2024-04-24T07:56:14.141 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T07:56:14.141 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_rishabh@teuthology --name rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} -- /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210/orig.config.yaml 2024-04-24T07:56:14.188 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T07:56:14.286 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210 branch: rishabh-main-apr17-a654945 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} email: null first_in_suite: false job_id: '7671210' kernel: flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-main-apr17-a654945 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Scrub error on dir - Metadata damage detected sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-main-apr17-a654945 sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 7954 sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 sleep_before_teardown: 0 subset: 364/512 suite: fs suite_branch: rishabh-main-apr17-a654945 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK20pgGhDUG25YTktz5xwXObPjKvwJGn16p16aQ/0z0+mcDD2fI1t/7FaYdn3Uc0KXME5axBn4v+GA4VH3lFBc8= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO0oFFrIDtsdmDzD5a+fGpNSauqw+ofxvqcq6IPtvi5gvN4205o5tQ8kZoJSeqobbF0W03bDbUz/qnqVr6oBns4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_07:32:23 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T07:56:14.287 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa; will attempt to use it 2024-04-24T07:56:14.287 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks 2024-04-24T07:56:14.287 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T07:56:14.289 INFO:teuthology.task.internal:Checking packages... 2024-04-24T07:56:14.312 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1f81fb397ae98da7563d451a78c61574c8f4e6e0' 2024-04-24T07:56:14.313 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T07:56:14.313 INFO:teuthology.packaging:ref: None 2024-04-24T07:56:14.313 INFO:teuthology.packaging:tag: None 2024-04-24T07:56:14.313 INFO:teuthology.packaging:branch: rishabh-main-apr17-a654945 2024-04-24T07:56:14.313 INFO:teuthology.packaging:sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:56:14.313 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-main-apr17-a654945 2024-04-24T07:56:14.457 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3116.g1f81fb39 2024-04-24T07:56:14.459 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T07:56:14.466 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T07:56:14.466 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T07:56:14.472 INFO:teuthology.task.internal:Saving configuration 2024-04-24T07:56:14.489 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T07:56:14.496 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T07:56:14.520 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:51:07.883968', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK20pgGhDUG25YTktz5xwXObPjKvwJGn16p16aQ/0z0+mcDD2fI1t/7FaYdn3Uc0KXME5axBn4v+GA4VH3lFBc8='} 2024-04-24T07:56:14.543 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:51:07.885359', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO0oFFrIDtsdmDzD5a+fGpNSauqw+ofxvqcq6IPtvi5gvN4205o5tQ8kZoJSeqobbF0W03bDbUz/qnqVr6oBns4='} 2024-04-24T07:56:14.543 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T07:56:14.550 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-24T07:56:14.550 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-24T07:56:14.550 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T07:56:14.618 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3393d44160>, signals=[15]) 2024-04-24T07:56:14.619 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T07:56:14.625 INFO:teuthology.task.internal:Opening connections... 2024-04-24T07:56:14.625 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T07:56:14.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:56:14.704 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T07:56:14.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:56:14.777 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T07:56:14.783 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-04-24T07:56:14.801 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-04-24T07:56:14.801 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-04-24T07:56:14.857 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-04-24T07:56:14.857 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-04-24T07:56:14.857 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-04-24T07:56:14.857 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-04-24T07:56:14.857 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-04-24T07:56:14.857 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-04-24T07:56:14.857 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T07:56:14.857 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-04-24T07:56:14.857 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-04-24T07:56:14.857 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T07:56:14.858 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-04-24T07:56:14.858 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T07:56:14.858 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T07:56:14.858 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T07:56:14.858 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-04-24T07:56:14.883 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-04-24T07:56:14.901 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-04-24T07:56:14.901 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-04-24T07:56:14.958 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-04-24T07:56:14.958 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-04-24T07:56:14.958 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T07:56:14.959 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T07:56:14.960 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-04-24T07:56:14.974 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T07:56:14.983 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T07:56:14.991 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T07:56:14.992 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-04-24T07:56:14.994 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-04-24T07:56:15.014 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T07:56:15.021 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T07:56:15.021 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-04-24T07:56:15.050 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-04-24T07:56:15.144 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T07:56:15.194 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T07:56:15.209 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-24T07:56:15.209 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}}, timeout 300 2024-04-24T07:56:15.209 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T07:56:15.210 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T07:56:15.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:56:15.225 DEBUG:teuthology.task.kernel:sha1 for mon.a is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:56:15.225 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-24T07:56:15.225 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-04-24T07:56:15.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:56:15.226 DEBUG:teuthology.task.kernel:sha1 for mon.b is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:56:15.227 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-24T07:56:15.227 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-04-24T07:56:15.281 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-437.el9.x86_64 2024-04-24T07:56:15.281 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:56:15.281 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-24T07:56:15.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:56:15.285 INFO:teuthology.orchestra.run.smithi121.stdout:5.14.0-437.el9.x86_64 2024-04-24T07:56:15.285 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:56:15.285 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-24T07:56:15.285 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:56:15.417 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum upgrade -y linux-firmware 2024-04-24T07:56:15.430 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum upgrade -y linux-firmware 2024-04-24T07:56:18.311 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 6.2 MB/s | 8.0 MB 00:01 2024-04-24T07:56:20.820 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-24T07:56:26.010 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 915 kB/s | 8.0 MB 00:09 2024-04-24T07:56:26.495 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.2 MB 00:02 2024-04-24T07:56:28.229 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-24T07:56:29.893 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 6.7 MB/s | 19 MB 00:02 2024-04-24T07:56:29.922 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-24T07:56:34.794 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.2 MB 00:01 2024-04-24T07:56:34.988 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-24T07:56:36.444 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-04-24T07:56:36.890 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-24T07:56:36.891 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-04-24T07:56:36.891 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-24T07:56:36.999 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on mon.b... 2024-04-24T07:56:36.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:56:37.151 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-24T07:56:37.151 DEBUG:teuthology.orchestra.run.smithi121:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-24T07:56:37.153 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi121.front.sepia.ceph.com to complete... 2024-04-24T07:56:38.564 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-04-24T07:56:43.687 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-24T07:56:45.638 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-24T07:56:45.639 INFO:teuthology.orchestra.run.smithi069.stdout:Nothing to do. 2024-04-24T07:56:45.639 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-24T07:56:45.733 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on mon.a... 2024-04-24T07:56:45.734 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:56:45.879 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-24T07:56:45.879 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-24T07:56:45.882 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi069.front.sepia.ceph.com to complete... 2024-04-24T07:56:50.531 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24 07:56:50 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-24T07:56:50.531 INFO:teuthology.orchestra.run.smithi121.stderr:FINISHED --2024-04-24 07:56:50-- 2024-04-24T07:56:50.532 INFO:teuthology.orchestra.run.smithi121.stderr:Total wall clock time: 13s 2024-04-24T07:56:50.532 INFO:teuthology.orchestra.run.smithi121.stderr:Downloaded: 1 files, 341M in 13s (26.1 MB/s) 2024-04-24T07:56:50.535 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on mon.b... 2024-04-24T07:56:50.535 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-24T07:56:52.616 INFO:teuthology.orchestra.run.smithi121.stdout:Verifying... ######################################## 2024-04-24T07:56:52.789 INFO:teuthology.orchestra.run.smithi121.stdout:Preparing... ######################################## 2024-04-24T07:56:52.894 INFO:teuthology.orchestra.run.smithi121.stdout:Updating / installing... 2024-04-24T07:57:00.778 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-24T07:57:02.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24 07:57:02 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-24T07:57:02.184 INFO:teuthology.orchestra.run.smithi069.stderr:FINISHED --2024-04-24 07:57:02-- 2024-04-24T07:57:02.184 INFO:teuthology.orchestra.run.smithi069.stderr:Total wall clock time: 16s 2024-04-24T07:57:02.184 INFO:teuthology.orchestra.run.smithi069.stderr:Downloaded: 1 files, 341M in 16s (21.7 MB/s) 2024-04-24T07:57:02.186 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on mon.a... 2024-04-24T07:57:02.187 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-24T07:57:04.277 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying... ######################################## 2024-04-24T07:57:04.537 INFO:teuthology.orchestra.run.smithi069.stdout:Preparing... ######################################## 2024-04-24T07:57:04.664 INFO:teuthology.orchestra.run.smithi069.stdout:Updating / installing... 2024-04-24T07:57:12.696 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-24T07:57:40.368 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-24T07:57:40.369 DEBUG:teuthology.orchestra.run.smithi121:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-24T07:57:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-24T07:57:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-24T07:57:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-24T07:57:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-24T07:57:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-24T07:57:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-24T07:57:40.443 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-24T07:57:40.444 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-24T07:57:40.445 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-24T07:57:40.446 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-24T07:57:40.446 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-24T07:57:40.446 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-24T07:57:40.446 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-24T07:57:40.446 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-24T07:57:40.446 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-24T07:57:40.446 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-24T07:57:40.446 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-24T07:57:40.446 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-24T07:57:40.446 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-24T07:57:40.447 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-24T07:57:40.448 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-24T07:57:40.449 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-24T07:57:40.450 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-24T07:57:40.451 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-24T07:57:40.452 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-24T07:57:40.452 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-24T07:57:40.452 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-24T07:57:40.452 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-24T07:57:40.452 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-24T07:57:40.452 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-24T07:57:40.452 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-24T07:57:40.452 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-24T07:57:40.452 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-24T07:57:40.453 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-24T07:57:40.453 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-24T07:57:40.453 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-24T07:57:40.453 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-24T07:57:40.453 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-24T07:57:40.453 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-24T07:57:40.453 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-24T07:57:40.453 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-24T07:57:40.453 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-24T07:57:40.454 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-24T07:57:40.454 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-24T07:57:40.454 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-24T07:57:40.454 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-24T07:57:40.454 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-24T07:57:40.454 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-24T07:57:40.454 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-24T07:57:40.454 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-24T07:57:40.455 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-24T07:57:40.455 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-24T07:57:40.455 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-24T07:57:40.455 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-24T07:57:40.455 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-24T07:57:40.455 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-24T07:57:40.455 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-24T07:57:40.455 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-24T07:57:40.455 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-24T07:57:40.456 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-24T07:57:40.456 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-24T07:57:40.456 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-24T07:57:40.456 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-24T07:57:40.456 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-24T07:57:40.456 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-24T07:57:40.456 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-24T07:57:40.456 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-24T07:57:40.456 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-24T07:57:40.456 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-24T07:57:40.457 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-24T07:57:40.457 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-24T07:57:40.457 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-24T07:57:40.457 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-24T07:57:40.457 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-24T07:57:40.457 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-24T07:57:40.457 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-24T07:57:40.457 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-24T07:57:40.458 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-24T07:57:40.458 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-24T07:57:40.458 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-24T07:57:40.458 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-24T07:57:40.458 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-24T07:57:40.458 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-24T07:57:40.458 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-24T07:57:40.458 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-24T07:57:40.458 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-24T07:57:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-24T07:57:40.460 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-24T07:57:40.461 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-24T07:57:40.462 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-24T07:57:40.463 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-24T07:57:40.464 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-24T07:57:40.465 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-24T07:57:40.466 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-24T07:57:40.467 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-24T07:57:40.468 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-24T07:57:40.469 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-24T07:57:40.470 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-24T07:57:40.471 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-24T07:57:40.472 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-24T07:57:40.473 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-24T07:57:40.474 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-24T07:57:40.475 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-24T07:57:40.476 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-24T07:57:40.477 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-24T07:57:40.478 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-24T07:57:40.478 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-24T07:57:40.478 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-24T07:57:40.478 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-24T07:57:40.478 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-24T07:57:40.478 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-24T07:57:40.478 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-24T07:57:40.478 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-24T07:57:40.478 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-24T07:57:40.479 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-24T07:57:40.480 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-24T07:57:40.481 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-24T07:57:40.482 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-24T07:57:40.482 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-24T07:57:40.482 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-24T07:57:40.482 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-24T07:57:40.482 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-24T07:57:40.482 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-24T07:57:40.482 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-24T07:57:40.482 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-24T07:57:40.482 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-24T07:57:40.482 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-24T07:57:40.483 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-24T07:57:40.484 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-24T07:57:40.485 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-24T07:57:40.486 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-24T07:57:40.486 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-24T07:57:40.486 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-24T07:57:40.486 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-24T07:57:40.486 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-24T07:57:40.486 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-24T07:57:40.486 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-24T07:57:40.486 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-24T07:57:40.486 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-24T07:57:40.487 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-24T07:57:40.488 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-24T07:57:40.488 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-24T07:57:40.488 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-24T07:57:40.488 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-24T07:57:40.488 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-24T07:57:40.488 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-24T07:57:40.488 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-24T07:57:40.488 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-24T07:57:40.488 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-24T07:57:40.488 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-24T07:57:40.489 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-24T07:57:40.489 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-24T07:57:40.489 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-24T07:57:40.489 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-24T07:57:40.489 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-24T07:57:40.489 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-24T07:57:40.489 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-24T07:57:40.489 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-24T07:57:40.489 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-24T07:57:40.489 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-24T07:57:40.490 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-24T07:57:40.490 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-24T07:57:40.490 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-24T07:57:40.490 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-24T07:57:40.490 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-24T07:57:40.490 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-24T07:57:40.490 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-24T07:57:40.490 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-24T07:57:40.490 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-24T07:57:40.491 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-24T07:57:40.492 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-24T07:57:40.493 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-24T07:57:40.494 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-24T07:57:40.495 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-24T07:57:40.496 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-24T07:57:40.497 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-24T07:57:40.498 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-24T07:57:40.499 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-24T07:57:40.500 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-24T07:57:40.501 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-24T07:57:40.502 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-24T07:57:40.503 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-24T07:57:40.504 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-24T07:57:40.505 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-24T07:57:40.506 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-24T07:57:40.507 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-24T07:57:40.508 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-24T07:57:40.509 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-24T07:57:40.510 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-24T07:57:40.511 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-24T07:57:40.512 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-24T07:57:40.512 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-24T07:57:40.512 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-24T07:57:40.512 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-24T07:57:40.512 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-24T07:57:40.512 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-24T07:57:40.512 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-24T07:57:40.512 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-24T07:57:40.512 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-24T07:57:40.512 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-24T07:57:40.513 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-24T07:57:40.514 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-24T07:57:40.515 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-24T07:57:40.516 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-24T07:57:40.517 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-24T07:57:40.518 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-24T07:57:40.519 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-24T07:57:40.520 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-24T07:57:40.521 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-24T07:57:40.522 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-24T07:57:40.523 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-24T07:57:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-24T07:57:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-24T07:57:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-24T07:57:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-24T07:57:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-24T07:57:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-24T07:57:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-24T07:57:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-24T07:57:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-24T07:57:40.524 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-24T07:57:40.525 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-24T07:57:40.526 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-24T07:57:40.527 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-24T07:57:40.528 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-24T07:57:40.528 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-24T07:57:40.528 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-24T07:57:40.528 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-24T07:57:40.528 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-24T07:57:40.528 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-24T07:57:40.528 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-24T07:57:40.529 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-24T07:57:40.529 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-24T07:57:40.529 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-24T07:57:40.529 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-24T07:57:40.529 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-24T07:57:40.529 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-24T07:57:40.530 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-24T07:57:40.530 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-24T07:57:40.530 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-24T07:57:40.530 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-24T07:57:40.530 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-24T07:57:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-24T07:57:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-24T07:57:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-24T07:57:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-24T07:57:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-24T07:57:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-24T07:57:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-24T07:57:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-24T07:57:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-24T07:57:40.531 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-24T07:57:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-24T07:57:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-24T07:57:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-24T07:57:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-24T07:57:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-24T07:57:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-24T07:57:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-24T07:57:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-24T07:57:40.532 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-24T07:57:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-24T07:57:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-24T07:57:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-24T07:57:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-24T07:57:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-24T07:57:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-24T07:57:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-24T07:57:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-24T07:57:40.533 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-24T07:57:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-24T07:57:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-24T07:57:40.534 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-24T07:57:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-24T07:57:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-24T07:57:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-24T07:57:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-24T07:57:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-24T07:57:40.535 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-24T07:57:40.536 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-24T07:57:40.536 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-24T07:57:40.536 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-24T07:57:40.536 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-24T07:57:40.536 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-24T07:57:40.536 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-24T07:57:40.536 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-24T07:57:40.537 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-24T07:57:40.537 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-24T07:57:40.537 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-24T07:57:40.537 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-24T07:57:40.537 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-24T07:57:40.537 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-24T07:57:40.537 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-24T07:57:40.537 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-24T07:57:40.538 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-24T07:57:40.538 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-24T07:57:40.538 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-24T07:57:40.538 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-24T07:57:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-24T07:57:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-24T07:57:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-24T07:57:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-24T07:57:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-24T07:57:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-24T07:57:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-24T07:57:40.539 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-24T07:57:40.540 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-24T07:57:40.541 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-24T07:57:40.542 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-24T07:57:40.542 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-24T07:57:40.542 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-24T07:57:40.543 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-24T07:57:40.545 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-24T07:57:40.546 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-24T07:57:40.546 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-24T07:57:40.547 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-24T07:57:40.547 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-24T07:57:40.547 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-24T07:57:40.547 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-24T07:57:40.547 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-24T07:57:40.548 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-24T07:57:40.548 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-24T07:57:40.549 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-24T07:57:40.549 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-24T07:57:40.549 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-24T07:57:40.549 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-24T07:57:40.549 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-24T07:57:40.549 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-24T07:57:40.550 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-24T07:57:40.550 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-24T07:57:40.550 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-24T07:57:40.550 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-24T07:57:40.550 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-24T07:57:40.551 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-24T07:57:40.551 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-24T07:57:40.552 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-24T07:57:40.553 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-24T07:57:40.553 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-24T07:57:40.554 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-24T07:57:40.554 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-24T07:57:40.554 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-24T07:57:40.554 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-24T07:57:40.555 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-24T07:57:40.555 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-24T07:57:40.555 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-24T07:57:40.556 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-24T07:57:40.556 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-24T07:57:40.557 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-24T07:57:40.557 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-24T07:57:40.558 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-24T07:57:40.558 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-24T07:57:40.558 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-24T07:57:40.559 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-24T07:57:40.559 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-24T07:57:40.559 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-24T07:57:40.559 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-24T07:57:40.560 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-24T07:57:40.560 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-24T07:57:40.560 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-24T07:57:40.560 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-24T07:57:40.560 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-24T07:57:40.560 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-24T07:57:40.560 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-24T07:57:40.560 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-24T07:57:40.560 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-24T07:57:40.560 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-24T07:57:40.561 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-24T07:57:40.562 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-24T07:57:40.563 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-24T07:57:40.564 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-24T07:57:40.565 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-24T07:57:40.566 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-24T07:57:40.567 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-24T07:57:40.568 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-24T07:57:40.569 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-24T07:57:40.570 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-24T07:57:40.571 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-24T07:57:40.572 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-24T07:57:40.573 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-24T07:57:40.574 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-24T07:57:40.575 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-24T07:57:40.576 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-24T07:57:40.577 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-24T07:57:40.578 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-24T07:57:40.579 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-24T07:57:40.579 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-24T07:57:40.579 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-24T07:57:40.579 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-24T07:57:40.579 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-24T07:57:40.579 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-24T07:57:40.579 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-24T07:57:40.579 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-24T07:57:40.579 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-24T07:57:40.579 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-24T07:57:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-24T07:57:40.581 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-24T07:57:40.582 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-24T07:57:40.582 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-24T07:57:40.582 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-24T07:57:40.582 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-24T07:57:40.582 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-24T07:57:40.582 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-24T07:57:40.582 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-24T07:57:40.582 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-24T07:57:40.582 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-24T07:57:40.583 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-24T07:57:40.584 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-24T07:57:40.585 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-24T07:57:40.586 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-24T07:57:40.587 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-24T07:57:40.588 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-24T07:57:40.589 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-24T07:57:40.590 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-24T07:57:40.591 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-24T07:57:40.592 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-24T07:57:40.593 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-24T07:57:40.593 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-24T07:57:40.593 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-24T07:57:40.593 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-24T07:57:40.593 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-24T07:57:40.593 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-24T07:57:40.594 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-24T07:57:40.594 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-24T07:57:40.594 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-24T07:57:40.594 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-24T07:57:40.594 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-24T07:57:40.594 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-24T07:57:40.594 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-24T07:57:40.594 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-24T07:57:40.594 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-24T07:57:40.594 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-24T07:57:40.595 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-24T07:57:40.596 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-24T07:57:40.597 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-24T07:57:40.598 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-24T07:57:40.599 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-24T07:57:40.600 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-24T07:57:40.600 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-24T07:57:40.600 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-24T07:57:40.600 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-24T07:57:40.600 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-24T07:57:40.600 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-24T07:57:40.600 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-24T07:57:40.600 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-24T07:57:40.600 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-24T07:57:40.600 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-24T07:57:40.601 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-24T07:57:40.602 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-24T07:57:40.603 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-24T07:57:40.604 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-24T07:57:40.605 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-24T07:57:40.606 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-24T07:57:40.607 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-24T07:57:40.608 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-24T07:57:40.609 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-24T07:57:40.610 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-24T07:57:40.611 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-24T07:57:40.612 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-24T07:57:40.613 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-24T07:57:40.613 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-24T07:57:40.613 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-24T07:57:40.613 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-24T07:57:40.613 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-24T07:57:40.613 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-24T07:57:40.613 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-24T07:57:40.613 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-24T07:57:40.613 INFO:teuthology.orchestra.run.smithi121.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-24T07:57:40.617 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:40.618 DEBUG:teuthology.orchestra.run.smithi121:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-24T07:57:40.642 INFO:teuthology.orchestra.run.smithi121.stdout:postinstall scriptlet (using /bin/sh): 2024-04-24T07:57:40.643 INFO:teuthology.orchestra.run.smithi121.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-24T07:57:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-24T07:57:40.643 INFO:teuthology.orchestra.run.smithi121.stdout:fi 2024-04-24T07:57:40.643 INFO:teuthology.orchestra.run.smithi121.stdout:for file in vmlinuz System.map config; do 2024-04-24T07:57:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-24T07:57:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:57:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: fi 2024-04-24T07:57:40.643 INFO:teuthology.orchestra.run.smithi121.stdout:done 2024-04-24T07:57:40.643 INFO:teuthology.orchestra.run.smithi121.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-24T07:57:40.643 INFO:teuthology.orchestra.run.smithi121.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-24T07:57:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:57:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:57:40.644 INFO:teuthology.orchestra.run.smithi121.stdout:fi 2024-04-24T07:57:40.644 INFO:teuthology.orchestra.run.smithi121.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-24T07:57:40.644 INFO:teuthology.orchestra.run.smithi121.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:40.644 INFO:teuthology.orchestra.run.smithi121.stdout:fi 2024-04-24T07:57:40.644 INFO:teuthology.orchestra.run.smithi121.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-24T07:57:40.644 INFO:teuthology.orchestra.run.smithi121.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-24T07:57:40.644 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:40.644 INFO:teuthology.orchestra.run.smithi121.stdout:fi 2024-04-24T07:57:40.645 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-04-24T07:57:40.724 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-04-24T07:57:40.724 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-04-24T07:57:40.724 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.06 2024-04-24T07:57:40.724 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 68.el9 2024-04-24T07:57:40.724 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-04-24T07:57:40.724 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-24T07:57:40.724 INFO:teuthology.orchestra.run.smithi121.stdout:Group : Unspecified 2024-04-24T07:57:40.724 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 8264254 2024-04-24T07:57:40.724 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : builder@centos.org 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T07:57:40.725 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-04-24T07:57:40.726 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-24T07:57:40.726 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:40.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-24T07:57:41.306 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-04-24T07:57:42.684 INFO:teuthology.orchestra.run.smithi121.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-24T07:57:42.709 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-04-24T07:57:42.711 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-04-24T07:57:42.725 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.J0oaZ0zLZb 2024-04-24T07:57:42.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.J0oaZ0zLZb 2024-04-24T07:57:42.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.J0oaZ0zLZb 2024-04-24T07:57:42.932 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.J0oaZ0zLZb is 6KB 2024-04-24T07:57:42.980 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.J0oaZ0zLZb 2024-04-24T07:57:42.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-04-24T07:57:43.055 INFO:teuthology.orchestra.run.smithi121.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-24T07:57:43.055 INFO:teuthology.orchestra.run.smithi121.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-24T07:57:43.055 INFO:teuthology.orchestra.run.smithi121.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-24T07:57:43.056 INFO:teuthology.orchestra.run.smithi121.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-24T07:57:43.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:43.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-04-24T07:57:51.715 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-24T07:57:51.715 DEBUG:teuthology.orchestra.run.smithi069:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-24T07:57:51.800 INFO:teuthology.orchestra.run.smithi069.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:51.800 INFO:teuthology.orchestra.run.smithi069.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:51.800 INFO:teuthology.orchestra.run.smithi069.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:51.800 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-24T07:57:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-24T07:57:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-24T07:57:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-24T07:57:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-24T07:57:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-24T07:57:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-24T07:57:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-24T07:57:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-24T07:57:51.801 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-24T07:57:51.802 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-24T07:57:51.802 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-24T07:57:51.802 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-24T07:57:51.802 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-24T07:57:51.802 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-24T07:57:51.802 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-24T07:57:51.802 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-24T07:57:51.802 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-24T07:57:51.802 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-24T07:57:51.802 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-24T07:57:51.803 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-24T07:57:51.804 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-24T07:57:51.805 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-24T07:57:51.806 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-24T07:57:51.807 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-24T07:57:51.808 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-24T07:57:51.809 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-24T07:57:51.810 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-24T07:57:51.811 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-24T07:57:51.812 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-24T07:57:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-24T07:57:51.814 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-24T07:57:51.815 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-24T07:57:51.816 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-24T07:57:51.817 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-24T07:57:51.818 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-24T07:57:51.819 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-24T07:57:51.820 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-24T07:57:51.821 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-24T07:57:51.822 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-24T07:57:51.823 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-24T07:57:51.824 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-24T07:57:51.825 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-24T07:57:51.826 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-24T07:57:51.827 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-24T07:57:51.828 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-24T07:57:51.829 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-24T07:57:51.830 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-24T07:57:51.831 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-24T07:57:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-24T07:57:51.833 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-24T07:57:51.834 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-24T07:57:51.835 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-24T07:57:51.836 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-24T07:57:51.837 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-24T07:57:51.838 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-24T07:57:51.839 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-24T07:57:51.840 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-24T07:57:51.841 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-24T07:57:51.842 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-24T07:57:51.843 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-24T07:57:51.844 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-24T07:57:51.845 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-24T07:57:51.846 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-24T07:57:51.847 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-24T07:57:51.848 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-24T07:57:51.849 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-24T07:57:51.850 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-24T07:57:51.851 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-24T07:57:51.852 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-24T07:57:51.853 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-24T07:57:51.854 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-24T07:57:51.855 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-24T07:57:51.856 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-24T07:57:51.857 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-24T07:57:51.858 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-24T07:57:51.859 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-24T07:57:51.860 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-24T07:57:51.861 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-24T07:57:51.862 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-24T07:57:51.863 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-24T07:57:51.864 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-24T07:57:51.865 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-24T07:57:51.866 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-24T07:57:51.867 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-24T07:57:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-24T07:57:51.869 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-24T07:57:51.870 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-24T07:57:51.871 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-24T07:57:51.872 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-24T07:57:51.873 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-24T07:57:51.874 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-24T07:57:51.875 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-24T07:57:51.876 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-24T07:57:51.877 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-24T07:57:51.878 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-24T07:57:51.879 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-24T07:57:51.880 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-24T07:57:51.881 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-24T07:57:51.882 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-24T07:57:51.883 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-24T07:57:51.884 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-24T07:57:51.885 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-24T07:57:51.886 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-24T07:57:51.887 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-24T07:57:51.888 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-24T07:57:51.889 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-24T07:57:51.890 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-24T07:57:51.891 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-24T07:57:51.892 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-24T07:57:51.893 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-24T07:57:51.894 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-24T07:57:51.895 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-24T07:57:51.896 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-24T07:57:51.897 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-24T07:57:51.898 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-24T07:57:51.899 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-24T07:57:51.900 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-24T07:57:51.900 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-24T07:57:51.900 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-24T07:57:51.900 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-24T07:57:51.900 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-24T07:57:51.900 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-24T07:57:51.900 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-24T07:57:51.900 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-24T07:57:51.900 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-24T07:57:51.900 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-24T07:57:51.901 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-24T07:57:51.902 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-24T07:57:51.903 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-24T07:57:51.904 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-24T07:57:51.905 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-24T07:57:51.906 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-24T07:57:51.907 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-24T07:57:51.908 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-24T07:57:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-24T07:57:51.910 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-24T07:57:51.911 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-24T07:57:51.912 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-24T07:57:51.913 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-24T07:57:51.914 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-24T07:57:51.915 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-24T07:57:51.916 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-24T07:57:51.917 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-24T07:57:51.918 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-24T07:57:51.919 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-24T07:57:51.920 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-24T07:57:51.921 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-24T07:57:51.922 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-24T07:57:51.923 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-24T07:57:51.924 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-24T07:57:51.925 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-24T07:57:51.926 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-24T07:57:51.927 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-24T07:57:51.928 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-24T07:57:51.929 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-24T07:57:51.930 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-24T07:57:51.931 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-24T07:57:51.932 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-24T07:57:51.933 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-24T07:57:51.934 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-24T07:57:51.935 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-24T07:57:51.935 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-24T07:57:51.935 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-24T07:57:51.935 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-24T07:57:51.935 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-24T07:57:51.935 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-24T07:57:51.935 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-24T07:57:51.935 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-24T07:57:51.935 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-24T07:57:51.935 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-24T07:57:51.936 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-24T07:57:51.937 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-24T07:57:51.937 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-24T07:57:51.937 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-24T07:57:51.937 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-24T07:57:51.937 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-24T07:57:51.937 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-24T07:57:51.937 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-24T07:57:51.937 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-24T07:57:51.937 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-24T07:57:51.937 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-24T07:57:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-24T07:57:51.939 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-24T07:57:51.939 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-24T07:57:51.939 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-24T07:57:51.939 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-24T07:57:51.939 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-24T07:57:51.939 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-24T07:57:51.939 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-24T07:57:51.939 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-24T07:57:51.939 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-24T07:57:51.939 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-24T07:57:51.940 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-24T07:57:51.941 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-24T07:57:51.942 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-24T07:57:51.943 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-24T07:57:51.944 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-24T07:57:51.945 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-24T07:57:51.946 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-24T07:57:51.947 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-24T07:57:51.948 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-24T07:57:51.949 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-24T07:57:51.950 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-24T07:57:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-24T07:57:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-24T07:57:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-24T07:57:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-24T07:57:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-24T07:57:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-24T07:57:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-24T07:57:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-24T07:57:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-24T07:57:51.951 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-24T07:57:51.953 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-24T07:57:51.954 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-24T07:57:51.955 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-24T07:57:51.956 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-24T07:57:51.957 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-24T07:57:51.958 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-24T07:57:51.959 INFO:teuthology.orchestra.run.smithi069.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-24T07:57:51.962 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:51.962 DEBUG:teuthology.orchestra.run.smithi069:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-24T07:57:51.985 INFO:teuthology.orchestra.run.smithi069.stdout:postinstall scriptlet (using /bin/sh): 2024-04-24T07:57:51.985 INFO:teuthology.orchestra.run.smithi069.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-24T07:57:51.985 INFO:teuthology.orchestra.run.smithi069.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-24T07:57:51.985 INFO:teuthology.orchestra.run.smithi069.stdout:fi 2024-04-24T07:57:51.985 INFO:teuthology.orchestra.run.smithi069.stdout:for file in vmlinuz System.map config; do 2024-04-24T07:57:51.985 INFO:teuthology.orchestra.run.smithi069.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-24T07:57:51.985 INFO:teuthology.orchestra.run.smithi069.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:57:51.985 INFO:teuthology.orchestra.run.smithi069.stdout: fi 2024-04-24T07:57:51.985 INFO:teuthology.orchestra.run.smithi069.stdout:done 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout:fi 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout:fi 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:51.986 INFO:teuthology.orchestra.run.smithi069.stdout:fi 2024-04-24T07:57:51.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2024-04-24T07:57:52.059 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2024-04-24T07:57:52.060 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2024-04-24T07:57:52.060 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2024-04-24T07:57:52.060 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2024-04-24T07:57:52.060 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2024-04-24T07:57:52.060 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-24T07:57:52.061 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2024-04-24T07:57:52.061 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2024-04-24T07:57:52.061 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2024-04-24T07:57:52.061 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-24T07:57:52.061 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-24T07:57:52.061 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-24T07:57:52.062 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-24T07:57:52.062 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2024-04-24T07:57:52.062 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2024-04-24T07:57:52.062 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-24T07:57:52.062 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2024-04-24T07:57:52.062 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2024-04-24T07:57:52.063 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T07:57:52.063 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-24T07:57:52.063 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-24T07:57:52.063 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-24T07:57:52.063 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2024-04-24T07:57:52.064 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T07:57:52.064 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2024-04-24T07:57:52.065 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-24T07:57:52.066 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:52.066 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-24T07:57:52.619 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-04-24T07:57:54.000 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-24T07:57:54.031 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-04-24T07:57:54.033 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-04-24T07:57:54.064 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.NVy63Ng1dn 2024-04-24T07:57:54.065 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NVy63Ng1dn 2024-04-24T07:57:54.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.NVy63Ng1dn 2024-04-24T07:57:54.261 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.NVy63Ng1dn is 6KB 2024-04-24T07:57:54.309 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.NVy63Ng1dn 2024-04-24T07:57:54.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2024-04-24T07:57:54.383 INFO:teuthology.orchestra.run.smithi069.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-24T07:57:54.384 INFO:teuthology.orchestra.run.smithi069.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-24T07:57:54.384 INFO:teuthology.orchestra.run.smithi069.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-24T07:57:54.384 INFO:teuthology.orchestra.run.smithi069.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-24T07:57:54.385 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-24T07:57:54.531 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-04-24T07:58:13.230 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-24T07:58:13.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-24T07:58:13.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:58:24.542 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-24T07:58:24.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-24T07:58:24.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:58:31.710 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-04-24T07:58:40.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-24T07:58:40.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:58:43.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-04-24T07:58:55.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-24T07:58:55.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:59:24.569 DEBUG:teuthology.orchestra.remote:timed out 2024-04-24T07:59:27.591 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-04-24T07:59:28.261 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-24T07:59:28.261 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.9.0-rc3-g806ec36b2112"... 2024-04-24T07:59:28.262 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-04-24T07:59:28.305 INFO:teuthology.orchestra.run.smithi121.stdout:6.9.0-rc3-g806ec36b2112 2024-04-24T07:59:28.305 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-24T07:59:28.305 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T07:59:28.305 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-24T07:59:29.305 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-24T07:59:29.306 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T07:59:29.440 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-04-24T07:59:29.487 DEBUG:teuthology.parallel:result is None 2024-04-24T07:59:33.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-24T07:59:33.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:59:41.330 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-24T07:59:41.910 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-24T07:59:41.910 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.9.0-rc3-g806ec36b2112"... 2024-04-24T07:59:41.910 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-04-24T07:59:41.958 INFO:teuthology.orchestra.run.smithi069.stdout:6.9.0-rc3-g806ec36b2112 2024-04-24T07:59:41.959 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-24T07:59:41.959 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T07:59:41.959 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-24T07:59:42.959 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-24T07:59:42.960 DEBUG:teuthology.orchestra.run.smithi069:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T07:59:43.080 INFO:teuthology.orchestra.run.smithi069.stdout:ttyS1 2024-04-24T07:59:43.137 DEBUG:teuthology.parallel:result is None 2024-04-24T07:59:43.137 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T07:59:43.145 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T07:59:43.145 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T07:59:43.150 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T07:59:43.232 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T07:59:43.240 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T07:59:43.288 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T07:59:43.288 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T07:59:43.295 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T07:59:43.395 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T07:59:43.402 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T07:59:43.402 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T07:59:43.410 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T07:59:43.497 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T07:59:43.522 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T07:59:43.540 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T07:59:43.565 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T07:59:43.568 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T07:59:43.576 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T07:59:43.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T07:59:43.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T07:59:43.661 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T07:59:43.685 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T07:59:43.685 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T07:59:43.690 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T07:59:43.733 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T07:59:43.843 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T07:59:43.933 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T07:59:44.002 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T07:59:44.078 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T07:59:44.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T07:59:44.138 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T07:59:44.218 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T07:59:44.308 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T07:59:44.381 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T07:59:44.451 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T07:59:44.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T07:59:44.511 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-04-24T07:59:44.518 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-04-24T07:59:44.606 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T07:59:44.641 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T07:59:45.023 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T07:59:45.032 INFO:teuthology.task.internal:Starting timer... 2024-04-24T07:59:45.033 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T07:59:45.127 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T07:59:45.149 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-24T07:59:45.150 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-04-24T07:59:45.252 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-04-24T07:59:45.258 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-04-24T07:59:45.527 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-04-24T07:59:45.532 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T07:59:45.532 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-04-24T07:59:45.578 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-04-24T07:59:45.579 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-04-24T07:59:45.627 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-04-24T07:59:45.628 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi069.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2024-04-24T07:59:45.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T07:59:45.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:59:45.698 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T07:59:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:59:45.781 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T07:59:45.781 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-04-24T07:59:45.853 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-04-24T07:59:45.932 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T07:59:45.942 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T07:59:45.959 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T07:59:45.960 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T08:04:53.921 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2024-04-24T08:04:53.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-24T08:04:53.923 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T08:04:54.029 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-04-24T08:04:54.181 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-04-24T08:04:54.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-24T08:04:54.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T08:04:54.290 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-04-24T08:04:54.443 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-04-24T08:04:54.443 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T08:04:54.455 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T08:04:54.456 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T08:04:54.456 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:04:54.464 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T08:04:54.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:04:54.548 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T08:04:54.551 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T08:04:54.587 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T08:04:54.592 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T08:04:54.660 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-04-24T08:04:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-04-24T08:04:54.693 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-04-24T08:04:54.696 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-04-24T08:04:54.734 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T08:04:54.735 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T08:04:54.774 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T08:04:54.775 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T08:04:54.905 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:04:54.913 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:04:54.913 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-04-24T08:04:54.913 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:04:54.913 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:04:54.913 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:04:54.913 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:04:54.920 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:04:54.930 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:04:54.930 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-04-24T08:04:54.930 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:04:54.930 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:04:54.930 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:04:54.930 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:04:54.931 INFO:teuthology.run_tasks:Running task install... 2024-04-24T08:04:54.946 DEBUG:teuthology.task.install:project ceph 2024-04-24T08:04:54.946 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0'}} 2024-04-24T08:04:54.946 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0'} 2024-04-24T08:04:54.946 INFO:teuthology.task.install:Using flavor: default 2024-04-24T08:04:54.953 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T08:04:54.953 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-24T08:04:54.954 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'tag': None, 'wait_for_package': False} 2024-04-24T08:04:54.954 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:04:54.956 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'tag': None, 'wait_for_package': False} 2024-04-24T08:04:54.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:04:55.107 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/ 2024-04-24T08:04:55.107 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3116.g1f81fb39 2024-04-24T08:04:55.117 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/ 2024-04-24T08:04:55.118 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3116.g1f81fb39 2024-04-24T08:04:55.223 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T08:04:55.223 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:04:55.223 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T08:04:55.286 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T08:04:55.286 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:04:55.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T08:04:55.313 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T08:04:55.313 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1f81fb397ae98da7563d451a78c61574c8f4e6e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T08:04:55.377 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T08:04:55.377 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1f81fb397ae98da7563d451a78c61574c8f4e6e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T08:04:55.443 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T08:04:55.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T08:04:55.549 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T08:04:55.617 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T08:04:55.623 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2024-04-24T08:04:55.627 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2024-04-24T08:04:55.698 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-04-24T08:04:55.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-04-24T08:04:55.945 INFO:teuthology.orchestra.run.smithi069.stdout:48 files removed 2024-04-24T08:04:55.984 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T08:04:56.018 INFO:teuthology.orchestra.run.smithi121.stdout:48 files removed 2024-04-24T08:04:56.056 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T08:04:56.853 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 191 kB/s | 83 kB 00:00 2024-04-24T08:04:57.009 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 156 kB/s | 83 kB 00:00 2024-04-24T08:04:57.593 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-04-24T08:04:58.519 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-04-24T08:04:58.528 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 7.3 kB/s | 12 kB 00:01 2024-04-24T08:04:59.037 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-04-24T08:04:59.375 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-24T08:04:59.685 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-24T08:05:21.839 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 917 kB/s | 19 MB 00:21 2024-04-24T08:05:22.124 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 918 kB/s | 19 MB 00:21 2024-04-24T08:05:26.692 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.2 MB 00:00 2024-04-24T08:05:27.047 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.2 MB 00:01 2024-04-24T08:05:28.510 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-24T08:05:28.873 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-24T08:05:29.806 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-24T08:05:30.783 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-24T08:05:35.745 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-04-24T08:05:36.657 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-24T08:05:38.127 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T08:05:38.129 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T08:05:38.146 INFO:teuthology.orchestra.run.smithi069.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T08:05:38.147 INFO:teuthology.orchestra.run.smithi069.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T08:05:38.149 INFO:teuthology.orchestra.run.smithi069.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T08:05:38.150 INFO:teuthology.orchestra.run.smithi069.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T08:05:38.152 INFO:teuthology.orchestra.run.smithi069.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T08:05:38.155 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T08:05:38.156 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T08:05:38.225 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-04-24T08:05:38.245 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-04-24T08:05:38.245 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 6.6 k 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 890 k 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 145 k 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 1.5 M 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 142 k 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 3.5 M 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 7.4 M 2024-04-24T08:05:38.246 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 49 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 11 M 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 49 M 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 273 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 762 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 225 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-shell x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 27 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 18 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 32 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 746 k 2024-04-24T08:05:38.247 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 127 k 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 161 k 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 320 k 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 298 k 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 99 k 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 85 k 2024-04-24T08:05:38.248 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 170 k 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.4 M 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 21 M 2024-04-24T08:05:38.249 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 34 k 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 2.4 M 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 249 k 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 4.7 M 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 17 M 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 16 k 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 25 k 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T08:05:38.250 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 163 k 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 501 k 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T08:05:38.251 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T08:05:38.252 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T08:05:38.253 INFO:teuthology.orchestra.run.smithi069.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T08:05:38.253 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T08:05:38.253 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T08:05:38.253 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T08:05:38.253 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 45 k 2024-04-24T08:05:38.253 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 135 k 2024-04-24T08:05:38.253 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T08:05:38.253 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T08:05:38.253 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T08:05:38.253 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T08:05:38.254 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T08:05:38.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T08:05:38.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T08:05:38.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T08:05:38.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T08:05:38.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T08:05:38.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T08:05:38.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T08:05:38.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T08:05:38.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T08:05:38.255 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T08:05:38.256 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T08:05:38.257 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T08:05:38.258 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout:Installing weak dependencies: 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout:Install 148 Packages 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 4 Packages 2024-04-24T08:05:38.259 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:05:38.260 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 222 M 2024-04-24T08:05:38.260 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-04-24T08:05:39.047 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T08:05:39.049 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T08:05:39.069 INFO:teuthology.orchestra.run.smithi121.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T08:05:39.071 INFO:teuthology.orchestra.run.smithi121.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T08:05:39.072 INFO:teuthology.orchestra.run.smithi121.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T08:05:39.073 INFO:teuthology.orchestra.run.smithi121.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T08:05:39.076 INFO:teuthology.orchestra.run.smithi121.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T08:05:39.079 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T08:05:39.080 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T08:05:39.152 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-04-24T08:05:39.174 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-04-24T08:05:39.174 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-04-24T08:05:39.174 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-04-24T08:05:39.174 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-04-24T08:05:39.174 INFO:teuthology.orchestra.run.smithi121.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T08:05:39.174 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 6.6 k 2024-04-24T08:05:39.174 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T08:05:39.174 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 890 k 2024-04-24T08:05:39.174 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 145 k 2024-04-24T08:05:39.174 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 1.5 M 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 142 k 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 3.5 M 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 7.4 M 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 49 k 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 11 M 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 49 M 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 273 k 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 762 k 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 225 k 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-shell x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 27 k 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 18 k 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T08:05:39.175 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 32 k 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 746 k 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 127 k 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 161 k 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 320 k 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 298 k 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 99 k 2024-04-24T08:05:39.176 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 85 k 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 170 k 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.4 M 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T08:05:39.177 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 21 M 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 34 k 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 2.4 M 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 249 k 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 4.7 M 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 17 M 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 16 k 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 25 k 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T08:05:39.178 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 163 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 501 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T08:05:39.179 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T08:05:39.180 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 45 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 135 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T08:05:39.181 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T08:05:39.182 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T08:05:39.183 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T08:05:39.184 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-04-24T08:05:39.185 INFO:teuthology.orchestra.run.smithi121.stdout:======================================================================================= 2024-04-24T08:05:39.186 INFO:teuthology.orchestra.run.smithi121.stdout:Install 148 Packages 2024-04-24T08:05:39.186 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 4 Packages 2024-04-24T08:05:39.186 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:05:39.186 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 222 M 2024-04-24T08:05:39.186 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-04-24T08:05:39.313 INFO:teuthology.orchestra.run.smithi069.stdout:(1/152): ceph-19.0.0-3116.g1f81fb39.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-04-24T08:05:39.505 INFO:teuthology.orchestra.run.smithi069.stdout:(2/152): ceph-fuse-19.0.0-3116.g1f81fb39.el9.x8 4.6 MB/s | 890 kB 00:00 2024-04-24T08:05:39.563 INFO:teuthology.orchestra.run.smithi069.stdout:(3/152): ceph-immutable-object-cache-19.0.0-311 2.4 MB/s | 145 kB 00:00 2024-04-24T08:05:39.889 INFO:teuthology.orchestra.run.smithi069.stdout:(4/152): ceph-mds-19.0.0-3116.g1f81fb39.el9.x86 7.3 MB/s | 2.4 MB 00:00 2024-04-24T08:05:40.117 INFO:teuthology.orchestra.run.smithi121.stdout:(1/152): ceph-19.0.0-3116.g1f81fb39.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-24T08:05:40.122 INFO:teuthology.orchestra.run.smithi069.stdout:(5/152): ceph-mgr-19.0.0-3116.g1f81fb39.el9.x86 6.4 MB/s | 1.5 MB 00:00 2024-04-24T08:05:40.231 INFO:teuthology.orchestra.run.smithi069.stdout:(6/152): ceph-base-19.0.0-3116.g1f81fb39.el9.x8 4.5 MB/s | 5.4 MB 00:01 2024-04-24T08:05:40.790 INFO:teuthology.orchestra.run.smithi069.stdout:(7/152): ceph-mon-19.0.0-3116.g1f81fb39.el9.x86 7.1 MB/s | 4.7 MB 00:00 2024-04-24T08:05:41.150 INFO:teuthology.orchestra.run.smithi121.stdout:(2/152): ceph-fuse-19.0.0-3116.g1f81fb39.el9.x8 862 kB/s | 890 kB 00:01 2024-04-24T08:05:41.268 INFO:teuthology.orchestra.run.smithi121.stdout:(3/152): ceph-immutable-object-cache-19.0.0-311 1.2 MB/s | 145 kB 00:00 2024-04-24T08:05:41.282 INFO:teuthology.orchestra.run.smithi069.stdout:(8/152): ceph-common-19.0.0-3116.g1f81fb39.el9. 9.5 MB/s | 21 MB 00:02 2024-04-24T08:05:41.348 INFO:teuthology.orchestra.run.smithi069.stdout:(9/152): ceph-selinux-19.0.0-3116.g1f81fb39.el9 387 kB/s | 25 kB 00:00 2024-04-24T08:05:42.232 INFO:teuthology.orchestra.run.smithi069.stdout:(10/152): ceph-radosgw-19.0.0-3116.g1f81fb39.el 7.4 MB/s | 11 MB 00:01 2024-04-24T08:05:42.324 INFO:teuthology.orchestra.run.smithi069.stdout:(11/152): cephfs-mirror-19.0.0-3116.g1f81fb39.e 2.4 MB/s | 225 kB 00:00 2024-04-24T08:05:42.399 INFO:teuthology.orchestra.run.smithi069.stdout:(12/152): cephfs-shell-19.0.0-3116.g1f81fb39.el 365 kB/s | 27 kB 00:00 2024-04-24T08:05:42.466 INFO:teuthology.orchestra.run.smithi069.stdout:(13/152): libcephfs-devel-19.0.0-3116.g1f81fb39 484 kB/s | 32 kB 00:00 2024-04-24T08:05:42.534 INFO:teuthology.orchestra.run.smithi121.stdout:(4/152): ceph-base-19.0.0-3116.g1f81fb39.el9.x8 1.9 MB/s | 5.4 MB 00:02 2024-04-24T08:05:42.635 INFO:teuthology.orchestra.run.smithi121.stdout:(5/152): ceph-mds-19.0.0-3116.g1f81fb39.el9.x86 1.7 MB/s | 2.4 MB 00:01 2024-04-24T08:05:42.641 INFO:teuthology.orchestra.run.smithi069.stdout:(14/152): libcephfs2-19.0.0-3116.g1f81fb39.el9. 4.2 MB/s | 746 kB 00:00 2024-04-24T08:05:42.742 INFO:teuthology.orchestra.run.smithi069.stdout:(15/152): libcephsqlite-19.0.0-3116.g1f81fb39.e 1.6 MB/s | 163 kB 00:00 2024-04-24T08:05:42.833 INFO:teuthology.orchestra.run.smithi069.stdout:(16/152): librados-devel-19.0.0-3116.g1f81fb39. 1.4 MB/s | 127 kB 00:00 2024-04-24T08:05:43.001 INFO:teuthology.orchestra.run.smithi069.stdout:(17/152): libradosstriper1-19.0.0-3116.g1f81fb3 2.9 MB/s | 501 kB 00:00 2024-04-24T08:05:43.293 INFO:teuthology.orchestra.run.smithi121.stdout:(6/152): ceph-mgr-19.0.0-3116.g1f81fb39.el9.x86 2.0 MB/s | 1.5 MB 00:00 2024-04-24T08:05:43.476 INFO:teuthology.orchestra.run.smithi069.stdout:(18/152): ceph-osd-19.0.0-3116.g1f81fb39.el9.x8 5.2 MB/s | 17 MB 00:03 2024-04-24T08:05:43.576 INFO:teuthology.orchestra.run.smithi069.stdout:(19/152): python3-ceph-argparse-19.0.0-3116.g1f 457 kB/s | 45 kB 00:00 2024-04-24T08:05:43.693 INFO:teuthology.orchestra.run.smithi069.stdout:(20/152): python3-ceph-common-19.0.0-3116.g1f81 1.1 MB/s | 135 kB 00:00 2024-04-24T08:05:43.810 INFO:teuthology.orchestra.run.smithi069.stdout:(21/152): python3-cephfs-19.0.0-3116.g1f81fb39. 1.4 MB/s | 161 kB 00:00 2024-04-24T08:05:43.952 INFO:teuthology.orchestra.run.smithi069.stdout:(22/152): librgw2-19.0.0-3116.g1f81fb39.el9.x86 5.7 MB/s | 5.4 MB 00:00 2024-04-24T08:05:43.993 INFO:teuthology.orchestra.run.smithi069.stdout:(23/152): python3-rados-19.0.0-3116.g1f81fb39.e 1.7 MB/s | 320 kB 00:00 2024-04-24T08:05:44.044 INFO:teuthology.orchestra.run.smithi069.stdout:(24/152): python3-rbd-19.0.0-3116.g1f81fb39.el9 3.2 MB/s | 298 kB 00:00 2024-04-24T08:05:44.111 INFO:teuthology.orchestra.run.smithi069.stdout:(25/152): python3-rgw-19.0.0-3116.g1f81fb39.el9 853 kB/s | 99 kB 00:00 2024-04-24T08:05:44.136 INFO:teuthology.orchestra.run.smithi069.stdout:(26/152): rbd-fuse-19.0.0-3116.g1f81fb39.el9.x8 934 kB/s | 85 kB 00:00 2024-04-24T08:05:44.795 INFO:teuthology.orchestra.run.smithi069.stdout:(27/152): rbd-mirror-19.0.0-3116.g1f81fb39.el9. 4.5 MB/s | 3.1 MB 00:00 2024-04-24T08:05:44.828 INFO:teuthology.orchestra.run.smithi069.stdout:(28/152): rbd-nbd-19.0.0-3116.g1f81fb39.el9.x86 246 kB/s | 170 kB 00:00 2024-04-24T08:05:44.887 INFO:teuthology.orchestra.run.smithi069.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3116.g 376 kB/s | 34 kB 00:00 2024-04-24T08:05:45.129 INFO:teuthology.orchestra.run.smithi069.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3116.g1f81fb3 475 kB/s | 142 kB 00:00 2024-04-24T08:05:45.629 INFO:teuthology.orchestra.run.smithi069.stdout:(31/152): ceph-test-19.0.0-3116.g1f81fb39.el9.x 11 MB/s | 49 MB 00:04 2024-04-24T08:05:45.746 INFO:teuthology.orchestra.run.smithi069.stdout:(32/152): ceph-mgr-dashboard-19.0.0-3116.g1f81f 4.1 MB/s | 3.5 MB 00:00 2024-04-24T08:05:45.780 INFO:teuthology.orchestra.run.smithi069.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3116.g1f 1.6 MB/s | 249 kB 00:00 2024-04-24T08:05:45.826 INFO:teuthology.orchestra.run.smithi121.stdout:(7/152): ceph-mon-19.0.0-3116.g1f81fb39.el9.x86 1.5 MB/s | 4.7 MB 00:03 2024-04-24T08:05:45.838 INFO:teuthology.orchestra.run.smithi069.stdout:(34/152): ceph-mgr-rook-19.0.0-3116.g1f81fb39.e 542 kB/s | 49 kB 00:00 2024-04-24T08:05:45.864 INFO:teuthology.orchestra.run.smithi069.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3116.g1 198 kB/s | 16 kB 00:00 2024-04-24T08:05:45.930 INFO:teuthology.orchestra.run.smithi069.stdout:(36/152): ceph-volume-19.0.0-3116.g1f81fb39.el9 2.9 MB/s | 273 kB 00:00 2024-04-24T08:05:46.023 INFO:teuthology.orchestra.run.smithi069.stdout:(37/152): cephadm-19.0.0-3116.g1f81fb39.el9.noa 4.7 MB/s | 762 kB 00:00 2024-04-24T08:05:46.048 INFO:teuthology.orchestra.run.smithi069.stdout:(38/152): cephfs-top-19.0.0-3116.g1f81fb39.el9. 157 kB/s | 18 kB 00:00 2024-04-24T08:05:46.148 INFO:teuthology.orchestra.run.smithi069.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 647 kB/s | 80 kB 00:00 2024-04-24T08:05:46.232 INFO:teuthology.orchestra.run.smithi069.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-04-24T08:05:46.257 INFO:teuthology.orchestra.run.smithi069.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-04-24T08:05:46.290 INFO:teuthology.orchestra.run.smithi069.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.3 MB/s | 253 kB 00:00 2024-04-24T08:05:46.341 INFO:teuthology.orchestra.run.smithi069.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-24T08:05:46.399 INFO:teuthology.orchestra.run.smithi069.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 8.7 MB/s | 1.2 MB 00:00 2024-04-24T08:05:46.448 INFO:teuthology.orchestra.run.smithi069.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-24T08:05:46.483 INFO:teuthology.orchestra.run.smithi069.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-04-24T08:05:46.517 INFO:teuthology.orchestra.run.smithi069.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.1 MB/s | 215 kB 00:00 2024-04-24T08:05:46.826 INFO:teuthology.orchestra.run.smithi069.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 2.0 MB/s | 689 kB 00:00 2024-04-24T08:05:46.893 INFO:teuthology.orchestra.run.smithi069.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-24T08:05:46.943 INFO:teuthology.orchestra.run.smithi069.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 438 kB/s | 21 kB 00:00 2024-04-24T08:05:47.010 INFO:teuthology.orchestra.run.smithi121.stdout:(8/152): ceph-common-19.0.0-3116.g1f81fb39.el9. 2.9 MB/s | 21 MB 00:07 2024-04-24T08:05:47.035 INFO:teuthology.orchestra.run.smithi069.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 3.4 MB/s | 313 kB 00:00 2024-04-24T08:05:47.060 INFO:teuthology.orchestra.run.smithi121.stdout:(9/152): ceph-selinux-19.0.0-3116.g1f81fb39.el9 515 kB/s | 25 kB 00:00 2024-04-24T08:05:47.110 INFO:teuthology.orchestra.run.smithi069.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 439 kB/s | 33 kB 00:00 2024-04-24T08:05:47.152 INFO:teuthology.orchestra.run.smithi069.stdout:(53/152): bison-3.7.4-5.el9.x86_64.rpm 1.5 MB/s | 943 kB 00:00 2024-04-24T08:05:47.219 INFO:teuthology.orchestra.run.smithi069.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 279 kB/s | 18 kB 00:00 2024-04-24T08:05:47.269 INFO:teuthology.orchestra.run.smithi069.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 227 kB/s | 11 kB 00:00 2024-04-24T08:05:47.361 INFO:teuthology.orchestra.run.smithi069.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-04-24T08:05:47.412 INFO:teuthology.orchestra.run.smithi069.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 922 kB/s | 45 kB 00:00 2024-04-24T08:05:47.487 INFO:teuthology.orchestra.run.smithi069.stdout:(58/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 8.0 MB/s | 3.0 MB 00:00 2024-04-24T08:05:47.595 INFO:teuthology.orchestra.run.smithi069.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 5.3 MB/s | 582 kB 00:00 2024-04-24T08:05:47.638 INFO:teuthology.orchestra.run.smithi069.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.9 MB/s | 662 kB 00:00 2024-04-24T08:05:47.663 INFO:teuthology.orchestra.run.smithi069.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 385 kB/s | 25 kB 00:00 2024-04-24T08:05:47.688 INFO:teuthology.orchestra.run.smithi069.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 960 kB/s | 47 kB 00:00 2024-04-24T08:05:47.710 INFO:teuthology.orchestra.run.smithi121.stdout:(10/152): ceph-osd-19.0.0-3116.g1f81fb39.el9.x8 3.8 MB/s | 17 MB 00:04 2024-04-24T08:05:47.755 INFO:teuthology.orchestra.run.smithi069.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-04-24T08:05:47.785 INFO:teuthology.orchestra.run.smithi121.stdout:(11/152): cephfs-mirror-19.0.0-3116.g1f81fb39.e 3.0 MB/s | 225 kB 00:00 2024-04-24T08:05:47.835 INFO:teuthology.orchestra.run.smithi121.stdout:(12/152): cephfs-shell-19.0.0-3116.g1f81fb39.el 552 kB/s | 27 kB 00:00 2024-04-24T08:05:47.886 INFO:teuthology.orchestra.run.smithi121.stdout:(13/152): libcephfs-devel-19.0.0-3116.g1f81fb39 646 kB/s | 32 kB 00:00 2024-04-24T08:05:47.888 INFO:teuthology.orchestra.run.smithi069.stdout:(64/152): ceph-mgr-diskprediction-local-19.0.0- 2.7 MB/s | 7.4 MB 00:02 2024-04-24T08:05:47.914 INFO:teuthology.orchestra.run.smithi069.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 223 kB/s | 35 kB 00:00 2024-04-24T08:05:47.947 INFO:teuthology.orchestra.run.smithi069.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 730 kB/s | 188 kB 00:00 2024-04-24T08:05:48.011 INFO:teuthology.orchestra.run.smithi121.stdout:(14/152): libcephfs2-19.0.0-3116.g1f81fb39.el9. 5.9 MB/s | 746 kB 00:00 2024-04-24T08:05:48.069 INFO:teuthology.orchestra.run.smithi121.stdout:(15/152): libcephsqlite-19.0.0-3116.g1f81fb39.e 2.8 MB/s | 163 kB 00:00 2024-04-24T08:05:48.128 INFO:teuthology.orchestra.run.smithi121.stdout:(16/152): librados-devel-19.0.0-3116.g1f81fb39. 2.2 MB/s | 127 kB 00:00 2024-04-24T08:05:48.206 INFO:teuthology.orchestra.run.smithi069.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-04-24T08:05:48.228 INFO:teuthology.orchestra.run.smithi121.stdout:(17/152): libradosstriper1-19.0.0-3116.g1f81fb3 4.9 MB/s | 501 kB 00:00 2024-04-24T08:05:48.306 INFO:teuthology.orchestra.run.smithi069.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 11 MB/s | 4.7 MB 00:00 2024-04-24T08:05:48.380 INFO:teuthology.orchestra.run.smithi069.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 3.7 MB/s | 1.6 MB 00:00 2024-04-24T08:05:48.418 INFO:teuthology.orchestra.run.smithi069.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 133 kB 00:00 2024-04-24T08:05:48.465 INFO:teuthology.orchestra.run.smithi069.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 3.4 MB/s | 911 kB 00:00 2024-04-24T08:05:48.524 INFO:teuthology.orchestra.run.smithi069.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 9.4 MB/s | 1.0 MB 00:00 2024-04-24T08:05:48.549 INFO:teuthology.orchestra.run.smithi069.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.0 MB/s | 87 kB 00:00 2024-04-24T08:05:48.608 INFO:teuthology.orchestra.run.smithi069.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.2 MB/s | 244 kB 00:00 2024-04-24T08:05:48.666 INFO:teuthology.orchestra.run.smithi069.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-24T08:05:48.717 INFO:teuthology.orchestra.run.smithi069.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 520 kB/s | 26 kB 00:00 2024-04-24T08:05:48.758 INFO:teuthology.orchestra.run.smithi069.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 473 kB/s | 19 kB 00:00 2024-04-24T08:05:48.876 INFO:teuthology.orchestra.run.smithi069.stdout:(78/152): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-04-24T08:05:48.909 INFO:teuthology.orchestra.run.smithi069.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-24T08:05:48.935 INFO:teuthology.orchestra.run.smithi069.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 602 kB/s | 35 kB 00:00 2024-04-24T08:05:49.027 INFO:teuthology.orchestra.run.smithi069.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.7 MB/s | 250 kB 00:00 2024-04-24T08:05:49.093 INFO:teuthology.orchestra.run.smithi069.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.3 MB/s | 222 kB 00:00 2024-04-24T08:05:49.145 INFO:teuthology.orchestra.run.smithi121.stdout:(18/152): librgw2-19.0.0-3116.g1f81fb39.el9.x86 5.9 MB/s | 5.4 MB 00:00 2024-04-24T08:05:49.195 INFO:teuthology.orchestra.run.smithi121.stdout:(19/152): python3-ceph-argparse-19.0.0-3116.g1f 924 kB/s | 45 kB 00:00 2024-04-24T08:05:49.210 INFO:teuthology.orchestra.run.smithi069.stdout:(83/152): postgresql-server-13.14-1.el9.x86_64. 7.0 MB/s | 5.8 MB 00:00 2024-04-24T08:05:49.236 INFO:teuthology.orchestra.run.smithi069.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 547 kB/s | 77 kB 00:00 2024-04-24T08:05:49.254 INFO:teuthology.orchestra.run.smithi121.stdout:(20/152): python3-ceph-common-19.0.0-3116.g1f81 2.3 MB/s | 135 kB 00:00 2024-04-24T08:05:49.261 INFO:teuthology.orchestra.run.smithi069.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 855 kB/s | 42 kB 00:00 2024-04-24T08:05:49.294 INFO:teuthology.orchestra.run.smithi069.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 269 kB 00:00 2024-04-24T08:05:49.313 INFO:teuthology.orchestra.run.smithi121.stdout:(21/152): python3-cephfs-19.0.0-3116.g1f81fb39. 2.7 MB/s | 161 kB 00:00 2024-04-24T08:05:49.328 INFO:teuthology.orchestra.run.smithi069.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-24T08:05:49.362 INFO:teuthology.orchestra.run.smithi069.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-04-24T08:05:49.387 INFO:teuthology.orchestra.run.smithi069.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 886 kB/s | 51 kB 00:00 2024-04-24T08:05:49.420 INFO:teuthology.orchestra.run.smithi069.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 932 kB/s | 54 kB 00:00 2024-04-24T08:05:49.438 INFO:teuthology.orchestra.run.smithi121.stdout:(22/152): python3-rados-19.0.0-3116.g1f81fb39.e 2.5 MB/s | 320 kB 00:00 2024-04-24T08:05:49.521 INFO:teuthology.orchestra.run.smithi069.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.3 MB/s | 5.1 MB 00:00 2024-04-24T08:05:49.546 INFO:teuthology.orchestra.run.smithi069.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 334 kB/s | 42 kB 00:00 2024-04-24T08:05:49.563 INFO:teuthology.orchestra.run.smithi121.stdout:(23/152): python3-rbd-19.0.0-3116.g1f81fb39.el9 2.3 MB/s | 298 kB 00:00 2024-04-24T08:05:49.572 INFO:teuthology.orchestra.run.smithi069.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 885 kB/s | 44 kB 00:00 2024-04-24T08:05:49.605 INFO:teuthology.orchestra.run.smithi069.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.8 MB/s | 221 kB 00:00 2024-04-24T08:05:49.622 INFO:teuthology.orchestra.run.smithi121.stdout:(24/152): python3-rgw-19.0.0-3116.g1f81fb39.el9 1.7 MB/s | 99 kB 00:00 2024-04-24T08:05:49.660 INFO:teuthology.orchestra.run.smithi069.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 416 kB/s | 36 kB 00:00 2024-04-24T08:05:49.672 INFO:teuthology.orchestra.run.smithi121.stdout:(25/152): rbd-fuse-19.0.0-3116.g1f81fb39.el9.x8 1.7 MB/s | 85 kB 00:00 2024-04-24T08:05:49.689 INFO:teuthology.orchestra.run.smithi069.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 569 kB/s | 47 kB 00:00 2024-04-24T08:05:49.731 INFO:teuthology.orchestra.run.smithi069.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.3 MB/s | 305 kB 00:00 2024-04-24T08:05:49.756 INFO:teuthology.orchestra.run.smithi069.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 880 kB/s | 58 kB 00:00 2024-04-24T08:05:49.782 INFO:teuthology.orchestra.run.smithi069.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-24T08:05:49.807 INFO:teuthology.orchestra.run.smithi069.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-04-24T08:05:49.832 INFO:teuthology.orchestra.run.smithi069.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 908 kB/s | 45 kB 00:00 2024-04-24T08:05:49.924 INFO:teuthology.orchestra.run.smithi069.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 193 kB/s | 22 kB 00:00 2024-04-24T08:05:49.983 INFO:teuthology.orchestra.run.smithi069.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 5.7 MB/s | 863 kB 00:00 2024-04-24T08:05:50.047 INFO:teuthology.orchestra.run.smithi121.stdout:(26/152): rbd-mirror-19.0.0-3116.g1f81fb39.el9. 8.3 MB/s | 3.1 MB 00:00 2024-04-24T08:05:50.106 INFO:teuthology.orchestra.run.smithi121.stdout:(27/152): rbd-nbd-19.0.0-3116.g1f81fb39.el9.x86 2.9 MB/s | 170 kB 00:00 2024-04-24T08:05:50.156 INFO:teuthology.orchestra.run.smithi121.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3116.g 703 kB/s | 34 kB 00:00 2024-04-24T08:05:50.215 INFO:teuthology.orchestra.run.smithi121.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3116.g1f81fb3 2.4 MB/s | 142 kB 00:00 2024-04-24T08:05:50.383 INFO:teuthology.orchestra.run.smithi069.stdout:(104/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-04-24T08:05:50.590 INFO:teuthology.orchestra.run.smithi121.stdout:(30/152): ceph-radosgw-19.0.0-3116.g1f81fb39.el 2.2 MB/s | 11 MB 00:04 2024-04-24T08:05:50.608 INFO:teuthology.orchestra.run.smithi069.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 803 kB/s | 548 kB 00:00 2024-04-24T08:05:50.650 INFO:teuthology.orchestra.run.smithi069.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-04-24T08:05:50.692 INFO:teuthology.orchestra.run.smithi069.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 436 kB/s | 309 kB 00:00 2024-04-24T08:05:50.740 INFO:teuthology.orchestra.run.smithi121.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3116.g1f81f 6.8 MB/s | 3.5 MB 00:00 2024-04-24T08:05:50.799 INFO:teuthology.orchestra.run.smithi121.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3116.g1f 4.3 MB/s | 249 kB 00:00 2024-04-24T08:05:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:(33/152): ceph-mgr-rook-19.0.0-3116.g1f81fb39.e 1.0 MB/s | 49 kB 00:00 2024-04-24T08:05:50.900 INFO:teuthology.orchestra.run.smithi121.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3116.g1 335 kB/s | 16 kB 00:00 2024-04-24T08:05:50.918 INFO:teuthology.orchestra.run.smithi069.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-04-24T08:05:50.983 INFO:teuthology.orchestra.run.smithi121.stdout:(35/152): ceph-volume-19.0.0-3116.g1f81fb39.el9 3.2 MB/s | 273 kB 00:00 2024-04-24T08:05:51.043 INFO:teuthology.orchestra.run.smithi069.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 139 kB/s | 49 kB 00:00 2024-04-24T08:05:51.091 INFO:teuthology.orchestra.run.smithi121.stdout:(36/152): cephadm-19.0.0-3116.g1f81fb39.el9.noa 7.0 MB/s | 762 kB 00:00 2024-04-24T08:05:51.268 INFO:teuthology.orchestra.run.smithi069.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 193 kB/s | 67 kB 00:00 2024-04-24T08:05:51.384 INFO:teuthology.orchestra.run.smithi121.stdout:(37/152): cephfs-top-19.0.0-3116.g1f81fb39.el9. 63 kB/s | 18 kB 00:00 2024-04-24T08:05:51.477 INFO:teuthology.orchestra.run.smithi069.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 349 kB/s | 151 kB 00:00 2024-04-24T08:05:51.493 INFO:teuthology.orchestra.run.smithi121.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 754 kB/s | 80 kB 00:00 2024-04-24T08:05:51.609 INFO:teuthology.orchestra.run.smithi121.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 6.8 MB/s | 803 kB 00:00 2024-04-24T08:05:51.660 INFO:teuthology.orchestra.run.smithi121.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 195 kB 00:00 2024-04-24T08:05:51.710 INFO:teuthology.orchestra.run.smithi121.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-24T08:05:51.744 INFO:teuthology.orchestra.run.smithi069.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-04-24T08:05:51.802 INFO:teuthology.orchestra.run.smithi121.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.2 MB 00:00 2024-04-24T08:05:51.844 INFO:teuthology.orchestra.run.smithi121.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.6 MB/s | 106 kB 00:00 2024-04-24T08:05:51.886 INFO:teuthology.orchestra.run.smithi121.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 3.3 MB/s | 135 kB 00:00 2024-04-24T08:05:51.929 INFO:teuthology.orchestra.run.smithi121.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 3.0 MB/s | 125 kB 00:00 2024-04-24T08:05:51.953 INFO:teuthology.orchestra.run.smithi069.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-04-24T08:05:51.978 INFO:teuthology.orchestra.run.smithi121.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.3 MB/s | 215 kB 00:00 2024-04-24T08:05:52.086 INFO:teuthology.orchestra.run.smithi069.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 127 kB/s | 43 kB 00:00 2024-04-24T08:05:52.144 INFO:teuthology.orchestra.run.smithi069.stdout:(115/152): python3-asyncssh-2.13.2-5.el9.noarch 822 kB/s | 548 kB 00:00 2024-04-24T08:05:52.211 INFO:teuthology.orchestra.run.smithi069.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 125 kB/s | 32 kB 00:00 2024-04-24T08:05:52.345 INFO:teuthology.orchestra.run.smithi069.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 58 kB/s | 15 kB 00:00 2024-04-24T08:05:52.595 INFO:teuthology.orchestra.run.smithi069.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 381 kB/s | 172 kB 00:00 2024-04-24T08:05:52.704 INFO:teuthology.orchestra.run.smithi121.stdout:(47/152): ceph-test-19.0.0-3116.g1f81fb39.el9.x 8.7 MB/s | 49 MB 00:05 2024-04-24T08:05:52.729 INFO:teuthology.orchestra.run.smithi069.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 694 kB/s | 358 kB 00:00 2024-04-24T08:05:52.745 INFO:teuthology.orchestra.run.smithi121.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 900 kB/s | 689 kB 00:00 2024-04-24T08:05:52.854 INFO:teuthology.orchestra.run.smithi069.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 137 kB/s | 35 kB 00:00 2024-04-24T08:05:52.887 INFO:teuthology.orchestra.run.smithi121.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 759 kB/s | 107 kB 00:00 2024-04-24T08:05:52.937 INFO:teuthology.orchestra.run.smithi069.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 512 kB/s | 303 kB 00:00 2024-04-24T08:05:53.054 INFO:teuthology.orchestra.run.smithi121.stdout:(50/152): ceph-mgr-diskprediction-local-19.0.0- 3.0 MB/s | 7.4 MB 00:02 2024-04-24T08:05:53.079 INFO:teuthology.orchestra.run.smithi121.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 112 kB/s | 21 kB 00:00 2024-04-24T08:05:53.138 INFO:teuthology.orchestra.run.smithi121.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 569 kB/s | 33 kB 00:00 2024-04-24T08:05:53.205 INFO:teuthology.orchestra.run.smithi121.stdout:(53/152): bison-3.7.4-5.el9.x86_64.rpm 1.8 MB/s | 943 kB 00:00 2024-04-24T08:05:53.229 INFO:teuthology.orchestra.run.smithi069.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 411 kB/s | 206 kB 00:00 2024-04-24T08:05:53.264 INFO:teuthology.orchestra.run.smithi121.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 316 kB/s | 18 kB 00:00 2024-04-24T08:05:53.322 INFO:teuthology.orchestra.run.smithi121.stdout:(55/152): flex-2.6.4-9.el9.x86_64.rpm 1.1 MB/s | 313 kB 00:00 2024-04-24T08:05:53.348 INFO:teuthology.orchestra.run.smithi121.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 135 kB/s | 11 kB 00:00 2024-04-24T08:05:53.363 INFO:teuthology.orchestra.run.smithi069.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 340 kB/s | 144 kB 00:00 2024-04-24T08:05:53.381 INFO:teuthology.orchestra.run.smithi121.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-24T08:05:53.406 INFO:teuthology.orchestra.run.smithi121.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 782 kB/s | 45 kB 00:00 2024-04-24T08:05:53.413 INFO:teuthology.orchestra.run.smithi069.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 58 kB/s | 11 kB 00:00 2024-04-24T08:05:53.540 INFO:teuthology.orchestra.run.smithi121.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-04-24T08:05:53.582 INFO:teuthology.orchestra.run.smithi121.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 3.3 MB/s | 582 kB 00:00 2024-04-24T08:05:53.599 INFO:teuthology.orchestra.run.smithi121.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 442 kB/s | 25 kB 00:00 2024-04-24T08:05:53.632 INFO:teuthology.orchestra.run.smithi121.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 962 kB/s | 47 kB 00:00 2024-04-24T08:05:53.638 INFO:teuthology.orchestra.run.smithi069.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.6 MB/s | 2.0 MB 00:00 2024-04-24T08:05:53.664 INFO:teuthology.orchestra.run.smithi069.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 59 kB/s | 18 kB 00:00 2024-04-24T08:05:53.683 INFO:teuthology.orchestra.run.smithi121.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-24T08:05:53.689 INFO:teuthology.orchestra.run.smithi069.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 84 kB/s | 23 kB 00:00 2024-04-24T08:05:53.741 INFO:teuthology.orchestra.run.smithi121.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-24T08:05:53.775 INFO:teuthology.orchestra.run.smithi121.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 385 kB/s | 35 kB 00:00 2024-04-24T08:05:53.842 INFO:teuthology.orchestra.run.smithi121.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.3 MB/s | 3.0 MB 00:00 2024-04-24T08:05:53.914 INFO:teuthology.orchestra.run.smithi069.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 71 kB/s | 19 kB 00:00 2024-04-24T08:05:53.939 INFO:teuthology.orchestra.run.smithi069.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 72 kB/s | 20 kB 00:00 2024-04-24T08:05:54.134 INFO:teuthology.orchestra.run.smithi121.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 5.5 MB/s | 1.6 MB 00:00 2024-04-24T08:05:54.265 INFO:teuthology.orchestra.run.smithi069.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 132 kB/s | 46 kB 00:00 2024-04-24T08:05:54.290 INFO:teuthology.orchestra.run.smithi069.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 225 kB/s | 79 kB 00:00 2024-04-24T08:05:54.301 INFO:teuthology.orchestra.run.smithi121.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 5.4 MB/s | 911 kB 00:00 2024-04-24T08:05:54.360 INFO:teuthology.orchestra.run.smithi121.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-24T08:05:54.465 INFO:teuthology.orchestra.run.smithi069.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 1.3 MB/s | 1.0 MB 00:00 2024-04-24T08:05:54.535 INFO:teuthology.orchestra.run.smithi121.stdout:(70/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 5.8 MB/s | 4.4 MB 00:00 2024-04-24T08:05:54.615 INFO:teuthology.orchestra.run.smithi069.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 165 kB/s | 58 kB 00:00 2024-04-24T08:05:54.710 INFO:teuthology.orchestra.run.smithi121.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 4.9 MB/s | 4.7 MB 00:00 2024-04-24T08:05:54.732 INFO:teuthology.orchestra.run.smithi069.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 62 kB/s | 16 kB 00:00 2024-04-24T08:05:54.769 INFO:teuthology.orchestra.run.smithi121.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-04-24T08:05:54.786 INFO:teuthology.orchestra.run.smithi121.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-04-24T08:05:54.808 INFO:teuthology.orchestra.run.smithi069.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 527 kB/s | 272 kB 00:00 2024-04-24T08:05:54.869 INFO:teuthology.orchestra.run.smithi121.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-24T08:05:54.953 INFO:teuthology.orchestra.run.smithi121.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-04-24T08:05:54.966 INFO:teuthology.orchestra.run.smithi069.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 257 kB/s | 90 kB 00:00 2024-04-24T08:05:55.000 INFO:teuthology.orchestra.run.smithi069.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 106 kB/s | 28 kB 00:00 2024-04-24T08:05:55.011 INFO:teuthology.orchestra.run.smithi121.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 444 kB/s | 26 kB 00:00 2024-04-24T08:05:55.070 INFO:teuthology.orchestra.run.smithi121.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 333 kB/s | 19 kB 00:00 2024-04-24T08:05:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 112 kB/s | 31 kB 00:00 2024-04-24T08:05:55.204 INFO:teuthology.orchestra.run.smithi121.stdout:(78/152): postgresql-server-13.14-1.el9.x86_64. 6.9 MB/s | 5.8 MB 00:00 2024-04-24T08:05:55.238 INFO:teuthology.orchestra.run.smithi121.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-04-24T08:05:55.263 INFO:teuthology.orchestra.run.smithi121.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 602 kB/s | 35 kB 00:00 2024-04-24T08:05:55.346 INFO:teuthology.orchestra.run.smithi121.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.0 MB/s | 250 kB 00:00 2024-04-24T08:05:55.350 INFO:teuthology.orchestra.run.smithi069.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 169 kB/s | 59 kB 00:00 2024-04-24T08:05:55.376 INFO:teuthology.orchestra.run.smithi069.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 123 kB/s | 36 kB 00:00 2024-04-24T08:05:55.430 INFO:teuthology.orchestra.run.smithi121.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-04-24T08:05:55.489 INFO:teuthology.orchestra.run.smithi121.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-24T08:05:55.492 INFO:teuthology.orchestra.run.smithi069.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 359 kB/s | 188 kB 00:00 2024-04-24T08:05:55.547 INFO:teuthology.orchestra.run.smithi121.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 739 kB/s | 42 kB 00:00 2024-04-24T08:05:55.664 INFO:teuthology.orchestra.run.smithi121.stdout:(85/152): python3-babel-2.9.1-2.el9.noarch.rpm 6.7 MB/s | 6.0 MB 00:00 2024-04-24T08:05:55.697 INFO:teuthology.orchestra.run.smithi121.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 1.8 MB/s | 269 kB 00:00 2024-04-24T08:05:55.709 INFO:teuthology.orchestra.run.smithi069.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 142 kB/s | 51 kB 00:00 2024-04-24T08:05:55.731 INFO:teuthology.orchestra.run.smithi121.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.4 MB/s | 159 kB 00:00 2024-04-24T08:05:55.843 INFO:teuthology.orchestra.run.smithi069.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 256 kB/s | 90 kB 00:00 2024-04-24T08:05:55.848 INFO:teuthology.orchestra.run.smithi121.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.4 MB/s | 5.1 MB 00:00 2024-04-24T08:05:55.882 INFO:teuthology.orchestra.run.smithi121.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.5 MB/s | 279 kB 00:00 2024-04-24T08:05:55.907 INFO:teuthology.orchestra.run.smithi121.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 292 kB/s | 51 kB 00:00 2024-04-24T08:05:55.918 INFO:teuthology.orchestra.run.smithi069.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 424 kB/s | 230 kB 00:00 2024-04-24T08:05:55.932 INFO:teuthology.orchestra.run.smithi121.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 648 kB/s | 54 kB 00:00 2024-04-24T08:05:55.966 INFO:teuthology.orchestra.run.smithi121.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 727 kB/s | 42 kB 00:00 2024-04-24T08:05:55.991 INFO:teuthology.orchestra.run.smithi121.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 761 kB/s | 44 kB 00:00 2024-04-24T08:05:56.033 INFO:teuthology.orchestra.run.smithi121.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.3 MB/s | 221 kB 00:00 2024-04-24T08:05:56.058 INFO:teuthology.orchestra.run.smithi121.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 540 kB/s | 36 kB 00:00 2024-04-24T08:05:56.084 INFO:teuthology.orchestra.run.smithi121.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 962 kB/s | 47 kB 00:00 2024-04-24T08:05:56.125 INFO:teuthology.orchestra.run.smithi121.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-24T08:05:56.144 INFO:teuthology.orchestra.run.smithi069.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 67 kB/s | 20 kB 00:00 2024-04-24T08:05:56.151 INFO:teuthology.orchestra.run.smithi121.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 873 kB/s | 58 kB 00:00 2024-04-24T08:05:56.210 INFO:teuthology.orchestra.run.smithi121.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-24T08:05:56.226 INFO:teuthology.orchestra.run.smithi121.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 854 kB/s | 64 kB 00:00 2024-04-24T08:05:56.260 INFO:teuthology.orchestra.run.smithi121.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 910 kB/s | 45 kB 00:00 2024-04-24T08:05:56.294 INFO:teuthology.orchestra.run.smithi069.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 732 kB/s | 427 kB 00:00 2024-04-24T08:05:56.352 INFO:teuthology.orchestra.run.smithi069.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 442 kB/s | 192 kB 00:00 2024-04-24T08:05:56.377 INFO:teuthology.orchestra.run.smithi121.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 149 kB/s | 22 kB 00:00 2024-04-24T08:05:56.603 INFO:teuthology.orchestra.run.smithi121.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-04-24T08:05:56.770 INFO:teuthology.orchestra.run.smithi121.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-04-24T08:05:56.786 INFO:teuthology.orchestra.run.smithi069.stdout:(148/152): librados2-19.0.0-3116.g1f81fb39.el9. 6.9 MB/s | 3.4 MB 00:00 2024-04-24T08:05:56.895 INFO:teuthology.orchestra.run.smithi069.stdout:(149/152): librbd1-19.0.0-3116.g1f81fb39.el9.x8 5.8 MB/s | 3.1 MB 00:00 2024-04-24T08:05:56.928 INFO:teuthology.orchestra.run.smithi121.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 125 kB/s | 19 kB 00:00 2024-04-24T08:05:56.961 INFO:teuthology.orchestra.run.smithi069.stdout:(150/152): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-04-24T08:05:57.062 INFO:teuthology.orchestra.run.smithi121.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 674 kB/s | 309 kB 00:00 2024-04-24T08:05:57.153 INFO:teuthology.orchestra.run.smithi069.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 3.7 MB/s | 1.3 MB 00:00 2024-04-24T08:05:57.154 INFO:teuthology.orchestra.run.smithi121.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 287 kB/s | 26 kB 00:00 2024-04-24T08:05:57.238 INFO:teuthology.orchestra.run.smithi069.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 7.0 MB/s | 2.4 MB 00:00 2024-04-24T08:05:57.243 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-04-24T08:05:57.244 INFO:teuthology.orchestra.run.smithi069.stdout:Total 12 MB/s | 222 MB 00:18 2024-04-24T08:05:57.254 INFO:teuthology.orchestra.run.smithi121.stdout:(108/152): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-24T08:05:57.279 INFO:teuthology.orchestra.run.smithi121.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 390 kB/s | 49 kB 00:00 2024-04-24T08:05:57.463 INFO:teuthology.orchestra.run.smithi121.stdout:(110/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-04-24T08:05:57.488 INFO:teuthology.orchestra.run.smithi121.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 289 kB/s | 67 kB 00:00 2024-04-24T08:05:57.513 INFO:teuthology.orchestra.run.smithi121.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 648 kB/s | 151 kB 00:00 2024-04-24T08:05:57.622 INFO:teuthology.orchestra.run.smithi121.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-04-24T08:05:57.648 INFO:teuthology.orchestra.run.smithi121.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 678 kB/s | 43 kB 00:00 2024-04-24T08:05:57.681 INFO:teuthology.orchestra.run.smithi121.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 581 kB/s | 32 kB 00:00 2024-04-24T08:05:57.706 INFO:teuthology.orchestra.run.smithi121.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 258 kB/s | 15 kB 00:00 2024-04-24T08:05:57.740 INFO:teuthology.orchestra.run.smithi121.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-24T08:05:57.807 INFO:teuthology.orchestra.run.smithi121.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-24T08:05:57.840 INFO:teuthology.orchestra.run.smithi121.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-04-24T08:05:57.874 INFO:teuthology.orchestra.run.smithi121.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 536 kB/s | 35 kB 00:00 2024-04-24T08:05:57.916 INFO:teuthology.orchestra.run.smithi121.stdout:(121/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 838 kB 00:00 2024-04-24T08:05:57.950 INFO:teuthology.orchestra.run.smithi121.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.9 MB/s | 206 kB 00:00 2024-04-24T08:05:57.983 INFO:teuthology.orchestra.run.smithi121.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-04-24T08:05:58.009 INFO:teuthology.orchestra.run.smithi121.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 186 kB/s | 11 kB 00:00 2024-04-24T08:05:58.051 INFO:teuthology.orchestra.run.smithi121.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 268 kB/s | 18 kB 00:00 2024-04-24T08:05:58.076 INFO:teuthology.orchestra.run.smithi121.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 350 kB/s | 23 kB 00:00 2024-04-24T08:05:58.134 INFO:teuthology.orchestra.run.smithi121.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 236 kB/s | 19 kB 00:00 2024-04-24T08:05:58.160 INFO:teuthology.orchestra.run.smithi121.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 237 kB/s | 20 kB 00:00 2024-04-24T08:05:58.218 INFO:teuthology.orchestra.run.smithi121.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 5.9 MB/s | 2.0 MB 00:00 2024-04-24T08:05:58.243 INFO:teuthology.orchestra.run.smithi121.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 559 kB/s | 46 kB 00:00 2024-04-24T08:05:58.333 INFO:teuthology.orchestra.run.smithi121.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 695 kB/s | 79 kB 00:00 2024-04-24T08:05:58.361 INFO:teuthology.orchestra.run.smithi121.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 497 kB/s | 58 kB 00:00 2024-04-24T08:05:58.411 INFO:teuthology.orchestra.run.smithi121.stdout:(133/152): python3-kubernetes-26.1.0-2.el9.noar 3.7 MB/s | 1.0 MB 00:00 2024-04-24T08:05:58.436 INFO:teuthology.orchestra.run.smithi121.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 220 kB/s | 16 kB 00:00 2024-04-24T08:05:58.470 INFO:teuthology.orchestra.run.smithi121.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-04-24T08:05:58.495 INFO:teuthology.orchestra.run.smithi121.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 488 kB/s | 28 kB 00:00 2024-04-24T08:05:58.520 INFO:teuthology.orchestra.run.smithi121.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 832 kB/s | 90 kB 00:00 2024-04-24T08:05:58.537 INFO:teuthology.orchestra.run.smithi121.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 468 kB/s | 31 kB 00:00 2024-04-24T08:05:58.571 INFO:teuthology.orchestra.run.smithi121.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-04-24T08:05:58.587 INFO:teuthology.orchestra.run.smithi121.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 893 kB/s | 59 kB 00:00 2024-04-24T08:05:58.605 INFO:teuthology.orchestra.run.smithi121.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 540 kB/s | 36 kB 00:00 2024-04-24T08:05:58.630 INFO:teuthology.orchestra.run.smithi121.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 881 kB/s | 51 kB 00:00 2024-04-24T08:05:58.664 INFO:teuthology.orchestra.run.smithi121.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-24T08:05:58.697 INFO:teuthology.orchestra.run.smithi121.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.5 MB/s | 230 kB 00:00 2024-04-24T08:05:58.731 INFO:teuthology.orchestra.run.smithi121.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-04-24T08:05:58.756 INFO:teuthology.orchestra.run.smithi121.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 219 kB/s | 20 kB 00:00 2024-04-24T08:05:58.798 INFO:teuthology.orchestra.run.smithi121.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-24T08:05:58.890 INFO:teuthology.orchestra.run.smithi121.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-24T08:05:59.082 INFO:teuthology.orchestra.run.smithi121.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 7.1 MB/s | 1.3 MB 00:00 2024-04-24T08:05:59.199 INFO:teuthology.orchestra.run.smithi121.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 21 MB/s | 2.4 MB 00:00 2024-04-24T08:05:59.332 INFO:teuthology.orchestra.run.smithi121.stdout:(151/152): librbd1-19.0.0-3116.g1f81fb39.el9.x8 5.9 MB/s | 3.1 MB 00:00 2024-04-24T08:05:59.916 INFO:teuthology.orchestra.run.smithi121.stdout:(152/152): librados2-19.0.0-3116.g1f81fb39.el9. 2.9 MB/s | 3.4 MB 00:01 2024-04-24T08:05:59.928 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-04-24T08:05:59.928 INFO:teuthology.orchestra.run.smithi121.stdout:Total 11 MB/s | 222 MB 00:20 2024-04-24T08:06:00.108 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-04-24T08:06:00.348 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-04-24T08:06:00.348 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-04-24T08:06:02.709 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-04-24T08:06:02.952 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-04-24T08:06:02.952 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-04-24T08:06:03.012 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-04-24T08:06:03.012 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-04-24T08:06:05.626 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-04-24T08:06:05.627 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-04-24T08:06:09.487 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-04-24T08:06:09.593 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T08:06:09.689 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T08:06:09.971 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T08:06:10.041 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T08:06:10.079 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T08:06:10.201 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T08:06:10.648 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T08:06:10.759 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:06:11.008 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:06:11.044 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:06:11.208 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:06:11.252 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T08:06:11.596 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T08:06:11.665 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T08:06:11.692 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:06:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:06:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T08:06:11.919 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-04-24T08:06:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T08:06:12.005 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T08:06:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T08:06:12.057 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T08:06:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T08:06:12.148 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T08:06:12.195 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T08:06:12.231 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T08:06:12.358 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T08:06:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T08:06:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T08:06:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T08:06:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T08:06:12.772 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T08:06:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T08:06:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T08:06:12.871 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:06:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T08:06:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T08:06:13.043 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:06:13.050 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T08:06:13.074 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:06:13.093 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T08:06:13.125 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T08:06:13.199 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:06:13.231 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T08:06:13.238 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T08:06:13.305 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T08:06:13.344 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T08:06:13.428 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T08:06:13.549 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T08:06:13.557 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T08:06:13.589 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 31/156 2024-04-24T08:06:13.620 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T08:06:13.645 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:06:13.645 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 32/156 2024-04-24T08:06:13.679 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 33/156 2024-04-24T08:06:13.793 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:06:13.796 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T08:06:13.829 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T08:06:13.859 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T08:06:13.916 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T08:06:13.946 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T08:06:13.966 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T08:06:14.003 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T08:06:14.140 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T08:06:14.209 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T08:06:14.303 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T08:06:14.364 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T08:06:14.591 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T08:06:14.712 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T08:06:14.756 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T08:06:14.814 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T08:06:14.891 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T08:06:14.917 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T08:06:14.960 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T08:06:14.997 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T08:06:15.021 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T08:06:15.035 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T08:06:15.073 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T08:06:15.081 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T08:06:15.112 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T08:06:15.117 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 43/156 2024-04-24T08:06:15.179 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T08:06:15.228 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T08:06:15.290 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T08:06:15.330 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T08:06:15.416 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T08:06:15.437 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T08:06:15.497 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T08:06:15.535 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T08:06:15.578 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T08:06:15.584 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 31/156 2024-04-24T08:06:15.635 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 32/156 2024-04-24T08:06:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 33/156 2024-04-24T08:06:15.670 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T08:06:15.712 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T08:06:15.751 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T08:06:15.782 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T08:06:15.820 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T08:06:15.852 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T08:06:15.859 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T08:06:15.920 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T08:06:15.951 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T08:06:16.004 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T08:06:16.119 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T08:06:16.196 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T08:06:16.201 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T08:06:16.257 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T08:06:16.295 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T08:06:16.344 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T08:06:16.480 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T08:06:16.528 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T08:06:16.578 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T08:06:16.618 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T08:06:16.651 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T08:06:16.840 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T08:06:16.877 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T08:06:16.978 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T08:06:17.013 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T08:06:17.019 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T08:06:17.058 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T08:06:17.074 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T08:06:17.099 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 43/156 2024-04-24T08:06:17.112 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T08:06:17.157 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T08:06:17.158 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T08:06:17.219 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T08:06:17.294 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T08:06:17.337 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T08:06:17.376 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T08:06:17.383 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T08:06:17.426 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T08:06:17.457 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T08:06:17.496 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T08:06:17.523 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T08:06:17.537 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T08:06:17.619 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T08:06:17.658 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T08:06:17.683 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T08:06:17.706 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T08:06:17.742 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T08:06:17.772 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T08:06:17.786 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T08:06:17.813 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T08:06:17.857 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T08:06:17.875 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T08:06:17.934 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T08:06:18.012 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T08:06:18.074 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T08:06:18.132 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T08:06:18.186 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T08:06:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T08:06:18.302 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T08:06:18.448 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T08:06:18.481 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T08:06:18.539 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T08:06:18.572 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T08:06:18.605 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T08:06:18.791 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T08:06:18.975 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T08:06:19.012 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T08:06:19.016 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T08:06:19.049 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T08:06:19.065 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T08:06:19.086 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T08:06:19.099 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T08:06:19.145 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T08:06:19.197 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T08:06:19.216 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T08:06:19.264 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T08:06:19.285 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T08:06:19.319 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T08:06:19.371 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T08:06:19.380 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T08:06:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T08:06:19.445 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T08:06:19.492 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T08:06:19.635 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T08:06:19.665 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T08:06:19.697 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T08:06:19.701 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T08:06:19.750 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T08:06:19.802 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T08:06:19.821 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T08:06:19.966 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T08:06:20.029 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T08:06:20.631 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T08:06:20.776 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T08:06:20.994 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T08:06:21.054 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T08:06:21.087 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T08:06:21.242 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T08:06:21.315 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T08:06:21.413 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T08:06:21.455 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T08:06:21.687 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T08:06:21.711 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T08:06:21.813 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T08:06:22.363 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T08:06:22.461 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T08:06:22.507 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T08:06:22.555 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T08:06:22.634 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T08:06:22.775 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T08:06:22.937 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T08:06:23.007 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T08:06:23.140 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T08:06:23.173 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 100/156 2024-04-24T08:06:23.239 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T08:06:23.821 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T08:06:23.852 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T08:06:23.986 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T08:06:24.012 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 104/156 2024-04-24T08:06:24.371 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T08:06:24.467 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T08:06:24.507 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T08:06:24.551 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T08:06:24.933 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T08:06:24.987 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T08:06:25.102 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T08:06:25.136 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 100/156 2024-04-24T08:06:25.201 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T08:06:25.745 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T08:06:25.782 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T08:06:25.873 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T08:06:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 104/156 2024-04-24T08:06:26.616 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:06:26.658 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:06:27.295 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:06:27.345 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T08:06:27.490 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T08:06:27.490 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T08:06:27.490 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T08:06:27.490 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:27.537 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T08:06:28.613 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:06:28.655 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:06:29.299 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T08:06:29.351 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T08:06:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T08:06:29.511 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T08:06:29.511 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T08:06:29.511 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:29.560 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T08:06:40.448 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T08:06:40.448 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2024-04-24T08:06:40.448 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2024-04-24T08:06:40.448 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2024-04-24T08:06:40.448 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2024-04-24T08:06:40.448 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2024-04-24T08:06:40.449 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2024-04-24T08:06:40.449 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2024-04-24T08:06:40.449 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:40.691 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T08:06:40.738 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T08:06:40.738 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:40.738 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T08:06:40.738 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:42.378 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T08:06:42.456 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T08:06:42.456 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:42.456 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T08:06:42.456 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T08:06:42.456 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T08:06:42.456 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:42.719 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T08:06:42.736 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T08:06:42.736 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-04-24T08:06:42.736 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-04-24T08:06:42.736 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-04-24T08:06:42.736 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-04-24T08:06:42.737 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-04-24T08:06:42.737 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-04-24T08:06:42.737 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-04-24T08:06:42.737 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:42.788 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T08:06:42.788 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:42.788 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T08:06:42.788 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T08:06:42.788 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T08:06:42.788 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:42.988 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T08:06:43.034 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T08:06:43.034 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:43.034 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T08:06:43.034 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:43.278 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T08:06:43.347 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T08:06:43.347 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:43.347 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T08:06:43.347 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T08:06:43.347 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T08:06:43.348 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:43.478 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T08:06:43.540 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T08:06:43.616 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T08:06:43.848 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T08:06:43.916 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T08:06:44.000 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T08:06:44.102 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T08:06:44.271 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T08:06:44.341 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T08:06:44.382 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T08:06:44.422 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T08:06:44.701 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T08:06:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T08:06:44.803 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:44.803 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T08:06:44.803 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T08:06:44.803 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T08:06:44.803 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:45.075 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T08:06:45.094 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T08:06:45.145 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T08:06:45.146 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T08:06:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T08:06:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T08:06:45.147 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T08:06:45.147 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:45.208 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 124/156 2024-04-24T08:06:45.319 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 125/156 2024-04-24T08:06:45.360 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T08:06:45.651 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T08:06:45.717 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T08:06:45.717 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:45.717 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T08:06:45.717 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T08:06:45.718 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T08:06:45.718 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:45.860 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T08:06:45.931 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T08:06:46.007 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T08:06:46.246 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T08:06:46.305 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T08:06:46.390 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T08:06:46.490 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T08:06:46.666 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T08:06:46.674 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T08:06:46.712 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T08:06:46.722 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T08:06:46.762 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T08:06:46.829 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T08:06:47.479 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T08:06:47.515 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T08:06:47.553 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 124/156 2024-04-24T08:06:47.645 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 125/156 2024-04-24T08:06:47.684 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T08:06:47.808 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T08:06:47.846 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T08:06:47.988 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T08:06:48.160 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 129/156 2024-04-24T08:06:48.189 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T08:06:48.254 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T08:06:48.257 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:48.257 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T08:06:48.257 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T08:06:48.257 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T08:06:48.257 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:48.307 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T08:06:48.344 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T08:06:48.448 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 132/156 2024-04-24T08:06:48.524 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 133/156 2024-04-24T08:06:48.977 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T08:06:49.019 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T08:06:49.520 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T08:06:49.551 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T08:06:49.619 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T08:06:49.619 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:49.619 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T08:06:49.619 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T08:06:49.619 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T08:06:49.619 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:49.668 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T08:06:49.744 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T08:06:49.744 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:49.744 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T08:06:49.745 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:49.811 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T08:06:49.882 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T08:06:49.882 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:49.882 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T08:06:49.883 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T08:06:49.883 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T08:06:49.883 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:50.113 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T08:06:50.144 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T08:06:50.274 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T08:06:50.288 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T08:06:50.365 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T08:06:50.365 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:50.365 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T08:06:50.365 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T08:06:50.365 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T08:06:50.365 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:06:50.467 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 129/156 2024-04-24T08:06:50.521 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T08:06:50.589 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T08:06:50.589 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:50.589 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T08:06:50.589 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T08:06:50.589 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T08:06:50.590 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:50.646 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T08:06:50.683 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T08:06:50.848 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 132/156 2024-04-24T08:06:50.917 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 133/156 2024-04-24T08:06:51.966 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T08:06:51.999 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T08:06:52.083 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T08:06:52.084 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:52.084 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T08:06:52.084 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T08:06:52.084 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T08:06:52.084 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:52.134 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T08:06:52.226 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T08:06:52.226 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:52.226 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T08:06:52.226 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:52.292 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T08:06:52.387 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T08:06:52.387 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:52.387 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T08:06:52.387 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T08:06:52.387 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T08:06:52.387 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:52.760 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T08:06:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T08:06:52.830 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T08:06:52.830 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T08:06:52.830 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T08:06:52.830 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T08:06:52.830 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:06:55.405 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 139/156 2024-04-24T08:06:55.464 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 140/156 2024-04-24T08:06:55.550 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 141/156 2024-04-24T08:06:55.596 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T08:06:55.646 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T08:06:55.952 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 144/156 2024-04-24T08:06:56.031 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T08:06:56.068 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 146/156 2024-04-24T08:06:56.954 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:06:56.997 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:06:57.051 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:06:57.806 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 139/156 2024-04-24T08:06:57.850 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 140/156 2024-04-24T08:06:57.932 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 141/156 2024-04-24T08:06:57.992 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T08:06:58.042 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T08:06:58.371 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 144/156 2024-04-24T08:06:58.384 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 148/156 2024-04-24T08:06:58.436 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T08:06:58.464 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 146/156 2024-04-24T08:06:58.495 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T08:06:58.585 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T08:06:58.768 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T08:06:58.808 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T08:06:58.808 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T08:06:58.927 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T08:06:58.974 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T08:06:58.975 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T08:06:59.115 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T08:06:59.245 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:06:59.350 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:06:59.404 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:06:59.460 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T08:06:59.542 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:06:59.542 INFO:teuthology.orchestra.run.smithi069.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-24T08:06:59.542 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:00.748 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 148/156 2024-04-24T08:07:00.852 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T08:07:00.947 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T08:07:01.144 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T08:07:01.191 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T08:07:01.191 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T08:07:01.344 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T08:07:01.390 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T08:07:01.391 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T08:07:01.539 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T08:07:01.669 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:07:01.935 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:07:01.935 INFO:teuthology.orchestra.run.smithi121.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-24T08:07:01.935 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:05.292 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 1/156 2024-04-24T08:07:05.292 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2/156 2024-04-24T08:07:05.292 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 3/156 2024-04-24T08:07:05.292 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 4/156 2024-04-24T08:07:05.292 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 5/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 6/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 7/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 11/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 12/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 14/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 15/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 16/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 17/156 2024-04-24T08:07:05.293 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 18/156 2024-04-24T08:07:05.294 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 19/156 2024-04-24T08:07:05.294 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 20/156 2024-04-24T08:07:05.294 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 21/156 2024-04-24T08:07:05.294 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 22/156 2024-04-24T08:07:05.294 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 23/156 2024-04-24T08:07:05.294 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 24/156 2024-04-24T08:07:05.294 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 25/156 2024-04-24T08:07:05.294 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 26/156 2024-04-24T08:07:05.294 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 27/156 2024-04-24T08:07:05.294 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 28/156 2024-04-24T08:07:05.305 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 29/156 2024-04-24T08:07:05.305 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 30/156 2024-04-24T08:07:05.305 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 31/156 2024-04-24T08:07:05.305 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 32/156 2024-04-24T08:07:05.305 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 33/156 2024-04-24T08:07:05.305 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 34/156 2024-04-24T08:07:05.305 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 35/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 36/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 37/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 38/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 39/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T08:07:05.306 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T08:07:05.307 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T08:07:05.308 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T08:07:05.309 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T08:07:05.310 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T08:07:05.311 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T08:07:05.312 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 149/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 151/156 2024-04-24T08:07:05.313 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T08:07:05.314 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T08:07:05.314 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T08:07:05.314 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T08:07:07.602 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 1/156 2024-04-24T08:07:07.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2/156 2024-04-24T08:07:07.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 3/156 2024-04-24T08:07:07.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 4/156 2024-04-24T08:07:07.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 5/156 2024-04-24T08:07:07.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 6/156 2024-04-24T08:07:07.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 7/156 2024-04-24T08:07:07.603 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 11/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 12/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 14/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 15/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 16/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 17/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 18/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 19/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 20/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 21/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 22/156 2024-04-24T08:07:07.604 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 23/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 24/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 25/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 26/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 27/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 28/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 29/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 30/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 31/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 32/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 33/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 34/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 35/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 36/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 37/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 38/156 2024-04-24T08:07:07.605 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 39/156 2024-04-24T08:07:07.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T08:07:07.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T08:07:07.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T08:07:07.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T08:07:07.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T08:07:07.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T08:07:07.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T08:07:07.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T08:07:07.606 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T08:07:07.607 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T08:07:07.608 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T08:07:07.609 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T08:07:07.610 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T08:07:07.611 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T08:07:07.612 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 149/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 151/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T08:07:07.613 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T08:07:07.614 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T08:07:07.614 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T08:07:07.803 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:07:07.803 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:07.803 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2024-04-24T08:07:07.803 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.803 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.803 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.804 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.805 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T08:07:07.806 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.807 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T08:07:07.808 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T08:07:07.809 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T08:07:07.810 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T08:07:07.811 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T08:07:07.811 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T08:07:07.811 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T08:07:07.811 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T08:07:07.811 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T08:07:07.811 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T08:07:07.811 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T08:07:07.811 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T08:07:07.812 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T08:07:07.812 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T08:07:07.812 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T08:07:07.812 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T08:07:07.812 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T08:07:07.812 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T08:07:07.813 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T08:07:07.813 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T08:07:07.813 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T08:07:07.813 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T08:07:07.813 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T08:07:07.813 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T08:07:07.813 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T08:07:07.814 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T08:07:07.814 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T08:07:07.814 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T08:07:07.814 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T08:07:07.814 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T08:07:07.814 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T08:07:07.815 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T08:07:07.815 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T08:07:07.815 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T08:07:07.815 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.815 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.815 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T08:07:07.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T08:07:07.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T08:07:07.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T08:07:07.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T08:07:07.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T08:07:07.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T08:07:07.816 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T08:07:07.817 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T08:07:07.817 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T08:07:07.817 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T08:07:07.817 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T08:07:07.817 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T08:07:07.818 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T08:07:07.818 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T08:07:07.818 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T08:07:07.818 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T08:07:07.818 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T08:07:07.818 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.818 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.819 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:07.819 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T08:07:07.819 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T08:07:07.819 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T08:07:07.819 INFO:teuthology.orchestra.run.smithi069.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T08:07:07.819 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T08:07:07.820 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T08:07:07.821 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T08:07:07.821 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:07.821 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-04-24T08:07:08.409 DEBUG:teuthology.parallel:result is None 2024-04-24T08:07:09.944 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T08:07:09.944 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:09.944 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-04-24T08:07:09.944 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.944 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.944 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T08:07:09.944 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T08:07:09.944 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:09.945 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:09.946 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T08:07:09.947 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T08:07:09.948 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T08:07:09.949 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T08:07:09.950 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T08:07:09.951 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T08:07:09.952 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T08:07:09.953 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.954 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:09.955 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-04-24T08:07:10.557 DEBUG:teuthology.parallel:result is None 2024-04-24T08:07:10.558 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:07:10.779 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T08:07:10.831 INFO:teuthology.orchestra.run.smithi069.stdout:19.0.0-3116.g1f81fb39.el9 2024-04-24T08:07:10.832 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3116.g1f81fb39.el9 2024-04-24T08:07:10.832 INFO:teuthology.task.install:The correct ceph version 19.0.0-3116.g1f81fb39 is installed. 2024-04-24T08:07:10.834 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:07:10.971 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T08:07:11.030 INFO:teuthology.orchestra.run.smithi121.stdout:19.0.0-3116.g1f81fb39.el9 2024-04-24T08:07:11.030 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3116.g1f81fb39.el9 2024-04-24T08:07:11.030 INFO:teuthology.task.install:The correct ceph version 19.0.0-3116.g1f81fb39 is installed. 2024-04-24T08:07:11.032 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T08:07:11.032 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:11.032 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T08:07:11.109 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:11.110 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T08:07:11.178 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T08:07:11.178 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:11.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T08:07:11.253 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T08:07:11.359 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:11.359 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T08:07:11.438 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T08:07:11.507 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T08:07:11.507 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:11.507 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T08:07:11.582 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T08:07:11.681 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:11.682 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T08:07:11.746 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T08:07:11.852 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T08:07:11.852 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:11.852 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T08:07:11.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T08:07:12.018 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:12.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T08:07:12.083 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T08:07:12.192 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T08:07:12.275 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-24T08:07:12.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /var/log/ceph 2024-04-24T08:07:12.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2024-04-24T08:07:12.351 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-24T08:07:12.352 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T08:07:12.357 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T08:07:12.420 INFO:tasks.ceph:Creating extra log directories... 2024-04-24T08:07:12.420 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T08:07:12.425 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T08:07:12.499 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-24T08:07:12.499 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-24T08:07:12.499 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-24T08:07:12.499 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:12.499 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-24T08:07:12.541 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-24T08:07:12.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-24T08:07:12.797 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:12.797 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-24T08:07:12.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-24T08:07:13.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-24T08:07:13.085 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-24T08:07:13.086 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T08:07:13.086 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210', 'branch': 'rishabh-main-apr17-a654945', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub}', 'email': None, 'first_in_suite': False, 'job_id': '7671210', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-main-apr17-a654945'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-main-apr17-a654945', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 7954, 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'sleep_before_teardown': 0, 'subset': '364/512', 'suite': 'fs', 'suite_branch': 'rishabh-main-apr17-a654945', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi069.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK20pgGhDUG25YTktz5xwXObPjKvwJGn16p16aQ/0z0+mcDD2fI1t/7FaYdn3Uc0KXME5axBn4v+GA4VH3lFBc8=', 'smithi121.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO0oFFrIDtsdmDzD5a+fGpNSauqw+ofxvqcq6IPtvi5gvN4205o5tQ8kZoJSeqobbF0W03bDbUz/qnqVr6oBns4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_forward_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_07:32:23', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T08:07:13.087 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T08:07:13.092 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T08:07:13.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T08:07:13.176 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T08:07:13.241 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:13.241 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T08:07:13.279 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T08:07:13.279 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-04-24T08:07:13.356 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T08:07:13.356 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:07:13.356 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 627 Links: 1 2024-04-24T08:07:13.356 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:07:13.356 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:07:13.356 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-24 08:07:00.025957620 +0000 2024-04-24T08:07:13.357 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-24 08:04:27.137617419 +0000 2024-04-24T08:07:13.357 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-24 08:04:27.137617419 +0000 2024-04-24T08:07:13.357 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-24 08:04:27.137617419 +0000 2024-04-24T08:07:13.357 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T08:07:13.451 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-24T08:07:13.452 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-24T08:07:13.452 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000371829 s, 1.4 MB/s 2024-04-24T08:07:13.454 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T08:07:13.494 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-04-24T08:07:13.576 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T08:07:13.577 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:07:13.577 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 646 Links: 1 2024-04-24T08:07:13.577 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:07:13.577 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:07:13.577 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-24 08:07:00.026957597 +0000 2024-04-24T08:07:13.577 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-24 08:04:27.735603666 +0000 2024-04-24T08:07:13.577 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-24 08:04:27.735603666 +0000 2024-04-24T08:07:13.577 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-24 08:04:27.735603666 +0000 2024-04-24T08:07:13.577 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T08:07:13.675 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-24T08:07:13.676 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-24T08:07:13.676 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000301658 s, 1.7 MB/s 2024-04-24T08:07:13.678 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T08:07:13.721 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-04-24T08:07:13.800 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T08:07:13.800 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:07:13.800 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 664 Links: 1 2024-04-24T08:07:13.800 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:07:13.800 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:07:13.800 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-24 08:07:00.027957574 +0000 2024-04-24T08:07:13.800 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-24 08:04:28.346589616 +0000 2024-04-24T08:07:13.800 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-24 08:04:28.346589616 +0000 2024-04-24T08:07:13.800 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-24 08:04:28.346589616 +0000 2024-04-24T08:07:13.801 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T08:07:13.898 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-24T08:07:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-24T08:07:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000282468 s, 1.8 MB/s 2024-04-24T08:07:13.901 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T08:07:13.940 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-04-24T08:07:14.018 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T08:07:14.018 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:07:14.018 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 6h/6d Inode: 684 Links: 1 2024-04-24T08:07:14.018 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:07:14.018 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:07:14.018 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-04-24 08:07:00.027957574 +0000 2024-04-24T08:07:14.018 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-04-24 08:04:28.907576715 +0000 2024-04-24T08:07:14.018 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-04-24 08:04:28.907576715 +0000 2024-04-24T08:07:14.018 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-04-24 08:04:28.907576715 +0000 2024-04-24T08:07:14.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T08:07:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-04-24T08:07:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-04-24T08:07:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000274388 s, 1.9 MB/s 2024-04-24T08:07:14.119 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T08:07:14.157 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T08:07:14.157 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:14.158 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T08:07:14.200 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T08:07:14.200 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-04-24T08:07:14.290 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T08:07:14.290 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:07:14.290 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 628 Links: 1 2024-04-24T08:07:14.290 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:07:14.291 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:07:14.291 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-24 08:07:02.452406350 +0000 2024-04-24T08:07:14.291 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-24 08:04:24.335197240 +0000 2024-04-24T08:07:14.291 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-24 08:04:24.335197240 +0000 2024-04-24T08:07:14.291 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-24 08:04:24.335197240 +0000 2024-04-24T08:07:14.291 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T08:07:14.358 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-24T08:07:14.358 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-24T08:07:14.358 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000455419 s, 1.1 MB/s 2024-04-24T08:07:14.361 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T08:07:14.409 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-04-24T08:07:14.454 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T08:07:14.454 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:07:14.454 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 646 Links: 1 2024-04-24T08:07:14.454 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:07:14.454 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:07:14.454 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-24 08:07:02.452406350 +0000 2024-04-24T08:07:14.455 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-24 08:04:24.905189847 +0000 2024-04-24T08:07:14.455 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-24 08:04:24.905189847 +0000 2024-04-24T08:07:14.455 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-24 08:04:24.905189847 +0000 2024-04-24T08:07:14.455 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T08:07:14.554 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-24T08:07:14.554 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-24T08:07:14.554 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000330352 s, 1.5 MB/s 2024-04-24T08:07:14.557 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T08:07:14.595 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-04-24T08:07:14.674 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T08:07:14.674 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:07:14.674 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 664 Links: 1 2024-04-24T08:07:14.674 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:07:14.674 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:07:14.674 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-24 08:07:02.453406337 +0000 2024-04-24T08:07:14.674 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-24 08:04:25.492182232 +0000 2024-04-24T08:07:14.674 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-24 08:04:25.492182232 +0000 2024-04-24T08:07:14.674 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-24 08:04:25.492182232 +0000 2024-04-24T08:07:14.675 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T08:07:14.779 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-24T08:07:14.779 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-24T08:07:14.779 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000337126 s, 1.5 MB/s 2024-04-24T08:07:14.781 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T08:07:14.822 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-04-24T08:07:14.903 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T08:07:14.903 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:07:14.903 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 684 Links: 1 2024-04-24T08:07:14.903 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:07:14.903 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:07:14.903 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-04-24 08:07:02.453406337 +0000 2024-04-24T08:07:14.903 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-04-24 08:04:26.061174851 +0000 2024-04-24T08:07:14.903 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-04-24 08:04:26.061174851 +0000 2024-04-24T08:07:14.903 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-04-24 08:04:26.061174851 +0000 2024-04-24T08:07:14.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T08:07:15.009 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-04-24T08:07:15.009 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-04-24T08:07:15.010 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000272581 s, 1.9 MB/s 2024-04-24T08:07:15.013 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T08:07:15.053 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T08:07:15.053 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-24T08:07:15.053 INFO:tasks.ceph:Generating config... 2024-04-24T08:07:15.054 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-24T08:07:15.054 INFO:tasks.ceph:[client] debug client = 20 2024-04-24T08:07:15.054 INFO:tasks.ceph:[client] debug ms = 1 2024-04-24T08:07:15.055 INFO:tasks.ceph:[client] pid file = 2024-04-24T08:07:15.055 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-24T08:07:15.055 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T08:07:15.055 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T08:07:15.056 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T08:07:15.056 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T08:07:15.056 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T08:07:15.056 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-24T08:07:15.056 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-24T08:07:15.056 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-24T08:07:15.056 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-24T08:07:15.056 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-24T08:07:15.056 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T08:07:15.056 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T08:07:15.056 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-24T08:07:15.056 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-24T08:07:15.056 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T08:07:15.056 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T08:07:15.056 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-24T08:07:15.057 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T08:07:15.057 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T08:07:15.057 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T08:07:15.057 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T08:07:15.057 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T08:07:15.057 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-24T08:07:15.057 INFO:tasks.ceph:Setting up mon.a... 2024-04-24T08:07:15.057 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-24T08:07:15.146 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.keyring 2024-04-24T08:07:15.153 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-24T08:07:15.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T08:07:15.301 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.69'), ('mon.b', '172.21.15.121'), ('mon.c', '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]')] 2024-04-24T08:07:15.302 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-24T08:07:15.303 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:15.303 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T08:07:15.340 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.69 --add b 172.21.15.121 --addv c '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:15.462 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:15.462 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: generated fsid 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:07:15.462 INFO:teuthology.orchestra.run.smithi069.stdout:setting min_mon_release = quincy 2024-04-24T08:07:15.462 INFO:teuthology.orchestra.run.smithi069.stdout:epoch 0 2024-04-24T08:07:15.462 INFO:teuthology.orchestra.run.smithi069.stdout:fsid 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:07:15.463 INFO:teuthology.orchestra.run.smithi069.stdout:last_changed 2024-04-24T08:07:15.462589+0000 2024-04-24T08:07:15.463 INFO:teuthology.orchestra.run.smithi069.stdout:created 2024-04-24T08:07:15.462589+0000 2024-04-24T08:07:15.463 INFO:teuthology.orchestra.run.smithi069.stdout:min_mon_release 17 (quincy) 2024-04-24T08:07:15.463 INFO:teuthology.orchestra.run.smithi069.stdout:election_strategy: 1 2024-04-24T08:07:15.463 INFO:teuthology.orchestra.run.smithi069.stdout:0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-04-24T08:07:15.463 INFO:teuthology.orchestra.run.smithi069.stdout:1: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.b 2024-04-24T08:07:15.463 INFO:teuthology.orchestra.run.smithi069.stdout:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2024-04-24T08:07:15.463 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-24T08:07:15.465 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T08:07:15.501 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3422a353-af9e-4ed5-a4b2-7365b9c531c6... 2024-04-24T08:07:15.502 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T08:07:15.546 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T08:07:15.660 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-04-24T08:07:15.660 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-04-24T08:07:15.660 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T08:07:15.660 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-04-24T08:07:15.660 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.660 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-04-24T08:07:15.660 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-04-24T08:07:15.661 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-04-24T08:07:15.662 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-04-24T08:07:15.663 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-04-24T08:07:15.664 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-04-24T08:07:15.665 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: bdev async discard = True 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: bdev enable discard = True 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore allocator = bitmap 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 1/20 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 1/20 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 4/10 2024-04-24T08:07:15.666 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:07:15.667 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.668 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: mon down mkfs grace = 300 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-04-24T08:07:15.669 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = "" 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-04-24T08:07:15.670 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds balancer = 20 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-24T08:07:15.671 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-24T08:07:15.672 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-24T08:07:15.672 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-24T08:07:15.672 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T08:07:15.672 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-24T08:07:15.672 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.672 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-24T08:07:15.672 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.672 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-24T08:07:15.672 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.673 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-24T08:07:15.674 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-24T08:07:15.675 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-24T08:07:15.676 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:07:15.677 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:07:15.678 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.679 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: mon down mkfs grace = 300 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-24T08:07:15.680 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-24T08:07:15.681 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-24T08:07:15.682 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-24T08:07:15.701 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-24T08:07:15.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-24T08:07:15.805 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T08:07:15.806 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:15.806 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-24T08:07:15.843 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:15.843 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-24T08:07:15.922 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:07:15.922 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:15.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T08:07:15.922 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T08:07:16.061 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:16.061 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:16.153 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:07:16.153 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:16.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T08:07:16.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T08:07:16.245 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:16.245 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:16.327 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T08:07:16.327 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T08:07:16.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-24T08:07:16.464 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-24T08:07:16.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T08:07:16.602 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T08:07:16.611 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T08:07:16.611 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-24T08:07:16.756 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-24T08:07:16.763 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-24T08:07:16.833 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-24T08:07:16.965 INFO:teuthology.orchestra.run.smithi069.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-24T08:07:16.973 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-24T08:07:17.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-24T08:07:17.175 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-24T08:07:17.182 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-24T08:07:17.245 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-24T08:07:17.360 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-24T08:07:17.368 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-24T08:07:17.429 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T08:07:17.430 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T08:07:17.549 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-24T08:07:17.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-24T08:07:17.690 INFO:teuthology.orchestra.run.smithi069.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-24T08:07:17.726 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-24T08:07:17.839 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-24T08:07:17.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-24T08:07:18.001 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-24T08:07:18.042 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T08:07:18.043 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi069.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-24T08:07:18.043 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-24T08:07:18.134 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T08:07:18.134 INFO:tasks.ceph:role: osd.0 2024-04-24T08:07:18.134 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:07:18.135 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T08:07:18.217 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T08:07:18.217 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T08:07:18.217 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T08:07:18.217 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T08:07:18.217 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T08:07:18.217 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-24T08:07:18.218 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T08:07:18.218 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T08:07:18.218 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T08:07:18.218 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T08:07:18.622 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-24T08:07:18.628 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-24T08:07:18.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-24T08:07:18.765 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-24T08:07:18.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-24T08:07:18.928 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T08:07:18.928 INFO:tasks.ceph:role: osd.1 2024-04-24T08:07:18.928 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:07:18.928 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T08:07:19.013 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T08:07:19.014 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T08:07:19.370 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-24T08:07:19.377 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-24T08:07:19.378 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-24T08:07:19.481 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-24T08:07:19.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-24T08:07:19.620 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T08:07:19.620 INFO:tasks.ceph:role: osd.2 2024-04-24T08:07:19.620 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:07:19.621 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T08:07:19.707 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T08:07:19.707 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T08:07:19.707 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T08:07:19.707 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T08:07:19.707 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T08:07:19.707 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-24T08:07:19.707 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T08:07:19.707 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T08:07:19.707 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T08:07:19.707 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T08:07:20.098 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-24T08:07:20.105 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-24T08:07:20.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-24T08:07:20.234 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-24T08:07:20.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-24T08:07:20.390 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T08:07:20.390 INFO:tasks.ceph:role: osd.3 2024-04-24T08:07:20.390 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:07:20.390 DEBUG:teuthology.orchestra.run.smithi069:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-24T08:07:20.472 INFO:teuthology.orchestra.run.smithi069.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T08:07:20.472 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T08:07:20.472 INFO:teuthology.orchestra.run.smithi069.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T08:07:20.472 INFO:teuthology.orchestra.run.smithi069.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T08:07:20.472 INFO:teuthology.orchestra.run.smithi069.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T08:07:20.472 INFO:teuthology.orchestra.run.smithi069.stdout: = sunit=0 swidth=0 blks 2024-04-24T08:07:20.472 INFO:teuthology.orchestra.run.smithi069.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T08:07:20.472 INFO:teuthology.orchestra.run.smithi069.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T08:07:20.472 INFO:teuthology.orchestra.run.smithi069.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T08:07:20.473 INFO:teuthology.orchestra.run.smithi069.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T08:07:20.880 INFO:teuthology.orchestra.run.smithi069.stdout:Discarding blocks...Done. 2024-04-24T08:07:20.888 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi069.front.sepia.ceph.com -o noatime 2024-04-24T08:07:20.888 DEBUG:teuthology.orchestra.run.smithi069:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-24T08:07:21.048 DEBUG:teuthology.orchestra.run.smithi069:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-24T08:07:21.145 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:21.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:21.299+0000 7f2ae1a11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-24T08:07:21.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:21.299+0000 7f2ae1a11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-24T08:07:21.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:21.299+0000 7f2ae1a11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T08:07:21.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:21.299+0000 7f2ae1a11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T08:07:21.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:21.299+0000 7f2ae1a11640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T08:07:21.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:21.303+0000 7f2ae1a11640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-24T08:07:24.540 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T08:07:24.620 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:24.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:24.749+0000 7f0cd3e32640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-24T08:07:24.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:24.749+0000 7f0cd3e32640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-24T08:07:24.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:24.749+0000 7f0cd3e32640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T08:07:24.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:24.749+0000 7f0cd3e32640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T08:07:24.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:24.749+0000 7f0cd3e32640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T08:07:24.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:24.751+0000 7f0cd3e32640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-24T08:07:27.706 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T08:07:27.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:27.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:27.920+0000 7fde7b25f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-24T08:07:27.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:27.920+0000 7fde7b25f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-24T08:07:27.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:27.920+0000 7fde7b25f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T08:07:27.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:27.921+0000 7fde7b25f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T08:07:27.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:27.921+0000 7fde7b25f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T08:07:27.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:27.923+0000 7fde7b25f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-24T08:07:30.895 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T08:07:30.983 DEBUG:teuthology.orchestra.run.smithi069:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:31.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:31.113+0000 7f13e9dc4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-24T08:07:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:31.113+0000 7f13e9dc4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-24T08:07:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:31.113+0000 7f13e9dc4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T08:07:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:31.114+0000 7f13e9dc4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T08:07:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:31.114+0000 7f13e9dc4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T08:07:31.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:31.116+0000 7f13e9dc4640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-24T08:07:34.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T08:07:34.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-24T08:07:34.241 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T08:07:34.241 INFO:tasks.ceph:role: osd.4 2024-04-24T08:07:34.241 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:07:34.242 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T08:07:34.331 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T08:07:34.332 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T08:07:34.332 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T08:07:34.332 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T08:07:34.332 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T08:07:34.332 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-24T08:07:34.332 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T08:07:34.332 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T08:07:34.332 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T08:07:34.332 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T08:07:34.744 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-24T08:07:34.751 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-24T08:07:34.752 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-24T08:07:34.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-24T08:07:34.966 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-24T08:07:35.032 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T08:07:35.033 INFO:tasks.ceph:role: osd.5 2024-04-24T08:07:35.033 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:07:35.033 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T08:07:35.101 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T08:07:35.101 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T08:07:35.101 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T08:07:35.101 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T08:07:35.101 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T08:07:35.101 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-24T08:07:35.101 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T08:07:35.101 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T08:07:35.101 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T08:07:35.101 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T08:07:35.458 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-24T08:07:35.464 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-24T08:07:35.464 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-24T08:07:35.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-24T08:07:35.637 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-24T08:07:35.698 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T08:07:35.698 INFO:tasks.ceph:role: osd.6 2024-04-24T08:07:35.698 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:07:35.698 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T08:07:35.769 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T08:07:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T08:07:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T08:07:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T08:07:35.769 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T08:07:35.769 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-24T08:07:35.769 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T08:07:35.769 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T08:07:35.770 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T08:07:35.770 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T08:07:36.156 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-24T08:07:36.162 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-24T08:07:36.162 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-24T08:07:36.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-24T08:07:36.344 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-24T08:07:36.414 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T08:07:36.415 INFO:tasks.ceph:role: osd.7 2024-04-24T08:07:36.415 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:07:36.415 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-24T08:07:36.499 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T08:07:36.500 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T08:07:36.500 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T08:07:36.500 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T08:07:36.500 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T08:07:36.500 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2024-04-24T08:07:36.500 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T08:07:36.500 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T08:07:36.500 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T08:07:36.500 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T08:07:36.934 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2024-04-24T08:07:36.941 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2024-04-24T08:07:36.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-24T08:07:37.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-24T08:07:37.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:37.301 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:37.300+0000 7f67752f0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-24T08:07:37.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:37.306+0000 7f67752f0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-24T08:07:37.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:37.306+0000 7f67752f0640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T08:07:37.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:37.306+0000 7f67752f0640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T08:07:37.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:37.306+0000 7f67752f0640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T08:07:37.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:37.319+0000 7f67752f0640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-24T08:07:40.276 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-24T08:07:40.360 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:40.492 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:40.491+0000 7fbbd2e63640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-24T08:07:40.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:40.492+0000 7fbbd2e63640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-24T08:07:40.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:40.492+0000 7fbbd2e63640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T08:07:40.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:40.492+0000 7fbbd2e63640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T08:07:40.493 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:40.492+0000 7fbbd2e63640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T08:07:40.495 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:40.494+0000 7fbbd2e63640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-24T08:07:43.088 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:07:43.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:07:43.697 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-24T08:07:43.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:43.913 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:43.912+0000 7f1611c18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-24T08:07:43.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:43.913+0000 7f1611c18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-24T08:07:43.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:43.913+0000 7f1611c18640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T08:07:43.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:43.913+0000 7f1611c18640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T08:07:43.914 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:43.913+0000 7f1611c18640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T08:07:43.916 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:43.915+0000 7f1611c18640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-24T08:07:46.882 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-24T08:07:46.965 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:47.092 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:47.092+0000 7efd24211640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-24T08:07:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:47.092+0000 7efd24211640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-24T08:07:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:47.092+0000 7efd24211640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T08:07:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:47.092+0000 7efd24211640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T08:07:47.093 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:47.092+0000 7efd24211640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T08:07:47.095 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:47.095+0000 7efd24211640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-24T08:07:50.047 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-24T08:07:50.132 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T08:07:50.132 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:50.132 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-24T08:07:50.221 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:50.221 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-24T08:07:50.305 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:50.305 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-24T08:07:50.388 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:50.388 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-24T08:07:50.515 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:50.515 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-24T08:07:50.602 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:50.602 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-24T08:07:50.729 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:50.729 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-24T08:07:50.813 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:50.813 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-24T08:07:50.900 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:50.901 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-24T08:07:50.977 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:50.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-24T08:07:51.050 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:51.050 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-24T08:07:51.171 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:51.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-24T08:07:51.246 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:51.246 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-24T08:07:51.375 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:51.375 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-24T08:07:51.457 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:51.457 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-24T08:07:51.513 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:51.514 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-24T08:07:51.567 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:51.567 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-24T08:07:51.625 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:51.625 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-24T08:07:51.675 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T08:07:51.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T08:07:51.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T08:07:51.758 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.y] 2024-04-24T08:07:51.758 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC0vShm2eatGxAAFQkL0GBsujypv4VnmSy5bw== 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC0vShmsqMeLRAApLPtHjtq78HzlVcsjfqXOg== 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC0vShmS+yLORAAHPCeBtq99IXEVTpcMT3c4A== 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.0] 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC5vShmMbvcERAANgogiCC/sXf1y0Ykv7ecNQ== 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.1] 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC8vShmDD+uLBAAJHXmxEm1cumJ3wjqe6S0ww== 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.2] 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC/vShmS63kNhAAXec5HVuwo8THKstXiQOL8Q== 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.3] 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDDvShmxcDJBhAA3V3I4C63EHPRUyhzBCZEzw== 2024-04-24T08:07:51.759 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr.x] 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC0vShmv4rwIxAA3rp+yy09qrs7HXip8bZ8xA== 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmkyd5ChAAGsJGU6GGMvvGzQnABuDx/g== 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShm0xmHFRAA3VfYoqbcV8kWF+4fCZMVLA== 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.4] 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDJvShm1pX9ERAAL2gjv/+M915Cf5OoqBhZVw== 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.5] 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDMvShmNIxbHRAA23gxiaGvagddVc2mE5DJJw== 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.6] 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDPvShmb+x4NhAA6aWmAzU6c0AZomVgWcFwig== 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout:[osd.7] 2024-04-24T08:07:51.760 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQDTvShmgcKMBRAA6fkJvNxBjlNhK9CrZ/FMSw== 2024-04-24T08:07:51.761 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-24T08:07:51.761 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw== 2024-04-24T08:07:51.761 INFO:teuthology.orchestra.run.smithi069.stdout:[client.2] 2024-04-24T08:07:51.761 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmW0MnKRAASzxCAKuApVvfu9kUfK9BJw== 2024-04-24T08:07:51.761 INFO:teuthology.orchestra.run.smithi069.stdout:[client.1] 2024-04-24T08:07:51.761 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmKAIHMhAAuTuFmTHvBteErsG23LMEyQ== 2024-04-24T08:07:51.761 INFO:teuthology.orchestra.run.smithi069.stdout:[client.3] 2024-04-24T08:07:51.761 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC2vShmfgoaABAAniS9oubv92R7Qe2EITGKxg== 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC0vShm2eatGxAAFQkL0GBsujypv4VnmSy5bw== 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC0vShmsqMeLRAApLPtHjtq78HzlVcsjfqXOg== 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC0vShmS+yLORAAHPCeBtq99IXEVTpcMT3c4A== 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC5vShmMbvcERAANgogiCC/sXf1y0Ykv7ecNQ== 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC8vShmDD+uLBAAJHXmxEm1cumJ3wjqe6S0ww== 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2024-04-24T08:07:51.764 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC/vShmS63kNhAAXec5HVuwo8THKstXiQOL8Q== 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDDvShmxcDJBhAA3V3I4C63EHPRUyhzBCZEzw== 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC0vShmv4rwIxAA3rp+yy09qrs7HXip8bZ8xA== 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC1vShmkyd5ChAAGsJGU6GGMvvGzQnABuDx/g== 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC1vShm0xmHFRAA3VfYoqbcV8kWF+4fCZMVLA== 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDJvShm1pX9ERAAL2gjv/+M915Cf5OoqBhZVw== 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDMvShmNIxbHRAA23gxiaGvagddVc2mE5DJJw== 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.6] 2024-04-24T08:07:51.765 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDPvShmb+x4NhAA6aWmAzU6c0AZomVgWcFwig== 2024-04-24T08:07:51.766 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.7] 2024-04-24T08:07:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDTvShmgcKMBRAA6fkJvNxBjlNhK9CrZ/FMSw== 2024-04-24T08:07:51.766 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2024-04-24T08:07:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw== 2024-04-24T08:07:51.766 INFO:teuthology.orchestra.run.smithi121.stdout:[client.2] 2024-04-24T08:07:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC1vShmW0MnKRAASzxCAKuApVvfu9kUfK9BJw== 2024-04-24T08:07:51.766 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2024-04-24T08:07:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC1vShmKAIHMhAAuTuFmTHvBteErsG23LMEyQ== 2024-04-24T08:07:51.766 INFO:teuthology.orchestra.run.smithi121.stdout:[client.3] 2024-04-24T08:07:51.766 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQC2vShmfgoaABAAniS9oubv92R7Qe2EITGKxg== 2024-04-24T08:07:51.768 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T08:07:51.811 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T08:07:51.951 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T08:07:51.959 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T08:07:52.088 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T08:07:52.095 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T08:07:52.225 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:52.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:52.360 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:52.367 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:52.496 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:52.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:52.634 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:52.640 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:52.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T08:07:52.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T08:07:52.908 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T08:07:52.913 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T08:07:53.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T08:07:53.055 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T08:07:53.191 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:53.197 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:53.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:53.328 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:53.458 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:53.465 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:53.598 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:53.604 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T08:07:53.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T08:07:53.738 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T08:07:53.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T08:07:53.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T08:07:54.008 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T08:07:54.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T08:07:54.150 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T08:07:54.155 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T08:07:54.295 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T08:07:54.296 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-24T08:07:54.381 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T08:07:54.523 DEBUG:teuthology.orchestra.run.smithi069:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-24T08:07:54.588 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-24T08:07:54.678 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T08:07:54.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-24T08:07:54.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-24T08:07:54.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T08:07:55.120 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-24T08:07:55.191 DEBUG:teuthology.orchestra.run.smithi069:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:55.199 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T08:07:55.249 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-24T08:07:55.249 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-24T08:07:55.249 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-24T08:07:55.254 INFO:tasks.ceph.mon.a:Started 2024-04-24T08:07:55.254 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-24T08:07:55.254 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-24T08:07:55.259 INFO:tasks.ceph.mon.b:Started 2024-04-24T08:07:55.259 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-24T08:07:55.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-24T08:07:55.264 INFO:tasks.ceph.mon.c:Started 2024-04-24T08:07:55.264 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-24T08:07:55.265 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-24T08:07:55.265 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-24T08:07:55.271 INFO:tasks.ceph.mgr.y:Started 2024-04-24T08:07:55.272 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-24T08:07:55.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-24T08:07:55.276 INFO:tasks.ceph.mgr.x:Started 2024-04-24T08:07:55.276 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T08:07:55.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph osd crush tunables default 2024-04-24T08:07:55.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.451+0000 7fa4fa200640 1 Processor -- start 2024-04-24T08:07:55.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.451+0000 7fa4fa200640 1 -- start start 2024-04-24T08:07:55.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.452+0000 7fa4fa200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 0x7fa4f415faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.452+0000 7fa4fa200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4f406cc30 con 0x7fa4f415e870 2024-04-24T08:07:55.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.452+0000 7fa4fa200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4f406ced0 con 0x7fa4f40d4e00 2024-04-24T08:07:55.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.452+0000 7fa4fa200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f406e9c0 con 0x7fa4f40d3350 2024-04-24T08:07:55.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.453+0000 7fa4f3e00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa4f40d4e00 0x7fa4f41615a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:53798/0 (socket says 172.21.15.69:53798) 2024-04-24T08:07:55.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.453+0000 7fa4f3e00640 1 -- 172.21.15.69:0/2955066223 learned_addr learned my addr 172.21.15.69:0/2955066223 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:07:55.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.453+0000 7fa4f2a00640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 0x7fa4f415faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.453+0000 7fa4f2a00640 1 -- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 msgr2=0x7fa4f415faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-24T08:07:55.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.453+0000 7fa4f2a00640 1 -- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 msgr2=0x7fa4f415faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T08:07:55.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.453+0000 7fa4f2a00640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 0x7fa4f415faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T08:07:55.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.453+0000 7fa4f2a00640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 0x7fa4f415faf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:07:55.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.467+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931040932 0 0) 0x7fa4f406cc30 con 0x7fa4f415e870 2024-04-24T08:07:55.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.467+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4d8003520 con 0x7fa4f415e870 2024-04-24T08:07:55.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2395176162 0 0) 0x7fa4f406ced0 con 0x7fa4f40d4e00 2024-04-24T08:07:55.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4f406cc30 con 0x7fa4f40d4e00 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2247236702 0 0) 0x7fa4d8003520 con 0x7fa4f415e870 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 msgr2=0x7fa4f415faf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 0x7fa4f415faf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 >> v1:172.21.15.121:6789/0 conn(0x7fa4f40d4e00 legacy=0x7fa4f41615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f406d1b0 con 0x7fa4f415e870 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fa4e4001670 con 0x7fa4f415e870 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 >> v1:172.21.15.69:6789/0 conn(0x7fa4f415e870 legacy=0x7fa4f40d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 0x7fa4d8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4d8004120 0x7fa4d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4d8004d40 0x7fa4d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d800bbf0 con 0x7fa4d8004d40 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d800bda0 con 0x7fa4d8004120 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d800bfa0 con 0x7fa4f40d3350 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.468+0000 7fa4f3e00640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4d8004d40 0x7fa4d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.469+0000 7fa4f3400640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 0x7fa4d8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.469+0000 7fa4f2a00640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4d8004120 0x7fa4d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.469+0000 7fa4f3e00640 1 -- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 msgr2=0x7fa4d8003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.469+0000 7fa4f3e00640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40d3350 0x7fa4d8003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.469+0000 7fa4f3e00640 1 -- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4d8004120 msgr2=0x7fa4d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.469+0000 7fa4f3e00640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4d8004120 0x7fa4d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.469+0000 7fa4f3e00640 1 -- 172.21.15.69:0/2955066223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d8003520 con 0x7fa4d8004d40 2024-04-24T08:07:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.469+0000 7fa4f2a00640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4d8004120 0x7fa4d8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:07:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.470+0000 7fa4f3e00640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4d8004d40 0x7fa4d80051c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00c320 tx=0x7fa4d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e654c2cff50708 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.470+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec036070 con 0x7fa4d8004d40 2024-04-24T08:07:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.470+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4ec031680 con 0x7fa4d8004d40 2024-04-24T08:07:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.470+0000 7fa4f2000640 1 -- 172.21.15.69:0/2955066223 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4ec002060 con 0x7fa4d8004d40 2024-04-24T08:07:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.471+0000 7fa4fa200640 1 -- 172.21.15.69:0/2955066223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4d8004d40 msgr2=0x7fa4d80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.471+0000 7fa4fa200640 1 --2- 172.21.15.69:0/2955066223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4d8004d40 0x7fa4d80051c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec00c320 tx=0x7fa4d8005830 comp rx=0 tx=0).stop 2024-04-24T08:07:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.471+0000 7fa4f3e00640 1 -- 172.21.15.69:0/2955066223 reap_dead start 2024-04-24T08:07:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.471+0000 7fa4fa200640 1 -- 172.21.15.69:0/2955066223 shutdown_connections 2024-04-24T08:07:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.471+0000 7fa4fa200640 1 -- 172.21.15.69:0/2955066223 >> 172.21.15.69:0/2955066223 conn(0x7fa4f408bf60 msgr2=0x7fa4f40d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:55.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.471+0000 7fa4fa200640 1 -- 172.21.15.69:0/2955066223 shutdown_connections 2024-04-24T08:07:55.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.471+0000 7fa4fa200640 1 -- 172.21.15.69:0/2955066223 wait complete. 2024-04-24T08:07:55.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.473+0000 7fa4fa200640 1 Processor -- start 2024-04-24T08:07:55.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.473+0000 7fa4fa200640 1 -- start start 2024-04-24T08:07:55.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.474+0000 7fa4fa200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4f40b2520 0x7fa4f409fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.474+0000 7fa4fa200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40b2900 0x7fa4f409e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.474+0000 7fa4fa200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4151e10 0x7fa4f414e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.474+0000 7fa4fa200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f40d55a0 con 0x7fa4f40b2520 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.474+0000 7fa4fa200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f40d5290 con 0x7fa4f4151e10 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.474+0000 7fa4fa200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa4f41602d0 con 0x7fa4f40b2900 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f3e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4151e10 0x7fa4f414e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f3e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4151e10 0x7fa4f414e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:58566/0 (socket says 172.21.15.69:58566) 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f3e00640 1 -- 172.21.15.69:0/56616835 learned_addr learned my addr 172.21.15.69:0/56616835 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f3400640 1 --2- 172.21.15.69:0/56616835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4f40b2520 0x7fa4f409fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f2a00640 1 --2- 172.21.15.69:0/56616835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40b2900 0x7fa4f409e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f3400640 1 -- 172.21.15.69:0/56616835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40b2900 msgr2=0x7fa4f409e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f3400640 1 --2- 172.21.15.69:0/56616835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40b2900 0x7fa4f409e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f3400640 1 -- 172.21.15.69:0/56616835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4151e10 msgr2=0x7fa4f414e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f3400640 1 --2- 172.21.15.69:0/56616835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4151e10 0x7fa4f414e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f3400640 1 -- 172.21.15.69:0/56616835 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f41522c0 con 0x7fa4f40b2520 2024-04-24T08:07:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.475+0000 7fa4f3400640 1 --2- 172.21.15.69:0/56616835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4f40b2520 0x7fa4f409fea0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400f770 tx=0x7fa4e400fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.476+0000 7fa4eb400640 1 -- 172.21.15.69:0/56616835 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e40050d0 con 0x7fa4f40b2520 2024-04-24T08:07:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.476+0000 7fa4eb400640 1 -- 172.21.15.69:0/56616835 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4e4018070 con 0x7fa4f40b2520 2024-04-24T08:07:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.476+0000 7fa4fa200640 1 -- 172.21.15.69:0/56616835 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f414f510 con 0x7fa4f40b2520 2024-04-24T08:07:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.476+0000 7fa4eb400640 1 -- 172.21.15.69:0/56616835 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400d6d0 con 0x7fa4f40b2520 2024-04-24T08:07:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.476+0000 7fa4fa200640 1 -- 172.21.15.69:0/56616835 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f414fad0 con 0x7fa4f40b2520 2024-04-24T08:07:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.476+0000 7fa4eb400640 1 -- 172.21.15.69:0/56616835 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa4e400da80 con 0x7fa4f40b2520 2024-04-24T08:07:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.476+0000 7fa4eb400640 1 -- 172.21.15.69:0/56616835 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa4e401d4f0 con 0x7fa4f40b2520 2024-04-24T08:07:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.477+0000 7fa4fa200640 1 -- 172.21.15.69:0/56616835 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4f4069190 con 0x7fa4f40b2520 2024-04-24T08:07:55.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.480+0000 7fa4eb400640 1 -- 172.21.15.69:0/56616835 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa4e40035f0 con 0x7fa4f40b2520 2024-04-24T08:07:55.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.552+0000 7fa4fa200640 1 -- 172.21.15.69:0/56616835 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fa4f40d1ec0 con 0x7fa4f40b2520 2024-04-24T08:07:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.556+0000 7fa4eb400640 1 -- 172.21.15.69:0/56616835 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fa4e40037d0 con 0x7fa4f40b2520 2024-04-24T08:07:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to default 2024-04-24T08:07:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.557+0000 7fa4fa200640 1 -- 172.21.15.69:0/56616835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4f40b2520 msgr2=0x7fa4f409fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.557+0000 7fa4fa200640 1 --2- 172.21.15.69:0/56616835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4f40b2520 0x7fa4f409fea0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400f770 tx=0x7fa4e400fc30 comp rx=0 tx=0).stop 2024-04-24T08:07:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.557+0000 7fa4fa200640 1 -- 172.21.15.69:0/56616835 shutdown_connections 2024-04-24T08:07:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.557+0000 7fa4fa200640 1 --2- 172.21.15.69:0/56616835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4151e10 0x7fa4f414e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.557+0000 7fa4fa200640 1 --2- 172.21.15.69:0/56616835 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4f40b2900 0x7fa4f409e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.557+0000 7fa4fa200640 1 --2- 172.21.15.69:0/56616835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4f40b2520 0x7fa4f409fea0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.557+0000 7fa4fa200640 1 -- 172.21.15.69:0/56616835 >> 172.21.15.69:0/56616835 conn(0x7fa4f408bf60 msgr2=0x7fa4f40d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.558+0000 7fa4fa200640 1 -- 172.21.15.69:0/56616835 shutdown_connections 2024-04-24T08:07:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:55.558+0000 7fa4fa200640 1 -- 172.21.15.69:0/56616835 wait complete. 2024-04-24T08:07:55.575 INFO:tasks.ceph:check_enable_crimson: False 2024-04-24T08:07:55.575 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-24T08:07:55.576 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:55.576 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-24T08:07:55.644 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:55.644 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-24T08:07:55.755 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:55.755 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-24T08:07:55.779 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:55.778+0000 7f96d3614040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:07:55.784 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:55.783+0000 7f6452f1c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:07:55.860 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:07:55.860 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-24T08:07:55.966 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:55.965+0000 7f96d3614040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:07:55.978 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:55.978 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-24T08:07:55.998 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:55.997+0000 7f6452f1c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:07:56.033 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:56.032+0000 7f96d3614040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:07:56.057 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:56.058 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-24T08:07:56.065 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:56.063+0000 7f6452f1c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:07:56.171 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:56.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-24T08:07:56.295 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:07:56.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-24T08:07:56.342 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:56.341+0000 7f96d3614040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:07:56.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new f892d466-0add-47f5-9758-28353c69b922 0 2024-04-24T08:07:56.387 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:56.386+0000 7f6452f1c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:07:56.398 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:56.397+0000 7f96d3614040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:07:56.441 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:56.440+0000 7f6452f1c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:07:56.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.585+0000 7f68a6a00640 1 Processor -- start 2024-04-24T08:07:56.586 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.586+0000 7f68a6a00640 1 -- start start 2024-04-24T08:07:56.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a6a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f68a015e790 0x7f68a00d2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a6a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d27d0 0x7f68a00d4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a6a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68a00d12a0 con 0x7f68a00d4b50 2024-04-24T08:07:56.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a6a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f68a006ca60 con 0x7f68a015e790 2024-04-24T08:07:56.588 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a6a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f68a00d4fb0 con 0x7f68a00d27d0 2024-04-24T08:07:56.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a4c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d27d0 0x7f68a00d4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a4c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d27d0 0x7f68a00d4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52772/0 (socket says 172.21.15.121:52772) 2024-04-24T08:07:56.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a4c00640 1 -- 172.21.15.121:0/4287923066 learned_addr learned my addr 172.21.15.121:0/4287923066 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:56.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a5600640 1 --2- 172.21.15.121:0/4287923066 >> v2:172.21.15.69:3300/0 conn(0x7f68a015e790 0x7f68a00d2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a4c00640 1 -- 172.21.15.121:0/4287923066 >> v2:172.21.15.69:3300/0 conn(0x7f68a015e790 msgr2=0x7f68a00d2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:56.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.587+0000 7f68a4c00640 1 --2- 172.21.15.121:0/4287923066 >> v2:172.21.15.69:3300/0 conn(0x7f68a015e790 0x7f68a00d2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.588+0000 7f68a4c00640 1 -- 172.21.15.121:0/4287923066 >> v1:172.21.15.69:6789/0 conn(0x7f68a00d4b50 legacy=0x7f68a0162350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:56.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.588+0000 7f68a4c00640 1 -- 172.21.15.121:0/4287923066 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a00d34d0 con 0x7f68a00d27d0 2024-04-24T08:07:56.589 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.589+0000 7f68a4c00640 1 --2- 172.21.15.121:0/4287923066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d27d0 0x7f68a00d4610 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f688c009870 tx=0x7f688c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8ed31fb35269e05 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:56.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.589+0000 7f689fe00640 1 -- 172.21.15.121:0/4287923066 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c037070 con 0x7f68a00d27d0 2024-04-24T08:07:56.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.589+0000 7f689fe00640 1 -- 172.21.15.121:0/4287923066 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f688c032440 con 0x7f68a00d27d0 2024-04-24T08:07:56.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.589+0000 7f689fe00640 1 -- 172.21.15.121:0/4287923066 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c032810 con 0x7f68a00d27d0 2024-04-24T08:07:56.590 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.589+0000 7f68a6a00640 1 -- 172.21.15.121:0/4287923066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d27d0 msgr2=0x7f68a00d4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.589+0000 7f68a6a00640 1 --2- 172.21.15.121:0/4287923066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d27d0 0x7f68a00d4610 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f688c009870 tx=0x7f688c02a210 comp rx=0 tx=0).stop 2024-04-24T08:07:56.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.589+0000 7f68a6a00640 1 -- 172.21.15.121:0/4287923066 shutdown_connections 2024-04-24T08:07:56.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.589+0000 7f68a6a00640 1 --2- 172.21.15.121:0/4287923066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d27d0 0x7f68a00d4610 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.589+0000 7f68a6a00640 1 --2- 172.21.15.121:0/4287923066 >> v2:172.21.15.69:3300/0 conn(0x7f68a015e790 0x7f68a00d2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.589+0000 7f68a6a00640 1 -- 172.21.15.121:0/4287923066 >> 172.21.15.121:0/4287923066 conn(0x7f68a00681e0 msgr2=0x7f68a009d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:56.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.590+0000 7f68a6a00640 1 -- 172.21.15.121:0/4287923066 shutdown_connections 2024-04-24T08:07:56.591 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.590+0000 7f68a6a00640 1 -- 172.21.15.121:0/4287923066 wait complete. 2024-04-24T08:07:56.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.591+0000 7f68a6a00640 1 Processor -- start 2024-04-24T08:07:56.592 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.591+0000 7f68a6a00640 1 -- start start 2024-04-24T08:07:56.593 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.592+0000 7f68a6a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d4b50 0x7f68a01f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.592+0000 7f68a6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a015e790 0x7f68a015f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.592+0000 7f68a6a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a01526d0 0x7f68a0161730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.592+0000 7f68a6a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f68a00d1b00 con 0x7f68a015e790 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.592+0000 7f68a6a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f68a00d1820 con 0x7f68a01526d0 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.592+0000 7f68a6a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f68a00d1de0 con 0x7f68a00d4b50 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.592+0000 7f68a5600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d4b50 0x7f68a01f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.592+0000 7f68a5600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d4b50 0x7f68a01f88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52786/0 (socket says 172.21.15.121:52786) 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.592+0000 7f68a5600640 1 -- 172.21.15.121:0/1643213397 learned_addr learned my addr 172.21.15.121:0/1643213397 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.592+0000 7f68a6000640 1 --2- 172.21.15.121:0/1643213397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a01526d0 0x7f68a0161730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f68a4c00640 1 --2- 172.21.15.121:0/1643213397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a015e790 0x7f68a015f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.594 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f68a5600640 1 -- 172.21.15.121:0/1643213397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a01526d0 msgr2=0x7f68a0161730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f68a5600640 1 --2- 172.21.15.121:0/1643213397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a01526d0 0x7f68a0161730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f68a5600640 1 -- 172.21.15.121:0/1643213397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a015e790 msgr2=0x7f68a015f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f68a5600640 1 --2- 172.21.15.121:0/1643213397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a015e790 0x7f68a015f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f68a5600640 1 -- 172.21.15.121:0/1643213397 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a0152b80 con 0x7f68a00d4b50 2024-04-24T08:07:56.595 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f68a5600640 1 --2- 172.21.15.121:0/1643213397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d4b50 0x7f68a01f88c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f689400d5f0 tx=0x7f689400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:56.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f689d600640 1 -- 172.21.15.121:0/1643213397 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68940125f0 con 0x7f68a00d4b50 2024-04-24T08:07:56.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f689d600640 1 -- 172.21.15.121:0/1643213397 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6894019070 con 0x7f68a00d4b50 2024-04-24T08:07:56.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f689d600640 1 -- 172.21.15.121:0/1643213397 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68940129d0 con 0x7f68a00d4b50 2024-04-24T08:07:56.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.593+0000 7f68a6a00640 1 -- 172.21.15.121:0/1643213397 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a0161c70 con 0x7f68a00d4b50 2024-04-24T08:07:56.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.594+0000 7f68a6a00640 1 -- 172.21.15.121:0/1643213397 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a0160120 con 0x7f68a00d4b50 2024-04-24T08:07:56.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.594+0000 7f689d600640 1 -- 172.21.15.121:0/1643213397 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6894012b70 con 0x7f68a00d4b50 2024-04-24T08:07:56.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.594+0000 7f689d600640 1 -- 172.21.15.121:0/1643213397 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f68940144c0 con 0x7f68a00d4b50 2024-04-24T08:07:56.596 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.595+0000 7f68a6a00640 1 -- 172.21.15.121:0/1643213397 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68a0075d50 con 0x7f68a00d4b50 2024-04-24T08:07:56.598 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.597+0000 7f689d600640 1 -- 172.21.15.121:0/1643213397 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f68a0161c70 con 0x7f68a00d4b50 2024-04-24T08:07:56.677 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.677+0000 7f68a6a00640 1 -- 172.21.15.121:0/1643213397 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "f892d466-0add-47f5-9758-28353c69b922", "id": 0} v 0) -- 0x7f68a006d9d0 con 0x7f68a00d4b50 2024-04-24T08:07:56.684 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.683+0000 7f689d600640 1 -- 172.21.15.121:0/1643213397 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f892d466-0add-47f5-9758-28353c69b922", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f689401e530 con 0x7f68a00d4b50 2024-04-24T08:07:56.684 INFO:teuthology.orchestra.run.smithi121.stdout:0 2024-04-24T08:07:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.685+0000 7f68a6a00640 1 -- 172.21.15.121:0/1643213397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d4b50 msgr2=0x7f68a01f88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.685+0000 7f68a6a00640 1 --2- 172.21.15.121:0/1643213397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d4b50 0x7f68a01f88c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f689400d5f0 tx=0x7f689400dab0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.685+0000 7f68a6a00640 1 -- 172.21.15.121:0/1643213397 shutdown_connections 2024-04-24T08:07:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.685+0000 7f68a6a00640 1 --2- 172.21.15.121:0/1643213397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68a015e790 0x7f68a015f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.685+0000 7f68a6a00640 1 --2- 172.21.15.121:0/1643213397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68a01526d0 0x7f68a0161730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.685+0000 7f68a6a00640 1 --2- 172.21.15.121:0/1643213397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68a00d4b50 0x7f68a01f88c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.685+0000 7f68a6a00640 1 -- 172.21.15.121:0/1643213397 >> 172.21.15.121:0/1643213397 conn(0x7f68a00681e0 msgr2=0x7f68a015c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:56.686 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.685+0000 7f68a6a00640 1 -- 172.21.15.121:0/1643213397 shutdown_connections 2024-04-24T08:07:56.687 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.685+0000 7f68a6a00640 1 -- 172.21.15.121:0/1643213397 wait complete. 2024-04-24T08:07:56.702 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 40de3817-1e3a-467e-850b-0027ceccdcb6 1 2024-04-24T08:07:56.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.873+0000 7ffa4ea00640 1 Processor -- start 2024-04-24T08:07:56.875 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.874+0000 7ffa4ea00640 1 -- start start 2024-04-24T08:07:56.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.875+0000 7ffa4ea00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480d4610 0x7ffa480d4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.875+0000 7ffa4ea00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa480d12a0 con 0x7ffa4815e790 2024-04-24T08:07:56.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.875+0000 7ffa4ea00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa480d29a0 con 0x7ffa480cf330 2024-04-24T08:07:56.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.875+0000 7ffa4ea00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffa4806ca60 con 0x7ffa480d4610 2024-04-24T08:07:56.876 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.875+0000 7ffa4cc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480d4610 0x7ffa480d4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.875+0000 7ffa4e000640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ffa480cf330 0x7ffa480cf7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33820/0 (socket says 172.21.15.121:33820) 2024-04-24T08:07:56.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.875+0000 7ffa4cc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480d4610 0x7ffa480d4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52792/0 (socket says 172.21.15.121:52792) 2024-04-24T08:07:56.877 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.875+0000 7ffa4e000640 1 -- 172.21.15.121:0/1355720833 learned_addr learned my addr 172.21.15.121:0/1355720833 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-04-24T08:07:56.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.876+0000 7ffa4cc00640 1 -- 172.21.15.121:0/1355720833 >> v1:172.21.15.121:6789/0 conn(0x7ffa480cf330 legacy=0x7ffa480cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.876+0000 7ffa4cc00640 1 -- 172.21.15.121:0/1355720833 >> v1:172.21.15.69:6789/0 conn(0x7ffa4815e790 legacy=0x7ffa480d2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.876+0000 7ffa47e00640 1 -- 172.21.15.121:0/1355720833 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2308901940 0 0) 0x7ffa480d29a0 con 0x7ffa480cf330 2024-04-24T08:07:56.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.876+0000 7ffa4cc00640 1 -- 172.21.15.121:0/1355720833 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa480d34d0 con 0x7ffa480d4610 2024-04-24T08:07:56.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.877+0000 7ffa4cc00640 1 --2- 172.21.15.121:0/1355720833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480d4610 0x7ffa480d4a50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffa340097d0 tx=0x7ffa3402a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d920cffb0decfec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:56.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.877+0000 7ffa47e00640 1 -- 172.21.15.121:0/1355720833 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa34037070 con 0x7ffa480d4610 2024-04-24T08:07:56.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.877+0000 7ffa47e00640 1 -- 172.21.15.121:0/1355720833 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa34032440 con 0x7ffa480d4610 2024-04-24T08:07:56.878 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.877+0000 7ffa47e00640 1 -- 172.21.15.121:0/1355720833 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa34032810 con 0x7ffa480d4610 2024-04-24T08:07:56.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.878+0000 7ffa4ea00640 1 -- 172.21.15.121:0/1355720833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480d4610 msgr2=0x7ffa480d4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.878+0000 7ffa4ea00640 1 --2- 172.21.15.121:0/1355720833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480d4610 0x7ffa480d4a50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ffa340097d0 tx=0x7ffa3402a200 comp rx=0 tx=0).stop 2024-04-24T08:07:56.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.878+0000 7ffa4ea00640 1 -- 172.21.15.121:0/1355720833 shutdown_connections 2024-04-24T08:07:56.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.878+0000 7ffa4ea00640 1 --2- 172.21.15.121:0/1355720833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480d4610 0x7ffa480d4a50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.878+0000 7ffa4ea00640 1 -- 172.21.15.121:0/1355720833 >> 172.21.15.121:0/1355720833 conn(0x7ffa480681e0 msgr2=0x7ffa4809d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:56.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.878+0000 7ffa4ea00640 1 -- 172.21.15.121:0/1355720833 shutdown_connections 2024-04-24T08:07:56.879 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.878+0000 7ffa4ea00640 1 -- 172.21.15.121:0/1355720833 wait complete. 2024-04-24T08:07:56.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.879+0000 7ffa4ea00640 1 Processor -- start 2024-04-24T08:07:56.880 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.880+0000 7ffa4ea00640 1 -- start start 2024-04-24T08:07:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.880+0000 7ffa4ea00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480cf330 0x7ffa480d0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.880+0000 7ffa4ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa480d4610 0x7ffa4815b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4ea00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4815e790 0x7ffa4815bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4ea00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffa480d1820 con 0x7ffa480d4610 2024-04-24T08:07:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4ea00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffa480d4f90 con 0x7ffa4815e790 2024-04-24T08:07:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4ea00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffa480d1b00 con 0x7ffa480cf330 2024-04-24T08:07:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480cf330 0x7ffa480d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.883 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4e000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4815e790 0x7ffa4815bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480cf330 0x7ffa480d0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52796/0 (socket says 172.21.15.121:52796) 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4d600640 1 -- 172.21.15.121:0/674896540 learned_addr learned my addr 172.21.15.121:0/674896540 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4cc00640 1 --2- 172.21.15.121:0/674896540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa480d4610 0x7ffa4815b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4d600640 1 -- 172.21.15.121:0/674896540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4815e790 msgr2=0x7ffa4815bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4d600640 1 --2- 172.21.15.121:0/674896540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4815e790 0x7ffa4815bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4d600640 1 -- 172.21.15.121:0/674896540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa480d4610 msgr2=0x7ffa4815b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4d600640 1 --2- 172.21.15.121:0/674896540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa480d4610 0x7ffa4815b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4d600640 1 -- 172.21.15.121:0/674896540 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa480b84f0 con 0x7ffa480cf330 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4cc00640 1 --2- 172.21.15.121:0/674896540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa480d4610 0x7ffa4815b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.881+0000 7ffa4e000640 1 --2- 172.21.15.121:0/674896540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4815e790 0x7ffa4815bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.882+0000 7ffa4d600640 1 --2- 172.21.15.121:0/674896540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480cf330 0x7ffa480d0540 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00c390 tx=0x7ffa3c00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.882+0000 7ffa45600640 1 -- 172.21.15.121:0/674896540 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c017070 con 0x7ffa480cf330 2024-04-24T08:07:56.884 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.882+0000 7ffa45600640 1 -- 172.21.15.121:0/674896540 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa3c0052d0 con 0x7ffa480cf330 2024-04-24T08:07:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.882+0000 7ffa45600640 1 -- 172.21.15.121:0/674896540 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c003830 con 0x7ffa480cf330 2024-04-24T08:07:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.882+0000 7ffa4ea00640 1 -- 172.21.15.121:0/674896540 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa480b87d0 con 0x7ffa480cf330 2024-04-24T08:07:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.882+0000 7ffa4ea00640 1 -- 172.21.15.121:0/674896540 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa480b8d90 con 0x7ffa480cf330 2024-04-24T08:07:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.882+0000 7ffa45600640 1 -- 172.21.15.121:0/674896540 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffa3c01a070 con 0x7ffa480cf330 2024-04-24T08:07:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.882+0000 7ffa45600640 1 -- 172.21.15.121:0/674896540 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7ffa3c011c00 con 0x7ffa480cf330 2024-04-24T08:07:56.885 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.883+0000 7ffa4ea00640 1 -- 172.21.15.121:0/674896540 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa48075360 con 0x7ffa480cf330 2024-04-24T08:07:56.886 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.886+0000 7ffa45600640 1 -- 172.21.15.121:0/674896540 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ffa3c021070 con 0x7ffa480cf330 2024-04-24T08:07:56.965 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.964+0000 7ffa4ea00640 1 -- 172.21.15.121:0/674896540 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "40de3817-1e3a-467e-850b-0027ceccdcb6", "id": 1} v 0) -- 0x7ffa480d1dd0 con 0x7ffa480cf330 2024-04-24T08:07:56.972 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.971+0000 7ffa45600640 1 -- 172.21.15.121:0/674896540 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "40de3817-1e3a-467e-850b-0027ceccdcb6", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7ffa3c011e00 con 0x7ffa480cf330 2024-04-24T08:07:56.972 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-04-24T08:07:56.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.973+0000 7ffa4ea00640 1 -- 172.21.15.121:0/674896540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480cf330 msgr2=0x7ffa480d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.973+0000 7ffa4ea00640 1 --2- 172.21.15.121:0/674896540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480cf330 0x7ffa480d0540 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c00c390 tx=0x7ffa3c00c850 comp rx=0 tx=0).stop 2024-04-24T08:07:56.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.973+0000 7ffa4ea00640 1 -- 172.21.15.121:0/674896540 shutdown_connections 2024-04-24T08:07:56.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.973+0000 7ffa4ea00640 1 --2- 172.21.15.121:0/674896540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa480d4610 0x7ffa4815b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.973+0000 7ffa4ea00640 1 --2- 172.21.15.121:0/674896540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa4815e790 0x7ffa4815bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.973+0000 7ffa4ea00640 1 --2- 172.21.15.121:0/674896540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffa480cf330 0x7ffa480d0540 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.973+0000 7ffa4ea00640 1 -- 172.21.15.121:0/674896540 >> 172.21.15.121:0/674896540 conn(0x7ffa480681e0 msgr2=0x7ffa480b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:56.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.973+0000 7ffa4ea00640 1 -- 172.21.15.121:0/674896540 shutdown_connections 2024-04-24T08:07:56.974 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:56.973+0000 7ffa4ea00640 1 -- 172.21.15.121:0/674896540 wait complete. 2024-04-24T08:07:56.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new e952a5f2-4652-482d-baa9-8193e47b28dc 2 2024-04-24T08:07:57.165 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.165+0000 7fa3c0600640 1 Processor -- start 2024-04-24T08:07:57.166 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.165+0000 7fa3c0600640 1 -- start start 2024-04-24T08:07:57.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.166+0000 7fa3c0600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3b815e870 0x7fa3b815faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.166+0000 7fa3c0600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3b806cc30 con 0x7fa3b80d3350 2024-04-24T08:07:57.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.166+0000 7fa3c0600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3b806ced0 con 0x7fa3b80d4e00 2024-04-24T08:07:57.167 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.166+0000 7fa3c0600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa3b806e9c0 con 0x7fa3b815e870 2024-04-24T08:07:57.169 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.167+0000 7fa3bd200640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fa3b80d4e00 0x7fa3b80d2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33828/0 (socket says 172.21.15.121:33828) 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.167+0000 7fa3bd200640 1 -- 172.21.15.121:0/3675052413 learned_addr learned my addr 172.21.15.121:0/3675052413 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.167+0000 7fa3bdc00640 1 --2- 172.21.15.121:0/3675052413 >> v2:172.21.15.69:3300/0 conn(0x7fa3b815e870 0x7fa3b815faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.167+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 222660052 0 0) 0x7fa3b806ced0 con 0x7fa3b80d4e00 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.167+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3a0003520 con 0x7fa3b80d4e00 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.167+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 775291520 0 0) 0x7fa3a0003520 con 0x7fa3b80d4e00 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.167+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 >> v2:172.21.15.69:3300/0 conn(0x7fa3b815e870 msgr2=0x7fa3b815faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.167+0000 7fa3b7e00640 1 --2- 172.21.15.121:0/3675052413 >> v2:172.21.15.69:3300/0 conn(0x7fa3b815e870 0x7fa3b815faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.167+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 >> v1:172.21.15.69:6789/0 conn(0x7fa3b80d3350 legacy=0x7fa3b81615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.167+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b806d1b0 con 0x7fa3b80d4e00 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fa3a4001670 con 0x7fa3b80d4e00 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 >> v1:172.21.15.121:6789/0 conn(0x7fa3b80d4e00 legacy=0x7fa3b80d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.170 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3b7e00640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a00039b0 0x7fa3a0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3b7e00640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a0004800 0x7fa3a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3b7e00640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a00051c0 0x7fa3a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a000c130 con 0x7fa3a00039b0 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a000c2e0 con 0x7fa3a0004800 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a000c4e0 con 0x7fa3a00051c0 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3bd200640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a0004800 0x7fa3a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3be600640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a00051c0 0x7fa3a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3bdc00640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a00039b0 0x7fa3a0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3bd200640 1 -- 172.21.15.121:0/3675052413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a00051c0 msgr2=0x7fa3a000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3bd200640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a00051c0 0x7fa3a000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3bd200640 1 -- 172.21.15.121:0/3675052413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a00039b0 msgr2=0x7fa3a0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3bd200640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a00039b0 0x7fa3a0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3bd200640 1 -- 172.21.15.121:0/3675052413 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a0003520 con 0x7fa3a0004800 2024-04-24T08:07:57.171 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.168+0000 7fa3be600640 1 -- 172.21.15.121:0/3675052413 reap_dead start 2024-04-24T08:07:57.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.169+0000 7fa3bd200640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a0004800 0x7fa3a0004c80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400c240 tx=0x7fa3a0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a23a175d9ec24d1f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:57.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.170+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4007d10 con 0x7fa3a0004800 2024-04-24T08:07:57.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.170+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a40058d0 con 0x7fa3a0004800 2024-04-24T08:07:57.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.170+0000 7fa3b7e00640 1 -- 172.21.15.121:0/3675052413 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4005d20 con 0x7fa3a0004800 2024-04-24T08:07:57.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.170+0000 7fa3c0600640 1 -- 172.21.15.121:0/3675052413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a0004800 msgr2=0x7fa3a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.170+0000 7fa3c0600640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a0004800 0x7fa3a0004c80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400c240 tx=0x7fa3a0005830 comp rx=0 tx=0).stop 2024-04-24T08:07:57.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.170+0000 7fa3c0600640 1 -- 172.21.15.121:0/3675052413 shutdown_connections 2024-04-24T08:07:57.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.170+0000 7fa3c0600640 1 --2- 172.21.15.121:0/3675052413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a0004800 0x7fa3a0004c80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.170+0000 7fa3c0600640 1 -- 172.21.15.121:0/3675052413 >> 172.21.15.121:0/3675052413 conn(0x7fa3b808bf60 msgr2=0x7fa3b80d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:57.173 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.170+0000 7fa3c0600640 1 -- 172.21.15.121:0/3675052413 shutdown_connections 2024-04-24T08:07:57.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.170+0000 7fa3c0600640 1 -- 172.21.15.121:0/3675052413 wait complete. 2024-04-24T08:07:57.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.172+0000 7fa3c0600640 1 Processor -- start 2024-04-24T08:07:57.174 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.172+0000 7fa3c0600640 1 -- start start 2024-04-24T08:07:57.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.173+0000 7fa3c0600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a0004800 0x7fa3b81f8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.173+0000 7fa3c0600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3b8155560 0x7fa3b80b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.173+0000 7fa3c0600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3b80cd9b0 0x7fa3b815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.173+0000 7fa3c0600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b80d5270 con 0x7fa3b8155560 2024-04-24T08:07:57.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.173+0000 7fa3c0600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b80d55a0 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.173+0000 7fa3c0600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b81602d0 con 0x7fa3a0004800 2024-04-24T08:07:57.176 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.173+0000 7fa3bdc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a0004800 0x7fa3b81f8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.173+0000 7fa3be600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3b80cd9b0 0x7fa3b815bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.173+0000 7fa3bdc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a0004800 0x7fa3b81f8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52812/0 (socket says 172.21.15.121:52812) 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.173+0000 7fa3bdc00640 1 -- 172.21.15.121:0/2308260135 learned_addr learned my addr 172.21.15.121:0/2308260135 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3bd200640 1 --2- 172.21.15.121:0/2308260135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3b8155560 0x7fa3b80b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3be600640 1 -- 172.21.15.121:0/2308260135 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a0004800 msgr2=0x7fa3b81f8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3be600640 1 --2- 172.21.15.121:0/2308260135 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a0004800 0x7fa3b81f8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3be600640 1 -- 172.21.15.121:0/2308260135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3b8155560 msgr2=0x7fa3b80b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3be600640 1 --2- 172.21.15.121:0/2308260135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3b8155560 0x7fa3b80b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3be600640 1 -- 172.21.15.121:0/2308260135 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b80b6bb0 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3be600640 1 --2- 172.21.15.121:0/2308260135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3b80cd9b0 0x7fa3b815bfd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa3b808edd0 tx=0x7fa3b000fc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3b5600640 1 -- 172.21.15.121:0/2308260135 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0003b80 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3b5600640 1 -- 172.21.15.121:0/2308260135 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b0018070 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3c0600640 1 -- 172.21.15.121:0/2308260135 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b80b6e90 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.177 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3b5600640 1 -- 172.21.15.121:0/2308260135 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00050d0 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.174+0000 7fa3c0600640 1 -- 172.21.15.121:0/2308260135 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b80b6810 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.175+0000 7fa3b5600640 1 -- 172.21.15.121:0/2308260135 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa3b0005270 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.175+0000 7fa3b5600640 1 -- 172.21.15.121:0/2308260135 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fa3b000dd50 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.178 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.175+0000 7fa3c0600640 1 -- 172.21.15.121:0/2308260135 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3b80b34f0 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.179 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.178+0000 7fa3b5600640 1 -- 172.21.15.121:0/2308260135 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa3b80b34f0 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.256+0000 7fa3c0600640 1 -- 172.21.15.121:0/2308260135 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e952a5f2-4652-482d-baa9-8193e47b28dc", "id": 2} v 0) -- 0x7fa3b8069190 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.264 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.263+0000 7fa3b5600640 1 -- 172.21.15.121:0/2308260135 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e952a5f2-4652-482d-baa9-8193e47b28dc", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fa3b00056a0 con 0x7fa3b80cd9b0 2024-04-24T08:07:57.264 INFO:teuthology.orchestra.run.smithi121.stdout:2 2024-04-24T08:07:57.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.264+0000 7fa3c0600640 1 -- 172.21.15.121:0/2308260135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3b80cd9b0 msgr2=0x7fa3b815bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.264+0000 7fa3c0600640 1 --2- 172.21.15.121:0/2308260135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3b80cd9b0 0x7fa3b815bfd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa3b808edd0 tx=0x7fa3b000fc70 comp rx=0 tx=0).stop 2024-04-24T08:07:57.265 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.264+0000 7fa3c0600640 1 -- 172.21.15.121:0/2308260135 shutdown_connections 2024-04-24T08:07:57.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.264+0000 7fa3c0600640 1 --2- 172.21.15.121:0/2308260135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3b8155560 0x7fa3b80b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.264+0000 7fa3c0600640 1 --2- 172.21.15.121:0/2308260135 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3b80cd9b0 0x7fa3b815bfd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.264+0000 7fa3c0600640 1 --2- 172.21.15.121:0/2308260135 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a0004800 0x7fa3b81f8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.264+0000 7fa3c0600640 1 -- 172.21.15.121:0/2308260135 >> 172.21.15.121:0/2308260135 conn(0x7fa3b808bf60 msgr2=0x7fa3b80d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:57.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.265+0000 7fa3c0600640 1 -- 172.21.15.121:0/2308260135 shutdown_connections 2024-04-24T08:07:57.266 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.265+0000 7fa3c0600640 1 -- 172.21.15.121:0/2308260135 wait complete. 2024-04-24T08:07:57.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef 3 2024-04-24T08:07:57.338 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:57.337+0000 7f96d3614040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:07:57.404 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:57.403+0000 7f6452f1c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:07:57.446 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:57.445+0000 7f96d3614040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:07:57.462 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.462+0000 7ff23f200640 1 Processor -- start 2024-04-24T08:07:57.463 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.462+0000 7ff23f200640 1 -- start start 2024-04-24T08:07:57.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.463+0000 7ff23f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff24015e790 0x7ff2400d2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.464 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.463+0000 7ff23f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 0x7ff2400cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.463+0000 7ff23f200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2400d12a0 con 0x7ff2400d27d0 2024-04-24T08:07:57.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.463+0000 7ff23f200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff24006ca60 con 0x7ff24015e790 2024-04-24T08:07:57.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.463+0000 7ff23f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff2400d34d0 con 0x7ff2400d4d20 2024-04-24T08:07:57.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.464+0000 7ff23de00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff24015e790 0x7ff2400d2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.464+0000 7ff23e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 0x7ff2400cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.464+0000 7ff23de00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff24015e790 0x7ff2400d2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60058/0 (socket says 172.21.15.121:60058) 2024-04-24T08:07:57.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.464+0000 7ff23de00640 1 -- 172.21.15.121:0/2963057217 learned_addr learned my addr 172.21.15.121:0/2963057217 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:57.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.464+0000 7ff23e800640 1 -- 172.21.15.121:0/2963057217 >> v2:172.21.15.121:3300/0 conn(0x7ff24015e790 msgr2=0x7ff2400d2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:57.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.464+0000 7ff23e800640 1 --2- 172.21.15.121:0/2963057217 >> v2:172.21.15.121:3300/0 conn(0x7ff24015e790 0x7ff2400d2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.464+0000 7ff23e800640 1 -- 172.21.15.121:0/2963057217 >> v1:172.21.15.121:6789/0 conn(0x7ff2400d27d0 legacy=0x7ff2400d4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.464+0000 7ff23e800640 1 -- 172.21.15.121:0/2963057217 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2400d1540 con 0x7ff2400d4d20 2024-04-24T08:07:57.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.464+0000 7ff23de00640 1 --2- 172.21.15.121:0/2963057217 >> v2:172.21.15.121:3300/0 conn(0x7ff24015e790 0x7ff2400d2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:57.466 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.465+0000 7ff23e800640 1 --2- 172.21.15.121:0/2963057217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 0x7ff2400cf330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff238009d90 tx=0x7ff23802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8d667f0652d8bee server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:57.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.466+0000 7ff23ca00640 1 -- 172.21.15.121:0/2963057217 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238007740 con 0x7ff2400d4d20 2024-04-24T08:07:57.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.466+0000 7ff23ca00640 1 -- 172.21.15.121:0/2963057217 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2380078e0 con 0x7ff2400d4d20 2024-04-24T08:07:57.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.466+0000 7ff23ca00640 1 -- 172.21.15.121:0/2963057217 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238007cb0 con 0x7ff2400d4d20 2024-04-24T08:07:57.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.466+0000 7ff23f200640 1 -- 172.21.15.121:0/2963057217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 msgr2=0x7ff2400cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.467 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.466+0000 7ff23f200640 1 --2- 172.21.15.121:0/2963057217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 0x7ff2400cf330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff238009d90 tx=0x7ff23802a660 comp rx=0 tx=0).stop 2024-04-24T08:07:57.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.466+0000 7ff23f200640 1 -- 172.21.15.121:0/2963057217 shutdown_connections 2024-04-24T08:07:57.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.466+0000 7ff23f200640 1 --2- 172.21.15.121:0/2963057217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 0x7ff2400cf330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.466+0000 7ff23f200640 1 --2- 172.21.15.121:0/2963057217 >> v2:172.21.15.121:3300/0 conn(0x7ff24015e790 0x7ff2400d2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.466+0000 7ff23f200640 1 -- 172.21.15.121:0/2963057217 >> 172.21.15.121:0/2963057217 conn(0x7ff2400681e0 msgr2=0x7ff24009d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:57.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.467+0000 7ff23f200640 1 -- 172.21.15.121:0/2963057217 shutdown_connections 2024-04-24T08:07:57.468 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.467+0000 7ff23f200640 1 -- 172.21.15.121:0/2963057217 wait complete. 2024-04-24T08:07:57.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.468+0000 7ff23f200640 1 Processor -- start 2024-04-24T08:07:57.469 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.469+0000 7ff23f200640 1 -- start start 2024-04-24T08:07:57.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.469+0000 7ff23f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 0x7ff2400b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24015e790 0x7ff24014dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2401f77e0 0x7ff2400b52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff2400d1de0 con 0x7ff2401f77e0 2024-04-24T08:07:57.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff2400d1b00 con 0x7ff24015e790 2024-04-24T08:07:57.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff24015eb70 con 0x7ff2400d4d20 2024-04-24T08:07:57.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23d400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24015e790 0x7ff24014dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.471 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 0x7ff2400b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23d400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24015e790 0x7ff24014dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60064/0 (socket says 172.21.15.121:60064) 2024-04-24T08:07:57.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23d400640 1 -- 172.21.15.121:0/3714601126 learned_addr learned my addr 172.21.15.121:0/3714601126 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:57.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23e800640 1 --2- 172.21.15.121:0/3714601126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2401f77e0 0x7ff2400b52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23d400640 1 -- 172.21.15.121:0/3714601126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 msgr2=0x7ff2400b7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23d400640 1 --2- 172.21.15.121:0/3714601126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 0x7ff2400b7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23d400640 1 -- 172.21.15.121:0/3714601126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2401f77e0 msgr2=0x7ff2400b52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23d400640 1 --2- 172.21.15.121:0/3714601126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2401f77e0 0x7ff2400b52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23d400640 1 -- 172.21.15.121:0/3714601126 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2401f8230 con 0x7ff24015e790 2024-04-24T08:07:57.472 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.470+0000 7ff23e800640 1 --2- 172.21.15.121:0/3714601126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2401f77e0 0x7ff2400b52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:07:57.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.471+0000 7ff23d400640 1 --2- 172.21.15.121:0/3714601126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24015e790 0x7ff24014dbe0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff22800c400 tx=0x7ff22800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:57.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.471+0000 7ff236000640 1 -- 172.21.15.121:0/3714601126 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2280052d0 con 0x7ff24015e790 2024-04-24T08:07:57.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.471+0000 7ff236000640 1 -- 172.21.15.121:0/3714601126 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff228005470 con 0x7ff24015e790 2024-04-24T08:07:57.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.471+0000 7ff236000640 1 -- 172.21.15.121:0/3714601126 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228007260 con 0x7ff24015e790 2024-04-24T08:07:57.473 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.471+0000 7ff23f200640 1 -- 172.21.15.121:0/3714601126 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2401f8510 con 0x7ff24015e790 2024-04-24T08:07:57.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.471+0000 7ff23f200640 1 -- 172.21.15.121:0/3714601126 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff24014e120 con 0x7ff24015e790 2024-04-24T08:07:57.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.471+0000 7ff236000640 1 -- 172.21.15.121:0/3714601126 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff228007400 con 0x7ff24015e790 2024-04-24T08:07:57.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.471+0000 7ff236000640 1 -- 172.21.15.121:0/3714601126 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7ff2280035f0 con 0x7ff24015e790 2024-04-24T08:07:57.474 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.472+0000 7ff23f200640 1 -- 172.21.15.121:0/3714601126 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff240075d50 con 0x7ff24015e790 2024-04-24T08:07:57.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.475+0000 7ff236000640 1 -- 172.21.15.121:0/3714601126 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff228003c30 con 0x7ff24015e790 2024-04-24T08:07:57.500 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:57.500+0000 7f96d3614040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:07:57.508 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:57.507+0000 7f6452f1c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:07:57.555 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.555+0000 7ff23f200640 1 -- 172.21.15.121:0/3714601126 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef", "id": 3} v 0) -- 0x7ff2401f86e0 con 0x7ff24015e790 2024-04-24T08:07:57.558 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:57.557+0000 7f6452f1c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:07:57.562 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.561+0000 7ff236000640 1 -- 172.21.15.121:0/3714601126 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7ff2280067d0 con 0x7ff24015e790 2024-04-24T08:07:57.562 INFO:teuthology.orchestra.run.smithi121.stdout:3 2024-04-24T08:07:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.562+0000 7ff23f200640 1 -- 172.21.15.121:0/3714601126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24015e790 msgr2=0x7ff24014dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.562+0000 7ff23f200640 1 --2- 172.21.15.121:0/3714601126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24015e790 0x7ff24014dbe0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff22800c400 tx=0x7ff22800c8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.563+0000 7ff23f200640 1 -- 172.21.15.121:0/3714601126 shutdown_connections 2024-04-24T08:07:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.563+0000 7ff23f200640 1 --2- 172.21.15.121:0/3714601126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2401f77e0 0x7ff2400b52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.563+0000 7ff23f200640 1 --2- 172.21.15.121:0/3714601126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff24015e790 0x7ff24014dbe0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.563+0000 7ff23f200640 1 --2- 172.21.15.121:0/3714601126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff2400d4d20 0x7ff2400b7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.563+0000 7ff23f200640 1 -- 172.21.15.121:0/3714601126 >> 172.21.15.121:0/3714601126 conn(0x7ff2400681e0 msgr2=0x7ff2400cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.563+0000 7ff23f200640 1 -- 172.21.15.121:0/3714601126 shutdown_connections 2024-04-24T08:07:57.564 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.563+0000 7ff23f200640 1 -- 172.21.15.121:0/3714601126 wait complete. 2024-04-24T08:07:57.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 027a943d-d007-4adf-8e00-8261810feaa9 4 2024-04-24T08:07:57.705 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:57.704+0000 7f96d3614040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:07:57.750 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:57.749+0000 7f6452f1c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:07:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.755+0000 7efeb7200640 1 Processor -- start 2024-04-24T08:07:57.756 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.755+0000 7efeb7200640 1 -- start start 2024-04-24T08:07:57.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.757+0000 7efeb7200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efeb806edb0 0x7efeb81602b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.757+0000 7efeb7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb806f5b0 0x7efeb806fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.757+0000 7efeb7200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeb8160a00 con 0x7efeb815c930 2024-04-24T08:07:57.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.757+0000 7efeb7200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7efeb806ff40 con 0x7efeb806edb0 2024-04-24T08:07:57.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.757+0000 7efeb7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efeb8160ce0 con 0x7efeb806f5b0 2024-04-24T08:07:57.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.757+0000 7efeb6800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb806f5b0 0x7efeb806fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.758 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.757+0000 7efead400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efeb806edb0 0x7efeb81602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.757+0000 7efeb5e00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7efeb815c930 0x7efeb80cfa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33838/0 (socket says 172.21.15.121:33838) 2024-04-24T08:07:57.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.757+0000 7efeb6800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb806f5b0 0x7efeb806fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52846/0 (socket says 172.21.15.121:52846) 2024-04-24T08:07:57.759 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.757+0000 7efeb5e00640 1 -- 172.21.15.121:0/2231236658 learned_addr learned my addr 172.21.15.121:0/2231236658 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-04-24T08:07:57.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.758+0000 7efeb6800640 1 -- 172.21.15.121:0/2231236658 >> v2:172.21.15.121:3300/0 conn(0x7efeb806edb0 msgr2=0x7efeb81602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:57.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.758+0000 7efeb6800640 1 --2- 172.21.15.121:0/2231236658 >> v2:172.21.15.121:3300/0 conn(0x7efeb806edb0 0x7efeb81602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.758+0000 7efeb6800640 1 -- 172.21.15.121:0/2231236658 >> v1:172.21.15.121:6789/0 conn(0x7efeb815c930 legacy=0x7efeb80cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.758+0000 7efeb6800640 1 -- 172.21.15.121:0/2231236658 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb8160fc0 con 0x7efeb806f5b0 2024-04-24T08:07:57.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.758+0000 7efead400640 1 --2- 172.21.15.121:0/2231236658 >> v2:172.21.15.121:3300/0 conn(0x7efeb806edb0 0x7efeb81602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:57.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.759+0000 7efeb6800640 1 --2- 172.21.15.121:0/2231236658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb806f5b0 0x7efeb806fa00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009d90 tx=0x7efeb002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95b3d51a4c0a8e43 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:57.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.759+0000 7efeb5400640 1 -- 172.21.15.121:0/2231236658 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb0007740 con 0x7efeb806f5b0 2024-04-24T08:07:57.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.759+0000 7efeb5400640 1 -- 172.21.15.121:0/2231236658 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeb00078e0 con 0x7efeb806f5b0 2024-04-24T08:07:57.760 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.759+0000 7efeb5400640 1 -- 172.21.15.121:0/2231236658 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeb0007cb0 con 0x7efeb806f5b0 2024-04-24T08:07:57.761 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.760+0000 7efeb7200640 1 -- 172.21.15.121:0/2231236658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb806f5b0 msgr2=0x7efeb806fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.760+0000 7efeb7200640 1 --2- 172.21.15.121:0/2231236658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb806f5b0 0x7efeb806fa00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009d90 tx=0x7efeb002a660 comp rx=0 tx=0).stop 2024-04-24T08:07:57.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.760+0000 7efeb7200640 1 -- 172.21.15.121:0/2231236658 shutdown_connections 2024-04-24T08:07:57.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.760+0000 7efeb7200640 1 --2- 172.21.15.121:0/2231236658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb806f5b0 0x7efeb806fa00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.760+0000 7efeb7200640 1 --2- 172.21.15.121:0/2231236658 >> v2:172.21.15.121:3300/0 conn(0x7efeb806edb0 0x7efeb81602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.760+0000 7efeb7200640 1 -- 172.21.15.121:0/2231236658 >> 172.21.15.121:0/2231236658 conn(0x7efeb806cb30 msgr2=0x7efeb806cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:57.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.760+0000 7efeb7200640 1 -- 172.21.15.121:0/2231236658 shutdown_connections 2024-04-24T08:07:57.762 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.760+0000 7efeb7200640 1 -- 172.21.15.121:0/2231236658 wait complete. 2024-04-24T08:07:57.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.762+0000 7efeb7200640 1 Processor -- start 2024-04-24T08:07:57.763 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.762+0000 7efeb7200640 1 -- start start 2024-04-24T08:07:57.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb7200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb806edb0 0x7efeb80caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efeb806f5b0 0x7efeb81f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb815c930 0x7efeb81f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efeb8073380 con 0x7efeb806f5b0 2024-04-24T08:07:57.765 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb7200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efeb8073200 con 0x7efeb806edb0 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efeb8073500 con 0x7efeb815c930 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb6800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb815c930 0x7efeb81f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb5e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb806edb0 0x7efeb80caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb6800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb815c930 0x7efeb81f42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52862/0 (socket says 172.21.15.121:52862) 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb6800640 1 -- 172.21.15.121:0/1180687851 learned_addr learned my addr 172.21.15.121:0/1180687851 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb5e00640 1 -- 172.21.15.121:0/1180687851 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb815c930 msgr2=0x7efeb81f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb5e00640 1 --2- 172.21.15.121:0/1180687851 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb815c930 0x7efeb81f42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb5e00640 1 -- 172.21.15.121:0/1180687851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efeb806f5b0 msgr2=0x7efeb81f5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb5e00640 1 --2- 172.21.15.121:0/1180687851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efeb806f5b0 0x7efeb81f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.763+0000 7efeb5e00640 1 -- 172.21.15.121:0/1180687851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb8150020 con 0x7efeb806edb0 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.764+0000 7efeb5e00640 1 --2- 172.21.15.121:0/1180687851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb806edb0 0x7efeb80caa50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efea800c390 tx=0x7efea800c790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.764+0000 7efeaea00640 1 -- 172.21.15.121:0/1180687851 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea8017070 con 0x7efeb806edb0 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.764+0000 7efeaea00640 1 -- 172.21.15.121:0/1180687851 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efea8005ce0 con 0x7efeb806edb0 2024-04-24T08:07:57.766 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.764+0000 7efeaea00640 1 -- 172.21.15.121:0/1180687851 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea8007310 con 0x7efeb806edb0 2024-04-24T08:07:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.764+0000 7efeb7200640 1 -- 172.21.15.121:0/1180687851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb80c9a20 con 0x7efeb806edb0 2024-04-24T08:07:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.764+0000 7efeb7200640 1 -- 172.21.15.121:0/1180687851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb81505c0 con 0x7efeb806edb0 2024-04-24T08:07:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.764+0000 7efeaea00640 1 -- 172.21.15.121:0/1180687851 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efea801a070 con 0x7efeb806edb0 2024-04-24T08:07:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.764+0000 7efeaea00640 1 -- 172.21.15.121:0/1180687851 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7efea80074b0 con 0x7efeb806edb0 2024-04-24T08:07:57.767 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.765+0000 7efeb7200640 1 -- 172.21.15.121:0/1180687851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb8075d50 con 0x7efeb806edb0 2024-04-24T08:07:57.768 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.768+0000 7efeaea00640 1 -- 172.21.15.121:0/1180687851 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efea8021070 con 0x7efeb806edb0 2024-04-24T08:07:57.773 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:57.773+0000 7f96d3614040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:07:57.806 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:57.805+0000 7f6452f1c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:07:57.837 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:57.836+0000 7f96d3614040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:07:57.847 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.846+0000 7efeb7200640 1 -- 172.21.15.121:0/1180687851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "027a943d-d007-4adf-8e00-8261810feaa9", "id": 4} v 0) -- 0x7efeb806f390 con 0x7efeb806edb0 2024-04-24T08:07:57.854 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.853+0000 7efeaea00640 1 -- 172.21.15.121:0/1180687851 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "027a943d-d007-4adf-8e00-8261810feaa9", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7efea801c440 con 0x7efeb806edb0 2024-04-24T08:07:57.854 INFO:teuthology.orchestra.run.smithi121.stdout:4 2024-04-24T08:07:57.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.855+0000 7efeb7200640 1 -- 172.21.15.121:0/1180687851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb806edb0 msgr2=0x7efeb80caa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.855+0000 7efeb7200640 1 --2- 172.21.15.121:0/1180687851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb806edb0 0x7efeb80caa50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efea800c390 tx=0x7efea800c790 comp rx=0 tx=0).stop 2024-04-24T08:07:57.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.855+0000 7efeb7200640 1 -- 172.21.15.121:0/1180687851 shutdown_connections 2024-04-24T08:07:57.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.855+0000 7efeb7200640 1 --2- 172.21.15.121:0/1180687851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efeb806f5b0 0x7efeb81f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.855+0000 7efeb7200640 1 --2- 172.21.15.121:0/1180687851 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeb815c930 0x7efeb81f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.855+0000 7efeb7200640 1 --2- 172.21.15.121:0/1180687851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeb806edb0 0x7efeb80caa50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.855+0000 7efeb7200640 1 -- 172.21.15.121:0/1180687851 >> 172.21.15.121:0/1180687851 conn(0x7efeb806cb30 msgr2=0x7efeb80d1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:57.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.855+0000 7efeb7200640 1 -- 172.21.15.121:0/1180687851 shutdown_connections 2024-04-24T08:07:57.856 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:57.855+0000 7efeb7200640 1 -- 172.21.15.121:0/1180687851 wait complete. 2024-04-24T08:07:57.869 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:57.868+0000 7f6452f1c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:07:57.871 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 9bbf8edc-5742-43a4-ba34-afe0e3f880bb 5 2024-04-24T08:07:57.972 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:57.972+0000 7f96d3614040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:07:57.996 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:57.995+0000 7f6452f1c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:07:58.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.053+0000 7f8725800640 1 Processor -- start 2024-04-24T08:07:58.054 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.053+0000 7f8725800640 1 -- start start 2024-04-24T08:07:58.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f8725800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f872015c930 0x7f872006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f8725800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f87201602b0 0x7f87200cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f8725800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8720160740 con 0x7f872006f5b0 2024-04-24T08:07:58.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f8725800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f87200cffc0 con 0x7f87201602b0 2024-04-24T08:07:58.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f8725800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8720160a00 con 0x7f872015c930 2024-04-24T08:07:58.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f871fe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f872015c930 0x7f872006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f871fe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f872015c930 0x7f872006edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60072/0 (socket says 172.21.15.121:60072) 2024-04-24T08:07:58.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f871fe00640 1 -- 172.21.15.121:0/3631299858 learned_addr learned my addr 172.21.15.121:0/3631299858 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:58.056 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f871fe00640 1 -- 172.21.15.121:0/3631299858 >> v2:172.21.15.69:3300/0 conn(0x7f87201602b0 msgr2=0x7f87200cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:58.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f871fe00640 1 --2- 172.21.15.121:0/3631299858 >> v2:172.21.15.69:3300/0 conn(0x7f87201602b0 0x7f87200cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f871fe00640 1 -- 172.21.15.121:0/3631299858 >> v1:172.21.15.69:6789/0 conn(0x7f872006f5b0 legacy=0x7f8720073200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:58.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.055+0000 7f871fe00640 1 -- 172.21.15.121:0/3631299858 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8720160ce0 con 0x7f872015c930 2024-04-24T08:07:58.057 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.056+0000 7f871fe00640 1 --2- 172.21.15.121:0/3631299858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872015c930 0x7f872006edb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f87200b2da0 tx=0x7f871002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1a7273ee36a5f39 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:58.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.057+0000 7f871ea00640 1 -- 172.21.15.121:0/3631299858 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710037070 con 0x7f872015c930 2024-04-24T08:07:58.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.057+0000 7f871ea00640 1 -- 172.21.15.121:0/3631299858 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8710032440 con 0x7f872015c930 2024-04-24T08:07:58.058 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.057+0000 7f871ea00640 1 -- 172.21.15.121:0/3631299858 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710032810 con 0x7f872015c930 2024-04-24T08:07:58.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.057+0000 7f8725800640 1 -- 172.21.15.121:0/3631299858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872015c930 msgr2=0x7f872006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.057+0000 7f8725800640 1 --2- 172.21.15.121:0/3631299858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872015c930 0x7f872006edb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f87200b2da0 tx=0x7f871002a230 comp rx=0 tx=0).stop 2024-04-24T08:07:58.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.057+0000 7f8725800640 1 -- 172.21.15.121:0/3631299858 shutdown_connections 2024-04-24T08:07:58.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.057+0000 7f8725800640 1 --2- 172.21.15.121:0/3631299858 >> v2:172.21.15.69:3300/0 conn(0x7f87201602b0 0x7f87200cfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.057+0000 7f8725800640 1 --2- 172.21.15.121:0/3631299858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872015c930 0x7f872006edb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.057+0000 7f8725800640 1 -- 172.21.15.121:0/3631299858 >> 172.21.15.121:0/3631299858 conn(0x7f872006cb30 msgr2=0x7f872006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:58.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.057+0000 7f8725800640 1 -- 172.21.15.121:0/3631299858 shutdown_connections 2024-04-24T08:07:58.059 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.058+0000 7f8725800640 1 -- 172.21.15.121:0/3631299858 wait complete. 2024-04-24T08:07:58.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.059+0000 7f8725800640 1 Processor -- start 2024-04-24T08:07:58.060 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.059+0000 7f8725800640 1 -- start start 2024-04-24T08:07:58.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.060+0000 7f8725800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872006f5b0 0x7f872015cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.060+0000 7f8725800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f872015c930 0x7f87200cabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.060+0000 7f8725800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87201602b0 0x7f872015e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.060+0000 7f8725800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f872006fa00 con 0x7f87201602b0 2024-04-24T08:07:58.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.060+0000 7f8725800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8720072ce0 con 0x7f872006f5b0 2024-04-24T08:07:58.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.060+0000 7f8725800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f872006fb80 con 0x7f872015c930 2024-04-24T08:07:58.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.060+0000 7f871fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872006f5b0 0x7f872015cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.060+0000 7f871f400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f872015c930 0x7f87200cabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.063 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.060+0000 7f871fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872006f5b0 0x7f872015cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60074/0 (socket says 172.21.15.121:60074) 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.060+0000 7f871fe00640 1 -- 172.21.15.121:0/361737431 learned_addr learned my addr 172.21.15.121:0/361737431 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f8724e00640 1 --2- 172.21.15.121:0/361737431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87201602b0 0x7f872015e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f871f400640 1 -- 172.21.15.121:0/361737431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872006f5b0 msgr2=0x7f872015cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f871f400640 1 --2- 172.21.15.121:0/361737431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872006f5b0 0x7f872015cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f871f400640 1 -- 172.21.15.121:0/361737431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87201602b0 msgr2=0x7f872015e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f871f400640 1 --2- 172.21.15.121:0/361737431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87201602b0 0x7f872015e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f871f400640 1 -- 172.21.15.121:0/361737431 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87200ae950 con 0x7f872015c930 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f871fe00640 1 --2- 172.21.15.121:0/361737431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872006f5b0 0x7f872015cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f8724e00640 1 --2- 172.21.15.121:0/361737431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87201602b0 0x7f872015e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f871f400640 1 --2- 172.21.15.121:0/361737431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f872015c930 0x7f87200cabf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f870c00b410 tx=0x7f870c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f8717e00640 1 -- 172.21.15.121:0/361737431 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c015020 con 0x7f872015c930 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f8717e00640 1 -- 172.21.15.121:0/361737431 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f870c011850 con 0x7f872015c930 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f8717e00640 1 -- 172.21.15.121:0/361737431 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c011c40 con 0x7f872015c930 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.061+0000 7f8725800640 1 -- 172.21.15.121:0/361737431 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872015d2f0 con 0x7f872015c930 2024-04-24T08:07:58.064 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.062+0000 7f8725800640 1 -- 172.21.15.121:0/361737431 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87200aeef0 con 0x7f872015c930 2024-04-24T08:07:58.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.062+0000 7f8717e00640 1 -- 172.21.15.121:0/361737431 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f870c018020 con 0x7f872015c930 2024-04-24T08:07:58.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.062+0000 7f8717e00640 1 -- 172.21.15.121:0/361737431 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f870c011de0 con 0x7f872015c930 2024-04-24T08:07:58.065 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.063+0000 7f8725800640 1 -- 172.21.15.121:0/361737431 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8720075d50 con 0x7f872015c930 2024-04-24T08:07:58.066 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.065+0000 7f8717e00640 1 -- 172.21.15.121:0/361737431 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f870c01f020 con 0x7f872015c930 2024-04-24T08:07:58.145 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.144+0000 7f8725800640 1 -- 172.21.15.121:0/361737431 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "9bbf8edc-5742-43a4-ba34-afe0e3f880bb", "id": 5} v 0) -- 0x7f872006f390 con 0x7f872015c930 2024-04-24T08:07:58.152 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.151+0000 7f8717e00640 1 -- 172.21.15.121:0/361737431 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9bbf8edc-5742-43a4-ba34-afe0e3f880bb", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f870c01a790 con 0x7f872015c930 2024-04-24T08:07:58.152 INFO:teuthology.orchestra.run.smithi121.stdout:5 2024-04-24T08:07:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.153+0000 7f8725800640 1 -- 172.21.15.121:0/361737431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f872015c930 msgr2=0x7f87200cabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.153+0000 7f8725800640 1 --2- 172.21.15.121:0/361737431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f872015c930 0x7f87200cabf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f870c00b410 tx=0x7f870c00b8d0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.153+0000 7f8725800640 1 -- 172.21.15.121:0/361737431 shutdown_connections 2024-04-24T08:07:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.153+0000 7f8725800640 1 --2- 172.21.15.121:0/361737431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87201602b0 0x7f872015e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.153+0000 7f8725800640 1 --2- 172.21.15.121:0/361737431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f872015c930 0x7f87200cabf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.153+0000 7f8725800640 1 --2- 172.21.15.121:0/361737431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f872006f5b0 0x7f872015cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.153+0000 7f8725800640 1 -- 172.21.15.121:0/361737431 >> 172.21.15.121:0/361737431 conn(0x7f872006cb30 msgr2=0x7f872006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.153+0000 7f8725800640 1 -- 172.21.15.121:0/361737431 shutdown_connections 2024-04-24T08:07:58.154 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.153+0000 7f8725800640 1 -- 172.21.15.121:0/361737431 wait complete. 2024-04-24T08:07:58.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 2edf233a-d123-4810-b90d-1a04dd625042 6 2024-04-24T08:07:58.340 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.340+0000 7f4021600640 1 Processor -- start 2024-04-24T08:07:58.341 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.340+0000 7f4021600640 1 -- start start 2024-04-24T08:07:58.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.341+0000 7f4021600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f401c06cfb0 0x7f401c06d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.341+0000 7f4021600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f401c15ed60 con 0x7f401c162f90 2024-04-24T08:07:58.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.342+0000 7f4021600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f401c0d1580 con 0x7f401c15e980 2024-04-24T08:07:58.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.342+0000 7f4021600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f401c1633d0 con 0x7f401c06cfb0 2024-04-24T08:07:58.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.342+0000 7f401be00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f401c15e980 0x7f401c0d0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:33852/0 (socket says 172.21.15.121:33852) 2024-04-24T08:07:58.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.342+0000 7f401be00640 1 -- 172.21.15.121:0/1397082407 learned_addr learned my addr 172.21.15.121:0/1397082407 (peer_addr_for_me v1:172.21.15.121:0/0) 2024-04-24T08:07:58.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.342+0000 7f4020c00640 1 --2- 172.21.15.121:0/1397082407 >> v2:172.21.15.69:3300/0 conn(0x7f401c06cfb0 0x7f401c06d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.342+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1780799135 0 0) 0x7f401c0d1580 con 0x7f401c15e980 2024-04-24T08:07:58.343 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.342+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4000003520 con 0x7f401c15e980 2024-04-24T08:07:58.345 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4017718478 0 0) 0x7f4000003520 con 0x7f401c15e980 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 >> v2:172.21.15.69:3300/0 conn(0x7f401c06cfb0 msgr2=0x7f401c06d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 --2- 172.21.15.121:0/1397082407 >> v2:172.21.15.69:3300/0 conn(0x7f401c06cfb0 0x7f401c06d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 >> v1:172.21.15.69:6789/0 conn(0x7f401c162f90 legacy=0x7f401c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401c0d1860 con 0x7f401c15e980 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f400c001670 con 0x7f401c15e980 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 >> v1:172.21.15.121:6789/0 conn(0x7f401c15e980 legacy=0x7f401c0d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4000003a00 0x7f4000003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000004800 0x7f4000004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40000051c0 0x7f400000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f401c1633d0 con 0x7f4000004800 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f400000c0a0 con 0x7f4000003a00 2024-04-24T08:07:58.346 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f400000c2a0 con 0x7f40000051c0 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401be00640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4000003a00 0x7f4000003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f4020c00640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40000051c0 0x7f400000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401b400640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000004800 0x7f4000004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401be00640 1 -- 172.21.15.121:0/1397082407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40000051c0 msgr2=0x7f400000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401be00640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40000051c0 0x7f400000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401be00640 1 -- 172.21.15.121:0/1397082407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000004800 msgr2=0x7f4000004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401be00640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000004800 0x7f4000004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401be00640 1 -- 172.21.15.121:0/1397082407 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4000003520 con 0x7f4000003a00 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.343+0000 7f401b400640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000004800 0x7f4000004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.344+0000 7f4020c00640 1 -- 172.21.15.121:0/1397082407 reap_dead start 2024-04-24T08:07:58.347 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.344+0000 7f401be00640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4000003a00 0x7f4000003e20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f400c00c110 tx=0x7f4000005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcd1420ae94c2b2c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:58.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.345+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c007c40 con 0x7f4000003a00 2024-04-24T08:07:58.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.345+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f400c005af0 con 0x7f4000003a00 2024-04-24T08:07:58.348 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.345+0000 7f401aa00640 1 -- 172.21.15.121:0/1397082407 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c007070 con 0x7f4000003a00 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.345+0000 7f4021600640 1 -- 172.21.15.121:0/1397082407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4000003a00 msgr2=0x7f4000003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.345+0000 7f4021600640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4000003a00 0x7f4000003e20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f400c00c110 tx=0x7f4000005830 comp rx=0 tx=0).stop 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.345+0000 7f4021600640 1 -- 172.21.15.121:0/1397082407 shutdown_connections 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.345+0000 7f4021600640 1 --2- 172.21.15.121:0/1397082407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4000003a00 0x7f4000003e20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.345+0000 7f4021600640 1 -- 172.21.15.121:0/1397082407 >> 172.21.15.121:0/1397082407 conn(0x7f401c09d920 msgr2=0x7f401c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.346+0000 7f4021600640 1 -- 172.21.15.121:0/1397082407 shutdown_connections 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.346+0000 7f4021600640 1 -- 172.21.15.121:0/1397082407 wait complete. 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.347+0000 7f4021600640 1 Processor -- start 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.347+0000 7f4021600640 1 -- start start 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.348+0000 7f4021600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000003a00 0x7f401c1f89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.351 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.348+0000 7f4021600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401c0b40e0 0x7f401c0b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.348+0000 7f4021600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f401c0b7350 0x7f401c0b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.348+0000 7f4021600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f401c0cf070 con 0x7f4000003a00 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.348+0000 7f4021600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f401c0cf730 con 0x7f401c0b7350 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.348+0000 7f4021600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f401c0cf1f0 con 0x7f401c0b40e0 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401b400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401c0b40e0 0x7f401c0b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f4020c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f401c0b7350 0x7f401c0b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401b400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401c0b40e0 0x7f401c0b13f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52898/0 (socket says 172.21.15.121:52898) 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401b400640 1 -- 172.21.15.121:0/2115293499 learned_addr learned my addr 172.21.15.121:0/2115293499 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401be00640 1 --2- 172.21.15.121:0/2115293499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000003a00 0x7f401c1f89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401b400640 1 -- 172.21.15.121:0/2115293499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f401c0b7350 msgr2=0x7f401c0b65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401b400640 1 --2- 172.21.15.121:0/2115293499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f401c0b7350 0x7f401c0b65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401b400640 1 -- 172.21.15.121:0/2115293499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000003a00 msgr2=0x7f401c1f89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.352 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401b400640 1 --2- 172.21.15.121:0/2115293499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000003a00 0x7f401c1f89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401b400640 1 -- 172.21.15.121:0/2115293499 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401c0b9b40 con 0x7f401c0b40e0 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f4020c00640 1 --2- 172.21.15.121:0/2115293499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f401c0b7350 0x7f401c0b65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401be00640 1 --2- 172.21.15.121:0/2115293499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000003a00 0x7f401c1f89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.349+0000 7f401b400640 1 --2- 172.21.15.121:0/2115293499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401c0b40e0 0x7f401c0b13f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f400800d4b0 tx=0x7f400800d970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.350+0000 7f4013e00640 1 -- 172.21.15.121:0/2115293499 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008005480 con 0x7f401c0b40e0 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.350+0000 7f4013e00640 1 -- 172.21.15.121:0/2115293499 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4008017070 con 0x7f401c0b40e0 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.350+0000 7f4013e00640 1 -- 172.21.15.121:0/2115293499 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4008003830 con 0x7f401c0b40e0 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.350+0000 7f4021600640 1 -- 172.21.15.121:0/2115293499 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f401c0b9e20 con 0x7f401c0b40e0 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.350+0000 7f4021600640 1 -- 172.21.15.121:0/2115293499 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f401c0b5da0 con 0x7f401c0b40e0 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.350+0000 7f4013e00640 1 -- 172.21.15.121:0/2115293499 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f40080039d0 con 0x7f401c0b40e0 2024-04-24T08:07:58.353 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.350+0000 7f4013e00640 1 -- 172.21.15.121:0/2115293499 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f4008003cc0 con 0x7f401c0b40e0 2024-04-24T08:07:58.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.351+0000 7f4021600640 1 -- 172.21.15.121:0/2115293499 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f401c075d50 con 0x7f401c0b40e0 2024-04-24T08:07:58.354 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.353+0000 7f4013e00640 1 -- 172.21.15.121:0/2115293499 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f400801c460 con 0x7f401c0b40e0 2024-04-24T08:07:58.433 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.433+0000 7f4021600640 1 -- 172.21.15.121:0/2115293499 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2edf233a-d123-4810-b90d-1a04dd625042", "id": 6} v 0) -- 0x7f401c0d1e40 con 0x7f401c0b40e0 2024-04-24T08:07:58.440 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.439+0000 7f4013e00640 1 -- 172.21.15.121:0/2115293499 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2edf233a-d123-4810-b90d-1a04dd625042", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f400801c710 con 0x7f401c0b40e0 2024-04-24T08:07:58.440 INFO:teuthology.orchestra.run.smithi121.stdout:6 2024-04-24T08:07:58.441 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.441+0000 7f4021600640 1 -- 172.21.15.121:0/2115293499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401c0b40e0 msgr2=0x7f401c0b13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.441+0000 7f4021600640 1 --2- 172.21.15.121:0/2115293499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401c0b40e0 0x7f401c0b13f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f400800d4b0 tx=0x7f400800d970 comp rx=0 tx=0).stop 2024-04-24T08:07:58.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.441+0000 7f4021600640 1 -- 172.21.15.121:0/2115293499 shutdown_connections 2024-04-24T08:07:58.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.441+0000 7f4021600640 1 --2- 172.21.15.121:0/2115293499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f401c0b7350 0x7f401c0b65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.441+0000 7f4021600640 1 --2- 172.21.15.121:0/2115293499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f401c0b40e0 0x7f401c0b13f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.441+0000 7f4021600640 1 --2- 172.21.15.121:0/2115293499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4000003a00 0x7f401c1f89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.441+0000 7f4021600640 1 -- 172.21.15.121:0/2115293499 >> 172.21.15.121:0/2115293499 conn(0x7f401c09d920 msgr2=0x7f401c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:58.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.441+0000 7f4021600640 1 -- 172.21.15.121:0/2115293499 shutdown_connections 2024-04-24T08:07:58.442 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.441+0000 7f4021600640 1 -- 172.21.15.121:0/2115293499 wait complete. 2024-04-24T08:07:58.456 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:58.454+0000 7f6452f1c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:07:58.459 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0 7 2024-04-24T08:07:58.459 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:58.458+0000 7f96d3614040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:07:58.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.634+0000 7fcca9800640 1 Processor -- start 2024-04-24T08:07:58.635 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.634+0000 7fcca9800640 1 -- start start 2024-04-24T08:07:58.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.636+0000 7fcca9800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcca406ef40 0x7fcca4073b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.636+0000 7fcca9800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca406ba20 0x7fcca406beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.636+0000 7fcca9800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcca415ee50 con 0x7fcca415ea70 2024-04-24T08:07:58.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.636+0000 7fcca9800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcca40cdcb0 con 0x7fcca406ef40 2024-04-24T08:07:58.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.636+0000 7fcca9800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcca406f4b0 con 0x7fcca406ba20 2024-04-24T08:07:58.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.636+0000 7fcc9be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcca406ef40 0x7fcca4073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.636+0000 7fcca8e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca406ba20 0x7fcca406beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.636+0000 7fcca8e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca406ba20 0x7fcca406beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:52914/0 (socket says 172.21.15.121:52914) 2024-04-24T08:07:58.637 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.636+0000 7fcca8e00640 1 -- 172.21.15.121:0/1272958125 learned_addr learned my addr 172.21.15.121:0/1272958125 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:58.638 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.637+0000 7fcca8e00640 1 -- 172.21.15.121:0/1272958125 >> v2:172.21.15.121:3300/0 conn(0x7fcca406ef40 msgr2=0x7fcca4073b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.637+0000 7fcca8e00640 1 --2- 172.21.15.121:0/1272958125 >> v2:172.21.15.121:3300/0 conn(0x7fcca406ef40 0x7fcca4073b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.637+0000 7fcca8e00640 1 -- 172.21.15.121:0/1272958125 >> v1:172.21.15.69:6789/0 conn(0x7fcca415ea70 legacy=0x7fcca40cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.637+0000 7fcca8e00640 1 -- 172.21.15.121:0/1272958125 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca406f790 con 0x7fcca406ba20 2024-04-24T08:07:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.637+0000 7fcc9be00640 1 --2- 172.21.15.121:0/1272958125 >> v2:172.21.15.121:3300/0 conn(0x7fcca406ef40 0x7fcca4073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.638+0000 7fcca8e00640 1 --2- 172.21.15.121:0/1272958125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca406ba20 0x7fcca406beb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009e50 tx=0x7fcc9c02a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55994d4030d9b48c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.638+0000 7fcca3400640 1 -- 172.21.15.121:0/1272958125 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c036070 con 0x7fcca406ba20 2024-04-24T08:07:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.638+0000 7fcca3400640 1 -- 172.21.15.121:0/1272958125 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc9c031600 con 0x7fcca406ba20 2024-04-24T08:07:58.639 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.638+0000 7fcca3400640 1 -- 172.21.15.121:0/1272958125 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c0319d0 con 0x7fcca406ba20 2024-04-24T08:07:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.639+0000 7fcca9800640 1 -- 172.21.15.121:0/1272958125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca406ba20 msgr2=0x7fcca406beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.639+0000 7fcca9800640 1 --2- 172.21.15.121:0/1272958125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca406ba20 0x7fcca406beb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009e50 tx=0x7fcc9c02a680 comp rx=0 tx=0).stop 2024-04-24T08:07:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.639+0000 7fcca9800640 1 -- 172.21.15.121:0/1272958125 shutdown_connections 2024-04-24T08:07:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.639+0000 7fcca9800640 1 --2- 172.21.15.121:0/1272958125 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca406ba20 0x7fcca406beb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.639+0000 7fcca9800640 1 --2- 172.21.15.121:0/1272958125 >> v2:172.21.15.121:3300/0 conn(0x7fcca406ef40 0x7fcca4073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.639+0000 7fcca9800640 1 -- 172.21.15.121:0/1272958125 >> 172.21.15.121:0/1272958125 conn(0x7fcca406cb30 msgr2=0x7fcca406cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:58.640 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.639+0000 7fcca9800640 1 -- 172.21.15.121:0/1272958125 shutdown_connections 2024-04-24T08:07:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.639+0000 7fcca9800640 1 -- 172.21.15.121:0/1272958125 wait complete. 2024-04-24T08:07:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.641+0000 7fcca9800640 1 Processor -- start 2024-04-24T08:07:58.641 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.641+0000 7fcca9800640 1 -- start start 2024-04-24T08:07:58.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca9800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcca406ba20 0x7fcca40c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca9800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca406ef40 0x7fcca415a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca9800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca415ea70 0x7fcca40b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.644 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca9800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcca40b0290 con 0x7fcca406ef40 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca9800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcca406fd50 con 0x7fcca406ba20 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca9800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcca40cfef0 con 0x7fcca415ea70 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca3e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcca406ba20 0x7fcca40c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca3e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcca406ba20 0x7fcca40c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:60096/0 (socket says 172.21.15.121:60096) 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca3e00640 1 -- 172.21.15.121:0/298249971 learned_addr learned my addr 172.21.15.121:0/298249971 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca8e00640 1 --2- 172.21.15.121:0/298249971 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca415ea70 0x7fcca40b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcc9be00640 1 --2- 172.21.15.121:0/298249971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca406ef40 0x7fcca415a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca3e00640 1 -- 172.21.15.121:0/298249971 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca415ea70 msgr2=0x7fcca40b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.642+0000 7fcca3e00640 1 --2- 172.21.15.121:0/298249971 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca415ea70 0x7fcca40b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.643+0000 7fcca3e00640 1 -- 172.21.15.121:0/298249971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca406ef40 msgr2=0x7fcca415a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.643+0000 7fcca3e00640 1 --2- 172.21.15.121:0/298249971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca406ef40 0x7fcca415a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.643+0000 7fcca3e00640 1 -- 172.21.15.121:0/298249971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca415f270 con 0x7fcca406ba20 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.643+0000 7fcc9be00640 1 --2- 172.21.15.121:0/298249971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca406ef40 0x7fcca415a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:07:58.645 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.643+0000 7fcca3e00640 1 --2- 172.21.15.121:0/298249971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcca406ba20 0x7fcca40c8340 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400c400 tx=0x7fcc9400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:58.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.643+0000 7fcca0c00640 1 -- 172.21.15.121:0/298249971 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc940052d0 con 0x7fcca406ba20 2024-04-24T08:07:58.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.643+0000 7fcca0c00640 1 -- 172.21.15.121:0/298249971 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc94005490 con 0x7fcca406ba20 2024-04-24T08:07:58.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.643+0000 7fcca0c00640 1 -- 172.21.15.121:0/298249971 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc94007260 con 0x7fcca406ba20 2024-04-24T08:07:58.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.643+0000 7fcca9800640 1 -- 172.21.15.121:0/298249971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca415f550 con 0x7fcca406ba20 2024-04-24T08:07:58.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.643+0000 7fcca9800640 1 -- 172.21.15.121:0/298249971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca415fb10 con 0x7fcca406ba20 2024-04-24T08:07:58.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.644+0000 7fcca0c00640 1 -- 172.21.15.121:0/298249971 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcc94007400 con 0x7fcca406ba20 2024-04-24T08:07:58.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.644+0000 7fcca0c00640 1 -- 172.21.15.121:0/298249971 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fcc940035f0 con 0x7fcca406ba20 2024-04-24T08:07:58.646 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.644+0000 7fcca9800640 1 -- 172.21.15.121:0/298249971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca4075d50 con 0x7fcca406ba20 2024-04-24T08:07:58.647 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.646+0000 7fcca0c00640 1 -- 172.21.15.121:0/298249971 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fcc94003a50 con 0x7fcca406ba20 2024-04-24T08:07:58.725 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.724+0000 7fcca9800640 1 -- 172.21.15.121:0/298249971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0", "id": 7} v 0) -- 0x7fcca415ba80 con 0x7fcca406ba20 2024-04-24T08:07:58.738 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.737+0000 7fcca0c00640 1 -- 172.21.15.121:0/298249971 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fcc94003c30 con 0x7fcca406ba20 2024-04-24T08:07:58.738 INFO:teuthology.orchestra.run.smithi121.stdout:7 2024-04-24T08:07:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.739+0000 7fcca9800640 1 -- 172.21.15.121:0/298249971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcca406ba20 msgr2=0x7fcca40c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.739+0000 7fcca9800640 1 --2- 172.21.15.121:0/298249971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcca406ba20 0x7fcca40c8340 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400c400 tx=0x7fcc9400c8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.739+0000 7fcca9800640 1 -- 172.21.15.121:0/298249971 shutdown_connections 2024-04-24T08:07:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.739+0000 7fcca9800640 1 --2- 172.21.15.121:0/298249971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcca406ef40 0x7fcca415a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.739+0000 7fcca9800640 1 --2- 172.21.15.121:0/298249971 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcca415ea70 0x7fcca40b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.739+0000 7fcca9800640 1 --2- 172.21.15.121:0/298249971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcca406ba20 0x7fcca40c8340 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.739+0000 7fcca9800640 1 -- 172.21.15.121:0/298249971 >> 172.21.15.121:0/298249971 conn(0x7fcca406cb30 msgr2=0x7fcca4072a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.739+0000 7fcca9800640 1 -- 172.21.15.121:0/298249971 shutdown_connections 2024-04-24T08:07:58.740 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:07:58.739+0000 7fcca9800640 1 -- 172.21.15.121:0/298249971 wait complete. 2024-04-24T08:07:58.753 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:58.751+0000 7f6452f1c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:07:58.756 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-24T08:07:58.756 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-24T08:07:58.762 INFO:tasks.ceph.osd.0:Started 2024-04-24T08:07:58.762 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-24T08:07:58.762 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-24T08:07:58.771 INFO:tasks.ceph.osd.1:Started 2024-04-24T08:07:58.771 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-24T08:07:58.772 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-24T08:07:58.772 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:58.770+0000 7f96d3614040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:07:58.777 INFO:tasks.ceph.osd.2:Started 2024-04-24T08:07:58.777 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-24T08:07:58.777 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-24T08:07:58.782 INFO:tasks.ceph.osd.3:Started 2024-04-24T08:07:58.782 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-24T08:07:58.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-24T08:07:58.789 INFO:tasks.ceph.osd.4:Started 2024-04-24T08:07:58.789 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-24T08:07:58.789 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-24T08:07:58.795 INFO:tasks.ceph.osd.5:Started 2024-04-24T08:07:58.795 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-24T08:07:58.796 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-24T08:07:58.800 INFO:tasks.ceph.osd.6:Started 2024-04-24T08:07:58.800 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-24T08:07:58.800 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-24T08:07:58.805 INFO:tasks.ceph.osd.7:Started 2024-04-24T08:07:58.805 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:07:58.831 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:58.830+0000 7f6452f1c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:07:58.844 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:58.843+0000 7f96d3614040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:07:59.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.004+0000 7f7b7d600640 1 Processor -- start 2024-04-24T08:07:59.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.004+0000 7f7b7d600640 1 -- start start 2024-04-24T08:07:59.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b7d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b780cea60 0x7f7b7806d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b7d600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7b7806dd90 0x7f7b7809a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b7d600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b780d0db0 con 0x7f7b780d0820 2024-04-24T08:07:59.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b7d600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7b780cee40 con 0x7f7b7806dd90 2024-04-24T08:07:59.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b7d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7b780d1090 con 0x7f7b780cea60 2024-04-24T08:07:59.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b77400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7b780d0820 0x7f7b7815e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:41910/0 (socket says 172.21.15.69:41910) 2024-04-24T08:07:59.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b77400640 1 -- 172.21.15.69:0/1156324544 learned_addr learned my addr 172.21.15.69:0/1156324544 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:07:59.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b6fe00640 1 --2- 172.21.15.69:0/1156324544 >> v2:172.21.15.121:3300/0 conn(0x7f7b7806dd90 0x7f7b7809a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b76a00640 1 --2- 172.21.15.69:0/1156324544 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b780cea60 0x7f7b7806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b76000640 1 -- 172.21.15.69:0/1156324544 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2789741162 0 0) 0x7f7b780d0db0 con 0x7f7b780d0820 2024-04-24T08:07:59.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b76000640 1 -- 172.21.15.69:0/1156324544 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b5c003520 con 0x7f7b780d0820 2024-04-24T08:07:59.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b6fe00640 1 -- 172.21.15.69:0/1156324544 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b780cea60 msgr2=0x7f7b7806d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:07:59.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b6fe00640 1 --2- 172.21.15.69:0/1156324544 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b780cea60 0x7f7b7806d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b6fe00640 1 -- 172.21.15.69:0/1156324544 >> v1:172.21.15.69:6789/0 conn(0x7f7b780d0820 legacy=0x7f7b7815e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.006+0000 7f7b6fe00640 1 -- 172.21.15.69:0/1156324544 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b780d1370 con 0x7f7b7806dd90 2024-04-24T08:07:59.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.007+0000 7f7b6fe00640 1 --2- 172.21.15.69:0/1156324544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b7806dd90 0x7f7b7809a9d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7b600097a0 tx=0x7f7b6002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f7d1794bd3b8907 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:59.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.008+0000 7f7b76000640 1 -- 172.21.15.69:0/1156324544 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60037070 con 0x7f7b7806dd90 2024-04-24T08:07:59.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.008+0000 7f7b76000640 1 -- 172.21.15.69:0/1156324544 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b60032440 con 0x7f7b7806dd90 2024-04-24T08:07:59.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.008+0000 7f7b76000640 1 -- 172.21.15.69:0/1156324544 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60032810 con 0x7f7b7806dd90 2024-04-24T08:07:59.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.008+0000 7f7b7d600640 1 -- 172.21.15.69:0/1156324544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b7806dd90 msgr2=0x7f7b7809a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.008+0000 7f7b7d600640 1 --2- 172.21.15.69:0/1156324544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b7806dd90 0x7f7b7809a9d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7b600097a0 tx=0x7f7b6002a180 comp rx=0 tx=0).stop 2024-04-24T08:07:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.008+0000 7f7b7d600640 1 -- 172.21.15.69:0/1156324544 shutdown_connections 2024-04-24T08:07:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.008+0000 7f7b7d600640 1 --2- 172.21.15.69:0/1156324544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b7806dd90 0x7f7b7809a9d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.008+0000 7f7b7d600640 1 --2- 172.21.15.69:0/1156324544 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b780cea60 0x7f7b7806d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.008+0000 7f7b7d600640 1 -- 172.21.15.69:0/1156324544 >> 172.21.15.69:0/1156324544 conn(0x7f7b780b4040 msgr2=0x7f7b780b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.008+0000 7f7b7d600640 1 -- 172.21.15.69:0/1156324544 shutdown_connections 2024-04-24T08:07:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.009+0000 7f7b7d600640 1 -- 172.21.15.69:0/1156324544 wait complete. 2024-04-24T08:07:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.010+0000 7f7b7d600640 1 Processor -- start 2024-04-24T08:07:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.010+0000 7f7b7d600640 1 -- start start 2024-04-24T08:07:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b7d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b7806dd90 0x7f7b78145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b7d600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b780cea60 0x7f7b78144470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b7d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b780d0820 0x7f7b780b7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b7d600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b7809af10 con 0x7f7b7806dd90 2024-04-24T08:07:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b7d600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7b780d1930 con 0x7f7b780cea60 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b7d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7b7809b090 con 0x7f7b780d0820 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b76a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b7806dd90 0x7f7b78145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b76a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b7806dd90 0x7f7b78145d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33836/0 (socket says 172.21.15.69:33836) 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b77400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b780d0820 0x7f7b780b7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b76a00640 1 -- 172.21.15.69:0/3931391825 learned_addr learned my addr 172.21.15.69:0/3931391825 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.011+0000 7f7b6fe00640 1 --2- 172.21.15.69:0/3931391825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b780cea60 0x7f7b78144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.012+0000 7f7b6fe00640 1 -- 172.21.15.69:0/3931391825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b780d0820 msgr2=0x7f7b780b7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.012+0000 7f7b6fe00640 1 --2- 172.21.15.69:0/3931391825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b780d0820 0x7f7b780b7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.012+0000 7f7b6fe00640 1 -- 172.21.15.69:0/3931391825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b7806dd90 msgr2=0x7f7b78145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.012+0000 7f7b6fe00640 1 --2- 172.21.15.69:0/3931391825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b7806dd90 0x7f7b78145d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.012+0000 7f7b6fe00640 1 -- 172.21.15.69:0/3931391825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b78146280 con 0x7f7b780cea60 2024-04-24T08:07:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.012+0000 7f7b6fe00640 1 --2- 172.21.15.69:0/3931391825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b780cea60 0x7f7b78144470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b780734e0 tx=0x7f7b600046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:59.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.012+0000 7f7b6ea00640 1 -- 172.21.15.69:0/3931391825 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60041030 con 0x7f7b780cea60 2024-04-24T08:07:59.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.012+0000 7f7b6ea00640 1 -- 172.21.15.69:0/3931391825 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b6003d4e0 con 0x7f7b780cea60 2024-04-24T08:07:59.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.012+0000 7f7b7d600640 1 -- 172.21.15.69:0/3931391825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b78144ca0 con 0x7f7b780cea60 2024-04-24T08:07:59.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.013+0000 7f7b7d600640 1 -- 172.21.15.69:0/3931391825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b78145260 con 0x7f7b780cea60 2024-04-24T08:07:59.016 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:59.013+0000 7f6452f1c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:07:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.013+0000 7f7b6ea00640 1 -- 172.21.15.69:0/3931391825 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b60037040 con 0x7f7b780cea60 2024-04-24T08:07:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.014+0000 7f7b6ea00640 1 -- 172.21.15.69:0/3931391825 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7b6003d950 con 0x7f7b780cea60 2024-04-24T08:07:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.014+0000 7f7b6ea00640 1 -- 172.21.15.69:0/3931391825 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f7b6003bac0 con 0x7f7b780cea60 2024-04-24T08:07:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.014+0000 7f7b7d600640 1 -- 172.21.15.69:0/3931391825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b780d2260 con 0x7f7b780cea60 2024-04-24T08:07:59.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.016+0000 7f7b6ea00640 1 -- 172.21.15.69:0/3931391825 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7b780d2260 con 0x7f7b780cea60 2024-04-24T08:07:59.023 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:59.022+0000 7f96d3614040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:07:59.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.080+0000 7f7b7d600640 1 -- 172.21.15.69:0/3931391825 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b780b4cd0 con 0x7f7b780cea60 2024-04-24T08:07:59.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.082+0000 7f7b6ea00640 1 -- 172.21.15.69:0/3931391825 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f7b6003ddd0 con 0x7f7b780cea60 2024-04-24T08:07:59.083 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:07:59.083 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":10,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:07:58.727306+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:07:59.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.083+0000 7f7b7d600640 1 -- 172.21.15.69:0/3931391825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b780cea60 msgr2=0x7f7b78144470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.083+0000 7f7b7d600640 1 --2- 172.21.15.69:0/3931391825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b780cea60 0x7f7b78144470 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7b780734e0 tx=0x7f7b600046c0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.083+0000 7f7b7d600640 1 -- 172.21.15.69:0/3931391825 shutdown_connections 2024-04-24T08:07:59.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.083+0000 7f7b7d600640 1 --2- 172.21.15.69:0/3931391825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7b780d0820 0x7f7b780b7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.083+0000 7f7b7d600640 1 --2- 172.21.15.69:0/3931391825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7b780cea60 0x7f7b78144470 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.083+0000 7f7b7d600640 1 --2- 172.21.15.69:0/3931391825 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b7806dd90 0x7f7b78145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.083+0000 7f7b7d600640 1 -- 172.21.15.69:0/3931391825 >> 172.21.15.69:0/3931391825 conn(0x7f7b780b4040 msgr2=0x7f7b78074b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:59.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.083+0000 7f7b7d600640 1 -- 172.21.15.69:0/3931391825 shutdown_connections 2024-04-24T08:07:59.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:07:59.083+0000 7f7b7d600640 1 -- 172.21.15.69:0/3931391825 wait complete. 2024-04-24T08:07:59.099 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-24T08:07:59.099 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T08:07:59.352 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:59.351+0000 7f6452f1c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:07:59.355 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:59.354+0000 7f96d3614040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:07:59.408 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:59.406+0000 7f6452f1c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:07:59.410 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:59.409+0000 7f96d3614040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:07:59.447 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-04-24T08:07:59.446+0000 7f9c9ce0e640 -1 Falling back to public interface 2024-04-24T08:07:59.476 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-24T08:07:59.476+0000 7f2ee8144640 -1 Falling back to public interface 2024-04-24T08:07:59.480 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-24T08:07:59.479+0000 7f696c063640 -1 Falling back to public interface 2024-04-24T08:07:59.481 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-24T08:07:59.480+0000 7f4eee6fd640 -1 Falling back to public interface 2024-04-24T08:07:59.489 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-04-24T08:07:59.488+0000 7f31ec832640 -1 Falling back to public interface 2024-04-24T08:07:59.490 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-04-24T08:07:59.489+0000 7ffbdda16640 -1 Falling back to public interface 2024-04-24T08:07:59.493 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-04-24T08:07:59.492+0000 7f8ded82c640 -1 Falling back to public interface 2024-04-24T08:07:59.497 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-24T08:07:59.496+0000 7f2df45c0640 -1 Falling back to public interface 2024-04-24T08:07:59.499 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-04-24T08:07:59.498+0000 7f31eb200640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6821/0: (98) Address already in use 2024-04-24T08:07:59.599 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:59.598+0000 7f6452f1c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:07:59.601 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:59.600+0000 7f96d3614040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:07:59.765 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:59.764+0000 7f96d3614040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:07:59.766 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:59.764+0000 7f6452f1c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:07:59.826 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:59.825+0000 7f96d3614040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:07:59.826 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:59.824+0000 7f6452f1c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:07:59.906 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T08:07:59.907 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:59.905+0000 7f96d3614040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:07:59.914 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:59.907+0000 7f6452f1c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:07:59.959 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:07:59.958+0000 7f96d3614040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:07:59.964 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:07:59.962+0000 7f6452f1c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:08:00.063 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.061+0000 7f32d3200640 1 Processor -- start 2024-04-24T08:08:00.063 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.061+0000 7f32d3200640 1 -- start start 2024-04-24T08:08:00.065 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.063+0000 7f32d3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f32cc0c49a0 0x7f32cc0b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.065 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.063+0000 7f32d3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32cc0cde30 con 0x7f32cc15c530 2024-04-24T08:08:00.065 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.063+0000 7f32d3200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32cc0ce110 con 0x7f32cc06c130 2024-04-24T08:08:00.065 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.063+0000 7f32d3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f32cc0ce3f0 con 0x7f32cc0c49a0 2024-04-24T08:08:00.065 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.063+0000 7f32d1e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f32cc15c530 0x7f32cc06ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:41916/0 (socket says 172.21.15.69:41916) 2024-04-24T08:08:00.065 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.063+0000 7f32d1e00640 1 -- 172.21.15.69:0/715232608 learned_addr learned my addr 172.21.15.69:0/715232608 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:00.065 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.063+0000 7f32d2800640 1 --2- 172.21.15.69:0/715232608 >> v2:172.21.15.69:3300/0 conn(0x7f32cc0c49a0 0x7f32cc0b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.066 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.064+0000 7f32d2800640 1 -- 172.21.15.69:0/715232608 >> v1:172.21.15.121:6789/0 conn(0x7f32cc06c130 legacy=0x7f32cc0b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.066 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.064+0000 7f32d2800640 1 -- 172.21.15.69:0/715232608 >> v1:172.21.15.69:6789/0 conn(0x7f32cc15c530 legacy=0x7f32cc06ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.066 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.064+0000 7f32d2800640 1 -- 172.21.15.69:0/715232608 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32cc0ce6d0 con 0x7f32cc0c49a0 2024-04-24T08:08:00.066 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.065+0000 7f32d2800640 1 --2- 172.21.15.69:0/715232608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc0c49a0 0x7f32cc0b13f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009d90 tx=0x7f32c802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8099f0ac72a9e52c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:00.067 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.066+0000 7f32d0a00640 1 -- 172.21.15.69:0/715232608 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8007740 con 0x7f32cc0c49a0 2024-04-24T08:08:00.067 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.066+0000 7f32d0a00640 1 -- 172.21.15.69:0/715232608 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c80078e0 con 0x7f32cc0c49a0 2024-04-24T08:08:00.067 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.066+0000 7f32d0a00640 1 -- 172.21.15.69:0/715232608 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8007cb0 con 0x7f32cc0c49a0 2024-04-24T08:08:00.068 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.066+0000 7f32d3200640 1 -- 172.21.15.69:0/715232608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc0c49a0 msgr2=0x7f32cc0b13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.069 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.066+0000 7f32d3200640 1 --2- 172.21.15.69:0/715232608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc0c49a0 0x7f32cc0b13f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009d90 tx=0x7f32c802a660 comp rx=0 tx=0).stop 2024-04-24T08:08:00.069 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.066+0000 7f32d3200640 1 -- 172.21.15.69:0/715232608 shutdown_connections 2024-04-24T08:08:00.069 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.066+0000 7f32d3200640 1 --2- 172.21.15.69:0/715232608 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc0c49a0 0x7f32cc0b13f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.069 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.066+0000 7f32d3200640 1 -- 172.21.15.69:0/715232608 >> 172.21.15.69:0/715232608 conn(0x7f32cc0b2e40 msgr2=0x7f32cc0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:00.069 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.066+0000 7f32d3200640 1 -- 172.21.15.69:0/715232608 shutdown_connections 2024-04-24T08:08:00.069 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.066+0000 7f32d3200640 1 -- 172.21.15.69:0/715232608 wait complete. 2024-04-24T08:08:00.070 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.068+0000 7f32d3200640 1 Processor -- start 2024-04-24T08:08:00.070 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.068+0000 7f32d3200640 1 -- start start 2024-04-24T08:08:00.072 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d3200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32cc06c130 0x7f32cc15f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.072 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32cc0c49a0 0x7f32cc15df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.072 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc15c530 0x7f32cc15a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.072 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc15ca90 con 0x7f32cc15c530 2024-04-24T08:08:00.072 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d3200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f32cc15c910 con 0x7f32cc06c130 2024-04-24T08:08:00.072 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f32cc0b3960 con 0x7f32cc0c49a0 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc15c530 0x7f32cc15a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc15c530 0x7f32cc15a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33852/0 (socket says 172.21.15.69:33852) 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d2800640 1 -- 172.21.15.69:0/932453062 learned_addr learned my addr 172.21.15.69:0/932453062 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d1e00640 1 --2- 172.21.15.69:0/932453062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32cc06c130 0x7f32cc15f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.069+0000 7f32d1400640 1 --2- 172.21.15.69:0/932453062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32cc0c49a0 0x7f32cc15df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32d2800640 1 -- 172.21.15.69:0/932453062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32cc0c49a0 msgr2=0x7f32cc15df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32d2800640 1 --2- 172.21.15.69:0/932453062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32cc0c49a0 0x7f32cc15df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32d2800640 1 -- 172.21.15.69:0/932453062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32cc06c130 msgr2=0x7f32cc15f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32d2800640 1 --2- 172.21.15.69:0/932453062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32cc06c130 0x7f32cc15f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32d2800640 1 -- 172.21.15.69:0/932453062 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32cc15ce90 con 0x7f32cc15c530 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32d1400640 1 --2- 172.21.15.69:0/932453062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32cc0c49a0 0x7f32cc15df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32d2800640 1 --2- 172.21.15.69:0/932453062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc15c530 0x7f32cc15a8d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009eb0 tx=0x7f32c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:00.073 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32c6000640 1 -- 172.21.15.69:0/932453062 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8040070 con 0x7f32cc15c530 2024-04-24T08:08:00.074 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32c6000640 1 -- 172.21.15.69:0/932453062 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c8031bc0 con 0x7f32cc15c530 2024-04-24T08:08:00.074 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32d3200640 1 -- 172.21.15.69:0/932453062 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32cc15d170 con 0x7f32cc15c530 2024-04-24T08:08:00.074 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32c6000640 1 -- 172.21.15.69:0/932453062 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c8045460 con 0x7f32cc15c530 2024-04-24T08:08:00.074 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.070+0000 7f32d3200640 1 -- 172.21.15.69:0/932453062 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32cc15d730 con 0x7f32cc15c530 2024-04-24T08:08:00.074 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.071+0000 7f32c6000640 1 -- 172.21.15.69:0/932453062 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f32c8039040 con 0x7f32cc15c530 2024-04-24T08:08:00.074 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.071+0000 7f32c6000640 1 -- 172.21.15.69:0/932453062 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f32c803caf0 con 0x7f32cc15c530 2024-04-24T08:08:00.075 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.072+0000 7f32d3200640 1 -- 172.21.15.69:0/932453062 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32cc04fd90 con 0x7f32cc15c530 2024-04-24T08:08:00.075 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.074+0000 7f32c6000640 1 -- 172.21.15.69:0/932453062 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f32c803cdd0 con 0x7f32cc15c530 2024-04-24T08:08:00.142 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.141+0000 7f32d3200640 1 -- 172.21.15.69:0/932453062 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32cc074f70 con 0x7f32cc15c530 2024-04-24T08:08:00.143 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.141+0000 7f32c6000640 1 -- 172.21.15.69:0/932453062 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f32c8007840 con 0x7f32cc15c530 2024-04-24T08:08:00.143 INFO:teuthology.misc.health.smithi069.stdout: 2024-04-24T08:08:00.143 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":10,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:07:58.727306+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:00.144 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.143+0000 7f32d3200640 1 -- 172.21.15.69:0/932453062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc15c530 msgr2=0x7f32cc15a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.144 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.143+0000 7f32d3200640 1 --2- 172.21.15.69:0/932453062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc15c530 0x7f32cc15a8d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f32c8009eb0 tx=0x7f32c8004a40 comp rx=0 tx=0).stop 2024-04-24T08:08:00.144 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.143+0000 7f32d3200640 1 -- 172.21.15.69:0/932453062 shutdown_connections 2024-04-24T08:08:00.145 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.143+0000 7f32d3200640 1 --2- 172.21.15.69:0/932453062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32cc15c530 0x7f32cc15a8d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.145 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.143+0000 7f32d3200640 1 --2- 172.21.15.69:0/932453062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f32cc0c49a0 0x7f32cc15df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.145 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.143+0000 7f32d3200640 1 --2- 172.21.15.69:0/932453062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32cc06c130 0x7f32cc15f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.145 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.143+0000 7f32d3200640 1 -- 172.21.15.69:0/932453062 >> 172.21.15.69:0/932453062 conn(0x7f32cc0b2e40 msgr2=0x7f32cc0b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:00.145 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.143+0000 7f32d3200640 1 -- 172.21.15.69:0/932453062 shutdown_connections 2024-04-24T08:08:00.145 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:00.143+0000 7f32d3200640 1 -- 172.21.15.69:0/932453062 wait complete. 2024-04-24T08:08:00.157 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-24T08:08:00.506 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:08:00.505+0000 7f96d3614040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:08:00.518 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:08:00.517+0000 7f6452f1c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:08:00.639 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:08:00.638+0000 7f96d3614040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:08:00.654 INFO:tasks.ceph.mgr.y.smithi069.stderr:2024-04-24T08:08:00.653+0000 7f6452f1c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:08:01.687 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:08:01.686+0000 7f9673a00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-24T08:08:01.687 INFO:tasks.ceph.mgr.x.smithi121.stderr:2024-04-24T08:08:01.686+0000 7f9673a00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-24T08:08:02.867 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-24T08:08:02.866+0000 7f2df45c0640 -1 osd.5 0 log_to_monitors true 2024-04-24T08:08:03.075 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-04-24T08:08:03.074+0000 7f9c9ce0e640 -1 osd.0 0 log_to_monitors true 2024-04-24T08:08:03.095 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-24T08:08:03.095+0000 7f2ee8144640 -1 osd.4 0 log_to_monitors true 2024-04-24T08:08:03.121 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-04-24T08:08:03.120+0000 7ffbdda16640 -1 osd.3 0 log_to_monitors true 2024-04-24T08:08:03.125 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-24T08:08:03.124+0000 7f696c063640 -1 osd.6 0 log_to_monitors true 2024-04-24T08:08:03.135 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-24T08:08:03.134+0000 7f4eee6fd640 -1 osd.7 0 log_to_monitors true 2024-04-24T08:08:03.152 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-04-24T08:08:03.150+0000 7f31ec832640 -1 osd.2 0 log_to_monitors true 2024-04-24T08:08:03.153 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-04-24T08:08:03.152+0000 7f8ded82c640 -1 osd.1 0 log_to_monitors true 2024-04-24T08:08:06.947 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-04-24T08:08:06.947+0000 7ffbd8200640 -1 osd.3 0 waiting for initial osdmap 2024-04-24T08:08:06.953 INFO:tasks.ceph.osd.3.smithi069.stderr:2024-04-24T08:08:06.953+0000 7ffbce200640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:08:06.964 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:06.979 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-04-24T08:08:06.979+0000 7f31e7600640 -1 osd.2 0 waiting for initial osdmap 2024-04-24T08:08:06.986 INFO:tasks.ceph.osd.2.smithi069.stderr:2024-04-24T08:08:06.986+0000 7f31dd600640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:08:06.988 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-04-24T08:08:06.987+0000 7f9c97600640 -1 osd.0 0 waiting for initial osdmap 2024-04-24T08:08:06.996 INFO:tasks.ceph.osd.0.smithi069.stderr:2024-04-24T08:08:06.996+0000 7f9c8d600640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:08:07.000 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-04-24T08:08:07.001+0000 7f8de7c00640 -1 osd.1 0 waiting for initial osdmap 2024-04-24T08:08:07.002 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-24T08:08:07.002+0000 7f6967000640 -1 osd.6 0 waiting for initial osdmap 2024-04-24T08:08:07.005 INFO:tasks.ceph.osd.1.smithi069.stderr:2024-04-24T08:08:07.005+0000 7f8dddc00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:08:07.010 INFO:tasks.ceph.osd.6.smithi121.stderr:2024-04-24T08:08:07.009+0000 7f695d000640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:08:07.037 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-24T08:08:07.037+0000 7f4ee9600640 -1 osd.7 0 waiting for initial osdmap 2024-04-24T08:08:07.042 INFO:tasks.ceph.osd.7.smithi121.stderr:2024-04-24T08:08:07.042+0000 7f4edf600640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:08:07.046 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-24T08:08:07.046+0000 7f2ee3a00640 -1 osd.4 0 waiting for initial osdmap 2024-04-24T08:08:07.060 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-24T08:08:07.060+0000 7f2def400640 -1 osd.5 0 waiting for initial osdmap 2024-04-24T08:08:07.064 INFO:tasks.ceph.osd.4.smithi121.stderr:2024-04-24T08:08:07.063+0000 7f2ed9000640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:08:07.069 INFO:tasks.ceph.osd.5.smithi121.stderr:2024-04-24T08:08:07.068+0000 7f2de5400640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:08:07.115 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.114+0000 7f0d07400640 1 Processor -- start 2024-04-24T08:08:07.115 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.114+0000 7f0d07400640 1 -- start start 2024-04-24T08:08:07.117 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0d07400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d00160990 0x7f0d001669e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:07.117 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0d07400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d000b4450 0x7f0d001673d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:07.117 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0d07400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d00164350 con 0x7f0d00166f20 2024-04-24T08:08:07.117 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0d07400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0d000b48a0 con 0x7f0d00160990 2024-04-24T08:08:07.117 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0d07400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0d00071ef0 con 0x7f0d000b4450 2024-04-24T08:08:07.117 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0d04a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d00160990 0x7f0d001669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:07.117 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0d04a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d00160990 0x7f0d001669e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33890/0 (socket says 172.21.15.69:33890) 2024-04-24T08:08:07.117 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0d04a00640 1 -- 172.21.15.69:0/319885762 learned_addr learned my addr 172.21.15.69:0/319885762 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:07.117 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0cf7e00640 1 --1- 172.21.15.69:0/319885762 >> v1:172.21.15.69:6789/0 conn(0x7f0d00166f20 0x7f0d0006e410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:41920/0 (socket says 172.21.15.69:41920) 2024-04-24T08:08:07.118 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0d04a00640 1 -- 172.21.15.69:0/319885762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d000b4450 msgr2=0x7f0d001673d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:07.118 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.116+0000 7f0d05400640 1 --2- 172.21.15.69:0/319885762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d000b4450 0x7f0d001673d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:07.118 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.117+0000 7f0d04a00640 1 --2- 172.21.15.69:0/319885762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d000b4450 0x7f0d001673d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:07.118 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.117+0000 7f0d04a00640 1 -- 172.21.15.69:0/319885762 >> v1:172.21.15.69:6789/0 conn(0x7f0d00166f20 legacy=0x7f0d0006e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:07.118 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.117+0000 7f0d04a00640 1 -- 172.21.15.69:0/319885762 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d000721d0 con 0x7f0d00160990 2024-04-24T08:08:07.118 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.117+0000 7f0cffe00640 1 -- 172.21.15.69:0/319885762 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2608761372 0 0) 0x7f0d00164350 con 0x7f0d00166f20 2024-04-24T08:08:07.118 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.117+0000 7f0d05400640 1 --2- 172.21.15.69:0/319885762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d000b4450 0x7f0d001673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:07.119 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0d04a00640 1 --2- 172.21.15.69:0/319885762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00160990 0x7f0d001669e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009870 tx=0x7f0cf002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f502f35cd566bd57 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:07.119 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0cffe00640 1 -- 172.21.15.69:0/319885762 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0037070 con 0x7f0d00160990 2024-04-24T08:08:07.119 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0cffe00640 1 -- 172.21.15.69:0/319885762 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf0032440 con 0x7f0d00160990 2024-04-24T08:08:07.119 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0cffe00640 1 -- 172.21.15.69:0/319885762 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0032810 con 0x7f0d00160990 2024-04-24T08:08:07.119 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0d07400640 1 -- 172.21.15.69:0/319885762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00160990 msgr2=0x7f0d001669e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:07.120 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0d07400640 1 --2- 172.21.15.69:0/319885762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00160990 0x7f0d001669e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009870 tx=0x7f0cf002a250 comp rx=0 tx=0).stop 2024-04-24T08:08:07.120 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0d07400640 1 -- 172.21.15.69:0/319885762 shutdown_connections 2024-04-24T08:08:07.120 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0d07400640 1 --2- 172.21.15.69:0/319885762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d000b4450 0x7f0d001673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:07.120 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0d07400640 1 --2- 172.21.15.69:0/319885762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00160990 0x7f0d001669e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:07.120 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0d07400640 1 -- 172.21.15.69:0/319885762 >> 172.21.15.69:0/319885762 conn(0x7f0d00075130 msgr2=0x7f0d0006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:07.120 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.118+0000 7f0d07400640 1 -- 172.21.15.69:0/319885762 shutdown_connections 2024-04-24T08:08:07.120 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.119+0000 7f0d07400640 1 -- 172.21.15.69:0/319885762 wait complete. 2024-04-24T08:08:07.121 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.120+0000 7f0d07400640 1 Processor -- start 2024-04-24T08:08:07.121 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.120+0000 7f0d07400640 1 -- start start 2024-04-24T08:08:07.123 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.121+0000 7f0d07400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d000b4450 0x7f0d0015c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:07.123 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.121+0000 7f0d07400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d00160990 0x7f0d0014a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:07.123 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.121+0000 7f0d07400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00166f20 0x7f0d00159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.121+0000 7f0d07400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0d00072a70 con 0x7f0d00166f20 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.121+0000 7f0d07400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0d00072790 con 0x7f0d000b4450 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.121+0000 7f0d07400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0d00160d70 con 0x7f0d00160990 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.121+0000 7f0d05400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00166f20 0x7f0d00159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.121+0000 7f0d05400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00166f20 0x7f0d00159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33894/0 (socket says 172.21.15.69:33894) 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.121+0000 7f0d05400640 1 -- 172.21.15.69:0/990346060 learned_addr learned my addr 172.21.15.69:0/990346060 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.121+0000 7f0cf7e00640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d00160990 0x7f0d0014a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0d04a00640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d000b4450 0x7f0d0015c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0d05400640 1 -- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d00160990 msgr2=0x7f0d0014a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0d05400640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d00160990 0x7f0d0014a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:07.124 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0d05400640 1 -- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d000b4450 msgr2=0x7f0d0015c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:07.125 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0d05400640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d000b4450 0x7f0d0015c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:07.125 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0d05400640 1 -- 172.21.15.69:0/990346060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d0015b250 con 0x7f0d00166f20 2024-04-24T08:08:07.125 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0cf7e00640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d00160990 0x7f0d0014a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:07.125 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0d05400640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00166f20 0x7f0d00159010 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800cf20 tx=0x7f0cf800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:07.125 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0cfd600640 1 -- 172.21.15.69:0/990346060 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf801a070 con 0x7f0d00166f20 2024-04-24T08:08:07.126 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0cfd600640 1 -- 172.21.15.69:0/990346060 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf800d840 con 0x7f0d00166f20 2024-04-24T08:08:07.126 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0d07400640 1 -- 172.21.15.69:0/990346060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d0015b530 con 0x7f0d00166f20 2024-04-24T08:08:07.126 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0cfd600640 1 -- 172.21.15.69:0/990346060 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf800dc30 con 0x7f0d00166f20 2024-04-24T08:08:07.126 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.122+0000 7f0d07400640 1 -- 172.21.15.69:0/990346060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d0015baf0 con 0x7f0d00166f20 2024-04-24T08:08:07.127 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.123+0000 7f0d07400640 1 -- 172.21.15.69:0/990346060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d0004fd90 con 0x7f0d00166f20 2024-04-24T08:08:07.127 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.124+0000 7f0cfd600640 1 -- 172.21.15.69:0/990346060 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f0cf801d070 con 0x7f0d00166f20 2024-04-24T08:08:07.127 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.125+0000 7f0cfd600640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0cdc06c860 0x7f0cdc06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:07.127 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.125+0000 7f0cfd600640 1 -- 172.21.15.69:0/990346060 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f0cf808b680 con 0x7f0d00166f20 2024-04-24T08:08:07.127 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.125+0000 7f0d04a00640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0cdc06c860 0x7f0cdc06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:07.127 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.126+0000 7f0d04a00640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0cdc06c860 0x7f0cdc06ed20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0034000 tx=0x7f0cf002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:07.130 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.130+0000 7f0cfd600640 1 -- 172.21.15.69:0/990346060 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0cf805a880 con 0x7f0d00166f20 2024-04-24T08:08:07.313 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.313+0000 7f0d07400640 1 -- 172.21.15.69:0/990346060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d000b0da0 con 0x7f0d00166f20 2024-04-24T08:08:07.315 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.315+0000 7f0cfd600640 1 -- 172.21.15.69:0/990346060 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f0cf805a180 con 0x7f0d00166f20 2024-04-24T08:08:07.315 INFO:teuthology.misc.health.smithi069.stdout: 2024-04-24T08:08:07.315 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":12,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:04.678850+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:07.317 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 -- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0cdc06c860 msgr2=0x7f0cdc06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:07.317 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0cdc06c860 0x7f0cdc06ed20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0034000 tx=0x7f0cf002e000 comp rx=0 tx=0).stop 2024-04-24T08:08:07.317 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 -- 172.21.15.69:0/990346060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00166f20 msgr2=0x7f0d00159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:07.317 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00166f20 0x7f0d00159010 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800cf20 tx=0x7f0cf800a7d0 comp rx=0 tx=0).stop 2024-04-24T08:08:07.318 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 -- 172.21.15.69:0/990346060 shutdown_connections 2024-04-24T08:08:07.318 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0cdc06c860 0x7f0cdc06ed20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:07.318 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d00166f20 0x7f0d00159010 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:07.318 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d00160990 0x7f0d0014a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:07.318 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 --2- 172.21.15.69:0/990346060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d000b4450 0x7f0d0015c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:07.318 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 -- 172.21.15.69:0/990346060 >> 172.21.15.69:0/990346060 conn(0x7f0d00075130 msgr2=0x7f0d00166260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:07.318 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 -- 172.21.15.69:0/990346060 shutdown_connections 2024-04-24T08:08:07.318 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:07.317+0000 7f0d07400640 1 -- 172.21.15.69:0/990346060 wait complete. 2024-04-24T08:08:07.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-24T08:08:13.234 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:08:13.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:08:14.139 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:14.324 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.324+0000 7fe2f7600640 1 Processor -- start 2024-04-24T08:08:14.324 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.324+0000 7fe2f7600640 1 -- start start 2024-04-24T08:08:14.326 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.325+0000 7fe2f7600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe2f00cfcd0 0x7fe2f0160390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.326 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.326+0000 7fe2f7600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f0161310 con 0x7fe2f00d03e0 2024-04-24T08:08:14.326 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.326+0000 7fe2f7600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f0146330 con 0x7fe2f0145ab0 2024-04-24T08:08:14.326 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.326+0000 7fe2f7600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f0160a70 con 0x7fe2f00cfcd0 2024-04-24T08:08:14.328 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.326+0000 7fe2f5600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe2f00d03e0 0x7fe2f006e950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37374/0 (socket says 172.21.15.69:37374) 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.326+0000 7fe2f5600640 1 -- 172.21.15.69:0/3372420226 learned_addr learned my addr 172.21.15.69:0/3372420226 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.326+0000 7fe2f4c00640 1 --2- 172.21.15.69:0/3372420226 >> v2:172.21.15.121:3300/0 conn(0x7fe2f00cfcd0 0x7fe2f0160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3912191514 0 0) 0x7fe2f0161310 con 0x7fe2f00d03e0 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d4003520 con 0x7fe2f00d03e0 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 930383221 0 0) 0x7fe2d4003520 con 0x7fe2f00d03e0 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 >> v2:172.21.15.121:3300/0 conn(0x7fe2f00cfcd0 msgr2=0x7fe2f0160390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 --2- 172.21.15.69:0/3372420226 >> v2:172.21.15.121:3300/0 conn(0x7fe2f00cfcd0 0x7fe2f0160390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 >> v1:172.21.15.121:6789/0 conn(0x7fe2f0145ab0 legacy=0x7fe2f015cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f0160d50 con 0x7fe2f00d03e0 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe2e80019f0 con 0x7fe2f00d03e0 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 >> v1:172.21.15.69:6789/0 conn(0x7fe2f00d03e0 legacy=0x7fe2f006e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.329 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2d4003a00 0x7fe2d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2d4005210 0x7fe2d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f0160a70 con 0x7fe2d4003a00 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d400c160 con 0x7fe2d4005210 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d400c360 con 0x7fe2d4004800 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.327+0000 7fe2f4c00640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2d4003a00 0x7fe2d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.328+0000 7fe2f4c00640 1 -- 172.21.15.69:0/3372420226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2d4004800 msgr2=0x7fe2d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.328+0000 7fe2e7e00640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.328+0000 7fe2f4c00640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.328+0000 7fe2f4c00640 1 -- 172.21.15.69:0/3372420226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2d4005210 msgr2=0x7fe2d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.328+0000 7fe2f4c00640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2d4005210 0x7fe2d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.328+0000 7fe2f4c00640 1 -- 172.21.15.69:0/3372420226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d4003520 con 0x7fe2d4003a00 2024-04-24T08:08:14.330 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.328+0000 7fe2f5600640 1 -- 172.21.15.69:0/3372420226 reap_dead start 2024-04-24T08:08:14.331 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.328+0000 7fe2e7e00640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2d4004800 0x7fe2d4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:14.332 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.329+0000 7fe2f4c00640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2d4003a00 0x7fe2d4003e20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009a90 tx=0x7fe2d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae8d2e14b376b79 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:14.332 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.329+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0036070 con 0x7fe2d4003a00 2024-04-24T08:08:14.332 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.329+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e0031400 con 0x7fe2d4003a00 2024-04-24T08:08:14.332 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.329+0000 7fe2efe00640 1 -- 172.21.15.69:0/3372420226 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0009690 con 0x7fe2d4003a00 2024-04-24T08:08:14.332 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.330+0000 7fe2f7600640 1 -- 172.21.15.69:0/3372420226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2d4003a00 msgr2=0x7fe2d4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.332 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.330+0000 7fe2f7600640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2d4003a00 0x7fe2d4003e20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009a90 tx=0x7fe2d4005830 comp rx=0 tx=0).stop 2024-04-24T08:08:14.332 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.330+0000 7fe2f7600640 1 -- 172.21.15.69:0/3372420226 shutdown_connections 2024-04-24T08:08:14.332 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.330+0000 7fe2f7600640 1 --2- 172.21.15.69:0/3372420226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2d4003a00 0x7fe2d4003e20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.332 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.330+0000 7fe2f7600640 1 -- 172.21.15.69:0/3372420226 >> 172.21.15.69:0/3372420226 conn(0x7fe2f0075130 msgr2=0x7fe2f006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:14.332 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.330+0000 7fe2f7600640 1 -- 172.21.15.69:0/3372420226 shutdown_connections 2024-04-24T08:08:14.333 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.330+0000 7fe2f7600640 1 -- 172.21.15.69:0/3372420226 wait complete. 2024-04-24T08:08:14.333 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.332+0000 7fe2f7600640 1 Processor -- start 2024-04-24T08:08:14.333 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.332+0000 7fe2f7600640 1 -- start start 2024-04-24T08:08:14.335 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2f7600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2d4003a00 0x7fe2f015bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.335 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2f7600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f00b3930 0x7fe2f01e6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.335 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2f7600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f00b3db0 0x7fe2f01e6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.335 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2f7600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f00d00b0 con 0x7fe2f00b3930 2024-04-24T08:08:14.335 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2f7600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f006e430 con 0x7fe2f00b3db0 2024-04-24T08:08:14.335 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2f7600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f0160390 con 0x7fe2d4003a00 2024-04-24T08:08:14.335 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2e7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f00b3930 0x7fe2f01e6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.335 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2e7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f00b3930 0x7fe2f01e6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60374/0 (socket says 172.21.15.69:60374) 2024-04-24T08:08:14.335 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2e7e00640 1 -- 172.21.15.69:0/2956038967 learned_addr learned my addr 172.21.15.69:0/2956038967 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:14.335 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2f4c00640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2d4003a00 0x7fe2f015bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2e7e00640 1 -- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2d4003a00 msgr2=0x7fe2f015bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2e7e00640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2d4003a00 0x7fe2f015bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2f5600640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f00b3db0 0x7fe2f01e6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2e7e00640 1 -- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f00b3db0 msgr2=0x7fe2f01e6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2e7e00640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f00b3db0 0x7fe2f01e6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.333+0000 7fe2e7e00640 1 -- 172.21.15.69:0/2956038967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f015c4f0 con 0x7fe2f00b3930 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.334+0000 7fe2e7e00640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f00b3930 0x7fe2f01e6aa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800d480 tx=0x7fe2d800d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.334+0000 7fe2ed600640 1 -- 172.21.15.69:0/2956038967 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8005480 con 0x7fe2f00b3930 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.334+0000 7fe2ed600640 1 -- 172.21.15.69:0/2956038967 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2d8017070 con 0x7fe2f00b3930 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.334+0000 7fe2ed600640 1 -- 172.21.15.69:0/2956038967 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8003830 con 0x7fe2f00b3930 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.334+0000 7fe2f7600640 1 -- 172.21.15.69:0/2956038967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f00b4310 con 0x7fe2f00b3930 2024-04-24T08:08:14.336 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.334+0000 7fe2f7600640 1 -- 172.21.15.69:0/2956038967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f00b48d0 con 0x7fe2f00b3930 2024-04-24T08:08:14.338 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.335+0000 7fe2f7600640 1 -- 172.21.15.69:0/2956038967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f004fd90 con 0x7fe2f00b3930 2024-04-24T08:08:14.338 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.336+0000 7fe2ed600640 1 -- 172.21.15.69:0/2956038967 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2d80039d0 con 0x7fe2f00b3930 2024-04-24T08:08:14.338 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.337+0000 7fe2ed600640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2cc06c8f0 0x7fe2cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.338 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.337+0000 7fe2ed600640 1 -- 172.21.15.69:0/2956038967 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fe2d808b650 con 0x7fe2f00b3930 2024-04-24T08:08:14.339 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.338+0000 7fe2f4c00640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2cc06c8f0 0x7fe2cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.339 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.339+0000 7fe2f4c00640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2cc06c8f0 0x7fe2cc06edb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0033000 tx=0x7fe2e0008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:14.342 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.342+0000 7fe2ed600640 1 -- 172.21.15.69:0/2956038967 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2f015c4f0 con 0x7fe2f00b3930 2024-04-24T08:08:14.546 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.546+0000 7fe2f7600640 1 -- 172.21.15.69:0/2956038967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe2f006c880 con 0x7fe2f00b3930 2024-04-24T08:08:14.548 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.548+0000 7fe2ed600640 1 -- 172.21.15.69:0/2956038967 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12860 (secure 0 0 0) 0x7fe2d8059e80 con 0x7fe2f00b3930 2024-04-24T08:08:14.548 INFO:teuthology.misc.health.smithi069.stdout: 2024-04-24T08:08:14.549 INFO:teuthology.misc.health.smithi069.stdout:{"epoch":16,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:11.018735+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:14.551 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.550+0000 7fe2f7600640 1 -- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2cc06c8f0 msgr2=0x7fe2cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.551 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.550+0000 7fe2f7600640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2cc06c8f0 0x7fe2cc06edb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0033000 tx=0x7fe2e0008030 comp rx=0 tx=0).stop 2024-04-24T08:08:14.551 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.550+0000 7fe2f7600640 1 -- 172.21.15.69:0/2956038967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f00b3930 msgr2=0x7fe2f01e6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.552 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.550+0000 7fe2f7600640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f00b3930 0x7fe2f01e6aa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800d480 tx=0x7fe2d800d940 comp rx=0 tx=0).stop 2024-04-24T08:08:14.552 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.550+0000 7fe2f7600640 1 -- 172.21.15.69:0/2956038967 shutdown_connections 2024-04-24T08:08:14.552 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.550+0000 7fe2f7600640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2cc06c8f0 0x7fe2cc06edb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.552 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.551+0000 7fe2f7600640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2f00b3930 0x7fe2f01e6aa0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.552 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.551+0000 7fe2f7600640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2f00b3db0 0x7fe2f01e6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.552 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.551+0000 7fe2f7600640 1 --2- 172.21.15.69:0/2956038967 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2d4003a00 0x7fe2f015bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.552 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.551+0000 7fe2f7600640 1 -- 172.21.15.69:0/2956038967 >> 172.21.15.69:0/2956038967 conn(0x7fe2f0075130 msgr2=0x7fe2f00b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:14.552 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.551+0000 7fe2f7600640 1 -- 172.21.15.69:0/2956038967 shutdown_connections 2024-04-24T08:08:14.552 INFO:teuthology.misc.health.smithi069.stderr:2024-04-24T08:08:14.551+0000 7fe2f7600640 1 -- 172.21.15.69:0/2956038967 wait complete. 2024-04-24T08:08:14.566 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-24T08:08:14.566 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-24T08:08:14.566 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T08:08:14.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T08:08:14.572 INFO:tasks.ceph.mds.a:Started 2024-04-24T08:08:14.572 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T08:08:14.573 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T08:08:14.578 INFO:tasks.ceph.mds.c:Started 2024-04-24T08:08:14.578 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T08:08:14.578 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T08:08:14.585 INFO:tasks.ceph.mds.b:Started 2024-04-24T08:08:14.586 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T08:08:14.586 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T08:08:14.593 INFO:tasks.ceph.mds.d:Started 2024-04-24T08:08:14.593 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-24T08:08:14.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:14.719 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:08:14.733 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:08:14.740 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-24T08:08:14.743 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:08:14.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.772+0000 7f3bf5600640 1 Processor -- start 2024-04-24T08:08:14.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.772+0000 7f3bf5600640 1 -- start start 2024-04-24T08:08:14.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.773+0000 7f3bf5600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf006cfb0 0x7f3bf00cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.773+0000 7f3bf5600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bf015ed40 con 0x7f3bf015e960 2024-04-24T08:08:14.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.773+0000 7f3bf5600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bf00cf050 con 0x7f3bf006d410 2024-04-24T08:08:14.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.773+0000 7f3bf5600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3bf0162b50 con 0x7f3bf006cfb0 2024-04-24T08:08:14.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.773+0000 7f3befe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3bf015e960 0x7f3bf0162440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37406/0 (socket says 172.21.15.69:37406) 2024-04-24T08:08:14.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.773+0000 7f3befe00640 1 -- 172.21.15.69:0/2355074508 learned_addr learned my addr 172.21.15.69:0/2355074508 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:14.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.773+0000 7f3bef400640 1 --2- 172.21.15.69:0/2355074508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf006cfb0 0x7f3bf00cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.774+0000 7f3beea00640 1 -- 172.21.15.69:0/2355074508 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2046802315 0 0) 0x7f3bf015ed40 con 0x7f3bf015e960 2024-04-24T08:08:14.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.774+0000 7f3beea00640 1 -- 172.21.15.69:0/2355074508 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bd4003520 con 0x7f3bf015e960 2024-04-24T08:08:14.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.774+0000 7f3beea00640 1 -- 172.21.15.69:0/2355074508 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323949180 0 0) 0x7f3bf00cf050 con 0x7f3bf006d410 2024-04-24T08:08:14.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.774+0000 7f3beea00640 1 -- 172.21.15.69:0/2355074508 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bf015ed40 con 0x7f3bf006d410 2024-04-24T08:08:14.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.774+0000 7f3bef400640 1 -- 172.21.15.69:0/2355074508 >> v1:172.21.15.121:6789/0 conn(0x7f3bf006d410 legacy=0x7f3bf00d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.774+0000 7f3bef400640 1 -- 172.21.15.69:0/2355074508 >> v1:172.21.15.69:6789/0 conn(0x7f3bf015e960 legacy=0x7f3bf0162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.774+0000 7f3bef400640 1 -- 172.21.15.69:0/2355074508 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf0162e20 con 0x7f3bf006cfb0 2024-04-24T08:08:14.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.775+0000 7f3bef400640 1 --2- 172.21.15.69:0/2355074508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf006cfb0 0x7f3bf00cfaf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0097d0 tx=0x7f3bdc02a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae811b2d823ce3d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:14.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.775+0000 7f3beea00640 1 -- 172.21.15.69:0/2355074508 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc037070 con 0x7f3bf006cfb0 2024-04-24T08:08:14.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.775+0000 7f3beea00640 1 -- 172.21.15.69:0/2355074508 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bdc032440 con 0x7f3bf006cfb0 2024-04-24T08:08:14.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.775+0000 7f3beea00640 1 -- 172.21.15.69:0/2355074508 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc0327f0 con 0x7f3bf006cfb0 2024-04-24T08:08:14.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.776+0000 7f3bf5600640 1 -- 172.21.15.69:0/2355074508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf006cfb0 msgr2=0x7f3bf00cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.776+0000 7f3bf5600640 1 --2- 172.21.15.69:0/2355074508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf006cfb0 0x7f3bf00cfaf0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0097d0 tx=0x7f3bdc02a200 comp rx=0 tx=0).stop 2024-04-24T08:08:14.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.776+0000 7f3bf5600640 1 -- 172.21.15.69:0/2355074508 shutdown_connections 2024-04-24T08:08:14.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.776+0000 7f3bf5600640 1 --2- 172.21.15.69:0/2355074508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf006cfb0 0x7f3bf00cfaf0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.776+0000 7f3bf5600640 1 -- 172.21.15.69:0/2355074508 >> 172.21.15.69:0/2355074508 conn(0x7f3bf009d900 msgr2=0x7f3bf008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:14.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.776+0000 7f3bf5600640 1 -- 172.21.15.69:0/2355074508 shutdown_connections 2024-04-24T08:08:14.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.776+0000 7f3bf5600640 1 -- 172.21.15.69:0/2355074508 wait complete. 2024-04-24T08:08:14.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.778+0000 7f3bf5600640 1 Processor -- start 2024-04-24T08:08:14.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.778+0000 7f3bf5600640 1 -- start start 2024-04-24T08:08:14.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3bf5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bf006cfb0 0x7f3bf014fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3bf5600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bf006d410 0x7f3bf00cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3bf5600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf015e960 0x7f3bf015f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3bf5600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf00cf5d0 con 0x7f3bf006cfb0 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3bf5600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf00cf2f0 con 0x7f3bf006d410 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3bf5600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3bf00b6170 con 0x7f3bf015e960 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3befe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bf006cfb0 0x7f3bf014fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3befe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bf006cfb0 0x7f3bf014fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60428/0 (socket says 172.21.15.69:60428) 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3befe00640 1 -- 172.21.15.69:0/1931529938 learned_addr learned my addr 172.21.15.69:0/1931529938 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3bef400640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bf006d410 0x7f3bf00cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3bf4c00640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf015e960 0x7f3bf015f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3befe00640 1 -- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf015e960 msgr2=0x7f3bf015f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3befe00640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf015e960 0x7f3bf015f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3befe00640 1 -- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bf006d410 msgr2=0x7f3bf00cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3befe00640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bf006d410 0x7f3bf00cb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.779+0000 7f3befe00640 1 -- 172.21.15.69:0/1931529938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf00c91e0 con 0x7f3bf006cfb0 2024-04-24T08:08:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.780+0000 7f3bef400640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bf006d410 0x7f3bf00cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.780+0000 7f3befe00640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bf006cfb0 0x7f3bf014fba0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3be000c410 tx=0x7f3be000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.780+0000 7f3be7e00640 1 -- 172.21.15.69:0/1931529938 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be0017070 con 0x7f3bf006cfb0 2024-04-24T08:08:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.780+0000 7f3be7e00640 1 -- 172.21.15.69:0/1931529938 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3be0005ce0 con 0x7f3bf006cfb0 2024-04-24T08:08:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.780+0000 7f3bf5600640 1 -- 172.21.15.69:0/1931529938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf00c94c0 con 0x7f3bf006cfb0 2024-04-24T08:08:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.780+0000 7f3be7e00640 1 -- 172.21.15.69:0/1931529938 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3be00072d0 con 0x7f3bf006cfb0 2024-04-24T08:08:14.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.780+0000 7f3bf5600640 1 -- 172.21.15.69:0/1931529938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf00c9a80 con 0x7f3bf006cfb0 2024-04-24T08:08:14.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.781+0000 7f3bf5600640 1 -- 172.21.15.69:0/1931529938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bf0075d50 con 0x7f3bf006cfb0 2024-04-24T08:08:14.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.782+0000 7f3be7e00640 1 -- 172.21.15.69:0/1931529938 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3be001a070 con 0x7f3bf006cfb0 2024-04-24T08:08:14.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.783+0000 7f3be7e00640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3bd006c920 0x7f3bd006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:14.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.783+0000 7f3be7e00640 1 -- 172.21.15.69:0/1931529938 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f3be008aa70 con 0x7f3bf006cfb0 2024-04-24T08:08:14.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.783+0000 7f3bef400640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3bd006c920 0x7f3bd006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:14.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.784+0000 7f3bef400640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3bd006c920 0x7f3bd006ede0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc034000 tx=0x7f3bdc02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:14.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.788+0000 7f3be7e00640 1 -- 172.21.15.69:0/1931529938 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3be0058b90 con 0x7f3bf006cfb0 2024-04-24T08:08:14.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.990+0000 7f3bf5600640 1 -- 172.21.15.69:0/1931529938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3bf00d1dc0 con 0x7f3bf006cfb0 2024-04-24T08:08:14.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.992+0000 7f3be7e00640 1 -- 172.21.15.69:0/1931529938 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12860 (secure 0 0 0) 0x7f3be005ce90 con 0x7f3bf006cfb0 2024-04-24T08:08:14.993 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:14.993 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":16,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:11.018735+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:14.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.994+0000 7f3bf5600640 1 -- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3bd006c920 msgr2=0x7f3bd006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.994+0000 7f3bf5600640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3bd006c920 0x7f3bd006ede0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc034000 tx=0x7f3bdc02e000 comp rx=0 tx=0).stop 2024-04-24T08:08:14.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.994+0000 7f3bf5600640 1 -- 172.21.15.69:0/1931529938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bf006cfb0 msgr2=0x7f3bf014fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:14.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.994+0000 7f3bf5600640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bf006cfb0 0x7f3bf014fba0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3be000c410 tx=0x7f3be000c8d0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.994+0000 7f3bf5600640 1 -- 172.21.15.69:0/1931529938 shutdown_connections 2024-04-24T08:08:14.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.994+0000 7f3bf5600640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3bd006c920 0x7f3bd006ede0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.994+0000 7f3bf5600640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3bf015e960 0x7f3bf015f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.994+0000 7f3bf5600640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3bf006d410 0x7f3bf00cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.994+0000 7f3bf5600640 1 --2- 172.21.15.69:0/1931529938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3bf006cfb0 0x7f3bf014fba0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:14.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.994+0000 7f3bf5600640 1 -- 172.21.15.69:0/1931529938 >> 172.21.15.69:0/1931529938 conn(0x7f3bf009d900 msgr2=0x7f3bf00b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:14.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.995+0000 7f3bf5600640 1 -- 172.21.15.69:0/1931529938 shutdown_connections 2024-04-24T08:08:14.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:14.995+0000 7f3bf5600640 1 -- 172.21.15.69:0/1931529938 wait complete. 2024-04-24T08:08:15.013 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:08:15.013 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:08:15.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.220+0000 7f3ce7a00640 1 Processor -- start 2024-04-24T08:08:15.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.220+0000 7f3ce7a00640 1 -- start start 2024-04-24T08:08:15.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.222+0000 7f3ce7a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3ce015e850 0x7f3ce00cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.222+0000 7f3ce7a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce015ec30 0x7f3ce00d1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.222+0000 7f3ce7a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce00d2710 con 0x7f3ce00d0450 2024-04-24T08:08:15.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.222+0000 7f3ce7a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce00d37b0 con 0x7f3ce015e850 2024-04-24T08:08:15.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.222+0000 7f3ce7a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce006eba0 con 0x7f3ce015ec30 2024-04-24T08:08:15.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.222+0000 7f3ce5000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3ce015e850 0x7f3ce00cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:15.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.222+0000 7f3ce5a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3ce00d0450 0x7f3ce0162560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37412/0 (socket says 172.21.15.69:37412) 2024-04-24T08:08:15.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.222+0000 7f3ce5a00640 1 -- 172.21.15.69:0/4025570836 learned_addr learned my addr 172.21.15.69:0/4025570836 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:15.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.223+0000 7f3cdfe00640 1 --2- 172.21.15.69:0/4025570836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce015ec30 0x7f3ce00d1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:15.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.223+0000 7f3ce5000640 1 -- 172.21.15.69:0/4025570836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce015ec30 msgr2=0x7f3ce00d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:15.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.223+0000 7f3ce5000640 1 --2- 172.21.15.69:0/4025570836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce015ec30 0x7f3ce00d1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.223+0000 7f3ce5000640 1 -- 172.21.15.69:0/4025570836 >> v1:172.21.15.69:6789/0 conn(0x7f3ce00d0450 legacy=0x7f3ce0162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.223+0000 7f3ce5000640 1 -- 172.21.15.69:0/4025570836 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00d29f0 con 0x7f3ce015e850 2024-04-24T08:08:15.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.223+0000 7f3cdf400640 1 -- 172.21.15.69:0/4025570836 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1711154260 0 0) 0x7f3ce00d2710 con 0x7f3ce00d0450 2024-04-24T08:08:15.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.223+0000 7f3cdfe00640 1 --2- 172.21.15.69:0/4025570836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce015ec30 0x7f3ce00d1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:15.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.224+0000 7f3ce5000640 1 --2- 172.21.15.69:0/4025570836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015e850 0x7f3ce00cff10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00097a0 tx=0x7f3cd002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35403d85c40e14a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:15.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.224+0000 7f3cdf400640 1 -- 172.21.15.69:0/4025570836 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0037070 con 0x7f3ce015e850 2024-04-24T08:08:15.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.224+0000 7f3cdf400640 1 -- 172.21.15.69:0/4025570836 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd0032440 con 0x7f3ce015e850 2024-04-24T08:08:15.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.224+0000 7f3cdf400640 1 -- 172.21.15.69:0/4025570836 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0032810 con 0x7f3ce015e850 2024-04-24T08:08:15.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.225+0000 7f3ce7a00640 1 -- 172.21.15.69:0/4025570836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015e850 msgr2=0x7f3ce00cff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.225+0000 7f3ce7a00640 1 --2- 172.21.15.69:0/4025570836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015e850 0x7f3ce00cff10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00097a0 tx=0x7f3cd002a180 comp rx=0 tx=0).stop 2024-04-24T08:08:15.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.225+0000 7f3ce7a00640 1 -- 172.21.15.69:0/4025570836 shutdown_connections 2024-04-24T08:08:15.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.225+0000 7f3ce7a00640 1 --2- 172.21.15.69:0/4025570836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce015ec30 0x7f3ce00d1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.225+0000 7f3ce7a00640 1 --2- 172.21.15.69:0/4025570836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015e850 0x7f3ce00cff10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.225+0000 7f3ce7a00640 1 -- 172.21.15.69:0/4025570836 >> 172.21.15.69:0/4025570836 conn(0x7f3ce008bf60 msgr2=0x7f3ce00d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:15.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.225+0000 7f3ce7a00640 1 -- 172.21.15.69:0/4025570836 shutdown_connections 2024-04-24T08:08:15.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.225+0000 7f3ce7a00640 1 -- 172.21.15.69:0/4025570836 wait complete. 2024-04-24T08:08:15.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.226+0000 7f3ce7a00640 1 Processor -- start 2024-04-24T08:08:15.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.227+0000 7f3ce7a00640 1 -- start start 2024-04-24T08:08:15.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.227+0000 7f3ce7a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00d0450 0x7f3ce00b8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.227+0000 7f3ce7a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ce015e850 0x7f3ce00b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce7a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015ec30 0x7f3ce01f86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce7a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00d3290 con 0x7f3ce015ec30 2024-04-24T08:08:15.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce7a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00d2fb0 con 0x7f3ce015e850 2024-04-24T08:08:15.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce7a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce00d2330 con 0x7f3ce00d0450 2024-04-24T08:08:15.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce5a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015ec30 0x7f3ce01f86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:15.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce5a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015ec30 0x7f3ce01f86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60444/0 (socket says 172.21.15.69:60444) 2024-04-24T08:08:15.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce5a00640 1 -- 172.21.15.69:0/2144597529 learned_addr learned my addr 172.21.15.69:0/2144597529 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce5000640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00d0450 0x7f3ce00b8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce5a00640 1 -- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00d0450 msgr2=0x7f3ce00b8270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce5a00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00d0450 0x7f3ce00b8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce5a00640 1 -- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ce015e850 msgr2=0x7f3ce00b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3cdfe00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ce015e850 0x7f3ce00b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce5a00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ce015e850 0x7f3ce00b4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.228+0000 7f3ce5a00640 1 -- 172.21.15.69:0/2144597529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00b8950 con 0x7f3ce015ec30 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.229+0000 7f3ce5a00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015ec30 0x7f3ce01f86a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800c450 tx=0x7f3cd800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.229+0000 7f3cdcc00640 1 -- 172.21.15.69:0/2144597529 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8018070 con 0x7f3ce015ec30 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.229+0000 7f3cdcc00640 1 -- 172.21.15.69:0/2144597529 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd8007500 con 0x7f3ce015ec30 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.229+0000 7f3cdcc00640 1 -- 172.21.15.69:0/2144597529 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8004a20 con 0x7f3ce015ec30 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.229+0000 7f3ce7a00640 1 -- 172.21.15.69:0/2144597529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce00b8ec0 con 0x7f3ce015ec30 2024-04-24T08:08:15.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.229+0000 7f3ce7a00640 1 -- 172.21.15.69:0/2144597529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce00c84a0 con 0x7f3ce015ec30 2024-04-24T08:08:15.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.230+0000 7f3ce7a00640 1 -- 172.21.15.69:0/2144597529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce00c86a0 con 0x7f3ce015ec30 2024-04-24T08:08:15.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.231+0000 7f3cdcc00640 1 -- 172.21.15.69:0/2144597529 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3cd8004bc0 con 0x7f3ce015ec30 2024-04-24T08:08:15.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.232+0000 7f3cdcc00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cbc06c8f0 0x7f3cbc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.232+0000 7f3cdcc00640 1 -- 172.21.15.69:0/2144597529 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f3cd808aa10 con 0x7f3ce015ec30 2024-04-24T08:08:15.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.232+0000 7f3ce5000640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cbc06c8f0 0x7f3cbc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:15.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.234+0000 7f3ce5000640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cbc06c8f0 0x7f3cbc06edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0034000 tx=0x7f3cd002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:15.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.236+0000 7f3cdcc00640 1 -- 172.21.15.69:0/2144597529 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3ce00c86a0 con 0x7f3ce015ec30 2024-04-24T08:08:15.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.434+0000 7f3ce7a00640 1 -- 172.21.15.69:0/2144597529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3ce0068a80 con 0x7f3ce015ec30 2024-04-24T08:08:15.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.435+0000 7f3cdcc00640 1 -- 172.21.15.69:0/2144597529 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f3cd8059180 con 0x7f3ce015ec30 2024-04-24T08:08:15.435 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:08:15.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 -- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cbc06c8f0 msgr2=0x7f3cbc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cbc06c8f0 0x7f3cbc06edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0034000 tx=0x7f3cd002e000 comp rx=0 tx=0).stop 2024-04-24T08:08:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 -- 172.21.15.69:0/2144597529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015ec30 msgr2=0x7f3ce01f86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015ec30 0x7f3ce01f86a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800c450 tx=0x7f3cd800c910 comp rx=0 tx=0).stop 2024-04-24T08:08:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 -- 172.21.15.69:0/2144597529 shutdown_connections 2024-04-24T08:08:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cbc06c8f0 0x7f3cbc06edb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ce015ec30 0x7f3ce01f86a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ce015e850 0x7f3ce00b4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 --2- 172.21.15.69:0/2144597529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00d0450 0x7f3ce00b8270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 -- 172.21.15.69:0/2144597529 >> 172.21.15.69:0/2144597529 conn(0x7f3ce008bf60 msgr2=0x7f3ce00b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 -- 172.21.15.69:0/2144597529 shutdown_connections 2024-04-24T08:08:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.438+0000 7f3ce7a00640 1 -- 172.21.15.69:0/2144597529 wait complete. 2024-04-24T08:08:15.458 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:08:15.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.654+0000 7f2c4b200640 1 Processor -- start 2024-04-24T08:08:15.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.654+0000 7f2c4b200640 1 -- start start 2024-04-24T08:08:15.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.655+0000 7f2c4b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2c4c06db90 0x7f2c4c073530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.655+0000 7f2c4b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c4c073a70 con 0x7f2c4c15b980 2024-04-24T08:08:15.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.655+0000 7f2c4b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c4c0aeac0 con 0x7f2c4c1523f0 2024-04-24T08:08:15.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.655+0000 7f2c4b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2c4c0aeda0 con 0x7f2c4c06db90 2024-04-24T08:08:15.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.656+0000 7f2c41400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2c4c06db90 0x7f2c4c073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:15.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.656+0000 7f2c4a800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2c4c15b980 0x7f2c4c15be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37426/0 (socket says 172.21.15.69:37426) 2024-04-24T08:08:15.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.656+0000 7f2c4a800640 1 -- 172.21.15.69:0/432692980 learned_addr learned my addr 172.21.15.69:0/432692980 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:15.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.656+0000 7f2c41400640 1 -- 172.21.15.69:0/432692980 >> v1:172.21.15.121:6789/0 conn(0x7f2c4c1523f0 legacy=0x7f2c4c072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:15.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.656+0000 7f2c41400640 1 -- 172.21.15.69:0/432692980 >> v1:172.21.15.69:6789/0 conn(0x7f2c4c15b980 legacy=0x7f2c4c15be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.656+0000 7f2c41400640 1 -- 172.21.15.69:0/432692980 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4c0af080 con 0x7f2c4c06db90 2024-04-24T08:08:15.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.657+0000 7f2c41400640 1 --2- 172.21.15.69:0/432692980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 0x7f2c4c073530 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2c340097a0 tx=0x7f2c3402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d5cb2478c31a45 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:15.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.657+0000 7f2c49400640 1 -- 172.21.15.69:0/432692980 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c34037070 con 0x7f2c4c06db90 2024-04-24T08:08:15.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.657+0000 7f2c49400640 1 -- 172.21.15.69:0/432692980 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c3402ae50 con 0x7f2c4c06db90 2024-04-24T08:08:15.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.658+0000 7f2c49400640 1 -- 172.21.15.69:0/432692980 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c34032660 con 0x7f2c4c06db90 2024-04-24T08:08:15.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.658+0000 7f2c4b200640 1 -- 172.21.15.69:0/432692980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 msgr2=0x7f2c4c073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.658+0000 7f2c4b200640 1 --2- 172.21.15.69:0/432692980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 0x7f2c4c073530 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2c340097a0 tx=0x7f2c3402a140 comp rx=0 tx=0).stop 2024-04-24T08:08:15.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.658+0000 7f2c4b200640 1 -- 172.21.15.69:0/432692980 shutdown_connections 2024-04-24T08:08:15.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.658+0000 7f2c4b200640 1 --2- 172.21.15.69:0/432692980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 0x7f2c4c073530 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.658+0000 7f2c4b200640 1 -- 172.21.15.69:0/432692980 >> 172.21.15.69:0/432692980 conn(0x7f2c4c0d0160 msgr2=0x7f2c4c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:15.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.658+0000 7f2c4b200640 1 -- 172.21.15.69:0/432692980 shutdown_connections 2024-04-24T08:08:15.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.658+0000 7f2c4b200640 1 -- 172.21.15.69:0/432692980 wait complete. 2024-04-24T08:08:15.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.660+0000 7f2c4b200640 1 Processor -- start 2024-04-24T08:08:15.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.660+0000 7f2c4b200640 1 -- start start 2024-04-24T08:08:15.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.661+0000 7f2c4b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 0x7f2c4c155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.661+0000 7f2c4b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4c1523f0 0x7f2c4c1f7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.661+0000 7f2c4b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4c15b980 0x7f2c4c1f8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.661+0000 7f2c4b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c06dfd0 con 0x7f2c4c06db90 2024-04-24T08:08:15.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.661+0000 7f2c4b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2c4c0b3190 con 0x7f2c4c1523f0 2024-04-24T08:08:15.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.661+0000 7f2c4b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2c4c1527d0 con 0x7f2c4c15b980 2024-04-24T08:08:15.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.661+0000 7f2c49e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 0x7f2c4c155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:15.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.661+0000 7f2c49e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 0x7f2c4c155480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60468/0 (socket says 172.21.15.69:60468) 2024-04-24T08:08:15.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.661+0000 7f2c49e00640 1 -- 172.21.15.69:0/2642154874 learned_addr learned my addr 172.21.15.69:0/2642154874 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:15.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c49e00640 1 -- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4c15b980 msgr2=0x7f2c4c1f8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c49e00640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4c15b980 0x7f2c4c1f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c49e00640 1 -- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4c1523f0 msgr2=0x7f2c4c1f7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c41400640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4c1523f0 0x7f2c4c1f7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:15.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c49e00640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4c1523f0 0x7f2c4c1f7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c49e00640 1 -- 172.21.15.69:0/2642154874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c4c1586c0 con 0x7f2c4c06db90 2024-04-24T08:08:15.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c49e00640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 0x7f2c4c155480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00c430 tx=0x7f2c3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:15.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c42a00640 1 -- 172.21.15.69:0/2642154874 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c011790 con 0x7f2c4c06db90 2024-04-24T08:08:15.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c42a00640 1 -- 172.21.15.69:0/2642154874 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c3c017070 con 0x7f2c4c06db90 2024-04-24T08:08:15.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c42a00640 1 -- 172.21.15.69:0/2642154874 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c011b70 con 0x7f2c4c06db90 2024-04-24T08:08:15.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c4b200640 1 -- 172.21.15.69:0/2642154874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c4c156290 con 0x7f2c4c06db90 2024-04-24T08:08:15.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.662+0000 7f2c4b200640 1 -- 172.21.15.69:0/2642154874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c4c158960 con 0x7f2c4c06db90 2024-04-24T08:08:15.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.664+0000 7f2c4b200640 1 -- 172.21.15.69:0/2642154874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c4c156020 con 0x7f2c4c06db90 2024-04-24T08:08:15.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.664+0000 7f2c42a00640 1 -- 172.21.15.69:0/2642154874 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c3c011d10 con 0x7f2c4c06db90 2024-04-24T08:08:15.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.665+0000 7f2c42a00640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c2806c8f0 0x7f2c2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:15.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.665+0000 7f2c42a00640 1 -- 172.21.15.69:0/2642154874 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f2c3c08b660 con 0x7f2c4c06db90 2024-04-24T08:08:15.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.666+0000 7f2c41400640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c2806c8f0 0x7f2c2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:15.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.667+0000 7f2c41400640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c2806c8f0 0x7f2c2806edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2c34034000 tx=0x7f2c3402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:15.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.670+0000 7f2c42a00640 1 -- 172.21.15.69:0/2642154874 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c3c059dd0 con 0x7f2c4c06db90 2024-04-24T08:08:15.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.913+0000 7f2c4b200640 1 -- 172.21.15.69:0/2642154874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c4c1567b0 con 0x7f2c4c06db90 2024-04-24T08:08:15.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.914+0000 7f2c42a00640 1 -- 172.21.15.69:0/2642154874 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3867 (secure 0 0 0) 0x7f2c3c0596d0 con 0x7f2c4c06db90 2024-04-24T08:08:15.914 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:15.914 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/934340733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":934340733},{"type":"v1","addr":"172.21.15.121:6835","nonce":934340733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3140867113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3140867113},{"type":"v1","addr":"172.21.15.121:6837","nonce":3140867113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/1059995973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1059995973},{"type":"v1","addr":"172.21.15.69:6833","nonce":1059995973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1003206529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1003206529},{"type":"v1","addr":"172.21.15.69:6835","nonce":1003206529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-24T08:08:15.915 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 3 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.916+0000 7f2c4b200640 1 -- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c2806c8f0 msgr2=0x7f2c2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.916+0000 7f2c4b200640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c2806c8f0 0x7f2c2806edb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2c34034000 tx=0x7f2c3402e000 comp rx=0 tx=0).stop 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.916+0000 7f2c4b200640 1 -- 172.21.15.69:0/2642154874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 msgr2=0x7f2c4c155480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.916+0000 7f2c4b200640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 0x7f2c4c155480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00c430 tx=0x7f2c3c00c8f0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.916+0000 7f2c4b200640 1 -- 172.21.15.69:0/2642154874 shutdown_connections 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.916+0000 7f2c4b200640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c2806c8f0 0x7f2c2806edb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.916+0000 7f2c4b200640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c4c15b980 0x7f2c4c1f8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.916+0000 7f2c4b200640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c4c1523f0 0x7f2c4c1f7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.916+0000 7f2c4b200640 1 --2- 172.21.15.69:0/2642154874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c4c06db90 0x7f2c4c155480 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.916+0000 7f2c4b200640 1 -- 172.21.15.69:0/2642154874 >> 172.21.15.69:0/2642154874 conn(0x7f2c4c0d0160 msgr2=0x7f2c4c1516f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.917+0000 7f2c4b200640 1 -- 172.21.15.69:0/2642154874 shutdown_connections 2024-04-24T08:08:15.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:15.917+0000 7f2c4b200640 1 -- 172.21.15.69:0/2642154874 wait complete. 2024-04-24T08:08:15.935 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.131+0000 7f58bf200640 1 Processor -- start 2024-04-24T08:08:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.131+0000 7f58bf200640 1 -- start start 2024-04-24T08:08:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.132+0000 7f58bf200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b80d31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.132+0000 7f58bf200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f58b81624d0 0x7f58b81628f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.133+0000 7f58bf200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58b80d2730 con 0x7f58b80d4690 2024-04-24T08:08:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.133+0000 7f58bf200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f58b815edc0 con 0x7f58b81624d0 2024-04-24T08:08:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.133+0000 7f58bf200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f58b8162f30 con 0x7f58b815e9e0 2024-04-24T08:08:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.133+0000 7f58bde00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b80d31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.133+0000 7f58bd400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f58b81624d0 0x7f58b81628f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.133+0000 7f58bde00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b80d31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:54704/0 (socket says 172.21.15.69:54704) 2024-04-24T08:08:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.133+0000 7f58bd400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f58b81624d0 0x7f58b81628f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:47542/0 (socket says 172.21.15.69:47542) 2024-04-24T08:08:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.133+0000 7f58bde00640 1 -- 172.21.15.69:0/3191251320 learned_addr learned my addr 172.21.15.69:0/3191251320 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.133+0000 7f58be800640 1 --1- 172.21.15.69:0/3191251320 >> v1:172.21.15.121:6789/0 conn(0x7f58b80d4690 0x7f58b80cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:58754/0 (socket says 172.21.15.69:58754) 2024-04-24T08:08:16.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.134+0000 7f58bd400640 1 -- 172.21.15.69:0/3191251320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 msgr2=0x7f58b80d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:16.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.134+0000 7f58bd400640 1 --2- 172.21.15.69:0/3191251320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b80d31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.134+0000 7f58bd400640 1 -- 172.21.15.69:0/3191251320 >> v1:172.21.15.121:6789/0 conn(0x7f58b80d4690 legacy=0x7f58b80cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.134+0000 7f58bd400640 1 -- 172.21.15.69:0/3191251320 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b8163210 con 0x7f58b81624d0 2024-04-24T08:08:16.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.134+0000 7f58bde00640 1 --2- 172.21.15.69:0/3191251320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b80d31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:16.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.135+0000 7f58bd400640 1 --2- 172.21.15.69:0/3191251320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b81624d0 0x7f58b81628f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58a80097a0 tx=0x7f58a802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf6dbdb37dce81ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bca00640 1 -- 172.21.15.69:0/3191251320 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a8037070 con 0x7f58b81624d0 2024-04-24T08:08:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bca00640 1 -- 172.21.15.69:0/3191251320 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58a8032440 con 0x7f58b81624d0 2024-04-24T08:08:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bca00640 1 -- 172.21.15.69:0/3191251320 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a8032810 con 0x7f58b81624d0 2024-04-24T08:08:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bf200640 1 -- 172.21.15.69:0/3191251320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b81624d0 msgr2=0x7f58b81628f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bf200640 1 --2- 172.21.15.69:0/3191251320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b81624d0 0x7f58b81628f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f58a80097a0 tx=0x7f58a802a180 comp rx=0 tx=0).stop 2024-04-24T08:08:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bf200640 1 -- 172.21.15.69:0/3191251320 shutdown_connections 2024-04-24T08:08:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bf200640 1 --2- 172.21.15.69:0/3191251320 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b81624d0 0x7f58b81628f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bf200640 1 --2- 172.21.15.69:0/3191251320 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b80d31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bf200640 1 -- 172.21.15.69:0/3191251320 >> 172.21.15.69:0/3191251320 conn(0x7f58b808bf60 msgr2=0x7f58b809d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bf200640 1 -- 172.21.15.69:0/3191251320 shutdown_connections 2024-04-24T08:08:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.136+0000 7f58bf200640 1 -- 172.21.15.69:0/3191251320 wait complete. 2024-04-24T08:08:16.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.138+0000 7f58bf200640 1 Processor -- start 2024-04-24T08:08:16.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.138+0000 7f58bf200640 1 -- start start 2024-04-24T08:08:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.139+0000 7f58bf200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b80d4690 0x7f58b80cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.139+0000 7f58bf200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b81f1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.139+0000 7f58bf200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b81624d0 0x7f58b80cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.139+0000 7f58bf200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58b80d3710 con 0x7f58b80d4690 2024-04-24T08:08:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.139+0000 7f58bf200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f58b80d2cb0 con 0x7f58b81624d0 2024-04-24T08:08:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.139+0000 7f58bf200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f58b815f1e0 con 0x7f58b815e9e0 2024-04-24T08:08:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.139+0000 7f58bde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b80d4690 0x7f58b80cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.139+0000 7f58bde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b80d4690 0x7f58b80cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60484/0 (socket says 172.21.15.69:60484) 2024-04-24T08:08:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.139+0000 7f58bde00640 1 -- 172.21.15.69:0/3549425078 learned_addr learned my addr 172.21.15.69:0/3549425078 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58bd400640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b81f1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58be800640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b81624d0 0x7f58b80cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58bde00640 1 -- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 msgr2=0x7f58b81f1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58bde00640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b81f1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58bde00640 1 -- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b81624d0 msgr2=0x7f58b80cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58bde00640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b81624d0 0x7f58b80cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58bde00640 1 -- 172.21.15.69:0/3549425078 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b80cc8c0 con 0x7f58b80d4690 2024-04-24T08:08:16.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58bd400640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b81f1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:16.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58bde00640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b80d4690 0x7f58b80cb7d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f58b000efc0 tx=0x7f58b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58ae000640 1 -- 172.21.15.69:0/3549425078 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0014480 con 0x7f58b80d4690 2024-04-24T08:08:16.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58ae000640 1 -- 172.21.15.69:0/3549425078 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b0019070 con 0x7f58b80d4690 2024-04-24T08:08:16.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58ae000640 1 -- 172.21.15.69:0/3549425078 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0014820 con 0x7f58b80d4690 2024-04-24T08:08:16.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58bf200640 1 -- 172.21.15.69:0/3549425078 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b80cc020 con 0x7f58b80d4690 2024-04-24T08:08:16.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.140+0000 7f58bf200640 1 -- 172.21.15.69:0/3549425078 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b81e8390 con 0x7f58b80d4690 2024-04-24T08:08:16.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.141+0000 7f58bf200640 1 -- 172.21.15.69:0/3549425078 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58b8075d50 con 0x7f58b80d4690 2024-04-24T08:08:16.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.142+0000 7f58ae000640 1 -- 172.21.15.69:0/3549425078 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f58b0005d10 con 0x7f58b80d4690 2024-04-24T08:08:16.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.143+0000 7f58ae000640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589c06c8f0 0x7f589c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.143+0000 7f58ae000640 1 -- 172.21.15.69:0/3549425078 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f58b008be10 con 0x7f58b80d4690 2024-04-24T08:08:16.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.144+0000 7f58bd400640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589c06c8f0 0x7f589c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.145+0000 7f58bd400640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589c06c8f0 0x7f589c06edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f58a8034000 tx=0x7f58a802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.147+0000 7f58ae000640 1 -- 172.21.15.69:0/3549425078 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58b008f030 con 0x7f58b80d4690 2024-04-24T08:08:16.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.345+0000 7f58bf200640 1 -- 172.21.15.69:0/3549425078 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58b80d0ef0 con 0x7f58b80d4690 2024-04-24T08:08:16.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.347+0000 7f58ae000640 1 -- 172.21.15.69:0/3549425078 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12860 (secure 0 0 0) 0x7f58b0010040 con 0x7f58b80d4690 2024-04-24T08:08:16.347 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:16.348 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":16,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:11.018735+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:16.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.349+0000 7f58bf200640 1 -- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589c06c8f0 msgr2=0x7f589c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.349+0000 7f58bf200640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589c06c8f0 0x7f589c06edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f58a8034000 tx=0x7f58a802e000 comp rx=0 tx=0).stop 2024-04-24T08:08:16.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.349+0000 7f58bf200640 1 -- 172.21.15.69:0/3549425078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b80d4690 msgr2=0x7f58b80cb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.349+0000 7f58bf200640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b80d4690 0x7f58b80cb7d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f58b000efc0 tx=0x7f58b000c490 comp rx=0 tx=0).stop 2024-04-24T08:08:16.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.350+0000 7f58bf200640 1 -- 172.21.15.69:0/3549425078 shutdown_connections 2024-04-24T08:08:16.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.350+0000 7f58bf200640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589c06c8f0 0x7f589c06edb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.350+0000 7f58bf200640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b81624d0 0x7f58b80cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.350+0000 7f58bf200640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b815e9e0 0x7f58b81f1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.350+0000 7f58bf200640 1 --2- 172.21.15.69:0/3549425078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b80d4690 0x7f58b80cb7d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.350+0000 7f58bf200640 1 -- 172.21.15.69:0/3549425078 >> 172.21.15.69:0/3549425078 conn(0x7f58b808bf60 msgr2=0x7f58b8160010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:16.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.350+0000 7f58bf200640 1 -- 172.21.15.69:0/3549425078 shutdown_connections 2024-04-24T08:08:16.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.350+0000 7f58bf200640 1 -- 172.21.15.69:0/3549425078 wait complete. 2024-04-24T08:08:16.368 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:08:16.369 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:08:16.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.565+0000 7f6ef7800640 1 Processor -- start 2024-04-24T08:08:16.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.565+0000 7f6ef7800640 1 -- start start 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.567+0000 7f6ef7800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6ef015f4b0 0x7f6ef00b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.567+0000 7f6ef7800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef015f890 0x7f6ef00732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.567+0000 7f6ef7800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6ef0073820 0x7f6ef00b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.567+0000 7f6ef7800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef0073cc0 con 0x7f6ef015f4b0 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.567+0000 7f6ef7800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef0073fa0 con 0x7f6ef0073820 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.567+0000 7f6ef7800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef00b3300 con 0x7f6ef015f890 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.567+0000 7f6ef4e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6ef015f4b0 0x7f6ef00b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.567+0000 7f6ef4e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6ef015f4b0 0x7f6ef00b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60488/0 (socket says 172.21.15.69:60488) 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.567+0000 7f6ef4e00640 1 -- 172.21.15.69:0/1421851840 learned_addr learned my addr 172.21.15.69:0/1421851840 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.568+0000 7f6ef5800640 1 --2- 172.21.15.69:0/1421851840 >> v2:172.21.15.121:3300/0 conn(0x7f6ef0073820 0x7f6ef00b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.568+0000 7f6ee7e00640 1 --2- 172.21.15.69:0/1421851840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef015f890 0x7f6ef00732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.568+0000 7f6ef4e00640 1 -- 172.21.15.69:0/1421851840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef015f890 msgr2=0x7f6ef00732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.568+0000 7f6ef4e00640 1 --2- 172.21.15.69:0/1421851840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef015f890 0x7f6ef00732e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.568+0000 7f6ef4e00640 1 -- 172.21.15.69:0/1421851840 >> v2:172.21.15.121:3300/0 conn(0x7f6ef0073820 msgr2=0x7f6ef00b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.568+0000 7f6ef4e00640 1 --2- 172.21.15.69:0/1421851840 >> v2:172.21.15.121:3300/0 conn(0x7f6ef0073820 0x7f6ef00b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.568+0000 7f6ef4e00640 1 -- 172.21.15.69:0/1421851840 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef00b35e0 con 0x7f6ef015f4b0 2024-04-24T08:08:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.569+0000 7f6ef4e00640 1 --2- 172.21.15.69:0/1421851840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef015f4b0 0x7f6ef00b3de0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00cdd90 tx=0x7f6ee002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d3160de94f9c30 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.569+0000 7f6eefe00640 1 -- 172.21.15.69:0/1421851840 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0037070 con 0x7f6ef015f4b0 2024-04-24T08:08:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6eefe00640 1 -- 172.21.15.69:0/1421851840 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee0032440 con 0x7f6ef015f4b0 2024-04-24T08:08:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6eefe00640 1 -- 172.21.15.69:0/1421851840 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0032810 con 0x7f6ef015f4b0 2024-04-24T08:08:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6ef7800640 1 -- 172.21.15.69:0/1421851840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef015f4b0 msgr2=0x7f6ef00b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6ef7800640 1 --2- 172.21.15.69:0/1421851840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef015f4b0 0x7f6ef00b3de0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6ef00cdd90 tx=0x7f6ee002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6ef7800640 1 -- 172.21.15.69:0/1421851840 shutdown_connections 2024-04-24T08:08:16.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6ef7800640 1 --2- 172.21.15.69:0/1421851840 >> v2:172.21.15.121:3300/0 conn(0x7f6ef0073820 0x7f6ef00b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6ef7800640 1 --2- 172.21.15.69:0/1421851840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef015f890 0x7f6ef00732e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6ef7800640 1 --2- 172.21.15.69:0/1421851840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef015f4b0 0x7f6ef00b3de0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6ef7800640 1 -- 172.21.15.69:0/1421851840 >> 172.21.15.69:0/1421851840 conn(0x7f6ef00d0350 msgr2=0x7f6ef00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:16.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6ef7800640 1 -- 172.21.15.69:0/1421851840 shutdown_connections 2024-04-24T08:08:16.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.570+0000 7f6ef7800640 1 -- 172.21.15.69:0/1421851840 wait complete. 2024-04-24T08:08:16.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.573+0000 7f6ef7800640 1 Processor -- start 2024-04-24T08:08:16.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.573+0000 7f6ef7800640 1 -- start start 2024-04-24T08:08:16.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.574+0000 7f6ef7800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef0073820 0x7f6ef00ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.574+0000 7f6ef7800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef015f4b0 0x7f6ef01e8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.574+0000 7f6ef7800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef015f890 0x7f6ef0099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.574+0000 7f6ef7800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef015fe90 con 0x7f6ef0073820 2024-04-24T08:08:16.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.574+0000 7f6ef7800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef015fd10 con 0x7f6ef015f4b0 2024-04-24T08:08:16.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.574+0000 7f6ef7800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef00b4be0 con 0x7f6ef015f890 2024-04-24T08:08:16.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ee7e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef015f4b0 0x7f6ef01e8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ef4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef0073820 0x7f6ef00ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ef5800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef015f890 0x7f6ef0099510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ee7e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef015f4b0 0x7f6ef01e8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:47568/0 (socket says 172.21.15.69:47568) 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ee7e00640 1 -- 172.21.15.69:0/832286447 learned_addr learned my addr 172.21.15.69:0/832286447 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ef4e00640 1 -- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef015f890 msgr2=0x7f6ef0099510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ef4e00640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef015f890 0x7f6ef0099510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ef4e00640 1 -- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef015f4b0 msgr2=0x7f6ef01e8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ef4e00640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef015f4b0 0x7f6ef01e8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ef4e00640 1 -- 172.21.15.69:0/832286447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef0154860 con 0x7f6ef0073820 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.575+0000 7f6ef4e00640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef0073820 0x7f6ef00ae760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6ef006d4c0 tx=0x7f6ee00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.576+0000 7f6eed600640 1 -- 172.21.15.69:0/832286447 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0041020 con 0x7f6ef0073820 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.576+0000 7f6eed600640 1 -- 172.21.15.69:0/832286447 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee0030700 con 0x7f6ef0073820 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.576+0000 7f6ef7800640 1 -- 172.21.15.69:0/832286447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef0143310 con 0x7f6ef0073820 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.576+0000 7f6eed600640 1 -- 172.21.15.69:0/832286447 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0037040 con 0x7f6ef0073820 2024-04-24T08:08:16.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.576+0000 7f6ef7800640 1 -- 172.21.15.69:0/832286447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef01438d0 con 0x7f6ef0073820 2024-04-24T08:08:16.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.577+0000 7f6ef7800640 1 -- 172.21.15.69:0/832286447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef0068a80 con 0x7f6ef0073820 2024-04-24T08:08:16.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.578+0000 7f6eed600640 1 -- 172.21.15.69:0/832286447 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ee0044020 con 0x7f6ef0073820 2024-04-24T08:08:16.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.579+0000 7f6eed600640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ecc06c8f0 0x7f6ecc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.579+0000 7f6eed600640 1 -- 172.21.15.69:0/832286447 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f6ee00abe00 con 0x7f6ef0073820 2024-04-24T08:08:16.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.580+0000 7f6ee7e00640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ecc06c8f0 0x7f6ecc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.581+0000 7f6ee7e00640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ecc06c8f0 0x7f6ecc06edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6ef01f7c10 tx=0x7f6ed8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.583+0000 7f6eed600640 1 -- 172.21.15.69:0/832286447 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ee004b030 con 0x7f6ef0073820 2024-04-24T08:08:16.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.774+0000 7f6ef7800640 1 -- 172.21.15.69:0/832286447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6ef00b4fb0 con 0x7f6ef0073820 2024-04-24T08:08:16.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.775+0000 7f6eed600640 1 -- 172.21.15.69:0/832286447 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f6ee0079f20 con 0x7f6ef0073820 2024-04-24T08:08:16.775 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:08:16.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.777+0000 7f6ef7800640 1 -- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ecc06c8f0 msgr2=0x7f6ecc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.777+0000 7f6ef7800640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ecc06c8f0 0x7f6ecc06edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6ef01f7c10 tx=0x7f6ed8008040 comp rx=0 tx=0).stop 2024-04-24T08:08:16.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.777+0000 7f6ef7800640 1 -- 172.21.15.69:0/832286447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef0073820 msgr2=0x7f6ef00ae760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.777+0000 7f6ef7800640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef0073820 0x7f6ef00ae760 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6ef006d4c0 tx=0x7f6ee00046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.777+0000 7f6ef7800640 1 -- 172.21.15.69:0/832286447 shutdown_connections 2024-04-24T08:08:16.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.777+0000 7f6ef7800640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ecc06c8f0 0x7f6ecc06edb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.777+0000 7f6ef7800640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ef015f890 0x7f6ef0099510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.777+0000 7f6ef7800640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ef015f4b0 0x7f6ef01e8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.777+0000 7f6ef7800640 1 --2- 172.21.15.69:0/832286447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ef0073820 0x7f6ef00ae760 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.777+0000 7f6ef7800640 1 -- 172.21.15.69:0/832286447 >> 172.21.15.69:0/832286447 conn(0x7f6ef00d0350 msgr2=0x7f6ef00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:16.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.778+0000 7f6ef7800640 1 -- 172.21.15.69:0/832286447 shutdown_connections 2024-04-24T08:08:16.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.778+0000 7f6ef7800640 1 -- 172.21.15.69:0/832286447 wait complete. 2024-04-24T08:08:16.796 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T08:08:16.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.981+0000 7f059a800640 1 Processor -- start 2024-04-24T08:08:16.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.981+0000 7f059a800640 1 -- start start 2024-04-24T08:08:16.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.982+0000 7f059a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594071e50 0x7f059415ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.982+0000 7f059a800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0594162300 0x7f059406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.982+0000 7f059a800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f059415f540 con 0x7f0594072230 2024-04-24T08:08:16.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.982+0000 7f059a800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f059415f820 con 0x7f0594162300 2024-04-24T08:08:16.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.982+0000 7f059a800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f059415fb00 con 0x7f0594071e50 2024-04-24T08:08:16.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.983+0000 7f0599e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0594072230 0x7f059406e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37434/0 (socket says 172.21.15.69:37434) 2024-04-24T08:08:16.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.983+0000 7f0599e00640 1 -- 172.21.15.69:0/1303807570 learned_addr learned my addr 172.21.15.69:0/1303807570 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:16.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.983+0000 7f0598a00640 1 --2- 172.21.15.69:0/1303807570 >> v2:172.21.15.121:3300/0 conn(0x7f0594162300 0x7f059406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.983+0000 7f0599400640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594071e50 0x7f059415ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.983+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2346905840 0 0) 0x7f059415f540 con 0x7f0594072230 2024-04-24T08:08:16.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.983+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f057c003520 con 0x7f0594072230 2024-04-24T08:08:16.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2786160267 0 0) 0x7f057c003520 con 0x7f0594072230 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594071e50 msgr2=0x7f059415ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594071e50 0x7f059415ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 >> v2:172.21.15.121:3300/0 conn(0x7f0594162300 msgr2=0x7f059406e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 --2- 172.21.15.69:0/1303807570 >> v2:172.21.15.121:3300/0 conn(0x7f0594162300 0x7f059406e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059406dda0 con 0x7f0594072230 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f058c0019f0 con 0x7f0594072230 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 >> v1:172.21.15.69:6789/0 conn(0x7f0594072230 legacy=0x7f059406e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594071e50 0x7f057c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c003f80 0x7f057c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f057c004d40 0x7f057c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f059415f820 con 0x7f057c004d40 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f059415fb00 con 0x7f057c003f80 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f057c00b9c0 con 0x7f0594071e50 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0599e00640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f057c004d40 0x7f057c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0598a00640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c003f80 0x7f057c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0599e00640 1 -- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594071e50 msgr2=0x7f057c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.984+0000 7f0599400640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594071e50 0x7f057c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.985+0000 7f0599e00640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594071e50 0x7f057c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.985+0000 7f0599e00640 1 -- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c003f80 msgr2=0x7f057c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.985+0000 7f0599e00640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c003f80 0x7f057c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.985+0000 7f0599e00640 1 -- 172.21.15.69:0/1303807570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057c003520 con 0x7f057c004d40 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.985+0000 7f0599400640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0594071e50 0x7f057c0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:16.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.985+0000 7f0598a00640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c003f80 0x7f057c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.985+0000 7f0599e00640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f057c004d40 0x7f057c0051c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f058c00a7d0 tx=0x7f057c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0ee16a6fa80c49 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.986+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c00a3f0 con 0x7f057c004d40 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.986+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f058c02f070 con 0x7f057c004d40 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.986+0000 7f0593e00640 1 -- 172.21.15.69:0/1303807570 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058c02bc50 con 0x7f057c004d40 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.986+0000 7f059a800640 1 -- 172.21.15.69:0/1303807570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f057c004d40 msgr2=0x7f057c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.986+0000 7f059a800640 1 --2- 172.21.15.69:0/1303807570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f057c004d40 0x7f057c0051c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f058c00a7d0 tx=0x7f057c005830 comp rx=0 tx=0).stop 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.986+0000 7f0599e00640 1 -- 172.21.15.69:0/1303807570 reap_dead start 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.986+0000 7f059a800640 1 -- 172.21.15.69:0/1303807570 shutdown_connections 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.986+0000 7f059a800640 1 -- 172.21.15.69:0/1303807570 >> 172.21.15.69:0/1303807570 conn(0x7f05940d0160 msgr2=0x7f05940d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.986+0000 7f059a800640 1 -- 172.21.15.69:0/1303807570 shutdown_connections 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.986+0000 7f059a800640 1 -- 172.21.15.69:0/1303807570 wait complete. 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.988+0000 7f059a800640 1 Processor -- start 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.988+0000 7f059a800640 1 -- start start 2024-04-24T08:08:16.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.989+0000 7f059a800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004d40 0x7f059409c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.989+0000 7f059a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0594159bc0 0x7f05941629b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.989+0000 7f059a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05940b0130 0x7f05940994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.989+0000 7f059a800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f059406f2a0 con 0x7f0594159bc0 2024-04-24T08:08:16.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.989+0000 7f059a800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f059406f120 con 0x7f057c004d40 2024-04-24T08:08:16.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.989+0000 7f059a800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f059406f420 con 0x7f05940b0130 2024-04-24T08:08:16.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.989+0000 7f0598a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0594159bc0 0x7f05941629b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.989+0000 7f0598a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0594159bc0 0x7f05941629b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60516/0 (socket says 172.21.15.69:60516) 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0598a00640 1 -- 172.21.15.69:0/3122699181 learned_addr learned my addr 172.21.15.69:0/3122699181 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0599400640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004d40 0x7f059409c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0599e00640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05940b0130 0x7f05940994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0598a00640 1 -- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05940b0130 msgr2=0x7f05940994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0598a00640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05940b0130 0x7f05940994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0598a00640 1 -- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004d40 msgr2=0x7f059409c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0598a00640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004d40 0x7f059409c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0598a00640 1 -- 172.21.15.69:0/3122699181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059409b300 con 0x7f0594159bc0 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0599e00640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05940b0130 0x7f05940994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0599400640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004d40 0x7f059409c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0598a00640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0594159bc0 0x7f05941629b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0580010a00 tx=0x7f0580010ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0591600640 1 -- 172.21.15.69:0/3122699181 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058000f040 con 0x7f0594159bc0 2024-04-24T08:08:16.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0591600640 1 -- 172.21.15.69:0/3122699181 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f058000ec20 con 0x7f0594159bc0 2024-04-24T08:08:16.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.990+0000 7f0591600640 1 -- 172.21.15.69:0/3122699181 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058001c400 con 0x7f0594159bc0 2024-04-24T08:08:16.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.991+0000 7f059a800640 1 -- 172.21.15.69:0/3122699181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f059409b570 con 0x7f0594159bc0 2024-04-24T08:08:16.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.991+0000 7f059a800640 1 -- 172.21.15.69:0/3122699181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f059409bb30 con 0x7f0594159bc0 2024-04-24T08:08:16.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.992+0000 7f059a800640 1 -- 172.21.15.69:0/3122699181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f059406fbf0 con 0x7f0594159bc0 2024-04-24T08:08:16.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.993+0000 7f0591600640 1 -- 172.21.15.69:0/3122699181 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f058001c5a0 con 0x7f0594159bc0 2024-04-24T08:08:16.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.993+0000 7f0591600640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f057406c8f0 0x7f057406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.994+0000 7f0591600640 1 -- 172.21.15.69:0/3122699181 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f058008bfe0 con 0x7f0594159bc0 2024-04-24T08:08:16.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.994+0000 7f0599400640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f057406c8f0 0x7f057406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.995+0000 7f0599400640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f057406c8f0 0x7f057406edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0588009aa0 tx=0x7f0588008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:16.998+0000 7f0591600640 1 -- 172.21.15.69:0/3122699181 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f059406fbf0 con 0x7f0594159bc0 2024-04-24T08:08:17.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.187+0000 7f059a800640 1 -- 172.21.15.69:0/3122699181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f05940d0d90 con 0x7f0594159bc0 2024-04-24T08:08:17.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.188+0000 7f0591600640 1 -- 172.21.15.69:0/3122699181 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f0580018030 con 0x7f0594159bc0 2024-04-24T08:08:17.188 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:17.188 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:08:17.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:08:17.188 INFO:teuthology.orchestra.run.smithi069.stdout: "poolnum": 1, 2024-04-24T08:08:17.188 INFO:teuthology.orchestra.run.smithi069.stdout: "poolname": ".mgr" 2024-04-24T08:08:17.188 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:08:17.188 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:08:17.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 -- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f057406c8f0 msgr2=0x7f057406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:17.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f057406c8f0 0x7f057406edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0588009aa0 tx=0x7f0588008040 comp rx=0 tx=0).stop 2024-04-24T08:08:17.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 -- 172.21.15.69:0/3122699181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0594159bc0 msgr2=0x7f05941629b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:17.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0594159bc0 0x7f05941629b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0580010a00 tx=0x7f0580010ec0 comp rx=0 tx=0).stop 2024-04-24T08:08:17.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 -- 172.21.15.69:0/3122699181 shutdown_connections 2024-04-24T08:08:17.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f057406c8f0 0x7f057406edb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:17.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0594159bc0 0x7f05941629b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:17.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05940b0130 0x7f05940994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:17.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 --2- 172.21.15.69:0/3122699181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f057c004d40 0x7f059409c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:17.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 -- 172.21.15.69:0/3122699181 >> 172.21.15.69:0/3122699181 conn(0x7f05940d0160 msgr2=0x7f05940b1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:17.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 -- 172.21.15.69:0/3122699181 shutdown_connections 2024-04-24T08:08:17.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.190+0000 7f059a800640 1 -- 172.21.15.69:0/3122699181 wait complete. 2024-04-24T08:08:17.208 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T08:08:17.208 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T08:08:17.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.411+0000 7f116f400640 1 Processor -- start 2024-04-24T08:08:17.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.412+0000 7f116f400640 1 -- start start 2024-04-24T08:08:17.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.413+0000 7f116f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f11681625d0 0x7f11680b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:17.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.413+0000 7f116f400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1168162b30 con 0x7f116806e320 2024-04-24T08:08:17.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.413+0000 7f116f400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1168162e10 con 0x7f11680d22f0 2024-04-24T08:08:17.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.413+0000 7f116f400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f11680d09c0 con 0x7f11681625d0 2024-04-24T08:08:17.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.413+0000 7f1167e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f11681625d0 0x7f11680b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:17.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.413+0000 7f116ca00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f116806e320 0x7f116806d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37442/0 (socket says 172.21.15.69:37442) 2024-04-24T08:08:17.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.413+0000 7f116ca00640 1 -- 172.21.15.69:0/2489588088 learned_addr learned my addr 172.21.15.69:0/2489588088 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:17.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.414+0000 7f1167e00640 1 -- 172.21.15.69:0/2489588088 >> v1:172.21.15.121:6789/0 conn(0x7f11680d22f0 legacy=0x7f11680d2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:17.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.414+0000 7f1167e00640 1 -- 172.21.15.69:0/2489588088 >> v1:172.21.15.69:6789/0 conn(0x7f116806e320 legacy=0x7f116806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:17.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.414+0000 7f1167e00640 1 -- 172.21.15.69:0/2489588088 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11681630f0 con 0x7f11681625d0 2024-04-24T08:08:17.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.414+0000 7f1167400640 1 -- 172.21.15.69:0/2489588088 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3086715641 0 0) 0x7f1168162b30 con 0x7f116806e320 2024-04-24T08:08:17.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.415+0000 7f1167e00640 1 --2- 172.21.15.69:0/2489588088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 0x7f11680b39c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f11500097a0 tx=0x7f115002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d408e21e564a624d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:17.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.415+0000 7f1167400640 1 -- 172.21.15.69:0/2489588088 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150037070 con 0x7f11681625d0 2024-04-24T08:08:17.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.415+0000 7f1167400640 1 -- 172.21.15.69:0/2489588088 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1150032440 con 0x7f11681625d0 2024-04-24T08:08:17.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.415+0000 7f1167400640 1 -- 172.21.15.69:0/2489588088 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150032810 con 0x7f11681625d0 2024-04-24T08:08:17.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.416+0000 7f116f400640 1 -- 172.21.15.69:0/2489588088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 msgr2=0x7f11680b39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:17.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.416+0000 7f116f400640 1 --2- 172.21.15.69:0/2489588088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 0x7f11680b39c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f11500097a0 tx=0x7f115002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:08:17.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.416+0000 7f116f400640 1 -- 172.21.15.69:0/2489588088 shutdown_connections 2024-04-24T08:08:17.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.416+0000 7f116f400640 1 --2- 172.21.15.69:0/2489588088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 0x7f11680b39c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:17.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.416+0000 7f116f400640 1 -- 172.21.15.69:0/2489588088 >> 172.21.15.69:0/2489588088 conn(0x7f116808bf60 msgr2=0x7f116808c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:17.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.416+0000 7f116f400640 1 -- 172.21.15.69:0/2489588088 shutdown_connections 2024-04-24T08:08:17.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.416+0000 7f116f400640 1 -- 172.21.15.69:0/2489588088 wait complete. 2024-04-24T08:08:17.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.418+0000 7f116f400640 1 Processor -- start 2024-04-24T08:08:17.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.418+0000 7f116f400640 1 -- start start 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f116f400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11680d22f0 0x7f1168158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f116f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 0x7f11681e85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f116f400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f116815c9b0 0x7f11681e8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f116f400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f11680b29d0 con 0x7f11681625d0 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f116f400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f11680b2540 con 0x7f116815c9b0 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f116f400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f11680b2b50 con 0x7f11680d22f0 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f1167e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 0x7f11681e85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f1167e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 0x7f11681e85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60532/0 (socket says 172.21.15.69:60532) 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f1167e00640 1 -- 172.21.15.69:0/729506051 learned_addr learned my addr 172.21.15.69:0/729506051 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f1167e00640 1 -- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11680d22f0 msgr2=0x7f1168158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f116ca00640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11680d22f0 0x7f1168158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f116d400640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f116815c9b0 0x7f11681e8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:17.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.419+0000 7f1167e00640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11680d22f0 0x7f1168158710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:17.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.420+0000 7f1167e00640 1 -- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f116815c9b0 msgr2=0x7f11681e8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:17.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.420+0000 7f1167e00640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f116815c9b0 0x7f11681e8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:17.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.420+0000 7f1167e00640 1 -- 172.21.15.69:0/729506051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11681e9140 con 0x7f11681625d0 2024-04-24T08:08:17.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.420+0000 7f116d400640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f116815c9b0 0x7f11681e8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:17.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.420+0000 7f1167e00640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 0x7f11681e85d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f11500098c0 tx=0x7f11500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:17.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.420+0000 7f1164c00640 1 -- 172.21.15.69:0/729506051 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150041070 con 0x7f11681625d0 2024-04-24T08:08:17.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.420+0000 7f1164c00640 1 -- 172.21.15.69:0/729506051 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f115003c870 con 0x7f11681625d0 2024-04-24T08:08:17.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.420+0000 7f1164c00640 1 -- 172.21.15.69:0/729506051 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150037040 con 0x7f11681625d0 2024-04-24T08:08:17.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.420+0000 7f116f400640 1 -- 172.21.15.69:0/729506051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11680c6d60 con 0x7f11681625d0 2024-04-24T08:08:17.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.420+0000 7f116f400640 1 -- 172.21.15.69:0/729506051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11680c7320 con 0x7f11681625d0 2024-04-24T08:08:17.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.421+0000 7f116f400640 1 -- 172.21.15.69:0/729506051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1168075dc0 con 0x7f11681625d0 2024-04-24T08:08:17.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.422+0000 7f1164c00640 1 -- 172.21.15.69:0/729506051 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1150044070 con 0x7f11681625d0 2024-04-24T08:08:17.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.423+0000 7f1164c00640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f114406c8f0 0x7f114406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:17.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.423+0000 7f1164c00640 1 -- 172.21.15.69:0/729506051 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f11500ac630 con 0x7f11681625d0 2024-04-24T08:08:17.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.424+0000 7f116ca00640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f114406c8f0 0x7f114406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:17.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.425+0000 7f116ca00640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f114406c8f0 0x7f114406edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1158007fd0 tx=0x7f1158009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:17.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.427+0000 7f1164c00640 1 -- 172.21.15.69:0/729506051 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f115007ae30 con 0x7f11681625d0 2024-04-24T08:08:17.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:17.629+0000 7f116f400640 1 -- 172.21.15.69:0/729506051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f11680b2290 con 0x7f11681625d0 2024-04-24T08:08:18.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.086+0000 7f1164c00640 1 -- 172.21.15.69:0/729506051 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7f115007a730 con 0x7f11681625d0 2024-04-24T08:08:18.087 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' created 2024-04-24T08:08:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 -- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f114406c8f0 msgr2=0x7f114406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f114406c8f0 0x7f114406edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1158007fd0 tx=0x7f1158009040 comp rx=0 tx=0).stop 2024-04-24T08:08:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 -- 172.21.15.69:0/729506051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 msgr2=0x7f11681e85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 0x7f11681e85d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f11500098c0 tx=0x7f11500046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 -- 172.21.15.69:0/729506051 shutdown_connections 2024-04-24T08:08:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f114406c8f0 0x7f114406edb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11681625d0 0x7f11681e85d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f116815c9b0 0x7f11681e8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 --2- 172.21.15.69:0/729506051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11680d22f0 0x7f1168158710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 -- 172.21.15.69:0/729506051 >> 172.21.15.69:0/729506051 conn(0x7f116808bf60 msgr2=0x7f11680715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:18.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 -- 172.21.15.69:0/729506051 shutdown_connections 2024-04-24T08:08:18.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.090+0000 7f116f400640 1 -- 172.21.15.69:0/729506051 wait complete. 2024-04-24T08:08:18.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T08:08:18.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.310+0000 7f4499a00640 1 Processor -- start 2024-04-24T08:08:18.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.310+0000 7f4499a00640 1 -- start start 2024-04-24T08:08:18.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.311+0000 7f4499a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44940722f0 0x7f449415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:18.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.311+0000 7f4499a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f44940cc4a0 0x7f449406e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:18.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.311+0000 7f4499a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4494073cd0 con 0x7f449406e5a0 2024-04-24T08:08:18.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.312+0000 7f4499a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f449415d1c0 con 0x7f44940cc4a0 2024-04-24T08:08:18.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.312+0000 7f4499a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f449415d4a0 con 0x7f44940722f0 2024-04-24T08:08:18.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.312+0000 7f4493e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44940722f0 0x7f449415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:18.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.312+0000 7f4493400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f44940cc4a0 0x7f449406e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:18.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.312+0000 7f4493e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44940722f0 0x7f449415cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:54778/0 (socket says 172.21.15.69:54778) 2024-04-24T08:08:18.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.312+0000 7f4493400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f44940cc4a0 0x7f449406e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:47614/0 (socket says 172.21.15.69:47614) 2024-04-24T08:08:18.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.312+0000 7f4493e00640 1 -- 172.21.15.69:0/1486095596 learned_addr learned my addr 172.21.15.69:0/1486095596 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:18.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.313+0000 7f4493400640 1 -- 172.21.15.69:0/1486095596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44940722f0 msgr2=0x7f449415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:18.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.313+0000 7f4493400640 1 --2- 172.21.15.69:0/1486095596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44940722f0 0x7f449415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:18.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.313+0000 7f4493400640 1 -- 172.21.15.69:0/1486095596 >> v1:172.21.15.121:6789/0 conn(0x7f449406e5a0 legacy=0x7f44940cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:18.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.313+0000 7f4493400640 1 -- 172.21.15.69:0/1486095596 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f449415d780 con 0x7f44940cc4a0 2024-04-24T08:08:18.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.313+0000 7f4493e00640 1 --2- 172.21.15.69:0/1486095596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44940722f0 0x7f449415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:18.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.313+0000 7f4492a00640 1 -- 172.21.15.69:0/1486095596 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2002231505 0 0) 0x7f4494073cd0 con 0x7f449406e5a0 2024-04-24T08:08:18.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.315+0000 7f4493400640 1 --2- 172.21.15.69:0/1486095596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44940cc4a0 0x7f449406e060 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f44800097a0 tx=0x7f448002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cba7d785c1d0fe7a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:18.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.315+0000 7f4492a00640 1 -- 172.21.15.69:0/1486095596 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480037070 con 0x7f44940cc4a0 2024-04-24T08:08:18.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.315+0000 7f4492a00640 1 -- 172.21.15.69:0/1486095596 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4480032440 con 0x7f44940cc4a0 2024-04-24T08:08:18.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.315+0000 7f4492a00640 1 -- 172.21.15.69:0/1486095596 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480032810 con 0x7f44940cc4a0 2024-04-24T08:08:18.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.316+0000 7f4499a00640 1 -- 172.21.15.69:0/1486095596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44940cc4a0 msgr2=0x7f449406e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:18.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.316+0000 7f4499a00640 1 --2- 172.21.15.69:0/1486095596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44940cc4a0 0x7f449406e060 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f44800097a0 tx=0x7f448002a180 comp rx=0 tx=0).stop 2024-04-24T08:08:18.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.316+0000 7f4499a00640 1 -- 172.21.15.69:0/1486095596 shutdown_connections 2024-04-24T08:08:18.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.316+0000 7f4499a00640 1 --2- 172.21.15.69:0/1486095596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44940cc4a0 0x7f449406e060 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:18.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.316+0000 7f4499a00640 1 --2- 172.21.15.69:0/1486095596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44940722f0 0x7f449415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:18.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.316+0000 7f4499a00640 1 -- 172.21.15.69:0/1486095596 >> 172.21.15.69:0/1486095596 conn(0x7f44940b1da0 msgr2=0x7f44940b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:18.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.316+0000 7f4499a00640 1 -- 172.21.15.69:0/1486095596 shutdown_connections 2024-04-24T08:08:18.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.316+0000 7f4499a00640 1 -- 172.21.15.69:0/1486095596 wait complete. 2024-04-24T08:08:18.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.317+0000 7f4499a00640 1 Processor -- start 2024-04-24T08:08:18.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.318+0000 7f4499a00640 1 -- start start 2024-04-24T08:08:18.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4499a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f449406e5a0 0x7f44940c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4499a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44940722f0 0x7f44940b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4499a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44940cc4a0 0x7f44941562a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4499a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44940cb420 con 0x7f44940722f0 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4499a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f44940cb2a0 con 0x7f44940cc4a0 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4499a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f44940ccbe0 con 0x7f449406e5a0 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4493400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44940722f0 0x7f44940b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4493400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44940722f0 0x7f44940b4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60544/0 (socket says 172.21.15.69:60544) 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4493400640 1 -- 172.21.15.69:0/2254268182 learned_addr learned my addr 172.21.15.69:0/2254268182 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4493e00640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f449406e5a0 0x7f44940c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4499000640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44940cc4a0 0x7f44941562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4493400640 1 -- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f449406e5a0 msgr2=0x7f44940c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4493400640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f449406e5a0 0x7f44940c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4493400640 1 -- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44940cc4a0 msgr2=0x7f44941562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4493400640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44940cc4a0 0x7f44941562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4493400640 1 -- 172.21.15.69:0/2254268182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44941f56e0 con 0x7f44940722f0 2024-04-24T08:08:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.319+0000 7f4499000640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44940cc4a0 0x7f44941562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.320+0000 7f4493400640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44940722f0 0x7f44940b4de0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4480035040 tx=0x7f44800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.320+0000 7f448be00640 1 -- 172.21.15.69:0/2254268182 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480037040 con 0x7f44940722f0 2024-04-24T08:08:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.320+0000 7f448be00640 1 -- 172.21.15.69:0/2254268182 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4480030500 con 0x7f44940722f0 2024-04-24T08:08:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.320+0000 7f4499a00640 1 -- 172.21.15.69:0/2254268182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44940c4d20 con 0x7f44940722f0 2024-04-24T08:08:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.320+0000 7f448be00640 1 -- 172.21.15.69:0/2254268182 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44800308d0 con 0x7f44940722f0 2024-04-24T08:08:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.320+0000 7f4499a00640 1 -- 172.21.15.69:0/2254268182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44940c52e0 con 0x7f44940722f0 2024-04-24T08:08:18.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.321+0000 7f4499a00640 1 -- 172.21.15.69:0/2254268182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4494075d50 con 0x7f44940722f0 2024-04-24T08:08:18.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.323+0000 7f448be00640 1 -- 172.21.15.69:0/2254268182 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4480044020 con 0x7f44940722f0 2024-04-24T08:08:18.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.323+0000 7f448be00640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f447406c920 0x7f447406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:18.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.324+0000 7f448be00640 1 -- 172.21.15.69:0/2254268182 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f44800ac6b0 con 0x7f44940722f0 2024-04-24T08:08:18.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.324+0000 7f4493e00640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f447406c920 0x7f447406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.325+0000 7f4493e00640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f447406c920 0x7f447406ede0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4484009b50 tx=0x7f4484008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:18.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.328+0000 7f448be00640 1 -- 172.21.15.69:0/2254268182 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f448007ad30 con 0x7f44940722f0 2024-04-24T08:08:18.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:18.529+0000 7f4499a00640 1 -- 172.21.15.69:0/2254268182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f44940b29d0 con 0x7f44940722f0 2024-04-24T08:08:19.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.100+0000 7f448be00640 1 -- 172.21.15.69:0/2254268182 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) ==== 170+0+0 (secure 0 0 0) 0x7f448007a630 con 0x7f44940722f0 2024-04-24T08:08:19.101 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' created 2024-04-24T08:08:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.103+0000 7f4499a00640 1 -- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f447406c920 msgr2=0x7f447406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.103+0000 7f4499a00640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f447406c920 0x7f447406ede0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4484009b50 tx=0x7f4484008040 comp rx=0 tx=0).stop 2024-04-24T08:08:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.103+0000 7f4499a00640 1 -- 172.21.15.69:0/2254268182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44940722f0 msgr2=0x7f44940b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.103+0000 7f4499a00640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44940722f0 0x7f44940b4de0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4480035040 tx=0x7f44800046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.104+0000 7f4499a00640 1 -- 172.21.15.69:0/2254268182 shutdown_connections 2024-04-24T08:08:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.104+0000 7f4499a00640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f447406c920 0x7f447406ede0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.104+0000 7f4499a00640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44940722f0 0x7f44940b4de0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.104+0000 7f4499a00640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44940cc4a0 0x7f44941562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.104+0000 7f4499a00640 1 --2- 172.21.15.69:0/2254268182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f449406e5a0 0x7f44940c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.104+0000 7f4499a00640 1 -- 172.21.15.69:0/2254268182 >> 172.21.15.69:0/2254268182 conn(0x7f44940b1da0 msgr2=0x7f44940b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.104+0000 7f4499a00640 1 -- 172.21.15.69:0/2254268182 shutdown_connections 2024-04-24T08:08:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.104+0000 7f4499a00640 1 -- 172.21.15.69:0/2254268182 wait complete. 2024-04-24T08:08:19.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T08:08:19.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.317+0000 7fda40a00640 1 Processor -- start 2024-04-24T08:08:19.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.317+0000 7fda40a00640 1 -- start start 2024-04-24T08:08:19.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.319+0000 7fda40a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fda3c0d1860 0x7fda3c169900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:19.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.319+0000 7fda40a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3c0d2710 con 0x7fda3c15e850 2024-04-24T08:08:19.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.319+0000 7fda40a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3c0d29f0 con 0x7fda3c15ec30 2024-04-24T08:08:19.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.319+0000 7fda40a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fda3c0d37b0 con 0x7fda3c0d1860 2024-04-24T08:08:19.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.319+0000 7fda3a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fda3c0d1860 0x7fda3c169900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:19.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.319+0000 7fda39e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fda3c15e850 0x7fda3c0cff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44578/0 (socket says 172.21.15.69:44578) 2024-04-24T08:08:19.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.319+0000 7fda39e00640 1 -- 172.21.15.69:0/946045862 learned_addr learned my addr 172.21.15.69:0/946045862 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:19.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.319+0000 7fda3a800640 1 -- 172.21.15.69:0/946045862 >> v1:172.21.15.121:6789/0 conn(0x7fda3c15ec30 legacy=0x7fda3c0d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:19.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.319+0000 7fda38a00640 1 -- 172.21.15.69:0/946045862 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 957503093 0 0) 0x7fda3c0d2710 con 0x7fda3c15e850 2024-04-24T08:08:19.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.319+0000 7fda3a800640 1 -- 172.21.15.69:0/946045862 >> v1:172.21.15.69:6789/0 conn(0x7fda3c15e850 legacy=0x7fda3c0cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:19.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.320+0000 7fda3a800640 1 -- 172.21.15.69:0/946045862 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c0d2cd0 con 0x7fda3c0d1860 2024-04-24T08:08:19.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.321+0000 7fda3a800640 1 --2- 172.21.15.69:0/946045862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c0d1860 0x7fda3c169900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fda34009d90 tx=0x7fda3402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2496002555a706a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:19.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.321+0000 7fda38a00640 1 -- 172.21.15.69:0/946045862 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34031600 con 0x7fda3c0d1860 2024-04-24T08:08:19.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.321+0000 7fda38a00640 1 -- 172.21.15.69:0/946045862 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda340317a0 con 0x7fda3c0d1860 2024-04-24T08:08:19.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.321+0000 7fda38a00640 1 -- 172.21.15.69:0/946045862 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda34031b70 con 0x7fda3c0d1860 2024-04-24T08:08:19.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.321+0000 7fda40a00640 1 -- 172.21.15.69:0/946045862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c0d1860 msgr2=0x7fda3c169900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:19.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.321+0000 7fda40a00640 1 --2- 172.21.15.69:0/946045862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c0d1860 0x7fda3c169900 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fda34009d90 tx=0x7fda3402a610 comp rx=0 tx=0).stop 2024-04-24T08:08:19.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.322+0000 7fda40a00640 1 -- 172.21.15.69:0/946045862 shutdown_connections 2024-04-24T08:08:19.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.322+0000 7fda40a00640 1 --2- 172.21.15.69:0/946045862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c0d1860 0x7fda3c169900 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:19.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.322+0000 7fda40a00640 1 -- 172.21.15.69:0/946045862 >> 172.21.15.69:0/946045862 conn(0x7fda3c08bf60 msgr2=0x7fda3c0d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:19.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.322+0000 7fda40a00640 1 -- 172.21.15.69:0/946045862 shutdown_connections 2024-04-24T08:08:19.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.322+0000 7fda40a00640 1 -- 172.21.15.69:0/946045862 wait complete. 2024-04-24T08:08:19.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.323+0000 7fda40a00640 1 Processor -- start 2024-04-24T08:08:19.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.323+0000 7fda40a00640 1 -- start start 2024-04-24T08:08:19.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.324+0000 7fda40a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c0d1860 0x7fda3c15cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:19.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.324+0000 7fda40a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c15e850 0x7fda3c0b5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:19.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.324+0000 7fda40a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c15ec30 0x7fda3c0b4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:19.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.324+0000 7fda40a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c0d3290 con 0x7fda3c15e850 2024-04-24T08:08:19.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.324+0000 7fda40a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c06eba0 con 0x7fda3c15ec30 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.324+0000 7fda40a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fda3c0d0620 con 0x7fda3c0d1860 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda39400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c15e850 0x7fda3c0b5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda39400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c15e850 0x7fda3c0b5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41096/0 (socket says 172.21.15.69:41096) 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda39400640 1 -- 172.21.15.69:0/3938820050 learned_addr learned my addr 172.21.15.69:0/3938820050 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda39e00640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c0d1860 0x7fda3c15cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda3a800640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c15ec30 0x7fda3c0b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda39400640 1 -- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c0d1860 msgr2=0x7fda3c15cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda39400640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c0d1860 0x7fda3c15cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda39400640 1 -- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c15ec30 msgr2=0x7fda3c0b4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda39400640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c15ec30 0x7fda3c0b4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda39400640 1 -- 172.21.15.69:0/3938820050 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c0b6840 con 0x7fda3c15e850 2024-04-24T08:08:19.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda3a800640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c15ec30 0x7fda3c0b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:19.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.325+0000 7fda39400640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c15e850 0x7fda3c0b5f80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fda2400b3f0 tx=0x7fda2400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:19.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.326+0000 7fda32000640 1 -- 172.21.15.69:0/3938820050 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda24015020 con 0x7fda3c15e850 2024-04-24T08:08:19.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.326+0000 7fda32000640 1 -- 172.21.15.69:0/3938820050 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda24011830 con 0x7fda3c15e850 2024-04-24T08:08:19.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.326+0000 7fda32000640 1 -- 172.21.15.69:0/3938820050 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda24011c20 con 0x7fda3c15e850 2024-04-24T08:08:19.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.326+0000 7fda40a00640 1 -- 172.21.15.69:0/3938820050 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda3c0b6b20 con 0x7fda3c15e850 2024-04-24T08:08:19.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.326+0000 7fda40a00640 1 -- 172.21.15.69:0/3938820050 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda3c0b4090 con 0x7fda3c15e850 2024-04-24T08:08:19.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.327+0000 7fda40a00640 1 -- 172.21.15.69:0/3938820050 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda3c068a80 con 0x7fda3c15e850 2024-04-24T08:08:19.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.328+0000 7fda32000640 1 -- 172.21.15.69:0/3938820050 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fda24018020 con 0x7fda3c15e850 2024-04-24T08:08:19.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.329+0000 7fda32000640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda1806c8f0 0x7fda1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:19.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.329+0000 7fda32000640 1 -- 172.21.15.69:0/3938820050 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7fda2408b7a0 con 0x7fda3c15e850 2024-04-24T08:08:19.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.330+0000 7fda39e00640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda1806c8f0 0x7fda1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:19.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.331+0000 7fda39e00640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda1806c8f0 0x7fda1806edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fda2c000f60 tx=0x7fda2c007b50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:19.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.334+0000 7fda32000640 1 -- 172.21.15.69:0/3938820050 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda2401f030 con 0x7fda3c15e850 2024-04-24T08:08:19.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:19.572+0000 7fda40a00640 1 -- 172.21.15.69:0/3938820050 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fda3c0b5d60 con 0x7fda3c15e850 2024-04-24T08:08:20.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.131+0000 7fda32000640 1 -- 172.21.15.69:0/3938820050 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fda240172c0 con 0x7fda3c15e850 2024-04-24T08:08:20.136 INFO:teuthology.orchestra.run.smithi069.stderr:new fs with metadata pool 2 and data pool 3 2024-04-24T08:08:20.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.137+0000 7fda40a00640 1 -- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda1806c8f0 msgr2=0x7fda1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.137+0000 7fda40a00640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda1806c8f0 0x7fda1806edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fda2c000f60 tx=0x7fda2c007b50 comp rx=0 tx=0).stop 2024-04-24T08:08:20.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.137+0000 7fda40a00640 1 -- 172.21.15.69:0/3938820050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c15e850 msgr2=0x7fda3c0b5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.137+0000 7fda40a00640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c15e850 0x7fda3c0b5f80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fda2400b3f0 tx=0x7fda2400b8b0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.138+0000 7fda40a00640 1 -- 172.21.15.69:0/3938820050 shutdown_connections 2024-04-24T08:08:20.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.138+0000 7fda40a00640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda1806c8f0 0x7fda1806edb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.138+0000 7fda40a00640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda3c15e850 0x7fda3c0b5f80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.138+0000 7fda40a00640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda3c15ec30 0x7fda3c0b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.138+0000 7fda40a00640 1 --2- 172.21.15.69:0/3938820050 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda3c0d1860 0x7fda3c15cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.138+0000 7fda40a00640 1 -- 172.21.15.69:0/3938820050 >> 172.21.15.69:0/3938820050 conn(0x7fda3c08bf60 msgr2=0x7fda3c0b2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:20.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.138+0000 7fda40a00640 1 -- 172.21.15.69:0/3938820050 shutdown_connections 2024-04-24T08:08:20.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.139+0000 7fda40a00640 1 -- 172.21.15.69:0/3938820050 wait complete. 2024-04-24T08:08:20.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:20.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.336+0000 7fd1cb600640 1 Processor -- start 2024-04-24T08:08:20.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.336+0000 7fd1cb600640 1 -- start start 2024-04-24T08:08:20.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.337+0000 7fd1cb600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c406f2d0 0x7fd1c406f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.337+0000 7fd1cb600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd1c40cfbe0 0x7fd1c40d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.337+0000 7fd1cb600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1c40b46e0 con 0x7fd1c415b490 2024-04-24T08:08:20.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.337+0000 7fd1cb600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd1c406fc70 con 0x7fd1c40cfbe0 2024-04-24T08:08:20.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.337+0000 7fd1cb600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c40b5890 con 0x7fd1c406f2d0 2024-04-24T08:08:20.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c8c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd1c415b490 0x7fd1c40b5180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44590/0 (socket says 172.21.15.69:44590) 2024-04-24T08:08:20.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c8c00640 1 -- 172.21.15.69:0/3627335700 learned_addr learned my addr 172.21.15.69:0/3627335700 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:20.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c9600640 1 --2- 172.21.15.69:0/3627335700 >> v2:172.21.15.121:3300/0 conn(0x7fd1c40cfbe0 0x7fd1c40d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2698364007 0 0) 0x7fd1c40b46e0 con 0x7fd1c415b490 2024-04-24T08:08:20.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1ac003520 con 0x7fd1c415b490 2024-04-24T08:08:20.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3e00640 1 --2- 172.21.15.69:0/3627335700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c406f2d0 0x7fd1c406f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 730281110 0 0) 0x7fd1ac003520 con 0x7fd1c415b490 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c406f2d0 msgr2=0x7fd1c406f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3400640 1 --2- 172.21.15.69:0/3627335700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c406f2d0 0x7fd1c406f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 >> v2:172.21.15.121:3300/0 conn(0x7fd1c40cfbe0 msgr2=0x7fd1c40d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3400640 1 --2- 172.21.15.69:0/3627335700 >> v2:172.21.15.121:3300/0 conn(0x7fd1c40cfbe0 0x7fd1c40d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c40d0960 con 0x7fd1c415b490 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fd1b8001670 con 0x7fd1c415b490 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.338+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 >> v1:172.21.15.69:6789/0 conn(0x7fd1c415b490 legacy=0x7fd1c40b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3400640 1 --2- 172.21.15.69:0/3627335700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c406f2d0 0x7fd1ac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3400640 1 --2- 172.21.15.69:0/3627335700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1ac003fe0 0x7fd1ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3400640 1 --2- 172.21.15.69:0/3627335700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ac004d40 0x7fd1ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c406fc70 con 0x7fd1ac003fe0 2024-04-24T08:08:20.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c40b5890 con 0x7fd1ac004d40 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd1ac00bac0 con 0x7fd1c406f2d0 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c9600640 1 --2- 172.21.15.69:0/3627335700 >> v2:172.21.15.121:3300/0 conn(0x7fd1c40cfbe0 0x7fd1c40d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3e00640 1 --2- 172.21.15.69:0/3627335700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1ac003fe0 0x7fd1ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3e00640 1 -- 172.21.15.69:0/3627335700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c406f2d0 msgr2=0x7fd1ac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3e00640 1 --2- 172.21.15.69:0/3627335700 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1c406f2d0 0x7fd1ac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3e00640 1 -- 172.21.15.69:0/3627335700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ac004d40 msgr2=0x7fd1ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3e00640 1 --2- 172.21.15.69:0/3627335700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1ac004d40 0x7fd1ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.339+0000 7fd1c3e00640 1 -- 172.21.15.69:0/3627335700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ac003520 con 0x7fd1ac003fe0 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.340+0000 7fd1c3e00640 1 --2- 172.21.15.69:0/3627335700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1ac003fe0 0x7fd1ac004800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000ba30 tx=0x7fd1ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7797081adacfebb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.340+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b0030070 con 0x7fd1ac003fe0 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.340+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b002ae00 con 0x7fd1ac003fe0 2024-04-24T08:08:20.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.340+0000 7fd1c3400640 1 -- 172.21.15.69:0/3627335700 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b000b600 con 0x7fd1ac003fe0 2024-04-24T08:08:20.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.341+0000 7fd1cb600640 1 -- 172.21.15.69:0/3627335700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1ac003fe0 msgr2=0x7fd1ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.341+0000 7fd1cb600640 1 --2- 172.21.15.69:0/3627335700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1ac003fe0 0x7fd1ac004800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd1b000ba30 tx=0x7fd1ac005830 comp rx=0 tx=0).stop 2024-04-24T08:08:20.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.341+0000 7fd1c9600640 1 -- 172.21.15.69:0/3627335700 reap_dead start 2024-04-24T08:08:20.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.341+0000 7fd1cb600640 1 -- 172.21.15.69:0/3627335700 shutdown_connections 2024-04-24T08:08:20.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.341+0000 7fd1cb600640 1 -- 172.21.15.69:0/3627335700 >> 172.21.15.69:0/3627335700 conn(0x7fd1c4067d90 msgr2=0x7fd1c406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:20.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.341+0000 7fd1cb600640 1 -- 172.21.15.69:0/3627335700 shutdown_connections 2024-04-24T08:08:20.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.341+0000 7fd1cb600640 1 -- 172.21.15.69:0/3627335700 wait complete. 2024-04-24T08:08:20.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.343+0000 7fd1cb600640 1 Processor -- start 2024-04-24T08:08:20.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.343+0000 7fd1cb600640 1 -- start start 2024-04-24T08:08:20.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1cb600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac003fe0 0x7fd1c4142f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1cb600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1c4153200 0x7fd1c40b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1cb600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1c40b3bf0 0x7fd1c414fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1cb600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c40d0c40 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1cb600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c40b4c60 con 0x7fd1c4153200 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1cb600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c415b870 con 0x7fd1ac003fe0 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1c9600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1c40b3bf0 0x7fd1c414fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1c9600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1c40b3bf0 0x7fd1c414fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41104/0 (socket says 172.21.15.69:41104) 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1c9600640 1 -- 172.21.15.69:0/88541669 learned_addr learned my addr 172.21.15.69:0/88541669 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1c3e00640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1c4153200 0x7fd1c40b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1c9600640 1 -- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac003fe0 msgr2=0x7fd1c4142f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1c8c00640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac003fe0 0x7fd1c4142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1c9600640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac003fe0 0x7fd1c4142f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1c9600640 1 -- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1c4153200 msgr2=0x7fd1c40b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1c9600640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1c4153200 0x7fd1c40b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.344+0000 7fd1c9600640 1 -- 172.21.15.69:0/88541669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c4156280 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.345+0000 7fd1c9600640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1c40b3bf0 0x7fd1c414fc80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40afb50 tx=0x7fd1bc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:20.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.345+0000 7fd1c0c00640 1 -- 172.21.15.69:0/88541669 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc01a070 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.345+0000 7fd1c0c00640 1 -- 172.21.15.69:0/88541669 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1bc00dcf0 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.345+0000 7fd1cb600640 1 -- 172.21.15.69:0/88541669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c4151550 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.345+0000 7fd1c0c00640 1 -- 172.21.15.69:0/88541669 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc00dea0 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.345+0000 7fd1cb600640 1 -- 172.21.15.69:0/88541669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c4153680 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.346+0000 7fd1cb600640 1 -- 172.21.15.69:0/88541669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c40b4070 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.347+0000 7fd1c0c00640 1 -- 172.21.15.69:0/88541669 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd1bc01d070 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.348+0000 7fd1c0c00640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1a406c8f0 0x7fd1a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.348+0000 7fd1c0c00640 1 -- 172.21.15.69:0/88541669 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fd1bc08ba10 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.349+0000 7fd1c8c00640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1a406c8f0 0x7fd1a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.350+0000 7fd1c8c00640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1a406c8f0 0x7fd1a406edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8007a30 tx=0x7fd1b8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:20.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.352+0000 7fd1c0c00640 1 -- 172.21.15.69:0/88541669 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd1bc010070 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.543+0000 7fd1cb600640 1 -- 172.21.15.69:0/88541669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd1c40cdcb0 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.545+0000 7fd1c0c00640 1 -- 172.21.15.69:0/88541669 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16594 (secure 0 0 0) 0x7fd1c40cdcb0 con 0x7fd1c40b3bf0 2024-04-24T08:08:20.546 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:20.547 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":19,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:19.574157+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:20.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 -- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1a406c8f0 msgr2=0x7fd1a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1a406c8f0 0x7fd1a406edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8007a30 tx=0x7fd1b8009040 comp rx=0 tx=0).stop 2024-04-24T08:08:20.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 -- 172.21.15.69:0/88541669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1c40b3bf0 msgr2=0x7fd1c414fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1c40b3bf0 0x7fd1c414fc80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd1c40afb50 tx=0x7fd1bc00d4a0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 -- 172.21.15.69:0/88541669 shutdown_connections 2024-04-24T08:08:20.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1a406c8f0 0x7fd1a406edb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1c40b3bf0 0x7fd1c414fc80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd1c4153200 0x7fd1c40b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 --2- 172.21.15.69:0/88541669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd1ac003fe0 0x7fd1c4142f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 -- 172.21.15.69:0/88541669 >> 172.21.15.69:0/88541669 conn(0x7fd1c4067d90 msgr2=0x7fd1c40ceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:20.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 -- 172.21.15.69:0/88541669 shutdown_connections 2024-04-24T08:08:20.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.548+0000 7fd1cb600640 1 -- 172.21.15.69:0/88541669 wait complete. 2024-04-24T08:08:20.566 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:20.744 INFO:tasks.ceph.mds.c.smithi069.stderr:2024-04-24T08:08:20.744+0000 7f4562e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:08:20.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.756+0000 7fb743600640 1 Processor -- start 2024-04-24T08:08:20.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.756+0000 7fb743600640 1 -- start start 2024-04-24T08:08:20.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.757+0000 7fb743600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb73c145fe0 0x7fb73c06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.757+0000 7fb743600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb73c0cf3d0 con 0x7fb73c0d0300 2024-04-24T08:08:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.757+0000 7fb743600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb73c0cf6b0 con 0x7fb73c145a90 2024-04-24T08:08:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.757+0000 7fb743600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb73c0cf990 con 0x7fb73c145fe0 2024-04-24T08:08:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.758+0000 7fb740c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb73c0d0300 0x7fb73c0d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44600/0 (socket says 172.21.15.69:44600) 2024-04-24T08:08:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.758+0000 7fb740c00640 1 -- 172.21.15.69:0/2027055900 learned_addr learned my addr 172.21.15.69:0/2027055900 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.758+0000 7fb741600640 1 --2- 172.21.15.69:0/2027055900 >> v2:172.21.15.69:3300/0 conn(0x7fb73c145fe0 0x7fb73c06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.758+0000 7fb741600640 1 -- 172.21.15.69:0/2027055900 >> v1:172.21.15.121:6789/0 conn(0x7fb73c145a90 legacy=0x7fb73c160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.758+0000 7fb741600640 1 -- 172.21.15.69:0/2027055900 >> v1:172.21.15.69:6789/0 conn(0x7fb73c0d0300 legacy=0x7fb73c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.758+0000 7fb741600640 1 -- 172.21.15.69:0/2027055900 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73c0cfc70 con 0x7fb73c145fe0 2024-04-24T08:08:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.759+0000 7fb741600640 1 --2- 172.21.15.69:0/2027055900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145fe0 0x7fb73c06e240 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb734009d90 tx=0x7fb73402a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0dbc0bb7328193e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.759+0000 7fb73b400640 1 -- 172.21.15.69:0/2027055900 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb734007740 con 0x7fb73c145fe0 2024-04-24T08:08:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.759+0000 7fb73b400640 1 -- 172.21.15.69:0/2027055900 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7340078e0 con 0x7fb73c145fe0 2024-04-24T08:08:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.759+0000 7fb73b400640 1 -- 172.21.15.69:0/2027055900 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb734007cb0 con 0x7fb73c145fe0 2024-04-24T08:08:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.760+0000 7fb743600640 1 -- 172.21.15.69:0/2027055900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145fe0 msgr2=0x7fb73c06e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.760+0000 7fb743600640 1 --2- 172.21.15.69:0/2027055900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145fe0 0x7fb73c06e240 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb734009d90 tx=0x7fb73402a5b0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.760+0000 7fb743600640 1 -- 172.21.15.69:0/2027055900 shutdown_connections 2024-04-24T08:08:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.760+0000 7fb743600640 1 --2- 172.21.15.69:0/2027055900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145fe0 0x7fb73c06e240 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.760+0000 7fb743600640 1 -- 172.21.15.69:0/2027055900 >> 172.21.15.69:0/2027055900 conn(0x7fb73c067d90 msgr2=0x7fb73c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.760+0000 7fb743600640 1 -- 172.21.15.69:0/2027055900 shutdown_connections 2024-04-24T08:08:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.760+0000 7fb743600640 1 -- 172.21.15.69:0/2027055900 wait complete. 2024-04-24T08:08:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.762+0000 7fb743600640 1 Processor -- start 2024-04-24T08:08:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.762+0000 7fb743600640 1 -- start start 2024-04-24T08:08:20.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.763+0000 7fb743600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb73c0d0300 0x7fb73c13d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.763+0000 7fb743600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145a90 0x7fb73c0b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.763+0000 7fb743600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb73c145fe0 0x7fb73c15bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.763+0000 7fb743600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c1458f0 con 0x7fb73c145a90 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.763+0000 7fb743600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c145770 con 0x7fb73c0d0300 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.763+0000 7fb743600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb73c0cdc30 con 0x7fb73c145fe0 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.763+0000 7fb73be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145a90 0x7fb73c0b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.763+0000 7fb73be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145a90 0x7fb73c0b13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41122/0 (socket says 172.21.15.69:41122) 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.763+0000 7fb73be00640 1 -- 172.21.15.69:0/343655562 learned_addr learned my addr 172.21.15.69:0/343655562 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb73be00640 1 -- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb73c145fe0 msgr2=0x7fb73c15bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb73be00640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb73c145fe0 0x7fb73c15bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb73be00640 1 -- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb73c0d0300 msgr2=0x7fb73c13d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb740c00640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb73c0d0300 0x7fb73c13d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb73be00640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb73c0d0300 0x7fb73c13d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb73be00640 1 -- 172.21.15.69:0/343655562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73c13c580 con 0x7fb73c145a90 2024-04-24T08:08:20.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb73be00640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145a90 0x7fb73c0b13a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb72800c460 tx=0x7fb72800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:20.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb738c00640 1 -- 172.21.15.69:0/343655562 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728016020 con 0x7fb73c145a90 2024-04-24T08:08:20.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb738c00640 1 -- 172.21.15.69:0/343655562 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb728012870 con 0x7fb73c145a90 2024-04-24T08:08:20.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb738c00640 1 -- 172.21.15.69:0/343655562 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb728012c60 con 0x7fb73c145a90 2024-04-24T08:08:20.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb743600640 1 -- 172.21.15.69:0/343655562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb73c13c820 con 0x7fb73c145a90 2024-04-24T08:08:20.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.764+0000 7fb743600640 1 -- 172.21.15.69:0/343655562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb73c13d120 con 0x7fb73c145a90 2024-04-24T08:08:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.765+0000 7fb743600640 1 -- 172.21.15.69:0/343655562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb73c068a80 con 0x7fb73c145a90 2024-04-24T08:08:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.766+0000 7fb738c00640 1 -- 172.21.15.69:0/343655562 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb728019020 con 0x7fb73c145a90 2024-04-24T08:08:20.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.767+0000 7fb738c00640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb71c06c8f0 0x7fb71c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:20.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.768+0000 7fb738c00640 1 -- 172.21.15.69:0/343655562 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fb72808b8a0 con 0x7fb73c145a90 2024-04-24T08:08:20.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.768+0000 7fb740c00640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb71c06c8f0 0x7fb71c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:20.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.769+0000 7fb740c00640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb71c06c8f0 0x7fb71c06edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb730007fd0 tx=0x7fb730009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:20.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.772+0000 7fb738c00640 1 -- 172.21.15.69:0/343655562 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb728020030 con 0x7fb73c145a90 2024-04-24T08:08:20.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.957+0000 7fb743600640 1 -- 172.21.15.69:0/343655562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb73c0726a0 con 0x7fb73c145a90 2024-04-24T08:08:20.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.960+0000 7fb738c00640 1 -- 172.21.15.69:0/343655562 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16594 (secure 0 0 0) 0x7fb728059cc0 con 0x7fb73c145a90 2024-04-24T08:08:20.960 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:20.961 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":19,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:19.574157+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:20.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.962+0000 7fb743600640 1 -- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb71c06c8f0 msgr2=0x7fb71c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.962+0000 7fb743600640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb71c06c8f0 0x7fb71c06edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb730007fd0 tx=0x7fb730009040 comp rx=0 tx=0).stop 2024-04-24T08:08:20.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.962+0000 7fb743600640 1 -- 172.21.15.69:0/343655562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145a90 msgr2=0x7fb73c0b13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:20.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.962+0000 7fb743600640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145a90 0x7fb73c0b13a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb72800c460 tx=0x7fb72800c920 comp rx=0 tx=0).stop 2024-04-24T08:08:20.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.963+0000 7fb743600640 1 -- 172.21.15.69:0/343655562 shutdown_connections 2024-04-24T08:08:20.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.963+0000 7fb743600640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb71c06c8f0 0x7fb71c06edb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.963+0000 7fb743600640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb73c145a90 0x7fb73c0b13a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.963+0000 7fb743600640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb73c145fe0 0x7fb73c15bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.963+0000 7fb743600640 1 --2- 172.21.15.69:0/343655562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb73c0d0300 0x7fb73c13d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:20.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.963+0000 7fb743600640 1 -- 172.21.15.69:0/343655562 >> 172.21.15.69:0/343655562 conn(0x7fb73c067d90 msgr2=0x7fb73c0b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:20.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.963+0000 7fb743600640 1 -- 172.21.15.69:0/343655562 shutdown_connections 2024-04-24T08:08:20.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:20.963+0000 7fb743600640 1 -- 172.21.15.69:0/343655562 wait complete. 2024-04-24T08:08:20.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T08:08:21.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.183+0000 7f1d94e00640 1 Processor -- start 2024-04-24T08:08:21.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.183+0000 7f1d94e00640 1 -- start start 2024-04-24T08:08:21.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d94e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d9015edf0 0x7f1d9015f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:21.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d94e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9006e000 0x7f1d9006e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:21.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d94e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d9015fbb0 con 0x7f1d90071d70 2024-04-24T08:08:21.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d94e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1d900d0a00 con 0x7f1d9015edf0 2024-04-24T08:08:21.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d94e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1d900d00d0 con 0x7f1d9006e000 2024-04-24T08:08:21.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d8ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d9015edf0 0x7f1d9015f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:21.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d8f400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1d90071d70 0x7f1d90072150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44606/0 (socket says 172.21.15.69:44606) 2024-04-24T08:08:21.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d8f400640 1 -- 172.21.15.69:0/2152697073 learned_addr learned my addr 172.21.15.69:0/2152697073 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:21.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d8ea00640 1 -- 172.21.15.69:0/2152697073 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9006e000 msgr2=0x7f1d9006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:21.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d8ea00640 1 --2- 172.21.15.69:0/2152697073 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9006e000 0x7f1d9006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:21.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d8ea00640 1 -- 172.21.15.69:0/2152697073 >> v1:172.21.15.69:6789/0 conn(0x7f1d90071d70 legacy=0x7f1d90072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:21.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d8ea00640 1 -- 172.21.15.69:0/2152697073 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d900d03b0 con 0x7f1d9015edf0 2024-04-24T08:08:21.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.185+0000 7f1d8e000640 1 -- 172.21.15.69:0/2152697073 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 788832909 0 0) 0x7f1d9015fbb0 con 0x7f1d90071d70 2024-04-24T08:08:21.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.186+0000 7f1d8ea00640 1 --2- 172.21.15.69:0/2152697073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9015edf0 0x7f1d9015f250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1d780097a0 tx=0x7f1d7802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ff65225895f4c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:21.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d8e000640 1 -- 172.21.15.69:0/2152697073 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d78037070 con 0x7f1d9015edf0 2024-04-24T08:08:21.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d8e000640 1 -- 172.21.15.69:0/2152697073 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d78032440 con 0x7f1d9015edf0 2024-04-24T08:08:21.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d8e000640 1 -- 172.21.15.69:0/2152697073 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d78032810 con 0x7f1d9015edf0 2024-04-24T08:08:21.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d94e00640 1 -- 172.21.15.69:0/2152697073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9015edf0 msgr2=0x7f1d9015f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:21.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d94e00640 1 --2- 172.21.15.69:0/2152697073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9015edf0 0x7f1d9015f250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1d780097a0 tx=0x7f1d7802a180 comp rx=0 tx=0).stop 2024-04-24T08:08:21.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d94e00640 1 -- 172.21.15.69:0/2152697073 shutdown_connections 2024-04-24T08:08:21.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d94e00640 1 --2- 172.21.15.69:0/2152697073 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d9006e000 0x7f1d9006e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:21.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d94e00640 1 --2- 172.21.15.69:0/2152697073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9015edf0 0x7f1d9015f250 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:21.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d94e00640 1 -- 172.21.15.69:0/2152697073 >> 172.21.15.69:0/2152697073 conn(0x7f1d90067b80 msgr2=0x7f1d9006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:21.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d94e00640 1 -- 172.21.15.69:0/2152697073 shutdown_connections 2024-04-24T08:08:21.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.187+0000 7f1d94e00640 1 -- 172.21.15.69:0/2152697073 wait complete. 2024-04-24T08:08:21.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.189+0000 7f1d94e00640 1 Processor -- start 2024-04-24T08:08:21.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.189+0000 7f1d94e00640 1 -- start start 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.190+0000 7f1d94e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9006e000 0x7f1d900cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.190+0000 7f1d94e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d90071d70 0x7f1d9015de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.190+0000 7f1d94e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9015edf0 0x7f1d9013d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.190+0000 7f1d94e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d900d1010 con 0x7f1d9006e000 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.190+0000 7f1d94e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1d900d0e90 con 0x7f1d9015edf0 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.190+0000 7f1d94e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1d901481a0 con 0x7f1d90071d70 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.190+0000 7f1d8f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9006e000 0x7f1d900cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.190+0000 7f1d8f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9006e000 0x7f1d900cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41136/0 (socket says 172.21.15.69:41136) 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.190+0000 7f1d8f400640 1 -- 172.21.15.69:0/1833941280 learned_addr learned my addr 172.21.15.69:0/1833941280 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d8f400640 1 -- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d90071d70 msgr2=0x7f1d9015de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d8f400640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d90071d70 0x7f1d9015de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d8f400640 1 -- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9015edf0 msgr2=0x7f1d9013d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d8f400640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9015edf0 0x7f1d9013d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:21.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d8f400640 1 -- 172.21.15.69:0/1833941280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d900cd7b0 con 0x7f1d9006e000 2024-04-24T08:08:21.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d8f400640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9006e000 0x7f1d900cc5c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000c500 tx=0x7f1d8000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:21.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d87400640 1 -- 172.21.15.69:0/1833941280 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d80018070 con 0x7f1d9006e000 2024-04-24T08:08:21.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d87400640 1 -- 172.21.15.69:0/1833941280 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d80007580 con 0x7f1d9006e000 2024-04-24T08:08:21.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d94e00640 1 -- 172.21.15.69:0/1833941280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d900ccba0 con 0x7f1d9006e000 2024-04-24T08:08:21.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d87400640 1 -- 172.21.15.69:0/1833941280 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d80004a20 con 0x7f1d9006e000 2024-04-24T08:08:21.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.191+0000 7f1d94e00640 1 -- 172.21.15.69:0/1833941280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d900cd160 con 0x7f1d9006e000 2024-04-24T08:08:21.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.192+0000 7f1d94e00640 1 -- 172.21.15.69:0/1833941280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d90075d50 con 0x7f1d9006e000 2024-04-24T08:08:21.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.193+0000 7f1d87400640 1 -- 172.21.15.69:0/1833941280 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d8001b070 con 0x7f1d9006e000 2024-04-24T08:08:21.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.194+0000 7f1d87400640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d6c06c8f0 0x7f1d6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:21.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.194+0000 7f1d87400640 1 -- 172.21.15.69:0/1833941280 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1d8008ae30 con 0x7f1d9006e000 2024-04-24T08:08:21.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.195+0000 7f1d8ea00640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d6c06c8f0 0x7f1d6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.196+0000 7f1d8ea00640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d6c06c8f0 0x7f1d6c06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1d78034000 tx=0x7f1d7802c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.198+0000 7f1d87400640 1 -- 172.21.15.69:0/1833941280 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d80059250 con 0x7f1d9006e000 2024-04-24T08:08:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:21.420+0000 7f1d94e00640 1 -- 172.21.15.69:0/1833941280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f1d9006be30 con 0x7f1d9006e000 2024-04-24T08:08:22.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.119+0000 7f1d87400640 1 -- 172.21.15.69:0/1833941280 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) ==== 118+0+0 (secure 0 0 0) 0x7f1d80058b50 con 0x7f1d9006e000 2024-04-24T08:08:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.122+0000 7f1d94e00640 1 -- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d6c06c8f0 msgr2=0x7f1d6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.122+0000 7f1d94e00640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d6c06c8f0 0x7f1d6c06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1d78034000 tx=0x7f1d7802c000 comp rx=0 tx=0).stop 2024-04-24T08:08:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.122+0000 7f1d94e00640 1 -- 172.21.15.69:0/1833941280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9006e000 msgr2=0x7f1d900cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.122+0000 7f1d94e00640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9006e000 0x7f1d900cc5c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1d8000c500 tx=0x7f1d8000c9c0 comp rx=0 tx=0).stop 2024-04-24T08:08:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.123+0000 7f1d94e00640 1 -- 172.21.15.69:0/1833941280 shutdown_connections 2024-04-24T08:08:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.123+0000 7f1d94e00640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d6c06c8f0 0x7f1d6c06edb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.123+0000 7f1d94e00640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d9015edf0 0x7f1d9013d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.123+0000 7f1d94e00640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d90071d70 0x7f1d9015de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.123+0000 7f1d94e00640 1 --2- 172.21.15.69:0/1833941280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d9006e000 0x7f1d900cc5c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.123+0000 7f1d94e00640 1 -- 172.21.15.69:0/1833941280 >> 172.21.15.69:0/1833941280 conn(0x7f1d90067b80 msgr2=0x7f1d90144640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.123+0000 7f1d94e00640 1 -- 172.21.15.69:0/1833941280 shutdown_connections 2024-04-24T08:08:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.123+0000 7f1d94e00640 1 -- 172.21.15.69:0/1833941280 wait complete. 2024-04-24T08:08:22.146 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-24T08:08:22.146 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T08:08:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.350+0000 7faff9400640 1 Processor -- start 2024-04-24T08:08:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.351+0000 7faff9400640 1 -- start start 2024-04-24T08:08:22.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.352+0000 7faff9400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff415ec30 0x7faff40d1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:22.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.352+0000 7faff9400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faff40d2710 con 0x7faff415e850 2024-04-24T08:08:22.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.352+0000 7faff9400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faff40d29f0 con 0x7faff40d1690 2024-04-24T08:08:22.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.352+0000 7faff9400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faff40d37b0 con 0x7faff415ec30 2024-04-24T08:08:22.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.352+0000 7faff2800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7faff415e850 0x7faff40cff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44622/0 (socket says 172.21.15.69:44622) 2024-04-24T08:08:22.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.352+0000 7faff2800640 1 -- 172.21.15.69:0/470717919 learned_addr learned my addr 172.21.15.69:0/470717919 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:22.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.353+0000 7fafe9e00640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff415ec30 0x7faff40d1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:22.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.353+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 810561435 0 0) 0x7faff40d2710 con 0x7faff415e850 2024-04-24T08:08:22.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.353+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafd8003520 con 0x7faff415e850 2024-04-24T08:08:22.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.353+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 974259330 0 0) 0x7fafd8003520 con 0x7faff415e850 2024-04-24T08:08:22.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.353+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff415ec30 msgr2=0x7faff40d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:22.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.353+0000 7faff1e00640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff415ec30 0x7faff40d1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:22.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.353+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 >> v1:172.21.15.121:6789/0 conn(0x7faff40d1690 legacy=0x7faff41688f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.353+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff40d2cd0 con 0x7faff415e850 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.353+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fafe40019f0 con 0x7faff415e850 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 >> v1:172.21.15.69:6789/0 conn(0x7faff415e850 legacy=0x7faff40cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff1e00640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 0x7fafd8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff1e00640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd8004800 0x7fafd8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff1e00640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff415ec30 0x7fafd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faff40d37b0 con 0x7fafd80039e0 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fafd800bad0 con 0x7fafd8004800 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fafd800bcd0 con 0x7faff415ec30 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff2800640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 0x7fafd8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7fafe9e00640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd8004800 0x7fafd8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:22.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff2800640 1 -- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff415ec30 msgr2=0x7fafd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:22.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff3200640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff415ec30 0x7fafd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:22.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff2800640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff415ec30 0x7fafd80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:22.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff2800640 1 -- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd8004800 msgr2=0x7fafd8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:22.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff2800640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd8004800 0x7fafd8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:22.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff2800640 1 -- 172.21.15.69:0/470717919 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd8003520 con 0x7fafd80039e0 2024-04-24T08:08:22.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7fafe9e00640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd8004800 0x7fafd8004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:22.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.354+0000 7faff3200640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff415ec30 0x7fafd80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:22.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.355+0000 7faff2800640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 0x7fafd8003e00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fafe400a740 tx=0x7fafd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd98018416e6290 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:22.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.356+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe400a390 con 0x7fafd80039e0 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.356+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafe402f070 con 0x7fafd80039e0 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.356+0000 7faff1e00640 1 -- 172.21.15.69:0/470717919 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe402bb90 con 0x7fafd80039e0 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.356+0000 7faff9400640 1 -- 172.21.15.69:0/470717919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 msgr2=0x7fafd8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.356+0000 7faff9400640 1 --2- 172.21.15.69:0/470717919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 0x7fafd8003e00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fafe400a740 tx=0x7fafd8005830 comp rx=0 tx=0).stop 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.356+0000 7faff3200640 1 -- 172.21.15.69:0/470717919 reap_dead start 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.356+0000 7faff9400640 1 -- 172.21.15.69:0/470717919 shutdown_connections 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.356+0000 7faff9400640 1 -- 172.21.15.69:0/470717919 >> 172.21.15.69:0/470717919 conn(0x7faff408bf60 msgr2=0x7faff40d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.356+0000 7faff9400640 1 -- 172.21.15.69:0/470717919 shutdown_connections 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.356+0000 7faff9400640 1 -- 172.21.15.69:0/470717919 wait complete. 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.358+0000 7faff9400640 1 Processor -- start 2024-04-24T08:08:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.358+0000 7faff9400640 1 -- start start 2024-04-24T08:08:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.359+0000 7faff9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 0x7faff414ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.359+0000 7faff9400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faff40cdce0 0x7faff40b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.359+0000 7faff9400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff414b4f0 0x7faff414de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.359+0000 7faff9400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faff40d1ae0 con 0x7fafd80039e0 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.359+0000 7faff9400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7faff406eba0 con 0x7faff40cdce0 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.359+0000 7faff9400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faff40d3290 con 0x7faff414b4f0 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.359+0000 7faff2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 0x7faff414ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7faff2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 0x7faff414ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41150/0 (socket says 172.21.15.69:41150) 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7faff2800640 1 -- 172.21.15.69:0/677184542 learned_addr learned my addr 172.21.15.69:0/677184542 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7fafe9e00640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faff40cdce0 0x7faff40b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7faff3200640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff414b4f0 0x7faff414de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7faff2800640 1 -- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff414b4f0 msgr2=0x7faff414de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7faff2800640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff414b4f0 0x7faff414de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7faff2800640 1 -- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faff40cdce0 msgr2=0x7faff40b7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7faff2800640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faff40cdce0 0x7faff40b7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7faff2800640 1 -- 172.21.15.69:0/677184542 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff40b78c0 con 0x7fafd80039e0 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7fafe9e00640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faff40cdce0 0x7faff40b7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.360+0000 7faff2800640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 0x7faff414ea00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fafe402b3e0 tx=0x7faff40d2ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.361+0000 7fafeb400640 1 -- 172.21.15.69:0/677184542 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe4001ee0 con 0x7fafd80039e0 2024-04-24T08:08:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.361+0000 7fafeb400640 1 -- 172.21.15.69:0/677184542 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafe4001250 con 0x7fafd80039e0 2024-04-24T08:08:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.361+0000 7fafeb400640 1 -- 172.21.15.69:0/677184542 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe40346b0 con 0x7fafd80039e0 2024-04-24T08:08:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.361+0000 7faff9400640 1 -- 172.21.15.69:0/677184542 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faff40b9f90 con 0x7fafd80039e0 2024-04-24T08:08:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.361+0000 7faff9400640 1 -- 172.21.15.69:0/677184542 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faff40cd4e0 con 0x7fafd80039e0 2024-04-24T08:08:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.362+0000 7faff9400640 1 -- 172.21.15.69:0/677184542 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faff4068a80 con 0x7fafd80039e0 2024-04-24T08:08:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.363+0000 7fafeb400640 1 -- 172.21.15.69:0/677184542 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fafe4034850 con 0x7fafd80039e0 2024-04-24T08:08:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.364+0000 7fafeb400640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafd006c940 0x7fafd006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.364+0000 7fafeb400640 1 -- 172.21.15.69:0/677184542 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fafe40ac460 con 0x7fafd80039e0 2024-04-24T08:08:22.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.365+0000 7fafe9e00640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafd006c940 0x7fafd006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:22.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.366+0000 7fafe9e00640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafd006c940 0x7fafd006ee00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faff4151e20 tx=0x7fafdc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:22.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.368+0000 7fafeb400640 1 -- 172.21.15.69:0/677184542 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fafe407a970 con 0x7fafd80039e0 2024-04-24T08:08:22.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:22.604+0000 7faff9400640 1 -- 172.21.15.69:0/677184542 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7faff41612d0 con 0x7fafd80039e0 2024-04-24T08:08:23.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.137+0000 7fafeb400640 1 -- 172.21.15.69:0/677184542 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7faff41612d0 con 0x7fafd80039e0 2024-04-24T08:08:23.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 -- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafd006c940 msgr2=0x7fafd006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafd006c940 0x7fafd006ee00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faff4151e20 tx=0x7fafdc008040 comp rx=0 tx=0).stop 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 -- 172.21.15.69:0/677184542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 msgr2=0x7faff414ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 0x7faff414ea00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fafe402b3e0 tx=0x7faff40d2ea0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 -- 172.21.15.69:0/677184542 shutdown_connections 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafd006c940 0x7fafd006ee00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faff414b4f0 0x7faff414de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faff40cdce0 0x7faff40b7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 --2- 172.21.15.69:0/677184542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd80039e0 0x7faff414ea00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 -- 172.21.15.69:0/677184542 >> 172.21.15.69:0/677184542 conn(0x7faff408bf60 msgr2=0x7faff4161940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 -- 172.21.15.69:0/677184542 shutdown_connections 2024-04-24T08:08:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.142+0000 7faff9400640 1 -- 172.21.15.69:0/677184542 wait complete. 2024-04-24T08:08:23.165 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T08:08:23.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:08:23.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.363+0000 7f0871e00640 1 Processor -- start 2024-04-24T08:08:23.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.364+0000 7f0871e00640 1 -- start start 2024-04-24T08:08:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.365+0000 7f0871e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f086c15d1d0 0x7f086c0ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.365+0000 7f0871e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f086c15d660 con 0x7f086c0b30f0 2024-04-24T08:08:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.365+0000 7f0871e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f086c06dc30 con 0x7f086c156c60 2024-04-24T08:08:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.365+0000 7f0871e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f086c06df10 con 0x7f086c15d1d0 2024-04-24T08:08:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.365+0000 7f0863e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f086c15d1d0 0x7f086c0ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.365+0000 7f0871400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f086c0b30f0 0x7f086c06e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44638/0 (socket says 172.21.15.69:44638) 2024-04-24T08:08:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.366+0000 7f0871400640 1 -- 172.21.15.69:0/629970659 learned_addr learned my addr 172.21.15.69:0/629970659 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:23.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.366+0000 7f0863e00640 1 -- 172.21.15.69:0/629970659 >> v1:172.21.15.121:6789/0 conn(0x7f086c156c60 legacy=0x7f086c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:23.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.366+0000 7f0863e00640 1 -- 172.21.15.69:0/629970659 >> v1:172.21.15.69:6789/0 conn(0x7f086c0b30f0 legacy=0x7f086c06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.366+0000 7f0863e00640 1 -- 172.21.15.69:0/629970659 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086c06e1f0 con 0x7f086c15d1d0 2024-04-24T08:08:23.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.367+0000 7f0863e00640 1 --2- 172.21.15.69:0/629970659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c15d1d0 0x7f086c0ad0d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08540097a0 tx=0x7f085402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97cc851e093f3bec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:23.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.367+0000 7f086be00640 1 -- 172.21.15.69:0/629970659 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854037070 con 0x7f086c15d1d0 2024-04-24T08:08:23.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.367+0000 7f086be00640 1 -- 172.21.15.69:0/629970659 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0854032440 con 0x7f086c15d1d0 2024-04-24T08:08:23.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.367+0000 7f086be00640 1 -- 172.21.15.69:0/629970659 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854032810 con 0x7f086c15d1d0 2024-04-24T08:08:23.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.368+0000 7f0871e00640 1 -- 172.21.15.69:0/629970659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c15d1d0 msgr2=0x7f086c0ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.368+0000 7f0871e00640 1 --2- 172.21.15.69:0/629970659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c15d1d0 0x7f086c0ad0d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08540097a0 tx=0x7f085402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.368+0000 7f0871e00640 1 -- 172.21.15.69:0/629970659 shutdown_connections 2024-04-24T08:08:23.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.368+0000 7f0871e00640 1 --2- 172.21.15.69:0/629970659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c15d1d0 0x7f086c0ad0d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.368+0000 7f0871e00640 1 -- 172.21.15.69:0/629970659 >> 172.21.15.69:0/629970659 conn(0x7f086c0d0160 msgr2=0x7f086c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:23.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.368+0000 7f0871e00640 1 -- 172.21.15.69:0/629970659 shutdown_connections 2024-04-24T08:08:23.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.368+0000 7f0871e00640 1 -- 172.21.15.69:0/629970659 wait complete. 2024-04-24T08:08:23.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.369+0000 7f0871e00640 1 Processor -- start 2024-04-24T08:08:23.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.370+0000 7f0871e00640 1 -- start start 2024-04-24T08:08:23.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.370+0000 7f0871e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f086c0b30f0 0x7f086c1f4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0871e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c156c60 0x7f086c0ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0871e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f086c155c80 0x7f086c0cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0871e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f086c0ad610 con 0x7f086c156c60 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0871e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f086c15d940 con 0x7f086c155c80 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0871e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f086c0ad790 con 0x7f086c0b30f0 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0863e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c156c60 0x7f086c0ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0863e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c156c60 0x7f086c0ca610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41166/0 (socket says 172.21.15.69:41166) 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0863e00640 1 -- 172.21.15.69:0/2124045591 learned_addr learned my addr 172.21.15.69:0/2124045591 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0870a00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f086c0b30f0 0x7f086c1f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0871400640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f086c155c80 0x7f086c0cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0863e00640 1 -- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f086c0b30f0 msgr2=0x7f086c1f4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0863e00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f086c0b30f0 0x7f086c1f4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0863e00640 1 -- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f086c155c80 msgr2=0x7f086c0cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0863e00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f086c155c80 0x7f086c0cab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0863e00640 1 -- 172.21.15.69:0/2124045591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086c072790 con 0x7f086c156c60 2024-04-24T08:08:23.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.371+0000 7f0870a00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f086c0b30f0 0x7f086c1f4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:23.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.372+0000 7f0863e00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c156c60 0x7f086c0ca610 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f08540097a0 tx=0x7f08540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:23.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.372+0000 7f0869600640 1 -- 172.21.15.69:0/2124045591 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854041070 con 0x7f086c156c60 2024-04-24T08:08:23.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.372+0000 7f0869600640 1 -- 172.21.15.69:0/2124045591 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085403c870 con 0x7f086c156c60 2024-04-24T08:08:23.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.372+0000 7f0869600640 1 -- 172.21.15.69:0/2124045591 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854037040 con 0x7f086c156c60 2024-04-24T08:08:23.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.372+0000 7f0871e00640 1 -- 172.21.15.69:0/2124045591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086c072a30 con 0x7f086c156c60 2024-04-24T08:08:23.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.372+0000 7f0871e00640 1 -- 172.21.15.69:0/2124045591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f086c072ff0 con 0x7f086c156c60 2024-04-24T08:08:23.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.373+0000 7f0871e00640 1 -- 172.21.15.69:0/2124045591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f086c075d50 con 0x7f086c156c60 2024-04-24T08:08:23.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.374+0000 7f0869600640 1 -- 172.21.15.69:0/2124045591 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0854044070 con 0x7f086c156c60 2024-04-24T08:08:23.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.375+0000 7f0869600640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f084806c920 0x7f084806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.375+0000 7f0869600640 1 -- 172.21.15.69:0/2124045591 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f08540ac720 con 0x7f086c156c60 2024-04-24T08:08:23.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.376+0000 7f0870a00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f084806c920 0x7f084806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.377+0000 7f0870a00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f084806c920 0x7f084806ede0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f085c003eb0 tx=0x7f085c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:23.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.379+0000 7f0869600640 1 -- 172.21.15.69:0/2124045591 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f085407abd0 con 0x7f086c156c60 2024-04-24T08:08:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.599+0000 7f0871e00640 1 -- 172.21.15.69:0/2124045591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f086c0b1b60 con 0x7f086c156c60 2024-04-24T08:08:23.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.600+0000 7f0869600640 1 -- 172.21.15.69:0/2124045591 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f085407a4d0 con 0x7f086c156c60 2024-04-24T08:08:23.601 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:23.601 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/934340733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":934340733},{"type":"v1","addr":"172.21.15.121:6835","nonce":934340733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3140867113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3140867113},{"type":"v1","addr":"172.21.15.121:6837","nonce":3140867113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/1059995973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1059995973},{"type":"v1","addr":"172.21.15.69:6833","nonce":1059995973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:08:19.574083+0000","modified":"2024-04-24T08:08:22.605668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/1003206529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1003206529},{"type":"v1","addr":"172.21.15.69:6835","nonce":1003206529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-24T08:08:23.601 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 7 2024-04-24T08:08:23.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 -- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f084806c920 msgr2=0x7f084806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f084806c920 0x7f084806ede0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f085c003eb0 tx=0x7f085c009040 comp rx=0 tx=0).stop 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 -- 172.21.15.69:0/2124045591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c156c60 msgr2=0x7f086c0ca610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c156c60 0x7f086c0ca610 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f08540097a0 tx=0x7f08540046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 -- 172.21.15.69:0/2124045591 shutdown_connections 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f084806c920 0x7f084806ede0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c156c60 0x7f086c0ca610 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f086c155c80 0x7f086c0cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 --2- 172.21.15.69:0/2124045591 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f086c0b30f0 0x7f086c1f4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 -- 172.21.15.69:0/2124045591 >> 172.21.15.69:0/2124045591 conn(0x7f086c0d0160 msgr2=0x7f086c0b34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 -- 172.21.15.69:0/2124045591 shutdown_connections 2024-04-24T08:08:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.603+0000 7f0871e00640 1 -- 172.21.15.69:0/2124045591 wait complete. 2024-04-24T08:08:23.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:23.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.802+0000 7f39b7e00640 1 Processor -- start 2024-04-24T08:08:23.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.802+0000 7f39b7e00640 1 -- start start 2024-04-24T08:08:23.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.804+0000 7f39b7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39b8156c60 0x7f39b815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.804+0000 7f39b7e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f39b80b30f0 0x7f39b815d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.804+0000 7f39b7e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39b815d660 con 0x7f39b80ad0d0 2024-04-24T08:08:23.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.804+0000 7f39b7e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f39b806dc30 con 0x7f39b8156c60 2024-04-24T08:08:23.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.804+0000 7f39b7e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f39b806df10 con 0x7f39b80b30f0 2024-04-24T08:08:23.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.804+0000 7f39b6a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39b8156c60 0x7f39b815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.804+0000 7f39b7400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f39b80ad0d0 0x7f39b81600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44654/0 (socket says 172.21.15.69:44654) 2024-04-24T08:08:23.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.804+0000 7f39b7400640 1 -- 172.21.15.69:0/1466987186 learned_addr learned my addr 172.21.15.69:0/1466987186 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:23.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.805+0000 7f39b6000640 1 --2- 172.21.15.69:0/1466987186 >> v2:172.21.15.121:3300/0 conn(0x7f39b80b30f0 0x7f39b815d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.805+0000 7f39b6a00640 1 -- 172.21.15.69:0/1466987186 >> v2:172.21.15.121:3300/0 conn(0x7f39b80b30f0 msgr2=0x7f39b815d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.805+0000 7f39b6a00640 1 --2- 172.21.15.69:0/1466987186 >> v2:172.21.15.121:3300/0 conn(0x7f39b80b30f0 0x7f39b815d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.805+0000 7f39b6a00640 1 -- 172.21.15.69:0/1466987186 >> v1:172.21.15.69:6789/0 conn(0x7f39b80ad0d0 legacy=0x7f39b81600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.805+0000 7f39b6a00640 1 -- 172.21.15.69:0/1466987186 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b806e1f0 con 0x7f39b8156c60 2024-04-24T08:08:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.805+0000 7f39b5600640 1 -- 172.21.15.69:0/1466987186 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4096407219 0 0) 0x7f39b815d660 con 0x7f39b80ad0d0 2024-04-24T08:08:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.806+0000 7f39b6a00640 1 --2- 172.21.15.69:0/1466987186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 0x7f39b815cac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f39b806d900 tx=0x7f39a802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9770ba6475f9b56f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.806+0000 7f39b5600640 1 -- 172.21.15.69:0/1466987186 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a8037070 con 0x7f39b8156c60 2024-04-24T08:08:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.806+0000 7f39b5600640 1 -- 172.21.15.69:0/1466987186 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a8032440 con 0x7f39b8156c60 2024-04-24T08:08:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.806+0000 7f39b5600640 1 -- 172.21.15.69:0/1466987186 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a8032810 con 0x7f39b8156c60 2024-04-24T08:08:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.807+0000 7f39b7e00640 1 -- 172.21.15.69:0/1466987186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 msgr2=0x7f39b815cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.807+0000 7f39b7e00640 1 --2- 172.21.15.69:0/1466987186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 0x7f39b815cac0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f39b806d900 tx=0x7f39a802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.807+0000 7f39b7e00640 1 -- 172.21.15.69:0/1466987186 shutdown_connections 2024-04-24T08:08:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.807+0000 7f39b7e00640 1 --2- 172.21.15.69:0/1466987186 >> v2:172.21.15.121:3300/0 conn(0x7f39b80b30f0 0x7f39b815d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.807+0000 7f39b7e00640 1 --2- 172.21.15.69:0/1466987186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 0x7f39b815cac0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.807+0000 7f39b7e00640 1 -- 172.21.15.69:0/1466987186 >> 172.21.15.69:0/1466987186 conn(0x7f39b80d0160 msgr2=0x7f39b80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.807+0000 7f39b7e00640 1 -- 172.21.15.69:0/1466987186 shutdown_connections 2024-04-24T08:08:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.807+0000 7f39b7e00640 1 -- 172.21.15.69:0/1466987186 wait complete. 2024-04-24T08:08:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.809+0000 7f39b7e00640 1 Processor -- start 2024-04-24T08:08:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.809+0000 7f39b7e00640 1 -- start start 2024-04-24T08:08:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.810+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b80ad0d0 0x7f39b81555f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.810+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39b80b30f0 0x7f39b8158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.810+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 0x7f39b8159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.810+0000 7f39b7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39b80ad540 con 0x7f39b8156c60 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.810+0000 7f39b7e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f39b815d940 con 0x7f39b80b30f0 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.810+0000 7f39b7e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f39b80ad6c0 con 0x7f39b80ad0d0 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.810+0000 7f39b7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 0x7f39b8159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.810+0000 7f39b7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 0x7f39b8159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41184/0 (socket says 172.21.15.69:41184) 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.810+0000 7f39b7400640 1 -- 172.21.15.69:0/1669586754 learned_addr learned my addr 172.21.15.69:0/1669586754 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.810+0000 7f39b6000640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39b80b30f0 0x7f39b8158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39b6a00640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b80ad0d0 0x7f39b81555f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39b7400640 1 -- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b80ad0d0 msgr2=0x7f39b81555f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39b7400640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b80ad0d0 0x7f39b81555f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39b7400640 1 -- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39b80b30f0 msgr2=0x7f39b8158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39b7400640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39b80b30f0 0x7f39b8158f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39b7400640 1 -- 172.21.15.69:0/1669586754 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b8143ed0 con 0x7f39b8156c60 2024-04-24T08:08:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39b7400640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 0x7f39b8159490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39b000c7d0 tx=0x7f39b000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39aea00640 1 -- 172.21.15.69:0/1669586754 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0018070 con 0x7f39b8156c60 2024-04-24T08:08:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39aea00640 1 -- 172.21.15.69:0/1669586754 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b000a5e0 con 0x7f39b8156c60 2024-04-24T08:08:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39aea00640 1 -- 172.21.15.69:0/1669586754 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b000a9d0 con 0x7f39b8156c60 2024-04-24T08:08:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39b7e00640 1 -- 172.21.15.69:0/1669586754 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b8144de0 con 0x7f39b8156c60 2024-04-24T08:08:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.811+0000 7f39b7e00640 1 -- 172.21.15.69:0/1669586754 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b81453a0 con 0x7f39b8156c60 2024-04-24T08:08:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.812+0000 7f39b7e00640 1 -- 172.21.15.69:0/1669586754 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39b81460c0 con 0x7f39b8156c60 2024-04-24T08:08:23.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.814+0000 7f39aea00640 1 -- 172.21.15.69:0/1669586754 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f39b000ab70 con 0x7f39b8156c60 2024-04-24T08:08:23.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.815+0000 7f39aea00640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f399806c920 0x7f399806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:23.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.815+0000 7f39aea00640 1 -- 172.21.15.69:0/1669586754 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f39b008b180 con 0x7f39b8156c60 2024-04-24T08:08:23.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.816+0000 7f39b6a00640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f399806c920 0x7f399806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:23.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.817+0000 7f39b6a00640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f399806c920 0x7f399806ede0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f39a8034000 tx=0x7f39a802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:23.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:23.820+0000 7f39aea00640 1 -- 172.21.15.69:0/1669586754 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f39b81460c0 con 0x7f39b8156c60 2024-04-24T08:08:24.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.012+0000 7f39b7e00640 1 -- 172.21.15.69:0/1669586754 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39b8075d50 con 0x7f39b8156c60 2024-04-24T08:08:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.014+0000 7f39aea00640 1 -- 172.21.15.69:0/1669586754 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7f39b00595a0 con 0x7f39b8156c60 2024-04-24T08:08:24.015 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:24.016 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 -- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f399806c920 msgr2=0x7f399806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f399806c920 0x7f399806ede0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f39a8034000 tx=0x7f39a802e000 comp rx=0 tx=0).stop 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 -- 172.21.15.69:0/1669586754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 msgr2=0x7f39b8159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 0x7f39b8159490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f39b000c7d0 tx=0x7f39b000cc90 comp rx=0 tx=0).stop 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 -- 172.21.15.69:0/1669586754 shutdown_connections 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f399806c920 0x7f399806ede0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39b8156c60 0x7f39b8159490 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39b80b30f0 0x7f39b8158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 --2- 172.21.15.69:0/1669586754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39b80ad0d0 0x7f39b81555f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 -- 172.21.15.69:0/1669586754 >> 172.21.15.69:0/1669586754 conn(0x7f39b80d0160 msgr2=0x7f39b815ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 -- 172.21.15.69:0/1669586754 shutdown_connections 2024-04-24T08:08:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.017+0000 7f39b7e00640 1 -- 172.21.15.69:0/1669586754 wait complete. 2024-04-24T08:08:24.035 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T08:08:24.035 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:08:24.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.224+0000 7f95a3200640 1 Processor -- start 2024-04-24T08:08:24.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.224+0000 7f95a3200640 1 -- start start 2024-04-24T08:08:24.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.225+0000 7f95a3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95a4156c60 0x7f95a415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.225+0000 7f95a3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f95a40b30f0 0x7f95a415d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.225+0000 7f95a3200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95a415d660 con 0x7f95a40ad0d0 2024-04-24T08:08:24.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.225+0000 7f95a3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f95a406dc30 con 0x7f95a40b30f0 2024-04-24T08:08:24.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.225+0000 7f95a3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f95a406df10 con 0x7f95a4156c60 2024-04-24T08:08:24.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.226+0000 7f95a1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f95a40b30f0 0x7f95a415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.226+0000 7f95a1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f95a40b30f0 0x7f95a415d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41194/0 (socket says 172.21.15.69:41194) 2024-04-24T08:08:24.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.226+0000 7f95a1400640 1 -- 172.21.15.69:0/1274205855 learned_addr learned my addr 172.21.15.69:0/1274205855 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.226+0000 7f95a1e00640 1 --2- 172.21.15.69:0/1274205855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95a4156c60 0x7f95a415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.226+0000 7f95a1400640 1 -- 172.21.15.69:0/1274205855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95a4156c60 msgr2=0x7f95a415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.226+0000 7f95a1400640 1 --2- 172.21.15.69:0/1274205855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95a4156c60 0x7f95a415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.226+0000 7f95a1400640 1 -- 172.21.15.69:0/1274205855 >> v1:172.21.15.121:6789/0 conn(0x7f95a40ad0d0 legacy=0x7f95a41600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:24.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.226+0000 7f95a1400640 1 -- 172.21.15.69:0/1274205855 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a406e1f0 con 0x7f95a40b30f0 2024-04-24T08:08:24.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.227+0000 7f95a1400640 1 --2- 172.21.15.69:0/1274205855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 0x7f95a415d000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9590009870 tx=0x7f959002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b36a9fcb6caac3c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:24.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.228+0000 7f95a0a00640 1 -- 172.21.15.69:0/1274205855 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590037070 con 0x7f95a40b30f0 2024-04-24T08:08:24.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.228+0000 7f95a0a00640 1 -- 172.21.15.69:0/1274205855 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9590032440 con 0x7f95a40b30f0 2024-04-24T08:08:24.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.228+0000 7f95a0a00640 1 -- 172.21.15.69:0/1274205855 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590032810 con 0x7f95a40b30f0 2024-04-24T08:08:24.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.228+0000 7f95a3200640 1 -- 172.21.15.69:0/1274205855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 msgr2=0x7f95a415d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.228+0000 7f95a3200640 1 --2- 172.21.15.69:0/1274205855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 0x7f95a415d000 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9590009870 tx=0x7f959002a270 comp rx=0 tx=0).stop 2024-04-24T08:08:24.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.229+0000 7f95a3200640 1 -- 172.21.15.69:0/1274205855 shutdown_connections 2024-04-24T08:08:24.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.229+0000 7f95a3200640 1 --2- 172.21.15.69:0/1274205855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 0x7f95a415d000 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.229+0000 7f95a3200640 1 --2- 172.21.15.69:0/1274205855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95a4156c60 0x7f95a415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.229+0000 7f95a3200640 1 -- 172.21.15.69:0/1274205855 >> 172.21.15.69:0/1274205855 conn(0x7f95a40d0160 msgr2=0x7f95a40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.229+0000 7f95a3200640 1 -- 172.21.15.69:0/1274205855 shutdown_connections 2024-04-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.229+0000 7f95a3200640 1 -- 172.21.15.69:0/1274205855 wait complete. 2024-04-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.230+0000 7f95a3200640 1 Processor -- start 2024-04-24T08:08:24.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.230+0000 7f95a3200640 1 -- start start 2024-04-24T08:08:24.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.231+0000 7f95a3200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95a40ad0d0 0x7f95a40c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.231+0000 7f95a3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 0x7f95a41f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.231+0000 7f95a3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95a4156c60 0x7f95a4072910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.231+0000 7f95a3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95a40ad530 con 0x7f95a40b30f0 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.231+0000 7f95a3200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f95a415d940 con 0x7f95a40ad0d0 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.231+0000 7f95a3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f95a40ad6b0 con 0x7f95a4156c60 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 0x7f95a41f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 0x7f95a41f42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41210/0 (socket says 172.21.15.69:41210) 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1400640 1 -- 172.21.15.69:0/212304843 learned_addr learned my addr 172.21.15.69:0/212304843 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1e00640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95a40ad0d0 0x7f95a40c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1400640 1 -- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95a4156c60 msgr2=0x7f95a4072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a2800640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95a4156c60 0x7f95a4072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1400640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95a4156c60 0x7f95a4072910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1400640 1 -- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95a40ad0d0 msgr2=0x7f95a40c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1400640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95a40ad0d0 0x7f95a40c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1400640 1 -- 172.21.15.69:0/212304843 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a40cbd00 con 0x7f95a40b30f0 2024-04-24T08:08:24.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1e00640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95a40ad0d0 0x7f95a40c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:24.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.232+0000 7f95a1400640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 0x7f95a41f42f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9590009870 tx=0x7f95900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:24.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.233+0000 7f959a000640 1 -- 172.21.15.69:0/212304843 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590041070 con 0x7f95a40b30f0 2024-04-24T08:08:24.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.233+0000 7f959a000640 1 -- 172.21.15.69:0/212304843 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f959003c890 con 0x7f95a40b30f0 2024-04-24T08:08:24.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.233+0000 7f959a000640 1 -- 172.21.15.69:0/212304843 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590037040 con 0x7f95a40b30f0 2024-04-24T08:08:24.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.233+0000 7f95a3200640 1 -- 172.21.15.69:0/212304843 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95a4071e30 con 0x7f95a40b30f0 2024-04-24T08:08:24.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.233+0000 7f95a3200640 1 -- 172.21.15.69:0/212304843 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a4072e50 con 0x7f95a40b30f0 2024-04-24T08:08:24.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.234+0000 7f95a3200640 1 -- 172.21.15.69:0/212304843 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a415b370 con 0x7f95a40b30f0 2024-04-24T08:08:24.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.235+0000 7f959a000640 1 -- 172.21.15.69:0/212304843 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9590044070 con 0x7f95a40b30f0 2024-04-24T08:08:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.236+0000 7f959a000640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f958406c8f0 0x7f958406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.236+0000 7f959a000640 1 -- 172.21.15.69:0/212304843 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f95900ac590 con 0x7f95a40b30f0 2024-04-24T08:08:24.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.236+0000 7f95a1e00640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f958406c8f0 0x7f958406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.237+0000 7f95a1e00640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f958406c8f0 0x7f958406edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9594006fd0 tx=0x7f9594008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:24.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.240+0000 7f959a000640 1 -- 172.21.15.69:0/212304843 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f95a415b370 con 0x7f95a40b30f0 2024-04-24T08:08:24.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.419+0000 7f95a3200640 1 -- 172.21.15.69:0/212304843 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f95a41f4830 con 0x7f958406c8f0 2024-04-24T08:08:24.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.440+0000 7f959a000640 1 -- 172.21.15.69:0/212304843 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240067 (secure 0 0 0) 0x7f95a41f4830 con 0x7f958406c8f0 2024-04-24T08:08:24.444 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:24.445 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:08:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.445+0000 7f95a3200640 1 -- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f958406c8f0 msgr2=0x7f958406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.445+0000 7f95a3200640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f958406c8f0 0x7f958406edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9594006fd0 tx=0x7f9594008040 comp rx=0 tx=0).stop 2024-04-24T08:08:24.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.445+0000 7f95a3200640 1 -- 172.21.15.69:0/212304843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 msgr2=0x7f95a41f42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.445+0000 7f95a3200640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 0x7f95a41f42f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9590009870 tx=0x7f95900046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.446+0000 7f95a3200640 1 -- 172.21.15.69:0/212304843 shutdown_connections 2024-04-24T08:08:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.446+0000 7f95a3200640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f958406c8f0 0x7f958406edb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.446+0000 7f95a3200640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95a40b30f0 0x7f95a41f42f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.446+0000 7f95a3200640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95a4156c60 0x7f95a4072910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.446+0000 7f95a3200640 1 --2- 172.21.15.69:0/212304843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95a40ad0d0 0x7f95a40c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.446+0000 7f95a3200640 1 -- 172.21.15.69:0/212304843 >> 172.21.15.69:0/212304843 conn(0x7f95a40d0160 msgr2=0x7f95a4154f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.446+0000 7f95a3200640 1 -- 172.21.15.69:0/212304843 shutdown_connections 2024-04-24T08:08:24.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.446+0000 7f95a3200640 1 -- 172.21.15.69:0/212304843 wait complete. 2024-04-24T08:08:24.464 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-04-24T08:08:22.660832+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218580,"kb_used_data":2400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756140,"statfs":{"total":773094113280,"available":772870287360,"internally_reserved":0,"allocated":2457600,"data_stored":1558506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.571409"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.095952+0000","last_change":"2024-04-24T08:08:18.095952+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.095952+0000","last_undegraded":"2024-04-24T08:08:18.095952+0000","last_fullsized":"2024-04-24T08:08:18.095952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.108975+0000","last_change":"2024-04-24T08:08:18.108975+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.108975+0000","last_undegraded":"2024-04-24T08:08:18.108975+0000","last_fullsized":"2024-04-24T08:08:18.108975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.095530+0000","last_change":"2024-04-24T08:08:18.095530+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.095530+0000","last_undegraded":"2024-04-24T08:08:18.095530+0000","last_fullsized":"2024-04-24T08:08:18.095530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.103138+0000","last_change":"2024-04-24T08:08:18.103138+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.103138+0000","last_undegraded":"2024-04-24T08:08:18.103138+0000","last_fullsized":"2024-04-24T08:08:18.103138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.110930+0000","last_change":"2024-04-24T08:08:18.110930+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.110930+0000","last_undegraded":"2024-04-24T08:08:18.110930+0000","last_fullsized":"2024-04-24T08:08:18.110930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.097728+0000","last_change":"2024-04-24T08:08:18.097728+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.097728+0000","last_undegraded":"2024-04-24T08:08:18.097728+0000","last_fullsized":"2024-04-24T08:08:18.097728+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.108111+0000","last_change":"2024-04-24T08:08:18.108111+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.108111+0000","last_undegraded":"2024-04-24T08:08:18.108111+0000","last_fullsized":"2024-04-24T08:08:18.108111+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.107940+0000","last_change":"2024-04-24T08:08:18.107940+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.107940+0000","last_undegraded":"2024-04-24T08:08:18.107940+0000","last_fullsized":"2024-04-24T08:08:18.107940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.100253+0000","last_change":"2024-04-24T08:08:18.100253+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.100253+0000","last_undegraded":"2024-04-24T08:08:18.100253+0000","last_fullsized":"2024-04-24T08:08:18.100253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.104449+0000","last_change":"2024-04-24T08:08:18.104449+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.104449+0000","last_undegraded":"2024-04-24T08:08:18.104449+0000","last_fullsized":"2024-04-24T08:08:18.104449+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.106133+0000","last_change":"2024-04-24T08:08:18.106133+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.106133+0000","last_undegraded":"2024-04-24T08:08:18.106133+0000","last_fullsized":"2024-04-24T08:08:18.106133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.111515+0000","last_change":"2024-04-24T08:08:18.111515+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.111515+0000","last_undegraded":"2024-04-24T08:08:18.111515+0000","last_fullsized":"2024-04-24T08:08:18.111515+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.109938+0000","last_change":"2024-04-24T08:08:18.109938+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.109938+0000","last_undegraded":"2024-04-24T08:08:18.109938+0000","last_fullsized":"2024-04-24T08:08:18.109938+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.110764+0000","last_change":"2024-04-24T08:08:18.110764+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.110764+0000","last_undegraded":"2024-04-24T08:08:18.110764+0000","last_fullsized":"2024-04-24T08:08:18.110764+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.104601+0000","last_change":"2024-04-24T08:08:18.104601+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.104601+0000","last_undegraded":"2024-04-24T08:08:18.104601+0000","last_fullsized":"2024-04-24T08:08:18.104601+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.110784+0000","last_change":"2024-04-24T08:08:18.110784+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.110784+0000","last_undegraded":"2024-04-24T08:08:18.110784+0000","last_fullsized":"2024-04-24T08:08:18.110784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-04-24T08:08:18.101112+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:08:18.101112+0000","last_peered":"2024-04-24T08:08:18.101112+0000","last_clean":"2024-04-24T08:08:18.101112+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:08:18.101112+0000","last_undegraded":"2024-04-24T08:08:18.101112+0000","last_fullsized":"2024-04-24T08:08:18.101112+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.096860+0000","last_change":"2024-04-24T08:08:18.096860+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.096860+0000","last_undegraded":"2024-04-24T08:08:18.096860+0000","last_fullsized":"2024-04-24T08:08:18.096860+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.096856+0000","last_change":"2024-04-24T08:08:18.096856+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.096856+0000","last_undegraded":"2024-04-24T08:08:18.096856+0000","last_fullsized":"2024-04-24T08:08:18.096856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.111070+0000","last_change":"2024-04-24T08:08:18.111070+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.111070+0000","last_undegraded":"2024-04-24T08:08:18.111070+0000","last_fullsized":"2024-04-24T08:08:18.111070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.099155+0000","last_change":"2024-04-24T08:08:18.099155+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.099155+0000","last_undegraded":"2024-04-24T08:08:18.099155+0000","last_fullsized":"2024-04-24T08:08:18.099155+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.103966+0000","last_change":"2024-04-24T08:08:18.103966+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.103966+0000","last_undegraded":"2024-04-24T08:08:18.103966+0000","last_fullsized":"2024-04-24T08:08:18.103966+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.098820+0000","last_change":"2024-04-24T08:08:18.098820+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.098820+0000","last_undegraded":"2024-04-24T08:08:18.098820+0000","last_fullsized":"2024-04-24T08:08:18.098820+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.104619+0000","last_change":"2024-04-24T08:08:18.104619+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.104619+0000","last_undegraded":"2024-04-24T08:08:18.104619+0000","last_fullsized":"2024-04-24T08:08:18.104619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.110088+0000","last_change":"2024-04-24T08:08:18.110088+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.110088+0000","last_undegraded":"2024-04-24T08:08:18.110088+0000","last_fullsized":"2024-04-24T08:08:18.110088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.099178+0000","last_change":"2024-04-24T08:08:18.099178+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.099178+0000","last_undegraded":"2024-04-24T08:08:18.099178+0000","last_fullsized":"2024-04-24T08:08:18.099178+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.104738+0000","last_change":"2024-04-24T08:08:18.104738+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.104738+0000","last_undegraded":"2024-04-24T08:08:18.104738+0000","last_fullsized":"2024-04-24T08:08:18.104738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.105140+0000","last_change":"2024-04-24T08:08:18.105140+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.105140+0000","last_undegraded":"2024-04-24T08:08:18.105140+0000","last_fullsized":"2024-04-24T08:08:18.105140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.096865+0000","last_change":"2024-04-24T08:08:18.096865+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.096865+0000","last_undegraded":"2024-04-24T08:08:18.096865+0000","last_fullsized":"2024-04-24T08:08:18.096865+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.096875+0000","last_change":"2024-04-24T08:08:18.096875+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.096875+0000","last_undegraded":"2024-04-24T08:08:18.096875+0000","last_fullsized":"2024-04-24T08:08:18.096875+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.100076+0000","last_change":"2024-04-24T08:08:18.100076+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.100076+0000","last_undegraded":"2024-04-24T08:08:18.100076+0000","last_fullsized":"2024-04-24T08:08:18.100076+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.095956+0000","last_change":"2024-04-24T08:08:18.095956+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.095956+0000","last_undegraded":"2024-04-24T08:08:18.095956+0000","last_fullsized":"2024-04-24T08:08:18.095956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:08:19.087922+0000","last_change":"2024-04-24T08:08:19.087922+0000","last_active":"2024-04-24T08:08:19.087922+0000","last_peered":"2024-04-24T08:08:19.087922+0000","last_clean":"2024-04-24T08:08:19.087922+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:19.087922+0000","last_undegraded":"2024-04-24T08:08:19.087922+0000","last_fullsized":"2024-04-24T08:08:19.087922+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-04-24T08:08:18.101336+0000","last_change":"2024-04-24T08:08:18.101336+0000","last_active":"2024-04-24T08:08:18.073365+0000","last_peered":"2024-04-24T08:08:18.073365+0000","last_clean":"2024-04-24T08:08:18.073365+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:08:18.101336+0000","last_undegraded":"2024-04-24T08:08:18.101336+0000","last_fullsized":"2024-04-24T08:08:18.101336+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":641952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":641952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:08:24.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:08:24.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.698+0000 7fa576c00640 1 Processor -- start 2024-04-24T08:08:24.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.699+0000 7fa576c00640 1 -- start start 2024-04-24T08:08:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.700+0000 7fa576c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570162150 0x7fa5700d5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.700+0000 7fa576c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa5700d10f0 0x7fa5700d1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.700+0000 7fa576c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5700d22d0 con 0x7fa5700cfe40 2024-04-24T08:08:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.700+0000 7fa576c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa5700d2900 con 0x7fa5700d10f0 2024-04-24T08:08:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.700+0000 7fa576c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa5700d03b0 con 0x7fa570162150 2024-04-24T08:08:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.700+0000 7fa56f400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa5700cfe40 0x7fa5700b7550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44664/0 (socket says 172.21.15.69:44664) 2024-04-24T08:08:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.700+0000 7fa56f400640 1 -- 172.21.15.69:0/757792527 learned_addr learned my addr 172.21.15.69:0/757792527 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.700+0000 7fa574c00640 1 --2- 172.21.15.69:0/757792527 >> v2:172.21.15.121:3300/0 conn(0x7fa5700d10f0 0x7fa5700d1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.700+0000 7fa56fe00640 1 --2- 172.21.15.69:0/757792527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570162150 0x7fa5700d5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.700+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 676458030 0 0) 0x7fa5700d22d0 con 0x7fa5700cfe40 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa554003520 con 0x7fa5700cfe40 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3412187704 0 0) 0x7fa554003520 con 0x7fa5700cfe40 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570162150 msgr2=0x7fa5700d5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 --2- 172.21.15.69:0/757792527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570162150 0x7fa5700d5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 >> v2:172.21.15.121:3300/0 conn(0x7fa5700d10f0 msgr2=0x7fa5700d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 --2- 172.21.15.69:0/757792527 >> v2:172.21.15.121:3300/0 conn(0x7fa5700d10f0 0x7fa5700d1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5700d0690 con 0x7fa5700cfe40 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fa55c001b70 con 0x7fa5700cfe40 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 >> v1:172.21.15.69:6789/0 conn(0x7fa5700cfe40 legacy=0x7fa5700b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 --2- 172.21.15.69:0/757792527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5540039b0 0x7fa554003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 --2- 172.21.15.69:0/757792527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570162150 0x7fa554004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 --2- 172.21.15.69:0/757792527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa554004d40 0x7fa5540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa5700d2900 con 0x7fa5540039b0 2024-04-24T08:08:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa5700d03b0 con 0x7fa554004d40 2024-04-24T08:08:24.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa55400b9c0 con 0x7fa570162150 2024-04-24T08:08:24.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.701+0000 7fa56fe00640 1 --2- 172.21.15.69:0/757792527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5540039b0 0x7fa554003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.702+0000 7fa56fe00640 1 -- 172.21.15.69:0/757792527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570162150 msgr2=0x7fa554004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:24.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.702+0000 7fa56fe00640 1 --2- 172.21.15.69:0/757792527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa570162150 0x7fa554004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.702+0000 7fa56fe00640 1 -- 172.21.15.69:0/757792527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa554004d40 msgr2=0x7fa5540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:24.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.702+0000 7fa56fe00640 1 --2- 172.21.15.69:0/757792527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa554004d40 0x7fa5540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.702+0000 7fa56fe00640 1 -- 172.21.15.69:0/757792527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa554003520 con 0x7fa5540039b0 2024-04-24T08:08:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa56fe00640 1 --2- 172.21.15.69:0/757792527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5540039b0 0x7fa554003dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa560009ab0 tx=0x7fa554005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=573fd18be0ae9690 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560036070 con 0x7fa5540039b0 2024-04-24T08:08:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa560005990 con 0x7fa5540039b0 2024-04-24T08:08:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa56ea00640 1 -- 172.21.15.69:0/757792527 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560009680 con 0x7fa5540039b0 2024-04-24T08:08:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa576c00640 1 -- 172.21.15.69:0/757792527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5540039b0 msgr2=0x7fa554003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa576c00640 1 --2- 172.21.15.69:0/757792527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5540039b0 0x7fa554003dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa560009ab0 tx=0x7fa554005830 comp rx=0 tx=0).stop 2024-04-24T08:08:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa574c00640 1 -- 172.21.15.69:0/757792527 reap_dead start 2024-04-24T08:08:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa576c00640 1 -- 172.21.15.69:0/757792527 shutdown_connections 2024-04-24T08:08:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa576c00640 1 -- 172.21.15.69:0/757792527 >> 172.21.15.69:0/757792527 conn(0x7fa57008bf60 msgr2=0x7fa5700d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa576c00640 1 -- 172.21.15.69:0/757792527 shutdown_connections 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.703+0000 7fa576c00640 1 -- 172.21.15.69:0/757792527 wait complete. 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.705+0000 7fa576c00640 1 Processor -- start 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.705+0000 7fa576c00640 1 -- start start 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.706+0000 7fa576c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5540039b0 0x7fa570155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.706+0000 7fa576c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57015af40 0x7fa5701532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.706+0000 7fa576c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5701537e0 0x7fa5700cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.706+0000 7fa576c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa5700d5020 con 0x7fa57015af40 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.706+0000 7fa576c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa5700b7c60 con 0x7fa5701537e0 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.706+0000 7fa576c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa5700d51a0 con 0x7fa5540039b0 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57015af40 0x7fa5701532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57015af40 0x7fa5701532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41230/0 (socket says 172.21.15.69:41230) 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56f400640 1 -- 172.21.15.69:0/1546994644 learned_addr learned my addr 172.21.15.69:0/1546994644 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa574c00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5701537e0 0x7fa5700cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56fe00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5540039b0 0x7fa570155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56f400640 1 -- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5540039b0 msgr2=0x7fa570155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56f400640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5540039b0 0x7fa570155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56f400640 1 -- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5701537e0 msgr2=0x7fa5700cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56f400640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5701537e0 0x7fa5700cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56f400640 1 -- 172.21.15.69:0/1546994644 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa570155b10 con 0x7fa57015af40 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56fe00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5540039b0 0x7fa570155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.707+0000 7fa56f400640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57015af40 0x7fa5701532a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa55c001460 tx=0x7fa55c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.708+0000 7fa567e00640 1 -- 172.21.15.69:0/1546994644 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c017070 con 0x7fa57015af40 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.708+0000 7fa567e00640 1 -- 172.21.15.69:0/1546994644 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55c00a890 con 0x7fa57015af40 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.708+0000 7fa567e00640 1 -- 172.21.15.69:0/1546994644 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c00ac80 con 0x7fa57015af40 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.708+0000 7fa576c00640 1 -- 172.21.15.69:0/1546994644 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa570155df0 con 0x7fa57015af40 2024-04-24T08:08:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.708+0000 7fa576c00640 1 -- 172.21.15.69:0/1546994644 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57015b3c0 con 0x7fa57015af40 2024-04-24T08:08:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.709+0000 7fa576c00640 1 -- 172.21.15.69:0/1546994644 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa570068400 con 0x7fa57015af40 2024-04-24T08:08:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.710+0000 7fa567e00640 1 -- 172.21.15.69:0/1546994644 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa55c01a070 con 0x7fa57015af40 2024-04-24T08:08:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.711+0000 7fa567e00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55006c920 0x7fa55006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.711+0000 7fa567e00640 1 -- 172.21.15.69:0/1546994644 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa55c08b570 con 0x7fa57015af40 2024-04-24T08:08:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.711+0000 7fa56fe00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55006c920 0x7fa55006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.713+0000 7fa56fe00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55006c920 0x7fa55006ede0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa560033000 tx=0x7fa560008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:24.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.716+0000 7fa567e00640 1 -- 172.21.15.69:0/1546994644 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa570068400 con 0x7fa57015af40 2024-04-24T08:08:24.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.912+0000 7fa576c00640 1 -- 172.21.15.69:0/1546994644 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa570068a80 con 0x7fa55006c920 2024-04-24T08:08:24.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.932+0000 7fa567e00640 1 -- 172.21.15.69:0/1546994644 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251947 (secure 0 0 0) 0x7fa570068a80 con 0x7fa55006c920 2024-04-24T08:08:24.937 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:24.938 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:08:24.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.938+0000 7fa576c00640 1 -- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55006c920 msgr2=0x7fa55006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55006c920 0x7fa55006ede0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa560033000 tx=0x7fa560008030 comp rx=0 tx=0).stop 2024-04-24T08:08:24.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 -- 172.21.15.69:0/1546994644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57015af40 msgr2=0x7fa5701532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:24.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57015af40 0x7fa5701532a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa55c001460 tx=0x7fa55c00a060 comp rx=0 tx=0).stop 2024-04-24T08:08:24.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 -- 172.21.15.69:0/1546994644 shutdown_connections 2024-04-24T08:08:24.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55006c920 0x7fa55006ede0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57015af40 0x7fa5701532a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5701537e0 0x7fa5700cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 --2- 172.21.15.69:0/1546994644 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5540039b0 0x7fa570155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:24.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 -- 172.21.15.69:0/1546994644 >> 172.21.15.69:0/1546994644 conn(0x7fa57008bf60 msgr2=0x7fa5700cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:24.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 -- 172.21.15.69:0/1546994644 shutdown_connections 2024-04-24T08:08:24.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:24.939+0000 7fa576c00640 1 -- 172.21.15.69:0/1546994644 wait complete. 2024-04-24T08:08:24.958 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-04-24T08:08:24.661519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218992,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755728,"statfs":{"total":773094113280,"available":772869865472,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001900"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539355+0000","last_change":"2024-04-24T08:08:20.145550+0000","last_active":"2024-04-24T08:08:22.539355+0000","last_peered":"2024-04-24T08:08:22.539355+0000","last_clean":"2024-04-24T08:08:22.539355+0000","last_became_active":"2024-04-24T08:08:20.144816+0000","last_became_peered":"2024-04-24T08:08:20.144816+0000","last_unstale":"2024-04-24T08:08:22.539355+0000","last_undegraded":"2024-04-24T08:08:22.539355+0000","last_fullsized":"2024-04-24T08:08:22.539355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:03:56.009457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.122230+0000","last_change":"2024-04-24T08:08:20.161217+0000","last_active":"2024-04-24T08:08:21.122230+0000","last_peered":"2024-04-24T08:08:21.122230+0000","last_clean":"2024-04-24T08:08:21.122230+0000","last_became_active":"2024-04-24T08:08:20.159365+0000","last_became_peered":"2024-04-24T08:08:20.159365+0000","last_unstale":"2024-04-24T08:08:21.122230+0000","last_undegraded":"2024-04-24T08:08:21.122230+0000","last_fullsized":"2024-04-24T08:08:21.122230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887063+0000","last_change":"2024-04-24T08:08:20.141345+0000","last_active":"2024-04-24T08:08:21.887063+0000","last_peered":"2024-04-24T08:08:21.887063+0000","last_clean":"2024-04-24T08:08:21.887063+0000","last_became_active":"2024-04-24T08:08:20.141134+0000","last_became_peered":"2024-04-24T08:08:20.141134+0000","last_unstale":"2024-04-24T08:08:21.887063+0000","last_undegraded":"2024-04-24T08:08:21.887063+0000","last_fullsized":"2024-04-24T08:08:21.887063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125297+0000","last_change":"2024-04-24T08:08:20.164461+0000","last_active":"2024-04-24T08:08:21.125297+0000","last_peered":"2024-04-24T08:08:21.125297+0000","last_clean":"2024-04-24T08:08:21.125297+0000","last_became_active":"2024-04-24T08:08:20.164118+0000","last_became_peered":"2024-04-24T08:08:20.164118+0000","last_unstale":"2024-04-24T08:08:21.125297+0000","last_undegraded":"2024-04-24T08:08:21.125297+0000","last_fullsized":"2024-04-24T08:08:21.125297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:34.119848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538852+0000","last_change":"2024-04-24T08:08:20.146319+0000","last_active":"2024-04-24T08:08:22.538852+0000","last_peered":"2024-04-24T08:08:22.538852+0000","last_clean":"2024-04-24T08:08:22.538852+0000","last_became_active":"2024-04-24T08:08:20.145969+0000","last_became_peered":"2024-04-24T08:08:20.145969+0000","last_unstale":"2024-04-24T08:08:22.538852+0000","last_undegraded":"2024-04-24T08:08:22.538852+0000","last_fullsized":"2024-04-24T08:08:22.538852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:32:29.020308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123393+0000","last_change":"2024-04-24T08:08:20.149172+0000","last_active":"2024-04-24T08:08:21.123393+0000","last_peered":"2024-04-24T08:08:21.123393+0000","last_clean":"2024-04-24T08:08:21.123393+0000","last_became_active":"2024-04-24T08:08:20.148447+0000","last_became_peered":"2024-04-24T08:08:20.148447+0000","last_unstale":"2024-04-24T08:08:21.123393+0000","last_undegraded":"2024-04-24T08:08:21.123393+0000","last_fullsized":"2024-04-24T08:08:21.123393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:53.941683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538379+0000","last_change":"2024-04-24T08:08:20.132800+0000","last_active":"2024-04-24T08:08:22.538379+0000","last_peered":"2024-04-24T08:08:22.538379+0000","last_clean":"2024-04-24T08:08:22.538379+0000","last_became_active":"2024-04-24T08:08:20.132442+0000","last_became_peered":"2024-04-24T08:08:20.132442+0000","last_unstale":"2024-04-24T08:08:22.538379+0000","last_undegraded":"2024-04-24T08:08:22.538379+0000","last_fullsized":"2024-04-24T08:08:22.538379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:34.119848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537521+0000","last_change":"2024-04-24T08:08:20.153541+0000","last_active":"2024-04-24T08:08:22.537521+0000","last_peered":"2024-04-24T08:08:22.537521+0000","last_clean":"2024-04-24T08:08:22.537521+0000","last_became_active":"2024-04-24T08:08:20.153214+0000","last_became_peered":"2024-04-24T08:08:20.153214+0000","last_unstale":"2024-04-24T08:08:22.537521+0000","last_undegraded":"2024-04-24T08:08:22.537521+0000","last_fullsized":"2024-04-24T08:08:22.537521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539513+0000","last_change":"2024-04-24T08:08:20.136784+0000","last_active":"2024-04-24T08:08:22.539513+0000","last_peered":"2024-04-24T08:08:22.539513+0000","last_clean":"2024-04-24T08:08:22.539513+0000","last_became_active":"2024-04-24T08:08:20.136552+0000","last_became_peered":"2024-04-24T08:08:20.136552+0000","last_unstale":"2024-04-24T08:08:22.539513+0000","last_undegraded":"2024-04-24T08:08:22.539513+0000","last_fullsized":"2024-04-24T08:08:22.539513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:02:22.840489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152584+0000","last_change":"2024-04-24T08:08:20.143039+0000","last_active":"2024-04-24T08:08:21.152584+0000","last_peered":"2024-04-24T08:08:21.152584+0000","last_clean":"2024-04-24T08:08:21.152584+0000","last_became_active":"2024-04-24T08:08:20.142388+0000","last_became_peered":"2024-04-24T08:08:20.142388+0000","last_unstale":"2024-04-24T08:08:21.152584+0000","last_undegraded":"2024-04-24T08:08:21.152584+0000","last_fullsized":"2024-04-24T08:08:21.152584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152537+0000","last_change":"2024-04-24T08:08:20.143995+0000","last_active":"2024-04-24T08:08:21.152537+0000","last_peered":"2024-04-24T08:08:21.152537+0000","last_clean":"2024-04-24T08:08:21.152537+0000","last_became_active":"2024-04-24T08:08:20.143674+0000","last_became_peered":"2024-04-24T08:08:20.143674+0000","last_unstale":"2024-04-24T08:08:21.152537+0000","last_undegraded":"2024-04-24T08:08:21.152537+0000","last_fullsized":"2024-04-24T08:08:21.152537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125663+0000","last_change":"2024-04-24T08:08:20.146375+0000","last_active":"2024-04-24T08:08:21.125663+0000","last_peered":"2024-04-24T08:08:21.125663+0000","last_clean":"2024-04-24T08:08:21.125663+0000","last_became_active":"2024-04-24T08:08:20.146053+0000","last_became_peered":"2024-04-24T08:08:20.146053+0000","last_unstale":"2024-04-24T08:08:21.125663+0000","last_undegraded":"2024-04-24T08:08:21.125663+0000","last_fullsized":"2024-04-24T08:08:21.125663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:50.353563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538060+0000","last_change":"2024-04-24T08:08:20.151989+0000","last_active":"2024-04-24T08:08:22.538060+0000","last_peered":"2024-04-24T08:08:22.538060+0000","last_clean":"2024-04-24T08:08:22.538060+0000","last_became_active":"2024-04-24T08:08:20.150076+0000","last_became_peered":"2024-04-24T08:08:20.150076+0000","last_unstale":"2024-04-24T08:08:22.538060+0000","last_undegraded":"2024-04-24T08:08:22.538060+0000","last_fullsized":"2024-04-24T08:08:22.538060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.886831+0000","last_change":"2024-04-24T08:08:20.157703+0000","last_active":"2024-04-24T08:08:21.886831+0000","last_peered":"2024-04-24T08:08:21.886831+0000","last_clean":"2024-04-24T08:08:21.886831+0000","last_became_active":"2024-04-24T08:08:20.151876+0000","last_became_peered":"2024-04-24T08:08:20.151876+0000","last_unstale":"2024-04-24T08:08:21.886831+0000","last_undegraded":"2024-04-24T08:08:21.886831+0000","last_fullsized":"2024-04-24T08:08:21.886831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:41.389051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121308+0000","last_change":"2024-04-24T08:08:20.146100+0000","last_active":"2024-04-24T08:08:21.121308+0000","last_peered":"2024-04-24T08:08:21.121308+0000","last_clean":"2024-04-24T08:08:21.121308+0000","last_became_active":"2024-04-24T08:08:20.144236+0000","last_became_peered":"2024-04-24T08:08:20.144236+0000","last_unstale":"2024-04-24T08:08:21.121308+0000","last_undegraded":"2024-04-24T08:08:21.121308+0000","last_fullsized":"2024-04-24T08:08:21.121308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123180+0000","last_change":"2024-04-24T08:08:20.137406+0000","last_active":"2024-04-24T08:08:21.123180+0000","last_peered":"2024-04-24T08:08:21.123180+0000","last_clean":"2024-04-24T08:08:21.123180+0000","last_became_active":"2024-04-24T08:08:20.137109+0000","last_became_peered":"2024-04-24T08:08:20.137109+0000","last_unstale":"2024-04-24T08:08:21.123180+0000","last_undegraded":"2024-04-24T08:08:21.123180+0000","last_fullsized":"2024-04-24T08:08:21.123180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126733+0000","last_change":"2024-04-24T08:08:20.141553+0000","last_active":"2024-04-24T08:08:21.126733+0000","last_peered":"2024-04-24T08:08:21.126733+0000","last_clean":"2024-04-24T08:08:21.126733+0000","last_became_active":"2024-04-24T08:08:20.141163+0000","last_became_peered":"2024-04-24T08:08:20.141163+0000","last_unstale":"2024-04-24T08:08:21.126733+0000","last_undegraded":"2024-04-24T08:08:21.126733+0000","last_fullsized":"2024-04-24T08:08:21.126733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:19:04.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362588+0000","last_change":"2024-04-24T08:08:20.143805+0000","last_active":"2024-04-24T08:08:21.362588+0000","last_peered":"2024-04-24T08:08:21.362588+0000","last_clean":"2024-04-24T08:08:21.362588+0000","last_became_active":"2024-04-24T08:08:20.143427+0000","last_became_peered":"2024-04-24T08:08:20.143427+0000","last_unstale":"2024-04-24T08:08:21.362588+0000","last_undegraded":"2024-04-24T08:08:21.362588+0000","last_fullsized":"2024-04-24T08:08:21.362588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:10:52.052957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539464+0000","last_change":"2024-04-24T08:08:20.139941+0000","last_active":"2024-04-24T08:08:22.539464+0000","last_peered":"2024-04-24T08:08:22.539464+0000","last_clean":"2024-04-24T08:08:22.539464+0000","last_became_active":"2024-04-24T08:08:20.139629+0000","last_became_peered":"2024-04-24T08:08:20.139629+0000","last_unstale":"2024-04-24T08:08:22.539464+0000","last_undegraded":"2024-04-24T08:08:22.539464+0000","last_fullsized":"2024-04-24T08:08:22.539464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:19:04.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152372+0000","last_change":"2024-04-24T08:08:20.148714+0000","last_active":"2024-04-24T08:08:21.152372+0000","last_peered":"2024-04-24T08:08:21.152372+0000","last_clean":"2024-04-24T08:08:21.152372+0000","last_became_active":"2024-04-24T08:08:20.148418+0000","last_became_peered":"2024-04-24T08:08:20.148418+0000","last_unstale":"2024-04-24T08:08:21.152372+0000","last_undegraded":"2024-04-24T08:08:21.152372+0000","last_fullsized":"2024-04-24T08:08:21.152372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887385+0000","last_change":"2024-04-24T08:08:20.131644+0000","last_active":"2024-04-24T08:08:21.887385+0000","last_peered":"2024-04-24T08:08:21.887385+0000","last_clean":"2024-04-24T08:08:21.887385+0000","last_became_active":"2024-04-24T08:08:20.131283+0000","last_became_peered":"2024-04-24T08:08:20.131283+0000","last_unstale":"2024-04-24T08:08:21.887385+0000","last_undegraded":"2024-04-24T08:08:21.887385+0000","last_fullsized":"2024-04-24T08:08:21.887385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123520+0000","last_change":"2024-04-24T08:08:20.137461+0000","last_active":"2024-04-24T08:08:21.123520+0000","last_peered":"2024-04-24T08:08:21.123520+0000","last_clean":"2024-04-24T08:08:21.123520+0000","last_became_active":"2024-04-24T08:08:20.137181+0000","last_became_peered":"2024-04-24T08:08:20.137181+0000","last_unstale":"2024-04-24T08:08:21.123520+0000","last_undegraded":"2024-04-24T08:08:21.123520+0000","last_fullsized":"2024-04-24T08:08:21.123520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:15.025368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537451+0000","last_change":"2024-04-24T08:08:20.146731+0000","last_active":"2024-04-24T08:08:22.537451+0000","last_peered":"2024-04-24T08:08:22.537451+0000","last_clean":"2024-04-24T08:08:22.537451+0000","last_became_active":"2024-04-24T08:08:20.146216+0000","last_became_peered":"2024-04-24T08:08:20.146216+0000","last_unstale":"2024-04-24T08:08:22.537451+0000","last_undegraded":"2024-04-24T08:08:22.537451+0000","last_fullsized":"2024-04-24T08:08:22.537451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123388+0000","last_change":"2024-04-24T08:08:20.159848+0000","last_active":"2024-04-24T08:08:21.123388+0000","last_peered":"2024-04-24T08:08:21.123388+0000","last_clean":"2024-04-24T08:08:21.123388+0000","last_became_active":"2024-04-24T08:08:20.159530+0000","last_became_peered":"2024-04-24T08:08:20.159530+0000","last_unstale":"2024-04-24T08:08:21.123388+0000","last_undegraded":"2024-04-24T08:08:21.123388+0000","last_fullsized":"2024-04-24T08:08:21.123388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537893+0000","last_change":"2024-04-24T08:08:20.153648+0000","last_active":"2024-04-24T08:08:22.537893+0000","last_peered":"2024-04-24T08:08:22.537893+0000","last_clean":"2024-04-24T08:08:22.537893+0000","last_became_active":"2024-04-24T08:08:20.153373+0000","last_became_peered":"2024-04-24T08:08:20.153373+0000","last_unstale":"2024-04-24T08:08:22.537893+0000","last_undegraded":"2024-04-24T08:08:22.537893+0000","last_fullsized":"2024-04-24T08:08:22.537893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152686+0000","last_change":"2024-04-24T08:08:20.136593+0000","last_active":"2024-04-24T08:08:21.152686+0000","last_peered":"2024-04-24T08:08:21.152686+0000","last_clean":"2024-04-24T08:08:21.152686+0000","last_became_active":"2024-04-24T08:08:20.136373+0000","last_became_peered":"2024-04-24T08:08:20.136373+0000","last_unstale":"2024-04-24T08:08:21.152686+0000","last_undegraded":"2024-04-24T08:08:21.152686+0000","last_fullsized":"2024-04-24T08:08:21.152686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121751+0000","last_change":"2024-04-24T08:08:20.161976+0000","last_active":"2024-04-24T08:08:21.121751+0000","last_peered":"2024-04-24T08:08:21.121751+0000","last_clean":"2024-04-24T08:08:21.121751+0000","last_became_active":"2024-04-24T08:08:20.161665+0000","last_became_peered":"2024-04-24T08:08:20.161665+0000","last_unstale":"2024-04-24T08:08:21.121751+0000","last_undegraded":"2024-04-24T08:08:21.121751+0000","last_fullsized":"2024-04-24T08:08:21.121751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538337+0000","last_change":"2024-04-24T08:08:20.155775+0000","last_active":"2024-04-24T08:08:22.538337+0000","last_peered":"2024-04-24T08:08:22.538337+0000","last_clean":"2024-04-24T08:08:22.538337+0000","last_became_active":"2024-04-24T08:08:20.152792+0000","last_became_peered":"2024-04-24T08:08:20.152792+0000","last_unstale":"2024-04-24T08:08:22.538337+0000","last_undegraded":"2024-04-24T08:08:22.538337+0000","last_fullsized":"2024-04-24T08:08:22.538337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:07:27.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887649+0000","last_change":"2024-04-24T08:08:20.152662+0000","last_active":"2024-04-24T08:08:21.887649+0000","last_peered":"2024-04-24T08:08:21.887649+0000","last_clean":"2024-04-24T08:08:21.887649+0000","last_became_active":"2024-04-24T08:08:20.143770+0000","last_became_peered":"2024-04-24T08:08:20.143770+0000","last_unstale":"2024-04-24T08:08:21.887649+0000","last_undegraded":"2024-04-24T08:08:21.887649+0000","last_fullsized":"2024-04-24T08:08:21.887649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123612+0000","last_change":"2024-04-24T08:08:20.164390+0000","last_active":"2024-04-24T08:08:21.123612+0000","last_peered":"2024-04-24T08:08:21.123612+0000","last_clean":"2024-04-24T08:08:21.123612+0000","last_became_active":"2024-04-24T08:08:20.164063+0000","last_became_peered":"2024-04-24T08:08:20.164063+0000","last_unstale":"2024-04-24T08:08:21.123612+0000","last_undegraded":"2024-04-24T08:08:21.123612+0000","last_fullsized":"2024-04-24T08:08:21.123612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:07:27.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362285+0000","last_change":"2024-04-24T08:08:20.140288+0000","last_active":"2024-04-24T08:08:21.362285+0000","last_peered":"2024-04-24T08:08:21.362285+0000","last_clean":"2024-04-24T08:08:21.362285+0000","last_became_active":"2024-04-24T08:08:20.139983+0000","last_became_peered":"2024-04-24T08:08:20.139983+0000","last_unstale":"2024-04-24T08:08:21.362285+0000","last_undegraded":"2024-04-24T08:08:21.362285+0000","last_fullsized":"2024-04-24T08:08:21.362285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:46.310762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151856+0000","last_change":"2024-04-24T08:08:20.143305+0000","last_active":"2024-04-24T08:08:21.151856+0000","last_peered":"2024-04-24T08:08:21.151856+0000","last_clean":"2024-04-24T08:08:21.151856+0000","last_became_active":"2024-04-24T08:08:20.142984+0000","last_became_peered":"2024-04-24T08:08:20.142984+0000","last_unstale":"2024-04-24T08:08:21.151856+0000","last_undegraded":"2024-04-24T08:08:21.151856+0000","last_fullsized":"2024-04-24T08:08:21.151856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:40.157193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124584+0000","last_change":"2024-04-24T08:08:20.139312+0000","last_active":"2024-04-24T08:08:21.124584+0000","last_peered":"2024-04-24T08:08:21.124584+0000","last_clean":"2024-04-24T08:08:21.124584+0000","last_became_active":"2024-04-24T08:08:20.137881+0000","last_became_peered":"2024-04-24T08:08:20.137881+0000","last_unstale":"2024-04-24T08:08:21.124584+0000","last_undegraded":"2024-04-24T08:08:21.124584+0000","last_fullsized":"2024-04-24T08:08:21.124584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:28.189776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538238+0000","last_change":"2024-04-24T08:08:19.144554+0000","last_active":"2024-04-24T08:08:22.538238+0000","last_peered":"2024-04-24T08:08:22.538238+0000","last_clean":"2024-04-24T08:08:22.538238+0000","last_became_active":"2024-04-24T08:08:19.144285+0000","last_became_peered":"2024-04-24T08:08:19.144285+0000","last_unstale":"2024-04-24T08:08:22.538238+0000","last_undegraded":"2024-04-24T08:08:22.538238+0000","last_fullsized":"2024-04-24T08:08:22.538238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:02:50.869449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887732+0000","last_change":"2024-04-24T08:08:20.153872+0000","last_active":"2024-04-24T08:08:21.887732+0000","last_peered":"2024-04-24T08:08:21.887732+0000","last_clean":"2024-04-24T08:08:21.887732+0000","last_became_active":"2024-04-24T08:08:20.153593+0000","last_became_peered":"2024-04-24T08:08:20.153593+0000","last_unstale":"2024-04-24T08:08:21.887732+0000","last_undegraded":"2024-04-24T08:08:21.887732+0000","last_fullsized":"2024-04-24T08:08:21.887732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:53.941683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151661+0000","last_change":"2024-04-24T08:08:19.145643+0000","last_active":"2024-04-24T08:08:21.151661+0000","last_peered":"2024-04-24T08:08:21.151661+0000","last_clean":"2024-04-24T08:08:21.151661+0000","last_became_active":"2024-04-24T08:08:19.141228+0000","last_became_peered":"2024-04-24T08:08:19.141228+0000","last_unstale":"2024-04-24T08:08:21.151661+0000","last_undegraded":"2024-04-24T08:08:21.151661+0000","last_fullsized":"2024-04-24T08:08:21.151661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537587+0000","last_change":"2024-04-24T08:08:19.150248+0000","last_active":"2024-04-24T08:08:22.537587+0000","last_peered":"2024-04-24T08:08:22.537587+0000","last_clean":"2024-04-24T08:08:22.537587+0000","last_became_active":"2024-04-24T08:08:19.149979+0000","last_became_peered":"2024-04-24T08:08:19.149979+0000","last_unstale":"2024-04-24T08:08:22.537587+0000","last_undegraded":"2024-04-24T08:08:22.537587+0000","last_fullsized":"2024-04-24T08:08:22.537587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:13:42.392189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538422+0000","last_change":"2024-04-24T08:08:20.155184+0000","last_active":"2024-04-24T08:08:22.538422+0000","last_peered":"2024-04-24T08:08:22.538422+0000","last_clean":"2024-04-24T08:08:22.538422+0000","last_became_active":"2024-04-24T08:08:20.154858+0000","last_became_peered":"2024-04-24T08:08:20.154858+0000","last_unstale":"2024-04-24T08:08:22.538422+0000","last_undegraded":"2024-04-24T08:08:22.538422+0000","last_fullsized":"2024-04-24T08:08:22.538422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:34.119848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123341+0000","last_change":"2024-04-24T08:08:19.138408+0000","last_active":"2024-04-24T08:08:21.123341+0000","last_peered":"2024-04-24T08:08:21.123341+0000","last_clean":"2024-04-24T08:08:21.123341+0000","last_became_active":"2024-04-24T08:08:19.137042+0000","last_became_peered":"2024-04-24T08:08:19.137042+0000","last_unstale":"2024-04-24T08:08:21.123341+0000","last_undegraded":"2024-04-24T08:08:21.123341+0000","last_fullsized":"2024-04-24T08:08:21.123341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121298+0000","last_change":"2024-04-24T08:08:20.164732+0000","last_active":"2024-04-24T08:08:21.121298+0000","last_peered":"2024-04-24T08:08:21.121298+0000","last_clean":"2024-04-24T08:08:21.121298+0000","last_became_active":"2024-04-24T08:08:20.164193+0000","last_became_peered":"2024-04-24T08:08:20.164193+0000","last_unstale":"2024-04-24T08:08:21.121298+0000","last_undegraded":"2024-04-24T08:08:21.121298+0000","last_fullsized":"2024-04-24T08:08:21.121298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152443+0000","last_change":"2024-04-24T08:08:19.151929+0000","last_active":"2024-04-24T08:08:21.152443+0000","last_peered":"2024-04-24T08:08:21.152443+0000","last_clean":"2024-04-24T08:08:21.152443+0000","last_became_active":"2024-04-24T08:08:19.151633+0000","last_became_peered":"2024-04-24T08:08:19.151633+0000","last_unstale":"2024-04-24T08:08:21.152443+0000","last_undegraded":"2024-04-24T08:08:21.152443+0000","last_fullsized":"2024-04-24T08:08:21.152443+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125502+0000","last_change":"2024-04-24T08:08:20.139547+0000","last_active":"2024-04-24T08:08:21.125502+0000","last_peered":"2024-04-24T08:08:21.125502+0000","last_clean":"2024-04-24T08:08:21.125502+0000","last_became_active":"2024-04-24T08:08:20.139316+0000","last_became_peered":"2024-04-24T08:08:20.139316+0000","last_unstale":"2024-04-24T08:08:21.125502+0000","last_undegraded":"2024-04-24T08:08:21.125502+0000","last_fullsized":"2024-04-24T08:08:21.125502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:37.292548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125476+0000","last_change":"2024-04-24T08:08:19.150665+0000","last_active":"2024-04-24T08:08:21.125476+0000","last_peered":"2024-04-24T08:08:21.125476+0000","last_clean":"2024-04-24T08:08:21.125476+0000","last_became_active":"2024-04-24T08:08:19.150327+0000","last_became_peered":"2024-04-24T08:08:19.150327+0000","last_unstale":"2024-04-24T08:08:21.125476+0000","last_undegraded":"2024-04-24T08:08:21.125476+0000","last_fullsized":"2024-04-24T08:08:21.125476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887629+0000","last_change":"2024-04-24T08:08:20.138279+0000","last_active":"2024-04-24T08:08:21.887629+0000","last_peered":"2024-04-24T08:08:21.887629+0000","last_clean":"2024-04-24T08:08:21.887629+0000","last_became_active":"2024-04-24T08:08:20.137776+0000","last_became_peered":"2024-04-24T08:08:20.137776+0000","last_unstale":"2024-04-24T08:08:21.887629+0000","last_undegraded":"2024-04-24T08:08:21.887629+0000","last_fullsized":"2024-04-24T08:08:21.887629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121623+0000","last_change":"2024-04-24T08:08:19.143886+0000","last_active":"2024-04-24T08:08:21.121623+0000","last_peered":"2024-04-24T08:08:21.121623+0000","last_clean":"2024-04-24T08:08:21.121623+0000","last_became_active":"2024-04-24T08:08:19.143606+0000","last_became_peered":"2024-04-24T08:08:19.143606+0000","last_unstale":"2024-04-24T08:08:21.121623+0000","last_undegraded":"2024-04-24T08:08:21.121623+0000","last_fullsized":"2024-04-24T08:08:21.121623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126085+0000","last_change":"2024-04-24T08:08:20.146770+0000","last_active":"2024-04-24T08:08:21.126085+0000","last_peered":"2024-04-24T08:08:21.126085+0000","last_clean":"2024-04-24T08:08:21.126085+0000","last_became_active":"2024-04-24T08:08:20.146470+0000","last_became_peered":"2024-04-24T08:08:20.146470+0000","last_unstale":"2024-04-24T08:08:21.126085+0000","last_undegraded":"2024-04-24T08:08:21.126085+0000","last_fullsized":"2024-04-24T08:08:21.126085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:46.310762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152147+0000","last_change":"2024-04-24T08:08:19.149833+0000","last_active":"2024-04-24T08:08:21.152147+0000","last_peered":"2024-04-24T08:08:21.152147+0000","last_clean":"2024-04-24T08:08:21.152147+0000","last_became_active":"2024-04-24T08:08:19.149391+0000","last_became_peered":"2024-04-24T08:08:19.149391+0000","last_unstale":"2024-04-24T08:08:21.152147+0000","last_undegraded":"2024-04-24T08:08:21.152147+0000","last_fullsized":"2024-04-24T08:08:21.152147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538676+0000","last_change":"2024-04-24T08:08:20.145962+0000","last_active":"2024-04-24T08:08:22.538676+0000","last_peered":"2024-04-24T08:08:22.538676+0000","last_clean":"2024-04-24T08:08:22.538676+0000","last_became_active":"2024-04-24T08:08:20.145644+0000","last_became_peered":"2024-04-24T08:08:20.145644+0000","last_unstale":"2024-04-24T08:08:22.538676+0000","last_undegraded":"2024-04-24T08:08:22.538676+0000","last_fullsized":"2024-04-24T08:08:22.538676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:37.292548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887814+0000","last_change":"2024-04-24T08:08:19.131316+0000","last_active":"2024-04-24T08:08:21.887814+0000","last_peered":"2024-04-24T08:08:21.887814+0000","last_clean":"2024-04-24T08:08:21.887814+0000","last_became_active":"2024-04-24T08:08:19.130895+0000","last_became_peered":"2024-04-24T08:08:19.130895+0000","last_unstale":"2024-04-24T08:08:21.887814+0000","last_undegraded":"2024-04-24T08:08:21.887814+0000","last_fullsized":"2024-04-24T08:08:21.887814+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:13:42.392189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.886980+0000","last_change":"2024-04-24T08:08:20.123273+0000","last_active":"2024-04-24T08:08:21.886980+0000","last_peered":"2024-04-24T08:08:21.886980+0000","last_clean":"2024-04-24T08:08:21.886980+0000","last_became_active":"2024-04-24T08:08:20.122506+0000","last_became_peered":"2024-04-24T08:08:20.122506+0000","last_unstale":"2024-04-24T08:08:21.886980+0000","last_undegraded":"2024-04-24T08:08:21.886980+0000","last_fullsized":"2024-04-24T08:08:21.886980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:40.157193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123661+0000","last_change":"2024-04-24T08:08:19.138216+0000","last_active":"2024-04-24T08:08:21.123661+0000","last_peered":"2024-04-24T08:08:21.123661+0000","last_clean":"2024-04-24T08:08:21.123661+0000","last_became_active":"2024-04-24T08:08:19.137882+0000","last_became_peered":"2024-04-24T08:08:19.137882+0000","last_unstale":"2024-04-24T08:08:21.123661+0000","last_undegraded":"2024-04-24T08:08:21.123661+0000","last_fullsized":"2024-04-24T08:08:21.123661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538907+0000","last_change":"2024-04-24T08:08:20.155343+0000","last_active":"2024-04-24T08:08:22.538907+0000","last_peered":"2024-04-24T08:08:22.538907+0000","last_clean":"2024-04-24T08:08:22.538907+0000","last_became_active":"2024-04-24T08:08:20.155112+0000","last_became_peered":"2024-04-24T08:08:20.155112+0000","last_unstale":"2024-04-24T08:08:22.538907+0000","last_undegraded":"2024-04-24T08:08:22.538907+0000","last_fullsized":"2024-04-24T08:08:22.538907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:36.215783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151585+0000","last_change":"2024-04-24T08:08:19.135440+0000","last_active":"2024-04-24T08:08:21.151585+0000","last_peered":"2024-04-24T08:08:21.151585+0000","last_clean":"2024-04-24T08:08:21.151585+0000","last_became_active":"2024-04-24T08:08:19.134946+0000","last_became_peered":"2024-04-24T08:08:19.134946+0000","last_unstale":"2024-04-24T08:08:21.151585+0000","last_undegraded":"2024-04-24T08:08:21.151585+0000","last_fullsized":"2024-04-24T08:08:21.151585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151891+0000","last_change":"2024-04-24T08:08:20.144523+0000","last_active":"2024-04-24T08:08:21.151891+0000","last_peered":"2024-04-24T08:08:21.151891+0000","last_clean":"2024-04-24T08:08:21.151891+0000","last_became_active":"2024-04-24T08:08:20.142266+0000","last_became_peered":"2024-04-24T08:08:20.142266+0000","last_unstale":"2024-04-24T08:08:21.151891+0000","last_undegraded":"2024-04-24T08:08:21.151891+0000","last_fullsized":"2024-04-24T08:08:21.151891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.361922+0000","last_change":"2024-04-24T08:08:19.110993+0000","last_active":"2024-04-24T08:08:21.361922+0000","last_peered":"2024-04-24T08:08:21.361922+0000","last_clean":"2024-04-24T08:08:21.361922+0000","last_became_active":"2024-04-24T08:08:19.110592+0000","last_became_peered":"2024-04-24T08:08:19.110592+0000","last_unstale":"2024-04-24T08:08:21.361922+0000","last_undegraded":"2024-04-24T08:08:21.361922+0000","last_fullsized":"2024-04-24T08:08:21.361922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538312+0000","last_change":"2024-04-24T08:08:20.149332+0000","last_active":"2024-04-24T08:08:22.538312+0000","last_peered":"2024-04-24T08:08:22.538312+0000","last_clean":"2024-04-24T08:08:22.538312+0000","last_became_active":"2024-04-24T08:08:20.148856+0000","last_became_peered":"2024-04-24T08:08:20.148856+0000","last_unstale":"2024-04-24T08:08:22.538312+0000","last_undegraded":"2024-04-24T08:08:22.538312+0000","last_fullsized":"2024-04-24T08:08:22.538312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:07:27.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125565+0000","last_change":"2024-04-24T08:08:19.128023+0000","last_active":"2024-04-24T08:08:21.125565+0000","last_peered":"2024-04-24T08:08:21.125565+0000","last_clean":"2024-04-24T08:08:21.125565+0000","last_became_active":"2024-04-24T08:08:19.126833+0000","last_became_peered":"2024-04-24T08:08:19.126833+0000","last_unstale":"2024-04-24T08:08:21.125565+0000","last_undegraded":"2024-04-24T08:08:21.125565+0000","last_fullsized":"2024-04-24T08:08:21.125565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.062494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539639+0000","last_change":"2024-04-24T08:08:20.144208+0000","last_active":"2024-04-24T08:08:22.539639+0000","last_peered":"2024-04-24T08:08:22.539639+0000","last_clean":"2024-04-24T08:08:22.539639+0000","last_became_active":"2024-04-24T08:08:20.143708+0000","last_became_peered":"2024-04-24T08:08:20.143708+0000","last_unstale":"2024-04-24T08:08:22.539639+0000","last_undegraded":"2024-04-24T08:08:22.539639+0000","last_fullsized":"2024-04-24T08:08:22.539639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:58:28.805520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362439+0000","last_change":"2024-04-24T08:08:19.137638+0000","last_active":"2024-04-24T08:08:21.362439+0000","last_peered":"2024-04-24T08:08:21.362439+0000","last_clean":"2024-04-24T08:08:21.362439+0000","last_became_active":"2024-04-24T08:08:19.137280+0000","last_became_peered":"2024-04-24T08:08:19.137280+0000","last_unstale":"2024-04-24T08:08:21.362439+0000","last_undegraded":"2024-04-24T08:08:21.362439+0000","last_fullsized":"2024-04-24T08:08:21.362439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:02:50.869449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539746+0000","last_change":"2024-04-24T08:08:20.140656+0000","last_active":"2024-04-24T08:08:22.539746+0000","last_peered":"2024-04-24T08:08:22.539746+0000","last_clean":"2024-04-24T08:08:22.539746+0000","last_became_active":"2024-04-24T08:08:20.140347+0000","last_became_peered":"2024-04-24T08:08:20.140347+0000","last_unstale":"2024-04-24T08:08:22.539746+0000","last_undegraded":"2024-04-24T08:08:22.539746+0000","last_fullsized":"2024-04-24T08:08:22.539746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:13:39.728625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121835+0000","last_change":"2024-04-24T08:08:19.127246+0000","last_active":"2024-04-24T08:08:21.121835+0000","last_peered":"2024-04-24T08:08:21.121835+0000","last_clean":"2024-04-24T08:08:21.121835+0000","last_became_active":"2024-04-24T08:08:19.126190+0000","last_became_peered":"2024-04-24T08:08:19.126190+0000","last_unstale":"2024-04-24T08:08:21.121835+0000","last_undegraded":"2024-04-24T08:08:21.121835+0000","last_fullsized":"2024-04-24T08:08:21.121835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126132+0000","last_change":"2024-04-24T08:08:20.161962+0000","last_active":"2024-04-24T08:08:21.126132+0000","last_peered":"2024-04-24T08:08:21.126132+0000","last_clean":"2024-04-24T08:08:21.126132+0000","last_became_active":"2024-04-24T08:08:20.161572+0000","last_became_peered":"2024-04-24T08:08:20.161572+0000","last_unstale":"2024-04-24T08:08:21.126132+0000","last_undegraded":"2024-04-24T08:08:21.126132+0000","last_fullsized":"2024-04-24T08:08:21.126132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:56:58.521686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362140+0000","last_change":"2024-04-24T08:08:19.140345+0000","last_active":"2024-04-24T08:08:21.362140+0000","last_peered":"2024-04-24T08:08:21.362140+0000","last_clean":"2024-04-24T08:08:21.362140+0000","last_became_active":"2024-04-24T08:08:19.140122+0000","last_became_peered":"2024-04-24T08:08:19.140122+0000","last_unstale":"2024-04-24T08:08:21.362140+0000","last_undegraded":"2024-04-24T08:08:21.362140+0000","last_fullsized":"2024-04-24T08:08:21.362140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539177+0000","last_change":"2024-04-24T08:08:20.137838+0000","last_active":"2024-04-24T08:08:22.539177+0000","last_peered":"2024-04-24T08:08:22.539177+0000","last_clean":"2024-04-24T08:08:22.539177+0000","last_became_active":"2024-04-24T08:08:20.137552+0000","last_became_peered":"2024-04-24T08:08:20.137552+0000","last_unstale":"2024-04-24T08:08:22.539177+0000","last_undegraded":"2024-04-24T08:08:22.539177+0000","last_fullsized":"2024-04-24T08:08:22.539177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:56:58.521686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538262+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:08:22.538262+0000","last_peered":"2024-04-24T08:08:22.538262+0000","last_clean":"2024-04-24T08:08:22.538262+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:08:22.538262+0000","last_undegraded":"2024-04-24T08:08:22.538262+0000","last_fullsized":"2024-04-24T08:08:22.538262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124383+0000","last_change":"2024-04-24T08:08:19.135677+0000","last_active":"2024-04-24T08:08:21.124383+0000","last_peered":"2024-04-24T08:08:21.124383+0000","last_clean":"2024-04-24T08:08:21.124383+0000","last_became_active":"2024-04-24T08:08:19.133935+0000","last_became_peered":"2024-04-24T08:08:19.133935+0000","last_unstale":"2024-04-24T08:08:21.124383+0000","last_undegraded":"2024-04-24T08:08:21.124383+0000","last_fullsized":"2024-04-24T08:08:21.124383+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537560+0000","last_change":"2024-04-24T08:08:20.148292+0000","last_active":"2024-04-24T08:08:22.537560+0000","last_peered":"2024-04-24T08:08:22.537560+0000","last_clean":"2024-04-24T08:08:22.537560+0000","last_became_active":"2024-04-24T08:08:20.147966+0000","last_became_peered":"2024-04-24T08:08:20.147966+0000","last_unstale":"2024-04-24T08:08:22.537560+0000","last_undegraded":"2024-04-24T08:08:22.537560+0000","last_fullsized":"2024-04-24T08:08:22.537560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537956+0000","last_change":"2024-04-24T08:08:19.148444+0000","last_active":"2024-04-24T08:08:22.537956+0000","last_peered":"2024-04-24T08:08:22.537956+0000","last_clean":"2024-04-24T08:08:22.537956+0000","last_became_active":"2024-04-24T08:08:19.148032+0000","last_became_peered":"2024-04-24T08:08:19.148032+0000","last_unstale":"2024-04-24T08:08:22.537956+0000","last_undegraded":"2024-04-24T08:08:22.537956+0000","last_fullsized":"2024-04-24T08:08:22.537956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.886751+0000","last_change":"2024-04-24T08:08:20.139133+0000","last_active":"2024-04-24T08:08:21.886751+0000","last_peered":"2024-04-24T08:08:21.886751+0000","last_clean":"2024-04-24T08:08:21.886751+0000","last_became_active":"2024-04-24T08:08:20.138841+0000","last_became_peered":"2024-04-24T08:08:20.138841+0000","last_unstale":"2024-04-24T08:08:21.886751+0000","last_undegraded":"2024-04-24T08:08:21.886751+0000","last_fullsized":"2024-04-24T08:08:21.886751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:29:21.637806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362188+0000","last_change":"2024-04-24T08:08:19.135322+0000","last_active":"2024-04-24T08:08:21.362188+0000","last_peered":"2024-04-24T08:08:21.362188+0000","last_clean":"2024-04-24T08:08:21.362188+0000","last_became_active":"2024-04-24T08:08:19.134982+0000","last_became_peered":"2024-04-24T08:08:19.134982+0000","last_unstale":"2024-04-24T08:08:21.362188+0000","last_undegraded":"2024-04-24T08:08:21.362188+0000","last_fullsized":"2024-04-24T08:08:21.362188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121142+0000","last_change":"2024-04-24T08:08:20.142759+0000","last_active":"2024-04-24T08:08:21.121142+0000","last_peered":"2024-04-24T08:08:21.121142+0000","last_clean":"2024-04-24T08:08:21.121142+0000","last_became_active":"2024-04-24T08:08:20.142474+0000","last_became_peered":"2024-04-24T08:08:20.142474+0000","last_unstale":"2024-04-24T08:08:21.121142+0000","last_undegraded":"2024-04-24T08:08:21.121142+0000","last_fullsized":"2024-04-24T08:08:21.121142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123701+0000","last_change":"2024-04-24T08:08:19.131093+0000","last_active":"2024-04-24T08:08:21.123701+0000","last_peered":"2024-04-24T08:08:21.123701+0000","last_clean":"2024-04-24T08:08:21.123701+0000","last_became_active":"2024-04-24T08:08:19.129809+0000","last_became_peered":"2024-04-24T08:08:19.129809+0000","last_unstale":"2024-04-24T08:08:21.123701+0000","last_undegraded":"2024-04-24T08:08:21.123701+0000","last_fullsized":"2024-04-24T08:08:21.123701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124958+0000","last_change":"2024-04-24T08:08:20.155888+0000","last_active":"2024-04-24T08:08:21.124958+0000","last_peered":"2024-04-24T08:08:21.124958+0000","last_clean":"2024-04-24T08:08:21.124958+0000","last_became_active":"2024-04-24T08:08:20.154101+0000","last_became_peered":"2024-04-24T08:08:20.154101+0000","last_unstale":"2024-04-24T08:08:21.124958+0000","last_undegraded":"2024-04-24T08:08:21.124958+0000","last_fullsized":"2024-04-24T08:08:21.124958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:49.918407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121420+0000","last_change":"2024-04-24T08:08:19.128197+0000","last_active":"2024-04-24T08:08:21.121420+0000","last_peered":"2024-04-24T08:08:21.121420+0000","last_clean":"2024-04-24T08:08:21.121420+0000","last_became_active":"2024-04-24T08:08:19.127218+0000","last_became_peered":"2024-04-24T08:08:19.127218+0000","last_unstale":"2024-04-24T08:08:21.121420+0000","last_undegraded":"2024-04-24T08:08:21.121420+0000","last_fullsized":"2024-04-24T08:08:21.121420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:45.296206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126415+0000","last_change":"2024-04-24T08:08:20.159839+0000","last_active":"2024-04-24T08:08:21.126415+0000","last_peered":"2024-04-24T08:08:21.126415+0000","last_clean":"2024-04-24T08:08:21.126415+0000","last_became_active":"2024-04-24T08:08:20.159475+0000","last_became_peered":"2024-04-24T08:08:20.159475+0000","last_unstale":"2024-04-24T08:08:21.126415+0000","last_undegraded":"2024-04-24T08:08:21.126415+0000","last_fullsized":"2024-04-24T08:08:21.126415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:03:56.009457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539328+0000","last_change":"2024-04-24T08:08:19.148608+0000","last_active":"2024-04-24T08:08:22.539328+0000","last_peered":"2024-04-24T08:08:22.539328+0000","last_clean":"2024-04-24T08:08:22.539328+0000","last_became_active":"2024-04-24T08:08:19.148299+0000","last_became_peered":"2024-04-24T08:08:19.148299+0000","last_unstale":"2024-04-24T08:08:22.539328+0000","last_undegraded":"2024-04-24T08:08:22.539328+0000","last_fullsized":"2024-04-24T08:08:22.539328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:45.296206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121527+0000","last_change":"2024-04-24T08:08:20.144511+0000","last_active":"2024-04-24T08:08:21.121527+0000","last_peered":"2024-04-24T08:08:21.121527+0000","last_clean":"2024-04-24T08:08:21.121527+0000","last_became_active":"2024-04-24T08:08:20.144184+0000","last_became_peered":"2024-04-24T08:08:20.144184+0000","last_unstale":"2024-04-24T08:08:21.121527+0000","last_undegraded":"2024-04-24T08:08:21.121527+0000","last_fullsized":"2024-04-24T08:08:21.121527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.122119+0000","last_change":"2024-04-24T08:08:19.149355+0000","last_active":"2024-04-24T08:08:21.122119+0000","last_peered":"2024-04-24T08:08:21.122119+0000","last_clean":"2024-04-24T08:08:21.122119+0000","last_became_active":"2024-04-24T08:08:19.148936+0000","last_became_peered":"2024-04-24T08:08:19.148936+0000","last_unstale":"2024-04-24T08:08:21.122119+0000","last_undegraded":"2024-04-24T08:08:21.122119+0000","last_fullsized":"2024-04-24T08:08:21.122119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:49.339006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538134+0000","last_change":"2024-04-24T08:08:20.153837+0000","last_active":"2024-04-24T08:08:22.538134+0000","last_peered":"2024-04-24T08:08:22.538134+0000","last_clean":"2024-04-24T08:08:22.538134+0000","last_became_active":"2024-04-24T08:08:20.150347+0000","last_became_peered":"2024-04-24T08:08:20.150347+0000","last_unstale":"2024-04-24T08:08:22.538134+0000","last_undegraded":"2024-04-24T08:08:22.538134+0000","last_fullsized":"2024-04-24T08:08:22.538134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:53.941683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152520+0000","last_change":"2024-04-24T08:08:19.143650+0000","last_active":"2024-04-24T08:08:21.152520+0000","last_peered":"2024-04-24T08:08:21.152520+0000","last_clean":"2024-04-24T08:08:21.152520+0000","last_became_active":"2024-04-24T08:08:19.143321+0000","last_became_peered":"2024-04-24T08:08:19.143321+0000","last_unstale":"2024-04-24T08:08:21.152520+0000","last_undegraded":"2024-04-24T08:08:21.152520+0000","last_fullsized":"2024-04-24T08:08:21.152520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887531+0000","last_change":"2024-04-24T08:08:20.142865+0000","last_active":"2024-04-24T08:08:21.887531+0000","last_peered":"2024-04-24T08:08:21.887531+0000","last_clean":"2024-04-24T08:08:21.887531+0000","last_became_active":"2024-04-24T08:08:20.142593+0000","last_became_peered":"2024-04-24T08:08:20.142593+0000","last_unstale":"2024-04-24T08:08:21.887531+0000","last_undegraded":"2024-04-24T08:08:21.887531+0000","last_fullsized":"2024-04-24T08:08:21.887531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537886+0000","last_change":"2024-04-24T08:08:19.146717+0000","last_active":"2024-04-24T08:08:22.537886+0000","last_peered":"2024-04-24T08:08:22.537886+0000","last_clean":"2024-04-24T08:08:22.537886+0000","last_became_active":"2024-04-24T08:08:19.146435+0000","last_became_peered":"2024-04-24T08:08:19.146435+0000","last_unstale":"2024-04-24T08:08:22.537886+0000","last_undegraded":"2024-04-24T08:08:22.537886+0000","last_fullsized":"2024-04-24T08:08:22.537886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124833+0000","last_change":"2024-04-24T08:08:20.153349+0000","last_active":"2024-04-24T08:08:21.124833+0000","last_peered":"2024-04-24T08:08:21.124833+0000","last_clean":"2024-04-24T08:08:21.124833+0000","last_became_active":"2024-04-24T08:08:20.153071+0000","last_became_peered":"2024-04-24T08:08:20.153071+0000","last_unstale":"2024-04-24T08:08:21.124833+0000","last_undegraded":"2024-04-24T08:08:21.124833+0000","last_fullsized":"2024-04-24T08:08:21.124833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:26.469631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121526+0000","last_change":"2024-04-24T08:08:19.141216+0000","last_active":"2024-04-24T08:08:21.121526+0000","last_peered":"2024-04-24T08:08:21.121526+0000","last_clean":"2024-04-24T08:08:21.121526+0000","last_became_active":"2024-04-24T08:08:19.140885+0000","last_became_peered":"2024-04-24T08:08:19.140885+0000","last_unstale":"2024-04-24T08:08:21.121526+0000","last_undegraded":"2024-04-24T08:08:21.121526+0000","last_fullsized":"2024-04-24T08:08:21.121526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537837+0000","last_change":"2024-04-24T08:08:20.139684+0000","last_active":"2024-04-24T08:08:22.537837+0000","last_peered":"2024-04-24T08:08:22.537837+0000","last_clean":"2024-04-24T08:08:22.537837+0000","last_became_active":"2024-04-24T08:08:20.139348+0000","last_became_peered":"2024-04-24T08:08:20.139348+0000","last_unstale":"2024-04-24T08:08:22.537837+0000","last_undegraded":"2024-04-24T08:08:22.537837+0000","last_fullsized":"2024-04-24T08:08:22.537837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121275+0000","last_change":"2024-04-24T08:08:19.137264+0000","last_active":"2024-04-24T08:08:21.121275+0000","last_peered":"2024-04-24T08:08:21.121275+0000","last_clean":"2024-04-24T08:08:21.121275+0000","last_became_active":"2024-04-24T08:08:19.136921+0000","last_became_peered":"2024-04-24T08:08:19.136921+0000","last_unstale":"2024-04-24T08:08:21.121275+0000","last_undegraded":"2024-04-24T08:08:21.121275+0000","last_fullsized":"2024-04-24T08:08:21.121275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538286+0000","last_change":"2024-04-24T08:08:20.158595+0000","last_active":"2024-04-24T08:08:22.538286+0000","last_peered":"2024-04-24T08:08:22.538286+0000","last_clean":"2024-04-24T08:08:22.538286+0000","last_became_active":"2024-04-24T08:08:20.158248+0000","last_became_peered":"2024-04-24T08:08:20.158248+0000","last_unstale":"2024-04-24T08:08:22.538286+0000","last_undegraded":"2024-04-24T08:08:22.538286+0000","last_fullsized":"2024-04-24T08:08:22.538286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:15.025368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539152+0000","last_change":"2024-04-24T08:08:19.143021+0000","last_active":"2024-04-24T08:08:22.539152+0000","last_peered":"2024-04-24T08:08:22.539152+0000","last_clean":"2024-04-24T08:08:22.539152+0000","last_became_active":"2024-04-24T08:08:19.142731+0000","last_became_peered":"2024-04-24T08:08:19.142731+0000","last_unstale":"2024-04-24T08:08:22.539152+0000","last_undegraded":"2024-04-24T08:08:22.539152+0000","last_fullsized":"2024-04-24T08:08:22.539152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:10:51.038401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151895+0000","last_change":"2024-04-24T08:08:20.141951+0000","last_active":"2024-04-24T08:08:21.151895+0000","last_peered":"2024-04-24T08:08:21.151895+0000","last_clean":"2024-04-24T08:08:21.151895+0000","last_became_active":"2024-04-24T08:08:20.141642+0000","last_became_peered":"2024-04-24T08:08:20.141642+0000","last_unstale":"2024-04-24T08:08:21.151895+0000","last_undegraded":"2024-04-24T08:08:21.151895+0000","last_fullsized":"2024-04-24T08:08:21.151895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126001+0000","last_change":"2024-04-24T08:08:20.152685+0000","last_active":"2024-04-24T08:08:21.126001+0000","last_peered":"2024-04-24T08:08:21.126001+0000","last_clean":"2024-04-24T08:08:21.126001+0000","last_became_active":"2024-04-24T08:08:20.152171+0000","last_became_peered":"2024-04-24T08:08:20.152171+0000","last_unstale":"2024-04-24T08:08:21.126001+0000","last_undegraded":"2024-04-24T08:08:21.126001+0000","last_fullsized":"2024-04-24T08:08:21.126001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:36.215783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538456+0000","last_change":"2024-04-24T08:08:19.147333+0000","last_active":"2024-04-24T08:08:22.538456+0000","last_peered":"2024-04-24T08:08:22.538456+0000","last_clean":"2024-04-24T08:08:22.538456+0000","last_became_active":"2024-04-24T08:08:19.147046+0000","last_became_peered":"2024-04-24T08:08:19.147046+0000","last_unstale":"2024-04-24T08:08:22.538456+0000","last_undegraded":"2024-04-24T08:08:22.538456+0000","last_fullsized":"2024-04-24T08:08:22.538456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:24:02.225409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125720+0000","last_change":"2024-04-24T08:08:20.152622+0000","last_active":"2024-04-24T08:08:21.125720+0000","last_peered":"2024-04-24T08:08:21.125720+0000","last_clean":"2024-04-24T08:08:21.125720+0000","last_became_active":"2024-04-24T08:08:20.151784+0000","last_became_peered":"2024-04-24T08:08:20.151784+0000","last_unstale":"2024-04-24T08:08:21.125720+0000","last_undegraded":"2024-04-24T08:08:21.125720+0000","last_fullsized":"2024-04-24T08:08:21.125720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:32:29.020308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538879+0000","last_change":"2024-04-24T08:08:19.150378+0000","last_active":"2024-04-24T08:08:22.538879+0000","last_peered":"2024-04-24T08:08:22.538879+0000","last_clean":"2024-04-24T08:08:22.538879+0000","last_became_active":"2024-04-24T08:08:19.149946+0000","last_became_peered":"2024-04-24T08:08:19.149946+0000","last_unstale":"2024-04-24T08:08:22.538879+0000","last_undegraded":"2024-04-24T08:08:22.538879+0000","last_fullsized":"2024-04-24T08:08:22.538879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:49.339006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887243+0000","last_change":"2024-04-24T08:08:20.138065+0000","last_active":"2024-04-24T08:08:21.887243+0000","last_peered":"2024-04-24T08:08:21.887243+0000","last_clean":"2024-04-24T08:08:21.887243+0000","last_became_active":"2024-04-24T08:08:20.137340+0000","last_became_peered":"2024-04-24T08:08:20.137340+0000","last_unstale":"2024-04-24T08:08:21.887243+0000","last_undegraded":"2024-04-24T08:08:21.887243+0000","last_fullsized":"2024-04-24T08:08:21.887243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538608+0000","last_change":"2024-04-24T08:08:19.134141+0000","last_active":"2024-04-24T08:08:22.538608+0000","last_peered":"2024-04-24T08:08:22.538608+0000","last_clean":"2024-04-24T08:08:22.538608+0000","last_became_active":"2024-04-24T08:08:19.133873+0000","last_became_peered":"2024-04-24T08:08:19.133873+0000","last_unstale":"2024-04-24T08:08:22.538608+0000","last_undegraded":"2024-04-24T08:08:22.538608+0000","last_fullsized":"2024-04-24T08:08:22.538608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538569+0000","last_change":"2024-04-24T08:08:20.146963+0000","last_active":"2024-04-24T08:08:22.538569+0000","last_peered":"2024-04-24T08:08:22.538569+0000","last_clean":"2024-04-24T08:08:22.538569+0000","last_became_active":"2024-04-24T08:08:20.146270+0000","last_became_peered":"2024-04-24T08:08:20.146270+0000","last_unstale":"2024-04-24T08:08:22.538569+0000","last_undegraded":"2024-04-24T08:08:22.538569+0000","last_fullsized":"2024-04-24T08:08:22.538569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:37.292548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887648+0000","last_change":"2024-04-24T08:08:19.142063+0000","last_active":"2024-04-24T08:08:21.887648+0000","last_peered":"2024-04-24T08:08:21.887648+0000","last_clean":"2024-04-24T08:08:21.887648+0000","last_became_active":"2024-04-24T08:08:19.141469+0000","last_became_peered":"2024-04-24T08:08:19.141469+0000","last_unstale":"2024-04-24T08:08:21.887648+0000","last_undegraded":"2024-04-24T08:08:21.887648+0000","last_fullsized":"2024-04-24T08:08:21.887648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:46.176456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:08:24.962 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T08:08:24.963 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T08:08:24.963 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-24T08:08:24.963 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-24T08:08:24.964 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T08:08:25.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.193+0000 7f9a9e800640 1 Processor -- start 2024-04-24T08:08:25.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.193+0000 7f9a9e800640 1 -- start start 2024-04-24T08:08:25.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.194+0000 7f9a9e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a980d0300 0x7f9a9806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.194+0000 7f9a9e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9a98145fe0 0x7f9a98160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.194+0000 7f9a9e800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a980cf3d0 con 0x7f9a98145a90 2024-04-24T08:08:25.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.194+0000 7f9a9e800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9a980cf6b0 con 0x7f9a98145fe0 2024-04-24T08:08:25.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.194+0000 7f9a9e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a980cf990 con 0x7f9a980d0300 2024-04-24T08:08:25.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.195+0000 7f9a96a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9a98145a90 0x7f9a980d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44670/0 (socket says 172.21.15.69:44670) 2024-04-24T08:08:25.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.195+0000 7f9a96a00640 1 -- 172.21.15.69:0/2361462041 learned_addr learned my addr 172.21.15.69:0/2361462041 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:25.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.195+0000 7f9a97e00640 1 --2- 172.21.15.69:0/2361462041 >> v2:172.21.15.69:3300/0 conn(0x7f9a98145fe0 0x7f9a98160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.195+0000 7f9a97400640 1 --2- 172.21.15.69:0/2361462041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a980d0300 0x7f9a9806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.195+0000 7f9a96000640 1 -- 172.21.15.69:0/2361462041 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2131487198 0 0) 0x7f9a980cf3d0 con 0x7f9a98145a90 2024-04-24T08:08:25.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.195+0000 7f9a96000640 1 -- 172.21.15.69:0/2361462041 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a7c003520 con 0x7f9a98145a90 2024-04-24T08:08:25.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.195+0000 7f9a97e00640 1 -- 172.21.15.69:0/2361462041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a980d0300 msgr2=0x7f9a9806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:25.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.196+0000 7f9a97e00640 1 --2- 172.21.15.69:0/2361462041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a980d0300 0x7f9a9806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.196+0000 7f9a97e00640 1 -- 172.21.15.69:0/2361462041 >> v1:172.21.15.69:6789/0 conn(0x7f9a98145a90 legacy=0x7f9a980d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.196+0000 7f9a97e00640 1 -- 172.21.15.69:0/2361462041 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a980cfc70 con 0x7f9a98145fe0 2024-04-24T08:08:25.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.196+0000 7f9a97400640 1 --2- 172.21.15.69:0/2361462041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a980d0300 0x7f9a9806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:25.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a97e00640 1 --2- 172.21.15.69:0/2361462041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a98145fe0 0x7f9a98160540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009d90 tx=0x7f9a9002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c17eb903c7a36b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:25.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a96000640 1 -- 172.21.15.69:0/2361462041 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90007740 con 0x7f9a98145fe0 2024-04-24T08:08:25.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a96000640 1 -- 172.21.15.69:0/2361462041 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a900078e0 con 0x7f9a98145fe0 2024-04-24T08:08:25.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a96000640 1 -- 172.21.15.69:0/2361462041 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90007cb0 con 0x7f9a98145fe0 2024-04-24T08:08:25.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a9e800640 1 -- 172.21.15.69:0/2361462041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a98145fe0 msgr2=0x7f9a98160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a9e800640 1 --2- 172.21.15.69:0/2361462041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a98145fe0 0x7f9a98160540 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9a90009d90 tx=0x7f9a9002a660 comp rx=0 tx=0).stop 2024-04-24T08:08:25.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a9e800640 1 -- 172.21.15.69:0/2361462041 shutdown_connections 2024-04-24T08:08:25.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a9e800640 1 --2- 172.21.15.69:0/2361462041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a98145fe0 0x7f9a98160540 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a9e800640 1 --2- 172.21.15.69:0/2361462041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a980d0300 0x7f9a9806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a9e800640 1 -- 172.21.15.69:0/2361462041 >> 172.21.15.69:0/2361462041 conn(0x7f9a98067d90 msgr2=0x7f9a9806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:25.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.197+0000 7f9a9e800640 1 -- 172.21.15.69:0/2361462041 shutdown_connections 2024-04-24T08:08:25.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.198+0000 7f9a9e800640 1 -- 172.21.15.69:0/2361462041 wait complete. 2024-04-24T08:08:25.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.199+0000 7f9a9e800640 1 Processor -- start 2024-04-24T08:08:25.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.199+0000 7f9a9e800640 1 -- start start 2024-04-24T08:08:25.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.200+0000 7f9a9e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a980d0300 0x7f9a981f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.200+0000 7f9a9e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a98145a90 0x7f9a980af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.200+0000 7f9a9e800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a98145fe0 0x7f9a980cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.200+0000 7f9a9e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9a981458f0 con 0x7f9a980d0300 2024-04-24T08:08:25.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.200+0000 7f9a9e800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9a98145770 con 0x7f9a98145fe0 2024-04-24T08:08:25.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.200+0000 7f9a9e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a980cdc30 con 0x7f9a98145a90 2024-04-24T08:08:25.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.200+0000 7f9a97400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a980d0300 0x7f9a981f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.200+0000 7f9a97400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a980d0300 0x7f9a981f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41246/0 (socket says 172.21.15.69:41246) 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.200+0000 7f9a97400640 1 -- 172.21.15.69:0/2258956329 learned_addr learned my addr 172.21.15.69:0/2258956329 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a97400640 1 -- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a98145a90 msgr2=0x7f9a980af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a97400640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a98145a90 0x7f9a980af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a97400640 1 -- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a98145fe0 msgr2=0x7f9a980cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a97e00640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a98145fe0 0x7f9a980cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a97400640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a98145fe0 0x7f9a980cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a97400640 1 -- 172.21.15.69:0/2258956329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a980ad6f0 con 0x7f9a980d0300 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a97400640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a980d0300 0x7f9a981f2c90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800d7d0 tx=0x7f9a8800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a8f400640 1 -- 172.21.15.69:0/2258956329 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a88004030 con 0x7f9a980d0300 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a8f400640 1 -- 172.21.15.69:0/2258956329 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a880041d0 con 0x7f9a980d0300 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a9e800640 1 -- 172.21.15.69:0/2258956329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a980ad9d0 con 0x7f9a980d0300 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.201+0000 7f9a8f400640 1 -- 172.21.15.69:0/2258956329 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a880096a0 con 0x7f9a980d0300 2024-04-24T08:08:25.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.202+0000 7f9a9e800640 1 -- 172.21.15.69:0/2258956329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a980ae270 con 0x7f9a980d0300 2024-04-24T08:08:25.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.203+0000 7f9a9e800640 1 -- 172.21.15.69:0/2258956329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a98068a80 con 0x7f9a980d0300 2024-04-24T08:08:25.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.203+0000 7f9a8f400640 1 -- 172.21.15.69:0/2258956329 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a88009840 con 0x7f9a980d0300 2024-04-24T08:08:25.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.205+0000 7f9a8f400640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a7806c8f0 0x7f9a7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.205+0000 7f9a8f400640 1 -- 172.21.15.69:0/2258956329 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9a8808bef0 con 0x7f9a980d0300 2024-04-24T08:08:25.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.205+0000 7f9a96a00640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a7806c8f0 0x7f9a7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.206+0000 7f9a96a00640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a7806c8f0 0x7f9a7806edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000f60 tx=0x7f9a840078f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:25.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.209+0000 7f9a8f400640 1 -- 172.21.15.69:0/2258956329 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a8805a280 con 0x7f9a980d0300 2024-04-24T08:08:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.459+0000 7f9a9e800640 1 -- 172.21.15.69:0/2258956329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f9a980d1120 con 0x7f9a980d0300 2024-04-24T08:08:25.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.463+0000 7f9a8f400640 1 -- 172.21.15.69:0/2258956329 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174141 (secure 0 0 0) 0x7f9a88059b80 con 0x7f9a980d0300 2024-04-24T08:08:25.467 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:25.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.466+0000 7f9a9e800640 1 -- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a7806c8f0 msgr2=0x7f9a7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.466+0000 7f9a9e800640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a7806c8f0 0x7f9a7806edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000f60 tx=0x7f9a840078f0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.466+0000 7f9a9e800640 1 -- 172.21.15.69:0/2258956329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a980d0300 msgr2=0x7f9a981f2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.466+0000 7f9a9e800640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a980d0300 0x7f9a981f2c90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9a8800d7d0 tx=0x7f9a8800dc90 comp rx=0 tx=0).stop 2024-04-24T08:08:25.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.467+0000 7f9a9e800640 1 -- 172.21.15.69:0/2258956329 shutdown_connections 2024-04-24T08:08:25.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.467+0000 7f9a9e800640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a7806c8f0 0x7f9a7806edb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.467+0000 7f9a9e800640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a98145fe0 0x7f9a980cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.467+0000 7f9a9e800640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a98145a90 0x7f9a980af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.467+0000 7f9a9e800640 1 --2- 172.21.15.69:0/2258956329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a980d0300 0x7f9a981f2c90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.467+0000 7f9a9e800640 1 -- 172.21.15.69:0/2258956329 >> 172.21.15.69:0/2258956329 conn(0x7f9a98067d90 msgr2=0x7f9a9806f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:25.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.467+0000 7f9a9e800640 1 -- 172.21.15.69:0/2258956329 shutdown_connections 2024-04-24T08:08:25.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.467+0000 7f9a9e800640 1 -- 172.21.15.69:0/2258956329 wait complete. 2024-04-24T08:08:25.487 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":5,"flags":0,"active_gid":4109,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2070338101},{"type":"v1","addr":"172.21.15.121:6833","nonce":2070338101}]},"active_addr":"172.21.15.121:6833/2070338101","active_change":"2024-04-24T08:08:00.642198+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":614108161}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":543210474}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":2518949136}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.121:0","nonce":332811835}]}]} 2024-04-24T08:08:25.490 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-24T08:08:25.490 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T08:08:25.490 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:25.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.688+0000 7f884ae00640 1 Processor -- start 2024-04-24T08:08:25.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.688+0000 7f884ae00640 1 -- start start 2024-04-24T08:08:25.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.690+0000 7f884ae00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f88440d0300 0x7f884406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.690+0000 7f884ae00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8844145fe0 0x7f8844160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.690+0000 7f884ae00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88440cf3d0 con 0x7f8844145a90 2024-04-24T08:08:25.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.690+0000 7f884ae00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f88440cf6b0 con 0x7f88440d0300 2024-04-24T08:08:25.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.690+0000 7f884ae00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f88440cf990 con 0x7f8844145fe0 2024-04-24T08:08:25.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.690+0000 7f8843e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f88440d0300 0x7f884406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.690+0000 7f8843e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f88440d0300 0x7f884406e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41260/0 (socket says 172.21.15.69:41260) 2024-04-24T08:08:25.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.690+0000 7f8843e00640 1 -- 172.21.15.69:0/250958070 learned_addr learned my addr 172.21.15.69:0/250958070 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:25.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.690+0000 7f8848e00640 1 --2- 172.21.15.69:0/250958070 >> v2:172.21.15.121:3300/0 conn(0x7f8844145fe0 0x7f8844160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.691+0000 7f8843e00640 1 -- 172.21.15.69:0/250958070 >> v2:172.21.15.121:3300/0 conn(0x7f8844145fe0 msgr2=0x7f8844160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:25.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.691+0000 7f8843e00640 1 --2- 172.21.15.69:0/250958070 >> v2:172.21.15.121:3300/0 conn(0x7f8844145fe0 0x7f8844160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.691+0000 7f8843e00640 1 -- 172.21.15.69:0/250958070 >> v1:172.21.15.121:6789/0 conn(0x7f8844145a90 legacy=0x7f88440d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:25.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.691+0000 7f8843e00640 1 -- 172.21.15.69:0/250958070 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88440cfc70 con 0x7f88440d0300 2024-04-24T08:08:25.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.692+0000 7f8843e00640 1 --2- 172.21.15.69:0/250958070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88440d0300 0x7f884406e240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f88440d1050 tx=0x7f883402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfd2fa53a93e3f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:25.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.692+0000 7f8842a00640 1 -- 172.21.15.69:0/250958070 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8834037070 con 0x7f88440d0300 2024-04-24T08:08:25.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.692+0000 7f8842a00640 1 -- 172.21.15.69:0/250958070 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8834032440 con 0x7f88440d0300 2024-04-24T08:08:25.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.693+0000 7f8842a00640 1 -- 172.21.15.69:0/250958070 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8834032810 con 0x7f88440d0300 2024-04-24T08:08:25.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.693+0000 7f884ae00640 1 -- 172.21.15.69:0/250958070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88440d0300 msgr2=0x7f884406e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.693+0000 7f884ae00640 1 --2- 172.21.15.69:0/250958070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88440d0300 0x7f884406e240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f88440d1050 tx=0x7f883402a230 comp rx=0 tx=0).stop 2024-04-24T08:08:25.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.693+0000 7f884ae00640 1 -- 172.21.15.69:0/250958070 shutdown_connections 2024-04-24T08:08:25.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.693+0000 7f884ae00640 1 --2- 172.21.15.69:0/250958070 >> v2:172.21.15.121:3300/0 conn(0x7f8844145fe0 0x7f8844160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.693+0000 7f884ae00640 1 --2- 172.21.15.69:0/250958070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88440d0300 0x7f884406e240 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.693+0000 7f884ae00640 1 -- 172.21.15.69:0/250958070 >> 172.21.15.69:0/250958070 conn(0x7f8844067d90 msgr2=0x7f884406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:25.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.693+0000 7f884ae00640 1 -- 172.21.15.69:0/250958070 shutdown_connections 2024-04-24T08:08:25.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.694+0000 7f884ae00640 1 -- 172.21.15.69:0/250958070 wait complete. 2024-04-24T08:08:25.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.695+0000 7f884ae00640 1 Processor -- start 2024-04-24T08:08:25.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.695+0000 7f884ae00640 1 -- start start 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.696+0000 7f884ae00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88440d0300 0x7f8844143960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.696+0000 7f884ae00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8844145a90 0x7f884415f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.696+0000 7f884ae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8844145fe0 0x7f884415e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.696+0000 7f884ae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f88441458f0 con 0x7f8844145fe0 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.696+0000 7f884ae00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8844145770 con 0x7f8844145a90 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.696+0000 7f884ae00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f88440cdc30 con 0x7f88440d0300 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.696+0000 7f8848e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8844145fe0 0x7f884415e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8848e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8844145fe0 0x7f884415e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41274/0 (socket says 172.21.15.69:41274) 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8848e00640 1 -- 172.21.15.69:0/714075703 learned_addr learned my addr 172.21.15.69:0/714075703 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8843400640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8844145a90 0x7f884415f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8843e00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88440d0300 0x7f8844143960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8848e00640 1 -- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88440d0300 msgr2=0x7f8844143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8848e00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88440d0300 0x7f8844143960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8848e00640 1 -- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8844145a90 msgr2=0x7f884415f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8848e00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8844145a90 0x7f884415f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8848e00640 1 -- 172.21.15.69:0/714075703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88441e65b0 con 0x7f8844145fe0 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8843e00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88440d0300 0x7f8844143960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f8848e00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8844145fe0 0x7f884415e4a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f883c007e70 tx=0x7f883c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.697+0000 7f883be00640 1 -- 172.21.15.69:0/714075703 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c014620 con 0x7f8844145fe0 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.698+0000 7f883be00640 1 -- 172.21.15.69:0/714075703 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f883c019070 con 0x7f8844145fe0 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.698+0000 7f883be00640 1 -- 172.21.15.69:0/714075703 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c014a20 con 0x7f8844145fe0 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.698+0000 7f884ae00640 1 -- 172.21.15.69:0/714075703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88441e6890 con 0x7f8844145fe0 2024-04-24T08:08:25.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.698+0000 7f884ae00640 1 -- 172.21.15.69:0/714075703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88441e6e50 con 0x7f8844145fe0 2024-04-24T08:08:25.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.699+0000 7f884ae00640 1 -- 172.21.15.69:0/714075703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8844068a80 con 0x7f8844145fe0 2024-04-24T08:08:25.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.700+0000 7f883be00640 1 -- 172.21.15.69:0/714075703 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f883c014bc0 con 0x7f8844145fe0 2024-04-24T08:08:25.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.701+0000 7f883be00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882006c8f0 0x7f882006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:25.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.701+0000 7f883be00640 1 -- 172.21.15.69:0/714075703 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f883c08c4a0 con 0x7f8844145fe0 2024-04-24T08:08:25.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.701+0000 7f8843e00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882006c8f0 0x7f882006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:25.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.702+0000 7f8843e00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882006c8f0 0x7f882006edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8834034000 tx=0x7f883402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:25.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.705+0000 7f883be00640 1 -- 172.21.15.69:0/714075703 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f883c011260 con 0x7f8844145fe0 2024-04-24T08:08:25.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.905+0000 7f884ae00640 1 -- 172.21.15.69:0/714075703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f88440ce980 con 0x7f8844145fe0 2024-04-24T08:08:25.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.906+0000 7f883be00640 1 -- 172.21.15.69:0/714075703 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7f883c05a8d0 con 0x7f8844145fe0 2024-04-24T08:08:25.908 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:25.908 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 -- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882006c8f0 msgr2=0x7f882006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882006c8f0 0x7f882006edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8834034000 tx=0x7f883402e000 comp rx=0 tx=0).stop 2024-04-24T08:08:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 -- 172.21.15.69:0/714075703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8844145fe0 msgr2=0x7f884415e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8844145fe0 0x7f884415e4a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f883c007e70 tx=0x7f883c00e4b0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 -- 172.21.15.69:0/714075703 shutdown_connections 2024-04-24T08:08:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882006c8f0 0x7f882006edb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8844145fe0 0x7f884415e4a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8844145a90 0x7f884415f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 --2- 172.21.15.69:0/714075703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88440d0300 0x7f8844143960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 -- 172.21.15.69:0/714075703 >> 172.21.15.69:0/714075703 conn(0x7f8844067d90 msgr2=0x7f88440b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:25.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 -- 172.21.15.69:0/714075703 shutdown_connections 2024-04-24T08:08:25.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:25.909+0000 7f884ae00640 1 -- 172.21.15.69:0/714075703 wait complete. 2024-04-24T08:08:25.929 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T08:08:25.929 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:26.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.127+0000 7f3e2f200640 1 Processor -- start 2024-04-24T08:08:26.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.127+0000 7f3e2f200640 1 -- start start 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e2f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3e28145a90 0x7f3e2806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e2f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3e28145fe0 0x7f3e28160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e2f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e280cf3d0 con 0x7f3e280d0300 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e2f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3e280cf6b0 con 0x7f3e28145fe0 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e2f200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3e280cf990 con 0x7f3e28145a90 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e2d200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3e28145fe0 0x7f3e28160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e27e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3e280d0300 0x7f3e280d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44676/0 (socket says 172.21.15.69:44676) 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e27e00640 1 -- 172.21.15.69:0/737500786 learned_addr learned my addr 172.21.15.69:0/737500786 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e1fe00640 1 --2- 172.21.15.69:0/737500786 >> v2:172.21.15.121:3300/0 conn(0x7f3e28145a90 0x7f3e2806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e2d200640 1 -- 172.21.15.69:0/737500786 >> v2:172.21.15.121:3300/0 conn(0x7f3e28145a90 msgr2=0x7f3e2806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e2d200640 1 --2- 172.21.15.69:0/737500786 >> v2:172.21.15.121:3300/0 conn(0x7f3e28145a90 0x7f3e2806e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e2d200640 1 -- 172.21.15.69:0/737500786 >> v1:172.21.15.69:6789/0 conn(0x7f3e280d0300 legacy=0x7f3e280d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e2d200640 1 -- 172.21.15.69:0/737500786 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e280cfc70 con 0x7f3e28145fe0 2024-04-24T08:08:26.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.129+0000 7f3e27400640 1 -- 172.21.15.69:0/737500786 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135016997 0 0) 0x7f3e280cf3d0 con 0x7f3e280d0300 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.130+0000 7f3e2d200640 1 --2- 172.21.15.69:0/737500786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e28145fe0 0x7f3e28160540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e280b5780 tx=0x7f3e2002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545ab154caf71eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e27400640 1 -- 172.21.15.69:0/737500786 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20007740 con 0x7f3e28145fe0 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e27400640 1 -- 172.21.15.69:0/737500786 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e200078e0 con 0x7f3e28145fe0 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e27400640 1 -- 172.21.15.69:0/737500786 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e20007cb0 con 0x7f3e28145fe0 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e2f200640 1 -- 172.21.15.69:0/737500786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e28145fe0 msgr2=0x7f3e28160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e2f200640 1 --2- 172.21.15.69:0/737500786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e28145fe0 0x7f3e28160540 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e280b5780 tx=0x7f3e2002a680 comp rx=0 tx=0).stop 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e2f200640 1 -- 172.21.15.69:0/737500786 shutdown_connections 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e2f200640 1 --2- 172.21.15.69:0/737500786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e28145fe0 0x7f3e28160540 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e2f200640 1 --2- 172.21.15.69:0/737500786 >> v2:172.21.15.121:3300/0 conn(0x7f3e28145a90 0x7f3e2806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e2f200640 1 -- 172.21.15.69:0/737500786 >> 172.21.15.69:0/737500786 conn(0x7f3e28067d90 msgr2=0x7f3e2806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e2f200640 1 -- 172.21.15.69:0/737500786 shutdown_connections 2024-04-24T08:08:26.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.131+0000 7f3e2f200640 1 -- 172.21.15.69:0/737500786 wait complete. 2024-04-24T08:08:26.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.133+0000 7f3e2f200640 1 Processor -- start 2024-04-24T08:08:26.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.133+0000 7f3e2f200640 1 -- start start 2024-04-24T08:08:26.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.134+0000 7f3e2f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e280d0300 0x7f3e2813b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.134+0000 7f3e2f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e28145a90 0x7f3e2815b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.134+0000 7f3e2f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e28145fe0 0x7f3e280b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.134+0000 7f3e2f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3e281458f0 con 0x7f3e280d0300 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.134+0000 7f3e2f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3e28145770 con 0x7f3e28145fe0 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.134+0000 7f3e2f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3e280cdc30 con 0x7f3e28145a90 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.134+0000 7f3e27e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e280d0300 0x7f3e2813b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.134+0000 7f3e27e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e280d0300 0x7f3e2813b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41298/0 (socket says 172.21.15.69:41298) 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.134+0000 7f3e27e00640 1 -- 172.21.15.69:0/3153723685 learned_addr learned my addr 172.21.15.69:0/3153723685 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e2d200640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e28145fe0 0x7f3e280b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e27e00640 1 -- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e28145a90 msgr2=0x7f3e2815b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e27e00640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e28145a90 0x7f3e2815b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e27e00640 1 -- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e28145fe0 msgr2=0x7f3e280b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e27e00640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e28145fe0 0x7f3e280b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e27e00640 1 -- 172.21.15.69:0/3153723685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e280ad9e0 con 0x7f3e280d0300 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e2d200640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e28145fe0 0x7f3e280b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:26.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e27e00640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e280d0300 0x7f3e2813b820 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800c430 tx=0x7f3e1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e24c00640 1 -- 172.21.15.69:0/3153723685 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e180074e0 con 0x7f3e280d0300 2024-04-24T08:08:26.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e24c00640 1 -- 172.21.15.69:0/3153723685 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e180047e0 con 0x7f3e280d0300 2024-04-24T08:08:26.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.135+0000 7f3e2f200640 1 -- 172.21.15.69:0/3153723685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e280adcc0 con 0x7f3e280d0300 2024-04-24T08:08:26.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.136+0000 7f3e24c00640 1 -- 172.21.15.69:0/3153723685 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e18004bd0 con 0x7f3e280d0300 2024-04-24T08:08:26.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.136+0000 7f3e2f200640 1 -- 172.21.15.69:0/3153723685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e280ae280 con 0x7f3e280d0300 2024-04-24T08:08:26.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.137+0000 7f3e2f200640 1 -- 172.21.15.69:0/3153723685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e28068a80 con 0x7f3e280d0300 2024-04-24T08:08:26.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.138+0000 7f3e24c00640 1 -- 172.21.15.69:0/3153723685 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e18011070 con 0x7f3e280d0300 2024-04-24T08:08:26.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.139+0000 7f3e24c00640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3e0406c8f0 0x7f3e0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.139+0000 7f3e24c00640 1 -- 172.21.15.69:0/3153723685 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3e1808b820 con 0x7f3e280d0300 2024-04-24T08:08:26.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.139+0000 7f3e1fe00640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3e0406c8f0 0x7f3e0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.140+0000 7f3e1fe00640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3e0406c8f0 0x7f3e0406edb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3e2815c7b0 tx=0x7f3e10008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.143+0000 7f3e24c00640 1 -- 172.21.15.69:0/3153723685 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e1800e280 con 0x7f3e280d0300 2024-04-24T08:08:26.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.344+0000 7f3e2f200640 1 -- 172.21.15.69:0/3153723685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e2806bc70 con 0x7f3e280d0300 2024-04-24T08:08:26.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.347+0000 7f3e24c00640 1 -- 172.21.15.69:0/3153723685 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7f3e18059bb0 con 0x7f3e280d0300 2024-04-24T08:08:26.349 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:26.349 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 -- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3e0406c8f0 msgr2=0x7f3e0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3e0406c8f0 0x7f3e0406edb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3e2815c7b0 tx=0x7f3e10008040 comp rx=0 tx=0).stop 2024-04-24T08:08:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 -- 172.21.15.69:0/3153723685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e280d0300 msgr2=0x7f3e2813b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e280d0300 0x7f3e2813b820 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800c430 tx=0x7f3e1800c8f0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 -- 172.21.15.69:0/3153723685 shutdown_connections 2024-04-24T08:08:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3e0406c8f0 0x7f3e0406edb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3e28145fe0 0x7f3e280b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3e28145a90 0x7f3e2815b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 --2- 172.21.15.69:0/3153723685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e280d0300 0x7f3e2813b820 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 -- 172.21.15.69:0/3153723685 >> 172.21.15.69:0/3153723685 conn(0x7f3e28067d90 msgr2=0x7f3e2806e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 -- 172.21.15.69:0/3153723685 shutdown_connections 2024-04-24T08:08:26.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.350+0000 7f3e2f200640 1 -- 172.21.15.69:0/3153723685 wait complete. 2024-04-24T08:08:26.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T08:08:26.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T08:08:26.372 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T08:08:26.372 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T08:08:26.372 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-24T08:08:26.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-24T08:08:26.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-24T08:08:26.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-24T08:08:26.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.652+0000 7f02bfe00640 1 Processor -- start 2024-04-24T08:08:26.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.653+0000 7f02bfe00640 1 -- start start 2024-04-24T08:08:26.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.655+0000 7f02bfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80e3160 0x7f02b80e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.655+0000 7f02bfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f02b8053cb0 0x7f02b8054120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.655+0000 7f02bfe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b8054660 con 0x7f02b803c0a0 2024-04-24T08:08:26.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.655+0000 7f02bfe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f02b8007b70 con 0x7f02b8053cb0 2024-04-24T08:08:26.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.655+0000 7f02bfe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f02b8058810 con 0x7f02b80e3160 2024-04-24T08:08:26.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.656+0000 7f02bf400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f02b8053cb0 0x7f02b8054120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.656+0000 7f02bf400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f02b8053cb0 0x7f02b8054120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41306/0 (socket says 172.21.15.69:41306) 2024-04-24T08:08:26.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.656+0000 7f02bf400640 1 -- 172.21.15.69:0/482062682 learned_addr learned my addr 172.21.15.69:0/482062682 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.656+0000 7f02bea00640 1 --2- 172.21.15.69:0/482062682 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80e3160 0x7f02b80e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.656+0000 7f02bf400640 1 -- 172.21.15.69:0/482062682 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80e3160 msgr2=0x7f02b80e7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:26.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.656+0000 7f02bf400640 1 --2- 172.21.15.69:0/482062682 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80e3160 0x7f02b80e7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.656+0000 7f02bf400640 1 -- 172.21.15.69:0/482062682 >> v1:172.21.15.121:6789/0 conn(0x7f02b803c0a0 legacy=0x7f02b8058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:26.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.656+0000 7f02bf400640 1 -- 172.21.15.69:0/482062682 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b8058af0 con 0x7f02b8053cb0 2024-04-24T08:08:26.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.658+0000 7f02bf400640 1 --2- 172.21.15.69:0/482062682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b8053cb0 0x7f02b8054120 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f02c0069ac0 tx=0x7f02c00a5d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b818b979e0cc60 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.658+0000 7f02bd600640 1 -- 172.21.15.69:0/482062682 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c00b3070 con 0x7f02b8053cb0 2024-04-24T08:08:26.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.658+0000 7f02bd600640 1 -- 172.21.15.69:0/482062682 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c00a6af0 con 0x7f02b8053cb0 2024-04-24T08:08:26.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.658+0000 7f02bd600640 1 -- 172.21.15.69:0/482062682 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c00ae440 con 0x7f02b8053cb0 2024-04-24T08:08:26.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.660+0000 7f02bfe00640 1 -- 172.21.15.69:0/482062682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b8053cb0 msgr2=0x7f02b8054120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.660+0000 7f02bfe00640 1 --2- 172.21.15.69:0/482062682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b8053cb0 0x7f02b8054120 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f02c0069ac0 tx=0x7f02c00a5d20 comp rx=0 tx=0).stop 2024-04-24T08:08:26.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.660+0000 7f02bfe00640 1 -- 172.21.15.69:0/482062682 shutdown_connections 2024-04-24T08:08:26.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.660+0000 7f02bfe00640 1 --2- 172.21.15.69:0/482062682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b8053cb0 0x7f02b8054120 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.661+0000 7f7e71a00640 1 Processor -- start 2024-04-24T08:08:26.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.661+0000 7f02bfe00640 1 --2- 172.21.15.69:0/482062682 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b80e3160 0x7f02b80e7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.661+0000 7f02bfe00640 1 -- 172.21.15.69:0/482062682 >> 172.21.15.69:0/482062682 conn(0x7f02b8058ee0 msgr2=0x7f02b80592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.662+0000 7f02bfe00640 1 -- 172.21.15.69:0/482062682 shutdown_connections 2024-04-24T08:08:26.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.662+0000 7f02bfe00640 1 -- 172.21.15.69:0/482062682 wait complete. 2024-04-24T08:08:26.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.661+0000 7f7e71a00640 1 -- start start 2024-04-24T08:08:26.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.663+0000 7f7e71a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7e6c0d10f0 0x7f7e6c0d1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.663+0000 7f7e71a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e6c0d22d0 con 0x7f7e6c162150 2024-04-24T08:08:26.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.663+0000 7f7e71a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e6c0d03b0 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.663+0000 7f7e71a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7e6c0d1a80 con 0x7f7e6c0d10f0 2024-04-24T08:08:26.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.664+0000 7f7e6b400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7e6c0d10f0 0x7f7e6c0d1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.664+0000 7f7e6b400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7e6c0d10f0 0x7f7e6c0d1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:42348/0 (socket says 172.21.15.69:42348) 2024-04-24T08:08:26.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.664+0000 7f7e6a000640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7e6c0cfe40 0x7f7e6c0b7550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:38366/0 (socket says 172.21.15.69:38366) 2024-04-24T08:08:26.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.664+0000 7f7e6a000640 1 -- 172.21.15.69:0/377338868 learned_addr learned my addr 172.21.15.69:0/377338868 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:26.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.665+0000 7f7e69600640 1 -- 172.21.15.69:0/377338868 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1462544240 0 0) 0x7f7e6c0d03b0 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.666+0000 7f7e69600640 1 -- 172.21.15.69:0/377338868 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e50003520 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.666+0000 7f7e6b400640 1 -- 172.21.15.69:0/377338868 >> v1:172.21.15.121:6789/0 conn(0x7f7e6c0cfe40 legacy=0x7f7e6c0b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.666+0000 7f02bfe00640 1 Processor -- start 2024-04-24T08:08:26.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.666+0000 7f7e6b400640 1 -- 172.21.15.69:0/377338868 >> v1:172.21.15.69:6789/0 conn(0x7f7e6c162150 legacy=0x7f7e6c0d5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.666+0000 7f02bfe00640 1 -- start start 2024-04-24T08:08:26.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.666+0000 7f7e6b400640 1 -- 172.21.15.69:0/377338868 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c0d0690 con 0x7f7e6c0d10f0 2024-04-24T08:08:26.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.667+0000 7f3046a00640 1 Processor -- start 2024-04-24T08:08:26.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.667+0000 7f3046a00640 1 -- start start 2024-04-24T08:08:26.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.668+0000 7f40a9200640 1 Processor -- start 2024-04-24T08:08:26.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.668+0000 7f40a9200640 1 -- start start 2024-04-24T08:08:26.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.668+0000 7f02bfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b803c0a0 0x7f02b80d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.668+0000 7f7e6b400640 1 --2- 172.21.15.69:0/377338868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c0d10f0 0x7f7e6c0d1540 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e580097a0 tx=0x7f7e5802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2fa1adfb80f2987 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f02bfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b8053cb0 0x7f02b8050950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f02bfe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02b80e3160 0x7f02b80d4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f02bfe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f02b80e3540 con 0x7f02b803c0a0 2024-04-24T08:08:26.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f02bfe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f02b8006640 con 0x7f02b80e3160 2024-04-24T08:08:26.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f02bfe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f02b8006b60 con 0x7f02b8053cb0 2024-04-24T08:08:26.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f3046a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3040162c70 0x7f30400b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f3046a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f304006f100 0x7f30400b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f3046a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30401669d0 con 0x7f30400b3d30 2024-04-24T08:08:26.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f3046a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3040166cb0 con 0x7f304006f100 2024-04-24T08:08:26.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f3046a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3040166f90 con 0x7f3040162c70 2024-04-24T08:08:26.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f02bea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b803c0a0 0x7f02b80d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f02bea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b803c0a0 0x7f02b80d0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41322/0 (socket says 172.21.15.69:41322) 2024-04-24T08:08:26.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f02bea00640 1 -- 172.21.15.69:0/607966712 learned_addr learned my addr 172.21.15.69:0/607966712 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f7e69600640 1 -- 172.21.15.69:0/377338868 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58037070 con 0x7f7e6c0d10f0 2024-04-24T08:08:26.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f7e69600640 1 -- 172.21.15.69:0/377338868 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e58032440 con 0x7f7e6c0d10f0 2024-04-24T08:08:26.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f7e69600640 1 -- 172.21.15.69:0/377338868 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58032850 con 0x7f7e6c0d10f0 2024-04-24T08:08:26.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f7e71a00640 1 -- 172.21.15.69:0/377338868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c0d10f0 msgr2=0x7f7e6c0d1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f7e71a00640 1 --2- 172.21.15.69:0/377338868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c0d10f0 0x7f7e6c0d1540 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7e580097a0 tx=0x7f7e5802a250 comp rx=0 tx=0).stop 2024-04-24T08:08:26.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f3044a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f304006f100 0x7f30400b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f3044a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f304006f100 0x7f30400b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41326/0 (socket says 172.21.15.69:41326) 2024-04-24T08:08:26.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f7e71a00640 1 -- 172.21.15.69:0/377338868 shutdown_connections 2024-04-24T08:08:26.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f7e71a00640 1 --2- 172.21.15.69:0/377338868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c0d10f0 0x7f7e6c0d1540 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f7e71a00640 1 -- 172.21.15.69:0/377338868 >> 172.21.15.69:0/377338868 conn(0x7f7e6c08bf60 msgr2=0x7f7e6c0d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f7e71a00640 1 -- 172.21.15.69:0/377338868 shutdown_connections 2024-04-24T08:08:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f02bf400640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02b80e3160 0x7f02b80d4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.669+0000 7f02be000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b8053cb0 0x7f02b8050950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f02bea00640 1 -- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b8053cb0 msgr2=0x7f02b8050950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f02bea00640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b8053cb0 0x7f02b8050950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f02bea00640 1 -- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02b80e3160 msgr2=0x7f02b80d4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f02bea00640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02b80e3160 0x7f02b80d4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f02bea00640 1 -- 172.21.15.69:0/607966712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b80ceb90 con 0x7f02b803c0a0 2024-04-24T08:08:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f02be000640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02b8053cb0 0x7f02b8050950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f3044a00640 1 -- 172.21.15.69:0/2764582224 learned_addr learned my addr 172.21.15.69:0/2764582224 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f303fe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3040162c70 0x7f30400b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f40a9200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4162c00 0x7f40a4162fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f40a9200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f40a40b30f0 0x7f40a4072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f40a9200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a41638e0 con 0x7f40a4073510 2024-04-24T08:08:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f40a9200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f40a4163bc0 con 0x7f40a40b30f0 2024-04-24T08:08:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f40a9200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f40a4163ea0 con 0x7f40a4162c00 2024-04-24T08:08:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f40a3e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f40a4073510 0x7f40a4166880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44696/0 (socket says 172.21.15.69:44696) 2024-04-24T08:08:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f40a3e00640 1 -- 172.21.15.69:0/1840975186 learned_addr learned my addr 172.21.15.69:0/1840975186 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.672+0000 7f40a2a00640 1 --2- 172.21.15.69:0/1840975186 >> v2:172.21.15.121:3300/0 conn(0x7f40a40b30f0 0x7f40a4072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.672+0000 7f5605200640 1 Processor -- start 2024-04-24T08:08:26.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f3044a00640 1 -- 172.21.15.69:0/2764582224 >> v2:172.21.15.121:3300/0 conn(0x7f3040162c70 msgr2=0x7f30400b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:26.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.672+0000 7f40a3400640 1 --2- 172.21.15.69:0/1840975186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4162c00 0x7f40a4162fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f3044a00640 1 --2- 172.21.15.69:0/2764582224 >> v2:172.21.15.121:3300/0 conn(0x7f3040162c70 0x7f30400b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f3044a00640 1 -- 172.21.15.69:0/2764582224 >> v1:172.21.15.121:6789/0 conn(0x7f30400b3d30 legacy=0x7f3040071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.671+0000 7f3044a00640 1 -- 172.21.15.69:0/2764582224 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3040167270 con 0x7f304006f100 2024-04-24T08:08:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.672+0000 7f5605200640 1 -- start start 2024-04-24T08:08:26.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.672+0000 7f02bea00640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b803c0a0 0x7f02b80d0db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02b000d5f0 tx=0x7f02b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.673+0000 7f02b6a00640 1 -- 172.21.15.69:0/607966712 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b00125f0 con 0x7f02b803c0a0 2024-04-24T08:08:26.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.673+0000 7f02b6a00640 1 -- 172.21.15.69:0/607966712 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02b0019070 con 0x7f02b803c0a0 2024-04-24T08:08:26.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.670+0000 7f7e71a00640 1 -- 172.21.15.69:0/377338868 wait complete. 2024-04-24T08:08:26.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.673+0000 7f7e71a00640 1 Processor -- start 2024-04-24T08:08:26.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.673+0000 7f7e71a00640 1 -- start start 2024-04-24T08:08:26.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.673+0000 7f02b6a00640 1 -- 172.21.15.69:0/607966712 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b00125f0 con 0x7f02b803c0a0 2024-04-24T08:08:26.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.672+0000 7f40a2a00640 1 -- 172.21.15.69:0/1840975186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4162c00 msgr2=0x7f40a4162fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:26.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.672+0000 7f40a2a00640 1 --2- 172.21.15.69:0/1840975186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4162c00 0x7f40a4162fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.672+0000 7f40a2a00640 1 -- 172.21.15.69:0/1840975186 >> v1:172.21.15.69:6789/0 conn(0x7f40a4073510 legacy=0x7f40a4166880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.672+0000 7f40a2a00640 1 -- 172.21.15.69:0/1840975186 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a4164180 con 0x7f40a40b30f0 2024-04-24T08:08:26.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.673+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b8051440 con 0x7f02b803c0a0 2024-04-24T08:08:26.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.673+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b8051a00 con 0x7f02b803c0a0 2024-04-24T08:08:26.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.674+0000 7f5605200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f56000b41c0 0x7f560006e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.674+0000 7f5605200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56001669d0 con 0x7f56000b3d30 2024-04-24T08:08:26.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.674+0000 7f5605200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5600166cb0 con 0x7f5600162c70 2024-04-24T08:08:26.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.674+0000 7f5605200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5600166f90 con 0x7f56000b41c0 2024-04-24T08:08:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.674+0000 7f3044a00640 1 --2- 172.21.15.69:0/2764582224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f304006f100 0x7f30400b41c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3040070120 tx=0x7f303802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c303b57d45b183 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.675+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f02b80cef20 con 0x7f02b803c0a0 2024-04-24T08:08:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.676+0000 7f40a2a00640 1 --2- 172.21.15.69:0/1840975186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a40b30f0 0x7f40a4072fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f40940097a0 tx=0x7f409402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7886330583146f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.675+0000 7f55fde00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5600162c70 0x7f56000b3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:38382/0 (socket says 172.21.15.69:38382) 2024-04-24T08:08:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.675+0000 7f55fe800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f56000b41c0 0x7f560006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.676+0000 7f55fde00640 1 -- 172.21.15.69:0/1940278454 learned_addr learned my addr 172.21.15.69:0/1940278454 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.676+0000 7f303ea00640 1 -- 172.21.15.69:0/2764582224 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3038036070 con 0x7f304006f100 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.675+0000 7f7e71a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0cfe40 0x7f7e6c15a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.675+0000 7f7e71a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c0d10f0 0x7f7e6c1f2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.675+0000 7f7e71a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c162150 0x7f7e6c1f2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.675+0000 7f7e71a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c0b7c60 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.675+0000 7f7e71a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c0d2900 con 0x7f7e6c0d10f0 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.675+0000 7f7e71a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6c0d2d90 con 0x7f7e6c162150 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.676+0000 7f7e6aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0cfe40 0x7f7e6c15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.676+0000 7f7e6aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0cfe40 0x7f7e6c15a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41330/0 (socket says 172.21.15.69:41330) 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.676+0000 7f7e6aa00640 1 -- 172.21.15.69:0/2954499185 learned_addr learned my addr 172.21.15.69:0/2954499185 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.676+0000 7f7e6a000640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c0d10f0 0x7f7e6c1f2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.676+0000 7f7e6b400640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c162150 0x7f7e6c1f2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.676+0000 7f303ea00640 1 -- 172.21.15.69:0/2764582224 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3038031600 con 0x7f304006f100 2024-04-24T08:08:26.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.676+0000 7f303ea00640 1 -- 172.21.15.69:0/2764582224 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30380319d0 con 0x7f304006f100 2024-04-24T08:08:26.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f55fca00640 1 -- 172.21.15.69:0/1940278454 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2740941052 0 0) 0x7f56001669d0 con 0x7f56000b3d30 2024-04-24T08:08:26.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f7e6aa00640 1 -- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c162150 msgr2=0x7f7e6c1f2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f7e6aa00640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c162150 0x7f7e6c1f2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f7e6aa00640 1 -- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c0d10f0 msgr2=0x7f7e6c1f2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f7e6aa00640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c0d10f0 0x7f7e6c1f2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f7e6aa00640 1 -- 172.21.15.69:0/2954499185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c1efa60 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f55fe800640 1 -- 172.21.15.69:0/1940278454 >> v1:172.21.15.121:6789/0 conn(0x7f5600162c70 legacy=0x7f56000b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f55fe800640 1 -- 172.21.15.69:0/1940278454 >> v1:172.21.15.69:6789/0 conn(0x7f56000b3d30 legacy=0x7f560006e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f55fe800640 1 -- 172.21.15.69:0/1940278454 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5600167270 con 0x7f56000b41c0 2024-04-24T08:08:26.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7f02b6a00640 1 -- 172.21.15.69:0/607966712 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f02b0012790 con 0x7f02b803c0a0 2024-04-24T08:08:26.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f7e6aa00640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0cfe40 0x7f7e6c15a860 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c730 tx=0x7f7e6400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.677+0000 7f40a2000640 1 -- 172.21.15.69:0/1840975186 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094037070 con 0x7f40a40b30f0 2024-04-24T08:08:26.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7f40a2000640 1 -- 172.21.15.69:0/1840975186 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4094032440 con 0x7f40a40b30f0 2024-04-24T08:08:26.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7f40a2000640 1 -- 172.21.15.69:0/1840975186 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094032810 con 0x7f40a40b30f0 2024-04-24T08:08:26.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7f3046a00640 1 -- 172.21.15.69:0/2764582224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f304006f100 msgr2=0x7f30400b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7f3046a00640 1 --2- 172.21.15.69:0/2764582224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f304006f100 0x7f30400b41c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3040070120 tx=0x7f303802a6f0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7f7e62a00640 1 -- 172.21.15.69:0/2954499185 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64016020 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7fe2a3400640 1 Processor -- start 2024-04-24T08:08:26.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7fe2a3400640 1 -- start start 2024-04-24T08:08:26.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7f3046a00640 1 -- 172.21.15.69:0/2764582224 shutdown_connections 2024-04-24T08:08:26.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7f3046a00640 1 --2- 172.21.15.69:0/2764582224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f304006f100 0x7f30400b41c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7f3046a00640 1 --2- 172.21.15.69:0/2764582224 >> v2:172.21.15.121:3300/0 conn(0x7f3040162c70 0x7f30400b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.678+0000 7f3046a00640 1 -- 172.21.15.69:0/2764582224 >> 172.21.15.69:0/2764582224 conn(0x7f3040067d90 msgr2=0x7f304006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f7e62a00640 1 -- 172.21.15.69:0/2954499185 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e64006d70 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f7e62a00640 1 -- 172.21.15.69:0/2954499185 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64003bb0 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e6c1efd40 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f3046a00640 1 -- 172.21.15.69:0/2764582224 shutdown_connections 2024-04-24T08:08:26.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f40a9200640 1 -- 172.21.15.69:0/1840975186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a40b30f0 msgr2=0x7f40a4072fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f40a9200640 1 --2- 172.21.15.69:0/1840975186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a40b30f0 0x7f40a4072fd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f40940097a0 tx=0x7f409402a180 comp rx=0 tx=0).stop 2024-04-24T08:08:26.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e6c1f0300 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f55fe800640 1 --2- 172.21.15.69:0/1940278454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56000b41c0 0x7f560006e800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009d90 tx=0x7f55f802a610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=202c0105d13ec063 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f02b6a00640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f02a006cb20 0x7f02a006efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f55fca00640 1 -- 172.21.15.69:0/1940278454 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8007740 con 0x7f56000b41c0 2024-04-24T08:08:26.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f55fca00640 1 -- 172.21.15.69:0/1940278454 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f80078e0 con 0x7f56000b41c0 2024-04-24T08:08:26.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f55fca00640 1 -- 172.21.15.69:0/1940278454 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8007cb0 con 0x7f56000b41c0 2024-04-24T08:08:26.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f40a9200640 1 -- 172.21.15.69:0/1840975186 shutdown_connections 2024-04-24T08:08:26.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f3046a00640 1 -- 172.21.15.69:0/2764582224 wait complete. 2024-04-24T08:08:26.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f40a9200640 1 --2- 172.21.15.69:0/1840975186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a40b30f0 0x7f40a4072fd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f40a9200640 1 --2- 172.21.15.69:0/1840975186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4162c00 0x7f40a4162fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f40a9200640 1 -- 172.21.15.69:0/1840975186 >> 172.21.15.69:0/1840975186 conn(0x7f40a40b5830 msgr2=0x7f40a40b1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.679+0000 7f02b6a00640 1 -- 172.21.15.69:0/607966712 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f02b008bb20 con 0x7f02b803c0a0 2024-04-24T08:08:26.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f02b6a00640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x7f02a0074d40 0x7f02a0077160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f02b6a00640 1 -- 172.21.15.69:0/607966712 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02b0001d30 con 0x7f02a0074d40 2024-04-24T08:08:26.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f02b6a00640 1 -- 172.21.15.69:0/607966712 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f02b0092050 con 0x7f02b803c0a0 2024-04-24T08:08:26.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f02be000640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f02a006cb20 0x7f02a006efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f45b1600640 1 Processor -- start 2024-04-24T08:08:26.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f45b1600640 1 -- start start 2024-04-24T08:08:26.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.680+0000 7f40a9200640 1 -- 172.21.15.69:0/1840975186 shutdown_connections 2024-04-24T08:08:26.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.681+0000 7f5605200640 1 -- 172.21.15.69:0/1940278454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56000b41c0 msgr2=0x7f560006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.681+0000 7f5605200640 1 --2- 172.21.15.69:0/1940278454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56000b41c0 0x7f560006e800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009d90 tx=0x7f55f802a610 comp rx=0 tx=0).stop 2024-04-24T08:08:26.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.681+0000 7f40a9200640 1 -- 172.21.15.69:0/1840975186 wait complete. 2024-04-24T08:08:26.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.681+0000 7fe2a3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe29c162150 0x7fe29c0d5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.681+0000 7fe2a3400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe29c0d10f0 0x7fe29c0d1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.681+0000 7fe2a3400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe29c0d22d0 con 0x7fe29c0cfe40 2024-04-24T08:08:26.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.681+0000 7fe2a3400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe29c0d1a80 con 0x7fe29c162150 2024-04-24T08:08:26.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.681+0000 7fe2a3400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe29c0d2900 con 0x7fe29c0d10f0 2024-04-24T08:08:26.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.682+0000 7f02bf400640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x7f02a0074d40 0x7f02a0077160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.682+0000 7f3046a00640 1 Processor -- start 2024-04-24T08:08:26.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.682+0000 7fe293e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe29c0cfe40 0x7fe29c0b7550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44720/0 (socket says 172.21.15.69:44720) 2024-04-24T08:08:26.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.682+0000 7f3046a00640 1 -- start start 2024-04-24T08:08:26.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.682+0000 7fe293e00640 1 -- 172.21.15.69:0/3950588751 learned_addr learned my addr 172.21.15.69:0/3950588751 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:26.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.682+0000 7f5605200640 1 -- 172.21.15.69:0/1940278454 shutdown_connections 2024-04-24T08:08:26.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.682+0000 7f5605200640 1 --2- 172.21.15.69:0/1940278454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56000b41c0 0x7f560006e800 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.682+0000 7f5605200640 1 -- 172.21.15.69:0/1940278454 >> 172.21.15.69:0/1940278454 conn(0x7f5600067d90 msgr2=0x7f560006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.681+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7e6c068a80 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.683+0000 7f5605200640 1 -- 172.21.15.69:0/1940278454 shutdown_connections 2024-04-24T08:08:26.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.683+0000 7f45b1600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f45ac0d0840 0x7f45ac072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.683+0000 7f45b1600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45ac0d0df0 con 0x7f45ac071d70 2024-04-24T08:08:26.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.683+0000 7f45b1600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45ac06e5a0 con 0x7f45ac151a60 2024-04-24T08:08:26.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.683+0000 7f45b1600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f45ac06ebf0 con 0x7f45ac0d0840 2024-04-24T08:08:26.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.683+0000 7f45aa000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f45ac071d70 0x7f45ac0d0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44736/0 (socket says 172.21.15.69:44736) 2024-04-24T08:08:26.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.683+0000 7f45aaa00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f45ac151a60 0x7f45ac0b3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:38398/0 (socket says 172.21.15.69:38398) 2024-04-24T08:08:26.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.683+0000 7f45aa000640 1 -- 172.21.15.69:0/2184178016 learned_addr learned my addr 172.21.15.69:0/2184178016 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:26.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7f40a9200640 1 Processor -- start 2024-04-24T08:08:26.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7f3d60a00640 1 Processor -- start 2024-04-24T08:08:26.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7f40a9200640 1 -- start start 2024-04-24T08:08:26.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.683+0000 7f5605200640 1 -- 172.21.15.69:0/1940278454 wait complete. 2024-04-24T08:08:26.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7f3d60a00640 1 -- start start 2024-04-24T08:08:26.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7f3046a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f304006f100 0x7f3040201320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7f3046a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30400b3d30 0x7f3040165b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3115605840 0 0) 0x7f45ac06e5a0 con 0x7f45ac151a60 2024-04-24T08:08:26.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f3046a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040162c70 0x7f3040161980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4590003520 con 0x7f45ac151a60 2024-04-24T08:08:26.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.682+0000 7f02be000640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f02a006cb20 0x7f02a006efe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f02b803a450 tx=0x7f02ac009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.682+0000 7fe2a1400640 1 --2- 172.21.15.69:0/3950588751 >> v2:172.21.15.121:3300/0 conn(0x7fe29c0d10f0 0x7fe29c0d1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f303fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f304006f100 0x7f3040201320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f303fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f304006f100 0x7f3040201320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:47304/0 (socket says 172.21.15.69:47304) 2024-04-24T08:08:26.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f30400cfc40 con 0x7f3040162c70 2024-04-24T08:08:26.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3040163050 con 0x7f30400b3d30 2024-04-24T08:08:26.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f30400cfdc0 con 0x7f304006f100 2024-04-24T08:08:26.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f303f400640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30400b3d30 0x7f3040165b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.683+0000 7fe2a0a00640 1 --2- 172.21.15.69:0/3950588751 >> v2:172.21.15.69:3300/0 conn(0x7fe29c162150 0x7fe29c0d5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7fe2a0a00640 1 -- 172.21.15.69:0/3950588751 >> v2:172.21.15.121:3300/0 conn(0x7fe29c0d10f0 msgr2=0x7fe29c0d1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:26.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7fe2a0a00640 1 --2- 172.21.15.69:0/3950588751 >> v2:172.21.15.121:3300/0 conn(0x7fe29c0d10f0 0x7fe29c0d1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7fe2a0a00640 1 -- 172.21.15.69:0/3950588751 >> v1:172.21.15.69:6789/0 conn(0x7fe29c0cfe40 legacy=0x7fe29c0b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.684+0000 7fe2a0a00640 1 -- 172.21.15.69:0/3950588751 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c0d03b0 con 0x7fe29c162150 2024-04-24T08:08:26.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f303fe00640 1 -- 172.21.15.69:0/3412288980 learned_addr learned my addr 172.21.15.69:0/3412288980 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f3044a00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040162c70 0x7f3040161980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 <== mon.1 v1:172.21.15.121:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 521065482 0 0) 0x7f4590003520 con 0x7f45ac151a60 2024-04-24T08:08:26.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 >> v2:172.21.15.121:3300/0 conn(0x7f45ac0d0840 msgr2=0x7f45ac072220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:08:26.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 --2- 172.21.15.69:0/2184178016 >> v2:172.21.15.121:3300/0 conn(0x7f45ac0d0840 0x7f45ac072220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 >> v1:172.21.15.69:6789/0 conn(0x7f45ac071d70 legacy=0x7f45ac0d0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 --> v1:172.21.15.121:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac06e880 con 0x7f45ac151a60 2024-04-24T08:08:26.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 <== mon.1 v1:172.21.15.121:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f45a40019f0 con 0x7f45ac151a60 2024-04-24T08:08:26.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 >> v1:172.21.15.121:6789/0 conn(0x7f45ac151a60 legacy=0x7f45ac0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 --2- 172.21.15.69:0/2184178016 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4590003a00 0x7f4590003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 --2- 172.21.15.69:0/2184178016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4590004800 0x7f4590004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 --2- 172.21.15.69:0/2184178016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45900053c0 0x7f4590005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7fe2a0a00640 1 --2- 172.21.15.69:0/3950588751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe29c162150 0x7fe29c0d5020 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe29c08d700 tx=0x7fe2840329d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b032df82fa80d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac06ebf0 con 0x7f4590004800 2024-04-24T08:08:26.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4590006270 con 0x7f45900053c0 2024-04-24T08:08:26.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.685+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4590006470 con 0x7f4590003a00 2024-04-24T08:08:26.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f40a9200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a4073510 0x7f40a42023d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f40a9200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a40b30f0 0x7f40a4205210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f40a9200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4162c00 0x7f40a4165bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f40a9200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f40a4163520 con 0x7f40a40b30f0 2024-04-24T08:08:26.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f40a9200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f40a4073cc0 con 0x7f40a4073510 2024-04-24T08:08:26.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f40a9200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f40a41636a0 con 0x7f40a4162c00 2024-04-24T08:08:26.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f3044a00640 1 -- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f304006f100 msgr2=0x7f3040201320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f3044a00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f304006f100 0x7f3040201320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f45ab400640 1 --2- 172.21.15.69:0/2184178016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45900053c0 0x7f4590005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f3d60a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d5c162c70 0x7f3d5c0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7fe29be00640 1 -- 172.21.15.69:0/3950588751 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28403b070 con 0x7fe29c162150 2024-04-24T08:08:26.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7fe29be00640 1 -- 172.21.15.69:0/3950588751 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe284037830 con 0x7fe29c162150 2024-04-24T08:08:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7fe29be00640 1 -- 172.21.15.69:0/3950588751 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284037be0 con 0x7fe29c162150 2024-04-24T08:08:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f3044a00640 1 -- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30400b3d30 msgr2=0x7f3040165b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f3044a00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30400b3d30 0x7f3040165b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f3044a00640 1 -- 172.21.15.69:0/3412288980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30400b26d0 con 0x7f3040162c70 2024-04-24T08:08:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f303f400640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30400b3d30 0x7f3040165b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f40a3400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a4073510 0x7f40a42023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f40a3400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a4073510 0x7f40a42023d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:42434/0 (socket says 172.21.15.69:42434) 2024-04-24T08:08:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f40a3400640 1 -- 172.21.15.69:0/2655388002 learned_addr learned my addr 172.21.15.69:0/2655388002 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f40a3e00640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4162c00 0x7f40a4165bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.686+0000 7f3d60a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3d5c0b3d30 0x7f3d5c0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3044a00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040162c70 0x7f3040161980 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3040072e70 tx=0x7f3038004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f45aaa00640 1 --2- 172.21.15.69:0/2184178016 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4590003a00 0x7f4590003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3d60a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5c1669d0 con 0x7f3d5c071d70 2024-04-24T08:08:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3d60a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3d5c0b46d0 con 0x7f3d5c0b3d30 2024-04-24T08:08:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3d60a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d5c166cb0 con 0x7f3d5c162c70 2024-04-24T08:08:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3d59400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3d5c0b3d30 0x7f3d5c0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3d5a800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3d5c071d70 0x7f3d5c0721a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44742/0 (socket says 172.21.15.69:44742) 2024-04-24T08:08:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3d5a800640 1 -- 172.21.15.69:0/1978507804 learned_addr learned my addr 172.21.15.69:0/1978507804 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:26.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3d59e00640 1 --2- 172.21.15.69:0/1978507804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d5c162c70 0x7f3d5c0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f7e62a00640 1 -- 172.21.15.69:0/2954499185 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e6400a7f0 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f5605200640 1 Processor -- start 2024-04-24T08:08:26.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3037e00640 1 -- 172.21.15.69:0/3412288980 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3038041020 con 0x7f3040162c70 2024-04-24T08:08:26.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7fe2a3400640 1 -- 172.21.15.69:0/3950588751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe29c162150 msgr2=0x7fe29c0d5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7fe2a3400640 1 --2- 172.21.15.69:0/3950588751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe29c162150 0x7fe29c0d5020 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe29c08d700 tx=0x7fe2840329d0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3037e00640 1 -- 172.21.15.69:0/3412288980 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3038008710 con 0x7f3040162c70 2024-04-24T08:08:26.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f3037e00640 1 -- 172.21.15.69:0/3412288980 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3038036040 con 0x7f3040162c70 2024-04-24T08:08:26.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f40a2a00640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a40b30f0 0x7f40a4205210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.687+0000 7f5605200640 1 -- start start 2024-04-24T08:08:26.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7fe2a3400640 1 -- 172.21.15.69:0/3950588751 shutdown_connections 2024-04-24T08:08:26.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7fe2a3400640 1 --2- 172.21.15.69:0/3950588751 >> v2:172.21.15.121:3300/0 conn(0x7fe29c0d10f0 0x7fe29c0d1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7fe2a3400640 1 --2- 172.21.15.69:0/3950588751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe29c162150 0x7fe29c0d5020 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7fe2a3400640 1 -- 172.21.15.69:0/3950588751 >> 172.21.15.69:0/3950588751 conn(0x7fe29c08bf60 msgr2=0x7fe29c0d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f40a3e00640 1 -- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a4073510 msgr2=0x7f40a42023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f40a3e00640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a4073510 0x7f40a42023d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f40a3e00640 1 -- 172.21.15.69:0/2655388002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a40b30f0 msgr2=0x7f40a4205210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f40a3e00640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a40b30f0 0x7f40a4205210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f40a3e00640 1 -- 172.21.15.69:0/2655388002 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a4205750 con 0x7f40a4162c00 2024-04-24T08:08:26.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f3d59400640 1 -- 172.21.15.69:0/1978507804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d5c162c70 msgr2=0x7f3d5c0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:26.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f3d59400640 1 --2- 172.21.15.69:0/1978507804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d5c162c70 0x7f3d5c0b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f3d59400640 1 -- 172.21.15.69:0/1978507804 >> v1:172.21.15.69:6789/0 conn(0x7f3d5c071d70 legacy=0x7f3d5c0721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f3d59400640 1 -- 172.21.15.69:0/1978507804 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c166f90 con 0x7f3d5c0b3d30 2024-04-24T08:08:26.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f45ab400640 1 -- 172.21.15.69:0/2184178016 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4590003a00 msgr2=0x7f4590003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:26.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f45ab400640 1 --2- 172.21.15.69:0/2184178016 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4590003a00 0x7f4590003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f45ab400640 1 -- 172.21.15.69:0/2184178016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4590004800 msgr2=0x7f4590004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:26.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f45ab400640 1 --2- 172.21.15.69:0/2184178016 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4590004800 0x7f4590004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f45ab400640 1 -- 172.21.15.69:0/2184178016 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4590003520 con 0x7f45900053c0 2024-04-24T08:08:26.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f40a3e00640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4162c00 0x7f40a4165bc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f409000c3c0 tx=0x7f409000c7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f02bf400640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x7f02a0074d40 0x7f02a0077160 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30400b29b0 con 0x7f3040162c70 2024-04-24T08:08:26.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.688+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30400b2360 con 0x7f3040162c70 2024-04-24T08:08:26.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f45ab400640 1 -- 172.21.15.69:0/2184178016 reap_dead start 2024-04-24T08:08:26.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7fe2a3400640 1 -- 172.21.15.69:0/3950588751 shutdown_connections 2024-04-24T08:08:26.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7fe2a3400640 1 -- 172.21.15.69:0/3950588751 wait complete. 2024-04-24T08:08:26.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f5605200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f56000b3d30 0x7f5600164680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f5605200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56000b41c0 0x7f56001fcb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f5605200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5600162c70 0x7f560014df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f5605200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f56000b45f0 con 0x7f5600162c70 2024-04-24T08:08:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f5605200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5600163050 con 0x7f56000b41c0 2024-04-24T08:08:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f5605200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f56000cfc40 con 0x7f56000b3d30 2024-04-24T08:08:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f409b400640 1 -- 172.21.15.69:0/2655388002 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4090017070 con 0x7f40a4162c00 2024-04-24T08:08:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f409b400640 1 -- 172.21.15.69:0/2655388002 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4090005ce0 con 0x7f40a4162c00 2024-04-24T08:08:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f409b400640 1 -- 172.21.15.69:0/2655388002 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40900072d0 con 0x7f40a4162c00 2024-04-24T08:08:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f55fe800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5600162c70 0x7f560014df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f55fe800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5600162c70 0x7f560014df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41364/0 (socket says 172.21.15.69:41364) 2024-04-24T08:08:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f55fe800640 1 -- 172.21.15.69:0/3874654166 learned_addr learned my addr 172.21.15.69:0/3874654166 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f55fd400640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56000b41c0 0x7f56001fcb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3018001030 con 0x7f3040162c70 2024-04-24T08:08:26.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a4153020 con 0x7f40a4162c00 2024-04-24T08:08:26.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a41535e0 con 0x7f40a4162c00 2024-04-24T08:08:26.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f02b6a00640 1 -- 172.21.15.69:0/607966712 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f02b0001d30 con 0x7f02a0074d40 2024-04-24T08:08:26.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f3d59400640 1 --2- 172.21.15.69:0/1978507804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d5c0b3d30 0x7f3d5c0b4190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3d480097a0 tx=0x7f3d4802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5eec9404805fc8c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f55fe800640 1 -- 172.21.15.69:0/3874654166 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f56000b3d30 msgr2=0x7f5600164680 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T08:08:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f55fe800640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f56000b3d30 0x7f5600164680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f55fe800640 1 -- 172.21.15.69:0/3874654166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56000b41c0 msgr2=0x7f56001fcb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f55fe800640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56000b41c0 0x7f56001fcb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.690+0000 7f55fe800640 1 -- 172.21.15.69:0/3874654166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5600165d90 con 0x7f5600162c70 2024-04-24T08:08:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f7e62a00640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4c06c8f0 0x7f7e4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.689+0000 7f7e62a00640 1 -- 172.21.15.69:0/2954499185 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7e6408b790 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.691+0000 7f55fe800640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5600162c70 0x7f560014df40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009eb0 tx=0x7f55f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.691+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f40a4153c30 con 0x7f40a4162c00 2024-04-24T08:08:26.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.691+0000 7f3d58a00640 1 -- 172.21.15.69:0/1978507804 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48037070 con 0x7f3d5c0b3d30 2024-04-24T08:08:26.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.691+0000 7f3d58a00640 1 -- 172.21.15.69:0/1978507804 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d48032440 con 0x7f3d5c0b3d30 2024-04-24T08:08:26.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.691+0000 7f3d58a00640 1 -- 172.21.15.69:0/1978507804 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48032810 con 0x7f3d5c0b3d30 2024-04-24T08:08:26.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.691+0000 7f45ab400640 1 --2- 172.21.15.69:0/2184178016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45900053c0 0x7f4590005990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4598009860 tx=0x7f4590005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e784cfcf62bf0019 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.691+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598037070 con 0x7f45900053c0 2024-04-24T08:08:26.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.691+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f459802a9f0 con 0x7f45900053c0 2024-04-24T08:08:26.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7fe2a3400640 1 Processor -- start 2024-04-24T08:08:26.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f3d60a00640 1 -- 172.21.15.69:0/1978507804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d5c0b3d30 msgr2=0x7f3d5c0b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f3d60a00640 1 --2- 172.21.15.69:0/1978507804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d5c0b3d30 0x7f3d5c0b4190 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3d480097a0 tx=0x7f3d4802a180 comp rx=0 tx=0).stop 2024-04-24T08:08:26.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7fe2a3400640 1 -- start start 2024-04-24T08:08:26.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f45b1600640 1 -- 172.21.15.69:0/2184178016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45900053c0 msgr2=0x7f4590005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f7e6a000640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4c06c8f0 0x7f7e4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f7e62a00640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x7f7e4c074b70 0x7f7e4c076f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f7e62a00640 1 -- 172.21.15.69:0/2954499185 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e6401eec0 con 0x7f7e4c074b70 2024-04-24T08:08:26.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f7e62a00640 1 -- 172.21.15.69:0/2954499185 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f7e64092050 con 0x7f7e6c0cfe40 2024-04-24T08:08:26.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f45a9600640 1 -- 172.21.15.69:0/2184178016 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598037070 con 0x7f45900053c0 2024-04-24T08:08:26.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f45b1600640 1 --2- 172.21.15.69:0/2184178016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45900053c0 0x7f4590005990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4598009860 tx=0x7f4590005830 comp rx=0 tx=0).stop 2024-04-24T08:08:26.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f3d60a00640 1 -- 172.21.15.69:0/1978507804 shutdown_connections 2024-04-24T08:08:26.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f3d60a00640 1 --2- 172.21.15.69:0/1978507804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d5c0b3d30 0x7f3d5c0b4190 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f3d60a00640 1 --2- 172.21.15.69:0/1978507804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d5c162c70 0x7f3d5c0b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.692+0000 7f3d60a00640 1 -- 172.21.15.69:0/1978507804 >> 172.21.15.69:0/1978507804 conn(0x7f3d5c067d90 msgr2=0x7f3d5c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f3d60a00640 1 -- 172.21.15.69:0/1978507804 shutdown_connections 2024-04-24T08:08:26.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f45b1600640 1 -- 172.21.15.69:0/2184178016 shutdown_connections 2024-04-24T08:08:26.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f45b1600640 1 --2- 172.21.15.69:0/2184178016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45900053c0 0x7f4590005990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f45b1600640 1 -- 172.21.15.69:0/2184178016 >> 172.21.15.69:0/2184178016 conn(0x7f45ac067d90 msgr2=0x7f45ac06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f55f6000640 1 -- 172.21.15.69:0/3874654166 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8040070 con 0x7f5600162c70 2024-04-24T08:08:26.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f409b400640 1 -- 172.21.15.69:0/2655388002 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f409001a070 con 0x7f40a4162c00 2024-04-24T08:08:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f45b1600640 1 -- 172.21.15.69:0/2184178016 shutdown_connections 2024-04-24T08:08:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f45b1600640 1 -- 172.21.15.69:0/2184178016 wait complete. 2024-04-24T08:08:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f7e6a000640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4c06c8f0 0x7f7e4c06edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0b9ff0 tx=0x7f7e5c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f55f6000640 1 -- 172.21.15.69:0/3874654166 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f8031bc0 con 0x7f5600162c70 2024-04-24T08:08:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f55f6000640 1 -- 172.21.15.69:0/3874654166 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f8045460 con 0x7f5600162c70 2024-04-24T08:08:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.694+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5600164cf0 con 0x7f5600162c70 2024-04-24T08:08:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.694+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56001652b0 con 0x7f5600162c70 2024-04-24T08:08:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.694+0000 7f409b400640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f408406c940 0x7f408406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.694+0000 7f409b400640 1 -- 172.21.15.69:0/2655388002 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f409008b840 con 0x7f40a4162c00 2024-04-24T08:08:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.694+0000 7f409b400640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x7f4084074bc0 0x7f4084076fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.694+0000 7f409b400640 1 -- 172.21.15.69:0/2655388002 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4090014ec0 con 0x7f4084074bc0 2024-04-24T08:08:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.694+0000 7f409b400640 1 -- 172.21.15.69:0/2655388002 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f409008bd20 con 0x7f40a4162c00 2024-04-24T08:08:26.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.693+0000 7f3d60a00640 1 -- 172.21.15.69:0/1978507804 wait complete. 2024-04-24T08:08:26.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.695+0000 7f40a2a00640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x7f4084074bc0 0x7f4084076fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.695+0000 7f40a3400640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f408406c940 0x7f408406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.695+0000 7f3d60a00640 1 Processor -- start 2024-04-24T08:08:26.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.695+0000 7f3d60a00640 1 -- start start 2024-04-24T08:08:26.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.695+0000 7f3037e00640 1 -- 172.21.15.69:0/3412288980 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3038044020 con 0x7f3040162c70 2024-04-24T08:08:26.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.696+0000 7f7e6b400640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x7f7e4c074b70 0x7f7e4c076f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.696+0000 7f45b1600640 1 Processor -- start 2024-04-24T08:08:26.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.696+0000 7f40a3400640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f408406c940 0x7f408406ee00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f409c0094c0 tx=0x7f409c00b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.696+0000 7f40a2a00640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x7f4084074bc0 0x7f4084076fe0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.696+0000 7fe2a3400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c0cfe40 0x7fe29c1509e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3d60a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d5c071d70 0x7f3d5c1519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3d60a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d5c0b3d30 0x7f3d5c157dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3d60a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d5c162c70 0x7f3d5c158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3d60a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c163050 con 0x7f3d5c162c70 2024-04-24T08:08:26.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3d60a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c167550 con 0x7f3d5c071d70 2024-04-24T08:08:26.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3037e00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f301c06c970 0x7f301c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3d60a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d5c0cfc40 con 0x7f3d5c0b3d30 2024-04-24T08:08:26.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7fe2a3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe29c0d10f0 0x7fe29c15b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3037e00640 1 -- 172.21.15.69:0/3412288980 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f30380aca20 con 0x7f3040162c70 2024-04-24T08:08:26.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5600068a80 con 0x7f5600162c70 2024-04-24T08:08:26.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3d59e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d5c071d70 0x7f3d5c1519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3037e00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x7f301c074b90 0x7f301c076fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3037e00640 1 -- 172.21.15.69:0/3412288980 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3038032e50 con 0x7f301c074b90 2024-04-24T08:08:26.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3037e00640 1 -- 172.21.15.69:0/3412288980 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f30380b12a0 con 0x7f3040162c70 2024-04-24T08:08:26.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f45b1600640 1 -- start start 2024-04-24T08:08:26.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7fe2a3400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe29c162150 0x7fe29c15d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7fe2a3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c0b7c60 con 0x7fe29c0d10f0 2024-04-24T08:08:26.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7fe2a3400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c0d2590 con 0x7fe29c0cfe40 2024-04-24T08:08:26.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7fe2a3400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe29c0d5560 con 0x7fe29c162150 2024-04-24T08:08:26.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7fe293e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe29c0d10f0 0x7fe29c15b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3d5a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d5c162c70 0x7f3d5c158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3d5a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d5c162c70 0x7f3d5c158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41394/0 (socket says 172.21.15.69:41394) 2024-04-24T08:08:26.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.697+0000 7f3d5a800640 1 -- 172.21.15.69:0/371892030 learned_addr learned my addr 172.21.15.69:0/371892030 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7f303fe00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f301c06c970 0x7f301c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7f3d59400640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d5c0b3d30 0x7f3d5c157dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7fe2a1400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe29c162150 0x7fe29c15d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7f7e6b400640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x7f7e4c074b70 0x7f7e4c076f90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7fe2a0a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c0cfe40 0x7fe29c1509e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7fe2a0a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c0cfe40 0x7fe29c1509e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:42456/0 (socket says 172.21.15.69:42456) 2024-04-24T08:08:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7fe2a0a00640 1 -- 172.21.15.69:0/3276624040 learned_addr learned my addr 172.21.15.69:0/3276624040 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7f55f6000640 1 -- 172.21.15.69:0/3874654166 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f55f8039040 con 0x7f5600162c70 2024-04-24T08:08:26.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7f3d59e00640 1 -- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d5c0b3d30 msgr2=0x7f3d5c157dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.698+0000 7f3d59e00640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d5c0b3d30 0x7f3d5c157dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7f3d59e00640 1 -- 172.21.15.69:0/371892030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d5c162c70 msgr2=0x7f3d5c158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7f3d59e00640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d5c162c70 0x7f3d5c158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7f3d59e00640 1 -- 172.21.15.69:0/371892030 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c164db0 con 0x7f3d5c071d70 2024-04-24T08:08:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7f3d5a800640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d5c162c70 0x7f3d5c158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:26.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7f45b1600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f45900053c0 0x7f45ac0ac540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7f303f400640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x7f301c074b90 0x7f301c076fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7fe2a0a00640 1 -- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe29c162150 msgr2=0x7fe29c15d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7f45b1600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45ac151e40 0x7f45ac1538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7fe2a0a00640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe29c162150 0x7fe29c15d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7f7e62a00640 1 -- 172.21.15.69:0/2954499185 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e6401eec0 con 0x7f7e4c074b70 2024-04-24T08:08:26.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7fe2a0a00640 1 -- 172.21.15.69:0/3276624040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe29c0d10f0 msgr2=0x7fe29c15b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7f45b1600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45ac1522a0 0x7f45ac153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7f3d59e00640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d5c071d70 0x7f3d5c1519e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00e870 tx=0x7f3d4c00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f3d52000640 1 -- 172.21.15.69:0/371892030 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c017070 con 0x7f3d5c071d70 2024-04-24T08:08:26.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f3d52000640 1 -- 172.21.15.69:0/371892030 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d4c00ee60 con 0x7f3d5c071d70 2024-04-24T08:08:26.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f3d52000640 1 -- 172.21.15.69:0/371892030 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c00cb60 con 0x7f3d5c071d70 2024-04-24T08:08:26.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f45b1600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac072220 con 0x7f45ac1522a0 2024-04-24T08:08:26.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.699+0000 7fe2a0a00640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe29c0d10f0 0x7fe29c15b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f45b1600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac070070 con 0x7f45ac151e40 2024-04-24T08:08:26.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f45b1600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f45ac0723a0 con 0x7f45900053c0 2024-04-24T08:08:26.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f45ab400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45ac1522a0 0x7f45ac153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7fe2a0a00640 1 -- 172.21.15.69:0/3276624040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c0ca6d0 con 0x7fe29c0cfe40 2024-04-24T08:08:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f45ab400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45ac1522a0 0x7f45ac153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41398/0 (socket says 172.21.15.69:41398) 2024-04-24T08:08:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f45ab400640 1 -- 172.21.15.69:0/4150042075 learned_addr learned my addr 172.21.15.69:0/4150042075 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f45aaa00640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f45900053c0 0x7f45ac0ac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5c158cd0 con 0x7f3d5c071d70 2024-04-24T08:08:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5c165330 con 0x7f3d5c071d70 2024-04-24T08:08:26.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7fe293e00640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe29c0d10f0 0x7fe29c15b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:26.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.700+0000 7f303fe00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f301c06c970 0x7f301c06ee30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3028007a60 tx=0x7f3028008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f409b400640 1 -- 172.21.15.69:0/2655388002 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4090014ec0 con 0x7f4084074bc0 2024-04-24T08:08:26.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f55f6000640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55e006c9c0 0x7f55e006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f45aa000640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45ac151e40 0x7f45ac1538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7fe2a0a00640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c0cfe40 0x7fe29c1509e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe284032fc0 tx=0x7fe2840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f303f400640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x7f301c074b90 0x7f301c076fb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f55f6000640 1 -- 172.21.15.69:0/3874654166 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f55f80ad4f0 con 0x7f5600162c70 2024-04-24T08:08:26.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f55f6000640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x7f55e0074be0 0x7f55e0077000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f55f6000640 1 -- 172.21.15.69:0/3874654166 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55e00776b0 con 0x7f55e0074be0 2024-04-24T08:08:26.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f55f6000640 1 -- 172.21.15.69:0/3874654166 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f55f80b4050 con 0x7f5600162c70 2024-04-24T08:08:26.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f45ab400640 1 -- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f45900053c0 msgr2=0x7f45ac0ac540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f45ab400640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f45900053c0 0x7f45ac0ac540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f45ab400640 1 -- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45ac151e40 msgr2=0x7f45ac1538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f45ab400640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45ac151e40 0x7f45ac1538f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.701+0000 7f45ab400640 1 -- 172.21.15.69:0/4150042075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac152850 con 0x7f45ac1522a0 2024-04-24T08:08:26.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.702+0000 7fe299600640 1 -- 172.21.15.69:0/3276624040 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28403b050 con 0x7fe29c0cfe40 2024-04-24T08:08:26.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.702+0000 7f55fd400640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x7f55e0074be0 0x7f55e0077000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.702+0000 7f55fde00640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55e006c9c0 0x7f55e006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.702+0000 7f45ab400640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45ac1522a0 0x7f45ac153e30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4598009950 tx=0x7f4598004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.702+0000 7fe299600640 1 -- 172.21.15.69:0/3276624040 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe284003810 con 0x7fe29c0cfe40 2024-04-24T08:08:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.702+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c0cc8c0 con 0x7fe29c0cfe40 2024-04-24T08:08:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.703+0000 7f3037e00640 1 -- 172.21.15.69:0/3412288980 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3038032e50 con 0x7f301c074b90 2024-04-24T08:08:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.702+0000 7f45a2a00640 1 -- 172.21.15.69:0/4150042075 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598037050 con 0x7f45ac1522a0 2024-04-24T08:08:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.703+0000 7f45a2a00640 1 -- 172.21.15.69:0/4150042075 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4598038040 con 0x7f45ac1522a0 2024-04-24T08:08:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.703+0000 7f45a2a00640 1 -- 172.21.15.69:0/4150042075 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45980093b0 con 0x7f45ac1522a0 2024-04-24T08:08:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.702+0000 7fe299600640 1 -- 172.21.15.69:0/3276624040 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284003be0 con 0x7fe29c0cfe40 2024-04-24T08:08:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.703+0000 7f55fd400640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x7f55e0074be0 0x7f55e0077000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.703+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c0cce80 con 0x7fe29c0cfe40 2024-04-24T08:08:26.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.703+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ac152b30 con 0x7f45ac1522a0 2024-04-24T08:08:26.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.703+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ac1530f0 con 0x7f45ac1522a0 2024-04-24T08:08:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.704+0000 7f55f6000640 1 -- 172.21.15.69:0/3874654166 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f55e00776b0 con 0x7f55e0074be0 2024-04-24T08:08:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.704+0000 7f55fde00640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55e006c9c0 0x7f55e006ee80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f55f0007fd0 tx=0x7f55f0007960 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.707+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe29c068a80 con 0x7fe29c0cfe40 2024-04-24T08:08:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.705+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f45ac068a80 con 0x7f45ac1522a0 2024-04-24T08:08:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.706+0000 7f45a2a00640 1 -- 172.21.15.69:0/4150042075 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f459803e8b0 con 0x7f45ac1522a0 2024-04-24T08:08:26.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.704+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3d5c165670 con 0x7f3d5c071d70 2024-04-24T08:08:26.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.708+0000 7f45a2a00640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f458c06c940 0x7f458c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.708+0000 7f45a2a00640 1 -- 172.21.15.69:0/4150042075 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f45980abce0 con 0x7f45ac1522a0 2024-04-24T08:08:26.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.708+0000 7f45a2a00640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x7f458c074b60 0x7f458c076f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.708+0000 7f45a2a00640 1 -- 172.21.15.69:0/4150042075 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45980067f0 con 0x7f458c074b60 2024-04-24T08:08:26.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.708+0000 7f45a2a00640 1 -- 172.21.15.69:0/4150042075 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f45980ae430 con 0x7f45ac1522a0 2024-04-24T08:08:26.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.708+0000 7f45aaa00640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f458c06c940 0x7f458c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.709+0000 7f45aa000640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x7f458c074b60 0x7f458c076f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.709+0000 7f3d52000640 1 -- 172.21.15.69:0/371892030 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d4c01a070 con 0x7f3d5c071d70 2024-04-24T08:08:26.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.709+0000 7fe299600640 1 -- 172.21.15.69:0/3276624040 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe28403e070 con 0x7fe29c0cfe40 2024-04-24T08:08:26.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.710+0000 7f45aaa00640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f458c06c940 0x7f458c06ee00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f45a4005e90 tx=0x7f45a4007f30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.710+0000 7f45aa000640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x7f458c074b60 0x7f458c076f80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.711+0000 7fe299600640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe27806c8f0 0x7fe27806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.711+0000 7fe299600640 1 -- 172.21.15.69:0/3276624040 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe2840b4a80 con 0x7fe29c0cfe40 2024-04-24T08:08:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.711+0000 7fe299600640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x7fe278074b70 0x7fe278076f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.711+0000 7fe299600640 1 -- 172.21.15.69:0/3276624040 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe278077640 con 0x7fe278074b70 2024-04-24T08:08:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.711+0000 7fe299600640 1 -- 172.21.15.69:0/3276624040 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe2840b4e70 con 0x7fe29c0cfe40 2024-04-24T08:08:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.712+0000 7fe293e00640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe27806c8f0 0x7fe27806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.712+0000 7fe2a1400640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x7fe278074b70 0x7fe278076f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.710+0000 7f3d52000640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d3c06c8f0 0x7f3d3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.710+0000 7f3d52000640 1 -- 172.21.15.69:0/371892030 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3d4c08b8f0 con 0x7f3d5c071d70 2024-04-24T08:08:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.710+0000 7f3d52000640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x7f3d3c074a90 0x7f3d3c076eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.710+0000 7f3d52000640 1 -- 172.21.15.69:0/371892030 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d4c004470 con 0x7f3d3c074a90 2024-04-24T08:08:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.710+0000 7f3d52000640 1 -- 172.21.15.69:0/371892030 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f3d4c0902c0 con 0x7f3d5c071d70 2024-04-24T08:08:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.713+0000 7f45a2a00640 1 -- 172.21.15.69:0/4150042075 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45980067f0 con 0x7f458c074b60 2024-04-24T08:08:26.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.713+0000 7f3d5a800640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x7f3d3c074a90 0x7f3d3c076eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.714+0000 7f3d59400640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d3c06c8f0 0x7f3d3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:26.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.714+0000 7fe293e00640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe27806c8f0 0x7fe27806edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe29c073120 tx=0x7fe28c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.714+0000 7fe2a1400640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x7fe278074b70 0x7fe278076f90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.715+0000 7f3d5a800640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x7f3d3c074a90 0x7f3d3c076eb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.715+0000 7f3d59400640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d3c06c8f0 0x7f3d3c06edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3d48034000 tx=0x7f3d4802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:26.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.717+0000 7fe299600640 1 -- 172.21.15.69:0/3276624040 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe278077640 con 0x7fe278074b70 2024-04-24T08:08:26.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.718+0000 7f3d52000640 1 -- 172.21.15.69:0/371892030 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d4c004470 con 0x7f3d3c074a90 2024-04-24T08:08:26.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.725+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f02b8005390 con 0x7f02a0074d40 2024-04-24T08:08:26.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.728+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7e6c0d1df0 con 0x7f7e4c074b70 2024-04-24T08:08:26.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.728+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f40a42011e0 con 0x7f4084074bc0 2024-04-24T08:08:26.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.728+0000 7f02b6a00640 1 -- 172.21.15.69:0/607966712 <== osd.0 v2:172.21.15.69:6800/146737935 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f02b8005390 con 0x7f02a0074d40 2024-04-24T08:08:26.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.729+0000 7f7e62a00640 1 -- 172.21.15.69:0/2954499185 <== osd.1 v2:172.21.15.69:6815/153736297 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7e6c0d1df0 con 0x7f7e4c074b70 2024-04-24T08:08:26.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.729+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x7f02a0074d40 msgr2=0x7f02a0077160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.729+0000 7f02bfe00640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x7f02a0074d40 0x7f02a0077160 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f02a006cb20 msgr2=0x7f02a006efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f02bfe00640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f02a006cb20 0x7f02a006efe0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f02b803a450 tx=0x7f02ac009040 comp rx=0 tx=0).stop 2024-04-24T08:08:26.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b803c0a0 msgr2=0x7f02b80d0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f02bfe00640 1 --2- 172.21.15.69:0/607966712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02b803c0a0 0x7f02b80d0db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02b000d5f0 tx=0x7f02b000dab0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f409b400640 1 -- 172.21.15.69:0/2655388002 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f40a42011e0 con 0x7f4084074bc0 2024-04-24T08:08:26.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f02bf400640 1 -- 172.21.15.69:0/607966712 reap_dead start 2024-04-24T08:08:26.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 shutdown_connections 2024-04-24T08:08:26.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 >> 172.21.15.69:0/607966712 conn(0x7f02b8058ee0 msgr2=0x7f02b803e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x7f7e4c074b70 msgr2=0x7f7e4c076f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f7e71a00640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x7f7e4c074b70 0x7f7e4c076f90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4c06c8f0 msgr2=0x7f7e4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f7e71a00640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4c06c8f0 0x7f7e4c06edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c0b9ff0 tx=0x7f7e5c009040 comp rx=0 tx=0).stop 2024-04-24T08:08:26.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0cfe40 msgr2=0x7f7e6c15a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f7e71a00640 1 --2- 172.21.15.69:0/2954499185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0cfe40 0x7f7e6c15a860 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c730 tx=0x7f7e6400cbf0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 shutdown_connections 2024-04-24T08:08:26.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f02bfe00640 1 -- 172.21.15.69:0/607966712 wait complete. 2024-04-24T08:08:26.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.730+0000 7f7e6b400640 1 -- 172.21.15.69:0/2954499185 reap_dead start 2024-04-24T08:08:26.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.731+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 shutdown_connections 2024-04-24T08:08:26.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.731+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 >> 172.21.15.69:0/2954499185 conn(0x7f7e6c08bf60 msgr2=0x7f7e6c0b9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.731+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x7f4084074bc0 msgr2=0x7f4084076fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.731+0000 7f40a9200640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x7f4084074bc0 0x7f4084076fe0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.732+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f408406c940 msgr2=0x7f408406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.732+0000 7f40a9200640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f408406c940 0x7f408406ee00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f409c0094c0 tx=0x7f409c00b040 comp rx=0 tx=0).stop 2024-04-24T08:08:26.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.732+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4162c00 msgr2=0x7f40a4165bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.732+0000 7f40a9200640 1 --2- 172.21.15.69:0/2655388002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4162c00 0x7f40a4165bc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f409000c3c0 tx=0x7f409000c7c0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.732+0000 7f40a3e00640 1 -- 172.21.15.69:0/2655388002 reap_dead start 2024-04-24T08:08:26.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.732+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 shutdown_connections 2024-04-24T08:08:26.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.732+0000 7f7e71a00640 1 -- 172.21.15.69:0/2954499185 wait complete. 2024-04-24T08:08:26.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.733+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3018002dd0 con 0x7f301c074b90 2024-04-24T08:08:26.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.733+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 shutdown_connections 2024-04-24T08:08:26.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.733+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 >> 172.21.15.69:0/2655388002 conn(0x7f40a40b5830 msgr2=0x7f40a4158300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.734+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 shutdown_connections 2024-04-24T08:08:26.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.734+0000 7f40a9200640 1 -- 172.21.15.69:0/2655388002 wait complete. 2024-04-24T08:08:26.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.734+0000 7f3037e00640 1 -- 172.21.15.69:0/3412288980 <== osd.3 v2:172.21.15.69:6811/1875112303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3018002dd0 con 0x7f301c074b90 2024-04-24T08:08:26.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.736+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x7f301c074b90 msgr2=0x7f301c076fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.736+0000 7f3046a00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x7f301c074b90 0x7f301c076fb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.735+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f560014e700 con 0x7f55e0074be0 2024-04-24T08:08:26.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.736+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f301c06c970 msgr2=0x7f301c06ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.736+0000 7f3046a00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f301c06c970 0x7f301c06ee30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3028007a60 tx=0x7f3028008040 comp rx=0 tx=0).stop 2024-04-24T08:08:26.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.736+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040162c70 msgr2=0x7f3040161980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.736+0000 7f3046a00640 1 --2- 172.21.15.69:0/3412288980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3040162c70 0x7f3040161980 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3040072e70 tx=0x7f3038004a40 comp rx=0 tx=0).stop 2024-04-24T08:08:26.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.736+0000 7f3044a00640 1 -- 172.21.15.69:0/3412288980 reap_dead start 2024-04-24T08:08:26.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.737+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 shutdown_connections 2024-04-24T08:08:26.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.737+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 >> 172.21.15.69:0/3412288980 conn(0x7f3040067d90 msgr2=0x7f30400cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.737+0000 7f55f6000640 1 -- 172.21.15.69:0/3874654166 <== osd.2 v2:172.21.15.69:6808/2630808717 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f560014e700 con 0x7f55e0074be0 2024-04-24T08:08:26.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.737+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 shutdown_connections 2024-04-24T08:08:26.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.737+0000 7f3046a00640 1 -- 172.21.15.69:0/3412288980 wait complete. 2024-04-24T08:08:26.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.738+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x7f55e0074be0 msgr2=0x7f55e0077000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.738+0000 7f5605200640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x7f55e0074be0 0x7f55e0077000 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.738+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55e006c9c0 msgr2=0x7f55e006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.738+0000 7f5605200640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55e006c9c0 0x7f55e006ee80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f55f0007fd0 tx=0x7f55f0007960 comp rx=0 tx=0).stop 2024-04-24T08:08:26.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.738+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5600162c70 msgr2=0x7f560014df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.738+0000 7f5605200640 1 --2- 172.21.15.69:0/3874654166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5600162c70 0x7f560014df40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f55f8009eb0 tx=0x7f55f8004a40 comp rx=0 tx=0).stop 2024-04-24T08:08:26.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.738+0000 7f55fe800640 1 -- 172.21.15.69:0/3874654166 reap_dead start 2024-04-24T08:08:26.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.739+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 shutdown_connections 2024-04-24T08:08:26.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.739+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 >> 172.21.15.69:0/3874654166 conn(0x7f5600067d90 msgr2=0x7f560006e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.739+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 shutdown_connections 2024-04-24T08:08:26.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.739+0000 7f5605200640 1 -- 172.21.15.69:0/3874654166 wait complete. 2024-04-24T08:08:26.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.741+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f45ac06e1e0 con 0x7f458c074b60 2024-04-24T08:08:26.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.742+0000 7f45a2a00640 1 -- 172.21.15.69:0/4150042075 <== osd.5 v2:172.21.15.121:6822/2187246263 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f45ac06e1e0 con 0x7f458c074b60 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.743+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x7f458c074b60 msgr2=0x7f458c076f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.743+0000 7f45b1600640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x7f458c074b60 0x7f458c076f80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.744+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f458c06c940 msgr2=0x7f458c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.744+0000 7f45b1600640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f458c06c940 0x7f458c06ee00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f45a4005e90 tx=0x7f45a4007f30 comp rx=0 tx=0).stop 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.744+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45ac1522a0 msgr2=0x7f45ac153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.744+0000 7f45b1600640 1 --2- 172.21.15.69:0/4150042075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45ac1522a0 0x7f45ac153e30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4598009950 tx=0x7f4598004700 comp rx=0 tx=0).stop 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.744+0000 7f45ab400640 1 -- 172.21.15.69:0/4150042075 reap_dead start 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.744+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 shutdown_connections 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.744+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 >> 172.21.15.69:0/4150042075 conn(0x7f45ac067d90 msgr2=0x7f45ac0b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.744+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 shutdown_connections 2024-04-24T08:08:26.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.744+0000 7f45b1600640 1 -- 172.21.15.69:0/4150042075 wait complete. 2024-04-24T08:08:26.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.752+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe29c06e990 con 0x7fe278074b70 2024-04-24T08:08:26.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.752+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3d5c068a80 con 0x7f3d3c074a90 2024-04-24T08:08:26.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.753+0000 7f3d52000640 1 -- 172.21.15.69:0/371892030 <== osd.6 v2:172.21.15.121:6807/3068198325 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3d5c068a80 con 0x7f3d3c074a90 2024-04-24T08:08:26.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.754+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x7f3d3c074a90 msgr2=0x7f3d3c076eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.754+0000 7f3d60a00640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x7f3d3c074a90 0x7f3d3c076eb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.754+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d3c06c8f0 msgr2=0x7f3d3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.754+0000 7f3d60a00640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d3c06c8f0 0x7f3d3c06edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3d48034000 tx=0x7f3d4802e000 comp rx=0 tx=0).stop 2024-04-24T08:08:26.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.754+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d5c071d70 msgr2=0x7f3d5c1519e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.754+0000 7f3d60a00640 1 --2- 172.21.15.69:0/371892030 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d5c071d70 0x7f3d5c1519e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00e870 tx=0x7f3d4c00ed30 comp rx=0 tx=0).stop 2024-04-24T08:08:26.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.754+0000 7f3d5a800640 1 -- 172.21.15.69:0/371892030 reap_dead start 2024-04-24T08:08:26.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.755+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 shutdown_connections 2024-04-24T08:08:26.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.755+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 >> 172.21.15.69:0/371892030 conn(0x7f3d5c067d90 msgr2=0x7f3d5c0b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.755+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 shutdown_connections 2024-04-24T08:08:26.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.755+0000 7f3d60a00640 1 -- 172.21.15.69:0/371892030 wait complete. 2024-04-24T08:08:26.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.755+0000 7fe299600640 1 -- 172.21.15.69:0/3276624040 <== osd.7 v2:172.21.15.121:6811/176144805 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe29c06e990 con 0x7fe278074b70 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.756+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x7fe278074b70 msgr2=0x7fe278076f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.756+0000 7fe2a3400640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x7fe278074b70 0x7fe278076f90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.756+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe27806c8f0 msgr2=0x7fe27806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.756+0000 7fe2a3400640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe27806c8f0 0x7fe27806edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe29c073120 tx=0x7fe28c009040 comp rx=0 tx=0).stop 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.756+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c0cfe40 msgr2=0x7fe29c1509e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.756+0000 7fe2a3400640 1 --2- 172.21.15.69:0/3276624040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe29c0cfe40 0x7fe29c1509e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe284032fc0 tx=0x7fe2840046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.756+0000 7fe2a1400640 1 -- 172.21.15.69:0/3276624040 reap_dead start 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.757+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 shutdown_connections 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.757+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 >> 172.21.15.69:0/3276624040 conn(0x7fe29c08bf60 msgr2=0x7fe29c0d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.757+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 shutdown_connections 2024-04-24T08:08:26.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:26.757+0000 7fe2a3400640 1 -- 172.21.15.69:0/3276624040 wait complete. 2024-04-24T08:08:26.785 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2024-04-24T08:08:26.785 INFO:teuthology.orchestra.run.smithi069.stdout:55834574855 2024-04-24T08:08:26.786 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T08:08:26.786 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T08:08:26.787 INFO:teuthology.orchestra.run.smithi069.stdout:55834574853 2024-04-24T08:08:26.788 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T08:08:26.788 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2024-04-24T08:08:26.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T08:08:26.790 INFO:teuthology.orchestra.run.smithi069.stdout:55834574855 2024-04-24T08:08:26.790 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T08:08:26.791 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2024-04-24T08:08:26.791 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T08:08:26.791 INFO:teuthology.orchestra.run.smithi069.stdout:55834574856 2024-04-24T08:08:26.792 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T08:08:26.793 INFO:teuthology.orchestra.run.smithi069.stdout:55834574856 2024-04-24T08:08:26.793 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-24T08:08:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.100+0000 7f3862c00640 1 Processor -- start 2024-04-24T08:08:27.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.100+0000 7f3862c00640 1 -- start start 2024-04-24T08:08:27.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.103+0000 7f3862c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f385c162e00 0x7f385c0d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.103+0000 7f3862c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f385c0d0840 0x7f385c0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.103+0000 7f3862c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385c167880 con 0x7f385c0b3b70 2024-04-24T08:08:27.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.103+0000 7f3862c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f385c0ccf90 con 0x7f385c162e00 2024-04-24T08:08:27.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.103+0000 7f3862c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f385c1631e0 con 0x7f385c0d0840 2024-04-24T08:08:27.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.103+0000 7f385be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f385c162e00 0x7f385c0d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.104+0000 7f385be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f385c162e00 0x7f385c0d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41406/0 (socket says 172.21.15.69:41406) 2024-04-24T08:08:27.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.104+0000 7f385be00640 1 -- 172.21.15.69:0/3067835888 learned_addr learned my addr 172.21.15.69:0/3067835888 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.104+0000 7f385b400640 1 --2- 172.21.15.69:0/3067835888 >> v2:172.21.15.121:3300/0 conn(0x7f385c0d0840 0x7f385c0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.105+0000 7f385be00640 1 -- 172.21.15.69:0/3067835888 >> v2:172.21.15.121:3300/0 conn(0x7f385c0d0840 msgr2=0x7f385c0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:27.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.105+0000 7f385be00640 1 --2- 172.21.15.69:0/3067835888 >> v2:172.21.15.121:3300/0 conn(0x7f385c0d0840 0x7f385c0b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.105+0000 7f385be00640 1 -- 172.21.15.69:0/3067835888 >> v1:172.21.15.69:6789/0 conn(0x7f385c0b3b70 legacy=0x7f385c071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.105+0000 7f385be00640 1 -- 172.21.15.69:0/3067835888 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c166ac0 con 0x7f385c162e00 2024-04-24T08:08:27.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.105+0000 7f385b400640 1 --2- 172.21.15.69:0/3067835888 >> v2:172.21.15.121:3300/0 conn(0x7f385c0d0840 0x7f385c0b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:27.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.106+0000 7fb7bda00640 1 Processor -- start 2024-04-24T08:08:27.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.107+0000 7fb7bda00640 1 -- start start 2024-04-24T08:08:27.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.107+0000 7f385be00640 1 --2- 172.21.15.69:0/3067835888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c162e00 0x7f385c0d0300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f385c0b5670 tx=0x7f384802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a0d98e4fd69cc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.108+0000 7f385aa00640 1 -- 172.21.15.69:0/3067835888 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848037070 con 0x7f385c162e00 2024-04-24T08:08:27.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.108+0000 7f385aa00640 1 -- 172.21.15.69:0/3067835888 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f384802ae10 con 0x7f385c162e00 2024-04-24T08:08:27.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.108+0000 7f385aa00640 1 -- 172.21.15.69:0/3067835888 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848037070 con 0x7f385c162e00 2024-04-24T08:08:27.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.109+0000 7f3862c00640 1 -- 172.21.15.69:0/3067835888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c162e00 msgr2=0x7f385c0d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.109+0000 7f3862c00640 1 --2- 172.21.15.69:0/3067835888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c162e00 0x7f385c0d0300 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f385c0b5670 tx=0x7f384802a180 comp rx=0 tx=0).stop 2024-04-24T08:08:27.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.109+0000 7fb7bda00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb7b80d1560 0x7fb7b806d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.109+0000 7fb7bda00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b80b3f60 con 0x7fb7b80d1100 2024-04-24T08:08:27.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.109+0000 7fb7bda00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b80d2750 con 0x7fb7b80b3b80 2024-04-24T08:08:27.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.109+0000 7fb7bda00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb7b80d3360 con 0x7fb7b80d1560 2024-04-24T08:08:27.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.109+0000 7fb7b6a00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb7b80b3b80 0x7fb7b80d0050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:38406/0 (socket says 172.21.15.69:38406) 2024-04-24T08:08:27.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.109+0000 7fb7b6000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb7b80d1100 0x7fb7b806cc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44758/0 (socket says 172.21.15.69:44758) 2024-04-24T08:08:27.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.109+0000 7fb7b6000640 1 -- 172.21.15.69:0/71907810 learned_addr learned my addr 172.21.15.69:0/71907810 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:27.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.110+0000 7fb7b7400640 1 --2- 172.21.15.69:0/71907810 >> v2:172.21.15.69:3300/0 conn(0x7fb7b80d1560 0x7fb7b806d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.110+0000 7fb7b5600640 1 -- 172.21.15.69:0/71907810 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1590308959 0 0) 0x7fb7b80b3f60 con 0x7fb7b80d1100 2024-04-24T08:08:27.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.110+0000 7f3862c00640 1 -- 172.21.15.69:0/3067835888 shutdown_connections 2024-04-24T08:08:27.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.110+0000 7f3862c00640 1 --2- 172.21.15.69:0/3067835888 >> v2:172.21.15.121:3300/0 conn(0x7f385c0d0840 0x7f385c0b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.110+0000 7f3862c00640 1 --2- 172.21.15.69:0/3067835888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c162e00 0x7f385c0d0300 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.110+0000 7f3862c00640 1 -- 172.21.15.69:0/3067835888 >> 172.21.15.69:0/3067835888 conn(0x7f385c067940 msgr2=0x7f385c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.111+0000 7fb7b7400640 1 -- 172.21.15.69:0/71907810 >> v1:172.21.15.121:6789/0 conn(0x7fb7b80b3b80 legacy=0x7fb7b80d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.111+0000 7fb7b7400640 1 -- 172.21.15.69:0/71907810 >> v1:172.21.15.69:6789/0 conn(0x7fb7b80d1100 legacy=0x7fb7b806cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.111+0000 7fb7b7400640 1 -- 172.21.15.69:0/71907810 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b80d2a30 con 0x7fb7b80d1560 2024-04-24T08:08:27.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.112+0000 7f9a40c00640 1 Processor -- start 2024-04-24T08:08:27.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.112+0000 7f9a40c00640 1 -- start start 2024-04-24T08:08:27.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.113+0000 7fb7b7400640 1 --2- 172.21.15.69:0/71907810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80d1560 0x7fb7b806d310 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009d90 tx=0x7fb7b002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a0ae0ba183d601 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.113+0000 7fb7b5600640 1 -- 172.21.15.69:0/71907810 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0007740 con 0x7fb7b80d1560 2024-04-24T08:08:27.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.113+0000 7fb7b5600640 1 -- 172.21.15.69:0/71907810 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b00078e0 con 0x7fb7b80d1560 2024-04-24T08:08:27.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.113+0000 7fb7b5600640 1 -- 172.21.15.69:0/71907810 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b0007cb0 con 0x7fb7b80d1560 2024-04-24T08:08:27.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.113+0000 7f9a40c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9a34053e50 0x7f9a340542e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.113+0000 7f9a40c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9a3404f930 0x7f9a3404fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.114+0000 7f3862c00640 1 -- 172.21.15.69:0/3067835888 shutdown_connections 2024-04-24T08:08:27.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.114+0000 7f9a40c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a34050280 con 0x7f9a34048100 2024-04-24T08:08:27.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.114+0000 7f9a40c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9a340e7b90 con 0x7f9a3404f930 2024-04-24T08:08:27.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.114+0000 7f9a40c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9a34006a90 con 0x7f9a34053e50 2024-04-24T08:08:27.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.114+0000 7f3862c00640 1 -- 172.21.15.69:0/3067835888 wait complete. 2024-04-24T08:08:27.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.114+0000 7f9a3a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9a3404f930 0x7f9a3404fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.114+0000 7f9a3a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9a3404f930 0x7f9a3404fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41436/0 (socket says 172.21.15.69:41436) 2024-04-24T08:08:27.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.114+0000 7f9a3a800640 1 -- 172.21.15.69:0/1993535960 learned_addr learned my addr 172.21.15.69:0/1993535960 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.114+0000 7f9a39400640 1 --2- 172.21.15.69:0/1993535960 >> v2:172.21.15.121:3300/0 conn(0x7f9a34053e50 0x7f9a340542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.115+0000 7fb7bda00640 1 -- 172.21.15.69:0/71907810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80d1560 msgr2=0x7fb7b806d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.115+0000 7fb7bda00640 1 --2- 172.21.15.69:0/71907810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80d1560 0x7fb7b806d310 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009d90 tx=0x7fb7b002a610 comp rx=0 tx=0).stop 2024-04-24T08:08:27.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.115+0000 7fb7bda00640 1 -- 172.21.15.69:0/71907810 shutdown_connections 2024-04-24T08:08:27.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.115+0000 7fb7bda00640 1 --2- 172.21.15.69:0/71907810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80d1560 0x7fb7b806d310 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.115+0000 7fb7bda00640 1 -- 172.21.15.69:0/71907810 >> 172.21.15.69:0/71907810 conn(0x7fb7b808bf60 msgr2=0x7fb7b80d4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.115+0000 7f9a3a800640 1 -- 172.21.15.69:0/1993535960 >> v2:172.21.15.121:3300/0 conn(0x7f9a34053e50 msgr2=0x7f9a340542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:27.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.115+0000 7f9a3a800640 1 --2- 172.21.15.69:0/1993535960 >> v2:172.21.15.121:3300/0 conn(0x7f9a34053e50 0x7f9a340542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.115+0000 7f9a3a800640 1 -- 172.21.15.69:0/1993535960 >> v1:172.21.15.121:6789/0 conn(0x7f9a34048100 legacy=0x7f9a34006f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.115+0000 7f9a3a800640 1 -- 172.21.15.69:0/1993535960 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a34005ec0 con 0x7f9a3404f930 2024-04-24T08:08:27.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.115+0000 7fd4c2200640 1 Processor -- start 2024-04-24T08:08:27.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.116+0000 7fd4c2200640 1 -- start start 2024-04-24T08:08:27.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.116+0000 7fb7bda00640 1 -- 172.21.15.69:0/71907810 shutdown_connections 2024-04-24T08:08:27.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.116+0000 7fb7bda00640 1 -- 172.21.15.69:0/71907810 wait complete. 2024-04-24T08:08:27.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.117+0000 7fac52e00640 1 Processor -- start 2024-04-24T08:08:27.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.117+0000 7fd4c2200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4bc163160 0x7fd4bc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.117+0000 7fd4c2200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd4bc0b66f0 0x7fd4bc06cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.117+0000 7fd4c2200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4bc0d23b0 con 0x7fd4bc162d80 2024-04-24T08:08:27.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.117+0000 7fd4c2200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd4bc0d2f80 con 0x7fd4bc0b66f0 2024-04-24T08:08:27.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.117+0000 7fd4c2200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd4bc0d2650 con 0x7fd4bc163160 2024-04-24T08:08:27.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.118+0000 7fac52e00640 1 -- start start 2024-04-24T08:08:27.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.118+0000 7f3862c00640 1 Processor -- start 2024-04-24T08:08:27.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.118+0000 7fd4bb400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd4bc162d80 0x7fd4bc0d3410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44772/0 (socket says 172.21.15.69:44772) 2024-04-24T08:08:27.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.118+0000 7fd4bb400640 1 -- 172.21.15.69:0/1779277251 learned_addr learned my addr 172.21.15.69:0/1779277251 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:27.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.118+0000 7f9a3a800640 1 --2- 172.21.15.69:0/1993535960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a3404f930 0x7f9a3404fd40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c069ac0 tx=0x7f9a3c0a5dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4eb321a86a9efb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.119+0000 7f9a38a00640 1 -- 172.21.15.69:0/1993535960 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0b3070 con 0x7f9a3404f930 2024-04-24T08:08:27.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.119+0000 7f9a38a00640 1 -- 172.21.15.69:0/1993535960 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a3c0a6c10 con 0x7f9a3404f930 2024-04-24T08:08:27.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.119+0000 7f9a38a00640 1 -- 172.21.15.69:0/1993535960 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0ae640 con 0x7f9a3404f930 2024-04-24T08:08:27.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.119+0000 7fd4bbe00640 1 --2- 172.21.15.69:0/1779277251 >> v2:172.21.15.121:3300/0 conn(0x7fd4bc0b66f0 0x7fd4bc06cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f9a40c00640 1 -- 172.21.15.69:0/1993535960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a3404f930 msgr2=0x7f9a3404fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f9a40c00640 1 --2- 172.21.15.69:0/1993535960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a3404f930 0x7f9a3404fd40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c069ac0 tx=0x7f9a3c0a5dc0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.119+0000 7fd4ba000640 1 -- 172.21.15.69:0/1779277251 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1559695187 0 0) 0x7fd4bc0d23b0 con 0x7fd4bc162d80 2024-04-24T08:08:27.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7fd4ba000640 1 -- 172.21.15.69:0/1779277251 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4a0003520 con 0x7fd4bc162d80 2024-04-24T08:08:27.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.119+0000 7f3862c00640 1 -- start start 2024-04-24T08:08:27.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f3862c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c0b3b70 0x7f385c1ec7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f3862c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f385c0d0840 0x7f385c1652c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f3862c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f385c14a850 0x7f385c165800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f3862c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f385c167360 con 0x7f385c0b3b70 2024-04-24T08:08:27.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f3862c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f385c167080 con 0x7f385c14a850 2024-04-24T08:08:27.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7fd4baa00640 1 --2- 172.21.15.69:0/1779277251 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4bc163160 0x7fd4bc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7fac52e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 0x7fac4c0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f3862c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f385c0b4360 con 0x7f385c0d0840 2024-04-24T08:08:27.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7fac52e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fac4c0b3d30 0x7fac4c0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f385be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c0b3b70 0x7f385c1ec7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7fb7bda00640 1 Processor -- start 2024-04-24T08:08:27.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f9a40c00640 1 -- 172.21.15.69:0/1993535960 shutdown_connections 2024-04-24T08:08:27.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f9a40c00640 1 --2- 172.21.15.69:0/1993535960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a3404f930 0x7f9a3404fd40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f9a40c00640 1 --2- 172.21.15.69:0/1993535960 >> v2:172.21.15.121:3300/0 conn(0x7f9a34053e50 0x7f9a340542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7f9a40c00640 1 -- 172.21.15.69:0/1993535960 >> 172.21.15.69:0/1993535960 conn(0x7f9a34009fe0 msgr2=0x7f9a3400a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7f385be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c0b3b70 0x7f385c1ec7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41444/0 (socket says 172.21.15.69:41444) 2024-04-24T08:08:27.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7f385be00640 1 -- 172.21.15.69:0/3326201556 learned_addr learned my addr 172.21.15.69:0/3326201556 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7f385b400640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f385c0d0840 0x7f385c1652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7f9a40c00640 1 -- 172.21.15.69:0/1993535960 shutdown_connections 2024-04-24T08:08:27.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7fb7bda00640 1 -- start start 2024-04-24T08:08:27.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7fac4be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 0x7fac4c0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7fac52e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c1669d0 con 0x7fac4c071d70 2024-04-24T08:08:27.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7fac4be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 0x7fac4c0b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:47374/0 (socket says 172.21.15.69:47374) 2024-04-24T08:08:27.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7fac4be00640 1 -- 172.21.15.69:0/4168597601 learned_addr learned my addr 172.21.15.69:0/4168597601 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7fac52e00640 1 -- 172.21.15.69:0/4168597601 --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fac4c0b46d0 con 0x7fac4c0b3d30 2024-04-24T08:08:27.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7fac52e00640 1 -- 172.21.15.69:0/4168597601 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c166cb0 con 0x7fac4c162c70 2024-04-24T08:08:27.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7fd4bbe00640 1 -- 172.21.15.69:0/1779277251 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4bc163160 msgr2=0x7fd4bc0b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:27.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7fac4b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fac4c0b3d30 0x7fac4c0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.122+0000 7f3860c00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f385c14a850 0x7f385c165800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7fd4bbe00640 1 --2- 172.21.15.69:0/1779277251 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4bc163160 0x7fd4bc0b61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7fd4bbe00640 1 -- 172.21.15.69:0/1779277251 >> v1:172.21.15.69:6789/0 conn(0x7fd4bc162d80 legacy=0x7fd4bc0d3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.120+0000 7fd4bbe00640 1 -- 172.21.15.69:0/1779277251 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4bc0d2930 con 0x7fd4bc0b66f0 2024-04-24T08:08:27.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.121+0000 7f9a40c00640 1 -- 172.21.15.69:0/1993535960 wait complete. 2024-04-24T08:08:27.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.122+0000 7f385be00640 1 -- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f385c0d0840 msgr2=0x7f385c1652c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.122+0000 7f385be00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f385c0d0840 0x7f385c1652c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.122+0000 7f385be00640 1 -- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f385c14a850 msgr2=0x7f385c165800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.122+0000 7f385be00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f385c14a850 0x7f385c165800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.122+0000 7f385be00640 1 -- 172.21.15.69:0/3326201556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385c1ed0c0 con 0x7f385c0b3b70 2024-04-24T08:08:27.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.122+0000 7fd4bbe00640 1 --2- 172.21.15.69:0/1779277251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4bc0b66f0 0x7fd4bc06cb10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009d90 tx=0x7fd4b402a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=201b3ea5b8ef7092 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.123+0000 7fd4ba000640 1 -- 172.21.15.69:0/1779277251 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4007740 con 0x7fd4bc0b66f0 2024-04-24T08:08:27.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.123+0000 7fd4ba000640 1 -- 172.21.15.69:0/1779277251 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4b40078e0 con 0x7fd4bc0b66f0 2024-04-24T08:08:27.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.123+0000 7fd4ba000640 1 -- 172.21.15.69:0/1779277251 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4007cb0 con 0x7fd4bc0b66f0 2024-04-24T08:08:27.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.124+0000 7fb7bda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80b3b80 0x7fb7b80ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.124+0000 7fb7bda00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b80d1100 0x7fb7b81f89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.124+0000 7f9a40c00640 1 Processor -- start 2024-04-24T08:08:27.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.123+0000 7fac4aa00640 1 -- 172.21.15.69:0/4168597601 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4096867653 0 0) 0x7fac4c1669d0 con 0x7fac4c071d70 2024-04-24T08:08:27.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.123+0000 7fac4aa00640 1 -- 172.21.15.69:0/4168597601 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac30003520 con 0x7fac4c071d70 2024-04-24T08:08:27.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.123+0000 7fac4b400640 1 -- 172.21.15.69:0/4168597601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 msgr2=0x7fac4c0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:27.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.123+0000 7fac4b400640 1 --2- 172.21.15.69:0/4168597601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 0x7fac4c0b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.123+0000 7fac4b400640 1 -- 172.21.15.69:0/4168597601 >> v1:172.21.15.121:6789/0 conn(0x7fac4c071d70 legacy=0x7fac4c0721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.123+0000 7fac4b400640 1 -- 172.21.15.69:0/4168597601 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c166f90 con 0x7fac4c0b3d30 2024-04-24T08:08:27.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7f9a40c00640 1 -- start start 2024-04-24T08:08:27.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.124+0000 7fb7bda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b80d1560 0x7fb7b815fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.124+0000 7fb7bda00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b80d0760 con 0x7fb7b80b3b80 2024-04-24T08:08:27.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.124+0000 7fb7bda00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b80d2ff0 con 0x7fb7b80d1100 2024-04-24T08:08:27.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.124+0000 7fb7bda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b80d19f0 con 0x7fb7b80d1560 2024-04-24T08:08:27.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7fb7b6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80b3b80 0x7fb7b80ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7fb7b6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80b3b80 0x7fb7b80ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41466/0 (socket says 172.21.15.69:41466) 2024-04-24T08:08:27.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7fb7b6a00640 1 -- 172.21.15.69:0/2406298476 learned_addr learned my addr 172.21.15.69:0/2406298476 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.124+0000 7f385be00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c0b3b70 0x7f385c1ec7c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3848000c00 tx=0x7f38480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7f3853e00640 1 -- 172.21.15.69:0/3326201556 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848037070 con 0x7f385c0b3b70 2024-04-24T08:08:27.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.124+0000 7fd4c2200640 1 -- 172.21.15.69:0/1779277251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4bc0b66f0 msgr2=0x7fd4bc06cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.124+0000 7fd4c2200640 1 --2- 172.21.15.69:0/1779277251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4bc0b66f0 0x7fd4bc06cb10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4009d90 tx=0x7fd4b402a660 comp rx=0 tx=0).stop 2024-04-24T08:08:27.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7fd4c2200640 1 -- 172.21.15.69:0/1779277251 shutdown_connections 2024-04-24T08:08:27.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7fd4c2200640 1 --2- 172.21.15.69:0/1779277251 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4bc0b66f0 0x7fd4bc06cb10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7fd4c2200640 1 --2- 172.21.15.69:0/1779277251 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4bc163160 0x7fd4bc0b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7fd4c2200640 1 -- 172.21.15.69:0/1779277251 >> 172.21.15.69:0/1779277251 conn(0x7fd4bc0d11b0 msgr2=0x7fd4bc0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7f7923400640 1 Processor -- start 2024-04-24T08:08:27.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7f3853e00640 1 -- 172.21.15.69:0/3326201556 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3848004030 con 0x7f385c0b3b70 2024-04-24T08:08:27.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7f3853e00640 1 -- 172.21.15.69:0/3326201556 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848037070 con 0x7f385c0b3b70 2024-04-24T08:08:27.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7fac4b400640 1 --2- 172.21.15.69:0/4168597601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 0x7fac4c0b4190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0097a0 tx=0x7fac3c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e2f2b94dd27de7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7fac4aa00640 1 -- 172.21.15.69:0/4168597601 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c037070 con 0x7fac4c0b3d30 2024-04-24T08:08:27.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7fd4c2200640 1 -- 172.21.15.69:0/1779277251 shutdown_connections 2024-04-24T08:08:27.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fb7b6a00640 1 -- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b80d1560 msgr2=0x7fb7b815fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fb7b6a00640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b80d1560 0x7fb7b815fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fb7b6a00640 1 -- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b80d1100 msgr2=0x7fb7b81f89a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T08:08:27.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fb7b6a00640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b80d1100 0x7fb7b81f89a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fb7b6a00640 1 -- 172.21.15.69:0/2406298476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b81f8ee0 con 0x7fb7b80b3b80 2024-04-24T08:08:27.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fd4c2200640 1 -- 172.21.15.69:0/1779277251 wait complete. 2024-04-24T08:08:27.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fac4aa00640 1 -- 172.21.15.69:0/4168597601 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac3c02ae40 con 0x7fac4c0b3d30 2024-04-24T08:08:27.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fac4aa00640 1 -- 172.21.15.69:0/4168597601 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c032660 con 0x7fac4c0b3d30 2024-04-24T08:08:27.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fb7b6a00640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80b3b80 0x7fb7b80ce8e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800c7b0 tx=0x7fb7a800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fb7aea00640 1 -- 172.21.15.69:0/2406298476 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8018070 con 0x7fb7b80b3b80 2024-04-24T08:08:27.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fac52e00640 1 -- 172.21.15.69:0/4168597601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 msgr2=0x7fac4c0b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.126+0000 7fac52e00640 1 --2- 172.21.15.69:0/4168597601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 0x7fac4c0b4190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0097a0 tx=0x7fac3c02a130 comp rx=0 tx=0).stop 2024-04-24T08:08:27.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fb7bda00640 1 -- 172.21.15.69:0/2406298476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b80c8020 con 0x7fb7b80b3b80 2024-04-24T08:08:27.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fac52e00640 1 -- 172.21.15.69:0/4168597601 shutdown_connections 2024-04-24T08:08:27.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fac52e00640 1 --2- 172.21.15.69:0/4168597601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 0x7fac4c0b4190 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fac52e00640 1 --2- 172.21.15.69:0/4168597601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 0x7fac4c0b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fac52e00640 1 -- 172.21.15.69:0/4168597601 >> 172.21.15.69:0/4168597601 conn(0x7fac4c067d90 msgr2=0x7fac4c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fac52e00640 1 -- 172.21.15.69:0/4168597601 shutdown_connections 2024-04-24T08:08:27.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fac52e00640 1 -- 172.21.15.69:0/4168597601 wait complete. 2024-04-24T08:08:27.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.125+0000 7f7923400640 1 -- start start 2024-04-24T08:08:27.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7f7923400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f791c0b3d30 0x7f791c0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7f7923400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 0x7f791c06e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7f7923400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f791c1669d0 con 0x7f791c162c70 2024-04-24T08:08:27.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7f7923400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f791c166cb0 con 0x7f791c0b3d30 2024-04-24T08:08:27.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fb7aea00640 1 -- 172.21.15.69:0/2406298476 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a80072d0 con 0x7fb7b80b3b80 2024-04-24T08:08:27.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7f7923400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f791c166f90 con 0x7f791c06e0f0 2024-04-24T08:08:27.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fbae4a00640 1 Processor -- start 2024-04-24T08:08:27.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fb7bda00640 1 -- 172.21.15.69:0/2406298476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b81f9460 con 0x7fb7b80b3b80 2024-04-24T08:08:27.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.127+0000 7fb7aea00640 1 -- 172.21.15.69:0/2406298476 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8007680 con 0x7fb7b80b3b80 2024-04-24T08:08:27.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f7921400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 0x7f791c06e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f3862c00640 1 -- 172.21.15.69:0/3326201556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f385c1ed3a0 con 0x7f385c0b3b70 2024-04-24T08:08:27.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f9a40c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a34048100 0x7f9a34172150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f9a40c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a3404f930 0x7f9a340dcee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f9a40c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a34053e50 0x7f9a34049760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f7913e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f791c0b3d30 0x7f791c0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f7913e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f791c0b3d30 0x7f791c0b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:42534/0 (socket says 172.21.15.69:42534) 2024-04-24T08:08:27.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f7913e00640 1 -- 172.21.15.69:0/4284847434 learned_addr learned my addr 172.21.15.69:0/4284847434 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f7920a00640 1 --1- 172.21.15.69:0/4284847434 >> v1:172.21.15.69:6789/0 conn(0x7f791c162c70 0x7f791c0b3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44786/0 (socket says 172.21.15.69:44786) 2024-04-24T08:08:27.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f3862c00640 1 -- 172.21.15.69:0/3326201556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f385c1ed960 con 0x7f385c0b3b70 2024-04-24T08:08:27.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f9a40c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34006440 con 0x7f9a3404f930 2024-04-24T08:08:27.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f9a40c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9a34006160 con 0x7f9a34053e50 2024-04-24T08:08:27.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7f9a40c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a34006720 con 0x7f9a34048100 2024-04-24T08:08:27.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.128+0000 7fbae4a00640 1 -- start start 2024-04-24T08:08:27.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.129+0000 7f9a3a800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a34053e50 0x7f9a34049760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.129+0000 7f9a3a800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a34053e50 0x7f9a34049760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:42538/0 (socket says 172.21.15.69:42538) 2024-04-24T08:08:27.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.129+0000 7f9a3a800640 1 -- 172.21.15.69:0/3832636318 learned_addr learned my addr 172.21.15.69:0/3832636318 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.129+0000 7f9a3a800640 1 -- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a34048100 msgr2=0x7f9a34172150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.129+0000 7f9a3a800640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a34048100 0x7f9a34172150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.129+0000 7f9a3a800640 1 -- 172.21.15.69:0/3832636318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a3404f930 msgr2=0x7f9a340dcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.130+0000 7f3862c00640 1 -- 172.21.15.69:0/3326201556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f385c068a80 con 0x7f385c0b3b70 2024-04-24T08:08:27.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.130+0000 7fbae4a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 0x7fbae00d1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.130+0000 7fbae4a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbae015ed80 0x7fbae015f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.130+0000 7fbae4a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbae015fb20 con 0x7fbae00d2740 2024-04-24T08:08:27.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.130+0000 7fbae4a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbae00b6400 con 0x7fbae015ed80 2024-04-24T08:08:27.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.130+0000 7fbae4a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbae00d10b0 con 0x7fbae015e9a0 2024-04-24T08:08:27.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.130+0000 7f9a39400640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a3404f930 0x7f9a340dcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.130+0000 7f9a3a800640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a3404f930 0x7f9a340dcee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.130+0000 7f9a3a800640 1 -- 172.21.15.69:0/3832636318 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a3404a040 con 0x7f9a34053e50 2024-04-24T08:08:27.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.131+0000 7fbadd400640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fbae00d2740 0x7fbae00d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:38430/0 (socket says 172.21.15.69:38430) 2024-04-24T08:08:27.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.131+0000 7fbadd400640 1 -- 172.21.15.69:0/780534749 learned_addr learned my addr 172.21.15.69:0/780534749 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:27.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.131+0000 7fd4c2200640 1 Processor -- start 2024-04-24T08:08:27.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.129+0000 7f7913e00640 1 -- 172.21.15.69:0/4284847434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 msgr2=0x7f791c06e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:27.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.129+0000 7f7913e00640 1 --2- 172.21.15.69:0/4284847434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 0x7f791c06e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.131+0000 7fac52e00640 1 Processor -- start 2024-04-24T08:08:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.129+0000 7f7913e00640 1 -- 172.21.15.69:0/4284847434 >> v1:172.21.15.69:6789/0 conn(0x7f791c162c70 legacy=0x7f791c0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.129+0000 7f7913e00640 1 -- 172.21.15.69:0/4284847434 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791c167270 con 0x7f791c0b3d30 2024-04-24T08:08:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.131+0000 7fbadde00640 1 --2- 172.21.15.69:0/780534749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 0x7fbae00d1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.132+0000 7fbade800640 1 --2- 172.21.15.69:0/780534749 >> v2:172.21.15.69:3300/0 conn(0x7fbae015ed80 0x7fbae015f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.132+0000 7f9a3a800640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a34053e50 0x7f9a34049760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0699a0 tx=0x7f9a3c076b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.132+0000 7f7921400640 1 --2- 172.21.15.69:0/4284847434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 0x7f791c06e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:27.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.132+0000 7fac52e00640 1 -- start start 2024-04-24T08:08:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.132+0000 7fbadca00640 1 -- 172.21.15.69:0/780534749 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1610818819 0 0) 0x7fbae015fb20 con 0x7fbae00d2740 2024-04-24T08:08:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.132+0000 7fbadca00640 1 -- 172.21.15.69:0/780534749 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbac4003520 con 0x7fbae00d2740 2024-04-24T08:08:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.132+0000 7f7913e00640 1 --2- 172.21.15.69:0/4284847434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f791c0b3d30 0x7f791c0b41c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7904009870 tx=0x7f790402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=850bc75bfc6e089c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.132+0000 7f9a32000640 1 -- 172.21.15.69:0/3832636318 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0b3050 con 0x7f9a34053e50 2024-04-24T08:08:27.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.132+0000 7f9a32000640 1 -- 172.21.15.69:0/3832636318 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a3c0b4040 con 0x7f9a34053e50 2024-04-24T08:08:27.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.132+0000 7f9a32000640 1 -- 172.21.15.69:0/3832636318 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a3c0b8810 con 0x7f9a34053e50 2024-04-24T08:08:27.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7f791be00640 1 -- 172.21.15.69:0/4284847434 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904037070 con 0x7f791c0b3d30 2024-04-24T08:08:27.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7f791be00640 1 -- 172.21.15.69:0/4284847434 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7904032440 con 0x7f791c0b3d30 2024-04-24T08:08:27.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7f791be00640 1 -- 172.21.15.69:0/4284847434 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904032810 con 0x7f791c0b3d30 2024-04-24T08:08:27.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7fbade800640 1 -- 172.21.15.69:0/780534749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 msgr2=0x7fbae00d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:27.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7fbade800640 1 --2- 172.21.15.69:0/780534749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 0x7fbae00d1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7fbade800640 1 -- 172.21.15.69:0/780534749 >> v1:172.21.15.121:6789/0 conn(0x7fbae00d2740 legacy=0x7fbae00d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7fbade800640 1 -- 172.21.15.69:0/780534749 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae00d1390 con 0x7fbae015ed80 2024-04-24T08:08:27.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7fd4c2200640 1 -- start start 2024-04-24T08:08:27.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7fbadde00640 1 --2- 172.21.15.69:0/780534749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 0x7fbae00d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:27.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7f9a40c00640 1 -- 172.21.15.69:0/3832636318 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a3404a320 con 0x7f9a34053e50 2024-04-24T08:08:27.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.133+0000 7f9a40c00640 1 -- 172.21.15.69:0/3832636318 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a3404a8e0 con 0x7f9a34053e50 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7fac52e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c071d70 0x7fac4c154a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7fac52e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 0x7fac4c156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7fac52e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 0x7fac4c163790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7fac52e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c163050 con 0x7fac4c0b3d30 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7fac52e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c167550 con 0x7fac4c071d70 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7fac52e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c0cfc40 con 0x7fac4c162c70 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7f7923400640 1 -- 172.21.15.69:0/4284847434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f791c0b3d30 msgr2=0x7f791c0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7f7923400640 1 --2- 172.21.15.69:0/4284847434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f791c0b3d30 0x7f791c0b41c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7904009870 tx=0x7f790402a270 comp rx=0 tx=0).stop 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7f7923400640 1 -- 172.21.15.69:0/4284847434 shutdown_connections 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7f7923400640 1 --2- 172.21.15.69:0/4284847434 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 0x7f791c06e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7f7923400640 1 --2- 172.21.15.69:0/4284847434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f791c0b3d30 0x7f791c0b41c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7f7923400640 1 -- 172.21.15.69:0/4284847434 >> 172.21.15.69:0/4284847434 conn(0x7f791c067d90 msgr2=0x7f791c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.134+0000 7fb7bda00640 1 -- 172.21.15.69:0/2406298476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb790005290 con 0x7fb7b80b3b80 2024-04-24T08:08:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7fac4b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 0x7fac4c156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7fac4be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c071d70 0x7fac4c154a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7fac4b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 0x7fac4c156240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41496/0 (socket says 172.21.15.69:41496) 2024-04-24T08:08:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7fac4b400640 1 -- 172.21.15.69:0/2872868632 learned_addr learned my addr 172.21.15.69:0/2872868632 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7fac50e00640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 0x7fac4c163790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7f7923400640 1 -- 172.21.15.69:0/4284847434 shutdown_connections 2024-04-24T08:08:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7fac4b400640 1 -- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 msgr2=0x7fac4c163790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7fac4b400640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 0x7fac4c163790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7fac4b400640 1 -- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c071d70 msgr2=0x7fac4c154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7fac4b400640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c071d70 0x7fac4c154a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7fac4b400640 1 -- 172.21.15.69:0/2872868632 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c0b0450 con 0x7fac4c0b3d30 2024-04-24T08:08:27.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.135+0000 7f7923400640 1 -- 172.21.15.69:0/4284847434 wait complete. 2024-04-24T08:08:27.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.136+0000 7f3853e00640 1 -- 172.21.15.69:0/3326201556 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f38480041e0 con 0x7f385c0b3b70 2024-04-24T08:08:27.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.136+0000 7fac4b400640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 0x7fac4c156240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fac3c035040 tx=0x7fac3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.136+0000 7fd4c2200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4bc0b66f0 0x7fd4bc0c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.136+0000 7fbade800640 1 --2- 172.21.15.69:0/780534749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 0x7fbae015f1f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009d90 tx=0x7fbad802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ca19f957b9aad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.136+0000 7fd4c2200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4bc162d80 0x7fd4bc1625d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.136+0000 7fd4c2200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4bc163160 0x7fd4bc1fc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.136+0000 7fd4c2200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd4bc06d1d0 con 0x7fd4bc162d80 2024-04-24T08:08:27.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.136+0000 7fd4c2200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd4bc06d050 con 0x7fd4bc163160 2024-04-24T08:08:27.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.136+0000 7fd4c2200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd4bc06d350 con 0x7fd4bc0b66f0 2024-04-24T08:08:27.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.137+0000 7fd4baa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4bc162d80 0x7fd4bc1625d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.137+0000 7fd4baa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4bc162d80 0x7fd4bc1625d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41504/0 (socket says 172.21.15.69:41504) 2024-04-24T08:08:27.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.137+0000 7fd4baa00640 1 -- 172.21.15.69:0/4257420855 learned_addr learned my addr 172.21.15.69:0/4257420855 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.138+0000 7fd4bbe00640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4bc163160 0x7fd4bc1fc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.139+0000 7f7923400640 1 Processor -- start 2024-04-24T08:08:27.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.139+0000 7f7923400640 1 -- start start 2024-04-24T08:08:27.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.138+0000 7f9a32000640 1 -- 172.21.15.69:0/3832636318 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a3c0c1020 con 0x7f9a34053e50 2024-04-24T08:08:27.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.138+0000 7f9a40c00640 1 -- 172.21.15.69:0/3832636318 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a14005290 con 0x7f9a34053e50 2024-04-24T08:08:27.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.138+0000 7fd4bbe00640 1 -- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4bc0b66f0 msgr2=0x7fd4bc0c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.138+0000 7fd4bbe00640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4bc0b66f0 0x7fd4bc0c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.137+0000 7fb7aea00640 1 -- 172.21.15.69:0/2406298476 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb7a801b070 con 0x7fb7b80b3b80 2024-04-24T08:08:27.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.138+0000 7fb7aea00640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb79806ca90 0x7fb79806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.139+0000 7fb7aea00640 1 -- 172.21.15.69:0/2406298476 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb7a808af40 con 0x7fb7b80b3b80 2024-04-24T08:08:27.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.139+0000 7f3853e00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f383c06cb20 0x7f383c06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.139+0000 7f3853e00640 1 -- 172.21.15.69:0/3326201556 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f38480abdf0 con 0x7f385c0b3b70 2024-04-24T08:08:27.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.139+0000 7f9a32000640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1c06c970 0x7f9a1c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.138+0000 7fd4bbe00640 1 -- 172.21.15.69:0/4257420855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4bc162d80 msgr2=0x7fd4bc1625d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f9a32000640 1 -- 172.21.15.69:0/3832636318 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9a3c127650 con 0x7f9a34053e50 2024-04-24T08:08:27.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.138+0000 7fd4bbe00640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4bc162d80 0x7fd4bc1625d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.139+0000 7fd4bbe00640 1 -- 172.21.15.69:0/4257420855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4bc1fcc30 con 0x7fd4bc163160 2024-04-24T08:08:27.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f7923400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 0x7f791c148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f7923400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f791c0b3d30 0x7f791c1492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f385b400640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f383c06cb20 0x7f383c06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f7923400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f791c162c70 0x7f791c0cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7fd4bbe00640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4bc163160 0x7fd4bc1fc1e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4031000 tx=0x7fd4b4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f7923400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f791c0cfc40 con 0x7f791c0b3d30 2024-04-24T08:08:27.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f7923400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f791c163050 con 0x7f791c162c70 2024-04-24T08:08:27.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f7923400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f791c0cfdc0 con 0x7f791c06e0f0 2024-04-24T08:08:27.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7fb7b6000640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb79806ca90 0x7fb79806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f7920a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 0x7f791c148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f7920a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 0x7f791c148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:47434/0 (socket says 172.21.15.69:47434) 2024-04-24T08:08:27.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.140+0000 7f7920a00640 1 -- 172.21.15.69:0/4083891309 learned_addr learned my addr 172.21.15.69:0/4083891309 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.141+0000 7f9a39e00640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1c06c970 0x7f9a1c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.141+0000 7fac43e00640 1 -- 172.21.15.69:0/2872868632 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c041070 con 0x7fac4c0b3d30 2024-04-24T08:08:27.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.141+0000 7f7921400640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f791c162c70 0x7f791c0cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.141+0000 7f7913e00640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f791c0b3d30 0x7f791c1492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.141+0000 7f3853e00640 1 -- 172.21.15.69:0/3326201556 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3848042020 con 0x7f385c0b3b70 2024-04-24T08:08:27.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.142+0000 7f7913e00640 1 -- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 msgr2=0x7f791c148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.142+0000 7f7913e00640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 0x7f791c148d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.142+0000 7f7913e00640 1 -- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f791c162c70 msgr2=0x7f791c0cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.142+0000 7f7913e00640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f791c162c70 0x7f791c0cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.142+0000 7f7913e00640 1 -- 172.21.15.69:0/4083891309 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791c1479c0 con 0x7f791c0b3d30 2024-04-24T08:08:27.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.142+0000 7fb7b6000640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb79806ca90 0x7fb79806ef50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4007fd0 tx=0x7fb7a4007920 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.142+0000 7fac52e00640 1 -- 172.21.15.69:0/2872868632 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4c0b0730 con 0x7fac4c0b3d30 2024-04-24T08:08:27.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.142+0000 7f385b400640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f383c06cb20 0x7f383c06efe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3854009bd0 tx=0x7f385400b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.142+0000 7fac52e00640 1 -- 172.21.15.69:0/2872868632 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac4c0b0cf0 con 0x7fac4c0b3d30 2024-04-24T08:08:27.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.143+0000 7f6763200640 1 Processor -- start 2024-04-24T08:08:27.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.142+0000 7f7913e00640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f791c0b3d30 0x7f791c1492c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7904031000 tx=0x7f79040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.143+0000 7f6763200640 1 -- start start 2024-04-24T08:08:27.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.143+0000 7f9a39e00640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1c06c970 0x7f9a1c06ee30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c000f60 tx=0x7f9a2c0079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.144+0000 7fac52e00640 1 -- 172.21.15.69:0/2872868632 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac4c068a80 con 0x7fac4c0b3d30 2024-04-24T08:08:27.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.144+0000 7f6763200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676406db90 0x7f6764073530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.144+0000 7f6763200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f676415b980 0x7f676415be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.144+0000 7f6763200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6764073a70 con 0x7f67641523f0 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.144+0000 7f6763200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f67640aeac0 con 0x7f676415b980 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.144+0000 7f6763200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f67640aeda0 con 0x7f676406db90 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.144+0000 7f6761e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f67641523f0 0x7f6764072e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44788/0 (socket says 172.21.15.69:44788) 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.144+0000 7f6761e00640 1 -- 172.21.15.69:0/4075820858 learned_addr learned my addr 172.21.15.69:0/4075820858 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.145+0000 7f6761400640 1 --2- 172.21.15.69:0/4075820858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676406db90 0x7f6764073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.145+0000 7f6762800640 1 --2- 172.21.15.69:0/4075820858 >> v2:172.21.15.121:3300/0 conn(0x7f676415b980 0x7f676415be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.146+0000 7f6761400640 1 -- 172.21.15.69:0/4075820858 >> v2:172.21.15.121:3300/0 conn(0x7f676415b980 msgr2=0x7f676415be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.146+0000 7f6761400640 1 --2- 172.21.15.69:0/4075820858 >> v2:172.21.15.121:3300/0 conn(0x7f676415b980 0x7f676415be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.146+0000 7f6761400640 1 -- 172.21.15.69:0/4075820858 >> v1:172.21.15.69:6789/0 conn(0x7f67641523f0 legacy=0x7f6764072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.146+0000 7f6761400640 1 -- 172.21.15.69:0/4075820858 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67640af080 con 0x7f676406db90 2024-04-24T08:08:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.147+0000 7f6761400640 1 --2- 172.21.15.69:0/4075820858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676406db90 0x7f6764073530 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f675000d7b0 tx=0x7f675002c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cbd391222e9632b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7f7919600640 1 -- 172.21.15.69:0/4083891309 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904041020 con 0x7f791c0b3d30 2024-04-24T08:08:27.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7f7923400640 1 -- 172.21.15.69:0/4083891309 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791c147c60 con 0x7f791c0b3d30 2024-04-24T08:08:27.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7f7923400640 1 -- 172.21.15.69:0/4083891309 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f791c148220 con 0x7f791c0b3d30 2024-04-24T08:08:27.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7fbadca00640 1 -- 172.21.15.69:0/780534749 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad8031600 con 0x7fbae015ed80 2024-04-24T08:08:27.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7fbadca00640 1 -- 172.21.15.69:0/780534749 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad80317a0 con 0x7fbae015ed80 2024-04-24T08:08:27.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7fbadca00640 1 -- 172.21.15.69:0/780534749 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad8031b70 con 0x7fbae015ed80 2024-04-24T08:08:27.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7fac43e00640 1 -- 172.21.15.69:0/2872868632 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac3c03c460 con 0x7fac4c0b3d30 2024-04-24T08:08:27.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7fac43e00640 1 -- 172.21.15.69:0/2872868632 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c037040 con 0x7fac4c0b3d30 2024-04-24T08:08:27.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.149+0000 7fb7aea00640 1 -- 172.21.15.69:0/2406298476 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7a8059360 con 0x7fb7b80b3b80 2024-04-24T08:08:27.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7f6760a00640 1 -- 172.21.15.69:0/4075820858 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6750038070 con 0x7f676406db90 2024-04-24T08:08:27.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7f6760a00640 1 -- 172.21.15.69:0/4075820858 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6750031710 con 0x7f676406db90 2024-04-24T08:08:27.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.148+0000 7f6760a00640 1 -- 172.21.15.69:0/4075820858 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6750031ae0 con 0x7f676406db90 2024-04-24T08:08:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.149+0000 7fbae4a00640 1 -- 172.21.15.69:0/780534749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 msgr2=0x7fbae015f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.149+0000 7fbae4a00640 1 --2- 172.21.15.69:0/780534749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 0x7fbae015f1f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009d90 tx=0x7fbad802a610 comp rx=0 tx=0).stop 2024-04-24T08:08:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.149+0000 7fbae4a00640 1 -- 172.21.15.69:0/780534749 shutdown_connections 2024-04-24T08:08:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.149+0000 7fbae4a00640 1 --2- 172.21.15.69:0/780534749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 0x7fbae015f1f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.149+0000 7fbae4a00640 1 --2- 172.21.15.69:0/780534749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 0x7fbae00d1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.149+0000 7fbae4a00640 1 -- 172.21.15.69:0/780534749 >> 172.21.15.69:0/780534749 conn(0x7fbae008bf60 msgr2=0x7fbae009d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.150+0000 7f7919600640 1 -- 172.21.15.69:0/4083891309 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7904032440 con 0x7f791c0b3d30 2024-04-24T08:08:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.150+0000 7f7919600640 1 -- 172.21.15.69:0/4083891309 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904037040 con 0x7f791c0b3d30 2024-04-24T08:08:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.150+0000 7f6763200640 1 -- 172.21.15.69:0/4075820858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676406db90 msgr2=0x7f6764073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.150+0000 7f6763200640 1 --2- 172.21.15.69:0/4075820858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676406db90 0x7f6764073530 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f675000d7b0 tx=0x7f675002c530 comp rx=0 tx=0).stop 2024-04-24T08:08:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.150+0000 7f6763200640 1 -- 172.21.15.69:0/4075820858 shutdown_connections 2024-04-24T08:08:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.150+0000 7f6763200640 1 --2- 172.21.15.69:0/4075820858 >> v2:172.21.15.121:3300/0 conn(0x7f676415b980 0x7f676415be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.150+0000 7f6763200640 1 --2- 172.21.15.69:0/4075820858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676406db90 0x7f6764073530 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.151+0000 7f6763200640 1 -- 172.21.15.69:0/4075820858 >> 172.21.15.69:0/4075820858 conn(0x7f67640d0160 msgr2=0x7f67640d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.151+0000 7fbae4a00640 1 -- 172.21.15.69:0/780534749 shutdown_connections 2024-04-24T08:08:27.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.151+0000 7f7923400640 1 -- 172.21.15.69:0/4083891309 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f4005290 con 0x7f791c0b3d30 2024-04-24T08:08:27.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.152+0000 7fac43e00640 1 -- 172.21.15.69:0/2872868632 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fac3c03a070 con 0x7fac4c0b3d30 2024-04-24T08:08:27.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.153+0000 7f6763200640 1 -- 172.21.15.69:0/4075820858 shutdown_connections 2024-04-24T08:08:27.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.153+0000 7f6763200640 1 -- 172.21.15.69:0/4075820858 wait complete. 2024-04-24T08:08:27.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.151+0000 7fbae4a00640 1 -- 172.21.15.69:0/780534749 wait complete. 2024-04-24T08:08:27.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.153+0000 7fac43e00640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fac2c06c8f0 0x7fac2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.153+0000 7fac43e00640 1 -- 172.21.15.69:0/2872868632 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fac3c0ac340 con 0x7fac4c0b3d30 2024-04-24T08:08:27.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.153+0000 7fbae4a00640 1 Processor -- start 2024-04-24T08:08:27.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.154+0000 7fbae4a00640 1 -- start start 2024-04-24T08:08:27.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.154+0000 7fac4be00640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fac2c06c8f0 0x7fac2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7fbae4a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbae00d2740 0x7fbae00ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7fbae4a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 0x7fbae00cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7fbae4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 0x7fbae02007b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7fbae4a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbae015f730 con 0x7fbae015ed80 2024-04-24T08:08:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7fbae4a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbae015fe00 con 0x7fbae00d2740 2024-04-24T08:08:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7fbae4a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbae015f8b0 con 0x7fbae015e9a0 2024-04-24T08:08:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7f6763200640 1 Processor -- start 2024-04-24T08:08:27.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7fbade800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 0x7fbae02007b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7fbade800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 0x7fbae02007b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41530/0 (socket says 172.21.15.69:41530) 2024-04-24T08:08:27.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7fbade800640 1 -- 172.21.15.69:0/4228250255 learned_addr learned my addr 172.21.15.69:0/4228250255 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.156+0000 7fac4be00640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fac2c06c8f0 0x7fac2c06edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fac38009ac0 tx=0x7fac38008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.156+0000 7fbadde00640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbae00d2740 0x7fbae00ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.156+0000 7fbadd400640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 0x7fbae00cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.156+0000 7fbade800640 1 -- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 msgr2=0x7fbae00cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.156+0000 7fbade800640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 0x7fbae00cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.156+0000 7fbade800640 1 -- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbae00d2740 msgr2=0x7fbae00ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.156+0000 7fbade800640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbae00d2740 0x7fbae00ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.156+0000 7fbade800640 1 -- 172.21.15.69:0/4228250255 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbae00cc9d0 con 0x7fbae015ed80 2024-04-24T08:08:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.156+0000 7fbade800640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 0x7fbae02007b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009eb0 tx=0x7fbad8004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.155+0000 7f6763200640 1 -- start start 2024-04-24T08:08:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.157+0000 7f6763200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f676406db90 0x7f676409d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.157+0000 7fac43e00640 1 -- 172.21.15.69:0/2872868632 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac3c04a030 con 0x7fac4c0b3d30 2024-04-24T08:08:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.157+0000 7f6763200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67641523f0 0x7f67641f0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.157+0000 7f6763200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676415b980 0x7f67640b0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.157+0000 7f6763200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f676406dfd0 con 0x7f676406db90 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.157+0000 7f6763200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f67640b3190 con 0x7f67641523f0 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.157+0000 7f6763200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f67641527d0 con 0x7f676415b980 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.157+0000 7f6761e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f676406db90 0x7f676409d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.157+0000 7f6761e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f676406db90 0x7f676409d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41544/0 (socket says 172.21.15.69:41544) 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.157+0000 7f6761e00640 1 -- 172.21.15.69:0/3427180938 learned_addr learned my addr 172.21.15.69:0/3427180938 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.158+0000 7f6761400640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67641523f0 0x7f67641f0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.158+0000 7f6762800640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676415b980 0x7f67640b0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.158+0000 7f6761e00640 1 -- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676415b980 msgr2=0x7f67640b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.158+0000 7f6761e00640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676415b980 0x7f67640b0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.158+0000 7f6761e00640 1 -- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67641523f0 msgr2=0x7f67641f0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.158+0000 7f6761e00640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67641523f0 0x7f67641f0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.158+0000 7f6761e00640 1 -- 172.21.15.69:0/3427180938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6764152e10 con 0x7f676406db90 2024-04-24T08:08:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.158+0000 7f6762800640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676415b980 0x7f67640b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.159+0000 7f6761e00640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f676406db90 0x7f676409d9e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f675c00cb40 tx=0x7f675c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.159+0000 7f7919600640 1 -- 172.21.15.69:0/4083891309 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7904044020 con 0x7f791c0b3d30 2024-04-24T08:08:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.160+0000 7f7919600640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78f806c8f0 0x7f78f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.161+0000 7f7919600640 1 -- 172.21.15.69:0/4083891309 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f79040ac630 con 0x7f791c0b3d30 2024-04-24T08:08:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.161+0000 7fbad6000640 1 -- 172.21.15.69:0/4228250255 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad803f070 con 0x7fbae015ed80 2024-04-24T08:08:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.161+0000 7fbae4a00640 1 -- 172.21.15.69:0/4228250255 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbae00cccb0 con 0x7fbae015ed80 2024-04-24T08:08:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.161+0000 7fbae4a00640 1 -- 172.21.15.69:0/4228250255 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbae00cd270 con 0x7fbae015ed80 2024-04-24T08:08:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.161+0000 7f7920a00640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78f806c8f0 0x7f78f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.162+0000 7f9a32000640 1 -- 172.21.15.69:0/3832636318 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a3c0c8020 con 0x7f9a34053e50 2024-04-24T08:08:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.162+0000 7fbae4a00640 1 -- 172.21.15.69:0/4228250255 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae0075d50 con 0x7fbae015ed80 2024-04-24T08:08:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.162+0000 7fd4b3400640 1 -- 172.21.15.69:0/4257420855 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b403f070 con 0x7fd4bc163160 2024-04-24T08:08:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.163+0000 7fd4b3400640 1 -- 172.21.15.69:0/4257420855 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4b4031b80 con 0x7fd4bc163160 2024-04-24T08:08:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.163+0000 7fd4b3400640 1 -- 172.21.15.69:0/4257420855 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4044460 con 0x7fd4bc163160 2024-04-24T08:08:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.164+0000 7fd4c2200640 1 -- 172.21.15.69:0/4257420855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4bc1fcf10 con 0x7fd4bc163160 2024-04-24T08:08:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.164+0000 7fd4c2200640 1 -- 172.21.15.69:0/4257420855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4bc1639c0 con 0x7fd4bc163160 2024-04-24T08:08:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.164+0000 7f7920a00640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78f806c8f0 0x7f78f806edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f790c000f60 tx=0x7f790c007f00 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.165+0000 7fd4c2200640 1 -- 172.21.15.69:0/4257420855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4bc069190 con 0x7fd4bc163160 2024-04-24T08:08:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.167+0000 7fd4b3400640 1 -- 172.21.15.69:0/4257420855 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd4b403b410 con 0x7fd4bc163160 2024-04-24T08:08:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.167+0000 7f7919600640 1 -- 172.21.15.69:0/4083891309 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f790404b030 con 0x7f791c0b3d30 2024-04-24T08:08:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.167+0000 7fbad6000640 1 -- 172.21.15.69:0/4228250255 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad8031710 con 0x7fbae015ed80 2024-04-24T08:08:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.170+0000 7fbad6000640 1 -- 172.21.15.69:0/4228250255 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad803bae0 con 0x7fbae015ed80 2024-04-24T08:08:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.170+0000 7fbad6000640 1 -- 172.21.15.69:0/4228250255 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbad803bc80 con 0x7fbae015ed80 2024-04-24T08:08:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.169+0000 7fd4b3400640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd49c06c940 0x7fd49c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.169+0000 7fd4b3400640 1 -- 172.21.15.69:0/4257420855 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd4b40ad580 con 0x7fd4bc163160 2024-04-24T08:08:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.171+0000 7fbad6000640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbac006c8d0 0x7fbac006ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.171+0000 7f675a000640 1 -- 172.21.15.69:0/3427180938 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c017070 con 0x7f676406db90 2024-04-24T08:08:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.171+0000 7fbad6000640 1 -- 172.21.15.69:0/4228250255 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbad80ad380 con 0x7fbae015ed80 2024-04-24T08:08:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.172+0000 7f675a000640 1 -- 172.21.15.69:0/3427180938 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f675c00aca0 con 0x7f676406db90 2024-04-24T08:08:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.172+0000 7f675a000640 1 -- 172.21.15.69:0/3427180938 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f675c012620 con 0x7f676406db90 2024-04-24T08:08:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.172+0000 7fd4bb400640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd49c06c940 0x7fd49c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.173+0000 7f6763200640 1 -- 172.21.15.69:0/3427180938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67641530f0 con 0x7f676406db90 2024-04-24T08:08:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.173+0000 7f6763200640 1 -- 172.21.15.69:0/3427180938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67641536b0 con 0x7f676406db90 2024-04-24T08:08:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.173+0000 7fd4bb400640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd49c06c940 0x7fd49c06ee00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc0d0c10 tx=0x7fd4ac0078f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.174+0000 7f6763200640 1 -- 172.21.15.69:0/3427180938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6764075d50 con 0x7f676406db90 2024-04-24T08:08:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.176+0000 7fbadde00640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbac006c8d0 0x7fbac006ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.176+0000 7fbad6000640 1 -- 172.21.15.69:0/4228250255 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbad807b820 con 0x7fbae015ed80 2024-04-24T08:08:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.177+0000 7fbadde00640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbac006c8d0 0x7fbac006ed90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbae00ceca0 tx=0x7fbad0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.184+0000 7f675a000640 1 -- 172.21.15.69:0/3427180938 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f675c01a070 con 0x7f676406db90 2024-04-24T08:08:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.185+0000 7fd4b3400640 1 -- 172.21.15.69:0/4257420855 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd4b407ba20 con 0x7fd4bc163160 2024-04-24T08:08:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.185+0000 7f675a000640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f674406c9c0 0x7f674406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.185+0000 7f675a000640 1 -- 172.21.15.69:0/3427180938 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f675c08beb0 con 0x7f676406db90 2024-04-24T08:08:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.185+0000 7f675a000640 1 -- 172.21.15.69:0/3427180938 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f675c021020 con 0x7f676406db90 2024-04-24T08:08:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.186+0000 7f6761400640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f674406c9c0 0x7f674406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.188+0000 7f6761400640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f674406c9c0 0x7f674406ee80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6750035000 tx=0x7f6750008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:27.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.571+0000 7f3862c00640 1 -- 172.21.15.69:0/3326201556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f385c06f830 con 0x7f385c0b3b70 2024-04-24T08:08:27.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.572+0000 7f3853e00640 1 -- 172.21.15.69:0/3326201556 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3848079d40 con 0x7f385c0b3b70 2024-04-24T08:08:27.576 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2024-04-24T08:08:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.576+0000 7f3862c00640 1 -- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f383c06cb20 msgr2=0x7f383c06efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.576+0000 7f3862c00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f383c06cb20 0x7f383c06efe0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3854009bd0 tx=0x7f385400b040 comp rx=0 tx=0).stop 2024-04-24T08:08:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.576+0000 7f3862c00640 1 -- 172.21.15.69:0/3326201556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c0b3b70 msgr2=0x7f385c1ec7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.576+0000 7f3862c00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c0b3b70 0x7f385c1ec7c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3848000c00 tx=0x7f38480046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.578+0000 7f3862c00640 1 -- 172.21.15.69:0/3326201556 shutdown_connections 2024-04-24T08:08:27.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.578+0000 7f3862c00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f383c06cb20 0x7f383c06efe0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.578+0000 7f3862c00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f385c14a850 0x7f385c165800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.578+0000 7f3862c00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f385c0d0840 0x7f385c1652c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.578+0000 7f3862c00640 1 --2- 172.21.15.69:0/3326201556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f385c0b3b70 0x7f385c1ec7c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.578+0000 7f3862c00640 1 -- 172.21.15.69:0/3326201556 >> 172.21.15.69:0/3326201556 conn(0x7f385c067940 msgr2=0x7f385c0b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.579+0000 7f3862c00640 1 -- 172.21.15.69:0/3326201556 shutdown_connections 2024-04-24T08:08:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.579+0000 7f3862c00640 1 -- 172.21.15.69:0/3326201556 wait complete. 2024-04-24T08:08:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.588+0000 7fb7bda00640 1 -- 172.21.15.69:0/2406298476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb790005580 con 0x7fb7b80b3b80 2024-04-24T08:08:27.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.589+0000 7fb7aea00640 1 -- 172.21.15.69:0/2406298476 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb7a8058c60 con 0x7fb7b80b3b80 2024-04-24T08:08:27.593 INFO:teuthology.orchestra.run.smithi069.stdout:55834574853 2024-04-24T08:08:27.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.593+0000 7fb7bda00640 1 -- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb79806ca90 msgr2=0x7fb79806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.593+0000 7fb7bda00640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb79806ca90 0x7fb79806ef50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb7a4007fd0 tx=0x7fb7a4007920 comp rx=0 tx=0).stop 2024-04-24T08:08:27.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.594+0000 7fb7bda00640 1 -- 172.21.15.69:0/2406298476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80b3b80 msgr2=0x7fb7b80ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.594+0000 7fb7bda00640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80b3b80 0x7fb7b80ce8e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800c7b0 tx=0x7fb7a800cc70 comp rx=0 tx=0).stop 2024-04-24T08:08:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.596+0000 7fb7bda00640 1 -- 172.21.15.69:0/2406298476 shutdown_connections 2024-04-24T08:08:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.596+0000 7fb7bda00640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb79806ca90 0x7fb79806ef50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.596+0000 7fb7bda00640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb7b80d1560 0x7fb7b815fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.596+0000 7fb7bda00640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7b80d1100 0x7fb7b81f89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.596+0000 7fb7bda00640 1 --2- 172.21.15.69:0/2406298476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb7b80b3b80 0x7fb7b80ce8e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.596+0000 7fb7bda00640 1 -- 172.21.15.69:0/2406298476 >> 172.21.15.69:0/2406298476 conn(0x7fb7b808bf60 msgr2=0x7fb7b81607d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.597+0000 7fb7bda00640 1 -- 172.21.15.69:0/2406298476 shutdown_connections 2024-04-24T08:08:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.597+0000 7fac52e00640 1 -- 172.21.15.69:0/2872868632 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fac4c06ddd0 con 0x7fac4c0b3d30 2024-04-24T08:08:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.597+0000 7fb7bda00640 1 -- 172.21.15.69:0/2406298476 wait complete. 2024-04-24T08:08:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.598+0000 7fac43e00640 1 -- 172.21.15.69:0/2872868632 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fac3c07a750 con 0x7fac4c0b3d30 2024-04-24T08:08:27.602 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2024-04-24T08:08:27.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.606+0000 7fac41600640 1 -- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fac2c06c8f0 msgr2=0x7fac2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.606+0000 7fac41600640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fac2c06c8f0 0x7fac2c06edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fac38009ac0 tx=0x7fac38008040 comp rx=0 tx=0).stop 2024-04-24T08:08:27.609 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.1 2024-04-24T08:08:27.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.606+0000 7f9a40c00640 1 -- 172.21.15.69:0/3832636318 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9a14005850 con 0x7f9a34053e50 2024-04-24T08:08:27.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.606+0000 7fac41600640 1 -- 172.21.15.69:0/2872868632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 msgr2=0x7fac4c156240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.606+0000 7fac41600640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 0x7fac4c156240 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fac3c035040 tx=0x7fac3c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.607+0000 7fac41600640 1 -- 172.21.15.69:0/2872868632 shutdown_connections 2024-04-24T08:08:27.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.607+0000 7fac41600640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fac2c06c8f0 0x7fac2c06edb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.607+0000 7fac41600640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac4c0b3d30 0x7fac4c156240 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.607+0000 7fac41600640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fac4c162c70 0x7fac4c163790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.607+0000 7f9a32000640 1 -- 172.21.15.69:0/3832636318 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9a3c0f5c80 con 0x7f9a34053e50 2024-04-24T08:08:27.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.607+0000 7fac41600640 1 --2- 172.21.15.69:0/2872868632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac4c071d70 0x7fac4c154a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.607+0000 7fac41600640 1 -- 172.21.15.69:0/2872868632 >> 172.21.15.69:0/2872868632 conn(0x7fac4c067d90 msgr2=0x7fac4c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.607+0000 7fac41600640 1 -- 172.21.15.69:0/2872868632 shutdown_connections 2024-04-24T08:08:27.612 INFO:teuthology.orchestra.run.smithi069.stdout:55834574852 2024-04-24T08:08:27.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.608+0000 7f6763200640 1 -- 172.21.15.69:0/3427180938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f67640b3b80 con 0x7f676406db90 2024-04-24T08:08:27.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.607+0000 7fac41600640 1 -- 172.21.15.69:0/2872868632 wait complete. 2024-04-24T08:08:27.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.609+0000 7f675a000640 1 -- 172.21.15.69:0/3427180938 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f675c05a230 con 0x7f676406db90 2024-04-24T08:08:27.613 INFO:teuthology.orchestra.run.smithi069.stdout:55834574853 2024-04-24T08:08:27.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.610+0000 7f7923400640 1 -- 172.21.15.69:0/4083891309 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f78f4005850 con 0x7f791c0b3d30 2024-04-24T08:08:27.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f9a40c00640 1 -- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1c06c970 msgr2=0x7f9a1c06ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f9a40c00640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1c06c970 0x7f9a1c06ee30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c000f60 tx=0x7f9a2c0079b0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f9a40c00640 1 -- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a34053e50 msgr2=0x7f9a34049760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f9a40c00640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a34053e50 0x7f9a34049760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9a3c0699a0 tx=0x7f9a3c076b60 comp rx=0 tx=0).stop 2024-04-24T08:08:27.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f7919600640 1 -- 172.21.15.69:0/4083891309 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f790407a510 con 0x7f791c0b3d30 2024-04-24T08:08:27.615 INFO:teuthology.orchestra.run.smithi069.stdout:55834574855 2024-04-24T08:08:27.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f9a40c00640 1 -- 172.21.15.69:0/3832636318 shutdown_connections 2024-04-24T08:08:27.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f9a40c00640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1c06c970 0x7f9a1c06ee30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f9a40c00640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a3404f930 0x7f9a340dcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f9a40c00640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a34053e50 0x7f9a34049760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f9a40c00640 1 --2- 172.21.15.69:0/3832636318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a34048100 0x7f9a34172150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.611+0000 7f9a40c00640 1 -- 172.21.15.69:0/3832636318 >> 172.21.15.69:0/3832636318 conn(0x7f9a34009fe0 msgr2=0x7f9a3400a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.612+0000 7f9a40c00640 1 -- 172.21.15.69:0/3832636318 shutdown_connections 2024-04-24T08:08:27.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.612+0000 7f9a40c00640 1 -- 172.21.15.69:0/3832636318 wait complete. 2024-04-24T08:08:27.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.613+0000 7fbae4a00640 1 -- 172.21.15.69:0/4228250255 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fbae006da60 con 0x7fbae015ed80 2024-04-24T08:08:27.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.614+0000 7f6763200640 1 -- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f674406c9c0 msgr2=0x7f674406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.614+0000 7f6763200640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f674406c9c0 0x7f674406ee80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6750035000 tx=0x7f6750008030 comp rx=0 tx=0).stop 2024-04-24T08:08:27.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.614+0000 7f6763200640 1 -- 172.21.15.69:0/3427180938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f676406db90 msgr2=0x7f676409d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.614+0000 7f6763200640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f676406db90 0x7f676409d9e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f675c00cb40 tx=0x7f675c00a060 comp rx=0 tx=0).stop 2024-04-24T08:08:27.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f7923400640 1 -- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78f806c8f0 msgr2=0x7f78f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f7923400640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78f806c8f0 0x7f78f806edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f790c000f60 tx=0x7f790c007f00 comp rx=0 tx=0).stop 2024-04-24T08:08:27.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f7923400640 1 -- 172.21.15.69:0/4083891309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f791c0b3d30 msgr2=0x7f791c1492c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f7923400640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f791c0b3d30 0x7f791c1492c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7904031000 tx=0x7f79040046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f7923400640 1 -- 172.21.15.69:0/4083891309 shutdown_connections 2024-04-24T08:08:27.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f7923400640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78f806c8f0 0x7f78f806edb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f7923400640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f791c0b3d30 0x7f791c1492c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f7923400640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f791c162c70 0x7f791c0cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f7923400640 1 --2- 172.21.15.69:0/4083891309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f791c06e0f0 0x7f791c148d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f7923400640 1 -- 172.21.15.69:0/4083891309 >> 172.21.15.69:0/4083891309 conn(0x7f791c067d90 msgr2=0x7f791c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7fbad6000640 1 -- 172.21.15.69:0/4228250255 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbad807b120 con 0x7fbae015ed80 2024-04-24T08:08:27.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f6763200640 1 -- 172.21.15.69:0/3427180938 shutdown_connections 2024-04-24T08:08:27.619 INFO:teuthology.orchestra.run.smithi069.stdout:55834574855 2024-04-24T08:08:27.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f6763200640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f674406c9c0 0x7f674406ee80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f6763200640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f676415b980 0x7f67640b0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.615+0000 7f6763200640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67641523f0 0x7f67641f0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.616+0000 7f6763200640 1 --2- 172.21.15.69:0/3427180938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f676406db90 0x7f676409d9e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.616+0000 7f6763200640 1 -- 172.21.15.69:0/3427180938 >> 172.21.15.69:0/3427180938 conn(0x7f67640d0160 msgr2=0x7f6764071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.616+0000 7f7923400640 1 -- 172.21.15.69:0/4083891309 shutdown_connections 2024-04-24T08:08:27.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.616+0000 7f7923400640 1 -- 172.21.15.69:0/4083891309 wait complete. 2024-04-24T08:08:27.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.616+0000 7f6763200640 1 -- 172.21.15.69:0/3427180938 shutdown_connections 2024-04-24T08:08:27.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.616+0000 7f6763200640 1 -- 172.21.15.69:0/3427180938 wait complete. 2024-04-24T08:08:27.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.619+0000 7fbae4a00640 1 -- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbac006c8d0 msgr2=0x7fbac006ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.619+0000 7fbae4a00640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbac006c8d0 0x7fbac006ed90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbae00ceca0 tx=0x7fbad0008040 comp rx=0 tx=0).stop 2024-04-24T08:08:27.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.619+0000 7fbae4a00640 1 -- 172.21.15.69:0/4228250255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 msgr2=0x7fbae02007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.619+0000 7fbae4a00640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 0x7fbae02007b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009eb0 tx=0x7fbad8004a70 comp rx=0 tx=0).stop 2024-04-24T08:08:27.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.620+0000 7fbae4a00640 1 -- 172.21.15.69:0/4228250255 shutdown_connections 2024-04-24T08:08:27.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.620+0000 7fbae4a00640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbac006c8d0 0x7fbac006ed90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.620+0000 7fbae4a00640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbae015ed80 0x7fbae02007b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.620+0000 7fbae4a00640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbae015e9a0 0x7fbae00cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.620+0000 7fbae4a00640 1 --2- 172.21.15.69:0/4228250255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbae00d2740 0x7fbae00ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.620+0000 7fbae4a00640 1 -- 172.21.15.69:0/4228250255 >> 172.21.15.69:0/4228250255 conn(0x7fbae008bf60 msgr2=0x7fbae006cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.620+0000 7fbae4a00640 1 -- 172.21.15.69:0/4228250255 shutdown_connections 2024-04-24T08:08:27.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.620+0000 7fbae4a00640 1 -- 172.21.15.69:0/4228250255 wait complete. 2024-04-24T08:08:27.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.0 2024-04-24T08:08:27.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.628+0000 7fd4c2200640 1 -- 172.21.15.69:0/4257420855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd4bc1fd0e0 con 0x7fd4bc163160 2024-04-24T08:08:27.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.629+0000 7fd4b3400640 1 -- 172.21.15.69:0/4257420855 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd4b407b320 con 0x7fd4bc163160 2024-04-24T08:08:27.630 INFO:teuthology.orchestra.run.smithi069.stdout:55834574853 2024-04-24T08:08:27.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 -- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd49c06c940 msgr2=0x7fd49c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd49c06c940 0x7fd49c06ee00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc0d0c10 tx=0x7fd4ac0078f0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 -- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4bc163160 msgr2=0x7fd4bc1fc1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4bc163160 0x7fd4bc1fc1e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4031000 tx=0x7fd4b4004a40 comp rx=0 tx=0).stop 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 -- 172.21.15.69:0/4257420855 shutdown_connections 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd49c06c940 0x7fd49c06ee00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd4bc162d80 0x7fd4bc1625d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4bc163160 0x7fd4bc1fc1e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 --2- 172.21.15.69:0/4257420855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd4bc0b66f0 0x7fd4bc0c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 -- 172.21.15.69:0/4257420855 >> 172.21.15.69:0/4257420855 conn(0x7fd4bc0d11b0 msgr2=0x7fd4bc0ceeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 -- 172.21.15.69:0/4257420855 shutdown_connections 2024-04-24T08:08:27.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:27.633+0000 7fd4c2200640 1 -- 172.21.15.69:0/4257420855 wait complete. 2024-04-24T08:08:27.636 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.2 2024-04-24T08:08:27.637 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-04-24T08:08:27.641 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-04-24T08:08:27.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.6 2024-04-24T08:08:27.645 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.7 2024-04-24T08:08:27.653 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-04-24T08:08:28.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T08:08:28.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T08:08:28.637 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T08:08:28.637 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T08:08:28.642 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T08:08:28.644 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T08:08:28.646 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-24T08:08:28.653 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T08:08:28.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.893+0000 7f1daf200640 1 Processor -- start 2024-04-24T08:08:28.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.893+0000 7f1daf200640 1 -- start start 2024-04-24T08:08:28.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.895+0000 7f1daf200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1da00e7a50 0x7f1da0049b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.895+0000 7f1daf200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 0x7f1da003c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.895+0000 7f1daf200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da00e7e30 con 0x7f1da004a0a0 2024-04-24T08:08:28.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.895+0000 7f1daf200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1da003b5a0 con 0x7f1da00e7a50 2024-04-24T08:08:28.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.895+0000 7f1daf200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1da003b880 con 0x7f1da003c360 2024-04-24T08:08:28.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.896+0000 7f1dad200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 0x7f1da003c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.896+0000 7f1da7e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1da00e7a50 0x7f1da0049b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.896+0000 7f1da7e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1da00e7a50 0x7f1da0049b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:60834/0 (socket says 172.21.15.69:60834) 2024-04-24T08:08:28.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.896+0000 7f1da7e00640 1 -- 172.21.15.69:0/980197979 learned_addr learned my addr 172.21.15.69:0/980197979 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:28.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.898+0000 7f1dad200640 1 -- 172.21.15.69:0/980197979 >> v2:172.21.15.121:3300/0 conn(0x7f1da00e7a50 msgr2=0x7f1da0049b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.898+0000 7f1dad200640 1 --2- 172.21.15.69:0/980197979 >> v2:172.21.15.121:3300/0 conn(0x7f1da00e7a50 0x7f1da0049b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.898+0000 7f1dad200640 1 -- 172.21.15.69:0/980197979 >> v1:172.21.15.121:6789/0 conn(0x7f1da004a0a0 legacy=0x7f1da00548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.898+0000 7f1dad200640 1 -- 172.21.15.69:0/980197979 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da003bb60 con 0x7f1da003c360 2024-04-24T08:08:28.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.898+0000 7f1da7e00640 1 --2- 172.21.15.69:0/980197979 >> v2:172.21.15.121:3300/0 conn(0x7f1da00e7a50 0x7f1da0049b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:28.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.899+0000 7f1dad200640 1 --2- 172.21.15.69:0/980197979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 0x7f1da003c810 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1da8051000 tx=0x7f1da80a5b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec065f84fcbc6faf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.900+0000 7f1da6a00640 1 -- 172.21.15.69:0/980197979 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da80a9930 con 0x7f1da003c360 2024-04-24T08:08:28.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.900+0000 7f1da6a00640 1 -- 172.21.15.69:0/980197979 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1da80a9ad0 con 0x7f1da003c360 2024-04-24T08:08:28.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.900+0000 7f1da6a00640 1 -- 172.21.15.69:0/980197979 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da80b1400 con 0x7f1da003c360 2024-04-24T08:08:28.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.901+0000 7f1daf200640 1 -- 172.21.15.69:0/980197979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 msgr2=0x7f1da003c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.901+0000 7f1daf200640 1 --2- 172.21.15.69:0/980197979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 0x7f1da003c810 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1da8051000 tx=0x7f1da80a5b70 comp rx=0 tx=0).stop 2024-04-24T08:08:28.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.901+0000 7f1daf200640 1 -- 172.21.15.69:0/980197979 shutdown_connections 2024-04-24T08:08:28.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.902+0000 7f1daf200640 1 --2- 172.21.15.69:0/980197979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 0x7f1da003c810 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.902+0000 7f1daf200640 1 --2- 172.21.15.69:0/980197979 >> v2:172.21.15.121:3300/0 conn(0x7f1da00e7a50 0x7f1da0049b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.902+0000 7f1daf200640 1 -- 172.21.15.69:0/980197979 >> 172.21.15.69:0/980197979 conn(0x7f1da0056cd0 msgr2=0x7f1da00570e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:28.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.903+0000 7f1daf200640 1 -- 172.21.15.69:0/980197979 shutdown_connections 2024-04-24T08:08:28.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.903+0000 7f1daf200640 1 -- 172.21.15.69:0/980197979 wait complete. 2024-04-24T08:08:28.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.907+0000 7f1daf200640 1 Processor -- start 2024-04-24T08:08:28.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.907+0000 7f1daf200640 1 -- start start 2024-04-24T08:08:28.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.909+0000 7f1daf200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 0x7f1da00d13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.909+0000 7f1daf200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1da004a0a0 0x7f1da00ccda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.910+0000 7f1daf200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da00e7a50 0x7f1da00cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.910+0000 7f1da7400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1da004a0a0 0x7f1da00ccda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.911+0000 7f1da7400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1da004a0a0 0x7f1da00ccda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:60844/0 (socket says 172.21.15.69:60844) 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.911+0000 7f1dad200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da00e7a50 0x7f1da00cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.911+0000 7f1da7400640 1 -- 172.21.15.69:0/2536526502 learned_addr learned my addr 172.21.15.69:0/2536526502 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.911+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1da0054fb0 con 0x7f1da00e7a50 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.911+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1da004a5b0 con 0x7f1da004a0a0 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.911+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1da00d0f50 con 0x7f1da003c360 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.913+0000 7f1da7e00640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 0x7f1da00d13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.913+0000 7f1dad200640 1 -- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 msgr2=0x7f1da00d13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.913+0000 7f1dad200640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 0x7f1da00d13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.913+0000 7f1dad200640 1 -- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1da004a0a0 msgr2=0x7f1da00ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.913+0000 7f1dad200640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1da004a0a0 0x7f1da00ccda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.913+0000 7f1dad200640 1 -- 172.21.15.69:0/2536526502 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da000a9a0 con 0x7f1da00e7a50 2024-04-24T08:08:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.913+0000 7f1da7e00640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 0x7f1da00d13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.914+0000 7f1dad200640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da00e7a50 0x7f1da00cd2e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1da806a740 tx=0x7f1da806b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.915+0000 7f1d9be00640 1 -- 172.21.15.69:0/2536526502 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da80c2070 con 0x7f1da00e7a50 2024-04-24T08:08:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.915+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da00d0c70 con 0x7f1da00e7a50 2024-04-24T08:08:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.915+0000 7f1d9be00640 1 -- 172.21.15.69:0/2536526502 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1da80b7040 con 0x7f1da00e7a50 2024-04-24T08:08:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.915+0000 7f1d9be00640 1 -- 172.21.15.69:0/2536526502 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da80c2070 con 0x7f1da00e7a50 2024-04-24T08:08:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.916+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da00d1f70 con 0x7f1da00e7a50 2024-04-24T08:08:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.920+0000 7f1d9be00640 1 -- 172.21.15.69:0/2536526502 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1da80a9930 con 0x7f1da00e7a50 2024-04-24T08:08:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.921+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da00057e0 con 0x7f1da00e7a50 2024-04-24T08:08:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.921+0000 7f1d9be00640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d8806cb20 0x7f1d8806efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.926+0000 7f1d9be00640 1 -- 172.21.15.69:0/2536526502 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1da80f34b0 con 0x7f1da00e7a50 2024-04-24T08:08:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.928+0000 7f1da7e00640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d8806cb20 0x7f1d8806efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.929+0000 7f1da7e00640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d8806cb20 0x7f1d8806efe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009630 tx=0x7f1d94008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.933+0000 7f1d9be00640 1 -- 172.21.15.69:0/2536526502 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1da80ad030 con 0x7f1da00e7a50 2024-04-24T08:08:28.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.945+0000 7f2452e00640 1 Processor -- start 2024-04-24T08:08:28.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.945+0000 7f2452e00640 1 -- start start 2024-04-24T08:08:28.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.948+0000 7f2452e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0d10f0 0x7f244c0d1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.948+0000 7f244be00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f244c162150 0x7f244c0d5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36566/0 (socket says 172.21.15.69:36566) 2024-04-24T08:08:28.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.948+0000 7f244be00640 1 -- 172.21.15.69:0/309980107 learned_addr learned my addr 172.21.15.69:0/309980107 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:28.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.948+0000 7f2452e00640 1 -- 172.21.15.69:0/309980107 --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f244c0d22d0 con 0x7f244c162150 2024-04-24T08:08:28.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.948+0000 7f2452e00640 1 -- 172.21.15.69:0/309980107 --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f244c0d03b0 con 0x7f244c0cfe40 2024-04-24T08:08:28.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.948+0000 7f2452e00640 1 -- 172.21.15.69:0/309980107 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f244c0d2900 con 0x7f244c0d10f0 2024-04-24T08:08:28.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.949+0000 7f2450e00640 1 --2- 172.21.15.69:0/309980107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0d10f0 0x7f244c0d1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.949+0000 7f2450e00640 1 -- 172.21.15.69:0/309980107 >> v1:172.21.15.121:6789/0 conn(0x7f244c0cfe40 legacy=0x7f244c0b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.949+0000 7f2450e00640 1 -- 172.21.15.69:0/309980107 >> v1:172.21.15.69:6789/0 conn(0x7f244c162150 legacy=0x7f244c0d5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.949+0000 7f2450e00640 1 -- 172.21.15.69:0/309980107 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c0d0690 con 0x7f244c0d10f0 2024-04-24T08:08:28.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.951+0000 7f4dcee00640 1 Processor -- start 2024-04-24T08:08:28.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.951+0000 7f5905800640 1 Processor -- start 2024-04-24T08:08:28.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.951+0000 7f4dcee00640 1 -- start start 2024-04-24T08:08:28.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.951+0000 7f2450e00640 1 --2- 172.21.15.69:0/309980107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0d10f0 0x7f244c0d1580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2444009dc0 tx=0x7f244402a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56c5df25d6645297 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.952+0000 7f244aa00640 1 -- 172.21.15.69:0/309980107 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444007740 con 0x7f244c0d10f0 2024-04-24T08:08:28.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.952+0000 7f244aa00640 1 -- 172.21.15.69:0/309980107 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24440078e0 con 0x7f244c0d10f0 2024-04-24T08:08:28.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.952+0000 7f244aa00640 1 -- 172.21.15.69:0/309980107 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2444007cb0 con 0x7f244c0d10f0 2024-04-24T08:08:28.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f4dcee00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4dc8154f50 0x7f4dc815ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f4dcee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80b3320 0x7f4dc806e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.951+0000 7f5905800640 1 -- start start 2024-04-24T08:08:28.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f5905800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f58f804fa90 0x7f58f804fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f5905800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58f80df420 0x7f58f80382c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f4dcee00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc8155330 con 0x7f4dc806ee50 2024-04-24T08:08:28.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f4dcee00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4dc806f2c0 con 0x7f4dc8154f50 2024-04-24T08:08:28.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f4dcee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc806f790 con 0x7f4dc80b3320 2024-04-24T08:08:28.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.954+0000 7f4dcce00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4dc806ee50 0x7f4dc8071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:59814/0 (socket says 172.21.15.69:59814) 2024-04-24T08:08:28.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.954+0000 7f5905800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58f80503b0 con 0x7f58f8038800 2024-04-24T08:08:28.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.954+0000 7f4dcce00640 1 -- 172.21.15.69:0/2120153665 learned_addr learned my addr 172.21.15.69:0/2120153665 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:28.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.954+0000 7f4dc7400640 1 --2- 172.21.15.69:0/2120153665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80b3320 0x7f4dc806e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.954+0000 7f5905800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f58f80068a0 con 0x7f58f804fa90 2024-04-24T08:08:28.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.954+0000 7f5905800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f58f8006af0 con 0x7f58f80df420 2024-04-24T08:08:28.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f2452e00640 1 -- 172.21.15.69:0/309980107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0d10f0 msgr2=0x7f244c0d1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f2452e00640 1 --2- 172.21.15.69:0/309980107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0d10f0 0x7f244c0d1580 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2444009dc0 tx=0x7f244402a580 comp rx=0 tx=0).stop 2024-04-24T08:08:28.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f2452e00640 1 -- 172.21.15.69:0/309980107 shutdown_connections 2024-04-24T08:08:28.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f2452e00640 1 --2- 172.21.15.69:0/309980107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0d10f0 0x7f244c0d1580 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.953+0000 7f2452e00640 1 -- 172.21.15.69:0/309980107 >> 172.21.15.69:0/309980107 conn(0x7f244c08bf60 msgr2=0x7f244c0d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:28.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.954+0000 7f2452e00640 1 -- 172.21.15.69:0/309980107 shutdown_connections 2024-04-24T08:08:28.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.955+0000 7f2452e00640 1 -- 172.21.15.69:0/309980107 wait complete. 2024-04-24T08:08:28.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.954+0000 7f5904e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f58f8038800 0x7f58f8038c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36572/0 (socket says 172.21.15.69:36572) 2024-04-24T08:08:28.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.954+0000 7f5904e00640 1 -- 172.21.15.69:0/2948757394 learned_addr learned my addr 172.21.15.69:0/2948757394 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:28.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.954+0000 7f58ff400640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58f80df420 0x7f58f80382c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.955+0000 7f58ffe00640 1 --2- 172.21.15.69:0/2948757394 >> v2:172.21.15.121:3300/0 conn(0x7f58f804fa90 0x7f58f804fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.955+0000 7f4dc6a00640 1 -- 172.21.15.69:0/2120153665 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 380638091 0 0) 0x7f4db4001110 con 0x7f4dc806ee50 2024-04-24T08:08:28.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.955+0000 7f4dc6a00640 1 -- 172.21.15.69:0/2120153665 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dac005730 con 0x7f4dc806ee50 2024-04-24T08:08:28.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.955+0000 7f4dc7400640 1 -- 172.21.15.69:0/2120153665 >> v2:172.21.15.69:3300/0 conn(0x7f4dc8154f50 msgr2=0x7f4dc815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.955+0000 7f4dc7400640 1 --2- 172.21.15.69:0/2120153665 >> v2:172.21.15.69:3300/0 conn(0x7f4dc8154f50 0x7f4dc815ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.955+0000 7f4dc7400640 1 -- 172.21.15.69:0/2120153665 >> v1:172.21.15.121:6789/0 conn(0x7f4dc806ee50 legacy=0x7f4dc8071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.955+0000 7f4dc7400640 1 -- 172.21.15.69:0/2120153665 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc80b37a0 con 0x7f4dc80b3320 2024-04-24T08:08:28.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.955+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1357329269 0 0) 0x7f58f80503b0 con 0x7f58f8038800 2024-04-24T08:08:28.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58e8003520 con 0x7f58f8038800 2024-04-24T08:08:28.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1630722057 0 0) 0x7f58e8003520 con 0x7f58f8038800 2024-04-24T08:08:28.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58f80df420 msgr2=0x7f58f80382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58f80df420 0x7f58f80382c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 >> v2:172.21.15.121:3300/0 conn(0x7f58f804fa90 msgr2=0x7f58f804fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 --2- 172.21.15.69:0/2948757394 >> v2:172.21.15.121:3300/0 conn(0x7f58f804fa90 0x7f58f804fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f8006dd0 con 0x7f58f8038800 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f5900075d50 con 0x7f58f8038800 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 >> v1:172.21.15.69:6789/0 conn(0x7f58f8038800 legacy=0x7f58f8038c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58e80039b0 0x7f58e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 0x7f58e8004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58e8004d60 0x7f58e80051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58f80068a0 con 0x7f58e8004350 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8006af0 con 0x7f58e8004d60 2024-04-24T08:08:28.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.956+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f58e80145c0 con 0x7f58e80039b0 2024-04-24T08:08:28.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.957+0000 7efe09200640 1 Processor -- start 2024-04-24T08:08:28.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.957+0000 7fcb40200640 1 Processor -- start 2024-04-24T08:08:28.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.957+0000 7f58ff400640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 0x7f58e8004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7f5904e00640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58e8004d60 0x7f58e80051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7f2452e00640 1 Processor -- start 2024-04-24T08:08:28.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7f2452e00640 1 -- start start 2024-04-24T08:08:28.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7efe09200640 1 -- start start 2024-04-24T08:08:28.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7f58ff400640 1 -- 172.21.15.69:0/2948757394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58e80039b0 msgr2=0x7f58e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7f58ff400640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58e80039b0 0x7f58e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7f58ff400640 1 -- 172.21.15.69:0/2948757394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58e8004d60 msgr2=0x7f58e80051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7f58ff400640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58e8004d60 0x7f58e80051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7f58ff400640 1 -- 172.21.15.69:0/2948757394 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f8006dd0 con 0x7f58e8004350 2024-04-24T08:08:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7f4dc7400640 1 --2- 172.21.15.69:0/2120153665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80b3320 0x7f4dc806e910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4db80090e0 tx=0x7f4db802a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63b3f676e9259044 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.958+0000 7fcb40200640 1 -- start start 2024-04-24T08:08:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7f4dc6a00640 1 -- 172.21.15.69:0/2120153665 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db8037070 con 0x7f4dc80b3320 2024-04-24T08:08:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7f4dc6a00640 1 -- 172.21.15.69:0/2120153665 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db8032440 con 0x7f4dc80b3320 2024-04-24T08:08:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7f4dc6a00640 1 -- 172.21.15.69:0/2120153665 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db8032810 con 0x7f4dc80b3320 2024-04-24T08:08:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7f2452e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0cfe40 0x7f244c0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7fcb40200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcb380b3b80 0x7fcb380d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7fcb40200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380d1810 0x7fcb3806cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7fcb40200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb380b3f60 con 0x7fcb380d0590 2024-04-24T08:08:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7fcb40200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcb380d3360 con 0x7fcb380b3b80 2024-04-24T08:08:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7fcb40200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcb380d2750 con 0x7fcb380d1810 2024-04-24T08:08:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7efe09200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efe0415ae50 0x7efe04154f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7efe09200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe040b3320 0x7efe04072480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7efe09200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0415d020 con 0x7efe04067df0 2024-04-24T08:08:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7efe09200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7efe0406f790 con 0x7efe0415ae50 2024-04-24T08:08:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7efe09200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efe040b3760 con 0x7efe040b3320 2024-04-24T08:08:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7f4dcee00640 1 -- 172.21.15.69:0/2120153665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80b3320 msgr2=0x7f4dc806e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7f4dcee00640 1 --2- 172.21.15.69:0/2120153665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80b3320 0x7f4dc806e910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4db80090e0 tx=0x7f4db802a200 comp rx=0 tx=0).stop 2024-04-24T08:08:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7f4856c00640 1 Processor -- start 2024-04-24T08:08:28.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7f58ff400640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 0x7f58e8004820 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009960 tx=0x7f58e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631d2af2936a2f82 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ec036070 con 0x7f58e8004350 2024-04-24T08:08:28.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7f2452e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f244c0d10f0 0x7f244c201e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7f2452e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f244c162150 0x7f244c1f7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7f2452e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f244c0d2d90 con 0x7f244c0d10f0 2024-04-24T08:08:28.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7f2452e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f244c0b7c60 con 0x7f244c162150 2024-04-24T08:08:28.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.959+0000 7f2452e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f244c0d2f10 con 0x7f244c0cfe40 2024-04-24T08:08:28.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7fcb3e200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380d1810 0x7fcb3806cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7f4856c00640 1 -- start start 2024-04-24T08:08:28.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f4dcee00640 1 -- 172.21.15.69:0/2120153665 shutdown_connections 2024-04-24T08:08:28.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f4dcee00640 1 --2- 172.21.15.69:0/2120153665 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80b3320 0x7f4dc806e910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f4dcee00640 1 --2- 172.21.15.69:0/2120153665 >> v2:172.21.15.69:3300/0 conn(0x7f4dc8154f50 0x7f4dc815ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f4dcee00640 1 -- 172.21.15.69:0/2120153665 >> 172.21.15.69:0/2120153665 conn(0x7f4dc8067940 msgr2=0x7f4dc806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:28.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7fcb3e200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380d1810 0x7fcb3806cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:33814/0 (socket says 172.21.15.69:33814) 2024-04-24T08:08:28.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7fcb3e200640 1 -- 172.21.15.69:0/550564639 learned_addr learned my addr 172.21.15.69:0/550564639 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:28.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58ec031400 con 0x7f58e8004350 2024-04-24T08:08:28.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f58fea00640 1 -- 172.21.15.69:0/2948757394 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ec009630 con 0x7f58e8004350 2024-04-24T08:08:28.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7fcb3d800640 1 --2- 172.21.15.69:0/550564639 >> v2:172.21.15.121:3300/0 conn(0x7fcb380b3b80 0x7fcb380d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f4dcee00640 1 -- 172.21.15.69:0/2120153665 shutdown_connections 2024-04-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7efe01e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7efe04067df0 0x7efe04071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36582/0 (socket says 172.21.15.69:36582) 2024-04-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7efe01e00640 1 -- 172.21.15.69:0/1014902792 learned_addr learned my addr 172.21.15.69:0/1014902792 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.960+0000 7efe02800640 1 --2- 172.21.15.69:0/1014902792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe040b3320 0x7efe04072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f7d11800640 1 Processor -- start 2024-04-24T08:08:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f7d11800640 1 -- start start 2024-04-24T08:08:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7fcb37e00640 1 -- 172.21.15.69:0/550564639 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 257388503 0 0) 0x7fcb380b3f60 con 0x7fcb380d0590 2024-04-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f4dcee00640 1 -- 172.21.15.69:0/2120153665 wait complete. 2024-04-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f244be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0cfe40 0x7f244c0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f244be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0cfe40 0x7f244c0cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:33784/0 (socket says 172.21.15.69:33784) 2024-04-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f244be00640 1 -- 172.21.15.69:0/3867439148 learned_addr learned my addr 172.21.15.69:0/3867439148 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.961+0000 7f2450e00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f244c162150 0x7f244c1f7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7f244be00640 1 -- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f244c162150 msgr2=0x7f244c1f7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7f244be00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f244c162150 0x7f244c1f7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7f244be00640 1 -- 172.21.15.69:0/3867439148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f244c0d10f0 msgr2=0x7f244c201e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7f244be00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f244c0d10f0 0x7f244c201e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7f244be00640 1 -- 172.21.15.69:0/3867439148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f244c2023a0 con 0x7f244c0cfe40 2024-04-24T08:08:28.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7efe01400640 1 --2- 172.21.15.69:0/1014902792 >> v2:172.21.15.69:3300/0 conn(0x7efe0415ae50 0x7efe04154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7f5905800640 1 -- 172.21.15.69:0/2948757394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 msgr2=0x7f58e8004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7f5905800640 1 --2- 172.21.15.69:0/2948757394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 0x7f58e8004820 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009960 tx=0x7f58e8005830 comp rx=0 tx=0).stop 2024-04-24T08:08:28.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7f5904e00640 1 -- 172.21.15.69:0/2948757394 reap_dead start 2024-04-24T08:08:28.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7fcb37e00640 1 -- 172.21.15.69:0/550564639 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb20003520 con 0x7fcb380d0590 2024-04-24T08:08:28.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7fcb3e200640 1 -- 172.21.15.69:0/550564639 >> v2:172.21.15.121:3300/0 conn(0x7fcb380b3b80 msgr2=0x7fcb380d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7fcb3e200640 1 --2- 172.21.15.69:0/550564639 >> v2:172.21.15.121:3300/0 conn(0x7fcb380b3b80 0x7fcb380d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7fcb3e200640 1 -- 172.21.15.69:0/550564639 >> v1:172.21.15.69:6789/0 conn(0x7fcb380d0590 legacy=0x7fcb380d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7fcb3e200640 1 -- 172.21.15.69:0/550564639 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb380d2a30 con 0x7fcb380d1810 2024-04-24T08:08:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.962+0000 7efe02800640 1 -- 172.21.15.69:0/1014902792 >> v2:172.21.15.69:3300/0 conn(0x7efe0415ae50 msgr2=0x7efe04154f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7f4856c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f48500b3d30 0x7f48500b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7f4856c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 0x7f485006e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7efe02800640 1 --2- 172.21.15.69:0/1014902792 >> v2:172.21.15.69:3300/0 conn(0x7efe0415ae50 0x7efe04154f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7efe02800640 1 -- 172.21.15.69:0/1014902792 >> v1:172.21.15.69:6789/0 conn(0x7efe04067df0 legacy=0x7efe04071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7efe02800640 1 -- 172.21.15.69:0/1014902792 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe040b3a20 con 0x7efe040b3320 2024-04-24T08:08:28.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7f4856c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48501669d0 con 0x7f4850162c70 2024-04-24T08:08:28.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7f4856c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4850166cb0 con 0x7f48500b3d30 2024-04-24T08:08:28.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7f7d11800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d0c15b590 0x7f7d0c0b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7f4856c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4850166f90 con 0x7f485006e0f0 2024-04-24T08:08:28.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7f484fe00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4850162c70 0x7f48500b3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:59818/0 (socket says 172.21.15.69:59818) 2024-04-24T08:08:28.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f7d11800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d0c0ccb50 0x7f7d0c15f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.963+0000 7f484fe00640 1 -- 172.21.15.69:0/3627957194 learned_addr learned my addr 172.21.15.69:0/3627957194 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:28.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7fcb3e200640 1 --2- 172.21.15.69:0/550564639 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380d1810 0x7fcb3806cc00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009e50 tx=0x7fcb2c02a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=880940fc8ef05592 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f244be00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0cfe40 0x7f244c0cccc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f243c00bdb0 tx=0x7f243c00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f5905800640 1 -- 172.21.15.69:0/2948757394 shutdown_connections 2024-04-24T08:08:28.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f5905800640 1 -- 172.21.15.69:0/2948757394 >> 172.21.15.69:0/2948757394 conn(0x7f58f8009fb0 msgr2=0x7f58f800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:28.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f4dcee00640 1 Processor -- start 2024-04-24T08:08:28.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7efe01400640 1 --2- 172.21.15.69:0/1014902792 >> v2:172.21.15.69:3300/0 conn(0x7efe0415ae50 0x7efe04154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:28.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f7d11800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d0c0cf070 con 0x7f7d0c0b1340 2024-04-24T08:08:28.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f7d11800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7d0c0cf680 con 0x7f7d0c0ccb50 2024-04-24T08:08:28.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f484f400640 1 --2- 172.21.15.69:0/3627957194 >> v2:172.21.15.121:3300/0 conn(0x7f48500b3d30 0x7f48500b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f4854c00640 1 --2- 172.21.15.69:0/3627957194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 0x7f485006e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f7d11800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0c0cd010 con 0x7f7d0c15b590 2024-04-24T08:08:28.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f7d02000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7d0c0b1340 0x7f7d0c0cfb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36594/0 (socket says 172.21.15.69:36594) 2024-04-24T08:08:28.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f7d02000640 1 -- 172.21.15.69:0/480004790 learned_addr learned my addr 172.21.15.69:0/480004790 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:28.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f7d0aa00640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d0c15b590 0x7f7d0c0b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f7d0b400640 1 --2- 172.21.15.69:0/480004790 >> v2:172.21.15.121:3300/0 conn(0x7f7d0c0ccb50 0x7f7d0c15f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f484ea00640 1 -- 172.21.15.69:0/3627957194 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 818707124 0 0) 0x7f48501669d0 con 0x7f4850162c70 2024-04-24T08:08:28.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f484ea00640 1 -- 172.21.15.69:0/3627957194 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4834003520 con 0x7f4850162c70 2024-04-24T08:08:28.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.964+0000 7f4dcee00640 1 -- start start 2024-04-24T08:08:28.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f5905800640 1 -- 172.21.15.69:0/2948757394 shutdown_connections 2024-04-24T08:08:28.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f5905800640 1 -- 172.21.15.69:0/2948757394 wait complete. 2024-04-24T08:08:28.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1758665590 0 0) 0x7f7d0c0cf070 con 0x7f7d0c0b1340 2024-04-24T08:08:28.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cf0003520 con 0x7f7d0c0b1340 2024-04-24T08:08:28.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f2443e00640 1 -- 172.21.15.69:0/3867439148 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c00ce10 con 0x7f244c0cfe40 2024-04-24T08:08:28.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f2443e00640 1 -- 172.21.15.69:0/3867439148 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243c010070 con 0x7f244c0cfe40 2024-04-24T08:08:28.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f2452e00640 1 -- 172.21.15.69:0/3867439148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f244c1f8300 con 0x7f244c0cfe40 2024-04-24T08:08:28.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f2452e00640 1 -- 172.21.15.69:0/3867439148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f244c15f1a0 con 0x7f244c0cfe40 2024-04-24T08:08:28.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7fcb37e00640 1 -- 172.21.15.69:0/550564639 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c036070 con 0x7fcb380d1810 2024-04-24T08:08:28.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7fcb37e00640 1 -- 172.21.15.69:0/550564639 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb2c031600 con 0x7fcb380d1810 2024-04-24T08:08:28.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7fcb37e00640 1 -- 172.21.15.69:0/550564639 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c031a10 con 0x7fcb380d1810 2024-04-24T08:08:28.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1857145355 0 0) 0x7f7cf0003520 con 0x7f7d0c0b1340 2024-04-24T08:08:28.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7efe02800640 1 --2- 172.21.15.69:0/1014902792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe040b3320 0x7efe04072480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00be90 tx=0x7efdfc02c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=836218cf2d0f6b25 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d0c15b590 msgr2=0x7f7d0c0b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f7d0a000640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d0c15b590 0x7f7d0c0b0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 >> v2:172.21.15.121:3300/0 conn(0x7f7d0c0ccb50 msgr2=0x7f7d0c15f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f7d0a000640 1 --2- 172.21.15.69:0/480004790 >> v2:172.21.15.121:3300/0 conn(0x7f7d0c0ccb50 0x7f7d0c15f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0c0cd2f0 con 0x7f7d0c0b1340 2024-04-24T08:08:28.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f4854c00640 1 -- 172.21.15.69:0/3627957194 >> v2:172.21.15.121:3300/0 conn(0x7f48500b3d30 msgr2=0x7f48500b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f4854c00640 1 --2- 172.21.15.69:0/3627957194 >> v2:172.21.15.121:3300/0 conn(0x7f48500b3d30 0x7f48500b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f4854c00640 1 -- 172.21.15.69:0/3627957194 >> v1:172.21.15.121:6789/0 conn(0x7f4850162c70 legacy=0x7f48500b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.965+0000 7f4854c00640 1 -- 172.21.15.69:0/3627957194 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4850167270 con 0x7f485006e0f0 2024-04-24T08:08:28.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.966+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f7cf40019d0 con 0x7f7d0c0b1340 2024-04-24T08:08:28.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.966+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 >> v1:172.21.15.69:6789/0 conn(0x7f7d0c0b1340 legacy=0x7f7d0c0cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.966+0000 7f7d0a000640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cf00039b0 0x7f7cf0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.966+0000 7f7d0a000640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cf0004800 0x7f7cf0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.966+0000 7f7d0a000640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d0c15b590 0x7f7cf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.966+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c0cf680 con 0x7f7cf0004800 2024-04-24T08:08:28.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.966+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c0cd010 con 0x7f7cf00039b0 2024-04-24T08:08:28.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.966+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf000b9f0 con 0x7f7d0c15b590 2024-04-24T08:08:28.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.966+0000 7f2443e00640 1 -- 172.21.15.69:0/3867439148 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c0156b0 con 0x7f244c0cfe40 2024-04-24T08:08:28.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f7d02000640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cf0004800 0x7f7cf0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f4854c00640 1 --2- 172.21.15.69:0/3627957194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 0x7f485006e520 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4848009d90 tx=0x7f484802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e75cd3df6145e68 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f7d0b400640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d0c15b590 0x7f7cf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7efe00a00640 1 -- 172.21.15.69:0/1014902792 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc03a070 con 0x7efe040b3320 2024-04-24T08:08:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7efe00a00640 1 -- 172.21.15.69:0/1014902792 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdfc032480 con 0x7efe040b3320 2024-04-24T08:08:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7efe00a00640 1 -- 172.21.15.69:0/1014902792 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc032850 con 0x7efe040b3320 2024-04-24T08:08:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f2452e00640 1 -- 172.21.15.69:0/3867439148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f244c068a80 con 0x7f244c0cfe40 2024-04-24T08:08:28.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7fcb40200640 1 -- 172.21.15.69:0/550564639 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380d1810 msgr2=0x7fcb3806cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7fcb40200640 1 --2- 172.21.15.69:0/550564639 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380d1810 0x7fcb3806cc00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009e50 tx=0x7fcb2c02a6f0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f7d0aa00640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cf00039b0 0x7f7cf0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f4dcee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dc806ee50 0x7f4dc81566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f4dcee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80b3320 0x7f4dc8156c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f4dcee00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8154f50 0x7f4dc80cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7f5905800640 1 Processor -- start 2024-04-24T08:08:28.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7f4dcee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc80b3d00 con 0x7f4dc806ee50 2024-04-24T08:08:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f7d02000640 1 -- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d0c15b590 msgr2=0x7f7cf00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f7d02000640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d0c15b590 0x7f7cf00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f7d02000640 1 -- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cf00039b0 msgr2=0x7f7cf0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f7d02000640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cf00039b0 0x7f7cf0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.967+0000 7f7d02000640 1 -- 172.21.15.69:0/480004790 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf0003520 con 0x7f7cf0004800 2024-04-24T08:08:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7f484ea00640 1 -- 172.21.15.69:0/3627957194 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848007740 con 0x7f485006e0f0 2024-04-24T08:08:28.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7f5905800640 1 -- start start 2024-04-24T08:08:28.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7efe09200640 1 -- 172.21.15.69:0/1014902792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe040b3320 msgr2=0x7efe04072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:28.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7efe09200640 1 --2- 172.21.15.69:0/1014902792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe040b3320 0x7efe04072480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00be90 tx=0x7efdfc02c6d0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7fcb40200640 1 -- 172.21.15.69:0/550564639 shutdown_connections 2024-04-24T08:08:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7fcb40200640 1 --2- 172.21.15.69:0/550564639 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380d1810 0x7fcb3806cc00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7fcb40200640 1 --2- 172.21.15.69:0/550564639 >> v2:172.21.15.121:3300/0 conn(0x7fcb380b3b80 0x7fcb380d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7f4dcee00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc80b3a20 con 0x7f4dc8154f50 2024-04-24T08:08:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7f4dcee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc80b3fe0 con 0x7f4dc80b3320 2024-04-24T08:08:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7f4dc7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dc806ee50 0x7f4dc81566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7f4dc7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dc806ee50 0x7f4dc81566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44358/0 (socket says 172.21.15.69:44358) 2024-04-24T08:08:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7f4dc7e00640 1 -- 172.21.15.69:0/1742093953 learned_addr learned my addr 172.21.15.69:0/1742093953 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7efe09200640 1 -- 172.21.15.69:0/1014902792 shutdown_connections 2024-04-24T08:08:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7efe09200640 1 --2- 172.21.15.69:0/1014902792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe040b3320 0x7efe04072480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7efe09200640 1 --2- 172.21.15.69:0/1014902792 >> v2:172.21.15.69:3300/0 conn(0x7efe0415ae50 0x7efe04154f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7efe09200640 1 -- 172.21.15.69:0/1014902792 >> 172.21.15.69:0/1014902792 conn(0x7efe04067940 msgr2=0x7efe0406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.968+0000 7fcb40200640 1 -- 172.21.15.69:0/550564639 >> 172.21.15.69:0/550564639 conn(0x7fcb3808bf60 msgr2=0x7fcb380d4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7fcb40200640 1 -- 172.21.15.69:0/550564639 shutdown_connections 2024-04-24T08:08:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7efe09200640 1 -- 172.21.15.69:0/1014902792 shutdown_connections 2024-04-24T08:08:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7f7d02000640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cf0004800 0x7f7cf0004c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0cf240 tx=0x7f7cf0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47528297d6d8e61 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.970+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf400a3b0 con 0x7f7cf0004800 2024-04-24T08:08:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.970+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf4033030 con 0x7f7cf0004800 2024-04-24T08:08:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.970+0000 7f7d0a000640 1 -- 172.21.15.69:0/480004790 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf40070d0 con 0x7f7cf0004800 2024-04-24T08:08:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.970+0000 7f2443e00640 1 -- 172.21.15.69:0/3867439148 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f243c015910 con 0x7f244c0cfe40 2024-04-24T08:08:28.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.970+0000 7f5905800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 0x7f58f8174190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f5905800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58f8175520 0x7f58f8049520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f5905800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58f8049a60 0x7f58f80da220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:28.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f5905800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8052b40 con 0x7f58e8004350 2024-04-24T08:08:28.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f5905800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8007390 con 0x7f58f8049a60 2024-04-24T08:08:28.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f5905800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f58f8052cc0 con 0x7f58f8175520 2024-04-24T08:08:28.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f58ffe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 0x7f58f8174190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f58ffe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 0x7f58f8174190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44362/0 (socket says 172.21.15.69:44362) 2024-04-24T08:08:28.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f58ffe00640 1 -- 172.21.15.69:0/2699700257 learned_addr learned my addr 172.21.15.69:0/2699700257 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:28.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f5904e00640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58f8049a60 0x7f58f80da220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:28.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7efe09200640 1 -- 172.21.15.69:0/1014902792 wait complete. 2024-04-24T08:08:29.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7f4dc7e00640 1 -- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80b3320 msgr2=0x7f4dc8156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7f4dc7e00640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80b3320 0x7f4dc8156c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7f4dc7e00640 1 -- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8154f50 msgr2=0x7f4dc80cf970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:08:29.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7f4dc7e00640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8154f50 0x7f4dc80cf970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7f4dc7e00640 1 -- 172.21.15.69:0/1742093953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc813e420 con 0x7f4dc806ee50 2024-04-24T08:08:29.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7f4dc7e00640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dc806ee50 0x7f4dc81566c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000ba50 tx=0x7f4dc000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7efe09200640 1 Processor -- start 2024-04-24T08:08:29.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f58ff400640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58f8175520 0x7f58f8049520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.970+0000 7f4dbfe00640 1 -- 172.21.15.69:0/1742093953 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0019070 con 0x7f4dc806ee50 2024-04-24T08:08:29.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.970+0000 7f4dcee00640 1 -- 172.21.15.69:0/1742093953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc813e790 con 0x7f4dc806ee50 2024-04-24T08:08:29.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.970+0000 7f4dbfe00640 1 -- 172.21.15.69:0/1742093953 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc000cbe0 con 0x7f4dc806ee50 2024-04-24T08:08:29.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.970+0000 7f4dcee00640 1 -- 172.21.15.69:0/1742093953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc813f030 con 0x7f4dc806ee50 2024-04-24T08:08:29.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.969+0000 7fcb40200640 1 -- 172.21.15.69:0/550564639 wait complete. 2024-04-24T08:08:29.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7fcb40200640 1 Processor -- start 2024-04-24T08:08:29.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f2443e00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f242c06caf0 0x7f242c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.971+0000 7f2443e00640 1 -- 172.21.15.69:0/3867439148 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f243c015470 con 0x7f244c0cfe40 2024-04-24T08:08:29.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.972+0000 7efe09200640 1 -- start start 2024-04-24T08:08:29.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.972+0000 7fcb40200640 1 -- start start 2024-04-24T08:08:29.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.972+0000 7f4dbfe00640 1 -- 172.21.15.69:0/1742093953 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0019070 con 0x7f4dc806ee50 2024-04-24T08:08:29.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.972+0000 7f58ffe00640 1 -- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58f8175520 msgr2=0x7f58f8049520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.972+0000 7f4dcee00640 1 -- 172.21.15.69:0/1742093953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da0005290 con 0x7f4dc806ee50 2024-04-24T08:08:29.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.972+0000 7f58ffe00640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58f8175520 0x7f58f8049520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.972+0000 7f58ffe00640 1 -- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58f8049a60 msgr2=0x7f58f80da220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.972+0000 7f58ffe00640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58f8049a60 0x7f58f80da220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.972+0000 7f58ffe00640 1 -- 172.21.15.69:0/2699700257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f8049f00 con 0x7f58e8004350 2024-04-24T08:08:29.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.972+0000 7f244b400640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f242c06caf0 0x7f242c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.973+0000 7f58ffe00640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 0x7f58f8174190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f58f400e860 tx=0x7f58f400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.973+0000 7f7d11800640 1 -- 172.21.15.69:0/480004790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cf0004800 msgr2=0x7f7cf0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.973+0000 7f7d11800640 1 --2- 172.21.15.69:0/480004790 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7cf0004800 0x7f7cf0004c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0cf240 tx=0x7f7cf0005830 comp rx=0 tx=0).stop 2024-04-24T08:08:29.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.973+0000 7f58f3e00640 1 -- 172.21.15.69:0/2699700257 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f4017070 con 0x7f58e8004350 2024-04-24T08:08:29.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.973+0000 7f58f3e00640 1 -- 172.21.15.69:0/2699700257 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58f400c940 con 0x7f58e8004350 2024-04-24T08:08:29.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.973+0000 7f58f3e00640 1 -- 172.21.15.69:0/2699700257 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f400cd10 con 0x7f58e8004350 2024-04-24T08:08:29.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.973+0000 7f7d0b400640 1 -- 172.21.15.69:0/480004790 reap_dead start 2024-04-24T08:08:29.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7f7d11800640 1 -- 172.21.15.69:0/480004790 shutdown_connections 2024-04-24T08:08:29.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7f7d11800640 1 -- 172.21.15.69:0/480004790 >> 172.21.15.69:0/480004790 conn(0x7f7d0c09a9d0 msgr2=0x7f7d0c0d14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7efe09200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe04067df0 0x7efe0414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7f5905800640 1 -- 172.21.15.69:0/2699700257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f8177dc0 con 0x7f58e8004350 2024-04-24T08:08:29.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7f5905800640 1 -- 172.21.15.69:0/2699700257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f8178380 con 0x7f58e8004350 2024-04-24T08:08:29.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7efe09200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe040b3320 0x7efe04157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7f4dbfe00640 1 -- 172.21.15.69:0/1742093953 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4dc001c070 con 0x7f4dc806ee50 2024-04-24T08:08:29.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7efe09200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0415ae50 0x7efe0406e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7efe09200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efe040b42c0 con 0x7efe0415ae50 2024-04-24T08:08:29.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7efe09200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efe040b3fe0 con 0x7efe040b3320 2024-04-24T08:08:29.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7efe09200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efe040b4ac0 con 0x7efe04067df0 2024-04-24T08:08:29.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7efe01e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe04067df0 0x7efe0414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7f7d11800640 1 -- 172.21.15.69:0/480004790 shutdown_connections 2024-04-24T08:08:29.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7efe01e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe04067df0 0x7efe0414fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:33878/0 (socket says 172.21.15.69:33878) 2024-04-24T08:08:29.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7efe01e00640 1 -- 172.21.15.69:0/2742328194 learned_addr learned my addr 172.21.15.69:0/2742328194 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:29.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.974+0000 7efe02800640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0415ae50 0x7efe0406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7efe01400640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe040b3320 0x7efe04157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7f244b400640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f242c06caf0 0x7f242c06efb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2438001550 tx=0x7f2438007990 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7f4dbfe00640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4da806caf0 0x7f4da806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7f7d11800640 1 -- 172.21.15.69:0/480004790 wait complete. 2024-04-24T08:08:29.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7fcb40200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380b3b80 0x7fcb381f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7f4dbfe00640 1 -- 172.21.15.69:0/1742093953 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4dc008b5c0 con 0x7f4dc806ee50 2024-04-24T08:08:29.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7efe02800640 1 -- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe04067df0 msgr2=0x7efe0414fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7fcb40200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb380d0590 0x7fcb380c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7efe02800640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe04067df0 0x7efe0414fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7efe02800640 1 -- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe040b3320 msgr2=0x7efe04157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7efe02800640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe040b3320 0x7efe04157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7efe02800640 1 -- 172.21.15.69:0/2742328194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0414ff60 con 0x7efe0415ae50 2024-04-24T08:08:29.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.975+0000 7fcb40200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb380d1810 0x7fcb3815e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.976+0000 7fcb3d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380b3b80 0x7fcb381f58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.976+0000 7efe02800640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0415ae50 0x7efe0406e910 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efdfc034000 tx=0x7efdfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.976+0000 7fcb3e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb380d1810 0x7fcb3815e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.976+0000 7fcb3d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380b3b80 0x7fcb381f58e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:33886/0 (socket says 172.21.15.69:33886) 2024-04-24T08:08:29.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.976+0000 7fcb3e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb380d1810 0x7fcb3815e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44382/0 (socket says 172.21.15.69:44382) 2024-04-24T08:08:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.976+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcb380d2300 con 0x7fcb380d1810 2024-04-24T08:08:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.976+0000 7fcb3d800640 1 -- 172.21.15.69:0/105082548 learned_addr learned my addr 172.21.15.69:0/105082548 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.976+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcb380d2ff0 con 0x7fcb380d0590 2024-04-24T08:08:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.976+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcb380d2480 con 0x7fcb380b3b80 2024-04-24T08:08:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.976+0000 7fcb3ce00640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb380d0590 0x7fcb380c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.977+0000 7f4dc7400640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4da806caf0 0x7f4da806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.977+0000 7fcb3e200640 1 -- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380b3b80 msgr2=0x7fcb381f58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.977+0000 7fcb3e200640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380b3b80 0x7fcb381f58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.977+0000 7fcb3e200640 1 -- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb380d0590 msgr2=0x7fcb380c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.977+0000 7fcb3e200640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb380d0590 0x7fcb380c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.977+0000 7f58f3e00640 1 -- 172.21.15.69:0/2699700257 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f58f401a070 con 0x7f58e8004350 2024-04-24T08:08:29.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.977+0000 7fcb3e200640 1 -- 172.21.15.69:0/105082548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb380b8790 con 0x7fcb380d1810 2024-04-24T08:08:29.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.977+0000 7fcb3ce00640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb380d0590 0x7fcb380c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:29.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.977+0000 7f5905800640 1 -- 172.21.15.69:0/2699700257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f80089d0 con 0x7f58e8004350 2024-04-24T08:08:29.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.977+0000 7fcb3e200640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb380d1810 0x7fcb3815e240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c030000 tx=0x7fcb2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.978+0000 7f58f3e00640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f58e006c940 0x7f58e006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.979+0000 7f4dc7400640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4da806caf0 0x7f4da806efb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4db8034000 tx=0x7f4db802c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.979+0000 7f7d11800640 1 Processor -- start 2024-04-24T08:08:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.979+0000 7f7d11800640 1 -- start start 2024-04-24T08:08:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.981+0000 7f7d11800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cf0004800 0x7f7d0c156c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.981+0000 7f7d11800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d0c158420 0x7f7d0c1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.981+0000 7f7d11800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c0c8700 0x7f7d0c0c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.981+0000 7f58ff400640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f58e006c940 0x7f58e006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.981+0000 7f58f3e00640 1 -- 172.21.15.69:0/2699700257 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f58f4090060 con 0x7f58e8004350 2024-04-24T08:08:29.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.981+0000 7f7d11800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c0d1f00 con 0x7f7d0c0c8700 2024-04-24T08:08:29.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.981+0000 7f7d11800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7d0c0d1d80 con 0x7f7d0c158420 2024-04-24T08:08:29.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.981+0000 7f7d11800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0c15b970 con 0x7f7cf0004800 2024-04-24T08:08:29.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7f4dbfe00640 1 -- 172.21.15.69:0/1742093953 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4dc00598d0 con 0x7f4dc806ee50 2024-04-24T08:08:29.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.981+0000 7efdfa000640 1 -- 172.21.15.69:0/2742328194 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc044020 con 0x7efe0415ae50 2024-04-24T08:08:29.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.981+0000 7efe09200640 1 -- 172.21.15.69:0/2742328194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe041e2380 con 0x7efe0415ae50 2024-04-24T08:08:29.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7efe09200640 1 -- 172.21.15.69:0/2742328194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe041e2940 con 0x7efe0415ae50 2024-04-24T08:08:29.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7fcb35600640 1 -- 172.21.15.69:0/105082548 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c041020 con 0x7fcb380d1810 2024-04-24T08:08:29.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7f7d0b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c0c8700 0x7f7d0c0c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7f7d0b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c0c8700 0x7f7d0c0c8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44386/0 (socket says 172.21.15.69:44386) 2024-04-24T08:08:29.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7f7d0b400640 1 -- 172.21.15.69:0/3059379845 learned_addr learned my addr 172.21.15.69:0/3059379845 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:29.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7f7d0aa00640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cf0004800 0x7f7d0c156c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7f7d02000640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d0c158420 0x7f7d0c1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7fcb35600640 1 -- 172.21.15.69:0/105082548 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb2c0077a0 con 0x7fcb380d1810 2024-04-24T08:08:29.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7efdfa000640 1 -- 172.21.15.69:0/2742328194 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdfc0359d0 con 0x7efe0415ae50 2024-04-24T08:08:29.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7efdfa000640 1 -- 172.21.15.69:0/2742328194 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc044020 con 0x7efe0415ae50 2024-04-24T08:08:29.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.983+0000 7f7d0b400640 1 -- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cf0004800 msgr2=0x7f7d0c156c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.983+0000 7f7d0b400640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cf0004800 0x7f7d0c156c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.983+0000 7f7d0b400640 1 -- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d0c158420 msgr2=0x7f7d0c1588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.983+0000 7f7d0b400640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d0c158420 0x7f7d0c1588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.983+0000 7f7d0b400640 1 -- 172.21.15.69:0/3059379845 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0c0c9480 con 0x7f7d0c0c8700 2024-04-24T08:08:29.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb380b8a70 con 0x7fcb380d1810 2024-04-24T08:08:29.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.982+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb380b9030 con 0x7fcb380d1810 2024-04-24T08:08:29.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.983+0000 7fcb35600640 1 -- 172.21.15.69:0/105082548 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c036040 con 0x7fcb380d1810 2024-04-24T08:08:29.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.983+0000 7f58ff400640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f58e006c940 0x7f58e006ee00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f58ec033000 tx=0x7f58ec008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.983+0000 7f7d0b400640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c0c8700 0x7f7d0c0c8b80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0cbaa0 tx=0x7f7d04011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.983+0000 7efe09200640 1 -- 172.21.15.69:0/2742328194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efddc005290 con 0x7efe0415ae50 2024-04-24T08:08:29.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.986+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb14005290 con 0x7fcb380d1810 2024-04-24T08:08:29.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.988+0000 7f58f3e00640 1 -- 172.21.15.69:0/2699700257 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58f4059120 con 0x7f58e8004350 2024-04-24T08:08:29.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.989+0000 7f484ea00640 1 -- 172.21.15.69:0/3627957194 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48480078e0 con 0x7f485006e0f0 2024-04-24T08:08:29.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.989+0000 7f7d03400640 1 -- 172.21.15.69:0/3059379845 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04004240 con 0x7f7d0c0c8700 2024-04-24T08:08:29.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.989+0000 7f7d11800640 1 -- 172.21.15.69:0/3059379845 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d0c0c9760 con 0x7f7d0c0c8700 2024-04-24T08:08:29.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.989+0000 7f7d11800640 1 -- 172.21.15.69:0/3059379845 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d0c0c9d20 con 0x7f7d0c0c8700 2024-04-24T08:08:29.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.990+0000 7f2443e00640 1 -- 172.21.15.69:0/3867439148 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f243c024080 con 0x7f244c0cfe40 2024-04-24T08:08:29.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.989+0000 7f484ea00640 1 -- 172.21.15.69:0/3627957194 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848007c90 con 0x7f485006e0f0 2024-04-24T08:08:29.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.990+0000 7f4856c00640 1 -- 172.21.15.69:0/3627957194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 msgr2=0x7f485006e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.990+0000 7f4856c00640 1 --2- 172.21.15.69:0/3627957194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 0x7f485006e520 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4848009d90 tx=0x7f484802a660 comp rx=0 tx=0).stop 2024-04-24T08:08:29.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.989+0000 7f7d03400640 1 -- 172.21.15.69:0/3059379845 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d0401b070 con 0x7f7d0c0c8700 2024-04-24T08:08:29.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.989+0000 7f7d03400640 1 -- 172.21.15.69:0/3059379845 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0400f710 con 0x7f7d0c0c8700 2024-04-24T08:08:29.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.991+0000 7f7d11800640 1 -- 172.21.15.69:0/3059379845 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce4005290 con 0x7f7d0c0c8700 2024-04-24T08:08:29.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.991+0000 7f4856c00640 1 -- 172.21.15.69:0/3627957194 shutdown_connections 2024-04-24T08:08:29.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.991+0000 7f4856c00640 1 --2- 172.21.15.69:0/3627957194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 0x7f485006e520 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.991+0000 7f4856c00640 1 --2- 172.21.15.69:0/3627957194 >> v2:172.21.15.121:3300/0 conn(0x7f48500b3d30 0x7f48500b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.991+0000 7f4856c00640 1 -- 172.21.15.69:0/3627957194 >> 172.21.15.69:0/3627957194 conn(0x7f4850067d90 msgr2=0x7f485006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.992+0000 7f4856c00640 1 -- 172.21.15.69:0/3627957194 shutdown_connections 2024-04-24T08:08:29.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.991+0000 7efdfa000640 1 -- 172.21.15.69:0/2742328194 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efdfc005d10 con 0x7efe0415ae50 2024-04-24T08:08:29.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.993+0000 7efdfa000640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efde006caf0 0x7efde006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.993+0000 7efdfa000640 1 -- 172.21.15.69:0/2742328194 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efdfc0acd10 con 0x7efe0415ae50 2024-04-24T08:08:29.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.992+0000 7f4856c00640 1 -- 172.21.15.69:0/3627957194 wait complete. 2024-04-24T08:08:29.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.993+0000 7efe01e00640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efde006caf0 0x7efde006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.994+0000 7fcb35600640 1 -- 172.21.15.69:0/105082548 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcb2c044020 con 0x7fcb380d1810 2024-04-24T08:08:29.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.994+0000 7f4856c00640 1 Processor -- start 2024-04-24T08:08:29.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.995+0000 7f4856c00640 1 -- start start 2024-04-24T08:08:29.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.995+0000 7efe01e00640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efde006caf0 0x7efde006efb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efdec000f60 tx=0x7efdec009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.995+0000 7fcb35600640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcb1806c940 0x7fcb1806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.996+0000 7fcb35600640 1 -- 172.21.15.69:0/105082548 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fcb2c0ac970 con 0x7fcb380d1810 2024-04-24T08:08:29.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.997+0000 7f4856c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 0x7f4850158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.996+0000 7fcb3d800640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcb1806c940 0x7fcb1806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.997+0000 7f4856c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48500b3d30 0x7f4850144c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.997+0000 7f4856c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4850162c70 0x7f48501451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.997+0000 7f4856c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f48500cfc40 con 0x7f4850162c70 2024-04-24T08:08:29.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.997+0000 7f4856c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4850163050 con 0x7f48500b3d30 2024-04-24T08:08:29.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.997+0000 7f4856c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f48500cfdc0 con 0x7f485006e0f0 2024-04-24T08:08:29.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.997+0000 7f4854c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4850162c70 0x7f48501451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.997+0000 7f4854c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4850162c70 0x7f48501451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44388/0 (socket says 172.21.15.69:44388) 2024-04-24T08:08:29.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.997+0000 7f4854c00640 1 -- 172.21.15.69:0/462444943 learned_addr learned my addr 172.21.15.69:0/462444943 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:29.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.997+0000 7f484fe00640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 0x7f4850158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.998+0000 7f484f400640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48500b3d30 0x7f4850144c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.998+0000 7f4854c00640 1 -- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 msgr2=0x7f4850158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.998+0000 7f4854c00640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 0x7f4850158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.998+0000 7f4854c00640 1 -- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48500b3d30 msgr2=0x7f4850144c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.998+0000 7f4854c00640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48500b3d30 0x7f4850144c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.998+0000 7f4854c00640 1 -- 172.21.15.69:0/462444943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48500c78a0 con 0x7f4850162c70 2024-04-24T08:08:29.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.998+0000 7efdfa000640 1 -- 172.21.15.69:0/2742328194 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efdfc04e020 con 0x7efe0415ae50 2024-04-24T08:08:29.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.999+0000 7fcb3d800640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcb1806c940 0x7fcb1806ee00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009c00 tx=0x7fcb30009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:28.999+0000 7f4854c00640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4850162c70 0x7f48501451c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4848009d90 tx=0x7f4848004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.007+0000 7fcb35600640 1 -- 172.21.15.69:0/105082548 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb2c04b030 con 0x7fcb380d1810 2024-04-24T08:08:29.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.009+0000 7f7d03400640 1 -- 172.21.15.69:0/3059379845 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d04012040 con 0x7f7d0c0c8700 2024-04-24T08:08:29.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.011+0000 7f7d03400640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ce806c920 0x7f7ce806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.011+0000 7f4847e00640 1 -- 172.21.15.69:0/462444943 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848009450 con 0x7f4850162c70 2024-04-24T08:08:29.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.012+0000 7f4847e00640 1 -- 172.21.15.69:0/462444943 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4848004240 con 0x7f4850162c70 2024-04-24T08:08:29.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.012+0000 7f4847e00640 1 -- 172.21.15.69:0/462444943 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848004630 con 0x7f4850162c70 2024-04-24T08:08:29.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.011+0000 7f7d03400640 1 -- 172.21.15.69:0/3059379845 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7d0408c330 con 0x7f7d0c0c8700 2024-04-24T08:08:29.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.013+0000 7f7d0aa00640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ce806c920 0x7f7ce806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.015+0000 7f7d0aa00640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ce806c920 0x7f7ce806ede0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009960 tx=0x7f7cfc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.013+0000 7f4856c00640 1 -- 172.21.15.69:0/462444943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4850145700 con 0x7f4850162c70 2024-04-24T08:08:29.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.013+0000 7f4856c00640 1 -- 172.21.15.69:0/462444943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4850145c50 con 0x7f4850162c70 2024-04-24T08:08:29.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.015+0000 7f4845600640 1 -- 172.21.15.69:0/462444943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4850068a80 con 0x7f4850162c70 2024-04-24T08:08:29.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.019+0000 7f4847e00640 1 -- 172.21.15.69:0/462444943 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4848039040 con 0x7f4850162c70 2024-04-24T08:08:29.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.020+0000 7f7d03400640 1 -- 172.21.15.69:0/3059379845 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d0405a0f0 con 0x7f7d0c0c8700 2024-04-24T08:08:29.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.021+0000 7f4847e00640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f483006c8f0 0x7f483006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.021+0000 7f4847e00640 1 -- 172.21.15.69:0/462444943 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f48480aceb0 con 0x7f4850162c70 2024-04-24T08:08:29.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.022+0000 7f484fe00640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f483006c8f0 0x7f483006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.027+0000 7f484fe00640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f483006c8f0 0x7f483006edb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4840005880 tx=0x7f4840009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.027+0000 7f4847e00640 1 -- 172.21.15.69:0/462444943 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f48500c78a0 con 0x7f4850162c70 2024-04-24T08:08:29.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.352+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f1da0052830 con 0x7f1da00e7a50 2024-04-24T08:08:29.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.352+0000 7f1d9be00640 1 -- 172.21.15.69:0/2536526502 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1da80f5410 con 0x7f1da00e7a50 2024-04-24T08:08:29.354 INFO:teuthology.orchestra.run.smithi069.stdout:55834574855 2024-04-24T08:08:29.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.356+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d8806cb20 msgr2=0x7f1d8806efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.356+0000 7f1daf200640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d8806cb20 0x7f1d8806efe0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009630 tx=0x7f1d94008040 comp rx=0 tx=0).stop 2024-04-24T08:08:29.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.356+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da00e7a50 msgr2=0x7f1da00cd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.356+0000 7f1daf200640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da00e7a50 0x7f1da00cd2e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1da806a740 tx=0x7f1da806b810 comp rx=0 tx=0).stop 2024-04-24T08:08:29.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.357+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 shutdown_connections 2024-04-24T08:08:29.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.357+0000 7f1daf200640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d8806cb20 0x7f1d8806efe0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.357+0000 7f1daf200640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da00e7a50 0x7f1da00cd2e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.357+0000 7f1daf200640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1da004a0a0 0x7f1da00ccda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.357+0000 7f1daf200640 1 --2- 172.21.15.69:0/2536526502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1da003c360 0x7f1da00d13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.357+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 >> 172.21.15.69:0/2536526502 conn(0x7f1da0056cd0 msgr2=0x7f1da00e6050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.357+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 shutdown_connections 2024-04-24T08:08:29.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.358+0000 7f1daf200640 1 -- 172.21.15.69:0/2536526502 wait complete. 2024-04-24T08:08:29.385 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-04-24T08:08:29.385 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:29.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.394+0000 7f4dcee00640 1 -- 172.21.15.69:0/1742093953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f4da0005580 con 0x7f4dc806ee50 2024-04-24T08:08:29.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.395+0000 7f4dbfe00640 1 -- 172.21.15.69:0/1742093953 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4dc00591d0 con 0x7f4dc806ee50 2024-04-24T08:08:29.396 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2024-04-24T08:08:29.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 -- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4da806caf0 msgr2=0x7f4da806efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4da806caf0 0x7f4da806efb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4db8034000 tx=0x7f4db802c000 comp rx=0 tx=0).stop 2024-04-24T08:08:29.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 -- 172.21.15.69:0/1742093953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dc806ee50 msgr2=0x7f4dc81566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dc806ee50 0x7f4dc81566c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4dc000ba50 tx=0x7f4dc000bf10 comp rx=0 tx=0).stop 2024-04-24T08:08:29.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 -- 172.21.15.69:0/1742093953 shutdown_connections 2024-04-24T08:08:29.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4da806caf0 0x7f4da806efb0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc8154f50 0x7f4dc80cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dc80b3320 0x7f4dc8156c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 --2- 172.21.15.69:0/1742093953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dc806ee50 0x7f4dc81566c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 -- 172.21.15.69:0/1742093953 >> 172.21.15.69:0/1742093953 conn(0x7f4dc8067940 msgr2=0x7f4dc806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.397+0000 7f4dcee00640 1 -- 172.21.15.69:0/1742093953 shutdown_connections 2024-04-24T08:08:29.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.398+0000 7f4dcee00640 1 -- 172.21.15.69:0/1742093953 wait complete. 2024-04-24T08:08:29.418 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-04-24T08:08:29.418 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:29.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.430+0000 7f2452e00640 1 -- 172.21.15.69:0/3867439148 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f244c0d1a00 con 0x7f244c0cfe40 2024-04-24T08:08:29.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.431+0000 7f2443e00640 1 -- 172.21.15.69:0/3867439148 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f244c2023a0 con 0x7f244c0cfe40 2024-04-24T08:08:29.432 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2024-04-24T08:08:29.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.434+0000 7f2452e00640 1 -- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f242c06caf0 msgr2=0x7f242c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.434+0000 7f2452e00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f242c06caf0 0x7f242c06efb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2438001550 tx=0x7f2438007990 comp rx=0 tx=0).stop 2024-04-24T08:08:29.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.434+0000 7f2452e00640 1 -- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0cfe40 msgr2=0x7f244c0cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.434+0000 7f2452e00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0cfe40 0x7f244c0cccc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f243c00bdb0 tx=0x7f243c00bea0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.434+0000 7f2452e00640 1 -- 172.21.15.69:0/3867439148 shutdown_connections 2024-04-24T08:08:29.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.435+0000 7f2452e00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f242c06caf0 0x7f242c06efb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.435+0000 7f2452e00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f244c0d10f0 0x7f244c201e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.435+0000 7f2452e00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f244c162150 0x7f244c1f7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.435+0000 7f2452e00640 1 --2- 172.21.15.69:0/3867439148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f244c0cfe40 0x7f244c0cccc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.435+0000 7f2452e00640 1 -- 172.21.15.69:0/3867439148 >> 172.21.15.69:0/3867439148 conn(0x7f244c08bf60 msgr2=0x7f244c0b7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.435+0000 7f2452e00640 1 -- 172.21.15.69:0/3867439148 shutdown_connections 2024-04-24T08:08:29.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.435+0000 7f2452e00640 1 -- 172.21.15.69:0/3867439148 wait complete. 2024-04-24T08:08:29.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.435+0000 7f5905800640 1 -- 172.21.15.69:0/2699700257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f58f8005390 con 0x7f58e8004350 2024-04-24T08:08:29.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.436+0000 7f58f3e00640 1 -- 172.21.15.69:0/2699700257 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f58f405d420 con 0x7f58e8004350 2024-04-24T08:08:29.437 INFO:teuthology.orchestra.run.smithi069.stdout:55834574855 2024-04-24T08:08:29.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.439+0000 7f5905800640 1 -- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f58e006c940 msgr2=0x7f58e006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.439+0000 7f5905800640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f58e006c940 0x7f58e006ee00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f58ec033000 tx=0x7f58ec008030 comp rx=0 tx=0).stop 2024-04-24T08:08:29.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.439+0000 7f5905800640 1 -- 172.21.15.69:0/2699700257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 msgr2=0x7f58f8174190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.439+0000 7f5905800640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 0x7f58f8174190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f58f400e860 tx=0x7f58f400ed20 comp rx=0 tx=0).stop 2024-04-24T08:08:29.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.440+0000 7f5905800640 1 -- 172.21.15.69:0/2699700257 shutdown_connections 2024-04-24T08:08:29.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.440+0000 7f5905800640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f58e006c940 0x7f58e006ee00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.440+0000 7f5905800640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58f8049a60 0x7f58f80da220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.440+0000 7f5905800640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58f8175520 0x7f58f8049520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.440+0000 7f5905800640 1 --2- 172.21.15.69:0/2699700257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58e8004350 0x7f58f8174190 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.440+0000 7f5905800640 1 -- 172.21.15.69:0/2699700257 >> 172.21.15.69:0/2699700257 conn(0x7f58f8009fb0 msgr2=0x7f58f804bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.440+0000 7f5905800640 1 -- 172.21.15.69:0/2699700257 shutdown_connections 2024-04-24T08:08:29.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.440+0000 7f5905800640 1 -- 172.21.15.69:0/2699700257 wait complete. 2024-04-24T08:08:29.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.442+0000 7efe09200640 1 -- 172.21.15.69:0/2742328194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7efddc005850 con 0x7efe0415ae50 2024-04-24T08:08:29.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.442+0000 7efdfa000640 1 -- 172.21.15.69:0/2742328194 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efdfc07b1b0 con 0x7efe0415ae50 2024-04-24T08:08:29.444 INFO:teuthology.orchestra.run.smithi069.stdout:55834574854 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 -- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efde006caf0 msgr2=0x7efde006efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efde006caf0 0x7efde006efb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7efdec000f60 tx=0x7efdec009040 comp rx=0 tx=0).stop 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 -- 172.21.15.69:0/2742328194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0415ae50 msgr2=0x7efe0406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0415ae50 0x7efe0406e910 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efdfc034000 tx=0x7efdfc004a40 comp rx=0 tx=0).stop 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 -- 172.21.15.69:0/2742328194 shutdown_connections 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efde006caf0 0x7efde006efb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0415ae50 0x7efe0406e910 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe040b3320 0x7efe04157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 --2- 172.21.15.69:0/2742328194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe04067df0 0x7efe0414fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 -- 172.21.15.69:0/2742328194 >> 172.21.15.69:0/2742328194 conn(0x7efe04067940 msgr2=0x7efe040b0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.446+0000 7efe09200640 1 -- 172.21.15.69:0/2742328194 shutdown_connections 2024-04-24T08:08:29.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.447+0000 7efe09200640 1 -- 172.21.15.69:0/2742328194 wait complete. 2024-04-24T08:08:29.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.447+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fcb14005850 con 0x7fcb380d1810 2024-04-24T08:08:29.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.448+0000 7fcb35600640 1 -- 172.21.15.69:0/105082548 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcb2c07b030 con 0x7fcb380d1810 2024-04-24T08:08:29.449 INFO:teuthology.orchestra.run.smithi069.stdout:55834574853 2024-04-24T08:08:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.451+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcb1806c940 msgr2=0x7fcb1806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.451+0000 7fcb40200640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcb1806c940 0x7fcb1806ee00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009c00 tx=0x7fcb30009190 comp rx=0 tx=0).stop 2024-04-24T08:08:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.451+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb380d1810 msgr2=0x7fcb3815e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.451+0000 7fcb40200640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb380d1810 0x7fcb3815e240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c030000 tx=0x7fcb2c004a40 comp rx=0 tx=0).stop 2024-04-24T08:08:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.452+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 shutdown_connections 2024-04-24T08:08:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.452+0000 7fcb40200640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcb1806c940 0x7fcb1806ee00 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009c00 tx=0x7fcb30009190 comp rx=0 tx=0).stop 2024-04-24T08:08:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.452+0000 7fcb40200640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcb380d1810 0x7fcb3815e240 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.452+0000 7fcb40200640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb380d0590 0x7fcb380c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.452+0000 7fcb40200640 1 --2- 172.21.15.69:0/105082548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcb380b3b80 0x7fcb381f58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.452+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 >> 172.21.15.69:0/105082548 conn(0x7fcb3808bf60 msgr2=0x7fcb380b6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.452+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 shutdown_connections 2024-04-24T08:08:29.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.452+0000 7fcb40200640 1 -- 172.21.15.69:0/105082548 wait complete. 2024-04-24T08:08:29.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.458+0000 7f7d11800640 1 -- 172.21.15.69:0/3059379845 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f7ce4005850 con 0x7f7d0c0c8700 2024-04-24T08:08:29.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.459+0000 7f7d03400640 1 -- 172.21.15.69:0/3059379845 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7d0405e3f0 con 0x7f7d0c0c8700 2024-04-24T08:08:29.460 INFO:teuthology.orchestra.run.smithi069.stdout:55834574856 2024-04-24T08:08:29.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.460+0000 7f4845600640 1 -- 172.21.15.69:0/462444943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f48500d0dc0 con 0x7f4850162c70 2024-04-24T08:08:29.463 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-04-24T08:08:29.463 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:29.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.462+0000 7f7d11800640 1 -- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ce806c920 msgr2=0x7f7ce806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.462+0000 7f7d11800640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ce806c920 0x7f7ce806ede0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009960 tx=0x7f7cfc008040 comp rx=0 tx=0).stop 2024-04-24T08:08:29.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.462+0000 7f4847e00640 1 -- 172.21.15.69:0/462444943 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4848040030 con 0x7f4850162c70 2024-04-24T08:08:29.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.462+0000 7f7d11800640 1 -- 172.21.15.69:0/3059379845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c0c8700 msgr2=0x7f7d0c0c8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.462+0000 7f7d11800640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c0c8700 0x7f7d0c0c8b80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c0cbaa0 tx=0x7f7d04011cf0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.463+0000 7f7d11800640 1 -- 172.21.15.69:0/3059379845 shutdown_connections 2024-04-24T08:08:29.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.463+0000 7f7d11800640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ce806c920 0x7f7ce806ede0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.463+0000 7f7d11800640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c0c8700 0x7f7d0c0c8b80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.463+0000 7f7d11800640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d0c158420 0x7f7d0c1588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.463+0000 7f7d11800640 1 --2- 172.21.15.69:0/3059379845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7cf0004800 0x7f7d0c156c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.463+0000 7f7d11800640 1 -- 172.21.15.69:0/3059379845 >> 172.21.15.69:0/3059379845 conn(0x7f7d0c09a9d0 msgr2=0x7f7d0c159820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.463+0000 7f7d11800640 1 -- 172.21.15.69:0/3059379845 shutdown_connections 2024-04-24T08:08:29.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.463+0000 7f7d11800640 1 -- 172.21.15.69:0/3059379845 wait complete. 2024-04-24T08:08:29.465 INFO:teuthology.orchestra.run.smithi069.stdout:55834574856 2024-04-24T08:08:29.467 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-04-24T08:08:29.468 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:29.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 -- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f483006c8f0 msgr2=0x7f483006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f483006c8f0 0x7f483006edb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4840005880 tx=0x7f4840009040 comp rx=0 tx=0).stop 2024-04-24T08:08:29.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 -- 172.21.15.69:0/462444943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4850162c70 msgr2=0x7f48501451c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4850162c70 0x7f48501451c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4848009d90 tx=0x7f4848004a40 comp rx=0 tx=0).stop 2024-04-24T08:08:29.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 -- 172.21.15.69:0/462444943 shutdown_connections 2024-04-24T08:08:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f483006c8f0 0x7f483006edb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4850162c70 0x7f48501451c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48500b3d30 0x7f4850144c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 --2- 172.21.15.69:0/462444943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f485006e0f0 0x7f4850158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 -- 172.21.15.69:0/462444943 >> 172.21.15.69:0/462444943 conn(0x7f4850067d90 msgr2=0x7f48500cd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 -- 172.21.15.69:0/462444943 shutdown_connections 2024-04-24T08:08:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.466+0000 7f4845600640 1 -- 172.21.15.69:0/462444943 wait complete. 2024-04-24T08:08:29.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-04-24T08:08:29.471 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:29.472 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-04-24T08:08:29.473 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:29.487 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-04-24T08:08:29.487 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:29.489 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2024-04-24T08:08:29.489 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:29.489 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T08:08:29.489 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:08:29.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.707+0000 7efe9d800640 1 Processor -- start 2024-04-24T08:08:29.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.707+0000 7efe9d800640 1 -- start start 2024-04-24T08:08:29.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.709+0000 7efe9d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efe980b3a00 0x7efe98073370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.709+0000 7efe9d800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe980b4070 con 0x7efe9806ca30 2024-04-24T08:08:29.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.709+0000 7efe9d800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe980b4350 con 0x7efe9815e750 2024-04-24T08:08:29.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.709+0000 7efe9d800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7efe980738b0 con 0x7efe980b3a00 2024-04-24T08:08:29.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.709+0000 7efe8fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efe980b3a00 0x7efe98073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.709+0000 7efe9ce00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7efe9806ca30 0x7efe980b5110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36598/0 (socket says 172.21.15.69:36598) 2024-04-24T08:08:29.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.709+0000 7efe9ce00640 1 -- 172.21.15.69:0/3038282066 learned_addr learned my addr 172.21.15.69:0/3038282066 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:29.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.710+0000 7efe8fe00640 1 -- 172.21.15.69:0/3038282066 >> v1:172.21.15.121:6789/0 conn(0x7efe9815e750 legacy=0x7efe980ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:29.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.710+0000 7efe8fe00640 1 -- 172.21.15.69:0/3038282066 >> v1:172.21.15.69:6789/0 conn(0x7efe9806ca30 legacy=0x7efe980b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.710+0000 7efe8fe00640 1 -- 172.21.15.69:0/3038282066 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe980b4630 con 0x7efe980b3a00 2024-04-24T08:08:29.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.710+0000 7efe97400640 1 -- 172.21.15.69:0/3038282066 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457414630 0 0) 0x7efe980b4070 con 0x7efe9806ca30 2024-04-24T08:08:29.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.711+0000 7efe8fe00640 1 --2- 172.21.15.69:0/3038282066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe980b3a00 0x7efe98073370 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efe800097a0 tx=0x7efe8002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9005856074db4eab server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.711+0000 7efe97400640 1 -- 172.21.15.69:0/3038282066 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe80037070 con 0x7efe980b3a00 2024-04-24T08:08:29.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.711+0000 7efe97400640 1 -- 172.21.15.69:0/3038282066 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe80032440 con 0x7efe980b3a00 2024-04-24T08:08:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.711+0000 7efe97400640 1 -- 172.21.15.69:0/3038282066 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe80032810 con 0x7efe980b3a00 2024-04-24T08:08:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.712+0000 7efe9d800640 1 -- 172.21.15.69:0/3038282066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe980b3a00 msgr2=0x7efe98073370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.712+0000 7efe9d800640 1 --2- 172.21.15.69:0/3038282066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe980b3a00 0x7efe98073370 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efe800097a0 tx=0x7efe8002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.712+0000 7efe9d800640 1 -- 172.21.15.69:0/3038282066 shutdown_connections 2024-04-24T08:08:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.712+0000 7efe9d800640 1 --2- 172.21.15.69:0/3038282066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe980b3a00 0x7efe98073370 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.712+0000 7efe9d800640 1 -- 172.21.15.69:0/3038282066 >> 172.21.15.69:0/3038282066 conn(0x7efe980722b0 msgr2=0x7efe980726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.712+0000 7efe9d800640 1 -- 172.21.15.69:0/3038282066 shutdown_connections 2024-04-24T08:08:29.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.712+0000 7efe9d800640 1 -- 172.21.15.69:0/3038282066 wait complete. 2024-04-24T08:08:29.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.714+0000 7efe9d800640 1 Processor -- start 2024-04-24T08:08:29.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.714+0000 7efe9d800640 1 -- start start 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.715+0000 7efe9d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe9806ca30 0x7efe980cf1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.715+0000 7efe9d800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe980b3a00 0x7efe98150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.715+0000 7efe9d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9815e750 0x7efe981f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.715+0000 7efe9d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efe980b4c80 con 0x7efe9815e750 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.715+0000 7efe9d800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efe980b5820 con 0x7efe980b3a00 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.715+0000 7efe9d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efe980b59a0 con 0x7efe9806ca30 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe9ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9815e750 0x7efe981f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe9ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9815e750 0x7efe981f81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44402/0 (socket says 172.21.15.69:44402) 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe9ce00640 1 -- 172.21.15.69:0/2259984587 learned_addr learned my addr 172.21.15.69:0/2259984587 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe9ce00640 1 -- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe9806ca30 msgr2=0x7efe980cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe9ce00640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe9806ca30 0x7efe980cf1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe9ce00640 1 -- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe980b3a00 msgr2=0x7efe98150260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe8fe00640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe980b3a00 0x7efe98150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe9ce00640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe980b3a00 0x7efe98150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe9ce00640 1 -- 172.21.15.69:0/2259984587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe981618f0 con 0x7efe9815e750 2024-04-24T08:08:29.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe9ce00640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9815e750 0x7efe981f81f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efe9000c7d0 tx=0x7efe9000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.716+0000 7efe94c00640 1 -- 172.21.15.69:0/2259984587 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe90018070 con 0x7efe9815e750 2024-04-24T08:08:29.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.717+0000 7efe94c00640 1 -- 172.21.15.69:0/2259984587 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe9000a640 con 0x7efe9815e750 2024-04-24T08:08:29.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.717+0000 7efe94c00640 1 -- 172.21.15.69:0/2259984587 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9000aa70 con 0x7efe9815e750 2024-04-24T08:08:29.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.717+0000 7efe9d800640 1 -- 172.21.15.69:0/2259984587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe98161bd0 con 0x7efe9815e750 2024-04-24T08:08:29.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.717+0000 7efe9d800640 1 -- 172.21.15.69:0/2259984587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe98152f80 con 0x7efe9815e750 2024-04-24T08:08:29.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.718+0000 7efe9d800640 1 -- 172.21.15.69:0/2259984587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe98075d50 con 0x7efe9815e750 2024-04-24T08:08:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.719+0000 7efe94c00640 1 -- 172.21.15.69:0/2259984587 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efe9001b070 con 0x7efe9815e750 2024-04-24T08:08:29.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.720+0000 7efe94c00640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe7406c940 0x7efe7406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:29.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.720+0000 7efe94c00640 1 -- 172.21.15.69:0/2259984587 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efe9008b160 con 0x7efe9815e750 2024-04-24T08:08:29.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.721+0000 7efe97e00640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe7406c940 0x7efe7406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:29.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.722+0000 7efe97e00640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe7406c940 0x7efe7406ee00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efe88006fd0 tx=0x7efe88006bc0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:29.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.724+0000 7efe94c00640 1 -- 172.21.15.69:0/2259984587 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe90059670 con 0x7efe9815e750 2024-04-24T08:08:29.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.909+0000 7efe9d800640 1 -- 172.21.15.69:0/2259984587 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efe9815ded0 con 0x7efe7406c940 2024-04-24T08:08:29.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.928+0000 7efe94c00640 1 -- 172.21.15.69:0/2259984587 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7efe9815ded0 con 0x7efe7406c940 2024-04-24T08:08:29.933 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:29.934 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:08:29.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 -- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe7406c940 msgr2=0x7efe7406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe7406c940 0x7efe7406ee00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efe88006fd0 tx=0x7efe88006bc0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 -- 172.21.15.69:0/2259984587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9815e750 msgr2=0x7efe981f81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:29.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9815e750 0x7efe981f81f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efe9000c7d0 tx=0x7efe9000cc90 comp rx=0 tx=0).stop 2024-04-24T08:08:29.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 -- 172.21.15.69:0/2259984587 shutdown_connections 2024-04-24T08:08:29.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe7406c940 0x7efe7406ee00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe9815e750 0x7efe981f81f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe980b3a00 0x7efe98150260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 --2- 172.21.15.69:0/2259984587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efe9806ca30 0x7efe980cf1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:29.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 -- 172.21.15.69:0/2259984587 >> 172.21.15.69:0/2259984587 conn(0x7efe980722b0 msgr2=0x7efe980ad840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:29.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 -- 172.21.15.69:0/2259984587 shutdown_connections 2024-04-24T08:08:29.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:29.933+0000 7efe9d800640 1 -- 172.21.15.69:0/2259984587 wait complete. 2024-04-24T08:08:29.951 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-24T08:08:28.663055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218992,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755728,"statfs":{"total":773094113280,"available":772869865472,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.555218"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539355+0000","last_change":"2024-04-24T08:08:20.145550+0000","last_active":"2024-04-24T08:08:22.539355+0000","last_peered":"2024-04-24T08:08:22.539355+0000","last_clean":"2024-04-24T08:08:22.539355+0000","last_became_active":"2024-04-24T08:08:20.144816+0000","last_became_peered":"2024-04-24T08:08:20.144816+0000","last_unstale":"2024-04-24T08:08:22.539355+0000","last_undegraded":"2024-04-24T08:08:22.539355+0000","last_fullsized":"2024-04-24T08:08:22.539355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:03:56.009457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.122230+0000","last_change":"2024-04-24T08:08:20.161217+0000","last_active":"2024-04-24T08:08:21.122230+0000","last_peered":"2024-04-24T08:08:21.122230+0000","last_clean":"2024-04-24T08:08:21.122230+0000","last_became_active":"2024-04-24T08:08:20.159365+0000","last_became_peered":"2024-04-24T08:08:20.159365+0000","last_unstale":"2024-04-24T08:08:21.122230+0000","last_undegraded":"2024-04-24T08:08:21.122230+0000","last_fullsized":"2024-04-24T08:08:21.122230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887063+0000","last_change":"2024-04-24T08:08:20.141345+0000","last_active":"2024-04-24T08:08:21.887063+0000","last_peered":"2024-04-24T08:08:21.887063+0000","last_clean":"2024-04-24T08:08:21.887063+0000","last_became_active":"2024-04-24T08:08:20.141134+0000","last_became_peered":"2024-04-24T08:08:20.141134+0000","last_unstale":"2024-04-24T08:08:21.887063+0000","last_undegraded":"2024-04-24T08:08:21.887063+0000","last_fullsized":"2024-04-24T08:08:21.887063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125297+0000","last_change":"2024-04-24T08:08:20.164461+0000","last_active":"2024-04-24T08:08:21.125297+0000","last_peered":"2024-04-24T08:08:21.125297+0000","last_clean":"2024-04-24T08:08:21.125297+0000","last_became_active":"2024-04-24T08:08:20.164118+0000","last_became_peered":"2024-04-24T08:08:20.164118+0000","last_unstale":"2024-04-24T08:08:21.125297+0000","last_undegraded":"2024-04-24T08:08:21.125297+0000","last_fullsized":"2024-04-24T08:08:21.125297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:34.119848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538852+0000","last_change":"2024-04-24T08:08:20.146319+0000","last_active":"2024-04-24T08:08:22.538852+0000","last_peered":"2024-04-24T08:08:22.538852+0000","last_clean":"2024-04-24T08:08:22.538852+0000","last_became_active":"2024-04-24T08:08:20.145969+0000","last_became_peered":"2024-04-24T08:08:20.145969+0000","last_unstale":"2024-04-24T08:08:22.538852+0000","last_undegraded":"2024-04-24T08:08:22.538852+0000","last_fullsized":"2024-04-24T08:08:22.538852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:32:29.020308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123393+0000","last_change":"2024-04-24T08:08:20.149172+0000","last_active":"2024-04-24T08:08:21.123393+0000","last_peered":"2024-04-24T08:08:21.123393+0000","last_clean":"2024-04-24T08:08:21.123393+0000","last_became_active":"2024-04-24T08:08:20.148447+0000","last_became_peered":"2024-04-24T08:08:20.148447+0000","last_unstale":"2024-04-24T08:08:21.123393+0000","last_undegraded":"2024-04-24T08:08:21.123393+0000","last_fullsized":"2024-04-24T08:08:21.123393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:53.941683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538379+0000","last_change":"2024-04-24T08:08:20.132800+0000","last_active":"2024-04-24T08:08:22.538379+0000","last_peered":"2024-04-24T08:08:22.538379+0000","last_clean":"2024-04-24T08:08:22.538379+0000","last_became_active":"2024-04-24T08:08:20.132442+0000","last_became_peered":"2024-04-24T08:08:20.132442+0000","last_unstale":"2024-04-24T08:08:22.538379+0000","last_undegraded":"2024-04-24T08:08:22.538379+0000","last_fullsized":"2024-04-24T08:08:22.538379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:34.119848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537521+0000","last_change":"2024-04-24T08:08:20.153541+0000","last_active":"2024-04-24T08:08:22.537521+0000","last_peered":"2024-04-24T08:08:22.537521+0000","last_clean":"2024-04-24T08:08:22.537521+0000","last_became_active":"2024-04-24T08:08:20.153214+0000","last_became_peered":"2024-04-24T08:08:20.153214+0000","last_unstale":"2024-04-24T08:08:22.537521+0000","last_undegraded":"2024-04-24T08:08:22.537521+0000","last_fullsized":"2024-04-24T08:08:22.537521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539513+0000","last_change":"2024-04-24T08:08:20.136784+0000","last_active":"2024-04-24T08:08:22.539513+0000","last_peered":"2024-04-24T08:08:22.539513+0000","last_clean":"2024-04-24T08:08:22.539513+0000","last_became_active":"2024-04-24T08:08:20.136552+0000","last_became_peered":"2024-04-24T08:08:20.136552+0000","last_unstale":"2024-04-24T08:08:22.539513+0000","last_undegraded":"2024-04-24T08:08:22.539513+0000","last_fullsized":"2024-04-24T08:08:22.539513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:02:22.840489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152584+0000","last_change":"2024-04-24T08:08:20.143039+0000","last_active":"2024-04-24T08:08:21.152584+0000","last_peered":"2024-04-24T08:08:21.152584+0000","last_clean":"2024-04-24T08:08:21.152584+0000","last_became_active":"2024-04-24T08:08:20.142388+0000","last_became_peered":"2024-04-24T08:08:20.142388+0000","last_unstale":"2024-04-24T08:08:21.152584+0000","last_undegraded":"2024-04-24T08:08:21.152584+0000","last_fullsized":"2024-04-24T08:08:21.152584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152537+0000","last_change":"2024-04-24T08:08:20.143995+0000","last_active":"2024-04-24T08:08:21.152537+0000","last_peered":"2024-04-24T08:08:21.152537+0000","last_clean":"2024-04-24T08:08:21.152537+0000","last_became_active":"2024-04-24T08:08:20.143674+0000","last_became_peered":"2024-04-24T08:08:20.143674+0000","last_unstale":"2024-04-24T08:08:21.152537+0000","last_undegraded":"2024-04-24T08:08:21.152537+0000","last_fullsized":"2024-04-24T08:08:21.152537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125663+0000","last_change":"2024-04-24T08:08:20.146375+0000","last_active":"2024-04-24T08:08:21.125663+0000","last_peered":"2024-04-24T08:08:21.125663+0000","last_clean":"2024-04-24T08:08:21.125663+0000","last_became_active":"2024-04-24T08:08:20.146053+0000","last_became_peered":"2024-04-24T08:08:20.146053+0000","last_unstale":"2024-04-24T08:08:21.125663+0000","last_undegraded":"2024-04-24T08:08:21.125663+0000","last_fullsized":"2024-04-24T08:08:21.125663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:50.353563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538060+0000","last_change":"2024-04-24T08:08:20.151989+0000","last_active":"2024-04-24T08:08:22.538060+0000","last_peered":"2024-04-24T08:08:22.538060+0000","last_clean":"2024-04-24T08:08:22.538060+0000","last_became_active":"2024-04-24T08:08:20.150076+0000","last_became_peered":"2024-04-24T08:08:20.150076+0000","last_unstale":"2024-04-24T08:08:22.538060+0000","last_undegraded":"2024-04-24T08:08:22.538060+0000","last_fullsized":"2024-04-24T08:08:22.538060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.886831+0000","last_change":"2024-04-24T08:08:20.157703+0000","last_active":"2024-04-24T08:08:21.886831+0000","last_peered":"2024-04-24T08:08:21.886831+0000","last_clean":"2024-04-24T08:08:21.886831+0000","last_became_active":"2024-04-24T08:08:20.151876+0000","last_became_peered":"2024-04-24T08:08:20.151876+0000","last_unstale":"2024-04-24T08:08:21.886831+0000","last_undegraded":"2024-04-24T08:08:21.886831+0000","last_fullsized":"2024-04-24T08:08:21.886831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:41.389051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121308+0000","last_change":"2024-04-24T08:08:20.146100+0000","last_active":"2024-04-24T08:08:21.121308+0000","last_peered":"2024-04-24T08:08:21.121308+0000","last_clean":"2024-04-24T08:08:21.121308+0000","last_became_active":"2024-04-24T08:08:20.144236+0000","last_became_peered":"2024-04-24T08:08:20.144236+0000","last_unstale":"2024-04-24T08:08:21.121308+0000","last_undegraded":"2024-04-24T08:08:21.121308+0000","last_fullsized":"2024-04-24T08:08:21.121308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123180+0000","last_change":"2024-04-24T08:08:20.137406+0000","last_active":"2024-04-24T08:08:21.123180+0000","last_peered":"2024-04-24T08:08:21.123180+0000","last_clean":"2024-04-24T08:08:21.123180+0000","last_became_active":"2024-04-24T08:08:20.137109+0000","last_became_peered":"2024-04-24T08:08:20.137109+0000","last_unstale":"2024-04-24T08:08:21.123180+0000","last_undegraded":"2024-04-24T08:08:21.123180+0000","last_fullsized":"2024-04-24T08:08:21.123180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126733+0000","last_change":"2024-04-24T08:08:20.141553+0000","last_active":"2024-04-24T08:08:21.126733+0000","last_peered":"2024-04-24T08:08:21.126733+0000","last_clean":"2024-04-24T08:08:21.126733+0000","last_became_active":"2024-04-24T08:08:20.141163+0000","last_became_peered":"2024-04-24T08:08:20.141163+0000","last_unstale":"2024-04-24T08:08:21.126733+0000","last_undegraded":"2024-04-24T08:08:21.126733+0000","last_fullsized":"2024-04-24T08:08:21.126733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:19:04.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362588+0000","last_change":"2024-04-24T08:08:20.143805+0000","last_active":"2024-04-24T08:08:21.362588+0000","last_peered":"2024-04-24T08:08:21.362588+0000","last_clean":"2024-04-24T08:08:21.362588+0000","last_became_active":"2024-04-24T08:08:20.143427+0000","last_became_peered":"2024-04-24T08:08:20.143427+0000","last_unstale":"2024-04-24T08:08:21.362588+0000","last_undegraded":"2024-04-24T08:08:21.362588+0000","last_fullsized":"2024-04-24T08:08:21.362588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:10:52.052957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539464+0000","last_change":"2024-04-24T08:08:20.139941+0000","last_active":"2024-04-24T08:08:22.539464+0000","last_peered":"2024-04-24T08:08:22.539464+0000","last_clean":"2024-04-24T08:08:22.539464+0000","last_became_active":"2024-04-24T08:08:20.139629+0000","last_became_peered":"2024-04-24T08:08:20.139629+0000","last_unstale":"2024-04-24T08:08:22.539464+0000","last_undegraded":"2024-04-24T08:08:22.539464+0000","last_fullsized":"2024-04-24T08:08:22.539464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:19:04.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152372+0000","last_change":"2024-04-24T08:08:20.148714+0000","last_active":"2024-04-24T08:08:21.152372+0000","last_peered":"2024-04-24T08:08:21.152372+0000","last_clean":"2024-04-24T08:08:21.152372+0000","last_became_active":"2024-04-24T08:08:20.148418+0000","last_became_peered":"2024-04-24T08:08:20.148418+0000","last_unstale":"2024-04-24T08:08:21.152372+0000","last_undegraded":"2024-04-24T08:08:21.152372+0000","last_fullsized":"2024-04-24T08:08:21.152372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887385+0000","last_change":"2024-04-24T08:08:20.131644+0000","last_active":"2024-04-24T08:08:21.887385+0000","last_peered":"2024-04-24T08:08:21.887385+0000","last_clean":"2024-04-24T08:08:21.887385+0000","last_became_active":"2024-04-24T08:08:20.131283+0000","last_became_peered":"2024-04-24T08:08:20.131283+0000","last_unstale":"2024-04-24T08:08:21.887385+0000","last_undegraded":"2024-04-24T08:08:21.887385+0000","last_fullsized":"2024-04-24T08:08:21.887385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123520+0000","last_change":"2024-04-24T08:08:20.137461+0000","last_active":"2024-04-24T08:08:21.123520+0000","last_peered":"2024-04-24T08:08:21.123520+0000","last_clean":"2024-04-24T08:08:21.123520+0000","last_became_active":"2024-04-24T08:08:20.137181+0000","last_became_peered":"2024-04-24T08:08:20.137181+0000","last_unstale":"2024-04-24T08:08:21.123520+0000","last_undegraded":"2024-04-24T08:08:21.123520+0000","last_fullsized":"2024-04-24T08:08:21.123520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:15.025368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537451+0000","last_change":"2024-04-24T08:08:20.146731+0000","last_active":"2024-04-24T08:08:22.537451+0000","last_peered":"2024-04-24T08:08:22.537451+0000","last_clean":"2024-04-24T08:08:22.537451+0000","last_became_active":"2024-04-24T08:08:20.146216+0000","last_became_peered":"2024-04-24T08:08:20.146216+0000","last_unstale":"2024-04-24T08:08:22.537451+0000","last_undegraded":"2024-04-24T08:08:22.537451+0000","last_fullsized":"2024-04-24T08:08:22.537451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123388+0000","last_change":"2024-04-24T08:08:20.159848+0000","last_active":"2024-04-24T08:08:21.123388+0000","last_peered":"2024-04-24T08:08:21.123388+0000","last_clean":"2024-04-24T08:08:21.123388+0000","last_became_active":"2024-04-24T08:08:20.159530+0000","last_became_peered":"2024-04-24T08:08:20.159530+0000","last_unstale":"2024-04-24T08:08:21.123388+0000","last_undegraded":"2024-04-24T08:08:21.123388+0000","last_fullsized":"2024-04-24T08:08:21.123388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537893+0000","last_change":"2024-04-24T08:08:20.153648+0000","last_active":"2024-04-24T08:08:22.537893+0000","last_peered":"2024-04-24T08:08:22.537893+0000","last_clean":"2024-04-24T08:08:22.537893+0000","last_became_active":"2024-04-24T08:08:20.153373+0000","last_became_peered":"2024-04-24T08:08:20.153373+0000","last_unstale":"2024-04-24T08:08:22.537893+0000","last_undegraded":"2024-04-24T08:08:22.537893+0000","last_fullsized":"2024-04-24T08:08:22.537893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152686+0000","last_change":"2024-04-24T08:08:20.136593+0000","last_active":"2024-04-24T08:08:21.152686+0000","last_peered":"2024-04-24T08:08:21.152686+0000","last_clean":"2024-04-24T08:08:21.152686+0000","last_became_active":"2024-04-24T08:08:20.136373+0000","last_became_peered":"2024-04-24T08:08:20.136373+0000","last_unstale":"2024-04-24T08:08:21.152686+0000","last_undegraded":"2024-04-24T08:08:21.152686+0000","last_fullsized":"2024-04-24T08:08:21.152686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121751+0000","last_change":"2024-04-24T08:08:20.161976+0000","last_active":"2024-04-24T08:08:21.121751+0000","last_peered":"2024-04-24T08:08:21.121751+0000","last_clean":"2024-04-24T08:08:21.121751+0000","last_became_active":"2024-04-24T08:08:20.161665+0000","last_became_peered":"2024-04-24T08:08:20.161665+0000","last_unstale":"2024-04-24T08:08:21.121751+0000","last_undegraded":"2024-04-24T08:08:21.121751+0000","last_fullsized":"2024-04-24T08:08:21.121751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538337+0000","last_change":"2024-04-24T08:08:20.155775+0000","last_active":"2024-04-24T08:08:22.538337+0000","last_peered":"2024-04-24T08:08:22.538337+0000","last_clean":"2024-04-24T08:08:22.538337+0000","last_became_active":"2024-04-24T08:08:20.152792+0000","last_became_peered":"2024-04-24T08:08:20.152792+0000","last_unstale":"2024-04-24T08:08:22.538337+0000","last_undegraded":"2024-04-24T08:08:22.538337+0000","last_fullsized":"2024-04-24T08:08:22.538337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:07:27.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887649+0000","last_change":"2024-04-24T08:08:20.152662+0000","last_active":"2024-04-24T08:08:21.887649+0000","last_peered":"2024-04-24T08:08:21.887649+0000","last_clean":"2024-04-24T08:08:21.887649+0000","last_became_active":"2024-04-24T08:08:20.143770+0000","last_became_peered":"2024-04-24T08:08:20.143770+0000","last_unstale":"2024-04-24T08:08:21.887649+0000","last_undegraded":"2024-04-24T08:08:21.887649+0000","last_fullsized":"2024-04-24T08:08:21.887649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123612+0000","last_change":"2024-04-24T08:08:20.164390+0000","last_active":"2024-04-24T08:08:21.123612+0000","last_peered":"2024-04-24T08:08:21.123612+0000","last_clean":"2024-04-24T08:08:21.123612+0000","last_became_active":"2024-04-24T08:08:20.164063+0000","last_became_peered":"2024-04-24T08:08:20.164063+0000","last_unstale":"2024-04-24T08:08:21.123612+0000","last_undegraded":"2024-04-24T08:08:21.123612+0000","last_fullsized":"2024-04-24T08:08:21.123612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:07:27.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362285+0000","last_change":"2024-04-24T08:08:20.140288+0000","last_active":"2024-04-24T08:08:21.362285+0000","last_peered":"2024-04-24T08:08:21.362285+0000","last_clean":"2024-04-24T08:08:21.362285+0000","last_became_active":"2024-04-24T08:08:20.139983+0000","last_became_peered":"2024-04-24T08:08:20.139983+0000","last_unstale":"2024-04-24T08:08:21.362285+0000","last_undegraded":"2024-04-24T08:08:21.362285+0000","last_fullsized":"2024-04-24T08:08:21.362285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:46.310762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151856+0000","last_change":"2024-04-24T08:08:20.143305+0000","last_active":"2024-04-24T08:08:21.151856+0000","last_peered":"2024-04-24T08:08:21.151856+0000","last_clean":"2024-04-24T08:08:21.151856+0000","last_became_active":"2024-04-24T08:08:20.142984+0000","last_became_peered":"2024-04-24T08:08:20.142984+0000","last_unstale":"2024-04-24T08:08:21.151856+0000","last_undegraded":"2024-04-24T08:08:21.151856+0000","last_fullsized":"2024-04-24T08:08:21.151856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:40.157193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124584+0000","last_change":"2024-04-24T08:08:20.139312+0000","last_active":"2024-04-24T08:08:21.124584+0000","last_peered":"2024-04-24T08:08:21.124584+0000","last_clean":"2024-04-24T08:08:21.124584+0000","last_became_active":"2024-04-24T08:08:20.137881+0000","last_became_peered":"2024-04-24T08:08:20.137881+0000","last_unstale":"2024-04-24T08:08:21.124584+0000","last_undegraded":"2024-04-24T08:08:21.124584+0000","last_fullsized":"2024-04-24T08:08:21.124584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:28.189776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538238+0000","last_change":"2024-04-24T08:08:19.144554+0000","last_active":"2024-04-24T08:08:22.538238+0000","last_peered":"2024-04-24T08:08:22.538238+0000","last_clean":"2024-04-24T08:08:22.538238+0000","last_became_active":"2024-04-24T08:08:19.144285+0000","last_became_peered":"2024-04-24T08:08:19.144285+0000","last_unstale":"2024-04-24T08:08:22.538238+0000","last_undegraded":"2024-04-24T08:08:22.538238+0000","last_fullsized":"2024-04-24T08:08:22.538238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:02:50.869449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887732+0000","last_change":"2024-04-24T08:08:20.153872+0000","last_active":"2024-04-24T08:08:21.887732+0000","last_peered":"2024-04-24T08:08:21.887732+0000","last_clean":"2024-04-24T08:08:21.887732+0000","last_became_active":"2024-04-24T08:08:20.153593+0000","last_became_peered":"2024-04-24T08:08:20.153593+0000","last_unstale":"2024-04-24T08:08:21.887732+0000","last_undegraded":"2024-04-24T08:08:21.887732+0000","last_fullsized":"2024-04-24T08:08:21.887732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:53.941683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151661+0000","last_change":"2024-04-24T08:08:19.145643+0000","last_active":"2024-04-24T08:08:21.151661+0000","last_peered":"2024-04-24T08:08:21.151661+0000","last_clean":"2024-04-24T08:08:21.151661+0000","last_became_active":"2024-04-24T08:08:19.141228+0000","last_became_peered":"2024-04-24T08:08:19.141228+0000","last_unstale":"2024-04-24T08:08:21.151661+0000","last_undegraded":"2024-04-24T08:08:21.151661+0000","last_fullsized":"2024-04-24T08:08:21.151661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537587+0000","last_change":"2024-04-24T08:08:19.150248+0000","last_active":"2024-04-24T08:08:22.537587+0000","last_peered":"2024-04-24T08:08:22.537587+0000","last_clean":"2024-04-24T08:08:22.537587+0000","last_became_active":"2024-04-24T08:08:19.149979+0000","last_became_peered":"2024-04-24T08:08:19.149979+0000","last_unstale":"2024-04-24T08:08:22.537587+0000","last_undegraded":"2024-04-24T08:08:22.537587+0000","last_fullsized":"2024-04-24T08:08:22.537587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:13:42.392189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538422+0000","last_change":"2024-04-24T08:08:20.155184+0000","last_active":"2024-04-24T08:08:22.538422+0000","last_peered":"2024-04-24T08:08:22.538422+0000","last_clean":"2024-04-24T08:08:22.538422+0000","last_became_active":"2024-04-24T08:08:20.154858+0000","last_became_peered":"2024-04-24T08:08:20.154858+0000","last_unstale":"2024-04-24T08:08:22.538422+0000","last_undegraded":"2024-04-24T08:08:22.538422+0000","last_fullsized":"2024-04-24T08:08:22.538422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:34.119848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123341+0000","last_change":"2024-04-24T08:08:19.138408+0000","last_active":"2024-04-24T08:08:21.123341+0000","last_peered":"2024-04-24T08:08:21.123341+0000","last_clean":"2024-04-24T08:08:21.123341+0000","last_became_active":"2024-04-24T08:08:19.137042+0000","last_became_peered":"2024-04-24T08:08:19.137042+0000","last_unstale":"2024-04-24T08:08:21.123341+0000","last_undegraded":"2024-04-24T08:08:21.123341+0000","last_fullsized":"2024-04-24T08:08:21.123341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121298+0000","last_change":"2024-04-24T08:08:20.164732+0000","last_active":"2024-04-24T08:08:21.121298+0000","last_peered":"2024-04-24T08:08:21.121298+0000","last_clean":"2024-04-24T08:08:21.121298+0000","last_became_active":"2024-04-24T08:08:20.164193+0000","last_became_peered":"2024-04-24T08:08:20.164193+0000","last_unstale":"2024-04-24T08:08:21.121298+0000","last_undegraded":"2024-04-24T08:08:21.121298+0000","last_fullsized":"2024-04-24T08:08:21.121298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152443+0000","last_change":"2024-04-24T08:08:19.151929+0000","last_active":"2024-04-24T08:08:21.152443+0000","last_peered":"2024-04-24T08:08:21.152443+0000","last_clean":"2024-04-24T08:08:21.152443+0000","last_became_active":"2024-04-24T08:08:19.151633+0000","last_became_peered":"2024-04-24T08:08:19.151633+0000","last_unstale":"2024-04-24T08:08:21.152443+0000","last_undegraded":"2024-04-24T08:08:21.152443+0000","last_fullsized":"2024-04-24T08:08:21.152443+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125502+0000","last_change":"2024-04-24T08:08:20.139547+0000","last_active":"2024-04-24T08:08:21.125502+0000","last_peered":"2024-04-24T08:08:21.125502+0000","last_clean":"2024-04-24T08:08:21.125502+0000","last_became_active":"2024-04-24T08:08:20.139316+0000","last_became_peered":"2024-04-24T08:08:20.139316+0000","last_unstale":"2024-04-24T08:08:21.125502+0000","last_undegraded":"2024-04-24T08:08:21.125502+0000","last_fullsized":"2024-04-24T08:08:21.125502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:37.292548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125476+0000","last_change":"2024-04-24T08:08:19.150665+0000","last_active":"2024-04-24T08:08:21.125476+0000","last_peered":"2024-04-24T08:08:21.125476+0000","last_clean":"2024-04-24T08:08:21.125476+0000","last_became_active":"2024-04-24T08:08:19.150327+0000","last_became_peered":"2024-04-24T08:08:19.150327+0000","last_unstale":"2024-04-24T08:08:21.125476+0000","last_undegraded":"2024-04-24T08:08:21.125476+0000","last_fullsized":"2024-04-24T08:08:21.125476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887629+0000","last_change":"2024-04-24T08:08:20.138279+0000","last_active":"2024-04-24T08:08:21.887629+0000","last_peered":"2024-04-24T08:08:21.887629+0000","last_clean":"2024-04-24T08:08:21.887629+0000","last_became_active":"2024-04-24T08:08:20.137776+0000","last_became_peered":"2024-04-24T08:08:20.137776+0000","last_unstale":"2024-04-24T08:08:21.887629+0000","last_undegraded":"2024-04-24T08:08:21.887629+0000","last_fullsized":"2024-04-24T08:08:21.887629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121623+0000","last_change":"2024-04-24T08:08:19.143886+0000","last_active":"2024-04-24T08:08:21.121623+0000","last_peered":"2024-04-24T08:08:21.121623+0000","last_clean":"2024-04-24T08:08:21.121623+0000","last_became_active":"2024-04-24T08:08:19.143606+0000","last_became_peered":"2024-04-24T08:08:19.143606+0000","last_unstale":"2024-04-24T08:08:21.121623+0000","last_undegraded":"2024-04-24T08:08:21.121623+0000","last_fullsized":"2024-04-24T08:08:21.121623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126085+0000","last_change":"2024-04-24T08:08:20.146770+0000","last_active":"2024-04-24T08:08:21.126085+0000","last_peered":"2024-04-24T08:08:21.126085+0000","last_clean":"2024-04-24T08:08:21.126085+0000","last_became_active":"2024-04-24T08:08:20.146470+0000","last_became_peered":"2024-04-24T08:08:20.146470+0000","last_unstale":"2024-04-24T08:08:21.126085+0000","last_undegraded":"2024-04-24T08:08:21.126085+0000","last_fullsized":"2024-04-24T08:08:21.126085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:46.310762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152147+0000","last_change":"2024-04-24T08:08:19.149833+0000","last_active":"2024-04-24T08:08:21.152147+0000","last_peered":"2024-04-24T08:08:21.152147+0000","last_clean":"2024-04-24T08:08:21.152147+0000","last_became_active":"2024-04-24T08:08:19.149391+0000","last_became_peered":"2024-04-24T08:08:19.149391+0000","last_unstale":"2024-04-24T08:08:21.152147+0000","last_undegraded":"2024-04-24T08:08:21.152147+0000","last_fullsized":"2024-04-24T08:08:21.152147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538676+0000","last_change":"2024-04-24T08:08:20.145962+0000","last_active":"2024-04-24T08:08:22.538676+0000","last_peered":"2024-04-24T08:08:22.538676+0000","last_clean":"2024-04-24T08:08:22.538676+0000","last_became_active":"2024-04-24T08:08:20.145644+0000","last_became_peered":"2024-04-24T08:08:20.145644+0000","last_unstale":"2024-04-24T08:08:22.538676+0000","last_undegraded":"2024-04-24T08:08:22.538676+0000","last_fullsized":"2024-04-24T08:08:22.538676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:37.292548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887814+0000","last_change":"2024-04-24T08:08:19.131316+0000","last_active":"2024-04-24T08:08:21.887814+0000","last_peered":"2024-04-24T08:08:21.887814+0000","last_clean":"2024-04-24T08:08:21.887814+0000","last_became_active":"2024-04-24T08:08:19.130895+0000","last_became_peered":"2024-04-24T08:08:19.130895+0000","last_unstale":"2024-04-24T08:08:21.887814+0000","last_undegraded":"2024-04-24T08:08:21.887814+0000","last_fullsized":"2024-04-24T08:08:21.887814+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:13:42.392189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.886980+0000","last_change":"2024-04-24T08:08:20.123273+0000","last_active":"2024-04-24T08:08:21.886980+0000","last_peered":"2024-04-24T08:08:21.886980+0000","last_clean":"2024-04-24T08:08:21.886980+0000","last_became_active":"2024-04-24T08:08:20.122506+0000","last_became_peered":"2024-04-24T08:08:20.122506+0000","last_unstale":"2024-04-24T08:08:21.886980+0000","last_undegraded":"2024-04-24T08:08:21.886980+0000","last_fullsized":"2024-04-24T08:08:21.886980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:40.157193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123661+0000","last_change":"2024-04-24T08:08:19.138216+0000","last_active":"2024-04-24T08:08:21.123661+0000","last_peered":"2024-04-24T08:08:21.123661+0000","last_clean":"2024-04-24T08:08:21.123661+0000","last_became_active":"2024-04-24T08:08:19.137882+0000","last_became_peered":"2024-04-24T08:08:19.137882+0000","last_unstale":"2024-04-24T08:08:21.123661+0000","last_undegraded":"2024-04-24T08:08:21.123661+0000","last_fullsized":"2024-04-24T08:08:21.123661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538907+0000","last_change":"2024-04-24T08:08:20.155343+0000","last_active":"2024-04-24T08:08:22.538907+0000","last_peered":"2024-04-24T08:08:22.538907+0000","last_clean":"2024-04-24T08:08:22.538907+0000","last_became_active":"2024-04-24T08:08:20.155112+0000","last_became_peered":"2024-04-24T08:08:20.155112+0000","last_unstale":"2024-04-24T08:08:22.538907+0000","last_undegraded":"2024-04-24T08:08:22.538907+0000","last_fullsized":"2024-04-24T08:08:22.538907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:36.215783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151585+0000","last_change":"2024-04-24T08:08:19.135440+0000","last_active":"2024-04-24T08:08:21.151585+0000","last_peered":"2024-04-24T08:08:21.151585+0000","last_clean":"2024-04-24T08:08:21.151585+0000","last_became_active":"2024-04-24T08:08:19.134946+0000","last_became_peered":"2024-04-24T08:08:19.134946+0000","last_unstale":"2024-04-24T08:08:21.151585+0000","last_undegraded":"2024-04-24T08:08:21.151585+0000","last_fullsized":"2024-04-24T08:08:21.151585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151891+0000","last_change":"2024-04-24T08:08:20.144523+0000","last_active":"2024-04-24T08:08:21.151891+0000","last_peered":"2024-04-24T08:08:21.151891+0000","last_clean":"2024-04-24T08:08:21.151891+0000","last_became_active":"2024-04-24T08:08:20.142266+0000","last_became_peered":"2024-04-24T08:08:20.142266+0000","last_unstale":"2024-04-24T08:08:21.151891+0000","last_undegraded":"2024-04-24T08:08:21.151891+0000","last_fullsized":"2024-04-24T08:08:21.151891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.361922+0000","last_change":"2024-04-24T08:08:19.110993+0000","last_active":"2024-04-24T08:08:21.361922+0000","last_peered":"2024-04-24T08:08:21.361922+0000","last_clean":"2024-04-24T08:08:21.361922+0000","last_became_active":"2024-04-24T08:08:19.110592+0000","last_became_peered":"2024-04-24T08:08:19.110592+0000","last_unstale":"2024-04-24T08:08:21.361922+0000","last_undegraded":"2024-04-24T08:08:21.361922+0000","last_fullsized":"2024-04-24T08:08:21.361922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538312+0000","last_change":"2024-04-24T08:08:20.149332+0000","last_active":"2024-04-24T08:08:22.538312+0000","last_peered":"2024-04-24T08:08:22.538312+0000","last_clean":"2024-04-24T08:08:22.538312+0000","last_became_active":"2024-04-24T08:08:20.148856+0000","last_became_peered":"2024-04-24T08:08:20.148856+0000","last_unstale":"2024-04-24T08:08:22.538312+0000","last_undegraded":"2024-04-24T08:08:22.538312+0000","last_fullsized":"2024-04-24T08:08:22.538312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:07:27.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125565+0000","last_change":"2024-04-24T08:08:19.128023+0000","last_active":"2024-04-24T08:08:21.125565+0000","last_peered":"2024-04-24T08:08:21.125565+0000","last_clean":"2024-04-24T08:08:21.125565+0000","last_became_active":"2024-04-24T08:08:19.126833+0000","last_became_peered":"2024-04-24T08:08:19.126833+0000","last_unstale":"2024-04-24T08:08:21.125565+0000","last_undegraded":"2024-04-24T08:08:21.125565+0000","last_fullsized":"2024-04-24T08:08:21.125565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.062494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539639+0000","last_change":"2024-04-24T08:08:20.144208+0000","last_active":"2024-04-24T08:08:22.539639+0000","last_peered":"2024-04-24T08:08:22.539639+0000","last_clean":"2024-04-24T08:08:22.539639+0000","last_became_active":"2024-04-24T08:08:20.143708+0000","last_became_peered":"2024-04-24T08:08:20.143708+0000","last_unstale":"2024-04-24T08:08:22.539639+0000","last_undegraded":"2024-04-24T08:08:22.539639+0000","last_fullsized":"2024-04-24T08:08:22.539639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:58:28.805520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362439+0000","last_change":"2024-04-24T08:08:19.137638+0000","last_active":"2024-04-24T08:08:21.362439+0000","last_peered":"2024-04-24T08:08:21.362439+0000","last_clean":"2024-04-24T08:08:21.362439+0000","last_became_active":"2024-04-24T08:08:19.137280+0000","last_became_peered":"2024-04-24T08:08:19.137280+0000","last_unstale":"2024-04-24T08:08:21.362439+0000","last_undegraded":"2024-04-24T08:08:21.362439+0000","last_fullsized":"2024-04-24T08:08:21.362439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:02:50.869449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539746+0000","last_change":"2024-04-24T08:08:20.140656+0000","last_active":"2024-04-24T08:08:22.539746+0000","last_peered":"2024-04-24T08:08:22.539746+0000","last_clean":"2024-04-24T08:08:22.539746+0000","last_became_active":"2024-04-24T08:08:20.140347+0000","last_became_peered":"2024-04-24T08:08:20.140347+0000","last_unstale":"2024-04-24T08:08:22.539746+0000","last_undegraded":"2024-04-24T08:08:22.539746+0000","last_fullsized":"2024-04-24T08:08:22.539746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:13:39.728625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121835+0000","last_change":"2024-04-24T08:08:19.127246+0000","last_active":"2024-04-24T08:08:21.121835+0000","last_peered":"2024-04-24T08:08:21.121835+0000","last_clean":"2024-04-24T08:08:21.121835+0000","last_became_active":"2024-04-24T08:08:19.126190+0000","last_became_peered":"2024-04-24T08:08:19.126190+0000","last_unstale":"2024-04-24T08:08:21.121835+0000","last_undegraded":"2024-04-24T08:08:21.121835+0000","last_fullsized":"2024-04-24T08:08:21.121835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126132+0000","last_change":"2024-04-24T08:08:20.161962+0000","last_active":"2024-04-24T08:08:21.126132+0000","last_peered":"2024-04-24T08:08:21.126132+0000","last_clean":"2024-04-24T08:08:21.126132+0000","last_became_active":"2024-04-24T08:08:20.161572+0000","last_became_peered":"2024-04-24T08:08:20.161572+0000","last_unstale":"2024-04-24T08:08:21.126132+0000","last_undegraded":"2024-04-24T08:08:21.126132+0000","last_fullsized":"2024-04-24T08:08:21.126132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:56:58.521686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362140+0000","last_change":"2024-04-24T08:08:19.140345+0000","last_active":"2024-04-24T08:08:21.362140+0000","last_peered":"2024-04-24T08:08:21.362140+0000","last_clean":"2024-04-24T08:08:21.362140+0000","last_became_active":"2024-04-24T08:08:19.140122+0000","last_became_peered":"2024-04-24T08:08:19.140122+0000","last_unstale":"2024-04-24T08:08:21.362140+0000","last_undegraded":"2024-04-24T08:08:21.362140+0000","last_fullsized":"2024-04-24T08:08:21.362140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539177+0000","last_change":"2024-04-24T08:08:20.137838+0000","last_active":"2024-04-24T08:08:22.539177+0000","last_peered":"2024-04-24T08:08:22.539177+0000","last_clean":"2024-04-24T08:08:22.539177+0000","last_became_active":"2024-04-24T08:08:20.137552+0000","last_became_peered":"2024-04-24T08:08:20.137552+0000","last_unstale":"2024-04-24T08:08:22.539177+0000","last_undegraded":"2024-04-24T08:08:22.539177+0000","last_fullsized":"2024-04-24T08:08:22.539177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:56:58.521686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538262+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:08:22.538262+0000","last_peered":"2024-04-24T08:08:22.538262+0000","last_clean":"2024-04-24T08:08:22.538262+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:08:22.538262+0000","last_undegraded":"2024-04-24T08:08:22.538262+0000","last_fullsized":"2024-04-24T08:08:22.538262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124383+0000","last_change":"2024-04-24T08:08:19.135677+0000","last_active":"2024-04-24T08:08:21.124383+0000","last_peered":"2024-04-24T08:08:21.124383+0000","last_clean":"2024-04-24T08:08:21.124383+0000","last_became_active":"2024-04-24T08:08:19.133935+0000","last_became_peered":"2024-04-24T08:08:19.133935+0000","last_unstale":"2024-04-24T08:08:21.124383+0000","last_undegraded":"2024-04-24T08:08:21.124383+0000","last_fullsized":"2024-04-24T08:08:21.124383+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537560+0000","last_change":"2024-04-24T08:08:20.148292+0000","last_active":"2024-04-24T08:08:22.537560+0000","last_peered":"2024-04-24T08:08:22.537560+0000","last_clean":"2024-04-24T08:08:22.537560+0000","last_became_active":"2024-04-24T08:08:20.147966+0000","last_became_peered":"2024-04-24T08:08:20.147966+0000","last_unstale":"2024-04-24T08:08:22.537560+0000","last_undegraded":"2024-04-24T08:08:22.537560+0000","last_fullsized":"2024-04-24T08:08:22.537560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537956+0000","last_change":"2024-04-24T08:08:19.148444+0000","last_active":"2024-04-24T08:08:22.537956+0000","last_peered":"2024-04-24T08:08:22.537956+0000","last_clean":"2024-04-24T08:08:22.537956+0000","last_became_active":"2024-04-24T08:08:19.148032+0000","last_became_peered":"2024-04-24T08:08:19.148032+0000","last_unstale":"2024-04-24T08:08:22.537956+0000","last_undegraded":"2024-04-24T08:08:22.537956+0000","last_fullsized":"2024-04-24T08:08:22.537956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.886751+0000","last_change":"2024-04-24T08:08:20.139133+0000","last_active":"2024-04-24T08:08:21.886751+0000","last_peered":"2024-04-24T08:08:21.886751+0000","last_clean":"2024-04-24T08:08:21.886751+0000","last_became_active":"2024-04-24T08:08:20.138841+0000","last_became_peered":"2024-04-24T08:08:20.138841+0000","last_unstale":"2024-04-24T08:08:21.886751+0000","last_undegraded":"2024-04-24T08:08:21.886751+0000","last_fullsized":"2024-04-24T08:08:21.886751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:29:21.637806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362188+0000","last_change":"2024-04-24T08:08:19.135322+0000","last_active":"2024-04-24T08:08:21.362188+0000","last_peered":"2024-04-24T08:08:21.362188+0000","last_clean":"2024-04-24T08:08:21.362188+0000","last_became_active":"2024-04-24T08:08:19.134982+0000","last_became_peered":"2024-04-24T08:08:19.134982+0000","last_unstale":"2024-04-24T08:08:21.362188+0000","last_undegraded":"2024-04-24T08:08:21.362188+0000","last_fullsized":"2024-04-24T08:08:21.362188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121142+0000","last_change":"2024-04-24T08:08:20.142759+0000","last_active":"2024-04-24T08:08:21.121142+0000","last_peered":"2024-04-24T08:08:21.121142+0000","last_clean":"2024-04-24T08:08:21.121142+0000","last_became_active":"2024-04-24T08:08:20.142474+0000","last_became_peered":"2024-04-24T08:08:20.142474+0000","last_unstale":"2024-04-24T08:08:21.121142+0000","last_undegraded":"2024-04-24T08:08:21.121142+0000","last_fullsized":"2024-04-24T08:08:21.121142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123701+0000","last_change":"2024-04-24T08:08:19.131093+0000","last_active":"2024-04-24T08:08:21.123701+0000","last_peered":"2024-04-24T08:08:21.123701+0000","last_clean":"2024-04-24T08:08:21.123701+0000","last_became_active":"2024-04-24T08:08:19.129809+0000","last_became_peered":"2024-04-24T08:08:19.129809+0000","last_unstale":"2024-04-24T08:08:21.123701+0000","last_undegraded":"2024-04-24T08:08:21.123701+0000","last_fullsized":"2024-04-24T08:08:21.123701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124958+0000","last_change":"2024-04-24T08:08:20.155888+0000","last_active":"2024-04-24T08:08:21.124958+0000","last_peered":"2024-04-24T08:08:21.124958+0000","last_clean":"2024-04-24T08:08:21.124958+0000","last_became_active":"2024-04-24T08:08:20.154101+0000","last_became_peered":"2024-04-24T08:08:20.154101+0000","last_unstale":"2024-04-24T08:08:21.124958+0000","last_undegraded":"2024-04-24T08:08:21.124958+0000","last_fullsized":"2024-04-24T08:08:21.124958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:49.918407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121420+0000","last_change":"2024-04-24T08:08:19.128197+0000","last_active":"2024-04-24T08:08:21.121420+0000","last_peered":"2024-04-24T08:08:21.121420+0000","last_clean":"2024-04-24T08:08:21.121420+0000","last_became_active":"2024-04-24T08:08:19.127218+0000","last_became_peered":"2024-04-24T08:08:19.127218+0000","last_unstale":"2024-04-24T08:08:21.121420+0000","last_undegraded":"2024-04-24T08:08:21.121420+0000","last_fullsized":"2024-04-24T08:08:21.121420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:45.296206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126415+0000","last_change":"2024-04-24T08:08:20.159839+0000","last_active":"2024-04-24T08:08:21.126415+0000","last_peered":"2024-04-24T08:08:21.126415+0000","last_clean":"2024-04-24T08:08:21.126415+0000","last_became_active":"2024-04-24T08:08:20.159475+0000","last_became_peered":"2024-04-24T08:08:20.159475+0000","last_unstale":"2024-04-24T08:08:21.126415+0000","last_undegraded":"2024-04-24T08:08:21.126415+0000","last_fullsized":"2024-04-24T08:08:21.126415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:03:56.009457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539328+0000","last_change":"2024-04-24T08:08:19.148608+0000","last_active":"2024-04-24T08:08:22.539328+0000","last_peered":"2024-04-24T08:08:22.539328+0000","last_clean":"2024-04-24T08:08:22.539328+0000","last_became_active":"2024-04-24T08:08:19.148299+0000","last_became_peered":"2024-04-24T08:08:19.148299+0000","last_unstale":"2024-04-24T08:08:22.539328+0000","last_undegraded":"2024-04-24T08:08:22.539328+0000","last_fullsized":"2024-04-24T08:08:22.539328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:45.296206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121527+0000","last_change":"2024-04-24T08:08:20.144511+0000","last_active":"2024-04-24T08:08:21.121527+0000","last_peered":"2024-04-24T08:08:21.121527+0000","last_clean":"2024-04-24T08:08:21.121527+0000","last_became_active":"2024-04-24T08:08:20.144184+0000","last_became_peered":"2024-04-24T08:08:20.144184+0000","last_unstale":"2024-04-24T08:08:21.121527+0000","last_undegraded":"2024-04-24T08:08:21.121527+0000","last_fullsized":"2024-04-24T08:08:21.121527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.122119+0000","last_change":"2024-04-24T08:08:19.149355+0000","last_active":"2024-04-24T08:08:21.122119+0000","last_peered":"2024-04-24T08:08:21.122119+0000","last_clean":"2024-04-24T08:08:21.122119+0000","last_became_active":"2024-04-24T08:08:19.148936+0000","last_became_peered":"2024-04-24T08:08:19.148936+0000","last_unstale":"2024-04-24T08:08:21.122119+0000","last_undegraded":"2024-04-24T08:08:21.122119+0000","last_fullsized":"2024-04-24T08:08:21.122119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:49.339006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538134+0000","last_change":"2024-04-24T08:08:20.153837+0000","last_active":"2024-04-24T08:08:22.538134+0000","last_peered":"2024-04-24T08:08:22.538134+0000","last_clean":"2024-04-24T08:08:22.538134+0000","last_became_active":"2024-04-24T08:08:20.150347+0000","last_became_peered":"2024-04-24T08:08:20.150347+0000","last_unstale":"2024-04-24T08:08:22.538134+0000","last_undegraded":"2024-04-24T08:08:22.538134+0000","last_fullsized":"2024-04-24T08:08:22.538134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:53.941683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152520+0000","last_change":"2024-04-24T08:08:19.143650+0000","last_active":"2024-04-24T08:08:21.152520+0000","last_peered":"2024-04-24T08:08:21.152520+0000","last_clean":"2024-04-24T08:08:21.152520+0000","last_became_active":"2024-04-24T08:08:19.143321+0000","last_became_peered":"2024-04-24T08:08:19.143321+0000","last_unstale":"2024-04-24T08:08:21.152520+0000","last_undegraded":"2024-04-24T08:08:21.152520+0000","last_fullsized":"2024-04-24T08:08:21.152520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887531+0000","last_change":"2024-04-24T08:08:20.142865+0000","last_active":"2024-04-24T08:08:21.887531+0000","last_peered":"2024-04-24T08:08:21.887531+0000","last_clean":"2024-04-24T08:08:21.887531+0000","last_became_active":"2024-04-24T08:08:20.142593+0000","last_became_peered":"2024-04-24T08:08:20.142593+0000","last_unstale":"2024-04-24T08:08:21.887531+0000","last_undegraded":"2024-04-24T08:08:21.887531+0000","last_fullsized":"2024-04-24T08:08:21.887531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537886+0000","last_change":"2024-04-24T08:08:19.146717+0000","last_active":"2024-04-24T08:08:22.537886+0000","last_peered":"2024-04-24T08:08:22.537886+0000","last_clean":"2024-04-24T08:08:22.537886+0000","last_became_active":"2024-04-24T08:08:19.146435+0000","last_became_peered":"2024-04-24T08:08:19.146435+0000","last_unstale":"2024-04-24T08:08:22.537886+0000","last_undegraded":"2024-04-24T08:08:22.537886+0000","last_fullsized":"2024-04-24T08:08:22.537886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124833+0000","last_change":"2024-04-24T08:08:20.153349+0000","last_active":"2024-04-24T08:08:21.124833+0000","last_peered":"2024-04-24T08:08:21.124833+0000","last_clean":"2024-04-24T08:08:21.124833+0000","last_became_active":"2024-04-24T08:08:20.153071+0000","last_became_peered":"2024-04-24T08:08:20.153071+0000","last_unstale":"2024-04-24T08:08:21.124833+0000","last_undegraded":"2024-04-24T08:08:21.124833+0000","last_fullsized":"2024-04-24T08:08:21.124833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:26.469631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121526+0000","last_change":"2024-04-24T08:08:19.141216+0000","last_active":"2024-04-24T08:08:21.121526+0000","last_peered":"2024-04-24T08:08:21.121526+0000","last_clean":"2024-04-24T08:08:21.121526+0000","last_became_active":"2024-04-24T08:08:19.140885+0000","last_became_peered":"2024-04-24T08:08:19.140885+0000","last_unstale":"2024-04-24T08:08:21.121526+0000","last_undegraded":"2024-04-24T08:08:21.121526+0000","last_fullsized":"2024-04-24T08:08:21.121526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537837+0000","last_change":"2024-04-24T08:08:20.139684+0000","last_active":"2024-04-24T08:08:22.537837+0000","last_peered":"2024-04-24T08:08:22.537837+0000","last_clean":"2024-04-24T08:08:22.537837+0000","last_became_active":"2024-04-24T08:08:20.139348+0000","last_became_peered":"2024-04-24T08:08:20.139348+0000","last_unstale":"2024-04-24T08:08:22.537837+0000","last_undegraded":"2024-04-24T08:08:22.537837+0000","last_fullsized":"2024-04-24T08:08:22.537837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121275+0000","last_change":"2024-04-24T08:08:19.137264+0000","last_active":"2024-04-24T08:08:21.121275+0000","last_peered":"2024-04-24T08:08:21.121275+0000","last_clean":"2024-04-24T08:08:21.121275+0000","last_became_active":"2024-04-24T08:08:19.136921+0000","last_became_peered":"2024-04-24T08:08:19.136921+0000","last_unstale":"2024-04-24T08:08:21.121275+0000","last_undegraded":"2024-04-24T08:08:21.121275+0000","last_fullsized":"2024-04-24T08:08:21.121275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538286+0000","last_change":"2024-04-24T08:08:20.158595+0000","last_active":"2024-04-24T08:08:22.538286+0000","last_peered":"2024-04-24T08:08:22.538286+0000","last_clean":"2024-04-24T08:08:22.538286+0000","last_became_active":"2024-04-24T08:08:20.158248+0000","last_became_peered":"2024-04-24T08:08:20.158248+0000","last_unstale":"2024-04-24T08:08:22.538286+0000","last_undegraded":"2024-04-24T08:08:22.538286+0000","last_fullsized":"2024-04-24T08:08:22.538286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:15.025368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539152+0000","last_change":"2024-04-24T08:08:19.143021+0000","last_active":"2024-04-24T08:08:22.539152+0000","last_peered":"2024-04-24T08:08:22.539152+0000","last_clean":"2024-04-24T08:08:22.539152+0000","last_became_active":"2024-04-24T08:08:19.142731+0000","last_became_peered":"2024-04-24T08:08:19.142731+0000","last_unstale":"2024-04-24T08:08:22.539152+0000","last_undegraded":"2024-04-24T08:08:22.539152+0000","last_fullsized":"2024-04-24T08:08:22.539152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:10:51.038401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151895+0000","last_change":"2024-04-24T08:08:20.141951+0000","last_active":"2024-04-24T08:08:21.151895+0000","last_peered":"2024-04-24T08:08:21.151895+0000","last_clean":"2024-04-24T08:08:21.151895+0000","last_became_active":"2024-04-24T08:08:20.141642+0000","last_became_peered":"2024-04-24T08:08:20.141642+0000","last_unstale":"2024-04-24T08:08:21.151895+0000","last_undegraded":"2024-04-24T08:08:21.151895+0000","last_fullsized":"2024-04-24T08:08:21.151895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126001+0000","last_change":"2024-04-24T08:08:20.152685+0000","last_active":"2024-04-24T08:08:21.126001+0000","last_peered":"2024-04-24T08:08:21.126001+0000","last_clean":"2024-04-24T08:08:21.126001+0000","last_became_active":"2024-04-24T08:08:20.152171+0000","last_became_peered":"2024-04-24T08:08:20.152171+0000","last_unstale":"2024-04-24T08:08:21.126001+0000","last_undegraded":"2024-04-24T08:08:21.126001+0000","last_fullsized":"2024-04-24T08:08:21.126001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:36.215783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538456+0000","last_change":"2024-04-24T08:08:19.147333+0000","last_active":"2024-04-24T08:08:22.538456+0000","last_peered":"2024-04-24T08:08:22.538456+0000","last_clean":"2024-04-24T08:08:22.538456+0000","last_became_active":"2024-04-24T08:08:19.147046+0000","last_became_peered":"2024-04-24T08:08:19.147046+0000","last_unstale":"2024-04-24T08:08:22.538456+0000","last_undegraded":"2024-04-24T08:08:22.538456+0000","last_fullsized":"2024-04-24T08:08:22.538456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:24:02.225409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125720+0000","last_change":"2024-04-24T08:08:20.152622+0000","last_active":"2024-04-24T08:08:21.125720+0000","last_peered":"2024-04-24T08:08:21.125720+0000","last_clean":"2024-04-24T08:08:21.125720+0000","last_became_active":"2024-04-24T08:08:20.151784+0000","last_became_peered":"2024-04-24T08:08:20.151784+0000","last_unstale":"2024-04-24T08:08:21.125720+0000","last_undegraded":"2024-04-24T08:08:21.125720+0000","last_fullsized":"2024-04-24T08:08:21.125720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:32:29.020308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538879+0000","last_change":"2024-04-24T08:08:19.150378+0000","last_active":"2024-04-24T08:08:22.538879+0000","last_peered":"2024-04-24T08:08:22.538879+0000","last_clean":"2024-04-24T08:08:22.538879+0000","last_became_active":"2024-04-24T08:08:19.149946+0000","last_became_peered":"2024-04-24T08:08:19.149946+0000","last_unstale":"2024-04-24T08:08:22.538879+0000","last_undegraded":"2024-04-24T08:08:22.538879+0000","last_fullsized":"2024-04-24T08:08:22.538879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:49.339006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887243+0000","last_change":"2024-04-24T08:08:20.138065+0000","last_active":"2024-04-24T08:08:21.887243+0000","last_peered":"2024-04-24T08:08:21.887243+0000","last_clean":"2024-04-24T08:08:21.887243+0000","last_became_active":"2024-04-24T08:08:20.137340+0000","last_became_peered":"2024-04-24T08:08:20.137340+0000","last_unstale":"2024-04-24T08:08:21.887243+0000","last_undegraded":"2024-04-24T08:08:21.887243+0000","last_fullsized":"2024-04-24T08:08:21.887243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538608+0000","last_change":"2024-04-24T08:08:19.134141+0000","last_active":"2024-04-24T08:08:22.538608+0000","last_peered":"2024-04-24T08:08:22.538608+0000","last_clean":"2024-04-24T08:08:22.538608+0000","last_became_active":"2024-04-24T08:08:19.133873+0000","last_became_peered":"2024-04-24T08:08:19.133873+0000","last_unstale":"2024-04-24T08:08:22.538608+0000","last_undegraded":"2024-04-24T08:08:22.538608+0000","last_fullsized":"2024-04-24T08:08:22.538608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538569+0000","last_change":"2024-04-24T08:08:20.146963+0000","last_active":"2024-04-24T08:08:22.538569+0000","last_peered":"2024-04-24T08:08:22.538569+0000","last_clean":"2024-04-24T08:08:22.538569+0000","last_became_active":"2024-04-24T08:08:20.146270+0000","last_became_peered":"2024-04-24T08:08:20.146270+0000","last_unstale":"2024-04-24T08:08:22.538569+0000","last_undegraded":"2024-04-24T08:08:22.538569+0000","last_fullsized":"2024-04-24T08:08:22.538569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:37.292548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887648+0000","last_change":"2024-04-24T08:08:19.142063+0000","last_active":"2024-04-24T08:08:21.887648+0000","last_peered":"2024-04-24T08:08:21.887648+0000","last_clean":"2024-04-24T08:08:21.887648+0000","last_became_active":"2024-04-24T08:08:19.141469+0000","last_became_peered":"2024-04-24T08:08:19.141469+0000","last_unstale":"2024-04-24T08:08:21.887648+0000","last_undegraded":"2024-04-24T08:08:21.887648+0000","last_fullsized":"2024-04-24T08:08:21.887648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:46.176456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:08:29.955 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:08:30.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.183+0000 7f72be400640 1 Processor -- start 2024-04-24T08:08:30.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.183+0000 7f72be400640 1 -- start start 2024-04-24T08:08:30.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.184+0000 7f72be400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f72b815d1d0 0x7f72b80ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.184+0000 7f72be400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f72b80b30f0 0x7f72b80ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.184+0000 7f72be400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72b815d660 con 0x7f72b8156c60 2024-04-24T08:08:30.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.184+0000 7f72be400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f72b806dc30 con 0x7f72b815d1d0 2024-04-24T08:08:30.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.184+0000 7f72be400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f72b806df10 con 0x7f72b80b30f0 2024-04-24T08:08:30.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.184+0000 7f72bd000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f72b8156c60 0x7f72b815cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36604/0 (socket says 172.21.15.69:36604) 2024-04-24T08:08:30.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.184+0000 7f72bd000640 1 -- 172.21.15.69:0/3695168100 learned_addr learned my addr 172.21.15.69:0/3695168100 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:30.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.184+0000 7f72b7e00640 1 --2- 172.21.15.69:0/3695168100 >> v2:172.21.15.69:3300/0 conn(0x7f72b815d1d0 0x7f72b80ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.185+0000 7f72b7400640 1 -- 172.21.15.69:0/3695168100 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2877784158 0 0) 0x7f72b815d660 con 0x7f72b8156c60 2024-04-24T08:08:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.185+0000 7f72bda00640 1 --2- 172.21.15.69:0/3695168100 >> v2:172.21.15.121:3300/0 conn(0x7f72b80b30f0 0x7f72b80ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.185+0000 7f72b7400640 1 -- 172.21.15.69:0/3695168100 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f729c003520 con 0x7f72b8156c60 2024-04-24T08:08:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.185+0000 7f72b7e00640 1 -- 172.21.15.69:0/3695168100 >> v2:172.21.15.121:3300/0 conn(0x7f72b80b30f0 msgr2=0x7f72b80ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.185+0000 7f72b7e00640 1 --2- 172.21.15.69:0/3695168100 >> v2:172.21.15.121:3300/0 conn(0x7f72b80b30f0 0x7f72b80ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.185+0000 7f72b7e00640 1 -- 172.21.15.69:0/3695168100 >> v1:172.21.15.69:6789/0 conn(0x7f72b8156c60 legacy=0x7f72b815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.185+0000 7f72b7e00640 1 -- 172.21.15.69:0/3695168100 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b806e1f0 con 0x7f72b815d1d0 2024-04-24T08:08:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.185+0000 7f72bda00640 1 --2- 172.21.15.69:0/3695168100 >> v2:172.21.15.121:3300/0 conn(0x7f72b80b30f0 0x7f72b80ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:30.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.186+0000 7f72b7e00640 1 --2- 172.21.15.69:0/3695168100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b815d1d0 0x7f72b80ad0d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f72b80b1d80 tx=0x7f72a002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75613c552ffcf1dd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:30.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.186+0000 7f72b7400640 1 -- 172.21.15.69:0/3695168100 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0037070 con 0x7f72b815d1d0 2024-04-24T08:08:30.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.186+0000 7f72b7400640 1 -- 172.21.15.69:0/3695168100 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72a0032440 con 0x7f72b815d1d0 2024-04-24T08:08:30.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.186+0000 7f72b7400640 1 -- 172.21.15.69:0/3695168100 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a00327f0 con 0x7f72b815d1d0 2024-04-24T08:08:30.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.186+0000 7f72be400640 1 -- 172.21.15.69:0/3695168100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b815d1d0 msgr2=0x7f72b80ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.186+0000 7f72be400640 1 --2- 172.21.15.69:0/3695168100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b815d1d0 0x7f72b80ad0d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f72b80b1d80 tx=0x7f72a002a1a0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.187+0000 7f72be400640 1 -- 172.21.15.69:0/3695168100 shutdown_connections 2024-04-24T08:08:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.187+0000 7f72be400640 1 --2- 172.21.15.69:0/3695168100 >> v2:172.21.15.121:3300/0 conn(0x7f72b80b30f0 0x7f72b80ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.187+0000 7f72be400640 1 --2- 172.21.15.69:0/3695168100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b815d1d0 0x7f72b80ad0d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.187+0000 7f72be400640 1 -- 172.21.15.69:0/3695168100 >> 172.21.15.69:0/3695168100 conn(0x7f72b80d0160 msgr2=0x7f72b80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.187+0000 7f72be400640 1 -- 172.21.15.69:0/3695168100 shutdown_connections 2024-04-24T08:08:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.187+0000 7f72be400640 1 -- 172.21.15.69:0/3695168100 wait complete. 2024-04-24T08:08:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.188+0000 7f72be400640 1 Processor -- start 2024-04-24T08:08:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.188+0000 7f72be400640 1 -- start start 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.189+0000 7f72be400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b80b30f0 0x7f72b8072890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.189+0000 7f72be400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b8156c60 0x7f72b815a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.189+0000 7f72be400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b81f8a90 0x7f72b8154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.189+0000 7f72be400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72b806e710 con 0x7f72b81f8a90 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.189+0000 7f72be400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f72b815d940 con 0x7f72b80b30f0 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.189+0000 7f72be400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72b806e890 con 0x7f72b8156c60 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72bda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b81f8a90 0x7f72b8154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72bda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b81f8a90 0x7f72b8154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44424/0 (socket says 172.21.15.69:44424) 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72bda00640 1 -- 172.21.15.69:0/4050795100 learned_addr learned my addr 172.21.15.69:0/4050795100 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72bd000640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b80b30f0 0x7f72b8072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72b7e00640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b8156c60 0x7f72b815a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72bda00640 1 -- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b8156c60 msgr2=0x7f72b815a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72bda00640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b8156c60 0x7f72b815a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72bda00640 1 -- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b80b30f0 msgr2=0x7f72b8072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72bda00640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b80b30f0 0x7f72b8072890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72bda00640 1 -- 172.21.15.69:0/4050795100 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b8073170 con 0x7f72b81f8a90 2024-04-24T08:08:30.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72b7e00640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b8156c60 0x7f72b815a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:30.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72bda00640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b81f8a90 0x7f72b8154980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f72b000cf20 tx=0x7f72b000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:30.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.190+0000 7f72b4c00640 1 -- 172.21.15.69:0/4050795100 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b001a070 con 0x7f72b81f8a90 2024-04-24T08:08:30.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.191+0000 7f72b4c00640 1 -- 172.21.15.69:0/4050795100 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72b000d840 con 0x7f72b81f8a90 2024-04-24T08:08:30.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.191+0000 7f72b4c00640 1 -- 172.21.15.69:0/4050795100 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b000dc30 con 0x7f72b81f8a90 2024-04-24T08:08:30.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.191+0000 7f72be400640 1 -- 172.21.15.69:0/4050795100 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b81f7fb0 con 0x7f72b81f8a90 2024-04-24T08:08:30.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.191+0000 7f72be400640 1 -- 172.21.15.69:0/4050795100 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b81f8570 con 0x7f72b81f8a90 2024-04-24T08:08:30.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.192+0000 7f72be400640 1 -- 172.21.15.69:0/4050795100 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72b80ae1e0 con 0x7f72b81f8a90 2024-04-24T08:08:30.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.193+0000 7f72b4c00640 1 -- 172.21.15.69:0/4050795100 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f72b001d070 con 0x7f72b81f8a90 2024-04-24T08:08:30.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.193+0000 7f72b4c00640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f729406c8f0 0x7f729406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.193+0000 7f72b4c00640 1 -- 172.21.15.69:0/4050795100 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f72b008c3c0 con 0x7f72b81f8a90 2024-04-24T08:08:30.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.194+0000 7f72bd000640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f729406c8f0 0x7f729406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.195+0000 7f72bd000640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f729406c8f0 0x7f729406edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f72a8005880 tx=0x7f72a8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:30.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.197+0000 7f72b4c00640 1 -- 172.21.15.69:0/4050795100 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72b005a8b0 con 0x7f72b81f8a90 2024-04-24T08:08:30.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.375+0000 7f72be400640 1 -- 172.21.15.69:0/4050795100 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f72b81f87b0 con 0x7f729406c8f0 2024-04-24T08:08:30.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.394+0000 7f72b4c00640 1 -- 172.21.15.69:0/4050795100 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f72a8001d00 con 0x7f729406c8f0 2024-04-24T08:08:30.398 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:30.400 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:08:30.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.398+0000 7f72be400640 1 -- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f729406c8f0 msgr2=0x7f729406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.398+0000 7f72be400640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f729406c8f0 0x7f729406edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f72a8005880 tx=0x7f72a8009040 comp rx=0 tx=0).stop 2024-04-24T08:08:30.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.398+0000 7f72be400640 1 -- 172.21.15.69:0/4050795100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b81f8a90 msgr2=0x7f72b8154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.398+0000 7f72be400640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b81f8a90 0x7f72b8154980 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f72b000cf20 tx=0x7f72b000a7d0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.399+0000 7f72be400640 1 -- 172.21.15.69:0/4050795100 shutdown_connections 2024-04-24T08:08:30.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.399+0000 7f72be400640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f729406c8f0 0x7f729406edb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.399+0000 7f72be400640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b81f8a90 0x7f72b8154980 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.399+0000 7f72be400640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72b8156c60 0x7f72b815a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.399+0000 7f72be400640 1 --2- 172.21.15.69:0/4050795100 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b80b30f0 0x7f72b8072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.399+0000 7f72be400640 1 -- 172.21.15.69:0/4050795100 >> 172.21.15.69:0/4050795100 conn(0x7f72b80d0160 msgr2=0x7f72b80b34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:30.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.399+0000 7f72be400640 1 -- 172.21.15.69:0/4050795100 shutdown_connections 2024-04-24T08:08:30.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.399+0000 7f72be400640 1 -- 172.21.15.69:0/4050795100 wait complete. 2024-04-24T08:08:30.419 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-24T08:08:28.663055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218992,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755728,"statfs":{"total":773094113280,"available":772869865472,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.555218"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539355+0000","last_change":"2024-04-24T08:08:20.145550+0000","last_active":"2024-04-24T08:08:22.539355+0000","last_peered":"2024-04-24T08:08:22.539355+0000","last_clean":"2024-04-24T08:08:22.539355+0000","last_became_active":"2024-04-24T08:08:20.144816+0000","last_became_peered":"2024-04-24T08:08:20.144816+0000","last_unstale":"2024-04-24T08:08:22.539355+0000","last_undegraded":"2024-04-24T08:08:22.539355+0000","last_fullsized":"2024-04-24T08:08:22.539355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:03:56.009457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.122230+0000","last_change":"2024-04-24T08:08:20.161217+0000","last_active":"2024-04-24T08:08:21.122230+0000","last_peered":"2024-04-24T08:08:21.122230+0000","last_clean":"2024-04-24T08:08:21.122230+0000","last_became_active":"2024-04-24T08:08:20.159365+0000","last_became_peered":"2024-04-24T08:08:20.159365+0000","last_unstale":"2024-04-24T08:08:21.122230+0000","last_undegraded":"2024-04-24T08:08:21.122230+0000","last_fullsized":"2024-04-24T08:08:21.122230+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887063+0000","last_change":"2024-04-24T08:08:20.141345+0000","last_active":"2024-04-24T08:08:21.887063+0000","last_peered":"2024-04-24T08:08:21.887063+0000","last_clean":"2024-04-24T08:08:21.887063+0000","last_became_active":"2024-04-24T08:08:20.141134+0000","last_became_peered":"2024-04-24T08:08:20.141134+0000","last_unstale":"2024-04-24T08:08:21.887063+0000","last_undegraded":"2024-04-24T08:08:21.887063+0000","last_fullsized":"2024-04-24T08:08:21.887063+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125297+0000","last_change":"2024-04-24T08:08:20.164461+0000","last_active":"2024-04-24T08:08:21.125297+0000","last_peered":"2024-04-24T08:08:21.125297+0000","last_clean":"2024-04-24T08:08:21.125297+0000","last_became_active":"2024-04-24T08:08:20.164118+0000","last_became_peered":"2024-04-24T08:08:20.164118+0000","last_unstale":"2024-04-24T08:08:21.125297+0000","last_undegraded":"2024-04-24T08:08:21.125297+0000","last_fullsized":"2024-04-24T08:08:21.125297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:34.119848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538852+0000","last_change":"2024-04-24T08:08:20.146319+0000","last_active":"2024-04-24T08:08:22.538852+0000","last_peered":"2024-04-24T08:08:22.538852+0000","last_clean":"2024-04-24T08:08:22.538852+0000","last_became_active":"2024-04-24T08:08:20.145969+0000","last_became_peered":"2024-04-24T08:08:20.145969+0000","last_unstale":"2024-04-24T08:08:22.538852+0000","last_undegraded":"2024-04-24T08:08:22.538852+0000","last_fullsized":"2024-04-24T08:08:22.538852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:32:29.020308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123393+0000","last_change":"2024-04-24T08:08:20.149172+0000","last_active":"2024-04-24T08:08:21.123393+0000","last_peered":"2024-04-24T08:08:21.123393+0000","last_clean":"2024-04-24T08:08:21.123393+0000","last_became_active":"2024-04-24T08:08:20.148447+0000","last_became_peered":"2024-04-24T08:08:20.148447+0000","last_unstale":"2024-04-24T08:08:21.123393+0000","last_undegraded":"2024-04-24T08:08:21.123393+0000","last_fullsized":"2024-04-24T08:08:21.123393+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:53.941683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538379+0000","last_change":"2024-04-24T08:08:20.132800+0000","last_active":"2024-04-24T08:08:22.538379+0000","last_peered":"2024-04-24T08:08:22.538379+0000","last_clean":"2024-04-24T08:08:22.538379+0000","last_became_active":"2024-04-24T08:08:20.132442+0000","last_became_peered":"2024-04-24T08:08:20.132442+0000","last_unstale":"2024-04-24T08:08:22.538379+0000","last_undegraded":"2024-04-24T08:08:22.538379+0000","last_fullsized":"2024-04-24T08:08:22.538379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:34.119848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537521+0000","last_change":"2024-04-24T08:08:20.153541+0000","last_active":"2024-04-24T08:08:22.537521+0000","last_peered":"2024-04-24T08:08:22.537521+0000","last_clean":"2024-04-24T08:08:22.537521+0000","last_became_active":"2024-04-24T08:08:20.153214+0000","last_became_peered":"2024-04-24T08:08:20.153214+0000","last_unstale":"2024-04-24T08:08:22.537521+0000","last_undegraded":"2024-04-24T08:08:22.537521+0000","last_fullsized":"2024-04-24T08:08:22.537521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539513+0000","last_change":"2024-04-24T08:08:20.136784+0000","last_active":"2024-04-24T08:08:22.539513+0000","last_peered":"2024-04-24T08:08:22.539513+0000","last_clean":"2024-04-24T08:08:22.539513+0000","last_became_active":"2024-04-24T08:08:20.136552+0000","last_became_peered":"2024-04-24T08:08:20.136552+0000","last_unstale":"2024-04-24T08:08:22.539513+0000","last_undegraded":"2024-04-24T08:08:22.539513+0000","last_fullsized":"2024-04-24T08:08:22.539513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:02:22.840489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152584+0000","last_change":"2024-04-24T08:08:20.143039+0000","last_active":"2024-04-24T08:08:21.152584+0000","last_peered":"2024-04-24T08:08:21.152584+0000","last_clean":"2024-04-24T08:08:21.152584+0000","last_became_active":"2024-04-24T08:08:20.142388+0000","last_became_peered":"2024-04-24T08:08:20.142388+0000","last_unstale":"2024-04-24T08:08:21.152584+0000","last_undegraded":"2024-04-24T08:08:21.152584+0000","last_fullsized":"2024-04-24T08:08:21.152584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152537+0000","last_change":"2024-04-24T08:08:20.143995+0000","last_active":"2024-04-24T08:08:21.152537+0000","last_peered":"2024-04-24T08:08:21.152537+0000","last_clean":"2024-04-24T08:08:21.152537+0000","last_became_active":"2024-04-24T08:08:20.143674+0000","last_became_peered":"2024-04-24T08:08:20.143674+0000","last_unstale":"2024-04-24T08:08:21.152537+0000","last_undegraded":"2024-04-24T08:08:21.152537+0000","last_fullsized":"2024-04-24T08:08:21.152537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125663+0000","last_change":"2024-04-24T08:08:20.146375+0000","last_active":"2024-04-24T08:08:21.125663+0000","last_peered":"2024-04-24T08:08:21.125663+0000","last_clean":"2024-04-24T08:08:21.125663+0000","last_became_active":"2024-04-24T08:08:20.146053+0000","last_became_peered":"2024-04-24T08:08:20.146053+0000","last_unstale":"2024-04-24T08:08:21.125663+0000","last_undegraded":"2024-04-24T08:08:21.125663+0000","last_fullsized":"2024-04-24T08:08:21.125663+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:50.353563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538060+0000","last_change":"2024-04-24T08:08:20.151989+0000","last_active":"2024-04-24T08:08:22.538060+0000","last_peered":"2024-04-24T08:08:22.538060+0000","last_clean":"2024-04-24T08:08:22.538060+0000","last_became_active":"2024-04-24T08:08:20.150076+0000","last_became_peered":"2024-04-24T08:08:20.150076+0000","last_unstale":"2024-04-24T08:08:22.538060+0000","last_undegraded":"2024-04-24T08:08:22.538060+0000","last_fullsized":"2024-04-24T08:08:22.538060+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.886831+0000","last_change":"2024-04-24T08:08:20.157703+0000","last_active":"2024-04-24T08:08:21.886831+0000","last_peered":"2024-04-24T08:08:21.886831+0000","last_clean":"2024-04-24T08:08:21.886831+0000","last_became_active":"2024-04-24T08:08:20.151876+0000","last_became_peered":"2024-04-24T08:08:20.151876+0000","last_unstale":"2024-04-24T08:08:21.886831+0000","last_undegraded":"2024-04-24T08:08:21.886831+0000","last_fullsized":"2024-04-24T08:08:21.886831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:07:41.389051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121308+0000","last_change":"2024-04-24T08:08:20.146100+0000","last_active":"2024-04-24T08:08:21.121308+0000","last_peered":"2024-04-24T08:08:21.121308+0000","last_clean":"2024-04-24T08:08:21.121308+0000","last_became_active":"2024-04-24T08:08:20.144236+0000","last_became_peered":"2024-04-24T08:08:20.144236+0000","last_unstale":"2024-04-24T08:08:21.121308+0000","last_undegraded":"2024-04-24T08:08:21.121308+0000","last_fullsized":"2024-04-24T08:08:21.121308+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123180+0000","last_change":"2024-04-24T08:08:20.137406+0000","last_active":"2024-04-24T08:08:21.123180+0000","last_peered":"2024-04-24T08:08:21.123180+0000","last_clean":"2024-04-24T08:08:21.123180+0000","last_became_active":"2024-04-24T08:08:20.137109+0000","last_became_peered":"2024-04-24T08:08:20.137109+0000","last_unstale":"2024-04-24T08:08:21.123180+0000","last_undegraded":"2024-04-24T08:08:21.123180+0000","last_fullsized":"2024-04-24T08:08:21.123180+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126733+0000","last_change":"2024-04-24T08:08:20.141553+0000","last_active":"2024-04-24T08:08:21.126733+0000","last_peered":"2024-04-24T08:08:21.126733+0000","last_clean":"2024-04-24T08:08:21.126733+0000","last_became_active":"2024-04-24T08:08:20.141163+0000","last_became_peered":"2024-04-24T08:08:20.141163+0000","last_unstale":"2024-04-24T08:08:21.126733+0000","last_undegraded":"2024-04-24T08:08:21.126733+0000","last_fullsized":"2024-04-24T08:08:21.126733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:19:04.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362588+0000","last_change":"2024-04-24T08:08:20.143805+0000","last_active":"2024-04-24T08:08:21.362588+0000","last_peered":"2024-04-24T08:08:21.362588+0000","last_clean":"2024-04-24T08:08:21.362588+0000","last_became_active":"2024-04-24T08:08:20.143427+0000","last_became_peered":"2024-04-24T08:08:20.143427+0000","last_unstale":"2024-04-24T08:08:21.362588+0000","last_undegraded":"2024-04-24T08:08:21.362588+0000","last_fullsized":"2024-04-24T08:08:21.362588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:10:52.052957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539464+0000","last_change":"2024-04-24T08:08:20.139941+0000","last_active":"2024-04-24T08:08:22.539464+0000","last_peered":"2024-04-24T08:08:22.539464+0000","last_clean":"2024-04-24T08:08:22.539464+0000","last_became_active":"2024-04-24T08:08:20.139629+0000","last_became_peered":"2024-04-24T08:08:20.139629+0000","last_unstale":"2024-04-24T08:08:22.539464+0000","last_undegraded":"2024-04-24T08:08:22.539464+0000","last_fullsized":"2024-04-24T08:08:22.539464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:19:04.318854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152372+0000","last_change":"2024-04-24T08:08:20.148714+0000","last_active":"2024-04-24T08:08:21.152372+0000","last_peered":"2024-04-24T08:08:21.152372+0000","last_clean":"2024-04-24T08:08:21.152372+0000","last_became_active":"2024-04-24T08:08:20.148418+0000","last_became_peered":"2024-04-24T08:08:20.148418+0000","last_unstale":"2024-04-24T08:08:21.152372+0000","last_undegraded":"2024-04-24T08:08:21.152372+0000","last_fullsized":"2024-04-24T08:08:21.152372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887385+0000","last_change":"2024-04-24T08:08:20.131644+0000","last_active":"2024-04-24T08:08:21.887385+0000","last_peered":"2024-04-24T08:08:21.887385+0000","last_clean":"2024-04-24T08:08:21.887385+0000","last_became_active":"2024-04-24T08:08:20.131283+0000","last_became_peered":"2024-04-24T08:08:20.131283+0000","last_unstale":"2024-04-24T08:08:21.887385+0000","last_undegraded":"2024-04-24T08:08:21.887385+0000","last_fullsized":"2024-04-24T08:08:21.887385+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123520+0000","last_change":"2024-04-24T08:08:20.137461+0000","last_active":"2024-04-24T08:08:21.123520+0000","last_peered":"2024-04-24T08:08:21.123520+0000","last_clean":"2024-04-24T08:08:21.123520+0000","last_became_active":"2024-04-24T08:08:20.137181+0000","last_became_peered":"2024-04-24T08:08:20.137181+0000","last_unstale":"2024-04-24T08:08:21.123520+0000","last_undegraded":"2024-04-24T08:08:21.123520+0000","last_fullsized":"2024-04-24T08:08:21.123520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:15.025368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537451+0000","last_change":"2024-04-24T08:08:20.146731+0000","last_active":"2024-04-24T08:08:22.537451+0000","last_peered":"2024-04-24T08:08:22.537451+0000","last_clean":"2024-04-24T08:08:22.537451+0000","last_became_active":"2024-04-24T08:08:20.146216+0000","last_became_peered":"2024-04-24T08:08:20.146216+0000","last_unstale":"2024-04-24T08:08:22.537451+0000","last_undegraded":"2024-04-24T08:08:22.537451+0000","last_fullsized":"2024-04-24T08:08:22.537451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123388+0000","last_change":"2024-04-24T08:08:20.159848+0000","last_active":"2024-04-24T08:08:21.123388+0000","last_peered":"2024-04-24T08:08:21.123388+0000","last_clean":"2024-04-24T08:08:21.123388+0000","last_became_active":"2024-04-24T08:08:20.159530+0000","last_became_peered":"2024-04-24T08:08:20.159530+0000","last_unstale":"2024-04-24T08:08:21.123388+0000","last_undegraded":"2024-04-24T08:08:21.123388+0000","last_fullsized":"2024-04-24T08:08:21.123388+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537893+0000","last_change":"2024-04-24T08:08:20.153648+0000","last_active":"2024-04-24T08:08:22.537893+0000","last_peered":"2024-04-24T08:08:22.537893+0000","last_clean":"2024-04-24T08:08:22.537893+0000","last_became_active":"2024-04-24T08:08:20.153373+0000","last_became_peered":"2024-04-24T08:08:20.153373+0000","last_unstale":"2024-04-24T08:08:22.537893+0000","last_undegraded":"2024-04-24T08:08:22.537893+0000","last_fullsized":"2024-04-24T08:08:22.537893+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152686+0000","last_change":"2024-04-24T08:08:20.136593+0000","last_active":"2024-04-24T08:08:21.152686+0000","last_peered":"2024-04-24T08:08:21.152686+0000","last_clean":"2024-04-24T08:08:21.152686+0000","last_became_active":"2024-04-24T08:08:20.136373+0000","last_became_peered":"2024-04-24T08:08:20.136373+0000","last_unstale":"2024-04-24T08:08:21.152686+0000","last_undegraded":"2024-04-24T08:08:21.152686+0000","last_fullsized":"2024-04-24T08:08:21.152686+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121751+0000","last_change":"2024-04-24T08:08:20.161976+0000","last_active":"2024-04-24T08:08:21.121751+0000","last_peered":"2024-04-24T08:08:21.121751+0000","last_clean":"2024-04-24T08:08:21.121751+0000","last_became_active":"2024-04-24T08:08:20.161665+0000","last_became_peered":"2024-04-24T08:08:20.161665+0000","last_unstale":"2024-04-24T08:08:21.121751+0000","last_undegraded":"2024-04-24T08:08:21.121751+0000","last_fullsized":"2024-04-24T08:08:21.121751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538337+0000","last_change":"2024-04-24T08:08:20.155775+0000","last_active":"2024-04-24T08:08:22.538337+0000","last_peered":"2024-04-24T08:08:22.538337+0000","last_clean":"2024-04-24T08:08:22.538337+0000","last_became_active":"2024-04-24T08:08:20.152792+0000","last_became_peered":"2024-04-24T08:08:20.152792+0000","last_unstale":"2024-04-24T08:08:22.538337+0000","last_undegraded":"2024-04-24T08:08:22.538337+0000","last_fullsized":"2024-04-24T08:08:22.538337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:07:27.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887649+0000","last_change":"2024-04-24T08:08:20.152662+0000","last_active":"2024-04-24T08:08:21.887649+0000","last_peered":"2024-04-24T08:08:21.887649+0000","last_clean":"2024-04-24T08:08:21.887649+0000","last_became_active":"2024-04-24T08:08:20.143770+0000","last_became_peered":"2024-04-24T08:08:20.143770+0000","last_unstale":"2024-04-24T08:08:21.887649+0000","last_undegraded":"2024-04-24T08:08:21.887649+0000","last_fullsized":"2024-04-24T08:08:21.887649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:38:26.223234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123612+0000","last_change":"2024-04-24T08:08:20.164390+0000","last_active":"2024-04-24T08:08:21.123612+0000","last_peered":"2024-04-24T08:08:21.123612+0000","last_clean":"2024-04-24T08:08:21.123612+0000","last_became_active":"2024-04-24T08:08:20.164063+0000","last_became_peered":"2024-04-24T08:08:20.164063+0000","last_unstale":"2024-04-24T08:08:21.123612+0000","last_undegraded":"2024-04-24T08:08:21.123612+0000","last_fullsized":"2024-04-24T08:08:21.123612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:07:27.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362285+0000","last_change":"2024-04-24T08:08:20.140288+0000","last_active":"2024-04-24T08:08:21.362285+0000","last_peered":"2024-04-24T08:08:21.362285+0000","last_clean":"2024-04-24T08:08:21.362285+0000","last_became_active":"2024-04-24T08:08:20.139983+0000","last_became_peered":"2024-04-24T08:08:20.139983+0000","last_unstale":"2024-04-24T08:08:21.362285+0000","last_undegraded":"2024-04-24T08:08:21.362285+0000","last_fullsized":"2024-04-24T08:08:21.362285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:46.310762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151856+0000","last_change":"2024-04-24T08:08:20.143305+0000","last_active":"2024-04-24T08:08:21.151856+0000","last_peered":"2024-04-24T08:08:21.151856+0000","last_clean":"2024-04-24T08:08:21.151856+0000","last_became_active":"2024-04-24T08:08:20.142984+0000","last_became_peered":"2024-04-24T08:08:20.142984+0000","last_unstale":"2024-04-24T08:08:21.151856+0000","last_undegraded":"2024-04-24T08:08:21.151856+0000","last_fullsized":"2024-04-24T08:08:21.151856+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:40.157193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124584+0000","last_change":"2024-04-24T08:08:20.139312+0000","last_active":"2024-04-24T08:08:21.124584+0000","last_peered":"2024-04-24T08:08:21.124584+0000","last_clean":"2024-04-24T08:08:21.124584+0000","last_became_active":"2024-04-24T08:08:20.137881+0000","last_became_peered":"2024-04-24T08:08:20.137881+0000","last_unstale":"2024-04-24T08:08:21.124584+0000","last_undegraded":"2024-04-24T08:08:21.124584+0000","last_fullsized":"2024-04-24T08:08:21.124584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:28.189776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538238+0000","last_change":"2024-04-24T08:08:19.144554+0000","last_active":"2024-04-24T08:08:22.538238+0000","last_peered":"2024-04-24T08:08:22.538238+0000","last_clean":"2024-04-24T08:08:22.538238+0000","last_became_active":"2024-04-24T08:08:19.144285+0000","last_became_peered":"2024-04-24T08:08:19.144285+0000","last_unstale":"2024-04-24T08:08:22.538238+0000","last_undegraded":"2024-04-24T08:08:22.538238+0000","last_fullsized":"2024-04-24T08:08:22.538238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:02:50.869449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887732+0000","last_change":"2024-04-24T08:08:20.153872+0000","last_active":"2024-04-24T08:08:21.887732+0000","last_peered":"2024-04-24T08:08:21.887732+0000","last_clean":"2024-04-24T08:08:21.887732+0000","last_became_active":"2024-04-24T08:08:20.153593+0000","last_became_peered":"2024-04-24T08:08:20.153593+0000","last_unstale":"2024-04-24T08:08:21.887732+0000","last_undegraded":"2024-04-24T08:08:21.887732+0000","last_fullsized":"2024-04-24T08:08:21.887732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:53.941683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151661+0000","last_change":"2024-04-24T08:08:19.145643+0000","last_active":"2024-04-24T08:08:21.151661+0000","last_peered":"2024-04-24T08:08:21.151661+0000","last_clean":"2024-04-24T08:08:21.151661+0000","last_became_active":"2024-04-24T08:08:19.141228+0000","last_became_peered":"2024-04-24T08:08:19.141228+0000","last_unstale":"2024-04-24T08:08:21.151661+0000","last_undegraded":"2024-04-24T08:08:21.151661+0000","last_fullsized":"2024-04-24T08:08:21.151661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537587+0000","last_change":"2024-04-24T08:08:19.150248+0000","last_active":"2024-04-24T08:08:22.537587+0000","last_peered":"2024-04-24T08:08:22.537587+0000","last_clean":"2024-04-24T08:08:22.537587+0000","last_became_active":"2024-04-24T08:08:19.149979+0000","last_became_peered":"2024-04-24T08:08:19.149979+0000","last_unstale":"2024-04-24T08:08:22.537587+0000","last_undegraded":"2024-04-24T08:08:22.537587+0000","last_fullsized":"2024-04-24T08:08:22.537587+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:13:42.392189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538422+0000","last_change":"2024-04-24T08:08:20.155184+0000","last_active":"2024-04-24T08:08:22.538422+0000","last_peered":"2024-04-24T08:08:22.538422+0000","last_clean":"2024-04-24T08:08:22.538422+0000","last_became_active":"2024-04-24T08:08:20.154858+0000","last_became_peered":"2024-04-24T08:08:20.154858+0000","last_unstale":"2024-04-24T08:08:22.538422+0000","last_undegraded":"2024-04-24T08:08:22.538422+0000","last_fullsized":"2024-04-24T08:08:22.538422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:34.119848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123341+0000","last_change":"2024-04-24T08:08:19.138408+0000","last_active":"2024-04-24T08:08:21.123341+0000","last_peered":"2024-04-24T08:08:21.123341+0000","last_clean":"2024-04-24T08:08:21.123341+0000","last_became_active":"2024-04-24T08:08:19.137042+0000","last_became_peered":"2024-04-24T08:08:19.137042+0000","last_unstale":"2024-04-24T08:08:21.123341+0000","last_undegraded":"2024-04-24T08:08:21.123341+0000","last_fullsized":"2024-04-24T08:08:21.123341+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121298+0000","last_change":"2024-04-24T08:08:20.164732+0000","last_active":"2024-04-24T08:08:21.121298+0000","last_peered":"2024-04-24T08:08:21.121298+0000","last_clean":"2024-04-24T08:08:21.121298+0000","last_became_active":"2024-04-24T08:08:20.164193+0000","last_became_peered":"2024-04-24T08:08:20.164193+0000","last_unstale":"2024-04-24T08:08:21.121298+0000","last_undegraded":"2024-04-24T08:08:21.121298+0000","last_fullsized":"2024-04-24T08:08:21.121298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152443+0000","last_change":"2024-04-24T08:08:19.151929+0000","last_active":"2024-04-24T08:08:21.152443+0000","last_peered":"2024-04-24T08:08:21.152443+0000","last_clean":"2024-04-24T08:08:21.152443+0000","last_became_active":"2024-04-24T08:08:19.151633+0000","last_became_peered":"2024-04-24T08:08:19.151633+0000","last_unstale":"2024-04-24T08:08:21.152443+0000","last_undegraded":"2024-04-24T08:08:21.152443+0000","last_fullsized":"2024-04-24T08:08:21.152443+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125502+0000","last_change":"2024-04-24T08:08:20.139547+0000","last_active":"2024-04-24T08:08:21.125502+0000","last_peered":"2024-04-24T08:08:21.125502+0000","last_clean":"2024-04-24T08:08:21.125502+0000","last_became_active":"2024-04-24T08:08:20.139316+0000","last_became_peered":"2024-04-24T08:08:20.139316+0000","last_unstale":"2024-04-24T08:08:21.125502+0000","last_undegraded":"2024-04-24T08:08:21.125502+0000","last_fullsized":"2024-04-24T08:08:21.125502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:37.292548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125476+0000","last_change":"2024-04-24T08:08:19.150665+0000","last_active":"2024-04-24T08:08:21.125476+0000","last_peered":"2024-04-24T08:08:21.125476+0000","last_clean":"2024-04-24T08:08:21.125476+0000","last_became_active":"2024-04-24T08:08:19.150327+0000","last_became_peered":"2024-04-24T08:08:19.150327+0000","last_unstale":"2024-04-24T08:08:21.125476+0000","last_undegraded":"2024-04-24T08:08:21.125476+0000","last_fullsized":"2024-04-24T08:08:21.125476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887629+0000","last_change":"2024-04-24T08:08:20.138279+0000","last_active":"2024-04-24T08:08:21.887629+0000","last_peered":"2024-04-24T08:08:21.887629+0000","last_clean":"2024-04-24T08:08:21.887629+0000","last_became_active":"2024-04-24T08:08:20.137776+0000","last_became_peered":"2024-04-24T08:08:20.137776+0000","last_unstale":"2024-04-24T08:08:21.887629+0000","last_undegraded":"2024-04-24T08:08:21.887629+0000","last_fullsized":"2024-04-24T08:08:21.887629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:09.887236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121623+0000","last_change":"2024-04-24T08:08:19.143886+0000","last_active":"2024-04-24T08:08:21.121623+0000","last_peered":"2024-04-24T08:08:21.121623+0000","last_clean":"2024-04-24T08:08:21.121623+0000","last_became_active":"2024-04-24T08:08:19.143606+0000","last_became_peered":"2024-04-24T08:08:19.143606+0000","last_unstale":"2024-04-24T08:08:21.121623+0000","last_undegraded":"2024-04-24T08:08:21.121623+0000","last_fullsized":"2024-04-24T08:08:21.121623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126085+0000","last_change":"2024-04-24T08:08:20.146770+0000","last_active":"2024-04-24T08:08:21.126085+0000","last_peered":"2024-04-24T08:08:21.126085+0000","last_clean":"2024-04-24T08:08:21.126085+0000","last_became_active":"2024-04-24T08:08:20.146470+0000","last_became_peered":"2024-04-24T08:08:20.146470+0000","last_unstale":"2024-04-24T08:08:21.126085+0000","last_undegraded":"2024-04-24T08:08:21.126085+0000","last_fullsized":"2024-04-24T08:08:21.126085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:46.310762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152147+0000","last_change":"2024-04-24T08:08:19.149833+0000","last_active":"2024-04-24T08:08:21.152147+0000","last_peered":"2024-04-24T08:08:21.152147+0000","last_clean":"2024-04-24T08:08:21.152147+0000","last_became_active":"2024-04-24T08:08:19.149391+0000","last_became_peered":"2024-04-24T08:08:19.149391+0000","last_unstale":"2024-04-24T08:08:21.152147+0000","last_undegraded":"2024-04-24T08:08:21.152147+0000","last_fullsized":"2024-04-24T08:08:21.152147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538676+0000","last_change":"2024-04-24T08:08:20.145962+0000","last_active":"2024-04-24T08:08:22.538676+0000","last_peered":"2024-04-24T08:08:22.538676+0000","last_clean":"2024-04-24T08:08:22.538676+0000","last_became_active":"2024-04-24T08:08:20.145644+0000","last_became_peered":"2024-04-24T08:08:20.145644+0000","last_unstale":"2024-04-24T08:08:22.538676+0000","last_undegraded":"2024-04-24T08:08:22.538676+0000","last_fullsized":"2024-04-24T08:08:22.538676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:37.292548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887814+0000","last_change":"2024-04-24T08:08:19.131316+0000","last_active":"2024-04-24T08:08:21.887814+0000","last_peered":"2024-04-24T08:08:21.887814+0000","last_clean":"2024-04-24T08:08:21.887814+0000","last_became_active":"2024-04-24T08:08:19.130895+0000","last_became_peered":"2024-04-24T08:08:19.130895+0000","last_unstale":"2024-04-24T08:08:21.887814+0000","last_undegraded":"2024-04-24T08:08:21.887814+0000","last_fullsized":"2024-04-24T08:08:21.887814+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:13:42.392189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.886980+0000","last_change":"2024-04-24T08:08:20.123273+0000","last_active":"2024-04-24T08:08:21.886980+0000","last_peered":"2024-04-24T08:08:21.886980+0000","last_clean":"2024-04-24T08:08:21.886980+0000","last_became_active":"2024-04-24T08:08:20.122506+0000","last_became_peered":"2024-04-24T08:08:20.122506+0000","last_unstale":"2024-04-24T08:08:21.886980+0000","last_undegraded":"2024-04-24T08:08:21.886980+0000","last_fullsized":"2024-04-24T08:08:21.886980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:26:40.157193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123661+0000","last_change":"2024-04-24T08:08:19.138216+0000","last_active":"2024-04-24T08:08:21.123661+0000","last_peered":"2024-04-24T08:08:21.123661+0000","last_clean":"2024-04-24T08:08:21.123661+0000","last_became_active":"2024-04-24T08:08:19.137882+0000","last_became_peered":"2024-04-24T08:08:19.137882+0000","last_unstale":"2024-04-24T08:08:21.123661+0000","last_undegraded":"2024-04-24T08:08:21.123661+0000","last_fullsized":"2024-04-24T08:08:21.123661+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538907+0000","last_change":"2024-04-24T08:08:20.155343+0000","last_active":"2024-04-24T08:08:22.538907+0000","last_peered":"2024-04-24T08:08:22.538907+0000","last_clean":"2024-04-24T08:08:22.538907+0000","last_became_active":"2024-04-24T08:08:20.155112+0000","last_became_peered":"2024-04-24T08:08:20.155112+0000","last_unstale":"2024-04-24T08:08:22.538907+0000","last_undegraded":"2024-04-24T08:08:22.538907+0000","last_fullsized":"2024-04-24T08:08:22.538907+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:36.215783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151585+0000","last_change":"2024-04-24T08:08:19.135440+0000","last_active":"2024-04-24T08:08:21.151585+0000","last_peered":"2024-04-24T08:08:21.151585+0000","last_clean":"2024-04-24T08:08:21.151585+0000","last_became_active":"2024-04-24T08:08:19.134946+0000","last_became_peered":"2024-04-24T08:08:19.134946+0000","last_unstale":"2024-04-24T08:08:21.151585+0000","last_undegraded":"2024-04-24T08:08:21.151585+0000","last_fullsized":"2024-04-24T08:08:21.151585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151891+0000","last_change":"2024-04-24T08:08:20.144523+0000","last_active":"2024-04-24T08:08:21.151891+0000","last_peered":"2024-04-24T08:08:21.151891+0000","last_clean":"2024-04-24T08:08:21.151891+0000","last_became_active":"2024-04-24T08:08:20.142266+0000","last_became_peered":"2024-04-24T08:08:20.142266+0000","last_unstale":"2024-04-24T08:08:21.151891+0000","last_undegraded":"2024-04-24T08:08:21.151891+0000","last_fullsized":"2024-04-24T08:08:21.151891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.361922+0000","last_change":"2024-04-24T08:08:19.110993+0000","last_active":"2024-04-24T08:08:21.361922+0000","last_peered":"2024-04-24T08:08:21.361922+0000","last_clean":"2024-04-24T08:08:21.361922+0000","last_became_active":"2024-04-24T08:08:19.110592+0000","last_became_peered":"2024-04-24T08:08:19.110592+0000","last_unstale":"2024-04-24T08:08:21.361922+0000","last_undegraded":"2024-04-24T08:08:21.361922+0000","last_fullsized":"2024-04-24T08:08:21.361922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538312+0000","last_change":"2024-04-24T08:08:20.149332+0000","last_active":"2024-04-24T08:08:22.538312+0000","last_peered":"2024-04-24T08:08:22.538312+0000","last_clean":"2024-04-24T08:08:22.538312+0000","last_became_active":"2024-04-24T08:08:20.148856+0000","last_became_peered":"2024-04-24T08:08:20.148856+0000","last_unstale":"2024-04-24T08:08:22.538312+0000","last_undegraded":"2024-04-24T08:08:22.538312+0000","last_fullsized":"2024-04-24T08:08:22.538312+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:07:27.577802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125565+0000","last_change":"2024-04-24T08:08:19.128023+0000","last_active":"2024-04-24T08:08:21.125565+0000","last_peered":"2024-04-24T08:08:21.125565+0000","last_clean":"2024-04-24T08:08:21.125565+0000","last_became_active":"2024-04-24T08:08:19.126833+0000","last_became_peered":"2024-04-24T08:08:19.126833+0000","last_unstale":"2024-04-24T08:08:21.125565+0000","last_undegraded":"2024-04-24T08:08:21.125565+0000","last_fullsized":"2024-04-24T08:08:21.125565+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:45.062494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539639+0000","last_change":"2024-04-24T08:08:20.144208+0000","last_active":"2024-04-24T08:08:22.539639+0000","last_peered":"2024-04-24T08:08:22.539639+0000","last_clean":"2024-04-24T08:08:22.539639+0000","last_became_active":"2024-04-24T08:08:20.143708+0000","last_became_peered":"2024-04-24T08:08:20.143708+0000","last_unstale":"2024-04-24T08:08:22.539639+0000","last_undegraded":"2024-04-24T08:08:22.539639+0000","last_fullsized":"2024-04-24T08:08:22.539639+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:58:28.805520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362439+0000","last_change":"2024-04-24T08:08:19.137638+0000","last_active":"2024-04-24T08:08:21.362439+0000","last_peered":"2024-04-24T08:08:21.362439+0000","last_clean":"2024-04-24T08:08:21.362439+0000","last_became_active":"2024-04-24T08:08:19.137280+0000","last_became_peered":"2024-04-24T08:08:19.137280+0000","last_unstale":"2024-04-24T08:08:21.362439+0000","last_undegraded":"2024-04-24T08:08:21.362439+0000","last_fullsized":"2024-04-24T08:08:21.362439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:02:50.869449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539746+0000","last_change":"2024-04-24T08:08:20.140656+0000","last_active":"2024-04-24T08:08:22.539746+0000","last_peered":"2024-04-24T08:08:22.539746+0000","last_clean":"2024-04-24T08:08:22.539746+0000","last_became_active":"2024-04-24T08:08:20.140347+0000","last_became_peered":"2024-04-24T08:08:20.140347+0000","last_unstale":"2024-04-24T08:08:22.539746+0000","last_undegraded":"2024-04-24T08:08:22.539746+0000","last_fullsized":"2024-04-24T08:08:22.539746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:13:39.728625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121835+0000","last_change":"2024-04-24T08:08:19.127246+0000","last_active":"2024-04-24T08:08:21.121835+0000","last_peered":"2024-04-24T08:08:21.121835+0000","last_clean":"2024-04-24T08:08:21.121835+0000","last_became_active":"2024-04-24T08:08:19.126190+0000","last_became_peered":"2024-04-24T08:08:19.126190+0000","last_unstale":"2024-04-24T08:08:21.121835+0000","last_undegraded":"2024-04-24T08:08:21.121835+0000","last_fullsized":"2024-04-24T08:08:21.121835+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126132+0000","last_change":"2024-04-24T08:08:20.161962+0000","last_active":"2024-04-24T08:08:21.126132+0000","last_peered":"2024-04-24T08:08:21.126132+0000","last_clean":"2024-04-24T08:08:21.126132+0000","last_became_active":"2024-04-24T08:08:20.161572+0000","last_became_peered":"2024-04-24T08:08:20.161572+0000","last_unstale":"2024-04-24T08:08:21.126132+0000","last_undegraded":"2024-04-24T08:08:21.126132+0000","last_fullsized":"2024-04-24T08:08:21.126132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:56:58.521686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362140+0000","last_change":"2024-04-24T08:08:19.140345+0000","last_active":"2024-04-24T08:08:21.362140+0000","last_peered":"2024-04-24T08:08:21.362140+0000","last_clean":"2024-04-24T08:08:21.362140+0000","last_became_active":"2024-04-24T08:08:19.140122+0000","last_became_peered":"2024-04-24T08:08:19.140122+0000","last_unstale":"2024-04-24T08:08:21.362140+0000","last_undegraded":"2024-04-24T08:08:21.362140+0000","last_fullsized":"2024-04-24T08:08:21.362140+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539177+0000","last_change":"2024-04-24T08:08:20.137838+0000","last_active":"2024-04-24T08:08:22.539177+0000","last_peered":"2024-04-24T08:08:22.539177+0000","last_clean":"2024-04-24T08:08:22.539177+0000","last_became_active":"2024-04-24T08:08:20.137552+0000","last_became_peered":"2024-04-24T08:08:20.137552+0000","last_unstale":"2024-04-24T08:08:22.539177+0000","last_undegraded":"2024-04-24T08:08:22.539177+0000","last_fullsized":"2024-04-24T08:08:22.539177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:56:58.521686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538262+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:08:22.538262+0000","last_peered":"2024-04-24T08:08:22.538262+0000","last_clean":"2024-04-24T08:08:22.538262+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:08:22.538262+0000","last_undegraded":"2024-04-24T08:08:22.538262+0000","last_fullsized":"2024-04-24T08:08:22.538262+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124383+0000","last_change":"2024-04-24T08:08:19.135677+0000","last_active":"2024-04-24T08:08:21.124383+0000","last_peered":"2024-04-24T08:08:21.124383+0000","last_clean":"2024-04-24T08:08:21.124383+0000","last_became_active":"2024-04-24T08:08:19.133935+0000","last_became_peered":"2024-04-24T08:08:19.133935+0000","last_unstale":"2024-04-24T08:08:21.124383+0000","last_undegraded":"2024-04-24T08:08:21.124383+0000","last_fullsized":"2024-04-24T08:08:21.124383+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537560+0000","last_change":"2024-04-24T08:08:20.148292+0000","last_active":"2024-04-24T08:08:22.537560+0000","last_peered":"2024-04-24T08:08:22.537560+0000","last_clean":"2024-04-24T08:08:22.537560+0000","last_became_active":"2024-04-24T08:08:20.147966+0000","last_became_peered":"2024-04-24T08:08:20.147966+0000","last_unstale":"2024-04-24T08:08:22.537560+0000","last_undegraded":"2024-04-24T08:08:22.537560+0000","last_fullsized":"2024-04-24T08:08:22.537560+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:21.241375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537956+0000","last_change":"2024-04-24T08:08:19.148444+0000","last_active":"2024-04-24T08:08:22.537956+0000","last_peered":"2024-04-24T08:08:22.537956+0000","last_clean":"2024-04-24T08:08:22.537956+0000","last_became_active":"2024-04-24T08:08:19.148032+0000","last_became_peered":"2024-04-24T08:08:19.148032+0000","last_unstale":"2024-04-24T08:08:22.537956+0000","last_undegraded":"2024-04-24T08:08:22.537956+0000","last_fullsized":"2024-04-24T08:08:22.537956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:48.903851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.886751+0000","last_change":"2024-04-24T08:08:20.139133+0000","last_active":"2024-04-24T08:08:21.886751+0000","last_peered":"2024-04-24T08:08:21.886751+0000","last_clean":"2024-04-24T08:08:21.886751+0000","last_became_active":"2024-04-24T08:08:20.138841+0000","last_became_peered":"2024-04-24T08:08:20.138841+0000","last_unstale":"2024-04-24T08:08:21.886751+0000","last_undegraded":"2024-04-24T08:08:21.886751+0000","last_fullsized":"2024-04-24T08:08:21.886751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:29:21.637806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.362188+0000","last_change":"2024-04-24T08:08:19.135322+0000","last_active":"2024-04-24T08:08:21.362188+0000","last_peered":"2024-04-24T08:08:21.362188+0000","last_clean":"2024-04-24T08:08:21.362188+0000","last_became_active":"2024-04-24T08:08:19.134982+0000","last_became_peered":"2024-04-24T08:08:19.134982+0000","last_unstale":"2024-04-24T08:08:21.362188+0000","last_undegraded":"2024-04-24T08:08:21.362188+0000","last_fullsized":"2024-04-24T08:08:21.362188+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121142+0000","last_change":"2024-04-24T08:08:20.142759+0000","last_active":"2024-04-24T08:08:21.121142+0000","last_peered":"2024-04-24T08:08:21.121142+0000","last_clean":"2024-04-24T08:08:21.121142+0000","last_became_active":"2024-04-24T08:08:20.142474+0000","last_became_peered":"2024-04-24T08:08:20.142474+0000","last_unstale":"2024-04-24T08:08:21.121142+0000","last_undegraded":"2024-04-24T08:08:21.121142+0000","last_fullsized":"2024-04-24T08:08:21.121142+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:04:14.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.123701+0000","last_change":"2024-04-24T08:08:19.131093+0000","last_active":"2024-04-24T08:08:21.123701+0000","last_peered":"2024-04-24T08:08:21.123701+0000","last_clean":"2024-04-24T08:08:21.123701+0000","last_became_active":"2024-04-24T08:08:19.129809+0000","last_became_peered":"2024-04-24T08:08:19.129809+0000","last_unstale":"2024-04-24T08:08:21.123701+0000","last_undegraded":"2024-04-24T08:08:21.123701+0000","last_fullsized":"2024-04-24T08:08:21.123701+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124958+0000","last_change":"2024-04-24T08:08:20.155888+0000","last_active":"2024-04-24T08:08:21.124958+0000","last_peered":"2024-04-24T08:08:21.124958+0000","last_clean":"2024-04-24T08:08:21.124958+0000","last_became_active":"2024-04-24T08:08:20.154101+0000","last_became_peered":"2024-04-24T08:08:20.154101+0000","last_unstale":"2024-04-24T08:08:21.124958+0000","last_undegraded":"2024-04-24T08:08:21.124958+0000","last_fullsized":"2024-04-24T08:08:21.124958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:16:49.918407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121420+0000","last_change":"2024-04-24T08:08:19.128197+0000","last_active":"2024-04-24T08:08:21.121420+0000","last_peered":"2024-04-24T08:08:21.121420+0000","last_clean":"2024-04-24T08:08:21.121420+0000","last_became_active":"2024-04-24T08:08:19.127218+0000","last_became_peered":"2024-04-24T08:08:19.127218+0000","last_unstale":"2024-04-24T08:08:21.121420+0000","last_undegraded":"2024-04-24T08:08:21.121420+0000","last_fullsized":"2024-04-24T08:08:21.121420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:45.296206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126415+0000","last_change":"2024-04-24T08:08:20.159839+0000","last_active":"2024-04-24T08:08:21.126415+0000","last_peered":"2024-04-24T08:08:21.126415+0000","last_clean":"2024-04-24T08:08:21.126415+0000","last_became_active":"2024-04-24T08:08:20.159475+0000","last_became_peered":"2024-04-24T08:08:20.159475+0000","last_unstale":"2024-04-24T08:08:21.126415+0000","last_undegraded":"2024-04-24T08:08:21.126415+0000","last_fullsized":"2024-04-24T08:08:21.126415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:03:56.009457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539328+0000","last_change":"2024-04-24T08:08:19.148608+0000","last_active":"2024-04-24T08:08:22.539328+0000","last_peered":"2024-04-24T08:08:22.539328+0000","last_clean":"2024-04-24T08:08:22.539328+0000","last_became_active":"2024-04-24T08:08:19.148299+0000","last_became_peered":"2024-04-24T08:08:19.148299+0000","last_unstale":"2024-04-24T08:08:22.539328+0000","last_undegraded":"2024-04-24T08:08:22.539328+0000","last_fullsized":"2024-04-24T08:08:22.539328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:45.296206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121527+0000","last_change":"2024-04-24T08:08:20.144511+0000","last_active":"2024-04-24T08:08:21.121527+0000","last_peered":"2024-04-24T08:08:21.121527+0000","last_clean":"2024-04-24T08:08:21.121527+0000","last_became_active":"2024-04-24T08:08:20.144184+0000","last_became_peered":"2024-04-24T08:08:20.144184+0000","last_unstale":"2024-04-24T08:08:21.121527+0000","last_undegraded":"2024-04-24T08:08:21.121527+0000","last_fullsized":"2024-04-24T08:08:21.121527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.122119+0000","last_change":"2024-04-24T08:08:19.149355+0000","last_active":"2024-04-24T08:08:21.122119+0000","last_peered":"2024-04-24T08:08:21.122119+0000","last_clean":"2024-04-24T08:08:21.122119+0000","last_became_active":"2024-04-24T08:08:19.148936+0000","last_became_peered":"2024-04-24T08:08:19.148936+0000","last_unstale":"2024-04-24T08:08:21.122119+0000","last_undegraded":"2024-04-24T08:08:21.122119+0000","last_fullsized":"2024-04-24T08:08:21.122119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:49.339006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538134+0000","last_change":"2024-04-24T08:08:20.153837+0000","last_active":"2024-04-24T08:08:22.538134+0000","last_peered":"2024-04-24T08:08:22.538134+0000","last_clean":"2024-04-24T08:08:22.538134+0000","last_became_active":"2024-04-24T08:08:20.150347+0000","last_became_peered":"2024-04-24T08:08:20.150347+0000","last_unstale":"2024-04-24T08:08:22.538134+0000","last_undegraded":"2024-04-24T08:08:22.538134+0000","last_fullsized":"2024-04-24T08:08:22.538134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:53.941683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.152520+0000","last_change":"2024-04-24T08:08:19.143650+0000","last_active":"2024-04-24T08:08:21.152520+0000","last_peered":"2024-04-24T08:08:21.152520+0000","last_clean":"2024-04-24T08:08:21.152520+0000","last_became_active":"2024-04-24T08:08:19.143321+0000","last_became_peered":"2024-04-24T08:08:19.143321+0000","last_unstale":"2024-04-24T08:08:21.152520+0000","last_undegraded":"2024-04-24T08:08:21.152520+0000","last_fullsized":"2024-04-24T08:08:21.152520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887531+0000","last_change":"2024-04-24T08:08:20.142865+0000","last_active":"2024-04-24T08:08:21.887531+0000","last_peered":"2024-04-24T08:08:21.887531+0000","last_clean":"2024-04-24T08:08:21.887531+0000","last_became_active":"2024-04-24T08:08:20.142593+0000","last_became_peered":"2024-04-24T08:08:20.142593+0000","last_unstale":"2024-04-24T08:08:21.887531+0000","last_undegraded":"2024-04-24T08:08:21.887531+0000","last_fullsized":"2024-04-24T08:08:21.887531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537886+0000","last_change":"2024-04-24T08:08:19.146717+0000","last_active":"2024-04-24T08:08:22.537886+0000","last_peered":"2024-04-24T08:08:22.537886+0000","last_clean":"2024-04-24T08:08:22.537886+0000","last_became_active":"2024-04-24T08:08:19.146435+0000","last_became_peered":"2024-04-24T08:08:19.146435+0000","last_unstale":"2024-04-24T08:08:22.537886+0000","last_undegraded":"2024-04-24T08:08:22.537886+0000","last_fullsized":"2024-04-24T08:08:22.537886+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:25.455075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.124833+0000","last_change":"2024-04-24T08:08:20.153349+0000","last_active":"2024-04-24T08:08:21.124833+0000","last_peered":"2024-04-24T08:08:21.124833+0000","last_clean":"2024-04-24T08:08:21.124833+0000","last_became_active":"2024-04-24T08:08:20.153071+0000","last_became_peered":"2024-04-24T08:08:20.153071+0000","last_unstale":"2024-04-24T08:08:21.124833+0000","last_undegraded":"2024-04-24T08:08:21.124833+0000","last_fullsized":"2024-04-24T08:08:21.124833+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:26.469631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121526+0000","last_change":"2024-04-24T08:08:19.141216+0000","last_active":"2024-04-24T08:08:21.121526+0000","last_peered":"2024-04-24T08:08:21.121526+0000","last_clean":"2024-04-24T08:08:21.121526+0000","last_became_active":"2024-04-24T08:08:19.140885+0000","last_became_peered":"2024-04-24T08:08:19.140885+0000","last_unstale":"2024-04-24T08:08:21.121526+0000","last_undegraded":"2024-04-24T08:08:21.121526+0000","last_fullsized":"2024-04-24T08:08:21.121526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:13:17.489343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.537837+0000","last_change":"2024-04-24T08:08:20.139684+0000","last_active":"2024-04-24T08:08:22.537837+0000","last_peered":"2024-04-24T08:08:22.537837+0000","last_clean":"2024-04-24T08:08:22.537837+0000","last_became_active":"2024-04-24T08:08:20.139348+0000","last_became_peered":"2024-04-24T08:08:20.139348+0000","last_unstale":"2024-04-24T08:08:22.537837+0000","last_undegraded":"2024-04-24T08:08:22.537837+0000","last_fullsized":"2024-04-24T08:08:22.537837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:14.554291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.121275+0000","last_change":"2024-04-24T08:08:19.137264+0000","last_active":"2024-04-24T08:08:21.121275+0000","last_peered":"2024-04-24T08:08:21.121275+0000","last_clean":"2024-04-24T08:08:21.121275+0000","last_became_active":"2024-04-24T08:08:19.136921+0000","last_became_peered":"2024-04-24T08:08:19.136921+0000","last_unstale":"2024-04-24T08:08:21.121275+0000","last_undegraded":"2024-04-24T08:08:21.121275+0000","last_fullsized":"2024-04-24T08:08:21.121275+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:36:27.175220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538286+0000","last_change":"2024-04-24T08:08:20.158595+0000","last_active":"2024-04-24T08:08:22.538286+0000","last_peered":"2024-04-24T08:08:22.538286+0000","last_clean":"2024-04-24T08:08:22.538286+0000","last_became_active":"2024-04-24T08:08:20.158248+0000","last_became_peered":"2024-04-24T08:08:20.158248+0000","last_unstale":"2024-04-24T08:08:22.538286+0000","last_undegraded":"2024-04-24T08:08:22.538286+0000","last_fullsized":"2024-04-24T08:08:22.538286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:15.025368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.539152+0000","last_change":"2024-04-24T08:08:19.143021+0000","last_active":"2024-04-24T08:08:22.539152+0000","last_peered":"2024-04-24T08:08:22.539152+0000","last_clean":"2024-04-24T08:08:22.539152+0000","last_became_active":"2024-04-24T08:08:19.142731+0000","last_became_peered":"2024-04-24T08:08:19.142731+0000","last_unstale":"2024-04-24T08:08:22.539152+0000","last_undegraded":"2024-04-24T08:08:22.539152+0000","last_fullsized":"2024-04-24T08:08:22.539152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:10:51.038401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.151895+0000","last_change":"2024-04-24T08:08:20.141951+0000","last_active":"2024-04-24T08:08:21.151895+0000","last_peered":"2024-04-24T08:08:21.151895+0000","last_clean":"2024-04-24T08:08:21.151895+0000","last_became_active":"2024-04-24T08:08:20.141642+0000","last_became_peered":"2024-04-24T08:08:20.141642+0000","last_unstale":"2024-04-24T08:08:21.151895+0000","last_undegraded":"2024-04-24T08:08:21.151895+0000","last_fullsized":"2024-04-24T08:08:21.151895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.126001+0000","last_change":"2024-04-24T08:08:20.152685+0000","last_active":"2024-04-24T08:08:21.126001+0000","last_peered":"2024-04-24T08:08:21.126001+0000","last_clean":"2024-04-24T08:08:21.126001+0000","last_became_active":"2024-04-24T08:08:20.152171+0000","last_became_peered":"2024-04-24T08:08:20.152171+0000","last_unstale":"2024-04-24T08:08:21.126001+0000","last_undegraded":"2024-04-24T08:08:21.126001+0000","last_fullsized":"2024-04-24T08:08:21.126001+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:36.215783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538456+0000","last_change":"2024-04-24T08:08:19.147333+0000","last_active":"2024-04-24T08:08:22.538456+0000","last_peered":"2024-04-24T08:08:22.538456+0000","last_clean":"2024-04-24T08:08:22.538456+0000","last_became_active":"2024-04-24T08:08:19.147046+0000","last_became_peered":"2024-04-24T08:08:19.147046+0000","last_unstale":"2024-04-24T08:08:22.538456+0000","last_undegraded":"2024-04-24T08:08:22.538456+0000","last_fullsized":"2024-04-24T08:08:22.538456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:24:02.225409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.125720+0000","last_change":"2024-04-24T08:08:20.152622+0000","last_active":"2024-04-24T08:08:21.125720+0000","last_peered":"2024-04-24T08:08:21.125720+0000","last_clean":"2024-04-24T08:08:21.125720+0000","last_became_active":"2024-04-24T08:08:20.151784+0000","last_became_peered":"2024-04-24T08:08:20.151784+0000","last_unstale":"2024-04-24T08:08:21.125720+0000","last_undegraded":"2024-04-24T08:08:21.125720+0000","last_fullsized":"2024-04-24T08:08:21.125720+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:32:29.020308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538879+0000","last_change":"2024-04-24T08:08:19.150378+0000","last_active":"2024-04-24T08:08:22.538879+0000","last_peered":"2024-04-24T08:08:22.538879+0000","last_clean":"2024-04-24T08:08:22.538879+0000","last_became_active":"2024-04-24T08:08:19.149946+0000","last_became_peered":"2024-04-24T08:08:19.149946+0000","last_unstale":"2024-04-24T08:08:22.538879+0000","last_undegraded":"2024-04-24T08:08:22.538879+0000","last_fullsized":"2024-04-24T08:08:22.538879+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:49.339006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887243+0000","last_change":"2024-04-24T08:08:20.138065+0000","last_active":"2024-04-24T08:08:21.887243+0000","last_peered":"2024-04-24T08:08:21.887243+0000","last_clean":"2024-04-24T08:08:21.887243+0000","last_became_active":"2024-04-24T08:08:20.137340+0000","last_became_peered":"2024-04-24T08:08:20.137340+0000","last_unstale":"2024-04-24T08:08:21.887243+0000","last_undegraded":"2024-04-24T08:08:21.887243+0000","last_fullsized":"2024-04-24T08:08:21.887243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:38:51.345194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538608+0000","last_change":"2024-04-24T08:08:19.134141+0000","last_active":"2024-04-24T08:08:22.538608+0000","last_peered":"2024-04-24T08:08:22.538608+0000","last_clean":"2024-04-24T08:08:22.538608+0000","last_became_active":"2024-04-24T08:08:19.133873+0000","last_became_peered":"2024-04-24T08:08:19.133873+0000","last_unstale":"2024-04-24T08:08:22.538608+0000","last_undegraded":"2024-04-24T08:08:22.538608+0000","last_fullsized":"2024-04-24T08:08:22.538608+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:53.483157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:22.538569+0000","last_change":"2024-04-24T08:08:20.146963+0000","last_active":"2024-04-24T08:08:22.538569+0000","last_peered":"2024-04-24T08:08:22.538569+0000","last_clean":"2024-04-24T08:08:22.538569+0000","last_became_active":"2024-04-24T08:08:20.146270+0000","last_became_peered":"2024-04-24T08:08:20.146270+0000","last_unstale":"2024-04-24T08:08:22.538569+0000","last_undegraded":"2024-04-24T08:08:22.538569+0000","last_fullsized":"2024-04-24T08:08:22.538569+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:19.087922+0000","last_clean_scrub_stamp":"2024-04-24T08:08:19.087922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:37.292548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-04-24T08:08:21.887648+0000","last_change":"2024-04-24T08:08:19.142063+0000","last_active":"2024-04-24T08:08:21.887648+0000","last_peered":"2024-04-24T08:08:21.887648+0000","last_clean":"2024-04-24T08:08:21.887648+0000","last_became_active":"2024-04-24T08:08:19.141469+0000","last_became_peered":"2024-04-24T08:08:19.141469+0000","last_unstale":"2024-04-24T08:08:21.887648+0000","last_undegraded":"2024-04-24T08:08:21.887648+0000","last_fullsized":"2024-04-24T08:08:21.887648+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:18.073365+0000","last_clean_scrub_stamp":"2024-04-24T08:08:18.073365+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:46.176456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:08:30.423 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T08:08:30.423 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T08:08:30.423 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-24T08:08:30.423 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T08:08:30.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.656+0000 7f115ca00640 1 Processor -- start 2024-04-24T08:08:30.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.656+0000 7f115ca00640 1 -- start start 2024-04-24T08:08:30.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.658+0000 7f115ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f115815aa90 0x7f1158072bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.658+0000 7f115ca00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f115806e4e0 con 0x7f11580726f0 2024-04-24T08:08:30.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.658+0000 7f115ca00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f115806e760 con 0x7f1158071b50 2024-04-24T08:08:30.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.658+0000 7f115ca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f115806ea40 con 0x7f115815aa90 2024-04-24T08:08:30.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.658+0000 7f1156800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f115815aa90 0x7f1158072bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.658+0000 7f1156800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f115815aa90 0x7f1158072bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44438/0 (socket says 172.21.15.69:44438) 2024-04-24T08:08:30.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.658+0000 7f1156800640 1 -- 172.21.15.69:0/1869458796 learned_addr learned my addr 172.21.15.69:0/1869458796 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:30.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.658+0000 7f1156800640 1 -- 172.21.15.69:0/1869458796 >> v1:172.21.15.121:6789/0 conn(0x7f1158071b50 legacy=0x7f1158071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:30.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.658+0000 7f1156800640 1 -- 172.21.15.69:0/1869458796 >> v1:172.21.15.69:6789/0 conn(0x7f11580726f0 legacy=0x7f11580b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.658+0000 7f1156800640 1 -- 172.21.15.69:0/1869458796 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f115806ed20 con 0x7f115815aa90 2024-04-24T08:08:30.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.659+0000 7f1156800640 1 --2- 172.21.15.69:0/1869458796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f115815aa90 0x7f1158072bd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f114800bb30 tx=0x7f114802c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c5c8eca900cb61 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:30.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.660+0000 7f1155400640 1 -- 172.21.15.69:0/1869458796 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114803a070 con 0x7f115815aa90 2024-04-24T08:08:30.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.660+0000 7f1155400640 1 -- 172.21.15.69:0/1869458796 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1148032480 con 0x7f115815aa90 2024-04-24T08:08:30.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.660+0000 7f1155400640 1 -- 172.21.15.69:0/1869458796 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148032850 con 0x7f115815aa90 2024-04-24T08:08:30.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.660+0000 7f115ca00640 1 -- 172.21.15.69:0/1869458796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f115815aa90 msgr2=0x7f1158072bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.660+0000 7f115ca00640 1 --2- 172.21.15.69:0/1869458796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f115815aa90 0x7f1158072bd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f114800bb30 tx=0x7f114802c770 comp rx=0 tx=0).stop 2024-04-24T08:08:30.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.660+0000 7f115ca00640 1 -- 172.21.15.69:0/1869458796 shutdown_connections 2024-04-24T08:08:30.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.660+0000 7f115ca00640 1 --2- 172.21.15.69:0/1869458796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f115815aa90 0x7f1158072bd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.660+0000 7f115ca00640 1 -- 172.21.15.69:0/1869458796 >> 172.21.15.69:0/1869458796 conn(0x7f115806be40 msgr2=0x7f115806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:30.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.660+0000 7f115ca00640 1 -- 172.21.15.69:0/1869458796 shutdown_connections 2024-04-24T08:08:30.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.660+0000 7f115ca00640 1 -- 172.21.15.69:0/1869458796 wait complete. 2024-04-24T08:08:30.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.662+0000 7f115ca00640 1 Processor -- start 2024-04-24T08:08:30.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.662+0000 7f115ca00640 1 -- start start 2024-04-24T08:08:30.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.663+0000 7f115ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1158071b50 0x7f1158156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.663+0000 7f115ca00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11580726f0 0x7f11581f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.663+0000 7f115ca00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f115815aa90 0x7f1158159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.663+0000 7f115ca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f115815ae70 con 0x7f1158071b50 2024-04-24T08:08:30.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.663+0000 7f115ca00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f115806f2e0 con 0x7f11580726f0 2024-04-24T08:08:30.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.663+0000 7f115ca00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f115815aff0 con 0x7f115815aa90 2024-04-24T08:08:30.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.663+0000 7f1156800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1158071b50 0x7f1158156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.663+0000 7f1156800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1158071b50 0x7f1158156220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44446/0 (socket says 172.21.15.69:44446) 2024-04-24T08:08:30.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.663+0000 7f1156800640 1 -- 172.21.15.69:0/699791215 learned_addr learned my addr 172.21.15.69:0/699791215 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f1157200640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f115815aa90 0x7f1158159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f1156800640 1 -- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f115815aa90 msgr2=0x7f1158159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f1155e00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11580726f0 0x7f11581f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f1156800640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f115815aa90 0x7f1158159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f1156800640 1 -- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11580726f0 msgr2=0x7f11581f46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f1156800640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11580726f0 0x7f11581f46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f1156800640 1 -- 172.21.15.69:0/699791215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1158158700 con 0x7f1158071b50 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f1155e00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11580726f0 0x7f11581f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f1156800640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1158071b50 0x7f1158156220 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1148038040 tx=0x7f11480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f114ea00640 1 -- 172.21.15.69:0/699791215 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114803a040 con 0x7f1158071b50 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f114ea00640 1 -- 172.21.15.69:0/699791215 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1148040450 con 0x7f1158071b50 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f115ca00640 1 -- 172.21.15.69:0/699791215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1158156760 con 0x7f1158071b50 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f114ea00640 1 -- 172.21.15.69:0/699791215 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148040820 con 0x7f1158071b50 2024-04-24T08:08:30.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.664+0000 7f115ca00640 1 -- 172.21.15.69:0/699791215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1158158c80 con 0x7f1158071b50 2024-04-24T08:08:30.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.666+0000 7f115ca00640 1 -- 172.21.15.69:0/699791215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1158075d50 con 0x7f1158071b50 2024-04-24T08:08:30.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.666+0000 7f114ea00640 1 -- 172.21.15.69:0/699791215 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1148047020 con 0x7f1158071b50 2024-04-24T08:08:30.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.668+0000 7f114ea00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f113406c8f0 0x7f113406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:30.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.668+0000 7f114ea00640 1 -- 172.21.15.69:0/699791215 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f11480ac350 con 0x7f1158071b50 2024-04-24T08:08:30.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.669+0000 7f1155e00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f113406c8f0 0x7f113406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:30.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.670+0000 7f1155e00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f113406c8f0 0x7f113406edb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1140007fd0 tx=0x7f1140009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:30.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.673+0000 7f114ea00640 1 -- 172.21.15.69:0/699791215 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f114804e020 con 0x7f1158071b50 2024-04-24T08:08:30.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.895+0000 7f115ca00640 1 -- 172.21.15.69:0/699791215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f11580cf8e0 con 0x7f1158071b50 2024-04-24T08:08:30.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.896+0000 7f114ea00640 1 -- 172.21.15.69:0/699791215 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f114807a650 con 0x7f1158071b50 2024-04-24T08:08:30.897 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:30.897 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T08:08:30.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.898+0000 7f115ca00640 1 -- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f113406c8f0 msgr2=0x7f113406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.898+0000 7f115ca00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f113406c8f0 0x7f113406edb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1140007fd0 tx=0x7f1140009040 comp rx=0 tx=0).stop 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.898+0000 7f115ca00640 1 -- 172.21.15.69:0/699791215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1158071b50 msgr2=0x7f1158156220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.898+0000 7f115ca00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1158071b50 0x7f1158156220 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1148038040 tx=0x7f11480046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.898+0000 7f115ca00640 1 -- 172.21.15.69:0/699791215 shutdown_connections 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.898+0000 7f115ca00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f113406c8f0 0x7f113406edb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.898+0000 7f115ca00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f115815aa90 0x7f1158159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.898+0000 7f115ca00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11580726f0 0x7f11581f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.898+0000 7f115ca00640 1 --2- 172.21.15.69:0/699791215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1158071b50 0x7f1158156220 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.898+0000 7f115ca00640 1 -- 172.21.15.69:0/699791215 >> 172.21.15.69:0/699791215 conn(0x7f115806be40 msgr2=0x7f11580ad9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.899+0000 7f115ca00640 1 -- 172.21.15.69:0/699791215 shutdown_connections 2024-04-24T08:08:30.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:30.899+0000 7f115ca00640 1 -- 172.21.15.69:0/699791215 wait complete. 2024-04-24T08:08:30.917 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-24T08:08:30.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:31.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.107+0000 7fa77e800640 1 Processor -- start 2024-04-24T08:08:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.107+0000 7fa77e800640 1 -- start start 2024-04-24T08:08:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.109+0000 7fa77e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa7780b3ba0 0x7fa778163180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.109+0000 7fa77e800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7780d0d60 con 0x7fa7780d0880 2024-04-24T08:08:31.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.109+0000 7fa77e800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7780d1040 con 0x7fa77809b0e0 2024-04-24T08:08:31.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.109+0000 7fa77e800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa7781636c0 con 0x7fa7780b3ba0 2024-04-24T08:08:31.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.109+0000 7fa76fe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa7780d0880 0x7fa77809a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36614/0 (socket says 172.21.15.69:36614) 2024-04-24T08:08:31.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.109+0000 7fa76fe00640 1 -- 172.21.15.69:0/4234700574 learned_addr learned my addr 172.21.15.69:0/4234700574 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:31.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.109+0000 7fa777400640 1 --2- 172.21.15.69:0/4234700574 >> v2:172.21.15.69:3300/0 conn(0x7fa7780b3ba0 0x7fa778163180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.110+0000 7fa776a00640 1 -- 172.21.15.69:0/4234700574 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1361987889 0 0) 0x7fa7780d0d60 con 0x7fa7780d0880 2024-04-24T08:08:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.110+0000 7fa776a00640 1 -- 172.21.15.69:0/4234700574 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa75c003520 con 0x7fa7780d0880 2024-04-24T08:08:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.110+0000 7fa777400640 1 -- 172.21.15.69:0/4234700574 >> v1:172.21.15.121:6789/0 conn(0x7fa77809b0e0 legacy=0x7fa7780b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.110+0000 7fa777400640 1 -- 172.21.15.69:0/4234700574 >> v1:172.21.15.69:6789/0 conn(0x7fa7780d0880 legacy=0x7fa77809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.110+0000 7fa777400640 1 -- 172.21.15.69:0/4234700574 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7780d1320 con 0x7fa7780b3ba0 2024-04-24T08:08:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.111+0000 7fa777400640 1 --2- 172.21.15.69:0/4234700574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 0x7fa778163180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa7680097a0 tx=0x7fa76802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0f009409a4b4df server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.111+0000 7fa776a00640 1 -- 172.21.15.69:0/4234700574 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa768037070 con 0x7fa7780b3ba0 2024-04-24T08:08:31.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.111+0000 7fa776a00640 1 -- 172.21.15.69:0/4234700574 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa768032440 con 0x7fa7780b3ba0 2024-04-24T08:08:31.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.111+0000 7fa776a00640 1 -- 172.21.15.69:0/4234700574 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa768032810 con 0x7fa7780b3ba0 2024-04-24T08:08:31.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.112+0000 7fa77e800640 1 -- 172.21.15.69:0/4234700574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 msgr2=0x7fa778163180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.112+0000 7fa77e800640 1 --2- 172.21.15.69:0/4234700574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 0x7fa778163180 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa7680097a0 tx=0x7fa76802a230 comp rx=0 tx=0).stop 2024-04-24T08:08:31.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.112+0000 7fa77e800640 1 -- 172.21.15.69:0/4234700574 shutdown_connections 2024-04-24T08:08:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.112+0000 7fa77e800640 1 --2- 172.21.15.69:0/4234700574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 0x7fa778163180 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.112+0000 7fa77e800640 1 -- 172.21.15.69:0/4234700574 >> 172.21.15.69:0/4234700574 conn(0x7fa7780b40a0 msgr2=0x7fa7780b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.112+0000 7fa77e800640 1 -- 172.21.15.69:0/4234700574 shutdown_connections 2024-04-24T08:08:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.112+0000 7fa77e800640 1 -- 172.21.15.69:0/4234700574 wait complete. 2024-04-24T08:08:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.114+0000 7fa77e800640 1 Processor -- start 2024-04-24T08:08:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.114+0000 7fa77e800640 1 -- start start 2024-04-24T08:08:31.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa77e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa77809b0e0 0x7fa77814fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa77e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 0x7fa7781f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa77e800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7780d0880 0x7fa7781f4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa77e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa77809b940 con 0x7fa7780b3ba0 2024-04-24T08:08:31.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa77e800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa7780d18e0 con 0x7fa7780d0880 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa77e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa7780b2620 con 0x7fa77809b0e0 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa76fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 0x7fa7781f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa76fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 0x7fa7781f4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44458/0 (socket says 172.21.15.69:44458) 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa76fe00640 1 -- 172.21.15.69:0/607055660 learned_addr learned my addr 172.21.15.69:0/607055660 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa76fe00640 1 -- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa77809b0e0 msgr2=0x7fa77814fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa777400640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa77809b0e0 0x7fa77814fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa76fe00640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa77809b0e0 0x7fa77814fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa76fe00640 1 -- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7780d0880 msgr2=0x7fa7781f4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa777e00640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7780d0880 0x7fa7781f4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa76fe00640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7780d0880 0x7fa7781f4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.116+0000 7fa76fe00640 1 -- 172.21.15.69:0/607055660 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7780b5ed0 con 0x7fa7780b3ba0 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.117+0000 7fa76fe00640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 0x7fa7781f4730 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa76000c410 tx=0x7fa76000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.117+0000 7fa76f400640 1 -- 172.21.15.69:0/607055660 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa760017070 con 0x7fa7780b3ba0 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.117+0000 7fa76f400640 1 -- 172.21.15.69:0/607055660 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa760005ce0 con 0x7fa7780b3ba0 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.117+0000 7fa76f400640 1 -- 172.21.15.69:0/607055660 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7600072f0 con 0x7fa7780b3ba0 2024-04-24T08:08:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.117+0000 7fa77e800640 1 -- 172.21.15.69:0/607055660 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7780b61b0 con 0x7fa7780b3ba0 2024-04-24T08:08:31.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.117+0000 7fa77e800640 1 -- 172.21.15.69:0/607055660 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7780b6770 con 0x7fa7780b3ba0 2024-04-24T08:08:31.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.118+0000 7fa77e800640 1 -- 172.21.15.69:0/607055660 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa778068a80 con 0x7fa7780b3ba0 2024-04-24T08:08:31.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.119+0000 7fa76f400640 1 -- 172.21.15.69:0/607055660 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa76001a070 con 0x7fa7780b3ba0 2024-04-24T08:08:31.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.120+0000 7fa76f400640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa75406c8f0 0x7fa75406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.120+0000 7fa76f400640 1 -- 172.21.15.69:0/607055660 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa76008afa0 con 0x7fa7780b3ba0 2024-04-24T08:08:31.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.121+0000 7fa777400640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa75406c8f0 0x7fa75406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.122+0000 7fa777400640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa75406c8f0 0x7fa75406edb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa768034000 tx=0x7fa76802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:31.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.124+0000 7fa76f400640 1 -- 172.21.15.69:0/607055660 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa760059480 con 0x7fa7780b3ba0 2024-04-24T08:08:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.311+0000 7fa77e800640 1 -- 172.21.15.69:0/607055660 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa7780b4cc0 con 0x7fa7780b3ba0 2024-04-24T08:08:31.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.312+0000 7fa76f400640 1 -- 172.21.15.69:0/607055660 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7fa760082e20 con 0x7fa7780b3ba0 2024-04-24T08:08:31.314 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:31.314 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:31.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.315+0000 7fa77e800640 1 -- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa75406c8f0 msgr2=0x7fa75406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.315+0000 7fa77e800640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa75406c8f0 0x7fa75406edb0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa768034000 tx=0x7fa76802e000 comp rx=0 tx=0).stop 2024-04-24T08:08:31.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.315+0000 7fa77e800640 1 -- 172.21.15.69:0/607055660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 msgr2=0x7fa7781f4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.315+0000 7fa77e800640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 0x7fa7781f4730 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa76000c410 tx=0x7fa76000c8d0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.316+0000 7fa77e800640 1 -- 172.21.15.69:0/607055660 shutdown_connections 2024-04-24T08:08:31.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.316+0000 7fa77e800640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa75406c8f0 0x7fa75406edb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.316+0000 7fa77e800640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7780b3ba0 0x7fa7781f4730 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.316+0000 7fa77e800640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7780d0880 0x7fa7781f4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.316+0000 7fa77e800640 1 --2- 172.21.15.69:0/607055660 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa77809b0e0 0x7fa77814fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.316+0000 7fa77e800640 1 -- 172.21.15.69:0/607055660 >> 172.21.15.69:0/607055660 conn(0x7fa7780b40a0 msgr2=0x7fa7780b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:31.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.316+0000 7fa77e800640 1 -- 172.21.15.69:0/607055660 shutdown_connections 2024-04-24T08:08:31.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.316+0000 7fa77e800640 1 -- 172.21.15.69:0/607055660 wait complete. 2024-04-24T08:08:31.334 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:08:31.334 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.512+0000 7f452d000640 1 Processor -- start 2024-04-24T08:08:31.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.512+0000 7f452d000640 1 -- start start 2024-04-24T08:08:31.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.514+0000 7f452d000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f45280d09f0 0x7f4528166020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.514+0000 7f452d000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45280ce640 con 0x7f452815ea60 2024-04-24T08:08:31.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.514+0000 7f452d000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45280ce910 con 0x7f45280b5300 2024-04-24T08:08:31.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.514+0000 7f452d000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f45280cebf0 con 0x7f45280d09f0 2024-04-24T08:08:31.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.514+0000 7f4527400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f452815ea60 0x7f45280cdf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36626/0 (socket says 172.21.15.69:36626) 2024-04-24T08:08:31.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.514+0000 7f4527400640 1 -- 172.21.15.69:0/1597424145 learned_addr learned my addr 172.21.15.69:0/1597424145 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:31.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4032846594 0 0) 0x7f45280ce640 con 0x7f452815ea60 2024-04-24T08:08:31.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4527e00640 1 --2- 172.21.15.69:0/1597424145 >> v2:172.21.15.121:3300/0 conn(0x7f45280d09f0 0x7f4528166020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f450c003520 con 0x7f452815ea60 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3605069807 0 0) 0x7f450c003520 con 0x7f452815ea60 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 >> v2:172.21.15.121:3300/0 conn(0x7f45280d09f0 msgr2=0x7f4528166020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 --2- 172.21.15.69:0/1597424145 >> v2:172.21.15.121:3300/0 conn(0x7f45280d09f0 0x7f4528166020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 >> v1:172.21.15.121:6789/0 conn(0x7f45280b5300 legacy=0x7f45280d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45280ceed0 con 0x7f452815ea60 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f4518001670 con 0x7f452815ea60 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 >> v1:172.21.15.69:6789/0 conn(0x7f452815ea60 legacy=0x7f45280cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 --2- 172.21.15.69:0/1597424145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0039b0 0x7f450c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 --2- 172.21.15.69:0/1597424145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f450c004800 0x7f450c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 --2- 172.21.15.69:0/1597424145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f450c0051c0 0x7f450c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f45280cebf0 con 0x7f450c0039b0 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f450c00c0f0 con 0x7f450c0051c0 2024-04-24T08:08:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f450c00c2f0 con 0x7f450c004800 2024-04-24T08:08:31.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.515+0000 7f4527e00640 1 --2- 172.21.15.69:0/1597424145 >> v2:172.21.15.121:3300/0 conn(0x7f45280d09f0 0x7f4528166020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:31.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.516+0000 7f4527400640 1 --2- 172.21.15.69:0/1597424145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0039b0 0x7f450c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.516+0000 7f4527400640 1 -- 172.21.15.69:0/1597424145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f450c004800 msgr2=0x7f450c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:31.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.516+0000 7f4527400640 1 --2- 172.21.15.69:0/1597424145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f450c004800 0x7f450c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.516+0000 7f4527400640 1 -- 172.21.15.69:0/1597424145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f450c0051c0 msgr2=0x7f450c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:31.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.516+0000 7f4527400640 1 --2- 172.21.15.69:0/1597424145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f450c0051c0 0x7f450c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.516+0000 7f4527400640 1 -- 172.21.15.69:0/1597424145 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450c003520 con 0x7f450c0039b0 2024-04-24T08:08:31.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.516+0000 7f4527e00640 1 -- 172.21.15.69:0/1597424145 reap_dead start 2024-04-24T08:08:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.517+0000 7f4527400640 1 --2- 172.21.15.69:0/1597424145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0039b0 0x7f450c003db0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f451800c210 tx=0x7f450c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d4d64a5b307331 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.517+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518007d10 con 0x7f450c0039b0 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.517+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45180058d0 con 0x7f450c0039b0 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.517+0000 7f4526a00640 1 -- 172.21.15.69:0/1597424145 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518005d20 con 0x7f450c0039b0 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.517+0000 7f452d000640 1 -- 172.21.15.69:0/1597424145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0039b0 msgr2=0x7f450c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.517+0000 7f452d000640 1 --2- 172.21.15.69:0/1597424145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0039b0 0x7f450c003db0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f451800c210 tx=0x7f450c005830 comp rx=0 tx=0).stop 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.518+0000 7f452d000640 1 -- 172.21.15.69:0/1597424145 shutdown_connections 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.518+0000 7f452d000640 1 --2- 172.21.15.69:0/1597424145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f450c0039b0 0x7f450c003db0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.518+0000 7f452d000640 1 -- 172.21.15.69:0/1597424145 >> 172.21.15.69:0/1597424145 conn(0x7f4528067b80 msgr2=0x7f452806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.518+0000 7f452d000640 1 -- 172.21.15.69:0/1597424145 shutdown_connections 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.518+0000 7f452d000640 1 -- 172.21.15.69:0/1597424145 wait complete. 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.519+0000 7f452d000640 1 Processor -- start 2024-04-24T08:08:31.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.519+0000 7f452d000640 1 -- start start 2024-04-24T08:08:31.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.520+0000 7f452d000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f450c0039b0 0x7f452814fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.520+0000 7f452d000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45281457e0 0x7f452814d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.520+0000 7f452d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f452814dae0 0x7f4528144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.520+0000 7f452d000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f45280d02e0 con 0x7f452814dae0 2024-04-24T08:08:31.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.520+0000 7f452d000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f45280cf490 con 0x7f45281457e0 2024-04-24T08:08:31.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.520+0000 7f452d000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f45280d0460 con 0x7f450c0039b0 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f4527e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f452814dae0 0x7f4528144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f4527e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f452814dae0 0x7f4528144ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44490/0 (socket says 172.21.15.69:44490) 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f4527e00640 1 -- 172.21.15.69:0/2680929558 learned_addr learned my addr 172.21.15.69:0/2680929558 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f4527400640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f450c0039b0 0x7f452814fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f4527e00640 1 -- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f450c0039b0 msgr2=0x7f452814fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f451ea00640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45281457e0 0x7f452814d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f4527e00640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f450c0039b0 0x7f452814fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f4527e00640 1 -- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45281457e0 msgr2=0x7f452814d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f4527e00640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45281457e0 0x7f452814d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f4527e00640 1 -- 172.21.15.69:0/2680929558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452814aac0 con 0x7f452814dae0 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f4527e00640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f452814dae0 0x7f4528144ba0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f452806c690 tx=0x7f452000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f451fe00640 1 -- 172.21.15.69:0/2680929558 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f452001a070 con 0x7f452814dae0 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f451fe00640 1 -- 172.21.15.69:0/2680929558 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4520007e20 con 0x7f452814dae0 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f452d000640 1 -- 172.21.15.69:0/2680929558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f452814e0a0 con 0x7f452814dae0 2024-04-24T08:08:31.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.521+0000 7f451fe00640 1 -- 172.21.15.69:0/2680929558 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4520003bd0 con 0x7f452814dae0 2024-04-24T08:08:31.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.522+0000 7f452d000640 1 -- 172.21.15.69:0/2680929558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f452814b040 con 0x7f452814dae0 2024-04-24T08:08:31.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.522+0000 7f452d000640 1 -- 172.21.15.69:0/2680929558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4528075d50 con 0x7f452814dae0 2024-04-24T08:08:31.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.523+0000 7f451fe00640 1 -- 172.21.15.69:0/2680929558 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f452001d070 con 0x7f452814dae0 2024-04-24T08:08:31.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.524+0000 7f451fe00640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f450406c8f0 0x7f450406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.524+0000 7f451fe00640 1 -- 172.21.15.69:0/2680929558 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f452008b940 con 0x7f452814dae0 2024-04-24T08:08:31.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.525+0000 7f4527400640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f450406c8f0 0x7f450406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.526+0000 7f4527400640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f450406c8f0 0x7f450406edb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4518033000 tx=0x7f4518009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:31.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.528+0000 7f451fe00640 1 -- 172.21.15.69:0/2680929558 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4520010070 con 0x7f452814dae0 2024-04-24T08:08:31.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.729+0000 7f452d000640 1 -- 172.21.15.69:0/2680929558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4528149050 con 0x7f452814dae0 2024-04-24T08:08:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.729+0000 7f451fe00640 1 -- 172.21.15.69:0/2680929558 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f4520059d60 con 0x7f452814dae0 2024-04-24T08:08:31.731 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:08:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 -- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f450406c8f0 msgr2=0x7f450406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f450406c8f0 0x7f450406edb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4518033000 tx=0x7f4518009030 comp rx=0 tx=0).stop 2024-04-24T08:08:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 -- 172.21.15.69:0/2680929558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f452814dae0 msgr2=0x7f4528144ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f452814dae0 0x7f4528144ba0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f452806c690 tx=0x7f452000d4a0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 -- 172.21.15.69:0/2680929558 shutdown_connections 2024-04-24T08:08:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f450406c8f0 0x7f450406edb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f452814dae0 0x7f4528144ba0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f45281457e0 0x7f452814d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 --2- 172.21.15.69:0/2680929558 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f450c0039b0 0x7f452814fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 -- 172.21.15.69:0/2680929558 >> 172.21.15.69:0/2680929558 conn(0x7f4528067b80 msgr2=0x7f45280b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 -- 172.21.15.69:0/2680929558 shutdown_connections 2024-04-24T08:08:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.732+0000 7f452d000640 1 -- 172.21.15.69:0/2680929558 wait complete. 2024-04-24T08:08:31.751 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:08:31.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.950+0000 7fb0b7c00640 1 Processor -- start 2024-04-24T08:08:31.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.951+0000 7fb0b7c00640 1 -- start start 2024-04-24T08:08:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.952+0000 7fb0b7c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b015eac0 0x7fb0b00b4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.952+0000 7fb0b7c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb0b00b4ec0 0x7fb0b00d46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.952+0000 7fb0b7c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0b00d4c30 0x7fb0b00d2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.952+0000 7fb0b7c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b00d50a0 con 0x7fb0b00d4c30 2024-04-24T08:08:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.952+0000 7fb0b7c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b00d2540 con 0x7fb0b00b4ec0 2024-04-24T08:08:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.952+0000 7fb0b7c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b00d0370 con 0x7fb0b015eac0 2024-04-24T08:08:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.952+0000 7fb0b5c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0b00d4c30 0x7fb0b00d2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.953+0000 7fb0b5c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb0b00d4c30 0x7fb0b00d2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44504/0 (socket says 172.21.15.69:44504) 2024-04-24T08:08:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.953+0000 7fb0b5c00640 1 -- 172.21.15.69:0/706903946 learned_addr learned my addr 172.21.15.69:0/706903946 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:31.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.953+0000 7fb0b5200640 1 --2- 172.21.15.69:0/706903946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b015eac0 0x7fb0b00b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.953+0000 7fb0afe00640 1 --2- 172.21.15.69:0/706903946 >> v2:172.21.15.121:3300/0 conn(0x7fb0b00b4ec0 0x7fb0b00d46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.953+0000 7fb0b5c00640 1 -- 172.21.15.69:0/706903946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b015eac0 msgr2=0x7fb0b00b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.953+0000 7fb0b5c00640 1 --2- 172.21.15.69:0/706903946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b015eac0 0x7fb0b00b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.953+0000 7fb0b5c00640 1 -- 172.21.15.69:0/706903946 >> v2:172.21.15.121:3300/0 conn(0x7fb0b00b4ec0 msgr2=0x7fb0b00d46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.953+0000 7fb0b5c00640 1 --2- 172.21.15.69:0/706903946 >> v2:172.21.15.121:3300/0 conn(0x7fb0b00b4ec0 0x7fb0b00d46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.953+0000 7fb0b5c00640 1 -- 172.21.15.69:0/706903946 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b00d0650 con 0x7fb0b00d4c30 2024-04-24T08:08:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.954+0000 7fb0b5c00640 1 --2- 172.21.15.69:0/706903946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b00d4c30 0x7fb0b00d2f90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00b7350 tx=0x7fb0a802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a9e175517ab4c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.954+0000 7fb0af400640 1 -- 172.21.15.69:0/706903946 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8037070 con 0x7fb0b00d4c30 2024-04-24T08:08:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.954+0000 7fb0af400640 1 -- 172.21.15.69:0/706903946 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a802ae00 con 0x7fb0b00d4c30 2024-04-24T08:08:31.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.954+0000 7fb0af400640 1 -- 172.21.15.69:0/706903946 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8032660 con 0x7fb0b00d4c30 2024-04-24T08:08:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.955+0000 7fb0b7c00640 1 -- 172.21.15.69:0/706903946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b00d4c30 msgr2=0x7fb0b00d2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.955+0000 7fb0b7c00640 1 --2- 172.21.15.69:0/706903946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b00d4c30 0x7fb0b00d2f90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00b7350 tx=0x7fb0a802a0b0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.955+0000 7fb0b7c00640 1 -- 172.21.15.69:0/706903946 shutdown_connections 2024-04-24T08:08:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.955+0000 7fb0b7c00640 1 --2- 172.21.15.69:0/706903946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b00d4c30 0x7fb0b00d2f90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.955+0000 7fb0b7c00640 1 --2- 172.21.15.69:0/706903946 >> v2:172.21.15.121:3300/0 conn(0x7fb0b00b4ec0 0x7fb0b00d46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.955+0000 7fb0b7c00640 1 --2- 172.21.15.69:0/706903946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b015eac0 0x7fb0b00b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.955+0000 7fb0b7c00640 1 -- 172.21.15.69:0/706903946 >> 172.21.15.69:0/706903946 conn(0x7fb0b008e150 msgr2=0x7fb0b009da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.955+0000 7fb0b7c00640 1 -- 172.21.15.69:0/706903946 shutdown_connections 2024-04-24T08:08:31.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.955+0000 7fb0b7c00640 1 -- 172.21.15.69:0/706903946 wait complete. 2024-04-24T08:08:31.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.957+0000 7fb0b7c00640 1 Processor -- start 2024-04-24T08:08:31.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.957+0000 7fb0b7c00640 1 -- start start 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.958+0000 7fb0b7c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0b00b4ec0 0x7fb0b00c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.958+0000 7fb0b7c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b00d4c30 0x7fb0b00c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.958+0000 7fb0b7c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b015eac0 0x7fb0b00ce240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.958+0000 7fb0b7c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b015eea0 con 0x7fb0b015eac0 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.958+0000 7fb0b7c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b00d2a70 con 0x7fb0b00b4ec0 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.958+0000 7fb0b7c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b00c89f0 con 0x7fb0b00d4c30 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.958+0000 7fb0b5c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b015eac0 0x7fb0b00ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.958+0000 7fb0b5c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b015eac0 0x7fb0b00ce240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44508/0 (socket says 172.21.15.69:44508) 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.958+0000 7fb0b5c00640 1 -- 172.21.15.69:0/118973136 learned_addr learned my addr 172.21.15.69:0/118973136 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0b5200640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0b00b4ec0 0x7fb0b00c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0afe00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b00d4c30 0x7fb0b00c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0b5c00640 1 -- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b00d4c30 msgr2=0x7fb0b00c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0b5c00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b00d4c30 0x7fb0b00c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0b5c00640 1 -- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0b00b4ec0 msgr2=0x7fb0b00c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0b5c00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0b00b4ec0 0x7fb0b00c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0b5c00640 1 -- 172.21.15.69:0/118973136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b00cde20 con 0x7fb0b015eac0 2024-04-24T08:08:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0afe00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b00d4c30 0x7fb0b00c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0b5200640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0b00b4ec0 0x7fb0b00c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0b5c00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b015eac0 0x7fb0b00ce240 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb0a802a0f0 tx=0x7fb0a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0acc00640 1 -- 172.21.15.69:0/118973136 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8041030 con 0x7fb0b015eac0 2024-04-24T08:08:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0acc00640 1 -- 172.21.15.69:0/118973136 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a802ac00 con 0x7fb0b015eac0 2024-04-24T08:08:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0acc00640 1 -- 172.21.15.69:0/118973136 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8037040 con 0x7fb0b015eac0 2024-04-24T08:08:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0b7c00640 1 -- 172.21.15.69:0/118973136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b00cdb40 con 0x7fb0b015eac0 2024-04-24T08:08:31.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.959+0000 7fb0b7c00640 1 -- 172.21.15.69:0/118973136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b00c8680 con 0x7fb0b015eac0 2024-04-24T08:08:31.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.960+0000 7fb0b7c00640 1 -- 172.21.15.69:0/118973136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b009da70 con 0x7fb0b015eac0 2024-04-24T08:08:31.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.961+0000 7fb0acc00640 1 -- 172.21.15.69:0/118973136 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb0a8044070 con 0x7fb0b015eac0 2024-04-24T08:08:31.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.962+0000 7fb0acc00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb08c06c8f0 0x7fb08c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:31.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.963+0000 7fb0acc00640 1 -- 172.21.15.69:0/118973136 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb0a80abef0 con 0x7fb0b015eac0 2024-04-24T08:08:31.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.963+0000 7fb0b5200640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb08c06c8f0 0x7fb08c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:31.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.964+0000 7fb0b5200640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb08c06c8f0 0x7fb08c06edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0007520 tx=0x7fb0a000b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:31.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:31.967+0000 7fb0acc00640 1 -- 172.21.15.69:0/118973136 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb0b009da70 con 0x7fb0b015eac0 2024-04-24T08:08:32.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.165+0000 7fb0b7c00640 1 -- 172.21.15.69:0/118973136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb0b0068a80 con 0x7fb0b015eac0 2024-04-24T08:08:32.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.165+0000 7fb0acc00640 1 -- 172.21.15.69:0/118973136 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7fb0a804b070 con 0x7fb0b015eac0 2024-04-24T08:08:32.167 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:08:32.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 -- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb08c06c8f0 msgr2=0x7fb08c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb08c06c8f0 0x7fb08c06edb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0007520 tx=0x7fb0a000b040 comp rx=0 tx=0).stop 2024-04-24T08:08:32.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 -- 172.21.15.69:0/118973136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b015eac0 msgr2=0x7fb0b00ce240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b015eac0 0x7fb0b00ce240 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb0a802a0f0 tx=0x7fb0a80046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 -- 172.21.15.69:0/118973136 shutdown_connections 2024-04-24T08:08:32.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb08c06c8f0 0x7fb08c06edb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0b015eac0 0x7fb0b00ce240 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0b00d4c30 0x7fb0b00c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 --2- 172.21.15.69:0/118973136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0b00b4ec0 0x7fb0b00c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 -- 172.21.15.69:0/118973136 >> 172.21.15.69:0/118973136 conn(0x7fb0b008e150 msgr2=0x7fb0b0161230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:32.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.168+0000 7fb0b7c00640 1 -- 172.21.15.69:0/118973136 shutdown_connections 2024-04-24T08:08:32.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.169+0000 7fb0b7c00640 1 -- 172.21.15.69:0/118973136 wait complete. 2024-04-24T08:08:32.187 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:08:32.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.386+0000 7fed88a00640 1 Processor -- start 2024-04-24T08:08:32.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.387+0000 7fed88a00640 1 -- start start 2024-04-24T08:08:32.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.388+0000 7fed88a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fed841420a0 0x7fed84071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.388+0000 7fed88a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fed84142480 0x7fed8415ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.388+0000 7fed88a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed84160920 con 0x7fed84160510 2024-04-24T08:08:32.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.388+0000 7fed88a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fed840cf1e0 con 0x7fed84142480 2024-04-24T08:08:32.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.388+0000 7fed88a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fed84072450 con 0x7fed841420a0 2024-04-24T08:08:32.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.389+0000 7fed83200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fed84160510 0x7fed8415c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36630/0 (socket says 172.21.15.69:36630) 2024-04-24T08:08:32.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.389+0000 7fed83200640 1 -- 172.21.15.69:0/2012306854 learned_addr learned my addr 172.21.15.69:0/2012306854 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:32.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.389+0000 7fed7be00640 1 --2- 172.21.15.69:0/2012306854 >> v2:172.21.15.69:3300/0 conn(0x7fed84142480 0x7fed8415ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.389+0000 7fed82800640 1 --2- 172.21.15.69:0/2012306854 >> v2:172.21.15.121:3300/0 conn(0x7fed841420a0 0x7fed84071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.389+0000 7fed81e00640 1 -- 172.21.15.69:0/2012306854 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4131548369 0 0) 0x7fed7c001490 con 0x7fed84160510 2024-04-24T08:08:32.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.389+0000 7fed7be00640 1 -- 172.21.15.69:0/2012306854 >> v2:172.21.15.121:3300/0 conn(0x7fed841420a0 msgr2=0x7fed84071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:32.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.389+0000 7fed7be00640 1 --2- 172.21.15.69:0/2012306854 >> v2:172.21.15.121:3300/0 conn(0x7fed841420a0 0x7fed84071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.389+0000 7fed7be00640 1 -- 172.21.15.69:0/2012306854 >> v1:172.21.15.69:6789/0 conn(0x7fed84160510 legacy=0x7fed8415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.389+0000 7fed7be00640 1 -- 172.21.15.69:0/2012306854 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed84072730 con 0x7fed84142480 2024-04-24T08:08:32.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.390+0000 7fed7be00640 1 --2- 172.21.15.69:0/2012306854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 0x7fed8415ffd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fed840b5770 tx=0x7fed6c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21d71a34dacebc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:32.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.390+0000 7fed81e00640 1 -- 172.21.15.69:0/2012306854 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c037070 con 0x7fed84142480 2024-04-24T08:08:32.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.390+0000 7fed81e00640 1 -- 172.21.15.69:0/2012306854 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed6c032440 con 0x7fed84142480 2024-04-24T08:08:32.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.390+0000 7fed81e00640 1 -- 172.21.15.69:0/2012306854 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c032810 con 0x7fed84142480 2024-04-24T08:08:32.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.391+0000 7fed88a00640 1 -- 172.21.15.69:0/2012306854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 msgr2=0x7fed8415ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.391+0000 7fed88a00640 1 --2- 172.21.15.69:0/2012306854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 0x7fed8415ffd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fed840b5770 tx=0x7fed6c02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.391+0000 7fed88a00640 1 -- 172.21.15.69:0/2012306854 shutdown_connections 2024-04-24T08:08:32.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.391+0000 7fed88a00640 1 --2- 172.21.15.69:0/2012306854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 0x7fed8415ffd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.391+0000 7fed88a00640 1 --2- 172.21.15.69:0/2012306854 >> v2:172.21.15.121:3300/0 conn(0x7fed841420a0 0x7fed84071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.391+0000 7fed88a00640 1 -- 172.21.15.69:0/2012306854 >> 172.21.15.69:0/2012306854 conn(0x7fed84067b80 msgr2=0x7fed8406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:32.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.391+0000 7fed88a00640 1 -- 172.21.15.69:0/2012306854 shutdown_connections 2024-04-24T08:08:32.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.391+0000 7fed88a00640 1 -- 172.21.15.69:0/2012306854 wait complete. 2024-04-24T08:08:32.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.393+0000 7fed88a00640 1 Processor -- start 2024-04-24T08:08:32.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.393+0000 7fed88a00640 1 -- start start 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.394+0000 7fed88a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed841420a0 0x7fed8406f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.394+0000 7fed88a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 0x7fed84140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.394+0000 7fed88a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed84160510 0x7fed841f73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.394+0000 7fed88a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed84142910 con 0x7fed84142480 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.394+0000 7fed88a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fed84072cf0 con 0x7fed84160510 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.394+0000 7fed88a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed840d0680 con 0x7fed841420a0 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.394+0000 7fed7be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 0x7fed84140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.394+0000 7fed7be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 0x7fed84140660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44520/0 (socket says 172.21.15.69:44520) 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.394+0000 7fed7be00640 1 -- 172.21.15.69:0/3936018917 learned_addr learned my addr 172.21.15.69:0/3936018917 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed83200640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed84160510 0x7fed841f73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed7be00640 1 -- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed841420a0 msgr2=0x7fed8406f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed7be00640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed841420a0 0x7fed8406f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed7be00640 1 -- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed84160510 msgr2=0x7fed841f73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed7be00640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed84160510 0x7fed841f73a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed7be00640 1 -- 172.21.15.69:0/3936018917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed841f78e0 con 0x7fed84142480 2024-04-24T08:08:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed83200640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed84160510 0x7fed841f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed7be00640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 0x7fed84140660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fed6c000c00 tx=0x7fed6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed7aa00640 1 -- 172.21.15.69:0/3936018917 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c037040 con 0x7fed84142480 2024-04-24T08:08:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed7aa00640 1 -- 172.21.15.69:0/3936018917 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed6c042070 con 0x7fed84142480 2024-04-24T08:08:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.395+0000 7fed88a00640 1 -- 172.21.15.69:0/3936018917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed84140dd0 con 0x7fed84142480 2024-04-24T08:08:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.396+0000 7fed7aa00640 1 -- 172.21.15.69:0/3936018917 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c030bd0 con 0x7fed84142480 2024-04-24T08:08:32.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.396+0000 7fed88a00640 1 -- 172.21.15.69:0/3936018917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed84141390 con 0x7fed84142480 2024-04-24T08:08:32.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.397+0000 7fed88a00640 1 -- 172.21.15.69:0/3936018917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed84075d50 con 0x7fed84142480 2024-04-24T08:08:32.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.398+0000 7fed7aa00640 1 -- 172.21.15.69:0/3936018917 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fed6c047460 con 0x7fed84142480 2024-04-24T08:08:32.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.399+0000 7fed7aa00640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed6006c8f0 0x7fed6006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.399+0000 7fed7aa00640 1 -- 172.21.15.69:0/3936018917 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fed6c0ac730 con 0x7fed84142480 2024-04-24T08:08:32.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.400+0000 7fed82800640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed6006c8f0 0x7fed6006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.401+0000 7fed82800640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed6006c8f0 0x7fed6006edb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fed740098f0 tx=0x7fed74008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:32.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.404+0000 7fed7aa00640 1 -- 172.21.15.69:0/3936018917 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fed6c032e20 con 0x7fed84142480 2024-04-24T08:08:32.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.605+0000 7fed88a00640 1 -- 172.21.15.69:0/3936018917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fed8406c470 con 0x7fed84142480 2024-04-24T08:08:32.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.606+0000 7fed7aa00640 1 -- 172.21.15.69:0/3936018917 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fed6c07ab50 con 0x7fed84142480 2024-04-24T08:08:32.607 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:08:32.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.608+0000 7fed88a00640 1 -- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed6006c8f0 msgr2=0x7fed6006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.608+0000 7fed88a00640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed6006c8f0 0x7fed6006edb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fed740098f0 tx=0x7fed74008040 comp rx=0 tx=0).stop 2024-04-24T08:08:32.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.608+0000 7fed88a00640 1 -- 172.21.15.69:0/3936018917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 msgr2=0x7fed84140660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.608+0000 7fed88a00640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 0x7fed84140660 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fed6c000c00 tx=0x7fed6c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.608+0000 7fed88a00640 1 -- 172.21.15.69:0/3936018917 shutdown_connections 2024-04-24T08:08:32.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.608+0000 7fed88a00640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed6006c8f0 0x7fed6006edb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.609+0000 7fed88a00640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed84142480 0x7fed84140660 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.609+0000 7fed88a00640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed84160510 0x7fed841f73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.609+0000 7fed88a00640 1 --2- 172.21.15.69:0/3936018917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed841420a0 0x7fed8406f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.609+0000 7fed88a00640 1 -- 172.21.15.69:0/3936018917 >> 172.21.15.69:0/3936018917 conn(0x7fed84067b80 msgr2=0x7fed841530e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:32.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.609+0000 7fed88a00640 1 -- 172.21.15.69:0/3936018917 shutdown_connections 2024-04-24T08:08:32.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.609+0000 7fed88a00640 1 -- 172.21.15.69:0/3936018917 wait complete. 2024-04-24T08:08:32.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:08:32.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.828+0000 7fc507200640 1 Processor -- start 2024-04-24T08:08:32.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.828+0000 7fc507200640 1 -- start start 2024-04-24T08:08:32.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.829+0000 7fc507200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5081523f0 0x7fc50806db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.829+0000 7fc507200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc508072e20 0x7fc508073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.829+0000 7fc507200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc5080737a0 0x7fc50815ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.829+0000 7fc507200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc5080aeac0 con 0x7fc5080737a0 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.829+0000 7fc507200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc5080aeda0 con 0x7fc508072e20 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.829+0000 7fc507200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc5080af080 con 0x7fc5081523f0 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.830+0000 7fc506800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc5080737a0 0x7fc50815ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.830+0000 7fc506800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc5080737a0 0x7fc50815ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44536/0 (socket says 172.21.15.69:44536) 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.830+0000 7fc506800640 1 -- 172.21.15.69:0/3975658616 learned_addr learned my addr 172.21.15.69:0/3975658616 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.830+0000 7fc4fd400640 1 --2- 172.21.15.69:0/3975658616 >> v2:172.21.15.121:3300/0 conn(0x7fc508072e20 0x7fc508073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.830+0000 7fc505e00640 1 --2- 172.21.15.69:0/3975658616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5081523f0 0x7fc50806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.830+0000 7fc506800640 1 -- 172.21.15.69:0/3975658616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5081523f0 msgr2=0x7fc50806db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.830+0000 7fc506800640 1 --2- 172.21.15.69:0/3975658616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5081523f0 0x7fc50806db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.830+0000 7fc506800640 1 -- 172.21.15.69:0/3975658616 >> v2:172.21.15.121:3300/0 conn(0x7fc508072e20 msgr2=0x7fc508073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.830+0000 7fc506800640 1 --2- 172.21.15.69:0/3975658616 >> v2:172.21.15.121:3300/0 conn(0x7fc508072e20 0x7fc508073260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.830+0000 7fc506800640 1 -- 172.21.15.69:0/3975658616 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5080af360 con 0x7fc5080737a0 2024-04-24T08:08:32.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.831+0000 7fc506800640 1 --2- 172.21.15.69:0/3975658616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5080737a0 0x7fc50815ef70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc50806d930 tx=0x7fc50002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2772326456cc68aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:32.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc505400640 1 -- 172.21.15.69:0/3975658616 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc500036070 con 0x7fc5080737a0 2024-04-24T08:08:32.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc505400640 1 -- 172.21.15.69:0/3975658616 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc500031600 con 0x7fc5080737a0 2024-04-24T08:08:32.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc505400640 1 -- 172.21.15.69:0/3975658616 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5000319d0 con 0x7fc5080737a0 2024-04-24T08:08:32.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc507200640 1 -- 172.21.15.69:0/3975658616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5080737a0 msgr2=0x7fc50815ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc507200640 1 --2- 172.21.15.69:0/3975658616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5080737a0 0x7fc50815ef70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc50806d930 tx=0x7fc50002a610 comp rx=0 tx=0).stop 2024-04-24T08:08:32.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc507200640 1 -- 172.21.15.69:0/3975658616 shutdown_connections 2024-04-24T08:08:32.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc507200640 1 --2- 172.21.15.69:0/3975658616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5080737a0 0x7fc50815ef70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc507200640 1 --2- 172.21.15.69:0/3975658616 >> v2:172.21.15.121:3300/0 conn(0x7fc508072e20 0x7fc508073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc507200640 1 --2- 172.21.15.69:0/3975658616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5081523f0 0x7fc50806db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc507200640 1 -- 172.21.15.69:0/3975658616 >> 172.21.15.69:0/3975658616 conn(0x7fc5080d0160 msgr2=0x7fc5080d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:32.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc507200640 1 -- 172.21.15.69:0/3975658616 shutdown_connections 2024-04-24T08:08:32.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.832+0000 7fc507200640 1 -- 172.21.15.69:0/3975658616 wait complete. 2024-04-24T08:08:32.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.834+0000 7fc507200640 1 Processor -- start 2024-04-24T08:08:32.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.834+0000 7fc507200640 1 -- start start 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.835+0000 7fc507200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc508072e20 0x7fc508143ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.835+0000 7fc507200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5080737a0 0x7fc50814abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.835+0000 7fc507200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5081523f0 0x7fc50809b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.835+0000 7fc507200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc50815f4b0 con 0x7fc508072e20 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.835+0000 7fc507200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc5081527d0 con 0x7fc5080737a0 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.835+0000 7fc507200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc50815f630 con 0x7fc5081523f0 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.835+0000 7fc505e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc508072e20 0x7fc508143ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.835+0000 7fc505e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc508072e20 0x7fc508143ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44546/0 (socket says 172.21.15.69:44546) 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.835+0000 7fc505e00640 1 -- 172.21.15.69:0/2605593277 learned_addr learned my addr 172.21.15.69:0/2605593277 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc4fd400640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5080737a0 0x7fc50814abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc506800640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5081523f0 0x7fc50809b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc505e00640 1 -- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5081523f0 msgr2=0x7fc50809b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc505e00640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5081523f0 0x7fc50809b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc505e00640 1 -- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5080737a0 msgr2=0x7fc50814abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:32.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc505e00640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5080737a0 0x7fc50814abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:32.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc505e00640 1 -- 172.21.15.69:0/2605593277 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc50809e200 con 0x7fc508072e20 2024-04-24T08:08:32.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc4fd400640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5080737a0 0x7fc50814abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:32.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc505e00640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc508072e20 0x7fc508143ce0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800c7f0 tx=0x7fc4f800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:32.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc4fea00640 1 -- 172.21.15.69:0/2605593277 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8015070 con 0x7fc508072e20 2024-04-24T08:08:32.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc4fea00640 1 -- 172.21.15.69:0/2605593277 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f8004220 con 0x7fc508072e20 2024-04-24T08:08:32.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc4fea00640 1 -- 172.21.15.69:0/2605593277 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f8007620 con 0x7fc508072e20 2024-04-24T08:08:32.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc507200640 1 -- 172.21.15.69:0/2605593277 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc50809dc40 con 0x7fc508072e20 2024-04-24T08:08:32.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.836+0000 7fc507200640 1 -- 172.21.15.69:0/2605593277 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc50809ad00 con 0x7fc508072e20 2024-04-24T08:08:32.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.838+0000 7fc507200640 1 -- 172.21.15.69:0/2605593277 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc508075d50 con 0x7fc508072e20 2024-04-24T08:08:32.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.838+0000 7fc4fea00640 1 -- 172.21.15.69:0/2605593277 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc4f80079d0 con 0x7fc508072e20 2024-04-24T08:08:32.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.839+0000 7fc4fea00640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc4e406c8f0 0x7fc4e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:32.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.839+0000 7fc4fea00640 1 -- 172.21.15.69:0/2605593277 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc4f808bc50 con 0x7fc508072e20 2024-04-24T08:08:32.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.840+0000 7fc4fd400640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc4e406c8f0 0x7fc4e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:32.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.841+0000 7fc4fd400640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc4e406c8f0 0x7fc4e406edb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc5080b4030 tx=0x7fc4f0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:32.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:32.844+0000 7fc4fea00640 1 -- 172.21.15.69:0/2605593277 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc4f8059fe0 con 0x7fc508072e20 2024-04-24T08:08:33.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.082+0000 7fc507200640 1 -- 172.21.15.69:0/2605593277 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc508151c10 con 0x7fc508072e20 2024-04-24T08:08:33.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.083+0000 7fc4fea00640 1 -- 172.21.15.69:0/2605593277 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7fc4f805ee20 con 0x7fc508072e20 2024-04-24T08:08:33.085 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:33.085 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/934340733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":934340733},{"type":"v1","addr":"172.21.15.121:6835","nonce":934340733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3140867113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3140867113},{"type":"v1","addr":"172.21.15.121:6837","nonce":3140867113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/1059995973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1059995973},{"type":"v1","addr":"172.21.15.69:6833","nonce":1059995973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:08:19.574083+0000","modified":"2024-04-24T08:08:22.605668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/1003206529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1003206529},{"type":"v1","addr":"172.21.15.69:6835","nonce":1003206529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-24T08:08:33.085 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 7 2024-04-24T08:08:33.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 -- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc4e406c8f0 msgr2=0x7fc4e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc4e406c8f0 0x7fc4e406edb0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc5080b4030 tx=0x7fc4f0008040 comp rx=0 tx=0).stop 2024-04-24T08:08:33.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 -- 172.21.15.69:0/2605593277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc508072e20 msgr2=0x7fc508143ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc508072e20 0x7fc508143ce0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc4f800c7f0 tx=0x7fc4f800ccb0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 -- 172.21.15.69:0/2605593277 shutdown_connections 2024-04-24T08:08:33.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc4e406c8f0 0x7fc4e406edb0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5081523f0 0x7fc50809b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5080737a0 0x7fc50814abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 --2- 172.21.15.69:0/2605593277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc508072e20 0x7fc508143ce0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 -- 172.21.15.69:0/2605593277 >> 172.21.15.69:0/2605593277 conn(0x7fc5080d0160 msgr2=0x7fc5080addf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:33.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 -- 172.21.15.69:0/2605593277 shutdown_connections 2024-04-24T08:08:33.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.086+0000 7fc507200640 1 -- 172.21.15.69:0/2605593277 wait complete. 2024-04-24T08:08:33.107 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:33.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.308+0000 7f77fda00640 1 Processor -- start 2024-04-24T08:08:33.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.308+0000 7f77fda00640 1 -- start start 2024-04-24T08:08:33.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.310+0000 7f77fda00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f77f8156c60 0x7f77f815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.310+0000 7f77fda00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f77f80b30f0 0x7f77f815d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.310+0000 7f77fda00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77f815d660 con 0x7f77f80ad0d0 2024-04-24T08:08:33.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.310+0000 7f77fda00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f77f806dc30 con 0x7f77f80b30f0 2024-04-24T08:08:33.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.310+0000 7f77fda00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f77f806df10 con 0x7f77f8156c60 2024-04-24T08:08:33.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.310+0000 7f77f7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f77f80b30f0 0x7f77f815d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.310+0000 7f77f7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f77f80b30f0 0x7f77f815d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44548/0 (socket says 172.21.15.69:44548) 2024-04-24T08:08:33.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.310+0000 7f77f7400640 1 -- 172.21.15.69:0/634699701 learned_addr learned my addr 172.21.15.69:0/634699701 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:33.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.310+0000 7f77f7e00640 1 --2- 172.21.15.69:0/634699701 >> v2:172.21.15.121:3300/0 conn(0x7f77f8156c60 0x7f77f815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.311+0000 7f77f7400640 1 -- 172.21.15.69:0/634699701 >> v2:172.21.15.121:3300/0 conn(0x7f77f8156c60 msgr2=0x7f77f815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:33.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.311+0000 7f77f7400640 1 --2- 172.21.15.69:0/634699701 >> v2:172.21.15.121:3300/0 conn(0x7f77f8156c60 0x7f77f815cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.311+0000 7f77f7400640 1 -- 172.21.15.69:0/634699701 >> v1:172.21.15.121:6789/0 conn(0x7f77f80ad0d0 legacy=0x7f77f81600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:33.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.311+0000 7f77f7400640 1 -- 172.21.15.69:0/634699701 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f806e1f0 con 0x7f77f80b30f0 2024-04-24T08:08:33.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.312+0000 7f77f7400640 1 --2- 172.21.15.69:0/634699701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80b30f0 0x7f77f815d000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f77f80b5330 tx=0x7f77e402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610735963c82eb21 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:33.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.312+0000 7f77f6a00640 1 -- 172.21.15.69:0/634699701 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e4037070 con 0x7f77f80b30f0 2024-04-24T08:08:33.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.312+0000 7f77f6a00640 1 -- 172.21.15.69:0/634699701 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77e4032440 con 0x7f77f80b30f0 2024-04-24T08:08:33.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.312+0000 7f77f6a00640 1 -- 172.21.15.69:0/634699701 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e4032810 con 0x7f77f80b30f0 2024-04-24T08:08:33.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.313+0000 7f77fda00640 1 -- 172.21.15.69:0/634699701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80b30f0 msgr2=0x7f77f815d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.313+0000 7f77fda00640 1 --2- 172.21.15.69:0/634699701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80b30f0 0x7f77f815d000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f77f80b5330 tx=0x7f77e402a230 comp rx=0 tx=0).stop 2024-04-24T08:08:33.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.313+0000 7f77fda00640 1 -- 172.21.15.69:0/634699701 shutdown_connections 2024-04-24T08:08:33.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.313+0000 7f77fda00640 1 --2- 172.21.15.69:0/634699701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80b30f0 0x7f77f815d000 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.313+0000 7f77fda00640 1 --2- 172.21.15.69:0/634699701 >> v2:172.21.15.121:3300/0 conn(0x7f77f8156c60 0x7f77f815cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.313+0000 7f77fda00640 1 -- 172.21.15.69:0/634699701 >> 172.21.15.69:0/634699701 conn(0x7f77f80d0160 msgr2=0x7f77f80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:33.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.313+0000 7f77fda00640 1 -- 172.21.15.69:0/634699701 shutdown_connections 2024-04-24T08:08:33.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.313+0000 7f77fda00640 1 -- 172.21.15.69:0/634699701 wait complete. 2024-04-24T08:08:33.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.315+0000 7f77fda00640 1 Processor -- start 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.315+0000 7f77fda00640 1 -- start start 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.316+0000 7f77fda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80ad0d0 0x7f77f8157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.316+0000 7f77fda00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77f80b30f0 0x7f77f81f7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.316+0000 7f77fda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77f8156c60 0x7f77f81595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.316+0000 7f77fda00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f77f80ad540 con 0x7f77f80ad0d0 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.316+0000 7f77fda00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f77f815d940 con 0x7f77f80b30f0 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.316+0000 7f77fda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f77f80ad6c0 con 0x7f77f8156c60 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.317+0000 7f77f7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80ad0d0 0x7f77f8157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.317+0000 7f77f7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80ad0d0 0x7f77f8157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44558/0 (socket says 172.21.15.69:44558) 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.317+0000 7f77f7e00640 1 -- 172.21.15.69:0/3928517467 learned_addr learned my addr 172.21.15.69:0/3928517467 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.317+0000 7f77f7e00640 1 -- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77f8156c60 msgr2=0x7f77f81595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.317+0000 7f77f7e00640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77f8156c60 0x7f77f81595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.317+0000 7f77f7e00640 1 -- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77f80b30f0 msgr2=0x7f77f81f7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.317+0000 7f77f7e00640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77f80b30f0 0x7f77f81f7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.317+0000 7f77f7e00640 1 -- 172.21.15.69:0/3928517467 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f8072090 con 0x7f77f80ad0d0 2024-04-24T08:08:33.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.317+0000 7f77f7e00640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80ad0d0 0x7f77f8157040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f77e800d5f0 tx=0x7f77e800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:33.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.318+0000 7f77efe00640 1 -- 172.21.15.69:0/3928517467 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e80125f0 con 0x7f77f80ad0d0 2024-04-24T08:08:33.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.318+0000 7f77efe00640 1 -- 172.21.15.69:0/3928517467 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77e8019070 con 0x7f77f80ad0d0 2024-04-24T08:08:33.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.318+0000 7f77efe00640 1 -- 172.21.15.69:0/3928517467 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77e80129d0 con 0x7f77f80ad0d0 2024-04-24T08:08:33.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.318+0000 7f77fda00640 1 -- 172.21.15.69:0/3928517467 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77f8072370 con 0x7f77f80ad0d0 2024-04-24T08:08:33.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.318+0000 7f77fda00640 1 -- 172.21.15.69:0/3928517467 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77f8072c10 con 0x7f77f80ad0d0 2024-04-24T08:08:33.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.319+0000 7f77fda00640 1 -- 172.21.15.69:0/3928517467 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77f8072540 con 0x7f77f80ad0d0 2024-04-24T08:08:33.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.320+0000 7f77efe00640 1 -- 172.21.15.69:0/3928517467 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f77e8012b70 con 0x7f77f80ad0d0 2024-04-24T08:08:33.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.321+0000 7f77efe00640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77d806c8f0 0x7f77d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.321+0000 7f77efe00640 1 -- 172.21.15.69:0/3928517467 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f77e808c070 con 0x7f77f80ad0d0 2024-04-24T08:08:33.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.322+0000 7f77f7400640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77d806c8f0 0x7f77d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.323+0000 7f77f7400640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77d806c8f0 0x7f77d806edb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f77e4034000 tx=0x7f77e402c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:33.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.325+0000 7f77efe00640 1 -- 172.21.15.69:0/3928517467 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f77e805a400 con 0x7f77f80ad0d0 2024-04-24T08:08:33.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.529+0000 7f77fda00640 1 -- 172.21.15.69:0/3928517467 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77f8075d50 con 0x7f77f80ad0d0 2024-04-24T08:08:33.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.530+0000 7f77efe00640 1 -- 172.21.15.69:0/3928517467 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7f77e8059d00 con 0x7f77f80ad0d0 2024-04-24T08:08:33.532 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:33.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 -- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77d806c8f0 msgr2=0x7f77d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77d806c8f0 0x7f77d806edb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f77e4034000 tx=0x7f77e402c000 comp rx=0 tx=0).stop 2024-04-24T08:08:33.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 -- 172.21.15.69:0/3928517467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80ad0d0 msgr2=0x7f77f8157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80ad0d0 0x7f77f8157040 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f77e800d5f0 tx=0x7f77e800dab0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 -- 172.21.15.69:0/3928517467 shutdown_connections 2024-04-24T08:08:33.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77d806c8f0 0x7f77d806edb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77f8156c60 0x7f77f81595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77f80b30f0 0x7f77f81f7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 --2- 172.21.15.69:0/3928517467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77f80ad0d0 0x7f77f8157040 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 -- 172.21.15.69:0/3928517467 >> 172.21.15.69:0/3928517467 conn(0x7f77f80d0160 msgr2=0x7f77f80afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:33.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 -- 172.21.15.69:0/3928517467 shutdown_connections 2024-04-24T08:08:33.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.533+0000 7f77fda00640 1 -- 172.21.15.69:0/3928517467 wait complete. 2024-04-24T08:08:33.553 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:08:33.554 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:08:33.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.757+0000 7f4322600640 1 Processor -- start 2024-04-24T08:08:33.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.757+0000 7f4322600640 1 -- start start 2024-04-24T08:08:33.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.759+0000 7f4322600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f431c15e980 0x7f431c162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.759+0000 7f4322600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f431c0d0da0 0x7f431c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.759+0000 7f4322600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f431c15ed60 con 0x7f431c0d0190 2024-04-24T08:08:33.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.759+0000 7f4322600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f431c0d1580 con 0x7f431c15e980 2024-04-24T08:08:33.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.759+0000 7f4322600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f431c0d1860 con 0x7f431c0d0da0 2024-04-24T08:08:33.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.759+0000 7f4321200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f431c15e980 0x7f431c162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.759+0000 7f4321200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f431c15e980 0x7f431c162f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44566/0 (socket says 172.21.15.69:44566) 2024-04-24T08:08:33.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.759+0000 7f4321200640 1 -- 172.21.15.69:0/436388368 learned_addr learned my addr 172.21.15.69:0/436388368 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:33.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.759+0000 7f4313e00640 1 --2- 172.21.15.69:0/436388368 >> v2:172.21.15.121:3300/0 conn(0x7f431c0d0da0 0x7f431c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.760+0000 7f4321200640 1 -- 172.21.15.69:0/436388368 >> v2:172.21.15.121:3300/0 conn(0x7f431c0d0da0 msgr2=0x7f431c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:33.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.760+0000 7f4321200640 1 --2- 172.21.15.69:0/436388368 >> v2:172.21.15.121:3300/0 conn(0x7f431c0d0da0 0x7f431c0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.760+0000 7f4321200640 1 -- 172.21.15.69:0/436388368 >> v1:172.21.15.121:6789/0 conn(0x7f431c0d0190 legacy=0x7f431c06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:33.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.760+0000 7f4321200640 1 -- 172.21.15.69:0/436388368 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431c0cf450 con 0x7f431c15e980 2024-04-24T08:08:33.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.761+0000 7f4321200640 1 --2- 172.21.15.69:0/436388368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 0x7f431c162f90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f431c06c480 tx=0x7f430c02a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81cc822eabbdbda0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:33.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.761+0000 7f431be00640 1 -- 172.21.15.69:0/436388368 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c037070 con 0x7f431c15e980 2024-04-24T08:08:33.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.761+0000 7f431be00640 1 -- 172.21.15.69:0/436388368 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f430c032440 con 0x7f431c15e980 2024-04-24T08:08:33.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.761+0000 7f431be00640 1 -- 172.21.15.69:0/436388368 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430c032810 con 0x7f431c15e980 2024-04-24T08:08:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.762+0000 7f4322600640 1 -- 172.21.15.69:0/436388368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 msgr2=0x7f431c162f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.762+0000 7f4322600640 1 --2- 172.21.15.69:0/436388368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 0x7f431c162f90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f431c06c480 tx=0x7f430c02a2c0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.762+0000 7f4322600640 1 -- 172.21.15.69:0/436388368 shutdown_connections 2024-04-24T08:08:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.762+0000 7f4322600640 1 --2- 172.21.15.69:0/436388368 >> v2:172.21.15.121:3300/0 conn(0x7f431c0d0da0 0x7f431c0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.762+0000 7f4322600640 1 --2- 172.21.15.69:0/436388368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 0x7f431c162f90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.762+0000 7f4322600640 1 -- 172.21.15.69:0/436388368 >> 172.21.15.69:0/436388368 conn(0x7f431c09d920 msgr2=0x7f431c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.762+0000 7f4322600640 1 -- 172.21.15.69:0/436388368 shutdown_connections 2024-04-24T08:08:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.762+0000 7f4322600640 1 -- 172.21.15.69:0/436388368 wait complete. 2024-04-24T08:08:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.764+0000 7f4322600640 1 Processor -- start 2024-04-24T08:08:33.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.764+0000 7f4322600640 1 -- start start 2024-04-24T08:08:33.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4322600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c0d0190 0x7f431c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4322600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f431c0d0da0 0x7f431c0b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4322600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 0x7f431c0ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4322600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f431c0d1360 con 0x7f431c15e980 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4322600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f431c0d11e0 con 0x7f431c0d0190 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4322600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f431c0cf070 con 0x7f431c0d0da0 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4321c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 0x7f431c0ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4321c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 0x7f431c0ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44572/0 (socket says 172.21.15.69:44572) 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4321c00640 1 -- 172.21.15.69:0/3541628760 learned_addr learned my addr 172.21.15.69:0/3541628760 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4313e00640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f431c0d0da0 0x7f431c0b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4321200640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c0d0190 0x7f431c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4321c00640 1 -- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f431c0d0da0 msgr2=0x7f431c0b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4321c00640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f431c0d0da0 0x7f431c0b8290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4321c00640 1 -- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c0d0190 msgr2=0x7f431c152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4321c00640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c0d0190 0x7f431c152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.765+0000 7f4321c00640 1 -- 172.21.15.69:0/3541628760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f431c0b71f0 con 0x7f431c15e980 2024-04-24T08:08:33.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.766+0000 7f4321c00640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 0x7f431c0ca590 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f431400b9a0 tx=0x7f431400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:33.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.766+0000 7f4319600640 1 -- 172.21.15.69:0/3541628760 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4314019070 con 0x7f431c15e980 2024-04-24T08:08:33.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.766+0000 7f4319600640 1 -- 172.21.15.69:0/3541628760 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f431400cba0 con 0x7f431c15e980 2024-04-24T08:08:33.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.766+0000 7f4319600640 1 -- 172.21.15.69:0/3541628760 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4314014440 con 0x7f431c15e980 2024-04-24T08:08:33.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.766+0000 7f4322600640 1 -- 172.21.15.69:0/3541628760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f431c0b74d0 con 0x7f431c15e980 2024-04-24T08:08:33.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.766+0000 7f4322600640 1 -- 172.21.15.69:0/3541628760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f431c0b7a90 con 0x7f431c15e980 2024-04-24T08:08:33.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.767+0000 7f4322600640 1 -- 172.21.15.69:0/3541628760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f431c075d50 con 0x7f431c15e980 2024-04-24T08:08:33.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.768+0000 7f4319600640 1 -- 172.21.15.69:0/3541628760 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f431401c070 con 0x7f431c15e980 2024-04-24T08:08:33.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.769+0000 7f4319600640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f42f806c970 0x7f42f806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.769+0000 7f4319600640 1 -- 172.21.15.69:0/3541628760 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f431408b820 con 0x7f431c15e980 2024-04-24T08:08:33.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.770+0000 7f4321200640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f42f806c970 0x7f42f806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.771+0000 7f4321200640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f42f806c970 0x7f42f806ee30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f430c034000 tx=0x7f430c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:33.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.774+0000 7f4319600640 1 -- 172.21.15.69:0/3541628760 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4314059c30 con 0x7f431c15e980 2024-04-24T08:08:33.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.972+0000 7f4322600640 1 -- 172.21.15.69:0/3541628760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f431c06d150 con 0x7f431c15e980 2024-04-24T08:08:33.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.973+0000 7f4319600640 1 -- 172.21.15.69:0/3541628760 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f4314059530 con 0x7f431c15e980 2024-04-24T08:08:33.974 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:08:33.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 -- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f42f806c970 msgr2=0x7f42f806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f42f806c970 0x7f42f806ee30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f430c034000 tx=0x7f430c02e000 comp rx=0 tx=0).stop 2024-04-24T08:08:33.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 -- 172.21.15.69:0/3541628760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 msgr2=0x7f431c0ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 0x7f431c0ca590 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f431400b9a0 tx=0x7f431400be60 comp rx=0 tx=0).stop 2024-04-24T08:08:33.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 -- 172.21.15.69:0/3541628760 shutdown_connections 2024-04-24T08:08:33.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f42f806c970 0x7f42f806ee30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f431c15e980 0x7f431c0ca590 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f431c0d0da0 0x7f431c0b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 --2- 172.21.15.69:0/3541628760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c0d0190 0x7f431c152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 -- 172.21.15.69:0/3541628760 >> 172.21.15.69:0/3541628760 conn(0x7f431c09d920 msgr2=0x7f431c0b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:33.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 -- 172.21.15.69:0/3541628760 shutdown_connections 2024-04-24T08:08:33.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:33.976+0000 7f4322600640 1 -- 172.21.15.69:0/3541628760 wait complete. 2024-04-24T08:08:33.997 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:08:34.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.198+0000 7f0912400640 1 Processor -- start 2024-04-24T08:08:34.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.199+0000 7f0912400640 1 -- start start 2024-04-24T08:08:34.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.200+0000 7f0912400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f090c15e850 0x7f090c0cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.200+0000 7f0912400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 0x7f090c1688f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.200+0000 7f0912400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f090c0d2710 con 0x7f090c15ec30 2024-04-24T08:08:34.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.200+0000 7f0912400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f090c0d37b0 con 0x7f090c15e850 2024-04-24T08:08:34.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.200+0000 7f0912400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f090c06eba0 con 0x7f090c0d1860 2024-04-24T08:08:34.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.200+0000 7f090b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f090c15e850 0x7f090c0cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.200+0000 7f090aa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f090c15ec30 0x7f090c0d1150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36640/0 (socket says 172.21.15.69:36640) 2024-04-24T08:08:34.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.200+0000 7f090aa00640 1 -- 172.21.15.69:0/4168655761 learned_addr learned my addr 172.21.15.69:0/4168655761 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:34.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.201+0000 7f090be00640 1 --2- 172.21.15.69:0/4168655761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 0x7f090c1688f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.201+0000 7f090b400640 1 -- 172.21.15.69:0/4168655761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 msgr2=0x7f090c1688f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:34.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.201+0000 7f090b400640 1 --2- 172.21.15.69:0/4168655761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 0x7f090c1688f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.201+0000 7f090b400640 1 -- 172.21.15.69:0/4168655761 >> v1:172.21.15.69:6789/0 conn(0x7f090c15ec30 legacy=0x7f090c0d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.201+0000 7f090b400640 1 -- 172.21.15.69:0/4168655761 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090c0d29f0 con 0x7f090c15e850 2024-04-24T08:08:34.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.201+0000 7f090a000640 1 -- 172.21.15.69:0/4168655761 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4181369368 0 0) 0x7f090c0d2710 con 0x7f090c15ec30 2024-04-24T08:08:34.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.201+0000 7f090be00640 1 --2- 172.21.15.69:0/4168655761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 0x7f090c1688f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:34.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.202+0000 7f090b400640 1 --2- 172.21.15.69:0/4168655761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 0x7f090c0cff10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009d90 tx=0x7f08fc02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500342cccc218eee server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.202+0000 7f090a000640 1 -- 172.21.15.69:0/4168655761 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc007740 con 0x7f090c15e850 2024-04-24T08:08:34.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.202+0000 7f090a000640 1 -- 172.21.15.69:0/4168655761 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08fc0078e0 con 0x7f090c15e850 2024-04-24T08:08:34.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.202+0000 7f090a000640 1 -- 172.21.15.69:0/4168655761 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08fc007cb0 con 0x7f090c15e850 2024-04-24T08:08:34.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.203+0000 7f0912400640 1 -- 172.21.15.69:0/4168655761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 msgr2=0x7f090c0cff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.203+0000 7f0912400640 1 --2- 172.21.15.69:0/4168655761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 0x7f090c0cff10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f08fc009d90 tx=0x7f08fc02a660 comp rx=0 tx=0).stop 2024-04-24T08:08:34.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.203+0000 7f0912400640 1 -- 172.21.15.69:0/4168655761 shutdown_connections 2024-04-24T08:08:34.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.203+0000 7f0912400640 1 --2- 172.21.15.69:0/4168655761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 0x7f090c1688f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.203+0000 7f0912400640 1 --2- 172.21.15.69:0/4168655761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 0x7f090c0cff10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.203+0000 7f0912400640 1 -- 172.21.15.69:0/4168655761 >> 172.21.15.69:0/4168655761 conn(0x7f090c08bf60 msgr2=0x7f090c0d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.203+0000 7f0912400640 1 -- 172.21.15.69:0/4168655761 shutdown_connections 2024-04-24T08:08:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.203+0000 7f0912400640 1 -- 172.21.15.69:0/4168655761 wait complete. 2024-04-24T08:08:34.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.204+0000 7f0912400640 1 Processor -- start 2024-04-24T08:08:34.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.205+0000 7f0912400640 1 -- start start 2024-04-24T08:08:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.205+0000 7f0912400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 0x7f090c0b83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.205+0000 7f0912400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 0x7f090c0cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.205+0000 7f0912400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f090c15ec30 0x7f090c0cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.205+0000 7f0912400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f090c0d3290 con 0x7f090c15e850 2024-04-24T08:08:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.205+0000 7f0912400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f090c0d2fb0 con 0x7f090c15ec30 2024-04-24T08:08:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.205+0000 7f0912400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f090c0d2330 con 0x7f090c0d1860 2024-04-24T08:08:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 0x7f090c0cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 0x7f090c0cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44584/0 (socket says 172.21.15.69:44584) 2024-04-24T08:08:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090aa00640 1 -- 172.21.15.69:0/785428953 learned_addr learned my addr 172.21.15.69:0/785428953 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090b400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 0x7f090c0b83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090aa00640 1 -- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 msgr2=0x7f090c0b83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090be00640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f090c15ec30 0x7f090c0cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090aa00640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 0x7f090c0b83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090aa00640 1 -- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f090c15ec30 msgr2=0x7f090c0cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090aa00640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f090c15ec30 0x7f090c0cbd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090aa00640 1 -- 172.21.15.69:0/785428953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f090c0b8d40 con 0x7f090c15e850 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090b400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 0x7f090c0b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.206+0000 7f090aa00640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 0x7f090c0cb830 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08f400c430 tx=0x7f08f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.207+0000 7f0903400640 1 -- 172.21.15.69:0/785428953 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f40074e0 con 0x7f090c15e850 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.207+0000 7f0903400640 1 -- 172.21.15.69:0/785428953 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08f40047e0 con 0x7f090c15e850 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.207+0000 7f0903400640 1 -- 172.21.15.69:0/785428953 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08f4004bd0 con 0x7f090c15e850 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.207+0000 7f0912400640 1 -- 172.21.15.69:0/785428953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f090c0b9020 con 0x7f090c15e850 2024-04-24T08:08:34.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.207+0000 7f0912400640 1 -- 172.21.15.69:0/785428953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f090c0b95e0 con 0x7f090c15e850 2024-04-24T08:08:34.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.208+0000 7f0912400640 1 -- 172.21.15.69:0/785428953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f090c068a80 con 0x7f090c15e850 2024-04-24T08:08:34.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.209+0000 7f0903400640 1 -- 172.21.15.69:0/785428953 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f08f4011070 con 0x7f090c15e850 2024-04-24T08:08:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.210+0000 7f0903400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08e806c920 0x7f08e806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.210+0000 7f0903400640 1 -- 172.21.15.69:0/785428953 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f08f408b870 con 0x7f090c15e850 2024-04-24T08:08:34.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.211+0000 7f090b400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08e806c920 0x7f08e806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.212+0000 7f090b400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08e806c920 0x7f08e806ede0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f08fc034000 tx=0x7f08fc02f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.214+0000 7f0903400640 1 -- 172.21.15.69:0/785428953 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f08f4059c00 con 0x7f090c15e850 2024-04-24T08:08:34.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.416+0000 7f0912400640 1 -- 172.21.15.69:0/785428953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f090c0d1f50 con 0x7f090c15e850 2024-04-24T08:08:34.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.417+0000 7f0903400640 1 -- 172.21.15.69:0/785428953 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f08f4059500 con 0x7f090c15e850 2024-04-24T08:08:34.418 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:08:34.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 -- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08e806c920 msgr2=0x7f08e806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08e806c920 0x7f08e806ede0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f08fc034000 tx=0x7f08fc02f000 comp rx=0 tx=0).stop 2024-04-24T08:08:34.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 -- 172.21.15.69:0/785428953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 msgr2=0x7f090c0cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 0x7f090c0cb830 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f08f400c430 tx=0x7f08f400c8f0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 -- 172.21.15.69:0/785428953 shutdown_connections 2024-04-24T08:08:34.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08e806c920 0x7f08e806ede0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f090c15e850 0x7f090c0cb830 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f090c15ec30 0x7f090c0cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 --2- 172.21.15.69:0/785428953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f090c0d1860 0x7f090c0b83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 -- 172.21.15.69:0/785428953 >> 172.21.15.69:0/785428953 conn(0x7f090c08bf60 msgr2=0x7f090c15f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:34.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 -- 172.21.15.69:0/785428953 shutdown_connections 2024-04-24T08:08:34.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.419+0000 7f0912400640 1 -- 172.21.15.69:0/785428953 wait complete. 2024-04-24T08:08:34.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:08:34.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.639+0000 7fd727c00640 1 Processor -- start 2024-04-24T08:08:34.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.640+0000 7fd727c00640 1 -- start start 2024-04-24T08:08:34.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.641+0000 7fd727c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd72015c820 0x7fd7200b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.641+0000 7fd727c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd7200b46a0 0x7fd72006f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.641+0000 7fd727c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd72006fd10 0x7fd7200ce250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.641+0000 7fd727c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd72015cc00 con 0x7fd72006fd10 2024-04-24T08:08:34.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.641+0000 7fd727c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd72006ed30 con 0x7fd7200b46a0 2024-04-24T08:08:34.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.641+0000 7fd727c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd7200b4e10 con 0x7fd72015c820 2024-04-24T08:08:34.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.641+0000 7fd725c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd72006fd10 0x7fd7200ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.642+0000 7fd725c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd72006fd10 0x7fd7200ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44598/0 (socket says 172.21.15.69:44598) 2024-04-24T08:08:34.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.642+0000 7fd725c00640 1 -- 172.21.15.69:0/3918913431 learned_addr learned my addr 172.21.15.69:0/3918913431 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:34.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.642+0000 7fd717e00640 1 --2- 172.21.15.69:0/3918913431 >> v2:172.21.15.121:3300/0 conn(0x7fd7200b46a0 0x7fd72006f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.642+0000 7fd725200640 1 --2- 172.21.15.69:0/3918913431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd72015c820 0x7fd7200b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.642+0000 7fd725c00640 1 -- 172.21.15.69:0/3918913431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd72015c820 msgr2=0x7fd7200b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:34.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.642+0000 7fd725c00640 1 --2- 172.21.15.69:0/3918913431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd72015c820 0x7fd7200b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.642+0000 7fd725c00640 1 -- 172.21.15.69:0/3918913431 >> v2:172.21.15.121:3300/0 conn(0x7fd7200b46a0 msgr2=0x7fd72006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:34.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.642+0000 7fd725c00640 1 --2- 172.21.15.69:0/3918913431 >> v2:172.21.15.121:3300/0 conn(0x7fd7200b46a0 0x7fd72006f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.642+0000 7fd725c00640 1 -- 172.21.15.69:0/3918913431 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7200b50f0 con 0x7fd72006fd10 2024-04-24T08:08:34.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.643+0000 7fd725c00640 1 --2- 172.21.15.69:0/3918913431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 0x7fd7200ce250 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd7200b2fe0 tx=0x7fd71802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664ade76e17ca2ec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.643+0000 7fd71fe00640 1 -- 172.21.15.69:0/3918913431 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718036070 con 0x7fd72006fd10 2024-04-24T08:08:34.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.643+0000 7fd71fe00640 1 -- 172.21.15.69:0/3918913431 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd718031600 con 0x7fd72006fd10 2024-04-24T08:08:34.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.643+0000 7fd71fe00640 1 -- 172.21.15.69:0/3918913431 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7180319d0 con 0x7fd72006fd10 2024-04-24T08:08:34.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.644+0000 7fd727c00640 1 -- 172.21.15.69:0/3918913431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 msgr2=0x7fd7200ce250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.644+0000 7fd727c00640 1 --2- 172.21.15.69:0/3918913431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 0x7fd7200ce250 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd7200b2fe0 tx=0x7fd71802a6a0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.644+0000 7fd727c00640 1 -- 172.21.15.69:0/3918913431 shutdown_connections 2024-04-24T08:08:34.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.644+0000 7fd727c00640 1 --2- 172.21.15.69:0/3918913431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 0x7fd7200ce250 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.644+0000 7fd727c00640 1 --2- 172.21.15.69:0/3918913431 >> v2:172.21.15.121:3300/0 conn(0x7fd7200b46a0 0x7fd72006f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.644+0000 7fd727c00640 1 --2- 172.21.15.69:0/3918913431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd72015c820 0x7fd7200b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.644+0000 7fd727c00640 1 -- 172.21.15.69:0/3918913431 >> 172.21.15.69:0/3918913431 conn(0x7fd7200d0350 msgr2=0x7fd7200d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:34.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.644+0000 7fd727c00640 1 -- 172.21.15.69:0/3918913431 shutdown_connections 2024-04-24T08:08:34.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.644+0000 7fd727c00640 1 -- 172.21.15.69:0/3918913431 wait complete. 2024-04-24T08:08:34.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.646+0000 7fd727c00640 1 Processor -- start 2024-04-24T08:08:34.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.646+0000 7fd727c00640 1 -- start start 2024-04-24T08:08:34.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.647+0000 7fd727c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 0x7fd7200726e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.647+0000 7fd727c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7200b46a0 0x7fd720151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.647+0000 7fd727c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd72015c820 0x7fd7201566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.647+0000 7fd727c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd7200b4b00 con 0x7fd72006fd10 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.647+0000 7fd727c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd72006f2b0 con 0x7fd7200b46a0 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.647+0000 7fd727c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd7200ce790 con 0x7fd72015c820 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.647+0000 7fd725200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 0x7fd7200726e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.647+0000 7fd725200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 0x7fd7200726e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44614/0 (socket says 172.21.15.69:44614) 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.647+0000 7fd725200640 1 -- 172.21.15.69:0/3045952485 learned_addr learned my addr 172.21.15.69:0/3045952485 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.647+0000 7fd725c00640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd72015c820 0x7fd7201566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd717e00640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7200b46a0 0x7fd720151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd725200640 1 -- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd72015c820 msgr2=0x7fd7201566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd725200640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd72015c820 0x7fd7201566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd725200640 1 -- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7200b46a0 msgr2=0x7fd720151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd725200640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7200b46a0 0x7fd720151180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd725200640 1 -- 172.21.15.69:0/3045952485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7201516c0 con 0x7fd72006fd10 2024-04-24T08:08:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd725200640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 0x7fd7200726e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd70800d5f0 tx=0x7fd70800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd71d600640 1 -- 172.21.15.69:0/3045952485 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080125f0 con 0x7fd72006fd10 2024-04-24T08:08:34.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd71d600640 1 -- 172.21.15.69:0/3045952485 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd708019070 con 0x7fd72006fd10 2024-04-24T08:08:34.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd71d600640 1 -- 172.21.15.69:0/3045952485 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080129d0 con 0x7fd72006fd10 2024-04-24T08:08:34.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd727c00640 1 -- 172.21.15.69:0/3045952485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd720073ca0 con 0x7fd72006fd10 2024-04-24T08:08:34.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.648+0000 7fd727c00640 1 -- 172.21.15.69:0/3045952485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7200c9b60 con 0x7fd72006fd10 2024-04-24T08:08:34.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.650+0000 7fd727c00640 1 -- 172.21.15.69:0/3045952485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7200cf4e0 con 0x7fd72006fd10 2024-04-24T08:08:34.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.650+0000 7fd71d600640 1 -- 172.21.15.69:0/3045952485 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd708012b70 con 0x7fd72006fd10 2024-04-24T08:08:34.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.651+0000 7fd71d600640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6fc06c8f0 0x7fd6fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.651+0000 7fd71d600640 1 -- 172.21.15.69:0/3045952485 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd70808bfe0 con 0x7fd72006fd10 2024-04-24T08:08:34.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.652+0000 7fd717e00640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6fc06c8f0 0x7fd6fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.653+0000 7fd717e00640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6fc06c8f0 0x7fd6fc06edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd7201573d0 tx=0x7fd710008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.656+0000 7fd71d600640 1 -- 172.21.15.69:0/3045952485 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7200cf4e0 con 0x7fd72006fd10 2024-04-24T08:08:34.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.855+0000 7fd727c00640 1 -- 172.21.15.69:0/3045952485 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd72015c080 con 0x7fd72006fd10 2024-04-24T08:08:34.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.856+0000 7fd71d600640 1 -- 172.21.15.69:0/3045952485 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fd72015c080 con 0x7fd72006fd10 2024-04-24T08:08:34.858 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 -- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6fc06c8f0 msgr2=0x7fd6fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6fc06c8f0 0x7fd6fc06edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd7201573d0 tx=0x7fd710008040 comp rx=0 tx=0).stop 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 -- 172.21.15.69:0/3045952485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 msgr2=0x7fd7200726e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 0x7fd7200726e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd70800d5f0 tx=0x7fd70800dab0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 -- 172.21.15.69:0/3045952485 shutdown_connections 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6fc06c8f0 0x7fd6fc06edb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd72015c820 0x7fd7201566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7200b46a0 0x7fd720151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 --2- 172.21.15.69:0/3045952485 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd72006fd10 0x7fd7200726e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 -- 172.21.15.69:0/3045952485 >> 172.21.15.69:0/3045952485 conn(0x7fd7200d0350 msgr2=0x7fd7200cedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 -- 172.21.15.69:0/3045952485 shutdown_connections 2024-04-24T08:08:34.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:34.859+0000 7fd727c00640 1 -- 172.21.15.69:0/3045952485 wait complete. 2024-04-24T08:08:34.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:08:35.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.071+0000 7fe7ae200640 1 Processor -- start 2024-04-24T08:08:35.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.071+0000 7fe7ae200640 1 -- start start 2024-04-24T08:08:35.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.072+0000 7fe7ae200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe7a80d02f0 0x7fe7a806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.072+0000 7fe7ae200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7a80b46e0 con 0x7fe7a806f2d0 2024-04-24T08:08:35.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.072+0000 7fe7ae200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7a80d0960 con 0x7fe7a815b490 2024-04-24T08:08:35.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.073+0000 7fe7ae200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe7a80d0720 con 0x7fe7a80d02f0 2024-04-24T08:08:35.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.073+0000 7fe79fe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe7a806f2d0 0x7fe7a80cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36656/0 (socket says 172.21.15.69:36656) 2024-04-24T08:08:35.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.073+0000 7fe79fe00640 1 -- 172.21.15.69:0/1467522215 learned_addr learned my addr 172.21.15.69:0/1467522215 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:35.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.073+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3096752370 0 0) 0x7fe7a80b46e0 con 0x7fe7a806f2d0 2024-04-24T08:08:35.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.073+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe78c005730 con 0x7fe7a806f2d0 2024-04-24T08:08:35.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2241964309 0 0) 0x7fe78c005730 con 0x7fe7a806f2d0 2024-04-24T08:08:35.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 >> v2:172.21.15.121:3300/0 conn(0x7fe7a80d02f0 msgr2=0x7fe7a806f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:35.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a7e00640 1 --2- 172.21.15.69:0/1467522215 >> v2:172.21.15.121:3300/0 conn(0x7fe7a80d02f0 0x7fe7a806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 --2- 172.21.15.69:0/1467522215 >> v2:172.21.15.121:3300/0 conn(0x7fe7a80d02f0 0x7fe7a806f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 >> v1:172.21.15.121:6789/0 conn(0x7fe7a815b490 legacy=0x7fe7a80b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a80b4980 con 0x7fe7a806f2d0 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe790001690 con 0x7fe7a806f2d0 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a7e00640 1 --2- 172.21.15.69:0/1467522215 >> v2:172.21.15.121:3300/0 conn(0x7fe7a80d02f0 0x7fe7a806f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 >> v1:172.21.15.69:6789/0 conn(0x7fe7a806f2d0 legacy=0x7fe7a80cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78c005c10 0x7fe78c006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c006a10 0x7fe78c006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe78c00dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a80d0720 con 0x7fe78c0075d0 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c00e440 con 0x7fe78c006a10 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe78c00e640 con 0x7fe78c005c10 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.074+0000 7fe7a7e00640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe78c00dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.075+0000 7fe79fe00640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c006a10 0x7fe78c006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.075+0000 7fe7a7400640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78c005c10 0x7fe78c006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.075+0000 7fe7a7e00640 1 -- 172.21.15.69:0/1467522215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78c005c10 msgr2=0x7fe78c006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:35.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.075+0000 7fe7a7e00640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe78c005c10 0x7fe78c006030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.075+0000 7fe7a7e00640 1 -- 172.21.15.69:0/1467522215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c006a10 msgr2=0x7fe78c006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:35.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.075+0000 7fe7a7e00640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c006a10 0x7fe78c006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.075+0000 7fe7a7e00640 1 -- 172.21.15.69:0/1467522215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78c005730 con 0x7fe78c0075d0 2024-04-24T08:08:35.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.075+0000 7fe79fe00640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe78c006a10 0x7fe78c006e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:35.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.076+0000 7fe7a7e00640 1 -- 172.21.15.69:0/1467522215 reap_dead start 2024-04-24T08:08:35.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.076+0000 7fe7a7e00640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe78c00dbd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000c490 tx=0x7fe78c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8346c4e12b0ae2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:35.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.076+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0038070 con 0x7fe78c0075d0 2024-04-24T08:08:35.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.076+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7a002bbb0 con 0x7fe78c0075d0 2024-04-24T08:08:35.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.076+0000 7fe7a6a00640 1 -- 172.21.15.69:0/1467522215 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0007c10 con 0x7fe78c0075d0 2024-04-24T08:08:35.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.077+0000 7fe7ae200640 1 -- 172.21.15.69:0/1467522215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 msgr2=0x7fe78c00dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.077+0000 7fe7ae200640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe78c00dbd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000c490 tx=0x7fe78c007a40 comp rx=0 tx=0).stop 2024-04-24T08:08:35.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.077+0000 7fe7ae200640 1 -- 172.21.15.69:0/1467522215 shutdown_connections 2024-04-24T08:08:35.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.077+0000 7fe7ae200640 1 --2- 172.21.15.69:0/1467522215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe78c00dbd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.077+0000 7fe7ae200640 1 -- 172.21.15.69:0/1467522215 >> 172.21.15.69:0/1467522215 conn(0x7fe7a8067d90 msgr2=0x7fe7a806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:35.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.077+0000 7fe7ae200640 1 -- 172.21.15.69:0/1467522215 shutdown_connections 2024-04-24T08:08:35.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.077+0000 7fe7ae200640 1 -- 172.21.15.69:0/1467522215 wait complete. 2024-04-24T08:08:35.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.078+0000 7fe7ae200640 1 Processor -- start 2024-04-24T08:08:35.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.079+0000 7fe7ae200640 1 -- start start 2024-04-24T08:08:35.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.079+0000 7fe7ae200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe7a80cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.079+0000 7fe7ae200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a81f0240 0x7fe7a81510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.079+0000 7fe7ae200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7a8151620 0x7fe7a8152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.079+0000 7fe7ae200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a815b870 con 0x7fe78c0075d0 2024-04-24T08:08:35.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.079+0000 7fe7ae200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a80d0c40 con 0x7fe7a81f0240 2024-04-24T08:08:35.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.079+0000 7fe7ae200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a806f730 con 0x7fe7a8151620 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe7a7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe7a80cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe7a7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe7a80cc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44640/0 (socket says 172.21.15.69:44640) 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe7a7400640 1 -- 172.21.15.69:0/3826354757 learned_addr learned my addr 172.21.15.69:0/3826354757 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe7a7e00640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7a8151620 0x7fe7a8152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe79fe00640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a81f0240 0x7fe7a81510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe7a7400640 1 -- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7a8151620 msgr2=0x7fe7a8152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe7a7400640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7a8151620 0x7fe7a8152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe7a7400640 1 -- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a81f0240 msgr2=0x7fe7a81510e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe7a7400640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a81f0240 0x7fe7a81510e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe7a7400640 1 -- 172.21.15.69:0/3826354757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a815ba90 con 0x7fe78c0075d0 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.080+0000 7fe7a7400640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe7a80cc840 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe79800d570 tx=0x7fe79800da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.081+0000 7fe79f400640 1 -- 172.21.15.69:0/3826354757 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe798003610 con 0x7fe78c0075d0 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.081+0000 7fe79f400640 1 -- 172.21.15.69:0/3826354757 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe798017070 con 0x7fe78c0075d0 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.081+0000 7fe79f400640 1 -- 172.21.15.69:0/3826354757 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7980039b0 con 0x7fe78c0075d0 2024-04-24T08:08:35.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.081+0000 7fe7ae200640 1 -- 172.21.15.69:0/3826354757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a815bd30 con 0x7fe78c0075d0 2024-04-24T08:08:35.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.081+0000 7fe7ae200640 1 -- 172.21.15.69:0/3826354757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a815c2f0 con 0x7fe78c0075d0 2024-04-24T08:08:35.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.082+0000 7fe7ae200640 1 -- 172.21.15.69:0/3826354757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7a8068a80 con 0x7fe78c0075d0 2024-04-24T08:08:35.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.083+0000 7fe79f400640 1 -- 172.21.15.69:0/3826354757 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe798003b50 con 0x7fe78c0075d0 2024-04-24T08:08:35.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.084+0000 7fe79f400640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe78406c920 0x7fe78406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.084+0000 7fe79f400640 1 -- 172.21.15.69:0/3826354757 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe79808b5d0 con 0x7fe78c0075d0 2024-04-24T08:08:35.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.084+0000 7fe79fe00640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe78406c920 0x7fe78406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.085+0000 7fe79fe00640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe78406c920 0x7fe78406ede0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe790001800 tx=0x7fe790009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:35.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.088+0000 7fe79f400640 1 -- 172.21.15.69:0/3826354757 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe798059ab0 con 0x7fe78c0075d0 2024-04-24T08:08:35.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.334+0000 7fe7ae200640 1 -- 172.21.15.69:0/3826354757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7a80b5040 con 0x7fe78c0075d0 2024-04-24T08:08:35.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.335+0000 7fe79f400640 1 -- 172.21.15.69:0/3826354757 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7fe7980593b0 con 0x7fe78c0075d0 2024-04-24T08:08:35.337 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:35.337 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/934340733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":934340733},{"type":"v1","addr":"172.21.15.121:6835","nonce":934340733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3140867113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3140867113},{"type":"v1","addr":"172.21.15.121:6837","nonce":3140867113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/1059995973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1059995973},{"type":"v1","addr":"172.21.15.69:6833","nonce":1059995973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:08:19.574083+0000","modified":"2024-04-24T08:08:22.605668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/1003206529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1003206529},{"type":"v1","addr":"172.21.15.69:6835","nonce":1003206529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-24T08:08:35.337 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 7 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 -- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe78406c920 msgr2=0x7fe78406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe78406c920 0x7fe78406ede0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe790001800 tx=0x7fe790009040 comp rx=0 tx=0).stop 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 -- 172.21.15.69:0/3826354757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 msgr2=0x7fe7a80cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe7a80cc840 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe79800d570 tx=0x7fe79800da30 comp rx=0 tx=0).stop 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 -- 172.21.15.69:0/3826354757 shutdown_connections 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe78406c920 0x7fe78406ede0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7a8151620 0x7fe7a8152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7a81f0240 0x7fe7a81510e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 --2- 172.21.15.69:0/3826354757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe78c0075d0 0x7fe7a80cc840 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 -- 172.21.15.69:0/3826354757 >> 172.21.15.69:0/3826354757 conn(0x7fe7a8067d90 msgr2=0x7fe7a8159220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 -- 172.21.15.69:0/3826354757 shutdown_connections 2024-04-24T08:08:35.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.338+0000 7fe7ae200640 1 -- 172.21.15.69:0/3826354757 wait complete. 2024-04-24T08:08:35.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:35.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.542+0000 7f605aa00640 1 Processor -- start 2024-04-24T08:08:35.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.543+0000 7f605aa00640 1 -- start start 2024-04-24T08:08:35.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.544+0000 7f605aa00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f605406f2d0 0x7f605406f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.544+0000 7f605aa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 0x7f60540d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.544+0000 7f605aa00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60540b46e0 con 0x7f605415b490 2024-04-24T08:08:35.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.544+0000 7f605aa00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f605406fc70 con 0x7f605406f2d0 2024-04-24T08:08:35.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.544+0000 7f605aa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f60540b5890 con 0x7f60540cfbe0 2024-04-24T08:08:35.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.545+0000 7f604b400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f605406f2d0 0x7f605406f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.545+0000 7f6058a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 0x7f60540d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.545+0000 7f604b400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f605406f2d0 0x7f605406f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:32916/0 (socket says 172.21.15.69:32916) 2024-04-24T08:08:35.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.545+0000 7f604b400640 1 -- 172.21.15.69:0/3913167641 learned_addr learned my addr 172.21.15.69:0/3913167641 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:35.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.546+0000 7f604b400640 1 -- 172.21.15.69:0/3913167641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 msgr2=0x7f60540d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:35.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.546+0000 7f604b400640 1 --2- 172.21.15.69:0/3913167641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 0x7f60540d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.546+0000 7f604b400640 1 -- 172.21.15.69:0/3913167641 >> v1:172.21.15.121:6789/0 conn(0x7f605415b490 legacy=0x7f60540b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.546+0000 7f604b400640 1 -- 172.21.15.69:0/3913167641 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60540d0960 con 0x7f605406f2d0 2024-04-24T08:08:35.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.546+0000 7f6058a00640 1 --2- 172.21.15.69:0/3913167641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 0x7f60540d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:35.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.547+0000 7f604b400640 1 --2- 172.21.15.69:0/3913167641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f605406f2d0 0x7f605406f730 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f603c009870 tx=0x7f603c02a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1ad89de89ba6f71 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:35.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f6053400640 1 -- 172.21.15.69:0/3913167641 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c037070 con 0x7f605406f2d0 2024-04-24T08:08:35.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f6053400640 1 -- 172.21.15.69:0/3913167641 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f603c032440 con 0x7f605406f2d0 2024-04-24T08:08:35.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f6053400640 1 -- 172.21.15.69:0/3913167641 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c032810 con 0x7f605406f2d0 2024-04-24T08:08:35.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f605aa00640 1 -- 172.21.15.69:0/3913167641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f605406f2d0 msgr2=0x7f605406f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f605aa00640 1 --2- 172.21.15.69:0/3913167641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f605406f2d0 0x7f605406f730 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f603c009870 tx=0x7f603c02a250 comp rx=0 tx=0).stop 2024-04-24T08:08:35.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f605aa00640 1 -- 172.21.15.69:0/3913167641 shutdown_connections 2024-04-24T08:08:35.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f605aa00640 1 --2- 172.21.15.69:0/3913167641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 0x7f60540d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f605aa00640 1 --2- 172.21.15.69:0/3913167641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f605406f2d0 0x7f605406f730 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f605aa00640 1 -- 172.21.15.69:0/3913167641 >> 172.21.15.69:0/3913167641 conn(0x7f6054067d90 msgr2=0x7f605406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:35.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f605aa00640 1 -- 172.21.15.69:0/3913167641 shutdown_connections 2024-04-24T08:08:35.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.548+0000 7f605aa00640 1 -- 172.21.15.69:0/3913167641 wait complete. 2024-04-24T08:08:35.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.550+0000 7f605aa00640 1 Processor -- start 2024-04-24T08:08:35.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.550+0000 7f605aa00640 1 -- start start 2024-04-24T08:08:35.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.551+0000 7f605aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f605406f2d0 0x7f605414cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.551+0000 7f605aa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 0x7f60540ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.551+0000 7f605aa00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f605415b490 0x7f60540ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.551+0000 7f605aa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60540d0c40 con 0x7f605406f2d0 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.551+0000 7f605aa00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f60540b4c60 con 0x7f605415b490 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.551+0000 7f605aa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f605415b870 con 0x7f60540cfbe0 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.551+0000 7f6053e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f605406f2d0 0x7f605414cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6053e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f605406f2d0 0x7f605414cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44648/0 (socket says 172.21.15.69:44648) 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6053e00640 1 -- 172.21.15.69:0/2343544531 learned_addr learned my addr 172.21.15.69:0/2343544531 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f604b400640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 0x7f60540ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6058a00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f605415b490 0x7f60540ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6053e00640 1 -- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 msgr2=0x7f60540ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6053e00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 0x7f60540ae090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6053e00640 1 -- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f605415b490 msgr2=0x7f60540ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6053e00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f605415b490 0x7f60540ae5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6053e00640 1 -- 172.21.15.69:0/2343544531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605415c900 con 0x7f605406f2d0 2024-04-24T08:08:35.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6058a00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f605415b490 0x7f60540ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:35.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6053e00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f605406f2d0 0x7f605414cdf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f604400c480 tx=0x7f604400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:35.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.552+0000 7f6050c00640 1 -- 172.21.15.69:0/2343544531 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60440052d0 con 0x7f605406f2d0 2024-04-24T08:08:35.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.553+0000 7f6050c00640 1 -- 172.21.15.69:0/2343544531 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6044005470 con 0x7f605406f2d0 2024-04-24T08:08:35.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.553+0000 7f605aa00640 1 -- 172.21.15.69:0/2343544531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605415d3c0 con 0x7f605406f2d0 2024-04-24T08:08:35.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.553+0000 7f6050c00640 1 -- 172.21.15.69:0/2343544531 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6044012870 con 0x7f605406f2d0 2024-04-24T08:08:35.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.553+0000 7f605aa00640 1 -- 172.21.15.69:0/2343544531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f605415d980 con 0x7f605406f2d0 2024-04-24T08:08:35.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.554+0000 7f605aa00640 1 -- 172.21.15.69:0/2343544531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f605415d1d0 con 0x7f605406f2d0 2024-04-24T08:08:35.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.555+0000 7f6050c00640 1 -- 172.21.15.69:0/2343544531 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6044012a10 con 0x7f605406f2d0 2024-04-24T08:08:35.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.556+0000 7f6050c00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f603006c8f0 0x7f603006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.556+0000 7f6050c00640 1 -- 172.21.15.69:0/2343544531 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f604408b890 con 0x7f605406f2d0 2024-04-24T08:08:35.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.556+0000 7f604b400640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f603006c8f0 0x7f603006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.557+0000 7f604b400640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f603006c8f0 0x7f603006edb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f603c034000 tx=0x7f603c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:35.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.561+0000 7f6050c00640 1 -- 172.21.15.69:0/2343544531 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6044059cb0 con 0x7f605406f2d0 2024-04-24T08:08:35.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.761+0000 7f605aa00640 1 -- 172.21.15.69:0/2343544531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6054072c60 con 0x7f605406f2d0 2024-04-24T08:08:35.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.764+0000 7f6050c00640 1 -- 172.21.15.69:0/2343544531 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7f60440595b0 con 0x7f605406f2d0 2024-04-24T08:08:35.766 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:35.766 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:35.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.766+0000 7f605aa00640 1 -- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f603006c8f0 msgr2=0x7f603006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.766+0000 7f605aa00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f603006c8f0 0x7f603006edb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f603c034000 tx=0x7f603c02e000 comp rx=0 tx=0).stop 2024-04-24T08:08:35.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.766+0000 7f605aa00640 1 -- 172.21.15.69:0/2343544531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f605406f2d0 msgr2=0x7f605414cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.766+0000 7f605aa00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f605406f2d0 0x7f605414cdf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f604400c480 tx=0x7f604400c940 comp rx=0 tx=0).stop 2024-04-24T08:08:35.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.766+0000 7f605aa00640 1 -- 172.21.15.69:0/2343544531 shutdown_connections 2024-04-24T08:08:35.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.766+0000 7f605aa00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f603006c8f0 0x7f603006edb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.766+0000 7f605aa00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f605415b490 0x7f60540ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.766+0000 7f605aa00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60540cfbe0 0x7f60540ae090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.766+0000 7f605aa00640 1 --2- 172.21.15.69:0/2343544531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f605406f2d0 0x7f605414cdf0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.766+0000 7f605aa00640 1 -- 172.21.15.69:0/2343544531 >> 172.21.15.69:0/2343544531 conn(0x7f6054067d90 msgr2=0x7f605406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.767+0000 7f605aa00640 1 -- 172.21.15.69:0/2343544531 shutdown_connections 2024-04-24T08:08:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.767+0000 7f605aa00640 1 -- 172.21.15.69:0/2343544531 wait complete. 2024-04-24T08:08:35.785 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:08:35.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.980+0000 7f0486400640 1 Processor -- start 2024-04-24T08:08:35.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.980+0000 7f0486400640 1 -- start start 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f0486400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04800d0300 0x7f048006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f0486400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0480145a90 0x7f04800d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f0486400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0480145fe0 0x7f0480160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f0486400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f04800d0c20 con 0x7f0480145fe0 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f0486400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f04800cf3d0 con 0x7f0480145a90 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f0486400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04800cf6b0 con 0x7f04800d0300 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f047fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0480145fe0 0x7f0480160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f047fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0480145fe0 0x7f0480160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44664/0 (socket says 172.21.15.69:44664) 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f047fe00640 1 -- 172.21.15.69:0/4154293103 learned_addr learned my addr 172.21.15.69:0/4154293103 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f047f400640 1 --2- 172.21.15.69:0/4154293103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04800d0300 0x7f048006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.982+0000 7f047ea00640 1 --2- 172.21.15.69:0/4154293103 >> v2:172.21.15.121:3300/0 conn(0x7f0480145a90 0x7f04800d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.983+0000 7f047fe00640 1 -- 172.21.15.69:0/4154293103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04800d0300 msgr2=0x7f048006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:35.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.983+0000 7f047fe00640 1 --2- 172.21.15.69:0/4154293103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04800d0300 0x7f048006e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.983+0000 7f047fe00640 1 -- 172.21.15.69:0/4154293103 >> v2:172.21.15.121:3300/0 conn(0x7f0480145a90 msgr2=0x7f04800d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:35.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.983+0000 7f047fe00640 1 --2- 172.21.15.69:0/4154293103 >> v2:172.21.15.121:3300/0 conn(0x7f0480145a90 0x7f04800d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.983+0000 7f047fe00640 1 -- 172.21.15.69:0/4154293103 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04800cf990 con 0x7f0480145fe0 2024-04-24T08:08:35.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.984+0000 7f047fe00640 1 --2- 172.21.15.69:0/4154293103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0480145fe0 0x7f0480160540 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0480073210 tx=0x7f047802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1de76be81153654 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:35.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.984+0000 7f047e000640 1 -- 172.21.15.69:0/4154293103 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0478036070 con 0x7f0480145fe0 2024-04-24T08:08:35.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.984+0000 7f047e000640 1 -- 172.21.15.69:0/4154293103 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0478031600 con 0x7f0480145fe0 2024-04-24T08:08:35.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.984+0000 7f047e000640 1 -- 172.21.15.69:0/4154293103 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04780319d0 con 0x7f0480145fe0 2024-04-24T08:08:35.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.985+0000 7f0486400640 1 -- 172.21.15.69:0/4154293103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0480145fe0 msgr2=0x7f0480160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.985+0000 7f0486400640 1 --2- 172.21.15.69:0/4154293103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0480145fe0 0x7f0480160540 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0480073210 tx=0x7f047802a6a0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.985+0000 7f0486400640 1 -- 172.21.15.69:0/4154293103 shutdown_connections 2024-04-24T08:08:35.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.985+0000 7f0486400640 1 --2- 172.21.15.69:0/4154293103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0480145fe0 0x7f0480160540 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.985+0000 7f0486400640 1 --2- 172.21.15.69:0/4154293103 >> v2:172.21.15.121:3300/0 conn(0x7f0480145a90 0x7f04800d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.985+0000 7f0486400640 1 --2- 172.21.15.69:0/4154293103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04800d0300 0x7f048006e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.985+0000 7f0486400640 1 -- 172.21.15.69:0/4154293103 >> 172.21.15.69:0/4154293103 conn(0x7f0480067d90 msgr2=0x7f048006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:35.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.985+0000 7f0486400640 1 -- 172.21.15.69:0/4154293103 shutdown_connections 2024-04-24T08:08:35.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.985+0000 7f0486400640 1 -- 172.21.15.69:0/4154293103 wait complete. 2024-04-24T08:08:35.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.987+0000 7f0486400640 1 Processor -- start 2024-04-24T08:08:35.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.987+0000 7f0486400640 1 -- start start 2024-04-24T08:08:35.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.988+0000 7f0486400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04800d0300 0x7f048013f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.988+0000 7f0486400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0480145a90 0x7f04800ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.988+0000 7f0486400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0480145fe0 0x7f04800ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.988+0000 7f0486400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0480145770 con 0x7f04800d0300 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.988+0000 7f0486400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04800cff50 con 0x7f0480145fe0 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.988+0000 7f0486400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04801458f0 con 0x7f0480145a90 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04800d0300 0x7f048013f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04800d0300 0x7f048013f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44668/0 (socket says 172.21.15.69:44668) 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047f400640 1 -- 172.21.15.69:0/3259140350 learned_addr learned my addr 172.21.15.69:0/3259140350 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047ea00640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0480145a90 0x7f04800ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047f400640 1 -- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0480145a90 msgr2=0x7f04800ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047f400640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0480145a90 0x7f04800ae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047fe00640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0480145fe0 0x7f04800ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047f400640 1 -- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0480145fe0 msgr2=0x7f04800ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047f400640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0480145fe0 0x7f04800ab8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:35.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047f400640 1 -- 172.21.15.69:0/3259140350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04800aea90 con 0x7f04800d0300 2024-04-24T08:08:35.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.989+0000 7f047f400640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04800d0300 0x7f048013f030 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f047000d800 tx=0x7f047000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:35.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.990+0000 7f0477400640 1 -- 172.21.15.69:0/3259140350 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470018030 con 0x7f04800d0300 2024-04-24T08:08:35.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.990+0000 7f0477400640 1 -- 172.21.15.69:0/3259140350 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04700096a0 con 0x7f04800d0300 2024-04-24T08:08:35.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.990+0000 7f0477400640 1 -- 172.21.15.69:0/3259140350 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470009870 con 0x7f04800d0300 2024-04-24T08:08:35.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.990+0000 7f0486400640 1 -- 172.21.15.69:0/3259140350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04800aed70 con 0x7f04800d0300 2024-04-24T08:08:35.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.990+0000 7f0486400640 1 -- 172.21.15.69:0/3259140350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0480142900 con 0x7f04800d0300 2024-04-24T08:08:35.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.991+0000 7f0486400640 1 -- 172.21.15.69:0/3259140350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04800af290 con 0x7f04800d0300 2024-04-24T08:08:35.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.992+0000 7f0477400640 1 -- 172.21.15.69:0/3259140350 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f047001b070 con 0x7f04800d0300 2024-04-24T08:08:35.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.993+0000 7f0477400640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f046006c8f0 0x7f046006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:35.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.993+0000 7f0477400640 1 -- 172.21.15.69:0/3259140350 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f047008be80 con 0x7f04800d0300 2024-04-24T08:08:35.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.993+0000 7f047ea00640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f046006c8f0 0x7f046006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:35.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.994+0000 7f047ea00640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f046006c8f0 0x7f046006edb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f046c006fd0 tx=0x7f046c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:36.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:35.999+0000 7f0477400640 1 -- 172.21.15.69:0/3259140350 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f047005a210 con 0x7f04800d0300 2024-04-24T08:08:36.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.244+0000 7f0486400640 1 -- 172.21.15.69:0/3259140350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0480068a80 con 0x7f04800d0300 2024-04-24T08:08:36.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.249+0000 7f0477400640 1 -- 172.21.15.69:0/3259140350 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f0470059b10 con 0x7f04800d0300 2024-04-24T08:08:36.251 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:36.251 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/934340733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":934340733},{"type":"v1","addr":"172.21.15.121:6835","nonce":934340733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3140867113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3140867113},{"type":"v1","addr":"172.21.15.121:6837","nonce":3140867113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/1059995973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1059995973},{"type":"v1","addr":"172.21.15.69:6833","nonce":1059995973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:08:19.574083+0000","modified":"2024-04-24T08:08:22.605668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/1003206529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1003206529},{"type":"v1","addr":"172.21.15.69:6835","nonce":1003206529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-24T08:08:36.251 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 7 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 -- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f046006c8f0 msgr2=0x7f046006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f046006c8f0 0x7f046006edb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f046c006fd0 tx=0x7f046c008040 comp rx=0 tx=0).stop 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 -- 172.21.15.69:0/3259140350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04800d0300 msgr2=0x7f048013f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04800d0300 0x7f048013f030 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f047000d800 tx=0x7f047000dcc0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 -- 172.21.15.69:0/3259140350 shutdown_connections 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f046006c8f0 0x7f046006edb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0480145fe0 0x7f04800ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0480145a90 0x7f04800ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 --2- 172.21.15.69:0/3259140350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04800d0300 0x7f048013f030 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 -- 172.21.15.69:0/3259140350 >> 172.21.15.69:0/3259140350 conn(0x7f0480067d90 msgr2=0x7f048006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.252+0000 7f0486400640 1 -- 172.21.15.69:0/3259140350 shutdown_connections 2024-04-24T08:08:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.253+0000 7f0486400640 1 -- 172.21.15.69:0/3259140350 wait complete. 2024-04-24T08:08:36.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:08:19.574083+0000', 'modified': '2024-04-24T08:08:22.605668+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4257}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4257': {'gid': 4257, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.69:6835/1003206529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 1003206529}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 1003206529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4257, 'qdb_cluster': [4257]} 2024-04-24T08:08:36.674 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:08:36.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-24T08:08:36.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.892+0000 7f79cfa00640 1 Processor -- start 2024-04-24T08:08:36.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.892+0000 7f79cfa00640 1 -- start start 2024-04-24T08:08:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.893+0000 7f79cfa00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f79c80b3d30 0x7f79c80b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.893+0000 7f79cfa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79c806e0f0 0x7f79c806e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.893+0000 7f79cfa00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79c81669d0 con 0x7f79c8162c70 2024-04-24T08:08:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.893+0000 7f79cfa00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f79c8166cb0 con 0x7f79c80b3d30 2024-04-24T08:08:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.893+0000 7f79cfa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f79c8166f90 con 0x7f79c806e0f0 2024-04-24T08:08:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.894+0000 7f79cd000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f79c8162c70 0x7f79c80b3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36660/0 (socket says 172.21.15.69:36660) 2024-04-24T08:08:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.894+0000 7f79cd000640 1 -- 172.21.15.69:0/316502465 learned_addr learned my addr 172.21.15.69:0/316502465 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:36.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.894+0000 7f79cda00640 1 --2- 172.21.15.69:0/316502465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79c806e0f0 0x7f79c806e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.894+0000 7f79bfe00640 1 --2- 172.21.15.69:0/316502465 >> v2:172.21.15.121:3300/0 conn(0x7f79c80b3d30 0x7f79c80b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.894+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1908413310 0 0) 0x7f79c81669d0 con 0x7f79c8162c70 2024-04-24T08:08:36.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.894+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79ac003520 con 0x7f79c8162c70 2024-04-24T08:08:36.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1025833549 0 0) 0x7f79ac003520 con 0x7f79c8162c70 2024-04-24T08:08:36.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79c806e0f0 msgr2=0x7f79c806e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 --2- 172.21.15.69:0/316502465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79c806e0f0 0x7f79c806e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 >> v2:172.21.15.121:3300/0 conn(0x7f79c80b3d30 msgr2=0x7f79c80b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 --2- 172.21.15.69:0/316502465 >> v2:172.21.15.121:3300/0 conn(0x7f79c80b3d30 0x7f79c80b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c8167270 con 0x7f79c8162c70 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f79b8001670 con 0x7f79c8162c70 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 >> v1:172.21.15.69:6789/0 conn(0x7f79c8162c70 legacy=0x7f79c80b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 --2- 172.21.15.69:0/316502465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac0039b0 0x7f79ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 --2- 172.21.15.69:0/316502465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79ac004800 0x7f79ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 --2- 172.21.15.69:0/316502465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79c806e0f0 0x7f79ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8166cb0 con 0x7f79ac0039b0 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8166f90 con 0x7f79ac004800 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f79ac00b9c0 con 0x7f79c806e0f0 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79cd000640 1 --2- 172.21.15.69:0/316502465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac0039b0 0x7f79ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79cd000640 1 -- 172.21.15.69:0/316502465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79c806e0f0 msgr2=0x7f79ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79cd000640 1 --2- 172.21.15.69:0/316502465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79c806e0f0 0x7f79ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79cd000640 1 -- 172.21.15.69:0/316502465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79ac004800 msgr2=0x7f79ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79cd000640 1 --2- 172.21.15.69:0/316502465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79ac004800 0x7f79ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.895+0000 7f79cd000640 1 -- 172.21.15.69:0/316502465 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac003520 con 0x7f79ac0039b0 2024-04-24T08:08:36.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.896+0000 7f79cd000640 1 --2- 172.21.15.69:0/316502465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac0039b0 0x7f79ac003dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f79b800c240 tx=0x7f79ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8d64cbd173e4e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:36.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.897+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b8007d60 con 0x7f79ac0039b0 2024-04-24T08:08:36.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.897+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b8005af0 con 0x7f79ac0039b0 2024-04-24T08:08:36.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.897+0000 7f79c7e00640 1 -- 172.21.15.69:0/316502465 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b8007070 con 0x7f79ac0039b0 2024-04-24T08:08:36.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.897+0000 7f79cfa00640 1 -- 172.21.15.69:0/316502465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac0039b0 msgr2=0x7f79ac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.897+0000 7f79cfa00640 1 --2- 172.21.15.69:0/316502465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79ac0039b0 0x7f79ac003dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f79b800c240 tx=0x7f79ac005830 comp rx=0 tx=0).stop 2024-04-24T08:08:36.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.897+0000 7f79cda00640 1 -- 172.21.15.69:0/316502465 reap_dead start 2024-04-24T08:08:36.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.897+0000 7f79cfa00640 1 -- 172.21.15.69:0/316502465 shutdown_connections 2024-04-24T08:08:36.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.897+0000 7f79cfa00640 1 -- 172.21.15.69:0/316502465 >> 172.21.15.69:0/316502465 conn(0x7f79c8067d90 msgr2=0x7f79c806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:36.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.897+0000 7f79cfa00640 1 -- 172.21.15.69:0/316502465 shutdown_connections 2024-04-24T08:08:36.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.897+0000 7f79cfa00640 1 -- 172.21.15.69:0/316502465 wait complete. 2024-04-24T08:08:36.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.899+0000 7f79cfa00640 1 Processor -- start 2024-04-24T08:08:36.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.899+0000 7f79cfa00640 1 -- start start 2024-04-24T08:08:36.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.900+0000 7f79cfa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79ac0039b0 0x7f79c81f8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.900+0000 7f79cfa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c8163be0 0x7f79c8164040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.900+0000 7f79cfa00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79c815dd30 0x7f79c815e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.900+0000 7f79cfa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f79c80cfc40 con 0x7f79c8163be0 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.900+0000 7f79cfa00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8163050 con 0x7f79c815dd30 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.900+0000 7f79cfa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f79c80cfdc0 con 0x7f79ac0039b0 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79bfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c8163be0 0x7f79c8164040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79bfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c8163be0 0x7f79c8164040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44676/0 (socket says 172.21.15.69:44676) 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79bfe00640 1 -- 172.21.15.69:0/3667402560 learned_addr learned my addr 172.21.15.69:0/3667402560 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79cd000640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79ac0039b0 0x7f79c81f8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79cda00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79c815dd30 0x7f79c815e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79bfe00640 1 -- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79ac0039b0 msgr2=0x7f79c81f8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79bfe00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79ac0039b0 0x7f79c81f8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79bfe00640 1 -- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79c815dd30 msgr2=0x7f79c815e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79bfe00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79c815dd30 0x7f79c815e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79bfe00640 1 -- 172.21.15.69:0/3667402560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c8164730 con 0x7f79c8163be0 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.901+0000 7f79cda00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79c815dd30 0x7f79c815e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:36.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.902+0000 7f79bfe00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c8163be0 0x7f79c8164040 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f79b000e7d0 tx=0x7f79b000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:36.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.902+0000 7f79c5600640 1 -- 172.21.15.69:0/3667402560 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0017070 con 0x7f79c8163be0 2024-04-24T08:08:36.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.902+0000 7f79c5600640 1 -- 172.21.15.69:0/3667402560 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b000c940 con 0x7f79c8163be0 2024-04-24T08:08:36.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.902+0000 7f79c5600640 1 -- 172.21.15.69:0/3667402560 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b000cd10 con 0x7f79c8163be0 2024-04-24T08:08:36.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.902+0000 7f79cfa00640 1 -- 172.21.15.69:0/3667402560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c8164a10 con 0x7f79c8163be0 2024-04-24T08:08:36.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.902+0000 7f79cfa00640 1 -- 172.21.15.69:0/3667402560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c8164fd0 con 0x7f79c8163be0 2024-04-24T08:08:36.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.904+0000 7f79c5600640 1 -- 172.21.15.69:0/3667402560 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f79b001a070 con 0x7f79c8163be0 2024-04-24T08:08:36.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.905+0000 7f79c5600640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79a406c920 0x7f79a406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.905+0000 7f79c5600640 1 -- 172.21.15.69:0/3667402560 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f79b008bde0 con 0x7f79c8163be0 2024-04-24T08:08:36.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.905+0000 7f79cd000640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79a406c920 0x7f79a406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.906+0000 7f79d2770740 1 Processor -- start 2024-04-24T08:08:36.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.906+0000 7f79d2770740 1 -- start start 2024-04-24T08:08:36.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79d2770740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a187f7b0 0x55d6a1832140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79d2770740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a18628d0 0x55d6a1861c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79d2770740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a1862190 0x55d6a1836fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79cd000640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79a406c920 0x7f79a406ede0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f79b8033000 tx=0x7f79b8009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:36.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79d2770740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d6a179cb10 con 0x55d6a18628d0 2024-04-24T08:08:36.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79d2770740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d6a17432b0 con 0x55d6a187f7b0 2024-04-24T08:08:36.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79d2770740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55d6a1798ad0 con 0x55d6a1862190 2024-04-24T08:08:36.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79bfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a18628d0 0x55d6a1861c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79bfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a18628d0 0x55d6a1861c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44688/0 (socket says 172.21.15.69:44688) 2024-04-24T08:08:36.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79bfe00640 1 -- 172.21.15.69:0/2881085459 learned_addr learned my addr 172.21.15.69:0/2881085459 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79cda00640 1 --2- 172.21.15.69:0/2881085459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a1862190 0x55d6a1836fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79bfe00640 1 -- 172.21.15.69:0/2881085459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a1862190 msgr2=0x55d6a1836fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79bfe00640 1 --2- 172.21.15.69:0/2881085459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a1862190 0x55d6a1836fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79bfe00640 1 -- 172.21.15.69:0/2881085459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a187f7b0 msgr2=0x55d6a1832140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79cd000640 1 --2- 172.21.15.69:0/2881085459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a187f7b0 0x55d6a1832140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79bfe00640 1 --2- 172.21.15.69:0/2881085459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a187f7b0 0x55d6a1832140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79bfe00640 1 -- 172.21.15.69:0/2881085459 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6a14af2d0 con 0x55d6a18628d0 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.907+0000 7f79bfe00640 1 --2- 172.21.15.69:0/2881085459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a18628d0 0x55d6a1861c50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f79b008a1e0 tx=0x7f79b0059400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a96869b9666d526 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79bd600640 1 -- 172.21.15.69:0/2881085459 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0021040 con 0x55d6a18628d0 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79bd600640 1 -- 172.21.15.69:0/2881085459 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b005dd90 con 0x55d6a18628d0 2024-04-24T08:08:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79bd600640 1 -- 172.21.15.69:0/2881085459 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0062650 con 0x55d6a18628d0 2024-04-24T08:08:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79d2770740 1 -- 172.21.15.69:0/2881085459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a18628d0 msgr2=0x55d6a1861c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79d2770740 1 --2- 172.21.15.69:0/2881085459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a18628d0 0x55d6a1861c50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f79b008a1e0 tx=0x7f79b0059400 comp rx=0 tx=0).stop 2024-04-24T08:08:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79d2770740 1 -- 172.21.15.69:0/2881085459 shutdown_connections 2024-04-24T08:08:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79d2770740 1 --2- 172.21.15.69:0/2881085459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a18628d0 0x55d6a1861c50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79d2770740 1 --2- 172.21.15.69:0/2881085459 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a1862190 0x55d6a1836fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79d2770740 1 --2- 172.21.15.69:0/2881085459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a187f7b0 0x55d6a1832140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79d2770740 1 -- 172.21.15.69:0/2881085459 >> 172.21.15.69:0/2881085459 conn(0x55d6a18343b0 msgr2=0x55d6a187cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.908+0000 7f79d2770740 1 -- 172.21.15.69:0/2881085459 shutdown_connections 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.909+0000 7f79d2770740 1 -- 172.21.15.69:0/2881085459 wait complete. 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.909+0000 7f79d2770740 1 -- start start 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.909+0000 7f79d2770740 1 Processor -- start 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.910+0000 7f79d2770740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a185d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.910+0000 7f79d2770740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 0x55d6a188cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.910+0000 7f79d2770740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a1863d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.910+0000 7f79d2770740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d6a1836560 con 0x55d6a187e600 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.910+0000 7f79d2770740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d6a1756590 con 0x55d6a18628d0 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.910+0000 7f79d2770740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55d6a1835950 con 0x55d6a187f7b0 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.910+0000 7f79d2770740 10 client.0 _ll_register_callbacks cb 0x55d6a17393a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a1863d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a1863d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44694/0 (socket says 172.21.15.69:44694) 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cda00640 1 -- 172.21.15.69:0/4167504882 learned_addr learned my addr 172.21.15.69:0/4167504882 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cd000640 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a185d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79bfe00640 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 0x55d6a188cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cda00640 1 -- 172.21.15.69:0/4167504882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 msgr2=0x55d6a188cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cda00640 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 0x55d6a188cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cda00640 1 -- 172.21.15.69:0/4167504882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 msgr2=0x55d6a185d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cda00640 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a185d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cda00640 1 -- 172.21.15.69:0/4167504882 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6a15796b0 con 0x55d6a187e600 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cd000640 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a185d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79cda00640 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a1863d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f79c80cc800 tx=0x7f79c0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f371c0d194b2134 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.911+0000 7f79b6000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79b6000640 1 -- 172.21.15.69:0/4167504882 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0009e60 con 0x55d6a187e600 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79b6000640 1 -- 172.21.15.69:0/4167504882 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f79c0010d30 con 0x55d6a187e600 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79d2770740 1 -- 172.21.15.69:0/4167504882 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79b005dd90 con 0x55d6a187e600 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79b6000640 1 -- 172.21.15.69:0/4167504882 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f79c0024050 con 0x55d6a187e600 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79d2770740 10 client.4659 fetch_fsmap learned FSMap version 7 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79d2770740 1 -- 172.21.15.69:0/4167504882 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55d6a18637e0 con 0x55d6a187e600 2024-04-24T08:08:36.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79b6000640 1 -- 172.21.15.69:0/4167504882 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b005dd90 con 0x55d6a187e600 2024-04-24T08:08:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79b6000640 1 -- 172.21.15.69:0/4167504882 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 7) ==== 3494+0+0 (secure 0 0 0) 0x7f79c0027020 con 0x55d6a187e600 2024-04-24T08:08:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79d2770740 10 client.4659 fetch_fsmap finished waiting for FSMap version 7 2024-04-24T08:08:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79d2770740 10 client.4659 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:08:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79d2770740 20 client.4659 populate_metadata read hostname 'smithi069' 2024-04-24T08:08:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79d2770740 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.69:6834/1003206529,v1:172.21.15.69:6835/1003206529] conn(0x55d6a187bd00 0x55d6a187d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79d2770740 4 client.4659 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:08:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.912+0000 7f79d2770740 1 -- 172.21.15.69:0/4167504882 --> [v2:172.21.15.69:6834/1003206529,v1:172.21.15.69:6835/1003206529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6a17aa530 con 0x55d6a187bd00 2024-04-24T08:08:36.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.913+0000 7f79cd000640 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.69:6834/1003206529,v1:172.21.15.69:6835/1003206529] conn(0x55d6a187bd00 0x55d6a187d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.913+0000 7f79cd000640 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.69:6834/1003206529,v1:172.21.15.69:6835/1003206529] conn(0x55d6a187bd00 0x55d6a187d690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d94ec4b61f7a261 server_cookie=8e5f61abf68fdf0 in_seq=0 out_seq=0 2024-04-24T08:08:36.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.913+0000 7f79b6000640 10 client.4659 ms_handle_connect on v2:172.21.15.69:6834/1003206529 2024-04-24T08:08:36.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79b6000640 1 -- 172.21.15.69:0/4167504882 <== mds.0 v2:172.21.15.69:6834/1003206529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d6a17aa530 con 0x55d6a187bd00 2024-04-24T08:08:36.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79b6000640 10 client.4659 handle_command_reply: tid=0 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79d2770740 1 client.4659 shutdown 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79d2770740 1 -- 172.21.15.69:0/4167504882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 msgr2=0x55d6a1863d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79d2770740 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a1863d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f79c80cc800 tx=0x7f79c0009750 comp rx=0 tx=0).stop 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79d2770740 1 -- 172.21.15.69:0/4167504882 shutdown_connections 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79d2770740 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.69:6834/1003206529,v1:172.21.15.69:6835/1003206529] conn(0x55d6a187bd00 0x55d6a187d690 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79d2770740 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a1863d00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79d2770740 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 0x55d6a188cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79d2770740 1 --2- 172.21.15.69:0/4167504882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a185d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.914+0000 7f79d2770740 1 -- 172.21.15.69:0/4167504882 >> 172.21.15.69:0/4167504882 conn(0x55d6a1862190 msgr2=0x55d6a18318c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.915+0000 7f79b6000640 0 client.4659 ms_handle_reset on v2:172.21.15.69:6834/1003206529 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.915+0000 7f79d2770740 1 -- 172.21.15.69:0/4167504882 shutdown_connections 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.915+0000 7f79d2770740 1 -- 172.21.15.69:0/4167504882 wait complete. 2024-04-24T08:08:36.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.915+0000 7f79d2770740 20 client.4659 trim_cache size 0 max 16384 2024-04-24T08:08:36.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.934+0000 7f79d2770740 1 Processor -- start 2024-04-24T08:08:36.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.934+0000 7f79d2770740 1 -- start start 2024-04-24T08:08:36.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.935+0000 7f79d2770740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a18ed9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.935+0000 7f79d2770740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18786f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.935+0000 7f79d2770740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 0x55d6a1878c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.935+0000 7f79d2770740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8204a60 con 0x55d6a187e600 2024-04-24T08:08:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.935+0000 7f79d2770740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f79c8143fa0 con 0x55d6a18628d0 2024-04-24T08:08:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.935+0000 7f79d2770740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f79c8204410 con 0x55d6a187f7b0 2024-04-24T08:08:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.935+0000 7f79bfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18786f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.935+0000 7f79bfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18786f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44700/0 (socket says 172.21.15.69:44700) 2024-04-24T08:08:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.935+0000 7f79bfe00640 1 -- 172.21.15.69:0/1672683819 learned_addr learned my addr 172.21.15.69:0/1672683819 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.936+0000 7f79bfe00640 1 -- 172.21.15.69:0/1672683819 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 msgr2=0x55d6a1878c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.936+0000 7f79bfe00640 1 --2- 172.21.15.69:0/1672683819 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 0x55d6a1878c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.936+0000 7f79bfe00640 1 -- 172.21.15.69:0/1672683819 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 msgr2=0x55d6a18ed9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.936+0000 7f79bfe00640 1 --2- 172.21.15.69:0/1672683819 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a18ed9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.936+0000 7f79bfe00640 1 -- 172.21.15.69:0/1672683819 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d6a187a060 con 0x55d6a187e600 2024-04-24T08:08:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.936+0000 7f79bfe00640 1 --2- 172.21.15.69:0/1672683819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18786f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f79b0000c00 tx=0x7f79b001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4eeb992560079d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.936+0000 7f79bcc00640 1 -- 172.21.15.69:0/1672683819 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0021040 con 0x55d6a187e600 2024-04-24T08:08:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.936+0000 7f79bcc00640 1 -- 172.21.15.69:0/1672683819 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b005dd90 con 0x55d6a187e600 2024-04-24T08:08:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.936+0000 7f79bcc00640 1 -- 172.21.15.69:0/1672683819 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0079080 con 0x55d6a187e600 2024-04-24T08:08:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.937+0000 7f79d2770740 1 -- 172.21.15.69:0/1672683819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 msgr2=0x55d6a18786f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.937+0000 7f79d2770740 1 --2- 172.21.15.69:0/1672683819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18786f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f79b0000c00 tx=0x7f79b001e000 comp rx=0 tx=0).stop 2024-04-24T08:08:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.937+0000 7f79d2770740 1 -- 172.21.15.69:0/1672683819 shutdown_connections 2024-04-24T08:08:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.937+0000 7f79d2770740 1 --2- 172.21.15.69:0/1672683819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18786f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.937+0000 7f79d2770740 1 --2- 172.21.15.69:0/1672683819 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 0x55d6a1878c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.937+0000 7f79d2770740 1 --2- 172.21.15.69:0/1672683819 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a18ed9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.937+0000 7f79d2770740 1 -- 172.21.15.69:0/1672683819 >> 172.21.15.69:0/1672683819 conn(0x55d6a1862190 msgr2=0x55d6a1837e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.937+0000 7f79d2770740 1 -- 172.21.15.69:0/1672683819 shutdown_connections 2024-04-24T08:08:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.937+0000 7f79d2770740 1 -- 172.21.15.69:0/1672683819 wait complete. 2024-04-24T08:08:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.938+0000 7f79d2770740 1 -- start start 2024-04-24T08:08:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.938+0000 7f79d2770740 1 Processor -- start 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79d2770740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a18da8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79d2770740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18725e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79d2770740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 0x55d6a1872b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79d2770740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d6a187bb80 con 0x55d6a187e600 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79d2770740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d6a188d480 con 0x55d6a18628d0 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79d2770740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55d6a187c1d0 con 0x55d6a187f7b0 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79d2770740 10 client.0 _ll_register_callbacks cb 0x7f79c0009b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79bfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79bfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44710/0 (socket says 172.21.15.69:44710) 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79bfe00640 1 -- 172.21.15.69:0/3643584800 learned_addr learned my addr 172.21.15.69:0/3643584800 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79bfe00640 1 -- 172.21.15.69:0/3643584800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 msgr2=0x55d6a1872b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79bfe00640 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 0x55d6a1872b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79bfe00640 1 -- 172.21.15.69:0/3643584800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 msgr2=0x55d6a18da8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79cd000640 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a18da8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.939+0000 7f79bfe00640 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a18da8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79bfe00640 1 -- 172.21.15.69:0/3643584800 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6a18ee3f0 con 0x55d6a187e600 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79bfe00640 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18725e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f79b0000c00 tx=0x7f79b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86881756074a1666 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79b6000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79b6000640 1 -- 172.21.15.69:0/3643584800 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b0082060 con 0x55d6a187e600 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79b6000640 1 -- 172.21.15.69:0/3643584800 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f79b0083de0 con 0x55d6a187e600 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79d2770740 1 -- 172.21.15.69:0/3643584800 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79b005dd90 con 0x55d6a187e600 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79b6000640 1 -- 172.21.15.69:0/3643584800 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f79b0021070 con 0x55d6a187e600 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79d2770740 10 client.4665 fetch_fsmap learned FSMap version 7 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79d2770740 1 -- 172.21.15.69:0/3643584800 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55d6a1896c10 con 0x55d6a187e600 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79b6000640 1 -- 172.21.15.69:0/3643584800 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b005dd90 con 0x55d6a187e600 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79b6000640 1 -- 172.21.15.69:0/3643584800 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 7) ==== 3494+0+0 (secure 0 0 0) 0x7f79b0058e30 con 0x55d6a187e600 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79d2770740 10 client.4665 fetch_fsmap finished waiting for FSMap version 7 2024-04-24T08:08:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79d2770740 10 client.4665 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.940+0000 7f79d2770740 20 client.4665 populate_metadata read hostname 'smithi069' 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.941+0000 7f79d2770740 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.69:6834/1003206529,v1:172.21.15.69:6835/1003206529] conn(0x55d6a1864c40 0x55d6a1865020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.941+0000 7f79d2770740 4 client.4665 mds_command: new command op to 4257 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.941+0000 7f79d2770740 1 -- 172.21.15.69:0/3643584800 --> [v2:172.21.15.69:6834/1003206529,v1:172.21.15.69:6835/1003206529] -- command(tid 0: {"prefix": "status"}) -- 0x55d6a17a43a0 con 0x55d6a1864c40 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.941+0000 7f79cd000640 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.69:6834/1003206529,v1:172.21.15.69:6835/1003206529] conn(0x55d6a1864c40 0x55d6a1865020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.941+0000 7f79b6000640 10 client.4665 ms_handle_connect on v2:172.21.15.69:6834/1003206529 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.941+0000 7f79cd000640 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.69:6834/1003206529,v1:172.21.15.69:6835/1003206529] conn(0x55d6a1864c40 0x55d6a1865020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff780035c0cf89cb server_cookie=a04dfe3c3fa6e2f7 in_seq=0 out_seq=0 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.941+0000 7f79b6000640 1 -- 172.21.15.69:0/3643584800 <== mds.0 v2:172.21.15.69:6834/1003206529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55d6a17a43a0 con 0x55d6a1864c40 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.941+0000 7f79b6000640 10 client.4665 handle_command_reply: tid=0 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 client.4665 shutdown 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 -- 172.21.15.69:0/3643584800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 msgr2=0x55d6a18725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18725e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f79b0000c00 tx=0x7f79b00046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 -- 172.21.15.69:0/3643584800 shutdown_connections 2024-04-24T08:08:36.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.69:6834/1003206529,v1:172.21.15.69:6835/1003206529] conn(0x55d6a1864c40 0x55d6a1865020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d6a187e600 0x55d6a18725e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d6a187f7b0 0x55d6a1872b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 --2- 172.21.15.69:0/3643584800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d6a18628d0 0x55d6a18da8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79b6000640 0 client.4665 ms_handle_reset on v2:172.21.15.69:6834/1003206529 2024-04-24T08:08:36.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 -- 172.21.15.69:0/3643584800 >> 172.21.15.69:0/3643584800 conn(0x55d6a1862190 msgr2=0x55d6a1880070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:36.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 -- 172.21.15.69:0/3643584800 shutdown_connections 2024-04-24T08:08:36.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 1 -- 172.21.15.69:0/3643584800 wait complete. 2024-04-24T08:08:36.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.942+0000 7f79d2770740 20 client.4665 trim_cache size 0 max 16384 2024-04-24T08:08:36.947 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4257, 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 16.808899213, 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 7, 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 20, 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 20, 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 22.206278569999998 2024-04-24T08:08:36.948 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:08:36.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 -- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79a406c920 msgr2=0x7f79a406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79a406c920 0x7f79a406ede0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f79b8033000 tx=0x7f79b8009030 comp rx=0 tx=0).stop 2024-04-24T08:08:36.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 -- 172.21.15.69:0/3667402560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c8163be0 msgr2=0x7f79c8164040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:36.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c8163be0 0x7f79c8164040 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f79b000e7d0 tx=0x7f79b000ec90 comp rx=0 tx=0).stop 2024-04-24T08:08:36.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 -- 172.21.15.69:0/3667402560 shutdown_connections 2024-04-24T08:08:36.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79a406c920 0x7f79a406ede0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79c8163be0 0x7f79c8164040 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79c815dd30 0x7f79c815e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 --2- 172.21.15.69:0/3667402560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79ac0039b0 0x7f79c81f8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:36.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 -- 172.21.15.69:0/3667402560 >> 172.21.15.69:0/3667402560 conn(0x7f79c8067d90 msgr2=0x7f79c80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:36.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 -- 172.21.15.69:0/3667402560 shutdown_connections 2024-04-24T08:08:36.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:36.943+0000 7f79cfa00640 1 -- 172.21.15.69:0/3667402560 wait complete. 2024-04-24T08:08:36.963 INFO:teuthology.run_tasks:Running task sequential... 2024-04-24T08:08:36.973 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-24T08:08:36.973 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-24T08:08:36.974 INFO:teuthology.task.print:Enabling mgr modules 2024-04-24T08:08:36.974 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-24T08:08:37.037 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-24T08:08:37.037 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-24T08:08:37.038 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-24T08:08:37.038 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T08:08:37.038 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:08:37.038 INFO:tasks.ceph_fuse:client.2 config is {} 2024-04-24T08:08:37.038 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T08:08:37.038 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-24T08:08:37.038 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-24T08:08:37.038 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T08:08:37.038 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-24T08:08:37.038 INFO:tasks.ceph_fuse:client.3 config is {} 2024-04-24T08:08:37.039 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T08:08:37.039 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-24T08:08:37.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:37.039 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:08:37.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:37.095 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-04-24T08:08:37.181 INFO:teuthology.orchestra.run.smithi069.stderr:Cannot find device "ceph-brx" 2024-04-24T08:08:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:08:37.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:37.184 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-24T08:08:37.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:37.243 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2024-04-24T08:08:37.335 INFO:teuthology.orchestra.run.smithi121.stderr:Cannot find device "ceph-brx" 2024-04-24T08:08:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:08:37.338 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-24T08:08:37.339 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:08:37.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.551+0000 7fe79b200640 1 Processor -- start 2024-04-24T08:08:37.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.551+0000 7fe79b200640 1 -- start start 2024-04-24T08:08:37.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.552+0000 7fe79b200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe79415d1d0 0x7fe7940ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.552+0000 7fe79b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7940b30f0 0x7fe7940ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.552+0000 7fe79b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe79415d660 con 0x7fe794156c60 2024-04-24T08:08:37.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.552+0000 7fe79b200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe79406dc30 con 0x7fe79415d1d0 2024-04-24T08:08:37.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.552+0000 7fe79b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe79406df10 con 0x7fe7940b30f0 2024-04-24T08:08:37.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.553+0000 7fe799400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe79415d1d0 0x7fe7940ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.553+0000 7fe799400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe79415d1d0 0x7fe7940ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:33030/0 (socket says 172.21.15.69:33030) 2024-04-24T08:08:37.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.553+0000 7fe799400640 1 -- 172.21.15.69:0/1290168399 learned_addr learned my addr 172.21.15.69:0/1290168399 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:37.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.553+0000 7fe79a800640 1 --2- 172.21.15.69:0/1290168399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7940b30f0 0x7fe7940ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.554+0000 7fe799400640 1 -- 172.21.15.69:0/1290168399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7940b30f0 msgr2=0x7fe7940ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:37.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.554+0000 7fe799400640 1 --2- 172.21.15.69:0/1290168399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7940b30f0 0x7fe7940ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.554+0000 7fe799400640 1 -- 172.21.15.69:0/1290168399 >> v1:172.21.15.121:6789/0 conn(0x7fe794156c60 legacy=0x7fe79415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.554+0000 7fe799400640 1 -- 172.21.15.69:0/1290168399 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79406e1f0 con 0x7fe79415d1d0 2024-04-24T08:08:37.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.555+0000 7fe799400640 1 --2- 172.21.15.69:0/1290168399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe79415d1d0 0x7fe7940ad0d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe784009870 tx=0x7fe78402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6140821e1f78e61 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:37.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.556+0000 7fe798a00640 1 -- 172.21.15.69:0/1290168399 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784037070 con 0x7fe79415d1d0 2024-04-24T08:08:37.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.556+0000 7fe798a00640 1 -- 172.21.15.69:0/1290168399 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe784032440 con 0x7fe79415d1d0 2024-04-24T08:08:37.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.556+0000 7fe798a00640 1 -- 172.21.15.69:0/1290168399 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784032810 con 0x7fe79415d1d0 2024-04-24T08:08:37.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.557+0000 7fe79b200640 1 -- 172.21.15.69:0/1290168399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe79415d1d0 msgr2=0x7fe7940ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.557+0000 7fe79b200640 1 --2- 172.21.15.69:0/1290168399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe79415d1d0 0x7fe7940ad0d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe784009870 tx=0x7fe78402a250 comp rx=0 tx=0).stop 2024-04-24T08:08:37.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.557+0000 7fe79b200640 1 -- 172.21.15.69:0/1290168399 shutdown_connections 2024-04-24T08:08:37.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.557+0000 7fe79b200640 1 --2- 172.21.15.69:0/1290168399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7940b30f0 0x7fe7940ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.557+0000 7fe79b200640 1 --2- 172.21.15.69:0/1290168399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe79415d1d0 0x7fe7940ad0d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.557+0000 7fe79b200640 1 -- 172.21.15.69:0/1290168399 >> 172.21.15.69:0/1290168399 conn(0x7fe7940d0160 msgr2=0x7fe7940d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:37.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.557+0000 7fe79b200640 1 -- 172.21.15.69:0/1290168399 shutdown_connections 2024-04-24T08:08:37.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.557+0000 7fe79b200640 1 -- 172.21.15.69:0/1290168399 wait complete. 2024-04-24T08:08:37.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.559+0000 7fe79b200640 1 Processor -- start 2024-04-24T08:08:37.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.559+0000 7fe79b200640 1 -- start start 2024-04-24T08:08:37.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe79b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7940b30f0 0x7fe7940c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe79b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe794156c60 0x7fe79415e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe79b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7941e7660 0x7fe794145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe79b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe79406e710 con 0x7fe794156c60 2024-04-24T08:08:37.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe79b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe79415d940 con 0x7fe7941e7660 2024-04-24T08:08:37.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe79b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe79406e890 con 0x7fe7940b30f0 2024-04-24T08:08:37.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe799400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe794156c60 0x7fe79415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe799400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe794156c60 0x7fe79415e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44716/0 (socket says 172.21.15.69:44716) 2024-04-24T08:08:37.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe799400640 1 -- 172.21.15.69:0/3172571587 learned_addr learned my addr 172.21.15.69:0/3172571587 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:37.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe799e00640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7940b30f0 0x7fe7940c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe79a800640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7941e7660 0x7fe794145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe799400640 1 -- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7940b30f0 msgr2=0x7fe7940c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe799400640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7940b30f0 0x7fe7940c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe799400640 1 -- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7941e7660 msgr2=0x7fe794145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe799400640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7941e7660 0x7fe794145370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.560+0000 7fe799400640 1 -- 172.21.15.69:0/3172571587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7940ca7d0 con 0x7fe794156c60 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.561+0000 7fe799400640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe794156c60 0x7fe79415e910 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe78402a290 tx=0x7fe7840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.561+0000 7fe78e000640 1 -- 172.21.15.69:0/3172571587 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784037040 con 0x7fe794156c60 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.561+0000 7fe78e000640 1 -- 172.21.15.69:0/3172571587 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe784042070 con 0x7fe794156c60 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.561+0000 7fe79b200640 1 -- 172.21.15.69:0/3172571587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7940caab0 con 0x7fe794156c60 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.561+0000 7fe78e000640 1 -- 172.21.15.69:0/3172571587 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe784030880 con 0x7fe794156c60 2024-04-24T08:08:37.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.562+0000 7fe79b200640 1 -- 172.21.15.69:0/3172571587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7940cb070 con 0x7fe794156c60 2024-04-24T08:08:37.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.563+0000 7fe79b200640 1 -- 172.21.15.69:0/3172571587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe794075d50 con 0x7fe794156c60 2024-04-24T08:08:37.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.564+0000 7fe78e000640 1 -- 172.21.15.69:0/3172571587 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe784030a20 con 0x7fe794156c60 2024-04-24T08:08:37.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.565+0000 7fe78e000640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe77806c8f0 0x7fe77806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.565+0000 7fe78e000640 1 -- 172.21.15.69:0/3172571587 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe7840ac370 con 0x7fe794156c60 2024-04-24T08:08:37.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.566+0000 7fe799e00640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe77806c8f0 0x7fe77806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.567+0000 7fe799e00640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe77806c8f0 0x7fe77806edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe788007fd0 tx=0x7fe788009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:37.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.570+0000 7fe78e000640 1 -- 172.21.15.69:0/3172571587 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe78407a7a0 con 0x7fe794156c60 2024-04-24T08:08:37.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.810+0000 7fe79b200640 1 -- 172.21.15.69:0/3172571587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe79406f9e0 con 0x7fe794156c60 2024-04-24T08:08:37.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.811+0000 7fe78e000640 1 -- 172.21.15.69:0/3172571587 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7fe78407a0a0 con 0x7fe794156c60 2024-04-24T08:08:37.812 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:08:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 -- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe77806c8f0 msgr2=0x7fe77806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe77806c8f0 0x7fe77806edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe788007fd0 tx=0x7fe788009040 comp rx=0 tx=0).stop 2024-04-24T08:08:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 -- 172.21.15.69:0/3172571587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe794156c60 msgr2=0x7fe79415e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe794156c60 0x7fe79415e910 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe78402a290 tx=0x7fe7840046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 -- 172.21.15.69:0/3172571587 shutdown_connections 2024-04-24T08:08:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe77806c8f0 0x7fe77806edb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe794156c60 0x7fe79415e910 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7941e7660 0x7fe794145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 --2- 172.21.15.69:0/3172571587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7940b30f0 0x7fe7940c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 -- 172.21.15.69:0/3172571587 >> 172.21.15.69:0/3172571587 conn(0x7fe7940d0160 msgr2=0x7fe7940b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 -- 172.21.15.69:0/3172571587 shutdown_connections 2024-04-24T08:08:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37.814+0000 7fe79b200640 1 -- 172.21.15.69:0/3172571587 wait complete. 2024-04-24T08:08:37.832 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:08:37.832 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:08:37.832 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:08:37.833 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:08:37.833 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:08:37.833 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:08:37.833 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:08:37.833 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:08:37.833 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:08:37.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:37.833 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42662sec preferred_lft 42662sec 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:37.883 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T08:08:37.883 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:37.883 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:08:37.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add name ceph-brx type bridge 2024-04-24T08:08:37.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr flush dev ceph-brx 2024-04-24T08:08:37.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set ceph-brx up 2024-04-24T08:08:37.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T08:08:37.883 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:08:37.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:38.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:38.344 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:08:38.407 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:08:38.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:38.411 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:08:38.468 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:08:38.469 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:08:38.469 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:08:38.469 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:08:38.469 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:08:38.469 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:38.469 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:08:38.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:08:38.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:08:38.470 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:08:38.470 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:08:38.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:39.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:39.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:39.237 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:08:39.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:39.280 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list-id 2024-04-24T08:08:39.366 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:39.366 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:08:39.367 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:08:39.367 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-24T08:08:39.367 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:08:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:39.587 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-24T08:08:39.588 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:39.588 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:08:39.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-24T08:08:39.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T08:08:39.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-24T08:08:39.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-24T08:08:39.588 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-24T08:08:39.588 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:08:39.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:39.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:39.853 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:39.853 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:08:39.853 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:08:39.853 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T08:08:39.853 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:08:39.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:40.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:40.041 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:08:40.041 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:08:40.041 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:08:40.082 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:08:40.082 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:08:40.082 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:08:40.167 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:08:40.273 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:08:40.320 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:08:40.321 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:08:40.322 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:08:40.322 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:08:40.322 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:08:40.322 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:08:40.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:40.323 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:08:40.367 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:08:40.367 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:08:40.415 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:08:40.485 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:08:40.526 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:08:40.527 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:08:40.528 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:08:40.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:40.529 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:08:40.594 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:08:40.593+0000 7f73a098d480 -1 init, newargv = 0x55dd650abd50 newargc=15 2024-04-24T08:08:40.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[41637]: starting ceph client 2024-04-24T08:08:40.605 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[41637]: starting fuse 2024-04-24T08:08:40.615 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:08:40.615 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:08:40.615 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:08:40.615 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:08:40.615 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:08:40.615 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:08:40.615 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:08:40.615 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:08:40.615 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:08:40.615 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:08:40.615 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:08:40.616 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:08:40.617 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:08:40.617 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:08:40.617 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:08:40.798 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.41637.asok 2024-04-24T08:08:40.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:40.815 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.41637.asok 2024-04-24T08:08:40.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:40.816 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.41637.asok status 2024-04-24T08:08:41.037 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "41637", 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4674, 2024-04-24T08:08:41.038 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 4674 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2852256800 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2852256800 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:08:41.039 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.4674 192.168.0.1:0/2852256800", 2024-04-24T08:08:41.040 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2852256800", 2024-04-24T08:08:41.040 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:08:41.040 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 7, 2024-04-24T08:08:41.040 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 20, 2024-04-24T08:08:41.040 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:08:41.040 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:08:41.040 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:08:41.040 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:08:41.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:08:41.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.240+0000 7fb3f3000640 1 Processor -- start 2024-04-24T08:08:41.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.240+0000 7fb3f3000640 1 -- start start 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3f3000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb3ec0b30f0 0x7fb3ec167470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3f3000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ec15d660 con 0x7fb3ec156c60 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3f3000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3ec06dc30 con 0x7fb3ec15d1d0 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3f3000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb3ec06df10 con 0x7fb3ec0b30f0 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3f1c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb3ec156c60 0x7fb3ec15cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48032/0 (socket says 172.21.15.69:48032) 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3f1c00640 1 -- 172.21.15.69:0/683086610 learned_addr learned my addr 172.21.15.69:0/683086610 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3f2600640 1 --2- 172.21.15.69:0/683086610 >> v2:172.21.15.69:3300/0 conn(0x7fb3ec0b30f0 0x7fb3ec167470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3ebe00640 1 -- 172.21.15.69:0/683086610 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3313533989 0 0) 0x7fb3ec15d660 con 0x7fb3ec156c60 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3ebe00640 1 -- 172.21.15.69:0/683086610 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3d4003520 con 0x7fb3ec156c60 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3f2600640 1 -- 172.21.15.69:0/683086610 >> v1:172.21.15.121:6789/0 conn(0x7fb3ec15d1d0 legacy=0x7fb3ec0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.242+0000 7fb3f2600640 1 -- 172.21.15.69:0/683086610 >> v1:172.21.15.69:6789/0 conn(0x7fb3ec156c60 legacy=0x7fb3ec15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.243+0000 7fb3f2600640 1 -- 172.21.15.69:0/683086610 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ec06e1f0 con 0x7fb3ec0b30f0 2024-04-24T08:08:41.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.243+0000 7fb3f2600640 1 --2- 172.21.15.69:0/683086610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec0b30f0 0x7fb3ec167470 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009d90 tx=0x7fb3e402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3311663cad9d7b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:41.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.244+0000 7fb3ebe00640 1 -- 172.21.15.69:0/683086610 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4007740 con 0x7fb3ec0b30f0 2024-04-24T08:08:41.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.244+0000 7fb3ebe00640 1 -- 172.21.15.69:0/683086610 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3e40078e0 con 0x7fb3ec0b30f0 2024-04-24T08:08:41.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.244+0000 7fb3ebe00640 1 -- 172.21.15.69:0/683086610 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4007cb0 con 0x7fb3ec0b30f0 2024-04-24T08:08:41.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.244+0000 7fb3f3000640 1 -- 172.21.15.69:0/683086610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec0b30f0 msgr2=0x7fb3ec167470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.244+0000 7fb3f3000640 1 --2- 172.21.15.69:0/683086610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec0b30f0 0x7fb3ec167470 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009d90 tx=0x7fb3e402a6a0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.244+0000 7fb3f3000640 1 -- 172.21.15.69:0/683086610 shutdown_connections 2024-04-24T08:08:41.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.244+0000 7fb3f3000640 1 --2- 172.21.15.69:0/683086610 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec0b30f0 0x7fb3ec167470 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.244+0000 7fb3f3000640 1 -- 172.21.15.69:0/683086610 >> 172.21.15.69:0/683086610 conn(0x7fb3ec0d0160 msgr2=0x7fb3ec0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:41.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.245+0000 7fb3f3000640 1 -- 172.21.15.69:0/683086610 shutdown_connections 2024-04-24T08:08:41.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.245+0000 7fb3f3000640 1 -- 172.21.15.69:0/683086610 wait complete. 2024-04-24T08:08:41.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.246+0000 7fb3f3000640 1 Processor -- start 2024-04-24T08:08:41.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.246+0000 7fb3f3000640 1 -- start start 2024-04-24T08:08:41.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.247+0000 7fb3f3000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3ec0b30f0 0x7fb3ec142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:41.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.247+0000 7fb3f3000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3ec156c60 0x7fb3ec153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:41.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.247+0000 7fb3f3000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec154b50 0x7fb3ec153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:41.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.247+0000 7fb3f3000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ec0ad7e0 con 0x7fb3ec154b50 2024-04-24T08:08:41.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.247+0000 7fb3f3000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ec15d940 con 0x7fb3ec156c60 2024-04-24T08:08:41.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.247+0000 7fb3f3000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb3ec0ad960 con 0x7fb3ec0b30f0 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.247+0000 7fb3f2600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec154b50 0x7fb3ec153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.248+0000 7fb3f2600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec154b50 0x7fb3ec153650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58810/0 (socket says 172.21.15.69:58810) 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.248+0000 7fb3f2600640 1 -- 172.21.15.69:0/3178511568 learned_addr learned my addr 172.21.15.69:0/3178511568 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.248+0000 7fb3f1c00640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3ec0b30f0 0x7fb3ec142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.248+0000 7fb3f1200640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3ec156c60 0x7fb3ec153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.248+0000 7fb3f2600640 1 -- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3ec0b30f0 msgr2=0x7fb3ec142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.248+0000 7fb3f2600640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3ec0b30f0 0x7fb3ec142e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.248+0000 7fb3f2600640 1 -- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3ec156c60 msgr2=0x7fb3ec153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.248+0000 7fb3f2600640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3ec156c60 0x7fb3ec153110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.248+0000 7fb3f2600640 1 -- 172.21.15.69:0/3178511568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ec1548b0 con 0x7fb3ec154b50 2024-04-24T08:08:41.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.248+0000 7fb3f2600640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec154b50 0x7fb3ec153650 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009d90 tx=0x7fb3e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:41.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.249+0000 7fb3e9600640 1 -- 172.21.15.69:0/3178511568 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4040070 con 0x7fb3ec154b50 2024-04-24T08:08:41.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.249+0000 7fb3e9600640 1 -- 172.21.15.69:0/3178511568 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3e4031bc0 con 0x7fb3ec154b50 2024-04-24T08:08:41.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.249+0000 7fb3e9600640 1 -- 172.21.15.69:0/3178511568 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4009450 con 0x7fb3ec154b50 2024-04-24T08:08:41.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.249+0000 7fb3f3000640 1 -- 172.21.15.69:0/3178511568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ec1542f0 con 0x7fb3ec154b50 2024-04-24T08:08:41.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.249+0000 7fb3f3000640 1 -- 172.21.15.69:0/3178511568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ec145560 con 0x7fb3ec154b50 2024-04-24T08:08:41.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.250+0000 7fb3f3000640 1 -- 172.21.15.69:0/3178511568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3ec075d50 con 0x7fb3ec154b50 2024-04-24T08:08:41.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.251+0000 7fb3e9600640 1 -- 172.21.15.69:0/3178511568 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb3e4039040 con 0x7fb3ec154b50 2024-04-24T08:08:41.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.252+0000 7fb3e9600640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3cc06c8f0 0x7fb3cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:41.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.252+0000 7fb3e9600640 1 -- 172.21.15.69:0/3178511568 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb3e40acd00 con 0x7fb3ec154b50 2024-04-24T08:08:41.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.253+0000 7fb3f1c00640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3cc06c8f0 0x7fb3cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:41.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.254+0000 7fb3f1c00640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3cc06c8f0 0x7fb3cc06edb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0000f60 tx=0x7fb3e0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:41.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.257+0000 7fb3e9600640 1 -- 172.21.15.69:0/3178511568 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3e407b090 con 0x7fb3ec154b50 2024-04-24T08:08:41.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.498+0000 7fb3f3000640 1 -- 172.21.15.69:0/3178511568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb3ec0b4f40 con 0x7fb3ec154b50 2024-04-24T08:08:41.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.499+0000 7fb3e9600640 1 -- 172.21.15.69:0/3178511568 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7fb3e407a990 con 0x7fb3ec154b50 2024-04-24T08:08:41.500 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:08:41.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.501+0000 7fb3f3000640 1 -- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3cc06c8f0 msgr2=0x7fb3cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.501+0000 7fb3f3000640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3cc06c8f0 0x7fb3cc06edb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0000f60 tx=0x7fb3e0009040 comp rx=0 tx=0).stop 2024-04-24T08:08:41.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.502+0000 7fb3f3000640 1 -- 172.21.15.69:0/3178511568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec154b50 msgr2=0x7fb3ec153650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.502+0000 7fb3f3000640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec154b50 0x7fb3ec153650 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009d90 tx=0x7fb3e4004a40 comp rx=0 tx=0).stop 2024-04-24T08:08:41.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.502+0000 7fb3f3000640 1 -- 172.21.15.69:0/3178511568 shutdown_connections 2024-04-24T08:08:41.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.502+0000 7fb3f3000640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3cc06c8f0 0x7fb3cc06edb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.502+0000 7fb3f3000640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3ec154b50 0x7fb3ec153650 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.502+0000 7fb3f3000640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3ec156c60 0x7fb3ec153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.502+0000 7fb3f3000640 1 --2- 172.21.15.69:0/3178511568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3ec0b30f0 0x7fb3ec142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.502+0000 7fb3f3000640 1 -- 172.21.15.69:0/3178511568 >> 172.21.15.69:0/3178511568 conn(0x7fb3ec0d0160 msgr2=0x7fb3ec15a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:41.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.502+0000 7fb3f3000640 1 -- 172.21.15.69:0/3178511568 shutdown_connections 2024-04-24T08:08:41.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41.502+0000 7fb3f3000640 1 -- 172.21.15.69:0/3178511568 wait complete. 2024-04-24T08:08:41.522 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:08:41.522 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:08:41.522 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:08:41.522 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-24T08:08:41.522 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-24T08:08:41.522 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:08:41.522 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:08:41.522 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:08:41.522 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-24T08:08:41.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:41.522 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:08:41.572 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:41.573 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42658sec preferred_lft 42658sec 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:41.574 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:08:41.575 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:08:41.575 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:08:41.575 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:41.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:41.576 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:08:41.646 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:08:41.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:41.650 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:08:41.697 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:08:41.697 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:08:41.697 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:08:41.697 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:08:41.697 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:08:41.698 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:41.698 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:08:41.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:08:41.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:08:41.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:08:41.698 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:08:41.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:41.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:41.896 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:08:41.940 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:08:41.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:41.941 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list-id 2024-04-24T08:08:42.036 INFO:teuthology.orchestra.run.smithi069.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-24T08:08:42.037 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:42.037 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:08:42.037 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:08:42.037 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-24T08:08:42.037 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:08:42.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:42.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:42.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:42.226 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-24T08:08:42.303 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 96:4e:3b:de:33:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::944e:3bff:fede:33f9/64 scope link 2024-04-24T08:08:42.304 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:42.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:42.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-24T08:08:42.386 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 96:4e:3b:de:33:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::944e:3bff:fede:33f9/64 scope link 2024-04-24T08:08:42.387 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:42.391 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-24T08:08:42.391 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:42.391 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:08:42.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-24T08:08:42.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-24T08:08:42.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-24T08:08:42.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-24T08:08:42.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-24T08:08:42.391 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:08:42.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:42.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:42.662 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:42.662 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:08:42.662 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.1 up 2024-04-24T08:08:42.662 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set dev brx.1 master ceph-brx 2024-04-24T08:08:42.662 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:08:42.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:42.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:42.867 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-24T08:08:42.867 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:08:42.867 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-24T08:08:42.913 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-24T08:08:42.914 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:08:42.914 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-24T08:08:43.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:08:43.088 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:08:43.135 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:08:43.135 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:08:43.135 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:08:43.135 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:08:43.135 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:08:43.136 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:08:43.137 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:08:43.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:43.139 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:08:43.186 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:08:43.186 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-04-24T08:08:43.186 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-04-24T08:08:43.233 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:08:43.305 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:08:43.348 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:08:43.349 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:08:43.350 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:08:43.350 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:08:43.350 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:08:43.350 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:08:43.350 INFO:teuthology.orchestra.run.smithi069.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-24T08:08:43.350 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:08:43.350 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:08:43.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:43.351 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:08:43.364 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:08:43.398 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:08:43.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi069.stderr:2024-04-24T08:08:43.405+0000 7f84b58b2480 -1 init, newargv = 0x564172a0add0 newargc=15 2024-04-24T08:08:43.407 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi069.stderr:ceph-fuse[42135]: starting ceph client 2024-04-24T08:08:43.417 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi069.stderr:ceph-fuse[42135]: starting fuse 2024-04-24T08:08:43.435 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:08:43.435 INFO:teuthology.orchestra.run.smithi069.stdout:43 2024-04-24T08:08:43.435 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 43] 2024-04-24T08:08:43.435 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:08:43.435 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:08:43.435 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:08:43.435 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:08:43.435 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:08:43.435 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:08:43.436 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:08:43.437 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:08:43.437 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.2")) 2024-04-24T08:08:43.437 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:08:43.620 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.2.42135.asok 2024-04-24T08:08:43.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:43.637 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.42135.asok 2024-04-24T08:08:43.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:43.638 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.42135.asok status 2024-04-24T08:08:43.817 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:08:43.817 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:08:43.817 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:08:43.817 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:08:43.817 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "2", 2024-04-24T08:08:43.817 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "42135", 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4689, 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 4689 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:08:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.2:0", 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1006449122 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.2:0", 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1006449122 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.4689 192.168.0.2:0/1006449122", 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.2:0/1006449122", 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 7, 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 20, 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:08:43.819 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:08:43.820 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:08:43.820 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:08:43.831 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:08:44.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.022+0000 7f43c7e00640 1 Processor -- start 2024-04-24T08:08:44.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.022+0000 7f43c7e00640 1 -- start start 2024-04-24T08:08:44.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.023+0000 7f43c7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c8156c60 0x7f43c815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.023+0000 7f43c7e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43c815d660 con 0x7f43c80b30f0 2024-04-24T08:08:44.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.024+0000 7f43c7e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43c806dc30 con 0x7f43c80ad7e0 2024-04-24T08:08:44.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.024+0000 7f43c7e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f43c806df10 con 0x7f43c8156c60 2024-04-24T08:08:44.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.024+0000 7f43c6000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f43c80b30f0 0x7f43c80ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48048/0 (socket says 172.21.15.69:48048) 2024-04-24T08:08:44.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.024+0000 7f43c6000640 1 -- 172.21.15.69:0/1850347441 learned_addr learned my addr 172.21.15.69:0/1850347441 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:44.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.024+0000 7f43c6a00640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c8156c60 0x7f43c815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.024+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1632776958 0 0) 0x7f43c815d660 con 0x7f43c80b30f0 2024-04-24T08:08:44.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.024+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43ac003520 con 0x7f43c80b30f0 2024-04-24T08:08:44.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3980996609 0 0) 0x7f43ac003520 con 0x7f43c80b30f0 2024-04-24T08:08:44.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c8156c60 msgr2=0x7f43c815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:44.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c8156c60 0x7f43c815cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 >> v1:172.21.15.121:6789/0 conn(0x7f43c80ad7e0 legacy=0x7f43c806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c806e1f0 con 0x7f43c80b30f0 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f43b4001670 con 0x7f43c80b30f0 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 >> v1:172.21.15.69:6789/0 conn(0x7f43c80b30f0 legacy=0x7f43c80ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c8156c60 0x7f43ac0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43ac003fb0 0x7f43ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43ac004d40 0x7f43ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43c806df10 con 0x7f43ac003fb0 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac00bb00 con 0x7f43ac004d40 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f43ac00bd00 con 0x7f43c8156c60 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c6000640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43ac003fb0 0x7f43ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.025+0000 7f43c6a00640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c8156c60 0x7f43ac0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.026+0000 7f43c7400640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43ac004d40 0x7f43ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.026+0000 7f43c6000640 1 -- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c8156c60 msgr2=0x7f43ac0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.026+0000 7f43c6000640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c8156c60 0x7f43ac0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.026+0000 7f43c6000640 1 -- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43ac004d40 msgr2=0x7f43ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.026+0000 7f43c6000640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43ac004d40 0x7f43ac0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.026+0000 7f43c6000640 1 -- 172.21.15.69:0/1850347441 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ac003520 con 0x7f43ac003fb0 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.027+0000 7f43c6000640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43ac003fb0 0x7f43ac004800 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f43b400c160 tx=0x7f43ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed86c58ea010cc78 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.027+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4007c60 con 0x7f43ac003fb0 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.027+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b4005ad0 con 0x7f43ac003fb0 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.027+0000 7f43c5600640 1 -- 172.21.15.69:0/1850347441 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4007070 con 0x7f43ac003fb0 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.027+0000 7f43c7e00640 1 -- 172.21.15.69:0/1850347441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43ac003fb0 msgr2=0x7f43ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.027+0000 7f43c7e00640 1 --2- 172.21.15.69:0/1850347441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43ac003fb0 0x7f43ac004800 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f43b400c160 tx=0x7f43ac005830 comp rx=0 tx=0).stop 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.027+0000 7f43c7400640 1 -- 172.21.15.69:0/1850347441 reap_dead start 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.028+0000 7f43c7e00640 1 -- 172.21.15.69:0/1850347441 shutdown_connections 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.028+0000 7f43c7e00640 1 -- 172.21.15.69:0/1850347441 >> 172.21.15.69:0/1850347441 conn(0x7f43c80d0160 msgr2=0x7f43c80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.028+0000 7f43c7e00640 1 -- 172.21.15.69:0/1850347441 shutdown_connections 2024-04-24T08:08:44.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.028+0000 7f43c7e00640 1 -- 172.21.15.69:0/1850347441 wait complete. 2024-04-24T08:08:44.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.029+0000 7f43c7e00640 1 Processor -- start 2024-04-24T08:08:44.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.030+0000 7f43c7e00640 1 -- start start 2024-04-24T08:08:44.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.030+0000 7f43c7e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43ac003fb0 0x7f43c80cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c81f3f10 0x7f43c80c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c80c9f80 0x7f43c80ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f43c806ee20 con 0x7f43c81f3f10 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c7e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f43c815d940 con 0x7f43ac003fb0 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c7e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f43c806efa0 con 0x7f43c80c9f80 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c81f3f10 0x7f43c80c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c81f3f10 0x7f43c80c9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58838/0 (socket says 172.21.15.69:58838) 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c6000640 1 -- 172.21.15.69:0/86592944 learned_addr learned my addr 172.21.15.69:0/86592944 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c6a00640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43ac003fb0 0x7f43c80cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c7400640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c80c9f80 0x7f43c80ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c6000640 1 -- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c80c9f80 msgr2=0x7f43c80ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c6000640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c80c9f80 0x7f43c80ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c6000640 1 -- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43ac003fb0 msgr2=0x7f43c80cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c6000640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43ac003fb0 0x7f43c80cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.031+0000 7f43c6000640 1 -- 172.21.15.69:0/86592944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c80ca590 con 0x7f43c81f3f10 2024-04-24T08:08:44.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.032+0000 7f43c6000640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c81f3f10 0x7f43c80c9a40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f43b402ac80 tx=0x7f43b4001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:44.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.032+0000 7f43bea00640 1 -- 172.21.15.69:0/86592944 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4040020 con 0x7f43c81f3f10 2024-04-24T08:08:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.032+0000 7f43bea00640 1 -- 172.21.15.69:0/86592944 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43b40013b0 con 0x7f43c81f3f10 2024-04-24T08:08:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.032+0000 7f43c7e00640 1 -- 172.21.15.69:0/86592944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c80ca870 con 0x7f43c81f3f10 2024-04-24T08:08:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.032+0000 7f43bea00640 1 -- 172.21.15.69:0/86592944 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b4037040 con 0x7f43c81f3f10 2024-04-24T08:08:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.032+0000 7f43c7e00640 1 -- 172.21.15.69:0/86592944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c80cae30 con 0x7f43c81f3f10 2024-04-24T08:08:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.033+0000 7f43c7e00640 1 -- 172.21.15.69:0/86592944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c8075d50 con 0x7f43c81f3f10 2024-04-24T08:08:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.034+0000 7f43bea00640 1 -- 172.21.15.69:0/86592944 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f43b4043020 con 0x7f43c81f3f10 2024-04-24T08:08:44.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.035+0000 7f43bea00640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43a806c920 0x7f43a806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.035+0000 7f43bea00640 1 -- 172.21.15.69:0/86592944 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f43b40abcf0 con 0x7f43c81f3f10 2024-04-24T08:08:44.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.036+0000 7f43c6a00640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43a806c920 0x7f43a806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.037+0000 7f43c6a00640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43a806c920 0x7f43a806ede0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009a10 tx=0x7f43b8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:44.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.040+0000 7f43bea00640 1 -- 172.21.15.69:0/86592944 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43b404a020 con 0x7f43c81f3f10 2024-04-24T08:08:44.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.288+0000 7f43c7e00640 1 -- 172.21.15.69:0/86592944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f43c806f6b0 con 0x7f43c81f3f10 2024-04-24T08:08:44.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.289+0000 7f43bea00640 1 -- 172.21.15.69:0/86592944 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f43b407a080 con 0x7f43c81f3f10 2024-04-24T08:08:44.290 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:08:44.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.291+0000 7f43c7e00640 1 -- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43a806c920 msgr2=0x7f43a806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.291+0000 7f43c7e00640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43a806c920 0x7f43a806ede0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f43b8009a10 tx=0x7f43b8008040 comp rx=0 tx=0).stop 2024-04-24T08:08:44.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.291+0000 7f43c7e00640 1 -- 172.21.15.69:0/86592944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c81f3f10 msgr2=0x7f43c80c9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.291+0000 7f43c7e00640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c81f3f10 0x7f43c80c9a40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f43b402ac80 tx=0x7f43b4001970 comp rx=0 tx=0).stop 2024-04-24T08:08:44.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.291+0000 7f43c7e00640 1 -- 172.21.15.69:0/86592944 shutdown_connections 2024-04-24T08:08:44.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.291+0000 7f43c7e00640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43a806c920 0x7f43a806ede0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.291+0000 7f43c7e00640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43c81f3f10 0x7f43c80c9a40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.291+0000 7f43c7e00640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43c80c9f80 0x7f43c80ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.291+0000 7f43c7e00640 1 --2- 172.21.15.69:0/86592944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43ac003fb0 0x7f43c80cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.291+0000 7f43c7e00640 1 -- 172.21.15.69:0/86592944 >> 172.21.15.69:0/86592944 conn(0x7f43c80d0160 msgr2=0x7f43c80b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:44.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.292+0000 7f43c7e00640 1 -- 172.21.15.69:0/86592944 shutdown_connections 2024-04-24T08:08:44.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:44.292+0000 7f43c7e00640 1 -- 172.21.15.69:0/86592944 wait complete. 2024-04-24T08:08:44.310 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:08:44.311 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:08:44.311 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi121.front.sepia.ceph.com 2024-04-24T08:08:44.311 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-24T08:08:44.311 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-24T08:08:44.311 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:08:44.311 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:08:44.311 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:08:44.311 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-24T08:08:44.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:44.311 DEBUG:teuthology.orchestra.run.smithi121:> ip addr 2024-04-24T08:08:44.368 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:08:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:08:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:08:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-04-24T08:08:44.368 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:82 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f0 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:83 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f1 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: inet 172.21.15.121/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft 42644sec preferred_lft 42644sec 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2024-04-24T08:08:44.369 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:44.370 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T08:08:44.371 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:44.371 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-24T08:08:44.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add name ceph-brx type bridge 2024-04-24T08:08:44.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr flush dev ceph-brx 2024-04-24T08:08:44.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set ceph-brx up 2024-04-24T08:08:44.371 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T08:08:44.371 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-24T08:08:44.475 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:44.852 DEBUG:teuthology.orchestra.run.smithi121:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:08:44.917 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-04-24T08:08:44.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:44.921 DEBUG:teuthology.orchestra.run.smithi121:> route 2024-04-24T08:08:44.976 INFO:teuthology.orchestra.run.smithi121.stdout:Kernel IP routing table 2024-04-24T08:08:44.976 INFO:teuthology.orchestra.run.smithi121.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:08:44.976 INFO:teuthology.orchestra.run.smithi121.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:08:44.976 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:08:44.976 INFO:teuthology.orchestra.run.smithi121.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:08:44.977 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:44.977 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-24T08:08:44.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:08:44.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:08:44.977 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:08:44.977 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-24T08:08:45.072 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:45.688 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:45.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:45.698 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-24T08:08:45.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:45.749 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list-id 2024-04-24T08:08:45.802 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:45.802 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-24T08:08:45.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:45.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-24T08:08:45.802 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-24T08:08:45.907 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:45.975 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:45.985 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-24T08:08:45.985 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:45.985 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-24T08:08:45.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-24T08:08:45.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T08:08:45.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-24T08:08:45.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-24T08:08:45.986 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-24T08:08:45.986 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-24T08:08:46.085 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:46.263 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:46.271 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:46.271 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-24T08:08:46.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set brx.0 up 2024-04-24T08:08:46.271 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T08:08:46.271 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-24T08:08:46.362 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:46.461 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:46.470 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-24T08:08:46.470 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:08:46.470 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-24T08:08:46.516 INFO:teuthology.orchestra.run.smithi121.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-24T08:08:46.516 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:08:46.516 DEBUG:teuthology.orchestra.run.smithi121:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-24T08:08:46.612 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2024-04-24T08:08:46.696 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:08:46.746 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:08:46.747 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-4 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-5 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-6 2024-04-24T08:08:46.748 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-7 2024-04-24T08:08:46.749 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2024-04-24T08:08:46.749 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:46.749 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:46.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:46.750 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2024-04-24T08:08:46.800 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:08:46.800 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-24T08:08:46.805 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2024-04-24T08:08:46.872 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:08:46.912 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2024-04-24T08:08:46.912 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2024-04-24T08:08:46.912 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2024-04-24T08:08:46.912 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2024-04-24T08:08:46.912 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2024-04-24T08:08:46.912 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2024-04-24T08:08:46.912 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2024-04-24T08:08:46.912 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2024-04-24T08:08:46.912 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2024-04-24T08:08:46.913 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-4 2024-04-24T08:08:46.914 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-5 2024-04-24T08:08:46.914 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-6 2024-04-24T08:08:46.914 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-7 2024-04-24T08:08:46.914 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2024-04-24T08:08:46.914 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:46.914 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:46.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:46.915 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2024-04-24T08:08:47.003 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi121.stderr:2024-04-24T08:08:47.003+0000 7f839bf29480 -1 init, newargv = 0x55fcaa1901d0 newargc=15 2024-04-24T08:08:47.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi121.stderr:ceph-fuse[39283]: starting ceph client 2024-04-24T08:08:47.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi121.stderr:ceph-fuse[39283]: starting fuse 2024-04-24T08:08:48.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2024-04-24T08:08:48.191 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:08:48.247 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2024-04-24T08:08:48.247 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2024-04-24T08:08:48.247 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2024-04-24T08:08:48.247 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2024-04-24T08:08:48.248 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-4 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-5 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-6 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-7 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:48.249 INFO:teuthology.orchestra.run.smithi121.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-24T08:08:48.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:48.250 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2024-04-24T08:08:48.302 INFO:teuthology.orchestra.run.smithi121.stdout:40 2024-04-24T08:08:48.302 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:08:48.302 DEBUG:teuthology.orchestra.run.smithi121:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> import re 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> import os 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> def _find_admin_socket(client_name): 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:08:48.303 DEBUG:teuthology.orchestra.run.smithi121:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:08:48.304 DEBUG:teuthology.orchestra.run.smithi121:> contents = proc_f.read() 2024-04-24T08:08:48.304 DEBUG:teuthology.orchestra.run.smithi121:> if mountpoint in contents: 2024-04-24T08:08:48.304 DEBUG:teuthology.orchestra.run.smithi121:> return f 2024-04-24T08:08:48.304 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:08:48.304 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-24T08:08:48.304 DEBUG:teuthology.orchestra.run.smithi121:> print(_find_admin_socket("client.1")) 2024-04-24T08:08:48.304 DEBUG:teuthology.orchestra.run.smithi121:> ' 2024-04-24T08:08:48.466 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.1.39283.asok 2024-04-24T08:08:48.470 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:48.483 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.39283.asok 2024-04-24T08:08:48.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:48.483 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.39283.asok status 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "1", 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "39283", 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-24T08:08:48.710 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4427, 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4427 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1110540226 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2024-04-24T08:08:48.711 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 1110540226 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.4427 192.168.0.1:0/1110540226", 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "192.168.0.1:0/1110540226", 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 7, 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 20, 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklisted": false, 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2024-04-24T08:08:48.712 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-24T08:08:48.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:08:48.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.935+0000 7f7327200640 1 Processor -- start 2024-04-24T08:08:48.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.935+0000 7f7327200640 1 -- start start 2024-04-24T08:08:48.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.936+0000 7f7327200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7320145a90 0x7f732006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:48.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.936+0000 7f7327200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7320145fe0 0x7f7320160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:48.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.936+0000 7f7327200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73200cf3d0 con 0x7f73200d0300 2024-04-24T08:08:48.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.936+0000 7f7327200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f73200cf6b0 con 0x7f7320145a90 2024-04-24T08:08:48.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.936+0000 7f7327200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f73200cf990 con 0x7f7320145fe0 2024-04-24T08:08:48.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.937+0000 7f731f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7320145a90 0x7f732006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:48.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.937+0000 7f731f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7320145a90 0x7f732006e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37768/0 (socket says 172.21.15.69:37768) 2024-04-24T08:08:48.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.937+0000 7f731f400640 1 -- 172.21.15.69:0/2928909943 learned_addr learned my addr 172.21.15.69:0/2928909943 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:48.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.937+0000 7f731f400640 1 -- 172.21.15.69:0/2928909943 >> v2:172.21.15.121:3300/0 conn(0x7f7320145fe0 msgr2=0x7f7320160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:48.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.937+0000 7f7325200640 1 --2- 172.21.15.69:0/2928909943 >> v2:172.21.15.121:3300/0 conn(0x7f7320145fe0 0x7f7320160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:48.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.937+0000 7f731f400640 1 --2- 172.21.15.69:0/2928909943 >> v2:172.21.15.121:3300/0 conn(0x7f7320145fe0 0x7f7320160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:48.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.937+0000 7f731f400640 1 -- 172.21.15.69:0/2928909943 >> v1:172.21.15.121:6789/0 conn(0x7f73200d0300 legacy=0x7f73200d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:48.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.937+0000 7f731f400640 1 -- 172.21.15.69:0/2928909943 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73200cfc70 con 0x7f7320145a90 2024-04-24T08:08:48.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.938+0000 7f731f400640 1 --2- 172.21.15.69:0/2928909943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 0x7f732006e240 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f73200d1050 tx=0x7f730c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768a93782d870c2c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:48.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f731ea00640 1 -- 172.21.15.69:0/2928909943 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c037070 con 0x7f7320145a90 2024-04-24T08:08:48.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f731ea00640 1 -- 172.21.15.69:0/2928909943 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f730c032440 con 0x7f7320145a90 2024-04-24T08:08:48.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f731ea00640 1 -- 172.21.15.69:0/2928909943 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c032810 con 0x7f7320145a90 2024-04-24T08:08:48.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f7327200640 1 -- 172.21.15.69:0/2928909943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 msgr2=0x7f732006e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:48.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f7327200640 1 --2- 172.21.15.69:0/2928909943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 0x7f732006e240 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f73200d1050 tx=0x7f730c02a230 comp rx=0 tx=0).stop 2024-04-24T08:08:48.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f7327200640 1 -- 172.21.15.69:0/2928909943 shutdown_connections 2024-04-24T08:08:48.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f7327200640 1 --2- 172.21.15.69:0/2928909943 >> v2:172.21.15.121:3300/0 conn(0x7f7320145fe0 0x7f7320160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:48.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f7327200640 1 --2- 172.21.15.69:0/2928909943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 0x7f732006e240 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:48.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f7327200640 1 -- 172.21.15.69:0/2928909943 >> 172.21.15.69:0/2928909943 conn(0x7f7320067d90 msgr2=0x7f732006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:48.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f7327200640 1 -- 172.21.15.69:0/2928909943 shutdown_connections 2024-04-24T08:08:48.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.939+0000 7f7327200640 1 -- 172.21.15.69:0/2928909943 wait complete. 2024-04-24T08:08:48.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.941+0000 7f7327200640 1 Processor -- start 2024-04-24T08:08:48.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.941+0000 7f7327200640 1 -- start start 2024-04-24T08:08:48.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.942+0000 7f7327200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73200d0300 0x7f73201e5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:48.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.942+0000 7f7327200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 0x7f732013d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:48.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.942+0000 7f7327200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7320145fe0 0x7f732015ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:48.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.942+0000 7f7327200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73201458f0 con 0x7f7320145a90 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.942+0000 7f7327200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7320145770 con 0x7f73200d0300 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.942+0000 7f7327200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f73200cdc30 con 0x7f7320145fe0 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.942+0000 7f731f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 0x7f732013d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.942+0000 7f731f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 0x7f732013d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37772/0 (socket says 172.21.15.69:37772) 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.942+0000 7f731f400640 1 -- 172.21.15.69:0/2363744730 learned_addr learned my addr 172.21.15.69:0/2363744730 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.943+0000 7f731fe00640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73200d0300 0x7f73201e5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.943+0000 7f7325200640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7320145fe0 0x7f732015ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.943+0000 7f731f400640 1 -- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7320145fe0 msgr2=0x7f732015ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.943+0000 7f731f400640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7320145fe0 0x7f732015ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.943+0000 7f731f400640 1 -- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73200d0300 msgr2=0x7f73201e5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.943+0000 7f731f400640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73200d0300 0x7f73201e5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.943+0000 7f731f400640 1 -- 172.21.15.69:0/2363744730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732013d5a0 con 0x7f7320145a90 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.943+0000 7f731fe00640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73200d0300 0x7f73201e5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:48.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.943+0000 7f7325200640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7320145fe0 0x7f732015ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:48.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.943+0000 7f731f400640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 0x7f732013d060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f730c02a270 tx=0x7f730c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:48.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.944+0000 7f7317e00640 1 -- 172.21.15.69:0/2363744730 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c037040 con 0x7f7320145a90 2024-04-24T08:08:48.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.944+0000 7f7317e00640 1 -- 172.21.15.69:0/2363744730 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f730c042070 con 0x7f7320145a90 2024-04-24T08:08:48.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.944+0000 7f7317e00640 1 -- 172.21.15.69:0/2363744730 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c030bb0 con 0x7f7320145a90 2024-04-24T08:08:48.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.944+0000 7f7327200640 1 -- 172.21.15.69:0/2363744730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f732014dca0 con 0x7f7320145a90 2024-04-24T08:08:48.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.944+0000 7f7327200640 1 -- 172.21.15.69:0/2363744730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f732014e260 con 0x7f7320145a90 2024-04-24T08:08:48.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.945+0000 7f7327200640 1 -- 172.21.15.69:0/2363744730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7320068a80 con 0x7f7320145a90 2024-04-24T08:08:48.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.946+0000 7f7317e00640 1 -- 172.21.15.69:0/2363744730 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f730c047460 con 0x7f7320145a90 2024-04-24T08:08:48.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.947+0000 7f7317e00640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c920 0x7f730006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:48.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.947+0000 7f7317e00640 1 -- 172.21.15.69:0/2363744730 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f730c0ac7d0 con 0x7f7320145a90 2024-04-24T08:08:48.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.947+0000 7f731fe00640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c920 0x7f730006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:48.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.949+0000 7f731fe00640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c920 0x7f730006ede0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f731000a810 tx=0x7f7310008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:48.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:48.951+0000 7f7317e00640 1 -- 172.21.15.69:0/2363744730 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f730c045360 con 0x7f7320145a90 2024-04-24T08:08:49.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.191+0000 7f7327200640 1 -- 172.21.15.69:0/2363744730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f73200d1120 con 0x7f7320145a90 2024-04-24T08:08:49.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.192+0000 7f7317e00640 1 -- 172.21.15.69:0/2363744730 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+73 (secure 0 0 0) 0x7f730c07abf0 con 0x7f7320145a90 2024-04-24T08:08:49.192 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:08:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.194+0000 7f7327200640 1 -- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c920 msgr2=0x7f730006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.194+0000 7f7327200640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c920 0x7f730006ede0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f731000a810 tx=0x7f7310008040 comp rx=0 tx=0).stop 2024-04-24T08:08:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.194+0000 7f7327200640 1 -- 172.21.15.69:0/2363744730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 msgr2=0x7f732013d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.194+0000 7f7327200640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 0x7f732013d060 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f730c02a270 tx=0x7f730c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.195+0000 7f7327200640 1 -- 172.21.15.69:0/2363744730 shutdown_connections 2024-04-24T08:08:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.195+0000 7f7327200640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c920 0x7f730006ede0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.195+0000 7f7327200640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7320145a90 0x7f732013d060 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:49.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.195+0000 7f7327200640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7320145fe0 0x7f732015ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:49.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.195+0000 7f7327200640 1 --2- 172.21.15.69:0/2363744730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73200d0300 0x7f73201e5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:49.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.195+0000 7f7327200640 1 -- 172.21.15.69:0/2363744730 >> 172.21.15.69:0/2363744730 conn(0x7f7320067d90 msgr2=0x7f732006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:49.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.195+0000 7f7327200640 1 -- 172.21.15.69:0/2363744730 shutdown_connections 2024-04-24T08:08:49.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:49.195+0000 7f7327200640 1 -- 172.21.15.69:0/2363744730 wait complete. 2024-04-24T08:08:49.214 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:08:49.214 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:08:49.214 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi121.front.sepia.ceph.com 2024-04-24T08:08:49.214 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-24T08:08:49.214 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-24T08:08:49.214 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:08:49.214 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:08:49.214 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:08:49.214 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-24T08:08:49.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:49.215 DEBUG:teuthology.orchestra.run.smithi121:> ip addr 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:82 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f0 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:d8:a4:83 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:49.273 INFO:teuthology.orchestra.run.smithi121.stdout: altname enp7s0f1 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 0c:c4:7a:8f:cb:a3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: inet 172.21.15.121/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft 42639sec preferred_lft 42639sec 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 8e:82:62:09:b1:cd brd ff:ff:ff:ff:ff:ff 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::c8a4:b1ff:fefc:e7fd/64 scope link 2024-04-24T08:08:49.274 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi121.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 8e:82:62:09:b1:cd brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::8c82:62ff:fe09:b1cd/64 scope link 2024-04-24T08:08:49.275 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:49.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:49.276 DEBUG:teuthology.orchestra.run.smithi121:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:08:49.354 INFO:teuthology.orchestra.run.smithi121.stdout:1 2024-04-24T08:08:49.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:49.358 DEBUG:teuthology.orchestra.run.smithi121:> route 2024-04-24T08:08:49.407 INFO:teuthology.orchestra.run.smithi121.stdout:Kernel IP routing table 2024-04-24T08:08:49.408 INFO:teuthology.orchestra.run.smithi121.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:08:49.408 INFO:teuthology.orchestra.run.smithi121.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:08:49.408 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:08:49.408 INFO:teuthology.orchestra.run.smithi121.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:08:49.408 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:49.409 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-24T08:08:49.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:08:49.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:08:49.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:08:49.409 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-24T08:08:49.507 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:49.603 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:49.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:49.612 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-24T08:08:49.658 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-24T08:08:49.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:49.659 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list-id 2024-04-24T08:08:49.755 INFO:teuthology.orchestra.run.smithi121.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-24T08:08:49.756 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:49.756 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-24T08:08:49.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T08:08:49.756 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-24T08:08:49.756 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-24T08:08:49.862 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:49.930 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:49.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:49.939 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-24T08:08:50.013 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:08:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:08:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:08:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-04-24T08:08:50.013 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:50.014 INFO:teuthology.orchestra.run.smithi121.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:08:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 26:84:af:d4:ba:d9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-24T08:08:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-24T08:08:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::2484:afff:fed4:bad9/64 scope link 2024-04-24T08:08:50.014 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:50.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:50.016 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-24T08:08:50.096 INFO:teuthology.orchestra.run.smithi121.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:08:50.096 INFO:teuthology.orchestra.run.smithi121.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:08:50.096 INFO:teuthology.orchestra.run.smithi121.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:08:50.096 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:50.096 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 ::1/128 scope host 2024-04-24T08:08:50.096 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:50.097 INFO:teuthology.orchestra.run.smithi121.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:08:50.097 INFO:teuthology.orchestra.run.smithi121.stdout: link/ether 26:84:af:d4:ba:d9 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-24T08:08:50.097 INFO:teuthology.orchestra.run.smithi121.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-24T08:08:50.097 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:50.097 INFO:teuthology.orchestra.run.smithi121.stdout: inet6 fe80::2484:afff:fed4:bad9/64 scope link 2024-04-24T08:08:50.097 INFO:teuthology.orchestra.run.smithi121.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:08:50.100 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-24T08:08:50.100 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:50.100 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-24T08:08:50.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-24T08:08:50.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-24T08:08:50.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-24T08:08:50.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-24T08:08:50.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-24T08:08:50.101 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-24T08:08:50.208 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:50.367 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:50.375 DEBUG:teuthology.orchestra.run.smithi121:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:08:50.376 DEBUG:teuthology.orchestra.run.smithi121:> set -e 2024-04-24T08:08:50.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set brx.1 up 2024-04-24T08:08:50.376 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link set dev brx.1 master ceph-brx 2024-04-24T08:08:50.376 DEBUG:teuthology.orchestra.run.smithi121:> ') 2024-04-24T08:08:50.465 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:08:50.561 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:50.569 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-24T08:08:50.570 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:08:50.570 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-24T08:08:50.615 INFO:teuthology.orchestra.run.smithi121.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-24T08:08:50.615 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:08:50.615 DEBUG:teuthology.orchestra.run.smithi121:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-24T08:08:50.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2024-04-24T08:08:50.794 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:08:50.847 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2024-04-24T08:08:50.847 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2024-04-24T08:08:50.848 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-4 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-5 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-6 2024-04-24T08:08:50.849 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-7 2024-04-24T08:08:50.850 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2024-04-24T08:08:50.850 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:50.850 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:50.850 INFO:teuthology.orchestra.run.smithi121.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-24T08:08:50.850 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T08:08:50.850 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T08:08:50.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:50.851 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2024-04-24T08:08:50.899 INFO:teuthology.orchestra.run.smithi121.stdout:40 2024-04-24T08:08:50.899 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-04-24T08:08:50.899 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-04-24T08:08:50.904 DEBUG:teuthology.orchestra.run.smithi121:> sudo modprobe fuse 2024-04-24T08:08:50.969 DEBUG:teuthology.orchestra.run.smithi121:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:08:51.009 INFO:teuthology.orchestra.run.smithi121.stdout:/proc 2024-04-24T08:08:51.009 INFO:teuthology.orchestra.run.smithi121.stdout:/sys 2024-04-24T08:08:51.009 INFO:teuthology.orchestra.run.smithi121.stdout:/dev 2024-04-24T08:08:51.009 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/security 2024-04-24T08:08:51.009 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/shm 2024-04-24T08:08:51.009 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/pts 2024-04-24T08:08:51.009 INFO:teuthology.orchestra.run.smithi121.stdout:/run 2024-04-24T08:08:51.009 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/cgroup 2024-04-24T08:08:51.009 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/pstore 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/bpf 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/ 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/selinux 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/mqueue 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/debug 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/dev/hugepages 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/tracing 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/fs/fuse/connections 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/sys/kernel/config 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:08:51.010 INFO:teuthology.orchestra.run.smithi121.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/run/user/1000 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-4 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-5 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-6 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/var/lib/ceph/osd/ceph-7 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T08:08:51.011 INFO:teuthology.orchestra.run.smithi121.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T08:08:51.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:51.013 DEBUG:teuthology.orchestra.run.smithi121:> ls /sys/fs/fuse/connections 2024-04-24T08:08:51.066 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi121.stderr:2024-04-24T08:08:51.066+0000 7f650d85b480 -1 init, newargv = 0x557c0dd326a0 newargc=15 2024-04-24T08:08:51.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi121.stderr:ceph-fuse[39806]: starting ceph client 2024-04-24T08:08:51.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi121.stderr:ceph-fuse[39806]: starting fuse 2024-04-24T08:08:51.095 INFO:teuthology.orchestra.run.smithi121.stdout:40 2024-04-24T08:08:51.096 INFO:teuthology.orchestra.run.smithi121.stdout:43 2024-04-24T08:08:51.096 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 43] 2024-04-24T08:08:51.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:08:51.096 DEBUG:teuthology.orchestra.run.smithi121:> import glob 2024-04-24T08:08:51.096 DEBUG:teuthology.orchestra.run.smithi121:> import re 2024-04-24T08:08:51.096 DEBUG:teuthology.orchestra.run.smithi121:> import os 2024-04-24T08:08:51.096 DEBUG:teuthology.orchestra.run.smithi121:> import subprocess 2024-04-24T08:08:51.096 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> def _find_admin_socket(client_name): 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> files = glob.glob(asok_path) 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> # Given a non-glob path, it better be there 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> if "*" not in asok_path: 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> assert(len(files) == 1) 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> return files[0] 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> for f in files: 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> contents = proc_f.read() 2024-04-24T08:08:51.097 DEBUG:teuthology.orchestra.run.smithi121:> if mountpoint in contents: 2024-04-24T08:08:51.098 DEBUG:teuthology.orchestra.run.smithi121:> return f 2024-04-24T08:08:51.098 DEBUG:teuthology.orchestra.run.smithi121:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:08:51.098 DEBUG:teuthology.orchestra.run.smithi121:> 2024-04-24T08:08:51.098 DEBUG:teuthology.orchestra.run.smithi121:> print(_find_admin_socket("client.3")) 2024-04-24T08:08:51.098 DEBUG:teuthology.orchestra.run.smithi121:> ' 2024-04-24T08:08:51.284 INFO:teuthology.orchestra.run.smithi121.stdout:/var/run/ceph/ceph-client.3.39806.asok 2024-04-24T08:08:51.288 INFO:teuthology.orchestra.run.smithi121.stderr:2024-04-24T08:08:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:08:51.300 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.39806.asok 2024-04-24T08:08:51.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:51.300 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.39806.asok status 2024-04-24T08:08:51.520 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-04-24T08:08:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "metadata": { 2024-04-24T08:08:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:08:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:08:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "entity_id": "3", 2024-04-24T08:08:51.520 INFO:teuthology.orchestra.run.smithi121.stdout: "hostname": "smithi121", 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "pid": "39806", 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "root": "/" 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_count": 0, 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "dentry_pinned_count": 0, 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "id": 4552, 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "inst": { 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "name": { 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "client", 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "num": 4552 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-04-24T08:08:51.521 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.2:0", 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2269598655 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": { 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "type": "v1", 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "addr": "192.168.0.2:0", 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "nonce": 2269598655 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "inst_str": "client.4552 192.168.0.2:0/2269598655", 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "addr_str": "192.168.0.2:0/2269598655", 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "inode_count": 1, 2024-04-24T08:08:51.522 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_epoch": 7, 2024-04-24T08:08:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch": 20, 2024-04-24T08:08:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:08:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "blocklisted": false, 2024-04-24T08:08:51.523 INFO:teuthology.orchestra.run.smithi121.stdout: "fs_name": "cephfs" 2024-04-24T08:08:51.523 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-04-24T08:08:51.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:51.534 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:08:51.534 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:08:51.593 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:08:51.593 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:08:51.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:51.593 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:08:51.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:51.680 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:08:51.680 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.2 2024-04-24T08:08:51.733 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:08:51.733 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-04-24T08:08:51.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:51.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-24T08:08:51.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:51.821 DEBUG:teuthology.orchestra.run.smithi121:> stat --file-system '--printf=%T 2024-04-24T08:08:51.821 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-24T08:08:51.882 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2024-04-24T08:08:51.883 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-24T08:08:51.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:51.883 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-24T08:08:51.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:51.968 DEBUG:teuthology.orchestra.run.smithi121:> stat --file-system '--printf=%T 2024-04-24T08:08:51.968 DEBUG:teuthology.orchestra.run.smithi121:> ' -- /home/ubuntu/cephtest/mnt.3 2024-04-24T08:08:52.022 INFO:teuthology.orchestra.run.smithi121.stdout:fuseblk 2024-04-24T08:08:52.023 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-04-24T08:08:52.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:08:52.023 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-24T08:08:52.099 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-24T08:08:52.158 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.365+0000 7f238ee00640 1 Processor -- start 2024-04-24T08:08:52.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.365+0000 7f238ee00640 1 -- start start 2024-04-24T08:08:52.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.367+0000 7f238ee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145a90 0x7f238806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.367+0000 7f238ee00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2388145fe0 0x7f2388160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.367+0000 7f238ee00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23880cf3d0 con 0x7f23880d0300 2024-04-24T08:08:52.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.367+0000 7f238ee00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f23880cf6b0 con 0x7f2388145fe0 2024-04-24T08:08:52.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.367+0000 7f238ee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f23880cf990 con 0x7f2388145a90 2024-04-24T08:08:52.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.368+0000 7f2387e00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f23880d0300 0x7f23880d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:50226/0 (socket says 172.21.15.69:50226) 2024-04-24T08:08:52.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.368+0000 7f2387e00640 1 -- 172.21.15.69:0/63556150 learned_addr learned my addr 172.21.15.69:0/63556150 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:52.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.368+0000 7f2387400640 1 --2- 172.21.15.69:0/63556150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145a90 0x7f238806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:52.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.368+0000 7f238ce00640 1 --2- 172.21.15.69:0/63556150 >> v2:172.21.15.121:3300/0 conn(0x7f2388145fe0 0x7f2388160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:52.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.369+0000 7f2387400640 1 -- 172.21.15.69:0/63556150 >> v2:172.21.15.121:3300/0 conn(0x7f2388145fe0 msgr2=0x7f2388160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:52.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.369+0000 7f2387400640 1 --2- 172.21.15.69:0/63556150 >> v2:172.21.15.121:3300/0 conn(0x7f2388145fe0 0x7f2388160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.369+0000 7f2387400640 1 -- 172.21.15.69:0/63556150 >> v1:172.21.15.121:6789/0 conn(0x7f23880d0300 legacy=0x7f23880d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:52.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.369+0000 7f2387400640 1 -- 172.21.15.69:0/63556150 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23880cfc70 con 0x7f2388145a90 2024-04-24T08:08:52.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.369+0000 7f2386a00640 1 -- 172.21.15.69:0/63556150 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2175265073 0 0) 0x7f23880cf3d0 con 0x7f23880d0300 2024-04-24T08:08:52.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.370+0000 7f2387400640 1 --2- 172.21.15.69:0/63556150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145a90 0x7f238806e240 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23740097a0 tx=0x7f237402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5617f75b26bb5e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:52.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.371+0000 7f2386a00640 1 -- 172.21.15.69:0/63556150 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374037070 con 0x7f2388145a90 2024-04-24T08:08:52.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.371+0000 7f2386a00640 1 -- 172.21.15.69:0/63556150 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2374032440 con 0x7f2388145a90 2024-04-24T08:08:52.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.371+0000 7f2386a00640 1 -- 172.21.15.69:0/63556150 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374032810 con 0x7f2388145a90 2024-04-24T08:08:52.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.371+0000 7f238ee00640 1 -- 172.21.15.69:0/63556150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145a90 msgr2=0x7f238806e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:52.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.371+0000 7f238ee00640 1 --2- 172.21.15.69:0/63556150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145a90 0x7f238806e240 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23740097a0 tx=0x7f237402a180 comp rx=0 tx=0).stop 2024-04-24T08:08:52.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.372+0000 7f238ee00640 1 -- 172.21.15.69:0/63556150 shutdown_connections 2024-04-24T08:08:52.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.372+0000 7f238ee00640 1 --2- 172.21.15.69:0/63556150 >> v2:172.21.15.121:3300/0 conn(0x7f2388145fe0 0x7f2388160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.372+0000 7f238ee00640 1 --2- 172.21.15.69:0/63556150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145a90 0x7f238806e240 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.372+0000 7f238ee00640 1 -- 172.21.15.69:0/63556150 >> 172.21.15.69:0/63556150 conn(0x7f2388067d90 msgr2=0x7f238806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:52.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.372+0000 7f238ee00640 1 -- 172.21.15.69:0/63556150 shutdown_connections 2024-04-24T08:08:52.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.372+0000 7f238ee00640 1 -- 172.21.15.69:0/63556150 wait complete. 2024-04-24T08:08:52.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.373+0000 7f238ee00640 1 Processor -- start 2024-04-24T08:08:52.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.374+0000 7f238ee00640 1 -- start start 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.374+0000 7f238ee00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23880d0300 0x7f2388137b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.374+0000 7f238ee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2388145a90 0x7f2388141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.374+0000 7f238ee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145fe0 0x7f2388072310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f238ee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f23881458f0 con 0x7f2388145a90 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f238ee00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2388145770 con 0x7f23880d0300 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f238ee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f23880cdc30 con 0x7f2388145fe0 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f2387400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2388145a90 0x7f2388141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f2387400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2388145a90 0x7f2388141320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37782/0 (socket says 172.21.15.69:37782) 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f2387400640 1 -- 172.21.15.69:0/3001705324 learned_addr learned my addr 172.21.15.69:0/3001705324 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f2387e00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23880d0300 0x7f2388137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f238ce00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145fe0 0x7f2388072310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:52.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f2387400640 1 -- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145fe0 msgr2=0x7f2388072310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:52.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f2387400640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145fe0 0x7f2388072310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f2387400640 1 -- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23880d0300 msgr2=0x7f2388137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:52.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f2387400640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23880d0300 0x7f2388137b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.375+0000 7f2387400640 1 -- 172.21.15.69:0/3001705324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23880b41d0 con 0x7f2388145a90 2024-04-24T08:08:52.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.376+0000 7f2387400640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2388145a90 0x7f2388141320 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2374035040 tx=0x7f23740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:52.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.376+0000 7f237fe00640 1 -- 172.21.15.69:0/3001705324 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374037040 con 0x7f2388145a90 2024-04-24T08:08:52.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.376+0000 7f237fe00640 1 -- 172.21.15.69:0/3001705324 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2374030500 con 0x7f2388145a90 2024-04-24T08:08:52.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.376+0000 7f238ee00640 1 -- 172.21.15.69:0/3001705324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23880b44b0 con 0x7f2388145a90 2024-04-24T08:08:52.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.376+0000 7f237fe00640 1 -- 172.21.15.69:0/3001705324 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374030910 con 0x7f2388145a90 2024-04-24T08:08:52.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.376+0000 7f238ee00640 1 -- 172.21.15.69:0/3001705324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23880b4a70 con 0x7f2388145a90 2024-04-24T08:08:52.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.377+0000 7f238ee00640 1 -- 172.21.15.69:0/3001705324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2388068a80 con 0x7f2388145a90 2024-04-24T08:08:52.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.378+0000 7f237fe00640 1 -- 172.21.15.69:0/3001705324 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2374044020 con 0x7f2388145a90 2024-04-24T08:08:52.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.379+0000 7f237fe00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f236806c8f0 0x7f236806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.379+0000 7f237fe00640 1 -- 172.21.15.69:0/3001705324 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f23740abec0 con 0x7f2388145a90 2024-04-24T08:08:52.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.380+0000 7f2387e00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f236806c8f0 0x7f236806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:52.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.381+0000 7f2387e00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f236806c8f0 0x7f236806edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2378000f60 tx=0x7f2378007c00 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:52.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.384+0000 7f237fe00640 1 -- 172.21.15.69:0/3001705324 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f237407a1c0 con 0x7f2388145a90 2024-04-24T08:08:52.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.587+0000 7f238ee00640 1 -- 172.21.15.69:0/3001705324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23881450e0 con 0x7f2388145a90 2024-04-24T08:08:52.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.588+0000 7f237fe00640 1 -- 172.21.15.69:0/3001705324 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7f2374079ac0 con 0x7f2388145a90 2024-04-24T08:08:52.589 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:52.589 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:52.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 -- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f236806c8f0 msgr2=0x7f236806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f236806c8f0 0x7f236806edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2378000f60 tx=0x7f2378007c00 comp rx=0 tx=0).stop 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 -- 172.21.15.69:0/3001705324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2388145a90 msgr2=0x7f2388141320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2388145a90 0x7f2388141320 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2374035040 tx=0x7f23740046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 -- 172.21.15.69:0/3001705324 shutdown_connections 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f236806c8f0 0x7f236806edb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2388145a90 0x7f2388141320 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2388145fe0 0x7f2388072310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 --2- 172.21.15.69:0/3001705324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f23880d0300 0x7f2388137b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 -- 172.21.15.69:0/3001705324 >> 172.21.15.69:0/3001705324 conn(0x7f2388067d90 msgr2=0x7f23880ccdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 -- 172.21.15.69:0/3001705324 shutdown_connections 2024-04-24T08:08:52.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.591+0000 7f238ee00640 1 -- 172.21.15.69:0/3001705324 wait complete. 2024-04-24T08:08:52.610 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:08:52.610 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:08:52.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.814+0000 7fdcf5200640 1 Processor -- start 2024-04-24T08:08:52.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.815+0000 7fdcf5200640 1 -- start start 2024-04-24T08:08:52.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.816+0000 7fdcf5200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcf015ea60 0x7fdcf00cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.816+0000 7fdcf5200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdcf00d36a0 0x7fdcf00d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.816+0000 7fdcf5200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf00d2600 con 0x7fdcf00d11b0 2024-04-24T08:08:52.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.816+0000 7fdcf5200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fdcf00d23b0 con 0x7fdcf00d36a0 2024-04-24T08:08:52.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.816+0000 7fdcf5200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf006dba0 con 0x7fdcf015ea60 2024-04-24T08:08:52.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.817+0000 7fdced400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdcf00d36a0 0x7fdcf00d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:52.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.817+0000 7fdced400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdcf00d36a0 0x7fdcf00d04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37796/0 (socket says 172.21.15.69:37796) 2024-04-24T08:08:52.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.817+0000 7fdced400640 1 -- 172.21.15.69:0/3400206562 learned_addr learned my addr 172.21.15.69:0/3400206562 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:52.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.817+0000 7fdcede00640 1 --2- 172.21.15.69:0/3400206562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcf015ea60 0x7fdcf00cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:52.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.817+0000 7fdced400640 1 -- 172.21.15.69:0/3400206562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcf015ea60 msgr2=0x7fdcf00cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:52.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.817+0000 7fdced400640 1 --2- 172.21.15.69:0/3400206562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcf015ea60 0x7fdcf00cff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.817+0000 7fdced400640 1 -- 172.21.15.69:0/3400206562 >> v1:172.21.15.69:6789/0 conn(0x7fdcf00d11b0 legacy=0x7fdcf006de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:52.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.817+0000 7fdced400640 1 -- 172.21.15.69:0/3400206562 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf00d28e0 con 0x7fdcf00d36a0 2024-04-24T08:08:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.818+0000 7fdced400640 1 --2- 172.21.15.69:0/3400206562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d36a0 0x7fdcf00d04b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009870 tx=0x7fdcdc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3612f11f5318c93 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.819+0000 7fdceca00640 1 -- 172.21.15.69:0/3400206562 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc037070 con 0x7fdcf00d36a0 2024-04-24T08:08:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.819+0000 7fdceca00640 1 -- 172.21.15.69:0/3400206562 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcdc032440 con 0x7fdcf00d36a0 2024-04-24T08:08:52.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.819+0000 7fdceca00640 1 -- 172.21.15.69:0/3400206562 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcdc032810 con 0x7fdcf00d36a0 2024-04-24T08:08:52.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.819+0000 7fdcf5200640 1 -- 172.21.15.69:0/3400206562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d36a0 msgr2=0x7fdcf00d04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:52.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.819+0000 7fdcf5200640 1 --2- 172.21.15.69:0/3400206562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d36a0 0x7fdcf00d04b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc009870 tx=0x7fdcdc02a250 comp rx=0 tx=0).stop 2024-04-24T08:08:52.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.820+0000 7fdcf5200640 1 -- 172.21.15.69:0/3400206562 shutdown_connections 2024-04-24T08:08:52.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.820+0000 7fdcf5200640 1 --2- 172.21.15.69:0/3400206562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d36a0 0x7fdcf00d04b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.820+0000 7fdcf5200640 1 --2- 172.21.15.69:0/3400206562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcf015ea60 0x7fdcf00cff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.820+0000 7fdcf5200640 1 -- 172.21.15.69:0/3400206562 >> 172.21.15.69:0/3400206562 conn(0x7fdcf008bf60 msgr2=0x7fdcf00d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.820+0000 7fdcf5200640 1 -- 172.21.15.69:0/3400206562 shutdown_connections 2024-04-24T08:08:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.820+0000 7fdcf5200640 1 -- 172.21.15.69:0/3400206562 wait complete. 2024-04-24T08:08:52.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.821+0000 7fdcf5200640 1 Processor -- start 2024-04-24T08:08:52.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.822+0000 7fdcf5200640 1 -- start start 2024-04-24T08:08:52.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcf5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d11b0 0x7fdcf0157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcf5200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcf00d36a0 0x7fdcf00b9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcf5200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcf015ea60 0x7fdcf0160ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcf5200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf00d3180 con 0x7fdcf00d11b0 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcf5200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf00d2ea0 con 0x7fdcf00d36a0 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcf5200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf015ee40 con 0x7fdcf015ea60 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcede00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d11b0 0x7fdcf0157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcede00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d11b0 0x7fdcf0157db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37800/0 (socket says 172.21.15.69:37800) 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcede00640 1 -- 172.21.15.69:0/1733083887 learned_addr learned my addr 172.21.15.69:0/1733083887 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcede00640 1 -- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcf015ea60 msgr2=0x7fdcf0160ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcede00640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcf015ea60 0x7fdcf0160ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcede00640 1 -- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcf00d36a0 msgr2=0x7fdcf00b9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcede00640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcf00d36a0 0x7fdcf00b9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.823+0000 7fdcede00640 1 -- 172.21.15.69:0/1733083887 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf01582f0 con 0x7fdcf00d11b0 2024-04-24T08:08:52.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.824+0000 7fdcede00640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d11b0 0x7fdcf0157db0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdce000d830 tx=0x7fdce000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:52.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.824+0000 7fdce6000640 1 -- 172.21.15.69:0/1733083887 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce000b6c0 con 0x7fdcf00d11b0 2024-04-24T08:08:52.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.824+0000 7fdce6000640 1 -- 172.21.15.69:0/1733083887 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce0019070 con 0x7fdcf00d11b0 2024-04-24T08:08:52.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.824+0000 7fdcf5200640 1 -- 172.21.15.69:0/1733083887 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf00b8380 con 0x7fdcf00d11b0 2024-04-24T08:08:52.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.824+0000 7fdce6000640 1 -- 172.21.15.69:0/1733083887 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce000ba80 con 0x7fdcf00d11b0 2024-04-24T08:08:52.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.824+0000 7fdcf5200640 1 -- 172.21.15.69:0/1733083887 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf00b8940 con 0x7fdcf00d11b0 2024-04-24T08:08:52.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.825+0000 7fdcf5200640 1 -- 172.21.15.69:0/1733083887 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcf0068a80 con 0x7fdcf00d11b0 2024-04-24T08:08:52.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.826+0000 7fdce6000640 1 -- 172.21.15.69:0/1733083887 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdce001e460 con 0x7fdcf00d11b0 2024-04-24T08:08:52.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.827+0000 7fdce6000640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcd006c8f0 0x7fdcd006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:52.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.827+0000 7fdce6000640 1 -- 172.21.15.69:0/1733083887 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fdce008c240 con 0x7fdcf00d11b0 2024-04-24T08:08:52.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.828+0000 7fdced400640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcd006c8f0 0x7fdcd006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:52.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.829+0000 7fdced400640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcd006c8f0 0x7fdcd006edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc034000 tx=0x7fdcdc02c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:52.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:52.832+0000 7fdce6000640 1 -- 172.21.15.69:0/1733083887 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdcf01582f0 con 0x7fdcf00d11b0 2024-04-24T08:08:53.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.030+0000 7fdcf5200640 1 -- 172.21.15.69:0/1733083887 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdcf006e920 con 0x7fdcf00d11b0 2024-04-24T08:08:53.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.031+0000 7fdce6000640 1 -- 172.21.15.69:0/1733083887 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fdce005a660 con 0x7fdcf00d11b0 2024-04-24T08:08:53.031 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:08:53.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.033+0000 7fdcf5200640 1 -- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcd006c8f0 msgr2=0x7fdcd006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.033+0000 7fdcf5200640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcd006c8f0 0x7fdcd006edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdcdc034000 tx=0x7fdcdc02c000 comp rx=0 tx=0).stop 2024-04-24T08:08:53.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.033+0000 7fdcf5200640 1 -- 172.21.15.69:0/1733083887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d11b0 msgr2=0x7fdcf0157db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.033+0000 7fdcf5200640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d11b0 0x7fdcf0157db0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdce000d830 tx=0x7fdce000dcf0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.033+0000 7fdcf5200640 1 -- 172.21.15.69:0/1733083887 shutdown_connections 2024-04-24T08:08:53.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.033+0000 7fdcf5200640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcd006c8f0 0x7fdcd006edb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.033+0000 7fdcf5200640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdcf015ea60 0x7fdcf0160ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.033+0000 7fdcf5200640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdcf00d36a0 0x7fdcf00b9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.033+0000 7fdcf5200640 1 --2- 172.21.15.69:0/1733083887 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdcf00d11b0 0x7fdcf0157db0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.033+0000 7fdcf5200640 1 -- 172.21.15.69:0/1733083887 >> 172.21.15.69:0/1733083887 conn(0x7fdcf008bf60 msgr2=0x7fdcf00b62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:53.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.034+0000 7fdcf5200640 1 -- 172.21.15.69:0/1733083887 shutdown_connections 2024-04-24T08:08:53.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.034+0000 7fdcf5200640 1 -- 172.21.15.69:0/1733083887 wait complete. 2024-04-24T08:08:53.052 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:08:53.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.261+0000 7f781cc00640 1 Processor -- start 2024-04-24T08:08:53.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.261+0000 7f781cc00640 1 -- start start 2024-04-24T08:08:53.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.262+0000 7f781cc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f781815c820 0x7f78180b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.262+0000 7f781cc00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f78180b46a0 0x7f781806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.263+0000 7f781cc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f781815cc00 con 0x7f781806fd10 2024-04-24T08:08:53.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.263+0000 7f781cc00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f781806ed30 con 0x7f78180b46a0 2024-04-24T08:08:53.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.263+0000 7f781cc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f78180b4e10 con 0x7f781815c820 2024-04-24T08:08:53.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.263+0000 7f7816800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f781806fd10 0x7f78180ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47466/0 (socket says 172.21.15.69:47466) 2024-04-24T08:08:53.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.263+0000 7f7816800640 1 -- 172.21.15.69:0/203379691 learned_addr learned my addr 172.21.15.69:0/203379691 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.263+0000 7f7815e00640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f781815c820 0x7f78180b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7815400640 1 --2- 172.21.15.69:0/203379691 >> v2:172.21.15.121:3300/0 conn(0x7f78180b46a0 0x7f781806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 914539377 0 0) 0x7f781815cc00 con 0x7f781806fd10 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77fc003520 con 0x7f781806fd10 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1988386711 0 0) 0x7f77fc003520 con 0x7f781806fd10 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f781815c820 msgr2=0x7f78180b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f781815c820 0x7f78180b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 >> v2:172.21.15.121:3300/0 conn(0x7f78180b46a0 msgr2=0x7f781806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 --2- 172.21.15.69:0/203379691 >> v2:172.21.15.121:3300/0 conn(0x7f78180b46a0 0x7f781806f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78180b50f0 con 0x7f781806fd10 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f78100019f0 con 0x7f781806fd10 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 >> v1:172.21.15.69:6789/0 conn(0x7f781806fd10 legacy=0x7f78180ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77fc0039b0 0x7f77fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f781815c820 0x7f77fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77fc004d40 0x7f77fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f781806ed30 con 0x7f77fc004d40 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f78180b4e10 con 0x7f77fc0039b0 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.264+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f77fc00b9c0 con 0x7f781815c820 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.265+0000 7f7816800640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77fc004d40 0x7f77fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.265+0000 7f7815e00640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77fc0039b0 0x7f77fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.265+0000 7f7815400640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f781815c820 0x7f77fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.265+0000 7f7816800640 1 -- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f781815c820 msgr2=0x7f77fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.265+0000 7f7816800640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f781815c820 0x7f77fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.265+0000 7f7816800640 1 -- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77fc0039b0 msgr2=0x7f77fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.265+0000 7f7816800640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77fc0039b0 0x7f77fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.265+0000 7f7816800640 1 -- 172.21.15.69:0/203379691 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77fc003520 con 0x7f77fc004d40 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.266+0000 7f7816800640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77fc004d40 0x7f77fc0051c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f781000a920 tx=0x7f77fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b414d35cb6457d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.266+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781000a4c0 con 0x7f77fc004d40 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.266+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f781002f070 con 0x7f77fc004d40 2024-04-24T08:08:53.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.266+0000 7f7814a00640 1 -- 172.21.15.69:0/203379691 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781002bd00 con 0x7f77fc004d40 2024-04-24T08:08:53.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.267+0000 7f781cc00640 1 -- 172.21.15.69:0/203379691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77fc004d40 msgr2=0x7f77fc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.267+0000 7f781cc00640 1 --2- 172.21.15.69:0/203379691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f77fc004d40 0x7f77fc0051c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f781000a920 tx=0x7f77fc005830 comp rx=0 tx=0).stop 2024-04-24T08:08:53.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.267+0000 7f7816800640 1 -- 172.21.15.69:0/203379691 reap_dead start 2024-04-24T08:08:53.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.267+0000 7f781cc00640 1 -- 172.21.15.69:0/203379691 shutdown_connections 2024-04-24T08:08:53.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.267+0000 7f781cc00640 1 -- 172.21.15.69:0/203379691 >> 172.21.15.69:0/203379691 conn(0x7f78180d0350 msgr2=0x7f78180d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:53.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.267+0000 7f781cc00640 1 -- 172.21.15.69:0/203379691 shutdown_connections 2024-04-24T08:08:53.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.267+0000 7f781cc00640 1 -- 172.21.15.69:0/203379691 wait complete. 2024-04-24T08:08:53.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.269+0000 7f781cc00640 1 Processor -- start 2024-04-24T08:08:53.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.269+0000 7f781cc00640 1 -- start start 2024-04-24T08:08:53.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.270+0000 7f781cc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7818149650 0x7f781815f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.270+0000 7f781cc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f781815d7f0 0x7f78181f06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.270+0000 7f781cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78181f0c20 0x7f7818157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.270+0000 7f781cc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f78180b4b00 con 0x7f78181f0c20 2024-04-24T08:08:53.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.270+0000 7f781cc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f781806f2b0 con 0x7f781815d7f0 2024-04-24T08:08:53.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.270+0000 7f781cc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f78180ce960 con 0x7f7818149650 2024-04-24T08:08:53.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.270+0000 7f7816800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78181f0c20 0x7f7818157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.270+0000 7f7816800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78181f0c20 0x7f7818157680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37826/0 (socket says 172.21.15.69:37826) 2024-04-24T08:08:53.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.270+0000 7f7816800640 1 -- 172.21.15.69:0/2411864906 learned_addr learned my addr 172.21.15.69:0/2411864906 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:53.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f7815400640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f781815d7f0 0x7f78181f06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f7816800640 1 -- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7818149650 msgr2=0x7f781815f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f7816800640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7818149650 0x7f781815f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f7816800640 1 -- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f781815d7f0 msgr2=0x7f78181f06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f7816800640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f781815d7f0 0x7f78181f06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f7816800640 1 -- 172.21.15.69:0/2411864906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781815dfd0 con 0x7f78181f0c20 2024-04-24T08:08:53.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f7815400640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f781815d7f0 0x7f78181f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:53.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f7816800640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78181f0c20 0x7f7818157680 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7810001cf0 tx=0x7f7810001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:53.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f780e000640 1 -- 172.21.15.69:0/2411864906 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781000a4c0 con 0x7f78181f0c20 2024-04-24T08:08:53.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f780e000640 1 -- 172.21.15.69:0/2411864906 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78100019e0 con 0x7f78181f0c20 2024-04-24T08:08:53.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f780e000640 1 -- 172.21.15.69:0/2411864906 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7810002000 con 0x7f78181f0c20 2024-04-24T08:08:53.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f781cc00640 1 -- 172.21.15.69:0/2411864906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f781815dcf0 con 0x7f78181f0c20 2024-04-24T08:08:53.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.271+0000 7f781cc00640 1 -- 172.21.15.69:0/2411864906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7818156e80 con 0x7f78181f0c20 2024-04-24T08:08:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.273+0000 7f781cc00640 1 -- 172.21.15.69:0/2411864906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7818068a80 con 0x7f78181f0c20 2024-04-24T08:08:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.273+0000 7f780e000640 1 -- 172.21.15.69:0/2411864906 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7810034670 con 0x7f78181f0c20 2024-04-24T08:08:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.274+0000 7f780e000640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77f806c8f0 0x7f77f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.274+0000 7f780e000640 1 -- 172.21.15.69:0/2411864906 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f78100abd90 con 0x7f78181f0c20 2024-04-24T08:08:53.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.275+0000 7f7815e00640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77f806c8f0 0x7f77f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.276+0000 7f7815e00640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77f806c8f0 0x7f77f806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7808009890 tx=0x7f7808008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:53.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.279+0000 7f780e000640 1 -- 172.21.15.69:0/2411864906 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7810046060 con 0x7f78181f0c20 2024-04-24T08:08:53.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.479+0000 7f781cc00640 1 -- 172.21.15.69:0/2411864906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f78180cddc0 con 0x7f78181f0c20 2024-04-24T08:08:53.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.480+0000 7f780e000640 1 -- 172.21.15.69:0/2411864906 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f781007a0a0 con 0x7f78181f0c20 2024-04-24T08:08:53.480 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:08:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.482+0000 7f781cc00640 1 -- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77f806c8f0 msgr2=0x7f77f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.482+0000 7f781cc00640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77f806c8f0 0x7f77f806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7808009890 tx=0x7f7808008040 comp rx=0 tx=0).stop 2024-04-24T08:08:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.483+0000 7f781cc00640 1 -- 172.21.15.69:0/2411864906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78181f0c20 msgr2=0x7f7818157680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.483+0000 7f781cc00640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78181f0c20 0x7f7818157680 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7810001cf0 tx=0x7f7810001dc0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.483+0000 7f781cc00640 1 -- 172.21.15.69:0/2411864906 shutdown_connections 2024-04-24T08:08:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.483+0000 7f781cc00640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77f806c8f0 0x7f77f806edb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.483+0000 7f781cc00640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78181f0c20 0x7f7818157680 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.483+0000 7f781cc00640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f781815d7f0 0x7f78181f06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.483+0000 7f781cc00640 1 --2- 172.21.15.69:0/2411864906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7818149650 0x7f781815f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.483+0000 7f781cc00640 1 -- 172.21.15.69:0/2411864906 >> 172.21.15.69:0/2411864906 conn(0x7f78180d0350 msgr2=0x7f78180cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.483+0000 7f781cc00640 1 -- 172.21.15.69:0/2411864906 shutdown_connections 2024-04-24T08:08:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.483+0000 7f781cc00640 1 -- 172.21.15.69:0/2411864906 wait complete. 2024-04-24T08:08:53.504 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:08:53.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.705+0000 7faf3be00640 1 Processor -- start 2024-04-24T08:08:53.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.705+0000 7faf3be00640 1 -- start start 2024-04-24T08:08:53.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.707+0000 7faf3be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7faf3c15ea60 0x7faf3c0b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.707+0000 7faf3be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf3c0cdf30 0x7faf3c0d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.707+0000 7faf3be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faf3c0ce390 0x7faf3c0d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.707+0000 7faf3be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7faf3c0ce910 con 0x7faf3c0ce390 2024-04-24T08:08:53.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.707+0000 7faf3be00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7faf3c0cebf0 con 0x7faf3c15ea60 2024-04-24T08:08:53.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.707+0000 7faf3be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faf3c0ceed0 con 0x7faf3c0cdf30 2024-04-24T08:08:53.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.707+0000 7faf3b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faf3c0ce390 0x7faf3c0d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.708+0000 7faf3b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7faf3c0ce390 0x7faf3c0d0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37830/0 (socket says 172.21.15.69:37830) 2024-04-24T08:08:53.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.708+0000 7faf3b400640 1 -- 172.21.15.69:0/2189409467 learned_addr learned my addr 172.21.15.69:0/2189409467 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:53.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.708+0000 7faf3aa00640 1 --2- 172.21.15.69:0/2189409467 >> v2:172.21.15.121:3300/0 conn(0x7faf3c15ea60 0x7faf3c0b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.708+0000 7faf33e00640 1 --2- 172.21.15.69:0/2189409467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf3c0cdf30 0x7faf3c0d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.708+0000 7faf3b400640 1 -- 172.21.15.69:0/2189409467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf3c0cdf30 msgr2=0x7faf3c0d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:53.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.708+0000 7faf3b400640 1 --2- 172.21.15.69:0/2189409467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf3c0cdf30 0x7faf3c0d02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.708+0000 7faf3b400640 1 -- 172.21.15.69:0/2189409467 >> v2:172.21.15.121:3300/0 conn(0x7faf3c15ea60 msgr2=0x7faf3c0b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:53.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.708+0000 7faf3b400640 1 --2- 172.21.15.69:0/2189409467 >> v2:172.21.15.121:3300/0 conn(0x7faf3c15ea60 0x7faf3c0b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.708+0000 7faf3b400640 1 -- 172.21.15.69:0/2189409467 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3c0cf1b0 con 0x7faf3c0ce390 2024-04-24T08:08:53.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.709+0000 7faf3b400640 1 --2- 172.21.15.69:0/2189409467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 0x7faf3c0d0820 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0730e0 tx=0x7faf3402c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d881e77d12cea7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.709+0000 7faf3a000640 1 -- 172.21.15.69:0/2189409467 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34039070 con 0x7faf3c0ce390 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3a000640 1 -- 172.21.15.69:0/2189409467 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf34034410 con 0x7faf3c0ce390 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3a000640 1 -- 172.21.15.69:0/2189409467 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf340347e0 con 0x7faf3c0ce390 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3be00640 1 -- 172.21.15.69:0/2189409467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 msgr2=0x7faf3c0d0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3be00640 1 --2- 172.21.15.69:0/2189409467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 0x7faf3c0d0820 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7faf3c0730e0 tx=0x7faf3402c820 comp rx=0 tx=0).stop 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3be00640 1 -- 172.21.15.69:0/2189409467 shutdown_connections 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3be00640 1 --2- 172.21.15.69:0/2189409467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 0x7faf3c0d0820 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3be00640 1 --2- 172.21.15.69:0/2189409467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf3c0cdf30 0x7faf3c0d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3be00640 1 --2- 172.21.15.69:0/2189409467 >> v2:172.21.15.121:3300/0 conn(0x7faf3c15ea60 0x7faf3c0b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3be00640 1 -- 172.21.15.69:0/2189409467 >> 172.21.15.69:0/2189409467 conn(0x7faf3c067b80 msgr2=0x7faf3c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:53.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3be00640 1 -- 172.21.15.69:0/2189409467 shutdown_connections 2024-04-24T08:08:53.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.710+0000 7faf3be00640 1 -- 172.21.15.69:0/2189409467 wait complete. 2024-04-24T08:08:53.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.712+0000 7faf3be00640 1 Processor -- start 2024-04-24T08:08:53.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.712+0000 7faf3be00640 1 -- start start 2024-04-24T08:08:53.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.713+0000 7faf3be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf3c0cdf30 0x7faf3c14f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.713+0000 7faf3be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 0x7faf3c14fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.713+0000 7faf3be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf3c15ea60 0x7faf3c152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.713+0000 7faf3be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c0b34a0 con 0x7faf3c0ce390 2024-04-24T08:08:53.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.713+0000 7faf3be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c0b3320 con 0x7faf3c0cdf30 2024-04-24T08:08:53.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.713+0000 7faf3be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7faf3c0b3620 con 0x7faf3c15ea60 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.713+0000 7faf33e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 0x7faf3c14fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.713+0000 7faf33e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 0x7faf3c14fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37834/0 (socket says 172.21.15.69:37834) 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.714+0000 7faf33e00640 1 -- 172.21.15.69:0/4231066301 learned_addr learned my addr 172.21.15.69:0/4231066301 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.714+0000 7faf3aa00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf3c0cdf30 0x7faf3c14f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.714+0000 7faf3b400640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf3c15ea60 0x7faf3c152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.714+0000 7faf33e00640 1 -- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf3c15ea60 msgr2=0x7faf3c152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.714+0000 7faf33e00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf3c15ea60 0x7faf3c152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.714+0000 7faf33e00640 1 -- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf3c0cdf30 msgr2=0x7faf3c14f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.714+0000 7faf33e00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf3c0cdf30 0x7faf3c14f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.714+0000 7faf33e00640 1 -- 172.21.15.69:0/4231066301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3c150ea0 con 0x7faf3c0ce390 2024-04-24T08:08:53.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.714+0000 7faf33e00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 0x7faf3c14fb30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faf2400d800 tx=0x7faf2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:53.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.715+0000 7faf32a00640 1 -- 172.21.15.69:0/4231066301 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24018020 con 0x7faf3c0ce390 2024-04-24T08:08:53.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.715+0000 7faf32a00640 1 -- 172.21.15.69:0/4231066301 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf24009640 con 0x7faf3c0ce390 2024-04-24T08:08:53.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.715+0000 7faf3be00640 1 -- 172.21.15.69:0/4231066301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf3c151180 con 0x7faf3c0ce390 2024-04-24T08:08:53.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.715+0000 7faf32a00640 1 -- 172.21.15.69:0/4231066301 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf24009a10 con 0x7faf3c0ce390 2024-04-24T08:08:53.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.715+0000 7faf3be00640 1 -- 172.21.15.69:0/4231066301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf3c14f0d0 con 0x7faf3c0ce390 2024-04-24T08:08:53.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.716+0000 7faf3be00640 1 -- 172.21.15.69:0/4231066301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf3c151580 con 0x7faf3c0ce390 2024-04-24T08:08:53.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.717+0000 7faf32a00640 1 -- 172.21.15.69:0/4231066301 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faf2401b020 con 0x7faf3c0ce390 2024-04-24T08:08:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.718+0000 7faf32a00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7faf1806c920 0x7faf1806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.718+0000 7faf32a00640 1 -- 172.21.15.69:0/4231066301 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7faf2408c130 con 0x7faf3c0ce390 2024-04-24T08:08:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.718+0000 7faf3aa00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7faf1806c920 0x7faf1806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.720+0000 7faf3aa00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7faf1806c920 0x7faf1806ede0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009a30 tx=0x7faf2c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.722+0000 7faf32a00640 1 -- 172.21.15.69:0/4231066301 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faf24022020 con 0x7faf3c0ce390 2024-04-24T08:08:53.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.920+0000 7faf3be00640 1 -- 172.21.15.69:0/4231066301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faf3c075d50 con 0x7faf3c0ce390 2024-04-24T08:08:53.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.921+0000 7faf32a00640 1 -- 172.21.15.69:0/4231066301 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7faf24011340 con 0x7faf3c0ce390 2024-04-24T08:08:53.922 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:08:53.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 -- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7faf1806c920 msgr2=0x7faf1806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7faf1806c920 0x7faf1806ede0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009a30 tx=0x7faf2c008040 comp rx=0 tx=0).stop 2024-04-24T08:08:53.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 -- 172.21.15.69:0/4231066301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 msgr2=0x7faf3c14fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 0x7faf3c14fb30 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faf2400d800 tx=0x7faf2400dcc0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 -- 172.21.15.69:0/4231066301 shutdown_connections 2024-04-24T08:08:53.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7faf1806c920 0x7faf1806ede0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faf3c0ce390 0x7faf3c14fb30 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7faf3c15ea60 0x7faf3c152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 --2- 172.21.15.69:0/4231066301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf3c0cdf30 0x7faf3c14f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 -- 172.21.15.69:0/4231066301 >> 172.21.15.69:0/4231066301 conn(0x7faf3c067b80 msgr2=0x7faf3c15dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:53.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.923+0000 7faf3be00640 1 -- 172.21.15.69:0/4231066301 shutdown_connections 2024-04-24T08:08:53.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:53.924+0000 7faf3be00640 1 -- 172.21.15.69:0/4231066301 wait complete. 2024-04-24T08:08:53.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:54.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.140+0000 7f5a0f200640 1 Processor -- start 2024-04-24T08:08:54.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.140+0000 7f5a0f200640 1 -- start start 2024-04-24T08:08:54.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.142+0000 7f5a0f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a1015d1d0 0x7f5a100ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.142+0000 7f5a0f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1015d660 con 0x7f5a100b30f0 2024-04-24T08:08:54.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.142+0000 7f5a0f200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a1006dc30 con 0x7f5a10156c60 2024-04-24T08:08:54.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.142+0000 7f5a0f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5a1006df10 con 0x7f5a1015d1d0 2024-04-24T08:08:54.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.142+0000 7f5a0e800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5a100b30f0 0x7f5a1006e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47482/0 (socket says 172.21.15.69:47482) 2024-04-24T08:08:54.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.142+0000 7f5a0e800640 1 -- 172.21.15.69:0/2171078084 learned_addr learned my addr 172.21.15.69:0/2171078084 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:54.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.142+0000 7f5a07e00640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a1015d1d0 0x7f5a100ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 34165503 0 0) 0x7f5a1015d660 con 0x7f5a100b30f0 2024-04-24T08:08:54.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59f4003520 con 0x7f5a100b30f0 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 309093539 0 0) 0x7f59f4003520 con 0x7f5a100b30f0 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a1015d1d0 msgr2=0x7f5a100ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5a1015d1d0 0x7f5a100ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 >> v1:172.21.15.121:6789/0 conn(0x7f5a10156c60 legacy=0x7f5a1015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a1006e1f0 con 0x7f5a100b30f0 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f5a080019f0 con 0x7f5a100b30f0 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 >> v1:172.21.15.69:6789/0 conn(0x7f5a100b30f0 legacy=0x7f5a1006e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59f40039e0 0x7f59f4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f4004800 0x7f59f4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59f40051c0 0x7f59f400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1006df10 con 0x7f59f40039e0 2024-04-24T08:08:54.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f59f400bf00 con 0x7f59f4004800 2024-04-24T08:08:54.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.143+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f59f400c080 con 0x7f59f40051c0 2024-04-24T08:08:54.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.144+0000 7f5a0de00640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59f40039e0 0x7f59f4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.144+0000 7f5a07e00640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f4004800 0x7f59f4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.144+0000 7f5a0de00640 1 -- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59f40051c0 msgr2=0x7f59f400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.144+0000 7f5a0e800640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59f40051c0 0x7f59f400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.144+0000 7f5a0de00640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59f40051c0 0x7f59f400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.144+0000 7f5a0de00640 1 -- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f4004800 msgr2=0x7f59f4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.144+0000 7f5a0de00640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f4004800 0x7f59f4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.144+0000 7f5a0de00640 1 -- 172.21.15.69:0/2171078084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f4003520 con 0x7f59f40039e0 2024-04-24T08:08:54.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.144+0000 7f5a07e00640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59f4004800 0x7f59f4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:54.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.145+0000 7f5a0de00640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59f40039e0 0x7f59f4003e00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5a00001750 tx=0x7f59f4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ce9bd5b1789b71 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:54.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.145+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00001370 con 0x7f59f40039e0 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.145+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a000052d0 con 0x7f59f40039e0 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.145+0000 7f5a0d400640 1 -- 172.21.15.69:0/2171078084 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0002abe0 con 0x7f59f40039e0 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.146+0000 7f5a0f200640 1 -- 172.21.15.69:0/2171078084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59f40039e0 msgr2=0x7f59f4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.146+0000 7f5a0f200640 1 --2- 172.21.15.69:0/2171078084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f59f40039e0 0x7f59f4003e00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5a00001750 tx=0x7f59f4005830 comp rx=0 tx=0).stop 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.146+0000 7f5a0e800640 1 -- 172.21.15.69:0/2171078084 reap_dead start 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.146+0000 7f5a0f200640 1 -- 172.21.15.69:0/2171078084 shutdown_connections 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.146+0000 7f5a0f200640 1 -- 172.21.15.69:0/2171078084 >> 172.21.15.69:0/2171078084 conn(0x7f5a100d0160 msgr2=0x7f5a100d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.146+0000 7f5a0f200640 1 -- 172.21.15.69:0/2171078084 shutdown_connections 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.146+0000 7f5a0f200640 1 -- 172.21.15.69:0/2171078084 wait complete. 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.148+0000 7f5a0f200640 1 Processor -- start 2024-04-24T08:08:54.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.148+0000 7f5a0f200640 1 -- start start 2024-04-24T08:08:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.149+0000 7f5a0f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59f40039e0 0x7f5a101f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.149+0000 7f5a0f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a10071be0 0x7f5a100c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.149+0000 7f5a0f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a10154be0 0x7f5a1015ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.149+0000 7f5a0f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1006ee20 con 0x7f5a10154be0 2024-04-24T08:08:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.149+0000 7f5a0f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5a1015d940 con 0x7f5a10071be0 2024-04-24T08:08:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.149+0000 7f5a0f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5a1006efa0 con 0x7f59f40039e0 2024-04-24T08:08:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.149+0000 7f5a0e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a10154be0 0x7f5a1015ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.149+0000 7f5a0e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a10154be0 0x7f5a1015ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37850/0 (socket says 172.21.15.69:37850) 2024-04-24T08:08:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.149+0000 7f5a0e800640 1 -- 172.21.15.69:0/1369586646 learned_addr learned my addr 172.21.15.69:0/1369586646 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:54.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a07e00640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a10071be0 0x7f5a100c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a0de00640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59f40039e0 0x7f5a101f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a0e800640 1 -- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59f40039e0 msgr2=0x7f5a101f7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a0e800640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59f40039e0 0x7f5a101f7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a0e800640 1 -- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a10071be0 msgr2=0x7f5a100c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a0e800640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a10071be0 0x7f5a100c9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a0e800640 1 -- 172.21.15.69:0/1369586646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a10072150 con 0x7f5a10154be0 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a07e00640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a10071be0 0x7f5a100c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a0e800640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a10154be0 0x7f5a1015ec40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a100b34d0 tx=0x7f5a0800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a06000640 1 -- 172.21.15.69:0/1369586646 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0800bdb0 con 0x7f5a10154be0 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a06000640 1 -- 172.21.15.69:0/1369586646 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a08004b60 con 0x7f5a10154be0 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a06000640 1 -- 172.21.15.69:0/1369586646 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a08007110 con 0x7f5a10154be0 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a0f200640 1 -- 172.21.15.69:0/1369586646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a10072430 con 0x7f5a10154be0 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.150+0000 7f5a0f200640 1 -- 172.21.15.69:0/1369586646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a100729f0 con 0x7f5a10154be0 2024-04-24T08:08:54.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.152+0000 7f5a0f200640 1 -- 172.21.15.69:0/1369586646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a100d0c90 con 0x7f5a10154be0 2024-04-24T08:08:54.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.152+0000 7f5a06000640 1 -- 172.21.15.69:0/1369586646 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5a0801b020 con 0x7f5a10154be0 2024-04-24T08:08:54.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.154+0000 7f5a06000640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f59ec06c920 0x7f59ec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.154+0000 7f5a06000640 1 -- 172.21.15.69:0/1369586646 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5a0808bce0 con 0x7f5a10154be0 2024-04-24T08:08:54.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.155+0000 7f5a0de00640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f59ec06c920 0x7f59ec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.156+0000 7f5a0de00640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f59ec06c920 0x7f59ec06ede0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5a00032000 tx=0x7f5a0002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:54.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.158+0000 7f5a06000640 1 -- 172.21.15.69:0/1369586646 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5a100d0c90 con 0x7f5a10154be0 2024-04-24T08:08:54.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.368+0000 7f5a0f200640 1 -- 172.21.15.69:0/1369586646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5a10075d50 con 0x7f5a10154be0 2024-04-24T08:08:54.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.371+0000 7f5a06000640 1 -- 172.21.15.69:0/1369586646 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7f5a0805a100 con 0x7f5a10154be0 2024-04-24T08:08:54.371 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:54.372 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:54.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 -- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f59ec06c920 msgr2=0x7f59ec06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f59ec06c920 0x7f59ec06ede0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5a00032000 tx=0x7f5a0002e000 comp rx=0 tx=0).stop 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 -- 172.21.15.69:0/1369586646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a10154be0 msgr2=0x7f5a1015ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a10154be0 0x7f5a1015ec40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a100b34d0 tx=0x7f5a0800b4a0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 -- 172.21.15.69:0/1369586646 shutdown_connections 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f59ec06c920 0x7f59ec06ede0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a10154be0 0x7f5a1015ec40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a10071be0 0x7f5a100c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 --2- 172.21.15.69:0/1369586646 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f59f40039e0 0x7f5a101f7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 -- 172.21.15.69:0/1369586646 >> 172.21.15.69:0/1369586646 conn(0x7f5a100d0160 msgr2=0x7f5a1015c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 -- 172.21.15.69:0/1369586646 shutdown_connections 2024-04-24T08:08:54.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.374+0000 7f5a0f200640 1 -- 172.21.15.69:0/1369586646 wait complete. 2024-04-24T08:08:54.393 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:08:54.394 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:08:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.604+0000 7f9adae00640 1 Processor -- start 2024-04-24T08:08:54.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.604+0000 7f9adae00640 1 -- start start 2024-04-24T08:08:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.605+0000 7f9adae00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9ad415b7c0 0x7f9ad40ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.605+0000 7f9adae00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad40cf7f0 con 0x7f9ad40cd0f0 2024-04-24T08:08:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.605+0000 7f9adae00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ad40cc110 con 0x7f9ad40cf010 2024-04-24T08:08:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.605+0000 7f9adae00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9ad40cbee0 con 0x7f9ad415b7c0 2024-04-24T08:08:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.605+0000 7f9ad3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9ad415b7c0 0x7f9ad40ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.605+0000 7f9ad3400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9ad40cd0f0 0x7f9ad40b0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47498/0 (socket says 172.21.15.69:47498) 2024-04-24T08:08:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.606+0000 7f9ad3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9ad415b7c0 0x7f9ad40ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37864/0 (socket says 172.21.15.69:37864) 2024-04-24T08:08:54.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.606+0000 7f9ad3400640 1 -- 172.21.15.69:0/3482029167 learned_addr learned my addr 172.21.15.69:0/3482029167 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:54.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.606+0000 7f9ad3e00640 1 -- 172.21.15.69:0/3482029167 >> v1:172.21.15.121:6789/0 conn(0x7f9ad40cf010 legacy=0x7f9ad415f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:54.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.606+0000 7f9ad3e00640 1 -- 172.21.15.69:0/3482029167 >> v1:172.21.15.69:6789/0 conn(0x7f9ad40cd0f0 legacy=0x7f9ad40b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.606+0000 7f9ad3e00640 1 -- 172.21.15.69:0/3482029167 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad40cf510 con 0x7f9ad415b7c0 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.607+0000 7f9ad3e00640 1 --2- 172.21.15.69:0/3482029167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad415b7c0 0x7f9ad40ccbb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9ac40097a0 tx=0x7f9ac402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74aac3c49b8be437 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.607+0000 7f9ad2a00640 1 -- 172.21.15.69:0/3482029167 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4037070 con 0x7f9ad415b7c0 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.607+0000 7f9ad2a00640 1 -- 172.21.15.69:0/3482029167 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac4032440 con 0x7f9ad415b7c0 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.607+0000 7f9ad2a00640 1 -- 172.21.15.69:0/3482029167 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4032810 con 0x7f9ad415b7c0 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.608+0000 7f9adae00640 1 -- 172.21.15.69:0/3482029167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad415b7c0 msgr2=0x7f9ad40ccbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.608+0000 7f9adae00640 1 --2- 172.21.15.69:0/3482029167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad415b7c0 0x7f9ad40ccbb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9ac40097a0 tx=0x7f9ac402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.608+0000 7f9adae00640 1 -- 172.21.15.69:0/3482029167 shutdown_connections 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.608+0000 7f9adae00640 1 --2- 172.21.15.69:0/3482029167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad415b7c0 0x7f9ad40ccbb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.608+0000 7f9adae00640 1 -- 172.21.15.69:0/3482029167 >> 172.21.15.69:0/3482029167 conn(0x7f9ad40b2f70 msgr2=0x7f9ad40b3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.608+0000 7f9adae00640 1 -- 172.21.15.69:0/3482029167 shutdown_connections 2024-04-24T08:08:54.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.608+0000 7f9adae00640 1 -- 172.21.15.69:0/3482029167 wait complete. 2024-04-24T08:08:54.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.610+0000 7f9adae00640 1 Processor -- start 2024-04-24T08:08:54.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.610+0000 7f9adae00640 1 -- start start 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9adae00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ad40cf010 0x7f9ad4156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9adae00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ad40b6110 0x7f9ad40b6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9adae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad40b1320 0x7f9ad40b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9adae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad415bba0 con 0x7f9ad40b1320 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9adae00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9ad40cc720 con 0x7f9ad40cf010 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9adae00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad40b3cc0 con 0x7f9ad40b6110 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9ad8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad40b1320 0x7f9ad40b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9ad8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad40b1320 0x7f9ad40b17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37876/0 (socket says 172.21.15.69:37876) 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9ad8e00640 1 -- 172.21.15.69:0/2340555805 learned_addr learned my addr 172.21.15.69:0/2340555805 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9ad3e00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ad40cf010 0x7f9ad4156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.611+0000 7f9ad3400640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ad40b6110 0x7f9ad40b6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9ad8e00640 1 -- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ad40b6110 msgr2=0x7f9ad40b6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9ad8e00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ad40b6110 0x7f9ad40b6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9ad8e00640 1 -- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ad40cf010 msgr2=0x7f9ad4156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9ad8e00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ad40cf010 0x7f9ad4156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9ad8e00640 1 -- 172.21.15.69:0/2340555805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad41e74a0 con 0x7f9ad40b1320 2024-04-24T08:08:54.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9ad8e00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad40b1320 0x7f9ad40b17a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00b4c0 tx=0x7f9acc00b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:54.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9acbe00640 1 -- 172.21.15.69:0/2340555805 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9acc015020 con 0x7f9ad40b1320 2024-04-24T08:08:54.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9acbe00640 1 -- 172.21.15.69:0/2340555805 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9acc011850 con 0x7f9ad40b1320 2024-04-24T08:08:54.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9acbe00640 1 -- 172.21.15.69:0/2340555805 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9acc011c40 con 0x7f9ad40b1320 2024-04-24T08:08:54.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9adae00640 1 -- 172.21.15.69:0/2340555805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ad40b43d0 con 0x7f9ad40b1320 2024-04-24T08:08:54.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.612+0000 7f9adae00640 1 -- 172.21.15.69:0/2340555805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ad40b4990 con 0x7f9ad40b1320 2024-04-24T08:08:54.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.614+0000 7f9adae00640 1 -- 172.21.15.69:0/2340555805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad4069190 con 0x7f9ad40b1320 2024-04-24T08:08:54.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.615+0000 7f9acbe00640 1 -- 172.21.15.69:0/2340555805 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9acc018020 con 0x7f9ad40b1320 2024-04-24T08:08:54.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.616+0000 7f9acbe00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ab006c920 0x7f9ab006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:54.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.616+0000 7f9acbe00640 1 -- 172.21.15.69:0/2340555805 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9acc08b990 con 0x7f9ad40b1320 2024-04-24T08:08:54.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.617+0000 7f9ad3e00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ab006c920 0x7f9ab006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:54.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.618+0000 7f9ad3e00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ab006c920 0x7f9ab006ede0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4034000 tx=0x7f9ac402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:54.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.621+0000 7f9acbe00640 1 -- 172.21.15.69:0/2340555805 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9acc01f020 con 0x7f9ad40b1320 2024-04-24T08:08:54.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.820+0000 7f9adae00640 1 -- 172.21.15.69:0/2340555805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9ad40cdb40 con 0x7f9ad40b1320 2024-04-24T08:08:54.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.821+0000 7f9acbe00640 1 -- 172.21.15.69:0/2340555805 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f9acc0172c0 con 0x7f9ad40b1320 2024-04-24T08:08:54.821 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:08:54.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.823+0000 7f9adae00640 1 -- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ab006c920 msgr2=0x7f9ab006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.823+0000 7f9adae00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ab006c920 0x7f9ab006ede0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9ac4034000 tx=0x7f9ac402e000 comp rx=0 tx=0).stop 2024-04-24T08:08:54.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.823+0000 7f9adae00640 1 -- 172.21.15.69:0/2340555805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad40b1320 msgr2=0x7f9ad40b17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:54.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.823+0000 7f9adae00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad40b1320 0x7f9ad40b17a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9acc00b4c0 tx=0x7f9acc00b980 comp rx=0 tx=0).stop 2024-04-24T08:08:54.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.824+0000 7f9adae00640 1 -- 172.21.15.69:0/2340555805 shutdown_connections 2024-04-24T08:08:54.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.824+0000 7f9adae00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ab006c920 0x7f9ab006ede0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.824+0000 7f9adae00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ad40b1320 0x7f9ad40b17a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.824+0000 7f9adae00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ad40b6110 0x7f9ad40b6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.824+0000 7f9adae00640 1 --2- 172.21.15.69:0/2340555805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ad40cf010 0x7f9ad4156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:54.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.824+0000 7f9adae00640 1 -- 172.21.15.69:0/2340555805 >> 172.21.15.69:0/2340555805 conn(0x7f9ad40b2f70 msgr2=0x7f9ad406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:54.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.824+0000 7f9adae00640 1 -- 172.21.15.69:0/2340555805 shutdown_connections 2024-04-24T08:08:54.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:54.824+0000 7f9adae00640 1 -- 172.21.15.69:0/2340555805 wait complete. 2024-04-24T08:08:54.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:08:55.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.042+0000 7efde5000640 1 Processor -- start 2024-04-24T08:08:55.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.042+0000 7efde5000640 1 -- start start 2024-04-24T08:08:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efde5000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efde00b3e90 0x7efde00b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efde5000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efde00ce250 0x7efde00ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efde5000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efde015cc00 con 0x7efde015c820 2024-04-24T08:08:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efde5000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7efde00b4be0 con 0x7efde00ce250 2024-04-24T08:08:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efde5000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efde00b42f0 con 0x7efde00b3e90 2024-04-24T08:08:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efdde800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efde00ce250 0x7efde00ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efdde800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efde00ce250 0x7efde00ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37884/0 (socket says 172.21.15.69:37884) 2024-04-24T08:08:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efdde800640 1 -- 172.21.15.69:0/2972413373 learned_addr learned my addr 172.21.15.69:0/2972413373 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:55.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efddde00640 1 --1- 172.21.15.69:0/2972413373 >> v1:172.21.15.69:6789/0 conn(0x7efde015c820 0x7efde006f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47502/0 (socket says 172.21.15.69:47502) 2024-04-24T08:08:55.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efddd400640 1 --2- 172.21.15.69:0/2972413373 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efde00b3e90 0x7efde00b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efdde800640 1 -- 172.21.15.69:0/2972413373 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efde00b3e90 msgr2=0x7efde00b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:55.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efdde800640 1 --2- 172.21.15.69:0/2972413373 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efde00b3e90 0x7efde00b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.044+0000 7efdde800640 1 -- 172.21.15.69:0/2972413373 >> v1:172.21.15.69:6789/0 conn(0x7efde015c820 legacy=0x7efde006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.045+0000 7efdde800640 1 -- 172.21.15.69:0/2972413373 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde006ed30 con 0x7efde00ce250 2024-04-24T08:08:55.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.045+0000 7efddd400640 1 --2- 172.21.15.69:0/2972413373 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efde00b3e90 0x7efde00b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.045+0000 7efddca00640 1 -- 172.21.15.69:0/2972413373 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3868419238 0 0) 0x7efde015cc00 con 0x7efde015c820 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.046+0000 7efdde800640 1 --2- 172.21.15.69:0/2972413373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 0x7efde00ce6e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009e50 tx=0x7efdd802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50820d75d27c9f4a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.046+0000 7efddca00640 1 -- 172.21.15.69:0/2972413373 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd8036070 con 0x7efde00ce250 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.046+0000 7efddca00640 1 -- 172.21.15.69:0/2972413373 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd8031600 con 0x7efde00ce250 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.046+0000 7efddca00640 1 -- 172.21.15.69:0/2972413373 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd80319d0 con 0x7efde00ce250 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.046+0000 7efde5000640 1 -- 172.21.15.69:0/2972413373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 msgr2=0x7efde00ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.046+0000 7efde5000640 1 --2- 172.21.15.69:0/2972413373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 0x7efde00ce6e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7efdd8009e50 tx=0x7efdd802a710 comp rx=0 tx=0).stop 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.046+0000 7efde5000640 1 -- 172.21.15.69:0/2972413373 shutdown_connections 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.046+0000 7efde5000640 1 --2- 172.21.15.69:0/2972413373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 0x7efde00ce6e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.046+0000 7efde5000640 1 --2- 172.21.15.69:0/2972413373 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efde00b3e90 0x7efde00b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.047+0000 7efde5000640 1 -- 172.21.15.69:0/2972413373 >> 172.21.15.69:0/2972413373 conn(0x7efde00d0350 msgr2=0x7efde00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.047+0000 7efde5000640 1 -- 172.21.15.69:0/2972413373 shutdown_connections 2024-04-24T08:08:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.047+0000 7efde5000640 1 -- 172.21.15.69:0/2972413373 wait complete. 2024-04-24T08:08:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.048+0000 7efde5000640 1 Processor -- start 2024-04-24T08:08:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.048+0000 7efde5000640 1 -- start start 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.049+0000 7efde5000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efde00b3e90 0x7efde01f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.049+0000 7efde5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 0x7efde015f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.049+0000 7efde5000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efde015c820 0x7efde0071d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.049+0000 7efde5000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efde006efd0 con 0x7efde00ce250 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.049+0000 7efde5000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efde00b50f0 con 0x7efde015c820 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.049+0000 7efde5000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efde006f2b0 con 0x7efde00b3e90 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efddd400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 0x7efde015f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efddd400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 0x7efde015f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37886/0 (socket says 172.21.15.69:37886) 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efddd400640 1 -- 172.21.15.69:0/2342543755 learned_addr learned my addr 172.21.15.69:0/2342543755 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efddd400640 1 -- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efde00b3e90 msgr2=0x7efde01f4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efddd400640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efde00b3e90 0x7efde01f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efddd400640 1 -- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efde015c820 msgr2=0x7efde0071d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efddd400640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efde015c820 0x7efde0071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efddd400640 1 -- 172.21.15.69:0/2342543755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde00735d0 con 0x7efde00ce250 2024-04-24T08:08:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efddd400640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 0x7efde015f730 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7efdc800c7f0 tx=0x7efdc800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efdd6000640 1 -- 172.21.15.69:0/2342543755 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc8015070 con 0x7efde00ce250 2024-04-24T08:08:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efdd6000640 1 -- 172.21.15.69:0/2342543755 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc8007a30 con 0x7efde00ce250 2024-04-24T08:08:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efdd6000640 1 -- 172.21.15.69:0/2342543755 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc8007e00 con 0x7efde00ce250 2024-04-24T08:08:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.050+0000 7efde5000640 1 -- 172.21.15.69:0/2342543755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde015fc70 con 0x7efde00ce250 2024-04-24T08:08:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.051+0000 7efde5000640 1 -- 172.21.15.69:0/2342543755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde0072810 con 0x7efde00ce250 2024-04-24T08:08:55.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.052+0000 7efde5000640 1 -- 172.21.15.69:0/2342543755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde0068a80 con 0x7efde00ce250 2024-04-24T08:08:55.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.053+0000 7efdd6000640 1 -- 172.21.15.69:0/2342543755 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efdc8018070 con 0x7efde00ce250 2024-04-24T08:08:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.054+0000 7efdd6000640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efdbc06c8f0 0x7efdbc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.054+0000 7efdd6000640 1 -- 172.21.15.69:0/2342543755 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efdc808bad0 con 0x7efde00ce250 2024-04-24T08:08:55.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.054+0000 7efddde00640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efdbc06c8f0 0x7efdbc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.055+0000 7efddde00640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efdbc06c8f0 0x7efdbc06edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efdd0007fd0 tx=0x7efdd0007960 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:55.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.058+0000 7efdd6000640 1 -- 172.21.15.69:0/2342543755 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efdc8059f90 con 0x7efde00ce250 2024-04-24T08:08:55.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.257+0000 7efde5000640 1 -- 172.21.15.69:0/2342543755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efde0072240 con 0x7efde00ce250 2024-04-24T08:08:55.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.258+0000 7efdd6000640 1 -- 172.21.15.69:0/2342543755 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7efdc8059890 con 0x7efde00ce250 2024-04-24T08:08:55.259 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:08:55.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 -- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efdbc06c8f0 msgr2=0x7efdbc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efdbc06c8f0 0x7efdbc06edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efdd0007fd0 tx=0x7efdd0007960 comp rx=0 tx=0).stop 2024-04-24T08:08:55.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 -- 172.21.15.69:0/2342543755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 msgr2=0x7efde015f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 0x7efde015f730 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7efdc800c7f0 tx=0x7efdc800cbf0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 -- 172.21.15.69:0/2342543755 shutdown_connections 2024-04-24T08:08:55.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efdbc06c8f0 0x7efdbc06edb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efde00ce250 0x7efde015f730 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efde015c820 0x7efde0071d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 --2- 172.21.15.69:0/2342543755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efde00b3e90 0x7efde01f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 -- 172.21.15.69:0/2342543755 >> 172.21.15.69:0/2342543755 conn(0x7efde00d0350 msgr2=0x7efde00cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:55.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 -- 172.21.15.69:0/2342543755 shutdown_connections 2024-04-24T08:08:55.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.261+0000 7efde5000640 1 -- 172.21.15.69:0/2342543755 wait complete. 2024-04-24T08:08:55.279 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:08:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.472+0000 7fc0f4e00640 1 Processor -- start 2024-04-24T08:08:55.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.472+0000 7fc0f4e00640 1 -- start start 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.473+0000 7fc0f4e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f0162f90 0x7fc0f00cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.473+0000 7fc0f4e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc0f00d0190 0x7fc0f006cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0f4e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0f015ed60 con 0x7fc0f015e980 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0f4e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc0f01633d0 con 0x7fc0f00d0190 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0f4e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f00d1580 con 0x7fc0f0162f90 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0ef400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc0f015e980 0x7fc0f00d0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47510/0 (socket says 172.21.15.69:47510) 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0ef400640 1 -- 172.21.15.69:0/2392476453 learned_addr learned my addr 172.21.15.69:0/2392476453 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0efe00640 1 --2- 172.21.15.69:0/2392476453 >> v2:172.21.15.69:3300/0 conn(0x7fc0f00d0190 0x7fc0f006cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0eea00640 1 -- 172.21.15.69:0/2392476453 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3864818282 0 0) 0x7fc0f015ed60 con 0x7fc0f015e980 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0eea00640 1 -- 172.21.15.69:0/2392476453 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0d0003520 con 0x7fc0f015e980 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0e7e00640 1 --2- 172.21.15.69:0/2392476453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f0162f90 0x7fc0f00cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0efe00640 1 -- 172.21.15.69:0/2392476453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f0162f90 msgr2=0x7fc0f00cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0efe00640 1 --2- 172.21.15.69:0/2392476453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f0162f90 0x7fc0f00cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0efe00640 1 -- 172.21.15.69:0/2392476453 >> v1:172.21.15.69:6789/0 conn(0x7fc0f015e980 legacy=0x7fc0f00d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.474+0000 7fc0efe00640 1 -- 172.21.15.69:0/2392476453 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f00d1860 con 0x7fc0f00d0190 2024-04-24T08:08:55.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.475+0000 7fc0efe00640 1 --2- 172.21.15.69:0/2392476453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f00d0190 0x7fc0f006cfb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009dc0 tx=0x7fc0e802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1738124ae7920426 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0eea00640 1 -- 172.21.15.69:0/2392476453 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8007740 con 0x7fc0f00d0190 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0eea00640 1 -- 172.21.15.69:0/2392476453 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0e80078e0 con 0x7fc0f00d0190 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0eea00640 1 -- 172.21.15.69:0/2392476453 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8007cb0 con 0x7fc0f00d0190 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0f4e00640 1 -- 172.21.15.69:0/2392476453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f00d0190 msgr2=0x7fc0f006cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0f4e00640 1 --2- 172.21.15.69:0/2392476453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f00d0190 0x7fc0f006cfb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009dc0 tx=0x7fc0e802a710 comp rx=0 tx=0).stop 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0f4e00640 1 -- 172.21.15.69:0/2392476453 shutdown_connections 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0f4e00640 1 --2- 172.21.15.69:0/2392476453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f00d0190 0x7fc0f006cfb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0f4e00640 1 --2- 172.21.15.69:0/2392476453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f0162f90 0x7fc0f00cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0f4e00640 1 -- 172.21.15.69:0/2392476453 >> 172.21.15.69:0/2392476453 conn(0x7fc0f009d920 msgr2=0x7fc0f008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0f4e00640 1 -- 172.21.15.69:0/2392476453 shutdown_connections 2024-04-24T08:08:55.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.476+0000 7fc0f4e00640 1 -- 172.21.15.69:0/2392476453 wait complete. 2024-04-24T08:08:55.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.478+0000 7fc0f4e00640 1 Processor -- start 2024-04-24T08:08:55.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.478+0000 7fc0f4e00640 1 -- start start 2024-04-24T08:08:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.479+0000 7fc0f4e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f00d0190 0x7fc0f015ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.479+0000 7fc0f4e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f015e980 0x7fc0f00b9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.479+0000 7fc0f4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f0162f90 0x7fc0f015ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.479+0000 7fc0f4e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f00cf070 con 0x7fc0f0162f90 2024-04-24T08:08:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.479+0000 7fc0f4e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f00cf730 con 0x7fc0f00d0190 2024-04-24T08:08:55.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.479+0000 7fc0f4e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc0f00cf1f0 con 0x7fc0f015e980 2024-04-24T08:08:55.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.479+0000 7fc0efe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f0162f90 0x7fc0f015ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.479+0000 7fc0efe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f0162f90 0x7fc0f015ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37906/0 (socket says 172.21.15.69:37906) 2024-04-24T08:08:55.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.479+0000 7fc0efe00640 1 -- 172.21.15.69:0/3295572999 learned_addr learned my addr 172.21.15.69:0/3295572999 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:55.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0efe00640 1 -- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f015e980 msgr2=0x7fc0f00b9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0efe00640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f015e980 0x7fc0f00b9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0efe00640 1 -- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f00d0190 msgr2=0x7fc0f015ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0efe00640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f00d0190 0x7fc0f015ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0efe00640 1 -- 172.21.15.69:0/3295572999 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f00b8f40 con 0x7fc0f0162f90 2024-04-24T08:08:55.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0efe00640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f0162f90 0x7fc0f015ffb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc0e802ad00 tx=0x7fc0e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:55.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0e7400640 1 -- 172.21.15.69:0/3295572999 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8040070 con 0x7fc0f0162f90 2024-04-24T08:08:55.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0e7400640 1 -- 172.21.15.69:0/3295572999 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0e8031bc0 con 0x7fc0f0162f90 2024-04-24T08:08:55.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0e7400640 1 -- 172.21.15.69:0/3295572999 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8009450 con 0x7fc0f0162f90 2024-04-24T08:08:55.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0f4e00640 1 -- 172.21.15.69:0/3295572999 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f00b7370 con 0x7fc0f0162f90 2024-04-24T08:08:55.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.480+0000 7fc0f4e00640 1 -- 172.21.15.69:0/3295572999 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f00b7930 con 0x7fc0f0162f90 2024-04-24T08:08:55.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.481+0000 7fc0f4e00640 1 -- 172.21.15.69:0/3295572999 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0f0075d50 con 0x7fc0f0162f90 2024-04-24T08:08:55.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.482+0000 7fc0e7400640 1 -- 172.21.15.69:0/3295572999 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc0e8039040 con 0x7fc0f0162f90 2024-04-24T08:08:55.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.483+0000 7fc0e7400640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0cc06c8f0 0x7fc0cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.483+0000 7fc0e7400640 1 -- 172.21.15.69:0/3295572999 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc0e80ad310 con 0x7fc0f0162f90 2024-04-24T08:08:55.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.484+0000 7fc0ef400640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0cc06c8f0 0x7fc0cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.485+0000 7fc0ef400640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0cc06c8f0 0x7fc0cc06edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0000f60 tx=0x7fc0e00078f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:55.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.488+0000 7fc0e7400640 1 -- 172.21.15.69:0/3295572999 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc0e807b7d0 con 0x7fc0f0162f90 2024-04-24T08:08:55.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.688+0000 7fc0f4e00640 1 -- 172.21.15.69:0/3295572999 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc0f00d1e40 con 0x7fc0f0162f90 2024-04-24T08:08:55.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.688+0000 7fc0e7400640 1 -- 172.21.15.69:0/3295572999 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fc0e807b0d0 con 0x7fc0f0162f90 2024-04-24T08:08:55.689 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:08:55.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.690+0000 7fc0f4e00640 1 -- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0cc06c8f0 msgr2=0x7fc0cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.690+0000 7fc0f4e00640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0cc06c8f0 0x7fc0cc06edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0000f60 tx=0x7fc0e00078f0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.690+0000 7fc0f4e00640 1 -- 172.21.15.69:0/3295572999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f0162f90 msgr2=0x7fc0f015ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.690+0000 7fc0f4e00640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f0162f90 0x7fc0f015ffb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc0e802ad00 tx=0x7fc0e8004a40 comp rx=0 tx=0).stop 2024-04-24T08:08:55.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.691+0000 7fc0f4e00640 1 -- 172.21.15.69:0/3295572999 shutdown_connections 2024-04-24T08:08:55.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.691+0000 7fc0f4e00640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0cc06c8f0 0x7fc0cc06edb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.691+0000 7fc0f4e00640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f0162f90 0x7fc0f015ffb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.691+0000 7fc0f4e00640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0f015e980 0x7fc0f00b9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.691+0000 7fc0f4e00640 1 --2- 172.21.15.69:0/3295572999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0f00d0190 0x7fc0f015ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.691+0000 7fc0f4e00640 1 -- 172.21.15.69:0/3295572999 >> 172.21.15.69:0/3295572999 conn(0x7fc0f009d920 msgr2=0x7fc0f00b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:55.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.691+0000 7fc0f4e00640 1 -- 172.21.15.69:0/3295572999 shutdown_connections 2024-04-24T08:08:55.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.691+0000 7fc0f4e00640 1 -- 172.21.15.69:0/3295572999 wait complete. 2024-04-24T08:08:55.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:55.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.906+0000 7efc79a00640 1 Processor -- start 2024-04-24T08:08:55.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.906+0000 7efc79a00640 1 -- start start 2024-04-24T08:08:55.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.907+0000 7efc79a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74156c60 0x7efc7415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.907+0000 7efc79a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efc740b30f0 0x7efc7415d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.907+0000 7efc79a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc7415d660 con 0x7efc740ad0d0 2024-04-24T08:08:55.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.907+0000 7efc79a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7efc7406dc30 con 0x7efc740b30f0 2024-04-24T08:08:55.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.907+0000 7efc79a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efc7406df10 con 0x7efc74156c60 2024-04-24T08:08:55.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc79000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7efc740ad0d0 0x7efc741600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47516/0 (socket says 172.21.15.69:47516) 2024-04-24T08:08:55.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc79000640 1 -- 172.21.15.69:0/3190982596 learned_addr learned my addr 172.21.15.69:0/3190982596 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:55.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc73e00640 1 --2- 172.21.15.69:0/3190982596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74156c60 0x7efc7415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc73400640 1 --2- 172.21.15.69:0/3190982596 >> v2:172.21.15.121:3300/0 conn(0x7efc740b30f0 0x7efc7415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3345831323 0 0) 0x7efc7415d660 con 0x7efc740ad0d0 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc58003520 con 0x7efc740ad0d0 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1983633988 0 0) 0x7efc58003520 con 0x7efc740ad0d0 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74156c60 msgr2=0x7efc7415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc72a00640 1 --2- 172.21.15.69:0/3190982596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74156c60 0x7efc7415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 >> v2:172.21.15.121:3300/0 conn(0x7efc740b30f0 msgr2=0x7efc7415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc72a00640 1 --2- 172.21.15.69:0/3190982596 >> v2:172.21.15.121:3300/0 conn(0x7efc740b30f0 0x7efc7415d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7406e1f0 con 0x7efc740ad0d0 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7efc6c0019f0 con 0x7efc740ad0d0 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.908+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 >> v1:172.21.15.69:6789/0 conn(0x7efc740ad0d0 legacy=0x7efc741600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc72a00640 1 --2- 172.21.15.69:0/3190982596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc580039b0 0x7efc58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc72a00640 1 --2- 172.21.15.69:0/3190982596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74156c60 0x7efc58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc72a00640 1 --2- 172.21.15.69:0/3190982596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc58004d40 0x7efc580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc7406dc30 con 0x7efc58004d40 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efc7406df10 con 0x7efc580039b0 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efc5800ba70 con 0x7efc74156c60 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc73400640 1 --2- 172.21.15.69:0/3190982596 >> v2:172.21.15.121:3300/0 conn(0x7efc740b30f0 0x7efc7415d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc79000640 1 --2- 172.21.15.69:0/3190982596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc58004d40 0x7efc580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc79000640 1 -- 172.21.15.69:0/3190982596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74156c60 msgr2=0x7efc58004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc79000640 1 --2- 172.21.15.69:0/3190982596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74156c60 0x7efc58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc79000640 1 -- 172.21.15.69:0/3190982596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc580039b0 msgr2=0x7efc58003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc79000640 1 --2- 172.21.15.69:0/3190982596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc580039b0 0x7efc58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.909+0000 7efc79000640 1 -- 172.21.15.69:0/3190982596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc58003520 con 0x7efc58004d40 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.910+0000 7efc79000640 1 --2- 172.21.15.69:0/3190982596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc58004d40 0x7efc580051c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00a8d0 tx=0x7efc58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ac42157f9e6e59 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.910+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c00a4c0 con 0x7efc58004d40 2024-04-24T08:08:55.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.911+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc6c02f070 con 0x7efc58004d40 2024-04-24T08:08:55.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.911+0000 7efc72a00640 1 -- 172.21.15.69:0/3190982596 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c02bd00 con 0x7efc58004d40 2024-04-24T08:08:55.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.911+0000 7efc79a00640 1 -- 172.21.15.69:0/3190982596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc58004d40 msgr2=0x7efc580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.911+0000 7efc79a00640 1 --2- 172.21.15.69:0/3190982596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc58004d40 0x7efc580051c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00a8d0 tx=0x7efc58005830 comp rx=0 tx=0).stop 2024-04-24T08:08:55.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.911+0000 7efc79000640 1 -- 172.21.15.69:0/3190982596 reap_dead start 2024-04-24T08:08:55.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.911+0000 7efc79a00640 1 -- 172.21.15.69:0/3190982596 shutdown_connections 2024-04-24T08:08:55.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.911+0000 7efc79a00640 1 -- 172.21.15.69:0/3190982596 >> 172.21.15.69:0/3190982596 conn(0x7efc740d0160 msgr2=0x7efc740d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:55.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.911+0000 7efc79a00640 1 -- 172.21.15.69:0/3190982596 shutdown_connections 2024-04-24T08:08:55.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.911+0000 7efc79a00640 1 -- 172.21.15.69:0/3190982596 wait complete. 2024-04-24T08:08:55.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.913+0000 7efc79a00640 1 Processor -- start 2024-04-24T08:08:55.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.913+0000 7efc79a00640 1 -- start start 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.914+0000 7efc79a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc58004d40 0x7efc740cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.914+0000 7efc79a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc7415aae0 0x7efc741449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.914+0000 7efc79a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74144ee0 0x7efc74145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.914+0000 7efc79a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc7415cac0 con 0x7efc7415aae0 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.914+0000 7efc79a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efc7415d940 con 0x7efc58004d40 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.914+0000 7efc79a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efc7415cc40 con 0x7efc74144ee0 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.914+0000 7efc73400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc7415aae0 0x7efc741449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.914+0000 7efc73400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc7415aae0 0x7efc741449a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37924/0 (socket says 172.21.15.69:37924) 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.914+0000 7efc73400640 1 -- 172.21.15.69:0/2797113742 learned_addr learned my addr 172.21.15.69:0/2797113742 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.914+0000 7efc73e00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc58004d40 0x7efc740cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc73400640 1 -- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74144ee0 msgr2=0x7efc74145390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc73400640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74144ee0 0x7efc74145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc73400640 1 -- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc58004d40 msgr2=0x7efc740cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc73400640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc58004d40 0x7efc740cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc73400640 1 -- 172.21.15.69:0/2797113742 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc740ca580 con 0x7efc7415aae0 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc73e00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc58004d40 0x7efc740cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc73400640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc7415aae0 0x7efc741449a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efc6000e810 tx=0x7efc6000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc6be00640 1 -- 172.21.15.69:0/2797113742 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6000f040 con 0x7efc7415aae0 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc6be00640 1 -- 172.21.15.69:0/2797113742 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc60004030 con 0x7efc7415aae0 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc6be00640 1 -- 172.21.15.69:0/2797113742 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc60004400 con 0x7efc7415aae0 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc79a00640 1 -- 172.21.15.69:0/2797113742 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc74073b50 con 0x7efc7415aae0 2024-04-24T08:08:55.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.915+0000 7efc79a00640 1 -- 172.21.15.69:0/2797113742 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc740cfa50 con 0x7efc7415aae0 2024-04-24T08:08:55.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.917+0000 7efc79a00640 1 -- 172.21.15.69:0/2797113742 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc74075d50 con 0x7efc7415aae0 2024-04-24T08:08:55.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.917+0000 7efc6be00640 1 -- 172.21.15.69:0/2797113742 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efc6000c650 con 0x7efc7415aae0 2024-04-24T08:08:55.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.918+0000 7efc6be00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efc5406c8f0 0x7efc5406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:55.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.918+0000 7efc6be00640 1 -- 172.21.15.69:0/2797113742 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efc6008b800 con 0x7efc7415aae0 2024-04-24T08:08:55.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.919+0000 7efc73e00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efc5406c8f0 0x7efc5406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:55.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.920+0000 7efc73e00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efc5406c8f0 0x7efc5406edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efc64006fd0 tx=0x7efc64008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:55.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:55.923+0000 7efc6be00640 1 -- 172.21.15.69:0/2797113742 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc60017020 con 0x7efc7415aae0 2024-04-24T08:08:56.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.121+0000 7efc79a00640 1 -- 172.21.15.69:0/2797113742 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc7406f9e0 con 0x7efc7415aae0 2024-04-24T08:08:56.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.124+0000 7efc6be00640 1 -- 172.21.15.69:0/2797113742 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7efc60059c30 con 0x7efc7415aae0 2024-04-24T08:08:56.125 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:56.125 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:56.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.126+0000 7efc79a00640 1 -- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efc5406c8f0 msgr2=0x7efc5406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.126+0000 7efc79a00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efc5406c8f0 0x7efc5406edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7efc64006fd0 tx=0x7efc64008040 comp rx=0 tx=0).stop 2024-04-24T08:08:56.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.127+0000 7efc79a00640 1 -- 172.21.15.69:0/2797113742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc7415aae0 msgr2=0x7efc741449a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.127+0000 7efc79a00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc7415aae0 0x7efc741449a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7efc6000e810 tx=0x7efc6000ecd0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.127+0000 7efc79a00640 1 -- 172.21.15.69:0/2797113742 shutdown_connections 2024-04-24T08:08:56.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.127+0000 7efc79a00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efc5406c8f0 0x7efc5406edb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.127+0000 7efc79a00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc7415aae0 0x7efc741449a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.127+0000 7efc79a00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc74144ee0 0x7efc74145390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.127+0000 7efc79a00640 1 --2- 172.21.15.69:0/2797113742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc58004d40 0x7efc740cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.127+0000 7efc79a00640 1 -- 172.21.15.69:0/2797113742 >> 172.21.15.69:0/2797113742 conn(0x7efc740d0160 msgr2=0x7efc740afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:56.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.127+0000 7efc79a00640 1 -- 172.21.15.69:0/2797113742 shutdown_connections 2024-04-24T08:08:56.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.127+0000 7efc79a00640 1 -- 172.21.15.69:0/2797113742 wait complete. 2024-04-24T08:08:56.146 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:08:56.147 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:08:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.345+0000 7f37ae200640 1 Processor -- start 2024-04-24T08:08:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.346+0000 7f37ae200640 1 -- start start 2024-04-24T08:08:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.347+0000 7f37ae200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f37a80b1bd0 0x7f37a80cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.347+0000 7f37ae200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37a80b05e0 0x7f37a80b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.347+0000 7f37ae200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37a80d0770 con 0x7f37a80b1070 2024-04-24T08:08:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.347+0000 7f37ae200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f37a80d12b0 con 0x7f37a80b1bd0 2024-04-24T08:08:56.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.347+0000 7f37ae200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f37a80d0a10 con 0x7f37a80b05e0 2024-04-24T08:08:56.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.348+0000 7f37a7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f37a80b1bd0 0x7f37a80cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:56.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.348+0000 7f37a7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f37a80b1bd0 0x7f37a80cd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37928/0 (socket says 172.21.15.69:37928) 2024-04-24T08:08:56.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.348+0000 7f37a7400640 1 -- 172.21.15.69:0/1458557547 learned_addr learned my addr 172.21.15.69:0/1458557547 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:56.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.348+0000 7f37a7e00640 1 --1- 172.21.15.69:0/1458557547 >> v1:172.21.15.69:6789/0 conn(0x7f37a80b1070 0x7f37a80ce350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47522/0 (socket says 172.21.15.69:47522) 2024-04-24T08:08:56.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.348+0000 7f37a7400640 1 -- 172.21.15.69:0/1458557547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37a80b05e0 msgr2=0x7f37a80b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:56.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.348+0000 7f37a7400640 1 --2- 172.21.15.69:0/1458557547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37a80b05e0 0x7f37a80b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.348+0000 7f37a7400640 1 -- 172.21.15.69:0/1458557547 >> v1:172.21.15.69:6789/0 conn(0x7f37a80b1070 legacy=0x7f37a80ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.348+0000 7f37a7400640 1 -- 172.21.15.69:0/1458557547 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a80d0cf0 con 0x7f37a80b1bd0 2024-04-24T08:08:56.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.349+0000 7f37a7400640 1 --2- 172.21.15.69:0/1458557547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b1bd0 0x7f37a80cd5f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3798009870 tx=0x7f379802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8060ae2d1515af3c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:56.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.349+0000 7f37a6000640 1 -- 172.21.15.69:0/1458557547 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798037070 con 0x7f37a80b1bd0 2024-04-24T08:08:56.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.349+0000 7f37a6000640 1 -- 172.21.15.69:0/1458557547 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3798032440 con 0x7f37a80b1bd0 2024-04-24T08:08:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.349+0000 7f37a6000640 1 -- 172.21.15.69:0/1458557547 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798032810 con 0x7f37a80b1bd0 2024-04-24T08:08:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.350+0000 7f37ae200640 1 -- 172.21.15.69:0/1458557547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b1bd0 msgr2=0x7f37a80cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.350+0000 7f37ae200640 1 --2- 172.21.15.69:0/1458557547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b1bd0 0x7f37a80cd5f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3798009870 tx=0x7f379802a1c0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.350+0000 7f37ae200640 1 -- 172.21.15.69:0/1458557547 shutdown_connections 2024-04-24T08:08:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.350+0000 7f37ae200640 1 --2- 172.21.15.69:0/1458557547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37a80b05e0 0x7f37a80b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.350+0000 7f37ae200640 1 --2- 172.21.15.69:0/1458557547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b1bd0 0x7f37a80cd5f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.350+0000 7f37ae200640 1 -- 172.21.15.69:0/1458557547 >> 172.21.15.69:0/1458557547 conn(0x7f37a80d1430 msgr2=0x7f37a80d1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.350+0000 7f37ae200640 1 -- 172.21.15.69:0/1458557547 shutdown_connections 2024-04-24T08:08:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.350+0000 7f37ae200640 1 -- 172.21.15.69:0/1458557547 wait complete. 2024-04-24T08:08:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.352+0000 7f37ae200640 1 Processor -- start 2024-04-24T08:08:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.352+0000 7f37ae200640 1 -- start start 2024-04-24T08:08:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37ae200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b05e0 0x7f37a81cff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37ae200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37a80b1070 0x7f37a81df010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37ae200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37a80b1bd0 0x7f37a81e0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37ae200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f37a80cea60 con 0x7f37a80b05e0 2024-04-24T08:08:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37ae200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f37a80b1fb0 con 0x7f37a80b1070 2024-04-24T08:08:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37ae200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f37a80cebe0 con 0x7f37a80b1bd0 2024-04-24T08:08:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37a7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b05e0 0x7f37a81cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37a7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b05e0 0x7f37a81cff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37930/0 (socket says 172.21.15.69:37930) 2024-04-24T08:08:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37a7400640 1 -- 172.21.15.69:0/1995600238 learned_addr learned my addr 172.21.15.69:0/1995600238 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37a7400640 1 -- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37a80b1bd0 msgr2=0x7f37a81e0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37a7400640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37a80b1bd0 0x7f37a81e0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37a7400640 1 -- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37a80b1070 msgr2=0x7f37a81df010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37a7400640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37a80b1070 0x7f37a81df010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.353+0000 7f37a7400640 1 -- 172.21.15.69:0/1995600238 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a81e2c20 con 0x7f37a80b05e0 2024-04-24T08:08:56.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.354+0000 7f37a7400640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b05e0 0x7f37a81cff00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3798009870 tx=0x7f37980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:56.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.354+0000 7f379f400640 1 -- 172.21.15.69:0/1995600238 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798041070 con 0x7f37a80b05e0 2024-04-24T08:08:56.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.354+0000 7f379f400640 1 -- 172.21.15.69:0/1995600238 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f379803c870 con 0x7f37a80b05e0 2024-04-24T08:08:56.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.354+0000 7f37ae200640 1 -- 172.21.15.69:0/1995600238 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a81e2ec0 con 0x7f37a80b05e0 2024-04-24T08:08:56.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.354+0000 7f379f400640 1 -- 172.21.15.69:0/1995600238 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798037040 con 0x7f37a80b05e0 2024-04-24T08:08:56.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.354+0000 7f37ae200640 1 -- 172.21.15.69:0/1995600238 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a81df550 con 0x7f37a80b05e0 2024-04-24T08:08:56.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.355+0000 7f37ae200640 1 -- 172.21.15.69:0/1995600238 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37a8069190 con 0x7f37a80b05e0 2024-04-24T08:08:56.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.356+0000 7f379f400640 1 -- 172.21.15.69:0/1995600238 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3798044070 con 0x7f37a80b05e0 2024-04-24T08:08:56.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.357+0000 7f379f400640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f378406c8f0 0x7f378406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.357+0000 7f379f400640 1 -- 172.21.15.69:0/1995600238 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f37980ac650 con 0x7f37a80b05e0 2024-04-24T08:08:56.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.358+0000 7f37a6a00640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f378406c8f0 0x7f378406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:56.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.359+0000 7f37a6a00640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f378406c8f0 0x7f378406edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f37a81e2540 tx=0x7f3790006bf0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:56.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.361+0000 7f379f400640 1 -- 172.21.15.69:0/1995600238 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f379807aa70 con 0x7f37a80b05e0 2024-04-24T08:08:56.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.543+0000 7f37ae200640 1 -- 172.21.15.69:0/1995600238 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f37a81e3260 con 0x7f37a80b05e0 2024-04-24T08:08:56.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.544+0000 7f379f400640 1 -- 172.21.15.69:0/1995600238 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f379807a370 con 0x7f37a80b05e0 2024-04-24T08:08:56.544 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 -- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f378406c8f0 msgr2=0x7f378406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f378406c8f0 0x7f378406edb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f37a81e2540 tx=0x7f3790006bf0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 -- 172.21.15.69:0/1995600238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b05e0 msgr2=0x7f37a81cff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b05e0 0x7f37a81cff00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3798009870 tx=0x7f37980046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 -- 172.21.15.69:0/1995600238 shutdown_connections 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f378406c8f0 0x7f378406edb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37a80b1bd0 0x7f37a81e0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37a80b1070 0x7f37a81df010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 --2- 172.21.15.69:0/1995600238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37a80b05e0 0x7f37a81cff00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 -- 172.21.15.69:0/1995600238 >> 172.21.15.69:0/1995600238 conn(0x7f37a80d1430 msgr2=0x7f37a806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.546+0000 7f37ae200640 1 -- 172.21.15.69:0/1995600238 shutdown_connections 2024-04-24T08:08:56.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.547+0000 7f37ae200640 1 -- 172.21.15.69:0/1995600238 wait complete. 2024-04-24T08:08:56.565 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:08:56.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.767+0000 7f4d4fe00640 1 Processor -- start 2024-04-24T08:08:56.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.767+0000 7f4d4fe00640 1 -- start start 2024-04-24T08:08:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.768+0000 7f4d4fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d50071d70 0x7f4d50072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.769+0000 7f4d4fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4d5006e000 0x7f4d5006e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.769+0000 7f4d4fe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d50072690 con 0x7f4d5015edf0 2024-04-24T08:08:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.769+0000 7f4d4fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4d500d0a00 con 0x7f4d5006e000 2024-04-24T08:08:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.769+0000 7f4d4fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4d5015f960 con 0x7f4d50071d70 2024-04-24T08:08:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.769+0000 7f4d4f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4d5006e000 0x7f4d5006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.769+0000 7f4d4f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4d5006e000 0x7f4d5006e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37944/0 (socket says 172.21.15.69:37944) 2024-04-24T08:08:56.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.769+0000 7f4d4f400640 1 -- 172.21.15.69:0/2809326122 learned_addr learned my addr 172.21.15.69:0/2809326122 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:56.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.769+0000 7f4d4ea00640 1 --2- 172.21.15.69:0/2809326122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d50071d70 0x7f4d50072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:56.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.769+0000 7f4d4f400640 1 -- 172.21.15.69:0/2809326122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d50071d70 msgr2=0x7f4d50072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:56.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.769+0000 7f4d4f400640 1 --2- 172.21.15.69:0/2809326122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d50071d70 0x7f4d50072150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.770+0000 7f4d4f400640 1 -- 172.21.15.69:0/2809326122 >> v1:172.21.15.121:6789/0 conn(0x7f4d5015edf0 legacy=0x7f4d5015f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.770+0000 7f4d4f400640 1 -- 172.21.15.69:0/2809326122 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5015fbb0 con 0x7f4d5006e000 2024-04-24T08:08:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.771+0000 7f4d4f400640 1 --2- 172.21.15.69:0/2809326122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5006e000 0x7f4d5006e490 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009e50 tx=0x7f4d4802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb9e025296f1212 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.771+0000 7f4d4d600640 1 -- 172.21.15.69:0/2809326122 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48036070 con 0x7f4d5006e000 2024-04-24T08:08:56.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.771+0000 7f4d4d600640 1 -- 172.21.15.69:0/2809326122 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d48031600 con 0x7f4d5006e000 2024-04-24T08:08:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.771+0000 7f4d4d600640 1 -- 172.21.15.69:0/2809326122 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d480319d0 con 0x7f4d5006e000 2024-04-24T08:08:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.771+0000 7f4d4fe00640 1 -- 172.21.15.69:0/2809326122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5006e000 msgr2=0x7f4d5006e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.771+0000 7f4d4fe00640 1 --2- 172.21.15.69:0/2809326122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5006e000 0x7f4d5006e490 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009e50 tx=0x7f4d4802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.772+0000 7f4d4fe00640 1 -- 172.21.15.69:0/2809326122 shutdown_connections 2024-04-24T08:08:56.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.772+0000 7f4d4fe00640 1 --2- 172.21.15.69:0/2809326122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5006e000 0x7f4d5006e490 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.772+0000 7f4d4fe00640 1 --2- 172.21.15.69:0/2809326122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d50071d70 0x7f4d50072150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.772+0000 7f4d4fe00640 1 -- 172.21.15.69:0/2809326122 >> 172.21.15.69:0/2809326122 conn(0x7f4d50067b80 msgr2=0x7f4d5006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:56.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.772+0000 7f4d4fe00640 1 -- 172.21.15.69:0/2809326122 shutdown_connections 2024-04-24T08:08:56.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.772+0000 7f4d4fe00640 1 -- 172.21.15.69:0/2809326122 wait complete. 2024-04-24T08:08:56.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.773+0000 7f4d4fe00640 1 Processor -- start 2024-04-24T08:08:56.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.773+0000 7f4d4fe00640 1 -- start start 2024-04-24T08:08:56.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.774+0000 7f4d4fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d5006e000 0x7f4d500af850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.774+0000 7f4d4fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d50071d70 0x7f4d500c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.774+0000 7f4d4fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5015edf0 0x7f4d500c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.774+0000 7f4d4fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4d500d0690 con 0x7f4d5015edf0 2024-04-24T08:08:56.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.774+0000 7f4d4fe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4d500d03b0 con 0x7f4d50071d70 2024-04-24T08:08:56.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.774+0000 7f4d4fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4d500d0e90 con 0x7f4d5006e000 2024-04-24T08:08:56.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.774+0000 7f4d4f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5015edf0 0x7f4d500c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:56.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.775+0000 7f4d4f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5015edf0 0x7f4d500c57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37960/0 (socket says 172.21.15.69:37960) 2024-04-24T08:08:56.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.775+0000 7f4d4f400640 1 -- 172.21.15.69:0/114938870 learned_addr learned my addr 172.21.15.69:0/114938870 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.775+0000 7f4d4ea00640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d5006e000 0x7f4d500af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.775+0000 7f4d4f400640 1 -- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d5006e000 msgr2=0x7f4d500af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.775+0000 7f4d4f400640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d5006e000 0x7f4d500af850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.775+0000 7f4d4f400640 1 -- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d50071d70 msgr2=0x7f4d500c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.775+0000 7f4d4e000640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d50071d70 0x7f4d500c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.775+0000 7f4d4f400640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d50071d70 0x7f4d500c5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.775+0000 7f4d4f400640 1 -- 172.21.15.69:0/114938870 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d50141fb0 con 0x7f4d5015edf0 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.775+0000 7f4d4f400640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5015edf0 0x7f4d500c57d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009f70 tx=0x7f4d48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.776+0000 7f4d46a00640 1 -- 172.21.15.69:0/114938870 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48036050 con 0x7f4d5015edf0 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.776+0000 7f4d46a00640 1 -- 172.21.15.69:0/114938870 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d48042070 con 0x7f4d5015edf0 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.776+0000 7f4d4fe00640 1 -- 172.21.15.69:0/114938870 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d500b02a0 con 0x7f4d5015edf0 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.776+0000 7f4d46a00640 1 -- 172.21.15.69:0/114938870 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48009510 con 0x7f4d5015edf0 2024-04-24T08:08:56.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.776+0000 7f4d4fe00640 1 -- 172.21.15.69:0/114938870 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d500b0860 con 0x7f4d5015edf0 2024-04-24T08:08:56.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.777+0000 7f4d4fe00640 1 -- 172.21.15.69:0/114938870 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d50075d50 con 0x7f4d5015edf0 2024-04-24T08:08:56.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.778+0000 7f4d46a00640 1 -- 172.21.15.69:0/114938870 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d48007bd0 con 0x7f4d5015edf0 2024-04-24T08:08:56.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.779+0000 7f4d46a00640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d3006c8f0 0x7f4d3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:56.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.779+0000 7f4d46a00640 1 -- 172.21.15.69:0/114938870 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4d480ac9d0 con 0x7f4d5015edf0 2024-04-24T08:08:56.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.780+0000 7f4d4ea00640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d3006c8f0 0x7f4d3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:56.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.781+0000 7f4d4ea00640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d3006c8f0 0x7f4d3006edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4d40006fd0 tx=0x7f4d40008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:56.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.784+0000 7f4d46a00640 1 -- 172.21.15.69:0/114938870 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d48045360 con 0x7f4d5015edf0 2024-04-24T08:08:56.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.983+0000 7f4d4fe00640 1 -- 172.21.15.69:0/114938870 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4d5006cd50 con 0x7f4d5015edf0 2024-04-24T08:08:56.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.984+0000 7f4d46a00640 1 -- 172.21.15.69:0/114938870 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f4d4807adf0 con 0x7f4d5015edf0 2024-04-24T08:08:56.984 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:08:56.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.986+0000 7f4d4fe00640 1 -- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d3006c8f0 msgr2=0x7f4d3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.986+0000 7f4d4fe00640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d3006c8f0 0x7f4d3006edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4d40006fd0 tx=0x7f4d40008040 comp rx=0 tx=0).stop 2024-04-24T08:08:56.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.986+0000 7f4d4fe00640 1 -- 172.21.15.69:0/114938870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5015edf0 msgr2=0x7f4d500c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:56.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.986+0000 7f4d4fe00640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5015edf0 0x7f4d500c57d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009f70 tx=0x7f4d48004a40 comp rx=0 tx=0).stop 2024-04-24T08:08:56.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.986+0000 7f4d4fe00640 1 -- 172.21.15.69:0/114938870 shutdown_connections 2024-04-24T08:08:56.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.986+0000 7f4d4fe00640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d3006c8f0 0x7f4d3006edb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.986+0000 7f4d4fe00640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d5015edf0 0x7f4d500c57d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.986+0000 7f4d4fe00640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d50071d70 0x7f4d500c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.986+0000 7f4d4fe00640 1 --2- 172.21.15.69:0/114938870 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d5006e000 0x7f4d500af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:56.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.986+0000 7f4d4fe00640 1 -- 172.21.15.69:0/114938870 >> 172.21.15.69:0/114938870 conn(0x7f4d50067b80 msgr2=0x7f4d5015f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:56.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.987+0000 7f4d4fe00640 1 -- 172.21.15.69:0/114938870 shutdown_connections 2024-04-24T08:08:56.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:56.987+0000 7f4d4fe00640 1 -- 172.21.15.69:0/114938870 wait complete. 2024-04-24T08:08:57.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:08:57.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.204+0000 7fd97fe00640 1 Processor -- start 2024-04-24T08:08:57.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.204+0000 7fd97fe00640 1 -- start start 2024-04-24T08:08:57.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.206+0000 7fd97fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd980071d70 0x7fd98015ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.206+0000 7fd97fe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd980160510 0x7fd9800cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.206+0000 7fd97fe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9800721d0 con 0x7fd9801420a0 2024-04-24T08:08:57.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.206+0000 7fd97fe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd980160960 con 0x7fd980160510 2024-04-24T08:08:57.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.206+0000 7fd97fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9800cf1e0 con 0x7fd980071d70 2024-04-24T08:08:57.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.206+0000 7fd97ea00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd9801420a0 0x7fd980142480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47536/0 (socket says 172.21.15.69:47536) 2024-04-24T08:08:57.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.206+0000 7fd97ea00640 1 -- 172.21.15.69:0/3301435360 learned_addr learned my addr 172.21.15.69:0/3301435360 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:57.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97f400640 1 --2- 172.21.15.69:0/3301435360 >> v2:172.21.15.121:3300/0 conn(0x7fd980160510 0x7fd9800cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd977e00640 1 --2- 172.21.15.69:0/3301435360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd980071d70 0x7fd98015ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1818857498 0 0) 0x7fd970001110 con 0x7fd9801420a0 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd964003520 con 0x7fd9801420a0 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 276728520 0 0) 0x7fd964003520 con 0x7fd9801420a0 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd980071d70 msgr2=0x7fd98015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 --2- 172.21.15.69:0/3301435360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd980071d70 0x7fd98015ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 >> v2:172.21.15.121:3300/0 conn(0x7fd980160510 msgr2=0x7fd9800cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 --2- 172.21.15.69:0/3301435360 >> v2:172.21.15.121:3300/0 conn(0x7fd980160510 0x7fd9800cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd980072450 con 0x7fd9801420a0 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fd970001840 con 0x7fd9801420a0 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 >> v1:172.21.15.69:6789/0 conn(0x7fd9801420a0 legacy=0x7fd980142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 --2- 172.21.15.69:0/3301435360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd980071d70 0x7fd9640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 --2- 172.21.15.69:0/3301435360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd964003f80 0x7fd964004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 --2- 172.21.15.69:0/3301435360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd964004d40 0x7fd9640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd980160960 con 0x7fd964003f80 2024-04-24T08:08:57.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd9800cf1e0 con 0x7fd964004d40 2024-04-24T08:08:57.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd96400b9c0 con 0x7fd980071d70 2024-04-24T08:08:57.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.207+0000 7fd977e00640 1 --2- 172.21.15.69:0/3301435360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd964003f80 0x7fd964004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.208+0000 7fd977e00640 1 -- 172.21.15.69:0/3301435360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd980071d70 msgr2=0x7fd9640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:57.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.208+0000 7fd977e00640 1 --2- 172.21.15.69:0/3301435360 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd980071d70 0x7fd9640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.208+0000 7fd977e00640 1 -- 172.21.15.69:0/3301435360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd964004d40 msgr2=0x7fd9640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:57.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.208+0000 7fd977e00640 1 --2- 172.21.15.69:0/3301435360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd964004d40 0x7fd9640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.208+0000 7fd977e00640 1 -- 172.21.15.69:0/3301435360 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd964003520 con 0x7fd964003f80 2024-04-24T08:08:57.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.209+0000 7fd977e00640 1 --2- 172.21.15.69:0/3301435360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd964003f80 0x7fd964004800 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd968009af0 tx=0x7fd964005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab7738f992b7b47 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:57.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.209+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968036070 con 0x7fd964003f80 2024-04-24T08:08:57.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.209+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd968031400 con 0x7fd964003f80 2024-04-24T08:08:57.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.209+0000 7fd97e000640 1 -- 172.21.15.69:0/3301435360 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd968009690 con 0x7fd964003f80 2024-04-24T08:08:57.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.210+0000 7fd97fe00640 1 -- 172.21.15.69:0/3301435360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd964003f80 msgr2=0x7fd964004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.210+0000 7fd97fe00640 1 --2- 172.21.15.69:0/3301435360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd964003f80 0x7fd964004800 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd968009af0 tx=0x7fd964005830 comp rx=0 tx=0).stop 2024-04-24T08:08:57.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.210+0000 7fd97f400640 1 -- 172.21.15.69:0/3301435360 reap_dead start 2024-04-24T08:08:57.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.210+0000 7fd97fe00640 1 -- 172.21.15.69:0/3301435360 shutdown_connections 2024-04-24T08:08:57.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.210+0000 7fd97fe00640 1 -- 172.21.15.69:0/3301435360 >> 172.21.15.69:0/3301435360 conn(0x7fd980067b80 msgr2=0x7fd98006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:57.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.210+0000 7fd97fe00640 1 -- 172.21.15.69:0/3301435360 shutdown_connections 2024-04-24T08:08:57.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.210+0000 7fd97fe00640 1 -- 172.21.15.69:0/3301435360 wait complete. 2024-04-24T08:08:57.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.212+0000 7fd97fe00640 1 Processor -- start 2024-04-24T08:08:57.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.212+0000 7fd97fe00640 1 -- start start 2024-04-24T08:08:57.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd964003f80 0x7fd9801568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd98015dbe0 0x7fd98015c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd98015cd80 0x7fd98015d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd980072cf0 con 0x7fd98015cd80 2024-04-24T08:08:57.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97fe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd980072a10 con 0x7fd964003f80 2024-04-24T08:08:57.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd98015ffd0 con 0x7fd98015dbe0 2024-04-24T08:08:57.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd98015cd80 0x7fd98015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd98015cd80 0x7fd98015d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37978/0 (socket says 172.21.15.69:37978) 2024-04-24T08:08:57.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97f400640 1 -- 172.21.15.69:0/3504821189 learned_addr learned my addr 172.21.15.69:0/3504821189 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:57.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97ea00640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd964003f80 0x7fd9801568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97f400640 1 -- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd98015dbe0 msgr2=0x7fd98015c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97f400640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd98015dbe0 0x7fd98015c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97f400640 1 -- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd964003f80 msgr2=0x7fd9801568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97f400640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd964003f80 0x7fd9801568f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.213+0000 7fd97f400640 1 -- 172.21.15.69:0/3504821189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9801e6d80 con 0x7fd98015cd80 2024-04-24T08:08:57.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.214+0000 7fd97f400640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd98015cd80 0x7fd98015d200 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd9800b5720 tx=0x7fd97800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:57.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.214+0000 7fd976a00640 1 -- 172.21.15.69:0/3504821189 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97800bb10 con 0x7fd98015cd80 2024-04-24T08:08:57.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.214+0000 7fd976a00640 1 -- 172.21.15.69:0/3504821189 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd97801a070 con 0x7fd98015cd80 2024-04-24T08:08:57.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.214+0000 7fd976a00640 1 -- 172.21.15.69:0/3504821189 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd978014400 con 0x7fd98015cd80 2024-04-24T08:08:57.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.214+0000 7fd97fe00640 1 -- 172.21.15.69:0/3504821189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9801e7060 con 0x7fd98015cd80 2024-04-24T08:08:57.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.214+0000 7fd97fe00640 1 -- 172.21.15.69:0/3504821189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9801e6a30 con 0x7fd98015cd80 2024-04-24T08:08:57.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.215+0000 7fd97fe00640 1 -- 172.21.15.69:0/3504821189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd980075d50 con 0x7fd98015cd80 2024-04-24T08:08:57.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.217+0000 7fd976a00640 1 -- 172.21.15.69:0/3504821189 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd978005d10 con 0x7fd98015cd80 2024-04-24T08:08:57.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.217+0000 7fd976a00640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd95c06c8f0 0x7fd95c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.218+0000 7fd976a00640 1 -- 172.21.15.69:0/3504821189 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd97808be90 con 0x7fd98015cd80 2024-04-24T08:08:57.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.218+0000 7fd97ea00640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd95c06c8f0 0x7fd95c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.219+0000 7fd97ea00640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd95c06c8f0 0x7fd95c06edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd970001eb0 tx=0x7fd970009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:57.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.222+0000 7fd976a00640 1 -- 172.21.15.69:0/3504821189 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd978024020 con 0x7fd98015cd80 2024-04-24T08:08:57.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.425+0000 7fd97fe00640 1 -- 172.21.15.69:0/3504821189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd98013f2d0 con 0x7fd98015cd80 2024-04-24T08:08:57.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.426+0000 7fd976a00640 1 -- 172.21.15.69:0/3504821189 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fd97808d070 con 0x7fd98015cd80 2024-04-24T08:08:57.426 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 -- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd95c06c8f0 msgr2=0x7fd95c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd95c06c8f0 0x7fd95c06edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd970001eb0 tx=0x7fd970009040 comp rx=0 tx=0).stop 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 -- 172.21.15.69:0/3504821189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd98015cd80 msgr2=0x7fd98015d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd98015cd80 0x7fd98015d200 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd9800b5720 tx=0x7fd97800b480 comp rx=0 tx=0).stop 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 -- 172.21.15.69:0/3504821189 shutdown_connections 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd95c06c8f0 0x7fd95c06edb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd98015cd80 0x7fd98015d200 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd98015dbe0 0x7fd98015c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 --2- 172.21.15.69:0/3504821189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd964003f80 0x7fd9801568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 -- 172.21.15.69:0/3504821189 >> 172.21.15.69:0/3504821189 conn(0x7fd980067b80 msgr2=0x7fd98006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 -- 172.21.15.69:0/3504821189 shutdown_connections 2024-04-24T08:08:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.429+0000 7fd97fe00640 1 -- 172.21.15.69:0/3504821189 wait complete. 2024-04-24T08:08:57.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:57.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.655+0000 7f5efb200640 1 Processor -- start 2024-04-24T08:08:57.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.656+0000 7f5efb200640 1 -- start start 2024-04-24T08:08:57.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.657+0000 7f5efb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ef4145a90 0x7f5ef406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.657+0000 7f5efb200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef40cf3d0 con 0x7f5ef4145fe0 2024-04-24T08:08:57.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.657+0000 7f5efb200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ef40cf6b0 con 0x7f5ef40d0300 2024-04-24T08:08:57.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.657+0000 7f5efb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5ef40cf990 con 0x7f5ef4145a90 2024-04-24T08:08:57.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.657+0000 7f5ef9200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5ef4145fe0 0x7f5ef4160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47550/0 (socket says 172.21.15.69:47550) 2024-04-24T08:08:57.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.657+0000 7f5ef9200640 1 -- 172.21.15.69:0/3864497794 learned_addr learned my addr 172.21.15.69:0/3864497794 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:57.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5eeb400640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ef4145a90 0x7f5ef406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1650529011 0 0) 0x7f5ef40cf3d0 con 0x7f5ef4145fe0 2024-04-24T08:08:57.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ed8003520 con 0x7f5ef4145fe0 2024-04-24T08:08:57.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1551353783 0 0) 0x7f5ed8003520 con 0x7f5ef4145fe0 2024-04-24T08:08:57.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ef4145a90 msgr2=0x7f5ef406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:57.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5ef3400640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ef4145a90 0x7f5ef406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 >> v1:172.21.15.121:6789/0 conn(0x7f5ef40d0300 legacy=0x7f5ef40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef40cfc70 con 0x7f5ef4145fe0 2024-04-24T08:08:57.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f5eec0019f0 con 0x7f5ef4145fe0 2024-04-24T08:08:57.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 >> v1:172.21.15.69:6789/0 conn(0x7f5ef4145fe0 legacy=0x7f5ef4160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.658+0000 7f5ef3400640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ed80039e0 0x7f5ed8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5ef3400640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ed8004390 0x7f5ed8004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5ef3400640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ef4145a90 0x7f5ed80050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef40cf990 con 0x7f5ed8004390 2024-04-24T08:08:57.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5ed800b9c0 con 0x7f5ed80039e0 2024-04-24T08:08:57.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5ed800bbc0 con 0x7f5ef4145a90 2024-04-24T08:08:57.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5eeb400640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ed8004390 0x7f5ed8004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5ef3e00640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ed80039e0 0x7f5ed8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5eeb400640 1 -- 172.21.15.69:0/3864497794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ef4145a90 msgr2=0x7f5ed80050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5eeb400640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ef4145a90 0x7f5ed80050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5eeb400640 1 -- 172.21.15.69:0/3864497794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ed80039e0 msgr2=0x7f5ed8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5eeb400640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ed80039e0 0x7f5ed8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.659+0000 7f5eeb400640 1 -- 172.21.15.69:0/3864497794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ed8003520 con 0x7f5ed8004390 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.660+0000 7f5eeb400640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ed8004390 0x7f5ed8004860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5edc009ac0 tx=0x7f5ed8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8628158f4aa0bd7d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.661+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc036070 con 0x7f5ed8004390 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.661+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5edc031400 con 0x7f5ed8004390 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.661+0000 7f5ef3400640 1 -- 172.21.15.69:0/3864497794 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc009690 con 0x7f5ed8004390 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.661+0000 7f5efb200640 1 -- 172.21.15.69:0/3864497794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ed8004390 msgr2=0x7f5ed8004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.661+0000 7f5efb200640 1 --2- 172.21.15.69:0/3864497794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ed8004390 0x7f5ed8004860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5edc009ac0 tx=0x7f5ed8005830 comp rx=0 tx=0).stop 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.661+0000 7f5ef9200640 1 -- 172.21.15.69:0/3864497794 reap_dead start 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.661+0000 7f5efb200640 1 -- 172.21.15.69:0/3864497794 shutdown_connections 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.661+0000 7f5efb200640 1 -- 172.21.15.69:0/3864497794 >> 172.21.15.69:0/3864497794 conn(0x7f5ef4067d90 msgr2=0x7f5ef406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:57.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.661+0000 7f5efb200640 1 -- 172.21.15.69:0/3864497794 shutdown_connections 2024-04-24T08:08:57.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.661+0000 7f5efb200640 1 -- 172.21.15.69:0/3864497794 wait complete. 2024-04-24T08:08:57.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.663+0000 7f5efb200640 1 Processor -- start 2024-04-24T08:08:57.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.663+0000 7f5efb200640 1 -- start start 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.664+0000 7f5efb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ed8004390 0x7f5ef40b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.664+0000 7f5efb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef412f8c0 0x7f5ef41ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.664+0000 7f5efb200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef40a9430 0x7f5ef40a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.664+0000 7f5efb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef41458f0 con 0x7f5ef412f8c0 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.664+0000 7f5efb200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef4145770 con 0x7f5ef40a9430 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.664+0000 7f5efb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5ef40cdc30 con 0x7f5ed8004390 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.664+0000 7f5eeb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef412f8c0 0x7f5ef41ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.664+0000 7f5eeb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef412f8c0 0x7f5ef41ef820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37998/0 (socket says 172.21.15.69:37998) 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.664+0000 7f5eeb400640 1 -- 172.21.15.69:0/3581315513 learned_addr learned my addr 172.21.15.69:0/3581315513 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5ef9200640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef40a9430 0x7f5ef40a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5ef3e00640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ed8004390 0x7f5ef40b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5eeb400640 1 -- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ed8004390 msgr2=0x7f5ef40b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5eeb400640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ed8004390 0x7f5ef40b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5eeb400640 1 -- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef40a9430 msgr2=0x7f5ef40a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5eeb400640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef40a9430 0x7f5ef40a98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5eeb400640 1 -- 172.21.15.69:0/3581315513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef41300f0 con 0x7f5ef412f8c0 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5ef9200640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef40a9430 0x7f5ef40a98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5eeb400640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef412f8c0 0x7f5ef41ef820 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5edc02abe0 tx=0x7f5edc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5ef0c00640 1 -- 172.21.15.69:0/3581315513 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc040020 con 0x7f5ef412f8c0 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5ef0c00640 1 -- 172.21.15.69:0/3581315513 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5edc0038d0 con 0x7f5ef412f8c0 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5ef0c00640 1 -- 172.21.15.69:0/3581315513 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc036040 con 0x7f5ef412f8c0 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.665+0000 7f5efb200640 1 -- 172.21.15.69:0/3581315513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef41303d0 con 0x7f5ef412f8c0 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.666+0000 7f5efb200640 1 -- 172.21.15.69:0/3581315513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef41efd60 con 0x7f5ef412f8c0 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.667+0000 7f5efb200640 1 -- 172.21.15.69:0/3581315513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ef412d880 con 0x7f5ef412f8c0 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.668+0000 7f5ef0c00640 1 -- 172.21.15.69:0/3581315513 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5edc043020 con 0x7f5ef412f8c0 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.669+0000 7f5ef0c00640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ed006c8f0 0x7f5ed006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.669+0000 7f5ef0c00640 1 -- 172.21.15.69:0/3581315513 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5edc0abc90 con 0x7f5ef412f8c0 2024-04-24T08:08:57.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.669+0000 7f5ef3e00640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ed006c8f0 0x7f5ed006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:57.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.670+0000 7f5ef3e00640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ed006c8f0 0x7f5ed006edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40cd3f0 tx=0x7f5ee4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:57.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.673+0000 7f5ef0c00640 1 -- 172.21.15.69:0/3581315513 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5edc04a070 con 0x7f5ef412f8c0 2024-04-24T08:08:57.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.874+0000 7f5efb200640 1 -- 172.21.15.69:0/3581315513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ef4068a80 con 0x7f5ef412f8c0 2024-04-24T08:08:57.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.876+0000 7f5ef0c00640 1 -- 172.21.15.69:0/3581315513 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7f5edc07a0b0 con 0x7f5ef412f8c0 2024-04-24T08:08:57.877 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:57.877 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.878+0000 7f5efb200640 1 -- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ed006c8f0 msgr2=0x7f5ed006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ed006c8f0 0x7f5ed006edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5ef40cd3f0 tx=0x7f5ee4009040 comp rx=0 tx=0).stop 2024-04-24T08:08:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 -- 172.21.15.69:0/3581315513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef412f8c0 msgr2=0x7f5ef41ef820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef412f8c0 0x7f5ef41ef820 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5edc02abe0 tx=0x7f5edc004730 comp rx=0 tx=0).stop 2024-04-24T08:08:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 -- 172.21.15.69:0/3581315513 shutdown_connections 2024-04-24T08:08:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ed006c8f0 0x7f5ed006edb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ef412f8c0 0x7f5ef41ef820 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ef40a9430 0x7f5ef40a98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 --2- 172.21.15.69:0/3581315513 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ed8004390 0x7f5ef40b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 -- 172.21.15.69:0/3581315513 >> 172.21.15.69:0/3581315513 conn(0x7f5ef4067d90 msgr2=0x7f5ef406e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 -- 172.21.15.69:0/3581315513 shutdown_connections 2024-04-24T08:08:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:57.879+0000 7f5efb200640 1 -- 172.21.15.69:0/3581315513 wait complete. 2024-04-24T08:08:57.899 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:08:57.899 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:08:58.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.102+0000 7f62bb200640 1 Processor -- start 2024-04-24T08:08:58.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.102+0000 7f62bb200640 1 -- start start 2024-04-24T08:08:58.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.104+0000 7f62bb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b41606e0 0x7f62b4167360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.104+0000 7f62bb200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62b4072450 con 0x7f62b4071d70 2024-04-24T08:08:58.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.104+0000 7f62bb200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62b4072730 con 0x7f62b41420a0 2024-04-24T08:08:58.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.104+0000 7f62bb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62b40cf1e0 con 0x7f62b41606e0 2024-04-24T08:08:58.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.104+0000 7f62b9400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f62b4071d70 0x7f62b415ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47564/0 (socket says 172.21.15.69:47564) 2024-04-24T08:08:58.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.104+0000 7f62b9400640 1 -- 172.21.15.69:0/530822228 learned_addr learned my addr 172.21.15.69:0/530822228 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.104+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592399499 0 0) 0x7f62b4072450 con 0x7f62b4071d70 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.104+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f629c003520 con 0x7f62b4071d70 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.104+0000 7f62ba800640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b41606e0 0x7f62b4167360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 711514593 0 0) 0x7f629c003520 con 0x7f62b4071d70 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b41606e0 msgr2=0x7f62b4167360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b41606e0 0x7f62b4167360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 >> v1:172.21.15.121:6789/0 conn(0x7f62b41420a0 legacy=0x7f62b4142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b4072a10 con 0x7f62b4071d70 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f62a4001670 con 0x7f62b4071d70 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 >> v1:172.21.15.69:6789/0 conn(0x7f62b4071d70 legacy=0x7f62b415ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 0x7f629c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b41606e0 0x7f629c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f629c004d40 0x7f629c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62b40cf1e0 con 0x7f629c0039e0 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f629c00bb50 con 0x7f629c004d40 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f629c00bd50 con 0x7f62b41606e0 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.105+0000 7f62b9e00640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 0x7f629c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.106+0000 7f62b9400640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b41606e0 0x7f629c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.106+0000 7f62ba800640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f629c004d40 0x7f629c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.106+0000 7f62b9e00640 1 -- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b41606e0 msgr2=0x7f629c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.106+0000 7f62b9e00640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b41606e0 0x7f629c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.106+0000 7f62b9e00640 1 -- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f629c004d40 msgr2=0x7f629c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.106+0000 7f62b9e00640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f629c004d40 0x7f629c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.106+0000 7f62b9e00640 1 -- 172.21.15.69:0/530822228 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629c003520 con 0x7f629c0039e0 2024-04-24T08:08:58.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.107+0000 7f62b9e00640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 0x7f629c003e00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f62a8001750 tx=0x7f629c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=349dd56bf9d8387 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.107+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8001370 con 0x7f629c0039e0 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.107+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a80052d0 con 0x7f629c0039e0 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.107+0000 7f62b8a00640 1 -- 172.21.15.69:0/530822228 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a802abe0 con 0x7f629c0039e0 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.108+0000 7f62bb200640 1 -- 172.21.15.69:0/530822228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 msgr2=0x7f629c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.108+0000 7f62bb200640 1 --2- 172.21.15.69:0/530822228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 0x7f629c003e00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f62a8001750 tx=0x7f629c005830 comp rx=0 tx=0).stop 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.108+0000 7f62ba800640 1 -- 172.21.15.69:0/530822228 reap_dead start 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.108+0000 7f62bb200640 1 -- 172.21.15.69:0/530822228 shutdown_connections 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.108+0000 7f62bb200640 1 -- 172.21.15.69:0/530822228 >> 172.21.15.69:0/530822228 conn(0x7f62b4067b80 msgr2=0x7f62b406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.108+0000 7f62bb200640 1 -- 172.21.15.69:0/530822228 shutdown_connections 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.108+0000 7f62bb200640 1 -- 172.21.15.69:0/530822228 wait complete. 2024-04-24T08:08:58.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.110+0000 7f62bb200640 1 Processor -- start 2024-04-24T08:08:58.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.110+0000 7f62bb200640 1 -- start start 2024-04-24T08:08:58.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62bb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 0x7f62b40aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62bb200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62b40b48d0 0x7f62b40b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62bb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b415b420 0x7f62b415b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62bb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62b40d0680 con 0x7f629c0039e0 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62bb200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f62b4072200 con 0x7f62b40b48d0 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62bb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62b40d0800 con 0x7f62b415b420 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62b9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 0x7f62b40aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62b9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 0x7f62b40aff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38014/0 (socket says 172.21.15.69:38014) 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62b9e00640 1 -- 172.21.15.69:0/2373001095 learned_addr learned my addr 172.21.15.69:0/2373001095 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62b9e00640 1 -- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b415b420 msgr2=0x7f62b415b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62b9e00640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b415b420 0x7f62b415b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62b9e00640 1 -- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62b40b48d0 msgr2=0x7f62b40b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62b9e00640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62b40b48d0 0x7f62b40b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.111+0000 7f62b9e00640 1 -- 172.21.15.69:0/2373001095 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b40b06e0 con 0x7f629c0039e0 2024-04-24T08:08:58.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.112+0000 7f62b9e00640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 0x7f62b40aff90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f62a8001870 tx=0x7f62a8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:58.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.112+0000 7f62ae000640 1 -- 172.21.15.69:0/2373001095 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8037040 con 0x7f629c0039e0 2024-04-24T08:08:58.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.112+0000 7f62ae000640 1 -- 172.21.15.69:0/2373001095 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a8042070 con 0x7f629c0039e0 2024-04-24T08:08:58.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.112+0000 7f62ae000640 1 -- 172.21.15.69:0/2373001095 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a8007a30 con 0x7f629c0039e0 2024-04-24T08:08:58.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.112+0000 7f62bb200640 1 -- 172.21.15.69:0/2373001095 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b40b09c0 con 0x7f629c0039e0 2024-04-24T08:08:58.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.112+0000 7f62bb200640 1 -- 172.21.15.69:0/2373001095 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b40b0f80 con 0x7f629c0039e0 2024-04-24T08:08:58.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.113+0000 7f62bb200640 1 -- 172.21.15.69:0/2373001095 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62b4075d50 con 0x7f629c0039e0 2024-04-24T08:08:58.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.114+0000 7f62ae000640 1 -- 172.21.15.69:0/2373001095 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f62a80052d0 con 0x7f629c0039e0 2024-04-24T08:08:58.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.115+0000 7f62ae000640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f629806c920 0x7f629806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.115+0000 7f62ae000640 1 -- 172.21.15.69:0/2373001095 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f62a80abec0 con 0x7f629c0039e0 2024-04-24T08:08:58.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.116+0000 7f62b9400640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f629806c920 0x7f629806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.117+0000 7f62b9400640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f629806c920 0x7f629806ede0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f62a4001050 tx=0x7f62a40011c0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:58.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.119+0000 7f62ae000640 1 -- 172.21.15.69:0/2373001095 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62a807a2f0 con 0x7f629c0039e0 2024-04-24T08:08:58.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.317+0000 7f62bb200640 1 -- 172.21.15.69:0/2373001095 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f62b41f4650 con 0x7f629c0039e0 2024-04-24T08:08:58.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.318+0000 7f62ae000640 1 -- 172.21.15.69:0/2373001095 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f62a8079bf0 con 0x7f629c0039e0 2024-04-24T08:08:58.319 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:08:58.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 -- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f629806c920 msgr2=0x7f629806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f629806c920 0x7f629806ede0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f62a4001050 tx=0x7f62a40011c0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 -- 172.21.15.69:0/2373001095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 msgr2=0x7f62b40aff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 0x7f62b40aff90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f62a8001870 tx=0x7f62a8001120 comp rx=0 tx=0).stop 2024-04-24T08:08:58.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 -- 172.21.15.69:0/2373001095 shutdown_connections 2024-04-24T08:08:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f629806c920 0x7f629806ede0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62b415b420 0x7f62b415b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62b40b48d0 0x7f62b40b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 --2- 172.21.15.69:0/2373001095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f629c0039e0 0x7f62b40aff90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 -- 172.21.15.69:0/2373001095 >> 172.21.15.69:0/2373001095 conn(0x7f62b4067b80 msgr2=0x7f62b4152f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 -- 172.21.15.69:0/2373001095 shutdown_connections 2024-04-24T08:08:58.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.321+0000 7f62bb200640 1 -- 172.21.15.69:0/2373001095 wait complete. 2024-04-24T08:08:58.341 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:08:58.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.537+0000 7f3a3f600640 1 Processor -- start 2024-04-24T08:08:58.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.538+0000 7f3a3f600640 1 -- start start 2024-04-24T08:08:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.539+0000 7f3a3f600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3a3815f4b0 0x7f3a380b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.539+0000 7f3a3f600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a380b46a0 0x7f3a380b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.539+0000 7f3a3f600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a380739f0 con 0x7f3a3815f890 2024-04-24T08:08:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.539+0000 7f3a3f600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3a380b5010 con 0x7f3a3815f4b0 2024-04-24T08:08:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.539+0000 7f3a3f600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3a38073cc0 con 0x7f3a380b46a0 2024-04-24T08:08:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.539+0000 7f3a3cc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3a3815f4b0 0x7f3a380b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.540+0000 7f3a3cc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3a3815f4b0 0x7f3a380b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38026/0 (socket says 172.21.15.69:38026) 2024-04-24T08:08:58.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.540+0000 7f3a3cc00640 1 -- 172.21.15.69:0/127805984 learned_addr learned my addr 172.21.15.69:0/127805984 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.540+0000 7f3a3cc00640 1 -- 172.21.15.69:0/127805984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a380b46a0 msgr2=0x7f3a380b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:58.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.540+0000 7f3a3d600640 1 --2- 172.21.15.69:0/127805984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a380b46a0 0x7f3a380b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.540+0000 7f3a3cc00640 1 --2- 172.21.15.69:0/127805984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a380b46a0 0x7f3a380b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.540+0000 7f3a3cc00640 1 -- 172.21.15.69:0/127805984 >> v1:172.21.15.121:6789/0 conn(0x7f3a3815f890 legacy=0x7f3a380732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:58.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.540+0000 7f3a3cc00640 1 -- 172.21.15.69:0/127805984 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a38073fa0 con 0x7f3a3815f4b0 2024-04-24T08:08:58.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.541+0000 7f3a3cc00640 1 --2- 172.21.15.69:0/127805984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f4b0 0x7f3a380b3de0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3a380cde60 tx=0x7f3a2802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a82e771e646fc45 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:58.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.541+0000 7f3a37e00640 1 -- 172.21.15.69:0/127805984 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28037070 con 0x7f3a3815f4b0 2024-04-24T08:08:58.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.541+0000 7f3a37e00640 1 -- 172.21.15.69:0/127805984 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a28032440 con 0x7f3a3815f4b0 2024-04-24T08:08:58.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.541+0000 7f3a37e00640 1 -- 172.21.15.69:0/127805984 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a28032810 con 0x7f3a3815f4b0 2024-04-24T08:08:58.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.542+0000 7f3a3f600640 1 -- 172.21.15.69:0/127805984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f4b0 msgr2=0x7f3a380b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.542+0000 7f3a3f600640 1 --2- 172.21.15.69:0/127805984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f4b0 0x7f3a380b3de0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3a380cde60 tx=0x7f3a2802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.542+0000 7f3a3f600640 1 -- 172.21.15.69:0/127805984 shutdown_connections 2024-04-24T08:08:58.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.542+0000 7f3a3f600640 1 --2- 172.21.15.69:0/127805984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a380b46a0 0x7f3a380b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.542+0000 7f3a3f600640 1 --2- 172.21.15.69:0/127805984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f4b0 0x7f3a380b3de0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.542+0000 7f3a3f600640 1 -- 172.21.15.69:0/127805984 >> 172.21.15.69:0/127805984 conn(0x7f3a380d0350 msgr2=0x7f3a380d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:58.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.542+0000 7f3a3f600640 1 -- 172.21.15.69:0/127805984 shutdown_connections 2024-04-24T08:08:58.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.542+0000 7f3a3f600640 1 -- 172.21.15.69:0/127805984 wait complete. 2024-04-24T08:08:58.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.544+0000 7f3a3f600640 1 Processor -- start 2024-04-24T08:08:58.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.545+0000 7f3a3f600640 1 -- start start 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3f600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a380b46a0 0x7f3a380cee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3f600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a3815f4b0 0x7f3a381e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3f600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f890 0x7f3a381e9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3f600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3a380b38c0 con 0x7f3a3815f890 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3f600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3a380b35e0 con 0x7f3a3815f4b0 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3f600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3a3815fd40 con 0x7f3a380b46a0 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f890 0x7f3a381e9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f890 0x7f3a381e9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38040/0 (socket says 172.21.15.69:38040) 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3d600640 1 -- 172.21.15.69:0/445737332 learned_addr learned my addr 172.21.15.69:0/445737332 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a2fe00640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a3815f4b0 0x7f3a381e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3d600640 1 -- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a380b46a0 msgr2=0x7f3a380cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3cc00640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a380b46a0 0x7f3a380cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3d600640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a380b46a0 0x7f3a380cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3d600640 1 -- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a3815f4b0 msgr2=0x7f3a381e8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3d600640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a3815f4b0 0x7f3a381e8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.546+0000 7f3a3d600640 1 -- 172.21.15.69:0/445737332 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a381569a0 con 0x7f3a3815f890 2024-04-24T08:08:58.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.547+0000 7f3a2fe00640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a3815f4b0 0x7f3a381e8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:58.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.547+0000 7f3a3d600640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f890 0x7f3a381e9250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000cc30 tx=0x7f3a3000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:58.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.547+0000 7f3a35600640 1 -- 172.21.15.69:0/445737332 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3000ce40 con 0x7f3a3815f890 2024-04-24T08:08:58.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.547+0000 7f3a35600640 1 -- 172.21.15.69:0/445737332 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a30003990 con 0x7f3a3815f890 2024-04-24T08:08:58.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.547+0000 7f3a3f600640 1 -- 172.21.15.69:0/445737332 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a381e9880 con 0x7f3a3815f890 2024-04-24T08:08:58.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.547+0000 7f3a35600640 1 -- 172.21.15.69:0/445737332 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30003d40 con 0x7f3a3815f890 2024-04-24T08:08:58.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.547+0000 7f3a3f600640 1 -- 172.21.15.69:0/445737332 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a381545b0 con 0x7f3a3815f890 2024-04-24T08:08:58.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.548+0000 7f3a3f600640 1 -- 172.21.15.69:0/445737332 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a38068a80 con 0x7f3a3815f890 2024-04-24T08:08:58.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.549+0000 7f3a35600640 1 -- 172.21.15.69:0/445737332 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3a30004000 con 0x7f3a3815f890 2024-04-24T08:08:58.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.550+0000 7f3a35600640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3a1406c8f0 0x7f3a1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.550+0000 7f3a35600640 1 -- 172.21.15.69:0/445737332 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3a3008c080 con 0x7f3a3815f890 2024-04-24T08:08:58.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.551+0000 7f3a3cc00640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3a1406c8f0 0x7f3a1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.552+0000 7f3a3cc00640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3a1406c8f0 0x7f3a1406edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3a28034000 tx=0x7f3a2802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:58.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.555+0000 7f3a35600640 1 -- 172.21.15.69:0/445737332 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a30014030 con 0x7f3a3815f890 2024-04-24T08:08:58.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.753+0000 7f3a3f600640 1 -- 172.21.15.69:0/445737332 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3a381606b0 con 0x7f3a3815f890 2024-04-24T08:08:58.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.754+0000 7f3a35600640 1 -- 172.21.15.69:0/445737332 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f3a3005a4b0 con 0x7f3a3815f890 2024-04-24T08:08:58.754 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:08:58.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.756+0000 7f3a3f600640 1 -- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3a1406c8f0 msgr2=0x7f3a1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.756+0000 7f3a3f600640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3a1406c8f0 0x7f3a1406edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3a28034000 tx=0x7f3a2802e000 comp rx=0 tx=0).stop 2024-04-24T08:08:58.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.756+0000 7f3a3f600640 1 -- 172.21.15.69:0/445737332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f890 msgr2=0x7f3a381e9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.756+0000 7f3a3f600640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f890 0x7f3a381e9250 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000cc30 tx=0x7f3a3000a5c0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.756+0000 7f3a3f600640 1 -- 172.21.15.69:0/445737332 shutdown_connections 2024-04-24T08:08:58.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.756+0000 7f3a3f600640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3a1406c8f0 0x7f3a1406edb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.756+0000 7f3a3f600640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3a3815f890 0x7f3a381e9250 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.756+0000 7f3a3f600640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3a3815f4b0 0x7f3a381e8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.756+0000 7f3a3f600640 1 --2- 172.21.15.69:0/445737332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3a380b46a0 0x7f3a380cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.756+0000 7f3a3f600640 1 -- 172.21.15.69:0/445737332 >> 172.21.15.69:0/445737332 conn(0x7f3a380d0350 msgr2=0x7f3a38160990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:58.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.757+0000 7f3a3f600640 1 -- 172.21.15.69:0/445737332 shutdown_connections 2024-04-24T08:08:58.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.757+0000 7f3a3f600640 1 -- 172.21.15.69:0/445737332 wait complete. 2024-04-24T08:08:58.774 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:08:58.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.978+0000 7f2a9a800640 1 Processor -- start 2024-04-24T08:08:58.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.978+0000 7f2a9a800640 1 -- start start 2024-04-24T08:08:58.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.980+0000 7f2a9a800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2a9415e980 0x7f2a94162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.980+0000 7f2a9a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a9406cfb0 0x7f2a9406d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.980+0000 7f2a9a800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a9415ed60 con 0x7f2a940d0da0 2024-04-24T08:08:58.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.980+0000 7f2a9a800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2a940d1580 con 0x7f2a9415e980 2024-04-24T08:08:58.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.980+0000 7f2a9a800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a940d1860 con 0x7f2a9406cfb0 2024-04-24T08:08:58.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.981+0000 7f2a99400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2a9415e980 0x7f2a94162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.981+0000 7f2a99400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2a9415e980 0x7f2a94162f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:33140/0 (socket says 172.21.15.69:33140) 2024-04-24T08:08:58.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.981+0000 7f2a99400640 1 -- 172.21.15.69:0/4281344797 learned_addr learned my addr 172.21.15.69:0/4281344797 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:58.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.981+0000 7f2a99e00640 1 --2- 172.21.15.69:0/4281344797 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a9406cfb0 0x7f2a9406d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.981+0000 7f2a99400640 1 -- 172.21.15.69:0/4281344797 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a9406cfb0 msgr2=0x7f2a9406d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:58.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.981+0000 7f2a99400640 1 --2- 172.21.15.69:0/4281344797 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a9406cfb0 0x7f2a9406d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.981+0000 7f2a99400640 1 -- 172.21.15.69:0/4281344797 >> v1:172.21.15.121:6789/0 conn(0x7f2a940d0da0 legacy=0x7f2a940cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.981+0000 7f2a99400640 1 -- 172.21.15.69:0/4281344797 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a940cf450 con 0x7f2a9415e980 2024-04-24T08:08:58.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.983+0000 7f2a99400640 1 --2- 172.21.15.69:0/4281344797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a9415e980 0x7f2a94162f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009870 tx=0x7f2a8802a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a7dcadd072d1e4d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:58.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.983+0000 7f2a93e00640 1 -- 172.21.15.69:0/4281344797 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88037070 con 0x7f2a9415e980 2024-04-24T08:08:58.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.983+0000 7f2a93e00640 1 -- 172.21.15.69:0/4281344797 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a8802ae50 con 0x7f2a9415e980 2024-04-24T08:08:58.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.983+0000 7f2a93e00640 1 -- 172.21.15.69:0/4281344797 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88032860 con 0x7f2a9415e980 2024-04-24T08:08:58.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.984+0000 7f2a9a800640 1 -- 172.21.15.69:0/4281344797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a9415e980 msgr2=0x7f2a94162f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.984+0000 7f2a9a800640 1 --2- 172.21.15.69:0/4281344797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a9415e980 0x7f2a94162f90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009870 tx=0x7f2a8802a300 comp rx=0 tx=0).stop 2024-04-24T08:08:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.984+0000 7f2a9a800640 1 -- 172.21.15.69:0/4281344797 shutdown_connections 2024-04-24T08:08:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.984+0000 7f2a9a800640 1 --2- 172.21.15.69:0/4281344797 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a9406cfb0 0x7f2a9406d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.984+0000 7f2a9a800640 1 --2- 172.21.15.69:0/4281344797 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a9415e980 0x7f2a94162f90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.984+0000 7f2a9a800640 1 -- 172.21.15.69:0/4281344797 >> 172.21.15.69:0/4281344797 conn(0x7f2a9409d920 msgr2=0x7f2a9408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.984+0000 7f2a9a800640 1 -- 172.21.15.69:0/4281344797 shutdown_connections 2024-04-24T08:08:58.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.984+0000 7f2a9a800640 1 -- 172.21.15.69:0/4281344797 wait complete. 2024-04-24T08:08:58.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.987+0000 7f2a9a800640 1 Processor -- start 2024-04-24T08:08:58.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.987+0000 7f2a9a800640 1 -- start start 2024-04-24T08:08:58.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.988+0000 7f2a9a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9406cfb0 0x7f2a94150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.988+0000 7f2a9a800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a940d0da0 0x7f2a94153080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.988+0000 7f2a9a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a9415e980 0x7f2a9415b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.988+0000 7f2a9a800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a940d11e0 con 0x7f2a9406cfb0 2024-04-24T08:08:58.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.988+0000 7f2a9a800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2a940d0360 con 0x7f2a940d0da0 2024-04-24T08:08:58.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.988+0000 7f2a9a800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a940d1360 con 0x7f2a9415e980 2024-04-24T08:08:58.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.988+0000 7f2a99400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9406cfb0 0x7f2a94150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.988+0000 7f2a99400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9406cfb0 0x7f2a94150d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34286/0 (socket says 172.21.15.69:34286) 2024-04-24T08:08:58.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.988+0000 7f2a99400640 1 -- 172.21.15.69:0/627530421 learned_addr learned my addr 172.21.15.69:0/627530421 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:58.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.989+0000 7f2a98a00640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a940d0da0 0x7f2a94153080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.989+0000 7f2a99400640 1 -- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a9415e980 msgr2=0x7f2a9415b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.989+0000 7f2a99400640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a9415e980 0x7f2a9415b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.989+0000 7f2a99400640 1 -- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a940d0da0 msgr2=0x7f2a94153080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:58.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.989+0000 7f2a99400640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a940d0da0 0x7f2a94153080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:58.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.989+0000 7f2a99400640 1 -- 172.21.15.69:0/627530421 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a940c92b0 con 0x7f2a9406cfb0 2024-04-24T08:08:58.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.989+0000 7f2a99400640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9406cfb0 0x7f2a94150d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2a88000c00 tx=0x7f2a880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:58.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.989+0000 7f2a91600640 1 -- 172.21.15.69:0/627530421 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88041070 con 0x7f2a9406cfb0 2024-04-24T08:08:58.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.990+0000 7f2a91600640 1 -- 172.21.15.69:0/627530421 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a8803c660 con 0x7f2a9406cfb0 2024-04-24T08:08:58.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.990+0000 7f2a91600640 1 -- 172.21.15.69:0/627530421 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88037040 con 0x7f2a9406cfb0 2024-04-24T08:08:58.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.990+0000 7f2a9a800640 1 -- 172.21.15.69:0/627530421 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a940c9590 con 0x7f2a9406cfb0 2024-04-24T08:08:58.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.990+0000 7f2a9a800640 1 -- 172.21.15.69:0/627530421 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a940c9b50 con 0x7f2a9406cfb0 2024-04-24T08:08:58.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.991+0000 7f2a9a800640 1 -- 172.21.15.69:0/627530421 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a94075d50 con 0x7f2a9406cfb0 2024-04-24T08:08:58.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.992+0000 7f2a91600640 1 -- 172.21.15.69:0/627530421 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a88044070 con 0x7f2a9406cfb0 2024-04-24T08:08:58.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.993+0000 7f2a91600640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a7406c8f0 0x7f2a7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:58.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.993+0000 7f2a91600640 1 -- 172.21.15.69:0/627530421 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2a880ac3a0 con 0x7f2a9406cfb0 2024-04-24T08:08:58.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.993+0000 7f2a98a00640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a7406c8f0 0x7f2a7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:58.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.995+0000 7f2a98a00640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a7406c8f0 0x7f2a7406edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2a940cb090 tx=0x7f2a80009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:58.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:58.997+0000 7f2a91600640 1 -- 172.21.15.69:0/627530421 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a8807a8e0 con 0x7f2a9406cfb0 2024-04-24T08:08:59.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.200+0000 7f2a9a800640 1 -- 172.21.15.69:0/627530421 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2a940d1e40 con 0x7f2a9406cfb0 2024-04-24T08:08:59.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.200+0000 7f2a91600640 1 -- 172.21.15.69:0/627530421 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f2a8807a1e0 con 0x7f2a9406cfb0 2024-04-24T08:08:59.201 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:08:59.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.203+0000 7f2a9a800640 1 -- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a7406c8f0 msgr2=0x7f2a7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.203+0000 7f2a9a800640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a7406c8f0 0x7f2a7406edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2a940cb090 tx=0x7f2a80009040 comp rx=0 tx=0).stop 2024-04-24T08:08:59.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.203+0000 7f2a9a800640 1 -- 172.21.15.69:0/627530421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9406cfb0 msgr2=0x7f2a94150d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.203+0000 7f2a9a800640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9406cfb0 0x7f2a94150d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2a88000c00 tx=0x7f2a880046c0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.203+0000 7f2a9a800640 1 -- 172.21.15.69:0/627530421 shutdown_connections 2024-04-24T08:08:59.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.203+0000 7f2a9a800640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a7406c8f0 0x7f2a7406edb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.203+0000 7f2a9a800640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a9415e980 0x7f2a9415b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.203+0000 7f2a9a800640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a940d0da0 0x7f2a94153080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.203+0000 7f2a9a800640 1 --2- 172.21.15.69:0/627530421 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9406cfb0 0x7f2a94150d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.203+0000 7f2a9a800640 1 -- 172.21.15.69:0/627530421 >> 172.21.15.69:0/627530421 conn(0x7f2a9409d920 msgr2=0x7f2a941e8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:59.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.204+0000 7f2a9a800640 1 -- 172.21.15.69:0/627530421 shutdown_connections 2024-04-24T08:08:59.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.204+0000 7f2a9a800640 1 -- 172.21.15.69:0/627530421 wait complete. 2024-04-24T08:08:59.243 INFO:tasks.cephfs_test_runner:Starting test: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-24T08:08:59.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:59.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.438+0000 7fb1e2c00640 1 Processor -- start 2024-04-24T08:08:59.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.438+0000 7fb1e2c00640 1 -- start start 2024-04-24T08:08:59.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.440+0000 7fb1e2c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1dc145a90 0x7fb1dc06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.440+0000 7fb1e2c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1dc0cf3d0 con 0x7fb1dc145fe0 2024-04-24T08:08:59.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.440+0000 7fb1e2c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1dc0cf6b0 con 0x7fb1dc0d0300 2024-04-24T08:08:59.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.440+0000 7fb1e2c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb1dc0cf990 con 0x7fb1dc145a90 2024-04-24T08:08:59.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.440+0000 7fb1db400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1dc145a90 0x7fb1dc06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.440+0000 7fb1db400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1dc145a90 0x7fb1dc06e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34294/0 (socket says 172.21.15.69:34294) 2024-04-24T08:08:59.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.440+0000 7fb1db400640 1 -- 172.21.15.69:0/2690041551 learned_addr learned my addr 172.21.15.69:0/2690041551 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:59.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.440+0000 7fb1e0c00640 1 --1- 172.21.15.69:0/2690041551 >> v1:172.21.15.69:6789/0 conn(0x7fb1dc145fe0 0x7fb1dc160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54856/0 (socket says 172.21.15.69:54856) 2024-04-24T08:08:59.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.441+0000 7fb1db400640 1 -- 172.21.15.69:0/2690041551 >> v1:172.21.15.121:6789/0 conn(0x7fb1dc0d0300 legacy=0x7fb1dc0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:59.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.441+0000 7fb1db400640 1 -- 172.21.15.69:0/2690041551 >> v1:172.21.15.69:6789/0 conn(0x7fb1dc145fe0 legacy=0x7fb1dc160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.441+0000 7fb1db400640 1 -- 172.21.15.69:0/2690041551 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1dc0cfc70 con 0x7fb1dc145a90 2024-04-24T08:08:59.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.442+0000 7fb1db400640 1 --2- 172.21.15.69:0/2690041551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145a90 0x7fb1dc06e240 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8009870 tx=0x7fb1c802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785cfe2ce0f443b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:59.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.442+0000 7fb1daa00640 1 -- 172.21.15.69:0/2690041551 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8037070 con 0x7fb1dc145a90 2024-04-24T08:08:59.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.442+0000 7fb1daa00640 1 -- 172.21.15.69:0/2690041551 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1c8032440 con 0x7fb1dc145a90 2024-04-24T08:08:59.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.442+0000 7fb1daa00640 1 -- 172.21.15.69:0/2690041551 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8032810 con 0x7fb1dc145a90 2024-04-24T08:08:59.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.443+0000 7fb1e2c00640 1 -- 172.21.15.69:0/2690041551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145a90 msgr2=0x7fb1dc06e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.443+0000 7fb1e2c00640 1 --2- 172.21.15.69:0/2690041551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145a90 0x7fb1dc06e240 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8009870 tx=0x7fb1c802a2a0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.443+0000 7fb1e2c00640 1 -- 172.21.15.69:0/2690041551 shutdown_connections 2024-04-24T08:08:59.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.443+0000 7fb1e2c00640 1 --2- 172.21.15.69:0/2690041551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145a90 0x7fb1dc06e240 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.443+0000 7fb1e2c00640 1 -- 172.21.15.69:0/2690041551 >> 172.21.15.69:0/2690041551 conn(0x7fb1dc067d90 msgr2=0x7fb1dc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:59.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.443+0000 7fb1e2c00640 1 -- 172.21.15.69:0/2690041551 shutdown_connections 2024-04-24T08:08:59.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.443+0000 7fb1e2c00640 1 -- 172.21.15.69:0/2690041551 wait complete. 2024-04-24T08:08:59.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.445+0000 7fb1e2c00640 1 Processor -- start 2024-04-24T08:08:59.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.445+0000 7fb1e2c00640 1 -- start start 2024-04-24T08:08:59.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.446+0000 7fb1e2c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1dc0d0300 0x7fb1dc1398e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.446+0000 7fb1e2c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1dc145a90 0x7fb1dc15ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.446+0000 7fb1e2c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145fe0 0x7fb1dc13a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.446+0000 7fb1e2c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc1458f0 con 0x7fb1dc145fe0 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.446+0000 7fb1e2c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb1dc145770 con 0x7fb1dc0d0300 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.446+0000 7fb1e2c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb1dc0cdc30 con 0x7fb1dc145a90 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.446+0000 7fb1e0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145fe0 0x7fb1dc13a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.446+0000 7fb1e0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145fe0 0x7fb1dc13a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34298/0 (socket says 172.21.15.69:34298) 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.446+0000 7fb1e0c00640 1 -- 172.21.15.69:0/1276820648 learned_addr learned my addr 172.21.15.69:0/1276820648 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1dbe00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1dc0d0300 0x7fb1dc1398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1e0c00640 1 -- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1dc145a90 msgr2=0x7fb1dc15ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1e0c00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1dc145a90 0x7fb1dc15ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1e0c00640 1 -- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1dc0d0300 msgr2=0x7fb1dc1398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1e0c00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1dc0d0300 0x7fb1dc1398e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1e0c00640 1 -- 172.21.15.69:0/1276820648 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1dc14ea40 con 0x7fb1dc145fe0 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1dbe00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1dc0d0300 0x7fb1dc1398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1e0c00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145fe0 0x7fb1dc13a190 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400c780 tx=0x7fb1d400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:59.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1d3e00640 1 -- 172.21.15.69:0/1276820648 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4017070 con 0x7fb1dc145fe0 2024-04-24T08:08:59.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1d3e00640 1 -- 172.21.15.69:0/1276820648 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1d400a640 con 0x7fb1dc145fe0 2024-04-24T08:08:59.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1d3e00640 1 -- 172.21.15.69:0/1276820648 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1d400aa30 con 0x7fb1dc145fe0 2024-04-24T08:08:59.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.447+0000 7fb1e2c00640 1 -- 172.21.15.69:0/1276820648 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1dc14ecd0 con 0x7fb1dc145fe0 2024-04-24T08:08:59.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.448+0000 7fb1e2c00640 1 -- 172.21.15.69:0/1276820648 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1dc14f290 con 0x7fb1dc145fe0 2024-04-24T08:08:59.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.449+0000 7fb1e2c00640 1 -- 172.21.15.69:0/1276820648 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1dc068a80 con 0x7fb1dc145fe0 2024-04-24T08:08:59.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.450+0000 7fb1d3e00640 1 -- 172.21.15.69:0/1276820648 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb1d401a070 con 0x7fb1dc145fe0 2024-04-24T08:08:59.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.450+0000 7fb1d3e00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1bc06c8f0 0x7fb1bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.451+0000 7fb1d3e00640 1 -- 172.21.15.69:0/1276820648 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb1d408b110 con 0x7fb1dc145fe0 2024-04-24T08:08:59.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.451+0000 7fb1dbe00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1bc06c8f0 0x7fb1bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.452+0000 7fb1dbe00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1bc06c8f0 0x7fb1bc06edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc144eb0 tx=0x7fb1cc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:59.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.455+0000 7fb1d3e00640 1 -- 172.21.15.69:0/1276820648 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb1d4058e50 con 0x7fb1dc145fe0 2024-04-24T08:08:59.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.656+0000 7fb1e2c00640 1 -- 172.21.15.69:0/1276820648 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb1dc0d1120 con 0x7fb1dc145fe0 2024-04-24T08:08:59.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.658+0000 7fb1d3e00640 1 -- 172.21.15.69:0/1276820648 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16581 (secure 0 0 0) 0x7fb1d405d150 con 0x7fb1dc145fe0 2024-04-24T08:08:59.659 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:08:59.659 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":20,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:08:21.105472+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:59.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 -- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1bc06c8f0 msgr2=0x7fb1bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1bc06c8f0 0x7fb1bc06edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc144eb0 tx=0x7fb1cc008040 comp rx=0 tx=0).stop 2024-04-24T08:08:59.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 -- 172.21.15.69:0/1276820648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145fe0 msgr2=0x7fb1dc13a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145fe0 0x7fb1dc13a190 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400c780 tx=0x7fb1d400cc40 comp rx=0 tx=0).stop 2024-04-24T08:08:59.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 -- 172.21.15.69:0/1276820648 shutdown_connections 2024-04-24T08:08:59.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1bc06c8f0 0x7fb1bc06edb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1dc145fe0 0x7fb1dc13a190 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1dc145a90 0x7fb1dc15ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 --2- 172.21.15.69:0/1276820648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1dc0d0300 0x7fb1dc1398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 -- 172.21.15.69:0/1276820648 >> 172.21.15.69:0/1276820648 conn(0x7fb1dc067d90 msgr2=0x7fb1dc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:59.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 -- 172.21.15.69:0/1276820648 shutdown_connections 2024-04-24T08:08:59.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.662+0000 7fb1e2c00640 1 -- 172.21.15.69:0/1276820648 wait complete. 2024-04-24T08:08:59.681 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:08:59.681 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:08:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.883+0000 7f110d000640 1 Processor -- start 2024-04-24T08:08:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.884+0000 7f110d000640 1 -- start start 2024-04-24T08:08:59.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.885+0000 7f110d000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11080b46a0 0x7f11080b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.885+0000 7f110d000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11080739f0 con 0x7f110815f4b0 2024-04-24T08:08:59.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.885+0000 7f110d000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1108073cc0 con 0x7f11080b3de0 2024-04-24T08:08:59.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.885+0000 7f110d000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f11080b5010 con 0x7f11080b46a0 2024-04-24T08:08:59.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.885+0000 7f1105e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f110815f4b0 0x7f110815f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54866/0 (socket says 172.21.15.69:54866) 2024-04-24T08:08:59.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.885+0000 7f1105e00640 1 -- 172.21.15.69:0/1214872147 learned_addr learned my addr 172.21.15.69:0/1214872147 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.886+0000 7f1106800640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11080b46a0 0x7f11080b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.886+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3324874614 0 0) 0x7f11080739f0 con 0x7f110815f4b0 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.886+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10ec003520 con 0x7f110815f4b0 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.886+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1317653063 0 0) 0x7f10ec003520 con 0x7f110815f4b0 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.886+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11080b46a0 msgr2=0x7f11080b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.886+0000 7f1104a00640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11080b46a0 0x7f11080b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.886+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 >> v1:172.21.15.121:6789/0 conn(0x7f11080b3de0 legacy=0x7f11080732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.886+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1108073fa0 con 0x7f110815f4b0 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.886+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f10f8001690 con 0x7f110815f4b0 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.886+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 >> v1:172.21.15.69:6789/0 conn(0x7f110815f4b0 legacy=0x7f110815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1104a00640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ec0039e0 0x7f10ec003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1104a00640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec004800 0x7f10ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1104a00640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11080b46a0 0x7f10ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f11080b5010 con 0x7f10ec0039e0 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec00bad0 con 0x7f10ec004800 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f10ec00bcd0 con 0x7f11080b46a0 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1105e00640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ec0039e0 0x7f10ec003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1106800640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11080b46a0 0x7f10ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1105e00640 1 -- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11080b46a0 msgr2=0x7f10ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1105e00640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11080b46a0 0x7f10ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1105e00640 1 -- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec004800 msgr2=0x7f10ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1105e00640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec004800 0x7f10ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1105e00640 1 -- 172.21.15.69:0/1214872147 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ec003520 con 0x7f10ec0039e0 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.887+0000 7f1106800640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11080b46a0 0x7f10ec0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.888+0000 7f1105e00640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ec0039e0 0x7f10ec003e00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f11080d1450 tx=0x7f10ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efaab69da2cf2a00 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.889+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8007c60 con 0x7f10ec0039e0 2024-04-24T08:08:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.889+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f8005ad0 con 0x7f10ec0039e0 2024-04-24T08:08:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.889+0000 7f1104a00640 1 -- 172.21.15.69:0/1214872147 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8007070 con 0x7f10ec0039e0 2024-04-24T08:08:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.889+0000 7f110d000640 1 -- 172.21.15.69:0/1214872147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ec0039e0 msgr2=0x7f10ec003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.889+0000 7f110d000640 1 --2- 172.21.15.69:0/1214872147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ec0039e0 0x7f10ec003e00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f11080d1450 tx=0x7f10ec005830 comp rx=0 tx=0).stop 2024-04-24T08:08:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.889+0000 7f1106800640 1 -- 172.21.15.69:0/1214872147 reap_dead start 2024-04-24T08:08:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.889+0000 7f110d000640 1 -- 172.21.15.69:0/1214872147 shutdown_connections 2024-04-24T08:08:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.889+0000 7f110d000640 1 -- 172.21.15.69:0/1214872147 >> 172.21.15.69:0/1214872147 conn(0x7f11080d0350 msgr2=0x7f11080d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.890+0000 7f110d000640 1 -- 172.21.15.69:0/1214872147 shutdown_connections 2024-04-24T08:08:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.890+0000 7f110d000640 1 -- 172.21.15.69:0/1214872147 wait complete. 2024-04-24T08:08:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.891+0000 7f110d000640 1 Processor -- start 2024-04-24T08:08:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.891+0000 7f110d000640 1 -- start start 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.892+0000 7f110d000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec0039e0 0x7f1108151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.892+0000 7f110d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1108152560 0x7f11081e8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.892+0000 7f110d000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11081e9cf0 0x7f11081608e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.892+0000 7f110d000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f11080b38c0 con 0x7f1108152560 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.892+0000 7f110d000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f11080b35e0 con 0x7f10ec0039e0 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.892+0000 7f110d000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f11080b4ad0 con 0x7f11081e9cf0 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1108152560 0x7f11081e8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1108152560 0x7f11081e8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34320/0 (socket says 172.21.15.69:34320) 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105400640 1 -- 172.21.15.69:0/3325370108 learned_addr learned my addr 172.21.15.69:0/3325370108 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105e00640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec0039e0 0x7f1108151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105400640 1 -- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11081e9cf0 msgr2=0x7f11081608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105400640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11081e9cf0 0x7f11081608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105400640 1 -- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec0039e0 msgr2=0x7f1108151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105400640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec0039e0 0x7f1108151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105400640 1 -- 172.21.15.69:0/3325370108 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11081e8c50 con 0x7f1108152560 2024-04-24T08:08:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105e00640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec0039e0 0x7f1108151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:08:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.893+0000 7f1105400640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1108152560 0x7f11081e8300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f10f000c480 tx=0x7f10f000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.894+0000 7f10fe000640 1 -- 172.21.15.69:0/3325370108 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f00052d0 con 0x7f1108152560 2024-04-24T08:08:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.894+0000 7f10fe000640 1 -- 172.21.15.69:0/3325370108 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f0005470 con 0x7f1108152560 2024-04-24T08:08:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.894+0000 7f10fe000640 1 -- 172.21.15.69:0/3325370108 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0012850 con 0x7f1108152560 2024-04-24T08:08:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.894+0000 7f110d000640 1 -- 172.21.15.69:0/3325370108 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11081e8f30 con 0x7f1108152560 2024-04-24T08:08:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.894+0000 7f110d000640 1 -- 172.21.15.69:0/3325370108 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11081e94f0 con 0x7f1108152560 2024-04-24T08:08:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.895+0000 7f110d000640 1 -- 172.21.15.69:0/3325370108 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1108068a80 con 0x7f1108152560 2024-04-24T08:08:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.896+0000 7f10fe000640 1 -- 172.21.15.69:0/3325370108 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f10f00129f0 con 0x7f1108152560 2024-04-24T08:08:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.897+0000 7f10fe000640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f10e406c8f0 0x7f10e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.897+0000 7f10fe000640 1 -- 172.21.15.69:0/3325370108 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f10f008b860 con 0x7f1108152560 2024-04-24T08:08:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.898+0000 7f1105e00640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f10e406c8f0 0x7f10e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.899+0000 7f1105e00640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f10e406c8f0 0x7f10e406edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f10f8032000 tx=0x7f10f802d000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:59.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:08:59.902+0000 7f10fe000640 1 -- 172.21.15.69:0/3325370108 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f10f0059c80 con 0x7f1108152560 2024-04-24T08:09:00.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.099+0000 7f110d000640 1 -- 172.21.15.69:0/3325370108 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f11080ad770 con 0x7f1108152560 2024-04-24T08:09:00.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.100+0000 7f10fe000640 1 -- 172.21.15.69:0/3325370108 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f10f0059580 con 0x7f1108152560 2024-04-24T08:09:00.102 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:00.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.102+0000 7f110d000640 1 -- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f10e406c8f0 msgr2=0x7f10e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.102+0000 7f110d000640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f10e406c8f0 0x7f10e406edb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f10f8032000 tx=0x7f10f802d000 comp rx=0 tx=0).stop 2024-04-24T08:09:00.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.102+0000 7f110d000640 1 -- 172.21.15.69:0/3325370108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1108152560 msgr2=0x7f11081e8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.102+0000 7f110d000640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1108152560 0x7f11081e8300 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f10f000c480 tx=0x7f10f000c940 comp rx=0 tx=0).stop 2024-04-24T08:09:00.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.102+0000 7f110d000640 1 -- 172.21.15.69:0/3325370108 shutdown_connections 2024-04-24T08:09:00.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.102+0000 7f110d000640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f10e406c8f0 0x7f10e406edb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.102+0000 7f110d000640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1108152560 0x7f11081e8300 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.102+0000 7f110d000640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11081e9cf0 0x7f11081608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.102+0000 7f110d000640 1 --2- 172.21.15.69:0/3325370108 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ec0039e0 0x7f1108151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.102+0000 7f110d000640 1 -- 172.21.15.69:0/3325370108 >> 172.21.15.69:0/3325370108 conn(0x7f11080d0350 msgr2=0x7f11080afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:00.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.103+0000 7f110d000640 1 -- 172.21.15.69:0/3325370108 shutdown_connections 2024-04-24T08:09:00.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.103+0000 7f110d000640 1 -- 172.21.15.69:0/3325370108 wait complete. 2024-04-24T08:09:00.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:00.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.319+0000 7f7f87200640 1 Processor -- start 2024-04-24T08:09:00.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.319+0000 7f7f87200640 1 -- start start 2024-04-24T08:09:00.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.321+0000 7f7f87200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7f800b3e90 0x7f7f800b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.321+0000 7f7f87200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f800ce250 0x7f7f800ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.321+0000 7f7f87200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f8015cc00 con 0x7f7f8015c820 2024-04-24T08:09:00.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.321+0000 7f7f87200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7f800b4be0 con 0x7f7f800b3e90 2024-04-24T08:09:00.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.321+0000 7f7f87200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7f8006ed30 con 0x7f7f800ce250 2024-04-24T08:09:00.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.321+0000 7f7f77400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7f800b3e90 0x7f7f800b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:00.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.321+0000 7f7f7fe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7f8015c820 0x7f7f8006f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54870/0 (socket says 172.21.15.69:54870) 2024-04-24T08:09:00.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.321+0000 7f7f7fe00640 1 -- 172.21.15.69:0/141880054 learned_addr learned my addr 172.21.15.69:0/141880054 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:00.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.322+0000 7f7f85200640 1 --2- 172.21.15.69:0/141880054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f800ce250 0x7f7f800ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:00.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.322+0000 7f7f77400640 1 -- 172.21.15.69:0/141880054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f800ce250 msgr2=0x7f7f800ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:00.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.322+0000 7f7f77400640 1 --2- 172.21.15.69:0/141880054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f800ce250 0x7f7f800ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.322+0000 7f7f77400640 1 -- 172.21.15.69:0/141880054 >> v1:172.21.15.69:6789/0 conn(0x7f7f8015c820 legacy=0x7f7f8006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.322+0000 7f7f77400640 1 -- 172.21.15.69:0/141880054 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f800b4e10 con 0x7f7f800b3e90 2024-04-24T08:09:00.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.323+0000 7f7f77400640 1 --2- 172.21.15.69:0/141880054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 0x7f7f800b46a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7f680097a0 tx=0x7f7f6802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28dc369aff99c704 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.323+0000 7f7f7f400640 1 -- 172.21.15.69:0/141880054 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68037070 con 0x7f7f800b3e90 2024-04-24T08:09:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.323+0000 7f7f7f400640 1 -- 172.21.15.69:0/141880054 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f68032440 con 0x7f7f800b3e90 2024-04-24T08:09:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.323+0000 7f7f7f400640 1 -- 172.21.15.69:0/141880054 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f68032810 con 0x7f7f800b3e90 2024-04-24T08:09:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.323+0000 7f7f87200640 1 -- 172.21.15.69:0/141880054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 msgr2=0x7f7f800b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.323+0000 7f7f87200640 1 --2- 172.21.15.69:0/141880054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 0x7f7f800b46a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7f680097a0 tx=0x7f7f6802a180 comp rx=0 tx=0).stop 2024-04-24T08:09:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.324+0000 7f7f87200640 1 -- 172.21.15.69:0/141880054 shutdown_connections 2024-04-24T08:09:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.324+0000 7f7f87200640 1 --2- 172.21.15.69:0/141880054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f800ce250 0x7f7f800ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.324+0000 7f7f87200640 1 --2- 172.21.15.69:0/141880054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 0x7f7f800b46a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.324+0000 7f7f87200640 1 -- 172.21.15.69:0/141880054 >> 172.21.15.69:0/141880054 conn(0x7f7f800d0350 msgr2=0x7f7f800d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.324+0000 7f7f87200640 1 -- 172.21.15.69:0/141880054 shutdown_connections 2024-04-24T08:09:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.324+0000 7f7f87200640 1 -- 172.21.15.69:0/141880054 wait complete. 2024-04-24T08:09:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.325+0000 7f7f87200640 1 Processor -- start 2024-04-24T08:09:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.326+0000 7f7f87200640 1 -- start start 2024-04-24T08:09:00.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.326+0000 7f7f87200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 0x7f7f801f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.326+0000 7f7f87200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f800ce250 0x7f7f800ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f87200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f8015c820 0x7f7f800aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f87200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8006f2b0 con 0x7f7f800b3e90 2024-04-24T08:09:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f87200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8006efd0 con 0x7f7f8015c820 2024-04-24T08:09:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f87200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7f8006fee0 con 0x7f7f800ce250 2024-04-24T08:09:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f7fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 0x7f7f801f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f7fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 0x7f7f801f1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34338/0 (socket says 172.21.15.69:34338) 2024-04-24T08:09:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f7fe00640 1 -- 172.21.15.69:0/4197746562 learned_addr learned my addr 172.21.15.69:0/4197746562 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f7fe00640 1 -- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f800ce250 msgr2=0x7f7f800ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f7fe00640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f800ce250 0x7f7f800ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f7fe00640 1 -- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f8015c820 msgr2=0x7f7f800aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f7fe00640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f8015c820 0x7f7f800aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f7fe00640 1 -- 172.21.15.69:0/4197746562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f801e6260 con 0x7f7f800b3e90 2024-04-24T08:09:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.327+0000 7f7f7fe00640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 0x7f7f801f1ed0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000c400 tx=0x7f7f7000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.328+0000 7f7f7cc00640 1 -- 172.21.15.69:0/4197746562 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f70017070 con 0x7f7f800b3e90 2024-04-24T08:09:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.328+0000 7f7f7cc00640 1 -- 172.21.15.69:0/4197746562 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f70005ce0 con 0x7f7f800b3e90 2024-04-24T08:09:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.328+0000 7f7f7cc00640 1 -- 172.21.15.69:0/4197746562 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700072f0 con 0x7f7f800b3e90 2024-04-24T08:09:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.328+0000 7f7f87200640 1 -- 172.21.15.69:0/4197746562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f801e6540 con 0x7f7f800b3e90 2024-04-24T08:09:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.328+0000 7f7f87200640 1 -- 172.21.15.69:0/4197746562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f801f32f0 con 0x7f7f800b3e90 2024-04-24T08:09:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.329+0000 7f7f87200640 1 -- 172.21.15.69:0/4197746562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f80068a80 con 0x7f7f800b3e90 2024-04-24T08:09:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.330+0000 7f7f7cc00640 1 -- 172.21.15.69:0/4197746562 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7f7001a070 con 0x7f7f800b3e90 2024-04-24T08:09:00.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.331+0000 7f7f7cc00640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f5c06c8f0 0x7f7f5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.331+0000 7f7f7cc00640 1 -- 172.21.15.69:0/4197746562 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7f7008afc0 con 0x7f7f800b3e90 2024-04-24T08:09:00.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.332+0000 7f7f77400640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f5c06c8f0 0x7f7f5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:00.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.333+0000 7f7f77400640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f5c06c8f0 0x7f7f5c06edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7f68034000 tx=0x7f7f6802c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:00.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.336+0000 7f7f7cc00640 1 -- 172.21.15.69:0/4197746562 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f70059360 con 0x7f7f800b3e90 2024-04-24T08:09:00.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.545+0000 7f7f87200640 1 -- 172.21.15.69:0/4197746562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7f800cddc0 con 0x7f7f800b3e90 2024-04-24T08:09:00.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.546+0000 7f7f7cc00640 1 -- 172.21.15.69:0/4197746562 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f7f70058c60 con 0x7f7f800b3e90 2024-04-24T08:09:00.546 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 -- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f5c06c8f0 msgr2=0x7f7f5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f5c06c8f0 0x7f7f5c06edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7f68034000 tx=0x7f7f6802c000 comp rx=0 tx=0).stop 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 -- 172.21.15.69:0/4197746562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 msgr2=0x7f7f801f1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 0x7f7f801f1ed0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7f7000c400 tx=0x7f7f7000c8c0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 -- 172.21.15.69:0/4197746562 shutdown_connections 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f5c06c8f0 0x7f7f5c06edb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f8015c820 0x7f7f800aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7f800ce250 0x7f7f800ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 --2- 172.21.15.69:0/4197746562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f800b3e90 0x7f7f801f1ed0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 -- 172.21.15.69:0/4197746562 >> 172.21.15.69:0/4197746562 conn(0x7f7f800d0350 msgr2=0x7f7f800cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:00.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 -- 172.21.15.69:0/4197746562 shutdown_connections 2024-04-24T08:09:00.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.549+0000 7f7f87200640 1 -- 172.21.15.69:0/4197746562 wait complete. 2024-04-24T08:09:00.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:00.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.764+0000 7f49aa400640 1 Processor -- start 2024-04-24T08:09:00.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.764+0000 7f49aa400640 1 -- start start 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.766+0000 7f49aa400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f49a415f4b0 0x7f49a40b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.766+0000 7f49aa400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f49a415f890 0x7f49a40732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.766+0000 7f49aa400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49a4073cc0 con 0x7f49a4073820 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.766+0000 7f49aa400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f49a4073fa0 con 0x7f49a415f890 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.766+0000 7f49aa400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f49a40b3300 con 0x7f49a415f4b0 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.766+0000 7f499aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f49a415f890 0x7f49a40732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.766+0000 7f499aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f49a415f890 0x7f49a40732e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34340/0 (socket says 172.21.15.69:34340) 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.766+0000 7f499aa00640 1 -- 172.21.15.69:0/1420233184 learned_addr learned my addr 172.21.15.69:0/1420233184 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.767+0000 7f499aa00640 1 -- 172.21.15.69:0/1420233184 >> v2:172.21.15.121:3300/0 conn(0x7f49a415f4b0 msgr2=0x7f49a40b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.767+0000 7f499aa00640 1 --2- 172.21.15.69:0/1420233184 >> v2:172.21.15.121:3300/0 conn(0x7f49a415f4b0 0x7f49a40b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.767+0000 7f499aa00640 1 -- 172.21.15.69:0/1420233184 >> v1:172.21.15.121:6789/0 conn(0x7f49a4073820 legacy=0x7f49a40b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:00.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.767+0000 7f499aa00640 1 -- 172.21.15.69:0/1420233184 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a40b35e0 con 0x7f49a415f890 2024-04-24T08:09:00.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.768+0000 7f499aa00640 1 --2- 172.21.15.69:0/1420233184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f890 0x7f49a40732e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f49a40cde60 tx=0x7f498c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b00a6b5a3b7141 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:00.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.768+0000 7f49a2a00640 1 -- 172.21.15.69:0/1420233184 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c037070 con 0x7f49a415f890 2024-04-24T08:09:00.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.768+0000 7f49a2a00640 1 -- 172.21.15.69:0/1420233184 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f498c032440 con 0x7f49a415f890 2024-04-24T08:09:00.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.768+0000 7f49a2a00640 1 -- 172.21.15.69:0/1420233184 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c032810 con 0x7f49a415f890 2024-04-24T08:09:00.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.769+0000 7f49aa400640 1 -- 172.21.15.69:0/1420233184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f890 msgr2=0x7f49a40732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.769+0000 7f49aa400640 1 --2- 172.21.15.69:0/1420233184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f890 0x7f49a40732e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f49a40cde60 tx=0x7f498c02a230 comp rx=0 tx=0).stop 2024-04-24T08:09:00.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.769+0000 7f49aa400640 1 -- 172.21.15.69:0/1420233184 shutdown_connections 2024-04-24T08:09:00.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.769+0000 7f49aa400640 1 --2- 172.21.15.69:0/1420233184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f890 0x7f49a40732e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.769+0000 7f49aa400640 1 --2- 172.21.15.69:0/1420233184 >> v2:172.21.15.121:3300/0 conn(0x7f49a415f4b0 0x7f49a40b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.769+0000 7f49aa400640 1 -- 172.21.15.69:0/1420233184 >> 172.21.15.69:0/1420233184 conn(0x7f49a40d0350 msgr2=0x7f49a40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:00.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.769+0000 7f49aa400640 1 -- 172.21.15.69:0/1420233184 shutdown_connections 2024-04-24T08:09:00.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.769+0000 7f49aa400640 1 -- 172.21.15.69:0/1420233184 wait complete. 2024-04-24T08:09:00.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.771+0000 7f49aa400640 1 Processor -- start 2024-04-24T08:09:00.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.771+0000 7f49aa400640 1 -- start start 2024-04-24T08:09:00.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f49aa400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49a4073820 0x7f49a4149780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f49aa400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f4b0 0x7f49a41e8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f49aa400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49a415f890 0x7f49a40ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f49aa400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f49a415fec0 con 0x7f49a415f4b0 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f49aa400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f49a415fd40 con 0x7f49a415f890 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f49aa400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f49a40b4db0 con 0x7f49a4073820 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f499aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f4b0 0x7f49a41e8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f499aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f4b0 0x7f49a41e8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34356/0 (socket says 172.21.15.69:34356) 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f499aa00640 1 -- 172.21.15.69:0/2417774272 learned_addr learned my addr 172.21.15.69:0/2417774272 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f49a3400640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49a4073820 0x7f49a4149780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.772+0000 7f499aa00640 1 -- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49a4073820 msgr2=0x7f49a4149780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f49a3e00640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49a415f890 0x7f49a40ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f499aa00640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49a4073820 0x7f49a4149780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f499aa00640 1 -- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49a415f890 msgr2=0x7f49a40ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f499aa00640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49a415f890 0x7f49a40ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f499aa00640 1 -- 172.21.15.69:0/2417774272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49a41e9540 con 0x7f49a415f4b0 2024-04-24T08:09:00.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f49a3400640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49a4073820 0x7f49a4149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:00.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f49a3e00640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49a415f890 0x7f49a40ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:00.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f499aa00640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f4b0 0x7f49a41e8d10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f498c02a270 tx=0x7f498c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:00.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f499be00640 1 -- 172.21.15.69:0/2417774272 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c041070 con 0x7f49a415f4b0 2024-04-24T08:09:00.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f499be00640 1 -- 172.21.15.69:0/2417774272 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f498c03c870 con 0x7f49a415f4b0 2024-04-24T08:09:00.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f499be00640 1 -- 172.21.15.69:0/2417774272 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c037040 con 0x7f49a415f4b0 2024-04-24T08:09:00.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f49aa400640 1 -- 172.21.15.69:0/2417774272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49a41e9820 con 0x7f49a415f4b0 2024-04-24T08:09:00.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.773+0000 7f49aa400640 1 -- 172.21.15.69:0/2417774272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49a41e9de0 con 0x7f49a415f4b0 2024-04-24T08:09:00.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.775+0000 7f49aa400640 1 -- 172.21.15.69:0/2417774272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a4068a80 con 0x7f49a415f4b0 2024-04-24T08:09:00.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.775+0000 7f499be00640 1 -- 172.21.15.69:0/2417774272 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f498c044070 con 0x7f49a415f4b0 2024-04-24T08:09:00.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.776+0000 7f499be00640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f498006c920 0x7f498006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:00.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.776+0000 7f499be00640 1 -- 172.21.15.69:0/2417774272 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f498c0ac650 con 0x7f49a415f4b0 2024-04-24T08:09:00.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.777+0000 7f49a3400640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f498006c920 0x7f498006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:00.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.778+0000 7f49a3400640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f498006c920 0x7f498006ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4994007eb0 tx=0x7f49940073c0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:00.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.781+0000 7f499be00640 1 -- 172.21.15.69:0/2417774272 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f498c07ab10 con 0x7f49a415f4b0 2024-04-24T08:09:00.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.964+0000 7f49aa400640 1 -- 172.21.15.69:0/2417774272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f49a40ceeb0 con 0x7f49a415f4b0 2024-04-24T08:09:00.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.965+0000 7f499be00640 1 -- 172.21.15.69:0/2417774272 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f498c07a410 con 0x7f49a415f4b0 2024-04-24T08:09:00.966 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 -- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f498006c920 msgr2=0x7f498006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f498006c920 0x7f498006ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4994007eb0 tx=0x7f49940073c0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 -- 172.21.15.69:0/2417774272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f4b0 msgr2=0x7f49a41e8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f4b0 0x7f49a41e8d10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f498c02a270 tx=0x7f498c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 -- 172.21.15.69:0/2417774272 shutdown_connections 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f498006c920 0x7f498006ede0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49a415f4b0 0x7f49a41e8d10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49a415f890 0x7f49a40ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 --2- 172.21.15.69:0/2417774272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49a4073820 0x7f49a4149780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 -- 172.21.15.69:0/2417774272 >> 172.21.15.69:0/2417774272 conn(0x7f49a40d0350 msgr2=0x7f49a4160550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:00.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 -- 172.21.15.69:0/2417774272 shutdown_connections 2024-04-24T08:09:00.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:00.968+0000 7f49aa400640 1 -- 172.21.15.69:0/2417774272 wait complete. 2024-04-24T08:09:00.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-04-24T08:09:01.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.178+0000 7f5166e00640 1 Processor -- start 2024-04-24T08:09:01.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.178+0000 7f5166e00640 1 -- start start 2024-04-24T08:09:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.180+0000 7f5166e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5160071e50 0x7f516015ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.180+0000 7f5166e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5160144db0 con 0x7f5160162300 2024-04-24T08:09:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.180+0000 7f5166e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f516015f540 con 0x7f5160072230 2024-04-24T08:09:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.180+0000 7f5166e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f516015f820 con 0x7f5160071e50 2024-04-24T08:09:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.180+0000 7f5165000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5160162300 0x7f516006e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54872/0 (socket says 172.21.15.69:54872) 2024-04-24T08:09:01.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.180+0000 7f5165000640 1 -- 172.21.15.69:0/1627438060 learned_addr learned my addr 172.21.15.69:0/1627438060 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.180+0000 7f5165a00640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5160071e50 0x7f516015ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 446518080 0 0) 0x7f5160144db0 con 0x7f5160162300 2024-04-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5148003520 con 0x7f5160162300 2024-04-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1576003775 0 0) 0x7f5148003520 con 0x7f5160162300 2024-04-24T08:09:01.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5160071e50 msgr2=0x7f516015ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5160071e50 0x7f516015ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 >> v1:172.21.15.121:6789/0 conn(0x7f5160072230 legacy=0x7f51601446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f516015fb00 con 0x7f5160162300 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f514c001670 con 0x7f5160162300 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 >> v1:172.21.15.69:6789/0 conn(0x7f5160162300 legacy=0x7f516006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51480039e0 0x7f5148003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 0x7f5148004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5148005210 0x7f514800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f516015f820 con 0x7f5148004800 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f514800c0c0 con 0x7f51480039e0 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f514800c2c0 con 0x7f5148005210 2024-04-24T08:09:01.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f5165a00640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5160071e50 0x7f516015ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.181+0000 7f5165000640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 0x7f5148004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.182+0000 7f5165a00640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51480039e0 0x7f5148003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.182+0000 7f5165000640 1 -- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5148005210 msgr2=0x7f514800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.182+0000 7f5165000640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5148005210 0x7f514800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.182+0000 7f5165000640 1 -- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51480039e0 msgr2=0x7f5148003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.182+0000 7f5165000640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51480039e0 0x7f5148003e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.182+0000 7f5165000640 1 -- 172.21.15.69:0/1627438060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5148003520 con 0x7f5148004800 2024-04-24T08:09:01.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.183+0000 7f5165000640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 0x7f5148004c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f514c00c110 tx=0x7f5148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e720353b4a905344 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:01.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.183+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c007c40 con 0x7f5148004800 2024-04-24T08:09:01.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.183+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f514c005af0 con 0x7f5148004800 2024-04-24T08:09:01.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.183+0000 7f515fe00640 1 -- 172.21.15.69:0/1627438060 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514c007070 con 0x7f5148004800 2024-04-24T08:09:01.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.184+0000 7f5166e00640 1 -- 172.21.15.69:0/1627438060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 msgr2=0x7f5148004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:01.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.184+0000 7f5166e00640 1 --2- 172.21.15.69:0/1627438060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 0x7f5148004c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f514c00c110 tx=0x7f5148005830 comp rx=0 tx=0).stop 2024-04-24T08:09:01.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.184+0000 7f5166400640 1 -- 172.21.15.69:0/1627438060 reap_dead start 2024-04-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.184+0000 7f5166e00640 1 -- 172.21.15.69:0/1627438060 shutdown_connections 2024-04-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.184+0000 7f5166e00640 1 -- 172.21.15.69:0/1627438060 >> 172.21.15.69:0/1627438060 conn(0x7f51600d0160 msgr2=0x7f51600d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.184+0000 7f5166e00640 1 -- 172.21.15.69:0/1627438060 shutdown_connections 2024-04-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.184+0000 7f5166e00640 1 -- 172.21.15.69:0/1627438060 wait complete. 2024-04-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.185+0000 7f5166e00640 1 Processor -- start 2024-04-24T08:09:01.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.186+0000 7f5166e00640 1 -- start start 2024-04-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5166e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 0x7f516015bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5166e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5160098950 0x7f51601539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5166e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5160160d90 0x7f5160161210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5166e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f516015ee40 con 0x7f5148004800 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5166e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f516015ecc0 con 0x7f5160098950 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5166e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f516015efc0 con 0x7f5160160d90 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5165a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 0x7f516015bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5165a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 0x7f516015bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34388/0 (socket says 172.21.15.69:34388) 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5165a00640 1 -- 172.21.15.69:0/4204660724 learned_addr learned my addr 172.21.15.69:0/4204660724 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5165a00640 1 -- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5160160d90 msgr2=0x7f5160161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5165a00640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5160160d90 0x7f5160161210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5165a00640 1 -- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5160098950 msgr2=0x7f51601539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5165a00640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5160098950 0x7f51601539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.187+0000 7f5165a00640 1 -- 172.21.15.69:0/4204660724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5160099130 con 0x7f5148004800 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.188+0000 7f5165a00640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 0x7f516015bf90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f515400f4c0 tx=0x7f515400f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.188+0000 7f515d600640 1 -- 172.21.15.69:0/4204660724 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5154018020 con 0x7f5148004800 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.188+0000 7f515d600640 1 -- 172.21.15.69:0/4204660724 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5154014890 con 0x7f5148004800 2024-04-24T08:09:01.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.188+0000 7f515d600640 1 -- 172.21.15.69:0/4204660724 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5154014c60 con 0x7f5148004800 2024-04-24T08:09:01.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.188+0000 7f5166e00640 1 -- 172.21.15.69:0/4204660724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5160099410 con 0x7f5148004800 2024-04-24T08:09:01.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.188+0000 7f5166e00640 1 -- 172.21.15.69:0/4204660724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5160095570 con 0x7f5148004800 2024-04-24T08:09:01.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.189+0000 7f5166e00640 1 -- 172.21.15.69:0/4204660724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5160075d50 con 0x7f5148004800 2024-04-24T08:09:01.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.190+0000 7f515d600640 1 -- 172.21.15.69:0/4204660724 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f515401b020 con 0x7f5148004800 2024-04-24T08:09:01.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.191+0000 7f515d600640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f514006c8f0 0x7f514006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.191+0000 7f515d600640 1 -- 172.21.15.69:0/4204660724 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f515408c0e0 con 0x7f5148004800 2024-04-24T08:09:01.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.192+0000 7f5165000640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f514006c8f0 0x7f514006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:01.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.193+0000 7f5165000640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f514006c8f0 0x7f514006edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f514c032000 tx=0x7f514c007000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:01.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.196+0000 7f515d600640 1 -- 172.21.15.69:0/4204660724 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5154022020 con 0x7f5148004800 2024-04-24T08:09:01.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:01.485+0000 7f5166e00640 1 -- 172.21.15.69:0/4204660724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) -- 0x7f51600d0d90 con 0x7f5148004800 2024-04-24T08:09:02.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.394+0000 7f515d600640 1 -- 172.21.15.69:0/4204660724 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v61) ==== 145+0+0 (secure 0 0 0) 0x7f5154024300 con 0x7f5148004800 2024-04-24T08:09:02.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.397+0000 7f5166e00640 1 -- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f514006c8f0 msgr2=0x7f514006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.397+0000 7f5166e00640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f514006c8f0 0x7f514006edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f514c032000 tx=0x7f514c007000 comp rx=0 tx=0).stop 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.397+0000 7f5166e00640 1 -- 172.21.15.69:0/4204660724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 msgr2=0x7f516015bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.397+0000 7f5166e00640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 0x7f516015bf90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f515400f4c0 tx=0x7f515400f980 comp rx=0 tx=0).stop 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.398+0000 7f5166e00640 1 -- 172.21.15.69:0/4204660724 shutdown_connections 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.398+0000 7f5166e00640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f514006c8f0 0x7f514006edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.398+0000 7f5166e00640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5160160d90 0x7f5160161210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.398+0000 7f5166e00640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5160098950 0x7f51601539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.398+0000 7f5166e00640 1 --2- 172.21.15.69:0/4204660724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5148004800 0x7f516015bf90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.398+0000 7f5166e00640 1 -- 172.21.15.69:0/4204660724 >> 172.21.15.69:0/4204660724 conn(0x7f51600d0160 msgr2=0x7f51600b0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.398+0000 7f5166e00640 1 -- 172.21.15.69:0/4204660724 shutdown_connections 2024-04-24T08:09:02.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.398+0000 7f5166e00640 1 -- 172.21.15.69:0/4204660724 wait complete. 2024-04-24T08:09:02.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-24T08:09:02.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.619+0000 7f97f3400640 1 Processor -- start 2024-04-24T08:09:02.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.619+0000 7f97f3400640 1 -- start start 2024-04-24T08:09:02.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.621+0000 7f97f3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f97ec0cf360 0x7f97ec0cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:02.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.621+0000 7f97f3400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97ec0cd6d0 con 0x7f97ec155f20 2024-04-24T08:09:02.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.621+0000 7f97f3400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97ec0cd9b0 con 0x7f97ec15c9b0 2024-04-24T08:09:02.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.621+0000 7f97f3400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f97ec06db90 con 0x7f97ec0cf360 2024-04-24T08:09:02.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.621+0000 7f97f0a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f97ec155f20 0x7f97ec0ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54888/0 (socket says 172.21.15.69:54888) 2024-04-24T08:09:02.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.621+0000 7f97f0a00640 1 -- 172.21.15.69:0/3775088965 learned_addr learned my addr 172.21.15.69:0/3775088965 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:02.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.621+0000 7f97f1400640 1 --2- 172.21.15.69:0/3775088965 >> v2:172.21.15.69:3300/0 conn(0x7f97ec0cf360 0x7f97ec0cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:02.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.622+0000 7f97f1400640 1 -- 172.21.15.69:0/3775088965 >> v1:172.21.15.121:6789/0 conn(0x7f97ec15c9b0 legacy=0x7f97ec0b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:02.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.622+0000 7f97f1400640 1 -- 172.21.15.69:0/3775088965 >> v1:172.21.15.69:6789/0 conn(0x7f97ec155f20 legacy=0x7f97ec0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:02.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.622+0000 7f97f1400640 1 -- 172.21.15.69:0/3775088965 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec0cdc90 con 0x7f97ec0cf360 2024-04-24T08:09:02.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.623+0000 7f97f1400640 1 --2- 172.21.15.69:0/3775088965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 0x7f97ec0cf810 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97e40097a0 tx=0x7f97e402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be762e0787ba36df server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:02.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.623+0000 7f97eb400640 1 -- 172.21.15.69:0/3775088965 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4037070 con 0x7f97ec0cf360 2024-04-24T08:09:02.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.623+0000 7f97eb400640 1 -- 172.21.15.69:0/3775088965 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e4032440 con 0x7f97ec0cf360 2024-04-24T08:09:02.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.623+0000 7f97eb400640 1 -- 172.21.15.69:0/3775088965 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4032810 con 0x7f97ec0cf360 2024-04-24T08:09:02.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.624+0000 7f97f3400640 1 -- 172.21.15.69:0/3775088965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 msgr2=0x7f97ec0cf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:02.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.624+0000 7f97f3400640 1 --2- 172.21.15.69:0/3775088965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 0x7f97ec0cf810 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f97e40097a0 tx=0x7f97e402a270 comp rx=0 tx=0).stop 2024-04-24T08:09:02.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.624+0000 7f97f3400640 1 -- 172.21.15.69:0/3775088965 shutdown_connections 2024-04-24T08:09:02.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.624+0000 7f97f3400640 1 --2- 172.21.15.69:0/3775088965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 0x7f97ec0cf810 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.624+0000 7f97f3400640 1 -- 172.21.15.69:0/3775088965 >> 172.21.15.69:0/3775088965 conn(0x7f97ec0d0350 msgr2=0x7f97ec0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:02.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.624+0000 7f97f3400640 1 -- 172.21.15.69:0/3775088965 shutdown_connections 2024-04-24T08:09:02.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.624+0000 7f97f3400640 1 -- 172.21.15.69:0/3775088965 wait complete. 2024-04-24T08:09:02.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.626+0000 7f97f3400640 1 Processor -- start 2024-04-24T08:09:02.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.626+0000 7f97f3400640 1 -- start start 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 0x7f97ec0aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f3400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97ec155f20 0x7f97ec1f8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f3400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97ec15c9b0 0x7f97ec15e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f3400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec0cd470 con 0x7f97ec0cf360 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f3400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec072070 con 0x7f97ec155f20 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f3400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f97ec0b4140 con 0x7f97ec15c9b0 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 0x7f97ec0aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 0x7f97ec0aeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34400/0 (socket says 172.21.15.69:34400) 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f0a00640 1 -- 172.21.15.69:0/1160806849 learned_addr learned my addr 172.21.15.69:0/1160806849 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f0a00640 1 -- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97ec15c9b0 msgr2=0x7f97ec15e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f0a00640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97ec15c9b0 0x7f97ec15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f0a00640 1 -- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97ec155f20 msgr2=0x7f97ec1f8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f0a00640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97ec155f20 0x7f97ec1f8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.627+0000 7f97f0a00640 1 -- 172.21.15.69:0/1160806849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec159a40 con 0x7f97ec0cf360 2024-04-24T08:09:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.628+0000 7f97f0a00640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 0x7f97ec0aeee0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c400 tx=0x7f97d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:02.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.628+0000 7f97e8c00640 1 -- 172.21.15.69:0/1160806849 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4017070 con 0x7f97ec0cf360 2024-04-24T08:09:02.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.628+0000 7f97e8c00640 1 -- 172.21.15.69:0/1160806849 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97d4005ce0 con 0x7f97ec0cf360 2024-04-24T08:09:02.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.628+0000 7f97e8c00640 1 -- 172.21.15.69:0/1160806849 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d40072f0 con 0x7f97ec0cf360 2024-04-24T08:09:02.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.628+0000 7f97f3400640 1 -- 172.21.15.69:0/1160806849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ec159d20 con 0x7f97ec0cf360 2024-04-24T08:09:02.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.628+0000 7f97f3400640 1 -- 172.21.15.69:0/1160806849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ec1f8240 con 0x7f97ec0cf360 2024-04-24T08:09:02.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.629+0000 7f97f3400640 1 -- 172.21.15.69:0/1160806849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97ec068a80 con 0x7f97ec0cf360 2024-04-24T08:09:02.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.630+0000 7f97e8c00640 1 -- 172.21.15.69:0/1160806849 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f97d401a070 con 0x7f97ec0cf360 2024-04-24T08:09:02.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.631+0000 7f97e8c00640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f97c806c8f0 0x7f97c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:02.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.631+0000 7f97e8c00640 1 -- 172.21.15.69:0/1160806849 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f97d408b060 con 0x7f97ec0cf360 2024-04-24T08:09:02.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.632+0000 7f97ebe00640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f97c806c8f0 0x7f97c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:02.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.633+0000 7f97ebe00640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f97c806c8f0 0x7f97c806edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f97ec0b1550 tx=0x7f97dc009250 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:02.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.636+0000 7f97e8c00640 1 -- 172.21.15.69:0/1160806849 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97d4059360 con 0x7f97ec0cf360 2024-04-24T08:09:02.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.836+0000 7f97f3400640 1 -- 172.21.15.69:0/1160806849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f97ec06ddc0 con 0x7f97ec0cf360 2024-04-24T08:09:02.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.842+0000 7f97e8c00640 1 -- 172.21.15.69:0/1160806849 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f97d4058c60 con 0x7f97ec0cf360 2024-04-24T08:09:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.844+0000 7f97f3400640 1 -- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f97c806c8f0 msgr2=0x7f97c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.844+0000 7f97f3400640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f97c806c8f0 0x7f97c806edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f97ec0b1550 tx=0x7f97dc009250 comp rx=0 tx=0).stop 2024-04-24T08:09:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.844+0000 7f97f3400640 1 -- 172.21.15.69:0/1160806849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 msgr2=0x7f97ec0aeee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.844+0000 7f97f3400640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 0x7f97ec0aeee0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c400 tx=0x7f97d400c8c0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.845+0000 7f97f3400640 1 -- 172.21.15.69:0/1160806849 shutdown_connections 2024-04-24T08:09:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.845+0000 7f97f3400640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f97c806c8f0 0x7f97c806edb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.845+0000 7f97f3400640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97ec15c9b0 0x7f97ec15e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.845+0000 7f97f3400640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f97ec155f20 0x7f97ec1f8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.845+0000 7f97f3400640 1 --2- 172.21.15.69:0/1160806849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97ec0cf360 0x7f97ec0aeee0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.845+0000 7f97f3400640 1 -- 172.21.15.69:0/1160806849 >> 172.21.15.69:0/1160806849 conn(0x7f97ec0d0350 msgr2=0x7f97ec0b4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.845+0000 7f97f3400640 1 -- 172.21.15.69:0/1160806849 shutdown_connections 2024-04-24T08:09:02.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:02.845+0000 7f97f3400640 1 -- 172.21.15.69:0/1160806849 wait complete. 2024-04-24T08:09:02.866 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:09:02.866 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:09:02.967 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:09:03.072 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:09:03.073 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:09:03.073 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:03.131 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:09:03.131 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:03.178 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:03.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.415+0000 7f7073e00640 1 Processor -- start 2024-04-24T08:09:03.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.416+0000 7f7073e00640 1 -- start start 2024-04-24T08:09:03.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.417+0000 7f7073e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 0x7f7074162250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.417+0000 7f7073e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f707415f320 0x7f707415f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.417+0000 7f7073e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7074070860 con 0x7f70740cf9a0 2024-04-24T08:09:03.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.417+0000 7f7073e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f707415ee90 con 0x7f707415f320 2024-04-24T08:09:03.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.417+0000 7f7073e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7074163180 con 0x7f70740710e0 2024-04-24T08:09:03.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.418+0000 7f7073400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f707415f320 0x7f707415f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.418+0000 7f7073400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f707415f320 0x7f707415f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:33230/0 (socket says 172.21.15.69:33230) 2024-04-24T08:09:03.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.418+0000 7f7072a00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f70740cf9a0 0x7f7074072af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:33986/0 (socket says 172.21.15.69:33986) 2024-04-24T08:09:03.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.418+0000 7f7073400640 1 -- 172.21.15.69:0/4081881233 learned_addr learned my addr 172.21.15.69:0/4081881233 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:03.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.418+0000 7f706be00640 1 --2- 172.21.15.69:0/4081881233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 0x7f7074162250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.419+0000 7f706be00640 1 -- 172.21.15.69:0/4081881233 >> v2:172.21.15.121:3300/0 conn(0x7f707415f320 msgr2=0x7f707415f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:03.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.419+0000 7f706be00640 1 --2- 172.21.15.69:0/4081881233 >> v2:172.21.15.121:3300/0 conn(0x7f707415f320 0x7f707415f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.419+0000 7f706be00640 1 -- 172.21.15.69:0/4081881233 >> v1:172.21.15.121:6789/0 conn(0x7f70740cf9a0 legacy=0x7f7074072af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.419+0000 7f706be00640 1 -- 172.21.15.69:0/4081881233 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70741628e0 con 0x7f70740710e0 2024-04-24T08:09:03.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.420+0000 7f706be00640 1 --2- 172.21.15.69:0/4081881233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 0x7f7074162250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f705c0097a0 tx=0x7f705c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c321b3c4a6bedcf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:03.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.420+0000 7f7072000640 1 -- 172.21.15.69:0/4081881233 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c037070 con 0x7f70740710e0 2024-04-24T08:09:03.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.421+0000 7f7072000640 1 -- 172.21.15.69:0/4081881233 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f705c032440 con 0x7f70740710e0 2024-04-24T08:09:03.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.421+0000 7f7072000640 1 -- 172.21.15.69:0/4081881233 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c032810 con 0x7f70740710e0 2024-04-24T08:09:03.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.421+0000 7f7073e00640 1 -- 172.21.15.69:0/4081881233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 msgr2=0x7f7074162250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.421+0000 7f7073e00640 1 --2- 172.21.15.69:0/4081881233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 0x7f7074162250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f705c0097a0 tx=0x7f705c02a180 comp rx=0 tx=0).stop 2024-04-24T08:09:03.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.421+0000 7f7073e00640 1 -- 172.21.15.69:0/4081881233 shutdown_connections 2024-04-24T08:09:03.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.421+0000 7f7073e00640 1 --2- 172.21.15.69:0/4081881233 >> v2:172.21.15.121:3300/0 conn(0x7f707415f320 0x7f707415f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.421+0000 7f7073e00640 1 --2- 172.21.15.69:0/4081881233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 0x7f7074162250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.421+0000 7f7073e00640 1 -- 172.21.15.69:0/4081881233 >> 172.21.15.69:0/4081881233 conn(0x7f70740d12c0 msgr2=0x7f70740d16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:03.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.421+0000 7f7073e00640 1 -- 172.21.15.69:0/4081881233 shutdown_connections 2024-04-24T08:09:03.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.421+0000 7f7073e00640 1 -- 172.21.15.69:0/4081881233 wait complete. 2024-04-24T08:09:03.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.423+0000 7f7073e00640 1 Processor -- start 2024-04-24T08:09:03.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.423+0000 7f7073e00640 1 -- start start 2024-04-24T08:09:03.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.424+0000 7f7073e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 0x7f7074152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.424+0000 7f7073e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70740cf9a0 0x7f70740ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.424+0000 7f7073e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f707415f320 0x7f70741579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.424+0000 7f7073e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7074073200 con 0x7f70740cf9a0 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.424+0000 7f7073e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7074162ea0 con 0x7f707415f320 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.424+0000 7f7073e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f70740af750 con 0x7f70740710e0 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f706be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70740cf9a0 0x7f70740ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f706be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70740cf9a0 0x7f70740ae260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34414/0 (socket says 172.21.15.69:34414) 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f706be00640 1 -- 172.21.15.69:0/2902726416 learned_addr learned my addr 172.21.15.69:0/2902726416 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f7072a00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 0x7f7074152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f7073400640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f707415f320 0x7f70741579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f706be00640 1 -- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 msgr2=0x7f7074152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f706be00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 0x7f7074152a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f706be00640 1 -- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f707415f320 msgr2=0x7f70741579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f706be00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f707415f320 0x7f70741579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f706be00640 1 -- 172.21.15.69:0/2902726416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7074152f60 con 0x7f70740cf9a0 2024-04-24T08:09:03.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f7073400640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f707415f320 0x7f70741579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:03.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.425+0000 7f706be00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70740cf9a0 0x7f70740ae260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f705c000c00 tx=0x7f705c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:03.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.426+0000 7f706aa00640 1 -- 172.21.15.69:0/2902726416 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c041070 con 0x7f70740cf9a0 2024-04-24T08:09:03.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.426+0000 7f706aa00640 1 -- 172.21.15.69:0/2902726416 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f705c03c870 con 0x7f70740cf9a0 2024-04-24T08:09:03.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.426+0000 7f706aa00640 1 -- 172.21.15.69:0/2902726416 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c037040 con 0x7f70740cf9a0 2024-04-24T08:09:03.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.426+0000 7f7073e00640 1 -- 172.21.15.69:0/2902726416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70740aebd0 con 0x7f70740cf9a0 2024-04-24T08:09:03.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.426+0000 7f7073e00640 1 -- 172.21.15.69:0/2902726416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70740af190 con 0x7f70740cf9a0 2024-04-24T08:09:03.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.427+0000 7f7073e00640 1 -- 172.21.15.69:0/2902726416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7074075d50 con 0x7f70740cf9a0 2024-04-24T08:09:03.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.428+0000 7f706aa00640 1 -- 172.21.15.69:0/2902726416 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f705c044070 con 0x7f70740cf9a0 2024-04-24T08:09:03.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.429+0000 7f706aa00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 0x7f705006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.429+0000 7f706aa00640 1 -- 172.21.15.69:0/2902726416 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f705c0ac7e0 con 0x7f70740cf9a0 2024-04-24T08:09:03.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.430+0000 7f7072a00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 0x7f705006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.431+0000 7f7072a00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 0x7f705006edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7074072cb0 tx=0x7f7064007ce0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:03.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.433+0000 7f706aa00640 1 -- 172.21.15.69:0/2902726416 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f705c07ac00 con 0x7f70740cf9a0 2024-04-24T08:09:03.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.634+0000 7f7073e00640 1 -- 172.21.15.69:0/2902726416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70740703a0 con 0x7f70740cf9a0 2024-04-24T08:09:03.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.637+0000 7f706aa00640 1 -- 172.21.15.69:0/2902726416 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7f705c07a500 con 0x7f70740cf9a0 2024-04-24T08:09:03.638 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:03.638 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:03.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 -- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 msgr2=0x7f705006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 0x7f705006edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7074072cb0 tx=0x7f7064007ce0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 -- 172.21.15.69:0/2902726416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70740cf9a0 msgr2=0x7f70740ae260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70740cf9a0 0x7f70740ae260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f705c000c00 tx=0x7f705c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 -- 172.21.15.69:0/2902726416 shutdown_connections 2024-04-24T08:09:03.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 0x7f705006edb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70740cf9a0 0x7f70740ae260 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f707415f320 0x7f70741579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 --2- 172.21.15.69:0/2902726416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70740710e0 0x7f7074152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 -- 172.21.15.69:0/2902726416 >> 172.21.15.69:0/2902726416 conn(0x7f70740d12c0 msgr2=0x7f70740b0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:03.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 -- 172.21.15.69:0/2902726416 shutdown_connections 2024-04-24T08:09:03.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.640+0000 7f7073e00640 1 -- 172.21.15.69:0/2902726416 wait complete. 2024-04-24T08:09:03.659 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:03.659 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:03.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.861+0000 7fbbcb200640 1 Processor -- start 2024-04-24T08:09:03.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.861+0000 7fbbcb200640 1 -- start start 2024-04-24T08:09:03.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.863+0000 7fbbcb200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbbcc1673e0 0x7fbbcc1697d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.863+0000 7fbbcb200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbcc15fbb0 con 0x7fbbcc15edf0 2024-04-24T08:09:03.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.863+0000 7fbbcb200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbcc0d00d0 con 0x7fbbcc071d70 2024-04-24T08:09:03.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.863+0000 7fbbcb200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbbcc0d0a00 con 0x7fbbcc1673e0 2024-04-24T08:09:03.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.863+0000 7fbbc1400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbbcc15edf0 0x7fbbcc15f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54894/0 (socket says 172.21.15.69:54894) 2024-04-24T08:09:03.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.863+0000 7fbbc1400640 1 -- 172.21.15.69:0/3736781679 learned_addr learned my addr 172.21.15.69:0/3736781679 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:03.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.863+0000 7fbbca800640 1 --2- 172.21.15.69:0/3736781679 >> v2:172.21.15.69:3300/0 conn(0x7fbbcc1673e0 0x7fbbcc1697d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.864+0000 7fbbc9400640 1 -- 172.21.15.69:0/3736781679 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2146393087 0 0) 0x7fbbcc15fbb0 con 0x7fbbcc15edf0 2024-04-24T08:09:03.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.864+0000 7fbbc9400640 1 -- 172.21.15.69:0/3736781679 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb0003520 con 0x7fbbcc15edf0 2024-04-24T08:09:03.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.864+0000 7fbbca800640 1 -- 172.21.15.69:0/3736781679 >> v1:172.21.15.121:6789/0 conn(0x7fbbcc071d70 legacy=0x7fbbcc072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.864+0000 7fbbca800640 1 -- 172.21.15.69:0/3736781679 >> v1:172.21.15.69:6789/0 conn(0x7fbbcc15edf0 legacy=0x7fbbcc15f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.864+0000 7fbbca800640 1 -- 172.21.15.69:0/3736781679 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbcc0d03b0 con 0x7fbbcc1673e0 2024-04-24T08:09:03.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.865+0000 7fbbca800640 1 --2- 172.21.15.69:0/3736781679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc1673e0 0x7fbbcc1697d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4009d90 tx=0x7fbbc402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c65cb9eb49f45ae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:03.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.865+0000 7fbbc9400640 1 -- 172.21.15.69:0/3736781679 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4007740 con 0x7fbbcc1673e0 2024-04-24T08:09:03.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.865+0000 7fbbc9400640 1 -- 172.21.15.69:0/3736781679 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbc40078e0 con 0x7fbbcc1673e0 2024-04-24T08:09:03.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.865+0000 7fbbc9400640 1 -- 172.21.15.69:0/3736781679 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4007cb0 con 0x7fbbcc1673e0 2024-04-24T08:09:03.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.866+0000 7fbbcb200640 1 -- 172.21.15.69:0/3736781679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc1673e0 msgr2=0x7fbbcc1697d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.866+0000 7fbbcb200640 1 --2- 172.21.15.69:0/3736781679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc1673e0 0x7fbbcc1697d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4009d90 tx=0x7fbbc402a6f0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.866+0000 7fbbcb200640 1 -- 172.21.15.69:0/3736781679 shutdown_connections 2024-04-24T08:09:03.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.866+0000 7fbbcb200640 1 --2- 172.21.15.69:0/3736781679 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc1673e0 0x7fbbcc1697d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.866+0000 7fbbcb200640 1 -- 172.21.15.69:0/3736781679 >> 172.21.15.69:0/3736781679 conn(0x7fbbcc067b80 msgr2=0x7fbbcc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:03.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.866+0000 7fbbcb200640 1 -- 172.21.15.69:0/3736781679 shutdown_connections 2024-04-24T08:09:03.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.866+0000 7fbbcb200640 1 -- 172.21.15.69:0/3736781679 wait complete. 2024-04-24T08:09:03.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.868+0000 7fbbcb200640 1 Processor -- start 2024-04-24T08:09:03.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.868+0000 7fbbcb200640 1 -- start start 2024-04-24T08:09:03.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.869+0000 7fbbcb200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbcc071d70 0x7fbbcc15bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.869+0000 7fbbcb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc15edf0 0x7fbbcc1664b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.869+0000 7fbbcb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbcc1673e0 0x7fbbcc163f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.869+0000 7fbbcb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc06f010 con 0x7fbbcc15edf0 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.869+0000 7fbbcb200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbbcc15f960 con 0x7fbbcc071d70 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.869+0000 7fbbcb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbbcc0d0e90 con 0x7fbbcc1673e0 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.869+0000 7fbbc1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc15edf0 0x7fbbcc1664b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.869+0000 7fbbc1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc15edf0 0x7fbbcc1664b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34422/0 (socket says 172.21.15.69:34422) 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.869+0000 7fbbc1400640 1 -- 172.21.15.69:0/2671587722 learned_addr learned my addr 172.21.15.69:0/2671587722 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbc1400640 1 -- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbcc1673e0 msgr2=0x7fbbcc163f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbc1400640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbcc1673e0 0x7fbbcc163f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbc9e00640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbcc071d70 0x7fbbcc15bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbc1400640 1 -- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbcc071d70 msgr2=0x7fbbcc15bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbc1400640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbcc071d70 0x7fbbcc15bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbc1400640 1 -- 172.21.15.69:0/2671587722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbcc164480 con 0x7fbbcc15edf0 2024-04-24T08:09:03.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbc1400640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc15edf0 0x7fbbcc1664b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400c500 tx=0x7fbbb400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:03.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbc2a00640 1 -- 172.21.15.69:0/2671587722 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4017070 con 0x7fbbcc15edf0 2024-04-24T08:09:03.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbc2a00640 1 -- 172.21.15.69:0/2671587722 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb4005ce0 con 0x7fbbcc15edf0 2024-04-24T08:09:03.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbc2a00640 1 -- 172.21.15.69:0/2671587722 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4007330 con 0x7fbbcc15edf0 2024-04-24T08:09:03.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbcb200640 1 -- 172.21.15.69:0/2671587722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbcc166b80 con 0x7fbbcc15edf0 2024-04-24T08:09:03.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.870+0000 7fbbcb200640 1 -- 172.21.15.69:0/2671587722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbcc167140 con 0x7fbbcc15edf0 2024-04-24T08:09:03.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.871+0000 7fbbcb200640 1 -- 172.21.15.69:0/2671587722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbcc075d50 con 0x7fbbcc15edf0 2024-04-24T08:09:03.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.872+0000 7fbbc2a00640 1 -- 172.21.15.69:0/2671587722 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbbb401a070 con 0x7fbbcc15edf0 2024-04-24T08:09:03.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.874+0000 7fbbc2a00640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbba806c8f0 0x7fbba806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.874+0000 7fbbc2a00640 1 -- 172.21.15.69:0/2671587722 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fbbb408aff0 con 0x7fbbcc15edf0 2024-04-24T08:09:03.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.874+0000 7fbbc9e00640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbba806c8f0 0x7fbba806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.876+0000 7fbbc9e00640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbba806c8f0 0x7fbba806edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0015a0 tx=0x7fbbbc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:03.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:03.879+0000 7fbbc2a00640 1 -- 172.21.15.69:0/2671587722 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbbb40594a0 con 0x7fbbcc15edf0 2024-04-24T08:09:04.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.076+0000 7fbbcb200640 1 -- 172.21.15.69:0/2671587722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbbcc141f30 con 0x7fbbcc15edf0 2024-04-24T08:09:04.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.077+0000 7fbbc2a00640 1 -- 172.21.15.69:0/2671587722 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fbbb4058da0 con 0x7fbbcc15edf0 2024-04-24T08:09:04.078 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:04.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 -- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbba806c8f0 msgr2=0x7fbba806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbba806c8f0 0x7fbba806edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc0015a0 tx=0x7fbbbc009040 comp rx=0 tx=0).stop 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 -- 172.21.15.69:0/2671587722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc15edf0 msgr2=0x7fbbcc1664b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc15edf0 0x7fbbcc1664b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400c500 tx=0x7fbbb400c9c0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 -- 172.21.15.69:0/2671587722 shutdown_connections 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbba806c8f0 0x7fbba806edb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbcc15edf0 0x7fbbcc1664b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbcc1673e0 0x7fbbcc163f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 --2- 172.21.15.69:0/2671587722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbcc071d70 0x7fbbcc15bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 -- 172.21.15.69:0/2671587722 >> 172.21.15.69:0/2671587722 conn(0x7fbbcc067b80 msgr2=0x7fbbcc140ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 -- 172.21.15.69:0/2671587722 shutdown_connections 2024-04-24T08:09:04.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.080+0000 7fbbcb200640 1 -- 172.21.15.69:0/2671587722 wait complete. 2024-04-24T08:09:04.099 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:04.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.298+0000 7f6cc4200640 1 Processor -- start 2024-04-24T08:09:04.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.298+0000 7f6cc4200640 1 -- start start 2024-04-24T08:09:04.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.300+0000 7f6cc4200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc15c820 0x7f6cbc0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.300+0000 7f6cc4200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6cbc0ce250 0x7f6cbc0ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.300+0000 7f6cc4200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cbc15cc00 con 0x7f6cbc0b46a0 2024-04-24T08:09:04.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.300+0000 7f6cc4200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6cbc06ed30 con 0x7f6cbc0ce250 2024-04-24T08:09:04.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.300+0000 7f6cc4200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6cbc0b4e10 con 0x7f6cbc15c820 2024-04-24T08:09:04.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.301+0000 7f6cc2200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6cbc0ce250 0x7f6cbc0ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.301+0000 7f6cc1800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc15c820 0x7f6cbc0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.301+0000 7f6cc1800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc15c820 0x7f6cbc0b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:58952/0 (socket says 172.21.15.69:58952) 2024-04-24T08:09:04.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.301+0000 7f6cc0e00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6cbc0b46a0 0x7f6cbc06f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:34012/0 (socket says 172.21.15.69:34012) 2024-04-24T08:09:04.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.301+0000 7f6cc1800640 1 -- 172.21.15.69:0/4103095240 learned_addr learned my addr 172.21.15.69:0/4103095240 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:04.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.301+0000 7f6cc2200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6cbc0ce250 0x7f6cbc0ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:33242/0 (socket says 172.21.15.69:33242) 2024-04-24T08:09:04.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.301+0000 7f6cc1800640 1 -- 172.21.15.69:0/4103095240 >> v2:172.21.15.121:3300/0 conn(0x7f6cbc0ce250 msgr2=0x7f6cbc0ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:04.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.301+0000 7f6cc1800640 1 --2- 172.21.15.69:0/4103095240 >> v2:172.21.15.121:3300/0 conn(0x7f6cbc0ce250 0x7f6cbc0ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.302+0000 7f6cc1800640 1 -- 172.21.15.69:0/4103095240 >> v1:172.21.15.121:6789/0 conn(0x7f6cbc0b46a0 legacy=0x7f6cbc06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.302+0000 7f6cc1800640 1 -- 172.21.15.69:0/4103095240 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc0b50f0 con 0x7f6cbc15c820 2024-04-24T08:09:04.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.303+0000 7f6cc1800640 1 --2- 172.21.15.69:0/4103095240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc15c820 0x7f6cbc0b3e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0009870 tx=0x7f6cb002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5802affbd0d633b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:04.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.303+0000 7f6cbbe00640 1 -- 172.21.15.69:0/4103095240 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0037070 con 0x7f6cbc15c820 2024-04-24T08:09:04.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.303+0000 7f6cbbe00640 1 -- 172.21.15.69:0/4103095240 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb0032440 con 0x7f6cbc15c820 2024-04-24T08:09:04.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.304+0000 7f6cbbe00640 1 -- 172.21.15.69:0/4103095240 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb0032810 con 0x7f6cbc15c820 2024-04-24T08:09:04.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.304+0000 7f6cc4200640 1 -- 172.21.15.69:0/4103095240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc15c820 msgr2=0x7f6cbc0b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.304+0000 7f6cc4200640 1 --2- 172.21.15.69:0/4103095240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc15c820 0x7f6cbc0b3e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0009870 tx=0x7f6cb002a250 comp rx=0 tx=0).stop 2024-04-24T08:09:04.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.304+0000 7f6cc4200640 1 -- 172.21.15.69:0/4103095240 shutdown_connections 2024-04-24T08:09:04.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.304+0000 7f6cc4200640 1 --2- 172.21.15.69:0/4103095240 >> v2:172.21.15.121:3300/0 conn(0x7f6cbc0ce250 0x7f6cbc0ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.304+0000 7f6cc4200640 1 --2- 172.21.15.69:0/4103095240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc15c820 0x7f6cbc0b3e90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.304+0000 7f6cc4200640 1 -- 172.21.15.69:0/4103095240 >> 172.21.15.69:0/4103095240 conn(0x7f6cbc0d0350 msgr2=0x7f6cbc0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:04.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.304+0000 7f6cc4200640 1 -- 172.21.15.69:0/4103095240 shutdown_connections 2024-04-24T08:09:04.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.304+0000 7f6cc4200640 1 -- 172.21.15.69:0/4103095240 wait complete. 2024-04-24T08:09:04.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.306+0000 7f6cc4200640 1 Processor -- start 2024-04-24T08:09:04.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.306+0000 7f6cc4200640 1 -- start start 2024-04-24T08:09:04.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.307+0000 7f6cc4200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc0b46a0 0x7f6cbc1503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.307+0000 7f6cc4200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cbc0ce250 0x7f6cbc15ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.307+0000 7f6cc4200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cbc15c820 0x7f6cbc15d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.307+0000 7f6cc4200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc0b4b00 con 0x7f6cbc0ce250 2024-04-24T08:09:04.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.307+0000 7f6cc4200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc06f2b0 con 0x7f6cbc15c820 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.307+0000 7f6cc4200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6cbc06fee0 con 0x7f6cbc0b46a0 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.307+0000 7f6cc0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cbc0ce250 0x7f6cbc15ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.307+0000 7f6cc0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cbc0ce250 0x7f6cbc15ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34432/0 (socket says 172.21.15.69:34432) 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.307+0000 7f6cc0e00640 1 -- 172.21.15.69:0/3232926355 learned_addr learned my addr 172.21.15.69:0/3232926355 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.308+0000 7f6cc1800640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc0b46a0 0x7f6cbc1503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.308+0000 7f6cc2200640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cbc15c820 0x7f6cbc15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.308+0000 7f6cc0e00640 1 -- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc0b46a0 msgr2=0x7f6cbc1503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.308+0000 7f6cc0e00640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc0b46a0 0x7f6cbc1503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.308+0000 7f6cc0e00640 1 -- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cbc15c820 msgr2=0x7f6cbc15d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.308+0000 7f6cc0e00640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cbc15c820 0x7f6cbc15d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.308+0000 7f6cc0e00640 1 -- 172.21.15.69:0/3232926355 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc073750 con 0x7f6cbc0ce250 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.308+0000 7f6cc2200640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cbc15c820 0x7f6cbc15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:04.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.308+0000 7f6cc0e00640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cbc0ce250 0x7f6cbc15ce00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800c400 tx=0x7f6ca800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:04.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.308+0000 7f6cb9600640 1 -- 172.21.15.69:0/3232926355 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80074e0 con 0x7f6cbc0ce250 2024-04-24T08:09:04.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.309+0000 7f6cb9600640 1 -- 172.21.15.69:0/3232926355 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ca8007680 con 0x7f6cbc0ce250 2024-04-24T08:09:04.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.309+0000 7f6cc4200640 1 -- 172.21.15.69:0/3232926355 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cbc150d00 con 0x7f6cbc0ce250 2024-04-24T08:09:04.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.309+0000 7f6cb9600640 1 -- 172.21.15.69:0/3232926355 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8004a00 con 0x7f6cbc0ce250 2024-04-24T08:09:04.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.309+0000 7f6cc4200640 1 -- 172.21.15.69:0/3232926355 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cbc0cb130 con 0x7f6cbc0ce250 2024-04-24T08:09:04.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.310+0000 7f6cc4200640 1 -- 172.21.15.69:0/3232926355 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cbc068a80 con 0x7f6cbc0ce250 2024-04-24T08:09:04.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.311+0000 7f6cb9600640 1 -- 172.21.15.69:0/3232926355 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ca801a020 con 0x7f6cbc0ce250 2024-04-24T08:09:04.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.312+0000 7f6cb9600640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c9c06c920 0x7f6c9c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.312+0000 7f6cb9600640 1 -- 172.21.15.69:0/3232926355 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f6ca808b3f0 con 0x7f6cbc0ce250 2024-04-24T08:09:04.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.312+0000 7f6cc1800640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c9c06c920 0x7f6c9c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.313+0000 7f6cc1800640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c9c06c920 0x7f6c9c06ede0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0034000 tx=0x7f6cb002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:04.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.316+0000 7f6cb9600640 1 -- 172.21.15.69:0/3232926355 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6cbc150d00 con 0x7f6cbc0ce250 2024-04-24T08:09:04.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.518+0000 7f6cc4200640 1 -- 172.21.15.69:0/3232926355 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6cbc0cddc0 con 0x7f6cbc0ce250 2024-04-24T08:09:04.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.519+0000 7f6cb9600640 1 -- 172.21.15.69:0/3232926355 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f6ca8010300 con 0x7f6cbc0ce250 2024-04-24T08:09:04.519 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:04.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 -- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c9c06c920 msgr2=0x7f6c9c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c9c06c920 0x7f6c9c06ede0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6cb0034000 tx=0x7f6cb002e000 comp rx=0 tx=0).stop 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 -- 172.21.15.69:0/3232926355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cbc0ce250 msgr2=0x7f6cbc15ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cbc0ce250 0x7f6cbc15ce00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800c400 tx=0x7f6ca800c8c0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 -- 172.21.15.69:0/3232926355 shutdown_connections 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c9c06c920 0x7f6c9c06ede0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cbc0ce250 0x7f6cbc15ce00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cbc15c820 0x7f6cbc15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 --2- 172.21.15.69:0/3232926355 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cbc0b46a0 0x7f6cbc1503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 -- 172.21.15.69:0/3232926355 >> 172.21.15.69:0/3232926355 conn(0x7f6cbc0d0350 msgr2=0x7f6cbc1601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 -- 172.21.15.69:0/3232926355 shutdown_connections 2024-04-24T08:09:04.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.522+0000 7f6cc4200640 1 -- 172.21.15.69:0/3232926355 wait complete. 2024-04-24T08:09:04.541 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:04.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.741+0000 7fa9d6e00640 1 Processor -- start 2024-04-24T08:09:04.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.742+0000 7fa9d6e00640 1 -- start start 2024-04-24T08:09:04.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.743+0000 7fa9d6e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa9d015e850 0x7fa9d00cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.743+0000 7fa9d6e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015ec30 0x7fa9d00d1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.743+0000 7fa9d6e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9d00d2710 con 0x7fa9d00d0450 2024-04-24T08:09:04.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.743+0000 7fa9d6e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa9d00d37b0 con 0x7fa9d015e850 2024-04-24T08:09:04.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.743+0000 7fa9d6e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d006eba0 con 0x7fa9d015ec30 2024-04-24T08:09:04.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.744+0000 7fa9c7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015ec30 0x7fa9d00d1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.744+0000 7fa9cfe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa9d015e850 0x7fa9d00cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.744+0000 7fa9c7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015ec30 0x7fa9d00d1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:58974/0 (socket says 172.21.15.69:58974) 2024-04-24T08:09:04.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.744+0000 7fa9c7e00640 1 -- 172.21.15.69:0/1143449968 learned_addr learned my addr 172.21.15.69:0/1143449968 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:04.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.745+0000 7fa9c7e00640 1 -- 172.21.15.69:0/1143449968 >> v2:172.21.15.121:3300/0 conn(0x7fa9d015e850 msgr2=0x7fa9d00cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:04.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.745+0000 7fa9c7e00640 1 --2- 172.21.15.69:0/1143449968 >> v2:172.21.15.121:3300/0 conn(0x7fa9d015e850 0x7fa9d00cff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.745+0000 7fa9c7e00640 1 -- 172.21.15.69:0/1143449968 >> v1:172.21.15.121:6789/0 conn(0x7fa9d00d0450 legacy=0x7fa9d0162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.745+0000 7fa9c7e00640 1 -- 172.21.15.69:0/1143449968 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d00d29f0 con 0x7fa9d015ec30 2024-04-24T08:09:04.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.745+0000 7fa9cfe00640 1 --2- 172.21.15.69:0/1143449968 >> v2:172.21.15.121:3300/0 conn(0x7fa9d015e850 0x7fa9d00cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:04.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.746+0000 7fa9c7e00640 1 --2- 172.21.15.69:0/1143449968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015ec30 0x7fa9d00d1150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009870 tx=0x7fa9b802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52e2fb939c5a8ec1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:04.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.747+0000 7fa9cf400640 1 -- 172.21.15.69:0/1143449968 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8037070 con 0x7fa9d015ec30 2024-04-24T08:09:04.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.747+0000 7fa9cf400640 1 -- 172.21.15.69:0/1143449968 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b8032440 con 0x7fa9d015ec30 2024-04-24T08:09:04.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.747+0000 7fa9cf400640 1 -- 172.21.15.69:0/1143449968 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8032810 con 0x7fa9d015ec30 2024-04-24T08:09:04.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.748+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1143449968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015ec30 msgr2=0x7fa9d00d1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.748+0000 7fa9d6e00640 1 --2- 172.21.15.69:0/1143449968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015ec30 0x7fa9d00d1150 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009870 tx=0x7fa9b802a270 comp rx=0 tx=0).stop 2024-04-24T08:09:04.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.748+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1143449968 shutdown_connections 2024-04-24T08:09:04.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.748+0000 7fa9d6e00640 1 --2- 172.21.15.69:0/1143449968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015ec30 0x7fa9d00d1150 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.748+0000 7fa9d6e00640 1 --2- 172.21.15.69:0/1143449968 >> v2:172.21.15.121:3300/0 conn(0x7fa9d015e850 0x7fa9d00cff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.748+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1143449968 >> 172.21.15.69:0/1143449968 conn(0x7fa9d008bf60 msgr2=0x7fa9d00d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:04.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.748+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1143449968 shutdown_connections 2024-04-24T08:09:04.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.748+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1143449968 wait complete. 2024-04-24T08:09:04.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.750+0000 7fa9d6e00640 1 Processor -- start 2024-04-24T08:09:04.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.750+0000 7fa9d6e00640 1 -- start start 2024-04-24T08:09:04.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9d6e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9d00d0450 0x7fa9d00c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9d6e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015e850 0x7fa9d0158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9d6e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d015ec30 0x7fa9d00cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9d6e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d00d3290 con 0x7fa9d015ec30 2024-04-24T08:09:04.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9d6e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d00d2fb0 con 0x7fa9d00d0450 2024-04-24T08:09:04.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9d6e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d00d2330 con 0x7fa9d015e850 2024-04-24T08:09:04.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9d4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d015ec30 0x7fa9d00cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9d4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d015ec30 0x7fa9d00cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34448/0 (socket says 172.21.15.69:34448) 2024-04-24T08:09:04.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9d4e00640 1 -- 172.21.15.69:0/1383879720 learned_addr learned my addr 172.21.15.69:0/1383879720 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:04.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9cfe00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9d00d0450 0x7fa9d00c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.752+0000 7fa9c7e00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015e850 0x7fa9d0158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9d4e00640 1 -- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015e850 msgr2=0x7fa9d0158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9d4e00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015e850 0x7fa9d0158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9d4e00640 1 -- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9d00d0450 msgr2=0x7fa9d00c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9d4e00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9d00d0450 0x7fa9d00c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9d4e00640 1 -- 172.21.15.69:0/1383879720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d00b82a0 con 0x7fa9d015ec30 2024-04-24T08:09:04.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9cfe00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9d00d0450 0x7fa9d00c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:09:04.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9d4e00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d015ec30 0x7fa9d00cad00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800c6b0 tx=0x7fa9c800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:04.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9ccc00640 1 -- 172.21.15.69:0/1383879720 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8003b70 con 0x7fa9d015ec30 2024-04-24T08:09:04.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9ccc00640 1 -- 172.21.15.69:0/1383879720 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9c8003d30 con 0x7fa9d015ec30 2024-04-24T08:09:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1383879720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d00b7720 con 0x7fa9d015ec30 2024-04-24T08:09:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9ccc00640 1 -- 172.21.15.69:0/1383879720 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8005310 con 0x7fa9d015ec30 2024-04-24T08:09:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.753+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1383879720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d00b7ce0 con 0x7fa9d015ec30 2024-04-24T08:09:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.755+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1383879720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d015f590 con 0x7fa9d015ec30 2024-04-24T08:09:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.755+0000 7fa9ccc00640 1 -- 172.21.15.69:0/1383879720 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa9c80054b0 con 0x7fa9d015ec30 2024-04-24T08:09:04.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.756+0000 7fa9ccc00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9ac06c8f0 0x7fa9ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:04.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.756+0000 7fa9ccc00640 1 -- 172.21.15.69:0/1383879720 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fa9c808ba40 con 0x7fa9d015ec30 2024-04-24T08:09:04.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.757+0000 7fa9cfe00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9ac06c8f0 0x7fa9ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:04.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.758+0000 7fa9cfe00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9ac06c8f0 0x7fa9ac06edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0009ba0 tx=0x7fa9c0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:04.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.761+0000 7fa9ccc00640 1 -- 172.21.15.69:0/1383879720 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9d015f590 con 0x7fa9d015ec30 2024-04-24T08:09:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.965+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1383879720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa9d0068a80 con 0x7fa9d015ec30 2024-04-24T08:09:04.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.966+0000 7fa9ccc00640 1 -- 172.21.15.69:0/1383879720 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7fa9c808be20 con 0x7fa9d015ec30 2024-04-24T08:09:04.966 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9ac06c8f0 msgr2=0x7fa9ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9ac06c8f0 0x7fa9ac06edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0009ba0 tx=0x7fa9c0008040 comp rx=0 tx=0).stop 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1383879720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d015ec30 msgr2=0x7fa9d00cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d015ec30 0x7fa9d00cad00 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800c6b0 tx=0x7fa9c800cb70 comp rx=0 tx=0).stop 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1383879720 shutdown_connections 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9ac06c8f0 0x7fa9ac06edb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d015ec30 0x7fa9d00cad00 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d015e850 0x7fa9d0158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 --2- 172.21.15.69:0/1383879720 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9d00d0450 0x7fa9d00c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1383879720 >> 172.21.15.69:0/1383879720 conn(0x7fa9d008bf60 msgr2=0x7fa9d00b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1383879720 shutdown_connections 2024-04-24T08:09:04.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:04.969+0000 7fa9d6e00640 1 -- 172.21.15.69:0/1383879720 wait complete. 2024-04-24T08:09:04.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:09:05.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.185+0000 7f1309a00640 1 Processor -- start 2024-04-24T08:09:05.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.186+0000 7f1309a00640 1 -- start start 2024-04-24T08:09:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.187+0000 7f1309a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f130415d1d0 0x7f13040ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.187+0000 7f1309a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13040b30f0 0x7f13040ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.187+0000 7f1309a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f130415d660 con 0x7f1304156c60 2024-04-24T08:09:05.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.187+0000 7f1309a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f130406dc30 con 0x7f130415d1d0 2024-04-24T08:09:05.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.187+0000 7f1309a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f130406df10 con 0x7f13040b30f0 2024-04-24T08:09:05.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.187+0000 7f1303e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1304156c60 0x7f130415cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54904/0 (socket says 172.21.15.69:54904) 2024-04-24T08:09:05.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.187+0000 7f1303e00640 1 -- 172.21.15.69:0/3500685965 learned_addr learned my addr 172.21.15.69:0/3500685965 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:05.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3436419815 0 0) 0x7f130415d660 con 0x7f1304156c60 2024-04-24T08:09:05.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1309000640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13040b30f0 0x7f13040ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12e8003520 con 0x7f1304156c60 2024-04-24T08:09:05.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1303400640 1 --2- 172.21.15.69:0/3500685965 >> v2:172.21.15.121:3300/0 conn(0x7f130415d1d0 0x7f13040ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 847431715 0 0) 0x7f12e8003520 con 0x7f1304156c60 2024-04-24T08:09:05.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13040b30f0 msgr2=0x7f13040ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:05.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13040b30f0 0x7f13040ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 >> v2:172.21.15.121:3300/0 conn(0x7f130415d1d0 msgr2=0x7f13040ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:05.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 --2- 172.21.15.69:0/3500685965 >> v2:172.21.15.121:3300/0 conn(0x7f130415d1d0 0x7f13040ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130406e1f0 con 0x7f1304156c60 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f12f4001670 con 0x7f1304156c60 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 >> v1:172.21.15.69:6789/0 conn(0x7f1304156c60 legacy=0x7f130415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12e80039b0 0x7f12e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13040b30f0 0x7f12e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8004d40 0x7f12e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f130406dc30 con 0x7f12e8004d40 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f130406df10 con 0x7f12e80039b0 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.188+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12e800ba70 con 0x7f13040b30f0 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.189+0000 7f1309000640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8004d40 0x7f12e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.189+0000 7f1303400640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13040b30f0 0x7f12e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.189+0000 7f1303e00640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12e80039b0 0x7f12e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.189+0000 7f1309000640 1 -- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13040b30f0 msgr2=0x7f12e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:05.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.189+0000 7f1309000640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13040b30f0 0x7f12e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.189+0000 7f1309000640 1 -- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12e80039b0 msgr2=0x7f12e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:05.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.189+0000 7f1309000640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12e80039b0 0x7f12e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.189+0000 7f1309000640 1 -- 172.21.15.69:0/3500685965 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e8003520 con 0x7f12e8004d40 2024-04-24T08:09:05.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.190+0000 7f1309000640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8004d40 0x7f12e80051c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00ba10 tx=0x7f12e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0c1d4d78817d20 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:05.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.190+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc030070 con 0x7f12e8004d40 2024-04-24T08:09:05.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.190+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12fc02adb0 con 0x7f12e8004d40 2024-04-24T08:09:05.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.190+0000 7f1302a00640 1 -- 172.21.15.69:0/3500685965 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12fc00b610 con 0x7f12e8004d40 2024-04-24T08:09:05.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.191+0000 7f1309a00640 1 -- 172.21.15.69:0/3500685965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8004d40 msgr2=0x7f12e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.191+0000 7f1309a00640 1 --2- 172.21.15.69:0/3500685965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12e8004d40 0x7f12e80051c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f12fc00ba10 tx=0x7f12e8005830 comp rx=0 tx=0).stop 2024-04-24T08:09:05.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.191+0000 7f1309000640 1 -- 172.21.15.69:0/3500685965 reap_dead start 2024-04-24T08:09:05.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.191+0000 7f1309a00640 1 -- 172.21.15.69:0/3500685965 shutdown_connections 2024-04-24T08:09:05.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.191+0000 7f1309a00640 1 -- 172.21.15.69:0/3500685965 >> 172.21.15.69:0/3500685965 conn(0x7f13040d0160 msgr2=0x7f13040d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:05.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.191+0000 7f1309a00640 1 -- 172.21.15.69:0/3500685965 shutdown_connections 2024-04-24T08:09:05.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.191+0000 7f1309a00640 1 -- 172.21.15.69:0/3500685965 wait complete. 2024-04-24T08:09:05.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.193+0000 7f1309a00640 1 Processor -- start 2024-04-24T08:09:05.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.193+0000 7f1309a00640 1 -- start start 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.194+0000 7f1309a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e8004d40 0x7f1304155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.194+0000 7f1309a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13041f41a0 0x7f13040caa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.194+0000 7f1309a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13041f4620 0x7f13040caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.194+0000 7f1309a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f13040ad610 con 0x7f13041f41a0 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.194+0000 7f1309a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f130415d940 con 0x7f13041f4620 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.194+0000 7f1309a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f13040ad790 con 0x7f12e8004d40 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.194+0000 7f1303400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13041f41a0 0x7f13040caa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.194+0000 7f1303400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13041f41a0 0x7f13040caa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34462/0 (socket says 172.21.15.69:34462) 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.194+0000 7f1303400640 1 -- 172.21.15.69:0/2525005410 learned_addr learned my addr 172.21.15.69:0/2525005410 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f1303e00640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e8004d40 0x7f1304155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f1309000640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13041f4620 0x7f13040caf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f1303400640 1 -- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e8004d40 msgr2=0x7f1304155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f1303400640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e8004d40 0x7f1304155890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f1303400640 1 -- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13041f4620 msgr2=0x7f13040caf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f1303400640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13041f4620 0x7f13040caf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f1303400640 1 -- 172.21.15.69:0/2525005410 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1304072950 con 0x7f13041f41a0 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f1303400640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13041f41a0 0x7f13040caa50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009ce0 tx=0x7f12ec007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f12fbe00640 1 -- 172.21.15.69:0/2525005410 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec014610 con 0x7f13041f41a0 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f12fbe00640 1 -- 172.21.15.69:0/2525005410 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12ec019070 con 0x7f13041f41a0 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f1309a00640 1 -- 172.21.15.69:0/2525005410 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1304072c30 con 0x7f13041f41a0 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.195+0000 7f12fbe00640 1 -- 172.21.15.69:0/2525005410 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec0149f0 con 0x7f13041f41a0 2024-04-24T08:09:05.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.196+0000 7f1309a00640 1 -- 172.21.15.69:0/2525005410 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13040731f0 con 0x7f13041f41a0 2024-04-24T08:09:05.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.197+0000 7f1309a00640 1 -- 172.21.15.69:0/2525005410 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1304075d50 con 0x7f13041f41a0 2024-04-24T08:09:05.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.198+0000 7f12fbe00640 1 -- 172.21.15.69:0/2525005410 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f12ec014b90 con 0x7f13041f41a0 2024-04-24T08:09:05.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.199+0000 7f12fbe00640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f12e006c8f0 0x7f12e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.199+0000 7f12fbe00640 1 -- 172.21.15.69:0/2525005410 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f12ec08c610 con 0x7f13041f41a0 2024-04-24T08:09:05.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.200+0000 7f1303e00640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f12e006c8f0 0x7f12e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.201+0000 7f1303e00640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f12e006c8f0 0x7f12e006edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f12f40058d0 tx=0x7f12f4000f10 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:05.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.204+0000 7f12fbe00640 1 -- 172.21.15.69:0/2525005410 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f12ec024250 con 0x7f13041f41a0 2024-04-24T08:09:05.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.419+0000 7f1309a00640 1 -- 172.21.15.69:0/2525005410 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f130406f9e0 con 0x7f13041f41a0 2024-04-24T08:09:05.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.421+0000 7f12fbe00640 1 -- 172.21.15.69:0/2525005410 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f12ec05a9a0 con 0x7f13041f41a0 2024-04-24T08:09:05.421 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:09:05.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 -- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f12e006c8f0 msgr2=0x7f12e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f12e006c8f0 0x7f12e006edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f12f40058d0 tx=0x7f12f4000f10 comp rx=0 tx=0).stop 2024-04-24T08:09:05.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 -- 172.21.15.69:0/2525005410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13041f41a0 msgr2=0x7f13040caa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13041f41a0 0x7f13040caa50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009ce0 tx=0x7f12ec007700 comp rx=0 tx=0).stop 2024-04-24T08:09:05.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 -- 172.21.15.69:0/2525005410 shutdown_connections 2024-04-24T08:09:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f12e006c8f0 0x7f12e006edb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13041f41a0 0x7f13040caa50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13041f4620 0x7f13040caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 --2- 172.21.15.69:0/2525005410 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12e8004d40 0x7f1304155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 -- 172.21.15.69:0/2525005410 >> 172.21.15.69:0/2525005410 conn(0x7f13040d0160 msgr2=0x7f130406ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 -- 172.21.15.69:0/2525005410 shutdown_connections 2024-04-24T08:09:05.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.424+0000 7f1309a00640 1 -- 172.21.15.69:0/2525005410 wait complete. 2024-04-24T08:09:05.442 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:09:05.443 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:05.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:05.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.736+0000 7fec67e00640 1 Processor -- start 2024-04-24T08:09:05.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.736+0000 7fec67e00640 1 -- start start 2024-04-24T08:09:05.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.738+0000 7fec67e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec600d0300 0x7fec6006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.738+0000 7fec67e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fec60145a90 0x7fec600d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.738+0000 7fec67e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec600d0c20 con 0x7fec60145fe0 2024-04-24T08:09:05.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.738+0000 7fec67e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fec600cf3d0 con 0x7fec60145a90 2024-04-24T08:09:05.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.738+0000 7fec67e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec600cf6b0 con 0x7fec600d0300 2024-04-24T08:09:05.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.738+0000 7fec65e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fec60145fe0 0x7fec60160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54910/0 (socket says 172.21.15.69:54910) 2024-04-24T08:09:05.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.738+0000 7fec65e00640 1 -- 172.21.15.69:0/2688526710 learned_addr learned my addr 172.21.15.69:0/2688526710 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:05.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec65400640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec600d0300 0x7fec6006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec64a00640 1 --2- 172.21.15.69:0/2688526710 >> v2:172.21.15.121:3300/0 conn(0x7fec60145a90 0x7fec600d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3407492447 0 0) 0x7fec600d0c20 con 0x7fec60145fe0 2024-04-24T08:09:05.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec48003520 con 0x7fec60145fe0 2024-04-24T08:09:05.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4292080551 0 0) 0x7fec48003520 con 0x7fec60145fe0 2024-04-24T08:09:05.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec600d0300 msgr2=0x7fec6006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec600d0300 0x7fec6006e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 >> v2:172.21.15.121:3300/0 conn(0x7fec60145a90 msgr2=0x7fec600d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 --2- 172.21.15.69:0/2688526710 >> v2:172.21.15.121:3300/0 conn(0x7fec60145a90 0x7fec600d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec600cf990 con 0x7fec60145fe0 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fec580019f0 con 0x7fec60145fe0 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 >> v1:172.21.15.69:6789/0 conn(0x7fec60145fe0 legacy=0x7fec60160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.739+0000 7fec5fe00640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 0x7fec48003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec5fe00640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec48004800 0x7fec48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec5fe00640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec600d0300 0x7fec480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fec600cf3d0 con 0x7fec480039b0 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fec600cf6b0 con 0x7fec48004800 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec4800b9c0 con 0x7fec600d0300 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec65400640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 0x7fec48003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec64a00640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec48004800 0x7fec48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec65400640 1 -- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec600d0300 msgr2=0x7fec480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec65e00640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec600d0300 0x7fec480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec65400640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec600d0300 0x7fec480051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec65400640 1 -- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec48004800 msgr2=0x7fec48004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec65400640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec48004800 0x7fec48004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec65400640 1 -- 172.21.15.69:0/2688526710 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec48003520 con 0x7fec480039b0 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.740+0000 7fec64a00640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec48004800 0x7fec48004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.741+0000 7fec65400640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 0x7fec48003dd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fec54009ac0 tx=0x7fec48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb2a536d632abd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.741+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec54036070 con 0x7fec480039b0 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.741+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec54031400 con 0x7fec480039b0 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.741+0000 7fec5fe00640 1 -- 172.21.15.69:0/2688526710 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec54009690 con 0x7fec480039b0 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.742+0000 7fec67e00640 1 -- 172.21.15.69:0/2688526710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 msgr2=0x7fec48003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.742+0000 7fec67e00640 1 --2- 172.21.15.69:0/2688526710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 0x7fec48003dd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fec54009ac0 tx=0x7fec48005830 comp rx=0 tx=0).stop 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.742+0000 7fec65e00640 1 -- 172.21.15.69:0/2688526710 reap_dead start 2024-04-24T08:09:05.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.742+0000 7fec67e00640 1 -- 172.21.15.69:0/2688526710 shutdown_connections 2024-04-24T08:09:05.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.742+0000 7fec67e00640 1 -- 172.21.15.69:0/2688526710 >> 172.21.15.69:0/2688526710 conn(0x7fec60067d90 msgr2=0x7fec6006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:05.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.742+0000 7fec67e00640 1 -- 172.21.15.69:0/2688526710 shutdown_connections 2024-04-24T08:09:05.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.742+0000 7fec67e00640 1 -- 172.21.15.69:0/2688526710 wait complete. 2024-04-24T08:09:05.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.744+0000 7fec67e00640 1 Processor -- start 2024-04-24T08:09:05.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.744+0000 7fec67e00640 1 -- start start 2024-04-24T08:09:05.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.745+0000 7fec67e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 0x7fec600aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.745+0000 7fec67e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec6013eec0 0x7fec600b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.745+0000 7fec67e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec600b5020 0x7fec601f46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.745+0000 7fec67e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fec60145770 con 0x7fec480039b0 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.745+0000 7fec67e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fec600cff50 con 0x7fec600b5020 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.745+0000 7fec67e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec601458f0 con 0x7fec6013eec0 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.745+0000 7fec65400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 0x7fec600aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.745+0000 7fec65400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 0x7fec600aea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34478/0 (socket says 172.21.15.69:34478) 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.745+0000 7fec65400640 1 -- 172.21.15.69:0/8632184 learned_addr learned my addr 172.21.15.69:0/8632184 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec65e00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec600b5020 0x7fec601f46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec65400640 1 -- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec6013eec0 msgr2=0x7fec600b4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec64a00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec6013eec0 0x7fec600b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec65400640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec6013eec0 0x7fec600b4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec65400640 1 -- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec600b5020 msgr2=0x7fec601f46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec65400640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec600b5020 0x7fec601f46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec65400640 1 -- 172.21.15.69:0/8632184 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec600ad9e0 con 0x7fec480039b0 2024-04-24T08:09:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec65e00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec600b5020 0x7fec601f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec65400640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 0x7fec600aea80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fec54034040 tx=0x7fec54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec5d600640 1 -- 172.21.15.69:0/8632184 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec54040020 con 0x7fec480039b0 2024-04-24T08:09:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec5d600640 1 -- 172.21.15.69:0/8632184 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec54003bc0 con 0x7fec480039b0 2024-04-24T08:09:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec67e00640 1 -- 172.21.15.69:0/8632184 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec600adcc0 con 0x7fec480039b0 2024-04-24T08:09:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.746+0000 7fec5d600640 1 -- 172.21.15.69:0/8632184 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec54036040 con 0x7fec480039b0 2024-04-24T08:09:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.747+0000 7fec67e00640 1 -- 172.21.15.69:0/8632184 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec600ae280 con 0x7fec480039b0 2024-04-24T08:09:05.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.748+0000 7fec67e00640 1 -- 172.21.15.69:0/8632184 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec600cdc30 con 0x7fec480039b0 2024-04-24T08:09:05.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.749+0000 7fec5d600640 1 -- 172.21.15.69:0/8632184 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fec54043020 con 0x7fec480039b0 2024-04-24T08:09:05.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.750+0000 7fec5d600640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec4006c920 0x7fec4006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:05.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.750+0000 7fec5d600640 1 -- 172.21.15.69:0/8632184 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fec540abcf0 con 0x7fec480039b0 2024-04-24T08:09:05.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.751+0000 7fec64a00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec4006c920 0x7fec4006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:05.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.752+0000 7fec64a00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec4006c920 0x7fec4006ede0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fec600b1d80 tx=0x7fec4c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:05.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.755+0000 7fec5d600640 1 -- 172.21.15.69:0/8632184 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec600cdc30 con 0x7fec480039b0 2024-04-24T08:09:05.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.956+0000 7fec67e00640 1 -- 172.21.15.69:0/8632184 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fec60068a80 con 0x7fec480039b0 2024-04-24T08:09:05.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.959+0000 7fec5d600640 1 -- 172.21.15.69:0/8632184 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7fec5404a070 con 0x7fec480039b0 2024-04-24T08:09:05.960 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:05.960 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 -- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec4006c920 msgr2=0x7fec4006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec4006c920 0x7fec4006ede0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fec600b1d80 tx=0x7fec4c008040 comp rx=0 tx=0).stop 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 -- 172.21.15.69:0/8632184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 msgr2=0x7fec600aea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 0x7fec600aea80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fec54034040 tx=0x7fec54004730 comp rx=0 tx=0).stop 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 -- 172.21.15.69:0/8632184 shutdown_connections 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec4006c920 0x7fec4006ede0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec600b5020 0x7fec601f46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec6013eec0 0x7fec600b4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 --2- 172.21.15.69:0/8632184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec480039b0 0x7fec600aea80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 -- 172.21.15.69:0/8632184 >> 172.21.15.69:0/8632184 conn(0x7fec60067d90 msgr2=0x7fec600b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 -- 172.21.15.69:0/8632184 shutdown_connections 2024-04-24T08:09:05.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:05.962+0000 7fec67e00640 1 -- 172.21.15.69:0/8632184 wait complete. 2024-04-24T08:09:05.982 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:05.983 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:06.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.187+0000 7f28eda00640 1 Processor -- start 2024-04-24T08:09:06.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.188+0000 7f28eda00640 1 -- start start 2024-04-24T08:09:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.189+0000 7f28eda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28e815e980 0x7f28e8162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.189+0000 7f28eda00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f28e80d0da0 0x7f28e80cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.189+0000 7f28eda00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f28e80d0190 0x7f28e806cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.189+0000 7f28eda00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f28e815ed60 con 0x7f28e80d0190 2024-04-24T08:09:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.189+0000 7f28eda00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f28e80d1580 con 0x7f28e80d0da0 2024-04-24T08:09:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.189+0000 7f28eda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f28e80d1860 con 0x7f28e815e980 2024-04-24T08:09:06.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.190+0000 7f28ed000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f28e80d0190 0x7f28e806cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.190+0000 7f28ed000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f28e80d0190 0x7f28e806cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34480/0 (socket says 172.21.15.69:34480) 2024-04-24T08:09:06.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.190+0000 7f28ed000640 1 -- 172.21.15.69:0/4175727530 learned_addr learned my addr 172.21.15.69:0/4175727530 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:06.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.190+0000 7f28e7e00640 1 --2- 172.21.15.69:0/4175727530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28e815e980 0x7f28e8162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.190+0000 7f28df400640 1 --2- 172.21.15.69:0/4175727530 >> v2:172.21.15.121:3300/0 conn(0x7f28e80d0da0 0x7f28e80cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.190+0000 7f28ed000640 1 -- 172.21.15.69:0/4175727530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28e815e980 msgr2=0x7f28e8162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.190+0000 7f28ed000640 1 --2- 172.21.15.69:0/4175727530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28e815e980 0x7f28e8162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.190+0000 7f28ed000640 1 -- 172.21.15.69:0/4175727530 >> v2:172.21.15.121:3300/0 conn(0x7f28e80d0da0 msgr2=0x7f28e80cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.190+0000 7f28ed000640 1 --2- 172.21.15.69:0/4175727530 >> v2:172.21.15.121:3300/0 conn(0x7f28e80d0da0 0x7f28e80cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.190+0000 7f28ed000640 1 -- 172.21.15.69:0/4175727530 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e80cf450 con 0x7f28e80d0190 2024-04-24T08:09:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.191+0000 7f28ed000640 1 --2- 172.21.15.69:0/4175727530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e80d0190 0x7f28e806cfb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f28e0009e50 tx=0x7f28e002a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee01467f7397c29 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.191+0000 7f28e7400640 1 -- 172.21.15.69:0/4175727530 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e0036070 con 0x7f28e80d0190 2024-04-24T08:09:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28e7400640 1 -- 172.21.15.69:0/4175727530 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e0031600 con 0x7f28e80d0190 2024-04-24T08:09:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28e7400640 1 -- 172.21.15.69:0/4175727530 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e00319d0 con 0x7f28e80d0190 2024-04-24T08:09:06.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28eda00640 1 -- 172.21.15.69:0/4175727530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e80d0190 msgr2=0x7f28e806cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28eda00640 1 --2- 172.21.15.69:0/4175727530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e80d0190 0x7f28e806cfb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f28e0009e50 tx=0x7f28e002a750 comp rx=0 tx=0).stop 2024-04-24T08:09:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28eda00640 1 -- 172.21.15.69:0/4175727530 shutdown_connections 2024-04-24T08:09:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28eda00640 1 --2- 172.21.15.69:0/4175727530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e80d0190 0x7f28e806cfb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28eda00640 1 --2- 172.21.15.69:0/4175727530 >> v2:172.21.15.121:3300/0 conn(0x7f28e80d0da0 0x7f28e80cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28eda00640 1 --2- 172.21.15.69:0/4175727530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28e815e980 0x7f28e8162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28eda00640 1 -- 172.21.15.69:0/4175727530 >> 172.21.15.69:0/4175727530 conn(0x7f28e809d920 msgr2=0x7f28e808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28eda00640 1 -- 172.21.15.69:0/4175727530 shutdown_connections 2024-04-24T08:09:06.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.192+0000 7f28eda00640 1 -- 172.21.15.69:0/4175727530 wait complete. 2024-04-24T08:09:06.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.194+0000 7f28eda00640 1 Processor -- start 2024-04-24T08:09:06.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.194+0000 7f28eda00640 1 -- start start 2024-04-24T08:09:06.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.195+0000 7f28eda00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e80d0190 0x7f28e80b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.195+0000 7f28eda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28e80d0da0 0x7f28e80b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.195+0000 7f28eda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e815e980 0x7f28e81f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.195+0000 7f28eda00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f28e80d1360 con 0x7f28e815e980 2024-04-24T08:09:06.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.195+0000 7f28eda00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f28e80d11e0 con 0x7f28e80d0190 2024-04-24T08:09:06.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.195+0000 7f28eda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f28e80cf070 con 0x7f28e80d0da0 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.195+0000 7f28ed000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e815e980 0x7f28e81f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.195+0000 7f28ed000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e815e980 0x7f28e81f49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34488/0 (socket says 172.21.15.69:34488) 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.195+0000 7f28ed000640 1 -- 172.21.15.69:0/891384013 learned_addr learned my addr 172.21.15.69:0/891384013 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28e7e00640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e80d0190 0x7f28e80b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28df400640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28e80d0da0 0x7f28e80b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28ed000640 1 -- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28e80d0da0 msgr2=0x7f28e80b40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28ed000640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28e80d0da0 0x7f28e80b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28ed000640 1 -- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e80d0190 msgr2=0x7f28e80b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28ed000640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e80d0190 0x7f28e80b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28ed000640 1 -- 172.21.15.69:0/891384013 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28e814eb20 con 0x7f28e815e980 2024-04-24T08:09:06.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28ed000640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e815e980 0x7f28e81f49f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f28e0009e50 tx=0x7f28e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:06.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28e4c00640 1 -- 172.21.15.69:0/891384013 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e0036050 con 0x7f28e815e980 2024-04-24T08:09:06.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28e4c00640 1 -- 172.21.15.69:0/891384013 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e0042070 con 0x7f28e815e980 2024-04-24T08:09:06.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28e4c00640 1 -- 172.21.15.69:0/891384013 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e0009510 con 0x7f28e815e980 2024-04-24T08:09:06.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.196+0000 7f28eda00640 1 -- 172.21.15.69:0/891384013 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28e80b6050 con 0x7f28e815e980 2024-04-24T08:09:06.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.197+0000 7f28eda00640 1 -- 172.21.15.69:0/891384013 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28e80b6610 con 0x7f28e815e980 2024-04-24T08:09:06.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.198+0000 7f28eda00640 1 -- 172.21.15.69:0/891384013 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28e8075d50 con 0x7f28e815e980 2024-04-24T08:09:06.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.199+0000 7f28e4c00640 1 -- 172.21.15.69:0/891384013 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f28e0007bf0 con 0x7f28e815e980 2024-04-24T08:09:06.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.199+0000 7f28e4c00640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f28c406c8f0 0x7f28c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.200+0000 7f28e4c00640 1 -- 172.21.15.69:0/891384013 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f28e00ac910 con 0x7f28e815e980 2024-04-24T08:09:06.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.200+0000 7f28e7e00640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f28c406c8f0 0x7f28c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.201+0000 7f28e7e00640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f28c406c8f0 0x7f28c406edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f28e81f2040 tx=0x7f28d8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:06.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.204+0000 7f28e4c00640 1 -- 172.21.15.69:0/891384013 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f28e007aca0 con 0x7f28e815e980 2024-04-24T08:09:06.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.397+0000 7f28eda00640 1 -- 172.21.15.69:0/891384013 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f28e80d1e40 con 0x7f28e815e980 2024-04-24T08:09:06.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.398+0000 7f28e4c00640 1 -- 172.21.15.69:0/891384013 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f28e007a5a0 con 0x7f28e815e980 2024-04-24T08:09:06.398 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:06.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 -- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f28c406c8f0 msgr2=0x7f28c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f28c406c8f0 0x7f28c406edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f28e81f2040 tx=0x7f28d8008040 comp rx=0 tx=0).stop 2024-04-24T08:09:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 -- 172.21.15.69:0/891384013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e815e980 msgr2=0x7f28e81f49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e815e980 0x7f28e81f49f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f28e0009e50 tx=0x7f28e0004a40 comp rx=0 tx=0).stop 2024-04-24T08:09:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 -- 172.21.15.69:0/891384013 shutdown_connections 2024-04-24T08:09:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f28c406c8f0 0x7f28c406edb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28e815e980 0x7f28e81f49f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28e80d0da0 0x7f28e80b40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 --2- 172.21.15.69:0/891384013 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28e80d0190 0x7f28e80b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 -- 172.21.15.69:0/891384013 >> 172.21.15.69:0/891384013 conn(0x7f28e809d920 msgr2=0x7f28e80b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 -- 172.21.15.69:0/891384013 shutdown_connections 2024-04-24T08:09:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.400+0000 7f28eda00640 1 -- 172.21.15.69:0/891384013 wait complete. 2024-04-24T08:09:06.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:06.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.622+0000 7fe860200640 1 Processor -- start 2024-04-24T08:09:06.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.622+0000 7fe860200640 1 -- start start 2024-04-24T08:09:06.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.624+0000 7fe860200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe85815c820 0x7fe8580b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.624+0000 7fe860200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8580ce250 0x7fe8580ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.624+0000 7fe860200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe85815cc00 con 0x7fe8580b46a0 2024-04-24T08:09:06.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.624+0000 7fe860200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe85806ed30 con 0x7fe85815c820 2024-04-24T08:09:06.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.624+0000 7fe860200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe8580b4e10 con 0x7fe8580ce250 2024-04-24T08:09:06.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.624+0000 7fe85ce00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe8580b46a0 0x7fe85806f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54914/0 (socket says 172.21.15.69:54914) 2024-04-24T08:09:06.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.624+0000 7fe85ce00640 1 -- 172.21.15.69:0/2228427126 learned_addr learned my addr 172.21.15.69:0/2228427126 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:06.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.624+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1572745563 0 0) 0x7fe85815cc00 con 0x7fe8580b46a0 2024-04-24T08:09:06.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe85d800640 1 --2- 172.21.15.69:0/2228427126 >> v2:172.21.15.121:3300/0 conn(0x7fe85815c820 0x7fe8580b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe85e200640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8580ce250 0x7fe8580ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe840003520 con 0x7fe8580b46a0 2024-04-24T08:09:06.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3458950888 0 0) 0x7fe840003520 con 0x7fe8580b46a0 2024-04-24T08:09:06.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8580ce250 msgr2=0x7fe8580ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:06.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8580ce250 0x7fe8580ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 >> v2:172.21.15.121:3300/0 conn(0x7fe85815c820 msgr2=0x7fe8580b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:06.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 --2- 172.21.15.69:0/2228427126 >> v2:172.21.15.121:3300/0 conn(0x7fe85815c820 0x7fe8580b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8580b50f0 con 0x7fe8580b46a0 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe844001670 con 0x7fe8580b46a0 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 >> v1:172.21.15.69:6789/0 conn(0x7fe8580b46a0 legacy=0x7fe85806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8580ce250 0x7fe8400039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe840003fe0 0x7fe840004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 0x7fe8400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe85806ed30 con 0x7fe840004d40 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe8580b4e10 con 0x7fe840003fe0 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe84000fed0 con 0x7fe8580ce250 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.625+0000 7fe85e200640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 0x7fe8400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.626+0000 7fe85ce00640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe840003fe0 0x7fe840004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.626+0000 7fe85d800640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8580ce250 0x7fe8400039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.626+0000 7fe85e200640 1 -- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8580ce250 msgr2=0x7fe8400039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.626+0000 7fe85e200640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8580ce250 0x7fe8400039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.626+0000 7fe85e200640 1 -- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe840003fe0 msgr2=0x7fe840004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:06.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.626+0000 7fe85e200640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe840003fe0 0x7fe840004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.626+0000 7fe85e200640 1 -- 172.21.15.69:0/2228427126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe840003520 con 0x7fe840004d40 2024-04-24T08:09:06.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.626+0000 7fe85ce00640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe840003fe0 0x7fe840004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:06.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.626+0000 7fe85d800640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8580ce250 0x7fe8400039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:06.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.627+0000 7fe85e200640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 0x7fe8400051c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe85000ba30 tx=0x7fe840005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440006cd2b7e2a9b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:06.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.627+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe850030070 con 0x7fe840004d40 2024-04-24T08:09:06.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.627+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85002add0 con 0x7fe840004d40 2024-04-24T08:09:06.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.627+0000 7fe857e00640 1 -- 172.21.15.69:0/2228427126 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85000b600 con 0x7fe840004d40 2024-04-24T08:09:06.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.628+0000 7fe860200640 1 -- 172.21.15.69:0/2228427126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 msgr2=0x7fe8400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.628+0000 7fe860200640 1 --2- 172.21.15.69:0/2228427126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 0x7fe8400051c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe85000ba30 tx=0x7fe840005830 comp rx=0 tx=0).stop 2024-04-24T08:09:06.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.628+0000 7fe85e200640 1 -- 172.21.15.69:0/2228427126 reap_dead start 2024-04-24T08:09:06.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.628+0000 7fe860200640 1 -- 172.21.15.69:0/2228427126 shutdown_connections 2024-04-24T08:09:06.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.628+0000 7fe860200640 1 -- 172.21.15.69:0/2228427126 >> 172.21.15.69:0/2228427126 conn(0x7fe8580d0350 msgr2=0x7fe8580d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:06.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.628+0000 7fe860200640 1 -- 172.21.15.69:0/2228427126 shutdown_connections 2024-04-24T08:09:06.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.628+0000 7fe860200640 1 -- 172.21.15.69:0/2228427126 wait complete. 2024-04-24T08:09:06.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.630+0000 7fe860200640 1 Processor -- start 2024-04-24T08:09:06.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.630+0000 7fe860200640 1 -- start start 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.631+0000 7fe860200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 0x7fe85815d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.631+0000 7fe860200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe85814e380 0x7fe8580aec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.631+0000 7fe860200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe858156ce0 0x7fe858157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.631+0000 7fe860200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe8580b4b30 con 0x7fe840004d40 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.631+0000 7fe860200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe85806f2b0 con 0x7fe858156ce0 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.631+0000 7fe860200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe85806fee0 con 0x7fe85814e380 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.631+0000 7fe85d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 0x7fe85815d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.631+0000 7fe85d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 0x7fe85815d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34502/0 (socket says 172.21.15.69:34502) 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.631+0000 7fe85d800640 1 -- 172.21.15.69:0/2615041017 learned_addr learned my addr 172.21.15.69:0/2615041017 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe85ce00640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe85814e380 0x7fe8580aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe85d800640 1 -- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe85814e380 msgr2=0x7fe8580aec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe85d800640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe85814e380 0x7fe8580aec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe85e200640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe858156ce0 0x7fe858157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe85d800640 1 -- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe858156ce0 msgr2=0x7fe858157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe85d800640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe858156ce0 0x7fe858157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe85d800640 1 -- 172.21.15.69:0/2615041017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85815dcf0 con 0x7fe840004d40 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe85ce00640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe85814e380 0x7fe8580aec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe85d800640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 0x7fe85815d610 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00f150 tx=0x7fe84c00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe855600640 1 -- 172.21.15.69:0/2615041017 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c018020 con 0x7fe840004d40 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe855600640 1 -- 172.21.15.69:0/2615041017 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe84c014600 con 0x7fe840004d40 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe855600640 1 -- 172.21.15.69:0/2615041017 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c0149b0 con 0x7fe840004d40 2024-04-24T08:09:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.632+0000 7fe860200640 1 -- 172.21.15.69:0/2615041017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe85815dfd0 con 0x7fe840004d40 2024-04-24T08:09:06.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.633+0000 7fe860200640 1 -- 172.21.15.69:0/2615041017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe85815e590 con 0x7fe840004d40 2024-04-24T08:09:06.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.634+0000 7fe860200640 1 -- 172.21.15.69:0/2615041017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe858068a80 con 0x7fe840004d40 2024-04-24T08:09:06.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.634+0000 7fe855600640 1 -- 172.21.15.69:0/2615041017 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe84c01b020 con 0x7fe840004d40 2024-04-24T08:09:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.635+0000 7fe855600640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe83806c8f0 0x7fe83806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.635+0000 7fe855600640 1 -- 172.21.15.69:0/2615041017 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fe84c08c990 con 0x7fe840004d40 2024-04-24T08:09:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.636+0000 7fe85ce00640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe83806c8f0 0x7fe83806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.637+0000 7fe85ce00640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe83806c8f0 0x7fe83806edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe8440017e0 tx=0x7fe844009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:06.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.640+0000 7fe855600640 1 -- 172.21.15.69:0/2615041017 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe84c022020 con 0x7fe840004d40 2024-04-24T08:09:06.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.840+0000 7fe860200640 1 -- 172.21.15.69:0/2615041017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe85814ed40 con 0x7fe840004d40 2024-04-24T08:09:06.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.841+0000 7fe855600640 1 -- 172.21.15.69:0/2615041017 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fe84c08c020 con 0x7fe840004d40 2024-04-24T08:09:06.841 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.843+0000 7fe860200640 1 -- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe83806c8f0 msgr2=0x7fe83806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.843+0000 7fe860200640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe83806c8f0 0x7fe83806edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe8440017e0 tx=0x7fe844009040 comp rx=0 tx=0).stop 2024-04-24T08:09:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.843+0000 7fe860200640 1 -- 172.21.15.69:0/2615041017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 msgr2=0x7fe85815d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.843+0000 7fe860200640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 0x7fe85815d610 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00f150 tx=0x7fe84c00f610 comp rx=0 tx=0).stop 2024-04-24T08:09:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.843+0000 7fe860200640 1 -- 172.21.15.69:0/2615041017 shutdown_connections 2024-04-24T08:09:06.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.843+0000 7fe860200640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe83806c8f0 0x7fe83806edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.843+0000 7fe860200640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe858156ce0 0x7fe858157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.843+0000 7fe860200640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe85814e380 0x7fe8580aec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.843+0000 7fe860200640 1 --2- 172.21.15.69:0/2615041017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe840004d40 0x7fe85815d610 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.843+0000 7fe860200640 1 -- 172.21.15.69:0/2615041017 >> 172.21.15.69:0/2615041017 conn(0x7fe8580d0350 msgr2=0x7fe858158370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.844+0000 7fe860200640 1 -- 172.21.15.69:0/2615041017 shutdown_connections 2024-04-24T08:09:06.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:06.844+0000 7fe860200640 1 -- 172.21.15.69:0/2615041017 wait complete. 2024-04-24T08:09:06.863 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.063+0000 7f92be800640 1 Processor -- start 2024-04-24T08:09:07.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.063+0000 7f92be800640 1 -- start start 2024-04-24T08:09:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.064+0000 7f92be800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f92b80b46a0 0x7f92b80ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.064+0000 7f92be800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92b815cc00 con 0x7f92b815c820 2024-04-24T08:09:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.064+0000 7f92be800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92b806ed30 con 0x7f92b80b3e90 2024-04-24T08:09:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.064+0000 7f92be800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f92b80b4e10 con 0x7f92b80b46a0 2024-04-24T08:09:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.065+0000 7f92b7400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f92b815c820 0x7f92b806f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54926/0 (socket says 172.21.15.69:54926) 2024-04-24T08:09:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.065+0000 7f92b7400640 1 -- 172.21.15.69:0/1291799509 learned_addr learned my addr 172.21.15.69:0/1291799509 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:07.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.065+0000 7f92b7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f92b80b46a0 0x7f92b80ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.065+0000 7f92b7e00640 1 -- 172.21.15.69:0/1291799509 >> v1:172.21.15.121:6789/0 conn(0x7f92b80b3e90 legacy=0x7f92b80ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.065+0000 7f92b7e00640 1 -- 172.21.15.69:0/1291799509 >> v1:172.21.15.69:6789/0 conn(0x7f92b815c820 legacy=0x7f92b806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.065+0000 7f92b7e00640 1 -- 172.21.15.69:0/1291799509 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b80b50f0 con 0x7f92b80b46a0 2024-04-24T08:09:07.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.065+0000 7f92b6a00640 1 -- 172.21.15.69:0/1291799509 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 409976440 0 0) 0x7f92b815cc00 con 0x7f92b815c820 2024-04-24T08:09:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.066+0000 7f92b7e00640 1 --2- 172.21.15.69:0/1291799509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 0x7f92b80ccd10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f92b815cdd0 tx=0x7f92b002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f143676fbb6dfffd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.067+0000 7f92b6a00640 1 -- 172.21.15.69:0/1291799509 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b0007740 con 0x7f92b80b46a0 2024-04-24T08:09:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.067+0000 7f92b6a00640 1 -- 172.21.15.69:0/1291799509 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b00078e0 con 0x7f92b80b46a0 2024-04-24T08:09:07.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.067+0000 7f92b6a00640 1 -- 172.21.15.69:0/1291799509 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b0007cb0 con 0x7f92b80b46a0 2024-04-24T08:09:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.067+0000 7f92be800640 1 -- 172.21.15.69:0/1291799509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 msgr2=0x7f92b80ccd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.067+0000 7f92be800640 1 --2- 172.21.15.69:0/1291799509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 0x7f92b80ccd10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f92b815cdd0 tx=0x7f92b002a6a0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.067+0000 7f92be800640 1 -- 172.21.15.69:0/1291799509 shutdown_connections 2024-04-24T08:09:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.067+0000 7f92be800640 1 --2- 172.21.15.69:0/1291799509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 0x7f92b80ccd10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.067+0000 7f92be800640 1 -- 172.21.15.69:0/1291799509 >> 172.21.15.69:0/1291799509 conn(0x7f92b80d0350 msgr2=0x7f92b80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.067+0000 7f92be800640 1 -- 172.21.15.69:0/1291799509 shutdown_connections 2024-04-24T08:09:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.067+0000 7f92be800640 1 -- 172.21.15.69:0/1291799509 wait complete. 2024-04-24T08:09:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.069+0000 7f92be800640 1 Processor -- start 2024-04-24T08:09:07.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.069+0000 7f92be800640 1 -- start start 2024-04-24T08:09:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.070+0000 7f92be800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92b80b3e90 0x7f92b8150b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.070+0000 7f92be800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 0x7f92b80b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.070+0000 7f92be800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92b815c820 0x7f92b80b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.070+0000 7f92be800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f92b80b4b10 con 0x7f92b80b46a0 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.070+0000 7f92be800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f92b806f2b0 con 0x7f92b80b3e90 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.070+0000 7f92be800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f92b806fee0 con 0x7f92b815c820 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.070+0000 7f92aea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 0x7f92b80b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.070+0000 7f92aea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 0x7f92b80b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34516/0 (socket says 172.21.15.69:34516) 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92aea00640 1 -- 172.21.15.69:0/2666284152 learned_addr learned my addr 172.21.15.69:0/2666284152 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92b7400640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92b80b3e90 0x7f92b8150b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92b7e00640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92b815c820 0x7f92b80b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92aea00640 1 -- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92b815c820 msgr2=0x7f92b80b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92aea00640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92b815c820 0x7f92b80b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92aea00640 1 -- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92b80b3e90 msgr2=0x7f92b8150b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92aea00640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92b80b3e90 0x7f92b8150b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92aea00640 1 -- 172.21.15.69:0/2666284152 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b8151050 con 0x7f92b80b46a0 2024-04-24T08:09:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92b7400640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92b80b3e90 0x7f92b8150b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92b7e00640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92b815c820 0x7f92b80b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:07.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92aea00640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 0x7f92b80b0860 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f92a000c430 tx=0x7f92a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.071+0000 7f92afe00640 1 -- 172.21.15.69:0/2666284152 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0011790 con 0x7f92b80b46a0 2024-04-24T08:09:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.072+0000 7f92afe00640 1 -- 172.21.15.69:0/2666284152 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a0017070 con 0x7f92b80b46a0 2024-04-24T08:09:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.072+0000 7f92be800640 1 -- 172.21.15.69:0/2666284152 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b81f2fa0 con 0x7f92b80b46a0 2024-04-24T08:09:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.072+0000 7f92afe00640 1 -- 172.21.15.69:0/2666284152 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0011b70 con 0x7f92b80b46a0 2024-04-24T08:09:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.072+0000 7f92be800640 1 -- 172.21.15.69:0/2666284152 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b81f3560 con 0x7f92b80b46a0 2024-04-24T08:09:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.073+0000 7f92be800640 1 -- 172.21.15.69:0/2666284152 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92b8068a80 con 0x7f92b80b46a0 2024-04-24T08:09:07.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.074+0000 7f92afe00640 1 -- 172.21.15.69:0/2666284152 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f92a0011d10 con 0x7f92b80b46a0 2024-04-24T08:09:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.075+0000 7f92afe00640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f929406c8f0 0x7f929406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.075+0000 7f92afe00640 1 -- 172.21.15.69:0/2666284152 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f92a008ba40 con 0x7f92b80b46a0 2024-04-24T08:09:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.075+0000 7f92b7400640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f929406c8f0 0x7f929406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.076+0000 7f92b7400640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f929406c8f0 0x7f929406edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f92a8000f60 tx=0x7f92a8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:07.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.079+0000 7f92afe00640 1 -- 172.21.15.69:0/2666284152 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f92a0059dd0 con 0x7f92b80b46a0 2024-04-24T08:09:07.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.280+0000 7f92be800640 1 -- 172.21.15.69:0/2666284152 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f92b80cddc0 con 0x7f92b80b46a0 2024-04-24T08:09:07.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.280+0000 7f92afe00640 1 -- 172.21.15.69:0/2666284152 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f92a00596d0 con 0x7f92b80b46a0 2024-04-24T08:09:07.281 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 -- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f929406c8f0 msgr2=0x7f929406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f929406c8f0 0x7f929406edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f92a8000f60 tx=0x7f92a8009040 comp rx=0 tx=0).stop 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 -- 172.21.15.69:0/2666284152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 msgr2=0x7f92b80b0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 0x7f92b80b0860 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f92a000c430 tx=0x7f92a000c8f0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 -- 172.21.15.69:0/2666284152 shutdown_connections 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f929406c8f0 0x7f929406edb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92b80b46a0 0x7f92b80b0860 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92b815c820 0x7f92b80b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 --2- 172.21.15.69:0/2666284152 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92b80b3e90 0x7f92b8150b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 -- 172.21.15.69:0/2666284152 >> 172.21.15.69:0/2666284152 conn(0x7f92b80d0350 msgr2=0x7f92b80cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 -- 172.21.15.69:0/2666284152 shutdown_connections 2024-04-24T08:09:07.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.283+0000 7f92be800640 1 -- 172.21.15.69:0/2666284152 wait complete. 2024-04-24T08:09:07.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:09:07.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.498+0000 7f407de00640 1 Processor -- start 2024-04-24T08:09:07.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.498+0000 7f407de00640 1 -- start start 2024-04-24T08:09:07.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.500+0000 7f407de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4078156c60 0x7f407815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.500+0000 7f407de00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f40780b30f0 0x7f407815d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.500+0000 7f407de00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f407815d660 con 0x7f40780ad0d0 2024-04-24T08:09:07.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.500+0000 7f407de00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f407806dc30 con 0x7f40780b30f0 2024-04-24T08:09:07.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.500+0000 7f407de00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f407806df10 con 0x7f4078156c60 2024-04-24T08:09:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.500+0000 7f407d400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f40780ad0d0 0x7f40781600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54940/0 (socket says 172.21.15.69:54940) 2024-04-24T08:09:07.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.500+0000 7f407d400640 1 -- 172.21.15.69:0/4020899973 learned_addr learned my addr 172.21.15.69:0/4020899973 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:07.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077e00640 1 --2- 172.21.15.69:0/4020899973 >> v2:172.21.15.121:3300/0 conn(0x7f40780b30f0 0x7f407815d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f407ca00640 1 --2- 172.21.15.69:0/4020899973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4078156c60 0x7f407815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 72724439 0 0) 0x7f407815d660 con 0x7f40780ad0d0 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4060003520 con 0x7f40780ad0d0 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3870076444 0 0) 0x7f4060003520 con 0x7f40780ad0d0 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4078156c60 msgr2=0x7f407815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 --2- 172.21.15.69:0/4020899973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4078156c60 0x7f407815cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 >> v2:172.21.15.121:3300/0 conn(0x7f40780b30f0 msgr2=0x7f407815d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 --2- 172.21.15.69:0/4020899973 >> v2:172.21.15.121:3300/0 conn(0x7f40780b30f0 0x7f407815d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407806e1f0 con 0x7f40780ad0d0 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f40700019f0 con 0x7f40780ad0d0 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 >> v1:172.21.15.69:6789/0 conn(0x7f40780ad0d0 legacy=0x7f40781600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 --2- 172.21.15.69:0/4020899973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 0x7f4060003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 --2- 172.21.15.69:0/4020899973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4078156c60 0x7f4060004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 --2- 172.21.15.69:0/4020899973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4060004d40 0x7f40600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f407806dc30 con 0x7f40600039b0 2024-04-24T08:09:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f407806df10 con 0x7f4060004d40 2024-04-24T08:09:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f406000b9c0 con 0x7f4078156c60 2024-04-24T08:09:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.501+0000 7f407ca00640 1 --2- 172.21.15.69:0/4020899973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 0x7f4060003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.502+0000 7f407ca00640 1 -- 172.21.15.69:0/4020899973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4078156c60 msgr2=0x7f4060004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.502+0000 7f407ca00640 1 --2- 172.21.15.69:0/4020899973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4078156c60 0x7f4060004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.502+0000 7f407ca00640 1 -- 172.21.15.69:0/4020899973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4060004d40 msgr2=0x7f40600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.502+0000 7f407ca00640 1 --2- 172.21.15.69:0/4020899973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4060004d40 0x7f40600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.502+0000 7f407ca00640 1 -- 172.21.15.69:0/4020899973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4060003520 con 0x7f40600039b0 2024-04-24T08:09:07.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.503+0000 7f407ca00640 1 --2- 172.21.15.69:0/4020899973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 0x7f4060003dd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f406c009ac0 tx=0x7f4060005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603b0ccf08a14bca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:07.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.503+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c036070 con 0x7f40600039b0 2024-04-24T08:09:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.503+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c031400 con 0x7f40600039b0 2024-04-24T08:09:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.503+0000 7f4077400640 1 -- 172.21.15.69:0/4020899973 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c009690 con 0x7f40600039b0 2024-04-24T08:09:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.504+0000 7f407de00640 1 -- 172.21.15.69:0/4020899973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 msgr2=0x7f4060003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.504+0000 7f407de00640 1 --2- 172.21.15.69:0/4020899973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 0x7f4060003dd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f406c009ac0 tx=0x7f4060005830 comp rx=0 tx=0).stop 2024-04-24T08:09:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.504+0000 7f407d400640 1 -- 172.21.15.69:0/4020899973 reap_dead start 2024-04-24T08:09:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.504+0000 7f407de00640 1 -- 172.21.15.69:0/4020899973 shutdown_connections 2024-04-24T08:09:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.504+0000 7f407de00640 1 -- 172.21.15.69:0/4020899973 >> 172.21.15.69:0/4020899973 conn(0x7f40780d0160 msgr2=0x7f40780d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.504+0000 7f407de00640 1 -- 172.21.15.69:0/4020899973 shutdown_connections 2024-04-24T08:09:07.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.504+0000 7f407de00640 1 -- 172.21.15.69:0/4020899973 wait complete. 2024-04-24T08:09:07.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.506+0000 7f407de00640 1 Processor -- start 2024-04-24T08:09:07.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.506+0000 7f407de00640 1 -- start start 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 0x7f40780736d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40780b18a0 0x7f40781f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407de00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40780c9d80 0x7f40781550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407de00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f407815cac0 con 0x7f40600039b0 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407de00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f407815d940 con 0x7f40780c9d80 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407de00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f407815cc40 con 0x7f40780b18a0 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 0x7f40780736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 0x7f40780736d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34544/0 (socket says 172.21.15.69:34544) 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407ca00640 1 -- 172.21.15.69:0/2793890972 learned_addr learned my addr 172.21.15.69:0/2793890972 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407ca00640 1 -- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40780b18a0 msgr2=0x7f40781f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407ca00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40780b18a0 0x7f40781f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407ca00640 1 -- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40780c9d80 msgr2=0x7f40781550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407ca00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40780c9d80 0x7f40781550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.507+0000 7f407ca00640 1 -- 172.21.15.69:0/2793890972 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4078073c10 con 0x7f40600039b0 2024-04-24T08:09:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.508+0000 7f407ca00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 0x7f40780736d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f406c009be0 tx=0x7f406c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:07.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.508+0000 7f4074c00640 1 -- 172.21.15.69:0/2793890972 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c040020 con 0x7f40600039b0 2024-04-24T08:09:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.508+0000 7f4074c00640 1 -- 172.21.15.69:0/2793890972 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f406c003a40 con 0x7f40600039b0 2024-04-24T08:09:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.508+0000 7f407de00640 1 -- 172.21.15.69:0/2793890972 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40780c97a0 con 0x7f40600039b0 2024-04-24T08:09:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.508+0000 7f4074c00640 1 -- 172.21.15.69:0/2793890972 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c036040 con 0x7f40600039b0 2024-04-24T08:09:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.508+0000 7f407de00640 1 -- 172.21.15.69:0/2793890972 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4078072ed0 con 0x7f40600039b0 2024-04-24T08:09:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.509+0000 7f407de00640 1 -- 172.21.15.69:0/2793890972 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4078075d50 con 0x7f40600039b0 2024-04-24T08:09:07.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.510+0000 7f4074c00640 1 -- 172.21.15.69:0/2793890972 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f406c043020 con 0x7f40600039b0 2024-04-24T08:09:07.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.511+0000 7f4074c00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f405c06c920 0x7f405c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.511+0000 7f4074c00640 1 -- 172.21.15.69:0/2793890972 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f406c0ab810 con 0x7f40600039b0 2024-04-24T08:09:07.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.512+0000 7f4077e00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f405c06c920 0x7f405c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.513+0000 7f4077e00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f405c06c920 0x7f405c06ede0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f40680093d0 tx=0x7f4068009300 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:07.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.515+0000 7f4074c00640 1 -- 172.21.15.69:0/2793890972 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4078073c10 con 0x7f40600039b0 2024-04-24T08:09:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.731+0000 7f407de00640 1 -- 172.21.15.69:0/2793890972 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f407806f9e0 con 0x7f40600039b0 2024-04-24T08:09:07.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.731+0000 7f4074c00640 1 -- 172.21.15.69:0/2793890972 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f406c079b10 con 0x7f40600039b0 2024-04-24T08:09:07.732 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 -- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f405c06c920 msgr2=0x7f405c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f405c06c920 0x7f405c06ede0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f40680093d0 tx=0x7f4068009300 comp rx=0 tx=0).stop 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 -- 172.21.15.69:0/2793890972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 msgr2=0x7f40780736d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 0x7f40780736d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f406c009be0 tx=0x7f406c004730 comp rx=0 tx=0).stop 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 -- 172.21.15.69:0/2793890972 shutdown_connections 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f405c06c920 0x7f405c06ede0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40780c9d80 0x7f40781550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40780b18a0 0x7f40781f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 --2- 172.21.15.69:0/2793890972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40600039b0 0x7f40780736d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 -- 172.21.15.69:0/2793890972 >> 172.21.15.69:0/2793890972 conn(0x7f40780d0160 msgr2=0x7f40780b0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 -- 172.21.15.69:0/2793890972 shutdown_connections 2024-04-24T08:09:07.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:07.734+0000 7f407de00640 1 -- 172.21.15.69:0/2793890972 wait complete. 2024-04-24T08:09:07.752 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:09:07.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:07.752 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:09:07.832 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:09:07.832 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:09:07.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:07.833 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:09:07.894 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:09:07.894 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:07.951 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:09:07.951 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:08.001 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:08.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.210+0000 7fcaab200640 1 Processor -- start 2024-04-24T08:09:08.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.211+0000 7fcaab200640 1 -- start start 2024-04-24T08:09:08.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.212+0000 7fcaab200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcaac15e960 0x7fcaac162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.212+0000 7fcaab200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac162980 0x7fcaac06cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.212+0000 7fcaab200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaac15ed40 con 0x7fcaac06d4f0 2024-04-24T08:09:08.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.212+0000 7fcaab200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcaac0cf050 con 0x7fcaac15e960 2024-04-24T08:09:08.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.212+0000 7fcaab200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcaac162e20 con 0x7fcaac162980 2024-04-24T08:09:08.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.212+0000 7fcaaa800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcaac06d4f0 0x7fcaac0cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54956/0 (socket says 172.21.15.69:54956) 2024-04-24T08:09:08.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.212+0000 7fcaaa800640 1 -- 172.21.15.69:0/2840425669 learned_addr learned my addr 172.21.15.69:0/2840425669 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:08.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa1400640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac162980 0x7fcaac06cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/2840425669 >> v2:172.21.15.121:3300/0 conn(0x7fcaac15e960 0x7fcaac162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2735612853 0 0) 0x7fcaac15ed40 con 0x7fcaac06d4f0 2024-04-24T08:09:08.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca90003520 con 0x7fcaac06d4f0 2024-04-24T08:09:08.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1442271150 0 0) 0x7fca90003520 con 0x7fcaac06d4f0 2024-04-24T08:09:08.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac162980 msgr2=0x7fcaac06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:08.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac162980 0x7fcaac06cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 >> v2:172.21.15.121:3300/0 conn(0x7fcaac15e960 msgr2=0x7fcaac162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:08.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 --2- 172.21.15.69:0/2840425669 >> v2:172.21.15.121:3300/0 conn(0x7fcaac15e960 0x7fcaac162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaac163100 con 0x7fcaac06d4f0 2024-04-24T08:09:08.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fcaa40019f0 con 0x7fcaac06d4f0 2024-04-24T08:09:08.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 >> v1:172.21.15.69:6789/0 conn(0x7fcaac06d4f0 legacy=0x7fcaac0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 0x7fca90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca90004800 0x7fca90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac162980 0x7fca900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcaac0cf050 con 0x7fca900039b0 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcaac162e20 con 0x7fca90004800 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.213+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fca9000ba10 con 0x7fcaac162980 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.214+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 0x7fca90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.214+0000 7fcaaa800640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac162980 0x7fca900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.214+0000 7fcaa1400640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca90004800 0x7fca90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.214+0000 7fcaa9e00640 1 -- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac162980 msgr2=0x7fca900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.214+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac162980 0x7fca900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.214+0000 7fcaa9e00640 1 -- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca90004800 msgr2=0x7fca90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.214+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca90004800 0x7fca90004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.214+0000 7fcaa9e00640 1 -- 172.21.15.69:0/2840425669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca90003520 con 0x7fca900039b0 2024-04-24T08:09:08.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.214+0000 7fcaa1400640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca90004800 0x7fca90004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:08.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.216+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 0x7fca90003dd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00ba30 tx=0x7fca90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d8faeb671f24101 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:08.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.216+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9c030070 con 0x7fca900039b0 2024-04-24T08:09:08.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.216+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca9c02ae10 con 0x7fca900039b0 2024-04-24T08:09:08.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.216+0000 7fcaa9400640 1 -- 172.21.15.69:0/2840425669 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9c00b600 con 0x7fca900039b0 2024-04-24T08:09:08.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.216+0000 7fcaab200640 1 -- 172.21.15.69:0/2840425669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 msgr2=0x7fca90003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.216+0000 7fcaab200640 1 --2- 172.21.15.69:0/2840425669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 0x7fca90003dd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00ba30 tx=0x7fca90005830 comp rx=0 tx=0).stop 2024-04-24T08:09:08.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.216+0000 7fcaaa800640 1 -- 172.21.15.69:0/2840425669 reap_dead start 2024-04-24T08:09:08.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.216+0000 7fcaab200640 1 -- 172.21.15.69:0/2840425669 shutdown_connections 2024-04-24T08:09:08.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.216+0000 7fcaab200640 1 -- 172.21.15.69:0/2840425669 >> 172.21.15.69:0/2840425669 conn(0x7fcaac09d900 msgr2=0x7fcaac08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:08.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.217+0000 7fcaab200640 1 -- 172.21.15.69:0/2840425669 shutdown_connections 2024-04-24T08:09:08.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.217+0000 7fcaab200640 1 -- 172.21.15.69:0/2840425669 wait complete. 2024-04-24T08:09:08.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.218+0000 7fcaab200640 1 Processor -- start 2024-04-24T08:09:08.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.218+0000 7fcaab200640 1 -- start start 2024-04-24T08:09:08.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.219+0000 7fcaab200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 0x7fcaac0b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.219+0000 7fcaab200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac15f7e0 0x7fcaac1f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.219+0000 7fcaab200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaac0b22e0 0x7fcaac0b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.219+0000 7fcaab200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcaac0b6170 con 0x7fca900039b0 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.219+0000 7fcaab200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcaac0cf5d0 con 0x7fcaac0b22e0 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.219+0000 7fcaab200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcaac0b62f0 con 0x7fcaac15f7e0 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.220+0000 7fcaa9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 0x7fcaac0b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.220+0000 7fcaa9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 0x7fcaac0b71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34566/0 (socket says 172.21.15.69:34566) 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.220+0000 7fcaa9e00640 1 -- 172.21.15.69:0/1183562353 learned_addr learned my addr 172.21.15.69:0/1183562353 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.220+0000 7fcaa9e00640 1 -- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac15f7e0 msgr2=0x7fcaac1f3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.220+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac15f7e0 0x7fcaac1f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.220+0000 7fcaa9e00640 1 -- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaac0b22e0 msgr2=0x7fcaac0b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.220+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaac0b22e0 0x7fcaac0b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.220+0000 7fcaa9e00640 1 -- 172.21.15.69:0/1183562353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaac0b7a40 con 0x7fca900039b0 2024-04-24T08:09:08.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.220+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 0x7fcaac0b71f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fca9c02a900 tx=0x7fca9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:08.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.220+0000 7fcaa2a00640 1 -- 172.21.15.69:0/1183562353 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9c042030 con 0x7fca900039b0 2024-04-24T08:09:08.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.221+0000 7fcaa2a00640 1 -- 172.21.15.69:0/1183562353 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca9c02ac10 con 0x7fca900039b0 2024-04-24T08:09:08.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.221+0000 7fcaa2a00640 1 -- 172.21.15.69:0/1183562353 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9c00b600 con 0x7fca900039b0 2024-04-24T08:09:08.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.221+0000 7fcaab200640 1 -- 172.21.15.69:0/1183562353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaac0b7d20 con 0x7fca900039b0 2024-04-24T08:09:08.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.221+0000 7fcaab200640 1 -- 172.21.15.69:0/1183562353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaac0b82e0 con 0x7fca900039b0 2024-04-24T08:09:08.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.222+0000 7fcaab200640 1 -- 172.21.15.69:0/1183562353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaac075d50 con 0x7fca900039b0 2024-04-24T08:09:08.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.223+0000 7fcaa2a00640 1 -- 172.21.15.69:0/1183562353 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fca9c045070 con 0x7fca900039b0 2024-04-24T08:09:08.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.224+0000 7fcaa2a00640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c8f0 0x7fca8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.224+0000 7fcaa2a00640 1 -- 172.21.15.69:0/1183562353 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fca9c0ac3d0 con 0x7fca900039b0 2024-04-24T08:09:08.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.225+0000 7fcaa1400640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c8f0 0x7fca8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.226+0000 7fcaa1400640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c8f0 0x7fca8806edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcaac15c770 tx=0x7fca94008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:08.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.228+0000 7fcaa2a00640 1 -- 172.21.15.69:0/1183562353 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca9c0463b0 con 0x7fca900039b0 2024-04-24T08:09:08.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.426+0000 7fcaab200640 1 -- 172.21.15.69:0/1183562353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcaac0d1dc0 con 0x7fca900039b0 2024-04-24T08:09:08.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.428+0000 7fcaa2a00640 1 -- 172.21.15.69:0/1183562353 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7fca9c07a640 con 0x7fca900039b0 2024-04-24T08:09:08.430 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:08.430 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:08.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 -- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c8f0 msgr2=0x7fca8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c8f0 0x7fca8806edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcaac15c770 tx=0x7fca94008040 comp rx=0 tx=0).stop 2024-04-24T08:09:08.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 -- 172.21.15.69:0/1183562353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 msgr2=0x7fcaac0b71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 0x7fcaac0b71f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fca9c02a900 tx=0x7fca9c004730 comp rx=0 tx=0).stop 2024-04-24T08:09:08.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 -- 172.21.15.69:0/1183562353 shutdown_connections 2024-04-24T08:09:08.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c8f0 0x7fca8806edb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaac0b22e0 0x7fcaac0b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac15f7e0 0x7fcaac1f3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 --2- 172.21.15.69:0/1183562353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca900039b0 0x7fcaac0b71f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 -- 172.21.15.69:0/1183562353 >> 172.21.15.69:0/1183562353 conn(0x7fcaac09d900 msgr2=0x7fcaac0d23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:08.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 -- 172.21.15.69:0/1183562353 shutdown_connections 2024-04-24T08:09:08.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.431+0000 7fcaab200640 1 -- 172.21.15.69:0/1183562353 wait complete. 2024-04-24T08:09:08.451 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:08.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:08.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.652+0000 7f4606c00640 1 Processor -- start 2024-04-24T08:09:08.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.653+0000 7f4606c00640 1 -- start start 2024-04-24T08:09:08.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.654+0000 7f4606c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f460015c820 0x7f46000b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.654+0000 7f4606c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 0x7f46000ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.654+0000 7f4606c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f460015cc00 con 0x7f46000b46a0 2024-04-24T08:09:08.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.654+0000 7f4606c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f460006ed30 con 0x7f460015c820 2024-04-24T08:09:08.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.654+0000 7f4606c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f46000b4e10 con 0x7f46000ce250 2024-04-24T08:09:08.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.655+0000 7f45ffe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f460015c820 0x7f46000b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.655+0000 7f45ffe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f460015c820 0x7f46000b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:33408/0 (socket says 172.21.15.69:33408) 2024-04-24T08:09:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.655+0000 7f45ffe00640 1 -- 172.21.15.69:0/3739082309 learned_addr learned my addr 172.21.15.69:0/3739082309 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.655+0000 7f4604c00640 1 --2- 172.21.15.69:0/3739082309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 0x7f46000ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.656+0000 7f45ffe00640 1 -- 172.21.15.69:0/3739082309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 msgr2=0x7f46000ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.656+0000 7f45ffe00640 1 --2- 172.21.15.69:0/3739082309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 0x7f46000ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.656+0000 7f45ffe00640 1 -- 172.21.15.69:0/3739082309 >> v1:172.21.15.121:6789/0 conn(0x7f46000b46a0 legacy=0x7f460006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.656+0000 7f45ffe00640 1 -- 172.21.15.69:0/3739082309 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46000b50f0 con 0x7f460015c820 2024-04-24T08:09:08.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.657+0000 7f45ffe00640 1 --2- 172.21.15.69:0/3739082309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460015c820 0x7f46000b3e90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009870 tx=0x7f45f002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d720e408043d20f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:08.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.657+0000 7f45ff400640 1 -- 172.21.15.69:0/3739082309 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0037070 con 0x7f460015c820 2024-04-24T08:09:08.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.658+0000 7f45ff400640 1 -- 172.21.15.69:0/3739082309 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45f0032440 con 0x7f460015c820 2024-04-24T08:09:08.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.658+0000 7f45ff400640 1 -- 172.21.15.69:0/3739082309 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0032810 con 0x7f460015c820 2024-04-24T08:09:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.658+0000 7f4606c00640 1 -- 172.21.15.69:0/3739082309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460015c820 msgr2=0x7f46000b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.658+0000 7f4606c00640 1 --2- 172.21.15.69:0/3739082309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460015c820 0x7f46000b3e90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009870 tx=0x7f45f002a270 comp rx=0 tx=0).stop 2024-04-24T08:09:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.658+0000 7f4606c00640 1 -- 172.21.15.69:0/3739082309 shutdown_connections 2024-04-24T08:09:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.658+0000 7f4606c00640 1 --2- 172.21.15.69:0/3739082309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 0x7f46000ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.658+0000 7f4606c00640 1 --2- 172.21.15.69:0/3739082309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460015c820 0x7f46000b3e90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.658+0000 7f4606c00640 1 -- 172.21.15.69:0/3739082309 >> 172.21.15.69:0/3739082309 conn(0x7f46000d0350 msgr2=0x7f46000d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:08.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.658+0000 7f4606c00640 1 -- 172.21.15.69:0/3739082309 shutdown_connections 2024-04-24T08:09:08.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.658+0000 7f4606c00640 1 -- 172.21.15.69:0/3739082309 wait complete. 2024-04-24T08:09:08.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.660+0000 7f4606c00640 1 Processor -- start 2024-04-24T08:09:08.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.661+0000 7f4606c00640 1 -- start start 2024-04-24T08:09:08.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.662+0000 7f4606c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000b46a0 0x7f460015f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.662+0000 7f4606c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 0x7f460015aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.662+0000 7f4606c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460015c820 0x7f460014a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.662+0000 7f4606c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f46000b4b30 con 0x7f46000b46a0 2024-04-24T08:09:08.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.662+0000 7f4606c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f460006f2b0 con 0x7f460015c820 2024-04-24T08:09:08.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.662+0000 7f4606c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f460006fee0 con 0x7f46000ce250 2024-04-24T08:09:08.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.662+0000 7f45ffe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000b46a0 0x7f460015f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.662+0000 7f45ffe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000b46a0 0x7f460015f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34580/0 (socket says 172.21.15.69:34580) 2024-04-24T08:09:08.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.662+0000 7f45ffe00640 1 -- 172.21.15.69:0/1012783257 learned_addr learned my addr 172.21.15.69:0/1012783257 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:08.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.662+0000 7f45f7e00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 0x7f460015aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.663+0000 7f4604c00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460015c820 0x7f460014a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.663+0000 7f45ffe00640 1 -- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 msgr2=0x7f460015aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.663+0000 7f45ffe00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 0x7f460015aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.663+0000 7f45ffe00640 1 -- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460015c820 msgr2=0x7f460014a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.663+0000 7f45ffe00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460015c820 0x7f460014a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.663+0000 7f45ffe00640 1 -- 172.21.15.69:0/1012783257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460015b530 con 0x7f46000b46a0 2024-04-24T08:09:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.663+0000 7f45f7e00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 0x7f460015aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:09:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.663+0000 7f45ffe00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000b46a0 0x7f460015f060 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f45f0000c00 tx=0x7f45f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.664+0000 7f45fcc00640 1 -- 172.21.15.69:0/1012783257 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0037040 con 0x7f46000b46a0 2024-04-24T08:09:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.664+0000 7f45fcc00640 1 -- 172.21.15.69:0/1012783257 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45f0042070 con 0x7f46000b46a0 2024-04-24T08:09:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.664+0000 7f45fcc00640 1 -- 172.21.15.69:0/1012783257 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0030880 con 0x7f46000b46a0 2024-04-24T08:09:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.664+0000 7f4606c00640 1 -- 172.21.15.69:0/1012783257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46001e6bd0 con 0x7f46000b46a0 2024-04-24T08:09:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.664+0000 7f4606c00640 1 -- 172.21.15.69:0/1012783257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46001e7190 con 0x7f46000b46a0 2024-04-24T08:09:08.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.665+0000 7f4606c00640 1 -- 172.21.15.69:0/1012783257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4600068a80 con 0x7f46000b46a0 2024-04-24T08:09:08.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.666+0000 7f45fcc00640 1 -- 172.21.15.69:0/1012783257 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f45f0030a20 con 0x7f46000b46a0 2024-04-24T08:09:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.667+0000 7f45fcc00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45dc06c920 0x7f45dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.667+0000 7f45fcc00640 1 -- 172.21.15.69:0/1012783257 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f45f00ac440 con 0x7f46000b46a0 2024-04-24T08:09:08.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.668+0000 7f45f7e00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45dc06c920 0x7f45dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:08.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.669+0000 7f45f7e00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45dc06c920 0x7f45dc06ede0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f460006c750 tx=0x7f45e8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:08.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.671+0000 7f45fcc00640 1 -- 172.21.15.69:0/1012783257 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f460015b530 con 0x7f46000b46a0 2024-04-24T08:09:08.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.872+0000 7f4606c00640 1 -- 172.21.15.69:0/1012783257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4600074d60 con 0x7f46000b46a0 2024-04-24T08:09:08.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.872+0000 7f45fcc00640 1 -- 172.21.15.69:0/1012783257 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f45f007a7d0 con 0x7f46000b46a0 2024-04-24T08:09:08.874 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:08.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 -- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45dc06c920 msgr2=0x7f45dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45dc06c920 0x7f45dc06ede0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f460006c750 tx=0x7f45e8009040 comp rx=0 tx=0).stop 2024-04-24T08:09:08.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 -- 172.21.15.69:0/1012783257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000b46a0 msgr2=0x7f460015f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:08.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000b46a0 0x7f460015f060 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f45f0000c00 tx=0x7f45f00046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 -- 172.21.15.69:0/1012783257 shutdown_connections 2024-04-24T08:09:08.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45dc06c920 0x7f45dc06ede0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460015c820 0x7f460014a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000ce250 0x7f460015aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 --2- 172.21.15.69:0/1012783257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000b46a0 0x7f460015f060 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:08.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 -- 172.21.15.69:0/1012783257 >> 172.21.15.69:0/1012783257 conn(0x7f46000d0350 msgr2=0x7f46000af250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:08.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 -- 172.21.15.69:0/1012783257 shutdown_connections 2024-04-24T08:09:08.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:08.875+0000 7f4606c00640 1 -- 172.21.15.69:0/1012783257 wait complete. 2024-04-24T08:09:08.897 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:09.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.101+0000 7f5cd1a00640 1 Processor -- start 2024-04-24T08:09:09.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.101+0000 7f5cd1a00640 1 -- start start 2024-04-24T08:09:09.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5cd1a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5ccc06dd90 0x7f5ccc15f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5cd1a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ccc0d02f0 0x7f5ccc071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5cd1a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ccc06e220 con 0x7f5ccc15e050 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5cd1a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5ccc15f660 con 0x7f5ccc06dd90 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5cd1a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5ccc15e430 con 0x7f5ccc0d02f0 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5ccbe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5ccc15e050 0x7f5ccc06f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55446/0 (socket says 172.21.15.69:55446) 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5ccbe00640 1 -- 172.21.15.69:0/2449245126 learned_addr learned my addr 172.21.15.69:0/2449245126 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5ccb400640 1 --2- 172.21.15.69:0/2449245126 >> v2:172.21.15.69:3300/0 conn(0x7f5ccc06dd90 0x7f5ccc15f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5ccaa00640 1 -- 172.21.15.69:0/2449245126 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553407769 0 0) 0x7f5ccc06e220 con 0x7f5ccc15e050 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5cd1000640 1 --2- 172.21.15.69:0/2449245126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ccc0d02f0 0x7f5ccc071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.103+0000 7f5ccaa00640 1 -- 172.21.15.69:0/2449245126 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cb0003520 con 0x7f5ccc15e050 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.104+0000 7f5ccb400640 1 -- 172.21.15.69:0/2449245126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ccc0d02f0 msgr2=0x7f5ccc071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.104+0000 7f5ccb400640 1 --2- 172.21.15.69:0/2449245126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ccc0d02f0 0x7f5ccc071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.104+0000 7f5ccb400640 1 -- 172.21.15.69:0/2449245126 >> v1:172.21.15.69:6789/0 conn(0x7f5ccc15e050 legacy=0x7f5ccc06f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.104+0000 7f5ccaa00640 1 -- 172.21.15.69:0/2449245126 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3212050680 0 0) 0x7f5cb0003520 con 0x7f5ccc15e050 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.104+0000 7f5ccb400640 1 -- 172.21.15.69:0/2449245126 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ccc06e490 con 0x7f5ccc06dd90 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.105+0000 7f5ccb400640 1 --2- 172.21.15.69:0/2449245126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc06dd90 0x7f5ccc15f120 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80097a0 tx=0x7f5cb802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9968aa0be747b45f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.105+0000 7f5ccaa00640 1 -- 172.21.15.69:0/2449245126 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8037070 con 0x7f5ccc06dd90 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.105+0000 7f5ccaa00640 1 -- 172.21.15.69:0/2449245126 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cb802ab10 con 0x7f5ccc06dd90 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.105+0000 7f5ccaa00640 1 -- 172.21.15.69:0/2449245126 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc001670 con 0x7f5ccc06dd90 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.106+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2449245126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc06dd90 msgr2=0x7f5ccc15f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.106+0000 7f5cd1a00640 1 --2- 172.21.15.69:0/2449245126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc06dd90 0x7f5ccc15f120 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80097a0 tx=0x7f5cb802a080 comp rx=0 tx=0).stop 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.106+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2449245126 shutdown_connections 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.106+0000 7f5cd1a00640 1 --2- 172.21.15.69:0/2449245126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ccc0d02f0 0x7f5ccc071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.106+0000 7f5cd1a00640 1 --2- 172.21.15.69:0/2449245126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc06dd90 0x7f5ccc15f120 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.106+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2449245126 >> 172.21.15.69:0/2449245126 conn(0x7f5ccc067b80 msgr2=0x7f5ccc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.106+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2449245126 shutdown_connections 2024-04-24T08:09:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.106+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2449245126 wait complete. 2024-04-24T08:09:09.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.108+0000 7f5cd1a00640 1 Processor -- start 2024-04-24T08:09:09.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.108+0000 7f5cd1a00640 1 -- start start 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5cd1a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ccc06dd90 0x7f5ccc0cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5cd1a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ccc0d02f0 0x7f5ccc14b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5cd1a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc15e050 0x7f5ccc141f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5cd1a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc06ed30 con 0x7f5ccc15e050 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5cd1a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5ccc06ea50 con 0x7f5ccc0d02f0 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5cd1a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5ccc06f010 con 0x7f5ccc06dd90 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5cd1000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc15e050 0x7f5ccc141f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5cd1000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc15e050 0x7f5ccc141f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35868/0 (socket says 172.21.15.69:35868) 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5cd1000640 1 -- 172.21.15.69:0/2168510807 learned_addr learned my addr 172.21.15.69:0/2168510807 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5ccbe00640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ccc06dd90 0x7f5ccc0cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.109+0000 7f5ccb400640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ccc0d02f0 0x7f5ccc14b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cd1000640 1 -- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ccc06dd90 msgr2=0x7f5ccc0cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cd1000640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ccc06dd90 0x7f5ccc0cf2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cd1000640 1 -- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ccc0d02f0 msgr2=0x7f5ccc14b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cd1000640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ccc0d02f0 0x7f5ccc14b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cd1000640 1 -- 172.21.15.69:0/2168510807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ccc14bd80 con 0x7f5ccc15e050 2024-04-24T08:09:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cd1000640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc15e050 0x7f5ccc141f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4007e40 tx=0x7f5cc400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cc3e00640 1 -- 172.21.15.69:0/2168510807 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4014620 con 0x7f5ccc15e050 2024-04-24T08:09:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cc3e00640 1 -- 172.21.15.69:0/2168510807 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc4019070 con 0x7f5ccc15e050 2024-04-24T08:09:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2168510807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ccc14ed10 con 0x7f5ccc15e050 2024-04-24T08:09:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cc3e00640 1 -- 172.21.15.69:0/2168510807 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4014a20 con 0x7f5ccc15e050 2024-04-24T08:09:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.110+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2168510807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ccc14f2d0 con 0x7f5ccc15e050 2024-04-24T08:09:09.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.111+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2168510807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ccc0d0760 con 0x7f5ccc15e050 2024-04-24T08:09:09.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.112+0000 7f5cc3e00640 1 -- 172.21.15.69:0/2168510807 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5cc4014bc0 con 0x7f5ccc15e050 2024-04-24T08:09:09.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.113+0000 7f5cc3e00640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5cac06c8f0 0x7f5cac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.113+0000 7f5cc3e00640 1 -- 172.21.15.69:0/2168510807 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f5cc408c4b0 con 0x7f5ccc15e050 2024-04-24T08:09:09.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.114+0000 7f5ccbe00640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5cac06c8f0 0x7f5cac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.115+0000 7f5ccbe00640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5cac06c8f0 0x7f5cac06edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc005880 tx=0x7f5cbc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:09.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.118+0000 7f5cc3e00640 1 -- 172.21.15.69:0/2168510807 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ccc0d0760 con 0x7f5ccc15e050 2024-04-24T08:09:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.314+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2168510807 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5ccc075d50 con 0x7f5ccc15e050 2024-04-24T08:09:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.315+0000 7f5cc3e00640 1 -- 172.21.15.69:0/2168510807 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f5ccc14bd80 con 0x7f5ccc15e050 2024-04-24T08:09:09.316 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.317+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5cac06c8f0 msgr2=0x7f5cac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.317+0000 7f5cd1a00640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5cac06c8f0 0x7f5cac06edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc005880 tx=0x7f5cbc009040 comp rx=0 tx=0).stop 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.317+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2168510807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc15e050 msgr2=0x7f5ccc141f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.317+0000 7f5cd1a00640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc15e050 0x7f5ccc141f30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4007e40 tx=0x7f5cc400e4b0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.317+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2168510807 shutdown_connections 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.317+0000 7f5cd1a00640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5cac06c8f0 0x7f5cac06edb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.317+0000 7f5cd1a00640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ccc15e050 0x7f5ccc141f30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.317+0000 7f5cd1a00640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ccc0d02f0 0x7f5ccc14b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.317+0000 7f5cd1a00640 1 --2- 172.21.15.69:0/2168510807 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ccc06dd90 0x7f5ccc0cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.317+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2168510807 >> 172.21.15.69:0/2168510807 conn(0x7f5ccc067b80 msgr2=0x7f5ccc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.318+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2168510807 shutdown_connections 2024-04-24T08:09:09.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.318+0000 7f5cd1a00640 1 -- 172.21.15.69:0/2168510807 wait complete. 2024-04-24T08:09:09.339 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:09.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.546+0000 7fe8c1000640 1 Processor -- start 2024-04-24T08:09:09.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.546+0000 7fe8c1000640 1 -- start start 2024-04-24T08:09:09.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.548+0000 7fe8c1000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8bc0b5300 0x7fe8bc0d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.548+0000 7fe8c1000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8bc0d0820 0x7fe8bc06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.548+0000 7fe8c1000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8bc0ce640 con 0x7fe8bc15ea60 2024-04-24T08:09:09.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.548+0000 7fe8c1000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe8bc0d0c90 con 0x7fe8bc0d0820 2024-04-24T08:09:09.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.548+0000 7fe8c1000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe8bc0ce910 con 0x7fe8bc0b5300 2024-04-24T08:09:09.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.549+0000 7fe8bbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8bc0d0820 0x7fe8bc06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.549+0000 7fe8bbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe8bc0d0820 0x7fe8bc06df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35876/0 (socket says 172.21.15.69:35876) 2024-04-24T08:09:09.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.549+0000 7fe8bbe00640 1 -- 172.21.15.69:0/2467856862 learned_addr learned my addr 172.21.15.69:0/2467856862 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:09.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.549+0000 7fe8b2a00640 1 --2- 172.21.15.69:0/2467856862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8bc0b5300 0x7fe8bc0d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.549+0000 7fe8bbe00640 1 -- 172.21.15.69:0/2467856862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8bc0b5300 msgr2=0x7fe8bc0d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:09.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.549+0000 7fe8bbe00640 1 --2- 172.21.15.69:0/2467856862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8bc0b5300 0x7fe8bc0d02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.549+0000 7fe8bbe00640 1 -- 172.21.15.69:0/2467856862 >> v1:172.21.15.121:6789/0 conn(0x7fe8bc15ea60 legacy=0x7fe8bc0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:09.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.549+0000 7fe8bbe00640 1 -- 172.21.15.69:0/2467856862 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc0cebf0 con 0x7fe8bc0d0820 2024-04-24T08:09:09.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.550+0000 7fe8bbe00640 1 --2- 172.21.15.69:0/2467856862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0d0820 0x7fe8bc06df40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009e50 tx=0x7fe8b402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9807bd19ec0b8a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:09.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.551+0000 7fe8baa00640 1 -- 172.21.15.69:0/2467856862 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4036070 con 0x7fe8bc0d0820 2024-04-24T08:09:09.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.551+0000 7fe8baa00640 1 -- 172.21.15.69:0/2467856862 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b4031600 con 0x7fe8bc0d0820 2024-04-24T08:09:09.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.551+0000 7fe8baa00640 1 -- 172.21.15.69:0/2467856862 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b40319d0 con 0x7fe8bc0d0820 2024-04-24T08:09:09.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.552+0000 7fe8c1000640 1 -- 172.21.15.69:0/2467856862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0d0820 msgr2=0x7fe8bc06df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.552+0000 7fe8c1000640 1 --2- 172.21.15.69:0/2467856862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0d0820 0x7fe8bc06df40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe8b4009e50 tx=0x7fe8b402a6c0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.552+0000 7fe8c1000640 1 -- 172.21.15.69:0/2467856862 shutdown_connections 2024-04-24T08:09:09.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.552+0000 7fe8c1000640 1 --2- 172.21.15.69:0/2467856862 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0d0820 0x7fe8bc06df40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.552+0000 7fe8c1000640 1 --2- 172.21.15.69:0/2467856862 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8bc0b5300 0x7fe8bc0d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.552+0000 7fe8c1000640 1 -- 172.21.15.69:0/2467856862 >> 172.21.15.69:0/2467856862 conn(0x7fe8bc067b80 msgr2=0x7fe8bc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:09.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.552+0000 7fe8c1000640 1 -- 172.21.15.69:0/2467856862 shutdown_connections 2024-04-24T08:09:09.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.552+0000 7fe8c1000640 1 -- 172.21.15.69:0/2467856862 wait complete. 2024-04-24T08:09:09.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.554+0000 7fe8c1000640 1 Processor -- start 2024-04-24T08:09:09.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.554+0000 7fe8c1000640 1 -- start start 2024-04-24T08:09:09.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8c1000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0b5300 0x7fe8bc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8c1000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8bc0d0820 0x7fe8bc1f4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8c1000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8bc15ea60 0x7fe8bc1f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8c1000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc0cf490 con 0x7fe8bc0b5300 2024-04-24T08:09:09.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8c1000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc0cf1b0 con 0x7fe8bc0d0820 2024-04-24T08:09:09.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8c1000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe8bc06e480 con 0x7fe8bc15ea60 2024-04-24T08:09:09.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8bb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0b5300 0x7fe8bc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8bb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0b5300 0x7fe8bc0b2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35890/0 (socket says 172.21.15.69:35890) 2024-04-24T08:09:09.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8bb400640 1 -- 172.21.15.69:0/66733288 learned_addr learned my addr 172.21.15.69:0/66733288 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:09.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8bb400640 1 -- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8bc15ea60 msgr2=0x7fe8bc1f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8bb400640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8bc15ea60 0x7fe8bc1f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8bb400640 1 -- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8bc0d0820 msgr2=0x7fe8bc1f4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8bb400640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8bc0d0820 0x7fe8bc1f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.555+0000 7fe8bb400640 1 -- 172.21.15.69:0/66733288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc0b26c0 con 0x7fe8bc0b5300 2024-04-24T08:09:09.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.556+0000 7fe8bb400640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0b5300 0x7fe8bc0b2000 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00b3f0 tx=0x7fe8ac00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:09.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.556+0000 7fe8b3e00640 1 -- 172.21.15.69:0/66733288 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac015020 con 0x7fe8bc0b5300 2024-04-24T08:09:09.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.556+0000 7fe8b3e00640 1 -- 172.21.15.69:0/66733288 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8ac011830 con 0x7fe8bc0b5300 2024-04-24T08:09:09.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.556+0000 7fe8b3e00640 1 -- 172.21.15.69:0/66733288 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac011c20 con 0x7fe8bc0b5300 2024-04-24T08:09:09.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.556+0000 7fe8c1000640 1 -- 172.21.15.69:0/66733288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8bc0b29a0 con 0x7fe8bc0b5300 2024-04-24T08:09:09.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.556+0000 7fe8c1000640 1 -- 172.21.15.69:0/66733288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8bc0b2f60 con 0x7fe8bc0b5300 2024-04-24T08:09:09.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.558+0000 7fe8c1000640 1 -- 172.21.15.69:0/66733288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8bc075d50 con 0x7fe8bc0b5300 2024-04-24T08:09:09.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.558+0000 7fe8b3e00640 1 -- 172.21.15.69:0/66733288 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe8ac018020 con 0x7fe8bc0b5300 2024-04-24T08:09:09.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.560+0000 7fe8b3e00640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe89806c8f0 0x7fe89806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.560+0000 7fe8b3e00640 1 -- 172.21.15.69:0/66733288 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fe8ac08b850 con 0x7fe8bc0b5300 2024-04-24T08:09:09.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.561+0000 7fe8b2a00640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe89806c8f0 0x7fe89806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.562+0000 7fe8b2a00640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe89806c8f0 0x7fe89806edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe8a40098f0 tx=0x7fe8a4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:09.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.565+0000 7fe8b3e00640 1 -- 172.21.15.69:0/66733288 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8ac01f030 con 0x7fe8bc0b5300 2024-04-24T08:09:09.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.765+0000 7fe8c1000640 1 -- 172.21.15.69:0/66733288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe8bc06cd50 con 0x7fe8bc0b5300 2024-04-24T08:09:09.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.765+0000 7fe8b3e00640 1 -- 172.21.15.69:0/66733288 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7fe8ac059b50 con 0x7fe8bc0b5300 2024-04-24T08:09:09.767 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:09.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 -- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe89806c8f0 msgr2=0x7fe89806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe89806c8f0 0x7fe89806edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fe8a40098f0 tx=0x7fe8a4008040 comp rx=0 tx=0).stop 2024-04-24T08:09:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 -- 172.21.15.69:0/66733288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0b5300 msgr2=0x7fe8bc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0b5300 0x7fe8bc0b2000 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00b3f0 tx=0x7fe8ac00b8b0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 -- 172.21.15.69:0/66733288 shutdown_connections 2024-04-24T08:09:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe89806c8f0 0x7fe89806edb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8bc15ea60 0x7fe8bc1f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8bc0d0820 0x7fe8bc1f4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 --2- 172.21.15.69:0/66733288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe8bc0b5300 0x7fe8bc0b2000 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 -- 172.21.15.69:0/66733288 >> 172.21.15.69:0/66733288 conn(0x7fe8bc067b80 msgr2=0x7fe8bc0b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 -- 172.21.15.69:0/66733288 shutdown_connections 2024-04-24T08:09:09.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.768+0000 7fe8c1000640 1 -- 172.21.15.69:0/66733288 wait complete. 2024-04-24T08:09:09.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:09:09.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.982+0000 7f3283a00640 1 Processor -- start 2024-04-24T08:09:09.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.983+0000 7f3283a00640 1 -- start start 2024-04-24T08:09:09.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.984+0000 7f3283a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f327c0d18c0 0x7f327c0cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.984+0000 7f3283a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f327c15ec70 0x7f327c0d0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.984+0000 7f3283a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f327c0d26c0 con 0x7f327c15e890 2024-04-24T08:09:09.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.984+0000 7f3283a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f327c0d32d0 con 0x7f327c0d18c0 2024-04-24T08:09:09.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.984+0000 7f3283a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f327c0d29a0 con 0x7f327c15ec70 2024-04-24T08:09:09.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.985+0000 7f327be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f327c0d18c0 0x7f327c0cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.985+0000 7f327be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f327c0d18c0 0x7f327c0cfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35896/0 (socket says 172.21.15.69:35896) 2024-04-24T08:09:09.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.985+0000 7f327be00640 1 -- 172.21.15.69:0/2504580259 learned_addr learned my addr 172.21.15.69:0/2504580259 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:09.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.985+0000 7f3281a00640 1 --2- 172.21.15.69:0/2504580259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f327c15ec70 0x7f327c0d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.985+0000 7f327be00640 1 -- 172.21.15.69:0/2504580259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f327c15ec70 msgr2=0x7f327c0d0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:09.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.985+0000 7f327be00640 1 --2- 172.21.15.69:0/2504580259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f327c15ec70 0x7f327c0d0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.985+0000 7f327be00640 1 -- 172.21.15.69:0/2504580259 >> v1:172.21.15.121:6789/0 conn(0x7f327c15e890 legacy=0x7f327c0d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:09.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.985+0000 7f327be00640 1 -- 172.21.15.69:0/2504580259 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327c0d2c80 con 0x7f327c0d18c0 2024-04-24T08:09:09.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.986+0000 7f327be00640 1 --2- 172.21.15.69:0/2504580259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0d18c0 0x7f327c0cfec0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3264009850 tx=0x7f326402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bacb2bebf647ac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:09.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.987+0000 7f327b400640 1 -- 172.21.15.69:0/2504580259 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3264004030 con 0x7f327c0d18c0 2024-04-24T08:09:09.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.987+0000 7f327b400640 1 -- 172.21.15.69:0/2504580259 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32640041d0 con 0x7f327c0d18c0 2024-04-24T08:09:09.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.987+0000 7f327b400640 1 -- 172.21.15.69:0/2504580259 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3264003610 con 0x7f327c0d18c0 2024-04-24T08:09:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.987+0000 7f3283a00640 1 -- 172.21.15.69:0/2504580259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0d18c0 msgr2=0x7f327c0cfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.987+0000 7f3283a00640 1 --2- 172.21.15.69:0/2504580259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0d18c0 0x7f327c0cfec0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3264009850 tx=0x7f326402a270 comp rx=0 tx=0).stop 2024-04-24T08:09:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.987+0000 7f3283a00640 1 -- 172.21.15.69:0/2504580259 shutdown_connections 2024-04-24T08:09:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.987+0000 7f3283a00640 1 --2- 172.21.15.69:0/2504580259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f327c15ec70 0x7f327c0d0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.987+0000 7f3283a00640 1 --2- 172.21.15.69:0/2504580259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c0d18c0 0x7f327c0cfec0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.987+0000 7f3283a00640 1 -- 172.21.15.69:0/2504580259 >> 172.21.15.69:0/2504580259 conn(0x7f327c08bf60 msgr2=0x7f327c0d47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.988+0000 7f3283a00640 1 -- 172.21.15.69:0/2504580259 shutdown_connections 2024-04-24T08:09:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.988+0000 7f3283a00640 1 -- 172.21.15.69:0/2504580259 wait complete. 2024-04-24T08:09:09.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.989+0000 7f3283a00640 1 Processor -- start 2024-04-24T08:09:09.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.989+0000 7f3283a00640 1 -- start start 2024-04-24T08:09:09.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.990+0000 7f3283a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f327c0d18c0 0x7f327c0b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.990+0000 7f3283a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c15e890 0x7f327c0a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.990+0000 7f3283a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f327c15ec70 0x7f327c0b9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.990+0000 7f3283a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f327c0d2370 con 0x7f327c15e890 2024-04-24T08:09:09.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.990+0000 7f3283a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f327c0d3760 con 0x7f327c0d18c0 2024-04-24T08:09:09.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.990+0000 7f3283a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f327c0b3b60 con 0x7f327c15ec70 2024-04-24T08:09:09.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.991+0000 7f327be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c15e890 0x7f327c0a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.991+0000 7f327be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c15e890 0x7f327c0a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35908/0 (socket says 172.21.15.69:35908) 2024-04-24T08:09:09.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.991+0000 7f327be00640 1 -- 172.21.15.69:0/592227702 learned_addr learned my addr 172.21.15.69:0/592227702 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.991+0000 7f327be00640 1 -- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f327c15ec70 msgr2=0x7f327c0b9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.991+0000 7f327be00640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f327c15ec70 0x7f327c0b9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.991+0000 7f327be00640 1 -- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f327c0d18c0 msgr2=0x7f327c0b4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.991+0000 7f327be00640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f327c0d18c0 0x7f327c0b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.991+0000 7f327be00640 1 -- 172.21.15.69:0/592227702 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327c0a05a0 con 0x7f327c15e890 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.992+0000 7f327be00640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c15e890 0x7f327c0a1640 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3264009970 tx=0x7f32640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.992+0000 7f3278c00640 1 -- 172.21.15.69:0/592227702 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3264041030 con 0x7f327c15e890 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.992+0000 7f3278c00640 1 -- 172.21.15.69:0/592227702 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3264003c50 con 0x7f327c15e890 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.992+0000 7f3283a00640 1 -- 172.21.15.69:0/592227702 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f327c0a0880 con 0x7f327c15e890 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.992+0000 7f3278c00640 1 -- 172.21.15.69:0/592227702 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3264004030 con 0x7f327c15e890 2024-04-24T08:09:09.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.992+0000 7f3283a00640 1 -- 172.21.15.69:0/592227702 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f327c0a0e40 con 0x7f327c15e890 2024-04-24T08:09:09.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.993+0000 7f3283a00640 1 -- 172.21.15.69:0/592227702 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f327c068a80 con 0x7f327c15e890 2024-04-24T08:09:09.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.994+0000 7f3278c00640 1 -- 172.21.15.69:0/592227702 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3264044070 con 0x7f327c15e890 2024-04-24T08:09:09.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.995+0000 7f3278c00640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f325806c8f0 0x7f325806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.995+0000 7f3278c00640 1 -- 172.21.15.69:0/592227702 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f32640ac760 con 0x7f327c15e890 2024-04-24T08:09:09.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.996+0000 7f3281000640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f325806c8f0 0x7f325806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:09.997+0000 7f3281000640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f325806c8f0 0x7f325806edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f326c003cc0 tx=0x7f326c0079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:10.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.000+0000 7f3278c00640 1 -- 172.21.15.69:0/592227702 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f326403b3b0 con 0x7f327c15e890 2024-04-24T08:09:10.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.211+0000 7f3283a00640 1 -- 172.21.15.69:0/592227702 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f327c0d4eb0 con 0x7f327c15e890 2024-04-24T08:09:10.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.212+0000 7f3278c00640 1 -- 172.21.15.69:0/592227702 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f326407ab80 con 0x7f327c15e890 2024-04-24T08:09:10.213 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:09:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.214+0000 7f3283a00640 1 -- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f325806c8f0 msgr2=0x7f325806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.214+0000 7f3283a00640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f325806c8f0 0x7f325806edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f326c003cc0 tx=0x7f326c0079b0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.214+0000 7f3283a00640 1 -- 172.21.15.69:0/592227702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c15e890 msgr2=0x7f327c0a1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.214+0000 7f3283a00640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c15e890 0x7f327c0a1640 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3264009970 tx=0x7f32640046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.215+0000 7f3283a00640 1 -- 172.21.15.69:0/592227702 shutdown_connections 2024-04-24T08:09:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.215+0000 7f3283a00640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f325806c8f0 0x7f325806edb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.215+0000 7f3283a00640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f327c15e890 0x7f327c0a1640 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.215+0000 7f3283a00640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f327c15ec70 0x7f327c0b9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.215+0000 7f3283a00640 1 --2- 172.21.15.69:0/592227702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f327c0d18c0 0x7f327c0b4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.215+0000 7f3283a00640 1 -- 172.21.15.69:0/592227702 >> 172.21.15.69:0/592227702 conn(0x7f327c08bf60 msgr2=0x7f327c0b3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:10.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.215+0000 7f3283a00640 1 -- 172.21.15.69:0/592227702 shutdown_connections 2024-04-24T08:09:10.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.215+0000 7f3283a00640 1 -- 172.21.15.69:0/592227702 wait complete. 2024-04-24T08:09:10.234 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:09:10.235 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:10.294 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.494+0000 7f4ab6c00640 1 Processor -- start 2024-04-24T08:09:10.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.494+0000 7f4ab6c00640 1 -- start start 2024-04-24T08:09:10.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab6c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4ab0156c60 0x7f4ab015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab6c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 0x7f4ab015d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab6c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ab015d660 con 0x7f4ab00b30f0 2024-04-24T08:09:10.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab6c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4ab006dc30 con 0x7f4ab0156c60 2024-04-24T08:09:10.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab6c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4ab006df10 con 0x7f4ab00ad7e0 2024-04-24T08:09:10.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab5800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4ab0156c60 0x7f4ab015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab4e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4ab00b30f0 0x7f4ab00ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55462/0 (socket says 172.21.15.69:55462) 2024-04-24T08:09:10.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab4e00640 1 -- 172.21.15.69:0/890872867 learned_addr learned my addr 172.21.15.69:0/890872867 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:10.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab5800640 1 -- 172.21.15.69:0/890872867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 msgr2=0x7f4ab015d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:10.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab6200640 1 --2- 172.21.15.69:0/890872867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 0x7f4ab015d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab5800640 1 --2- 172.21.15.69:0/890872867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 0x7f4ab015d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab5800640 1 -- 172.21.15.69:0/890872867 >> v1:172.21.15.69:6789/0 conn(0x7f4ab00b30f0 legacy=0x7f4ab00ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4aafe00640 1 -- 172.21.15.69:0/890872867 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3114469406 0 0) 0x7f4ab015d660 con 0x7f4ab00b30f0 2024-04-24T08:09:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.496+0000 7f4ab5800640 1 -- 172.21.15.69:0/890872867 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab006e1f0 con 0x7f4ab0156c60 2024-04-24T08:09:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.497+0000 7f4ab6200640 1 --2- 172.21.15.69:0/890872867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 0x7f4ab015d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:10.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.497+0000 7f4ab5800640 1 --2- 172.21.15.69:0/890872867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 0x7f4ab015cac0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40097a0 tx=0x7f4aa402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f1ff0359754748 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4aafe00640 1 -- 172.21.15.69:0/890872867 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4037070 con 0x7f4ab0156c60 2024-04-24T08:09:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4aafe00640 1 -- 172.21.15.69:0/890872867 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aa4032440 con 0x7f4ab0156c60 2024-04-24T08:09:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4aafe00640 1 -- 172.21.15.69:0/890872867 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4032810 con 0x7f4ab0156c60 2024-04-24T08:09:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4ab6c00640 1 -- 172.21.15.69:0/890872867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 msgr2=0x7f4ab015cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4ab6c00640 1 --2- 172.21.15.69:0/890872867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 0x7f4ab015cac0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40097a0 tx=0x7f4aa402a180 comp rx=0 tx=0).stop 2024-04-24T08:09:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4ab6c00640 1 -- 172.21.15.69:0/890872867 shutdown_connections 2024-04-24T08:09:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4ab6c00640 1 --2- 172.21.15.69:0/890872867 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 0x7f4ab015d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4ab6c00640 1 --2- 172.21.15.69:0/890872867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 0x7f4ab015cac0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4ab6c00640 1 -- 172.21.15.69:0/890872867 >> 172.21.15.69:0/890872867 conn(0x7f4ab00d0160 msgr2=0x7f4ab00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:10.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4ab6c00640 1 -- 172.21.15.69:0/890872867 shutdown_connections 2024-04-24T08:09:10.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.498+0000 7f4ab6c00640 1 -- 172.21.15.69:0/890872867 wait complete. 2024-04-24T08:09:10.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.500+0000 7f4ab6c00640 1 Processor -- start 2024-04-24T08:09:10.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.500+0000 7f4ab6c00640 1 -- start start 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.501+0000 7f4ab6c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 0x7f4ab0071be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.501+0000 7f4ab6c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ab00b30f0 0x7f4ab0073730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.501+0000 7f4ab6c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 0x7f4ab0098250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.501+0000 7f4ab6c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab006e710 con 0x7f4ab0156c60 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.501+0000 7f4ab6c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab015d940 con 0x7f4ab00b30f0 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.501+0000 7f4ab6c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4ab006e890 con 0x7f4ab00ad7e0 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.501+0000 7f4ab6200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 0x7f4ab0098250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab6200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 0x7f4ab0098250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35926/0 (socket says 172.21.15.69:35926) 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab6200640 1 -- 172.21.15.69:0/366609933 learned_addr learned my addr 172.21.15.69:0/366609933 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab5800640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 0x7f4ab0071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab4e00640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ab00b30f0 0x7f4ab0073730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab6200640 1 -- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 msgr2=0x7f4ab0071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab6200640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 0x7f4ab0071be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab6200640 1 -- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ab00b30f0 msgr2=0x7f4ab0073730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab6200640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ab00b30f0 0x7f4ab0073730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab6200640 1 -- 172.21.15.69:0/366609933 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab01506c0 con 0x7f4ab0156c60 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab4e00640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ab00b30f0 0x7f4ab0073730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.502+0000 7f4ab6200640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 0x7f4ab0098250 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800ce80 tx=0x7f4aa800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.503+0000 7f4aad600640 1 -- 172.21.15.69:0/366609933 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa801a070 con 0x7f4ab0156c60 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.503+0000 7f4aad600640 1 -- 172.21.15.69:0/366609933 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4aa800d880 con 0x7f4ab0156c60 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.503+0000 7f4aad600640 1 -- 172.21.15.69:0/366609933 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa800dc70 con 0x7f4ab0156c60 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.503+0000 7f4ab6c00640 1 -- 172.21.15.69:0/366609933 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab00990b0 con 0x7f4ab0156c60 2024-04-24T08:09:10.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.503+0000 7f4ab6c00640 1 -- 172.21.15.69:0/366609933 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab0151cc0 con 0x7f4ab0156c60 2024-04-24T08:09:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.504+0000 7f4ab6c00640 1 -- 172.21.15.69:0/366609933 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab015fa60 con 0x7f4ab0156c60 2024-04-24T08:09:10.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.505+0000 7f4aad600640 1 -- 172.21.15.69:0/366609933 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4aa801d070 con 0x7f4ab0156c60 2024-04-24T08:09:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.506+0000 7f4aad600640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a9006c920 0x7f4a9006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.506+0000 7f4aad600640 1 -- 172.21.15.69:0/366609933 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f4aa808c2b0 con 0x7f4ab0156c60 2024-04-24T08:09:10.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.506+0000 7f4ab5800640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a9006c920 0x7f4a9006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.508+0000 7f4ab5800640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a9006c920 0x7f4a9006ede0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4034000 tx=0x7f4aa402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:10.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.511+0000 7f4aad600640 1 -- 172.21.15.69:0/366609933 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ab015fa60 con 0x7f4ab0156c60 2024-04-24T08:09:10.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.713+0000 7f4ab6c00640 1 -- 172.21.15.69:0/366609933 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ab0146080 con 0x7f4ab0156c60 2024-04-24T08:09:10.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.715+0000 7f4aad600640 1 -- 172.21.15.69:0/366609933 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7f4aa8059ff0 con 0x7f4ab0156c60 2024-04-24T08:09:10.717 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:10.717 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.718+0000 7f4ab6c00640 1 -- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a9006c920 msgr2=0x7f4a9006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.718+0000 7f4ab6c00640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a9006c920 0x7f4a9006ede0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4034000 tx=0x7f4aa402e000 comp rx=0 tx=0).stop 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.718+0000 7f4ab6c00640 1 -- 172.21.15.69:0/366609933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 msgr2=0x7f4ab0098250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.718+0000 7f4ab6c00640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 0x7f4ab0098250 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800ce80 tx=0x7f4aa800a710 comp rx=0 tx=0).stop 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.718+0000 7f4ab6c00640 1 -- 172.21.15.69:0/366609933 shutdown_connections 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.718+0000 7f4ab6c00640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a9006c920 0x7f4a9006ede0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.718+0000 7f4ab6c00640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ab0156c60 0x7f4ab0098250 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.718+0000 7f4ab6c00640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ab00b30f0 0x7f4ab0073730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.718+0000 7f4ab6c00640 1 --2- 172.21.15.69:0/366609933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ab00ad7e0 0x7f4ab0071be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.718+0000 7f4ab6c00640 1 -- 172.21.15.69:0/366609933 >> 172.21.15.69:0/366609933 conn(0x7f4ab00d0160 msgr2=0x7f4ab015c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:10.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.719+0000 7f4ab6c00640 1 -- 172.21.15.69:0/366609933 shutdown_connections 2024-04-24T08:09:10.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.719+0000 7f4ab6c00640 1 -- 172.21.15.69:0/366609933 wait complete. 2024-04-24T08:09:10.742 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:10.742 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:10.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.931+0000 7ff664000640 1 Processor -- start 2024-04-24T08:09:10.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.931+0000 7ff664000640 1 -- start start 2024-04-24T08:09:10.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.932+0000 7ff664000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff65c15f4b0 0x7ff65c0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.932+0000 7ff664000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff65c15f890 0x7ff65c0732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.933+0000 7ff664000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff65c073820 0x7ff65c0b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.933+0000 7ff664000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff65c073cc0 con 0x7ff65c15f4b0 2024-04-24T08:09:10.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.933+0000 7ff664000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff65c073fa0 con 0x7ff65c15f890 2024-04-24T08:09:10.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.933+0000 7ff664000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff65c0b3300 con 0x7ff65c073820 2024-04-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.933+0000 7ff661600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff65c15f4b0 0x7ff65c0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.933+0000 7ff661600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff65c15f4b0 0x7ff65c0b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35934/0 (socket says 172.21.15.69:35934) 2024-04-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.933+0000 7ff661600640 1 -- 172.21.15.69:0/3582972118 learned_addr learned my addr 172.21.15.69:0/3582972118 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.933+0000 7ff660c00640 1 --2- 172.21.15.69:0/3582972118 >> v2:172.21.15.121:3300/0 conn(0x7ff65c15f890 0x7ff65c0732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.933+0000 7ff662000640 1 --2- 172.21.15.69:0/3582972118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff65c073820 0x7ff65c0b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.934+0000 7ff661600640 1 -- 172.21.15.69:0/3582972118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff65c073820 msgr2=0x7ff65c0b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.934+0000 7ff661600640 1 --2- 172.21.15.69:0/3582972118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff65c073820 0x7ff65c0b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.934+0000 7ff661600640 1 -- 172.21.15.69:0/3582972118 >> v2:172.21.15.121:3300/0 conn(0x7ff65c15f890 msgr2=0x7ff65c0732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.934+0000 7ff661600640 1 --2- 172.21.15.69:0/3582972118 >> v2:172.21.15.121:3300/0 conn(0x7ff65c15f890 0x7ff65c0732e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.934+0000 7ff661600640 1 -- 172.21.15.69:0/3582972118 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c0b35e0 con 0x7ff65c15f4b0 2024-04-24T08:09:10.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.935+0000 7ff661600640 1 --2- 172.21.15.69:0/3582972118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 0x7ff65c0b3de0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff65c0701f0 tx=0x7ff65002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7837aadd3f6a05 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:10.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.935+0000 7ff65be00640 1 -- 172.21.15.69:0/3582972118 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650037070 con 0x7ff65c15f4b0 2024-04-24T08:09:10.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.935+0000 7ff65be00640 1 -- 172.21.15.69:0/3582972118 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff650032440 con 0x7ff65c15f4b0 2024-04-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.935+0000 7ff65be00640 1 -- 172.21.15.69:0/3582972118 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650032810 con 0x7ff65c15f4b0 2024-04-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.936+0000 7ff664000640 1 -- 172.21.15.69:0/3582972118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 msgr2=0x7ff65c0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.936+0000 7ff664000640 1 --2- 172.21.15.69:0/3582972118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 0x7ff65c0b3de0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ff65c0701f0 tx=0x7ff65002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.936+0000 7ff664000640 1 -- 172.21.15.69:0/3582972118 shutdown_connections 2024-04-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.936+0000 7ff664000640 1 --2- 172.21.15.69:0/3582972118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff65c073820 0x7ff65c0b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.936+0000 7ff664000640 1 --2- 172.21.15.69:0/3582972118 >> v2:172.21.15.121:3300/0 conn(0x7ff65c15f890 0x7ff65c0732e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.936+0000 7ff664000640 1 --2- 172.21.15.69:0/3582972118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 0x7ff65c0b3de0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.936+0000 7ff664000640 1 -- 172.21.15.69:0/3582972118 >> 172.21.15.69:0/3582972118 conn(0x7ff65c0d0350 msgr2=0x7ff65c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.936+0000 7ff664000640 1 -- 172.21.15.69:0/3582972118 shutdown_connections 2024-04-24T08:09:10.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.936+0000 7ff664000640 1 -- 172.21.15.69:0/3582972118 wait complete. 2024-04-24T08:09:10.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.938+0000 7ff664000640 1 Processor -- start 2024-04-24T08:09:10.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.939+0000 7ff664000640 1 -- start start 2024-04-24T08:09:10.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff664000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff65c073820 0x7ff65c14d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff664000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 0x7ff65c154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff664000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff65c15f890 0x7ff65c154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff664000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c15fec0 con 0x7ff65c15f4b0 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff664000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff65c15fd40 con 0x7ff65c15f890 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff664000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff65c0b4be0 con 0x7ff65c073820 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff660c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 0x7ff65c154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff660c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 0x7ff65c154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35942/0 (socket says 172.21.15.69:35942) 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff660c00640 1 -- 172.21.15.69:0/2942900091 learned_addr learned my addr 172.21.15.69:0/2942900091 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff661600640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff65c073820 0x7ff65c14d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff662000640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff65c15f890 0x7ff65c154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff660c00640 1 -- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff65c073820 msgr2=0x7ff65c14d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.940+0000 7ff660c00640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff65c073820 0x7ff65c14d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.941+0000 7ff660c00640 1 -- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff65c15f890 msgr2=0x7ff65c154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.941+0000 7ff660c00640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff65c15f890 0x7ff65c154a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.941+0000 7ff660c00640 1 -- 172.21.15.69:0/2942900091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65c06da80 con 0x7ff65c15f4b0 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.941+0000 7ff660c00640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 0x7ff65c154500 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff64800e9b0 tx=0x7ff64800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:10.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.941+0000 7ff659600640 1 -- 172.21.15.69:0/2942900091 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff648017070 con 0x7ff65c15f4b0 2024-04-24T08:09:10.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.941+0000 7ff664000640 1 -- 172.21.15.69:0/2942900091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65c156370 con 0x7ff65c15f4b0 2024-04-24T08:09:10.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.941+0000 7ff659600640 1 -- 172.21.15.69:0/2942900091 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff64800c9a0 con 0x7ff65c15f4b0 2024-04-24T08:09:10.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.941+0000 7ff659600640 1 -- 172.21.15.69:0/2942900091 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff64800cd90 con 0x7ff65c15f4b0 2024-04-24T08:09:10.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.941+0000 7ff664000640 1 -- 172.21.15.69:0/2942900091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff65c06dd40 con 0x7ff65c15f4b0 2024-04-24T08:09:10.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.942+0000 7ff664000640 1 -- 172.21.15.69:0/2942900091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff65c068a80 con 0x7ff65c15f4b0 2024-04-24T08:09:10.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.943+0000 7ff659600640 1 -- 172.21.15.69:0/2942900091 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff64801a070 con 0x7ff65c15f4b0 2024-04-24T08:09:10.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.944+0000 7ff659600640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff63c06c8f0 0x7ff63c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:10.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.945+0000 7ff659600640 1 -- 172.21.15.69:0/2942900091 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7ff64808bc60 con 0x7ff65c15f4b0 2024-04-24T08:09:10.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.945+0000 7ff661600640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff63c06c8f0 0x7ff63c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:10.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.946+0000 7ff661600640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff63c06c8f0 0x7ff63c06edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff650034000 tx=0x7ff65002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:10.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:10.949+0000 7ff659600640 1 -- 172.21.15.69:0/2942900091 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff648059ed0 con 0x7ff65c15f4b0 2024-04-24T08:09:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.150+0000 7ff664000640 1 -- 172.21.15.69:0/2942900091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff65c15ef20 con 0x7ff65c15f4b0 2024-04-24T08:09:11.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.151+0000 7ff659600640 1 -- 172.21.15.69:0/2942900091 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7ff6480597d0 con 0x7ff65c15f4b0 2024-04-24T08:09:11.152 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:11.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.153+0000 7ff664000640 1 -- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff63c06c8f0 msgr2=0x7ff63c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.153+0000 7ff664000640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff63c06c8f0 0x7ff63c06edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff650034000 tx=0x7ff65002e000 comp rx=0 tx=0).stop 2024-04-24T08:09:11.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.153+0000 7ff664000640 1 -- 172.21.15.69:0/2942900091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 msgr2=0x7ff65c154500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.153+0000 7ff664000640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 0x7ff65c154500 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff64800e9b0 tx=0x7ff64800ee70 comp rx=0 tx=0).stop 2024-04-24T08:09:11.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.154+0000 7ff664000640 1 -- 172.21.15.69:0/2942900091 shutdown_connections 2024-04-24T08:09:11.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.154+0000 7ff664000640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff63c06c8f0 0x7ff63c06edb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.154+0000 7ff664000640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff65c15f4b0 0x7ff65c154500 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.154+0000 7ff664000640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff65c15f890 0x7ff65c154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.154+0000 7ff664000640 1 --2- 172.21.15.69:0/2942900091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff65c073820 0x7ff65c14d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.154+0000 7ff664000640 1 -- 172.21.15.69:0/2942900091 >> 172.21.15.69:0/2942900091 conn(0x7ff65c0d0350 msgr2=0x7ff65c1617a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:11.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.154+0000 7ff664000640 1 -- 172.21.15.69:0/2942900091 shutdown_connections 2024-04-24T08:09:11.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.154+0000 7ff664000640 1 -- 172.21.15.69:0/2942900091 wait complete. 2024-04-24T08:09:11.174 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:11.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.384+0000 7f992d600640 1 Processor -- start 2024-04-24T08:09:11.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.384+0000 7f992d600640 1 -- start start 2024-04-24T08:09:11.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.385+0000 7f992d600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f992815c820 0x7f99280b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.385+0000 7f992d600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f99280b46a0 0x7f992806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.385+0000 7f992d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806fd10 0x7f99280ce250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.385+0000 7f992d600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f992815cc00 con 0x7f992815c820 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.385+0000 7f992d600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f992806ed30 con 0x7f99280b46a0 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.385+0000 7f992d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99280b4e10 con 0x7f992806fd10 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9926a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f992815c820 0x7f99280b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9926a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f992815c820 0x7f99280b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35944/0 (socket says 172.21.15.69:35944) 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9926a00640 1 -- 172.21.15.69:0/1657949433 learned_addr learned my addr 172.21.15.69:0/1657949433 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9926000640 1 --2- 172.21.15.69:0/1657949433 >> v2:172.21.15.121:3300/0 conn(0x7f99280b46a0 0x7f992806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9927400640 1 --2- 172.21.15.69:0/1657949433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806fd10 0x7f99280ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9926a00640 1 -- 172.21.15.69:0/1657949433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806fd10 msgr2=0x7f99280ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9926a00640 1 --2- 172.21.15.69:0/1657949433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806fd10 0x7f99280ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9926a00640 1 -- 172.21.15.69:0/1657949433 >> v2:172.21.15.121:3300/0 conn(0x7f99280b46a0 msgr2=0x7f992806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9926a00640 1 --2- 172.21.15.69:0/1657949433 >> v2:172.21.15.121:3300/0 conn(0x7f99280b46a0 0x7f992806f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9926a00640 1 -- 172.21.15.69:0/1657949433 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99280b50f0 con 0x7f992815c820 2024-04-24T08:09:11.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.386+0000 7f9927400640 1 --2- 172.21.15.69:0/1657949433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806fd10 0x7f99280ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.387+0000 7f9926a00640 1 --2- 172.21.15.69:0/1657949433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 0x7f99280b3e90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f992806e200 tx=0x7f991802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38aad1db4ac4b960 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.388+0000 7f9925600640 1 -- 172.21.15.69:0/1657949433 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918037070 con 0x7f992815c820 2024-04-24T08:09:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.388+0000 7f9925600640 1 -- 172.21.15.69:0/1657949433 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9918032440 con 0x7f992815c820 2024-04-24T08:09:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.388+0000 7f9925600640 1 -- 172.21.15.69:0/1657949433 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918032810 con 0x7f992815c820 2024-04-24T08:09:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.388+0000 7f992d600640 1 -- 172.21.15.69:0/1657949433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 msgr2=0x7f99280b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.388+0000 7f992d600640 1 --2- 172.21.15.69:0/1657949433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 0x7f99280b3e90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f992806e200 tx=0x7f991802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.388+0000 7f992d600640 1 -- 172.21.15.69:0/1657949433 shutdown_connections 2024-04-24T08:09:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.388+0000 7f992d600640 1 --2- 172.21.15.69:0/1657949433 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f992806fd10 0x7f99280ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.388+0000 7f992d600640 1 --2- 172.21.15.69:0/1657949433 >> v2:172.21.15.121:3300/0 conn(0x7f99280b46a0 0x7f992806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.388+0000 7f992d600640 1 --2- 172.21.15.69:0/1657949433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 0x7f99280b3e90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.388+0000 7f992d600640 1 -- 172.21.15.69:0/1657949433 >> 172.21.15.69:0/1657949433 conn(0x7f99280d0350 msgr2=0x7f99280d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.389+0000 7f992d600640 1 -- 172.21.15.69:0/1657949433 shutdown_connections 2024-04-24T08:09:11.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.389+0000 7f992d600640 1 -- 172.21.15.69:0/1657949433 wait complete. 2024-04-24T08:09:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.390+0000 7f992d600640 1 Processor -- start 2024-04-24T08:09:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.391+0000 7f992d600640 1 -- start start 2024-04-24T08:09:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.391+0000 7f992d600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f992806fd10 0x7f9928150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.391+0000 7f992d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99280b46a0 0x7f992815f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.391+0000 7f992d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 0x7f99280ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.391+0000 7f992d600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f99280b4b30 con 0x7f992815c820 2024-04-24T08:09:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f992d600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f992806f2b0 con 0x7f992806fd10 2024-04-24T08:09:11.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f992d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99280ce790 con 0x7f99280b46a0 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f9927400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 0x7f99280ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f9927400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 0x7f99280ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35952/0 (socket says 172.21.15.69:35952) 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f9927400640 1 -- 172.21.15.69:0/1848575345 learned_addr learned my addr 172.21.15.69:0/1848575345 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f9926a00640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f992806fd10 0x7f9928150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f9926000640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99280b46a0 0x7f992815f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f9927400640 1 -- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99280b46a0 msgr2=0x7f992815f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f9927400640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99280b46a0 0x7f992815f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f9927400640 1 -- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f992806fd10 msgr2=0x7f9928150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f9927400640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f992806fd10 0x7f9928150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.392+0000 7f9927400640 1 -- 172.21.15.69:0/1848575345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992815fc60 con 0x7f992815c820 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.393+0000 7f9927400640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 0x7f99280ca310 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f992000c820 tx=0x7f992000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.393+0000 7f991ea00640 1 -- 172.21.15.69:0/1848575345 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920015070 con 0x7f992815c820 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.393+0000 7f991ea00640 1 -- 172.21.15.69:0/1848575345 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9920007a30 con 0x7f992815c820 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.393+0000 7f991ea00640 1 -- 172.21.15.69:0/1848575345 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920007e20 con 0x7f992815c820 2024-04-24T08:09:11.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.393+0000 7f992d600640 1 -- 172.21.15.69:0/1848575345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9928150da0 con 0x7f992815c820 2024-04-24T08:09:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.393+0000 7f992d600640 1 -- 172.21.15.69:0/1848575345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99280c9b10 con 0x7f992815c820 2024-04-24T08:09:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.394+0000 7f992d600640 1 -- 172.21.15.69:0/1848575345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9928068a80 con 0x7f992815c820 2024-04-24T08:09:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.395+0000 7f991ea00640 1 -- 172.21.15.69:0/1848575345 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9920018070 con 0x7f992815c820 2024-04-24T08:09:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.396+0000 7f991ea00640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990406c8f0 0x7f990406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.396+0000 7f991ea00640 1 -- 172.21.15.69:0/1848575345 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f992008ba70 con 0x7f992815c820 2024-04-24T08:09:11.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.397+0000 7f9926a00640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990406c8f0 0x7f990406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.398+0000 7f9926a00640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990406c8f0 0x7f990406edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9918034000 tx=0x7f991802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:11.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.400+0000 7f991ea00640 1 -- 172.21.15.69:0/1848575345 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9920059ec0 con 0x7f992815c820 2024-04-24T08:09:11.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.600+0000 7f992d600640 1 -- 172.21.15.69:0/1848575345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f99280cddc0 con 0x7f992815c820 2024-04-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.601+0000 7f991ea00640 1 -- 172.21.15.69:0/1848575345 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f99200597c0 con 0x7f992815c820 2024-04-24T08:09:11.602 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 -- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990406c8f0 msgr2=0x7f990406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990406c8f0 0x7f990406edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9918034000 tx=0x7f991802e000 comp rx=0 tx=0).stop 2024-04-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 -- 172.21.15.69:0/1848575345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 msgr2=0x7f99280ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 0x7f99280ca310 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f992000c820 tx=0x7f992000cce0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 -- 172.21.15.69:0/1848575345 shutdown_connections 2024-04-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990406c8f0 0x7f990406edb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f992815c820 0x7f99280ca310 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99280b46a0 0x7f992815f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 --2- 172.21.15.69:0/1848575345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f992806fd10 0x7f9928150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 -- 172.21.15.69:0/1848575345 >> 172.21.15.69:0/1848575345 conn(0x7f99280d0350 msgr2=0x7f99280b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 -- 172.21.15.69:0/1848575345 shutdown_connections 2024-04-24T08:09:11.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.604+0000 7f992d600640 1 -- 172.21.15.69:0/1848575345 wait complete. 2024-04-24T08:09:11.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:11.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.826+0000 7f87e1c00640 1 Processor -- start 2024-04-24T08:09:11.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.826+0000 7f87e1c00640 1 -- start start 2024-04-24T08:09:11.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.827+0000 7f87e1c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f87dc15c820 0x7f87dc0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.827+0000 7f87e1c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87dc15cc00 con 0x7f87dc0ce250 2024-04-24T08:09:11.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.828+0000 7f87e1c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87dc06ed30 con 0x7f87dc0b46a0 2024-04-24T08:09:11.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.828+0000 7f87e1c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f87dc0b4e10 con 0x7f87dc15c820 2024-04-24T08:09:11.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.828+0000 7f87daa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f87dc15c820 0x7f87dc0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.828+0000 7f87db400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f87dc0ce250 0x7f87dc0ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55474/0 (socket says 172.21.15.69:55474) 2024-04-24T08:09:11.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.828+0000 7f87db400640 1 -- 172.21.15.69:0/4284224250 learned_addr learned my addr 172.21.15.69:0/4284224250 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:11.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.828+0000 7f87daa00640 1 -- 172.21.15.69:0/4284224250 >> v1:172.21.15.121:6789/0 conn(0x7f87dc0b46a0 legacy=0x7f87dc06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:11.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.828+0000 7f87daa00640 1 -- 172.21.15.69:0/4284224250 >> v1:172.21.15.69:6789/0 conn(0x7f87dc0ce250 legacy=0x7f87dc0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.828+0000 7f87da000640 1 -- 172.21.15.69:0/4284224250 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 320233527 0 0) 0x7f87dc15cc00 con 0x7f87dc0ce250 2024-04-24T08:09:11.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.828+0000 7f87daa00640 1 -- 172.21.15.69:0/4284224250 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc0b50f0 con 0x7f87dc15c820 2024-04-24T08:09:11.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.829+0000 7f87daa00640 1 --2- 172.21.15.69:0/4284224250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 0x7f87dc0b3e90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f87dc15cdd0 tx=0x7f87cc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97378feaa4b74196 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:11.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.830+0000 7f87da000640 1 -- 172.21.15.69:0/4284224250 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc037070 con 0x7f87dc15c820 2024-04-24T08:09:11.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.830+0000 7f87da000640 1 -- 172.21.15.69:0/4284224250 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87cc032440 con 0x7f87dc15c820 2024-04-24T08:09:11.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.830+0000 7f87da000640 1 -- 172.21.15.69:0/4284224250 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc032810 con 0x7f87dc15c820 2024-04-24T08:09:11.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.830+0000 7f87e1c00640 1 -- 172.21.15.69:0/4284224250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 msgr2=0x7f87dc0b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.830+0000 7f87e1c00640 1 --2- 172.21.15.69:0/4284224250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 0x7f87dc0b3e90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f87dc15cdd0 tx=0x7f87cc02a1c0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.830+0000 7f87e1c00640 1 -- 172.21.15.69:0/4284224250 shutdown_connections 2024-04-24T08:09:11.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.830+0000 7f87e1c00640 1 --2- 172.21.15.69:0/4284224250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 0x7f87dc0b3e90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.830+0000 7f87e1c00640 1 -- 172.21.15.69:0/4284224250 >> 172.21.15.69:0/4284224250 conn(0x7f87dc0d0350 msgr2=0x7f87dc0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:11.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.830+0000 7f87e1c00640 1 -- 172.21.15.69:0/4284224250 shutdown_connections 2024-04-24T08:09:11.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.830+0000 7f87e1c00640 1 -- 172.21.15.69:0/4284224250 wait complete. 2024-04-24T08:09:11.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.832+0000 7f87e1c00640 1 Processor -- start 2024-04-24T08:09:11.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.832+0000 7f87e1c00640 1 -- start start 2024-04-24T08:09:11.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.833+0000 7f87e1c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87dc0b46a0 0x7f87dc1f5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.833+0000 7f87e1c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87dc0ce250 0x7f87dc152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.833+0000 7f87e1c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 0x7f87dc152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.833+0000 7f87e1c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc0b4b30 con 0x7f87dc15c820 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.833+0000 7f87e1c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f87dc06f2b0 con 0x7f87dc0ce250 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.833+0000 7f87e1c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f87dc06fee0 con 0x7f87dc0b46a0 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.833+0000 7f87db400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 0x7f87dc152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.833+0000 7f87db400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 0x7f87dc152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35964/0 (socket says 172.21.15.69:35964) 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.833+0000 7f87db400640 1 -- 172.21.15.69:0/3243148315 learned_addr learned my addr 172.21.15.69:0/3243148315 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87d3e00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87dc0ce250 0x7f87dc152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87db400640 1 -- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87dc0b46a0 msgr2=0x7f87dc1f5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87db400640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87dc0b46a0 0x7f87dc1f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87db400640 1 -- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87dc0ce250 msgr2=0x7f87dc152120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:11.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87db400640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87dc0ce250 0x7f87dc152120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87db400640 1 -- 172.21.15.69:0/3243148315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87dc0af0c0 con 0x7f87dc15c820 2024-04-24T08:09:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87d3e00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87dc0ce250 0x7f87dc152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87db400640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 0x7f87dc152660 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f87d400c7d0 tx=0x7f87d400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87d2a00640 1 -- 172.21.15.69:0/3243148315 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d4018070 con 0x7f87dc15c820 2024-04-24T08:09:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87d2a00640 1 -- 172.21.15.69:0/3243148315 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87d400a640 con 0x7f87dc15c820 2024-04-24T08:09:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87d2a00640 1 -- 172.21.15.69:0/3243148315 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d400aa70 con 0x7f87dc15c820 2024-04-24T08:09:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.834+0000 7f87e1c00640 1 -- 172.21.15.69:0/3243148315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87dc152c90 con 0x7f87dc15c820 2024-04-24T08:09:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.835+0000 7f87e1c00640 1 -- 172.21.15.69:0/3243148315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87dc153250 con 0x7f87dc15c820 2024-04-24T08:09:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.836+0000 7f87e1c00640 1 -- 172.21.15.69:0/3243148315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87dc068a80 con 0x7f87dc15c820 2024-04-24T08:09:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.837+0000 7f87d2a00640 1 -- 172.21.15.69:0/3243148315 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f87d401b070 con 0x7f87dc15c820 2024-04-24T08:09:11.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.838+0000 7f87d2a00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f87b806c8f0 0x7f87b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:11.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.839+0000 7f87d2a00640 1 -- 172.21.15.69:0/3243148315 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f87d408b2e0 con 0x7f87dc15c820 2024-04-24T08:09:11.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.839+0000 7f87daa00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f87b806c8f0 0x7f87b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:11.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.840+0000 7f87daa00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f87b806c8f0 0x7f87b806edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f87cc034000 tx=0x7f87cc02c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:11.843+0000 7f87d2a00640 1 -- 172.21.15.69:0/3243148315 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87d40595e0 con 0x7f87dc15c820 2024-04-24T08:09:12.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.043+0000 7f87e1c00640 1 -- 172.21.15.69:0/3243148315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f87dc0cddc0 con 0x7f87dc15c820 2024-04-24T08:09:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.044+0000 7f87d2a00640 1 -- 172.21.15.69:0/3243148315 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f87d4058ee0 con 0x7f87dc15c820 2024-04-24T08:09:12.045 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:12.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.046+0000 7f87e1c00640 1 -- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f87b806c8f0 msgr2=0x7f87b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.046+0000 7f87e1c00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f87b806c8f0 0x7f87b806edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f87cc034000 tx=0x7f87cc02c000 comp rx=0 tx=0).stop 2024-04-24T08:09:12.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.046+0000 7f87e1c00640 1 -- 172.21.15.69:0/3243148315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 msgr2=0x7f87dc152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.046+0000 7f87e1c00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 0x7f87dc152660 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f87d400c7d0 tx=0x7f87d400cc90 comp rx=0 tx=0).stop 2024-04-24T08:09:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.047+0000 7f87e1c00640 1 -- 172.21.15.69:0/3243148315 shutdown_connections 2024-04-24T08:09:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.047+0000 7f87e1c00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f87b806c8f0 0x7f87b806edb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.047+0000 7f87e1c00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f87dc15c820 0x7f87dc152660 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.047+0000 7f87e1c00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87dc0ce250 0x7f87dc152120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.047+0000 7f87e1c00640 1 --2- 172.21.15.69:0/3243148315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f87dc0b46a0 0x7f87dc1f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.047+0000 7f87e1c00640 1 -- 172.21.15.69:0/3243148315 >> 172.21.15.69:0/3243148315 conn(0x7f87dc0d0350 msgr2=0x7f87dc0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.047+0000 7f87e1c00640 1 -- 172.21.15.69:0/3243148315 shutdown_connections 2024-04-24T08:09:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.047+0000 7f87e1c00640 1 -- 172.21.15.69:0/3243148315 wait complete. 2024-04-24T08:09:12.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:09:12.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.256+0000 7fae36c00640 1 Processor -- start 2024-04-24T08:09:12.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.256+0000 7fae36c00640 1 -- start start 2024-04-24T08:09:12.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.258+0000 7fae36c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fae30145fe0 0x7fae3006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.258+0000 7fae36c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae300cf3d0 con 0x7fae300d0300 2024-04-24T08:09:12.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.258+0000 7fae36c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae300cf6b0 con 0x7fae30145a90 2024-04-24T08:09:12.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.258+0000 7fae36c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fae300cf990 con 0x7fae30145fe0 2024-04-24T08:09:12.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.258+0000 7fae34c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fae30145fe0 0x7fae3006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.258+0000 7fae2fe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fae300d0300 0x7fae300d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55478/0 (socket says 172.21.15.69:55478) 2024-04-24T08:09:12.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.258+0000 7fae2fe00640 1 -- 172.21.15.69:0/3326963056 learned_addr learned my addr 172.21.15.69:0/3326963056 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:12.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.259+0000 7fae34c00640 1 -- 172.21.15.69:0/3326963056 >> v1:172.21.15.121:6789/0 conn(0x7fae30145a90 legacy=0x7fae30160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:12.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.259+0000 7fae34c00640 1 -- 172.21.15.69:0/3326963056 >> v1:172.21.15.69:6789/0 conn(0x7fae300d0300 legacy=0x7fae300d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.259+0000 7fae34c00640 1 -- 172.21.15.69:0/3326963056 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae300cfc70 con 0x7fae30145fe0 2024-04-24T08:09:12.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.259+0000 7fae2f400640 1 -- 172.21.15.69:0/3326963056 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4279461274 0 0) 0x7fae300cf3d0 con 0x7fae300d0300 2024-04-24T08:09:12.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.260+0000 7fae34c00640 1 --2- 172.21.15.69:0/3326963056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 0x7fae3006e240 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fae28009d90 tx=0x7fae2802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b975946546efa144 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:12.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.260+0000 7fae2f400640 1 -- 172.21.15.69:0/3326963056 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28007740 con 0x7fae30145fe0 2024-04-24T08:09:12.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.260+0000 7fae2f400640 1 -- 172.21.15.69:0/3326963056 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae280078e0 con 0x7fae30145fe0 2024-04-24T08:09:12.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.260+0000 7fae2f400640 1 -- 172.21.15.69:0/3326963056 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28007cb0 con 0x7fae30145fe0 2024-04-24T08:09:12.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.261+0000 7fae36c00640 1 -- 172.21.15.69:0/3326963056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 msgr2=0x7fae3006e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.261+0000 7fae36c00640 1 --2- 172.21.15.69:0/3326963056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 0x7fae3006e240 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fae28009d90 tx=0x7fae2802a6a0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.261+0000 7fae36c00640 1 -- 172.21.15.69:0/3326963056 shutdown_connections 2024-04-24T08:09:12.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.261+0000 7fae36c00640 1 --2- 172.21.15.69:0/3326963056 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 0x7fae3006e240 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.261+0000 7fae36c00640 1 -- 172.21.15.69:0/3326963056 >> 172.21.15.69:0/3326963056 conn(0x7fae30067d90 msgr2=0x7fae3006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:12.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.261+0000 7fae36c00640 1 -- 172.21.15.69:0/3326963056 shutdown_connections 2024-04-24T08:09:12.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.261+0000 7fae36c00640 1 -- 172.21.15.69:0/3326963056 wait complete. 2024-04-24T08:09:12.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.263+0000 7fae36c00640 1 Processor -- start 2024-04-24T08:09:12.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.263+0000 7fae36c00640 1 -- start start 2024-04-24T08:09:12.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae36c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae300d0300 0x7fae3015c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae36c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae30145a90 0x7fae301f5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae36c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 0x7fae301f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae36c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fae301458f0 con 0x7fae30145fe0 2024-04-24T08:09:12.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae36c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fae30145770 con 0x7fae30145a90 2024-04-24T08:09:12.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae36c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fae300cdc30 con 0x7fae300d0300 2024-04-24T08:09:12.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae34c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 0x7fae301f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae34c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 0x7fae301f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35990/0 (socket says 172.21.15.69:35990) 2024-04-24T08:09:12.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae34c00640 1 -- 172.21.15.69:0/1129087948 learned_addr learned my addr 172.21.15.69:0/1129087948 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:12.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae27400640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae30145a90 0x7fae301f5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.264+0000 7fae2fe00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae300d0300 0x7fae3015c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae34c00640 1 -- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae300d0300 msgr2=0x7fae3015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae34c00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae300d0300 0x7fae3015c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae34c00640 1 -- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae30145a90 msgr2=0x7fae301f5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae34c00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae30145a90 0x7fae301f5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae34c00640 1 -- 172.21.15.69:0/1129087948 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae300c5330 con 0x7fae30145fe0 2024-04-24T08:09:12.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae2fe00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae300d0300 0x7fae3015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:12.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae34c00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 0x7fae301f5f50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fae28009eb0 tx=0x7fae28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:12.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae2cc00640 1 -- 172.21.15.69:0/1129087948 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28040070 con 0x7fae30145fe0 2024-04-24T08:09:12.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae2cc00640 1 -- 172.21.15.69:0/1129087948 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae28031c90 con 0x7fae30145fe0 2024-04-24T08:09:12.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae36c00640 1 -- 172.21.15.69:0/1129087948 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae300aee30 con 0x7fae30145fe0 2024-04-24T08:09:12.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae2cc00640 1 -- 172.21.15.69:0/1129087948 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2803c640 con 0x7fae30145fe0 2024-04-24T08:09:12.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.265+0000 7fae36c00640 1 -- 172.21.15.69:0/1129087948 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae300af3f0 con 0x7fae30145fe0 2024-04-24T08:09:12.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.266+0000 7fae36c00640 1 -- 172.21.15.69:0/1129087948 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae30068a80 con 0x7fae30145fe0 2024-04-24T08:09:12.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.268+0000 7fae2cc00640 1 -- 172.21.15.69:0/1129087948 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fae28039040 con 0x7fae30145fe0 2024-04-24T08:09:12.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.268+0000 7fae2cc00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae0c06c8f0 0x7fae0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.269+0000 7fae2cc00640 1 -- 172.21.15.69:0/1129087948 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fae280ad250 con 0x7fae30145fe0 2024-04-24T08:09:12.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.269+0000 7fae2fe00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae0c06c8f0 0x7fae0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.270+0000 7fae2fe00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae0c06c8f0 0x7fae0c06edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fae20000f60 tx=0x7fae20009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:12.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.273+0000 7fae2cc00640 1 -- 172.21.15.69:0/1129087948 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae2807b6a0 con 0x7fae30145fe0 2024-04-24T08:09:12.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.487+0000 7fae36c00640 1 -- 172.21.15.69:0/1129087948 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fae3006ebb0 con 0x7fae30145fe0 2024-04-24T08:09:12.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.488+0000 7fae2cc00640 1 -- 172.21.15.69:0/1129087948 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7fae2807afa0 con 0x7fae30145fe0 2024-04-24T08:09:12.489 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:09:12.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.490+0000 7fae36c00640 1 -- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae0c06c8f0 msgr2=0x7fae0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.490+0000 7fae36c00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae0c06c8f0 0x7fae0c06edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fae20000f60 tx=0x7fae20009040 comp rx=0 tx=0).stop 2024-04-24T08:09:12.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.491+0000 7fae36c00640 1 -- 172.21.15.69:0/1129087948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 msgr2=0x7fae301f5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.491+0000 7fae36c00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 0x7fae301f5f50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fae28009eb0 tx=0x7fae28004a40 comp rx=0 tx=0).stop 2024-04-24T08:09:12.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.491+0000 7fae36c00640 1 -- 172.21.15.69:0/1129087948 shutdown_connections 2024-04-24T08:09:12.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.491+0000 7fae36c00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae0c06c8f0 0x7fae0c06edb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.491+0000 7fae36c00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae30145fe0 0x7fae301f5f50 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.491+0000 7fae36c00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae30145a90 0x7fae301f5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.491+0000 7fae36c00640 1 --2- 172.21.15.69:0/1129087948 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae300d0300 0x7fae3015c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.491+0000 7fae36c00640 1 -- 172.21.15.69:0/1129087948 >> 172.21.15.69:0/1129087948 conn(0x7fae30067d90 msgr2=0x7fae3006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:12.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.491+0000 7fae36c00640 1 -- 172.21.15.69:0/1129087948 shutdown_connections 2024-04-24T08:09:12.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.491+0000 7fae36c00640 1 -- 172.21.15.69:0/1129087948 wait complete. 2024-04-24T08:09:12.510 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2024-04-24T08:09:12.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:12.511 DEBUG:teuthology.orchestra.run.smithi121:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-24T08:09:12.601 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:09:12.602 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:09:12.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:12.602 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-24T08:09:12.673 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:09:12.673 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:12.729 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:09:12.729 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:12.780 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:12.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.962+0000 7f9eeda00640 1 Processor -- start 2024-04-24T08:09:12.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.962+0000 7f9eeda00640 1 -- start start 2024-04-24T08:09:12.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.963+0000 7f9eeda00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9ee8071e50 0x7f9ee815ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.963+0000 7f9eeda00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9ee8162300 0x7f9ee806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.963+0000 7f9eeda00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ee815f540 con 0x7f9ee8072230 2024-04-24T08:09:12.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.963+0000 7f9eeda00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee815f820 con 0x7f9ee8071e50 2024-04-24T08:09:12.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.963+0000 7f9eeda00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee815fb00 con 0x7f9ee8162300 2024-04-24T08:09:12.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.963+0000 7f9eed000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9ee8072230 0x7f9ee806e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55488/0 (socket says 172.21.15.69:55488) 2024-04-24T08:09:12.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.964+0000 7f9eed000640 1 -- 172.21.15.69:0/3340477284 learned_addr learned my addr 172.21.15.69:0/3340477284 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:12.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.964+0000 7f9ee7e00640 1 --2- 172.21.15.69:0/3340477284 >> v2:172.21.15.69:3300/0 conn(0x7f9ee8071e50 0x7f9ee815ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.964+0000 7f9ee7400640 1 --2- 172.21.15.69:0/3340477284 >> v2:172.21.15.121:3300/0 conn(0x7f9ee8162300 0x7f9ee806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.964+0000 7f9ee7e00640 1 -- 172.21.15.69:0/3340477284 >> v2:172.21.15.121:3300/0 conn(0x7f9ee8162300 msgr2=0x7f9ee806e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:12.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.964+0000 7f9ee7e00640 1 --2- 172.21.15.69:0/3340477284 >> v2:172.21.15.121:3300/0 conn(0x7f9ee8162300 0x7f9ee806e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.964+0000 7f9ee7e00640 1 -- 172.21.15.69:0/3340477284 >> v1:172.21.15.69:6789/0 conn(0x7f9ee8072230 legacy=0x7f9ee806e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.964+0000 7f9ee7e00640 1 -- 172.21.15.69:0/3340477284 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee806dda0 con 0x7f9ee8071e50 2024-04-24T08:09:12.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.964+0000 7f9ee6a00640 1 -- 172.21.15.69:0/3340477284 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2228625609 0 0) 0x7f9ee815f540 con 0x7f9ee8072230 2024-04-24T08:09:12.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.964+0000 7f9ee7400640 1 --2- 172.21.15.69:0/3340477284 >> v2:172.21.15.121:3300/0 conn(0x7f9ee8162300 0x7f9ee806e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:12.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.965+0000 7f9ee7e00640 1 --2- 172.21.15.69:0/3340477284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8071e50 0x7f9ee815ecc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9ee806da20 tx=0x7f9ed802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1545d530e27c9ca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:12.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.965+0000 7f9ee6a00640 1 -- 172.21.15.69:0/3340477284 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8037070 con 0x7f9ee8071e50 2024-04-24T08:09:12.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.965+0000 7f9ee6a00640 1 -- 172.21.15.69:0/3340477284 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed8032440 con 0x7f9ee8071e50 2024-04-24T08:09:12.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.965+0000 7f9ee6a00640 1 -- 172.21.15.69:0/3340477284 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8032810 con 0x7f9ee8071e50 2024-04-24T08:09:12.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.966+0000 7f9eeda00640 1 -- 172.21.15.69:0/3340477284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8071e50 msgr2=0x7f9ee815ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.966+0000 7f9eeda00640 1 --2- 172.21.15.69:0/3340477284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8071e50 0x7f9ee815ecc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9ee806da20 tx=0x7f9ed802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.966+0000 7f9eeda00640 1 -- 172.21.15.69:0/3340477284 shutdown_connections 2024-04-24T08:09:12.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.966+0000 7f9eeda00640 1 --2- 172.21.15.69:0/3340477284 >> v2:172.21.15.121:3300/0 conn(0x7f9ee8162300 0x7f9ee806e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.966+0000 7f9eeda00640 1 --2- 172.21.15.69:0/3340477284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8071e50 0x7f9ee815ecc0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.966+0000 7f9eeda00640 1 -- 172.21.15.69:0/3340477284 >> 172.21.15.69:0/3340477284 conn(0x7f9ee80d0160 msgr2=0x7f9ee80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:12.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.966+0000 7f9eeda00640 1 -- 172.21.15.69:0/3340477284 shutdown_connections 2024-04-24T08:09:12.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.966+0000 7f9eeda00640 1 -- 172.21.15.69:0/3340477284 wait complete. 2024-04-24T08:09:12.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.968+0000 7f9eeda00640 1 Processor -- start 2024-04-24T08:09:12.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.968+0000 7f9eeda00640 1 -- start start 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9eeda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ee8071e50 0x7f9ee809d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9eeda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8072230 0x7f9ee81f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9eeda00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee8162300 0x7f9ee81f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9eeda00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee806f090 con 0x7f9ee8072230 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9eeda00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee806ef10 con 0x7f9ee8162300 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9eeda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee806f210 con 0x7f9ee8071e50 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9ee7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8072230 0x7f9ee81f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9ee7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8072230 0x7f9ee81f84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36004/0 (socket says 172.21.15.69:36004) 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9ee7400640 1 -- 172.21.15.69:0/1085371670 learned_addr learned my addr 172.21.15.69:0/1085371670 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9ee7400640 1 -- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ee8071e50 msgr2=0x7f9ee809d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9ee7400640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ee8071e50 0x7f9ee809d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.969+0000 7f9eed000640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee8162300 0x7f9ee81f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.970+0000 7f9ee7400640 1 -- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee8162300 msgr2=0x7f9ee81f71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.970+0000 7f9ee7400640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee8162300 0x7f9ee81f71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.970+0000 7f9ee7400640 1 -- 172.21.15.69:0/1085371670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee809d7f0 con 0x7f9ee8072230 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.970+0000 7f9ee7400640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8072230 0x7f9ee81f84c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400c820 tx=0x7f9ed400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.970+0000 7f9edfe00640 1 -- 172.21.15.69:0/1085371670 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4015070 con 0x7f9ee8072230 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.970+0000 7f9edfe00640 1 -- 172.21.15.69:0/1085371670 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ed4007a30 con 0x7f9ee8072230 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.970+0000 7f9eeda00640 1 -- 172.21.15.69:0/1085371670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee809dad0 con 0x7f9ee8072230 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.970+0000 7f9edfe00640 1 -- 172.21.15.69:0/1085371670 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4007de0 con 0x7f9ee8072230 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.970+0000 7f9eeda00640 1 -- 172.21.15.69:0/1085371670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee809e090 con 0x7f9ee8072230 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.972+0000 7f9eeda00640 1 -- 172.21.15.69:0/1085371670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee81f56c0 con 0x7f9ee8072230 2024-04-24T08:09:12.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.973+0000 7f9edfe00640 1 -- 172.21.15.69:0/1085371670 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9ed4018070 con 0x7f9ee8072230 2024-04-24T08:09:12.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.974+0000 7f9edfe00640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ec806c920 0x7f9ec806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.974+0000 7f9edfe00640 1 -- 172.21.15.69:0/1085371670 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f9ed408bb20 con 0x7f9ee8072230 2024-04-24T08:09:12.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.975+0000 7f9ee7e00640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ec806c920 0x7f9ec806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.976+0000 7f9ee7e00640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ec806c920 0x7f9ec806ede0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8034000 tx=0x7f9ed802c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:12.979+0000 7f9edfe00640 1 -- 172.21.15.69:0/1085371670 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ed4059e20 con 0x7f9ee8072230 2024-04-24T08:09:13.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.184+0000 7f9eeda00640 1 -- 172.21.15.69:0/1085371670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ee8075d50 con 0x7f9ee8072230 2024-04-24T08:09:13.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.187+0000 7f9edfe00640 1 -- 172.21.15.69:0/1085371670 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7f9ed4059720 con 0x7f9ee8072230 2024-04-24T08:09:13.189 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:13.190 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.190+0000 7f9eeda00640 1 -- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ec806c920 msgr2=0x7f9ec806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.190+0000 7f9eeda00640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ec806c920 0x7f9ec806ede0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8034000 tx=0x7f9ed802c000 comp rx=0 tx=0).stop 2024-04-24T08:09:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.190+0000 7f9eeda00640 1 -- 172.21.15.69:0/1085371670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8072230 msgr2=0x7f9ee81f84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.190+0000 7f9eeda00640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8072230 0x7f9ee81f84c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400c820 tx=0x7f9ed400cce0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.191+0000 7f9eeda00640 1 -- 172.21.15.69:0/1085371670 shutdown_connections 2024-04-24T08:09:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.191+0000 7f9eeda00640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ec806c920 0x7f9ec806ede0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.191+0000 7f9eeda00640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ee8072230 0x7f9ee81f84c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.191+0000 7f9eeda00640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee8162300 0x7f9ee81f71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.191+0000 7f9eeda00640 1 --2- 172.21.15.69:0/1085371670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ee8071e50 0x7f9ee809d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.191+0000 7f9eeda00640 1 -- 172.21.15.69:0/1085371670 >> 172.21.15.69:0/1085371670 conn(0x7f9ee80d0160 msgr2=0x7f9ee81627a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.191+0000 7f9eeda00640 1 -- 172.21.15.69:0/1085371670 shutdown_connections 2024-04-24T08:09:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.191+0000 7f9eeda00640 1 -- 172.21.15.69:0/1085371670 wait complete. 2024-04-24T08:09:13.210 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:13.211 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:13.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.410+0000 7f462fe00640 1 Processor -- start 2024-04-24T08:09:13.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.410+0000 7f462fe00640 1 -- start start 2024-04-24T08:09:13.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.411+0000 7f462fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f463015c9d0 0x7f46300b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.411+0000 7f462fe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f46300cf400 0x7f46300cd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.411+0000 7f462fe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f463015cdb0 con 0x7f46301604c0 2024-04-24T08:09:13.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.411+0000 7f462fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f46300d1ee0 con 0x7f463015c9d0 2024-04-24T08:09:13.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.411+0000 7f462fe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4630160a80 con 0x7f46300cf400 2024-04-24T08:09:13.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.411+0000 7f462ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f463015c9d0 0x7f46300b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.411+0000 7f462e000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f46301604c0 0x7f46300cecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55494/0 (socket says 172.21.15.69:55494) 2024-04-24T08:09:13.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.411+0000 7f462e000640 1 -- 172.21.15.69:0/2291974114 learned_addr learned my addr 172.21.15.69:0/2291974114 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:13.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.412+0000 7f462f400640 1 --2- 172.21.15.69:0/2291974114 >> v2:172.21.15.121:3300/0 conn(0x7f46300cf400 0x7f46300cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.412+0000 7f462ea00640 1 -- 172.21.15.69:0/2291974114 >> v2:172.21.15.121:3300/0 conn(0x7f46300cf400 msgr2=0x7f46300cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:13.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.412+0000 7f462ea00640 1 --2- 172.21.15.69:0/2291974114 >> v2:172.21.15.121:3300/0 conn(0x7f46300cf400 0x7f46300cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.412+0000 7f462ea00640 1 -- 172.21.15.69:0/2291974114 >> v1:172.21.15.69:6789/0 conn(0x7f46301604c0 legacy=0x7f46300cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.412+0000 7f462d600640 1 -- 172.21.15.69:0/2291974114 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2452052303 0 0) 0x7f463015cdb0 con 0x7f46301604c0 2024-04-24T08:09:13.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.412+0000 7f462ea00640 1 -- 172.21.15.69:0/2291974114 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4630160d60 con 0x7f463015c9d0 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.413+0000 7f462ea00640 1 --2- 172.21.15.69:0/2291974114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f463015c9d0 0x7f46300b5270 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f46300b81f0 tx=0x7f462002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942f10526af752e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.413+0000 7f462d600640 1 -- 172.21.15.69:0/2291974114 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620007740 con 0x7f463015c9d0 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.413+0000 7f462d600640 1 -- 172.21.15.69:0/2291974114 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46200078e0 con 0x7f463015c9d0 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.413+0000 7f462d600640 1 -- 172.21.15.69:0/2291974114 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620007cb0 con 0x7f463015c9d0 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.414+0000 7f462fe00640 1 -- 172.21.15.69:0/2291974114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f463015c9d0 msgr2=0x7f46300b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.414+0000 7f462fe00640 1 --2- 172.21.15.69:0/2291974114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f463015c9d0 0x7f46300b5270 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f46300b81f0 tx=0x7f462002a5f0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.414+0000 7f462fe00640 1 -- 172.21.15.69:0/2291974114 shutdown_connections 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.414+0000 7f462fe00640 1 --2- 172.21.15.69:0/2291974114 >> v2:172.21.15.121:3300/0 conn(0x7f46300cf400 0x7f46300cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.414+0000 7f462fe00640 1 --2- 172.21.15.69:0/2291974114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f463015c9d0 0x7f46300b5270 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.414+0000 7f462fe00640 1 -- 172.21.15.69:0/2291974114 >> 172.21.15.69:0/2291974114 conn(0x7f4630067760 msgr2=0x7f4630067b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.414+0000 7f462fe00640 1 -- 172.21.15.69:0/2291974114 shutdown_connections 2024-04-24T08:09:13.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.414+0000 7f462fe00640 1 -- 172.21.15.69:0/2291974114 wait complete. 2024-04-24T08:09:13.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.416+0000 7f462fe00640 1 Processor -- start 2024-04-24T08:09:13.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.416+0000 7f462fe00640 1 -- start start 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.417+0000 7f462fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cf400 0x7f46300c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.417+0000 7f462fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f463015c9d0 0x7f46301f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.417+0000 7f462fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46301604c0 0x7f46301f61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.417+0000 7f462fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f46300d0530 con 0x7f46300cf400 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.417+0000 7f462fe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4630161320 con 0x7f46301604c0 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.417+0000 7f462fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f46300d0a50 con 0x7f463015c9d0 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.417+0000 7f462ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cf400 0x7f46300c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.417+0000 7f462ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cf400 0x7f46300c77f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36030/0 (socket says 172.21.15.69:36030) 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.417+0000 7f462ea00640 1 -- 172.21.15.69:0/629580587 learned_addr learned my addr 172.21.15.69:0/629580587 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f462f400640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46301604c0 0x7f46301f61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f462ea00640 1 -- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f463015c9d0 msgr2=0x7f46301f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f462ea00640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f463015c9d0 0x7f46301f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f462ea00640 1 -- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46301604c0 msgr2=0x7f46301f61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f462ea00640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46301604c0 0x7f46301f61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f462ea00640 1 -- 172.21.15.69:0/629580587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463015b590 con 0x7f46300cf400 2024-04-24T08:09:13.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f462ea00640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cf400 0x7f46300c77f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4620000f80 tx=0x7f4620004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:13.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f4626a00640 1 -- 172.21.15.69:0/629580587 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620040070 con 0x7f46300cf400 2024-04-24T08:09:13.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f4626a00640 1 -- 172.21.15.69:0/629580587 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4620031c40 con 0x7f46300cf400 2024-04-24T08:09:13.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f462fe00640 1 -- 172.21.15.69:0/629580587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46300c7dd0 con 0x7f46300cf400 2024-04-24T08:09:13.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f4626a00640 1 -- 172.21.15.69:0/629580587 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620009450 con 0x7f46300cf400 2024-04-24T08:09:13.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.418+0000 7f462fe00640 1 -- 172.21.15.69:0/629580587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46300c8390 con 0x7f46300cf400 2024-04-24T08:09:13.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.420+0000 7f462fe00640 1 -- 172.21.15.69:0/629580587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4630075d50 con 0x7f46300cf400 2024-04-24T08:09:13.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.421+0000 7f4626a00640 1 -- 172.21.15.69:0/629580587 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4620039040 con 0x7f46300cf400 2024-04-24T08:09:13.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.421+0000 7f4626a00640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f460c06c8f0 0x7f460c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.421+0000 7f4626a00640 1 -- 172.21.15.69:0/629580587 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f46200acef0 con 0x7f46300cf400 2024-04-24T08:09:13.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.422+0000 7f462e000640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f460c06c8f0 0x7f460c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.423+0000 7f462e000640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f460c06c8f0 0x7f460c06edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4618007fd0 tx=0x7f4618007960 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:13.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.426+0000 7f4626a00640 1 -- 172.21.15.69:0/629580587 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f462007b1f0 con 0x7f46300cf400 2024-04-24T08:09:13.526 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:09:13.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:09:13.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.630+0000 7f462fe00640 1 -- 172.21.15.69:0/629580587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f46300b16d0 con 0x7f46300cf400 2024-04-24T08:09:13.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.630+0000 7f4626a00640 1 -- 172.21.15.69:0/629580587 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f462007aaf0 con 0x7f46300cf400 2024-04-24T08:09:13.632 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:13.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 -- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f460c06c8f0 msgr2=0x7f460c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f460c06c8f0 0x7f460c06edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4618007fd0 tx=0x7f4618007960 comp rx=0 tx=0).stop 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 -- 172.21.15.69:0/629580587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cf400 msgr2=0x7f46300c77f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cf400 0x7f46300c77f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4620000f80 tx=0x7f4620004a40 comp rx=0 tx=0).stop 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 -- 172.21.15.69:0/629580587 shutdown_connections 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f460c06c8f0 0x7f460c06edb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46301604c0 0x7f46301f61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f463015c9d0 0x7f46301f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 --2- 172.21.15.69:0/629580587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46300cf400 0x7f46300c77f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 -- 172.21.15.69:0/629580587 >> 172.21.15.69:0/629580587 conn(0x7f4630067760 msgr2=0x7f46300cd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 -- 172.21.15.69:0/629580587 shutdown_connections 2024-04-24T08:09:13.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.633+0000 7f462fe00640 1 -- 172.21.15.69:0/629580587 wait complete. 2024-04-24T08:09:13.654 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:13.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.853+0000 7f9d91c00640 1 Processor -- start 2024-04-24T08:09:13.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.853+0000 7f9d91c00640 1 -- start start 2024-04-24T08:09:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.855+0000 7f9d91c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c0b3de0 0x7f9d8c0732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.855+0000 7f9d91c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9d8c073820 0x7f9d8c0b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.855+0000 7f9d91c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d8c073cc0 con 0x7f9d8c15f4b0 2024-04-24T08:09:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.855+0000 7f9d91c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9d8c0b4260 con 0x7f9d8c073820 2024-04-24T08:09:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.855+0000 7f9d91c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d8c073fa0 con 0x7f9d8c0b3de0 2024-04-24T08:09:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.855+0000 7f9d8aa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9d8c15f4b0 0x7f9d8c15f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55506/0 (socket says 172.21.15.69:55506) 2024-04-24T08:09:13.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.855+0000 7f9d8aa00640 1 -- 172.21.15.69:0/1951509566 learned_addr learned my addr 172.21.15.69:0/1951509566 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:13.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.855+0000 7f9d8b400640 1 --2- 172.21.15.69:0/1951509566 >> v2:172.21.15.121:3300/0 conn(0x7f9d8c073820 0x7f9d8c0b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1690579941 0 0) 0x7f9d8c073cc0 con 0x7f9d8c15f4b0 2024-04-24T08:09:13.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d83e00640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c0b3de0 0x7f9d8c0732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d70003520 con 0x7f9d8c15f4b0 2024-04-24T08:09:13.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1850886505 0 0) 0x7f9d70003520 con 0x7f9d8c15f4b0 2024-04-24T08:09:13.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c0b3de0 msgr2=0x7f9d8c0732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:13.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c0b3de0 0x7f9d8c0732e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 >> v2:172.21.15.121:3300/0 conn(0x7f9d8c073820 msgr2=0x7f9d8c0b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:13.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 --2- 172.21.15.69:0/1951509566 >> v2:172.21.15.121:3300/0 conn(0x7f9d8c073820 0x7f9d8c0b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c0b3300 con 0x7f9d8c15f4b0 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f9d7c001670 con 0x7f9d8c15f4b0 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 >> v1:172.21.15.69:6789/0 conn(0x7f9d8c15f4b0 legacy=0x7f9d8c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 0x7f9d70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c0b3de0 0x7f9d70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d70004d40 0x7f9d700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c0b4260 con 0x7f9d700039b0 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c073fa0 con 0x7f9d70004d40 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.856+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d7000b9c0 con 0x7f9d8c0b3de0 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.857+0000 7f9d8aa00640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 0x7f9d70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.857+0000 7f9d8b400640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d70004d40 0x7f9d700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.857+0000 7f9d83e00640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c0b3de0 0x7f9d70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.857+0000 7f9d8aa00640 1 -- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c0b3de0 msgr2=0x7f9d70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:13.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.857+0000 7f9d8aa00640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c0b3de0 0x7f9d70004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.857+0000 7f9d8aa00640 1 -- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d70004d40 msgr2=0x7f9d700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:13.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.857+0000 7f9d8aa00640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d70004d40 0x7f9d700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.857+0000 7f9d8aa00640 1 -- 172.21.15.69:0/1951509566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d70003520 con 0x7f9d700039b0 2024-04-24T08:09:13.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.857+0000 7f9d8b400640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d70004d40 0x7f9d700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:13.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.858+0000 7f9d8aa00640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 0x7f9d70003dd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c270 tx=0x7f9d70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31655b23251a057 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:13.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.859+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c007d30 con 0x7f9d700039b0 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.859+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d7c005ad0 con 0x7f9d700039b0 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.859+0000 7f9d8a000640 1 -- 172.21.15.69:0/1951509566 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c007070 con 0x7f9d700039b0 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.859+0000 7f9d91c00640 1 -- 172.21.15.69:0/1951509566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 msgr2=0x7f9d70003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.859+0000 7f9d91c00640 1 --2- 172.21.15.69:0/1951509566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 0x7f9d70003dd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c270 tx=0x7f9d70005830 comp rx=0 tx=0).stop 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.859+0000 7f9d8b400640 1 -- 172.21.15.69:0/1951509566 reap_dead start 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.859+0000 7f9d91c00640 1 -- 172.21.15.69:0/1951509566 shutdown_connections 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.859+0000 7f9d91c00640 1 -- 172.21.15.69:0/1951509566 >> 172.21.15.69:0/1951509566 conn(0x7f9d8c0d0350 msgr2=0x7f9d8c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.859+0000 7f9d91c00640 1 -- 172.21.15.69:0/1951509566 shutdown_connections 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.859+0000 7f9d91c00640 1 -- 172.21.15.69:0/1951509566 wait complete. 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.861+0000 7f9d91c00640 1 Processor -- start 2024-04-24T08:09:13.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.861+0000 7f9d91c00640 1 -- start start 2024-04-24T08:09:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.862+0000 7f9d91c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 0x7f9d8c151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.862+0000 7f9d91c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c153e40 0x7f9d8c1426d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.862+0000 7f9d91c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d8c1542c0 0x7f9d8c142c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.862+0000 7f9d91c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c0732e0 con 0x7f9d700039b0 2024-04-24T08:09:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.862+0000 7f9d91c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c0b38c0 con 0x7f9d8c1542c0 2024-04-24T08:09:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.862+0000 7f9d91c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d8c073460 con 0x7f9d8c153e40 2024-04-24T08:09:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.862+0000 7f9d8aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 0x7f9d8c151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d8aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 0x7f9d8c151880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36048/0 (socket says 172.21.15.69:36048) 2024-04-24T08:09:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d8aa00640 1 -- 172.21.15.69:0/860715113 learned_addr learned my addr 172.21.15.69:0/860715113 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d8aa00640 1 -- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c153e40 msgr2=0x7f9d8c1426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d8aa00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c153e40 0x7f9d8c1426d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d8aa00640 1 -- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d8c1542c0 msgr2=0x7f9d8c142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d8aa00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d8c1542c0 0x7f9d8c142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d8aa00640 1 -- 172.21.15.69:0/860715113 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c151dc0 con 0x7f9d700039b0 2024-04-24T08:09:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d8aa00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 0x7f9d8c151880 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c270 tx=0x7f9d8c073e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d82a00640 1 -- 172.21.15.69:0/860715113 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c042070 con 0x7f9d700039b0 2024-04-24T08:09:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d82a00640 1 -- 172.21.15.69:0/860715113 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d7c007480 con 0x7f9d700039b0 2024-04-24T08:09:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.863+0000 7f9d82a00640 1 -- 172.21.15.69:0/860715113 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c007d30 con 0x7f9d700039b0 2024-04-24T08:09:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.864+0000 7f9d91c00640 1 -- 172.21.15.69:0/860715113 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8c1567d0 con 0x7f9d700039b0 2024-04-24T08:09:13.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.864+0000 7f9d91c00640 1 -- 172.21.15.69:0/860715113 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d8c143580 con 0x7f9d700039b0 2024-04-24T08:09:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.865+0000 7f9d91c00640 1 -- 172.21.15.69:0/860715113 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d8c0cdde0 con 0x7f9d700039b0 2024-04-24T08:09:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.866+0000 7f9d82a00640 1 -- 172.21.15.69:0/860715113 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9d7c045070 con 0x7f9d700039b0 2024-04-24T08:09:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.867+0000 7f9d82a00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d6806c920 0x7f9d6806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.867+0000 7f9d82a00640 1 -- 172.21.15.69:0/860715113 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f9d7c0ac700 con 0x7f9d700039b0 2024-04-24T08:09:13.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.867+0000 7f9d83e00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d6806c920 0x7f9d6806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:13.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.868+0000 7f9d83e00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d6806c920 0x7f9d6806ede0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0cfab0 tx=0x7f9d74008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:13.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:13.872+0000 7f9d82a00640 1 -- 172.21.15.69:0/860715113 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d7c07aa80 con 0x7f9d700039b0 2024-04-24T08:09:14.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.073+0000 7f9d91c00640 1 -- 172.21.15.69:0/860715113 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9d8c0ce940 con 0x7f9d700039b0 2024-04-24T08:09:14.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.074+0000 7f9d82a00640 1 -- 172.21.15.69:0/860715113 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f9d7c07a380 con 0x7f9d700039b0 2024-04-24T08:09:14.075 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:14.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.076+0000 7f9d91c00640 1 -- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d6806c920 msgr2=0x7f9d6806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.076+0000 7f9d91c00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d6806c920 0x7f9d6806ede0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0cfab0 tx=0x7f9d74008040 comp rx=0 tx=0).stop 2024-04-24T08:09:14.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.076+0000 7f9d91c00640 1 -- 172.21.15.69:0/860715113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 msgr2=0x7f9d8c151880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.076+0000 7f9d91c00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 0x7f9d8c151880 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c270 tx=0x7f9d8c073e90 comp rx=0 tx=0).stop 2024-04-24T08:09:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.076+0000 7f9d91c00640 1 -- 172.21.15.69:0/860715113 shutdown_connections 2024-04-24T08:09:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.076+0000 7f9d91c00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d6806c920 0x7f9d6806ede0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.076+0000 7f9d91c00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d8c1542c0 0x7f9d8c142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.076+0000 7f9d91c00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d8c153e40 0x7f9d8c1426d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.076+0000 7f9d91c00640 1 --2- 172.21.15.69:0/860715113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d700039b0 0x7f9d8c151880 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.076+0000 7f9d91c00640 1 -- 172.21.15.69:0/860715113 >> 172.21.15.69:0/860715113 conn(0x7f9d8c0d0350 msgr2=0x7f9d8c157310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.077+0000 7f9d91c00640 1 -- 172.21.15.69:0/860715113 shutdown_connections 2024-04-24T08:09:14.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.077+0000 7f9d91c00640 1 -- 172.21.15.69:0/860715113 wait complete. 2024-04-24T08:09:14.096 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:14.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.298+0000 7fd2d7200640 1 Processor -- start 2024-04-24T08:09:14.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.299+0000 7fd2d7200640 1 -- start start 2024-04-24T08:09:14.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.300+0000 7fd2d7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d015ea60 0x7fd2d00b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.300+0000 7fd2d7200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd2d00d09f0 0x7fd2d006df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.300+0000 7fd2d7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2d00ce910 con 0x7fd2d00cdf30 2024-04-24T08:09:14.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.300+0000 7fd2d7200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd2d00cebf0 con 0x7fd2d00d09f0 2024-04-24T08:09:14.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.300+0000 7fd2d7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d00ceed0 con 0x7fd2d015ea60 2024-04-24T08:09:14.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.301+0000 7fd2d5400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd2d00cdf30 0x7fd2d00d02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55522/0 (socket says 172.21.15.69:55522) 2024-04-24T08:09:14.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.301+0000 7fd2d5400640 1 -- 172.21.15.69:0/363095804 learned_addr learned my addr 172.21.15.69:0/363095804 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:14.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.301+0000 7fd2d6800640 1 --2- 172.21.15.69:0/363095804 >> v2:172.21.15.121:3300/0 conn(0x7fd2d00d09f0 0x7fd2d006df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.301+0000 7fd2d5e00640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d015ea60 0x7fd2d00b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.301+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102504359 0 0) 0x7fd2d00ce910 con 0x7fd2d00cdf30 2024-04-24T08:09:14.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.301+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2b8003520 con 0x7fd2d00cdf30 2024-04-24T08:09:14.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3602769195 0 0) 0x7fd2b8003520 con 0x7fd2d00cdf30 2024-04-24T08:09:14.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d015ea60 msgr2=0x7fd2d00b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:14.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d015ea60 0x7fd2d00b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 >> v2:172.21.15.121:3300/0 conn(0x7fd2d00d09f0 msgr2=0x7fd2d006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:14.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 --2- 172.21.15.69:0/363095804 >> v2:172.21.15.121:3300/0 conn(0x7fd2d00d09f0 0x7fd2d006df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d00cf1b0 con 0x7fd2d00cdf30 2024-04-24T08:09:14.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fd2bc001670 con 0x7fd2d00cdf30 2024-04-24T08:09:14.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 >> v1:172.21.15.69:6789/0 conn(0x7fd2d00cdf30 legacy=0x7fd2d00d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b80039b0 0x7fd2b8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d015ea60 0x7fd2b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8004d40 0x7fd2b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d00cebf0 con 0x7fd2b8004d40 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d00ceed0 con 0x7fd2b80039b0 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2b800dbd0 con 0x7fd2d015ea60 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d6800640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8004d40 0x7fd2b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.302+0000 7fd2d5e00640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b80039b0 0x7fd2b8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.303+0000 7fd2d5400640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d015ea60 0x7fd2b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.303+0000 7fd2d6800640 1 -- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d015ea60 msgr2=0x7fd2b8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.303+0000 7fd2d6800640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d015ea60 0x7fd2b8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.303+0000 7fd2d6800640 1 -- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b80039b0 msgr2=0x7fd2b8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.303+0000 7fd2d6800640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b80039b0 0x7fd2b8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.303+0000 7fd2d6800640 1 -- 172.21.15.69:0/363095804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b8003520 con 0x7fd2b8004d40 2024-04-24T08:09:14.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.304+0000 7fd2d6800640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8004d40 0x7fd2b80051c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009f90 tx=0x7fd2b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85f4a52ee97b57c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:14.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.304+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc009c10 con 0x7fd2b8004d40 2024-04-24T08:09:14.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.304+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2cc036070 con 0x7fd2b8004d40 2024-04-24T08:09:14.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.304+0000 7fd2d4a00640 1 -- 172.21.15.69:0/363095804 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc031880 con 0x7fd2b8004d40 2024-04-24T08:09:14.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.305+0000 7fd2d7200640 1 -- 172.21.15.69:0/363095804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8004d40 msgr2=0x7fd2b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.305+0000 7fd2d7200640 1 --2- 172.21.15.69:0/363095804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2b8004d40 0x7fd2b80051c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009f90 tx=0x7fd2b8005830 comp rx=0 tx=0).stop 2024-04-24T08:09:14.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.305+0000 7fd2d6800640 1 -- 172.21.15.69:0/363095804 reap_dead start 2024-04-24T08:09:14.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.305+0000 7fd2d7200640 1 -- 172.21.15.69:0/363095804 shutdown_connections 2024-04-24T08:09:14.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.305+0000 7fd2d7200640 1 -- 172.21.15.69:0/363095804 >> 172.21.15.69:0/363095804 conn(0x7fd2d0067b80 msgr2=0x7fd2d006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:14.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.305+0000 7fd2d7200640 1 -- 172.21.15.69:0/363095804 shutdown_connections 2024-04-24T08:09:14.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.305+0000 7fd2d7200640 1 -- 172.21.15.69:0/363095804 wait complete. 2024-04-24T08:09:14.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.307+0000 7fd2d7200640 1 Processor -- start 2024-04-24T08:09:14.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.307+0000 7fd2d7200640 1 -- start start 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.308+0000 7fd2d7200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b8004d40 0x7fd2d014fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.308+0000 7fd2d7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d00cbdb0 0x7fd2d00cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.308+0000 7fd2d7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d00add10 0x7fd2d00ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.308+0000 7fd2d7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d00ce510 con 0x7fd2d00add10 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.308+0000 7fd2d7200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d00ce390 con 0x7fd2b8004d40 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.308+0000 7fd2d7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2d00ce690 con 0x7fd2d00cbdb0 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.308+0000 7fd2d6800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d00add10 0x7fd2d00ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.308+0000 7fd2d6800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d00add10 0x7fd2d00ae190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36066/0 (socket says 172.21.15.69:36066) 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.308+0000 7fd2d6800640 1 -- 172.21.15.69:0/2509628288 learned_addr learned my addr 172.21.15.69:0/2509628288 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.308+0000 7fd2d5e00640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b8004d40 0x7fd2d014fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2d5400640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d00cbdb0 0x7fd2d00cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2d6800640 1 -- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d00cbdb0 msgr2=0x7fd2d00cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2d6800640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d00cbdb0 0x7fd2d00cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2d6800640 1 -- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b8004d40 msgr2=0x7fd2d014fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2d6800640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b8004d40 0x7fd2d014fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2d6800640 1 -- 172.21.15.69:0/2509628288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d00cc870 con 0x7fd2d00add10 2024-04-24T08:09:14.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2d6800640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d00add10 0x7fd2d00ae190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc033040 tx=0x7fd2cc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:14.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2ca000640 1 -- 172.21.15.69:0/2509628288 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc009c10 con 0x7fd2d00add10 2024-04-24T08:09:14.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2ca000640 1 -- 172.21.15.69:0/2509628288 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2cc031db0 con 0x7fd2d00add10 2024-04-24T08:09:14.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2ca000640 1 -- 172.21.15.69:0/2509628288 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc003bd0 con 0x7fd2d00add10 2024-04-24T08:09:14.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.309+0000 7fd2d7200640 1 -- 172.21.15.69:0/2509628288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d00ccb50 con 0x7fd2d00add10 2024-04-24T08:09:14.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.310+0000 7fd2d7200640 1 -- 172.21.15.69:0/2509628288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d00c94d0 con 0x7fd2d00add10 2024-04-24T08:09:14.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.311+0000 7fd2d7200640 1 -- 172.21.15.69:0/2509628288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2d0072ad0 con 0x7fd2d00add10 2024-04-24T08:09:14.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.312+0000 7fd2ca000640 1 -- 172.21.15.69:0/2509628288 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd2cc005d10 con 0x7fd2d00add10 2024-04-24T08:09:14.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.313+0000 7fd2ca000640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2b006c970 0x7fd2b006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.314+0000 7fd2ca000640 1 -- 172.21.15.69:0/2509628288 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fd2cc0ad1a0 con 0x7fd2d00add10 2024-04-24T08:09:14.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.314+0000 7fd2d5e00640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2b006c970 0x7fd2b006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.315+0000 7fd2d5e00640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2b006c970 0x7fd2b006ee30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd2d00aecb0 tx=0x7fd2c4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:14.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.320+0000 7fd2ca000640 1 -- 172.21.15.69:0/2509628288 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd2d0072ad0 con 0x7fd2d00add10 2024-04-24T08:09:14.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.521+0000 7fd2d7200640 1 -- 172.21.15.69:0/2509628288 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd2d0075d50 con 0x7fd2d00add10 2024-04-24T08:09:14.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.522+0000 7fd2ca000640 1 -- 172.21.15.69:0/2509628288 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7fd2cc04a060 con 0x7fd2d00add10 2024-04-24T08:09:14.523 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:14.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 -- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2b006c970 msgr2=0x7fd2b006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2b006c970 0x7fd2b006ee30 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd2d00aecb0 tx=0x7fd2c4008040 comp rx=0 tx=0).stop 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 -- 172.21.15.69:0/2509628288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d00add10 msgr2=0x7fd2d00ae190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d00add10 0x7fd2d00ae190 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc033040 tx=0x7fd2cc009750 comp rx=0 tx=0).stop 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 -- 172.21.15.69:0/2509628288 shutdown_connections 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2b006c970 0x7fd2b006ee30 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d00add10 0x7fd2d00ae190 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2d00cbdb0 0x7fd2d00cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 --2- 172.21.15.69:0/2509628288 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2b8004d40 0x7fd2d014fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 -- 172.21.15.69:0/2509628288 >> 172.21.15.69:0/2509628288 conn(0x7fd2d0067b80 msgr2=0x7fd2d00b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 -- 172.21.15.69:0/2509628288 shutdown_connections 2024-04-24T08:09:14.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.524+0000 7fd2d7200640 1 -- 172.21.15.69:0/2509628288 wait complete. 2024-04-24T08:09:14.545 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:09:14.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.747+0000 7f30b8800640 1 Processor -- start 2024-04-24T08:09:14.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.747+0000 7f30b8800640 1 -- start start 2024-04-24T08:09:14.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.749+0000 7f30b8800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f30b00d0300 0x7f30b006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.749+0000 7f30b8800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 0x7f30b00d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.749+0000 7f30b8800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30b00d0c20 con 0x7f30b0145fe0 2024-04-24T08:09:14.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.749+0000 7f30b8800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f30b00cf3d0 con 0x7f30b00d0300 2024-04-24T08:09:14.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.749+0000 7f30b8800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f30b00cf6b0 con 0x7f30b0145a90 2024-04-24T08:09:14.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.750+0000 7f30b5e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f30b00d0300 0x7f30b006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.750+0000 7f30b6800640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f30b0145fe0 0x7f30b0160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:58840/0 (socket says 172.21.15.69:58840) 2024-04-24T08:09:14.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.750+0000 7f30b5400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 0x7f30b00d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.750+0000 7f30b6800640 1 -- 172.21.15.69:0/2804011214 learned_addr learned my addr 172.21.15.69:0/2804011214 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:14.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.751+0000 7f30b4a00640 1 -- 172.21.15.69:0/2804011214 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690083639 0 0) 0x7f30b00d0c20 con 0x7f30b0145fe0 2024-04-24T08:09:14.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.751+0000 7f30b4a00640 1 -- 172.21.15.69:0/2804011214 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3098003520 con 0x7f30b0145fe0 2024-04-24T08:09:14.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.751+0000 7f30b5400640 1 -- 172.21.15.69:0/2804011214 >> v2:172.21.15.121:3300/0 conn(0x7f30b00d0300 msgr2=0x7f30b006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:14.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.751+0000 7f30b5400640 1 --2- 172.21.15.69:0/2804011214 >> v2:172.21.15.121:3300/0 conn(0x7f30b00d0300 0x7f30b006e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.751+0000 7f30b5400640 1 -- 172.21.15.69:0/2804011214 >> v1:172.21.15.121:6789/0 conn(0x7f30b0145fe0 legacy=0x7f30b0160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.751+0000 7f30b5400640 1 -- 172.21.15.69:0/2804011214 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b00cf990 con 0x7f30b0145a90 2024-04-24T08:09:14.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.751+0000 7f30b5e00640 1 --2- 172.21.15.69:0/2804011214 >> v2:172.21.15.121:3300/0 conn(0x7f30b00d0300 0x7f30b006e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:14.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.752+0000 7f30b5400640 1 --2- 172.21.15.69:0/2804011214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 0x7f30b00d06e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f30a00097a0 tx=0x7f30a002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35898cce5cd11968 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:14.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.753+0000 7f30b4a00640 1 -- 172.21.15.69:0/2804011214 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a0037070 con 0x7f30b0145a90 2024-04-24T08:09:14.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.753+0000 7f30b4a00640 1 -- 172.21.15.69:0/2804011214 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30a0032440 con 0x7f30b0145a90 2024-04-24T08:09:14.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.753+0000 7f30b4a00640 1 -- 172.21.15.69:0/2804011214 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30a00327f0 con 0x7f30b0145a90 2024-04-24T08:09:14.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.754+0000 7f30b8800640 1 -- 172.21.15.69:0/2804011214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 msgr2=0x7f30b00d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.754+0000 7f30b8800640 1 --2- 172.21.15.69:0/2804011214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 0x7f30b00d06e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f30a00097a0 tx=0x7f30a002a180 comp rx=0 tx=0).stop 2024-04-24T08:09:14.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.754+0000 7f30b8800640 1 -- 172.21.15.69:0/2804011214 shutdown_connections 2024-04-24T08:09:14.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.754+0000 7f30b8800640 1 --2- 172.21.15.69:0/2804011214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 0x7f30b00d06e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.754+0000 7f30b8800640 1 --2- 172.21.15.69:0/2804011214 >> v2:172.21.15.121:3300/0 conn(0x7f30b00d0300 0x7f30b006e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.754+0000 7f30b8800640 1 -- 172.21.15.69:0/2804011214 >> 172.21.15.69:0/2804011214 conn(0x7f30b0067d90 msgr2=0x7f30b006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:14.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.754+0000 7f30b8800640 1 -- 172.21.15.69:0/2804011214 shutdown_connections 2024-04-24T08:09:14.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.754+0000 7f30b8800640 1 -- 172.21.15.69:0/2804011214 wait complete. 2024-04-24T08:09:14.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.756+0000 7f30b8800640 1 Processor -- start 2024-04-24T08:09:14.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.756+0000 7f30b8800640 1 -- start start 2024-04-24T08:09:14.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b8800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30b00d0300 0x7f30b01f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b8800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 0x7f30b01f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b8800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30b0145fe0 0x7f30b014e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b8800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f30b0145770 con 0x7f30b0145fe0 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b8800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f30b00cff50 con 0x7f30b00d0300 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b8800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f30b01458f0 con 0x7f30b0145a90 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b6800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30b0145fe0 0x7f30b014e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b6800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30b0145fe0 0x7f30b014e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36070/0 (socket says 172.21.15.69:36070) 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b6800640 1 -- 172.21.15.69:0/605044126 learned_addr learned my addr 172.21.15.69:0/605044126 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b5e00640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30b00d0300 0x7f30b01f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.757+0000 7f30b5400640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 0x7f30b01f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30b6800640 1 -- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 msgr2=0x7f30b01f3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30b6800640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 0x7f30b01f3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30b6800640 1 -- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30b00d0300 msgr2=0x7f30b01f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30b6800640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30b00d0300 0x7f30b01f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30b6800640 1 -- 172.21.15.69:0/605044126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b01f3260 con 0x7f30b0145fe0 2024-04-24T08:09:14.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30b6800640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30b0145fe0 0x7f30b014e170 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f30ac00cb40 tx=0x7f30ac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:14.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30aa000640 1 -- 172.21.15.69:0/605044126 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac017070 con 0x7f30b0145fe0 2024-04-24T08:09:14.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30aa000640 1 -- 172.21.15.69:0/605044126 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30ac00aca0 con 0x7f30b0145fe0 2024-04-24T08:09:14.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30aa000640 1 -- 172.21.15.69:0/605044126 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30ac012620 con 0x7f30b0145fe0 2024-04-24T08:09:14.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30b8800640 1 -- 172.21.15.69:0/605044126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30b014d3b0 con 0x7f30b0145fe0 2024-04-24T08:09:14.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.758+0000 7f30b8800640 1 -- 172.21.15.69:0/605044126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30b014d970 con 0x7f30b0145fe0 2024-04-24T08:09:14.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.760+0000 7f30b8800640 1 -- 172.21.15.69:0/605044126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30b0068a80 con 0x7f30b0145fe0 2024-04-24T08:09:14.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.761+0000 7f30aa000640 1 -- 172.21.15.69:0/605044126 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f30ac01a070 con 0x7f30b0145fe0 2024-04-24T08:09:14.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.761+0000 7f30aa000640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f309406c8f0 0x7f309406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.762+0000 7f30aa000640 1 -- 172.21.15.69:0/605044126 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f30ac08bb30 con 0x7f30b0145fe0 2024-04-24T08:09:14.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.762+0000 7f30b5e00640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f309406c8f0 0x7f309406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.763+0000 7f30b5e00640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f309406c8f0 0x7f309406edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009a30 tx=0x7f30a4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:14.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.766+0000 7f30aa000640 1 -- 172.21.15.69:0/605044126 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f30b01f3260 con 0x7f30b0145fe0 2024-04-24T08:09:14.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.981+0000 7f30b8800640 1 -- 172.21.15.69:0/605044126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f30b00d1120 con 0x7f30b0145fe0 2024-04-24T08:09:14.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.982+0000 7f30aa000640 1 -- 172.21.15.69:0/605044126 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f30ac059e30 con 0x7f30b0145fe0 2024-04-24T08:09:14.983 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:09:14.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 -- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f309406c8f0 msgr2=0x7f309406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f309406c8f0 0x7f309406edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009a30 tx=0x7f30a4008040 comp rx=0 tx=0).stop 2024-04-24T08:09:14.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 -- 172.21.15.69:0/605044126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30b0145fe0 msgr2=0x7f30b014e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30b0145fe0 0x7f30b014e170 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f30ac00cb40 tx=0x7f30ac00a060 comp rx=0 tx=0).stop 2024-04-24T08:09:14.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 -- 172.21.15.69:0/605044126 shutdown_connections 2024-04-24T08:09:14.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f309406c8f0 0x7f309406edb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30b0145fe0 0x7f30b014e170 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30b0145a90 0x7f30b01f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 --2- 172.21.15.69:0/605044126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30b00d0300 0x7f30b01f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 -- 172.21.15.69:0/605044126 >> 172.21.15.69:0/605044126 conn(0x7f30b0067d90 msgr2=0x7f30b00d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:14.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 -- 172.21.15.69:0/605044126 shutdown_connections 2024-04-24T08:09:14.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:14.985+0000 7f30b8800640 1 -- 172.21.15.69:0/605044126 wait complete. 2024-04-24T08:09:15.006 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:09:15.006 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:15.057 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:15.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.254+0000 7f4a1ce00640 1 Processor -- start 2024-04-24T08:09:15.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.254+0000 7f4a1ce00640 1 -- start start 2024-04-24T08:09:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.255+0000 7f4a1ce00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4a180b30f0 0x7f4a18167470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.255+0000 7f4a1ce00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a1815d660 con 0x7f4a18156c60 2024-04-24T08:09:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.255+0000 7f4a1ce00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a1806dc30 con 0x7f4a1815d1d0 2024-04-24T08:09:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.255+0000 7f4a1ce00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4a1806df10 con 0x7f4a180b30f0 2024-04-24T08:09:15.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.255+0000 7f4a17e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4a180b30f0 0x7f4a18167470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:15.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.255+0000 7f4a17400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4a18156c60 0x7f4a1815cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55534/0 (socket says 172.21.15.69:55534) 2024-04-24T08:09:15.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.255+0000 7f4a17400640 1 -- 172.21.15.69:0/3381859929 learned_addr learned my addr 172.21.15.69:0/3381859929 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:15.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.256+0000 7f4a17e00640 1 -- 172.21.15.69:0/3381859929 >> v1:172.21.15.121:6789/0 conn(0x7f4a1815d1d0 legacy=0x7f4a180ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:15.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.256+0000 7f4a17e00640 1 -- 172.21.15.69:0/3381859929 >> v1:172.21.15.69:6789/0 conn(0x7f4a18156c60 legacy=0x7f4a1815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.256+0000 7f4a17e00640 1 -- 172.21.15.69:0/3381859929 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1806e1f0 con 0x7f4a180b30f0 2024-04-24T08:09:15.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.257+0000 7f4a17e00640 1 --2- 172.21.15.69:0/3381859929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a180b30f0 0x7f4a18167470 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009d90 tx=0x7f4a1002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82add5acba995af3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:15.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.257+0000 7f4a16000640 1 -- 172.21.15.69:0/3381859929 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a10007740 con 0x7f4a180b30f0 2024-04-24T08:09:15.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.257+0000 7f4a16000640 1 -- 172.21.15.69:0/3381859929 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a100078e0 con 0x7f4a180b30f0 2024-04-24T08:09:15.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.257+0000 7f4a16000640 1 -- 172.21.15.69:0/3381859929 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a10007cb0 con 0x7f4a180b30f0 2024-04-24T08:09:15.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.258+0000 7f4a1ce00640 1 -- 172.21.15.69:0/3381859929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a180b30f0 msgr2=0x7f4a18167470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.258+0000 7f4a1ce00640 1 --2- 172.21.15.69:0/3381859929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a180b30f0 0x7f4a18167470 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4a10009d90 tx=0x7f4a1002a6a0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.258+0000 7f4a1ce00640 1 -- 172.21.15.69:0/3381859929 shutdown_connections 2024-04-24T08:09:15.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.258+0000 7f4a1ce00640 1 --2- 172.21.15.69:0/3381859929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a180b30f0 0x7f4a18167470 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.258+0000 7f4a1ce00640 1 -- 172.21.15.69:0/3381859929 >> 172.21.15.69:0/3381859929 conn(0x7f4a180d0160 msgr2=0x7f4a180d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:15.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.258+0000 7f4a1ce00640 1 -- 172.21.15.69:0/3381859929 shutdown_connections 2024-04-24T08:09:15.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.258+0000 7f4a1ce00640 1 -- 172.21.15.69:0/3381859929 wait complete. 2024-04-24T08:09:15.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.259+0000 7f4a1ce00640 1 Processor -- start 2024-04-24T08:09:15.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.260+0000 7f4a1ce00640 1 -- start start 2024-04-24T08:09:15.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.260+0000 7f4a1ce00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a180b30f0 0x7f4a181f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.260+0000 7f4a1ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a18156c60 0x7f4a18155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.260+0000 7f4a1ce00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a18155950 0x7f4a180c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.260+0000 7f4a1ce00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a180ad7e0 con 0x7f4a18156c60 2024-04-24T08:09:15.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.260+0000 7f4a1ce00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1815d940 con 0x7f4a18155950 2024-04-24T08:09:15.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.260+0000 7f4a1ce00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a180ad960 con 0x7f4a180b30f0 2024-04-24T08:09:15.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.261+0000 7f4a16a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a18156c60 0x7f4a18155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:15.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.261+0000 7f4a16a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a18156c60 0x7f4a18155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36092/0 (socket says 172.21.15.69:36092) 2024-04-24T08:09:15.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.261+0000 7f4a16a00640 1 -- 172.21.15.69:0/2340634844 learned_addr learned my addr 172.21.15.69:0/2340634844 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:15.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.261+0000 7f4a17e00640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a18155950 0x7f4a180c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.261+0000 7f4a16a00640 1 -- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a180b30f0 msgr2=0x7f4a181f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.261+0000 7f4a16a00640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a180b30f0 0x7f4a181f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.261+0000 7f4a16a00640 1 -- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a18155950 msgr2=0x7f4a180c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.261+0000 7f4a16a00640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a18155950 0x7f4a180c9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.261+0000 7f4a16a00640 1 -- 172.21.15.69:0/2340634844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a180ca1a0 con 0x7f4a18156c60 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.261+0000 7f4a16a00640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a18156c60 0x7f4a18155410 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400b420 tx=0x7f4a0400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.262+0000 7f4a0f400640 1 -- 172.21.15.69:0/2340634844 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a04011830 con 0x7f4a18156c60 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.262+0000 7f4a0f400640 1 -- 172.21.15.69:0/2340634844 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a04016070 con 0x7f4a18156c60 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.262+0000 7f4a0f400640 1 -- 172.21.15.69:0/2340634844 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a04011c10 con 0x7f4a18156c60 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.262+0000 7f4a1ce00640 1 -- 172.21.15.69:0/2340634844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a180ca480 con 0x7f4a18156c60 2024-04-24T08:09:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.262+0000 7f4a1ce00640 1 -- 172.21.15.69:0/2340634844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a180caa40 con 0x7f4a18156c60 2024-04-24T08:09:15.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.263+0000 7f4a1ce00640 1 -- 172.21.15.69:0/2340634844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a180aeba0 con 0x7f4a18156c60 2024-04-24T08:09:15.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.264+0000 7f4a0f400640 1 -- 172.21.15.69:0/2340634844 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a0401b460 con 0x7f4a18156c60 2024-04-24T08:09:15.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.265+0000 7f4a0f400640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49f806c8f0 0x7f49f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.265+0000 7f4a0f400640 1 -- 172.21.15.69:0/2340634844 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f4a0408b7b0 con 0x7f4a18156c60 2024-04-24T08:09:15.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.265+0000 7f4a17400640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49f806c8f0 0x7f49f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:15.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.267+0000 7f4a17400640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49f806c8f0 0x7f49f806edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4a08000f60 tx=0x7f4a08007a80 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:15.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.270+0000 7f4a0f400640 1 -- 172.21.15.69:0/2340634844 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a180aeba0 con 0x7f4a18156c60 2024-04-24T08:09:15.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.468+0000 7f4a1ce00640 1 -- 172.21.15.69:0/2340634844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a18075d50 con 0x7f4a18156c60 2024-04-24T08:09:15.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.471+0000 7f4a0f400640 1 -- 172.21.15.69:0/2340634844 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7f4a04020020 con 0x7f4a18156c60 2024-04-24T08:09:15.473 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:15.473 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 -- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49f806c8f0 msgr2=0x7f49f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49f806c8f0 0x7f49f806edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4a08000f60 tx=0x7f4a08007a80 comp rx=0 tx=0).stop 2024-04-24T08:09:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 -- 172.21.15.69:0/2340634844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a18156c60 msgr2=0x7f4a18155410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a18156c60 0x7f4a18155410 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400b420 tx=0x7f4a0400b8e0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 -- 172.21.15.69:0/2340634844 shutdown_connections 2024-04-24T08:09:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49f806c8f0 0x7f49f806edb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a18156c60 0x7f4a18155410 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a18155950 0x7f4a180c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 --2- 172.21.15.69:0/2340634844 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a180b30f0 0x7f4a181f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 -- 172.21.15.69:0/2340634844 >> 172.21.15.69:0/2340634844 conn(0x7f4a180d0160 msgr2=0x7f4a1815b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.474+0000 7f4a1ce00640 1 -- 172.21.15.69:0/2340634844 shutdown_connections 2024-04-24T08:09:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.475+0000 7f4a1ce00640 1 -- 172.21.15.69:0/2340634844 wait complete. 2024-04-24T08:09:15.495 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:15.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:15.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.693+0000 7fa383e00640 1 Processor -- start 2024-04-24T08:09:15.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.694+0000 7fa383e00640 1 -- start start 2024-04-24T08:09:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.695+0000 7fa383e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38415e790 0x7fa3840d2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.695+0000 7fa383e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3840d4d20 0x7fa3840cf330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.695+0000 7fa383e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3840d12a0 con 0x7fa3840d27d0 2024-04-24T08:09:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.695+0000 7fa383e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa38406ca60 con 0x7fa3840d4d20 2024-04-24T08:09:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.695+0000 7fa383e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa3840d34d0 con 0x7fa38415e790 2024-04-24T08:09:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.696+0000 7fa383400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3840d4d20 0x7fa3840cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.696+0000 7fa383400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3840d4d20 0x7fa3840cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36096/0 (socket says 172.21.15.69:36096) 2024-04-24T08:09:15.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.696+0000 7fa383400640 1 -- 172.21.15.69:0/282957541 learned_addr learned my addr 172.21.15.69:0/282957541 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.696+0000 7fa382a00640 1 --2- 172.21.15.69:0/282957541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38415e790 0x7fa3840d2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.696+0000 7fa383400640 1 -- 172.21.15.69:0/282957541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38415e790 msgr2=0x7fa3840d2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.696+0000 7fa383400640 1 --2- 172.21.15.69:0/282957541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38415e790 0x7fa3840d2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.696+0000 7fa383400640 1 -- 172.21.15.69:0/282957541 >> v1:172.21.15.121:6789/0 conn(0x7fa3840d27d0 legacy=0x7fa3840d4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:15.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.696+0000 7fa383400640 1 -- 172.21.15.69:0/282957541 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3840d1540 con 0x7fa3840d4d20 2024-04-24T08:09:15.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.697+0000 7fa383400640 1 --2- 172.21.15.69:0/282957541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 0x7fa3840cf330 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009e50 tx=0x7fa37c02a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef42658723293594 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:15.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.698+0000 7fa381600640 1 -- 172.21.15.69:0/282957541 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c036070 con 0x7fa3840d4d20 2024-04-24T08:09:15.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.698+0000 7fa381600640 1 -- 172.21.15.69:0/282957541 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa37c031600 con 0x7fa3840d4d20 2024-04-24T08:09:15.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.698+0000 7fa381600640 1 -- 172.21.15.69:0/282957541 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa37c0319d0 con 0x7fa3840d4d20 2024-04-24T08:09:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.698+0000 7fa383e00640 1 -- 172.21.15.69:0/282957541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 msgr2=0x7fa3840cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.698+0000 7fa383e00640 1 --2- 172.21.15.69:0/282957541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 0x7fa3840cf330 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa37c009e50 tx=0x7fa37c02a6d0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.698+0000 7fa383e00640 1 -- 172.21.15.69:0/282957541 shutdown_connections 2024-04-24T08:09:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.698+0000 7fa383e00640 1 --2- 172.21.15.69:0/282957541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 0x7fa3840cf330 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.698+0000 7fa383e00640 1 --2- 172.21.15.69:0/282957541 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa38415e790 0x7fa3840d2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.699+0000 7fa383e00640 1 -- 172.21.15.69:0/282957541 >> 172.21.15.69:0/282957541 conn(0x7fa3840681e0 msgr2=0x7fa38409d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.699+0000 7fa383e00640 1 -- 172.21.15.69:0/282957541 shutdown_connections 2024-04-24T08:09:15.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.699+0000 7fa383e00640 1 -- 172.21.15.69:0/282957541 wait complete. 2024-04-24T08:09:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.701+0000 7fa383e00640 1 Processor -- start 2024-04-24T08:09:15.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.701+0000 7fa383e00640 1 -- start start 2024-04-24T08:09:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.702+0000 7fa383e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 0x7fa384156e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.702+0000 7fa383e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38415e790 0x7fa3840b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.702+0000 7fa383e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3840b54c0 0x7fa38415ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.702+0000 7fa383e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3840d1de0 con 0x7fa3840d4d20 2024-04-24T08:09:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.702+0000 7fa383e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa3840d1b00 con 0x7fa38415e790 2024-04-24T08:09:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.702+0000 7fa383e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa38415eb70 con 0x7fa3840b54c0 2024-04-24T08:09:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.702+0000 7fa382a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 0x7fa384156e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:15.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.702+0000 7fa382a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 0x7fa384156e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36102/0 (socket says 172.21.15.69:36102) 2024-04-24T08:09:15.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.702+0000 7fa382a00640 1 -- 172.21.15.69:0/2532342793 learned_addr learned my addr 172.21.15.69:0/2532342793 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:15.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.703+0000 7fa383400640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3840b54c0 0x7fa38415ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:15.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.703+0000 7fa382000640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38415e790 0x7fa3840b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:15.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.703+0000 7fa382a00640 1 -- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3840b54c0 msgr2=0x7fa38415ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.703+0000 7fa382a00640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3840b54c0 0x7fa38415ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.703+0000 7fa382a00640 1 -- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38415e790 msgr2=0x7fa3840b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.703+0000 7fa382a00640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38415e790 0x7fa3840b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.703+0000 7fa382a00640 1 -- 172.21.15.69:0/2532342793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa38415fac0 con 0x7fa3840d4d20 2024-04-24T08:09:15.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.703+0000 7fa382a00640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 0x7fa384156e90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa37400d5f0 tx=0x7fa37400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:15.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.703+0000 7fa37aa00640 1 -- 172.21.15.69:0/2532342793 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3740125f0 con 0x7fa3840d4d20 2024-04-24T08:09:15.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.704+0000 7fa37aa00640 1 -- 172.21.15.69:0/2532342793 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa374019070 con 0x7fa3840d4d20 2024-04-24T08:09:15.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.704+0000 7fa37aa00640 1 -- 172.21.15.69:0/2532342793 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3740129d0 con 0x7fa3840d4d20 2024-04-24T08:09:15.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.704+0000 7fa383e00640 1 -- 172.21.15.69:0/2532342793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa38415fda0 con 0x7fa3840d4d20 2024-04-24T08:09:15.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.704+0000 7fa383e00640 1 -- 172.21.15.69:0/2532342793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa384160360 con 0x7fa3840d4d20 2024-04-24T08:09:15.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.705+0000 7fa383e00640 1 -- 172.21.15.69:0/2532342793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3840b58d0 con 0x7fa3840d4d20 2024-04-24T08:09:15.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.706+0000 7fa37aa00640 1 -- 172.21.15.69:0/2532342793 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa374012b70 con 0x7fa3840d4d20 2024-04-24T08:09:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.707+0000 7fa37aa00640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa36006c8f0 0x7fa36006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:15.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.707+0000 7fa37aa00640 1 -- 172.21.15.69:0/2532342793 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fa37408c060 con 0x7fa3840d4d20 2024-04-24T08:09:15.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.708+0000 7fa382000640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa36006c8f0 0x7fa36006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:15.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.709+0000 7fa382000640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa36006c8f0 0x7fa36006edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa3840ba2b0 tx=0x7fa36c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:15.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.712+0000 7fa37aa00640 1 -- 172.21.15.69:0/2532342793 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3840b58d0 con 0x7fa3840d4d20 2024-04-24T08:09:15.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.903+0000 7fa383e00640 1 -- 172.21.15.69:0/2532342793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa384075d50 con 0x7fa3840d4d20 2024-04-24T08:09:15.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.904+0000 7fa37aa00640 1 -- 172.21.15.69:0/2532342793 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fa37405a400 con 0x7fa3840d4d20 2024-04-24T08:09:15.905 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:15.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.907+0000 7fa383e00640 1 -- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa36006c8f0 msgr2=0x7fa36006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.907+0000 7fa383e00640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa36006c8f0 0x7fa36006edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa3840ba2b0 tx=0x7fa36c009040 comp rx=0 tx=0).stop 2024-04-24T08:09:15.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.907+0000 7fa383e00640 1 -- 172.21.15.69:0/2532342793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 msgr2=0x7fa384156e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:15.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.907+0000 7fa383e00640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 0x7fa384156e90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa37400d5f0 tx=0x7fa37400dab0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.907+0000 7fa383e00640 1 -- 172.21.15.69:0/2532342793 shutdown_connections 2024-04-24T08:09:15.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.907+0000 7fa383e00640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa36006c8f0 0x7fa36006edb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.907+0000 7fa383e00640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3840b54c0 0x7fa38415ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.907+0000 7fa383e00640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa38415e790 0x7fa3840b4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.907+0000 7fa383e00640 1 --2- 172.21.15.69:0/2532342793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3840d4d20 0x7fa384156e90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:15.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.907+0000 7fa383e00640 1 -- 172.21.15.69:0/2532342793 >> 172.21.15.69:0/2532342793 conn(0x7fa3840681e0 msgr2=0x7fa38415dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:15.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.908+0000 7fa383e00640 1 -- 172.21.15.69:0/2532342793 shutdown_connections 2024-04-24T08:09:15.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:15.908+0000 7fa383e00640 1 -- 172.21.15.69:0/2532342793 wait complete. 2024-04-24T08:09:15.928 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:16.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.122+0000 7fb217e00640 1 Processor -- start 2024-04-24T08:09:16.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.123+0000 7fb217e00640 1 -- start start 2024-04-24T08:09:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.124+0000 7fb217e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21815e980 0x7fb218162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.124+0000 7fb217e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb2180d0da0 0x7fb2180cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.124+0000 7fb217e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb2180d0190 0x7fb21806cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.124+0000 7fb217e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb21815ed60 con 0x7fb2180d0190 2024-04-24T08:09:16.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.124+0000 7fb217e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb2180d1580 con 0x7fb2180d0da0 2024-04-24T08:09:16.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.124+0000 7fb217e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb2180d1860 con 0x7fb21815e980 2024-04-24T08:09:16.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.124+0000 7fb217400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb2180d0190 0x7fb21806cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.124+0000 7fb217400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb2180d0190 0x7fb21806cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36110/0 (socket says 172.21.15.69:36110) 2024-04-24T08:09:16.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.124+0000 7fb217400640 1 -- 172.21.15.69:0/3394621048 learned_addr learned my addr 172.21.15.69:0/3394621048 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.125+0000 7fb216000640 1 --2- 172.21.15.69:0/3394621048 >> v2:172.21.15.121:3300/0 conn(0x7fb2180d0da0 0x7fb2180cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.125+0000 7fb216a00640 1 --2- 172.21.15.69:0/3394621048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21815e980 0x7fb218162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.125+0000 7fb217400640 1 -- 172.21.15.69:0/3394621048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21815e980 msgr2=0x7fb218162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.125+0000 7fb217400640 1 --2- 172.21.15.69:0/3394621048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21815e980 0x7fb218162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.125+0000 7fb217400640 1 -- 172.21.15.69:0/3394621048 >> v2:172.21.15.121:3300/0 conn(0x7fb2180d0da0 msgr2=0x7fb2180cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.125+0000 7fb217400640 1 --2- 172.21.15.69:0/3394621048 >> v2:172.21.15.121:3300/0 conn(0x7fb2180d0da0 0x7fb2180cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.125+0000 7fb217400640 1 -- 172.21.15.69:0/3394621048 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2180cf450 con 0x7fb2180d0190 2024-04-24T08:09:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.126+0000 7fb217400640 1 --2- 172.21.15.69:0/3394621048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0190 0x7fb21806cfb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb210009e50 tx=0x7fb21002a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216e161c7f239f54 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.126+0000 7fb215600640 1 -- 172.21.15.69:0/3394621048 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210036070 con 0x7fb2180d0190 2024-04-24T08:09:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.126+0000 7fb215600640 1 -- 172.21.15.69:0/3394621048 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb210031600 con 0x7fb2180d0190 2024-04-24T08:09:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.126+0000 7fb215600640 1 -- 172.21.15.69:0/3394621048 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2100319d0 con 0x7fb2180d0190 2024-04-24T08:09:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.127+0000 7fb217e00640 1 -- 172.21.15.69:0/3394621048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0190 msgr2=0x7fb21806cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.127+0000 7fb217e00640 1 --2- 172.21.15.69:0/3394621048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0190 0x7fb21806cfb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb210009e50 tx=0x7fb21002a750 comp rx=0 tx=0).stop 2024-04-24T08:09:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.127+0000 7fb217e00640 1 -- 172.21.15.69:0/3394621048 shutdown_connections 2024-04-24T08:09:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.127+0000 7fb217e00640 1 --2- 172.21.15.69:0/3394621048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0190 0x7fb21806cfb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.127+0000 7fb217e00640 1 --2- 172.21.15.69:0/3394621048 >> v2:172.21.15.121:3300/0 conn(0x7fb2180d0da0 0x7fb2180cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.127+0000 7fb217e00640 1 --2- 172.21.15.69:0/3394621048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21815e980 0x7fb218162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.127+0000 7fb217e00640 1 -- 172.21.15.69:0/3394621048 >> 172.21.15.69:0/3394621048 conn(0x7fb21809d920 msgr2=0x7fb21808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.127+0000 7fb217e00640 1 -- 172.21.15.69:0/3394621048 shutdown_connections 2024-04-24T08:09:16.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.127+0000 7fb217e00640 1 -- 172.21.15.69:0/3394621048 wait complete. 2024-04-24T08:09:16.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.129+0000 7fb217e00640 1 Processor -- start 2024-04-24T08:09:16.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.129+0000 7fb217e00640 1 -- start start 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb217e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2180d0190 0x7fb2180b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb217e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0da0 0x7fb2180b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb217e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21815e980 0x7fb2181f49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb217e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2180d1360 con 0x7fb2180d0da0 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb217e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb2180d11e0 con 0x7fb2180d0190 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb217e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb2180cf070 con 0x7fb21815e980 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb216000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0da0 0x7fb2180b40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb216000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0da0 0x7fb2180b40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36120/0 (socket says 172.21.15.69:36120) 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb216000640 1 -- 172.21.15.69:0/2513520997 learned_addr learned my addr 172.21.15.69:0/2513520997 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb216a00640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2180d0190 0x7fb2180b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.130+0000 7fb217400640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21815e980 0x7fb2181f49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb216000640 1 -- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21815e980 msgr2=0x7fb2181f49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb216000640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21815e980 0x7fb2181f49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb216000640 1 -- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2180d0190 msgr2=0x7fb2180b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb216000640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2180d0190 0x7fb2180b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb216000640 1 -- 172.21.15.69:0/2513520997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21814eb20 con 0x7fb2180d0da0 2024-04-24T08:09:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb216000640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0da0 0x7fb2180b40d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb20000d800 tx=0x7fb20000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb20f400640 1 -- 172.21.15.69:0/2513520997 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20000b6c0 con 0x7fb2180d0da0 2024-04-24T08:09:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb20f400640 1 -- 172.21.15.69:0/2513520997 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb200019070 con 0x7fb2180d0da0 2024-04-24T08:09:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb20f400640 1 -- 172.21.15.69:0/2513520997 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20000ba80 con 0x7fb2180d0da0 2024-04-24T08:09:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb217e00640 1 -- 172.21.15.69:0/2513520997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2180b6050 con 0x7fb2180d0da0 2024-04-24T08:09:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.131+0000 7fb217e00640 1 -- 172.21.15.69:0/2513520997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2180b6610 con 0x7fb2180d0da0 2024-04-24T08:09:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.132+0000 7fb217e00640 1 -- 172.21.15.69:0/2513520997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb218075d50 con 0x7fb2180d0da0 2024-04-24T08:09:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.133+0000 7fb20f400640 1 -- 172.21.15.69:0/2513520997 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb20000bc20 con 0x7fb2180d0da0 2024-04-24T08:09:16.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.134+0000 7fb20f400640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f406c8f0 0x7fb1f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.134+0000 7fb20f400640 1 -- 172.21.15.69:0/2513520997 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fb20008c040 con 0x7fb2180d0da0 2024-04-24T08:09:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.135+0000 7fb216a00640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f406c8f0 0x7fb1f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.136+0000 7fb216a00640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f406c8f0 0x7fb1f406edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb208006fd0 tx=0x7fb208008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:16.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.138+0000 7fb20f400640 1 -- 172.21.15.69:0/2513520997 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb20001c360 con 0x7fb2180d0da0 2024-04-24T08:09:16.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.340+0000 7fb217e00640 1 -- 172.21.15.69:0/2513520997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb2180d1e40 con 0x7fb2180d0da0 2024-04-24T08:09:16.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.341+0000 7fb20f400640 1 -- 172.21.15.69:0/2513520997 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7fb20005a470 con 0x7fb2180d0da0 2024-04-24T08:09:16.342 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 -- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f406c8f0 msgr2=0x7fb1f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f406c8f0 0x7fb1f406edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb208006fd0 tx=0x7fb208008040 comp rx=0 tx=0).stop 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 -- 172.21.15.69:0/2513520997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0da0 msgr2=0x7fb2180b40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0da0 0x7fb2180b40d0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb20000d800 tx=0x7fb20000dcc0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 -- 172.21.15.69:0/2513520997 shutdown_connections 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f406c8f0 0x7fb1f406edb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2180d0da0 0x7fb2180b40d0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21815e980 0x7fb2181f49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 --2- 172.21.15.69:0/2513520997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb2180d0190 0x7fb2180b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 -- 172.21.15.69:0/2513520997 >> 172.21.15.69:0/2513520997 conn(0x7fb21809d920 msgr2=0x7fb2180b2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:16.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 -- 172.21.15.69:0/2513520997 shutdown_connections 2024-04-24T08:09:16.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.343+0000 7fb217e00640 1 -- 172.21.15.69:0/2513520997 wait complete. 2024-04-24T08:09:16.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:16.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.557+0000 7fafe0000640 1 Processor -- start 2024-04-24T08:09:16.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.557+0000 7fafe0000640 1 -- start start 2024-04-24T08:09:16.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.559+0000 7fafe0000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fafd80b3de0 0x7fafd80732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.559+0000 7fafe0000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafd8073820 0x7fafd80b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.559+0000 7fafe0000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafd8073cc0 con 0x7fafd815f4b0 2024-04-24T08:09:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.559+0000 7fafe0000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fafd8073fa0 con 0x7fafd80b3de0 2024-04-24T08:09:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.559+0000 7fafe0000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fafd80b3300 con 0x7fafd8073820 2024-04-24T08:09:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.559+0000 7fafdd600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fafd815f4b0 0x7fafd815f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55544/0 (socket says 172.21.15.69:55544) 2024-04-24T08:09:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.559+0000 7fafdd600640 1 -- 172.21.15.69:0/3965556698 learned_addr learned my addr 172.21.15.69:0/3965556698 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.559+0000 7fafde000640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafd8073820 0x7fafd80b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.559+0000 7fafdcc00640 1 --2- 172.21.15.69:0/3965556698 >> v2:172.21.15.121:3300/0 conn(0x7fafd80b3de0 0x7fafd80732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.559+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2327968876 0 0) 0x7fafd8073cc0 con 0x7fafd815f4b0 2024-04-24T08:09:16.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafc0003520 con 0x7fafd815f4b0 2024-04-24T08:09:16.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3658516823 0 0) 0x7fafc0003520 con 0x7fafd815f4b0 2024-04-24T08:09:16.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafd8073820 msgr2=0x7fafd80b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:16.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafd8073820 0x7fafd80b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 >> v2:172.21.15.121:3300/0 conn(0x7fafd80b3de0 msgr2=0x7fafd80732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:16.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 --2- 172.21.15.69:0/3965556698 >> v2:172.21.15.121:3300/0 conn(0x7fafd80b3de0 0x7fafd80732e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd80b35e0 con 0x7fafd815f4b0 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fafcc001670 con 0x7fafd815f4b0 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 >> v1:172.21.15.69:6789/0 conn(0x7fafd815f4b0 legacy=0x7fafd815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafd8073820 0x7fafc00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc0003f30 0x7fafc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafc0004d40 0x7fafc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fafd8073fa0 con 0x7fafc0004d40 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fafd80b3300 con 0x7fafc0003f30 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fafc000ba10 con 0x7fafd8073820 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.560+0000 7fafde000640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafc0004d40 0x7fafc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.561+0000 7fafdd600640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafd8073820 0x7fafc00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.561+0000 7fafdcc00640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc0003f30 0x7fafc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.561+0000 7fafde000640 1 -- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafd8073820 msgr2=0x7fafc00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:16.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.561+0000 7fafde000640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafd8073820 0x7fafc00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.561+0000 7fafde000640 1 -- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc0003f30 msgr2=0x7fafc0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:16.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.561+0000 7fafde000640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc0003f30 0x7fafc0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.561+0000 7fafde000640 1 -- 172.21.15.69:0/3965556698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc0003520 con 0x7fafc0004d40 2024-04-24T08:09:16.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.561+0000 7fafdcc00640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafc0003f30 0x7fafc0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:09:16.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.562+0000 7fafde000640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafc0004d40 0x7fafc00051c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009f90 tx=0x7fafc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfe2fe329fba0a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:16.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.562+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0009c10 con 0x7fafc0004d40 2024-04-24T08:09:16.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.563+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafd0036070 con 0x7fafc0004d40 2024-04-24T08:09:16.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.563+0000 7fafd7e00640 1 -- 172.21.15.69:0/3965556698 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0031810 con 0x7fafc0004d40 2024-04-24T08:09:16.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.563+0000 7fafe0000640 1 -- 172.21.15.69:0/3965556698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafc0004d40 msgr2=0x7fafc00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.563+0000 7fafe0000640 1 --2- 172.21.15.69:0/3965556698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafc0004d40 0x7fafc00051c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009f90 tx=0x7fafc0005830 comp rx=0 tx=0).stop 2024-04-24T08:09:16.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.563+0000 7fafde000640 1 -- 172.21.15.69:0/3965556698 reap_dead start 2024-04-24T08:09:16.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.563+0000 7fafe0000640 1 -- 172.21.15.69:0/3965556698 shutdown_connections 2024-04-24T08:09:16.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.563+0000 7fafe0000640 1 -- 172.21.15.69:0/3965556698 >> 172.21.15.69:0/3965556698 conn(0x7fafd80d0350 msgr2=0x7fafd80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:16.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.563+0000 7fafe0000640 1 -- 172.21.15.69:0/3965556698 shutdown_connections 2024-04-24T08:09:16.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.563+0000 7fafe0000640 1 -- 172.21.15.69:0/3965556698 wait complete. 2024-04-24T08:09:16.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.565+0000 7fafe0000640 1 Processor -- start 2024-04-24T08:09:16.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.565+0000 7fafe0000640 1 -- start start 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.566+0000 7fafe0000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafc0004d40 0x7fafd806f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.566+0000 7fafe0000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd8149780 0x7fafd8161670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.566+0000 7fafe0000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd8161bb0 0x7fafd80cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.566+0000 7fafe0000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fafd80b4820 con 0x7fafd8161bb0 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.566+0000 7fafe0000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fafd80b46a0 con 0x7fafd8149780 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.566+0000 7fafe0000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fafd80b49a0 con 0x7fafc0004d40 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.566+0000 7fafde000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd8161bb0 0x7fafd80cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.566+0000 7fafde000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd8161bb0 0x7fafd80cabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36136/0 (socket says 172.21.15.69:36136) 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.566+0000 7fafde000640 1 -- 172.21.15.69:0/2594474017 learned_addr learned my addr 172.21.15.69:0/2594474017 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.566+0000 7fafdd600640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafc0004d40 0x7fafd806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafdcc00640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd8149780 0x7fafd8161670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafde000640 1 -- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafc0004d40 msgr2=0x7fafd806f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafde000640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafc0004d40 0x7fafd806f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafde000640 1 -- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd8149780 msgr2=0x7fafd8161670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafde000640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd8149780 0x7fafd8161670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafde000640 1 -- 172.21.15.69:0/2594474017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd806fd30 con 0x7fafd8161bb0 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafde000640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd8161bb0 0x7fafd80cabc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009f90 tx=0x7fafd0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafd5600640 1 -- 172.21.15.69:0/2594474017 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0009c10 con 0x7fafd8161bb0 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafd5600640 1 -- 172.21.15.69:0/2594474017 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafd0045430 con 0x7fafd8161bb0 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafd5600640 1 -- 172.21.15.69:0/2594474017 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0045800 con 0x7fafd8161bb0 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.567+0000 7fafe0000640 1 -- 172.21.15.69:0/2594474017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd806fa50 con 0x7fafd8161bb0 2024-04-24T08:09:16.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.568+0000 7fafe0000640 1 -- 172.21.15.69:0/2594474017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd806e810 con 0x7fafd8161bb0 2024-04-24T08:09:16.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.569+0000 7fafe0000640 1 -- 172.21.15.69:0/2594474017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd8068a80 con 0x7fafd8161bb0 2024-04-24T08:09:16.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.570+0000 7fafd5600640 1 -- 172.21.15.69:0/2594474017 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fafd0005d10 con 0x7fafd8161bb0 2024-04-24T08:09:16.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.571+0000 7fafd5600640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafb806c8f0 0x7fafb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.571+0000 7fafd5600640 1 -- 172.21.15.69:0/2594474017 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fafd00ac960 con 0x7fafd8161bb0 2024-04-24T08:09:16.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.572+0000 7fafdd600640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafb806c8f0 0x7fafb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.573+0000 7fafdd600640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafb806c8f0 0x7fafb806edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fafcc001050 tx=0x7fafcc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:16.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.575+0000 7fafd5600640 1 -- 172.21.15.69:0/2594474017 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fafd002e050 con 0x7fafd8161bb0 2024-04-24T08:09:16.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.775+0000 7fafe0000640 1 -- 172.21.15.69:0/2594474017 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fafd8070610 con 0x7fafd8161bb0 2024-04-24T08:09:16.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.775+0000 7fafd5600640 1 -- 172.21.15.69:0/2594474017 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7fafd007abd0 con 0x7fafd8161bb0 2024-04-24T08:09:16.777 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:16.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 -- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafb806c8f0 msgr2=0x7fafb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafb806c8f0 0x7fafb806edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fafcc001050 tx=0x7fafcc009040 comp rx=0 tx=0).stop 2024-04-24T08:09:16.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 -- 172.21.15.69:0/2594474017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd8161bb0 msgr2=0x7fafd80cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd8161bb0 0x7fafd80cabc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009f90 tx=0x7fafd0009750 comp rx=0 tx=0).stop 2024-04-24T08:09:16.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 -- 172.21.15.69:0/2594474017 shutdown_connections 2024-04-24T08:09:16.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fafb806c8f0 0x7fafb806edb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafd8161bb0 0x7fafd80cabc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fafd8149780 0x7fafd8161670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 --2- 172.21.15.69:0/2594474017 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fafc0004d40 0x7fafd806f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 -- 172.21.15.69:0/2594474017 >> 172.21.15.69:0/2594474017 conn(0x7fafd80d0350 msgr2=0x7fafd8074b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:16.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 -- 172.21.15.69:0/2594474017 shutdown_connections 2024-04-24T08:09:16.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.778+0000 7fafe0000640 1 -- 172.21.15.69:0/2594474017 wait complete. 2024-04-24T08:09:16.797 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:09:16.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.989+0000 7f72fae00640 1 Processor -- start 2024-04-24T08:09:16.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.990+0000 7f72fae00640 1 -- start start 2024-04-24T08:09:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.992+0000 7f72fae00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415b490 0x7f72f40b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.992+0000 7f72fae00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f40b46e0 con 0x7f72f40d02f0 2024-04-24T08:09:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.992+0000 7f72fae00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72f40d0960 con 0x7f72f406f2d0 2024-04-24T08:09:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.992+0000 7f72fae00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72f40b4980 con 0x7f72f415b490 2024-04-24T08:09:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.992+0000 7f72f8e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f72f40d02f0 0x7f72f40ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55546/0 (socket says 172.21.15.69:55546) 2024-04-24T08:09:16.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.992+0000 7f72f8e00640 1 -- 172.21.15.69:0/734475276 learned_addr learned my addr 172.21.15.69:0/734475276 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.992+0000 7f72f3e00640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415b490 0x7f72f40b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2974663791 0 0) 0x7f72f40b46e0 con 0x7f72f40d02f0 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72d8003520 con 0x7f72f40d02f0 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3321912753 0 0) 0x7f72d8003520 con 0x7f72f40d02f0 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415b490 msgr2=0x7f72f40b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415b490 0x7f72f40b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 >> v1:172.21.15.121:6789/0 conn(0x7f72f406f2d0 legacy=0x7f72f40cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f40b4c60 con 0x7f72f40d02f0 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f72ec001670 con 0x7f72f40d02f0 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 >> v1:172.21.15.69:6789/0 conn(0x7f72f40d02f0 legacy=0x7f72f40ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 0x7f72d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415b490 0x7f72d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d8004d40 0x7f72d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72f40b4980 con 0x7f72d80039e0 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f72d800baa0 con 0x7f72d8004d40 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.993+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72d800deb0 con 0x7f72f415b490 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.994+0000 7f72f3e00640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 0x7f72d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.994+0000 7f72f8e00640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d8004d40 0x7f72d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.994+0000 7f72eb400640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415b490 0x7f72d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.994+0000 7f72f3e00640 1 -- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415b490 msgr2=0x7f72d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.994+0000 7f72f3e00640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415b490 0x7f72d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.994+0000 7f72f3e00640 1 -- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d8004d40 msgr2=0x7f72d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.994+0000 7f72f3e00640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d8004d40 0x7f72d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.994+0000 7f72f3e00640 1 -- 172.21.15.69:0/734475276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d8003520 con 0x7f72d80039e0 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.994+0000 7f72f8e00640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d8004d40 0x7f72d80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.994+0000 7f72eb400640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415b490 0x7f72d8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.995+0000 7f72f3e00640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 0x7f72d8003e00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f72e4009a90 tx=0x7f72d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a771d3bea6446983 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.995+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e4036070 con 0x7f72d80039e0 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.995+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72e4031400 con 0x7f72d80039e0 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.995+0000 7f72f3400640 1 -- 172.21.15.69:0/734475276 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e4009690 con 0x7f72d80039e0 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.996+0000 7f72fae00640 1 -- 172.21.15.69:0/734475276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 msgr2=0x7f72d8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.996+0000 7f72fae00640 1 --2- 172.21.15.69:0/734475276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 0x7f72d8003e00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f72e4009a90 tx=0x7f72d8005830 comp rx=0 tx=0).stop 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.996+0000 7f72f8e00640 1 -- 172.21.15.69:0/734475276 reap_dead start 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.996+0000 7f72fae00640 1 -- 172.21.15.69:0/734475276 shutdown_connections 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.996+0000 7f72fae00640 1 -- 172.21.15.69:0/734475276 >> 172.21.15.69:0/734475276 conn(0x7f72f4067d90 msgr2=0x7f72f406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.996+0000 7f72fae00640 1 -- 172.21.15.69:0/734475276 shutdown_connections 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.996+0000 7f72fae00640 1 -- 172.21.15.69:0/734475276 wait complete. 2024-04-24T08:09:16.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.998+0000 7f72fae00640 1 Processor -- start 2024-04-24T08:09:17.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.998+0000 7f72fae00640 1 -- start start 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.999+0000 7f72fae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 0x7f72f41411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.999+0000 7f72fae00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415a8a0 0x7f72f414e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.999+0000 7f72fae00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f415dfe0 0x7f72f415e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.999+0000 7f72fae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72f40ce7e0 con 0x7f72d80039e0 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.999+0000 7f72fae00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f72f415b870 con 0x7f72f415dfe0 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.999+0000 7f72fae00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72f40ce960 con 0x7f72f415a8a0 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.999+0000 7f72f3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 0x7f72f41411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.999+0000 7f72f3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 0x7f72f41411f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36164/0 (socket says 172.21.15.69:36164) 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:16.999+0000 7f72f3e00640 1 -- 172.21.15.69:0/594914617 learned_addr learned my addr 172.21.15.69:0/594914617 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72f3e00640 1 -- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415a8a0 msgr2=0x7f72f414e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72f3e00640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415a8a0 0x7f72f414e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72f3e00640 1 -- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f415dfe0 msgr2=0x7f72f415e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72f3e00640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f415dfe0 0x7f72f415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72f3e00640 1 -- 172.21.15.69:0/594914617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f41417d0 con 0x7f72d80039e0 2024-04-24T08:09:17.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72f3e00640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 0x7f72f41411f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f72e4034040 tx=0x7f72e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:17.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72f0c00640 1 -- 172.21.15.69:0/594914617 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e4040020 con 0x7f72d80039e0 2024-04-24T08:09:17.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72f0c00640 1 -- 172.21.15.69:0/594914617 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72e4003c40 con 0x7f72d80039e0 2024-04-24T08:09:17.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72fae00640 1 -- 172.21.15.69:0/594914617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f4141ab0 con 0x7f72d80039e0 2024-04-24T08:09:17.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72f0c00640 1 -- 172.21.15.69:0/594914617 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e4036040 con 0x7f72d80039e0 2024-04-24T08:09:17.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.000+0000 7f72fae00640 1 -- 172.21.15.69:0/594914617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f4142070 con 0x7f72d80039e0 2024-04-24T08:09:17.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.001+0000 7f72fae00640 1 -- 172.21.15.69:0/594914617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72f406cae0 con 0x7f72d80039e0 2024-04-24T08:09:17.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.002+0000 7f72f0c00640 1 -- 172.21.15.69:0/594914617 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f72e4043020 con 0x7f72d80039e0 2024-04-24T08:09:17.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.003+0000 7f72f0c00640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72d006c8f0 0x7f72d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:17.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.003+0000 7f72f0c00640 1 -- 172.21.15.69:0/594914617 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f72e40abe80 con 0x7f72d80039e0 2024-04-24T08:09:17.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.004+0000 7f72eb400640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72d006c8f0 0x7f72d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:17.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.005+0000 7f72eb400640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72d006c8f0 0x7f72d006edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f72dc007fd0 tx=0x7f72dc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:17.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.008+0000 7f72f0c00640 1 -- 172.21.15.69:0/594914617 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72f406cae0 con 0x7f72d80039e0 2024-04-24T08:09:17.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.204+0000 7f72fae00640 1 -- 172.21.15.69:0/594914617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f72f4068a80 con 0x7f72d80039e0 2024-04-24T08:09:17.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.205+0000 7f72f0c00640 1 -- 172.21.15.69:0/594914617 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7f72e404a070 con 0x7f72d80039e0 2024-04-24T08:09:17.206 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:09:17.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 -- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72d006c8f0 msgr2=0x7f72d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72d006c8f0 0x7f72d006edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f72dc007fd0 tx=0x7f72dc009040 comp rx=0 tx=0).stop 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 -- 172.21.15.69:0/594914617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 msgr2=0x7f72f41411f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 0x7f72f41411f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f72e4034040 tx=0x7f72e4004730 comp rx=0 tx=0).stop 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 -- 172.21.15.69:0/594914617 shutdown_connections 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72d006c8f0 0x7f72d006edb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f415dfe0 0x7f72f415e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72f415a8a0 0x7f72f414e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 --2- 172.21.15.69:0/594914617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d80039e0 0x7f72f41411f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 -- 172.21.15.69:0/594914617 >> 172.21.15.69:0/594914617 conn(0x7f72f4067d90 msgr2=0x7f72f40ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 -- 172.21.15.69:0/594914617 shutdown_connections 2024-04-24T08:09:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.208+0000 7f72fae00640 1 -- 172.21.15.69:0/594914617 wait complete. 2024-04-24T08:09:17.228 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:09:17.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:17.228 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-24T08:09:17.311 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:09:17.312 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:09:17.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:17.312 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-24T08:09:17.360 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:09:17.360 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:17.415 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:09:17.415 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:17.465 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:17.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.674+0000 7fddce400640 1 Processor -- start 2024-04-24T08:09:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.674+0000 7fddce400640 1 -- start start 2024-04-24T08:09:17.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.676+0000 7fddce400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fddc80d0300 0x7fddc806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:17.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.676+0000 7fddce400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddc8145fe0 0x7fddc8160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:17.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.676+0000 7fddce400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddc80cf3d0 con 0x7fddc8145a90 2024-04-24T08:09:17.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.676+0000 7fddce400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fddc80cf6b0 con 0x7fddc80d0300 2024-04-24T08:09:17.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.676+0000 7fddce400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fddc80cf990 con 0x7fddc8145fe0 2024-04-24T08:09:17.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.676+0000 7fddc6a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fddc8145a90 0x7fddc80d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55548/0 (socket says 172.21.15.69:55548) 2024-04-24T08:09:17.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.676+0000 7fddc6a00640 1 -- 172.21.15.69:0/4193989985 learned_addr learned my addr 172.21.15.69:0/4193989985 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:17.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.676+0000 7fddc7400640 1 --2- 172.21.15.69:0/4193989985 >> v2:172.21.15.69:3300/0 conn(0x7fddc80d0300 0x7fddc806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:17.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.677+0000 7fddc7e00640 1 --2- 172.21.15.69:0/4193989985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddc8145fe0 0x7fddc8160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:17.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.677+0000 7fddc7400640 1 -- 172.21.15.69:0/4193989985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddc8145fe0 msgr2=0x7fddc8160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:17.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.677+0000 7fddc7400640 1 --2- 172.21.15.69:0/4193989985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddc8145fe0 0x7fddc8160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.677+0000 7fddc6000640 1 -- 172.21.15.69:0/4193989985 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374767005 0 0) 0x7fddc80cf3d0 con 0x7fddc8145a90 2024-04-24T08:09:17.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.677+0000 7fddc7400640 1 -- 172.21.15.69:0/4193989985 >> v1:172.21.15.69:6789/0 conn(0x7fddc8145a90 legacy=0x7fddc80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.677+0000 7fddc7400640 1 -- 172.21.15.69:0/4193989985 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc80cfc70 con 0x7fddc80d0300 2024-04-24T08:09:17.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.678+0000 7fddc7400640 1 --2- 172.21.15.69:0/4193989985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc80d0300 0x7fddc806e240 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fddb80097a0 tx=0x7fddb802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68254a5d2ffe053 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:17.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.678+0000 7fddc6000640 1 -- 172.21.15.69:0/4193989985 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8037070 con 0x7fddc80d0300 2024-04-24T08:09:17.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.678+0000 7fddc6000640 1 -- 172.21.15.69:0/4193989985 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddb8032440 con 0x7fddc80d0300 2024-04-24T08:09:17.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.678+0000 7fddc6000640 1 -- 172.21.15.69:0/4193989985 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8032810 con 0x7fddc80d0300 2024-04-24T08:09:17.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.679+0000 7fddce400640 1 -- 172.21.15.69:0/4193989985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc80d0300 msgr2=0x7fddc806e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.679+0000 7fddce400640 1 --2- 172.21.15.69:0/4193989985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc80d0300 0x7fddc806e240 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fddb80097a0 tx=0x7fddb802a180 comp rx=0 tx=0).stop 2024-04-24T08:09:17.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.679+0000 7fddce400640 1 -- 172.21.15.69:0/4193989985 shutdown_connections 2024-04-24T08:09:17.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.679+0000 7fddce400640 1 --2- 172.21.15.69:0/4193989985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddc8145fe0 0x7fddc8160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.679+0000 7fddce400640 1 --2- 172.21.15.69:0/4193989985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc80d0300 0x7fddc806e240 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.679+0000 7fddce400640 1 -- 172.21.15.69:0/4193989985 >> 172.21.15.69:0/4193989985 conn(0x7fddc8067d90 msgr2=0x7fddc806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:17.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.679+0000 7fddce400640 1 -- 172.21.15.69:0/4193989985 shutdown_connections 2024-04-24T08:09:17.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.679+0000 7fddce400640 1 -- 172.21.15.69:0/4193989985 wait complete. 2024-04-24T08:09:17.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.681+0000 7fddce400640 1 Processor -- start 2024-04-24T08:09:17.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.681+0000 7fddce400640 1 -- start start 2024-04-24T08:09:17.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddce400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddc80d0300 0x7fddc80b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:17.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddce400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc8145a90 0x7fddc815c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:17.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddce400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddc8145fe0 0x7fddc80b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:17.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddce400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fddc81458f0 con 0x7fddc8145a90 2024-04-24T08:09:17.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddce400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fddc8145770 con 0x7fddc8145fe0 2024-04-24T08:09:17.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddce400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fddc80cdc30 con 0x7fddc80d0300 2024-04-24T08:09:17.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddc6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc8145a90 0x7fddc815c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:17.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddc6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc8145a90 0x7fddc815c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36180/0 (socket says 172.21.15.69:36180) 2024-04-24T08:09:17.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddc6a00640 1 -- 172.21.15.69:0/1356891002 learned_addr learned my addr 172.21.15.69:0/1356891002 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:17.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddc7400640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddc80d0300 0x7fddc80b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:17.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.682+0000 7fddc7e00640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddc8145fe0 0x7fddc80b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:17.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddc6a00640 1 -- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddc80d0300 msgr2=0x7fddc80b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddc6a00640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddc80d0300 0x7fddc80b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddc6a00640 1 -- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddc8145fe0 msgr2=0x7fddc80b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddc6a00640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddc8145fe0 0x7fddc80b4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddc6a00640 1 -- 172.21.15.69:0/1356891002 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc815c750 con 0x7fddc8145a90 2024-04-24T08:09:17.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddc6a00640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc8145a90 0x7fddc815c210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fddb400c4d0 tx=0x7fddb400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:17.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddbf400640 1 -- 172.21.15.69:0/1356891002 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb4018070 con 0x7fddc8145a90 2024-04-24T08:09:17.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddbf400640 1 -- 172.21.15.69:0/1356891002 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddb4007500 con 0x7fddc8145a90 2024-04-24T08:09:17.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddce400640 1 -- 172.21.15.69:0/1356891002 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc80b3a00 con 0x7fddc8145a90 2024-04-24T08:09:17.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddbf400640 1 -- 172.21.15.69:0/1356891002 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb4004a20 con 0x7fddc8145a90 2024-04-24T08:09:17.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.683+0000 7fddce400640 1 -- 172.21.15.69:0/1356891002 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc80b3fc0 con 0x7fddc8145a90 2024-04-24T08:09:17.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.685+0000 7fddce400640 1 -- 172.21.15.69:0/1356891002 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc8068a80 con 0x7fddc8145a90 2024-04-24T08:09:17.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.685+0000 7fddbf400640 1 -- 172.21.15.69:0/1356891002 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fddb401b070 con 0x7fddc8145a90 2024-04-24T08:09:17.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.686+0000 7fddbf400640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdda806c8f0 0x7fdda806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:17.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.686+0000 7fddbf400640 1 -- 172.21.15.69:0/1356891002 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fddb408aeb0 con 0x7fddc8145a90 2024-04-24T08:09:17.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.687+0000 7fddc7400640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdda806c8f0 0x7fdda806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:17.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.688+0000 7fddc7400640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdda806c8f0 0x7fdda806edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fddb8034000 tx=0x7fddb802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:17.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.691+0000 7fddbf400640 1 -- 172.21.15.69:0/1356891002 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fddc815c750 con 0x7fddc8145a90 2024-04-24T08:09:17.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.896+0000 7fddce400640 1 -- 172.21.15.69:0/1356891002 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fddc80d1120 con 0x7fddc8145a90 2024-04-24T08:09:17.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.897+0000 7fddbf400640 1 -- 172.21.15.69:0/1356891002 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7fddb40592e0 con 0x7fddc8145a90 2024-04-24T08:09:17.899 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:17.899 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:17.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.900+0000 7fddce400640 1 -- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdda806c8f0 msgr2=0x7fdda806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.900+0000 7fddce400640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdda806c8f0 0x7fdda806edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fddb8034000 tx=0x7fddb802e000 comp rx=0 tx=0).stop 2024-04-24T08:09:17.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.900+0000 7fddce400640 1 -- 172.21.15.69:0/1356891002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc8145a90 msgr2=0x7fddc815c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.900+0000 7fddce400640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc8145a90 0x7fddc815c210 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fddb400c4d0 tx=0x7fddb400c990 comp rx=0 tx=0).stop 2024-04-24T08:09:17.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.901+0000 7fddce400640 1 -- 172.21.15.69:0/1356891002 shutdown_connections 2024-04-24T08:09:17.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.901+0000 7fddce400640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdda806c8f0 0x7fdda806edb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.901+0000 7fddce400640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddc8145a90 0x7fddc815c210 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.901+0000 7fddce400640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fddc8145fe0 0x7fddc80b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.901+0000 7fddce400640 1 --2- 172.21.15.69:0/1356891002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fddc80d0300 0x7fddc80b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.901+0000 7fddce400640 1 -- 172.21.15.69:0/1356891002 >> 172.21.15.69:0/1356891002 conn(0x7fddc8067d90 msgr2=0x7fddc80b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:17.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.901+0000 7fddce400640 1 -- 172.21.15.69:0/1356891002 shutdown_connections 2024-04-24T08:09:17.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:17.901+0000 7fddce400640 1 -- 172.21.15.69:0/1356891002 wait complete. 2024-04-24T08:09:17.922 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:17.923 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:18.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.115+0000 7f3177200640 1 Processor -- start 2024-04-24T08:09:18.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.115+0000 7f3177200640 1 -- start start 2024-04-24T08:09:18.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.117+0000 7f3177200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f317815e980 0x7f3178162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.117+0000 7f3177200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31780d0da0 0x7f31780cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.117+0000 7f3177200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f317815ed60 con 0x7f31780d0190 2024-04-24T08:09:18.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.117+0000 7f3177200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f31780d1580 con 0x7f317815e980 2024-04-24T08:09:18.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.117+0000 7f3177200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f31780d1860 con 0x7f31780d0da0 2024-04-24T08:09:18.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.117+0000 7f3175e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f317815e980 0x7f3178162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.117+0000 7f3176800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f31780d0190 0x7f317806cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55562/0 (socket says 172.21.15.69:55562) 2024-04-24T08:09:18.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.117+0000 7f3176800640 1 -- 172.21.15.69:0/41993559 learned_addr learned my addr 172.21.15.69:0/41993559 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:18.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.118+0000 7f316fe00640 1 --2- 172.21.15.69:0/41993559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31780d0da0 0x7f31780cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.118+0000 7f3175e00640 1 -- 172.21.15.69:0/41993559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31780d0da0 msgr2=0x7f31780cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:18.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.118+0000 7f3175e00640 1 --2- 172.21.15.69:0/41993559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31780d0da0 0x7f31780cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.118+0000 7f3175e00640 1 -- 172.21.15.69:0/41993559 >> v1:172.21.15.69:6789/0 conn(0x7f31780d0190 legacy=0x7f317806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.118+0000 7f3175e00640 1 -- 172.21.15.69:0/41993559 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31780cf450 con 0x7f317815e980 2024-04-24T08:09:18.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.118+0000 7f3175400640 1 -- 172.21.15.69:0/41993559 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374844500 0 0) 0x7f317815ed60 con 0x7f31780d0190 2024-04-24T08:09:18.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.119+0000 7f3175e00640 1 --2- 172.21.15.69:0/41993559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f317815e980 0x7f3178162f90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f31680097d0 tx=0x7f316802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8eb4c825c54aab server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:18.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.119+0000 7f3175400640 1 -- 172.21.15.69:0/41993559 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168037070 con 0x7f317815e980 2024-04-24T08:09:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.119+0000 7f3175400640 1 -- 172.21.15.69:0/41993559 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3168032440 con 0x7f317815e980 2024-04-24T08:09:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.119+0000 7f3175400640 1 -- 172.21.15.69:0/41993559 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168032810 con 0x7f317815e980 2024-04-24T08:09:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.120+0000 7f3177200640 1 -- 172.21.15.69:0/41993559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f317815e980 msgr2=0x7f3178162f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.120+0000 7f3177200640 1 --2- 172.21.15.69:0/41993559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f317815e980 0x7f3178162f90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f31680097d0 tx=0x7f316802a250 comp rx=0 tx=0).stop 2024-04-24T08:09:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.120+0000 7f3177200640 1 -- 172.21.15.69:0/41993559 shutdown_connections 2024-04-24T08:09:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.120+0000 7f3177200640 1 --2- 172.21.15.69:0/41993559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31780d0da0 0x7f31780cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.120+0000 7f3177200640 1 --2- 172.21.15.69:0/41993559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f317815e980 0x7f3178162f90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.120+0000 7f3177200640 1 -- 172.21.15.69:0/41993559 >> 172.21.15.69:0/41993559 conn(0x7f317809d920 msgr2=0x7f317808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.120+0000 7f3177200640 1 -- 172.21.15.69:0/41993559 shutdown_connections 2024-04-24T08:09:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.120+0000 7f3177200640 1 -- 172.21.15.69:0/41993559 wait complete. 2024-04-24T08:09:18.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.122+0000 7f3177200640 1 Processor -- start 2024-04-24T08:09:18.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.122+0000 7f3177200640 1 -- start start 2024-04-24T08:09:18.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.123+0000 7f3177200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0190 0x7f31781525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.123+0000 7f3177200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31780d0da0 0x7f31780ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.123+0000 7f3177200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f317815e980 0x7f31780c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.123+0000 7f3177200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31780d1360 con 0x7f31780d0190 2024-04-24T08:09:18.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.123+0000 7f3177200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f31780d11e0 con 0x7f31780d0da0 2024-04-24T08:09:18.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.123+0000 7f3177200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f31780cf070 con 0x7f317815e980 2024-04-24T08:09:18.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.123+0000 7f3175e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0190 0x7f31781525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.123+0000 7f3175e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0190 0x7f31781525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36200/0 (socket says 172.21.15.69:36200) 2024-04-24T08:09:18.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.123+0000 7f3175e00640 1 -- 172.21.15.69:0/2361881181 learned_addr learned my addr 172.21.15.69:0/2361881181 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:18.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.124+0000 7f3175e00640 1 -- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f317815e980 msgr2=0x7f31780c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.124+0000 7f3175e00640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f317815e980 0x7f31780c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.124+0000 7f3175e00640 1 -- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31780d0da0 msgr2=0x7f31780ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.124+0000 7f3175e00640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31780d0da0 0x7f31780ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.124+0000 7f3175e00640 1 -- 172.21.15.69:0/2361881181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31780ce560 con 0x7f31780d0190 2024-04-24T08:09:18.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.124+0000 7f3175e00640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0190 0x7f31781525c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3168035040 tx=0x7f31680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:18.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.124+0000 7f316e000640 1 -- 172.21.15.69:0/2361881181 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168037040 con 0x7f31780d0190 2024-04-24T08:09:18.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.124+0000 7f316e000640 1 -- 172.21.15.69:0/2361881181 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3168030500 con 0x7f31780d0190 2024-04-24T08:09:18.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.124+0000 7f316e000640 1 -- 172.21.15.69:0/2361881181 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168030910 con 0x7f31780d0190 2024-04-24T08:09:18.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.124+0000 7f3177200640 1 -- 172.21.15.69:0/2361881181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31780c8b90 con 0x7f31780d0190 2024-04-24T08:09:18.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.125+0000 7f3177200640 1 -- 172.21.15.69:0/2361881181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31780ccf80 con 0x7f31780d0190 2024-04-24T08:09:18.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.126+0000 7f3177200640 1 -- 172.21.15.69:0/2361881181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3178075d50 con 0x7f31780d0190 2024-04-24T08:09:18.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.127+0000 7f316e000640 1 -- 172.21.15.69:0/2361881181 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3168044020 con 0x7f31780d0190 2024-04-24T08:09:18.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.128+0000 7f316e000640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f315406c8f0 0x7f315406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.128+0000 7f316e000640 1 -- 172.21.15.69:0/2361881181 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f31680abeb0 con 0x7f31780d0190 2024-04-24T08:09:18.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.128+0000 7f316fe00640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f315406c8f0 0x7f315406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.129+0000 7f316fe00640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f315406c8f0 0x7f315406edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3160006a90 tx=0x7f3160006a20 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:18.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.132+0000 7f316e000640 1 -- 172.21.15.69:0/2361881181 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f316807a250 con 0x7f31780d0190 2024-04-24T08:09:18.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.330+0000 7f3177200640 1 -- 172.21.15.69:0/2361881181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f31780d1e40 con 0x7f31780d0190 2024-04-24T08:09:18.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.331+0000 7f316e000640 1 -- 172.21.15.69:0/2361881181 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f3168079b50 con 0x7f31780d0190 2024-04-24T08:09:18.332 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 -- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f315406c8f0 msgr2=0x7f315406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f315406c8f0 0x7f315406edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3160006a90 tx=0x7f3160006a20 comp rx=0 tx=0).stop 2024-04-24T08:09:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 -- 172.21.15.69:0/2361881181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0190 msgr2=0x7f31781525c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0190 0x7f31781525c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3168035040 tx=0x7f31680046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 -- 172.21.15.69:0/2361881181 shutdown_connections 2024-04-24T08:09:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f315406c8f0 0x7f315406edb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f317815e980 0x7f31780c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31780d0da0 0x7f31780ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 --2- 172.21.15.69:0/2361881181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31780d0190 0x7f31781525c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 -- 172.21.15.69:0/2361881181 >> 172.21.15.69:0/2361881181 conn(0x7f317809d920 msgr2=0x7f317808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:18.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 -- 172.21.15.69:0/2361881181 shutdown_connections 2024-04-24T08:09:18.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.333+0000 7f3177200640 1 -- 172.21.15.69:0/2361881181 wait complete. 2024-04-24T08:09:18.353 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:18.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.554+0000 7f4a3be00640 1 Processor -- start 2024-04-24T08:09:18.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.555+0000 7f4a3be00640 1 -- start start 2024-04-24T08:09:18.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.556+0000 7f4a3be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 0x7f4a3c162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.556+0000 7f4a3be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4a3c0d0da0 0x7f4a3c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.556+0000 7f4a3be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4a3c0d0190 0x7f4a3c06cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.556+0000 7f4a3be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4a3c15ed60 con 0x7f4a3c0d0190 2024-04-24T08:09:18.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.556+0000 7f4a3be00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4a3c0d1580 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.556+0000 7f4a3be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a3c0d1860 con 0x7f4a3c15e980 2024-04-24T08:09:18.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.556+0000 7f4a3b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4a3c0d0190 0x7f4a3c06cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.556+0000 7f4a3b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4a3c0d0190 0x7f4a3c06cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36210/0 (socket says 172.21.15.69:36210) 2024-04-24T08:09:18.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.557+0000 7f4a3b400640 1 -- 172.21.15.69:0/3417465848 learned_addr learned my addr 172.21.15.69:0/3417465848 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:18.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.557+0000 7f4a33e00640 1 --2- 172.21.15.69:0/3417465848 >> v2:172.21.15.121:3300/0 conn(0x7f4a3c0d0da0 0x7f4a3c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.557+0000 7f4a3aa00640 1 --2- 172.21.15.69:0/3417465848 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 0x7f4a3c162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.557+0000 7f4a3b400640 1 -- 172.21.15.69:0/3417465848 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 msgr2=0x7f4a3c162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:18.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.557+0000 7f4a3b400640 1 --2- 172.21.15.69:0/3417465848 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 0x7f4a3c162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.557+0000 7f4a3b400640 1 -- 172.21.15.69:0/3417465848 >> v2:172.21.15.121:3300/0 conn(0x7f4a3c0d0da0 msgr2=0x7f4a3c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:18.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.557+0000 7f4a3b400640 1 --2- 172.21.15.69:0/3417465848 >> v2:172.21.15.121:3300/0 conn(0x7f4a3c0d0da0 0x7f4a3c0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.557+0000 7f4a3b400640 1 -- 172.21.15.69:0/3417465848 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c0cf450 con 0x7f4a3c0d0190 2024-04-24T08:09:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.557+0000 7f4a3aa00640 1 --2- 172.21.15.69:0/3417465848 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 0x7f4a3c162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.557+0000 7f4a33e00640 1 --2- 172.21.15.69:0/3417465848 >> v2:172.21.15.121:3300/0 conn(0x7f4a3c0d0da0 0x7f4a3c0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.558+0000 7f4a3b400640 1 --2- 172.21.15.69:0/3417465848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0190 0x7f4a3c06cfb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4a34009e50 tx=0x7f4a3402a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e22a4e5d071ba2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.558+0000 7f4a3a000640 1 -- 172.21.15.69:0/3417465848 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a34036070 con 0x7f4a3c0d0190 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.558+0000 7f4a3a000640 1 -- 172.21.15.69:0/3417465848 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a34031600 con 0x7f4a3c0d0190 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.558+0000 7f4a3a000640 1 -- 172.21.15.69:0/3417465848 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a340319d0 con 0x7f4a3c0d0190 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.559+0000 7f4a3be00640 1 -- 172.21.15.69:0/3417465848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0190 msgr2=0x7f4a3c06cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.559+0000 7f4a3be00640 1 --2- 172.21.15.69:0/3417465848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0190 0x7f4a3c06cfb0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4a34009e50 tx=0x7f4a3402a760 comp rx=0 tx=0).stop 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.559+0000 7f4a3be00640 1 -- 172.21.15.69:0/3417465848 shutdown_connections 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.559+0000 7f4a3be00640 1 --2- 172.21.15.69:0/3417465848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0190 0x7f4a3c06cfb0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.559+0000 7f4a3be00640 1 --2- 172.21.15.69:0/3417465848 >> v2:172.21.15.121:3300/0 conn(0x7f4a3c0d0da0 0x7f4a3c0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.559+0000 7f4a3be00640 1 --2- 172.21.15.69:0/3417465848 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 0x7f4a3c162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.559+0000 7f4a3be00640 1 -- 172.21.15.69:0/3417465848 >> 172.21.15.69:0/3417465848 conn(0x7f4a3c09d920 msgr2=0x7f4a3c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.559+0000 7f4a3be00640 1 -- 172.21.15.69:0/3417465848 shutdown_connections 2024-04-24T08:09:18.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.559+0000 7f4a3be00640 1 -- 172.21.15.69:0/3417465848 wait complete. 2024-04-24T08:09:18.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.561+0000 7f4a3be00640 1 Processor -- start 2024-04-24T08:09:18.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.561+0000 7f4a3be00640 1 -- start start 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.562+0000 7f4a3be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3c0d0190 0x7f4a3c0b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.562+0000 7f4a3be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0da0 0x7f4a3c1f3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.562+0000 7f4a3be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 0x7f4a3c0b9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.562+0000 7f4a3be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c0d1360 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.562+0000 7f4a3be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3c0d11e0 con 0x7f4a3c0d0190 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.562+0000 7f4a3be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a3c0cf070 con 0x7f4a3c15e980 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.562+0000 7f4a33e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0da0 0x7f4a3c1f3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.562+0000 7f4a33e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0da0 0x7f4a3c1f3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36222/0 (socket says 172.21.15.69:36222) 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.562+0000 7f4a33e00640 1 -- 172.21.15.69:0/1267204212 learned_addr learned my addr 172.21.15.69:0/1267204212 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a3b400640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 0x7f4a3c0b9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a3aa00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3c0d0190 0x7f4a3c0b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a33e00640 1 -- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 msgr2=0x7f4a3c0b9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a33e00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 0x7f4a3c0b9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a33e00640 1 -- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3c0d0190 msgr2=0x7f4a3c0b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a33e00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3c0d0190 0x7f4a3c0b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a33e00640 1 -- 172.21.15.69:0/1267204212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3c0b56f0 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a3aa00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3c0d0190 0x7f4a3c0b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a33e00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0da0 0x7f4a3c1f3920 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400c7f0 tx=0x7f4a2400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a32a00640 1 -- 172.21.15.69:0/1267204212 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a24015070 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a32a00640 1 -- 172.21.15.69:0/1267204212 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a240079d0 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.563+0000 7f4a32a00640 1 -- 172.21.15.69:0/1267204212 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a24007da0 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.564+0000 7f4a3be00640 1 -- 172.21.15.69:0/1267204212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3c0b5990 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.564+0000 7f4a3be00640 1 -- 172.21.15.69:0/1267204212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3c0b5f50 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.565+0000 7f4a3be00640 1 -- 172.21.15.69:0/1267204212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a3c075d50 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.566+0000 7f4a32a00640 1 -- 172.21.15.69:0/1267204212 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a24018070 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.567+0000 7f4a32a00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a1806c8f0 0x7f4a1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.567+0000 7f4a32a00640 1 -- 172.21.15.69:0/1267204212 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f4a2408bbc0 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.567+0000 7f4a3aa00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a1806c8f0 0x7f4a1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.568+0000 7f4a3aa00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a1806c8f0 0x7f4a1806edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0088b0 tx=0x7f4a2c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.571+0000 7f4a32a00640 1 -- 172.21.15.69:0/1267204212 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a24059ec0 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.768+0000 7f4a3be00640 1 -- 172.21.15.69:0/1267204212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4a3c0008d0 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.769+0000 7f4a32a00640 1 -- 172.21.15.69:0/1267204212 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f4a240597c0 con 0x7f4a3c0d0da0 2024-04-24T08:09:18.770 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:18.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 -- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a1806c8f0 msgr2=0x7f4a1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a1806c8f0 0x7f4a1806edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0088b0 tx=0x7f4a2c008040 comp rx=0 tx=0).stop 2024-04-24T08:09:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 -- 172.21.15.69:0/1267204212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0da0 msgr2=0x7f4a3c1f3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0da0 0x7f4a3c1f3920 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4a2400c7f0 tx=0x7f4a2400cbf0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 -- 172.21.15.69:0/1267204212 shutdown_connections 2024-04-24T08:09:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a1806c8f0 0x7f4a1806edb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a3c0d0da0 0x7f4a3c1f3920 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a3c15e980 0x7f4a3c0b9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 --2- 172.21.15.69:0/1267204212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a3c0d0190 0x7f4a3c0b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 -- 172.21.15.69:0/1267204212 >> 172.21.15.69:0/1267204212 conn(0x7f4a3c09d920 msgr2=0x7f4a3c0cd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 -- 172.21.15.69:0/1267204212 shutdown_connections 2024-04-24T08:09:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.772+0000 7f4a3be00640 1 -- 172.21.15.69:0/1267204212 wait complete. 2024-04-24T08:09:18.791 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:18.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.995+0000 7f3d29c00640 1 Processor -- start 2024-04-24T08:09:18.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.995+0000 7f3d29c00640 1 -- start start 2024-04-24T08:09:18.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.996+0000 7f3d29c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3d241420a0 0x7f3d24071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.996+0000 7f3d29c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3d24142480 0x7f3d2415ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:18.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.996+0000 7f3d29c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d24160920 con 0x7f3d24160510 2024-04-24T08:09:18.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.996+0000 7f3d29c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3d240cf1e0 con 0x7f3d24142480 2024-04-24T08:09:18.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.996+0000 7f3d29c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3d24072450 con 0x7f3d241420a0 2024-04-24T08:09:18.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.997+0000 7f3d1be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3d24142480 0x7f3d2415ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.997+0000 7f3d1be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3d24142480 0x7f3d2415ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43496/0 (socket says 172.21.15.69:43496) 2024-04-24T08:09:18.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.997+0000 7f3d1be00640 1 -- 172.21.15.69:0/1878776428 learned_addr learned my addr 172.21.15.69:0/1878776428 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:18.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.997+0000 7f3d23e00640 1 --2- 172.21.15.69:0/1878776428 >> v2:172.21.15.121:3300/0 conn(0x7f3d241420a0 0x7f3d24071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:18.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.997+0000 7f3d1be00640 1 -- 172.21.15.69:0/1878776428 >> v2:172.21.15.121:3300/0 conn(0x7f3d241420a0 msgr2=0x7f3d24071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:18.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.997+0000 7f3d1be00640 1 --2- 172.21.15.69:0/1878776428 >> v2:172.21.15.121:3300/0 conn(0x7f3d241420a0 0x7f3d24071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:18.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.997+0000 7f3d1be00640 1 -- 172.21.15.69:0/1878776428 >> v1:172.21.15.121:6789/0 conn(0x7f3d24160510 legacy=0x7f3d2415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:18.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.997+0000 7f3d1be00640 1 -- 172.21.15.69:0/1878776428 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d24072730 con 0x7f3d24142480 2024-04-24T08:09:19.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.998+0000 7f3d1be00640 1 --2- 172.21.15.69:0/1878776428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 0x7f3d2415ffd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3d240b5770 tx=0x7f3d0c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7b7ee3e1a4be32 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:19.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.999+0000 7f3d23400640 1 -- 172.21.15.69:0/1878776428 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c037070 con 0x7f3d24142480 2024-04-24T08:09:19.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.999+0000 7f3d23400640 1 -- 172.21.15.69:0/1878776428 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d0c032440 con 0x7f3d24142480 2024-04-24T08:09:19.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.999+0000 7f3d23400640 1 -- 172.21.15.69:0/1878776428 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c032810 con 0x7f3d24142480 2024-04-24T08:09:19.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.999+0000 7f3d29c00640 1 -- 172.21.15.69:0/1878776428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 msgr2=0x7f3d2415ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:18.999+0000 7f3d29c00640 1 --2- 172.21.15.69:0/1878776428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 0x7f3d2415ffd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3d240b5770 tx=0x7f3d0c02a230 comp rx=0 tx=0).stop 2024-04-24T08:09:19.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.000+0000 7f3d29c00640 1 -- 172.21.15.69:0/1878776428 shutdown_connections 2024-04-24T08:09:19.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.000+0000 7f3d29c00640 1 --2- 172.21.15.69:0/1878776428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 0x7f3d2415ffd0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.000+0000 7f3d29c00640 1 --2- 172.21.15.69:0/1878776428 >> v2:172.21.15.121:3300/0 conn(0x7f3d241420a0 0x7f3d24071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.000+0000 7f3d29c00640 1 -- 172.21.15.69:0/1878776428 >> 172.21.15.69:0/1878776428 conn(0x7f3d24067b80 msgr2=0x7f3d2406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:19.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.000+0000 7f3d29c00640 1 -- 172.21.15.69:0/1878776428 shutdown_connections 2024-04-24T08:09:19.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.000+0000 7f3d29c00640 1 -- 172.21.15.69:0/1878776428 wait complete. 2024-04-24T08:09:19.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.001+0000 7f3d29c00640 1 Processor -- start 2024-04-24T08:09:19.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.002+0000 7f3d29c00640 1 -- start start 2024-04-24T08:09:19.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.002+0000 7f3d29c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d241420a0 0x7f3d241532f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.002+0000 7f3d29c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 0x7f3d2413ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d29c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d24160510 0x7f3d2406efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d29c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3d24142910 con 0x7f3d24142480 2024-04-24T08:09:19.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d29c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d24072cf0 con 0x7f3d241420a0 2024-04-24T08:09:19.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d29c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d240d0680 con 0x7f3d24160510 2024-04-24T08:09:19.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d1be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 0x7f3d2413ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d1be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 0x7f3d2413ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43512/0 (socket says 172.21.15.69:43512) 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d1be00640 1 -- 172.21.15.69:0/137567442 learned_addr learned my addr 172.21.15.69:0/137567442 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d23e00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d241420a0 0x7f3d241532f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d29200640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d24160510 0x7f3d2406efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d1be00640 1 -- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d24160510 msgr2=0x7f3d2406efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d1be00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d24160510 0x7f3d2406efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d1be00640 1 -- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d241420a0 msgr2=0x7f3d241532f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d1be00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d241420a0 0x7f3d241532f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d1be00640 1 -- 172.21.15.69:0/137567442 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d24154fe0 con 0x7f3d24142480 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.003+0000 7f3d23e00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d241420a0 0x7f3d241532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.004+0000 7f3d1be00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 0x7f3d2413ecb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c000c00 tx=0x7f3d0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.004+0000 7f3d20c00640 1 -- 172.21.15.69:0/137567442 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c041020 con 0x7f3d24142480 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.004+0000 7f3d20c00640 1 -- 172.21.15.69:0/137567442 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d0c0306c0 con 0x7f3d24142480 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.004+0000 7f3d20c00640 1 -- 172.21.15.69:0/137567442 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0c037040 con 0x7f3d24142480 2024-04-24T08:09:19.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.004+0000 7f3d29c00640 1 -- 172.21.15.69:0/137567442 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d241552c0 con 0x7f3d24142480 2024-04-24T08:09:19.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.004+0000 7f3d29c00640 1 -- 172.21.15.69:0/137567442 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d24155880 con 0x7f3d24142480 2024-04-24T08:09:19.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.005+0000 7f3d29c00640 1 -- 172.21.15.69:0/137567442 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d24075d50 con 0x7f3d24142480 2024-04-24T08:09:19.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.006+0000 7f3d20c00640 1 -- 172.21.15.69:0/137567442 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d0c044020 con 0x7f3d24142480 2024-04-24T08:09:19.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.007+0000 7f3d20c00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d0006c8f0 0x7f3d0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.007+0000 7f3d20c00640 1 -- 172.21.15.69:0/137567442 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f3d0c0ac450 con 0x7f3d24142480 2024-04-24T08:09:19.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.008+0000 7f3d23e00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d0006c8f0 0x7f3d0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.009+0000 7f3d23e00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d0006c8f0 0x7f3d0006edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3d14006fd0 tx=0x7f3d14008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:19.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.012+0000 7f3d20c00640 1 -- 172.21.15.69:0/137567442 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d0c04b030 con 0x7f3d24142480 2024-04-24T08:09:19.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.203+0000 7f3d29c00640 1 -- 172.21.15.69:0/137567442 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3d240afbb0 con 0x7f3d24142480 2024-04-24T08:09:19.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.204+0000 7f3d20c00640 1 -- 172.21.15.69:0/137567442 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f3d24160920 con 0x7f3d24142480 2024-04-24T08:09:19.206 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:19.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 -- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d0006c8f0 msgr2=0x7f3d0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d0006c8f0 0x7f3d0006edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3d14006fd0 tx=0x7f3d14008040 comp rx=0 tx=0).stop 2024-04-24T08:09:19.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 -- 172.21.15.69:0/137567442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 msgr2=0x7f3d2413ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 0x7f3d2413ecb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c000c00 tx=0x7f3d0c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 -- 172.21.15.69:0/137567442 shutdown_connections 2024-04-24T08:09:19.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3d0006c8f0 0x7f3d0006edb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d24142480 0x7f3d2413ecb0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d24160510 0x7f3d2406efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 --2- 172.21.15.69:0/137567442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d241420a0 0x7f3d241532f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 -- 172.21.15.69:0/137567442 >> 172.21.15.69:0/137567442 conn(0x7f3d24067b80 msgr2=0x7f3d2406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:19.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 -- 172.21.15.69:0/137567442 shutdown_connections 2024-04-24T08:09:19.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.207+0000 7f3d29c00640 1 -- 172.21.15.69:0/137567442 wait complete. 2024-04-24T08:09:19.226 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:09:19.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.418+0000 7fc4a5400640 1 Processor -- start 2024-04-24T08:09:19.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.418+0000 7fc4a5400640 1 -- start start 2024-04-24T08:09:19.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.419+0000 7fc4a5400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4a00b41c0 0x7fc4a006e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.419+0000 7fc4a5400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a01669d0 con 0x7fc4a0162c70 2024-04-24T08:09:19.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.419+0000 7fc4a5400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a0166cb0 con 0x7fc4a00b3d30 2024-04-24T08:09:19.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.419+0000 7fc4a5400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc4a0166f90 con 0x7fc4a00b41c0 2024-04-24T08:09:19.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.420+0000 7fc49e800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc4a0162c70 0x7fc4a00b3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52596/0 (socket says 172.21.15.69:52596) 2024-04-24T08:09:19.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.420+0000 7fc49e800640 1 -- 172.21.15.69:0/785446259 learned_addr learned my addr 172.21.15.69:0/785446259 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:19.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.420+0000 7fc49f200640 1 --2- 172.21.15.69:0/785446259 >> v2:172.21.15.69:3300/0 conn(0x7fc4a00b41c0 0x7fc4a006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.420+0000 7fc49de00640 1 -- 172.21.15.69:0/785446259 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1092144810 0 0) 0x7fc4a01669d0 con 0x7fc4a0162c70 2024-04-24T08:09:19.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.420+0000 7fc49f200640 1 -- 172.21.15.69:0/785446259 >> v1:172.21.15.121:6789/0 conn(0x7fc4a00b3d30 legacy=0x7fc4a006e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:19.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.420+0000 7fc49f200640 1 -- 172.21.15.69:0/785446259 >> v1:172.21.15.69:6789/0 conn(0x7fc4a0162c70 legacy=0x7fc4a00b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.420+0000 7fc49f200640 1 -- 172.21.15.69:0/785446259 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a0167270 con 0x7fc4a00b41c0 2024-04-24T08:09:19.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.421+0000 7fc49f200640 1 --2- 172.21.15.69:0/785446259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a00b41c0 0x7fc4a006e800 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc4980097a0 tx=0x7fc49802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec9a56635f0c88e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:19.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.421+0000 7fc49de00640 1 -- 172.21.15.69:0/785446259 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498037070 con 0x7fc4a00b41c0 2024-04-24T08:09:19.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.422+0000 7fc49de00640 1 -- 172.21.15.69:0/785446259 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc498032440 con 0x7fc4a00b41c0 2024-04-24T08:09:19.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.422+0000 7fc49de00640 1 -- 172.21.15.69:0/785446259 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498032810 con 0x7fc4a00b41c0 2024-04-24T08:09:19.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.422+0000 7fc4a5400640 1 -- 172.21.15.69:0/785446259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a00b41c0 msgr2=0x7fc4a006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.422+0000 7fc4a5400640 1 --2- 172.21.15.69:0/785446259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a00b41c0 0x7fc4a006e800 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc4980097a0 tx=0x7fc49802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.422+0000 7fc4a5400640 1 -- 172.21.15.69:0/785446259 shutdown_connections 2024-04-24T08:09:19.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.422+0000 7fc4a5400640 1 --2- 172.21.15.69:0/785446259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a00b41c0 0x7fc4a006e800 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.422+0000 7fc4a5400640 1 -- 172.21.15.69:0/785446259 >> 172.21.15.69:0/785446259 conn(0x7fc4a0067d90 msgr2=0x7fc4a006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:19.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.422+0000 7fc4a5400640 1 -- 172.21.15.69:0/785446259 shutdown_connections 2024-04-24T08:09:19.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.422+0000 7fc4a5400640 1 -- 172.21.15.69:0/785446259 wait complete. 2024-04-24T08:09:19.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.424+0000 7fc4a5400640 1 Processor -- start 2024-04-24T08:09:19.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.424+0000 7fc4a5400640 1 -- start start 2024-04-24T08:09:19.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.425+0000 7fc4a5400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4a00b3d30 0x7fc4a00b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.425+0000 7fc4a5400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4a00b41c0 0x7fc4a0163690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.425+0000 7fc4a5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a0162c70 0x7fc4a0150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.425+0000 7fc4a5400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a00b45f0 con 0x7fc4a0162c70 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.425+0000 7fc4a5400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a0163050 con 0x7fc4a00b3d30 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.425+0000 7fc4a5400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a00cfc40 con 0x7fc4a00b41c0 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.425+0000 7fc49f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a0162c70 0x7fc4a0150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.425+0000 7fc49f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a0162c70 0x7fc4a0150b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43526/0 (socket says 172.21.15.69:43526) 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.425+0000 7fc49f200640 1 -- 172.21.15.69:0/1325381963 learned_addr learned my addr 172.21.15.69:0/1325381963 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc49e800640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4a00b3d30 0x7fc4a00b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc49f200640 1 -- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4a00b41c0 msgr2=0x7fc4a0163690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc49f200640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4a00b41c0 0x7fc4a0163690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc49f200640 1 -- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4a00b3d30 msgr2=0x7fc4a00b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc49f200640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4a00b3d30 0x7fc4a00b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc49f200640 1 -- 172.21.15.69:0/1325381963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a0163bd0 con 0x7fc4a0162c70 2024-04-24T08:09:19.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc49e800640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4a00b3d30 0x7fc4a00b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:19.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc49f200640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a0162c70 0x7fc4a0150b00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc498035040 tx=0x7fc4980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:19.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc497400640 1 -- 172.21.15.69:0/1325381963 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498037040 con 0x7fc4a0162c70 2024-04-24T08:09:19.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc497400640 1 -- 172.21.15.69:0/1325381963 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc498004230 con 0x7fc4a0162c70 2024-04-24T08:09:19.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc4a5400640 1 -- 172.21.15.69:0/1325381963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a015e190 con 0x7fc4a0162c70 2024-04-24T08:09:19.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.426+0000 7fc497400640 1 -- 172.21.15.69:0/1325381963 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498032e20 con 0x7fc4a0162c70 2024-04-24T08:09:19.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.427+0000 7fc4a5400640 1 -- 172.21.15.69:0/1325381963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a015cdc0 con 0x7fc4a0162c70 2024-04-24T08:09:19.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.428+0000 7fc4a5400640 1 -- 172.21.15.69:0/1325381963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a0068a80 con 0x7fc4a0162c70 2024-04-24T08:09:19.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.429+0000 7fc497400640 1 -- 172.21.15.69:0/1325381963 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc498030770 con 0x7fc4a0162c70 2024-04-24T08:09:19.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.429+0000 7fc497400640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc47c06c8f0 0x7fc47c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.430+0000 7fc497400640 1 -- 172.21.15.69:0/1325381963 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fc4980acf00 con 0x7fc4a0162c70 2024-04-24T08:09:19.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.430+0000 7fc49e800640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc47c06c8f0 0x7fc47c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.431+0000 7fc49e800640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc47c06c8f0 0x7fc47c06edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc490007fd0 tx=0x7fc490009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.434+0000 7fc497400640 1 -- 172.21.15.69:0/1325381963 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc49807b200 con 0x7fc4a0162c70 2024-04-24T08:09:19.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.638+0000 7fc4a5400640 1 -- 172.21.15.69:0/1325381963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc4a006c640 con 0x7fc4a0162c70 2024-04-24T08:09:19.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.639+0000 7fc497400640 1 -- 172.21.15.69:0/1325381963 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7fc49807ab00 con 0x7fc4a0162c70 2024-04-24T08:09:19.640 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:09:19.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.641+0000 7fc4a5400640 1 -- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc47c06c8f0 msgr2=0x7fc47c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.641+0000 7fc4a5400640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc47c06c8f0 0x7fc47c06edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc490007fd0 tx=0x7fc490009040 comp rx=0 tx=0).stop 2024-04-24T08:09:19.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.641+0000 7fc4a5400640 1 -- 172.21.15.69:0/1325381963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a0162c70 msgr2=0x7fc4a0150b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.641+0000 7fc4a5400640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a0162c70 0x7fc4a0150b00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc498035040 tx=0x7fc4980046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.642+0000 7fc4a5400640 1 -- 172.21.15.69:0/1325381963 shutdown_connections 2024-04-24T08:09:19.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.642+0000 7fc4a5400640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc47c06c8f0 0x7fc47c06edb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.642+0000 7fc4a5400640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4a0162c70 0x7fc4a0150b00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.642+0000 7fc4a5400640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4a00b41c0 0x7fc4a0163690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.642+0000 7fc4a5400640 1 --2- 172.21.15.69:0/1325381963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4a00b3d30 0x7fc4a00b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.642+0000 7fc4a5400640 1 -- 172.21.15.69:0/1325381963 >> 172.21.15.69:0/1325381963 conn(0x7fc4a0067d90 msgr2=0x7fc4a00cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:19.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.642+0000 7fc4a5400640 1 -- 172.21.15.69:0/1325381963 shutdown_connections 2024-04-24T08:09:19.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.642+0000 7fc4a5400640 1 -- 172.21.15.69:0/1325381963 wait complete. 2024-04-24T08:09:19.661 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:09:19.662 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:19.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.922+0000 7f8b1d600640 1 Processor -- start 2024-04-24T08:09:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.922+0000 7f8b1d600640 1 -- start start 2024-04-24T08:09:19.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.924+0000 7f8b1d600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8b18156c60 0x7f8b1815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.924+0000 7f8b1d600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b1815d660 con 0x7f8b180ad7e0 2024-04-24T08:09:19.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.924+0000 7f8b1d600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b1806dc30 con 0x7f8b180b30f0 2024-04-24T08:09:19.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.924+0000 7f8b1d600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8b1806df10 con 0x7f8b18156c60 2024-04-24T08:09:19.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.924+0000 7f8b17e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8b18156c60 0x7f8b1815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.924+0000 7f8b1cc00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8b180ad7e0 0x7f8b1806e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52598/0 (socket says 172.21.15.69:52598) 2024-04-24T08:09:19.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.924+0000 7f8b1cc00640 1 -- 172.21.15.69:0/911944009 learned_addr learned my addr 172.21.15.69:0/911944009 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:19.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.925+0000 7f8b17e00640 1 -- 172.21.15.69:0/911944009 >> v1:172.21.15.121:6789/0 conn(0x7f8b180b30f0 legacy=0x7f8b180ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:19.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.925+0000 7f8b17e00640 1 -- 172.21.15.69:0/911944009 >> v1:172.21.15.69:6789/0 conn(0x7f8b180ad7e0 legacy=0x7f8b1806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.925+0000 7f8b17e00640 1 -- 172.21.15.69:0/911944009 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1806e1f0 con 0x7f8b18156c60 2024-04-24T08:09:19.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.925+0000 7f8b16a00640 1 -- 172.21.15.69:0/911944009 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544770364 0 0) 0x7f8b1815d660 con 0x7f8b180ad7e0 2024-04-24T08:09:19.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.926+0000 7f8b17e00640 1 --2- 172.21.15.69:0/911944009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b18156c60 0x7f8b1815cac0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8b080097a0 tx=0x7f8b0802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86f3b35bb7689fc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:19.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.926+0000 7f8b16a00640 1 -- 172.21.15.69:0/911944009 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08037070 con 0x7f8b18156c60 2024-04-24T08:09:19.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.926+0000 7f8b16a00640 1 -- 172.21.15.69:0/911944009 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b08032440 con 0x7f8b18156c60 2024-04-24T08:09:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.926+0000 7f8b16a00640 1 -- 172.21.15.69:0/911944009 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08032810 con 0x7f8b18156c60 2024-04-24T08:09:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.926+0000 7f8b1d600640 1 -- 172.21.15.69:0/911944009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b18156c60 msgr2=0x7f8b1815cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.926+0000 7f8b1d600640 1 --2- 172.21.15.69:0/911944009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b18156c60 0x7f8b1815cac0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8b080097a0 tx=0x7f8b0802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.927+0000 7f8b1d600640 1 -- 172.21.15.69:0/911944009 shutdown_connections 2024-04-24T08:09:19.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.927+0000 7f8b1d600640 1 --2- 172.21.15.69:0/911944009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b18156c60 0x7f8b1815cac0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.927+0000 7f8b1d600640 1 -- 172.21.15.69:0/911944009 >> 172.21.15.69:0/911944009 conn(0x7f8b180d0160 msgr2=0x7f8b180d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:19.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.927+0000 7f8b1d600640 1 -- 172.21.15.69:0/911944009 shutdown_connections 2024-04-24T08:09:19.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.927+0000 7f8b1d600640 1 -- 172.21.15.69:0/911944009 wait complete. 2024-04-24T08:09:19.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.928+0000 7f8b1d600640 1 Processor -- start 2024-04-24T08:09:19.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.929+0000 7f8b1d600640 1 -- start start 2024-04-24T08:09:19.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.929+0000 7f8b1d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b180ad7e0 0x7f8b180cb830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.929+0000 7f8b1d600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b180b30f0 0x7f8b18155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.929+0000 7f8b1d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b18156c60 0x7f8b181578c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.929+0000 7f8b1d600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1815d000 con 0x7f8b180ad7e0 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.929+0000 7f8b1d600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8b1815d940 con 0x7f8b180b30f0 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.929+0000 7f8b1d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8b1815d180 con 0x7f8b18156c60 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.930+0000 7f8b17e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b180ad7e0 0x7f8b180cb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.930+0000 7f8b17e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b180ad7e0 0x7f8b180cb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43540/0 (socket says 172.21.15.69:43540) 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.930+0000 7f8b17e00640 1 -- 172.21.15.69:0/404203804 learned_addr learned my addr 172.21.15.69:0/404203804 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.930+0000 7f8b17e00640 1 -- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b18156c60 msgr2=0x7f8b181578c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.930+0000 7f8b17e00640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b18156c60 0x7f8b181578c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.930+0000 7f8b17e00640 1 -- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b180b30f0 msgr2=0x7f8b18155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.930+0000 7f8b17e00640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b180b30f0 0x7f8b18155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.930+0000 7f8b17e00640 1 -- 172.21.15.69:0/404203804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b180cbd70 con 0x7f8b180ad7e0 2024-04-24T08:09:19.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.930+0000 7f8b17e00640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b180ad7e0 0x7f8b180cb830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8b080097a0 tx=0x7f8b080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:19.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.931+0000 7f8b0fe00640 1 -- 172.21.15.69:0/404203804 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08037040 con 0x7f8b180ad7e0 2024-04-24T08:09:19.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.931+0000 7f8b0fe00640 1 -- 172.21.15.69:0/404203804 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b08004230 con 0x7f8b180ad7e0 2024-04-24T08:09:19.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.931+0000 7f8b0fe00640 1 -- 172.21.15.69:0/404203804 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b08032e20 con 0x7f8b180ad7e0 2024-04-24T08:09:19.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.931+0000 7f8b1d600640 1 -- 172.21.15.69:0/404203804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b1814eb20 con 0x7f8b180ad7e0 2024-04-24T08:09:19.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.931+0000 7f8b1d600640 1 -- 172.21.15.69:0/404203804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b1814f0e0 con 0x7f8b180ad7e0 2024-04-24T08:09:19.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.932+0000 7f8b1d600640 1 -- 172.21.15.69:0/404203804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b18075d50 con 0x7f8b180ad7e0 2024-04-24T08:09:19.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.933+0000 7f8b0fe00640 1 -- 172.21.15.69:0/404203804 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8b08030770 con 0x7f8b180ad7e0 2024-04-24T08:09:19.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.934+0000 7f8b0fe00640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8af806c8f0 0x7f8af806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:19.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.934+0000 7f8b0fe00640 1 -- 172.21.15.69:0/404203804 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f8b080acf50 con 0x7f8b180ad7e0 2024-04-24T08:09:19.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.935+0000 7f8b17400640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8af806c8f0 0x7f8af806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:19.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.936+0000 7f8b17400640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8af806c8f0 0x7f8af806edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8b181572d0 tx=0x7f8b04007990 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:19.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:19.938+0000 7f8b0fe00640 1 -- 172.21.15.69:0/404203804 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b0807b370 con 0x7f8b180ad7e0 2024-04-24T08:09:20.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.142+0000 7f8b1d600640 1 -- 172.21.15.69:0/404203804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b180ae880 con 0x7f8b180ad7e0 2024-04-24T08:09:20.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.145+0000 7f8b0fe00640 1 -- 172.21.15.69:0/404203804 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7f8b08081030 con 0x7f8b180ad7e0 2024-04-24T08:09:20.147 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:20.147 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:20.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.147+0000 7f8b1d600640 1 -- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8af806c8f0 msgr2=0x7f8af806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.147+0000 7f8b1d600640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8af806c8f0 0x7f8af806edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8b181572d0 tx=0x7f8b04007990 comp rx=0 tx=0).stop 2024-04-24T08:09:20.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.147+0000 7f8b1d600640 1 -- 172.21.15.69:0/404203804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b180ad7e0 msgr2=0x7f8b180cb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.147+0000 7f8b1d600640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b180ad7e0 0x7f8b180cb830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8b080097a0 tx=0x7f8b080046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.147+0000 7f8b1d600640 1 -- 172.21.15.69:0/404203804 shutdown_connections 2024-04-24T08:09:20.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.147+0000 7f8b1d600640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8af806c8f0 0x7f8af806edb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.147+0000 7f8b1d600640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8b18156c60 0x7f8b181578c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.147+0000 7f8b1d600640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8b180b30f0 0x7f8b18155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.147+0000 7f8b1d600640 1 --2- 172.21.15.69:0/404203804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b180ad7e0 0x7f8b180cb830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.148+0000 7f8b1d600640 1 -- 172.21.15.69:0/404203804 >> 172.21.15.69:0/404203804 conn(0x7f8b180d0160 msgr2=0x7f8b1806e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:20.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.148+0000 7f8b1d600640 1 -- 172.21.15.69:0/404203804 shutdown_connections 2024-04-24T08:09:20.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.148+0000 7f8b1d600640 1 -- 172.21.15.69:0/404203804 wait complete. 2024-04-24T08:09:20.169 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:20.169 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:20.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.360+0000 7f1057000640 1 Processor -- start 2024-04-24T08:09:20.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.360+0000 7f1057000640 1 -- start start 2024-04-24T08:09:20.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1057000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f105015c9b0 0x7f10500b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1057000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10500b3f70 0x7f10500cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1057000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10500cd6d0 con 0x7f1050155f20 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1057000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f105006db90 con 0x7f105015c9b0 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1057000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1050072070 con 0x7f10500b3f70 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1047e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f105015c9b0 0x7f10500b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1047e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f105015c9b0 0x7f10500b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43550/0 (socket says 172.21.15.69:43550) 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1047e00640 1 -- 172.21.15.69:0/3976950728 learned_addr learned my addr 172.21.15.69:0/3976950728 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1047e00640 1 -- 172.21.15.69:0/3976950728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10500b3f70 msgr2=0x7f10500cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1047e00640 1 --2- 172.21.15.69:0/3976950728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10500b3f70 0x7f10500cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1047e00640 1 -- 172.21.15.69:0/3976950728 >> v1:172.21.15.121:6789/0 conn(0x7f1050155f20 legacy=0x7f10500ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:20.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.362+0000 7f1047e00640 1 -- 172.21.15.69:0/3976950728 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10500cd9b0 con 0x7f105015c9b0 2024-04-24T08:09:20.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.363+0000 7f1047e00640 1 --2- 172.21.15.69:0/3976950728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f105015c9b0 0x7f10500b3a30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1038009870 tx=0x7f103802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e7445858d4ca37 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:20.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f104f400640 1 -- 172.21.15.69:0/3976950728 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038037070 con 0x7f105015c9b0 2024-04-24T08:09:20.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f104f400640 1 -- 172.21.15.69:0/3976950728 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1038032440 con 0x7f105015c9b0 2024-04-24T08:09:20.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f104f400640 1 -- 172.21.15.69:0/3976950728 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038032810 con 0x7f105015c9b0 2024-04-24T08:09:20.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f1057000640 1 -- 172.21.15.69:0/3976950728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f105015c9b0 msgr2=0x7f10500b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f1057000640 1 --2- 172.21.15.69:0/3976950728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f105015c9b0 0x7f10500b3a30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1038009870 tx=0x7f103802a2a0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f1057000640 1 -- 172.21.15.69:0/3976950728 shutdown_connections 2024-04-24T08:09:20.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f1057000640 1 --2- 172.21.15.69:0/3976950728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10500b3f70 0x7f10500cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f1057000640 1 --2- 172.21.15.69:0/3976950728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f105015c9b0 0x7f10500b3a30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f1057000640 1 -- 172.21.15.69:0/3976950728 >> 172.21.15.69:0/3976950728 conn(0x7f10500d0350 msgr2=0x7f10500d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:20.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f1057000640 1 -- 172.21.15.69:0/3976950728 shutdown_connections 2024-04-24T08:09:20.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.364+0000 7f1057000640 1 -- 172.21.15.69:0/3976950728 wait complete. 2024-04-24T08:09:20.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.366+0000 7f1057000640 1 Processor -- start 2024-04-24T08:09:20.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.366+0000 7f1057000640 1 -- start start 2024-04-24T08:09:20.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.367+0000 7f1057000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10500b3f70 0x7f1050151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.367+0000 7f1057000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1050155f20 0x7f10500c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.367+0000 7f1057000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f105015c9b0 0x7f10500c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.367+0000 7f1057000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f10500cf8a0 con 0x7f1050155f20 2024-04-24T08:09:20.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.367+0000 7f1057000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1050071d00 con 0x7f105015c9b0 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.367+0000 7f1057000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f10500cfa20 con 0x7f10500b3f70 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.367+0000 7f1047e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1050155f20 0x7f10500c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.367+0000 7f1047e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1050155f20 0x7f10500c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43562/0 (socket says 172.21.15.69:43562) 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.367+0000 7f1047e00640 1 -- 172.21.15.69:0/3909355702 learned_addr learned my addr 172.21.15.69:0/3909355702 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f1055000640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f105015c9b0 0x7f10500c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f104fe00640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10500b3f70 0x7f1050151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f1047e00640 1 -- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10500b3f70 msgr2=0x7f1050151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f1047e00640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10500b3f70 0x7f1050151790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f1047e00640 1 -- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f105015c9b0 msgr2=0x7f10500c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f1047e00640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f105015c9b0 0x7f10500c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f1047e00640 1 -- 172.21.15.69:0/3909355702 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10501f4d00 con 0x7f1050155f20 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f1055000640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f105015c9b0 0x7f10500c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:20.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f1047e00640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1050155f20 0x7f10500c95e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1038009870 tx=0x7f10380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:20.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f104cc00640 1 -- 172.21.15.69:0/3909355702 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038041070 con 0x7f1050155f20 2024-04-24T08:09:20.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f104cc00640 1 -- 172.21.15.69:0/3909355702 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103803c870 con 0x7f1050155f20 2024-04-24T08:09:20.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.368+0000 7f104cc00640 1 -- 172.21.15.69:0/3909355702 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1038037040 con 0x7f1050155f20 2024-04-24T08:09:20.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.369+0000 7f1057000640 1 -- 172.21.15.69:0/3909355702 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10501f4fe0 con 0x7f1050155f20 2024-04-24T08:09:20.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.369+0000 7f1057000640 1 -- 172.21.15.69:0/3909355702 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10501f55a0 con 0x7f1050155f20 2024-04-24T08:09:20.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.370+0000 7f1057000640 1 -- 172.21.15.69:0/3909355702 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1050068a80 con 0x7f1050155f20 2024-04-24T08:09:20.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.371+0000 7f104cc00640 1 -- 172.21.15.69:0/3909355702 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1038044070 con 0x7f1050155f20 2024-04-24T08:09:20.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.371+0000 7f104cc00640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f102c06c8f0 0x7f102c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.372+0000 7f104cc00640 1 -- 172.21.15.69:0/3909355702 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f10380ac850 con 0x7f1050155f20 2024-04-24T08:09:20.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.372+0000 7f104fe00640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f102c06c8f0 0x7f102c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.373+0000 7f104fe00640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f102c06c8f0 0x7f102c06edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f10400015d0 tx=0x7f1040009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:20.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.376+0000 7f104cc00640 1 -- 172.21.15.69:0/3909355702 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f103807ac70 con 0x7f1050155f20 2024-04-24T08:09:20.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.574+0000 7f1057000640 1 -- 172.21.15.69:0/3909355702 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f105006c260 con 0x7f1050155f20 2024-04-24T08:09:20.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.575+0000 7f104cc00640 1 -- 172.21.15.69:0/3909355702 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f103807a570 con 0x7f1050155f20 2024-04-24T08:09:20.576 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:20.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 -- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f102c06c8f0 msgr2=0x7f102c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f102c06c8f0 0x7f102c06edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f10400015d0 tx=0x7f1040009040 comp rx=0 tx=0).stop 2024-04-24T08:09:20.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 -- 172.21.15.69:0/3909355702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1050155f20 msgr2=0x7f10500c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1050155f20 0x7f10500c95e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1038009870 tx=0x7f10380046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 -- 172.21.15.69:0/3909355702 shutdown_connections 2024-04-24T08:09:20.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f102c06c8f0 0x7f102c06edb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1050155f20 0x7f10500c95e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f105015c9b0 0x7f10500c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 --2- 172.21.15.69:0/3909355702 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10500b3f70 0x7f1050151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 -- 172.21.15.69:0/3909355702 >> 172.21.15.69:0/3909355702 conn(0x7f10500d0350 msgr2=0x7f105015f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:20.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 -- 172.21.15.69:0/3909355702 shutdown_connections 2024-04-24T08:09:20.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.577+0000 7f1057000640 1 -- 172.21.15.69:0/3909355702 wait complete. 2024-04-24T08:09:20.598 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:20.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.795+0000 7f8267200640 1 Processor -- start 2024-04-24T08:09:20.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.795+0000 7f8267200640 1 -- start start 2024-04-24T08:09:20.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.796+0000 7f8267200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f826815edf0 0x7f826815f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.797+0000 7f8267200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826815fbb0 con 0x7f826806e000 2024-04-24T08:09:20.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.797+0000 7f8267200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82680d00d0 con 0x7f8268071d70 2024-04-24T08:09:20.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.797+0000 7f8267200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f82680d0a00 con 0x7f826815edf0 2024-04-24T08:09:20.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.797+0000 7f825fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f826815edf0 0x7f826815f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.797+0000 7f8266800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f826806e000 0x7f826806e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52608/0 (socket says 172.21.15.69:52608) 2024-04-24T08:09:20.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.797+0000 7f8266800640 1 -- 172.21.15.69:0/2671861583 learned_addr learned my addr 172.21.15.69:0/2671861583 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:20.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.797+0000 7f825fe00640 1 -- 172.21.15.69:0/2671861583 >> v1:172.21.15.121:6789/0 conn(0x7f8268071d70 legacy=0x7f8268072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:20.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.797+0000 7f825fe00640 1 -- 172.21.15.69:0/2671861583 >> v1:172.21.15.69:6789/0 conn(0x7f826806e000 legacy=0x7f826806e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.797+0000 7f8265400640 1 -- 172.21.15.69:0/2671861583 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4175788983 0 0) 0x7f826815fbb0 con 0x7f826806e000 2024-04-24T08:09:20.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.797+0000 7f825fe00640 1 -- 172.21.15.69:0/2671861583 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82680d03b0 con 0x7f826815edf0 2024-04-24T08:09:20.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.798+0000 7f825fe00640 1 --2- 172.21.15.69:0/2671861583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826815edf0 0x7f826815f250 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f82500097a0 tx=0x7f825002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1809d356f9fcf675 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:20.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.799+0000 7f8265400640 1 -- 172.21.15.69:0/2671861583 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250037070 con 0x7f826815edf0 2024-04-24T08:09:20.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.799+0000 7f8265400640 1 -- 172.21.15.69:0/2671861583 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8250032440 con 0x7f826815edf0 2024-04-24T08:09:20.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.799+0000 7f8265400640 1 -- 172.21.15.69:0/2671861583 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250032810 con 0x7f826815edf0 2024-04-24T08:09:20.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.799+0000 7f8267200640 1 -- 172.21.15.69:0/2671861583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826815edf0 msgr2=0x7f826815f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.799+0000 7f8267200640 1 --2- 172.21.15.69:0/2671861583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826815edf0 0x7f826815f250 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f82500097a0 tx=0x7f825002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.799+0000 7f8267200640 1 -- 172.21.15.69:0/2671861583 shutdown_connections 2024-04-24T08:09:20.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.799+0000 7f8267200640 1 --2- 172.21.15.69:0/2671861583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826815edf0 0x7f826815f250 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.799+0000 7f8267200640 1 -- 172.21.15.69:0/2671861583 >> 172.21.15.69:0/2671861583 conn(0x7f8268067b80 msgr2=0x7f826806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:20.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.799+0000 7f8267200640 1 -- 172.21.15.69:0/2671861583 shutdown_connections 2024-04-24T08:09:20.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.800+0000 7f8267200640 1 -- 172.21.15.69:0/2671861583 wait complete. 2024-04-24T08:09:20.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.801+0000 7f8267200640 1 Processor -- start 2024-04-24T08:09:20.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.801+0000 7f8267200640 1 -- start start 2024-04-24T08:09:20.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.802+0000 7f8267200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826806e000 0x7f826815bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.802+0000 7f8267200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8268071d70 0x7f826813fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.802+0000 7f8267200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f826815edf0 0x7f826813c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.802+0000 7f8267200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f82680d1010 con 0x7f826806e000 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.802+0000 7f8267200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f82680d0e90 con 0x7f826815edf0 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.802+0000 7f8267200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f82681481a0 con 0x7f8268071d70 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.802+0000 7f8265e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826806e000 0x7f826815bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.802+0000 7f8265e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826806e000 0x7f826815bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43584/0 (socket says 172.21.15.69:43584) 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.802+0000 7f8265e00640 1 -- 172.21.15.69:0/1513486359 learned_addr learned my addr 172.21.15.69:0/1513486359 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f825fe00640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8268071d70 0x7f826813fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f8265e00640 1 -- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8268071d70 msgr2=0x7f826813fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f8265e00640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8268071d70 0x7f826813fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f8265e00640 1 -- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f826815edf0 msgr2=0x7f826813c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f8265e00640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f826815edf0 0x7f826813c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f8265e00640 1 -- 172.21.15.69:0/1513486359 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826813ebf0 con 0x7f826806e000 2024-04-24T08:09:20.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f8265e00640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826806e000 0x7f826815bfc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f825800c430 tx=0x7f825800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:20.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f825e000640 1 -- 172.21.15.69:0/1513486359 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258011790 con 0x7f826806e000 2024-04-24T08:09:20.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f825e000640 1 -- 172.21.15.69:0/1513486359 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8258017070 con 0x7f826806e000 2024-04-24T08:09:20.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f825e000640 1 -- 172.21.15.69:0/1513486359 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258011bb0 con 0x7f826806e000 2024-04-24T08:09:20.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.803+0000 7f8267200640 1 -- 172.21.15.69:0/1513486359 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826813eed0 con 0x7f826806e000 2024-04-24T08:09:20.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.804+0000 7f8267200640 1 -- 172.21.15.69:0/1513486359 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f826813f490 con 0x7f826806e000 2024-04-24T08:09:20.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.805+0000 7f8267200640 1 -- 172.21.15.69:0/1513486359 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8268075d50 con 0x7f826806e000 2024-04-24T08:09:20.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.806+0000 7f825e000640 1 -- 172.21.15.69:0/1513486359 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f825801c460 con 0x7f826806e000 2024-04-24T08:09:20.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.806+0000 7f825e000640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824406c8f0 0x7f824406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.807+0000 7f825e000640 1 -- 172.21.15.69:0/1513486359 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f825808bb50 con 0x7f826806e000 2024-04-24T08:09:20.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.807+0000 7f825fe00640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824406c8f0 0x7f824406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.808+0000 7f825fe00640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824406c8f0 0x7f824406edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8250034000 tx=0x7f825002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:20.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.811+0000 7f825e000640 1 -- 172.21.15.69:0/1513486359 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8258059e50 con 0x7f826806e000 2024-04-24T08:09:20.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.995+0000 7f8267200640 1 -- 172.21.15.69:0/1513486359 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f826806f730 con 0x7f826806e000 2024-04-24T08:09:20.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.996+0000 7f825e000640 1 -- 172.21.15.69:0/1513486359 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f8258059750 con 0x7f826806e000 2024-04-24T08:09:20.998 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:21.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.998+0000 7f8267200640 1 -- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824406c8f0 msgr2=0x7f824406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.998+0000 7f8267200640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824406c8f0 0x7f824406edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8250034000 tx=0x7f825002e000 comp rx=0 tx=0).stop 2024-04-24T08:09:21.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.998+0000 7f8267200640 1 -- 172.21.15.69:0/1513486359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826806e000 msgr2=0x7f826815bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.998+0000 7f8267200640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826806e000 0x7f826815bfc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f825800c430 tx=0x7f825800c8f0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.999+0000 7f8267200640 1 -- 172.21.15.69:0/1513486359 shutdown_connections 2024-04-24T08:09:21.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.999+0000 7f8267200640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824406c8f0 0x7f824406edb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.999+0000 7f8267200640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f826815edf0 0x7f826813c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.999+0000 7f8267200640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8268071d70 0x7f826813fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.999+0000 7f8267200640 1 --2- 172.21.15.69:0/1513486359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f826806e000 0x7f826815bfc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.999+0000 7f8267200640 1 -- 172.21.15.69:0/1513486359 >> 172.21.15.69:0/1513486359 conn(0x7f8268067b80 msgr2=0x7f8268142830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:21.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.999+0000 7f8267200640 1 -- 172.21.15.69:0/1513486359 shutdown_connections 2024-04-24T08:09:21.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:20.999+0000 7f8267200640 1 -- 172.21.15.69:0/1513486359 wait complete. 2024-04-24T08:09:21.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:21.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.214+0000 7efc0fc00640 1 Processor -- start 2024-04-24T08:09:21.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.215+0000 7efc0fc00640 1 -- start start 2024-04-24T08:09:21.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.216+0000 7efc0fc00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efc080d3350 0x7efc0815faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.216+0000 7efc0fc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 0x7efc081615a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.216+0000 7efc0fc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc0806cc30 con 0x7efc0815e870 2024-04-24T08:09:21.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.216+0000 7efc0fc00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7efc0806e9c0 con 0x7efc080d3350 2024-04-24T08:09:21.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.216+0000 7efc0fc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efc0806ced0 con 0x7efc080d4e00 2024-04-24T08:09:21.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.217+0000 7efc0dc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 0x7efc081615a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:21.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.217+0000 7efbffe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efc080d3350 0x7efc0815faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:21.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.217+0000 7efc0d200640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7efc0815e870 0x7efc080d2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:36376/0 (socket says 172.21.15.69:36376) 2024-04-24T08:09:21.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.217+0000 7efbffe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efc080d3350 0x7efc0815faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:37798/0 (socket says 172.21.15.69:37798) 2024-04-24T08:09:21.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.217+0000 7efc0dc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 0x7efc081615a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:43140/0 (socket says 172.21.15.69:43140) 2024-04-24T08:09:21.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.217+0000 7efc0d200640 1 -- 172.21.15.69:0/1402697559 learned_addr learned my addr 172.21.15.69:0/1402697559 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:21.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.218+0000 7efc0dc00640 1 -- 172.21.15.69:0/1402697559 >> v2:172.21.15.121:3300/0 conn(0x7efc080d3350 msgr2=0x7efc0815faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:21.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.218+0000 7efc0dc00640 1 --2- 172.21.15.69:0/1402697559 >> v2:172.21.15.121:3300/0 conn(0x7efc080d3350 0x7efc0815faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.218+0000 7efc0dc00640 1 -- 172.21.15.69:0/1402697559 >> v1:172.21.15.121:6789/0 conn(0x7efc0815e870 legacy=0x7efc080d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.218+0000 7efc0dc00640 1 -- 172.21.15.69:0/1402697559 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0806d1b0 con 0x7efc080d4e00 2024-04-24T08:09:21.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.219+0000 7efc0dc00640 1 --2- 172.21.15.69:0/1402697559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 0x7efc081615a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efc00009d90 tx=0x7efc0002a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f13094e39464ae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:21.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.220+0000 7efc07e00640 1 -- 172.21.15.69:0/1402697559 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00031600 con 0x7efc080d4e00 2024-04-24T08:09:21.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.220+0000 7efc07e00640 1 -- 172.21.15.69:0/1402697559 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc000317a0 con 0x7efc080d4e00 2024-04-24T08:09:21.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.220+0000 7efc07e00640 1 -- 172.21.15.69:0/1402697559 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00031b70 con 0x7efc080d4e00 2024-04-24T08:09:21.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.220+0000 7efc0fc00640 1 -- 172.21.15.69:0/1402697559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 msgr2=0x7efc081615a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.220+0000 7efc0fc00640 1 --2- 172.21.15.69:0/1402697559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 0x7efc081615a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efc00009d90 tx=0x7efc0002a5d0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.221+0000 7efc0fc00640 1 -- 172.21.15.69:0/1402697559 shutdown_connections 2024-04-24T08:09:21.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.221+0000 7efc0fc00640 1 --2- 172.21.15.69:0/1402697559 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 0x7efc081615a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.221+0000 7efc0fc00640 1 --2- 172.21.15.69:0/1402697559 >> v2:172.21.15.121:3300/0 conn(0x7efc080d3350 0x7efc0815faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.221+0000 7efc0fc00640 1 -- 172.21.15.69:0/1402697559 >> 172.21.15.69:0/1402697559 conn(0x7efc0808bf60 msgr2=0x7efc080d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:21.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.221+0000 7efc0fc00640 1 -- 172.21.15.69:0/1402697559 shutdown_connections 2024-04-24T08:09:21.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.221+0000 7efc0fc00640 1 -- 172.21.15.69:0/1402697559 wait complete. 2024-04-24T08:09:21.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.222+0000 7efc0fc00640 1 Processor -- start 2024-04-24T08:09:21.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.223+0000 7efc0fc00640 1 -- start start 2024-04-24T08:09:21.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.223+0000 7efc0fc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc080d3350 0x7efc080b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.223+0000 7efc0fc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 0x7efc08155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.223+0000 7efc0fc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0815e870 0x7efc080b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.223+0000 7efc0fc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc080d5270 con 0x7efc080d3350 2024-04-24T08:09:21.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.223+0000 7efc0fc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efc080d55a0 con 0x7efc0815e870 2024-04-24T08:09:21.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.223+0000 7efc0fc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efc081602d0 con 0x7efc080d4e00 2024-04-24T08:09:21.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efc0d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc080d3350 0x7efc080b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:21.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efc0d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc080d3350 0x7efc080b50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43588/0 (socket says 172.21.15.69:43588) 2024-04-24T08:09:21.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efc0d200640 1 -- 172.21.15.69:0/3658995557 learned_addr learned my addr 172.21.15.69:0/3658995557 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efc0dc00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0815e870 0x7efc080b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efbffe00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 0x7efc08155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efc0d200640 1 -- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 msgr2=0x7efc08155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efc0d200640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 0x7efc08155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efc0d200640 1 -- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0815e870 msgr2=0x7efc080b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efc0d200640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0815e870 0x7efc080b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efc0d200640 1 -- 172.21.15.69:0/3658995557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc080b3f90 con 0x7efc080d3350 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.224+0000 7efc0dc00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0815e870 0x7efc080b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.225+0000 7efc0d200640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc080d3350 0x7efc080b50f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efbf800c3c0 tx=0x7efbf800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.225+0000 7efc05600640 1 -- 172.21.15.69:0/3658995557 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8017070 con 0x7efc080d3350 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.225+0000 7efc05600640 1 -- 172.21.15.69:0/3658995557 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf8005ce0 con 0x7efc080d3350 2024-04-24T08:09:21.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.225+0000 7efc05600640 1 -- 172.21.15.69:0/3658995557 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8007310 con 0x7efc080d3350 2024-04-24T08:09:21.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.225+0000 7efc0fc00640 1 -- 172.21.15.69:0/3658995557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc080b5630 con 0x7efc080d3350 2024-04-24T08:09:21.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.225+0000 7efc0fc00640 1 -- 172.21.15.69:0/3658995557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc08157810 con 0x7efc080d3350 2024-04-24T08:09:21.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.226+0000 7efc0fc00640 1 -- 172.21.15.69:0/3658995557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc08069190 con 0x7efc080d3350 2024-04-24T08:09:21.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.227+0000 7efc05600640 1 -- 172.21.15.69:0/3658995557 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efbf801a070 con 0x7efc080d3350 2024-04-24T08:09:21.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.228+0000 7efc05600640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efbe406c8f0 0x7efbe406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.228+0000 7efc05600640 1 -- 172.21.15.69:0/3658995557 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7efbf808af40 con 0x7efc080d3350 2024-04-24T08:09:21.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.229+0000 7efbffe00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efbe406c8f0 0x7efbe406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:21.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.230+0000 7efbffe00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efbe406c8f0 0x7efbe406edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efc0815d4e0 tx=0x7efbf0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:21.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.233+0000 7efc05600640 1 -- 172.21.15.69:0/3658995557 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efbf80592e0 con 0x7efc080d3350 2024-04-24T08:09:21.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.431+0000 7efc0fc00640 1 -- 172.21.15.69:0/3658995557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efc080d1ec0 con 0x7efc080d3350 2024-04-24T08:09:21.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.432+0000 7efc05600640 1 -- 172.21.15.69:0/3658995557 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7efbf8058be0 con 0x7efc080d3350 2024-04-24T08:09:21.434 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 -- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efbe406c8f0 msgr2=0x7efbe406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efbe406c8f0 0x7efbe406edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7efc0815d4e0 tx=0x7efbf0008040 comp rx=0 tx=0).stop 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 -- 172.21.15.69:0/3658995557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc080d3350 msgr2=0x7efc080b50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc080d3350 0x7efc080b50f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7efbf800c3c0 tx=0x7efbf800c880 comp rx=0 tx=0).stop 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 -- 172.21.15.69:0/3658995557 shutdown_connections 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efbe406c8f0 0x7efbe406edb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc0815e870 0x7efc080b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efc080d4e00 0x7efc08155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 --2- 172.21.15.69:0/3658995557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc080d3350 0x7efc080b50f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 -- 172.21.15.69:0/3658995557 >> 172.21.15.69:0/3658995557 conn(0x7efc0808bf60 msgr2=0x7efc080d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:21.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 -- 172.21.15.69:0/3658995557 shutdown_connections 2024-04-24T08:09:21.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.435+0000 7efc0fc00640 1 -- 172.21.15.69:0/3658995557 wait complete. 2024-04-24T08:09:21.454 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:09:21.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.658+0000 7fec16c00640 1 Processor -- start 2024-04-24T08:09:21.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.658+0000 7fec16c00640 1 -- start start 2024-04-24T08:09:21.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.660+0000 7fec16c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec1015e960 0x7fec10162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.660+0000 7fec16c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec1015ed40 con 0x7fec10162980 2024-04-24T08:09:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.660+0000 7fec16c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec100cf050 con 0x7fec100cfaf0 2024-04-24T08:09:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.660+0000 7fec16c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fec10162e20 con 0x7fec1015e960 2024-04-24T08:09:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.660+0000 7fec15800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec1015e960 0x7fec10162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.660+0000 7fec14e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fec10162980 0x7fec1006cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52624/0 (socket says 172.21.15.69:52624) 2024-04-24T08:09:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.660+0000 7fec14e00640 1 -- 172.21.15.69:0/3963940777 learned_addr learned my addr 172.21.15.69:0/3963940777 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.661+0000 7fec15800640 1 -- 172.21.15.69:0/3963940777 >> v1:172.21.15.121:6789/0 conn(0x7fec100cfaf0 legacy=0x7fec100d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.661+0000 7fec15800640 1 -- 172.21.15.69:0/3963940777 >> v1:172.21.15.69:6789/0 conn(0x7fec10162980 legacy=0x7fec1006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.661+0000 7fec15800640 1 -- 172.21.15.69:0/3963940777 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec10163100 con 0x7fec1015e960 2024-04-24T08:09:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.661+0000 7fec0fe00640 1 -- 172.21.15.69:0/3963940777 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1038897924 0 0) 0x7fec1015ed40 con 0x7fec10162980 2024-04-24T08:09:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.662+0000 7fec15800640 1 --2- 172.21.15.69:0/3963940777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec1015e960 0x7fec10162440 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fec1015ef10 tx=0x7fec0402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e78deab2905a2e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.662+0000 7fec0fe00640 1 -- 172.21.15.69:0/3963940777 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04037070 con 0x7fec1015e960 2024-04-24T08:09:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.662+0000 7fec0fe00640 1 -- 172.21.15.69:0/3963940777 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec04032440 con 0x7fec1015e960 2024-04-24T08:09:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.662+0000 7fec0fe00640 1 -- 172.21.15.69:0/3963940777 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04032810 con 0x7fec1015e960 2024-04-24T08:09:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.663+0000 7fec16c00640 1 -- 172.21.15.69:0/3963940777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec1015e960 msgr2=0x7fec10162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.663+0000 7fec16c00640 1 --2- 172.21.15.69:0/3963940777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec1015e960 0x7fec10162440 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fec1015ef10 tx=0x7fec0402a1c0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.663+0000 7fec16c00640 1 -- 172.21.15.69:0/3963940777 shutdown_connections 2024-04-24T08:09:21.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.663+0000 7fec16c00640 1 --2- 172.21.15.69:0/3963940777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec1015e960 0x7fec10162440 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.663+0000 7fec16c00640 1 -- 172.21.15.69:0/3963940777 >> 172.21.15.69:0/3963940777 conn(0x7fec1009d900 msgr2=0x7fec1008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:21.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.663+0000 7fec16c00640 1 -- 172.21.15.69:0/3963940777 shutdown_connections 2024-04-24T08:09:21.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.663+0000 7fec16c00640 1 -- 172.21.15.69:0/3963940777 wait complete. 2024-04-24T08:09:21.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.665+0000 7fec16c00640 1 Processor -- start 2024-04-24T08:09:21.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.665+0000 7fec16c00640 1 -- start start 2024-04-24T08:09:21.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec16c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec100cfaf0 0x7fec100ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec16c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec1015e960 0x7fec100b9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec16c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec10162980 0x7fec101610b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec16c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fec1006d6c0 con 0x7fec100cfaf0 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec16c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fec100cf5d0 con 0x7fec1015e960 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec16c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec100b6170 con 0x7fec10162980 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec15800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec100cfaf0 0x7fec100ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec15800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec100cfaf0 0x7fec100ca610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43602/0 (socket says 172.21.15.69:43602) 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec15800640 1 -- 172.21.15.69:0/4128049365 learned_addr learned my addr 172.21.15.69:0/4128049365 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec15800640 1 -- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec10162980 msgr2=0x7fec101610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec15800640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec10162980 0x7fec101610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec15800640 1 -- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec1015e960 msgr2=0x7fec100b9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec15800640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec1015e960 0x7fec100b9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.666+0000 7fec15800640 1 -- 172.21.15.69:0/4128049365 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec100b30c0 con 0x7fec100cfaf0 2024-04-24T08:09:21.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.667+0000 7fec15800640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec100cfaf0 0x7fec100ca610 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fec04000c00 tx=0x7fec040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:21.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.667+0000 7fec0d600640 1 -- 172.21.15.69:0/4128049365 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04041070 con 0x7fec100cfaf0 2024-04-24T08:09:21.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.667+0000 7fec0d600640 1 -- 172.21.15.69:0/4128049365 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec0403c950 con 0x7fec100cfaf0 2024-04-24T08:09:21.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.667+0000 7fec0d600640 1 -- 172.21.15.69:0/4128049365 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04037040 con 0x7fec100cfaf0 2024-04-24T08:09:21.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.667+0000 7fec16c00640 1 -- 172.21.15.69:0/4128049365 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec100b33a0 con 0x7fec100cfaf0 2024-04-24T08:09:21.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.667+0000 7fec16c00640 1 -- 172.21.15.69:0/4128049365 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec100b3960 con 0x7fec100cfaf0 2024-04-24T08:09:21.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.668+0000 7fec16c00640 1 -- 172.21.15.69:0/4128049365 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec10075d50 con 0x7fec100cfaf0 2024-04-24T08:09:21.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.669+0000 7fec0d600640 1 -- 172.21.15.69:0/4128049365 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fec04044070 con 0x7fec100cfaf0 2024-04-24T08:09:21.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.670+0000 7fec0d600640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febf406c8f0 0x7febf406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:21.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.671+0000 7fec0d600640 1 -- 172.21.15.69:0/4128049365 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fec040ac8f0 con 0x7fec100cfaf0 2024-04-24T08:09:21.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.671+0000 7fec14e00640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febf406c8f0 0x7febf406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:21.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.672+0000 7fec14e00640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febf406c8f0 0x7febf406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fec00007fd0 tx=0x7fec00007960 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:21.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.674+0000 7fec0d600640 1 -- 172.21.15.69:0/4128049365 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec0407ac80 con 0x7fec100cfaf0 2024-04-24T08:09:21.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.887+0000 7fec16c00640 1 -- 172.21.15.69:0/4128049365 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fec100d1dc0 con 0x7fec100cfaf0 2024-04-24T08:09:21.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.888+0000 7fec0d600640 1 -- 172.21.15.69:0/4128049365 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) ==== 80+0+0 (secure 0 0 0) 0x7fec0407a580 con 0x7fec100cfaf0 2024-04-24T08:09:21.889 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:09:21.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.890+0000 7fec16c00640 1 -- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febf406c8f0 msgr2=0x7febf406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.890+0000 7fec16c00640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febf406c8f0 0x7febf406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fec00007fd0 tx=0x7fec00007960 comp rx=0 tx=0).stop 2024-04-24T08:09:21.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.890+0000 7fec16c00640 1 -- 172.21.15.69:0/4128049365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec100cfaf0 msgr2=0x7fec100ca610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.890+0000 7fec16c00640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec100cfaf0 0x7fec100ca610 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fec04000c00 tx=0x7fec040046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.890+0000 7fec16c00640 1 -- 172.21.15.69:0/4128049365 shutdown_connections 2024-04-24T08:09:21.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.890+0000 7fec16c00640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febf406c8f0 0x7febf406edb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.890+0000 7fec16c00640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec10162980 0x7fec101610b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.890+0000 7fec16c00640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec1015e960 0x7fec100b9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.890+0000 7fec16c00640 1 --2- 172.21.15.69:0/4128049365 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec100cfaf0 0x7fec100ca610 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.890+0000 7fec16c00640 1 -- 172.21.15.69:0/4128049365 >> 172.21.15.69:0/4128049365 conn(0x7fec1009d900 msgr2=0x7fec1015cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:21.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.891+0000 7fec16c00640 1 -- 172.21.15.69:0/4128049365 shutdown_connections 2024-04-24T08:09:21.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:21.891+0000 7fec16c00640 1 -- 172.21.15.69:0/4128049365 wait complete. 2024-04-24T08:09:21.911 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi121.front.sepia.ceph.com... 2024-04-24T08:09:21.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:21.911 DEBUG:teuthology.orchestra.run.smithi121:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-24T08:09:22.007 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:09:22.007 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:09:22.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:22.007 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-24T08:09:22.060 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-24T08:09:22.060 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:09:22.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.265+0000 7f384f000640 1 Processor -- start 2024-04-24T08:09:22.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.265+0000 7f384f000640 1 -- start start 2024-04-24T08:09:22.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.267+0000 7f384f000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38480b30f0 0x7f3848167470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.267+0000 7f384f000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f384815d660 con 0x7f384815d1d0 2024-04-24T08:09:22.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.267+0000 7f384f000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f384806dc30 con 0x7f3848156c60 2024-04-24T08:09:22.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.267+0000 7f384f000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f384806df10 con 0x7f38480b30f0 2024-04-24T08:09:22.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.267+0000 7f384d200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f384815d1d0 0x7f38480ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52632/0 (socket says 172.21.15.69:52632) 2024-04-24T08:09:22.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.267+0000 7f384d200640 1 -- 172.21.15.69:0/2920355657 learned_addr learned my addr 172.21.15.69:0/2920355657 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:22.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.267+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1329106646 0 0) 0x7f384815d660 con 0x7f384815d1d0 2024-04-24T08:09:22.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f384e600640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38480b30f0 0x7f3848167470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3830003520 con 0x7f384815d1d0 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3170734750 0 0) 0x7f3830003520 con 0x7f384815d1d0 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38480b30f0 msgr2=0x7f3848167470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38480b30f0 0x7f3848167470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 >> v1:172.21.15.121:6789/0 conn(0x7f3848156c60 legacy=0x7f384815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f384806e1f0 con 0x7f384815d1d0 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f3834001670 con 0x7f384815d1d0 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 >> v1:172.21.15.69:6789/0 conn(0x7f384815d1d0 legacy=0x7f38480ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38480b30f0 0x7f38300039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3830004010 0x7f3830004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3830004d40 0x7f38300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f384806df10 con 0x7f3830004d40 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f383000bb80 con 0x7f3830004010 2024-04-24T08:09:22.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f383000bd80 con 0x7f38480b30f0 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.268+0000 7f384e600640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3830004d40 0x7f38300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.269+0000 7f384d200640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3830004010 0x7f3830004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.269+0000 7f384dc00640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38480b30f0 0x7f38300039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.269+0000 7f384e600640 1 -- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38480b30f0 msgr2=0x7f38300039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.269+0000 7f384e600640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38480b30f0 0x7f38300039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.269+0000 7f384e600640 1 -- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3830004010 msgr2=0x7f3830004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.269+0000 7f384e600640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3830004010 0x7f3830004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.269+0000 7f384e600640 1 -- 172.21.15.69:0/2920355657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3830003520 con 0x7f3830004d40 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.270+0000 7f384e600640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3830004d40 0x7f38300051c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3840007330 tx=0x7f3830005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224dcdff3ae7af32 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.270+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3840033070 con 0x7f3830004d40 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.270+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f384002f650 con 0x7f3830004d40 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.270+0000 7f3847e00640 1 -- 172.21.15.69:0/2920355657 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3840009be0 con 0x7f3830004d40 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.271+0000 7f384f000640 1 -- 172.21.15.69:0/2920355657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3830004d40 msgr2=0x7f38300051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.271+0000 7f384f000640 1 --2- 172.21.15.69:0/2920355657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3830004d40 0x7f38300051c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3840007330 tx=0x7f3830005830 comp rx=0 tx=0).stop 2024-04-24T08:09:22.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.271+0000 7f384e600640 1 -- 172.21.15.69:0/2920355657 reap_dead start 2024-04-24T08:09:22.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.271+0000 7f384f000640 1 -- 172.21.15.69:0/2920355657 shutdown_connections 2024-04-24T08:09:22.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.271+0000 7f384f000640 1 -- 172.21.15.69:0/2920355657 >> 172.21.15.69:0/2920355657 conn(0x7f38480d0160 msgr2=0x7f38480d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:22.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.271+0000 7f384f000640 1 -- 172.21.15.69:0/2920355657 shutdown_connections 2024-04-24T08:09:22.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.271+0000 7f384f000640 1 -- 172.21.15.69:0/2920355657 wait complete. 2024-04-24T08:09:22.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.273+0000 7f384f000640 1 Processor -- start 2024-04-24T08:09:22.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.273+0000 7f384f000640 1 -- start start 2024-04-24T08:09:22.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.274+0000 7f384f000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3830004d40 0x7f3848097a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.274+0000 7f384f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38480996f0 0x7f384815f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.274+0000 7f384f000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f384815dac0 0x7f3848143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.274+0000 7f384f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f38480ad7e0 con 0x7f38480996f0 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.274+0000 7f384f000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f384815d940 con 0x7f384815dac0 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.274+0000 7f384f000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f38480ad960 con 0x7f3830004d40 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.274+0000 7f384d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38480996f0 0x7f384815f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.274+0000 7f384d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38480996f0 0x7f384815f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43622/0 (socket says 172.21.15.69:43622) 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.274+0000 7f384d200640 1 -- 172.21.15.69:0/4104957438 learned_addr learned my addr 172.21.15.69:0/4104957438 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f384d200640 1 -- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3830004d40 msgr2=0x7f3848097a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f384d200640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3830004d40 0x7f3848097a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f384d200640 1 -- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f384815dac0 msgr2=0x7f3848143eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f384d200640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f384815dac0 0x7f3848143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f384d200640 1 -- 172.21.15.69:0/4104957438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f384815bea0 con 0x7f38480996f0 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f384d200640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38480996f0 0x7f384815f5c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f383400e800 tx=0x7f383400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:22.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f3845600640 1 -- 172.21.15.69:0/4104957438 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383400c850 con 0x7f38480996f0 2024-04-24T08:09:22.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f3845600640 1 -- 172.21.15.69:0/4104957438 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3834018070 con 0x7f38480996f0 2024-04-24T08:09:22.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f384f000640 1 -- 172.21.15.69:0/4104957438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f384815c180 con 0x7f38480996f0 2024-04-24T08:09:22.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f3845600640 1 -- 172.21.15.69:0/4104957438 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383400cc30 con 0x7f38480996f0 2024-04-24T08:09:22.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.275+0000 7f384f000640 1 -- 172.21.15.69:0/4104957438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f384815df50 con 0x7f38480996f0 2024-04-24T08:09:22.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.276+0000 7f384f000640 1 -- 172.21.15.69:0/4104957438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3848075d50 con 0x7f38480996f0 2024-04-24T08:09:22.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.277+0000 7f3845600640 1 -- 172.21.15.69:0/4104957438 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f383401d460 con 0x7f38480996f0 2024-04-24T08:09:22.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.278+0000 7f3845600640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f382806c920 0x7f382806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.278+0000 7f3845600640 1 -- 172.21.15.69:0/4104957438 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f383408bbc0 con 0x7f38480996f0 2024-04-24T08:09:22.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.279+0000 7f384dc00640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f382806c920 0x7f382806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.280+0000 7f384dc00640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f382806c920 0x7f382806ede0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f383c007fd0 tx=0x7f383c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:22.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.283+0000 7f3845600640 1 -- 172.21.15.69:0/4104957438 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3834059ec0 con 0x7f38480996f0 2024-04-24T08:09:22.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.521+0000 7f384f000640 1 -- 172.21.15.69:0/4104957438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38480b4690 con 0x7f38480996f0 2024-04-24T08:09:22.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.522+0000 7f3845600640 1 -- 172.21.15.69:0/4104957438 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f38340597c0 con 0x7f38480996f0 2024-04-24T08:09:22.524 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:22.524 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/934340733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":934340733},{"type":"v1","addr":"172.21.15.121:6835","nonce":934340733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3140867113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3140867113},{"type":"v1","addr":"172.21.15.121:6837","nonce":3140867113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/1059995973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1059995973},{"type":"v1","addr":"172.21.15.69:6833","nonce":1059995973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:08:19.574083+0000","modified":"2024-04-24T08:08:22.605668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/1003206529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1003206529},{"type":"v1","addr":"172.21.15.69:6835","nonce":1003206529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-24T08:09:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 7 2024-04-24T08:09:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 -- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f382806c920 msgr2=0x7f382806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f382806c920 0x7f382806ede0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f383c007fd0 tx=0x7f383c009040 comp rx=0 tx=0).stop 2024-04-24T08:09:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 -- 172.21.15.69:0/4104957438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38480996f0 msgr2=0x7f384815f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38480996f0 0x7f384815f5c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f383400e800 tx=0x7f383400ecc0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 -- 172.21.15.69:0/4104957438 shutdown_connections 2024-04-24T08:09:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f382806c920 0x7f382806ede0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38480996f0 0x7f384815f5c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f384815dac0 0x7f3848143eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 --2- 172.21.15.69:0/4104957438 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3830004d40 0x7f3848097a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 -- 172.21.15.69:0/4104957438 >> 172.21.15.69:0/4104957438 conn(0x7f38480d0160 msgr2=0x7f384815fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:22.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 -- 172.21.15.69:0/4104957438 shutdown_connections 2024-04-24T08:09:22.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.525+0000 7f384f000640 1 -- 172.21.15.69:0/4104957438 wait complete. 2024-04-24T08:09:22.546 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:22.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.735+0000 7f284e200640 1 Processor -- start 2024-04-24T08:09:22.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.736+0000 7f284e200640 1 -- start start 2024-04-24T08:09:22.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.737+0000 7f284e200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f28480d0200 0x7f28480d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.737+0000 7f284e200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f284815ed40 con 0x7f284806cfb0 2024-04-24T08:09:22.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.737+0000 7f284e200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28480cf050 con 0x7f284815e960 2024-04-24T08:09:22.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.737+0000 7f284e200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2848162b50 con 0x7f28480d0200 2024-04-24T08:09:22.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.737+0000 7f284d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f28480d0200 0x7f28480d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.737+0000 7f283fe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f284806cfb0 0x7f28480cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52638/0 (socket says 172.21.15.69:52638) 2024-04-24T08:09:22.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.737+0000 7f283fe00640 1 -- 172.21.15.69:0/3291823874 learned_addr learned my addr 172.21.15.69:0/3291823874 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:22.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.738+0000 7f284d800640 1 -- 172.21.15.69:0/3291823874 >> v1:172.21.15.121:6789/0 conn(0x7f284815e960 legacy=0x7f2848162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:22.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.738+0000 7f284d800640 1 -- 172.21.15.69:0/3291823874 >> v1:172.21.15.69:6789/0 conn(0x7f284806cfb0 legacy=0x7f28480cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.738+0000 7f284d800640 1 -- 172.21.15.69:0/3291823874 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2848162e20 con 0x7f28480d0200 2024-04-24T08:09:22.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.739+0000 7f284d800640 1 --2- 172.21.15.69:0/3291823874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28480d0200 0x7f28480d2980 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f28480cee40 tx=0x7f284002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1850fbe4784f1825 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:22.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.739+0000 7f2847e00640 1 -- 172.21.15.69:0/3291823874 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284002ae60 con 0x7f28480d0200 2024-04-24T08:09:22.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.739+0000 7f2847e00640 1 -- 172.21.15.69:0/3291823874 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2840007740 con 0x7f28480d0200 2024-04-24T08:09:22.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.739+0000 7f2847e00640 1 -- 172.21.15.69:0/3291823874 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2840007b10 con 0x7f28480d0200 2024-04-24T08:09:22.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.740+0000 7f284e200640 1 -- 172.21.15.69:0/3291823874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28480d0200 msgr2=0x7f28480d2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.740+0000 7f284e200640 1 --2- 172.21.15.69:0/3291823874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28480d0200 0x7f28480d2980 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f28480cee40 tx=0x7f284002a680 comp rx=0 tx=0).stop 2024-04-24T08:09:22.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.740+0000 7f284e200640 1 -- 172.21.15.69:0/3291823874 shutdown_connections 2024-04-24T08:09:22.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.740+0000 7f284e200640 1 --2- 172.21.15.69:0/3291823874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28480d0200 0x7f28480d2980 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.740+0000 7f284e200640 1 -- 172.21.15.69:0/3291823874 >> 172.21.15.69:0/3291823874 conn(0x7f284809d900 msgr2=0x7f284808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:22.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.740+0000 7f284e200640 1 -- 172.21.15.69:0/3291823874 shutdown_connections 2024-04-24T08:09:22.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.740+0000 7f284e200640 1 -- 172.21.15.69:0/3291823874 wait complete. 2024-04-24T08:09:22.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.741+0000 7f284e200640 1 Processor -- start 2024-04-24T08:09:22.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.741+0000 7f284e200640 1 -- start start 2024-04-24T08:09:22.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.742+0000 7f284e200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f284806cfb0 0x7f28480ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.742+0000 7f284e200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28480d0200 0x7f28480ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.742+0000 7f284e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f284815e960 0x7f28480cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.742+0000 7f284e200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f28480cf5d0 con 0x7f284815e960 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.742+0000 7f284e200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f28480cf2f0 con 0x7f284806cfb0 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.742+0000 7f284e200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f28480b6170 con 0x7f28480d0200 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f284d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f284815e960 0x7f28480cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f284d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f284815e960 0x7f28480cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43648/0 (socket says 172.21.15.69:43648) 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f284d800640 1 -- 172.21.15.69:0/3460485043 learned_addr learned my addr 172.21.15.69:0/3460485043 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f284ce00640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f284806cfb0 0x7f28480ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f283fe00640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28480d0200 0x7f28480ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f284d800640 1 -- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28480d0200 msgr2=0x7f28480ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f284d800640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28480d0200 0x7f28480ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f284d800640 1 -- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f284806cfb0 msgr2=0x7f28480ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f284d800640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f284806cfb0 0x7f28480ca880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f284d800640 1 -- 172.21.15.69:0/3460485043 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28480cd6a0 con 0x7f284815e960 2024-04-24T08:09:22.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.743+0000 7f284d800640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f284815e960 0x7f28480cd040 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2840009e50 tx=0x7f2840004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:22.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.744+0000 7f2845600640 1 -- 172.21.15.69:0/3460485043 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2840040070 con 0x7f284815e960 2024-04-24T08:09:22.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.744+0000 7f2845600640 1 -- 172.21.15.69:0/3460485043 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f284002ae60 con 0x7f284815e960 2024-04-24T08:09:22.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.744+0000 7f2845600640 1 -- 172.21.15.69:0/3460485043 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2840045460 con 0x7f284815e960 2024-04-24T08:09:22.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.744+0000 7f284e200640 1 -- 172.21.15.69:0/3460485043 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28480cd980 con 0x7f284815e960 2024-04-24T08:09:22.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.744+0000 7f284e200640 1 -- 172.21.15.69:0/3460485043 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f284815f3a0 con 0x7f284815e960 2024-04-24T08:09:22.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.745+0000 7f284e200640 1 -- 172.21.15.69:0/3460485043 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2848075d50 con 0x7f284815e960 2024-04-24T08:09:22.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.746+0000 7f2845600640 1 -- 172.21.15.69:0/3460485043 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2840039040 con 0x7f284815e960 2024-04-24T08:09:22.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.747+0000 7f2845600640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f282406c8f0 0x7f282406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:22.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.747+0000 7f2845600640 1 -- 172.21.15.69:0/3460485043 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f28400ad4b0 con 0x7f284815e960 2024-04-24T08:09:22.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.747+0000 7f284ce00640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f282406c8f0 0x7f282406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:22.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.748+0000 7f284ce00640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f282406c8f0 0x7f282406edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2838006fd0 tx=0x7f2838008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:22.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.751+0000 7f2845600640 1 -- 172.21.15.69:0/3460485043 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f284007b990 con 0x7f284815e960 2024-04-24T08:09:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.949+0000 7f284e200640 1 -- 172.21.15.69:0/3460485043 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28480cdb50 con 0x7f284815e960 2024-04-24T08:09:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.951+0000 7f2845600640 1 -- 172.21.15.69:0/3460485043 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7f284007b290 con 0x7f284815e960 2024-04-24T08:09:22.953 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:22.953 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:22.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 -- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f282406c8f0 msgr2=0x7f282406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f282406c8f0 0x7f282406edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2838006fd0 tx=0x7f2838008040 comp rx=0 tx=0).stop 2024-04-24T08:09:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 -- 172.21.15.69:0/3460485043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f284815e960 msgr2=0x7f28480cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f284815e960 0x7f28480cd040 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2840009e50 tx=0x7f2840004a40 comp rx=0 tx=0).stop 2024-04-24T08:09:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 -- 172.21.15.69:0/3460485043 shutdown_connections 2024-04-24T08:09:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f282406c8f0 0x7f282406edb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f284815e960 0x7f28480cd040 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f28480d0200 0x7f28480ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 --2- 172.21.15.69:0/3460485043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f284806cfb0 0x7f28480ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 -- 172.21.15.69:0/3460485043 >> 172.21.15.69:0/3460485043 conn(0x7f284809d900 msgr2=0x7f284806db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 -- 172.21.15.69:0/3460485043 shutdown_connections 2024-04-24T08:09:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:22.954+0000 7f284e200640 1 -- 172.21.15.69:0/3460485043 wait complete. 2024-04-24T08:09:22.972 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:08:17.641214+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:08:18.544062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:22.973 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:23.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.174+0000 7fa57c600640 1 Processor -- start 2024-04-24T08:09:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.174+0000 7fa57c600640 1 -- start start 2024-04-24T08:09:23.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.175+0000 7fa57c600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa57415f4b0 0x7fa5740b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.175+0000 7fa57c600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa5740b46a0 0x7fa5740b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa57c600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5740739f0 con 0x7fa57415f890 2024-04-24T08:09:23.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa57c600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa5740b5010 con 0x7fa57415f4b0 2024-04-24T08:09:23.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa57c600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa574073cc0 con 0x7fa5740b46a0 2024-04-24T08:09:23.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa579c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa57415f4b0 0x7fa5740b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:23.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa579c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa57415f4b0 0x7fa5740b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43650/0 (socket says 172.21.15.69:43650) 2024-04-24T08:09:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa579c00640 1 -- 172.21.15.69:0/308496154 learned_addr learned my addr 172.21.15.69:0/308496154 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa579c00640 1 -- 172.21.15.69:0/308496154 >> v2:172.21.15.121:3300/0 conn(0x7fa5740b46a0 msgr2=0x7fa5740b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa57a600640 1 --2- 172.21.15.69:0/308496154 >> v2:172.21.15.121:3300/0 conn(0x7fa5740b46a0 0x7fa5740b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa579c00640 1 --2- 172.21.15.69:0/308496154 >> v2:172.21.15.121:3300/0 conn(0x7fa5740b46a0 0x7fa5740b4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa579c00640 1 -- 172.21.15.69:0/308496154 >> v1:172.21.15.121:6789/0 conn(0x7fa57415f890 legacy=0x7fa5740732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.176+0000 7fa579c00640 1 -- 172.21.15.69:0/308496154 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa574073fa0 con 0x7fa57415f4b0 2024-04-24T08:09:23.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.177+0000 7fa579c00640 1 --2- 172.21.15.69:0/308496154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f4b0 0x7fa5740b3de0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa5740cda60 tx=0x7fa56802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587f3e7e31c42d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:23.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.178+0000 7fa573e00640 1 -- 172.21.15.69:0/308496154 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568037070 con 0x7fa57415f4b0 2024-04-24T08:09:23.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.178+0000 7fa573e00640 1 -- 172.21.15.69:0/308496154 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa568032440 con 0x7fa57415f4b0 2024-04-24T08:09:23.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.178+0000 7fa573e00640 1 -- 172.21.15.69:0/308496154 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa568032810 con 0x7fa57415f4b0 2024-04-24T08:09:23.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.178+0000 7fa57c600640 1 -- 172.21.15.69:0/308496154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f4b0 msgr2=0x7fa5740b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:23.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.178+0000 7fa57c600640 1 --2- 172.21.15.69:0/308496154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f4b0 0x7fa5740b3de0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa5740cda60 tx=0x7fa56802a230 comp rx=0 tx=0).stop 2024-04-24T08:09:23.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.179+0000 7fa57c600640 1 -- 172.21.15.69:0/308496154 shutdown_connections 2024-04-24T08:09:23.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.179+0000 7fa57c600640 1 --2- 172.21.15.69:0/308496154 >> v2:172.21.15.121:3300/0 conn(0x7fa5740b46a0 0x7fa5740b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.179+0000 7fa57c600640 1 --2- 172.21.15.69:0/308496154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f4b0 0x7fa5740b3de0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.179+0000 7fa57c600640 1 -- 172.21.15.69:0/308496154 >> 172.21.15.69:0/308496154 conn(0x7fa5740d0350 msgr2=0x7fa5740d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:23.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.179+0000 7fa57c600640 1 -- 172.21.15.69:0/308496154 shutdown_connections 2024-04-24T08:09:23.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.179+0000 7fa57c600640 1 -- 172.21.15.69:0/308496154 wait complete. 2024-04-24T08:09:23.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.180+0000 7fa57c600640 1 Processor -- start 2024-04-24T08:09:23.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.180+0000 7fa57c600640 1 -- start start 2024-04-24T08:09:23.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.181+0000 7fa57c600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5740b46a0 0x7fa574149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.181+0000 7fa57c600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57415f4b0 0x7fa5741559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.181+0000 7fa57c600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f890 0x7fa57409a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.181+0000 7fa57c600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa5740b38c0 con 0x7fa57415f890 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.181+0000 7fa57c600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa5740b35e0 con 0x7fa57415f4b0 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.181+0000 7fa57c600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa57415fd40 con 0x7fa5740b46a0 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa57a600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f890 0x7fa57409a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa57a600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f890 0x7fa57409a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43662/0 (socket says 172.21.15.69:43662) 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa57a600640 1 -- 172.21.15.69:0/2230485372 learned_addr learned my addr 172.21.15.69:0/2230485372 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa579c00640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5740b46a0 0x7fa574149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa579200640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57415f4b0 0x7fa5741559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa57a600640 1 -- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5740b46a0 msgr2=0x7fa574149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa57a600640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5740b46a0 0x7fa574149df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa57a600640 1 -- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57415f4b0 msgr2=0x7fa5741559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:23.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa57a600640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57415f4b0 0x7fa5741559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa57a600640 1 -- 172.21.15.69:0/2230485372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57409c540 con 0x7fa57415f890 2024-04-24T08:09:23.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.182+0000 7fa579200640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57415f4b0 0x7fa5741559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:23.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.183+0000 7fa57a600640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f890 0x7fa57409a510 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00cc30 tx=0x7fa56c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:23.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.183+0000 7fa571600640 1 -- 172.21.15.69:0/2230485372 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c00ce40 con 0x7fa57415f890 2024-04-24T08:09:23.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.183+0000 7fa571600640 1 -- 172.21.15.69:0/2230485372 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa56c003990 con 0x7fa57415f890 2024-04-24T08:09:23.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.183+0000 7fa571600640 1 -- 172.21.15.69:0/2230485372 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c003d40 con 0x7fa57415f890 2024-04-24T08:09:23.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.183+0000 7fa57c600640 1 -- 172.21.15.69:0/2230485372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa574156460 con 0x7fa57415f890 2024-04-24T08:09:23.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.183+0000 7fa57c600640 1 -- 172.21.15.69:0/2230485372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57409c7e0 con 0x7fa57415f890 2024-04-24T08:09:23.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.184+0000 7fa57c600640 1 -- 172.21.15.69:0/2230485372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa574156980 con 0x7fa57415f890 2024-04-24T08:09:23.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.185+0000 7fa571600640 1 -- 172.21.15.69:0/2230485372 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa56c003ee0 con 0x7fa57415f890 2024-04-24T08:09:23.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.186+0000 7fa571600640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55406c920 0x7fa55406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.187+0000 7fa571600640 1 -- 172.21.15.69:0/2230485372 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fa56c08c0f0 con 0x7fa57415f890 2024-04-24T08:09:23.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.187+0000 7fa579c00640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55406c920 0x7fa55406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:23.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.188+0000 7fa579c00640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55406c920 0x7fa55406ede0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa568034000 tx=0x7fa56802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:23.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.191+0000 7fa571600640 1 -- 172.21.15.69:0/2230485372 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa574156980 con 0x7fa57415f890 2024-04-24T08:09:23.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.373+0000 7fa57c600640 1 -- 172.21.15.69:0/2230485372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa574068a80 con 0x7fa57415f890 2024-04-24T08:09:23.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.374+0000 7fa571600640 1 -- 172.21.15.69:0/2230485372 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7fa56c014030 con 0x7fa57415f890 2024-04-24T08:09:23.375 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:23.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.376+0000 7fa57c600640 1 -- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55406c920 msgr2=0x7fa55406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:23.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.376+0000 7fa57c600640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55406c920 0x7fa55406ede0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa568034000 tx=0x7fa56802e000 comp rx=0 tx=0).stop 2024-04-24T08:09:23.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.376+0000 7fa57c600640 1 -- 172.21.15.69:0/2230485372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f890 msgr2=0x7fa57409a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:23.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.376+0000 7fa57c600640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f890 0x7fa57409a510 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00cc30 tx=0x7fa56c00a5c0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.376+0000 7fa57c600640 1 -- 172.21.15.69:0/2230485372 shutdown_connections 2024-04-24T08:09:23.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.376+0000 7fa57c600640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa55406c920 0x7fa55406ede0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.376+0000 7fa57c600640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa57415f890 0x7fa57409a510 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.376+0000 7fa57c600640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa57415f4b0 0x7fa5741559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.376+0000 7fa57c600640 1 --2- 172.21.15.69:0/2230485372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5740b46a0 0x7fa574149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.377+0000 7fa57c600640 1 -- 172.21.15.69:0/2230485372 >> 172.21.15.69:0/2230485372 conn(0x7fa5740d0350 msgr2=0x7fa5740732a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:23.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.377+0000 7fa57c600640 1 -- 172.21.15.69:0/2230485372 shutdown_connections 2024-04-24T08:09:23.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.377+0000 7fa57c600640 1 -- 172.21.15.69:0/2230485372 wait complete. 2024-04-24T08:09:23.396 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:23.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.592+0000 7f8f3e600640 1 Processor -- start 2024-04-24T08:09:23.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.592+0000 7f8f3e600640 1 -- start start 2024-04-24T08:09:23.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.594+0000 7f8f3e600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8f380b3de0 0x7f8f380732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.594+0000 7f8f3e600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8f38073820 0x7f8f380b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.594+0000 7f8f3e600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f38073cc0 con 0x7f8f3815f4b0 2024-04-24T08:09:23.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.594+0000 7f8f3e600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8f38073fa0 con 0x7f8f380b3de0 2024-04-24T08:09:23.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.594+0000 7f8f3e600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8f380b3300 con 0x7f8f38073820 2024-04-24T08:09:23.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.594+0000 7f8f2fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8f380b3de0 0x7f8f380732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:23.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.594+0000 7f8f2fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8f380b3de0 0x7f8f380732e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43664/0 (socket says 172.21.15.69:43664) 2024-04-24T08:09:23.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.594+0000 7f8f2fe00640 1 -- 172.21.15.69:0/2948095025 learned_addr learned my addr 172.21.15.69:0/2948095025 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:23.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.594+0000 7f8f37e00640 1 --2- 172.21.15.69:0/2948095025 >> v2:172.21.15.121:3300/0 conn(0x7f8f38073820 0x7f8f380b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:23.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.594+0000 7f8f2fe00640 1 -- 172.21.15.69:0/2948095025 >> v2:172.21.15.121:3300/0 conn(0x7f8f38073820 msgr2=0x7f8f380b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:23.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.595+0000 7f8f2fe00640 1 --2- 172.21.15.69:0/2948095025 >> v2:172.21.15.121:3300/0 conn(0x7f8f38073820 0x7f8f380b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.595+0000 7f8f2fe00640 1 -- 172.21.15.69:0/2948095025 >> v1:172.21.15.121:6789/0 conn(0x7f8f3815f4b0 legacy=0x7f8f3815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:23.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.595+0000 7f8f2fe00640 1 -- 172.21.15.69:0/2948095025 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f380b35e0 con 0x7f8f380b3de0 2024-04-24T08:09:23.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.595+0000 7f8f37e00640 1 --2- 172.21.15.69:0/2948095025 >> v2:172.21.15.121:3300/0 conn(0x7f8f38073820 0x7f8f380b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:23.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.596+0000 7f8f2fe00640 1 --2- 172.21.15.69:0/2948095025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 0x7f8f380732e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8f380cda60 tx=0x7f8f2002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e43ace4a147780 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:23.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.596+0000 7f8f36a00640 1 -- 172.21.15.69:0/2948095025 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f20037070 con 0x7f8f380b3de0 2024-04-24T08:09:23.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.596+0000 7f8f36a00640 1 -- 172.21.15.69:0/2948095025 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f20032440 con 0x7f8f380b3de0 2024-04-24T08:09:23.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.596+0000 7f8f36a00640 1 -- 172.21.15.69:0/2948095025 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f20032810 con 0x7f8f380b3de0 2024-04-24T08:09:23.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.597+0000 7f8f3e600640 1 -- 172.21.15.69:0/2948095025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 msgr2=0x7f8f380732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.597+0000 7f8f3e600640 1 --2- 172.21.15.69:0/2948095025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 0x7f8f380732e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8f380cda60 tx=0x7f8f2002a230 comp rx=0 tx=0).stop 2024-04-24T08:09:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.597+0000 7f8f3e600640 1 -- 172.21.15.69:0/2948095025 shutdown_connections 2024-04-24T08:09:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.597+0000 7f8f3e600640 1 --2- 172.21.15.69:0/2948095025 >> v2:172.21.15.121:3300/0 conn(0x7f8f38073820 0x7f8f380b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.597+0000 7f8f3e600640 1 --2- 172.21.15.69:0/2948095025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 0x7f8f380732e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.597+0000 7f8f3e600640 1 -- 172.21.15.69:0/2948095025 >> 172.21.15.69:0/2948095025 conn(0x7f8f380d0350 msgr2=0x7f8f380d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.597+0000 7f8f3e600640 1 -- 172.21.15.69:0/2948095025 shutdown_connections 2024-04-24T08:09:23.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.597+0000 7f8f3e600640 1 -- 172.21.15.69:0/2948095025 wait complete. 2024-04-24T08:09:23.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.598+0000 7f8f3e600640 1 Processor -- start 2024-04-24T08:09:23.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.599+0000 7f8f3e600640 1 -- start start 2024-04-24T08:09:23.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.599+0000 7f8f3e600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f38073820 0x7f8f381533a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f3e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 0x7f8f3806f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f3e600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f3815f4b0 0x7f8f381466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f3e600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f380b4d60 con 0x7f8f380b3de0 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f3e600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8f380b4be0 con 0x7f8f3815f4b0 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f3e600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8f380b4ee0 con 0x7f8f38073820 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f2fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 0x7f8f3806f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f2fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 0x7f8f3806f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43678/0 (socket says 172.21.15.69:43678) 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f2fe00640 1 -- 172.21.15.69:0/1392052652 learned_addr learned my addr 172.21.15.69:0/1392052652 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f2fe00640 1 -- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f38073820 msgr2=0x7f8f381533a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f2fe00640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f38073820 0x7f8f381533a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f2fe00640 1 -- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f3815f4b0 msgr2=0x7f8f381466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f2fe00640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f3815f4b0 0x7f8f381466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.600+0000 7f8f2fe00640 1 -- 172.21.15.69:0/1392052652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f381539d0 con 0x7f8f380b3de0 2024-04-24T08:09:23.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.601+0000 7f8f2fe00640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 0x7f8f3806f620 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8f20000c00 tx=0x7f8f200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:23.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.601+0000 7f8f2f400640 1 -- 172.21.15.69:0/1392052652 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f20041020 con 0x7f8f380b3de0 2024-04-24T08:09:23.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.601+0000 7f8f2f400640 1 -- 172.21.15.69:0/1392052652 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f20030970 con 0x7f8f380b3de0 2024-04-24T08:09:23.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.601+0000 7f8f2f400640 1 -- 172.21.15.69:0/1392052652 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f20037040 con 0x7f8f380b3de0 2024-04-24T08:09:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.601+0000 7f8f3e600640 1 -- 172.21.15.69:0/1392052652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f38153cb0 con 0x7f8f380b3de0 2024-04-24T08:09:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.601+0000 7f8f3e600640 1 -- 172.21.15.69:0/1392052652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f38154270 con 0x7f8f380b3de0 2024-04-24T08:09:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.602+0000 7f8f3e600640 1 -- 172.21.15.69:0/1392052652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f38068a80 con 0x7f8f380b3de0 2024-04-24T08:09:23.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.603+0000 7f8f2f400640 1 -- 172.21.15.69:0/1392052652 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8f20044020 con 0x7f8f380b3de0 2024-04-24T08:09:23.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.604+0000 7f8f2f400640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8f1406c8f0 0x7f8f1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:23.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.604+0000 7f8f2f400640 1 -- 172.21.15.69:0/1392052652 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f8f200ac710 con 0x7f8f380b3de0 2024-04-24T08:09:23.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.605+0000 7f8f37400640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8f1406c8f0 0x7f8f1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:23.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.606+0000 7f8f37400640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8f1406c8f0 0x7f8f1406edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f28003cc0 tx=0x7f8f280079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:23.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.608+0000 7f8f2f400640 1 -- 172.21.15.69:0/1392052652 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f2004b030 con 0x7f8f380b3de0 2024-04-24T08:09:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.805+0000 7f8f3e600640 1 -- 172.21.15.69:0/1392052652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8f3815fae0 con 0x7f8f380b3de0 2024-04-24T08:09:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.806+0000 7f8f2f400640 1 -- 172.21.15.69:0/1392052652 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f8f2007aa10 con 0x7f8f380b3de0 2024-04-24T08:09:23.807 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 -- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8f1406c8f0 msgr2=0x7f8f1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8f1406c8f0 0x7f8f1406edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8f28003cc0 tx=0x7f8f280079b0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 -- 172.21.15.69:0/1392052652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 msgr2=0x7f8f3806f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 0x7f8f3806f620 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8f20000c00 tx=0x7f8f200046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 -- 172.21.15.69:0/1392052652 shutdown_connections 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8f1406c8f0 0x7f8f1406edb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f380b3de0 0x7f8f3806f620 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f3815f4b0 0x7f8f381466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 --2- 172.21.15.69:0/1392052652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8f38073820 0x7f8f381533a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 -- 172.21.15.69:0/1392052652 >> 172.21.15.69:0/1392052652 conn(0x7f8f380d0350 msgr2=0x7f8f381608c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 -- 172.21.15.69:0/1392052652 shutdown_connections 2024-04-24T08:09:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:23.808+0000 7f8f3e600640 1 -- 172.21.15.69:0/1392052652 wait complete. 2024-04-24T08:09:23.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:24.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.028+0000 7f3776400640 1 Processor -- start 2024-04-24T08:09:24.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.028+0000 7f3776400640 1 -- start start 2024-04-24T08:09:24.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.029+0000 7f3776400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f37700d09f0 0x7f3770166020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.029+0000 7f3776400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37700ce640 con 0x7f377015ea60 2024-04-24T08:09:24.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.029+0000 7f3776400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37700ce910 con 0x7f37700b5300 2024-04-24T08:09:24.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.029+0000 7f3776400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f37700cebf0 con 0x7f37700d09f0 2024-04-24T08:09:24.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.030+0000 7f3775000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f377015ea60 0x7f37700cdf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52646/0 (socket says 172.21.15.69:52646) 2024-04-24T08:09:24.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.030+0000 7f3775000640 1 -- 172.21.15.69:0/1766658364 learned_addr learned my addr 172.21.15.69:0/1766658364 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:24.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.030+0000 7f3775a00640 1 --2- 172.21.15.69:0/1766658364 >> v2:172.21.15.69:3300/0 conn(0x7f37700d09f0 0x7f3770166020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.030+0000 7f376f400640 1 -- 172.21.15.69:0/1766658364 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2306816069 0 0) 0x7f37700ce640 con 0x7f377015ea60 2024-04-24T08:09:24.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.030+0000 7f376f400640 1 -- 172.21.15.69:0/1766658364 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3758003520 con 0x7f377015ea60 2024-04-24T08:09:24.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.030+0000 7f3775a00640 1 -- 172.21.15.69:0/1766658364 >> v1:172.21.15.121:6789/0 conn(0x7f37700b5300 legacy=0x7f37700d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:24.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.030+0000 7f3775a00640 1 -- 172.21.15.69:0/1766658364 >> v1:172.21.15.69:6789/0 conn(0x7f377015ea60 legacy=0x7f37700cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.030+0000 7f3775a00640 1 -- 172.21.15.69:0/1766658364 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37700ceed0 con 0x7f37700d09f0 2024-04-24T08:09:24.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.031+0000 7f3775a00640 1 --2- 172.21.15.69:0/1766658364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37700d09f0 0x7f3770166020 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3768009d90 tx=0x7f376802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ada71610b2bd36c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.031+0000 7f376f400640 1 -- 172.21.15.69:0/1766658364 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768007740 con 0x7f37700d09f0 2024-04-24T08:09:24.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.031+0000 7f376f400640 1 -- 172.21.15.69:0/1766658364 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37680078e0 con 0x7f37700d09f0 2024-04-24T08:09:24.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.031+0000 7f376f400640 1 -- 172.21.15.69:0/1766658364 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768007cb0 con 0x7f37700d09f0 2024-04-24T08:09:24.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.032+0000 7f3776400640 1 -- 172.21.15.69:0/1766658364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37700d09f0 msgr2=0x7f3770166020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.032+0000 7f3776400640 1 --2- 172.21.15.69:0/1766658364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37700d09f0 0x7f3770166020 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3768009d90 tx=0x7f376802a6a0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.032+0000 7f3776400640 1 -- 172.21.15.69:0/1766658364 shutdown_connections 2024-04-24T08:09:24.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.032+0000 7f3776400640 1 --2- 172.21.15.69:0/1766658364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37700d09f0 0x7f3770166020 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.032+0000 7f3776400640 1 -- 172.21.15.69:0/1766658364 >> 172.21.15.69:0/1766658364 conn(0x7f3770067b80 msgr2=0x7f377006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:24.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.032+0000 7f3776400640 1 -- 172.21.15.69:0/1766658364 shutdown_connections 2024-04-24T08:09:24.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.032+0000 7f3776400640 1 -- 172.21.15.69:0/1766658364 wait complete. 2024-04-24T08:09:24.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.034+0000 7f3776400640 1 Processor -- start 2024-04-24T08:09:24.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.034+0000 7f3776400640 1 -- start start 2024-04-24T08:09:24.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3776400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37700b5300 0x7f37701450c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3776400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37700d09f0 0x7f3770168d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3776400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f377015ea60 0x7f37700b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3776400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f377006ef50 con 0x7f377015ea60 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3776400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f37700cf490 con 0x7f37700b5300 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3776400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f377006f0d0 con 0x7f37700d09f0 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3775a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f377015ea60 0x7f37700b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3775a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f377015ea60 0x7f37700b3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43686/0 (socket says 172.21.15.69:43686) 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3775a00640 1 -- 172.21.15.69:0/1632722759 learned_addr learned my addr 172.21.15.69:0/1632722759 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3775000640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37700b5300 0x7f37701450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3775a00640 1 -- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37700d09f0 msgr2=0x7f3770168d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3775a00640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37700d09f0 0x7f3770168d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3775a00640 1 -- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37700b5300 msgr2=0x7f37701450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3775a00640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37700b5300 0x7f37701450c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.035+0000 7f3775a00640 1 -- 172.21.15.69:0/1632722759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37700b23c0 con 0x7f377015ea60 2024-04-24T08:09:24.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.036+0000 7f3775a00640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f377015ea60 0x7f37700b3460 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3768009eb0 tx=0x7f3768004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.036+0000 7f376cc00640 1 -- 172.21.15.69:0/1632722759 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768040070 con 0x7f377015ea60 2024-04-24T08:09:24.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.036+0000 7f376cc00640 1 -- 172.21.15.69:0/1632722759 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3768031bc0 con 0x7f377015ea60 2024-04-24T08:09:24.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.036+0000 7f376cc00640 1 -- 172.21.15.69:0/1632722759 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3768009450 con 0x7f377015ea60 2024-04-24T08:09:24.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.036+0000 7f3776400640 1 -- 172.21.15.69:0/1632722759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37700b26a0 con 0x7f377015ea60 2024-04-24T08:09:24.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.036+0000 7f3776400640 1 -- 172.21.15.69:0/1632722759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37700b2c60 con 0x7f377015ea60 2024-04-24T08:09:24.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.037+0000 7f3776400640 1 -- 172.21.15.69:0/1632722759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3770075d50 con 0x7f377015ea60 2024-04-24T08:09:24.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.038+0000 7f376cc00640 1 -- 172.21.15.69:0/1632722759 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3768039040 con 0x7f377015ea60 2024-04-24T08:09:24.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.040+0000 7f376cc00640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f375006c920 0x7f375006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.040+0000 7f376cc00640 1 -- 172.21.15.69:0/1632722759 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f37680ad070 con 0x7f377015ea60 2024-04-24T08:09:24.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.040+0000 7f3775000640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f375006c920 0x7f375006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.042+0000 7f3775000640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f375006c920 0x7f375006ede0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3764000f60 tx=0x7f3764009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.044+0000 7f376cc00640 1 -- 172.21.15.69:0/1632722759 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f376807b410 con 0x7f377015ea60 2024-04-24T08:09:24.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.250+0000 7f3776400640 1 -- 172.21.15.69:0/1632722759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f377015bdf0 con 0x7f377015ea60 2024-04-24T08:09:24.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.251+0000 7f376cc00640 1 -- 172.21.15.69:0/1632722759 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f376807ad10 con 0x7f377015ea60 2024-04-24T08:09:24.252 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.253+0000 7f3776400640 1 -- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f375006c920 msgr2=0x7f375006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.253+0000 7f3776400640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f375006c920 0x7f375006ede0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3764000f60 tx=0x7f3764009040 comp rx=0 tx=0).stop 2024-04-24T08:09:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.253+0000 7f3776400640 1 -- 172.21.15.69:0/1632722759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f377015ea60 msgr2=0x7f37700b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.253+0000 7f3776400640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f377015ea60 0x7f37700b3460 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3768009eb0 tx=0x7f3768004a40 comp rx=0 tx=0).stop 2024-04-24T08:09:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.254+0000 7f3776400640 1 -- 172.21.15.69:0/1632722759 shutdown_connections 2024-04-24T08:09:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.254+0000 7f3776400640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f375006c920 0x7f375006ede0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.254+0000 7f3776400640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f377015ea60 0x7f37700b3460 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.254+0000 7f3776400640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f37700d09f0 0x7f3770168d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.254+0000 7f3776400640 1 --2- 172.21.15.69:0/1632722759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37700b5300 0x7f37701450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.254+0000 7f3776400640 1 -- 172.21.15.69:0/1632722759 >> 172.21.15.69:0/1632722759 conn(0x7f3770067b80 msgr2=0x7f377015d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.254+0000 7f3776400640 1 -- 172.21.15.69:0/1632722759 shutdown_connections 2024-04-24T08:09:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.254+0000 7f3776400640 1 -- 172.21.15.69:0/1632722759 wait complete. 2024-04-24T08:09:24.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:09:24.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.464+0000 7f0f08200640 1 Processor -- start 2024-04-24T08:09:24.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.464+0000 7f0f08200640 1 -- start start 2024-04-24T08:09:24.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.466+0000 7f0f08200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0f00145a90 0x7f0f0006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.466+0000 7f0f08200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00145fe0 0x7f0f00160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.466+0000 7f0f08200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f000cf3d0 con 0x7f0f000d0300 2024-04-24T08:09:24.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.466+0000 7f0f08200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0f000cf6b0 con 0x7f0f00145a90 2024-04-24T08:09:24.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.466+0000 7f0f08200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0f000cf990 con 0x7f0f00145fe0 2024-04-24T08:09:24.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.466+0000 7f0f06200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00145fe0 0x7f0f00160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.466+0000 7f0f04e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0f00145a90 0x7f0f0006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.467+0000 7f0f04e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0f00145a90 0x7f0f0006e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:37884/0 (socket says 172.21.15.69:37884) 2024-04-24T08:09:24.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.467+0000 7f0f06200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00145fe0 0x7f0f00160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:43220/0 (socket says 172.21.15.69:43220) 2024-04-24T08:09:24.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.467+0000 7f0f04e00640 1 -- 172.21.15.69:0/316638687 learned_addr learned my addr 172.21.15.69:0/316638687 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:24.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.467+0000 7f0f04e00640 1 -- 172.21.15.69:0/316638687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00145fe0 msgr2=0x7f0f00160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:24.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.467+0000 7f0f04e00640 1 --2- 172.21.15.69:0/316638687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00145fe0 0x7f0f00160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.467+0000 7f0f04e00640 1 -- 172.21.15.69:0/316638687 >> v1:172.21.15.121:6789/0 conn(0x7f0f000d0300 legacy=0x7f0f000d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.467+0000 7f0f04e00640 1 -- 172.21.15.69:0/316638687 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f000cfc70 con 0x7f0f00145a90 2024-04-24T08:09:24.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.469+0000 7f0f04e00640 1 --2- 172.21.15.69:0/316638687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f00145a90 0x7f0f0006e240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009870 tx=0x7f0eec02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6165604b0795e2d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.469+0000 7f0effe00640 1 -- 172.21.15.69:0/316638687 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec037070 con 0x7f0f00145a90 2024-04-24T08:09:24.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.469+0000 7f0effe00640 1 -- 172.21.15.69:0/316638687 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec032440 con 0x7f0f00145a90 2024-04-24T08:09:24.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.469+0000 7f0effe00640 1 -- 172.21.15.69:0/316638687 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec032810 con 0x7f0f00145a90 2024-04-24T08:09:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.470+0000 7f0f08200640 1 -- 172.21.15.69:0/316638687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f00145a90 msgr2=0x7f0f0006e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.470+0000 7f0f08200640 1 --2- 172.21.15.69:0/316638687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f00145a90 0x7f0f0006e240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009870 tx=0x7f0eec02a270 comp rx=0 tx=0).stop 2024-04-24T08:09:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.470+0000 7f0f08200640 1 -- 172.21.15.69:0/316638687 shutdown_connections 2024-04-24T08:09:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.470+0000 7f0f08200640 1 --2- 172.21.15.69:0/316638687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00145fe0 0x7f0f00160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.470+0000 7f0f08200640 1 --2- 172.21.15.69:0/316638687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f00145a90 0x7f0f0006e240 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.470+0000 7f0f08200640 1 -- 172.21.15.69:0/316638687 >> 172.21.15.69:0/316638687 conn(0x7f0f00067d90 msgr2=0x7f0f0006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:24.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.470+0000 7f0f08200640 1 -- 172.21.15.69:0/316638687 shutdown_connections 2024-04-24T08:09:24.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.470+0000 7f0f08200640 1 -- 172.21.15.69:0/316638687 wait complete. 2024-04-24T08:09:24.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.472+0000 7f0f08200640 1 Processor -- start 2024-04-24T08:09:24.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.472+0000 7f0f08200640 1 -- start start 2024-04-24T08:09:24.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f08200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f000d0300 0x7f0f001f5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f08200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f00145a90 0x7f0f0013cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f08200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00145fe0 0x7f0f001f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f08200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0f001458f0 con 0x7f0f00145fe0 2024-04-24T08:09:24.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f08200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00145770 con 0x7f0f00145a90 2024-04-24T08:09:24.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f08200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0f000cdc30 con 0x7f0f000d0300 2024-04-24T08:09:24.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f06200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00145fe0 0x7f0f001f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f06200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00145fe0 0x7f0f001f7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43700/0 (socket says 172.21.15.69:43700) 2024-04-24T08:09:24.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f06200640 1 -- 172.21.15.69:0/3433475530 learned_addr learned my addr 172.21.15.69:0/3433475530 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:24.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f04e00640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f00145a90 0x7f0f0013cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f05800640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f000d0300 0x7f0f001f5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f06200640 1 -- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f000d0300 msgr2=0x7f0f001f5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f06200640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f000d0300 0x7f0f001f5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f06200640 1 -- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f00145a90 msgr2=0x7f0f0013cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f06200640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f00145a90 0x7f0f0013cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.473+0000 7f0f06200640 1 -- 172.21.15.69:0/3433475530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f001f4710 con 0x7f0f00145fe0 2024-04-24T08:09:24.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.474+0000 7f0f06200640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00145fe0 0x7f0f001f7d00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800efc0 tx=0x7f0ef800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.474+0000 7f0efd600640 1 -- 172.21.15.69:0/3433475530 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8018020 con 0x7f0f00145fe0 2024-04-24T08:09:24.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.474+0000 7f0efd600640 1 -- 172.21.15.69:0/3433475530 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef8014480 con 0x7f0f00145fe0 2024-04-24T08:09:24.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.474+0000 7f0efd600640 1 -- 172.21.15.69:0/3433475530 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8014870 con 0x7f0f00145fe0 2024-04-24T08:09:24.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.474+0000 7f0f08200640 1 -- 172.21.15.69:0/3433475530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f001f87c0 con 0x7f0f00145fe0 2024-04-24T08:09:24.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.474+0000 7f0f08200640 1 -- 172.21.15.69:0/3433475530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f001f4990 con 0x7f0f00145fe0 2024-04-24T08:09:24.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.475+0000 7f0f08200640 1 -- 172.21.15.69:0/3433475530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f00068a80 con 0x7f0f00145fe0 2024-04-24T08:09:24.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.476+0000 7f0efd600640 1 -- 172.21.15.69:0/3433475530 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ef8005d10 con 0x7f0f00145fe0 2024-04-24T08:09:24.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.477+0000 7f0efd600640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 0x7f0ee006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.478+0000 7f0efd600640 1 -- 172.21.15.69:0/3433475530 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f0ef808c650 con 0x7f0f00145fe0 2024-04-24T08:09:24.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.478+0000 7f0f05800640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 0x7f0ee006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.479+0000 7f0f05800640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 0x7f0ee006edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0ef40015a0 tx=0x7f0ef4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.482+0000 7f0efd600640 1 -- 172.21.15.69:0/3433475530 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ef8022070 con 0x7f0f00145fe0 2024-04-24T08:09:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.708+0000 7f0f08200640 1 -- 172.21.15.69:0/3433475530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f000d1120 con 0x7f0f00145fe0 2024-04-24T08:09:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.709+0000 7f0efd600640 1 -- 172.21.15.69:0/3433475530 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7f0f001f4710 con 0x7f0f00145fe0 2024-04-24T08:09:24.711 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:24.711 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/934340733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":934340733},{"type":"v1","addr":"172.21.15.121:6835","nonce":934340733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3140867113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3140867113},{"type":"v1","addr":"172.21.15.121:6837","nonce":3140867113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/1059995973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1059995973},{"type":"v1","addr":"172.21.15.69:6833","nonce":1059995973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:08:19.574083+0000","modified":"2024-04-24T08:08:22.605668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/1003206529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1003206529},{"type":"v1","addr":"172.21.15.69:6835","nonce":1003206529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-24T08:09:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 7 2024-04-24T08:09:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.711+0000 7f0f08200640 1 -- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 msgr2=0x7f0ee006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.711+0000 7f0f08200640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 0x7f0ee006edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0ef40015a0 tx=0x7f0ef4009040 comp rx=0 tx=0).stop 2024-04-24T08:09:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.711+0000 7f0f08200640 1 -- 172.21.15.69:0/3433475530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00145fe0 msgr2=0x7f0f001f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.711+0000 7f0f08200640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00145fe0 0x7f0f001f7d00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800efc0 tx=0x7f0ef800c490 comp rx=0 tx=0).stop 2024-04-24T08:09:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.712+0000 7f0f08200640 1 -- 172.21.15.69:0/3433475530 shutdown_connections 2024-04-24T08:09:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.712+0000 7f0f08200640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 0x7f0ee006edb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.712+0000 7f0f08200640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00145fe0 0x7f0f001f7d00 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.712+0000 7f0f08200640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f00145a90 0x7f0f0013cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.712+0000 7f0f08200640 1 --2- 172.21.15.69:0/3433475530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f000d0300 0x7f0f001f5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.712+0000 7f0f08200640 1 -- 172.21.15.69:0/3433475530 >> 172.21.15.69:0/3433475530 conn(0x7f0f00067d90 msgr2=0x7f0f0006f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.712+0000 7f0f08200640 1 -- 172.21.15.69:0/3433475530 shutdown_connections 2024-04-24T08:09:24.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.712+0000 7f0f08200640 1 -- 172.21.15.69:0/3433475530 wait complete. 2024-04-24T08:09:24.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:24.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.918+0000 7fb54f200640 1 Processor -- start 2024-04-24T08:09:24.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.918+0000 7fb54f200640 1 -- start start 2024-04-24T08:09:24.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.920+0000 7fb54f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55015e960 0x7fb550162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.920+0000 7fb54f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb550162980 0x7fb55006cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.920+0000 7fb54f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb55006d4f0 0x7fb5500cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.920+0000 7fb54f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb55015ed40 con 0x7fb55006d4f0 2024-04-24T08:09:24.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.920+0000 7fb54f200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb5500cf050 con 0x7fb550162980 2024-04-24T08:09:24.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.920+0000 7fb54f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb550162e20 con 0x7fb55015e960 2024-04-24T08:09:24.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.920+0000 7fb54e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb55006d4f0 0x7fb5500cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.920+0000 7fb54e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb55006d4f0 0x7fb5500cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43706/0 (socket says 172.21.15.69:43706) 2024-04-24T08:09:24.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.920+0000 7fb54e800640 1 -- 172.21.15.69:0/3833510839 learned_addr learned my addr 172.21.15.69:0/3833510839 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:24.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.921+0000 7fb54e800640 1 -- 172.21.15.69:0/3833510839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55015e960 msgr2=0x7fb550162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:24.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.921+0000 7fb54e800640 1 --2- 172.21.15.69:0/3833510839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55015e960 0x7fb550162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.921+0000 7fb54e800640 1 -- 172.21.15.69:0/3833510839 >> v2:172.21.15.121:3300/0 conn(0x7fb550162980 msgr2=0x7fb55006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:24.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.921+0000 7fb54e800640 1 --2- 172.21.15.69:0/3833510839 >> v2:172.21.15.121:3300/0 conn(0x7fb550162980 0x7fb55006cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.921+0000 7fb54e800640 1 -- 172.21.15.69:0/3833510839 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb550163100 con 0x7fb55006d4f0 2024-04-24T08:09:24.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.922+0000 7fb54e800640 1 --2- 172.21.15.69:0/3833510839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 0x7fb5500cfaf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb55009e030 tx=0x7fb54802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8da566309658a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.922+0000 7fb54d400640 1 -- 172.21.15.69:0/3833510839 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb548036070 con 0x7fb55006d4f0 2024-04-24T08:09:24.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.922+0000 7fb54d400640 1 -- 172.21.15.69:0/3833510839 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb548031600 con 0x7fb55006d4f0 2024-04-24T08:09:24.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.922+0000 7fb54d400640 1 -- 172.21.15.69:0/3833510839 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5480319d0 con 0x7fb55006d4f0 2024-04-24T08:09:24.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.923+0000 7fb54f200640 1 -- 172.21.15.69:0/3833510839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 msgr2=0x7fb5500cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.923+0000 7fb54f200640 1 --2- 172.21.15.69:0/3833510839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 0x7fb5500cfaf0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb55009e030 tx=0x7fb54802a6a0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.923+0000 7fb54f200640 1 -- 172.21.15.69:0/3833510839 shutdown_connections 2024-04-24T08:09:24.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.923+0000 7fb54f200640 1 --2- 172.21.15.69:0/3833510839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 0x7fb5500cfaf0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.923+0000 7fb54f200640 1 --2- 172.21.15.69:0/3833510839 >> v2:172.21.15.121:3300/0 conn(0x7fb550162980 0x7fb55006cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.923+0000 7fb54f200640 1 --2- 172.21.15.69:0/3833510839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb55015e960 0x7fb550162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.923+0000 7fb54f200640 1 -- 172.21.15.69:0/3833510839 >> 172.21.15.69:0/3833510839 conn(0x7fb55009d900 msgr2=0x7fb55008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:24.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.923+0000 7fb54f200640 1 -- 172.21.15.69:0/3833510839 shutdown_connections 2024-04-24T08:09:24.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.923+0000 7fb54f200640 1 -- 172.21.15.69:0/3833510839 wait complete. 2024-04-24T08:09:24.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.924+0000 7fb54f200640 1 Processor -- start 2024-04-24T08:09:24.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.925+0000 7fb54f200640 1 -- start start 2024-04-24T08:09:24.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.925+0000 7fb54f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 0x7fb5501619b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.925+0000 7fb54f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55015e960 0x7fb5500cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.925+0000 7fb54f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb550162980 0x7fb5500cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.925+0000 7fb54f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb5500b6170 con 0x7fb55006d4f0 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.925+0000 7fb54f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb5500cf5d0 con 0x7fb55015e960 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.925+0000 7fb54f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb5500b62f0 con 0x7fb550162980 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.926+0000 7fb54de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 0x7fb5501619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.926+0000 7fb54de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 0x7fb5501619b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43722/0 (socket says 172.21.15.69:43722) 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.926+0000 7fb54de00640 1 -- 172.21.15.69:0/3379029619 learned_addr learned my addr 172.21.15.69:0/3379029619 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.926+0000 7fb54de00640 1 -- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb550162980 msgr2=0x7fb5500cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.926+0000 7fb54de00640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb550162980 0x7fb5500cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.926+0000 7fb54de00640 1 -- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55015e960 msgr2=0x7fb5500cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.926+0000 7fb54de00640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55015e960 0x7fb5500cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.926+0000 7fb54de00640 1 -- 172.21.15.69:0/3379029619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5501e83d0 con 0x7fb55006d4f0 2024-04-24T08:09:24.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.926+0000 7fb54de00640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 0x7fb5501619b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb54000b4d0 tx=0x7fb54000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.927+0000 7fb546a00640 1 -- 172.21.15.69:0/3379029619 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb540014070 con 0x7fb55006d4f0 2024-04-24T08:09:24.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.927+0000 7fb546a00640 1 -- 172.21.15.69:0/3379029619 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb540010870 con 0x7fb55006d4f0 2024-04-24T08:09:24.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.927+0000 7fb546a00640 1 -- 172.21.15.69:0/3379029619 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb540010c60 con 0x7fb55006d4f0 2024-04-24T08:09:24.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.927+0000 7fb54f200640 1 -- 172.21.15.69:0/3379029619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5500cdd10 con 0x7fb55006d4f0 2024-04-24T08:09:24.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.927+0000 7fb54f200640 1 -- 172.21.15.69:0/3379029619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb550148c00 con 0x7fb55006d4f0 2024-04-24T08:09:24.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.928+0000 7fb54f200640 1 -- 172.21.15.69:0/3379029619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb550075d50 con 0x7fb55006d4f0 2024-04-24T08:09:24.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.929+0000 7fb546a00640 1 -- 172.21.15.69:0/3379029619 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb540017070 con 0x7fb55006d4f0 2024-04-24T08:09:24.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.930+0000 7fb546a00640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb52c06c8f0 0x7fb52c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.930+0000 7fb546a00640 1 -- 172.21.15.69:0/3379029619 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fb54008ba40 con 0x7fb55006d4f0 2024-04-24T08:09:24.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.930+0000 7fb545400640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb52c06c8f0 0x7fb52c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.931+0000 7fb545400640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb52c06c8f0 0x7fb52c06edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb550148ad0 tx=0x7fb538006bf0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:24.934+0000 7fb546a00640 1 -- 172.21.15.69:0/3379029619 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb540059d40 con 0x7fb55006d4f0 2024-04-24T08:09:25.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.121+0000 7fb54f200640 1 -- 172.21.15.69:0/3379029619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5500d1dc0 con 0x7fb55006d4f0 2024-04-24T08:09:25.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.124+0000 7fb546a00640 1 -- 172.21.15.69:0/3379029619 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7fb540059640 con 0x7fb55006d4f0 2024-04-24T08:09:25.125 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:25.125 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:25.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.126+0000 7fb54f200640 1 -- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb52c06c8f0 msgr2=0x7fb52c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.126+0000 7fb54f200640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb52c06c8f0 0x7fb52c06edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb550148ad0 tx=0x7fb538006bf0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.126+0000 7fb54f200640 1 -- 172.21.15.69:0/3379029619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 msgr2=0x7fb5501619b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.126+0000 7fb54f200640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 0x7fb5501619b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb54000b4d0 tx=0x7fb54000b990 comp rx=0 tx=0).stop 2024-04-24T08:09:25.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.126+0000 7fb54f200640 1 -- 172.21.15.69:0/3379029619 shutdown_connections 2024-04-24T08:09:25.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.126+0000 7fb54f200640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb52c06c8f0 0x7fb52c06edb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.126+0000 7fb54f200640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb550162980 0x7fb5500cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.126+0000 7fb54f200640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb55015e960 0x7fb5500cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.126+0000 7fb54f200640 1 --2- 172.21.15.69:0/3379029619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb55006d4f0 0x7fb5501619b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.126+0000 7fb54f200640 1 -- 172.21.15.69:0/3379029619 >> 172.21.15.69:0/3379029619 conn(0x7fb55009d900 msgr2=0x7fb5500d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:25.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.127+0000 7fb54f200640 1 -- 172.21.15.69:0/3379029619 shutdown_connections 2024-04-24T08:09:25.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.127+0000 7fb54f200640 1 -- 172.21.15.69:0/3379029619 wait complete. 2024-04-24T08:09:25.146 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:09:25.147 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:09:25.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.341+0000 7fe2cee00640 1 Processor -- start 2024-04-24T08:09:25.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.341+0000 7fe2cee00640 1 -- start start 2024-04-24T08:09:25.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.343+0000 7fe2cee00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2c80d0200 0x7fe2c80d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.343+0000 7fe2cee00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2c815ed40 con 0x7fe2c806cfb0 2024-04-24T08:09:25.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.343+0000 7fe2cee00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2c80cf050 con 0x7fe2c815e960 2024-04-24T08:09:25.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.343+0000 7fe2cee00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe2c8162b50 con 0x7fe2c80d0200 2024-04-24T08:09:25.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.343+0000 7fe2ce400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe2c80d0200 0x7fe2c80d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:25.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.343+0000 7fe2cd000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe2c806cfb0 0x7fe2c80cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52652/0 (socket says 172.21.15.69:52652) 2024-04-24T08:09:25.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.343+0000 7fe2cd000640 1 -- 172.21.15.69:0/1247929999 learned_addr learned my addr 172.21.15.69:0/1247929999 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:25.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.343+0000 7fe2c7e00640 1 -- 172.21.15.69:0/1247929999 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 71125152 0 0) 0x7fe2c815ed40 con 0x7fe2c806cfb0 2024-04-24T08:09:25.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.343+0000 7fe2ce400640 1 -- 172.21.15.69:0/1247929999 >> v1:172.21.15.121:6789/0 conn(0x7fe2c815e960 legacy=0x7fe2c8162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:25.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.343+0000 7fe2ce400640 1 -- 172.21.15.69:0/1247929999 >> v1:172.21.15.69:6789/0 conn(0x7fe2c806cfb0 legacy=0x7fe2c80cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.344+0000 7fe2ce400640 1 -- 172.21.15.69:0/1247929999 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2c8162e20 con 0x7fe2c80d0200 2024-04-24T08:09:25.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.345+0000 7fe2ce400640 1 --2- 172.21.15.69:0/1247929999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 0x7fe2c80d2980 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80cee40 tx=0x7fe2c002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb3c9f5a0530d52 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:25.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.345+0000 7fe2c7e00640 1 -- 172.21.15.69:0/1247929999 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c002ae60 con 0x7fe2c80d0200 2024-04-24T08:09:25.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.345+0000 7fe2c7e00640 1 -- 172.21.15.69:0/1247929999 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c0007740 con 0x7fe2c80d0200 2024-04-24T08:09:25.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.345+0000 7fe2c7e00640 1 -- 172.21.15.69:0/1247929999 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0007b10 con 0x7fe2c80d0200 2024-04-24T08:09:25.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.345+0000 7fe2cee00640 1 -- 172.21.15.69:0/1247929999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 msgr2=0x7fe2c80d2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.345+0000 7fe2cee00640 1 --2- 172.21.15.69:0/1247929999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 0x7fe2c80d2980 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80cee40 tx=0x7fe2c002a680 comp rx=0 tx=0).stop 2024-04-24T08:09:25.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.345+0000 7fe2cee00640 1 -- 172.21.15.69:0/1247929999 shutdown_connections 2024-04-24T08:09:25.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.345+0000 7fe2cee00640 1 --2- 172.21.15.69:0/1247929999 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 0x7fe2c80d2980 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.345+0000 7fe2cee00640 1 -- 172.21.15.69:0/1247929999 >> 172.21.15.69:0/1247929999 conn(0x7fe2c809d900 msgr2=0x7fe2c808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:25.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.346+0000 7fe2cee00640 1 -- 172.21.15.69:0/1247929999 shutdown_connections 2024-04-24T08:09:25.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.346+0000 7fe2cee00640 1 -- 172.21.15.69:0/1247929999 wait complete. 2024-04-24T08:09:25.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.347+0000 7fe2cee00640 1 Processor -- start 2024-04-24T08:09:25.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.347+0000 7fe2cee00640 1 -- start start 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.348+0000 7fe2cee00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2c806cfb0 0x7fe2c80cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.348+0000 7fe2cee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 0x7fe2c815d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.348+0000 7fe2cee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2c815e960 0x7fe2c8160f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.348+0000 7fe2cee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c80cf5d0 con 0x7fe2c80d0200 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.348+0000 7fe2cee00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2c80cf2f0 con 0x7fe2c806cfb0 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.348+0000 7fe2cee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe2c80b6170 con 0x7fe2c815e960 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.348+0000 7fe2cd000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 0x7fe2c815d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2cd000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 0x7fe2c815d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43746/0 (socket says 172.21.15.69:43746) 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2cd000640 1 -- 172.21.15.69:0/2990946477 learned_addr learned my addr 172.21.15.69:0/2990946477 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2cda00640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2c806cfb0 0x7fe2c80cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2cd000640 1 -- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2c815e960 msgr2=0x7fe2c8160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2ce400640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2c815e960 0x7fe2c8160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:25.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2cd000640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2c815e960 0x7fe2c8160f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2cd000640 1 -- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2c806cfb0 msgr2=0x7fe2c80cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2cd000640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2c806cfb0 0x7fe2c80cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2cd000640 1 -- 172.21.15.69:0/2990946477 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2c80cc820 con 0x7fe2c80d0200 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2cda00640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2c806cfb0 0x7fe2c80cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2cd000640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 0x7fe2c815d8a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400c4d0 tx=0x7fe2b400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.349+0000 7fe2c5600640 1 -- 172.21.15.69:0/2990946477 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4018070 con 0x7fe2c80d0200 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.350+0000 7fe2c5600640 1 -- 172.21.15.69:0/2990946477 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2b4007500 con 0x7fe2c80d0200 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.350+0000 7fe2cee00640 1 -- 172.21.15.69:0/2990946477 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2c80ccb00 con 0x7fe2c80d0200 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.350+0000 7fe2c5600640 1 -- 172.21.15.69:0/2990946477 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4004a20 con 0x7fe2c80d0200 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.350+0000 7fe2cee00640 1 -- 172.21.15.69:0/2990946477 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2c80cd0c0 con 0x7fe2c80d0200 2024-04-24T08:09:25.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.351+0000 7fe2cee00640 1 -- 172.21.15.69:0/2990946477 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2c80b4a70 con 0x7fe2c80d0200 2024-04-24T08:09:25.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.352+0000 7fe2c5600640 1 -- 172.21.15.69:0/2990946477 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2b401b070 con 0x7fe2c80d0200 2024-04-24T08:09:25.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.353+0000 7fe2c5600640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2a806c8f0 0x7fe2a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.353+0000 7fe2c5600640 1 -- 172.21.15.69:0/2990946477 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7fe2b408af30 con 0x7fe2c80d0200 2024-04-24T08:09:25.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.353+0000 7fe2cda00640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2a806c8f0 0x7fe2a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:25.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.354+0000 7fe2cda00640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2a806c8f0 0x7fe2a806edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80d0110 tx=0x7fe2bc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:25.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.357+0000 7fe2c5600640 1 -- 172.21.15.69:0/2990946477 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2c80b4a70 con 0x7fe2c80d0200 2024-04-24T08:09:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.599+0000 7fe2cee00640 1 -- 172.21.15.69:0/2990946477 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2c8075d50 con 0x7fe2c80d0200 2024-04-24T08:09:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.600+0000 7fe2c5600640 1 -- 172.21.15.69:0/2990946477 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7fe2b40592c0 con 0x7fe2c80d0200 2024-04-24T08:09:25.602 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:25.602 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/934340733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":934340733},{"type":"v1","addr":"172.21.15.121:6835","nonce":934340733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3140867113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3140867113},{"type":"v1","addr":"172.21.15.121:6837","nonce":3140867113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/1059995973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1059995973},{"type":"v1","addr":"172.21.15.69:6833","nonce":1059995973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:08:19.574083+0000","modified":"2024-04-24T08:08:22.605668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/1003206529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1003206529},{"type":"v1","addr":"172.21.15.69:6835","nonce":1003206529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-24T08:09:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 7 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 -- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2a806c8f0 msgr2=0x7fe2a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2a806c8f0 0x7fe2a806edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe2c80d0110 tx=0x7fe2bc008040 comp rx=0 tx=0).stop 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 -- 172.21.15.69:0/2990946477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 msgr2=0x7fe2c815d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 0x7fe2c815d8a0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400c4d0 tx=0x7fe2b400c990 comp rx=0 tx=0).stop 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 -- 172.21.15.69:0/2990946477 shutdown_connections 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe2a806c8f0 0x7fe2a806edb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2c80d0200 0x7fe2c815d8a0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2c815e960 0x7fe2c8160f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 --2- 172.21.15.69:0/2990946477 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2c806cfb0 0x7fe2c80cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 -- 172.21.15.69:0/2990946477 >> 172.21.15.69:0/2990946477 conn(0x7fe2c809d900 msgr2=0x7fe2c80b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 -- 172.21.15.69:0/2990946477 shutdown_connections 2024-04-24T08:09:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.603+0000 7fe2cee00640 1 -- 172.21.15.69:0/2990946477 wait complete. 2024-04-24T08:09:25.622 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:09:25.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.806+0000 7efffc800640 1 Processor -- start 2024-04-24T08:09:25.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.806+0000 7efffc800640 1 -- start start 2024-04-24T08:09:25.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.808+0000 7efffc800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efff40d0300 0x7efff406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.808+0000 7efffc800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efff4145fe0 0x7efff4160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.808+0000 7efffc800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efff40cf3d0 con 0x7efff4145a90 2024-04-24T08:09:25.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.808+0000 7efffc800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7efff40cf6b0 con 0x7efff40d0300 2024-04-24T08:09:25.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.808+0000 7efffc800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7efff40cf990 con 0x7efff4145fe0 2024-04-24T08:09:25.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.808+0000 7efff9e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efff40d0300 0x7efff406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:25.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.808+0000 7efff9e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7efff40d0300 0x7efff406e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43758/0 (socket says 172.21.15.69:43758) 2024-04-24T08:09:25.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.808+0000 7efff9e00640 1 -- 172.21.15.69:0/3343053300 learned_addr learned my addr 172.21.15.69:0/3343053300 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:25.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.809+0000 7efffa800640 1 --2- 172.21.15.69:0/3343053300 >> v2:172.21.15.121:3300/0 conn(0x7efff4145fe0 0x7efff4160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:25.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.809+0000 7efff9e00640 1 -- 172.21.15.69:0/3343053300 >> v2:172.21.15.121:3300/0 conn(0x7efff4145fe0 msgr2=0x7efff4160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:25.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.809+0000 7efff9e00640 1 --2- 172.21.15.69:0/3343053300 >> v2:172.21.15.121:3300/0 conn(0x7efff4145fe0 0x7efff4160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.809+0000 7efff9e00640 1 -- 172.21.15.69:0/3343053300 >> v1:172.21.15.121:6789/0 conn(0x7efff4145a90 legacy=0x7efff40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:25.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.809+0000 7efff9e00640 1 -- 172.21.15.69:0/3343053300 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff40cfc70 con 0x7efff40d0300 2024-04-24T08:09:25.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.810+0000 7efff9e00640 1 --2- 172.21.15.69:0/3343053300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 0x7efff406e240 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7efff40d1050 tx=0x7effe802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0167277c313f509 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:25.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.811+0000 7efff8a00640 1 -- 172.21.15.69:0/3343053300 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8037070 con 0x7efff40d0300 2024-04-24T08:09:25.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.811+0000 7efff8a00640 1 -- 172.21.15.69:0/3343053300 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe8032440 con 0x7efff40d0300 2024-04-24T08:09:25.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.811+0000 7efff8a00640 1 -- 172.21.15.69:0/3343053300 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8032810 con 0x7efff40d0300 2024-04-24T08:09:25.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.811+0000 7efffc800640 1 -- 172.21.15.69:0/3343053300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 msgr2=0x7efff406e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.811+0000 7efffc800640 1 --2- 172.21.15.69:0/3343053300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 0x7efff406e240 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7efff40d1050 tx=0x7effe802a230 comp rx=0 tx=0).stop 2024-04-24T08:09:25.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.812+0000 7efffc800640 1 -- 172.21.15.69:0/3343053300 shutdown_connections 2024-04-24T08:09:25.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.812+0000 7efffc800640 1 --2- 172.21.15.69:0/3343053300 >> v2:172.21.15.121:3300/0 conn(0x7efff4145fe0 0x7efff4160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.812+0000 7efffc800640 1 --2- 172.21.15.69:0/3343053300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 0x7efff406e240 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.812+0000 7efffc800640 1 -- 172.21.15.69:0/3343053300 >> 172.21.15.69:0/3343053300 conn(0x7efff4067d90 msgr2=0x7efff406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:25.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.812+0000 7efffc800640 1 -- 172.21.15.69:0/3343053300 shutdown_connections 2024-04-24T08:09:25.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.812+0000 7efffc800640 1 -- 172.21.15.69:0/3343053300 wait complete. 2024-04-24T08:09:25.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.813+0000 7efffc800640 1 Processor -- start 2024-04-24T08:09:25.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.814+0000 7efffc800640 1 -- start start 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.814+0000 7efffc800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 0x7efff41e51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.814+0000 7efffc800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efff4145a90 0x7efff415c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.814+0000 7efffc800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efff4145fe0 0x7efff41e6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.814+0000 7efffc800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efff41458f0 con 0x7efff40d0300 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.814+0000 7efffc800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efff4145770 con 0x7efff4145a90 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.814+0000 7efffc800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efff40cdc30 con 0x7efff4145fe0 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.815+0000 7efff9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 0x7efff41e51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.815+0000 7efff9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 0x7efff41e51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43760/0 (socket says 172.21.15.69:43760) 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.815+0000 7efff9e00640 1 -- 172.21.15.69:0/3901605205 learned_addr learned my addr 172.21.15.69:0/3901605205 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.815+0000 7efff9e00640 1 -- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efff4145fe0 msgr2=0x7efff41e6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.815+0000 7efff9e00640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efff4145fe0 0x7efff41e6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.815+0000 7efff9e00640 1 -- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efff4145a90 msgr2=0x7efff415c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.815+0000 7efff9400640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efff4145a90 0x7efff415c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:25.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.815+0000 7efff9e00640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efff4145a90 0x7efff415c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.815+0000 7efff9e00640 1 -- 172.21.15.69:0/3901605205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff41e6dc0 con 0x7efff40d0300 2024-04-24T08:09:25.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.816+0000 7efff9e00640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 0x7efff41e51c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7effe802a270 tx=0x7effe80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:25.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.816+0000 7effee000640 1 -- 172.21.15.69:0/3901605205 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8041020 con 0x7efff40d0300 2024-04-24T08:09:25.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.816+0000 7effee000640 1 -- 172.21.15.69:0/3901605205 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe80306c0 con 0x7efff40d0300 2024-04-24T08:09:25.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.816+0000 7effee000640 1 -- 172.21.15.69:0/3901605205 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8037040 con 0x7efff40d0300 2024-04-24T08:09:25.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.816+0000 7efffc800640 1 -- 172.21.15.69:0/3901605205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff41e57c0 con 0x7efff40d0300 2024-04-24T08:09:25.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.816+0000 7efffc800640 1 -- 172.21.15.69:0/3901605205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efff41e5d80 con 0x7efff40d0300 2024-04-24T08:09:25.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.817+0000 7efffc800640 1 -- 172.21.15.69:0/3901605205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efff4068a80 con 0x7efff40d0300 2024-04-24T08:09:25.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.818+0000 7effee000640 1 -- 172.21.15.69:0/3901605205 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7effe8044020 con 0x7efff40d0300 2024-04-24T08:09:25.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.819+0000 7effee000640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effd806c8f0 0x7effd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:25.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.819+0000 7effee000640 1 -- 172.21.15.69:0/3901605205 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7effe80ac4d0 con 0x7efff40d0300 2024-04-24T08:09:25.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.820+0000 7efff9400640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effd806c8f0 0x7effd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:25.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.821+0000 7efff9400640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effd806c8f0 0x7effd806edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efff40ccda0 tx=0x7effe4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:25.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:25.824+0000 7effee000640 1 -- 172.21.15.69:0/3901605205 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7effe804b030 con 0x7efff40d0300 2024-04-24T08:09:26.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.063+0000 7efffc800640 1 -- 172.21.15.69:0/3901605205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efff406ba20 con 0x7efff40d0300 2024-04-24T08:09:26.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.064+0000 7effee000640 1 -- 172.21.15.69:0/3901605205 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5178 (secure 0 0 0) 0x7effe807a7d0 con 0x7efff40d0300 2024-04-24T08:09:26.065 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:26.065 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4241,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/934340733","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":934340733},{"type":"v1","addr":"172.21.15.121:6835","nonce":934340733}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3140867113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3140867113},{"type":"v1","addr":"172.21.15.121:6837","nonce":3140867113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/1059995973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":1059995973},{"type":"v1","addr":"172.21.15.69:6833","nonce":1059995973}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:08:19.574083+0000","modified":"2024-04-24T08:08:22.605668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4257":{"gid":4257,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/1003206529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1003206529},{"type":"v1","addr":"172.21.15.69:6835","nonce":1003206529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4257,"qdb_cluster":[4257]},"id":1}]} 2024-04-24T08:09:26.066 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 7 2024-04-24T08:09:26.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.066+0000 7efffc800640 1 -- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effd806c8f0 msgr2=0x7effd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.066+0000 7efffc800640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effd806c8f0 0x7effd806edb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7efff40ccda0 tx=0x7effe4008040 comp rx=0 tx=0).stop 2024-04-24T08:09:26.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.066+0000 7efffc800640 1 -- 172.21.15.69:0/3901605205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 msgr2=0x7efff41e51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.066+0000 7efffc800640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 0x7efff41e51c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7effe802a270 tx=0x7effe80046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.066+0000 7efffc800640 1 -- 172.21.15.69:0/3901605205 shutdown_connections 2024-04-24T08:09:26.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.066+0000 7efffc800640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effd806c8f0 0x7effd806edb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.066+0000 7efffc800640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efff4145fe0 0x7efff41e6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.066+0000 7efffc800640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efff4145a90 0x7efff415c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.066+0000 7efffc800640 1 --2- 172.21.15.69:0/3901605205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efff40d0300 0x7efff41e51c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.066+0000 7efffc800640 1 -- 172.21.15.69:0/3901605205 >> 172.21.15.69:0/3901605205 conn(0x7efff4067d90 msgr2=0x7efff40b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:26.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.067+0000 7efffc800640 1 -- 172.21.15.69:0/3901605205 shutdown_connections 2024-04-24T08:09:26.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.067+0000 7efffc800640 1 -- 172.21.15.69:0/3901605205 wait complete. 2024-04-24T08:09:26.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:26.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.281+0000 7f6008e00640 1 Processor -- start 2024-04-24T08:09:26.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.281+0000 7f6008e00640 1 -- start start 2024-04-24T08:09:26.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.283+0000 7f6008e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f600415e960 0x7f6004162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.283+0000 7f6008e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6004162980 0x7f600406cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.283+0000 7f6008e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f600406d4f0 0x7f60040cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.283+0000 7f6008e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f600415ed40 con 0x7f6004162980 2024-04-24T08:09:26.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.283+0000 7f6008e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f60040cf050 con 0x7f600406d4f0 2024-04-24T08:09:26.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.283+0000 7f6008e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6004162e20 con 0x7f600415e960 2024-04-24T08:09:26.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.283+0000 7f5ffaa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6004162980 0x7f600406cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.283+0000 7f5ffaa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6004162980 0x7f600406cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43770/0 (socket says 172.21.15.69:43770) 2024-04-24T08:09:26.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.283+0000 7f5ffaa00640 1 -- 172.21.15.69:0/41098773 learned_addr learned my addr 172.21.15.69:0/41098773 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:26.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.284+0000 7f6003e00640 1 --2- 172.21.15.69:0/41098773 >> v2:172.21.15.121:3300/0 conn(0x7f600406d4f0 0x7f60040cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.284+0000 7f5ffaa00640 1 -- 172.21.15.69:0/41098773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f600415e960 msgr2=0x7f6004162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:26.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.284+0000 7f6003400640 1 --2- 172.21.15.69:0/41098773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f600415e960 0x7f6004162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.284+0000 7f5ffaa00640 1 --2- 172.21.15.69:0/41098773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f600415e960 0x7f6004162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.284+0000 7f5ffaa00640 1 -- 172.21.15.69:0/41098773 >> v2:172.21.15.121:3300/0 conn(0x7f600406d4f0 msgr2=0x7f60040cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:26.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.284+0000 7f5ffaa00640 1 --2- 172.21.15.69:0/41098773 >> v2:172.21.15.121:3300/0 conn(0x7f600406d4f0 0x7f60040cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.284+0000 7f5ffaa00640 1 -- 172.21.15.69:0/41098773 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6004163100 con 0x7f6004162980 2024-04-24T08:09:26.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.285+0000 7f5ffaa00640 1 --2- 172.21.15.69:0/41098773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6004162980 0x7f600406cfb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f60040c7fc0 tx=0x7f5fec02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4a60259cb65d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:26.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.285+0000 7f6002a00640 1 -- 172.21.15.69:0/41098773 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec037070 con 0x7f6004162980 2024-04-24T08:09:26.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.285+0000 7f6002a00640 1 -- 172.21.15.69:0/41098773 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fec032440 con 0x7f6004162980 2024-04-24T08:09:26.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.285+0000 7f6002a00640 1 -- 172.21.15.69:0/41098773 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec032810 con 0x7f6004162980 2024-04-24T08:09:26.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.286+0000 7f6008e00640 1 -- 172.21.15.69:0/41098773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6004162980 msgr2=0x7f600406cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.286+0000 7f6008e00640 1 --2- 172.21.15.69:0/41098773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6004162980 0x7f600406cfb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f60040c7fc0 tx=0x7f5fec02a1e0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.286+0000 7f6008e00640 1 -- 172.21.15.69:0/41098773 shutdown_connections 2024-04-24T08:09:26.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.286+0000 7f6008e00640 1 --2- 172.21.15.69:0/41098773 >> v2:172.21.15.121:3300/0 conn(0x7f600406d4f0 0x7f60040cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.286+0000 7f6008e00640 1 --2- 172.21.15.69:0/41098773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6004162980 0x7f600406cfb0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.286+0000 7f6008e00640 1 --2- 172.21.15.69:0/41098773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f600415e960 0x7f6004162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.286+0000 7f6008e00640 1 -- 172.21.15.69:0/41098773 >> 172.21.15.69:0/41098773 conn(0x7f600409d900 msgr2=0x7f600408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:26.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.286+0000 7f6008e00640 1 -- 172.21.15.69:0/41098773 shutdown_connections 2024-04-24T08:09:26.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.286+0000 7f6008e00640 1 -- 172.21.15.69:0/41098773 wait complete. 2024-04-24T08:09:26.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.288+0000 7f6008e00640 1 Processor -- start 2024-04-24T08:09:26.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.288+0000 7f6008e00640 1 -- start start 2024-04-24T08:09:26.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.289+0000 7f6008e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f600406d4f0 0x7f6004161770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.289+0000 7f6008e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600415e960 0x7f60040b8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.289+0000 7f6008e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6004162980 0x7f60040b9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.289+0000 7f6008e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60040b6170 con 0x7f600415e960 2024-04-24T08:09:26.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.289+0000 7f6008e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f60040cf5d0 con 0x7f6004162980 2024-04-24T08:09:26.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.289+0000 7f6008e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f60040b62f0 con 0x7f600406d4f0 2024-04-24T08:09:26.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.290+0000 7f5ffaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600415e960 0x7f60040b8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.290+0000 7f5ffaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600415e960 0x7f60040b8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43780/0 (socket says 172.21.15.69:43780) 2024-04-24T08:09:26.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.290+0000 7f5ffaa00640 1 -- 172.21.15.69:0/1796281869 learned_addr learned my addr 172.21.15.69:0/1796281869 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:26.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.290+0000 7f5ffaa00640 1 -- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f600406d4f0 msgr2=0x7f6004161770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.290+0000 7f5ffaa00640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f600406d4f0 0x7f6004161770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.290+0000 7f5ffaa00640 1 -- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6004162980 msgr2=0x7f60040b9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.290+0000 7f5ffaa00640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6004162980 0x7f60040b9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.290+0000 7f5ffaa00640 1 -- 172.21.15.69:0/1796281869 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600415c110 con 0x7f600415e960 2024-04-24T08:09:26.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.291+0000 7f5ffaa00640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600415e960 0x7f60040b8d50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5fec02a220 tx=0x7f5fec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:26.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.291+0000 7f5ffbe00640 1 -- 172.21.15.69:0/1796281869 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec037040 con 0x7f600415e960 2024-04-24T08:09:26.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.291+0000 7f5ffbe00640 1 -- 172.21.15.69:0/1796281869 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fec042070 con 0x7f600415e960 2024-04-24T08:09:26.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.291+0000 7f5ffbe00640 1 -- 172.21.15.69:0/1796281869 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec030bb0 con 0x7f600415e960 2024-04-24T08:09:26.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.291+0000 7f6008e00640 1 -- 172.21.15.69:0/1796281869 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60040b76f0 con 0x7f600415e960 2024-04-24T08:09:26.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.291+0000 7f6008e00640 1 -- 172.21.15.69:0/1796281869 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60040ca070 con 0x7f600415e960 2024-04-24T08:09:26.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.292+0000 7f6008e00640 1 -- 172.21.15.69:0/1796281869 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6004075d50 con 0x7f600415e960 2024-04-24T08:09:26.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.293+0000 7f5ffbe00640 1 -- 172.21.15.69:0/1796281869 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5fec047460 con 0x7f600415e960 2024-04-24T08:09:26.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.294+0000 7f5ffbe00640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5fe006c8f0 0x7f5fe006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.294+0000 7f5ffbe00640 1 -- 172.21.15.69:0/1796281869 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7f5fec0ac9b0 con 0x7f600415e960 2024-04-24T08:09:26.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.295+0000 7f6003400640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5fe006c8f0 0x7f5fe006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.296+0000 7f6003400640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5fe006c8f0 0x7f5fe006edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4006fd0 tx=0x7f5ff4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:26.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.298+0000 7f5ffbe00640 1 -- 172.21.15.69:0/1796281869 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5fec045360 con 0x7f600415e960 2024-04-24T08:09:26.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.495+0000 7f6008e00640 1 -- 172.21.15.69:0/1796281869 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60040d1dc0 con 0x7f600415e960 2024-04-24T08:09:26.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.498+0000 7f5ffbe00640 1 -- 172.21.15.69:0/1796281869 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16877 (secure 0 0 0) 0x7f5fec07ac20 con 0x7f600415e960 2024-04-24T08:09:26.499 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:26.499 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":22,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:02.376137+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:08:17.641214+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:08:18.544062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.8","mappings":[{"from":3,"to":2}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:26.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 -- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5fe006c8f0 msgr2=0x7f5fe006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5fe006c8f0 0x7f5fe006edb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4006fd0 tx=0x7f5ff4008040 comp rx=0 tx=0).stop 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 -- 172.21.15.69:0/1796281869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600415e960 msgr2=0x7f60040b8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600415e960 0x7f60040b8d50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5fec02a220 tx=0x7f5fec0046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 -- 172.21.15.69:0/1796281869 shutdown_connections 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5fe006c8f0 0x7f5fe006edb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f600415e960 0x7f60040b8d50 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6004162980 0x7f60040b9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 --2- 172.21.15.69:0/1796281869 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f600406d4f0 0x7f6004161770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 -- 172.21.15.69:0/1796281869 >> 172.21.15.69:0/1796281869 conn(0x7f600409d900 msgr2=0x7f6004162180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 -- 172.21.15.69:0/1796281869 shutdown_connections 2024-04-24T08:09:26.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.500+0000 7f6008e00640 1 -- 172.21.15.69:0/1796281869 wait complete. 2024-04-24T08:09:26.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:09:26.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.710+0000 7febdda00640 1 Processor -- start 2024-04-24T08:09:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.710+0000 7febdda00640 1 -- start start 2024-04-24T08:09:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.712+0000 7febdda00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7febd8156c60 0x7febd815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.712+0000 7febdda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 0x7febd815d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.712+0000 7febdda00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febd815d660 con 0x7febd80ad0d0 2024-04-24T08:09:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.712+0000 7febdda00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7febd806dc30 con 0x7febd8156c60 2024-04-24T08:09:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.712+0000 7febdda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7febd806df10 con 0x7febd80b30f0 2024-04-24T08:09:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.712+0000 7febd7e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7febd8156c60 0x7febd815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.712+0000 7febd7400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 0x7febd815d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.713+0000 7febd7e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7febd8156c60 0x7febd815cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:37960/0 (socket says 172.21.15.69:37960) 2024-04-24T08:09:26.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.713+0000 7febd7e00640 1 -- 172.21.15.69:0/1151905796 learned_addr learned my addr 172.21.15.69:0/1151905796 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:26.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.713+0000 7febd7e00640 1 -- 172.21.15.69:0/1151905796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 msgr2=0x7febd815d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:26.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.713+0000 7febd7e00640 1 --2- 172.21.15.69:0/1151905796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 0x7febd815d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.713+0000 7febd7e00640 1 -- 172.21.15.69:0/1151905796 >> v1:172.21.15.121:6789/0 conn(0x7febd80ad0d0 legacy=0x7febd81600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.713+0000 7febd7e00640 1 -- 172.21.15.69:0/1151905796 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd806e1f0 con 0x7febd8156c60 2024-04-24T08:09:26.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.713+0000 7febd7400640 1 --2- 172.21.15.69:0/1151905796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 0x7febd815d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:26.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.715+0000 7febd7e00640 1 --2- 172.21.15.69:0/1151905796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd8156c60 0x7febd815cac0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7febc8009870 tx=0x7febc802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aac25e7bd28b3e8b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:26.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.715+0000 7febd6a00640 1 -- 172.21.15.69:0/1151905796 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc8037070 con 0x7febd8156c60 2024-04-24T08:09:26.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.715+0000 7febd6a00640 1 -- 172.21.15.69:0/1151905796 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febc8032440 con 0x7febd8156c60 2024-04-24T08:09:26.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.715+0000 7febd6a00640 1 -- 172.21.15.69:0/1151905796 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc8032810 con 0x7febd8156c60 2024-04-24T08:09:26.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.716+0000 7febdda00640 1 -- 172.21.15.69:0/1151905796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd8156c60 msgr2=0x7febd815cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.716+0000 7febdda00640 1 --2- 172.21.15.69:0/1151905796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd8156c60 0x7febd815cac0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7febc8009870 tx=0x7febc802a250 comp rx=0 tx=0).stop 2024-04-24T08:09:26.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.716+0000 7febdda00640 1 -- 172.21.15.69:0/1151905796 shutdown_connections 2024-04-24T08:09:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.716+0000 7febdda00640 1 --2- 172.21.15.69:0/1151905796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 0x7febd815d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.716+0000 7febdda00640 1 --2- 172.21.15.69:0/1151905796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd8156c60 0x7febd815cac0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.716+0000 7febdda00640 1 -- 172.21.15.69:0/1151905796 >> 172.21.15.69:0/1151905796 conn(0x7febd80d0160 msgr2=0x7febd80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.716+0000 7febdda00640 1 -- 172.21.15.69:0/1151905796 shutdown_connections 2024-04-24T08:09:26.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.716+0000 7febdda00640 1 -- 172.21.15.69:0/1151905796 wait complete. 2024-04-24T08:09:26.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.718+0000 7febdda00640 1 Processor -- start 2024-04-24T08:09:26.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.718+0000 7febdda00640 1 -- start start 2024-04-24T08:09:26.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.719+0000 7febdda00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd80ad0d0 0x7febd815b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.719+0000 7febdda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 0x7febd81418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.719+0000 7febdda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd8156c60 0x7febd81f8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.719+0000 7febdda00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7febd80ad530 con 0x7febd8156c60 2024-04-24T08:09:26.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.719+0000 7febdda00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7febd815d940 con 0x7febd80ad0d0 2024-04-24T08:09:26.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.719+0000 7febdda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7febd80ad6b0 con 0x7febd80b30f0 2024-04-24T08:09:26.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.719+0000 7febdd000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd8156c60 0x7febd81f8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.719+0000 7febdd000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd8156c60 0x7febd81f8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43792/0 (socket says 172.21.15.69:43792) 2024-04-24T08:09:26.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.719+0000 7febdd000640 1 -- 172.21.15.69:0/2940551285 learned_addr learned my addr 172.21.15.69:0/2940551285 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febd7e00640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd80ad0d0 0x7febd815b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febd7400640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 0x7febd81418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febdd000640 1 -- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 msgr2=0x7febd81418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febdd000640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 0x7febd81418e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febdd000640 1 -- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd80ad0d0 msgr2=0x7febd815b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febdd000640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd80ad0d0 0x7febd815b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febdd000640 1 -- 172.21.15.69:0/2940551285 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd8141f10 con 0x7febd8156c60 2024-04-24T08:09:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febdd000640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd8156c60 0x7febd81f8aa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7febd000c7a0 tx=0x7febd000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febcfe00640 1 -- 172.21.15.69:0/2940551285 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd0003b70 con 0x7febd8156c60 2024-04-24T08:09:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febcfe00640 1 -- 172.21.15.69:0/2940551285 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd0003d30 con 0x7febd8156c60 2024-04-24T08:09:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febcfe00640 1 -- 172.21.15.69:0/2940551285 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd000aa30 con 0x7febd8156c60 2024-04-24T08:09:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febdda00640 1 -- 172.21.15.69:0/2940551285 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd81421f0 con 0x7febd8156c60 2024-04-24T08:09:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.720+0000 7febdda00640 1 -- 172.21.15.69:0/2940551285 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd81427b0 con 0x7febd8156c60 2024-04-24T08:09:26.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.722+0000 7febdda00640 1 -- 172.21.15.69:0/2940551285 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd80694b0 con 0x7febd8156c60 2024-04-24T08:09:26.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.723+0000 7febcfe00640 1 -- 172.21.15.69:0/2940551285 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febd000abd0 con 0x7febd8156c60 2024-04-24T08:09:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.724+0000 7febcfe00640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febb806c8f0 0x7febb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.724+0000 7febcfe00640 1 -- 172.21.15.69:0/2940551285 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5854+0+0 (secure 0 0 0) 0x7febd008bc00 con 0x7febd8156c60 2024-04-24T08:09:26.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.724+0000 7febd7e00640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febb806c8f0 0x7febb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:26.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.725+0000 7febd7e00640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febb806c8f0 0x7febb806edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7febc8034000 tx=0x7febc802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:26.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.728+0000 7febcfe00640 1 -- 172.21.15.69:0/2940551285 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febd80694b0 con 0x7febd8156c60 2024-04-24T08:09:26.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.971+0000 7febdda00640 1 -- 172.21.15.69:0/2940551285 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7febd80b42b0 con 0x7febd8156c60 2024-04-24T08:09:26.974 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:09:26.973+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:09:26.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.977+0000 7febcfe00640 1 -- 172.21.15.69:0/2940551285 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) ==== 161+0+0 (secure 0 0 0) 0x7febd80b42b0 con 0x7febd8156c60 2024-04-24T08:09:26.978 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:09:26.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 -- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febb806c8f0 msgr2=0x7febb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febb806c8f0 0x7febb806edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7febc8034000 tx=0x7febc802e000 comp rx=0 tx=0).stop 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 -- 172.21.15.69:0/2940551285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd8156c60 msgr2=0x7febd81f8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd8156c60 0x7febd81f8aa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7febd000c7a0 tx=0x7febd000cc60 comp rx=0 tx=0).stop 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 -- 172.21.15.69:0/2940551285 shutdown_connections 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febb806c8f0 0x7febb806edb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febd8156c60 0x7febd81f8aa0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febd80b30f0 0x7febd81418e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 --2- 172.21.15.69:0/2940551285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febd80ad0d0 0x7febd815b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 -- 172.21.15.69:0/2940551285 >> 172.21.15.69:0/2940551285 conn(0x7febd80d0160 msgr2=0x7febd8160450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 -- 172.21.15.69:0/2940551285 shutdown_connections 2024-04-24T08:09:26.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:26.981+0000 7febdda00640 1 -- 172.21.15.69:0/2940551285 wait complete. 2024-04-24T08:09:26.994 INFO:tasks.ceph.mds.c.smithi069.stderr: -4259> 2024-04-24T08:08:20.744+0000 7f4562e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:09:27.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T08:09:27.051 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:09:27.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.188+0000 7f2c1c200640 1 Processor -- start 2024-04-24T08:09:27.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.188+0000 7f2c1c200640 1 -- start start 2024-04-24T08:09:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.190+0000 7f2c1c200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1415ec30 0x7f2c140d1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.190+0000 7f2c1c200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2c140d1690 0x7f2c141688f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.190+0000 7f2c1c200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c140d2710 con 0x7f2c1415e850 2024-04-24T08:09:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.190+0000 7f2c1c200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2c140d37b0 con 0x7f2c140d1690 2024-04-24T08:09:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.190+0000 7f2c1c200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2c1406eba0 con 0x7f2c1415ec30 2024-04-24T08:09:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.190+0000 7f2c1a200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2c140d1690 0x7f2c141688f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.190+0000 7f2c19800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2c1415e850 0x7f2c140cff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52680/0 (socket says 172.21.15.69:52680) 2024-04-24T08:09:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.190+0000 7f2c19800640 1 -- 172.21.15.69:0/4142291624 learned_addr learned my addr 172.21.15.69:0/4142291624 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.191+0000 7f2c18e00640 1 --2- 172.21.15.69:0/4142291624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1415ec30 0x7f2c140d1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.191+0000 7f2c1a200640 1 -- 172.21.15.69:0/4142291624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1415ec30 msgr2=0x7f2c140d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.191+0000 7f2c1a200640 1 --2- 172.21.15.69:0/4142291624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1415ec30 0x7f2c140d1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.191+0000 7f2c1a200640 1 -- 172.21.15.69:0/4142291624 >> v1:172.21.15.69:6789/0 conn(0x7f2c1415e850 legacy=0x7f2c140cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.191+0000 7f2c1a200640 1 -- 172.21.15.69:0/4142291624 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c140d29f0 con 0x7f2c140d1690 2024-04-24T08:09:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.191+0000 7f2c13e00640 1 -- 172.21.15.69:0/4142291624 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212286373 0 0) 0x7f2c140d2710 con 0x7f2c1415e850 2024-04-24T08:09:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.192+0000 7f2c1a200640 1 --2- 172.21.15.69:0/4142291624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 0x7f2c141688f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009d90 tx=0x7f2c0c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc261b1de2fa7c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.192+0000 7f2c13e00640 1 -- 172.21.15.69:0/4142291624 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c007740 con 0x7f2c140d1690 2024-04-24T08:09:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.192+0000 7f2c13e00640 1 -- 172.21.15.69:0/4142291624 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c0c0078e0 con 0x7f2c140d1690 2024-04-24T08:09:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.192+0000 7f2c13e00640 1 -- 172.21.15.69:0/4142291624 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c007cb0 con 0x7f2c140d1690 2024-04-24T08:09:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.193+0000 7f2c1c200640 1 -- 172.21.15.69:0/4142291624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 msgr2=0x7f2c141688f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.193+0000 7f2c1c200640 1 --2- 172.21.15.69:0/4142291624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 0x7f2c141688f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009d90 tx=0x7f2c0c02a660 comp rx=0 tx=0).stop 2024-04-24T08:09:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.193+0000 7f2c1c200640 1 -- 172.21.15.69:0/4142291624 shutdown_connections 2024-04-24T08:09:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.193+0000 7f2c1c200640 1 --2- 172.21.15.69:0/4142291624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 0x7f2c141688f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.193+0000 7f2c1c200640 1 --2- 172.21.15.69:0/4142291624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1415ec30 0x7f2c140d1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.193+0000 7f2c1c200640 1 -- 172.21.15.69:0/4142291624 >> 172.21.15.69:0/4142291624 conn(0x7f2c1408bf60 msgr2=0x7f2c140d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.193+0000 7f2c1c200640 1 -- 172.21.15.69:0/4142291624 shutdown_connections 2024-04-24T08:09:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.193+0000 7f2c1c200640 1 -- 172.21.15.69:0/4142291624 wait complete. 2024-04-24T08:09:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.195+0000 7f2c1c200640 1 Processor -- start 2024-04-24T08:09:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.195+0000 7f2c1c200640 1 -- start start 2024-04-24T08:09:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.196+0000 7f2c1c200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 0x7f2c141575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.196+0000 7f2c1c200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1415e850 0x7f2c140caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.196+0000 7f2c1c200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1415ec30 0x7f2c140b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.196+0000 7f2c1c200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2c140d2fb0 con 0x7f2c140d1690 2024-04-24T08:09:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.196+0000 7f2c1c200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2c140d1ae0 con 0x7f2c1415ec30 2024-04-24T08:09:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.196+0000 7f2c1c200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2c140d3290 con 0x7f2c1415e850 2024-04-24T08:09:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.196+0000 7f2c19800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 0x7f2c141575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.196+0000 7f2c19800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 0x7f2c141575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43832/0 (socket says 172.21.15.69:43832) 2024-04-24T08:09:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.196+0000 7f2c19800640 1 -- 172.21.15.69:0/2324735749 learned_addr learned my addr 172.21.15.69:0/2324735749 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c18e00640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1415e850 0x7f2c140caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c1a200640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1415ec30 0x7f2c140b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c19800640 1 -- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1415e850 msgr2=0x7f2c140caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c19800640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1415e850 0x7f2c140caef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c19800640 1 -- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1415ec30 msgr2=0x7f2c140b7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c19800640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1415ec30 0x7f2c140b7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c19800640 1 -- 172.21.15.69:0/2324735749 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c140cda30 con 0x7f2c140d1690 2024-04-24T08:09:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c19800640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 0x7f2c141575b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800c4d0 tx=0x7f2c0800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c11600640 1 -- 172.21.15.69:0/2324735749 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c08018070 con 0x7f2c140d1690 2024-04-24T08:09:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c11600640 1 -- 172.21.15.69:0/2324735749 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c08007500 con 0x7f2c140d1690 2024-04-24T08:09:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c11600640 1 -- 172.21.15.69:0/2324735749 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c08004a20 con 0x7f2c140d1690 2024-04-24T08:09:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.197+0000 7f2c1c200640 1 -- 172.21.15.69:0/2324735749 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c140cdd10 con 0x7f2c140d1690 2024-04-24T08:09:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.198+0000 7f2c1c200640 1 -- 172.21.15.69:0/2324735749 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c140b98f0 con 0x7f2c140d1690 2024-04-24T08:09:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.199+0000 7f2c1c200640 1 -- 172.21.15.69:0/2324735749 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c140ce230 con 0x7f2c140d1690 2024-04-24T08:09:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.200+0000 7f2c11600640 1 -- 172.21.15.69:0/2324735749 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c08004bc0 con 0x7f2c140d1690 2024-04-24T08:09:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.201+0000 7f2c11600640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2bf406c8f0 0x7f2bf406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.201+0000 7f2c11600640 1 -- 172.21.15.69:0/2324735749 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5940+0+0 (secure 0 0 0) 0x7f2c0808afd0 con 0x7f2c140d1690 2024-04-24T08:09:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.201+0000 7f2c18e00640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2bf406c8f0 0x7f2bf406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.202+0000 7f2c18e00640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2bf406c8f0 0x7f2bf406edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2c00006fd0 tx=0x7f2c00008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:27.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.205+0000 7f2c11600640 1 -- 172.21.15.69:0/2324735749 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c08059270 con 0x7f2c140d1690 2024-04-24T08:09:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.449+0000 7f2c1c200640 1 -- 172.21.15.69:0/2324735749 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f2c14068a80 con 0x7f2c140d1690 2024-04-24T08:09:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.980+0000 7f2c11600640 1 -- 172.21.15.69:0/2324735749 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) ==== 104+0+0 (secure 0 0 0) 0x7f2c08058b70 con 0x7f2c140d1690 2024-04-24T08:09:27.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.983+0000 7f2c1c200640 1 -- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2bf406c8f0 msgr2=0x7f2bf406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:27.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.983+0000 7f2c1c200640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2bf406c8f0 0x7f2bf406edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2c00006fd0 tx=0x7f2c00008040 comp rx=0 tx=0).stop 2024-04-24T08:09:27.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.983+0000 7f2c1c200640 1 -- 172.21.15.69:0/2324735749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 msgr2=0x7f2c141575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:27.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.983+0000 7f2c1c200640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 0x7f2c141575b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2c0800c4d0 tx=0x7f2c0800c990 comp rx=0 tx=0).stop 2024-04-24T08:09:27.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.983+0000 7f2c1c200640 1 -- 172.21.15.69:0/2324735749 shutdown_connections 2024-04-24T08:09:27.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.983+0000 7f2c1c200640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2bf406c8f0 0x7f2bf406edb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:27.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.983+0000 7f2c1c200640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2c1415ec30 0x7f2c140b7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:27.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.983+0000 7f2c1c200640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2c1415e850 0x7f2c140caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:27.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.983+0000 7f2c1c200640 1 --2- 172.21.15.69:0/2324735749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2c140d1690 0x7f2c141575b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:27.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.983+0000 7f2c1c200640 1 -- 172.21.15.69:0/2324735749 >> 172.21.15.69:0/2324735749 conn(0x7f2c1408bf60 msgr2=0x7f2c140b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:27.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.984+0000 7f2c1c200640 1 -- 172.21.15.69:0/2324735749 shutdown_connections 2024-04-24T08:09:27.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:27.984+0000 7f2c1c200640 1 -- 172.21.15.69:0/2324735749 wait complete. 2024-04-24T08:09:28.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T08:09:28.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.256+0000 7ff942800640 1 Processor -- start 2024-04-24T08:09:28.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.257+0000 7ff942800640 1 -- start start 2024-04-24T08:09:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.258+0000 7ff942800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff93c156810 0x7ff93c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.258+0000 7ff942800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff93c06fa50 con 0x7ff93c160200 2024-04-24T08:09:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.258+0000 7ff942800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff93c160750 con 0x7ff93c0b3610 2024-04-24T08:09:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.258+0000 7ff942800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff93c0b44e0 con 0x7ff93c156810 2024-04-24T08:09:28.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.258+0000 7ff941400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff93c156810 0x7ff93c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:28.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.258+0000 7ff940a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff93c160200 0x7ff93c0cb810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52692/0 (socket says 172.21.15.69:52692) 2024-04-24T08:09:28.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.258+0000 7ff940a00640 1 -- 172.21.15.69:0/3949179331 learned_addr learned my addr 172.21.15.69:0/3949179331 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:28.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.259+0000 7ff941400640 1 -- 172.21.15.69:0/3949179331 >> v1:172.21.15.121:6789/0 conn(0x7ff93c0b3610 legacy=0x7ff93c0d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:28.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.259+0000 7ff941400640 1 -- 172.21.15.69:0/3949179331 >> v1:172.21.15.69:6789/0 conn(0x7ff93c160200 legacy=0x7ff93c0cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:28.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.259+0000 7ff941400640 1 -- 172.21.15.69:0/3949179331 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93c160a30 con 0x7ff93c156810 2024-04-24T08:09:28.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.260+0000 7ff941400640 1 --2- 172.21.15.69:0/3949179331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 0x7ff93c156bf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff9300097a0 tx=0x7ff93002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acf60b91e71a1bf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:28.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.260+0000 7ff93be00640 1 -- 172.21.15.69:0/3949179331 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930037070 con 0x7ff93c156810 2024-04-24T08:09:28.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.260+0000 7ff93be00640 1 -- 172.21.15.69:0/3949179331 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff930032440 con 0x7ff93c156810 2024-04-24T08:09:28.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.260+0000 7ff93be00640 1 -- 172.21.15.69:0/3949179331 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff930032810 con 0x7ff93c156810 2024-04-24T08:09:28.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.261+0000 7ff942800640 1 -- 172.21.15.69:0/3949179331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 msgr2=0x7ff93c156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:28.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.261+0000 7ff942800640 1 --2- 172.21.15.69:0/3949179331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 0x7ff93c156bf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ff9300097a0 tx=0x7ff93002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.261+0000 7ff942800640 1 -- 172.21.15.69:0/3949179331 shutdown_connections 2024-04-24T08:09:28.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.261+0000 7ff942800640 1 --2- 172.21.15.69:0/3949179331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 0x7ff93c156bf0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.261+0000 7ff942800640 1 -- 172.21.15.69:0/3949179331 >> 172.21.15.69:0/3949179331 conn(0x7ff93c06be40 msgr2=0x7ff93c06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:28.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.261+0000 7ff942800640 1 -- 172.21.15.69:0/3949179331 shutdown_connections 2024-04-24T08:09:28.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.261+0000 7ff942800640 1 -- 172.21.15.69:0/3949179331 wait complete. 2024-04-24T08:09:28.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.264+0000 7ff942800640 1 Processor -- start 2024-04-24T08:09:28.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.264+0000 7ff942800640 1 -- start start 2024-04-24T08:09:28.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.265+0000 7ff942800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff93c0b3610 0x7ff93c0c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:28.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.265+0000 7ff942800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 0x7ff93c0cea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:28.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.265+0000 7ff942800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff93c160200 0x7ff93c15dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:28.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.265+0000 7ff942800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff93c0cbf20 con 0x7ff93c156810 2024-04-24T08:09:28.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.265+0000 7ff942800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff93c0b3180 con 0x7ff93c0b3610 2024-04-24T08:09:28.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.265+0000 7ff942800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff93c160ff0 con 0x7ff93c160200 2024-04-24T08:09:28.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.265+0000 7ff940a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 0x7ff93c0cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:28.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.265+0000 7ff940a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 0x7ff93c0cea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43856/0 (socket says 172.21.15.69:43856) 2024-04-24T08:09:28.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.265+0000 7ff940a00640 1 -- 172.21.15.69:0/3734381533 learned_addr learned my addr 172.21.15.69:0/3734381533 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:28.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff941400640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff93c0b3610 0x7ff93c0c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:28.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff941e00640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff93c160200 0x7ff93c15dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:28.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff940a00640 1 -- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff93c160200 msgr2=0x7ff93c15dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:28.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff940a00640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff93c160200 0x7ff93c15dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff940a00640 1 -- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff93c0b3610 msgr2=0x7ff93c0c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:28.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff940a00640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff93c0b3610 0x7ff93c0c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff940a00640 1 -- 172.21.15.69:0/3734381533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93c0ae010 con 0x7ff93c156810 2024-04-24T08:09:28.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff940a00640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 0x7ff93c0cea60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff92800c440 tx=0x7ff92800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:28.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff939600640 1 -- 172.21.15.69:0/3734381533 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff928017070 con 0x7ff93c156810 2024-04-24T08:09:28.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff939600640 1 -- 172.21.15.69:0/3734381533 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff928005ce0 con 0x7ff93c156810 2024-04-24T08:09:28.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.266+0000 7ff942800640 1 -- 172.21.15.69:0/3734381533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff93c0ae2f0 con 0x7ff93c156810 2024-04-24T08:09:28.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.267+0000 7ff939600640 1 -- 172.21.15.69:0/3734381533 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9280072f0 con 0x7ff93c156810 2024-04-24T08:09:28.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.267+0000 7ff942800640 1 -- 172.21.15.69:0/3734381533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff93c0ae8b0 con 0x7ff93c156810 2024-04-24T08:09:28.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.268+0000 7ff942800640 1 -- 172.21.15.69:0/3734381533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff93c068a80 con 0x7ff93c156810 2024-04-24T08:09:28.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.269+0000 7ff939600640 1 -- 172.21.15.69:0/3734381533 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff92801a070 con 0x7ff93c156810 2024-04-24T08:09:28.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.270+0000 7ff939600640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff91c06c920 0x7ff91c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:28.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.270+0000 7ff939600640 1 -- 172.21.15.69:0/3734381533 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5940+0+0 (secure 0 0 0) 0x7ff92808aec0 con 0x7ff93c156810 2024-04-24T08:09:28.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.271+0000 7ff941400640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff91c06c920 0x7ff91c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:28.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.272+0000 7ff941400640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff91c06c920 0x7ff91c06ede0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff930034000 tx=0x7ff93002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:28.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.281+0000 7ff939600640 1 -- 172.21.15.69:0/3734381533 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff928059290 con 0x7ff93c156810 2024-04-24T08:09:28.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.483+0000 7ff942800640 1 -- 172.21.15.69:0/3734381533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff93c06e5c0 con 0x7ff93c156810 2024-04-24T08:09:28.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.984+0000 7ff939600640 1 -- 172.21.15.69:0/3734381533 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) ==== 181+0+0 (secure 0 0 0) 0x7ff928058b90 con 0x7ff93c156810 2024-04-24T08:09:28.985 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' removed 2024-04-24T08:09:28.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.987+0000 7ff942800640 1 -- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff91c06c920 msgr2=0x7ff91c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.987+0000 7ff942800640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff91c06c920 0x7ff91c06ede0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff930034000 tx=0x7ff93002e000 comp rx=0 tx=0).stop 2024-04-24T08:09:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.987+0000 7ff942800640 1 -- 172.21.15.69:0/3734381533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 msgr2=0x7ff93c0cea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.987+0000 7ff942800640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 0x7ff93c0cea60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7ff92800c440 tx=0x7ff92800c900 comp rx=0 tx=0).stop 2024-04-24T08:09:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.987+0000 7ff942800640 1 -- 172.21.15.69:0/3734381533 shutdown_connections 2024-04-24T08:09:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.987+0000 7ff942800640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff91c06c920 0x7ff91c06ede0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.987+0000 7ff942800640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff93c156810 0x7ff93c0cea60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.987+0000 7ff942800640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff93c160200 0x7ff93c15dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.987+0000 7ff942800640 1 --2- 172.21.15.69:0/3734381533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff93c0b3610 0x7ff93c0c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.988+0000 7ff942800640 1 -- 172.21.15.69:0/3734381533 >> 172.21.15.69:0/3734381533 conn(0x7ff93c06be40 msgr2=0x7ff93c167be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:28.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.988+0000 7ff942800640 1 -- 172.21.15.69:0/3734381533 shutdown_connections 2024-04-24T08:09:28.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:28.988+0000 7ff942800640 1 -- 172.21.15.69:0/3734381533 wait complete. 2024-04-24T08:09:29.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T08:09:29.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.212+0000 7f7328400640 1 Processor -- start 2024-04-24T08:09:29.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.212+0000 7f7328400640 1 -- start start 2024-04-24T08:09:29.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.213+0000 7f7328400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73200c3380 0x7f73201671d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:29.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.213+0000 7f7328400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f73200b3980 0x7f73200b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:29.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.213+0000 7f7328400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f732006e2d0 con 0x7f73200c3760 2024-04-24T08:09:29.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.213+0000 7f7328400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7320167710 con 0x7f73200b3980 2024-04-24T08:09:29.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.213+0000 7f7328400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f732006e5b0 con 0x7f73200c3380 2024-04-24T08:09:29.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.214+0000 7f7325000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f73200c3760 0x7f732006dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32798/0 (socket says 172.21.15.69:32798) 2024-04-24T08:09:29.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.214+0000 7f7325000640 1 -- 172.21.15.69:0/4167504970 learned_addr learned my addr 172.21.15.69:0/4167504970 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.214+0000 7f7326400640 1 --2- 172.21.15.69:0/4167504970 >> v2:172.21.15.121:3300/0 conn(0x7f73200b3980 0x7f73200b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.214+0000 7f7325a00640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73200c3380 0x7f73201671d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.214+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2987142480 0 0) 0x7f732006e2d0 con 0x7f73200c3760 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.214+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7308003520 con 0x7f73200c3760 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2956210823 0 0) 0x7f7308003520 con 0x7f73200c3760 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73200c3380 msgr2=0x7f73201671d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73200c3380 0x7f73201671d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 >> v2:172.21.15.121:3300/0 conn(0x7f73200b3980 msgr2=0x7f73200b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 --2- 172.21.15.69:0/4167504970 >> v2:172.21.15.121:3300/0 conn(0x7f73200b3980 0x7f73200b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732006e890 con 0x7f73200c3760 2024-04-24T08:09:29.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f730c001670 con 0x7f73200c3760 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 >> v1:172.21.15.69:6789/0 conn(0x7f73200c3760 legacy=0x7f732006dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73080039b0 0x7f7308003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 0x7f7308004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73200c3380 0x7f73080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7320167710 con 0x7f7308004800 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f732006e5b0 con 0x7f73080039b0 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f730800b9c0 con 0x7f73200c3380 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f7325000640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 0x7f7308004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f7325000640 1 -- 172.21.15.69:0/4167504970 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73200c3380 msgr2=0x7f73080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f7325a00640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73080039b0 0x7f7308003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f7325000640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73200c3380 0x7f73080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.215+0000 7f7325000640 1 -- 172.21.15.69:0/4167504970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73080039b0 msgr2=0x7f7308003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:29.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.216+0000 7f7325000640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73080039b0 0x7f7308003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.216+0000 7f7325000640 1 -- 172.21.15.69:0/4167504970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7308003520 con 0x7f7308004800 2024-04-24T08:09:29.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.217+0000 7f7325000640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 0x7f7308004c80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f730c00c240 tx=0x7f7308005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ff46d9110ef3c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:29.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.217+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c007d50 con 0x7f7308004800 2024-04-24T08:09:29.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.217+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f730c005af0 con 0x7f7308004800 2024-04-24T08:09:29.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.217+0000 7f731fe00640 1 -- 172.21.15.69:0/4167504970 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f730c007070 con 0x7f7308004800 2024-04-24T08:09:29.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.217+0000 7f7328400640 1 -- 172.21.15.69:0/4167504970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 msgr2=0x7f7308004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:29.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.217+0000 7f7328400640 1 --2- 172.21.15.69:0/4167504970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 0x7f7308004c80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f730c00c240 tx=0x7f7308005830 comp rx=0 tx=0).stop 2024-04-24T08:09:29.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.217+0000 7f7326400640 1 -- 172.21.15.69:0/4167504970 reap_dead start 2024-04-24T08:09:29.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.217+0000 7f7328400640 1 -- 172.21.15.69:0/4167504970 shutdown_connections 2024-04-24T08:09:29.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.217+0000 7f7328400640 1 -- 172.21.15.69:0/4167504970 >> 172.21.15.69:0/4167504970 conn(0x7f73200b2380 msgr2=0x7f73200b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:29.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.218+0000 7f7328400640 1 -- 172.21.15.69:0/4167504970 shutdown_connections 2024-04-24T08:09:29.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.218+0000 7f7328400640 1 -- 172.21.15.69:0/4167504970 wait complete. 2024-04-24T08:09:29.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.219+0000 7f7328400640 1 Processor -- start 2024-04-24T08:09:29.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.219+0000 7f7328400640 1 -- start start 2024-04-24T08:09:29.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.220+0000 7f7328400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 0x7f73201fc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:29.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.220+0000 7f7328400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7320072f40 0x7f73201611d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:29.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.220+0000 7f7328400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73201f8710 0x7f732015df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:29.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.220+0000 7f7328400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73200c3bc0 con 0x7f7308004800 2024-04-24T08:09:29.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.220+0000 7f7328400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f732006ee50 con 0x7f7320072f40 2024-04-24T08:09:29.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.220+0000 7f7328400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f73200c3d40 con 0x7f73201f8710 2024-04-24T08:09:29.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.221+0000 7f7325a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 0x7f73201fc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:29.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.221+0000 7f7325a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 0x7f73201fc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58534/0 (socket says 172.21.15.69:58534) 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.221+0000 7f7325a00640 1 -- 172.21.15.69:0/1253767524 learned_addr learned my addr 172.21.15.69:0/1253767524 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.221+0000 7f7325a00640 1 -- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73201f8710 msgr2=0x7f732015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.221+0000 7f7325a00640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73201f8710 0x7f732015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.221+0000 7f7325a00640 1 -- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7320072f40 msgr2=0x7f73201611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.221+0000 7f7325000640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7320072f40 0x7f73201611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.221+0000 7f7325a00640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7320072f40 0x7f73201611d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.221+0000 7f7325a00640 1 -- 172.21.15.69:0/1253767524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73201f9670 con 0x7f7308004800 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.221+0000 7f7325a00640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 0x7f73201fc640 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f731400f380 tx=0x7f731400f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.222+0000 7f731d600640 1 -- 172.21.15.69:0/1253767524 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731401c070 con 0x7f7308004800 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.222+0000 7f731d600640 1 -- 172.21.15.69:0/1253767524 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7314017620 con 0x7f7308004800 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.222+0000 7f731d600640 1 -- 172.21.15.69:0/1253767524 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7314017a10 con 0x7f7308004800 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.222+0000 7f7328400640 1 -- 172.21.15.69:0/1253767524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73201fcb80 con 0x7f7308004800 2024-04-24T08:09:29.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.222+0000 7f7328400640 1 -- 172.21.15.69:0/1253767524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73201f9390 con 0x7f7308004800 2024-04-24T08:09:29.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.223+0000 7f7328400640 1 -- 172.21.15.69:0/1253767524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7320068a80 con 0x7f7308004800 2024-04-24T08:09:29.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.224+0000 7f731d600640 1 -- 172.21.15.69:0/1253767524 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7314005d10 con 0x7f7308004800 2024-04-24T08:09:29.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.225+0000 7f731d600640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c8f0 0x7f730006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:29.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.225+0000 7f731d600640 1 -- 172.21.15.69:0/1253767524 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5502+0+0 (secure 0 0 0) 0x7f731408bf90 con 0x7f7308004800 2024-04-24T08:09:29.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.226+0000 7f7325000640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c8f0 0x7f730006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:29.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.227+0000 7f7325000640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c8f0 0x7f730006edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f730c033000 tx=0x7f730c009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:29.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.230+0000 7f731d600640 1 -- 172.21.15.69:0/1253767524 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f73201f9670 con 0x7f7308004800 2024-04-24T08:09:29.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.440+0000 7f7328400640 1 -- 172.21.15.69:0/1253767524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f73200b33f0 con 0x7f7308004800 2024-04-24T08:09:29.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.987+0000 7f731d600640 1 -- 172.21.15.69:0/1253767524 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) ==== 169+0+0 (secure 0 0 0) 0x7f7314012070 con 0x7f7308004800 2024-04-24T08:09:29.989 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' removed 2024-04-24T08:09:29.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 -- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c8f0 msgr2=0x7f730006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c8f0 0x7f730006edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f730c033000 tx=0x7f730c009030 comp rx=0 tx=0).stop 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 -- 172.21.15.69:0/1253767524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 msgr2=0x7f73201fc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 0x7f73201fc640 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f731400f380 tx=0x7f731400f840 comp rx=0 tx=0).stop 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 -- 172.21.15.69:0/1253767524 shutdown_connections 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f730006c8f0 0x7f730006edb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73201f8710 0x7f732015df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7320072f40 0x7f73201611d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 --2- 172.21.15.69:0/1253767524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7308004800 0x7f73201fc640 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 -- 172.21.15.69:0/1253767524 >> 172.21.15.69:0/1253767524 conn(0x7f73200b2380 msgr2=0x7f73200afed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 -- 172.21.15.69:0/1253767524 shutdown_connections 2024-04-24T08:09:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:29.992+0000 7f7328400640 1 -- 172.21.15.69:0/1253767524 wait complete. 2024-04-24T08:09:30.032 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T08:09:30.033 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-24T08:09:30.033 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T08:09:30.033 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:09:30.033 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T08:09:30.033 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-24T08:09:30.034 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T08:09:30.034 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:09:30.034 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T08:09:30.034 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-24T08:09:30.034 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T08:09:30.034 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:09:30.034 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T08:09:30.034 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-24T08:09:30.035 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T08:09:30.035 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:09:30.049 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T08:09:30.049 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T08:09:30.050 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T08:09:30.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T08:09:30.053 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T08:09:30.053 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T08:09:30.054 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T08:09:30.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T08:09:30.094 INFO:tasks.ceph.mds.a:Started 2024-04-24T08:09:30.095 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:30.099 INFO:tasks.ceph.mds.c:Started 2024-04-24T08:09:30.099 INFO:tasks.ceph.mds.b:Started 2024-04-24T08:09:30.099 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:30.099 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:30.101 INFO:tasks.ceph.mds.d:Started 2024-04-24T08:09:30.101 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:30.102 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-24T08:09:30.237 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:09:30.243 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:09:30.245 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-24T08:09:30.245 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:09:30.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.280+0000 7f1aafa00640 1 Processor -- start 2024-04-24T08:09:30.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.281+0000 7f1aafa00640 1 -- start start 2024-04-24T08:09:30.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.282+0000 7f1aafa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1aa80d0300 0x7f1aa806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.282+0000 7f1aafa00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1aa8145a90 0x7f1aa80d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.282+0000 7f1aafa00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aa80d0c20 con 0x7f1aa8145fe0 2024-04-24T08:09:30.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.282+0000 7f1aafa00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1aa80cf3d0 con 0x7f1aa80d0300 2024-04-24T08:09:30.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.282+0000 7f1aafa00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1aa80cf6b0 con 0x7f1aa8145a90 2024-04-24T08:09:30.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.282+0000 7f1aad000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1aa80d0300 0x7f1aa806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.282+0000 7f1aada00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1aa8145fe0 0x7f1aa8160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32804/0 (socket says 172.21.15.69:32804) 2024-04-24T08:09:30.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.282+0000 7f1aada00640 1 -- 172.21.15.69:0/3193318728 learned_addr learned my addr 172.21.15.69:0/3193318728 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:30.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.282+0000 7f1a9fe00640 1 --2- 172.21.15.69:0/3193318728 >> v2:172.21.15.121:3300/0 conn(0x7f1aa8145a90 0x7f1aa80d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.283+0000 7f1aad000640 1 -- 172.21.15.69:0/3193318728 >> v2:172.21.15.121:3300/0 conn(0x7f1aa8145a90 msgr2=0x7f1aa80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:30.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.283+0000 7f1aad000640 1 --2- 172.21.15.69:0/3193318728 >> v2:172.21.15.121:3300/0 conn(0x7f1aa8145a90 0x7f1aa80d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.283+0000 7f1aad000640 1 -- 172.21.15.69:0/3193318728 >> v1:172.21.15.69:6789/0 conn(0x7f1aa8145fe0 legacy=0x7f1aa8160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.283+0000 7f1aad000640 1 -- 172.21.15.69:0/3193318728 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa80cf990 con 0x7f1aa80d0300 2024-04-24T08:09:30.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.283+0000 7f1aa7e00640 1 -- 172.21.15.69:0/3193318728 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2171920433 0 0) 0x7f1aa80d0c20 con 0x7f1aa8145fe0 2024-04-24T08:09:30.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.284+0000 7f1aad000640 1 --2- 172.21.15.69:0/3193318728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa80d0300 0x7f1aa806e240 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80b5780 tx=0x7f1a9802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982a2d3211540e14 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:30.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.284+0000 7f1aa7e00640 1 -- 172.21.15.69:0/3193318728 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98037070 con 0x7f1aa80d0300 2024-04-24T08:09:30.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.284+0000 7f1aa7e00640 1 -- 172.21.15.69:0/3193318728 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a98032440 con 0x7f1aa80d0300 2024-04-24T08:09:30.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.284+0000 7f1aa7e00640 1 -- 172.21.15.69:0/3193318728 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a98032810 con 0x7f1aa80d0300 2024-04-24T08:09:30.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.284+0000 7f1aafa00640 1 -- 172.21.15.69:0/3193318728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa80d0300 msgr2=0x7f1aa806e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.285+0000 7f1aafa00640 1 --2- 172.21.15.69:0/3193318728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa80d0300 0x7f1aa806e240 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80b5780 tx=0x7f1a9802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.285+0000 7f1aafa00640 1 -- 172.21.15.69:0/3193318728 shutdown_connections 2024-04-24T08:09:30.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.285+0000 7f1aafa00640 1 --2- 172.21.15.69:0/3193318728 >> v2:172.21.15.121:3300/0 conn(0x7f1aa8145a90 0x7f1aa80d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.285+0000 7f1aafa00640 1 --2- 172.21.15.69:0/3193318728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa80d0300 0x7f1aa806e240 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.285+0000 7f1aafa00640 1 -- 172.21.15.69:0/3193318728 >> 172.21.15.69:0/3193318728 conn(0x7f1aa8067d90 msgr2=0x7f1aa806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:30.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.285+0000 7f1aafa00640 1 -- 172.21.15.69:0/3193318728 shutdown_connections 2024-04-24T08:09:30.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.285+0000 7f1aafa00640 1 -- 172.21.15.69:0/3193318728 wait complete. 2024-04-24T08:09:30.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.286+0000 7f1aafa00640 1 Processor -- start 2024-04-24T08:09:30.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.287+0000 7f1aafa00640 1 -- start start 2024-04-24T08:09:30.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.287+0000 7f1aafa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1aa80d0300 0x7f1aa80acf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.287+0000 7f1aafa00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8145a90 0x7f1aa8138b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.287+0000 7f1aafa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa8145fe0 0x7f1aa81f2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.287+0000 7f1aafa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa8145770 con 0x7f1aa8145fe0 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.287+0000 7f1aafa00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa80cff50 con 0x7f1aa8145a90 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.287+0000 7f1aafa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa81458f0 con 0x7f1aa80d0300 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.288+0000 7f1aada00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa8145fe0 0x7f1aa81f2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.288+0000 7f1aada00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa8145fe0 0x7f1aa81f2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58588/0 (socket says 172.21.15.69:58588) 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.288+0000 7f1aada00640 1 -- 172.21.15.69:0/633079290 learned_addr learned my addr 172.21.15.69:0/633079290 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.288+0000 7f1a9fe00640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8145a90 0x7f1aa8138b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.288+0000 7f1aad000640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1aa80d0300 0x7f1aa80acf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.288+0000 7f1aada00640 1 -- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1aa80d0300 msgr2=0x7f1aa80acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.288+0000 7f1aada00640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1aa80d0300 0x7f1aa80acf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.288+0000 7f1aada00640 1 -- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8145a90 msgr2=0x7f1aa8138b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.288+0000 7f1aada00640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8145a90 0x7f1aa8138b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.288+0000 7f1aada00640 1 -- 172.21.15.69:0/633079290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa80ad810 con 0x7f1aa8145fe0 2024-04-24T08:09:30.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.289+0000 7f1aada00640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa8145fe0 0x7f1aa81f2830 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000c7a0 tx=0x7f1aa000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:30.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.289+0000 7f1aa5600640 1 -- 172.21.15.69:0/633079290 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0018070 con 0x7f1aa8145fe0 2024-04-24T08:09:30.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.289+0000 7f1aa5600640 1 -- 172.21.15.69:0/633079290 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa000a640 con 0x7f1aa8145fe0 2024-04-24T08:09:30.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.289+0000 7f1aa5600640 1 -- 172.21.15.69:0/633079290 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa000aab0 con 0x7f1aa8145fe0 2024-04-24T08:09:30.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.289+0000 7f1aafa00640 1 -- 172.21.15.69:0/633079290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa80adaf0 con 0x7f1aa8145fe0 2024-04-24T08:09:30.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.289+0000 7f1aafa00640 1 -- 172.21.15.69:0/633079290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa80ae0b0 con 0x7f1aa8145fe0 2024-04-24T08:09:30.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.290+0000 7f1aafa00640 1 -- 172.21.15.69:0/633079290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa8068a80 con 0x7f1aa8145fe0 2024-04-24T08:09:30.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.291+0000 7f1aa5600640 1 -- 172.21.15.69:0/633079290 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1aa001b070 con 0x7f1aa8145fe0 2024-04-24T08:09:30.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.292+0000 7f1aa5600640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a8406c8f0 0x7f1a8406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.292+0000 7f1aa5600640 1 -- 172.21.15.69:0/633079290 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 4943+0+0 (secure 0 0 0) 0x7f1aa008ae70 con 0x7f1aa8145fe0 2024-04-24T08:09:30.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.293+0000 7f1aad000640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a8406c8f0 0x7f1a8406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.294+0000 7f1aad000640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a8406c8f0 0x7f1a8406edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1a98034000 tx=0x7f1a9802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:30.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.298+0000 7f1aa5600640 1 -- 172.21.15.69:0/633079290 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1aa0058eb0 con 0x7f1aa8145fe0 2024-04-24T08:09:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:30.511+0000 7f1aafa00640 1 -- 172.21.15.69:0/633079290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f1aa806c640 con 0x7f1aa8145fe0 2024-04-24T08:09:31.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.028+0000 7f1aa5600640 1 -- 172.21.15.69:0/633079290 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) ==== 97+0+0 (secure 0 0 0) 0x7f1aa005d1b0 con 0x7f1aa8145fe0 2024-04-24T08:09:31.028 INFO:teuthology.orchestra.run.smithi069.stderr: removed all blocklist entries 2024-04-24T08:09:31.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 -- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a8406c8f0 msgr2=0x7f1a8406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a8406c8f0 0x7f1a8406edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1a98034000 tx=0x7f1a9802e000 comp rx=0 tx=0).stop 2024-04-24T08:09:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 -- 172.21.15.69:0/633079290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa8145fe0 msgr2=0x7f1aa81f2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa8145fe0 0x7f1aa81f2830 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1aa000c7a0 tx=0x7f1aa000cc60 comp rx=0 tx=0).stop 2024-04-24T08:09:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 -- 172.21.15.69:0/633079290 shutdown_connections 2024-04-24T08:09:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a8406c8f0 0x7f1a8406edb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1aa8145fe0 0x7f1aa81f2830 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1aa8145a90 0x7f1aa8138b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 --2- 172.21.15.69:0/633079290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1aa80d0300 0x7f1aa80acf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 -- 172.21.15.69:0/633079290 >> 172.21.15.69:0/633079290 conn(0x7f1aa8067d90 msgr2=0x7f1aa80d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 -- 172.21.15.69:0/633079290 shutdown_connections 2024-04-24T08:09:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.032+0000 7f1aafa00640 1 -- 172.21.15.69:0/633079290 wait complete. 2024-04-24T08:09:31.053 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-24T08:09:31.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.248+0000 7fe54a800640 1 Processor -- start 2024-04-24T08:09:31.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.249+0000 7fe54a800640 1 -- start start 2024-04-24T08:09:31.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.250+0000 7fe54a800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe544162c70 0x7fe5440b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.250+0000 7fe54a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5440b3d30 0x7fe5440b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.250+0000 7fe54a800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5441669d0 con 0x7fe544071d70 2024-04-24T08:09:31.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.250+0000 7fe54a800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe5440b46d0 con 0x7fe544162c70 2024-04-24T08:09:31.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.250+0000 7fe54a800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe544166cb0 con 0x7fe5440b3d30 2024-04-24T08:09:31.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.250+0000 7fe543e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe544071d70 0x7fe5440721a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32816/0 (socket says 172.21.15.69:32816) 2024-04-24T08:09:31.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.250+0000 7fe543e00640 1 -- 172.21.15.69:0/1029029306 learned_addr learned my addr 172.21.15.69:0/1029029306 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542a00640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5440b3d30 0x7fe5440b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe543400640 1 --2- 172.21.15.69:0/1029029306 >> v2:172.21.15.121:3300/0 conn(0x7fe544162c70 0x7fe5440b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2609537806 0 0) 0x7fe5441669d0 con 0x7fe544071d70 2024-04-24T08:09:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe528003520 con 0x7fe544071d70 2024-04-24T08:09:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 539838578 0 0) 0x7fe528003520 con 0x7fe544071d70 2024-04-24T08:09:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5440b3d30 msgr2=0x7fe5440b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5440b3d30 0x7fe5440b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 >> v2:172.21.15.121:3300/0 conn(0x7fe544162c70 msgr2=0x7fe5440b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 --2- 172.21.15.69:0/1029029306 >> v2:172.21.15.121:3300/0 conn(0x7fe544162c70 0x7fe5440b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe544166f90 con 0x7fe544071d70 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe53c0019f0 con 0x7fe544071d70 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 >> v1:172.21.15.69:6789/0 conn(0x7fe544071d70 legacy=0x7fe5440721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5280039b0 0x7fe528003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe528004800 0x7fe528004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5280051c0 0x7fe52800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.251+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe544166cb0 con 0x7fe5280039b0 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.252+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe52800bf00 con 0x7fe5280051c0 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.252+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe52800c0b0 con 0x7fe528004800 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.252+0000 7fe542a00640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5440b3d30 0x7fe5440b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.252+0000 7fe543400640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5280039b0 0x7fe528003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.252+0000 7fe543e00640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5280051c0 0x7fe52800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.252+0000 7fe543400640 1 -- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe528004800 msgr2=0x7fe528004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:31.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.252+0000 7fe543400640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe528004800 0x7fe528004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.252+0000 7fe543400640 1 -- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5280051c0 msgr2=0x7fe52800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:31.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.252+0000 7fe543400640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5280051c0 0x7fe52800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.252+0000 7fe543400640 1 -- 172.21.15.69:0/1029029306 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe528003520 con 0x7fe5280039b0 2024-04-24T08:09:31.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.253+0000 7fe543400640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5280039b0 0x7fe528003dd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009ac0 tx=0x7fe528005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2fee1bacb69a71 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.253+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c036070 con 0x7fe5280039b0 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.253+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe52c031400 con 0x7fe5280039b0 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.253+0000 7fe542000640 1 -- 172.21.15.69:0/1029029306 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52c009690 con 0x7fe5280039b0 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.254+0000 7fe54a800640 1 -- 172.21.15.69:0/1029029306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5280039b0 msgr2=0x7fe528003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.254+0000 7fe54a800640 1 --2- 172.21.15.69:0/1029029306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5280039b0 0x7fe528003dd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009ac0 tx=0x7fe528005830 comp rx=0 tx=0).stop 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.254+0000 7fe543e00640 1 -- 172.21.15.69:0/1029029306 reap_dead start 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.254+0000 7fe54a800640 1 -- 172.21.15.69:0/1029029306 shutdown_connections 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.254+0000 7fe54a800640 1 -- 172.21.15.69:0/1029029306 >> 172.21.15.69:0/1029029306 conn(0x7fe544067d90 msgr2=0x7fe54406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.254+0000 7fe54a800640 1 -- 172.21.15.69:0/1029029306 shutdown_connections 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.254+0000 7fe54a800640 1 -- 172.21.15.69:0/1029029306 wait complete. 2024-04-24T08:09:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.256+0000 7fe54a800640 1 Processor -- start 2024-04-24T08:09:31.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.256+0000 7fe54a800640 1 -- start start 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.257+0000 7fe54a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5280039b0 0x7fe54415f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.257+0000 7fe54a800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe544158550 0x7fe54415bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.257+0000 7fe54a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54415c0f0 0x7fe5441f8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.257+0000 7fe54a800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe544163050 con 0x7fe54415c0f0 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.257+0000 7fe54a800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe544167550 con 0x7fe544158550 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.257+0000 7fe54a800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe5440cfc40 con 0x7fe5280039b0 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.257+0000 7fe543e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54415c0f0 0x7fe5441f8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.257+0000 7fe543e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54415c0f0 0x7fe5441f8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58610/0 (socket says 172.21.15.69:58610) 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.257+0000 7fe543e00640 1 -- 172.21.15.69:0/1567205507 learned_addr learned my addr 172.21.15.69:0/1567205507 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.258+0000 7fe543400640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5280039b0 0x7fe54415f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.258+0000 7fe542a00640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe544158550 0x7fe54415bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.258+0000 7fe543e00640 1 -- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5280039b0 msgr2=0x7fe54415f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.258+0000 7fe543e00640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5280039b0 0x7fe54415f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.258+0000 7fe543e00640 1 -- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe544158550 msgr2=0x7fe54415bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.258+0000 7fe543e00640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe544158550 0x7fe54415bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.258+0000 7fe543e00640 1 -- 172.21.15.69:0/1567205507 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54415e1f0 con 0x7fe54415c0f0 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.258+0000 7fe543e00640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54415c0f0 0x7fe5441f8b70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe5440b30b0 tx=0x7fe53c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.259+0000 7fe53b400640 1 -- 172.21.15.69:0/1567205507 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c00bb80 con 0x7fe54415c0f0 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.259+0000 7fe53b400640 1 -- 172.21.15.69:0/1567205507 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe53c00bd20 con 0x7fe54415c0f0 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.259+0000 7fe53b400640 1 -- 172.21.15.69:0/1567205507 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53c0072b0 con 0x7fe54415c0f0 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.259+0000 7fe54a800640 1 -- 172.21.15.69:0/1567205507 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe54415e4d0 con 0x7fe54415c0f0 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.259+0000 7fe54a800640 1 -- 172.21.15.69:0/1567205507 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe54415ea90 con 0x7fe54415c0f0 2024-04-24T08:09:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.260+0000 7fe54a800640 1 -- 172.21.15.69:0/1567205507 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe544068a80 con 0x7fe54415c0f0 2024-04-24T08:09:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.261+0000 7fe53b400640 1 -- 172.21.15.69:0/1567205507 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe53c01b020 con 0x7fe54415c0f0 2024-04-24T08:09:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.262+0000 7fe53b400640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe52006c920 0x7fe52006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.262+0000 7fe53b400640 1 -- 172.21.15.69:0/1567205507 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fe53c08bae0 con 0x7fe54415c0f0 2024-04-24T08:09:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.263+0000 7fe543400640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe52006c920 0x7fe52006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.264+0000 7fe543400640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe52006c920 0x7fe52006ede0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009ac0 tx=0x7fe52c008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.269+0000 7fe53b400640 1 -- 172.21.15.69:0/1567205507 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe53c05a1c0 con 0x7fe54415c0f0 2024-04-24T08:09:31.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.526+0000 7fe54a800640 1 -- 172.21.15.69:0/1567205507 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fe5440ce280 con 0x7fe54415c0f0 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.528+0000 7fe53b400640 1 -- 172.21.15.69:0/1567205507 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v3) ==== 80+0+6645 (secure 0 0 0) 0x7fe53c059ac0 con 0x7fe54415c0f0 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_dump": [ 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.a", 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmsqMeLRAApLPtHjtq78HzlVcsjfqXOg==", 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.b", 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmkyd5ChAAGsJGU6GGMvvGzQnABuDx/g==", 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.c", 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmS+yLORAAHPCeBtq99IXEVTpcMT3c4A==", 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.d", 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShm0xmHFRAA3VfYoqbcV8kWF+4fCZMVLA==", 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.0", 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC5vShmMbvcERAANgogiCC/sXf1y0Ykv7ecNQ==", 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.1", 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC8vShmDD+uLBAAJHXmxEm1cumJ3wjqe6S0ww==", 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.2", 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC/vShmS63kNhAAXec5HVuwo8THKstXiQOL8Q==", 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.532 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.3", 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDDvShmxcDJBhAA3V3I4C63EHPRUyhzBCZEzw==", 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.4", 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDJvShm1pX9ERAAL2gjv/+M915Cf5OoqBhZVw==", 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.5", 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDMvShmNIxbHRAA23gxiaGvagddVc2mE5DJJw==", 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.534 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.6", 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDPvShmb+x4NhAA6aWmAzU6c0AZomVgWcFwig==", 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.7", 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDTvShmgcKMBRAA6fkJvNxBjlNhK9CrZ/FMSw==", 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:09:31.535 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.0", 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw==", 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.536 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.1", 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmKAIHMhAAuTuFmTHvBteErsG23LMEyQ==", 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.2", 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmW0MnKRAASzxCAKuApVvfu9kUfK9BJw==", 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:09:31.537 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.3", 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC2vShmfgoaABAAniS9oubv92R7Qe2EITGKxg==", 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.admin", 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQCzvShmLh6yLxAAoH7NhoEAUKcdXp6n+ai2Ug==", 2024-04-24T08:09:31.538 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.bootstrap-mds", 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDbvShmlfh9GhAAF71JbqQrSVmiZdIRwxBCbQ==", 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile bootstrap-mds" 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.bootstrap-mgr", 2024-04-24T08:09:31.539 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDbvShmYAJ+GhAAOoMvfHLWqwEeJw1wwPnVbA==", 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.bootstrap-osd", 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDbvShmDwx+GhAAMys6neq8RDEh9tEJV5oyTw==", 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile bootstrap-osd" 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.bootstrap-rbd", 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDbvShmyRR+GhAAh9XxTASmeugZYQpJpCVnTg==", 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.540 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDbvShmhB1+GhAAyoW9VVQrVpndfTZJtiIPxg==", 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.bootstrap-rgw", 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDbvShmYiZ+GhAAhSql/GSRjpKvlH/pNKOrQA==", 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.541 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.x", 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmv4rwIxAA3rp+yy09qrs7HXip8bZ8xA==", 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.y", 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShm2eatGxAAFQkL0GBsujypv4VnmSy5bw==", 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:09:31.542 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:09:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:09:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:31.543 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-24T08:09:31.543 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:09:31.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.530+0000 7fe54a800640 1 -- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe52006c920 msgr2=0x7fe52006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.530+0000 7fe54a800640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe52006c920 0x7fe52006ede0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe52c009ac0 tx=0x7fe52c008030 comp rx=0 tx=0).stop 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.530+0000 7fe54a800640 1 -- 172.21.15.69:0/1567205507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54415c0f0 msgr2=0x7fe5441f8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.530+0000 7fe54a800640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54415c0f0 0x7fe5441f8b70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe5440b30b0 tx=0x7fe53c00b4a0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.530+0000 7fe54a800640 1 -- 172.21.15.69:0/1567205507 shutdown_connections 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.530+0000 7fe54a800640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe52006c920 0x7fe52006ede0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.530+0000 7fe54a800640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe54415c0f0 0x7fe5441f8b70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.530+0000 7fe54a800640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe544158550 0x7fe54415bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.530+0000 7fe54a800640 1 --2- 172.21.15.69:0/1567205507 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5280039b0 0x7fe54415f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.530+0000 7fe54a800640 1 -- 172.21.15.69:0/1567205507 >> 172.21.15.69:0/1567205507 conn(0x7fe544067d90 msgr2=0x7fe544164040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.531+0000 7fe54a800640 1 -- 172.21.15.69:0/1567205507 shutdown_connections 2024-04-24T08:09:31.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.531+0000 7fe54a800640 1 -- 172.21.15.69:0/1567205507 wait complete. 2024-04-24T08:09:31.547 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-24T08:09:31.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.739+0000 7f0bbbe00640 1 Processor -- start 2024-04-24T08:09:31.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.740+0000 7f0bbbe00640 1 -- start start 2024-04-24T08:09:31.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.741+0000 7f0bbbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0bbc06eb30 0x7f0bbc144090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.741+0000 7f0bbbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bbc071be0 0x7f0bbc0b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.741+0000 7f0bbbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bbc0b3190 con 0x7f0bbc15ab10 2024-04-24T08:09:31.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.741+0000 7f0bbbe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0bbc0ad590 con 0x7f0bbc06eb30 2024-04-24T08:09:31.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.741+0000 7f0bbbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0bbc0b3400 con 0x7f0bbc071be0 2024-04-24T08:09:31.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.741+0000 7f0bbaa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0bbc15ab10 0x7f0bbc06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32822/0 (socket says 172.21.15.69:32822) 2024-04-24T08:09:31.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.741+0000 7f0bbaa00640 1 -- 172.21.15.69:0/2231782351 learned_addr learned my addr 172.21.15.69:0/2231782351 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:31.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.741+0000 7f0bb2000640 1 --2- 172.21.15.69:0/2231782351 >> v2:172.21.15.69:3300/0 conn(0x7f0bbc06eb30 0x7f0bbc144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.742+0000 7f0bbb400640 1 --2- 172.21.15.69:0/2231782351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bbc071be0 0x7f0bbc0b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.742+0000 7f0bba000640 1 -- 172.21.15.69:0/2231782351 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1357675203 0 0) 0x7f0bbc0b3190 con 0x7f0bbc15ab10 2024-04-24T08:09:31.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.742+0000 7f0bb2000640 1 -- 172.21.15.69:0/2231782351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bbc071be0 msgr2=0x7f0bbc0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:31.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.742+0000 7f0bb2000640 1 --2- 172.21.15.69:0/2231782351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bbc071be0 0x7f0bbc0b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.742+0000 7f0bb2000640 1 -- 172.21.15.69:0/2231782351 >> v1:172.21.15.69:6789/0 conn(0x7f0bbc15ab10 legacy=0x7f0bbc06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.742+0000 7f0bb2000640 1 -- 172.21.15.69:0/2231782351 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bbc0b36e0 con 0x7f0bbc06eb30 2024-04-24T08:09:31.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.743+0000 7f0bb2000640 1 --2- 172.21.15.69:0/2231782351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc06eb30 0x7f0bbc144090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40097a0 tx=0x7f0ba402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbb12c712b046ca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.743+0000 7f0bba000640 1 -- 172.21.15.69:0/2231782351 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4037070 con 0x7f0bbc06eb30 2024-04-24T08:09:31.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.743+0000 7f0bba000640 1 -- 172.21.15.69:0/2231782351 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba4032440 con 0x7f0bbc06eb30 2024-04-24T08:09:31.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.743+0000 7f0bba000640 1 -- 172.21.15.69:0/2231782351 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4032810 con 0x7f0bbc06eb30 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.744+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2231782351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc06eb30 msgr2=0x7f0bbc144090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.744+0000 7f0bbbe00640 1 --2- 172.21.15.69:0/2231782351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc06eb30 0x7f0bbc144090 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40097a0 tx=0x7f0ba402a180 comp rx=0 tx=0).stop 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.744+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2231782351 shutdown_connections 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.744+0000 7f0bbbe00640 1 --2- 172.21.15.69:0/2231782351 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bbc071be0 0x7f0bbc0b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.744+0000 7f0bbbe00640 1 --2- 172.21.15.69:0/2231782351 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc06eb30 0x7f0bbc144090 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.744+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2231782351 >> 172.21.15.69:0/2231782351 conn(0x7f0bbc0d0160 msgr2=0x7f0bbc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.744+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2231782351 shutdown_connections 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.744+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2231782351 wait complete. 2024-04-24T08:09:31.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.746+0000 7f0bbbe00640 1 Processor -- start 2024-04-24T08:09:31.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.746+0000 7f0bbbe00640 1 -- start start 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bbbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bbc06eb30 0x7f0bbc1f8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bbbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc071be0 0x7f0bbc15d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bbbe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bbc15ab10 0x7f0bbc14d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bbbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc15aef0 con 0x7f0bbc071be0 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bbbe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0bbc072150 con 0x7f0bbc15ab10 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bbbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0bbc06f950 con 0x7f0bbc06eb30 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bb2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc071be0 0x7f0bbc15d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bb2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc071be0 0x7f0bbc15d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58618/0 (socket says 172.21.15.69:58618) 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bb2000640 1 -- 172.21.15.69:0/2329773220 learned_addr learned my addr 172.21.15.69:0/2329773220 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bb2000640 1 -- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bbc06eb30 msgr2=0x7f0bbc1f8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bbaa00640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bbc06eb30 0x7f0bbc1f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bbb400640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bbc15ab10 0x7f0bbc14d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bb2000640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bbc06eb30 0x7f0bbc1f8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bb2000640 1 -- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bbc15ab10 msgr2=0x7f0bbc14d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bb2000640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bbc15ab10 0x7f0bbc14d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.747+0000 7f0bb2000640 1 -- 172.21.15.69:0/2329773220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bbc1f8b80 con 0x7f0bbc071be0 2024-04-24T08:09:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.748+0000 7f0bb2000640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc071be0 0x7f0bbc15d4b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4035040 tx=0x7f0ba40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.748+0000 7f0bb3400640 1 -- 172.21.15.69:0/2329773220 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4037040 con 0x7f0bbc071be0 2024-04-24T08:09:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.748+0000 7f0bb3400640 1 -- 172.21.15.69:0/2329773220 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba4030770 con 0x7f0bbc071be0 2024-04-24T08:09:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.748+0000 7f0bb3400640 1 -- 172.21.15.69:0/2329773220 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4030b80 con 0x7f0bbc071be0 2024-04-24T08:09:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.748+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2329773220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bbc14a360 con 0x7f0bbc071be0 2024-04-24T08:09:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.748+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2329773220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bbc14a920 con 0x7f0bbc071be0 2024-04-24T08:09:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.749+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2329773220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bbc075d50 con 0x7f0bbc071be0 2024-04-24T08:09:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.750+0000 7f0bb3400640 1 -- 172.21.15.69:0/2329773220 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ba4044020 con 0x7f0bbc071be0 2024-04-24T08:09:31.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.751+0000 7f0bb3400640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b9806c8f0 0x7f0b9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.752+0000 7f0bb3400640 1 -- 172.21.15.69:0/2329773220 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f0ba40abd50 con 0x7f0bbc071be0 2024-04-24T08:09:31.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.752+0000 7f0bbaa00640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b9806c8f0 0x7f0b9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.753+0000 7f0bbaa00640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b9806c8f0 0x7f0b9806edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0bac000f60 tx=0x7f0bac009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:31.756+0000 7f0bb3400640 1 -- 172.21.15.69:0/2329773220 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0bbc1f8b80 con 0x7f0bbc071be0 2024-04-24T08:09:32.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.007+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2329773220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f0bbc06f020 con 0x7f0bbc071be0 2024-04-24T08:09:32.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.012+0000 7f0bb3400640 1 -- 172.21.15.69:0/2329773220 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v4) ==== 90+0+0 (secure 0 0 0) 0x7f0ba4032e20 con 0x7f0bbc071be0 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.014+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b9806c8f0 msgr2=0x7f0b9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.014+0000 7f0bbbe00640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b9806c8f0 0x7f0b9806edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0bac000f60 tx=0x7f0bac009040 comp rx=0 tx=0).stop 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.014+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2329773220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc071be0 msgr2=0x7f0bbc15d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.014+0000 7f0bbbe00640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc071be0 0x7f0bbc15d4b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4035040 tx=0x7f0ba40046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.014+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2329773220 shutdown_connections 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.014+0000 7f0bbbe00640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b9806c8f0 0x7f0b9806edb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.014+0000 7f0bbbe00640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bbc071be0 0x7f0bbc15d4b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.014+0000 7f0bbbe00640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bbc15ab10 0x7f0bbc14d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.014+0000 7f0bbbe00640 1 --2- 172.21.15.69:0/2329773220 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bbc06eb30 0x7f0bbc1f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.014+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2329773220 >> 172.21.15.69:0/2329773220 conn(0x7f0bbc0d0160 msgr2=0x7f0bbc158470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.015+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2329773220 shutdown_connections 2024-04-24T08:09:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.015+0000 7f0bbbe00640 1 -- 172.21.15.69:0/2329773220 wait complete. 2024-04-24T08:09:32.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-24T08:09:32.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.220+0000 7f6cd5400640 1 Processor -- start 2024-04-24T08:09:32.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.220+0000 7f6cd5400640 1 -- start start 2024-04-24T08:09:32.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6cd5400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6cd00d0300 0x7f6cd006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6cd5400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6cd0145fe0 0x7f6cd0160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6cd5400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cd00cf3d0 con 0x7f6cd0145a90 2024-04-24T08:09:32.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6cd5400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6cd00cf6b0 con 0x7f6cd00d0300 2024-04-24T08:09:32.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6cd5400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6cd00cf990 con 0x7f6cd0145fe0 2024-04-24T08:09:32.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6cce800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6cd00d0300 0x7f6cd006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6cce800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6cd00d0300 0x7f6cd006e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58622/0 (socket says 172.21.15.69:58622) 2024-04-24T08:09:32.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6cce800640 1 -- 172.21.15.69:0/2293453163 learned_addr learned my addr 172.21.15.69:0/2293453163 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:32.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6cce800640 1 -- 172.21.15.69:0/2293453163 >> v2:172.21.15.121:3300/0 conn(0x7f6cd0145fe0 msgr2=0x7f6cd0160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:32.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6ccf200640 1 --2- 172.21.15.69:0/2293453163 >> v2:172.21.15.121:3300/0 conn(0x7f6cd0145fe0 0x7f6cd0160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.222+0000 7f6cce800640 1 --2- 172.21.15.69:0/2293453163 >> v2:172.21.15.121:3300/0 conn(0x7f6cd0145fe0 0x7f6cd0160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.223+0000 7f6cce800640 1 -- 172.21.15.69:0/2293453163 >> v1:172.21.15.121:6789/0 conn(0x7f6cd0145a90 legacy=0x7f6cd00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:32.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.223+0000 7f6cce800640 1 -- 172.21.15.69:0/2293453163 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd00cfc70 con 0x7f6cd00d0300 2024-04-24T08:09:32.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.224+0000 7f6cce800640 1 --2- 172.21.15.69:0/2293453163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd00d0300 0x7f6cd006e240 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00d1050 tx=0x7f6cc002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797cc8a3f8bfb044 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:32.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.224+0000 7f6ccd400640 1 -- 172.21.15.69:0/2293453163 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0037070 con 0x7f6cd00d0300 2024-04-24T08:09:32.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.224+0000 7f6ccd400640 1 -- 172.21.15.69:0/2293453163 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc0032440 con 0x7f6cd00d0300 2024-04-24T08:09:32.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.224+0000 7f6ccd400640 1 -- 172.21.15.69:0/2293453163 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0032810 con 0x7f6cd00d0300 2024-04-24T08:09:32.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.224+0000 7f6cd5400640 1 -- 172.21.15.69:0/2293453163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd00d0300 msgr2=0x7f6cd006e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.224+0000 7f6cd5400640 1 --2- 172.21.15.69:0/2293453163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd00d0300 0x7f6cd006e240 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00d1050 tx=0x7f6cc002a230 comp rx=0 tx=0).stop 2024-04-24T08:09:32.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.225+0000 7f6cd5400640 1 -- 172.21.15.69:0/2293453163 shutdown_connections 2024-04-24T08:09:32.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.225+0000 7f6cd5400640 1 --2- 172.21.15.69:0/2293453163 >> v2:172.21.15.121:3300/0 conn(0x7f6cd0145fe0 0x7f6cd0160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.225+0000 7f6cd5400640 1 --2- 172.21.15.69:0/2293453163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd00d0300 0x7f6cd006e240 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.225+0000 7f6cd5400640 1 -- 172.21.15.69:0/2293453163 >> 172.21.15.69:0/2293453163 conn(0x7f6cd0067d90 msgr2=0x7f6cd006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:32.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.225+0000 7f6cd5400640 1 -- 172.21.15.69:0/2293453163 shutdown_connections 2024-04-24T08:09:32.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.225+0000 7f6cd5400640 1 -- 172.21.15.69:0/2293453163 wait complete. 2024-04-24T08:09:32.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.226+0000 7f6cd5400640 1 Processor -- start 2024-04-24T08:09:32.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.226+0000 7f6cd5400640 1 -- start start 2024-04-24T08:09:32.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.227+0000 7f6cd5400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd00d0300 0x7f6cd01e6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.227+0000 7f6cd5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd0145a90 0x7f6cd00ac6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.227+0000 7f6cd5400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cd0145fe0 0x7f6cd0130a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.227+0000 7f6cd5400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd01458f0 con 0x7f6cd0145a90 2024-04-24T08:09:32.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.227+0000 7f6cd5400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0145770 con 0x7f6cd00d0300 2024-04-24T08:09:32.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.227+0000 7f6cd5400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd00cdc30 con 0x7f6cd0145fe0 2024-04-24T08:09:32.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.227+0000 7f6ccde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd0145a90 0x7f6cd00ac6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6ccde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd0145a90 0x7f6cd00ac6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58626/0 (socket says 172.21.15.69:58626) 2024-04-24T08:09:32.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6ccde00640 1 -- 172.21.15.69:0/1886770128 learned_addr learned my addr 172.21.15.69:0/1886770128 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:32.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6cce800640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd00d0300 0x7f6cd01e6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6ccde00640 1 -- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cd0145fe0 msgr2=0x7f6cd0130a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6ccde00640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cd0145fe0 0x7f6cd0130a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6ccde00640 1 -- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd00d0300 msgr2=0x7f6cd01e6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6ccde00640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd00d0300 0x7f6cd01e6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6ccde00640 1 -- 172.21.15.69:0/1886770128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd01e7440 con 0x7f6cd0145a90 2024-04-24T08:09:32.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6ccde00640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd0145a90 0x7f6cd00ac6b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00b3e0 tx=0x7f6cbc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:32.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6cc6a00640 1 -- 172.21.15.69:0/1886770128 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc015020 con 0x7f6cd0145a90 2024-04-24T08:09:32.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6cc6a00640 1 -- 172.21.15.69:0/1886770128 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cbc011850 con 0x7f6cd0145a90 2024-04-24T08:09:32.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6cc6a00640 1 -- 172.21.15.69:0/1886770128 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc011c40 con 0x7f6cd0145a90 2024-04-24T08:09:32.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.228+0000 7f6cd5400640 1 -- 172.21.15.69:0/1886770128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd01e7720 con 0x7f6cd0145a90 2024-04-24T08:09:32.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.229+0000 7f6cd5400640 1 -- 172.21.15.69:0/1886770128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd01e7ce0 con 0x7f6cd0145a90 2024-04-24T08:09:32.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.230+0000 7f6cd5400640 1 -- 172.21.15.69:0/1886770128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd00afcb0 con 0x7f6cd0145a90 2024-04-24T08:09:32.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.231+0000 7f6cc6a00640 1 -- 172.21.15.69:0/1886770128 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6cbc018020 con 0x7f6cd0145a90 2024-04-24T08:09:32.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.231+0000 7f6cc6a00640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6cb006c8f0 0x7f6cb006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.231+0000 7f6cc6a00640 1 -- 172.21.15.69:0/1886770128 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f6cbc08b860 con 0x7f6cd0145a90 2024-04-24T08:09:32.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.232+0000 7f6cce800640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6cb006c8f0 0x7f6cb006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.233+0000 7f6cce800640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6cb006c8f0 0x7f6cb006edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0034000 tx=0x7f6cc002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:32.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.236+0000 7f6cc6a00640 1 -- 172.21.15.69:0/1886770128 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6cd00afcb0 con 0x7f6cd0145a90 2024-04-24T08:09:32.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.494+0000 7f6cd5400640 1 -- 172.21.15.69:0/1886770128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f6cd0068a80 con 0x7f6cd0145a90 2024-04-24T08:09:32.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.498+0000 7f6cc6a00640 1 -- 172.21.15.69:0/1886770128 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f6cbc01f020 con 0x7f6cd0145a90 2024-04-24T08:09:32.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 -- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6cb006c8f0 msgr2=0x7f6cb006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6cb006c8f0 0x7f6cb006edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0034000 tx=0x7f6cc002e000 comp rx=0 tx=0).stop 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 -- 172.21.15.69:0/1886770128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd0145a90 msgr2=0x7f6cd00ac6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd0145a90 0x7f6cd00ac6b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00b3e0 tx=0x7f6cbc00b8a0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 -- 172.21.15.69:0/1886770128 shutdown_connections 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6cb006c8f0 0x7f6cb006edb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6cd0145a90 0x7f6cd00ac6b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6cd0145fe0 0x7f6cd0130a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 --2- 172.21.15.69:0/1886770128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6cd00d0300 0x7f6cd01e6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 -- 172.21.15.69:0/1886770128 >> 172.21.15.69:0/1886770128 conn(0x7f6cd0067d90 msgr2=0x7f6cd006f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 -- 172.21.15.69:0/1886770128 shutdown_connections 2024-04-24T08:09:32.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.501+0000 7f6cd5400640 1 -- 172.21.15.69:0/1886770128 wait complete. 2024-04-24T08:09:32.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-24T08:09:32.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.706+0000 7f7891000640 1 Processor -- start 2024-04-24T08:09:32.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.707+0000 7f7891000640 1 -- start start 2024-04-24T08:09:32.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.708+0000 7f7891000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f788c15e960 0x7f788c162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.708+0000 7f7891000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f788c162980 0x7f788c06cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.708+0000 7f7891000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f788c06d4f0 0x7f788c0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.708+0000 7f7891000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f788c15ed40 con 0x7f788c06d4f0 2024-04-24T08:09:32.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.708+0000 7f7891000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f788c0cf050 con 0x7f788c162980 2024-04-24T08:09:32.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.708+0000 7f7891000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f788c162e20 con 0x7f788c15e960 2024-04-24T08:09:32.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.708+0000 7f788be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f788c06d4f0 0x7f788c0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.709+0000 7f788be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f788c06d4f0 0x7f788c0cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58636/0 (socket says 172.21.15.69:58636) 2024-04-24T08:09:32.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.709+0000 7f788be00640 1 -- 172.21.15.69:0/1696299245 learned_addr learned my addr 172.21.15.69:0/1696299245 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:32.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.709+0000 7f788be00640 1 -- 172.21.15.69:0/1696299245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f788c15e960 msgr2=0x7f788c162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:32.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.709+0000 7f788be00640 1 --2- 172.21.15.69:0/1696299245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f788c15e960 0x7f788c162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.709+0000 7f788be00640 1 -- 172.21.15.69:0/1696299245 >> v2:172.21.15.121:3300/0 conn(0x7f788c162980 msgr2=0x7f788c06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:32.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.709+0000 7f788be00640 1 --2- 172.21.15.69:0/1696299245 >> v2:172.21.15.121:3300/0 conn(0x7f788c162980 0x7f788c06cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.709+0000 7f788be00640 1 -- 172.21.15.69:0/1696299245 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788c163100 con 0x7f788c06d4f0 2024-04-24T08:09:32.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.710+0000 7f788be00640 1 --2- 172.21.15.69:0/1696299245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 0x7f788c0cfaf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f788c09e030 tx=0x7f788402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75e7b645a76e1d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:32.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.710+0000 7f788a000640 1 -- 172.21.15.69:0/1696299245 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7884037070 con 0x7f788c06d4f0 2024-04-24T08:09:32.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.710+0000 7f788a000640 1 -- 172.21.15.69:0/1696299245 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7884032440 con 0x7f788c06d4f0 2024-04-24T08:09:32.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.710+0000 7f788a000640 1 -- 172.21.15.69:0/1696299245 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7884032810 con 0x7f788c06d4f0 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.711+0000 7f7891000640 1 -- 172.21.15.69:0/1696299245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 msgr2=0x7f788c0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.711+0000 7f7891000640 1 --2- 172.21.15.69:0/1696299245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 0x7f788c0cfaf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f788c09e030 tx=0x7f788402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.711+0000 7f7891000640 1 -- 172.21.15.69:0/1696299245 shutdown_connections 2024-04-24T08:09:32.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.711+0000 7f7891000640 1 --2- 172.21.15.69:0/1696299245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 0x7f788c0cfaf0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.711+0000 7f7891000640 1 --2- 172.21.15.69:0/1696299245 >> v2:172.21.15.121:3300/0 conn(0x7f788c162980 0x7f788c06cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.711+0000 7f7891000640 1 --2- 172.21.15.69:0/1696299245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f788c15e960 0x7f788c162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.711+0000 7f7891000640 1 -- 172.21.15.69:0/1696299245 >> 172.21.15.69:0/1696299245 conn(0x7f788c09d900 msgr2=0x7f788c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:32.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.711+0000 7f7891000640 1 -- 172.21.15.69:0/1696299245 shutdown_connections 2024-04-24T08:09:32.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.711+0000 7f7891000640 1 -- 172.21.15.69:0/1696299245 wait complete. 2024-04-24T08:09:32.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.713+0000 7f7891000640 1 Processor -- start 2024-04-24T08:09:32.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.713+0000 7f7891000640 1 -- start start 2024-04-24T08:09:32.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.714+0000 7f7891000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 0x7f788c1619b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.714+0000 7f7891000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f788c15e960 0x7f788c0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.714+0000 7f7891000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f788c162980 0x7f788c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.714+0000 7f7891000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f788c0b6170 con 0x7f788c06d4f0 2024-04-24T08:09:32.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.714+0000 7f7891000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f788c0cf5d0 con 0x7f788c15e960 2024-04-24T08:09:32.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.714+0000 7f7891000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f788c0b62f0 con 0x7f788c162980 2024-04-24T08:09:32.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.714+0000 7f788b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 0x7f788c1619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.714+0000 7f788b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 0x7f788c1619b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58652/0 (socket says 172.21.15.69:58652) 2024-04-24T08:09:32.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.714+0000 7f788b400640 1 -- 172.21.15.69:0/575910776 learned_addr learned my addr 172.21.15.69:0/575910776 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:32.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f788b400640 1 -- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f788c162980 msgr2=0x7f788c0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f788b400640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f788c162980 0x7f788c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f788b400640 1 -- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f788c15e960 msgr2=0x7f788c0cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f788b400640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f788c15e960 0x7f788c0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f788b400640 1 -- 172.21.15.69:0/575910776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788c1e83d0 con 0x7f788c06d4f0 2024-04-24T08:09:32.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f788b400640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 0x7f788c1619b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f787c00b4d0 tx=0x7f787c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:32.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f7883400640 1 -- 172.21.15.69:0/575910776 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c014070 con 0x7f788c06d4f0 2024-04-24T08:09:32.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f7883400640 1 -- 172.21.15.69:0/575910776 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f787c010870 con 0x7f788c06d4f0 2024-04-24T08:09:32.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f7883400640 1 -- 172.21.15.69:0/575910776 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c010c60 con 0x7f788c06d4f0 2024-04-24T08:09:32.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f7891000640 1 -- 172.21.15.69:0/575910776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f788c0cdd10 con 0x7f788c06d4f0 2024-04-24T08:09:32.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.715+0000 7f7891000640 1 -- 172.21.15.69:0/575910776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f788c148c00 con 0x7f788c06d4f0 2024-04-24T08:09:32.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.716+0000 7f7891000640 1 -- 172.21.15.69:0/575910776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f788c075d50 con 0x7f788c06d4f0 2024-04-24T08:09:32.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.717+0000 7f7883400640 1 -- 172.21.15.69:0/575910776 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f787c017070 con 0x7f788c06d4f0 2024-04-24T08:09:32.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.718+0000 7f7883400640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f786806c8f0 0x7f786806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.718+0000 7f7883400640 1 -- 172.21.15.69:0/575910776 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f787c08b660 con 0x7f788c06d4f0 2024-04-24T08:09:32.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.719+0000 7f788aa00640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f786806c8f0 0x7f786806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.720+0000 7f788aa00640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f786806c8f0 0x7f786806edb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f788c148ad0 tx=0x7f7874009250 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:32.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.722+0000 7f7883400640 1 -- 172.21.15.69:0/575910776 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f787c059d40 con 0x7f788c06d4f0 2024-04-24T08:09:32.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.975+0000 7f7891000640 1 -- 172.21.15.69:0/575910776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f788c0d1dc0 con 0x7f788c06d4f0 2024-04-24T08:09:32.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.980+0000 7f7883400640 1 -- 172.21.15.69:0/575910776 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f787c059640 con 0x7f788c06d4f0 2024-04-24T08:09:32.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 -- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f786806c8f0 msgr2=0x7f786806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f786806c8f0 0x7f786806edb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f788c148ad0 tx=0x7f7874009250 comp rx=0 tx=0).stop 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 -- 172.21.15.69:0/575910776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 msgr2=0x7f788c1619b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 0x7f788c1619b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f787c00b4d0 tx=0x7f787c00b990 comp rx=0 tx=0).stop 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 -- 172.21.15.69:0/575910776 shutdown_connections 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f786806c8f0 0x7f786806edb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f788c162980 0x7f788c0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f788c15e960 0x7f788c0cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 --2- 172.21.15.69:0/575910776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f788c06d4f0 0x7f788c1619b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 -- 172.21.15.69:0/575910776 >> 172.21.15.69:0/575910776 conn(0x7f788c09d900 msgr2=0x7f788c0d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 -- 172.21.15.69:0/575910776 shutdown_connections 2024-04-24T08:09:32.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:32.982+0000 7f7891000640 1 -- 172.21.15.69:0/575910776 wait complete. 2024-04-24T08:09:33.001 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-24T08:09:33.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.191+0000 7f6adde00640 1 Processor -- start 2024-04-24T08:09:33.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.191+0000 7f6adde00640 1 -- start start 2024-04-24T08:09:33.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.193+0000 7f6adde00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6ad815b490 0x7f6ad80b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.193+0000 7f6adde00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ad80d02f0 0x7f6ad806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.193+0000 7f6adde00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ad80b46e0 con 0x7f6ad806f2d0 2024-04-24T08:09:33.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.193+0000 7f6adde00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6ad806fc70 con 0x7f6ad815b490 2024-04-24T08:09:33.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.193+0000 7f6adde00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6ad80d0960 con 0x7f6ad80d02f0 2024-04-24T08:09:33.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.193+0000 7f6ad6a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6ad815b490 0x7f6ad80b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.194+0000 7f6ad7400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ad80d02f0 0x7f6ad806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.194+0000 7f6ad6a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6ad815b490 0x7f6ad80b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:34140/0 (socket says 172.21.15.69:34140) 2024-04-24T08:09:33.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.194+0000 7f6ad6a00640 1 -- 172.21.15.69:0/210911456 learned_addr learned my addr 172.21.15.69:0/210911456 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:33.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.194+0000 7f6ad6a00640 1 -- 172.21.15.69:0/210911456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ad80d02f0 msgr2=0x7f6ad806f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:33.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.194+0000 7f6ad6a00640 1 --2- 172.21.15.69:0/210911456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ad80d02f0 0x7f6ad806f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.194+0000 7f6ad6a00640 1 -- 172.21.15.69:0/210911456 >> v1:172.21.15.121:6789/0 conn(0x7f6ad806f2d0 legacy=0x7f6ad80cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.194+0000 7f6ad6a00640 1 -- 172.21.15.69:0/210911456 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad80b4980 con 0x7f6ad815b490 2024-04-24T08:09:33.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.196+0000 7f6ad6a00640 1 --2- 172.21.15.69:0/210911456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ad815b490 0x7f6ad80b5180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009870 tx=0x7f6ac802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56c2f86fd97e7ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:33.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.196+0000 7f6ad5600640 1 -- 172.21.15.69:0/210911456 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8037070 con 0x7f6ad815b490 2024-04-24T08:09:33.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.196+0000 7f6ad5600640 1 -- 172.21.15.69:0/210911456 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac8032440 con 0x7f6ad815b490 2024-04-24T08:09:33.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.196+0000 7f6ad5600640 1 -- 172.21.15.69:0/210911456 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8032810 con 0x7f6ad815b490 2024-04-24T08:09:33.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.197+0000 7f6adde00640 1 -- 172.21.15.69:0/210911456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ad815b490 msgr2=0x7f6ad80b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.197+0000 7f6adde00640 1 --2- 172.21.15.69:0/210911456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ad815b490 0x7f6ad80b5180 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009870 tx=0x7f6ac802a270 comp rx=0 tx=0).stop 2024-04-24T08:09:33.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.197+0000 7f6adde00640 1 -- 172.21.15.69:0/210911456 shutdown_connections 2024-04-24T08:09:33.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.197+0000 7f6adde00640 1 --2- 172.21.15.69:0/210911456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ad80d02f0 0x7f6ad806f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.197+0000 7f6adde00640 1 --2- 172.21.15.69:0/210911456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ad815b490 0x7f6ad80b5180 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.197+0000 7f6adde00640 1 -- 172.21.15.69:0/210911456 >> 172.21.15.69:0/210911456 conn(0x7f6ad8067d90 msgr2=0x7f6ad806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:33.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.197+0000 7f6adde00640 1 -- 172.21.15.69:0/210911456 shutdown_connections 2024-04-24T08:09:33.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.197+0000 7f6adde00640 1 -- 172.21.15.69:0/210911456 wait complete. 2024-04-24T08:09:33.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.199+0000 7f6adde00640 1 Processor -- start 2024-04-24T08:09:33.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.199+0000 7f6adde00640 1 -- start start 2024-04-24T08:09:33.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6adde00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ad806f2d0 0x7f6ad80721f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6adde00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ad80d02f0 0x7f6ad80aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6adde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad815b490 0x7f6ad814bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6adde00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad815b870 con 0x7f6ad815b490 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6adde00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad80d0c40 con 0x7f6ad806f2d0 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6adde00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6ad80b56c0 con 0x7f6ad80d02f0 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6ad7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad815b490 0x7f6ad814bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6ad7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad815b490 0x7f6ad814bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58656/0 (socket says 172.21.15.69:58656) 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6ad7400640 1 -- 172.21.15.69:0/2478825198 learned_addr learned my addr 172.21.15.69:0/2478825198 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6ad6a00640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ad806f2d0 0x7f6ad80721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6ad6000640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ad80d02f0 0x7f6ad80aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6ad7400640 1 -- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ad80d02f0 msgr2=0x7f6ad80aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6ad7400640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ad80d02f0 0x7f6ad80aed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6ad7400640 1 -- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ad806f2d0 msgr2=0x7f6ad80721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6ad7400640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ad806f2d0 0x7f6ad80721f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.201+0000 7f6ad7400640 1 -- 172.21.15.69:0/2478825198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad814cac0 con 0x7f6ad815b490 2024-04-24T08:09:33.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.202+0000 7f6ad7400640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad815b490 0x7f6ad814bb70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6ad000efc0 tx=0x7f6ad000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:33.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.202+0000 7f6acf400640 1 -- 172.21.15.69:0/2478825198 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0018020 con 0x7f6ad815b490 2024-04-24T08:09:33.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.202+0000 7f6acf400640 1 -- 172.21.15.69:0/2478825198 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ad0014480 con 0x7f6ad815b490 2024-04-24T08:09:33.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.202+0000 7f6acf400640 1 -- 172.21.15.69:0/2478825198 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0014870 con 0x7f6ad815b490 2024-04-24T08:09:33.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.202+0000 7f6adde00640 1 -- 172.21.15.69:0/2478825198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad80af730 con 0x7f6ad815b490 2024-04-24T08:09:33.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.202+0000 7f6adde00640 1 -- 172.21.15.69:0/2478825198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad815bf20 con 0x7f6ad815b490 2024-04-24T08:09:33.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.203+0000 7f6adde00640 1 -- 172.21.15.69:0/2478825198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ad8068a80 con 0x7f6ad815b490 2024-04-24T08:09:33.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.204+0000 7f6acf400640 1 -- 172.21.15.69:0/2478825198 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ad0005d10 con 0x7f6ad815b490 2024-04-24T08:09:33.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.205+0000 7f6acf400640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ab806c8f0 0x7f6ab806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.205+0000 7f6acf400640 1 -- 172.21.15.69:0/2478825198 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f6ad008c750 con 0x7f6ad815b490 2024-04-24T08:09:33.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.206+0000 7f6ad6a00640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ab806c8f0 0x7f6ab806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.207+0000 7f6ad6a00640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ab806c8f0 0x7f6ab806edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8034000 tx=0x7f6ac802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:33.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.210+0000 7f6acf400640 1 -- 172.21.15.69:0/2478825198 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ad0022030 con 0x7f6ad815b490 2024-04-24T08:09:33.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.474+0000 7f6adde00640 1 -- 172.21.15.69:0/2478825198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f6ad80cec10 con 0x7f6ad815b490 2024-04-24T08:09:33.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.479+0000 7f6acf400640 1 -- 172.21.15.69:0/2478825198 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f6ad0010040 con 0x7f6ad815b490 2024-04-24T08:09:33.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 -- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ab806c8f0 msgr2=0x7f6ab806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ab806c8f0 0x7f6ab806edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8034000 tx=0x7f6ac802e000 comp rx=0 tx=0).stop 2024-04-24T08:09:33.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 -- 172.21.15.69:0/2478825198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad815b490 msgr2=0x7f6ad814bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad815b490 0x7f6ad814bb70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f6ad000efc0 tx=0x7f6ad000c490 comp rx=0 tx=0).stop 2024-04-24T08:09:33.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 -- 172.21.15.69:0/2478825198 shutdown_connections 2024-04-24T08:09:33.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ab806c8f0 0x7f6ab806edb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ad815b490 0x7f6ad814bb70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ad80d02f0 0x7f6ad80aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 --2- 172.21.15.69:0/2478825198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ad806f2d0 0x7f6ad80721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 -- 172.21.15.69:0/2478825198 >> 172.21.15.69:0/2478825198 conn(0x7f6ad8067d90 msgr2=0x7f6ad80cf130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:33.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 -- 172.21.15.69:0/2478825198 shutdown_connections 2024-04-24T08:09:33.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.482+0000 7f6adde00640 1 -- 172.21.15.69:0/2478825198 wait complete. 2024-04-24T08:09:33.499 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-24T08:09:33.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.691+0000 7fe3e5c00640 1 Processor -- start 2024-04-24T08:09:33.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.691+0000 7fe3e5c00640 1 -- start start 2024-04-24T08:09:33.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.693+0000 7fe3e5c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e014c650 0x7fe3e0071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.693+0000 7fe3e5c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe3e006db90 0x7fe3e00721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.693+0000 7fe3e5c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3e0163fb0 con 0x7fe3e01666e0 2024-04-24T08:09:33.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.693+0000 7fe3e5c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe3e006e130 con 0x7fe3e006db90 2024-04-24T08:09:33.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.693+0000 7fe3e5c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e006e410 con 0x7fe3e014c650 2024-04-24T08:09:33.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.694+0000 7fe3dfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e014c650 0x7fe3e0071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.694+0000 7fe3e5200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe3e006db90 0x7fe3e00721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.694+0000 7fe3dfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e014c650 0x7fe3e0071c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:48032/0 (socket says 172.21.15.69:48032) 2024-04-24T08:09:33.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.694+0000 7fe3dfe00640 1 -- 172.21.15.69:0/3730086796 learned_addr learned my addr 172.21.15.69:0/3730086796 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:33.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.695+0000 7fe3dfe00640 1 -- 172.21.15.69:0/3730086796 >> v2:172.21.15.121:3300/0 conn(0x7fe3e006db90 msgr2=0x7fe3e00721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:33.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.695+0000 7fe3dfe00640 1 --2- 172.21.15.69:0/3730086796 >> v2:172.21.15.121:3300/0 conn(0x7fe3e006db90 0x7fe3e00721d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.695+0000 7fe3dfe00640 1 -- 172.21.15.69:0/3730086796 >> v1:172.21.15.121:6789/0 conn(0x7fe3e01666e0 legacy=0x7fe3e00b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.695+0000 7fe3dfe00640 1 -- 172.21.15.69:0/3730086796 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e006e6f0 con 0x7fe3e014c650 2024-04-24T08:09:33.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.696+0000 7fe3dfe00640 1 --2- 172.21.15.69:0/3730086796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e014c650 0x7fe3e0071c90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009e50 tx=0x7fe3d002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ebe19d12a2da36 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:33.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.696+0000 7fe3df400640 1 -- 172.21.15.69:0/3730086796 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d0036070 con 0x7fe3e014c650 2024-04-24T08:09:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.696+0000 7fe3df400640 1 -- 172.21.15.69:0/3730086796 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3d0031600 con 0x7fe3e014c650 2024-04-24T08:09:33.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.696+0000 7fe3df400640 1 -- 172.21.15.69:0/3730086796 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3d00319d0 con 0x7fe3e014c650 2024-04-24T08:09:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.697+0000 7fe3e5c00640 1 -- 172.21.15.69:0/3730086796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e014c650 msgr2=0x7fe3e0071c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.697+0000 7fe3e5c00640 1 --2- 172.21.15.69:0/3730086796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e014c650 0x7fe3e0071c90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0009e50 tx=0x7fe3d002a680 comp rx=0 tx=0).stop 2024-04-24T08:09:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.697+0000 7fe3e5c00640 1 -- 172.21.15.69:0/3730086796 shutdown_connections 2024-04-24T08:09:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.697+0000 7fe3e5c00640 1 --2- 172.21.15.69:0/3730086796 >> v2:172.21.15.121:3300/0 conn(0x7fe3e006db90 0x7fe3e00721d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.697+0000 7fe3e5c00640 1 --2- 172.21.15.69:0/3730086796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e014c650 0x7fe3e0071c90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.697+0000 7fe3e5c00640 1 -- 172.21.15.69:0/3730086796 >> 172.21.15.69:0/3730086796 conn(0x7fe3e00d0160 msgr2=0x7fe3e00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.697+0000 7fe3e5c00640 1 -- 172.21.15.69:0/3730086796 shutdown_connections 2024-04-24T08:09:33.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.697+0000 7fe3e5c00640 1 -- 172.21.15.69:0/3730086796 wait complete. 2024-04-24T08:09:33.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.699+0000 7fe3e5c00640 1 Processor -- start 2024-04-24T08:09:33.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.699+0000 7fe3e5c00640 1 -- start start 2024-04-24T08:09:33.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.700+0000 7fe3e5c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3e006db90 0x7fe3e0154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.700+0000 7fe3e5c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3e014c650 0x7fe3e00cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.700+0000 7fe3e5c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e01666e0 0x7fe3e00ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.700+0000 7fe3e5c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e014ca30 con 0x7fe3e014c650 2024-04-24T08:09:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.700+0000 7fe3e5c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe3e006ecb0 con 0x7fe3e006db90 2024-04-24T08:09:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.700+0000 7fe3e5c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe3e0072830 con 0x7fe3e01666e0 2024-04-24T08:09:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.700+0000 7fe3d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3e014c650 0x7fe3e00cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.700+0000 7fe3d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3e014c650 0x7fe3e00cc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58668/0 (socket says 172.21.15.69:58668) 2024-04-24T08:09:33.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.700+0000 7fe3d7400640 1 -- 172.21.15.69:0/86335752 learned_addr learned my addr 172.21.15.69:0/86335752 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3e5200640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e01666e0 0x7fe3e00ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3dfe00640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3e006db90 0x7fe3e0154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3d7400640 1 -- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e01666e0 msgr2=0x7fe3e00ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3d7400640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e01666e0 0x7fe3e00ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3d7400640 1 -- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3e006db90 msgr2=0x7fe3e0154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3d7400640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3e006db90 0x7fe3e0154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3d7400640 1 -- 172.21.15.69:0/86335752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3e015ed40 con 0x7fe3e014c650 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3d7400640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3e014c650 0x7fe3e00cc8b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800c430 tx=0x7fe3c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3dcc00640 1 -- 172.21.15.69:0/86335752 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8017070 con 0x7fe3e014c650 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3dcc00640 1 -- 172.21.15.69:0/86335752 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c80052d0 con 0x7fe3e014c650 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3dcc00640 1 -- 172.21.15.69:0/86335752 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8003850 con 0x7fe3e014c650 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.701+0000 7fe3e5c00640 1 -- 172.21.15.69:0/86335752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3e0149e40 con 0x7fe3e014c650 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.702+0000 7fe3e5c00640 1 -- 172.21.15.69:0/86335752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3e014a0e0 con 0x7fe3e014c650 2024-04-24T08:09:33.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.703+0000 7fe3e5c00640 1 -- 172.21.15.69:0/86335752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3e0075d50 con 0x7fe3e014c650 2024-04-24T08:09:33.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.703+0000 7fe3dcc00640 1 -- 172.21.15.69:0/86335752 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe3c801a070 con 0x7fe3e014c650 2024-04-24T08:09:33.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.704+0000 7fe3dcc00640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3bc06c8f0 0x7fe3bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.704+0000 7fe3dcc00640 1 -- 172.21.15.69:0/86335752 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fe3c808ac30 con 0x7fe3e014c650 2024-04-24T08:09:33.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.705+0000 7fe3dfe00640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3bc06c8f0 0x7fe3bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.706+0000 7fe3dfe00640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3bc06c8f0 0x7fe3bc06edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0033000 tx=0x7fe3d0009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:33.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.708+0000 7fe3dcc00640 1 -- 172.21.15.69:0/86335752 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3e015ed40 con 0x7fe3e014c650 2024-04-24T08:09:33.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.959+0000 7fe3e5c00640 1 -- 172.21.15.69:0/86335752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fe3e00d0d90 con 0x7fe3e014c650 2024-04-24T08:09:33.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.963+0000 7fe3dcc00640 1 -- 172.21.15.69:0/86335752 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v8) ==== 97+0+0 (secure 0 0 0) 0x7fe3c8059310 con 0x7fe3e014c650 2024-04-24T08:09:33.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.965+0000 7fe3e5c00640 1 -- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3bc06c8f0 msgr2=0x7fe3bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.965+0000 7fe3e5c00640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3bc06c8f0 0x7fe3bc06edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0033000 tx=0x7fe3d0009000 comp rx=0 tx=0).stop 2024-04-24T08:09:33.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.965+0000 7fe3e5c00640 1 -- 172.21.15.69:0/86335752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3e014c650 msgr2=0x7fe3e00cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.965+0000 7fe3e5c00640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3e014c650 0x7fe3e00cc8b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800c430 tx=0x7fe3c800c8f0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.966+0000 7fe3e5c00640 1 -- 172.21.15.69:0/86335752 shutdown_connections 2024-04-24T08:09:33.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.966+0000 7fe3e5c00640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3bc06c8f0 0x7fe3bc06edb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.966+0000 7fe3e5c00640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe3e014c650 0x7fe3e00cc8b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.966+0000 7fe3e5c00640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe3e01666e0 0x7fe3e00ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.966+0000 7fe3e5c00640 1 --2- 172.21.15.69:0/86335752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe3e006db90 0x7fe3e0154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.966+0000 7fe3e5c00640 1 -- 172.21.15.69:0/86335752 >> 172.21.15.69:0/86335752 conn(0x7fe3e00d0160 msgr2=0x7fe3e0074b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:33.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.966+0000 7fe3e5c00640 1 -- 172.21.15.69:0/86335752 shutdown_connections 2024-04-24T08:09:33.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:33.966+0000 7fe3e5c00640 1 -- 172.21.15.69:0/86335752 wait complete. 2024-04-24T08:09:33.984 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-24T08:09:34.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.171+0000 7f763d800640 1 Processor -- start 2024-04-24T08:09:34.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.172+0000 7f763d800640 1 -- start start 2024-04-24T08:09:34.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.173+0000 7f763d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f763806eb30 0x7f7638144090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.173+0000 7f763d800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76380b3190 con 0x7f7638071be0 2024-04-24T08:09:34.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.173+0000 7f763d800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76380ad590 con 0x7f763815ab10 2024-04-24T08:09:34.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.173+0000 7f763d800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f76380b3400 con 0x7f763806eb30 2024-04-24T08:09:34.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.173+0000 7f762fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f763806eb30 0x7f7638144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.173+0000 7f762fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f763806eb30 0x7f7638144090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58684/0 (socket says 172.21.15.69:58684) 2024-04-24T08:09:34.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.174+0000 7f762fe00640 1 -- 172.21.15.69:0/1261215211 learned_addr learned my addr 172.21.15.69:0/1261215211 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:34.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.173+0000 7f763ce00640 1 --1- 172.21.15.69:0/1261215211 >> v1:172.21.15.69:6789/0 conn(0x7f7638071be0 0x7f76380b3ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32828/0 (socket says 172.21.15.69:32828) 2024-04-24T08:09:34.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.174+0000 7f762fe00640 1 -- 172.21.15.69:0/1261215211 >> v1:172.21.15.121:6789/0 conn(0x7f763815ab10 legacy=0x7f763806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:34.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.174+0000 7f762fe00640 1 -- 172.21.15.69:0/1261215211 >> v1:172.21.15.69:6789/0 conn(0x7f7638071be0 legacy=0x7f76380b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.174+0000 7f762fe00640 1 -- 172.21.15.69:0/1261215211 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76380b36e0 con 0x7f763806eb30 2024-04-24T08:09:34.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.174+0000 7f7637400640 1 -- 172.21.15.69:0/1261215211 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 141685240 0 0) 0x7f76380b3190 con 0x7f7638071be0 2024-04-24T08:09:34.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.175+0000 7f762fe00640 1 --2- 172.21.15.69:0/1261215211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763806eb30 0x7f7638144090 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7620009870 tx=0x7f762002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f120452fb6bf17b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:34.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.175+0000 7f7637400640 1 -- 172.21.15.69:0/1261215211 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620037070 con 0x7f763806eb30 2024-04-24T08:09:34.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.175+0000 7f7637400640 1 -- 172.21.15.69:0/1261215211 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7620032440 con 0x7f763806eb30 2024-04-24T08:09:34.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.175+0000 7f7637400640 1 -- 172.21.15.69:0/1261215211 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7620032810 con 0x7f763806eb30 2024-04-24T08:09:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.176+0000 7f763d800640 1 -- 172.21.15.69:0/1261215211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763806eb30 msgr2=0x7f7638144090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.176+0000 7f763d800640 1 --2- 172.21.15.69:0/1261215211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763806eb30 0x7f7638144090 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7620009870 tx=0x7f762002a2a0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.176+0000 7f763d800640 1 -- 172.21.15.69:0/1261215211 shutdown_connections 2024-04-24T08:09:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.176+0000 7f763d800640 1 --2- 172.21.15.69:0/1261215211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763806eb30 0x7f7638144090 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.176+0000 7f763d800640 1 -- 172.21.15.69:0/1261215211 >> 172.21.15.69:0/1261215211 conn(0x7f76380d0160 msgr2=0x7f76380d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.176+0000 7f763d800640 1 -- 172.21.15.69:0/1261215211 shutdown_connections 2024-04-24T08:09:34.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.176+0000 7f763d800640 1 -- 172.21.15.69:0/1261215211 wait complete. 2024-04-24T08:09:34.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.178+0000 7f763d800640 1 Processor -- start 2024-04-24T08:09:34.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.178+0000 7f763d800640 1 -- start start 2024-04-24T08:09:34.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.179+0000 7f763d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f763806eb30 0x7f76381f3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.179+0000 7f763d800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7638071be0 0x7f763814fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.179+0000 7f763d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763815ab10 0x7f763815d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.179+0000 7f763d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f763815aef0 con 0x7f763815ab10 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.179+0000 7f763d800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7638072150 con 0x7f7638071be0 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.179+0000 7f763d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f763806f950 con 0x7f763806eb30 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.179+0000 7f763ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763815ab10 0x7f763815d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.179+0000 7f763ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763815ab10 0x7f763815d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58696/0 (socket says 172.21.15.69:58696) 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.179+0000 7f763ce00640 1 -- 172.21.15.69:0/2285961806 learned_addr learned my addr 172.21.15.69:0/2285961806 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f762fe00640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7638071be0 0x7f763814fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f763ce00640 1 -- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f763806eb30 msgr2=0x7f76381f3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f7637e00640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f763806eb30 0x7f76381f3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f763ce00640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f763806eb30 0x7f76381f3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f763ce00640 1 -- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7638071be0 msgr2=0x7f763814fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f763ce00640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7638071be0 0x7f763814fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f763ce00640 1 -- 172.21.15.69:0/2285961806 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f763814db70 con 0x7f763815ab10 2024-04-24T08:09:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f7637e00640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f763806eb30 0x7f76381f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f763ce00640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763815ab10 0x7f763815d240 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f763000c7a0 tx=0x7f763000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f7634c00640 1 -- 172.21.15.69:0/2285961806 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7630018070 con 0x7f763815ab10 2024-04-24T08:09:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f7634c00640 1 -- 172.21.15.69:0/2285961806 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f763000a640 con 0x7f763815ab10 2024-04-24T08:09:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f7634c00640 1 -- 172.21.15.69:0/2285961806 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f763000aab0 con 0x7f763815ab10 2024-04-24T08:09:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f763d800640 1 -- 172.21.15.69:0/2285961806 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f763814de50 con 0x7f763815ab10 2024-04-24T08:09:34.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.180+0000 7f763d800640 1 -- 172.21.15.69:0/2285961806 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f763814e410 con 0x7f763815ab10 2024-04-24T08:09:34.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.182+0000 7f763d800640 1 -- 172.21.15.69:0/2285961806 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76380694b0 con 0x7f763815ab10 2024-04-24T08:09:34.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.182+0000 7f7634c00640 1 -- 172.21.15.69:0/2285961806 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f763000ac50 con 0x7f763815ab10 2024-04-24T08:09:34.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.184+0000 7f7634c00640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f761406c8f0 0x7f761406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.184+0000 7f7634c00640 1 -- 172.21.15.69:0/2285961806 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f763008af20 con 0x7f763815ab10 2024-04-24T08:09:34.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.184+0000 7f7637e00640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f761406c8f0 0x7f761406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.186+0000 7f7637e00640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f761406c8f0 0x7f761406edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f762800a830 tx=0x7f7628008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:34.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.188+0000 7f7634c00640 1 -- 172.21.15.69:0/2285961806 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f76380694b0 con 0x7f763815ab10 2024-04-24T08:09:34.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.449+0000 7f763d800640 1 -- 172.21.15.69:0/2285961806 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f763806f020 con 0x7f763815ab10 2024-04-24T08:09:34.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.453+0000 7f7634c00640 1 -- 172.21.15.69:0/2285961806 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v9) ==== 90+0+0 (secure 0 0 0) 0x7f7630059600 con 0x7f763815ab10 2024-04-24T08:09:34.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 -- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f761406c8f0 msgr2=0x7f761406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f761406c8f0 0x7f761406edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f762800a830 tx=0x7f7628008040 comp rx=0 tx=0).stop 2024-04-24T08:09:34.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 -- 172.21.15.69:0/2285961806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763815ab10 msgr2=0x7f763815d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763815ab10 0x7f763815d240 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f763000c7a0 tx=0x7f763000cc60 comp rx=0 tx=0).stop 2024-04-24T08:09:34.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 -- 172.21.15.69:0/2285961806 shutdown_connections 2024-04-24T08:09:34.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f761406c8f0 0x7f761406edb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f763815ab10 0x7f763815d240 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7638071be0 0x7f763814fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 --2- 172.21.15.69:0/2285961806 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f763806eb30 0x7f76381f3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 -- 172.21.15.69:0/2285961806 >> 172.21.15.69:0/2285961806 conn(0x7f76380d0160 msgr2=0x7f76380b0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:34.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 -- 172.21.15.69:0/2285961806 shutdown_connections 2024-04-24T08:09:34.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.457+0000 7f763d800640 1 -- 172.21.15.69:0/2285961806 wait complete. 2024-04-24T08:09:34.475 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:34.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.664+0000 7fc1dfe00640 1 Processor -- start 2024-04-24T08:09:34.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.664+0000 7fc1dfe00640 1 -- start start 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.665+0000 7fc1dfe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc1e01523f0 0x7fc1e006db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.665+0000 7fc1dfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc1e015b980 0x7fc1e015be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.665+0000 7fc1dfe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e00aeac0 con 0x7fc1e0072e20 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.666+0000 7fc1dfe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc1e00aeda0 con 0x7fc1e015b980 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.666+0000 7fc1dfe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc1e00af080 con 0x7fc1e01523f0 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.666+0000 7fc1de000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc1e0072e20 0x7fc1e0073260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32836/0 (socket says 172.21.15.69:32836) 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.666+0000 7fc1df400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc1e015b980 0x7fc1e015be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.666+0000 7fc1de000640 1 -- 172.21.15.69:0/1690997557 learned_addr learned my addr 172.21.15.69:0/1690997557 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.666+0000 7fc1dea00640 1 --2- 172.21.15.69:0/1690997557 >> v2:172.21.15.121:3300/0 conn(0x7fc1e01523f0 0x7fc1e006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.666+0000 7fc1df400640 1 -- 172.21.15.69:0/1690997557 >> v2:172.21.15.121:3300/0 conn(0x7fc1e01523f0 msgr2=0x7fc1e006db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.666+0000 7fc1df400640 1 --2- 172.21.15.69:0/1690997557 >> v2:172.21.15.121:3300/0 conn(0x7fc1e01523f0 0x7fc1e006db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.667+0000 7fc1df400640 1 -- 172.21.15.69:0/1690997557 >> v1:172.21.15.69:6789/0 conn(0x7fc1e0072e20 legacy=0x7fc1e0073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.667+0000 7fc1dd600640 1 -- 172.21.15.69:0/1690997557 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3960106397 0 0) 0x7fc1e00aeac0 con 0x7fc1e0072e20 2024-04-24T08:09:34.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.667+0000 7fc1df400640 1 -- 172.21.15.69:0/1690997557 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e00af360 con 0x7fc1e015b980 2024-04-24T08:09:34.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.668+0000 7fc1df400640 1 --2- 172.21.15.69:0/1690997557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 0x7fc1e015be10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc1e006cb50 tx=0x7fc1d802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5e2678ace921ce server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:34.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.668+0000 7fc1dd600640 1 -- 172.21.15.69:0/1690997557 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8007740 con 0x7fc1e015b980 2024-04-24T08:09:34.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.668+0000 7fc1dd600640 1 -- 172.21.15.69:0/1690997557 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d80078e0 con 0x7fc1e015b980 2024-04-24T08:09:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.668+0000 7fc1dd600640 1 -- 172.21.15.69:0/1690997557 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8007cb0 con 0x7fc1e015b980 2024-04-24T08:09:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.669+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1690997557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 msgr2=0x7fc1e015be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.669+0000 7fc1dfe00640 1 --2- 172.21.15.69:0/1690997557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 0x7fc1e015be10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fc1e006cb50 tx=0x7fc1d802a5f0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.669+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1690997557 shutdown_connections 2024-04-24T08:09:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.669+0000 7fc1dfe00640 1 --2- 172.21.15.69:0/1690997557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 0x7fc1e015be10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.669+0000 7fc1dfe00640 1 --2- 172.21.15.69:0/1690997557 >> v2:172.21.15.121:3300/0 conn(0x7fc1e01523f0 0x7fc1e006db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.669+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1690997557 >> 172.21.15.69:0/1690997557 conn(0x7fc1e00d0160 msgr2=0x7fc1e00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.669+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1690997557 shutdown_connections 2024-04-24T08:09:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.669+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1690997557 wait complete. 2024-04-24T08:09:34.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.670+0000 7fc1dfe00640 1 Processor -- start 2024-04-24T08:09:34.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.671+0000 7fc1dfe00640 1 -- start start 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.671+0000 7fc1dfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0072e20 0x7fc1e0148450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.671+0000 7fc1dfe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e01523f0 0x7fc1e00b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.671+0000 7fc1dfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 0x7fc1e00b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.671+0000 7fc1dfe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e0073970 con 0x7fc1e015b980 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.671+0000 7fc1dfe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e01527d0 con 0x7fc1e01523f0 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.671+0000 7fc1dfe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e0073af0 con 0x7fc1e0072e20 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1df400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 0x7fc1e00b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1df400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 0x7fc1e00b4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58712/0 (socket says 172.21.15.69:58712) 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1df400640 1 -- 172.21.15.69:0/1285831204 learned_addr learned my addr 172.21.15.69:0/1285831204 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1dea00640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0072e20 0x7fc1e0148450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1de000640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e01523f0 0x7fc1e00b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1df400640 1 -- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0072e20 msgr2=0x7fc1e0148450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1df400640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0072e20 0x7fc1e0148450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1df400640 1 -- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e01523f0 msgr2=0x7fc1e00b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1df400640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e01523f0 0x7fc1e00b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1df400640 1 -- 172.21.15.69:0/1285831204 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e0148990 con 0x7fc1e015b980 2024-04-24T08:09:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1dea00640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0072e20 0x7fc1e0148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.672+0000 7fc1de000640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e01523f0 0x7fc1e00b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:34.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.673+0000 7fc1df400640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 0x7fc1e00b4240 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009e80 tx=0x7fc1d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.673+0000 7fc1d7400640 1 -- 172.21.15.69:0/1285831204 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d803f070 con 0x7fc1e015b980 2024-04-24T08:09:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.673+0000 7fc1d7400640 1 -- 172.21.15.69:0/1285831204 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d8031c40 con 0x7fc1e015b980 2024-04-24T08:09:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.673+0000 7fc1d7400640 1 -- 172.21.15.69:0/1285831204 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8009450 con 0x7fc1e015b980 2024-04-24T08:09:34.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.673+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1285831204 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e009cd80 con 0x7fc1e015b980 2024-04-24T08:09:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.673+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1285831204 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1e009d340 con 0x7fc1e015b980 2024-04-24T08:09:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.674+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1285831204 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1e0075d50 con 0x7fc1e015b980 2024-04-24T08:09:34.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.675+0000 7fc1d7400640 1 -- 172.21.15.69:0/1285831204 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc1d8003b90 con 0x7fc1e015b980 2024-04-24T08:09:34.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.677+0000 7fc1d7400640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc1c006c8f0 0x7fc1c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.677+0000 7fc1d7400640 1 -- 172.21.15.69:0/1285831204 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fc1d80aca00 con 0x7fc1e015b980 2024-04-24T08:09:34.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.677+0000 7fc1dea00640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc1c006c8f0 0x7fc1c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.679+0000 7fc1dea00640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc1c006c8f0 0x7fc1c006edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0006fd0 tx=0x7fc1d0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:34.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.681+0000 7fc1d7400640 1 -- 172.21.15.69:0/1285831204 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc1d807b0e0 con 0x7fc1e015b980 2024-04-24T08:09:34.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.865+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1285831204 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1e01595a0 con 0x7fc1e015b980 2024-04-24T08:09:34.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.867+0000 7fc1d7400640 1 -- 172.21.15.69:0/1285831204 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+12867 (secure 0 0 0) 0x7fc1d807a9e0 con 0x7fc1e015b980 2024-04-24T08:09:34.868 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:34.868 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":29,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:31.042459+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc1c006c8f0 msgr2=0x7fc1c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc1c006c8f0 0x7fc1c006edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0006fd0 tx=0x7fc1d0008040 comp rx=0 tx=0).stop 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1285831204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 msgr2=0x7fc1e00b4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 0x7fc1e00b4240 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009e80 tx=0x7fc1d8004a40 comp rx=0 tx=0).stop 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1285831204 shutdown_connections 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc1c006c8f0 0x7fc1c006edb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc1e015b980 0x7fc1e00b4240 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc1e01523f0 0x7fc1e00b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 --2- 172.21.15.69:0/1285831204 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc1e0072e20 0x7fc1e0148450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1285831204 >> 172.21.15.69:0/1285831204 conn(0x7fc1e00d0160 msgr2=0x7fc1e0073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1285831204 shutdown_connections 2024-04-24T08:09:34.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:34.870+0000 7fc1dfe00640 1 -- 172.21.15.69:0/1285831204 wait complete. 2024-04-24T08:09:34.889 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:34.889 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:35.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.087+0000 7f4df6200640 1 Processor -- start 2024-04-24T08:09:35.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.087+0000 7f4df6200640 1 -- start start 2024-04-24T08:09:35.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.089+0000 7f4df6200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4df0071d70 0x7f4df015ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.089+0000 7f4df6200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df0160920 con 0x7f4df01420a0 2024-04-24T08:09:35.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.089+0000 7f4df6200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df0072450 con 0x7f4df0160510 2024-04-24T08:09:35.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.089+0000 7f4df6200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4df00cf1e0 con 0x7f4df0071d70 2024-04-24T08:09:35.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.089+0000 7f4df4e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4df01420a0 0x7f4df0142480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32846/0 (socket says 172.21.15.69:32846) 2024-04-24T08:09:35.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.089+0000 7f4df4e00640 1 -- 172.21.15.69:0/699903319 learned_addr learned my addr 172.21.15.69:0/699903319 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4de7e00640 1 --2- 172.21.15.69:0/699903319 >> v2:172.21.15.121:3300/0 conn(0x7f4df0071d70 0x7f4df015ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3592597519 0 0) 0x7f4de0001490 con 0x7f4df01420a0 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4dd4003520 con 0x7f4df01420a0 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 358622715 0 0) 0x7f4dd4003520 con 0x7f4df01420a0 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 >> v2:172.21.15.121:3300/0 conn(0x7f4df0071d70 msgr2=0x7f4df015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 --2- 172.21.15.69:0/699903319 >> v2:172.21.15.121:3300/0 conn(0x7f4df0071d70 0x7f4df015ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 >> v1:172.21.15.121:6789/0 conn(0x7f4df0160510 legacy=0x7f4df00cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df0072730 con 0x7f4df01420a0 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f4de0001bc0 con 0x7f4df01420a0 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 >> v1:172.21.15.69:6789/0 conn(0x7f4df01420a0 legacy=0x7f4df0142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 --2- 172.21.15.69:0/699903319 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dd4003a00 0x7f4dd4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.090+0000 7f4defe00640 1 --2- 172.21.15.69:0/699903319 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dd4004800 0x7f4dd4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4defe00640 1 --2- 172.21.15.69:0/699903319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4dd400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4df00cf1e0 con 0x7f4dd40051c0 2024-04-24T08:09:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd400c0c0 con 0x7f4dd4004800 2024-04-24T08:09:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd40106d0 con 0x7f4dd4003a00 2024-04-24T08:09:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4df5800640 1 --2- 172.21.15.69:0/699903319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4dd400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4de7e00640 1 --2- 172.21.15.69:0/699903319 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dd4004800 0x7f4dd4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4df5800640 1 -- 172.21.15.69:0/699903319 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dd4003a00 msgr2=0x7f4dd4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4df5800640 1 --2- 172.21.15.69:0/699903319 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4dd4003a00 0x7f4dd4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4df5800640 1 -- 172.21.15.69:0/699903319 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dd4004800 msgr2=0x7f4dd4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4df5800640 1 --2- 172.21.15.69:0/699903319 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dd4004800 0x7f4dd4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.091+0000 7f4df5800640 1 -- 172.21.15.69:0/699903319 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd4003520 con 0x7f4dd40051c0 2024-04-24T08:09:35.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.092+0000 7f4df5800640 1 -- 172.21.15.69:0/699903319 reap_dead start 2024-04-24T08:09:35.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.092+0000 7f4df5800640 1 --2- 172.21.15.69:0/699903319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4dd400b9c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4de8001750 tx=0x7f4dd4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28dcb0cfbe45f7b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.092+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de8001370 con 0x7f4dd40051c0 2024-04-24T08:09:35.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.092+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4de80052f0 con 0x7f4dd40051c0 2024-04-24T08:09:35.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.092+0000 7f4defe00640 1 -- 172.21.15.69:0/699903319 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de802abe0 con 0x7f4dd40051c0 2024-04-24T08:09:35.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.093+0000 7f4df6200640 1 -- 172.21.15.69:0/699903319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 msgr2=0x7f4dd400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.093+0000 7f4df6200640 1 --2- 172.21.15.69:0/699903319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4dd400b9c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f4de8001750 tx=0x7f4dd4005830 comp rx=0 tx=0).stop 2024-04-24T08:09:35.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.093+0000 7f4df6200640 1 -- 172.21.15.69:0/699903319 shutdown_connections 2024-04-24T08:09:35.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.093+0000 7f4df6200640 1 --2- 172.21.15.69:0/699903319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4dd400b9c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.093+0000 7f4df6200640 1 -- 172.21.15.69:0/699903319 >> 172.21.15.69:0/699903319 conn(0x7f4df0067b80 msgr2=0x7f4df006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.093+0000 7f4df6200640 1 -- 172.21.15.69:0/699903319 shutdown_connections 2024-04-24T08:09:35.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.093+0000 7f4df6200640 1 -- 172.21.15.69:0/699903319 wait complete. 2024-04-24T08:09:35.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.095+0000 7f4df6200640 1 Processor -- start 2024-04-24T08:09:35.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.095+0000 7f4df6200640 1 -- start start 2024-04-24T08:09:35.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df6200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4df0153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df6200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4df013e790 0x7f4df013e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df6200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4df0156fe0 0x7f4df0157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df6200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4df0072cf0 con 0x7f4dd40051c0 2024-04-24T08:09:35.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df6200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4df0072200 con 0x7f4df0156fe0 2024-04-24T08:09:35.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df6200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4df015ffd0 con 0x7f4df013e790 2024-04-24T08:09:35.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4df0153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4df0153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58720/0 (socket says 172.21.15.69:58720) 2024-04-24T08:09:35.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df4e00640 1 -- 172.21.15.69:0/2223717038 learned_addr learned my addr 172.21.15.69:0/2223717038 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:35.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df4e00640 1 -- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4df013e790 msgr2=0x7f4df013e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df4e00640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4df013e790 0x7f4df013e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df4e00640 1 -- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4df0156fe0 msgr2=0x7f4df0157460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.096+0000 7f4df4e00640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4df0156fe0 0x7f4df0157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.097+0000 7f4df4e00640 1 -- 172.21.15.69:0/2223717038 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df013e540 con 0x7f4dd40051c0 2024-04-24T08:09:35.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.097+0000 7f4df4e00640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4df0153350 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4de000c9c0 tx=0x7f4de000ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.097+0000 7f4ded600640 1 -- 172.21.15.69:0/2223717038 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de0018070 con 0x7f4dd40051c0 2024-04-24T08:09:35.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.097+0000 7f4ded600640 1 -- 172.21.15.69:0/2223717038 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4de000a760 con 0x7f4dd40051c0 2024-04-24T08:09:35.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.097+0000 7f4ded600640 1 -- 172.21.15.69:0/2223717038 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de000ab50 con 0x7f4dd40051c0 2024-04-24T08:09:35.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.097+0000 7f4df6200640 1 -- 172.21.15.69:0/2223717038 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df01547f0 con 0x7f4dd40051c0 2024-04-24T08:09:35.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.097+0000 7f4df6200640 1 -- 172.21.15.69:0/2223717038 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df0154db0 con 0x7f4dd40051c0 2024-04-24T08:09:35.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.098+0000 7f4df6200640 1 -- 172.21.15.69:0/2223717038 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df0075d50 con 0x7f4dd40051c0 2024-04-24T08:09:35.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.099+0000 7f4ded600640 1 -- 172.21.15.69:0/2223717038 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4de001b070 con 0x7f4dd40051c0 2024-04-24T08:09:35.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.100+0000 7f4ded600640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4dcc06c920 0x7f4dcc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.100+0000 7f4ded600640 1 -- 172.21.15.69:0/2223717038 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f4de008b030 con 0x7f4dd40051c0 2024-04-24T08:09:35.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.101+0000 7f4de7e00640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4dcc06c920 0x7f4dcc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.102+0000 7f4de7e00640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4dcc06c920 0x7f4dcc06ede0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4df015f820 tx=0x7f4dd8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.105+0000 7f4ded600640 1 -- 172.21.15.69:0/2223717038 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4df0160920 con 0x7f4dd40051c0 2024-04-24T08:09:35.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.301+0000 7f4df6200640 1 -- 172.21.15.69:0/2223717038 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4df006c470 con 0x7f4dd40051c0 2024-04-24T08:09:35.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.302+0000 7f4ded600640 1 -- 172.21.15.69:0/2223717038 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f4de00597b0 con 0x7f4dd40051c0 2024-04-24T08:09:35.303 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:35.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 -- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4dcc06c920 msgr2=0x7f4dcc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4dcc06c920 0x7f4dcc06ede0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4df015f820 tx=0x7f4dd8008040 comp rx=0 tx=0).stop 2024-04-24T08:09:35.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 -- 172.21.15.69:0/2223717038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 msgr2=0x7f4df0153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4df0153350 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f4de000c9c0 tx=0x7f4de000ce80 comp rx=0 tx=0).stop 2024-04-24T08:09:35.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 -- 172.21.15.69:0/2223717038 shutdown_connections 2024-04-24T08:09:35.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4dcc06c920 0x7f4dcc06ede0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4df0156fe0 0x7f4df0157460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4df013e790 0x7f4df013e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 --2- 172.21.15.69:0/2223717038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4dd40051c0 0x7f4df0153350 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 -- 172.21.15.69:0/2223717038 >> 172.21.15.69:0/2223717038 conn(0x7f4df0067b80 msgr2=0x7f4df00b4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 -- 172.21.15.69:0/2223717038 shutdown_connections 2024-04-24T08:09:35.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.305+0000 7f4df6200640 1 -- 172.21.15.69:0/2223717038 wait complete. 2024-04-24T08:09:35.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T08:09:35.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.516+0000 7fc97c800640 1 Processor -- start 2024-04-24T08:09:35.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.516+0000 7fc97c800640 1 -- start start 2024-04-24T08:09:35.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.517+0000 7fc97c800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc974145a90 0x7fc97406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.517+0000 7fc97c800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc974145fe0 0x7fc974160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.517+0000 7fc97c800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9740cf3d0 con 0x7fc9740d0300 2024-04-24T08:09:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.517+0000 7fc97c800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc9740cf6b0 con 0x7fc974145fe0 2024-04-24T08:09:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.517+0000 7fc97c800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc9740cf990 con 0x7fc974145a90 2024-04-24T08:09:35.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.518+0000 7fc979400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc974145a90 0x7fc97406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.518+0000 7fc979e00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc9740d0300 0x7fc9740d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:39018/0 (socket says 172.21.15.69:39018) 2024-04-24T08:09:35.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.518+0000 7fc979e00640 1 -- 172.21.15.69:0/1287969377 learned_addr learned my addr 172.21.15.69:0/1287969377 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:35.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.518+0000 7fc97a800640 1 --2- 172.21.15.69:0/1287969377 >> v2:172.21.15.121:3300/0 conn(0x7fc974145fe0 0x7fc974160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.519+0000 7fc978a00640 1 -- 172.21.15.69:0/1287969377 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3156543534 0 0) 0x7fc9740cf3d0 con 0x7fc9740d0300 2024-04-24T08:09:35.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.519+0000 7fc978a00640 1 -- 172.21.15.69:0/1287969377 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc95c003520 con 0x7fc9740d0300 2024-04-24T08:09:35.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.519+0000 7fc979400640 1 -- 172.21.15.69:0/1287969377 >> v2:172.21.15.121:3300/0 conn(0x7fc974145fe0 msgr2=0x7fc974160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.519+0000 7fc979400640 1 --2- 172.21.15.69:0/1287969377 >> v2:172.21.15.121:3300/0 conn(0x7fc974145fe0 0x7fc974160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.519+0000 7fc979400640 1 -- 172.21.15.69:0/1287969377 >> v1:172.21.15.121:6789/0 conn(0x7fc9740d0300 legacy=0x7fc9740d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.519+0000 7fc979400640 1 -- 172.21.15.69:0/1287969377 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9740cfc70 con 0x7fc974145a90 2024-04-24T08:09:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.520+0000 7fc979400640 1 --2- 172.21.15.69:0/1287969377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc974145a90 0x7fc97406e240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc9640097a0 tx=0x7fc96402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e673baa3d60f6a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.521+0000 7fc978a00640 1 -- 172.21.15.69:0/1287969377 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc964037070 con 0x7fc974145a90 2024-04-24T08:09:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.521+0000 7fc978a00640 1 -- 172.21.15.69:0/1287969377 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc964032440 con 0x7fc974145a90 2024-04-24T08:09:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.521+0000 7fc978a00640 1 -- 172.21.15.69:0/1287969377 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9640327f0 con 0x7fc974145a90 2024-04-24T08:09:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.521+0000 7fc97c800640 1 -- 172.21.15.69:0/1287969377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc974145a90 msgr2=0x7fc97406e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.521+0000 7fc97c800640 1 --2- 172.21.15.69:0/1287969377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc974145a90 0x7fc97406e240 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc9640097a0 tx=0x7fc96402a180 comp rx=0 tx=0).stop 2024-04-24T08:09:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.521+0000 7fc97c800640 1 -- 172.21.15.69:0/1287969377 shutdown_connections 2024-04-24T08:09:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.521+0000 7fc97c800640 1 --2- 172.21.15.69:0/1287969377 >> v2:172.21.15.121:3300/0 conn(0x7fc974145fe0 0x7fc974160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.521+0000 7fc97c800640 1 --2- 172.21.15.69:0/1287969377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc974145a90 0x7fc97406e240 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.521+0000 7fc97c800640 1 -- 172.21.15.69:0/1287969377 >> 172.21.15.69:0/1287969377 conn(0x7fc974067d90 msgr2=0x7fc97406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.521+0000 7fc97c800640 1 -- 172.21.15.69:0/1287969377 shutdown_connections 2024-04-24T08:09:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.522+0000 7fc97c800640 1 -- 172.21.15.69:0/1287969377 wait complete. 2024-04-24T08:09:35.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.523+0000 7fc97c800640 1 Processor -- start 2024-04-24T08:09:35.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.523+0000 7fc97c800640 1 -- start start 2024-04-24T08:09:35.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.524+0000 7fc97c800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9740d0300 0x7fc974141960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.524+0000 7fc97c800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974145a90 0x7fc9740afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.524+0000 7fc97c800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc974145fe0 0x7fc9740b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.524+0000 7fc97c800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9741458f0 con 0x7fc974145fe0 2024-04-24T08:09:35.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.524+0000 7fc97c800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc974145770 con 0x7fc974145a90 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.524+0000 7fc97c800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc9740cdc30 con 0x7fc9740d0300 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc97a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc974145fe0 0x7fc9740b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc97a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc974145fe0 0x7fc9740b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58724/0 (socket says 172.21.15.69:58724) 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc97a800640 1 -- 172.21.15.69:0/611975547 learned_addr learned my addr 172.21.15.69:0/611975547 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc979e00640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9740d0300 0x7fc974141960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc979400640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974145a90 0x7fc9740afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc97a800640 1 -- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9740d0300 msgr2=0x7fc974141960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc97a800640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9740d0300 0x7fc974141960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc97a800640 1 -- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974145a90 msgr2=0x7fc9740afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc97a800640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974145a90 0x7fc9740afeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc97a800640 1 -- 172.21.15.69:0/611975547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc974141ea0 con 0x7fc974145fe0 2024-04-24T08:09:35.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.525+0000 7fc97a800640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc974145fe0 0x7fc9740b03f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc97000efc0 tx=0x7fc97000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.526+0000 7fc96e000640 1 -- 172.21.15.69:0/611975547 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970018020 con 0x7fc974145fe0 2024-04-24T08:09:35.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.526+0000 7fc96e000640 1 -- 172.21.15.69:0/611975547 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc970014480 con 0x7fc974145fe0 2024-04-24T08:09:35.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.526+0000 7fc97c800640 1 -- 172.21.15.69:0/611975547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9740ace50 con 0x7fc974145fe0 2024-04-24T08:09:35.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.526+0000 7fc96e000640 1 -- 172.21.15.69:0/611975547 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970014870 con 0x7fc974145fe0 2024-04-24T08:09:35.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.526+0000 7fc97c800640 1 -- 172.21.15.69:0/611975547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9740ad410 con 0x7fc974145fe0 2024-04-24T08:09:35.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.527+0000 7fc97c800640 1 -- 172.21.15.69:0/611975547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc974068a80 con 0x7fc974145fe0 2024-04-24T08:09:35.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.528+0000 7fc96e000640 1 -- 172.21.15.69:0/611975547 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc970005d10 con 0x7fc974145fe0 2024-04-24T08:09:35.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.529+0000 7fc96e000640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc95806c8f0 0x7fc95806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.529+0000 7fc96e000640 1 -- 172.21.15.69:0/611975547 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fc97008c770 con 0x7fc974145fe0 2024-04-24T08:09:35.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.530+0000 7fc979e00640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc95806c8f0 0x7fc95806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.531+0000 7fc979e00640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc95806c8f0 0x7fc95806edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc9680011e0 tx=0x7fc968009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.534+0000 7fc96e000640 1 -- 172.21.15.69:0/611975547 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc970022030 con 0x7fc974145fe0 2024-04-24T08:09:35.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.738+0000 7fc97c800640 1 -- 172.21.15.69:0/611975547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc97406dd70 con 0x7fc974145fe0 2024-04-24T08:09:35.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.739+0000 7fc96e000640 1 -- 172.21.15.69:0/611975547 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) ==== 84+0+65 (secure 0 0 0) 0x7fc970010040 con 0x7fc974145fe0 2024-04-24T08:09:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:35.740 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:09:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:09:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: "poolnum": 1, 2024-04-24T08:09:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: "poolname": ".mgr" 2024-04-24T08:09:35.740 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:35.740 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:09:35.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 -- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc95806c8f0 msgr2=0x7fc95806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc95806c8f0 0x7fc95806edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc9680011e0 tx=0x7fc968009040 comp rx=0 tx=0).stop 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 -- 172.21.15.69:0/611975547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc974145fe0 msgr2=0x7fc9740b03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc974145fe0 0x7fc9740b03f0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc97000efc0 tx=0x7fc97000c490 comp rx=0 tx=0).stop 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 -- 172.21.15.69:0/611975547 shutdown_connections 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc95806c8f0 0x7fc95806edb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc974145fe0 0x7fc9740b03f0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc974145a90 0x7fc9740afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 --2- 172.21.15.69:0/611975547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9740d0300 0x7fc974141960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 -- 172.21.15.69:0/611975547 >> 172.21.15.69:0/611975547 conn(0x7fc974067d90 msgr2=0x7fc97406f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 -- 172.21.15.69:0/611975547 shutdown_connections 2024-04-24T08:09:35.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.742+0000 7fc97c800640 1 -- 172.21.15.69:0/611975547 wait complete. 2024-04-24T08:09:35.761 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T08:09:35.762 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T08:09:35.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.964+0000 7f1228600640 1 Processor -- start 2024-04-24T08:09:35.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.964+0000 7f1228600640 1 -- start start 2024-04-24T08:09:35.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.965+0000 7f1228600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f122015e950 0x7f12200d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.965+0000 7f1228600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12200d0d00 0x7f12200b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.965+0000 7f1228600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12200cf150 con 0x7f12200b4a20 2024-04-24T08:09:35.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.965+0000 7f1228600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f12200d01f0 con 0x7f122015e950 2024-04-24T08:09:35.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.965+0000 7f1228600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f12200cfd60 con 0x7f12200d0d00 2024-04-24T08:09:35.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.966+0000 7f1225c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f122015e950 0x7f12200d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.966+0000 7f1225c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f122015e950 0x7f12200d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58726/0 (socket says 172.21.15.69:58726) 2024-04-24T08:09:35.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.966+0000 7f1225c00640 1 -- 172.21.15.69:0/2477137430 learned_addr learned my addr 172.21.15.69:0/2477137430 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:35.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.966+0000 7f1225200640 1 --2- 172.21.15.69:0/2477137430 >> v2:172.21.15.121:3300/0 conn(0x7f12200d0d00 0x7f12200b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.966+0000 7f1225c00640 1 -- 172.21.15.69:0/2477137430 >> v2:172.21.15.121:3300/0 conn(0x7f12200d0d00 msgr2=0x7f12200b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.966+0000 7f1225c00640 1 --2- 172.21.15.69:0/2477137430 >> v2:172.21.15.121:3300/0 conn(0x7f12200d0d00 0x7f12200b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.966+0000 7f1225c00640 1 -- 172.21.15.69:0/2477137430 >> v1:172.21.15.121:6789/0 conn(0x7f12200b4a20 legacy=0x7f12200ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.966+0000 7f1225c00640 1 -- 172.21.15.69:0/2477137430 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12200cf430 con 0x7f122015e950 2024-04-24T08:09:35.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.967+0000 7f1225c00640 1 --2- 172.21.15.69:0/2477137430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f122015e950 0x7f12200d07c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f122009e520 tx=0x7f121402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4439c81e4a1c7f40 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.968+0000 7f121fe00640 1 -- 172.21.15.69:0/2477137430 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214037070 con 0x7f122015e950 2024-04-24T08:09:35.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.968+0000 7f121fe00640 1 -- 172.21.15.69:0/2477137430 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1214032440 con 0x7f122015e950 2024-04-24T08:09:35.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.968+0000 7f121fe00640 1 -- 172.21.15.69:0/2477137430 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214032810 con 0x7f122015e950 2024-04-24T08:09:35.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.968+0000 7f1228600640 1 -- 172.21.15.69:0/2477137430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f122015e950 msgr2=0x7f12200d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.968+0000 7f1228600640 1 --2- 172.21.15.69:0/2477137430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f122015e950 0x7f12200d07c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f122009e520 tx=0x7f121402a2c0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.969+0000 7f1228600640 1 -- 172.21.15.69:0/2477137430 shutdown_connections 2024-04-24T08:09:35.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.969+0000 7f1228600640 1 --2- 172.21.15.69:0/2477137430 >> v2:172.21.15.121:3300/0 conn(0x7f12200d0d00 0x7f12200b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.969+0000 7f1228600640 1 --2- 172.21.15.69:0/2477137430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f122015e950 0x7f12200d07c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.969+0000 7f1228600640 1 -- 172.21.15.69:0/2477137430 >> 172.21.15.69:0/2477137430 conn(0x7f122006be40 msgr2=0x7f122006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.969+0000 7f1228600640 1 -- 172.21.15.69:0/2477137430 shutdown_connections 2024-04-24T08:09:35.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.969+0000 7f1228600640 1 -- 172.21.15.69:0/2477137430 wait complete. 2024-04-24T08:09:35.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.971+0000 7f1228600640 1 Processor -- start 2024-04-24T08:09:35.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.971+0000 7f1228600640 1 -- start start 2024-04-24T08:09:35.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.972+0000 7f1228600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12200b4a20 0x7f122015cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.972+0000 7f1228600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12200d0d00 0x7f12200c6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.972+0000 7f1228600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f122015e950 0x7f12201f3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.972+0000 7f1228600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f122006ea70 con 0x7f12200b4a20 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.972+0000 7f1228600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f12200cf9f0 con 0x7f122015e950 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.972+0000 7f1228600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f122015ed30 con 0x7f12200d0d00 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.972+0000 7f1225c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12200b4a20 0x7f122015cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.972+0000 7f1225c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12200b4a20 0x7f122015cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58728/0 (socket says 172.21.15.69:58728) 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.972+0000 7f1225c00640 1 -- 172.21.15.69:0/601020880 learned_addr learned my addr 172.21.15.69:0/601020880 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.973+0000 7f1225200640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12200d0d00 0x7f12200c6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.973+0000 7f1226600640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f122015e950 0x7f12201f3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.973+0000 7f1225c00640 1 -- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12200d0d00 msgr2=0x7f12200c6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.973+0000 7f1225c00640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12200d0d00 0x7f12200c6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.973+0000 7f1225c00640 1 -- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f122015e950 msgr2=0x7f12201f3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.973+0000 7f1225c00640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f122015e950 0x7f12201f3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.973+0000 7f1225c00640 1 -- 172.21.15.69:0/601020880 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12201578e0 con 0x7f12200b4a20 2024-04-24T08:09:35.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.973+0000 7f1226600640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f122015e950 0x7f12201f3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:35.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.973+0000 7f1225c00640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12200b4a20 0x7f122015cfa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f121402a300 tx=0x7f12140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.974+0000 7f121d600640 1 -- 172.21.15.69:0/601020880 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214041020 con 0x7f12200b4a20 2024-04-24T08:09:35.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.974+0000 7f121d600640 1 -- 172.21.15.69:0/601020880 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12140306c0 con 0x7f12200b4a20 2024-04-24T08:09:35.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.974+0000 7f121d600640 1 -- 172.21.15.69:0/601020880 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1214037040 con 0x7f12200b4a20 2024-04-24T08:09:35.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.974+0000 7f1228600640 1 -- 172.21.15.69:0/601020880 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12200c6de0 con 0x7f12200b4a20 2024-04-24T08:09:35.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.974+0000 7f1228600640 1 -- 172.21.15.69:0/601020880 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1220155b50 con 0x7f12200b4a20 2024-04-24T08:09:35.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.975+0000 7f1228600640 1 -- 172.21.15.69:0/601020880 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1220075dc0 con 0x7f12200b4a20 2024-04-24T08:09:35.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.976+0000 7f121d600640 1 -- 172.21.15.69:0/601020880 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1214044020 con 0x7f12200b4a20 2024-04-24T08:09:35.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.977+0000 7f121d600640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f120006c920 0x7f120006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.977+0000 7f121d600640 1 -- 172.21.15.69:0/601020880 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f12140ac0d0 con 0x7f12200b4a20 2024-04-24T08:09:35.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.978+0000 7f1225200640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f120006c920 0x7f120006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.979+0000 7f1225200640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f120006c920 0x7f120006ede0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f12201f7a60 tx=0x7f120c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:35.981+0000 7f121d600640 1 -- 172.21.15.69:0/601020880 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f121404b030 con 0x7f12200b4a20 2024-04-24T08:09:36.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:36.183+0000 7f1228600640 1 -- 172.21.15.69:0/601020880 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f12200b3ed0 con 0x7f12200b4a20 2024-04-24T08:09:37.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.055+0000 7f121d600640 1 -- 172.21.15.69:0/601020880 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) ==== 138+0+0 (secure 0 0 0) 0x7f121407a7b0 con 0x7f12200b4a20 2024-04-24T08:09:37.055 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' created 2024-04-24T08:09:37.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.058+0000 7f1228600640 1 -- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f120006c920 msgr2=0x7f120006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:37.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.058+0000 7f1228600640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f120006c920 0x7f120006ede0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f12201f7a60 tx=0x7f120c008040 comp rx=0 tx=0).stop 2024-04-24T08:09:37.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.058+0000 7f1228600640 1 -- 172.21.15.69:0/601020880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12200b4a20 msgr2=0x7f122015cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:37.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.058+0000 7f1228600640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12200b4a20 0x7f122015cfa0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f121402a300 tx=0x7f12140046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:37.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.058+0000 7f1228600640 1 -- 172.21.15.69:0/601020880 shutdown_connections 2024-04-24T08:09:37.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.058+0000 7f1228600640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f120006c920 0x7f120006ede0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:37.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.058+0000 7f1228600640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f122015e950 0x7f12201f3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:37.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.058+0000 7f1228600640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12200d0d00 0x7f12200c6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:37.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.058+0000 7f1228600640 1 --2- 172.21.15.69:0/601020880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12200b4a20 0x7f122015cfa0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:37.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.058+0000 7f1228600640 1 -- 172.21.15.69:0/601020880 >> 172.21.15.69:0/601020880 conn(0x7f122006be40 msgr2=0x7f1220071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:37.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.059+0000 7f1228600640 1 -- 172.21.15.69:0/601020880 shutdown_connections 2024-04-24T08:09:37.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.059+0000 7f1228600640 1 -- 172.21.15.69:0/601020880 wait complete. 2024-04-24T08:09:37.084 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T08:09:37.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.274+0000 7fcaab200640 1 Processor -- start 2024-04-24T08:09:37.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.275+0000 7fcaab200640 1 -- start start 2024-04-24T08:09:37.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.276+0000 7fcaab200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcaac06cd50 0x7fcaac0d4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:37.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.276+0000 7fcaab200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaac0d2ce0 con 0x7fcaac15ecf0 2024-04-24T08:09:37.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.276+0000 7fcaab200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaac0d2fc0 con 0x7fcaac15e910 2024-04-24T08:09:37.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.276+0000 7fcaab200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcaac0d32a0 con 0x7fcaac06cd50 2024-04-24T08:09:37.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.276+0000 7fcaa1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcaac06cd50 0x7fcaac0d4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:37.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.276+0000 7fcaaa800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcaac15ecf0 0x7fcaac0cf880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32848/0 (socket says 172.21.15.69:32848) 2024-04-24T08:09:37.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.276+0000 7fcaaa800640 1 -- 172.21.15.69:0/2617325631 learned_addr learned my addr 172.21.15.69:0/2617325631 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:37.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.277+0000 7fcaa1400640 1 -- 172.21.15.69:0/2617325631 >> v1:172.21.15.121:6789/0 conn(0x7fcaac15e910 legacy=0x7fcaac0d2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:37.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.277+0000 7fcaa1400640 1 -- 172.21.15.69:0/2617325631 >> v1:172.21.15.69:6789/0 conn(0x7fcaac15ecf0 legacy=0x7fcaac0cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:37.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.277+0000 7fcaa1400640 1 -- 172.21.15.69:0/2617325631 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaac0d3580 con 0x7fcaac06cd50 2024-04-24T08:09:37.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.277+0000 7fcaa9400640 1 -- 172.21.15.69:0/2617325631 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2596483788 0 0) 0x7fcaac0d2ce0 con 0x7fcaac15ecf0 2024-04-24T08:09:37.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.278+0000 7fcaa1400640 1 --2- 172.21.15.69:0/2617325631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 0x7fcaac0d4650 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fca940097a0 tx=0x7fca9402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7603001c25ee1d63 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:37.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.278+0000 7fcaa9400640 1 -- 172.21.15.69:0/2617325631 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94037070 con 0x7fcaac06cd50 2024-04-24T08:09:37.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.278+0000 7fcaa9400640 1 -- 172.21.15.69:0/2617325631 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca9402ae50 con 0x7fcaac06cd50 2024-04-24T08:09:37.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.278+0000 7fcaa9400640 1 -- 172.21.15.69:0/2617325631 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca94032660 con 0x7fcaac06cd50 2024-04-24T08:09:37.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.279+0000 7fcaab200640 1 -- 172.21.15.69:0/2617325631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 msgr2=0x7fcaac0d4650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:37.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.279+0000 7fcaab200640 1 --2- 172.21.15.69:0/2617325631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 0x7fcaac0d4650 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fca940097a0 tx=0x7fca9402a140 comp rx=0 tx=0).stop 2024-04-24T08:09:37.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.279+0000 7fcaab200640 1 -- 172.21.15.69:0/2617325631 shutdown_connections 2024-04-24T08:09:37.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.279+0000 7fcaab200640 1 --2- 172.21.15.69:0/2617325631 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 0x7fcaac0d4650 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:37.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.279+0000 7fcaab200640 1 -- 172.21.15.69:0/2617325631 >> 172.21.15.69:0/2617325631 conn(0x7fcaac08bf60 msgr2=0x7fcaac09d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:37.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.279+0000 7fcaab200640 1 -- 172.21.15.69:0/2617325631 shutdown_connections 2024-04-24T08:09:37.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.279+0000 7fcaab200640 1 -- 172.21.15.69:0/2617325631 wait complete. 2024-04-24T08:09:37.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.281+0000 7fcaab200640 1 Processor -- start 2024-04-24T08:09:37.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.281+0000 7fcaab200640 1 -- start start 2024-04-24T08:09:37.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.282+0000 7fcaab200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 0x7fcaac151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:37.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.282+0000 7fcaab200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaac15e910 0x7fcaac161660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:37.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.282+0000 7fcaab200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac15ecf0 0x7fcaac0b8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:37.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.282+0000 7fcaab200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcaac0cf3e0 con 0x7fcaac06cd50 2024-04-24T08:09:37.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.282+0000 7fcaab200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcaac0cf260 con 0x7fcaac15e910 2024-04-24T08:09:37.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.282+0000 7fcaab200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcaac0b3ae0 con 0x7fcaac15ecf0 2024-04-24T08:09:37.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.282+0000 7fcaa9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 0x7fcaac151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:37.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.282+0000 7fcaa9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 0x7fcaac151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58746/0 (socket says 172.21.15.69:58746) 2024-04-24T08:09:37.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.282+0000 7fcaa9e00640 1 -- 172.21.15.69:0/263279652 learned_addr learned my addr 172.21.15.69:0/263279652 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:37.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaa9e00640 1 -- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac15ecf0 msgr2=0x7fcaac0b8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:37.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac15ecf0 0x7fcaac0b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:37.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaa9e00640 1 -- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaac15e910 msgr2=0x7fcaac161660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:37.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaac15e910 0x7fcaac161660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:37.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaa9e00640 1 -- 172.21.15.69:0/263279652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaac1602e0 con 0x7fcaac06cd50 2024-04-24T08:09:37.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaa9e00640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 0x7fcaac151ba0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00c460 tx=0x7fca9c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:37.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaa2a00640 1 -- 172.21.15.69:0/263279652 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9c016020 con 0x7fcaac06cd50 2024-04-24T08:09:37.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaa2a00640 1 -- 172.21.15.69:0/263279652 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca9c012870 con 0x7fcaac06cd50 2024-04-24T08:09:37.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaab200640 1 -- 172.21.15.69:0/263279652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaac1605c0 con 0x7fcaac06cd50 2024-04-24T08:09:37.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaa2a00640 1 -- 172.21.15.69:0/263279652 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9c012c60 con 0x7fcaac06cd50 2024-04-24T08:09:37.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.283+0000 7fcaab200640 1 -- 172.21.15.69:0/263279652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaac160b80 con 0x7fcaac06cd50 2024-04-24T08:09:37.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.284+0000 7fcaab200640 1 -- 172.21.15.69:0/263279652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaac075d50 con 0x7fcaac06cd50 2024-04-24T08:09:37.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.285+0000 7fcaa2a00640 1 -- 172.21.15.69:0/263279652 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fca9c019020 con 0x7fcaac06cd50 2024-04-24T08:09:37.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.286+0000 7fcaa2a00640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 0x7fca8806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:37.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.286+0000 7fcaa2a00640 1 -- 172.21.15.69:0/263279652 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5227+0+0 (secure 0 0 0) 0x7fca9c08b7e0 con 0x7fcaac06cd50 2024-04-24T08:09:37.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.287+0000 7fcaa1400640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 0x7fca8806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:37.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.288+0000 7fcaa1400640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 0x7fca8806ede0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fca94034000 tx=0x7fca9402c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.291+0000 7fcaa2a00640 1 -- 172.21.15.69:0/263279652 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca9c020030 con 0x7fcaac06cd50 2024-04-24T08:09:37.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:37.493+0000 7fcaab200640 1 -- 172.21.15.69:0/263279652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fcaac068db0 con 0x7fcaac06cd50 2024-04-24T08:09:38.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.072+0000 7fcaa2a00640 1 -- 172.21.15.69:0/263279652 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) ==== 170+0+0 (secure 0 0 0) 0x7fca9c059cc0 con 0x7fcaac06cd50 2024-04-24T08:09:38.073 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' created 2024-04-24T08:09:38.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.078+0000 7fcaab200640 1 -- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 msgr2=0x7fca8806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:38.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.078+0000 7fcaab200640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 0x7fca8806ede0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fca94034000 tx=0x7fca9402c000 comp rx=0 tx=0).stop 2024-04-24T08:09:38.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.078+0000 7fcaab200640 1 -- 172.21.15.69:0/263279652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 msgr2=0x7fcaac151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:38.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.078+0000 7fcaab200640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 0x7fcaac151ba0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00c460 tx=0x7fca9c00c920 comp rx=0 tx=0).stop 2024-04-24T08:09:38.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.079+0000 7fcaab200640 1 -- 172.21.15.69:0/263279652 shutdown_connections 2024-04-24T08:09:38.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.079+0000 7fcaab200640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 0x7fca8806ede0 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fca94034000 tx=0x7fca9402c000 comp rx=0 tx=0).stop 2024-04-24T08:09:38.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.079+0000 7fcaab200640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaac15ecf0 0x7fcaac0b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:38.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.079+0000 7fcaab200640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaac15e910 0x7fcaac161660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:38.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.079+0000 7fcaab200640 1 --2- 172.21.15.69:0/263279652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaac06cd50 0x7fcaac151ba0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:38.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.079+0000 7fcaab200640 1 -- 172.21.15.69:0/263279652 >> 172.21.15.69:0/263279652 conn(0x7fcaac08bf60 msgr2=0x7fcaac0d1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:38.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.080+0000 7fcaab200640 1 -- 172.21.15.69:0/263279652 shutdown_connections 2024-04-24T08:09:38.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.080+0000 7fcaab200640 1 -- 172.21.15.69:0/263279652 wait complete. 2024-04-24T08:09:38.114 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T08:09:38.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.294+0000 7f378f600640 1 Processor -- start 2024-04-24T08:09:38.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.294+0000 7f378f600640 1 -- start start 2024-04-24T08:09:38.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.296+0000 7f378f600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f378815ec30 0x7f37880d1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:38.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.296+0000 7f378f600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37880d2710 con 0x7f378815e850 2024-04-24T08:09:38.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.296+0000 7f378f600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37880d29f0 con 0x7f37880d1690 2024-04-24T08:09:38.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.296+0000 7f378f600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f37880d37b0 con 0x7f378815ec30 2024-04-24T08:09:38.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.296+0000 7f3787e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f378815ec30 0x7f37880d1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:38.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.296+0000 7f378cc00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f378815e850 0x7f37880cff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32860/0 (socket says 172.21.15.69:32860) 2024-04-24T08:09:38.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.296+0000 7f378cc00640 1 -- 172.21.15.69:0/886545530 learned_addr learned my addr 172.21.15.69:0/886545530 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:38.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.297+0000 7f3787e00640 1 -- 172.21.15.69:0/886545530 >> v1:172.21.15.121:6789/0 conn(0x7f37880d1690 legacy=0x7f37881688f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:38.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.297+0000 7f3787e00640 1 -- 172.21.15.69:0/886545530 >> v1:172.21.15.69:6789/0 conn(0x7f378815e850 legacy=0x7f37880cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:38.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.297+0000 7f3787e00640 1 -- 172.21.15.69:0/886545530 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37880d2cd0 con 0x7f378815ec30 2024-04-24T08:09:38.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.298+0000 7f3787e00640 1 --2- 172.21.15.69:0/886545530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 0x7f37880d1150 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f37700097a0 tx=0x7f377002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6d30e4c3971705 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:38.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.298+0000 7f3787400640 1 -- 172.21.15.69:0/886545530 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770037070 con 0x7f378815ec30 2024-04-24T08:09:38.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.298+0000 7f3787400640 1 -- 172.21.15.69:0/886545530 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f377002ae50 con 0x7f378815ec30 2024-04-24T08:09:38.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.298+0000 7f3787400640 1 -- 172.21.15.69:0/886545530 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3770032660 con 0x7f378815ec30 2024-04-24T08:09:38.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.299+0000 7f378f600640 1 -- 172.21.15.69:0/886545530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 msgr2=0x7f37880d1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:38.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.299+0000 7f378f600640 1 --2- 172.21.15.69:0/886545530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 0x7f37880d1150 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f37700097a0 tx=0x7f377002a140 comp rx=0 tx=0).stop 2024-04-24T08:09:38.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.299+0000 7f378f600640 1 -- 172.21.15.69:0/886545530 shutdown_connections 2024-04-24T08:09:38.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.299+0000 7f378f600640 1 --2- 172.21.15.69:0/886545530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 0x7f37880d1150 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:38.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.299+0000 7f378f600640 1 -- 172.21.15.69:0/886545530 >> 172.21.15.69:0/886545530 conn(0x7f378808bf60 msgr2=0x7f37880d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:38.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.299+0000 7f378f600640 1 -- 172.21.15.69:0/886545530 shutdown_connections 2024-04-24T08:09:38.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.299+0000 7f378f600640 1 -- 172.21.15.69:0/886545530 wait complete. 2024-04-24T08:09:38.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.301+0000 7f378f600640 1 Processor -- start 2024-04-24T08:09:38.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.301+0000 7f378f600640 1 -- start start 2024-04-24T08:09:38.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.302+0000 7f378f600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37880d1690 0x7f3788156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:38.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.302+0000 7f378f600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f378815e850 0x7f37880b51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.302+0000 7f378f600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 0x7f37880cc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.302+0000 7f378f600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f37880d1b00 con 0x7f378815ec30 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.302+0000 7f378f600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f378806eba0 con 0x7f37880d1690 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.302+0000 7f378f600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f37880d3290 con 0x7f378815e850 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.302+0000 7f378d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 0x7f37880cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.302+0000 7f378d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 0x7f37880cc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58762/0 (socket says 172.21.15.69:58762) 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.302+0000 7f378d600640 1 -- 172.21.15.69:0/3370261976 learned_addr learned my addr 172.21.15.69:0/3370261976 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f378d600640 1 -- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f378815e850 msgr2=0x7f37880b51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f378d600640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f378815e850 0x7f37880b51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f378d600640 1 -- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37880d1690 msgr2=0x7f3788156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f378d600640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37880d1690 0x7f3788156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f378d600640 1 -- 172.21.15.69:0/3370261976 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37880b4430 con 0x7f378815ec30 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f378d600640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 0x7f37880cc880 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f378000c410 tx=0x7f378000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:38.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f3784c00640 1 -- 172.21.15.69:0/3370261976 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3780011790 con 0x7f378815ec30 2024-04-24T08:09:38.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f3784c00640 1 -- 172.21.15.69:0/3370261976 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3780017070 con 0x7f378815ec30 2024-04-24T08:09:38.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f3784c00640 1 -- 172.21.15.69:0/3370261976 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3780011bb0 con 0x7f378815ec30 2024-04-24T08:09:38.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f378f600640 1 -- 172.21.15.69:0/3370261976 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37880b4710 con 0x7f378815ec30 2024-04-24T08:09:38.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.303+0000 7f378f600640 1 -- 172.21.15.69:0/3370261976 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3788157380 con 0x7f378815ec30 2024-04-24T08:09:38.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.304+0000 7f378f600640 1 -- 172.21.15.69:0/3370261976 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3788068a80 con 0x7f378815ec30 2024-04-24T08:09:38.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.306+0000 7f3784c00640 1 -- 172.21.15.69:0/3370261976 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f378001c460 con 0x7f378815ec30 2024-04-24T08:09:38.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.307+0000 7f3784c00640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f376406c8f0 0x7f376406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:38.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.307+0000 7f3784c00640 1 -- 172.21.15.69:0/3370261976 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5609+0+0 (secure 0 0 0) 0x7f378008b6e0 con 0x7f378815ec30 2024-04-24T08:09:38.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.307+0000 7f378cc00640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f376406c8f0 0x7f376406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:38.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.309+0000 7f378cc00640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f376406c8f0 0x7f376406edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f37780012e0 tx=0x7f3778001630 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:38.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.312+0000 7f3784c00640 1 -- 172.21.15.69:0/3370261976 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3780021020 con 0x7f378815ec30 2024-04-24T08:09:38.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:38.548+0000 7f378f600640 1 -- 172.21.15.69:0/3370261976 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f37881614a0 con 0x7f378815ec30 2024-04-24T08:09:39.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.076+0000 7f3784c00640 1 -- 172.21.15.69:0/3370261976 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) ==== 172+0+0 (secure 0 0 0) 0x7f3780059b60 con 0x7f378815ec30 2024-04-24T08:09:39.077 INFO:teuthology.orchestra.run.smithi069.stderr:new fs with metadata pool 4 and data pool 5 2024-04-24T08:09:39.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.081+0000 7f378f600640 1 -- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f376406c8f0 msgr2=0x7f376406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.081+0000 7f378f600640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f376406c8f0 0x7f376406edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f37780012e0 tx=0x7f3778001630 comp rx=0 tx=0).stop 2024-04-24T08:09:39.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.081+0000 7f378f600640 1 -- 172.21.15.69:0/3370261976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 msgr2=0x7f37880cc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.081+0000 7f378f600640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 0x7f37880cc880 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f378000c410 tx=0x7f378000c8d0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.081+0000 7f378f600640 1 -- 172.21.15.69:0/3370261976 shutdown_connections 2024-04-24T08:09:39.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.081+0000 7f378f600640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f376406c8f0 0x7f376406edb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.081+0000 7f378f600640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f378815ec30 0x7f37880cc880 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.081+0000 7f378f600640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f378815e850 0x7f37880b51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.081+0000 7f378f600640 1 --2- 172.21.15.69:0/3370261976 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37880d1690 0x7f3788156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.081+0000 7f378f600640 1 -- 172.21.15.69:0/3370261976 >> 172.21.15.69:0/3370261976 conn(0x7f378808bf60 msgr2=0x7f378815f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:39.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.082+0000 7f378f600640 1 -- 172.21.15.69:0/3370261976 shutdown_connections 2024-04-24T08:09:39.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.082+0000 7f378f600640 1 -- 172.21.15.69:0/3370261976 wait complete. 2024-04-24T08:09:39.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:39.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.292+0000 7fa16b000640 1 Processor -- start 2024-04-24T08:09:39.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.292+0000 7fa16b000640 1 -- start start 2024-04-24T08:09:39.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.293+0000 7fa16b000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa164156c60 0x7fa16415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.293+0000 7fa16b000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa1640b30f0 0x7fa16415d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.293+0000 7fa16b000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1640ad0d0 0x7fa1640ad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.293+0000 7fa16b000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa16415d660 con 0x7fa164156c60 2024-04-24T08:09:39.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.293+0000 7fa16b000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa16406dc30 con 0x7fa1640b30f0 2024-04-24T08:09:39.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.293+0000 7fa16b000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa16406df10 con 0x7fa1640ad0d0 2024-04-24T08:09:39.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.294+0000 7fa169c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa164156c60 0x7fa16415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.294+0000 7fa169c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa164156c60 0x7fa16415cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60538/0 (socket says 172.21.15.69:60538) 2024-04-24T08:09:39.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.294+0000 7fa169c00640 1 -- 172.21.15.69:0/2378555509 learned_addr learned my addr 172.21.15.69:0/2378555509 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:39.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.294+0000 7fa169200640 1 --2- 172.21.15.69:0/2378555509 >> v2:172.21.15.121:3300/0 conn(0x7fa1640b30f0 0x7fa16415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.294+0000 7fa16a600640 1 --2- 172.21.15.69:0/2378555509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1640ad0d0 0x7fa1640ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.294+0000 7fa169c00640 1 -- 172.21.15.69:0/2378555509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1640ad0d0 msgr2=0x7fa1640ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:39.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.294+0000 7fa169c00640 1 --2- 172.21.15.69:0/2378555509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1640ad0d0 0x7fa1640ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.294+0000 7fa169c00640 1 -- 172.21.15.69:0/2378555509 >> v2:172.21.15.121:3300/0 conn(0x7fa1640b30f0 msgr2=0x7fa16415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:39.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.294+0000 7fa169c00640 1 --2- 172.21.15.69:0/2378555509 >> v2:172.21.15.121:3300/0 conn(0x7fa1640b30f0 0x7fa16415d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.294+0000 7fa169c00640 1 -- 172.21.15.69:0/2378555509 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16406e1f0 con 0x7fa164156c60 2024-04-24T08:09:39.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.296+0000 7fa169c00640 1 --2- 172.21.15.69:0/2378555509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa164156c60 0x7fa16415cac0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa16406d900 tx=0x7fa15802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae269b041884c29 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:39.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.296+0000 7fa163e00640 1 -- 172.21.15.69:0/2378555509 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158037070 con 0x7fa164156c60 2024-04-24T08:09:39.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.296+0000 7fa163e00640 1 -- 172.21.15.69:0/2378555509 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa158032440 con 0x7fa164156c60 2024-04-24T08:09:39.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.296+0000 7fa163e00640 1 -- 172.21.15.69:0/2378555509 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158032810 con 0x7fa164156c60 2024-04-24T08:09:39.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.297+0000 7fa16b000640 1 -- 172.21.15.69:0/2378555509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa164156c60 msgr2=0x7fa16415cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.297+0000 7fa16b000640 1 --2- 172.21.15.69:0/2378555509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa164156c60 0x7fa16415cac0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa16406d900 tx=0x7fa15802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.297+0000 7fa16b000640 1 -- 172.21.15.69:0/2378555509 shutdown_connections 2024-04-24T08:09:39.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.297+0000 7fa16b000640 1 --2- 172.21.15.69:0/2378555509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1640ad0d0 0x7fa1640ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.297+0000 7fa16b000640 1 --2- 172.21.15.69:0/2378555509 >> v2:172.21.15.121:3300/0 conn(0x7fa1640b30f0 0x7fa16415d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.297+0000 7fa16b000640 1 --2- 172.21.15.69:0/2378555509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa164156c60 0x7fa16415cac0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.297+0000 7fa16b000640 1 -- 172.21.15.69:0/2378555509 >> 172.21.15.69:0/2378555509 conn(0x7fa1640d0160 msgr2=0x7fa1640d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:39.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.297+0000 7fa16b000640 1 -- 172.21.15.69:0/2378555509 shutdown_connections 2024-04-24T08:09:39.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.297+0000 7fa16b000640 1 -- 172.21.15.69:0/2378555509 wait complete. 2024-04-24T08:09:39.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.299+0000 7fa16b000640 1 Processor -- start 2024-04-24T08:09:39.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.299+0000 7fa16b000640 1 -- start start 2024-04-24T08:09:39.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.300+0000 7fa16b000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1640ad0d0 0x7fa1641f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.300+0000 7fa16b000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1640b30f0 0x7fa164155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.300+0000 7fa16b000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa164156c60 0x7fa164143490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.300+0000 7fa16b000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa1641600e0 con 0x7fa1640ad0d0 2024-04-24T08:09:39.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.300+0000 7fa16b000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa16415d940 con 0x7fa164156c60 2024-04-24T08:09:39.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.300+0000 7fa16b000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa164160260 con 0x7fa1640b30f0 2024-04-24T08:09:39.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.300+0000 7fa169c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1640ad0d0 0x7fa1641f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.300+0000 7fa169c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1640ad0d0 0x7fa1641f42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60550/0 (socket says 172.21.15.69:60550) 2024-04-24T08:09:39.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.300+0000 7fa169c00640 1 -- 172.21.15.69:0/3934308893 learned_addr learned my addr 172.21.15.69:0/3934308893 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:39.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa169c00640 1 -- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1640b30f0 msgr2=0x7fa164155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa169200640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1640b30f0 0x7fa164155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa169c00640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1640b30f0 0x7fa164155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa169c00640 1 -- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa164156c60 msgr2=0x7fa164143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa169c00640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa164156c60 0x7fa164143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa169c00640 1 -- 172.21.15.69:0/3934308893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1640c9e50 con 0x7fa1640ad0d0 2024-04-24T08:09:39.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa169c00640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1640ad0d0 0x7fa1641f42f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa15802a220 tx=0x7fa1580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:39.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa161600640 1 -- 172.21.15.69:0/3934308893 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158041020 con 0x7fa1640ad0d0 2024-04-24T08:09:39.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa161600640 1 -- 172.21.15.69:0/3934308893 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1580306c0 con 0x7fa1640ad0d0 2024-04-24T08:09:39.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa16b000640 1 -- 172.21.15.69:0/3934308893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16415a9d0 con 0x7fa1640ad0d0 2024-04-24T08:09:39.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.301+0000 7fa161600640 1 -- 172.21.15.69:0/3934308893 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa158037040 con 0x7fa1640ad0d0 2024-04-24T08:09:39.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.302+0000 7fa16b000640 1 -- 172.21.15.69:0/3934308893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1640ca3b0 con 0x7fa1640ad0d0 2024-04-24T08:09:39.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.303+0000 7fa16b000640 1 -- 172.21.15.69:0/3934308893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa164075d50 con 0x7fa1640ad0d0 2024-04-24T08:09:39.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.304+0000 7fa161600640 1 -- 172.21.15.69:0/3934308893 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa158044020 con 0x7fa1640ad0d0 2024-04-24T08:09:39.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.305+0000 7fa161600640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa14406c8f0 0x7fa14406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.305+0000 7fa161600640 1 -- 172.21.15.69:0/3934308893 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fa1580ac3b0 con 0x7fa1640ad0d0 2024-04-24T08:09:39.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.305+0000 7fa169200640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa14406c8f0 0x7fa14406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.307+0000 7fa169200640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa14406c8f0 0x7fa14406edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa1640b01b0 tx=0x7fa150008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:39.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.309+0000 7fa161600640 1 -- 172.21.15.69:0/3934308893 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa15804b030 con 0x7fa1640ad0d0 2024-04-24T08:09:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.496+0000 7fa16b000640 1 -- 172.21.15.69:0/3934308893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa16406f9e0 con 0x7fa1640ad0d0 2024-04-24T08:09:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.499+0000 7fa161600640 1 -- 172.21.15.69:0/3934308893 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16658 (secure 0 0 0) 0x7fa1640c9e50 con 0x7fa1640ad0d0 2024-04-24T08:09:39.500 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:39.500 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":32,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:38.549313+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:09:36.197195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-24T08:09:37.506730+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.501+0000 7fa16b000640 1 -- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa14406c8f0 msgr2=0x7fa14406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.501+0000 7fa16b000640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa14406c8f0 0x7fa14406edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa1640b01b0 tx=0x7fa150008040 comp rx=0 tx=0).stop 2024-04-24T08:09:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.501+0000 7fa16b000640 1 -- 172.21.15.69:0/3934308893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1640ad0d0 msgr2=0x7fa1641f42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.501+0000 7fa16b000640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1640ad0d0 0x7fa1641f42f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa15802a220 tx=0x7fa1580046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.502+0000 7fa16b000640 1 -- 172.21.15.69:0/3934308893 shutdown_connections 2024-04-24T08:09:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.502+0000 7fa16b000640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa14406c8f0 0x7fa14406edb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.502+0000 7fa16b000640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa164156c60 0x7fa164143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.502+0000 7fa16b000640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1640b30f0 0x7fa164155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.502+0000 7fa16b000640 1 --2- 172.21.15.69:0/3934308893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa1640ad0d0 0x7fa1641f42f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.502+0000 7fa16b000640 1 -- 172.21.15.69:0/3934308893 >> 172.21.15.69:0/3934308893 conn(0x7fa1640d0160 msgr2=0x7fa164160560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:39.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.502+0000 7fa16b000640 1 -- 172.21.15.69:0/3934308893 shutdown_connections 2024-04-24T08:09:39.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.502+0000 7fa16b000640 1 -- 172.21.15.69:0/3934308893 wait complete. 2024-04-24T08:09:39.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:39.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.700+0000 7f1f56000640 1 Processor -- start 2024-04-24T08:09:39.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.700+0000 7f1f56000640 1 -- start start 2024-04-24T08:09:39.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.701+0000 7f1f56000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f50156c60 0x7f1f5015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.701+0000 7f1f56000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 0x7f1f5015d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.701+0000 7f1f56000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f5015d660 con 0x7f1f500ad0d0 2024-04-24T08:09:39.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.701+0000 7f1f56000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1f5006dc30 con 0x7f1f50156c60 2024-04-24T08:09:39.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.701+0000 7f1f56000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f5006df10 con 0x7f1f500b30f0 2024-04-24T08:09:39.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.701+0000 7f1f55600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1f500ad0d0 0x7f1f501600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53258/0 (socket says 172.21.15.69:53258) 2024-04-24T08:09:39.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.701+0000 7f1f55600640 1 -- 172.21.15.69:0/2055533079 learned_addr learned my addr 172.21.15.69:0/2055533079 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:39.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.701+0000 7f1f54c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f50156c60 0x7f1f5015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.702+0000 7f1f4fe00640 1 --2- 172.21.15.69:0/2055533079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 0x7f1f5015d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.702+0000 7f1f54c00640 1 -- 172.21.15.69:0/2055533079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 msgr2=0x7f1f5015d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:39.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.702+0000 7f1f54c00640 1 --2- 172.21.15.69:0/2055533079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 0x7f1f5015d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.702+0000 7f1f54c00640 1 -- 172.21.15.69:0/2055533079 >> v1:172.21.15.69:6789/0 conn(0x7f1f500ad0d0 legacy=0x7f1f501600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.702+0000 7f1f54c00640 1 -- 172.21.15.69:0/2055533079 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5006e1f0 con 0x7f1f50156c60 2024-04-24T08:09:39.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.702+0000 7f1f4fe00640 1 --2- 172.21.15.69:0/2055533079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 0x7f1f5015d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:39.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.702+0000 7f1f4f400640 1 -- 172.21.15.69:0/2055533079 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3529690004 0 0) 0x7f1f5015d660 con 0x7f1f500ad0d0 2024-04-24T08:09:39.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.703+0000 7f1f54c00640 1 --2- 172.21.15.69:0/2055533079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 0x7f1f5015cac0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009d90 tx=0x7f1f4002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4393198dced6de0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:39.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.703+0000 7f1f4f400640 1 -- 172.21.15.69:0/2055533079 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40007740 con 0x7f1f50156c60 2024-04-24T08:09:39.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.703+0000 7f1f4f400640 1 -- 172.21.15.69:0/2055533079 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f400078e0 con 0x7f1f50156c60 2024-04-24T08:09:39.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.703+0000 7f1f4f400640 1 -- 172.21.15.69:0/2055533079 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40007cb0 con 0x7f1f50156c60 2024-04-24T08:09:39.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.704+0000 7f1f56000640 1 -- 172.21.15.69:0/2055533079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 msgr2=0x7f1f5015cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.704+0000 7f1f56000640 1 --2- 172.21.15.69:0/2055533079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 0x7f1f5015cac0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1f40009d90 tx=0x7f1f4002a660 comp rx=0 tx=0).stop 2024-04-24T08:09:39.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.704+0000 7f1f56000640 1 -- 172.21.15.69:0/2055533079 shutdown_connections 2024-04-24T08:09:39.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.704+0000 7f1f56000640 1 --2- 172.21.15.69:0/2055533079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 0x7f1f5015d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.704+0000 7f1f56000640 1 --2- 172.21.15.69:0/2055533079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 0x7f1f5015cac0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.704+0000 7f1f56000640 1 -- 172.21.15.69:0/2055533079 >> 172.21.15.69:0/2055533079 conn(0x7f1f500d0160 msgr2=0x7f1f500d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:39.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.704+0000 7f1f56000640 1 -- 172.21.15.69:0/2055533079 shutdown_connections 2024-04-24T08:09:39.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.704+0000 7f1f56000640 1 -- 172.21.15.69:0/2055533079 wait complete. 2024-04-24T08:09:39.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.706+0000 7f1f56000640 1 Processor -- start 2024-04-24T08:09:39.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.706+0000 7f1f56000640 1 -- start start 2024-04-24T08:09:39.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f56000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ad0d0 0x7f1f50158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f56000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 0x7f1f501597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f56000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 0x7f1f501f7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f56000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1f500ad530 con 0x7f1f50156c60 2024-04-24T08:09:39.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f56000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5015d940 con 0x7f1f500ad0d0 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f56000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f500ad6b0 con 0x7f1f500b30f0 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f55600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 0x7f1f501f7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f55600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 0x7f1f501f7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60558/0 (socket says 172.21.15.69:60558) 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f55600640 1 -- 172.21.15.69:0/4046482973 learned_addr learned my addr 172.21.15.69:0/4046482973 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f54c00640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ad0d0 0x7f1f50158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f4fe00640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 0x7f1f501597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f55600640 1 -- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 msgr2=0x7f1f501597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f55600640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 0x7f1f501597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f55600640 1 -- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ad0d0 msgr2=0x7f1f50158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f55600640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ad0d0 0x7f1f50158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f55600640 1 -- 172.21.15.69:0/4046482973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f50156270 con 0x7f1f50156c60 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.707+0000 7f1f54c00640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ad0d0 0x7f1f50158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.708+0000 7f1f55600640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 0x7f1f501f7660 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800c480 tx=0x7f1f4800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:39.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.708+0000 7f1f4cc00640 1 -- 172.21.15.69:0/4046482973 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f48018070 con 0x7f1f50156c60 2024-04-24T08:09:39.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.708+0000 7f1f4cc00640 1 -- 172.21.15.69:0/4046482973 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f48007500 con 0x7f1f50156c60 2024-04-24T08:09:39.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.708+0000 7f1f4cc00640 1 -- 172.21.15.69:0/4046482973 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f48004a20 con 0x7f1f50156c60 2024-04-24T08:09:39.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.708+0000 7f1f56000640 1 -- 172.21.15.69:0/4046482973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f50158720 con 0x7f1f50156c60 2024-04-24T08:09:39.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.708+0000 7f1f56000640 1 -- 172.21.15.69:0/4046482973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f50158ce0 con 0x7f1f50156c60 2024-04-24T08:09:39.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.709+0000 7f1f56000640 1 -- 172.21.15.69:0/4046482973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f50075d50 con 0x7f1f50156c60 2024-04-24T08:09:39.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.710+0000 7f1f4cc00640 1 -- 172.21.15.69:0/4046482973 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f48004bc0 con 0x7f1f50156c60 2024-04-24T08:09:39.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.711+0000 7f1f4cc00640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f3006c8f0 0x7f1f3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.711+0000 7f1f4cc00640 1 -- 172.21.15.69:0/4046482973 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f1f4808ae10 con 0x7f1f50156c60 2024-04-24T08:09:39.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.712+0000 7f1f54c00640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f3006c8f0 0x7f1f3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.713+0000 7f1f54c00640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f3006c8f0 0x7f1f3006edb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1f40034000 tx=0x7f1f4002f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:39.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.715+0000 7f1f4cc00640 1 -- 172.21.15.69:0/4046482973 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f4808e030 con 0x7f1f50156c60 2024-04-24T08:09:39.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.916+0000 7f1f56000640 1 -- 172.21.15.69:0/4046482973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1f5006f9e0 con 0x7f1f50156c60 2024-04-24T08:09:39.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.918+0000 7f1f4cc00640 1 -- 172.21.15.69:0/4046482973 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16658 (secure 0 0 0) 0x7f1f48059230 con 0x7f1f50156c60 2024-04-24T08:09:39.919 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:39.919 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":32,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:38.549313+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:09:36.197195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-24T08:09:37.506730+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 -- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f3006c8f0 msgr2=0x7f1f3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f3006c8f0 0x7f1f3006edb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1f40034000 tx=0x7f1f4002f000 comp rx=0 tx=0).stop 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 -- 172.21.15.69:0/4046482973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 msgr2=0x7f1f501f7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 0x7f1f501f7660 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800c480 tx=0x7f1f4800c940 comp rx=0 tx=0).stop 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 -- 172.21.15.69:0/4046482973 shutdown_connections 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f3006c8f0 0x7f1f3006edb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f50156c60 0x7f1f501f7660 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f500b30f0 0x7f1f501597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 --2- 172.21.15.69:0/4046482973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f500ad0d0 0x7f1f50158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 -- 172.21.15.69:0/4046482973 >> 172.21.15.69:0/4046482973 conn(0x7f1f500d0160 msgr2=0x7f1f50160070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 -- 172.21.15.69:0/4046482973 shutdown_connections 2024-04-24T08:09:39.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:39.921+0000 7f1f56000640 1 -- 172.21.15.69:0/4046482973 wait complete. 2024-04-24T08:09:39.941 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T08:09:40.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.137+0000 7fd9c4000640 1 Processor -- start 2024-04-24T08:09:40.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.138+0000 7fd9c4000640 1 -- start start 2024-04-24T08:09:40.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.139+0000 7fd9c4000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9bc15f4b0 0x7fd9bc0b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:40.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.139+0000 7fd9c4000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9bc15f890 0x7fd9bc0732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:40.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.139+0000 7fd9c4000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9bc073cc0 con 0x7fd9bc073820 2024-04-24T08:09:40.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.139+0000 7fd9c4000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd9bc073fa0 con 0x7fd9bc15f4b0 2024-04-24T08:09:40.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.139+0000 7fd9c4000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9bc0b3300 con 0x7fd9bc15f890 2024-04-24T08:09:40.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.139+0000 7fd9c1600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9bc15f4b0 0x7fd9bc0b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:40.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.139+0000 7fd9c1600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9bc15f4b0 0x7fd9bc0b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60564/0 (socket says 172.21.15.69:60564) 2024-04-24T08:09:40.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.139+0000 7fd9c1600640 1 -- 172.21.15.69:0/1115090397 learned_addr learned my addr 172.21.15.69:0/1115090397 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:40.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.140+0000 7fd9c1600640 1 -- 172.21.15.69:0/1115090397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9bc15f890 msgr2=0x7fd9bc0732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:40.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.140+0000 7fd9c1600640 1 --2- 172.21.15.69:0/1115090397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9bc15f890 0x7fd9bc0732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:40.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.140+0000 7fd9c1600640 1 -- 172.21.15.69:0/1115090397 >> v1:172.21.15.121:6789/0 conn(0x7fd9bc073820 legacy=0x7fd9bc0b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:40.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.140+0000 7fd9c1600640 1 -- 172.21.15.69:0/1115090397 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9bc0b35e0 con 0x7fd9bc15f4b0 2024-04-24T08:09:40.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.141+0000 7fd9c1600640 1 --2- 172.21.15.69:0/1115090397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f4b0 0x7fd9bc0b3de0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac009870 tx=0x7fd9ac02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e0bde032222629 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:40.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.141+0000 7fd9bbe00640 1 -- 172.21.15.69:0/1115090397 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac037070 con 0x7fd9bc15f4b0 2024-04-24T08:09:40.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.141+0000 7fd9bbe00640 1 -- 172.21.15.69:0/1115090397 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9ac032440 con 0x7fd9bc15f4b0 2024-04-24T08:09:40.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.141+0000 7fd9bbe00640 1 -- 172.21.15.69:0/1115090397 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac032810 con 0x7fd9bc15f4b0 2024-04-24T08:09:40.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.142+0000 7fd9c4000640 1 -- 172.21.15.69:0/1115090397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f4b0 msgr2=0x7fd9bc0b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:40.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.142+0000 7fd9c4000640 1 --2- 172.21.15.69:0/1115090397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f4b0 0x7fd9bc0b3de0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac009870 tx=0x7fd9ac02a210 comp rx=0 tx=0).stop 2024-04-24T08:09:40.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.142+0000 7fd9c4000640 1 -- 172.21.15.69:0/1115090397 shutdown_connections 2024-04-24T08:09:40.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.142+0000 7fd9c4000640 1 --2- 172.21.15.69:0/1115090397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9bc15f890 0x7fd9bc0732e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:40.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.142+0000 7fd9c4000640 1 --2- 172.21.15.69:0/1115090397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f4b0 0x7fd9bc0b3de0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:40.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.142+0000 7fd9c4000640 1 -- 172.21.15.69:0/1115090397 >> 172.21.15.69:0/1115090397 conn(0x7fd9bc0d0350 msgr2=0x7fd9bc0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:40.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.142+0000 7fd9c4000640 1 -- 172.21.15.69:0/1115090397 shutdown_connections 2024-04-24T08:09:40.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.142+0000 7fd9c4000640 1 -- 172.21.15.69:0/1115090397 wait complete. 2024-04-24T08:09:40.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.143+0000 7fd9c4000640 1 Processor -- start 2024-04-24T08:09:40.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.144+0000 7fd9c4000640 1 -- start start 2024-04-24T08:09:40.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.144+0000 7fd9c4000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9bc073820 0x7fd9bc1548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:40.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.144+0000 7fd9c4000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9bc15f4b0 0x7fd9bc1e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.144+0000 7fd9c4000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f890 0x7fd9bc1e90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.144+0000 7fd9c4000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc15fe90 con 0x7fd9bc15f890 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.144+0000 7fd9c4000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc15fd10 con 0x7fd9bc15f4b0 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.144+0000 7fd9c4000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9bc0b4db0 con 0x7fd9bc073820 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.145+0000 7fd9c2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f890 0x7fd9bc1e90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.145+0000 7fd9c2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f890 0x7fd9bc1e90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60580/0 (socket says 172.21.15.69:60580) 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.145+0000 7fd9c2000640 1 -- 172.21.15.69:0/312929800 learned_addr learned my addr 172.21.15.69:0/312929800 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.145+0000 7fd9c2000640 1 -- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9bc073820 msgr2=0x7fd9bc1548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.145+0000 7fd9c2000640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9bc073820 0x7fd9bc1548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.145+0000 7fd9c2000640 1 -- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9bc15f4b0 msgr2=0x7fd9bc1e8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.145+0000 7fd9c2000640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9bc15f4b0 0x7fd9bc1e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.145+0000 7fd9c2000640 1 -- 172.21.15.69:0/312929800 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9bc154de0 con 0x7fd9bc15f890 2024-04-24T08:09:40.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.145+0000 7fd9c2000640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f890 0x7fd9bc1e90c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400b9c0 tx=0x7fd9b400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.146+0000 7fd9b9600640 1 -- 172.21.15.69:0/312929800 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4019070 con 0x7fd9bc15f890 2024-04-24T08:09:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.146+0000 7fd9b9600640 1 -- 172.21.15.69:0/312929800 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b400cba0 con 0x7fd9bc15f890 2024-04-24T08:09:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.146+0000 7fd9b9600640 1 -- 172.21.15.69:0/312929800 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4014440 con 0x7fd9bc15f890 2024-04-24T08:09:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.146+0000 7fd9c4000640 1 -- 172.21.15.69:0/312929800 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9bc06f3d0 con 0x7fd9bc15f890 2024-04-24T08:09:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.146+0000 7fd9c4000640 1 -- 172.21.15.69:0/312929800 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9bc06f990 con 0x7fd9bc15f890 2024-04-24T08:09:40.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.147+0000 7fd9c4000640 1 -- 172.21.15.69:0/312929800 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9bc068a80 con 0x7fd9bc15f890 2024-04-24T08:09:40.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.148+0000 7fd9b9600640 1 -- 172.21.15.69:0/312929800 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9b401c070 con 0x7fd9bc15f890 2024-04-24T08:09:40.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.149+0000 7fd9b9600640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99c06c8f0 0x7fd99c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:40.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.149+0000 7fd9b9600640 1 -- 172.21.15.69:0/312929800 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fd9b408b8f0 con 0x7fd9bc15f890 2024-04-24T08:09:40.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.150+0000 7fd9c1600640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99c06c8f0 0x7fd99c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:40.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.151+0000 7fd9c1600640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99c06c8f0 0x7fd99c06edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac034000 tx=0x7fd9ac02c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:40.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.154+0000 7fd9b9600640 1 -- 172.21.15.69:0/312929800 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9bc154de0 con 0x7fd9bc15f890 2024-04-24T08:09:40.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:40.389+0000 7fd9c4000640 1 -- 172.21.15.69:0/312929800 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd9bc0b4a20 con 0x7fd9bc15f890 2024-04-24T08:09:41.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.109+0000 7fd9b9600640 1 -- 172.21.15.69:0/312929800 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7fd9b4059d10 con 0x7fd9bc15f890 2024-04-24T08:09:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 -- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99c06c8f0 msgr2=0x7fd99c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99c06c8f0 0x7fd99c06edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac034000 tx=0x7fd9ac02c000 comp rx=0 tx=0).stop 2024-04-24T08:09:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 -- 172.21.15.69:0/312929800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f890 msgr2=0x7fd9bc1e90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f890 0x7fd9bc1e90c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400b9c0 tx=0x7fd9b400be80 comp rx=0 tx=0).stop 2024-04-24T08:09:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 -- 172.21.15.69:0/312929800 shutdown_connections 2024-04-24T08:09:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99c06c8f0 0x7fd99c06edb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9bc15f890 0x7fd9bc1e90c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9bc15f4b0 0x7fd9bc1e8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 --2- 172.21.15.69:0/312929800 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9bc073820 0x7fd9bc1548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 -- 172.21.15.69:0/312929800 >> 172.21.15.69:0/312929800 conn(0x7fd9bc0d0350 msgr2=0x7fd9bc160070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:41.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 -- 172.21.15.69:0/312929800 shutdown_connections 2024-04-24T08:09:41.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.112+0000 7fd9c4000640 1 -- 172.21.15.69:0/312929800 wait complete. 2024-04-24T08:09:41.134 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-24T08:09:41.134 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T08:09:41.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.326+0000 7f0063200640 1 Processor -- start 2024-04-24T08:09:41.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.327+0000 7f0063200640 1 -- start start 2024-04-24T08:09:41.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.328+0000 7f0063200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f006415e980 0x7f0064162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:41.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.328+0000 7f0063200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f006406cfb0 0x7f006406d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:41.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.328+0000 7f0063200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f006415ed60 con 0x7f00640d0da0 2024-04-24T08:09:41.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.328+0000 7f0063200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f00640d1580 con 0x7f006406cfb0 2024-04-24T08:09:41.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.328+0000 7f0063200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f00640d1860 con 0x7f006415e980 2024-04-24T08:09:41.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.328+0000 7f0062800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f006406cfb0 0x7f006406d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:41.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.328+0000 7f0061400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f00640d0da0 0x7f00640cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53268/0 (socket says 172.21.15.69:53268) 2024-04-24T08:09:41.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.328+0000 7f0061400640 1 -- 172.21.15.69:0/1280280848 learned_addr learned my addr 172.21.15.69:0/1280280848 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:41.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.329+0000 7f0061e00640 1 --2- 172.21.15.69:0/1280280848 >> v2:172.21.15.121:3300/0 conn(0x7f006415e980 0x7f0064162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:41.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.329+0000 7f0062800640 1 -- 172.21.15.69:0/1280280848 >> v2:172.21.15.121:3300/0 conn(0x7f006415e980 msgr2=0x7f0064162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:41.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.329+0000 7f0062800640 1 --2- 172.21.15.69:0/1280280848 >> v2:172.21.15.121:3300/0 conn(0x7f006415e980 0x7f0064162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:41.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.329+0000 7f0062800640 1 -- 172.21.15.69:0/1280280848 >> v1:172.21.15.69:6789/0 conn(0x7f00640d0da0 legacy=0x7f00640cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:41.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.329+0000 7f0062800640 1 -- 172.21.15.69:0/1280280848 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00640cf450 con 0x7f006406cfb0 2024-04-24T08:09:41.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.329+0000 7f0060a00640 1 -- 172.21.15.69:0/1280280848 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3478394380 0 0) 0x7f006415ed60 con 0x7f00640d0da0 2024-04-24T08:09:41.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.329+0000 7f0061e00640 1 --2- 172.21.15.69:0/1280280848 >> v2:172.21.15.121:3300/0 conn(0x7f006415e980 0x7f0064162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:41.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.330+0000 7f0062800640 1 --2- 172.21.15.69:0/1280280848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f006406cfb0 0x7f006406d440 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f006409de50 tx=0x7f005c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea253c0c3c8141c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:41.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.330+0000 7f0060a00640 1 -- 172.21.15.69:0/1280280848 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005c037070 con 0x7f006406cfb0 2024-04-24T08:09:41.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.330+0000 7f0060a00640 1 -- 172.21.15.69:0/1280280848 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f005c032440 con 0x7f006406cfb0 2024-04-24T08:09:41.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.330+0000 7f0060a00640 1 -- 172.21.15.69:0/1280280848 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005c032810 con 0x7f006406cfb0 2024-04-24T08:09:41.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.331+0000 7f0063200640 1 -- 172.21.15.69:0/1280280848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f006406cfb0 msgr2=0x7f006406d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:41.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.331+0000 7f0063200640 1 --2- 172.21.15.69:0/1280280848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f006406cfb0 0x7f006406d440 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f006409de50 tx=0x7f005c02a230 comp rx=0 tx=0).stop 2024-04-24T08:09:41.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.331+0000 7f0063200640 1 -- 172.21.15.69:0/1280280848 shutdown_connections 2024-04-24T08:09:41.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.331+0000 7f0063200640 1 --2- 172.21.15.69:0/1280280848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f006406cfb0 0x7f006406d440 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:41.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.331+0000 7f0063200640 1 --2- 172.21.15.69:0/1280280848 >> v2:172.21.15.121:3300/0 conn(0x7f006415e980 0x7f0064162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:41.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.331+0000 7f0063200640 1 -- 172.21.15.69:0/1280280848 >> 172.21.15.69:0/1280280848 conn(0x7f006409d920 msgr2=0x7f006408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:41.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.331+0000 7f0063200640 1 -- 172.21.15.69:0/1280280848 shutdown_connections 2024-04-24T08:09:41.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.331+0000 7f0063200640 1 -- 172.21.15.69:0/1280280848 wait complete. 2024-04-24T08:09:41.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.333+0000 7f0063200640 1 Processor -- start 2024-04-24T08:09:41.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.333+0000 7f0063200640 1 -- start start 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0063200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f006406cfb0 0x7f00641615d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0063200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00640d0da0 0x7f00640b9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0063200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f006415e980 0x7f00640b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0063200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00640d11e0 con 0x7f00640d0da0 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0063200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f00640d0360 con 0x7f006406cfb0 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0063200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f00640d1360 con 0x7f006415e980 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0061400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00640d0da0 0x7f00640b9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0061400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00640d0da0 0x7f00640b9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60598/0 (socket says 172.21.15.69:60598) 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0061400640 1 -- 172.21.15.69:0/160408119 learned_addr learned my addr 172.21.15.69:0/160408119 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0061e00640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f006406cfb0 0x7f00641615d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.334+0000 7f0061400640 1 -- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f006415e980 msgr2=0x7f00640b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:41.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f0061400640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f006415e980 0x7f00640b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:41.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f0061400640 1 -- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f006406cfb0 msgr2=0x7f00641615d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:41.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f0061400640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f006406cfb0 0x7f00641615d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:41.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f0061400640 1 -- 172.21.15.69:0/160408119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00640b8380 con 0x7f00640d0da0 2024-04-24T08:09:41.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f0061e00640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f006406cfb0 0x7f00641615d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:41.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f0061400640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00640d0da0 0x7f00640b9800 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f004c00c890 tx=0x7f004c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:41.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f005a000640 1 -- 172.21.15.69:0/160408119 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c018070 con 0x7f00640d0da0 2024-04-24T08:09:41.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f005a000640 1 -- 172.21.15.69:0/160408119 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f004c00a640 con 0x7f00640d0da0 2024-04-24T08:09:41.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f0063200640 1 -- 172.21.15.69:0/160408119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00640cabf0 con 0x7f00640d0da0 2024-04-24T08:09:41.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f005a000640 1 -- 172.21.15.69:0/160408119 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f004c00aa30 con 0x7f00640d0da0 2024-04-24T08:09:41.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.335+0000 7f0063200640 1 -- 172.21.15.69:0/160408119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00640ca110 con 0x7f00640d0da0 2024-04-24T08:09:41.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.336+0000 7f0063200640 1 -- 172.21.15.69:0/160408119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0064075d50 con 0x7f00640d0da0 2024-04-24T08:09:41.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.337+0000 7f005a000640 1 -- 172.21.15.69:0/160408119 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f004c01b070 con 0x7f00640d0da0 2024-04-24T08:09:41.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.338+0000 7f005a000640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004006c8f0 0x7f004006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:41.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.338+0000 7f005a000640 1 -- 172.21.15.69:0/160408119 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f004c08b250 con 0x7f00640d0da0 2024-04-24T08:09:41.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.339+0000 7f0061e00640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004006c8f0 0x7f004006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:41.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.340+0000 7f0061e00640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004006c8f0 0x7f004006edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0054006fd0 tx=0x7f0054008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:41.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.343+0000 7f005a000640 1 -- 172.21.15.69:0/160408119 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f004c059670 con 0x7f00640d0da0 2024-04-24T08:09:41.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:41.565+0000 7f0063200640 1 -- 172.21.15.69:0/160408119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f00640d1e40 con 0x7f00640d0da0 2024-04-24T08:09:42.040 INFO:tasks.ceph.mds.c.smithi069.stderr:2024-04-24T08:09:42.039+0000 7fb31ec00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:09:42.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.128+0000 7f005a000640 1 -- 172.21.15.69:0/160408119 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f004c058f70 con 0x7f00640d0da0 2024-04-24T08:09:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 -- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004006c8f0 msgr2=0x7f004006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004006c8f0 0x7f004006edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0054006fd0 tx=0x7f0054008040 comp rx=0 tx=0).stop 2024-04-24T08:09:42.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 -- 172.21.15.69:0/160408119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00640d0da0 msgr2=0x7f00640b9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00640d0da0 0x7f00640b9800 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f004c00c890 tx=0x7f004c00cd50 comp rx=0 tx=0).stop 2024-04-24T08:09:42.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 -- 172.21.15.69:0/160408119 shutdown_connections 2024-04-24T08:09:42.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004006c8f0 0x7f004006edb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00640d0da0 0x7f00640b9800 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f006415e980 0x7f00640b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 --2- 172.21.15.69:0/160408119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f006406cfb0 0x7f00641615d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 -- 172.21.15.69:0/160408119 >> 172.21.15.69:0/160408119 conn(0x7f006409d920 msgr2=0x7f006408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:42.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 -- 172.21.15.69:0/160408119 shutdown_connections 2024-04-24T08:09:42.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.132+0000 7f0063200640 1 -- 172.21.15.69:0/160408119 wait complete. 2024-04-24T08:09:42.155 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T08:09:42.155 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:09:42.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.357+0000 7f66ad000640 1 Processor -- start 2024-04-24T08:09:42.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.357+0000 7f66ad000640 1 -- start start 2024-04-24T08:09:42.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.358+0000 7f66ad000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a806db90 0x7f66a8073530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.358+0000 7f66ad000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f66a815b980 0x7f66a815be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.358+0000 7f66ad000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66a8073a70 con 0x7f66a81523f0 2024-04-24T08:09:42.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.358+0000 7f66ad000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f66a80aeac0 con 0x7f66a815b980 2024-04-24T08:09:42.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.358+0000 7f66ad000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f66a80aeda0 con 0x7f66a806db90 2024-04-24T08:09:42.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.359+0000 7f66a7400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f66a81523f0 0x7f66a8072e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53274/0 (socket says 172.21.15.69:53274) 2024-04-24T08:09:42.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.359+0000 7f66a7400640 1 -- 172.21.15.69:0/2075500649 learned_addr learned my addr 172.21.15.69:0/2075500649 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:42.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.359+0000 7f66a6a00640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a806db90 0x7f66a8073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.359+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1990403304 0 0) 0x7f66a8073a70 con 0x7f66a81523f0 2024-04-24T08:09:42.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.359+0000 7f66a7e00640 1 --2- 172.21.15.69:0/2075500649 >> v2:172.21.15.121:3300/0 conn(0x7f66a815b980 0x7f66a815be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.359+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f668c003520 con 0x7f66a81523f0 2024-04-24T08:09:42.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1942828791 0 0) 0x7f668c003520 con 0x7f66a81523f0 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a806db90 msgr2=0x7f66a8073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a806db90 0x7f66a8073530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 >> v2:172.21.15.121:3300/0 conn(0x7f66a815b980 msgr2=0x7f66a815be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 --2- 172.21.15.69:0/2075500649 >> v2:172.21.15.121:3300/0 conn(0x7f66a815b980 0x7f66a815be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a80af080 con 0x7f66a81523f0 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f6698001670 con 0x7f66a81523f0 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 >> v1:172.21.15.69:6789/0 conn(0x7f66a81523f0 legacy=0x7f66a8072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a806db90 0x7f668c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c003f80 0x7f668c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c004d40 0x7f668c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f66a80aeac0 con 0x7f668c004d40 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f66a80aeda0 con 0x7f668c003f80 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f668c00ff30 con 0x7f66a806db90 2024-04-24T08:09:42.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a7e00640 1 --2- 172.21.15.69:0/2075500649 >> v2:172.21.15.121:3300/0 conn(0x7f66a815b980 0x7f66a815be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:09:42.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a7e00640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c004d40 0x7f668c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.360+0000 7f66a7400640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a806db90 0x7f668c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.361+0000 7f66a6a00640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c003f80 0x7f668c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.361+0000 7f66a7e00640 1 -- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a806db90 msgr2=0x7f668c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:42.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.361+0000 7f66a7e00640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a806db90 0x7f668c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.361+0000 7f66a7e00640 1 -- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c003f80 msgr2=0x7f668c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:42.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.361+0000 7f66a7e00640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c003f80 0x7f668c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.361+0000 7f66a7e00640 1 -- 172.21.15.69:0/2075500649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668c003520 con 0x7f668c004d40 2024-04-24T08:09:42.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.361+0000 7f66a6a00640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c003f80 0x7f668c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:09:42.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.362+0000 7f66a7e00640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c004d40 0x7f668c0051c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f66a00073e0 tx=0x7f668c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dac7359d2e1ccb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:42.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.363+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0009c70 con 0x7f668c004d40 2024-04-24T08:09:42.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.363+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a002c680 con 0x7f668c004d40 2024-04-24T08:09:42.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.363+0000 7f66a6000640 1 -- 172.21.15.69:0/2075500649 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a002ca30 con 0x7f668c004d40 2024-04-24T08:09:42.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.363+0000 7f66ad000640 1 -- 172.21.15.69:0/2075500649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c004d40 msgr2=0x7f668c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.363+0000 7f66ad000640 1 --2- 172.21.15.69:0/2075500649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c004d40 0x7f668c0051c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f66a00073e0 tx=0x7f668c005830 comp rx=0 tx=0).stop 2024-04-24T08:09:42.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.363+0000 7f66a7e00640 1 -- 172.21.15.69:0/2075500649 reap_dead start 2024-04-24T08:09:42.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.363+0000 7f66ad000640 1 -- 172.21.15.69:0/2075500649 shutdown_connections 2024-04-24T08:09:42.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.363+0000 7f66ad000640 1 -- 172.21.15.69:0/2075500649 >> 172.21.15.69:0/2075500649 conn(0x7f66a80d0160 msgr2=0x7f66a80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:42.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.363+0000 7f66ad000640 1 -- 172.21.15.69:0/2075500649 shutdown_connections 2024-04-24T08:09:42.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.363+0000 7f66ad000640 1 -- 172.21.15.69:0/2075500649 wait complete. 2024-04-24T08:09:42.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.365+0000 7f66ad000640 1 Processor -- start 2024-04-24T08:09:42.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.365+0000 7f66ad000640 1 -- start start 2024-04-24T08:09:42.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.366+0000 7f66ad000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66a81efc60 0x7f66a81428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.366+0000 7f66ad000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66a81f0040 0x7f66a8140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.366+0000 7f66ad000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a809b100 0x7f66a80b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.366+0000 7f66ad000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f66a806dfd0 con 0x7f66a81efc60 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.366+0000 7f66ad000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f66a80b3190 con 0x7f66a81f0040 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.366+0000 7f66ad000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f66a81527d0 con 0x7f66a809b100 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66a81efc60 0x7f66a81428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66a81efc60 0x7f66a81428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60622/0 (socket says 172.21.15.69:60622) 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a7400640 1 -- 172.21.15.69:0/723336066 learned_addr learned my addr 172.21.15.69:0/723336066 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a7e00640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a809b100 0x7f66a80b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a6a00640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66a81f0040 0x7f66a8140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a7400640 1 -- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a809b100 msgr2=0x7f66a80b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a7400640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a809b100 0x7f66a80b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a7400640 1 -- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66a81f0040 msgr2=0x7f66a8140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a7400640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66a81f0040 0x7f66a8140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a7400640 1 -- 172.21.15.69:0/723336066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66a8141810 con 0x7f66a81efc60 2024-04-24T08:09:42.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.367+0000 7f66a7400640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66a81efc60 0x7f66a81428b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f66980072f0 tx=0x7f66980077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:42.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.368+0000 7f669f400640 1 -- 172.21.15.69:0/723336066 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669801a070 con 0x7f66a81efc60 2024-04-24T08:09:42.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.368+0000 7f669f400640 1 -- 172.21.15.69:0/723336066 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f669800f960 con 0x7f66a81efc60 2024-04-24T08:09:42.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.368+0000 7f66ad000640 1 -- 172.21.15.69:0/723336066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66a8141af0 con 0x7f66a81efc60 2024-04-24T08:09:42.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.368+0000 7f669f400640 1 -- 172.21.15.69:0/723336066 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f669800fd30 con 0x7f66a81efc60 2024-04-24T08:09:42.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.368+0000 7f66ad000640 1 -- 172.21.15.69:0/723336066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66a81420b0 con 0x7f66a81efc60 2024-04-24T08:09:42.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.369+0000 7f66ad000640 1 -- 172.21.15.69:0/723336066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66a8075d50 con 0x7f66a81efc60 2024-04-24T08:09:42.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.370+0000 7f669f400640 1 -- 172.21.15.69:0/723336066 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f669801d070 con 0x7f66a81efc60 2024-04-24T08:09:42.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.371+0000 7f669f400640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f668806c8f0 0x7f668806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.371+0000 7f669f400640 1 -- 172.21.15.69:0/723336066 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f669808b6a0 con 0x7f66a81efc60 2024-04-24T08:09:42.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.372+0000 7f66a6a00640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f668806c8f0 0x7f668806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.373+0000 7f66a6a00640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f668806c8f0 0x7f668806edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f66a80b19f0 tx=0x7f6694008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:42.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.375+0000 7f669f400640 1 -- 172.21.15.69:0/723336066 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6698059a30 con 0x7f66a81efc60 2024-04-24T08:09:42.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.622+0000 7f66ad000640 1 -- 172.21.15.69:0/723336066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66a8158cb0 con 0x7f66a81efc60 2024-04-24T08:09:42.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.623+0000 7f669f400640 1 -- 172.21.15.69:0/723336066 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5181 (secure 0 0 0) 0x7f6698059330 con 0x7f66a81efc60 2024-04-24T08:09:42.623 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:42.623 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2295273533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2295273533},{"type":"v1","addr":"172.21.15.121:6836","nonce":2295273533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1461471292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1461471292},{"type":"v1","addr":"172.21.15.121:6837","nonce":1461471292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/626688145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":626688145},{"type":"v1","addr":"172.21.15.69:6833","nonce":626688145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:09:38.549277+0000","modified":"2024-04-24T08:09:41.567377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/991803145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":991803145},{"type":"v1","addr":"172.21.15.69:6835","nonce":991803145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5241,"qdb_cluster":[5241]},"id":2}]} 2024-04-24T08:09:42.624 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 16 2024-04-24T08:09:42.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.625+0000 7f66ad000640 1 -- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f668806c8f0 msgr2=0x7f668806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.625+0000 7f66ad000640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f668806c8f0 0x7f668806edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f66a80b19f0 tx=0x7f6694008040 comp rx=0 tx=0).stop 2024-04-24T08:09:42.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.625+0000 7f66ad000640 1 -- 172.21.15.69:0/723336066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66a81efc60 msgr2=0x7f66a81428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.625+0000 7f66ad000640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66a81efc60 0x7f66a81428b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f66980072f0 tx=0x7f66980077b0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.625+0000 7f66ad000640 1 -- 172.21.15.69:0/723336066 shutdown_connections 2024-04-24T08:09:42.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.625+0000 7f66ad000640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f668806c8f0 0x7f668806edb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.625+0000 7f66ad000640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66a809b100 0x7f66a80b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.626+0000 7f66ad000640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66a81f0040 0x7f66a8140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.626+0000 7f66ad000640 1 --2- 172.21.15.69:0/723336066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66a81efc60 0x7f66a81428b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.626+0000 7f66ad000640 1 -- 172.21.15.69:0/723336066 >> 172.21.15.69:0/723336066 conn(0x7f66a80d0160 msgr2=0x7f66a80addf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:42.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.626+0000 7f66ad000640 1 -- 172.21.15.69:0/723336066 shutdown_connections 2024-04-24T08:09:42.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.626+0000 7f66ad000640 1 -- 172.21.15.69:0/723336066 wait complete. 2024-04-24T08:09:42.644 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:42.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.835+0000 7f4981e00640 1 Processor -- start 2024-04-24T08:09:42.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.835+0000 7f4981e00640 1 -- start start 2024-04-24T08:09:42.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.837+0000 7f4981e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f497c145a90 0x7f497c06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.837+0000 7f4981e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f497c145fe0 0x7f497c160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.837+0000 7f4981e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f497c0cf3d0 con 0x7f497c0d0300 2024-04-24T08:09:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.837+0000 7f4981e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f497c0cf6b0 con 0x7f497c145fe0 2024-04-24T08:09:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.837+0000 7f4981e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f497c0cf990 con 0x7f497c145a90 2024-04-24T08:09:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.837+0000 7f497b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f497c145fe0 0x7f497c160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.837+0000 7f497b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f497c145fe0 0x7f497c160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60638/0 (socket says 172.21.15.69:60638) 2024-04-24T08:09:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.837+0000 7f497b400640 1 -- 172.21.15.69:0/755193769 learned_addr learned my addr 172.21.15.69:0/755193769 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:42.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.838+0000 7f497a000640 1 --2- 172.21.15.69:0/755193769 >> v2:172.21.15.121:3300/0 conn(0x7f497c145a90 0x7f497c06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.838+0000 7f497b400640 1 -- 172.21.15.69:0/755193769 >> v2:172.21.15.121:3300/0 conn(0x7f497c145a90 msgr2=0x7f497c06e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.838+0000 7f497b400640 1 --2- 172.21.15.69:0/755193769 >> v2:172.21.15.121:3300/0 conn(0x7f497c145a90 0x7f497c06e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.838+0000 7f497b400640 1 -- 172.21.15.69:0/755193769 >> v1:172.21.15.121:6789/0 conn(0x7f497c0d0300 legacy=0x7f497c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.838+0000 7f497b400640 1 -- 172.21.15.69:0/755193769 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c0cfc70 con 0x7f497c145fe0 2024-04-24T08:09:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.839+0000 7f497b400640 1 --2- 172.21.15.69:0/755193769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c145fe0 0x7f497c160540 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f497c0b5780 tx=0x7f497402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdbe4c7dba2a6cd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.839+0000 7f4979600640 1 -- 172.21.15.69:0/755193769 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974036070 con 0x7f497c145fe0 2024-04-24T08:09:42.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.839+0000 7f4979600640 1 -- 172.21.15.69:0/755193769 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4974031600 con 0x7f497c145fe0 2024-04-24T08:09:42.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.839+0000 7f4979600640 1 -- 172.21.15.69:0/755193769 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49740319d0 con 0x7f497c145fe0 2024-04-24T08:09:42.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.840+0000 7f4981e00640 1 -- 172.21.15.69:0/755193769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c145fe0 msgr2=0x7f497c160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.840+0000 7f4981e00640 1 --2- 172.21.15.69:0/755193769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c145fe0 0x7f497c160540 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f497c0b5780 tx=0x7f497402a6f0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.840+0000 7f4981e00640 1 -- 172.21.15.69:0/755193769 shutdown_connections 2024-04-24T08:09:42.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.840+0000 7f4981e00640 1 --2- 172.21.15.69:0/755193769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c145fe0 0x7f497c160540 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.840+0000 7f4981e00640 1 --2- 172.21.15.69:0/755193769 >> v2:172.21.15.121:3300/0 conn(0x7f497c145a90 0x7f497c06e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.840+0000 7f4981e00640 1 -- 172.21.15.69:0/755193769 >> 172.21.15.69:0/755193769 conn(0x7f497c067d90 msgr2=0x7f497c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:42.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.840+0000 7f4981e00640 1 -- 172.21.15.69:0/755193769 shutdown_connections 2024-04-24T08:09:42.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.840+0000 7f4981e00640 1 -- 172.21.15.69:0/755193769 wait complete. 2024-04-24T08:09:42.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.842+0000 7f4981e00640 1 Processor -- start 2024-04-24T08:09:42.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.842+0000 7f4981e00640 1 -- start start 2024-04-24T08:09:42.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f4981e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c0d0300 0x7f497c1415e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f4981e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f497c145a90 0x7f497c141b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f4981e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f497c145fe0 0x7f497c0b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f4981e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f497c1458f0 con 0x7f497c0d0300 2024-04-24T08:09:42.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f4981e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f497c145770 con 0x7f497c145fe0 2024-04-24T08:09:42.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f4981e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f497c0cdc30 con 0x7f497c145a90 2024-04-24T08:09:42.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f497aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c0d0300 0x7f497c1415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f497aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c0d0300 0x7f497c1415e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60640/0 (socket says 172.21.15.69:60640) 2024-04-24T08:09:42.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f497aa00640 1 -- 172.21.15.69:0/1207471127 learned_addr learned my addr 172.21.15.69:0/1207471127 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:42.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f497aa00640 1 -- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f497c145a90 msgr2=0x7f497c141b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.843+0000 7f497a000640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f497c145a90 0x7f497c141b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.844+0000 7f497aa00640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f497c145a90 0x7f497c141b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.844+0000 7f497aa00640 1 -- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f497c145fe0 msgr2=0x7f497c0b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:42.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.844+0000 7f497aa00640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f497c145fe0 0x7f497c0b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:42.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.844+0000 7f497aa00640 1 -- 172.21.15.69:0/1207471127 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f497c0b4ac0 con 0x7f497c0d0300 2024-04-24T08:09:42.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.844+0000 7f497aa00640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c0d0300 0x7f497c1415e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f496c00b3e0 tx=0x7f496c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:42.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.844+0000 7f4973400640 1 -- 172.21.15.69:0/1207471127 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c015020 con 0x7f497c0d0300 2024-04-24T08:09:42.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.844+0000 7f4973400640 1 -- 172.21.15.69:0/1207471127 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f496c011850 con 0x7f497c0d0300 2024-04-24T08:09:42.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.844+0000 7f4981e00640 1 -- 172.21.15.69:0/1207471127 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f497c0b4da0 con 0x7f497c0d0300 2024-04-24T08:09:42.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.844+0000 7f4973400640 1 -- 172.21.15.69:0/1207471127 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c011c40 con 0x7f497c0d0300 2024-04-24T08:09:42.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.844+0000 7f4981e00640 1 -- 172.21.15.69:0/1207471127 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f497c0b5360 con 0x7f497c0d0300 2024-04-24T08:09:42.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.846+0000 7f4981e00640 1 -- 172.21.15.69:0/1207471127 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f497c068a80 con 0x7f497c0d0300 2024-04-24T08:09:42.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.847+0000 7f4973400640 1 -- 172.21.15.69:0/1207471127 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f496c018020 con 0x7f497c0d0300 2024-04-24T08:09:42.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.847+0000 7f4973400640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f495806c8f0 0x7f495806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:42.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.848+0000 7f4973400640 1 -- 172.21.15.69:0/1207471127 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f496c08bb20 con 0x7f497c0d0300 2024-04-24T08:09:42.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.848+0000 7f497a000640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f495806c8f0 0x7f495806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:42.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.849+0000 7f497a000640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f495806c8f0 0x7f495806edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4964006fd0 tx=0x7f4964008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:42.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:42.852+0000 7f4973400640 1 -- 172.21.15.69:0/1207471127 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f496c01f020 con 0x7f497c0d0300 2024-04-24T08:09:43.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.059+0000 7f4981e00640 1 -- 172.21.15.69:0/1207471127 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f497c06bb50 con 0x7f497c0d0300 2024-04-24T08:09:43.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.061+0000 7f4973400640 1 -- 172.21.15.69:0/1207471127 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16645 (secure 0 0 0) 0x7f496c059f40 con 0x7f497c0d0300 2024-04-24T08:09:43.062 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:43.062 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":33,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:40.096668+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:09:36.197195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-24T08:09:37.506730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:43.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.064+0000 7f4981e00640 1 -- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f495806c8f0 msgr2=0x7f495806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.064+0000 7f4981e00640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f495806c8f0 0x7f495806edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4964006fd0 tx=0x7f4964008040 comp rx=0 tx=0).stop 2024-04-24T08:09:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.064+0000 7f4981e00640 1 -- 172.21.15.69:0/1207471127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c0d0300 msgr2=0x7f497c1415e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.064+0000 7f4981e00640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c0d0300 0x7f497c1415e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f496c00b3e0 tx=0x7f496c00b8a0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.065+0000 7f4981e00640 1 -- 172.21.15.69:0/1207471127 shutdown_connections 2024-04-24T08:09:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.065+0000 7f4981e00640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f495806c8f0 0x7f495806edb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.065+0000 7f4981e00640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f497c145fe0 0x7f497c0b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.065+0000 7f4981e00640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f497c145a90 0x7f497c141b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.065+0000 7f4981e00640 1 --2- 172.21.15.69:0/1207471127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f497c0d0300 0x7f497c1415e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.065+0000 7f4981e00640 1 -- 172.21.15.69:0/1207471127 >> 172.21.15.69:0/1207471127 conn(0x7f497c067d90 msgr2=0x7f497c0d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.065+0000 7f4981e00640 1 -- 172.21.15.69:0/1207471127 shutdown_connections 2024-04-24T08:09:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.065+0000 7f4981e00640 1 -- 172.21.15.69:0/1207471127 wait complete. 2024-04-24T08:09:43.083 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T08:09:43.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:09:43.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.287+0000 7f120f400640 1 Processor -- start 2024-04-24T08:09:43.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.287+0000 7f120f400640 1 -- start start 2024-04-24T08:09:43.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.289+0000 7f120f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1208162150 0x7f12080d5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.289+0000 7f120f400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12080cfe40 0x7f12080b7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.289+0000 7f120f400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f12080b7a90 0x7f12080d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.289+0000 7f120f400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f12080d2900 con 0x7f1208162150 2024-04-24T08:09:43.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.289+0000 7f120f400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f12080d22d0 con 0x7f12080b7a90 2024-04-24T08:09:43.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.289+0000 7f120f400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12080d03b0 con 0x7f12080cfe40 2024-04-24T08:09:43.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.289+0000 7f120ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1208162150 0x7f12080d5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.290+0000 7f120ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1208162150 0x7f12080d5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60654/0 (socket says 172.21.15.69:60654) 2024-04-24T08:09:43.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.290+0000 7f120ca00640 1 -- 172.21.15.69:0/3840956643 learned_addr learned my addr 172.21.15.69:0/3840956643 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:43.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.290+0000 7f120d400640 1 --2- 172.21.15.69:0/3840956643 >> v2:172.21.15.121:3300/0 conn(0x7f12080b7a90 0x7f12080d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.290+0000 7f1207e00640 1 --2- 172.21.15.69:0/3840956643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12080cfe40 0x7f12080b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.290+0000 7f120ca00640 1 -- 172.21.15.69:0/3840956643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12080cfe40 msgr2=0x7f12080b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:43.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.290+0000 7f120ca00640 1 --2- 172.21.15.69:0/3840956643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12080cfe40 0x7f12080b7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.290+0000 7f120ca00640 1 -- 172.21.15.69:0/3840956643 >> v2:172.21.15.121:3300/0 conn(0x7f12080b7a90 msgr2=0x7f12080d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:43.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.290+0000 7f120ca00640 1 --2- 172.21.15.69:0/3840956643 >> v2:172.21.15.121:3300/0 conn(0x7f12080b7a90 0x7f12080d10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.290+0000 7f120ca00640 1 -- 172.21.15.69:0/3840956643 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12080d0690 con 0x7f1208162150 2024-04-24T08:09:43.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.292+0000 7f120ca00640 1 --2- 172.21.15.69:0/3840956643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 0x7f12080d5020 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1208074170 tx=0x7f11fc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976555549df74ec3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:43.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.292+0000 7f1207400640 1 -- 172.21.15.69:0/3840956643 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc004030 con 0x7f1208162150 2024-04-24T08:09:43.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.292+0000 7f1207400640 1 -- 172.21.15.69:0/3840956643 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11fc0041d0 con 0x7f1208162150 2024-04-24T08:09:43.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.292+0000 7f1207400640 1 -- 172.21.15.69:0/3840956643 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11fc003610 con 0x7f1208162150 2024-04-24T08:09:43.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.293+0000 7f120f400640 1 -- 172.21.15.69:0/3840956643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 msgr2=0x7f12080d5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.293+0000 7f120f400640 1 --2- 172.21.15.69:0/3840956643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 0x7f12080d5020 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f1208074170 tx=0x7f11fc02a1c0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.293+0000 7f120f400640 1 -- 172.21.15.69:0/3840956643 shutdown_connections 2024-04-24T08:09:43.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.293+0000 7f120f400640 1 --2- 172.21.15.69:0/3840956643 >> v2:172.21.15.121:3300/0 conn(0x7f12080b7a90 0x7f12080d10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.293+0000 7f120f400640 1 --2- 172.21.15.69:0/3840956643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12080cfe40 0x7f12080b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.293+0000 7f120f400640 1 --2- 172.21.15.69:0/3840956643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 0x7f12080d5020 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.293+0000 7f120f400640 1 -- 172.21.15.69:0/3840956643 >> 172.21.15.69:0/3840956643 conn(0x7f120808bf60 msgr2=0x7f12080d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:43.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.293+0000 7f120f400640 1 -- 172.21.15.69:0/3840956643 shutdown_connections 2024-04-24T08:09:43.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.293+0000 7f120f400640 1 -- 172.21.15.69:0/3840956643 wait complete. 2024-04-24T08:09:43.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.295+0000 7f120f400640 1 Processor -- start 2024-04-24T08:09:43.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.296+0000 7f120f400640 1 -- start start 2024-04-24T08:09:43.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f120f400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12080b7a90 0x7f1208153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f120f400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12080cfe40 0x7f120815e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f120f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 0x7f120815eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f120f400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12080d1630 con 0x7f1208162150 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f120f400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f12080d5560 con 0x7f12080cfe40 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f120f400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f12080d17b0 con 0x7f12080b7a90 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f120d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 0x7f120815eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f120d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 0x7f120815eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60668/0 (socket says 172.21.15.69:60668) 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f120d400640 1 -- 172.21.15.69:0/986789515 learned_addr learned my addr 172.21.15.69:0/986789515 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f120ca00640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12080b7a90 0x7f1208153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.297+0000 7f1207e00640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12080cfe40 0x7f120815e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f120d400640 1 -- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12080b7a90 msgr2=0x7f1208153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f120d400640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12080b7a90 0x7f1208153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f120d400640 1 -- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12080cfe40 msgr2=0x7f120815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f120d400640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12080cfe40 0x7f120815e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f120d400640 1 -- 172.21.15.69:0/986789515 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120815e320 con 0x7f1208162150 2024-04-24T08:09:43.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f1207e00640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12080cfe40 0x7f120815e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:43.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f120d400640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 0x7f120815eec0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f120000ef90 tx=0x7f120000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:43.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f1204c00640 1 -- 172.21.15.69:0/986789515 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200018020 con 0x7f1208162150 2024-04-24T08:09:43.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f1204c00640 1 -- 172.21.15.69:0/986789515 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1200014480 con 0x7f1208162150 2024-04-24T08:09:43.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f1204c00640 1 -- 172.21.15.69:0/986789515 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1200014870 con 0x7f1208162150 2024-04-24T08:09:43.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f120f400640 1 -- 172.21.15.69:0/986789515 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12080cc880 con 0x7f1208162150 2024-04-24T08:09:43.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.298+0000 7f120f400640 1 -- 172.21.15.69:0/986789515 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12080ccb20 con 0x7f1208162150 2024-04-24T08:09:43.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.300+0000 7f120f400640 1 -- 172.21.15.69:0/986789515 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12080cad90 con 0x7f1208162150 2024-04-24T08:09:43.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.301+0000 7f1204c00640 1 -- 172.21.15.69:0/986789515 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1200005d10 con 0x7f1208162150 2024-04-24T08:09:43.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.301+0000 7f1204c00640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11e806c8f0 0x7f11e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.302+0000 7f1204c00640 1 -- 172.21.15.69:0/986789515 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f120008cac0 con 0x7f1208162150 2024-04-24T08:09:43.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.302+0000 7f120ca00640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11e806c8f0 0x7f11e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.303+0000 7f120ca00640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11e806c8f0 0x7f11e806edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f11fc033000 tx=0x7f11fc02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:43.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.306+0000 7f1204c00640 1 -- 172.21.15.69:0/986789515 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f12080cad90 con 0x7f1208162150 2024-04-24T08:09:43.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.501+0000 7f120f400640 1 -- 172.21.15.69:0/986789515 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f12080d1df0 con 0x7f11e806c8f0 2024-04-24T08:09:43.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.522+0000 7f1204c00640 1 -- 172.21.15.69:0/986789515 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246635 (secure 0 0 0) 0x7f12080d1df0 con 0x7f11e806c8f0 2024-04-24T08:09:43.526 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:43.527 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:09:43.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 -- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11e806c8f0 msgr2=0x7f11e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11e806c8f0 0x7f11e806edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f11fc033000 tx=0x7f11fc02e000 comp rx=0 tx=0).stop 2024-04-24T08:09:43.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 -- 172.21.15.69:0/986789515 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 msgr2=0x7f120815eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 0x7f120815eec0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f120000ef90 tx=0x7f120000c490 comp rx=0 tx=0).stop 2024-04-24T08:09:43.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 -- 172.21.15.69:0/986789515 shutdown_connections 2024-04-24T08:09:43.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11e806c8f0 0x7f11e806edb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1208162150 0x7f120815eec0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f12080cfe40 0x7f120815e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 --2- 172.21.15.69:0/986789515 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f12080b7a90 0x7f1208153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 -- 172.21.15.69:0/986789515 >> 172.21.15.69:0/986789515 conn(0x7f120808bf60 msgr2=0x7f12080b85d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:43.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 -- 172.21.15.69:0/986789515 shutdown_connections 2024-04-24T08:09:43.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.528+0000 7f120f400640 1 -- 172.21.15.69:0/986789515 wait complete. 2024-04-24T08:09:43.547 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-04-24T08:09:42.690265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220040,"kb_used_data":3856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754680,"statfs":{"total":773094113280,"available":772868792320,"internally_reserved":0,"allocated":3948544,"data_stored":2237762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.631516"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.091995+0000","last_change":"2024-04-24T08:09:38.091995+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.091995+0000","last_undegraded":"2024-04-24T08:09:38.091995+0000","last_fullsized":"2024-04-24T08:09:38.091995+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090575+0000","last_change":"2024-04-24T08:09:38.090575+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090575+0000","last_undegraded":"2024-04-24T08:09:38.090575+0000","last_fullsized":"2024-04-24T08:09:38.090575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.085165+0000","last_change":"2024-04-24T08:09:38.085165+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.085165+0000","last_undegraded":"2024-04-24T08:09:38.085165+0000","last_fullsized":"2024-04-24T08:09:38.085165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092285+0000","last_change":"2024-04-24T08:09:38.092285+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092285+0000","last_undegraded":"2024-04-24T08:09:38.092285+0000","last_fullsized":"2024-04-24T08:09:38.092285+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087102+0000","last_change":"2024-04-24T08:09:38.087102+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087102+0000","last_undegraded":"2024-04-24T08:09:38.087102+0000","last_fullsized":"2024-04-24T08:09:38.087102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088583+0000","last_change":"2024-04-24T08:09:38.088583+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088583+0000","last_undegraded":"2024-04-24T08:09:38.088583+0000","last_fullsized":"2024-04-24T08:09:38.088583+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093698+0000","last_change":"2024-04-24T08:09:38.093698+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093698+0000","last_undegraded":"2024-04-24T08:09:38.093698+0000","last_fullsized":"2024-04-24T08:09:38.093698+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.098131+0000","last_change":"2024-04-24T08:09:38.098131+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.098131+0000","last_undegraded":"2024-04-24T08:09:38.098131+0000","last_fullsized":"2024-04-24T08:09:38.098131+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.095971+0000","last_change":"2024-04-24T08:09:38.095971+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.095971+0000","last_undegraded":"2024-04-24T08:09:38.095971+0000","last_fullsized":"2024-04-24T08:09:38.095971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093218+0000","last_change":"2024-04-24T08:09:38.093218+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093218+0000","last_undegraded":"2024-04-24T08:09:38.093218+0000","last_fullsized":"2024-04-24T08:09:38.093218+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.098579+0000","last_change":"2024-04-24T08:09:38.098579+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.098579+0000","last_undegraded":"2024-04-24T08:09:38.098579+0000","last_fullsized":"2024-04-24T08:09:38.098579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084662+0000","last_change":"2024-04-24T08:09:38.084662+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084662+0000","last_undegraded":"2024-04-24T08:09:38.084662+0000","last_fullsized":"2024-04-24T08:09:38.084662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086610+0000","last_change":"2024-04-24T08:09:38.086610+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086610+0000","last_undegraded":"2024-04-24T08:09:38.086610+0000","last_fullsized":"2024-04-24T08:09:38.086610+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087851+0000","last_change":"2024-04-24T08:09:38.087851+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087851+0000","last_undegraded":"2024-04-24T08:09:38.087851+0000","last_fullsized":"2024-04-24T08:09:38.087851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088469+0000","last_change":"2024-04-24T08:09:38.088469+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088469+0000","last_undegraded":"2024-04-24T08:09:38.088469+0000","last_fullsized":"2024-04-24T08:09:38.088469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088695+0000","last_change":"2024-04-24T08:09:38.088695+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088695+0000","last_undegraded":"2024-04-24T08:09:38.088695+0000","last_fullsized":"2024-04-24T08:09:38.088695+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.102476+0000","last_change":"2024-04-24T08:09:38.102476+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.102476+0000","last_undegraded":"2024-04-24T08:09:38.102476+0000","last_fullsized":"2024-04-24T08:09:38.102476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089933+0000","last_change":"2024-04-24T08:09:38.089933+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089933+0000","last_undegraded":"2024-04-24T08:09:38.089933+0000","last_fullsized":"2024-04-24T08:09:38.089933+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086859+0000","last_change":"2024-04-24T08:09:38.086859+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086859+0000","last_undegraded":"2024-04-24T08:09:38.086859+0000","last_fullsized":"2024-04-24T08:09:38.086859+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093820+0000","last_change":"2024-04-24T08:09:38.093820+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093820+0000","last_undegraded":"2024-04-24T08:09:38.093820+0000","last_fullsized":"2024-04-24T08:09:38.093820+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.096265+0000","last_change":"2024-04-24T08:09:38.096265+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.096265+0000","last_undegraded":"2024-04-24T08:09:38.096265+0000","last_fullsized":"2024-04-24T08:09:38.096265+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.094464+0000","last_change":"2024-04-24T08:09:38.094464+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.094464+0000","last_undegraded":"2024-04-24T08:09:38.094464+0000","last_fullsized":"2024-04-24T08:09:38.094464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081017+0000","last_change":"2024-04-24T08:09:38.081017+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081017+0000","last_undegraded":"2024-04-24T08:09:38.081017+0000","last_fullsized":"2024-04-24T08:09:38.081017+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083984+0000","last_change":"2024-04-24T08:09:38.083984+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083984+0000","last_undegraded":"2024-04-24T08:09:38.083984+0000","last_fullsized":"2024-04-24T08:09:38.083984+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092642+0000","last_change":"2024-04-24T08:09:38.092642+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092642+0000","last_undegraded":"2024-04-24T08:09:38.092642+0000","last_fullsized":"2024-04-24T08:09:38.092642+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089128+0000","last_change":"2024-04-24T08:09:38.089128+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089128+0000","last_undegraded":"2024-04-24T08:09:38.089128+0000","last_fullsized":"2024-04-24T08:09:38.089128+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.094332+0000","last_change":"2024-04-24T08:09:38.094332+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.094332+0000","last_undegraded":"2024-04-24T08:09:38.094332+0000","last_fullsized":"2024-04-24T08:09:38.094332+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083051+0000","last_change":"2024-04-24T08:09:38.083051+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083051+0000","last_undegraded":"2024-04-24T08:09:38.083051+0000","last_fullsized":"2024-04-24T08:09:38.083051+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087579+0000","last_change":"2024-04-24T08:09:38.087579+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087579+0000","last_undegraded":"2024-04-24T08:09:38.087579+0000","last_fullsized":"2024-04-24T08:09:38.087579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.094793+0000","last_change":"2024-04-24T08:09:38.094793+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.094793+0000","last_undegraded":"2024-04-24T08:09:38.094793+0000","last_fullsized":"2024-04-24T08:09:38.094793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.095355+0000","last_change":"2024-04-24T08:09:38.095355+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.095355+0000","last_undegraded":"2024-04-24T08:09:38.095355+0000","last_fullsized":"2024-04-24T08:09:38.095355+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.097440+0000","last_change":"2024-04-24T08:09:38.097440+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.097440+0000","last_undegraded":"2024-04-24T08:09:38.097440+0000","last_fullsized":"2024-04-24T08:09:38.097440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081278+0000","last_change":"2024-04-24T08:09:38.081278+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081278+0000","last_undegraded":"2024-04-24T08:09:38.081278+0000","last_fullsized":"2024-04-24T08:09:38.081278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.112465+0000","last_change":"2024-04-24T08:09:38.106509+0000","last_active":"2024-04-24T08:09:38.112465+0000","last_peered":"2024-04-24T08:09:38.112465+0000","last_clean":"2024-04-24T08:09:38.112465+0000","last_became_active":"2024-04-24T08:09:38.106216+0000","last_became_peered":"2024-04-24T08:09:38.106216+0000","last_unstale":"2024-04-24T08:09:38.112465+0000","last_undegraded":"2024-04-24T08:09:38.112465+0000","last_fullsized":"2024-04-24T08:09:38.112465+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:05.837977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090079+0000","last_change":"2024-04-24T08:09:38.090079+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090079+0000","last_undegraded":"2024-04-24T08:09:38.090079+0000","last_fullsized":"2024-04-24T08:09:38.090079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109428+0000","last_change":"2024-04-24T08:09:38.107316+0000","last_active":"2024-04-24T08:09:38.109428+0000","last_peered":"2024-04-24T08:09:38.109428+0000","last_clean":"2024-04-24T08:09:38.109428+0000","last_became_active":"2024-04-24T08:09:38.106891+0000","last_became_peered":"2024-04-24T08:09:38.106891+0000","last_unstale":"2024-04-24T08:09:38.109428+0000","last_undegraded":"2024-04-24T08:09:38.109428+0000","last_fullsized":"2024-04-24T08:09:38.109428+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092529+0000","last_change":"2024-04-24T08:09:38.092529+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092529+0000","last_undegraded":"2024-04-24T08:09:38.092529+0000","last_fullsized":"2024-04-24T08:09:38.092529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108092+0000","last_change":"2024-04-24T08:09:38.105975+0000","last_active":"2024-04-24T08:09:38.108092+0000","last_peered":"2024-04-24T08:09:38.108092+0000","last_clean":"2024-04-24T08:09:38.108092+0000","last_became_active":"2024-04-24T08:09:38.105684+0000","last_became_peered":"2024-04-24T08:09:38.105684+0000","last_unstale":"2024-04-24T08:09:38.108092+0000","last_undegraded":"2024-04-24T08:09:38.108092+0000","last_fullsized":"2024-04-24T08:09:38.108092+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087273+0000","last_change":"2024-04-24T08:09:38.087273+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087273+0000","last_undegraded":"2024-04-24T08:09:38.087273+0000","last_fullsized":"2024-04-24T08:09:38.087273+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.103902+0000","last_change":"2024-04-24T08:09:38.102564+0000","last_active":"2024-04-24T08:09:38.103902+0000","last_peered":"2024-04-24T08:09:38.103902+0000","last_clean":"2024-04-24T08:09:38.103902+0000","last_became_active":"2024-04-24T08:09:38.101598+0000","last_became_peered":"2024-04-24T08:09:38.101598+0000","last_unstale":"2024-04-24T08:09:38.103902+0000","last_undegraded":"2024-04-24T08:09:38.103902+0000","last_fullsized":"2024-04-24T08:09:38.103902+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:19:44.006523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093880+0000","last_change":"2024-04-24T08:09:38.093880+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093880+0000","last_undegraded":"2024-04-24T08:09:38.093880+0000","last_fullsized":"2024-04-24T08:09:38.093880+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109733+0000","last_change":"2024-04-24T08:09:38.101731+0000","last_active":"2024-04-24T08:09:38.109733+0000","last_peered":"2024-04-24T08:09:38.109733+0000","last_clean":"2024-04-24T08:09:38.109733+0000","last_became_active":"2024-04-24T08:09:38.100536+0000","last_became_peered":"2024-04-24T08:09:38.100536+0000","last_unstale":"2024-04-24T08:09:38.109733+0000","last_undegraded":"2024-04-24T08:09:38.109733+0000","last_fullsized":"2024-04-24T08:09:38.109733+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:17:11.041488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086737+0000","last_change":"2024-04-24T08:09:38.086737+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086737+0000","last_undegraded":"2024-04-24T08:09:38.086737+0000","last_fullsized":"2024-04-24T08:09:38.086737+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.112314+0000","last_change":"2024-04-24T08:09:38.108989+0000","last_active":"2024-04-24T08:09:38.112314+0000","last_peered":"2024-04-24T08:09:38.112314+0000","last_clean":"2024-04-24T08:09:38.112314+0000","last_became_active":"2024-04-24T08:09:38.108726+0000","last_became_peered":"2024-04-24T08:09:38.108726+0000","last_unstale":"2024-04-24T08:09:38.112314+0000","last_undegraded":"2024-04-24T08:09:38.112314+0000","last_fullsized":"2024-04-24T08:09:38.112314+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:40:46.556408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.091473+0000","last_change":"2024-04-24T08:09:38.091473+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.091473+0000","last_undegraded":"2024-04-24T08:09:38.091473+0000","last_fullsized":"2024-04-24T08:09:38.091473+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.097976+0000","last_change":"2024-04-24T08:09:38.093313+0000","last_active":"2024-04-24T08:09:38.097976+0000","last_peered":"2024-04-24T08:09:38.097976+0000","last_clean":"2024-04-24T08:09:38.097976+0000","last_became_active":"2024-04-24T08:09:38.092910+0000","last_became_peered":"2024-04-24T08:09:38.092910+0000","last_unstale":"2024-04-24T08:09:38.097976+0000","last_undegraded":"2024-04-24T08:09:38.097976+0000","last_fullsized":"2024-04-24T08:09:38.097976+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083631+0000","last_change":"2024-04-24T08:09:38.083631+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083631+0000","last_undegraded":"2024-04-24T08:09:38.083631+0000","last_fullsized":"2024-04-24T08:09:38.083631+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111408+0000","last_change":"2024-04-24T08:09:38.107454+0000","last_active":"2024-04-24T08:09:38.111408+0000","last_peered":"2024-04-24T08:09:38.111408+0000","last_clean":"2024-04-24T08:09:38.111408+0000","last_became_active":"2024-04-24T08:09:38.107149+0000","last_became_peered":"2024-04-24T08:09:38.107149+0000","last_unstale":"2024-04-24T08:09:38.111408+0000","last_undegraded":"2024-04-24T08:09:38.111408+0000","last_fullsized":"2024-04-24T08:09:38.111408+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:46:32.216293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087267+0000","last_change":"2024-04-24T08:09:38.087267+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087267+0000","last_undegraded":"2024-04-24T08:09:38.087267+0000","last_fullsized":"2024-04-24T08:09:38.087267+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111453+0000","last_change":"2024-04-24T08:09:38.110451+0000","last_active":"2024-04-24T08:09:38.111453+0000","last_peered":"2024-04-24T08:09:38.111453+0000","last_clean":"2024-04-24T08:09:38.111453+0000","last_became_active":"2024-04-24T08:09:38.110191+0000","last_became_peered":"2024-04-24T08:09:38.110191+0000","last_unstale":"2024-04-24T08:09:38.111453+0000","last_undegraded":"2024-04-24T08:09:38.111453+0000","last_fullsized":"2024-04-24T08:09:38.111453+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:10.779345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090571+0000","last_change":"2024-04-24T08:09:38.090571+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090571+0000","last_undegraded":"2024-04-24T08:09:38.090571+0000","last_fullsized":"2024-04-24T08:09:38.090571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.095062+0000","last_change":"2024-04-24T08:09:38.093711+0000","last_active":"2024-04-24T08:09:38.095062+0000","last_peered":"2024-04-24T08:09:38.095062+0000","last_clean":"2024-04-24T08:09:38.095062+0000","last_became_active":"2024-04-24T08:09:38.091626+0000","last_became_peered":"2024-04-24T08:09:38.091626+0000","last_unstale":"2024-04-24T08:09:38.095062+0000","last_undegraded":"2024-04-24T08:09:38.095062+0000","last_fullsized":"2024-04-24T08:09:38.095062+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.085881+0000","last_change":"2024-04-24T08:09:38.085881+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.085881+0000","last_undegraded":"2024-04-24T08:09:38.085881+0000","last_fullsized":"2024-04-24T08:09:38.085881+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109912+0000","last_change":"2024-04-24T08:09:38.107829+0000","last_active":"2024-04-24T08:09:38.109912+0000","last_peered":"2024-04-24T08:09:38.109912+0000","last_clean":"2024-04-24T08:09:38.109912+0000","last_became_active":"2024-04-24T08:09:38.107551+0000","last_became_peered":"2024-04-24T08:09:38.107551+0000","last_unstale":"2024-04-24T08:09:38.109912+0000","last_undegraded":"2024-04-24T08:09:38.109912+0000","last_fullsized":"2024-04-24T08:09:38.109912+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086224+0000","last_change":"2024-04-24T08:09:38.086224+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086224+0000","last_undegraded":"2024-04-24T08:09:38.086224+0000","last_fullsized":"2024-04-24T08:09:38.086224+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108172+0000","last_change":"2024-04-24T08:09:38.106294+0000","last_active":"2024-04-24T08:09:38.108172+0000","last_peered":"2024-04-24T08:09:38.108172+0000","last_clean":"2024-04-24T08:09:38.108172+0000","last_became_active":"2024-04-24T08:09:38.105994+0000","last_became_peered":"2024-04-24T08:09:38.105994+0000","last_unstale":"2024-04-24T08:09:38.108172+0000","last_undegraded":"2024-04-24T08:09:38.108172+0000","last_fullsized":"2024-04-24T08:09:38.108172+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:11.064191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084768+0000","last_change":"2024-04-24T08:09:38.084768+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084768+0000","last_undegraded":"2024-04-24T08:09:38.084768+0000","last_fullsized":"2024-04-24T08:09:38.084768+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111341+0000","last_change":"2024-04-24T08:09:38.105998+0000","last_active":"2024-04-24T08:09:38.111341+0000","last_peered":"2024-04-24T08:09:38.111341+0000","last_clean":"2024-04-24T08:09:38.111341+0000","last_became_active":"2024-04-24T08:09:38.105710+0000","last_became_peered":"2024-04-24T08:09:38.105710+0000","last_unstale":"2024-04-24T08:09:38.111341+0000","last_undegraded":"2024-04-24T08:09:38.111341+0000","last_fullsized":"2024-04-24T08:09:38.111341+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:58:11.279574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084682+0000","last_change":"2024-04-24T08:09:38.084682+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084682+0000","last_undegraded":"2024-04-24T08:09:38.084682+0000","last_fullsized":"2024-04-24T08:09:38.084682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109784+0000","last_change":"2024-04-24T08:09:38.107964+0000","last_active":"2024-04-24T08:09:38.109784+0000","last_peered":"2024-04-24T08:09:38.109784+0000","last_clean":"2024-04-24T08:09:38.109784+0000","last_became_active":"2024-04-24T08:09:38.107620+0000","last_became_peered":"2024-04-24T08:09:38.107620+0000","last_unstale":"2024-04-24T08:09:38.109784+0000","last_undegraded":"2024-04-24T08:09:38.109784+0000","last_fullsized":"2024-04-24T08:09:38.109784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.096631+0000","last_change":"2024-04-24T08:09:38.096631+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.096631+0000","last_undegraded":"2024-04-24T08:09:38.096631+0000","last_fullsized":"2024-04-24T08:09:38.096631+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.106220+0000","last_change":"2024-04-24T08:09:38.103294+0000","last_active":"2024-04-24T08:09:38.106220+0000","last_peered":"2024-04-24T08:09:38.106220+0000","last_clean":"2024-04-24T08:09:38.106220+0000","last_became_active":"2024-04-24T08:09:38.101762+0000","last_became_peered":"2024-04-24T08:09:38.101762+0000","last_unstale":"2024-04-24T08:09:38.106220+0000","last_undegraded":"2024-04-24T08:09:38.106220+0000","last_fullsized":"2024-04-24T08:09:38.106220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088094+0000","last_change":"2024-04-24T08:09:38.088094+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088094+0000","last_undegraded":"2024-04-24T08:09:38.088094+0000","last_fullsized":"2024-04-24T08:09:38.088094+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.103953+0000","last_change":"2024-04-24T08:09:38.100426+0000","last_active":"2024-04-24T08:09:38.103953+0000","last_peered":"2024-04-24T08:09:38.103953+0000","last_clean":"2024-04-24T08:09:38.103953+0000","last_became_active":"2024-04-24T08:09:38.100138+0000","last_became_peered":"2024-04-24T08:09:38.100138+0000","last_unstale":"2024-04-24T08:09:38.103953+0000","last_undegraded":"2024-04-24T08:09:38.103953+0000","last_fullsized":"2024-04-24T08:09:38.103953+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089848+0000","last_change":"2024-04-24T08:09:38.089848+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089848+0000","last_undegraded":"2024-04-24T08:09:38.089848+0000","last_fullsized":"2024-04-24T08:09:38.089848+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.100103+0000","last_change":"2024-04-24T08:09:38.096536+0000","last_active":"2024-04-24T08:09:38.100103+0000","last_peered":"2024-04-24T08:09:38.100103+0000","last_clean":"2024-04-24T08:09:38.100103+0000","last_became_active":"2024-04-24T08:09:38.095915+0000","last_became_peered":"2024-04-24T08:09:38.095915+0000","last_unstale":"2024-04-24T08:09:38.100103+0000","last_undegraded":"2024-04-24T08:09:38.100103+0000","last_fullsized":"2024-04-24T08:09:38.100103+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:05.837977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088654+0000","last_change":"2024-04-24T08:09:38.088654+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088654+0000","last_undegraded":"2024-04-24T08:09:38.088654+0000","last_fullsized":"2024-04-24T08:09:38.088654+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109578+0000","last_change":"2024-04-24T08:09:38.101152+0000","last_active":"2024-04-24T08:09:38.109578+0000","last_peered":"2024-04-24T08:09:38.109578+0000","last_clean":"2024-04-24T08:09:38.109578+0000","last_became_active":"2024-04-24T08:09:38.099292+0000","last_became_peered":"2024-04-24T08:09:38.099292+0000","last_unstale":"2024-04-24T08:09:38.109578+0000","last_undegraded":"2024-04-24T08:09:38.109578+0000","last_fullsized":"2024-04-24T08:09:38.109578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:05:05.064870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087579+0000","last_change":"2024-04-24T08:09:38.087579+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087579+0000","last_undegraded":"2024-04-24T08:09:38.087579+0000","last_fullsized":"2024-04-24T08:09:38.087579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109637+0000","last_change":"2024-04-24T08:09:38.100668+0000","last_active":"2024-04-24T08:09:38.109637+0000","last_peered":"2024-04-24T08:09:38.109637+0000","last_clean":"2024-04-24T08:09:38.109637+0000","last_became_active":"2024-04-24T08:09:38.100303+0000","last_became_peered":"2024-04-24T08:09:38.100303+0000","last_unstale":"2024-04-24T08:09:38.109637+0000","last_undegraded":"2024-04-24T08:09:38.109637+0000","last_fullsized":"2024-04-24T08:09:38.109637+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:55.861909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089628+0000","last_change":"2024-04-24T08:09:38.089628+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089628+0000","last_undegraded":"2024-04-24T08:09:38.089628+0000","last_fullsized":"2024-04-24T08:09:38.089628+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111334+0000","last_change":"2024-04-24T08:09:38.105765+0000","last_active":"2024-04-24T08:09:38.111334+0000","last_peered":"2024-04-24T08:09:38.111334+0000","last_clean":"2024-04-24T08:09:38.111334+0000","last_became_active":"2024-04-24T08:09:38.100242+0000","last_became_peered":"2024-04-24T08:09:38.100242+0000","last_unstale":"2024-04-24T08:09:38.111334+0000","last_undegraded":"2024-04-24T08:09:38.111334+0000","last_fullsized":"2024-04-24T08:09:38.111334+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:19:44.006523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084377+0000","last_change":"2024-04-24T08:09:38.084377+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084377+0000","last_undegraded":"2024-04-24T08:09:38.084377+0000","last_fullsized":"2024-04-24T08:09:38.084377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.103077+0000","last_change":"2024-04-24T08:09:38.097858+0000","last_active":"2024-04-24T08:09:38.103077+0000","last_peered":"2024-04-24T08:09:38.103077+0000","last_clean":"2024-04-24T08:09:38.103077+0000","last_became_active":"2024-04-24T08:09:38.097599+0000","last_became_peered":"2024-04-24T08:09:38.097599+0000","last_unstale":"2024-04-24T08:09:38.103077+0000","last_undegraded":"2024-04-24T08:09:38.103077+0000","last_fullsized":"2024-04-24T08:09:38.103077+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086095+0000","last_change":"2024-04-24T08:09:38.086095+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086095+0000","last_undegraded":"2024-04-24T08:09:38.086095+0000","last_fullsized":"2024-04-24T08:09:38.086095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.115052+0000","last_change":"2024-04-24T08:09:38.110759+0000","last_active":"2024-04-24T08:09:38.115052+0000","last_peered":"2024-04-24T08:09:38.115052+0000","last_clean":"2024-04-24T08:09:38.115052+0000","last_became_active":"2024-04-24T08:09:38.110393+0000","last_became_peered":"2024-04-24T08:09:38.110393+0000","last_unstale":"2024-04-24T08:09:38.115052+0000","last_undegraded":"2024-04-24T08:09:38.115052+0000","last_fullsized":"2024-04-24T08:09:38.115052+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087968+0000","last_change":"2024-04-24T08:09:38.087968+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087968+0000","last_undegraded":"2024-04-24T08:09:38.087968+0000","last_fullsized":"2024-04-24T08:09:38.087968+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111161+0000","last_change":"2024-04-24T08:09:38.104959+0000","last_active":"2024-04-24T08:09:38.111161+0000","last_peered":"2024-04-24T08:09:38.111161+0000","last_clean":"2024-04-24T08:09:38.111161+0000","last_became_active":"2024-04-24T08:09:38.104606+0000","last_became_peered":"2024-04-24T08:09:38.104606+0000","last_unstale":"2024-04-24T08:09:38.111161+0000","last_undegraded":"2024-04-24T08:09:38.111161+0000","last_fullsized":"2024-04-24T08:09:38.111161+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:32.287730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.095292+0000","last_change":"2024-04-24T08:09:38.095292+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.095292+0000","last_undegraded":"2024-04-24T08:09:38.095292+0000","last_fullsized":"2024-04-24T08:09:38.095292+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.106663+0000","last_change":"2024-04-24T08:09:38.104201+0000","last_active":"2024-04-24T08:09:38.106663+0000","last_peered":"2024-04-24T08:09:38.106663+0000","last_clean":"2024-04-24T08:09:38.106663+0000","last_became_active":"2024-04-24T08:09:38.103332+0000","last_became_peered":"2024-04-24T08:09:38.103332+0000","last_unstale":"2024-04-24T08:09:38.106663+0000","last_undegraded":"2024-04-24T08:09:38.106663+0000","last_fullsized":"2024-04-24T08:09:38.106663+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090440+0000","last_change":"2024-04-24T08:09:38.090440+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090440+0000","last_undegraded":"2024-04-24T08:09:38.090440+0000","last_fullsized":"2024-04-24T08:09:38.090440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108613+0000","last_change":"2024-04-24T08:09:38.105102+0000","last_active":"2024-04-24T08:09:38.108613+0000","last_peered":"2024-04-24T08:09:38.108613+0000","last_clean":"2024-04-24T08:09:38.108613+0000","last_became_active":"2024-04-24T08:09:38.104814+0000","last_became_peered":"2024-04-24T08:09:38.104814+0000","last_unstale":"2024-04-24T08:09:38.108613+0000","last_undegraded":"2024-04-24T08:09:38.108613+0000","last_fullsized":"2024-04-24T08:09:38.108613+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087435+0000","last_change":"2024-04-24T08:09:38.087435+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087435+0000","last_undegraded":"2024-04-24T08:09:38.087435+0000","last_fullsized":"2024-04-24T08:09:38.087435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.104504+0000","last_change":"2024-04-24T08:09:38.102570+0000","last_active":"2024-04-24T08:09:38.104504+0000","last_peered":"2024-04-24T08:09:38.104504+0000","last_clean":"2024-04-24T08:09:38.104504+0000","last_became_active":"2024-04-24T08:09:38.101279+0000","last_became_peered":"2024-04-24T08:09:38.101279+0000","last_unstale":"2024-04-24T08:09:38.104504+0000","last_undegraded":"2024-04-24T08:09:38.104504+0000","last_fullsized":"2024-04-24T08:09:38.104504+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081042+0000","last_change":"2024-04-24T08:09:38.081042+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081042+0000","last_undegraded":"2024-04-24T08:09:38.081042+0000","last_fullsized":"2024-04-24T08:09:38.081042+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.105348+0000","last_change":"2024-04-24T08:09:38.101682+0000","last_active":"2024-04-24T08:09:38.105348+0000","last_peered":"2024-04-24T08:09:38.105348+0000","last_clean":"2024-04-24T08:09:38.105348+0000","last_became_active":"2024-04-24T08:09:38.099204+0000","last_became_peered":"2024-04-24T08:09:38.099204+0000","last_unstale":"2024-04-24T08:09:38.105348+0000","last_undegraded":"2024-04-24T08:09:38.105348+0000","last_fullsized":"2024-04-24T08:09:38.105348+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089049+0000","last_change":"2024-04-24T08:09:38.089049+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089049+0000","last_undegraded":"2024-04-24T08:09:38.089049+0000","last_fullsized":"2024-04-24T08:09:38.089049+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.104376+0000","last_change":"2024-04-24T08:09:38.101668+0000","last_active":"2024-04-24T08:09:38.104376+0000","last_peered":"2024-04-24T08:09:38.104376+0000","last_clean":"2024-04-24T08:09:38.104376+0000","last_became_active":"2024-04-24T08:09:38.101318+0000","last_became_peered":"2024-04-24T08:09:38.101318+0000","last_unstale":"2024-04-24T08:09:38.104376+0000","last_undegraded":"2024-04-24T08:09:38.104376+0000","last_fullsized":"2024-04-24T08:09:38.104376+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081090+0000","last_change":"2024-04-24T08:09:38.081090+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081090+0000","last_undegraded":"2024-04-24T08:09:38.081090+0000","last_fullsized":"2024-04-24T08:09:38.081090+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108737+0000","last_change":"2024-04-24T08:09:38.105797+0000","last_active":"2024-04-24T08:09:38.108737+0000","last_peered":"2024-04-24T08:09:38.108737+0000","last_clean":"2024-04-24T08:09:38.108737+0000","last_became_active":"2024-04-24T08:09:38.104048+0000","last_became_peered":"2024-04-24T08:09:38.104048+0000","last_unstale":"2024-04-24T08:09:38.108737+0000","last_undegraded":"2024-04-24T08:09:38.108737+0000","last_fullsized":"2024-04-24T08:09:38.108737+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083903+0000","last_change":"2024-04-24T08:09:38.083903+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083903+0000","last_undegraded":"2024-04-24T08:09:38.083903+0000","last_fullsized":"2024-04-24T08:09:38.083903+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.110901+0000","last_change":"2024-04-24T08:09:38.103639+0000","last_active":"2024-04-24T08:09:38.110901+0000","last_peered":"2024-04-24T08:09:38.110901+0000","last_clean":"2024-04-24T08:09:38.110901+0000","last_became_active":"2024-04-24T08:09:38.103230+0000","last_became_peered":"2024-04-24T08:09:38.103230+0000","last_unstale":"2024-04-24T08:09:38.110901+0000","last_undegraded":"2024-04-24T08:09:38.110901+0000","last_fullsized":"2024-04-24T08:09:38.110901+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:27.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092331+0000","last_change":"2024-04-24T08:09:38.092331+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092331+0000","last_undegraded":"2024-04-24T08:09:38.092331+0000","last_fullsized":"2024-04-24T08:09:38.092331+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108778+0000","last_change":"2024-04-24T08:09:38.107831+0000","last_active":"2024-04-24T08:09:38.108778+0000","last_peered":"2024-04-24T08:09:38.108778+0000","last_clean":"2024-04-24T08:09:38.108778+0000","last_became_active":"2024-04-24T08:09:38.107509+0000","last_became_peered":"2024-04-24T08:09:38.107509+0000","last_unstale":"2024-04-24T08:09:38.108778+0000","last_undegraded":"2024-04-24T08:09:38.108778+0000","last_fullsized":"2024-04-24T08:09:38.108778+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:27.457454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.099092+0000","last_change":"2024-04-24T08:09:38.099092+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.099092+0000","last_undegraded":"2024-04-24T08:09:38.099092+0000","last_fullsized":"2024-04-24T08:09:38.099092+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.117716+0000","last_change":"2024-04-24T08:09:38.112443+0000","last_active":"2024-04-24T08:09:38.117716+0000","last_peered":"2024-04-24T08:09:38.117716+0000","last_clean":"2024-04-24T08:09:38.117716+0000","last_became_active":"2024-04-24T08:09:38.111901+0000","last_became_peered":"2024-04-24T08:09:38.111901+0000","last_unstale":"2024-04-24T08:09:38.117716+0000","last_undegraded":"2024-04-24T08:09:38.117716+0000","last_fullsized":"2024-04-24T08:09:38.117716+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.106310+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:09:38.106310+0000","last_peered":"2024-04-24T08:09:38.106310+0000","last_clean":"2024-04-24T08:09:38.106310+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:09:38.106310+0000","last_undegraded":"2024-04-24T08:09:38.106310+0000","last_fullsized":"2024-04-24T08:09:38.106310+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":724063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:09:43.552 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:09:43.666 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:09:43.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:09:43.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.790+0000 7f580ee00640 1 Processor -- start 2024-04-24T08:09:43.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.790+0000 7f580ee00640 1 -- start start 2024-04-24T08:09:43.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.792+0000 7f580ee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808162c70 0x7f58080b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.792+0000 7f580ee00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58080b3d30 0x7f58080b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.792+0000 7f580ee00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58081669d0 con 0x7f5808071d70 2024-04-24T08:09:43.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.792+0000 7f580ee00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f58080b46d0 con 0x7f58080b3d30 2024-04-24T08:09:43.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.792+0000 7f580ee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5808166cb0 con 0x7f5808162c70 2024-04-24T08:09:43.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.792+0000 7f580ce00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5808071d70 0x7f58080721a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53286/0 (socket says 172.21.15.69:53286) 2024-04-24T08:09:43.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.792+0000 7f57ffe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58080b3d30 0x7f58080b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.792+0000 7f580ce00640 1 -- 172.21.15.69:0/1653716007 learned_addr learned my addr 172.21.15.69:0/1653716007 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:43.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.793+0000 7f5807e00640 1 --2- 172.21.15.69:0/1653716007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808162c70 0x7f58080b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.793+0000 7f57ffe00640 1 -- 172.21.15.69:0/1653716007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808162c70 msgr2=0x7f58080b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:43.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.793+0000 7f57ffe00640 1 --2- 172.21.15.69:0/1653716007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808162c70 0x7f58080b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.793+0000 7f57ffe00640 1 -- 172.21.15.69:0/1653716007 >> v1:172.21.15.69:6789/0 conn(0x7f5808071d70 legacy=0x7f58080721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.793+0000 7f57ffe00640 1 -- 172.21.15.69:0/1653716007 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5808166f90 con 0x7f58080b3d30 2024-04-24T08:09:43.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.794+0000 7f57ffe00640 1 --2- 172.21.15.69:0/1653716007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58080b3d30 0x7f58080b4190 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f57f00097a0 tx=0x7f57f002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfba356eb52e10f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:43.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.795+0000 7f5807400640 1 -- 172.21.15.69:0/1653716007 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f0037070 con 0x7f58080b3d30 2024-04-24T08:09:43.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.795+0000 7f5807400640 1 -- 172.21.15.69:0/1653716007 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57f0032440 con 0x7f58080b3d30 2024-04-24T08:09:43.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.795+0000 7f5807400640 1 -- 172.21.15.69:0/1653716007 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f0032810 con 0x7f58080b3d30 2024-04-24T08:09:43.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.795+0000 7f580ee00640 1 -- 172.21.15.69:0/1653716007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58080b3d30 msgr2=0x7f58080b4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.795+0000 7f580ee00640 1 --2- 172.21.15.69:0/1653716007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58080b3d30 0x7f58080b4190 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f57f00097a0 tx=0x7f57f002a180 comp rx=0 tx=0).stop 2024-04-24T08:09:43.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.795+0000 7f580ee00640 1 -- 172.21.15.69:0/1653716007 shutdown_connections 2024-04-24T08:09:43.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.795+0000 7f580ee00640 1 --2- 172.21.15.69:0/1653716007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58080b3d30 0x7f58080b4190 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.795+0000 7f580ee00640 1 --2- 172.21.15.69:0/1653716007 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808162c70 0x7f58080b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.795+0000 7f580ee00640 1 -- 172.21.15.69:0/1653716007 >> 172.21.15.69:0/1653716007 conn(0x7f5808067d90 msgr2=0x7f580806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:43.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.796+0000 7f580ee00640 1 -- 172.21.15.69:0/1653716007 shutdown_connections 2024-04-24T08:09:43.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.796+0000 7f580ee00640 1 -- 172.21.15.69:0/1653716007 wait complete. 2024-04-24T08:09:43.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.797+0000 7f580ee00640 1 Processor -- start 2024-04-24T08:09:43.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.797+0000 7f580ee00640 1 -- start start 2024-04-24T08:09:43.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.798+0000 7f580ee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808071d70 0x7f580815e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.798+0000 7f580ee00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58080b3d30 0x7f580815c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.798+0000 7f580ee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808162c70 0x7f58081561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.798+0000 7f580ee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5808163050 con 0x7f5808162c70 2024-04-24T08:09:43.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.798+0000 7f580ee00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5808167550 con 0x7f58080b3d30 2024-04-24T08:09:43.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.798+0000 7f580ee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f58080cfc40 con 0x7f5808071d70 2024-04-24T08:09:43.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.799+0000 7f5807e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808071d70 0x7f580815e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.799+0000 7f57ffe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58080b3d30 0x7f580815c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.799+0000 7f580ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808162c70 0x7f58081561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.799+0000 7f5807e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808071d70 0x7f580815e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:36344/0 (socket says 172.21.15.69:36344) 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.799+0000 7f5807e00640 1 -- 172.21.15.69:0/1572528415 learned_addr learned my addr 172.21.15.69:0/1572528415 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.799+0000 7f580ce00640 1 -- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808071d70 msgr2=0x7f580815e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.800+0000 7f580ce00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808071d70 0x7f580815e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.800+0000 7f580ce00640 1 -- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58080b3d30 msgr2=0x7f580815c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.800+0000 7f580ce00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58080b3d30 0x7f580815c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.800+0000 7f580ce00640 1 -- 172.21.15.69:0/1572528415 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58081420b0 con 0x7f5808162c70 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.800+0000 7f580ce00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808162c70 0x7f58081561e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f580000cad0 tx=0x7f580000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.800+0000 7f5804c00640 1 -- 172.21.15.69:0/1572528415 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5800017070 con 0x7f5808162c70 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.800+0000 7f5804c00640 1 -- 172.21.15.69:0/1572528415 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f580000ac30 con 0x7f5808162c70 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.800+0000 7f5804c00640 1 -- 172.21.15.69:0/1572528415 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5800012400 con 0x7f5808162c70 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.800+0000 7f580ee00640 1 -- 172.21.15.69:0/1572528415 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5808142390 con 0x7f5808162c70 2024-04-24T08:09:43.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.800+0000 7f580ee00640 1 -- 172.21.15.69:0/1572528415 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5808142950 con 0x7f5808162c70 2024-04-24T08:09:43.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.802+0000 7f580ee00640 1 -- 172.21.15.69:0/1572528415 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5808068a80 con 0x7f5808162c70 2024-04-24T08:09:43.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.803+0000 7f5804c00640 1 -- 172.21.15.69:0/1572528415 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f580001a070 con 0x7f5808162c70 2024-04-24T08:09:43.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.803+0000 7f5804c00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57e406c8f0 0x7f57e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.804+0000 7f5804c00640 1 -- 172.21.15.69:0/1572528415 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f580008b6c0 con 0x7f5808162c70 2024-04-24T08:09:43.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.804+0000 7f5807e00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57e406c8f0 0x7f57e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.805+0000 7f5807e00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57e406c8f0 0x7f57e406edb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f57f8007a60 tx=0x7f57f8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:43.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:43.808+0000 7f5804c00640 1 -- 172.21.15.69:0/1572528415 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5800059530 con 0x7f5808162c70 2024-04-24T08:09:44.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.008+0000 7f580ee00640 1 -- 172.21.15.69:0/1572528415 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5808156720 con 0x7f57e406c8f0 2024-04-24T08:09:44.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.027+0000 7f5804c00640 1 -- 172.21.15.69:0/1572528415 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246635 (secure 0 0 0) 0x7f57f8010990 con 0x7f57e406c8f0 2024-04-24T08:09:44.032 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:44.033 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:09:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.034+0000 7f580ee00640 1 -- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57e406c8f0 msgr2=0x7f57e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.034+0000 7f580ee00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57e406c8f0 0x7f57e406edb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f57f8007a60 tx=0x7f57f8008040 comp rx=0 tx=0).stop 2024-04-24T08:09:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.034+0000 7f580ee00640 1 -- 172.21.15.69:0/1572528415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808162c70 msgr2=0x7f58081561e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.034+0000 7f580ee00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808162c70 0x7f58081561e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f580000cad0 tx=0x7f580000cf90 comp rx=0 tx=0).stop 2024-04-24T08:09:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.034+0000 7f580ee00640 1 -- 172.21.15.69:0/1572528415 shutdown_connections 2024-04-24T08:09:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.034+0000 7f580ee00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57e406c8f0 0x7f57e406edb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.034+0000 7f580ee00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808162c70 0x7f58081561e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.034+0000 7f580ee00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58080b3d30 0x7f580815c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.034+0000 7f580ee00640 1 --2- 172.21.15.69:0/1572528415 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5808071d70 0x7f580815e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.034+0000 7f580ee00640 1 -- 172.21.15.69:0/1572528415 >> 172.21.15.69:0/1572528415 conn(0x7f5808067d90 msgr2=0x7f580815cee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:44.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.035+0000 7f580ee00640 1 -- 172.21.15.69:0/1572528415 shutdown_connections 2024-04-24T08:09:44.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.035+0000 7f580ee00640 1 -- 172.21.15.69:0/1572528415 wait complete. 2024-04-24T08:09:44.055 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-04-24T08:09:42.690265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220040,"kb_used_data":3856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754680,"statfs":{"total":773094113280,"available":772868792320,"internally_reserved":0,"allocated":3948544,"data_stored":2237762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.631516"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.091995+0000","last_change":"2024-04-24T08:09:38.091995+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.091995+0000","last_undegraded":"2024-04-24T08:09:38.091995+0000","last_fullsized":"2024-04-24T08:09:38.091995+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090575+0000","last_change":"2024-04-24T08:09:38.090575+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090575+0000","last_undegraded":"2024-04-24T08:09:38.090575+0000","last_fullsized":"2024-04-24T08:09:38.090575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.085165+0000","last_change":"2024-04-24T08:09:38.085165+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.085165+0000","last_undegraded":"2024-04-24T08:09:38.085165+0000","last_fullsized":"2024-04-24T08:09:38.085165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092285+0000","last_change":"2024-04-24T08:09:38.092285+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092285+0000","last_undegraded":"2024-04-24T08:09:38.092285+0000","last_fullsized":"2024-04-24T08:09:38.092285+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087102+0000","last_change":"2024-04-24T08:09:38.087102+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087102+0000","last_undegraded":"2024-04-24T08:09:38.087102+0000","last_fullsized":"2024-04-24T08:09:38.087102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088583+0000","last_change":"2024-04-24T08:09:38.088583+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088583+0000","last_undegraded":"2024-04-24T08:09:38.088583+0000","last_fullsized":"2024-04-24T08:09:38.088583+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093698+0000","last_change":"2024-04-24T08:09:38.093698+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093698+0000","last_undegraded":"2024-04-24T08:09:38.093698+0000","last_fullsized":"2024-04-24T08:09:38.093698+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.098131+0000","last_change":"2024-04-24T08:09:38.098131+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.098131+0000","last_undegraded":"2024-04-24T08:09:38.098131+0000","last_fullsized":"2024-04-24T08:09:38.098131+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.095971+0000","last_change":"2024-04-24T08:09:38.095971+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.095971+0000","last_undegraded":"2024-04-24T08:09:38.095971+0000","last_fullsized":"2024-04-24T08:09:38.095971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093218+0000","last_change":"2024-04-24T08:09:38.093218+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093218+0000","last_undegraded":"2024-04-24T08:09:38.093218+0000","last_fullsized":"2024-04-24T08:09:38.093218+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.098579+0000","last_change":"2024-04-24T08:09:38.098579+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.098579+0000","last_undegraded":"2024-04-24T08:09:38.098579+0000","last_fullsized":"2024-04-24T08:09:38.098579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084662+0000","last_change":"2024-04-24T08:09:38.084662+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084662+0000","last_undegraded":"2024-04-24T08:09:38.084662+0000","last_fullsized":"2024-04-24T08:09:38.084662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086610+0000","last_change":"2024-04-24T08:09:38.086610+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086610+0000","last_undegraded":"2024-04-24T08:09:38.086610+0000","last_fullsized":"2024-04-24T08:09:38.086610+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087851+0000","last_change":"2024-04-24T08:09:38.087851+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087851+0000","last_undegraded":"2024-04-24T08:09:38.087851+0000","last_fullsized":"2024-04-24T08:09:38.087851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088469+0000","last_change":"2024-04-24T08:09:38.088469+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088469+0000","last_undegraded":"2024-04-24T08:09:38.088469+0000","last_fullsized":"2024-04-24T08:09:38.088469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088695+0000","last_change":"2024-04-24T08:09:38.088695+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088695+0000","last_undegraded":"2024-04-24T08:09:38.088695+0000","last_fullsized":"2024-04-24T08:09:38.088695+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.102476+0000","last_change":"2024-04-24T08:09:38.102476+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.102476+0000","last_undegraded":"2024-04-24T08:09:38.102476+0000","last_fullsized":"2024-04-24T08:09:38.102476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089933+0000","last_change":"2024-04-24T08:09:38.089933+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089933+0000","last_undegraded":"2024-04-24T08:09:38.089933+0000","last_fullsized":"2024-04-24T08:09:38.089933+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086859+0000","last_change":"2024-04-24T08:09:38.086859+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086859+0000","last_undegraded":"2024-04-24T08:09:38.086859+0000","last_fullsized":"2024-04-24T08:09:38.086859+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093820+0000","last_change":"2024-04-24T08:09:38.093820+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093820+0000","last_undegraded":"2024-04-24T08:09:38.093820+0000","last_fullsized":"2024-04-24T08:09:38.093820+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.096265+0000","last_change":"2024-04-24T08:09:38.096265+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.096265+0000","last_undegraded":"2024-04-24T08:09:38.096265+0000","last_fullsized":"2024-04-24T08:09:38.096265+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.094464+0000","last_change":"2024-04-24T08:09:38.094464+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.094464+0000","last_undegraded":"2024-04-24T08:09:38.094464+0000","last_fullsized":"2024-04-24T08:09:38.094464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081017+0000","last_change":"2024-04-24T08:09:38.081017+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081017+0000","last_undegraded":"2024-04-24T08:09:38.081017+0000","last_fullsized":"2024-04-24T08:09:38.081017+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083984+0000","last_change":"2024-04-24T08:09:38.083984+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083984+0000","last_undegraded":"2024-04-24T08:09:38.083984+0000","last_fullsized":"2024-04-24T08:09:38.083984+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092642+0000","last_change":"2024-04-24T08:09:38.092642+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092642+0000","last_undegraded":"2024-04-24T08:09:38.092642+0000","last_fullsized":"2024-04-24T08:09:38.092642+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089128+0000","last_change":"2024-04-24T08:09:38.089128+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089128+0000","last_undegraded":"2024-04-24T08:09:38.089128+0000","last_fullsized":"2024-04-24T08:09:38.089128+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.094332+0000","last_change":"2024-04-24T08:09:38.094332+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.094332+0000","last_undegraded":"2024-04-24T08:09:38.094332+0000","last_fullsized":"2024-04-24T08:09:38.094332+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083051+0000","last_change":"2024-04-24T08:09:38.083051+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083051+0000","last_undegraded":"2024-04-24T08:09:38.083051+0000","last_fullsized":"2024-04-24T08:09:38.083051+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087579+0000","last_change":"2024-04-24T08:09:38.087579+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087579+0000","last_undegraded":"2024-04-24T08:09:38.087579+0000","last_fullsized":"2024-04-24T08:09:38.087579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.094793+0000","last_change":"2024-04-24T08:09:38.094793+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.094793+0000","last_undegraded":"2024-04-24T08:09:38.094793+0000","last_fullsized":"2024-04-24T08:09:38.094793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.095355+0000","last_change":"2024-04-24T08:09:38.095355+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.095355+0000","last_undegraded":"2024-04-24T08:09:38.095355+0000","last_fullsized":"2024-04-24T08:09:38.095355+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.097440+0000","last_change":"2024-04-24T08:09:38.097440+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.097440+0000","last_undegraded":"2024-04-24T08:09:38.097440+0000","last_fullsized":"2024-04-24T08:09:38.097440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081278+0000","last_change":"2024-04-24T08:09:38.081278+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081278+0000","last_undegraded":"2024-04-24T08:09:38.081278+0000","last_fullsized":"2024-04-24T08:09:38.081278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.112465+0000","last_change":"2024-04-24T08:09:38.106509+0000","last_active":"2024-04-24T08:09:38.112465+0000","last_peered":"2024-04-24T08:09:38.112465+0000","last_clean":"2024-04-24T08:09:38.112465+0000","last_became_active":"2024-04-24T08:09:38.106216+0000","last_became_peered":"2024-04-24T08:09:38.106216+0000","last_unstale":"2024-04-24T08:09:38.112465+0000","last_undegraded":"2024-04-24T08:09:38.112465+0000","last_fullsized":"2024-04-24T08:09:38.112465+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:05.837977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090079+0000","last_change":"2024-04-24T08:09:38.090079+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090079+0000","last_undegraded":"2024-04-24T08:09:38.090079+0000","last_fullsized":"2024-04-24T08:09:38.090079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109428+0000","last_change":"2024-04-24T08:09:38.107316+0000","last_active":"2024-04-24T08:09:38.109428+0000","last_peered":"2024-04-24T08:09:38.109428+0000","last_clean":"2024-04-24T08:09:38.109428+0000","last_became_active":"2024-04-24T08:09:38.106891+0000","last_became_peered":"2024-04-24T08:09:38.106891+0000","last_unstale":"2024-04-24T08:09:38.109428+0000","last_undegraded":"2024-04-24T08:09:38.109428+0000","last_fullsized":"2024-04-24T08:09:38.109428+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092529+0000","last_change":"2024-04-24T08:09:38.092529+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092529+0000","last_undegraded":"2024-04-24T08:09:38.092529+0000","last_fullsized":"2024-04-24T08:09:38.092529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108092+0000","last_change":"2024-04-24T08:09:38.105975+0000","last_active":"2024-04-24T08:09:38.108092+0000","last_peered":"2024-04-24T08:09:38.108092+0000","last_clean":"2024-04-24T08:09:38.108092+0000","last_became_active":"2024-04-24T08:09:38.105684+0000","last_became_peered":"2024-04-24T08:09:38.105684+0000","last_unstale":"2024-04-24T08:09:38.108092+0000","last_undegraded":"2024-04-24T08:09:38.108092+0000","last_fullsized":"2024-04-24T08:09:38.108092+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087273+0000","last_change":"2024-04-24T08:09:38.087273+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087273+0000","last_undegraded":"2024-04-24T08:09:38.087273+0000","last_fullsized":"2024-04-24T08:09:38.087273+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.103902+0000","last_change":"2024-04-24T08:09:38.102564+0000","last_active":"2024-04-24T08:09:38.103902+0000","last_peered":"2024-04-24T08:09:38.103902+0000","last_clean":"2024-04-24T08:09:38.103902+0000","last_became_active":"2024-04-24T08:09:38.101598+0000","last_became_peered":"2024-04-24T08:09:38.101598+0000","last_unstale":"2024-04-24T08:09:38.103902+0000","last_undegraded":"2024-04-24T08:09:38.103902+0000","last_fullsized":"2024-04-24T08:09:38.103902+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:19:44.006523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093880+0000","last_change":"2024-04-24T08:09:38.093880+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093880+0000","last_undegraded":"2024-04-24T08:09:38.093880+0000","last_fullsized":"2024-04-24T08:09:38.093880+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109733+0000","last_change":"2024-04-24T08:09:38.101731+0000","last_active":"2024-04-24T08:09:38.109733+0000","last_peered":"2024-04-24T08:09:38.109733+0000","last_clean":"2024-04-24T08:09:38.109733+0000","last_became_active":"2024-04-24T08:09:38.100536+0000","last_became_peered":"2024-04-24T08:09:38.100536+0000","last_unstale":"2024-04-24T08:09:38.109733+0000","last_undegraded":"2024-04-24T08:09:38.109733+0000","last_fullsized":"2024-04-24T08:09:38.109733+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:17:11.041488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086737+0000","last_change":"2024-04-24T08:09:38.086737+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086737+0000","last_undegraded":"2024-04-24T08:09:38.086737+0000","last_fullsized":"2024-04-24T08:09:38.086737+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.112314+0000","last_change":"2024-04-24T08:09:38.108989+0000","last_active":"2024-04-24T08:09:38.112314+0000","last_peered":"2024-04-24T08:09:38.112314+0000","last_clean":"2024-04-24T08:09:38.112314+0000","last_became_active":"2024-04-24T08:09:38.108726+0000","last_became_peered":"2024-04-24T08:09:38.108726+0000","last_unstale":"2024-04-24T08:09:38.112314+0000","last_undegraded":"2024-04-24T08:09:38.112314+0000","last_fullsized":"2024-04-24T08:09:38.112314+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:40:46.556408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.091473+0000","last_change":"2024-04-24T08:09:38.091473+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.091473+0000","last_undegraded":"2024-04-24T08:09:38.091473+0000","last_fullsized":"2024-04-24T08:09:38.091473+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.097976+0000","last_change":"2024-04-24T08:09:38.093313+0000","last_active":"2024-04-24T08:09:38.097976+0000","last_peered":"2024-04-24T08:09:38.097976+0000","last_clean":"2024-04-24T08:09:38.097976+0000","last_became_active":"2024-04-24T08:09:38.092910+0000","last_became_peered":"2024-04-24T08:09:38.092910+0000","last_unstale":"2024-04-24T08:09:38.097976+0000","last_undegraded":"2024-04-24T08:09:38.097976+0000","last_fullsized":"2024-04-24T08:09:38.097976+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083631+0000","last_change":"2024-04-24T08:09:38.083631+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083631+0000","last_undegraded":"2024-04-24T08:09:38.083631+0000","last_fullsized":"2024-04-24T08:09:38.083631+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111408+0000","last_change":"2024-04-24T08:09:38.107454+0000","last_active":"2024-04-24T08:09:38.111408+0000","last_peered":"2024-04-24T08:09:38.111408+0000","last_clean":"2024-04-24T08:09:38.111408+0000","last_became_active":"2024-04-24T08:09:38.107149+0000","last_became_peered":"2024-04-24T08:09:38.107149+0000","last_unstale":"2024-04-24T08:09:38.111408+0000","last_undegraded":"2024-04-24T08:09:38.111408+0000","last_fullsized":"2024-04-24T08:09:38.111408+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:46:32.216293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087267+0000","last_change":"2024-04-24T08:09:38.087267+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087267+0000","last_undegraded":"2024-04-24T08:09:38.087267+0000","last_fullsized":"2024-04-24T08:09:38.087267+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111453+0000","last_change":"2024-04-24T08:09:38.110451+0000","last_active":"2024-04-24T08:09:38.111453+0000","last_peered":"2024-04-24T08:09:38.111453+0000","last_clean":"2024-04-24T08:09:38.111453+0000","last_became_active":"2024-04-24T08:09:38.110191+0000","last_became_peered":"2024-04-24T08:09:38.110191+0000","last_unstale":"2024-04-24T08:09:38.111453+0000","last_undegraded":"2024-04-24T08:09:38.111453+0000","last_fullsized":"2024-04-24T08:09:38.111453+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:10.779345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090571+0000","last_change":"2024-04-24T08:09:38.090571+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090571+0000","last_undegraded":"2024-04-24T08:09:38.090571+0000","last_fullsized":"2024-04-24T08:09:38.090571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.095062+0000","last_change":"2024-04-24T08:09:38.093711+0000","last_active":"2024-04-24T08:09:38.095062+0000","last_peered":"2024-04-24T08:09:38.095062+0000","last_clean":"2024-04-24T08:09:38.095062+0000","last_became_active":"2024-04-24T08:09:38.091626+0000","last_became_peered":"2024-04-24T08:09:38.091626+0000","last_unstale":"2024-04-24T08:09:38.095062+0000","last_undegraded":"2024-04-24T08:09:38.095062+0000","last_fullsized":"2024-04-24T08:09:38.095062+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.085881+0000","last_change":"2024-04-24T08:09:38.085881+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.085881+0000","last_undegraded":"2024-04-24T08:09:38.085881+0000","last_fullsized":"2024-04-24T08:09:38.085881+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109912+0000","last_change":"2024-04-24T08:09:38.107829+0000","last_active":"2024-04-24T08:09:38.109912+0000","last_peered":"2024-04-24T08:09:38.109912+0000","last_clean":"2024-04-24T08:09:38.109912+0000","last_became_active":"2024-04-24T08:09:38.107551+0000","last_became_peered":"2024-04-24T08:09:38.107551+0000","last_unstale":"2024-04-24T08:09:38.109912+0000","last_undegraded":"2024-04-24T08:09:38.109912+0000","last_fullsized":"2024-04-24T08:09:38.109912+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086224+0000","last_change":"2024-04-24T08:09:38.086224+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086224+0000","last_undegraded":"2024-04-24T08:09:38.086224+0000","last_fullsized":"2024-04-24T08:09:38.086224+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108172+0000","last_change":"2024-04-24T08:09:38.106294+0000","last_active":"2024-04-24T08:09:38.108172+0000","last_peered":"2024-04-24T08:09:38.108172+0000","last_clean":"2024-04-24T08:09:38.108172+0000","last_became_active":"2024-04-24T08:09:38.105994+0000","last_became_peered":"2024-04-24T08:09:38.105994+0000","last_unstale":"2024-04-24T08:09:38.108172+0000","last_undegraded":"2024-04-24T08:09:38.108172+0000","last_fullsized":"2024-04-24T08:09:38.108172+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:11.064191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084768+0000","last_change":"2024-04-24T08:09:38.084768+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084768+0000","last_undegraded":"2024-04-24T08:09:38.084768+0000","last_fullsized":"2024-04-24T08:09:38.084768+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111341+0000","last_change":"2024-04-24T08:09:38.105998+0000","last_active":"2024-04-24T08:09:38.111341+0000","last_peered":"2024-04-24T08:09:38.111341+0000","last_clean":"2024-04-24T08:09:38.111341+0000","last_became_active":"2024-04-24T08:09:38.105710+0000","last_became_peered":"2024-04-24T08:09:38.105710+0000","last_unstale":"2024-04-24T08:09:38.111341+0000","last_undegraded":"2024-04-24T08:09:38.111341+0000","last_fullsized":"2024-04-24T08:09:38.111341+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:58:11.279574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084682+0000","last_change":"2024-04-24T08:09:38.084682+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084682+0000","last_undegraded":"2024-04-24T08:09:38.084682+0000","last_fullsized":"2024-04-24T08:09:38.084682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109784+0000","last_change":"2024-04-24T08:09:38.107964+0000","last_active":"2024-04-24T08:09:38.109784+0000","last_peered":"2024-04-24T08:09:38.109784+0000","last_clean":"2024-04-24T08:09:38.109784+0000","last_became_active":"2024-04-24T08:09:38.107620+0000","last_became_peered":"2024-04-24T08:09:38.107620+0000","last_unstale":"2024-04-24T08:09:38.109784+0000","last_undegraded":"2024-04-24T08:09:38.109784+0000","last_fullsized":"2024-04-24T08:09:38.109784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.096631+0000","last_change":"2024-04-24T08:09:38.096631+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.096631+0000","last_undegraded":"2024-04-24T08:09:38.096631+0000","last_fullsized":"2024-04-24T08:09:38.096631+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.106220+0000","last_change":"2024-04-24T08:09:38.103294+0000","last_active":"2024-04-24T08:09:38.106220+0000","last_peered":"2024-04-24T08:09:38.106220+0000","last_clean":"2024-04-24T08:09:38.106220+0000","last_became_active":"2024-04-24T08:09:38.101762+0000","last_became_peered":"2024-04-24T08:09:38.101762+0000","last_unstale":"2024-04-24T08:09:38.106220+0000","last_undegraded":"2024-04-24T08:09:38.106220+0000","last_fullsized":"2024-04-24T08:09:38.106220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088094+0000","last_change":"2024-04-24T08:09:38.088094+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088094+0000","last_undegraded":"2024-04-24T08:09:38.088094+0000","last_fullsized":"2024-04-24T08:09:38.088094+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.103953+0000","last_change":"2024-04-24T08:09:38.100426+0000","last_active":"2024-04-24T08:09:38.103953+0000","last_peered":"2024-04-24T08:09:38.103953+0000","last_clean":"2024-04-24T08:09:38.103953+0000","last_became_active":"2024-04-24T08:09:38.100138+0000","last_became_peered":"2024-04-24T08:09:38.100138+0000","last_unstale":"2024-04-24T08:09:38.103953+0000","last_undegraded":"2024-04-24T08:09:38.103953+0000","last_fullsized":"2024-04-24T08:09:38.103953+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089848+0000","last_change":"2024-04-24T08:09:38.089848+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089848+0000","last_undegraded":"2024-04-24T08:09:38.089848+0000","last_fullsized":"2024-04-24T08:09:38.089848+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.100103+0000","last_change":"2024-04-24T08:09:38.096536+0000","last_active":"2024-04-24T08:09:38.100103+0000","last_peered":"2024-04-24T08:09:38.100103+0000","last_clean":"2024-04-24T08:09:38.100103+0000","last_became_active":"2024-04-24T08:09:38.095915+0000","last_became_peered":"2024-04-24T08:09:38.095915+0000","last_unstale":"2024-04-24T08:09:38.100103+0000","last_undegraded":"2024-04-24T08:09:38.100103+0000","last_fullsized":"2024-04-24T08:09:38.100103+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:05.837977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088654+0000","last_change":"2024-04-24T08:09:38.088654+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088654+0000","last_undegraded":"2024-04-24T08:09:38.088654+0000","last_fullsized":"2024-04-24T08:09:38.088654+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109578+0000","last_change":"2024-04-24T08:09:38.101152+0000","last_active":"2024-04-24T08:09:38.109578+0000","last_peered":"2024-04-24T08:09:38.109578+0000","last_clean":"2024-04-24T08:09:38.109578+0000","last_became_active":"2024-04-24T08:09:38.099292+0000","last_became_peered":"2024-04-24T08:09:38.099292+0000","last_unstale":"2024-04-24T08:09:38.109578+0000","last_undegraded":"2024-04-24T08:09:38.109578+0000","last_fullsized":"2024-04-24T08:09:38.109578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:05:05.064870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087579+0000","last_change":"2024-04-24T08:09:38.087579+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087579+0000","last_undegraded":"2024-04-24T08:09:38.087579+0000","last_fullsized":"2024-04-24T08:09:38.087579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109637+0000","last_change":"2024-04-24T08:09:38.100668+0000","last_active":"2024-04-24T08:09:38.109637+0000","last_peered":"2024-04-24T08:09:38.109637+0000","last_clean":"2024-04-24T08:09:38.109637+0000","last_became_active":"2024-04-24T08:09:38.100303+0000","last_became_peered":"2024-04-24T08:09:38.100303+0000","last_unstale":"2024-04-24T08:09:38.109637+0000","last_undegraded":"2024-04-24T08:09:38.109637+0000","last_fullsized":"2024-04-24T08:09:38.109637+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:55.861909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089628+0000","last_change":"2024-04-24T08:09:38.089628+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089628+0000","last_undegraded":"2024-04-24T08:09:38.089628+0000","last_fullsized":"2024-04-24T08:09:38.089628+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111334+0000","last_change":"2024-04-24T08:09:38.105765+0000","last_active":"2024-04-24T08:09:38.111334+0000","last_peered":"2024-04-24T08:09:38.111334+0000","last_clean":"2024-04-24T08:09:38.111334+0000","last_became_active":"2024-04-24T08:09:38.100242+0000","last_became_peered":"2024-04-24T08:09:38.100242+0000","last_unstale":"2024-04-24T08:09:38.111334+0000","last_undegraded":"2024-04-24T08:09:38.111334+0000","last_fullsized":"2024-04-24T08:09:38.111334+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:19:44.006523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084377+0000","last_change":"2024-04-24T08:09:38.084377+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084377+0000","last_undegraded":"2024-04-24T08:09:38.084377+0000","last_fullsized":"2024-04-24T08:09:38.084377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.103077+0000","last_change":"2024-04-24T08:09:38.097858+0000","last_active":"2024-04-24T08:09:38.103077+0000","last_peered":"2024-04-24T08:09:38.103077+0000","last_clean":"2024-04-24T08:09:38.103077+0000","last_became_active":"2024-04-24T08:09:38.097599+0000","last_became_peered":"2024-04-24T08:09:38.097599+0000","last_unstale":"2024-04-24T08:09:38.103077+0000","last_undegraded":"2024-04-24T08:09:38.103077+0000","last_fullsized":"2024-04-24T08:09:38.103077+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086095+0000","last_change":"2024-04-24T08:09:38.086095+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086095+0000","last_undegraded":"2024-04-24T08:09:38.086095+0000","last_fullsized":"2024-04-24T08:09:38.086095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.115052+0000","last_change":"2024-04-24T08:09:38.110759+0000","last_active":"2024-04-24T08:09:38.115052+0000","last_peered":"2024-04-24T08:09:38.115052+0000","last_clean":"2024-04-24T08:09:38.115052+0000","last_became_active":"2024-04-24T08:09:38.110393+0000","last_became_peered":"2024-04-24T08:09:38.110393+0000","last_unstale":"2024-04-24T08:09:38.115052+0000","last_undegraded":"2024-04-24T08:09:38.115052+0000","last_fullsized":"2024-04-24T08:09:38.115052+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087968+0000","last_change":"2024-04-24T08:09:38.087968+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087968+0000","last_undegraded":"2024-04-24T08:09:38.087968+0000","last_fullsized":"2024-04-24T08:09:38.087968+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111161+0000","last_change":"2024-04-24T08:09:38.104959+0000","last_active":"2024-04-24T08:09:38.111161+0000","last_peered":"2024-04-24T08:09:38.111161+0000","last_clean":"2024-04-24T08:09:38.111161+0000","last_became_active":"2024-04-24T08:09:38.104606+0000","last_became_peered":"2024-04-24T08:09:38.104606+0000","last_unstale":"2024-04-24T08:09:38.111161+0000","last_undegraded":"2024-04-24T08:09:38.111161+0000","last_fullsized":"2024-04-24T08:09:38.111161+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:32.287730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.095292+0000","last_change":"2024-04-24T08:09:38.095292+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.095292+0000","last_undegraded":"2024-04-24T08:09:38.095292+0000","last_fullsized":"2024-04-24T08:09:38.095292+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.106663+0000","last_change":"2024-04-24T08:09:38.104201+0000","last_active":"2024-04-24T08:09:38.106663+0000","last_peered":"2024-04-24T08:09:38.106663+0000","last_clean":"2024-04-24T08:09:38.106663+0000","last_became_active":"2024-04-24T08:09:38.103332+0000","last_became_peered":"2024-04-24T08:09:38.103332+0000","last_unstale":"2024-04-24T08:09:38.106663+0000","last_undegraded":"2024-04-24T08:09:38.106663+0000","last_fullsized":"2024-04-24T08:09:38.106663+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090440+0000","last_change":"2024-04-24T08:09:38.090440+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090440+0000","last_undegraded":"2024-04-24T08:09:38.090440+0000","last_fullsized":"2024-04-24T08:09:38.090440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108613+0000","last_change":"2024-04-24T08:09:38.105102+0000","last_active":"2024-04-24T08:09:38.108613+0000","last_peered":"2024-04-24T08:09:38.108613+0000","last_clean":"2024-04-24T08:09:38.108613+0000","last_became_active":"2024-04-24T08:09:38.104814+0000","last_became_peered":"2024-04-24T08:09:38.104814+0000","last_unstale":"2024-04-24T08:09:38.108613+0000","last_undegraded":"2024-04-24T08:09:38.108613+0000","last_fullsized":"2024-04-24T08:09:38.108613+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087435+0000","last_change":"2024-04-24T08:09:38.087435+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087435+0000","last_undegraded":"2024-04-24T08:09:38.087435+0000","last_fullsized":"2024-04-24T08:09:38.087435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.104504+0000","last_change":"2024-04-24T08:09:38.102570+0000","last_active":"2024-04-24T08:09:38.104504+0000","last_peered":"2024-04-24T08:09:38.104504+0000","last_clean":"2024-04-24T08:09:38.104504+0000","last_became_active":"2024-04-24T08:09:38.101279+0000","last_became_peered":"2024-04-24T08:09:38.101279+0000","last_unstale":"2024-04-24T08:09:38.104504+0000","last_undegraded":"2024-04-24T08:09:38.104504+0000","last_fullsized":"2024-04-24T08:09:38.104504+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081042+0000","last_change":"2024-04-24T08:09:38.081042+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081042+0000","last_undegraded":"2024-04-24T08:09:38.081042+0000","last_fullsized":"2024-04-24T08:09:38.081042+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.105348+0000","last_change":"2024-04-24T08:09:38.101682+0000","last_active":"2024-04-24T08:09:38.105348+0000","last_peered":"2024-04-24T08:09:38.105348+0000","last_clean":"2024-04-24T08:09:38.105348+0000","last_became_active":"2024-04-24T08:09:38.099204+0000","last_became_peered":"2024-04-24T08:09:38.099204+0000","last_unstale":"2024-04-24T08:09:38.105348+0000","last_undegraded":"2024-04-24T08:09:38.105348+0000","last_fullsized":"2024-04-24T08:09:38.105348+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089049+0000","last_change":"2024-04-24T08:09:38.089049+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089049+0000","last_undegraded":"2024-04-24T08:09:38.089049+0000","last_fullsized":"2024-04-24T08:09:38.089049+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.104376+0000","last_change":"2024-04-24T08:09:38.101668+0000","last_active":"2024-04-24T08:09:38.104376+0000","last_peered":"2024-04-24T08:09:38.104376+0000","last_clean":"2024-04-24T08:09:38.104376+0000","last_became_active":"2024-04-24T08:09:38.101318+0000","last_became_peered":"2024-04-24T08:09:38.101318+0000","last_unstale":"2024-04-24T08:09:38.104376+0000","last_undegraded":"2024-04-24T08:09:38.104376+0000","last_fullsized":"2024-04-24T08:09:38.104376+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081090+0000","last_change":"2024-04-24T08:09:38.081090+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081090+0000","last_undegraded":"2024-04-24T08:09:38.081090+0000","last_fullsized":"2024-04-24T08:09:38.081090+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108737+0000","last_change":"2024-04-24T08:09:38.105797+0000","last_active":"2024-04-24T08:09:38.108737+0000","last_peered":"2024-04-24T08:09:38.108737+0000","last_clean":"2024-04-24T08:09:38.108737+0000","last_became_active":"2024-04-24T08:09:38.104048+0000","last_became_peered":"2024-04-24T08:09:38.104048+0000","last_unstale":"2024-04-24T08:09:38.108737+0000","last_undegraded":"2024-04-24T08:09:38.108737+0000","last_fullsized":"2024-04-24T08:09:38.108737+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083903+0000","last_change":"2024-04-24T08:09:38.083903+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083903+0000","last_undegraded":"2024-04-24T08:09:38.083903+0000","last_fullsized":"2024-04-24T08:09:38.083903+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.110901+0000","last_change":"2024-04-24T08:09:38.103639+0000","last_active":"2024-04-24T08:09:38.110901+0000","last_peered":"2024-04-24T08:09:38.110901+0000","last_clean":"2024-04-24T08:09:38.110901+0000","last_became_active":"2024-04-24T08:09:38.103230+0000","last_became_peered":"2024-04-24T08:09:38.103230+0000","last_unstale":"2024-04-24T08:09:38.110901+0000","last_undegraded":"2024-04-24T08:09:38.110901+0000","last_fullsized":"2024-04-24T08:09:38.110901+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:27.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092331+0000","last_change":"2024-04-24T08:09:38.092331+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092331+0000","last_undegraded":"2024-04-24T08:09:38.092331+0000","last_fullsized":"2024-04-24T08:09:38.092331+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108778+0000","last_change":"2024-04-24T08:09:38.107831+0000","last_active":"2024-04-24T08:09:38.108778+0000","last_peered":"2024-04-24T08:09:38.108778+0000","last_clean":"2024-04-24T08:09:38.108778+0000","last_became_active":"2024-04-24T08:09:38.107509+0000","last_became_peered":"2024-04-24T08:09:38.107509+0000","last_unstale":"2024-04-24T08:09:38.108778+0000","last_undegraded":"2024-04-24T08:09:38.108778+0000","last_fullsized":"2024-04-24T08:09:38.108778+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:27.457454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.099092+0000","last_change":"2024-04-24T08:09:38.099092+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.099092+0000","last_undegraded":"2024-04-24T08:09:38.099092+0000","last_fullsized":"2024-04-24T08:09:38.099092+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.117716+0000","last_change":"2024-04-24T08:09:38.112443+0000","last_active":"2024-04-24T08:09:38.117716+0000","last_peered":"2024-04-24T08:09:38.117716+0000","last_clean":"2024-04-24T08:09:38.117716+0000","last_became_active":"2024-04-24T08:09:38.111901+0000","last_became_peered":"2024-04-24T08:09:38.111901+0000","last_unstale":"2024-04-24T08:09:38.117716+0000","last_undegraded":"2024-04-24T08:09:38.117716+0000","last_fullsized":"2024-04-24T08:09:38.117716+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.106310+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:09:38.106310+0000","last_peered":"2024-04-24T08:09:38.106310+0000","last_clean":"2024-04-24T08:09:38.106310+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:09:38.106310+0000","last_undegraded":"2024-04-24T08:09:38.106310+0000","last_fullsized":"2024-04-24T08:09:38.106310+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":724063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:09:44.059 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:09:44.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.299+0000 7f1065800640 1 Processor -- start 2024-04-24T08:09:44.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.299+0000 7f1065800640 1 -- start start 2024-04-24T08:09:44.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.300+0000 7f1065800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f106015b490 0x7f10600b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f1065800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f106006f2d0 0x7f106006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f1065800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10600b46e0 con 0x7f10600cfbe0 2024-04-24T08:09:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f1065800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f106006fc70 con 0x7f106015b490 2024-04-24T08:09:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f1065800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f10600d0720 con 0x7f106006f2d0 2024-04-24T08:09:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f105ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f106015b490 0x7f10600b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f105ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f106015b490 0x7f10600b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60690/0 (socket says 172.21.15.69:60690) 2024-04-24T08:09:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f105ea00640 1 -- 172.21.15.69:0/250379648 learned_addr learned my addr 172.21.15.69:0/250379648 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f105ea00640 1 -- 172.21.15.69:0/250379648 >> v2:172.21.15.121:3300/0 conn(0x7f106006f2d0 msgr2=0x7f106006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f105ea00640 1 --2- 172.21.15.69:0/250379648 >> v2:172.21.15.121:3300/0 conn(0x7f106006f2d0 0x7f106006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f105ea00640 1 -- 172.21.15.69:0/250379648 >> v1:172.21.15.121:6789/0 conn(0x7f10600cfbe0 legacy=0x7f10600d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:44.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.301+0000 7f105ea00640 1 -- 172.21.15.69:0/250379648 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10600d0960 con 0x7f106015b490 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.302+0000 7f105ea00640 1 --2- 172.21.15.69:0/250379648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 0x7f10600b5180 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1060072fa0 tx=0x7f105002c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6947b2af1f2ae7d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f105e000640 1 -- 172.21.15.69:0/250379648 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1050038070 con 0x7f106015b490 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f105e000640 1 -- 172.21.15.69:0/250379648 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1050031710 con 0x7f106015b490 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f105e000640 1 -- 172.21.15.69:0/250379648 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1050031ae0 con 0x7f106015b490 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f1065800640 1 -- 172.21.15.69:0/250379648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 msgr2=0x7f10600b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f1065800640 1 --2- 172.21.15.69:0/250379648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 0x7f10600b5180 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1060072fa0 tx=0x7f105002c420 comp rx=0 tx=0).stop 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f1065800640 1 -- 172.21.15.69:0/250379648 shutdown_connections 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f1065800640 1 --2- 172.21.15.69:0/250379648 >> v2:172.21.15.121:3300/0 conn(0x7f106006f2d0 0x7f106006f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f1065800640 1 --2- 172.21.15.69:0/250379648 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 0x7f10600b5180 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f1065800640 1 -- 172.21.15.69:0/250379648 >> 172.21.15.69:0/250379648 conn(0x7f1060067d90 msgr2=0x7f106006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f1065800640 1 -- 172.21.15.69:0/250379648 shutdown_connections 2024-04-24T08:09:44.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.303+0000 7f1065800640 1 -- 172.21.15.69:0/250379648 wait complete. 2024-04-24T08:09:44.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.305+0000 7f1065800640 1 Processor -- start 2024-04-24T08:09:44.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.305+0000 7f1065800640 1 -- start start 2024-04-24T08:09:44.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.306+0000 7f1065800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106006f2d0 0x7f106014e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.306+0000 7f1065800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10600cfbe0 0x7f10601f87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.306+0000 7f1065800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 0x7f10601f4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.306+0000 7f1065800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f10600d0c40 con 0x7f106015b490 2024-04-24T08:09:44.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.306+0000 7f1065800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f10600b4c60 con 0x7f10600cfbe0 2024-04-24T08:09:44.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.306+0000 7f1065800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f106015b870 con 0x7f106006f2d0 2024-04-24T08:09:44.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.307+0000 7f105f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 0x7f10601f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.307+0000 7f105f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 0x7f10601f4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60694/0 (socket says 172.21.15.69:60694) 2024-04-24T08:09:44.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.307+0000 7f105f400640 1 -- 172.21.15.69:0/42186248 learned_addr learned my addr 172.21.15.69:0/42186248 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:44.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.307+0000 7f105ea00640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106006f2d0 0x7f106014e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.307+0000 7f1057e00640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10600cfbe0 0x7f10601f87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.307+0000 7f105f400640 1 -- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106006f2d0 msgr2=0x7f106014e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.307+0000 7f105f400640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106006f2d0 0x7f106014e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.307+0000 7f105f400640 1 -- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10600cfbe0 msgr2=0x7f10601f87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.307+0000 7f105f400640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10600cfbe0 0x7f10601f87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.307+0000 7f105f400640 1 -- 172.21.15.69:0/42186248 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10601f8ce0 con 0x7f106015b490 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.308+0000 7f105f400640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 0x7f10601f4830 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f105800c780 tx=0x7f105800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.308+0000 7f1056a00640 1 -- 172.21.15.69:0/42186248 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058017070 con 0x7f106015b490 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.308+0000 7f1056a00640 1 -- 172.21.15.69:0/42186248 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1058003b70 con 0x7f106015b490 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.308+0000 7f1056a00640 1 -- 172.21.15.69:0/42186248 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058005070 con 0x7f106015b490 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.308+0000 7f1065800640 1 -- 172.21.15.69:0/42186248 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10601f5210 con 0x7f106015b490 2024-04-24T08:09:44.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.308+0000 7f1065800640 1 -- 172.21.15.69:0/42186248 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10601f7fa0 con 0x7f106015b490 2024-04-24T08:09:44.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.309+0000 7f1065800640 1 -- 172.21.15.69:0/42186248 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10601f5730 con 0x7f106015b490 2024-04-24T08:09:44.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.310+0000 7f1056a00640 1 -- 172.21.15.69:0/42186248 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1058005210 con 0x7f106015b490 2024-04-24T08:09:44.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.311+0000 7f1056a00640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f103c06c920 0x7f103c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.311+0000 7f1056a00640 1 -- 172.21.15.69:0/42186248 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f105808af80 con 0x7f106015b490 2024-04-24T08:09:44.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.312+0000 7f105ea00640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f103c06c920 0x7f103c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.313+0000 7f105ea00640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f103c06c920 0x7f103c06ede0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1050035000 tx=0x7f1050008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:44.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.315+0000 7f1056a00640 1 -- 172.21.15.69:0/42186248 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f10601f8ce0 con 0x7f106015b490 2024-04-24T08:09:44.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.516+0000 7f1065800640 1 -- 172.21.15.69:0/42186248 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1060068a80 con 0x7f103c06c920 2024-04-24T08:09:44.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.534+0000 7f1056a00640 1 -- 172.21.15.69:0/42186248 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246635 (secure 0 0 0) 0x7f1060068a80 con 0x7f103c06c920 2024-04-24T08:09:44.537 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:44.538 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:09:44.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.538+0000 7f1065800640 1 -- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f103c06c920 msgr2=0x7f103c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.538+0000 7f1065800640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f103c06c920 0x7f103c06ede0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1050035000 tx=0x7f1050008030 comp rx=0 tx=0).stop 2024-04-24T08:09:44.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.538+0000 7f1065800640 1 -- 172.21.15.69:0/42186248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 msgr2=0x7f10601f4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.538+0000 7f1065800640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 0x7f10601f4830 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f105800c780 tx=0x7f105800cb80 comp rx=0 tx=0).stop 2024-04-24T08:09:44.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.538+0000 7f1065800640 1 -- 172.21.15.69:0/42186248 shutdown_connections 2024-04-24T08:09:44.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.539+0000 7f1065800640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f103c06c920 0x7f103c06ede0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.539+0000 7f1065800640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f106015b490 0x7f10601f4830 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.539+0000 7f1065800640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10600cfbe0 0x7f10601f87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.539+0000 7f1065800640 1 --2- 172.21.15.69:0/42186248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f106006f2d0 0x7f106014e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.539+0000 7f1065800640 1 -- 172.21.15.69:0/42186248 >> 172.21.15.69:0/42186248 conn(0x7f1060067d90 msgr2=0x7f1060072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:44.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.539+0000 7f1065800640 1 -- 172.21.15.69:0/42186248 shutdown_connections 2024-04-24T08:09:44.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.539+0000 7f1065800640 1 -- 172.21.15.69:0/42186248 wait complete. 2024-04-24T08:09:44.556 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-04-24T08:09:42.690265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":58,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220040,"kb_used_data":3856,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754680,"statfs":{"total":773094113280,"available":772868792320,"internally_reserved":0,"allocated":3948544,"data_stored":2237762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.631516"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.091995+0000","last_change":"2024-04-24T08:09:38.091995+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.091995+0000","last_undegraded":"2024-04-24T08:09:38.091995+0000","last_fullsized":"2024-04-24T08:09:38.091995+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090575+0000","last_change":"2024-04-24T08:09:38.090575+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090575+0000","last_undegraded":"2024-04-24T08:09:38.090575+0000","last_fullsized":"2024-04-24T08:09:38.090575+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.085165+0000","last_change":"2024-04-24T08:09:38.085165+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.085165+0000","last_undegraded":"2024-04-24T08:09:38.085165+0000","last_fullsized":"2024-04-24T08:09:38.085165+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092285+0000","last_change":"2024-04-24T08:09:38.092285+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092285+0000","last_undegraded":"2024-04-24T08:09:38.092285+0000","last_fullsized":"2024-04-24T08:09:38.092285+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087102+0000","last_change":"2024-04-24T08:09:38.087102+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087102+0000","last_undegraded":"2024-04-24T08:09:38.087102+0000","last_fullsized":"2024-04-24T08:09:38.087102+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088583+0000","last_change":"2024-04-24T08:09:38.088583+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088583+0000","last_undegraded":"2024-04-24T08:09:38.088583+0000","last_fullsized":"2024-04-24T08:09:38.088583+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093698+0000","last_change":"2024-04-24T08:09:38.093698+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093698+0000","last_undegraded":"2024-04-24T08:09:38.093698+0000","last_fullsized":"2024-04-24T08:09:38.093698+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.098131+0000","last_change":"2024-04-24T08:09:38.098131+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.098131+0000","last_undegraded":"2024-04-24T08:09:38.098131+0000","last_fullsized":"2024-04-24T08:09:38.098131+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.095971+0000","last_change":"2024-04-24T08:09:38.095971+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.095971+0000","last_undegraded":"2024-04-24T08:09:38.095971+0000","last_fullsized":"2024-04-24T08:09:38.095971+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093218+0000","last_change":"2024-04-24T08:09:38.093218+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093218+0000","last_undegraded":"2024-04-24T08:09:38.093218+0000","last_fullsized":"2024-04-24T08:09:38.093218+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.098579+0000","last_change":"2024-04-24T08:09:38.098579+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.098579+0000","last_undegraded":"2024-04-24T08:09:38.098579+0000","last_fullsized":"2024-04-24T08:09:38.098579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084662+0000","last_change":"2024-04-24T08:09:38.084662+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084662+0000","last_undegraded":"2024-04-24T08:09:38.084662+0000","last_fullsized":"2024-04-24T08:09:38.084662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086610+0000","last_change":"2024-04-24T08:09:38.086610+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086610+0000","last_undegraded":"2024-04-24T08:09:38.086610+0000","last_fullsized":"2024-04-24T08:09:38.086610+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087851+0000","last_change":"2024-04-24T08:09:38.087851+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087851+0000","last_undegraded":"2024-04-24T08:09:38.087851+0000","last_fullsized":"2024-04-24T08:09:38.087851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088469+0000","last_change":"2024-04-24T08:09:38.088469+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088469+0000","last_undegraded":"2024-04-24T08:09:38.088469+0000","last_fullsized":"2024-04-24T08:09:38.088469+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088695+0000","last_change":"2024-04-24T08:09:38.088695+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088695+0000","last_undegraded":"2024-04-24T08:09:38.088695+0000","last_fullsized":"2024-04-24T08:09:38.088695+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.102476+0000","last_change":"2024-04-24T08:09:38.102476+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.102476+0000","last_undegraded":"2024-04-24T08:09:38.102476+0000","last_fullsized":"2024-04-24T08:09:38.102476+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089933+0000","last_change":"2024-04-24T08:09:38.089933+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089933+0000","last_undegraded":"2024-04-24T08:09:38.089933+0000","last_fullsized":"2024-04-24T08:09:38.089933+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086859+0000","last_change":"2024-04-24T08:09:38.086859+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086859+0000","last_undegraded":"2024-04-24T08:09:38.086859+0000","last_fullsized":"2024-04-24T08:09:38.086859+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093820+0000","last_change":"2024-04-24T08:09:38.093820+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093820+0000","last_undegraded":"2024-04-24T08:09:38.093820+0000","last_fullsized":"2024-04-24T08:09:38.093820+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.096265+0000","last_change":"2024-04-24T08:09:38.096265+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.096265+0000","last_undegraded":"2024-04-24T08:09:38.096265+0000","last_fullsized":"2024-04-24T08:09:38.096265+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.094464+0000","last_change":"2024-04-24T08:09:38.094464+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.094464+0000","last_undegraded":"2024-04-24T08:09:38.094464+0000","last_fullsized":"2024-04-24T08:09:38.094464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081017+0000","last_change":"2024-04-24T08:09:38.081017+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081017+0000","last_undegraded":"2024-04-24T08:09:38.081017+0000","last_fullsized":"2024-04-24T08:09:38.081017+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083984+0000","last_change":"2024-04-24T08:09:38.083984+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083984+0000","last_undegraded":"2024-04-24T08:09:38.083984+0000","last_fullsized":"2024-04-24T08:09:38.083984+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092642+0000","last_change":"2024-04-24T08:09:38.092642+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092642+0000","last_undegraded":"2024-04-24T08:09:38.092642+0000","last_fullsized":"2024-04-24T08:09:38.092642+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089128+0000","last_change":"2024-04-24T08:09:38.089128+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089128+0000","last_undegraded":"2024-04-24T08:09:38.089128+0000","last_fullsized":"2024-04-24T08:09:38.089128+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.094332+0000","last_change":"2024-04-24T08:09:38.094332+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.094332+0000","last_undegraded":"2024-04-24T08:09:38.094332+0000","last_fullsized":"2024-04-24T08:09:38.094332+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083051+0000","last_change":"2024-04-24T08:09:38.083051+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083051+0000","last_undegraded":"2024-04-24T08:09:38.083051+0000","last_fullsized":"2024-04-24T08:09:38.083051+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087579+0000","last_change":"2024-04-24T08:09:38.087579+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087579+0000","last_undegraded":"2024-04-24T08:09:38.087579+0000","last_fullsized":"2024-04-24T08:09:38.087579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.094793+0000","last_change":"2024-04-24T08:09:38.094793+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.094793+0000","last_undegraded":"2024-04-24T08:09:38.094793+0000","last_fullsized":"2024-04-24T08:09:38.094793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.095355+0000","last_change":"2024-04-24T08:09:38.095355+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.095355+0000","last_undegraded":"2024-04-24T08:09:38.095355+0000","last_fullsized":"2024-04-24T08:09:38.095355+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.097440+0000","last_change":"2024-04-24T08:09:38.097440+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.097440+0000","last_undegraded":"2024-04-24T08:09:38.097440+0000","last_fullsized":"2024-04-24T08:09:38.097440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081278+0000","last_change":"2024-04-24T08:09:38.081278+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081278+0000","last_undegraded":"2024-04-24T08:09:38.081278+0000","last_fullsized":"2024-04-24T08:09:38.081278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.112465+0000","last_change":"2024-04-24T08:09:38.106509+0000","last_active":"2024-04-24T08:09:38.112465+0000","last_peered":"2024-04-24T08:09:38.112465+0000","last_clean":"2024-04-24T08:09:38.112465+0000","last_became_active":"2024-04-24T08:09:38.106216+0000","last_became_peered":"2024-04-24T08:09:38.106216+0000","last_unstale":"2024-04-24T08:09:38.112465+0000","last_undegraded":"2024-04-24T08:09:38.112465+0000","last_fullsized":"2024-04-24T08:09:38.112465+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:05.837977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090079+0000","last_change":"2024-04-24T08:09:38.090079+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090079+0000","last_undegraded":"2024-04-24T08:09:38.090079+0000","last_fullsized":"2024-04-24T08:09:38.090079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109428+0000","last_change":"2024-04-24T08:09:38.107316+0000","last_active":"2024-04-24T08:09:38.109428+0000","last_peered":"2024-04-24T08:09:38.109428+0000","last_clean":"2024-04-24T08:09:38.109428+0000","last_became_active":"2024-04-24T08:09:38.106891+0000","last_became_peered":"2024-04-24T08:09:38.106891+0000","last_unstale":"2024-04-24T08:09:38.109428+0000","last_undegraded":"2024-04-24T08:09:38.109428+0000","last_fullsized":"2024-04-24T08:09:38.109428+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092529+0000","last_change":"2024-04-24T08:09:38.092529+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092529+0000","last_undegraded":"2024-04-24T08:09:38.092529+0000","last_fullsized":"2024-04-24T08:09:38.092529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108092+0000","last_change":"2024-04-24T08:09:38.105975+0000","last_active":"2024-04-24T08:09:38.108092+0000","last_peered":"2024-04-24T08:09:38.108092+0000","last_clean":"2024-04-24T08:09:38.108092+0000","last_became_active":"2024-04-24T08:09:38.105684+0000","last_became_peered":"2024-04-24T08:09:38.105684+0000","last_unstale":"2024-04-24T08:09:38.108092+0000","last_undegraded":"2024-04-24T08:09:38.108092+0000","last_fullsized":"2024-04-24T08:09:38.108092+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087273+0000","last_change":"2024-04-24T08:09:38.087273+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087273+0000","last_undegraded":"2024-04-24T08:09:38.087273+0000","last_fullsized":"2024-04-24T08:09:38.087273+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.103902+0000","last_change":"2024-04-24T08:09:38.102564+0000","last_active":"2024-04-24T08:09:38.103902+0000","last_peered":"2024-04-24T08:09:38.103902+0000","last_clean":"2024-04-24T08:09:38.103902+0000","last_became_active":"2024-04-24T08:09:38.101598+0000","last_became_peered":"2024-04-24T08:09:38.101598+0000","last_unstale":"2024-04-24T08:09:38.103902+0000","last_undegraded":"2024-04-24T08:09:38.103902+0000","last_fullsized":"2024-04-24T08:09:38.103902+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:19:44.006523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.093880+0000","last_change":"2024-04-24T08:09:38.093880+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.093880+0000","last_undegraded":"2024-04-24T08:09:38.093880+0000","last_fullsized":"2024-04-24T08:09:38.093880+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109733+0000","last_change":"2024-04-24T08:09:38.101731+0000","last_active":"2024-04-24T08:09:38.109733+0000","last_peered":"2024-04-24T08:09:38.109733+0000","last_clean":"2024-04-24T08:09:38.109733+0000","last_became_active":"2024-04-24T08:09:38.100536+0000","last_became_peered":"2024-04-24T08:09:38.100536+0000","last_unstale":"2024-04-24T08:09:38.109733+0000","last_undegraded":"2024-04-24T08:09:38.109733+0000","last_fullsized":"2024-04-24T08:09:38.109733+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:17:11.041488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086737+0000","last_change":"2024-04-24T08:09:38.086737+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086737+0000","last_undegraded":"2024-04-24T08:09:38.086737+0000","last_fullsized":"2024-04-24T08:09:38.086737+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.112314+0000","last_change":"2024-04-24T08:09:38.108989+0000","last_active":"2024-04-24T08:09:38.112314+0000","last_peered":"2024-04-24T08:09:38.112314+0000","last_clean":"2024-04-24T08:09:38.112314+0000","last_became_active":"2024-04-24T08:09:38.108726+0000","last_became_peered":"2024-04-24T08:09:38.108726+0000","last_unstale":"2024-04-24T08:09:38.112314+0000","last_undegraded":"2024-04-24T08:09:38.112314+0000","last_fullsized":"2024-04-24T08:09:38.112314+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:40:46.556408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.091473+0000","last_change":"2024-04-24T08:09:38.091473+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.091473+0000","last_undegraded":"2024-04-24T08:09:38.091473+0000","last_fullsized":"2024-04-24T08:09:38.091473+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.097976+0000","last_change":"2024-04-24T08:09:38.093313+0000","last_active":"2024-04-24T08:09:38.097976+0000","last_peered":"2024-04-24T08:09:38.097976+0000","last_clean":"2024-04-24T08:09:38.097976+0000","last_became_active":"2024-04-24T08:09:38.092910+0000","last_became_peered":"2024-04-24T08:09:38.092910+0000","last_unstale":"2024-04-24T08:09:38.097976+0000","last_undegraded":"2024-04-24T08:09:38.097976+0000","last_fullsized":"2024-04-24T08:09:38.097976+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083631+0000","last_change":"2024-04-24T08:09:38.083631+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083631+0000","last_undegraded":"2024-04-24T08:09:38.083631+0000","last_fullsized":"2024-04-24T08:09:38.083631+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111408+0000","last_change":"2024-04-24T08:09:38.107454+0000","last_active":"2024-04-24T08:09:38.111408+0000","last_peered":"2024-04-24T08:09:38.111408+0000","last_clean":"2024-04-24T08:09:38.111408+0000","last_became_active":"2024-04-24T08:09:38.107149+0000","last_became_peered":"2024-04-24T08:09:38.107149+0000","last_unstale":"2024-04-24T08:09:38.111408+0000","last_undegraded":"2024-04-24T08:09:38.111408+0000","last_fullsized":"2024-04-24T08:09:38.111408+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:46:32.216293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087267+0000","last_change":"2024-04-24T08:09:38.087267+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087267+0000","last_undegraded":"2024-04-24T08:09:38.087267+0000","last_fullsized":"2024-04-24T08:09:38.087267+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111453+0000","last_change":"2024-04-24T08:09:38.110451+0000","last_active":"2024-04-24T08:09:38.111453+0000","last_peered":"2024-04-24T08:09:38.111453+0000","last_clean":"2024-04-24T08:09:38.111453+0000","last_became_active":"2024-04-24T08:09:38.110191+0000","last_became_peered":"2024-04-24T08:09:38.110191+0000","last_unstale":"2024-04-24T08:09:38.111453+0000","last_undegraded":"2024-04-24T08:09:38.111453+0000","last_fullsized":"2024-04-24T08:09:38.111453+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:10.779345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090571+0000","last_change":"2024-04-24T08:09:38.090571+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090571+0000","last_undegraded":"2024-04-24T08:09:38.090571+0000","last_fullsized":"2024-04-24T08:09:38.090571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.095062+0000","last_change":"2024-04-24T08:09:38.093711+0000","last_active":"2024-04-24T08:09:38.095062+0000","last_peered":"2024-04-24T08:09:38.095062+0000","last_clean":"2024-04-24T08:09:38.095062+0000","last_became_active":"2024-04-24T08:09:38.091626+0000","last_became_peered":"2024-04-24T08:09:38.091626+0000","last_unstale":"2024-04-24T08:09:38.095062+0000","last_undegraded":"2024-04-24T08:09:38.095062+0000","last_fullsized":"2024-04-24T08:09:38.095062+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.085881+0000","last_change":"2024-04-24T08:09:38.085881+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.085881+0000","last_undegraded":"2024-04-24T08:09:38.085881+0000","last_fullsized":"2024-04-24T08:09:38.085881+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109912+0000","last_change":"2024-04-24T08:09:38.107829+0000","last_active":"2024-04-24T08:09:38.109912+0000","last_peered":"2024-04-24T08:09:38.109912+0000","last_clean":"2024-04-24T08:09:38.109912+0000","last_became_active":"2024-04-24T08:09:38.107551+0000","last_became_peered":"2024-04-24T08:09:38.107551+0000","last_unstale":"2024-04-24T08:09:38.109912+0000","last_undegraded":"2024-04-24T08:09:38.109912+0000","last_fullsized":"2024-04-24T08:09:38.109912+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086224+0000","last_change":"2024-04-24T08:09:38.086224+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086224+0000","last_undegraded":"2024-04-24T08:09:38.086224+0000","last_fullsized":"2024-04-24T08:09:38.086224+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108172+0000","last_change":"2024-04-24T08:09:38.106294+0000","last_active":"2024-04-24T08:09:38.108172+0000","last_peered":"2024-04-24T08:09:38.108172+0000","last_clean":"2024-04-24T08:09:38.108172+0000","last_became_active":"2024-04-24T08:09:38.105994+0000","last_became_peered":"2024-04-24T08:09:38.105994+0000","last_unstale":"2024-04-24T08:09:38.108172+0000","last_undegraded":"2024-04-24T08:09:38.108172+0000","last_fullsized":"2024-04-24T08:09:38.108172+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:11.064191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084768+0000","last_change":"2024-04-24T08:09:38.084768+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084768+0000","last_undegraded":"2024-04-24T08:09:38.084768+0000","last_fullsized":"2024-04-24T08:09:38.084768+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111341+0000","last_change":"2024-04-24T08:09:38.105998+0000","last_active":"2024-04-24T08:09:38.111341+0000","last_peered":"2024-04-24T08:09:38.111341+0000","last_clean":"2024-04-24T08:09:38.111341+0000","last_became_active":"2024-04-24T08:09:38.105710+0000","last_became_peered":"2024-04-24T08:09:38.105710+0000","last_unstale":"2024-04-24T08:09:38.111341+0000","last_undegraded":"2024-04-24T08:09:38.111341+0000","last_fullsized":"2024-04-24T08:09:38.111341+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:58:11.279574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084682+0000","last_change":"2024-04-24T08:09:38.084682+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084682+0000","last_undegraded":"2024-04-24T08:09:38.084682+0000","last_fullsized":"2024-04-24T08:09:38.084682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109784+0000","last_change":"2024-04-24T08:09:38.107964+0000","last_active":"2024-04-24T08:09:38.109784+0000","last_peered":"2024-04-24T08:09:38.109784+0000","last_clean":"2024-04-24T08:09:38.109784+0000","last_became_active":"2024-04-24T08:09:38.107620+0000","last_became_peered":"2024-04-24T08:09:38.107620+0000","last_unstale":"2024-04-24T08:09:38.109784+0000","last_undegraded":"2024-04-24T08:09:38.109784+0000","last_fullsized":"2024-04-24T08:09:38.109784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.096631+0000","last_change":"2024-04-24T08:09:38.096631+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.096631+0000","last_undegraded":"2024-04-24T08:09:38.096631+0000","last_fullsized":"2024-04-24T08:09:38.096631+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.106220+0000","last_change":"2024-04-24T08:09:38.103294+0000","last_active":"2024-04-24T08:09:38.106220+0000","last_peered":"2024-04-24T08:09:38.106220+0000","last_clean":"2024-04-24T08:09:38.106220+0000","last_became_active":"2024-04-24T08:09:38.101762+0000","last_became_peered":"2024-04-24T08:09:38.101762+0000","last_unstale":"2024-04-24T08:09:38.106220+0000","last_undegraded":"2024-04-24T08:09:38.106220+0000","last_fullsized":"2024-04-24T08:09:38.106220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088094+0000","last_change":"2024-04-24T08:09:38.088094+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088094+0000","last_undegraded":"2024-04-24T08:09:38.088094+0000","last_fullsized":"2024-04-24T08:09:38.088094+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.103953+0000","last_change":"2024-04-24T08:09:38.100426+0000","last_active":"2024-04-24T08:09:38.103953+0000","last_peered":"2024-04-24T08:09:38.103953+0000","last_clean":"2024-04-24T08:09:38.103953+0000","last_became_active":"2024-04-24T08:09:38.100138+0000","last_became_peered":"2024-04-24T08:09:38.100138+0000","last_unstale":"2024-04-24T08:09:38.103953+0000","last_undegraded":"2024-04-24T08:09:38.103953+0000","last_fullsized":"2024-04-24T08:09:38.103953+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089848+0000","last_change":"2024-04-24T08:09:38.089848+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089848+0000","last_undegraded":"2024-04-24T08:09:38.089848+0000","last_fullsized":"2024-04-24T08:09:38.089848+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.100103+0000","last_change":"2024-04-24T08:09:38.096536+0000","last_active":"2024-04-24T08:09:38.100103+0000","last_peered":"2024-04-24T08:09:38.100103+0000","last_clean":"2024-04-24T08:09:38.100103+0000","last_became_active":"2024-04-24T08:09:38.095915+0000","last_became_peered":"2024-04-24T08:09:38.095915+0000","last_unstale":"2024-04-24T08:09:38.100103+0000","last_undegraded":"2024-04-24T08:09:38.100103+0000","last_fullsized":"2024-04-24T08:09:38.100103+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:05.837977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.088654+0000","last_change":"2024-04-24T08:09:38.088654+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.088654+0000","last_undegraded":"2024-04-24T08:09:38.088654+0000","last_fullsized":"2024-04-24T08:09:38.088654+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109578+0000","last_change":"2024-04-24T08:09:38.101152+0000","last_active":"2024-04-24T08:09:38.109578+0000","last_peered":"2024-04-24T08:09:38.109578+0000","last_clean":"2024-04-24T08:09:38.109578+0000","last_became_active":"2024-04-24T08:09:38.099292+0000","last_became_peered":"2024-04-24T08:09:38.099292+0000","last_unstale":"2024-04-24T08:09:38.109578+0000","last_undegraded":"2024-04-24T08:09:38.109578+0000","last_fullsized":"2024-04-24T08:09:38.109578+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:05:05.064870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087579+0000","last_change":"2024-04-24T08:09:38.087579+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087579+0000","last_undegraded":"2024-04-24T08:09:38.087579+0000","last_fullsized":"2024-04-24T08:09:38.087579+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.109637+0000","last_change":"2024-04-24T08:09:38.100668+0000","last_active":"2024-04-24T08:09:38.109637+0000","last_peered":"2024-04-24T08:09:38.109637+0000","last_clean":"2024-04-24T08:09:38.109637+0000","last_became_active":"2024-04-24T08:09:38.100303+0000","last_became_peered":"2024-04-24T08:09:38.100303+0000","last_unstale":"2024-04-24T08:09:38.109637+0000","last_undegraded":"2024-04-24T08:09:38.109637+0000","last_fullsized":"2024-04-24T08:09:38.109637+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:55.861909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089628+0000","last_change":"2024-04-24T08:09:38.089628+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089628+0000","last_undegraded":"2024-04-24T08:09:38.089628+0000","last_fullsized":"2024-04-24T08:09:38.089628+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111334+0000","last_change":"2024-04-24T08:09:38.105765+0000","last_active":"2024-04-24T08:09:38.111334+0000","last_peered":"2024-04-24T08:09:38.111334+0000","last_clean":"2024-04-24T08:09:38.111334+0000","last_became_active":"2024-04-24T08:09:38.100242+0000","last_became_peered":"2024-04-24T08:09:38.100242+0000","last_unstale":"2024-04-24T08:09:38.111334+0000","last_undegraded":"2024-04-24T08:09:38.111334+0000","last_fullsized":"2024-04-24T08:09:38.111334+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:19:44.006523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.084377+0000","last_change":"2024-04-24T08:09:38.084377+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.084377+0000","last_undegraded":"2024-04-24T08:09:38.084377+0000","last_fullsized":"2024-04-24T08:09:38.084377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.103077+0000","last_change":"2024-04-24T08:09:38.097858+0000","last_active":"2024-04-24T08:09:38.103077+0000","last_peered":"2024-04-24T08:09:38.103077+0000","last_clean":"2024-04-24T08:09:38.103077+0000","last_became_active":"2024-04-24T08:09:38.097599+0000","last_became_peered":"2024-04-24T08:09:38.097599+0000","last_unstale":"2024-04-24T08:09:38.103077+0000","last_undegraded":"2024-04-24T08:09:38.103077+0000","last_fullsized":"2024-04-24T08:09:38.103077+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.086095+0000","last_change":"2024-04-24T08:09:38.086095+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.086095+0000","last_undegraded":"2024-04-24T08:09:38.086095+0000","last_fullsized":"2024-04-24T08:09:38.086095+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.115052+0000","last_change":"2024-04-24T08:09:38.110759+0000","last_active":"2024-04-24T08:09:38.115052+0000","last_peered":"2024-04-24T08:09:38.115052+0000","last_clean":"2024-04-24T08:09:38.115052+0000","last_became_active":"2024-04-24T08:09:38.110393+0000","last_became_peered":"2024-04-24T08:09:38.110393+0000","last_unstale":"2024-04-24T08:09:38.115052+0000","last_undegraded":"2024-04-24T08:09:38.115052+0000","last_fullsized":"2024-04-24T08:09:38.115052+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087968+0000","last_change":"2024-04-24T08:09:38.087968+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087968+0000","last_undegraded":"2024-04-24T08:09:38.087968+0000","last_fullsized":"2024-04-24T08:09:38.087968+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.111161+0000","last_change":"2024-04-24T08:09:38.104959+0000","last_active":"2024-04-24T08:09:38.111161+0000","last_peered":"2024-04-24T08:09:38.111161+0000","last_clean":"2024-04-24T08:09:38.111161+0000","last_became_active":"2024-04-24T08:09:38.104606+0000","last_became_peered":"2024-04-24T08:09:38.104606+0000","last_unstale":"2024-04-24T08:09:38.111161+0000","last_undegraded":"2024-04-24T08:09:38.111161+0000","last_fullsized":"2024-04-24T08:09:38.111161+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:32.287730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.095292+0000","last_change":"2024-04-24T08:09:38.095292+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.095292+0000","last_undegraded":"2024-04-24T08:09:38.095292+0000","last_fullsized":"2024-04-24T08:09:38.095292+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.106663+0000","last_change":"2024-04-24T08:09:38.104201+0000","last_active":"2024-04-24T08:09:38.106663+0000","last_peered":"2024-04-24T08:09:38.106663+0000","last_clean":"2024-04-24T08:09:38.106663+0000","last_became_active":"2024-04-24T08:09:38.103332+0000","last_became_peered":"2024-04-24T08:09:38.103332+0000","last_unstale":"2024-04-24T08:09:38.106663+0000","last_undegraded":"2024-04-24T08:09:38.106663+0000","last_fullsized":"2024-04-24T08:09:38.106663+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.090440+0000","last_change":"2024-04-24T08:09:38.090440+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.090440+0000","last_undegraded":"2024-04-24T08:09:38.090440+0000","last_fullsized":"2024-04-24T08:09:38.090440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108613+0000","last_change":"2024-04-24T08:09:38.105102+0000","last_active":"2024-04-24T08:09:38.108613+0000","last_peered":"2024-04-24T08:09:38.108613+0000","last_clean":"2024-04-24T08:09:38.108613+0000","last_became_active":"2024-04-24T08:09:38.104814+0000","last_became_peered":"2024-04-24T08:09:38.104814+0000","last_unstale":"2024-04-24T08:09:38.108613+0000","last_undegraded":"2024-04-24T08:09:38.108613+0000","last_fullsized":"2024-04-24T08:09:38.108613+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.087435+0000","last_change":"2024-04-24T08:09:38.087435+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.087435+0000","last_undegraded":"2024-04-24T08:09:38.087435+0000","last_fullsized":"2024-04-24T08:09:38.087435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.104504+0000","last_change":"2024-04-24T08:09:38.102570+0000","last_active":"2024-04-24T08:09:38.104504+0000","last_peered":"2024-04-24T08:09:38.104504+0000","last_clean":"2024-04-24T08:09:38.104504+0000","last_became_active":"2024-04-24T08:09:38.101279+0000","last_became_peered":"2024-04-24T08:09:38.101279+0000","last_unstale":"2024-04-24T08:09:38.104504+0000","last_undegraded":"2024-04-24T08:09:38.104504+0000","last_fullsized":"2024-04-24T08:09:38.104504+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081042+0000","last_change":"2024-04-24T08:09:38.081042+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081042+0000","last_undegraded":"2024-04-24T08:09:38.081042+0000","last_fullsized":"2024-04-24T08:09:38.081042+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.105348+0000","last_change":"2024-04-24T08:09:38.101682+0000","last_active":"2024-04-24T08:09:38.105348+0000","last_peered":"2024-04-24T08:09:38.105348+0000","last_clean":"2024-04-24T08:09:38.105348+0000","last_became_active":"2024-04-24T08:09:38.099204+0000","last_became_peered":"2024-04-24T08:09:38.099204+0000","last_unstale":"2024-04-24T08:09:38.105348+0000","last_undegraded":"2024-04-24T08:09:38.105348+0000","last_fullsized":"2024-04-24T08:09:38.105348+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.089049+0000","last_change":"2024-04-24T08:09:38.089049+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.089049+0000","last_undegraded":"2024-04-24T08:09:38.089049+0000","last_fullsized":"2024-04-24T08:09:38.089049+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.104376+0000","last_change":"2024-04-24T08:09:38.101668+0000","last_active":"2024-04-24T08:09:38.104376+0000","last_peered":"2024-04-24T08:09:38.104376+0000","last_clean":"2024-04-24T08:09:38.104376+0000","last_became_active":"2024-04-24T08:09:38.101318+0000","last_became_peered":"2024-04-24T08:09:38.101318+0000","last_unstale":"2024-04-24T08:09:38.104376+0000","last_undegraded":"2024-04-24T08:09:38.104376+0000","last_fullsized":"2024-04-24T08:09:38.104376+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.081090+0000","last_change":"2024-04-24T08:09:38.081090+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.081090+0000","last_undegraded":"2024-04-24T08:09:38.081090+0000","last_fullsized":"2024-04-24T08:09:38.081090+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108737+0000","last_change":"2024-04-24T08:09:38.105797+0000","last_active":"2024-04-24T08:09:38.108737+0000","last_peered":"2024-04-24T08:09:38.108737+0000","last_clean":"2024-04-24T08:09:38.108737+0000","last_became_active":"2024-04-24T08:09:38.104048+0000","last_became_peered":"2024-04-24T08:09:38.104048+0000","last_unstale":"2024-04-24T08:09:38.108737+0000","last_undegraded":"2024-04-24T08:09:38.108737+0000","last_fullsized":"2024-04-24T08:09:38.108737+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.083903+0000","last_change":"2024-04-24T08:09:38.083903+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.083903+0000","last_undegraded":"2024-04-24T08:09:38.083903+0000","last_fullsized":"2024-04-24T08:09:38.083903+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.110901+0000","last_change":"2024-04-24T08:09:38.103639+0000","last_active":"2024-04-24T08:09:38.110901+0000","last_peered":"2024-04-24T08:09:38.110901+0000","last_clean":"2024-04-24T08:09:38.110901+0000","last_became_active":"2024-04-24T08:09:38.103230+0000","last_became_peered":"2024-04-24T08:09:38.103230+0000","last_unstale":"2024-04-24T08:09:38.110901+0000","last_undegraded":"2024-04-24T08:09:38.110901+0000","last_fullsized":"2024-04-24T08:09:38.110901+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:27.127550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.092331+0000","last_change":"2024-04-24T08:09:38.092331+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.092331+0000","last_undegraded":"2024-04-24T08:09:38.092331+0000","last_fullsized":"2024-04-24T08:09:38.092331+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.108778+0000","last_change":"2024-04-24T08:09:38.107831+0000","last_active":"2024-04-24T08:09:38.108778+0000","last_peered":"2024-04-24T08:09:38.108778+0000","last_clean":"2024-04-24T08:09:38.108778+0000","last_became_active":"2024-04-24T08:09:38.107509+0000","last_became_peered":"2024-04-24T08:09:38.107509+0000","last_unstale":"2024-04-24T08:09:38.108778+0000","last_undegraded":"2024-04-24T08:09:38.108778+0000","last_fullsized":"2024-04-24T08:09:38.108778+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:27.457454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":31,"state":"creating+peering","last_fresh":"2024-04-24T08:09:38.099092+0000","last_change":"2024-04-24T08:09:38.099092+0000","last_active":"2024-04-24T08:09:38.056115+0000","last_peered":"2024-04-24T08:09:38.056115+0000","last_clean":"2024-04-24T08:09:38.056115+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:09:38.099092+0000","last_undegraded":"2024-04-24T08:09:38.099092+0000","last_fullsized":"2024-04-24T08:09:38.099092+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.117716+0000","last_change":"2024-04-24T08:09:38.112443+0000","last_active":"2024-04-24T08:09:38.117716+0000","last_peered":"2024-04-24T08:09:38.117716+0000","last_clean":"2024-04-24T08:09:38.117716+0000","last_became_active":"2024-04-24T08:09:38.111901+0000","last_became_peered":"2024-04-24T08:09:38.111901+0000","last_unstale":"2024-04-24T08:09:38.117716+0000","last_undegraded":"2024-04-24T08:09:38.117716+0000","last_fullsized":"2024-04-24T08:09:38.117716+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":189,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-04-24T08:09:38.106310+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:09:38.106310+0000","last_peered":"2024-04-24T08:09:38.106310+0000","last_clean":"2024-04-24T08:09:38.106310+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:09:38.106310+0000","last_undegraded":"2024-04-24T08:09:38.106310+0000","last_fullsized":"2024-04-24T08:09:38.106310+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343896,"statfs":{"total":96636764160,"available":96608149504,"internally_reserved":0,"allocated":942080,"data_stored":724063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":133695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:09:44.560 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2024-04-24T08:09:44.560 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.091995+0000', 'last_change': '2024-04-24T08:09:38.091995+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.091995+0000', 'last_undegraded': '2024-04-24T08:09:38.091995+0000', 'last_fullsized': '2024-04-24T08:09:38.091995+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-24T08:09:44.560 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2024-04-24T08:09:44.560 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.090575+0000', 'last_change': '2024-04-24T08:09:38.090575+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.090575+0000', 'last_undegraded': '2024-04-24T08:09:38.090575+0000', 'last_fullsized': '2024-04-24T08:09:38.090575+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.085165+0000', 'last_change': '2024-04-24T08:09:38.085165+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.085165+0000', 'last_undegraded': '2024-04-24T08:09:38.085165+0000', 'last_fullsized': '2024-04-24T08:09:38.085165+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.092285+0000', 'last_change': '2024-04-24T08:09:38.092285+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.092285+0000', 'last_undegraded': '2024-04-24T08:09:38.092285+0000', 'last_fullsized': '2024-04-24T08:09:38.092285+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.087102+0000', 'last_change': '2024-04-24T08:09:38.087102+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.087102+0000', 'last_undegraded': '2024-04-24T08:09:38.087102+0000', 'last_fullsized': '2024-04-24T08:09:38.087102+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.088583+0000', 'last_change': '2024-04-24T08:09:38.088583+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.088583+0000', 'last_undegraded': '2024-04-24T08:09:38.088583+0000', 'last_fullsized': '2024-04-24T08:09:38.088583+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.093698+0000', 'last_change': '2024-04-24T08:09:38.093698+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.093698+0000', 'last_undegraded': '2024-04-24T08:09:38.093698+0000', 'last_fullsized': '2024-04-24T08:09:38.093698+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.561 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.098131+0000', 'last_change': '2024-04-24T08:09:38.098131+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.098131+0000', 'last_undegraded': '2024-04-24T08:09:38.098131+0000', 'last_fullsized': '2024-04-24T08:09:38.098131+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.095971+0000', 'last_change': '2024-04-24T08:09:38.095971+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.095971+0000', 'last_undegraded': '2024-04-24T08:09:38.095971+0000', 'last_fullsized': '2024-04-24T08:09:38.095971+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.093218+0000', 'last_change': '2024-04-24T08:09:38.093218+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.093218+0000', 'last_undegraded': '2024-04-24T08:09:38.093218+0000', 'last_fullsized': '2024-04-24T08:09:38.093218+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.098579+0000', 'last_change': '2024-04-24T08:09:38.098579+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.098579+0000', 'last_undegraded': '2024-04-24T08:09:38.098579+0000', 'last_fullsized': '2024-04-24T08:09:38.098579+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.084662+0000', 'last_change': '2024-04-24T08:09:38.084662+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.084662+0000', 'last_undegraded': '2024-04-24T08:09:38.084662+0000', 'last_fullsized': '2024-04-24T08:09:38.084662+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.086610+0000', 'last_change': '2024-04-24T08:09:38.086610+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.086610+0000', 'last_undegraded': '2024-04-24T08:09:38.086610+0000', 'last_fullsized': '2024-04-24T08:09:38.086610+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2024-04-24T08:09:44.562 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.087851+0000', 'last_change': '2024-04-24T08:09:38.087851+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.087851+0000', 'last_undegraded': '2024-04-24T08:09:38.087851+0000', 'last_fullsized': '2024-04-24T08:09:38.087851+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.088469+0000', 'last_change': '2024-04-24T08:09:38.088469+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.088469+0000', 'last_undegraded': '2024-04-24T08:09:38.088469+0000', 'last_fullsized': '2024-04-24T08:09:38.088469+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.088695+0000', 'last_change': '2024-04-24T08:09:38.088695+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.088695+0000', 'last_undegraded': '2024-04-24T08:09:38.088695+0000', 'last_fullsized': '2024-04-24T08:09:38.088695+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.102476+0000', 'last_change': '2024-04-24T08:09:38.102476+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.102476+0000', 'last_undegraded': '2024-04-24T08:09:38.102476+0000', 'last_fullsized': '2024-04-24T08:09:38.102476+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.089933+0000', 'last_change': '2024-04-24T08:09:38.089933+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.089933+0000', 'last_undegraded': '2024-04-24T08:09:38.089933+0000', 'last_fullsized': '2024-04-24T08:09:38.089933+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.086859+0000', 'last_change': '2024-04-24T08:09:38.086859+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.086859+0000', 'last_undegraded': '2024-04-24T08:09:38.086859+0000', 'last_fullsized': '2024-04-24T08:09:38.086859+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2024-04-24T08:09:44.563 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.093820+0000', 'last_change': '2024-04-24T08:09:38.093820+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.093820+0000', 'last_undegraded': '2024-04-24T08:09:38.093820+0000', 'last_fullsized': '2024-04-24T08:09:38.093820+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.096265+0000', 'last_change': '2024-04-24T08:09:38.096265+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.096265+0000', 'last_undegraded': '2024-04-24T08:09:38.096265+0000', 'last_fullsized': '2024-04-24T08:09:38.096265+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.094464+0000', 'last_change': '2024-04-24T08:09:38.094464+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.094464+0000', 'last_undegraded': '2024-04-24T08:09:38.094464+0000', 'last_fullsized': '2024-04-24T08:09:38.094464+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.081017+0000', 'last_change': '2024-04-24T08:09:38.081017+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.081017+0000', 'last_undegraded': '2024-04-24T08:09:38.081017+0000', 'last_fullsized': '2024-04-24T08:09:38.081017+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.083984+0000', 'last_change': '2024-04-24T08:09:38.083984+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.083984+0000', 'last_undegraded': '2024-04-24T08:09:38.083984+0000', 'last_fullsized': '2024-04-24T08:09:38.083984+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.092642+0000', 'last_change': '2024-04-24T08:09:38.092642+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.092642+0000', 'last_undegraded': '2024-04-24T08:09:38.092642+0000', 'last_fullsized': '2024-04-24T08:09:38.092642+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.089128+0000', 'last_change': '2024-04-24T08:09:38.089128+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.089128+0000', 'last_undegraded': '2024-04-24T08:09:38.089128+0000', 'last_fullsized': '2024-04-24T08:09:38.089128+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T08:09:44.564 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.094332+0000', 'last_change': '2024-04-24T08:09:38.094332+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.094332+0000', 'last_undegraded': '2024-04-24T08:09:38.094332+0000', 'last_fullsized': '2024-04-24T08:09:38.094332+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.083051+0000', 'last_change': '2024-04-24T08:09:38.083051+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.083051+0000', 'last_undegraded': '2024-04-24T08:09:38.083051+0000', 'last_fullsized': '2024-04-24T08:09:38.083051+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.087579+0000', 'last_change': '2024-04-24T08:09:38.087579+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.087579+0000', 'last_undegraded': '2024-04-24T08:09:38.087579+0000', 'last_fullsized': '2024-04-24T08:09:38.087579+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.094793+0000', 'last_change': '2024-04-24T08:09:38.094793+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.094793+0000', 'last_undegraded': '2024-04-24T08:09:38.094793+0000', 'last_fullsized': '2024-04-24T08:09:38.094793+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.095355+0000', 'last_change': '2024-04-24T08:09:38.095355+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.095355+0000', 'last_undegraded': '2024-04-24T08:09:38.095355+0000', 'last_fullsized': '2024-04-24T08:09:38.095355+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.097440+0000', 'last_change': '2024-04-24T08:09:38.097440+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.097440+0000', 'last_undegraded': '2024-04-24T08:09:38.097440+0000', 'last_fullsized': '2024-04-24T08:09:38.097440+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-04-24T08:09:44.565 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.081278+0000', 'last_change': '2024-04-24T08:09:38.081278+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.081278+0000', 'last_undegraded': '2024-04-24T08:09:38.081278+0000', 'last_fullsized': '2024-04-24T08:09:38.081278+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.090079+0000', 'last_change': '2024-04-24T08:09:38.090079+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.090079+0000', 'last_undegraded': '2024-04-24T08:09:38.090079+0000', 'last_fullsized': '2024-04-24T08:09:38.090079+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.092529+0000', 'last_change': '2024-04-24T08:09:38.092529+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.092529+0000', 'last_undegraded': '2024-04-24T08:09:38.092529+0000', 'last_fullsized': '2024-04-24T08:09:38.092529+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.087273+0000', 'last_change': '2024-04-24T08:09:38.087273+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.087273+0000', 'last_undegraded': '2024-04-24T08:09:38.087273+0000', 'last_fullsized': '2024-04-24T08:09:38.087273+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.093880+0000', 'last_change': '2024-04-24T08:09:38.093880+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.093880+0000', 'last_undegraded': '2024-04-24T08:09:38.093880+0000', 'last_fullsized': '2024-04-24T08:09:38.093880+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.086737+0000', 'last_change': '2024-04-24T08:09:38.086737+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.086737+0000', 'last_undegraded': '2024-04-24T08:09:38.086737+0000', 'last_fullsized': '2024-04-24T08:09:38.086737+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T08:09:44.566 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.091473+0000', 'last_change': '2024-04-24T08:09:38.091473+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.091473+0000', 'last_undegraded': '2024-04-24T08:09:38.091473+0000', 'last_fullsized': '2024-04-24T08:09:38.091473+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.083631+0000', 'last_change': '2024-04-24T08:09:38.083631+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.083631+0000', 'last_undegraded': '2024-04-24T08:09:38.083631+0000', 'last_fullsized': '2024-04-24T08:09:38.083631+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.087267+0000', 'last_change': '2024-04-24T08:09:38.087267+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.087267+0000', 'last_undegraded': '2024-04-24T08:09:38.087267+0000', 'last_fullsized': '2024-04-24T08:09:38.087267+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.090571+0000', 'last_change': '2024-04-24T08:09:38.090571+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.090571+0000', 'last_undegraded': '2024-04-24T08:09:38.090571+0000', 'last_fullsized': '2024-04-24T08:09:38.090571+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.085881+0000', 'last_change': '2024-04-24T08:09:38.085881+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.085881+0000', 'last_undegraded': '2024-04-24T08:09:38.085881+0000', 'last_fullsized': '2024-04-24T08:09:38.085881+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-04-24T08:09:44.567 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.086224+0000', 'last_change': '2024-04-24T08:09:38.086224+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.086224+0000', 'last_undegraded': '2024-04-24T08:09:38.086224+0000', 'last_fullsized': '2024-04-24T08:09:38.086224+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.084768+0000', 'last_change': '2024-04-24T08:09:38.084768+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.084768+0000', 'last_undegraded': '2024-04-24T08:09:38.084768+0000', 'last_fullsized': '2024-04-24T08:09:38.084768+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.084682+0000', 'last_change': '2024-04-24T08:09:38.084682+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.084682+0000', 'last_undegraded': '2024-04-24T08:09:38.084682+0000', 'last_fullsized': '2024-04-24T08:09:38.084682+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.096631+0000', 'last_change': '2024-04-24T08:09:38.096631+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.096631+0000', 'last_undegraded': '2024-04-24T08:09:38.096631+0000', 'last_fullsized': '2024-04-24T08:09:38.096631+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.088094+0000', 'last_change': '2024-04-24T08:09:38.088094+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.088094+0000', 'last_undegraded': '2024-04-24T08:09:38.088094+0000', 'last_fullsized': '2024-04-24T08:09:38.088094+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.089848+0000', 'last_change': '2024-04-24T08:09:38.089848+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.089848+0000', 'last_undegraded': '2024-04-24T08:09:38.089848+0000', 'last_fullsized': '2024-04-24T08:09:38.089848+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-04-24T08:09:44.568 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.088654+0000', 'last_change': '2024-04-24T08:09:38.088654+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.088654+0000', 'last_undegraded': '2024-04-24T08:09:38.088654+0000', 'last_fullsized': '2024-04-24T08:09:38.088654+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.087579+0000', 'last_change': '2024-04-24T08:09:38.087579+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.087579+0000', 'last_undegraded': '2024-04-24T08:09:38.087579+0000', 'last_fullsized': '2024-04-24T08:09:38.087579+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.089628+0000', 'last_change': '2024-04-24T08:09:38.089628+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.089628+0000', 'last_undegraded': '2024-04-24T08:09:38.089628+0000', 'last_fullsized': '2024-04-24T08:09:38.089628+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.084377+0000', 'last_change': '2024-04-24T08:09:38.084377+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.084377+0000', 'last_undegraded': '2024-04-24T08:09:38.084377+0000', 'last_fullsized': '2024-04-24T08:09:38.084377+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.086095+0000', 'last_change': '2024-04-24T08:09:38.086095+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.086095+0000', 'last_undegraded': '2024-04-24T08:09:38.086095+0000', 'last_fullsized': '2024-04-24T08:09:38.086095+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.087968+0000', 'last_change': '2024-04-24T08:09:38.087968+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.087968+0000', 'last_undegraded': '2024-04-24T08:09:38.087968+0000', 'last_fullsized': '2024-04-24T08:09:38.087968+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.095292+0000', 'last_change': '2024-04-24T08:09:38.095292+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.095292+0000', 'last_undegraded': '2024-04-24T08:09:38.095292+0000', 'last_fullsized': '2024-04-24T08:09:38.095292+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.569 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-04-24T08:09:44.570 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.090440+0000', 'last_change': '2024-04-24T08:09:38.090440+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.090440+0000', 'last_undegraded': '2024-04-24T08:09:38.090440+0000', 'last_fullsized': '2024-04-24T08:09:38.090440+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T08:09:44.570 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-04-24T08:09:44.570 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.087435+0000', 'last_change': '2024-04-24T08:09:38.087435+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.087435+0000', 'last_undegraded': '2024-04-24T08:09:38.087435+0000', 'last_fullsized': '2024-04-24T08:09:38.087435+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T08:09:44.570 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-04-24T08:09:44.570 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.081042+0000', 'last_change': '2024-04-24T08:09:38.081042+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.081042+0000', 'last_undegraded': '2024-04-24T08:09:38.081042+0000', 'last_fullsized': '2024-04-24T08:09:38.081042+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-24T08:09:44.570 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-04-24T08:09:44.570 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.089049+0000', 'last_change': '2024-04-24T08:09:38.089049+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.089049+0000', 'last_undegraded': '2024-04-24T08:09:38.089049+0000', 'last_fullsized': '2024-04-24T08:09:38.089049+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-24T08:09:44.570 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-04-24T08:09:44.570 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.081090+0000', 'last_change': '2024-04-24T08:09:38.081090+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.081090+0000', 'last_undegraded': '2024-04-24T08:09:38.081090+0000', 'last_fullsized': '2024-04-24T08:09:38.081090+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-24T08:09:44.570 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-04-24T08:09:44.571 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.083903+0000', 'last_change': '2024-04-24T08:09:38.083903+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.083903+0000', 'last_undegraded': '2024-04-24T08:09:38.083903+0000', 'last_fullsized': '2024-04-24T08:09:38.083903+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T08:09:44.571 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-04-24T08:09:44.571 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.092331+0000', 'last_change': '2024-04-24T08:09:38.092331+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.092331+0000', 'last_undegraded': '2024-04-24T08:09:38.092331+0000', 'last_fullsized': '2024-04-24T08:09:38.092331+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T08:09:44.571 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-04-24T08:09:44.571 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 31, 'state': 'creating+peering', 'last_fresh': '2024-04-24T08:09:38.099092+0000', 'last_change': '2024-04-24T08:09:38.099092+0000', 'last_active': '2024-04-24T08:09:38.056115+0000', 'last_peered': '2024-04-24T08:09:38.056115+0000', 'last_clean': '2024-04-24T08:09:38.056115+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T08:09:38.099092+0000', 'last_undegraded': '2024-04-24T08:09:38.099092+0000', 'last_fullsized': '2024-04-24T08:09:38.099092+0000', 'mapping_epoch': 31, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 31, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'last_clean_scrub_stamp': '2024-04-24T08:09:38.056115+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T08:09:44.571 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-24T08:09:44.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.791+0000 7fb8e7c00640 1 Processor -- start 2024-04-24T08:09:44.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.791+0000 7fb8e7c00640 1 -- start start 2024-04-24T08:09:44.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.792+0000 7fb8e7c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb8e00cb400 0x7fb8e00d0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.792+0000 7fb8e7c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e00d0cd0 0x7fb8e0071db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.792+0000 7fb8e7c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8e00ce980 con 0x7fb8e015b680 2024-04-24T08:09:44.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.792+0000 7fb8e7c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb8e00d0130 con 0x7fb8e00cb400 2024-04-24T08:09:44.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.792+0000 7fb8e7c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e00d1140 con 0x7fb8e00d0cd0 2024-04-24T08:09:44.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.792+0000 7fb8e5200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb8e015b680 0x7fb8e00cfa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53290/0 (socket says 172.21.15.69:53290) 2024-04-24T08:09:44.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.792+0000 7fb8e5200640 1 -- 172.21.15.69:0/4108711622 learned_addr learned my addr 172.21.15.69:0/4108711622 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:44.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.793+0000 7fb8d7e00640 1 --2- 172.21.15.69:0/4108711622 >> v2:172.21.15.69:3300/0 conn(0x7fb8e00cb400 0x7fb8e00d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.793+0000 7fb8e5c00640 1 --2- 172.21.15.69:0/4108711622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e00d0cd0 0x7fb8e0071db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.793+0000 7fb8dfe00640 1 -- 172.21.15.69:0/4108711622 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 963360208 0 0) 0x7fb8e00ce980 con 0x7fb8e015b680 2024-04-24T08:09:44.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.793+0000 7fb8dfe00640 1 -- 172.21.15.69:0/4108711622 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8c4003520 con 0x7fb8e015b680 2024-04-24T08:09:44.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.793+0000 7fb8d7e00640 1 -- 172.21.15.69:0/4108711622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e00d0cd0 msgr2=0x7fb8e0071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:44.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.793+0000 7fb8d7e00640 1 --2- 172.21.15.69:0/4108711622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e00d0cd0 0x7fb8e0071db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.793+0000 7fb8d7e00640 1 -- 172.21.15.69:0/4108711622 >> v1:172.21.15.69:6789/0 conn(0x7fb8e015b680 legacy=0x7fb8e00cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.793+0000 7fb8d7e00640 1 -- 172.21.15.69:0/4108711622 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e00cec60 con 0x7fb8e00cb400 2024-04-24T08:09:44.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.794+0000 7fb8d7e00640 1 --2- 172.21.15.69:0/4108711622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 0x7fb8e00d0790 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80097d0 tx=0x7fb8c802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e015dc85f986cd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:44.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.795+0000 7fb8dfe00640 1 -- 172.21.15.69:0/4108711622 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8037070 con 0x7fb8e00cb400 2024-04-24T08:09:44.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.795+0000 7fb8dfe00640 1 -- 172.21.15.69:0/4108711622 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8c8032440 con 0x7fb8e00cb400 2024-04-24T08:09:44.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.795+0000 7fb8dfe00640 1 -- 172.21.15.69:0/4108711622 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c80327f0 con 0x7fb8e00cb400 2024-04-24T08:09:44.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.795+0000 7fb8e7c00640 1 -- 172.21.15.69:0/4108711622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 msgr2=0x7fb8e00d0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.795+0000 7fb8e7c00640 1 --2- 172.21.15.69:0/4108711622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 0x7fb8e00d0790 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fb8c80097d0 tx=0x7fb8c802a250 comp rx=0 tx=0).stop 2024-04-24T08:09:44.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.796+0000 7fb8e7c00640 1 -- 172.21.15.69:0/4108711622 shutdown_connections 2024-04-24T08:09:44.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.796+0000 7fb8e7c00640 1 --2- 172.21.15.69:0/4108711622 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e00d0cd0 0x7fb8e0071db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.796+0000 7fb8e7c00640 1 --2- 172.21.15.69:0/4108711622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 0x7fb8e00d0790 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.796+0000 7fb8e7c00640 1 -- 172.21.15.69:0/4108711622 >> 172.21.15.69:0/4108711622 conn(0x7fb8e006be40 msgr2=0x7fb8e006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:44.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.796+0000 7fb8e7c00640 1 -- 172.21.15.69:0/4108711622 shutdown_connections 2024-04-24T08:09:44.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.796+0000 7fb8e7c00640 1 -- 172.21.15.69:0/4108711622 wait complete. 2024-04-24T08:09:44.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.797+0000 7fb8e7c00640 1 Processor -- start 2024-04-24T08:09:44.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.797+0000 7fb8e7c00640 1 -- start start 2024-04-24T08:09:44.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.798+0000 7fb8e7c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 0x7fb8e01f6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.798+0000 7fb8e7c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8e00d0cd0 0x7fb8e00c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.798+0000 7fb8e7c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e015b680 0x7fb8e009a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.798+0000 7fb8e7c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e00cf220 con 0x7fb8e00cb400 2024-04-24T08:09:44.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.798+0000 7fb8e7c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e00cf590 con 0x7fb8e00d0cd0 2024-04-24T08:09:44.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.798+0000 7fb8e7c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e00cb860 con 0x7fb8e015b680 2024-04-24T08:09:44.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 0x7fb8e01f6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 0x7fb8e01f6eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60722/0 (socket says 172.21.15.69:60722) 2024-04-24T08:09:44.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5200640 1 -- 172.21.15.69:0/1681492651 learned_addr learned my addr 172.21.15.69:0/1681492651 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:44.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5c00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e015b680 0x7fb8e009a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8d7e00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8e00d0cd0 0x7fb8e00c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5200640 1 -- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e015b680 msgr2=0x7fb8e009a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5200640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e015b680 0x7fb8e009a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5200640 1 -- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8e00d0cd0 msgr2=0x7fb8e00c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5200640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8e00d0cd0 0x7fb8e00c8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5200640 1 -- 172.21.15.69:0/1681492651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e0158d80 con 0x7fb8e00cb400 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5c00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e015b680 0x7fb8e009a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.799+0000 7fb8e5200640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 0x7fb8e01f6eb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb8d00010e0 tx=0x7fb8d000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.800+0000 7fb8dd600640 1 -- 172.21.15.69:0/1681492651 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d0017070 con 0x7fb8e00cb400 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.800+0000 7fb8dd600640 1 -- 172.21.15.69:0/1681492651 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d0005d00 con 0x7fb8e00cb400 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.800+0000 7fb8e7c00640 1 -- 172.21.15.69:0/1681492651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e0159060 con 0x7fb8e00cb400 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.800+0000 7fb8dd600640 1 -- 172.21.15.69:0/1681492651 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d00072b0 con 0x7fb8e00cb400 2024-04-24T08:09:44.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.800+0000 7fb8e7c00640 1 -- 172.21.15.69:0/1681492651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e0159620 con 0x7fb8e00cb400 2024-04-24T08:09:44.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.801+0000 7fb8e7c00640 1 -- 172.21.15.69:0/1681492651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8e0075dc0 con 0x7fb8e00cb400 2024-04-24T08:09:44.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.802+0000 7fb8dd600640 1 -- 172.21.15.69:0/1681492651 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb8d001a070 con 0x7fb8e00cb400 2024-04-24T08:09:44.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.803+0000 7fb8dd600640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8bc06c8f0 0x7fb8bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:44.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.803+0000 7fb8dd600640 1 -- 172.21.15.69:0/1681492651 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fb8d008af80 con 0x7fb8e00cb400 2024-04-24T08:09:44.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.804+0000 7fb8d7e00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8bc06c8f0 0x7fb8bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:44.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.805+0000 7fb8d7e00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8bc06c8f0 0x7fb8bc06edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8034000 tx=0x7fb8c802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:44.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:44.807+0000 7fb8dd600640 1 -- 172.21.15.69:0/1681492651 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb8d00593a0 con 0x7fb8e00cb400 2024-04-24T08:09:45.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.086+0000 7fb8e7c00640 1 -- 172.21.15.69:0/1681492651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7fb8e0159860 con 0x7fb8e00cb400 2024-04-24T08:09:45.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.087+0000 7fb8dd600640 1 -- 172.21.15.69:0/1681492651 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1034 (secure 0 0 0) 0x7fb8d0058ca0 con 0x7fb8e00cb400 2024-04-24T08:09:45.087 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:45.087 INFO:teuthology.orchestra.run.smithi069.stdout:{"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":109,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":8,"num_up_osds":8,"osd_up_since":1713946087,"num_in_osds":8,"osd_in_since":1713946078,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225320960,"bytes_avail":772868792320,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":16,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":5241}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-24T08:09:26.685489+0000","services":{}},"progress_events":{}} 2024-04-24T08:09:45.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.089+0000 7fb8e7c00640 1 -- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8bc06c8f0 msgr2=0x7fb8bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:45.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8bc06c8f0 0x7fb8bc06edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb8c8034000 tx=0x7fb8c802e000 comp rx=0 tx=0).stop 2024-04-24T08:09:45.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 -- 172.21.15.69:0/1681492651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 msgr2=0x7fb8e01f6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:45.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 0x7fb8e01f6eb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb8d00010e0 tx=0x7fb8d000c960 comp rx=0 tx=0).stop 2024-04-24T08:09:45.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 -- 172.21.15.69:0/1681492651 shutdown_connections 2024-04-24T08:09:45.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8bc06c8f0 0x7fb8bc06edb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8e015b680 0x7fb8e009a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8e00d0cd0 0x7fb8e00c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 --2- 172.21.15.69:0/1681492651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8e00cb400 0x7fb8e01f6eb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 -- 172.21.15.69:0/1681492651 >> 172.21.15.69:0/1681492651 conn(0x7fb8e006be40 msgr2=0x7fb8e015e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:45.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 -- 172.21.15.69:0/1681492651 shutdown_connections 2024-04-24T08:09:45.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.090+0000 7fb8e7c00640 1 -- 172.21.15.69:0/1681492651 wait complete. 2024-04-24T08:09:45.109 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-24T08:09:45.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:09:45.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.302+0000 7fec03e00640 1 Processor -- start 2024-04-24T08:09:45.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.303+0000 7fec03e00640 1 -- start start 2024-04-24T08:09:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.304+0000 7fec03e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec0406cfb0 0x7fec040cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.304+0000 7fec03e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fec0406d410 0x7fec040d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.304+0000 7fec03e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec0415ed40 con 0x7fec0415e960 2024-04-24T08:09:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.304+0000 7fec03e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fec040cf050 con 0x7fec0406cfb0 2024-04-24T08:09:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.304+0000 7fec03e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fec04162b50 con 0x7fec0406d410 2024-04-24T08:09:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.304+0000 7fec02a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fec0415e960 0x7fec04162440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53306/0 (socket says 172.21.15.69:53306) 2024-04-24T08:09:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.304+0000 7febfa000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec0406cfb0 0x7fec040cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.304+0000 7fec02a00640 1 -- 172.21.15.69:0/3512956528 learned_addr learned my addr 172.21.15.69:0/3512956528 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:45.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.305+0000 7fec03400640 1 --2- 172.21.15.69:0/3512956528 >> v2:172.21.15.121:3300/0 conn(0x7fec0406d410 0x7fec040d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.305+0000 7febfa000640 1 -- 172.21.15.69:0/3512956528 >> v2:172.21.15.121:3300/0 conn(0x7fec0406d410 msgr2=0x7fec040d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.305+0000 7febfa000640 1 --2- 172.21.15.69:0/3512956528 >> v2:172.21.15.121:3300/0 conn(0x7fec0406d410 0x7fec040d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.305+0000 7febfa000640 1 -- 172.21.15.69:0/3512956528 >> v1:172.21.15.69:6789/0 conn(0x7fec0415e960 legacy=0x7fec04162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.305+0000 7febfa000640 1 -- 172.21.15.69:0/3512956528 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec04162e20 con 0x7fec0406cfb0 2024-04-24T08:09:45.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.305+0000 7fec02000640 1 -- 172.21.15.69:0/3512956528 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 992893874 0 0) 0x7fec0415ed40 con 0x7fec0415e960 2024-04-24T08:09:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.306+0000 7febfa000640 1 --2- 172.21.15.69:0/3512956528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0406cfb0 0x7fec040cfaf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fec040b3fe0 tx=0x7febec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c255dcec1d0768c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.306+0000 7fec02000640 1 -- 172.21.15.69:0/3512956528 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec037070 con 0x7fec0406cfb0 2024-04-24T08:09:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.306+0000 7fec02000640 1 -- 172.21.15.69:0/3512956528 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febec032440 con 0x7fec0406cfb0 2024-04-24T08:09:45.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.306+0000 7fec02000640 1 -- 172.21.15.69:0/3512956528 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec032810 con 0x7fec0406cfb0 2024-04-24T08:09:45.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.307+0000 7fec03e00640 1 -- 172.21.15.69:0/3512956528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0406cfb0 msgr2=0x7fec040cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:45.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.307+0000 7fec03e00640 1 --2- 172.21.15.69:0/3512956528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0406cfb0 0x7fec040cfaf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fec040b3fe0 tx=0x7febec02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.307+0000 7fec03e00640 1 -- 172.21.15.69:0/3512956528 shutdown_connections 2024-04-24T08:09:45.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.307+0000 7fec03e00640 1 --2- 172.21.15.69:0/3512956528 >> v2:172.21.15.121:3300/0 conn(0x7fec0406d410 0x7fec040d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.307+0000 7fec03e00640 1 --2- 172.21.15.69:0/3512956528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0406cfb0 0x7fec040cfaf0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.307+0000 7fec03e00640 1 -- 172.21.15.69:0/3512956528 >> 172.21.15.69:0/3512956528 conn(0x7fec0409d900 msgr2=0x7fec0408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:45.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.307+0000 7fec03e00640 1 -- 172.21.15.69:0/3512956528 shutdown_connections 2024-04-24T08:09:45.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.307+0000 7fec03e00640 1 -- 172.21.15.69:0/3512956528 wait complete. 2024-04-24T08:09:45.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.309+0000 7fec03e00640 1 Processor -- start 2024-04-24T08:09:45.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.309+0000 7fec03e00640 1 -- start start 2024-04-24T08:09:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.310+0000 7fec03e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec0406cfb0 0x7fec040ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.310+0000 7fec03e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec0406d410 0x7fec040b87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.310+0000 7fec03e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0415e960 0x7fec040b8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.310+0000 7fec03e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fec040cf5d0 con 0x7fec0415e960 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.310+0000 7fec03e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fec040cf2f0 con 0x7fec0406cfb0 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.310+0000 7fec03e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec040b6170 con 0x7fec0406d410 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.310+0000 7fec03400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0415e960 0x7fec040b8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7fec03400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0415e960 0x7fec040b8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60736/0 (socket says 172.21.15.69:60736) 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7fec03400640 1 -- 172.21.15.69:0/4095998675 learned_addr learned my addr 172.21.15.69:0/4095998675 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7fec02a00640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec0406cfb0 0x7fec040ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7febfa000640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec0406d410 0x7fec040b87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7fec03400640 1 -- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec0406d410 msgr2=0x7fec040b87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7fec03400640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec0406d410 0x7fec040b87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7fec03400640 1 -- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec0406cfb0 msgr2=0x7fec040ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7fec03400640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec0406cfb0 0x7fec040ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7fec03400640 1 -- 172.21.15.69:0/4095998675 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec04152340 con 0x7fec0415e960 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7fec02a00640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec0406cfb0 0x7fec040ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:45.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.311+0000 7fec03400640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0415e960 0x7fec040b8d30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7febfc007e70 tx=0x7febfc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.312+0000 7febfb400640 1 -- 172.21.15.69:0/4095998675 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc014620 con 0x7fec0415e960 2024-04-24T08:09:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.312+0000 7febfb400640 1 -- 172.21.15.69:0/4095998675 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febfc019070 con 0x7fec0415e960 2024-04-24T08:09:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.312+0000 7fec03e00640 1 -- 172.21.15.69:0/4095998675 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec040cbb80 con 0x7fec0415e960 2024-04-24T08:09:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.312+0000 7febfb400640 1 -- 172.21.15.69:0/4095998675 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc014a20 con 0x7fec0415e960 2024-04-24T08:09:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.312+0000 7fec03e00640 1 -- 172.21.15.69:0/4095998675 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec040cc140 con 0x7fec0415e960 2024-04-24T08:09:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.313+0000 7fec03e00640 1 -- 172.21.15.69:0/4095998675 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec04075d50 con 0x7fec0415e960 2024-04-24T08:09:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.314+0000 7febfb400640 1 -- 172.21.15.69:0/4095998675 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febfc014bc0 con 0x7fec0415e960 2024-04-24T08:09:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.315+0000 7febfb400640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febe006c920 0x7febe006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.315+0000 7febfb400640 1 -- 172.21.15.69:0/4095998675 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7febfc08c500 con 0x7fec0415e960 2024-04-24T08:09:45.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.315+0000 7fec02a00640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febe006c920 0x7febe006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:45.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.316+0000 7fec02a00640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febe006c920 0x7febe006ede0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7febf4000f60 tx=0x7febf4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:45.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.319+0000 7febfb400640 1 -- 172.21.15.69:0/4095998675 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febfc011260 con 0x7fec0415e960 2024-04-24T08:09:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.515+0000 7fec03e00640 1 -- 172.21.15.69:0/4095998675 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fec040d1dc0 con 0x7febe006c920 2024-04-24T08:09:45.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.525+0000 7febfb400640 1 -- 172.21.15.69:0/4095998675 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251955 (secure 0 0 0) 0x7fec040d1dc0 con 0x7febe006c920 2024-04-24T08:09:45.529 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:45.530 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:09:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 -- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febe006c920 msgr2=0x7febe006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febe006c920 0x7febe006ede0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7febf4000f60 tx=0x7febf4009040 comp rx=0 tx=0).stop 2024-04-24T08:09:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 -- 172.21.15.69:0/4095998675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0415e960 msgr2=0x7fec040b8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0415e960 0x7fec040b8d30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7febfc007e70 tx=0x7febfc00e4b0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 -- 172.21.15.69:0/4095998675 shutdown_connections 2024-04-24T08:09:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febe006c920 0x7febe006ede0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec0415e960 0x7fec040b8d30 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec0406d410 0x7fec040b87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 --2- 172.21.15.69:0/4095998675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec0406cfb0 0x7fec040ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:45.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 -- 172.21.15.69:0/4095998675 >> 172.21.15.69:0/4095998675 conn(0x7fec0409d900 msgr2=0x7fec040b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:45.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 -- 172.21.15.69:0/4095998675 shutdown_connections 2024-04-24T08:09:45.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:45.531+0000 7fec03e00640 1 -- 172.21.15.69:0/4095998675 wait complete. 2024-04-24T08:09:45.550 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-04-24T08:09:44.690878+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220312,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754408,"statfs":{"total":773094113280,"available":772868513792,"internally_reserved":0,"allocated":4235264,"data_stored":2359716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.615389"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311306+0000","last_change":"2024-04-24T08:09:39.125219+0000","last_active":"2024-04-24T08:09:40.311306+0000","last_peered":"2024-04-24T08:09:40.311306+0000","last_clean":"2024-04-24T08:09:40.311306+0000","last_became_active":"2024-04-24T08:09:39.124763+0000","last_became_peered":"2024-04-24T08:09:39.124763+0000","last_unstale":"2024-04-24T08:09:40.311306+0000","last_undegraded":"2024-04-24T08:09:40.311306+0000","last_fullsized":"2024-04-24T08:09:40.311306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:54.979294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310409+0000","last_change":"2024-04-24T08:09:39.103863+0000","last_active":"2024-04-24T08:09:40.310409+0000","last_peered":"2024-04-24T08:09:40.310409+0000","last_clean":"2024-04-24T08:09:40.310409+0000","last_became_active":"2024-04-24T08:09:39.103592+0000","last_became_peered":"2024-04-24T08:09:39.103592+0000","last_unstale":"2024-04-24T08:09:40.310409+0000","last_undegraded":"2024-04-24T08:09:40.310409+0000","last_fullsized":"2024-04-24T08:09:40.310409+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:19:54.217101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.116541+0000","last_change":"2024-04-24T08:09:39.111170+0000","last_active":"2024-04-24T08:09:40.116541+0000","last_peered":"2024-04-24T08:09:40.116541+0000","last_clean":"2024-04-24T08:09:40.116541+0000","last_became_active":"2024-04-24T08:09:39.110658+0000","last_became_peered":"2024-04-24T08:09:39.110658+0000","last_unstale":"2024-04-24T08:09:40.116541+0000","last_undegraded":"2024-04-24T08:09:40.116541+0000","last_fullsized":"2024-04-24T08:09:40.116541+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:16:48.624830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284665+0000","last_change":"2024-04-24T08:09:39.108422+0000","last_active":"2024-04-24T08:09:40.284665+0000","last_peered":"2024-04-24T08:09:40.284665+0000","last_clean":"2024-04-24T08:09:40.284665+0000","last_became_active":"2024-04-24T08:09:39.107895+0000","last_became_peered":"2024-04-24T08:09:39.107895+0000","last_unstale":"2024-04-24T08:09:40.284665+0000","last_undegraded":"2024-04-24T08:09:40.284665+0000","last_fullsized":"2024-04-24T08:09:40.284665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:17.553795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309432+0000","last_change":"2024-04-24T08:09:39.107555+0000","last_active":"2024-04-24T08:09:40.309432+0000","last_peered":"2024-04-24T08:09:40.309432+0000","last_clean":"2024-04-24T08:09:40.309432+0000","last_became_active":"2024-04-24T08:09:39.105910+0000","last_became_peered":"2024-04-24T08:09:39.105910+0000","last_unstale":"2024-04-24T08:09:40.309432+0000","last_undegraded":"2024-04-24T08:09:40.309432+0000","last_fullsized":"2024-04-24T08:09:40.309432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:13.439187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311055+0000","last_change":"2024-04-24T08:09:39.120619+0000","last_active":"2024-04-24T08:09:40.311055+0000","last_peered":"2024-04-24T08:09:40.311055+0000","last_clean":"2024-04-24T08:09:40.311055+0000","last_became_active":"2024-04-24T08:09:39.120309+0000","last_became_peered":"2024-04-24T08:09:39.120309+0000","last_unstale":"2024-04-24T08:09:40.311055+0000","last_undegraded":"2024-04-24T08:09:40.311055+0000","last_fullsized":"2024-04-24T08:09:40.311055+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:11:52.251677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309862+0000","last_change":"2024-04-24T08:09:39.125840+0000","last_active":"2024-04-24T08:09:40.309862+0000","last_peered":"2024-04-24T08:09:40.309862+0000","last_clean":"2024-04-24T08:09:40.309862+0000","last_became_active":"2024-04-24T08:09:39.125236+0000","last_became_peered":"2024-04-24T08:09:39.125236+0000","last_unstale":"2024-04-24T08:09:40.309862+0000","last_undegraded":"2024-04-24T08:09:40.309862+0000","last_fullsized":"2024-04-24T08:09:40.309862+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:16:51.790913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309221+0000","last_change":"2024-04-24T08:09:39.117265+0000","last_active":"2024-04-24T08:09:40.309221+0000","last_peered":"2024-04-24T08:09:40.309221+0000","last_clean":"2024-04-24T08:09:40.309221+0000","last_became_active":"2024-04-24T08:09:39.116973+0000","last_became_peered":"2024-04-24T08:09:39.116973+0000","last_unstale":"2024-04-24T08:09:40.309221+0000","last_undegraded":"2024-04-24T08:09:40.309221+0000","last_fullsized":"2024-04-24T08:09:40.309221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:32.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.122413+0000","last_change":"2024-04-24T08:09:39.106586+0000","last_active":"2024-04-24T08:09:40.122413+0000","last_peered":"2024-04-24T08:09:40.122413+0000","last_clean":"2024-04-24T08:09:40.122413+0000","last_became_active":"2024-04-24T08:09:39.106096+0000","last_became_peered":"2024-04-24T08:09:39.106096+0000","last_unstale":"2024-04-24T08:09:40.122413+0000","last_undegraded":"2024-04-24T08:09:40.122413+0000","last_fullsized":"2024-04-24T08:09:40.122413+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:32.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285229+0000","last_change":"2024-04-24T08:09:39.117928+0000","last_active":"2024-04-24T08:09:40.285229+0000","last_peered":"2024-04-24T08:09:40.285229+0000","last_clean":"2024-04-24T08:09:40.285229+0000","last_became_active":"2024-04-24T08:09:39.117634+0000","last_became_peered":"2024-04-24T08:09:39.117634+0000","last_unstale":"2024-04-24T08:09:40.285229+0000","last_undegraded":"2024-04-24T08:09:40.285229+0000","last_fullsized":"2024-04-24T08:09:40.285229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:45:39.110898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310529+0000","last_change":"2024-04-24T08:09:39.127475+0000","last_active":"2024-04-24T08:09:40.310529+0000","last_peered":"2024-04-24T08:09:40.310529+0000","last_clean":"2024-04-24T08:09:40.310529+0000","last_became_active":"2024-04-24T08:09:39.127153+0000","last_became_peered":"2024-04-24T08:09:39.127153+0000","last_unstale":"2024-04-24T08:09:40.310529+0000","last_undegraded":"2024-04-24T08:09:40.310529+0000","last_fullsized":"2024-04-24T08:09:40.310529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:38:26.155101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.116166+0000","last_change":"2024-04-24T08:09:39.121787+0000","last_active":"2024-04-24T08:09:40.116166+0000","last_peered":"2024-04-24T08:09:40.116166+0000","last_clean":"2024-04-24T08:09:40.116166+0000","last_became_active":"2024-04-24T08:09:39.121504+0000","last_became_peered":"2024-04-24T08:09:39.121504+0000","last_unstale":"2024-04-24T08:09:40.116166+0000","last_undegraded":"2024-04-24T08:09:40.116166+0000","last_fullsized":"2024-04-24T08:09:40.116166+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:16:51.790913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309817+0000","last_change":"2024-04-24T08:09:39.101508+0000","last_active":"2024-04-24T08:09:40.309817+0000","last_peered":"2024-04-24T08:09:40.309817+0000","last_clean":"2024-04-24T08:09:40.309817+0000","last_became_active":"2024-04-24T08:09:39.101181+0000","last_became_peered":"2024-04-24T08:09:39.101181+0000","last_unstale":"2024-04-24T08:09:40.309817+0000","last_undegraded":"2024-04-24T08:09:40.309817+0000","last_fullsized":"2024-04-24T08:09:40.309817+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:53.588357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310992+0000","last_change":"2024-04-24T08:09:39.106541+0000","last_active":"2024-04-24T08:09:40.310992+0000","last_peered":"2024-04-24T08:09:40.310992+0000","last_clean":"2024-04-24T08:09:40.310992+0000","last_became_active":"2024-04-24T08:09:39.106096+0000","last_became_peered":"2024-04-24T08:09:39.106096+0000","last_unstale":"2024-04-24T08:09:40.310992+0000","last_undegraded":"2024-04-24T08:09:40.310992+0000","last_fullsized":"2024-04-24T08:09:40.310992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:00:29.600153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.115480+0000","last_change":"2024-04-24T08:09:39.118355+0000","last_active":"2024-04-24T08:09:40.115480+0000","last_peered":"2024-04-24T08:09:40.115480+0000","last_clean":"2024-04-24T08:09:40.115480+0000","last_became_active":"2024-04-24T08:09:39.118091+0000","last_became_peered":"2024-04-24T08:09:39.118091+0000","last_unstale":"2024-04-24T08:09:40.115480+0000","last_undegraded":"2024-04-24T08:09:40.115480+0000","last_fullsized":"2024-04-24T08:09:40.115480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:45:39.110898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310097+0000","last_change":"2024-04-24T08:09:39.118631+0000","last_active":"2024-04-24T08:09:40.310097+0000","last_peered":"2024-04-24T08:09:40.310097+0000","last_clean":"2024-04-24T08:09:40.310097+0000","last_became_active":"2024-04-24T08:09:39.118352+0000","last_became_peered":"2024-04-24T08:09:39.118352+0000","last_unstale":"2024-04-24T08:09:40.310097+0000","last_undegraded":"2024-04-24T08:09:40.310097+0000","last_fullsized":"2024-04-24T08:09:40.310097+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:16:48.624830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311029+0000","last_change":"2024-04-24T08:09:39.120668+0000","last_active":"2024-04-24T08:09:40.311029+0000","last_peered":"2024-04-24T08:09:40.311029+0000","last_clean":"2024-04-24T08:09:40.311029+0000","last_became_active":"2024-04-24T08:09:39.119974+0000","last_became_peered":"2024-04-24T08:09:39.119974+0000","last_unstale":"2024-04-24T08:09:40.311029+0000","last_undegraded":"2024-04-24T08:09:40.311029+0000","last_fullsized":"2024-04-24T08:09:40.311029+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:23:17.124621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310462+0000","last_change":"2024-04-24T08:09:39.106863+0000","last_active":"2024-04-24T08:09:40.310462+0000","last_peered":"2024-04-24T08:09:40.310462+0000","last_clean":"2024-04-24T08:09:40.310462+0000","last_became_active":"2024-04-24T08:09:39.106186+0000","last_became_peered":"2024-04-24T08:09:39.106186+0000","last_unstale":"2024-04-24T08:09:40.310462+0000","last_undegraded":"2024-04-24T08:09:40.310462+0000","last_fullsized":"2024-04-24T08:09:40.310462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:54:36.719416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310033+0000","last_change":"2024-04-24T08:09:39.119111+0000","last_active":"2024-04-24T08:09:40.310033+0000","last_peered":"2024-04-24T08:09:40.310033+0000","last_clean":"2024-04-24T08:09:40.310033+0000","last_became_active":"2024-04-24T08:09:39.115194+0000","last_became_peered":"2024-04-24T08:09:39.115194+0000","last_unstale":"2024-04-24T08:09:40.310033+0000","last_undegraded":"2024-04-24T08:09:40.310033+0000","last_fullsized":"2024-04-24T08:09:40.310033+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:23:17.124621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310549+0000","last_change":"2024-04-24T08:09:39.121570+0000","last_active":"2024-04-24T08:09:40.310549+0000","last_peered":"2024-04-24T08:09:40.310549+0000","last_clean":"2024-04-24T08:09:40.310549+0000","last_became_active":"2024-04-24T08:09:39.121118+0000","last_became_peered":"2024-04-24T08:09:39.121118+0000","last_unstale":"2024-04-24T08:09:40.310549+0000","last_undegraded":"2024-04-24T08:09:40.310549+0000","last_fullsized":"2024-04-24T08:09:40.310549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:28.471092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121434+0000","last_change":"2024-04-24T08:09:39.107878+0000","last_active":"2024-04-24T08:09:40.121434+0000","last_peered":"2024-04-24T08:09:40.121434+0000","last_clean":"2024-04-24T08:09:40.121434+0000","last_became_active":"2024-04-24T08:09:39.106637+0000","last_became_peered":"2024-04-24T08:09:39.106637+0000","last_unstale":"2024-04-24T08:09:40.121434+0000","last_undegraded":"2024-04-24T08:09:40.121434+0000","last_fullsized":"2024-04-24T08:09:40.121434+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:58.147840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.120561+0000","last_change":"2024-04-24T08:09:39.110476+0000","last_active":"2024-04-24T08:09:40.120561+0000","last_peered":"2024-04-24T08:09:40.120561+0000","last_clean":"2024-04-24T08:09:40.120561+0000","last_became_active":"2024-04-24T08:09:39.110161+0000","last_became_peered":"2024-04-24T08:09:39.110161+0000","last_unstale":"2024-04-24T08:09:40.120561+0000","last_undegraded":"2024-04-24T08:09:40.120561+0000","last_fullsized":"2024-04-24T08:09:40.120561+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:47:51.470569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.111473+0000","last_change":"2024-04-24T08:09:39.105741+0000","last_active":"2024-04-24T08:09:40.111473+0000","last_peered":"2024-04-24T08:09:40.111473+0000","last_clean":"2024-04-24T08:09:40.111473+0000","last_became_active":"2024-04-24T08:09:39.105447+0000","last_became_peered":"2024-04-24T08:09:39.105447+0000","last_unstale":"2024-04-24T08:09:40.111473+0000","last_undegraded":"2024-04-24T08:09:40.111473+0000","last_fullsized":"2024-04-24T08:09:40.111473+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:54.979294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.115432+0000","last_change":"2024-04-24T08:09:39.110197+0000","last_active":"2024-04-24T08:09:40.115432+0000","last_peered":"2024-04-24T08:09:40.115432+0000","last_clean":"2024-04-24T08:09:40.115432+0000","last_became_active":"2024-04-24T08:09:39.109709+0000","last_became_peered":"2024-04-24T08:09:39.109709+0000","last_unstale":"2024-04-24T08:09:40.115432+0000","last_undegraded":"2024-04-24T08:09:40.115432+0000","last_fullsized":"2024-04-24T08:09:40.115432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:32.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285470+0000","last_change":"2024-04-24T08:09:39.111278+0000","last_active":"2024-04-24T08:09:40.285470+0000","last_peered":"2024-04-24T08:09:40.285470+0000","last_clean":"2024-04-24T08:09:40.285470+0000","last_became_active":"2024-04-24T08:09:39.110843+0000","last_became_peered":"2024-04-24T08:09:39.110843+0000","last_unstale":"2024-04-24T08:09:40.285470+0000","last_undegraded":"2024-04-24T08:09:40.285470+0000","last_fullsized":"2024-04-24T08:09:40.285470+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:16:48.624830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310811+0000","last_change":"2024-04-24T08:09:39.116987+0000","last_active":"2024-04-24T08:09:40.310811+0000","last_peered":"2024-04-24T08:09:40.310811+0000","last_clean":"2024-04-24T08:09:40.310811+0000","last_became_active":"2024-04-24T08:09:39.116660+0000","last_became_peered":"2024-04-24T08:09:39.116660+0000","last_unstale":"2024-04-24T08:09:40.310811+0000","last_undegraded":"2024-04-24T08:09:40.310811+0000","last_fullsized":"2024-04-24T08:09:40.310811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:45:10.083436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285401+0000","last_change":"2024-04-24T08:09:39.119204+0000","last_active":"2024-04-24T08:09:40.285401+0000","last_peered":"2024-04-24T08:09:40.285401+0000","last_clean":"2024-04-24T08:09:40.285401+0000","last_became_active":"2024-04-24T08:09:39.118896+0000","last_became_peered":"2024-04-24T08:09:39.118896+0000","last_unstale":"2024-04-24T08:09:40.285401+0000","last_undegraded":"2024-04-24T08:09:40.285401+0000","last_fullsized":"2024-04-24T08:09:40.285401+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:43:09.912013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.116302+0000","last_change":"2024-04-24T08:09:39.116617+0000","last_active":"2024-04-24T08:09:40.116302+0000","last_peered":"2024-04-24T08:09:40.116302+0000","last_clean":"2024-04-24T08:09:40.116302+0000","last_became_active":"2024-04-24T08:09:39.116298+0000","last_became_peered":"2024-04-24T08:09:39.116298+0000","last_unstale":"2024-04-24T08:09:40.116302+0000","last_undegraded":"2024-04-24T08:09:40.116302+0000","last_fullsized":"2024-04-24T08:09:40.116302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:43:09.912013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119992+0000","last_change":"2024-04-24T08:09:39.109527+0000","last_active":"2024-04-24T08:09:40.119992+0000","last_peered":"2024-04-24T08:09:40.119992+0000","last_clean":"2024-04-24T08:09:40.119992+0000","last_became_active":"2024-04-24T08:09:39.109192+0000","last_became_peered":"2024-04-24T08:09:39.109192+0000","last_unstale":"2024-04-24T08:09:40.119992+0000","last_undegraded":"2024-04-24T08:09:40.119992+0000","last_fullsized":"2024-04-24T08:09:40.119992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:37:08.649309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310110+0000","last_change":"2024-04-24T08:09:39.126432+0000","last_active":"2024-04-24T08:09:40.310110+0000","last_peered":"2024-04-24T08:09:40.310110+0000","last_clean":"2024-04-24T08:09:40.310110+0000","last_became_active":"2024-04-24T08:09:39.126157+0000","last_became_peered":"2024-04-24T08:09:39.126157+0000","last_unstale":"2024-04-24T08:09:40.310110+0000","last_undegraded":"2024-04-24T08:09:40.310110+0000","last_fullsized":"2024-04-24T08:09:40.310110+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:43:09.912013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309299+0000","last_change":"2024-04-24T08:09:39.111790+0000","last_active":"2024-04-24T08:09:40.309299+0000","last_peered":"2024-04-24T08:09:40.309299+0000","last_clean":"2024-04-24T08:09:40.309299+0000","last_became_active":"2024-04-24T08:09:39.110665+0000","last_became_peered":"2024-04-24T08:09:39.110665+0000","last_unstale":"2024-04-24T08:09:40.309299+0000","last_undegraded":"2024-04-24T08:09:40.309299+0000","last_fullsized":"2024-04-24T08:09:40.309299+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:34:51.226727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309336+0000","last_change":"2024-04-24T08:09:39.112112+0000","last_active":"2024-04-24T08:09:40.309336+0000","last_peered":"2024-04-24T08:09:40.309336+0000","last_clean":"2024-04-24T08:09:40.309336+0000","last_became_active":"2024-04-24T08:09:39.111791+0000","last_became_peered":"2024-04-24T08:09:39.111791+0000","last_unstale":"2024-04-24T08:09:40.309336+0000","last_undegraded":"2024-04-24T08:09:40.309336+0000","last_fullsized":"2024-04-24T08:09:40.309336+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:17.553795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112778+0000","last_change":"2024-04-24T08:09:39.106665+0000","last_active":"2024-04-24T08:09:40.112778+0000","last_peered":"2024-04-24T08:09:40.112778+0000","last_clean":"2024-04-24T08:09:40.112778+0000","last_became_active":"2024-04-24T08:09:39.105378+0000","last_became_peered":"2024-04-24T08:09:39.105378+0000","last_unstale":"2024-04-24T08:09:40.112778+0000","last_undegraded":"2024-04-24T08:09:40.112778+0000","last_fullsized":"2024-04-24T08:09:40.112778+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:58.147840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311474+0000","last_change":"2024-04-24T08:09:38.106509+0000","last_active":"2024-04-24T08:09:40.311474+0000","last_peered":"2024-04-24T08:09:40.311474+0000","last_clean":"2024-04-24T08:09:40.311474+0000","last_became_active":"2024-04-24T08:09:38.106216+0000","last_became_peered":"2024-04-24T08:09:38.106216+0000","last_unstale":"2024-04-24T08:09:40.311474+0000","last_undegraded":"2024-04-24T08:09:40.311474+0000","last_fullsized":"2024-04-24T08:09:40.311474+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311454+0000","last_change":"2024-04-24T08:09:39.108958+0000","last_active":"2024-04-24T08:09:40.311454+0000","last_peered":"2024-04-24T08:09:40.311454+0000","last_clean":"2024-04-24T08:09:40.311454+0000","last_became_active":"2024-04-24T08:09:39.107924+0000","last_became_peered":"2024-04-24T08:09:39.107924+0000","last_unstale":"2024-04-24T08:09:40.311454+0000","last_undegraded":"2024-04-24T08:09:40.311454+0000","last_fullsized":"2024-04-24T08:09:40.311454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:37:08.649309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112677+0000","last_change":"2024-04-24T08:09:38.107316+0000","last_active":"2024-04-24T08:09:40.112677+0000","last_peered":"2024-04-24T08:09:40.112677+0000","last_clean":"2024-04-24T08:09:40.112677+0000","last_became_active":"2024-04-24T08:09:38.106891+0000","last_became_peered":"2024-04-24T08:09:38.106891+0000","last_unstale":"2024-04-24T08:09:40.112677+0000","last_undegraded":"2024-04-24T08:09:40.112677+0000","last_fullsized":"2024-04-24T08:09:40.112677+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121657+0000","last_change":"2024-04-24T08:09:39.115826+0000","last_active":"2024-04-24T08:09:40.121657+0000","last_peered":"2024-04-24T08:09:40.121657+0000","last_clean":"2024-04-24T08:09:40.121657+0000","last_became_active":"2024-04-24T08:09:39.115503+0000","last_became_peered":"2024-04-24T08:09:39.115503+0000","last_unstale":"2024-04-24T08:09:40.121657+0000","last_undegraded":"2024-04-24T08:09:40.121657+0000","last_fullsized":"2024-04-24T08:09:40.121657+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:55:27.468741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309859+0000","last_change":"2024-04-24T08:09:38.105975+0000","last_active":"2024-04-24T08:09:40.309859+0000","last_peered":"2024-04-24T08:09:40.309859+0000","last_clean":"2024-04-24T08:09:40.309859+0000","last_became_active":"2024-04-24T08:09:38.105684+0000","last_became_peered":"2024-04-24T08:09:38.105684+0000","last_unstale":"2024-04-24T08:09:40.309859+0000","last_undegraded":"2024-04-24T08:09:40.309859+0000","last_fullsized":"2024-04-24T08:09:40.309859+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:37:07.635670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285447+0000","last_change":"2024-04-24T08:09:39.119133+0000","last_active":"2024-04-24T08:09:40.285447+0000","last_peered":"2024-04-24T08:09:40.285447+0000","last_clean":"2024-04-24T08:09:40.285447+0000","last_became_active":"2024-04-24T08:09:39.118926+0000","last_became_peered":"2024-04-24T08:09:39.118926+0000","last_unstale":"2024-04-24T08:09:40.285447+0000","last_undegraded":"2024-04-24T08:09:40.285447+0000","last_fullsized":"2024-04-24T08:09:40.285447+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:16:51.790913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310800+0000","last_change":"2024-04-24T08:09:38.102564+0000","last_active":"2024-04-24T08:09:40.310800+0000","last_peered":"2024-04-24T08:09:40.310800+0000","last_clean":"2024-04-24T08:09:40.310800+0000","last_became_active":"2024-04-24T08:09:38.101598+0000","last_became_peered":"2024-04-24T08:09:38.101598+0000","last_unstale":"2024-04-24T08:09:40.310800+0000","last_undegraded":"2024-04-24T08:09:40.310800+0000","last_fullsized":"2024-04-24T08:09:40.310800+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:58:11.279574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310554+0000","last_change":"2024-04-24T08:09:39.121972+0000","last_active":"2024-04-24T08:09:40.310554+0000","last_peered":"2024-04-24T08:09:40.310554+0000","last_clean":"2024-04-24T08:09:40.310554+0000","last_became_active":"2024-04-24T08:09:39.120976+0000","last_became_peered":"2024-04-24T08:09:39.120976+0000","last_unstale":"2024-04-24T08:09:40.310554+0000","last_undegraded":"2024-04-24T08:09:40.310554+0000","last_fullsized":"2024-04-24T08:09:40.310554+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:53.687771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.120197+0000","last_change":"2024-04-24T08:09:38.101731+0000","last_active":"2024-04-24T08:09:40.120197+0000","last_peered":"2024-04-24T08:09:40.120197+0000","last_clean":"2024-04-24T08:09:40.120197+0000","last_became_active":"2024-04-24T08:09:38.100536+0000","last_became_peered":"2024-04-24T08:09:38.100536+0000","last_unstale":"2024-04-24T08:09:40.120197+0000","last_undegraded":"2024-04-24T08:09:40.120197+0000","last_fullsized":"2024-04-24T08:09:40.120197+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:05.837977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.115503+0000","last_change":"2024-04-24T08:09:39.122882+0000","last_active":"2024-04-24T08:09:40.115503+0000","last_peered":"2024-04-24T08:09:40.115503+0000","last_clean":"2024-04-24T08:09:40.115503+0000","last_became_active":"2024-04-24T08:09:39.122534+0000","last_became_peered":"2024-04-24T08:09:39.122534+0000","last_unstale":"2024-04-24T08:09:40.115503+0000","last_undegraded":"2024-04-24T08:09:40.115503+0000","last_fullsized":"2024-04-24T08:09:40.115503+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:06:14.738196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.120423+0000","last_change":"2024-04-24T08:09:38.108989+0000","last_active":"2024-04-24T08:09:40.120423+0000","last_peered":"2024-04-24T08:09:40.120423+0000","last_clean":"2024-04-24T08:09:40.120423+0000","last_became_active":"2024-04-24T08:09:38.108726+0000","last_became_peered":"2024-04-24T08:09:38.108726+0000","last_unstale":"2024-04-24T08:09:40.120423+0000","last_undegraded":"2024-04-24T08:09:40.120423+0000","last_fullsized":"2024-04-24T08:09:40.120423+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:42.105437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119781+0000","last_change":"2024-04-24T08:09:39.115338+0000","last_active":"2024-04-24T08:09:40.119781+0000","last_peered":"2024-04-24T08:09:40.119781+0000","last_clean":"2024-04-24T08:09:40.119781+0000","last_became_active":"2024-04-24T08:09:39.115065+0000","last_became_peered":"2024-04-24T08:09:39.115065+0000","last_unstale":"2024-04-24T08:09:40.119781+0000","last_undegraded":"2024-04-24T08:09:40.119781+0000","last_fullsized":"2024-04-24T08:09:40.119781+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:54.979294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308969+0000","last_change":"2024-04-24T08:09:38.093313+0000","last_active":"2024-04-24T08:09:40.308969+0000","last_peered":"2024-04-24T08:09:40.308969+0000","last_clean":"2024-04-24T08:09:40.308969+0000","last_became_active":"2024-04-24T08:09:38.092910+0000","last_became_peered":"2024-04-24T08:09:38.092910+0000","last_unstale":"2024-04-24T08:09:40.308969+0000","last_undegraded":"2024-04-24T08:09:40.308969+0000","last_fullsized":"2024-04-24T08:09:40.308969+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:42:29.816207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284451+0000","last_change":"2024-04-24T08:09:39.125735+0000","last_active":"2024-04-24T08:09:40.284451+0000","last_peered":"2024-04-24T08:09:40.284451+0000","last_clean":"2024-04-24T08:09:40.284451+0000","last_became_active":"2024-04-24T08:09:39.125449+0000","last_became_peered":"2024-04-24T08:09:39.125449+0000","last_unstale":"2024-04-24T08:09:40.284451+0000","last_undegraded":"2024-04-24T08:09:40.284451+0000","last_fullsized":"2024-04-24T08:09:40.284451+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:32.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119602+0000","last_change":"2024-04-24T08:09:38.107454+0000","last_active":"2024-04-24T08:09:40.119602+0000","last_peered":"2024-04-24T08:09:40.119602+0000","last_clean":"2024-04-24T08:09:40.119602+0000","last_became_active":"2024-04-24T08:09:38.107149+0000","last_became_peered":"2024-04-24T08:09:38.107149+0000","last_unstale":"2024-04-24T08:09:40.119602+0000","last_undegraded":"2024-04-24T08:09:40.119602+0000","last_fullsized":"2024-04-24T08:09:40.119602+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.117340+0000","last_change":"2024-04-24T08:09:39.121519+0000","last_active":"2024-04-24T08:09:40.117340+0000","last_peered":"2024-04-24T08:09:40.117340+0000","last_clean":"2024-04-24T08:09:40.117340+0000","last_became_active":"2024-04-24T08:09:39.120897+0000","last_became_peered":"2024-04-24T08:09:39.120897+0000","last_unstale":"2024-04-24T08:09:40.117340+0000","last_undegraded":"2024-04-24T08:09:40.117340+0000","last_fullsized":"2024-04-24T08:09:40.117340+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:48.329141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310742+0000","last_change":"2024-04-24T08:09:38.110451+0000","last_active":"2024-04-24T08:09:40.310742+0000","last_peered":"2024-04-24T08:09:40.310742+0000","last_clean":"2024-04-24T08:09:40.310742+0000","last_became_active":"2024-04-24T08:09:38.110191+0000","last_became_peered":"2024-04-24T08:09:38.110191+0000","last_unstale":"2024-04-24T08:09:40.310742+0000","last_undegraded":"2024-04-24T08:09:40.310742+0000","last_fullsized":"2024-04-24T08:09:40.310742+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:47:50.456931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310331+0000","last_change":"2024-04-24T08:09:39.116365+0000","last_active":"2024-04-24T08:09:40.310331+0000","last_peered":"2024-04-24T08:09:40.310331+0000","last_clean":"2024-04-24T08:09:40.310331+0000","last_became_active":"2024-04-24T08:09:39.115283+0000","last_became_peered":"2024-04-24T08:09:39.115283+0000","last_unstale":"2024-04-24T08:09:40.310331+0000","last_undegraded":"2024-04-24T08:09:40.310331+0000","last_fullsized":"2024-04-24T08:09:40.310331+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:26:44.551602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112782+0000","last_change":"2024-04-24T08:09:38.093711+0000","last_active":"2024-04-24T08:09:40.112782+0000","last_peered":"2024-04-24T08:09:40.112782+0000","last_clean":"2024-04-24T08:09:40.112782+0000","last_became_active":"2024-04-24T08:09:38.091626+0000","last_became_peered":"2024-04-24T08:09:38.091626+0000","last_unstale":"2024-04-24T08:09:40.112782+0000","last_undegraded":"2024-04-24T08:09:40.112782+0000","last_fullsized":"2024-04-24T08:09:40.112782+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:11.064191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309629+0000","last_change":"2024-04-24T08:09:39.111382+0000","last_active":"2024-04-24T08:09:40.309629+0000","last_peered":"2024-04-24T08:09:40.309629+0000","last_clean":"2024-04-24T08:09:40.309629+0000","last_became_active":"2024-04-24T08:09:39.108541+0000","last_became_peered":"2024-04-24T08:09:39.108541+0000","last_unstale":"2024-04-24T08:09:40.309629+0000","last_undegraded":"2024-04-24T08:09:40.309629+0000","last_fullsized":"2024-04-24T08:09:40.309629+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:38:26.155101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.116469+0000","last_change":"2024-04-24T08:09:38.107829+0000","last_active":"2024-04-24T08:09:40.116469+0000","last_peered":"2024-04-24T08:09:40.116469+0000","last_clean":"2024-04-24T08:09:40.116469+0000","last_became_active":"2024-04-24T08:09:38.107551+0000","last_became_peered":"2024-04-24T08:09:38.107551+0000","last_unstale":"2024-04-24T08:09:40.116469+0000","last_undegraded":"2024-04-24T08:09:40.116469+0000","last_fullsized":"2024-04-24T08:09:40.116469+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:10.779345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.115482+0000","last_change":"2024-04-24T08:09:39.111833+0000","last_active":"2024-04-24T08:09:40.115482+0000","last_peered":"2024-04-24T08:09:40.115482+0000","last_clean":"2024-04-24T08:09:40.115482+0000","last_became_active":"2024-04-24T08:09:39.111444+0000","last_became_peered":"2024-04-24T08:09:39.111444+0000","last_unstale":"2024-04-24T08:09:40.115482+0000","last_undegraded":"2024-04-24T08:09:40.115482+0000","last_fullsized":"2024-04-24T08:09:40.115482+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:17.553795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309727+0000","last_change":"2024-04-24T08:09:38.106294+0000","last_active":"2024-04-24T08:09:40.309727+0000","last_peered":"2024-04-24T08:09:40.309727+0000","last_clean":"2024-04-24T08:09:40.309727+0000","last_became_active":"2024-04-24T08:09:38.105994+0000","last_became_peered":"2024-04-24T08:09:38.105994+0000","last_unstale":"2024-04-24T08:09:40.309727+0000","last_undegraded":"2024-04-24T08:09:40.309727+0000","last_fullsized":"2024-04-24T08:09:40.309727+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:53.965656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284266+0000","last_change":"2024-04-24T08:09:39.121621+0000","last_active":"2024-04-24T08:09:40.284266+0000","last_peered":"2024-04-24T08:09:40.284266+0000","last_clean":"2024-04-24T08:09:40.284266+0000","last_became_active":"2024-04-24T08:09:39.121289+0000","last_became_peered":"2024-04-24T08:09:39.121289+0000","last_unstale":"2024-04-24T08:09:40.284266+0000","last_undegraded":"2024-04-24T08:09:40.284266+0000","last_fullsized":"2024-04-24T08:09:40.284266+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:43:19.492962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121197+0000","last_change":"2024-04-24T08:09:38.105998+0000","last_active":"2024-04-24T08:09:40.121197+0000","last_peered":"2024-04-24T08:09:40.121197+0000","last_clean":"2024-04-24T08:09:40.121197+0000","last_became_active":"2024-04-24T08:09:38.105710+0000","last_became_peered":"2024-04-24T08:09:38.105710+0000","last_unstale":"2024-04-24T08:09:40.121197+0000","last_undegraded":"2024-04-24T08:09:40.121197+0000","last_fullsized":"2024-04-24T08:09:40.121197+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309926+0000","last_change":"2024-04-24T08:09:39.104007+0000","last_active":"2024-04-24T08:09:40.309926+0000","last_peered":"2024-04-24T08:09:40.309926+0000","last_clean":"2024-04-24T08:09:40.309926+0000","last_became_active":"2024-04-24T08:09:39.103739+0000","last_became_peered":"2024-04-24T08:09:39.103739+0000","last_unstale":"2024-04-24T08:09:40.309926+0000","last_undegraded":"2024-04-24T08:09:40.309926+0000","last_fullsized":"2024-04-24T08:09:40.309926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:53.687771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284378+0000","last_change":"2024-04-24T08:09:38.107964+0000","last_active":"2024-04-24T08:09:40.284378+0000","last_peered":"2024-04-24T08:09:40.284378+0000","last_clean":"2024-04-24T08:09:40.284378+0000","last_became_active":"2024-04-24T08:09:38.107620+0000","last_became_peered":"2024-04-24T08:09:38.107620+0000","last_unstale":"2024-04-24T08:09:40.284378+0000","last_undegraded":"2024-04-24T08:09:40.284378+0000","last_fullsized":"2024-04-24T08:09:40.284378+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309665+0000","last_change":"2024-04-24T08:09:39.118278+0000","last_active":"2024-04-24T08:09:40.309665+0000","last_peered":"2024-04-24T08:09:40.309665+0000","last_clean":"2024-04-24T08:09:40.309665+0000","last_became_active":"2024-04-24T08:09:39.117897+0000","last_became_peered":"2024-04-24T08:09:39.117897+0000","last_unstale":"2024-04-24T08:09:40.309665+0000","last_undegraded":"2024-04-24T08:09:40.309665+0000","last_fullsized":"2024-04-24T08:09:40.309665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:45:39.110898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308902+0000","last_change":"2024-04-24T08:09:38.103294+0000","last_active":"2024-04-24T08:09:40.308902+0000","last_peered":"2024-04-24T08:09:40.308902+0000","last_clean":"2024-04-24T08:09:40.308902+0000","last_became_active":"2024-04-24T08:09:38.101762+0000","last_became_peered":"2024-04-24T08:09:38.101762+0000","last_unstale":"2024-04-24T08:09:40.308902+0000","last_undegraded":"2024-04-24T08:09:40.308902+0000","last_fullsized":"2024-04-24T08:09:40.308902+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:57.918072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309463+0000","last_change":"2024-04-24T08:09:39.109911+0000","last_active":"2024-04-24T08:09:40.309463+0000","last_peered":"2024-04-24T08:09:40.309463+0000","last_clean":"2024-04-24T08:09:40.309463+0000","last_became_active":"2024-04-24T08:09:39.106934+0000","last_became_peered":"2024-04-24T08:09:39.106934+0000","last_unstale":"2024-04-24T08:09:40.309463+0000","last_undegraded":"2024-04-24T08:09:40.309463+0000","last_fullsized":"2024-04-24T08:09:40.309463+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:58.931710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308682+0000","last_change":"2024-04-24T08:09:38.100426+0000","last_active":"2024-04-24T08:09:40.308682+0000","last_peered":"2024-04-24T08:09:40.308682+0000","last_clean":"2024-04-24T08:09:40.308682+0000","last_became_active":"2024-04-24T08:09:38.100138+0000","last_became_peered":"2024-04-24T08:09:38.100138+0000","last_unstale":"2024-04-24T08:09:40.308682+0000","last_undegraded":"2024-04-24T08:09:40.308682+0000","last_fullsized":"2024-04-24T08:09:40.308682+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:34:50.213088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.122004+0000","last_change":"2024-04-24T08:09:39.122438+0000","last_active":"2024-04-24T08:09:40.122004+0000","last_peered":"2024-04-24T08:09:40.122004+0000","last_clean":"2024-04-24T08:09:40.122004+0000","last_became_active":"2024-04-24T08:09:39.121827+0000","last_became_peered":"2024-04-24T08:09:39.121827+0000","last_unstale":"2024-04-24T08:09:40.122004+0000","last_undegraded":"2024-04-24T08:09:40.122004+0000","last_fullsized":"2024-04-24T08:09:40.122004+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:43:19.492962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285624+0000","last_change":"2024-04-24T08:09:38.096536+0000","last_active":"2024-04-24T08:09:40.285624+0000","last_peered":"2024-04-24T08:09:40.285624+0000","last_clean":"2024-04-24T08:09:40.285624+0000","last_became_active":"2024-04-24T08:09:38.095915+0000","last_became_peered":"2024-04-24T08:09:38.095915+0000","last_unstale":"2024-04-24T08:09:40.285624+0000","last_undegraded":"2024-04-24T08:09:40.285624+0000","last_fullsized":"2024-04-24T08:09:40.285624+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:10.779345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.117569+0000","last_change":"2024-04-24T08:09:39.120695+0000","last_active":"2024-04-24T08:09:40.117569+0000","last_peered":"2024-04-24T08:09:40.117569+0000","last_clean":"2024-04-24T08:09:40.117569+0000","last_became_active":"2024-04-24T08:09:39.115473+0000","last_became_peered":"2024-04-24T08:09:39.115473+0000","last_unstale":"2024-04-24T08:09:40.117569+0000","last_undegraded":"2024-04-24T08:09:40.117569+0000","last_fullsized":"2024-04-24T08:09:40.117569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:38:26.155101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121882+0000","last_change":"2024-04-24T08:09:38.101152+0000","last_active":"2024-04-24T08:09:40.121882+0000","last_peered":"2024-04-24T08:09:40.121882+0000","last_clean":"2024-04-24T08:09:40.121882+0000","last_became_active":"2024-04-24T08:09:38.099292+0000","last_became_peered":"2024-04-24T08:09:38.099292+0000","last_unstale":"2024-04-24T08:09:40.121882+0000","last_undegraded":"2024-04-24T08:09:40.121882+0000","last_fullsized":"2024-04-24T08:09:40.121882+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284867+0000","last_change":"2024-04-24T08:09:39.108177+0000","last_active":"2024-04-24T08:09:40.284867+0000","last_peered":"2024-04-24T08:09:40.284867+0000","last_clean":"2024-04-24T08:09:40.284867+0000","last_became_active":"2024-04-24T08:09:39.107524+0000","last_became_peered":"2024-04-24T08:09:39.107524+0000","last_unstale":"2024-04-24T08:09:40.284867+0000","last_undegraded":"2024-04-24T08:09:40.284867+0000","last_fullsized":"2024-04-24T08:09:40.284867+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:06:14.738196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121973+0000","last_change":"2024-04-24T08:09:38.100668+0000","last_active":"2024-04-24T08:09:40.121973+0000","last_peered":"2024-04-24T08:09:40.121973+0000","last_clean":"2024-04-24T08:09:40.121973+0000","last_became_active":"2024-04-24T08:09:38.100303+0000","last_became_peered":"2024-04-24T08:09:38.100303+0000","last_unstale":"2024-04-24T08:09:40.121973+0000","last_undegraded":"2024-04-24T08:09:40.121973+0000","last_fullsized":"2024-04-24T08:09:40.121973+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:33.580704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311324+0000","last_change":"2024-04-24T08:09:39.109433+0000","last_active":"2024-04-24T08:09:40.311324+0000","last_peered":"2024-04-24T08:09:40.311324+0000","last_clean":"2024-04-24T08:09:40.311324+0000","last_became_active":"2024-04-24T08:09:39.108287+0000","last_became_peered":"2024-04-24T08:09:39.108287+0000","last_unstale":"2024-04-24T08:09:40.311324+0000","last_undegraded":"2024-04-24T08:09:40.311324+0000","last_fullsized":"2024-04-24T08:09:40.311324+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:22.010195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121571+0000","last_change":"2024-04-24T08:09:38.105765+0000","last_active":"2024-04-24T08:09:40.121571+0000","last_peered":"2024-04-24T08:09:40.121571+0000","last_clean":"2024-04-24T08:09:40.121571+0000","last_became_active":"2024-04-24T08:09:38.100242+0000","last_became_peered":"2024-04-24T08:09:38.100242+0000","last_unstale":"2024-04-24T08:09:40.121571+0000","last_undegraded":"2024-04-24T08:09:40.121571+0000","last_fullsized":"2024-04-24T08:09:40.121571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309041+0000","last_change":"2024-04-24T08:09:39.099317+0000","last_active":"2024-04-24T08:09:40.309041+0000","last_peered":"2024-04-24T08:09:40.309041+0000","last_clean":"2024-04-24T08:09:40.309041+0000","last_became_active":"2024-04-24T08:09:39.099056+0000","last_became_peered":"2024-04-24T08:09:39.099056+0000","last_unstale":"2024-04-24T08:09:40.309041+0000","last_undegraded":"2024-04-24T08:09:40.309041+0000","last_fullsized":"2024-04-24T08:09:40.309041+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:03.587948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112026+0000","last_change":"2024-04-24T08:09:38.097858+0000","last_active":"2024-04-24T08:09:40.112026+0000","last_peered":"2024-04-24T08:09:40.112026+0000","last_clean":"2024-04-24T08:09:40.112026+0000","last_became_active":"2024-04-24T08:09:38.097599+0000","last_became_peered":"2024-04-24T08:09:38.097599+0000","last_unstale":"2024-04-24T08:09:40.112026+0000","last_undegraded":"2024-04-24T08:09:40.112026+0000","last_fullsized":"2024-04-24T08:09:40.112026+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:10.779345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310172+0000","last_change":"2024-04-24T08:09:39.103765+0000","last_active":"2024-04-24T08:09:40.310172+0000","last_peered":"2024-04-24T08:09:40.310172+0000","last_clean":"2024-04-24T08:09:40.310172+0000","last_became_active":"2024-04-24T08:09:39.103468+0000","last_became_peered":"2024-04-24T08:09:39.103468+0000","last_unstale":"2024-04-24T08:09:40.310172+0000","last_undegraded":"2024-04-24T08:09:40.310172+0000","last_fullsized":"2024-04-24T08:09:40.310172+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:12.077829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310934+0000","last_change":"2024-04-24T08:09:38.110759+0000","last_active":"2024-04-24T08:09:40.310934+0000","last_peered":"2024-04-24T08:09:40.310934+0000","last_clean":"2024-04-24T08:09:40.310934+0000","last_became_active":"2024-04-24T08:09:38.110393+0000","last_became_peered":"2024-04-24T08:09:38.110393+0000","last_unstale":"2024-04-24T08:09:40.310934+0000","last_undegraded":"2024-04-24T08:09:40.310934+0000","last_fullsized":"2024-04-24T08:09:40.310934+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308561+0000","last_change":"2024-04-24T08:09:39.101642+0000","last_active":"2024-04-24T08:09:40.308561+0000","last_peered":"2024-04-24T08:09:40.308561+0000","last_clean":"2024-04-24T08:09:40.308561+0000","last_became_active":"2024-04-24T08:09:39.101382+0000","last_became_peered":"2024-04-24T08:09:39.101382+0000","last_unstale":"2024-04-24T08:09:40.308561+0000","last_undegraded":"2024-04-24T08:09:40.308561+0000","last_fullsized":"2024-04-24T08:09:40.308561+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:57.650910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119632+0000","last_change":"2024-04-24T08:09:38.104959+0000","last_active":"2024-04-24T08:09:40.119632+0000","last_peered":"2024-04-24T08:09:40.119632+0000","last_clean":"2024-04-24T08:09:40.119632+0000","last_became_active":"2024-04-24T08:09:38.104606+0000","last_became_peered":"2024-04-24T08:09:38.104606+0000","last_unstale":"2024-04-24T08:09:40.119632+0000","last_undegraded":"2024-04-24T08:09:40.119632+0000","last_fullsized":"2024-04-24T08:09:40.119632+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:04:53.285545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309446+0000","last_change":"2024-04-24T08:09:39.127115+0000","last_active":"2024-04-24T08:09:40.309446+0000","last_peered":"2024-04-24T08:09:40.309446+0000","last_clean":"2024-04-24T08:09:40.309446+0000","last_became_active":"2024-04-24T08:09:39.125800+0000","last_became_peered":"2024-04-24T08:09:39.125800+0000","last_unstale":"2024-04-24T08:09:40.309446+0000","last_undegraded":"2024-04-24T08:09:40.309446+0000","last_fullsized":"2024-04-24T08:09:40.309446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:06:14.738196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.114996+0000","last_change":"2024-04-24T08:09:38.104201+0000","last_active":"2024-04-24T08:09:40.114996+0000","last_peered":"2024-04-24T08:09:40.114996+0000","last_clean":"2024-04-24T08:09:40.114996+0000","last_became_active":"2024-04-24T08:09:38.103332+0000","last_became_peered":"2024-04-24T08:09:38.103332+0000","last_unstale":"2024-04-24T08:09:40.114996+0000","last_undegraded":"2024-04-24T08:09:40.114996+0000","last_fullsized":"2024-04-24T08:09:40.114996+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:27.457454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310983+0000","last_change":"2024-04-24T08:09:39.116271+0000","last_active":"2024-04-24T08:09:40.310983+0000","last_peered":"2024-04-24T08:09:40.310983+0000","last_clean":"2024-04-24T08:09:40.310983+0000","last_became_active":"2024-04-24T08:09:39.116002+0000","last_became_peered":"2024-04-24T08:09:39.116002+0000","last_unstale":"2024-04-24T08:09:40.310983+0000","last_undegraded":"2024-04-24T08:09:40.310983+0000","last_fullsized":"2024-04-24T08:09:40.310983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:11.792983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285653+0000","last_change":"2024-04-24T08:09:38.105102+0000","last_active":"2024-04-24T08:09:40.285653+0000","last_peered":"2024-04-24T08:09:40.285653+0000","last_clean":"2024-04-24T08:09:40.285653+0000","last_became_active":"2024-04-24T08:09:38.104814+0000","last_became_peered":"2024-04-24T08:09:38.104814+0000","last_unstale":"2024-04-24T08:09:40.285653+0000","last_undegraded":"2024-04-24T08:09:40.285653+0000","last_fullsized":"2024-04-24T08:09:40.285653+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:20.996557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.117636+0000","last_change":"2024-04-24T08:09:39.115185+0000","last_active":"2024-04-24T08:09:40.117636+0000","last_peered":"2024-04-24T08:09:40.117636+0000","last_clean":"2024-04-24T08:09:40.117636+0000","last_became_active":"2024-04-24T08:09:39.114901+0000","last_became_peered":"2024-04-24T08:09:39.114901+0000","last_unstale":"2024-04-24T08:09:40.117636+0000","last_undegraded":"2024-04-24T08:09:40.117636+0000","last_fullsized":"2024-04-24T08:09:40.117636+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:53.687771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112929+0000","last_change":"2024-04-24T08:09:38.102570+0000","last_active":"2024-04-24T08:09:40.112929+0000","last_peered":"2024-04-24T08:09:40.112929+0000","last_clean":"2024-04-24T08:09:40.112929+0000","last_became_active":"2024-04-24T08:09:38.101279+0000","last_became_peered":"2024-04-24T08:09:38.101279+0000","last_unstale":"2024-04-24T08:09:40.112929+0000","last_undegraded":"2024-04-24T08:09:40.112929+0000","last_fullsized":"2024-04-24T08:09:40.112929+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:27.457454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112274+0000","last_change":"2024-04-24T08:09:39.103942+0000","last_active":"2024-04-24T08:09:40.112274+0000","last_peered":"2024-04-24T08:09:40.112274+0000","last_clean":"2024-04-24T08:09:40.112274+0000","last_became_active":"2024-04-24T08:09:39.103686+0000","last_became_peered":"2024-04-24T08:09:39.103686+0000","last_unstale":"2024-04-24T08:09:40.112274+0000","last_undegraded":"2024-04-24T08:09:40.112274+0000","last_fullsized":"2024-04-24T08:09:40.112274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:47:51.470569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284982+0000","last_change":"2024-04-24T08:09:38.101682+0000","last_active":"2024-04-24T08:09:40.284982+0000","last_peered":"2024-04-24T08:09:40.284982+0000","last_clean":"2024-04-24T08:09:40.284982+0000","last_became_active":"2024-04-24T08:09:38.099204+0000","last_became_peered":"2024-04-24T08:09:38.099204+0000","last_unstale":"2024-04-24T08:09:40.284982+0000","last_undegraded":"2024-04-24T08:09:40.284982+0000","last_fullsized":"2024-04-24T08:09:40.284982+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:27.457454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310932+0000","last_change":"2024-04-24T08:09:39.114705+0000","last_active":"2024-04-24T08:09:40.310932+0000","last_peered":"2024-04-24T08:09:40.310932+0000","last_clean":"2024-04-24T08:09:40.310932+0000","last_became_active":"2024-04-24T08:09:39.114454+0000","last_became_peered":"2024-04-24T08:09:39.114454+0000","last_unstale":"2024-04-24T08:09:40.310932+0000","last_undegraded":"2024-04-24T08:09:40.310932+0000","last_fullsized":"2024-04-24T08:09:40.310932+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:42:30.829845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112199+0000","last_change":"2024-04-24T08:09:38.101668+0000","last_active":"2024-04-24T08:09:40.112199+0000","last_peered":"2024-04-24T08:09:40.112199+0000","last_clean":"2024-04-24T08:09:40.112199+0000","last_became_active":"2024-04-24T08:09:38.101318+0000","last_became_peered":"2024-04-24T08:09:38.101318+0000","last_unstale":"2024-04-24T08:09:40.112199+0000","last_undegraded":"2024-04-24T08:09:40.112199+0000","last_fullsized":"2024-04-24T08:09:40.112199+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:20.996557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.111985+0000","last_change":"2024-04-24T08:09:39.112842+0000","last_active":"2024-04-24T08:09:40.111985+0000","last_peered":"2024-04-24T08:09:40.111985+0000","last_clean":"2024-04-24T08:09:40.111985+0000","last_became_active":"2024-04-24T08:09:39.112556+0000","last_became_peered":"2024-04-24T08:09:39.112556+0000","last_unstale":"2024-04-24T08:09:40.111985+0000","last_undegraded":"2024-04-24T08:09:40.111985+0000","last_fullsized":"2024-04-24T08:09:40.111985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:37:08.649309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284545+0000","last_change":"2024-04-24T08:09:38.105797+0000","last_active":"2024-04-24T08:09:40.284545+0000","last_peered":"2024-04-24T08:09:40.284545+0000","last_clean":"2024-04-24T08:09:40.284545+0000","last_became_active":"2024-04-24T08:09:38.104048+0000","last_became_peered":"2024-04-24T08:09:38.104048+0000","last_unstale":"2024-04-24T08:09:40.284545+0000","last_undegraded":"2024-04-24T08:09:40.284545+0000","last_fullsized":"2024-04-24T08:09:40.284545+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:11.064191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308808+0000","last_change":"2024-04-24T08:09:39.113158+0000","last_active":"2024-04-24T08:09:40.308808+0000","last_peered":"2024-04-24T08:09:40.308808+0000","last_clean":"2024-04-24T08:09:40.308808+0000","last_became_active":"2024-04-24T08:09:39.112663+0000","last_became_peered":"2024-04-24T08:09:39.112663+0000","last_unstale":"2024-04-24T08:09:40.308808+0000","last_undegraded":"2024-04-24T08:09:40.308808+0000","last_fullsized":"2024-04-24T08:09:40.308808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:48.329141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119878+0000","last_change":"2024-04-24T08:09:38.103639+0000","last_active":"2024-04-24T08:09:40.119878+0000","last_peered":"2024-04-24T08:09:40.119878+0000","last_clean":"2024-04-24T08:09:40.119878+0000","last_became_active":"2024-04-24T08:09:38.103230+0000","last_became_peered":"2024-04-24T08:09:38.103230+0000","last_unstale":"2024-04-24T08:09:40.119878+0000","last_undegraded":"2024-04-24T08:09:40.119878+0000","last_fullsized":"2024-04-24T08:09:40.119878+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119603+0000","last_change":"2024-04-24T08:09:39.106605+0000","last_active":"2024-04-24T08:09:40.119603+0000","last_peered":"2024-04-24T08:09:40.119603+0000","last_clean":"2024-04-24T08:09:40.119603+0000","last_became_active":"2024-04-24T08:09:39.105437+0000","last_became_peered":"2024-04-24T08:09:39.105437+0000","last_unstale":"2024-04-24T08:09:40.119603+0000","last_undegraded":"2024-04-24T08:09:40.119603+0000","last_fullsized":"2024-04-24T08:09:40.119603+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:42:30.829845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308573+0000","last_change":"2024-04-24T08:09:38.107831+0000","last_active":"2024-04-24T08:09:40.308573+0000","last_peered":"2024-04-24T08:09:40.308573+0000","last_clean":"2024-04-24T08:09:40.308573+0000","last_became_active":"2024-04-24T08:09:38.107509+0000","last_became_peered":"2024-04-24T08:09:38.107509+0000","last_unstale":"2024-04-24T08:09:40.308573+0000","last_undegraded":"2024-04-24T08:09:40.308573+0000","last_fullsized":"2024-04-24T08:09:40.308573+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:44.585295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310367+0000","last_change":"2024-04-24T08:09:39.126373+0000","last_active":"2024-04-24T08:09:40.310367+0000","last_peered":"2024-04-24T08:09:40.310367+0000","last_clean":"2024-04-24T08:09:40.310367+0000","last_became_active":"2024-04-24T08:09:39.125744+0000","last_became_peered":"2024-04-24T08:09:39.125744+0000","last_unstale":"2024-04-24T08:09:40.310367+0000","last_undegraded":"2024-04-24T08:09:40.310367+0000","last_fullsized":"2024-04-24T08:09:40.310367+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:48.329141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311083+0000","last_change":"2024-04-24T08:09:38.112443+0000","last_active":"2024-04-24T08:09:40.311083+0000","last_peered":"2024-04-24T08:09:40.311083+0000","last_clean":"2024-04-24T08:09:40.311083+0000","last_became_active":"2024-04-24T08:09:38.111901+0000","last_became_peered":"2024-04-24T08:09:38.111901+0000","last_unstale":"2024-04-24T08:09:40.311083+0000","last_undegraded":"2024-04-24T08:09:40.311083+0000","last_fullsized":"2024-04-24T08:09:40.311083+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":193,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309991+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:09:40.309991+0000","last_peered":"2024-04-24T08:09:40.309991+0000","last_clean":"2024-04-24T08:09:40.309991+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:09:40.309991+0000","last_undegraded":"2024-04-24T08:09:40.309991+0000","last_fullsized":"2024-04-24T08:09:40.309991+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":737228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":147442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":146840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":737138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:09:48.555 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:09:48.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.767+0000 7f1943200640 1 Processor -- start 2024-04-24T08:09:48.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.767+0000 7f1943200640 1 -- start start 2024-04-24T08:09:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.768+0000 7f1943200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f19440b30f0 0x7f1944167470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.768+0000 7f1943200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f194415d660 con 0x7f1944156c60 2024-04-24T08:09:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.768+0000 7f1943200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f194406dc30 con 0x7f194415d1d0 2024-04-24T08:09:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.768+0000 7f1943200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f194406df10 con 0x7f19440b30f0 2024-04-24T08:09:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.769+0000 7f1941e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1944156c60 0x7f194415cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53320/0 (socket says 172.21.15.69:53320) 2024-04-24T08:09:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.769+0000 7f1941e00640 1 -- 172.21.15.69:0/530062330 learned_addr learned my addr 172.21.15.69:0/530062330 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:48.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.769+0000 7f1942800640 1 --2- 172.21.15.69:0/530062330 >> v2:172.21.15.121:3300/0 conn(0x7f19440b30f0 0x7f1944167470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:48.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.769+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3368661110 0 0) 0x7f194415d660 con 0x7f1944156c60 2024-04-24T08:09:48.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.769+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1928003520 con 0x7f1944156c60 2024-04-24T08:09:48.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2875858846 0 0) 0x7f1928003520 con 0x7f1944156c60 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 >> v2:172.21.15.121:3300/0 conn(0x7f19440b30f0 msgr2=0x7f1944167470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 --2- 172.21.15.69:0/530062330 >> v2:172.21.15.121:3300/0 conn(0x7f19440b30f0 0x7f1944167470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 >> v1:172.21.15.121:6789/0 conn(0x7f194415d1d0 legacy=0x7f19440ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f194406e1f0 con 0x7f1944156c60 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f1934001670 con 0x7f1944156c60 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 >> v1:172.21.15.69:6789/0 conn(0x7f1944156c60 legacy=0x7f194415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19280039b0 0x7f1928003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1928004800 0x7f1928004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f192800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f194406df10 con 0x7f19280051c0 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f192800c060 con 0x7f19280039b0 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f192800c260 con 0x7f1928004800 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.770+0000 7f1942800640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f192800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:48.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.771+0000 7f1941400640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1928004800 0x7f1928004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.771+0000 7f1942800640 1 -- 172.21.15.69:0/530062330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1928004800 msgr2=0x7f1928004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.771+0000 7f1941e00640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19280039b0 0x7f1928003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.771+0000 7f1942800640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1928004800 0x7f1928004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.771+0000 7f1942800640 1 -- 172.21.15.69:0/530062330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19280039b0 msgr2=0x7f1928003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.771+0000 7f1942800640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19280039b0 0x7f1928003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.771+0000 7f1942800640 1 -- 172.21.15.69:0/530062330 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1928003520 con 0x7f19280051c0 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.771+0000 7f1941400640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1928004800 0x7f1928004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.771+0000 7f1941e00640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19280039b0 0x7f1928003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.772+0000 7f1942800640 1 -- 172.21.15.69:0/530062330 reap_dead start 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.772+0000 7f1942800640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f192800b9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f194406d590 tx=0x7f1928005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ecc928ad639e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.772+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c009c10 con 0x7f19280051c0 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.772+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f193c031410 con 0x7f19280051c0 2024-04-24T08:09:48.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.772+0000 7f1940a00640 1 -- 172.21.15.69:0/530062330 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f193c0317e0 con 0x7f19280051c0 2024-04-24T08:09:48.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.773+0000 7f1943200640 1 -- 172.21.15.69:0/530062330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 msgr2=0x7f192800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:48.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.773+0000 7f1943200640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f192800b9c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f194406d590 tx=0x7f1928005830 comp rx=0 tx=0).stop 2024-04-24T08:09:48.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.773+0000 7f1943200640 1 -- 172.21.15.69:0/530062330 shutdown_connections 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.773+0000 7f1943200640 1 --2- 172.21.15.69:0/530062330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f192800b9c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.773+0000 7f1943200640 1 -- 172.21.15.69:0/530062330 >> 172.21.15.69:0/530062330 conn(0x7f19440d0160 msgr2=0x7f19440d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.773+0000 7f1943200640 1 -- 172.21.15.69:0/530062330 shutdown_connections 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.773+0000 7f1943200640 1 -- 172.21.15.69:0/530062330 wait complete. 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.775+0000 7f1943200640 1 Processor -- start 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.775+0000 7f1943200640 1 -- start start 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1943200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f19441f5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1943200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1944143830 0x7f1944142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1943200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19441f43f0 0x7f19440b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1943200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f194415cac0 con 0x7f19280051c0 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1943200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f194415d940 con 0x7f1944143830 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1943200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f194415cc40 con 0x7f19441f43f0 2024-04-24T08:09:48.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1941e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f19441f5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1941e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f19441f5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60744/0 (socket says 172.21.15.69:60744) 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1941e00640 1 -- 172.21.15.69:0/2286265897 learned_addr learned my addr 172.21.15.69:0/2286265897 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1942800640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19441f43f0 0x7f19440b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1941400640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1944143830 0x7f1944142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1941e00640 1 -- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19441f43f0 msgr2=0x7f19440b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1941e00640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19441f43f0 0x7f19440b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1941e00640 1 -- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1944143830 msgr2=0x7f1944142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1941e00640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1944143830 0x7f1944142e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1941e00640 1 -- 172.21.15.69:0/2286265897 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1944143cb0 con 0x7f19280051c0 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1942800640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19441f43f0 0x7f19440b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.776+0000 7f1941400640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1944143830 0x7f1944142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.777+0000 7f1941e00640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f19441f5460 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f193400d8b0 tx=0x7f193400dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.777+0000 7f193a000640 1 -- 172.21.15.69:0/2286265897 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934019070 con 0x7f19280051c0 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.777+0000 7f193a000640 1 -- 172.21.15.69:0/2286265897 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19340074e0 con 0x7f19280051c0 2024-04-24T08:09:48.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.777+0000 7f193a000640 1 -- 172.21.15.69:0/2286265897 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934004a00 con 0x7f19280051c0 2024-04-24T08:09:48.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.777+0000 7f1943200640 1 -- 172.21.15.69:0/2286265897 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1944143f30 con 0x7f19280051c0 2024-04-24T08:09:48.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.777+0000 7f1943200640 1 -- 172.21.15.69:0/2286265897 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19441f4c60 con 0x7f19280051c0 2024-04-24T08:09:48.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.778+0000 7f1943200640 1 -- 172.21.15.69:0/2286265897 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1944075d50 con 0x7f19280051c0 2024-04-24T08:09:48.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.779+0000 7f193a000640 1 -- 172.21.15.69:0/2286265897 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1934004ba0 con 0x7f19280051c0 2024-04-24T08:09:48.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.781+0000 7f193a000640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192406c920 0x7f192406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:48.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.781+0000 7f193a000640 1 -- 172.21.15.69:0/2286265897 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f193408b630 con 0x7f19280051c0 2024-04-24T08:09:48.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.782+0000 7f1941400640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192406c920 0x7f192406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:48.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.783+0000 7f1941400640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192406c920 0x7f192406ede0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f194415c610 tx=0x7f1930009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:48.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.786+0000 7f193a000640 1 -- 172.21.15.69:0/2286265897 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1944143cb0 con 0x7f19280051c0 2024-04-24T08:09:48.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:48.985+0000 7f1943200640 1 -- 172.21.15.69:0/2286265897 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f194406f9e0 con 0x7f192406c920 2024-04-24T08:09:49.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.004+0000 7f193a000640 1 -- 172.21.15.69:0/2286265897 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251955 (secure 0 0 0) 0x7f194406f9e0 con 0x7f192406c920 2024-04-24T08:09:49.008 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:49.010 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:09:49.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.010+0000 7f1943200640 1 -- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192406c920 msgr2=0x7f192406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.010+0000 7f1943200640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192406c920 0x7f192406ede0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f194415c610 tx=0x7f1930009040 comp rx=0 tx=0).stop 2024-04-24T08:09:49.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.010+0000 7f1943200640 1 -- 172.21.15.69:0/2286265897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 msgr2=0x7f19441f5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.010+0000 7f1943200640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f19441f5460 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f193400d8b0 tx=0x7f193400dd70 comp rx=0 tx=0).stop 2024-04-24T08:09:49.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.011+0000 7f1943200640 1 -- 172.21.15.69:0/2286265897 shutdown_connections 2024-04-24T08:09:49.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.011+0000 7f1943200640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192406c920 0x7f192406ede0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.011+0000 7f1943200640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19441f43f0 0x7f19440b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.011+0000 7f1943200640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1944143830 0x7f1944142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.011+0000 7f1943200640 1 --2- 172.21.15.69:0/2286265897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19280051c0 0x7f19441f5460 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.011+0000 7f1943200640 1 -- 172.21.15.69:0/2286265897 >> 172.21.15.69:0/2286265897 conn(0x7f19440d0160 msgr2=0x7f19440aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:49.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.011+0000 7f1943200640 1 -- 172.21.15.69:0/2286265897 shutdown_connections 2024-04-24T08:09:49.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.011+0000 7f1943200640 1 -- 172.21.15.69:0/2286265897 wait complete. 2024-04-24T08:09:49.030 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-24T08:09:48.692532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220312,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754408,"statfs":{"total":773094113280,"available":772868513792,"internally_reserved":0,"allocated":4235264,"data_stored":2359716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.612168"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311306+0000","last_change":"2024-04-24T08:09:39.125219+0000","last_active":"2024-04-24T08:09:40.311306+0000","last_peered":"2024-04-24T08:09:40.311306+0000","last_clean":"2024-04-24T08:09:40.311306+0000","last_became_active":"2024-04-24T08:09:39.124763+0000","last_became_peered":"2024-04-24T08:09:39.124763+0000","last_unstale":"2024-04-24T08:09:40.311306+0000","last_undegraded":"2024-04-24T08:09:40.311306+0000","last_fullsized":"2024-04-24T08:09:40.311306+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:54.979294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310409+0000","last_change":"2024-04-24T08:09:39.103863+0000","last_active":"2024-04-24T08:09:40.310409+0000","last_peered":"2024-04-24T08:09:40.310409+0000","last_clean":"2024-04-24T08:09:40.310409+0000","last_became_active":"2024-04-24T08:09:39.103592+0000","last_became_peered":"2024-04-24T08:09:39.103592+0000","last_unstale":"2024-04-24T08:09:40.310409+0000","last_undegraded":"2024-04-24T08:09:40.310409+0000","last_fullsized":"2024-04-24T08:09:40.310409+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:19:54.217101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.116541+0000","last_change":"2024-04-24T08:09:39.111170+0000","last_active":"2024-04-24T08:09:40.116541+0000","last_peered":"2024-04-24T08:09:40.116541+0000","last_clean":"2024-04-24T08:09:40.116541+0000","last_became_active":"2024-04-24T08:09:39.110658+0000","last_became_peered":"2024-04-24T08:09:39.110658+0000","last_unstale":"2024-04-24T08:09:40.116541+0000","last_undegraded":"2024-04-24T08:09:40.116541+0000","last_fullsized":"2024-04-24T08:09:40.116541+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:16:48.624830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284665+0000","last_change":"2024-04-24T08:09:39.108422+0000","last_active":"2024-04-24T08:09:40.284665+0000","last_peered":"2024-04-24T08:09:40.284665+0000","last_clean":"2024-04-24T08:09:40.284665+0000","last_became_active":"2024-04-24T08:09:39.107895+0000","last_became_peered":"2024-04-24T08:09:39.107895+0000","last_unstale":"2024-04-24T08:09:40.284665+0000","last_undegraded":"2024-04-24T08:09:40.284665+0000","last_fullsized":"2024-04-24T08:09:40.284665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:17.553795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309432+0000","last_change":"2024-04-24T08:09:39.107555+0000","last_active":"2024-04-24T08:09:40.309432+0000","last_peered":"2024-04-24T08:09:40.309432+0000","last_clean":"2024-04-24T08:09:40.309432+0000","last_became_active":"2024-04-24T08:09:39.105910+0000","last_became_peered":"2024-04-24T08:09:39.105910+0000","last_unstale":"2024-04-24T08:09:40.309432+0000","last_undegraded":"2024-04-24T08:09:40.309432+0000","last_fullsized":"2024-04-24T08:09:40.309432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:13.439187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311055+0000","last_change":"2024-04-24T08:09:39.120619+0000","last_active":"2024-04-24T08:09:40.311055+0000","last_peered":"2024-04-24T08:09:40.311055+0000","last_clean":"2024-04-24T08:09:40.311055+0000","last_became_active":"2024-04-24T08:09:39.120309+0000","last_became_peered":"2024-04-24T08:09:39.120309+0000","last_unstale":"2024-04-24T08:09:40.311055+0000","last_undegraded":"2024-04-24T08:09:40.311055+0000","last_fullsized":"2024-04-24T08:09:40.311055+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:11:52.251677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309862+0000","last_change":"2024-04-24T08:09:39.125840+0000","last_active":"2024-04-24T08:09:40.309862+0000","last_peered":"2024-04-24T08:09:40.309862+0000","last_clean":"2024-04-24T08:09:40.309862+0000","last_became_active":"2024-04-24T08:09:39.125236+0000","last_became_peered":"2024-04-24T08:09:39.125236+0000","last_unstale":"2024-04-24T08:09:40.309862+0000","last_undegraded":"2024-04-24T08:09:40.309862+0000","last_fullsized":"2024-04-24T08:09:40.309862+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:16:51.790913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309221+0000","last_change":"2024-04-24T08:09:39.117265+0000","last_active":"2024-04-24T08:09:40.309221+0000","last_peered":"2024-04-24T08:09:40.309221+0000","last_clean":"2024-04-24T08:09:40.309221+0000","last_became_active":"2024-04-24T08:09:39.116973+0000","last_became_peered":"2024-04-24T08:09:39.116973+0000","last_unstale":"2024-04-24T08:09:40.309221+0000","last_undegraded":"2024-04-24T08:09:40.309221+0000","last_fullsized":"2024-04-24T08:09:40.309221+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:32.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.122413+0000","last_change":"2024-04-24T08:09:39.106586+0000","last_active":"2024-04-24T08:09:40.122413+0000","last_peered":"2024-04-24T08:09:40.122413+0000","last_clean":"2024-04-24T08:09:40.122413+0000","last_became_active":"2024-04-24T08:09:39.106096+0000","last_became_peered":"2024-04-24T08:09:39.106096+0000","last_unstale":"2024-04-24T08:09:40.122413+0000","last_undegraded":"2024-04-24T08:09:40.122413+0000","last_fullsized":"2024-04-24T08:09:40.122413+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:32.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285229+0000","last_change":"2024-04-24T08:09:39.117928+0000","last_active":"2024-04-24T08:09:40.285229+0000","last_peered":"2024-04-24T08:09:40.285229+0000","last_clean":"2024-04-24T08:09:40.285229+0000","last_became_active":"2024-04-24T08:09:39.117634+0000","last_became_peered":"2024-04-24T08:09:39.117634+0000","last_unstale":"2024-04-24T08:09:40.285229+0000","last_undegraded":"2024-04-24T08:09:40.285229+0000","last_fullsized":"2024-04-24T08:09:40.285229+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:45:39.110898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310529+0000","last_change":"2024-04-24T08:09:39.127475+0000","last_active":"2024-04-24T08:09:40.310529+0000","last_peered":"2024-04-24T08:09:40.310529+0000","last_clean":"2024-04-24T08:09:40.310529+0000","last_became_active":"2024-04-24T08:09:39.127153+0000","last_became_peered":"2024-04-24T08:09:39.127153+0000","last_unstale":"2024-04-24T08:09:40.310529+0000","last_undegraded":"2024-04-24T08:09:40.310529+0000","last_fullsized":"2024-04-24T08:09:40.310529+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:38:26.155101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.116166+0000","last_change":"2024-04-24T08:09:39.121787+0000","last_active":"2024-04-24T08:09:40.116166+0000","last_peered":"2024-04-24T08:09:40.116166+0000","last_clean":"2024-04-24T08:09:40.116166+0000","last_became_active":"2024-04-24T08:09:39.121504+0000","last_became_peered":"2024-04-24T08:09:39.121504+0000","last_unstale":"2024-04-24T08:09:40.116166+0000","last_undegraded":"2024-04-24T08:09:40.116166+0000","last_fullsized":"2024-04-24T08:09:40.116166+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:16:51.790913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309817+0000","last_change":"2024-04-24T08:09:39.101508+0000","last_active":"2024-04-24T08:09:40.309817+0000","last_peered":"2024-04-24T08:09:40.309817+0000","last_clean":"2024-04-24T08:09:40.309817+0000","last_became_active":"2024-04-24T08:09:39.101181+0000","last_became_peered":"2024-04-24T08:09:39.101181+0000","last_unstale":"2024-04-24T08:09:40.309817+0000","last_undegraded":"2024-04-24T08:09:40.309817+0000","last_fullsized":"2024-04-24T08:09:40.309817+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:53.588357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310992+0000","last_change":"2024-04-24T08:09:39.106541+0000","last_active":"2024-04-24T08:09:40.310992+0000","last_peered":"2024-04-24T08:09:40.310992+0000","last_clean":"2024-04-24T08:09:40.310992+0000","last_became_active":"2024-04-24T08:09:39.106096+0000","last_became_peered":"2024-04-24T08:09:39.106096+0000","last_unstale":"2024-04-24T08:09:40.310992+0000","last_undegraded":"2024-04-24T08:09:40.310992+0000","last_fullsized":"2024-04-24T08:09:40.310992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:00:29.600153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.115480+0000","last_change":"2024-04-24T08:09:39.118355+0000","last_active":"2024-04-24T08:09:40.115480+0000","last_peered":"2024-04-24T08:09:40.115480+0000","last_clean":"2024-04-24T08:09:40.115480+0000","last_became_active":"2024-04-24T08:09:39.118091+0000","last_became_peered":"2024-04-24T08:09:39.118091+0000","last_unstale":"2024-04-24T08:09:40.115480+0000","last_undegraded":"2024-04-24T08:09:40.115480+0000","last_fullsized":"2024-04-24T08:09:40.115480+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:45:39.110898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310097+0000","last_change":"2024-04-24T08:09:39.118631+0000","last_active":"2024-04-24T08:09:40.310097+0000","last_peered":"2024-04-24T08:09:40.310097+0000","last_clean":"2024-04-24T08:09:40.310097+0000","last_became_active":"2024-04-24T08:09:39.118352+0000","last_became_peered":"2024-04-24T08:09:39.118352+0000","last_unstale":"2024-04-24T08:09:40.310097+0000","last_undegraded":"2024-04-24T08:09:40.310097+0000","last_fullsized":"2024-04-24T08:09:40.310097+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:16:48.624830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311029+0000","last_change":"2024-04-24T08:09:39.120668+0000","last_active":"2024-04-24T08:09:40.311029+0000","last_peered":"2024-04-24T08:09:40.311029+0000","last_clean":"2024-04-24T08:09:40.311029+0000","last_became_active":"2024-04-24T08:09:39.119974+0000","last_became_peered":"2024-04-24T08:09:39.119974+0000","last_unstale":"2024-04-24T08:09:40.311029+0000","last_undegraded":"2024-04-24T08:09:40.311029+0000","last_fullsized":"2024-04-24T08:09:40.311029+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:23:17.124621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310462+0000","last_change":"2024-04-24T08:09:39.106863+0000","last_active":"2024-04-24T08:09:40.310462+0000","last_peered":"2024-04-24T08:09:40.310462+0000","last_clean":"2024-04-24T08:09:40.310462+0000","last_became_active":"2024-04-24T08:09:39.106186+0000","last_became_peered":"2024-04-24T08:09:39.106186+0000","last_unstale":"2024-04-24T08:09:40.310462+0000","last_undegraded":"2024-04-24T08:09:40.310462+0000","last_fullsized":"2024-04-24T08:09:40.310462+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:54:36.719416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310033+0000","last_change":"2024-04-24T08:09:39.119111+0000","last_active":"2024-04-24T08:09:40.310033+0000","last_peered":"2024-04-24T08:09:40.310033+0000","last_clean":"2024-04-24T08:09:40.310033+0000","last_became_active":"2024-04-24T08:09:39.115194+0000","last_became_peered":"2024-04-24T08:09:39.115194+0000","last_unstale":"2024-04-24T08:09:40.310033+0000","last_undegraded":"2024-04-24T08:09:40.310033+0000","last_fullsized":"2024-04-24T08:09:40.310033+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:23:17.124621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310549+0000","last_change":"2024-04-24T08:09:39.121570+0000","last_active":"2024-04-24T08:09:40.310549+0000","last_peered":"2024-04-24T08:09:40.310549+0000","last_clean":"2024-04-24T08:09:40.310549+0000","last_became_active":"2024-04-24T08:09:39.121118+0000","last_became_peered":"2024-04-24T08:09:39.121118+0000","last_unstale":"2024-04-24T08:09:40.310549+0000","last_undegraded":"2024-04-24T08:09:40.310549+0000","last_fullsized":"2024-04-24T08:09:40.310549+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:28.471092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121434+0000","last_change":"2024-04-24T08:09:39.107878+0000","last_active":"2024-04-24T08:09:40.121434+0000","last_peered":"2024-04-24T08:09:40.121434+0000","last_clean":"2024-04-24T08:09:40.121434+0000","last_became_active":"2024-04-24T08:09:39.106637+0000","last_became_peered":"2024-04-24T08:09:39.106637+0000","last_unstale":"2024-04-24T08:09:40.121434+0000","last_undegraded":"2024-04-24T08:09:40.121434+0000","last_fullsized":"2024-04-24T08:09:40.121434+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:58.147840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.120561+0000","last_change":"2024-04-24T08:09:39.110476+0000","last_active":"2024-04-24T08:09:40.120561+0000","last_peered":"2024-04-24T08:09:40.120561+0000","last_clean":"2024-04-24T08:09:40.120561+0000","last_became_active":"2024-04-24T08:09:39.110161+0000","last_became_peered":"2024-04-24T08:09:39.110161+0000","last_unstale":"2024-04-24T08:09:40.120561+0000","last_undegraded":"2024-04-24T08:09:40.120561+0000","last_fullsized":"2024-04-24T08:09:40.120561+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:47:51.470569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.111473+0000","last_change":"2024-04-24T08:09:39.105741+0000","last_active":"2024-04-24T08:09:40.111473+0000","last_peered":"2024-04-24T08:09:40.111473+0000","last_clean":"2024-04-24T08:09:40.111473+0000","last_became_active":"2024-04-24T08:09:39.105447+0000","last_became_peered":"2024-04-24T08:09:39.105447+0000","last_unstale":"2024-04-24T08:09:40.111473+0000","last_undegraded":"2024-04-24T08:09:40.111473+0000","last_fullsized":"2024-04-24T08:09:40.111473+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:54.979294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.115432+0000","last_change":"2024-04-24T08:09:39.110197+0000","last_active":"2024-04-24T08:09:40.115432+0000","last_peered":"2024-04-24T08:09:40.115432+0000","last_clean":"2024-04-24T08:09:40.115432+0000","last_became_active":"2024-04-24T08:09:39.109709+0000","last_became_peered":"2024-04-24T08:09:39.109709+0000","last_unstale":"2024-04-24T08:09:40.115432+0000","last_undegraded":"2024-04-24T08:09:40.115432+0000","last_fullsized":"2024-04-24T08:09:40.115432+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:32.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285470+0000","last_change":"2024-04-24T08:09:39.111278+0000","last_active":"2024-04-24T08:09:40.285470+0000","last_peered":"2024-04-24T08:09:40.285470+0000","last_clean":"2024-04-24T08:09:40.285470+0000","last_became_active":"2024-04-24T08:09:39.110843+0000","last_became_peered":"2024-04-24T08:09:39.110843+0000","last_unstale":"2024-04-24T08:09:40.285470+0000","last_undegraded":"2024-04-24T08:09:40.285470+0000","last_fullsized":"2024-04-24T08:09:40.285470+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:16:48.624830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310811+0000","last_change":"2024-04-24T08:09:39.116987+0000","last_active":"2024-04-24T08:09:40.310811+0000","last_peered":"2024-04-24T08:09:40.310811+0000","last_clean":"2024-04-24T08:09:40.310811+0000","last_became_active":"2024-04-24T08:09:39.116660+0000","last_became_peered":"2024-04-24T08:09:39.116660+0000","last_unstale":"2024-04-24T08:09:40.310811+0000","last_undegraded":"2024-04-24T08:09:40.310811+0000","last_fullsized":"2024-04-24T08:09:40.310811+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:45:10.083436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285401+0000","last_change":"2024-04-24T08:09:39.119204+0000","last_active":"2024-04-24T08:09:40.285401+0000","last_peered":"2024-04-24T08:09:40.285401+0000","last_clean":"2024-04-24T08:09:40.285401+0000","last_became_active":"2024-04-24T08:09:39.118896+0000","last_became_peered":"2024-04-24T08:09:39.118896+0000","last_unstale":"2024-04-24T08:09:40.285401+0000","last_undegraded":"2024-04-24T08:09:40.285401+0000","last_fullsized":"2024-04-24T08:09:40.285401+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:43:09.912013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.116302+0000","last_change":"2024-04-24T08:09:39.116617+0000","last_active":"2024-04-24T08:09:40.116302+0000","last_peered":"2024-04-24T08:09:40.116302+0000","last_clean":"2024-04-24T08:09:40.116302+0000","last_became_active":"2024-04-24T08:09:39.116298+0000","last_became_peered":"2024-04-24T08:09:39.116298+0000","last_unstale":"2024-04-24T08:09:40.116302+0000","last_undegraded":"2024-04-24T08:09:40.116302+0000","last_fullsized":"2024-04-24T08:09:40.116302+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:43:09.912013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119992+0000","last_change":"2024-04-24T08:09:39.109527+0000","last_active":"2024-04-24T08:09:40.119992+0000","last_peered":"2024-04-24T08:09:40.119992+0000","last_clean":"2024-04-24T08:09:40.119992+0000","last_became_active":"2024-04-24T08:09:39.109192+0000","last_became_peered":"2024-04-24T08:09:39.109192+0000","last_unstale":"2024-04-24T08:09:40.119992+0000","last_undegraded":"2024-04-24T08:09:40.119992+0000","last_fullsized":"2024-04-24T08:09:40.119992+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:37:08.649309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310110+0000","last_change":"2024-04-24T08:09:39.126432+0000","last_active":"2024-04-24T08:09:40.310110+0000","last_peered":"2024-04-24T08:09:40.310110+0000","last_clean":"2024-04-24T08:09:40.310110+0000","last_became_active":"2024-04-24T08:09:39.126157+0000","last_became_peered":"2024-04-24T08:09:39.126157+0000","last_unstale":"2024-04-24T08:09:40.310110+0000","last_undegraded":"2024-04-24T08:09:40.310110+0000","last_fullsized":"2024-04-24T08:09:40.310110+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:43:09.912013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309299+0000","last_change":"2024-04-24T08:09:39.111790+0000","last_active":"2024-04-24T08:09:40.309299+0000","last_peered":"2024-04-24T08:09:40.309299+0000","last_clean":"2024-04-24T08:09:40.309299+0000","last_became_active":"2024-04-24T08:09:39.110665+0000","last_became_peered":"2024-04-24T08:09:39.110665+0000","last_unstale":"2024-04-24T08:09:40.309299+0000","last_undegraded":"2024-04-24T08:09:40.309299+0000","last_fullsized":"2024-04-24T08:09:40.309299+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:34:51.226727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309336+0000","last_change":"2024-04-24T08:09:39.112112+0000","last_active":"2024-04-24T08:09:40.309336+0000","last_peered":"2024-04-24T08:09:40.309336+0000","last_clean":"2024-04-24T08:09:40.309336+0000","last_became_active":"2024-04-24T08:09:39.111791+0000","last_became_peered":"2024-04-24T08:09:39.111791+0000","last_unstale":"2024-04-24T08:09:40.309336+0000","last_undegraded":"2024-04-24T08:09:40.309336+0000","last_fullsized":"2024-04-24T08:09:40.309336+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:17.553795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112778+0000","last_change":"2024-04-24T08:09:39.106665+0000","last_active":"2024-04-24T08:09:40.112778+0000","last_peered":"2024-04-24T08:09:40.112778+0000","last_clean":"2024-04-24T08:09:40.112778+0000","last_became_active":"2024-04-24T08:09:39.105378+0000","last_became_peered":"2024-04-24T08:09:39.105378+0000","last_unstale":"2024-04-24T08:09:40.112778+0000","last_undegraded":"2024-04-24T08:09:40.112778+0000","last_fullsized":"2024-04-24T08:09:40.112778+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:58.147840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311474+0000","last_change":"2024-04-24T08:09:38.106509+0000","last_active":"2024-04-24T08:09:40.311474+0000","last_peered":"2024-04-24T08:09:40.311474+0000","last_clean":"2024-04-24T08:09:40.311474+0000","last_became_active":"2024-04-24T08:09:38.106216+0000","last_became_peered":"2024-04-24T08:09:38.106216+0000","last_unstale":"2024-04-24T08:09:40.311474+0000","last_undegraded":"2024-04-24T08:09:40.311474+0000","last_fullsized":"2024-04-24T08:09:40.311474+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311454+0000","last_change":"2024-04-24T08:09:39.108958+0000","last_active":"2024-04-24T08:09:40.311454+0000","last_peered":"2024-04-24T08:09:40.311454+0000","last_clean":"2024-04-24T08:09:40.311454+0000","last_became_active":"2024-04-24T08:09:39.107924+0000","last_became_peered":"2024-04-24T08:09:39.107924+0000","last_unstale":"2024-04-24T08:09:40.311454+0000","last_undegraded":"2024-04-24T08:09:40.311454+0000","last_fullsized":"2024-04-24T08:09:40.311454+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:37:08.649309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112677+0000","last_change":"2024-04-24T08:09:38.107316+0000","last_active":"2024-04-24T08:09:40.112677+0000","last_peered":"2024-04-24T08:09:40.112677+0000","last_clean":"2024-04-24T08:09:40.112677+0000","last_became_active":"2024-04-24T08:09:38.106891+0000","last_became_peered":"2024-04-24T08:09:38.106891+0000","last_unstale":"2024-04-24T08:09:40.112677+0000","last_undegraded":"2024-04-24T08:09:40.112677+0000","last_fullsized":"2024-04-24T08:09:40.112677+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121657+0000","last_change":"2024-04-24T08:09:39.115826+0000","last_active":"2024-04-24T08:09:40.121657+0000","last_peered":"2024-04-24T08:09:40.121657+0000","last_clean":"2024-04-24T08:09:40.121657+0000","last_became_active":"2024-04-24T08:09:39.115503+0000","last_became_peered":"2024-04-24T08:09:39.115503+0000","last_unstale":"2024-04-24T08:09:40.121657+0000","last_undegraded":"2024-04-24T08:09:40.121657+0000","last_fullsized":"2024-04-24T08:09:40.121657+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:55:27.468741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309859+0000","last_change":"2024-04-24T08:09:38.105975+0000","last_active":"2024-04-24T08:09:40.309859+0000","last_peered":"2024-04-24T08:09:40.309859+0000","last_clean":"2024-04-24T08:09:40.309859+0000","last_became_active":"2024-04-24T08:09:38.105684+0000","last_became_peered":"2024-04-24T08:09:38.105684+0000","last_unstale":"2024-04-24T08:09:40.309859+0000","last_undegraded":"2024-04-24T08:09:40.309859+0000","last_fullsized":"2024-04-24T08:09:40.309859+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:37:07.635670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285447+0000","last_change":"2024-04-24T08:09:39.119133+0000","last_active":"2024-04-24T08:09:40.285447+0000","last_peered":"2024-04-24T08:09:40.285447+0000","last_clean":"2024-04-24T08:09:40.285447+0000","last_became_active":"2024-04-24T08:09:39.118926+0000","last_became_peered":"2024-04-24T08:09:39.118926+0000","last_unstale":"2024-04-24T08:09:40.285447+0000","last_undegraded":"2024-04-24T08:09:40.285447+0000","last_fullsized":"2024-04-24T08:09:40.285447+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:16:51.790913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310800+0000","last_change":"2024-04-24T08:09:38.102564+0000","last_active":"2024-04-24T08:09:40.310800+0000","last_peered":"2024-04-24T08:09:40.310800+0000","last_clean":"2024-04-24T08:09:40.310800+0000","last_became_active":"2024-04-24T08:09:38.101598+0000","last_became_peered":"2024-04-24T08:09:38.101598+0000","last_unstale":"2024-04-24T08:09:40.310800+0000","last_undegraded":"2024-04-24T08:09:40.310800+0000","last_fullsized":"2024-04-24T08:09:40.310800+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:58:11.279574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310554+0000","last_change":"2024-04-24T08:09:39.121972+0000","last_active":"2024-04-24T08:09:40.310554+0000","last_peered":"2024-04-24T08:09:40.310554+0000","last_clean":"2024-04-24T08:09:40.310554+0000","last_became_active":"2024-04-24T08:09:39.120976+0000","last_became_peered":"2024-04-24T08:09:39.120976+0000","last_unstale":"2024-04-24T08:09:40.310554+0000","last_undegraded":"2024-04-24T08:09:40.310554+0000","last_fullsized":"2024-04-24T08:09:40.310554+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:53.687771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.120197+0000","last_change":"2024-04-24T08:09:38.101731+0000","last_active":"2024-04-24T08:09:40.120197+0000","last_peered":"2024-04-24T08:09:40.120197+0000","last_clean":"2024-04-24T08:09:40.120197+0000","last_became_active":"2024-04-24T08:09:38.100536+0000","last_became_peered":"2024-04-24T08:09:38.100536+0000","last_unstale":"2024-04-24T08:09:40.120197+0000","last_undegraded":"2024-04-24T08:09:40.120197+0000","last_fullsized":"2024-04-24T08:09:40.120197+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:05.837977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.115503+0000","last_change":"2024-04-24T08:09:39.122882+0000","last_active":"2024-04-24T08:09:40.115503+0000","last_peered":"2024-04-24T08:09:40.115503+0000","last_clean":"2024-04-24T08:09:40.115503+0000","last_became_active":"2024-04-24T08:09:39.122534+0000","last_became_peered":"2024-04-24T08:09:39.122534+0000","last_unstale":"2024-04-24T08:09:40.115503+0000","last_undegraded":"2024-04-24T08:09:40.115503+0000","last_fullsized":"2024-04-24T08:09:40.115503+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:06:14.738196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.120423+0000","last_change":"2024-04-24T08:09:38.108989+0000","last_active":"2024-04-24T08:09:40.120423+0000","last_peered":"2024-04-24T08:09:40.120423+0000","last_clean":"2024-04-24T08:09:40.120423+0000","last_became_active":"2024-04-24T08:09:38.108726+0000","last_became_peered":"2024-04-24T08:09:38.108726+0000","last_unstale":"2024-04-24T08:09:40.120423+0000","last_undegraded":"2024-04-24T08:09:40.120423+0000","last_fullsized":"2024-04-24T08:09:40.120423+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:42.105437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119781+0000","last_change":"2024-04-24T08:09:39.115338+0000","last_active":"2024-04-24T08:09:40.119781+0000","last_peered":"2024-04-24T08:09:40.119781+0000","last_clean":"2024-04-24T08:09:40.119781+0000","last_became_active":"2024-04-24T08:09:39.115065+0000","last_became_peered":"2024-04-24T08:09:39.115065+0000","last_unstale":"2024-04-24T08:09:40.119781+0000","last_undegraded":"2024-04-24T08:09:40.119781+0000","last_fullsized":"2024-04-24T08:09:40.119781+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:54.979294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308969+0000","last_change":"2024-04-24T08:09:38.093313+0000","last_active":"2024-04-24T08:09:40.308969+0000","last_peered":"2024-04-24T08:09:40.308969+0000","last_clean":"2024-04-24T08:09:40.308969+0000","last_became_active":"2024-04-24T08:09:38.092910+0000","last_became_peered":"2024-04-24T08:09:38.092910+0000","last_unstale":"2024-04-24T08:09:40.308969+0000","last_undegraded":"2024-04-24T08:09:40.308969+0000","last_fullsized":"2024-04-24T08:09:40.308969+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:42:29.816207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284451+0000","last_change":"2024-04-24T08:09:39.125735+0000","last_active":"2024-04-24T08:09:40.284451+0000","last_peered":"2024-04-24T08:09:40.284451+0000","last_clean":"2024-04-24T08:09:40.284451+0000","last_became_active":"2024-04-24T08:09:39.125449+0000","last_became_peered":"2024-04-24T08:09:39.125449+0000","last_unstale":"2024-04-24T08:09:40.284451+0000","last_undegraded":"2024-04-24T08:09:40.284451+0000","last_fullsized":"2024-04-24T08:09:40.284451+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:35:32.146851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119602+0000","last_change":"2024-04-24T08:09:38.107454+0000","last_active":"2024-04-24T08:09:40.119602+0000","last_peered":"2024-04-24T08:09:40.119602+0000","last_clean":"2024-04-24T08:09:40.119602+0000","last_became_active":"2024-04-24T08:09:38.107149+0000","last_became_peered":"2024-04-24T08:09:38.107149+0000","last_unstale":"2024-04-24T08:09:40.119602+0000","last_undegraded":"2024-04-24T08:09:40.119602+0000","last_fullsized":"2024-04-24T08:09:40.119602+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:56.637271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.117340+0000","last_change":"2024-04-24T08:09:39.121519+0000","last_active":"2024-04-24T08:09:40.117340+0000","last_peered":"2024-04-24T08:09:40.117340+0000","last_clean":"2024-04-24T08:09:40.117340+0000","last_became_active":"2024-04-24T08:09:39.120897+0000","last_became_peered":"2024-04-24T08:09:39.120897+0000","last_unstale":"2024-04-24T08:09:40.117340+0000","last_undegraded":"2024-04-24T08:09:40.117340+0000","last_fullsized":"2024-04-24T08:09:40.117340+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:48.329141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310742+0000","last_change":"2024-04-24T08:09:38.110451+0000","last_active":"2024-04-24T08:09:40.310742+0000","last_peered":"2024-04-24T08:09:40.310742+0000","last_clean":"2024-04-24T08:09:40.310742+0000","last_became_active":"2024-04-24T08:09:38.110191+0000","last_became_peered":"2024-04-24T08:09:38.110191+0000","last_unstale":"2024-04-24T08:09:40.310742+0000","last_undegraded":"2024-04-24T08:09:40.310742+0000","last_fullsized":"2024-04-24T08:09:40.310742+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:47:50.456931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310331+0000","last_change":"2024-04-24T08:09:39.116365+0000","last_active":"2024-04-24T08:09:40.310331+0000","last_peered":"2024-04-24T08:09:40.310331+0000","last_clean":"2024-04-24T08:09:40.310331+0000","last_became_active":"2024-04-24T08:09:39.115283+0000","last_became_peered":"2024-04-24T08:09:39.115283+0000","last_unstale":"2024-04-24T08:09:40.310331+0000","last_undegraded":"2024-04-24T08:09:40.310331+0000","last_fullsized":"2024-04-24T08:09:40.310331+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:26:44.551602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112782+0000","last_change":"2024-04-24T08:09:38.093711+0000","last_active":"2024-04-24T08:09:40.112782+0000","last_peered":"2024-04-24T08:09:40.112782+0000","last_clean":"2024-04-24T08:09:40.112782+0000","last_became_active":"2024-04-24T08:09:38.091626+0000","last_became_peered":"2024-04-24T08:09:38.091626+0000","last_unstale":"2024-04-24T08:09:40.112782+0000","last_undegraded":"2024-04-24T08:09:40.112782+0000","last_fullsized":"2024-04-24T08:09:40.112782+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:11.064191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309629+0000","last_change":"2024-04-24T08:09:39.111382+0000","last_active":"2024-04-24T08:09:40.309629+0000","last_peered":"2024-04-24T08:09:40.309629+0000","last_clean":"2024-04-24T08:09:40.309629+0000","last_became_active":"2024-04-24T08:09:39.108541+0000","last_became_peered":"2024-04-24T08:09:39.108541+0000","last_unstale":"2024-04-24T08:09:40.309629+0000","last_undegraded":"2024-04-24T08:09:40.309629+0000","last_fullsized":"2024-04-24T08:09:40.309629+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:38:26.155101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.116469+0000","last_change":"2024-04-24T08:09:38.107829+0000","last_active":"2024-04-24T08:09:40.116469+0000","last_peered":"2024-04-24T08:09:40.116469+0000","last_clean":"2024-04-24T08:09:40.116469+0000","last_became_active":"2024-04-24T08:09:38.107551+0000","last_became_peered":"2024-04-24T08:09:38.107551+0000","last_unstale":"2024-04-24T08:09:40.116469+0000","last_undegraded":"2024-04-24T08:09:40.116469+0000","last_fullsized":"2024-04-24T08:09:40.116469+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:10.779345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.115482+0000","last_change":"2024-04-24T08:09:39.111833+0000","last_active":"2024-04-24T08:09:40.115482+0000","last_peered":"2024-04-24T08:09:40.115482+0000","last_clean":"2024-04-24T08:09:40.115482+0000","last_became_active":"2024-04-24T08:09:39.111444+0000","last_became_peered":"2024-04-24T08:09:39.111444+0000","last_unstale":"2024-04-24T08:09:40.115482+0000","last_undegraded":"2024-04-24T08:09:40.115482+0000","last_fullsized":"2024-04-24T08:09:40.115482+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:17.553795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309727+0000","last_change":"2024-04-24T08:09:38.106294+0000","last_active":"2024-04-24T08:09:40.309727+0000","last_peered":"2024-04-24T08:09:40.309727+0000","last_clean":"2024-04-24T08:09:40.309727+0000","last_became_active":"2024-04-24T08:09:38.105994+0000","last_became_peered":"2024-04-24T08:09:38.105994+0000","last_unstale":"2024-04-24T08:09:40.309727+0000","last_undegraded":"2024-04-24T08:09:40.309727+0000","last_fullsized":"2024-04-24T08:09:40.309727+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:53.965656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284266+0000","last_change":"2024-04-24T08:09:39.121621+0000","last_active":"2024-04-24T08:09:40.284266+0000","last_peered":"2024-04-24T08:09:40.284266+0000","last_clean":"2024-04-24T08:09:40.284266+0000","last_became_active":"2024-04-24T08:09:39.121289+0000","last_became_peered":"2024-04-24T08:09:39.121289+0000","last_unstale":"2024-04-24T08:09:40.284266+0000","last_undegraded":"2024-04-24T08:09:40.284266+0000","last_fullsized":"2024-04-24T08:09:40.284266+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:43:19.492962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121197+0000","last_change":"2024-04-24T08:09:38.105998+0000","last_active":"2024-04-24T08:09:40.121197+0000","last_peered":"2024-04-24T08:09:40.121197+0000","last_clean":"2024-04-24T08:09:40.121197+0000","last_became_active":"2024-04-24T08:09:38.105710+0000","last_became_peered":"2024-04-24T08:09:38.105710+0000","last_unstale":"2024-04-24T08:09:40.121197+0000","last_undegraded":"2024-04-24T08:09:40.121197+0000","last_fullsized":"2024-04-24T08:09:40.121197+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:55.438998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309926+0000","last_change":"2024-04-24T08:09:39.104007+0000","last_active":"2024-04-24T08:09:40.309926+0000","last_peered":"2024-04-24T08:09:40.309926+0000","last_clean":"2024-04-24T08:09:40.309926+0000","last_became_active":"2024-04-24T08:09:39.103739+0000","last_became_peered":"2024-04-24T08:09:39.103739+0000","last_unstale":"2024-04-24T08:09:40.309926+0000","last_undegraded":"2024-04-24T08:09:40.309926+0000","last_fullsized":"2024-04-24T08:09:40.309926+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:53.687771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284378+0000","last_change":"2024-04-24T08:09:38.107964+0000","last_active":"2024-04-24T08:09:40.284378+0000","last_peered":"2024-04-24T08:09:40.284378+0000","last_clean":"2024-04-24T08:09:40.284378+0000","last_became_active":"2024-04-24T08:09:38.107620+0000","last_became_peered":"2024-04-24T08:09:38.107620+0000","last_unstale":"2024-04-24T08:09:40.284378+0000","last_undegraded":"2024-04-24T08:09:40.284378+0000","last_fullsized":"2024-04-24T08:09:40.284378+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309665+0000","last_change":"2024-04-24T08:09:39.118278+0000","last_active":"2024-04-24T08:09:40.309665+0000","last_peered":"2024-04-24T08:09:40.309665+0000","last_clean":"2024-04-24T08:09:40.309665+0000","last_became_active":"2024-04-24T08:09:39.117897+0000","last_became_peered":"2024-04-24T08:09:39.117897+0000","last_unstale":"2024-04-24T08:09:40.309665+0000","last_undegraded":"2024-04-24T08:09:40.309665+0000","last_fullsized":"2024-04-24T08:09:40.309665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:45:39.110898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308902+0000","last_change":"2024-04-24T08:09:38.103294+0000","last_active":"2024-04-24T08:09:40.308902+0000","last_peered":"2024-04-24T08:09:40.308902+0000","last_clean":"2024-04-24T08:09:40.308902+0000","last_became_active":"2024-04-24T08:09:38.101762+0000","last_became_peered":"2024-04-24T08:09:38.101762+0000","last_unstale":"2024-04-24T08:09:40.308902+0000","last_undegraded":"2024-04-24T08:09:40.308902+0000","last_fullsized":"2024-04-24T08:09:40.308902+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:57.918072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309463+0000","last_change":"2024-04-24T08:09:39.109911+0000","last_active":"2024-04-24T08:09:40.309463+0000","last_peered":"2024-04-24T08:09:40.309463+0000","last_clean":"2024-04-24T08:09:40.309463+0000","last_became_active":"2024-04-24T08:09:39.106934+0000","last_became_peered":"2024-04-24T08:09:39.106934+0000","last_unstale":"2024-04-24T08:09:40.309463+0000","last_undegraded":"2024-04-24T08:09:40.309463+0000","last_fullsized":"2024-04-24T08:09:40.309463+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:46:58.931710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308682+0000","last_change":"2024-04-24T08:09:38.100426+0000","last_active":"2024-04-24T08:09:40.308682+0000","last_peered":"2024-04-24T08:09:40.308682+0000","last_clean":"2024-04-24T08:09:40.308682+0000","last_became_active":"2024-04-24T08:09:38.100138+0000","last_became_peered":"2024-04-24T08:09:38.100138+0000","last_unstale":"2024-04-24T08:09:40.308682+0000","last_undegraded":"2024-04-24T08:09:40.308682+0000","last_fullsized":"2024-04-24T08:09:40.308682+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:34:50.213088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.122004+0000","last_change":"2024-04-24T08:09:39.122438+0000","last_active":"2024-04-24T08:09:40.122004+0000","last_peered":"2024-04-24T08:09:40.122004+0000","last_clean":"2024-04-24T08:09:40.122004+0000","last_became_active":"2024-04-24T08:09:39.121827+0000","last_became_peered":"2024-04-24T08:09:39.121827+0000","last_unstale":"2024-04-24T08:09:40.122004+0000","last_undegraded":"2024-04-24T08:09:40.122004+0000","last_fullsized":"2024-04-24T08:09:40.122004+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:43:19.492962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285624+0000","last_change":"2024-04-24T08:09:38.096536+0000","last_active":"2024-04-24T08:09:40.285624+0000","last_peered":"2024-04-24T08:09:40.285624+0000","last_clean":"2024-04-24T08:09:40.285624+0000","last_became_active":"2024-04-24T08:09:38.095915+0000","last_became_peered":"2024-04-24T08:09:38.095915+0000","last_unstale":"2024-04-24T08:09:40.285624+0000","last_undegraded":"2024-04-24T08:09:40.285624+0000","last_fullsized":"2024-04-24T08:09:40.285624+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:10.779345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.117569+0000","last_change":"2024-04-24T08:09:39.120695+0000","last_active":"2024-04-24T08:09:40.117569+0000","last_peered":"2024-04-24T08:09:40.117569+0000","last_clean":"2024-04-24T08:09:40.117569+0000","last_became_active":"2024-04-24T08:09:39.115473+0000","last_became_peered":"2024-04-24T08:09:39.115473+0000","last_unstale":"2024-04-24T08:09:40.117569+0000","last_undegraded":"2024-04-24T08:09:40.117569+0000","last_fullsized":"2024-04-24T08:09:40.117569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:38:26.155101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121882+0000","last_change":"2024-04-24T08:09:38.101152+0000","last_active":"2024-04-24T08:09:40.121882+0000","last_peered":"2024-04-24T08:09:40.121882+0000","last_clean":"2024-04-24T08:09:40.121882+0000","last_became_active":"2024-04-24T08:09:38.099292+0000","last_became_peered":"2024-04-24T08:09:38.099292+0000","last_unstale":"2024-04-24T08:09:40.121882+0000","last_undegraded":"2024-04-24T08:09:40.121882+0000","last_fullsized":"2024-04-24T08:09:40.121882+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:50:12.425548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284867+0000","last_change":"2024-04-24T08:09:39.108177+0000","last_active":"2024-04-24T08:09:40.284867+0000","last_peered":"2024-04-24T08:09:40.284867+0000","last_clean":"2024-04-24T08:09:40.284867+0000","last_became_active":"2024-04-24T08:09:39.107524+0000","last_became_peered":"2024-04-24T08:09:39.107524+0000","last_unstale":"2024-04-24T08:09:40.284867+0000","last_undegraded":"2024-04-24T08:09:40.284867+0000","last_fullsized":"2024-04-24T08:09:40.284867+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:06:14.738196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121973+0000","last_change":"2024-04-24T08:09:38.100668+0000","last_active":"2024-04-24T08:09:40.121973+0000","last_peered":"2024-04-24T08:09:40.121973+0000","last_clean":"2024-04-24T08:09:40.121973+0000","last_became_active":"2024-04-24T08:09:38.100303+0000","last_became_peered":"2024-04-24T08:09:38.100303+0000","last_unstale":"2024-04-24T08:09:40.121973+0000","last_undegraded":"2024-04-24T08:09:40.121973+0000","last_fullsized":"2024-04-24T08:09:40.121973+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:57:33.580704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311324+0000","last_change":"2024-04-24T08:09:39.109433+0000","last_active":"2024-04-24T08:09:40.311324+0000","last_peered":"2024-04-24T08:09:40.311324+0000","last_clean":"2024-04-24T08:09:40.311324+0000","last_became_active":"2024-04-24T08:09:39.108287+0000","last_became_peered":"2024-04-24T08:09:39.108287+0000","last_unstale":"2024-04-24T08:09:40.311324+0000","last_undegraded":"2024-04-24T08:09:40.311324+0000","last_fullsized":"2024-04-24T08:09:40.311324+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:22.010195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.121571+0000","last_change":"2024-04-24T08:09:38.105765+0000","last_active":"2024-04-24T08:09:40.121571+0000","last_peered":"2024-04-24T08:09:40.121571+0000","last_clean":"2024-04-24T08:09:40.121571+0000","last_became_active":"2024-04-24T08:09:38.100242+0000","last_became_peered":"2024-04-24T08:09:38.100242+0000","last_unstale":"2024-04-24T08:09:40.121571+0000","last_undegraded":"2024-04-24T08:09:40.121571+0000","last_fullsized":"2024-04-24T08:09:40.121571+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309041+0000","last_change":"2024-04-24T08:09:39.099317+0000","last_active":"2024-04-24T08:09:40.309041+0000","last_peered":"2024-04-24T08:09:40.309041+0000","last_clean":"2024-04-24T08:09:40.309041+0000","last_became_active":"2024-04-24T08:09:39.099056+0000","last_became_peered":"2024-04-24T08:09:39.099056+0000","last_unstale":"2024-04-24T08:09:40.309041+0000","last_undegraded":"2024-04-24T08:09:40.309041+0000","last_fullsized":"2024-04-24T08:09:40.309041+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:03.587948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112026+0000","last_change":"2024-04-24T08:09:38.097858+0000","last_active":"2024-04-24T08:09:40.112026+0000","last_peered":"2024-04-24T08:09:40.112026+0000","last_clean":"2024-04-24T08:09:40.112026+0000","last_became_active":"2024-04-24T08:09:38.097599+0000","last_became_peered":"2024-04-24T08:09:38.097599+0000","last_unstale":"2024-04-24T08:09:40.112026+0000","last_undegraded":"2024-04-24T08:09:40.112026+0000","last_fullsized":"2024-04-24T08:09:40.112026+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:10.779345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310172+0000","last_change":"2024-04-24T08:09:39.103765+0000","last_active":"2024-04-24T08:09:40.310172+0000","last_peered":"2024-04-24T08:09:40.310172+0000","last_clean":"2024-04-24T08:09:40.310172+0000","last_became_active":"2024-04-24T08:09:39.103468+0000","last_became_peered":"2024-04-24T08:09:39.103468+0000","last_unstale":"2024-04-24T08:09:40.310172+0000","last_undegraded":"2024-04-24T08:09:40.310172+0000","last_fullsized":"2024-04-24T08:09:40.310172+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:12.077829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310934+0000","last_change":"2024-04-24T08:09:38.110759+0000","last_active":"2024-04-24T08:09:40.310934+0000","last_peered":"2024-04-24T08:09:40.310934+0000","last_clean":"2024-04-24T08:09:40.310934+0000","last_became_active":"2024-04-24T08:09:38.110393+0000","last_became_peered":"2024-04-24T08:09:38.110393+0000","last_unstale":"2024-04-24T08:09:40.310934+0000","last_undegraded":"2024-04-24T08:09:40.310934+0000","last_fullsized":"2024-04-24T08:09:40.310934+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:26:52.574718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308561+0000","last_change":"2024-04-24T08:09:39.101642+0000","last_active":"2024-04-24T08:09:40.308561+0000","last_peered":"2024-04-24T08:09:40.308561+0000","last_clean":"2024-04-24T08:09:40.308561+0000","last_became_active":"2024-04-24T08:09:39.101382+0000","last_became_peered":"2024-04-24T08:09:39.101382+0000","last_unstale":"2024-04-24T08:09:40.308561+0000","last_undegraded":"2024-04-24T08:09:40.308561+0000","last_fullsized":"2024-04-24T08:09:40.308561+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:57.650910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119632+0000","last_change":"2024-04-24T08:09:38.104959+0000","last_active":"2024-04-24T08:09:40.119632+0000","last_peered":"2024-04-24T08:09:40.119632+0000","last_clean":"2024-04-24T08:09:40.119632+0000","last_became_active":"2024-04-24T08:09:38.104606+0000","last_became_peered":"2024-04-24T08:09:38.104606+0000","last_unstale":"2024-04-24T08:09:40.119632+0000","last_undegraded":"2024-04-24T08:09:40.119632+0000","last_fullsized":"2024-04-24T08:09:40.119632+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:04:53.285545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309446+0000","last_change":"2024-04-24T08:09:39.127115+0000","last_active":"2024-04-24T08:09:40.309446+0000","last_peered":"2024-04-24T08:09:40.309446+0000","last_clean":"2024-04-24T08:09:40.309446+0000","last_became_active":"2024-04-24T08:09:39.125800+0000","last_became_peered":"2024-04-24T08:09:39.125800+0000","last_unstale":"2024-04-24T08:09:40.309446+0000","last_undegraded":"2024-04-24T08:09:40.309446+0000","last_fullsized":"2024-04-24T08:09:40.309446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:06:14.738196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.114996+0000","last_change":"2024-04-24T08:09:38.104201+0000","last_active":"2024-04-24T08:09:40.114996+0000","last_peered":"2024-04-24T08:09:40.114996+0000","last_clean":"2024-04-24T08:09:40.114996+0000","last_became_active":"2024-04-24T08:09:38.103332+0000","last_became_peered":"2024-04-24T08:09:38.103332+0000","last_unstale":"2024-04-24T08:09:40.114996+0000","last_undegraded":"2024-04-24T08:09:40.114996+0000","last_fullsized":"2024-04-24T08:09:40.114996+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:27.457454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310983+0000","last_change":"2024-04-24T08:09:39.116271+0000","last_active":"2024-04-24T08:09:40.310983+0000","last_peered":"2024-04-24T08:09:40.310983+0000","last_clean":"2024-04-24T08:09:40.310983+0000","last_became_active":"2024-04-24T08:09:39.116002+0000","last_became_peered":"2024-04-24T08:09:39.116002+0000","last_unstale":"2024-04-24T08:09:40.310983+0000","last_undegraded":"2024-04-24T08:09:40.310983+0000","last_fullsized":"2024-04-24T08:09:40.310983+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:46:11.792983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.285653+0000","last_change":"2024-04-24T08:09:38.105102+0000","last_active":"2024-04-24T08:09:40.285653+0000","last_peered":"2024-04-24T08:09:40.285653+0000","last_clean":"2024-04-24T08:09:40.285653+0000","last_became_active":"2024-04-24T08:09:38.104814+0000","last_became_peered":"2024-04-24T08:09:38.104814+0000","last_unstale":"2024-04-24T08:09:40.285653+0000","last_undegraded":"2024-04-24T08:09:40.285653+0000","last_fullsized":"2024-04-24T08:09:40.285653+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:20.996557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.117636+0000","last_change":"2024-04-24T08:09:39.115185+0000","last_active":"2024-04-24T08:09:40.117636+0000","last_peered":"2024-04-24T08:09:40.117636+0000","last_clean":"2024-04-24T08:09:40.117636+0000","last_became_active":"2024-04-24T08:09:39.114901+0000","last_became_peered":"2024-04-24T08:09:39.114901+0000","last_unstale":"2024-04-24T08:09:40.117636+0000","last_undegraded":"2024-04-24T08:09:40.117636+0000","last_fullsized":"2024-04-24T08:09:40.117636+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:43:53.687771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112929+0000","last_change":"2024-04-24T08:09:38.102570+0000","last_active":"2024-04-24T08:09:40.112929+0000","last_peered":"2024-04-24T08:09:40.112929+0000","last_clean":"2024-04-24T08:09:40.112929+0000","last_became_active":"2024-04-24T08:09:38.101279+0000","last_became_peered":"2024-04-24T08:09:38.101279+0000","last_unstale":"2024-04-24T08:09:40.112929+0000","last_undegraded":"2024-04-24T08:09:40.112929+0000","last_fullsized":"2024-04-24T08:09:40.112929+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:27.457454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112274+0000","last_change":"2024-04-24T08:09:39.103942+0000","last_active":"2024-04-24T08:09:40.112274+0000","last_peered":"2024-04-24T08:09:40.112274+0000","last_clean":"2024-04-24T08:09:40.112274+0000","last_became_active":"2024-04-24T08:09:39.103686+0000","last_became_peered":"2024-04-24T08:09:39.103686+0000","last_unstale":"2024-04-24T08:09:40.112274+0000","last_undegraded":"2024-04-24T08:09:40.112274+0000","last_fullsized":"2024-04-24T08:09:40.112274+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:47:51.470569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284982+0000","last_change":"2024-04-24T08:09:38.101682+0000","last_active":"2024-04-24T08:09:40.284982+0000","last_peered":"2024-04-24T08:09:40.284982+0000","last_clean":"2024-04-24T08:09:40.284982+0000","last_became_active":"2024-04-24T08:09:38.099204+0000","last_became_peered":"2024-04-24T08:09:38.099204+0000","last_unstale":"2024-04-24T08:09:40.284982+0000","last_undegraded":"2024-04-24T08:09:40.284982+0000","last_fullsized":"2024-04-24T08:09:40.284982+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:40:27.457454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310932+0000","last_change":"2024-04-24T08:09:39.114705+0000","last_active":"2024-04-24T08:09:40.310932+0000","last_peered":"2024-04-24T08:09:40.310932+0000","last_clean":"2024-04-24T08:09:40.310932+0000","last_became_active":"2024-04-24T08:09:39.114454+0000","last_became_peered":"2024-04-24T08:09:39.114454+0000","last_unstale":"2024-04-24T08:09:40.310932+0000","last_undegraded":"2024-04-24T08:09:40.310932+0000","last_fullsized":"2024-04-24T08:09:40.310932+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:42:30.829845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.112199+0000","last_change":"2024-04-24T08:09:38.101668+0000","last_active":"2024-04-24T08:09:40.112199+0000","last_peered":"2024-04-24T08:09:40.112199+0000","last_clean":"2024-04-24T08:09:40.112199+0000","last_became_active":"2024-04-24T08:09:38.101318+0000","last_became_peered":"2024-04-24T08:09:38.101318+0000","last_unstale":"2024-04-24T08:09:40.112199+0000","last_undegraded":"2024-04-24T08:09:40.112199+0000","last_fullsized":"2024-04-24T08:09:40.112199+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:20.996557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.111985+0000","last_change":"2024-04-24T08:09:39.112842+0000","last_active":"2024-04-24T08:09:40.111985+0000","last_peered":"2024-04-24T08:09:40.111985+0000","last_clean":"2024-04-24T08:09:40.111985+0000","last_became_active":"2024-04-24T08:09:39.112556+0000","last_became_peered":"2024-04-24T08:09:39.112556+0000","last_unstale":"2024-04-24T08:09:40.111985+0000","last_undegraded":"2024-04-24T08:09:40.111985+0000","last_fullsized":"2024-04-24T08:09:40.111985+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:37:08.649309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.284545+0000","last_change":"2024-04-24T08:09:38.105797+0000","last_active":"2024-04-24T08:09:40.284545+0000","last_peered":"2024-04-24T08:09:40.284545+0000","last_clean":"2024-04-24T08:09:40.284545+0000","last_became_active":"2024-04-24T08:09:38.104048+0000","last_became_peered":"2024-04-24T08:09:38.104048+0000","last_unstale":"2024-04-24T08:09:40.284545+0000","last_undegraded":"2024-04-24T08:09:40.284545+0000","last_fullsized":"2024-04-24T08:09:40.284545+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:27:11.064191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308808+0000","last_change":"2024-04-24T08:09:39.113158+0000","last_active":"2024-04-24T08:09:40.308808+0000","last_peered":"2024-04-24T08:09:40.308808+0000","last_clean":"2024-04-24T08:09:40.308808+0000","last_became_active":"2024-04-24T08:09:39.112663+0000","last_became_peered":"2024-04-24T08:09:39.112663+0000","last_unstale":"2024-04-24T08:09:40.308808+0000","last_undegraded":"2024-04-24T08:09:40.308808+0000","last_fullsized":"2024-04-24T08:09:40.308808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:48.329141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119878+0000","last_change":"2024-04-24T08:09:38.103639+0000","last_active":"2024-04-24T08:09:40.119878+0000","last_peered":"2024-04-24T08:09:40.119878+0000","last_clean":"2024-04-24T08:09:40.119878+0000","last_became_active":"2024-04-24T08:09:38.103230+0000","last_became_peered":"2024-04-24T08:09:38.103230+0000","last_unstale":"2024-04-24T08:09:40.119878+0000","last_undegraded":"2024-04-24T08:09:40.119878+0000","last_fullsized":"2024-04-24T08:09:40.119878+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:37.571787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.119603+0000","last_change":"2024-04-24T08:09:39.106605+0000","last_active":"2024-04-24T08:09:40.119603+0000","last_peered":"2024-04-24T08:09:40.119603+0000","last_clean":"2024-04-24T08:09:40.119603+0000","last_became_active":"2024-04-24T08:09:39.105437+0000","last_became_peered":"2024-04-24T08:09:39.105437+0000","last_unstale":"2024-04-24T08:09:40.119603+0000","last_undegraded":"2024-04-24T08:09:40.119603+0000","last_fullsized":"2024-04-24T08:09:40.119603+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:42:30.829845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.308573+0000","last_change":"2024-04-24T08:09:38.107831+0000","last_active":"2024-04-24T08:09:40.308573+0000","last_peered":"2024-04-24T08:09:40.308573+0000","last_clean":"2024-04-24T08:09:40.308573+0000","last_became_active":"2024-04-24T08:09:38.107509+0000","last_became_peered":"2024-04-24T08:09:38.107509+0000","last_unstale":"2024-04-24T08:09:40.308573+0000","last_undegraded":"2024-04-24T08:09:40.308573+0000","last_fullsized":"2024-04-24T08:09:40.308573+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:44.585295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.310367+0000","last_change":"2024-04-24T08:09:39.126373+0000","last_active":"2024-04-24T08:09:40.310367+0000","last_peered":"2024-04-24T08:09:40.310367+0000","last_clean":"2024-04-24T08:09:40.310367+0000","last_became_active":"2024-04-24T08:09:39.125744+0000","last_became_peered":"2024-04-24T08:09:39.125744+0000","last_unstale":"2024-04-24T08:09:40.310367+0000","last_undegraded":"2024-04-24T08:09:40.310367+0000","last_fullsized":"2024-04-24T08:09:40.310367+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:38.056115+0000","last_clean_scrub_stamp":"2024-04-24T08:09:38.056115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:04:48.329141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.311083+0000","last_change":"2024-04-24T08:09:38.112443+0000","last_active":"2024-04-24T08:09:40.311083+0000","last_peered":"2024-04-24T08:09:40.311083+0000","last_clean":"2024-04-24T08:09:40.311083+0000","last_became_active":"2024-04-24T08:09:38.111901+0000","last_became_peered":"2024-04-24T08:09:38.111901+0000","last_unstale":"2024-04-24T08:09:40.311083+0000","last_undegraded":"2024-04-24T08:09:40.311083+0000","last_fullsized":"2024-04-24T08:09:40.311083+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:09:37.042476+0000","last_clean_scrub_stamp":"2024-04-24T08:09:37.042476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:28:02.574310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":193,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-24T08:09:40.309991+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:09:40.309991+0000","last_peered":"2024-04-24T08:09:40.309991+0000","last_clean":"2024-04-24T08:09:40.309991+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:09:40.309991+0000","last_undegraded":"2024-04-24T08:09:40.309991+0000","last_fullsized":"2024-04-24T08:09:40.309991+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":970752,"data_stored":737228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":147442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":146840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":966656,"data_stored":737138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:09:49.034 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T08:09:49.035 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-24T08:09:49.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.284+0000 7fc67f400640 1 Processor -- start 2024-04-24T08:09:49.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.284+0000 7fc67f400640 1 -- start start 2024-04-24T08:09:49.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.285+0000 7fc67f400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc678145a20 0x7fc678145eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.285+0000 7fc67f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6780d0300 0x7fc6780d0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.285+0000 7fc67f400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6780cf610 con 0x7fc6780cd8e0 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.285+0000 7fc67f400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc6780d0c90 con 0x7fc6780d0300 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.285+0000 7fc67f400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc6780cf8f0 con 0x7fc678145a20 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.286+0000 7fc67ca00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc6780cd8e0 0x7fc67806e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44660/0 (socket says 172.21.15.69:44660) 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.286+0000 7fc67d400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6780d0300 0x7fc6780d0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.286+0000 7fc67ca00640 1 -- 172.21.15.69:0/3980078195 learned_addr learned my addr 172.21.15.69:0/3980078195 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.286+0000 7fc67d400640 1 -- 172.21.15.69:0/3980078195 >> v2:172.21.15.121:3300/0 conn(0x7fc678145a20 msgr2=0x7fc678145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.286+0000 7fc67d400640 1 --2- 172.21.15.69:0/3980078195 >> v2:172.21.15.121:3300/0 conn(0x7fc678145a20 0x7fc678145eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.286+0000 7fc67d400640 1 -- 172.21.15.69:0/3980078195 >> v1:172.21.15.69:6789/0 conn(0x7fc6780cd8e0 legacy=0x7fc67806e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.286+0000 7fc67d400640 1 -- 172.21.15.69:0/3980078195 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6780cfbd0 con 0x7fc6780d0300 2024-04-24T08:09:49.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.286+0000 7fc677400640 1 -- 172.21.15.69:0/3980078195 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744985996 0 0) 0x7fc6780cf610 con 0x7fc6780cd8e0 2024-04-24T08:09:49.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.287+0000 7fc67d400640 1 --2- 172.21.15.69:0/3980078195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 0x7fc6780d0750 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc67806d660 tx=0x7fc67002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df48cf0acd12594 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:49.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.288+0000 7fc677400640 1 -- 172.21.15.69:0/3980078195 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670007740 con 0x7fc6780d0300 2024-04-24T08:09:49.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.288+0000 7fc677400640 1 -- 172.21.15.69:0/3980078195 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6700078e0 con 0x7fc6780d0300 2024-04-24T08:09:49.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.288+0000 7fc677400640 1 -- 172.21.15.69:0/3980078195 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670007cb0 con 0x7fc6780d0300 2024-04-24T08:09:49.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.288+0000 7fc67f400640 1 -- 172.21.15.69:0/3980078195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 msgr2=0x7fc6780d0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.288+0000 7fc67f400640 1 --2- 172.21.15.69:0/3980078195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 0x7fc6780d0750 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc67806d660 tx=0x7fc67002a680 comp rx=0 tx=0).stop 2024-04-24T08:09:49.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.288+0000 7fc67f400640 1 -- 172.21.15.69:0/3980078195 shutdown_connections 2024-04-24T08:09:49.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.288+0000 7fc67f400640 1 --2- 172.21.15.69:0/3980078195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 0x7fc6780d0750 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.288+0000 7fc67f400640 1 --2- 172.21.15.69:0/3980078195 >> v2:172.21.15.121:3300/0 conn(0x7fc678145a20 0x7fc678145eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.288+0000 7fc67f400640 1 -- 172.21.15.69:0/3980078195 >> 172.21.15.69:0/3980078195 conn(0x7fc67806ba20 msgr2=0x7fc67806be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:49.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.289+0000 7fc67f400640 1 -- 172.21.15.69:0/3980078195 shutdown_connections 2024-04-24T08:09:49.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.289+0000 7fc67f400640 1 -- 172.21.15.69:0/3980078195 wait complete. 2024-04-24T08:09:49.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.290+0000 7fc67f400640 1 Processor -- start 2024-04-24T08:09:49.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.291+0000 7fc67f400640 1 -- start start 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.291+0000 7fc67f400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6780cd8e0 0x7fc6780c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc67f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 0x7fc6780adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc67f400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc678145a20 0x7fc67815f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc67f400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc6780cf230 con 0x7fc6780d0300 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc67f400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc6780cdcc0 con 0x7fc6780cd8e0 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc67f400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc6780cf3b0 con 0x7fc678145a20 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc677e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 0x7fc6780adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc677e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 0x7fc6780adab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45582/0 (socket says 172.21.15.69:45582) 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc677e00640 1 -- 172.21.15.69:0/3875598392 learned_addr learned my addr 172.21.15.69:0/3875598392 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc67ca00640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6780cd8e0 0x7fc6780c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc67d400640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc678145a20 0x7fc67815f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:49.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc677e00640 1 -- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc678145a20 msgr2=0x7fc67815f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc677e00640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc678145a20 0x7fc67815f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc677e00640 1 -- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6780cd8e0 msgr2=0x7fc6780c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc677e00640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6780cd8e0 0x7fc6780c61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc677e00640 1 -- 172.21.15.69:0/3875598392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6780c5430 con 0x7fc6780d0300 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.292+0000 7fc67ca00640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6780cd8e0 0x7fc6780c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.293+0000 7fc677e00640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 0x7fc6780adab0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc66400b410 tx=0x7fc66400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.293+0000 7fc674c00640 1 -- 172.21.15.69:0/3875598392 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664015020 con 0x7fc6780d0300 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.293+0000 7fc674c00640 1 -- 172.21.15.69:0/3875598392 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc664011850 con 0x7fc6780d0300 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.293+0000 7fc674c00640 1 -- 172.21.15.69:0/3875598392 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664011c40 con 0x7fc6780d0300 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.293+0000 7fc67f400640 1 -- 172.21.15.69:0/3875598392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6780c5710 con 0x7fc6780d0300 2024-04-24T08:09:49.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.293+0000 7fc67f400640 1 -- 172.21.15.69:0/3875598392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6780adff0 con 0x7fc6780d0300 2024-04-24T08:09:49.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.295+0000 7fc67f400640 1 -- 172.21.15.69:0/3875598392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc678075e30 con 0x7fc6780d0300 2024-04-24T08:09:49.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.295+0000 7fc674c00640 1 -- 172.21.15.69:0/3875598392 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc664018020 con 0x7fc6780d0300 2024-04-24T08:09:49.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.296+0000 7fc674c00640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc65806c920 0x7fc65806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.296+0000 7fc674c00640 1 -- 172.21.15.69:0/3875598392 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc66408ba30 con 0x7fc6780d0300 2024-04-24T08:09:49.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.297+0000 7fc67ca00640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc65806c920 0x7fc65806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:49.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.298+0000 7fc67ca00640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc65806c920 0x7fc65806ede0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc66c000f60 tx=0x7fc66c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:49.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.301+0000 7fc674c00640 1 -- 172.21.15.69:0/3875598392 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc66401f020 con 0x7fc6780d0300 2024-04-24T08:09:49.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.575+0000 7fc67f400640 1 -- 172.21.15.69:0/3875598392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fc6780d1050 con 0x7fc6780d0300 2024-04-24T08:09:49.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.580+0000 7fc674c00640 1 -- 172.21.15.69:0/3875598392 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) ==== 190+0+147 (secure 0 0 0) 0x7fc6640172c0 con 0x7fc6780d0300 2024-04-24T08:09:49.580 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-24T08:09:49.580 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw== 2024-04-24T08:09:49.580 INFO:teuthology.orchestra.run.smithi069.stdout: caps mds = "allow" 2024-04-24T08:09:49.580 INFO:teuthology.orchestra.run.smithi069.stdout: caps mon = "allow r" 2024-04-24T08:09:49.580 INFO:teuthology.orchestra.run.smithi069.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-24T08:09:49.581 INFO:teuthology.orchestra.run.smithi069.stderr:updated caps for client.0 2024-04-24T08:09:49.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.582+0000 7fc67f400640 1 -- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc65806c920 msgr2=0x7fc65806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.582+0000 7fc67f400640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc65806c920 0x7fc65806ede0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc66c000f60 tx=0x7fc66c009040 comp rx=0 tx=0).stop 2024-04-24T08:09:49.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.582+0000 7fc67f400640 1 -- 172.21.15.69:0/3875598392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 msgr2=0x7fc6780adab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.582+0000 7fc67f400640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 0x7fc6780adab0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc66400b410 tx=0x7fc66400b8d0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.583+0000 7fc67f400640 1 -- 172.21.15.69:0/3875598392 shutdown_connections 2024-04-24T08:09:49.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.583+0000 7fc67f400640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc65806c920 0x7fc65806ede0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.583+0000 7fc67f400640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6780d0300 0x7fc6780adab0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.583+0000 7fc67f400640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc678145a20 0x7fc67815f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.583+0000 7fc67f400640 1 --2- 172.21.15.69:0/3875598392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6780cd8e0 0x7fc6780c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.583+0000 7fc67f400640 1 -- 172.21.15.69:0/3875598392 >> 172.21.15.69:0/3875598392 conn(0x7fc67806ba20 msgr2=0x7fc6780ae580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:49.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.583+0000 7fc67f400640 1 -- 172.21.15.69:0/3875598392 shutdown_connections 2024-04-24T08:09:49.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.583+0000 7fc67f400640 1 -- 172.21.15.69:0/3875598392 wait complete. 2024-04-24T08:09:49.602 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:09:49.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.794+0000 7fe7d9200640 1 Processor -- start 2024-04-24T08:09:49.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.795+0000 7fe7d9200640 1 -- start start 2024-04-24T08:09:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.796+0000 7fe7d9200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe7d40b3ba0 0x7fe7d4163180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.796+0000 7fe7d9200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7d40d0880 0x7fe7d409a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.796+0000 7fe7d9200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d40d0d60 con 0x7fe7d409af10 2024-04-24T08:09:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.796+0000 7fe7d9200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe7d41636c0 con 0x7fe7d40d0880 2024-04-24T08:09:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.796+0000 7fe7d9200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe7d40d1040 con 0x7fe7d40b3ba0 2024-04-24T08:09:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.796+0000 7fe7d1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7d40d0880 0x7fe7d409a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.796+0000 7fe7d1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe7d40d0880 0x7fe7d409a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45588/0 (socket says 172.21.15.69:45588) 2024-04-24T08:09:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.796+0000 7fe7d1400640 1 -- 172.21.15.69:0/2624446892 learned_addr learned my addr 172.21.15.69:0/2624446892 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:49.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.797+0000 7fe7d1e00640 1 --2- 172.21.15.69:0/2624446892 >> v2:172.21.15.121:3300/0 conn(0x7fe7d40b3ba0 0x7fe7d4163180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:49.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.797+0000 7fe7d1400640 1 -- 172.21.15.69:0/2624446892 >> v2:172.21.15.121:3300/0 conn(0x7fe7d40b3ba0 msgr2=0x7fe7d4163180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:49.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.797+0000 7fe7d1400640 1 --2- 172.21.15.69:0/2624446892 >> v2:172.21.15.121:3300/0 conn(0x7fe7d40b3ba0 0x7fe7d4163180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.797+0000 7fe7d1400640 1 -- 172.21.15.69:0/2624446892 >> v1:172.21.15.121:6789/0 conn(0x7fe7d409af10 legacy=0x7fe7d406ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:49.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.797+0000 7fe7d1400640 1 -- 172.21.15.69:0/2624446892 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d40d1320 con 0x7fe7d40d0880 2024-04-24T08:09:49.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.797+0000 7fe7d1e00640 1 --2- 172.21.15.69:0/2624446892 >> v2:172.21.15.121:3300/0 conn(0x7fe7d40b3ba0 0x7fe7d4163180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:49.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.798+0000 7fe7d1400640 1 --2- 172.21.15.69:0/2624446892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d40d0880 0x7fe7d409a9d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40d2580 tx=0x7fe7bc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9972db20215761d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:49.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.798+0000 7fe7d0a00640 1 -- 172.21.15.69:0/2624446892 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc037070 con 0x7fe7d40d0880 2024-04-24T08:09:49.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.798+0000 7fe7d0a00640 1 -- 172.21.15.69:0/2624446892 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7bc032440 con 0x7fe7d40d0880 2024-04-24T08:09:49.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.798+0000 7fe7d0a00640 1 -- 172.21.15.69:0/2624446892 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7bc032810 con 0x7fe7d40d0880 2024-04-24T08:09:49.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.799+0000 7fe7d9200640 1 -- 172.21.15.69:0/2624446892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d40d0880 msgr2=0x7fe7d409a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.799+0000 7fe7d9200640 1 --2- 172.21.15.69:0/2624446892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d40d0880 0x7fe7d409a9d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe7d40d2580 tx=0x7fe7bc02a230 comp rx=0 tx=0).stop 2024-04-24T08:09:49.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.799+0000 7fe7d9200640 1 -- 172.21.15.69:0/2624446892 shutdown_connections 2024-04-24T08:09:49.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.799+0000 7fe7d9200640 1 --2- 172.21.15.69:0/2624446892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d40d0880 0x7fe7d409a9d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.799+0000 7fe7d9200640 1 --2- 172.21.15.69:0/2624446892 >> v2:172.21.15.121:3300/0 conn(0x7fe7d40b3ba0 0x7fe7d4163180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.799+0000 7fe7d9200640 1 -- 172.21.15.69:0/2624446892 >> 172.21.15.69:0/2624446892 conn(0x7fe7d40b40a0 msgr2=0x7fe7d40b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:49.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.799+0000 7fe7d9200640 1 -- 172.21.15.69:0/2624446892 shutdown_connections 2024-04-24T08:09:49.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.799+0000 7fe7d9200640 1 -- 172.21.15.69:0/2624446892 wait complete. 2024-04-24T08:09:49.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.801+0000 7fe7d9200640 1 Processor -- start 2024-04-24T08:09:49.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.801+0000 7fe7d9200640 1 -- start start 2024-04-24T08:09:49.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d9200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d409af10 0x7fe7d41e8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d9200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7d40b3ba0 0x7fe7d414a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d9200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7d40d0880 0x7fe7d40b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d9200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d409b940 con 0x7fe7d409af10 2024-04-24T08:09:49.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d9200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d40d18e0 con 0x7fe7d40d0880 2024-04-24T08:09:49.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d9200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d40b2f20 con 0x7fe7d40b3ba0 2024-04-24T08:09:49.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d409af10 0x7fe7d41e8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:49.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d409af10 0x7fe7d41e8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45592/0 (socket says 172.21.15.69:45592) 2024-04-24T08:09:49.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d1e00640 1 -- 172.21.15.69:0/839649533 learned_addr learned my addr 172.21.15.69:0/839649533 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:49.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d1e00640 1 -- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7d40b3ba0 msgr2=0x7fe7d414a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d1e00640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7d40b3ba0 0x7fe7d414a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d1e00640 1 -- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7d40d0880 msgr2=0x7fe7d40b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:49.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d1e00640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7d40d0880 0x7fe7d40b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:49.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.803+0000 7fe7d1e00640 1 -- 172.21.15.69:0/839649533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d414a9d0 con 0x7fe7d409af10 2024-04-24T08:09:49.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.804+0000 7fe7d1e00640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d409af10 0x7fe7d41e8550 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400cf90 tx=0x7fe7c400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:49.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.804+0000 7fe7ca000640 1 -- 172.21.15.69:0/839649533 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c401a070 con 0x7fe7d409af10 2024-04-24T08:09:49.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.804+0000 7fe7ca000640 1 -- 172.21.15.69:0/839649533 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c400d840 con 0x7fe7d409af10 2024-04-24T08:09:49.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.804+0000 7fe7ca000640 1 -- 172.21.15.69:0/839649533 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c400dc10 con 0x7fe7d409af10 2024-04-24T08:09:49.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.804+0000 7fe7d9200640 1 -- 172.21.15.69:0/839649533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d414acb0 con 0x7fe7d409af10 2024-04-24T08:09:49.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.804+0000 7fe7d9200640 1 -- 172.21.15.69:0/839649533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d414b270 con 0x7fe7d409af10 2024-04-24T08:09:49.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.805+0000 7fe7d9200640 1 -- 172.21.15.69:0/839649533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d4068a80 con 0x7fe7d409af10 2024-04-24T08:09:49.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.806+0000 7fe7ca000640 1 -- 172.21.15.69:0/839649533 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe7c401d070 con 0x7fe7d409af10 2024-04-24T08:09:49.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.807+0000 7fe7ca000640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7b006c8f0 0x7fe7b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:49.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.807+0000 7fe7ca000640 1 -- 172.21.15.69:0/839649533 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe7c408c3c0 con 0x7fe7d409af10 2024-04-24T08:09:49.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.808+0000 7fe7d1400640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7b006c8f0 0x7fe7b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:49.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.809+0000 7fe7d1400640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7b006c8f0 0x7fe7b006edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc034000 tx=0x7fe7bc02c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:49.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:49.811+0000 7fe7ca000640 1 -- 172.21.15.69:0/839649533 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe7c405a100 con 0x7fe7d409af10 2024-04-24T08:09:50.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.052+0000 7fe7d9200640 1 -- 172.21.15.69:0/839649533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe7d40b4cc0 con 0x7fe7d409af10 2024-04-24T08:09:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.053+0000 7fe7ca000640 1 -- 172.21.15.69:0/839649533 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5181 (secure 0 0 0) 0x7fe7c405e400 con 0x7fe7d409af10 2024-04-24T08:09:50.054 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:50.054 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2295273533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2295273533},{"type":"v1","addr":"172.21.15.121:6836","nonce":2295273533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1461471292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1461471292},{"type":"v1","addr":"172.21.15.121:6837","nonce":1461471292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/626688145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":626688145},{"type":"v1","addr":"172.21.15.69:6833","nonce":626688145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:09:38.549277+0000","modified":"2024-04-24T08:09:41.567377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/991803145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":991803145},{"type":"v1","addr":"172.21.15.69:6835","nonce":991803145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5241,"qdb_cluster":[5241]},"id":2}]} 2024-04-24T08:09:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 16 2024-04-24T08:09:50.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 -- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7b006c8f0 msgr2=0x7fe7b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7b006c8f0 0x7fe7b006edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc034000 tx=0x7fe7bc02c000 comp rx=0 tx=0).stop 2024-04-24T08:09:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 -- 172.21.15.69:0/839649533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d409af10 msgr2=0x7fe7d41e8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d409af10 0x7fe7d41e8550 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fe7c400cf90 tx=0x7fe7c400a7d0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 -- 172.21.15.69:0/839649533 shutdown_connections 2024-04-24T08:09:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7b006c8f0 0x7fe7b006edb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe7d40d0880 0x7fe7d40b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7d40b3ba0 0x7fe7d414a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 --2- 172.21.15.69:0/839649533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7d409af10 0x7fe7d41e8550 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 -- 172.21.15.69:0/839649533 >> 172.21.15.69:0/839649533 conn(0x7fe7d40b40a0 msgr2=0x7fe7d4070930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 -- 172.21.15.69:0/839649533 shutdown_connections 2024-04-24T08:09:50.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.056+0000 7fe7d9200640 1 -- 172.21.15.69:0/839649533 wait complete. 2024-04-24T08:09:50.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:09:38.549277+0000', 'modified': '2024-04-24T08:09:41.567377+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5241}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5241': {'gid': 5241, 'name': 'c', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.69:6835/991803145', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 991803145}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 991803145}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5241, 'qdb_cluster': [5241]} 2024-04-24T08:09:50.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:09:50.478 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-24T08:09:50.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.688+0000 7f0996c00640 1 Processor -- start 2024-04-24T08:09:50.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.688+0000 7f0996c00640 1 -- start start 2024-04-24T08:09:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.689+0000 7f0996c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f09901447d0 0x7f0990070cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.689+0000 7f0996c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f099006f230 0x7f09900ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.689+0000 7f0996c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f099015fe90 con 0x7f09900d0670 2024-04-24T08:09:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.689+0000 7f0996c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f099015f310 con 0x7f099006f230 2024-04-24T08:09:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.689+0000 7f0996c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f099015f5f0 con 0x7f09901447d0 2024-04-24T08:09:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.690+0000 7f0995800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f09900d0670 0x7f099006eb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44670/0 (socket says 172.21.15.69:44670) 2024-04-24T08:09:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.690+0000 7f0995800640 1 -- 172.21.15.69:0/3971673559 learned_addr learned my addr 172.21.15.69:0/3971673559 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:50.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.690+0000 7f0996200640 1 --2- 172.21.15.69:0/3971673559 >> v2:172.21.15.69:3300/0 conn(0x7f099006f230 0x7f09900ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.690+0000 7f0994e00640 1 --2- 172.21.15.69:0/3971673559 >> v2:172.21.15.121:3300/0 conn(0x7f09901447d0 0x7f0990070cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.690+0000 7f098fe00640 1 -- 172.21.15.69:0/3971673559 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378195937 0 0) 0x7f099015fe90 con 0x7f09900d0670 2024-04-24T08:09:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.690+0000 7f098fe00640 1 -- 172.21.15.69:0/3971673559 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0974003520 con 0x7f09900d0670 2024-04-24T08:09:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.690+0000 7f0996200640 1 -- 172.21.15.69:0/3971673559 >> v2:172.21.15.121:3300/0 conn(0x7f09901447d0 msgr2=0x7f0990070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.690+0000 7f0996200640 1 --2- 172.21.15.69:0/3971673559 >> v2:172.21.15.121:3300/0 conn(0x7f09901447d0 0x7f0990070cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.690+0000 7f0996200640 1 -- 172.21.15.69:0/3971673559 >> v1:172.21.15.69:6789/0 conn(0x7f09900d0670 legacy=0x7f099006eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.690+0000 7f0996200640 1 -- 172.21.15.69:0/3971673559 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099015f8d0 con 0x7f099006f230 2024-04-24T08:09:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.691+0000 7f0996200640 1 --2- 172.21.15.69:0/3971673559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f099006f230 0x7f09900ccb50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f09900680b0 tx=0x7f098802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7c9ff641e22219 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.692+0000 7f098fe00640 1 -- 172.21.15.69:0/3971673559 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0988037070 con 0x7f099006f230 2024-04-24T08:09:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.692+0000 7f098fe00640 1 -- 172.21.15.69:0/3971673559 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0988032440 con 0x7f099006f230 2024-04-24T08:09:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.692+0000 7f098fe00640 1 -- 172.21.15.69:0/3971673559 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098802ae60 con 0x7f099006f230 2024-04-24T08:09:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.692+0000 7f0996c00640 1 -- 172.21.15.69:0/3971673559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f099006f230 msgr2=0x7f09900ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.692+0000 7f0996c00640 1 --2- 172.21.15.69:0/3971673559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f099006f230 0x7f09900ccb50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f09900680b0 tx=0x7f098802a110 comp rx=0 tx=0).stop 2024-04-24T08:09:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.692+0000 7f0996c00640 1 -- 172.21.15.69:0/3971673559 shutdown_connections 2024-04-24T08:09:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.692+0000 7f0996c00640 1 --2- 172.21.15.69:0/3971673559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f099006f230 0x7f09900ccb50 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.692+0000 7f0996c00640 1 --2- 172.21.15.69:0/3971673559 >> v2:172.21.15.121:3300/0 conn(0x7f09901447d0 0x7f0990070cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.692+0000 7f0996c00640 1 -- 172.21.15.69:0/3971673559 >> 172.21.15.69:0/3971673559 conn(0x7f09900d12c0 msgr2=0x7f09900d16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.692+0000 7f0996c00640 1 -- 172.21.15.69:0/3971673559 shutdown_connections 2024-04-24T08:09:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.693+0000 7f0996c00640 1 -- 172.21.15.69:0/3971673559 wait complete. 2024-04-24T08:09:50.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.694+0000 7f0996c00640 1 Processor -- start 2024-04-24T08:09:50.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.694+0000 7f0996c00640 1 -- start start 2024-04-24T08:09:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.695+0000 7f0996c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f099006f230 0x7f0990150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.695+0000 7f0996c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09900d0670 0x7f09901436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.695+0000 7f0996c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09901447d0 0x7f09901532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.695+0000 7f0996c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09900cee90 con 0x7f09901447d0 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.695+0000 7f0996c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f099015efc0 con 0x7f09900d0670 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.695+0000 7f0996c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09900cf010 con 0x7f099006f230 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.696+0000 7f0996200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09901447d0 0x7f09901532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.696+0000 7f0996200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09901447d0 0x7f09901532e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45612/0 (socket says 172.21.15.69:45612) 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.696+0000 7f0996200640 1 -- 172.21.15.69:0/1911326520 learned_addr learned my addr 172.21.15.69:0/1911326520 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.696+0000 7f0996200640 1 -- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f099006f230 msgr2=0x7f0990150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.696+0000 7f0994e00640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09900d0670 0x7f09901436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.696+0000 7f0996200640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f099006f230 0x7f0990150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.696+0000 7f0996200640 1 -- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09900d0670 msgr2=0x7f09901436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.696+0000 7f0996200640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09900d0670 0x7f09901436f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.696+0000 7f0996200640 1 -- 172.21.15.69:0/1911326520 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0990166710 con 0x7f09901447d0 2024-04-24T08:09:50.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.696+0000 7f0996200640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09901447d0 0x7f09901532e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0988009890 tx=0x7f09880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.697+0000 7f098d600640 1 -- 172.21.15.69:0/1911326520 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0988037040 con 0x7f09901447d0 2024-04-24T08:09:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.697+0000 7f098d600640 1 -- 172.21.15.69:0/1911326520 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0988042070 con 0x7f09901447d0 2024-04-24T08:09:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.697+0000 7f0996c00640 1 -- 172.21.15.69:0/1911326520 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09901669f0 con 0x7f09901447d0 2024-04-24T08:09:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.697+0000 7f098d600640 1 -- 172.21.15.69:0/1911326520 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09880308a0 con 0x7f09901447d0 2024-04-24T08:09:50.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.697+0000 7f0996c00640 1 -- 172.21.15.69:0/1911326520 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0990151510 con 0x7f09901447d0 2024-04-24T08:09:50.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.699+0000 7f098d600640 1 -- 172.21.15.69:0/1911326520 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0988030a40 con 0x7f09901447d0 2024-04-24T08:09:50.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.700+0000 7f098d600640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f097006c8f0 0x7f097006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.700+0000 7f098d600640 1 -- 172.21.15.69:0/1911326520 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f09880ac4a0 con 0x7f09901447d0 2024-04-24T08:09:50.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.701+0000 7f0995800640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f097006c8f0 0x7f097006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.701+0000 7f099b1c8740 1 Processor -- start 2024-04-24T08:09:50.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.701+0000 7f099b1c8740 1 -- start start 2024-04-24T08:09:50.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.702+0000 7f0995800640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f097006c8f0 0x7f097006edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0980000f60 tx=0x7f09800078f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.702+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce0649980 0x564ce05fc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.702+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce062caa0 0x564ce062be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.702+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce062c360 0x564ce0601170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.702+0000 7f099b1c8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564ce0566b10 con 0x564ce062caa0 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.702+0000 7f099b1c8740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564ce050d2b0 con 0x564ce0649980 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.702+0000 7f099b1c8740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564ce0562ad0 con 0x564ce062c360 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.702+0000 7f0994e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce062caa0 0x564ce062be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0994e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce062caa0 0x564ce062be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45620/0 (socket says 172.21.15.69:45620) 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0994e00640 1 -- 172.21.15.69:0/930560737 learned_addr learned my addr 172.21.15.69:0/930560737 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0995800640 1 --2- 172.21.15.69:0/930560737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce0649980 0x564ce05fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0996200640 1 --2- 172.21.15.69:0/930560737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce062c360 0x564ce0601170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0994e00640 1 -- 172.21.15.69:0/930560737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce062c360 msgr2=0x564ce0601170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0994e00640 1 --2- 172.21.15.69:0/930560737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce062c360 0x564ce0601170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0994e00640 1 -- 172.21.15.69:0/930560737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce0649980 msgr2=0x564ce05fc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0994e00640 1 --2- 172.21.15.69:0/930560737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce0649980 0x564ce05fc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0994e00640 1 -- 172.21.15.69:0/930560737 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ce02792d0 con 0x564ce062caa0 2024-04-24T08:09:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0994e00640 1 --2- 172.21.15.69:0/930560737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce062caa0 0x564ce062be20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f097c009f30 tx=0x7f097c0077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=451e9a11a3b0cb0e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.703+0000 7f0986000640 1 -- 172.21.15.69:0/930560737 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c013780 con 0x564ce062caa0 2024-04-24T08:09:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f0986000640 1 -- 172.21.15.69:0/930560737 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f097c019070 con 0x564ce062caa0 2024-04-24T08:09:50.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f0986000640 1 -- 172.21.15.69:0/930560737 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c013b60 con 0x564ce062caa0 2024-04-24T08:09:50.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f099b1c8740 1 -- 172.21.15.69:0/930560737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce062caa0 msgr2=0x564ce062be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f099b1c8740 1 --2- 172.21.15.69:0/930560737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce062caa0 0x564ce062be20 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f097c009f30 tx=0x7f097c0077c0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f099b1c8740 1 -- 172.21.15.69:0/930560737 shutdown_connections 2024-04-24T08:09:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f099b1c8740 1 --2- 172.21.15.69:0/930560737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce062caa0 0x564ce062be20 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f099b1c8740 1 --2- 172.21.15.69:0/930560737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce062c360 0x564ce0601170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f099b1c8740 1 --2- 172.21.15.69:0/930560737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce0649980 0x564ce05fc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f099b1c8740 1 -- 172.21.15.69:0/930560737 >> 172.21.15.69:0/930560737 conn(0x564ce05fe580 msgr2=0x564ce0646e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f099b1c8740 1 -- 172.21.15.69:0/930560737 shutdown_connections 2024-04-24T08:09:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.704+0000 7f099b1c8740 1 -- 172.21.15.69:0/930560737 wait complete. 2024-04-24T08:09:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.705+0000 7f099b1c8740 1 -- start start 2024-04-24T08:09:50.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.705+0000 7f099b1c8740 1 Processor -- start 2024-04-24T08:09:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.706+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0627b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.706+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce0656ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.706+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0646dd0 0x564ce0647250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.706+0000 7f099b1c8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564ce0600730 con 0x564ce0649980 2024-04-24T08:09:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.706+0000 7f099b1c8740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564ce0520590 con 0x564ce062caa0 2024-04-24T08:09:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.706+0000 7f099b1c8740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564ce05ffb20 con 0x564ce0646dd0 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f099b1c8740 10 client.0 _ll_register_callbacks cb 0x564ce05033a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f0994e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce0656ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f0994e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce0656ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45628/0 (socket says 172.21.15.69:45628) 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f0994e00640 1 -- 172.21.15.69:0/1508574364 learned_addr learned my addr 172.21.15.69:0/1508574364 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f0995800640 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0627b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f0994e00640 1 -- 172.21.15.69:0/1508574364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0646dd0 msgr2=0x564ce0647250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f0994e00640 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0646dd0 0x564ce0647250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f0994e00640 1 -- 172.21.15.69:0/1508574364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 msgr2=0x564ce0627b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f0994e00640 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0627b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f0994e00640 1 -- 172.21.15.69:0/1508574364 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ce03436b0 con 0x564ce0649980 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f0994e00640 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce0656ee0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f097c007cc0 tx=0x7f097c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb45f82990403f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.707+0000 7f097aa00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f097aa00640 1 -- 172.21.15.69:0/1508574364 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c003b90 con 0x564ce0649980 2024-04-24T08:09:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f097aa00640 1 -- 172.21.15.69:0/1508574364 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f097c003d30 con 0x564ce0649980 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f099b1c8740 1 -- 172.21.15.69:0/1508574364 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f097c019070 con 0x564ce0649980 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f097aa00640 1 -- 172.21.15.69:0/1508574364 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f097c030050 con 0x564ce0649980 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f099b1c8740 10 client.5457 fetch_fsmap learned FSMap version 16 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f099b1c8740 1 -- 172.21.15.69:0/1508574364 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x564ce0657cf0 con 0x564ce0649980 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f097aa00640 1 -- 172.21.15.69:0/1508574364 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f097c019070 con 0x564ce0649980 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f097aa00640 1 -- 172.21.15.69:0/1508574364 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f097c033020 con 0x564ce0649980 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f099b1c8740 10 client.5457 fetch_fsmap finished waiting for FSMap version 16 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f099b1c8740 10 client.5457 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.708+0000 7f099b1c8740 20 client.5457 populate_metadata read hostname 'smithi069' 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.709+0000 7f099b1c8740 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.69:6834/991803145,v1:172.21.15.69:6835/991803145] conn(0x564ce062f910 0x564ce0665820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.709+0000 7f099b1c8740 4 client.5457 mds_command: new command op to 5241 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.709+0000 7f099b1c8740 1 -- 172.21.15.69:0/1508574364 --> [v2:172.21.15.69:6834/991803145,v1:172.21.15.69:6835/991803145] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ce0574530 con 0x564ce062f910 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.709+0000 7f0995800640 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.69:6834/991803145,v1:172.21.15.69:6835/991803145] conn(0x564ce062f910 0x564ce0665820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.709+0000 7f0995800640 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.69:6834/991803145,v1:172.21.15.69:6835/991803145] conn(0x564ce062f910 0x564ce0665820 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efa7d5a1a8d92d25 server_cookie=ab7f1c341e2f8022 in_seq=0 out_seq=0 2024-04-24T08:09:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.709+0000 7f097aa00640 10 client.5457 ms_handle_connect on v2:172.21.15.69:6834/991803145 2024-04-24T08:09:50.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.710+0000 7f097aa00640 1 -- 172.21.15.69:0/1508574364 <== mds.0 v2:172.21.15.69:6834/991803145 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ce0574530 con 0x564ce062f910 2024-04-24T08:09:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.710+0000 7f097aa00640 10 client.5457 handle_command_reply: tid=0 2024-04-24T08:09:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.710+0000 7f099b1c8740 1 client.5457 shutdown 2024-04-24T08:09:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 1 -- 172.21.15.69:0/1508574364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 msgr2=0x564ce0656ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce0656ee0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f097c007cc0 tx=0x7f097c004ab0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 1 -- 172.21.15.69:0/1508574364 shutdown_connections 2024-04-24T08:09:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.69:6834/991803145,v1:172.21.15.69:6835/991803145] conn(0x564ce062f910 0x564ce0665820 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce0656ee0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0646dd0 0x564ce0647250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f097aa00640 0 client.5457 ms_handle_reset on v2:172.21.15.69:6834/991803145 2024-04-24T08:09:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 1 --2- 172.21.15.69:0/1508574364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0627b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 1 -- 172.21.15.69:0/1508574364 >> 172.21.15.69:0/1508574364 conn(0x564ce062c360 msgr2=0x564ce05fb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 1 -- 172.21.15.69:0/1508574364 shutdown_connections 2024-04-24T08:09:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 1 -- 172.21.15.69:0/1508574364 wait complete. 2024-04-24T08:09:50.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.711+0000 7f099b1c8740 20 client.5457 trim_cache size 0 max 16384 2024-04-24T08:09:50.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.731+0000 7f099b1c8740 1 Processor -- start 2024-04-24T08:09:50.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.731+0000 7f099b1c8740 1 -- start start 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0602450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0646dd0 0x564ce0621ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0649980 0x564ce0644730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f099b1c8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0990200b50 con 0x564ce0646dd0 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f099b1c8740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f09901ffb90 con 0x564ce062caa0 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f099b1c8740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09901fef20 con 0x564ce0649980 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f0994e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0646dd0 0x564ce0621ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f0994e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0646dd0 0x564ce0621ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45642/0 (socket says 172.21.15.69:45642) 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f0994e00640 1 -- 172.21.15.69:0/3933664781 learned_addr learned my addr 172.21.15.69:0/3933664781 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f0994e00640 1 -- 172.21.15.69:0/3933664781 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0649980 msgr2=0x564ce0644730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f0994e00640 1 --2- 172.21.15.69:0/3933664781 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0649980 0x564ce0644730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f0994e00640 1 -- 172.21.15.69:0/3933664781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 msgr2=0x564ce0602450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f0994e00640 1 --2- 172.21.15.69:0/3933664781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0602450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.732+0000 7f0994e00640 1 -- 172.21.15.69:0/3933664781 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ce050c8b0 con 0x564ce0646dd0 2024-04-24T08:09:50.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.733+0000 7f0994e00640 1 --2- 172.21.15.69:0/3933664781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0646dd0 0x564ce0621ed0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f097c000f80 tx=0x7f097c021070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f51416a21f6ca10 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.733+0000 7f0985600640 1 -- 172.21.15.69:0/3933664781 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c003c50 con 0x564ce0646dd0 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.733+0000 7f0985600640 1 -- 172.21.15.69:0/3933664781 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f097c039070 con 0x564ce0646dd0 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.733+0000 7f0985600640 1 -- 172.21.15.69:0/3933664781 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097c0042e0 con 0x564ce0646dd0 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.734+0000 7f099b1c8740 1 -- 172.21.15.69:0/3933664781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0646dd0 msgr2=0x564ce0621ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.734+0000 7f099b1c8740 1 --2- 172.21.15.69:0/3933664781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0646dd0 0x564ce0621ed0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f097c000f80 tx=0x7f097c021070 comp rx=0 tx=0).stop 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.734+0000 7f099b1c8740 1 -- 172.21.15.69:0/3933664781 shutdown_connections 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.734+0000 7f099b1c8740 1 --2- 172.21.15.69:0/3933664781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0646dd0 0x564ce0621ed0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.734+0000 7f099b1c8740 1 --2- 172.21.15.69:0/3933664781 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0649980 0x564ce0644730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.734+0000 7f099b1c8740 1 --2- 172.21.15.69:0/3933664781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0602450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.734+0000 7f099b1c8740 1 -- 172.21.15.69:0/3933664781 >> 172.21.15.69:0/3933664781 conn(0x564ce062c360 msgr2=0x564ce0626a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.734+0000 7f099b1c8740 1 -- 172.21.15.69:0/3933664781 shutdown_connections 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.734+0000 7f099b1c8740 1 -- 172.21.15.69:0/3933664781 wait complete. 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.734+0000 7f099b1c8740 1 -- start start 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.735+0000 7f099b1c8740 1 Processor -- start 2024-04-24T08:09:50.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0648f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0646dd0 0x564ce06bb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f099b1c8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce05fc120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f099b1c8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564ce062d520 con 0x564ce0649980 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f099b1c8740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f09901ff540 con 0x564ce062caa0 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f099b1c8740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564ce0657420 con 0x564ce0646dd0 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f099b1c8740 10 client.0 _ll_register_callbacks cb 0x7f097c013890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f0996200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce05fc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f0996200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce05fc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45648/0 (socket says 172.21.15.69:45648) 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f0996200640 1 -- 172.21.15.69:0/2025294576 learned_addr learned my addr 172.21.15.69:0/2025294576 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f0996200640 1 -- 172.21.15.69:0/2025294576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0646dd0 msgr2=0x564ce06bb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f0996200640 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0646dd0 0x564ce06bb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f0996200640 1 -- 172.21.15.69:0/2025294576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 msgr2=0x564ce0648f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f0995800640 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0648f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.736+0000 7f0996200640 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0648f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.737+0000 7f0996200640 1 -- 172.21.15.69:0/2025294576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ce0668f10 con 0x564ce0649980 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.737+0000 7f0996200640 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce05fc120 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0988079f50 tx=0x7f0988032440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0174ad5720e108 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.737+0000 7f097aa00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.737+0000 7f097aa00640 1 -- 172.21.15.69:0/2025294576 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098807a250 con 0x564ce0649980 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.737+0000 7f097aa00640 1 -- 172.21.15.69:0/2025294576 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f0988030d30 con 0x564ce0649980 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.737+0000 7f099b1c8740 1 -- 172.21.15.69:0/2025294576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f097c039070 con 0x564ce0649980 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.737+0000 7f097aa00640 1 -- 172.21.15.69:0/2025294576 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f098807e600 con 0x564ce0649980 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.737+0000 7f099b1c8740 10 client.5463 fetch_fsmap learned FSMap version 16 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.737+0000 7f099b1c8740 1 -- 172.21.15.69:0/2025294576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x564ce06c5600 con 0x564ce0649980 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.737+0000 7f097aa00640 1 -- 172.21.15.69:0/2025294576 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f097c039070 con 0x564ce0649980 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.738+0000 7f097aa00640 1 -- 172.21.15.69:0/2025294576 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f098803a070 con 0x564ce0649980 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.738+0000 7f099b1c8740 10 client.5463 fetch_fsmap finished waiting for FSMap version 16 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.738+0000 7f099b1c8740 10 client.5463 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.738+0000 7f099b1c8740 20 client.5463 populate_metadata read hostname 'smithi069' 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.738+0000 7f099b1c8740 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.69:6834/991803145,v1:172.21.15.69:6835/991803145] conn(0x564ce06433b0 0x564ce0643790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.738+0000 7f099b1c8740 4 client.5463 mds_command: new command op to 5241 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.738+0000 7f099b1c8740 1 -- 172.21.15.69:0/2025294576 --> [v2:172.21.15.69:6834/991803145,v1:172.21.15.69:6835/991803145] -- command(tid 0: {"prefix": "status"}) -- 0x564ce063e5c0 con 0x564ce06433b0 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.738+0000 7f0995800640 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.69:6834/991803145,v1:172.21.15.69:6835/991803145] conn(0x564ce06433b0 0x564ce0643790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.738+0000 7f0995800640 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.69:6834/991803145,v1:172.21.15.69:6835/991803145] conn(0x564ce06433b0 0x564ce0643790 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3847145f23f97f22 server_cookie=f351f7eb4792f84c in_seq=0 out_seq=0 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.738+0000 7f097aa00640 10 client.5463 ms_handle_connect on v2:172.21.15.69:6834/991803145 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f097aa00640 1 -- 172.21.15.69:0/2025294576 <== mds.0 v2:172.21.15.69:6834/991803145 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x564ce063e5c0 con 0x564ce06433b0 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f097aa00640 10 client.5463 handle_command_reply: tid=0 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 client.5463 shutdown 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 -- 172.21.15.69:0/2025294576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 msgr2=0x564ce05fc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce05fc120 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0988079f50 tx=0x7f0988032440 comp rx=0 tx=0).stop 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 -- 172.21.15.69:0/2025294576 shutdown_connections 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.69:6834/991803145,v1:172.21.15.69:6835/991803145] conn(0x564ce06433b0 0x564ce0643790 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ce0649980 0x564ce05fc120 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564ce0646dd0 0x564ce06bb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 --2- 172.21.15.69:0/2025294576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564ce062caa0 0x564ce0648f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f097aa00640 0 client.5463 ms_handle_reset on v2:172.21.15.69:6834/991803145 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 -- 172.21.15.69:0/2025294576 >> 172.21.15.69:0/2025294576 conn(0x564ce062c360 msgr2=0x564ce0621ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 -- 172.21.15.69:0/2025294576 shutdown_connections 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 1 -- 172.21.15.69:0/2025294576 wait complete. 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.739+0000 7f099b1c8740 20 client.5463 trim_cache size 0 max 16384 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 5241, 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 11.660747992999999, 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 16, 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 33, 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 33, 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 20.491107064000001 2024-04-24T08:09:50.744 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:09:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 -- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f097006c8f0 msgr2=0x7f097006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f097006c8f0 0x7f097006edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0980000f60 tx=0x7f09800078f0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 -- 172.21.15.69:0/1911326520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09901447d0 msgr2=0x7f09901532e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09901447d0 0x7f09901532e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0988009890 tx=0x7f09880046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 -- 172.21.15.69:0/1911326520 shutdown_connections 2024-04-24T08:09:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f097006c8f0 0x7f097006edb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09901447d0 0x7f09901532e0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09900d0670 0x7f09901436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 --2- 172.21.15.69:0/1911326520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f099006f230 0x7f0990150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 -- 172.21.15.69:0/1911326520 >> 172.21.15.69:0/1911326520 conn(0x7f09900d12c0 msgr2=0x7f0990073280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 -- 172.21.15.69:0/1911326520 shutdown_connections 2024-04-24T08:09:50.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.740+0000 7f0996c00640 1 -- 172.21.15.69:0/1911326520 wait complete. 2024-04-24T08:09:50.758 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:09:50.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.947+0000 7f57c3e00640 1 Processor -- start 2024-04-24T08:09:50.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.947+0000 7f57c3e00640 1 -- start start 2024-04-24T08:09:50.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.948+0000 7f57c3e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f57c415d1d0 0x7f57c40ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.948+0000 7f57c3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f57c40b30f0 0x7f57c40ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.948+0000 7f57c3e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57c415d660 con 0x7f57c4156c60 2024-04-24T08:09:50.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.948+0000 7f57c3e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f57c406dc30 con 0x7f57c40b30f0 2024-04-24T08:09:50.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.948+0000 7f57c3e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f57c406df10 con 0x7f57c415d1d0 2024-04-24T08:09:50.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.949+0000 7f57c3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f57c40b30f0 0x7f57c40ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.949+0000 7f57c3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f57c40b30f0 0x7f57c40ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45660/0 (socket says 172.21.15.69:45660) 2024-04-24T08:09:50.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.949+0000 7f57c3400640 1 -- 172.21.15.69:0/617589222 learned_addr learned my addr 172.21.15.69:0/617589222 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:50.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.949+0000 7f57bbe00640 1 --2- 172.21.15.69:0/617589222 >> v2:172.21.15.121:3300/0 conn(0x7f57c415d1d0 0x7f57c40ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.949+0000 7f57c3400640 1 -- 172.21.15.69:0/617589222 >> v2:172.21.15.121:3300/0 conn(0x7f57c415d1d0 msgr2=0x7f57c40ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.949+0000 7f57c3400640 1 --2- 172.21.15.69:0/617589222 >> v2:172.21.15.121:3300/0 conn(0x7f57c415d1d0 0x7f57c40ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.949+0000 7f57c3400640 1 -- 172.21.15.69:0/617589222 >> v1:172.21.15.121:6789/0 conn(0x7f57c4156c60 legacy=0x7f57c415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:50.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.949+0000 7f57c3400640 1 -- 172.21.15.69:0/617589222 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c406e1f0 con 0x7f57c40b30f0 2024-04-24T08:09:50.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.950+0000 7f57c3400640 1 --2- 172.21.15.69:0/617589222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40b30f0 0x7f57c40ad610 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f57c406d900 tx=0x7f57bc02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d46a16f9b4296e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.950+0000 7f57c2000640 1 -- 172.21.15.69:0/617589222 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc036070 con 0x7f57c40b30f0 2024-04-24T08:09:50.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.950+0000 7f57c2000640 1 -- 172.21.15.69:0/617589222 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57bc031600 con 0x7f57c40b30f0 2024-04-24T08:09:50.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.950+0000 7f57c2000640 1 -- 172.21.15.69:0/617589222 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0319d0 con 0x7f57c40b30f0 2024-04-24T08:09:50.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.951+0000 7f57c3e00640 1 -- 172.21.15.69:0/617589222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40b30f0 msgr2=0x7f57c40ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.951+0000 7f57c3e00640 1 --2- 172.21.15.69:0/617589222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40b30f0 0x7f57c40ad610 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f57c406d900 tx=0x7f57bc02a6f0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.951+0000 7f57c3e00640 1 -- 172.21.15.69:0/617589222 shutdown_connections 2024-04-24T08:09:50.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.951+0000 7f57c3e00640 1 --2- 172.21.15.69:0/617589222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40b30f0 0x7f57c40ad610 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.951+0000 7f57c3e00640 1 --2- 172.21.15.69:0/617589222 >> v2:172.21.15.121:3300/0 conn(0x7f57c415d1d0 0x7f57c40ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.951+0000 7f57c3e00640 1 -- 172.21.15.69:0/617589222 >> 172.21.15.69:0/617589222 conn(0x7f57c40d0160 msgr2=0x7f57c40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.951+0000 7f57c3e00640 1 -- 172.21.15.69:0/617589222 shutdown_connections 2024-04-24T08:09:50.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.951+0000 7f57c3e00640 1 -- 172.21.15.69:0/617589222 wait complete. 2024-04-24T08:09:50.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.953+0000 7f57c3e00640 1 Processor -- start 2024-04-24T08:09:50.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.953+0000 7f57c3e00640 1 -- start start 2024-04-24T08:09:50.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.954+0000 7f57c3e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57c40b30f0 0x7f57c4155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.954+0000 7f57c3e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57c4156c60 0x7f57c41e83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.954+0000 7f57c3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40963c0 0x7f57c4096870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.954+0000 7f57c3e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f57c406e710 con 0x7f57c40963c0 2024-04-24T08:09:50.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.954+0000 7f57c3e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f57c415d940 con 0x7f57c40b30f0 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.954+0000 7f57c3e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f57c406e890 con 0x7f57c4156c60 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.954+0000 7f57c3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40963c0 0x7f57c4096870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.954+0000 7f57c3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40963c0 0x7f57c4096870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45668/0 (socket says 172.21.15.69:45668) 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.954+0000 7f57c3400640 1 -- 172.21.15.69:0/2830135589 learned_addr learned my addr 172.21.15.69:0/2830135589 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.955+0000 7f57bbe00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57c4156c60 0x7f57c41e83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.955+0000 7f57c2a00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57c40b30f0 0x7f57c4155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.955+0000 7f57c3400640 1 -- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57c4156c60 msgr2=0x7f57c41e83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.955+0000 7f57c3400640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57c4156c60 0x7f57c41e83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.955+0000 7f57c3400640 1 -- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57c40b30f0 msgr2=0x7f57c4155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.955+0000 7f57c3400640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57c40b30f0 0x7f57c4155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.955+0000 7f57c3400640 1 -- 172.21.15.69:0/2830135589 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c4155550 con 0x7f57c40963c0 2024-04-24T08:09:50.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.955+0000 7f57c3400640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40963c0 0x7f57c4096870 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009f40 tx=0x7f57bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.955+0000 7f57baa00640 1 -- 172.21.15.69:0/2830135589 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc036050 con 0x7f57c40963c0 2024-04-24T08:09:50.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.956+0000 7f57baa00640 1 -- 172.21.15.69:0/2830135589 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57bc042070 con 0x7f57c40963c0 2024-04-24T08:09:50.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.956+0000 7f57c3e00640 1 -- 172.21.15.69:0/2830135589 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c41e7310 con 0x7f57c40963c0 2024-04-24T08:09:50.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.956+0000 7f57baa00640 1 -- 172.21.15.69:0/2830135589 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc009510 con 0x7f57c40963c0 2024-04-24T08:09:50.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.956+0000 7f57c3e00640 1 -- 172.21.15.69:0/2830135589 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c41e78d0 con 0x7f57c40963c0 2024-04-24T08:09:50.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.957+0000 7f57c3e00640 1 -- 172.21.15.69:0/2830135589 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c4075d50 con 0x7f57c40963c0 2024-04-24T08:09:50.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.958+0000 7f57baa00640 1 -- 172.21.15.69:0/2830135589 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f57bc007bf0 con 0x7f57c40963c0 2024-04-24T08:09:50.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.959+0000 7f57baa00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57a006c920 0x7f57a006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.959+0000 7f57baa00640 1 -- 172.21.15.69:0/2830135589 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f57bc0ac530 con 0x7f57c40963c0 2024-04-24T08:09:50.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.959+0000 7f57c2a00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57a006c920 0x7f57a006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.961+0000 7f57c2a00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57a006c920 0x7f57a006ede0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f57b4007fd0 tx=0x7f57b4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:50.964+0000 7f57baa00640 1 -- 172.21.15.69:0/2830135589 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f57bc04b020 con 0x7f57c40963c0 2024-04-24T08:09:51.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.197+0000 7f57c3e00640 1 -- 172.21.15.69:0/2830135589 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f57c40b1790 con 0x7f57c40963c0 2024-04-24T08:09:51.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.198+0000 7f57baa00640 1 -- 172.21.15.69:0/2830135589 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f57bc07a8c0 con 0x7f57c40963c0 2024-04-24T08:09:51.198 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 -- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57a006c920 msgr2=0x7f57a006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57a006c920 0x7f57a006ede0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f57b4007fd0 tx=0x7f57b4009040 comp rx=0 tx=0).stop 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 -- 172.21.15.69:0/2830135589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40963c0 msgr2=0x7f57c4096870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40963c0 0x7f57c4096870 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009f40 tx=0x7f57bc004a40 comp rx=0 tx=0).stop 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 -- 172.21.15.69:0/2830135589 shutdown_connections 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f57a006c920 0x7f57a006ede0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57c40963c0 0x7f57c4096870 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57c4156c60 0x7f57c41e83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 --2- 172.21.15.69:0/2830135589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57c40b30f0 0x7f57c4155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 -- 172.21.15.69:0/2830135589 >> 172.21.15.69:0/2830135589 conn(0x7f57c40d0160 msgr2=0x7f57c415a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.200+0000 7f57c3e00640 1 -- 172.21.15.69:0/2830135589 shutdown_connections 2024-04-24T08:09:51.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51.201+0000 7f57c3e00640 1 -- 172.21.15.69:0/2830135589 wait complete. 2024-04-24T08:09:51.218 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:09:51.218 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:09:51.218 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:09:51.218 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:09:51.218 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:09:51.218 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:09:51.218 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:09:51.218 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:09:51.218 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:09:51.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:51.219 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:09:51.262 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:09:51.262 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:09:51.262 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42588sec preferred_lft 42588sec 2024-04-24T08:09:51.263 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:09:51.264 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:09:51.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:51.265 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:09:51.378 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:09:51.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:51.382 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:09:51.430 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:09:51.430 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:09:51.430 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:09:51.430 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:09:51.431 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:09:51.431 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:09:51.432 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:09:51.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:09:51.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:09:51.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:09:51.432 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:09:51.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:09:51.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:09:51.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:51.642 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:09:51.693 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:09:51.693 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:09:51.694 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:09:51.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:51.694 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:09:51.779 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:09:51.779 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:09:51.779 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:09:51.832 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:09:51.832 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:09:51.832 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:09:51.881 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:09:51.963 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:09:52.017 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:09:52.017 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:09:52.017 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:09:52.018 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:09:52.019 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:09:52.020 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:09:52.020 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:09:52.020 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:09:52.020 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:09:52.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:52.021 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:09:52.069 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:09:52.070 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:09:52.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:09:52.142 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:09:52.186 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:09:52.186 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:09:52.186 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:09:52.186 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:09:52.186 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:09:52.186 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:09:52.187 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:09:52.188 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:09:52.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:52.189 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:09:52.245 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:09:52.244+0000 7fe947fc8480 -1 init, newargv = 0x5555ff996290 newargc=15 2024-04-24T08:09:52.246 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[47460]: starting ceph client 2024-04-24T08:09:52.253 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[47460]: starting fuse 2024-04-24T08:09:52.273 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:09:52.273 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:09:52.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:09:52.273 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:09:52.273 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:09:52.274 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:09:52.275 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:09:52.275 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:09:52.275 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:09:52.275 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:09:52.275 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:09:52.275 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:09:52.275 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:09:52.275 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:09:52.465 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.47460.asok 2024-04-24T08:09:52.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:09:52.482 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47460.asok 2024-04-24T08:09:52.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:52.482 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47460.asok status 2024-04-24T08:09:52.708 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:09:52.708 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:09:52.708 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:09:52.708 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:09:52.708 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "47460", 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 5478, 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 5478 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:09:52.709 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3504588662 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3504588662 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.5478 192.168.0.1:0/3504588662", 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/3504588662", 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 16, 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 33, 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:09:52.710 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:09:52.711 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:09:52.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:52.723 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:09:52.723 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:09:52.774 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:09:52.774 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:09:52.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:52.774 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:09:52.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:52.856 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir) 2024-04-24T08:09:52.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:52.921 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir/childdir) 2024-04-24T08:09:52.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:52.977 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch rfile) 2024-04-24T08:09:53.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:53.033 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/pfile) 2024-04-24T08:09:53.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:53.093 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/childdir/cfile) 2024-04-24T08:09:53.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:53.149 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-04-24T08:09:53.215 INFO:teuthology.orchestra.run.smithi069.stdout:./ 2024-04-24T08:09:53.215 INFO:teuthology.orchestra.run.smithi069.stdout:./rfile 2024-04-24T08:09:53.216 INFO:teuthology.orchestra.run.smithi069.stdout:./parentdir 2024-04-24T08:09:53.216 INFO:teuthology.orchestra.run.smithi069.stdout:./parentdir/pfile 2024-04-24T08:09:53.216 INFO:teuthology.orchestra.run.smithi069.stdout:./parentdir/childdir 2024-04-24T08:09:53.216 INFO:teuthology.orchestra.run.smithi069.stdout:./parentdir/childdir/cfile 2024-04-24T08:09:53.217 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:09:53.217 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:09:53.217 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:09:53.217 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:09:53.217 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-04-24T08:09:53.218 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:09:53.355 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:09:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:09:53.369 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:09:53.369 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:09:53.369 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:09:53.369 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:09:53.369 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./rfile").st_ino) 2024-04-24T08:09:53.369 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:09:53.550 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627778 2024-04-24T08:09:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:09:53.563 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:09:53.563 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:09:53.563 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:09:53.563 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:09:53.563 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir").st_ino) 2024-04-24T08:09:53.563 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:09:53.746 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627776 2024-04-24T08:09:53.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:09:53.759 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:09:53.759 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:09:53.759 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:09:53.759 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:09:53.759 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/pfile").st_ino) 2024-04-24T08:09:53.759 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:09:53.944 INFO:teuthology.orchestra.run.smithi069.stdout:1099511628279 2024-04-24T08:09:53.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:09:53.958 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:09:53.958 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:09:53.958 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:09:53.958 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:09:53.958 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir").st_ino) 2024-04-24T08:09:53.958 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:09:54.142 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627777 2024-04-24T08:09:54.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:09:54.155 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:09:54.155 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:09:54.156 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:09:54.156 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:09:54.156 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir/cfile").st_ino) 2024-04-24T08:09:54.156 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:09:54.342 INFO:teuthology.orchestra.run.smithi069.stdout:1099511628280 2024-04-24T08:09:54.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:09:54.355 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:09:54.356 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:54.453 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:54.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.654+0000 7f58b8400640 1 Processor -- start 2024-04-24T08:09:54.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.654+0000 7f58b8400640 1 -- start start 2024-04-24T08:09:54.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.656+0000 7f58b8400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00d0300 0x7f58b006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.656+0000 7f58b8400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58b00cf3d0 con 0x7f58b0145fe0 2024-04-24T08:09:54.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.656+0000 7f58b8400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58b00cf6b0 con 0x7f58b0145a90 2024-04-24T08:09:54.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.656+0000 7f58b8400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f58b00cf990 con 0x7f58b00d0300 2024-04-24T08:09:54.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.656+0000 7f58b6400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f58b0145fe0 0x7f58b0160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44676/0 (socket says 172.21.15.69:44676) 2024-04-24T08:09:54.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.656+0000 7f58b6400640 1 -- 172.21.15.69:0/244149283 learned_addr learned my addr 172.21.15.69:0/244149283 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:54.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.656+0000 7f58b5a00640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00d0300 0x7f58b006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2858664017 0 0) 0x7f58b00cf3d0 con 0x7f58b0145fe0 2024-04-24T08:09:54.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5898003520 con 0x7f58b0145fe0 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 147192889 0 0) 0x7f5898003520 con 0x7f58b0145fe0 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00d0300 msgr2=0x7f58b006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00d0300 0x7f58b006e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 >> v1:172.21.15.121:6789/0 conn(0x7f58b0145a90 legacy=0x7f58b00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b00cfc70 con 0x7f58b0145fe0 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f58a80019f0 con 0x7f58b0145fe0 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 >> v1:172.21.15.69:6789/0 conn(0x7f58b0145fe0 legacy=0x7f58b0160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00d0300 0x7f58980039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5898003f60 0x7f5898004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5898004d40 0x7f58980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58b00cf990 con 0x7f5898004d40 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f589800baa0 con 0x7f5898003f60 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f589800bca0 con 0x7f58b00d0300 2024-04-24T08:09:54.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.657+0000 7f58b6400640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5898004d40 0x7f58980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.658+0000 7f58b5000640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5898003f60 0x7f5898004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.658+0000 7f58b6400640 1 -- 172.21.15.69:0/244149283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00d0300 msgr2=0x7f58980039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:54.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.658+0000 7f58b6400640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00d0300 0x7f58980039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.658+0000 7f58b6400640 1 -- 172.21.15.69:0/244149283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5898003f60 msgr2=0x7f5898004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:54.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.658+0000 7f58b6400640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5898003f60 0x7f5898004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.658+0000 7f58b6400640 1 -- 172.21.15.69:0/244149283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5898003520 con 0x7f5898004d40 2024-04-24T08:09:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.659+0000 7f58b6400640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5898004d40 0x7f58980051c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f58a800a860 tx=0x7f5898005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d17f4402aeb023c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.659+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a800a480 con 0x7f5898004d40 2024-04-24T08:09:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.659+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58a802f070 con 0x7f5898004d40 2024-04-24T08:09:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.659+0000 7f58afe00640 1 -- 172.21.15.69:0/244149283 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a802bd20 con 0x7f5898004d40 2024-04-24T08:09:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.660+0000 7f58b8400640 1 -- 172.21.15.69:0/244149283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5898004d40 msgr2=0x7f58980051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.660+0000 7f58b8400640 1 --2- 172.21.15.69:0/244149283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5898004d40 0x7f58980051c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f58a800a860 tx=0x7f5898005830 comp rx=0 tx=0).stop 2024-04-24T08:09:54.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.660+0000 7f58b6400640 1 -- 172.21.15.69:0/244149283 reap_dead start 2024-04-24T08:09:54.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.660+0000 7f58b8400640 1 -- 172.21.15.69:0/244149283 shutdown_connections 2024-04-24T08:09:54.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.660+0000 7f58b8400640 1 -- 172.21.15.69:0/244149283 >> 172.21.15.69:0/244149283 conn(0x7f58b0067d90 msgr2=0x7f58b006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:54.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.660+0000 7f58b8400640 1 -- 172.21.15.69:0/244149283 shutdown_connections 2024-04-24T08:09:54.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.660+0000 7f58b8400640 1 -- 172.21.15.69:0/244149283 wait complete. 2024-04-24T08:09:54.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.662+0000 7f58b8400640 1 Processor -- start 2024-04-24T08:09:54.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.662+0000 7f58b8400640 1 -- start start 2024-04-24T08:09:54.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b8400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5898004d40 0x7f58b01f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b8400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00ab1b0 0x7f58b01f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b8400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b01f2a40 0x7f58b015ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b8400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58b01458f0 con 0x7f58b01f2a40 2024-04-24T08:09:54.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b8400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f58b0145770 con 0x7f5898004d40 2024-04-24T08:09:54.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b8400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f58b00cdc30 con 0x7f58b00ab1b0 2024-04-24T08:09:54.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b6400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b01f2a40 0x7f58b015ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b6400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b01f2a40 0x7f58b015ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45684/0 (socket says 172.21.15.69:45684) 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b6400640 1 -- 172.21.15.69:0/3915868727 learned_addr learned my addr 172.21.15.69:0/3915868727 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b5a00640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5898004d40 0x7f58b01f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b6400640 1 -- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00ab1b0 msgr2=0x7f58b01f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.663+0000 7f58b5000640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00ab1b0 0x7f58b01f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58b6400640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00ab1b0 0x7f58b01f1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58b6400640 1 -- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5898004d40 msgr2=0x7f58b01f33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58b6400640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5898004d40 0x7f58b01f33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58b6400640 1 -- 172.21.15.69:0/3915868727 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b0132810 con 0x7f58b01f2a40 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58b5a00640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5898004d40 0x7f58b01f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58b5000640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00ab1b0 0x7f58b01f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58b6400640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b01f2a40 0x7f58b015ecc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f58a800a980 tx=0x7f58b00cf5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58ad600640 1 -- 172.21.15.69:0/3915868727 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a800a480 con 0x7f58b01f2a40 2024-04-24T08:09:54.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58ad600640 1 -- 172.21.15.69:0/3915868727 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58a8004b60 con 0x7f58b01f2a40 2024-04-24T08:09:54.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58ad600640 1 -- 172.21.15.69:0/3915868727 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a8034470 con 0x7f58b01f2a40 2024-04-24T08:09:54.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58b8400640 1 -- 172.21.15.69:0/3915868727 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b0132af0 con 0x7f58b01f2a40 2024-04-24T08:09:54.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.664+0000 7f58b8400640 1 -- 172.21.15.69:0/3915868727 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b01330b0 con 0x7f58b01f2a40 2024-04-24T08:09:54.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.666+0000 7f58b8400640 1 -- 172.21.15.69:0/3915868727 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58b0068a80 con 0x7f58b01f2a40 2024-04-24T08:09:54.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.667+0000 7f58ad600640 1 -- 172.21.15.69:0/3915868727 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f58a8034610 con 0x7f58b01f2a40 2024-04-24T08:09:54.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.668+0000 7f58ad600640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 0x7f589006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.668+0000 7f58ad600640 1 -- 172.21.15.69:0/3915868727 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f58a80aba30 con 0x7f58b01f2a40 2024-04-24T08:09:54.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.669+0000 7f58b5a00640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 0x7f589006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.670+0000 7f58b5a00640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 0x7f589006edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f58a40099e0 tx=0x7f58a4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:54.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.673+0000 7f58ad600640 1 -- 172.21.15.69:0/3915868727 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58a8079dc0 con 0x7f58b01f2a40 2024-04-24T08:09:54.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.873+0000 7f58b8400640 1 -- 172.21.15.69:0/3915868727 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58b006eeb0 con 0x7f58b01f2a40 2024-04-24T08:09:54.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.876+0000 7f58ad600640 1 -- 172.21.15.69:0/3915868727 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16645 (secure 0 0 0) 0x7f58a80796c0 con 0x7f58b01f2a40 2024-04-24T08:09:54.878 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:54.878 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":33,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:40.096668+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:09:36.197195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-24T08:09:37.506730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:54.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.878+0000 7f58b8400640 1 -- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 msgr2=0x7f589006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.878+0000 7f58b8400640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 0x7f589006edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f58a40099e0 tx=0x7f58a4008040 comp rx=0 tx=0).stop 2024-04-24T08:09:54.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.878+0000 7f58b8400640 1 -- 172.21.15.69:0/3915868727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b01f2a40 msgr2=0x7f58b015ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.878+0000 7f58b8400640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b01f2a40 0x7f58b015ecc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f58a800a980 tx=0x7f58b00cf5a0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.879+0000 7f58b8400640 1 -- 172.21.15.69:0/3915868727 shutdown_connections 2024-04-24T08:09:54.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.879+0000 7f58b8400640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 0x7f589006edb0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.879+0000 7f58b8400640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b01f2a40 0x7f58b015ecc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.879+0000 7f58b8400640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00ab1b0 0x7f58b01f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.879+0000 7f58b8400640 1 --2- 172.21.15.69:0/3915868727 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5898004d40 0x7f58b01f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.879+0000 7f58b8400640 1 -- 172.21.15.69:0/3915868727 >> 172.21.15.69:0/3915868727 conn(0x7f58b0067d90 msgr2=0x7f58b00aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:54.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.879+0000 7f58b8400640 1 -- 172.21.15.69:0/3915868727 shutdown_connections 2024-04-24T08:09:54.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:54.879+0000 7f58b8400640 1 -- 172.21.15.69:0/3915868727 wait complete. 2024-04-24T08:09:54.900 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:09:36.197195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:09:37.506730+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:54.901 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:55.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.107+0000 7f34b7e00640 1 Processor -- start 2024-04-24T08:09:55.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.107+0000 7f34b7e00640 1 -- start start 2024-04-24T08:09:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.108+0000 7f34b7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f34b80d09f0 0x7f34b8166020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.108+0000 7f34b7e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b80ce640 con 0x7f34b80b5300 2024-04-24T08:09:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.108+0000 7f34b7e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b80ce910 con 0x7f34b815ea60 2024-04-24T08:09:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.108+0000 7f34b7e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f34b80cebf0 con 0x7f34b80d09f0 2024-04-24T08:09:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.108+0000 7f34b7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f34b80d09f0 0x7f34b8166020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.109+0000 7f34b7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f34b80d09f0 0x7f34b8166020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45694/0 (socket says 172.21.15.69:45694) 2024-04-24T08:09:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.109+0000 7f34b7400640 1 -- 172.21.15.69:0/586235806 learned_addr learned my addr 172.21.15.69:0/586235806 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.109+0000 7f34b6000640 1 --1- 172.21.15.69:0/586235806 >> v1:172.21.15.69:6789/0 conn(0x7f34b80b5300 0x7f34b80d02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44692/0 (socket says 172.21.15.69:44692) 2024-04-24T08:09:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.109+0000 7f34b7400640 1 -- 172.21.15.69:0/586235806 >> v1:172.21.15.121:6789/0 conn(0x7f34b815ea60 legacy=0x7f34b80cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.109+0000 7f34b7400640 1 -- 172.21.15.69:0/586235806 >> v1:172.21.15.69:6789/0 conn(0x7f34b80b5300 legacy=0x7f34b80d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.109+0000 7f34b7400640 1 -- 172.21.15.69:0/586235806 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b80ceed0 con 0x7f34b80d09f0 2024-04-24T08:09:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.109+0000 7f34b5600640 1 -- 172.21.15.69:0/586235806 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 829706252 0 0) 0x7f34b80ce640 con 0x7f34b80b5300 2024-04-24T08:09:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.110+0000 7f34b7400640 1 --2- 172.21.15.69:0/586235806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80d09f0 0x7f34b8166020 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009e50 tx=0x7f34b002a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4f61bd37f6a968 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.111+0000 7f34b5600640 1 -- 172.21.15.69:0/586235806 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0036070 con 0x7f34b80d09f0 2024-04-24T08:09:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.111+0000 7f34b5600640 1 -- 172.21.15.69:0/586235806 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b0031600 con 0x7f34b80d09f0 2024-04-24T08:09:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.111+0000 7f34b5600640 1 -- 172.21.15.69:0/586235806 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00319d0 con 0x7f34b80d09f0 2024-04-24T08:09:55.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.112+0000 7f34b7e00640 1 -- 172.21.15.69:0/586235806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80d09f0 msgr2=0x7f34b8166020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.112+0000 7f34b7e00640 1 --2- 172.21.15.69:0/586235806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80d09f0 0x7f34b8166020 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009e50 tx=0x7f34b002a760 comp rx=0 tx=0).stop 2024-04-24T08:09:55.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.112+0000 7f34b7e00640 1 -- 172.21.15.69:0/586235806 shutdown_connections 2024-04-24T08:09:55.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.112+0000 7f34b7e00640 1 --2- 172.21.15.69:0/586235806 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80d09f0 0x7f34b8166020 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.112+0000 7f34b7e00640 1 -- 172.21.15.69:0/586235806 >> 172.21.15.69:0/586235806 conn(0x7f34b8067b80 msgr2=0x7f34b806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:55.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.112+0000 7f34b7e00640 1 -- 172.21.15.69:0/586235806 shutdown_connections 2024-04-24T08:09:55.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.112+0000 7f34b7e00640 1 -- 172.21.15.69:0/586235806 wait complete. 2024-04-24T08:09:55.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.114+0000 7f34b7e00640 1 Processor -- start 2024-04-24T08:09:55.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.114+0000 7f34b7e00640 1 -- start start 2024-04-24T08:09:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.116+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80b5300 0x7f34b814df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.116+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34b80d09f0 0x7f34b8168be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.116+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34b815ea60 0x7f34b8169120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.116+0000 7f34b7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f34b806ef50 con 0x7f34b80b5300 2024-04-24T08:09:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.116+0000 7f34b7e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f34b80cf490 con 0x7f34b80d09f0 2024-04-24T08:09:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.116+0000 7f34b7e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f34b806f0d0 con 0x7f34b815ea60 2024-04-24T08:09:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.116+0000 7f34b6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80b5300 0x7f34b814df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.116+0000 7f34b6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80b5300 0x7f34b814df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45702/0 (socket says 172.21.15.69:45702) 2024-04-24T08:09:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.116+0000 7f34b6a00640 1 -- 172.21.15.69:0/3261687868 learned_addr learned my addr 172.21.15.69:0/3261687868 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34b6a00640 1 -- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34b815ea60 msgr2=0x7f34b8169120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34b6a00640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34b815ea60 0x7f34b8169120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34b6a00640 1 -- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34b80d09f0 msgr2=0x7f34b8168be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34b6a00640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34b80d09f0 0x7f34b8168be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34b6a00640 1 -- 172.21.15.69:0/3261687868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b815b750 con 0x7f34b80b5300 2024-04-24T08:09:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34b6a00640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80b5300 0x7f34b814df30 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f34a800c430 tx=0x7f34a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34aea00640 1 -- 172.21.15.69:0/3261687868 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8011790 con 0x7f34b80b5300 2024-04-24T08:09:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34aea00640 1 -- 172.21.15.69:0/3261687868 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a8017070 con 0x7f34b80b5300 2024-04-24T08:09:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34b7e00640 1 -- 172.21.15.69:0/3261687868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b80afe40 con 0x7f34b80b5300 2024-04-24T08:09:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34aea00640 1 -- 172.21.15.69:0/3261687868 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a8011bb0 con 0x7f34b80b5300 2024-04-24T08:09:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.117+0000 7f34b7e00640 1 -- 172.21.15.69:0/3261687868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b80b0400 con 0x7f34b80b5300 2024-04-24T08:09:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.119+0000 7f34b7e00640 1 -- 172.21.15.69:0/3261687868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b8075d50 con 0x7f34b80b5300 2024-04-24T08:09:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.119+0000 7f34aea00640 1 -- 172.21.15.69:0/3261687868 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f34a801c460 con 0x7f34b80b5300 2024-04-24T08:09:55.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.120+0000 7f34aea00640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f349806c8f0 0x7f349806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.121+0000 7f34aea00640 1 -- 172.21.15.69:0/3261687868 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f34a808bb10 con 0x7f34b80b5300 2024-04-24T08:09:55.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.121+0000 7f34b6000640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f349806c8f0 0x7f349806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.123+0000 7f34b6000640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f349806c8f0 0x7f349806edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f34a4007fd0 tx=0x7f34a4007960 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:55.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.125+0000 7f34aea00640 1 -- 172.21.15.69:0/3261687868 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34a8059ea0 con 0x7f34b80b5300 2024-04-24T08:09:55.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.326+0000 7f34b7e00640 1 -- 172.21.15.69:0/3261687868 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f34b814f220 con 0x7f34b80b5300 2024-04-24T08:09:55.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.327+0000 7f34aea00640 1 -- 172.21.15.69:0/3261687868 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) ==== 93+0+10 (secure 0 0 0) 0x7f34a80597a0 con 0x7f34b80b5300 2024-04-24T08:09:55.328 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:55.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 -- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f349806c8f0 msgr2=0x7f349806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f349806c8f0 0x7f349806edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f34a4007fd0 tx=0x7f34a4007960 comp rx=0 tx=0).stop 2024-04-24T08:09:55.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 -- 172.21.15.69:0/3261687868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80b5300 msgr2=0x7f34b814df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80b5300 0x7f34b814df30 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f34a800c430 tx=0x7f34a800c8f0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 -- 172.21.15.69:0/3261687868 shutdown_connections 2024-04-24T08:09:55.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f349806c8f0 0x7f349806edb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f34b815ea60 0x7f34b8169120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34b80d09f0 0x7f34b8168be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 --2- 172.21.15.69:0/3261687868 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34b80b5300 0x7f34b814df30 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 -- 172.21.15.69:0/3261687868 >> 172.21.15.69:0/3261687868 conn(0x7f34b8067b80 msgr2=0x7f34b814e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.329+0000 7f34b7e00640 1 -- 172.21.15.69:0/3261687868 shutdown_connections 2024-04-24T08:09:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.330+0000 7f34b7e00640 1 -- 172.21.15.69:0/3261687868 wait complete. 2024-04-24T08:09:55.350 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:55.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.544+0000 7f4bc6800640 1 Processor -- start 2024-04-24T08:09:55.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.544+0000 7f4bc6800640 1 -- start start 2024-04-24T08:09:55.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.546+0000 7f4bc6800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc015c820 0x7f4bc00b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.546+0000 7f4bc6800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4bc00ce250 0x7f4bc00ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.546+0000 7f4bc6800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bc015cc00 con 0x7f4bc00b46a0 2024-04-24T08:09:55.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.546+0000 7f4bc6800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4bc006ed30 con 0x7f4bc00ce250 2024-04-24T08:09:55.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.546+0000 7f4bc6800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4bc00b4e10 con 0x7f4bc015c820 2024-04-24T08:09:55.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.546+0000 7f4bbf400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc015c820 0x7f4bc00b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:55.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.547+0000 7f4bbfe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4bc00ce250 0x7f4bc00ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:55.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.547+0000 7f4bbfe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4bc00ce250 0x7f4bc00ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:58614/0 (socket says 172.21.15.69:58614) 2024-04-24T08:09:55.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.547+0000 7f4bbfe00640 1 -- 172.21.15.69:0/4035331579 learned_addr learned my addr 172.21.15.69:0/4035331579 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:55.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.547+0000 7f4bbf400640 1 -- 172.21.15.69:0/4035331579 >> v2:172.21.15.121:3300/0 conn(0x7f4bc00ce250 msgr2=0x7f4bc00ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:55.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.547+0000 7f4bbf400640 1 --2- 172.21.15.69:0/4035331579 >> v2:172.21.15.121:3300/0 conn(0x7f4bc00ce250 0x7f4bc00ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.547+0000 7f4bbf400640 1 -- 172.21.15.69:0/4035331579 >> v1:172.21.15.121:6789/0 conn(0x7f4bc00b46a0 legacy=0x7f4bc006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.547+0000 7f4bbf400640 1 -- 172.21.15.69:0/4035331579 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc00b50f0 con 0x7f4bc015c820 2024-04-24T08:09:55.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.549+0000 7f4bbf400640 1 --2- 172.21.15.69:0/4035331579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc015c820 0x7f4bc00b3e90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4bb00097a0 tx=0x7f4bb002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d20c676e9086d963 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:55.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.549+0000 7f4bbe000640 1 -- 172.21.15.69:0/4035331579 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0037070 con 0x7f4bc015c820 2024-04-24T08:09:55.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.549+0000 7f4bbe000640 1 -- 172.21.15.69:0/4035331579 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bb0032440 con 0x7f4bc015c820 2024-04-24T08:09:55.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.549+0000 7f4bbe000640 1 -- 172.21.15.69:0/4035331579 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0032810 con 0x7f4bc015c820 2024-04-24T08:09:55.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.550+0000 7f4bc6800640 1 -- 172.21.15.69:0/4035331579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc015c820 msgr2=0x7f4bc00b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.550+0000 7f4bc6800640 1 --2- 172.21.15.69:0/4035331579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc015c820 0x7f4bc00b3e90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4bb00097a0 tx=0x7f4bb002a180 comp rx=0 tx=0).stop 2024-04-24T08:09:55.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.550+0000 7f4bc6800640 1 -- 172.21.15.69:0/4035331579 shutdown_connections 2024-04-24T08:09:55.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.550+0000 7f4bc6800640 1 --2- 172.21.15.69:0/4035331579 >> v2:172.21.15.121:3300/0 conn(0x7f4bc00ce250 0x7f4bc00ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.550+0000 7f4bc6800640 1 --2- 172.21.15.69:0/4035331579 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc015c820 0x7f4bc00b3e90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.550+0000 7f4bc6800640 1 -- 172.21.15.69:0/4035331579 >> 172.21.15.69:0/4035331579 conn(0x7f4bc00d0350 msgr2=0x7f4bc00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.550+0000 7f4bc6800640 1 -- 172.21.15.69:0/4035331579 shutdown_connections 2024-04-24T08:09:55.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.550+0000 7f4bc6800640 1 -- 172.21.15.69:0/4035331579 wait complete. 2024-04-24T08:09:55.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.552+0000 7f4bc6800640 1 Processor -- start 2024-04-24T08:09:55.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.552+0000 7f4bc6800640 1 -- start start 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.553+0000 7f4bc6800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc00b46a0 0x7f4bc01503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.553+0000 7f4bc6800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc00ce250 0x7f4bc015ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.553+0000 7f4bc6800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bc015c820 0x7f4bc015d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.553+0000 7f4bc6800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc00b4b00 con 0x7f4bc00ce250 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.553+0000 7f4bc6800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc006f2b0 con 0x7f4bc015c820 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.553+0000 7f4bc6800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4bc006fee0 con 0x7f4bc00b46a0 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.553+0000 7f4bbea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc00ce250 0x7f4bc015ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.553+0000 7f4bbea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc00ce250 0x7f4bc015ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45708/0 (socket says 172.21.15.69:45708) 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.553+0000 7f4bbea00640 1 -- 172.21.15.69:0/1866688568 learned_addr learned my addr 172.21.15.69:0/1866688568 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.553+0000 7f4bbf400640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc00b46a0 0x7f4bc01503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bbfe00640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bc015c820 0x7f4bc015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:55.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bbea00640 1 -- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc00b46a0 msgr2=0x7f4bc01503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bbea00640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc00b46a0 0x7f4bc01503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bbea00640 1 -- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bc015c820 msgr2=0x7f4bc015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bbea00640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bc015c820 0x7f4bc015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bbea00640 1 -- 172.21.15.69:0/1866688568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc0073750 con 0x7f4bc00ce250 2024-04-24T08:09:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bbea00640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc00ce250 0x7f4bc015ce00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c400 tx=0x7f4ba800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bb7400640 1 -- 172.21.15.69:0/1866688568 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80052d0 con 0x7f4bc00ce250 2024-04-24T08:09:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bb7400640 1 -- 172.21.15.69:0/1866688568 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba8005490 con 0x7f4bc00ce250 2024-04-24T08:09:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bb7400640 1 -- 172.21.15.69:0/1866688568 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8007260 con 0x7f4bc00ce250 2024-04-24T08:09:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.554+0000 7f4bc6800640 1 -- 172.21.15.69:0/1866688568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc0150d00 con 0x7f4bc00ce250 2024-04-24T08:09:55.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.555+0000 7f4bc6800640 1 -- 172.21.15.69:0/1866688568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc00cb130 con 0x7f4bc00ce250 2024-04-24T08:09:55.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.556+0000 7f4bc6800640 1 -- 172.21.15.69:0/1866688568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc0068a80 con 0x7f4bc00ce250 2024-04-24T08:09:55.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.557+0000 7f4bb7400640 1 -- 172.21.15.69:0/1866688568 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ba8007400 con 0x7f4bc00ce250 2024-04-24T08:09:55.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.557+0000 7f4bb7400640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9c06c920 0x7f4b9c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.558+0000 7f4bb7400640 1 -- 172.21.15.69:0/1866688568 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f4ba808b060 con 0x7f4bc00ce250 2024-04-24T08:09:55.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.558+0000 7f4bbf400640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9c06c920 0x7f4b9c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:55.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.560+0000 7f4bbf400640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9c06c920 0x7f4b9c06ede0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0034000 tx=0x7f4bb002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:55.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.562+0000 7f4bb7400640 1 -- 172.21.15.69:0/1866688568 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4bc0150d00 con 0x7f4bc00ce250 2024-04-24T08:09:55.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.766+0000 7f4bc6800640 1 -- 172.21.15.69:0/1866688568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4bc00cddc0 con 0x7f4bc00ce250 2024-04-24T08:09:55.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.767+0000 7f4bb7400640 1 -- 172.21.15.69:0/1866688568 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) ==== 104+0+11 (secure 0 0 0) 0x7f4ba8059490 con 0x7f4bc00ce250 2024-04-24T08:09:55.769 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 -- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9c06c920 msgr2=0x7f4b9c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9c06c920 0x7f4b9c06ede0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4bb0034000 tx=0x7f4bb002e000 comp rx=0 tx=0).stop 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 -- 172.21.15.69:0/1866688568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc00ce250 msgr2=0x7f4bc015ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc00ce250 0x7f4bc015ce00 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800c400 tx=0x7f4ba800c8c0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 -- 172.21.15.69:0/1866688568 shutdown_connections 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9c06c920 0x7f4b9c06ede0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bc00ce250 0x7f4bc015ce00 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bc015c820 0x7f4bc015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 --2- 172.21.15.69:0/1866688568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bc00b46a0 0x7f4bc01503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 -- 172.21.15.69:0/1866688568 >> 172.21.15.69:0/1866688568 conn(0x7f4bc00d0350 msgr2=0x7f4bc01601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 -- 172.21.15.69:0/1866688568 shutdown_connections 2024-04-24T08:09:55.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.770+0000 7f4bc6800640 1 -- 172.21.15.69:0/1866688568 wait complete. 2024-04-24T08:09:55.790 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:55.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.994+0000 7fa638e00640 1 Processor -- start 2024-04-24T08:09:55.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.994+0000 7fa638e00640 1 -- start start 2024-04-24T08:09:55.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.995+0000 7fa638e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6340b5300 0x7fa6340d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.995+0000 7fa638e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6340d0820 0x7fa63406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:55.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.995+0000 7fa638e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6340ce640 con 0x7fa63415ea60 2024-04-24T08:09:55.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.995+0000 7fa638e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa6340d0c90 con 0x7fa6340b5300 2024-04-24T08:09:55.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.995+0000 7fa638e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa6340ce910 con 0x7fa6340d0820 2024-04-24T08:09:55.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.996+0000 7fa62aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6340b5300 0x7fa6340d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:55.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.996+0000 7fa633400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa63415ea60 0x7fa6340cdf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44694/0 (socket says 172.21.15.69:44694) 2024-04-24T08:09:55.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.996+0000 7fa633400640 1 -- 172.21.15.69:0/478546290 learned_addr learned my addr 172.21.15.69:0/478546290 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:55.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.996+0000 7fa62aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6340b5300 0x7fa6340d02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45710/0 (socket says 172.21.15.69:45710) 2024-04-24T08:09:55.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.996+0000 7fa62aa00640 1 -- 172.21.15.69:0/478546290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6340d0820 msgr2=0x7fa63406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:55.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.996+0000 7fa62aa00640 1 --2- 172.21.15.69:0/478546290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6340d0820 0x7fa63406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:55.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.996+0000 7fa62aa00640 1 -- 172.21.15.69:0/478546290 >> v1:172.21.15.69:6789/0 conn(0x7fa63415ea60 legacy=0x7fa6340cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:55.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.996+0000 7fa62aa00640 1 -- 172.21.15.69:0/478546290 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6340cebf0 con 0x7fa6340b5300 2024-04-24T08:09:55.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.996+0000 7fa632a00640 1 -- 172.21.15.69:0/478546290 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 180665112 0 0) 0x7fa6340ce640 con 0x7fa63415ea60 2024-04-24T08:09:56.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.997+0000 7fa62aa00640 1 --2- 172.21.15.69:0/478546290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 0x7fa6340d02e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00ba40 tx=0x7fa61c02c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8339eec2967b40 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.998+0000 7fa632a00640 1 -- 172.21.15.69:0/478546290 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c032750 con 0x7fa6340b5300 2024-04-24T08:09:56.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.998+0000 7fa632a00640 1 -- 172.21.15.69:0/478546290 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa61c030050 con 0x7fa6340b5300 2024-04-24T08:09:56.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.998+0000 7fa632a00640 1 -- 172.21.15.69:0/478546290 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c032b10 con 0x7fa6340b5300 2024-04-24T08:09:56.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.998+0000 7fa638e00640 1 -- 172.21.15.69:0/478546290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 msgr2=0x7fa6340d02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.998+0000 7fa638e00640 1 --2- 172.21.15.69:0/478546290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 0x7fa6340d02e0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00ba40 tx=0x7fa61c02c850 comp rx=0 tx=0).stop 2024-04-24T08:09:56.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.998+0000 7fa638e00640 1 -- 172.21.15.69:0/478546290 shutdown_connections 2024-04-24T08:09:56.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.998+0000 7fa638e00640 1 --2- 172.21.15.69:0/478546290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6340d0820 0x7fa63406df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.998+0000 7fa638e00640 1 --2- 172.21.15.69:0/478546290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 0x7fa6340d02e0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.998+0000 7fa638e00640 1 -- 172.21.15.69:0/478546290 >> 172.21.15.69:0/478546290 conn(0x7fa634067b80 msgr2=0x7fa63406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:56.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.999+0000 7fa638e00640 1 -- 172.21.15.69:0/478546290 shutdown_connections 2024-04-24T08:09:56.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:55.999+0000 7fa638e00640 1 -- 172.21.15.69:0/478546290 wait complete. 2024-04-24T08:09:56.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.000+0000 7fa638e00640 1 Processor -- start 2024-04-24T08:09:56.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.000+0000 7fa638e00640 1 -- start start 2024-04-24T08:09:56.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.001+0000 7fa638e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 0x7fa6340b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.001+0000 7fa638e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6340d0820 0x7fa6340aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.001+0000 7fa638e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa63415ea60 0x7fa634148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.001+0000 7fa638e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa6340cf490 con 0x7fa6340b5300 2024-04-24T08:09:56.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.001+0000 7fa638e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa6340cf1b0 con 0x7fa63415ea60 2024-04-24T08:09:56.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.001+0000 7fa638e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa63406e480 con 0x7fa6340d0820 2024-04-24T08:09:56.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 0x7fa6340b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 0x7fa6340b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45722/0 (socket says 172.21.15.69:45722) 2024-04-24T08:09:56.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633400640 1 -- 172.21.15.69:0/181751389 learned_addr learned my addr 172.21.15.69:0/181751389 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:56.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633e00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa63415ea60 0x7fa634148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa62aa00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6340d0820 0x7fa6340aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633400640 1 -- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6340d0820 msgr2=0x7fa6340aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633400640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6340d0820 0x7fa6340aed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633400640 1 -- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa63415ea60 msgr2=0x7fa634148f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633400640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa63415ea60 0x7fa634148f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633400640 1 -- 172.21.15.69:0/181751389 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6340af2a0 con 0x7fa6340b5300 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633e00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa63415ea60 0x7fa634148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.002+0000 7fa633400640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 0x7fa6340b3f10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa62400cb10 tx=0x7fa62400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.003+0000 7fa62be00640 1 -- 172.21.15.69:0/181751389 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624017070 con 0x7fa6340b5300 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.003+0000 7fa62be00640 1 -- 172.21.15.69:0/181751389 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa62400ac70 con 0x7fa6340b5300 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.003+0000 7fa62be00640 1 -- 172.21.15.69:0/181751389 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624012620 con 0x7fa6340b5300 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.003+0000 7fa638e00640 1 -- 172.21.15.69:0/181751389 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa634149440 con 0x7fa6340b5300 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.003+0000 7fa638e00640 1 -- 172.21.15.69:0/181751389 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6341531a0 con 0x7fa6340b5300 2024-04-24T08:09:56.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.004+0000 7fa638e00640 1 -- 172.21.15.69:0/181751389 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa634075d50 con 0x7fa6340b5300 2024-04-24T08:09:56.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.005+0000 7fa62be00640 1 -- 172.21.15.69:0/181751389 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa62401a070 con 0x7fa6340b5300 2024-04-24T08:09:56.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.006+0000 7fa62be00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa61006c8f0 0x7fa61006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.006+0000 7fa62be00640 1 -- 172.21.15.69:0/181751389 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fa62408b9f0 con 0x7fa6340b5300 2024-04-24T08:09:56.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.006+0000 7fa62aa00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa61006c8f0 0x7fa61006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.008+0000 7fa62aa00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa61006c8f0 0x7fa61006edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa61c037000 tx=0x7fa61c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.010+0000 7fa62be00640 1 -- 172.21.15.69:0/181751389 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa62400ae20 con 0x7fa6340b5300 2024-04-24T08:09:56.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.212+0000 7fa638e00640 1 -- 172.21.15.69:0/181751389 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa63406cd50 con 0x7fa6340b5300 2024-04-24T08:09:56.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.212+0000 7fa62be00640 1 -- 172.21.15.69:0/181751389 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) ==== 100+0+11 (secure 0 0 0) 0x7fa624059d80 con 0x7fa6340b5300 2024-04-24T08:09:56.215 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:56.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 -- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa61006c8f0 msgr2=0x7fa61006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa61006c8f0 0x7fa61006edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa61c037000 tx=0x7fa61c008040 comp rx=0 tx=0).stop 2024-04-24T08:09:56.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 -- 172.21.15.69:0/181751389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 msgr2=0x7fa6340b3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 0x7fa6340b3f10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa62400cb10 tx=0x7fa62400a060 comp rx=0 tx=0).stop 2024-04-24T08:09:56.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 -- 172.21.15.69:0/181751389 shutdown_connections 2024-04-24T08:09:56.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa61006c8f0 0x7fa61006edb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa63415ea60 0x7fa634148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6340d0820 0x7fa6340aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 --2- 172.21.15.69:0/181751389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6340b5300 0x7fa6340b3f10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 -- 172.21.15.69:0/181751389 >> 172.21.15.69:0/181751389 conn(0x7fa634067b80 msgr2=0x7fa63415d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:56.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 -- 172.21.15.69:0/181751389 shutdown_connections 2024-04-24T08:09:56.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.215+0000 7fa638e00640 1 -- 172.21.15.69:0/181751389 wait complete. 2024-04-24T08:09:56.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:09:56.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.429+0000 7f6d0f600640 1 Processor -- start 2024-04-24T08:09:56.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.430+0000 7f6d0f600640 1 -- start start 2024-04-24T08:09:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.431+0000 7f6d0f600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6d08071d70 0x7f6d08072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.431+0000 7f6d0f600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 0x7f6d0806e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.431+0000 7f6d0f600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d08072690 con 0x7f6d0806f730 2024-04-24T08:09:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.431+0000 7f6d0f600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6d080b4f80 con 0x7f6d08071d70 2024-04-24T08:09:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.431+0000 7f6d0f600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6d080b4370 con 0x7f6d080b3320 2024-04-24T08:09:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.431+0000 7f6d07e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6d0806f730 0x7f6d0806dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44696/0 (socket says 172.21.15.69:44696) 2024-04-24T08:09:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.431+0000 7f6d07e00640 1 -- 172.21.15.69:0/1958224653 learned_addr learned my addr 172.21.15.69:0/1958224653 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.431+0000 7f6d0cc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6d08071d70 0x7f6d08072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.432+0000 7f6d0d600640 1 --2- 172.21.15.69:0/1958224653 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 0x7f6d0806e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.432+0000 7f6d07400640 1 -- 172.21.15.69:0/1958224653 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3486821097 0 0) 0x7f6d08072690 con 0x7f6d0806f730 2024-04-24T08:09:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.432+0000 7f6d07400640 1 -- 172.21.15.69:0/1958224653 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cf0003520 con 0x7f6d0806f730 2024-04-24T08:09:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.432+0000 7f6d0cc00640 1 -- 172.21.15.69:0/1958224653 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 msgr2=0x7f6d0806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.432+0000 7f6d0cc00640 1 --2- 172.21.15.69:0/1958224653 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 0x7f6d0806e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.432+0000 7f6d0cc00640 1 -- 172.21.15.69:0/1958224653 >> v1:172.21.15.69:6789/0 conn(0x7f6d0806f730 legacy=0x7f6d0806dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.432+0000 7f6d0cc00640 1 -- 172.21.15.69:0/1958224653 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d080b4650 con 0x7f6d08071d70 2024-04-24T08:09:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.433+0000 7f6d0cc00640 1 --2- 172.21.15.69:0/1958224653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d08071d70 0x7f6d08072150 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00a940 tx=0x7f6cfc02b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4244ea606c0638b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.433+0000 7f6d07400640 1 -- 172.21.15.69:0/1958224653 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc00b040 con 0x7f6d08071d70 2024-04-24T08:09:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.433+0000 7f6d07400640 1 -- 172.21.15.69:0/1958224653 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cfc030710 con 0x7f6d08071d70 2024-04-24T08:09:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.433+0000 7f6d07400640 1 -- 172.21.15.69:0/1958224653 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc030ac0 con 0x7f6d08071d70 2024-04-24T08:09:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.434+0000 7f6d0f600640 1 -- 172.21.15.69:0/1958224653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d08071d70 msgr2=0x7f6d08072150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.434+0000 7f6d0f600640 1 --2- 172.21.15.69:0/1958224653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d08071d70 0x7f6d08072150 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc00a940 tx=0x7f6cfc02b540 comp rx=0 tx=0).stop 2024-04-24T08:09:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.434+0000 7f6d0f600640 1 -- 172.21.15.69:0/1958224653 shutdown_connections 2024-04-24T08:09:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.434+0000 7f6d0f600640 1 --2- 172.21.15.69:0/1958224653 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 0x7f6d0806e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.434+0000 7f6d0f600640 1 --2- 172.21.15.69:0/1958224653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d08071d70 0x7f6d08072150 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.434+0000 7f6d0f600640 1 -- 172.21.15.69:0/1958224653 >> 172.21.15.69:0/1958224653 conn(0x7f6d08067d90 msgr2=0x7f6d0806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.434+0000 7f6d0f600640 1 -- 172.21.15.69:0/1958224653 shutdown_connections 2024-04-24T08:09:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.434+0000 7f6d0f600640 1 -- 172.21.15.69:0/1958224653 wait complete. 2024-04-24T08:09:56.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.436+0000 7f6d0f600640 1 Processor -- start 2024-04-24T08:09:56.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.436+0000 7f6d0f600640 1 -- start start 2024-04-24T08:09:56.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.437+0000 7f6d0f600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0806f730 0x7f6d081e3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.437+0000 7f6d0f600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d08071d70 0x7f6d08154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.437+0000 7f6d0f600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 0x7f6d081f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.437+0000 7f6d0f600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0806ee60 con 0x7f6d0806f730 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.437+0000 7f6d0f600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6d080b4c10 con 0x7f6d08071d70 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.437+0000 7f6d0f600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6d0815b040 con 0x7f6d080b3320 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.437+0000 7f6d0cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0806f730 0x7f6d081e3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.437+0000 7f6d0cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0806f730 0x7f6d081e3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45750/0 (socket says 172.21.15.69:45750) 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.437+0000 7f6d0cc00640 1 -- 172.21.15.69:0/1226395461 learned_addr learned my addr 172.21.15.69:0/1226395461 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d07e00640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d08071d70 0x7f6d08154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d0d600640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 0x7f6d081f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d0cc00640 1 -- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 msgr2=0x7f6d081f2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d0cc00640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 0x7f6d081f2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d0cc00640 1 -- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d08071d70 msgr2=0x7f6d08154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d0cc00640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d08071d70 0x7f6d08154fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d0cc00640 1 -- 172.21.15.69:0/1226395461 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d081f3450 con 0x7f6d0806f730 2024-04-24T08:09:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d0d600640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 0x7f6d081f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d0cc00640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0806f730 0x7f6d081e3580 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc000c00 tx=0x7f6cfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d04c00640 1 -- 172.21.15.69:0/1226395461 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc03f020 con 0x7f6d0806f730 2024-04-24T08:09:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d04c00640 1 -- 172.21.15.69:0/1226395461 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cfc033a50 con 0x7f6d0806f730 2024-04-24T08:09:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d04c00640 1 -- 172.21.15.69:0/1226395461 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc033e20 con 0x7f6d0806f730 2024-04-24T08:09:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d0f600640 1 -- 172.21.15.69:0/1226395461 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d081f39d0 con 0x7f6d0806f730 2024-04-24T08:09:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.438+0000 7f6d0f600640 1 -- 172.21.15.69:0/1226395461 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d081f3f90 con 0x7f6d0806f730 2024-04-24T08:09:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.440+0000 7f6d0f600640 1 -- 172.21.15.69:0/1226395461 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d08068a80 con 0x7f6d0806f730 2024-04-24T08:09:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.440+0000 7f6d04c00640 1 -- 172.21.15.69:0/1226395461 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6cfc042020 con 0x7f6d0806f730 2024-04-24T08:09:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.441+0000 7f6d04c00640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ce806c920 0x7f6ce806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.441+0000 7f6d04c00640 1 -- 172.21.15.69:0/1226395461 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f6cfc0ac2a0 con 0x7f6d0806f730 2024-04-24T08:09:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.442+0000 7f6d07e00640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ce806c920 0x7f6ce806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.443+0000 7f6d07e00640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ce806c920 0x7f6ce806ede0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6d080cb310 tx=0x7f6cf4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.446+0000 7f6d04c00640 1 -- 172.21.15.69:0/1226395461 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6cfc049030 con 0x7f6d0806f730 2024-04-24T08:09:56.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.660+0000 7f6d0f600640 1 -- 172.21.15.69:0/1226395461 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6d080b5720 con 0x7f6d0806f730 2024-04-24T08:09:56.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.661+0000 7f6d04c00640 1 -- 172.21.15.69:0/1226395461 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) ==== 80+0+0 (secure 0 0 0) 0x7f6d081f3450 con 0x7f6d0806f730 2024-04-24T08:09:56.662 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:09:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.663+0000 7f6d0f600640 1 -- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ce806c920 msgr2=0x7f6ce806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.663+0000 7f6d0f600640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ce806c920 0x7f6ce806ede0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6d080cb310 tx=0x7f6cf4009040 comp rx=0 tx=0).stop 2024-04-24T08:09:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.663+0000 7f6d0f600640 1 -- 172.21.15.69:0/1226395461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0806f730 msgr2=0x7f6d081e3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.663+0000 7f6d0f600640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0806f730 0x7f6d081e3580 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc000c00 tx=0x7f6cfc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.664+0000 7f6d0f600640 1 -- 172.21.15.69:0/1226395461 shutdown_connections 2024-04-24T08:09:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.664+0000 7f6d0f600640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ce806c920 0x7f6ce806ede0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.664+0000 7f6d0f600640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6d080b3320 0x7f6d081f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.664+0000 7f6d0f600640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6d08071d70 0x7f6d08154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.664+0000 7f6d0f600640 1 --2- 172.21.15.69:0/1226395461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0806f730 0x7f6d081e3580 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.664+0000 7f6d0f600640 1 -- 172.21.15.69:0/1226395461 >> 172.21.15.69:0/1226395461 conn(0x7f6d08067d90 msgr2=0x7f6d081e7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.664+0000 7f6d0f600640 1 -- 172.21.15.69:0/1226395461 shutdown_connections 2024-04-24T08:09:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.664+0000 7f6d0f600640 1 -- 172.21.15.69:0/1226395461 wait complete. 2024-04-24T08:09:56.685 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:09:56.685 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:09:56.734 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.929+0000 7f3d09e00640 1 Processor -- start 2024-04-24T08:09:56.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.929+0000 7f3d09e00640 1 -- start start 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.930+0000 7f3d09e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3d041523f0 0x7f3d0406db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.930+0000 7f3d09e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d040aeac0 con 0x7f3d0415b980 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.930+0000 7f3d09e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d040aeda0 con 0x7f3d04072e20 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.930+0000 7f3d09e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3d040af080 con 0x7f3d041523f0 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.931+0000 7f3d08a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3d041523f0 0x7f3d0406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.931+0000 7f3d09400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3d0415b980 0x7f3d0415be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44708/0 (socket says 172.21.15.69:44708) 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.931+0000 7f3d08a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3d041523f0 0x7f3d0406db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45756/0 (socket says 172.21.15.69:45756) 2024-04-24T08:09:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.931+0000 7f3d09400640 1 -- 172.21.15.69:0/834891171 learned_addr learned my addr 172.21.15.69:0/834891171 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:56.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.931+0000 7f3d08a00640 1 -- 172.21.15.69:0/834891171 >> v1:172.21.15.121:6789/0 conn(0x7f3d04072e20 legacy=0x7f3d04073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:56.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.931+0000 7f3d08a00640 1 -- 172.21.15.69:0/834891171 >> v1:172.21.15.69:6789/0 conn(0x7f3d0415b980 legacy=0x7f3d0415be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.931+0000 7f3d08a00640 1 -- 172.21.15.69:0/834891171 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d040af360 con 0x7f3d041523f0 2024-04-24T08:09:56.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.932+0000 7f3d08a00640 1 --2- 172.21.15.69:0/834891171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d041523f0 0x7f3d0406db90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80097a0 tx=0x7f3cf802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8032fb152576e58 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.933+0000 7f3d03400640 1 -- 172.21.15.69:0/834891171 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8037070 con 0x7f3d041523f0 2024-04-24T08:09:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.933+0000 7f3d03400640 1 -- 172.21.15.69:0/834891171 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cf802ae50 con 0x7f3d041523f0 2024-04-24T08:09:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.933+0000 7f3d03400640 1 -- 172.21.15.69:0/834891171 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8032660 con 0x7f3d041523f0 2024-04-24T08:09:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.933+0000 7f3d09e00640 1 -- 172.21.15.69:0/834891171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d041523f0 msgr2=0x7f3d0406db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.933+0000 7f3d09e00640 1 --2- 172.21.15.69:0/834891171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d041523f0 0x7f3d0406db90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80097a0 tx=0x7f3cf802a140 comp rx=0 tx=0).stop 2024-04-24T08:09:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.933+0000 7f3d09e00640 1 -- 172.21.15.69:0/834891171 shutdown_connections 2024-04-24T08:09:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.933+0000 7f3d09e00640 1 --2- 172.21.15.69:0/834891171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d041523f0 0x7f3d0406db90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.934+0000 7f3d09e00640 1 -- 172.21.15.69:0/834891171 >> 172.21.15.69:0/834891171 conn(0x7f3d040d0160 msgr2=0x7f3d040d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.934+0000 7f3d09e00640 1 -- 172.21.15.69:0/834891171 shutdown_connections 2024-04-24T08:09:56.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.934+0000 7f3d09e00640 1 -- 172.21.15.69:0/834891171 wait complete. 2024-04-24T08:09:56.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.935+0000 7f3d09e00640 1 Processor -- start 2024-04-24T08:09:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.935+0000 7f3d09e00640 1 -- start start 2024-04-24T08:09:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.936+0000 7f3d09e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d04072e20 0x7f3d041569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.936+0000 7f3d09e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d041523f0 0x7f3d041514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.936+0000 7f3d09e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d0415b980 0x7f3d04158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.936+0000 7f3d09e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3d04073970 con 0x7f3d04072e20 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.936+0000 7f3d09e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d041527d0 con 0x7f3d0415b980 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.936+0000 7f3d09e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d04073af0 con 0x7f3d041523f0 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.937+0000 7f3d08a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d04072e20 0x7f3d041569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.937+0000 7f3d08a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d04072e20 0x7f3d041569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45766/0 (socket says 172.21.15.69:45766) 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.937+0000 7f3d08a00640 1 -- 172.21.15.69:0/1109986091 learned_addr learned my addr 172.21.15.69:0/1109986091 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.937+0000 7f3d08a00640 1 -- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d041523f0 msgr2=0x7f3d041514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.937+0000 7f3d08a00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d041523f0 0x7f3d041514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.937+0000 7f3d09400640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d0415b980 0x7f3d04158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.937+0000 7f3d08a00640 1 -- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d0415b980 msgr2=0x7f3d04158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.937+0000 7f3d08a00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d0415b980 0x7f3d04158400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.937+0000 7f3d08a00640 1 -- 172.21.15.69:0/1109986091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d041564d0 con 0x7f3d04072e20 2024-04-24T08:09:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.937+0000 7f3d08a00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d04072e20 0x7f3d041569d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8000c00 tx=0x7f3cf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.938+0000 7f3d00c00640 1 -- 172.21.15.69:0/1109986091 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8041070 con 0x7f3d04072e20 2024-04-24T08:09:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.938+0000 7f3d00c00640 1 -- 172.21.15.69:0/1109986091 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cf803c670 con 0x7f3d04072e20 2024-04-24T08:09:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.938+0000 7f3d00c00640 1 -- 172.21.15.69:0/1109986091 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8037040 con 0x7f3d04072e20 2024-04-24T08:09:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.938+0000 7f3d09e00640 1 -- 172.21.15.69:0/1109986091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d041e9cc0 con 0x7f3d04072e20 2024-04-24T08:09:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.938+0000 7f3d09e00640 1 -- 172.21.15.69:0/1109986091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d041e8e20 con 0x7f3d04072e20 2024-04-24T08:09:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.939+0000 7f3d09e00640 1 -- 172.21.15.69:0/1109986091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d04075d50 con 0x7f3d04072e20 2024-04-24T08:09:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.940+0000 7f3d00c00640 1 -- 172.21.15.69:0/1109986091 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3cf8044070 con 0x7f3d04072e20 2024-04-24T08:09:56.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.941+0000 7f3d00c00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ce406c8f0 0x7f3ce406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:56.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.941+0000 7f3d00c00640 1 -- 172.21.15.69:0/1109986091 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f3cf80ac200 con 0x7f3d04072e20 2024-04-24T08:09:56.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.941+0000 7f3d03e00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ce406c8f0 0x7f3ce406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:56.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.943+0000 7f3d03e00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ce406c8f0 0x7f3ce406edb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3d0409e420 tx=0x7f3cf00079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:56.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:56.945+0000 7f3d00c00640 1 -- 172.21.15.69:0/1109986091 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cf807a620 con 0x7f3d04072e20 2024-04-24T08:09:57.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.144+0000 7f3d09e00640 1 -- 172.21.15.69:0/1109986091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d040ae400 con 0x7f3d04072e20 2024-04-24T08:09:57.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.146+0000 7f3d00c00640 1 -- 172.21.15.69:0/1109986091 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16645 (secure 0 0 0) 0x7f3cf8079f20 con 0x7f3d04072e20 2024-04-24T08:09:57.148 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:09:57.148 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":33,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:40.096668+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:09:36.197195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-24T08:09:37.506730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:57.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 -- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ce406c8f0 msgr2=0x7f3ce406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ce406c8f0 0x7f3ce406edb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3d0409e420 tx=0x7f3cf00079b0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 -- 172.21.15.69:0/1109986091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d04072e20 msgr2=0x7f3d041569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d04072e20 0x7f3d041569d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8000c00 tx=0x7f3cf80046c0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 -- 172.21.15.69:0/1109986091 shutdown_connections 2024-04-24T08:09:57.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ce406c8f0 0x7f3ce406edb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d0415b980 0x7f3d04158400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d041523f0 0x7f3d041514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 --2- 172.21.15.69:0/1109986091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d04072e20 0x7f3d041569d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 -- 172.21.15.69:0/1109986091 >> 172.21.15.69:0/1109986091 conn(0x7f3d040d0160 msgr2=0x7f3d040d0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:57.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 -- 172.21.15.69:0/1109986091 shutdown_connections 2024-04-24T08:09:57.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.149+0000 7f3d09e00640 1 -- 172.21.15.69:0/1109986091 wait complete. 2024-04-24T08:09:57.170 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:09:36.197195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:09:37.506730+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:57.171 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:57.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.359+0000 7f01e9800640 1 Processor -- start 2024-04-24T08:09:57.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.359+0000 7f01e9800640 1 -- start start 2024-04-24T08:09:57.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.360+0000 7f01e9800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f01e40d1860 0x7f01e4169900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.360+0000 7f01e9800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e40d2710 con 0x7f01e415ec30 2024-04-24T08:09:57.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.360+0000 7f01e9800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e40d29f0 con 0x7f01e415e850 2024-04-24T08:09:57.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.360+0000 7f01e9800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f01e40d37b0 con 0x7f01e40d1860 2024-04-24T08:09:57.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.361+0000 7f01da000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f01e415ec30 0x7f01e40d1150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44724/0 (socket says 172.21.15.69:44724) 2024-04-24T08:09:57.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.361+0000 7f01da000640 1 -- 172.21.15.69:0/2861559859 learned_addr learned my addr 172.21.15.69:0/2861559859 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:57.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.361+0000 7f01e3400640 1 --2- 172.21.15.69:0/2861559859 >> v2:172.21.15.69:3300/0 conn(0x7f01e40d1860 0x7f01e4169900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.361+0000 7f01e3400640 1 -- 172.21.15.69:0/2861559859 >> v1:172.21.15.121:6789/0 conn(0x7f01e415e850 legacy=0x7f01e40cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:57.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.361+0000 7f01e3400640 1 -- 172.21.15.69:0/2861559859 >> v1:172.21.15.69:6789/0 conn(0x7f01e415ec30 legacy=0x7f01e40d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.361+0000 7f01e3400640 1 -- 172.21.15.69:0/2861559859 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e40d2cd0 con 0x7f01e40d1860 2024-04-24T08:09:57.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.361+0000 7f01e2000640 1 -- 172.21.15.69:0/2861559859 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3348255039 0 0) 0x7f01e40d2710 con 0x7f01e415ec30 2024-04-24T08:09:57.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.362+0000 7f01e3400640 1 --2- 172.21.15.69:0/2861559859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 0x7f01e4169900 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009d90 tx=0x7f01dc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1a2a37032dcb28 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:57.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.362+0000 7f01e2000640 1 -- 172.21.15.69:0/2861559859 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc031600 con 0x7f01e40d1860 2024-04-24T08:09:57.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.363+0000 7f01e2000640 1 -- 172.21.15.69:0/2861559859 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01dc0317a0 con 0x7f01e40d1860 2024-04-24T08:09:57.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.363+0000 7f01e2000640 1 -- 172.21.15.69:0/2861559859 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc031b70 con 0x7f01e40d1860 2024-04-24T08:09:57.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.363+0000 7f01e9800640 1 -- 172.21.15.69:0/2861559859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 msgr2=0x7f01e4169900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.363+0000 7f01e9800640 1 --2- 172.21.15.69:0/2861559859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 0x7f01e4169900 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009d90 tx=0x7f01dc02a610 comp rx=0 tx=0).stop 2024-04-24T08:09:57.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.363+0000 7f01e9800640 1 -- 172.21.15.69:0/2861559859 shutdown_connections 2024-04-24T08:09:57.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.363+0000 7f01e9800640 1 --2- 172.21.15.69:0/2861559859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 0x7f01e4169900 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.363+0000 7f01e9800640 1 -- 172.21.15.69:0/2861559859 >> 172.21.15.69:0/2861559859 conn(0x7f01e408bf60 msgr2=0x7f01e40d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:57.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.363+0000 7f01e9800640 1 -- 172.21.15.69:0/2861559859 shutdown_connections 2024-04-24T08:09:57.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.363+0000 7f01e9800640 1 -- 172.21.15.69:0/2861559859 wait complete. 2024-04-24T08:09:57.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.365+0000 7f01e9800640 1 Processor -- start 2024-04-24T08:09:57.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.365+0000 7f01e9800640 1 -- start start 2024-04-24T08:09:57.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.366+0000 7f01e9800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 0x7f01e40b9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.366+0000 7f01e9800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e415e850 0x7f01e40cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.366+0000 7f01e9800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e415ec30 0x7f01e4157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.366+0000 7f01e9800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01e40d3290 con 0x7f01e40d1860 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.366+0000 7f01e9800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f01e406eba0 con 0x7f01e415ec30 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.366+0000 7f01e9800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f01e40d0620 con 0x7f01e415e850 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01e2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 0x7f01e40b9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01e2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 0x7f01e40b9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45776/0 (socket says 172.21.15.69:45776) 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01e2a00640 1 -- 172.21.15.69:0/430141375 learned_addr learned my addr 172.21.15.69:0/430141375 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01da000640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e415e850 0x7f01e40cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01e3400640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e415ec30 0x7f01e4157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01e2a00640 1 -- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e415e850 msgr2=0x7f01e40cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01e2a00640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e415e850 0x7f01e40cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01e2a00640 1 -- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e415ec30 msgr2=0x7f01e4157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01e2a00640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e415ec30 0x7f01e4157890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01e2a00640 1 -- 172.21.15.69:0/430141375 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e40b9fc0 con 0x7f01e40d1860 2024-04-24T08:09:57.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.367+0000 7f01e2a00640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 0x7f01e40b9a80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f01d400c430 tx=0x7f01d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:57.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.368+0000 7f01db400640 1 -- 172.21.15.69:0/430141375 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4011790 con 0x7f01e40d1860 2024-04-24T08:09:57.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.368+0000 7f01db400640 1 -- 172.21.15.69:0/430141375 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01d4017070 con 0x7f01e40d1860 2024-04-24T08:09:57.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.368+0000 7f01db400640 1 -- 172.21.15.69:0/430141375 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d4011bb0 con 0x7f01e40d1860 2024-04-24T08:09:57.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.368+0000 7f01e9800640 1 -- 172.21.15.69:0/430141375 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e40ccae0 con 0x7f01e40d1860 2024-04-24T08:09:57.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.368+0000 7f01e9800640 1 -- 172.21.15.69:0/430141375 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e40cd0a0 con 0x7f01e40d1860 2024-04-24T08:09:57.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.369+0000 7f01e9800640 1 -- 172.21.15.69:0/430141375 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e4068a80 con 0x7f01e40d1860 2024-04-24T08:09:57.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.370+0000 7f01db400640 1 -- 172.21.15.69:0/430141375 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f01d401c460 con 0x7f01e40d1860 2024-04-24T08:09:57.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.371+0000 7f01db400640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f01c006c920 0x7f01c006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.371+0000 7f01db400640 1 -- 172.21.15.69:0/430141375 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f01d408ba90 con 0x7f01e40d1860 2024-04-24T08:09:57.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.371+0000 7f01da000640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f01c006c920 0x7f01c006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.373+0000 7f01da000640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f01c006c920 0x7f01c006ede0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f01e40d1830 tx=0x7f01cc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:57.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.375+0000 7f01db400640 1 -- 172.21.15.69:0/430141375 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f01d4059e20 con 0x7f01e40d1860 2024-04-24T08:09:57.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.579+0000 7f01e9800640 1 -- 172.21.15.69:0/430141375 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f01e415e1b0 con 0x7f01e40d1860 2024-04-24T08:09:57.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.580+0000 7f01db400640 1 -- 172.21.15.69:0/430141375 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) ==== 93+0+10 (secure 0 0 0) 0x7f01d4059720 con 0x7f01e40d1860 2024-04-24T08:09:57.581 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:09:57.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.582+0000 7f01e9800640 1 -- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f01c006c920 msgr2=0x7f01c006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.582+0000 7f01e9800640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f01c006c920 0x7f01c006ede0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f01e40d1830 tx=0x7f01cc009040 comp rx=0 tx=0).stop 2024-04-24T08:09:57.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.582+0000 7f01e9800640 1 -- 172.21.15.69:0/430141375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 msgr2=0x7f01e40b9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.582+0000 7f01e9800640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 0x7f01e40b9a80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f01d400c430 tx=0x7f01d400c8f0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.582+0000 7f01e9800640 1 -- 172.21.15.69:0/430141375 shutdown_connections 2024-04-24T08:09:57.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.582+0000 7f01e9800640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f01c006c920 0x7f01c006ede0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.582+0000 7f01e9800640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e415ec30 0x7f01e4157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.582+0000 7f01e9800640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e415e850 0x7f01e40cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.582+0000 7f01e9800640 1 --2- 172.21.15.69:0/430141375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01e40d1860 0x7f01e40b9a80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.582+0000 7f01e9800640 1 -- 172.21.15.69:0/430141375 >> 172.21.15.69:0/430141375 conn(0x7f01e408bf60 msgr2=0x7f01e40d45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:57.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.583+0000 7f01e9800640 1 -- 172.21.15.69:0/430141375 shutdown_connections 2024-04-24T08:09:57.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.583+0000 7f01e9800640 1 -- 172.21.15.69:0/430141375 wait complete. 2024-04-24T08:09:57.601 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:57.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.808+0000 7fdfbe800640 1 Processor -- start 2024-04-24T08:09:57.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.808+0000 7fdfbe800640 1 -- start start 2024-04-24T08:09:57.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.810+0000 7fdfbe800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb81420a0 0x7fdfb8071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.810+0000 7fdfbe800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdfb8142480 0x7fdfb815ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.810+0000 7fdfbe800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb8072450 con 0x7fdfb8160510 2024-04-24T08:09:57.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.810+0000 7fdfbe800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb8160960 con 0x7fdfb8142480 2024-04-24T08:09:57.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.810+0000 7fdfbe800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb80cf1e0 con 0x7fdfb81420a0 2024-04-24T08:09:57.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.810+0000 7fdfbde00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdfb8160510 0x7fdfb815c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44730/0 (socket says 172.21.15.69:44730) 2024-04-24T08:09:57.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.810+0000 7fdfbde00640 1 -- 172.21.15.69:0/2866940415 learned_addr learned my addr 172.21.15.69:0/2866940415 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:57.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfbca00640 1 --2- 172.21.15.69:0/2866940415 >> v2:172.21.15.121:3300/0 conn(0x7fdfb8142480 0x7fdfb815ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfbd400640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb81420a0 0x7fdfb8071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 579024801 0 0) 0x7fdfb8072450 con 0x7fdfb8160510 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfa0003520 con 0x7fdfb8160510 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1296279272 0 0) 0x7fdfa0003520 con 0x7fdfb8160510 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb81420a0 msgr2=0x7fdfb8071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb81420a0 0x7fdfb8071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 >> v2:172.21.15.121:3300/0 conn(0x7fdfb8142480 msgr2=0x7fdfb815ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 --2- 172.21.15.69:0/2866940415 >> v2:172.21.15.121:3300/0 conn(0x7fdfb8142480 0x7fdfb815ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb8072730 con 0x7fdfb8160510 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fdfb00019f0 con 0x7fdfb8160510 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 >> v1:172.21.15.69:6789/0 conn(0x7fdfb8160510 legacy=0x7fdfb815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfa00039b0 0x7fdfa0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfa0004800 0x7fdfa0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb81420a0 0x7fdfa00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb8160960 con 0x7fdfa00039b0 2024-04-24T08:09:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb80cf1e0 con 0x7fdfa0004800 2024-04-24T08:09:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.811+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa000b9c0 con 0x7fdfb81420a0 2024-04-24T08:09:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.812+0000 7fdfbd400640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfa00039b0 0x7fdfa0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.812+0000 7fdfbde00640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb81420a0 0x7fdfa00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.812+0000 7fdfbca00640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfa0004800 0x7fdfa0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.812+0000 7fdfbd400640 1 -- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb81420a0 msgr2=0x7fdfa00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.812+0000 7fdfbd400640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb81420a0 0x7fdfa00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.812+0000 7fdfbd400640 1 -- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfa0004800 msgr2=0x7fdfa0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.812+0000 7fdfbd400640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfa0004800 0x7fdfa0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.812+0000 7fdfbd400640 1 -- 172.21.15.69:0/2866940415 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa0003520 con 0x7fdfa00039b0 2024-04-24T08:09:57.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.813+0000 7fdfbd400640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfa00039b0 0x7fdfa0003dd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009ac0 tx=0x7fdfa0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14eed0104180ed65 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:57.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.814+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac036070 con 0x7fdfa00039b0 2024-04-24T08:09:57.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.814+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfac031400 con 0x7fdfa00039b0 2024-04-24T08:09:57.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.814+0000 7fdfb7e00640 1 -- 172.21.15.69:0/2866940415 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfac009690 con 0x7fdfa00039b0 2024-04-24T08:09:57.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.814+0000 7fdfbe800640 1 -- 172.21.15.69:0/2866940415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfa00039b0 msgr2=0x7fdfa0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.814+0000 7fdfbe800640 1 --2- 172.21.15.69:0/2866940415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfa00039b0 0x7fdfa0003dd0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009ac0 tx=0x7fdfa0005830 comp rx=0 tx=0).stop 2024-04-24T08:09:57.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.814+0000 7fdfbde00640 1 -- 172.21.15.69:0/2866940415 reap_dead start 2024-04-24T08:09:57.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.814+0000 7fdfbe800640 1 -- 172.21.15.69:0/2866940415 shutdown_connections 2024-04-24T08:09:57.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.814+0000 7fdfbe800640 1 -- 172.21.15.69:0/2866940415 >> 172.21.15.69:0/2866940415 conn(0x7fdfb8067b80 msgr2=0x7fdfb806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:57.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.814+0000 7fdfbe800640 1 -- 172.21.15.69:0/2866940415 shutdown_connections 2024-04-24T08:09:57.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.814+0000 7fdfbe800640 1 -- 172.21.15.69:0/2866940415 wait complete. 2024-04-24T08:09:57.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.816+0000 7fdfbe800640 1 Processor -- start 2024-04-24T08:09:57.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.816+0000 7fdfbe800640 1 -- start start 2024-04-24T08:09:57.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.817+0000 7fdfbe800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfa00039b0 0x7fdfb81525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.817+0000 7fdfbe800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb815b420 0x7fdfb815b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.817+0000 7fdfbe800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb815e2a0 0x7fdfb815e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.817+0000 7fdfbe800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb81428e0 con 0x7fdfb815b420 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.817+0000 7fdfbe800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb8072cf0 con 0x7fdfa00039b0 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.817+0000 7fdfbe800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb815ec60 con 0x7fdfb815e2a0 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb815b420 0x7fdfb815b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb815b420 0x7fdfb815b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45796/0 (socket says 172.21.15.69:45796) 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbca00640 1 -- 172.21.15.69:0/3127606650 learned_addr learned my addr 172.21.15.69:0/3127606650 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbd400640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfa00039b0 0x7fdfb81525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbde00640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb815e2a0 0x7fdfb815e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbca00640 1 -- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb815e2a0 msgr2=0x7fdfb815e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbca00640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb815e2a0 0x7fdfb815e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbca00640 1 -- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfa00039b0 msgr2=0x7fdfb81525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbca00640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfa00039b0 0x7fdfb81525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbca00640 1 -- 172.21.15.69:0/3127606650 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb8152b10 con 0x7fdfb815b420 2024-04-24T08:09:57.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.818+0000 7fdfbca00640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb815b420 0x7fdfb815b8a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40116d0 tx=0x7fdfa4011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:57.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.819+0000 7fdfb5600640 1 -- 172.21.15.69:0/3127606650 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa400f9c0 con 0x7fdfb815b420 2024-04-24T08:09:57.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.819+0000 7fdfb5600640 1 -- 172.21.15.69:0/3127606650 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa400fb60 con 0x7fdfb815b420 2024-04-24T08:09:57.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.819+0000 7fdfb5600640 1 -- 172.21.15.69:0/3127606650 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa4015460 con 0x7fdfb815b420 2024-04-24T08:09:57.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.819+0000 7fdfbe800640 1 -- 172.21.15.69:0/3127606650 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb815eeb0 con 0x7fdfb815b420 2024-04-24T08:09:57.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.819+0000 7fdfbe800640 1 -- 172.21.15.69:0/3127606650 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb8153e80 con 0x7fdfb815b420 2024-04-24T08:09:57.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.820+0000 7fdfbe800640 1 -- 172.21.15.69:0/3127606650 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb8075d50 con 0x7fdfb815b420 2024-04-24T08:09:57.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.821+0000 7fdfb5600640 1 -- 172.21.15.69:0/3127606650 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdfa4015600 con 0x7fdfb815b420 2024-04-24T08:09:57.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.822+0000 7fdfb5600640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf9806c940 0x7fdf9806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:57.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.822+0000 7fdfb5600640 1 -- 172.21.15.69:0/3127606650 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fdfa408c440 con 0x7fdfb815b420 2024-04-24T08:09:57.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.823+0000 7fdfbd400640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf9806c940 0x7fdf9806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:57.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.824+0000 7fdfbd400640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf9806c940 0x7fdf9806ee00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdfac033000 tx=0x7fdfac008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:57.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:57.827+0000 7fdfb5600640 1 -- 172.21.15.69:0/3127606650 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdfb8152b10 con 0x7fdfb815b420 2024-04-24T08:09:58.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.027+0000 7fdfbe800640 1 -- 172.21.15.69:0/3127606650 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdfb806c470 con 0x7fdfb815b420 2024-04-24T08:09:58.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.028+0000 7fdfb5600640 1 -- 172.21.15.69:0/3127606650 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) ==== 104+0+11 (secure 0 0 0) 0x7fdfa4019030 con 0x7fdfb815b420 2024-04-24T08:09:58.029 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 -- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf9806c940 msgr2=0x7fdf9806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf9806c940 0x7fdf9806ee00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdfac033000 tx=0x7fdfac008030 comp rx=0 tx=0).stop 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 -- 172.21.15.69:0/3127606650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb815b420 msgr2=0x7fdfb815b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb815b420 0x7fdfb815b8a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40116d0 tx=0x7fdfa4011b90 comp rx=0 tx=0).stop 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 -- 172.21.15.69:0/3127606650 shutdown_connections 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf9806c940 0x7fdf9806ee00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfb815b420 0x7fdfb815b8a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdfb815e2a0 0x7fdfb815e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 --2- 172.21.15.69:0/3127606650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdfa00039b0 0x7fdfb81525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 -- 172.21.15.69:0/3127606650 >> 172.21.15.69:0/3127606650 conn(0x7fdfb8067b80 msgr2=0x7fdfb806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 -- 172.21.15.69:0/3127606650 shutdown_connections 2024-04-24T08:09:58.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.030+0000 7fdfbe800640 1 -- 172.21.15.69:0/3127606650 wait complete. 2024-04-24T08:09:58.051 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:58.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.256+0000 7fab29800640 1 Processor -- start 2024-04-24T08:09:58.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.257+0000 7fab29800640 1 -- start start 2024-04-24T08:09:58.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.258+0000 7fab29800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fab2415eca0 0x7fab240d0530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.258+0000 7fab29800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fab241632e0 0x7fab2406cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab29800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab240cca70 con 0x7fab2406d190 2024-04-24T08:09:58.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab29800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fab240d1690 con 0x7fab2415eca0 2024-04-24T08:09:58.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab29800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fab2406d5c0 con 0x7fab241632e0 2024-04-24T08:09:58.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab23e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fab2415eca0 0x7fab240d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab28e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fab2406d190 0x7fab240d23b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44732/0 (socket says 172.21.15.69:44732) 2024-04-24T08:09:58.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab28e00640 1 -- 172.21.15.69:0/1596546541 learned_addr learned my addr 172.21.15.69:0/1596546541 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:58.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab23400640 1 --2- 172.21.15.69:0/1596546541 >> v2:172.21.15.121:3300/0 conn(0x7fab241632e0 0x7fab2406cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab22a00640 1 -- 172.21.15.69:0/1596546541 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2525508349 0 0) 0x7fab240cca70 con 0x7fab2406d190 2024-04-24T08:09:58.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab22a00640 1 -- 172.21.15.69:0/1596546541 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab08003520 con 0x7fab2406d190 2024-04-24T08:09:58.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab23e00640 1 -- 172.21.15.69:0/1596546541 >> v2:172.21.15.121:3300/0 conn(0x7fab241632e0 msgr2=0x7fab2406cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:58.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.259+0000 7fab23e00640 1 --2- 172.21.15.69:0/1596546541 >> v2:172.21.15.121:3300/0 conn(0x7fab241632e0 0x7fab2406cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.260+0000 7fab23e00640 1 -- 172.21.15.69:0/1596546541 >> v1:172.21.15.69:6789/0 conn(0x7fab2406d190 legacy=0x7fab240d23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.260+0000 7fab23e00640 1 -- 172.21.15.69:0/1596546541 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab240d0b10 con 0x7fab2415eca0 2024-04-24T08:09:58.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.261+0000 7fab23e00640 1 --2- 172.21.15.69:0/1596546541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2415eca0 0x7fab240d0530 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fab14009d90 tx=0x7fab1402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4f35678139c622 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:58.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.261+0000 7fab22a00640 1 -- 172.21.15.69:0/1596546541 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14007740 con 0x7fab2415eca0 2024-04-24T08:09:58.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.261+0000 7fab22a00640 1 -- 172.21.15.69:0/1596546541 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab140078e0 con 0x7fab2415eca0 2024-04-24T08:09:58.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.261+0000 7fab22a00640 1 -- 172.21.15.69:0/1596546541 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14007c90 con 0x7fab2415eca0 2024-04-24T08:09:58.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.262+0000 7fab29800640 1 -- 172.21.15.69:0/1596546541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2415eca0 msgr2=0x7fab240d0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.262+0000 7fab29800640 1 --2- 172.21.15.69:0/1596546541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2415eca0 0x7fab240d0530 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fab14009d90 tx=0x7fab1402a6a0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.262+0000 7fab29800640 1 -- 172.21.15.69:0/1596546541 shutdown_connections 2024-04-24T08:09:58.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.262+0000 7fab29800640 1 --2- 172.21.15.69:0/1596546541 >> v2:172.21.15.121:3300/0 conn(0x7fab241632e0 0x7fab2406cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.262+0000 7fab29800640 1 --2- 172.21.15.69:0/1596546541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2415eca0 0x7fab240d0530 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.262+0000 7fab29800640 1 -- 172.21.15.69:0/1596546541 >> 172.21.15.69:0/1596546541 conn(0x7fab24067760 msgr2=0x7fab24067b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:58.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.262+0000 7fab29800640 1 -- 172.21.15.69:0/1596546541 shutdown_connections 2024-04-24T08:09:58.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.262+0000 7fab29800640 1 -- 172.21.15.69:0/1596546541 wait complete. 2024-04-24T08:09:58.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.264+0000 7fab29800640 1 Processor -- start 2024-04-24T08:09:58.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.264+0000 7fab29800640 1 -- start start 2024-04-24T08:09:58.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab29800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2406d190 0x7fab2414d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab29800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab2415eca0 0x7fab2415b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab29800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab241632e0 0x7fab2415c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab29800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab240d13b0 con 0x7fab2406d190 2024-04-24T08:09:58.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab29800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fab240d10d0 con 0x7fab241632e0 2024-04-24T08:09:58.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab29800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fab24163750 con 0x7fab2415eca0 2024-04-24T08:09:58.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab23e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2406d190 0x7fab2414d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab23e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2406d190 0x7fab2414d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45814/0 (socket says 172.21.15.69:45814) 2024-04-24T08:09:58.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab23e00640 1 -- 172.21.15.69:0/1548018289 learned_addr learned my addr 172.21.15.69:0/1548018289 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:58.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab23400640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab2415eca0 0x7fab2415b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab28e00640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab241632e0 0x7fab2415c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab23e00640 1 -- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab2415eca0 msgr2=0x7fab2415b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab23e00640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab2415eca0 0x7fab2415b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab23e00640 1 -- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab241632e0 msgr2=0x7fab2415c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab23e00640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab241632e0 0x7fab2415c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab23e00640 1 -- 172.21.15.69:0/1548018289 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2415bad0 con 0x7fab2406d190 2024-04-24T08:09:58.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.265+0000 7fab23400640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab2415eca0 0x7fab2415b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:58.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.266+0000 7fab23e00640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2406d190 0x7fab2414d870 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fab1402ac90 tx=0x7fab14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:58.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.266+0000 7fab1be00640 1 -- 172.21.15.69:0/1548018289 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14040070 con 0x7fab2406d190 2024-04-24T08:09:58.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.266+0000 7fab1be00640 1 -- 172.21.15.69:0/1548018289 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab14031c40 con 0x7fab2406d190 2024-04-24T08:09:58.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.266+0000 7fab1be00640 1 -- 172.21.15.69:0/1548018289 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14009450 con 0x7fab2406d190 2024-04-24T08:09:58.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.266+0000 7fab29800640 1 -- 172.21.15.69:0/1548018289 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab2415bdb0 con 0x7fab2406d190 2024-04-24T08:09:58.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.266+0000 7fab29800640 1 -- 172.21.15.69:0/1548018289 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab2415c370 con 0x7fab2406d190 2024-04-24T08:09:58.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.267+0000 7fab29800640 1 -- 172.21.15.69:0/1548018289 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab24075d50 con 0x7fab2406d190 2024-04-24T08:09:58.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.268+0000 7fab1be00640 1 -- 172.21.15.69:0/1548018289 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fab14039040 con 0x7fab2406d190 2024-04-24T08:09:58.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.269+0000 7fab1be00640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab0006c8f0 0x7fab0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.269+0000 7fab1be00640 1 -- 172.21.15.69:0/1548018289 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fab140ad400 con 0x7fab2406d190 2024-04-24T08:09:58.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.270+0000 7fab23400640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab0006c8f0 0x7fab0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.271+0000 7fab23400640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab0006c8f0 0x7fab0006edb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fab0c006fd0 tx=0x7fab0c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:58.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.274+0000 7fab1be00640 1 -- 172.21.15.69:0/1548018289 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fab1407b820 con 0x7fab2406d190 2024-04-24T08:09:58.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.472+0000 7fab29800640 1 -- 172.21.15.69:0/1548018289 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fab2406cde0 con 0x7fab2406d190 2024-04-24T08:09:58.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.473+0000 7fab1be00640 1 -- 172.21.15.69:0/1548018289 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) ==== 100+0+11 (secure 0 0 0) 0x7fab1407b120 con 0x7fab2406d190 2024-04-24T08:09:58.474 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:09:58.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.475+0000 7fab29800640 1 -- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab0006c8f0 msgr2=0x7fab0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.475+0000 7fab29800640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab0006c8f0 0x7fab0006edb0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fab0c006fd0 tx=0x7fab0c008040 comp rx=0 tx=0).stop 2024-04-24T08:09:58.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.476+0000 7fab29800640 1 -- 172.21.15.69:0/1548018289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2406d190 msgr2=0x7fab2414d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.476+0000 7fab29800640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2406d190 0x7fab2414d870 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fab1402ac90 tx=0x7fab14004a40 comp rx=0 tx=0).stop 2024-04-24T08:09:58.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.476+0000 7fab29800640 1 -- 172.21.15.69:0/1548018289 shutdown_connections 2024-04-24T08:09:58.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.476+0000 7fab29800640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab0006c8f0 0x7fab0006edb0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.476+0000 7fab29800640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab241632e0 0x7fab2415c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.476+0000 7fab29800640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab2415eca0 0x7fab2415b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.476+0000 7fab29800640 1 --2- 172.21.15.69:0/1548018289 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab2406d190 0x7fab2414d870 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.476+0000 7fab29800640 1 -- 172.21.15.69:0/1548018289 >> 172.21.15.69:0/1548018289 conn(0x7fab24067760 msgr2=0x7fab240b7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:58.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.476+0000 7fab29800640 1 -- 172.21.15.69:0/1548018289 shutdown_connections 2024-04-24T08:09:58.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.476+0000 7fab29800640 1 -- 172.21.15.69:0/1548018289 wait complete. 2024-04-24T08:09:58.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:09:58.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.694+0000 7fd611000640 1 Processor -- start 2024-04-24T08:09:58.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.695+0000 7fd611000640 1 -- start start 2024-04-24T08:09:58.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.696+0000 7fd611000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c15b490 0x7fd60c0b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.696+0000 7fd611000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd60c0b46e0 con 0x7fd60c0d02f0 2024-04-24T08:09:58.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.696+0000 7fd611000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd60c0d0960 con 0x7fd60c06f2d0 2024-04-24T08:09:58.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.696+0000 7fd611000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd60c0b4980 con 0x7fd60c15b490 2024-04-24T08:09:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.697+0000 7fd60a800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd60c0d02f0 0x7fd60c0ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:44742/0 (socket says 172.21.15.69:44742) 2024-04-24T08:09:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.697+0000 7fd60a800640 1 -- 172.21.15.69:0/3324751009 learned_addr learned my addr 172.21.15.69:0/3324751009 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:09:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.697+0000 7fd609e00640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c15b490 0x7fd60c0b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.697+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4243927228 0 0) 0x7fd60c0b46e0 con 0x7fd60c0d02f0 2024-04-24T08:09:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.697+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5f0003520 con 0x7fd60c0d02f0 2024-04-24T08:09:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 760942914 0 0) 0x7fd5f0003520 con 0x7fd60c0d02f0 2024-04-24T08:09:58.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c15b490 msgr2=0x7fd60c0b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c15b490 0x7fd60c0b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 >> v1:172.21.15.121:6789/0 conn(0x7fd60c06f2d0 legacy=0x7fd60c0cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60c0b4c60 con 0x7fd60c0d02f0 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fd604001670 con 0x7fd60c0d02f0 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 >> v1:172.21.15.69:6789/0 conn(0x7fd60c0d02f0 legacy=0x7fd60c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 0x7fd5f0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c15b490 0x7fd5f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5f0004d40 0x7fd5f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c0b4980 con 0x7fd5f00039e0 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f000baa0 con 0x7fd5f0004d40 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f000bca0 con 0x7fd60c15b490 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.698+0000 7fd609e00640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 0x7fd5f0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.699+0000 7fd609e00640 1 -- 172.21.15.69:0/3324751009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c15b490 msgr2=0x7fd5f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:58.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.699+0000 7fd609e00640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c15b490 0x7fd5f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.699+0000 7fd60a800640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5f0004d40 0x7fd5f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.699+0000 7fd609e00640 1 -- 172.21.15.69:0/3324751009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5f0004d40 msgr2=0x7fd5f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:58.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.699+0000 7fd609e00640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd5f0004d40 0x7fd5f00051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.699+0000 7fd609e00640 1 -- 172.21.15.69:0/3324751009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f0003520 con 0x7fd5f00039e0 2024-04-24T08:09:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.700+0000 7fd609e00640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 0x7fd5f0003e00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009f90 tx=0x7fd5f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f31e63fed64ced server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.700+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc009c10 con 0x7fd5f00039e0 2024-04-24T08:09:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.700+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5fc036070 con 0x7fd5f00039e0 2024-04-24T08:09:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.700+0000 7fd608a00640 1 -- 172.21.15.69:0/3324751009 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc031880 con 0x7fd5f00039e0 2024-04-24T08:09:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.701+0000 7fd611000640 1 -- 172.21.15.69:0/3324751009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 msgr2=0x7fd5f0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.701+0000 7fd611000640 1 --2- 172.21.15.69:0/3324751009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 0x7fd5f0003e00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc009f90 tx=0x7fd5f0005830 comp rx=0 tx=0).stop 2024-04-24T08:09:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.701+0000 7fd60a800640 1 -- 172.21.15.69:0/3324751009 reap_dead start 2024-04-24T08:09:58.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.701+0000 7fd611000640 1 -- 172.21.15.69:0/3324751009 shutdown_connections 2024-04-24T08:09:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.701+0000 7fd611000640 1 -- 172.21.15.69:0/3324751009 >> 172.21.15.69:0/3324751009 conn(0x7fd60c067d90 msgr2=0x7fd60c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.701+0000 7fd611000640 1 -- 172.21.15.69:0/3324751009 shutdown_connections 2024-04-24T08:09:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.701+0000 7fd611000640 1 -- 172.21.15.69:0/3324751009 wait complete. 2024-04-24T08:09:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.703+0000 7fd611000640 1 Processor -- start 2024-04-24T08:09:58.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.703+0000 7fd611000640 1 -- start start 2024-04-24T08:09:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd611000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 0x7fd60c14ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd611000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c0ab2b0 0x7fd60c15dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd611000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd60c15e260 0x7fd60c0a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd611000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c0ce7e0 con 0x7fd5f00039e0 2024-04-24T08:09:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd611000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd60c15b870 con 0x7fd60c15e260 2024-04-24T08:09:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd611000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd60c0ce960 con 0x7fd60c0ab2b0 2024-04-24T08:09:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd609e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 0x7fd60c14ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd609e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 0x7fd60c14ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45836/0 (socket says 172.21.15.69:45836) 2024-04-24T08:09:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd609e00640 1 -- 172.21.15.69:0/754379929 learned_addr learned my addr 172.21.15.69:0/754379929 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:09:58.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd60a800640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd60c15e260 0x7fd60c0a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd609400640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c0ab2b0 0x7fd60c15dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd609e00640 1 -- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c0ab2b0 msgr2=0x7fd60c15dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.704+0000 7fd609e00640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c0ab2b0 0x7fd60c15dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.705+0000 7fd609e00640 1 -- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd60c15e260 msgr2=0x7fd60c0a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.705+0000 7fd609e00640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd60c15e260 0x7fd60c0a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.705+0000 7fd609e00640 1 -- 172.21.15.69:0/754379929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60c0aa210 con 0x7fd5f00039e0 2024-04-24T08:09:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.705+0000 7fd60a800640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd60c15e260 0x7fd60c0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:58.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.705+0000 7fd609e00640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 0x7fd60c14ae40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc02aee0 tx=0x7fd5fc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:58.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.705+0000 7fd602000640 1 -- 172.21.15.69:0/754379929 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc009c10 con 0x7fd5f00039e0 2024-04-24T08:09:58.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.705+0000 7fd602000640 1 -- 172.21.15.69:0/754379929 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5fc040070 con 0x7fd5f00039e0 2024-04-24T08:09:58.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.705+0000 7fd602000640 1 -- 172.21.15.69:0/754379929 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0458b0 con 0x7fd5f00039e0 2024-04-24T08:09:58.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.705+0000 7fd611000640 1 -- 172.21.15.69:0/754379929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd60c0aa4f0 con 0x7fd5f00039e0 2024-04-24T08:09:58.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.705+0000 7fd611000640 1 -- 172.21.15.69:0/754379929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd60c0aaab0 con 0x7fd5f00039e0 2024-04-24T08:09:58.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.707+0000 7fd611000640 1 -- 172.21.15.69:0/754379929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd60c068a80 con 0x7fd5f00039e0 2024-04-24T08:09:58.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.707+0000 7fd602000640 1 -- 172.21.15.69:0/754379929 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd5fc005d10 con 0x7fd5f00039e0 2024-04-24T08:09:58.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.708+0000 7fd602000640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd5ec06c8f0 0x7fd5ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.708+0000 7fd602000640 1 -- 172.21.15.69:0/754379929 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fd5fc0acf60 con 0x7fd5f00039e0 2024-04-24T08:09:58.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.709+0000 7fd609400640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd5ec06c8f0 0x7fd5ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.710+0000 7fd609400640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd5ec06c8f0 0x7fd5ec06edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd60c15ec10 tx=0x7fd5f8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:58.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.713+0000 7fd602000640 1 -- 172.21.15.69:0/754379929 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd5fc07b380 con 0x7fd5f00039e0 2024-04-24T08:09:58.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.922+0000 7fd611000640 1 -- 172.21.15.69:0/754379929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd60c0b1370 con 0x7fd5f00039e0 2024-04-24T08:09:58.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.923+0000 7fd602000640 1 -- 172.21.15.69:0/754379929 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) ==== 80+0+0 (secure 0 0 0) 0x7fd5fc07ac80 con 0x7fd5f00039e0 2024-04-24T08:09:58.924 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.925+0000 7fd611000640 1 -- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd5ec06c8f0 msgr2=0x7fd5ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.925+0000 7fd611000640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd5ec06c8f0 0x7fd5ec06edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd60c15ec10 tx=0x7fd5f8009040 comp rx=0 tx=0).stop 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.925+0000 7fd611000640 1 -- 172.21.15.69:0/754379929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 msgr2=0x7fd60c14ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.925+0000 7fd611000640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 0x7fd60c14ae40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc02aee0 tx=0x7fd5fc009750 comp rx=0 tx=0).stop 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.925+0000 7fd611000640 1 -- 172.21.15.69:0/754379929 shutdown_connections 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.925+0000 7fd611000640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd5ec06c8f0 0x7fd5ec06edb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.925+0000 7fd611000640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd60c15e260 0x7fd60c0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.925+0000 7fd611000640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd60c0ab2b0 0x7fd60c15dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.925+0000 7fd611000640 1 --2- 172.21.15.69:0/754379929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5f00039e0 0x7fd60c14ae40 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.925+0000 7fd611000640 1 -- 172.21.15.69:0/754379929 >> 172.21.15.69:0/754379929 conn(0x7fd60c067d90 msgr2=0x7fd60c0b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.926+0000 7fd611000640 1 -- 172.21.15.69:0/754379929 shutdown_connections 2024-04-24T08:09:58.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:09:58.926+0000 7fd611000640 1 -- 172.21.15.69:0/754379929 wait complete. 2024-04-24T08:09:58.945 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:09:58.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:09:58.945 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:09:59.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[47460]: fuse finished with error 0 and tester_r 0 2024-04-24T08:09:59.026 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:10:05.128 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:10:05.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:05.128 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:10:05.187 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:05.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.390+0000 7ff113800640 1 Processor -- start 2024-04-24T08:10:05.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.390+0000 7ff113800640 1 -- start start 2024-04-24T08:10:05.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.392+0000 7ff113800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c0d0300 0x7ff10c06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.392+0000 7ff113800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff10c145a90 0x7ff10c0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.392+0000 7ff113800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff10c0d0c20 con 0x7ff10c145fe0 2024-04-24T08:10:05.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.392+0000 7ff113800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff10c0cf3d0 con 0x7ff10c145a90 2024-04-24T08:10:05.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.392+0000 7ff113800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff10c0cf6b0 con 0x7ff10c0d0300 2024-04-24T08:10:05.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.393+0000 7ff103e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff10c145a90 0x7ff10c0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.393+0000 7ff110e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c0d0300 0x7ff10c06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.393+0000 7ff103e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff10c145a90 0x7ff10c0d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:37124/0 (socket says 172.21.15.69:37124) 2024-04-24T08:10:05.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.393+0000 7ff103e00640 1 -- 172.21.15.69:0/3276806126 learned_addr learned my addr 172.21.15.69:0/3276806126 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:05.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.393+0000 7ff103e00640 1 -- 172.21.15.69:0/3276806126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c0d0300 msgr2=0x7ff10c06e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:05.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.393+0000 7ff103e00640 1 --2- 172.21.15.69:0/3276806126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c0d0300 0x7ff10c06e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.393+0000 7ff103e00640 1 -- 172.21.15.69:0/3276806126 >> v1:172.21.15.121:6789/0 conn(0x7ff10c145fe0 legacy=0x7ff10c160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.393+0000 7ff103e00640 1 -- 172.21.15.69:0/3276806126 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10c0cf990 con 0x7ff10c145a90 2024-04-24T08:10:05.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.393+0000 7ff110e00640 1 --2- 172.21.15.69:0/3276806126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c0d0300 0x7ff10c06e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:05.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.395+0000 7ff103e00640 1 --2- 172.21.15.69:0/3276806126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c145a90 0x7ff10c0d06e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009870 tx=0x7ff0f402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc7740aa8093c5e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:05.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.395+0000 7ff10be00640 1 -- 172.21.15.69:0/3276806126 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4037070 con 0x7ff10c145a90 2024-04-24T08:10:05.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.395+0000 7ff10be00640 1 -- 172.21.15.69:0/3276806126 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f4032440 con 0x7ff10c145a90 2024-04-24T08:10:05.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.395+0000 7ff10be00640 1 -- 172.21.15.69:0/3276806126 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4032810 con 0x7ff10c145a90 2024-04-24T08:10:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.396+0000 7ff113800640 1 -- 172.21.15.69:0/3276806126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c145a90 msgr2=0x7ff10c0d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.396+0000 7ff113800640 1 --2- 172.21.15.69:0/3276806126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c145a90 0x7ff10c0d06e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4009870 tx=0x7ff0f402a250 comp rx=0 tx=0).stop 2024-04-24T08:10:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.396+0000 7ff113800640 1 -- 172.21.15.69:0/3276806126 shutdown_connections 2024-04-24T08:10:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.396+0000 7ff113800640 1 --2- 172.21.15.69:0/3276806126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c145a90 0x7ff10c0d06e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.396+0000 7ff113800640 1 --2- 172.21.15.69:0/3276806126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c0d0300 0x7ff10c06e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.396+0000 7ff113800640 1 -- 172.21.15.69:0/3276806126 >> 172.21.15.69:0/3276806126 conn(0x7ff10c067d90 msgr2=0x7ff10c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.396+0000 7ff113800640 1 -- 172.21.15.69:0/3276806126 shutdown_connections 2024-04-24T08:10:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.396+0000 7ff113800640 1 -- 172.21.15.69:0/3276806126 wait complete. 2024-04-24T08:10:05.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.398+0000 7ff113800640 1 Processor -- start 2024-04-24T08:10:05.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.398+0000 7ff113800640 1 -- start start 2024-04-24T08:10:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.399+0000 7ff113800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff10c0d0300 0x7ff10c15c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.399+0000 7ff113800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c145a90 0x7ff10c142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.399+0000 7ff113800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c145fe0 0x7ff10c0c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.399+0000 7ff113800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c145770 con 0x7ff10c0d0300 2024-04-24T08:10:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.399+0000 7ff113800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff10c0cff50 con 0x7ff10c145fe0 2024-04-24T08:10:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.399+0000 7ff113800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff10c1458f0 con 0x7ff10c145a90 2024-04-24T08:10:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.399+0000 7ff110e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff10c0d0300 0x7ff10c15c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.399+0000 7ff110e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff10c0d0300 0x7ff10c15c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53114/0 (socket says 172.21.15.69:53114) 2024-04-24T08:10:05.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.399+0000 7ff110e00640 1 -- 172.21.15.69:0/628748698 learned_addr learned my addr 172.21.15.69:0/628748698 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.399+0000 7ff111800640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c145fe0 0x7ff10c0c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff103e00640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c145a90 0x7ff10c142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff110e00640 1 -- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c145a90 msgr2=0x7ff10c142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff110e00640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c145a90 0x7ff10c142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff110e00640 1 -- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c145fe0 msgr2=0x7ff10c0c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff110e00640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c145fe0 0x7ff10c0c5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff110e00640 1 -- 172.21.15.69:0/628748698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff10c15c850 con 0x7ff10c0d0300 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff103e00640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c145a90 0x7ff10c142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff110e00640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff10c0d0300 0x7ff10c15c310 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00e990 tx=0x7ff0fc00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff109600640 1 -- 172.21.15.69:0/628748698 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc017070 con 0x7ff10c0d0300 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff109600640 1 -- 172.21.15.69:0/628748698 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0fc00c9e0 con 0x7ff10c0d0300 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff109600640 1 -- 172.21.15.69:0/628748698 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc00cdb0 con 0x7ff10c0d0300 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff113800640 1 -- 172.21.15.69:0/628748698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff10c15fbc0 con 0x7ff10c0d0300 2024-04-24T08:10:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.400+0000 7ff113800640 1 -- 172.21.15.69:0/628748698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff10c15ee00 con 0x7ff10c0d0300 2024-04-24T08:10:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.401+0000 7ff113800640 1 -- 172.21.15.69:0/628748698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff10c068a80 con 0x7ff10c0d0300 2024-04-24T08:10:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.403+0000 7ff109600640 1 -- 172.21.15.69:0/628748698 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0fc01a070 con 0x7ff10c0d0300 2024-04-24T08:10:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.403+0000 7ff109600640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0e806c8f0 0x7ff0e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.404+0000 7ff109600640 1 -- 172.21.15.69:0/628748698 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7ff0fc08ba90 con 0x7ff10c0d0300 2024-04-24T08:10:05.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.404+0000 7ff103e00640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0e806c8f0 0x7ff0e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.405+0000 7ff103e00640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0e806c8f0 0x7ff0e806edb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4034000 tx=0x7ff0f402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:05.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.408+0000 7ff109600640 1 -- 172.21.15.69:0/628748698 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff0fc059ec0 con 0x7ff10c0d0300 2024-04-24T08:10:05.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.651+0000 7ff113800640 1 -- 172.21.15.69:0/628748698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff10c0d1120 con 0x7ff10c0d0300 2024-04-24T08:10:05.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.651+0000 7ff109600640 1 -- 172.21.15.69:0/628748698 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5181 (secure 0 0 0) 0x7ff0fc0597c0 con 0x7ff10c0d0300 2024-04-24T08:10:05.653 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:05.653 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2295273533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2295273533},{"type":"v1","addr":"172.21.15.121:6836","nonce":2295273533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1461471292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1461471292},{"type":"v1","addr":"172.21.15.121:6837","nonce":1461471292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/626688145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":626688145},{"type":"v1","addr":"172.21.15.69:6833","nonce":626688145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:09:38.549277+0000","modified":"2024-04-24T08:09:41.567377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/991803145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":991803145},{"type":"v1","addr":"172.21.15.69:6835","nonce":991803145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5241,"qdb_cluster":[5241]},"id":2}]} 2024-04-24T08:10:05.653 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 16 2024-04-24T08:10:05.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 -- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0e806c8f0 msgr2=0x7ff0e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0e806c8f0 0x7ff0e806edb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4034000 tx=0x7ff0f402e000 comp rx=0 tx=0).stop 2024-04-24T08:10:05.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 -- 172.21.15.69:0/628748698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff10c0d0300 msgr2=0x7ff10c15c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff10c0d0300 0x7ff10c15c310 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00e990 tx=0x7ff0fc00ee50 comp rx=0 tx=0).stop 2024-04-24T08:10:05.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 -- 172.21.15.69:0/628748698 shutdown_connections 2024-04-24T08:10:05.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0e806c8f0 0x7ff0e806edb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff10c145fe0 0x7ff10c0c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff10c145a90 0x7ff10c142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 --2- 172.21.15.69:0/628748698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff10c0d0300 0x7ff10c15c310 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 -- 172.21.15.69:0/628748698 >> 172.21.15.69:0/628748698 conn(0x7ff10c067d90 msgr2=0x7ff10c0b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:05.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 -- 172.21.15.69:0/628748698 shutdown_connections 2024-04-24T08:10:05.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:05.654+0000 7ff113800640 1 -- 172.21.15.69:0/628748698 wait complete. 2024-04-24T08:10:05.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-04-24T08:10:05.892 INFO:teuthology.orchestra.run.smithi069.stdout:{"message":"","return_code":0} 2024-04-24T08:10:05.892 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-24T08:10:05.892 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.078+0000 7f5e15200640 1 Processor -- start 2024-04-24T08:10:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.078+0000 7f5e15200640 1 -- start start 2024-04-24T08:10:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.080+0000 7f5e15200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5e1015e960 0x7f5e10162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.080+0000 7f5e15200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e10162980 0x7f5e1006cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.080+0000 7f5e15200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e1015ed40 con 0x7f5e1006d4f0 2024-04-24T08:10:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.080+0000 7f5e15200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5e100cf050 con 0x7f5e1015e960 2024-04-24T08:10:06.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.080+0000 7f5e15200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5e10162e20 con 0x7f5e10162980 2024-04-24T08:10:06.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.080+0000 7f5e0f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5e1015e960 0x7f5e10162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.080+0000 7f5e0fe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5e1006d4f0 0x7f5e100cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48054/0 (socket says 172.21.15.69:48054) 2024-04-24T08:10:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.080+0000 7f5e0fe00640 1 -- 172.21.15.69:0/87914676 learned_addr learned my addr 172.21.15.69:0/87914676 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.081+0000 7f5e07e00640 1 --2- 172.21.15.69:0/87914676 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e10162980 0x7f5e1006cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.081+0000 7f5e0f400640 1 -- 172.21.15.69:0/87914676 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e10162980 msgr2=0x7f5e1006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.081+0000 7f5e0f400640 1 --2- 172.21.15.69:0/87914676 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e10162980 0x7f5e1006cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.081+0000 7f5e0f400640 1 -- 172.21.15.69:0/87914676 >> v1:172.21.15.69:6789/0 conn(0x7f5e1006d4f0 legacy=0x7f5e100cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.081+0000 7f5e0f400640 1 -- 172.21.15.69:0/87914676 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e10163100 con 0x7f5e1015e960 2024-04-24T08:10:06.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.081+0000 7f5e0ea00640 1 -- 172.21.15.69:0/87914676 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506354904 0 0) 0x7f5e1015ed40 con 0x7f5e1006d4f0 2024-04-24T08:10:06.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.082+0000 7f5e0f400640 1 --2- 172.21.15.69:0/87914676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1015e960 0x7f5e10162440 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5e000097a0 tx=0x7f5e0002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df92b305b207478e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:06.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.082+0000 7f5e0ea00640 1 -- 172.21.15.69:0/87914676 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00037070 con 0x7f5e1015e960 2024-04-24T08:10:06.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.082+0000 7f5e0ea00640 1 -- 172.21.15.69:0/87914676 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e00032440 con 0x7f5e1015e960 2024-04-24T08:10:06.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.082+0000 7f5e0ea00640 1 -- 172.21.15.69:0/87914676 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00032810 con 0x7f5e1015e960 2024-04-24T08:10:06.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.083+0000 7f5e15200640 1 -- 172.21.15.69:0/87914676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1015e960 msgr2=0x7f5e10162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.083+0000 7f5e15200640 1 --2- 172.21.15.69:0/87914676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1015e960 0x7f5e10162440 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5e000097a0 tx=0x7f5e0002a180 comp rx=0 tx=0).stop 2024-04-24T08:10:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.083+0000 7f5e15200640 1 -- 172.21.15.69:0/87914676 shutdown_connections 2024-04-24T08:10:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.083+0000 7f5e15200640 1 --2- 172.21.15.69:0/87914676 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e10162980 0x7f5e1006cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.083+0000 7f5e15200640 1 --2- 172.21.15.69:0/87914676 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1015e960 0x7f5e10162440 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.083+0000 7f5e15200640 1 -- 172.21.15.69:0/87914676 >> 172.21.15.69:0/87914676 conn(0x7f5e1009d900 msgr2=0x7f5e1008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.083+0000 7f5e15200640 1 -- 172.21.15.69:0/87914676 shutdown_connections 2024-04-24T08:10:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.083+0000 7f5e15200640 1 -- 172.21.15.69:0/87914676 wait complete. 2024-04-24T08:10:06.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.084+0000 7f5e15200640 1 Processor -- start 2024-04-24T08:10:06.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.085+0000 7f5e15200640 1 -- start start 2024-04-24T08:10:06.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.085+0000 7f5e15200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1006d4f0 0x7f5e101616c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.085+0000 7f5e15200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e1015e960 0x7f5e1015ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.085+0000 7f5e15200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e10162980 0x7f5e1015f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.085+0000 7f5e15200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5e100b6170 con 0x7f5e1006d4f0 2024-04-24T08:10:06.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.085+0000 7f5e15200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5e100cf5d0 con 0x7f5e10162980 2024-04-24T08:10:06.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.085+0000 7f5e15200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5e100b62f0 con 0x7f5e1015e960 2024-04-24T08:10:06.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.086+0000 7f5e0f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1006d4f0 0x7f5e101616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.086+0000 7f5e0f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1006d4f0 0x7f5e101616c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53140/0 (socket says 172.21.15.69:53140) 2024-04-24T08:10:06.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.086+0000 7f5e0f400640 1 -- 172.21.15.69:0/3282640793 learned_addr learned my addr 172.21.15.69:0/3282640793 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:06.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.086+0000 7f5e0f400640 1 -- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e1015e960 msgr2=0x7f5e1015ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.086+0000 7f5e0f400640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e1015e960 0x7f5e1015ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.086+0000 7f5e0f400640 1 -- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e10162980 msgr2=0x7f5e1015f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.086+0000 7f5e0f400640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e10162980 0x7f5e1015f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.086+0000 7f5e0f400640 1 -- 172.21.15.69:0/3282640793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e100b7e10 con 0x7f5e1006d4f0 2024-04-24T08:10:06.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.086+0000 7f5e0f400640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1006d4f0 0x7f5e101616c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5e00035040 tx=0x7f5e000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:06.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.087+0000 7f5e07400640 1 -- 172.21.15.69:0/3282640793 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00037050 con 0x7f5e1006d4f0 2024-04-24T08:10:06.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.087+0000 7f5e07400640 1 -- 172.21.15.69:0/3282640793 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e00038040 con 0x7f5e1006d4f0 2024-04-24T08:10:06.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.087+0000 7f5e07400640 1 -- 172.21.15.69:0/3282640793 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e00004470 con 0x7f5e1006d4f0 2024-04-24T08:10:06.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.087+0000 7f5e15200640 1 -- 172.21.15.69:0/3282640793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e100b80b0 con 0x7f5e1006d4f0 2024-04-24T08:10:06.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.087+0000 7f5e15200640 1 -- 172.21.15.69:0/3282640793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e100b8670 con 0x7f5e1006d4f0 2024-04-24T08:10:06.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.088+0000 7f5e15200640 1 -- 172.21.15.69:0/3282640793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e10075d50 con 0x7f5e1006d4f0 2024-04-24T08:10:06.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.089+0000 7f5e07400640 1 -- 172.21.15.69:0/3282640793 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5e00030900 con 0x7f5e1006d4f0 2024-04-24T08:10:06.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.090+0000 7f5e07400640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5dec06c8f0 0x7f5dec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.090+0000 7f5e07400640 1 -- 172.21.15.69:0/3282640793 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f5e000ac670 con 0x7f5e1006d4f0 2024-04-24T08:10:06.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.091+0000 7f5e07e00640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5dec06c8f0 0x7f5dec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.092+0000 7f5e07e00640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5dec06c8f0 0x7f5dec06edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5df8006fd0 tx=0x7f5df8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:06.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.094+0000 7f5e07400640 1 -- 172.21.15.69:0/3282640793 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5e0007aa90 con 0x7f5e1006d4f0 2024-04-24T08:10:06.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.335+0000 7f5e15200640 1 -- 172.21.15.69:0/3282640793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e1015c8d0 con 0x7f5e1006d4f0 2024-04-24T08:10:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.336+0000 7f5e07400640 1 -- 172.21.15.69:0/3282640793 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5181 (secure 0 0 0) 0x7f5e0007a390 con 0x7f5e1006d4f0 2024-04-24T08:10:06.337 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:06.337 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2295273533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2295273533},{"type":"v1","addr":"172.21.15.121:6836","nonce":2295273533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1461471292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1461471292},{"type":"v1","addr":"172.21.15.121:6837","nonce":1461471292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/626688145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":626688145},{"type":"v1","addr":"172.21.15.69:6833","nonce":626688145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:09:38.549277+0000","modified":"2024-04-24T08:09:41.567377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/991803145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":991803145},{"type":"v1","addr":"172.21.15.69:6835","nonce":991803145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5241,"qdb_cluster":[5241]},"id":2}]} 2024-04-24T08:10:06.337 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 16 2024-04-24T08:10:06.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 -- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5dec06c8f0 msgr2=0x7f5dec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5dec06c8f0 0x7f5dec06edb0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5df8006fd0 tx=0x7f5df8008040 comp rx=0 tx=0).stop 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 -- 172.21.15.69:0/3282640793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1006d4f0 msgr2=0x7f5e101616c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1006d4f0 0x7f5e101616c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5e00035040 tx=0x7f5e000046c0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 -- 172.21.15.69:0/3282640793 shutdown_connections 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5dec06c8f0 0x7f5dec06edb0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e10162980 0x7f5e1015f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e1015e960 0x7f5e1015ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 --2- 172.21.15.69:0/3282640793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e1006d4f0 0x7f5e101616c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 -- 172.21.15.69:0/3282640793 >> 172.21.15.69:0/3282640793 conn(0x7f5e1009d900 msgr2=0x7f5e100cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 -- 172.21.15.69:0/3282640793 shutdown_connections 2024-04-24T08:10:06.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:06.338+0000 7f5e15200640 1 -- 172.21.15.69:0/3282640793 wait complete. 2024-04-24T08:10:06.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json tag path /parentdir mytag 2024-04-24T08:10:06.581 INFO:teuthology.orchestra.run.smithi069.stdout:{"return_code":0,"scrub_tag":"mytag","mode":"asynchronous"} 2024-04-24T08:10:06.581 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag" } 2024-04-24T08:10:13.817 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:10:13.825 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:10:16.583 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000000.00000000 scrub_tag 2024-04-24T08:10:16.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.698+0000 7f00f931bf40 1 Processor -- start 2024-04-24T08:10:16.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.698+0000 7f00f931bf40 1 -- start start 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.699+0000 7f00f931bf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a603b91ec0 0x55a603c33970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.699+0000 7f00f931bf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603c3f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.699+0000 7f00f931bf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a603c3cc20 0x55a603c3fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.699+0000 7f00f931bf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a603b5e710 con 0x55a603c36340 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.699+0000 7f00f931bf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a603b07cc0 con 0x55a603b91ec0 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.699+0000 7f00f931bf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a603b90e50 con 0x55a603c3cc20 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.699+0000 7f00f5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603c3f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.700+0000 7f00f5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603c3f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56722/0 (socket says 172.21.15.69:56722) 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.700+0000 7f00f5000640 1 -- 172.21.15.69:0/1829062435 learned_addr learned my addr 172.21.15.69:0/1829062435 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.700+0000 7f00f5000640 1 -- 172.21.15.69:0/1829062435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a603c3cc20 msgr2=0x55a603c3fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.700+0000 7f00f5000640 1 --2- 172.21.15.69:0/1829062435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a603c3cc20 0x55a603c3fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.700+0000 7f00f5000640 1 -- 172.21.15.69:0/1829062435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a603b91ec0 msgr2=0x55a603c33970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:16.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.700+0000 7f00f5a00640 1 --2- 172.21.15.69:0/1829062435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a603b91ec0 0x55a603c33970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.700+0000 7f00f5000640 1 --2- 172.21.15.69:0/1829062435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a603b91ec0 0x55a603c33970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.700+0000 7f00f5000640 1 -- 172.21.15.69:0/1829062435 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a603c3e780 con 0x55a603c36340 2024-04-24T08:10:16.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.700+0000 7f00f5000640 1 --2- 172.21.15.69:0/1829062435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603c3f590 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f00e0001d90 tx=0x7f00e000b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e51c6657c2ba75e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.700+0000 7f00ef400640 1 -- 172.21.15.69:0/1829062435 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e0012020 con 0x55a603c36340 2024-04-24T08:10:16.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00ef400640 1 -- 172.21.15.69:0/1829062435 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00e000e670 con 0x55a603c36340 2024-04-24T08:10:16.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00ef400640 1 -- 172.21.15.69:0/1829062435 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e000eac0 con 0x55a603c36340 2024-04-24T08:10:16.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00f931bf40 1 -- 172.21.15.69:0/1829062435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 msgr2=0x55a603c3f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00f931bf40 1 --2- 172.21.15.69:0/1829062435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603c3f590 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f00e0001d90 tx=0x7f00e000b6c0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00f931bf40 1 -- 172.21.15.69:0/1829062435 shutdown_connections 2024-04-24T08:10:16.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00f931bf40 1 --2- 172.21.15.69:0/1829062435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603c3f590 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00f931bf40 1 --2- 172.21.15.69:0/1829062435 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a603c3cc20 0x55a603c3fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00f931bf40 1 --2- 172.21.15.69:0/1829062435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a603b91ec0 0x55a603c33970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00f931bf40 1 -- 172.21.15.69:0/1829062435 >> 172.21.15.69:0/1829062435 conn(0x55a603b8fb50 msgr2=0x55a603c3d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:16.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00f931bf40 1 -- 172.21.15.69:0/1829062435 shutdown_connections 2024-04-24T08:10:16.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.701+0000 7f00f931bf40 1 -- 172.21.15.69:0/1829062435 wait complete. 2024-04-24T08:10:16.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.702+0000 7f00f931bf40 1 Processor -- start 2024-04-24T08:10:16.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.702+0000 7f00f931bf40 1 -- start start 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.703+0000 7f00f931bf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a603b91ec0 0x55a603c30130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.703+0000 7f00f931bf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603b62930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.703+0000 7f00f931bf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a603c3cc20 0x55a603b62e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.703+0000 7f00f931bf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a603b644d0 con 0x55a603c36340 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.703+0000 7f00f931bf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a603b90400 con 0x55a603b91ec0 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.703+0000 7f00f931bf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a603b64720 con 0x55a603c3cc20 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603b62930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603b62930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56730/0 (socket says 172.21.15.69:56730) 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f5000640 1 -- 172.21.15.69:0/528691265 learned_addr learned my addr 172.21.15.69:0/528691265 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f5a00640 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a603b91ec0 0x55a603c30130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f6400640 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a603c3cc20 0x55a603b62e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f5000640 1 -- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a603c3cc20 msgr2=0x55a603b62e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f5000640 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a603c3cc20 0x55a603b62e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f5000640 1 -- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a603b91ec0 msgr2=0x55a603c30130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f5000640 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a603b91ec0 0x55a603c30130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f5000640 1 -- 172.21.15.69:0/528691265 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a603c3dd10 con 0x55a603c36340 2024-04-24T08:10:16.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.704+0000 7f00f5000640 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603b62930 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x55a603b91400 tx=0x7f00e0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.705+0000 7f00ee000640 1 -- 172.21.15.69:0/528691265 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e0027070 con 0x55a603c36340 2024-04-24T08:10:16.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.705+0000 7f00ee000640 1 -- 172.21.15.69:0/528691265 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00e000be90 con 0x55a603c36340 2024-04-24T08:10:16.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.705+0000 7f00f931bf40 1 -- 172.21.15.69:0/528691265 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a603b60af0 con 0x55a603c36340 2024-04-24T08:10:16.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.705+0000 7f00ee000640 1 -- 172.21.15.69:0/528691265 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e00197f0 con 0x55a603c36340 2024-04-24T08:10:16.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.705+0000 7f00f931bf40 1 -- 172.21.15.69:0/528691265 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a603c3dff0 con 0x55a603c36340 2024-04-24T08:10:16.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.707+0000 7f00ee000640 1 -- 172.21.15.69:0/528691265 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f00e002a070 con 0x55a603c36340 2024-04-24T08:10:16.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.708+0000 7f00ee000640 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00d406b300 0x7f00d406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.708+0000 7f00ee000640 1 -- 172.21.15.69:0/528691265 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f00e008be50 con 0x55a603c36340 2024-04-24T08:10:16.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.708+0000 7f00f931bf40 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55a603b919d0 0x55a603b61800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.708+0000 7f00f931bf40 1 -- 172.21.15.69:0/528691265 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:1 4.10 4:0cf6ad0f:::10000000000.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) -- 0x55a603b61d40 con 0x55a603b919d0 2024-04-24T08:10:16.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.708+0000 7f00f5a00640 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00d406b300 0x7f00d406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.709+0000 7f00f6400640 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55a603b919d0 0x55a603b61800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.709+0000 7f00f6400640 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55a603b919d0 0x55a603b61800 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f6400640 1 -- 172.21.15.69:0/528691265 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (9) out=9b] v0'0 uv11 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7f00f00096e0 con 0x55a603b919d0 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f931bf40 1 -- 172.21.15.69:0/528691265 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55a603b919d0 msgr2=0x55a603b61800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f931bf40 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55a603b919d0 0x55a603b61800 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f931bf40 1 -- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00d406b300 msgr2=0x7f00d406d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f5a00640 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00d406b300 0x7f00d406d7c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f00e8057000 tx=0x7f00e805e040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f931bf40 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00d406b300 0x7f00d406d7c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f00e8057000 tx=0x7f00e805e040 comp rx=0 tx=0).stop 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f931bf40 1 -- 172.21.15.69:0/528691265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 msgr2=0x55a603b62930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f931bf40 1 --2- 172.21.15.69:0/528691265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a603c36340 0x55a603b62930 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x55a603b91400 tx=0x7f00e0000bc0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f6400640 1 -- 172.21.15.69:0/528691265 reap_dead start 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f931bf40 1 -- 172.21.15.69:0/528691265 shutdown_connections 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f931bf40 1 -- 172.21.15.69:0/528691265 >> 172.21.15.69:0/528691265 conn(0x55a603b8fb50 msgr2=0x55a603c3c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f931bf40 1 -- 172.21.15.69:0/528691265 shutdown_connections 2024-04-24T08:10:16.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.710+0000 7f00f931bf40 1 -- 172.21.15.69:0/528691265 wait complete. 2024-04-24T08:10:16.715 INFO:teuthology.orchestra.run.smithi069.stdout:mytag 2024-04-24T08:10:16.715 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000001.00000000 scrub_tag 2024-04-24T08:10:16.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.808+0000 7f721ad01f40 1 Processor -- start 2024-04-24T08:10:16.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.808+0000 7f721ad01f40 1 -- start start 2024-04-24T08:10:16.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.809+0000 7f721ad01f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e1285825a0 0x55e128650790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.809+0000 7f721ad01f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e1285cd940 0x55e1285cc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.809+0000 7f721ad01f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e128653f50 0x55e128580680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.809+0000 7f721ad01f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e12865ad60 con 0x55e128653f50 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.809+0000 7f721ad01f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e1285b3180 con 0x55e1285cd940 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.809+0000 7f721ad01f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55e1285829b0 con 0x55e1285825a0 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.809+0000 7f7218a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e128653f50 0x55e128580680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.809+0000 7f7218a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e128653f50 0x55e128580680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56754/0 (socket says 172.21.15.69:56754) 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.809+0000 7f7218a00640 1 -- 172.21.15.69:0/1598829248 learned_addr learned my addr 172.21.15.69:0/1598829248 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7217600640 1 --2- 172.21.15.69:0/1598829248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e1285cd940 0x55e1285cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7218000640 1 --2- 172.21.15.69:0/1598829248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e1285825a0 0x55e128650790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7218a00640 1 -- 172.21.15.69:0/1598829248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e1285825a0 msgr2=0x55e128650790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7218a00640 1 --2- 172.21.15.69:0/1598829248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e1285825a0 0x55e128650790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7218a00640 1 -- 172.21.15.69:0/1598829248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e1285cd940 msgr2=0x55e1285cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:16.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7218a00640 1 --2- 172.21.15.69:0/1598829248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e1285cd940 0x55e1285cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7218a00640 1 -- 172.21.15.69:0/1598829248 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1284adb30 con 0x55e128653f50 2024-04-24T08:10:16.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7218a00640 1 --2- 172.21.15.69:0/1598829248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e128653f50 0x55e128580680 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f721000a460 tx=0x7f721000a920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efcaa47602c82aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7216200640 1 -- 172.21.15.69:0/1598829248 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7210008940 con 0x55e128653f50 2024-04-24T08:10:16.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7216200640 1 -- 172.21.15.69:0/1598829248 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7210008b00 con 0x55e128653f50 2024-04-24T08:10:16.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.810+0000 7f7216200640 1 -- 172.21.15.69:0/1598829248 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7210013440 con 0x55e128653f50 2024-04-24T08:10:16.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.811+0000 7f721ad01f40 1 -- 172.21.15.69:0/1598829248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e128653f50 msgr2=0x55e128580680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.811+0000 7f721ad01f40 1 --2- 172.21.15.69:0/1598829248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e128653f50 0x55e128580680 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f721000a460 tx=0x7f721000a920 comp rx=0 tx=0).stop 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.811+0000 7f721ad01f40 1 -- 172.21.15.69:0/1598829248 shutdown_connections 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.811+0000 7f721ad01f40 1 --2- 172.21.15.69:0/1598829248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e128653f50 0x55e128580680 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.811+0000 7f721ad01f40 1 --2- 172.21.15.69:0/1598829248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e1285cd940 0x55e1285cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.811+0000 7f721ad01f40 1 --2- 172.21.15.69:0/1598829248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e1285825a0 0x55e128650790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.811+0000 7f721ad01f40 1 -- 172.21.15.69:0/1598829248 >> 172.21.15.69:0/1598829248 conn(0x55e1285b1b50 msgr2=0x55e1285b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.811+0000 7f721ad01f40 1 -- 172.21.15.69:0/1598829248 shutdown_connections 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.811+0000 7f721ad01f40 1 -- 172.21.15.69:0/1598829248 wait complete. 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.812+0000 7f721ad01f40 1 Processor -- start 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.812+0000 7f721ad01f40 1 -- start start 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.813+0000 7f721ad01f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e1285825a0 0x55e1286543c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.813+0000 7f721ad01f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e1285cd940 0x55e1286e6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.813+0000 7f721ad01f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e128653f50 0x55e128640f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.813+0000 7f721ad01f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e128652cc0 con 0x55e1285cd940 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.813+0000 7f721ad01f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e128650cd0 con 0x55e128653f50 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.813+0000 7f721ad01f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55e1286535d0 con 0x55e1285825a0 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.813+0000 7f7217600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e1285cd940 0x55e1286e6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.813+0000 7f7217600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e1285cd940 0x55e1286e6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56762/0 (socket says 172.21.15.69:56762) 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.813+0000 7f7217600640 1 -- 172.21.15.69:0/1617217226 learned_addr learned my addr 172.21.15.69:0/1617217226 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7218000640 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e1285825a0 0x55e1286543c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7217600640 1 -- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e1285825a0 msgr2=0x55e1286543c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7217600640 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e1285825a0 0x55e1286543c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7217600640 1 -- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e128653f50 msgr2=0x55e128640f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7218a00640 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e128653f50 0x55e128640f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7217600640 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e128653f50 0x55e128640f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7217600640 1 -- 172.21.15.69:0/1617217226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e128654900 con 0x55e1285cd940 2024-04-24T08:10:16.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7217600640 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e1285cd940 0x55e1286e6080 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f7200007e70 tx=0x7f7200006750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7214e00640 1 -- 172.21.15.69:0/1617217226 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200018070 con 0x55e1285cd940 2024-04-24T08:10:16.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7214e00640 1 -- 172.21.15.69:0/1617217226 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7200011700 con 0x55e1285cd940 2024-04-24T08:10:16.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f7214e00640 1 -- 172.21.15.69:0/1617217226 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200011ad0 con 0x55e1285cd940 2024-04-24T08:10:16.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.814+0000 7f721ad01f40 1 -- 172.21.15.69:0/1617217226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e1286f2a10 con 0x55e1285cd940 2024-04-24T08:10:16.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.815+0000 7f721ad01f40 1 -- 172.21.15.69:0/1617217226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1286f2fd0 con 0x55e1285cd940 2024-04-24T08:10:16.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.817+0000 7f7214e00640 1 -- 172.21.15.69:0/1617217226 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f720001b070 con 0x55e1285cd940 2024-04-24T08:10:16.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.817+0000 7f7214e00640 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f71f406b300 0x7f71f406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.818+0000 7f7214e00640 1 -- 172.21.15.69:0/1617217226 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f7200089b50 con 0x55e1285cd940 2024-04-24T08:10:16.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.818+0000 7f721ad01f40 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55e1286f2550 0x55e1286eff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.818+0000 7f721ad01f40 1 -- 172.21.15.69:0/1617217226 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:1 4.7 4:e2e07930:::10000000001.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) -- 0x55e1286e65c0 con 0x55e1286f2550 2024-04-24T08:10:16.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.818+0000 7f7218000640 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f71f406b300 0x7f71f406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.818+0000 7f7218a00640 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55e1286f2550 0x55e1286eff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.819+0000 7f7218a00640 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55e1286f2550 0x55e1286eff30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.819+0000 7f7218a00640 1 -- 172.21.15.69:0/1617217226 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (9) out=9b] v0'0 uv5 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7f7210008d30 con 0x55e1286f2550 2024-04-24T08:10:16.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.819+0000 7f721ad01f40 1 -- 172.21.15.69:0/1617217226 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55e1286f2550 msgr2=0x55e1286eff30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.819+0000 7f721ad01f40 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55e1286f2550 0x55e1286eff30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.819+0000 7f7218000640 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f71f406b300 0x7f71f406d7c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7208056000 tx=0x7f7208007030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.819+0000 7f721ad01f40 1 -- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f71f406b300 msgr2=0x7f71f406d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.819+0000 7f721ad01f40 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f71f406b300 0x7f71f406d7c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7208056000 tx=0x7f7208007030 comp rx=0 tx=0).stop 2024-04-24T08:10:16.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.819+0000 7f721ad01f40 1 -- 172.21.15.69:0/1617217226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e1285cd940 msgr2=0x55e1286e6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.819+0000 7f721ad01f40 1 --2- 172.21.15.69:0/1617217226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e1285cd940 0x55e1286e6080 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f7200007e70 tx=0x7f7200006750 comp rx=0 tx=0).stop 2024-04-24T08:10:16.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.819+0000 7f7218a00640 1 -- 172.21.15.69:0/1617217226 reap_dead start 2024-04-24T08:10:16.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.820+0000 7f721ad01f40 1 -- 172.21.15.69:0/1617217226 shutdown_connections 2024-04-24T08:10:16.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.820+0000 7f721ad01f40 1 -- 172.21.15.69:0/1617217226 >> 172.21.15.69:0/1617217226 conn(0x55e1285b1b50 msgr2=0x55e128650e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:16.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.820+0000 7f721ad01f40 1 -- 172.21.15.69:0/1617217226 shutdown_connections 2024-04-24T08:10:16.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.820+0000 7f721ad01f40 1 -- 172.21.15.69:0/1617217226 wait complete. 2024-04-24T08:10:16.825 INFO:teuthology.orchestra.run.smithi069.stdout:mytag 2024-04-24T08:10:16.825 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-04-24T08:10:16.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.913+0000 7fe0b3887f40 1 Processor -- start 2024-04-24T08:10:16.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.913+0000 7fe0b3887f40 1 -- start start 2024-04-24T08:10:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.914+0000 7fe0b3887f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5602ba743ad0 0x5602ba8132e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.914+0000 7fe0b3887f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba770380 0x5602ba781250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.914+0000 7fe0b3887f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5602ba8154b0 0x5602ba781790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.914+0000 7fe0b3887f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5602ba815aa0 con 0x5602ba770380 2024-04-24T08:10:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.914+0000 7fe0b3887f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5602ba6e7cc0 con 0x5602ba8154b0 2024-04-24T08:10:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.914+0000 7fe0b3887f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5602ba815c20 con 0x5602ba743ad0 2024-04-24T08:10:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.914+0000 7fe0abe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba770380 0x5602ba781250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.914+0000 7fe0abe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba770380 0x5602ba781250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56776/0 (socket says 172.21.15.69:56776) 2024-04-24T08:10:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.914+0000 7fe0abe00640 1 -- 172.21.15.69:0/3525828462 learned_addr learned my addr 172.21.15.69:0/3525828462 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.915+0000 7fe0b1400640 1 --2- 172.21.15.69:0/3525828462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5602ba8154b0 0x5602ba781790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.915+0000 7fe0abe00640 1 -- 172.21.15.69:0/3525828462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5602ba743ad0 msgr2=0x5602ba8132e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.915+0000 7fe0abe00640 1 --2- 172.21.15.69:0/3525828462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5602ba743ad0 0x5602ba8132e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.915+0000 7fe0abe00640 1 -- 172.21.15.69:0/3525828462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5602ba8154b0 msgr2=0x5602ba781790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.915+0000 7fe0abe00640 1 --2- 172.21.15.69:0/3525828462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5602ba8154b0 0x5602ba781790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.915+0000 7fe0abe00640 1 -- 172.21.15.69:0/3525828462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ba814220 con 0x5602ba770380 2024-04-24T08:10:16.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.915+0000 7fe0abe00640 1 --2- 172.21.15.69:0/3525828462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba770380 0x5602ba781250 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00b310 tx=0x7fe09c00b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34e0eb7661b3eb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.915+0000 7fe0aaa00640 1 -- 172.21.15.69:0/3525828462 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c008070 con 0x5602ba770380 2024-04-24T08:10:16.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.915+0000 7fe0aaa00640 1 -- 172.21.15.69:0/3525828462 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe09c00d500 con 0x5602ba770380 2024-04-24T08:10:16.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.915+0000 7fe0aaa00640 1 -- 172.21.15.69:0/3525828462 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c00d8b0 con 0x5602ba770380 2024-04-24T08:10:16.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.916+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3525828462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba770380 msgr2=0x5602ba781250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.916+0000 7fe0b3887f40 1 --2- 172.21.15.69:0/3525828462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba770380 0x5602ba781250 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00b310 tx=0x7fe09c00b7d0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.916+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3525828462 shutdown_connections 2024-04-24T08:10:16.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.916+0000 7fe0b3887f40 1 --2- 172.21.15.69:0/3525828462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba770380 0x5602ba781250 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.916+0000 7fe0b3887f40 1 --2- 172.21.15.69:0/3525828462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5602ba8154b0 0x5602ba781790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.916+0000 7fe0b3887f40 1 --2- 172.21.15.69:0/3525828462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5602ba743ad0 0x5602ba8132e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.916+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3525828462 >> 172.21.15.69:0/3525828462 conn(0x5602ba76de40 msgr2=0x5602ba76e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.916+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3525828462 shutdown_connections 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.916+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3525828462 wait complete. 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.917+0000 7fe0b3887f40 1 Processor -- start 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.917+0000 7fe0b3887f40 1 -- start start 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0b3887f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5602ba743ad0 0x5602ba7fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0b3887f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5602ba770380 0x5602ba8132e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0b3887f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba8154b0 0x5602ba813820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0b3887f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5602ba7870a0 con 0x5602ba8154b0 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0b3887f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5602ba743390 con 0x5602ba770380 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0b3887f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5602ba8a5a60 con 0x5602ba743ad0 2024-04-24T08:10:16.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0b1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba8154b0 0x5602ba813820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0b1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba8154b0 0x5602ba813820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56780/0 (socket says 172.21.15.69:56780) 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0b1400640 1 -- 172.21.15.69:0/3794845213 learned_addr learned my addr 172.21.15.69:0/3794845213 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0b0a00640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5602ba743ad0 0x5602ba7fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.918+0000 7fe0abe00640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5602ba770380 0x5602ba8132e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0b1400640 1 -- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5602ba743ad0 msgr2=0x5602ba7fd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0b1400640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5602ba743ad0 0x5602ba7fd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0b1400640 1 -- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5602ba770380 msgr2=0x5602ba8132e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0b1400640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5602ba770380 0x5602ba8132e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0b1400640 1 -- 172.21.15.69:0/3794845213 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ba8a6780 con 0x5602ba8154b0 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0abe00640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5602ba770380 0x5602ba8132e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:16.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0b1400640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba8154b0 0x5602ba813820 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac025ac0 tx=0x7fe0ac003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0a9600640 1 -- 172.21.15.69:0/3794845213 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac06a070 con 0x5602ba8154b0 2024-04-24T08:10:16.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0a9600640 1 -- 172.21.15.69:0/3794845213 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0ac051470 con 0x5602ba8154b0 2024-04-24T08:10:16.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0a9600640 1 -- 172.21.15.69:0/3794845213 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac055040 con 0x5602ba8154b0 2024-04-24T08:10:16.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3794845213 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5602ba8a6a60 con 0x5602ba8154b0 2024-04-24T08:10:16.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.919+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3794845213 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602ba8a5630 con 0x5602ba8154b0 2024-04-24T08:10:16.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.921+0000 7fe0a9600640 1 -- 172.21.15.69:0/3794845213 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe0ac06d070 con 0x5602ba8154b0 2024-04-24T08:10:16.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.922+0000 7fe0a9600640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe09006b2d0 0x7fe09006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.922+0000 7fe0a9600640 1 -- 172.21.15.69:0/3794845213 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fe0ac0ce480 con 0x5602ba8154b0 2024-04-24T08:10:16.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.923+0000 7fe0b3887f40 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5602ba812970 0x5602ba7826c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:16.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.923+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3794845213 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:1 5.36 5:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) -- 0x5602ba782c00 con 0x5602ba812970 2024-04-24T08:10:16.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.923+0000 7fe0b0a00640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe09006b2d0 0x7fe09006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.923+0000 7fe0abe00640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5602ba812970 0x5602ba7826c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:16.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.923+0000 7fe0abe00640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5602ba812970 0x5602ba7826c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0abe00640 1 -- 172.21.15.69:0/3794845213 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr (9) out=9b] v0'0 uv17 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7fe09c00d7c0 con 0x5602ba812970 2024-04-24T08:10:16.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b0a00640 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe09006b2d0 0x7fe09006d790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40030a0 tx=0x7fe0a4005770 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:16.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3794845213 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5602ba812970 msgr2=0x5602ba7826c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b3887f40 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5602ba812970 0x5602ba7826c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:16.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe09006b2d0 msgr2=0x7fe09006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b3887f40 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe09006b2d0 0x7fe09006d790 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe0a40030a0 tx=0x7fe0a4005770 comp rx=0 tx=0).stop 2024-04-24T08:10:16.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3794845213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba8154b0 msgr2=0x5602ba813820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:16.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b3887f40 1 --2- 172.21.15.69:0/3794845213 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602ba8154b0 0x5602ba813820 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac025ac0 tx=0x7fe0ac003330 comp rx=0 tx=0).stop 2024-04-24T08:10:16.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b1400640 1 -- 172.21.15.69:0/3794845213 reap_dead start 2024-04-24T08:10:16.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3794845213 shutdown_connections 2024-04-24T08:10:16.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3794845213 >> 172.21.15.69:0/3794845213 conn(0x5602ba76de40 msgr2=0x5602ba76e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:16.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3794845213 shutdown_connections 2024-04-24T08:10:16.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:16.924+0000 7fe0b3887f40 1 -- 172.21.15.69:0/3794845213 wait complete. 2024-04-24T08:10:16.929 INFO:teuthology.orchestra.run.smithi069.stdout:mytag 2024-04-24T08:10:16.929 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-04-24T08:10:17.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.014+0000 7f9a1b33cf40 1 Processor -- start 2024-04-24T08:10:17.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.014+0000 7f9a1b33cf40 1 -- start start 2024-04-24T08:10:17.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.015+0000 7f9a1b33cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd84fc770 0x557bd85c32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.015+0000 7f9a1b33cf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8547940 0x557bd852b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.015+0000 7f9a1b33cf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd85cdf50 0x557bd85c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.015+0000 7f9a1b33cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557bd852d180 con 0x557bd84fc770 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.015+0000 7f9a1b33cf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557bd84a3cc0 con 0x557bd85cdf50 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.015+0000 7f9a1b33cf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557bd85d4d60 con 0x557bd8547940 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.015+0000 7f9a18600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd84fc770 0x557bd85c32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.015+0000 7f9a18600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd84fc770 0x557bd85c32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56798/0 (socket says 172.21.15.69:56798) 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.015+0000 7f9a18600640 1 -- 172.21.15.69:0/1002279268 learned_addr learned my addr 172.21.15.69:0/1002279268 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.016+0000 7f9a18600640 1 -- 172.21.15.69:0/1002279268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8547940 msgr2=0x557bd852b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.016+0000 7f9a18600640 1 --2- 172.21.15.69:0/1002279268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8547940 0x557bd852b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.016+0000 7f9a18600640 1 -- 172.21.15.69:0/1002279268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd85cdf50 msgr2=0x557bd85c3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.016+0000 7f9a18600640 1 --2- 172.21.15.69:0/1002279268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd85cdf50 0x557bd85c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.016+0000 7f9a18600640 1 -- 172.21.15.69:0/1002279268 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bd85cbe80 con 0x557bd84fc770 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.016+0000 7f9a18600640 1 --2- 172.21.15.69:0/1002279268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd84fc770 0x557bd85c32d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800c480 tx=0x7f9a0800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80afe02d32bbfd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:17.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.016+0000 7f9a16800640 1 -- 172.21.15.69:0/1002279268 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080116b0 con 0x557bd84fc770 2024-04-24T08:10:17.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.016+0000 7f9a16800640 1 -- 172.21.15.69:0/1002279268 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a08017070 con 0x557bd84fc770 2024-04-24T08:10:17.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.016+0000 7f9a16800640 1 -- 172.21.15.69:0/1002279268 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a08011a50 con 0x557bd84fc770 2024-04-24T08:10:17.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.017+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/1002279268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd84fc770 msgr2=0x557bd85c32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.017+0000 7f9a1b33cf40 1 --2- 172.21.15.69:0/1002279268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd84fc770 0x557bd85c32d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800c480 tx=0x7f9a0800c940 comp rx=0 tx=0).stop 2024-04-24T08:10:17.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.017+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/1002279268 shutdown_connections 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.017+0000 7f9a1b33cf40 1 --2- 172.21.15.69:0/1002279268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd85cdf50 0x557bd85c3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.017+0000 7f9a1b33cf40 1 --2- 172.21.15.69:0/1002279268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8547940 0x557bd852b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.017+0000 7f9a1b33cf40 1 --2- 172.21.15.69:0/1002279268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd84fc770 0x557bd85c32d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.017+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/1002279268 >> 172.21.15.69:0/1002279268 conn(0x557bd852bb50 msgr2=0x557bd84fc040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.017+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/1002279268 shutdown_connections 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.017+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/1002279268 wait complete. 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.018+0000 7f9a1b33cf40 1 Processor -- start 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.018+0000 7f9a1b33cf40 1 -- start start 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a1b33cf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd84fc770 0x557bd85bcb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a1b33cf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8547940 0x557bd85bd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a1b33cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd85cdf50 0x557bd8663640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a1b33cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557bd852fa60 con 0x557bd85cdf50 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a1b33cf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557bd85cc540 con 0x557bd84fc770 2024-04-24T08:10:17.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a1b33cf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557bd85cc260 con 0x557bd8547940 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a19000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd85cdf50 0x557bd8663640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a19000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd85cdf50 0x557bd8663640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56804/0 (socket says 172.21.15.69:56804) 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a19000640 1 -- 172.21.15.69:0/3635496808 learned_addr learned my addr 172.21.15.69:0/3635496808 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a18600640 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd84fc770 0x557bd85bcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a17c00640 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8547940 0x557bd85bd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a19000640 1 -- 172.21.15.69:0/3635496808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8547940 msgr2=0x557bd85bd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a19000640 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8547940 0x557bd85bd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a19000640 1 -- 172.21.15.69:0/3635496808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd84fc770 msgr2=0x557bd85bcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a19000640 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd84fc770 0x557bd85bcb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.019+0000 7f9a19000640 1 -- 172.21.15.69:0/3635496808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bd8664310 con 0x557bd85cdf50 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.020+0000 7f9a19000640 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd85cdf50 0x557bd8663640 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9a10025cc0 tx=0x7f9a100058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.020+0000 7f9a15400640 1 -- 172.21.15.69:0/3635496808 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10054800 con 0x557bd85cdf50 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.020+0000 7f9a15400640 1 -- 172.21.15.69:0/3635496808 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a100549a0 con 0x557bd85cdf50 2024-04-24T08:10:17.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.020+0000 7f9a15400640 1 -- 172.21.15.69:0/3635496808 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a10054d70 con 0x557bd85cdf50 2024-04-24T08:10:17.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.020+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/3635496808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557bd84fc0f0 con 0x557bd85cdf50 2024-04-24T08:10:17.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.020+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/3635496808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bd85c2970 con 0x557bd85cdf50 2024-04-24T08:10:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.022+0000 7f9a15400640 1 -- 172.21.15.69:0/3635496808 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a1005c040 con 0x557bd85cdf50 2024-04-24T08:10:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.023+0000 7f9a15400640 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f99f806b2d0 0x7f99f806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.023+0000 7f9a15400640 1 -- 172.21.15.69:0/3635496808 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f9a100ce900 con 0x557bd85cdf50 2024-04-24T08:10:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.023+0000 7f9a1b33cf40 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x557bd852cbe0 0x557bd85befc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.023+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/3635496808 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:1 5.19 5:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) -- 0x557bd85d5650 con 0x557bd852cbe0 2024-04-24T08:10:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.024+0000 7f9a18600640 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f99f806b2d0 0x7f99f806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:17.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.024+0000 7f9a17c00640 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x557bd852cbe0 0x557bd85befc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:17.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.024+0000 7f9a17c00640 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x557bd852cbe0 0x557bd85befc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:17.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a17c00640 1 -- 172.21.15.69:0/3635496808 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (9) out=9b] v0'0 uv12 ondisk = 0) ==== 164+0+9 (crc 0 0 0) 0x7f9a0400f3a0 con 0x557bd852cbe0 2024-04-24T08:10:17.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/3635496808 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x557bd852cbe0 msgr2=0x557bd85befc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a1b33cf40 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x557bd852cbe0 0x557bd85befc0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/3635496808 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f99f806b2d0 msgr2=0x7f99f806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a1b33cf40 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f99f806b2d0 0x7f99f806d790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x557bd852b440 tx=0x7f9a08011000 comp rx=0 tx=0).stop 2024-04-24T08:10:17.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/3635496808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd85cdf50 msgr2=0x557bd8663640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a1b33cf40 1 --2- 172.21.15.69:0/3635496808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd85cdf50 0x557bd8663640 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f9a10025cc0 tx=0x7f9a100058b0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a19000640 1 -- 172.21.15.69:0/3635496808 reap_dead start 2024-04-24T08:10:17.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/3635496808 shutdown_connections 2024-04-24T08:10:17.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/3635496808 >> 172.21.15.69:0/3635496808 conn(0x557bd852bb50 msgr2=0x557bd85d4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:17.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/3635496808 shutdown_connections 2024-04-24T08:10:17.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.025+0000 7f9a1b33cf40 1 -- 172.21.15.69:0/3635496808 wait complete. 2024-04-24T08:10:17.030 INFO:teuthology.orchestra.run.smithi069.stdout:mytag 2024-04-24T08:10:17.031 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-04-24T08:10:17.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.112+0000 7fc0dda7cf40 1 Processor -- start 2024-04-24T08:10:17.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.112+0000 7fc0dda7cf40 1 -- start start 2024-04-24T08:10:17.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0dda7cf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5654b726c5a0 0x5654b73cc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0dda7cf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5654b72b7940 0x5654b733abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0dda7cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b733df50 0x5654b7344f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0dda7cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5654b729d180 con 0x5654b733df50 2024-04-24T08:10:17.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0dda7cf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5654b7213cc0 con 0x5654b72b7940 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0dda7cf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5654b7344d60 con 0x5654b726c5a0 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0db600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b733df50 0x5654b7344f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0db600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b733df50 0x5654b7344f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56814/0 (socket says 172.21.15.69:56814) 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0db600640 1 -- 172.21.15.69:0/1692129751 learned_addr learned my addr 172.21.15.69:0/1692129751 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0db600640 1 -- 172.21.15.69:0/1692129751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5654b726c5a0 msgr2=0x5654b73cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0db600640 1 --2- 172.21.15.69:0/1692129751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5654b726c5a0 0x5654b73cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0db600640 1 -- 172.21.15.69:0/1692129751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5654b72b7940 msgr2=0x5654b733abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0db600640 1 --2- 172.21.15.69:0/1692129751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5654b72b7940 0x5654b733abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.113+0000 7fc0db600640 1 -- 172.21.15.69:0/1692129751 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654b729b360 con 0x5654b733df50 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.114+0000 7fc0db600640 1 --2- 172.21.15.69:0/1692129751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b733df50 0x5654b7344f10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4006e50 tx=0x7fc0d400b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=affe396b76e15d4a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.114+0000 7fc0d8e00640 1 -- 172.21.15.69:0/1692129751 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4012070 con 0x5654b733df50 2024-04-24T08:10:17.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.114+0000 7fc0d8e00640 1 -- 172.21.15.69:0/1692129751 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d400e670 con 0x5654b733df50 2024-04-24T08:10:17.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.114+0000 7fc0d8e00640 1 -- 172.21.15.69:0/1692129751 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d400ea60 con 0x5654b733df50 2024-04-24T08:10:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.115+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/1692129751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b733df50 msgr2=0x5654b7344f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.115+0000 7fc0dda7cf40 1 --2- 172.21.15.69:0/1692129751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b733df50 0x5654b7344f10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4006e50 tx=0x7fc0d400b6c0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.115+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/1692129751 shutdown_connections 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.115+0000 7fc0dda7cf40 1 --2- 172.21.15.69:0/1692129751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b733df50 0x5654b7344f10 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.115+0000 7fc0dda7cf40 1 --2- 172.21.15.69:0/1692129751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5654b72b7940 0x5654b733abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.115+0000 7fc0dda7cf40 1 --2- 172.21.15.69:0/1692129751 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5654b726c5a0 0x5654b73cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.115+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/1692129751 >> 172.21.15.69:0/1692129751 conn(0x5654b729bb50 msgr2=0x5654b729a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.115+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/1692129751 shutdown_connections 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.115+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/1692129751 wait complete. 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.115+0000 7fc0dda7cf40 1 Processor -- start 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.116+0000 7fc0dda7cf40 1 -- start start 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.116+0000 7fc0dda7cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b726c5a0 0x5654b7332df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.116+0000 7fc0dda7cf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5654b72b7940 0x5654b73cf500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.116+0000 7fc0dda7cf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5654b733df50 0x5654b73cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.116+0000 7fc0dda7cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5654b733a780 con 0x5654b726c5a0 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.116+0000 7fc0dda7cf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5654b729a300 con 0x5654b72b7940 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.116+0000 7fc0dda7cf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5654b73457f0 con 0x5654b733df50 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.117+0000 7fc0dac00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b726c5a0 0x5654b7332df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.117+0000 7fc0dac00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b726c5a0 0x5654b7332df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56820/0 (socket says 172.21.15.69:56820) 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.117+0000 7fc0dac00640 1 -- 172.21.15.69:0/3078075041 learned_addr learned my addr 172.21.15.69:0/3078075041 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.117+0000 7fc0dac00640 1 -- 172.21.15.69:0/3078075041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5654b733df50 msgr2=0x5654b73cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.117+0000 7fc0dac00640 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5654b733df50 0x5654b73cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.117+0000 7fc0dac00640 1 -- 172.21.15.69:0/3078075041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5654b72b7940 msgr2=0x5654b73cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.117+0000 7fc0dac00640 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5654b72b7940 0x5654b73cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.117+0000 7fc0dac00640 1 -- 172.21.15.69:0/3078075041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654b73cff80 con 0x5654b726c5a0 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.117+0000 7fc0dac00640 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b726c5a0 0x5654b7332df0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0460e0 tx=0x7fc0cc003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.118+0000 7fc0d3400640 1 -- 172.21.15.69:0/3078075041 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc067070 con 0x5654b726c5a0 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.118+0000 7fc0d3400640 1 -- 172.21.15.69:0/3078075041 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0cc062440 con 0x5654b726c5a0 2024-04-24T08:10:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.118+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/3078075041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5654b73334b0 con 0x5654b726c5a0 2024-04-24T08:10:17.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.118+0000 7fc0d3400640 1 -- 172.21.15.69:0/3078075041 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc062830 con 0x5654b726c5a0 2024-04-24T08:10:17.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.118+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/3078075041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654b73cc3d0 con 0x5654b726c5a0 2024-04-24T08:10:17.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.120+0000 7fc0d3400640 1 -- 172.21.15.69:0/3078075041 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc0cc06a070 con 0x5654b726c5a0 2024-04-24T08:10:17.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.121+0000 7fc0d3400640 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0b806b2d0 0x7fc0b806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.121+0000 7fc0d3400640 1 -- 172.21.15.69:0/3078075041 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc0cc0ce300 con 0x5654b726c5a0 2024-04-24T08:10:17.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.121+0000 7fc0dda7cf40 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5654b729ea20 0x5654b7338330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:17.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.121+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/3078075041 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:1 5.12 5:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e33) -- 0x5654b7338870 con 0x5654b729ea20 2024-04-24T08:10:17.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.121+0000 7fc0da200640 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0b806b2d0 0x7fc0b806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:17.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.121+0000 7fc0db600640 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5654b729ea20 0x5654b7338330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.122+0000 7fc0db600640 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5654b729ea20 0x5654b7338330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.122+0000 7fc0db600640 1 -- 172.21.15.69:0/3078075041 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) ==== 164+0+0 (crc 0 0 0) 0x7fc0d4014030 con 0x5654b729ea20 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:error getting xattr cephfs_data/10000000002.00000000/scrub_tag: (61) No data available 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.122+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/3078075041 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5654b729ea20 msgr2=0x5654b7338330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.122+0000 7fc0dda7cf40 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5654b729ea20 0x5654b7338330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.122+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/3078075041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0b806b2d0 msgr2=0x7fc0b806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.122+0000 7fc0da200640 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0b806b2d0 0x7fc0b806d790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x5654b72b64a0 tx=0x7fc0c4005580 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.123+0000 7fc0dda7cf40 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc0b806b2d0 0x7fc0b806d790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x5654b72b64a0 tx=0x7fc0c4005580 comp rx=0 tx=0).stop 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.123+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/3078075041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b726c5a0 msgr2=0x5654b7332df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.123+0000 7fc0dda7cf40 1 --2- 172.21.15.69:0/3078075041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5654b726c5a0 0x5654b7332df0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0460e0 tx=0x7fc0cc003330 comp rx=0 tx=0).stop 2024-04-24T08:10:17.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.123+0000 7fc0db600640 1 -- 172.21.15.69:0/3078075041 reap_dead start 2024-04-24T08:10:17.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.123+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/3078075041 shutdown_connections 2024-04-24T08:10:17.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.123+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/3078075041 >> 172.21.15.69:0/3078075041 conn(0x5654b729bb50 msgr2=0x5654b729a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:17.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.123+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/3078075041 shutdown_connections 2024-04-24T08:10:17.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:17.123+0000 7fc0dda7cf40 1 -- 172.21.15.69:0/3078075041 wait complete. 2024-04-24T08:10:17.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:10:17.128 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:10:17.128 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:10:17.226 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:10:17.339 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:10:17.339 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:10:17.339 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:10:17.339 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:10:17.398 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:10:17.398 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:10:17.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:17.398 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:10:17.452 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-24T08:10:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:10:17.453 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:10:17.453 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:10:17.454 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:10:17.510 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:10:17.510 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:10:17.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:17.511 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-24T08:10:17.561 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-24T08:10:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:10:17.563 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:10:17.563 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:10:17.563 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:10:17.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:10:17.622 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:10:17.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:17.622 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-24T08:10:17.673 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-24T08:10:17.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:10:17.674 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:10:17.674 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:10:17.674 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:10:17.732 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:10:17.732 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:10:17.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:17.733 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-24T08:10:17.788 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-24T08:10:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:10:17.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:18.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.000+0000 7fbef6e00640 1 Processor -- start 2024-04-24T08:10:18.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.001+0000 7fbef6e00640 1 -- start start 2024-04-24T08:10:18.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.002+0000 7fbef6e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00b2bf0 0x7fbef00ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.002+0000 7fbef6e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef009b120 con 0x7fbef015c8c0 2024-04-24T08:10:18.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.002+0000 7fbef6e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef009b400 con 0x7fbef00d12a0 2024-04-24T08:10:18.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.002+0000 7fbef6e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbef009b6e0 con 0x7fbef00b2bf0 2024-04-24T08:10:18.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.002+0000 7fbeefe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbef015c8c0 0x7fbef00d0b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43544/0 (socket says 172.21.15.69:43544) 2024-04-24T08:10:18.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.002+0000 7fbeefe00640 1 -- 172.21.15.69:0/3009797272 learned_addr learned my addr 172.21.15.69:0/3009797272 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:18.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbef4e00640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00b2bf0 0x7fbef00ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2548542645 0 0) 0x7fbef009b120 con 0x7fbef015c8c0 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbed4003520 con 0x7fbef015c8c0 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2838612724 0 0) 0x7fbed4003520 con 0x7fbef015c8c0 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00b2bf0 msgr2=0x7fbef00ce0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00b2bf0 0x7fbef00ce0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 >> v1:172.21.15.121:6789/0 conn(0x7fbef00d12a0 legacy=0x7fbef00b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef009b9c0 con 0x7fbef015c8c0 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fbee0001670 con 0x7fbef015c8c0 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 >> v1:172.21.15.69:6789/0 conn(0x7fbef015c8c0 legacy=0x7fbef00d0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00b2bf0 0x7fbed40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbed4004010 0x7fbed4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 0x7fbed40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbef009b6e0 con 0x7fbed4004d40 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbed400bc10 con 0x7fbed4004010 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.003+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbed400be10 con 0x7fbef00b2bf0 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.004+0000 7fbef4e00640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 0x7fbed40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.004+0000 7fbee7400640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbed4004010 0x7fbed4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.004+0000 7fbeefe00640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00b2bf0 0x7fbed40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.004+0000 7fbef4e00640 1 -- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00b2bf0 msgr2=0x7fbed40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.004+0000 7fbef4e00640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00b2bf0 0x7fbed40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.004+0000 7fbef4e00640 1 -- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbed4004010 msgr2=0x7fbed4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.004+0000 7fbef4e00640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbed4004010 0x7fbed4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.004+0000 7fbef4e00640 1 -- 172.21.15.69:0/3009797272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed4003520 con 0x7fbed4004d40 2024-04-24T08:10:18.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.005+0000 7fbef4e00640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 0x7fbed40051c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbee8007360 tx=0x7fbed4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3196e67a8db94e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:18.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.005+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8033070 con 0x7fbed4004d40 2024-04-24T08:10:18.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.005+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee802f670 con 0x7fbed4004d40 2024-04-24T08:10:18.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.005+0000 7fbeef400640 1 -- 172.21.15.69:0/3009797272 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee8009be0 con 0x7fbed4004d40 2024-04-24T08:10:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.006+0000 7fbef6e00640 1 -- 172.21.15.69:0/3009797272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 msgr2=0x7fbed40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.006+0000 7fbef6e00640 1 --2- 172.21.15.69:0/3009797272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 0x7fbed40051c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fbee8007360 tx=0x7fbed4005830 comp rx=0 tx=0).stop 2024-04-24T08:10:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.006+0000 7fbef4e00640 1 -- 172.21.15.69:0/3009797272 reap_dead start 2024-04-24T08:10:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.006+0000 7fbef6e00640 1 -- 172.21.15.69:0/3009797272 shutdown_connections 2024-04-24T08:10:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.006+0000 7fbef6e00640 1 -- 172.21.15.69:0/3009797272 >> 172.21.15.69:0/3009797272 conn(0x7fbef00b40a0 msgr2=0x7fbef00b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.006+0000 7fbef6e00640 1 -- 172.21.15.69:0/3009797272 shutdown_connections 2024-04-24T08:10:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.006+0000 7fbef6e00640 1 -- 172.21.15.69:0/3009797272 wait complete. 2024-04-24T08:10:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.008+0000 7fbef6e00640 1 Processor -- start 2024-04-24T08:10:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.008+0000 7fbef6e00640 1 -- start start 2024-04-24T08:10:18.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.009+0000 7fbef6e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 0x7fbef0151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.009+0000 7fbef6e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbef00b1470 0x7fbef00b18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.009+0000 7fbef6e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00af440 0x7fbef00af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.009+0000 7fbef6e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbef015ce20 con 0x7fbed4004d40 2024-04-24T08:10:18.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.009+0000 7fbef6e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbef015cca0 con 0x7fbef00b1470 2024-04-24T08:10:18.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.009+0000 7fbef6e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbef00ce0b0 con 0x7fbef00af440 2024-04-24T08:10:18.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.009+0000 7fbeefe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 0x7fbef0151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.009+0000 7fbeefe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 0x7fbef0151dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56842/0 (socket says 172.21.15.69:56842) 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.009+0000 7fbeefe00640 1 -- 172.21.15.69:0/2018809471 learned_addr learned my addr 172.21.15.69:0/2018809471 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbeefe00640 1 -- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00af440 msgr2=0x7fbef00af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbeefe00640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00af440 0x7fbef00af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbeefe00640 1 -- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbef00b1470 msgr2=0x7fbef00b18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbeefe00640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbef00b1470 0x7fbef00b18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbeefe00640 1 -- 172.21.15.69:0/2018809471 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef00b01c0 con 0x7fbed4004d40 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbeefe00640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 0x7fbef0151dd0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbee000e800 tx=0x7fbee000ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbeecc00640 1 -- 172.21.15.69:0/2018809471 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee000c850 con 0x7fbed4004d40 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbeecc00640 1 -- 172.21.15.69:0/2018809471 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee0018070 con 0x7fbed4004d40 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbef6e00640 1 -- 172.21.15.69:0/2018809471 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef00b04a0 con 0x7fbed4004d40 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbeecc00640 1 -- 172.21.15.69:0/2018809471 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee000cc30 con 0x7fbed4004d40 2024-04-24T08:10:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.010+0000 7fbef6e00640 1 -- 172.21.15.69:0/2018809471 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef00b0a60 con 0x7fbed4004d40 2024-04-24T08:10:18.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.011+0000 7fbef6e00640 1 -- 172.21.15.69:0/2018809471 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbef0068a80 con 0x7fbed4004d40 2024-04-24T08:10:18.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.012+0000 7fbeecc00640 1 -- 172.21.15.69:0/2018809471 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbee001d460 con 0x7fbed4004d40 2024-04-24T08:10:18.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.013+0000 7fbeecc00640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbecc06c920 0x7fbecc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.013+0000 7fbeecc00640 1 -- 172.21.15.69:0/2018809471 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fbee008bb40 con 0x7fbed4004d40 2024-04-24T08:10:18.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.014+0000 7fbee7400640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbecc06c920 0x7fbecc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.015+0000 7fbee7400640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbecc06c920 0x7fbecc06ede0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbed8007fd0 tx=0x7fbed8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:18.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.018+0000 7fbeecc00640 1 -- 172.21.15.69:0/2018809471 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbee0059ed0 con 0x7fbed4004d40 2024-04-24T08:10:18.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.264+0000 7fbef6e00640 1 -- 172.21.15.69:0/2018809471 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbef0075350 con 0x7fbed4004d40 2024-04-24T08:10:18.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.265+0000 7fbeecc00640 1 -- 172.21.15.69:0/2018809471 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5181 (secure 0 0 0) 0x7fbee00597d0 con 0x7fbed4004d40 2024-04-24T08:10:18.265 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:18.266 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2295273533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2295273533},{"type":"v1","addr":"172.21.15.121:6836","nonce":2295273533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1461471292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1461471292},{"type":"v1","addr":"172.21.15.121:6837","nonce":1461471292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/626688145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":626688145},{"type":"v1","addr":"172.21.15.69:6833","nonce":626688145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:09:38.549277+0000","modified":"2024-04-24T08:09:41.567377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/991803145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":991803145},{"type":"v1","addr":"172.21.15.69:6835","nonce":991803145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5241,"qdb_cluster":[5241]},"id":2}]} 2024-04-24T08:10:18.266 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 16 2024-04-24T08:10:18.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 -- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbecc06c920 msgr2=0x7fbecc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbecc06c920 0x7fbecc06ede0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbed8007fd0 tx=0x7fbed8009040 comp rx=0 tx=0).stop 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 -- 172.21.15.69:0/2018809471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 msgr2=0x7fbef0151dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 0x7fbef0151dd0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbee000e800 tx=0x7fbee000ecc0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 -- 172.21.15.69:0/2018809471 shutdown_connections 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbecc06c920 0x7fbecc06ede0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef00af440 0x7fbef00af8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbef00b1470 0x7fbef00b18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 --2- 172.21.15.69:0/2018809471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbed4004d40 0x7fbef0151dd0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 -- 172.21.15.69:0/2018809471 >> 172.21.15.69:0/2018809471 conn(0x7fbef00b40a0 msgr2=0x7fbef0070840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 -- 172.21.15.69:0/2018809471 shutdown_connections 2024-04-24T08:10:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.268+0000 7fbef6e00640 1 -- 172.21.15.69:0/2018809471 wait complete. 2024-04-24T08:10:18.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:18.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.485+0000 7fc33f800640 1 Processor -- start 2024-04-24T08:10:18.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.486+0000 7fc33f800640 1 -- start start 2024-04-24T08:10:18.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.487+0000 7fc33f800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3380d1100 0x7fc3380d1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.487+0000 7fc33f800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3380b3f60 con 0x7fc33806cc00 2024-04-24T08:10:18.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.487+0000 7fc33f800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3380d2750 con 0x7fc3380b3b80 2024-04-24T08:10:18.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.487+0000 7fc33f800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc3380d1aa0 con 0x7fc3380d1100 2024-04-24T08:10:18.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.488+0000 7fc33d800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc33806cc00 0x7fc33806d060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43550/0 (socket says 172.21.15.69:43550) 2024-04-24T08:10:18.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.488+0000 7fc33d800640 1 -- 172.21.15.69:0/3211644058 learned_addr learned my addr 172.21.15.69:0/3211644058 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:18.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.488+0000 7fc337e00640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3380d1100 0x7fc3380d1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.488+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2078291572 0 0) 0x7fc3380b3f60 con 0x7fc33806cc00 2024-04-24T08:10:18.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.488+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc320003520 con 0x7fc33806cc00 2024-04-24T08:10:18.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3604662456 0 0) 0x7fc320003520 con 0x7fc33806cc00 2024-04-24T08:10:18.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3380d1100 msgr2=0x7fc3380d1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:18.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3380d1100 0x7fc3380d1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 >> v1:172.21.15.121:6789/0 conn(0x7fc3380b3b80 legacy=0x7fc3380d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3380d2a30 con 0x7fc33806cc00 2024-04-24T08:10:18.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fc3300019f0 con 0x7fc33806cc00 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 >> v1:172.21.15.69:6789/0 conn(0x7fc33806cc00 legacy=0x7fc33806d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3200039e0 0x7fc320003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3380d1100 0x7fc320004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc320004d40 0x7fc3200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc32000ba70 con 0x7fc320004d40 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc32000bbf0 con 0x7fc3200039e0 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc32000bda0 con 0x7fc3380d1100 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc33d800640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc320004d40 0x7fc3200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc337e00640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3380d1100 0x7fc320004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.489+0000 7fc33ce00640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3200039e0 0x7fc320003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.490+0000 7fc33d800640 1 -- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3380d1100 msgr2=0x7fc320004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.490+0000 7fc33d800640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3380d1100 0x7fc320004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.490+0000 7fc33d800640 1 -- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3200039e0 msgr2=0x7fc320003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.490+0000 7fc33d800640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3200039e0 0x7fc320003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.490+0000 7fc33d800640 1 -- 172.21.15.69:0/3211644058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc320003520 con 0x7fc320004d40 2024-04-24T08:10:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.491+0000 7fc33d800640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc320004d40 0x7fc3200051c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc33000a7a0 tx=0x7fc320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508758f009a3de93 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:18.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.491+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33000a3f0 con 0x7fc320004d40 2024-04-24T08:10:18.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.491+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc33002f070 con 0x7fc320004d40 2024-04-24T08:10:18.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.491+0000 7fc337400640 1 -- 172.21.15.69:0/3211644058 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33002bc00 con 0x7fc320004d40 2024-04-24T08:10:18.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.491+0000 7fc33f800640 1 -- 172.21.15.69:0/3211644058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc320004d40 msgr2=0x7fc3200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.491+0000 7fc33f800640 1 --2- 172.21.15.69:0/3211644058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc320004d40 0x7fc3200051c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc33000a7a0 tx=0x7fc320005830 comp rx=0 tx=0).stop 2024-04-24T08:10:18.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.492+0000 7fc33d800640 1 -- 172.21.15.69:0/3211644058 reap_dead start 2024-04-24T08:10:18.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.492+0000 7fc33f800640 1 -- 172.21.15.69:0/3211644058 shutdown_connections 2024-04-24T08:10:18.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.492+0000 7fc33f800640 1 -- 172.21.15.69:0/3211644058 >> 172.21.15.69:0/3211644058 conn(0x7fc33808bf60 msgr2=0x7fc3380d4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:18.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.492+0000 7fc33f800640 1 -- 172.21.15.69:0/3211644058 shutdown_connections 2024-04-24T08:10:18.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.492+0000 7fc33f800640 1 -- 172.21.15.69:0/3211644058 wait complete. 2024-04-24T08:10:18.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.493+0000 7fc33f800640 1 Processor -- start 2024-04-24T08:10:18.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.494+0000 7fc33f800640 1 -- start start 2024-04-24T08:10:18.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.494+0000 7fc33f800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc320004d40 0x7fc33815c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.494+0000 7fc33f800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3380b87f0 0x7fc338152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.494+0000 7fc33f800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3380b1300 0x7fc3380b1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.494+0000 7fc33f800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc3380d2ff0 con 0x7fc3380b1300 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.494+0000 7fc33f800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc3380d3360 con 0x7fc3380b87f0 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.494+0000 7fc33f800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc3380d0760 con 0x7fc320004d40 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc33d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3380b1300 0x7fc3380b1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc33d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3380b1300 0x7fc3380b1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56870/0 (socket says 172.21.15.69:56870) 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc33d800640 1 -- 172.21.15.69:0/1293854417 learned_addr learned my addr 172.21.15.69:0/1293854417 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc33ce00640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc320004d40 0x7fc33815c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc337e00640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3380b87f0 0x7fc338152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc33d800640 1 -- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc320004d40 msgr2=0x7fc33815c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc33d800640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc320004d40 0x7fc33815c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc33d800640 1 -- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3380b87f0 msgr2=0x7fc338152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc33d800640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3380b87f0 0x7fc338152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc33d800640 1 -- 172.21.15.69:0/1293854417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3380b8c50 con 0x7fc3380b1300 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.495+0000 7fc337e00640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3380b87f0 0x7fc338152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.496+0000 7fc33d800640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3380b1300 0x7fc3380b1780 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc33000a8c0 tx=0x7fc330001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:18.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.496+0000 7fc334c00640 1 -- 172.21.15.69:0/1293854417 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33002bbb0 con 0x7fc3380b1300 2024-04-24T08:10:18.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.496+0000 7fc334c00640 1 -- 172.21.15.69:0/1293854417 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc33002bd50 con 0x7fc3380b1300 2024-04-24T08:10:18.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.496+0000 7fc334c00640 1 -- 172.21.15.69:0/1293854417 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc330034670 con 0x7fc3380b1300 2024-04-24T08:10:18.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.496+0000 7fc33f800640 1 -- 172.21.15.69:0/1293854417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3380b8ed0 con 0x7fc3380b1300 2024-04-24T08:10:18.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.496+0000 7fc33f800640 1 -- 172.21.15.69:0/1293854417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3380b9490 con 0x7fc3380b1300 2024-04-24T08:10:18.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.497+0000 7fc33f800640 1 -- 172.21.15.69:0/1293854417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc338069190 con 0x7fc3380b1300 2024-04-24T08:10:18.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.498+0000 7fc334c00640 1 -- 172.21.15.69:0/1293854417 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc330034810 con 0x7fc3380b1300 2024-04-24T08:10:18.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.499+0000 7fc334c00640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc31806c920 0x7fc31806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.499+0000 7fc334c00640 1 -- 172.21.15.69:0/1293854417 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fc3300abcf0 con 0x7fc3380b1300 2024-04-24T08:10:18.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.500+0000 7fc33ce00640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc31806c920 0x7fc31806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.501+0000 7fc33ce00640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc31806c920 0x7fc31806ede0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc32c001720 tx=0x7fc32c007b70 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:18.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.504+0000 7fc334c00640 1 -- 172.21.15.69:0/1293854417 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3380b8c50 con 0x7fc3380b1300 2024-04-24T08:10:18.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.709+0000 7fc33f800640 1 -- 172.21.15.69:0/1293854417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc338161ef0 con 0x7fc3380b1300 2024-04-24T08:10:18.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.712+0000 7fc334c00640 1 -- 172.21.15.69:0/1293854417 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16645 (secure 0 0 0) 0x7fc33007a080 con 0x7fc3380b1300 2024-04-24T08:10:18.713 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:18.713 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":33,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:40.096668+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:09:36.197195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-24T08:09:37.506730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:18.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.715+0000 7fc33f800640 1 -- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc31806c920 msgr2=0x7fc31806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.715+0000 7fc33f800640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc31806c920 0x7fc31806ede0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc32c001720 tx=0x7fc32c007b70 comp rx=0 tx=0).stop 2024-04-24T08:10:18.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.715+0000 7fc33f800640 1 -- 172.21.15.69:0/1293854417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3380b1300 msgr2=0x7fc3380b1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.715+0000 7fc33f800640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3380b1300 0x7fc3380b1780 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc33000a8c0 tx=0x7fc330001cf0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.715+0000 7fc33f800640 1 -- 172.21.15.69:0/1293854417 shutdown_connections 2024-04-24T08:10:18.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.715+0000 7fc33f800640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc31806c920 0x7fc31806ede0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.715+0000 7fc33f800640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3380b1300 0x7fc3380b1780 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.715+0000 7fc33f800640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3380b87f0 0x7fc338152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.715+0000 7fc33f800640 1 --2- 172.21.15.69:0/1293854417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc320004d40 0x7fc33815c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.716+0000 7fc33f800640 1 -- 172.21.15.69:0/1293854417 >> 172.21.15.69:0/1293854417 conn(0x7fc33808bf60 msgr2=0x7fc3380b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:18.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.716+0000 7fc33f800640 1 -- 172.21.15.69:0/1293854417 shutdown_connections 2024-04-24T08:10:18.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.716+0000 7fc33f800640 1 -- 172.21.15.69:0/1293854417 wait complete. 2024-04-24T08:10:18.736 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:09:36.197195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:09:37.506730+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:10:18.736 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:10:18.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.941+0000 7f205b200640 1 Processor -- start 2024-04-24T08:10:18.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.941+0000 7f205b200640 1 -- start start 2024-04-24T08:10:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.943+0000 7f205b200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f205c071d70 0x7f205c15ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.943+0000 7f205b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205c160510 0x7f205c0cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.943+0000 7f205b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f205c072450 con 0x7f205c1420a0 2024-04-24T08:10:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.943+0000 7f205b200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f205c160960 con 0x7f205c071d70 2024-04-24T08:10:18.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.943+0000 7f205b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f205c0cf1e0 con 0x7f205c160510 2024-04-24T08:10:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.944+0000 7f205a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205c160510 0x7f205c0cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.944+0000 7f2059400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f205c071d70 0x7f205c15ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.944+0000 7f2059400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f205c071d70 0x7f205c15ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:60666/0 (socket says 172.21.15.69:60666) 2024-04-24T08:10:18.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.944+0000 7f2059400640 1 -- 172.21.15.69:0/1379032226 learned_addr learned my addr 172.21.15.69:0/1379032226 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:18.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.944+0000 7f2059400640 1 -- 172.21.15.69:0/1379032226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205c160510 msgr2=0x7f205c0cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:18.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.944+0000 7f2059400640 1 --2- 172.21.15.69:0/1379032226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205c160510 0x7f205c0cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.944+0000 7f2059400640 1 -- 172.21.15.69:0/1379032226 >> v1:172.21.15.121:6789/0 conn(0x7f205c1420a0 legacy=0x7f205c142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.944+0000 7f2059400640 1 -- 172.21.15.69:0/1379032226 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f205c072730 con 0x7f205c071d70 2024-04-24T08:10:18.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.946+0000 7f2059400640 1 --2- 172.21.15.69:0/1379032226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c071d70 0x7f205c15ffd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2048009870 tx=0x7f204802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f60a56ee699fdb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:18.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.946+0000 7f2058a00640 1 -- 172.21.15.69:0/1379032226 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2048037070 con 0x7f205c071d70 2024-04-24T08:10:18.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.946+0000 7f2058a00640 1 -- 172.21.15.69:0/1379032226 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2048032440 con 0x7f205c071d70 2024-04-24T08:10:18.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.946+0000 7f2058a00640 1 -- 172.21.15.69:0/1379032226 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2048032810 con 0x7f205c071d70 2024-04-24T08:10:18.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.947+0000 7f205b200640 1 -- 172.21.15.69:0/1379032226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c071d70 msgr2=0x7f205c15ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.947+0000 7f205b200640 1 --2- 172.21.15.69:0/1379032226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c071d70 0x7f205c15ffd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2048009870 tx=0x7f204802a250 comp rx=0 tx=0).stop 2024-04-24T08:10:18.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.947+0000 7f205b200640 1 -- 172.21.15.69:0/1379032226 shutdown_connections 2024-04-24T08:10:18.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.947+0000 7f205b200640 1 --2- 172.21.15.69:0/1379032226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205c160510 0x7f205c0cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.947+0000 7f205b200640 1 --2- 172.21.15.69:0/1379032226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c071d70 0x7f205c15ffd0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.947+0000 7f205b200640 1 -- 172.21.15.69:0/1379032226 >> 172.21.15.69:0/1379032226 conn(0x7f205c067b80 msgr2=0x7f205c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:18.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.947+0000 7f205b200640 1 -- 172.21.15.69:0/1379032226 shutdown_connections 2024-04-24T08:10:18.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.947+0000 7f205b200640 1 -- 172.21.15.69:0/1379032226 wait complete. 2024-04-24T08:10:18.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.949+0000 7f205b200640 1 Processor -- start 2024-04-24T08:10:18.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.949+0000 7f205b200640 1 -- start start 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f205b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205c071d70 0x7f205c13e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f205b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c1420a0 0x7f205c15b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f205b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c160510 0x7f205c06ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f205b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f205c072cf0 con 0x7f205c1420a0 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f205b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f205c072200 con 0x7f205c160510 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f205b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f205c0cfbb0 con 0x7f205c071d70 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f2059400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c1420a0 0x7f205c15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f2059400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c1420a0 0x7f205c15b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54414/0 (socket says 172.21.15.69:54414) 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f2059400640 1 -- 172.21.15.69:0/4054472020 learned_addr learned my addr 172.21.15.69:0/4054472020 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f2059400640 1 -- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205c071d70 msgr2=0x7f205c13e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f2059400640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205c071d70 0x7f205c13e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f2059400640 1 -- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c160510 msgr2=0x7f205c06ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:18.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f2059400640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c160510 0x7f205c06ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:18.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.951+0000 7f2059400640 1 -- 172.21.15.69:0/4054472020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f205c15bac0 con 0x7f205c1420a0 2024-04-24T08:10:18.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.952+0000 7f2059400640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c1420a0 0x7f205c15b220 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f204802a290 tx=0x7f20480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:18.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.952+0000 7f2052000640 1 -- 172.21.15.69:0/4054472020 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2048037040 con 0x7f205c1420a0 2024-04-24T08:10:18.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.952+0000 7f2052000640 1 -- 172.21.15.69:0/4054472020 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2048042070 con 0x7f205c1420a0 2024-04-24T08:10:18.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.952+0000 7f2052000640 1 -- 172.21.15.69:0/4054472020 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204803b850 con 0x7f205c1420a0 2024-04-24T08:10:18.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.952+0000 7f205b200640 1 -- 172.21.15.69:0/4054472020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f205c15bda0 con 0x7f205c1420a0 2024-04-24T08:10:18.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.952+0000 7f205b200640 1 -- 172.21.15.69:0/4054472020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f205c1574d0 con 0x7f205c1420a0 2024-04-24T08:10:18.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.954+0000 7f205b200640 1 -- 172.21.15.69:0/4054472020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f205c156fe0 con 0x7f205c1420a0 2024-04-24T08:10:18.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.954+0000 7f2052000640 1 -- 172.21.15.69:0/4054472020 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f204803b9f0 con 0x7f205c1420a0 2024-04-24T08:10:18.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.955+0000 7f2052000640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f203c06c8f0 0x7f203c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:18.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.955+0000 7f2052000640 1 -- 172.21.15.69:0/4054472020 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f20480ac350 con 0x7f205c1420a0 2024-04-24T08:10:18.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.956+0000 7f2059e00640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f203c06c8f0 0x7f203c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:18.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.957+0000 7f2059e00640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f203c06c8f0 0x7f203c06edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f204c000f60 tx=0x7f204c0079d0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:18.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:18.960+0000 7f2052000640 1 -- 172.21.15.69:0/4054472020 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f204807a770 con 0x7f205c1420a0 2024-04-24T08:10:19.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.168+0000 7f205b200640 1 -- 172.21.15.69:0/4054472020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f205c075d50 con 0x7f205c1420a0 2024-04-24T08:10:19.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.169+0000 7f2052000640 1 -- 172.21.15.69:0/4054472020 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) ==== 93+0+10 (secure 0 0 0) 0x7f204807a070 con 0x7f205c1420a0 2024-04-24T08:10:19.170 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:10:19.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 -- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f203c06c8f0 msgr2=0x7f203c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f203c06c8f0 0x7f203c06edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f204c000f60 tx=0x7f204c0079d0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 -- 172.21.15.69:0/4054472020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c1420a0 msgr2=0x7f205c15b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c1420a0 0x7f205c15b220 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f204802a290 tx=0x7f20480046c0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 -- 172.21.15.69:0/4054472020 shutdown_connections 2024-04-24T08:10:19.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f203c06c8f0 0x7f203c06edb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f205c1420a0 0x7f205c15b220 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f205c160510 0x7f205c06ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 --2- 172.21.15.69:0/4054472020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f205c071d70 0x7f205c13e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 -- 172.21.15.69:0/4054472020 >> 172.21.15.69:0/4054472020 conn(0x7f205c067b80 msgr2=0x7f205c15e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:19.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 -- 172.21.15.69:0/4054472020 shutdown_connections 2024-04-24T08:10:19.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.172+0000 7f205b200640 1 -- 172.21.15.69:0/4054472020 wait complete. 2024-04-24T08:10:19.191 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:10:19.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.392+0000 7f9c26000640 1 Processor -- start 2024-04-24T08:10:19.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.392+0000 7f9c26000640 1 -- start start 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.393+0000 7f9c26000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9c20155f20 0x7f9c200ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c26000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c2015c9b0 0x7f9c200b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c26000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c200b3f70 0x7f9c200cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c26000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9c2006db90 con 0x7f9c2015c9b0 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c26000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9c20072070 con 0x7f9c20155f20 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c26000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9c200cd6d0 con 0x7f9c200b3f70 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c17e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c2015c9b0 0x7f9c200b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c17e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c2015c9b0 0x7f9c200b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54426/0 (socket says 172.21.15.69:54426) 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c17e00640 1 -- 172.21.15.69:0/1339312133 learned_addr learned my addr 172.21.15.69:0/1339312133 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c17e00640 1 -- 172.21.15.69:0/1339312133 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c200b3f70 msgr2=0x7f9c200cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c1f400640 1 --2- 172.21.15.69:0/1339312133 >> v2:172.21.15.121:3300/0 conn(0x7f9c20155f20 0x7f9c200ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c1fe00640 1 --2- 172.21.15.69:0/1339312133 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c200b3f70 0x7f9c200cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c17e00640 1 --2- 172.21.15.69:0/1339312133 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c200b3f70 0x7f9c200cf360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c17e00640 1 -- 172.21.15.69:0/1339312133 >> v2:172.21.15.121:3300/0 conn(0x7f9c20155f20 msgr2=0x7f9c200ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:19.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c17e00640 1 --2- 172.21.15.69:0/1339312133 >> v2:172.21.15.121:3300/0 conn(0x7f9c20155f20 0x7f9c200ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.394+0000 7f9c17e00640 1 -- 172.21.15.69:0/1339312133 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c200cd9b0 con 0x7f9c2015c9b0 2024-04-24T08:10:19.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.395+0000 7f9c17e00640 1 --2- 172.21.15.69:0/1339312133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c2015c9b0 0x7f9c200b3a30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9c20159870 tx=0x7f9c0802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ff286dc375752e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:19.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.396+0000 7f9c1ea00640 1 -- 172.21.15.69:0/1339312133 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08037070 con 0x7f9c2015c9b0 2024-04-24T08:10:19.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.396+0000 7f9c1ea00640 1 -- 172.21.15.69:0/1339312133 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c08032440 con 0x7f9c2015c9b0 2024-04-24T08:10:19.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.396+0000 7f9c1ea00640 1 -- 172.21.15.69:0/1339312133 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c08032810 con 0x7f9c2015c9b0 2024-04-24T08:10:19.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.396+0000 7f9c26000640 1 -- 172.21.15.69:0/1339312133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c2015c9b0 msgr2=0x7f9c200b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.396+0000 7f9c26000640 1 --2- 172.21.15.69:0/1339312133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c2015c9b0 0x7f9c200b3a30 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f9c20159870 tx=0x7f9c0802a270 comp rx=0 tx=0).stop 2024-04-24T08:10:19.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.396+0000 7f9c26000640 1 -- 172.21.15.69:0/1339312133 shutdown_connections 2024-04-24T08:10:19.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.397+0000 7f9c26000640 1 --2- 172.21.15.69:0/1339312133 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c200b3f70 0x7f9c200cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.397+0000 7f9c26000640 1 --2- 172.21.15.69:0/1339312133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c2015c9b0 0x7f9c200b3a30 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.397+0000 7f9c26000640 1 --2- 172.21.15.69:0/1339312133 >> v2:172.21.15.121:3300/0 conn(0x7f9c20155f20 0x7f9c200ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.397+0000 7f9c26000640 1 -- 172.21.15.69:0/1339312133 >> 172.21.15.69:0/1339312133 conn(0x7f9c200d0350 msgr2=0x7f9c200d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:19.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.397+0000 7f9c26000640 1 -- 172.21.15.69:0/1339312133 shutdown_connections 2024-04-24T08:10:19.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.397+0000 7f9c26000640 1 -- 172.21.15.69:0/1339312133 wait complete. 2024-04-24T08:10:19.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.398+0000 7f9c26000640 1 Processor -- start 2024-04-24T08:10:19.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.398+0000 7f9c26000640 1 -- start start 2024-04-24T08:10:19.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.399+0000 7f9c26000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c200b3f70 0x7f9c200b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.399+0000 7f9c26000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c20155f20 0x7f9c200b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.399+0000 7f9c26000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c2015c9b0 0x7f9c200ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.399+0000 7f9c26000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c200cd2a0 con 0x7f9c200b3f70 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.399+0000 7f9c26000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9c20071d00 con 0x7f9c2015c9b0 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.399+0000 7f9c26000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9c200cd420 con 0x7f9c20155f20 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.400+0000 7f9c1f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c200b3f70 0x7f9c200b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.400+0000 7f9c1f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c200b3f70 0x7f9c200b0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54428/0 (socket says 172.21.15.69:54428) 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.400+0000 7f9c1f400640 1 -- 172.21.15.69:0/2006622329 learned_addr learned my addr 172.21.15.69:0/2006622329 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.400+0000 7f9c17e00640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c20155f20 0x7f9c200b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.400+0000 7f9c1fe00640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c2015c9b0 0x7f9c200ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.400+0000 7f9c1f400640 1 -- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c20155f20 msgr2=0x7f9c200b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.400+0000 7f9c1f400640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c20155f20 0x7f9c200b1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.400+0000 7f9c1f400640 1 -- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c2015c9b0 msgr2=0x7f9c200ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.400+0000 7f9c1f400640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c2015c9b0 0x7f9c200ad240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.400+0000 7f9c1f400640 1 -- 172.21.15.69:0/2006622329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c20140680 con 0x7f9c200b3f70 2024-04-24T08:10:19.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.401+0000 7f9c1f400640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c200b3f70 0x7f9c200b0f00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000c820 tx=0x7f9c1000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:19.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.401+0000 7f9c17400640 1 -- 172.21.15.69:0/2006622329 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c10015070 con 0x7f9c200b3f70 2024-04-24T08:10:19.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.401+0000 7f9c17400640 1 -- 172.21.15.69:0/2006622329 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c10004240 con 0x7f9c200b3f70 2024-04-24T08:10:19.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.401+0000 7f9c17400640 1 -- 172.21.15.69:0/2006622329 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c1000ce10 con 0x7f9c200b3f70 2024-04-24T08:10:19.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.401+0000 7f9c26000640 1 -- 172.21.15.69:0/2006622329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c201400c0 con 0x7f9c200b3f70 2024-04-24T08:10:19.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.401+0000 7f9c26000640 1 -- 172.21.15.69:0/2006622329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c20153e30 con 0x7f9c200b3f70 2024-04-24T08:10:19.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.402+0000 7f9c26000640 1 -- 172.21.15.69:0/2006622329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c200cf630 con 0x7f9c200b3f70 2024-04-24T08:10:19.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.403+0000 7f9c17400640 1 -- 172.21.15.69:0/2006622329 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9c10007830 con 0x7f9c200b3f70 2024-04-24T08:10:19.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.404+0000 7f9c17400640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9bfc06c940 0x7f9bfc06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.404+0000 7f9c17400640 1 -- 172.21.15.69:0/2006622329 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f9c1008ba00 con 0x7f9c200b3f70 2024-04-24T08:10:19.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.405+0000 7f9c17e00640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9bfc06c940 0x7f9bfc06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.406+0000 7f9c17e00640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9bfc06c940 0x7f9bfc06ee00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9c08034000 tx=0x7f9c0802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:19.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.408+0000 7f9c17400640 1 -- 172.21.15.69:0/2006622329 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9c200cf630 con 0x7f9c200b3f70 2024-04-24T08:10:19.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.607+0000 7f9c26000640 1 -- 172.21.15.69:0/2006622329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9c20068a80 con 0x7f9c200b3f70 2024-04-24T08:10:19.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.608+0000 7f9c17400640 1 -- 172.21.15.69:0/2006622329 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) ==== 104+0+11 (secure 0 0 0) 0x7f9c10059ea0 con 0x7f9c200b3f70 2024-04-24T08:10:19.608 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.610+0000 7f9c26000640 1 -- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9bfc06c940 msgr2=0x7f9bfc06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.610+0000 7f9c26000640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9bfc06c940 0x7f9bfc06ee00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9c08034000 tx=0x7f9c0802e000 comp rx=0 tx=0).stop 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.610+0000 7f9c26000640 1 -- 172.21.15.69:0/2006622329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c200b3f70 msgr2=0x7f9c200b0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.610+0000 7f9c26000640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c200b3f70 0x7f9c200b0f00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000c820 tx=0x7f9c1000cce0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.611+0000 7f9c26000640 1 -- 172.21.15.69:0/2006622329 shutdown_connections 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.611+0000 7f9c26000640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9bfc06c940 0x7f9bfc06ee00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.611+0000 7f9c26000640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c2015c9b0 0x7f9c200ad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.611+0000 7f9c26000640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c20155f20 0x7f9c200b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.611+0000 7f9c26000640 1 --2- 172.21.15.69:0/2006622329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c200b3f70 0x7f9c200b0f00 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.611+0000 7f9c26000640 1 -- 172.21.15.69:0/2006622329 >> 172.21.15.69:0/2006622329 conn(0x7f9c200d0350 msgr2=0x7f9c2015bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:19.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.611+0000 7f9c26000640 1 -- 172.21.15.69:0/2006622329 shutdown_connections 2024-04-24T08:10:19.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.611+0000 7f9c26000640 1 -- 172.21.15.69:0/2006622329 wait complete. 2024-04-24T08:10:19.630 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:10:19.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.834+0000 7f6667600640 1 Processor -- start 2024-04-24T08:10:19.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.835+0000 7f6667600640 1 -- start start 2024-04-24T08:10:19.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.836+0000 7f6667600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6660155f20 0x7f66600ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.836+0000 7f6667600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66600cd6d0 con 0x7f66600cf360 2024-04-24T08:10:19.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.836+0000 7f6667600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66600cd9b0 con 0x7f666015c9b0 2024-04-24T08:10:19.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.836+0000 7f6667600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f666006db90 con 0x7f6660155f20 2024-04-24T08:10:19.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.836+0000 7f6665600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f66600cf360 0x7f6660072500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51156/0 (socket says 172.21.15.69:51156) 2024-04-24T08:10:19.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.836+0000 7f6665600640 1 -- 172.21.15.69:0/707607124 learned_addr learned my addr 172.21.15.69:0/707607124 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f6664c00640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6660155f20 0x7f66600ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 241529104 0 0) 0x7f66600cd6d0 con 0x7f66600cf360 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6644003520 con 0x7f66600cf360 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3658585561 0 0) 0x7f6644003520 con 0x7f66600cf360 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6660155f20 msgr2=0x7f66600ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6660155f20 0x7f66600ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 >> v1:172.21.15.121:6789/0 conn(0x7f666015c9b0 legacy=0x7f66600b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66600cdc90 con 0x7f66600cf360 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f66580019f0 con 0x7f66600cf360 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 >> v1:172.21.15.69:6789/0 conn(0x7f66600cf360 legacy=0x7f6660072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6660155f20 0x7f66440039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6644003f60 0x7f6644004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.837+0000 7f665fe00640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6644004d40 0x7f66440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.838+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f664400bb00 con 0x7f6644003f60 2024-04-24T08:10:19.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.838+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f664400bcb0 con 0x7f6644004d40 2024-04-24T08:10:19.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.838+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f664400e0c0 con 0x7f6660155f20 2024-04-24T08:10:19.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.838+0000 7f6657e00640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6644003f60 0x7f6644004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.838+0000 7f6657e00640 1 -- 172.21.15.69:0/707607124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6660155f20 msgr2=0x7f66440039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:19.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.838+0000 7f6664c00640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6660155f20 0x7f66440039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.838+0000 7f6657e00640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6660155f20 0x7f66440039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.838+0000 7f6657e00640 1 -- 172.21.15.69:0/707607124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6644004d40 msgr2=0x7f66440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:19.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.838+0000 7f6657e00640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6644004d40 0x7f66440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.838+0000 7f6657e00640 1 -- 172.21.15.69:0/707607124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6644003520 con 0x7f6644003f60 2024-04-24T08:10:19.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.839+0000 7f6657e00640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6644003f60 0x7f6644004800 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6648001720 tx=0x7f6644005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49084a3a5a8c82a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.839+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6648001370 con 0x7f6644003f60 2024-04-24T08:10:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.839+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66480052d0 con 0x7f6644003f60 2024-04-24T08:10:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.839+0000 7f665fe00640 1 -- 172.21.15.69:0/707607124 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664802abc0 con 0x7f6644003f60 2024-04-24T08:10:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.840+0000 7f6667600640 1 -- 172.21.15.69:0/707607124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6644003f60 msgr2=0x7f6644004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.840+0000 7f6667600640 1 --2- 172.21.15.69:0/707607124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6644003f60 0x7f6644004800 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6648001720 tx=0x7f6644005830 comp rx=0 tx=0).stop 2024-04-24T08:10:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.840+0000 7f6665600640 1 -- 172.21.15.69:0/707607124 reap_dead start 2024-04-24T08:10:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.840+0000 7f6667600640 1 -- 172.21.15.69:0/707607124 shutdown_connections 2024-04-24T08:10:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.840+0000 7f6667600640 1 -- 172.21.15.69:0/707607124 >> 172.21.15.69:0/707607124 conn(0x7f66600d0350 msgr2=0x7f66600d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.840+0000 7f6667600640 1 -- 172.21.15.69:0/707607124 shutdown_connections 2024-04-24T08:10:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.840+0000 7f6667600640 1 -- 172.21.15.69:0/707607124 wait complete. 2024-04-24T08:10:19.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.842+0000 7f6667600640 1 Processor -- start 2024-04-24T08:10:19.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.842+0000 7f6667600640 1 -- start start 2024-04-24T08:10:19.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.843+0000 7f6667600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6644003f60 0x7f6660150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.843+0000 7f6667600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66601589d0 0x7f6660158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.843+0000 7f6667600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66600b1280 0x7f66600b1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.843+0000 7f6667600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f66600b4140 con 0x7f66601589d0 2024-04-24T08:10:19.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.843+0000 7f6667600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6660072070 con 0x7f6644003f60 2024-04-24T08:10:19.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.843+0000 7f6667600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6660072c10 con 0x7f66600b1280 2024-04-24T08:10:19.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.843+0000 7f6657e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66601589d0 0x7f6660158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.843+0000 7f6657e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66601589d0 0x7f6660158e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54438/0 (socket says 172.21.15.69:54438) 2024-04-24T08:10:19.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.843+0000 7f6657e00640 1 -- 172.21.15.69:0/201612272 learned_addr learned my addr 172.21.15.69:0/201612272 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:19.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f6664c00640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6644003f60 0x7f6660150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f6665600640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66600b1280 0x7f66600b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f6657e00640 1 -- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66600b1280 msgr2=0x7f66600b1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f6657e00640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66600b1280 0x7f66600b1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f6657e00640 1 -- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6644003f60 msgr2=0x7f6660150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f6657e00640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6644003f60 0x7f6660150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f6657e00640 1 -- 172.21.15.69:0/201612272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66601570f0 con 0x7f66601589d0 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f6664c00640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6644003f60 0x7f6660150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f6657e00640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66601589d0 0x7f6660158e50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6648008fd0 tx=0x7f6648001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f665d600640 1 -- 172.21.15.69:0/201612272 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6648037040 con 0x7f66601589d0 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f665d600640 1 -- 172.21.15.69:0/201612272 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6648005480 con 0x7f66601589d0 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f6667600640 1 -- 172.21.15.69:0/201612272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66601573d0 con 0x7f66601589d0 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.844+0000 7f665d600640 1 -- 172.21.15.69:0/201612272 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6648007a30 con 0x7f66601589d0 2024-04-24T08:10:19.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.845+0000 7f6667600640 1 -- 172.21.15.69:0/201612272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f666006df20 con 0x7f66601589d0 2024-04-24T08:10:19.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.846+0000 7f6667600640 1 -- 172.21.15.69:0/201612272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6660068a80 con 0x7f66601589d0 2024-04-24T08:10:19.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.847+0000 7f665d600640 1 -- 172.21.15.69:0/201612272 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6648044020 con 0x7f66601589d0 2024-04-24T08:10:19.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.847+0000 7f665d600640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f663c06c920 0x7f663c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.848+0000 7f665d600640 1 -- 172.21.15.69:0/201612272 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f66480ab6a0 con 0x7f66601589d0 2024-04-24T08:10:19.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.848+0000 7f6664c00640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f663c06c920 0x7f663c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.849+0000 7f6664c00640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f663c06c920 0x7f663c06ede0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f66500099e0 tx=0x7f6650008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:19.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:19.852+0000 7f665d600640 1 -- 172.21.15.69:0/201612272 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f664804b020 con 0x7f66601589d0 2024-04-24T08:10:20.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.052+0000 7f6667600640 1 -- 172.21.15.69:0/201612272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6660151d90 con 0x7f66601589d0 2024-04-24T08:10:20.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.053+0000 7f665d600640 1 -- 172.21.15.69:0/201612272 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) ==== 100+0+11 (secure 0 0 0) 0x7f6648079a30 con 0x7f66601589d0 2024-04-24T08:10:20.053 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 -- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f663c06c920 msgr2=0x7f663c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f663c06c920 0x7f663c06ede0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f66500099e0 tx=0x7f6650008040 comp rx=0 tx=0).stop 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 -- 172.21.15.69:0/201612272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66601589d0 msgr2=0x7f6660158e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66601589d0 0x7f6660158e50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f6648008fd0 tx=0x7f6648001120 comp rx=0 tx=0).stop 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 -- 172.21.15.69:0/201612272 shutdown_connections 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f663c06c920 0x7f663c06ede0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66601589d0 0x7f6660158e50 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66600b1280 0x7f66600b1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 --2- 172.21.15.69:0/201612272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6644003f60 0x7f6660150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 -- 172.21.15.69:0/201612272 >> 172.21.15.69:0/201612272 conn(0x7f66600d0350 msgr2=0x7f66600b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 -- 172.21.15.69:0/201612272 shutdown_connections 2024-04-24T08:10:20.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.055+0000 7f6667600640 1 -- 172.21.15.69:0/201612272 wait complete. 2024-04-24T08:10:20.076 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:20.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.277+0000 7f2b8e200640 1 Processor -- start 2024-04-24T08:10:20.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.277+0000 7f2b8e200640 1 -- start start 2024-04-24T08:10:20.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.278+0000 7f2b8e200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b8806db90 0x7f2b88073530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.278+0000 7f2b8e200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2b8815b980 0x7f2b8815be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.278+0000 7f2b8e200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b88073a70 con 0x7f2b881523f0 2024-04-24T08:10:20.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.278+0000 7f2b8e200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2b880aeac0 con 0x7f2b8815b980 2024-04-24T08:10:20.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.278+0000 7f2b8e200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2b880aeda0 con 0x7f2b8806db90 2024-04-24T08:10:20.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.278+0000 7f2b8d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2b8815b980 0x7f2b8815be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.278+0000 7f2b8d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2b8815b980 0x7f2b8815be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54452/0 (socket says 172.21.15.69:54452) 2024-04-24T08:10:20.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.278+0000 7f2b8d800640 1 -- 172.21.15.69:0/541554422 learned_addr learned my addr 172.21.15.69:0/541554422 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:20.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.279+0000 7f2b87e00640 1 --2- 172.21.15.69:0/541554422 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b8806db90 0x7f2b88073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.279+0000 7f2b8d800640 1 -- 172.21.15.69:0/541554422 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b8806db90 msgr2=0x7f2b88073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:20.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.279+0000 7f2b8d800640 1 --2- 172.21.15.69:0/541554422 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b8806db90 0x7f2b88073530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.279+0000 7f2b8d800640 1 -- 172.21.15.69:0/541554422 >> v1:172.21.15.121:6789/0 conn(0x7f2b881523f0 legacy=0x7f2b88072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.279+0000 7f2b8d800640 1 -- 172.21.15.69:0/541554422 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b880af080 con 0x7f2b8815b980 2024-04-24T08:10:20.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.280+0000 7f2b8d800640 1 --2- 172.21.15.69:0/541554422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8815b980 0x7f2b8815be10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009e50 tx=0x7f2b8002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e12fc5ce8def95 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:20.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b87400640 1 -- 172.21.15.69:0/541554422 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b80036070 con 0x7f2b8815b980 2024-04-24T08:10:20.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b87400640 1 -- 172.21.15.69:0/541554422 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b80031600 con 0x7f2b8815b980 2024-04-24T08:10:20.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b87400640 1 -- 172.21.15.69:0/541554422 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b800319d0 con 0x7f2b8815b980 2024-04-24T08:10:20.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b8e200640 1 -- 172.21.15.69:0/541554422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8815b980 msgr2=0x7f2b8815be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b8e200640 1 --2- 172.21.15.69:0/541554422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8815b980 0x7f2b8815be10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009e50 tx=0x7f2b8002a680 comp rx=0 tx=0).stop 2024-04-24T08:10:20.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b8e200640 1 -- 172.21.15.69:0/541554422 shutdown_connections 2024-04-24T08:10:20.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b8e200640 1 --2- 172.21.15.69:0/541554422 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8815b980 0x7f2b8815be10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b8e200640 1 --2- 172.21.15.69:0/541554422 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b8806db90 0x7f2b88073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b8e200640 1 -- 172.21.15.69:0/541554422 >> 172.21.15.69:0/541554422 conn(0x7f2b880d0160 msgr2=0x7f2b880d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:20.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b8e200640 1 -- 172.21.15.69:0/541554422 shutdown_connections 2024-04-24T08:10:20.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.281+0000 7f2b8e200640 1 -- 172.21.15.69:0/541554422 wait complete. 2024-04-24T08:10:20.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.283+0000 7f2b8e200640 1 Processor -- start 2024-04-24T08:10:20.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.283+0000 7f2b8e200640 1 -- start start 2024-04-24T08:10:20.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.284+0000 7f2b8e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8806db90 0x7f2b880b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.284+0000 7f2b8e200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b881523f0 0x7f2b8814a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.284+0000 7f2b8e200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b8815b980 0x7f2b88099280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.284+0000 7f2b8e200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8806dfd0 con 0x7f2b8806db90 2024-04-24T08:10:20.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.284+0000 7f2b8e200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2b880b3190 con 0x7f2b8815b980 2024-04-24T08:10:20.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.284+0000 7f2b8e200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2b881527d0 con 0x7f2b881523f0 2024-04-24T08:10:20.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.284+0000 7f2b8ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8806db90 0x7f2b880b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b8ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8806db90 0x7f2b880b1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54460/0 (socket says 172.21.15.69:54460) 2024-04-24T08:10:20.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b8ce00640 1 -- 172.21.15.69:0/450534453 learned_addr learned my addr 172.21.15.69:0/450534453 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b87e00640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b881523f0 0x7f2b8814a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b8d800640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b8815b980 0x7f2b88099280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b8ce00640 1 -- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b881523f0 msgr2=0x7f2b8814a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b8ce00640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b881523f0 0x7f2b8814a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b8ce00640 1 -- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b8815b980 msgr2=0x7f2b88099280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b8ce00640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b8815b980 0x7f2b88099280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b8ce00640 1 -- 172.21.15.69:0/450534453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b880b0a70 con 0x7f2b8806db90 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b8ce00640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8806db90 0x7f2b880b1830 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00c400 tx=0x7f2b7c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.285+0000 7f2b84c00640 1 -- 172.21.15.69:0/450534453 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c0052d0 con 0x7f2b8806db90 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.286+0000 7f2b84c00640 1 -- 172.21.15.69:0/450534453 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b7c005490 con 0x7f2b8806db90 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.286+0000 7f2b84c00640 1 -- 172.21.15.69:0/450534453 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c007260 con 0x7f2b8806db90 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.286+0000 7f2b8e200640 1 -- 172.21.15.69:0/450534453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b880b0d50 con 0x7f2b8806db90 2024-04-24T08:10:20.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.286+0000 7f2b8e200640 1 -- 172.21.15.69:0/450534453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b880b0760 con 0x7f2b8806db90 2024-04-24T08:10:20.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.287+0000 7f2b8e200640 1 -- 172.21.15.69:0/450534453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b88075d50 con 0x7f2b8806db90 2024-04-24T08:10:20.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.288+0000 7f2b84c00640 1 -- 172.21.15.69:0/450534453 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2b7c007400 con 0x7f2b8806db90 2024-04-24T08:10:20.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.288+0000 7f2b84c00640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2b6806c8f0 0x7f2b6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.289+0000 7f2b84c00640 1 -- 172.21.15.69:0/450534453 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f2b7c08b120 con 0x7f2b8806db90 2024-04-24T08:10:20.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.289+0000 7f2b87e00640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2b6806c8f0 0x7f2b6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.290+0000 7f2b87e00640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2b6806c8f0 0x7f2b6806edb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2b881f76c0 tx=0x7f2b74008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:20.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.293+0000 7f2b84c00640 1 -- 172.21.15.69:0/450534453 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b7c0594b0 con 0x7f2b8806db90 2024-04-24T08:10:20.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.538+0000 7f2b8e200640 1 -- 172.21.15.69:0/450534453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b88072a00 con 0x7f2b8806db90 2024-04-24T08:10:20.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.539+0000 7f2b84c00640 1 -- 172.21.15.69:0/450534453 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5181 (secure 0 0 0) 0x7f2b7c058db0 con 0x7f2b8806db90 2024-04-24T08:10:20.539 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:20.539 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2295273533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2295273533},{"type":"v1","addr":"172.21.15.121:6836","nonce":2295273533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1461471292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1461471292},{"type":"v1","addr":"172.21.15.121:6837","nonce":1461471292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/626688145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":626688145},{"type":"v1","addr":"172.21.15.69:6833","nonce":626688145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:09:38.549277+0000","modified":"2024-04-24T08:09:41.567377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/991803145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":991803145},{"type":"v1","addr":"172.21.15.69:6835","nonce":991803145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5241,"qdb_cluster":[5241]},"id":2}]} 2024-04-24T08:10:20.540 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 16 2024-04-24T08:10:20.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.541+0000 7f2b8e200640 1 -- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2b6806c8f0 msgr2=0x7f2b6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.541+0000 7f2b8e200640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2b6806c8f0 0x7f2b6806edb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2b881f76c0 tx=0x7f2b74008040 comp rx=0 tx=0).stop 2024-04-24T08:10:20.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.541+0000 7f2b8e200640 1 -- 172.21.15.69:0/450534453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8806db90 msgr2=0x7f2b880b1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.541+0000 7f2b8e200640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8806db90 0x7f2b880b1830 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00c400 tx=0x7f2b7c00c8c0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.542+0000 7f2b8e200640 1 -- 172.21.15.69:0/450534453 shutdown_connections 2024-04-24T08:10:20.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.542+0000 7f2b8e200640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2b6806c8f0 0x7f2b6806edb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.542+0000 7f2b8e200640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b8815b980 0x7f2b88099280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.542+0000 7f2b8e200640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2b881523f0 0x7f2b8814a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.542+0000 7f2b8e200640 1 --2- 172.21.15.69:0/450534453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2b8806db90 0x7f2b880b1830 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.542+0000 7f2b8e200640 1 -- 172.21.15.69:0/450534453 >> 172.21.15.69:0/450534453 conn(0x7f2b880d0160 msgr2=0x7f2b880b41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:20.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.542+0000 7f2b8e200640 1 -- 172.21.15.69:0/450534453 shutdown_connections 2024-04-24T08:10:20.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.542+0000 7f2b8e200640 1 -- 172.21.15.69:0/450534453 wait complete. 2024-04-24T08:10:20.561 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:20.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.756+0000 7feec6e00640 1 Processor -- start 2024-04-24T08:10:20.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.756+0000 7feec6e00640 1 -- start start 2024-04-24T08:10:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.758+0000 7feec6e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7feec015ea10 0x7feec00d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.758+0000 7feec6e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7feec01631a0 0x7feec00af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.758+0000 7feec6e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feec00adab0 con 0x7feec00af7d0 2024-04-24T08:10:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.758+0000 7feec6e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7feec00afbb0 con 0x7feec015ea10 2024-04-24T08:10:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.758+0000 7feec6e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7feec00add50 con 0x7feec01631a0 2024-04-24T08:10:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.758+0000 7feebfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7feec015ea10 0x7feec00d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.758+0000 7feec4e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7feec00af7d0 0x7feec0071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51172/0 (socket says 172.21.15.69:51172) 2024-04-24T08:10:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.758+0000 7feec4e00640 1 -- 172.21.15.69:0/2110000314 learned_addr learned my addr 172.21.15.69:0/2110000314 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.759+0000 7feebf400640 1 --2- 172.21.15.69:0/2110000314 >> v2:172.21.15.121:3300/0 conn(0x7feec01631a0 0x7feec00af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.759+0000 7feebfe00640 1 -- 172.21.15.69:0/2110000314 >> v2:172.21.15.121:3300/0 conn(0x7feec01631a0 msgr2=0x7feec00af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.759+0000 7feebfe00640 1 --2- 172.21.15.69:0/2110000314 >> v2:172.21.15.121:3300/0 conn(0x7feec01631a0 0x7feec00af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.759+0000 7feebfe00640 1 -- 172.21.15.69:0/2110000314 >> v1:172.21.15.69:6789/0 conn(0x7feec00af7d0 legacy=0x7feec0071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.759+0000 7feebfe00640 1 -- 172.21.15.69:0/2110000314 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec00ae030 con 0x7feec015ea10 2024-04-24T08:10:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.760+0000 7feebfe00640 1 --2- 172.21.15.69:0/2110000314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec015ea10 0x7feec00d0830 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7feec006f5d0 tx=0x7feeb002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d8532ddb18e941 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.761+0000 7feebea00640 1 -- 172.21.15.69:0/2110000314 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0037070 con 0x7feec015ea10 2024-04-24T08:10:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.761+0000 7feebea00640 1 -- 172.21.15.69:0/2110000314 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb0032440 con 0x7feec015ea10 2024-04-24T08:10:20.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.761+0000 7feebea00640 1 -- 172.21.15.69:0/2110000314 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0032810 con 0x7feec015ea10 2024-04-24T08:10:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.761+0000 7feec6e00640 1 -- 172.21.15.69:0/2110000314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec015ea10 msgr2=0x7feec00d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.761+0000 7feec6e00640 1 --2- 172.21.15.69:0/2110000314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec015ea10 0x7feec00d0830 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7feec006f5d0 tx=0x7feeb002a1a0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.761+0000 7feec6e00640 1 -- 172.21.15.69:0/2110000314 shutdown_connections 2024-04-24T08:10:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.761+0000 7feec6e00640 1 --2- 172.21.15.69:0/2110000314 >> v2:172.21.15.121:3300/0 conn(0x7feec01631a0 0x7feec00af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.761+0000 7feec6e00640 1 --2- 172.21.15.69:0/2110000314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec015ea10 0x7feec00d0830 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.761+0000 7feec6e00640 1 -- 172.21.15.69:0/2110000314 >> 172.21.15.69:0/2110000314 conn(0x7feec0067d90 msgr2=0x7feec006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.762+0000 7feec6e00640 1 -- 172.21.15.69:0/2110000314 shutdown_connections 2024-04-24T08:10:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.762+0000 7feec6e00640 1 -- 172.21.15.69:0/2110000314 wait complete. 2024-04-24T08:10:20.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.763+0000 7feec6e00640 1 Processor -- start 2024-04-24T08:10:20.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.764+0000 7feec6e00640 1 -- start start 2024-04-24T08:10:20.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.765+0000 7feec6e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00af7d0 0x7feec0156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.765+0000 7feec6e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feec015ea10 0x7feec0150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.765+0000 7feec6e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec01631a0 0x7feec0150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.765+0000 7feec6e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feec00ae8d0 con 0x7feec01631a0 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.765+0000 7feec6e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7feec00ae5f0 con 0x7feec00af7d0 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.765+0000 7feec6e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7feec00aedf0 con 0x7feec015ea10 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.765+0000 7feec4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec01631a0 0x7feec0150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.765+0000 7feec4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec01631a0 0x7feec0150e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54472/0 (socket says 172.21.15.69:54472) 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.765+0000 7feec4e00640 1 -- 172.21.15.69:0/1771191506 learned_addr learned my addr 172.21.15.69:0/1771191506 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.765+0000 7feebfe00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00af7d0 0x7feec0156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feec4e00640 1 -- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feec015ea10 msgr2=0x7feec0150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feec4e00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feec015ea10 0x7feec0150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feec4e00640 1 -- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00af7d0 msgr2=0x7feec0156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feec4e00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00af7d0 0x7feec0156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feec4e00640 1 -- 172.21.15.69:0/1771191506 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feec01f4b20 con 0x7feec01631a0 2024-04-24T08:10:20.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feebfe00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00af7d0 0x7feec0156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:20.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feec4e00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec01631a0 0x7feec0150e90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7feeb800cae0 tx=0x7feeb800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:20.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feeb7e00640 1 -- 172.21.15.69:0/1771191506 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8017070 con 0x7feec01631a0 2024-04-24T08:10:20.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feeb7e00640 1 -- 172.21.15.69:0/1771191506 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb800ac40 con 0x7feec01631a0 2024-04-24T08:10:20.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feeb7e00640 1 -- 172.21.15.69:0/1771191506 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb8012400 con 0x7feec01631a0 2024-04-24T08:10:20.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feec6e00640 1 -- 172.21.15.69:0/1771191506 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feec0153570 con 0x7feec01631a0 2024-04-24T08:10:20.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.766+0000 7feec6e00640 1 -- 172.21.15.69:0/1771191506 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feec01f50c0 con 0x7feec01631a0 2024-04-24T08:10:20.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.768+0000 7feec6e00640 1 -- 172.21.15.69:0/1771191506 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feec0068a80 con 0x7feec01631a0 2024-04-24T08:10:20.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.769+0000 7feeb7e00640 1 -- 172.21.15.69:0/1771191506 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feeb801a070 con 0x7feec01631a0 2024-04-24T08:10:20.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.770+0000 7feeb7e00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feea006c8f0 0x7feea006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.770+0000 7feeb7e00640 1 -- 172.21.15.69:0/1771191506 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7feeb808b890 con 0x7feec01631a0 2024-04-24T08:10:20.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.770+0000 7feebfe00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feea006c8f0 0x7feea006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.771+0000 7feebfe00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feea006c8f0 0x7feea006edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feeb0034000 tx=0x7feeb002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:20.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.775+0000 7feeb7e00640 1 -- 172.21.15.69:0/1771191506 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feeb8059d50 con 0x7feec01631a0 2024-04-24T08:10:20.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.977+0000 7feec6e00640 1 -- 172.21.15.69:0/1771191506 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feec006ddd0 con 0x7feec01631a0 2024-04-24T08:10:20.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.979+0000 7feeb7e00640 1 -- 172.21.15.69:0/1771191506 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16645 (secure 0 0 0) 0x7feeb8059650 con 0x7feec01631a0 2024-04-24T08:10:20.980 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:20.980 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":33,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:40.096668+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:09:36.197195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-24T08:09:37.506730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:20.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 -- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feea006c8f0 msgr2=0x7feea006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feea006c8f0 0x7feea006edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feeb0034000 tx=0x7feeb002e000 comp rx=0 tx=0).stop 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 -- 172.21.15.69:0/1771191506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec01631a0 msgr2=0x7feec0150e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec01631a0 0x7feec0150e90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7feeb800cae0 tx=0x7feeb800a060 comp rx=0 tx=0).stop 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 -- 172.21.15.69:0/1771191506 shutdown_connections 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feea006c8f0 0x7feea006edb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feec01631a0 0x7feec0150e90 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7feec015ea10 0x7feec0150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 --2- 172.21.15.69:0/1771191506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feec00af7d0 0x7feec0156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 -- 172.21.15.69:0/1771191506 >> 172.21.15.69:0/1771191506 conn(0x7feec0067d90 msgr2=0x7feec0159a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 -- 172.21.15.69:0/1771191506 shutdown_connections 2024-04-24T08:10:20.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:20.982+0000 7feec6e00640 1 -- 172.21.15.69:0/1771191506 wait complete. 2024-04-24T08:10:21.002 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:10:21.002 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:21.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.188+0000 7f7ef7800640 1 Processor -- start 2024-04-24T08:10:21.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.188+0000 7f7ef7800640 1 -- start start 2024-04-24T08:10:21.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.189+0000 7f7ef7800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 0x7f7ef006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.189+0000 7f7ef7800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7ef0145a90 0x7f7ef00d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.189+0000 7f7ef7800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ef00d0c20 con 0x7f7ef0145fe0 2024-04-24T08:10:21.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.190+0000 7f7ef7800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7ef00cf3d0 con 0x7f7ef0145a90 2024-04-24T08:10:21.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.190+0000 7f7ef7800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef00cf6b0 con 0x7f7ef00d0300 2024-04-24T08:10:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.190+0000 7f7ef4e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 0x7f7ef006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.190+0000 7f7ee7e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7ef0145a90 0x7f7ef00d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.190+0000 7f7ef4e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 0x7f7ef006e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:33948/0 (socket says 172.21.15.69:33948) 2024-04-24T08:10:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.190+0000 7f7ef4e00640 1 -- 172.21.15.69:0/955255744 learned_addr learned my addr 172.21.15.69:0/955255744 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.191+0000 7f7ef4e00640 1 -- 172.21.15.69:0/955255744 >> v2:172.21.15.121:3300/0 conn(0x7f7ef0145a90 msgr2=0x7f7ef00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:21.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.191+0000 7f7ef4e00640 1 --2- 172.21.15.69:0/955255744 >> v2:172.21.15.121:3300/0 conn(0x7f7ef0145a90 0x7f7ef00d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.191+0000 7f7ef4e00640 1 -- 172.21.15.69:0/955255744 >> v1:172.21.15.121:6789/0 conn(0x7f7ef0145fe0 legacy=0x7f7ef0160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.191+0000 7f7ef4e00640 1 -- 172.21.15.69:0/955255744 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef00cf990 con 0x7f7ef00d0300 2024-04-24T08:10:21.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.192+0000 7f7ef4e00640 1 --2- 172.21.15.69:0/955255744 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 0x7f7ef006e240 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009870 tx=0x7f7ee002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=213c76750c18c236 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.193+0000 7f7eefe00640 1 -- 172.21.15.69:0/955255744 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0037070 con 0x7f7ef00d0300 2024-04-24T08:10:21.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.193+0000 7f7eefe00640 1 -- 172.21.15.69:0/955255744 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee0032440 con 0x7f7ef00d0300 2024-04-24T08:10:21.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.193+0000 7f7eefe00640 1 -- 172.21.15.69:0/955255744 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0032810 con 0x7f7ef00d0300 2024-04-24T08:10:21.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.194+0000 7f7ef7800640 1 -- 172.21.15.69:0/955255744 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 msgr2=0x7f7ef006e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.194+0000 7f7ef7800640 1 --2- 172.21.15.69:0/955255744 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 0x7f7ef006e240 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0009870 tx=0x7f7ee002a270 comp rx=0 tx=0).stop 2024-04-24T08:10:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.194+0000 7f7ef7800640 1 -- 172.21.15.69:0/955255744 shutdown_connections 2024-04-24T08:10:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.194+0000 7f7ef7800640 1 --2- 172.21.15.69:0/955255744 >> v2:172.21.15.121:3300/0 conn(0x7f7ef0145a90 0x7f7ef00d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.194+0000 7f7ef7800640 1 --2- 172.21.15.69:0/955255744 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 0x7f7ef006e240 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.194+0000 7f7ef7800640 1 -- 172.21.15.69:0/955255744 >> 172.21.15.69:0/955255744 conn(0x7f7ef0067d90 msgr2=0x7f7ef006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.194+0000 7f7ef7800640 1 -- 172.21.15.69:0/955255744 shutdown_connections 2024-04-24T08:10:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.194+0000 7f7ef7800640 1 -- 172.21.15.69:0/955255744 wait complete. 2024-04-24T08:10:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.195+0000 7f7ef7800640 1 Processor -- start 2024-04-24T08:10:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.196+0000 7f7ef7800640 1 -- start start 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.196+0000 7f7ef7800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 0x7f7ef0071fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ef7800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ef0145a90 0x7f7ef00ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ef7800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef0145fe0 0x7f7ef00adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ef7800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef0145770 con 0x7f7ef0145a90 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ef7800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef00cff50 con 0x7f7ef0145fe0 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ef7800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef01458f0 con 0x7f7ef00d0300 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ee7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ef0145a90 0x7f7ef00ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ee7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ef0145a90 0x7f7ef00ad5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54474/0 (socket says 172.21.15.69:54474) 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ee7e00640 1 -- 172.21.15.69:0/3104723556 learned_addr learned my addr 172.21.15.69:0/3104723556 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ef4e00640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 0x7f7ef0071fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ee7e00640 1 -- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 msgr2=0x7f7ef0071fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ee7e00640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 0x7f7ef0071fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ee7e00640 1 -- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef0145fe0 msgr2=0x7f7ef00adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ee7e00640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef0145fe0 0x7f7ef00adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.197+0000 7f7ee7e00640 1 -- 172.21.15.69:0/3104723556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef00ae1a0 con 0x7f7ef0145a90 2024-04-24T08:10:21.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.198+0000 7f7ee7e00640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ef0145a90 0x7f7ef00ad5c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800e9e0 tx=0x7f7ed800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.198+0000 7f7eed600640 1 -- 172.21.15.69:0/3104723556 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8017070 con 0x7f7ef0145a90 2024-04-24T08:10:21.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.198+0000 7f7eed600640 1 -- 172.21.15.69:0/3104723556 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ed800c9d0 con 0x7f7ef0145a90 2024-04-24T08:10:21.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.198+0000 7f7eed600640 1 -- 172.21.15.69:0/3104723556 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed800cdc0 con 0x7f7ef0145a90 2024-04-24T08:10:21.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.198+0000 7f7ef7800640 1 -- 172.21.15.69:0/3104723556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef00ae480 con 0x7f7ef0145a90 2024-04-24T08:10:21.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.198+0000 7f7ef7800640 1 -- 172.21.15.69:0/3104723556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef00aea40 con 0x7f7ef0145a90 2024-04-24T08:10:21.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.199+0000 7f7ef7800640 1 -- 172.21.15.69:0/3104723556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef0068a80 con 0x7f7ef0145a90 2024-04-24T08:10:21.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.200+0000 7f7eed600640 1 -- 172.21.15.69:0/3104723556 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ed801a070 con 0x7f7ef0145a90 2024-04-24T08:10:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.201+0000 7f7eed600640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ecc06c8f0 0x7f7ecc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.201+0000 7f7eed600640 1 -- 172.21.15.69:0/3104723556 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f7ed808bef0 con 0x7f7ef0145a90 2024-04-24T08:10:21.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.202+0000 7f7ef4e00640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ecc06c8f0 0x7f7ecc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.203+0000 7f7ef4e00640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ecc06c8f0 0x7f7ecc06edb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0034000 tx=0x7f7ee002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.206+0000 7f7eed600640 1 -- 172.21.15.69:0/3104723556 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ed805a320 con 0x7f7ef0145a90 2024-04-24T08:10:21.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.442+0000 7f7ef7800640 1 -- 172.21.15.69:0/3104723556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ef006ba20 con 0x7f7ef0145a90 2024-04-24T08:10:21.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.443+0000 7f7eed600640 1 -- 172.21.15.69:0/3104723556 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5181 (secure 0 0 0) 0x7f7ed8059c20 con 0x7f7ef0145a90 2024-04-24T08:10:21.444 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:21.444 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2295273533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2295273533},{"type":"v1","addr":"172.21.15.121:6836","nonce":2295273533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1461471292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1461471292},{"type":"v1","addr":"172.21.15.121:6837","nonce":1461471292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/626688145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":626688145},{"type":"v1","addr":"172.21.15.69:6833","nonce":626688145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:09:38.549277+0000","modified":"2024-04-24T08:09:41.567377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/991803145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":991803145},{"type":"v1","addr":"172.21.15.69:6835","nonce":991803145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5241,"qdb_cluster":[5241]},"id":2}]} 2024-04-24T08:10:21.444 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 16 2024-04-24T08:10:21.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.445+0000 7f7ef7800640 1 -- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ecc06c8f0 msgr2=0x7f7ecc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.445+0000 7f7ef7800640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ecc06c8f0 0x7f7ecc06edb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0034000 tx=0x7f7ee002e000 comp rx=0 tx=0).stop 2024-04-24T08:10:21.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.445+0000 7f7ef7800640 1 -- 172.21.15.69:0/3104723556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ef0145a90 msgr2=0x7f7ef00ad5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.445+0000 7f7ef7800640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ef0145a90 0x7f7ef00ad5c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800e9e0 tx=0x7f7ed800eea0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.446+0000 7f7ef7800640 1 -- 172.21.15.69:0/3104723556 shutdown_connections 2024-04-24T08:10:21.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.446+0000 7f7ef7800640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ecc06c8f0 0x7f7ecc06edb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.446+0000 7f7ef7800640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ef0145a90 0x7f7ef00ad5c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.446+0000 7f7ef7800640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ef0145fe0 0x7f7ef00adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.446+0000 7f7ef7800640 1 --2- 172.21.15.69:0/3104723556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ef00d0300 0x7f7ef0071fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.446+0000 7f7ef7800640 1 -- 172.21.15.69:0/3104723556 >> 172.21.15.69:0/3104723556 conn(0x7f7ef0067d90 msgr2=0x7f7ef0144d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:21.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.446+0000 7f7ef7800640 1 -- 172.21.15.69:0/3104723556 shutdown_connections 2024-04-24T08:10:21.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.446+0000 7f7ef7800640 1 -- 172.21.15.69:0/3104723556 wait complete. 2024-04-24T08:10:21.464 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:21.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.654+0000 7fd658c00640 1 Processor -- start 2024-04-24T08:10:21.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.654+0000 7fd658c00640 1 -- start start 2024-04-24T08:10:21.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.655+0000 7fd658c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd654162300 0x7fd654143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.656+0000 7fd658c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65415f540 con 0x7fd654072230 2024-04-24T08:10:21.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.656+0000 7fd658c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65415f820 con 0x7fd654071e50 2024-04-24T08:10:21.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.656+0000 7fd658c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd65415fb00 con 0x7fd654162300 2024-04-24T08:10:21.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.656+0000 7fd652a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd654162300 0x7fd654143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.656+0000 7fd653e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd654072230 0x7fd654143bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51180/0 (socket says 172.21.15.69:51180) 2024-04-24T08:10:21.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.656+0000 7fd653e00640 1 -- 172.21.15.69:0/3466618916 learned_addr learned my addr 172.21.15.69:0/3466618916 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:21.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.656+0000 7fd652a00640 1 -- 172.21.15.69:0/3466618916 >> v1:172.21.15.121:6789/0 conn(0x7fd654071e50 legacy=0x7fd65415ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:21.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.656+0000 7fd652a00640 1 -- 172.21.15.69:0/3466618916 >> v1:172.21.15.69:6789/0 conn(0x7fd654072230 legacy=0x7fd654143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.656+0000 7fd652a00640 1 -- 172.21.15.69:0/3466618916 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65406dda0 con 0x7fd654162300 2024-04-24T08:10:21.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.656+0000 7fd652000640 1 -- 172.21.15.69:0/3466618916 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1197838227 0 0) 0x7fd65415f540 con 0x7fd654072230 2024-04-24T08:10:21.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.657+0000 7fd652a00640 1 --2- 172.21.15.69:0/3466618916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654162300 0x7fd654143690 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd6400097a0 tx=0x7fd64002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a560c89b54a404a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.658+0000 7fd652000640 1 -- 172.21.15.69:0/3466618916 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd640037070 con 0x7fd654162300 2024-04-24T08:10:21.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.658+0000 7fd652000640 1 -- 172.21.15.69:0/3466618916 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd640032440 con 0x7fd654162300 2024-04-24T08:10:21.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.658+0000 7fd652000640 1 -- 172.21.15.69:0/3466618916 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd640032810 con 0x7fd654162300 2024-04-24T08:10:21.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.658+0000 7fd658c00640 1 -- 172.21.15.69:0/3466618916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654162300 msgr2=0x7fd654143690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.658+0000 7fd658c00640 1 --2- 172.21.15.69:0/3466618916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654162300 0x7fd654143690 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd6400097a0 tx=0x7fd64002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.658+0000 7fd658c00640 1 -- 172.21.15.69:0/3466618916 shutdown_connections 2024-04-24T08:10:21.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.658+0000 7fd658c00640 1 --2- 172.21.15.69:0/3466618916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654162300 0x7fd654143690 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.658+0000 7fd658c00640 1 -- 172.21.15.69:0/3466618916 >> 172.21.15.69:0/3466618916 conn(0x7fd6540d0160 msgr2=0x7fd6540d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:21.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.658+0000 7fd658c00640 1 -- 172.21.15.69:0/3466618916 shutdown_connections 2024-04-24T08:10:21.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.658+0000 7fd658c00640 1 -- 172.21.15.69:0/3466618916 wait complete. 2024-04-24T08:10:21.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.660+0000 7fd658c00640 1 Processor -- start 2024-04-24T08:10:21.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.660+0000 7fd658c00640 1 -- start start 2024-04-24T08:10:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.661+0000 7fd658c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd654071e50 0x7fd654158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.661+0000 7fd658c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654072230 0x7fd654157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.661+0000 7fd658c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd654162300 0x7fd654099890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.661+0000 7fd658c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd654144460 con 0x7fd654072230 2024-04-24T08:10:21.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.661+0000 7fd658c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd6541442e0 con 0x7fd654071e50 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.661+0000 7fd658c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd6541445e0 con 0x7fd654162300 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.661+0000 7fd652a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654072230 0x7fd654157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd652a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654072230 0x7fd654157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54494/0 (socket says 172.21.15.69:54494) 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd652a00640 1 -- 172.21.15.69:0/4046531275 learned_addr learned my addr 172.21.15.69:0/4046531275 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd653400640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd654071e50 0x7fd654158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd653e00640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd654162300 0x7fd654099890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd652a00640 1 -- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd654162300 msgr2=0x7fd654099890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd652a00640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd654162300 0x7fd654099890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd652a00640 1 -- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd654071e50 msgr2=0x7fd654158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd652a00640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd654071e50 0x7fd654158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd652a00640 1 -- 172.21.15.69:0/4046531275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd654157590 con 0x7fd654072230 2024-04-24T08:10:21.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd653400640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd654071e50 0x7fd654158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.662+0000 7fd652a00640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654072230 0x7fd654157050 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd64002a7d0 tx=0x7fd6400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.663+0000 7fd64b400640 1 -- 172.21.15.69:0/4046531275 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd640042020 con 0x7fd654072230 2024-04-24T08:10:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.663+0000 7fd64b400640 1 -- 172.21.15.69:0/4046531275 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6400308f0 con 0x7fd654072230 2024-04-24T08:10:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.663+0000 7fd64b400640 1 -- 172.21.15.69:0/4046531275 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd640037040 con 0x7fd654072230 2024-04-24T08:10:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.663+0000 7fd658c00640 1 -- 172.21.15.69:0/4046531275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd654158ad0 con 0x7fd654072230 2024-04-24T08:10:21.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.663+0000 7fd658c00640 1 -- 172.21.15.69:0/4046531275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6540b3280 con 0x7fd654072230 2024-04-24T08:10:21.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.664+0000 7fd658c00640 1 -- 172.21.15.69:0/4046531275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd654075d50 con 0x7fd654072230 2024-04-24T08:10:21.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.665+0000 7fd64b400640 1 -- 172.21.15.69:0/4046531275 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd640045020 con 0x7fd654072230 2024-04-24T08:10:21.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.666+0000 7fd64b400640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd63406c920 0x7fd63406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.666+0000 7fd64b400640 1 -- 172.21.15.69:0/4046531275 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fd6400ac460 con 0x7fd654072230 2024-04-24T08:10:21.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.666+0000 7fd653400640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd63406c920 0x7fd63406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.667+0000 7fd653400640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd63406c920 0x7fd63406ede0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd644003eb0 tx=0x7fd644009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.670+0000 7fd64b400640 1 -- 172.21.15.69:0/4046531275 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd64004c070 con 0x7fd654072230 2024-04-24T08:10:21.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.904+0000 7fd658c00640 1 -- 172.21.15.69:0/4046531275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6540d0d90 con 0x7fd654072230 2024-04-24T08:10:21.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.905+0000 7fd64b400640 1 -- 172.21.15.69:0/4046531275 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5181 (secure 0 0 0) 0x7fd654157590 con 0x7fd654072230 2024-04-24T08:10:21.906 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:21.906 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2295273533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2295273533},{"type":"v1","addr":"172.21.15.121:6836","nonce":2295273533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1461471292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1461471292},{"type":"v1","addr":"172.21.15.121:6837","nonce":1461471292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/626688145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":626688145},{"type":"v1","addr":"172.21.15.69:6833","nonce":626688145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:09:38.549277+0000","modified":"2024-04-24T08:09:41.567377+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5241},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5241":{"gid":5241,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/991803145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":991803145},{"type":"v1","addr":"172.21.15.69:6835","nonce":991803145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5241,"qdb_cluster":[5241]},"id":2}]} 2024-04-24T08:10:21.906 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 16 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 -- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd63406c920 msgr2=0x7fd63406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd63406c920 0x7fd63406ede0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd644003eb0 tx=0x7fd644009040 comp rx=0 tx=0).stop 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 -- 172.21.15.69:0/4046531275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654072230 msgr2=0x7fd654157050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654072230 0x7fd654157050 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fd64002a7d0 tx=0x7fd6400046c0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 -- 172.21.15.69:0/4046531275 shutdown_connections 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd63406c920 0x7fd63406ede0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd654072230 0x7fd654157050 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd654162300 0x7fd654099890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 --2- 172.21.15.69:0/4046531275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd654071e50 0x7fd654158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 -- 172.21.15.69:0/4046531275 >> 172.21.15.69:0/4046531275 conn(0x7fd6540d0160 msgr2=0x7fd6540ae030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 -- 172.21.15.69:0/4046531275 shutdown_connections 2024-04-24T08:10:21.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:21.908+0000 7fd658c00640 1 -- 172.21.15.69:0/4046531275 wait complete. 2024-04-24T08:10:21.928 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:22.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.118+0000 7f85ed400640 1 Processor -- start 2024-04-24T08:10:22.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.118+0000 7f85ed400640 1 -- start start 2024-04-24T08:10:22.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.120+0000 7f85ed400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f85e8072230 0x7f85e8169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.120+0000 7f85ed400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85e8144db0 con 0x7f85e8071e50 2024-04-24T08:10:22.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.120+0000 7f85ed400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85e815f540 con 0x7f85e8162300 2024-04-24T08:10:22.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.120+0000 7f85ed400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f85e815f820 con 0x7f85e8072230 2024-04-24T08:10:22.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.120+0000 7f85e7e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f85e8071e50 0x7f85e815ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51184/0 (socket says 172.21.15.69:51184) 2024-04-24T08:10:22.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.120+0000 7f85e7e00640 1 -- 172.21.15.69:0/1417920412 learned_addr learned my addr 172.21.15.69:0/1417920412 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.120+0000 7f85eca00640 1 --2- 172.21.15.69:0/1417920412 >> v2:172.21.15.121:3300/0 conn(0x7f85e8072230 0x7f85e8169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1169368756 0 0) 0x7f85e8144db0 con 0x7f85e8071e50 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85cc003520 con 0x7f85e8071e50 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4264918645 0 0) 0x7f85cc003520 con 0x7f85e8071e50 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 >> v2:172.21.15.121:3300/0 conn(0x7f85e8072230 msgr2=0x7f85e8169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 --2- 172.21.15.69:0/1417920412 >> v2:172.21.15.121:3300/0 conn(0x7f85e8072230 0x7f85e8169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 >> v1:172.21.15.121:6789/0 conn(0x7f85e8162300 legacy=0x7f85e8143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e815fb00 con 0x7f85e8071e50 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f85d8001670 con 0x7f85e8071e50 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 >> v1:172.21.15.69:6789/0 conn(0x7f85e8071e50 legacy=0x7f85e815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85cc0039b0 0x7f85cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85cc004800 0x7f85cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85e815f820 con 0x7f85cc0051c0 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f85cc00c060 con 0x7f85cc004800 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f85cc00c260 con 0x7f85cc0039b0 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.121+0000 7f85eca00640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.122+0000 7f85eca00640 1 -- 172.21.15.69:0/1417920412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85cc0039b0 msgr2=0x7f85cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.122+0000 7f85e7e00640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85cc0039b0 0x7f85cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.122+0000 7f85e7400640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85cc004800 0x7f85cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.122+0000 7f85eca00640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85cc0039b0 0x7f85cc003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.122+0000 7f85eca00640 1 -- 172.21.15.69:0/1417920412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85cc004800 msgr2=0x7f85cc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.122+0000 7f85eca00640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85cc004800 0x7f85cc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.122+0000 7f85eca00640 1 -- 172.21.15.69:0/1417920412 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85cc003520 con 0x7f85cc0051c0 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.122+0000 7f85e7400640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85cc004800 0x7f85cc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:22.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.123+0000 7f85eca00640 1 -- 172.21.15.69:0/1417920412 reap_dead start 2024-04-24T08:10:22.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.123+0000 7f85eca00640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85cc00b9c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f85e0009fc0 tx=0x7f85cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edef557cbff131af server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:22.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.123+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e0035020 con 0x7f85cc0051c0 2024-04-24T08:10:22.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.123+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e0003bd0 con 0x7f85cc0051c0 2024-04-24T08:10:22.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.123+0000 7f85e6a00640 1 -- 172.21.15.69:0/1417920412 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e0009c10 con 0x7f85cc0051c0 2024-04-24T08:10:22.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.124+0000 7f85ed400640 1 -- 172.21.15.69:0/1417920412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 msgr2=0x7f85cc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.124+0000 7f85ed400640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85cc00b9c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f85e0009fc0 tx=0x7f85cc005830 comp rx=0 tx=0).stop 2024-04-24T08:10:22.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.124+0000 7f85ed400640 1 -- 172.21.15.69:0/1417920412 shutdown_connections 2024-04-24T08:10:22.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.124+0000 7f85ed400640 1 --2- 172.21.15.69:0/1417920412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85cc00b9c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.124+0000 7f85ed400640 1 -- 172.21.15.69:0/1417920412 >> 172.21.15.69:0/1417920412 conn(0x7f85e80d0160 msgr2=0x7f85e80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:22.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.124+0000 7f85ed400640 1 -- 172.21.15.69:0/1417920412 shutdown_connections 2024-04-24T08:10:22.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.124+0000 7f85ed400640 1 -- 172.21.15.69:0/1417920412 wait complete. 2024-04-24T08:10:22.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.126+0000 7f85ed400640 1 Processor -- start 2024-04-24T08:10:22.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.126+0000 7f85ed400640 1 -- start start 2024-04-24T08:10:22.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85ed400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85e8147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85ed400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85e815c1f0 0x7f85e80b12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85ed400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85e8099410 0x7f85e80998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85ed400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85e815ee40 con 0x7f85cc0051c0 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85ed400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f85e815ecc0 con 0x7f85e815c1f0 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85ed400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f85e815efc0 con 0x7f85e8099410 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85e7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85e8147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85e7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85e8147a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54514/0 (socket says 172.21.15.69:54514) 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85e7e00640 1 -- 172.21.15.69:0/2529090955 learned_addr learned my addr 172.21.15.69:0/2529090955 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85e7e00640 1 -- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85e8099410 msgr2=0x7f85e80998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85e7e00640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85e8099410 0x7f85e80998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85e7e00640 1 -- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85e815c1f0 msgr2=0x7f85e80b12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85e7e00640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85e815c1f0 0x7f85e80b12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.127+0000 7f85e7e00640 1 -- 172.21.15.69:0/2529090955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e80b4090 con 0x7f85cc0051c0 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.128+0000 7f85e7e00640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85e8147a40 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f85d800d7a0 tx=0x7f85d800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.128+0000 7f85dfe00640 1 -- 172.21.15.69:0/2529090955 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8019070 con 0x7f85cc0051c0 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.128+0000 7f85dfe00640 1 -- 172.21.15.69:0/2529090955 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d800de50 con 0x7f85cc0051c0 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.128+0000 7f85dfe00640 1 -- 172.21.15.69:0/2529090955 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d80047e0 con 0x7f85cc0051c0 2024-04-24T08:10:22.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.128+0000 7f85ed400640 1 -- 172.21.15.69:0/2529090955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e80b4370 con 0x7f85cc0051c0 2024-04-24T08:10:22.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.128+0000 7f85ed400640 1 -- 172.21.15.69:0/2529090955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e80b4930 con 0x7f85cc0051c0 2024-04-24T08:10:22.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.129+0000 7f85ed400640 1 -- 172.21.15.69:0/2529090955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e8075d50 con 0x7f85cc0051c0 2024-04-24T08:10:22.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.130+0000 7f85dfe00640 1 -- 172.21.15.69:0/2529090955 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f85d8004980 con 0x7f85cc0051c0 2024-04-24T08:10:22.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.131+0000 7f85dfe00640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85c806c8f0 0x7f85c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.131+0000 7f85dfe00640 1 -- 172.21.15.69:0/2529090955 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f85d808b3f0 con 0x7f85cc0051c0 2024-04-24T08:10:22.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.132+0000 7f85e7400640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85c806c8f0 0x7f85c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.133+0000 7f85e7400640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85c806c8f0 0x7f85c806edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f85d4000f60 tx=0x7f85d4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:22.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.135+0000 7f85dfe00640 1 -- 172.21.15.69:0/2529090955 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f85d8059130 con 0x7f85cc0051c0 2024-04-24T08:10:22.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.335+0000 7f85ed400640 1 -- 172.21.15.69:0/2529090955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85e80d0d90 con 0x7f85cc0051c0 2024-04-24T08:10:22.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.337+0000 7f85dfe00640 1 -- 172.21.15.69:0/2529090955 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16645 (secure 0 0 0) 0x7f85d805d430 con 0x7f85cc0051c0 2024-04-24T08:10:22.338 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:22.338 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":33,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:09:40.096668+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:09:36.197195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-24T08:09:37.506730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 -- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85c806c8f0 msgr2=0x7f85c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85c806c8f0 0x7f85c806edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f85d4000f60 tx=0x7f85d4009040 comp rx=0 tx=0).stop 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 -- 172.21.15.69:0/2529090955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 msgr2=0x7f85e8147a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85e8147a40 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f85d800d7a0 tx=0x7f85d800dc60 comp rx=0 tx=0).stop 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 -- 172.21.15.69:0/2529090955 shutdown_connections 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85c806c8f0 0x7f85c806edb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f85e8099410 0x7f85e80998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f85e815c1f0 0x7f85e80b12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 --2- 172.21.15.69:0/2529090955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85cc0051c0 0x7f85e8147a40 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 -- 172.21.15.69:0/2529090955 >> 172.21.15.69:0/2529090955 conn(0x7f85e80d0160 msgr2=0x7f85e8072fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 -- 172.21.15.69:0/2529090955 shutdown_connections 2024-04-24T08:10:22.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.340+0000 7f85ed400640 1 -- 172.21.15.69:0/2529090955 wait complete. 2024-04-24T08:10:22.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:10:22.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.557+0000 7fa0f6400640 1 Processor -- start 2024-04-24T08:10:22.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.557+0000 7fa0f6400640 1 -- start start 2024-04-24T08:10:22.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.558+0000 7fa0f6400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa0f00cfe40 0x7fa0f00b7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.558+0000 7fa0f6400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa0f00b7a90 0x7fa0f00d10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.558+0000 7fa0f6400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f00d22d0 con 0x7fa0f0162150 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.558+0000 7fa0f6400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa0f00d2900 con 0x7fa0f00b7a90 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.558+0000 7fa0f6400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa0f00d03b0 con 0x7fa0f00cfe40 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.559+0000 7fa0efe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa0f00b7a90 0x7fa0f00d10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.559+0000 7fa0efe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa0f00b7a90 0x7fa0f00d10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54518/0 (socket says 172.21.15.69:54518) 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.559+0000 7fa0efe00640 1 -- 172.21.15.69:0/2477160279 learned_addr learned my addr 172.21.15.69:0/2477160279 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.559+0000 7fa0eea00640 1 --2- 172.21.15.69:0/2477160279 >> v2:172.21.15.121:3300/0 conn(0x7fa0f00cfe40 0x7fa0f00b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.559+0000 7fa0efe00640 1 -- 172.21.15.69:0/2477160279 >> v2:172.21.15.121:3300/0 conn(0x7fa0f00cfe40 msgr2=0x7fa0f00b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.559+0000 7fa0efe00640 1 --2- 172.21.15.69:0/2477160279 >> v2:172.21.15.121:3300/0 conn(0x7fa0f00cfe40 0x7fa0f00b7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.559+0000 7fa0efe00640 1 -- 172.21.15.69:0/2477160279 >> v1:172.21.15.121:6789/0 conn(0x7fa0f0162150 legacy=0x7fa0f00d5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:22.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.559+0000 7fa0efe00640 1 -- 172.21.15.69:0/2477160279 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f00d0690 con 0x7fa0f00b7a90 2024-04-24T08:10:22.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.560+0000 7fa0efe00640 1 --2- 172.21.15.69:0/2477160279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00b7a90 0x7fa0f00d10f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00b6010 tx=0x7fa0e802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be403793db1927a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:22.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.560+0000 7fa0ee000640 1 -- 172.21.15.69:0/2477160279 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8036070 con 0x7fa0f00b7a90 2024-04-24T08:10:22.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.561+0000 7fa0ee000640 1 -- 172.21.15.69:0/2477160279 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e8031600 con 0x7fa0f00b7a90 2024-04-24T08:10:22.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.561+0000 7fa0ee000640 1 -- 172.21.15.69:0/2477160279 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e80319d0 con 0x7fa0f00b7a90 2024-04-24T08:10:22.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.561+0000 7fa0f6400640 1 -- 172.21.15.69:0/2477160279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00b7a90 msgr2=0x7fa0f00d10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.561+0000 7fa0f6400640 1 --2- 172.21.15.69:0/2477160279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00b7a90 0x7fa0f00d10f0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00b6010 tx=0x7fa0e802a6f0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.561+0000 7fa0f6400640 1 -- 172.21.15.69:0/2477160279 shutdown_connections 2024-04-24T08:10:22.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.561+0000 7fa0f6400640 1 --2- 172.21.15.69:0/2477160279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00b7a90 0x7fa0f00d10f0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.561+0000 7fa0f6400640 1 --2- 172.21.15.69:0/2477160279 >> v2:172.21.15.121:3300/0 conn(0x7fa0f00cfe40 0x7fa0f00b7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.561+0000 7fa0f6400640 1 -- 172.21.15.69:0/2477160279 >> 172.21.15.69:0/2477160279 conn(0x7fa0f008bf60 msgr2=0x7fa0f00d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:22.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.562+0000 7fa0f6400640 1 -- 172.21.15.69:0/2477160279 shutdown_connections 2024-04-24T08:10:22.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.562+0000 7fa0f6400640 1 -- 172.21.15.69:0/2477160279 wait complete. 2024-04-24T08:10:22.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.563+0000 7fa0f6400640 1 Processor -- start 2024-04-24T08:10:22.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.563+0000 7fa0f6400640 1 -- start start 2024-04-24T08:10:22.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.564+0000 7fa0f6400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0f00b7a90 0x7fa0f01f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.564+0000 7fa0f6400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00cfe40 0x7fa0f00c8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.564+0000 7fa0f6400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa0f0162150 0x7fa0f00cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.564+0000 7fa0f6400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f00d17b0 con 0x7fa0f00cfe40 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.564+0000 7fa0f6400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f00d1630 con 0x7fa0f00b7a90 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.564+0000 7fa0f6400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f00d1930 con 0x7fa0f0162150 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.565+0000 7fa0eea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00cfe40 0x7fa0f00c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.565+0000 7fa0eea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00cfe40 0x7fa0f00c8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54522/0 (socket says 172.21.15.69:54522) 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.565+0000 7fa0eea00640 1 -- 172.21.15.69:0/1308995946 learned_addr learned my addr 172.21.15.69:0/1308995946 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.565+0000 7fa0ef400640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0f00b7a90 0x7fa0f01f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.565+0000 7fa0eea00640 1 -- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa0f0162150 msgr2=0x7fa0f00cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.565+0000 7fa0eea00640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa0f0162150 0x7fa0f00cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.565+0000 7fa0eea00640 1 -- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0f00b7a90 msgr2=0x7fa0f01f55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.565+0000 7fa0eea00640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0f00b7a90 0x7fa0f01f55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.565+0000 7fa0eea00640 1 -- 172.21.15.69:0/1308995946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f01f5b20 con 0x7fa0f00cfe40 2024-04-24T08:10:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.565+0000 7fa0eea00640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00cfe40 0x7fa0f00c8230 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00c770 tx=0x7fa0dc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.566+0000 7fa0e7400640 1 -- 172.21.15.69:0/1308995946 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc015070 con 0x7fa0f00cfe40 2024-04-24T08:10:22.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.566+0000 7fa0e7400640 1 -- 172.21.15.69:0/1308995946 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0dc004220 con 0x7fa0f00cfe40 2024-04-24T08:10:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.566+0000 7fa0e7400640 1 -- 172.21.15.69:0/1308995946 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc007620 con 0x7fa0f00cfe40 2024-04-24T08:10:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.566+0000 7fa0f6400640 1 -- 172.21.15.69:0/1308995946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f00c8b30 con 0x7fa0f00cfe40 2024-04-24T08:10:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.566+0000 7fa0f6400640 1 -- 172.21.15.69:0/1308995946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f01f4de0 con 0x7fa0f00cfe40 2024-04-24T08:10:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.567+0000 7fa0f6400640 1 -- 172.21.15.69:0/1308995946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f01f4c00 con 0x7fa0f00cfe40 2024-04-24T08:10:22.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.568+0000 7fa0e7400640 1 -- 172.21.15.69:0/1308995946 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa0dc0077c0 con 0x7fa0f00cfe40 2024-04-24T08:10:22.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.569+0000 7fa0e7400640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0d006c8f0 0x7fa0d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.569+0000 7fa0e7400640 1 -- 172.21.15.69:0/1308995946 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fa0dc08bc30 con 0x7fa0f00cfe40 2024-04-24T08:10:22.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.569+0000 7fa0ef400640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0d006c8f0 0x7fa0d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.571+0000 7fa0ef400640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0d006c8f0 0x7fa0d006edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000b7f0 tx=0x7fa0e0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:22.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.573+0000 7fa0e7400640 1 -- 172.21.15.69:0/1308995946 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0f01f4c00 con 0x7fa0f00cfe40 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.812+0000 7fa0f6400640 1 -- 172.21.15.69:0/1308995946 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fa0f0068ab0 con 0x7fa0f00cfe40 2024-04-24T08:10:22.813 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:10:22.813+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:10:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.817+0000 7fa0e7400640 1 -- 172.21.15.69:0/1308995946 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7fa0f0068ab0 con 0x7fa0f00cfe40 2024-04-24T08:10:22.818 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:10:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.820+0000 7fa0f6400640 1 -- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0d006c8f0 msgr2=0x7fa0d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.820+0000 7fa0f6400640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0d006c8f0 0x7fa0d006edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000b7f0 tx=0x7fa0e0008040 comp rx=0 tx=0).stop 2024-04-24T08:10:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.820+0000 7fa0f6400640 1 -- 172.21.15.69:0/1308995946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00cfe40 msgr2=0x7fa0f00c8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.820+0000 7fa0f6400640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00cfe40 0x7fa0f00c8230 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00c770 tx=0x7fa0dc00cc30 comp rx=0 tx=0).stop 2024-04-24T08:10:22.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.820+0000 7fa0f6400640 1 -- 172.21.15.69:0/1308995946 shutdown_connections 2024-04-24T08:10:22.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.820+0000 7fa0f6400640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0d006c8f0 0x7fa0d006edb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.820+0000 7fa0f6400640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa0f00cfe40 0x7fa0f00c8230 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.820+0000 7fa0f6400640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa0f0162150 0x7fa0f00cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.820+0000 7fa0f6400640 1 --2- 172.21.15.69:0/1308995946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa0f00b7a90 0x7fa0f01f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.820+0000 7fa0f6400640 1 -- 172.21.15.69:0/1308995946 >> 172.21.15.69:0/1308995946 conn(0x7fa0f008bf60 msgr2=0x7fa0f00b84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:22.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.821+0000 7fa0f6400640 1 -- 172.21.15.69:0/1308995946 shutdown_connections 2024-04-24T08:10:22.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:22.821+0000 7fa0f6400640 1 -- 172.21.15.69:0/1308995946 wait complete. 2024-04-24T08:10:22.838 INFO:tasks.ceph.mds.c.smithi069.stderr: -6391> 2024-04-24T08:09:42.039+0000 7fb31ec00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:10:22.842 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T08:10:22.903 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:10:23.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.019+0000 7f6c15e00640 1 Processor -- start 2024-04-24T08:10:23.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.019+0000 7f6c15e00640 1 -- start start 2024-04-24T08:10:23.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.021+0000 7f6c15e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c100d18c0 0x7f6c1006de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.021+0000 7f6c15e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c100d2600 con 0x7f6c1015ea60 2024-04-24T08:10:23.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.021+0000 7f6c15e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c100d28e0 con 0x7f6c100d36a0 2024-04-24T08:10:23.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.021+0000 7f6c15e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6c100d23b0 con 0x7f6c100d18c0 2024-04-24T08:10:23.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.021+0000 7f6c0ea00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6c1015ea60 0x7f6c100cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51202/0 (socket says 172.21.15.69:51202) 2024-04-24T08:10:23.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.021+0000 7f6c0ea00640 1 -- 172.21.15.69:0/1043512424 learned_addr learned my addr 172.21.15.69:0/1043512424 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:23.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.021+0000 7f6c0f400640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c100d18c0 0x7f6c1006de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.021+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1183950243 0 0) 0x7f6c100d2600 con 0x7f6c1015ea60 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bf4003520 con 0x7f6c1015ea60 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 605421109 0 0) 0x7f6bf4003520 con 0x7f6c1015ea60 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c100d18c0 msgr2=0x7f6c1006de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c100d18c0 0x7f6c1006de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 >> v1:172.21.15.121:6789/0 conn(0x7f6c100d36a0 legacy=0x7f6c100d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c100d2bc0 con 0x7f6c1015ea60 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f6c00001670 con 0x7f6c1015ea60 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 >> v1:172.21.15.69:6789/0 conn(0x7f6c1015ea60 legacy=0x7f6c100cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bf40039e0 0x7f6bf4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bf4004800 0x7f6bf4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c100d18c0 0x7f6bf40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c100d23b0 con 0x7f6bf40039e0 2024-04-24T08:10:23.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf400baa0 con 0x7f6bf4004800 2024-04-24T08:10:23.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf400bca0 con 0x7f6c100d18c0 2024-04-24T08:10:23.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.022+0000 7f6c0ea00640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bf40039e0 0x7f6bf4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.023+0000 7f6c0ea00640 1 -- 172.21.15.69:0/1043512424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c100d18c0 msgr2=0x7f6bf40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:23.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.023+0000 7f6c0ea00640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c100d18c0 0x7f6bf40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.023+0000 7f6c0ea00640 1 -- 172.21.15.69:0/1043512424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bf4004800 msgr2=0x7f6bf4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:23.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.023+0000 7f6c0e000640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bf4004800 0x7f6bf4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.023+0000 7f6c0ea00640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bf4004800 0x7f6bf4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.023+0000 7f6c0ea00640 1 -- 172.21.15.69:0/1043512424 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf4003520 con 0x7f6bf40039e0 2024-04-24T08:10:23.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.024+0000 7f6c0ea00640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bf40039e0 0x7f6bf4003e00 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000c210 tx=0x7f6bf4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5073cf5d0497e101 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.024+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c00007d10 con 0x7f6bf40039e0 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.024+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c00005ad0 con 0x7f6bf40039e0 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.024+0000 7f6c0d600640 1 -- 172.21.15.69:0/1043512424 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c00007070 con 0x7f6bf40039e0 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.025+0000 7f6c15e00640 1 -- 172.21.15.69:0/1043512424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bf40039e0 msgr2=0x7f6bf4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.025+0000 7f6c15e00640 1 --2- 172.21.15.69:0/1043512424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6bf40039e0 0x7f6bf4003e00 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6c0000c210 tx=0x7f6bf4005830 comp rx=0 tx=0).stop 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.025+0000 7f6c0f400640 1 -- 172.21.15.69:0/1043512424 reap_dead start 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.025+0000 7f6c15e00640 1 -- 172.21.15.69:0/1043512424 shutdown_connections 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.025+0000 7f6c15e00640 1 -- 172.21.15.69:0/1043512424 >> 172.21.15.69:0/1043512424 conn(0x7f6c1008bf60 msgr2=0x7f6c100d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.025+0000 7f6c15e00640 1 -- 172.21.15.69:0/1043512424 shutdown_connections 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.025+0000 7f6c15e00640 1 -- 172.21.15.69:0/1043512424 wait complete. 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.026+0000 7f6c15e00640 1 Processor -- start 2024-04-24T08:10:23.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.027+0000 7f6c15e00640 1 -- start start 2024-04-24T08:10:23.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.027+0000 7f6c15e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bf40039e0 0x7f6c100c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.027+0000 7f6c15e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1015d880 0x7f6c1014fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c15e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1014ff90 0x7f6c100cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c15e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c100d3180 con 0x7f6c1015d880 2024-04-24T08:10:23.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c15e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1006dba0 con 0x7f6bf40039e0 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c15e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1015ee40 con 0x7f6c1014ff90 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c0e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1015d880 0x7f6c1014fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c0e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1015d880 0x7f6c1014fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54560/0 (socket says 172.21.15.69:54560) 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c0e000640 1 -- 172.21.15.69:0/3806555671 learned_addr learned my addr 172.21.15.69:0/3806555671 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c0e000640 1 -- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1014ff90 msgr2=0x7f6c100cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c0e000640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1014ff90 0x7f6c100cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c0ea00640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bf40039e0 0x7f6c100c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c0e000640 1 -- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bf40039e0 msgr2=0x7f6c100c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c0e000640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bf40039e0 0x7f6c100c84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c0e000640 1 -- 172.21.15.69:0/3806555671 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c100c8a30 con 0x7f6c1015d880 2024-04-24T08:10:23.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.028+0000 7f6c0e000640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1015d880 0x7f6c1014fa50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800d4f0 tx=0x7f6bf800d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:23.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.029+0000 7f6c07400640 1 -- 172.21.15.69:0/3806555671 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8018070 con 0x7f6c1015d880 2024-04-24T08:10:23.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.029+0000 7f6c07400640 1 -- 172.21.15.69:0/3806555671 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf8003610 con 0x7f6c1015d880 2024-04-24T08:10:23.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.029+0000 7f6c07400640 1 -- 172.21.15.69:0/3806555671 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8003a00 con 0x7f6c1015d880 2024-04-24T08:10:23.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.029+0000 7f6c15e00640 1 -- 172.21.15.69:0/3806555671 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c100cb460 con 0x7f6c1015d880 2024-04-24T08:10:23.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.029+0000 7f6c15e00640 1 -- 172.21.15.69:0/3806555671 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1014de80 con 0x7f6c1015d880 2024-04-24T08:10:23.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.030+0000 7f6c15e00640 1 -- 172.21.15.69:0/3806555671 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c10068a80 con 0x7f6c1015d880 2024-04-24T08:10:23.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.031+0000 7f6c07400640 1 -- 172.21.15.69:0/3806555671 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6bf801b070 con 0x7f6c1015d880 2024-04-24T08:10:23.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.032+0000 7f6c07400640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6bec06c920 0x7f6bec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.032+0000 7f6c07400640 1 -- 172.21.15.69:0/3806555671 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5795+0+0 (secure 0 0 0) 0x7f6bf808b370 con 0x7f6c1015d880 2024-04-24T08:10:23.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.033+0000 7f6c0ea00640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6bec06c920 0x7f6bec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.034+0000 7f6c0ea00640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6bec06c920 0x7f6bec06ede0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6c00032000 tx=0x7f6c0002d000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:23.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.037+0000 7f6c07400640 1 -- 172.21.15.69:0/3806555671 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6bf8059120 con 0x7f6c1015d880 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.278+0000 7f6c15e00640 1 -- 172.21.15.69:0/3806555671 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6c101503b0 con 0x7f6c1015d880 2024-04-24T08:10:23.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.821+0000 7f6c07400640 1 -- 172.21.15.69:0/3806555671 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7f6bf805d420 con 0x7f6c1015d880 2024-04-24T08:10:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 -- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6bec06c920 msgr2=0x7f6bec06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6bec06c920 0x7f6bec06ede0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6c00032000 tx=0x7f6c0002d000 comp rx=0 tx=0).stop 2024-04-24T08:10:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 -- 172.21.15.69:0/3806555671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1015d880 msgr2=0x7f6c1014fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1015d880 0x7f6c1014fa50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800d4f0 tx=0x7f6bf800d9b0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 -- 172.21.15.69:0/3806555671 shutdown_connections 2024-04-24T08:10:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6bec06c920 0x7f6bec06ede0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1015d880 0x7f6c1014fa50 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c1014ff90 0x7f6c100cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 --2- 172.21.15.69:0/3806555671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6bf40039e0 0x7f6c100c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 -- 172.21.15.69:0/3806555671 >> 172.21.15.69:0/3806555671 conn(0x7f6c1008bf60 msgr2=0x7f6c100b5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:23.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 -- 172.21.15.69:0/3806555671 shutdown_connections 2024-04-24T08:10:23.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:23.825+0000 7f6c15e00640 1 -- 172.21.15.69:0/3806555671 wait complete. 2024-04-24T08:10:23.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T08:10:24.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.042+0000 7f1133400640 1 Processor -- start 2024-04-24T08:10:24.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.042+0000 7f1133400640 1 -- start start 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1133400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f112c15c690 0x7f112c0d3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1133400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f112c0d1070 0x7f112c0d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1133400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f112c15f0d0 con 0x7f112c0d2920 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1133400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f112c15fa00 con 0x7f112c15c690 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1133400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f112c15f3b0 con 0x7f112c0d1070 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1130a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f112c15c690 0x7f112c0d3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1130a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f112c15c690 0x7f112c0d3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54570/0 (socket says 172.21.15.69:54570) 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1130a00640 1 -- 172.21.15.69:0/2934689119 learned_addr learned my addr 172.21.15.69:0/2934689119 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f112be00640 1 --2- 172.21.15.69:0/2934689119 >> v2:172.21.15.121:3300/0 conn(0x7f112c0d1070 0x7f112c0d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1130a00640 1 -- 172.21.15.69:0/2934689119 >> v2:172.21.15.121:3300/0 conn(0x7f112c0d1070 msgr2=0x7f112c0d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1130a00640 1 --2- 172.21.15.69:0/2934689119 >> v2:172.21.15.121:3300/0 conn(0x7f112c0d1070 0x7f112c0d14f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1130a00640 1 -- 172.21.15.69:0/2934689119 >> v1:172.21.15.121:6789/0 conn(0x7f112c0d2920 legacy=0x7f112c0d2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:24.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.044+0000 7f1130a00640 1 -- 172.21.15.69:0/2934689119 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112c15f690 con 0x7f112c15c690 2024-04-24T08:10:24.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.045+0000 7f1130a00640 1 --2- 172.21.15.69:0/2934689119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c15c690 0x7f112c0d3b60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f112c0d21b0 tx=0x7f112002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121c69a6464711dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:24.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.046+0000 7f112b400640 1 -- 172.21.15.69:0/2934689119 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120037070 con 0x7f112c15c690 2024-04-24T08:10:24.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.046+0000 7f112b400640 1 -- 172.21.15.69:0/2934689119 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1120032440 con 0x7f112c15c690 2024-04-24T08:10:24.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.046+0000 7f112b400640 1 -- 172.21.15.69:0/2934689119 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120032810 con 0x7f112c15c690 2024-04-24T08:10:24.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.046+0000 7f1133400640 1 -- 172.21.15.69:0/2934689119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c15c690 msgr2=0x7f112c0d3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.046+0000 7f1133400640 1 --2- 172.21.15.69:0/2934689119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c15c690 0x7f112c0d3b60 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f112c0d21b0 tx=0x7f112002a230 comp rx=0 tx=0).stop 2024-04-24T08:10:24.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.047+0000 7f1133400640 1 -- 172.21.15.69:0/2934689119 shutdown_connections 2024-04-24T08:10:24.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.047+0000 7f1133400640 1 --2- 172.21.15.69:0/2934689119 >> v2:172.21.15.121:3300/0 conn(0x7f112c0d1070 0x7f112c0d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.047+0000 7f1133400640 1 --2- 172.21.15.69:0/2934689119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c15c690 0x7f112c0d3b60 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.047+0000 7f1133400640 1 -- 172.21.15.69:0/2934689119 >> 172.21.15.69:0/2934689119 conn(0x7f112c067760 msgr2=0x7f112c067b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:24.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.047+0000 7f1133400640 1 -- 172.21.15.69:0/2934689119 shutdown_connections 2024-04-24T08:10:24.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.047+0000 7f1133400640 1 -- 172.21.15.69:0/2934689119 wait complete. 2024-04-24T08:10:24.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.048+0000 7f1133400640 1 Processor -- start 2024-04-24T08:10:24.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.048+0000 7f1133400640 1 -- start start 2024-04-24T08:10:24.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.049+0000 7f1133400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f112c0d1070 0x7f112c0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.049+0000 7f1133400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c0d2920 0x7f112c0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.049+0000 7f1133400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f112c15c690 0x7f112c0c87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.049+0000 7f1133400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f112c0b7140 con 0x7f112c0d2920 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.049+0000 7f1133400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f112c15fe90 con 0x7f112c0d1070 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.049+0000 7f1133400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f112c0b6da0 con 0x7f112c15c690 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f112be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c0d2920 0x7f112c0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f112be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c0d2920 0x7f112c0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54584/0 (socket says 172.21.15.69:54584) 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f112be00640 1 -- 172.21.15.69:0/1654896588 learned_addr learned my addr 172.21.15.69:0/1654896588 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f1130a00640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f112c0d1070 0x7f112c0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f112be00640 1 -- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f112c15c690 msgr2=0x7f112c0c87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f112be00640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f112c15c690 0x7f112c0c87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f112be00640 1 -- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f112c0d1070 msgr2=0x7f112c0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f112be00640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f112c0d1070 0x7f112c0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f112be00640 1 -- 172.21.15.69:0/1654896588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112c0caa80 con 0x7f112c0d2920 2024-04-24T08:10:24.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f1130a00640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f112c0d1070 0x7f112c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:24.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.050+0000 7f112be00640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c0d2920 0x7f112c0ca540 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f111800c7f0 tx=0x7f111800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:24.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.051+0000 7f1128c00640 1 -- 172.21.15.69:0/1654896588 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118015070 con 0x7f112c0d2920 2024-04-24T08:10:24.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.051+0000 7f1128c00640 1 -- 172.21.15.69:0/1654896588 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1118004240 con 0x7f112c0d2920 2024-04-24T08:10:24.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.051+0000 7f1128c00640 1 -- 172.21.15.69:0/1654896588 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118007620 con 0x7f112c0d2920 2024-04-24T08:10:24.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.051+0000 7f1133400640 1 -- 172.21.15.69:0/1654896588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f112c0cad20 con 0x7f112c0d2920 2024-04-24T08:10:24.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.051+0000 7f1133400640 1 -- 172.21.15.69:0/1654896588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f112c0cb5c0 con 0x7f112c0d2920 2024-04-24T08:10:24.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.052+0000 7f1133400640 1 -- 172.21.15.69:0/1654896588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f112c068a80 con 0x7f112c0d2920 2024-04-24T08:10:24.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.053+0000 7f1128c00640 1 -- 172.21.15.69:0/1654896588 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f11180079d0 con 0x7f112c0d2920 2024-04-24T08:10:24.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.054+0000 7f1128c00640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f110c06c8f0 0x7f110c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.054+0000 7f1128c00640 1 -- 172.21.15.69:0/1654896588 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5795+0+0 (secure 0 0 0) 0x7f111808bff0 con 0x7f112c0d2920 2024-04-24T08:10:24.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.055+0000 7f1130a00640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f110c06c8f0 0x7f110c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.056+0000 7f1130a00640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f110c06c8f0 0x7f110c06edb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1120034000 tx=0x7f112002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:24.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.059+0000 7f1128c00640 1 -- 172.21.15.69:0/1654896588 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f111805a410 con 0x7f112c0d2920 2024-04-24T08:10:24.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.263+0000 7f1133400640 1 -- 172.21.15.69:0/1654896588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f112c0d41b0 con 0x7f112c0d2920 2024-04-24T08:10:24.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.825+0000 7f1128c00640 1 -- 172.21.15.69:0/1654896588 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) ==== 181+0+0 (secure 0 0 0) 0x7f1118059d10 con 0x7f112c0d2920 2024-04-24T08:10:24.826 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' removed 2024-04-24T08:10:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.829+0000 7f1133400640 1 -- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f110c06c8f0 msgr2=0x7f110c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.829+0000 7f1133400640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f110c06c8f0 0x7f110c06edb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1120034000 tx=0x7f112002e000 comp rx=0 tx=0).stop 2024-04-24T08:10:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.829+0000 7f1133400640 1 -- 172.21.15.69:0/1654896588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c0d2920 msgr2=0x7f112c0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.829+0000 7f1133400640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c0d2920 0x7f112c0ca540 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f111800c7f0 tx=0x7f111800ccb0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.830+0000 7f1133400640 1 -- 172.21.15.69:0/1654896588 shutdown_connections 2024-04-24T08:10:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.830+0000 7f1133400640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f110c06c8f0 0x7f110c06edb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.830+0000 7f1133400640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f112c0d2920 0x7f112c0ca540 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.830+0000 7f1133400640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f112c15c690 0x7f112c0c87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.830+0000 7f1133400640 1 --2- 172.21.15.69:0/1654896588 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f112c0d1070 0x7f112c0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.830+0000 7f1133400640 1 -- 172.21.15.69:0/1654896588 >> 172.21.15.69:0/1654896588 conn(0x7f112c067760 msgr2=0x7f112c160010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:24.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.832+0000 7f1133400640 1 -- 172.21.15.69:0/1654896588 shutdown_connections 2024-04-24T08:10:24.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:24.833+0000 7f1133400640 1 -- 172.21.15.69:0/1654896588 wait complete. 2024-04-24T08:10:24.856 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T08:10:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.051+0000 7f5758c00640 1 Processor -- start 2024-04-24T08:10:25.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.051+0000 7f5758c00640 1 -- start start 2024-04-24T08:10:25.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.053+0000 7f5758c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5754160fb0 0x7f575409b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.053+0000 7f5758c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f575409a9d0 con 0x7f5754160410 2024-04-24T08:10:25.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.053+0000 7f5758c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57540ce3f0 con 0x7f575415fc10 2024-04-24T08:10:25.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.053+0000 7f5758c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f575409b2c0 con 0x7f5754160fb0 2024-04-24T08:10:25.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.053+0000 7f5752800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5754160410 0x7f57540b1e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51204/0 (socket says 172.21.15.69:51204) 2024-04-24T08:10:25.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.053+0000 7f5752800640 1 -- 172.21.15.69:0/1915220164 learned_addr learned my addr 172.21.15.69:0/1915220164 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:25.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.053+0000 7f5749400640 1 --2- 172.21.15.69:0/1915220164 >> v2:172.21.15.69:3300/0 conn(0x7f5754160fb0 0x7f575409b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.054+0000 7f5751400640 1 -- 172.21.15.69:0/1915220164 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1939813604 0 0) 0x7f575409a9d0 con 0x7f5754160410 2024-04-24T08:10:25.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.054+0000 7f5749400640 1 -- 172.21.15.69:0/1915220164 >> v1:172.21.15.121:6789/0 conn(0x7f575415fc10 legacy=0x7f57540cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:25.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.054+0000 7f5749400640 1 -- 172.21.15.69:0/1915220164 >> v1:172.21.15.69:6789/0 conn(0x7f5754160410 legacy=0x7f57540b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.054+0000 7f5749400640 1 -- 172.21.15.69:0/1915220164 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f575409ac70 con 0x7f5754160fb0 2024-04-24T08:10:25.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.055+0000 7f5749400640 1 --2- 172.21.15.69:0/1915220164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160fb0 0x7f575409b750 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f573c0097a0 tx=0x7f573c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b8626dd9acccc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:25.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.056+0000 7f5751400640 1 -- 172.21.15.69:0/1915220164 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c037070 con 0x7f5754160fb0 2024-04-24T08:10:25.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.056+0000 7f5751400640 1 -- 172.21.15.69:0/1915220164 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f573c032440 con 0x7f5754160fb0 2024-04-24T08:10:25.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.056+0000 7f5751400640 1 -- 172.21.15.69:0/1915220164 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c032810 con 0x7f5754160fb0 2024-04-24T08:10:25.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.056+0000 7f5758c00640 1 -- 172.21.15.69:0/1915220164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160fb0 msgr2=0x7f575409b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.056+0000 7f5758c00640 1 --2- 172.21.15.69:0/1915220164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160fb0 0x7f575409b750 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f573c0097a0 tx=0x7f573c02a1e0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.056+0000 7f5758c00640 1 -- 172.21.15.69:0/1915220164 shutdown_connections 2024-04-24T08:10:25.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.056+0000 7f5758c00640 1 --2- 172.21.15.69:0/1915220164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160fb0 0x7f575409b750 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.056+0000 7f5758c00640 1 -- 172.21.15.69:0/1915220164 >> 172.21.15.69:0/1915220164 conn(0x7f57540b4090 msgr2=0x7f57540b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:25.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.057+0000 7f5758c00640 1 -- 172.21.15.69:0/1915220164 shutdown_connections 2024-04-24T08:10:25.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.057+0000 7f5758c00640 1 -- 172.21.15.69:0/1915220164 wait complete. 2024-04-24T08:10:25.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.059+0000 7f5758c00640 1 Processor -- start 2024-04-24T08:10:25.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.059+0000 7f5758c00640 1 -- start start 2024-04-24T08:10:25.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.060+0000 7f5758c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575415fc10 0x7f57540b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.060+0000 7f5758c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160410 0x7f575414eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.060+0000 7f5758c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5754160fb0 0x7f57540c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.060+0000 7f5758c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5754160170 con 0x7f5754160410 2024-04-24T08:10:25.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.060+0000 7f5758c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f575415fff0 con 0x7f5754160fb0 2024-04-24T08:10:25.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.060+0000 7f5758c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f575409bc90 con 0x7f575415fc10 2024-04-24T08:10:25.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.060+0000 7f5749400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160410 0x7f575414eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5749400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160410 0x7f575414eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54596/0 (socket says 172.21.15.69:54596) 2024-04-24T08:10:25.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5749400640 1 -- 172.21.15.69:0/3874934798 learned_addr learned my addr 172.21.15.69:0/3874934798 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5752800640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5754160fb0 0x7f57540c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5751e00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575415fc10 0x7f57540b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5749400640 1 -- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575415fc10 msgr2=0x7f57540b2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5749400640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575415fc10 0x7f57540b2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5749400640 1 -- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5754160fb0 msgr2=0x7f57540c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5749400640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5754160fb0 0x7f57540c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5749400640 1 -- 172.21.15.69:0/3874934798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57540b1c40 con 0x7f5754160410 2024-04-24T08:10:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5752800640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5754160fb0 0x7f57540c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:25.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5751e00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575415fc10 0x7f57540b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:25.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.061+0000 7f5749400640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160410 0x7f575414eb00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f573c000c00 tx=0x7f573c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:25.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.062+0000 7f574aa00640 1 -- 172.21.15.69:0/3874934798 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c037040 con 0x7f5754160410 2024-04-24T08:10:25.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.062+0000 7f574aa00640 1 -- 172.21.15.69:0/3874934798 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f573c004230 con 0x7f5754160410 2024-04-24T08:10:25.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.062+0000 7f574aa00640 1 -- 172.21.15.69:0/3874934798 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c032e40 con 0x7f5754160410 2024-04-24T08:10:25.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.062+0000 7f5758c00640 1 -- 172.21.15.69:0/3874934798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57540b1f20 con 0x7f5754160410 2024-04-24T08:10:25.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.062+0000 7f5758c00640 1 -- 172.21.15.69:0/3874934798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57540b24e0 con 0x7f5754160410 2024-04-24T08:10:25.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.063+0000 7f5758c00640 1 -- 172.21.15.69:0/3874934798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5754068a80 con 0x7f5754160410 2024-04-24T08:10:25.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.064+0000 7f574aa00640 1 -- 172.21.15.69:0/3874934798 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f573c030770 con 0x7f5754160410 2024-04-24T08:10:25.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.065+0000 7f574aa00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f573006c8f0 0x7f573006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.065+0000 7f574aa00640 1 -- 172.21.15.69:0/3874934798 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5357+0+0 (secure 0 0 0) 0x7f573c0acd10 con 0x7f5754160410 2024-04-24T08:10:25.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.066+0000 7f5751e00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f573006c8f0 0x7f573006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.067+0000 7f5751e00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f573006c8f0 0x7f573006edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5744004520 tx=0x7f5744009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:25.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.069+0000 7f574aa00640 1 -- 172.21.15.69:0/3874934798 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f573c0b2020 con 0x7f5754160410 2024-04-24T08:10:25.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.269+0000 7f5758c00640 1 -- 172.21.15.69:0/3874934798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f575409ba10 con 0x7f5754160410 2024-04-24T08:10:25.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.828+0000 7f574aa00640 1 -- 172.21.15.69:0/3874934798 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) ==== 169+0+0 (secure 0 0 0) 0x7f573c07b200 con 0x7f5754160410 2024-04-24T08:10:25.829 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' removed 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 -- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f573006c8f0 msgr2=0x7f573006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f573006c8f0 0x7f573006edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f5744004520 tx=0x7f5744009040 comp rx=0 tx=0).stop 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 -- 172.21.15.69:0/3874934798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160410 msgr2=0x7f575414eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160410 0x7f575414eb00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f573c000c00 tx=0x7f573c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 -- 172.21.15.69:0/3874934798 shutdown_connections 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f573006c8f0 0x7f573006edb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5754160410 0x7f575414eb00 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5754160fb0 0x7f57540c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 --2- 172.21.15.69:0/3874934798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f575415fc10 0x7f57540b2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 -- 172.21.15.69:0/3874934798 >> 172.21.15.69:0/3874934798 conn(0x7f57540b4090 msgr2=0x7f57540ce710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 -- 172.21.15.69:0/3874934798 shutdown_connections 2024-04-24T08:10:25.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:25.832+0000 7f5758c00640 1 -- 172.21.15.69:0/3874934798 wait complete. 2024-04-24T08:10:25.858 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-24T08:10:26.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.052+0000 7f9ccbe00640 1 Processor -- start 2024-04-24T08:10:26.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.052+0000 7f9ccbe00640 1 -- start start 2024-04-24T08:10:26.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.053+0000 7f9ccbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9cc415c820 0x7f9cc40b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.053+0000 7f9ccbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cc40b46a0 0x7f9cc406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.053+0000 7f9ccbe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc415cc00 con 0x7f9cc406fd10 2024-04-24T08:10:26.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.054+0000 7f9ccbe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc406ed30 con 0x7f9cc415c820 2024-04-24T08:10:26.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.054+0000 7f9ccbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc40b4e10 con 0x7f9cc40b46a0 2024-04-24T08:10:26.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.054+0000 7f9cc9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9cc415c820 0x7f9cc40b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.054+0000 7f9cc9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9cc415c820 0x7f9cc40b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54612/0 (socket says 172.21.15.69:54612) 2024-04-24T08:10:26.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.054+0000 7f9cc9400640 1 -- 172.21.15.69:0/447246419 learned_addr learned my addr 172.21.15.69:0/447246419 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:26.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.054+0000 7f9cc9400640 1 -- 172.21.15.69:0/447246419 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cc40b46a0 msgr2=0x7f9cc406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:26.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.054+0000 7f9cc9400640 1 --2- 172.21.15.69:0/447246419 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cc40b46a0 0x7f9cc406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.054+0000 7f9cc9400640 1 -- 172.21.15.69:0/447246419 >> v1:172.21.15.121:6789/0 conn(0x7f9cc406fd10 legacy=0x7f9cc40ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:26.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.054+0000 7f9cc9400640 1 -- 172.21.15.69:0/447246419 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc40b50f0 con 0x7f9cc415c820 2024-04-24T08:10:26.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.055+0000 7f9cc9400640 1 --2- 172.21.15.69:0/447246419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 0x7f9cc40b3e90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009e50 tx=0x7f9cb802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae908a026c81428 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:26.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.056+0000 7f9cc3e00640 1 -- 172.21.15.69:0/447246419 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8036070 con 0x7f9cc415c820 2024-04-24T08:10:26.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.056+0000 7f9cc3e00640 1 -- 172.21.15.69:0/447246419 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb8031600 con 0x7f9cc415c820 2024-04-24T08:10:26.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.056+0000 7f9cc3e00640 1 -- 172.21.15.69:0/447246419 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80319d0 con 0x7f9cc415c820 2024-04-24T08:10:26.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.056+0000 7f9ccbe00640 1 -- 172.21.15.69:0/447246419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 msgr2=0x7f9cc40b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.056+0000 7f9ccbe00640 1 --2- 172.21.15.69:0/447246419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 0x7f9cc40b3e90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009e50 tx=0x7f9cb802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.056+0000 7f9ccbe00640 1 -- 172.21.15.69:0/447246419 shutdown_connections 2024-04-24T08:10:26.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.056+0000 7f9ccbe00640 1 --2- 172.21.15.69:0/447246419 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cc40b46a0 0x7f9cc406f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.056+0000 7f9ccbe00640 1 --2- 172.21.15.69:0/447246419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 0x7f9cc40b3e90 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.056+0000 7f9ccbe00640 1 -- 172.21.15.69:0/447246419 >> 172.21.15.69:0/447246419 conn(0x7f9cc40d0350 msgr2=0x7f9cc40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:26.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.057+0000 7f9ccbe00640 1 -- 172.21.15.69:0/447246419 shutdown_connections 2024-04-24T08:10:26.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.057+0000 7f9ccbe00640 1 -- 172.21.15.69:0/447246419 wait complete. 2024-04-24T08:10:26.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.058+0000 7f9ccbe00640 1 Processor -- start 2024-04-24T08:10:26.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.058+0000 7f9ccbe00640 1 -- start start 2024-04-24T08:10:26.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.059+0000 7f9ccbe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cc406fd10 0x7f9cc414e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.059+0000 7f9ccbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cc40b46a0 0x7f9cc406ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.059+0000 7f9ccbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 0x7f9cc406bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.059+0000 7f9ccbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc40b4b00 con 0x7f9cc415c820 2024-04-24T08:10:26.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.059+0000 7f9ccbe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc406f2b0 con 0x7f9cc406fd10 2024-04-24T08:10:26.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.059+0000 7f9ccbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc40ce960 con 0x7f9cc40b46a0 2024-04-24T08:10:26.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 0x7f9cc406bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 0x7f9cc406bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54626/0 (socket says 172.21.15.69:54626) 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc9e00640 1 -- 172.21.15.69:0/2710669546 learned_addr learned my addr 172.21.15.69:0/2710669546 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc9400640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cc406fd10 0x7f9cc414e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc8a00640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cc40b46a0 0x7f9cc406ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc9e00640 1 -- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cc40b46a0 msgr2=0x7f9cc406ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc9e00640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cc40b46a0 0x7f9cc406ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc9e00640 1 -- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cc406fd10 msgr2=0x7f9cc414e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc9e00640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cc406fd10 0x7f9cc414e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc9e00640 1 -- 172.21.15.69:0/2710669546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc415cea0 con 0x7f9cc415c820 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.060+0000 7f9cc9e00640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 0x7f9cc406bf60 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00b4c0 tx=0x7f9cbc00b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.061+0000 7f9cc1600640 1 -- 172.21.15.69:0/2710669546 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc015020 con 0x7f9cc415c820 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.061+0000 7f9cc1600640 1 -- 172.21.15.69:0/2710669546 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cbc011850 con 0x7f9cc415c820 2024-04-24T08:10:26.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.061+0000 7f9cc1600640 1 -- 172.21.15.69:0/2710669546 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc011c40 con 0x7f9cc415c820 2024-04-24T08:10:26.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.061+0000 7f9ccbe00640 1 -- 172.21.15.69:0/2710669546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc415d180 con 0x7f9cc415c820 2024-04-24T08:10:26.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.061+0000 7f9ccbe00640 1 -- 172.21.15.69:0/2710669546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc414eb50 con 0x7f9cc415c820 2024-04-24T08:10:26.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.062+0000 7f9ccbe00640 1 -- 172.21.15.69:0/2710669546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc4068a80 con 0x7f9cc415c820 2024-04-24T08:10:26.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.063+0000 7f9cc1600640 1 -- 172.21.15.69:0/2710669546 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9cbc018020 con 0x7f9cc415c820 2024-04-24T08:10:26.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.064+0000 7f9cc1600640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ca406c940 0x7f9ca406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.064+0000 7f9cc1600640 1 -- 172.21.15.69:0/2710669546 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f9cbc08b710 con 0x7f9cc415c820 2024-04-24T08:10:26.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.065+0000 7f9cc9400640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ca406c940 0x7f9ca406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.066+0000 7f9cc9400640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ca406c940 0x7f9ca406ee00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8033000 tx=0x7f9cb8009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:26.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.069+0000 7f9cc1600640 1 -- 172.21.15.69:0/2710669546 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9cbc01f020 con 0x7f9cc415c820 2024-04-24T08:10:26.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.264+0000 7f9ccbe00640 1 -- 172.21.15.69:0/2710669546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f9cc40cddc0 con 0x7f9cc415c820 2024-04-24T08:10:26.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.269+0000 7f9cc1600640 1 -- 172.21.15.69:0/2710669546 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f9cbc0172c0 con 0x7f9cc415c820 2024-04-24T08:10:26.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.271+0000 7f9ccbe00640 1 -- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ca406c940 msgr2=0x7f9ca406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.271+0000 7f9ccbe00640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ca406c940 0x7f9ca406ee00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8033000 tx=0x7f9cb8009000 comp rx=0 tx=0).stop 2024-04-24T08:10:26.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.272+0000 7f9ccbe00640 1 -- 172.21.15.69:0/2710669546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 msgr2=0x7f9cc406bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.272+0000 7f9ccbe00640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 0x7f9cc406bf60 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00b4c0 tx=0x7f9cbc00b980 comp rx=0 tx=0).stop 2024-04-24T08:10:26.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.272+0000 7f9ccbe00640 1 -- 172.21.15.69:0/2710669546 shutdown_connections 2024-04-24T08:10:26.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.272+0000 7f9ccbe00640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9ca406c940 0x7f9ca406ee00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.272+0000 7f9ccbe00640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cc415c820 0x7f9cc406bf60 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.272+0000 7f9ccbe00640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9cc40b46a0 0x7f9cc406ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.272+0000 7f9ccbe00640 1 --2- 172.21.15.69:0/2710669546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cc406fd10 0x7f9cc414e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.272+0000 7f9ccbe00640 1 -- 172.21.15.69:0/2710669546 >> 172.21.15.69:0/2710669546 conn(0x7f9cc40d0350 msgr2=0x7f9cc415a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:26.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.272+0000 7f9ccbe00640 1 -- 172.21.15.69:0/2710669546 shutdown_connections 2024-04-24T08:10:26.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.272+0000 7f9ccbe00640 1 -- 172.21.15.69:0/2710669546 wait complete. 2024-04-24T08:10:26.289 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-04-24T08:10:26.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.477+0000 7f05d3200640 1 Processor -- start 2024-04-24T08:10:26.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.477+0000 7f05d3200640 1 -- start start 2024-04-24T08:10:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.479+0000 7f05d3200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f05d4162300 0x7f05d4143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.479+0000 7f05d3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05d415f540 con 0x7f05d4071e50 2024-04-24T08:10:26.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.479+0000 7f05d3200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05d415f820 con 0x7f05d4072230 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.479+0000 7f05d3200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f05d415fb00 con 0x7f05d4162300 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.479+0000 7f05d1e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f05d4071e50 0x7f05d415ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51210/0 (socket says 172.21.15.69:51210) 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.479+0000 7f05d1e00640 1 -- 172.21.15.69:0/3568330499 learned_addr learned my addr 172.21.15.69:0/3568330499 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d1400640 1 --2- 172.21.15.69:0/3568330499 >> v2:172.21.15.121:3300/0 conn(0x7f05d4162300 0x7f05d4143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3853551978 0 0) 0x7f05d415f540 con 0x7f05d4071e50 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05b8003520 con 0x7f05d4071e50 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1192576262 0 0) 0x7f05b8003520 con 0x7f05d4071e50 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 >> v2:172.21.15.121:3300/0 conn(0x7f05d4162300 msgr2=0x7f05d4143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 --2- 172.21.15.69:0/3568330499 >> v2:172.21.15.121:3300/0 conn(0x7f05d4162300 0x7f05d4143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 >> v1:172.21.15.121:6789/0 conn(0x7f05d4072230 legacy=0x7f05d4143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d406dda0 con 0x7f05d4071e50 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f05c4001670 con 0x7f05d4071e50 2024-04-24T08:10:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 >> v1:172.21.15.69:6789/0 conn(0x7f05d4071e50 legacy=0x7f05d415ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 --2- 172.21.15.69:0/3568330499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b8003a00 0x7f05b8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 --2- 172.21.15.69:0/3568330499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05b8004800 0x7f05b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 --2- 172.21.15.69:0/3568330499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b80051c0 0x7f05b800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f05d415fb00 con 0x7f05b8003a00 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f05b800c0c0 con 0x7f05b80051c0 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f05b800c2c0 con 0x7f05b8004800 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.480+0000 7f05d1e00640 1 --2- 172.21.15.69:0/3568330499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b8003a00 0x7f05b8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.481+0000 7f05d1e00640 1 -- 172.21.15.69:0/3568330499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05b8004800 msgr2=0x7f05b8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.481+0000 7f05d1e00640 1 --2- 172.21.15.69:0/3568330499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05b8004800 0x7f05b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.481+0000 7f05d1e00640 1 -- 172.21.15.69:0/3568330499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b80051c0 msgr2=0x7f05b800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.481+0000 7f05d1e00640 1 --2- 172.21.15.69:0/3568330499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b80051c0 0x7f05b800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.481+0000 7f05d1e00640 1 -- 172.21.15.69:0/3568330499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b8003520 con 0x7f05b8003a00 2024-04-24T08:10:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.481+0000 7f05d2800640 1 -- 172.21.15.69:0/3568330499 reap_dead start 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.482+0000 7f05d1e00640 1 --2- 172.21.15.69:0/3568330499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b8003a00 0x7f05b8003e20 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f05c400c0e0 tx=0x7f05b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c65562a90655566 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.482+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c4007c20 con 0x7f05b8003a00 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.482+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05c4005ad0 con 0x7f05b8003a00 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.482+0000 7f05d0a00640 1 -- 172.21.15.69:0/3568330499 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c4007070 con 0x7f05b8003a00 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.483+0000 7f05d3200640 1 -- 172.21.15.69:0/3568330499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b8003a00 msgr2=0x7f05b8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.483+0000 7f05d3200640 1 --2- 172.21.15.69:0/3568330499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b8003a00 0x7f05b8003e20 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f05c400c0e0 tx=0x7f05b8005830 comp rx=0 tx=0).stop 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.483+0000 7f05d3200640 1 -- 172.21.15.69:0/3568330499 shutdown_connections 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.483+0000 7f05d3200640 1 --2- 172.21.15.69:0/3568330499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05b8003a00 0x7f05b8003e20 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.483+0000 7f05d3200640 1 -- 172.21.15.69:0/3568330499 >> 172.21.15.69:0/3568330499 conn(0x7f05d40d0160 msgr2=0x7f05d40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.483+0000 7f05d3200640 1 -- 172.21.15.69:0/3568330499 shutdown_connections 2024-04-24T08:10:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.483+0000 7f05d3200640 1 -- 172.21.15.69:0/3568330499 wait complete. 2024-04-24T08:10:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.485+0000 7f05d3200640 1 Processor -- start 2024-04-24T08:10:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.485+0000 7f05d3200640 1 -- start start 2024-04-24T08:10:26.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d3200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b8003a00 0x7f05d409b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05d415a270 0x7f05d40af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05d40ae120 0x7f05d40ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f05d415f050 con 0x7f05d415a270 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d3200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f05d415eed0 con 0x7f05b8003a00 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f05d415f1d0 con 0x7f05d40ae120 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05d415a270 0x7f05d40af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05d415a270 0x7f05d40af220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54642/0 (socket says 172.21.15.69:54642) 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d1400640 1 -- 172.21.15.69:0/2717377818 learned_addr learned my addr 172.21.15.69:0/2717377818 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d1e00640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b8003a00 0x7f05d409b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d1400640 1 -- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05d40ae120 msgr2=0x7f05d40ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d1400640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05d40ae120 0x7f05d40ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d1400640 1 -- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b8003a00 msgr2=0x7f05d409b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d1400640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b8003a00 0x7f05d409b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.486+0000 7f05d1400640 1 -- 172.21.15.69:0/2717377818 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d41e8e80 con 0x7f05d415a270 2024-04-24T08:10:26.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.487+0000 7f05d1400640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05d415a270 0x7f05d40af220 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f05c000e7a0 tx=0x7f05c000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:26.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.487+0000 7f05ca000640 1 -- 172.21.15.69:0/2717377818 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c0017070 con 0x7f05d415a270 2024-04-24T08:10:26.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.487+0000 7f05ca000640 1 -- 172.21.15.69:0/2717377818 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05c000c730 con 0x7f05d415a270 2024-04-24T08:10:26.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.487+0000 7f05ca000640 1 -- 172.21.15.69:0/2717377818 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c000cb00 con 0x7f05d415a270 2024-04-24T08:10:26.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.487+0000 7f05d3200640 1 -- 172.21.15.69:0/2717377818 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d41e9160 con 0x7f05d415a270 2024-04-24T08:10:26.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.487+0000 7f05d3200640 1 -- 172.21.15.69:0/2717377818 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d41e9720 con 0x7f05d415a270 2024-04-24T08:10:26.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.488+0000 7f05d3200640 1 -- 172.21.15.69:0/2717377818 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05d4075d50 con 0x7f05d415a270 2024-04-24T08:10:26.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.489+0000 7f05ca000640 1 -- 172.21.15.69:0/2717377818 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f05c001a070 con 0x7f05d415a270 2024-04-24T08:10:26.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.490+0000 7f05ca000640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05b406c8f0 0x7f05b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.490+0000 7f05ca000640 1 -- 172.21.15.69:0/2717377818 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f05c008ba00 con 0x7f05d415a270 2024-04-24T08:10:26.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.491+0000 7f05d1e00640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05b406c8f0 0x7f05b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.492+0000 7f05d1e00640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05b406c8f0 0x7f05b406edb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f05c4032000 tx=0x7f05c4009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:26.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.495+0000 7f05ca000640 1 -- 172.21.15.69:0/2717377818 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05c005a090 con 0x7f05d415a270 2024-04-24T08:10:26.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.769+0000 7f05d3200640 1 -- 172.21.15.69:0/2717377818 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) -- 0x7f05d40b4fe0 con 0x7f05d415a270 2024-04-24T08:10:26.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.847+0000 7f05ca000640 1 -- 172.21.15.69:0/2717377818 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v132) ==== 142+0+0 (secure 0 0 0) 0x7f05c0059990 con 0x7f05d415a270 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 -- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05b406c8f0 msgr2=0x7f05b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05b406c8f0 0x7f05b406edb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f05c4032000 tx=0x7f05c4009030 comp rx=0 tx=0).stop 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 -- 172.21.15.69:0/2717377818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05d415a270 msgr2=0x7f05d40af220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05d415a270 0x7f05d40af220 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f05c000e7a0 tx=0x7f05c000eba0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 -- 172.21.15.69:0/2717377818 shutdown_connections 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05b406c8f0 0x7f05b406edb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05d415a270 0x7f05d40af220 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05d40ae120 0x7f05d40ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 --2- 172.21.15.69:0/2717377818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05b8003a00 0x7f05d409b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 -- 172.21.15.69:0/2717377818 >> 172.21.15.69:0/2717377818 conn(0x7f05d40d0160 msgr2=0x7f05d40b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 -- 172.21.15.69:0/2717377818 shutdown_connections 2024-04-24T08:10:26.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:26.850+0000 7f05d3200640 1 -- 172.21.15.69:0/2717377818 wait complete. 2024-04-24T08:10:26.870 INFO:tasks.cephfs_test_runner:test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-04-24T08:10:26.870 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) That the MDS can repair an inodes backtrace in the data pool 2024-04-24T08:10:26.870 INFO:tasks.cephfs_test_runner:test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-24T08:10:26.871 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:27.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.045+0000 7f6f03200640 1 Processor -- start 2024-04-24T08:10:27.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.046+0000 7f6f03200640 1 -- start start 2024-04-24T08:10:27.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.047+0000 7f6f03200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6f04156c60 0x7f6f0415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.047+0000 7f6f03200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f040b30f0 0x7f6f0415d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.047+0000 7f6f03200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0415d660 con 0x7f6f040ad0d0 2024-04-24T08:10:27.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.047+0000 7f6f03200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0406dc30 con 0x7f6f04156c60 2024-04-24T08:10:27.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.047+0000 7f6f03200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0406df10 con 0x7f6f040b30f0 2024-04-24T08:10:27.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.047+0000 7f6f02800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6f040ad0d0 0x7f6f041600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51220/0 (socket says 172.21.15.69:51220) 2024-04-24T08:10:27.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6f02800640 1 -- 172.21.15.69:0/1520248137 learned_addr learned my addr 172.21.15.69:0/1520248137 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:27.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6f01e00640 1 --2- 172.21.15.69:0/1520248137 >> v2:172.21.15.121:3300/0 conn(0x7f6f04156c60 0x7f6f0415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6efbe00640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f040b30f0 0x7f6f0415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1223305607 0 0) 0x7f6f0415d660 con 0x7f6f040ad0d0 2024-04-24T08:10:27.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ee8003520 con 0x7f6f040ad0d0 2024-04-24T08:10:27.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3244435095 0 0) 0x7f6ee8003520 con 0x7f6f040ad0d0 2024-04-24T08:10:27.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f040b30f0 msgr2=0x7f6f0415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:27.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6f01400640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f040b30f0 0x7f6f0415d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 >> v2:172.21.15.121:3300/0 conn(0x7f6f04156c60 msgr2=0x7f6f0415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6f01400640 1 --2- 172.21.15.69:0/1520248137 >> v2:172.21.15.121:3300/0 conn(0x7f6f04156c60 0x7f6f0415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.048+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0406e1f0 con 0x7f6f040ad0d0 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f6efc0019f0 con 0x7f6f040ad0d0 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 >> v1:172.21.15.69:6789/0 conn(0x7f6f040ad0d0 legacy=0x7f6f041600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f01400640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ee80039b0 0x7f6ee8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f01400640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f040b30f0 0x7f6ee8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f01400640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ee8004d40 0x7f6ee80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0406dc30 con 0x7f6ee8004d40 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0406df10 con 0x7f6ee80039b0 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6ee800b9c0 con 0x7f6f040b30f0 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f01e00640 1 --2- 172.21.15.69:0/1520248137 >> v2:172.21.15.121:3300/0 conn(0x7f6f04156c60 0x7f6f0415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f02800640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ee8004d40 0x7f6ee80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6efbe00640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f040b30f0 0x7f6ee8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f02800640 1 -- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f040b30f0 msgr2=0x7f6ee8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:27.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f02800640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f040b30f0 0x7f6ee8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f02800640 1 -- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ee80039b0 msgr2=0x7f6ee8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:27.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.049+0000 7f6f01e00640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ee80039b0 0x7f6ee8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.050+0000 7f6f02800640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ee80039b0 0x7f6ee8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.050+0000 7f6f02800640 1 -- 172.21.15.69:0/1520248137 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee8003520 con 0x7f6ee8004d40 2024-04-24T08:10:27.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.050+0000 7f6f01e00640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ee80039b0 0x7f6ee8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:27.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.051+0000 7f6f02800640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ee8004d40 0x7f6ee80051c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00a8a0 tx=0x7f6ee8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24625b54d92b4c8b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.051+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc00a4c0 con 0x7f6ee8004d40 2024-04-24T08:10:27.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.051+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6efc02f070 con 0x7f6ee8004d40 2024-04-24T08:10:27.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.051+0000 7f6f01400640 1 -- 172.21.15.69:0/1520248137 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc02bd30 con 0x7f6ee8004d40 2024-04-24T08:10:27.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.051+0000 7f6f03200640 1 -- 172.21.15.69:0/1520248137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ee8004d40 msgr2=0x7f6ee80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.051+0000 7f6f03200640 1 --2- 172.21.15.69:0/1520248137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6ee8004d40 0x7f6ee80051c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00a8a0 tx=0x7f6ee8005830 comp rx=0 tx=0).stop 2024-04-24T08:10:27.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.051+0000 7f6f02800640 1 -- 172.21.15.69:0/1520248137 reap_dead start 2024-04-24T08:10:27.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.051+0000 7f6f03200640 1 -- 172.21.15.69:0/1520248137 shutdown_connections 2024-04-24T08:10:27.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.051+0000 7f6f03200640 1 -- 172.21.15.69:0/1520248137 >> 172.21.15.69:0/1520248137 conn(0x7f6f040d0160 msgr2=0x7f6f040d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.052+0000 7f6f03200640 1 -- 172.21.15.69:0/1520248137 shutdown_connections 2024-04-24T08:10:27.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.052+0000 7f6f03200640 1 -- 172.21.15.69:0/1520248137 wait complete. 2024-04-24T08:10:27.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.053+0000 7f6f03200640 1 Processor -- start 2024-04-24T08:10:27.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.053+0000 7f6f03200640 1 -- start start 2024-04-24T08:10:27.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.054+0000 7f6f03200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ee8004d40 0x7f6f041607f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.054+0000 7f6f03200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f04143180 0x7f6f04143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.054+0000 7f6f03200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04155820 0x7f6f04155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.054+0000 7f6f03200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0415d000 con 0x7f6f04143180 2024-04-24T08:10:27.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.054+0000 7f6f03200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0415d940 con 0x7f6f04155820 2024-04-24T08:10:27.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.054+0000 7f6f03200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0415d180 con 0x7f6ee8004d40 2024-04-24T08:10:27.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6efbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f04143180 0x7f6f04143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6efbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f04143180 0x7f6f04143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54670/0 (socket says 172.21.15.69:54670) 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6efbe00640 1 -- 172.21.15.69:0/3856471553 learned_addr learned my addr 172.21.15.69:0/3856471553 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6f01e00640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ee8004d40 0x7f6f041607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6f02800640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04155820 0x7f6f04155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6efbe00640 1 -- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ee8004d40 msgr2=0x7f6f041607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6efbe00640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ee8004d40 0x7f6f041607f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6efbe00640 1 -- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04155820 msgr2=0x7f6f04155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6efbe00640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04155820 0x7f6f04155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6efbe00640 1 -- 172.21.15.69:0/3856471553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f041f4190 con 0x7f6f04143180 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.055+0000 7f6efbe00640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f04143180 0x7f6f04143600 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0115d0 tx=0x7f6eec011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.056+0000 7f6efa000640 1 -- 172.21.15.69:0/3856471553 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec015870 con 0x7f6f04143180 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.056+0000 7f6efa000640 1 -- 172.21.15.69:0/3856471553 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eec015a10 con 0x7f6f04143180 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.056+0000 7f6efa000640 1 -- 172.21.15.69:0/3856471553 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec015de0 con 0x7f6f04143180 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.056+0000 7f6f03200640 1 -- 172.21.15.69:0/3856471553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f041f4450 con 0x7f6f04143180 2024-04-24T08:10:27.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.056+0000 7f6f03200640 1 -- 172.21.15.69:0/3856471553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f041f4a10 con 0x7f6f04143180 2024-04-24T08:10:27.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.057+0000 7f6f03200640 1 -- 172.21.15.69:0/3856471553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f04075d50 con 0x7f6f04143180 2024-04-24T08:10:27.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.058+0000 7f6efa000640 1 -- 172.21.15.69:0/3856471553 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6eec00f480 con 0x7f6f04143180 2024-04-24T08:10:27.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.059+0000 7f6efa000640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ee006c8f0 0x7f6ee006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.059+0000 7f6efa000640 1 -- 172.21.15.69:0/3856471553 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f6eec08c200 con 0x7f6f04143180 2024-04-24T08:10:27.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.059+0000 7f6f01e00640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ee006c8f0 0x7f6ee006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.061+0000 7f6f01e00640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ee006c8f0 0x7f6ee006edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6f0415c800 tx=0x7f6ef4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.063+0000 7f6efa000640 1 -- 172.21.15.69:0/3856471553 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6eec019030 con 0x7f6f04143180 2024-04-24T08:10:27.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.262+0000 7f6f03200640 1 -- 172.21.15.69:0/3856471553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f0406f9e0 con 0x7f6f04143180 2024-04-24T08:10:27.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.264+0000 7f6efa000640 1 -- 172.21.15.69:0/3856471553 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+12994 (secure 0 0 0) 0x7f6eec05a920 con 0x7f6f04143180 2024-04-24T08:10:27.265 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:27.265 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":36,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:10:25.825350+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/991803145":"2024-04-25T08:10:22.813791+0000","172.21.15.69:6834/991803145":"2024-04-25T08:10:22.813791+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:27.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 -- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ee006c8f0 msgr2=0x7f6ee006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ee006c8f0 0x7f6ee006edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6f0415c800 tx=0x7f6ef4008040 comp rx=0 tx=0).stop 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 -- 172.21.15.69:0/3856471553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f04143180 msgr2=0x7f6f04143600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f04143180 0x7f6f04143600 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0115d0 tx=0x7f6eec011a90 comp rx=0 tx=0).stop 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 -- 172.21.15.69:0/3856471553 shutdown_connections 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6ee006c8f0 0x7f6ee006edb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f04143180 0x7f6f04143600 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f04155820 0x7f6f04155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 --2- 172.21.15.69:0/3856471553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6ee8004d40 0x7f6f041607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 -- 172.21.15.69:0/3856471553 >> 172.21.15.69:0/3856471553 conn(0x7f6f040d0160 msgr2=0x7f6f040aefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 -- 172.21.15.69:0/3856471553 shutdown_connections 2024-04-24T08:10:27.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.267+0000 7f6f03200640 1 -- 172.21.15.69:0/3856471553 wait complete. 2024-04-24T08:10:27.286 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:10:27.287 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:10:27.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.489+0000 7f9e01e00640 1 Processor -- start 2024-04-24T08:10:27.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.490+0000 7f9e01e00640 1 -- start start 2024-04-24T08:10:27.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.491+0000 7f9e01e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9dfc15ea70 0x7f9dfc0d5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.491+0000 7f9e01e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9dfc0d02f0 0x7f9dfc0d0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.491+0000 7f9e01e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dfc0d55a0 con 0x7f9dfc0d1960 2024-04-24T08:10:27.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.491+0000 7f9e01e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9dfc0d1f20 con 0x7f9dfc15ea70 2024-04-24T08:10:27.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.491+0000 7f9e01e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9dfc0cf810 con 0x7f9dfc0d02f0 2024-04-24T08:10:27.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.491+0000 7f9dfa000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9dfc0d1960 0x7f9dfc06cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51224/0 (socket says 172.21.15.69:51224) 2024-04-24T08:10:27.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.491+0000 7f9dfa000640 1 -- 172.21.15.69:0/811740983 learned_addr learned my addr 172.21.15.69:0/811740983 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:27.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.491+0000 7f9dfaa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9dfc15ea70 0x7f9dfc0d5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.492+0000 7f9dfb400640 1 --2- 172.21.15.69:0/811740983 >> v2:172.21.15.121:3300/0 conn(0x7f9dfc0d02f0 0x7f9dfc0d0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.492+0000 7f9df9600640 1 -- 172.21.15.69:0/811740983 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3444874930 0 0) 0x7f9dfc0d55a0 con 0x7f9dfc0d1960 2024-04-24T08:10:27.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.492+0000 7f9df9600640 1 -- 172.21.15.69:0/811740983 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ddc003520 con 0x7f9dfc0d1960 2024-04-24T08:10:27.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.492+0000 7f9dfaa00640 1 -- 172.21.15.69:0/811740983 >> v2:172.21.15.121:3300/0 conn(0x7f9dfc0d02f0 msgr2=0x7f9dfc0d0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:27.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.492+0000 7f9dfaa00640 1 --2- 172.21.15.69:0/811740983 >> v2:172.21.15.121:3300/0 conn(0x7f9dfc0d02f0 0x7f9dfc0d0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.492+0000 7f9dfaa00640 1 -- 172.21.15.69:0/811740983 >> v1:172.21.15.69:6789/0 conn(0x7f9dfc0d1960 legacy=0x7f9dfc06cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.492+0000 7f9dfaa00640 1 -- 172.21.15.69:0/811740983 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dfc0cfaf0 con 0x7f9dfc15ea70 2024-04-24T08:10:27.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.493+0000 7f9dfaa00640 1 --2- 172.21.15.69:0/811740983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 0x7f9dfc0d5060 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc06ecb0 tx=0x7f9dec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968bb9b0846b647c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.493+0000 7f9df9600640 1 -- 172.21.15.69:0/811740983 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec037070 con 0x7f9dfc15ea70 2024-04-24T08:10:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.493+0000 7f9df9600640 1 -- 172.21.15.69:0/811740983 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dec032440 con 0x7f9dfc15ea70 2024-04-24T08:10:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.493+0000 7f9df9600640 1 -- 172.21.15.69:0/811740983 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dec032810 con 0x7f9dfc15ea70 2024-04-24T08:10:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.494+0000 7f9e01e00640 1 -- 172.21.15.69:0/811740983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 msgr2=0x7f9dfc0d5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.494+0000 7f9e01e00640 1 --2- 172.21.15.69:0/811740983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 0x7f9dfc0d5060 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc06ecb0 tx=0x7f9dec02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.494+0000 7f9e01e00640 1 -- 172.21.15.69:0/811740983 shutdown_connections 2024-04-24T08:10:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.494+0000 7f9e01e00640 1 --2- 172.21.15.69:0/811740983 >> v2:172.21.15.121:3300/0 conn(0x7f9dfc0d02f0 0x7f9dfc0d0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.494+0000 7f9e01e00640 1 --2- 172.21.15.69:0/811740983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 0x7f9dfc0d5060 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.494+0000 7f9e01e00640 1 -- 172.21.15.69:0/811740983 >> 172.21.15.69:0/811740983 conn(0x7f9dfc08bf60 msgr2=0x7f9dfc0d46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.494+0000 7f9e01e00640 1 -- 172.21.15.69:0/811740983 shutdown_connections 2024-04-24T08:10:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.494+0000 7f9e01e00640 1 -- 172.21.15.69:0/811740983 wait complete. 2024-04-24T08:10:27.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.496+0000 7f9e01e00640 1 Processor -- start 2024-04-24T08:10:27.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.496+0000 7f9e01e00640 1 -- start start 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.497+0000 7f9e01e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dfc0d02f0 0x7f9dfc0b5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.497+0000 7f9e01e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0d1960 0x7f9dfc15b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.497+0000 7f9e01e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 0x7f9dfc157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.497+0000 7f9e01e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9dfc0cf4c0 con 0x7f9dfc15ea70 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.497+0000 7f9e01e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9dfc15ee50 con 0x7f9dfc0d1960 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.497+0000 7f9e01e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9dfc06d360 con 0x7f9dfc0d02f0 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.497+0000 7f9dfb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 0x7f9dfc157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.497+0000 7f9dfb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 0x7f9dfc157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54686/0 (socket says 172.21.15.69:54686) 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.497+0000 7f9dfb400640 1 -- 172.21.15.69:0/3670888655 learned_addr learned my addr 172.21.15.69:0/3670888655 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9dfa000640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0d1960 0x7f9dfc15b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9dfaa00640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dfc0d02f0 0x7f9dfc0b5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9dfb400640 1 -- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dfc0d02f0 msgr2=0x7f9dfc0b5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9dfb400640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dfc0d02f0 0x7f9dfc0b5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9dfb400640 1 -- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0d1960 msgr2=0x7f9dfc15b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9dfb400640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0d1960 0x7f9dfc15b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9dfb400640 1 -- 172.21.15.69:0/3670888655 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dfc157fb0 con 0x7f9dfc15ea70 2024-04-24T08:10:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9dfb400640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 0x7f9dfc157a70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9df400d830 tx=0x7f9df400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9df3400640 1 -- 172.21.15.69:0/3670888655 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df400b6c0 con 0x7f9dfc15ea70 2024-04-24T08:10:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9df3400640 1 -- 172.21.15.69:0/3670888655 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df4019070 con 0x7f9dfc15ea70 2024-04-24T08:10:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9df3400640 1 -- 172.21.15.69:0/3670888655 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df400bac0 con 0x7f9dfc15ea70 2024-04-24T08:10:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.498+0000 7f9e01e00640 1 -- 172.21.15.69:0/3670888655 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dfc0b6290 con 0x7f9dfc15ea70 2024-04-24T08:10:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.499+0000 7f9e01e00640 1 -- 172.21.15.69:0/3670888655 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dfc0b6850 con 0x7f9dfc15ea70 2024-04-24T08:10:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.500+0000 7f9e01e00640 1 -- 172.21.15.69:0/3670888655 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dfc068a80 con 0x7f9dfc15ea70 2024-04-24T08:10:27.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.501+0000 7f9df3400640 1 -- 172.21.15.69:0/3670888655 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9df401e460 con 0x7f9dfc15ea70 2024-04-24T08:10:27.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.502+0000 7f9df3400640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dd806c8f0 0x7f9dd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.502+0000 7f9df3400640 1 -- 172.21.15.69:0/3670888655 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f9df408be70 con 0x7f9dfc15ea70 2024-04-24T08:10:27.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.502+0000 7f9dfaa00640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dd806c8f0 0x7f9dd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.504+0000 7f9dfaa00640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dd806c8f0 0x7f9dd806edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9dec034000 tx=0x7f9dec02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.506+0000 7f9df3400640 1 -- 172.21.15.69:0/3670888655 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9df405a630 con 0x7f9dfc15ea70 2024-04-24T08:10:27.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.706+0000 7f9e01e00640 1 -- 172.21.15.69:0/3670888655 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9dfc0ba650 con 0x7f9dfc15ea70 2024-04-24T08:10:27.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.707+0000 7f9df3400640 1 -- 172.21.15.69:0/3670888655 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f9df4059f30 con 0x7f9dfc15ea70 2024-04-24T08:10:27.707 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:10:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.709+0000 7f9e01e00640 1 -- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dd806c8f0 msgr2=0x7f9dd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.709+0000 7f9e01e00640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dd806c8f0 0x7f9dd806edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9dec034000 tx=0x7f9dec02e000 comp rx=0 tx=0).stop 2024-04-24T08:10:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.709+0000 7f9e01e00640 1 -- 172.21.15.69:0/3670888655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 msgr2=0x7f9dfc157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.709+0000 7f9e01e00640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 0x7f9dfc157a70 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9df400d830 tx=0x7f9df400dcf0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.710+0000 7f9e01e00640 1 -- 172.21.15.69:0/3670888655 shutdown_connections 2024-04-24T08:10:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.710+0000 7f9e01e00640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dd806c8f0 0x7f9dd806edb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.710+0000 7f9e01e00640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9dfc15ea70 0x7f9dfc157a70 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.710+0000 7f9e01e00640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dfc0d1960 0x7f9dfc15b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.710+0000 7f9e01e00640 1 --2- 172.21.15.69:0/3670888655 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9dfc0d02f0 0x7f9dfc0b5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.710+0000 7f9e01e00640 1 -- 172.21.15.69:0/3670888655 >> 172.21.15.69:0/3670888655 conn(0x7f9dfc08bf60 msgr2=0x7f9dfc0b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.710+0000 7f9e01e00640 1 -- 172.21.15.69:0/3670888655 shutdown_connections 2024-04-24T08:10:27.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.710+0000 7f9e01e00640 1 -- 172.21.15.69:0/3670888655 wait complete. 2024-04-24T08:10:27.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-04-24T08:10:27.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.930+0000 7f67bda00640 1 Processor -- start 2024-04-24T08:10:27.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.930+0000 7f67bda00640 1 -- start start 2024-04-24T08:10:27.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.931+0000 7f67bda00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67b80d1100 0x7f67b80d1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.931+0000 7f67bda00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67b80b3f60 con 0x7f67b806cc00 2024-04-24T08:10:27.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.931+0000 7f67bda00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67b80d2750 con 0x7f67b80b3b80 2024-04-24T08:10:27.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.931+0000 7f67bda00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f67b80d1aa0 con 0x7f67b80d1100 2024-04-24T08:10:27.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.932+0000 7f67b7400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f67b806cc00 0x7f67b806d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51230/0 (socket says 172.21.15.69:51230) 2024-04-24T08:10:27.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.932+0000 7f67b6000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f67b80d1100 0x7f67b80d1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.932+0000 7f67b7400640 1 -- 172.21.15.69:0/1881356568 learned_addr learned my addr 172.21.15.69:0/1881356568 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:27.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.932+0000 7f67b6000640 1 -- 172.21.15.69:0/1881356568 >> v1:172.21.15.121:6789/0 conn(0x7f67b80b3b80 legacy=0x7f67b80d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:27.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.932+0000 7f67b6000640 1 -- 172.21.15.69:0/1881356568 >> v1:172.21.15.69:6789/0 conn(0x7f67b806cc00 legacy=0x7f67b806d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.932+0000 7f67b5600640 1 -- 172.21.15.69:0/1881356568 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 160691374 0 0) 0x7f67b80b3f60 con 0x7f67b806cc00 2024-04-24T08:10:27.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.932+0000 7f67b6000640 1 -- 172.21.15.69:0/1881356568 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b80d2a30 con 0x7f67b80d1100 2024-04-24T08:10:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.933+0000 7f67b6000640 1 --2- 172.21.15.69:0/1881356568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b80d1100 0x7f67b80d1560 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009d90 tx=0x7f67a002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32aecaae983e993a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.933+0000 7f67b5600640 1 -- 172.21.15.69:0/1881356568 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a0007740 con 0x7f67b80d1100 2024-04-24T08:10:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.933+0000 7f67b5600640 1 -- 172.21.15.69:0/1881356568 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a00078e0 con 0x7f67b80d1100 2024-04-24T08:10:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.933+0000 7f67b5600640 1 -- 172.21.15.69:0/1881356568 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a0007cb0 con 0x7f67b80d1100 2024-04-24T08:10:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.934+0000 7f67bda00640 1 -- 172.21.15.69:0/1881356568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b80d1100 msgr2=0x7f67b80d1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.934+0000 7f67bda00640 1 --2- 172.21.15.69:0/1881356568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b80d1100 0x7f67b80d1560 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009d90 tx=0x7f67a002a6a0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.934+0000 7f67bda00640 1 -- 172.21.15.69:0/1881356568 shutdown_connections 2024-04-24T08:10:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.934+0000 7f67bda00640 1 --2- 172.21.15.69:0/1881356568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b80d1100 0x7f67b80d1560 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.934+0000 7f67bda00640 1 -- 172.21.15.69:0/1881356568 >> 172.21.15.69:0/1881356568 conn(0x7f67b808bf60 msgr2=0x7f67b80d4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.934+0000 7f67bda00640 1 -- 172.21.15.69:0/1881356568 shutdown_connections 2024-04-24T08:10:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.934+0000 7f67bda00640 1 -- 172.21.15.69:0/1881356568 wait complete. 2024-04-24T08:10:27.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.936+0000 7f67bda00640 1 Processor -- start 2024-04-24T08:10:27.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.936+0000 7f67bda00640 1 -- start start 2024-04-24T08:10:27.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.937+0000 7f67bda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b806cc00 0x7f67b815d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.937+0000 7f67bda00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67b80b3b80 0x7f67b8162050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.937+0000 7f67bda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67b80d1100 0x7f67b815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.937+0000 7f67bda00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67b80d2ff0 con 0x7f67b806cc00 2024-04-24T08:10:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.937+0000 7f67bda00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f67b80d3360 con 0x7f67b80b3b80 2024-04-24T08:10:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.937+0000 7f67bda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f67b80d0760 con 0x7f67b80d1100 2024-04-24T08:10:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.937+0000 7f67b6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b806cc00 0x7f67b815d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b806cc00 0x7f67b815d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54698/0 (socket says 172.21.15.69:54698) 2024-04-24T08:10:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b6a00640 1 -- 172.21.15.69:0/3088392202 learned_addr learned my addr 172.21.15.69:0/3088392202 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b6000640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67b80b3b80 0x7f67b8162050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b7400640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67b80d1100 0x7f67b815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b6a00640 1 -- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67b80d1100 msgr2=0x7f67b815a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b6a00640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67b80d1100 0x7f67b815a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b6a00640 1 -- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67b80b3b80 msgr2=0x7f67b8162050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b6a00640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67b80b3b80 0x7f67b8162050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b6a00640 1 -- 172.21.15.69:0/3088392202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b80c8520 con 0x7f67b806cc00 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b6000640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67b80b3b80 0x7f67b8162050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.938+0000 7f67b6a00640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b806cc00 0x7f67b815d450 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f67a800c430 tx=0x7f67a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.939+0000 7f67aea00640 1 -- 172.21.15.69:0/3088392202 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a8011790 con 0x7f67b806cc00 2024-04-24T08:10:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.939+0000 7f67aea00640 1 -- 172.21.15.69:0/3088392202 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67a8017070 con 0x7f67b806cc00 2024-04-24T08:10:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.939+0000 7f67bda00640 1 -- 172.21.15.69:0/3088392202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b80c8ae0 con 0x7f67b806cc00 2024-04-24T08:10:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.939+0000 7f67aea00640 1 -- 172.21.15.69:0/3088392202 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67a8011bb0 con 0x7f67b806cc00 2024-04-24T08:10:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.939+0000 7f67bda00640 1 -- 172.21.15.69:0/3088392202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b80c90a0 con 0x7f67b806cc00 2024-04-24T08:10:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.940+0000 7f67bda00640 1 -- 172.21.15.69:0/3088392202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b8069190 con 0x7f67b806cc00 2024-04-24T08:10:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.941+0000 7f67aea00640 1 -- 172.21.15.69:0/3088392202 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f67a801c460 con 0x7f67b806cc00 2024-04-24T08:10:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.942+0000 7f67aea00640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f679406c940 0x7f679406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.942+0000 7f67aea00640 1 -- 172.21.15.69:0/3088392202 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f67a808b690 con 0x7f67b806cc00 2024-04-24T08:10:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.942+0000 7f67b6000640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f679406c940 0x7f679406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.944+0000 7f67b6000640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f679406c940 0x7f679406ee00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f67a0034000 tx=0x7f67a0033040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:27.947+0000 7f67aea00640 1 -- 172.21.15.69:0/3088392202 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67a8059e30 con 0x7f67b806cc00 2024-04-24T08:10:28.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.216+0000 7f67bda00640 1 -- 172.21.15.69:0/3088392202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) -- 0x7f67b80b7010 con 0x7f67b806cc00 2024-04-24T08:10:28.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.862+0000 7f67aea00640 1 -- 172.21.15.69:0/3088392202 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v134) ==== 152+0+0 (secure 0 0 0) 0x7f67a8059730 con 0x7f67b806cc00 2024-04-24T08:10:28.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.865+0000 7f67bda00640 1 -- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f679406c940 msgr2=0x7f679406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.865+0000 7f67bda00640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f679406c940 0x7f679406ee00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f67a0034000 tx=0x7f67a0033040 comp rx=0 tx=0).stop 2024-04-24T08:10:28.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.865+0000 7f67bda00640 1 -- 172.21.15.69:0/3088392202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b806cc00 msgr2=0x7f67b815d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.865+0000 7f67bda00640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b806cc00 0x7f67b815d450 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f67a800c430 tx=0x7f67a800c8f0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.866+0000 7f67bda00640 1 -- 172.21.15.69:0/3088392202 shutdown_connections 2024-04-24T08:10:28.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.866+0000 7f67bda00640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f679406c940 0x7f679406ee00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.866+0000 7f67bda00640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67b80d1100 0x7f67b815a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.866+0000 7f67bda00640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67b80b3b80 0x7f67b8162050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.866+0000 7f67bda00640 1 --2- 172.21.15.69:0/3088392202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67b806cc00 0x7f67b815d450 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.866+0000 7f67bda00640 1 -- 172.21.15.69:0/3088392202 >> 172.21.15.69:0/3088392202 conn(0x7f67b808bf60 msgr2=0x7f67b8074b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:28.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.866+0000 7f67bda00640 1 -- 172.21.15.69:0/3088392202 shutdown_connections 2024-04-24T08:10:28.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:28.866+0000 7f67bda00640 1 -- 172.21.15.69:0/3088392202 wait complete. 2024-04-24T08:10:28.889 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-24T08:10:29.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.094+0000 7f627e600640 1 Processor -- start 2024-04-24T08:10:29.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.094+0000 7f627e600640 1 -- start start 2024-04-24T08:10:29.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.096+0000 7f627e600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62780d3350 0x7f627815faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.096+0000 7f627e600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f62780d4e00 0x7f62781615a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.096+0000 7f627e600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f627806cc30 con 0x7f627815e870 2024-04-24T08:10:29.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.096+0000 7f627e600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f627806e9c0 con 0x7f62780d4e00 2024-04-24T08:10:29.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.096+0000 7f627e600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f627806ced0 con 0x7f62780d3350 2024-04-24T08:10:29.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.096+0000 7f6277400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f627815e870 0x7f62780d2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37184/0 (socket says 172.21.15.69:37184) 2024-04-24T08:10:29.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.096+0000 7f6277400640 1 -- 172.21.15.69:0/1805251467 learned_addr learned my addr 172.21.15.69:0/1805251467 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f626fe00640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62780d3350 0x7f627815faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6277e00640 1 --2- 172.21.15.69:0/1805251467 >> v2:172.21.15.121:3300/0 conn(0x7f62780d4e00 0x7f62781615a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2102849965 0 0) 0x7f627806cc30 con 0x7f627815e870 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f625c003520 con 0x7f627815e870 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1301164266 0 0) 0x7f625c003520 con 0x7f627815e870 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62780d3350 msgr2=0x7f627815faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62780d3350 0x7f627815faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 >> v2:172.21.15.121:3300/0 conn(0x7f62780d4e00 msgr2=0x7f62781615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 --2- 172.21.15.69:0/1805251467 >> v2:172.21.15.121:3300/0 conn(0x7f62780d4e00 0x7f62781615a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627806d1b0 con 0x7f627815e870 2024-04-24T08:10:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f6268001670 con 0x7f627815e870 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 >> v1:172.21.15.69:6789/0 conn(0x7f627815e870 legacy=0x7f62780d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625c0039b0 0x7f625c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62780d3350 0x7f625c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f625c004d40 0x7f625c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f627806ced0 con 0x7f625c0039b0 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f625c00b9c0 con 0x7f625c004d40 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.097+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f625c00bb70 con 0x7f62780d3350 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.098+0000 7f6277400640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625c0039b0 0x7f625c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.098+0000 7f626fe00640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62780d3350 0x7f625c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.098+0000 7f6277e00640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f625c004d40 0x7f625c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.098+0000 7f6277400640 1 -- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62780d3350 msgr2=0x7f625c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.098+0000 7f6277400640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62780d3350 0x7f625c004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.098+0000 7f6277400640 1 -- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f625c004d40 msgr2=0x7f625c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.098+0000 7f6277400640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f625c004d40 0x7f625c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.098+0000 7f6277400640 1 -- 172.21.15.69:0/1805251467 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f625c003520 con 0x7f625c0039b0 2024-04-24T08:10:29.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.099+0000 7f6277400640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625c0039b0 0x7f625c003dd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6268001050 tx=0x7f625c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f77c4e662634d3f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.099+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268007ca0 con 0x7f625c0039b0 2024-04-24T08:10:29.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.099+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6268005ad0 con 0x7f625c0039b0 2024-04-24T08:10:29.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.099+0000 7f6276a00640 1 -- 172.21.15.69:0/1805251467 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268007070 con 0x7f625c0039b0 2024-04-24T08:10:29.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.100+0000 7f627e600640 1 -- 172.21.15.69:0/1805251467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625c0039b0 msgr2=0x7f625c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.100+0000 7f627e600640 1 --2- 172.21.15.69:0/1805251467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625c0039b0 0x7f625c003dd0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f6268001050 tx=0x7f625c005830 comp rx=0 tx=0).stop 2024-04-24T08:10:29.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.100+0000 7f6277e00640 1 -- 172.21.15.69:0/1805251467 reap_dead start 2024-04-24T08:10:29.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.100+0000 7f627e600640 1 -- 172.21.15.69:0/1805251467 shutdown_connections 2024-04-24T08:10:29.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.100+0000 7f627e600640 1 -- 172.21.15.69:0/1805251467 >> 172.21.15.69:0/1805251467 conn(0x7f627808bf60 msgr2=0x7f62780d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.100+0000 7f627e600640 1 -- 172.21.15.69:0/1805251467 shutdown_connections 2024-04-24T08:10:29.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.100+0000 7f627e600640 1 -- 172.21.15.69:0/1805251467 wait complete. 2024-04-24T08:10:29.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.102+0000 7f627e600640 1 Processor -- start 2024-04-24T08:10:29.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.102+0000 7f627e600640 1 -- start start 2024-04-24T08:10:29.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f627e600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f625c0039b0 0x7f62780ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f627e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62780a0c10 0x7f627809e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f627e600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f627815a680 0x7f627815ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f627e600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62780d55a0 con 0x7f62780a0c10 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f627e600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f62780d5290 con 0x7f627815a680 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f627e600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62781602d0 con 0x7f625c0039b0 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f626fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62780a0c10 0x7f627809e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f626fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62780a0c10 0x7f627809e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59224/0 (socket says 172.21.15.69:59224) 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f626fe00640 1 -- 172.21.15.69:0/2263303575 learned_addr learned my addr 172.21.15.69:0/2263303575 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f6277400640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f625c0039b0 0x7f62780ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f626fe00640 1 -- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f625c0039b0 msgr2=0x7f62780ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f626fe00640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f625c0039b0 0x7f62780ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f626fe00640 1 -- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f627815a680 msgr2=0x7f627815ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.103+0000 7f6277e00640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f627815a680 0x7f627815ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.104+0000 7f626fe00640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f627815a680 0x7f627815ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.104+0000 7f626fe00640 1 -- 172.21.15.69:0/2263303575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62780a1180 con 0x7f62780a0c10 2024-04-24T08:10:29.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.104+0000 7f626fe00640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62780a0c10 0x7f627809e5e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6260010890 tx=0x7f6260010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.104+0000 7f626f400640 1 -- 172.21.15.69:0/2263303575 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626000f040 con 0x7f62780a0c10 2024-04-24T08:10:29.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.104+0000 7f626f400640 1 -- 172.21.15.69:0/2263303575 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f626000eb60 con 0x7f62780a0c10 2024-04-24T08:10:29.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.104+0000 7f627e600640 1 -- 172.21.15.69:0/2263303575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62780a1460 con 0x7f62780a0c10 2024-04-24T08:10:29.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.104+0000 7f626f400640 1 -- 172.21.15.69:0/2263303575 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626001c400 con 0x7f62780a0c10 2024-04-24T08:10:29.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.104+0000 7f627e600640 1 -- 172.21.15.69:0/2263303575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62780a1a20 con 0x7f62780a0c10 2024-04-24T08:10:29.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.106+0000 7f627e600640 1 -- 172.21.15.69:0/2263303575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f627806c660 con 0x7f62780a0c10 2024-04-24T08:10:29.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.106+0000 7f626f400640 1 -- 172.21.15.69:0/2263303575 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f626001c5a0 con 0x7f62780a0c10 2024-04-24T08:10:29.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.107+0000 7f626f400640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f625406c920 0x7f625406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.107+0000 7f626f400640 1 -- 172.21.15.69:0/2263303575 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f626008bf80 con 0x7f62780a0c10 2024-04-24T08:10:29.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.108+0000 7f6277400640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f625406c920 0x7f625406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.109+0000 7f6277400640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f625406c920 0x7f625406ede0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6268032000 tx=0x7f6268007000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.111+0000 7f626f400640 1 -- 172.21.15.69:0/2263303575 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f627806c660 con 0x7f62780a0c10 2024-04-24T08:10:29.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.309+0000 7f627e600640 1 -- 172.21.15.69:0/2263303575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f62780a1dc0 con 0x7f62780a0c10 2024-04-24T08:10:29.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.314+0000 7f626f400640 1 -- 172.21.15.69:0/2263303575 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f6260018030 con 0x7f62780a0c10 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 -- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f625406c920 msgr2=0x7f625406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f625406c920 0x7f625406ede0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6268032000 tx=0x7f6268007000 comp rx=0 tx=0).stop 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 -- 172.21.15.69:0/2263303575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62780a0c10 msgr2=0x7f627809e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62780a0c10 0x7f627809e5e0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f6260010890 tx=0x7f6260010d50 comp rx=0 tx=0).stop 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 -- 172.21.15.69:0/2263303575 shutdown_connections 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f625406c920 0x7f625406ede0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62780a0c10 0x7f627809e5e0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f627815a680 0x7f627815ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 --2- 172.21.15.69:0/2263303575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f625c0039b0 0x7f62780ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 -- 172.21.15.69:0/2263303575 >> 172.21.15.69:0/2263303575 conn(0x7f627808bf60 msgr2=0x7f62780b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 -- 172.21.15.69:0/2263303575 shutdown_connections 2024-04-24T08:10:29.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.317+0000 7f627e600640 1 -- 172.21.15.69:0/2263303575 wait complete. 2024-04-24T08:10:29.336 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:10:29.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:10:29.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:10:29.545 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:10:29.546 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:10:29.546 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:10:29.604 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:10:29.604 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:10:29.604 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:10:29.661 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:10:29.661 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:10:29.661 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:10:29.714 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:10:29.714 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:10:29.715 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:10:29.770 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:10:29.770 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-24T08:10:29.770 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:29.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.976+0000 7f8441200640 1 Processor -- start 2024-04-24T08:10:29.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.977+0000 7f8441200640 1 -- start start 2024-04-24T08:10:29.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.978+0000 7f8441200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f843c0cfe40 0x7f843c0b7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.978+0000 7f8441200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f843c0d22d0 con 0x7f843c162150 2024-04-24T08:10:29.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.978+0000 7f8441200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f843c0d03b0 con 0x7f843c0b7a90 2024-04-24T08:10:29.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.978+0000 7f8441200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f843c0d2900 con 0x7f843c0cfe40 2024-04-24T08:10:29.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.978+0000 7f8439e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f843c162150 0x7f843c0d5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37200/0 (socket says 172.21.15.69:37200) 2024-04-24T08:10:29.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.978+0000 7f8439e00640 1 -- 172.21.15.69:0/550782268 learned_addr learned my addr 172.21.15.69:0/550782268 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:29.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3308245750 0 0) 0x7f843c0d22d0 con 0x7f843c162150 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f841c003520 con 0x7f843c162150 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8439400640 1 --2- 172.21.15.69:0/550782268 >> v2:172.21.15.121:3300/0 conn(0x7f843c0cfe40 0x7f843c0b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4250102482 0 0) 0x7f841c003520 con 0x7f843c162150 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 >> v2:172.21.15.121:3300/0 conn(0x7f843c0cfe40 msgr2=0x7f843c0b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 --2- 172.21.15.69:0/550782268 >> v2:172.21.15.121:3300/0 conn(0x7f843c0cfe40 0x7f843c0b7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 >> v1:172.21.15.121:6789/0 conn(0x7f843c0b7a90 legacy=0x7f843c0d10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843c0d0690 con 0x7f843c162150 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f842c0017f0 con 0x7f843c162150 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 >> v1:172.21.15.69:6789/0 conn(0x7f843c162150 legacy=0x7f843c0d5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 --2- 172.21.15.69:0/550782268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f841c003a00 0x7f841c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 --2- 172.21.15.69:0/550782268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c004800 0x7f841c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 --2- 172.21.15.69:0/550782268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f841c005210 0x7f841c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f843c0d2900 con 0x7f841c003a00 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f841c00c140 con 0x7f841c004800 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.979+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f841c00c340 con 0x7f841c005210 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.980+0000 7f8439e00640 1 --2- 172.21.15.69:0/550782268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f841c003a00 0x7f841c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.980+0000 7f8439e00640 1 -- 172.21.15.69:0/550782268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f841c005210 msgr2=0x7f841c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.980+0000 7f8439e00640 1 --2- 172.21.15.69:0/550782268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f841c005210 0x7f841c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.980+0000 7f8439e00640 1 -- 172.21.15.69:0/550782268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c004800 msgr2=0x7f841c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.980+0000 7f8439e00640 1 --2- 172.21.15.69:0/550782268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c004800 0x7f841c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.980+0000 7f8439e00640 1 -- 172.21.15.69:0/550782268 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841c003520 con 0x7f841c003a00 2024-04-24T08:10:29.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.980+0000 7f843a800640 1 -- 172.21.15.69:0/550782268 reap_dead start 2024-04-24T08:10:29.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.981+0000 7f8439e00640 1 --2- 172.21.15.69:0/550782268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f841c003a00 0x7f841c003e00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f842c00c160 tx=0x7f841c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a8df63d720b553a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.981+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c007c70 con 0x7f841c003a00 2024-04-24T08:10:29.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.981+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f842c007270 con 0x7f841c003a00 2024-04-24T08:10:29.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.981+0000 7f8438a00640 1 -- 172.21.15.69:0/550782268 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842c0057a0 con 0x7f841c003a00 2024-04-24T08:10:29.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.982+0000 7f8441200640 1 -- 172.21.15.69:0/550782268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f841c003a00 msgr2=0x7f841c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.982+0000 7f8441200640 1 --2- 172.21.15.69:0/550782268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f841c003a00 0x7f841c003e00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f842c00c160 tx=0x7f841c005830 comp rx=0 tx=0).stop 2024-04-24T08:10:29.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.982+0000 7f8441200640 1 -- 172.21.15.69:0/550782268 shutdown_connections 2024-04-24T08:10:29.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.982+0000 7f8441200640 1 --2- 172.21.15.69:0/550782268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f841c003a00 0x7f841c003e00 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.982+0000 7f8441200640 1 -- 172.21.15.69:0/550782268 >> 172.21.15.69:0/550782268 conn(0x7f843c08bf60 msgr2=0x7f843c0d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.982+0000 7f8441200640 1 -- 172.21.15.69:0/550782268 shutdown_connections 2024-04-24T08:10:29.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.982+0000 7f8441200640 1 -- 172.21.15.69:0/550782268 wait complete. 2024-04-24T08:10:29.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.984+0000 7f8441200640 1 Processor -- start 2024-04-24T08:10:29.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.984+0000 7f8441200640 1 -- start start 2024-04-24T08:10:29.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f8441200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c003a00 0x7f843c1f8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f8441200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843c0c9ca0 0x7f843c0cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f8441200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843c156790 0x7f843c0cd990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f8441200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f843c0b76d0 con 0x7f843c156790 2024-04-24T08:10:29.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f8441200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f843c0b7550 con 0x7f841c003a00 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f8441200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f843c0b7850 con 0x7f843c0c9ca0 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f843a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843c156790 0x7f843c0cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f843a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843c156790 0x7f843c0cd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59238/0 (socket says 172.21.15.69:59238) 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f843a800640 1 -- 172.21.15.69:0/1203107315 learned_addr learned my addr 172.21.15.69:0/1203107315 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f8439400640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843c0c9ca0 0x7f843c0cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.985+0000 7f8439e00640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c003a00 0x7f843c1f8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f843a800640 1 -- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843c0c9ca0 msgr2=0x7f843c0cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f843a800640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843c0c9ca0 0x7f843c0cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f843a800640 1 -- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c003a00 msgr2=0x7f843c1f8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f843a800640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c003a00 0x7f843c1f8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f843a800640 1 -- 172.21.15.69:0/1203107315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843c1f9280 con 0x7f843c156790 2024-04-24T08:10:29.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f843a800640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843c156790 0x7f843c0cd990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f843c0cee00 tx=0x7f843400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f8432000640 1 -- 172.21.15.69:0/1203107315 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84340044a0 con 0x7f843c156790 2024-04-24T08:10:29.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f8432000640 1 -- 172.21.15.69:0/1203107315 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8434007480 con 0x7f843c156790 2024-04-24T08:10:29.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f8441200640 1 -- 172.21.15.69:0/1203107315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f843c1f4f50 con 0x7f843c156790 2024-04-24T08:10:29.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f8432000640 1 -- 172.21.15.69:0/1203107315 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8434005a00 con 0x7f843c156790 2024-04-24T08:10:29.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.986+0000 7f8441200640 1 -- 172.21.15.69:0/1203107315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f843c1f5510 con 0x7f843c156790 2024-04-24T08:10:29.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.987+0000 7f8441200640 1 -- 172.21.15.69:0/1203107315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f843c068a80 con 0x7f843c156790 2024-04-24T08:10:29.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.988+0000 7f8432000640 1 -- 172.21.15.69:0/1203107315 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8434011070 con 0x7f843c156790 2024-04-24T08:10:29.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.989+0000 7f8432000640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 0x7f841806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.989+0000 7f8432000640 1 -- 172.21.15.69:0/1203107315 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f843408aab0 con 0x7f843c156790 2024-04-24T08:10:29.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.990+0000 7f8439e00640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 0x7f841806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.991+0000 7f8439e00640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 0x7f841806edb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f842c032000 tx=0x7f842c02d000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:29.994+0000 7f8432000640 1 -- 172.21.15.69:0/1203107315 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8434019060 con 0x7f843c156790 2024-04-24T08:10:30.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.232+0000 7f8441200640 1 -- 172.21.15.69:0/1203107315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f843c0cb040 con 0x7f843c156790 2024-04-24T08:10:30.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.233+0000 7f8432000640 1 -- 172.21.15.69:0/1203107315 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+3872 (secure 0 0 0) 0x7f843c1f9280 con 0x7f843c156790 2024-04-24T08:10:30.233 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:30.233 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4622,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2295273533","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2295273533},{"type":"v1","addr":"172.21.15.121:6836","nonce":2295273533}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4625,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1461471292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1461471292},{"type":"v1","addr":"172.21.15.121:6837","nonce":1461471292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":5238,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/626688145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":626688145},{"type":"v1","addr":"172.21.15.69:6833","nonce":626688145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5691,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/1300056009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":1300056009},{"type":"v1","addr":"172.21.15.69:6835","nonce":1300056009}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[]} 2024-04-24T08:10:30.234 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 18 2024-04-24T08:10:30.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.235+0000 7f8441200640 1 -- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 msgr2=0x7f841806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.235+0000 7f8441200640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 0x7f841806edb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f842c032000 tx=0x7f842c02d000 comp rx=0 tx=0).stop 2024-04-24T08:10:30.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.235+0000 7f8441200640 1 -- 172.21.15.69:0/1203107315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843c156790 msgr2=0x7f843c0cd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.235+0000 7f8441200640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843c156790 0x7f843c0cd990 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f843c0cee00 tx=0x7f843400c820 comp rx=0 tx=0).stop 2024-04-24T08:10:30.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.236+0000 7f8441200640 1 -- 172.21.15.69:0/1203107315 shutdown_connections 2024-04-24T08:10:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.236+0000 7f8441200640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 0x7f841806edb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.236+0000 7f8441200640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843c156790 0x7f843c0cd990 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.236+0000 7f8441200640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843c0c9ca0 0x7f843c0cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.236+0000 7f8441200640 1 --2- 172.21.15.69:0/1203107315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f841c003a00 0x7f843c1f8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.236+0000 7f8441200640 1 -- 172.21.15.69:0/1203107315 >> 172.21.15.69:0/1203107315 conn(0x7f843c08bf60 msgr2=0x7f843c0b6040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.236+0000 7f8441200640 1 -- 172.21.15.69:0/1203107315 shutdown_connections 2024-04-24T08:10:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.236+0000 7f8441200640 1 -- 172.21.15.69:0/1203107315 wait complete. 2024-04-24T08:10:30.254 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T08:10:30.254 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-24T08:10:30.254 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T08:10:30.254 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:10:30.255 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T08:10:30.255 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-24T08:10:30.255 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T08:10:30.255 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:10:30.255 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T08:10:30.255 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-24T08:10:30.255 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T08:10:30.256 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:10:30.256 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T08:10:30.256 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-24T08:10:30.256 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T08:10:30.256 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:10:30.274 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T08:10:30.274 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T08:10:30.275 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T08:10:30.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T08:10:30.276 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T08:10:30.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T08:10:30.277 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T08:10:30.278 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T08:10:30.320 INFO:tasks.ceph.mds.c:Started 2024-04-24T08:10:30.320 DEBUG:teuthology.parallel:result is None 2024-04-24T08:10:30.322 INFO:tasks.ceph.mds.b:Started 2024-04-24T08:10:30.323 DEBUG:teuthology.parallel:result is None 2024-04-24T08:10:30.324 INFO:tasks.ceph.mds.a:Started 2024-04-24T08:10:30.324 DEBUG:teuthology.parallel:result is None 2024-04-24T08:10:30.326 INFO:tasks.ceph.mds.d:Started 2024-04-24T08:10:30.326 DEBUG:teuthology.parallel:result is None 2024-04-24T08:10:30.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-24T08:10:30.466 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:10:30.470 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:10:30.477 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:10:30.477 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-24T08:10:30.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.503+0000 7f9a39600640 1 Processor -- start 2024-04-24T08:10:30.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.503+0000 7f9a39600640 1 -- start start 2024-04-24T08:10:30.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.504+0000 7f9a39600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a3415ca20 0x7f9a340b1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.504+0000 7f9a39600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a340ce150 con 0x7f9a3409bc60 2024-04-24T08:10:30.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.504+0000 7f9a39600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a340ce430 con 0x7f9a340b23f0 2024-04-24T08:10:30.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.504+0000 7f9a39600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a3415ce00 con 0x7f9a3415ca20 2024-04-24T08:10:30.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.505+0000 7f9a33400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9a3409bc60 0x7f9a340d0b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37222/0 (socket says 172.21.15.69:37222) 2024-04-24T08:10:30.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.505+0000 7f9a33400640 1 -- 172.21.15.69:0/1693795476 learned_addr learned my addr 172.21.15.69:0/1693795476 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:30.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.505+0000 7f9a32a00640 1 --2- 172.21.15.69:0/1693795476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a3415ca20 0x7f9a340b1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.505+0000 7f9a31600640 1 -- 172.21.15.69:0/1693795476 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1991517761 0 0) 0x7f9a340ce150 con 0x7f9a3409bc60 2024-04-24T08:10:30.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.505+0000 7f9a31600640 1 -- 172.21.15.69:0/1693795476 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a18003520 con 0x7f9a3409bc60 2024-04-24T08:10:30.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.505+0000 7f9a31600640 1 -- 172.21.15.69:0/1693795476 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1546298840 0 0) 0x7f9a340ce430 con 0x7f9a340b23f0 2024-04-24T08:10:30.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.505+0000 7f9a31600640 1 -- 172.21.15.69:0/1693795476 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a340ce150 con 0x7f9a340b23f0 2024-04-24T08:10:30.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.505+0000 7f9a32a00640 1 -- 172.21.15.69:0/1693795476 >> v1:172.21.15.121:6789/0 conn(0x7f9a340b23f0 legacy=0x7f9a3409b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.505+0000 7f9a32a00640 1 -- 172.21.15.69:0/1693795476 >> v1:172.21.15.69:6789/0 conn(0x7f9a3409bc60 legacy=0x7f9a340d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.505+0000 7f9a32a00640 1 -- 172.21.15.69:0/1693795476 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a3409aa70 con 0x7f9a3415ca20 2024-04-24T08:10:30.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.506+0000 7f9a32a00640 1 --2- 172.21.15.69:0/1693795476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a3415ca20 0x7f9a340b1eb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9a240097d0 tx=0x7f9a2402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e35980ce04cc60e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.507+0000 7f9a31600640 1 -- 172.21.15.69:0/1693795476 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24037070 con 0x7f9a3415ca20 2024-04-24T08:10:30.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.507+0000 7f9a31600640 1 -- 172.21.15.69:0/1693795476 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a24032440 con 0x7f9a3415ca20 2024-04-24T08:10:30.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.507+0000 7f9a31600640 1 -- 172.21.15.69:0/1693795476 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24032810 con 0x7f9a3415ca20 2024-04-24T08:10:30.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.507+0000 7f9a39600640 1 -- 172.21.15.69:0/1693795476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a3415ca20 msgr2=0x7f9a340b1eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.507+0000 7f9a39600640 1 --2- 172.21.15.69:0/1693795476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a3415ca20 0x7f9a340b1eb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9a240097d0 tx=0x7f9a2402a210 comp rx=0 tx=0).stop 2024-04-24T08:10:30.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.507+0000 7f9a39600640 1 -- 172.21.15.69:0/1693795476 shutdown_connections 2024-04-24T08:10:30.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.507+0000 7f9a39600640 1 --2- 172.21.15.69:0/1693795476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a3415ca20 0x7f9a340b1eb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.507+0000 7f9a39600640 1 -- 172.21.15.69:0/1693795476 >> 172.21.15.69:0/1693795476 conn(0x7f9a340b4040 msgr2=0x7f9a340b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.507+0000 7f9a39600640 1 -- 172.21.15.69:0/1693795476 shutdown_connections 2024-04-24T08:10:30.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.507+0000 7f9a39600640 1 -- 172.21.15.69:0/1693795476 wait complete. 2024-04-24T08:10:30.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.509+0000 7f9a39600640 1 Processor -- start 2024-04-24T08:10:30.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.509+0000 7f9a39600640 1 -- start start 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.510+0000 7f9a39600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a340b23f0 0x7f9a340cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.510+0000 7f9a39600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a3415ca20 0x7f9a34157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.510+0000 7f9a39600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a340b6470 0x7f9a340704a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.510+0000 7f9a39600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9a340d0510 con 0x7f9a340b23f0 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.510+0000 7f9a39600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9a3409b030 con 0x7f9a3415ca20 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.510+0000 7f9a39600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a340d0690 con 0x7f9a340b6470 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.510+0000 7f9a32a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a340b23f0 0x7f9a340cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.510+0000 7f9a33400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a340b6470 0x7f9a340704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.510+0000 7f9a32000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a3415ca20 0x7f9a34157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.511+0000 7f9a32a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a340b23f0 0x7f9a340cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59274/0 (socket says 172.21.15.69:59274) 2024-04-24T08:10:30.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.511+0000 7f9a32a00640 1 -- 172.21.15.69:0/2380815712 learned_addr learned my addr 172.21.15.69:0/2380815712 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:30.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.511+0000 7f9a32a00640 1 -- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a340b6470 msgr2=0x7f9a340704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.511+0000 7f9a32a00640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a340b6470 0x7f9a340704a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.511+0000 7f9a32a00640 1 -- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a3415ca20 msgr2=0x7f9a34157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.511+0000 7f9a32a00640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a3415ca20 0x7f9a34157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.511+0000 7f9a32a00640 1 -- 172.21.15.69:0/2380815712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a340b6c40 con 0x7f9a340b23f0 2024-04-24T08:10:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.511+0000 7f9a32a00640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a340b23f0 0x7f9a340cc2c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9a24035040 tx=0x7f9a240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.512+0000 7f9a2aa00640 1 -- 172.21.15.69:0/2380815712 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24042020 con 0x7f9a340b23f0 2024-04-24T08:10:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.512+0000 7f9a2aa00640 1 -- 172.21.15.69:0/2380815712 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a2403d750 con 0x7f9a340b23f0 2024-04-24T08:10:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.512+0000 7f9a2aa00640 1 -- 172.21.15.69:0/2380815712 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a24037040 con 0x7f9a340b23f0 2024-04-24T08:10:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.512+0000 7f9a39600640 1 -- 172.21.15.69:0/2380815712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a340b6f20 con 0x7f9a340b23f0 2024-04-24T08:10:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.512+0000 7f9a39600640 1 -- 172.21.15.69:0/2380815712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a340b74e0 con 0x7f9a340b23f0 2024-04-24T08:10:30.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.513+0000 7f9a39600640 1 -- 172.21.15.69:0/2380815712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a34068a80 con 0x7f9a340b23f0 2024-04-24T08:10:30.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.514+0000 7f9a2aa00640 1 -- 172.21.15.69:0/2380815712 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a24045020 con 0x7f9a340b23f0 2024-04-24T08:10:30.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.515+0000 7f9a2aa00640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1406c920 0x7f9a1406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.515+0000 7f9a2aa00640 1 -- 172.21.15.69:0/2380815712 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 4943+0+0 (secure 0 0 0) 0x7f9a240abfe0 con 0x7f9a340b23f0 2024-04-24T08:10:30.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.516+0000 7f9a32000640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1406c920 0x7f9a1406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.516+0000 7f9a32000640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1406c920 0x7f9a1406ede0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9a20007fd0 tx=0x7f9a20009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.520+0000 7f9a2aa00640 1 -- 172.21.15.69:0/2380815712 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a2404c070 con 0x7f9a340b23f0 2024-04-24T08:10:30.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.726+0000 7f9a39600640 1 -- 172.21.15.69:0/2380815712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f9a340b4cd0 con 0x7f9a340b23f0 2024-04-24T08:10:30.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.889+0000 7f9a2aa00640 1 -- 172.21.15.69:0/2380815712 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v38) ==== 97+0+0 (secure 0 0 0) 0x7f9a2407a050 con 0x7f9a340b23f0 2024-04-24T08:10:30.890 INFO:teuthology.orchestra.run.smithi069.stderr: removed all blocklist entries 2024-04-24T08:10:30.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 -- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1406c920 msgr2=0x7f9a1406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1406c920 0x7f9a1406ede0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9a20007fd0 tx=0x7f9a20009040 comp rx=0 tx=0).stop 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 -- 172.21.15.69:0/2380815712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a340b23f0 msgr2=0x7f9a340cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a340b23f0 0x7f9a340cc2c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9a24035040 tx=0x7f9a240046c0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 -- 172.21.15.69:0/2380815712 shutdown_connections 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a1406c920 0x7f9a1406ede0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a340b6470 0x7f9a340704a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a3415ca20 0x7f9a34157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 --2- 172.21.15.69:0/2380815712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a340b23f0 0x7f9a340cc2c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 -- 172.21.15.69:0/2380815712 >> 172.21.15.69:0/2380815712 conn(0x7f9a340b4040 msgr2=0x7f9a340ce5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 -- 172.21.15.69:0/2380815712 shutdown_connections 2024-04-24T08:10:30.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:30.893+0000 7f9a39600640 1 -- 172.21.15.69:0/2380815712 wait complete. 2024-04-24T08:10:30.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-24T08:10:31.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.104+0000 7fb912200640 1 Processor -- start 2024-04-24T08:10:31.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.104+0000 7fb912200640 1 -- start start 2024-04-24T08:10:31.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.106+0000 7fb912200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb90c1523f0 0x7fb90c06db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.106+0000 7fb912200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb90c0aeac0 con 0x7fb90c072e20 2024-04-24T08:10:31.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.106+0000 7fb912200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb90c0aeda0 con 0x7fb90c15b980 2024-04-24T08:10:31.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.106+0000 7fb912200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb90c0af080 con 0x7fb90c1523f0 2024-04-24T08:10:31.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.106+0000 7fb903e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb90c072e20 0x7fb90c073260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37232/0 (socket says 172.21.15.69:37232) 2024-04-24T08:10:31.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.106+0000 7fb903e00640 1 -- 172.21.15.69:0/1096210654 learned_addr learned my addr 172.21.15.69:0/1096210654 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb910e00640 1 --2- 172.21.15.69:0/1096210654 >> v2:172.21.15.121:3300/0 conn(0x7fb90c1523f0 0x7fb90c06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3602872462 0 0) 0x7fb90c0aeac0 con 0x7fb90c072e20 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8f0003520 con 0x7fb90c072e20 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1618254360 0 0) 0x7fb8f0003520 con 0x7fb90c072e20 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 >> v2:172.21.15.121:3300/0 conn(0x7fb90c1523f0 msgr2=0x7fb90c06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 --2- 172.21.15.69:0/1096210654 >> v2:172.21.15.121:3300/0 conn(0x7fb90c1523f0 0x7fb90c06db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 >> v1:172.21.15.121:6789/0 conn(0x7fb90c15b980 legacy=0x7fb90c15be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90c0af360 con 0x7fb90c072e20 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fb8f4001670 con 0x7fb90c072e20 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 >> v1:172.21.15.69:6789/0 conn(0x7fb90c072e20 legacy=0x7fb90c073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8f0003a00 0x7fb8f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0004800 0x7fb8f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8f0005210 0x7fb8f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb90c0af080 con 0x7fb8f0003a00 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f000c160 con 0x7fb8f0004800 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.107+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb8f000c360 con 0x7fb8f0005210 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.108+0000 7fb910e00640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8f0003a00 0x7fb8f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.108+0000 7fb903e00640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0004800 0x7fb8f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.108+0000 7fb910e00640 1 -- 172.21.15.69:0/1096210654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8f0005210 msgr2=0x7fb8f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.108+0000 7fb910e00640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8f0005210 0x7fb8f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.108+0000 7fb910e00640 1 -- 172.21.15.69:0/1096210654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0004800 msgr2=0x7fb8f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.108+0000 7fb910e00640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0004800 0x7fb8f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.108+0000 7fb910e00640 1 -- 172.21.15.69:0/1096210654 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8f0003520 con 0x7fb8f0003a00 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.108+0000 7fb911800640 1 -- 172.21.15.69:0/1096210654 reap_dead start 2024-04-24T08:10:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.108+0000 7fb903e00640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8f0004800 0x7fb8f0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:31.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.109+0000 7fb910e00640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8f0003a00 0x7fb8f0003e20 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009a90 tx=0x7fb8f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28999bde5751d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.109+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc036070 con 0x7fb8f0003a00 2024-04-24T08:10:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.109+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8fc031400 con 0x7fb8f0003a00 2024-04-24T08:10:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.109+0000 7fb90be00640 1 -- 172.21.15.69:0/1096210654 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc009690 con 0x7fb8f0003a00 2024-04-24T08:10:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.110+0000 7fb912200640 1 -- 172.21.15.69:0/1096210654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8f0003a00 msgr2=0x7fb8f0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.110+0000 7fb912200640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8f0003a00 0x7fb8f0003e20 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc009a90 tx=0x7fb8f0005830 comp rx=0 tx=0).stop 2024-04-24T08:10:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.110+0000 7fb912200640 1 -- 172.21.15.69:0/1096210654 shutdown_connections 2024-04-24T08:10:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.110+0000 7fb912200640 1 --2- 172.21.15.69:0/1096210654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8f0003a00 0x7fb8f0003e20 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.110+0000 7fb912200640 1 -- 172.21.15.69:0/1096210654 >> 172.21.15.69:0/1096210654 conn(0x7fb90c0d0160 msgr2=0x7fb90c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.110+0000 7fb912200640 1 -- 172.21.15.69:0/1096210654 shutdown_connections 2024-04-24T08:10:31.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.110+0000 7fb912200640 1 -- 172.21.15.69:0/1096210654 wait complete. 2024-04-24T08:10:31.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.112+0000 7fb912200640 1 Processor -- start 2024-04-24T08:10:31.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.112+0000 7fb912200640 1 -- start start 2024-04-24T08:10:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.113+0000 7fb912200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8f0003a00 0x7fb90c14d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.113+0000 7fb912200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90c14d400 0x7fb90c06e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.113+0000 7fb912200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb90c149290 0x7fb90c149710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.113+0000 7fb912200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb90c06db90 con 0x7fb90c149290 2024-04-24T08:10:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.113+0000 7fb912200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb90c1527d0 con 0x7fb90c14d400 2024-04-24T08:10:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.113+0000 7fb912200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb90c06dd10 con 0x7fb8f0003a00 2024-04-24T08:10:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.113+0000 7fb911800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb90c149290 0x7fb90c149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.113+0000 7fb911800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb90c149290 0x7fb90c149710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59292/0 (socket says 172.21.15.69:59292) 2024-04-24T08:10:31.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.113+0000 7fb911800640 1 -- 172.21.15.69:0/3061656526 learned_addr learned my addr 172.21.15.69:0/3061656526 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb910e00640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8f0003a00 0x7fb90c14d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb903e00640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90c14d400 0x7fb90c06e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb911800640 1 -- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8f0003a00 msgr2=0x7fb90c14d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb911800640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8f0003a00 0x7fb90c14d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb911800640 1 -- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90c14d400 msgr2=0x7fb90c06e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb911800640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90c14d400 0x7fb90c06e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb911800640 1 -- 172.21.15.69:0/3061656526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb90c149c50 con 0x7fb90c149290 2024-04-24T08:10:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb903e00640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90c14d400 0x7fb90c06e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:31.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb911800640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb90c149290 0x7fb90c149710 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb90c15bd60 tx=0x7fb90400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:31.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb909600640 1 -- 172.21.15.69:0/3061656526 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904017070 con 0x7fb90c149290 2024-04-24T08:10:31.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb909600640 1 -- 172.21.15.69:0/3061656526 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb904005d40 con 0x7fb90c149290 2024-04-24T08:10:31.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb909600640 1 -- 172.21.15.69:0/3061656526 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90400a850 con 0x7fb90c149290 2024-04-24T08:10:31.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.114+0000 7fb912200640 1 -- 172.21.15.69:0/3061656526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb90c14f410 con 0x7fb90c149290 2024-04-24T08:10:31.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.115+0000 7fb912200640 1 -- 172.21.15.69:0/3061656526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb90c14f9d0 con 0x7fb90c149290 2024-04-24T08:10:31.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.116+0000 7fb912200640 1 -- 172.21.15.69:0/3061656526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb90c075d50 con 0x7fb90c149290 2024-04-24T08:10:31.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.117+0000 7fb909600640 1 -- 172.21.15.69:0/3061656526 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb90400a9f0 con 0x7fb90c149290 2024-04-24T08:10:31.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.117+0000 7fb909600640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8e806c920 0x7fb8e806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.117+0000 7fb909600640 1 -- 172.21.15.69:0/3061656526 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7fb90408a880 con 0x7fb90c149290 2024-04-24T08:10:31.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.118+0000 7fb910e00640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8e806c920 0x7fb8e806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.119+0000 7fb910e00640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8e806c920 0x7fb8e806ede0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc033000 tx=0x7fb8fc008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:31.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.122+0000 7fb909600640 1 -- 172.21.15.69:0/3061656526 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb904058ff0 con 0x7fb90c149290 2024-04-24T08:10:31.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.379+0000 7fb912200640 1 -- 172.21.15.69:0/3061656526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fb90c0b4270 con 0x7fb90c149290 2024-04-24T08:10:31.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.380+0000 7fb909600640 1 -- 172.21.15.69:0/3061656526 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) ==== 80+0+5304 (secure 0 0 0) 0x7fb9040588f0 con 0x7fb90c149290 2024-04-24T08:10:31.380 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:31.380 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_dump": [ 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.a", 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmsqMeLRAApLPtHjtq78HzlVcsjfqXOg==", 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.b", 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmkyd5ChAAGsJGU6GGMvvGzQnABuDx/g==", 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.381 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:10:31.382 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:10:31.382 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:10:31.382 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.382 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.382 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.c", 2024-04-24T08:10:31.382 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmS+yLORAAHPCeBtq99IXEVTpcMT3c4A==", 2024-04-24T08:10:31.382 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.382 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.d", 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShm0xmHFRAA3VfYoqbcV8kWF+4fCZMVLA==", 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:10:31.383 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.0", 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC5vShmMbvcERAANgogiCC/sXf1y0Ykv7ecNQ==", 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.1", 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC8vShmDD+uLBAAJHXmxEm1cumJ3wjqe6S0ww==", 2024-04-24T08:10:31.384 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.2", 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC/vShmS63kNhAAXec5HVuwo8THKstXiQOL8Q==", 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.385 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.3", 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDDvShmxcDJBhAA3V3I4C63EHPRUyhzBCZEzw==", 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.4", 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDJvShm1pX9ERAAL2gjv/+M915Cf5OoqBhZVw==", 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.386 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.5", 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDMvShmNIxbHRAA23gxiaGvagddVc2mE5DJJw==", 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.387 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.6", 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDPvShmb+x4NhAA6aWmAzU6c0AZomVgWcFwig==", 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.7", 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDTvShmgcKMBRAA6fkJvNxBjlNhK9CrZ/FMSw==", 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:10:31.388 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.0", 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw==", 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow r", 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.1", 2024-04-24T08:10:31.389 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmKAIHMhAAuTuFmTHvBteErsG23LMEyQ==", 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.2", 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmW0MnKRAASzxCAKuApVvfu9kUfK9BJw==", 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:10:31.390 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.3", 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC2vShmfgoaABAAniS9oubv92R7Qe2EITGKxg==", 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:10:31.391 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.admin", 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQCzvShmLh6yLxAAoH7NhoEAUKcdXp6n+ai2Ug==", 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.x", 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmv4rwIxAA3rp+yy09qrs7HXip8bZ8xA==", 2024-04-24T08:10:31.392 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.y", 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShm2eatGxAAFQkL0GBsujypv4VnmSy5bw==", 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:10:31.393 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.394 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:31.394 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-24T08:10:31.394 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 -- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8e806c920 msgr2=0x7fb8e806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8e806c920 0x7fb8e806ede0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc033000 tx=0x7fb8fc008030 comp rx=0 tx=0).stop 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 -- 172.21.15.69:0/3061656526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb90c149290 msgr2=0x7fb90c149710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb90c149290 0x7fb90c149710 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb90c15bd60 tx=0x7fb90400cb90 comp rx=0 tx=0).stop 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 -- 172.21.15.69:0/3061656526 shutdown_connections 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8e806c920 0x7fb8e806ede0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb90c149290 0x7fb90c149710 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90c14d400 0x7fb90c06e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 --2- 172.21.15.69:0/3061656526 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8f0003a00 0x7fb90c14d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 -- 172.21.15.69:0/3061656526 >> 172.21.15.69:0/3061656526 conn(0x7fb90c0d0160 msgr2=0x7fb90c0b3b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.382+0000 7fb912200640 1 -- 172.21.15.69:0/3061656526 shutdown_connections 2024-04-24T08:10:31.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.383+0000 7fb912200640 1 -- 172.21.15.69:0/3061656526 wait complete. 2024-04-24T08:10:31.401 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:31.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.586+0000 7fa547200640 1 Processor -- start 2024-04-24T08:10:31.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.587+0000 7fa547200640 1 -- start start 2024-04-24T08:10:31.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.588+0000 7fa547200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5480d0200 0x7fa5480d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.588+0000 7fa547200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54815ed40 con 0x7fa54806cfb0 2024-04-24T08:10:31.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.588+0000 7fa547200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5480cf050 con 0x7fa54815e960 2024-04-24T08:10:31.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.588+0000 7fa547200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa548162b50 con 0x7fa5480d0200 2024-04-24T08:10:31.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.588+0000 7fa53d400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa54806cfb0 0x7fa5480cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37248/0 (socket says 172.21.15.69:37248) 2024-04-24T08:10:31.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.588+0000 7fa53d400640 1 -- 172.21.15.69:0/2062776404 learned_addr learned my addr 172.21.15.69:0/2062776404 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:31.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.588+0000 7fa546800640 1 --2- 172.21.15.69:0/2062776404 >> v2:172.21.15.69:3300/0 conn(0x7fa5480d0200 0x7fa5480d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.589+0000 7fa545400640 1 -- 172.21.15.69:0/2062776404 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686431058 0 0) 0x7fa54815ed40 con 0x7fa54806cfb0 2024-04-24T08:10:31.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.589+0000 7fa545400640 1 -- 172.21.15.69:0/2062776404 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa52c003520 con 0x7fa54806cfb0 2024-04-24T08:10:31.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.589+0000 7fa546800640 1 -- 172.21.15.69:0/2062776404 >> v1:172.21.15.121:6789/0 conn(0x7fa54815e960 legacy=0x7fa548162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:31.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.589+0000 7fa546800640 1 -- 172.21.15.69:0/2062776404 >> v1:172.21.15.69:6789/0 conn(0x7fa54806cfb0 legacy=0x7fa5480cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.589+0000 7fa546800640 1 -- 172.21.15.69:0/2062776404 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa548162e20 con 0x7fa5480d0200 2024-04-24T08:10:31.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.590+0000 7fa546800640 1 --2- 172.21.15.69:0/2062776404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5480d0200 0x7fa5480d2980 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa5480cee40 tx=0x7fa54002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8789b610a635e0b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:31.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.590+0000 7fa545400640 1 -- 172.21.15.69:0/2062776404 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54002ae60 con 0x7fa5480d0200 2024-04-24T08:10:31.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.590+0000 7fa545400640 1 -- 172.21.15.69:0/2062776404 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa540007740 con 0x7fa5480d0200 2024-04-24T08:10:31.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.590+0000 7fa545400640 1 -- 172.21.15.69:0/2062776404 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540007b10 con 0x7fa5480d0200 2024-04-24T08:10:31.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.591+0000 7fa547200640 1 -- 172.21.15.69:0/2062776404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5480d0200 msgr2=0x7fa5480d2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.591+0000 7fa547200640 1 --2- 172.21.15.69:0/2062776404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5480d0200 0x7fa5480d2980 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa5480cee40 tx=0x7fa54002a680 comp rx=0 tx=0).stop 2024-04-24T08:10:31.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.591+0000 7fa547200640 1 -- 172.21.15.69:0/2062776404 shutdown_connections 2024-04-24T08:10:31.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.591+0000 7fa547200640 1 --2- 172.21.15.69:0/2062776404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5480d0200 0x7fa5480d2980 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.591+0000 7fa547200640 1 -- 172.21.15.69:0/2062776404 >> 172.21.15.69:0/2062776404 conn(0x7fa54809d900 msgr2=0x7fa54808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:31.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.591+0000 7fa547200640 1 -- 172.21.15.69:0/2062776404 shutdown_connections 2024-04-24T08:10:31.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.591+0000 7fa547200640 1 -- 172.21.15.69:0/2062776404 wait complete. 2024-04-24T08:10:31.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.593+0000 7fa547200640 1 Processor -- start 2024-04-24T08:10:31.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.593+0000 7fa547200640 1 -- start start 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.594+0000 7fa547200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa54806cfb0 0x7fa5481619f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.594+0000 7fa547200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5480d0200 0x7fa548150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.594+0000 7fa547200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa54815e960 0x7fa54815ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.594+0000 7fa547200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa5480cf5d0 con 0x7fa54815e960 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.594+0000 7fa547200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa5480cf2f0 con 0x7fa5480d0200 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.594+0000 7fa547200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa5480b6170 con 0x7fa54806cfb0 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.594+0000 7fa546800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa54815e960 0x7fa54815ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.594+0000 7fa546800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa54815e960 0x7fa54815ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59308/0 (socket says 172.21.15.69:59308) 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.594+0000 7fa546800640 1 -- 172.21.15.69:0/57235156 learned_addr learned my addr 172.21.15.69:0/57235156 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa53d400640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5480d0200 0x7fa548150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa545e00640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa54806cfb0 0x7fa5481619f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa546800640 1 -- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa54806cfb0 msgr2=0x7fa5481619f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa546800640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa54806cfb0 0x7fa5481619f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa546800640 1 -- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5480d0200 msgr2=0x7fa548150a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa546800640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5480d0200 0x7fa548150a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa546800640 1 -- 172.21.15.69:0/57235156 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa548160950 con 0x7fa54815e960 2024-04-24T08:10:31.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa546800640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa54815e960 0x7fa54815ff90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa54002ac70 tx=0x7fa540004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:31.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa53ea00640 1 -- 172.21.15.69:0/57235156 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540040070 con 0x7fa54815e960 2024-04-24T08:10:31.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa53ea00640 1 -- 172.21.15.69:0/57235156 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa54002ae60 con 0x7fa54815e960 2024-04-24T08:10:31.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa547200640 1 -- 172.21.15.69:0/57235156 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa548160c30 con 0x7fa54815e960 2024-04-24T08:10:31.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.595+0000 7fa53ea00640 1 -- 172.21.15.69:0/57235156 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540031be0 con 0x7fa54815e960 2024-04-24T08:10:31.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.596+0000 7fa547200640 1 -- 172.21.15.69:0/57235156 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5481611f0 con 0x7fa54815e960 2024-04-24T08:10:31.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.597+0000 7fa547200640 1 -- 172.21.15.69:0/57235156 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa548075d50 con 0x7fa54815e960 2024-04-24T08:10:31.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.597+0000 7fa53ea00640 1 -- 172.21.15.69:0/57235156 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa540039040 con 0x7fa54815e960 2024-04-24T08:10:31.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.598+0000 7fa53ea00640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52406c8f0 0x7fa52406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.599+0000 7fa53ea00640 1 -- 172.21.15.69:0/57235156 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7fa5400acf20 con 0x7fa54815e960 2024-04-24T08:10:31.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.599+0000 7fa545e00640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52406c8f0 0x7fa52406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.600+0000 7fa545e00640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52406c8f0 0x7fa52406edb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa538006fd0 tx=0x7fa538008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:31.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.603+0000 7fa53ea00640 1 -- 172.21.15.69:0/57235156 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa54007b600 con 0x7fa54815e960 2024-04-24T08:10:31.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.788+0000 7fa547200640 1 -- 172.21.15.69:0/57235156 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5480d1dc0 con 0x7fa54815e960 2024-04-24T08:10:31.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.790+0000 7fa53ea00640 1 -- 172.21.15.69:0/57235156 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+12867 (secure 0 0 0) 0x7fa54007af00 con 0x7fa54815e960 2024-04-24T08:10:31.790 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:31.790 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":40,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:10:30.901513+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:31.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 -- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52406c8f0 msgr2=0x7fa52406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52406c8f0 0x7fa52406edb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa538006fd0 tx=0x7fa538008040 comp rx=0 tx=0).stop 2024-04-24T08:10:31.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 -- 172.21.15.69:0/57235156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa54815e960 msgr2=0x7fa54815ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa54815e960 0x7fa54815ff90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa54002ac70 tx=0x7fa540004a40 comp rx=0 tx=0).stop 2024-04-24T08:10:31.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 -- 172.21.15.69:0/57235156 shutdown_connections 2024-04-24T08:10:31.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52406c8f0 0x7fa52406edb0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa54815e960 0x7fa54815ff90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5480d0200 0x7fa548150a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 --2- 172.21.15.69:0/57235156 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa54806cfb0 0x7fa5481619f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 -- 172.21.15.69:0/57235156 >> 172.21.15.69:0/57235156 conn(0x7fa54809d900 msgr2=0x7fa54806dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 -- 172.21.15.69:0/57235156 shutdown_connections 2024-04-24T08:10:31.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.792+0000 7fa547200640 1 -- 172.21.15.69:0/57235156 wait complete. 2024-04-24T08:10:31.810 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:10:31.811 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:10:31.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.997+0000 7f72d5a00640 1 Processor -- start 2024-04-24T08:10:31.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.997+0000 7f72d5a00640 1 -- start start 2024-04-24T08:10:31.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.999+0000 7f72d5a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d0071d70 0x7f72d015ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.999+0000 7f72d5a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d00721d0 con 0x7f72d0160510 2024-04-24T08:10:31.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.999+0000 7f72d5a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d0072450 con 0x7f72d01420a0 2024-04-24T08:10:31.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.999+0000 7f72d5a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72d0160960 con 0x7f72d0071d70 2024-04-24T08:10:31.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.999+0000 7f72d5000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f72d0160510 0x7f72d00cf670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37260/0 (socket says 172.21.15.69:37260) 2024-04-24T08:10:32.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.999+0000 7f72d5000640 1 -- 172.21.15.69:0/2833548849 learned_addr learned my addr 172.21.15.69:0/2833548849 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:32.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:31.999+0000 7f72cf400640 1 --2- 172.21.15.69:0/2833548849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d0071d70 0x7f72d015ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2300546462 0 0) 0x7f72c8001490 con 0x7f72d0160510 2024-04-24T08:10:32.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f72b4003520 con 0x7f72d0160510 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1459362635 0 0) 0x7f72b4003520 con 0x7f72d0160510 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d0071d70 msgr2=0x7f72d015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 --2- 172.21.15.69:0/2833548849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d0071d70 0x7f72d015ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 >> v1:172.21.15.121:6789/0 conn(0x7f72d01420a0 legacy=0x7f72d0142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d0072730 con 0x7f72d0160510 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f72c8001bc0 con 0x7f72d0160510 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 >> v1:172.21.15.69:6789/0 conn(0x7f72d0160510 legacy=0x7f72d00cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 --2- 172.21.15.69:0/2833548849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40039e0 0x7f72b4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 --2- 172.21.15.69:0/2833548849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d0071d70 0x7f72b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 --2- 172.21.15.69:0/2833548849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b4004d40 0x7f72b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0160960 con 0x7f72b40039e0 2024-04-24T08:10:32.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f72b400ba70 con 0x7f72b4004d40 2024-04-24T08:10:32.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72b400bc70 con 0x7f72d0071d70 2024-04-24T08:10:32.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.000+0000 7f72cfe00640 1 --2- 172.21.15.69:0/2833548849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40039e0 0x7f72b4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.001+0000 7f72cfe00640 1 -- 172.21.15.69:0/2833548849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d0071d70 msgr2=0x7f72b4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:32.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.001+0000 7f72cfe00640 1 --2- 172.21.15.69:0/2833548849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d0071d70 0x7f72b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.001+0000 7f72cfe00640 1 -- 172.21.15.69:0/2833548849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b4004d40 msgr2=0x7f72b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:32.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.001+0000 7f72cfe00640 1 --2- 172.21.15.69:0/2833548849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b4004d40 0x7f72b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.001+0000 7f72cfe00640 1 -- 172.21.15.69:0/2833548849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b4003520 con 0x7f72b40039e0 2024-04-24T08:10:32.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.002+0000 7f72cfe00640 1 --2- 172.21.15.69:0/2833548849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40039e0 0x7f72b4003e00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f72c0001720 tx=0x7f72b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa6243b78943bb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:32.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.002+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c0001370 con 0x7f72b40039e0 2024-04-24T08:10:32.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.002+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72c00052d0 con 0x7f72b40039e0 2024-04-24T08:10:32.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.002+0000 7f72cea00640 1 -- 172.21.15.69:0/2833548849 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c002abc0 con 0x7f72b40039e0 2024-04-24T08:10:32.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.003+0000 7f72d5a00640 1 -- 172.21.15.69:0/2833548849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40039e0 msgr2=0x7f72b4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.003+0000 7f72d5a00640 1 --2- 172.21.15.69:0/2833548849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72b40039e0 0x7f72b4003e00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f72c0001720 tx=0x7f72b4005830 comp rx=0 tx=0).stop 2024-04-24T08:10:32.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.003+0000 7f72d5000640 1 -- 172.21.15.69:0/2833548849 reap_dead start 2024-04-24T08:10:32.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.003+0000 7f72d5a00640 1 -- 172.21.15.69:0/2833548849 shutdown_connections 2024-04-24T08:10:32.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.003+0000 7f72d5a00640 1 -- 172.21.15.69:0/2833548849 >> 172.21.15.69:0/2833548849 conn(0x7f72d0067b80 msgr2=0x7f72d006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:32.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.003+0000 7f72d5a00640 1 -- 172.21.15.69:0/2833548849 shutdown_connections 2024-04-24T08:10:32.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.003+0000 7f72d5a00640 1 -- 172.21.15.69:0/2833548849 wait complete. 2024-04-24T08:10:32.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.005+0000 7f72d5a00640 1 Processor -- start 2024-04-24T08:10:32.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.005+0000 7f72d5a00640 1 -- start start 2024-04-24T08:10:32.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72d5a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b40039e0 0x7f72d01f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72d5a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d013e700 0x7f72d01e8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72d5a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d013db70 0x7f72d013dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72d5a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0072cf0 con 0x7f72d013e700 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72d5a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f72d00cf1e0 con 0x7f72b40039e0 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72d5a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72d015ffd0 con 0x7f72d013db70 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72cf400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d013e700 0x7f72d01e8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72cf400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d013e700 0x7f72d01e8470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59338/0 (socket says 172.21.15.69:59338) 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72cf400640 1 -- 172.21.15.69:0/2107946910 learned_addr learned my addr 172.21.15.69:0/2107946910 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72cf400640 1 -- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d013db70 msgr2=0x7f72d013dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72cf400640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d013db70 0x7f72d013dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72cf400640 1 -- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b40039e0 msgr2=0x7f72d01f6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72cfe00640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b40039e0 0x7f72d01f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72cf400640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b40039e0 0x7f72d01f6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.006+0000 7f72cf400640 1 -- 172.21.15.69:0/2107946910 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d01550a0 con 0x7f72d013e700 2024-04-24T08:10:32.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.007+0000 7f72cf400640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d013e700 0x7f72d01e8470 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00e820 tx=0x7f72bc00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:32.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.007+0000 7f72c7e00640 1 -- 172.21.15.69:0/2107946910 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc017070 con 0x7f72d013e700 2024-04-24T08:10:32.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.007+0000 7f72c7e00640 1 -- 172.21.15.69:0/2107946910 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72bc00c8e0 con 0x7f72d013e700 2024-04-24T08:10:32.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.007+0000 7f72c7e00640 1 -- 172.21.15.69:0/2107946910 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc00ccb0 con 0x7f72d013e700 2024-04-24T08:10:32.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.007+0000 7f72d5a00640 1 -- 172.21.15.69:0/2107946910 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d0155380 con 0x7f72d013e700 2024-04-24T08:10:32.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.007+0000 7f72d5a00640 1 -- 172.21.15.69:0/2107946910 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d0155940 con 0x7f72d013e700 2024-04-24T08:10:32.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.008+0000 7f72d5a00640 1 -- 172.21.15.69:0/2107946910 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d0075d50 con 0x7f72d013e700 2024-04-24T08:10:32.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.009+0000 7f72c7e00640 1 -- 172.21.15.69:0/2107946910 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f72bc01a070 con 0x7f72d013e700 2024-04-24T08:10:32.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.010+0000 7f72c7e00640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72b006c8f0 0x7f72b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.010+0000 7f72c7e00640 1 -- 172.21.15.69:0/2107946910 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7f72bc08b930 con 0x7f72d013e700 2024-04-24T08:10:32.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.011+0000 7f72cfe00640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72b006c8f0 0x7f72b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.012+0000 7f72cfe00640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72b006c8f0 0x7f72b006edb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f72c0032000 tx=0x7f72c002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:32.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.015+0000 7f72c7e00640 1 -- 172.21.15.69:0/2107946910 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72bc05a010 con 0x7f72d013e700 2024-04-24T08:10:32.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.215+0000 7f72d5a00640 1 -- 172.21.15.69:0/2107946910 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f72d006c470 con 0x7f72d013e700 2024-04-24T08:10:32.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.215+0000 7f72c7e00640 1 -- 172.21.15.69:0/2107946910 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) ==== 93+0+10 (secure 0 0 0) 0x7f72bc059910 con 0x7f72d013e700 2024-04-24T08:10:32.216 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:10:32.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 -- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72b006c8f0 msgr2=0x7f72b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72b006c8f0 0x7f72b006edb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f72c0032000 tx=0x7f72c002e000 comp rx=0 tx=0).stop 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 -- 172.21.15.69:0/2107946910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d013e700 msgr2=0x7f72d01e8470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d013e700 0x7f72d01e8470 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00e820 tx=0x7f72bc00ece0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 -- 172.21.15.69:0/2107946910 shutdown_connections 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f72b006c8f0 0x7f72b006edb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72d013e700 0x7f72d01e8470 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72d013db70 0x7f72d013dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 --2- 172.21.15.69:0/2107946910 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72b40039e0 0x7f72d01f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 -- 172.21.15.69:0/2107946910 >> 172.21.15.69:0/2107946910 conn(0x7f72d0067b80 msgr2=0x7f72d015e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 -- 172.21.15.69:0/2107946910 shutdown_connections 2024-04-24T08:10:32.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.218+0000 7f72d5a00640 1 -- 172.21.15.69:0/2107946910 wait complete. 2024-04-24T08:10:32.238 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T08:10:32.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.432+0000 7f6c32000640 1 Processor -- start 2024-04-24T08:10:32.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.432+0000 7f6c32000640 1 -- start start 2024-04-24T08:10:32.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c32000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6c2c15ab10 0x7f6c2c166a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c32000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c2c0b3190 con 0x7f6c2c071be0 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c32000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c2c0ad590 con 0x7f6c2c144090 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c32000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6c2c15f4d0 con 0x7f6c2c15ab10 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c30c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6c2c071be0 0x7f6c2c15e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37274/0 (socket says 172.21.15.69:37274) 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c30c00640 1 -- 172.21.15.69:0/2214381932 learned_addr learned my addr 172.21.15.69:0/2214381932 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c31600640 1 --2- 172.21.15.69:0/2214381932 >> v2:172.21.15.69:3300/0 conn(0x7f6c2c15ab10 0x7f6c2c166a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c2b400640 1 -- 172.21.15.69:0/2214381932 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 138580258 0 0) 0x7f6c2c0b3190 con 0x7f6c2c071be0 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c2b400640 1 -- 172.21.15.69:0/2214381932 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c14003520 con 0x7f6c2c071be0 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c31600640 1 -- 172.21.15.69:0/2214381932 >> v1:172.21.15.121:6789/0 conn(0x7f6c2c144090 legacy=0x7f6c2c15edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c31600640 1 -- 172.21.15.69:0/2214381932 >> v1:172.21.15.69:6789/0 conn(0x7f6c2c071be0 legacy=0x7f6c2c15e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.434+0000 7f6c31600640 1 -- 172.21.15.69:0/2214381932 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2c15afa0 con 0x7f6c2c15ab10 2024-04-24T08:10:32.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.436+0000 7f6c31600640 1 --2- 172.21.15.69:0/2214381932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c15ab10 0x7f6c2c166a30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009d90 tx=0x7f6c2402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c49581c5578224 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:32.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.436+0000 7f6c2b400640 1 -- 172.21.15.69:0/2214381932 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24007740 con 0x7f6c2c15ab10 2024-04-24T08:10:32.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.436+0000 7f6c2b400640 1 -- 172.21.15.69:0/2214381932 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c240078e0 con 0x7f6c2c15ab10 2024-04-24T08:10:32.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.436+0000 7f6c2b400640 1 -- 172.21.15.69:0/2214381932 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c24007cb0 con 0x7f6c2c15ab10 2024-04-24T08:10:32.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.437+0000 7f6c32000640 1 -- 172.21.15.69:0/2214381932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c15ab10 msgr2=0x7f6c2c166a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.437+0000 7f6c32000640 1 --2- 172.21.15.69:0/2214381932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c15ab10 0x7f6c2c166a30 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009d90 tx=0x7f6c2402a6a0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.437+0000 7f6c32000640 1 -- 172.21.15.69:0/2214381932 shutdown_connections 2024-04-24T08:10:32.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.437+0000 7f6c32000640 1 --2- 172.21.15.69:0/2214381932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c15ab10 0x7f6c2c166a30 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.437+0000 7f6c32000640 1 -- 172.21.15.69:0/2214381932 >> 172.21.15.69:0/2214381932 conn(0x7f6c2c0d0160 msgr2=0x7f6c2c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:32.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.437+0000 7f6c32000640 1 -- 172.21.15.69:0/2214381932 shutdown_connections 2024-04-24T08:10:32.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.437+0000 7f6c32000640 1 -- 172.21.15.69:0/2214381932 wait complete. 2024-04-24T08:10:32.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.438+0000 7f6c32000640 1 Processor -- start 2024-04-24T08:10:32.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.439+0000 7f6c32000640 1 -- start start 2024-04-24T08:10:32.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.439+0000 7f6c32000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c071be0 0x7f6c2c0b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.439+0000 7f6c32000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c2c144090 0x7f6c2c099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.439+0000 7f6c32000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c2c15ab10 0x7f6c2c15bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c32000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c0b39c0 con 0x7f6c2c071be0 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c32000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c0b36e0 con 0x7f6c2c144090 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c32000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2c072150 con 0x7f6c2c15ab10 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c30c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c071be0 0x7f6c2c0b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c30c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c071be0 0x7f6c2c0b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59348/0 (socket says 172.21.15.69:59348) 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c30c00640 1 -- 172.21.15.69:0/3608866703 learned_addr learned my addr 172.21.15.69:0/3608866703 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c30c00640 1 -- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c2c15ab10 msgr2=0x7f6c2c15bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c30c00640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c2c15ab10 0x7f6c2c15bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c30c00640 1 -- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c2c144090 msgr2=0x7f6c2c099a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c30c00640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c2c144090 0x7f6c2c099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c30c00640 1 -- 172.21.15.69:0/3608866703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2c1e8fe0 con 0x7f6c2c071be0 2024-04-24T08:10:32.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.440+0000 7f6c30c00640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c071be0 0x7f6c2c0b00f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000c430 tx=0x7f6c2000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:32.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.441+0000 7f6c28c00640 1 -- 172.21.15.69:0/3608866703 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20017070 con 0x7f6c2c071be0 2024-04-24T08:10:32.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.441+0000 7f6c28c00640 1 -- 172.21.15.69:0/3608866703 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c20005ce0 con 0x7f6c2c071be0 2024-04-24T08:10:32.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.441+0000 7f6c28c00640 1 -- 172.21.15.69:0/3608866703 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c200072d0 con 0x7f6c2c071be0 2024-04-24T08:10:32.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.441+0000 7f6c32000640 1 -- 172.21.15.69:0/3608866703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c2c15d880 con 0x7f6c2c071be0 2024-04-24T08:10:32.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.441+0000 7f6c32000640 1 -- 172.21.15.69:0/3608866703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c2c1e6890 con 0x7f6c2c071be0 2024-04-24T08:10:32.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.442+0000 7f6c32000640 1 -- 172.21.15.69:0/3608866703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c2c075d50 con 0x7f6c2c071be0 2024-04-24T08:10:32.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.443+0000 7f6c28c00640 1 -- 172.21.15.69:0/3608866703 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6c2001a070 con 0x7f6c2c071be0 2024-04-24T08:10:32.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.444+0000 7f6c28c00640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c0c06c920 0x7f6c0c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.444+0000 7f6c28c00640 1 -- 172.21.15.69:0/3608866703 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7f6c2008ab40 con 0x7f6c2c071be0 2024-04-24T08:10:32.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.444+0000 7f6c2be00640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c0c06c920 0x7f6c0c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.446+0000 7f6c2be00640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c0c06c920 0x7f6c0c06ede0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c153610 tx=0x7f6c180079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:32.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.448+0000 7f6c28c00640 1 -- 172.21.15.69:0/3608866703 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6c20058bd0 con 0x7f6c2c071be0 2024-04-24T08:10:32.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.641+0000 7f6c32000640 1 -- 172.21.15.69:0/3608866703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f6c2c06f020 con 0x7f6c2c071be0 2024-04-24T08:10:32.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.642+0000 7f6c28c00640 1 -- 172.21.15.69:0/3608866703 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v40) ==== 84+0+65 (secure 0 0 0) 0x7f6c2005ced0 con 0x7f6c2c071be0 2024-04-24T08:10:32.642 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:32.642 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:10:32.642 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:32.643 INFO:teuthology.orchestra.run.smithi069.stdout: "poolnum": 1, 2024-04-24T08:10:32.643 INFO:teuthology.orchestra.run.smithi069.stdout: "poolname": ".mgr" 2024-04-24T08:10:32.643 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:32.643 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:10:32.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.644+0000 7f6c32000640 1 -- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c0c06c920 msgr2=0x7f6c0c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c0c06c920 0x7f6c0c06ede0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c153610 tx=0x7f6c180079b0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 -- 172.21.15.69:0/3608866703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c071be0 msgr2=0x7f6c2c0b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c071be0 0x7f6c2c0b00f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000c430 tx=0x7f6c2000c8f0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 -- 172.21.15.69:0/3608866703 shutdown_connections 2024-04-24T08:10:32.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6c0c06c920 0x7f6c0c06ede0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6c2c15ab10 0x7f6c2c15bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c2c144090 0x7f6c2c099a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 --2- 172.21.15.69:0/3608866703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c2c071be0 0x7f6c2c0b00f0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 -- 172.21.15.69:0/3608866703 >> 172.21.15.69:0/3608866703 conn(0x7f6c2c0d0160 msgr2=0x7f6c2c152cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:32.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 -- 172.21.15.69:0/3608866703 shutdown_connections 2024-04-24T08:10:32.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.645+0000 7f6c32000640 1 -- 172.21.15.69:0/3608866703 wait complete. 2024-04-24T08:10:32.664 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T08:10:32.664 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T08:10:32.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.863+0000 7f89d5200640 1 Processor -- start 2024-04-24T08:10:32.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.863+0000 7f89d5200640 1 -- start start 2024-04-24T08:10:32.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.865+0000 7f89d5200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f89d015c590 0x7f89d00b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.865+0000 7f89d5200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d00b4070 0x7f89d00d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.865+0000 7f89d5200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89d00b4840 con 0x7f89d00d0d00 2024-04-24T08:10:32.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.865+0000 7f89d5200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f89d00d1150 con 0x7f89d015c590 2024-04-24T08:10:32.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.865+0000 7f89d5200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f89d00b4560 con 0x7f89d00b4070 2024-04-24T08:10:32.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.865+0000 7f89ce800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f89d00d0d00 0x7f89d00cdf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37282/0 (socket says 172.21.15.69:37282) 2024-04-24T08:10:32.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.865+0000 7f89ce800640 1 -- 172.21.15.69:0/4249653924 learned_addr learned my addr 172.21.15.69:0/4249653924 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:32.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.865+0000 7f89cd400640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d00b4070 0x7f89d00d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.865+0000 7f89cde00640 1 --2- 172.21.15.69:0/4249653924 >> v2:172.21.15.121:3300/0 conn(0x7f89d015c590 0x7f89d00b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 741430260 0 0) 0x7f89d00b4840 con 0x7f89d00d0d00 2024-04-24T08:10:32.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89b4003520 con 0x7f89d00d0d00 2024-04-24T08:10:32.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1933368959 0 0) 0x7f89b4003520 con 0x7f89d00d0d00 2024-04-24T08:10:32.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d00b4070 msgr2=0x7f89d00d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:32.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d00b4070 0x7f89d00d07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 >> v2:172.21.15.121:3300/0 conn(0x7f89d015c590 msgr2=0x7f89d00b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 --2- 172.21.15.69:0/4249653924 >> v2:172.21.15.121:3300/0 conn(0x7f89d015c590 0x7f89d00b3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d00ad2e0 con 0x7f89d00d0d00 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f89c80019f0 con 0x7f89d00d0d00 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 >> v1:172.21.15.69:6789/0 conn(0x7f89d00d0d00 legacy=0x7f89d00cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89b40039b0 0x7f89b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 0x7f89b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d00b4070 0x7f89b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89d00d1150 con 0x7f89b4004800 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f89d00b4560 con 0x7f89b40039b0 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.866+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f89b400b9c0 con 0x7f89d00b4070 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.867+0000 7f89cd400640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 0x7f89b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.867+0000 7f89cde00640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89b40039b0 0x7f89b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.867+0000 7f89cd400640 1 -- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d00b4070 msgr2=0x7f89b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:32.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.867+0000 7f89cd400640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d00b4070 0x7f89b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.867+0000 7f89cd400640 1 -- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89b40039b0 msgr2=0x7f89b4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:32.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.867+0000 7f89cd400640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89b40039b0 0x7f89b4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.867+0000 7f89cd400640 1 -- 172.21.15.69:0/4249653924 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b4003520 con 0x7f89b4004800 2024-04-24T08:10:32.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.867+0000 7f89cde00640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89b40039b0 0x7f89b4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:32.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.868+0000 7f89cd400640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 0x7f89b4004c80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009a90 tx=0x7f89b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce66726c166395a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:32.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.868+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8036070 con 0x7f89b4004800 2024-04-24T08:10:32.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.868+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b8031400 con 0x7f89b4004800 2024-04-24T08:10:32.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.868+0000 7f89cca00640 1 -- 172.21.15.69:0/4249653924 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b8009690 con 0x7f89b4004800 2024-04-24T08:10:32.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.869+0000 7f89d5200640 1 -- 172.21.15.69:0/4249653924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 msgr2=0x7f89b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.869+0000 7f89d5200640 1 --2- 172.21.15.69:0/4249653924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 0x7f89b4004c80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f89b8009a90 tx=0x7f89b4005830 comp rx=0 tx=0).stop 2024-04-24T08:10:32.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.869+0000 7f89ce800640 1 -- 172.21.15.69:0/4249653924 reap_dead start 2024-04-24T08:10:32.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.869+0000 7f89d5200640 1 -- 172.21.15.69:0/4249653924 shutdown_connections 2024-04-24T08:10:32.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.869+0000 7f89d5200640 1 -- 172.21.15.69:0/4249653924 >> 172.21.15.69:0/4249653924 conn(0x7f89d006be40 msgr2=0x7f89d006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:32.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.869+0000 7f89d5200640 1 -- 172.21.15.69:0/4249653924 shutdown_connections 2024-04-24T08:10:32.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.869+0000 7f89d5200640 1 -- 172.21.15.69:0/4249653924 wait complete. 2024-04-24T08:10:32.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.871+0000 7f89d5200640 1 Processor -- start 2024-04-24T08:10:32.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.871+0000 7f89d5200640 1 -- start start 2024-04-24T08:10:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.872+0000 7f89d5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 0x7f89d0150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.872+0000 7f89d5200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d01e4e60 0x7f89d01e52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.872+0000 7f89d5200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d015e980 0x7f89d015ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.872+0000 7f89d5200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89d00b5160 con 0x7f89b4004800 2024-04-24T08:10:32.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.872+0000 7f89d5200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f89d00ad8a0 con 0x7f89d01e4e60 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.872+0000 7f89d5200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f89d00b52e0 con 0x7f89d015e980 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.872+0000 7f89cde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 0x7f89d0150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.872+0000 7f89cde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 0x7f89d0150090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59372/0 (socket says 172.21.15.69:59372) 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.872+0000 7f89cde00640 1 -- 172.21.15.69:0/942715178 learned_addr learned my addr 172.21.15.69:0/942715178 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89cd400640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d01e4e60 0x7f89d01e52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89ce800640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d015e980 0x7f89d015ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89cde00640 1 -- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d015e980 msgr2=0x7f89d015ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89cde00640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d015e980 0x7f89d015ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89cde00640 1 -- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d01e4e60 msgr2=0x7f89d01e52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89cde00640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d01e4e60 0x7f89d01e52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89cde00640 1 -- 172.21.15.69:0/942715178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d01e5920 con 0x7f89b4004800 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89cd400640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d01e4e60 0x7f89d01e52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89cde00640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 0x7f89d0150090 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f89c00109d0 tx=0x7f89c0010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:32.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89c6000640 1 -- 172.21.15.69:0/942715178 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c000f040 con 0x7f89b4004800 2024-04-24T08:10:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89c6000640 1 -- 172.21.15.69:0/942715178 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89c000ec20 con 0x7f89b4004800 2024-04-24T08:10:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89d5200640 1 -- 172.21.15.69:0/942715178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d01e5c00 con 0x7f89b4004800 2024-04-24T08:10:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.873+0000 7f89c6000640 1 -- 172.21.15.69:0/942715178 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89c001c400 con 0x7f89b4004800 2024-04-24T08:10:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.874+0000 7f89d5200640 1 -- 172.21.15.69:0/942715178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d015b5b0 con 0x7f89b4004800 2024-04-24T08:10:32.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.875+0000 7f89d5200640 1 -- 172.21.15.69:0/942715178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d01e5e20 con 0x7f89b4004800 2024-04-24T08:10:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.876+0000 7f89c6000640 1 -- 172.21.15.69:0/942715178 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f89c001c5a0 con 0x7f89b4004800 2024-04-24T08:10:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.876+0000 7f89c6000640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89ac06c920 0x7f89ac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.877+0000 7f89c6000640 1 -- 172.21.15.69:0/942715178 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4857+0+0 (secure 0 0 0) 0x7f89c008bf70 con 0x7f89b4004800 2024-04-24T08:10:32.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.877+0000 7f89cd400640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89ac06c920 0x7f89ac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:32.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.878+0000 7f89cd400640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89ac06c920 0x7f89ac06ede0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f89b8033000 tx=0x7f89b8008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:32.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:32.881+0000 7f89c6000640 1 -- 172.21.15.69:0/942715178 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89c0018030 con 0x7f89b4004800 2024-04-24T08:10:33.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.084+0000 7f89d5200640 1 -- 172.21.15.69:0/942715178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f89d0075dc0 con 0x7f89b4004800 2024-04-24T08:10:33.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.914+0000 7f89c6000640 1 -- 172.21.15.69:0/942715178 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v41) ==== 138+0+0 (secure 0 0 0) 0x7f89c005a6e0 con 0x7f89b4004800 2024-04-24T08:10:33.915 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' created 2024-04-24T08:10:33.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.917+0000 7f89d5200640 1 -- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89ac06c920 msgr2=0x7f89ac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:33.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.917+0000 7f89d5200640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89ac06c920 0x7f89ac06ede0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f89b8033000 tx=0x7f89b8008030 comp rx=0 tx=0).stop 2024-04-24T08:10:33.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.917+0000 7f89d5200640 1 -- 172.21.15.69:0/942715178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 msgr2=0x7f89d0150090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:33.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.917+0000 7f89d5200640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 0x7f89d0150090 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f89c00109d0 tx=0x7f89c0010e90 comp rx=0 tx=0).stop 2024-04-24T08:10:33.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.918+0000 7f89d5200640 1 -- 172.21.15.69:0/942715178 shutdown_connections 2024-04-24T08:10:33.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.918+0000 7f89d5200640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89ac06c920 0x7f89ac06ede0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:33.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.918+0000 7f89d5200640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89d015e980 0x7f89d015ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:33.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.918+0000 7f89d5200640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89d01e4e60 0x7f89d01e52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:33.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.918+0000 7f89d5200640 1 --2- 172.21.15.69:0/942715178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89b4004800 0x7f89d0150090 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:33.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.918+0000 7f89d5200640 1 -- 172.21.15.69:0/942715178 >> 172.21.15.69:0/942715178 conn(0x7f89d006be40 msgr2=0x7f89d0071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:33.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.918+0000 7f89d5200640 1 -- 172.21.15.69:0/942715178 shutdown_connections 2024-04-24T08:10:33.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:33.918+0000 7f89d5200640 1 -- 172.21.15.69:0/942715178 wait complete. 2024-04-24T08:10:33.947 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T08:10:34.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.141+0000 7f9452600640 1 Processor -- start 2024-04-24T08:10:34.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.141+0000 7f9452600640 1 -- start start 2024-04-24T08:10:34.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.143+0000 7f9452600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c0ccc50 0x7f944c072cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:34.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.143+0000 7f9452600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f944c0cd0b0 0x7f944c0731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:34.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.143+0000 7f9452600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f944c073c80 con 0x7f944c15ab10 2024-04-24T08:10:34.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.143+0000 7f9452600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f944c0ca8c0 con 0x7f944c0cd0b0 2024-04-24T08:10:34.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.143+0000 7f9452600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f944c0caba0 con 0x7f944c0ccc50 2024-04-24T08:10:34.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.143+0000 7f9451200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f944c15ab10 0x7f944c0cb960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37292/0 (socket says 172.21.15.69:37292) 2024-04-24T08:10:34.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.143+0000 7f9451200640 1 -- 172.21.15.69:0/4016565573 learned_addr learned my addr 172.21.15.69:0/4016565573 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:34.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.143+0000 7f9451c00640 1 --2- 172.21.15.69:0/4016565573 >> v2:172.21.15.121:3300/0 conn(0x7f944c0cd0b0 0x7f944c0731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:34.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.143+0000 7f944be00640 1 --2- 172.21.15.69:0/4016565573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c0ccc50 0x7f944c072cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:34.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.144+0000 7f944b400640 1 -- 172.21.15.69:0/4016565573 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3519168432 0 0) 0x7f944c073c80 con 0x7f944c15ab10 2024-04-24T08:10:34.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.144+0000 7f944b400640 1 -- 172.21.15.69:0/4016565573 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9434003520 con 0x7f944c15ab10 2024-04-24T08:10:34.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.144+0000 7f9451c00640 1 -- 172.21.15.69:0/4016565573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c0ccc50 msgr2=0x7f944c072cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:34.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.144+0000 7f9451c00640 1 --2- 172.21.15.69:0/4016565573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c0ccc50 0x7f944c072cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:34.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.144+0000 7f944b400640 1 -- 172.21.15.69:0/4016565573 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3162257463 0 0) 0x7f9434003520 con 0x7f944c15ab10 2024-04-24T08:10:34.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.144+0000 7f9451c00640 1 -- 172.21.15.69:0/4016565573 >> v1:172.21.15.69:6789/0 conn(0x7f944c15ab10 legacy=0x7f944c0cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:34.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.144+0000 7f9451c00640 1 -- 172.21.15.69:0/4016565573 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f944c0cae80 con 0x7f944c0cd0b0 2024-04-24T08:10:34.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.146+0000 7f9451c00640 1 --2- 172.21.15.69:0/4016565573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f944c0cd0b0 0x7f944c0731f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9444009d90 tx=0x7f944402a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=626907873137ef36 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:34.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.146+0000 7f944b400640 1 -- 172.21.15.69:0/4016565573 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9440001690 con 0x7f944c0cd0b0 2024-04-24T08:10:34.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.146+0000 7f944b400640 1 -- 172.21.15.69:0/4016565573 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9444007540 con 0x7f944c0cd0b0 2024-04-24T08:10:34.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.146+0000 7f944b400640 1 -- 172.21.15.69:0/4016565573 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9444007910 con 0x7f944c0cd0b0 2024-04-24T08:10:34.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.147+0000 7f9452600640 1 -- 172.21.15.69:0/4016565573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f944c0cd0b0 msgr2=0x7f944c0731f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:34.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.147+0000 7f9452600640 1 --2- 172.21.15.69:0/4016565573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f944c0cd0b0 0x7f944c0731f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9444009d90 tx=0x7f944402a560 comp rx=0 tx=0).stop 2024-04-24T08:10:34.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.147+0000 7f9452600640 1 -- 172.21.15.69:0/4016565573 shutdown_connections 2024-04-24T08:10:34.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.147+0000 7f9452600640 1 --2- 172.21.15.69:0/4016565573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f944c0cd0b0 0x7f944c0731f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:34.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.147+0000 7f9452600640 1 --2- 172.21.15.69:0/4016565573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c0ccc50 0x7f944c072cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:34.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.147+0000 7f9452600640 1 -- 172.21.15.69:0/4016565573 >> 172.21.15.69:0/4016565573 conn(0x7f944c0b1da0 msgr2=0x7f944c0b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:34.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.147+0000 7f9452600640 1 -- 172.21.15.69:0/4016565573 shutdown_connections 2024-04-24T08:10:34.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.147+0000 7f9452600640 1 -- 172.21.15.69:0/4016565573 wait complete. 2024-04-24T08:10:34.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.149+0000 7f9452600640 1 Processor -- start 2024-04-24T08:10:34.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.149+0000 7f9452600640 1 -- start start 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.150+0000 7f9452600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944c0ccc50 0x7f944c0aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.150+0000 7f9452600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f944c0cd0b0 0x7f944c1493b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.150+0000 7f9452600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c15ab10 0x7f944c0c2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.150+0000 7f9452600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f944c15aef0 con 0x7f944c0ccc50 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.150+0000 7f9452600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f944c0cb440 con 0x7f944c0cd0b0 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.150+0000 7f9452600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f944c15b070 con 0x7f944c15ab10 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.150+0000 7f9451200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944c0ccc50 0x7f944c0aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.150+0000 7f9451200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944c0ccc50 0x7f944c0aec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59386/0 (socket says 172.21.15.69:59386) 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.150+0000 7f9451200640 1 -- 172.21.15.69:0/3944668445 learned_addr learned my addr 172.21.15.69:0/3944668445 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f944be00640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f944c0cd0b0 0x7f944c1493b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9451c00640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c15ab10 0x7f944c0c2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:34.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9451200640 1 -- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c15ab10 msgr2=0x7f944c0c2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9451200640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c15ab10 0x7f944c0c2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9451200640 1 -- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f944c0cd0b0 msgr2=0x7f944c1493b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9451200640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f944c0cd0b0 0x7f944c1493b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9451200640 1 -- 172.21.15.69:0/3944668445 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f944c0c69d0 con 0x7f944c0ccc50 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9451c00640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c15ab10 0x7f944c0c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9451200640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944c0ccc50 0x7f944c0aec50 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f944000c500 tx=0x7f944000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9448c00640 1 -- 172.21.15.69:0/3944668445 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9440018070 con 0x7f944c0ccc50 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9448c00640 1 -- 172.21.15.69:0/3944668445 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9440007560 con 0x7f944c0ccc50 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9448c00640 1 -- 172.21.15.69:0/3944668445 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94400049e0 con 0x7f944c0ccc50 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.151+0000 7f9452600640 1 -- 172.21.15.69:0/3944668445 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f944c0af230 con 0x7f944c0ccc50 2024-04-24T08:10:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.152+0000 7f9452600640 1 -- 172.21.15.69:0/3944668445 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f944c0c61d0 con 0x7f944c0ccc50 2024-04-24T08:10:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.153+0000 7f9452600640 1 -- 172.21.15.69:0/3944668445 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f944c075d50 con 0x7f944c0ccc50 2024-04-24T08:10:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.154+0000 7f9448c00640 1 -- 172.21.15.69:0/3944668445 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f944001b070 con 0x7f944c0ccc50 2024-04-24T08:10:34.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.155+0000 7f9448c00640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f942c06c8f0 0x7f942c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:34.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.155+0000 7f9448c00640 1 -- 172.21.15.69:0/3944668445 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 5227+0+0 (secure 0 0 0) 0x7f944008ae50 con 0x7f944c0ccc50 2024-04-24T08:10:34.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.156+0000 7f944be00640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f942c06c8f0 0x7f942c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:34.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.157+0000 7f944be00640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f942c06c8f0 0x7f942c06edb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f944c14ccb0 tx=0x7f9438008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:34.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.160+0000 7f9448c00640 1 -- 172.21.15.69:0/3944668445 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f944004fb80 con 0x7f944c0ccc50 2024-04-24T08:10:34.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.363+0000 7f9452600640 1 -- 172.21.15.69:0/3944668445 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f944c14dc80 con 0x7f944c0ccc50 2024-04-24T08:10:34.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.927+0000 7f9448c00640 1 -- 172.21.15.69:0/3944668445 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v42) ==== 170+0+0 (secure 0 0 0) 0x7f9440059450 con 0x7f944c0ccc50 2024-04-24T08:10:34.929 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' created 2024-04-24T08:10:34.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 -- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f942c06c8f0 msgr2=0x7f942c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:34.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f942c06c8f0 0x7f942c06edb0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f944c14ccb0 tx=0x7f9438008040 comp rx=0 tx=0).stop 2024-04-24T08:10:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 -- 172.21.15.69:0/3944668445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944c0ccc50 msgr2=0x7f944c0aec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944c0ccc50 0x7f944c0aec50 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f944000c500 tx=0x7f944000c9c0 comp rx=0 tx=0).stop 2024-04-24T08:10:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 -- 172.21.15.69:0/3944668445 shutdown_connections 2024-04-24T08:10:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f942c06c8f0 0x7f942c06edb0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f944c15ab10 0x7f944c0c2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f944c0cd0b0 0x7f944c1493b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 --2- 172.21.15.69:0/3944668445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f944c0ccc50 0x7f944c0aec50 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 -- 172.21.15.69:0/3944668445 >> 172.21.15.69:0/3944668445 conn(0x7f944c0b1da0 msgr2=0x7f944c158100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 -- 172.21.15.69:0/3944668445 shutdown_connections 2024-04-24T08:10:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:34.931+0000 7f9452600640 1 -- 172.21.15.69:0/3944668445 wait complete. 2024-04-24T08:10:34.965 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T08:10:35.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.139+0000 7f00bb200640 1 Processor -- start 2024-04-24T08:10:35.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.139+0000 7f00bb200640 1 -- start start 2024-04-24T08:10:35.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.141+0000 7f00bb200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f00b41420a0 0x7f00b4071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:35.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.141+0000 7f00bb200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00b4072450 con 0x7f00b41606e0 2024-04-24T08:10:35.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.141+0000 7f00bb200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00b4072730 con 0x7f00b4142480 2024-04-24T08:10:35.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.141+0000 7f00bb200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f00b40cf1e0 con 0x7f00b41420a0 2024-04-24T08:10:35.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.141+0000 7f00ba800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f00b41606e0 0x7f00b40cf670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37306/0 (socket says 172.21.15.69:37306) 2024-04-24T08:10:35.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.141+0000 7f00ba800640 1 -- 172.21.15.69:0/1313616121 learned_addr learned my addr 172.21.15.69:0/1313616121 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:35.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b9e00640 1 --2- 172.21.15.69:0/1313616121 >> v2:172.21.15.121:3300/0 conn(0x7f00b41420a0 0x7f00b4071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:35.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2399673751 0 0) 0x7f00b4072450 con 0x7f00b41606e0 2024-04-24T08:10:35.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f009c003520 con 0x7f00b41606e0 2024-04-24T08:10:35.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1289291220 0 0) 0x7f009c003520 con 0x7f00b41606e0 2024-04-24T08:10:35.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 >> v2:172.21.15.121:3300/0 conn(0x7f00b41420a0 msgr2=0x7f00b4071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:35.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 --2- 172.21.15.69:0/1313616121 >> v2:172.21.15.121:3300/0 conn(0x7f00b41420a0 0x7f00b4071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 >> v1:172.21.15.121:6789/0 conn(0x7f00b4142480 legacy=0x7f00b415ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:35.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b4072a10 con 0x7f00b41606e0 2024-04-24T08:10:35.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f00b00019f0 con 0x7f00b41606e0 2024-04-24T08:10:35.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 >> v1:172.21.15.69:6789/0 conn(0x7f00b41606e0 legacy=0x7f00b40cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f009c0039b0 0x7f009c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f009c004800 0x7f009c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0051c0 0x7f009c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00b40cf1e0 con 0x7f009c0039b0 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f009c00c110 con 0x7f009c004800 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f009c00c310 con 0x7f009c0051c0 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.142+0000 7f00b9e00640 1 --2- 172.21.15.69:0/1313616121 >> v2:172.21.15.121:3300/0 conn(0x7f00b41420a0 0x7f00b4071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.143+0000 7f00b9e00640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f009c0039b0 0x7f009c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.143+0000 7f00ba800640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0051c0 0x7f009c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.143+0000 7f00b9400640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f009c004800 0x7f009c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.143+0000 7f00b9e00640 1 -- 172.21.15.69:0/1313616121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0051c0 msgr2=0x7f009c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:35.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.143+0000 7f00b9e00640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0051c0 0x7f009c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.143+0000 7f00b9e00640 1 -- 172.21.15.69:0/1313616121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f009c004800 msgr2=0x7f009c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:35.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.143+0000 7f00b9e00640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f009c004800 0x7f009c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.143+0000 7f00b9e00640 1 -- 172.21.15.69:0/1313616121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009c003520 con 0x7f009c0039b0 2024-04-24T08:10:35.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.143+0000 7f00ba800640 1 -- 172.21.15.69:0/1313616121 reap_dead start 2024-04-24T08:10:35.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.144+0000 7f00b9e00640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f009c0039b0 0x7f009c003dd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f00a80098a0 tx=0x7f009c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a20355168bc50f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:35.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.144+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a8037070 con 0x7f009c0039b0 2024-04-24T08:10:35.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.145+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a802aca0 con 0x7f009c0039b0 2024-04-24T08:10:35.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.145+0000 7f00b8a00640 1 -- 172.21.15.69:0/1313616121 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a800f040 con 0x7f009c0039b0 2024-04-24T08:10:35.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.145+0000 7f00bb200640 1 -- 172.21.15.69:0/1313616121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f009c0039b0 msgr2=0x7f009c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:35.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.145+0000 7f00bb200640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f009c0039b0 0x7f009c003dd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f00a80098a0 tx=0x7f009c005830 comp rx=0 tx=0).stop 2024-04-24T08:10:35.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.145+0000 7f00bb200640 1 -- 172.21.15.69:0/1313616121 shutdown_connections 2024-04-24T08:10:35.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.145+0000 7f00bb200640 1 --2- 172.21.15.69:0/1313616121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f009c0039b0 0x7f009c003dd0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.145+0000 7f00bb200640 1 -- 172.21.15.69:0/1313616121 >> 172.21.15.69:0/1313616121 conn(0x7f00b4067b80 msgr2=0x7f00b406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.145+0000 7f00bb200640 1 -- 172.21.15.69:0/1313616121 shutdown_connections 2024-04-24T08:10:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.145+0000 7f00bb200640 1 -- 172.21.15.69:0/1313616121 wait complete. 2024-04-24T08:10:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.147+0000 7f00bb200640 1 Processor -- start 2024-04-24T08:10:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.147+0000 7f00bb200640 1 -- start start 2024-04-24T08:10:35.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.148+0000 7f00bb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0039b0 0x7f00b4155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:35.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.148+0000 7f00bb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b4153b00 0x7f00b413ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:35.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.148+0000 7f00bb200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b40aee20 0x7f00b40aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:35.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.148+0000 7f00bb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00b4071d70 con 0x7f00b4153b00 2024-04-24T08:10:35.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.148+0000 7f00bb200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f00b4142910 con 0x7f00b40aee20 2024-04-24T08:10:35.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.148+0000 7f00bb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f00b4071ef0 con 0x7f009c0039b0 2024-04-24T08:10:35.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00b9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b4153b00 0x7f00b413ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:35.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00b9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b4153b00 0x7f00b413ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59404/0 (socket says 172.21.15.69:59404) 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00b9400640 1 -- 172.21.15.69:0/1177565911 learned_addr learned my addr 172.21.15.69:0/1177565911 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00b9e00640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0039b0 0x7f00b4155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00ba800640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b40aee20 0x7f00b40aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00b9400640 1 -- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0039b0 msgr2=0x7f00b4155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00b9400640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0039b0 0x7f00b4155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00b9400640 1 -- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b40aee20 msgr2=0x7f00b40aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00b9400640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b40aee20 0x7f00b40aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00b9400640 1 -- 172.21.15.69:0/1177565911 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b4153f90 con 0x7f00b4153b00 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.149+0000 7f00b9400640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b4153b00 0x7f00b413ed60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f00a400e410 tx=0x7f00a400e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.150+0000 7f00ae000640 1 -- 172.21.15.69:0/1177565911 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4003a70 con 0x7f00b4153b00 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.150+0000 7f00ae000640 1 -- 172.21.15.69:0/1177565911 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a4018070 con 0x7f00b4153b00 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.150+0000 7f00ae000640 1 -- 172.21.15.69:0/1177565911 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4003e50 con 0x7f00b4153b00 2024-04-24T08:10:35.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.150+0000 7f00bb200640 1 -- 172.21.15.69:0/1177565911 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00b4154270 con 0x7f00b4153b00 2024-04-24T08:10:35.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.150+0000 7f00bb200640 1 -- 172.21.15.69:0/1177565911 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00b4154830 con 0x7f00b4153b00 2024-04-24T08:10:35.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.151+0000 7f00bb200640 1 -- 172.21.15.69:0/1177565911 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00b4075d50 con 0x7f00b4153b00 2024-04-24T08:10:35.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.152+0000 7f00ae000640 1 -- 172.21.15.69:0/1177565911 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f00a4004050 con 0x7f00b4153b00 2024-04-24T08:10:35.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.153+0000 7f00ae000640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f009806ca90 0x7f009806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:35.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.153+0000 7f00ae000640 1 -- 172.21.15.69:0/1177565911 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 5609+0+0 (secure 0 0 0) 0x7f00a408b890 con 0x7f00b4153b00 2024-04-24T08:10:35.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.154+0000 7f00b9e00640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f009806ca90 0x7f009806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:35.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.155+0000 7f00b9e00640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f009806ca90 0x7f009806ef50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f00a8034000 tx=0x7f00a802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:35.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.158+0000 7f00ae000640 1 -- 172.21.15.69:0/1177565911 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00a4059c80 con 0x7f00b4153b00 2024-04-24T08:10:35.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.398+0000 7f00bb200640 1 -- 172.21.15.69:0/1177565911 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f00b415f850 con 0x7f00b4153b00 2024-04-24T08:10:35.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.932+0000 7f00ae000640 1 -- 172.21.15.69:0/1177565911 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) ==== 172+0+0 (secure 0 0 0) 0x7f00a4059580 con 0x7f00b4153b00 2024-04-24T08:10:35.934 INFO:teuthology.orchestra.run.smithi069.stderr:new fs with metadata pool 6 and data pool 7 2024-04-24T08:10:35.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.936+0000 7f00bb200640 1 -- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f009806ca90 msgr2=0x7f009806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:35.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.936+0000 7f00bb200640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f009806ca90 0x7f009806ef50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f00a8034000 tx=0x7f00a802e000 comp rx=0 tx=0).stop 2024-04-24T08:10:35.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.936+0000 7f00bb200640 1 -- 172.21.15.69:0/1177565911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b4153b00 msgr2=0x7f00b413ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:35.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.936+0000 7f00bb200640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b4153b00 0x7f00b413ed60 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f00a400e410 tx=0x7f00a400e8d0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.936+0000 7f00bb200640 1 -- 172.21.15.69:0/1177565911 shutdown_connections 2024-04-24T08:10:35.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.936+0000 7f00bb200640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f009806ca90 0x7f009806ef50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.936+0000 7f00bb200640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b4153b00 0x7f00b413ed60 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.936+0000 7f00bb200640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b40aee20 0x7f00b40aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.936+0000 7f00bb200640 1 --2- 172.21.15.69:0/1177565911 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f009c0039b0 0x7f00b4155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:35.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.936+0000 7f00bb200640 1 -- 172.21.15.69:0/1177565911 >> 172.21.15.69:0/1177565911 conn(0x7f00b4067b80 msgr2=0x7f00b415ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:35.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.938+0000 7f00bb200640 1 -- 172.21.15.69:0/1177565911 shutdown_connections 2024-04-24T08:10:35.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:35.938+0000 7f00bb200640 1 -- 172.21.15.69:0/1177565911 wait complete. 2024-04-24T08:10:35.974 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:36.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.143+0000 7f8a9fa00640 1 Processor -- start 2024-04-24T08:10:36.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.144+0000 7f8a9fa00640 1 -- start start 2024-04-24T08:10:36.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.145+0000 7f8a9fa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a980b3d30 0x7f8a980b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.145+0000 7f8a9fa00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8a9806e0f0 0x7f8a9806e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.145+0000 7f8a9fa00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a981669d0 con 0x7f8a98162c70 2024-04-24T08:10:36.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.145+0000 7f8a9fa00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8a98166cb0 con 0x7f8a980b3d30 2024-04-24T08:10:36.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.145+0000 7f8a9fa00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8a98166f90 con 0x7f8a9806e0f0 2024-04-24T08:10:36.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.146+0000 7f8a8fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a980b3d30 0x7f8a980b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.146+0000 7f8a8fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a980b3d30 0x7f8a980b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59406/0 (socket says 172.21.15.69:59406) 2024-04-24T08:10:36.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.146+0000 7f8a8fe00640 1 -- 172.21.15.69:0/1793700245 learned_addr learned my addr 172.21.15.69:0/1793700245 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:36.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.146+0000 7f8a9da00640 1 --2- 172.21.15.69:0/1793700245 >> v2:172.21.15.121:3300/0 conn(0x7f8a9806e0f0 0x7f8a9806e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.146+0000 7f8a8fe00640 1 -- 172.21.15.69:0/1793700245 >> v2:172.21.15.121:3300/0 conn(0x7f8a9806e0f0 msgr2=0x7f8a9806e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:36.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.146+0000 7f8a8fe00640 1 --2- 172.21.15.69:0/1793700245 >> v2:172.21.15.121:3300/0 conn(0x7f8a9806e0f0 0x7f8a9806e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.146+0000 7f8a8fe00640 1 -- 172.21.15.69:0/1793700245 >> v1:172.21.15.121:6789/0 conn(0x7f8a98162c70 legacy=0x7f8a980b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:36.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.146+0000 7f8a8fe00640 1 -- 172.21.15.69:0/1793700245 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a98167270 con 0x7f8a980b3d30 2024-04-24T08:10:36.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.147+0000 7f8a8fe00640 1 --2- 172.21.15.69:0/1793700245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a980b3d30 0x7f8a980b41c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8a980aff60 tx=0x7f8a8802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a445448a6e1b2394 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:36.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.147+0000 7f8a97e00640 1 -- 172.21.15.69:0/1793700245 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88037070 con 0x7f8a980b3d30 2024-04-24T08:10:36.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.148+0000 7f8a97e00640 1 -- 172.21.15.69:0/1793700245 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a88032440 con 0x7f8a980b3d30 2024-04-24T08:10:36.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.148+0000 7f8a97e00640 1 -- 172.21.15.69:0/1793700245 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a88032810 con 0x7f8a980b3d30 2024-04-24T08:10:36.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.148+0000 7f8a9fa00640 1 -- 172.21.15.69:0/1793700245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a980b3d30 msgr2=0x7f8a980b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.148+0000 7f8a9fa00640 1 --2- 172.21.15.69:0/1793700245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a980b3d30 0x7f8a980b41c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8a980aff60 tx=0x7f8a8802a230 comp rx=0 tx=0).stop 2024-04-24T08:10:36.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.148+0000 7f8a9fa00640 1 -- 172.21.15.69:0/1793700245 shutdown_connections 2024-04-24T08:10:36.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.148+0000 7f8a9fa00640 1 --2- 172.21.15.69:0/1793700245 >> v2:172.21.15.121:3300/0 conn(0x7f8a9806e0f0 0x7f8a9806e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.148+0000 7f8a9fa00640 1 --2- 172.21.15.69:0/1793700245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a980b3d30 0x7f8a980b41c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.148+0000 7f8a9fa00640 1 -- 172.21.15.69:0/1793700245 >> 172.21.15.69:0/1793700245 conn(0x7f8a98067d90 msgr2=0x7f8a9806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:36.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.148+0000 7f8a9fa00640 1 -- 172.21.15.69:0/1793700245 shutdown_connections 2024-04-24T08:10:36.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.148+0000 7f8a9fa00640 1 -- 172.21.15.69:0/1793700245 wait complete. 2024-04-24T08:10:36.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.150+0000 7f8a9fa00640 1 Processor -- start 2024-04-24T08:10:36.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.150+0000 7f8a9fa00640 1 -- start start 2024-04-24T08:10:36.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.151+0000 7f8a9fa00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9806e0f0 0x7f8a9815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.151+0000 7f8a9fa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a980b3d30 0x7f8a98144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.151+0000 7f8a9fa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a98162c70 0x7f8a98144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.151+0000 7f8a9fa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a980cfc40 con 0x7f8a98162c70 2024-04-24T08:10:36.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.151+0000 7f8a9fa00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8a98163050 con 0x7f8a9806e0f0 2024-04-24T08:10:36.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.151+0000 7f8a9fa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8a980cfdc0 con 0x7f8a980b3d30 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.151+0000 7f8a9da00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a98162c70 0x7f8a98144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.151+0000 7f8a9da00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a98162c70 0x7f8a98144570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59412/0 (socket says 172.21.15.69:59412) 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.151+0000 7f8a9da00640 1 -- 172.21.15.69:0/2122748548 learned_addr learned my addr 172.21.15.69:0/2122748548 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.152+0000 7f8a8fe00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a980b3d30 0x7f8a98144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.152+0000 7f8a9d000640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9806e0f0 0x7f8a9815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.152+0000 7f8a9da00640 1 -- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a980b3d30 msgr2=0x7f8a98144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.152+0000 7f8a9da00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a980b3d30 0x7f8a98144030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.152+0000 7f8a9da00640 1 -- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9806e0f0 msgr2=0x7f8a9815b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.152+0000 7f8a9da00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9806e0f0 0x7f8a9815b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.152+0000 7f8a9da00640 1 -- 172.21.15.69:0/2122748548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a98156760 con 0x7f8a98162c70 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.152+0000 7f8a8fe00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a980b3d30 0x7f8a98144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:36.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.152+0000 7f8a9da00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a98162c70 0x7f8a98144570 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000efc0 tx=0x7f8a9000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:36.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.152+0000 7f8a95600640 1 -- 172.21.15.69:0/2122748548 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90018020 con 0x7f8a98162c70 2024-04-24T08:10:36.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.153+0000 7f8a95600640 1 -- 172.21.15.69:0/2122748548 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a90014480 con 0x7f8a98162c70 2024-04-24T08:10:36.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.153+0000 7f8a95600640 1 -- 172.21.15.69:0/2122748548 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90014870 con 0x7f8a98162c70 2024-04-24T08:10:36.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.153+0000 7f8a9fa00640 1 -- 172.21.15.69:0/2122748548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a98165310 con 0x7f8a98162c70 2024-04-24T08:10:36.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.153+0000 7f8a9fa00640 1 -- 172.21.15.69:0/2122748548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a98156fc0 con 0x7f8a98162c70 2024-04-24T08:10:36.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.154+0000 7f8a9fa00640 1 -- 172.21.15.69:0/2122748548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a98156b10 con 0x7f8a98162c70 2024-04-24T08:10:36.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.155+0000 7f8a95600640 1 -- 172.21.15.69:0/2122748548 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8a90005d10 con 0x7f8a98162c70 2024-04-24T08:10:36.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.156+0000 7f8a95600640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a7406c8f0 0x7f8a7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.156+0000 7f8a95600640 1 -- 172.21.15.69:0/2122748548 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f8a9008c780 con 0x7f8a98162c70 2024-04-24T08:10:36.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.157+0000 7f8a9d000640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a7406c8f0 0x7f8a7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.158+0000 7f8a9d000640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a7406c8f0 0x7f8a7406edb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8a80006fd0 tx=0x7f8a80008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:36.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.161+0000 7f8a95600640 1 -- 172.21.15.69:0/2122748548 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a90022070 con 0x7f8a98162c70 2024-04-24T08:10:36.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.346+0000 7f8a9fa00640 1 -- 172.21.15.69:0/2122748548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a9806ece0 con 0x7f8a98162c70 2024-04-24T08:10:36.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.347+0000 7f8a95600640 1 -- 172.21.15.69:0/2122748548 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+16602 (secure 0 0 0) 0x7f8a9806ece0 con 0x7f8a98162c70 2024-04-24T08:10:36.349 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:36.349 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":43,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:10:35.400856+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:10:33.100132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-24T08:10:34.379302+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:36.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 -- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a7406c8f0 msgr2=0x7f8a7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a7406c8f0 0x7f8a7406edb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8a80006fd0 tx=0x7f8a80008040 comp rx=0 tx=0).stop 2024-04-24T08:10:36.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 -- 172.21.15.69:0/2122748548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a98162c70 msgr2=0x7f8a98144570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a98162c70 0x7f8a98144570 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000efc0 tx=0x7f8a9000c490 comp rx=0 tx=0).stop 2024-04-24T08:10:36.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 -- 172.21.15.69:0/2122748548 shutdown_connections 2024-04-24T08:10:36.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a7406c8f0 0x7f8a7406edb0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a98162c70 0x7f8a98144570 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a980b3d30 0x7f8a98144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 --2- 172.21.15.69:0/2122748548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a9806e0f0 0x7f8a9815b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 -- 172.21.15.69:0/2122748548 >> 172.21.15.69:0/2122748548 conn(0x7f8a98067d90 msgr2=0x7f8a9806ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:36.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 -- 172.21.15.69:0/2122748548 shutdown_connections 2024-04-24T08:10:36.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.350+0000 7f8a9fa00640 1 -- 172.21.15.69:0/2122748548 wait complete. 2024-04-24T08:10:36.369 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:36.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.559+0000 7f2ac6600640 1 Processor -- start 2024-04-24T08:10:36.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.559+0000 7f2ac6600640 1 -- start start 2024-04-24T08:10:36.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.560+0000 7f2ac6600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2ac0156c60 0x7f2ac015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.560+0000 7f2ac6600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2ac00b30f0 0x7f2ac015d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.560+0000 7f2ac6600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ac015d660 con 0x7f2ac00ad0d0 2024-04-24T08:10:36.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.560+0000 7f2ac6600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2ac006dc30 con 0x7f2ac0156c60 2024-04-24T08:10:36.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.560+0000 7f2ac6600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2ac006df10 con 0x7f2ac00b30f0 2024-04-24T08:10:36.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.561+0000 7f2ac5c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2ac00ad0d0 0x7f2ac01600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37320/0 (socket says 172.21.15.69:37320) 2024-04-24T08:10:36.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.561+0000 7f2ac5c00640 1 -- 172.21.15.69:0/1693744996 learned_addr learned my addr 172.21.15.69:0/1693744996 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:36.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.561+0000 7f2ac5200640 1 --2- 172.21.15.69:0/1693744996 >> v2:172.21.15.69:3300/0 conn(0x7f2ac0156c60 0x7f2ac015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.561+0000 7f2ab7e00640 1 --2- 172.21.15.69:0/1693744996 >> v2:172.21.15.121:3300/0 conn(0x7f2ac00b30f0 0x7f2ac015d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.561+0000 7f2abfe00640 1 -- 172.21.15.69:0/1693744996 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1236361424 0 0) 0x7f2ac015d660 con 0x7f2ac00ad0d0 2024-04-24T08:10:36.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.561+0000 7f2abfe00640 1 -- 172.21.15.69:0/1693744996 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa4003520 con 0x7f2ac00ad0d0 2024-04-24T08:10:36.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.561+0000 7f2ac5200640 1 -- 172.21.15.69:0/1693744996 >> v2:172.21.15.121:3300/0 conn(0x7f2ac00b30f0 msgr2=0x7f2ac015d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:36.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.561+0000 7f2ac5200640 1 --2- 172.21.15.69:0/1693744996 >> v2:172.21.15.121:3300/0 conn(0x7f2ac00b30f0 0x7f2ac015d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.561+0000 7f2ac5200640 1 -- 172.21.15.69:0/1693744996 >> v1:172.21.15.69:6789/0 conn(0x7f2ac00ad0d0 legacy=0x7f2ac01600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.562+0000 7f2ac5200640 1 -- 172.21.15.69:0/1693744996 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac006e1f0 con 0x7f2ac0156c60 2024-04-24T08:10:36.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.563+0000 7f2ac5200640 1 --2- 172.21.15.69:0/1693744996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac0156c60 0x7f2ac015cac0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2ac006d900 tx=0x7f2ab002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b7f665a0469486 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:36.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.563+0000 7f2abfe00640 1 -- 172.21.15.69:0/1693744996 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0037070 con 0x7f2ac0156c60 2024-04-24T08:10:36.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.563+0000 7f2abfe00640 1 -- 172.21.15.69:0/1693744996 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ab0032440 con 0x7f2ac0156c60 2024-04-24T08:10:36.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.563+0000 7f2abfe00640 1 -- 172.21.15.69:0/1693744996 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0032810 con 0x7f2ac0156c60 2024-04-24T08:10:36.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.564+0000 7f2ac6600640 1 -- 172.21.15.69:0/1693744996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac0156c60 msgr2=0x7f2ac015cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.564+0000 7f2ac6600640 1 --2- 172.21.15.69:0/1693744996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac0156c60 0x7f2ac015cac0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2ac006d900 tx=0x7f2ab002a1a0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.564+0000 7f2ac6600640 1 -- 172.21.15.69:0/1693744996 shutdown_connections 2024-04-24T08:10:36.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.564+0000 7f2ac6600640 1 --2- 172.21.15.69:0/1693744996 >> v2:172.21.15.121:3300/0 conn(0x7f2ac00b30f0 0x7f2ac015d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.564+0000 7f2ac6600640 1 --2- 172.21.15.69:0/1693744996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac0156c60 0x7f2ac015cac0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.564+0000 7f2ac6600640 1 -- 172.21.15.69:0/1693744996 >> 172.21.15.69:0/1693744996 conn(0x7f2ac00d0160 msgr2=0x7f2ac00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:36.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.564+0000 7f2ac6600640 1 -- 172.21.15.69:0/1693744996 shutdown_connections 2024-04-24T08:10:36.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.564+0000 7f2ac6600640 1 -- 172.21.15.69:0/1693744996 wait complete. 2024-04-24T08:10:36.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.566+0000 7f2ac6600640 1 Processor -- start 2024-04-24T08:10:36.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.566+0000 7f2ac6600640 1 -- start start 2024-04-24T08:10:36.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.567+0000 7f2ac6600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ac00ad0d0 0x7f2ac0158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.567+0000 7f2ac6600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac00b30f0 0x7f2ac00cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.567+0000 7f2ac6600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ac0156c60 0x7f2ac01f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.567+0000 7f2ac6600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac00ad540 con 0x7f2ac00b30f0 2024-04-24T08:10:36.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.567+0000 7f2ac6600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2ac015d940 con 0x7f2ac0156c60 2024-04-24T08:10:36.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.567+0000 7f2ac6600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2ac00ad6c0 con 0x7f2ac00ad0d0 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.567+0000 7f2ab7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac00b30f0 0x7f2ac00cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.567+0000 7f2ab7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac00b30f0 0x7f2ac00cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59432/0 (socket says 172.21.15.69:59432) 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.567+0000 7f2ab7e00640 1 -- 172.21.15.69:0/230095840 learned_addr learned my addr 172.21.15.69:0/230095840 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2ac5200640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ac00ad0d0 0x7f2ac0158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2ab7e00640 1 -- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ac00ad0d0 msgr2=0x7f2ac0158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2ac5c00640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ac0156c60 0x7f2ac01f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2ab7e00640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ac00ad0d0 0x7f2ac0158360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2ab7e00640 1 -- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ac0156c60 msgr2=0x7f2ac01f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2ab7e00640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ac0156c60 0x7f2ac01f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2ab7e00640 1 -- 172.21.15.69:0/230095840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ac00ca7d0 con 0x7f2ac00b30f0 2024-04-24T08:10:36.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2ab7e00640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac00b30f0 0x7f2ac00cb870 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800d7d0 tx=0x7f2aa800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:36.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2abd600640 1 -- 172.21.15.69:0/230095840 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa800b6c0 con 0x7f2ac00b30f0 2024-04-24T08:10:36.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2abd600640 1 -- 172.21.15.69:0/230095840 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa8019070 con 0x7f2ac00b30f0 2024-04-24T08:10:36.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2abd600640 1 -- 172.21.15.69:0/230095840 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa800ba80 con 0x7f2ac00b30f0 2024-04-24T08:10:36.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.568+0000 7f2ac6600640 1 -- 172.21.15.69:0/230095840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ac00caab0 con 0x7f2ac00b30f0 2024-04-24T08:10:36.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.569+0000 7f2ac6600640 1 -- 172.21.15.69:0/230095840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ac00cb350 con 0x7f2ac00b30f0 2024-04-24T08:10:36.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.570+0000 7f2ac6600640 1 -- 172.21.15.69:0/230095840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ac00cac80 con 0x7f2ac00b30f0 2024-04-24T08:10:36.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.571+0000 7f2abd600640 1 -- 172.21.15.69:0/230095840 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2aa800bc20 con 0x7f2ac00b30f0 2024-04-24T08:10:36.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.571+0000 7f2abd600640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a9c06c8f0 0x7f2a9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.572+0000 7f2abd600640 1 -- 172.21.15.69:0/230095840 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5709+0+0 (secure 0 0 0) 0x7f2aa808bfe0 con 0x7f2ac00b30f0 2024-04-24T08:10:36.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.572+0000 7f2ac5200640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a9c06c8f0 0x7f2a9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.573+0000 7f2ac5200640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a9c06c8f0 0x7f2a9c06edb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0034000 tx=0x7f2ab002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:36.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.576+0000 7f2abd600640 1 -- 172.21.15.69:0/230095840 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2aa8011340 con 0x7f2ac00b30f0 2024-04-24T08:10:36.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.761+0000 7f2ac6600640 1 -- 172.21.15.69:0/230095840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ac0075d50 con 0x7f2ac00b30f0 2024-04-24T08:10:36.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.764+0000 7f2abd600640 1 -- 172.21.15.69:0/230095840 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+16602 (secure 0 0 0) 0x7f2aa805a3d0 con 0x7f2ac00b30f0 2024-04-24T08:10:36.765 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:36.766 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":43,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:10:35.400856+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:10:33.100132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-24T08:10:34.379302+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:36.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 -- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a9c06c8f0 msgr2=0x7f2a9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a9c06c8f0 0x7f2a9c06edb0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0034000 tx=0x7f2ab002e000 comp rx=0 tx=0).stop 2024-04-24T08:10:36.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 -- 172.21.15.69:0/230095840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac00b30f0 msgr2=0x7f2ac00cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac00b30f0 0x7f2ac00cb870 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800d7d0 tx=0x7f2aa800dc90 comp rx=0 tx=0).stop 2024-04-24T08:10:36.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 -- 172.21.15.69:0/230095840 shutdown_connections 2024-04-24T08:10:36.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a9c06c8f0 0x7f2a9c06edb0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ac00b30f0 0x7f2ac00cb870 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ac0156c60 0x7f2ac01f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 --2- 172.21.15.69:0/230095840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2ac00ad0d0 0x7f2ac0158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 -- 172.21.15.69:0/230095840 >> 172.21.15.69:0/230095840 conn(0x7f2ac00d0160 msgr2=0x7f2ac00b3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:36.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 -- 172.21.15.69:0/230095840 shutdown_connections 2024-04-24T08:10:36.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.767+0000 7f2ac6600640 1 -- 172.21.15.69:0/230095840 wait complete. 2024-04-24T08:10:36.786 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T08:10:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.982+0000 7f6395000640 1 Processor -- start 2024-04-24T08:10:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.982+0000 7f6395000640 1 -- start start 2024-04-24T08:10:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.984+0000 7f6395000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6390071d70 0x7f639015ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.984+0000 7f6395000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6390160510 0x7f63900cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.984+0000 7f6395000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63900721d0 con 0x7f63901420a0 2024-04-24T08:10:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.984+0000 7f6395000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6390160960 con 0x7f6390160510 2024-04-24T08:10:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.984+0000 7f6395000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f63900cf1e0 con 0x7f6390071d70 2024-04-24T08:10:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.984+0000 7f638f400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f63901420a0 0x7f6390142480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37326/0 (socket says 172.21.15.69:37326) 2024-04-24T08:10:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.984+0000 7f638f400640 1 -- 172.21.15.69:0/3522993585 learned_addr learned my addr 172.21.15.69:0/3522993585 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.984+0000 7f638fe00640 1 --2- 172.21.15.69:0/3522993585 >> v2:172.21.15.121:3300/0 conn(0x7f6390160510 0x7f63900cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.984+0000 7f6386a00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6390071d70 0x7f639015ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.984+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3680844629 0 0) 0x7f6380001110 con 0x7f63901420a0 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6374003520 con 0x7f63901420a0 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 691995348 0 0) 0x7f6374003520 con 0x7f63901420a0 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6390071d70 msgr2=0x7f639015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6390071d70 0x7f639015ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 >> v2:172.21.15.121:3300/0 conn(0x7f6390160510 msgr2=0x7f63900cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 --2- 172.21.15.69:0/3522993585 >> v2:172.21.15.121:3300/0 conn(0x7f6390160510 0x7f63900cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6390072450 con 0x7f63901420a0 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f6380001840 con 0x7f63901420a0 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 >> v1:172.21.15.69:6789/0 conn(0x7f63901420a0 legacy=0x7f6390142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63740039b0 0x7f6374003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63740043a0 0x7f6374004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6374004fa0 0x7f637400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6390160960 con 0x7f6374004fa0 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f63900cf1e0 con 0x7f63740043a0 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f637400bf30 con 0x7f63740039b0 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f6386a00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6390071d70 0x7f639015ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.985+0000 7f638fe00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6374004fa0 0x7f637400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.986+0000 7f6386a00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63740043a0 0x7f6374004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.986+0000 7f638fe00640 1 -- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63740039b0 msgr2=0x7f6374003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.986+0000 7f638fe00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63740039b0 0x7f6374003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.986+0000 7f638fe00640 1 -- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63740043a0 msgr2=0x7f6374004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.986+0000 7f638fe00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63740043a0 0x7f6374004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.986+0000 7f638fe00640 1 -- 172.21.15.69:0/3522993585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6374003520 con 0x7f6374004fa0 2024-04-24T08:10:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.986+0000 7f6386a00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63740043a0 0x7f6374004880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:36.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.987+0000 7f638fe00640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6374004fa0 0x7f637400b9c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6388009fc0 tx=0x7f6374005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398d941c15b96c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:36.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.987+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6388009c10 con 0x7f6374004fa0 2024-04-24T08:10:36.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.987+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6388036070 con 0x7f6374004fa0 2024-04-24T08:10:36.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.987+0000 7f638ea00640 1 -- 172.21.15.69:0/3522993585 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638802d070 con 0x7f6374004fa0 2024-04-24T08:10:36.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.988+0000 7f6395000640 1 -- 172.21.15.69:0/3522993585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6374004fa0 msgr2=0x7f637400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.988+0000 7f6395000640 1 --2- 172.21.15.69:0/3522993585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6374004fa0 0x7f637400b9c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6388009fc0 tx=0x7f6374005830 comp rx=0 tx=0).stop 2024-04-24T08:10:36.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.988+0000 7f638fe00640 1 -- 172.21.15.69:0/3522993585 reap_dead start 2024-04-24T08:10:36.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.988+0000 7f6395000640 1 -- 172.21.15.69:0/3522993585 shutdown_connections 2024-04-24T08:10:36.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.988+0000 7f6395000640 1 -- 172.21.15.69:0/3522993585 >> 172.21.15.69:0/3522993585 conn(0x7f6390067b80 msgr2=0x7f639006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:36.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.988+0000 7f6395000640 1 -- 172.21.15.69:0/3522993585 shutdown_connections 2024-04-24T08:10:36.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.988+0000 7f6395000640 1 -- 172.21.15.69:0/3522993585 wait complete. 2024-04-24T08:10:36.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.989+0000 7f6395000640 1 Processor -- start 2024-04-24T08:10:36.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.990+0000 7f6395000640 1 -- start start 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.990+0000 7f6395000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6374004fa0 0x7f63900b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.990+0000 7f6395000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f639015b690 0x7f63901e78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.990+0000 7f6395000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f639015bad0 0x7f63901e7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.990+0000 7f6395000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6390072cf0 con 0x7f639015b690 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.990+0000 7f6395000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6390072a10 con 0x7f6374004fa0 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.990+0000 7f6395000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f639015ffd0 con 0x7f639015bad0 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.991+0000 7f6386a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f639015b690 0x7f63901e78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.991+0000 7f6386a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f639015b690 0x7f63901e78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59446/0 (socket says 172.21.15.69:59446) 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.991+0000 7f6386a00640 1 -- 172.21.15.69:0/3951622468 learned_addr learned my addr 172.21.15.69:0/3951622468 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.991+0000 7f638f400640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6374004fa0 0x7f63900b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.991+0000 7f6386a00640 1 -- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f639015bad0 msgr2=0x7f63901e7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.991+0000 7f6386a00640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f639015bad0 0x7f63901e7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.991+0000 7f6386a00640 1 -- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6374004fa0 msgr2=0x7f63900b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:36.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.991+0000 7f6386a00640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6374004fa0 0x7f63900b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:36.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.991+0000 7f6386a00640 1 -- 172.21.15.69:0/3951622468 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63901e70c0 con 0x7f639015b690 2024-04-24T08:10:36.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.991+0000 7f6386a00640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f639015b690 0x7f63901e78a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f63780105e0 tx=0x7f6378010aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:36.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.992+0000 7f6387e00640 1 -- 172.21.15.69:0/3951622468 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637801a020 con 0x7f639015b690 2024-04-24T08:10:36.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.992+0000 7f6387e00640 1 -- 172.21.15.69:0/3951622468 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6378015880 con 0x7f639015b690 2024-04-24T08:10:36.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.992+0000 7f6387e00640 1 -- 172.21.15.69:0/3951622468 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f637800f040 con 0x7f639015b690 2024-04-24T08:10:36.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.992+0000 7f6395000640 1 -- 172.21.15.69:0/3951622468 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63901e6040 con 0x7f639015b690 2024-04-24T08:10:36.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.992+0000 7f6395000640 1 -- 172.21.15.69:0/3951622468 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63901e6600 con 0x7f639015b690 2024-04-24T08:10:36.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.993+0000 7f6395000640 1 -- 172.21.15.69:0/3951622468 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6390155790 con 0x7f639015b690 2024-04-24T08:10:36.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.994+0000 7f6387e00640 1 -- 172.21.15.69:0/3951622468 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f637801d020 con 0x7f639015b690 2024-04-24T08:10:36.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.995+0000 7f6387e00640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f636c06c8f0 0x7f636c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:36.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.995+0000 7f6387e00640 1 -- 172.21.15.69:0/3951622468 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f637808c340 con 0x7f639015b690 2024-04-24T08:10:36.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.996+0000 7f638f400640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f636c06c8f0 0x7f636c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:36.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.997+0000 7f638f400640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f636c06c8f0 0x7f636c06edb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6380000ff0 tx=0x7f6380001c90 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:37.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:36.999+0000 7f6387e00640 1 -- 172.21.15.69:0/3951622468 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6378024020 con 0x7f639015b690 2024-04-24T08:10:37.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.243+0000 7f6395000640 1 -- 172.21.15.69:0/3951622468 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f6390075d50 con 0x7f639015b690 2024-04-24T08:10:37.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.964+0000 7f6387e00640 1 -- 172.21.15.69:0/3951622468 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f6378015c70 con 0x7f639015b690 2024-04-24T08:10:37.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.968+0000 7f6395000640 1 -- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f636c06c8f0 msgr2=0x7f636c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:37.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.968+0000 7f6395000640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f636c06c8f0 0x7f636c06edb0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6380000ff0 tx=0x7f6380001c90 comp rx=0 tx=0).stop 2024-04-24T08:10:37.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.968+0000 7f6395000640 1 -- 172.21.15.69:0/3951622468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f639015b690 msgr2=0x7f63901e78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:37.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.968+0000 7f6395000640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f639015b690 0x7f63901e78a0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f63780105e0 tx=0x7f6378010aa0 comp rx=0 tx=0).stop 2024-04-24T08:10:37.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.969+0000 7f6395000640 1 -- 172.21.15.69:0/3951622468 shutdown_connections 2024-04-24T08:10:37.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.969+0000 7f6395000640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f636c06c8f0 0x7f636c06edb0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:37.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.969+0000 7f6395000640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f639015b690 0x7f63901e78a0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:37.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.969+0000 7f6395000640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f639015bad0 0x7f63901e7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:37.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.969+0000 7f6395000640 1 --2- 172.21.15.69:0/3951622468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6374004fa0 0x7f63900b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:37.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.969+0000 7f6395000640 1 -- 172.21.15.69:0/3951622468 >> 172.21.15.69:0/3951622468 conn(0x7f6390067b80 msgr2=0x7f639006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:37.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.969+0000 7f6395000640 1 -- 172.21.15.69:0/3951622468 shutdown_connections 2024-04-24T08:10:37.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:37.969+0000 7f6395000640 1 -- 172.21.15.69:0/3951622468 wait complete. 2024-04-24T08:10:37.995 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-24T08:10:37.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T08:10:38.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.200+0000 7fe9c9c00640 1 Processor -- start 2024-04-24T08:10:38.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.200+0000 7fe9c9c00640 1 -- start start 2024-04-24T08:10:38.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.202+0000 7fe9c9c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe9c415d030 0x7fe9c4163b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:38.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.202+0000 7fe9c9c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9c41640a0 con 0x7fe9c4072220 2024-04-24T08:10:38.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.202+0000 7fe9c9c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9c41674f0 con 0x7fe9c4071d70 2024-04-24T08:10:38.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.202+0000 7fe9c9c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe9c40b39e0 con 0x7fe9c415d030 2024-04-24T08:10:38.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.202+0000 7fe9c3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe9c415d030 0x7fe9c4163b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:38.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.202+0000 7fe9c9200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe9c4072220 0x7fe9c4166de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37336/0 (socket says 172.21.15.69:37336) 2024-04-24T08:10:38.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.202+0000 7fe9c9200640 1 -- 172.21.15.69:0/523545880 learned_addr learned my addr 172.21.15.69:0/523545880 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:38.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.202+0000 7fe9c3e00640 1 -- 172.21.15.69:0/523545880 >> v1:172.21.15.121:6789/0 conn(0x7fe9c4071d70 legacy=0x7fe9c41666d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:38.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.203+0000 7fe9c3e00640 1 -- 172.21.15.69:0/523545880 >> v1:172.21.15.69:6789/0 conn(0x7fe9c4072220 legacy=0x7fe9c4166de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:38.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.203+0000 7fe9c3e00640 1 -- 172.21.15.69:0/523545880 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c4072690 con 0x7fe9c415d030 2024-04-24T08:10:38.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.204+0000 7fe9c3e00640 1 --2- 172.21.15.69:0/523545880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 0x7fe9c4163b60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40097a0 tx=0x7fe9b402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cc4c4d821c0cb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:38.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.204+0000 7fe9c2a00640 1 -- 172.21.15.69:0/523545880 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4037070 con 0x7fe9c415d030 2024-04-24T08:10:38.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.204+0000 7fe9c2a00640 1 -- 172.21.15.69:0/523545880 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9b4032440 con 0x7fe9c415d030 2024-04-24T08:10:38.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.204+0000 7fe9c2a00640 1 -- 172.21.15.69:0/523545880 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4032810 con 0x7fe9c415d030 2024-04-24T08:10:38.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.204+0000 7fe9c9c00640 1 -- 172.21.15.69:0/523545880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 msgr2=0x7fe9c4163b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:38.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.204+0000 7fe9c9c00640 1 --2- 172.21.15.69:0/523545880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 0x7fe9c4163b60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe9b40097a0 tx=0x7fe9b402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:10:38.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.205+0000 7fe9c9c00640 1 -- 172.21.15.69:0/523545880 shutdown_connections 2024-04-24T08:10:38.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.205+0000 7fe9c9c00640 1 --2- 172.21.15.69:0/523545880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 0x7fe9c4163b60 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:38.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.205+0000 7fe9c9c00640 1 -- 172.21.15.69:0/523545880 >> 172.21.15.69:0/523545880 conn(0x7fe9c4067b80 msgr2=0x7fe9c406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:38.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.205+0000 7fe9c9c00640 1 -- 172.21.15.69:0/523545880 shutdown_connections 2024-04-24T08:10:38.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.205+0000 7fe9c9c00640 1 -- 172.21.15.69:0/523545880 wait complete. 2024-04-24T08:10:38.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.207+0000 7fe9c9c00640 1 Processor -- start 2024-04-24T08:10:38.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.207+0000 7fe9c9c00640 1 -- start start 2024-04-24T08:10:38.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.208+0000 7fe9c9c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c4071d70 0x7fe9c415f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:38.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.208+0000 7fe9c9c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9c4072220 0x7fe9c41fb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.208+0000 7fe9c9c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 0x7fe9c41fb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.208+0000 7fe9c9c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c40b4100 con 0x7fe9c415d030 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.208+0000 7fe9c9c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c40b3e20 con 0x7fe9c4071d70 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.208+0000 7fe9c9c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe9c40b43e0 con 0x7fe9c4072220 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.208+0000 7fe9c9200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 0x7fe9c41fb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.209+0000 7fe9c9200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 0x7fe9c41fb880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59474/0 (socket says 172.21.15.69:59474) 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.209+0000 7fe9c9200640 1 -- 172.21.15.69:0/2769408160 learned_addr learned my addr 172.21.15.69:0/2769408160 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.209+0000 7fe9c3e00640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c4071d70 0x7fe9c415f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.209+0000 7fe9c3400640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9c4072220 0x7fe9c41fb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.209+0000 7fe9c9200640 1 -- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9c4072220 msgr2=0x7fe9c41fb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.209+0000 7fe9c9200640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9c4072220 0x7fe9c41fb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.209+0000 7fe9c9200640 1 -- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c4071d70 msgr2=0x7fe9c415f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.209+0000 7fe9c9200640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c4071d70 0x7fe9c415f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.209+0000 7fe9c9200640 1 -- 172.21.15.69:0/2769408160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c4159d60 con 0x7fe9c415d030 2024-04-24T08:10:38.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.209+0000 7fe9c9200640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 0x7fe9c41fb880 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00cbf0 tx=0x7fe9bc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:38.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.210+0000 7fe9bbe00640 1 -- 172.21.15.69:0/2769408160 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc017070 con 0x7fe9c415d030 2024-04-24T08:10:38.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.210+0000 7fe9bbe00640 1 -- 172.21.15.69:0/2769408160 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9bc00ad40 con 0x7fe9c415d030 2024-04-24T08:10:38.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.210+0000 7fe9bbe00640 1 -- 172.21.15.69:0/2769408160 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc012600 con 0x7fe9c415d030 2024-04-24T08:10:38.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.210+0000 7fe9c9c00640 1 -- 172.21.15.69:0/2769408160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c415a040 con 0x7fe9c415d030 2024-04-24T08:10:38.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.210+0000 7fe9c9c00640 1 -- 172.21.15.69:0/2769408160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c415a600 con 0x7fe9c415d030 2024-04-24T08:10:38.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.211+0000 7fe9c9c00640 1 -- 172.21.15.69:0/2769408160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c40b5610 con 0x7fe9c415d030 2024-04-24T08:10:38.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.212+0000 7fe9bbe00640 1 -- 172.21.15.69:0/2769408160 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe9bc01a070 con 0x7fe9c415d030 2024-04-24T08:10:38.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.213+0000 7fe9bbe00640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe9a406c8f0 0x7fe9a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:38.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.213+0000 7fe9bbe00640 1 -- 172.21.15.69:0/2769408160 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fe9bc08b7c0 con 0x7fe9c415d030 2024-04-24T08:10:38.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.214+0000 7fe9c3e00640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe9a406c8f0 0x7fe9a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:38.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.215+0000 7fe9c3e00640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe9a406c8f0 0x7fe9a406edb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4034000 tx=0x7fe9b402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:38.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.217+0000 7fe9bbe00640 1 -- 172.21.15.69:0/2769408160 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe9c40b5610 con 0x7fe9c415d030 2024-04-24T08:10:38.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.455+0000 7fe9c9c00640 1 -- 172.21.15.69:0/2769408160 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fe9c40b3330 con 0x7fe9c415d030 2024-04-24T08:10:38.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.982+0000 7fe9bbe00640 1 -- 172.21.15.69:0/2769408160 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 122+0+0 (secure 0 0 0) 0x7fe9c41640a0 con 0x7fe9c415d030 2024-04-24T08:10:38.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 -- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe9a406c8f0 msgr2=0x7fe9a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe9a406c8f0 0x7fe9a406edb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4034000 tx=0x7fe9b402e000 comp rx=0 tx=0).stop 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 -- 172.21.15.69:0/2769408160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 msgr2=0x7fe9c41fb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 0x7fe9c41fb880 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00cbf0 tx=0x7fe9bc00a430 comp rx=0 tx=0).stop 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 -- 172.21.15.69:0/2769408160 shutdown_connections 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe9a406c8f0 0x7fe9a406edb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe9c415d030 0x7fe9c41fb880 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe9c4072220 0x7fe9c41fb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 --2- 172.21.15.69:0/2769408160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9c4071d70 0x7fe9c415f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 -- 172.21.15.69:0/2769408160 >> 172.21.15.69:0/2769408160 conn(0x7fe9c4067b80 msgr2=0x7fe9c4166b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 -- 172.21.15.69:0/2769408160 shutdown_connections 2024-04-24T08:10:38.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:38.986+0000 7fe9c9c00640 1 -- 172.21.15.69:0/2769408160 wait complete. 2024-04-24T08:10:39.009 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T08:10:39.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:39.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.207+0000 7f5da4800640 1 Processor -- start 2024-04-24T08:10:39.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.208+0000 7f5da4800640 1 -- start start 2024-04-24T08:10:39.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.209+0000 7f5da4800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d9c15b490 0x7f5d9c0b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.209+0000 7f5da4800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d9c0b46e0 con 0x7f5d9c06f2d0 2024-04-24T08:10:39.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.209+0000 7f5da4800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d9c0d0960 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.209+0000 7f5da4800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5d9c0d0720 con 0x7f5d9c15b490 2024-04-24T08:10:39.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.209+0000 7f5da1e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d9c15b490 0x7f5d9c0b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:39.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.209+0000 7f5da1400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5d9c06f2d0 0x7f5d9c0cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49256/0 (socket says 172.21.15.69:49256) 2024-04-24T08:10:39.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.209+0000 7f5da1400640 1 -- 172.21.15.69:0/3668841366 learned_addr learned my addr 172.21.15.69:0/3668841366 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:39.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.210+0000 7f5da1e00640 1 -- 172.21.15.69:0/3668841366 >> v1:172.21.15.121:6789/0 conn(0x7f5d9c0d02f0 legacy=0x7f5d9c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:39.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.210+0000 7f5da1e00640 1 -- 172.21.15.69:0/3668841366 >> v1:172.21.15.69:6789/0 conn(0x7f5d9c06f2d0 legacy=0x7f5d9c0cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.210+0000 7f5da1e00640 1 -- 172.21.15.69:0/3668841366 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d9c0b4980 con 0x7f5d9c15b490 2024-04-24T08:10:39.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.211+0000 7f5da1e00640 1 --2- 172.21.15.69:0/3668841366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c15b490 0x7f5d9c0b5180 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5d900097a0 tx=0x7f5d9002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac041916520881a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:39.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.211+0000 7f5da0a00640 1 -- 172.21.15.69:0/3668841366 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d90037070 con 0x7f5d9c15b490 2024-04-24T08:10:39.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.211+0000 7f5da0a00640 1 -- 172.21.15.69:0/3668841366 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d90032440 con 0x7f5d9c15b490 2024-04-24T08:10:39.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.212+0000 7f5da0a00640 1 -- 172.21.15.69:0/3668841366 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d90032810 con 0x7f5d9c15b490 2024-04-24T08:10:39.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.212+0000 7f5da4800640 1 -- 172.21.15.69:0/3668841366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c15b490 msgr2=0x7f5d9c0b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.212+0000 7f5da4800640 1 --2- 172.21.15.69:0/3668841366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c15b490 0x7f5d9c0b5180 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5d900097a0 tx=0x7f5d9002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.212+0000 7f5da4800640 1 -- 172.21.15.69:0/3668841366 shutdown_connections 2024-04-24T08:10:39.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.212+0000 7f5da4800640 1 --2- 172.21.15.69:0/3668841366 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c15b490 0x7f5d9c0b5180 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.212+0000 7f5da4800640 1 -- 172.21.15.69:0/3668841366 >> 172.21.15.69:0/3668841366 conn(0x7f5d9c067d90 msgr2=0x7f5d9c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:39.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.212+0000 7f5da4800640 1 -- 172.21.15.69:0/3668841366 shutdown_connections 2024-04-24T08:10:39.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.212+0000 7f5da4800640 1 -- 172.21.15.69:0/3668841366 wait complete. 2024-04-24T08:10:39.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.214+0000 7f5da4800640 1 Processor -- start 2024-04-24T08:10:39.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.214+0000 7f5da4800640 1 -- start start 2024-04-24T08:10:39.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.215+0000 7f5da4800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d9c06f2d0 0x7f5d9c14e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.215+0000 7f5da4800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c0d02f0 0x7f5d9c14afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.215+0000 7f5da4800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d9c15b490 0x7f5d9c158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.215+0000 7f5da4800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9c15b870 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.215+0000 7f5da4800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5d9c0d0c40 con 0x7f5d9c15b490 2024-04-24T08:10:39.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.215+0000 7f5da4800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5d9c0b56c0 con 0x7f5d9c06f2d0 2024-04-24T08:10:39.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.215+0000 7f5da1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c0d02f0 0x7f5d9c14afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:39.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.215+0000 7f5da1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c0d02f0 0x7f5d9c14afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57820/0 (socket says 172.21.15.69:57820) 2024-04-24T08:10:39.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.215+0000 7f5da1400640 1 -- 172.21.15.69:0/3313639067 learned_addr learned my addr 172.21.15.69:0/3313639067 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da2800640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d9c15b490 0x7f5d9c158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da1e00640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d9c06f2d0 0x7f5d9c14e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da1400640 1 -- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d9c06f2d0 msgr2=0x7f5d9c14e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da1400640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d9c06f2d0 0x7f5d9c14e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da1400640 1 -- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d9c15b490 msgr2=0x7f5d9c158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da1400640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d9c15b490 0x7f5d9c158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da1400640 1 -- 172.21.15.69:0/3313639067 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d9c14b520 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da2800640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d9c15b490 0x7f5d9c158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da1400640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c0d02f0 0x7f5d9c14afe0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00c400 tx=0x7f5d8c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5d96000640 1 -- 172.21.15.69:0/3313639067 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c017070 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5d96000640 1 -- 172.21.15.69:0/3313639067 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d8c005ce0 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5d96000640 1 -- 172.21.15.69:0/3313639067 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c007310 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da4800640 1 -- 172.21.15.69:0/3313639067 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d9c14f940 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.216+0000 7f5da4800640 1 -- 172.21.15.69:0/3313639067 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d9c14ff00 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.218+0000 7f5da4800640 1 -- 172.21.15.69:0/3313639067 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d9c068a80 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.218+0000 7f5d96000640 1 -- 172.21.15.69:0/3313639067 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d8c01a070 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.219+0000 7f5d96000640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d8006c8f0 0x7f5d8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.219+0000 7f5d96000640 1 -- 172.21.15.69:0/3313639067 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f5d8c08afa0 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.220+0000 7f5da1e00640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d8006c8f0 0x7f5d8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:39.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.221+0000 7f5da1e00640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d8006c8f0 0x7f5d8006edb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5d90034000 tx=0x7f5d9002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:39.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.224+0000 7f5d96000640 1 -- 172.21.15.69:0/3313639067 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d8c059330 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.444+0000 7f5da4800640 1 -- 172.21.15.69:0/3313639067 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d9c0750e0 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.446+0000 7f5d96000640 1 -- 172.21.15.69:0/3313639067 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5184 (secure 0 0 0) 0x7f5d8c058c30 con 0x7f5d9c0d02f0 2024-04-24T08:10:39.447 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:39.447 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/376722301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":376722301},{"type":"v1","addr":"172.21.15.69:6835","nonce":376722301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4990,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1327624292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1327624292},{"type":"v1","addr":"172.21.15.121:6836","nonce":1327624292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/3244002416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3244002416},{"type":"v1","addr":"172.21.15.69:6834","nonce":3244002416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:10:35.400788+0000","modified":"2024-04-24T08:10:38.457641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5793},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5793":{"gid":5793,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/3424934990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3424934990},{"type":"v1","addr":"172.21.15.121:6837","nonce":3424934990}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5793,"qdb_cluster":[5793]},"id":3}]} 2024-04-24T08:10:39.448 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-04-24T08:10:39.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.448+0000 7f5da4800640 1 -- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d8006c8f0 msgr2=0x7f5d8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.448+0000 7f5da4800640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d8006c8f0 0x7f5d8006edb0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5d90034000 tx=0x7f5d9002e000 comp rx=0 tx=0).stop 2024-04-24T08:10:39.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.448+0000 7f5da4800640 1 -- 172.21.15.69:0/3313639067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c0d02f0 msgr2=0x7f5d9c14afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.448+0000 7f5da4800640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c0d02f0 0x7f5d9c14afe0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00c400 tx=0x7f5d8c00c8c0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.449+0000 7f5da4800640 1 -- 172.21.15.69:0/3313639067 shutdown_connections 2024-04-24T08:10:39.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.449+0000 7f5da4800640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d8006c8f0 0x7f5d8006edb0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.449+0000 7f5da4800640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d9c0d02f0 0x7f5d9c14afe0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.449+0000 7f5da4800640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d9c15b490 0x7f5d9c158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.449+0000 7f5da4800640 1 --2- 172.21.15.69:0/3313639067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d9c06f2d0 0x7f5d9c14e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.449+0000 7f5da4800640 1 -- 172.21.15.69:0/3313639067 >> 172.21.15.69:0/3313639067 conn(0x7f5d9c067d90 msgr2=0x7f5d9c0b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:39.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.449+0000 7f5da4800640 1 -- 172.21.15.69:0/3313639067 shutdown_connections 2024-04-24T08:10:39.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.449+0000 7f5da4800640 1 -- 172.21.15.69:0/3313639067 wait complete. 2024-04-24T08:10:39.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:39.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.654+0000 7f6760e00640 1 Processor -- start 2024-04-24T08:10:39.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.654+0000 7f6760e00640 1 -- start start 2024-04-24T08:10:39.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.655+0000 7f6760e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f675c15d1d0 0x7f675c0ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.655+0000 7f6760e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 0x7f675c0ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.655+0000 7f6760e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f675c15d660 con 0x7f675c156c60 2024-04-24T08:10:39.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.655+0000 7f6760e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f675c06dc30 con 0x7f675c15d1d0 2024-04-24T08:10:39.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.655+0000 7f6760e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f675c06df10 con 0x7f675c0b30f0 2024-04-24T08:10:39.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.656+0000 7f675aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f675c15d1d0 0x7f675c0ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:39.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.656+0000 7f675b400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f675c156c60 0x7f675c15cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49266/0 (socket says 172.21.15.69:49266) 2024-04-24T08:10:39.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.656+0000 7f675b400640 1 -- 172.21.15.69:0/1101544287 learned_addr learned my addr 172.21.15.69:0/1101544287 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:39.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.656+0000 7f675a000640 1 -- 172.21.15.69:0/1101544287 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3963299685 0 0) 0x7f675c15d660 con 0x7f675c156c60 2024-04-24T08:10:39.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.656+0000 7f675be00640 1 --2- 172.21.15.69:0/1101544287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 0x7f675c0ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:39.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.656+0000 7f675a000640 1 -- 172.21.15.69:0/1101544287 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6740003520 con 0x7f675c156c60 2024-04-24T08:10:39.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.656+0000 7f675aa00640 1 -- 172.21.15.69:0/1101544287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 msgr2=0x7f675c0ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:39.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.656+0000 7f675aa00640 1 --2- 172.21.15.69:0/1101544287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 0x7f675c0ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.656+0000 7f675aa00640 1 -- 172.21.15.69:0/1101544287 >> v1:172.21.15.69:6789/0 conn(0x7f675c156c60 legacy=0x7f675c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.656+0000 7f675aa00640 1 -- 172.21.15.69:0/1101544287 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675c06e1f0 con 0x7f675c15d1d0 2024-04-24T08:10:39.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.657+0000 7f675aa00640 1 --2- 172.21.15.69:0/1101544287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c15d1d0 0x7f675c0ad0d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f67480097a0 tx=0x7f674802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c0ecd7faf9b18c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f675a000640 1 -- 172.21.15.69:0/1101544287 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748037070 con 0x7f675c15d1d0 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f675a000640 1 -- 172.21.15.69:0/1101544287 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6748032440 con 0x7f675c15d1d0 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f675a000640 1 -- 172.21.15.69:0/1101544287 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748032810 con 0x7f675c15d1d0 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f6760e00640 1 -- 172.21.15.69:0/1101544287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c15d1d0 msgr2=0x7f675c0ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f6760e00640 1 --2- 172.21.15.69:0/1101544287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c15d1d0 0x7f675c0ad0d0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f67480097a0 tx=0x7f674802a180 comp rx=0 tx=0).stop 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f6760e00640 1 -- 172.21.15.69:0/1101544287 shutdown_connections 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f6760e00640 1 --2- 172.21.15.69:0/1101544287 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 0x7f675c0ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f6760e00640 1 --2- 172.21.15.69:0/1101544287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c15d1d0 0x7f675c0ad0d0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f6760e00640 1 -- 172.21.15.69:0/1101544287 >> 172.21.15.69:0/1101544287 conn(0x7f675c0d0160 msgr2=0x7f675c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f6760e00640 1 -- 172.21.15.69:0/1101544287 shutdown_connections 2024-04-24T08:10:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.658+0000 7f6760e00640 1 -- 172.21.15.69:0/1101544287 wait complete. 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.660+0000 7f6760e00640 1 Processor -- start 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.660+0000 7f6760e00640 1 -- start start 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.661+0000 7f6760e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 0x7f675c155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.661+0000 7f6760e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c156c60 0x7f675c0c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.661+0000 7f6760e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c15b0b0 0x7f675c0ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.661+0000 7f6760e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f675c06e710 con 0x7f675c156c60 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.661+0000 7f6760e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f675c15d940 con 0x7f675c15b0b0 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.661+0000 7f6760e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f675c06e890 con 0x7f675c0b30f0 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.661+0000 7f675aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c156c60 0x7f675c0c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.662+0000 7f675aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c156c60 0x7f675c0c9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57838/0 (socket says 172.21.15.69:57838) 2024-04-24T08:10:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.662+0000 7f675aa00640 1 -- 172.21.15.69:0/2866203326 learned_addr learned my addr 172.21.15.69:0/2866203326 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.662+0000 7f675b400640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 0x7f675c155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.662+0000 7f675aa00640 1 -- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 msgr2=0x7f675c155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.662+0000 7f675aa00640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 0x7f675c155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.662+0000 7f675aa00640 1 -- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c15b0b0 msgr2=0x7f675c0ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.662+0000 7f675aa00640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c15b0b0 0x7f675c0ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.662+0000 7f675aa00640 1 -- 172.21.15.69:0/2866203326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f675c0b3d30 con 0x7f675c156c60 2024-04-24T08:10:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.662+0000 7f675b400640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 0x7f675c155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.662+0000 7f675aa00640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c156c60 0x7f675c0c9f30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6748035040 tx=0x7f67480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.663+0000 7f6753400640 1 -- 172.21.15.69:0/2866203326 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6748037040 con 0x7f675c156c60 2024-04-24T08:10:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.663+0000 7f6753400640 1 -- 172.21.15.69:0/2866203326 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6748030500 con 0x7f675c156c60 2024-04-24T08:10:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.663+0000 7f6753400640 1 -- 172.21.15.69:0/2866203326 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67480308d0 con 0x7f675c156c60 2024-04-24T08:10:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.663+0000 7f6760e00640 1 -- 172.21.15.69:0/2866203326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f675c072d20 con 0x7f675c156c60 2024-04-24T08:10:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.663+0000 7f6760e00640 1 -- 172.21.15.69:0/2866203326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f675c0732e0 con 0x7f675c156c60 2024-04-24T08:10:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.664+0000 7f6760e00640 1 -- 172.21.15.69:0/2866203326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f675c075d50 con 0x7f675c156c60 2024-04-24T08:10:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.665+0000 7f6753400640 1 -- 172.21.15.69:0/2866203326 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6748044020 con 0x7f675c156c60 2024-04-24T08:10:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.666+0000 7f6753400640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f673c06c8f0 0x7f673c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.666+0000 7f6753400640 1 -- 172.21.15.69:0/2866203326 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f67480abfb0 con 0x7f675c156c60 2024-04-24T08:10:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.667+0000 7f675b400640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f673c06c8f0 0x7f673c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.668+0000 7f675b400640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f673c06c8f0 0x7f673c06edb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f674c005880 tx=0x7f674c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:39.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.670+0000 7f6753400640 1 -- 172.21.15.69:0/2866203326 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f674807a2b0 con 0x7f675c156c60 2024-04-24T08:10:39.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.871+0000 7f6760e00640 1 -- 172.21.15.69:0/2866203326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f675c146080 con 0x7f675c156c60 2024-04-24T08:10:39.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.872+0000 7f6753400640 1 -- 172.21.15.69:0/2866203326 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+16589 (secure 0 0 0) 0x7f6748079bb0 con 0x7f675c156c60 2024-04-24T08:10:39.874 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:39.874 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":44,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:10:36.952304+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:10:33.100132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-24T08:10:34.379302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:39.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 -- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f673c06c8f0 msgr2=0x7f673c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f673c06c8f0 0x7f673c06edb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f674c005880 tx=0x7f674c009040 comp rx=0 tx=0).stop 2024-04-24T08:10:39.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 -- 172.21.15.69:0/2866203326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c156c60 msgr2=0x7f675c0c9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:39.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c156c60 0x7f675c0c9f30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6748035040 tx=0x7f67480046c0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 -- 172.21.15.69:0/2866203326 shutdown_connections 2024-04-24T08:10:39.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f673c06c8f0 0x7f673c06edb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f675c156c60 0x7f675c0c9f30 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f675c15b0b0 0x7f675c0ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 --2- 172.21.15.69:0/2866203326 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f675c0b30f0 0x7f675c155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:39.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 -- 172.21.15.69:0/2866203326 >> 172.21.15.69:0/2866203326 conn(0x7f675c0d0160 msgr2=0x7f675c0b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:39.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 -- 172.21.15.69:0/2866203326 shutdown_connections 2024-04-24T08:10:39.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:39.875+0000 7f6760e00640 1 -- 172.21.15.69:0/2866203326 wait complete. 2024-04-24T08:10:39.896 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T08:10:39.896 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:10:40.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.094+0000 7f4d93200640 1 Processor -- start 2024-04-24T08:10:40.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.094+0000 7f4d93200640 1 -- start start 2024-04-24T08:10:40.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.095+0000 7f4d93200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4d941523f0 0x7f4d9406db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.095+0000 7f4d93200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d94072e20 0x7f4d94073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.095+0000 7f4d93200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4d940737a0 0x7f4d9415ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.095+0000 7f4d93200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4d940aeac0 con 0x7f4d940737a0 2024-04-24T08:10:40.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.095+0000 7f4d93200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4d940aeda0 con 0x7f4d941523f0 2024-04-24T08:10:40.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.095+0000 7f4d93200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4d940af080 con 0x7f4d94072e20 2024-04-24T08:10:40.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d92800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4d940737a0 0x7f4d9415ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d92800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4d940737a0 0x7f4d9415ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57848/0 (socket says 172.21.15.69:57848) 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d92800640 1 -- 172.21.15.69:0/135479322 learned_addr learned my addr 172.21.15.69:0/135479322 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d91e00640 1 --2- 172.21.15.69:0/135479322 >> v2:172.21.15.121:3300/0 conn(0x7f4d941523f0 0x7f4d9406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d8be00640 1 --2- 172.21.15.69:0/135479322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d94072e20 0x7f4d94073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d92800640 1 -- 172.21.15.69:0/135479322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d94072e20 msgr2=0x7f4d94073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d92800640 1 --2- 172.21.15.69:0/135479322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d94072e20 0x7f4d94073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d92800640 1 -- 172.21.15.69:0/135479322 >> v2:172.21.15.121:3300/0 conn(0x7f4d941523f0 msgr2=0x7f4d9406db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d92800640 1 --2- 172.21.15.69:0/135479322 >> v2:172.21.15.121:3300/0 conn(0x7f4d941523f0 0x7f4d9406db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d92800640 1 -- 172.21.15.69:0/135479322 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d940af360 con 0x7f4d940737a0 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.096+0000 7f4d91e00640 1 --2- 172.21.15.69:0/135479322 >> v2:172.21.15.121:3300/0 conn(0x7f4d941523f0 0x7f4d9406db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.097+0000 7f4d92800640 1 --2- 172.21.15.69:0/135479322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d940737a0 0x7f4d9415ef70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4d9406cb50 tx=0x7f4d8c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e0202f4f463df server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:40.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d91400640 1 -- 172.21.15.69:0/135479322 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c036070 con 0x7f4d940737a0 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d91400640 1 -- 172.21.15.69:0/135479322 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d8c031600 con 0x7f4d940737a0 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d91400640 1 -- 172.21.15.69:0/135479322 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8c0319d0 con 0x7f4d940737a0 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d93200640 1 -- 172.21.15.69:0/135479322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d940737a0 msgr2=0x7f4d9415ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d93200640 1 --2- 172.21.15.69:0/135479322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d940737a0 0x7f4d9415ef70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4d9406cb50 tx=0x7f4d8c02a610 comp rx=0 tx=0).stop 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d93200640 1 -- 172.21.15.69:0/135479322 shutdown_connections 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d93200640 1 --2- 172.21.15.69:0/135479322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d940737a0 0x7f4d9415ef70 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d93200640 1 --2- 172.21.15.69:0/135479322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d94072e20 0x7f4d94073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d93200640 1 --2- 172.21.15.69:0/135479322 >> v2:172.21.15.121:3300/0 conn(0x7f4d941523f0 0x7f4d9406db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d93200640 1 -- 172.21.15.69:0/135479322 >> 172.21.15.69:0/135479322 conn(0x7f4d940d0160 msgr2=0x7f4d940d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d93200640 1 -- 172.21.15.69:0/135479322 shutdown_connections 2024-04-24T08:10:40.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.098+0000 7f4d93200640 1 -- 172.21.15.69:0/135479322 wait complete. 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.100+0000 7f4d93200640 1 Processor -- start 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.100+0000 7f4d93200640 1 -- start start 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.101+0000 7f4d93200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d94072e20 0x7f4d94150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.101+0000 7f4d93200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d940737a0 0x7f4d9409dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.101+0000 7f4d93200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d941523f0 0x7f4d9414d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.101+0000 7f4d93200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9415f4b0 con 0x7f4d94072e20 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.101+0000 7f4d93200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4d941527d0 con 0x7f4d940737a0 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.101+0000 7f4d93200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4d9415f630 con 0x7f4d941523f0 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.101+0000 7f4d91e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d94072e20 0x7f4d94150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.101+0000 7f4d91e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d94072e20 0x7f4d94150930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57864/0 (socket says 172.21.15.69:57864) 2024-04-24T08:10:40.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.101+0000 7f4d91e00640 1 -- 172.21.15.69:0/4236014668 learned_addr learned my addr 172.21.15.69:0/4236014668 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d8be00640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d940737a0 0x7f4d9409dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d91e00640 1 -- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d941523f0 msgr2=0x7f4d9414d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d91e00640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d941523f0 0x7f4d9414d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d91e00640 1 -- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d940737a0 msgr2=0x7f4d9409dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d91e00640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d940737a0 0x7f4d9409dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d91e00640 1 -- 172.21.15.69:0/4236014668 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d94150f50 con 0x7f4d94072e20 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d8be00640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d940737a0 0x7f4d9409dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d91e00640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d94072e20 0x7f4d94150930 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400c7f0 tx=0x7f4d8400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d8a000640 1 -- 172.21.15.69:0/4236014668 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84015070 con 0x7f4d94072e20 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d8a000640 1 -- 172.21.15.69:0/4236014668 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d84007820 con 0x7f4d94072e20 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d8a000640 1 -- 172.21.15.69:0/4236014668 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84007bd0 con 0x7f4d94072e20 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.102+0000 7f4d93200640 1 -- 172.21.15.69:0/4236014668 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d94150370 con 0x7f4d94072e20 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.103+0000 7f4d93200640 1 -- 172.21.15.69:0/4236014668 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d9409b500 con 0x7f4d94072e20 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.104+0000 7f4d93200640 1 -- 172.21.15.69:0/4236014668 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d94075d50 con 0x7f4d94072e20 2024-04-24T08:10:40.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.105+0000 7f4d8a000640 1 -- 172.21.15.69:0/4236014668 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d84018070 con 0x7f4d94072e20 2024-04-24T08:10:40.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.105+0000 7f4d8a000640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d7006c8f0 0x7f4d7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.106+0000 7f4d8a000640 1 -- 172.21.15.69:0/4236014668 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f4d8408bbd0 con 0x7f4d94072e20 2024-04-24T08:10:40.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.106+0000 7f4d8be00640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d7006c8f0 0x7f4d7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.107+0000 7f4d8be00640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d7006c8f0 0x7f4d7006edb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c006fd0 tx=0x7f4d7c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:40.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.110+0000 7f4d8a000640 1 -- 172.21.15.69:0/4236014668 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d8405a090 con 0x7f4d94072e20 2024-04-24T08:10:40.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.306+0000 7f4d93200640 1 -- 172.21.15.69:0/4236014668 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4d94072a50 con 0x7f4d7006c8f0 2024-04-24T08:10:40.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.326+0000 7f4d8a000640 1 -- 172.21.15.69:0/4236014668 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+237202 (secure 0 0 0) 0x7f4d94072a50 con 0x7f4d7006c8f0 2024-04-24T08:10:40.331 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:40.332 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:10:40.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 -- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d7006c8f0 msgr2=0x7f4d7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d7006c8f0 0x7f4d7006edb0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c006fd0 tx=0x7f4d7c008040 comp rx=0 tx=0).stop 2024-04-24T08:10:40.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 -- 172.21.15.69:0/4236014668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d94072e20 msgr2=0x7f4d94150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d94072e20 0x7f4d94150930 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4d8400c7f0 tx=0x7f4d8400ccb0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 -- 172.21.15.69:0/4236014668 shutdown_connections 2024-04-24T08:10:40.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d7006c8f0 0x7f4d7006edb0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d941523f0 0x7f4d9414d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d940737a0 0x7f4d9409dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 --2- 172.21.15.69:0/4236014668 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d94072e20 0x7f4d94150930 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 -- 172.21.15.69:0/4236014668 >> 172.21.15.69:0/4236014668 conn(0x7f4d940d0160 msgr2=0x7f4d9414e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:40.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 -- 172.21.15.69:0/4236014668 shutdown_connections 2024-04-24T08:10:40.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.331+0000 7f4d93200640 1 -- 172.21.15.69:0/4236014668 wait complete. 2024-04-24T08:10:40.352 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-04-24T08:10:38.710718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":221016,"kb_used_data":4808,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753704,"statfs":{"total":773094113280,"available":772867792896,"internally_reserved":0,"allocated":4923392,"data_stored":2659160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.793089"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:34.915692+0000","last_change":"2024-04-24T08:10:34.915692+0000","last_active":"2024-04-24T08:10:34.915692+0000","last_peered":"2024-04-24T08:10:34.915692+0000","last_clean":"2024-04-24T08:10:34.915692+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:34.915692+0000","last_undegraded":"2024-04-24T08:10:34.915692+0000","last_fullsized":"2024-04-24T08:10:34.915692+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T08:10:33.899914+0000","last_change":"2024-04-24T08:10:33.899914+0000","last_active":"2024-04-24T08:10:33.899914+0000","last_peered":"2024-04-24T08:10:33.899914+0000","last_clean":"2024-04-24T08:10:33.899914+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:10:33.899914+0000","last_undegraded":"2024-04-24T08:10:33.899914+0000","last_fullsized":"2024-04-24T08:10:33.899914+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":205,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-04-24T08:10:31.674433+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:10:31.674433+0000","last_peered":"2024-04-24T08:10:31.674433+0000","last_clean":"2024-04-24T08:10:31.674433+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:10:31.674433+0000","last_undegraded":"2024-04-24T08:10:31.674433+0000","last_fullsized":"2024-04-24T08:10:31.674433+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1060864,"data_stored":775171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":184803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27488,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":466944,"data_stored":184803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":184803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":184803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":184803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":184803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1060864,"data_stored":775171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:10:40.356 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:10:40.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.593+0000 7fda02400640 1 Processor -- start 2024-04-24T08:10:40.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.594+0000 7fda02400640 1 -- start start 2024-04-24T08:10:40.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.595+0000 7fda02400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9fc0d02f0 0x7fd9fc06f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.595+0000 7fda02400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9fc0b46e0 con 0x7fd9fc06f2d0 2024-04-24T08:10:40.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.595+0000 7fda02400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9fc0d0960 con 0x7fd9fc15b490 2024-04-24T08:10:40.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.595+0000 7fda02400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9fc06fc70 con 0x7fd9fc0d02f0 2024-04-24T08:10:40.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9f2a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd9fc06f2d0 0x7fd9fc0cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49270/0 (socket says 172.21.15.69:49270) 2024-04-24T08:10:40.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9f2a00640 1 -- 172.21.15.69:0/784773189 learned_addr learned my addr 172.21.15.69:0/784773189 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:40.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4134545689 0 0) 0x7fd9fc0b46e0 con 0x7fd9fc06f2d0 2024-04-24T08:10:40.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9fbe00640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9fc0d02f0 0x7fd9fc06f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9e0003520 con 0x7fd9fc06f2d0 2024-04-24T08:10:40.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 161201145 0 0) 0x7fd9e0003520 con 0x7fd9fc06f2d0 2024-04-24T08:10:40.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9fc0d02f0 msgr2=0x7fd9fc06f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:40.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9faa00640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9fc0d02f0 0x7fd9fc06f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 >> v1:172.21.15.121:6789/0 conn(0x7fd9fc15b490 legacy=0x7fd9fc0b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9fc0b4980 con 0x7fd9fc06f2d0 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fd9e4001670 con 0x7fd9fc06f2d0 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 >> v1:172.21.15.69:6789/0 conn(0x7fd9fc06f2d0 legacy=0x7fd9fc0cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.596+0000 7fd9faa00640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9fc0d02f0 0x7fd9e00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9faa00640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9e0004010 0x7fd9e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9faa00640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9e0004d40 0x7fd9e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc06fc70 con 0x7fd9e0004d40 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e000bb80 con 0x7fd9e0004010 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e000df90 con 0x7fd9fc0d02f0 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9fbe00640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9e0004d40 0x7fd9e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9fb400640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9fc0d02f0 0x7fd9e00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9f2a00640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9e0004010 0x7fd9e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9fbe00640 1 -- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9fc0d02f0 msgr2=0x7fd9e00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9fbe00640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9fc0d02f0 0x7fd9e00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9fbe00640 1 -- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9e0004010 msgr2=0x7fd9e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:40.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9fbe00640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9e0004010 0x7fd9e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.597+0000 7fd9fbe00640 1 -- 172.21.15.69:0/784773189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e0003520 con 0x7fd9e0004d40 2024-04-24T08:10:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.598+0000 7fd9fbe00640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9e0004d40 0x7fd9e00051c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4007360 tx=0x7fd9e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7734bc70f1ae0e2f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.599+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f4009be0 con 0x7fd9e0004d40 2024-04-24T08:10:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.599+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9f4036600 con 0x7fd9e0004d40 2024-04-24T08:10:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.599+0000 7fd9faa00640 1 -- 172.21.15.69:0/784773189 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f40369d0 con 0x7fd9e0004d40 2024-04-24T08:10:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.599+0000 7fda02400640 1 -- 172.21.15.69:0/784773189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9e0004d40 msgr2=0x7fd9e00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.599+0000 7fda02400640 1 --2- 172.21.15.69:0/784773189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9e0004d40 0x7fd9e00051c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4007360 tx=0x7fd9e0005830 comp rx=0 tx=0).stop 2024-04-24T08:10:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.599+0000 7fd9fbe00640 1 -- 172.21.15.69:0/784773189 reap_dead start 2024-04-24T08:10:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.599+0000 7fda02400640 1 -- 172.21.15.69:0/784773189 shutdown_connections 2024-04-24T08:10:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.599+0000 7fda02400640 1 -- 172.21.15.69:0/784773189 >> 172.21.15.69:0/784773189 conn(0x7fd9fc067d90 msgr2=0x7fd9fc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.600+0000 7fda02400640 1 -- 172.21.15.69:0/784773189 shutdown_connections 2024-04-24T08:10:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.600+0000 7fda02400640 1 -- 172.21.15.69:0/784773189 wait complete. 2024-04-24T08:10:40.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.601+0000 7fda02400640 1 Processor -- start 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.601+0000 7fda02400640 1 -- start start 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.602+0000 7fda02400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9e0004d40 0x7fd9fc14df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.602+0000 7fda02400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9fc15bae0 0x7fd9fc15bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.602+0000 7fda02400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9fc15d320 0x7fd9fc15d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.602+0000 7fda02400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc0d0c40 con 0x7fd9fc15d320 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.602+0000 7fda02400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc0b5890 con 0x7fd9fc15bae0 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.602+0000 7fda02400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9fc15b870 con 0x7fd9e0004d40 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9fc15d320 0x7fd9fc15d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9fc15d320 0x7fd9fc15d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57874/0 (socket says 172.21.15.69:57874) 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fbe00640 1 -- 172.21.15.69:0/1649944944 learned_addr learned my addr 172.21.15.69:0/1649944944 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9f2a00640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9fc15bae0 0x7fd9fc15bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fb400640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9e0004d40 0x7fd9fc14df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fbe00640 1 -- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9e0004d40 msgr2=0x7fd9fc14df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fbe00640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9e0004d40 0x7fd9fc14df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fbe00640 1 -- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9fc15bae0 msgr2=0x7fd9fc15bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fbe00640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9fc15bae0 0x7fd9fc15bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fbe00640 1 -- 172.21.15.69:0/1649944944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9fc15c5a0 con 0x7fd9fc15d320 2024-04-24T08:10:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fb400640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9e0004d40 0x7fd9fc14df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.603+0000 7fd9fbe00640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9fc15d320 0x7fd9fc15d7a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4007450 tx=0x7fd9f4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.604+0000 7fd9f3e00640 1 -- 172.21.15.69:0/1649944944 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f4036e60 con 0x7fd9fc15d320 2024-04-24T08:10:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.604+0000 7fd9f3e00640 1 -- 172.21.15.69:0/1649944944 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9f4003990 con 0x7fd9fc15d320 2024-04-24T08:10:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.604+0000 7fda02400640 1 -- 172.21.15.69:0/1649944944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9fc15c880 con 0x7fd9fc15d320 2024-04-24T08:10:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.604+0000 7fd9f3e00640 1 -- 172.21.15.69:0/1649944944 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f4009be0 con 0x7fd9fc15d320 2024-04-24T08:10:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.604+0000 7fda02400640 1 -- 172.21.15.69:0/1649944944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9fc15ce40 con 0x7fd9fc15d320 2024-04-24T08:10:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.605+0000 7fda02400640 1 -- 172.21.15.69:0/1649944944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9fc068a80 con 0x7fd9fc15d320 2024-04-24T08:10:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.606+0000 7fd9f3e00640 1 -- 172.21.15.69:0/1649944944 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9f4048020 con 0x7fd9fc15d320 2024-04-24T08:10:40.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.607+0000 7fd9f3e00640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d806c8f0 0x7fd9d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:40.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.607+0000 7fd9f3e00640 1 -- 172.21.15.69:0/1649944944 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7fd9f40ac410 con 0x7fd9fc15d320 2024-04-24T08:10:40.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.608+0000 7fd9fb400640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d806c8f0 0x7fd9d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:40.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.609+0000 7fd9fb400640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d806c8f0 0x7fd9d806edb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0016b0 tx=0x7fd9ec009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:40.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.612+0000 7fd9f3e00640 1 -- 172.21.15.69:0/1649944944 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9f403d070 con 0x7fd9fc15d320 2024-04-24T08:10:40.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.804+0000 7fda02400640 1 -- 172.21.15.69:0/1649944944 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd9fc072b00 con 0x7fd9d806c8f0 2024-04-24T08:10:40.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.823+0000 7fd9f3e00640 1 -- 172.21.15.69:0/1649944944 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251959 (secure 0 0 0) 0x7fd9fc072b00 con 0x7fd9d806c8f0 2024-04-24T08:10:40.828 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:40.829 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:10:40.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 -- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d806c8f0 msgr2=0x7fd9d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d806c8f0 0x7fd9d806edb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec0016b0 tx=0x7fd9ec009040 comp rx=0 tx=0).stop 2024-04-24T08:10:40.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 -- 172.21.15.69:0/1649944944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9fc15d320 msgr2=0x7fd9fc15d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:40.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9fc15d320 0x7fd9fc15d7a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4007450 tx=0x7fd9f4004ab0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 -- 172.21.15.69:0/1649944944 shutdown_connections 2024-04-24T08:10:40.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d806c8f0 0x7fd9d806edb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9fc15d320 0x7fd9fc15d7a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9fc15bae0 0x7fd9fc15bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 --2- 172.21.15.69:0/1649944944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9e0004d40 0x7fd9fc14df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:40.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 -- 172.21.15.69:0/1649944944 >> 172.21.15.69:0/1649944944 conn(0x7fd9fc067d90 msgr2=0x7fd9fc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:40.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 -- 172.21.15.69:0/1649944944 shutdown_connections 2024-04-24T08:10:40.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:40.828+0000 7fda02400640 1 -- 172.21.15.69:0/1649944944 wait complete. 2024-04-24T08:10:40.847 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-04-24T08:10:40.711336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":225592,"kb_used_data":5256,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754749128,"statfs":{"total":773094113280,"available":772863107072,"internally_reserved":0,"allocated":5382144,"data_stored":2865252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":225496624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001820"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183587+0000","last_change":"2024-04-24T08:10:35.957733+0000","last_active":"2024-04-24T08:10:37.183587+0000","last_peered":"2024-04-24T08:10:37.183587+0000","last_clean":"2024-04-24T08:10:37.183587+0000","last_became_active":"2024-04-24T08:10:35.956882+0000","last_became_peered":"2024-04-24T08:10:35.956882+0000","last_unstale":"2024-04-24T08:10:37.183587+0000","last_undegraded":"2024-04-24T08:10:37.183587+0000","last_fullsized":"2024-04-24T08:10:37.183587+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:36:42.084494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.968688+0000","last_change":"2024-04-24T08:10:35.968125+0000","last_active":"2024-04-24T08:10:36.968688+0000","last_peered":"2024-04-24T08:10:36.968688+0000","last_clean":"2024-04-24T08:10:36.968688+0000","last_became_active":"2024-04-24T08:10:35.967346+0000","last_became_peered":"2024-04-24T08:10:35.967346+0000","last_unstale":"2024-04-24T08:10:36.968688+0000","last_undegraded":"2024-04-24T08:10:36.968688+0000","last_fullsized":"2024-04-24T08:10:36.968688+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:45:50.540457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181484+0000","last_change":"2024-04-24T08:10:35.961462+0000","last_active":"2024-04-24T08:10:37.181484+0000","last_peered":"2024-04-24T08:10:37.181484+0000","last_clean":"2024-04-24T08:10:37.181484+0000","last_became_active":"2024-04-24T08:10:35.961169+0000","last_became_peered":"2024-04-24T08:10:35.961169+0000","last_unstale":"2024-04-24T08:10:37.181484+0000","last_undegraded":"2024-04-24T08:10:37.181484+0000","last_fullsized":"2024-04-24T08:10:37.181484+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:38:08.679522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.182387+0000","last_change":"2024-04-24T08:10:35.974048+0000","last_active":"2024-04-24T08:10:37.182387+0000","last_peered":"2024-04-24T08:10:37.182387+0000","last_clean":"2024-04-24T08:10:37.182387+0000","last_became_active":"2024-04-24T08:10:35.973661+0000","last_became_peered":"2024-04-24T08:10:35.973661+0000","last_unstale":"2024-04-24T08:10:37.182387+0000","last_undegraded":"2024-04-24T08:10:37.182387+0000","last_fullsized":"2024-04-24T08:10:37.182387+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:23:53.251345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967368+0000","last_change":"2024-04-24T08:10:35.968947+0000","last_active":"2024-04-24T08:10:36.967368+0000","last_peered":"2024-04-24T08:10:36.967368+0000","last_clean":"2024-04-24T08:10:36.967368+0000","last_became_active":"2024-04-24T08:10:35.967709+0000","last_became_peered":"2024-04-24T08:10:35.967709+0000","last_unstale":"2024-04-24T08:10:36.967368+0000","last_undegraded":"2024-04-24T08:10:36.967368+0000","last_fullsized":"2024-04-24T08:10:36.967368+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:57:09.305687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.184591+0000","last_change":"2024-04-24T08:10:35.976251+0000","last_active":"2024-04-24T08:10:37.184591+0000","last_peered":"2024-04-24T08:10:37.184591+0000","last_clean":"2024-04-24T08:10:37.184591+0000","last_became_active":"2024-04-24T08:10:35.975982+0000","last_became_peered":"2024-04-24T08:10:35.975982+0000","last_unstale":"2024-04-24T08:10:37.184591+0000","last_undegraded":"2024-04-24T08:10:37.184591+0000","last_fullsized":"2024-04-24T08:10:37.184591+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:16:06.489585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183481+0000","last_change":"2024-04-24T08:10:35.968589+0000","last_active":"2024-04-24T08:10:37.183481+0000","last_peered":"2024-04-24T08:10:37.183481+0000","last_clean":"2024-04-24T08:10:37.183481+0000","last_became_active":"2024-04-24T08:10:35.968283+0000","last_became_peered":"2024-04-24T08:10:35.968283+0000","last_unstale":"2024-04-24T08:10:37.183481+0000","last_undegraded":"2024-04-24T08:10:37.183481+0000","last_fullsized":"2024-04-24T08:10:37.183481+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:55:47.037493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.180660+0000","last_change":"2024-04-24T08:10:35.970786+0000","last_active":"2024-04-24T08:10:37.180660+0000","last_peered":"2024-04-24T08:10:37.180660+0000","last_clean":"2024-04-24T08:10:37.180660+0000","last_became_active":"2024-04-24T08:10:35.970012+0000","last_became_peered":"2024-04-24T08:10:35.970012+0000","last_unstale":"2024-04-24T08:10:37.180660+0000","last_undegraded":"2024-04-24T08:10:37.180660+0000","last_fullsized":"2024-04-24T08:10:37.180660+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:15.175009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.182395+0000","last_change":"2024-04-24T08:10:35.968408+0000","last_active":"2024-04-24T08:10:37.182395+0000","last_peered":"2024-04-24T08:10:37.182395+0000","last_clean":"2024-04-24T08:10:37.182395+0000","last_became_active":"2024-04-24T08:10:35.968114+0000","last_became_peered":"2024-04-24T08:10:35.968114+0000","last_unstale":"2024-04-24T08:10:37.182395+0000","last_undegraded":"2024-04-24T08:10:37.182395+0000","last_fullsized":"2024-04-24T08:10:37.182395+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:24.043510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.182392+0000","last_change":"2024-04-24T08:10:35.973146+0000","last_active":"2024-04-24T08:10:37.182392+0000","last_peered":"2024-04-24T08:10:37.182392+0000","last_clean":"2024-04-24T08:10:37.182392+0000","last_became_active":"2024-04-24T08:10:35.970449+0000","last_became_peered":"2024-04-24T08:10:35.970449+0000","last_unstale":"2024-04-24T08:10:37.182392+0000","last_undegraded":"2024-04-24T08:10:37.182392+0000","last_fullsized":"2024-04-24T08:10:37.182392+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:36:49.165446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181804+0000","last_change":"2024-04-24T08:10:35.968630+0000","last_active":"2024-04-24T08:10:37.181804+0000","last_peered":"2024-04-24T08:10:37.181804+0000","last_clean":"2024-04-24T08:10:37.181804+0000","last_became_active":"2024-04-24T08:10:35.968329+0000","last_became_peered":"2024-04-24T08:10:35.968329+0000","last_unstale":"2024-04-24T08:10:37.181804+0000","last_undegraded":"2024-04-24T08:10:37.181804+0000","last_fullsized":"2024-04-24T08:10:37.181804+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:01.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967556+0000","last_change":"2024-04-24T08:10:35.963165+0000","last_active":"2024-04-24T08:10:36.967556+0000","last_peered":"2024-04-24T08:10:36.967556+0000","last_clean":"2024-04-24T08:10:36.967556+0000","last_became_active":"2024-04-24T08:10:35.961793+0000","last_became_peered":"2024-04-24T08:10:35.961793+0000","last_unstale":"2024-04-24T08:10:36.967556+0000","last_undegraded":"2024-04-24T08:10:36.967556+0000","last_fullsized":"2024-04-24T08:10:36.967556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:38:08.679522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967529+0000","last_change":"2024-04-24T08:10:35.959735+0000","last_active":"2024-04-24T08:10:36.967529+0000","last_peered":"2024-04-24T08:10:36.967529+0000","last_clean":"2024-04-24T08:10:36.967529+0000","last_became_active":"2024-04-24T08:10:35.959208+0000","last_became_peered":"2024-04-24T08:10:35.959208+0000","last_unstale":"2024-04-24T08:10:36.967529+0000","last_undegraded":"2024-04-24T08:10:36.967529+0000","last_fullsized":"2024-04-24T08:10:36.967529+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:22:45.794403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181174+0000","last_change":"2024-04-24T08:10:35.970799+0000","last_active":"2024-04-24T08:10:37.181174+0000","last_peered":"2024-04-24T08:10:37.181174+0000","last_clean":"2024-04-24T08:10:37.181174+0000","last_became_active":"2024-04-24T08:10:35.969442+0000","last_became_peered":"2024-04-24T08:10:35.969442+0000","last_unstale":"2024-04-24T08:10:37.181174+0000","last_undegraded":"2024-04-24T08:10:37.181174+0000","last_fullsized":"2024-04-24T08:10:37.181174+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:30:47.202350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.184252+0000","last_change":"2024-04-24T08:10:35.961372+0000","last_active":"2024-04-24T08:10:37.184252+0000","last_peered":"2024-04-24T08:10:37.184252+0000","last_clean":"2024-04-24T08:10:37.184252+0000","last_became_active":"2024-04-24T08:10:35.960794+0000","last_became_peered":"2024-04-24T08:10:35.960794+0000","last_unstale":"2024-04-24T08:10:37.184252+0000","last_undegraded":"2024-04-24T08:10:37.184252+0000","last_fullsized":"2024-04-24T08:10:37.184252+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:53:51.510905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.969901+0000","last_change":"2024-04-24T08:10:35.976971+0000","last_active":"2024-04-24T08:10:36.969901+0000","last_peered":"2024-04-24T08:10:36.969901+0000","last_clean":"2024-04-24T08:10:36.969901+0000","last_became_active":"2024-04-24T08:10:35.976667+0000","last_became_peered":"2024-04-24T08:10:35.976667+0000","last_unstale":"2024-04-24T08:10:36.969901+0000","last_undegraded":"2024-04-24T08:10:36.969901+0000","last_fullsized":"2024-04-24T08:10:36.969901+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:05:30.605710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181804+0000","last_change":"2024-04-24T08:10:35.958552+0000","last_active":"2024-04-24T08:10:37.181804+0000","last_peered":"2024-04-24T08:10:37.181804+0000","last_clean":"2024-04-24T08:10:37.181804+0000","last_became_active":"2024-04-24T08:10:35.958149+0000","last_became_peered":"2024-04-24T08:10:35.958149+0000","last_unstale":"2024-04-24T08:10:37.181804+0000","last_undegraded":"2024-04-24T08:10:37.181804+0000","last_fullsized":"2024-04-24T08:10:37.181804+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:15.175009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181888+0000","last_change":"2024-04-24T08:10:35.970582+0000","last_active":"2024-04-24T08:10:37.181888+0000","last_peered":"2024-04-24T08:10:37.181888+0000","last_clean":"2024-04-24T08:10:37.181888+0000","last_became_active":"2024-04-24T08:10:35.970254+0000","last_became_peered":"2024-04-24T08:10:35.970254+0000","last_unstale":"2024-04-24T08:10:37.181888+0000","last_undegraded":"2024-04-24T08:10:37.181888+0000","last_fullsized":"2024-04-24T08:10:37.181888+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:45:50.540457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.971182+0000","last_change":"2024-04-24T08:10:35.979490+0000","last_active":"2024-04-24T08:10:36.971182+0000","last_peered":"2024-04-24T08:10:36.971182+0000","last_clean":"2024-04-24T08:10:36.971182+0000","last_became_active":"2024-04-24T08:10:35.979204+0000","last_became_peered":"2024-04-24T08:10:35.979204+0000","last_unstale":"2024-04-24T08:10:36.971182+0000","last_undegraded":"2024-04-24T08:10:36.971182+0000","last_fullsized":"2024-04-24T08:10:36.971182+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:01:59.032681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967967+0000","last_change":"2024-04-24T08:10:35.978805+0000","last_active":"2024-04-24T08:10:36.967967+0000","last_peered":"2024-04-24T08:10:36.967967+0000","last_clean":"2024-04-24T08:10:36.967967+0000","last_became_active":"2024-04-24T08:10:35.977728+0000","last_became_peered":"2024-04-24T08:10:35.977728+0000","last_unstale":"2024-04-24T08:10:36.967967+0000","last_undegraded":"2024-04-24T08:10:36.967967+0000","last_fullsized":"2024-04-24T08:10:36.967967+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:27:17.191256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.184893+0000","last_change":"2024-04-24T08:10:35.967357+0000","last_active":"2024-04-24T08:10:37.184893+0000","last_peered":"2024-04-24T08:10:37.184893+0000","last_clean":"2024-04-24T08:10:37.184893+0000","last_became_active":"2024-04-24T08:10:35.967086+0000","last_became_peered":"2024-04-24T08:10:35.967086+0000","last_unstale":"2024-04-24T08:10:37.184893+0000","last_undegraded":"2024-04-24T08:10:37.184893+0000","last_fullsized":"2024-04-24T08:10:37.184893+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:00:16.871775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181555+0000","last_change":"2024-04-24T08:10:35.971187+0000","last_active":"2024-04-24T08:10:37.181555+0000","last_peered":"2024-04-24T08:10:37.181555+0000","last_clean":"2024-04-24T08:10:37.181555+0000","last_became_active":"2024-04-24T08:10:35.970271+0000","last_became_peered":"2024-04-24T08:10:35.970271+0000","last_unstale":"2024-04-24T08:10:37.181555+0000","last_undegraded":"2024-04-24T08:10:37.181555+0000","last_fullsized":"2024-04-24T08:10:37.181555+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:36:42.084494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.970456+0000","last_change":"2024-04-24T08:10:35.978138+0000","last_active":"2024-04-24T08:10:36.970456+0000","last_peered":"2024-04-24T08:10:36.970456+0000","last_clean":"2024-04-24T08:10:36.970456+0000","last_became_active":"2024-04-24T08:10:35.977056+0000","last_became_peered":"2024-04-24T08:10:35.977056+0000","last_unstale":"2024-04-24T08:10:36.970456+0000","last_undegraded":"2024-04-24T08:10:36.970456+0000","last_fullsized":"2024-04-24T08:10:36.970456+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:18:07.124356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.969970+0000","last_change":"2024-04-24T08:10:35.969765+0000","last_active":"2024-04-24T08:10:36.969970+0000","last_peered":"2024-04-24T08:10:36.969970+0000","last_clean":"2024-04-24T08:10:36.969970+0000","last_became_active":"2024-04-24T08:10:35.969588+0000","last_became_peered":"2024-04-24T08:10:35.969588+0000","last_unstale":"2024-04-24T08:10:36.969970+0000","last_undegraded":"2024-04-24T08:10:36.969970+0000","last_fullsized":"2024-04-24T08:10:36.969970+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:52:13.033599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.971453+0000","last_change":"2024-04-24T08:10:35.972120+0000","last_active":"2024-04-24T08:10:36.971453+0000","last_peered":"2024-04-24T08:10:36.971453+0000","last_clean":"2024-04-24T08:10:36.971453+0000","last_became_active":"2024-04-24T08:10:35.971717+0000","last_became_peered":"2024-04-24T08:10:35.971717+0000","last_unstale":"2024-04-24T08:10:36.971453+0000","last_undegraded":"2024-04-24T08:10:36.971453+0000","last_fullsized":"2024-04-24T08:10:36.971453+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:09:20.580556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967999+0000","last_change":"2024-04-24T08:10:35.972341+0000","last_active":"2024-04-24T08:10:36.967999+0000","last_peered":"2024-04-24T08:10:36.967999+0000","last_clean":"2024-04-24T08:10:36.967999+0000","last_became_active":"2024-04-24T08:10:35.968979+0000","last_became_peered":"2024-04-24T08:10:35.968979+0000","last_unstale":"2024-04-24T08:10:36.967999+0000","last_undegraded":"2024-04-24T08:10:36.967999+0000","last_fullsized":"2024-04-24T08:10:36.967999+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:33:52.941874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181932+0000","last_change":"2024-04-24T08:10:35.954386+0000","last_active":"2024-04-24T08:10:37.181932+0000","last_peered":"2024-04-24T08:10:37.181932+0000","last_clean":"2024-04-24T08:10:37.181932+0000","last_became_active":"2024-04-24T08:10:35.953477+0000","last_became_peered":"2024-04-24T08:10:35.953477+0000","last_unstale":"2024-04-24T08:10:37.181932+0000","last_undegraded":"2024-04-24T08:10:37.181932+0000","last_fullsized":"2024-04-24T08:10:37.181932+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:55:47.037493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967949+0000","last_change":"2024-04-24T08:10:35.954307+0000","last_active":"2024-04-24T08:10:36.967949+0000","last_peered":"2024-04-24T08:10:36.967949+0000","last_clean":"2024-04-24T08:10:36.967949+0000","last_became_active":"2024-04-24T08:10:35.954002+0000","last_became_peered":"2024-04-24T08:10:35.954002+0000","last_unstale":"2024-04-24T08:10:36.967949+0000","last_undegraded":"2024-04-24T08:10:36.967949+0000","last_fullsized":"2024-04-24T08:10:36.967949+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:48.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.970650+0000","last_change":"2024-04-24T08:10:35.971378+0000","last_active":"2024-04-24T08:10:36.970650+0000","last_peered":"2024-04-24T08:10:36.970650+0000","last_clean":"2024-04-24T08:10:36.970650+0000","last_became_active":"2024-04-24T08:10:35.971084+0000","last_became_peered":"2024-04-24T08:10:35.971084+0000","last_unstale":"2024-04-24T08:10:36.970650+0000","last_undegraded":"2024-04-24T08:10:36.970650+0000","last_fullsized":"2024-04-24T08:10:36.970650+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:54:45.297903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181162+0000","last_change":"2024-04-24T08:10:35.966544+0000","last_active":"2024-04-24T08:10:37.181162+0000","last_peered":"2024-04-24T08:10:37.181162+0000","last_clean":"2024-04-24T08:10:37.181162+0000","last_became_active":"2024-04-24T08:10:35.966205+0000","last_became_peered":"2024-04-24T08:10:35.966205+0000","last_unstale":"2024-04-24T08:10:37.181162+0000","last_undegraded":"2024-04-24T08:10:37.181162+0000","last_fullsized":"2024-04-24T08:10:37.181162+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:57:09.305687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.184280+0000","last_change":"2024-04-24T08:10:35.961147+0000","last_active":"2024-04-24T08:10:37.184280+0000","last_peered":"2024-04-24T08:10:37.184280+0000","last_clean":"2024-04-24T08:10:37.184280+0000","last_became_active":"2024-04-24T08:10:35.960467+0000","last_became_peered":"2024-04-24T08:10:35.960467+0000","last_unstale":"2024-04-24T08:10:37.184280+0000","last_undegraded":"2024-04-24T08:10:37.184280+0000","last_fullsized":"2024-04-24T08:10:37.184280+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:45:46.412557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967048+0000","last_change":"2024-04-24T08:10:35.961581+0000","last_active":"2024-04-24T08:10:36.967048+0000","last_peered":"2024-04-24T08:10:36.967048+0000","last_clean":"2024-04-24T08:10:36.967048+0000","last_became_active":"2024-04-24T08:10:35.961299+0000","last_became_peered":"2024-04-24T08:10:35.961299+0000","last_unstale":"2024-04-24T08:10:36.967048+0000","last_undegraded":"2024-04-24T08:10:36.967048+0000","last_fullsized":"2024-04-24T08:10:36.967048+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:54:24.417526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.182828+0000","last_change":"2024-04-24T08:10:35.974264+0000","last_active":"2024-04-24T08:10:37.182828+0000","last_peered":"2024-04-24T08:10:37.182828+0000","last_clean":"2024-04-24T08:10:37.182828+0000","last_became_active":"2024-04-24T08:10:35.974001+0000","last_became_peered":"2024-04-24T08:10:35.974001+0000","last_unstale":"2024-04-24T08:10:37.182828+0000","last_undegraded":"2024-04-24T08:10:37.182828+0000","last_fullsized":"2024-04-24T08:10:37.182828+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:54:45.297903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"43'4","reported_seq":18,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.185681+0000","last_change":"2024-04-24T08:10:34.965946+0000","last_active":"2024-04-24T08:10:37.185681+0000","last_peered":"2024-04-24T08:10:37.185681+0000","last_clean":"2024-04-24T08:10:37.185681+0000","last_became_active":"2024-04-24T08:10:34.965700+0000","last_became_peered":"2024-04-24T08:10:34.965700+0000","last_unstale":"2024-04-24T08:10:37.185681+0000","last_undegraded":"2024-04-24T08:10:37.185681+0000","last_fullsized":"2024-04-24T08:10:37.185681+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:00.217257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.971785+0000","last_change":"2024-04-24T08:10:35.977508+0000","last_active":"2024-04-24T08:10:36.971785+0000","last_peered":"2024-04-24T08:10:36.971785+0000","last_clean":"2024-04-24T08:10:36.971785+0000","last_became_active":"2024-04-24T08:10:35.976871+0000","last_became_peered":"2024-04-24T08:10:35.976871+0000","last_unstale":"2024-04-24T08:10:36.971785+0000","last_undegraded":"2024-04-24T08:10:36.971785+0000","last_fullsized":"2024-04-24T08:10:36.971785+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:01:18.747814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.182001+0000","last_change":"2024-04-24T08:10:34.966536+0000","last_active":"2024-04-24T08:10:37.182001+0000","last_peered":"2024-04-24T08:10:37.182001+0000","last_clean":"2024-04-24T08:10:37.182001+0000","last_became_active":"2024-04-24T08:10:34.965711+0000","last_became_peered":"2024-04-24T08:10:34.965711+0000","last_unstale":"2024-04-24T08:10:37.182001+0000","last_undegraded":"2024-04-24T08:10:37.182001+0000","last_fullsized":"2024-04-24T08:10:37.182001+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:50:15.371818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.968157+0000","last_change":"2024-04-24T08:10:35.973603+0000","last_active":"2024-04-24T08:10:36.968157+0000","last_peered":"2024-04-24T08:10:36.968157+0000","last_clean":"2024-04-24T08:10:36.968157+0000","last_became_active":"2024-04-24T08:10:35.972153+0000","last_became_peered":"2024-04-24T08:10:35.972153+0000","last_unstale":"2024-04-24T08:10:36.968157+0000","last_undegraded":"2024-04-24T08:10:36.968157+0000","last_fullsized":"2024-04-24T08:10:36.968157+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:15.175009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183612+0000","last_change":"2024-04-24T08:10:34.955935+0000","last_active":"2024-04-24T08:10:37.183612+0000","last_peered":"2024-04-24T08:10:37.183612+0000","last_clean":"2024-04-24T08:10:37.183612+0000","last_became_active":"2024-04-24T08:10:34.955681+0000","last_became_peered":"2024-04-24T08:10:34.955681+0000","last_unstale":"2024-04-24T08:10:37.183612+0000","last_undegraded":"2024-04-24T08:10:37.183612+0000","last_fullsized":"2024-04-24T08:10:37.183612+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:05:29.589932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183379+0000","last_change":"2024-04-24T08:10:35.968711+0000","last_active":"2024-04-24T08:10:37.183379+0000","last_peered":"2024-04-24T08:10:37.183379+0000","last_clean":"2024-04-24T08:10:37.183379+0000","last_became_active":"2024-04-24T08:10:35.967355+0000","last_became_peered":"2024-04-24T08:10:35.967355+0000","last_unstale":"2024-04-24T08:10:37.183379+0000","last_undegraded":"2024-04-24T08:10:37.183379+0000","last_fullsized":"2024-04-24T08:10:37.183379+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:35:48.724638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183236+0000","last_change":"2024-04-24T08:10:34.972660+0000","last_active":"2024-04-24T08:10:37.183236+0000","last_peered":"2024-04-24T08:10:37.183236+0000","last_clean":"2024-04-24T08:10:37.183236+0000","last_became_active":"2024-04-24T08:10:34.972303+0000","last_became_peered":"2024-04-24T08:10:34.972303+0000","last_unstale":"2024-04-24T08:10:37.183236+0000","last_undegraded":"2024-04-24T08:10:37.183236+0000","last_fullsized":"2024-04-24T08:10:37.183236+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:35:47.708860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181494+0000","last_change":"2024-04-24T08:10:35.967815+0000","last_active":"2024-04-24T08:10:37.181494+0000","last_peered":"2024-04-24T08:10:37.181494+0000","last_clean":"2024-04-24T08:10:37.181494+0000","last_became_active":"2024-04-24T08:10:35.967454+0000","last_became_peered":"2024-04-24T08:10:35.967454+0000","last_unstale":"2024-04-24T08:10:37.181494+0000","last_undegraded":"2024-04-24T08:10:37.181494+0000","last_fullsized":"2024-04-24T08:10:37.181494+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:35:34.379491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.185092+0000","last_change":"2024-04-24T08:10:34.961532+0000","last_active":"2024-04-24T08:10:37.185092+0000","last_peered":"2024-04-24T08:10:37.185092+0000","last_clean":"2024-04-24T08:10:37.185092+0000","last_became_active":"2024-04-24T08:10:34.961259+0000","last_became_peered":"2024-04-24T08:10:34.961259+0000","last_unstale":"2024-04-24T08:10:37.185092+0000","last_undegraded":"2024-04-24T08:10:37.185092+0000","last_fullsized":"2024-04-24T08:10:37.185092+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:14.159231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.180882+0000","last_change":"2024-04-24T08:10:35.965402+0000","last_active":"2024-04-24T08:10:37.180882+0000","last_peered":"2024-04-24T08:10:37.180882+0000","last_clean":"2024-04-24T08:10:37.180882+0000","last_became_active":"2024-04-24T08:10:35.965138+0000","last_became_peered":"2024-04-24T08:10:35.965138+0000","last_unstale":"2024-04-24T08:10:37.180882+0000","last_undegraded":"2024-04-24T08:10:37.180882+0000","last_fullsized":"2024-04-24T08:10:37.180882+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:45:50.540457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181339+0000","last_change":"2024-04-24T08:10:34.958748+0000","last_active":"2024-04-24T08:10:37.181339+0000","last_peered":"2024-04-24T08:10:37.181339+0000","last_clean":"2024-04-24T08:10:37.181339+0000","last_became_active":"2024-04-24T08:10:34.954291+0000","last_became_peered":"2024-04-24T08:10:34.954291+0000","last_unstale":"2024-04-24T08:10:37.181339+0000","last_undegraded":"2024-04-24T08:10:37.181339+0000","last_fullsized":"2024-04-24T08:10:37.181339+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:27:16.175478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.180427+0000","last_change":"2024-04-24T08:10:35.970559+0000","last_active":"2024-04-24T08:10:37.180427+0000","last_peered":"2024-04-24T08:10:37.180427+0000","last_clean":"2024-04-24T08:10:37.180427+0000","last_became_active":"2024-04-24T08:10:35.969519+0000","last_became_peered":"2024-04-24T08:10:35.969519+0000","last_unstale":"2024-04-24T08:10:37.180427+0000","last_undegraded":"2024-04-24T08:10:37.180427+0000","last_fullsized":"2024-04-24T08:10:37.180427+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:48.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967147+0000","last_change":"2024-04-24T08:10:34.964215+0000","last_active":"2024-04-24T08:10:36.967147+0000","last_peered":"2024-04-24T08:10:36.967147+0000","last_clean":"2024-04-24T08:10:36.967147+0000","last_became_active":"2024-04-24T08:10:34.963879+0000","last_became_peered":"2024-04-24T08:10:34.963879+0000","last_unstale":"2024-04-24T08:10:36.967147+0000","last_undegraded":"2024-04-24T08:10:36.967147+0000","last_fullsized":"2024-04-24T08:10:36.967147+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:52:12.017821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.970309+0000","last_change":"2024-04-24T08:10:35.978923+0000","last_active":"2024-04-24T08:10:36.970309+0000","last_peered":"2024-04-24T08:10:36.970309+0000","last_clean":"2024-04-24T08:10:36.970309+0000","last_became_active":"2024-04-24T08:10:35.977067+0000","last_became_peered":"2024-04-24T08:10:35.977067+0000","last_unstale":"2024-04-24T08:10:36.970309+0000","last_undegraded":"2024-04-24T08:10:36.970309+0000","last_fullsized":"2024-04-24T08:10:36.970309+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:36:05.800223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.180550+0000","last_change":"2024-04-24T08:10:34.957101+0000","last_active":"2024-04-24T08:10:37.180550+0000","last_peered":"2024-04-24T08:10:37.180550+0000","last_clean":"2024-04-24T08:10:37.180550+0000","last_became_active":"2024-04-24T08:10:34.956790+0000","last_became_peered":"2024-04-24T08:10:34.956790+0000","last_unstale":"2024-04-24T08:10:37.180550+0000","last_undegraded":"2024-04-24T08:10:37.180550+0000","last_fullsized":"2024-04-24T08:10:37.180550+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:22:44.778625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181684+0000","last_change":"2024-04-24T08:10:35.970590+0000","last_active":"2024-04-24T08:10:37.181684+0000","last_peered":"2024-04-24T08:10:37.181684+0000","last_clean":"2024-04-24T08:10:37.181684+0000","last_became_active":"2024-04-24T08:10:35.970301+0000","last_became_peered":"2024-04-24T08:10:35.970301+0000","last_unstale":"2024-04-24T08:10:37.181684+0000","last_undegraded":"2024-04-24T08:10:37.181684+0000","last_fullsized":"2024-04-24T08:10:37.181684+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:48.374193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.182100+0000","last_change":"2024-04-24T08:10:34.955647+0000","last_active":"2024-04-24T08:10:37.182100+0000","last_peered":"2024-04-24T08:10:37.182100+0000","last_clean":"2024-04-24T08:10:37.182100+0000","last_became_active":"2024-04-24T08:10:34.954906+0000","last_became_peered":"2024-04-24T08:10:34.954906+0000","last_unstale":"2024-04-24T08:10:37.182100+0000","last_undegraded":"2024-04-24T08:10:37.182100+0000","last_fullsized":"2024-04-24T08:10:37.182100+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:52:12.017821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183808+0000","last_change":"2024-04-24T08:10:35.968626+0000","last_active":"2024-04-24T08:10:37.183808+0000","last_peered":"2024-04-24T08:10:37.183808+0000","last_clean":"2024-04-24T08:10:37.183808+0000","last_became_active":"2024-04-24T08:10:35.968294+0000","last_became_peered":"2024-04-24T08:10:35.968294+0000","last_unstale":"2024-04-24T08:10:37.183808+0000","last_undegraded":"2024-04-24T08:10:37.183808+0000","last_fullsized":"2024-04-24T08:10:37.183808+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:58:01.233035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183381+0000","last_change":"2024-04-24T08:10:34.961221+0000","last_active":"2024-04-24T08:10:37.183381+0000","last_peered":"2024-04-24T08:10:37.183381+0000","last_clean":"2024-04-24T08:10:37.183381+0000","last_became_active":"2024-04-24T08:10:34.960962+0000","last_became_peered":"2024-04-24T08:10:34.960962+0000","last_unstale":"2024-04-24T08:10:37.183381+0000","last_undegraded":"2024-04-24T08:10:37.183381+0000","last_fullsized":"2024-04-24T08:10:37.183381+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:50:15.371818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.185569+0000","last_change":"2024-04-24T08:10:35.966859+0000","last_active":"2024-04-24T08:10:37.185569+0000","last_peered":"2024-04-24T08:10:37.185569+0000","last_clean":"2024-04-24T08:10:37.185569+0000","last_became_active":"2024-04-24T08:10:35.966476+0000","last_became_peered":"2024-04-24T08:10:35.966476+0000","last_unstale":"2024-04-24T08:10:37.185569+0000","last_undegraded":"2024-04-24T08:10:37.185569+0000","last_fullsized":"2024-04-24T08:10:37.185569+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:51.884627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"43'3","reported_seq":17,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183899+0000","last_change":"2024-04-24T08:10:34.973840+0000","last_active":"2024-04-24T08:10:37.183899+0000","last_peered":"2024-04-24T08:10:37.183899+0000","last_clean":"2024-04-24T08:10:37.183899+0000","last_became_active":"2024-04-24T08:10:34.973568+0000","last_became_peered":"2024-04-24T08:10:34.973568+0000","last_unstale":"2024-04-24T08:10:37.183899+0000","last_undegraded":"2024-04-24T08:10:37.183899+0000","last_fullsized":"2024-04-24T08:10:37.183899+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:07.163341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183597+0000","last_change":"2024-04-24T08:10:35.973360+0000","last_active":"2024-04-24T08:10:37.183597+0000","last_peered":"2024-04-24T08:10:37.183597+0000","last_clean":"2024-04-24T08:10:37.183597+0000","last_became_active":"2024-04-24T08:10:35.972827+0000","last_became_peered":"2024-04-24T08:10:35.972827+0000","last_unstale":"2024-04-24T08:10:37.183597+0000","last_undegraded":"2024-04-24T08:10:37.183597+0000","last_fullsized":"2024-04-24T08:10:37.183597+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:30:47.202350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"43'6","reported_seq":20,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.968130+0000","last_change":"2024-04-24T08:10:34.962930+0000","last_active":"2024-04-24T08:10:36.968130+0000","last_peered":"2024-04-24T08:10:36.968130+0000","last_clean":"2024-04-24T08:10:36.968130+0000","last_became_active":"2024-04-24T08:10:34.962627+0000","last_became_peered":"2024-04-24T08:10:34.962627+0000","last_unstale":"2024-04-24T08:10:36.968130+0000","last_undegraded":"2024-04-24T08:10:36.968130+0000","last_fullsized":"2024-04-24T08:10:36.968130+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:32.563215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.970292+0000","last_change":"2024-04-24T08:10:35.971394+0000","last_active":"2024-04-24T08:10:36.970292+0000","last_peered":"2024-04-24T08:10:36.970292+0000","last_clean":"2024-04-24T08:10:36.970292+0000","last_became_active":"2024-04-24T08:10:35.971063+0000","last_became_peered":"2024-04-24T08:10:35.971063+0000","last_unstale":"2024-04-24T08:10:36.970292+0000","last_undegraded":"2024-04-24T08:10:36.970292+0000","last_fullsized":"2024-04-24T08:10:36.970292+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:39:12.042558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.185203+0000","last_change":"2024-04-24T08:10:34.963141+0000","last_active":"2024-04-24T08:10:37.185203+0000","last_peered":"2024-04-24T08:10:37.185203+0000","last_clean":"2024-04-24T08:10:37.185203+0000","last_became_active":"2024-04-24T08:10:34.962957+0000","last_became_peered":"2024-04-24T08:10:34.962957+0000","last_unstale":"2024-04-24T08:10:37.185203+0000","last_undegraded":"2024-04-24T08:10:37.185203+0000","last_fullsized":"2024-04-24T08:10:37.185203+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:45:49.524678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181034+0000","last_change":"2024-04-24T08:10:35.971442+0000","last_active":"2024-04-24T08:10:37.181034+0000","last_peered":"2024-04-24T08:10:37.181034+0000","last_clean":"2024-04-24T08:10:37.181034+0000","last_became_active":"2024-04-24T08:10:35.970016+0000","last_became_peered":"2024-04-24T08:10:35.970016+0000","last_unstale":"2024-04-24T08:10:37.181034+0000","last_undegraded":"2024-04-24T08:10:37.181034+0000","last_fullsized":"2024-04-24T08:10:37.181034+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:55:47.037493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183885+0000","last_change":"2024-04-24T08:10:34.970428+0000","last_active":"2024-04-24T08:10:37.183885+0000","last_peered":"2024-04-24T08:10:37.183885+0000","last_clean":"2024-04-24T08:10:37.183885+0000","last_became_active":"2024-04-24T08:10:34.969515+0000","last_became_peered":"2024-04-24T08:10:34.969515+0000","last_unstale":"2024-04-24T08:10:37.183885+0000","last_undegraded":"2024-04-24T08:10:37.183885+0000","last_fullsized":"2024-04-24T08:10:37.183885+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:09:19.564778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.184049+0000","last_change":"2024-04-24T08:10:35.964091+0000","last_active":"2024-04-24T08:10:37.184049+0000","last_peered":"2024-04-24T08:10:37.184049+0000","last_clean":"2024-04-24T08:10:37.184049+0000","last_became_active":"2024-04-24T08:10:35.963388+0000","last_became_peered":"2024-04-24T08:10:35.963388+0000","last_unstale":"2024-04-24T08:10:37.184049+0000","last_undegraded":"2024-04-24T08:10:37.184049+0000","last_fullsized":"2024-04-24T08:10:37.184049+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:50:35.605054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.184531+0000","last_change":"2024-04-24T08:10:34.958523+0000","last_active":"2024-04-24T08:10:37.184531+0000","last_peered":"2024-04-24T08:10:37.184531+0000","last_clean":"2024-04-24T08:10:37.184531+0000","last_became_active":"2024-04-24T08:10:34.958222+0000","last_became_peered":"2024-04-24T08:10:34.958222+0000","last_unstale":"2024-04-24T08:10:37.184531+0000","last_undegraded":"2024-04-24T08:10:37.184531+0000","last_fullsized":"2024-04-24T08:10:37.184531+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:47.358414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183839+0000","last_change":"2024-04-24T08:10:35.969796+0000","last_active":"2024-04-24T08:10:37.183839+0000","last_peered":"2024-04-24T08:10:37.183839+0000","last_clean":"2024-04-24T08:10:37.183839+0000","last_became_active":"2024-04-24T08:10:35.969503+0000","last_became_peered":"2024-04-24T08:10:35.969503+0000","last_unstale":"2024-04-24T08:10:37.183839+0000","last_undegraded":"2024-04-24T08:10:37.183839+0000","last_fullsized":"2024-04-24T08:10:37.183839+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:24.043510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"43'8","reported_seq":22,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.180675+0000","last_change":"2024-04-24T08:10:34.954252+0000","last_active":"2024-04-24T08:10:37.180675+0000","last_peered":"2024-04-24T08:10:37.180675+0000","last_clean":"2024-04-24T08:10:37.180675+0000","last_became_active":"2024-04-24T08:10:34.954002+0000","last_became_peered":"2024-04-24T08:10:34.954002+0000","last_unstale":"2024-04-24T08:10:37.180675+0000","last_undegraded":"2024-04-24T08:10:37.180675+0000","last_fullsized":"2024-04-24T08:10:37.180675+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:35.002937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.968334+0000","last_change":"2024-04-24T08:10:35.953584+0000","last_active":"2024-04-24T08:10:36.968334+0000","last_peered":"2024-04-24T08:10:36.968334+0000","last_clean":"2024-04-24T08:10:36.968334+0000","last_became_active":"2024-04-24T08:10:35.953233+0000","last_became_peered":"2024-04-24T08:10:35.953233+0000","last_unstale":"2024-04-24T08:10:36.968334+0000","last_undegraded":"2024-04-24T08:10:36.968334+0000","last_fullsized":"2024-04-24T08:10:36.968334+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:50:16.387597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967806+0000","last_change":"2024-04-24T08:10:34.961766+0000","last_active":"2024-04-24T08:10:36.967806+0000","last_peered":"2024-04-24T08:10:36.967806+0000","last_clean":"2024-04-24T08:10:36.967806+0000","last_became_active":"2024-04-24T08:10:34.960778+0000","last_became_peered":"2024-04-24T08:10:34.960778+0000","last_unstale":"2024-04-24T08:10:36.967806+0000","last_undegraded":"2024-04-24T08:10:36.967806+0000","last_fullsized":"2024-04-24T08:10:36.967806+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:46:05.927235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.182798+0000","last_change":"2024-04-24T08:10:35.970705+0000","last_active":"2024-04-24T08:10:37.182798+0000","last_peered":"2024-04-24T08:10:37.182798+0000","last_clean":"2024-04-24T08:10:37.182798+0000","last_became_active":"2024-04-24T08:10:35.970257+0000","last_became_peered":"2024-04-24T08:10:35.970257+0000","last_unstale":"2024-04-24T08:10:37.182798+0000","last_undegraded":"2024-04-24T08:10:37.182798+0000","last_fullsized":"2024-04-24T08:10:37.182798+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:42:13.545323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.182120+0000","last_change":"2024-04-24T08:10:34.961451+0000","last_active":"2024-04-24T08:10:37.182120+0000","last_peered":"2024-04-24T08:10:37.182120+0000","last_clean":"2024-04-24T08:10:37.182120+0000","last_became_active":"2024-04-24T08:10:34.961162+0000","last_became_peered":"2024-04-24T08:10:34.961162+0000","last_unstale":"2024-04-24T08:10:37.182120+0000","last_undegraded":"2024-04-24T08:10:37.182120+0000","last_fullsized":"2024-04-24T08:10:37.182120+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:05:29.589932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.970811+0000","last_change":"2024-04-24T08:10:35.970412+0000","last_active":"2024-04-24T08:10:36.970811+0000","last_peered":"2024-04-24T08:10:36.970811+0000","last_clean":"2024-04-24T08:10:36.970811+0000","last_became_active":"2024-04-24T08:10:35.970129+0000","last_became_peered":"2024-04-24T08:10:35.970129+0000","last_unstale":"2024-04-24T08:10:36.970811+0000","last_undegraded":"2024-04-24T08:10:36.970811+0000","last_fullsized":"2024-04-24T08:10:36.970811+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:35:48.724638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"43'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.968408+0000","last_change":"2024-04-24T08:10:34.961605+0000","last_active":"2024-04-24T08:10:36.968408+0000","last_peered":"2024-04-24T08:10:36.968408+0000","last_clean":"2024-04-24T08:10:36.968408+0000","last_became_active":"2024-04-24T08:10:34.960969+0000","last_became_peered":"2024-04-24T08:10:34.960969+0000","last_unstale":"2024-04-24T08:10:36.968408+0000","last_undegraded":"2024-04-24T08:10:36.968408+0000","last_fullsized":"2024-04-24T08:10:36.968408+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:36:04.784445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.970990+0000","last_change":"2024-04-24T08:10:35.972961+0000","last_active":"2024-04-24T08:10:36.970990+0000","last_peered":"2024-04-24T08:10:36.970990+0000","last_clean":"2024-04-24T08:10:36.970990+0000","last_became_active":"2024-04-24T08:10:35.972716+0000","last_became_peered":"2024-04-24T08:10:35.972716+0000","last_unstale":"2024-04-24T08:10:36.970990+0000","last_undegraded":"2024-04-24T08:10:36.970990+0000","last_fullsized":"2024-04-24T08:10:36.970990+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:08.179119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.185680+0000","last_change":"2024-04-24T08:10:34.966254+0000","last_active":"2024-04-24T08:10:37.185680+0000","last_peered":"2024-04-24T08:10:37.185680+0000","last_clean":"2024-04-24T08:10:37.185680+0000","last_became_active":"2024-04-24T08:10:34.965947+0000","last_became_peered":"2024-04-24T08:10:34.965947+0000","last_unstale":"2024-04-24T08:10:37.185680+0000","last_undegraded":"2024-04-24T08:10:37.185680+0000","last_fullsized":"2024-04-24T08:10:37.185680+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:36:41.068716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.184835+0000","last_change":"2024-04-24T08:10:35.964032+0000","last_active":"2024-04-24T08:10:37.184835+0000","last_peered":"2024-04-24T08:10:37.184835+0000","last_clean":"2024-04-24T08:10:37.184835+0000","last_became_active":"2024-04-24T08:10:35.963726+0000","last_became_peered":"2024-04-24T08:10:35.963726+0000","last_unstale":"2024-04-24T08:10:37.184835+0000","last_undegraded":"2024-04-24T08:10:37.184835+0000","last_fullsized":"2024-04-24T08:10:37.184835+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:18.621201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.185401+0000","last_change":"2024-04-24T08:10:34.961886+0000","last_active":"2024-04-24T08:10:37.185401+0000","last_peered":"2024-04-24T08:10:37.185401+0000","last_clean":"2024-04-24T08:10:37.185401+0000","last_became_active":"2024-04-24T08:10:34.960505+0000","last_became_peered":"2024-04-24T08:10:34.960505+0000","last_unstale":"2024-04-24T08:10:37.185401+0000","last_undegraded":"2024-04-24T08:10:37.185401+0000","last_fullsized":"2024-04-24T08:10:37.185401+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:55:46.021715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181728+0000","last_change":"2024-04-24T08:10:35.959328+0000","last_active":"2024-04-24T08:10:37.181728+0000","last_peered":"2024-04-24T08:10:37.181728+0000","last_clean":"2024-04-24T08:10:37.181728+0000","last_became_active":"2024-04-24T08:10:35.957729+0000","last_became_peered":"2024-04-24T08:10:35.957729+0000","last_unstale":"2024-04-24T08:10:37.181728+0000","last_undegraded":"2024-04-24T08:10:37.181728+0000","last_fullsized":"2024-04-24T08:10:37.181728+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:18:07.124356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967459+0000","last_change":"2024-04-24T08:10:34.962992+0000","last_active":"2024-04-24T08:10:36.967459+0000","last_peered":"2024-04-24T08:10:36.967459+0000","last_clean":"2024-04-24T08:10:36.967459+0000","last_became_active":"2024-04-24T08:10:34.962701+0000","last_became_peered":"2024-04-24T08:10:34.962701+0000","last_unstale":"2024-04-24T08:10:36.967459+0000","last_undegraded":"2024-04-24T08:10:36.967459+0000","last_fullsized":"2024-04-24T08:10:36.967459+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:18:06.108578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967205+0000","last_change":"2024-04-24T08:10:35.970617+0000","last_active":"2024-04-24T08:10:36.967205+0000","last_peered":"2024-04-24T08:10:36.967205+0000","last_clean":"2024-04-24T08:10:36.967205+0000","last_became_active":"2024-04-24T08:10:35.970260+0000","last_became_peered":"2024-04-24T08:10:35.970260+0000","last_unstale":"2024-04-24T08:10:36.967205+0000","last_undegraded":"2024-04-24T08:10:36.967205+0000","last_fullsized":"2024-04-24T08:10:36.967205+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:14:36.018715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181185+0000","last_change":"2024-04-24T08:10:34.965942+0000","last_active":"2024-04-24T08:10:37.181185+0000","last_peered":"2024-04-24T08:10:37.181185+0000","last_clean":"2024-04-24T08:10:37.181185+0000","last_became_active":"2024-04-24T08:10:34.965688+0000","last_became_peered":"2024-04-24T08:10:34.965688+0000","last_unstale":"2024-04-24T08:10:37.181185+0000","last_undegraded":"2024-04-24T08:10:37.181185+0000","last_fullsized":"2024-04-24T08:10:37.181185+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:27:16.175478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183076+0000","last_change":"2024-04-24T08:10:35.968840+0000","last_active":"2024-04-24T08:10:37.183076+0000","last_peered":"2024-04-24T08:10:37.183076+0000","last_clean":"2024-04-24T08:10:37.183076+0000","last_became_active":"2024-04-24T08:10:35.968558+0000","last_became_peered":"2024-04-24T08:10:35.968558+0000","last_unstale":"2024-04-24T08:10:37.183076+0000","last_undegraded":"2024-04-24T08:10:37.183076+0000","last_fullsized":"2024-04-24T08:10:37.183076+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:40:50.268468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183117+0000","last_change":"2024-04-24T08:10:34.969611+0000","last_active":"2024-04-24T08:10:37.183117+0000","last_peered":"2024-04-24T08:10:37.183117+0000","last_clean":"2024-04-24T08:10:37.183117+0000","last_became_active":"2024-04-24T08:10:34.969312+0000","last_became_peered":"2024-04-24T08:10:34.969312+0000","last_unstale":"2024-04-24T08:10:37.183117+0000","last_undegraded":"2024-04-24T08:10:37.183117+0000","last_fullsized":"2024-04-24T08:10:37.183117+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:01:17.732036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.971781+0000","last_change":"2024-04-24T08:10:35.972414+0000","last_active":"2024-04-24T08:10:36.971781+0000","last_peered":"2024-04-24T08:10:36.971781+0000","last_clean":"2024-04-24T08:10:36.971781+0000","last_became_active":"2024-04-24T08:10:35.971294+0000","last_became_peered":"2024-04-24T08:10:35.971294+0000","last_unstale":"2024-04-24T08:10:36.971781+0000","last_undegraded":"2024-04-24T08:10:36.971781+0000","last_fullsized":"2024-04-24T08:10:36.971781+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:57:09.305687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"44'1","reported_seq":15,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.950047+0000","last_change":"2024-04-24T08:10:34.964190+0000","last_active":"2024-04-24T08:10:37.950047+0000","last_peered":"2024-04-24T08:10:37.950047+0000","last_clean":"2024-04-24T08:10:37.950047+0000","last_became_active":"2024-04-24T08:10:34.963833+0000","last_became_peered":"2024-04-24T08:10:34.963833+0000","last_unstale":"2024-04-24T08:10:37.950047+0000","last_undegraded":"2024-04-24T08:10:37.950047+0000","last_fullsized":"2024-04-24T08:10:37.950047+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:30:46.186572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.182781+0000","last_change":"2024-04-24T08:10:35.965299+0000","last_active":"2024-04-24T08:10:37.182781+0000","last_peered":"2024-04-24T08:10:37.182781+0000","last_clean":"2024-04-24T08:10:37.182781+0000","last_became_active":"2024-04-24T08:10:35.963358+0000","last_became_peered":"2024-04-24T08:10:35.963358+0000","last_unstale":"2024-04-24T08:10:37.182781+0000","last_undegraded":"2024-04-24T08:10:37.182781+0000","last_fullsized":"2024-04-24T08:10:37.182781+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:58:43.254440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.184028+0000","last_change":"2024-04-24T08:10:34.972453+0000","last_active":"2024-04-24T08:10:37.184028+0000","last_peered":"2024-04-24T08:10:37.184028+0000","last_clean":"2024-04-24T08:10:37.184028+0000","last_became_active":"2024-04-24T08:10:34.972143+0000","last_became_peered":"2024-04-24T08:10:34.972143+0000","last_unstale":"2024-04-24T08:10:37.184028+0000","last_undegraded":"2024-04-24T08:10:37.184028+0000","last_fullsized":"2024-04-24T08:10:37.184028+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:01:58.016903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183820+0000","last_change":"2024-04-24T08:10:35.973578+0000","last_active":"2024-04-24T08:10:37.183820+0000","last_peered":"2024-04-24T08:10:37.183820+0000","last_clean":"2024-04-24T08:10:37.183820+0000","last_became_active":"2024-04-24T08:10:35.973290+0000","last_became_peered":"2024-04-24T08:10:35.973290+0000","last_unstale":"2024-04-24T08:10:37.183820+0000","last_undegraded":"2024-04-24T08:10:37.183820+0000","last_fullsized":"2024-04-24T08:10:37.183820+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:23:53.251345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181420+0000","last_change":"2024-04-24T08:10:34.956279+0000","last_active":"2024-04-24T08:10:37.181420+0000","last_peered":"2024-04-24T08:10:37.181420+0000","last_clean":"2024-04-24T08:10:37.181420+0000","last_became_active":"2024-04-24T08:10:34.955986+0000","last_became_peered":"2024-04-24T08:10:34.955986+0000","last_unstale":"2024-04-24T08:10:37.181420+0000","last_undegraded":"2024-04-24T08:10:37.181420+0000","last_fullsized":"2024-04-24T08:10:37.181420+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:33:51.926096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.184763+0000","last_change":"2024-04-24T08:10:35.972614+0000","last_active":"2024-04-24T08:10:37.184763+0000","last_peered":"2024-04-24T08:10:37.184763+0000","last_clean":"2024-04-24T08:10:37.184763+0000","last_became_active":"2024-04-24T08:10:35.972193+0000","last_became_peered":"2024-04-24T08:10:35.972193+0000","last_unstale":"2024-04-24T08:10:37.184763+0000","last_undegraded":"2024-04-24T08:10:37.184763+0000","last_fullsized":"2024-04-24T08:10:37.184763+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:28.637770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183562+0000","last_change":"2024-04-24T08:10:34.972888+0000","last_active":"2024-04-24T08:10:37.183562+0000","last_peered":"2024-04-24T08:10:37.183562+0000","last_clean":"2024-04-24T08:10:37.183562+0000","last_became_active":"2024-04-24T08:10:34.972068+0000","last_became_peered":"2024-04-24T08:10:34.972068+0000","last_unstale":"2024-04-24T08:10:37.183562+0000","last_undegraded":"2024-04-24T08:10:37.183562+0000","last_fullsized":"2024-04-24T08:10:37.183562+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:54:44.282125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.185331+0000","last_change":"2024-04-24T08:10:35.964059+0000","last_active":"2024-04-24T08:10:37.185331+0000","last_peered":"2024-04-24T08:10:37.185331+0000","last_clean":"2024-04-24T08:10:37.185331+0000","last_became_active":"2024-04-24T08:10:35.963769+0000","last_became_peered":"2024-04-24T08:10:35.963769+0000","last_unstale":"2024-04-24T08:10:37.185331+0000","last_undegraded":"2024-04-24T08:10:37.185331+0000","last_fullsized":"2024-04-24T08:10:37.185331+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:47:42.446716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181600+0000","last_change":"2024-04-24T08:10:34.958374+0000","last_active":"2024-04-24T08:10:37.181600+0000","last_peered":"2024-04-24T08:10:37.181600+0000","last_clean":"2024-04-24T08:10:37.181600+0000","last_became_active":"2024-04-24T08:10:34.958090+0000","last_became_peered":"2024-04-24T08:10:34.958090+0000","last_unstale":"2024-04-24T08:10:37.181600+0000","last_undegraded":"2024-04-24T08:10:37.181600+0000","last_fullsized":"2024-04-24T08:10:37.181600+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:33:51.926096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.967880+0000","last_change":"2024-04-24T08:10:35.967122+0000","last_active":"2024-04-24T08:10:36.967880+0000","last_peered":"2024-04-24T08:10:36.967880+0000","last_clean":"2024-04-24T08:10:36.967880+0000","last_became_active":"2024-04-24T08:10:35.966705+0000","last_became_peered":"2024-04-24T08:10:35.966705+0000","last_unstale":"2024-04-24T08:10:36.967880+0000","last_undegraded":"2024-04-24T08:10:36.967880+0000","last_fullsized":"2024-04-24T08:10:36.967880+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:35:34.379491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181994+0000","last_change":"2024-04-24T08:10:34.961446+0000","last_active":"2024-04-24T08:10:37.181994+0000","last_peered":"2024-04-24T08:10:37.181994+0000","last_clean":"2024-04-24T08:10:37.181994+0000","last_became_active":"2024-04-24T08:10:34.961135+0000","last_became_peered":"2024-04-24T08:10:34.961135+0000","last_unstale":"2024-04-24T08:10:37.181994+0000","last_undegraded":"2024-04-24T08:10:37.181994+0000","last_fullsized":"2024-04-24T08:10:37.181994+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:50:15.371818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.181103+0000","last_change":"2024-04-24T08:10:35.962884+0000","last_active":"2024-04-24T08:10:37.181103+0000","last_peered":"2024-04-24T08:10:37.181103+0000","last_clean":"2024-04-24T08:10:37.181103+0000","last_became_active":"2024-04-24T08:10:35.962558+0000","last_became_peered":"2024-04-24T08:10:35.962558+0000","last_unstale":"2024-04-24T08:10:37.181103+0000","last_undegraded":"2024-04-24T08:10:37.181103+0000","last_fullsized":"2024-04-24T08:10:37.181103+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:01:18.747814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":14,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183016+0000","last_change":"2024-04-24T08:10:34.957031+0000","last_active":"2024-04-24T08:10:37.183016+0000","last_peered":"2024-04-24T08:10:37.183016+0000","last_clean":"2024-04-24T08:10:37.183016+0000","last_became_active":"2024-04-24T08:10:34.956363+0000","last_became_peered":"2024-04-24T08:10:34.956363+0000","last_unstale":"2024-04-24T08:10:37.183016+0000","last_undegraded":"2024-04-24T08:10:37.183016+0000","last_fullsized":"2024-04-24T08:10:37.183016+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:33:51.926096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.183137+0000","last_change":"2024-04-24T08:10:35.973508+0000","last_active":"2024-04-24T08:10:37.183137+0000","last_peered":"2024-04-24T08:10:37.183137+0000","last_clean":"2024-04-24T08:10:37.183137+0000","last_became_active":"2024-04-24T08:10:35.970886+0000","last_became_peered":"2024-04-24T08:10:35.970886+0000","last_unstale":"2024-04-24T08:10:37.183137+0000","last_undegraded":"2024-04-24T08:10:37.183137+0000","last_fullsized":"2024-04-24T08:10:37.183137+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:34.915692+0000","last_clean_scrub_stamp":"2024-04-24T08:10:34.915692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:10:54.133150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"43'2","reported_seq":16,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:36.968506+0000","last_change":"2024-04-24T08:10:34.961134+0000","last_active":"2024-04-24T08:10:36.968506+0000","last_peered":"2024-04-24T08:10:36.968506+0000","last_clean":"2024-04-24T08:10:36.968506+0000","last_became_active":"2024-04-24T08:10:34.959564+0000","last_became_peered":"2024-04-24T08:10:34.959564+0000","last_unstale":"2024-04-24T08:10:36.968506+0000","last_undegraded":"2024-04-24T08:10:36.968506+0000","last_fullsized":"2024-04-24T08:10:36.968506+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:10:33.899914+0000","last_clean_scrub_stamp":"2024-04-24T08:10:33.899914+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:39:11.026779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":213,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-24T08:10:37.182866+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:10:37.182866+0000","last_peered":"2024-04-24T08:10:37.182866+0000","last_clean":"2024-04-24T08:10:37.182866+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:10:37.182866+0000","last_undegraded":"2024-04-24T08:10:37.182866+0000","last_fullsized":"2024-04-24T08:10:37.182866+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343732,"statfs":{"total":96636764160,"available":96607981568,"internally_reserved":0,"allocated":1114112,"data_stored":800420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":524288,"data_stored":210634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":210032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":528384,"data_stored":211574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":528384,"data_stored":210600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":532480,"data_stored":211596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27532,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":520192,"data_stored":210032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1114112,"data_stored":800364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:10:40.851 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T08:10:40.851 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-24T08:10:41.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.093+0000 7f1bbbe00640 1 Processor -- start 2024-04-24T08:10:41.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.093+0000 7f1bbbe00640 1 -- start start 2024-04-24T08:10:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.095+0000 7f1bbbe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1bbc0d0a60 0x7f1bbc0cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.095+0000 7f1bbbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 0x7f1bbc162420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.095+0000 7f1bbbe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bbc1631a0 con 0x7f1bbc15f550 2024-04-24T08:10:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.095+0000 7f1bbbe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1bbc0cf060 con 0x7f1bbc0d0a60 2024-04-24T08:10:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.095+0000 7f1bbbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1bbc0cf340 con 0x7f1bbc15f930 2024-04-24T08:10:41.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.095+0000 7f1bb3e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1bbc0d0a60 0x7f1bbc0cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.096+0000 7f1bbb400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 0x7f1bbc162420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.096+0000 7f1bb3e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1bbc0d0a60 0x7f1bbc0cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:58296/0 (socket says 172.21.15.69:58296) 2024-04-24T08:10:41.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.096+0000 7f1bb3e00640 1 -- 172.21.15.69:0/2206935341 learned_addr learned my addr 172.21.15.69:0/2206935341 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:41.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.096+0000 7f1bbb400640 1 -- 172.21.15.69:0/2206935341 >> v2:172.21.15.121:3300/0 conn(0x7f1bbc0d0a60 msgr2=0x7f1bbc0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:41.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.096+0000 7f1bbb400640 1 --2- 172.21.15.69:0/2206935341 >> v2:172.21.15.121:3300/0 conn(0x7f1bbc0d0a60 0x7f1bbc0cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.096+0000 7f1bbb400640 1 -- 172.21.15.69:0/2206935341 >> v1:172.21.15.121:6789/0 conn(0x7f1bbc15f550 legacy=0x7f1bbc0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.096+0000 7f1bbb400640 1 -- 172.21.15.69:0/2206935341 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bbc0cf620 con 0x7f1bbc15f930 2024-04-24T08:10:41.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.096+0000 7f1bb3e00640 1 --2- 172.21.15.69:0/2206935341 >> v2:172.21.15.121:3300/0 conn(0x7f1bbc0d0a60 0x7f1bbc0cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:41.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.098+0000 7f1bbb400640 1 --2- 172.21.15.69:0/2206935341 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 0x7f1bbc162420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009d90 tx=0x7f1bb402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a6572df6b1ce604 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:41.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.098+0000 7f1bba000640 1 -- 172.21.15.69:0/2206935341 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4031600 con 0x7f1bbc15f930 2024-04-24T08:10:41.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.098+0000 7f1bba000640 1 -- 172.21.15.69:0/2206935341 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bb40317a0 con 0x7f1bbc15f930 2024-04-24T08:10:41.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.098+0000 7f1bba000640 1 -- 172.21.15.69:0/2206935341 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb4031b70 con 0x7f1bbc15f930 2024-04-24T08:10:41.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.099+0000 7f1bbbe00640 1 -- 172.21.15.69:0/2206935341 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 msgr2=0x7f1bbc162420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.099+0000 7f1bbbe00640 1 --2- 172.21.15.69:0/2206935341 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 0x7f1bbc162420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4009d90 tx=0x7f1bb402a5d0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.099+0000 7f1bbbe00640 1 -- 172.21.15.69:0/2206935341 shutdown_connections 2024-04-24T08:10:41.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.099+0000 7f1bbbe00640 1 --2- 172.21.15.69:0/2206935341 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 0x7f1bbc162420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.099+0000 7f1bbbe00640 1 --2- 172.21.15.69:0/2206935341 >> v2:172.21.15.121:3300/0 conn(0x7f1bbc0d0a60 0x7f1bbc0cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.099+0000 7f1bbbe00640 1 -- 172.21.15.69:0/2206935341 >> 172.21.15.69:0/2206935341 conn(0x7f1bbc06bc00 msgr2=0x7f1bbc06c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:41.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.099+0000 7f1bbbe00640 1 -- 172.21.15.69:0/2206935341 shutdown_connections 2024-04-24T08:10:41.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.099+0000 7f1bbbe00640 1 -- 172.21.15.69:0/2206935341 wait complete. 2024-04-24T08:10:41.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.101+0000 7f1bbbe00640 1 Processor -- start 2024-04-24T08:10:41.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.101+0000 7f1bbbe00640 1 -- start start 2024-04-24T08:10:41.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.102+0000 7f1bbbe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bbc0d0a60 0x7f1bbc099a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.102+0000 7f1bbbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bbc15f550 0x7f1bbc0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.102+0000 7f1bbbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 0x7f1bbc15fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.102+0000 7f1bbbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc162ae0 con 0x7f1bbc15f550 2024-04-24T08:10:41.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.102+0000 7f1bbbe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1bbc162960 con 0x7f1bbc0d0a60 2024-04-24T08:10:41.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.102+0000 7f1bbbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1bbc162c60 con 0x7f1bbc15f930 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.102+0000 7f1bb3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bbc15f550 0x7f1bbc0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.102+0000 7f1bb3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bbc15f550 0x7f1bbc0c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57876/0 (socket says 172.21.15.69:57876) 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.102+0000 7f1bb3e00640 1 -- 172.21.15.69:0/953872008 learned_addr learned my addr 172.21.15.69:0/953872008 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bbaa00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bbc0d0a60 0x7f1bbc099a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bbb400640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 0x7f1bbc15fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bb3e00640 1 -- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 msgr2=0x7f1bbc15fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bb3e00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 0x7f1bbc15fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bb3e00640 1 -- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bbc0d0a60 msgr2=0x7f1bbc099a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bb3e00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bbc0d0a60 0x7f1bbc099a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bb3e00640 1 -- 172.21.15.69:0/953872008 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bbc160720 con 0x7f1bbc15f550 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bbb400640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 0x7f1bbc15fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bb3e00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bbc15f550 0x7f1bbc0c9a10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400e8d0 tx=0x7f1ba400ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bb2a00640 1 -- 172.21.15.69:0/953872008 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4017070 con 0x7f1bbc15f550 2024-04-24T08:10:41.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bb2a00640 1 -- 172.21.15.69:0/953872008 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba400c940 con 0x7f1bbc15f550 2024-04-24T08:10:41.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.103+0000 7f1bb2a00640 1 -- 172.21.15.69:0/953872008 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba400cd30 con 0x7f1bbc15f550 2024-04-24T08:10:41.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.104+0000 7f1bbbe00640 1 -- 172.21.15.69:0/953872008 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bbc160a00 con 0x7f1bbc15f550 2024-04-24T08:10:41.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.104+0000 7f1bbbe00640 1 -- 172.21.15.69:0/953872008 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bbc0c9f50 con 0x7f1bbc15f550 2024-04-24T08:10:41.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.105+0000 7f1bbbe00640 1 -- 172.21.15.69:0/953872008 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bbc075d50 con 0x7f1bbc15f550 2024-04-24T08:10:41.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.106+0000 7f1bb2a00640 1 -- 172.21.15.69:0/953872008 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1ba401a070 con 0x7f1bbc15f550 2024-04-24T08:10:41.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.107+0000 7f1bb2a00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b9806c8f0 0x7f1b9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.107+0000 7f1bb2a00640 1 -- 172.21.15.69:0/953872008 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f1ba408be90 con 0x7f1bbc15f550 2024-04-24T08:10:41.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.107+0000 7f1bbaa00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b9806c8f0 0x7f1b9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.108+0000 7f1bbaa00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b9806c8f0 0x7f1b9806edb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1bac000f60 tx=0x7f1bac009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:41.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.112+0000 7f1bb2a00640 1 -- 172.21.15.69:0/953872008 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ba405a190 con 0x7f1bbc15f550 2024-04-24T08:10:41.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.383+0000 7f1bbbe00640 1 -- 172.21.15.69:0/953872008 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f1bbc06e360 con 0x7f1bbc15f550 2024-04-24T08:10:41.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.388+0000 7f1bb2a00640 1 -- 172.21.15.69:0/953872008 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f1ba408d0e0 con 0x7f1bbc15f550 2024-04-24T08:10:41.390 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-24T08:10:41.390 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw== 2024-04-24T08:10:41.390 INFO:teuthology.orchestra.run.smithi069.stdout: caps mds = "allow" 2024-04-24T08:10:41.390 INFO:teuthology.orchestra.run.smithi069.stdout: caps mon = "allow r" 2024-04-24T08:10:41.390 INFO:teuthology.orchestra.run.smithi069.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-24T08:10:41.390 INFO:teuthology.orchestra.run.smithi069.stderr:updated caps for client.0 2024-04-24T08:10:41.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.390+0000 7f1bbbe00640 1 -- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b9806c8f0 msgr2=0x7f1b9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b9806c8f0 0x7f1b9806edb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1bac000f60 tx=0x7f1bac009040 comp rx=0 tx=0).stop 2024-04-24T08:10:41.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 -- 172.21.15.69:0/953872008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bbc15f550 msgr2=0x7f1bbc0c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bbc15f550 0x7f1bbc0c9a10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f1ba400e8d0 tx=0x7f1ba400ed90 comp rx=0 tx=0).stop 2024-04-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 -- 172.21.15.69:0/953872008 shutdown_connections 2024-04-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b9806c8f0 0x7f1b9806edb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bbc15f550 0x7f1bbc0c9a10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1bbc15f930 0x7f1bbc15fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 --2- 172.21.15.69:0/953872008 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bbc0d0a60 0x7f1bbc099a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 -- 172.21.15.69:0/953872008 >> 172.21.15.69:0/953872008 conn(0x7f1bbc06bc00 msgr2=0x7f1bbc06bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 -- 172.21.15.69:0/953872008 shutdown_connections 2024-04-24T08:10:41.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.391+0000 7f1bbbe00640 1 -- 172.21.15.69:0/953872008 wait complete. 2024-04-24T08:10:41.412 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:41.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.612+0000 7f049f200640 1 Processor -- start 2024-04-24T08:10:41.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.612+0000 7f049f200640 1 -- start start 2024-04-24T08:10:41.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04a009b0e0 0x7f04a00b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04a015ed50 0x7f04a009b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049f200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04a00d1320 con 0x7f04a015e970 2024-04-24T08:10:41.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f04a009bc50 con 0x7f04a009b0e0 2024-04-24T08:10:41.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049f200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f04a00cc9f0 con 0x7f04a015ed50 2024-04-24T08:10:41.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049d400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04a009b0e0 0x7f04a00b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049d400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04a009b0e0 0x7f04a00b35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57886/0 (socket says 172.21.15.69:57886) 2024-04-24T08:10:41.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049d400640 1 -- 172.21.15.69:0/573301675 learned_addr learned my addr 172.21.15.69:0/573301675 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:41.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049d400640 1 -- 172.21.15.69:0/573301675 >> v2:172.21.15.121:3300/0 conn(0x7f04a015ed50 msgr2=0x7f04a009b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:41.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049d400640 1 --2- 172.21.15.69:0/573301675 >> v2:172.21.15.121:3300/0 conn(0x7f04a015ed50 0x7f04a009b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049d400640 1 -- 172.21.15.69:0/573301675 >> v1:172.21.15.121:6789/0 conn(0x7f04a015e970 legacy=0x7f04a009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:41.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.614+0000 7f049d400640 1 -- 172.21.15.69:0/573301675 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a00d1600 con 0x7f04a009b0e0 2024-04-24T08:10:41.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.615+0000 7f049d400640 1 --2- 172.21.15.69:0/573301675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 0x7f04a00b35f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f04a00d05c0 tx=0x7f048c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71baf99b0dc834b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:41.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.616+0000 7f049ca00640 1 -- 172.21.15.69:0/573301675 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c037070 con 0x7f04a009b0e0 2024-04-24T08:10:41.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.616+0000 7f049ca00640 1 -- 172.21.15.69:0/573301675 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f048c032440 con 0x7f04a009b0e0 2024-04-24T08:10:41.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.616+0000 7f049ca00640 1 -- 172.21.15.69:0/573301675 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f048c032810 con 0x7f04a009b0e0 2024-04-24T08:10:41.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.616+0000 7f049f200640 1 -- 172.21.15.69:0/573301675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 msgr2=0x7f04a00b35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.616+0000 7f049f200640 1 --2- 172.21.15.69:0/573301675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 0x7f04a00b35f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f04a00d05c0 tx=0x7f048c02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.616+0000 7f049f200640 1 -- 172.21.15.69:0/573301675 shutdown_connections 2024-04-24T08:10:41.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.616+0000 7f049f200640 1 --2- 172.21.15.69:0/573301675 >> v2:172.21.15.121:3300/0 conn(0x7f04a015ed50 0x7f04a009b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.616+0000 7f049f200640 1 --2- 172.21.15.69:0/573301675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 0x7f04a00b35f0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.617+0000 7f049f200640 1 -- 172.21.15.69:0/573301675 >> 172.21.15.69:0/573301675 conn(0x7f04a00b3e90 msgr2=0x7f04a00b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:41.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.617+0000 7f049f200640 1 -- 172.21.15.69:0/573301675 shutdown_connections 2024-04-24T08:10:41.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.617+0000 7f049f200640 1 -- 172.21.15.69:0/573301675 wait complete. 2024-04-24T08:10:41.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.619+0000 7f049f200640 1 Processor -- start 2024-04-24T08:10:41.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.619+0000 7f049f200640 1 -- start start 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.620+0000 7f049f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 0x7f04a0151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.620+0000 7f049f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a015e970 0x7f04a0161a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.620+0000 7f049f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a015ed50 0x7f04a00ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.620+0000 7f049f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04a00cd000 con 0x7f04a009b0e0 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.620+0000 7f049f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04a00cce80 con 0x7f04a015e970 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.620+0000 7f049f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04a00cd180 con 0x7f04a015ed50 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.620+0000 7f049de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 0x7f04a0151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.620+0000 7f049de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 0x7f04a0151e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57892/0 (socket says 172.21.15.69:57892) 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.620+0000 7f049de00640 1 -- 172.21.15.69:0/2834024071 learned_addr learned my addr 172.21.15.69:0/2834024071 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f049e800640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a015ed50 0x7f04a00ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f049de00640 1 -- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a015ed50 msgr2=0x7f04a00ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f049de00640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a015ed50 0x7f04a00ca5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f049de00640 1 -- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a015e970 msgr2=0x7f04a0161a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f049d400640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a015e970 0x7f04a0161a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f049de00640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a015e970 0x7f04a0161a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f049de00640 1 -- 172.21.15.69:0/2834024071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a00cb280 con 0x7f04a009b0e0 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f049de00640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 0x7f04a0151e80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f049000c400 tx=0x7f049000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f0496000640 1 -- 172.21.15.69:0/2834024071 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490017070 con 0x7f04a009b0e0 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f0496000640 1 -- 172.21.15.69:0/2834024071 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0490003640 con 0x7f04a009b0e0 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f0496000640 1 -- 172.21.15.69:0/2834024071 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490003a30 con 0x7f04a009b0e0 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.621+0000 7f049f200640 1 -- 172.21.15.69:0/2834024071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a00cb560 con 0x7f04a009b0e0 2024-04-24T08:10:41.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.622+0000 7f049f200640 1 -- 172.21.15.69:0/2834024071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a00c5ed0 con 0x7f04a009b0e0 2024-04-24T08:10:41.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.623+0000 7f049f200640 1 -- 172.21.15.69:0/2834024071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a0075d50 con 0x7f04a009b0e0 2024-04-24T08:10:41.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.624+0000 7f0496000640 1 -- 172.21.15.69:0/2834024071 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f049001a070 con 0x7f04a009b0e0 2024-04-24T08:10:41.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.625+0000 7f0496000640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048006c8f0 0x7f048006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:41.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.625+0000 7f0496000640 1 -- 172.21.15.69:0/2834024071 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f049008afd0 con 0x7f04a009b0e0 2024-04-24T08:10:41.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.625+0000 7f049d400640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048006c8f0 0x7f048006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:41.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.627+0000 7f049d400640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048006c8f0 0x7f048006edb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f048c034000 tx=0x7f048c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:41.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.629+0000 7f0496000640 1 -- 172.21.15.69:0/2834024071 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04900593f0 con 0x7f04a009b0e0 2024-04-24T08:10:41.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.868+0000 7f049f200640 1 -- 172.21.15.69:0/2834024071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04a00d1f00 con 0x7f04a009b0e0 2024-04-24T08:10:41.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.869+0000 7f0496000640 1 -- 172.21.15.69:0/2834024071 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5184 (secure 0 0 0) 0x7f0490058cf0 con 0x7f04a009b0e0 2024-04-24T08:10:41.871 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:41.871 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/376722301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":376722301},{"type":"v1","addr":"172.21.15.69:6835","nonce":376722301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4990,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1327624292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1327624292},{"type":"v1","addr":"172.21.15.121:6836","nonce":1327624292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/3244002416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3244002416},{"type":"v1","addr":"172.21.15.69:6834","nonce":3244002416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:10:35.400788+0000","modified":"2024-04-24T08:10:38.457641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5793},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5793":{"gid":5793,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/3424934990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3424934990},{"type":"v1","addr":"172.21.15.121:6837","nonce":3424934990}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5793,"qdb_cluster":[5793]},"id":3}]} 2024-04-24T08:10:41.871 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 -- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048006c8f0 msgr2=0x7f048006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048006c8f0 0x7f048006edb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f048c034000 tx=0x7f048c02e000 comp rx=0 tx=0).stop 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 -- 172.21.15.69:0/2834024071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 msgr2=0x7f04a0151e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 0x7f04a0151e80 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f049000c400 tx=0x7f049000c8c0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 -- 172.21.15.69:0/2834024071 shutdown_connections 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048006c8f0 0x7f048006edb0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a015ed50 0x7f04a00ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a015e970 0x7f04a0161a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 --2- 172.21.15.69:0/2834024071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a009b0e0 0x7f04a0151e80 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 -- 172.21.15.69:0/2834024071 >> 172.21.15.69:0/2834024071 conn(0x7f04a00b3e90 msgr2=0x7f04a0071720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 -- 172.21.15.69:0/2834024071 shutdown_connections 2024-04-24T08:10:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:41.872+0000 7f049f200640 1 -- 172.21.15.69:0/2834024071 wait complete. 2024-04-24T08:10:41.903 INFO:tasks.ceph.mds.d.smithi121.stderr:2024-04-24T08:10:41.902+0000 7f612ce00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:10:42.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:10:35.400788+0000', 'modified': '2024-04-24T08:10:38.457641+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5793}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5793': {'gid': 5793, 'name': 'd', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6837/3424934990', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6835', 'nonce': 3424934990}, {'type': 'v1', 'addr': '172.21.15.121:6837', 'nonce': 3424934990}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5793, 'qdb_cluster': [5793]} 2024-04-24T08:10:42.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:10:42.296 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-24T08:10:42.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.506+0000 7f7a12600640 1 Processor -- start 2024-04-24T08:10:42.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.506+0000 7f7a12600640 1 -- start start 2024-04-24T08:10:42.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.508+0000 7f7a12600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7a0c15b490 0x7f7a0c0b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.508+0000 7f7a12600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a0c0b46e0 con 0x7f7a0c0d02f0 2024-04-24T08:10:42.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.508+0000 7f7a12600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a0c0d0960 con 0x7f7a0c06f2d0 2024-04-24T08:10:42.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.508+0000 7f7a12600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7a0c0d0720 con 0x7f7a0c15b490 2024-04-24T08:10:42.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.508+0000 7f7a0be00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7a0c0d02f0 0x7f7a0c0ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49276/0 (socket says 172.21.15.69:49276) 2024-04-24T08:10:42.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.508+0000 7f7a0be00640 1 -- 172.21.15.69:0/1859043875 learned_addr learned my addr 172.21.15.69:0/1859043875 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0b400640 1 --2- 172.21.15.69:0/1859043875 >> v2:172.21.15.121:3300/0 conn(0x7f7a0c15b490 0x7f7a0c0b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1525017059 0 0) 0x7f7a0c0b46e0 con 0x7f7a0c0d02f0 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79f0003520 con 0x7f7a0c0d02f0 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1495806910 0 0) 0x7f79f0003520 con 0x7f7a0c0d02f0 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 >> v2:172.21.15.121:3300/0 conn(0x7f7a0c15b490 msgr2=0x7f7a0c0b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 --2- 172.21.15.69:0/1859043875 >> v2:172.21.15.121:3300/0 conn(0x7f7a0c15b490 0x7f7a0c0b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 >> v1:172.21.15.121:6789/0 conn(0x7f7a0c06f2d0 legacy=0x7f7a0c0cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0c0b4980 con 0x7f7a0c0d02f0 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f7a04001a10 con 0x7f7a0c0d02f0 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 >> v1:172.21.15.69:6789/0 conn(0x7f7a0c0d02f0 legacy=0x7f7a0c0ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79f00039b0 0x7f79f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f79f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f00051c0 0x7f79f000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0c0d0720 con 0x7f79f0004800 2024-04-24T08:10:42.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f79f000e220 con 0x7f79f00039b0 2024-04-24T08:10:42.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.509+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f79f000e420 con 0x7f79f00051c0 2024-04-24T08:10:42.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.510+0000 7f7a03e00640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f79f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.510+0000 7f7a0b400640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79f00039b0 0x7f79f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.510+0000 7f7a0be00640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f00051c0 0x7f79f000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.510+0000 7f7a03e00640 1 -- 172.21.15.69:0/1859043875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f00051c0 msgr2=0x7f79f000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.510+0000 7f7a03e00640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f00051c0 0x7f79f000dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.510+0000 7f7a03e00640 1 -- 172.21.15.69:0/1859043875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79f00039b0 msgr2=0x7f79f0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.510+0000 7f7a03e00640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79f00039b0 0x7f79f0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.510+0000 7f7a03e00640 1 -- 172.21.15.69:0/1859043875 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0c0b4980 con 0x7f79f0004800 2024-04-24T08:10:42.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.510+0000 7f7a0be00640 1 -- 172.21.15.69:0/1859043875 reap_dead start 2024-04-24T08:10:42.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.511+0000 7f7a03e00640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f79f0004c80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f79f4001750 tx=0x7f79f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c0377d0ff5ac2e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.511+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f4001370 con 0x7f79f0004800 2024-04-24T08:10:42.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.512+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79f40052d0 con 0x7f79f0004800 2024-04-24T08:10:42.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.512+0000 7f7a0aa00640 1 -- 172.21.15.69:0/1859043875 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f402abc0 con 0x7f79f0004800 2024-04-24T08:10:42.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.512+0000 7f7a12600640 1 -- 172.21.15.69:0/1859043875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 msgr2=0x7f79f0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.512+0000 7f7a12600640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f79f0004c80 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f79f4001750 tx=0x7f79f0005830 comp rx=0 tx=0).stop 2024-04-24T08:10:42.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.512+0000 7f7a12600640 1 -- 172.21.15.69:0/1859043875 shutdown_connections 2024-04-24T08:10:42.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.512+0000 7f7a12600640 1 --2- 172.21.15.69:0/1859043875 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f79f0004c80 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.512+0000 7f7a12600640 1 -- 172.21.15.69:0/1859043875 >> 172.21.15.69:0/1859043875 conn(0x7f7a0c067d90 msgr2=0x7f7a0c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:42.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.512+0000 7f7a12600640 1 -- 172.21.15.69:0/1859043875 shutdown_connections 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.512+0000 7f7a12600640 1 -- 172.21.15.69:0/1859043875 wait complete. 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.514+0000 7f7a12600640 1 Processor -- start 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.514+0000 7f7a12600640 1 -- start start 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.515+0000 7f7a12600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f7a0c15df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.515+0000 7f7a12600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a0c0c6f90 0x7f7a0c0b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.515+0000 7f7a12600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a0c0b3ea0 0x7f7a0c0b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.515+0000 7f7a12600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0c15b870 con 0x7f79f0004800 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.515+0000 7f7a12600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0c0d0c40 con 0x7f7a0c0c6f90 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.515+0000 7f7a12600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a0c06f730 con 0x7f7a0c0b3ea0 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a0b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f7a0c15df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a0b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f7a0c15df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57906/0 (socket says 172.21.15.69:57906) 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a0b400640 1 -- 172.21.15.69:0/1398708569 learned_addr learned my addr 172.21.15.69:0/1398708569 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:42.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a0b400640 1 -- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a0c0b3ea0 msgr2=0x7f7a0c0b1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a0be00640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a0c0b3ea0 0x7f7a0c0b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a0b400640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a0c0b3ea0 0x7f7a0c0b1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a03e00640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a0c0c6f90 0x7f7a0c0b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a0b400640 1 -- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a0c0c6f90 msgr2=0x7f7a0c0b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a0b400640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a0c0c6f90 0x7f7a0c0b1340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a0b400640 1 -- 172.21.15.69:0/1398708569 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a0c0c5c50 con 0x7f79f0004800 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.516+0000 7f7a0b400640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f7a0c15df20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f79fc011540 tx=0x7f79fc011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.517+0000 7f7a03400640 1 -- 172.21.15.69:0/1398708569 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79fc015870 con 0x7f79f0004800 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.517+0000 7f7a03400640 1 -- 172.21.15.69:0/1398708569 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79fc015a10 con 0x7f79f0004800 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.517+0000 7f7a03400640 1 -- 172.21.15.69:0/1398708569 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79fc015dc0 con 0x7f79f0004800 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.517+0000 7f7a12600640 1 -- 172.21.15.69:0/1398708569 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a0c0c5ef0 con 0x7f79f0004800 2024-04-24T08:10:42.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.517+0000 7f7a12600640 1 -- 172.21.15.69:0/1398708569 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a0c0c64b0 con 0x7f79f0004800 2024-04-24T08:10:42.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.519+0000 7f7a03400640 1 -- 172.21.15.69:0/1398708569 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f79fc013400 con 0x7f79f0004800 2024-04-24T08:10:42.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.520+0000 7f7a03400640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79e806c8f0 0x7f79e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.520+0000 7f7a03400640 1 -- 172.21.15.69:0/1398708569 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f79fc08c480 con 0x7f79f0004800 2024-04-24T08:10:42.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.520+0000 7f7a03e00640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79e806c8f0 0x7f79e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.521+0000 7f7a15597740 1 Processor -- start 2024-04-24T08:10:42.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.521+0000 7f7a15597740 1 -- start start 2024-04-24T08:10:42.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a03e00640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79e806c8f0 0x7f79e806edb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f79f4032000 tx=0x7f79f402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a15597740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3b57b0 0x55631f368140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a15597740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f397c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a15597740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f398190 0x55631f36cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a15597740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55631f2d2b10 con 0x55631f3988d0 2024-04-24T08:10:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a15597740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55631f2792b0 con 0x55631f398190 2024-04-24T08:10:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a15597740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55631f2cead0 con 0x55631f3b57b0 2024-04-24T08:10:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a03e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f397c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a03e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f397c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57916/0 (socket says 172.21.15.69:57916) 2024-04-24T08:10:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a03e00640 1 -- 172.21.15.69:0/1619071659 learned_addr learned my addr 172.21.15.69:0/1619071659 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:42.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.522+0000 7f7a0be00640 1 --2- 172.21.15.69:0/1619071659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f398190 0x55631f36cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.523+0000 7f7a0b400640 1 --2- 172.21.15.69:0/1619071659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3b57b0 0x55631f368140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.523+0000 7f7a03e00640 1 -- 172.21.15.69:0/1619071659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3b57b0 msgr2=0x55631f368140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.523+0000 7f7a03e00640 1 --2- 172.21.15.69:0/1619071659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3b57b0 0x55631f368140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.523+0000 7f7a03e00640 1 -- 172.21.15.69:0/1619071659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f398190 msgr2=0x55631f36cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.523+0000 7f7a03e00640 1 --2- 172.21.15.69:0/1619071659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f398190 0x55631f36cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.523+0000 7f7a03e00640 1 -- 172.21.15.69:0/1619071659 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55631efe52d0 con 0x55631f3988d0 2024-04-24T08:10:42.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.523+0000 7f7a03e00640 1 --2- 172.21.15.69:0/1619071659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f397c50 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f79f402a570 tx=0x7f79f402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dcc5d16d0bb80f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.523+0000 7f79fbe00640 1 -- 172.21.15.69:0/1619071659 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f40073c0 con 0x55631f3988d0 2024-04-24T08:10:42.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.523+0000 7f79fbe00640 1 -- 172.21.15.69:0/1619071659 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79f4007560 con 0x55631f3988d0 2024-04-24T08:10:42.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.523+0000 7f79fbe00640 1 -- 172.21.15.69:0/1619071659 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79f40069d0 con 0x55631f3988d0 2024-04-24T08:10:42.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.524+0000 7f7a15597740 1 -- 172.21.15.69:0/1619071659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 msgr2=0x55631f397c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.524+0000 7f7a15597740 1 --2- 172.21.15.69:0/1619071659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f397c50 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f79f402a570 tx=0x7f79f402c000 comp rx=0 tx=0).stop 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.524+0000 7f7a15597740 1 -- 172.21.15.69:0/1619071659 shutdown_connections 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.524+0000 7f7a15597740 1 --2- 172.21.15.69:0/1619071659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f397c50 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.524+0000 7f7a15597740 1 --2- 172.21.15.69:0/1619071659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f398190 0x55631f36cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.524+0000 7f7a15597740 1 --2- 172.21.15.69:0/1619071659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3b57b0 0x55631f368140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.524+0000 7f7a15597740 1 -- 172.21.15.69:0/1619071659 >> 172.21.15.69:0/1619071659 conn(0x55631f36a3b0 msgr2=0x55631f3b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.524+0000 7f7a15597740 1 -- 172.21.15.69:0/1619071659 shutdown_connections 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.524+0000 7f7a15597740 1 -- 172.21.15.69:0/1619071659 wait complete. 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.525+0000 7f7a15597740 1 -- start start 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.525+0000 7f7a15597740 1 Processor -- start 2024-04-24T08:10:42.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.526+0000 7f7a15597740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3988d0 0x55631f3aa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.526+0000 7f7a15597740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 0x55631f3ab000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.526+0000 7f7a15597740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f3ab540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.526+0000 7f7a15597740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55631f36c560 con 0x55631f39a5b0 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.526+0000 7f7a15597740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55631f28c590 con 0x55631f3b57b0 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.526+0000 7f7a15597740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55631f36b950 con 0x55631f3988d0 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.526+0000 7f7a15597740 10 client.0 _ll_register_callbacks cb 0x55631f26f3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.526+0000 7f7a0be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f3ab540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a0be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f3ab540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57926/0 (socket says 172.21.15.69:57926) 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a0be00640 1 -- 172.21.15.69:0/4245720043 learned_addr learned my addr 172.21.15.69:0/4245720043 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a03e00640 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 0x55631f3ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a0b400640 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3988d0 0x55631f3aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a0be00640 1 -- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3988d0 msgr2=0x55631f3aa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a0be00640 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3988d0 0x55631f3aa870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a0be00640 1 -- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 msgr2=0x55631f3ab000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a0be00640 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 0x55631f3ab000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a0be00640 1 -- 172.21.15.69:0/4245720043 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55631f0af6b0 con 0x55631f39a5b0 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a03e00640 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 0x55631f3ab000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f7a0be00640 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f3ab540 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0cfa70 tx=0x7f7a04007a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15563566483f3aab server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f79f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.527+0000 7f79f8c00640 1 -- 172.21.15.69:0/4245720043 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a04010910 con 0x55631f39a5b0 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f79f8c00640 1 -- 172.21.15.69:0/4245720043 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f7a04010ab0 con 0x55631f39a5b0 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f7a15597740 1 -- 172.21.15.69:0/4245720043 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79f4007560 con 0x55631f39a5b0 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f79f8c00640 1 -- 172.21.15.69:0/4245720043 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7a04021050 con 0x55631f39a5b0 2024-04-24T08:10:42.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f7a15597740 10 client.5928 fetch_fsmap learned FSMap version 25 2024-04-24T08:10:42.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f7a15597740 1 -- 172.21.15.69:0/4245720043 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55631f399db0 con 0x55631f39a5b0 2024-04-24T08:10:42.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f79f8c00640 1 -- 172.21.15.69:0/4245720043 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79f4007560 con 0x55631f39a5b0 2024-04-24T08:10:42.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f79f8c00640 1 -- 172.21.15.69:0/4245720043 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f7a04024020 con 0x55631f39a5b0 2024-04-24T08:10:42.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f7a15597740 10 client.5928 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:10:42.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f7a15597740 10 client.5928 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-24T08:10:42.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f7a15597740 20 client.5928 populate_metadata read hostname 'smithi069' 2024-04-24T08:10:42.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f7a15597740 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55631f386290 0x55631f39b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f7a15597740 4 client.5928 mds_command: new command op to 5793 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:10:42.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.528+0000 7f7a15597740 1 -- 172.21.15.69:0/4245720043 --> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55631f2e0530 con 0x55631f386290 2024-04-24T08:10:42.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.529+0000 7f7a0b400640 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55631f386290 0x55631f39b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.530+0000 7f7a0b400640 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55631f386290 0x55631f39b6a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b833242b6a53ea7 server_cookie=19e0f7ed99e7960 in_seq=0 out_seq=0 2024-04-24T08:10:42.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.530+0000 7f79f8c00640 10 client.5928 ms_handle_connect on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:42.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f79f8c00640 1 -- 172.21.15.69:0/4245720043 <== mds.0 v2:172.21.15.121:6835/3424934990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55631f2e0530 con 0x55631f386290 2024-04-24T08:10:42.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f79f8c00640 10 client.5928 handle_command_reply: tid=0 2024-04-24T08:10:42.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 client.5928 shutdown 2024-04-24T08:10:42.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 -- 172.21.15.69:0/4245720043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 msgr2=0x55631f3ab540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f3ab540 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0cfa70 tx=0x7f7a04007a80 comp rx=0 tx=0).stop 2024-04-24T08:10:42.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 -- 172.21.15.69:0/4245720043 shutdown_connections 2024-04-24T08:10:42.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55631f386290 0x55631f39b6a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f3ab540 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 0x55631f3ab000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 --2- 172.21.15.69:0/4245720043 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3988d0 0x55631f3aa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f79f8c00640 0 client.5928 ms_handle_reset on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:42.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 -- 172.21.15.69:0/4245720043 >> 172.21.15.69:0/4245720043 conn(0x55631f398190 msgr2=0x55631f3b3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:42.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 -- 172.21.15.69:0/4245720043 shutdown_connections 2024-04-24T08:10:42.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 1 -- 172.21.15.69:0/4245720043 wait complete. 2024-04-24T08:10:42.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.532+0000 7f7a15597740 20 client.5928 trim_cache size 0 max 16384 2024-04-24T08:10:42.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.552+0000 7f7a15597740 1 Processor -- start 2024-04-24T08:10:42.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.552+0000 7f7a15597740 1 -- start start 2024-04-24T08:10:42.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.553+0000 7f7a15597740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f3c5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.553+0000 7f7a15597740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f39a5b0 0x55631f36ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.553+0000 7f7a15597740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 0x55631f36d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.553+0000 7f7a15597740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0c2006e0 con 0x55631f3988d0 2024-04-24T08:10:42.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.553+0000 7f7a15597740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a0c14f510 con 0x55631f3b57b0 2024-04-24T08:10:42.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.553+0000 7f7a15597740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a0c200090 con 0x55631f39a5b0 2024-04-24T08:10:42.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.553+0000 7f7a0b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f3c5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.553+0000 7f7a0b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f3c5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57930/0 (socket says 172.21.15.69:57930) 2024-04-24T08:10:42.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.553+0000 7f7a0b400640 1 -- 172.21.15.69:0/2139071420 learned_addr learned my addr 172.21.15.69:0/2139071420 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:42.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f7a0be00640 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 0x55631f36d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f7a0b400640 1 -- 172.21.15.69:0/2139071420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f39a5b0 msgr2=0x55631f36ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f7a03e00640 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f39a5b0 0x55631f36ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f7a0b400640 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f39a5b0 0x55631f36ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f7a0b400640 1 -- 172.21.15.69:0/2139071420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 msgr2=0x55631f36d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f7a0b400640 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 0x55631f36d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f7a0b400640 1 -- 172.21.15.69:0/2139071420 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55631f36b4f0 con 0x55631f3988d0 2024-04-24T08:10:42.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f7a0be00640 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 0x55631f36d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:42.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f7a03e00640 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f39a5b0 0x55631f36ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:42.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f7a0b400640 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f3c5450 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f79fc059e10 tx=0x7f79fc08a7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1068c73fdc3faafc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f79fb400640 1 -- 172.21.15.69:0/2139071420 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79fc056780 con 0x55631f3988d0 2024-04-24T08:10:42.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f79fb400640 1 -- 172.21.15.69:0/2139071420 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79fc05e120 con 0x55631f3988d0 2024-04-24T08:10:42.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.554+0000 7f79fb400640 1 -- 172.21.15.69:0/2139071420 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79fc056920 con 0x55631f3988d0 2024-04-24T08:10:42.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.555+0000 7f7a15597740 1 -- 172.21.15.69:0/2139071420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 msgr2=0x55631f3c5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.555+0000 7f7a15597740 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f3c5450 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f79fc059e10 tx=0x7f79fc08a7e0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.555+0000 7f7a15597740 1 -- 172.21.15.69:0/2139071420 shutdown_connections 2024-04-24T08:10:42.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.555+0000 7f7a15597740 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f3b57b0 0x55631f36d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.555+0000 7f7a15597740 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f39a5b0 0x55631f36ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.555+0000 7f7a15597740 1 --2- 172.21.15.69:0/2139071420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f3988d0 0x55631f3c5450 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.555+0000 7f7a15597740 1 -- 172.21.15.69:0/2139071420 >> 172.21.15.69:0/2139071420 conn(0x55631f398190 msgr2=0x55631f3b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:42.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.555+0000 7f7a15597740 1 -- 172.21.15.69:0/2139071420 shutdown_connections 2024-04-24T08:10:42.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.555+0000 7f7a15597740 1 -- 172.21.15.69:0/2139071420 wait complete. 2024-04-24T08:10:42.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.556+0000 7f7a15597740 1 -- start start 2024-04-24T08:10:42.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.556+0000 7f7a15597740 1 Processor -- start 2024-04-24T08:10:42.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a15597740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f42e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a15597740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3b57b0 0x55631f383ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a15597740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f416a00 0x55631f3b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a15597740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55631f386110 con 0x55631f39a5b0 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a15597740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55631f386aa0 con 0x55631f416a00 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a15597740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55631f386760 con 0x55631f3b57b0 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a15597740 10 client.0 _ll_register_callbacks cb 0x7f7a04007280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a0b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f42e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a0b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f42e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57932/0 (socket says 172.21.15.69:57932) 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a0b400640 1 -- 172.21.15.69:0/516408142 learned_addr learned my addr 172.21.15.69:0/516408142 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a0b400640 1 -- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3b57b0 msgr2=0x55631f383ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a03e00640 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3b57b0 0x55631f383ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.557+0000 7f7a0b400640 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3b57b0 0x55631f383ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f7a0b400640 1 -- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f416a00 msgr2=0x55631f3b46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f7a0b400640 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f416a00 0x55631f3b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f7a0b400640 1 -- 172.21.15.69:0/516408142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55631f384000 con 0x55631f39a5b0 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f7a0b400640 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f42e640 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f79fc059e10 tx=0x7f79fc08c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a363ef1b5e7a99 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f79f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f79f8c00640 1 -- 172.21.15.69:0/516408142 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79fc062cf0 con 0x55631f39a5b0 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f79f8c00640 1 -- 172.21.15.69:0/516408142 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f79fc062860 con 0x55631f39a5b0 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f7a15597740 1 -- 172.21.15.69:0/516408142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f79fc05e120 con 0x55631f39a5b0 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f79f8c00640 1 -- 172.21.15.69:0/516408142 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f79fc08c2b0 con 0x55631f39a5b0 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f7a15597740 10 client.5934 fetch_fsmap learned FSMap version 25 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f7a15597740 1 -- 172.21.15.69:0/516408142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55631f3b3f10 con 0x55631f39a5b0 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.558+0000 7f79f8c00640 1 -- 172.21.15.69:0/516408142 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79fc05e120 con 0x55631f39a5b0 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.559+0000 7f79f8c00640 1 -- 172.21.15.69:0/516408142 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f79fc01cbf0 con 0x55631f39a5b0 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.559+0000 7f7a15597740 10 client.5934 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.559+0000 7f7a15597740 10 client.5934 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-24T08:10:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.559+0000 7f7a15597740 20 client.5934 populate_metadata read hostname 'smithi069' 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.559+0000 7f7a15597740 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55631f3686e0 0x55631f3b1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.559+0000 7f7a15597740 4 client.5934 mds_command: new command op to 5793 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.559+0000 7f7a15597740 1 -- 172.21.15.69:0/516408142 --> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] -- command(tid 0: {"prefix": "status"}) -- 0x55631f395720 con 0x55631f3686e0 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.559+0000 7f7a03e00640 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55631f3686e0 0x55631f3b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.560+0000 7f79f8c00640 10 client.5934 ms_handle_connect on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.560+0000 7f7a03e00640 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55631f3686e0 0x55631f3b1350 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96b6c7ab1905639 server_cookie=c3834086adf8d36a in_seq=0 out_seq=0 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f79f8c00640 1 -- 172.21.15.69:0/516408142 <== mds.0 v2:172.21.15.121:6835/3424934990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55631f395720 con 0x55631f3686e0 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f79f8c00640 10 client.5934 handle_command_reply: tid=0 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 client.5934 shutdown 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 -- 172.21.15.69:0/516408142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 msgr2=0x55631f42e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f42e640 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f79fc059e10 tx=0x7f79fc08c040 comp rx=0 tx=0).stop 2024-04-24T08:10:42.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 -- 172.21.15.69:0/516408142 shutdown_connections 2024-04-24T08:10:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55631f3686e0 0x55631f3b1350 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55631f416a00 0x55631f3b46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55631f3b57b0 0x55631f383ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 --2- 172.21.15.69:0/516408142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55631f39a5b0 0x55631f42e640 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f79f8c00640 0 client.5934 ms_handle_reset on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 -- 172.21.15.69:0/516408142 >> 172.21.15.69:0/516408142 conn(0x55631f398190 msgr2=0x55631f3637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 -- 172.21.15.69:0/516408142 shutdown_connections 2024-04-24T08:10:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 1 -- 172.21.15.69:0/516408142 wait complete. 2024-04-24T08:10:42.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.561+0000 7f7a15597740 20 client.5934 trim_cache size 0 max 16384 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 5793, 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 6.6266787359999997, 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 25, 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 44, 2024-04-24T08:10:42.568 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 44, 2024-04-24T08:10:42.569 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 12.079754922999999 2024-04-24T08:10:42.569 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:10:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 -- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79e806c8f0 msgr2=0x7f79e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79e806c8f0 0x7f79e806edb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f79f4032000 tx=0x7f79f402e000 comp rx=0 tx=0).stop 2024-04-24T08:10:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 -- 172.21.15.69:0/1398708569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 msgr2=0x7f7a0c15df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f7a0c15df20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f79fc011540 tx=0x7f79fc011a00 comp rx=0 tx=0).stop 2024-04-24T08:10:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 -- 172.21.15.69:0/1398708569 shutdown_connections 2024-04-24T08:10:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79e806c8f0 0x7f79e806edb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a0c0b3ea0 0x7f7a0c0b1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a0c0c6f90 0x7f7a0c0b1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 --2- 172.21.15.69:0/1398708569 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f0004800 0x7f7a0c15df20 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 -- 172.21.15.69:0/1398708569 >> 172.21.15.69:0/1398708569 conn(0x7f7a0c067d90 msgr2=0x7f7a0c15a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:42.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 -- 172.21.15.69:0/1398708569 shutdown_connections 2024-04-24T08:10:42.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.562+0000 7f7a12600640 1 -- 172.21.15.69:0/1398708569 wait complete. 2024-04-24T08:10:42.581 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:10:42.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.778+0000 7ff62b800640 1 Processor -- start 2024-04-24T08:10:42.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.778+0000 7ff62b800640 1 -- start start 2024-04-24T08:10:42.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.779+0000 7ff62b800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff624145a90 0x7ff62406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.779+0000 7ff62b800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff624145fe0 0x7ff624160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.779+0000 7ff62b800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6240cf3d0 con 0x7ff6240d0300 2024-04-24T08:10:42.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.779+0000 7ff62b800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff6240cf6b0 con 0x7ff624145a90 2024-04-24T08:10:42.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.779+0000 7ff62b800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff6240cf990 con 0x7ff624145fe0 2024-04-24T08:10:42.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.780+0000 7ff61be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff624145a90 0x7ff62406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.780+0000 7ff629800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff624145fe0 0x7ff624160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.780+0000 7ff61be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff624145a90 0x7ff62406e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:58404/0 (socket says 172.21.15.69:58404) 2024-04-24T08:10:42.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.780+0000 7ff61be00640 1 -- 172.21.15.69:0/3038493603 learned_addr learned my addr 172.21.15.69:0/3038493603 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:42.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.781+0000 7ff61be00640 1 -- 172.21.15.69:0/3038493603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff624145fe0 msgr2=0x7ff624160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:42.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.781+0000 7ff61be00640 1 --2- 172.21.15.69:0/3038493603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff624145fe0 0x7ff624160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.781+0000 7ff61be00640 1 -- 172.21.15.69:0/3038493603 >> v1:172.21.15.121:6789/0 conn(0x7ff6240d0300 legacy=0x7ff6240d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.781+0000 7ff61be00640 1 -- 172.21.15.69:0/3038493603 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6240cfc70 con 0x7ff624145a90 2024-04-24T08:10:42.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.782+0000 7ff61be00640 1 --2- 172.21.15.69:0/3038493603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff624145a90 0x7ff62406e240 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff610009870 tx=0x7ff61002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cdc0087ebdf2529 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.783+0000 7ff623e00640 1 -- 172.21.15.69:0/3038493603 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610037070 con 0x7ff624145a90 2024-04-24T08:10:42.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.783+0000 7ff623e00640 1 -- 172.21.15.69:0/3038493603 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff610032440 con 0x7ff624145a90 2024-04-24T08:10:42.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.783+0000 7ff623e00640 1 -- 172.21.15.69:0/3038493603 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610032810 con 0x7ff624145a90 2024-04-24T08:10:42.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.784+0000 7ff62b800640 1 -- 172.21.15.69:0/3038493603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff624145a90 msgr2=0x7ff62406e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.784+0000 7ff62b800640 1 --2- 172.21.15.69:0/3038493603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff624145a90 0x7ff62406e240 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff610009870 tx=0x7ff61002a270 comp rx=0 tx=0).stop 2024-04-24T08:10:42.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.784+0000 7ff62b800640 1 -- 172.21.15.69:0/3038493603 shutdown_connections 2024-04-24T08:10:42.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.784+0000 7ff62b800640 1 --2- 172.21.15.69:0/3038493603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff624145fe0 0x7ff624160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.784+0000 7ff62b800640 1 --2- 172.21.15.69:0/3038493603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff624145a90 0x7ff62406e240 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.784+0000 7ff62b800640 1 -- 172.21.15.69:0/3038493603 >> 172.21.15.69:0/3038493603 conn(0x7ff624067d90 msgr2=0x7ff62406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:42.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.784+0000 7ff62b800640 1 -- 172.21.15.69:0/3038493603 shutdown_connections 2024-04-24T08:10:42.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.784+0000 7ff62b800640 1 -- 172.21.15.69:0/3038493603 wait complete. 2024-04-24T08:10:42.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.786+0000 7ff62b800640 1 Processor -- start 2024-04-24T08:10:42.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.786+0000 7ff62b800640 1 -- start start 2024-04-24T08:10:42.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff62b800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6240d0300 0x7ff62413c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff62b800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff624145a90 0x7ff62413ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff62b800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff624145fe0 0x7ff6241514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff62b800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6241458f0 con 0x7ff624145a90 2024-04-24T08:10:42.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff62b800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff624145770 con 0x7ff6240d0300 2024-04-24T08:10:42.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff62b800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff6240cdc30 con 0x7ff624145fe0 2024-04-24T08:10:42.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff61be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff624145a90 0x7ff62413ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff61be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff624145a90 0x7ff62413ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57936/0 (socket says 172.21.15.69:57936) 2024-04-24T08:10:42.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff61be00640 1 -- 172.21.15.69:0/2848227232 learned_addr learned my addr 172.21.15.69:0/2848227232 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:42.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff628e00640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6240d0300 0x7ff62413c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.787+0000 7ff629800640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff624145fe0 0x7ff6241514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff61be00640 1 -- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff624145fe0 msgr2=0x7ff6241514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff61be00640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff624145fe0 0x7ff6241514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff61be00640 1 -- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6240d0300 msgr2=0x7ff62413c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:42.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff61be00640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6240d0300 0x7ff62413c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:42.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff61be00640 1 -- 172.21.15.69:0/2848227232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6240b42b0 con 0x7ff624145a90 2024-04-24T08:10:42.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff628e00640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6240d0300 0x7ff62413c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:42.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff61be00640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff624145a90 0x7ff62413ce30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff610000c00 tx=0x7ff6100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff621600640 1 -- 172.21.15.69:0/2848227232 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610037040 con 0x7ff624145a90 2024-04-24T08:10:42.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff621600640 1 -- 172.21.15.69:0/2848227232 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff610042070 con 0x7ff624145a90 2024-04-24T08:10:42.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff621600640 1 -- 172.21.15.69:0/2848227232 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff610030940 con 0x7ff624145a90 2024-04-24T08:10:42.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff62b800640 1 -- 172.21.15.69:0/2848227232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6240b4590 con 0x7ff624145a90 2024-04-24T08:10:42.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.788+0000 7ff62b800640 1 -- 172.21.15.69:0/2848227232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6240b4b50 con 0x7ff624145a90 2024-04-24T08:10:42.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.790+0000 7ff62b800640 1 -- 172.21.15.69:0/2848227232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff624068a80 con 0x7ff624145a90 2024-04-24T08:10:42.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.791+0000 7ff621600640 1 -- 172.21.15.69:0/2848227232 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff610030ae0 con 0x7ff624145a90 2024-04-24T08:10:42.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.792+0000 7ff621600640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5f806c8f0 0x7ff5f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:42.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.792+0000 7ff621600640 1 -- 172.21.15.69:0/2848227232 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7ff6100ac370 con 0x7ff624145a90 2024-04-24T08:10:42.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.792+0000 7ff628e00640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5f806c8f0 0x7ff5f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:42.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.793+0000 7ff628e00640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5f806c8f0 0x7ff5f806edb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff608001680 tx=0x7ff608009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:42.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:42.796+0000 7ff621600640 1 -- 172.21.15.69:0/2848227232 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff61007a7a0 con 0x7ff624145a90 2024-04-24T08:10:43.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.039+0000 7ff62b800640 1 -- 172.21.15.69:0/2848227232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff62406be10 con 0x7ff624145a90 2024-04-24T08:10:43.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.040+0000 7ff621600640 1 -- 172.21.15.69:0/2848227232 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7ff61007a0a0 con 0x7ff624145a90 2024-04-24T08:10:43.041 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:10:43.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 -- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5f806c8f0 msgr2=0x7ff5f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:43.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5f806c8f0 0x7ff5f806edb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff608001680 tx=0x7ff608009040 comp rx=0 tx=0).stop 2024-04-24T08:10:43.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 -- 172.21.15.69:0/2848227232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff624145a90 msgr2=0x7ff62413ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:43.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff624145a90 0x7ff62413ce30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff610000c00 tx=0x7ff6100046c0 comp rx=0 tx=0).stop 2024-04-24T08:10:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 -- 172.21.15.69:0/2848227232 shutdown_connections 2024-04-24T08:10:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5f806c8f0 0x7ff5f806edb0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff624145a90 0x7ff62413ce30 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff624145fe0 0x7ff6241514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 --2- 172.21.15.69:0/2848227232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6240d0300 0x7ff62413c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 -- 172.21.15.69:0/2848227232 >> 172.21.15.69:0/2848227232 conn(0x7ff624067d90 msgr2=0x7ff62406f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 -- 172.21.15.69:0/2848227232 shutdown_connections 2024-04-24T08:10:43.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43.043+0000 7ff62b800640 1 -- 172.21.15.69:0/2848227232 wait complete. 2024-04-24T08:10:43.063 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:10:43.063 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:10:43.063 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:10:43.063 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:10:43.063 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:43.063 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:10:43.063 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:10:43.063 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:10:43.063 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:10:43.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:43.064 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:10:43.112 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42537sec preferred_lft 42537sec 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:10:43.113 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:10:43.114 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:10:43.114 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:10:43.114 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:10:43.114 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:10:43.114 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:10:43.114 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:10:43.114 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:10:43.114 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:10:43.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:43.115 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:10:43.188 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:10:43.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:43.192 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:10:43.242 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:10:43.242 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:10:43.242 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:10:43.242 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:10:43.242 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:10:43.243 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:10:43.243 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:10:43.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:10:43.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:10:43.243 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:10:43.243 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:10:43.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:10:43.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:43.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:43.450 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:10:43.498 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:10:43.498 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:10:43.499 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:10:43.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:43.500 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:10:43.581 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:10:43.581 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:10:43.581 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:43.633 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:10:43.633 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:10:43.633 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:43.684 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:10:43.764 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:10:43.820 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:10:43.820 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:10:43.820 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:10:43.820 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:10:43.820 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:10:43.820 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:10:43.821 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:10:43.822 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:10:43.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:43.823 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:10:43.875 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:10:43.875 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:10:43.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:10:43.945 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:10:43.975 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:10:43.990 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:10:43.990 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:10:43.990 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:10:43.990 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:10:43.991 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:10:43.992 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:10:43.993 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:10:43.993 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:10:43.993 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:10:43.993 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:10:43.993 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:10:43.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:43.994 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:10:43.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:10:44.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:10:44.045+0000 7f3153a46480 -1 init, newargv = 0x55a58d79e3e0 newargc=15 2024-04-24T08:10:44.046 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[50855]: starting ceph client 2024-04-24T08:10:44.056 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[50855]: starting fuse 2024-04-24T08:10:45.136 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:10:45.228 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:10:45.280 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:10:45.280 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:10:45.281 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:10:45.282 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:10:45.283 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:10:45.283 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:10:45.283 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:10:45.283 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:10:45.283 INFO:teuthology.orchestra.run.smithi069.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:45.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:45.284 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:10:45.326 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:10:45.326 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:10:45.327 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:10:45.328 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:10:45.536 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.50855.asok 2024-04-24T08:10:45.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:45.553 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.50855.asok 2024-04-24T08:10:45.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:45.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.50855.asok status 2024-04-24T08:10:45.779 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "50855", 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 5940, 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:10:45.780 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 5940 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2959973163 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2959973163 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:45.781 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.5940 192.168.0.1:0/2959973163", 2024-04-24T08:10:45.782 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2959973163", 2024-04-24T08:10:45.782 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:10:45.782 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 25, 2024-04-24T08:10:45.782 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 44, 2024-04-24T08:10:45.782 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:10:45.782 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:10:45.782 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:10:45.782 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:10:45.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:45.794 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:10:45.794 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:45.849 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:10:45.849 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:45.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:45.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:45.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:45.932 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_a) 2024-04-24T08:10:46.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:46.003 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parent_a/alpha) 2024-04-24T08:10:46.065 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:10:46.065 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:10:46.065 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:10:46.065 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:10:46.065 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/parent_a/alpha").st_ino) 2024-04-24T08:10:46.065 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:10:46.207 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627777 2024-04-24T08:10:46.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:46.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:46.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.465+0000 7f5840c00640 1 Processor -- start 2024-04-24T08:10:46.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.465+0000 7f5840c00640 1 -- start start 2024-04-24T08:10:46.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.467+0000 7f5840c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c156c60 0x7f583c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:46.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.467+0000 7f5840c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f583c15d660 con 0x7f583c0b30f0 2024-04-24T08:10:46.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.467+0000 7f5840c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f583c06dc30 con 0x7f583c0ad7e0 2024-04-24T08:10:46.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.467+0000 7f5840c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f583c06df10 con 0x7f583c156c60 2024-04-24T08:10:46.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.467+0000 7f583aa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f583c0b30f0 0x7f583c0ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49288/0 (socket says 172.21.15.69:49288) 2024-04-24T08:10:46.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.467+0000 7f583aa00640 1 -- 172.21.15.69:0/355443455 learned_addr learned my addr 172.21.15.69:0/355443455 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.467+0000 7f583b400640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c156c60 0x7f583c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.467+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704390946 0 0) 0x7f583c15d660 con 0x7f583c0b30f0 2024-04-24T08:10:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5820003520 con 0x7f583c0b30f0 2024-04-24T08:10:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 204104037 0 0) 0x7f5820003520 con 0x7f583c0b30f0 2024-04-24T08:10:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c156c60 msgr2=0x7f583c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c156c60 0x7f583c15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 >> v1:172.21.15.121:6789/0 conn(0x7f583c0ad7e0 legacy=0x7f583c06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:46.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583c06e1f0 con 0x7f583c0b30f0 2024-04-24T08:10:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f5828001670 con 0x7f583c0b30f0 2024-04-24T08:10:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 >> v1:172.21.15.69:6789/0 conn(0x7f583c0b30f0 legacy=0x7f583c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58200039e0 0x7f5820003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5820004800 0x7f5820004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c156c60 0x7f58200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f583c06df10 con 0x7f5820004800 2024-04-24T08:10:46.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f582000baa0 con 0x7f58200039e0 2024-04-24T08:10:46.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f582000bca0 con 0x7f583c156c60 2024-04-24T08:10:46.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.468+0000 7f583aa00640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5820004800 0x7f5820004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:46.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.469+0000 7f583b400640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58200039e0 0x7f5820003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:46.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.469+0000 7f583aa00640 1 -- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c156c60 msgr2=0x7f58200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:46.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.469+0000 7f583aa00640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c156c60 0x7f58200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:46.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.469+0000 7f583aa00640 1 -- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58200039e0 msgr2=0x7f5820003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:46.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.469+0000 7f583aa00640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58200039e0 0x7f5820003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:46.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.469+0000 7f583aa00640 1 -- 172.21.15.69:0/355443455 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5820003520 con 0x7f5820004800 2024-04-24T08:10:46.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.469+0000 7f583b400640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58200039e0 0x7f5820003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.470+0000 7f583aa00640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5820004800 0x7f5820004c80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f583c0b34d0 tx=0x7f5820005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238c987838aa2273 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.470+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828007c60 con 0x7f5820004800 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.470+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5828005af0 con 0x7f5820004800 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.470+0000 7f583a000640 1 -- 172.21.15.69:0/355443455 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828007070 con 0x7f5820004800 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.471+0000 7f5840c00640 1 -- 172.21.15.69:0/355443455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5820004800 msgr2=0x7f5820004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.471+0000 7f5840c00640 1 --2- 172.21.15.69:0/355443455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5820004800 0x7f5820004c80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f583c0b34d0 tx=0x7f5820005830 comp rx=0 tx=0).stop 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.471+0000 7f583be00640 1 -- 172.21.15.69:0/355443455 reap_dead start 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.471+0000 7f5840c00640 1 -- 172.21.15.69:0/355443455 shutdown_connections 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.471+0000 7f5840c00640 1 -- 172.21.15.69:0/355443455 >> 172.21.15.69:0/355443455 conn(0x7f583c0d0160 msgr2=0x7f583c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.471+0000 7f5840c00640 1 -- 172.21.15.69:0/355443455 shutdown_connections 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.471+0000 7f5840c00640 1 -- 172.21.15.69:0/355443455 wait complete. 2024-04-24T08:10:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.473+0000 7f5840c00640 1 Processor -- start 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.473+0000 7f5840c00640 1 -- start start 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.474+0000 7f5840c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5820004800 0x7f583c145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.474+0000 7f5840c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c150ac0 0x7f583c159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.474+0000 7f5840c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f583c155630 0x7f583c155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.474+0000 7f5840c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f583c06ee20 con 0x7f583c155630 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.474+0000 7f5840c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f583c15d940 con 0x7f5820004800 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.474+0000 7f5840c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f583c06efa0 con 0x7f583c150ac0 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.474+0000 7f583be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f583c155630 0x7f583c155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.474+0000 7f583be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f583c155630 0x7f583c155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57952/0 (socket says 172.21.15.69:57952) 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.474+0000 7f583be00640 1 -- 172.21.15.69:0/1726059012 learned_addr learned my addr 172.21.15.69:0/1726059012 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f583b400640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5820004800 0x7f583c145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f583be00640 1 -- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c150ac0 msgr2=0x7f583c159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f583aa00640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c150ac0 0x7f583c159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f583be00640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c150ac0 0x7f583c159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f583be00640 1 -- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5820004800 msgr2=0x7f583c145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f583be00640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5820004800 0x7f583c145610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f583be00640 1 -- 172.21.15.69:0/1726059012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f583c159bd0 con 0x7f583c155630 2024-04-24T08:10:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f583be00640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f583c155630 0x7f583c155ab0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f583c0b5300 tx=0x7f583400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f5833400640 1 -- 172.21.15.69:0/1726059012 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5834017070 con 0x7f583c155630 2024-04-24T08:10:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f5833400640 1 -- 172.21.15.69:0/1726059012 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5834005d40 con 0x7f583c155630 2024-04-24T08:10:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.475+0000 7f5840c00640 1 -- 172.21.15.69:0/1726059012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f583c151150 con 0x7f583c155630 2024-04-24T08:10:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.476+0000 7f5833400640 1 -- 172.21.15.69:0/1726059012 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583400a850 con 0x7f583c155630 2024-04-24T08:10:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.476+0000 7f5840c00640 1 -- 172.21.15.69:0/1726059012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f583c151710 con 0x7f583c155630 2024-04-24T08:10:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.477+0000 7f5840c00640 1 -- 172.21.15.69:0/1726059012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f583c075d50 con 0x7f583c155630 2024-04-24T08:10:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.478+0000 7f5833400640 1 -- 172.21.15.69:0/1726059012 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f583401a070 con 0x7f583c155630 2024-04-24T08:10:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.479+0000 7f5833400640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f581c06c920 0x7f581c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:46.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.479+0000 7f5833400640 1 -- 172.21.15.69:0/1726059012 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f583408ae50 con 0x7f583c155630 2024-04-24T08:10:46.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.480+0000 7f583b400640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f581c06c920 0x7f581c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:46.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.481+0000 7f583b400640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f581c06c920 0x7f581c06ede0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f582c009ab0 tx=0x7f582c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.484+0000 7f5833400640 1 -- 172.21.15.69:0/1726059012 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58340591e0 con 0x7f583c155630 2024-04-24T08:10:46.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.727+0000 7f5840c00640 1 -- 172.21.15.69:0/1726059012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f583c150ea0 con 0x7f583c155630 2024-04-24T08:10:46.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.728+0000 7f5833400640 1 -- 172.21.15.69:0/1726059012 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5184 (secure 0 0 0) 0x7f5834058ae0 con 0x7f583c155630 2024-04-24T08:10:46.730 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:46.730 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/376722301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":376722301},{"type":"v1","addr":"172.21.15.69:6835","nonce":376722301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4990,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1327624292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1327624292},{"type":"v1","addr":"172.21.15.121:6836","nonce":1327624292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/3244002416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3244002416},{"type":"v1","addr":"172.21.15.69:6834","nonce":3244002416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:10:35.400788+0000","modified":"2024-04-24T08:10:38.457641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5793},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5793":{"gid":5793,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/3424934990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3424934990},{"type":"v1","addr":"172.21.15.121:6837","nonce":3424934990}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5793,"qdb_cluster":[5793]},"id":3}]} 2024-04-24T08:10:46.730 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-04-24T08:10:46.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 -- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f581c06c920 msgr2=0x7f581c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:46.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f581c06c920 0x7f581c06ede0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f582c009ab0 tx=0x7f582c008040 comp rx=0 tx=0).stop 2024-04-24T08:10:46.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 -- 172.21.15.69:0/1726059012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f583c155630 msgr2=0x7f583c155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:46.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f583c155630 0x7f583c155ab0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f583c0b5300 tx=0x7f583400cb90 comp rx=0 tx=0).stop 2024-04-24T08:10:46.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 -- 172.21.15.69:0/1726059012 shutdown_connections 2024-04-24T08:10:46.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f581c06c920 0x7f581c06ede0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:46.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f583c155630 0x7f583c155ab0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:46.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f583c150ac0 0x7f583c159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:46.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 --2- 172.21.15.69:0/1726059012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5820004800 0x7f583c145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:46.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 -- 172.21.15.69:0/1726059012 >> 172.21.15.69:0/1726059012 conn(0x7f583c0d0160 msgr2=0x7f583c15c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:46.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 -- 172.21.15.69:0/1726059012 shutdown_connections 2024-04-24T08:10:46.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:46.731+0000 7f5840c00640 1 -- 172.21.15.69:0/1726059012 wait complete. 2024-04-24T08:10:46.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-04-24T08:10:46.998 INFO:teuthology.orchestra.run.smithi121.stdout:{"message":"","return_code":0} 2024-04-24T08:10:46.998 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-24T08:10:46.999 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-04-24T08:10:47.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.111+0000 7ff9e951cf40 1 Processor -- start 2024-04-24T08:10:47.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.111+0000 7ff9e951cf40 1 -- start start 2024-04-24T08:10:47.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.112+0000 7ff9e951cf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dffa3c5ad0 0x55dffa499100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.112+0000 7ff9e951cf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dffa3f2380 0x55dffa497890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.112+0000 7ff9e951cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa4974b0 0x55dffa497dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.112+0000 7ff9e951cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dffa3c67b0 con 0x55dffa4974b0 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.112+0000 7ff9e951cf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55dffa369cc0 con 0x55dffa3c5ad0 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.112+0000 7ff9e951cf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55dffa40b150 con 0x55dffa3f2380 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e6600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa4974b0 0x55dffa497dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e6600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa4974b0 0x55dffa497dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57966/0 (socket says 172.21.15.69:57966) 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e6600640 1 -- 172.21.15.69:0/2128252276 learned_addr learned my addr 172.21.15.69:0/2128252276 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e5200640 1 --2- 172.21.15.69:0/2128252276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dffa3f2380 0x55dffa497890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e5c00640 1 --2- 172.21.15.69:0/2128252276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dffa3c5ad0 0x55dffa499100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e6600640 1 -- 172.21.15.69:0/2128252276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dffa3f2380 msgr2=0x55dffa497890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e6600640 1 --2- 172.21.15.69:0/2128252276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dffa3f2380 0x55dffa497890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e6600640 1 -- 172.21.15.69:0/2128252276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dffa3c5ad0 msgr2=0x55dffa499100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e6600640 1 --2- 172.21.15.69:0/2128252276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dffa3c5ad0 0x55dffa499100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e6600640 1 -- 172.21.15.69:0/2128252276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dffa48c970 con 0x55dffa4974b0 2024-04-24T08:10:47.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.113+0000 7ff9e6600640 1 --2- 172.21.15.69:0/2128252276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa4974b0 0x55dffa497dd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0001660 tx=0x7ff9e0001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd045bf7de575a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9df400640 1 -- 172.21.15.69:0/2128252276 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0003650 con 0x55dffa4974b0 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9df400640 1 -- 172.21.15.69:0/2128252276 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9e0003830 con 0x55dffa4974b0 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9df400640 1 -- 172.21.15.69:0/2128252276 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9e0003ca0 con 0x55dffa4974b0 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9e951cf40 1 -- 172.21.15.69:0/2128252276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa4974b0 msgr2=0x55dffa497dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9e951cf40 1 --2- 172.21.15.69:0/2128252276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa4974b0 0x55dffa497dd0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0001660 tx=0x7ff9e0001b20 comp rx=0 tx=0).stop 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9e951cf40 1 -- 172.21.15.69:0/2128252276 shutdown_connections 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9e951cf40 1 --2- 172.21.15.69:0/2128252276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa4974b0 0x55dffa497dd0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9e951cf40 1 --2- 172.21.15.69:0/2128252276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dffa3f2380 0x55dffa497890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9e951cf40 1 --2- 172.21.15.69:0/2128252276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dffa3c5ad0 0x55dffa499100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9e951cf40 1 -- 172.21.15.69:0/2128252276 >> 172.21.15.69:0/2128252276 conn(0x55dffa3efe40 msgr2=0x55dffa408e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9e951cf40 1 -- 172.21.15.69:0/2128252276 shutdown_connections 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.114+0000 7ff9e951cf40 1 -- 172.21.15.69:0/2128252276 wait complete. 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.115+0000 7ff9e951cf40 1 Processor -- start 2024-04-24T08:10:47.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.115+0000 7ff9e951cf40 1 -- start start 2024-04-24T08:10:47.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.116+0000 7ff9e951cf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa3c5ad0 0x55dffa48e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.116+0000 7ff9e951cf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dffa3f2380 0x55dffa409210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.116+0000 7ff9e951cf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dffa4974b0 0x55dffa47f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.116+0000 7ff9e951cf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dffa498540 con 0x55dffa3c5ad0 2024-04-24T08:10:47.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.116+0000 7ff9e951cf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55dffa499640 con 0x55dffa4974b0 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.116+0000 7ff9e951cf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55dffa406e50 con 0x55dffa3f2380 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.116+0000 7ff9e5c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa3c5ad0 0x55dffa48e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.116+0000 7ff9e5c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa3c5ad0 0x55dffa48e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57968/0 (socket says 172.21.15.69:57968) 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.116+0000 7ff9e5c00640 1 -- 172.21.15.69:0/3119170224 learned_addr learned my addr 172.21.15.69:0/3119170224 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.117+0000 7ff9e5200640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dffa3f2380 0x55dffa409210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.117+0000 7ff9e6600640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dffa4974b0 0x55dffa47f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.117+0000 7ff9e5c00640 1 -- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dffa3f2380 msgr2=0x55dffa409210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.117+0000 7ff9e5c00640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dffa3f2380 0x55dffa409210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.117+0000 7ff9e5c00640 1 -- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dffa4974b0 msgr2=0x55dffa47f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.117+0000 7ff9e5c00640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dffa4974b0 0x55dffa47f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.117+0000 7ff9e5c00640 1 -- 172.21.15.69:0/3119170224 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dffa407130 con 0x55dffa3c5ad0 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.117+0000 7ff9e5200640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dffa3f2380 0x55dffa409210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.117+0000 7ff9e6600640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dffa4974b0 0x55dffa47f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:47.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.117+0000 7ff9e5c00640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa3c5ad0 0x55dffa48e410 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8045df0 tx=0x7ff9d8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.118+0000 7ff9de000640 1 -- 172.21.15.69:0/3119170224 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8064020 con 0x55dffa3c5ad0 2024-04-24T08:10:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.118+0000 7ff9de000640 1 -- 172.21.15.69:0/3119170224 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9d805a040 con 0x55dffa3c5ad0 2024-04-24T08:10:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.118+0000 7ff9e951cf40 1 -- 172.21.15.69:0/3119170224 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55dffa407410 con 0x55dffa3c5ad0 2024-04-24T08:10:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.118+0000 7ff9de000640 1 -- 172.21.15.69:0/3119170224 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8059070 con 0x55dffa3c5ad0 2024-04-24T08:10:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.118+0000 7ff9e951cf40 1 -- 172.21.15.69:0/3119170224 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dffa4066e0 con 0x55dffa3c5ad0 2024-04-24T08:10:47.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.120+0000 7ff9de000640 1 -- 172.21.15.69:0/3119170224 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff9d8067020 con 0x55dffa3c5ad0 2024-04-24T08:10:47.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.121+0000 7ff9de000640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff9c406b2d0 0x7ff9c406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.121+0000 7ff9de000640 1 -- 172.21.15.69:0/3119170224 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7ff9d80ce450 con 0x55dffa3c5ad0 2024-04-24T08:10:47.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.121+0000 7ff9e951cf40 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55dffa409750 0x55dffa407c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.121+0000 7ff9e951cf40 1 -- 172.21.15.69:0/3119170224 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:1 7.7 7:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) -- 0x55dffa47f800 con 0x55dffa409750 2024-04-24T08:10:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.121+0000 7ff9e5200640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff9c406b2d0 0x7ff9c406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.122+0000 7ff9e6600640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55dffa409750 0x55dffa407c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.123+0000 7ff9e6600640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55dffa409750 0x55dffa407c10 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.123+0000 7ff9e5200640 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff9c406b2d0 0x7ff9c406d790 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0005fd0 tx=0x7ff9d000c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.123+0000 7ff9e6600640 1 -- 172.21.15.69:0/3119170224 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv1 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x7ff9e000f030 con 0x55dffa409750 2024-04-24T08:10:47.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e951cf40 1 -- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55dffa409750 msgr2=0x55dffa407c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e951cf40 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55dffa409750 0x55dffa407c10 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e951cf40 1 -- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff9c406b2d0 msgr2=0x7ff9c406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e951cf40 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff9c406b2d0 0x7ff9c406d790 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0005fd0 tx=0x7ff9d000c040 comp rx=0 tx=0).stop 2024-04-24T08:10:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e951cf40 1 -- 172.21.15.69:0/3119170224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa3c5ad0 msgr2=0x55dffa48e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e951cf40 1 --2- 172.21.15.69:0/3119170224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dffa3c5ad0 0x55dffa48e410 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8045df0 tx=0x7ff9d8002fb0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e6600640 1 -- 172.21.15.69:0/3119170224 reap_dead start 2024-04-24T08:10:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e951cf40 1 -- 172.21.15.69:0/3119170224 shutdown_connections 2024-04-24T08:10:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e951cf40 1 -- 172.21.15.69:0/3119170224 >> 172.21.15.69:0/3119170224 conn(0x55dffa3efe40 msgr2=0x55dffa497890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e951cf40 1 -- 172.21.15.69:0/3119170224 shutdown_connections 2024-04-24T08:10:47.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.124+0000 7ff9e951cf40 1 -- 172.21.15.69:0/3119170224 wait complete. 2024-04-24T08:10:47.130 INFO:teuthology.orchestra.run.smithi069.stdout:Yalphaparent_a 2024-04-24T08:10:47.130 DEBUG:teuthology.orchestra.run.smithi069:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-04-24T08:10:47.272 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:10:47.272 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511627777, 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: "ancestors": [ 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: "dirino": 1099511627776, 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: "dname": "alpha", 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: "version": 2 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: "dirino": 1, 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: "dname": "parent_a", 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: "version": 4 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-24T08:10:47.273 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 7, 2024-04-24T08:10:47.274 INFO:teuthology.orchestra.run.smithi069.stdout: "old_pools": [] 2024-04-24T08:10:47.274 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:10:47.274 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:47.282 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'oh i'"'"'m sorry did i overwrite your xattr?' 2024-04-24T08:10:47.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.394+0000 7f4e2cf1df40 1 Processor -- start 2024-04-24T08:10:47.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.394+0000 7f4e2cf1df40 1 -- start start 2024-04-24T08:10:47.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.394+0000 7f4e2cf1df40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e100022d0 0x56433d79ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.394+0000 7f4e2cf1df40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56433d7a8f00 0x56433d71ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2cf1df40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d71f140 0x56433d7007d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2cf1df40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56433d679cc0 con 0x56433d71f140 2024-04-24T08:10:47.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2cf1df40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56433d67dd00 con 0x7f4e100022d0 2024-04-24T08:10:47.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2cf1df40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56433d7aa8a0 con 0x56433d7a8f00 2024-04-24T08:10:47.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d71f140 0x56433d7007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d71f140 0x56433d7007d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57980/0 (socket says 172.21.15.69:57980) 2024-04-24T08:10:47.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2a000640 1 -- 172.21.15.69:0/1543140161 learned_addr learned my addr 172.21.15.69:0/1543140161 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2a000640 1 -- 172.21.15.69:0/1543140161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56433d7a8f00 msgr2=0x56433d71ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e29600640 1 --2- 172.21.15.69:0/1543140161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e100022d0 0x56433d79ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2a000640 1 --2- 172.21.15.69:0/1543140161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56433d7a8f00 0x56433d71ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2a000640 1 -- 172.21.15.69:0/1543140161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e100022d0 msgr2=0x56433d79ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2a000640 1 --2- 172.21.15.69:0/1543140161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e100022d0 0x56433d79ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.395+0000 7f4e2a000640 1 -- 172.21.15.69:0/1543140161 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56433d6ffcb0 con 0x56433d71f140 2024-04-24T08:10:47.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e2a000640 1 --2- 172.21.15.69:0/1543140161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d71f140 0x56433d7007d0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x56433d6bc590 tx=0x7f4e24005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807345949def69ba server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e23400640 1 -- 172.21.15.69:0/1543140161 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2401a070 con 0x56433d71f140 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e23400640 1 -- 172.21.15.69:0/1543140161 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e24009e50 con 0x56433d71f140 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e23400640 1 -- 172.21.15.69:0/1543140161 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24015880 con 0x56433d71f140 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/1543140161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d71f140 msgr2=0x56433d7007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e2cf1df40 1 --2- 172.21.15.69:0/1543140161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d71f140 0x56433d7007d0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x56433d6bc590 tx=0x7f4e24005a00 comp rx=0 tx=0).stop 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/1543140161 shutdown_connections 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e2cf1df40 1 --2- 172.21.15.69:0/1543140161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d71f140 0x56433d7007d0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e2cf1df40 1 --2- 172.21.15.69:0/1543140161 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56433d7a8f00 0x56433d71ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e2cf1df40 1 --2- 172.21.15.69:0/1543140161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e100022d0 0x56433d79ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.396+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/1543140161 >> 172.21.15.69:0/1543140161 conn(0x56433d701be0 msgr2=0x56433d6d7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.397+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/1543140161 shutdown_connections 2024-04-24T08:10:47.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.397+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/1543140161 wait complete. 2024-04-24T08:10:47.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.397+0000 7f4e2cf1df40 1 Processor -- start 2024-04-24T08:10:47.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.398+0000 7f4e2cf1df40 1 -- start start 2024-04-24T08:10:47.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.398+0000 7f4e2cf1df40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56433d71f140 0x56433d7a4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.398+0000 7f4e2cf1df40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d7a8f00 0x56433d841370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.398+0000 7f4e2cf1df40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e100022d0 0x56433d6d0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.398+0000 7f4e2cf1df40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56433d7a2da0 con 0x56433d7a8f00 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.398+0000 7f4e2cf1df40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56433d700440 con 0x7f4e100022d0 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.398+0000 7f4e2cf1df40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56433d6fff20 con 0x56433d71f140 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e28c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d7a8f00 0x56433d841370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e28c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d7a8f00 0x56433d841370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57988/0 (socket says 172.21.15.69:57988) 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e28c00640 1 -- 172.21.15.69:0/966809084 learned_addr learned my addr 172.21.15.69:0/966809084 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e29600640 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56433d71f140 0x56433d7a4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e2a000640 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e100022d0 0x56433d6d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e28c00640 1 -- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56433d71f140 msgr2=0x56433d7a4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e28c00640 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56433d71f140 0x56433d7a4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e28c00640 1 -- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e100022d0 msgr2=0x56433d6d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e28c00640 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e100022d0 0x56433d6d0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e28c00640 1 -- 172.21.15.69:0/966809084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56433d841ee0 con 0x56433d7a8f00 2024-04-24T08:10:47.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.399+0000 7f4e28c00640 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d7a8f00 0x56433d841370 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4e14001620 tx=0x7f4e1400b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.400+0000 7f4e22000640 1 -- 172.21.15.69:0/966809084 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e14013070 con 0x56433d7a8f00 2024-04-24T08:10:47.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.400+0000 7f4e22000640 1 -- 172.21.15.69:0/966809084 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e1400f860 con 0x56433d7a8f00 2024-04-24T08:10:47.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.400+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/966809084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56433d8421c0 con 0x56433d7a8f00 2024-04-24T08:10:47.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.400+0000 7f4e22000640 1 -- 172.21.15.69:0/966809084 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1400fc50 con 0x56433d7a8f00 2024-04-24T08:10:47.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.400+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/966809084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56433d842780 con 0x56433d7a8f00 2024-04-24T08:10:47.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.402+0000 7f4e22000640 1 -- 172.21.15.69:0/966809084 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4e14016070 con 0x56433d7a8f00 2024-04-24T08:10:47.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.403+0000 7f4e22000640 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e0806b2d0 0x7f4e0806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.403+0000 7f4e22000640 1 -- 172.21.15.69:0/966809084 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f4e1408abf0 con 0x56433d7a8f00 2024-04-24T08:10:47.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.403+0000 7f4e2cf1df40 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x56433d6d06c0 0x56433d842f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.403+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/966809084 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:1 7.7 7:e2e07930:::10000000001.00000000:head [setxattr parent (40) in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44) -- 0x56433d8418b0 con 0x56433d6d06c0 2024-04-24T08:10:47.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.403+0000 7f4e29600640 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e0806b2d0 0x7f4e0806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.404+0000 7f4e2a000640 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x56433d6d06c0 0x56433d842f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.405+0000 7f4e29600640 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e0806b2d0 0x7f4e0806d790 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c055000 tx=0x7f4e1c008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.405+0000 7f4e2a000640 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x56433d6d06c0 0x56433d842f80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.407+0000 7f4e2a000640 1 -- 172.21.15.69:0/966809084 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (40)] v44'2 uv2 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f4e2401d040 con 0x56433d6d06c0 2024-04-24T08:10:47.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x56433d6d06c0 msgr2=0x56433d842f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2cf1df40 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x56433d6d06c0 0x56433d842f80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e0806b2d0 msgr2=0x7f4e0806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2cf1df40 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e0806b2d0 0x7f4e0806d790 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c055000 tx=0x7f4e1c008030 comp rx=0 tx=0).stop 2024-04-24T08:10:47.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/966809084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d7a8f00 msgr2=0x56433d841370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2cf1df40 1 --2- 172.21.15.69:0/966809084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56433d7a8f00 0x56433d841370 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4e14001620 tx=0x7f4e1400b6c0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2a000640 1 -- 172.21.15.69:0/966809084 reap_dead start 2024-04-24T08:10:47.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/966809084 shutdown_connections 2024-04-24T08:10:47.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/966809084 >> 172.21.15.69:0/966809084 conn(0x56433d701be0 msgr2=0x56433d6d7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/966809084 shutdown_connections 2024-04-24T08:10:47.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.408+0000 7f4e2cf1df40 1 -- 172.21.15.69:0/966809084 wait complete. 2024-04-24T08:10:47.414 DEBUG:teuthology.orchestra.run.smithi069:> sudo daemon-helper kill ceph --cluster ceph -w 2024-04-24T08:10:47.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:47.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub start / repair,recursive 2024-04-24T08:10:47.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.594+0000 7f094ea00640 1 Processor -- start 2024-04-24T08:10:47.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.595+0000 7f094ea00640 1 -- start start 2024-04-24T08:10:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.596+0000 7f094ea00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 0x7f094806c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.596+0000 7f094ea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09480ce540 0x7f09480b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.596+0000 7f094ea00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09480b2c30 con 0x7f09480b27a0 2024-04-24T08:10:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.596+0000 7f094ea00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f09480ce0b0 con 0x7f09480ce540 2024-04-24T08:10:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.596+0000 7f094ea00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09480cd4c0 con 0x7f094806bf00 2024-04-24T08:10:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.596+0000 7f094cc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09480ce540 0x7f09480b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.596+0000 7f094cc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09480ce540 0x7f09480b2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57990/0 (socket says 172.21.15.69:57990) 2024-04-24T08:10:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.596+0000 7f094cc00640 1 -- 172.21.15.69:0/3349791023 learned_addr learned my addr 172.21.15.69:0/3349791023 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.597+0000 7f094d600640 1 --2- 172.21.15.69:0/3349791023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 0x7f094806c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.597+0000 7f094cc00640 1 -- 172.21.15.69:0/3349791023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 msgr2=0x7f094806c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.597+0000 7f094cc00640 1 --2- 172.21.15.69:0/3349791023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 0x7f094806c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.597+0000 7f094cc00640 1 -- 172.21.15.69:0/3349791023 >> v1:172.21.15.69:6789/0 conn(0x7f09480b27a0 legacy=0x7f09480d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.597+0000 7f094cc00640 1 -- 172.21.15.69:0/3349791023 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09480cd780 con 0x7f09480ce540 2024-04-24T08:10:47.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.598+0000 7f094cc00640 1 --2- 172.21.15.69:0/3349791023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480ce540 0x7f09480b2260 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f093c009870 tx=0x7f093c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63dae46360265c49 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.599+0000 7f0947e00640 1 -- 172.21.15.69:0/3349791023 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c037070 con 0x7f09480ce540 2024-04-24T08:10:47.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.599+0000 7f0947e00640 1 -- 172.21.15.69:0/3349791023 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093c032440 con 0x7f09480ce540 2024-04-24T08:10:47.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.599+0000 7f0947e00640 1 -- 172.21.15.69:0/3349791023 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c032810 con 0x7f09480ce540 2024-04-24T08:10:47.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.599+0000 7f094ea00640 1 -- 172.21.15.69:0/3349791023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480ce540 msgr2=0x7f09480b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.599+0000 7f094ea00640 1 --2- 172.21.15.69:0/3349791023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480ce540 0x7f09480b2260 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f093c009870 tx=0x7f093c02a1c0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.600+0000 7f094ea00640 1 -- 172.21.15.69:0/3349791023 shutdown_connections 2024-04-24T08:10:47.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.600+0000 7f094ea00640 1 --2- 172.21.15.69:0/3349791023 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480ce540 0x7f09480b2260 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.600+0000 7f094ea00640 1 --2- 172.21.15.69:0/3349791023 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 0x7f094806c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.600+0000 7f094ea00640 1 -- 172.21.15.69:0/3349791023 >> 172.21.15.69:0/3349791023 conn(0x7f09480b2e00 msgr2=0x7f09480b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.600+0000 7f094ea00640 1 -- 172.21.15.69:0/3349791023 shutdown_connections 2024-04-24T08:10:47.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.600+0000 7f094ea00640 1 -- 172.21.15.69:0/3349791023 wait complete. 2024-04-24T08:10:47.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.601+0000 7f094ea00640 1 Processor -- start 2024-04-24T08:10:47.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.602+0000 7f094ea00640 1 -- start start 2024-04-24T08:10:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.602+0000 7f094ea00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 0x7f09480b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.602+0000 7f094ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480b27a0 0x7f09480cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.602+0000 7f094ea00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480ce540 0x7f09481602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.602+0000 7f094ea00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09480ce9a0 con 0x7f09480b27a0 2024-04-24T08:10:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.602+0000 7f094ea00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f09480cdd40 con 0x7f09480ce540 2024-04-24T08:10:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.602+0000 7f094ea00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09480ceb20 con 0x7f094806bf00 2024-04-24T08:10:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480b27a0 0x7f09480cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480b27a0 0x7f09480cb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58004/0 (socket says 172.21.15.69:58004) 2024-04-24T08:10:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094cc00640 1 -- 172.21.15.69:0/3226429998 learned_addr learned my addr 172.21.15.69:0/3226429998 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094d600640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 0x7f09480b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094cc00640 1 -- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 msgr2=0x7f09480b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094cc00640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 0x7f09480b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094cc00640 1 -- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480ce540 msgr2=0x7f09481602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094e000640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480ce540 0x7f09481602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094cc00640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480ce540 0x7f09481602b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094cc00640 1 -- 172.21.15.69:0/3226429998 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09480b1ae0 con 0x7f09480b27a0 2024-04-24T08:10:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.603+0000 7f094d600640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 0x7f09480b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:47.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.604+0000 7f094cc00640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480b27a0 0x7f09480cb4a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f093c009870 tx=0x7f093c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.604+0000 7f0945600640 1 -- 172.21.15.69:0/3226429998 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c041070 con 0x7f09480b27a0 2024-04-24T08:10:47.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.604+0000 7f0945600640 1 -- 172.21.15.69:0/3226429998 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093c03c890 con 0x7f09480b27a0 2024-04-24T08:10:47.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.604+0000 7f094ea00640 1 -- 172.21.15.69:0/3226429998 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09480b0cf0 con 0x7f09480b27a0 2024-04-24T08:10:47.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.604+0000 7f0945600640 1 -- 172.21.15.69:0/3226429998 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c037040 con 0x7f09480b27a0 2024-04-24T08:10:47.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.604+0000 7f094ea00640 1 -- 172.21.15.69:0/3226429998 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09480b12b0 con 0x7f09480b27a0 2024-04-24T08:10:47.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.607+0000 7f0945600640 1 -- 172.21.15.69:0/3226429998 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f093c044070 con 0x7f09480b27a0 2024-04-24T08:10:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.608+0000 7f0952f91740 1 Processor -- start 2024-04-24T08:10:47.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.608+0000 7f0952f91740 1 -- start start 2024-04-24T08:10:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.609+0000 7f0945600640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f092c06c8f0 0x7f092c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.609+0000 7f0945600640 1 -- 172.21.15.69:0/3226429998 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f093c0ac580 con 0x7f09480b27a0 2024-04-24T08:10:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f094e000640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f092c06c8f0 0x7f092c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f0952f91740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad25fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f0952f91740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad56760 0x561f9ad55ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f0952f91740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56020 0x561f9ad2ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f0952f91740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561f9ac8fe70 con 0x561f9ad73640 2024-04-24T08:10:47.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f0952f91740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561f9ac36380 con 0x561f9ad56020 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f0952f91740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x561f9ac8be30 con 0x561f9ad56760 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f094d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad25fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f094d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad25fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58008/0 (socket says 172.21.15.69:58008) 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f094d600640 1 -- 172.21.15.69:0/3940253018 learned_addr learned my addr 172.21.15.69:0/3940253018 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.610+0000 7f094cc00640 1 --2- 172.21.15.69:0/3940253018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad56760 0x561f9ad55ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f094e000640 1 --2- 172.21.15.69:0/3940253018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56020 0x561f9ad2ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f094d600640 1 -- 172.21.15.69:0/3940253018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad56760 msgr2=0x561f9ad55ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f094d600640 1 --2- 172.21.15.69:0/3940253018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad56760 0x561f9ad55ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f094d600640 1 -- 172.21.15.69:0/3940253018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56020 msgr2=0x561f9ad2ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f094d600640 1 --2- 172.21.15.69:0/3940253018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56020 0x561f9ad2ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f094d600640 1 -- 172.21.15.69:0/3940253018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f9aa6a7e0 con 0x561f9ad73640 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f094e000640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f092c06c8f0 0x7f092c06edb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0940007fd0 tx=0x7f0940001770 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f094d600640 1 --2- 172.21.15.69:0/3940253018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad25fd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0938009a10 tx=0x7f0938009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8261d8d4423f7a90 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f0935c00640 1 -- 172.21.15.69:0/3940253018 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938017020 con 0x561f9ad73640 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f0935c00640 1 -- 172.21.15.69:0/3940253018 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093800ea70 con 0x561f9ad73640 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.611+0000 7f0935c00640 1 -- 172.21.15.69:0/3940253018 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093800f040 con 0x561f9ad73640 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.612+0000 7f0952f91740 1 -- 172.21.15.69:0/3940253018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 msgr2=0x561f9ad25fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.612+0000 7f0952f91740 1 --2- 172.21.15.69:0/3940253018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad25fd0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0938009a10 tx=0x7f0938009ed0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.612+0000 7f0952f91740 1 -- 172.21.15.69:0/3940253018 shutdown_connections 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.612+0000 7f0952f91740 1 --2- 172.21.15.69:0/3940253018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56020 0x561f9ad2ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.612+0000 7f0952f91740 1 --2- 172.21.15.69:0/3940253018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad56760 0x561f9ad55ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.612+0000 7f0952f91740 1 --2- 172.21.15.69:0/3940253018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad25fd0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.612+0000 7f0952f91740 1 -- 172.21.15.69:0/3940253018 >> 172.21.15.69:0/3940253018 conn(0x561f9ad28240 msgr2=0x561f9ad70ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.612+0000 7f0952f91740 1 -- 172.21.15.69:0/3940253018 shutdown_connections 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.612+0000 7f0952f91740 1 -- 172.21.15.69:0/3940253018 wait complete. 2024-04-24T08:10:47.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.613+0000 7f0952f91740 1 -- start start 2024-04-24T08:10:47.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.613+0000 7f0952f91740 1 Processor -- start 2024-04-24T08:10:47.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.614+0000 7f0952f91740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56760 0x561f9ad696b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.614+0000 7f0952f91740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 0x561f9ad58a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.614+0000 7f0952f91740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9ad71c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.614+0000 7f0952f91740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561f9ad2a3f0 con 0x561f9ad818f0 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.614+0000 7f0952f91740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561f9ac49660 con 0x561f9ad56760 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.614+0000 7f0952f91740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x561f9ad297e0 con 0x561f9ad73640 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f0952f91740 10 client.0 _ll_register_callbacks cb 0x561f9ac37aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9ad71c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9ad71c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58022/0 (socket says 172.21.15.69:58022) 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094e000640 1 -- 172.21.15.69:0/324268928 learned_addr learned my addr 172.21.15.69:0/324268928 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094d600640 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56760 0x561f9ad696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094cc00640 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 0x561f9ad58a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094e000640 1 -- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 msgr2=0x561f9ad58a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094e000640 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 0x561f9ad58a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094e000640 1 -- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56760 msgr2=0x561f9ad696b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094e000640 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56760 0x561f9ad696b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094e000640 1 -- 172.21.15.69:0/324268928 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f9aacaf70 con 0x561f9ad818f0 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.615+0000 7f094cc00640 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 0x561f9ad58a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.616+0000 7f094e000640 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9ad71c10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0940003c00 tx=0x7f0940001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c38085984786bef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.616+0000 7f092a000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.616+0000 7f092a000640 1 -- 172.21.15.69:0/324268928 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940010040 con 0x561f9ad818f0 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.616+0000 7f092a000640 1 -- 172.21.15.69:0/324268928 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f0940021920 con 0x561f9ad818f0 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.616+0000 7f0952f91740 1 -- 172.21.15.69:0/324268928 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f093800ea70 con 0x561f9ad818f0 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.616+0000 7f092a000640 1 -- 172.21.15.69:0/324268928 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f09400057e0 con 0x561f9ad818f0 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.616+0000 7f0952f91740 10 client.5976 fetch_fsmap learned FSMap version 25 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.616+0000 7f0952f91740 1 -- 172.21.15.69:0/324268928 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x561f9a962170 con 0x561f9ad818f0 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.617+0000 7f092a000640 1 -- 172.21.15.69:0/324268928 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093800ea70 con 0x561f9ad818f0 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.617+0000 7f092a000640 1 -- 172.21.15.69:0/324268928 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f094002e070 con 0x561f9ad818f0 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.617+0000 7f0952f91740 10 client.5976 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:10:47.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.617+0000 7f0952f91740 10 client.5976 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-24T08:10:47.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.617+0000 7f0952f91740 20 client.5976 populate_metadata read hostname 'smithi069' 2024-04-24T08:10:47.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.617+0000 7f0952f91740 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x561f9ad846a0 0x561f9ad56c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.617+0000 7f0952f91740 4 client.5976 mds_command: new command op to 5793 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:10:47.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.617+0000 7f0952f91740 1 -- 172.21.15.69:0/324268928 --> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f9ac9d890 con 0x561f9ad846a0 2024-04-24T08:10:47.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.618+0000 7f094d600640 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x561f9ad846a0 0x561f9ad56c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.618+0000 7f094d600640 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x561f9ad846a0 0x561f9ad56c50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c68498033c37671 server_cookie=30e396f6180a3a8e in_seq=0 out_seq=0 2024-04-24T08:10:47.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.618+0000 7f092a000640 10 client.5976 ms_handle_connect on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:47.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.620+0000 7f092a000640 1 -- 172.21.15.69:0/324268928 <== mds.0 v2:172.21.15.121:6835/3424934990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f9ac9d890 con 0x561f9ad846a0 2024-04-24T08:10:47.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.620+0000 7f092a000640 10 client.5976 handle_command_reply: tid=0 2024-04-24T08:10:47.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.620+0000 7f0952f91740 1 client.5976 shutdown 2024-04-24T08:10:47.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.620+0000 7f0952f91740 1 -- 172.21.15.69:0/324268928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 msgr2=0x561f9ad71c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.620+0000 7f0952f91740 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9ad71c10 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f0940003c00 tx=0x7f0940001250 comp rx=0 tx=0).stop 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.621+0000 7f0952f91740 1 -- 172.21.15.69:0/324268928 shutdown_connections 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.621+0000 7f0952f91740 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x561f9ad846a0 0x561f9ad56c50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.621+0000 7f0952f91740 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9ad71c10 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.621+0000 7f0952f91740 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 0x561f9ad58a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.621+0000 7f0952f91740 1 --2- 172.21.15.69:0/324268928 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56760 0x561f9ad696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.621+0000 7f0952f91740 1 -- 172.21.15.69:0/324268928 >> 172.21.15.69:0/324268928 conn(0x561f9ad56020 msgr2=0x561f9ad72cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.621+0000 7f092a000640 0 client.5976 ms_handle_reset on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.621+0000 7f0952f91740 1 -- 172.21.15.69:0/324268928 shutdown_connections 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.621+0000 7f0952f91740 1 -- 172.21.15.69:0/324268928 wait complete. 2024-04-24T08:10:47.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.621+0000 7f0952f91740 20 client.5976 trim_cache size 0 max 16384 2024-04-24T08:10:47.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.625+0000 7f7146200640 1 Processor -- start 2024-04-24T08:10:47.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.625+0000 7f7146200640 1 -- start start 2024-04-24T08:10:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.626+0000 7f7146200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71400d2b80 0x7f71400d3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.626+0000 7f7146200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714015ecc0 con 0x7f7140162ff0 2024-04-24T08:10:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.626+0000 7f7146200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714006cd50 con 0x7f714015e8e0 2024-04-24T08:10:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.626+0000 7f7146200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f714006d080 con 0x7f71400d2b80 2024-04-24T08:10:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713fe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7140162ff0 0x7f71400cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49322/0 (socket says 172.21.15.69:49322) 2024-04-24T08:10:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713fe00640 1 -- 172.21.15.69:0/2790300281 learned_addr learned my addr 172.21.15.69:0/2790300281 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f7145800640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71400d2b80 0x7f71400d3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3991022460 0 0) 0x7f714015ecc0 con 0x7f7140162ff0 2024-04-24T08:10:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7128003520 con 0x7f7140162ff0 2024-04-24T08:10:47.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 410926430 0 0) 0x7f7128003520 con 0x7f7140162ff0 2024-04-24T08:10:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71400d2b80 msgr2=0x7f71400d3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713f400640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71400d2b80 0x7f71400d3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 >> v1:172.21.15.121:6789/0 conn(0x7f714015e8e0 legacy=0x7f71400d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71400d17f0 con 0x7f7140162ff0 2024-04-24T08:10:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f712c001670 con 0x7f7140162ff0 2024-04-24T08:10:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.627+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 >> v1:172.21.15.69:6789/0 conn(0x7f7140162ff0 legacy=0x7f71400cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f713f400640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71400d2b80 0x7f71280039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f713f400640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7128003fb0 0x7f7128004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f713f400640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7128004d40 0x7f71280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f714006d080 con 0x7f7128004d40 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f712800bb00 con 0x7f7128003fb0 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f712800bd00 con 0x7f71400d2b80 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f7145800640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7128004d40 0x7f71280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f713fe00640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7128003fb0 0x7f7128004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f7144e00640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71400d2b80 0x7f71280039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f7145800640 1 -- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71400d2b80 msgr2=0x7f71280039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f7145800640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71400d2b80 0x7f71280039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f7145800640 1 -- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7128003fb0 msgr2=0x7f7128004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f7145800640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7128003fb0 0x7f7128004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f7145800640 1 -- 172.21.15.69:0/2790300281 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7128003520 con 0x7f7128004d40 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.628+0000 7f7144e00640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f71400d2b80 0x7f71280039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.629+0000 7f7145800640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7128004d40 0x7f71280051c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f7138009fc0 tx=0x7f7128005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b576723490b0a800 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.630+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138009c10 con 0x7f7128004d40 2024-04-24T08:10:47.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.630+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7138036070 con 0x7f7128004d40 2024-04-24T08:10:47.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.630+0000 7f713f400640 1 -- 172.21.15.69:0/2790300281 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138031880 con 0x7f7128004d40 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.630+0000 7f7146200640 1 -- 172.21.15.69:0/2790300281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7128004d40 msgr2=0x7f71280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.630+0000 7f7146200640 1 --2- 172.21.15.69:0/2790300281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7128004d40 0x7f71280051c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f7138009fc0 tx=0x7f7128005830 comp rx=0 tx=0).stop 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.630+0000 7f7145800640 1 -- 172.21.15.69:0/2790300281 reap_dead start 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.630+0000 7f7146200640 1 -- 172.21.15.69:0/2790300281 shutdown_connections 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.630+0000 7f7146200640 1 -- 172.21.15.69:0/2790300281 >> 172.21.15.69:0/2790300281 conn(0x7f714008e030 msgr2=0x7f714009d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.631+0000 7f7146200640 1 -- 172.21.15.69:0/2790300281 shutdown_connections 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.631+0000 7f7146200640 1 -- 172.21.15.69:0/2790300281 wait complete. 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.632+0000 7f7146200640 1 Processor -- start 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.632+0000 7f7146200640 1 -- start start 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.633+0000 7f7146200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128004d40 0x7f71400b8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.633+0000 7f7146200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7140158b90 0x7f71401f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.633+0000 7f7146200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71401f46b0 0x7f71400d0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.633+0000 7f7146200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f71400d1db0 con 0x7f71401f46b0 2024-04-24T08:10:47.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.633+0000 7f7146200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f71400d1ad0 con 0x7f7140158b90 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.633+0000 7f7146200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f71400d3030 con 0x7f7128004d40 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f7145800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71401f46b0 0x7f71400d0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f7145800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71401f46b0 0x7f71400d0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58032/0 (socket says 172.21.15.69:58032) 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f7145800640 1 -- 172.21.15.69:0/1136875335 learned_addr learned my addr 172.21.15.69:0/1136875335 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f713fe00640 1 --2- 172.21.15.69:0/1136875335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7140158b90 0x7f71401f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f7144e00640 1 --2- 172.21.15.69:0/1136875335 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128004d40 0x7f71400b8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f7145800640 1 -- 172.21.15.69:0/1136875335 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128004d40 msgr2=0x7f71400b8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f7145800640 1 --2- 172.21.15.69:0/1136875335 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7128004d40 0x7f71400b8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f7145800640 1 -- 172.21.15.69:0/1136875335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7140158b90 msgr2=0x7f71401f4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f7145800640 1 --2- 172.21.15.69:0/1136875335 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7140158b90 0x7f71401f4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f7145800640 1 -- 172.21.15.69:0/1136875335 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71401f4c30 con 0x7f71401f46b0 2024-04-24T08:10:47.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.634+0000 7f7145800640 1 --2- 172.21.15.69:0/1136875335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f71401f46b0 0x7f71400d0c00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f713800e230 tx=0x7f7138009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.635+0000 7f713cc00640 1 -- 172.21.15.69:0/1136875335 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138009c10 con 0x7f71401f46b0 2024-04-24T08:10:47.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.635+0000 7f713cc00640 1 -- 172.21.15.69:0/1136875335 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7138031dd0 con 0x7f71401f46b0 2024-04-24T08:10:47.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.635+0000 7f7146200640 1 -- 172.21.15.69:0/1136875335 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71401f4f10 con 0x7f71401f46b0 2024-04-24T08:10:47.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.635+0000 7f713cc00640 1 -- 172.21.15.69:0/1136875335 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138003bd0 con 0x7f71401f46b0 2024-04-24T08:10:47.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.635+0000 7f7146200640 1 -- 172.21.15.69:0/1136875335 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71400b8990 con 0x7f71401f46b0 2024-04-24T08:10:47.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.636+0000 7f7146200640 1 -- 172.21.15.69:0/1136875335 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f7140075d50 con 0x7f71401f46b0 2024-04-24T08:10:47.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.637+0000 7f713cc00640 1 -- 172.21.15.69:0/1136875335 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7138005d10 con 0x7f71401f46b0 2024-04-24T08:10:47.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.638+0000 7f713cc00640 1 --2- 172.21.15.69:0/1136875335 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f712006c8f0 0x7f712006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.638+0000 7f713cc00640 1 -- 172.21.15.69:0/1136875335 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f71380acfd0 con 0x7f71401f46b0 2024-04-24T08:10:47.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.639+0000 7f713cc00640 1 -- 172.21.15.69:0/1136875335 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+501 (secure 0 0 0) 0x7f713804a060 con 0x7f71401f46b0 2024-04-24T08:10:47.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.639+0000 7f7146200640 1 -- 172.21.15.69:0/1136875335 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f714006cb30 con 0x7f71401f46b0 2024-04-24T08:10:47.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.639+0000 7f7144e00640 1 --2- 172.21.15.69:0/1136875335 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f712006c8f0 0x7f712006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.640+0000 7f7144e00640 1 --2- 172.21.15.69:0/1136875335 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f712006c8f0 0x7f712006edb0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7134001630 tx=0x7f7134009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.643+0000 7f0952f91740 1 Processor -- start 2024-04-24T08:10:47.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.643+0000 7f0952f91740 1 -- start start 2024-04-24T08:10:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.643+0000 7f0952f91740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad56760 0x561f9ad718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.643+0000 7f0952f91740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad59740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.643+0000 7f0952f91740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad818f0 0x561f9ad24e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.643+0000 7f0952f91740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0948200d20 con 0x561f9ad73640 2024-04-24T08:10:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.643+0000 7f0952f91740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f09481ffd20 con 0x561f9ad818f0 2024-04-24T08:10:47.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.643+0000 7f0952f91740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09480aeec0 con 0x561f9ad56760 2024-04-24T08:10:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.644+0000 7f094cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad59740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.644+0000 7f094cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad59740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58036/0 (socket says 172.21.15.69:58036) 2024-04-24T08:10:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.644+0000 7f094cc00640 1 -- 172.21.15.69:0/4287255346 learned_addr learned my addr 172.21.15.69:0/4287255346 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.644+0000 7f094cc00640 1 -- 172.21.15.69:0/4287255346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad56760 msgr2=0x561f9ad718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.644+0000 7f094cc00640 1 --2- 172.21.15.69:0/4287255346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad56760 0x561f9ad718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.644+0000 7f094cc00640 1 -- 172.21.15.69:0/4287255346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad818f0 msgr2=0x561f9ad24e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.644+0000 7f094cc00640 1 --2- 172.21.15.69:0/4287255346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad818f0 0x561f9ad24e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.644+0000 7f094cc00640 1 -- 172.21.15.69:0/4287255346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f9ad2d1d0 con 0x561f9ad73640 2024-04-24T08:10:47.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.644+0000 7f094cc00640 1 --2- 172.21.15.69:0/4287255346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad59740 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f093c0aaa50 tx=0x7f093c079c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb65b0e9b4f0b5a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.645+0000 7f0935200640 1 -- 172.21.15.69:0/4287255346 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c04b040 con 0x561f9ad73640 2024-04-24T08:10:47.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.645+0000 7f0935200640 1 -- 172.21.15.69:0/4287255346 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093c07a2b0 con 0x561f9ad73640 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.645+0000 7f0935200640 1 -- 172.21.15.69:0/4287255346 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c07e660 con 0x561f9ad73640 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.645+0000 7f0952f91740 1 -- 172.21.15.69:0/4287255346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 msgr2=0x561f9ad59740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.645+0000 7f0952f91740 1 --2- 172.21.15.69:0/4287255346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad59740 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f093c0aaa50 tx=0x7f093c079c10 comp rx=0 tx=0).stop 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.645+0000 7f0952f91740 1 -- 172.21.15.69:0/4287255346 shutdown_connections 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.645+0000 7f0952f91740 1 --2- 172.21.15.69:0/4287255346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad73640 0x561f9ad59740 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.645+0000 7f0952f91740 1 --2- 172.21.15.69:0/4287255346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad818f0 0x561f9ad24e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.645+0000 7f0952f91740 1 --2- 172.21.15.69:0/4287255346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad56760 0x561f9ad718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.645+0000 7f0952f91740 1 -- 172.21.15.69:0/4287255346 >> 172.21.15.69:0/4287255346 conn(0x561f9ad56020 msgr2=0x561f9ad68480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.646+0000 7f0952f91740 1 -- 172.21.15.69:0/4287255346 shutdown_connections 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.646+0000 7f0952f91740 1 -- 172.21.15.69:0/4287255346 wait complete. 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.646+0000 7f0952f91740 1 -- start start 2024-04-24T08:10:47.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.646+0000 7f0952f91740 1 Processor -- start 2024-04-24T08:10:47.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.647+0000 7f0952f91740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56760 0x561f9adef290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.647+0000 7f0952f91740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 0x561f9adec2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.647+0000 7f0952f91740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9adec810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.647+0000 7f0952f91740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561f9ad70ff0 con 0x561f9ad818f0 2024-04-24T08:10:47.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.647+0000 7f0952f91740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x561f9ad58d70 con 0x561f9ad56760 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.647+0000 7f0952f91740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x561f9ade2ba0 con 0x561f9ad73640 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.647+0000 7f0952f91740 10 client.0 _ll_register_callbacks cb 0x7f0940001ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9adec810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9adec810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58048/0 (socket says 172.21.15.69:58048) 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094e000640 1 -- 172.21.15.69:0/1674806803 learned_addr learned my addr 172.21.15.69:0/1674806803 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094d600640 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56760 0x561f9adef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094cc00640 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 0x561f9adec2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094e000640 1 -- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 msgr2=0x561f9adec2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094e000640 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 0x561f9adec2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094e000640 1 -- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56760 msgr2=0x561f9adef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094e000640 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56760 0x561f9adef290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094e000640 1 -- 172.21.15.69:0/1674806803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f9adef7d0 con 0x561f9ad818f0 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f094e000640 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9adec810 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f094000f620 tx=0x7f09400014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66acaa0e08f0039c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.648+0000 7f092a000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f092a000640 1 -- 172.21.15.69:0/1674806803 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0940034020 con 0x561f9ad818f0 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f092a000640 1 -- 172.21.15.69:0/1674806803 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f0940021460 con 0x561f9ad818f0 2024-04-24T08:10:47.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f0952f91740 1 -- 172.21.15.69:0/1674806803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f093c07a2b0 con 0x561f9ad818f0 2024-04-24T08:10:47.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f092a000640 1 -- 172.21.15.69:0/1674806803 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0940021660 con 0x561f9ad818f0 2024-04-24T08:10:47.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f0952f91740 10 client.5988 fetch_fsmap learned FSMap version 25 2024-04-24T08:10:47.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f0952f91740 1 -- 172.21.15.69:0/1674806803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x561f9ad59b20 con 0x561f9ad818f0 2024-04-24T08:10:47.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f092a000640 1 -- 172.21.15.69:0/1674806803 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093c07a2b0 con 0x561f9ad818f0 2024-04-24T08:10:47.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f092a000640 1 -- 172.21.15.69:0/1674806803 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f094002e070 con 0x561f9ad818f0 2024-04-24T08:10:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f0952f91740 10 client.5988 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:10:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f0952f91740 10 client.5988 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-24T08:10:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f0952f91740 20 client.5988 populate_metadata read hostname 'smithi069' 2024-04-24T08:10:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f0952f91740 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x561f9ad53100 0x561f9ad73f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f0952f91740 4 client.5988 mds_command: new command op to 5793 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-04-24T08:10:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.649+0000 7f0952f91740 1 -- 172.21.15.69:0/1674806803 --> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) -- 0x561f9ac97700 con 0x561f9ad53100 2024-04-24T08:10:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.650+0000 7f094d600640 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x561f9ad53100 0x561f9ad73f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.651+0000 7f094d600640 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x561f9ad53100 0x561f9ad73f00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43fd443bc1217809 server_cookie=7ce42ceadcf30548 in_seq=0 out_seq=0 2024-04-24T08:10:47.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.651+0000 7f092a000640 10 client.5988 ms_handle_connect on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:47.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f092a000640 1 -- 172.21.15.69:0/1674806803 <== mds.0 v2:172.21.15.121:6835/3424934990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561f9ac97700 con 0x561f9ad53100 2024-04-24T08:10:47.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f092a000640 10 client.5988 handle_command_reply: tid=0 2024-04-24T08:10:47.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 client.5988 shutdown 2024-04-24T08:10:47.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 -- 172.21.15.69:0/1674806803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 msgr2=0x561f9adec810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9adec810 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f094000f620 tx=0x7f09400014a0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 -- 172.21.15.69:0/1674806803 shutdown_connections 2024-04-24T08:10:47.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x561f9ad53100 0x561f9ad73f00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561f9ad818f0 0x561f9adec810 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x561f9ad73640 0x561f9adec2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f092a000640 0 client.5988 ms_handle_reset on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:47.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 --2- 172.21.15.69:0/1674806803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x561f9ad56760 0x561f9adef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 -- 172.21.15.69:0/1674806803 >> 172.21.15.69:0/1674806803 conn(0x561f9ad56020 msgr2=0x561f9ad45b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.655 INFO:tasks.ceph.mds.d.smithi121.stderr:2024-04-24T08:10:47.654+0000 7f6127e00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={5940=0-4194304@1} caps={5940=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5940 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x55eaeeaa2c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-24T08:10:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 -- 172.21.15.69:0/1674806803 shutdown_connections 2024-04-24T08:10:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 1 -- 172.21.15.69:0/1674806803 wait complete. 2024-04-24T08:10:47.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.653+0000 7f0952f91740 20 client.5988 trim_cache size 0 max 16384 2024-04-24T08:10:47.657 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:10:47.657 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-04-24T08:10:47.657 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "cdfd7ecb-f66d-4ffd-8dee-2cc8c69b1c6c", 2024-04-24T08:10:47.657 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-04-24T08:10:47.657 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 -- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f092c06c8f0 msgr2=0x7f092c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f092c06c8f0 0x7f092c06edb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0940007fd0 tx=0x7f0940001770 comp rx=0 tx=0).stop 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 -- 172.21.15.69:0/3226429998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480b27a0 msgr2=0x7f09480cb4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480b27a0 0x7f09480cb4a0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f093c009870 tx=0x7f093c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 -- 172.21.15.69:0/3226429998 shutdown_connections 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f092c06c8f0 0x7f092c06edb0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09480b27a0 0x7f09480cb4a0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09480ce540 0x7f09481602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 --2- 172.21.15.69:0/3226429998 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f094806bf00 0x7f09480b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 -- 172.21.15.69:0/3226429998 >> 172.21.15.69:0/3226429998 conn(0x7f09480b2e00 msgr2=0x7f09480d0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 -- 172.21.15.69:0/3226429998 shutdown_connections 2024-04-24T08:10:47.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.654+0000 7f094ea00640 1 -- 172.21.15.69:0/3226429998 wait complete. 2024-04-24T08:10:47.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:47.673 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub status 2024-04-24T08:10:47.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.851+0000 7f73a4c00640 1 Processor -- start 2024-04-24T08:10:47.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.851+0000 7f73a4c00640 1 -- start start 2024-04-24T08:10:47.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.852+0000 7f73a4c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f73a00b3e90 0x7f73a00b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f73a4c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73a00ce250 0x7f73a00ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f73a4c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a015cc00 con 0x7f73a015c820 2024-04-24T08:10:47.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f73a4c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f73a00b4be0 con 0x7f73a00b3e90 2024-04-24T08:10:47.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f73a4c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f73a006ed30 con 0x7f73a00ce250 2024-04-24T08:10:47.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f739de00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f73a015c820 0x7f73a006f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49332/0 (socket says 172.21.15.69:49332) 2024-04-24T08:10:47.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f739de00640 1 -- 172.21.15.69:0/4101105561 learned_addr learned my addr 172.21.15.69:0/4101105561 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:10:47.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f7397e00640 1 --2- 172.21.15.69:0/4101105561 >> v2:172.21.15.69:3300/0 conn(0x7f73a00b3e90 0x7f73a00b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f739e800640 1 --2- 172.21.15.69:0/4101105561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73a00ce250 0x7f73a00ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f7397e00640 1 -- 172.21.15.69:0/4101105561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73a00ce250 msgr2=0x7f73a00ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f7397e00640 1 --2- 172.21.15.69:0/4101105561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73a00ce250 0x7f73a00ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f7397e00640 1 -- 172.21.15.69:0/4101105561 >> v1:172.21.15.69:6789/0 conn(0x7f73a015c820 legacy=0x7f73a006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.853+0000 7f7397e00640 1 -- 172.21.15.69:0/4101105561 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a00b4e10 con 0x7f73a00b3e90 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.854+0000 7f7397e00640 1 --2- 172.21.15.69:0/4101105561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 0x7f73a00b46a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f73880097a0 tx=0x7f738802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f60d8f9d106ca8e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.855+0000 7f739d400640 1 -- 172.21.15.69:0/4101105561 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388037070 con 0x7f73a00b3e90 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.855+0000 7f739d400640 1 -- 172.21.15.69:0/4101105561 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7388032440 con 0x7f73a00b3e90 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.855+0000 7f739d400640 1 -- 172.21.15.69:0/4101105561 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7388032810 con 0x7f73a00b3e90 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.855+0000 7f73a4c00640 1 -- 172.21.15.69:0/4101105561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 msgr2=0x7f73a00b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.855+0000 7f73a4c00640 1 --2- 172.21.15.69:0/4101105561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 0x7f73a00b46a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f73880097a0 tx=0x7f738802a180 comp rx=0 tx=0).stop 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.856+0000 7f73a4c00640 1 -- 172.21.15.69:0/4101105561 shutdown_connections 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.856+0000 7f73a4c00640 1 --2- 172.21.15.69:0/4101105561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73a00ce250 0x7f73a00ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.856+0000 7f73a4c00640 1 --2- 172.21.15.69:0/4101105561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 0x7f73a00b46a0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.856+0000 7f73a4c00640 1 -- 172.21.15.69:0/4101105561 >> 172.21.15.69:0/4101105561 conn(0x7f73a00d0350 msgr2=0x7f73a00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.856+0000 7f73a4c00640 1 -- 172.21.15.69:0/4101105561 shutdown_connections 2024-04-24T08:10:47.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.856+0000 7f73a4c00640 1 -- 172.21.15.69:0/4101105561 wait complete. 2024-04-24T08:10:47.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.857+0000 7f73a4c00640 1 Processor -- start 2024-04-24T08:10:47.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.858+0000 7f73a4c00640 1 -- start start 2024-04-24T08:10:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.858+0000 7f73a4c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 0x7f73a01f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.858+0000 7f73a4c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73a00ce250 0x7f73a00ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.858+0000 7f73a4c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73a015c820 0x7f73a00aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f73a4c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73a006f2b0 con 0x7f73a00b3e90 2024-04-24T08:10:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f73a4c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f73a006efd0 con 0x7f73a015c820 2024-04-24T08:10:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f73a4c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f73a006fee0 con 0x7f73a00ce250 2024-04-24T08:10:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f739de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 0x7f73a01f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f739de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 0x7f73a01f1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58070/0 (socket says 172.21.15.69:58070) 2024-04-24T08:10:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f739de00640 1 -- 172.21.15.69:0/1616681980 learned_addr learned my addr 172.21.15.69:0/1616681980 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f7397e00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73a00ce250 0x7f73a00ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f739de00640 1 -- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73a00ce250 msgr2=0x7f73a00ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f739de00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73a00ce250 0x7f73a00ae8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f739de00640 1 -- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73a015c820 msgr2=0x7f73a00aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f739de00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73a015c820 0x7f73a00aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f739de00640 1 -- 172.21.15.69:0/1616681980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a01e6260 con 0x7f73a00b3e90 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.859+0000 7f739de00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 0x7f73a01f1ed0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f739000c400 tx=0x7f739000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.860+0000 7f7396000640 1 -- 172.21.15.69:0/1616681980 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7390017070 con 0x7f73a00b3e90 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.860+0000 7f7396000640 1 -- 172.21.15.69:0/1616681980 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7390005ce0 con 0x7f73a00b3e90 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.860+0000 7f73a4c00640 1 -- 172.21.15.69:0/1616681980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a01e6540 con 0x7f73a00b3e90 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.860+0000 7f7396000640 1 -- 172.21.15.69:0/1616681980 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900072f0 con 0x7f73a00b3e90 2024-04-24T08:10:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.860+0000 7f73a4c00640 1 -- 172.21.15.69:0/1616681980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a01f32f0 con 0x7f73a00b3e90 2024-04-24T08:10:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.862+0000 7f7396000640 1 -- 172.21.15.69:0/1616681980 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f739001a070 con 0x7f73a00b3e90 2024-04-24T08:10:47.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.863+0000 7f7396000640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f737c06c8f0 0x7f737c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.863+0000 7f7396000640 1 -- 172.21.15.69:0/1616681980 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f739008b060 con 0x7f73a00b3e90 2024-04-24T08:10:47.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.864+0000 7f73a7a24740 1 Processor -- start 2024-04-24T08:10:47.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.864+0000 7f7397e00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f737c06c8f0 0x7f737c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.864+0000 7f73a7a24740 1 -- start start 2024-04-24T08:10:47.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fbbac30 0x55667fb6d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fb9dd50 0x55667fb9d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fb9d610 0x55667fb72420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f73a7a24740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55667fad71f0 con 0x55667fbbac30 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f73a7a24740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55667fa7d350 con 0x55667fb9d610 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f73a7a24740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55667fad31b0 con 0x55667fb9dd50 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f7397e00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f737c06c8f0 0x7f737c06edb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7388034000 tx=0x7f738802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f739de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fbbac30 0x55667fb6d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f739de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fbbac30 0x55667fb6d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58082/0 (socket says 172.21.15.69:58082) 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f739de00640 1 -- 172.21.15.69:0/973086518 learned_addr learned my addr 172.21.15.69:0/973086518 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f739de00640 1 -- 172.21.15.69:0/973086518 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fb9dd50 msgr2=0x55667fb9d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.865+0000 7f739e800640 1 --2- 172.21.15.69:0/973086518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fb9d610 0x55667fb72420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.866+0000 7f7397e00640 1 --2- 172.21.15.69:0/973086518 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fb9dd50 0x55667fb9d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.866+0000 7f739de00640 1 --2- 172.21.15.69:0/973086518 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fb9dd50 0x55667fb9d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.866+0000 7f739de00640 1 -- 172.21.15.69:0/973086518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fb9d610 msgr2=0x55667fb72420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.866+0000 7f739de00640 1 --2- 172.21.15.69:0/973086518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fb9d610 0x55667fb72420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.866+0000 7f739de00640 1 -- 172.21.15.69:0/973086518 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55667fad1f50 con 0x55667fbbac30 2024-04-24T08:10:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.866+0000 7f739e800640 1 --2- 172.21.15.69:0/973086518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fb9d610 0x55667fb72420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:47.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.866+0000 7f739de00640 1 --2- 172.21.15.69:0/973086518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fbbac30 0x55667fb6d5c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7390089490 tx=0x7f7390058da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5938fdf1746d5e6d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.866+0000 7f738e600640 1 -- 172.21.15.69:0/973086518 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7390056f60 con 0x55667fbbac30 2024-04-24T08:10:47.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.866+0000 7f738e600640 1 -- 172.21.15.69:0/973086518 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7390082e50 con 0x55667fbbac30 2024-04-24T08:10:47.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.866+0000 7f738e600640 1 -- 172.21.15.69:0/973086518 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900552d0 con 0x55667fbbac30 2024-04-24T08:10:47.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.867+0000 7f73a7a24740 1 -- 172.21.15.69:0/973086518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fbbac30 msgr2=0x55667fb6d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.867+0000 7f73a7a24740 1 --2- 172.21.15.69:0/973086518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fbbac30 0x55667fb6d5c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f7390089490 tx=0x7f7390058da0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.867+0000 7f73a7a24740 1 -- 172.21.15.69:0/973086518 shutdown_connections 2024-04-24T08:10:47.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.867+0000 7f73a7a24740 1 --2- 172.21.15.69:0/973086518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fb9d610 0x55667fb72420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.867+0000 7f73a7a24740 1 --2- 172.21.15.69:0/973086518 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fb9dd50 0x55667fb9d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.867+0000 7f73a7a24740 1 --2- 172.21.15.69:0/973086518 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fbbac30 0x55667fb6d5c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.867+0000 7f73a7a24740 1 -- 172.21.15.69:0/973086518 >> 172.21.15.69:0/973086518 conn(0x55667fb6f830 msgr2=0x55667fbb80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.867+0000 7f73a7a24740 1 -- 172.21.15.69:0/973086518 shutdown_connections 2024-04-24T08:10:47.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.867+0000 7f73a7a24740 1 -- 172.21.15.69:0/973086518 wait complete. 2024-04-24T08:10:47.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.868+0000 7f73a7a24740 1 -- start start 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.868+0000 7f73a7a24740 1 Processor -- start 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.869+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fbb7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.869+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fbbac30 0x55667fb98c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.869+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fba0190 0x55667fbb80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.869+0000 7f73a7a24740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55667fb719e0 con 0x55667fb9dd50 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.869+0000 7f73a7a24740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55667fa90690 con 0x55667fbbac30 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.869+0000 7f73a7a24740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55667fb70dd0 con 0x55667fba0190 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.869+0000 7f73a7a24740 10 client.0 _ll_register_callbacks cb 0x55667facc650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.869+0000 7f739de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fbb7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.869+0000 7f739de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fbb7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58088/0 (socket says 172.21.15.69:58088) 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.869+0000 7f739de00640 1 -- 172.21.15.69:0/861097527 learned_addr learned my addr 172.21.15.69:0/861097527 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f739de00640 1 -- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fba0190 msgr2=0x55667fbb80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f739de00640 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fba0190 0x55667fbb80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f7397e00640 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fbbac30 0x55667fb98c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f739de00640 1 -- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fbbac30 msgr2=0x55667fb98c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f739de00640 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fbbac30 0x55667fb98c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f739de00640 1 -- 172.21.15.69:0/861097527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55667fa8f920 con 0x55667fb9dd50 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f7397e00640 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fbbac30 0x55667fb98c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f739de00640 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fbb7700 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f7390058960 tx=0x7f7390000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bdcf8a2e11d0f6a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f7382a00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f7382a00640 1 -- 172.21.15.69:0/861097527 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7390088070 con 0x55667fb9dd50 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f7382a00640 1 -- 172.21.15.69:0/861097527 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f7390078170 con 0x55667fb9dd50 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f73a7a24740 1 -- 172.21.15.69:0/861097527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7390082e50 con 0x55667fb9dd50 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f7382a00640 1 -- 172.21.15.69:0/861097527 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7390007490 con 0x55667fb9dd50 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f73a7a24740 10 client.6003 fetch_fsmap learned FSMap version 25 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.870+0000 7f73a7a24740 1 -- 172.21.15.69:0/861097527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55667f8b37d0 con 0x55667fb9dd50 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.871+0000 7f7382a00640 1 -- 172.21.15.69:0/861097527 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7390082e50 con 0x55667fb9dd50 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.871+0000 7f7382a00640 1 -- 172.21.15.69:0/861097527 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f73900580b0 con 0x55667fb9dd50 2024-04-24T08:10:47.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.871+0000 7f73a7a24740 10 client.6003 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:10:47.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.871+0000 7f73a7a24740 10 client.6003 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-24T08:10:47.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.871+0000 7f73a7a24740 20 client.6003 populate_metadata read hostname 'smithi069' 2024-04-24T08:10:47.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.871+0000 7f73a7a24740 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55667fbb3430 0x55667fbb38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.871+0000 7f73a7a24740 4 client.6003 mds_command: new command op to 5793 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:10:47.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.871+0000 7f73a7a24740 1 -- 172.21.15.69:0/861097527 --> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55667fae4c10 con 0x55667fbb3430 2024-04-24T08:10:47.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.872+0000 7f7397e00640 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55667fbb3430 0x55667fbb38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.872+0000 7f7397e00640 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55667fbb3430 0x55667fbb38a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4514ae1e0d14fc21 server_cookie=28f543c7d8955b7a in_seq=0 out_seq=0 2024-04-24T08:10:47.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.872+0000 7f7382a00640 10 client.6003 ms_handle_connect on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:47.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.874+0000 7f7382a00640 1 -- 172.21.15.69:0/861097527 <== mds.0 v2:172.21.15.121:6835/3424934990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55667fae4c10 con 0x55667fbb3430 2024-04-24T08:10:47.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.874+0000 7f7382a00640 10 client.6003 handle_command_reply: tid=0 2024-04-24T08:10:47.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.874+0000 7f73a7a24740 1 client.6003 shutdown 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 1 -- 172.21.15.69:0/861097527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 msgr2=0x55667fbb7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fbb7700 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f7390058960 tx=0x7f7390000bc0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 1 -- 172.21.15.69:0/861097527 shutdown_connections 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55667fbb3430 0x55667fbb38a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fba0190 0x55667fbb80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fbbac30 0x55667fb98c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 1 --2- 172.21.15.69:0/861097527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fbb7700 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f7382a00640 0 client.6003 ms_handle_reset on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 1 -- 172.21.15.69:0/861097527 >> 172.21.15.69:0/861097527 conn(0x55667fb9d610 msgr2=0x55667fb99750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 1 -- 172.21.15.69:0/861097527 shutdown_connections 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 1 -- 172.21.15.69:0/861097527 wait complete. 2024-04-24T08:10:47.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.875+0000 7f73a7a24740 20 client.6003 trim_cache size 0 max 16384 2024-04-24T08:10:47.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.894+0000 7f73a7a24740 1 Processor -- start 2024-04-24T08:10:47.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.894+0000 7f73a7a24740 1 -- start start 2024-04-24T08:10:47.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fb99a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 0x55667fc33df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fbbac30 0x55667fb6c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f73a7a24740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f73a01fcc20 con 0x55667fb9dd50 2024-04-24T08:10:47.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f73a7a24740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f73a0073030 con 0x55667fba0190 2024-04-24T08:10:47.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f73a7a24740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f73a01f67b0 con 0x55667fbbac30 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f739de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fb99a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f739de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fb99a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58094/0 (socket says 172.21.15.69:58094) 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f739de00640 1 -- 172.21.15.69:0/1593005926 learned_addr learned my addr 172.21.15.69:0/1593005926 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f7397e00640 1 --2- 172.21.15.69:0/1593005926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 0x55667fc33df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f739de00640 1 -- 172.21.15.69:0/1593005926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fbbac30 msgr2=0x55667fb6c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f739e800640 1 --2- 172.21.15.69:0/1593005926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fbbac30 0x55667fb6c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f739de00640 1 --2- 172.21.15.69:0/1593005926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fbbac30 0x55667fb6c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f739de00640 1 -- 172.21.15.69:0/1593005926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 msgr2=0x55667fc33df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.895+0000 7f739de00640 1 --2- 172.21.15.69:0/1593005926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 0x55667fc33df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.896+0000 7f739de00640 1 -- 172.21.15.69:0/1593005926 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55667f7e93f0 con 0x55667fb9dd50 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.896+0000 7f7397e00640 1 --2- 172.21.15.69:0/1593005926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 0x55667fc33df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.896+0000 7f739de00640 1 --2- 172.21.15.69:0/1593005926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fb99a00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f739000cdc0 tx=0x7f7390088000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f95f28ba475d0757 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.896+0000 7f738dc00640 1 -- 172.21.15.69:0/1593005926 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900580b0 con 0x55667fb9dd50 2024-04-24T08:10:47.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.896+0000 7f738dc00640 1 -- 172.21.15.69:0/1593005926 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7390082e50 con 0x55667fb9dd50 2024-04-24T08:10:47.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.896+0000 7f738dc00640 1 -- 172.21.15.69:0/1593005926 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900549b0 con 0x55667fb9dd50 2024-04-24T08:10:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.897+0000 7f73a7a24740 1 -- 172.21.15.69:0/1593005926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 msgr2=0x55667fb99a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.897+0000 7f73a7a24740 1 --2- 172.21.15.69:0/1593005926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fb99a00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f739000cdc0 tx=0x7f7390088000 comp rx=0 tx=0).stop 2024-04-24T08:10:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.897+0000 7f73a7a24740 1 -- 172.21.15.69:0/1593005926 shutdown_connections 2024-04-24T08:10:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.897+0000 7f73a7a24740 1 --2- 172.21.15.69:0/1593005926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fbbac30 0x55667fb6c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.897+0000 7f73a7a24740 1 --2- 172.21.15.69:0/1593005926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 0x55667fc33df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.897+0000 7f73a7a24740 1 --2- 172.21.15.69:0/1593005926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fb99a00 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.897+0000 7f73a7a24740 1 -- 172.21.15.69:0/1593005926 >> 172.21.15.69:0/1593005926 conn(0x55667fb9d610 msgr2=0x55667fb98a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.897+0000 7f73a7a24740 1 -- 172.21.15.69:0/1593005926 shutdown_connections 2024-04-24T08:10:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.897+0000 7f73a7a24740 1 -- 172.21.15.69:0/1593005926 wait complete. 2024-04-24T08:10:47.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.897+0000 7f73a7a24740 1 -- start start 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.898+0000 7f73a7a24740 1 Processor -- start 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.898+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fc32e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.898+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 0x55667fb96f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.898+0000 7f73a7a24740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fbbac30 0x55667fb974d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.898+0000 7f73a7a24740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55667fb9f500 con 0x55667fb9dd50 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.898+0000 7f73a7a24740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f73a0158f10 con 0x55667fba0190 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f73a7a24740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55667fbb8350 con 0x55667fbbac30 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f73a7a24740 10 client.0 _ll_register_callbacks cb 0x7f73900593f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f739de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fc32e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f739de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fc32e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58110/0 (socket says 172.21.15.69:58110) 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f739de00640 1 -- 172.21.15.69:0/2996497210 learned_addr learned my addr 172.21.15.69:0/2996497210 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f7397e00640 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 0x55667fb96f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f739e800640 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fbbac30 0x55667fb974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f739de00640 1 -- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fbbac30 msgr2=0x55667fb974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f739de00640 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fbbac30 0x55667fb974d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f739de00640 1 -- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 msgr2=0x55667fb96f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f739de00640 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 0x55667fb96f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f739de00640 1 -- 172.21.15.69:0/2996497210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55667f7a90f0 con 0x55667fb9dd50 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.899+0000 7f7397e00640 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 0x55667fb96f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f739de00640 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fc32e80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f73900550c0 tx=0x7f7390000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e35597a083ea5a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f7382a00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f7382a00640 1 -- 172.21.15.69:0/2996497210 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7390088070 con 0x55667fb9dd50 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f7382a00640 1 -- 172.21.15.69:0/2996497210 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(44..44 src has 1..44) ==== 5709+0+0 (secure 0 0 0) 0x7f7390054bb0 con 0x55667fb9dd50 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f73a7a24740 1 -- 172.21.15.69:0/2996497210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7390082e50 con 0x55667fb9dd50 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f7382a00640 1 -- 172.21.15.69:0/2996497210 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7390057b20 con 0x55667fb9dd50 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f73a7a24740 10 client.6009 fetch_fsmap learned FSMap version 25 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f73a7a24740 1 -- 172.21.15.69:0/2996497210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x55667fa9c810 con 0x55667fb9dd50 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f7382a00640 1 -- 172.21.15.69:0/2996497210 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7390082e50 con 0x55667fb9dd50 2024-04-24T08:10:47.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f7382a00640 1 -- 172.21.15.69:0/2996497210 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f7390057cc0 con 0x55667fb9dd50 2024-04-24T08:10:47.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f73a7a24740 10 client.6009 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:10:47.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f73a7a24740 10 client.6009 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-24T08:10:47.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f73a7a24740 20 client.6009 populate_metadata read hostname 'smithi069' 2024-04-24T08:10:47.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f73a7a24740 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55667fbb9af0 0x55667fb99a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:47.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f73a7a24740 4 client.6009 mds_command: new command op to 5793 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:10:47.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.900+0000 7f73a7a24740 1 -- 172.21.15.69:0/2996497210 --> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55667fadea80 con 0x55667fbb9af0 2024-04-24T08:10:47.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.901+0000 7f7397e00640 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55667fbb9af0 0x55667fb99a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:47.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.902+0000 7f7397e00640 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55667fbb9af0 0x55667fb99a00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6097bc8f115b59a9 server_cookie=44b763beb50e9da6 in_seq=0 out_seq=0 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.902+0000 7f7382a00640 10 client.6009 ms_handle_connect on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f7382a00640 1 -- 172.21.15.69:0/2996497210 <== mds.0 v2:172.21.15.121:6835/3424934990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+260 (crc 0 0 0) 0x55667fadea80 con 0x55667fbb9af0 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f7382a00640 10 client.6009 handle_command_reply: tid=0 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 client.6009 shutdown 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 -- 172.21.15.69:0/2996497210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 msgr2=0x55667fc32e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fc32e80 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f73900550c0 tx=0x7f7390000ed0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 -- 172.21.15.69:0/2996497210 shutdown_connections 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x55667fbb9af0 0x55667fb99a00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55667fbbac30 0x55667fb974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55667fba0190 0x55667fb96f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 --2- 172.21.15.69:0/2996497210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55667fb9dd50 0x55667fc32e80 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f7382a00640 0 client.6009 ms_handle_reset on v2:172.21.15.121:6835/3424934990 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 -- 172.21.15.69:0/2996497210 >> 172.21.15.69:0/2996497210 conn(0x55667fb9d610 msgr2=0x55667fbcba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 -- 172.21.15.69:0/2996497210 shutdown_connections 2024-04-24T08:10:47.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 1 -- 172.21.15.69:0/2996497210 wait complete. 2024-04-24T08:10:47.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.903+0000 7f73a7a24740 20 client.6009 trim_cache size 0 max 16384 2024-04-24T08:10:47.908 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:10:47.908 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:10:47.908 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-04-24T08:10:47.908 INFO:teuthology.orchestra.run.smithi069.stdout: "cdfd7ecb-f66d-4ffd-8dee-2cc8c69b1c6c": { 2024-04-24T08:10:47.908 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/", 2024-04-24T08:10:47.908 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "cdfd7ecb-f66d-4ffd-8dee-2cc8c69b1c6c", 2024-04-24T08:10:47.908 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,repair" 2024-04-24T08:10:47.908 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:47.908 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:10:47.909 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:10:47.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 -- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f737c06c8f0 msgr2=0x7f737c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f737c06c8f0 0x7f737c06edb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7388034000 tx=0x7f738802e000 comp rx=0 tx=0).stop 2024-04-24T08:10:47.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 -- 172.21.15.69:0/1616681980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 msgr2=0x7f73a01f1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:47.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 0x7f73a01f1ed0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f739000c400 tx=0x7f739000c8c0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 -- 172.21.15.69:0/1616681980 shutdown_connections 2024-04-24T08:10:47.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f737c06c8f0 0x7f737c06edb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73a015c820 0x7f73a00aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f73a00ce250 0x7f73a00ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 --2- 172.21.15.69:0/1616681980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f73a00b3e90 0x7f73a01f1ed0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:47.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 -- 172.21.15.69:0/1616681980 >> 172.21.15.69:0/1616681980 conn(0x7f73a00d0350 msgr2=0x7f73a00cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:47.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 -- 172.21.15.69:0/1616681980 shutdown_connections 2024-04-24T08:10:47.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:47.904+0000 7f73a4c00640 1 -- 172.21.15.69:0/1616681980 wait complete. 2024-04-24T08:10:47.921 INFO:tasks.cephfs.filesystem:scrub status for tag:cdfd7ecb-f66d-4ffd-8dee-2cc8c69b1c6c - {'path': '/', 'tag': 'cdfd7ecb-f66d-4ffd-8dee-2cc8c69b1c6c', 'options': 'recursive,repair'} 2024-04-24T08:10:49.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:49.035+0000 7f713cc00640 1 -- 172.21.15.69:0/1136875335 <== mon.0 v2:172.21.15.69:3300/0 7 ==== log(6 entries from seq 1 at 2024-04-24T08:10:47.653581+0000) ==== 1078+0+0 (secure 0 0 0) 0x7f7138078d00 con 0x7f71401f46b0 2024-04-24T08:10:49.036 INFO:teuthology.orchestra.run.smithi069.stdout: cluster: 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: id: 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: health: HEALTH_OK 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: services: 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: mgr: x(active, since 2m), standbys: y 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: mds: 1/1 daemons up, 3 standby 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: data: 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: volumes: 1/1 healthy 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: pools: 3 pools, 97 pgs 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: objects: 24 objects, 579 KiB 2024-04-24T08:10:49.037 INFO:teuthology.orchestra.run.smithi069.stdout: usage: 220 MiB used, 720 GiB / 720 GiB avail 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout: pgs: 97 active+clean 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout: io: 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout: client: 1.4 KiB/s wr, 0 op/s rd, 4 op/s wr 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:10:47.653581+0000 mds.d [INF] scrub queued for path: / 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:10:47.653588+0000 mds.d [INF] scrub summary: idle+waiting paths [/] 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:10:47.653599+0000 mds.d [INF] scrub summary: active paths [/] 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:10:47.655647+0000 mds.d [WRN] bad backtrace on inode 0x10000000001(/parent_a/alpha), rewriting it 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:10:47.655670+0000 mds.d [INF] Scrub repaired inode 0x10000000001 (/parent_a/alpha) 2024-04-24T08:10:49.038 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:10:47.656386+0000 mds.d [INF] scrub summary: idle+waiting paths [/] 2024-04-24T08:10:52.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:10:52.044+0000 7f713cc00640 1 -- 172.21.15.69:0/1136875335 <== mon.0 v2:172.21.15.69:3300/0 8 ==== log(1 entries from seq 7 at 2024-04-24T08:10:50.486691+0000) ==== 189+0+0 (secure 0 0 0) 0x7f7138078f60 con 0x7f71401f46b0 2024-04-24T08:10:52.046 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:10:50.486691+0000 mds.d [INF] scrub summary: idle 2024-04-24T08:11:02.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:02.077+0000 7f713cc00640 1 -- 172.21.15.69:0/1136875335 <== mon.0 v2:172.21.15.69:3300/0 9 ==== log(4 entries from seq 122 at 2024-04-24T08:11:01.420432+0000) ==== 1120+0+0 (secure 0 0 0) 0x7f71380ab570 con 0x7f71401f46b0 2024-04-24T08:11:03.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:03.097+0000 7f713cc00640 1 -- 172.21.15.69:0/1136875335 <== mon.0 v2:172.21.15.69:3300/0 10 ==== log(2 entries from seq 474 at 2024-04-24T08:11:02.084608+0000) ==== 557+0+0 (secure 0 0 0) 0x7f71380772d0 con 0x7f71401f46b0 2024-04-24T08:11:14.121 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:11:14.129 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:11:17.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:17.923 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.3:0 scrub status 2024-04-24T08:11:18.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.134+0000 7f2a71800640 1 Processor -- start 2024-04-24T08:11:18.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.135+0000 7f2a71800640 1 -- start start 2024-04-24T08:11:18.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.136+0000 7f2a71800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a6c071d70 0x7f2a6c072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.136+0000 7f2a71800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2a6c15edf0 0x7f2a6c15f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.136+0000 7f2a71800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a6c072690 con 0x7f2a6c162270 2024-04-24T08:11:18.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.136+0000 7f2a71800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2a6c0d0a00 con 0x7f2a6c071d70 2024-04-24T08:11:18.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.136+0000 7f2a71800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2a6c06f010 con 0x7f2a6c15edf0 2024-04-24T08:11:18.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.137+0000 7f2a6be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a6c071d70 0x7f2a6c072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.137+0000 7f2a6be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2a6c071d70 0x7f2a6c072150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55112/0 (socket says 172.21.15.69:55112) 2024-04-24T08:11:18.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.137+0000 7f2a6be00640 1 -- 172.21.15.69:0/847139176 learned_addr learned my addr 172.21.15.69:0/847139176 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:18.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.137+0000 7f2a6be00640 1 -- 172.21.15.69:0/847139176 >> v2:172.21.15.121:3300/0 conn(0x7f2a6c15edf0 msgr2=0x7f2a6c15f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.137+0000 7f2a6be00640 1 --2- 172.21.15.69:0/847139176 >> v2:172.21.15.121:3300/0 conn(0x7f2a6c15edf0 0x7f2a6c15f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.137+0000 7f2a6be00640 1 -- 172.21.15.69:0/847139176 >> v1:172.21.15.121:6789/0 conn(0x7f2a6c162270 legacy=0x7f2a6c162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.137+0000 7f2a6be00640 1 -- 172.21.15.69:0/847139176 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c15fbb0 con 0x7f2a6c071d70 2024-04-24T08:11:18.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.138+0000 7f2a6be00640 1 --2- 172.21.15.69:0/847139176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 0x7f2a6c072150 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c0730b0 tx=0x7f2a5c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d55db1d09f1df2f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.138+0000 7f2a6aa00640 1 -- 172.21.15.69:0/847139176 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c037070 con 0x7f2a6c071d70 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.139+0000 7f2a6aa00640 1 -- 172.21.15.69:0/847139176 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c032440 con 0x7f2a6c071d70 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.139+0000 7f2a6aa00640 1 -- 172.21.15.69:0/847139176 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c032810 con 0x7f2a6c071d70 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.139+0000 7f2a71800640 1 -- 172.21.15.69:0/847139176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 msgr2=0x7f2a6c072150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.139+0000 7f2a71800640 1 --2- 172.21.15.69:0/847139176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 0x7f2a6c072150 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c0730b0 tx=0x7f2a5c02a230 comp rx=0 tx=0).stop 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.139+0000 7f2a71800640 1 -- 172.21.15.69:0/847139176 shutdown_connections 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.139+0000 7f2a71800640 1 --2- 172.21.15.69:0/847139176 >> v2:172.21.15.121:3300/0 conn(0x7f2a6c15edf0 0x7f2a6c15f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.139+0000 7f2a71800640 1 --2- 172.21.15.69:0/847139176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 0x7f2a6c072150 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.139+0000 7f2a71800640 1 -- 172.21.15.69:0/847139176 >> 172.21.15.69:0/847139176 conn(0x7f2a6c067b80 msgr2=0x7f2a6c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.139+0000 7f2a71800640 1 -- 172.21.15.69:0/847139176 shutdown_connections 2024-04-24T08:11:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.140+0000 7f2a71800640 1 -- 172.21.15.69:0/847139176 wait complete. 2024-04-24T08:11:18.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.141+0000 7f2a71800640 1 Processor -- start 2024-04-24T08:11:18.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.141+0000 7f2a71800640 1 -- start start 2024-04-24T08:11:18.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.142+0000 7f2a71800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 0x7f2a6c0cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.142+0000 7f2a71800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a6c15edf0 0x7f2a6c135bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.142+0000 7f2a71800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a6c162270 0x7f2a6c1f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.142+0000 7f2a71800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c0d0690 con 0x7f2a6c071d70 2024-04-24T08:11:18.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.142+0000 7f2a71800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c0d03b0 con 0x7f2a6c162270 2024-04-24T08:11:18.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.142+0000 7f2a71800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6c0d0e90 con 0x7f2a6c15edf0 2024-04-24T08:11:18.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.143+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 0x7f2a6c0cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.143+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 0x7f2a6c0cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55124/0 (socket says 172.21.15.69:55124) 2024-04-24T08:11:18.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.143+0000 7f2a6be00640 1 -- 172.21.15.69:0/1056310760 learned_addr learned my addr 172.21.15.69:0/1056310760 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.143+0000 7f2a70e00640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a6c162270 0x7f2a6c1f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.143+0000 7f2a6b400640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a6c15edf0 0x7f2a6c135bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.143+0000 7f2a6be00640 1 -- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a6c15edf0 msgr2=0x7f2a6c135bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.143+0000 7f2a6be00640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a6c15edf0 0x7f2a6c135bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.143+0000 7f2a6be00640 1 -- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a6c162270 msgr2=0x7f2a6c1f7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.143+0000 7f2a6be00640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a6c162270 0x7f2a6c1f7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.143+0000 7f2a6be00640 1 -- 172.21.15.69:0/1056310760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6c0adc00 con 0x7f2a6c071d70 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.144+0000 7f2a6be00640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 0x7f2a6c0cf6b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c000c00 tx=0x7f2a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.144+0000 7f2a63e00640 1 -- 172.21.15.69:0/1056310760 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c041070 con 0x7f2a6c071d70 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.144+0000 7f2a63e00640 1 -- 172.21.15.69:0/1056310760 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c03c870 con 0x7f2a6c071d70 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.144+0000 7f2a63e00640 1 -- 172.21.15.69:0/1056310760 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c037040 con 0x7f2a6c071d70 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.144+0000 7f2a71800640 1 -- 172.21.15.69:0/1056310760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a6c0cd130 con 0x7f2a6c071d70 2024-04-24T08:11:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.144+0000 7f2a71800640 1 -- 172.21.15.69:0/1056310760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a6c13ff30 con 0x7f2a6c071d70 2024-04-24T08:11:18.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.146+0000 7f2a63e00640 1 -- 172.21.15.69:0/1056310760 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a5c044070 con 0x7f2a6c071d70 2024-04-24T08:11:18.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.147+0000 7f2a63e00640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a4c06c8f0 0x7f2a4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.148+0000 7f2a63e00640 1 -- 172.21.15.69:0/1056310760 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f2a5c0ac820 con 0x7f2a6c071d70 2024-04-24T08:11:18.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.148+0000 7f2a6b400640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a4c06c8f0 0x7f2a4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.148+0000 7f2a75b98740 1 Processor -- start 2024-04-24T08:11:18.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.148+0000 7f2a75b98740 1 -- start start 2024-04-24T08:11:18.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.149+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d107e00 0x56529d0ba790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.149+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d0eaf20 0x56529d0ea2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.149+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0ea7e0 0x56529d0bf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.149+0000 7f2a75b98740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56529d0241f0 con 0x56529d107e00 2024-04-24T08:11:18.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.149+0000 7f2a75b98740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56529cfca350 con 0x56529d0ea7e0 2024-04-24T08:11:18.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.149+0000 7f2a75b98740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56529d0201b0 con 0x56529d0eaf20 2024-04-24T08:11:18.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.149+0000 7f2a6b400640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a4c06c8f0 0x7f2a4c06edb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c146fe0 tx=0x7f2a58008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:18.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d107e00 0x56529d0ba790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d107e00 0x56529d0ba790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55126/0 (socket says 172.21.15.69:55126) 2024-04-24T08:11:18.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a6be00640 1 -- 172.21.15.69:0/704272628 learned_addr learned my addr 172.21.15.69:0/704272628 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:18.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a70e00640 1 --2- 172.21.15.69:0/704272628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0ea7e0 0x56529d0bf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a6b400640 1 --2- 172.21.15.69:0/704272628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d0eaf20 0x56529d0ea2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a6be00640 1 -- 172.21.15.69:0/704272628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d0eaf20 msgr2=0x56529d0ea2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a6be00640 1 --2- 172.21.15.69:0/704272628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d0eaf20 0x56529d0ea2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a6be00640 1 -- 172.21.15.69:0/704272628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0ea7e0 msgr2=0x56529d0bf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a6be00640 1 --2- 172.21.15.69:0/704272628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0ea7e0 0x56529d0bf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a6be00640 1 -- 172.21.15.69:0/704272628 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56529d01ef50 con 0x56529d107e00 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.150+0000 7f2a6be00640 1 --2- 172.21.15.69:0/704272628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d107e00 0x56529d0ba790 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c082cd0 tx=0x7f2a5c07a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfee00ca4536ac19 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.151+0000 7f2a60c00640 1 -- 172.21.15.69:0/704272628 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c04b070 con 0x56529d107e00 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.151+0000 7f2a60c00640 1 -- 172.21.15.69:0/704272628 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c07e9d0 con 0x56529d107e00 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.151+0000 7f2a60c00640 1 -- 172.21.15.69:0/704272628 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c083200 con 0x56529d107e00 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.151+0000 7f2a75b98740 1 -- 172.21.15.69:0/704272628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d107e00 msgr2=0x56529d0ba790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.151+0000 7f2a75b98740 1 --2- 172.21.15.69:0/704272628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d107e00 0x56529d0ba790 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c082cd0 tx=0x7f2a5c07a040 comp rx=0 tx=0).stop 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.151+0000 7f2a75b98740 1 -- 172.21.15.69:0/704272628 shutdown_connections 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.151+0000 7f2a75b98740 1 --2- 172.21.15.69:0/704272628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0ea7e0 0x56529d0bf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.151+0000 7f2a75b98740 1 --2- 172.21.15.69:0/704272628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d0eaf20 0x56529d0ea2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.151+0000 7f2a75b98740 1 --2- 172.21.15.69:0/704272628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d107e00 0x56529d0ba790 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.151+0000 7f2a75b98740 1 -- 172.21.15.69:0/704272628 >> 172.21.15.69:0/704272628 conn(0x56529d0bca00 msgr2=0x56529d1052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.152+0000 7f2a75b98740 1 -- 172.21.15.69:0/704272628 shutdown_connections 2024-04-24T08:11:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.152+0000 7f2a75b98740 1 -- 172.21.15.69:0/704272628 wait complete. 2024-04-24T08:11:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.152+0000 7f2a75b98740 1 -- start start 2024-04-24T08:11:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.152+0000 7f2a75b98740 1 Processor -- start 2024-04-24T08:11:18.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.153+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d0e5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.153+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d107e00 0x56529d0eca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.153+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0fd5d0 0x56529d0fda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.153+0000 7f2a75b98740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56529d0241f0 con 0x56529d0eaf20 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a75b98740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56529cfdd690 con 0x56529d0fd5d0 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a75b98740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56529d0bebb0 con 0x56529d107e00 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a75b98740 10 client.0 _ll_register_callbacks cb 0x56529d019650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d0e5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d0e5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55130/0 (socket says 172.21.15.69:55130) 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a6be00640 1 -- 172.21.15.69:0/1872178606 learned_addr learned my addr 172.21.15.69:0/1872178606 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a6be00640 1 -- 172.21.15.69:0/1872178606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d107e00 msgr2=0x56529d0eca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a6be00640 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d107e00 0x56529d0eca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a6be00640 1 -- 172.21.15.69:0/1872178606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0fd5d0 msgr2=0x56529d0fda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a6be00640 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0fd5d0 0x56529d0fda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a6be00640 1 -- 172.21.15.69:0/1872178606 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56529cfdc920 con 0x56529d0eaf20 2024-04-24T08:11:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a6be00640 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d0e5a60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c07a5b0 tx=0x7f2a5c04b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237dd1b5a4af26ab server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.154+0000 7f2a55600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a55600640 1 -- 172.21.15.69:0/1872178606 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c03ce60 con 0x56529d0eaf20 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a55600640 1 -- 172.21.15.69:0/1872178606 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f2a5c082d70 con 0x56529d0eaf20 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a75b98740 1 -- 172.21.15.69:0/1872178606 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a5c07e9d0 con 0x56529d0eaf20 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a55600640 1 -- 172.21.15.69:0/1872178606 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2a5c083200 con 0x56529d0eaf20 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a75b98740 10 client.6021 fetch_fsmap learned FSMap version 25 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a75b98740 1 -- 172.21.15.69:0/1872178606 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x56529ce007d0 con 0x56529d0eaf20 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a55600640 1 -- 172.21.15.69:0/1872178606 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c07e9d0 con 0x56529d0eaf20 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a55600640 1 -- 172.21.15.69:0/1872178606 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f2a5c07fb60 con 0x56529d0eaf20 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a75b98740 10 client.6021 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a75b98740 10 client.6021 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a75b98740 20 client.6021 populate_metadata read hostname 'smithi069' 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a75b98740 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x56529d106b30 0x56529d1189d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a75b98740 4 client.6021 mds_command: new command op to 5793 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.155+0000 7f2a75b98740 1 -- 172.21.15.69:0/1872178606 --> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56529d031c10 con 0x56529d106b30 2024-04-24T08:11:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.156+0000 7f2a6b400640 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x56529d106b30 0x56529d1189d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.157+0000 7f2a6b400640 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x56529d106b30 0x56529d1189d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84fd5c50977c34ba server_cookie=e90089f1a0c23572 in_seq=0 out_seq=0 2024-04-24T08:11:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.157+0000 7f2a55600640 10 client.6021 ms_handle_connect on v2:172.21.15.121:6835/3424934990 2024-04-24T08:11:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a55600640 1 -- 172.21.15.69:0/1872178606 <== mds.0 v2:172.21.15.121:6835/3424934990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56529d031c10 con 0x56529d106b30 2024-04-24T08:11:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a55600640 10 client.6021 handle_command_reply: tid=0 2024-04-24T08:11:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 client.6021 shutdown 2024-04-24T08:11:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 -- 172.21.15.69:0/1872178606 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 msgr2=0x56529d0e5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d0e5a60 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c07a5b0 tx=0x7f2a5c04b000 comp rx=0 tx=0).stop 2024-04-24T08:11:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 -- 172.21.15.69:0/1872178606 shutdown_connections 2024-04-24T08:11:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x56529d106b30 0x56529d1189d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0fd5d0 0x56529d0fda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d107e00 0x56529d0eca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 --2- 172.21.15.69:0/1872178606 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d0e5a60 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a55600640 0 client.6021 ms_handle_reset on v2:172.21.15.121:6835/3424934990 2024-04-24T08:11:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 -- 172.21.15.69:0/1872178606 >> 172.21.15.69:0/1872178606 conn(0x56529d0ea7e0 msgr2=0x56529d0be1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 -- 172.21.15.69:0/1872178606 shutdown_connections 2024-04-24T08:11:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.159+0000 7f2a75b98740 1 -- 172.21.15.69:0/1872178606 wait complete. 2024-04-24T08:11:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.160+0000 7f2a75b98740 20 client.6021 trim_cache size 0 max 16384 2024-04-24T08:11:18.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.177+0000 7f2a75b98740 1 Processor -- start 2024-04-24T08:11:18.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.178+0000 7f2a75b98740 1 -- start start 2024-04-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.178+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d1292c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.178+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d0fd5d0 0x56529d0ede80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.178+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d107e00 0x56529d0ee3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.178+0000 7f2a75b98740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c200dc0 con 0x56529d0eaf20 2024-04-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.178+0000 7f2a75b98740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c0cee80 con 0x56529d107e00 2024-04-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.178+0000 7f2a75b98740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6c145e80 con 0x56529d0fd5d0 2024-04-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.179+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d1292c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.179+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d1292c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55142/0 (socket says 172.21.15.69:55142) 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.179+0000 7f2a6be00640 1 -- 172.21.15.69:0/1130568981 learned_addr learned my addr 172.21.15.69:0/1130568981 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.179+0000 7f2a70e00640 1 --2- 172.21.15.69:0/1130568981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d107e00 0x56529d0ee3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.179+0000 7f2a6b400640 1 --2- 172.21.15.69:0/1130568981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d0fd5d0 0x56529d0ede80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.179+0000 7f2a6be00640 1 -- 172.21.15.69:0/1130568981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d0fd5d0 msgr2=0x56529d0ede80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.179+0000 7f2a6be00640 1 --2- 172.21.15.69:0/1130568981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d0fd5d0 0x56529d0ede80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.179+0000 7f2a6be00640 1 -- 172.21.15.69:0/1130568981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d107e00 msgr2=0x56529d0ee3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.179+0000 7f2a6be00640 1 --2- 172.21.15.69:0/1130568981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d107e00 0x56529d0ee3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.179+0000 7f2a6be00640 1 -- 172.21.15.69:0/1130568981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56529cd363f0 con 0x56529d0eaf20 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.180+0000 7f2a6be00640 1 --2- 172.21.15.69:0/1130568981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d1292c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c082cd0 tx=0x7f2a5c0a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6edabfd1fb98aac server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.180+0000 7f2a57e00640 1 -- 172.21.15.69:0/1130568981 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c07e9d0 con 0x56529d0eaf20 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.180+0000 7f2a57e00640 1 -- 172.21.15.69:0/1130568981 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c089070 con 0x56529d0eaf20 2024-04-24T08:11:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.180+0000 7f2a57e00640 1 -- 172.21.15.69:0/1130568981 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c07fb60 con 0x56529d0eaf20 2024-04-24T08:11:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.180+0000 7f2a75b98740 1 -- 172.21.15.69:0/1130568981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 msgr2=0x56529d1292c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.180+0000 7f2a75b98740 1 --2- 172.21.15.69:0/1130568981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d1292c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c082cd0 tx=0x7f2a5c0a7000 comp rx=0 tx=0).stop 2024-04-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.181+0000 7f2a75b98740 1 -- 172.21.15.69:0/1130568981 shutdown_connections 2024-04-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.181+0000 7f2a75b98740 1 --2- 172.21.15.69:0/1130568981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d107e00 0x56529d0ee3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.181+0000 7f2a75b98740 1 --2- 172.21.15.69:0/1130568981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d0fd5d0 0x56529d0ede80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.181+0000 7f2a75b98740 1 --2- 172.21.15.69:0/1130568981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d1292c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.181+0000 7f2a75b98740 1 -- 172.21.15.69:0/1130568981 >> 172.21.15.69:0/1130568981 conn(0x56529d0ea7e0 msgr2=0x56529d0fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.181+0000 7f2a75b98740 1 -- 172.21.15.69:0/1130568981 shutdown_connections 2024-04-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.181+0000 7f2a75b98740 1 -- 172.21.15.69:0/1130568981 wait complete. 2024-04-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.181+0000 7f2a75b98740 1 -- start start 2024-04-24T08:11:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.182+0000 7f2a75b98740 1 Processor -- start 2024-04-24T08:11:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d17ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0fd5d0 0x56529d0bf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a75b98740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d107e00 0x56529d0bf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a75b98740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56529d107a70 con 0x56529d0eaf20 2024-04-24T08:11:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a75b98740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6c0ce830 con 0x56529d0fd5d0 2024-04-24T08:11:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a75b98740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56529cd363f0 con 0x56529d107e00 2024-04-24T08:11:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a75b98740 10 client.0 _ll_register_callbacks cb 0x7f2a5c07a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d17ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a6be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d17ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55152/0 (socket says 172.21.15.69:55152) 2024-04-24T08:11:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a6be00640 1 -- 172.21.15.69:0/2483614301 learned_addr learned my addr 172.21.15.69:0/2483614301 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a6be00640 1 -- 172.21.15.69:0/2483614301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d107e00 msgr2=0x56529d0bf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a6be00640 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d107e00 0x56529d0bf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a6be00640 1 -- 172.21.15.69:0/2483614301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0fd5d0 msgr2=0x56529d0bf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a6be00640 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0fd5d0 0x56529d0bf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a6be00640 1 -- 172.21.15.69:0/2483614301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56529ccf60f0 con 0x56529d0eaf20 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a6be00640 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d17ddd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c07a320 tx=0x7f2a5c076970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c2ba56d7a520b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.183+0000 7f2a55600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a55600640 1 -- 172.21.15.69:0/2483614301 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c09c060 con 0x56529d0eaf20 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a55600640 1 -- 172.21.15.69:0/2483614301 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f2a5c079c80 con 0x56529d0eaf20 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a75b98740 1 -- 172.21.15.69:0/2483614301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a5c089070 con 0x56529d0eaf20 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a55600640 1 -- 172.21.15.69:0/2483614301 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2a5c07e9d0 con 0x56529d0eaf20 2024-04-24T08:11:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a75b98740 10 client.6027 fetch_fsmap learned FSMap version 25 2024-04-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a75b98740 1 -- 172.21.15.69:0/2483614301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) -- 0x56529cfe9810 con 0x56529d0eaf20 2024-04-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a55600640 1 -- 172.21.15.69:0/2483614301 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c089070 con 0x56529d0eaf20 2024-04-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a55600640 1 -- 172.21.15.69:0/2483614301 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 3494+0+0 (secure 0 0 0) 0x7f2a5c076370 con 0x56529d0eaf20 2024-04-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a75b98740 10 client.6027 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a75b98740 10 client.6027 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a75b98740 20 client.6027 populate_metadata read hostname 'smithi069' 2024-04-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a75b98740 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x56529d17e310 0x56529d114720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a75b98740 4 client.6027 mds_command: new command op to 5793 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.184+0000 7f2a75b98740 1 -- 172.21.15.69:0/2483614301 --> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56529d02ba80 con 0x56529d17e310 2024-04-24T08:11:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.185+0000 7f2a6b400640 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x56529d17e310 0x56529d114720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.186+0000 7f2a6b400640 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x56529d17e310 0x56529d114720 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59548e41173dc404 server_cookie=8dc6a73e23036ee4 in_seq=0 out_seq=0 2024-04-24T08:11:18.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.186+0000 7f2a55600640 10 client.6027 ms_handle_connect on v2:172.21.15.121:6835/3424934990 2024-04-24T08:11:18.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a55600640 1 -- 172.21.15.69:0/2483614301 <== mds.0 v2:172.21.15.121:6835/3424934990 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56529d02ba80 con 0x56529d17e310 2024-04-24T08:11:18.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a55600640 10 client.6027 handle_command_reply: tid=0 2024-04-24T08:11:18.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 client.6027 shutdown 2024-04-24T08:11:18.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 -- 172.21.15.69:0/2483614301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 msgr2=0x56529d17ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d17ddd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c07a320 tx=0x7f2a5c076970 comp rx=0 tx=0).stop 2024-04-24T08:11:18.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 -- 172.21.15.69:0/2483614301 shutdown_connections 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.121:6835/3424934990,v1:172.21.15.121:6837/3424934990] conn(0x56529d17e310 0x56529d114720 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56529d107e00 0x56529d0bf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56529d0fd5d0 0x56529d0bf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 --2- 172.21.15.69:0/2483614301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56529d0eaf20 0x56529d17ddd0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 -- 172.21.15.69:0/2483614301 >> 172.21.15.69:0/2483614301 conn(0x56529d0ea7e0 msgr2=0x56529d0ed530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a55600640 0 client.6027 ms_handle_reset on v2:172.21.15.121:6835/3424934990 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 -- 172.21.15.69:0/2483614301 shutdown_connections 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 1 -- 172.21.15.69:0/2483614301 wait complete. 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.187+0000 7f2a75b98740 20 client.6027 trim_cache size 0 max 16384 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:11:18.192 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-04-24T08:11:18.193 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-04-24T08:11:18.193 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:11:18.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 -- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a4c06c8f0 msgr2=0x7f2a4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a4c06c8f0 0x7f2a4c06edb0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c146fe0 tx=0x7f2a58008040 comp rx=0 tx=0).stop 2024-04-24T08:11:18.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 -- 172.21.15.69:0/1056310760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 msgr2=0x7f2a6c0cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 0x7f2a6c0cf6b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c000c00 tx=0x7f2a5c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 -- 172.21.15.69:0/1056310760 shutdown_connections 2024-04-24T08:11:18.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a4c06c8f0 0x7f2a4c06edb0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a6c162270 0x7f2a6c1f7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a6c15edf0 0x7f2a6c135bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 --2- 172.21.15.69:0/1056310760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a6c071d70 0x7f2a6c0cf6b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 -- 172.21.15.69:0/1056310760 >> 172.21.15.69:0/1056310760 conn(0x7f2a6c067b80 msgr2=0x7f2a6c1430c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:18.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 -- 172.21.15.69:0/1056310760 shutdown_connections 2024-04-24T08:11:18.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.188+0000 7f2a71800640 1 -- 172.21.15.69:0/1056310760 wait complete. 2024-04-24T08:11:18.206 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:11:18.308 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.506+0000 7f39c7200640 1 Processor -- start 2024-04-24T08:11:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.506+0000 7f39c7200640 1 -- start start 2024-04-24T08:11:18.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39c7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39c80b67a0 0x7f39c80b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39c7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39c80d0580 con 0x7f39c806e110 2024-04-24T08:11:18.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39c7200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39c80d0860 con 0x7f39c8161f90 2024-04-24T08:11:18.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39c7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f39c8162370 con 0x7f39c80b67a0 2024-04-24T08:11:18.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39bd400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39c80b67a0 0x7f39c80b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39bd400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39c80b67a0 0x7f39c80b3a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55158/0 (socket says 172.21.15.69:55158) 2024-04-24T08:11:18.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39bd400640 1 -- 172.21.15.69:0/1818407057 learned_addr learned my addr 172.21.15.69:0/1818407057 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:18.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39bd400640 1 -- 172.21.15.69:0/1818407057 >> v1:172.21.15.121:6789/0 conn(0x7f39c8161f90 legacy=0x7f39c80b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:18.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39bd400640 1 -- 172.21.15.69:0/1818407057 >> v1:172.21.15.69:6789/0 conn(0x7f39c806e110 legacy=0x7f39c80ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39bd400640 1 -- 172.21.15.69:0/1818407057 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c80d0b40 con 0x7f39c80b67a0 2024-04-24T08:11:18.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.508+0000 7f39c5400640 1 -- 172.21.15.69:0/1818407057 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3165653138 0 0) 0x7f39c80d0580 con 0x7f39c806e110 2024-04-24T08:11:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.509+0000 7f39bd400640 1 --2- 172.21.15.69:0/1818407057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c80b67a0 0x7f39c80b3a40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009850 tx=0x7f39b002a320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a082c29d4f654a37 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.510+0000 7f39c5400640 1 -- 172.21.15.69:0/1818407057 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0004230 con 0x7f39c80b67a0 2024-04-24T08:11:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.510+0000 7f39c5400640 1 -- 172.21.15.69:0/1818407057 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b00043d0 con 0x7f39c80b67a0 2024-04-24T08:11:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.510+0000 7f39c5400640 1 -- 172.21.15.69:0/1818407057 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0003810 con 0x7f39c80b67a0 2024-04-24T08:11:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.510+0000 7f39c7200640 1 -- 172.21.15.69:0/1818407057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c80b67a0 msgr2=0x7f39c80b3a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.510+0000 7f39c7200640 1 --2- 172.21.15.69:0/1818407057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c80b67a0 0x7f39c80b3a40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009850 tx=0x7f39b002a320 comp rx=0 tx=0).stop 2024-04-24T08:11:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.510+0000 7f39c7200640 1 -- 172.21.15.69:0/1818407057 shutdown_connections 2024-04-24T08:11:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.510+0000 7f39c7200640 1 --2- 172.21.15.69:0/1818407057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c80b67a0 0x7f39c80b3a40 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.510+0000 7f39c7200640 1 -- 172.21.15.69:0/1818407057 >> 172.21.15.69:0/1818407057 conn(0x7f39c808bf60 msgr2=0x7f39c80d4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.510+0000 7f39c7200640 1 -- 172.21.15.69:0/1818407057 shutdown_connections 2024-04-24T08:11:18.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.511+0000 7f39c7200640 1 -- 172.21.15.69:0/1818407057 wait complete. 2024-04-24T08:11:18.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.512+0000 7f39c7200640 1 Processor -- start 2024-04-24T08:11:18.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.512+0000 7f39c7200640 1 -- start start 2024-04-24T08:11:18.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.513+0000 7f39c7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c806e110 0x7f39c815a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.513+0000 7f39c7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39c80b67a0 0x7f39c815b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.513+0000 7f39c7200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39c8161f90 0x7f39c80b5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.513+0000 7f39c7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39c80d53d0 con 0x7f39c806e110 2024-04-24T08:11:18.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.513+0000 7f39c7200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f39c806cbe0 con 0x7f39c8161f90 2024-04-24T08:11:18.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.513+0000 7f39c7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f39c80cf5c0 con 0x7f39c80b67a0 2024-04-24T08:11:18.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.513+0000 7f39c5e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c806e110 0x7f39c815a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39c5e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c806e110 0x7f39c815a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55162/0 (socket says 172.21.15.69:55162) 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39c5e00640 1 -- 172.21.15.69:0/1305095916 learned_addr learned my addr 172.21.15.69:0/1305095916 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39bd400640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39c80b67a0 0x7f39c815b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39c5e00640 1 -- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39c80b67a0 msgr2=0x7f39c815b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39c5e00640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39c80b67a0 0x7f39c815b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39c5e00640 1 -- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39c8161f90 msgr2=0x7f39c80b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39c6800640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39c8161f90 0x7f39c80b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39c5e00640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39c8161f90 0x7f39c80b5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39c5e00640 1 -- 172.21.15.69:0/1305095916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c80b9130 con 0x7f39c806e110 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39c6800640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39c8161f90 0x7f39c80b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.514+0000 7f39c5e00640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c806e110 0x7f39c815a130 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f39b800b410 tx=0x7f39b800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.515+0000 7f39bea00640 1 -- 172.21.15.69:0/1305095916 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8015020 con 0x7f39c806e110 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.515+0000 7f39bea00640 1 -- 172.21.15.69:0/1305095916 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b8011830 con 0x7f39c806e110 2024-04-24T08:11:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.515+0000 7f39c7200640 1 -- 172.21.15.69:0/1305095916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c80cb880 con 0x7f39c806e110 2024-04-24T08:11:18.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.515+0000 7f39bea00640 1 -- 172.21.15.69:0/1305095916 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b8011c20 con 0x7f39c806e110 2024-04-24T08:11:18.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.515+0000 7f39c7200640 1 -- 172.21.15.69:0/1305095916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c80cbe40 con 0x7f39c806e110 2024-04-24T08:11:18.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.516+0000 7f39c7200640 1 -- 172.21.15.69:0/1305095916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c8075d50 con 0x7f39c806e110 2024-04-24T08:11:18.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.517+0000 7f39bea00640 1 -- 172.21.15.69:0/1305095916 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f39b8018020 con 0x7f39c806e110 2024-04-24T08:11:18.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.518+0000 7f39bea00640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a406c8f0 0x7f39a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:18.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.518+0000 7f39bea00640 1 -- 172.21.15.69:0/1305095916 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f39b808bbb0 con 0x7f39c806e110 2024-04-24T08:11:18.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.519+0000 7f39bd400640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a406c8f0 0x7f39a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:18.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.520+0000 7f39bd400640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a406c8f0 0x7f39a406edb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f39b0033000 tx=0x7f39b002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:18.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.523+0000 7f39bea00640 1 -- 172.21.15.69:0/1305095916 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f39b801f020 con 0x7f39c806e110 2024-04-24T08:11:18.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.767+0000 7f39c7200640 1 -- 172.21.15.69:0/1305095916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39c815ffe0 con 0x7f39c806e110 2024-04-24T08:11:18.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.768+0000 7f39bea00640 1 -- 172.21.15.69:0/1305095916 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5184 (secure 0 0 0) 0x7f39b80172c0 con 0x7f39c806e110 2024-04-24T08:11:18.770 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:18.770 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/376722301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":376722301},{"type":"v1","addr":"172.21.15.69:6835","nonce":376722301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4990,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1327624292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1327624292},{"type":"v1","addr":"172.21.15.121:6836","nonce":1327624292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/3244002416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3244002416},{"type":"v1","addr":"172.21.15.69:6834","nonce":3244002416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:10:35.400788+0000","modified":"2024-04-24T08:10:38.457641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5793},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5793":{"gid":5793,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/3424934990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3424934990},{"type":"v1","addr":"172.21.15.121:6837","nonce":3424934990}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5793,"qdb_cluster":[5793]},"id":3}]} 2024-04-24T08:11:18.770 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-04-24T08:11:18.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.771+0000 7f39c7200640 1 -- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a406c8f0 msgr2=0x7f39a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.771+0000 7f39c7200640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a406c8f0 0x7f39a406edb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f39b0033000 tx=0x7f39b002e000 comp rx=0 tx=0).stop 2024-04-24T08:11:18.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.771+0000 7f39c7200640 1 -- 172.21.15.69:0/1305095916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c806e110 msgr2=0x7f39c815a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:18.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.771+0000 7f39c7200640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c806e110 0x7f39c815a130 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f39b800b410 tx=0x7f39b800b8d0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.771+0000 7f39c7200640 1 -- 172.21.15.69:0/1305095916 shutdown_connections 2024-04-24T08:11:18.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.771+0000 7f39c7200640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a406c8f0 0x7f39a406edb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.771+0000 7f39c7200640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39c8161f90 0x7f39c80b5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.771+0000 7f39c7200640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39c80b67a0 0x7f39c815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.771+0000 7f39c7200640 1 --2- 172.21.15.69:0/1305095916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39c806e110 0x7f39c815a130 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.771+0000 7f39c7200640 1 -- 172.21.15.69:0/1305095916 >> 172.21.15.69:0/1305095916 conn(0x7f39c808bf60 msgr2=0x7f39c80d49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.772+0000 7f39c7200640 1 -- 172.21.15.69:0/1305095916 shutdown_connections 2024-04-24T08:11:18.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:18.772+0000 7f39c7200640 1 -- 172.21.15.69:0/1305095916 wait complete. 2024-04-24T08:11:18.794 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-04-24T08:11:19.038 INFO:teuthology.orchestra.run.smithi121.stdout:{"message":"","return_code":0} 2024-04-24T08:11:19.039 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-24T08:11:19.039 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-04-24T08:11:19.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.136+0000 7fb687218f40 1 Processor -- start 2024-04-24T08:11:19.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.136+0000 7fb687218f40 1 -- start start 2024-04-24T08:11:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.137+0000 7fb687218f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c04ec28770 0x55c04ecf7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.137+0000 7fb687218f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c04ec73940 0x55c04ec26f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.137+0000 7fb687218f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04ecf33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.137+0000 7fb687218f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c04ec59180 con 0x55c04ecf9f50 2024-04-24T08:11:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.137+0000 7fb687218f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c04ebcfcc0 con 0x55c04ec73940 2024-04-24T08:11:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.137+0000 7fb687218f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c04ed00d60 con 0x55c04ec28770 2024-04-24T08:11:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.138+0000 7fb684200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04ecf33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.138+0000 7fb684200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04ecf33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44866/0 (socket says 172.21.15.69:44866) 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.138+0000 7fb684200640 1 -- 172.21.15.69:0/356395994 learned_addr learned my addr 172.21.15.69:0/356395994 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.138+0000 7fb683800640 1 --2- 172.21.15.69:0/356395994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c04ec28770 0x55c04ecf7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.138+0000 7fb682e00640 1 --2- 172.21.15.69:0/356395994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c04ec73940 0x55c04ec26f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.138+0000 7fb684200640 1 -- 172.21.15.69:0/356395994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c04ec28770 msgr2=0x55c04ecf7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.138+0000 7fb684200640 1 --2- 172.21.15.69:0/356395994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c04ec28770 0x55c04ecf7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.138+0000 7fb684200640 1 -- 172.21.15.69:0/356395994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c04ec73940 msgr2=0x55c04ec26f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.138+0000 7fb684200640 1 --2- 172.21.15.69:0/356395994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c04ec73940 0x55c04ec26f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.138+0000 7fb684200640 1 -- 172.21.15.69:0/356395994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c04ecef4b0 con 0x55c04ecf9f50 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb684200640 1 --2- 172.21.15.69:0/356395994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04ecf33b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00b610 tx=0x7fb67c00bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17e5f5af956eb27 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb681a00640 1 -- 172.21.15.69:0/356395994 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67c014020 con 0x55c04ecf9f50 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb681a00640 1 -- 172.21.15.69:0/356395994 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb67c010840 con 0x55c04ecf9f50 2024-04-24T08:11:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb681a00640 1 -- 172.21.15.69:0/356395994 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67c010c30 con 0x55c04ecf9f50 2024-04-24T08:11:19.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb687218f40 1 -- 172.21.15.69:0/356395994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 msgr2=0x55c04ecf33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb687218f40 1 --2- 172.21.15.69:0/356395994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04ecf33b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00b610 tx=0x7fb67c00bad0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb687218f40 1 -- 172.21.15.69:0/356395994 shutdown_connections 2024-04-24T08:11:19.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb687218f40 1 --2- 172.21.15.69:0/356395994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04ecf33b0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb687218f40 1 --2- 172.21.15.69:0/356395994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c04ec73940 0x55c04ec26f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb687218f40 1 --2- 172.21.15.69:0/356395994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c04ec28770 0x55c04ecf7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.139+0000 7fb687218f40 1 -- 172.21.15.69:0/356395994 >> 172.21.15.69:0/356395994 conn(0x55c04ec57b50 msgr2=0x55c04ec57360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:19.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.140+0000 7fb687218f40 1 -- 172.21.15.69:0/356395994 shutdown_connections 2024-04-24T08:11:19.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.140+0000 7fb687218f40 1 -- 172.21.15.69:0/356395994 wait complete. 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.140+0000 7fb687218f40 1 Processor -- start 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.141+0000 7fb687218f40 1 -- start start 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.141+0000 7fb687218f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c04ec28770 0x55c04ecf71e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb687218f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c04ec73940 0x55c04ecf7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb687218f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04eceed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb687218f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c04ec280e0 con 0x55c04ecf9f50 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb687218f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c04ecf85a0 con 0x55c04ec73940 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb687218f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c04ed02060 con 0x55c04ec28770 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb684200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04eceed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb684200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04eceed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44870/0 (socket says 172.21.15.69:44870) 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb684200640 1 -- 172.21.15.69:0/4190901555 learned_addr learned my addr 172.21.15.69:0/4190901555 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb682e00640 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c04ec73940 0x55c04ecf7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb684200640 1 -- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c04ec28770 msgr2=0x55c04ecf71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb684200640 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c04ec28770 0x55c04ecf71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb684200640 1 -- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c04ec73940 msgr2=0x55c04ecf7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb684200640 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c04ec73940 0x55c04ecf7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.142+0000 7fb684200640 1 -- 172.21.15.69:0/4190901555 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c04ecf27a0 con 0x55c04ecf9f50 2024-04-24T08:11:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.143+0000 7fb684200640 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04eceed10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00b1e0 tx=0x7fb67c0033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.144+0000 7fb67be00640 1 -- 172.21.15.69:0/4190901555 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67c01bc40 con 0x55c04ecf9f50 2024-04-24T08:11:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.144+0000 7fb67be00640 1 -- 172.21.15.69:0/4190901555 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb67c01bde0 con 0x55c04ecf9f50 2024-04-24T08:11:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.144+0000 7fb67be00640 1 -- 172.21.15.69:0/4190901555 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67c019690 con 0x55c04ecf9f50 2024-04-24T08:11:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.144+0000 7fb687218f40 1 -- 172.21.15.69:0/4190901555 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c04ecf2a80 con 0x55c04ecf9f50 2024-04-24T08:11:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.144+0000 7fb687218f40 1 -- 172.21.15.69:0/4190901555 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c04ecef250 con 0x55c04ecf9f50 2024-04-24T08:11:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.146+0000 7fb67be00640 1 -- 172.21.15.69:0/4190901555 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb67c017020 con 0x55c04ecf9f50 2024-04-24T08:11:19.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.147+0000 7fb67be00640 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb66406b300 0x7fb66406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.147+0000 7fb67be00640 1 -- 172.21.15.69:0/4190901555 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fb67c08c8a0 con 0x55c04ecf9f50 2024-04-24T08:11:19.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.148+0000 7fb687218f40 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55c04ed966a0 0x55c04ed96aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.148+0000 7fb687218f40 1 -- 172.21.15.69:0/4190901555 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:1 7.7 7:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c04ed96fe0 con 0x55c04ed966a0 2024-04-24T08:11:19.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.148+0000 7fb683800640 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb66406b300 0x7fb66406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.148+0000 7fb682e00640 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55c04ed966a0 0x55c04ed96aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.150+0000 7fb682e00640 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55c04ed966a0 0x55c04ed96aa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:19.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.150+0000 7fb683800640 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb66406b300 0x7fb66406d7c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb674057000 tx=0x7fb674051000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:19.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.150+0000 7fb682e00640 1 -- 172.21.15.69:0/4190901555 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv3 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x7fb67000b6f0 con 0x55c04ed966a0 2024-04-24T08:11:19.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.150+0000 7fb687218f40 1 -- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55c04ed966a0 msgr2=0x55c04ed96aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.150+0000 7fb687218f40 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55c04ed966a0 0x55c04ed96aa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.151+0000 7fb687218f40 1 -- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb66406b300 msgr2=0x7fb66406d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.151+0000 7fb687218f40 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb66406b300 0x7fb66406d7c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb674057000 tx=0x7fb674051000 comp rx=0 tx=0).stop 2024-04-24T08:11:19.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.151+0000 7fb687218f40 1 -- 172.21.15.69:0/4190901555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 msgr2=0x55c04eceed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.151+0000 7fb687218f40 1 --2- 172.21.15.69:0/4190901555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c04ecf9f50 0x55c04eceed10 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00b1e0 tx=0x7fb67c0033a0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.151+0000 7fb684200640 1 -- 172.21.15.69:0/4190901555 reap_dead start 2024-04-24T08:11:19.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.151+0000 7fb687218f40 1 -- 172.21.15.69:0/4190901555 shutdown_connections 2024-04-24T08:11:19.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.151+0000 7fb687218f40 1 -- 172.21.15.69:0/4190901555 >> 172.21.15.69:0/4190901555 conn(0x55c04ec57b50 msgr2=0x55c04ed028d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:19.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.151+0000 7fb687218f40 1 -- 172.21.15.69:0/4190901555 shutdown_connections 2024-04-24T08:11:19.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.151+0000 7fb687218f40 1 -- 172.21.15.69:0/4190901555 wait complete. 2024-04-24T08:11:19.157 INFO:teuthology.orchestra.run.smithi069.stdout:Yalphaparent_a 2024-04-24T08:11:19.157 DEBUG:teuthology.orchestra.run.smithi069:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-04-24T08:11:19.245 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:11:19.245 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511627777, 2024-04-24T08:11:19.245 INFO:teuthology.orchestra.run.smithi069.stdout: "ancestors": [ 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: "dirino": 1099511627776, 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: "dname": "alpha", 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: "version": 4 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: "dirino": 1, 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: "dname": "parent_a", 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: "version": 7 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 7, 2024-04-24T08:11:19.246 INFO:teuthology.orchestra.run.smithi069.stdout: "old_pools": [] 2024-04-24T08:11:19.247 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:11:19.247 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:19.253 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:11:19.253 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:11:19.363 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:11:19.467 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:11:19.467 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:11:19.467 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:11:19.467 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:11:19.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:11:19.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.719+0000 7fd267400640 1 Processor -- start 2024-04-24T08:11:19.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.719+0000 7fd267400640 1 -- start start 2024-04-24T08:11:19.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.721+0000 7fd267400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd260145a90 0x7fd26006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.721+0000 7fd267400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd260145fe0 0x7fd260160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.721+0000 7fd267400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2600cf3d0 con 0x7fd2600d0300 2024-04-24T08:11:19.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.721+0000 7fd267400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd2600cf6b0 con 0x7fd260145a90 2024-04-24T08:11:19.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.721+0000 7fd267400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2600cf990 con 0x7fd260145fe0 2024-04-24T08:11:19.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.721+0000 7fd25fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd260145a90 0x7fd26006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.721+0000 7fd25fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd260145a90 0x7fd26006e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44880/0 (socket says 172.21.15.69:44880) 2024-04-24T08:11:19.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.721+0000 7fd25fe00640 1 -- 172.21.15.69:0/511510908 learned_addr learned my addr 172.21.15.69:0/511510908 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:19.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.722+0000 7fd25fe00640 1 -- 172.21.15.69:0/511510908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd260145fe0 msgr2=0x7fd260160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:19.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.722+0000 7fd265400640 1 --2- 172.21.15.69:0/511510908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd260145fe0 0x7fd260160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.722+0000 7fd25fe00640 1 --2- 172.21.15.69:0/511510908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd260145fe0 0x7fd260160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.722+0000 7fd25fe00640 1 -- 172.21.15.69:0/511510908 >> v1:172.21.15.121:6789/0 conn(0x7fd2600d0300 legacy=0x7fd2600d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:19.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.722+0000 7fd25fe00640 1 -- 172.21.15.69:0/511510908 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2600cfc70 con 0x7fd260145a90 2024-04-24T08:11:19.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.723+0000 7fd25fe00640 1 --2- 172.21.15.69:0/511510908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 0x7fd26006e240 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd248009e50 tx=0x7fd24802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51ae2a083bdbda57 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:19.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.723+0000 7fd25f400640 1 -- 172.21.15.69:0/511510908 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd248036070 con 0x7fd260145a90 2024-04-24T08:11:19.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.723+0000 7fd25f400640 1 -- 172.21.15.69:0/511510908 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd248031600 con 0x7fd260145a90 2024-04-24T08:11:19.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.723+0000 7fd25f400640 1 -- 172.21.15.69:0/511510908 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2480319d0 con 0x7fd260145a90 2024-04-24T08:11:19.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.724+0000 7fd267400640 1 -- 172.21.15.69:0/511510908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 msgr2=0x7fd26006e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.724+0000 7fd267400640 1 --2- 172.21.15.69:0/511510908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 0x7fd26006e240 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd248009e50 tx=0x7fd24802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.724+0000 7fd267400640 1 -- 172.21.15.69:0/511510908 shutdown_connections 2024-04-24T08:11:19.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.724+0000 7fd267400640 1 --2- 172.21.15.69:0/511510908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd260145fe0 0x7fd260160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.724+0000 7fd267400640 1 --2- 172.21.15.69:0/511510908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 0x7fd26006e240 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.724+0000 7fd267400640 1 -- 172.21.15.69:0/511510908 >> 172.21.15.69:0/511510908 conn(0x7fd260067d90 msgr2=0x7fd26006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:19.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.724+0000 7fd267400640 1 -- 172.21.15.69:0/511510908 shutdown_connections 2024-04-24T08:11:19.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.724+0000 7fd267400640 1 -- 172.21.15.69:0/511510908 wait complete. 2024-04-24T08:11:19.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.726+0000 7fd267400640 1 Processor -- start 2024-04-24T08:11:19.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.726+0000 7fd267400640 1 -- start start 2024-04-24T08:11:19.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd267400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2600d0300 0x7fd2601f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd267400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 0x7fd26013c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd267400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd260145fe0 0x7fd26013cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd267400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2601458f0 con 0x7fd260145a90 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd267400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd260145770 con 0x7fd260145fe0 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd267400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2600cdc30 con 0x7fd2600d0300 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd25fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 0x7fd26013c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd25fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 0x7fd26013c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44892/0 (socket says 172.21.15.69:44892) 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd25fe00640 1 -- 172.21.15.69:0/3532967298 learned_addr learned my addr 172.21.15.69:0/3532967298 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd264a00640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2600d0300 0x7fd2601f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd25fe00640 1 -- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2600d0300 msgr2=0x7fd2601f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd25fe00640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2600d0300 0x7fd2601f7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd265400640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd260145fe0 0x7fd26013cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd25fe00640 1 -- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd260145fe0 msgr2=0x7fd26013cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd25fe00640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd260145fe0 0x7fd26013cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd25fe00640 1 -- 172.21.15.69:0/3532967298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2600aee60 con 0x7fd260145a90 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.727+0000 7fd265400640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd260145fe0 0x7fd26013cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.728+0000 7fd25fe00640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 0x7fd26013c600 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd248009e50 tx=0x7fd248004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.728+0000 7fd25cc00640 1 -- 172.21.15.69:0/3532967298 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd248036050 con 0x7fd260145a90 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.728+0000 7fd25cc00640 1 -- 172.21.15.69:0/3532967298 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd248042070 con 0x7fd260145a90 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.728+0000 7fd25cc00640 1 -- 172.21.15.69:0/3532967298 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd248009510 con 0x7fd260145a90 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.728+0000 7fd267400640 1 -- 172.21.15.69:0/3532967298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2600af140 con 0x7fd260145a90 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.728+0000 7fd267400640 1 -- 172.21.15.69:0/3532967298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2600b3de0 con 0x7fd260145a90 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.729+0000 7fd267400640 1 -- 172.21.15.69:0/3532967298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd260068a80 con 0x7fd260145a90 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.730+0000 7fd25cc00640 1 -- 172.21.15.69:0/3532967298 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd248007c50 con 0x7fd260145a90 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.731+0000 7fd25cc00640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd23c06c8f0 0x7fd23c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.731+0000 7fd25cc00640 1 -- 172.21.15.69:0/3532967298 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fd2480acc10 con 0x7fd260145a90 2024-04-24T08:11:19.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.732+0000 7fd264a00640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd23c06c8f0 0x7fd23c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:19.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.733+0000 7fd264a00640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd23c06c8f0 0x7fd23c06edb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd250003d90 tx=0x7fd250009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:19.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.736+0000 7fd25cc00640 1 -- 172.21.15.69:0/3532967298 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd248045360 con 0x7fd260145a90 2024-04-24T08:11:19.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.933+0000 7fd267400640 1 -- 172.21.15.69:0/3532967298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2600d1120 con 0x7fd260145a90 2024-04-24T08:11:19.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.935+0000 7fd25cc00640 1 -- 172.21.15.69:0/3532967298 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+16740 (secure 0 0 0) 0x7fd24807af60 con 0x7fd260145a90 2024-04-24T08:11:19.937 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:19.937 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":46,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:03.085538+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:10:33.100132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-24T08:10:34.379302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":5}]},{"pgid":"7.36","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:11:19.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 -- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd23c06c8f0 msgr2=0x7fd23c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd23c06c8f0 0x7fd23c06edb0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fd250003d90 tx=0x7fd250009040 comp rx=0 tx=0).stop 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 -- 172.21.15.69:0/3532967298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 msgr2=0x7fd26013c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 0x7fd26013c600 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd248009e50 tx=0x7fd248004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 -- 172.21.15.69:0/3532967298 shutdown_connections 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd23c06c8f0 0x7fd23c06edb0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd260145a90 0x7fd26013c600 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd260145fe0 0x7fd26013cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 --2- 172.21.15.69:0/3532967298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2600d0300 0x7fd2601f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 -- 172.21.15.69:0/3532967298 >> 172.21.15.69:0/3532967298 conn(0x7fd260067d90 msgr2=0x7fd26006f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 -- 172.21.15.69:0/3532967298 shutdown_connections 2024-04-24T08:11:19.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:19.938+0000 7fd267400640 1 -- 172.21.15.69:0/3532967298 wait complete. 2024-04-24T08:11:19.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:10:33.100132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:10:34.379302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:11:19.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:11:20.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.160+0000 7fbe4b200640 1 Processor -- start 2024-04-24T08:11:20.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.160+0000 7fbe4b200640 1 -- start start 2024-04-24T08:11:20.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.161+0000 7fbe4b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbe4c15ea70 0x7fbe4c0cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.161+0000 7fbe4b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe4c06c130 0x7fbe4c073b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.161+0000 7fbe4b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe4c15ee50 con 0x7fbe4c06ef40 2024-04-24T08:11:20.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.161+0000 7fbe4b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbe4c0cdcb0 con 0x7fbe4c15ea70 2024-04-24T08:11:20.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.161+0000 7fbe4b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbe4c06f4b0 con 0x7fbe4c06c130 2024-04-24T08:11:20.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.161+0000 7fbe49e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbe4c15ea70 0x7fbe4c0cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:20.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.161+0000 7fbe49e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbe4c15ea70 0x7fbe4c0cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44894/0 (socket says 172.21.15.69:44894) 2024-04-24T08:11:20.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.161+0000 7fbe49e00640 1 -- 172.21.15.69:0/1669316258 learned_addr learned my addr 172.21.15.69:0/1669316258 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:20.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.162+0000 7fbe4a800640 1 --2- 172.21.15.69:0/1669316258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe4c06c130 0x7fbe4c073b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:20.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.162+0000 7fbe49e00640 1 -- 172.21.15.69:0/1669316258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe4c06c130 msgr2=0x7fbe4c073b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:20.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.162+0000 7fbe49e00640 1 --2- 172.21.15.69:0/1669316258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe4c06c130 0x7fbe4c073b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.162+0000 7fbe49e00640 1 -- 172.21.15.69:0/1669316258 >> v1:172.21.15.121:6789/0 conn(0x7fbe4c06ef40 legacy=0x7fbe4c06ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:20.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.162+0000 7fbe49e00640 1 -- 172.21.15.69:0/1669316258 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4c06f790 con 0x7fbe4c15ea70 2024-04-24T08:11:20.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.163+0000 7fbe49e00640 1 --2- 172.21.15.69:0/1669316258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 0x7fbe4c0cf7e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c009870 tx=0x7fbe3c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d6283e9c7ff497 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:20.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.163+0000 7fbe48a00640 1 -- 172.21.15.69:0/1669316258 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c037070 con 0x7fbe4c15ea70 2024-04-24T08:11:20.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.163+0000 7fbe48a00640 1 -- 172.21.15.69:0/1669316258 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe3c032440 con 0x7fbe4c15ea70 2024-04-24T08:11:20.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.163+0000 7fbe48a00640 1 -- 172.21.15.69:0/1669316258 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c032810 con 0x7fbe4c15ea70 2024-04-24T08:11:20.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.164+0000 7fbe4b200640 1 -- 172.21.15.69:0/1669316258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 msgr2=0x7fbe4c0cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:20.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.164+0000 7fbe4b200640 1 --2- 172.21.15.69:0/1669316258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 0x7fbe4c0cf7e0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c009870 tx=0x7fbe3c02a1c0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.164+0000 7fbe4b200640 1 -- 172.21.15.69:0/1669316258 shutdown_connections 2024-04-24T08:11:20.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.164+0000 7fbe4b200640 1 --2- 172.21.15.69:0/1669316258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe4c06c130 0x7fbe4c073b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.164+0000 7fbe4b200640 1 --2- 172.21.15.69:0/1669316258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 0x7fbe4c0cf7e0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.164+0000 7fbe4b200640 1 -- 172.21.15.69:0/1669316258 >> 172.21.15.69:0/1669316258 conn(0x7fbe4c06cb30 msgr2=0x7fbe4c06cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:20.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.164+0000 7fbe4b200640 1 -- 172.21.15.69:0/1669316258 shutdown_connections 2024-04-24T08:11:20.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.164+0000 7fbe4b200640 1 -- 172.21.15.69:0/1669316258 wait complete. 2024-04-24T08:11:20.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.166+0000 7fbe4b200640 1 Processor -- start 2024-04-24T08:11:20.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.166+0000 7fbe4b200640 1 -- start start 2024-04-24T08:11:20.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe4c06c130 0x7fbe4c0caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe4c06ef40 0x7fbe4c1f4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 0x7fbe4c0b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4c0b0290 con 0x7fbe4c15ea70 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4c06fd50 con 0x7fbe4c06c130 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbe4c0cfd20 con 0x7fbe4c06ef40 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 0x7fbe4c0b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 0x7fbe4c0b4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44908/0 (socket says 172.21.15.69:44908) 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4a800640 1 -- 172.21.15.69:0/988457885 learned_addr learned my addr 172.21.15.69:0/988457885 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4a800640 1 -- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe4c06ef40 msgr2=0x7fbe4c1f4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4a800640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe4c06ef40 0x7fbe4c1f4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4a800640 1 -- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe4c06c130 msgr2=0x7fbe4c0caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4a800640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe4c06c130 0x7fbe4c0caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.167+0000 7fbe4a800640 1 -- 172.21.15.69:0/988457885 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4c15d140 con 0x7fbe4c15ea70 2024-04-24T08:11:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.168+0000 7fbe4a800640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 0x7fbe4c0b4ee0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fbe44007de0 tx=0x7fbe4400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:20.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.168+0000 7fbe42000640 1 -- 172.21.15.69:0/988457885 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe44014620 con 0x7fbe4c15ea70 2024-04-24T08:11:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.168+0000 7fbe42000640 1 -- 172.21.15.69:0/988457885 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe44019070 con 0x7fbe4c15ea70 2024-04-24T08:11:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.168+0000 7fbe42000640 1 -- 172.21.15.69:0/988457885 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe44014a20 con 0x7fbe4c15ea70 2024-04-24T08:11:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.168+0000 7fbe4b200640 1 -- 172.21.15.69:0/988457885 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe4c15bd20 con 0x7fbe4c15ea70 2024-04-24T08:11:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.168+0000 7fbe4b200640 1 -- 172.21.15.69:0/988457885 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe4c15c2e0 con 0x7fbe4c15ea70 2024-04-24T08:11:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.169+0000 7fbe4b200640 1 -- 172.21.15.69:0/988457885 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe4c075d50 con 0x7fbe4c15ea70 2024-04-24T08:11:20.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.170+0000 7fbe42000640 1 -- 172.21.15.69:0/988457885 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe44014bc0 con 0x7fbe4c15ea70 2024-04-24T08:11:20.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.171+0000 7fbe42000640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe2806c920 0x7fbe2806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.171+0000 7fbe42000640 1 -- 172.21.15.69:0/988457885 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fbe4408c4e0 con 0x7fbe4c15ea70 2024-04-24T08:11:20.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.171+0000 7fbe49e00640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe2806c920 0x7fbe2806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:20.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.173+0000 7fbe49e00640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe2806c920 0x7fbe2806ede0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c034000 tx=0x7fbe3c02c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:20.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.175+0000 7fbe42000640 1 -- 172.21.15.69:0/988457885 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe4405a8d0 con 0x7fbe4c15ea70 2024-04-24T08:11:20.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.376+0000 7fbe4b200640 1 -- 172.21.15.69:0/988457885 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbe4c0737d0 con 0x7fbe4c15ea70 2024-04-24T08:11:20.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.377+0000 7fbe42000640 1 -- 172.21.15.69:0/988457885 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) ==== 93+0+10 (secure 0 0 0) 0x7fbe4405a1d0 con 0x7fbe4c15ea70 2024-04-24T08:11:20.378 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:11:20.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 -- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe2806c920 msgr2=0x7fbe2806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:20.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe2806c920 0x7fbe2806ede0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c034000 tx=0x7fbe3c02c000 comp rx=0 tx=0).stop 2024-04-24T08:11:20.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 -- 172.21.15.69:0/988457885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 msgr2=0x7fbe4c0b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:20.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 0x7fbe4c0b4ee0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fbe44007de0 tx=0x7fbe4400e4b0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 -- 172.21.15.69:0/988457885 shutdown_connections 2024-04-24T08:11:20.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe2806c920 0x7fbe2806ede0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe4c15ea70 0x7fbe4c0b4ee0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe4c06ef40 0x7fbe4c1f4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 --2- 172.21.15.69:0/988457885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe4c06c130 0x7fbe4c0caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 -- 172.21.15.69:0/988457885 >> 172.21.15.69:0/988457885 conn(0x7fbe4c06cb30 msgr2=0x7fbe4c0ce140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:20.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 -- 172.21.15.69:0/988457885 shutdown_connections 2024-04-24T08:11:20.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.380+0000 7fbe4b200640 1 -- 172.21.15.69:0/988457885 wait complete. 2024-04-24T08:11:20.401 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:11:20.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.603+0000 7fd218800640 1 Processor -- start 2024-04-24T08:11:20.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.604+0000 7fd218800640 1 -- start start 2024-04-24T08:11:20.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.605+0000 7fd218800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2100cd3d0 0x7fd2100b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.605+0000 7fd218800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2100b4780 0x7fd2100b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.605+0000 7fd218800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2101631c0 con 0x7fd210073db0 2024-04-24T08:11:20.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.605+0000 7fd218800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd2100cf4d0 con 0x7fd2100b4780 2024-04-24T08:11:20.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.605+0000 7fd218800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2100cf7b0 con 0x7fd2100cd3d0 2024-04-24T08:11:20.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.606+0000 7fd216800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2100b4780 0x7fd2100b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:20.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.606+0000 7fd216800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd2100b4780 0x7fd2100b4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44924/0 (socket says 172.21.15.69:44924) 2024-04-24T08:11:20.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.606+0000 7fd216800640 1 -- 172.21.15.69:0/707626944 learned_addr learned my addr 172.21.15.69:0/707626944 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:20.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.606+0000 7fd215400640 1 --2- 172.21.15.69:0/707626944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2100cd3d0 0x7fd2100b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:20.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.606+0000 7fd216800640 1 -- 172.21.15.69:0/707626944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2100cd3d0 msgr2=0x7fd2100b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:20.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.606+0000 7fd216800640 1 --2- 172.21.15.69:0/707626944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2100cd3d0 0x7fd2100b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.606+0000 7fd216800640 1 -- 172.21.15.69:0/707626944 >> v1:172.21.15.121:6789/0 conn(0x7fd210073db0 legacy=0x7fd2100cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:20.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.606+0000 7fd216800640 1 -- 172.21.15.69:0/707626944 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2100cfa90 con 0x7fd2100b4780 2024-04-24T08:11:20.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.606+0000 7fd215400640 1 --2- 172.21.15.69:0/707626944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2100cd3d0 0x7fd2100b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:20.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.607+0000 7fd216800640 1 --2- 172.21.15.69:0/707626944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100b4780 0x7fd2100b4c30 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009e50 tx=0x7fd20c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e7d8f7210bba45 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:20.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.607+0000 7fd214a00640 1 -- 172.21.15.69:0/707626944 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c036070 con 0x7fd2100b4780 2024-04-24T08:11:20.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.608+0000 7fd214a00640 1 -- 172.21.15.69:0/707626944 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd20c031600 con 0x7fd2100b4780 2024-04-24T08:11:20.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.608+0000 7fd214a00640 1 -- 172.21.15.69:0/707626944 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c0319d0 con 0x7fd2100b4780 2024-04-24T08:11:20.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.608+0000 7fd218800640 1 -- 172.21.15.69:0/707626944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100b4780 msgr2=0x7fd2100b4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:20.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.608+0000 7fd218800640 1 --2- 172.21.15.69:0/707626944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100b4780 0x7fd2100b4c30 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd20c009e50 tx=0x7fd20c02a6c0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.608+0000 7fd218800640 1 -- 172.21.15.69:0/707626944 shutdown_connections 2024-04-24T08:11:20.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.608+0000 7fd218800640 1 --2- 172.21.15.69:0/707626944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100b4780 0x7fd2100b4c30 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.608+0000 7fd218800640 1 --2- 172.21.15.69:0/707626944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2100cd3d0 0x7fd2100b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.608+0000 7fd218800640 1 -- 172.21.15.69:0/707626944 >> 172.21.15.69:0/707626944 conn(0x7fd2100d0350 msgr2=0x7fd2100d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:20.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.608+0000 7fd218800640 1 -- 172.21.15.69:0/707626944 shutdown_connections 2024-04-24T08:11:20.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.608+0000 7fd218800640 1 -- 172.21.15.69:0/707626944 wait complete. 2024-04-24T08:11:20.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.610+0000 7fd218800640 1 Processor -- start 2024-04-24T08:11:20.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.610+0000 7fd218800640 1 -- start start 2024-04-24T08:11:20.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.611+0000 7fd218800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd210073db0 0x7fd210160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.611+0000 7fd218800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2100b4780 0x7fd2101ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.611+0000 7fd218800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100cd3d0 0x7fd2101eef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.611+0000 7fd218800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2100b5170 con 0x7fd2100cd3d0 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.611+0000 7fd218800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd2100d0050 con 0x7fd2100b4780 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.611+0000 7fd218800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2100ce470 con 0x7fd210073db0 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.611+0000 7fd216800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100cd3d0 0x7fd2101eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd216800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100cd3d0 0x7fd2101eef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44936/0 (socket says 172.21.15.69:44936) 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd216800640 1 -- 172.21.15.69:0/2608245243 learned_addr learned my addr 172.21.15.69:0/2608245243 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd215400640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2100b4780 0x7fd2101ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd215e00640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd210073db0 0x7fd210160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd216800640 1 -- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd210073db0 msgr2=0x7fd210160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd216800640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd210073db0 0x7fd210160410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd216800640 1 -- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2100b4780 msgr2=0x7fd2101ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:20.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd216800640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2100b4780 0x7fd2101ee9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd216800640 1 -- 172.21.15.69:0/2608245243 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd210164180 con 0x7fd2100cd3d0 2024-04-24T08:11:20.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd215400640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2100b4780 0x7fd2101ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:20.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.612+0000 7fd216800640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100cd3d0 0x7fd2101eef20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd20c034040 tx=0x7fd20c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:20.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.613+0000 7fd20a000640 1 -- 172.21.15.69:0/2608245243 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c036050 con 0x7fd2100cd3d0 2024-04-24T08:11:20.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.613+0000 7fd20a000640 1 -- 172.21.15.69:0/2608245243 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd20c042070 con 0x7fd2100cd3d0 2024-04-24T08:11:20.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.613+0000 7fd20a000640 1 -- 172.21.15.69:0/2608245243 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c009510 con 0x7fd2100cd3d0 2024-04-24T08:11:20.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.613+0000 7fd218800640 1 -- 172.21.15.69:0/2608245243 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd210164460 con 0x7fd2100cd3d0 2024-04-24T08:11:20.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.613+0000 7fd218800640 1 -- 172.21.15.69:0/2608245243 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd210164a20 con 0x7fd2100cd3d0 2024-04-24T08:11:20.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.614+0000 7fd218800640 1 -- 172.21.15.69:0/2608245243 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd210068a80 con 0x7fd2100cd3d0 2024-04-24T08:11:20.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.615+0000 7fd20a000640 1 -- 172.21.15.69:0/2608245243 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd20c007bd0 con 0x7fd2100cd3d0 2024-04-24T08:11:20.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.616+0000 7fd20a000640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1f406c8f0 0x7fd1f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:20.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.616+0000 7fd20a000640 1 -- 172.21.15.69:0/2608245243 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fd20c0acba0 con 0x7fd2100cd3d0 2024-04-24T08:11:20.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.617+0000 7fd215e00640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1f406c8f0 0x7fd1f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:20.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.618+0000 7fd215e00640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1f406c8f0 0x7fd1f406edb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd204003d90 tx=0x7fd204009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:20.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.620+0000 7fd20a000640 1 -- 172.21.15.69:0/2608245243 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd20c07aef0 con 0x7fd2100cd3d0 2024-04-24T08:11:20.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.805+0000 7fd218800640 1 -- 172.21.15.69:0/2608245243 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd2100997c0 con 0x7fd2100cd3d0 2024-04-24T08:11:20.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.806+0000 7fd20a000640 1 -- 172.21.15.69:0/2608245243 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) ==== 104+0+11 (secure 0 0 0) 0x7fd20c0810c0 con 0x7fd2100cd3d0 2024-04-24T08:11:20.808 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:11:20.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.808+0000 7fd218800640 1 -- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1f406c8f0 msgr2=0x7fd1f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:20.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.808+0000 7fd218800640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1f406c8f0 0x7fd1f406edb0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd204003d90 tx=0x7fd204009040 comp rx=0 tx=0).stop 2024-04-24T08:11:20.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.808+0000 7fd218800640 1 -- 172.21.15.69:0/2608245243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100cd3d0 msgr2=0x7fd2101eef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:20.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.808+0000 7fd218800640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100cd3d0 0x7fd2101eef20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd20c034040 tx=0x7fd20c004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:20.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.809+0000 7fd218800640 1 -- 172.21.15.69:0/2608245243 shutdown_connections 2024-04-24T08:11:20.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.809+0000 7fd218800640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd1f406c8f0 0x7fd1f406edb0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.809+0000 7fd218800640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2100cd3d0 0x7fd2101eef20 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.809+0000 7fd218800640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2100b4780 0x7fd2101ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.809+0000 7fd218800640 1 --2- 172.21.15.69:0/2608245243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd210073db0 0x7fd210160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:20.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.809+0000 7fd218800640 1 -- 172.21.15.69:0/2608245243 >> 172.21.15.69:0/2608245243 conn(0x7fd2100d0350 msgr2=0x7fd21006edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:20.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.809+0000 7fd218800640 1 -- 172.21.15.69:0/2608245243 shutdown_connections 2024-04-24T08:11:20.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:20.809+0000 7fd218800640 1 -- 172.21.15.69:0/2608245243 wait complete. 2024-04-24T08:11:20.828 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:11:21.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.023+0000 7f06c9400640 1 Processor -- start 2024-04-24T08:11:21.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.024+0000 7f06c9400640 1 -- start start 2024-04-24T08:11:21.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.025+0000 7f06c9400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c40d0690 0x7f06c40d23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.025+0000 7f06c9400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c40d2db0 con 0x7f06c40d2920 2024-04-24T08:11:21.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.025+0000 7f06c9400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06c40b2530 con 0x7f06c415ea70 2024-04-24T08:11:21.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.025+0000 7f06c9400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f06c40d18d0 con 0x7f06c40d0690 2024-04-24T08:11:21.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.025+0000 7f06c3200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f06c40d2920 0x7f06c40cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33914/0 (socket says 172.21.15.69:33914) 2024-04-24T08:11:21.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.025+0000 7f06c3200640 1 -- 172.21.15.69:0/972662701 learned_addr learned my addr 172.21.15.69:0/972662701 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:21.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.025+0000 7f06c1e00640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c40d0690 0x7f06c40d23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1538991304 0 0) 0x7f06c40d2db0 con 0x7f06c40d2920 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a8003520 con 0x7f06c40d2920 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2649648735 0 0) 0x7f06a8003520 con 0x7f06c40d2920 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c40d0690 msgr2=0x7f06c40d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c40d0690 0x7f06c40d23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 >> v1:172.21.15.121:6789/0 conn(0x7f06c415ea70 legacy=0x7f06c40cf940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c40d0cc0 con 0x7f06c40d2920 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f06bc001670 con 0x7f06c40d2920 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 >> v1:172.21.15.69:6789/0 conn(0x7f06c40d2920 legacy=0x7f06c40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a80039e0 0x7f06a8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c40d0690 0x7f06a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 0x7f06a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f06c40d18d0 con 0x7f06a8004d40 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f06a800baa0 con 0x7f06a80039e0 2024-04-24T08:11:21.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f06a800bca0 con 0x7f06c40d0690 2024-04-24T08:11:21.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.026+0000 7f06c3200640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 0x7f06a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.027+0000 7f06c2800640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a80039e0 0x7f06a8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.027+0000 7f06c3200640 1 -- 172.21.15.69:0/972662701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c40d0690 msgr2=0x7f06a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:21.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.027+0000 7f06c3200640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c40d0690 0x7f06a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.027+0000 7f06c3200640 1 -- 172.21.15.69:0/972662701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a80039e0 msgr2=0x7f06a8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:21.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.027+0000 7f06c3200640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a80039e0 0x7f06a8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.027+0000 7f06c3200640 1 -- 172.21.15.69:0/972662701 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a8003520 con 0x7f06a8004d40 2024-04-24T08:11:21.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.028+0000 7f06c3200640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 0x7f06a80051c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f06bc007fd0 tx=0x7f06a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9bd684b9d21d5af server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:21.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.028+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc007c20 con 0x7f06a8004d40 2024-04-24T08:11:21.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.028+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06bc005af0 con 0x7f06a8004d40 2024-04-24T08:11:21.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.028+0000 7f06c1400640 1 -- 172.21.15.69:0/972662701 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06bc007070 con 0x7f06a8004d40 2024-04-24T08:11:21.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.029+0000 7f06c9400640 1 -- 172.21.15.69:0/972662701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 msgr2=0x7f06a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.029+0000 7f06c9400640 1 --2- 172.21.15.69:0/972662701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 0x7f06a80051c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f06bc007fd0 tx=0x7f06a8005830 comp rx=0 tx=0).stop 2024-04-24T08:11:21.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.029+0000 7f06c3200640 1 -- 172.21.15.69:0/972662701 reap_dead start 2024-04-24T08:11:21.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.029+0000 7f06c9400640 1 -- 172.21.15.69:0/972662701 shutdown_connections 2024-04-24T08:11:21.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.029+0000 7f06c9400640 1 -- 172.21.15.69:0/972662701 >> 172.21.15.69:0/972662701 conn(0x7f06c408bf60 msgr2=0x7f06c408c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:21.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.029+0000 7f06c9400640 1 -- 172.21.15.69:0/972662701 shutdown_connections 2024-04-24T08:11:21.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.029+0000 7f06c9400640 1 -- 172.21.15.69:0/972662701 wait complete. 2024-04-24T08:11:21.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.031+0000 7f06c9400640 1 Processor -- start 2024-04-24T08:11:21.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.031+0000 7f06c9400640 1 -- start start 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 0x7f06c4157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c9400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c4159510 0x7f06c40c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c9400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06c40c91e0 0x7f06c40c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c9400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f06c40d1560 con 0x7f06a8004d40 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c9400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f06c40d1280 con 0x7f06c40c91e0 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c9400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f06c415ee50 con 0x7f06c4159510 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 0x7f06c4157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 0x7f06c4157140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44948/0 (socket says 172.21.15.69:44948) 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c2800640 1 -- 172.21.15.69:0/1756547982 learned_addr learned my addr 172.21.15.69:0/1756547982 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c2800640 1 -- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c4159510 msgr2=0x7f06c40c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c2800640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c4159510 0x7f06c40c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c2800640 1 -- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06c40c91e0 msgr2=0x7f06c40c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c2800640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06c40c91e0 0x7f06c40c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.032+0000 7f06c2800640 1 -- 172.21.15.69:0/1756547982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c40ca320 con 0x7f06a8004d40 2024-04-24T08:11:21.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.033+0000 7f06c2800640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 0x7f06c4157140 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f06b400e840 tx=0x7f06b400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:21.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.033+0000 7f06baa00640 1 -- 172.21.15.69:0/1756547982 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4003b80 con 0x7f06a8004d40 2024-04-24T08:11:21.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.033+0000 7f06baa00640 1 -- 172.21.15.69:0/1756547982 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b4018070 con 0x7f06a8004d40 2024-04-24T08:11:21.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.033+0000 7f06baa00640 1 -- 172.21.15.69:0/1756547982 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b40050d0 con 0x7f06a8004d40 2024-04-24T08:11:21.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.033+0000 7f06c9400640 1 -- 172.21.15.69:0/1756547982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c4158a30 con 0x7f06a8004d40 2024-04-24T08:11:21.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.033+0000 7f06c9400640 1 -- 172.21.15.69:0/1756547982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c4157680 con 0x7f06a8004d40 2024-04-24T08:11:21.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.034+0000 7f06c9400640 1 -- 172.21.15.69:0/1756547982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06c4068a80 con 0x7f06a8004d40 2024-04-24T08:11:21.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.035+0000 7f06baa00640 1 -- 172.21.15.69:0/1756547982 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f06b4005270 con 0x7f06a8004d40 2024-04-24T08:11:21.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.036+0000 7f06baa00640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f06a006c8f0 0x7f06a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.036+0000 7f06baa00640 1 -- 172.21.15.69:0/1756547982 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f06b408bb90 con 0x7f06a8004d40 2024-04-24T08:11:21.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.037+0000 7f06c1e00640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f06a006c8f0 0x7f06a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.038+0000 7f06c1e00640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f06a006c8f0 0x7f06a006edb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f06c415a280 tx=0x7f06ac009500 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:21.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.041+0000 7f06baa00640 1 -- 172.21.15.69:0/1756547982 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06b401a310 con 0x7f06a8004d40 2024-04-24T08:11:21.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.244+0000 7f06c9400640 1 -- 172.21.15.69:0/1756547982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f06c40b83e0 con 0x7f06a8004d40 2024-04-24T08:11:21.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.245+0000 7f06baa00640 1 -- 172.21.15.69:0/1756547982 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) ==== 100+0+11 (secure 0 0 0) 0x7f06b4059f70 con 0x7f06a8004d40 2024-04-24T08:11:21.247 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:11:21.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 -- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f06a006c8f0 msgr2=0x7f06a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f06a006c8f0 0x7f06a006edb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f06c415a280 tx=0x7f06ac009500 comp rx=0 tx=0).stop 2024-04-24T08:11:21.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 -- 172.21.15.69:0/1756547982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 msgr2=0x7f06c4157140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 0x7f06c4157140 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f06b400e840 tx=0x7f06b400ed00 comp rx=0 tx=0).stop 2024-04-24T08:11:21.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 -- 172.21.15.69:0/1756547982 shutdown_connections 2024-04-24T08:11:21.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f06a006c8f0 0x7f06a006edb0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06c40c91e0 0x7f06c40c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06c4159510 0x7f06c40c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 --2- 172.21.15.69:0/1756547982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06a8004d40 0x7f06c4157140 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 -- 172.21.15.69:0/1756547982 >> 172.21.15.69:0/1756547982 conn(0x7f06c408bf60 msgr2=0x7f06c40b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:21.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 -- 172.21.15.69:0/1756547982 shutdown_connections 2024-04-24T08:11:21.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.248+0000 7f06c9400640 1 -- 172.21.15.69:0/1756547982 wait complete. 2024-04-24T08:11:21.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:11:21.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.462+0000 7f480de00640 1 Processor -- start 2024-04-24T08:11:21.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.463+0000 7f480de00640 1 -- start start 2024-04-24T08:11:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.464+0000 7f480de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f480815d1d0 0x7f48080ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.464+0000 7f480de00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48080b30f0 0x7f48080ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.464+0000 7f480de00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f480815d660 con 0x7f4808156c60 2024-04-24T08:11:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.464+0000 7f480de00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f480806dc30 con 0x7f48080b30f0 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.464+0000 7f480de00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f480806df10 con 0x7f480815d1d0 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.464+0000 7f480d400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48080b30f0 0x7f48080ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.465+0000 7f480d400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48080b30f0 0x7f48080ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44958/0 (socket says 172.21.15.69:44958) 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.465+0000 7f480d400640 1 -- 172.21.15.69:0/3583085980 learned_addr learned my addr 172.21.15.69:0/3583085980 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.465+0000 7f480ca00640 1 --1- 172.21.15.69:0/3583085980 >> v1:172.21.15.69:6789/0 conn(0x7f4808156c60 0x7f480815cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33924/0 (socket says 172.21.15.69:33924) 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.465+0000 7f47ffe00640 1 --2- 172.21.15.69:0/3583085980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f480815d1d0 0x7f48080ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.465+0000 7f480d400640 1 -- 172.21.15.69:0/3583085980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f480815d1d0 msgr2=0x7f48080ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.465+0000 7f480d400640 1 --2- 172.21.15.69:0/3583085980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f480815d1d0 0x7f48080ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.465+0000 7f480d400640 1 -- 172.21.15.69:0/3583085980 >> v1:172.21.15.69:6789/0 conn(0x7f4808156c60 legacy=0x7f480815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.465+0000 7f480d400640 1 -- 172.21.15.69:0/3583085980 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480806e1f0 con 0x7f48080b30f0 2024-04-24T08:11:21.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.465+0000 7f4807e00640 1 -- 172.21.15.69:0/3583085980 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2699997110 0 0) 0x7f480815d660 con 0x7f4808156c60 2024-04-24T08:11:21.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.466+0000 7f480d400640 1 --2- 172.21.15.69:0/3583085980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48080b30f0 0x7f48080ad610 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4800009e50 tx=0x7f480002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87ecbab36e3731b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:21.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.466+0000 7f4807e00640 1 -- 172.21.15.69:0/3583085980 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800036070 con 0x7f48080b30f0 2024-04-24T08:11:21.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.466+0000 7f4807e00640 1 -- 172.21.15.69:0/3583085980 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4800031600 con 0x7f48080b30f0 2024-04-24T08:11:21.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.466+0000 7f4807e00640 1 -- 172.21.15.69:0/3583085980 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48000319d0 con 0x7f48080b30f0 2024-04-24T08:11:21.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.467+0000 7f480de00640 1 -- 172.21.15.69:0/3583085980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48080b30f0 msgr2=0x7f48080ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.467+0000 7f480de00640 1 --2- 172.21.15.69:0/3583085980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48080b30f0 0x7f48080ad610 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f4800009e50 tx=0x7f480002a710 comp rx=0 tx=0).stop 2024-04-24T08:11:21.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.467+0000 7f480de00640 1 -- 172.21.15.69:0/3583085980 shutdown_connections 2024-04-24T08:11:21.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.467+0000 7f480de00640 1 --2- 172.21.15.69:0/3583085980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48080b30f0 0x7f48080ad610 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.467+0000 7f480de00640 1 --2- 172.21.15.69:0/3583085980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f480815d1d0 0x7f48080ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.467+0000 7f480de00640 1 -- 172.21.15.69:0/3583085980 >> 172.21.15.69:0/3583085980 conn(0x7f48080d0160 msgr2=0x7f48080d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:21.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.467+0000 7f480de00640 1 -- 172.21.15.69:0/3583085980 shutdown_connections 2024-04-24T08:11:21.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.467+0000 7f480de00640 1 -- 172.21.15.69:0/3583085980 wait complete. 2024-04-24T08:11:21.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.469+0000 7f480de00640 1 Processor -- start 2024-04-24T08:11:21.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.469+0000 7f480de00640 1 -- start start 2024-04-24T08:11:21.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.470+0000 7f480de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48080b30f0 0x7f4808071be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.470+0000 7f480de00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4808156c60 0x7f48080735a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4808099910 0x7f48080c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480de00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f480806e710 con 0x7f4808099910 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480de00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f480815d940 con 0x7f4808156c60 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480de00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f480806e890 con 0x7f48080b30f0 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4808099910 0x7f48080c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4808099910 0x7f48080c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44970/0 (socket says 172.21.15.69:44970) 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480d400640 1 -- 172.21.15.69:0/1796959448 learned_addr learned my addr 172.21.15.69:0/1796959448 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f47ffe00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4808156c60 0x7f48080735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480ca00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48080b30f0 0x7f4808071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480d400640 1 -- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48080b30f0 msgr2=0x7f4808071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480d400640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48080b30f0 0x7f4808071be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480d400640 1 -- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4808156c60 msgr2=0x7f48080735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480d400640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4808156c60 0x7f48080735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f480d400640 1 -- 172.21.15.69:0/1796959448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4808155760 con 0x7f4808099910 2024-04-24T08:11:21.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.471+0000 7f47ffe00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4808156c60 0x7f48080735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:21.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.472+0000 7f480d400640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4808099910 0x7f48080c9ec0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4800034040 tx=0x7f4800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:21.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.472+0000 7f4805600640 1 -- 172.21.15.69:0/1796959448 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480003f020 con 0x7f4808099910 2024-04-24T08:11:21.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.472+0000 7f4805600640 1 -- 172.21.15.69:0/1796959448 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4800037040 con 0x7f4808099910 2024-04-24T08:11:21.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.472+0000 7f4805600640 1 -- 172.21.15.69:0/1796959448 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800036070 con 0x7f4808099910 2024-04-24T08:11:21.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.472+0000 7f480de00640 1 -- 172.21.15.69:0/1796959448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4808155a40 con 0x7f4808099910 2024-04-24T08:11:21.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.472+0000 7f480de00640 1 -- 172.21.15.69:0/1796959448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4808073ae0 con 0x7f4808099910 2024-04-24T08:11:21.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.474+0000 7f480de00640 1 -- 172.21.15.69:0/1796959448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4808075d50 con 0x7f4808099910 2024-04-24T08:11:21.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.474+0000 7f4805600640 1 -- 172.21.15.69:0/1796959448 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4800042020 con 0x7f4808099910 2024-04-24T08:11:21.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.475+0000 7f4805600640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f47e406c8f0 0x7f47e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:21.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.475+0000 7f4805600640 1 -- 172.21.15.69:0/1796959448 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f48000ac050 con 0x7f4808099910 2024-04-24T08:11:21.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.476+0000 7f480ca00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f47e406c8f0 0x7f47e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:21.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.477+0000 7f480ca00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f47e406c8f0 0x7f47e406edb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f47f0000f60 tx=0x7f47f0007b50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:21.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.480+0000 7f4805600640 1 -- 172.21.15.69:0/1796959448 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4800049030 con 0x7f4808099910 2024-04-24T08:11:21.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.696+0000 7f480de00640 1 -- 172.21.15.69:0/1796959448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f480806f9e0 con 0x7f4808099910 2024-04-24T08:11:21.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.696+0000 7f4805600640 1 -- 172.21.15.69:0/1796959448 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v46) ==== 80+0+0 (secure 0 0 0) 0x7f480007a440 con 0x7f4808099910 2024-04-24T08:11:21.698 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:11:21.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.699+0000 7f480de00640 1 -- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f47e406c8f0 msgr2=0x7f47e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.699+0000 7f480de00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f47e406c8f0 0x7f47e406edb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f47f0000f60 tx=0x7f47f0007b50 comp rx=0 tx=0).stop 2024-04-24T08:11:21.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.699+0000 7f480de00640 1 -- 172.21.15.69:0/1796959448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4808099910 msgr2=0x7f48080c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:21.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.699+0000 7f480de00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4808099910 0x7f48080c9ec0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4800034040 tx=0x7f4800004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:21.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.699+0000 7f480de00640 1 -- 172.21.15.69:0/1796959448 shutdown_connections 2024-04-24T08:11:21.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.699+0000 7f480de00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f47e406c8f0 0x7f47e406edb0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.699+0000 7f480de00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4808099910 0x7f48080c9ec0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.699+0000 7f480de00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4808156c60 0x7f48080735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.699+0000 7f480de00640 1 --2- 172.21.15.69:0/1796959448 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48080b30f0 0x7f4808071be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:21.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.699+0000 7f480de00640 1 -- 172.21.15.69:0/1796959448 >> 172.21.15.69:0/1796959448 conn(0x7f48080d0160 msgr2=0x7f48080b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:21.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.700+0000 7f480de00640 1 -- 172.21.15.69:0/1796959448 shutdown_connections 2024-04-24T08:11:21.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:21.700+0000 7f480de00640 1 -- 172.21.15.69:0/1796959448 wait complete. 2024-04-24T08:11:21.719 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:11:21.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:21.720 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:11:21.795 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[50855]: fuse finished with error 0 and tester_r 0 2024-04-24T08:11:21.800 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:11:21.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:21.800 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:11:21.884 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:11:21.884 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:11:21.884 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:11:21.923 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:11:21.923 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:11:21.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:21.923 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-24T08:11:22.006 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-24T08:11:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:11:22.007 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:11:22.007 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:11:22.007 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:11:22.058 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:11:22.058 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:11:22.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:22.059 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-24T08:11:22.156 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-24T08:11:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:11:22.157 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:11:22.157 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:11:22.157 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:11:22.214 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:11:22.214 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:11:22.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:22.215 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-24T08:11:22.267 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-24T08:11:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:11:22.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:22.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.475+0000 7f1d1cc00640 1 Processor -- start 2024-04-24T08:11:22.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.475+0000 7f1d1cc00640 1 -- start start 2024-04-24T08:11:22.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.477+0000 7f1d1cc00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1d1815d1d0 0x7f1d180ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.477+0000 7f1d1cc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d1815d660 con 0x7f1d180b30f0 2024-04-24T08:11:22.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.477+0000 7f1d1cc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d1806dc30 con 0x7f1d18156c60 2024-04-24T08:11:22.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.477+0000 7f1d1cc00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1d1806df10 con 0x7f1d1815d1d0 2024-04-24T08:11:22.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.477+0000 7f1d17e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1d180b30f0 0x7f1d1806e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33928/0 (socket says 172.21.15.69:33928) 2024-04-24T08:11:22.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.477+0000 7f1d17e00640 1 -- 172.21.15.69:0/3052207474 learned_addr learned my addr 172.21.15.69:0/3052207474 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.477+0000 7f1d16a00640 1 --2- 172.21.15.69:0/3052207474 >> v2:172.21.15.121:3300/0 conn(0x7f1d1815d1d0 0x7f1d180ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1285197731 0 0) 0x7f1d1815d660 con 0x7f1d180b30f0 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cfc003520 con 0x7f1d180b30f0 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2911178478 0 0) 0x7f1cfc003520 con 0x7f1d180b30f0 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 >> v2:172.21.15.121:3300/0 conn(0x7f1d1815d1d0 msgr2=0x7f1d180ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 --2- 172.21.15.69:0/3052207474 >> v2:172.21.15.121:3300/0 conn(0x7f1d1815d1d0 0x7f1d180ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 >> v1:172.21.15.121:6789/0 conn(0x7f1d18156c60 legacy=0x7f1d1815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1806e1f0 con 0x7f1d180b30f0 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f1d100019f0 con 0x7f1d180b30f0 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 >> v1:172.21.15.69:6789/0 conn(0x7f1d180b30f0 legacy=0x7f1d1806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1cfc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cfc004800 0x7f1cfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cfc0051c0 0x7f1cfc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1806df10 con 0x7f1cfc0039b0 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1cfc00c060 con 0x7f1cfc004800 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1cfc00c260 con 0x7f1cfc0051c0 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.478+0000 7f1d17400640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1cfc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.479+0000 7f1d16a00640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cfc004800 0x7f1cfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.479+0000 7f1d17400640 1 -- 172.21.15.69:0/3052207474 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cfc0051c0 msgr2=0x7f1cfc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.479+0000 7f1d17400640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1cfc0051c0 0x7f1cfc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.479+0000 7f1d17400640 1 -- 172.21.15.69:0/3052207474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cfc004800 msgr2=0x7f1cfc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.479+0000 7f1d17400640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cfc004800 0x7f1cfc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.479+0000 7f1d17400640 1 -- 172.21.15.69:0/3052207474 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cfc003520 con 0x7f1cfc0039b0 2024-04-24T08:11:22.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.479+0000 7f1d16a00640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cfc004800 0x7f1cfc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:22.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.479+0000 7f1d17e00640 1 -- 172.21.15.69:0/3052207474 reap_dead start 2024-04-24T08:11:22.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.480+0000 7f1d17400640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1cfc003dd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f1d08001750 tx=0x7f1cfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618886ea8e82ab93 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:22.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.480+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d08001370 con 0x7f1cfc0039b0 2024-04-24T08:11:22.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.480+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d080052d0 con 0x7f1cfc0039b0 2024-04-24T08:11:22.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.480+0000 7f1d16000640 1 -- 172.21.15.69:0/3052207474 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0802abc0 con 0x7f1cfc0039b0 2024-04-24T08:11:22.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.481+0000 7f1d1cc00640 1 -- 172.21.15.69:0/3052207474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 msgr2=0x7f1cfc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:22.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.481+0000 7f1d1cc00640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1cfc003dd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f1d08001750 tx=0x7f1cfc005830 comp rx=0 tx=0).stop 2024-04-24T08:11:22.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.481+0000 7f1d1cc00640 1 -- 172.21.15.69:0/3052207474 shutdown_connections 2024-04-24T08:11:22.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.481+0000 7f1d1cc00640 1 --2- 172.21.15.69:0/3052207474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1cfc003dd0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.481+0000 7f1d1cc00640 1 -- 172.21.15.69:0/3052207474 >> 172.21.15.69:0/3052207474 conn(0x7f1d180d0160 msgr2=0x7f1d180d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:22.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.481+0000 7f1d1cc00640 1 -- 172.21.15.69:0/3052207474 shutdown_connections 2024-04-24T08:11:22.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.481+0000 7f1d1cc00640 1 -- 172.21.15.69:0/3052207474 wait complete. 2024-04-24T08:11:22.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.483+0000 7f1d1cc00640 1 Processor -- start 2024-04-24T08:11:22.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.483+0000 7f1d1cc00640 1 -- start start 2024-04-24T08:11:22.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d1cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1d18142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d1cc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18159a80 0x7f1d181549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d1cc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d18145970 0x7f1d181447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d1cc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1815cac0 con 0x7f1cfc0039b0 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d1cc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1d1815d940 con 0x7f1d18159a80 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d1cc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1d1815cc40 con 0x7f1d18145970 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d17400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1d18142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d17400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1d18142e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44988/0 (socket says 172.21.15.69:44988) 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d17400640 1 -- 172.21.15.69:0/2666756538 learned_addr learned my addr 172.21.15.69:0/2666756538 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d16a00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18159a80 0x7f1d181549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.484+0000 7f1d17e00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d18145970 0x7f1d181447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d17400640 1 -- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d18145970 msgr2=0x7f1d181447f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d17400640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d18145970 0x7f1d181447f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d17400640 1 -- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18159a80 msgr2=0x7f1d181549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d17400640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18159a80 0x7f1d181549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d17400640 1 -- 172.21.15.69:0/2666756538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d1815ed90 con 0x7f1cfc0039b0 2024-04-24T08:11:22.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d17e00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d18145970 0x7f1d181447f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:22.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d17400640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1d18142e20 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1d08005040 tx=0x7f1d08001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:22.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d0f400640 1 -- 172.21.15.69:0/2666756538 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d0803f030 con 0x7f1cfc0039b0 2024-04-24T08:11:22.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d0f400640 1 -- 172.21.15.69:0/2666756538 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d08005480 con 0x7f1cfc0039b0 2024-04-24T08:11:22.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d1cc00640 1 -- 172.21.15.69:0/2666756538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d1815f070 con 0x7f1cfc0039b0 2024-04-24T08:11:22.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.485+0000 7f1d0f400640 1 -- 172.21.15.69:0/2666756538 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d08037040 con 0x7f1cfc0039b0 2024-04-24T08:11:22.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.486+0000 7f1d1cc00640 1 -- 172.21.15.69:0/2666756538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d1815f350 con 0x7f1cfc0039b0 2024-04-24T08:11:22.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.487+0000 7f1d1cc00640 1 -- 172.21.15.69:0/2666756538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d18075d50 con 0x7f1cfc0039b0 2024-04-24T08:11:22.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.489+0000 7f1d0f400640 1 -- 172.21.15.69:0/2666756538 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d08042070 con 0x7f1cfc0039b0 2024-04-24T08:11:22.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.489+0000 7f1d0f400640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1cf806c920 0x7f1cf806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.489+0000 7f1d0f400640 1 -- 172.21.15.69:0/2666756538 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f1d080ab1f0 con 0x7f1cfc0039b0 2024-04-24T08:11:22.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.490+0000 7f1d16a00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1cf806c920 0x7f1cf806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.491+0000 7f1d16a00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1cf806c920 0x7f1cf806ede0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1d1815fa20 tx=0x7f1d04008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:22.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.494+0000 7f1d0f400640 1 -- 172.21.15.69:0/2666756538 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d080433b0 con 0x7f1cfc0039b0 2024-04-24T08:11:22.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.732+0000 7f1d1cc00640 1 -- 172.21.15.69:0/2666756538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d1806f9e0 con 0x7f1cfc0039b0 2024-04-24T08:11:22.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.732+0000 7f1d0f400640 1 -- 172.21.15.69:0/2666756538 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5184 (secure 0 0 0) 0x7f1d080794b0 con 0x7f1cfc0039b0 2024-04-24T08:11:22.734 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:22.734 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/376722301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":376722301},{"type":"v1","addr":"172.21.15.69:6835","nonce":376722301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4990,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1327624292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1327624292},{"type":"v1","addr":"172.21.15.121:6836","nonce":1327624292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/3244002416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3244002416},{"type":"v1","addr":"172.21.15.69:6834","nonce":3244002416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:10:35.400788+0000","modified":"2024-04-24T08:10:38.457641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5793},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5793":{"gid":5793,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/3424934990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3424934990},{"type":"v1","addr":"172.21.15.121:6837","nonce":3424934990}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5793,"qdb_cluster":[5793]},"id":3}]} 2024-04-24T08:11:22.734 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-04-24T08:11:22.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.735+0000 7f1d1cc00640 1 -- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1cf806c920 msgr2=0x7f1cf806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:22.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.735+0000 7f1d1cc00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1cf806c920 0x7f1cf806ede0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1d1815fa20 tx=0x7f1d04008040 comp rx=0 tx=0).stop 2024-04-24T08:11:22.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.735+0000 7f1d1cc00640 1 -- 172.21.15.69:0/2666756538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 msgr2=0x7f1d18142e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:22.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.735+0000 7f1d1cc00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1d18142e20 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1d08005040 tx=0x7f1d08001120 comp rx=0 tx=0).stop 2024-04-24T08:11:22.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.736+0000 7f1d1cc00640 1 -- 172.21.15.69:0/2666756538 shutdown_connections 2024-04-24T08:11:22.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.736+0000 7f1d1cc00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1cf806c920 0x7f1cf806ede0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.736+0000 7f1d1cc00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d18145970 0x7f1d181447f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.736+0000 7f1d1cc00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d18159a80 0x7f1d181549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.736+0000 7f1d1cc00640 1 --2- 172.21.15.69:0/2666756538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1cfc0039b0 0x7f1d18142e20 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.736+0000 7f1d1cc00640 1 -- 172.21.15.69:0/2666756538 >> 172.21.15.69:0/2666756538 conn(0x7f1d180d0160 msgr2=0x7f1d1815a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:22.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.736+0000 7f1d1cc00640 1 -- 172.21.15.69:0/2666756538 shutdown_connections 2024-04-24T08:11:22.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.736+0000 7f1d1cc00640 1 -- 172.21.15.69:0/2666756538 wait complete. 2024-04-24T08:11:22.757 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:11:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.945+0000 7fc813200640 1 Processor -- start 2024-04-24T08:11:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.945+0000 7fc813200640 1 -- start start 2024-04-24T08:11:22.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.947+0000 7fc813200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc8141523f0 0x7fc81406db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.947+0000 7fc813200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc81415b980 0x7fc81415be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.947+0000 7fc813200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8140aeac0 con 0x7fc814072e20 2024-04-24T08:11:22.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.947+0000 7fc813200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc8140aeda0 con 0x7fc8141523f0 2024-04-24T08:11:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.947+0000 7fc813200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc8140af080 con 0x7fc81415b980 2024-04-24T08:11:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.947+0000 7fc811e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc8141523f0 0x7fc81406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.947+0000 7fc811e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc8141523f0 0x7fc81406db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45002/0 (socket says 172.21.15.69:45002) 2024-04-24T08:11:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.947+0000 7fc811e00640 1 -- 172.21.15.69:0/3154440657 learned_addr learned my addr 172.21.15.69:0/3154440657 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.948+0000 7fc811e00640 1 -- 172.21.15.69:0/3154440657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc81415b980 msgr2=0x7fc81415be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.948+0000 7fc812800640 1 --2- 172.21.15.69:0/3154440657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc81415b980 0x7fc81415be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.948+0000 7fc811e00640 1 --2- 172.21.15.69:0/3154440657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc81415b980 0x7fc81415be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.948+0000 7fc811e00640 1 -- 172.21.15.69:0/3154440657 >> v1:172.21.15.121:6789/0 conn(0x7fc814072e20 legacy=0x7fc814073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.948+0000 7fc811e00640 1 -- 172.21.15.69:0/3154440657 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8140af360 con 0x7fc8141523f0 2024-04-24T08:11:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.948+0000 7fc812800640 1 --2- 172.21.15.69:0/3154440657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc81415b980 0x7fc81415be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.949+0000 7fc811e00640 1 --2- 172.21.15.69:0/3154440657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 0x7fc81406db90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc804009870 tx=0x7fc80402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d82b22c68f5ace1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.949+0000 7fc810a00640 1 -- 172.21.15.69:0/3154440657 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804037070 con 0x7fc8141523f0 2024-04-24T08:11:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.949+0000 7fc810a00640 1 -- 172.21.15.69:0/3154440657 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc804032440 con 0x7fc8141523f0 2024-04-24T08:11:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.949+0000 7fc810a00640 1 -- 172.21.15.69:0/3154440657 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc804032810 con 0x7fc8141523f0 2024-04-24T08:11:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.950+0000 7fc813200640 1 -- 172.21.15.69:0/3154440657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 msgr2=0x7fc81406db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.950+0000 7fc813200640 1 --2- 172.21.15.69:0/3154440657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 0x7fc81406db90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc804009870 tx=0x7fc80402a180 comp rx=0 tx=0).stop 2024-04-24T08:11:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.950+0000 7fc813200640 1 -- 172.21.15.69:0/3154440657 shutdown_connections 2024-04-24T08:11:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.950+0000 7fc813200640 1 --2- 172.21.15.69:0/3154440657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc81415b980 0x7fc81415be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.950+0000 7fc813200640 1 --2- 172.21.15.69:0/3154440657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 0x7fc81406db90 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.950+0000 7fc813200640 1 -- 172.21.15.69:0/3154440657 >> 172.21.15.69:0/3154440657 conn(0x7fc8140d0160 msgr2=0x7fc8140d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.950+0000 7fc813200640 1 -- 172.21.15.69:0/3154440657 shutdown_connections 2024-04-24T08:11:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.950+0000 7fc813200640 1 -- 172.21.15.69:0/3154440657 wait complete. 2024-04-24T08:11:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.951+0000 7fc813200640 1 Processor -- start 2024-04-24T08:11:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.952+0000 7fc813200640 1 -- start start 2024-04-24T08:11:22.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.952+0000 7fc813200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc814072e20 0x7fc81414a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.952+0000 7fc813200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 0x7fc81409ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc813200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc81415b980 0x7fc814153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc813200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc814073970 con 0x7fc8141523f0 2024-04-24T08:11:22.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc813200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc8141527d0 con 0x7fc814072e20 2024-04-24T08:11:22.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc813200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc814073af0 con 0x7fc81415b980 2024-04-24T08:11:22.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc811400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 0x7fc81409ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc811400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 0x7fc81409ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45010/0 (socket says 172.21.15.69:45010) 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc811400640 1 -- 172.21.15.69:0/1744221318 learned_addr learned my addr 172.21.15.69:0/1744221318 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc811400640 1 -- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc81415b980 msgr2=0x7fc814153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc811400640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc81415b980 0x7fc814153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc811400640 1 -- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc814072e20 msgr2=0x7fc81414a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc811e00640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc814072e20 0x7fc81414a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc811400640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc814072e20 0x7fc81414a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc811400640 1 -- 172.21.15.69:0/1744221318 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8141579a0 con 0x7fc8141523f0 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.953+0000 7fc811400640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 0x7fc81409ab60 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc80000c430 tx=0x7fc80000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.954+0000 7fc80a000640 1 -- 172.21.15.69:0/1744221318 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800011790 con 0x7fc8141523f0 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.954+0000 7fc80a000640 1 -- 172.21.15.69:0/1744221318 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc800017070 con 0x7fc8141523f0 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.954+0000 7fc80a000640 1 -- 172.21.15.69:0/1744221318 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc800011bb0 con 0x7fc8141523f0 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.954+0000 7fc813200640 1 -- 172.21.15.69:0/1744221318 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc814157c10 con 0x7fc8141523f0 2024-04-24T08:11:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.954+0000 7fc813200640 1 -- 172.21.15.69:0/1744221318 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8141581d0 con 0x7fc8141523f0 2024-04-24T08:11:22.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.955+0000 7fc813200640 1 -- 172.21.15.69:0/1744221318 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc814075d50 con 0x7fc8141523f0 2024-04-24T08:11:22.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.956+0000 7fc80a000640 1 -- 172.21.15.69:0/1744221318 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc80001c460 con 0x7fc8141523f0 2024-04-24T08:11:22.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.957+0000 7fc80a000640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc7f406c8f0 0x7fc7f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:22.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.957+0000 7fc80a000640 1 -- 172.21.15.69:0/1744221318 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fc80008bb50 con 0x7fc8141523f0 2024-04-24T08:11:22.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.958+0000 7fc811e00640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc7f406c8f0 0x7fc7f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:22.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.959+0000 7fc811e00640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc7f406c8f0 0x7fc7f406edb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc804034000 tx=0x7fc80402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:22.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:22.962+0000 7fc80a000640 1 -- 172.21.15.69:0/1744221318 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc800059ea0 con 0x7fc8141523f0 2024-04-24T08:11:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.161+0000 7fc813200640 1 -- 172.21.15.69:0/1744221318 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc81406dda0 con 0x7fc8141523f0 2024-04-24T08:11:23.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.164+0000 7fc80a000640 1 -- 172.21.15.69:0/1744221318 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+16740 (secure 0 0 0) 0x7fc8000597a0 con 0x7fc8141523f0 2024-04-24T08:11:23.166 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:23.166 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":46,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:03.085538+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:10:33.100132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-24T08:10:34.379302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":5}]},{"pgid":"7.36","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:11:23.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 -- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc7f406c8f0 msgr2=0x7fc7f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc7f406c8f0 0x7fc7f406edb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fc804034000 tx=0x7fc80402e000 comp rx=0 tx=0).stop 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 -- 172.21.15.69:0/1744221318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 msgr2=0x7fc81409ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 0x7fc81409ab60 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc80000c430 tx=0x7fc80000c8f0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 -- 172.21.15.69:0/1744221318 shutdown_connections 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc7f406c8f0 0x7fc7f406edb0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8141523f0 0x7fc81409ab60 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc81415b980 0x7fc814153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 --2- 172.21.15.69:0/1744221318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc814072e20 0x7fc81414a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 -- 172.21.15.69:0/1744221318 >> 172.21.15.69:0/1744221318 conn(0x7fc8140d0160 msgr2=0x7fc814073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 -- 172.21.15.69:0/1744221318 shutdown_connections 2024-04-24T08:11:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.167+0000 7fc813200640 1 -- 172.21.15.69:0/1744221318 wait complete. 2024-04-24T08:11:23.187 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:10:33.100132+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:10:34.379302+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:11:23.188 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:11:23.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.385+0000 7f22ab000640 1 Processor -- start 2024-04-24T08:11:23.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.385+0000 7f22ab000640 1 -- start start 2024-04-24T08:11:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.386+0000 7f22ab000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f22a415e850 0x7f22a40cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.387+0000 7f22ab000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a40d2710 con 0x7f22a40d1860 2024-04-24T08:11:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.387+0000 7f22ab000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a40d29f0 con 0x7f22a415ec30 2024-04-24T08:11:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.387+0000 7f22ab000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f22a40d37b0 con 0x7f22a415e850 2024-04-24T08:11:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.387+0000 7f22a3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f22a415e850 0x7f22a40cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:23.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.387+0000 7f22a9000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f22a40d1860 0x7f22a41688f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33942/0 (socket says 172.21.15.69:33942) 2024-04-24T08:11:23.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.387+0000 7f22a9000640 1 -- 172.21.15.69:0/3365020349 learned_addr learned my addr 172.21.15.69:0/3365020349 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:23.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.387+0000 7f22a3e00640 1 -- 172.21.15.69:0/3365020349 >> v1:172.21.15.121:6789/0 conn(0x7f22a415ec30 legacy=0x7f22a40d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:23.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.387+0000 7f22a3e00640 1 -- 172.21.15.69:0/3365020349 >> v1:172.21.15.69:6789/0 conn(0x7f22a40d1860 legacy=0x7f22a41688f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.387+0000 7f22a3e00640 1 -- 172.21.15.69:0/3365020349 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a40d2cd0 con 0x7f22a415e850 2024-04-24T08:11:23.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.388+0000 7f22a3e00640 1 --2- 172.21.15.69:0/3365020349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415e850 0x7f22a40cff10 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f22940097a0 tx=0x7f229402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe6dee63ccd790a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:23.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.389+0000 7f22a2a00640 1 -- 172.21.15.69:0/3365020349 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294037070 con 0x7f22a415e850 2024-04-24T08:11:23.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.389+0000 7f22a2a00640 1 -- 172.21.15.69:0/3365020349 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f229402ae50 con 0x7f22a415e850 2024-04-24T08:11:23.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.389+0000 7f22a2a00640 1 -- 172.21.15.69:0/3365020349 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294032660 con 0x7f22a415e850 2024-04-24T08:11:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.389+0000 7f22ab000640 1 -- 172.21.15.69:0/3365020349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415e850 msgr2=0x7f22a40cff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.389+0000 7f22ab000640 1 --2- 172.21.15.69:0/3365020349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415e850 0x7f22a40cff10 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f22940097a0 tx=0x7f229402a140 comp rx=0 tx=0).stop 2024-04-24T08:11:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.389+0000 7f22ab000640 1 -- 172.21.15.69:0/3365020349 shutdown_connections 2024-04-24T08:11:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.389+0000 7f22ab000640 1 --2- 172.21.15.69:0/3365020349 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415e850 0x7f22a40cff10 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.390+0000 7f22ab000640 1 -- 172.21.15.69:0/3365020349 >> 172.21.15.69:0/3365020349 conn(0x7f22a408bf60 msgr2=0x7f22a40d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.390+0000 7f22ab000640 1 -- 172.21.15.69:0/3365020349 shutdown_connections 2024-04-24T08:11:23.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.390+0000 7f22ab000640 1 -- 172.21.15.69:0/3365020349 wait complete. 2024-04-24T08:11:23.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.391+0000 7f22ab000640 1 Processor -- start 2024-04-24T08:11:23.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.391+0000 7f22ab000640 1 -- start start 2024-04-24T08:11:23.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.392+0000 7f22ab000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a40d1860 0x7f22a40b9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.392+0000 7f22ab000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a415e850 0x7f22a40b6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.392+0000 7f22ab000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415ec30 0x7f22a4151ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.392+0000 7f22ab000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f22a40d3290 con 0x7f22a415ec30 2024-04-24T08:11:23.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.392+0000 7f22ab000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f22a406eba0 con 0x7f22a40d1860 2024-04-24T08:11:23.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.392+0000 7f22ab000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f22a40d2330 con 0x7f22a415e850 2024-04-24T08:11:23.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.393+0000 7f22a9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415ec30 0x7f22a4151ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:23.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.393+0000 7f22a9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415ec30 0x7f22a4151ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45028/0 (socket says 172.21.15.69:45028) 2024-04-24T08:11:23.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.393+0000 7f22a9000640 1 -- 172.21.15.69:0/4244645033 learned_addr learned my addr 172.21.15.69:0/4244645033 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.393+0000 7f22a3e00640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a40d1860 0x7f22a40b9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.393+0000 7f22a9000640 1 -- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a415e850 msgr2=0x7f22a40b6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.393+0000 7f22a9000640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a415e850 0x7f22a40b6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.393+0000 7f22a9000640 1 -- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a40d1860 msgr2=0x7f22a40b9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.393+0000 7f22a9000640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a40d1860 0x7f22a40b9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.393+0000 7f22a9000640 1 -- 172.21.15.69:0/4244645033 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a40b88d0 con 0x7f22a415ec30 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.393+0000 7f22a9000640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415ec30 0x7f22a4151ab0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f229c00cb40 tx=0x7f229c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.394+0000 7f229be00640 1 -- 172.21.15.69:0/4244645033 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c017070 con 0x7f22a415ec30 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.394+0000 7f229be00640 1 -- 172.21.15.69:0/4244645033 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f229c00aca0 con 0x7f22a415ec30 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.394+0000 7f229be00640 1 -- 172.21.15.69:0/4244645033 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229c012600 con 0x7f22a415ec30 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.394+0000 7f22ab000640 1 -- 172.21.15.69:0/4244645033 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a40b8bb0 con 0x7f22a415ec30 2024-04-24T08:11:23.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.394+0000 7f22ab000640 1 -- 172.21.15.69:0/4244645033 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a40b9170 con 0x7f22a415ec30 2024-04-24T08:11:23.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.395+0000 7f22ab000640 1 -- 172.21.15.69:0/4244645033 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a4068a80 con 0x7f22a415ec30 2024-04-24T08:11:23.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.396+0000 7f229be00640 1 -- 172.21.15.69:0/4244645033 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f229c01a070 con 0x7f22a415ec30 2024-04-24T08:11:23.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.397+0000 7f229be00640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f228006c8f0 0x7f228006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.397+0000 7f229be00640 1 -- 172.21.15.69:0/4244645033 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f229c08b840 con 0x7f22a415ec30 2024-04-24T08:11:23.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.398+0000 7f22a3e00640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f228006c8f0 0x7f228006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:23.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.399+0000 7f22a3e00640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f228006c8f0 0x7f228006edb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2294034000 tx=0x7f229402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:23.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.401+0000 7f229be00640 1 -- 172.21.15.69:0/4244645033 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f229c0595e0 con 0x7f22a415ec30 2024-04-24T08:11:23.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.600+0000 7f22ab000640 1 -- 172.21.15.69:0/4244645033 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f22a40b26f0 con 0x7f22a415ec30 2024-04-24T08:11:23.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.601+0000 7f229be00640 1 -- 172.21.15.69:0/4244645033 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) ==== 93+0+10 (secure 0 0 0) 0x7f229c05d8e0 con 0x7f22a415ec30 2024-04-24T08:11:23.602 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:11:23.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.603+0000 7f22ab000640 1 -- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f228006c8f0 msgr2=0x7f228006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.603+0000 7f22ab000640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f228006c8f0 0x7f228006edb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2294034000 tx=0x7f229402e000 comp rx=0 tx=0).stop 2024-04-24T08:11:23.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.603+0000 7f22ab000640 1 -- 172.21.15.69:0/4244645033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415ec30 msgr2=0x7f22a4151ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.603+0000 7f22ab000640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415ec30 0x7f22a4151ab0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f229c00cb40 tx=0x7f229c00a060 comp rx=0 tx=0).stop 2024-04-24T08:11:23.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.604+0000 7f22ab000640 1 -- 172.21.15.69:0/4244645033 shutdown_connections 2024-04-24T08:11:23.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.604+0000 7f22ab000640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f228006c8f0 0x7f228006edb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.604+0000 7f22ab000640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22a415ec30 0x7f22a4151ab0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.604+0000 7f22ab000640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a415e850 0x7f22a40b6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.604+0000 7f22ab000640 1 --2- 172.21.15.69:0/4244645033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22a40d1860 0x7f22a40b9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.604+0000 7f22ab000640 1 -- 172.21.15.69:0/4244645033 >> 172.21.15.69:0/4244645033 conn(0x7f22a408bf60 msgr2=0x7f22a40d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:23.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.604+0000 7f22ab000640 1 -- 172.21.15.69:0/4244645033 shutdown_connections 2024-04-24T08:11:23.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.604+0000 7f22ab000640 1 -- 172.21.15.69:0/4244645033 wait complete. 2024-04-24T08:11:23.622 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:11:23.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.822+0000 7fdc56a00640 1 Processor -- start 2024-04-24T08:11:23.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.822+0000 7fdc56a00640 1 -- start start 2024-04-24T08:11:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.824+0000 7fdc56a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdc50073db0 0x7fdc500b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.824+0000 7fdc56a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdc500cccc0 0x7fdc500cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.824+0000 7fdc56a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc501631c0 con 0x7fdc500b4780 2024-04-24T08:11:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.824+0000 7fdc56a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fdc500cd690 con 0x7fdc500cccc0 2024-04-24T08:11:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.824+0000 7fdc56a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fdc500cf4d0 con 0x7fdc50073db0 2024-04-24T08:11:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.824+0000 7fdc47400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdc500cccc0 0x7fdc500cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:23.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.824+0000 7fdc47400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdc500cccc0 0x7fdc500cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45044/0 (socket says 172.21.15.69:45044) 2024-04-24T08:11:23.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.824+0000 7fdc47400640 1 -- 172.21.15.69:0/1624876473 learned_addr learned my addr 172.21.15.69:0/1624876473 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:23.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.825+0000 7fdc4fe00640 1 --2- 172.21.15.69:0/1624876473 >> v2:172.21.15.121:3300/0 conn(0x7fdc50073db0 0x7fdc500b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:23.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.825+0000 7fdc47400640 1 -- 172.21.15.69:0/1624876473 >> v2:172.21.15.121:3300/0 conn(0x7fdc50073db0 msgr2=0x7fdc500b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:23.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.825+0000 7fdc47400640 1 --2- 172.21.15.69:0/1624876473 >> v2:172.21.15.121:3300/0 conn(0x7fdc50073db0 0x7fdc500b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.825+0000 7fdc47400640 1 -- 172.21.15.69:0/1624876473 >> v1:172.21.15.121:6789/0 conn(0x7fdc500b4780 legacy=0x7fdc500b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:23.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.825+0000 7fdc47400640 1 -- 172.21.15.69:0/1624876473 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc500cf7b0 con 0x7fdc500cccc0 2024-04-24T08:11:23.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.826+0000 7fdc47400640 1 --2- 172.21.15.69:0/1624876473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500cccc0 0x7fdc500cd150 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdc500cde60 tx=0x7fdc3802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6affef02e54b55de server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:23.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.826+0000 7fdc4f400640 1 -- 172.21.15.69:0/1624876473 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38037070 con 0x7fdc500cccc0 2024-04-24T08:11:23.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.826+0000 7fdc4f400640 1 -- 172.21.15.69:0/1624876473 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc38032440 con 0x7fdc500cccc0 2024-04-24T08:11:23.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.826+0000 7fdc4f400640 1 -- 172.21.15.69:0/1624876473 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38032810 con 0x7fdc500cccc0 2024-04-24T08:11:23.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.827+0000 7fdc56a00640 1 -- 172.21.15.69:0/1624876473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500cccc0 msgr2=0x7fdc500cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.827+0000 7fdc56a00640 1 --2- 172.21.15.69:0/1624876473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500cccc0 0x7fdc500cd150 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdc500cde60 tx=0x7fdc3802a230 comp rx=0 tx=0).stop 2024-04-24T08:11:23.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.827+0000 7fdc56a00640 1 -- 172.21.15.69:0/1624876473 shutdown_connections 2024-04-24T08:11:23.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.827+0000 7fdc56a00640 1 --2- 172.21.15.69:0/1624876473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500cccc0 0x7fdc500cd150 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.827+0000 7fdc56a00640 1 --2- 172.21.15.69:0/1624876473 >> v2:172.21.15.121:3300/0 conn(0x7fdc50073db0 0x7fdc500b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.827+0000 7fdc56a00640 1 -- 172.21.15.69:0/1624876473 >> 172.21.15.69:0/1624876473 conn(0x7fdc500d0350 msgr2=0x7fdc500d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:23.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.827+0000 7fdc56a00640 1 -- 172.21.15.69:0/1624876473 shutdown_connections 2024-04-24T08:11:23.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.827+0000 7fdc56a00640 1 -- 172.21.15.69:0/1624876473 wait complete. 2024-04-24T08:11:23.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.829+0000 7fdc56a00640 1 Processor -- start 2024-04-24T08:11:23.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.830+0000 7fdc56a00640 1 -- start start 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc56a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc50073db0 0x7fdc5015ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc56a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500b4780 0x7fdc50165190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc56a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc500cccc0 0x7fdc501f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc56a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdc500d0050 con 0x7fdc500b4780 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc56a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdc500cfd70 con 0x7fdc500cccc0 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc56a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdc500ce470 con 0x7fdc50073db0 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc47400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500b4780 0x7fdc50165190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc47400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500b4780 0x7fdc50165190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45048/0 (socket says 172.21.15.69:45048) 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc47400640 1 -- 172.21.15.69:0/1329414685 learned_addr learned my addr 172.21.15.69:0/1329414685 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc47400640 1 -- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc50073db0 msgr2=0x7fdc5015ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc47400640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc50073db0 0x7fdc5015ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc47400640 1 -- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc500cccc0 msgr2=0x7fdc501f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:23.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc47400640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc500cccc0 0x7fdc501f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:23.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.831+0000 7fdc47400640 1 -- 172.21.15.69:0/1329414685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc500ae7e0 con 0x7fdc500b4780 2024-04-24T08:11:23.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.832+0000 7fdc47400640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500b4780 0x7fdc50165190 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdc3802a270 tx=0x7fdc380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:23.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.832+0000 7fdc4cc00640 1 -- 172.21.15.69:0/1329414685 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38041020 con 0x7fdc500b4780 2024-04-24T08:11:23.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.832+0000 7fdc4cc00640 1 -- 172.21.15.69:0/1329414685 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc380306c0 con 0x7fdc500b4780 2024-04-24T08:11:23.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.832+0000 7fdc4cc00640 1 -- 172.21.15.69:0/1329414685 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38037040 con 0x7fdc500b4780 2024-04-24T08:11:23.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.832+0000 7fdc56a00640 1 -- 172.21.15.69:0/1329414685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc500aeac0 con 0x7fdc500b4780 2024-04-24T08:11:23.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.832+0000 7fdc56a00640 1 -- 172.21.15.69:0/1329414685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc500af080 con 0x7fdc500b4780 2024-04-24T08:11:23.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.833+0000 7fdc56a00640 1 -- 172.21.15.69:0/1329414685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc50068a80 con 0x7fdc500b4780 2024-04-24T08:11:23.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.834+0000 7fdc4cc00640 1 -- 172.21.15.69:0/1329414685 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdc38044020 con 0x7fdc500b4780 2024-04-24T08:11:23.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.835+0000 7fdc4cc00640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc2c06c8f0 0x7fdc2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:23.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.835+0000 7fdc4cc00640 1 -- 172.21.15.69:0/1329414685 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fdc380ac380 con 0x7fdc500b4780 2024-04-24T08:11:23.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.836+0000 7fdc4fe00640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc2c06c8f0 0x7fdc2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:23.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.837+0000 7fdc4fe00640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc2c06c8f0 0x7fdc2c06edb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdc400098f0 tx=0x7fdc40008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:23.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:23.842+0000 7fdc4cc00640 1 -- 172.21.15.69:0/1329414685 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc3804b030 con 0x7fdc500b4780 2024-04-24T08:11:24.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.038+0000 7fdc56a00640 1 -- 172.21.15.69:0/1329414685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdc50072190 con 0x7fdc500b4780 2024-04-24T08:11:24.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.039+0000 7fdc4cc00640 1 -- 172.21.15.69:0/1329414685 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v46) ==== 104+0+11 (secure 0 0 0) 0x7fdc3807a800 con 0x7fdc500b4780 2024-04-24T08:11:24.041 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:11:24.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 -- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc2c06c8f0 msgr2=0x7fdc2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc2c06c8f0 0x7fdc2c06edb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdc400098f0 tx=0x7fdc40008040 comp rx=0 tx=0).stop 2024-04-24T08:11:24.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 -- 172.21.15.69:0/1329414685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500b4780 msgr2=0x7fdc50165190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500b4780 0x7fdc50165190 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fdc3802a270 tx=0x7fdc380046c0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 -- 172.21.15.69:0/1329414685 shutdown_connections 2024-04-24T08:11:24.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc2c06c8f0 0x7fdc2c06edb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc500b4780 0x7fdc50165190 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc500cccc0 0x7fdc501f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 --2- 172.21.15.69:0/1329414685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc50073db0 0x7fdc5015ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 -- 172.21.15.69:0/1329414685 >> 172.21.15.69:0/1329414685 conn(0x7fdc500d0350 msgr2=0x7fdc50072620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:24.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 -- 172.21.15.69:0/1329414685 shutdown_connections 2024-04-24T08:11:24.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.042+0000 7fdc56a00640 1 -- 172.21.15.69:0/1329414685 wait complete. 2024-04-24T08:11:24.060 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:11:24.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.250+0000 7fca3f200640 1 Processor -- start 2024-04-24T08:11:24.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.250+0000 7fca3f200640 1 -- start start 2024-04-24T08:11:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.252+0000 7fca3f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fca3815f4b0 0x7fca380b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.252+0000 7fca3f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca380b46a0 0x7fca380b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.252+0000 7fca3f200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca380739f0 con 0x7fca3815f890 2024-04-24T08:11:24.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.252+0000 7fca3f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fca380b5010 con 0x7fca3815f4b0 2024-04-24T08:11:24.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.252+0000 7fca3f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fca38073cc0 con 0x7fca380b46a0 2024-04-24T08:11:24.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.252+0000 7fca37e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fca3815f4b0 0x7fca380b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.253+0000 7fca37e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fca3815f4b0 0x7fca380b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45062/0 (socket says 172.21.15.69:45062) 2024-04-24T08:11:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.253+0000 7fca37e00640 1 -- 172.21.15.69:0/1715876512 learned_addr learned my addr 172.21.15.69:0/1715876512 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.253+0000 7fca3d200640 1 --2- 172.21.15.69:0/1715876512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca380b46a0 0x7fca380b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.253+0000 7fca37e00640 1 -- 172.21.15.69:0/1715876512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca380b46a0 msgr2=0x7fca380b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.253+0000 7fca37e00640 1 --2- 172.21.15.69:0/1715876512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca380b46a0 0x7fca380b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.253+0000 7fca37e00640 1 -- 172.21.15.69:0/1715876512 >> v1:172.21.15.121:6789/0 conn(0x7fca3815f890 legacy=0x7fca380732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.253+0000 7fca37e00640 1 -- 172.21.15.69:0/1715876512 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca38073fa0 con 0x7fca3815f4b0 2024-04-24T08:11:24.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.254+0000 7fca37e00640 1 --2- 172.21.15.69:0/1715876512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca3815f4b0 0x7fca380b3de0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fca28009870 tx=0x7fca2802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a94b80d095394c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.255+0000 7fca37400640 1 -- 172.21.15.69:0/1715876512 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28037070 con 0x7fca3815f4b0 2024-04-24T08:11:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.255+0000 7fca37400640 1 -- 172.21.15.69:0/1715876512 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca28032440 con 0x7fca3815f4b0 2024-04-24T08:11:24.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.255+0000 7fca37400640 1 -- 172.21.15.69:0/1715876512 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28032810 con 0x7fca3815f4b0 2024-04-24T08:11:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.255+0000 7fca3f200640 1 -- 172.21.15.69:0/1715876512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca3815f4b0 msgr2=0x7fca380b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.255+0000 7fca3f200640 1 --2- 172.21.15.69:0/1715876512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca3815f4b0 0x7fca380b3de0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fca28009870 tx=0x7fca2802a270 comp rx=0 tx=0).stop 2024-04-24T08:11:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.256+0000 7fca3f200640 1 -- 172.21.15.69:0/1715876512 shutdown_connections 2024-04-24T08:11:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.256+0000 7fca3f200640 1 --2- 172.21.15.69:0/1715876512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca380b46a0 0x7fca380b4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.256+0000 7fca3f200640 1 --2- 172.21.15.69:0/1715876512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca3815f4b0 0x7fca380b3de0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.256+0000 7fca3f200640 1 -- 172.21.15.69:0/1715876512 >> 172.21.15.69:0/1715876512 conn(0x7fca380d0350 msgr2=0x7fca380d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:24.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.256+0000 7fca3f200640 1 -- 172.21.15.69:0/1715876512 shutdown_connections 2024-04-24T08:11:24.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.256+0000 7fca3f200640 1 -- 172.21.15.69:0/1715876512 wait complete. 2024-04-24T08:11:24.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.257+0000 7fca3f200640 1 Processor -- start 2024-04-24T08:11:24.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.257+0000 7fca3f200640 1 -- start start 2024-04-24T08:11:24.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.258+0000 7fca3f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca380b46a0 0x7fca38149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.258+0000 7fca3f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca3815f4b0 0x7fca3806f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.258+0000 7fca3f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3815f890 0x7fca380ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.258+0000 7fca3f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fca380b38c0 con 0x7fca380b46a0 2024-04-24T08:11:24.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.258+0000 7fca3f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fca380b35e0 con 0x7fca3815f890 2024-04-24T08:11:24.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.258+0000 7fca3f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fca3815fd40 con 0x7fca3815f4b0 2024-04-24T08:11:24.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca37e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca380b46a0 0x7fca38149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:24.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca37e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca380b46a0 0x7fca38149650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45064/0 (socket says 172.21.15.69:45064) 2024-04-24T08:11:24.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca37e00640 1 -- 172.21.15.69:0/2545711210 learned_addr learned my addr 172.21.15.69:0/2545711210 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:24.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca37e00640 1 -- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca3815f4b0 msgr2=0x7fca3806f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca37e00640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca3815f4b0 0x7fca3806f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca37e00640 1 -- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3815f890 msgr2=0x7fca380ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca37e00640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3815f890 0x7fca380ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca37e00640 1 -- 172.21.15.69:0/2545711210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3806ddb0 con 0x7fca380b46a0 2024-04-24T08:11:24.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca37e00640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca380b46a0 0x7fca38149650 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fca2802a2b0 tx=0x7fca280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:24.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca34c00640 1 -- 172.21.15.69:0/2545711210 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28041020 con 0x7fca380b46a0 2024-04-24T08:11:24.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca34c00640 1 -- 172.21.15.69:0/2545711210 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca280306c0 con 0x7fca380b46a0 2024-04-24T08:11:24.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.259+0000 7fca34c00640 1 -- 172.21.15.69:0/2545711210 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28037040 con 0x7fca380b46a0 2024-04-24T08:11:24.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.260+0000 7fca3f200640 1 -- 172.21.15.69:0/2545711210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3806e090 con 0x7fca380b46a0 2024-04-24T08:11:24.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.260+0000 7fca3f200640 1 -- 172.21.15.69:0/2545711210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca3806e650 con 0x7fca380b46a0 2024-04-24T08:11:24.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.261+0000 7fca3f200640 1 -- 172.21.15.69:0/2545711210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca38068a80 con 0x7fca380b46a0 2024-04-24T08:11:24.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.262+0000 7fca34c00640 1 -- 172.21.15.69:0/2545711210 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fca28044020 con 0x7fca380b46a0 2024-04-24T08:11:24.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.263+0000 7fca34c00640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca1406c920 0x7fca1406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.263+0000 7fca34c00640 1 -- 172.21.15.69:0/2545711210 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fca280ac4a0 con 0x7fca380b46a0 2024-04-24T08:11:24.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.264+0000 7fca2fe00640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca1406c920 0x7fca1406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:24.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.265+0000 7fca2fe00640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca1406c920 0x7fca1406ede0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca3809b7a0 tx=0x7fca20007970 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:24.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.268+0000 7fca34c00640 1 -- 172.21.15.69:0/2545711210 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca2804b030 con 0x7fca380b46a0 2024-04-24T08:11:24.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.466+0000 7fca3f200640 1 -- 172.21.15.69:0/2545711210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fca380af470 con 0x7fca380b46a0 2024-04-24T08:11:24.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.466+0000 7fca34c00640 1 -- 172.21.15.69:0/2545711210 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v46) ==== 100+0+11 (secure 0 0 0) 0x7fca2807a760 con 0x7fca380b46a0 2024-04-24T08:11:24.468 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 -- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca1406c920 msgr2=0x7fca1406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca1406c920 0x7fca1406ede0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca3809b7a0 tx=0x7fca20007970 comp rx=0 tx=0).stop 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 -- 172.21.15.69:0/2545711210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca380b46a0 msgr2=0x7fca38149650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca380b46a0 0x7fca38149650 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fca2802a2b0 tx=0x7fca280046c0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 -- 172.21.15.69:0/2545711210 shutdown_connections 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca1406c920 0x7fca1406ede0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fca3815f890 0x7fca380ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fca3815f4b0 0x7fca3806f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 --2- 172.21.15.69:0/2545711210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fca380b46a0 0x7fca38149650 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 -- 172.21.15.69:0/2545711210 >> 172.21.15.69:0/2545711210 conn(0x7fca380d0350 msgr2=0x7fca38161a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 -- 172.21.15.69:0/2545711210 shutdown_connections 2024-04-24T08:11:24.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.469+0000 7fca3f200640 1 -- 172.21.15.69:0/2545711210 wait complete. 2024-04-24T08:11:24.490 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:24.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.688+0000 7f414dc00640 1 Processor -- start 2024-04-24T08:11:24.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.688+0000 7f414dc00640 1 -- start start 2024-04-24T08:11:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.690+0000 7f414dc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4148162300 0x7f4148143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.690+0000 7f414dc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4148072230 0x7f4148143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.690+0000 7f414dc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f414815f540 con 0x7f4148071e50 2024-04-24T08:11:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.690+0000 7f414dc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f414815f820 con 0x7f4148072230 2024-04-24T08:11:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.690+0000 7f414dc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f414815fb00 con 0x7f4148162300 2024-04-24T08:11:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.690+0000 7f414d200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4148072230 0x7f4148143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.690+0000 7f414d200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4148072230 0x7f4148143bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45072/0 (socket says 172.21.15.69:45072) 2024-04-24T08:11:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.690+0000 7f414d200640 1 -- 172.21.15.69:0/2663316332 learned_addr learned my addr 172.21.15.69:0/2663316332 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.690+0000 7f4147400640 1 --2- 172.21.15.69:0/2663316332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4148162300 0x7f4148143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:24.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.691+0000 7f414d200640 1 -- 172.21.15.69:0/2663316332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4148162300 msgr2=0x7f4148143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.691+0000 7f414d200640 1 --2- 172.21.15.69:0/2663316332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4148162300 0x7f4148143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.691+0000 7f414d200640 1 -- 172.21.15.69:0/2663316332 >> v1:172.21.15.121:6789/0 conn(0x7f4148071e50 legacy=0x7f414815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.691+0000 7f414d200640 1 -- 172.21.15.69:0/2663316332 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414806dda0 con 0x7f4148072230 2024-04-24T08:11:24.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.691+0000 7f4147400640 1 --2- 172.21.15.69:0/2663316332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4148162300 0x7f4148143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:24.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.692+0000 7f414d200640 1 --2- 172.21.15.69:0/2663316332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148072230 0x7f4148143bd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4140009e50 tx=0x7f414002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f013bb1b822413 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:24.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.692+0000 7f4146a00640 1 -- 172.21.15.69:0/2663316332 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4140036070 con 0x7f4148072230 2024-04-24T08:11:24.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.692+0000 7f4146a00640 1 -- 172.21.15.69:0/2663316332 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4140031600 con 0x7f4148072230 2024-04-24T08:11:24.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.692+0000 7f4146a00640 1 -- 172.21.15.69:0/2663316332 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41400319d0 con 0x7f4148072230 2024-04-24T08:11:24.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.692+0000 7f414dc00640 1 -- 172.21.15.69:0/2663316332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148072230 msgr2=0x7f4148143bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.692+0000 7f414dc00640 1 --2- 172.21.15.69:0/2663316332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148072230 0x7f4148143bd0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f4140009e50 tx=0x7f414002a6c0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.693+0000 7f414dc00640 1 -- 172.21.15.69:0/2663316332 shutdown_connections 2024-04-24T08:11:24.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.693+0000 7f414dc00640 1 --2- 172.21.15.69:0/2663316332 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148072230 0x7f4148143bd0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.693+0000 7f414dc00640 1 --2- 172.21.15.69:0/2663316332 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4148162300 0x7f4148143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.693+0000 7f414dc00640 1 -- 172.21.15.69:0/2663316332 >> 172.21.15.69:0/2663316332 conn(0x7f41480d0160 msgr2=0x7f41480d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:24.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.693+0000 7f414dc00640 1 -- 172.21.15.69:0/2663316332 shutdown_connections 2024-04-24T08:11:24.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.693+0000 7f414dc00640 1 -- 172.21.15.69:0/2663316332 wait complete. 2024-04-24T08:11:24.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.694+0000 7f414dc00640 1 Processor -- start 2024-04-24T08:11:24.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.694+0000 7f414dc00640 1 -- start start 2024-04-24T08:11:24.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.695+0000 7f414dc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4148071e50 0x7f41481e8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.695+0000 7f414dc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4148072230 0x7f4148099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.695+0000 7f414dc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148162300 0x7f4148161300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.695+0000 7f414dc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4148144290 con 0x7f4148162300 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.695+0000 7f414dc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4148144110 con 0x7f4148072230 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.695+0000 7f414dc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4148144410 con 0x7f4148071e50 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.696+0000 7f414d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148162300 0x7f4148161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.696+0000 7f414d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148162300 0x7f4148161300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45074/0 (socket says 172.21.15.69:45074) 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.696+0000 7f414d200640 1 -- 172.21.15.69:0/173710428 learned_addr learned my addr 172.21.15.69:0/173710428 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.696+0000 7f4147400640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4148072230 0x7f4148099920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.696+0000 7f414d200640 1 -- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4148071e50 msgr2=0x7f41481e8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.696+0000 7f414d200640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4148071e50 0x7f41481e8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.696+0000 7f414d200640 1 -- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4148072230 msgr2=0x7f4148099920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.696+0000 7f414d200640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4148072230 0x7f4148099920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.696+0000 7f414d200640 1 -- 172.21.15.69:0/173710428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4148099e60 con 0x7f4148162300 2024-04-24T08:11:24.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.696+0000 7f4147400640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4148072230 0x7f4148099920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:24.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.697+0000 7f414d200640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148162300 0x7f4148161300 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4140009e50 tx=0x7f4140004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:24.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.697+0000 7f413fe00640 1 -- 172.21.15.69:0/173710428 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4140036050 con 0x7f4148162300 2024-04-24T08:11:24.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.697+0000 7f413fe00640 1 -- 172.21.15.69:0/173710428 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4140042070 con 0x7f4148162300 2024-04-24T08:11:24.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.697+0000 7f413fe00640 1 -- 172.21.15.69:0/173710428 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4140009510 con 0x7f4148162300 2024-04-24T08:11:24.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.697+0000 7f414dc00640 1 -- 172.21.15.69:0/173710428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41481f4a30 con 0x7f4148162300 2024-04-24T08:11:24.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.697+0000 7f414dc00640 1 -- 172.21.15.69:0/173710428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4148161840 con 0x7f4148162300 2024-04-24T08:11:24.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.698+0000 7f414dc00640 1 -- 172.21.15.69:0/173710428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4148075d50 con 0x7f4148162300 2024-04-24T08:11:24.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.699+0000 7f413fe00640 1 -- 172.21.15.69:0/173710428 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4140007bd0 con 0x7f4148162300 2024-04-24T08:11:24.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.700+0000 7f413fe00640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f412806c8f0 0x7f412806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.700+0000 7f413fe00640 1 -- 172.21.15.69:0/173710428 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f41400acbc0 con 0x7f4148162300 2024-04-24T08:11:24.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.701+0000 7f4147e00640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f412806c8f0 0x7f412806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:24.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.702+0000 7f4147e00640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f412806c8f0 0x7f412806edb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4138003cc0 tx=0x7f41380079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:24.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.705+0000 7f413fe00640 1 -- 172.21.15.69:0/173710428 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4148099e60 con 0x7f4148162300 2024-04-24T08:11:24.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.947+0000 7f414dc00640 1 -- 172.21.15.69:0/173710428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41480d0d90 con 0x7f4148162300 2024-04-24T08:11:24.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.948+0000 7f413fe00640 1 -- 172.21.15.69:0/173710428 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5184 (secure 0 0 0) 0x7f414007af10 con 0x7f4148162300 2024-04-24T08:11:24.950 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:24.950 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/376722301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":376722301},{"type":"v1","addr":"172.21.15.69:6835","nonce":376722301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4990,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1327624292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1327624292},{"type":"v1","addr":"172.21.15.121:6836","nonce":1327624292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/3244002416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3244002416},{"type":"v1","addr":"172.21.15.69:6834","nonce":3244002416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:10:35.400788+0000","modified":"2024-04-24T08:10:38.457641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5793},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5793":{"gid":5793,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/3424934990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3424934990},{"type":"v1","addr":"172.21.15.121:6837","nonce":3424934990}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5793,"qdb_cluster":[5793]},"id":3}]} 2024-04-24T08:11:24.950 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-04-24T08:11:24.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 -- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f412806c8f0 msgr2=0x7f412806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f412806c8f0 0x7f412806edb0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4138003cc0 tx=0x7f41380079b0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 -- 172.21.15.69:0/173710428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148162300 msgr2=0x7f4148161300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:24.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148162300 0x7f4148161300 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4140009e50 tx=0x7f4140004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:24.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 -- 172.21.15.69:0/173710428 shutdown_connections 2024-04-24T08:11:24.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f412806c8f0 0x7f412806edb0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4148162300 0x7f4148161300 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4148072230 0x7f4148099920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 --2- 172.21.15.69:0/173710428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4148071e50 0x7f41481e8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:24.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 -- 172.21.15.69:0/173710428 >> 172.21.15.69:0/173710428 conn(0x7f41480d0160 msgr2=0x7f41480b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:24.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 -- 172.21.15.69:0/173710428 shutdown_connections 2024-04-24T08:11:24.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:24.951+0000 7f414dc00640 1 -- 172.21.15.69:0/173710428 wait complete. 2024-04-24T08:11:24.971 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:11:25.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.161+0000 7f461e600640 1 Processor -- start 2024-04-24T08:11:25.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.161+0000 7f461e600640 1 -- start start 2024-04-24T08:11:25.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.163+0000 7f461e600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f461815e960 0x7f4618162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.163+0000 7f461e600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461815ed40 con 0x7f46180cfaf0 2024-04-24T08:11:25.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.163+0000 7f461e600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46180cf050 con 0x7f4618162980 2024-04-24T08:11:25.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.163+0000 7f461e600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4618162e20 con 0x7f461815e960 2024-04-24T08:11:25.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.163+0000 7f461dc00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f46180cfaf0 0x7f46180d2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33952/0 (socket says 172.21.15.69:33952) 2024-04-24T08:11:25.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.163+0000 7f461dc00640 1 -- 172.21.15.69:0/693079405 learned_addr learned my addr 172.21.15.69:0/693079405 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.163+0000 7f461d200640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f461815e960 0x7f4618162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.163+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1947407273 0 0) 0x7f461815ed40 con 0x7f46180cfaf0 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4600003520 con 0x7f46180cfaf0 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4183103867 0 0) 0x7f4600003520 con 0x7f46180cfaf0 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f461815e960 msgr2=0x7f4618162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f461815e960 0x7f4618162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 >> v1:172.21.15.121:6789/0 conn(0x7f4618162980 legacy=0x7f461806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4618163100 con 0x7f46180cfaf0 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f46100019f0 con 0x7f46180cfaf0 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 >> v1:172.21.15.69:6789/0 conn(0x7f46180cfaf0 legacy=0x7f46180d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000039e0 0x7f4600003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f461815e960 0x7f4600004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4600004d40 0x7f46000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4618162e20 con 0x7f46000039e0 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f460000baa0 con 0x7f4600004d40 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f460000bca0 con 0x7f461815e960 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.164+0000 7f461d200640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000039e0 0x7f4600003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.165+0000 7f461dc00640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4600004d40 0x7f46000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.165+0000 7f461d200640 1 -- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f461815e960 msgr2=0x7f4600004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.165+0000 7f461d200640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f461815e960 0x7f4600004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.165+0000 7f461d200640 1 -- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4600004d40 msgr2=0x7f46000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.165+0000 7f461d200640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4600004d40 0x7f46000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.165+0000 7f461d200640 1 -- 172.21.15.69:0/693079405 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4600003520 con 0x7f46000039e0 2024-04-24T08:11:25.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.165+0000 7f461dc00640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4600004d40 0x7f46000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.166+0000 7f461d200640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000039e0 0x7f4600003e00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f460c009a90 tx=0x7f4600005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999f2d01dc0d71b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.166+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c036070 con 0x7f46000039e0 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.166+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f460c031400 con 0x7f46000039e0 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.166+0000 7f4617400640 1 -- 172.21.15.69:0/693079405 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c009690 con 0x7f46000039e0 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.167+0000 7f461e600640 1 -- 172.21.15.69:0/693079405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000039e0 msgr2=0x7f4600003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.167+0000 7f461e600640 1 --2- 172.21.15.69:0/693079405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46000039e0 0x7f4600003e00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f460c009a90 tx=0x7f4600005830 comp rx=0 tx=0).stop 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.167+0000 7f461dc00640 1 -- 172.21.15.69:0/693079405 reap_dead start 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.167+0000 7f461e600640 1 -- 172.21.15.69:0/693079405 shutdown_connections 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.167+0000 7f461e600640 1 -- 172.21.15.69:0/693079405 >> 172.21.15.69:0/693079405 conn(0x7f461809d900 msgr2=0x7f461808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.167+0000 7f461e600640 1 -- 172.21.15.69:0/693079405 shutdown_connections 2024-04-24T08:11:25.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.167+0000 7f461e600640 1 -- 172.21.15.69:0/693079405 wait complete. 2024-04-24T08:11:25.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.169+0000 7f461e600640 1 Processor -- start 2024-04-24T08:11:25.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.169+0000 7f461e600640 1 -- start start 2024-04-24T08:11:25.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.170+0000 7f461e600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000039e0 0x7f46181528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.170+0000 7f461e600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461815fb10 0x7f46180b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.170+0000 7f461e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f461815ff70 0x7f46180b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.170+0000 7f461e600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f461806d6c0 con 0x7f461815ff70 2024-04-24T08:11:25.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.170+0000 7f461e600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f46180cf5d0 con 0x7f461815fb10 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.170+0000 7f461e600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f46180b6170 con 0x7f46000039e0 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.170+0000 7f461dc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f461815ff70 0x7f46180b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.170+0000 7f461dc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f461815ff70 0x7f46180b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45096/0 (socket says 172.21.15.69:45096) 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.170+0000 7f461dc00640 1 -- 172.21.15.69:0/4258773387 learned_addr learned my addr 172.21.15.69:0/4258773387 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f461d200640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000039e0 0x7f46181528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f4617e00640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461815fb10 0x7f46180b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f461dc00640 1 -- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000039e0 msgr2=0x7f46181528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f461dc00640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000039e0 0x7f46181528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f461dc00640 1 -- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461815fb10 msgr2=0x7f46180b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f461dc00640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461815fb10 0x7f46180b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f461dc00640 1 -- 172.21.15.69:0/4258773387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46180b7370 con 0x7f461815ff70 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f461d200640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000039e0 0x7f46181528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:25.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f461dc00640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f461815ff70 0x7f46180b26f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f46180c7d50 tx=0x7f461000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:25.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f4614c00640 1 -- 172.21.15.69:0/4258773387 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461000bb20 con 0x7f461815ff70 2024-04-24T08:11:25.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f4614c00640 1 -- 172.21.15.69:0/4258773387 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f461000bcc0 con 0x7f461815ff70 2024-04-24T08:11:25.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f4614c00640 1 -- 172.21.15.69:0/4258773387 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610007290 con 0x7f461815ff70 2024-04-24T08:11:25.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.171+0000 7f461e600640 1 -- 172.21.15.69:0/4258773387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46180b7650 con 0x7f461815ff70 2024-04-24T08:11:25.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.172+0000 7f461e600640 1 -- 172.21.15.69:0/4258773387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46180b7c10 con 0x7f461815ff70 2024-04-24T08:11:25.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.173+0000 7f461e600640 1 -- 172.21.15.69:0/4258773387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4618075d50 con 0x7f461815ff70 2024-04-24T08:11:25.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.173+0000 7f4614c00640 1 -- 172.21.15.69:0/4258773387 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f461001b020 con 0x7f461815ff70 2024-04-24T08:11:25.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.175+0000 7f4614c00640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45f806c920 0x7f45f806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.175+0000 7f4614c00640 1 -- 172.21.15.69:0/4258773387 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f461008bce0 con 0x7f461815ff70 2024-04-24T08:11:25.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.176+0000 7f461d200640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45f806c920 0x7f45f806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.177+0000 7f461d200640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45f806c920 0x7f45f806ede0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f460c033000 tx=0x7f460c008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:25.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.179+0000 7f4614c00640 1 -- 172.21.15.69:0/4258773387 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f461005a030 con 0x7f461815ff70 2024-04-24T08:11:25.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.366+0000 7f461e600640 1 -- 172.21.15.69:0/4258773387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46180d1dc0 con 0x7f461815ff70 2024-04-24T08:11:25.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.369+0000 7f4614c00640 1 -- 172.21.15.69:0/4258773387 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+16740 (secure 0 0 0) 0x7f4610059930 con 0x7f461815ff70 2024-04-24T08:11:25.370 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:25.371 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":46,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:03.085538+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:10:33.100132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-24T08:10:34.379302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":5}]},{"pgid":"7.36","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:11:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 -- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45f806c920 msgr2=0x7f45f806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45f806c920 0x7f45f806ede0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f460c033000 tx=0x7f460c008030 comp rx=0 tx=0).stop 2024-04-24T08:11:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 -- 172.21.15.69:0/4258773387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f461815ff70 msgr2=0x7f46180b26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f461815ff70 0x7f46180b26f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f46180c7d50 tx=0x7f461000b4a0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 -- 172.21.15.69:0/4258773387 shutdown_connections 2024-04-24T08:11:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f45f806c920 0x7f45f806ede0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f461815ff70 0x7f46180b26f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f461815fb10 0x7f46180b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 --2- 172.21.15.69:0/4258773387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f46000039e0 0x7f46181528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 -- 172.21.15.69:0/4258773387 >> 172.21.15.69:0/4258773387 conn(0x7f461809d900 msgr2=0x7f46180b49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:25.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 -- 172.21.15.69:0/4258773387 shutdown_connections 2024-04-24T08:11:25.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.371+0000 7f461e600640 1 -- 172.21.15.69:0/4258773387 wait complete. 2024-04-24T08:11:25.390 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:11:25.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:25.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.587+0000 7f1ae6800640 1 Processor -- start 2024-04-24T08:11:25.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.587+0000 7f1ae6800640 1 -- start start 2024-04-24T08:11:25.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.589+0000 7f1ae6800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1ae00cdbd0 0x7f1ae009b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.589+0000 7f1ae6800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae00ce150 con 0x7f1ae015ca20 2024-04-24T08:11:25.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.589+0000 7f1ae6800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae00ce430 con 0x7f1ae00b14b0 2024-04-24T08:11:25.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.589+0000 7f1ae6800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae015ce00 con 0x7f1ae00cdbd0 2024-04-24T08:11:25.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.589+0000 7f1adf400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1ae015ca20 0x7f1ae00b1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33962/0 (socket says 172.21.15.69:33962) 2024-04-24T08:11:25.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.589+0000 7f1adf400640 1 -- 172.21.15.69:0/4181814356 learned_addr learned my addr 172.21.15.69:0/4181814356 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:25.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1ad7e00640 1 --2- 172.21.15.69:0/4181814356 >> v2:172.21.15.121:3300/0 conn(0x7f1ae00cdbd0 0x7f1ae009b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1383728586 0 0) 0x7f1ae00ce150 con 0x7f1ae015ca20 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ac4003520 con 0x7f1ae015ca20 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 810561440 0 0) 0x7f1ac4003520 con 0x7f1ae015ca20 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 >> v2:172.21.15.121:3300/0 conn(0x7f1ae00cdbd0 msgr2=0x7f1ae009b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 --2- 172.21.15.69:0/4181814356 >> v2:172.21.15.121:3300/0 conn(0x7f1ae00cdbd0 0x7f1ae009b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 >> v1:172.21.15.121:6789/0 conn(0x7f1ae00b14b0 legacy=0x7f1ae00d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae009aa70 con 0x7f1ae015ca20 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f1ad0001670 con 0x7f1ae015ca20 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 >> v1:172.21.15.69:6789/0 conn(0x7f1ae015ca20 legacy=0x7f1ae00b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 --2- 172.21.15.69:0/4181814356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac40039b0 0x7f1ac4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 --2- 172.21.15.69:0/4181814356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ac4004800 0x7f1ac4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 --2- 172.21.15.69:0/4181814356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ac40051c0 0x7f1ac400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae015ce00 con 0x7f1ac4004800 2024-04-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac400c060 con 0x7f1ac40039b0 2024-04-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1ac400c260 con 0x7f1ac40051c0 2024-04-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.590+0000 7f1ad7e00640 1 --2- 172.21.15.69:0/4181814356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ac4004800 0x7f1ac4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.591+0000 7f1ad7e00640 1 -- 172.21.15.69:0/4181814356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ac40051c0 msgr2=0x7f1ac400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.591+0000 7f1ad7e00640 1 --2- 172.21.15.69:0/4181814356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ac40051c0 0x7f1ac400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.591+0000 7f1ad7e00640 1 -- 172.21.15.69:0/4181814356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac40039b0 msgr2=0x7f1ac4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.591+0000 7f1ad7e00640 1 --2- 172.21.15.69:0/4181814356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac40039b0 0x7f1ac4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.591+0000 7f1ad7e00640 1 -- 172.21.15.69:0/4181814356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac4003520 con 0x7f1ac4004800 2024-04-24T08:11:25.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.591+0000 7f1adfe00640 1 -- 172.21.15.69:0/4181814356 reap_dead start 2024-04-24T08:11:25.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.592+0000 7f1ad7e00640 1 --2- 172.21.15.69:0/4181814356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ac4004800 0x7f1ac4004c80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009ac0 tx=0x7f1ac4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2510ee9a629734 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:25.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.592+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8036070 con 0x7f1ac4004800 2024-04-24T08:11:25.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.592+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ac8031400 con 0x7f1ac4004800 2024-04-24T08:11:25.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.592+0000 7f1adea00640 1 -- 172.21.15.69:0/4181814356 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8009690 con 0x7f1ac4004800 2024-04-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.593+0000 7f1ae6800640 1 -- 172.21.15.69:0/4181814356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ac4004800 msgr2=0x7f1ac4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.593+0000 7f1ae6800640 1 --2- 172.21.15.69:0/4181814356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ac4004800 0x7f1ac4004c80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009ac0 tx=0x7f1ac4005830 comp rx=0 tx=0).stop 2024-04-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.593+0000 7f1ae6800640 1 -- 172.21.15.69:0/4181814356 shutdown_connections 2024-04-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.593+0000 7f1ae6800640 1 --2- 172.21.15.69:0/4181814356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ac4004800 0x7f1ac4004c80 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.593+0000 7f1ae6800640 1 -- 172.21.15.69:0/4181814356 >> 172.21.15.69:0/4181814356 conn(0x7f1ae00b4040 msgr2=0x7f1ae00b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.593+0000 7f1ae6800640 1 -- 172.21.15.69:0/4181814356 shutdown_connections 2024-04-24T08:11:25.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.593+0000 7f1ae6800640 1 -- 172.21.15.69:0/4181814356 wait complete. 2024-04-24T08:11:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.594+0000 7f1ae6800640 1 Processor -- start 2024-04-24T08:11:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.595+0000 7f1ae6800640 1 -- start start 2024-04-24T08:11:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.595+0000 7f1ae6800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac4004800 0x7f1ae0150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.595+0000 7f1ae6800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ae00b5170 0x7f1ae00b55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.595+0000 7f1ae6800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae006d500 0x7f1ae006d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.595+0000 7f1ae6800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae00d0510 con 0x7f1ae006d500 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.595+0000 7f1ae6800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae009b030 con 0x7f1ac4004800 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.595+0000 7f1ae6800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae00d0690 con 0x7f1ae00b5170 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.596+0000 7f1adfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae006d500 0x7f1ae006d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.596+0000 7f1adfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae006d500 0x7f1ae006d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45116/0 (socket says 172.21.15.69:45116) 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.596+0000 7f1adfe00640 1 -- 172.21.15.69:0/3775129184 learned_addr learned my addr 172.21.15.69:0/3775129184 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.596+0000 7f1adf400640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac4004800 0x7f1ae0150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.596+0000 7f1ad7e00640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ae00b5170 0x7f1ae00b55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.596+0000 7f1adfe00640 1 -- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ae00b5170 msgr2=0x7f1ae00b55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.596+0000 7f1adfe00640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ae00b5170 0x7f1ae00b55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.596+0000 7f1adfe00640 1 -- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac4004800 msgr2=0x7f1ae0150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.596+0000 7f1adfe00640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac4004800 0x7f1ae0150d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.596+0000 7f1adfe00640 1 -- 172.21.15.69:0/3775129184 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae0151280 con 0x7f1ae006d500 2024-04-24T08:11:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.597+0000 7f1adfe00640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae006d500 0x7f1ae006d980 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0074200 tx=0x7f1ad800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.597+0000 7f1ad7400640 1 -- 172.21.15.69:0/3775129184 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8017070 con 0x7f1ae006d500 2024-04-24T08:11:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.597+0000 7f1ad7400640 1 -- 172.21.15.69:0/3775129184 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad8007070 con 0x7f1ae006d500 2024-04-24T08:11:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.597+0000 7f1ae6800640 1 -- 172.21.15.69:0/3775129184 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae006e2d0 con 0x7f1ae006d500 2024-04-24T08:11:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.597+0000 7f1ad7400640 1 -- 172.21.15.69:0/3775129184 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8007460 con 0x7f1ae006d500 2024-04-24T08:11:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.597+0000 7f1ae6800640 1 -- 172.21.15.69:0/3775129184 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae00b5fc0 con 0x7f1ae006d500 2024-04-24T08:11:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.598+0000 7f1ae6800640 1 -- 172.21.15.69:0/3775129184 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ae0075520 con 0x7f1ae006d500 2024-04-24T08:11:25.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.599+0000 7f1ad7400640 1 -- 172.21.15.69:0/3775129184 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1ad801a070 con 0x7f1ae006d500 2024-04-24T08:11:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.600+0000 7f1ad7400640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1abc06c920 0x7f1abc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.600+0000 7f1ad7400640 1 -- 172.21.15.69:0/3775129184 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f1ad808ad70 con 0x7f1ae006d500 2024-04-24T08:11:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.601+0000 7f1adf400640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1abc06c920 0x7f1abc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:25.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.602+0000 7f1adf400640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1abc06c920 0x7f1abc06ede0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0000f60 tx=0x7f1ad0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:25.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.605+0000 7f1ad7400640 1 -- 172.21.15.69:0/3775129184 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ad8059150 con 0x7f1ae006d500 2024-04-24T08:11:25.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.851+0000 7f1ae6800640 1 -- 172.21.15.69:0/3775129184 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ae0068a80 con 0x7f1ae006d500 2024-04-24T08:11:25.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.852+0000 7f1ad7400640 1 -- 172.21.15.69:0/3775129184 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5184 (secure 0 0 0) 0x7f1ad8058a50 con 0x7f1ae006d500 2024-04-24T08:11:25.854 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:25.855 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/376722301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":376722301},{"type":"v1","addr":"172.21.15.69:6835","nonce":376722301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4990,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1327624292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1327624292},{"type":"v1","addr":"172.21.15.121:6836","nonce":1327624292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/3244002416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3244002416},{"type":"v1","addr":"172.21.15.69:6834","nonce":3244002416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:10:35.400788+0000","modified":"2024-04-24T08:10:38.457641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5793},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5793":{"gid":5793,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/3424934990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3424934990},{"type":"v1","addr":"172.21.15.121:6837","nonce":3424934990}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5793,"qdb_cluster":[5793]},"id":3}]} 2024-04-24T08:11:25.855 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-04-24T08:11:25.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 -- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1abc06c920 msgr2=0x7f1abc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1abc06c920 0x7f1abc06ede0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0000f60 tx=0x7f1ad0009040 comp rx=0 tx=0).stop 2024-04-24T08:11:25.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 -- 172.21.15.69:0/3775129184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae006d500 msgr2=0x7f1ae006d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:25.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae006d500 0x7f1ae006d980 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0074200 tx=0x7f1ad800c980 comp rx=0 tx=0).stop 2024-04-24T08:11:25.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 -- 172.21.15.69:0/3775129184 shutdown_connections 2024-04-24T08:11:25.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1abc06c920 0x7f1abc06ede0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ae006d500 0x7f1ae006d980 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ae00b5170 0x7f1ae00b55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 --2- 172.21.15.69:0/3775129184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ac4004800 0x7f1ae0150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:25.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 -- 172.21.15.69:0/3775129184 >> 172.21.15.69:0/3775129184 conn(0x7f1ae00b4040 msgr2=0x7f1ae00affa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:25.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 -- 172.21.15.69:0/3775129184 shutdown_connections 2024-04-24T08:11:25.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:25.856+0000 7f1ae6800640 1 -- 172.21.15.69:0/3775129184 wait complete. 2024-04-24T08:11:25.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:26.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.072+0000 7fbe40c00640 1 Processor -- start 2024-04-24T08:11:26.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.072+0000 7fbe40c00640 1 -- start start 2024-04-24T08:11:26.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.074+0000 7fbe40c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbe3c0cdbd0 0x7fbe3c09b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.074+0000 7fbe40c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe3c0ce150 con 0x7fbe3c0b14b0 2024-04-24T08:11:26.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.074+0000 7fbe40c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe3c0ce430 con 0x7fbe3c15ca20 2024-04-24T08:11:26.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.074+0000 7fbe40c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fbe3c15ce00 con 0x7fbe3c0cdbd0 2024-04-24T08:11:26.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.074+0000 7fbe3a800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbe3c0b14b0 0x7fbe3c0d0b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33976/0 (socket says 172.21.15.69:33976) 2024-04-24T08:11:26.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.074+0000 7fbe3a800640 1 -- 172.21.15.69:0/2210388899 learned_addr learned my addr 172.21.15.69:0/2210388899 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe39400640 1 --2- 172.21.15.69:0/2210388899 >> v2:172.21.15.121:3300/0 conn(0x7fbe3c0cdbd0 0x7fbe3c09b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2202335687 0 0) 0x7fbe3c0ce150 con 0x7fbe3c0b14b0 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe20003520 con 0x7fbe3c0b14b0 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1698257866 0 0) 0x7fbe20003520 con 0x7fbe3c0b14b0 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 >> v2:172.21.15.121:3300/0 conn(0x7fbe3c0cdbd0 msgr2=0x7fbe3c09b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 --2- 172.21.15.69:0/2210388899 >> v2:172.21.15.121:3300/0 conn(0x7fbe3c0cdbd0 0x7fbe3c09b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 >> v1:172.21.15.121:6789/0 conn(0x7fbe3c15ca20 legacy=0x7fbe3c0b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3c09aa70 con 0x7fbe3c0b14b0 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fbe34001670 con 0x7fbe3c0b14b0 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 >> v1:172.21.15.69:6789/0 conn(0x7fbe3c0b14b0 legacy=0x7fbe3c0d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe200039b0 0x7fbe20003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe20004800 0x7fbe20004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe200051c0 0x7fbe2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c15ce00 con 0x7fbe200039b0 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbe2000c060 con 0x7fbe20004800 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.075+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbe2000c260 con 0x7fbe200051c0 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.076+0000 7fbe39e00640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe200039b0 0x7fbe20003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.076+0000 7fbe39400640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe20004800 0x7fbe20004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.076+0000 7fbe3a800640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe200051c0 0x7fbe2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.076+0000 7fbe39e00640 1 -- 172.21.15.69:0/2210388899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe200051c0 msgr2=0x7fbe2000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.076+0000 7fbe39e00640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe200051c0 0x7fbe2000b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.076+0000 7fbe39e00640 1 -- 172.21.15.69:0/2210388899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe20004800 msgr2=0x7fbe20004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.076+0000 7fbe39e00640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe20004800 0x7fbe20004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.076+0000 7fbe39e00640 1 -- 172.21.15.69:0/2210388899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe20003520 con 0x7fbe200039b0 2024-04-24T08:11:26.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.076+0000 7fbe39400640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe20004800 0x7fbe20004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:26.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.076+0000 7fbe3a800640 1 -- 172.21.15.69:0/2210388899 reap_dead start 2024-04-24T08:11:26.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.077+0000 7fbe39e00640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe200039b0 0x7fbe20003dd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbe24001750 tx=0x7fbe20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84b0a05ebd56616 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:26.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.077+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe24001370 con 0x7fbe200039b0 2024-04-24T08:11:26.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.077+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe240052d0 con 0x7fbe200039b0 2024-04-24T08:11:26.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.077+0000 7fbe38a00640 1 -- 172.21.15.69:0/2210388899 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2402abe0 con 0x7fbe200039b0 2024-04-24T08:11:26.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.078+0000 7fbe40c00640 1 -- 172.21.15.69:0/2210388899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe200039b0 msgr2=0x7fbe20003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.078+0000 7fbe40c00640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe200039b0 0x7fbe20003dd0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbe24001750 tx=0x7fbe20005830 comp rx=0 tx=0).stop 2024-04-24T08:11:26.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.078+0000 7fbe40c00640 1 -- 172.21.15.69:0/2210388899 shutdown_connections 2024-04-24T08:11:26.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.078+0000 7fbe40c00640 1 --2- 172.21.15.69:0/2210388899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe200039b0 0x7fbe20003dd0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.078+0000 7fbe40c00640 1 -- 172.21.15.69:0/2210388899 >> 172.21.15.69:0/2210388899 conn(0x7fbe3c0b4040 msgr2=0x7fbe3c0b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:26.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.078+0000 7fbe40c00640 1 -- 172.21.15.69:0/2210388899 shutdown_connections 2024-04-24T08:11:26.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.078+0000 7fbe40c00640 1 -- 172.21.15.69:0/2210388899 wait complete. 2024-04-24T08:11:26.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.080+0000 7fbe40c00640 1 Processor -- start 2024-04-24T08:11:26.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.080+0000 7fbe40c00640 1 -- start start 2024-04-24T08:11:26.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe40c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe200039b0 0x7fbe3c15aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe40c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe3c0c8650 0x7fbe3c0c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe40c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe3c1576a0 0x7fbe3c157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe40c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c0d0510 con 0x7fbe3c0c8650 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe40c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c09b030 con 0x7fbe200039b0 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe40c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbe3c0d0690 con 0x7fbe3c1576a0 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe39400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe3c0c8650 0x7fbe3c0c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe39400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe3c0c8650 0x7fbe3c0c8ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45124/0 (socket says 172.21.15.69:45124) 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe39400640 1 -- 172.21.15.69:0/1918908105 learned_addr learned my addr 172.21.15.69:0/1918908105 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe3a800640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe3c1576a0 0x7fbe3c157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe39e00640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe200039b0 0x7fbe3c15aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe39400640 1 -- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe3c1576a0 msgr2=0x7fbe3c157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe39400640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe3c1576a0 0x7fbe3c157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe39400640 1 -- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe200039b0 msgr2=0x7fbe3c15aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe39400640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe200039b0 0x7fbe3c15aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.081+0000 7fbe39400640 1 -- 172.21.15.69:0/1918908105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3c0c75d0 con 0x7fbe3c0c8650 2024-04-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.082+0000 7fbe39400640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe3c0c8650 0x7fbe3c0c8ab0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00f3b0 tx=0x7fbe2c00f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.082+0000 7fbe32000640 1 -- 172.21.15.69:0/1918908105 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c003990 con 0x7fbe3c0c8650 2024-04-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.082+0000 7fbe32000640 1 -- 172.21.15.69:0/1918908105 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe2c003b30 con 0x7fbe3c0c8650 2024-04-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.082+0000 7fbe32000640 1 -- 172.21.15.69:0/1918908105 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c004020 con 0x7fbe3c0c8650 2024-04-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.082+0000 7fbe40c00640 1 -- 172.21.15.69:0/1918908105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe3c0c78b0 con 0x7fbe3c0c8650 2024-04-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.082+0000 7fbe40c00640 1 -- 172.21.15.69:0/1918908105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe3c0c7e70 con 0x7fbe3c0c8650 2024-04-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.083+0000 7fbe40c00640 1 -- 172.21.15.69:0/1918908105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe3c068a80 con 0x7fbe3c0c8650 2024-04-24T08:11:26.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.084+0000 7fbe32000640 1 -- 172.21.15.69:0/1918908105 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe2c01d020 con 0x7fbe3c0c8650 2024-04-24T08:11:26.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.085+0000 7fbe32000640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe1806c920 0x7fbe1806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.085+0000 7fbe32000640 1 -- 172.21.15.69:0/1918908105 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fbe2c08c410 con 0x7fbe3c0c8650 2024-04-24T08:11:26.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.086+0000 7fbe39e00640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe1806c920 0x7fbe1806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.087+0000 7fbe39e00640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe1806c920 0x7fbe1806ede0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbe24032000 tx=0x7fbe2402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:26.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.090+0000 7fbe32000640 1 -- 172.21.15.69:0/1918908105 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe2c05a7f0 con 0x7fbe3c0c8650 2024-04-24T08:11:26.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.312+0000 7fbe40c00640 1 -- 172.21.15.69:0/1918908105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe3c0b4cd0 con 0x7fbe3c0c8650 2024-04-24T08:11:26.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.313+0000 7fbe32000640 1 -- 172.21.15.69:0/1918908105 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+5184 (secure 0 0 0) 0x7fbe2c05a0f0 con 0x7fbe3c0c8650 2024-04-24T08:11:26.315 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:26.315 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/376722301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":376722301},{"type":"v1","addr":"172.21.15.69:6835","nonce":376722301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4990,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1327624292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1327624292},{"type":"v1","addr":"172.21.15.121:6836","nonce":1327624292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/3244002416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3244002416},{"type":"v1","addr":"172.21.15.69:6834","nonce":3244002416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:10:35.400788+0000","modified":"2024-04-24T08:10:38.457641+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5793},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5793":{"gid":5793,"name":"d","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/3424934990","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3424934990},{"type":"v1","addr":"172.21.15.121:6837","nonce":3424934990}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5793,"qdb_cluster":[5793]},"id":3}]} 2024-04-24T08:11:26.315 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 -- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe1806c920 msgr2=0x7fbe1806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe1806c920 0x7fbe1806ede0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbe24032000 tx=0x7fbe2402e000 comp rx=0 tx=0).stop 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 -- 172.21.15.69:0/1918908105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe3c0c8650 msgr2=0x7fbe3c0c8ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe3c0c8650 0x7fbe3c0c8ab0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00f3b0 tx=0x7fbe2c00f870 comp rx=0 tx=0).stop 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 -- 172.21.15.69:0/1918908105 shutdown_connections 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbe1806c920 0x7fbe1806ede0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe3c0c8650 0x7fbe3c0c8ab0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbe3c1576a0 0x7fbe3c157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 --2- 172.21.15.69:0/1918908105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe200039b0 0x7fbe3c15aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 -- 172.21.15.69:0/1918908105 >> 172.21.15.69:0/1918908105 conn(0x7fbe3c0b4040 msgr2=0x7fbe3c0af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 -- 172.21.15.69:0/1918908105 shutdown_connections 2024-04-24T08:11:26.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.316+0000 7fbe40c00640 1 -- 172.21.15.69:0/1918908105 wait complete. 2024-04-24T08:11:26.335 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:11:26.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.523+0000 7f062f200640 1 Processor -- start 2024-04-24T08:11:26.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.523+0000 7f062f200640 1 -- start start 2024-04-24T08:11:26.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.524+0000 7f062f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f06280d0300 0x7f062806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.525+0000 7f062f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145fe0 0x7f0628160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.525+0000 7f062f200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06280cf3d0 con 0x7f0628145a90 2024-04-24T08:11:26.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.525+0000 7f062f200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f06280cf6b0 con 0x7f06280d0300 2024-04-24T08:11:26.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.525+0000 7f062f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f06280cf990 con 0x7f0628145fe0 2024-04-24T08:11:26.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.525+0000 7f0627e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f06280d0300 0x7f062806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.525+0000 7f062d200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145fe0 0x7f0628160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.526+0000 7f062d200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145fe0 0x7f0628160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:40518/0 (socket says 172.21.15.69:40518) 2024-04-24T08:11:26.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.526+0000 7f0627e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f06280d0300 0x7f062806e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:40750/0 (socket says 172.21.15.69:40750) 2024-04-24T08:11:26.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.526+0000 7f062d200640 1 -- 172.21.15.69:0/691166965 learned_addr learned my addr 172.21.15.69:0/691166965 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.526+0000 7f062d200640 1 -- 172.21.15.69:0/691166965 >> v2:172.21.15.121:3300/0 conn(0x7f06280d0300 msgr2=0x7f062806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.526+0000 7f062d200640 1 --2- 172.21.15.69:0/691166965 >> v2:172.21.15.121:3300/0 conn(0x7f06280d0300 0x7f062806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.526+0000 7f062d200640 1 -- 172.21.15.69:0/691166965 >> v1:172.21.15.121:6789/0 conn(0x7f0628145a90 legacy=0x7f06280d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.526+0000 7f062d200640 1 -- 172.21.15.69:0/691166965 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06280cfc70 con 0x7f0628145fe0 2024-04-24T08:11:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.526+0000 7f0627e00640 1 --2- 172.21.15.69:0/691166965 >> v2:172.21.15.121:3300/0 conn(0x7f06280d0300 0x7f062806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.528+0000 7f062d200640 1 --2- 172.21.15.69:0/691166965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145fe0 0x7f0628160540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0620009e50 tx=0x7f062002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b12a9a46036a1a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:26.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.528+0000 7f0626a00640 1 -- 172.21.15.69:0/691166965 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620036070 con 0x7f0628145fe0 2024-04-24T08:11:26.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.528+0000 7f0626a00640 1 -- 172.21.15.69:0/691166965 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0620031600 con 0x7f0628145fe0 2024-04-24T08:11:26.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.528+0000 7f0626a00640 1 -- 172.21.15.69:0/691166965 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06200319d0 con 0x7f0628145fe0 2024-04-24T08:11:26.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.529+0000 7f062f200640 1 -- 172.21.15.69:0/691166965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145fe0 msgr2=0x7f0628160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.529+0000 7f062f200640 1 --2- 172.21.15.69:0/691166965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145fe0 0x7f0628160540 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0620009e50 tx=0x7f062002a710 comp rx=0 tx=0).stop 2024-04-24T08:11:26.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.529+0000 7f062f200640 1 -- 172.21.15.69:0/691166965 shutdown_connections 2024-04-24T08:11:26.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.529+0000 7f062f200640 1 --2- 172.21.15.69:0/691166965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145fe0 0x7f0628160540 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.529+0000 7f062f200640 1 --2- 172.21.15.69:0/691166965 >> v2:172.21.15.121:3300/0 conn(0x7f06280d0300 0x7f062806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.529+0000 7f062f200640 1 -- 172.21.15.69:0/691166965 >> 172.21.15.69:0/691166965 conn(0x7f0628067d90 msgr2=0x7f062806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:26.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.529+0000 7f062f200640 1 -- 172.21.15.69:0/691166965 shutdown_connections 2024-04-24T08:11:26.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.529+0000 7f062f200640 1 -- 172.21.15.69:0/691166965 wait complete. 2024-04-24T08:11:26.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.530+0000 7f062f200640 1 Processor -- start 2024-04-24T08:11:26.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.531+0000 7f062f200640 1 -- start start 2024-04-24T08:11:26.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.531+0000 7f062f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06280d0300 0x7f062815c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.531+0000 7f062f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145a90 0x7f06280cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.531+0000 7f062f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0628145fe0 0x7f06280ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.531+0000 7f062f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f06281458f0 con 0x7f0628145fe0 2024-04-24T08:11:26.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.531+0000 7f062f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0628145770 con 0x7f06280d0300 2024-04-24T08:11:26.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.531+0000 7f062f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f06280cdc30 con 0x7f0628145a90 2024-04-24T08:11:26.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.532+0000 7f062d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0628145fe0 0x7f06280ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.532+0000 7f062d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0628145fe0 0x7f06280ad550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45132/0 (socket says 172.21.15.69:45132) 2024-04-24T08:11:26.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.532+0000 7f062d200640 1 -- 172.21.15.69:0/1505200815 learned_addr learned my addr 172.21.15.69:0/1505200815 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:26.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.532+0000 7f0627e00640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06280d0300 0x7f062815c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.532+0000 7f0627400640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145a90 0x7f06280cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.532+0000 7f062d200640 1 -- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145a90 msgr2=0x7f06280cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.532+0000 7f062d200640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145a90 0x7f06280cad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.532+0000 7f062d200640 1 -- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06280d0300 msgr2=0x7f062815c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.532+0000 7f062d200640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06280d0300 0x7f062815c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.532+0000 7f062d200640 1 -- 172.21.15.69:0/1505200815 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06280adfa0 con 0x7f0628145fe0 2024-04-24T08:11:26.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.533+0000 7f062d200640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0628145fe0 0x7f06280ad550 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f0620034040 tx=0x7f0620004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:26.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.533+0000 7f061fe00640 1 -- 172.21.15.69:0/1505200815 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620041020 con 0x7f0628145fe0 2024-04-24T08:11:26.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.533+0000 7f061fe00640 1 -- 172.21.15.69:0/1505200815 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06200077a0 con 0x7f0628145fe0 2024-04-24T08:11:26.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.533+0000 7f061fe00640 1 -- 172.21.15.69:0/1505200815 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0620036040 con 0x7f0628145fe0 2024-04-24T08:11:26.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.533+0000 7f062f200640 1 -- 172.21.15.69:0/1505200815 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06280b34c0 con 0x7f0628145fe0 2024-04-24T08:11:26.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.533+0000 7f062f200640 1 -- 172.21.15.69:0/1505200815 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06280b3a80 con 0x7f0628145fe0 2024-04-24T08:11:26.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.534+0000 7f062f200640 1 -- 172.21.15.69:0/1505200815 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0628068a80 con 0x7f0628145fe0 2024-04-24T08:11:26.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.535+0000 7f061fe00640 1 -- 172.21.15.69:0/1505200815 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0620044020 con 0x7f0628145fe0 2024-04-24T08:11:26.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.537+0000 7f061fe00640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f060806c920 0x7f060806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.537+0000 7f061fe00640 1 -- 172.21.15.69:0/1505200815 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7f06200ac800 con 0x7f0628145fe0 2024-04-24T08:11:26.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.537+0000 7f0627e00640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f060806c920 0x7f060806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.538+0000 7f0627e00640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f060806c920 0x7f060806ede0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0618009560 tx=0x7f0618008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:26.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.541+0000 7f061fe00640 1 -- 172.21.15.69:0/1505200815 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f062004b030 con 0x7f0628145fe0 2024-04-24T08:11:26.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.725+0000 7f062f200640 1 -- 172.21.15.69:0/1505200815 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06280d1120 con 0x7f0628145fe0 2024-04-24T08:11:26.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.728+0000 7f061fe00640 1 -- 172.21.15.69:0/1505200815 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+16740 (secure 0 0 0) 0x7f062007ab50 con 0x7f0628145fe0 2024-04-24T08:11:26.730 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:26.730 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":46,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:03.085538+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:10:33.100132+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-04-24T08:10:34.379302+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.7","mappings":[{"from":0,"to":5}]},{"pgid":"7.36","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:11:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 -- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f060806c920 msgr2=0x7f060806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f060806c920 0x7f060806ede0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0618009560 tx=0x7f0618008040 comp rx=0 tx=0).stop 2024-04-24T08:11:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 -- 172.21.15.69:0/1505200815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0628145fe0 msgr2=0x7f06280ad550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0628145fe0 0x7f06280ad550 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f0620034040 tx=0x7f0620004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 -- 172.21.15.69:0/1505200815 shutdown_connections 2024-04-24T08:11:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f060806c920 0x7f060806ede0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0628145fe0 0x7f06280ad550 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0628145a90 0x7f06280cad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 --2- 172.21.15.69:0/1505200815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06280d0300 0x7f062815c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 -- 172.21.15.69:0/1505200815 >> 172.21.15.69:0/1505200815 conn(0x7f0628067d90 msgr2=0x7f062806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.730+0000 7f062f200640 1 -- 172.21.15.69:0/1505200815 shutdown_connections 2024-04-24T08:11:26.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.731+0000 7f062f200640 1 -- 172.21.15.69:0/1505200815 wait complete. 2024-04-24T08:11:26.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:11:26.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.946+0000 7fe912c00640 1 Processor -- start 2024-04-24T08:11:26.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.946+0000 7fe912c00640 1 -- start start 2024-04-24T08:11:26.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.948+0000 7fe912c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe90c15b490 0x7fe90c0b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.948+0000 7fe912c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe90c06f2d0 0x7fe90c06f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.948+0000 7fe912c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe90c0cfbe0 0x7fe90c0d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.948+0000 7fe912c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe90c06fc70 con 0x7fe90c0cfbe0 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.948+0000 7fe912c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe90c0b46e0 con 0x7fe90c06f2d0 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.948+0000 7fe912c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe90c0d0960 con 0x7fe90c15b490 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.948+0000 7fe910c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe90c0cfbe0 0x7fe90c0d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.948+0000 7fe910c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe90c0cfbe0 0x7fe90c0d0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45142/0 (socket says 172.21.15.69:45142) 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.948+0000 7fe910c00640 1 -- 172.21.15.69:0/1019346124 learned_addr learned my addr 172.21.15.69:0/1019346124 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.949+0000 7fe90be00640 1 --2- 172.21.15.69:0/1019346124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe90c15b490 0x7fe90c0b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.949+0000 7fe910c00640 1 -- 172.21.15.69:0/1019346124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe90c15b490 msgr2=0x7fe90c0b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.949+0000 7fe90b400640 1 --2- 172.21.15.69:0/1019346124 >> v2:172.21.15.121:3300/0 conn(0x7fe90c06f2d0 0x7fe90c06f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.949+0000 7fe910c00640 1 --2- 172.21.15.69:0/1019346124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe90c15b490 0x7fe90c0b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.949+0000 7fe910c00640 1 -- 172.21.15.69:0/1019346124 >> v2:172.21.15.121:3300/0 conn(0x7fe90c06f2d0 msgr2=0x7fe90c06f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.949+0000 7fe910c00640 1 --2- 172.21.15.69:0/1019346124 >> v2:172.21.15.121:3300/0 conn(0x7fe90c06f2d0 0x7fe90c06f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.949+0000 7fe910c00640 1 -- 172.21.15.69:0/1019346124 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90c0b4980 con 0x7fe90c0cfbe0 2024-04-24T08:11:26.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.949+0000 7fe90b400640 1 --2- 172.21.15.69:0/1019346124 >> v2:172.21.15.121:3300/0 conn(0x7fe90c06f2d0 0x7fe90c06f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:26.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.950+0000 7fe910c00640 1 --2- 172.21.15.69:0/1019346124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c0cfbe0 0x7fe90c0d0070 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe90c073050 tx=0x7fe90402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01ac2b419622c74 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:26.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.950+0000 7fe90aa00640 1 -- 172.21.15.69:0/1019346124 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904037070 con 0x7fe90c0cfbe0 2024-04-24T08:11:26.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.950+0000 7fe90aa00640 1 -- 172.21.15.69:0/1019346124 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe904032440 con 0x7fe90c0cfbe0 2024-04-24T08:11:26.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.950+0000 7fe90aa00640 1 -- 172.21.15.69:0/1019346124 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe904032810 con 0x7fe90c0cfbe0 2024-04-24T08:11:26.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.951+0000 7fe912c00640 1 -- 172.21.15.69:0/1019346124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c0cfbe0 msgr2=0x7fe90c0d0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.951+0000 7fe912c00640 1 --2- 172.21.15.69:0/1019346124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c0cfbe0 0x7fe90c0d0070 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fe90c073050 tx=0x7fe90402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.951+0000 7fe912c00640 1 -- 172.21.15.69:0/1019346124 shutdown_connections 2024-04-24T08:11:26.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.951+0000 7fe912c00640 1 --2- 172.21.15.69:0/1019346124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c0cfbe0 0x7fe90c0d0070 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.951+0000 7fe912c00640 1 --2- 172.21.15.69:0/1019346124 >> v2:172.21.15.121:3300/0 conn(0x7fe90c06f2d0 0x7fe90c06f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.951+0000 7fe912c00640 1 --2- 172.21.15.69:0/1019346124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe90c15b490 0x7fe90c0b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.951+0000 7fe912c00640 1 -- 172.21.15.69:0/1019346124 >> 172.21.15.69:0/1019346124 conn(0x7fe90c067d90 msgr2=0x7fe90c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:26.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.951+0000 7fe912c00640 1 -- 172.21.15.69:0/1019346124 shutdown_connections 2024-04-24T08:11:26.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.951+0000 7fe912c00640 1 -- 172.21.15.69:0/1019346124 wait complete. 2024-04-24T08:11:26.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.953+0000 7fe912c00640 1 Processor -- start 2024-04-24T08:11:26.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.953+0000 7fe912c00640 1 -- start start 2024-04-24T08:11:26.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe912c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c06f2d0 0x7fe90c1f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe912c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe90c0cfbe0 0x7fe90c1f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe912c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe90c15b490 0x7fe90c155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe912c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c15b870 con 0x7fe90c06f2d0 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe912c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe90c0d0c40 con 0x7fe90c15b490 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe912c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe90c0d05b0 con 0x7fe90c0cfbe0 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe90be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c06f2d0 0x7fe90c1f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe90be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c06f2d0 0x7fe90c1f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45148/0 (socket says 172.21.15.69:45148) 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe90be00640 1 -- 172.21.15.69:0/263303222 learned_addr learned my addr 172.21.15.69:0/263303222 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe90be00640 1 -- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe90c0cfbe0 msgr2=0x7fe90c1f55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe90be00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe90c0cfbe0 0x7fe90c1f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe90be00640 1 -- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe90c15b490 msgr2=0x7fe90c155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:26.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe910c00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe90c15b490 0x7fe90c155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe90be00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe90c15b490 0x7fe90c155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:26.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.954+0000 7fe90be00640 1 -- 172.21.15.69:0/263303222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe90c15e100 con 0x7fe90c06f2d0 2024-04-24T08:11:26.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.955+0000 7fe90be00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c06f2d0 0x7fe90c1f28a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00ccd0 tx=0x7fe8fc00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:26.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.955+0000 7fe903e00640 1 -- 172.21.15.69:0/263303222 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc003990 con 0x7fe90c06f2d0 2024-04-24T08:11:26.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.955+0000 7fe903e00640 1 -- 172.21.15.69:0/263303222 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8fc003b30 con 0x7fe90c06f2d0 2024-04-24T08:11:26.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.955+0000 7fe903e00640 1 -- 172.21.15.69:0/263303222 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8fc004020 con 0x7fe90c06f2d0 2024-04-24T08:11:26.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.955+0000 7fe912c00640 1 -- 172.21.15.69:0/263303222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe90c153680 con 0x7fe90c06f2d0 2024-04-24T08:11:26.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.955+0000 7fe912c00640 1 -- 172.21.15.69:0/263303222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe90c153c40 con 0x7fe90c06f2d0 2024-04-24T08:11:26.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.956+0000 7fe912c00640 1 -- 172.21.15.69:0/263303222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe90c068a80 con 0x7fe90c06f2d0 2024-04-24T08:11:26.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.957+0000 7fe903e00640 1 -- 172.21.15.69:0/263303222 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe8fc0041c0 con 0x7fe90c06f2d0 2024-04-24T08:11:26.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.958+0000 7fe903e00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe8e806c8f0 0x7fe8e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:26.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.958+0000 7fe903e00640 1 -- 172.21.15.69:0/263303222 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5767+0+0 (secure 0 0 0) 0x7fe8fc08c120 con 0x7fe90c06f2d0 2024-04-24T08:11:26.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.959+0000 7fe90b400640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe8e806c8f0 0x7fe8e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:26.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.960+0000 7fe90b400640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe8e806c8f0 0x7fe8e806edb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4006fd0 tx=0x7fe8f4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:26.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:26.962+0000 7fe903e00640 1 -- 172.21.15.69:0/263303222 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8fc014030 con 0x7fe90c06f2d0 2024-04-24T08:11:27.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.210+0000 7fe912c00640 1 -- 172.21.15.69:0/263303222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe90c0cd6b0 con 0x7fe90c06f2d0 2024-04-24T08:11:27.213 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:11:27.212+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:11:27.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.215+0000 7fe903e00640 1 -- 172.21.15.69:0/263303222 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v26) ==== 161+0+0 (secure 0 0 0) 0x7fe8fc05a470 con 0x7fe90c06f2d0 2024-04-24T08:11:27.217 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 -- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe8e806c8f0 msgr2=0x7fe8e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe8e806c8f0 0x7fe8e806edb0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4006fd0 tx=0x7fe8f4008040 comp rx=0 tx=0).stop 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 -- 172.21.15.69:0/263303222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c06f2d0 msgr2=0x7fe90c1f28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c06f2d0 0x7fe90c1f28a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc00ccd0 tx=0x7fe8fc00a5c0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 -- 172.21.15.69:0/263303222 shutdown_connections 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe8e806c8f0 0x7fe8e806edb0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe90c15b490 0x7fe90c155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe90c0cfbe0 0x7fe90c1f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 --2- 172.21.15.69:0/263303222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe90c06f2d0 0x7fe90c1f28a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 -- 172.21.15.69:0/263303222 >> 172.21.15.69:0/263303222 conn(0x7fe90c067d90 msgr2=0x7fe90c0b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 -- 172.21.15.69:0/263303222 shutdown_connections 2024-04-24T08:11:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.220+0000 7fe912c00640 1 -- 172.21.15.69:0/263303222 wait complete. 2024-04-24T08:11:27.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T08:11:27.253 INFO:tasks.ceph.mds.d.smithi121.stderr: -5035> 2024-04-24T08:10:41.902+0000 7f612ce00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:11:27.264 INFO:tasks.ceph.mds.d.smithi121.stderr: -3281> 2024-04-24T08:10:47.654+0000 7f6127e00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={5940=0-4194304@1} caps={5940=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=5940 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x55eaeeaa2c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-24T08:11:27.336 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-24T08:11:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.435+0000 7fe64e200640 1 Processor -- start 2024-04-24T08:11:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.436+0000 7fe64e200640 1 -- start start 2024-04-24T08:11:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.437+0000 7fe64e200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe6480b5300 0x7fe6480d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.437+0000 7fe64e200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6480d0c40 con 0x7fe6480d0820 2024-04-24T08:11:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.437+0000 7fe64e200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6480ce640 con 0x7fe64815ea60 2024-04-24T08:11:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.437+0000 7fe64e200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe6480ce910 con 0x7fe6480b5300 2024-04-24T08:11:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.438+0000 7fe64d800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe6480d0820 0x7fe64806df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33986/0 (socket says 172.21.15.69:33986) 2024-04-24T08:11:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.438+0000 7fe64d800640 1 -- 172.21.15.69:0/4249731670 learned_addr learned my addr 172.21.15.69:0/4249731670 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.438+0000 7fe647e00640 1 --2- 172.21.15.69:0/4249731670 >> v2:172.21.15.121:3300/0 conn(0x7fe6480b5300 0x7fe6480d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.438+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3434366355 0 0) 0x7fe6480d0c40 con 0x7fe6480d0820 2024-04-24T08:11:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.438+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe630005730 con 0x7fe6480d0820 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1762694148 0 0) 0x7fe630005730 con 0x7fe6480d0820 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 >> v2:172.21.15.121:3300/0 conn(0x7fe6480b5300 msgr2=0x7fe6480d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 --2- 172.21.15.69:0/4249731670 >> v2:172.21.15.121:3300/0 conn(0x7fe6480b5300 0x7fe6480d02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 >> v1:172.21.15.121:6789/0 conn(0x7fe64815ea60 legacy=0x7fe6480cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6480cebf0 con 0x7fe6480d0820 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe6400019f0 con 0x7fe6480d0820 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 >> v1:172.21.15.69:6789/0 conn(0x7fe6480d0820 legacy=0x7fe64806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 --2- 172.21.15.69:0/4249731670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe630005c10 0x7fe630006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 --2- 172.21.15.69:0/4249731670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe630006a10 0x7fe630006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 --2- 172.21.15.69:0/4249731670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6300073d0 0x7fe63000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe6480ce910 con 0x7fe630005c10 2024-04-24T08:11:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe63000e2a0 con 0x7fe6300073d0 2024-04-24T08:11:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe63000e4a0 con 0x7fe630006a10 2024-04-24T08:11:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe64ce00640 1 --2- 172.21.15.69:0/4249731670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe630005c10 0x7fe630006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe647e00640 1 --2- 172.21.15.69:0/4249731670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe630006a10 0x7fe630006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe64ce00640 1 -- 172.21.15.69:0/4249731670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe630006a10 msgr2=0x7fe630006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe64ce00640 1 --2- 172.21.15.69:0/4249731670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe630006a10 0x7fe630006e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe64ce00640 1 -- 172.21.15.69:0/4249731670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6300073d0 msgr2=0x7fe63000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe64ce00640 1 --2- 172.21.15.69:0/4249731670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6300073d0 0x7fe63000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.440+0000 7fe64d800640 1 -- 172.21.15.69:0/4249731670 reap_dead start 2024-04-24T08:11:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.439+0000 7fe64ce00640 1 -- 172.21.15.69:0/4249731670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe630005730 con 0x7fe630005c10 2024-04-24T08:11:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.440+0000 7fe64ce00640 1 --2- 172.21.15.69:0/4249731670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe630005c10 0x7fe630006010 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe63c003db0 tx=0x7fe630007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa43cc83e979c38 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:27.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.441+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c037070 con 0x7fe630005c10 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.441+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe63c02aa00 con 0x7fe630005c10 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.441+0000 7fe647400640 1 -- 172.21.15.69:0/4249731670 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe63c003980 con 0x7fe630005c10 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.441+0000 7fe64e200640 1 -- 172.21.15.69:0/4249731670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe630005c10 msgr2=0x7fe630006010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.441+0000 7fe64e200640 1 --2- 172.21.15.69:0/4249731670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe630005c10 0x7fe630006010 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe63c003db0 tx=0x7fe630007a40 comp rx=0 tx=0).stop 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.441+0000 7fe64e200640 1 -- 172.21.15.69:0/4249731670 shutdown_connections 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.441+0000 7fe64e200640 1 --2- 172.21.15.69:0/4249731670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe630005c10 0x7fe630006010 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.441+0000 7fe64e200640 1 -- 172.21.15.69:0/4249731670 >> 172.21.15.69:0/4249731670 conn(0x7fe648067b80 msgr2=0x7fe64806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.441+0000 7fe64e200640 1 -- 172.21.15.69:0/4249731670 shutdown_connections 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.441+0000 7fe64e200640 1 -- 172.21.15.69:0/4249731670 wait complete. 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.443+0000 7fe64e200640 1 Processor -- start 2024-04-24T08:11:27.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.443+0000 7fe64e200640 1 -- start start 2024-04-24T08:11:27.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.444+0000 7fe64e200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe630005c10 0x7fe648145d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:27.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.444+0000 7fe64e200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe64815b5d0 0x7fe648159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:27.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.444+0000 7fe64e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe648156f90 0x7fe648157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:27.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.444+0000 7fe64e200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe6480cf490 con 0x7fe648156f90 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.444+0000 7fe64e200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe6480cf1b0 con 0x7fe630005c10 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.444+0000 7fe64e200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe6480d02e0 con 0x7fe64815b5d0 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe64d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe648156f90 0x7fe648157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe64d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe648156f90 0x7fe648157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45180/0 (socket says 172.21.15.69:45180) 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe64d800640 1 -- 172.21.15.69:0/2524791228 learned_addr learned my addr 172.21.15.69:0/2524791228 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe64ce00640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe630005c10 0x7fe648145d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe647e00640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe64815b5d0 0x7fe648159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe64d800640 1 -- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe64815b5d0 msgr2=0x7fe648159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe64d800640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe64815b5d0 0x7fe648159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe64d800640 1 -- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe630005c10 msgr2=0x7fe648145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe64d800640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe630005c10 0x7fe648145d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:27.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe64d800640 1 -- 172.21.15.69:0/2524791228 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64814f6a0 con 0x7fe648156f90 2024-04-24T08:11:27.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.445+0000 7fe64d800640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe648156f90 0x7fe648157420 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe6480d10e0 tx=0x7fe64000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:27.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.446+0000 7fe644c00640 1 -- 172.21.15.69:0/2524791228 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64000ce70 con 0x7fe648156f90 2024-04-24T08:11:27.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.446+0000 7fe644c00640 1 -- 172.21.15.69:0/2524791228 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe64000a610 con 0x7fe648156f90 2024-04-24T08:11:27.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.446+0000 7fe64e200640 1 -- 172.21.15.69:0/2524791228 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe64815bab0 con 0x7fe648156f90 2024-04-24T08:11:27.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.446+0000 7fe644c00640 1 -- 172.21.15.69:0/2524791228 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64000aa00 con 0x7fe648156f90 2024-04-24T08:11:27.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.446+0000 7fe64e200640 1 -- 172.21.15.69:0/2524791228 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe64815c070 con 0x7fe648156f90 2024-04-24T08:11:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.447+0000 7fe64e200640 1 -- 172.21.15.69:0/2524791228 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe648075d50 con 0x7fe648156f90 2024-04-24T08:11:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.448+0000 7fe644c00640 1 -- 172.21.15.69:0/2524791228 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe640011070 con 0x7fe648156f90 2024-04-24T08:11:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.449+0000 7fe644c00640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe62806c920 0x7fe62806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.449+0000 7fe644c00640 1 -- 172.21.15.69:0/2524791228 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5853+0+0 (secure 0 0 0) 0x7fe64008b6a0 con 0x7fe648156f90 2024-04-24T08:11:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.449+0000 7fe64ce00640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe62806c920 0x7fe62806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.451+0000 7fe64ce00640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe62806c920 0x7fe62806ede0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe63c034000 tx=0x7fe63c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.453+0000 7fe644c00640 1 -- 172.21.15.69:0/2524791228 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe64001a060 con 0x7fe648156f90 2024-04-24T08:11:27.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:27.703+0000 7fe64e200640 1 -- 172.21.15.69:0/2524791228 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe64815dec0 con 0x7fe648156f90 2024-04-24T08:11:28.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.219+0000 7fe644c00640 1 -- 172.21.15.69:0/2524791228 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v27) ==== 104+0+0 (secure 0 0 0) 0x7fe64814f6a0 con 0x7fe648156f90 2024-04-24T08:11:28.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.222+0000 7fe64e200640 1 -- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe62806c920 msgr2=0x7fe62806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:28.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.222+0000 7fe64e200640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe62806c920 0x7fe62806ede0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe63c034000 tx=0x7fe63c02e000 comp rx=0 tx=0).stop 2024-04-24T08:11:28.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.222+0000 7fe64e200640 1 -- 172.21.15.69:0/2524791228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe648156f90 msgr2=0x7fe648157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:28.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.222+0000 7fe64e200640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe648156f90 0x7fe648157420 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe6480d10e0 tx=0x7fe64000cc20 comp rx=0 tx=0).stop 2024-04-24T08:11:28.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.223+0000 7fe64e200640 1 -- 172.21.15.69:0/2524791228 shutdown_connections 2024-04-24T08:11:28.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.223+0000 7fe64e200640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe62806c920 0x7fe62806ede0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:28.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.223+0000 7fe64e200640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe648156f90 0x7fe648157420 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:28.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.223+0000 7fe64e200640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe64815b5d0 0x7fe648159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:28.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.223+0000 7fe64e200640 1 --2- 172.21.15.69:0/2524791228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe630005c10 0x7fe648145d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:28.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.223+0000 7fe64e200640 1 -- 172.21.15.69:0/2524791228 >> 172.21.15.69:0/2524791228 conn(0x7fe648067b80 msgr2=0x7fe64806eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:28.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.223+0000 7fe64e200640 1 -- 172.21.15.69:0/2524791228 shutdown_connections 2024-04-24T08:11:28.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.223+0000 7fe64e200640 1 -- 172.21.15.69:0/2524791228 wait complete. 2024-04-24T08:11:28.246 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T08:11:28.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.452+0000 7fef3cc00640 1 Processor -- start 2024-04-24T08:11:28.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.452+0000 7fef3cc00640 1 -- start start 2024-04-24T08:11:28.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.453+0000 7fef3cc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fef380b32d0 0x7fef3815f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.453+0000 7fef3cc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef3815fad0 0x7fef380b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.453+0000 7fef3cc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef3815edb0 con 0x7fef3806f5e0 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.453+0000 7fef3cc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fef3806f9c0 con 0x7fef380b32d0 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.453+0000 7fef3cc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fef38162bb0 con 0x7fef3815fad0 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.454+0000 7fef35400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fef380b32d0 0x7fef3815f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.454+0000 7fef35400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fef380b32d0 0x7fef3815f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45196/0 (socket says 172.21.15.69:45196) 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.454+0000 7fef35400640 1 -- 172.21.15.69:0/1424413424 learned_addr learned my addr 172.21.15.69:0/1424413424 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.454+0000 7fef35400640 1 -- 172.21.15.69:0/1424413424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef3815fad0 msgr2=0x7fef380b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.454+0000 7fef35400640 1 --2- 172.21.15.69:0/1424413424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef3815fad0 0x7fef380b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.454+0000 7fef35400640 1 -- 172.21.15.69:0/1424413424 >> v1:172.21.15.69:6789/0 conn(0x7fef3806f5e0 legacy=0x7fef381623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:28.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.454+0000 7fef35400640 1 -- 172.21.15.69:0/1424413424 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef38162e90 con 0x7fef380b32d0 2024-04-24T08:11:28.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.455+0000 7fef35400640 1 --2- 172.21.15.69:0/1424413424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef380b32d0 0x7fef3815f590 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fef24009870 tx=0x7fef2402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c1ff6d97cb3700 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:28.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.455+0000 7fef34a00640 1 -- 172.21.15.69:0/1424413424 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef24037070 con 0x7fef380b32d0 2024-04-24T08:11:28.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.456+0000 7fef34a00640 1 -- 172.21.15.69:0/1424413424 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef24032440 con 0x7fef380b32d0 2024-04-24T08:11:28.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.456+0000 7fef34a00640 1 -- 172.21.15.69:0/1424413424 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef24032810 con 0x7fef380b32d0 2024-04-24T08:11:28.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.456+0000 7fef3cc00640 1 -- 172.21.15.69:0/1424413424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef380b32d0 msgr2=0x7fef3815f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:28.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.456+0000 7fef3cc00640 1 --2- 172.21.15.69:0/1424413424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef380b32d0 0x7fef3815f590 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fef24009870 tx=0x7fef2402a250 comp rx=0 tx=0).stop 2024-04-24T08:11:28.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.456+0000 7fef3cc00640 1 -- 172.21.15.69:0/1424413424 shutdown_connections 2024-04-24T08:11:28.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.456+0000 7fef3cc00640 1 --2- 172.21.15.69:0/1424413424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef3815fad0 0x7fef380b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:28.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.456+0000 7fef3cc00640 1 --2- 172.21.15.69:0/1424413424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef380b32d0 0x7fef3815f590 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:28.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.456+0000 7fef3cc00640 1 -- 172.21.15.69:0/1424413424 >> 172.21.15.69:0/1424413424 conn(0x7fef380b2380 msgr2=0x7fef380b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:28.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.456+0000 7fef3cc00640 1 -- 172.21.15.69:0/1424413424 shutdown_connections 2024-04-24T08:11:28.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.456+0000 7fef3cc00640 1 -- 172.21.15.69:0/1424413424 wait complete. 2024-04-24T08:11:28.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.458+0000 7fef3cc00640 1 Processor -- start 2024-04-24T08:11:28.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.458+0000 7fef3cc00640 1 -- start start 2024-04-24T08:11:28.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.459+0000 7fef3cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef3806f5e0 0x7fef3815b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:28.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.459+0000 7fef3cc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef380b32d0 0x7fef381596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:28.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.459+0000 7fef3cc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef3815c7b0 0x7fef381e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:28.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.459+0000 7fef3cc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fef380b4b30 con 0x7fef3806f5e0 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.459+0000 7fef3cc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fef3815f070 con 0x7fef380b32d0 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.459+0000 7fef3cc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fef380b4cb0 con 0x7fef3815c7b0 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef35e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef3806f5e0 0x7fef3815b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef35e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef3806f5e0 0x7fef3815b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45210/0 (socket says 172.21.15.69:45210) 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef35e00640 1 -- 172.21.15.69:0/3440217189 learned_addr learned my addr 172.21.15.69:0/3440217189 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef35400640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef380b32d0 0x7fef381596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef36800640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef3815c7b0 0x7fef381e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef35e00640 1 -- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef3815c7b0 msgr2=0x7fef381e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef35e00640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef3815c7b0 0x7fef381e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef35e00640 1 -- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef380b32d0 msgr2=0x7fef381596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef35e00640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef380b32d0 0x7fef381596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef35e00640 1 -- 172.21.15.69:0/3440217189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef38159c30 con 0x7fef3806f5e0 2024-04-24T08:11:28.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.460+0000 7fef35e00640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef3806f5e0 0x7fef3815b9e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fef2800c430 tx=0x7fef2800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:28.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.461+0000 7fef2e000640 1 -- 172.21.15.69:0/3440217189 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef280052d0 con 0x7fef3806f5e0 2024-04-24T08:11:28.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.461+0000 7fef2e000640 1 -- 172.21.15.69:0/3440217189 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef28005490 con 0x7fef3806f5e0 2024-04-24T08:11:28.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.461+0000 7fef2e000640 1 -- 172.21.15.69:0/3440217189 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28007260 con 0x7fef3806f5e0 2024-04-24T08:11:28.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.461+0000 7fef3cc00640 1 -- 172.21.15.69:0/3440217189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef381e8720 con 0x7fef3806f5e0 2024-04-24T08:11:28.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.461+0000 7fef3cc00640 1 -- 172.21.15.69:0/3440217189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef3815af00 con 0x7fef3806f5e0 2024-04-24T08:11:28.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.462+0000 7fef3cc00640 1 -- 172.21.15.69:0/3440217189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef3815acf0 con 0x7fef3806f5e0 2024-04-24T08:11:28.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.463+0000 7fef2e000640 1 -- 172.21.15.69:0/3440217189 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fef28007400 con 0x7fef3806f5e0 2024-04-24T08:11:28.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.464+0000 7fef2e000640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fef1806c8f0 0x7fef1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:28.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.465+0000 7fef2e000640 1 -- 172.21.15.69:0/3440217189 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5853+0+0 (secure 0 0 0) 0x7fef2808b130 con 0x7fef3806f5e0 2024-04-24T08:11:28.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.465+0000 7fef35400640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fef1806c8f0 0x7fef1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:28.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.466+0000 7fef35400640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fef1806c8f0 0x7fef1806edb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fef24034000 tx=0x7fef2402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:28.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.469+0000 7fef2e000640 1 -- 172.21.15.69:0/3440217189 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef28058de0 con 0x7fef3806f5e0 2024-04-24T08:11:28.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:28.669+0000 7fef3cc00640 1 -- 172.21.15.69:0/3440217189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fef38068a80 con 0x7fef3806f5e0 2024-04-24T08:11:29.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.228+0000 7fef2e000640 1 -- 172.21.15.69:0/3440217189 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v48) ==== 181+0+0 (secure 0 0 0) 0x7fef2805d0e0 con 0x7fef3806f5e0 2024-04-24T08:11:29.230 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' removed 2024-04-24T08:11:29.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.232+0000 7fef3cc00640 1 -- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fef1806c8f0 msgr2=0x7fef1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:29.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.232+0000 7fef3cc00640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fef1806c8f0 0x7fef1806edb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fef24034000 tx=0x7fef2402e000 comp rx=0 tx=0).stop 2024-04-24T08:11:29.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.232+0000 7fef3cc00640 1 -- 172.21.15.69:0/3440217189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef3806f5e0 msgr2=0x7fef3815b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:29.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.232+0000 7fef3cc00640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef3806f5e0 0x7fef3815b9e0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fef2800c430 tx=0x7fef2800c8f0 comp rx=0 tx=0).stop 2024-04-24T08:11:29.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.234+0000 7fef3cc00640 1 -- 172.21.15.69:0/3440217189 shutdown_connections 2024-04-24T08:11:29.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.234+0000 7fef3cc00640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fef1806c8f0 0x7fef1806edb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:29.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.234+0000 7fef3cc00640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fef3815c7b0 0x7fef381e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:29.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.234+0000 7fef3cc00640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef380b32d0 0x7fef381596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:29.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.234+0000 7fef3cc00640 1 --2- 172.21.15.69:0/3440217189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef3806f5e0 0x7fef3815b9e0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:29.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.234+0000 7fef3cc00640 1 -- 172.21.15.69:0/3440217189 >> 172.21.15.69:0/3440217189 conn(0x7fef380b2380 msgr2=0x7fef380b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:29.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.235+0000 7fef3cc00640 1 -- 172.21.15.69:0/3440217189 shutdown_connections 2024-04-24T08:11:29.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.235+0000 7fef3cc00640 1 -- 172.21.15.69:0/3440217189 wait complete. 2024-04-24T08:11:29.258 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T08:11:29.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.448+0000 7f1f68400640 1 Processor -- start 2024-04-24T08:11:29.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.449+0000 7f1f68400640 1 -- start start 2024-04-24T08:11:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.451+0000 7f1f68400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f600b4d40 0x7f1f6006cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.451+0000 7f1f68400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f6006e8a0 con 0x7f1f6006ca60 2024-04-24T08:11:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.451+0000 7f1f68400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f6006eb80 con 0x7f1f6015e740 2024-04-24T08:11:29.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.451+0000 7f1f68400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6006e650 con 0x7f1f600b4d40 2024-04-24T08:11:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.451+0000 7f1f65000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1f6006ca60 0x7f1f600b4630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48836/0 (socket says 172.21.15.69:48836) 2024-04-24T08:11:29.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.451+0000 7f1f65000640 1 -- 172.21.15.69:0/781180845 learned_addr learned my addr 172.21.15.69:0/781180845 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:29.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.451+0000 7f1f66400640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f600b4d40 0x7f1f6006cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.451+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2724142819 0 0) 0x7f1f6006e8a0 con 0x7f1f6006ca60 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.451+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1f48003520 con 0x7f1f6006ca60 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 448778108 0 0) 0x7f1f48003520 con 0x7f1f6006ca60 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f600b4d40 msgr2=0x7f1f6006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f600b4d40 0x7f1f6006cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 >> v1:172.21.15.121:6789/0 conn(0x7f1f6015e740 legacy=0x7f1f600adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6006ee60 con 0x7f1f6006ca60 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f1f4c001670 con 0x7f1f6006ca60 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 >> v1:172.21.15.69:6789/0 conn(0x7f1f6006ca60 legacy=0x7f1f600b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f480039e0 0x7f1f48003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 0x7f1f48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f600b4d40 0x7f1f480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6006e650 con 0x7f1f48004800 2024-04-24T08:11:29.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f4800baa0 con 0x7f1f480039e0 2024-04-24T08:11:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f4800bca0 con 0x7f1f600b4d40 2024-04-24T08:11:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.452+0000 7f1f65000640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 0x7f1f48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.453+0000 7f1f66400640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f600b4d40 0x7f1f480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.453+0000 7f1f65a00640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f480039e0 0x7f1f48003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.453+0000 7f1f65000640 1 -- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f600b4d40 msgr2=0x7f1f480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.453+0000 7f1f65000640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f600b4d40 0x7f1f480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.453+0000 7f1f65000640 1 -- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f480039e0 msgr2=0x7f1f48003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.453+0000 7f1f65000640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f480039e0 0x7f1f48003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:29.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.453+0000 7f1f65000640 1 -- 172.21.15.69:0/781180845 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f48003520 con 0x7f1f48004800 2024-04-24T08:11:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.454+0000 7f1f65000640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 0x7f1f48004c80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c00c210 tx=0x7f1f48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a396df2cfdd33f00 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.454+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c007cf0 con 0x7f1f48004800 2024-04-24T08:11:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.454+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f4c005ad0 con 0x7f1f48004800 2024-04-24T08:11:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.454+0000 7f1f5fe00640 1 -- 172.21.15.69:0/781180845 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4c007070 con 0x7f1f48004800 2024-04-24T08:11:29.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.455+0000 7f1f68400640 1 -- 172.21.15.69:0/781180845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 msgr2=0x7f1f48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:29.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.455+0000 7f1f68400640 1 --2- 172.21.15.69:0/781180845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 0x7f1f48004c80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c00c210 tx=0x7f1f48005830 comp rx=0 tx=0).stop 2024-04-24T08:11:29.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.455+0000 7f1f66400640 1 -- 172.21.15.69:0/781180845 reap_dead start 2024-04-24T08:11:29.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.455+0000 7f1f68400640 1 -- 172.21.15.69:0/781180845 shutdown_connections 2024-04-24T08:11:29.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.455+0000 7f1f68400640 1 -- 172.21.15.69:0/781180845 >> 172.21.15.69:0/781180845 conn(0x7f1f600b3230 msgr2=0x7f1f600d0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:29.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.455+0000 7f1f68400640 1 -- 172.21.15.69:0/781180845 shutdown_connections 2024-04-24T08:11:29.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.455+0000 7f1f68400640 1 -- 172.21.15.69:0/781180845 wait complete. 2024-04-24T08:11:29.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.457+0000 7f1f68400640 1 Processor -- start 2024-04-24T08:11:29.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.457+0000 7f1f68400640 1 -- start start 2024-04-24T08:11:29.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f68400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 0x7f1f60141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:29.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f68400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f60160450 0x7f1f601608d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:29.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f68400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f6015cd40 0x7f1f6015d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:29.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f68400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6015eb20 con 0x7f1f48004800 2024-04-24T08:11:29.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f68400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6006f420 con 0x7f1f6015cd40 2024-04-24T08:11:29.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f68400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6006cef0 con 0x7f1f60160450 2024-04-24T08:11:29.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f65a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 0x7f1f60141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:29.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f65a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 0x7f1f60141740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:32996/0 (socket says 172.21.15.69:32996) 2024-04-24T08:11:29.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f65a00640 1 -- 172.21.15.69:0/2697119479 learned_addr learned my addr 172.21.15.69:0/2697119479 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:29.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f65000640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f60160450 0x7f1f601608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:29.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f66400640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f6015cd40 0x7f1f6015d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:29.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f65a00640 1 -- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f60160450 msgr2=0x7f1f601608d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:29.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f65a00640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f60160450 0x7f1f601608d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:29.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f65a00640 1 -- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f6015cd40 msgr2=0x7f1f6015d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:29.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f65a00640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f6015cd40 0x7f1f6015d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:29.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.458+0000 7f1f65a00640 1 -- 172.21.15.69:0/2697119479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f60141c80 con 0x7f1f48004800 2024-04-24T08:11:29.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.459+0000 7f1f65a00640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 0x7f1f60141740 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400e8c0 tx=0x7f1f5400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:29.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.459+0000 7f1f5d600640 1 -- 172.21.15.69:0/2697119479 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54003b80 con 0x7f1f48004800 2024-04-24T08:11:29.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.459+0000 7f1f5d600640 1 -- 172.21.15.69:0/2697119479 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f54018070 con 0x7f1f48004800 2024-04-24T08:11:29.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.459+0000 7f1f68400640 1 -- 172.21.15.69:0/2697119479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f60160ec0 con 0x7f1f48004800 2024-04-24T08:11:29.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.459+0000 7f1f5d600640 1 -- 172.21.15.69:0/2697119479 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f540050d0 con 0x7f1f48004800 2024-04-24T08:11:29.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.459+0000 7f1f68400640 1 -- 172.21.15.69:0/2697119479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f60161480 con 0x7f1f48004800 2024-04-24T08:11:29.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.460+0000 7f1f68400640 1 -- 172.21.15.69:0/2697119479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f60069190 con 0x7f1f48004800 2024-04-24T08:11:29.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.461+0000 7f1f5d600640 1 -- 172.21.15.69:0/2697119479 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f54005270 con 0x7f1f48004800 2024-04-24T08:11:29.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.462+0000 7f1f5d600640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4006c920 0x7f1f4006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:29.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.462+0000 7f1f5d600640 1 -- 172.21.15.69:0/2697119479 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5415+0+0 (secure 0 0 0) 0x7f1f5408bb80 con 0x7f1f48004800 2024-04-24T08:11:29.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.463+0000 7f1f65000640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4006c920 0x7f1f4006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:29.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.464+0000 7f1f65000640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4006c920 0x7f1f4006ede0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c032000 tx=0x7f1f4c009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:29.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.466+0000 7f1f5d600640 1 -- 172.21.15.69:0/2697119479 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f5401a310 con 0x7f1f48004800 2024-04-24T08:11:29.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:29.669+0000 7f1f68400640 1 -- 172.21.15.69:0/2697119479 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1f60153670 con 0x7f1f48004800 2024-04-24T08:11:30.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.232+0000 7f1f5d600640 1 -- 172.21.15.69:0/2697119479 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v49) ==== 169+0+0 (secure 0 0 0) 0x7f1f5405a030 con 0x7f1f48004800 2024-04-24T08:11:30.234 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' removed 2024-04-24T08:11:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 -- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4006c920 msgr2=0x7f1f4006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4006c920 0x7f1f4006ede0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c032000 tx=0x7f1f4c009030 comp rx=0 tx=0).stop 2024-04-24T08:11:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 -- 172.21.15.69:0/2697119479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 msgr2=0x7f1f60141740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 0x7f1f60141740 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1f5400e8c0 tx=0x7f1f5400ed80 comp rx=0 tx=0).stop 2024-04-24T08:11:30.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 -- 172.21.15.69:0/2697119479 shutdown_connections 2024-04-24T08:11:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4006c920 0x7f1f4006ede0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f6015cd40 0x7f1f6015d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f60160450 0x7f1f601608d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 --2- 172.21.15.69:0/2697119479 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f48004800 0x7f1f60141740 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 -- 172.21.15.69:0/2697119479 >> 172.21.15.69:0/2697119479 conn(0x7f1f600b3230 msgr2=0x7f1f60152320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 -- 172.21.15.69:0/2697119479 shutdown_connections 2024-04-24T08:11:30.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.236+0000 7f1f68400640 1 -- 172.21.15.69:0/2697119479 wait complete. 2024-04-24T08:11:30.266 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-24T08:11:30.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.462+0000 7f5d68e00640 1 Processor -- start 2024-04-24T08:11:30.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.462+0000 7f5d68e00640 1 -- start start 2024-04-24T08:11:30.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.464+0000 7f5d68e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5d64155f20 0x7f5d640ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.464+0000 7f5d68e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d6415c9b0 0x7f5d640b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.464+0000 7f5d68e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d640b3f70 0x7f5d640cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.464+0000 7f5d68e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5d6406db90 con 0x7f5d640b3f70 2024-04-24T08:11:30.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.464+0000 7f5d68e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5d64072070 con 0x7f5d64155f20 2024-04-24T08:11:30.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.464+0000 7f5d68e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5d640cd6d0 con 0x7f5d6415c9b0 2024-04-24T08:11:30.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.464+0000 7f5d62800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d640b3f70 0x7f5d640cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.464+0000 7f5d62800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d640b3f70 0x7f5d640cf360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33006/0 (socket says 172.21.15.69:33006) 2024-04-24T08:11:30.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.464+0000 7f5d62800640 1 -- 172.21.15.69:0/3570651399 learned_addr learned my addr 172.21.15.69:0/3570651399 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:30.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.465+0000 7f5d59400640 1 --2- 172.21.15.69:0/3570651399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d6415c9b0 0x7f5d640b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.465+0000 7f5d61e00640 1 --2- 172.21.15.69:0/3570651399 >> v2:172.21.15.121:3300/0 conn(0x7f5d64155f20 0x7f5d640ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.465+0000 7f5d62800640 1 -- 172.21.15.69:0/3570651399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d6415c9b0 msgr2=0x7f5d640b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:30.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.465+0000 7f5d62800640 1 --2- 172.21.15.69:0/3570651399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d6415c9b0 0x7f5d640b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.465+0000 7f5d62800640 1 -- 172.21.15.69:0/3570651399 >> v2:172.21.15.121:3300/0 conn(0x7f5d64155f20 msgr2=0x7f5d640ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:30.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.465+0000 7f5d62800640 1 --2- 172.21.15.69:0/3570651399 >> v2:172.21.15.121:3300/0 conn(0x7f5d64155f20 0x7f5d640ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.465+0000 7f5d62800640 1 -- 172.21.15.69:0/3570651399 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d640cd9b0 con 0x7f5d640b3f70 2024-04-24T08:11:30.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.465+0000 7f5d59400640 1 --2- 172.21.15.69:0/3570651399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d6415c9b0 0x7f5d640b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:30.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.466+0000 7f5d62800640 1 --2- 172.21.15.69:0/3570651399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d640b3f70 0x7f5d640cf360 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5d64159870 tx=0x7f5d5c02c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df018d1841bc803 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:30.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.466+0000 7f5d61400640 1 -- 172.21.15.69:0/3570651399 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c038070 con 0x7f5d640b3f70 2024-04-24T08:11:30.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.466+0000 7f5d61400640 1 -- 172.21.15.69:0/3570651399 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d5c031710 con 0x7f5d640b3f70 2024-04-24T08:11:30.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.466+0000 7f5d61400640 1 -- 172.21.15.69:0/3570651399 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c031ae0 con 0x7f5d640b3f70 2024-04-24T08:11:30.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.467+0000 7f5d68e00640 1 -- 172.21.15.69:0/3570651399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d640b3f70 msgr2=0x7f5d640cf360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.467+0000 7f5d68e00640 1 --2- 172.21.15.69:0/3570651399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d640b3f70 0x7f5d640cf360 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5d64159870 tx=0x7f5d5c02c480 comp rx=0 tx=0).stop 2024-04-24T08:11:30.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.467+0000 7f5d68e00640 1 -- 172.21.15.69:0/3570651399 shutdown_connections 2024-04-24T08:11:30.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.467+0000 7f5d68e00640 1 --2- 172.21.15.69:0/3570651399 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d640b3f70 0x7f5d640cf360 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.467+0000 7f5d68e00640 1 --2- 172.21.15.69:0/3570651399 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d6415c9b0 0x7f5d640b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.467+0000 7f5d68e00640 1 --2- 172.21.15.69:0/3570651399 >> v2:172.21.15.121:3300/0 conn(0x7f5d64155f20 0x7f5d640ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.467+0000 7f5d68e00640 1 -- 172.21.15.69:0/3570651399 >> 172.21.15.69:0/3570651399 conn(0x7f5d640d0350 msgr2=0x7f5d640d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:30.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.467+0000 7f5d68e00640 1 -- 172.21.15.69:0/3570651399 shutdown_connections 2024-04-24T08:11:30.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.467+0000 7f5d68e00640 1 -- 172.21.15.69:0/3570651399 wait complete. 2024-04-24T08:11:30.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.469+0000 7f5d68e00640 1 Processor -- start 2024-04-24T08:11:30.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.469+0000 7f5d68e00640 1 -- start start 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d68e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d640b3f70 0x7f5d64154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d68e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d64155f20 0x7f5d641f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d68e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d6415c9b0 0x7f5d6415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d68e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5d640cd2a0 con 0x7f5d6415c9b0 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d68e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5d64071d00 con 0x7f5d64155f20 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d68e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5d640cd420 con 0x7f5d640b3f70 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d62800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d6415c9b0 0x7f5d6415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d62800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d6415c9b0 0x7f5d6415bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33012/0 (socket says 172.21.15.69:33012) 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d62800640 1 -- 172.21.15.69:0/3790566025 learned_addr learned my addr 172.21.15.69:0/3790566025 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d62800640 1 -- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d640b3f70 msgr2=0x7f5d64154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d59400640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d64155f20 0x7f5d641f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d62800640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d640b3f70 0x7f5d64154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d62800640 1 -- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d64155f20 msgr2=0x7f5d641f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d62800640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d64155f20 0x7f5d641f41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.470+0000 7f5d62800640 1 -- 172.21.15.69:0/3790566025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d640ad570 con 0x7f5d6415c9b0 2024-04-24T08:11:30.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.471+0000 7f5d62800640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d6415c9b0 0x7f5d6415bc80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0090b0 tx=0x7f5d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:30.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.471+0000 7f5d5aa00640 1 -- 172.21.15.69:0/3790566025 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c038040 con 0x7f5d6415c9b0 2024-04-24T08:11:30.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.471+0000 7f5d5aa00640 1 -- 172.21.15.69:0/3790566025 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d5c033830 con 0x7f5d6415c9b0 2024-04-24T08:11:30.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.471+0000 7f5d5aa00640 1 -- 172.21.15.69:0/3790566025 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c033c20 con 0x7f5d6415c9b0 2024-04-24T08:11:30.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.471+0000 7f5d68e00640 1 -- 172.21.15.69:0/3790566025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d640ad850 con 0x7f5d6415c9b0 2024-04-24T08:11:30.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.471+0000 7f5d68e00640 1 -- 172.21.15.69:0/3790566025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d640ade10 con 0x7f5d6415c9b0 2024-04-24T08:11:30.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.472+0000 7f5d68e00640 1 -- 172.21.15.69:0/3790566025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d64068a80 con 0x7f5d6415c9b0 2024-04-24T08:11:30.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.473+0000 7f5d5aa00640 1 -- 172.21.15.69:0/3790566025 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d5c03e430 con 0x7f5d6415c9b0 2024-04-24T08:11:30.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.474+0000 7f5d5aa00640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d4006c8f0 0x7f5d4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.474+0000 7f5d5aa00640 1 -- 172.21.15.69:0/3790566025 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f5d5c0ac500 con 0x7f5d6415c9b0 2024-04-24T08:11:30.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.475+0000 7f5d61e00640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d4006c8f0 0x7f5d4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.476+0000 7f5d61e00640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d4006c8f0 0x7f5d4006edb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5d54007130 tx=0x7f5d54007310 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:30.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.479+0000 7f5d5aa00640 1 -- 172.21.15.69:0/3790566025 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d5c07aba0 con 0x7f5d6415c9b0 2024-04-24T08:11:30.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.688+0000 7f5d68e00640 1 -- 172.21.15.69:0/3790566025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f5d640cf660 con 0x7f5d6415c9b0 2024-04-24T08:11:30.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.695+0000 7f5d5aa00640 1 -- 172.21.15.69:0/3790566025 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7f5d5c07a4a0 con 0x7f5d6415c9b0 2024-04-24T08:11:30.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 -- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d4006c8f0 msgr2=0x7f5d4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d4006c8f0 0x7f5d4006edb0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5d54007130 tx=0x7f5d54007310 comp rx=0 tx=0).stop 2024-04-24T08:11:30.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 -- 172.21.15.69:0/3790566025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d6415c9b0 msgr2=0x7f5d6415bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d6415c9b0 0x7f5d6415bc80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c0090b0 tx=0x7f5d5c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 -- 172.21.15.69:0/3790566025 shutdown_connections 2024-04-24T08:11:30.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d4006c8f0 0x7f5d4006edb0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d6415c9b0 0x7f5d6415bc80 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d64155f20 0x7f5d641f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 --2- 172.21.15.69:0/3790566025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d640b3f70 0x7f5d64154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 -- 172.21.15.69:0/3790566025 >> 172.21.15.69:0/3790566025 conn(0x7f5d640d0350 msgr2=0x7f5d640ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:30.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 -- 172.21.15.69:0/3790566025 shutdown_connections 2024-04-24T08:11:30.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.698+0000 7f5d68e00640 1 -- 172.21.15.69:0/3790566025 wait complete. 2024-04-24T08:11:30.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-04-24T08:11:30.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.912+0000 7f1022400640 1 Processor -- start 2024-04-24T08:11:30.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.912+0000 7f1022400640 1 -- start start 2024-04-24T08:11:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f1022400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f101c15f520 0x7f101c15f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f1022400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f101c06ba20 0x7f101c0b4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f1022400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f101c06d350 con 0x7f101c0b4ad0 2024-04-24T08:11:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f1022400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f101c0b4f20 con 0x7f101c15f520 2024-04-24T08:11:30.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f1022400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f101c06c2b0 con 0x7f101c06ba20 2024-04-24T08:11:30.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f101b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f101c15f520 0x7f101c15f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f101be00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f101c0b4ad0 0x7f101c162380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48850/0 (socket says 172.21.15.69:48850) 2024-04-24T08:11:30.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f101be00640 1 -- 172.21.15.69:0/3638874059 learned_addr learned my addr 172.21.15.69:0/3638874059 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:30.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f101b400640 1 -- 172.21.15.69:0/3638874059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f101c06ba20 msgr2=0x7f101c0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:30.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f101aa00640 1 --2- 172.21.15.69:0/3638874059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f101c06ba20 0x7f101c0b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.914+0000 7f101a000640 1 -- 172.21.15.69:0/3638874059 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1901469559 0 0) 0x7f101c06d350 con 0x7f101c0b4ad0 2024-04-24T08:11:30.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.915+0000 7f101b400640 1 --2- 172.21.15.69:0/3638874059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f101c06ba20 0x7f101c0b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.915+0000 7f101b400640 1 -- 172.21.15.69:0/3638874059 >> v1:172.21.15.69:6789/0 conn(0x7f101c0b4ad0 legacy=0x7f101c162380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.915+0000 7f101b400640 1 -- 172.21.15.69:0/3638874059 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101c06c590 con 0x7f101c15f520 2024-04-24T08:11:30.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.915+0000 7f101aa00640 1 --2- 172.21.15.69:0/3638874059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f101c06ba20 0x7f101c0b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:30.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.916+0000 7f101b400640 1 --2- 172.21.15.69:0/3638874059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 0x7f101c15f900 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f100c0097a0 tx=0x7f100c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4723075afce500df server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:30.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.916+0000 7f101a000640 1 -- 172.21.15.69:0/3638874059 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c037070 con 0x7f101c15f520 2024-04-24T08:11:30.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.916+0000 7f101a000640 1 -- 172.21.15.69:0/3638874059 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f100c032440 con 0x7f101c15f520 2024-04-24T08:11:30.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.916+0000 7f101a000640 1 -- 172.21.15.69:0/3638874059 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c032810 con 0x7f101c15f520 2024-04-24T08:11:30.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.917+0000 7f1022400640 1 -- 172.21.15.69:0/3638874059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 msgr2=0x7f101c15f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.917+0000 7f1022400640 1 --2- 172.21.15.69:0/3638874059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 0x7f101c15f900 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f100c0097a0 tx=0x7f100c02a180 comp rx=0 tx=0).stop 2024-04-24T08:11:30.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.917+0000 7f1022400640 1 -- 172.21.15.69:0/3638874059 shutdown_connections 2024-04-24T08:11:30.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.917+0000 7f1022400640 1 --2- 172.21.15.69:0/3638874059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f101c06ba20 0x7f101c0b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.917+0000 7f1022400640 1 --2- 172.21.15.69:0/3638874059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 0x7f101c15f900 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.917+0000 7f1022400640 1 -- 172.21.15.69:0/3638874059 >> 172.21.15.69:0/3638874059 conn(0x7f101c0b3230 msgr2=0x7f101c0d0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:30.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.917+0000 7f1022400640 1 -- 172.21.15.69:0/3638874059 shutdown_connections 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.917+0000 7f1022400640 1 -- 172.21.15.69:0/3638874059 wait complete. 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.919+0000 7f1022400640 1 Processor -- start 2024-04-24T08:11:30.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.919+0000 7f1022400640 1 -- start start 2024-04-24T08:11:30.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f1022400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f101c06ba20 0x7f101c14e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f1022400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f101c0b4ad0 0x7f101c1f4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f1022400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 0x7f101c0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f1022400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f101c06ce30 con 0x7f101c15f520 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f1022400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f101c06cb50 con 0x7f101c0b4ad0 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f1022400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f101c06f890 con 0x7f101c06ba20 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f101be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 0x7f101c0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f101be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 0x7f101c0aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33032/0 (socket says 172.21.15.69:33032) 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f101be00640 1 -- 172.21.15.69:0/3096883955 learned_addr learned my addr 172.21.15.69:0/3096883955 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f101be00640 1 -- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f101c06ba20 msgr2=0x7f101c14e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f101be00640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f101c06ba20 0x7f101c14e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f101be00640 1 -- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f101c0b4ad0 msgr2=0x7f101c1f4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f101be00640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f101c0b4ad0 0x7f101c1f4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.920+0000 7f101be00640 1 -- 172.21.15.69:0/3096883955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101c1f5180 con 0x7f101c15f520 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.921+0000 7f101be00640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 0x7f101c0aeb10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f101400c7a0 tx=0x7f101400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.921+0000 7f1013400640 1 -- 172.21.15.69:0/3096883955 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1014018070 con 0x7f101c15f520 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.921+0000 7f1013400640 1 -- 172.21.15.69:0/3096883955 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f101400a640 con 0x7f101c15f520 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.921+0000 7f1022400640 1 -- 172.21.15.69:0/3096883955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101c150fb0 con 0x7f101c15f520 2024-04-24T08:11:30.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.921+0000 7f1013400640 1 -- 172.21.15.69:0/3096883955 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f101400aa70 con 0x7f101c15f520 2024-04-24T08:11:30.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.921+0000 7f1022400640 1 -- 172.21.15.69:0/3096883955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101c151570 con 0x7f101c15f520 2024-04-24T08:11:30.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.922+0000 7f1022400640 1 -- 172.21.15.69:0/3096883955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f101c068a80 con 0x7f101c15f520 2024-04-24T08:11:30.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.923+0000 7f1013400640 1 -- 172.21.15.69:0/3096883955 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f101401b070 con 0x7f101c15f520 2024-04-24T08:11:30.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.924+0000 7f1013400640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ffc06c8f0 0x7f0ffc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:30.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.924+0000 7f1013400640 1 -- 172.21.15.69:0/3096883955 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f101408aef0 con 0x7f101c15f520 2024-04-24T08:11:30.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.925+0000 7f101b400640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ffc06c8f0 0x7f0ffc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:30.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.926+0000 7f101b400640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ffc06c8f0 0x7f0ffc06edb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f100c034000 tx=0x7f100c02c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:30.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:30.928+0000 7f1013400640 1 -- 172.21.15.69:0/3096883955 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f101c1f5180 con 0x7f101c15f520 2024-04-24T08:11:31.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.197+0000 7f1022400640 1 -- 172.21.15.69:0/3096883955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) -- 0x7f101c0d05a0 con 0x7f101c15f520 2024-04-24T08:11:31.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.249+0000 7f1013400640 1 -- 172.21.15.69:0/3096883955 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v181) ==== 149+0+0 (secure 0 0 0) 0x7f1014059580 con 0x7f101c15f520 2024-04-24T08:11:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.251+0000 7f1022400640 1 -- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ffc06c8f0 msgr2=0x7f0ffc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.251+0000 7f1022400640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ffc06c8f0 0x7f0ffc06edb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f100c034000 tx=0x7f100c02c000 comp rx=0 tx=0).stop 2024-04-24T08:11:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.251+0000 7f1022400640 1 -- 172.21.15.69:0/3096883955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 msgr2=0x7f101c0aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.251+0000 7f1022400640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 0x7f101c0aeb10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f101400c7a0 tx=0x7f101400cc60 comp rx=0 tx=0).stop 2024-04-24T08:11:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.252+0000 7f1022400640 1 -- 172.21.15.69:0/3096883955 shutdown_connections 2024-04-24T08:11:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.252+0000 7f1022400640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ffc06c8f0 0x7f0ffc06edb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.252+0000 7f1022400640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f101c15f520 0x7f101c0aeb10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.252+0000 7f1022400640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f101c0b4ad0 0x7f101c1f4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.252+0000 7f1022400640 1 --2- 172.21.15.69:0/3096883955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f101c06ba20 0x7f101c14e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.252+0000 7f1022400640 1 -- 172.21.15.69:0/3096883955 >> 172.21.15.69:0/3096883955 conn(0x7f101c0b3230 msgr2=0x7f101c0b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.252+0000 7f1022400640 1 -- 172.21.15.69:0/3096883955 shutdown_connections 2024-04-24T08:11:31.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.252+0000 7f1022400640 1 -- 172.21.15.69:0/3096883955 wait complete. 2024-04-24T08:11:31.272 INFO:tasks.cephfs_test_runner:That the MDS can repair an inodes backtrace in the data pool ... ok 2024-04-24T08:11:31.273 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-04-24T08:11:31.273 INFO:tasks.cephfs_test_runner:test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-24T08:11:31.273 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:11:31.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.464+0000 7f181f200640 1 Processor -- start 2024-04-24T08:11:31.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.464+0000 7f181f200640 1 -- start start 2024-04-24T08:11:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.465+0000 7f181f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f182015d1d0 0x7f18200ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.465+0000 7f181f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18200b30f0 0x7f18200ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.465+0000 7f181f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f182015d660 con 0x7f1820156c60 2024-04-24T08:11:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.466+0000 7f181f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f182006dc30 con 0x7f182015d1d0 2024-04-24T08:11:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.466+0000 7f181f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f182006df10 con 0x7f18200b30f0 2024-04-24T08:11:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.466+0000 7f181d400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f182015d1d0 0x7f18200ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.466+0000 7f181d400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f182015d1d0 0x7f18200ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33044/0 (socket says 172.21.15.69:33044) 2024-04-24T08:11:31.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.466+0000 7f181d400640 1 -- 172.21.15.69:0/764216154 learned_addr learned my addr 172.21.15.69:0/764216154 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:31.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.466+0000 7f181d400640 1 -- 172.21.15.69:0/764216154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18200b30f0 msgr2=0x7f18200ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:31.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.466+0000 7f181d400640 1 --2- 172.21.15.69:0/764216154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18200b30f0 0x7f18200ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.466+0000 7f181d400640 1 -- 172.21.15.69:0/764216154 >> v1:172.21.15.69:6789/0 conn(0x7f1820156c60 legacy=0x7f182015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.466+0000 7f181d400640 1 -- 172.21.15.69:0/764216154 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182006e1f0 con 0x7f182015d1d0 2024-04-24T08:11:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.467+0000 7f181d400640 1 --2- 172.21.15.69:0/764216154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f182015d1d0 0x7f18200ad0d0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f180c009870 tx=0x7f180c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e7b776d5defb85 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181ca00640 1 -- 172.21.15.69:0/764216154 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c037070 con 0x7f182015d1d0 2024-04-24T08:11:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181ca00640 1 -- 172.21.15.69:0/764216154 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f180c032440 con 0x7f182015d1d0 2024-04-24T08:11:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181ca00640 1 -- 172.21.15.69:0/764216154 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180c032810 con 0x7f182015d1d0 2024-04-24T08:11:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181f200640 1 -- 172.21.15.69:0/764216154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f182015d1d0 msgr2=0x7f18200ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181f200640 1 --2- 172.21.15.69:0/764216154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f182015d1d0 0x7f18200ad0d0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f180c009870 tx=0x7f180c02a270 comp rx=0 tx=0).stop 2024-04-24T08:11:31.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181f200640 1 -- 172.21.15.69:0/764216154 shutdown_connections 2024-04-24T08:11:31.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181f200640 1 --2- 172.21.15.69:0/764216154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18200b30f0 0x7f18200ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181f200640 1 --2- 172.21.15.69:0/764216154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f182015d1d0 0x7f18200ad0d0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181f200640 1 -- 172.21.15.69:0/764216154 >> 172.21.15.69:0/764216154 conn(0x7f18200d0160 msgr2=0x7f18200d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:31.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181f200640 1 -- 172.21.15.69:0/764216154 shutdown_connections 2024-04-24T08:11:31.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.468+0000 7f181f200640 1 -- 172.21.15.69:0/764216154 wait complete. 2024-04-24T08:11:31.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.470+0000 7f181f200640 1 Processor -- start 2024-04-24T08:11:31.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.471+0000 7f181f200640 1 -- start start 2024-04-24T08:11:31.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.472+0000 7f181f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18200b30f0 0x7f18200736d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.472+0000 7f181f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1820156c60 0x7f1820099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.472+0000 7f181f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820071be0 0x7f1820155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.472+0000 7f181f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f182006e710 con 0x7f1820071be0 2024-04-24T08:11:31.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.472+0000 7f181f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f182015d940 con 0x7f1820156c60 2024-04-24T08:11:31.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.472+0000 7f181f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f182006e890 con 0x7f18200b30f0 2024-04-24T08:11:31.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.472+0000 7f181e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820071be0 0x7f1820155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:31.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.472+0000 7f181e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820071be0 0x7f1820155100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33060/0 (socket says 172.21.15.69:33060) 2024-04-24T08:11:31.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.472+0000 7f181e800640 1 -- 172.21.15.69:0/1629099321 learned_addr learned my addr 172.21.15.69:0/1629099321 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:31.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.472+0000 7f181e800640 1 -- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18200b30f0 msgr2=0x7f18200736d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f181e800640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18200b30f0 0x7f18200736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f181d400640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1820156c60 0x7f1820099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:31.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f181e800640 1 -- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1820156c60 msgr2=0x7f1820099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f181e800640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1820156c60 0x7f1820099580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f181e800640 1 -- 172.21.15.69:0/1629099321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1820073c10 con 0x7f1820071be0 2024-04-24T08:11:31.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f181e800640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820071be0 0x7f1820155100 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f181800b9e0 tx=0x7f181800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:31.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f1816000640 1 -- 172.21.15.69:0/1629099321 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818019070 con 0x7f1820071be0 2024-04-24T08:11:31.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f1816000640 1 -- 172.21.15.69:0/1629099321 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181800cb20 con 0x7f1820071be0 2024-04-24T08:11:31.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f1816000640 1 -- 172.21.15.69:0/1629099321 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818014440 con 0x7f1820071be0 2024-04-24T08:11:31.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f181f200640 1 -- 172.21.15.69:0/1629099321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1820072910 con 0x7f1820071be0 2024-04-24T08:11:31.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.473+0000 7f181f200640 1 -- 172.21.15.69:0/1629099321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1820072ed0 con 0x7f1820071be0 2024-04-24T08:11:31.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.475+0000 7f181f200640 1 -- 172.21.15.69:0/1629099321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1820075d50 con 0x7f1820071be0 2024-04-24T08:11:31.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.475+0000 7f1816000640 1 -- 172.21.15.69:0/1629099321 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f181801c070 con 0x7f1820071be0 2024-04-24T08:11:31.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.476+0000 7f1816000640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f180006c8f0 0x7f180006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.476+0000 7f1816000640 1 -- 172.21.15.69:0/1629099321 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f181808b550 con 0x7f1820071be0 2024-04-24T08:11:31.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.477+0000 7f181de00640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f180006c8f0 0x7f180006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:31.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.478+0000 7f181de00640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f180006c8f0 0x7f180006edb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1810000f60 tx=0x7f18100079d0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:31.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.481+0000 7f1816000640 1 -- 172.21.15.69:0/1629099321 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1820073c10 con 0x7f1820071be0 2024-04-24T08:11:31.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.664+0000 7f181f200640 1 -- 172.21.15.69:0/1629099321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f182006f9e0 con 0x7f1820071be0 2024-04-24T08:11:31.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.667+0000 7f1816000640 1 -- 172.21.15.69:0/1629099321 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+12998 (secure 0 0 0) 0x7f1818059c70 con 0x7f1820071be0 2024-04-24T08:11:31.668 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:31.668 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":49,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:30.229687+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6837/3424934990":"2024-04-25T08:11:27.213036+0000","172.21.15.121:6835/3424934990":"2024-04-25T08:11:27.213036+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:11:31.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.669+0000 7f181f200640 1 -- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f180006c8f0 msgr2=0x7f180006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.669+0000 7f181f200640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f180006c8f0 0x7f180006edb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1810000f60 tx=0x7f18100079d0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.669+0000 7f181f200640 1 -- 172.21.15.69:0/1629099321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820071be0 msgr2=0x7f1820155100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.669+0000 7f181f200640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820071be0 0x7f1820155100 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f181800b9e0 tx=0x7f181800bea0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.669+0000 7f181f200640 1 -- 172.21.15.69:0/1629099321 shutdown_connections 2024-04-24T08:11:31.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.669+0000 7f181f200640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f180006c8f0 0x7f180006edb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.669+0000 7f181f200640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1820071be0 0x7f1820155100 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.669+0000 7f181f200640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1820156c60 0x7f1820099580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.669+0000 7f181f200640 1 --2- 172.21.15.69:0/1629099321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18200b30f0 0x7f18200736d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.669+0000 7f181f200640 1 -- 172.21.15.69:0/1629099321 >> 172.21.15.69:0/1629099321 conn(0x7f18200d0160 msgr2=0x7f18200b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:31.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.670+0000 7f181f200640 1 -- 172.21.15.69:0/1629099321 shutdown_connections 2024-04-24T08:11:31.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.670+0000 7f181f200640 1 -- 172.21.15.69:0/1629099321 wait complete. 2024-04-24T08:11:31.689 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:11:31.690 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:11:31.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.887+0000 7f925ea00640 1 Processor -- start 2024-04-24T08:11:31.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.887+0000 7f925ea00640 1 -- start start 2024-04-24T08:11:31.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.889+0000 7f925ea00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f925815c820 0x7f92580b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.889+0000 7f925ea00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f92580ce250 0x7f92580ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.889+0000 7f925ea00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925815cc00 con 0x7f92580b46a0 2024-04-24T08:11:31.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.889+0000 7f925ea00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f925806ed30 con 0x7f92580ce250 2024-04-24T08:11:31.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.889+0000 7f925ea00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f92580b4e10 con 0x7f925815c820 2024-04-24T08:11:31.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.890+0000 7f925ca00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f92580ce250 0x7f92580ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:31.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.890+0000 7f9257e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f925815c820 0x7f92580b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:31.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.890+0000 7f925ca00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f92580ce250 0x7f92580ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:57066/0 (socket says 172.21.15.69:57066) 2024-04-24T08:11:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.890+0000 7f925ca00640 1 -- 172.21.15.69:0/3852675102 learned_addr learned my addr 172.21.15.69:0/3852675102 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.891+0000 7f925ca00640 1 -- 172.21.15.69:0/3852675102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f925815c820 msgr2=0x7f92580b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.891+0000 7f925ca00640 1 --2- 172.21.15.69:0/3852675102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f925815c820 0x7f92580b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.891+0000 7f925ca00640 1 -- 172.21.15.69:0/3852675102 >> v1:172.21.15.121:6789/0 conn(0x7f92580b46a0 legacy=0x7f925806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.891+0000 7f925ca00640 1 -- 172.21.15.69:0/3852675102 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92580b50f0 con 0x7f92580ce250 2024-04-24T08:11:31.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.891+0000 7f9257e00640 1 --2- 172.21.15.69:0/3852675102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f925815c820 0x7f92580b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:31.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.892+0000 7f925ca00640 1 --2- 172.21.15.69:0/3852675102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92580ce250 0x7f92580ce6e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9250009e50 tx=0x7f925002a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c29b572bbeb1a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:31.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.892+0000 7f9257400640 1 -- 172.21.15.69:0/3852675102 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250036070 con 0x7f92580ce250 2024-04-24T08:11:31.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.892+0000 7f9257400640 1 -- 172.21.15.69:0/3852675102 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9250031600 con 0x7f92580ce250 2024-04-24T08:11:31.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.892+0000 7f9257400640 1 -- 172.21.15.69:0/3852675102 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500319d0 con 0x7f92580ce250 2024-04-24T08:11:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.893+0000 7f925ea00640 1 -- 172.21.15.69:0/3852675102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92580ce250 msgr2=0x7f92580ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.893+0000 7f925ea00640 1 --2- 172.21.15.69:0/3852675102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92580ce250 0x7f92580ce6e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9250009e50 tx=0x7f925002a730 comp rx=0 tx=0).stop 2024-04-24T08:11:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.893+0000 7f925ea00640 1 -- 172.21.15.69:0/3852675102 shutdown_connections 2024-04-24T08:11:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.893+0000 7f925ea00640 1 --2- 172.21.15.69:0/3852675102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92580ce250 0x7f92580ce6e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.893+0000 7f925ea00640 1 --2- 172.21.15.69:0/3852675102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f925815c820 0x7f92580b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.893+0000 7f925ea00640 1 -- 172.21.15.69:0/3852675102 >> 172.21.15.69:0/3852675102 conn(0x7f92580d0350 msgr2=0x7f92580d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.893+0000 7f925ea00640 1 -- 172.21.15.69:0/3852675102 shutdown_connections 2024-04-24T08:11:31.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.893+0000 7f925ea00640 1 -- 172.21.15.69:0/3852675102 wait complete. 2024-04-24T08:11:31.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.895+0000 7f925ea00640 1 Processor -- start 2024-04-24T08:11:31.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.895+0000 7f925ea00640 1 -- start start 2024-04-24T08:11:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.896+0000 7f925ea00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92580b46a0 0x7f92581f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.896+0000 7f925ea00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92580ce250 0x7f92581f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.896+0000 7f925ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925815c820 0x7f925814b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.896+0000 7f925ea00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f92580b4b00 con 0x7f925815c820 2024-04-24T08:11:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.896+0000 7f925ea00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f925806f2b0 con 0x7f92580b46a0 2024-04-24T08:11:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.896+0000 7f925ea00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f925806fee0 con 0x7f92580ce250 2024-04-24T08:11:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.896+0000 7f925ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925815c820 0x7f925814b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.896+0000 7f925ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925815c820 0x7f925814b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33062/0 (socket says 172.21.15.69:33062) 2024-04-24T08:11:31.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.896+0000 7f925ca00640 1 -- 172.21.15.69:0/3682182685 learned_addr learned my addr 172.21.15.69:0/3682182685 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:31.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f9257e00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92580b46a0 0x7f92581f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f925ca00640 1 -- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92580ce250 msgr2=0x7f92581f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f924f400640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92580ce250 0x7f92581f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f925ca00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92580ce250 0x7f92581f6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f925ca00640 1 -- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92580b46a0 msgr2=0x7f92581f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f925ca00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92580b46a0 0x7f92581f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f925ca00640 1 -- 172.21.15.69:0/3682182685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925814bc70 con 0x7f925815c820 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f9257e00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92580b46a0 0x7f92581f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f925ca00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925815c820 0x7f925814b450 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f9250009f70 tx=0x7f9250004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f9254c00640 1 -- 172.21.15.69:0/3682182685 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250041020 con 0x7f925815c820 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f9254c00640 1 -- 172.21.15.69:0/3682182685 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92500077a0 con 0x7f925815c820 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f925ea00640 1 -- 172.21.15.69:0/3682182685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925814bf50 con 0x7f925815c820 2024-04-24T08:11:31.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.897+0000 7f9254c00640 1 -- 172.21.15.69:0/3682182685 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250036040 con 0x7f925815c820 2024-04-24T08:11:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.898+0000 7f925ea00640 1 -- 172.21.15.69:0/3682182685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925814b990 con 0x7f925815c820 2024-04-24T08:11:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.899+0000 7f925ea00640 1 -- 172.21.15.69:0/3682182685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9258149400 con 0x7f925815c820 2024-04-24T08:11:31.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.899+0000 7f9254c00640 1 -- 172.21.15.69:0/3682182685 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9250044020 con 0x7f925815c820 2024-04-24T08:11:31.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.900+0000 7f9254c00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f923406c8f0 0x7f923406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:31.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.900+0000 7f9254c00640 1 -- 172.21.15.69:0/3682182685 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f92500ac430 con 0x7f925815c820 2024-04-24T08:11:31.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.901+0000 7f9257e00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f923406c8f0 0x7f923406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:31.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.902+0000 7f9257e00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f923406c8f0 0x7f923406edb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9248009a20 tx=0x7f9248008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:31.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:31.905+0000 7f9254c00640 1 -- 172.21.15.69:0/3682182685 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f925004b030 con 0x7f925815c820 2024-04-24T08:11:32.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.105+0000 7f925ea00640 1 -- 172.21.15.69:0/3682182685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f92580cddc0 con 0x7f925815c820 2024-04-24T08:11:32.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.106+0000 7f9254c00640 1 -- 172.21.15.69:0/3682182685 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) ==== 93+0+10 (secure 0 0 0) 0x7f925007ac10 con 0x7f925815c820 2024-04-24T08:11:32.108 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:11:32.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 -- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f923406c8f0 msgr2=0x7f923406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f923406c8f0 0x7f923406edb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9248009a20 tx=0x7f9248008040 comp rx=0 tx=0).stop 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 -- 172.21.15.69:0/3682182685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925815c820 msgr2=0x7f925814b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925815c820 0x7f925814b450 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f9250009f70 tx=0x7f9250004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 -- 172.21.15.69:0/3682182685 shutdown_connections 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f923406c8f0 0x7f923406edb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f925815c820 0x7f925814b450 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92580ce250 0x7f92581f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 --2- 172.21.15.69:0/3682182685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92580b46a0 0x7f92581f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 -- 172.21.15.69:0/3682182685 >> 172.21.15.69:0/3682182685 conn(0x7f92580d0350 msgr2=0x7f92581601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 -- 172.21.15.69:0/3682182685 shutdown_connections 2024-04-24T08:11:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.109+0000 7f925ea00640 1 -- 172.21.15.69:0/3682182685 wait complete. 2024-04-24T08:11:32.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-04-24T08:11:32.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.327+0000 7fbbc1400640 1 Processor -- start 2024-04-24T08:11:32.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.327+0000 7fbbc1400640 1 -- start start 2024-04-24T08:11:32.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.329+0000 7fbbc1400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbbbc0d0300 0x7fbbbc06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:32.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.329+0000 7fbbc1400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc0cf3d0 con 0x7fbbbc145a90 2024-04-24T08:11:32.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.329+0000 7fbbc1400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc0cf6b0 con 0x7fbbbc145fe0 2024-04-24T08:11:32.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.329+0000 7fbbc1400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fbbbc0cf990 con 0x7fbbbc0d0300 2024-04-24T08:11:32.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.329+0000 7fbbb1e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbbbc145a90 0x7fbbbc0d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48866/0 (socket says 172.21.15.69:48866) 2024-04-24T08:11:32.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.329+0000 7fbbb1e00640 1 -- 172.21.15.69:0/3615919773 learned_addr learned my addr 172.21.15.69:0/3615919773 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:32.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbba800640 1 --2- 172.21.15.69:0/3615919773 >> v2:172.21.15.121:3300/0 conn(0x7fbbbc0d0300 0x7fbbbc06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:32.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129267857 0 0) 0x7fbbbc0cf3d0 con 0x7fbbbc145a90 2024-04-24T08:11:32.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbba0003520 con 0x7fbbbc145a90 2024-04-24T08:11:32.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1986837522 0 0) 0x7fbba0003520 con 0x7fbbbc145a90 2024-04-24T08:11:32.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 >> v2:172.21.15.121:3300/0 conn(0x7fbbbc0d0300 msgr2=0x7fbbbc06e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:32.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 --2- 172.21.15.69:0/3615919773 >> v2:172.21.15.121:3300/0 conn(0x7fbbbc0d0300 0x7fbbbc06e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:32.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 >> v1:172.21.15.121:6789/0 conn(0x7fbbbc145fe0 legacy=0x7fbbbc160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:32.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc0cfc70 con 0x7fbbbc145a90 2024-04-24T08:11:32.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fbba4001670 con 0x7fbbbc145a90 2024-04-24T08:11:32.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 >> v1:172.21.15.69:6789/0 conn(0x7fbbbc145a90 legacy=0x7fbbbc0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbba00039b0 0x7fbba0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba0004800 0x7fbba0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbba00051c0 0x7fbba000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc0cf990 con 0x7fbba00039b0 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbba000c110 con 0x7fbba0004800 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.330+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbba000c310 con 0x7fbba00051c0 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.331+0000 7fbbba800640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbba00039b0 0x7fbba0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.331+0000 7fbbb1e00640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba0004800 0x7fbba0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.331+0000 7fbbba800640 1 -- 172.21.15.69:0/3615919773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbba00051c0 msgr2=0x7fbba000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.331+0000 7fbbba800640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbba00051c0 0x7fbba000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.331+0000 7fbbba800640 1 -- 172.21.15.69:0/3615919773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba0004800 msgr2=0x7fbba0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:32.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.331+0000 7fbbba800640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba0004800 0x7fbba0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:32.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.331+0000 7fbbba800640 1 -- 172.21.15.69:0/3615919773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba0003520 con 0x7fbba00039b0 2024-04-24T08:11:32.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.331+0000 7fbbb1e00640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba0004800 0x7fbba0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:32.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.331+0000 7fbbbb200640 1 -- 172.21.15.69:0/3615919773 reap_dead start 2024-04-24T08:11:32.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.332+0000 7fbbba800640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbba00039b0 0x7fbba0003dd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00ba30 tx=0x7fbba0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce28e12a831de4f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:32.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.332+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac030070 con 0x7fbba00039b0 2024-04-24T08:11:32.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.332+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbac02ade0 con 0x7fbba00039b0 2024-04-24T08:11:32.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.332+0000 7fbbb9e00640 1 -- 172.21.15.69:0/3615919773 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac00b600 con 0x7fbba00039b0 2024-04-24T08:11:32.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.333+0000 7fbbc1400640 1 -- 172.21.15.69:0/3615919773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbba00039b0 msgr2=0x7fbba0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:32.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.333+0000 7fbbc1400640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbba00039b0 0x7fbba0003dd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00ba30 tx=0x7fbba0005830 comp rx=0 tx=0).stop 2024-04-24T08:11:32.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.333+0000 7fbbc1400640 1 -- 172.21.15.69:0/3615919773 shutdown_connections 2024-04-24T08:11:32.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.333+0000 7fbbc1400640 1 --2- 172.21.15.69:0/3615919773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbba00039b0 0x7fbba0003dd0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:32.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.333+0000 7fbbc1400640 1 -- 172.21.15.69:0/3615919773 >> 172.21.15.69:0/3615919773 conn(0x7fbbbc067d90 msgr2=0x7fbbbc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:32.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.333+0000 7fbbc1400640 1 -- 172.21.15.69:0/3615919773 shutdown_connections 2024-04-24T08:11:32.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.333+0000 7fbbc1400640 1 -- 172.21.15.69:0/3615919773 wait complete. 2024-04-24T08:11:32.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.335+0000 7fbbc1400640 1 Processor -- start 2024-04-24T08:11:32.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.335+0000 7fbbc1400640 1 -- start start 2024-04-24T08:11:32.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.336+0000 7fbbc1400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba00039b0 0x7fbbbc1e7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:32.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.336+0000 7fbbc1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbbc15c210 0x7fbbbc1e7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:32.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.336+0000 7fbbc1400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbbc1e8500 0x7fbbbc1e5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:32.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.336+0000 7fbbc1400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc1458f0 con 0x7fbbbc15c210 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.336+0000 7fbbc1400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc145770 con 0x7fbba00039b0 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.336+0000 7fbbc1400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbbbc0cdc30 con 0x7fbbbc1e8500 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.336+0000 7fbbb1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbbc15c210 0x7fbbbc1e7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.336+0000 7fbbb1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbbc15c210 0x7fbbbc1e7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33074/0 (socket says 172.21.15.69:33074) 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.336+0000 7fbbb1e00640 1 -- 172.21.15.69:0/1939414263 learned_addr learned my addr 172.21.15.69:0/1939414263 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbba800640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba00039b0 0x7fbbbc1e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbb1e00640 1 -- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbbc1e8500 msgr2=0x7fbbbc1e5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbb1e00640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbbc1e8500 0x7fbbbc1e5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbb1e00640 1 -- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba00039b0 msgr2=0x7fbbbc1e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbb1e00640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba00039b0 0x7fbbbc1e7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:32.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbb1e00640 1 -- 172.21.15.69:0/1939414263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc1e65e0 con 0x7fbbbc15c210 2024-04-24T08:11:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbba800640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba00039b0 0x7fbbbc1e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbb1e00640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbbc15c210 0x7fbbbc1e7fc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fbba400d890 tx=0x7fbba400dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbb3400640 1 -- 172.21.15.69:0/1939414263 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4019070 con 0x7fbbbc15c210 2024-04-24T08:11:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbb3400640 1 -- 172.21.15.69:0/1939414263 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbba40074e0 con 0x7fbbbc15c210 2024-04-24T08:11:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbb3400640 1 -- 172.21.15.69:0/1939414263 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba40049e0 con 0x7fbbbc15c210 2024-04-24T08:11:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.337+0000 7fbbc1400640 1 -- 172.21.15.69:0/1939414263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbbc1e68c0 con 0x7fbbbc15c210 2024-04-24T08:11:32.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.338+0000 7fbbc1400640 1 -- 172.21.15.69:0/1939414263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbbc1e6e80 con 0x7fbbbc15c210 2024-04-24T08:11:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.339+0000 7fbbc1400640 1 -- 172.21.15.69:0/1939414263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbbc1e8920 con 0x7fbbbc15c210 2024-04-24T08:11:32.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.340+0000 7fbbb3400640 1 -- 172.21.15.69:0/1939414263 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbba4004b80 con 0x7fbbbc15c210 2024-04-24T08:11:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.341+0000 7fbbb3400640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbb9806c920 0x7fbb9806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.342+0000 7fbbb3400640 1 -- 172.21.15.69:0/1939414263 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7fbba408b350 con 0x7fbbbc15c210 2024-04-24T08:11:32.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.342+0000 7fbbba800640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbb9806c920 0x7fbb9806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:32.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.343+0000 7fbbba800640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbb9806c920 0x7fbb9806ede0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00ba00 tx=0x7fbbac030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:32.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.346+0000 7fbbb3400640 1 -- 172.21.15.69:0/1939414263 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbba4059950 con 0x7fbbbc15c210 2024-04-24T08:11:32.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:32.617+0000 7fbbc1400640 1 -- 172.21.15.69:0/1939414263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) -- 0x7fbbbc15fbd0 con 0x7fbbbc15c210 2024-04-24T08:11:33.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.264+0000 7fbbb3400640 1 -- 172.21.15.69:0/1939414263 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v183) ==== 172+0+0 (secure 0 0 0) 0x7fbba4059250 con 0x7fbbbc15c210 2024-04-24T08:11:33.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 -- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbb9806c920 msgr2=0x7fbb9806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbb9806c920 0x7fbb9806ede0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00ba00 tx=0x7fbbac030000 comp rx=0 tx=0).stop 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 -- 172.21.15.69:0/1939414263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbbc15c210 msgr2=0x7fbbbc1e7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbbc15c210 0x7fbbbc1e7fc0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fbba400d890 tx=0x7fbba400dd50 comp rx=0 tx=0).stop 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 -- 172.21.15.69:0/1939414263 shutdown_connections 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbb9806c920 0x7fbb9806ede0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbbc15c210 0x7fbbbc1e7fc0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbbc1e8500 0x7fbbbc1e5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 --2- 172.21.15.69:0/1939414263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbba00039b0 0x7fbbbc1e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 -- 172.21.15.69:0/1939414263 >> 172.21.15.69:0/1939414263 conn(0x7fbbbc067d90 msgr2=0x7fbbbc0b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 -- 172.21.15.69:0/1939414263 shutdown_connections 2024-04-24T08:11:33.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.268+0000 7fbbc1400640 1 -- 172.21.15.69:0/1939414263 wait complete. 2024-04-24T08:11:33.291 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-24T08:11:33.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.489+0000 7f10b1a00640 1 Processor -- start 2024-04-24T08:11:33.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.489+0000 7f10b1a00640 1 -- start start 2024-04-24T08:11:33.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.491+0000 7f10b1a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10ac06dd90 0x7f10ac15f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.491+0000 7f10b1a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10ac06e220 con 0x7f10ac071d70 2024-04-24T08:11:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.491+0000 7f10b1a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10ac15e430 con 0x7f10ac15e050 2024-04-24T08:11:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.491+0000 7f10b1a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f10ac15f660 con 0x7f10ac06dd90 2024-04-24T08:11:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.491+0000 7f10a3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10ac06dd90 0x7f10ac15f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.491+0000 7f10b1000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f10ac071d70 0x7f10ac0cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48880/0 (socket says 172.21.15.69:48880) 2024-04-24T08:11:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.491+0000 7f10a3400640 1 --2- 172.21.15.69:0/2114941593 >> v2:172.21.15.69:3300/0 conn(0x7f10ac06dd90 0x7f10ac15f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33078/0 (socket says 172.21.15.69:33078) 2024-04-24T08:11:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.491+0000 7f10b1000640 1 -- 172.21.15.69:0/2114941593 learned_addr learned my addr 172.21.15.69:0/2114941593 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.492+0000 7f10a3400640 1 -- 172.21.15.69:0/2114941593 >> v1:172.21.15.121:6789/0 conn(0x7f10ac15e050 legacy=0x7f10ac06f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.492+0000 7f10a3400640 1 -- 172.21.15.69:0/2114941593 >> v1:172.21.15.69:6789/0 conn(0x7f10ac071d70 legacy=0x7f10ac0cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.492+0000 7f10a3400640 1 -- 172.21.15.69:0/2114941593 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac06e490 con 0x7f10ac06dd90 2024-04-24T08:11:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.493+0000 7f10a3400640 1 --2- 172.21.15.69:0/2114941593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 0x7f10ac15f120 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f10ac15e600 tx=0x7f109402b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb09c3de901046dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.493+0000 7f10ab400640 1 -- 172.21.15.69:0/2114941593 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109400b040 con 0x7f10ac06dd90 2024-04-24T08:11:33.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.493+0000 7f10ab400640 1 -- 172.21.15.69:0/2114941593 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1094030710 con 0x7f10ac06dd90 2024-04-24T08:11:33.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.493+0000 7f10ab400640 1 -- 172.21.15.69:0/2114941593 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094030ae0 con 0x7f10ac06dd90 2024-04-24T08:11:33.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.494+0000 7f10b1a00640 1 -- 172.21.15.69:0/2114941593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 msgr2=0x7f10ac15f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.494+0000 7f10b1a00640 1 --2- 172.21.15.69:0/2114941593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 0x7f10ac15f120 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f10ac15e600 tx=0x7f109402b540 comp rx=0 tx=0).stop 2024-04-24T08:11:33.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.494+0000 7f10b1a00640 1 -- 172.21.15.69:0/2114941593 shutdown_connections 2024-04-24T08:11:33.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.494+0000 7f10b1a00640 1 --2- 172.21.15.69:0/2114941593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 0x7f10ac15f120 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.494+0000 7f10b1a00640 1 -- 172.21.15.69:0/2114941593 >> 172.21.15.69:0/2114941593 conn(0x7f10ac067b80 msgr2=0x7f10ac06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:33.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.494+0000 7f10b1a00640 1 -- 172.21.15.69:0/2114941593 shutdown_connections 2024-04-24T08:11:33.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.494+0000 7f10b1a00640 1 -- 172.21.15.69:0/2114941593 wait complete. 2024-04-24T08:11:33.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.496+0000 7f10b1a00640 1 Processor -- start 2024-04-24T08:11:33.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.496+0000 7f10b1a00640 1 -- start start 2024-04-24T08:11:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10b1a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 0x7f10ac158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10b1a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10ac071d70 0x7f10ac1424b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10b1a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ac15e050 0x7f10ac1f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10b1a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac06ed30 con 0x7f10ac06dd90 2024-04-24T08:11:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10b1a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac06ea50 con 0x7f10ac15e050 2024-04-24T08:11:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10b1a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f10ac06f010 con 0x7f10ac071d70 2024-04-24T08:11:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10abe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 0x7f10ac158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10abe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 0x7f10ac158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33082/0 (socket says 172.21.15.69:33082) 2024-04-24T08:11:33.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10abe00640 1 -- 172.21.15.69:0/3704769853 learned_addr learned my addr 172.21.15.69:0/3704769853 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:33.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10a3400640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10ac071d70 0x7f10ac1424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10abe00640 1 -- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10ac071d70 msgr2=0x7f10ac1424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10abe00640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10ac071d70 0x7f10ac1424b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10abe00640 1 -- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ac15e050 msgr2=0x7f10ac1f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.497+0000 7f10abe00640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ac15e050 0x7f10ac1f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.498+0000 7f10abe00640 1 -- 172.21.15.69:0/3704769853 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac159160 con 0x7f10ac06dd90 2024-04-24T08:11:33.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.498+0000 7f10abe00640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 0x7f10ac158af0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f109c00c8a0 tx=0x7f109c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.498+0000 7f10a8c00640 1 -- 172.21.15.69:0/3704769853 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c003b70 con 0x7f10ac06dd90 2024-04-24T08:11:33.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.498+0000 7f10a8c00640 1 -- 172.21.15.69:0/3704769853 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f109c003d30 con 0x7f10ac06dd90 2024-04-24T08:11:33.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.498+0000 7f10a8c00640 1 -- 172.21.15.69:0/3704769853 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c00aa90 con 0x7f10ac06dd90 2024-04-24T08:11:33.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.498+0000 7f10b1a00640 1 -- 172.21.15.69:0/3704769853 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ac159440 con 0x7f10ac06dd90 2024-04-24T08:11:33.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.498+0000 7f10b1a00640 1 -- 172.21.15.69:0/3704769853 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ac159a00 con 0x7f10ac06dd90 2024-04-24T08:11:33.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.499+0000 7f10b1a00640 1 -- 172.21.15.69:0/3704769853 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ac075d50 con 0x7f10ac06dd90 2024-04-24T08:11:33.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.500+0000 7f10a8c00640 1 -- 172.21.15.69:0/3704769853 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f109c00ac30 con 0x7f10ac06dd90 2024-04-24T08:11:33.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.501+0000 7f10a8c00640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f108806c940 0x7f108806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.501+0000 7f10a8c00640 1 -- 172.21.15.69:0/3704769853 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f109c08b780 con 0x7f10ac06dd90 2024-04-24T08:11:33.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.502+0000 7f10a3400640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f108806c940 0x7f108806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.503+0000 7f10a3400640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f108806c940 0x7f108806ee00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1094035000 tx=0x7f109400c040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.506+0000 7f10a8c00640 1 -- 172.21.15.69:0/3704769853 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f109c059f20 con 0x7f10ac06dd90 2024-04-24T08:11:33.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.711+0000 7f10b1a00640 1 -- 172.21.15.69:0/3704769853 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f10ac0728a0 con 0x7f10ac06dd90 2024-04-24T08:11:33.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.716+0000 7f10a8c00640 1 -- 172.21.15.69:0/3704769853 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) ==== 122+0+0 (secure 0 0 0) 0x7f109c059820 con 0x7f10ac06dd90 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 -- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f108806c940 msgr2=0x7f108806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f108806c940 0x7f108806ee00 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1094035000 tx=0x7f109400c040 comp rx=0 tx=0).stop 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 -- 172.21.15.69:0/3704769853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 msgr2=0x7f10ac158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 0x7f10ac158af0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f109c00c8a0 tx=0x7f109c00cd60 comp rx=0 tx=0).stop 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 -- 172.21.15.69:0/3704769853 shutdown_connections 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f108806c940 0x7f108806ee00 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10ac15e050 0x7f10ac1f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f10ac071d70 0x7f10ac1424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 --2- 172.21.15.69:0/3704769853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10ac06dd90 0x7f10ac158af0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 -- 172.21.15.69:0/3704769853 >> 172.21.15.69:0/3704769853 conn(0x7f10ac067b80 msgr2=0x7f10ac0b0420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 -- 172.21.15.69:0/3704769853 shutdown_connections 2024-04-24T08:11:33.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:33.719+0000 7f10b1a00640 1 -- 172.21.15.69:0/3704769853 wait complete. 2024-04-24T08:11:33.740 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:11:33.740 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:11:33.844 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:11:33.956 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:11:33.957 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:11:33.957 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:11:34.011 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:11:34.012 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:11:34.012 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:11:34.069 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:11:34.069 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:11:34.069 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:11:34.126 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:11:34.126 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:11:34.126 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:11:34.182 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:11:34.182 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-24T08:11:34.183 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:34.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.394+0000 7fa546e00640 1 Processor -- start 2024-04-24T08:11:34.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.394+0000 7fa546e00640 1 -- start start 2024-04-24T08:11:34.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.395+0000 7fa546e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa540156c60 0x7fa54015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.396+0000 7fa546e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54015d660 con 0x7fa5400ad7e0 2024-04-24T08:11:34.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.396+0000 7fa546e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54006dc30 con 0x7fa5400b30f0 2024-04-24T08:11:34.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.396+0000 7fa546e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa54006df10 con 0x7fa540156c60 2024-04-24T08:11:34.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.396+0000 7fa545a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa540156c60 0x7fa54015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.396+0000 7fa546400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa5400ad7e0 0x7fa54006e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48894/0 (socket says 172.21.15.69:48894) 2024-04-24T08:11:34.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.396+0000 7fa546400640 1 -- 172.21.15.69:0/1252244432 learned_addr learned my addr 172.21.15.69:0/1252244432 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:34.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.396+0000 7fa545a00640 1 -- 172.21.15.69:0/1252244432 >> v1:172.21.15.121:6789/0 conn(0x7fa5400b30f0 legacy=0x7fa5400ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.396+0000 7fa545a00640 1 -- 172.21.15.69:0/1252244432 >> v1:172.21.15.69:6789/0 conn(0x7fa5400ad7e0 legacy=0x7fa54006e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.396+0000 7fa545a00640 1 -- 172.21.15.69:0/1252244432 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa54006e1f0 con 0x7fa540156c60 2024-04-24T08:11:34.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.396+0000 7fa53fe00640 1 -- 172.21.15.69:0/1252244432 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3069201250 0 0) 0x7fa54015d660 con 0x7fa5400ad7e0 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.397+0000 7fa545a00640 1 --2- 172.21.15.69:0/1252244432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 0x7fa54015cac0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa5340097a0 tx=0x7fa53402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf603b56b031cfb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.398+0000 7fa53fe00640 1 -- 172.21.15.69:0/1252244432 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534037070 con 0x7fa540156c60 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.398+0000 7fa53fe00640 1 -- 172.21.15.69:0/1252244432 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa534032440 con 0x7fa540156c60 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.398+0000 7fa53fe00640 1 -- 172.21.15.69:0/1252244432 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534032810 con 0x7fa540156c60 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.398+0000 7fa546e00640 1 -- 172.21.15.69:0/1252244432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 msgr2=0x7fa54015cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.398+0000 7fa546e00640 1 --2- 172.21.15.69:0/1252244432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 0x7fa54015cac0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fa5340097a0 tx=0x7fa53402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.398+0000 7fa546e00640 1 -- 172.21.15.69:0/1252244432 shutdown_connections 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.398+0000 7fa546e00640 1 --2- 172.21.15.69:0/1252244432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 0x7fa54015cac0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.398+0000 7fa546e00640 1 -- 172.21.15.69:0/1252244432 >> 172.21.15.69:0/1252244432 conn(0x7fa5400d0160 msgr2=0x7fa5400d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.399+0000 7fa546e00640 1 -- 172.21.15.69:0/1252244432 shutdown_connections 2024-04-24T08:11:34.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.399+0000 7fa546e00640 1 -- 172.21.15.69:0/1252244432 wait complete. 2024-04-24T08:11:34.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.400+0000 7fa546e00640 1 Processor -- start 2024-04-24T08:11:34.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.400+0000 7fa546e00640 1 -- start start 2024-04-24T08:11:34.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.401+0000 7fa546e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5400ad7e0 0x7fa54015dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.401+0000 7fa546e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5400b30f0 0x7fa5401f4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.401+0000 7fa546e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 0x7fa5400cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.401+0000 7fa546e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa54015d000 con 0x7fa540156c60 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.401+0000 7fa546e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa54015d940 con 0x7fa5400b30f0 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.401+0000 7fa546e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa54015d180 con 0x7fa5400ad7e0 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa546400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 0x7fa5400cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa546400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 0x7fa5400cb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33088/0 (socket says 172.21.15.69:33088) 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa546400640 1 -- 172.21.15.69:0/2044926613 learned_addr learned my addr 172.21.15.69:0/2044926613 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa545a00640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5400ad7e0 0x7fa54015dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa545000640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5400b30f0 0x7fa5401f4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa546400640 1 -- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5400ad7e0 msgr2=0x7fa54015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa546400640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5400ad7e0 0x7fa54015dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa546400640 1 -- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5400b30f0 msgr2=0x7fa5401f4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa546400640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5400b30f0 0x7fa5401f4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa546400640 1 -- 172.21.15.69:0/2044926613 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa54015e1c0 con 0x7fa540156c60 2024-04-24T08:11:34.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.402+0000 7fa546400640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 0x7fa5400cb3b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa53800c730 tx=0x7fa53800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.403+0000 7fa53d600640 1 -- 172.21.15.69:0/2044926613 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa538018070 con 0x7fa540156c60 2024-04-24T08:11:34.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.403+0000 7fa53d600640 1 -- 172.21.15.69:0/2044926613 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa53800ce40 con 0x7fa540156c60 2024-04-24T08:11:34.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.403+0000 7fa546e00640 1 -- 172.21.15.69:0/2044926613 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa54015e4a0 con 0x7fa540156c60 2024-04-24T08:11:34.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.403+0000 7fa53d600640 1 -- 172.21.15.69:0/2044926613 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53800a880 con 0x7fa540156c60 2024-04-24T08:11:34.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.403+0000 7fa546e00640 1 -- 172.21.15.69:0/2044926613 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa54015ea60 con 0x7fa540156c60 2024-04-24T08:11:34.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.404+0000 7fa546e00640 1 -- 172.21.15.69:0/2044926613 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa540075d50 con 0x7fa540156c60 2024-04-24T08:11:34.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.405+0000 7fa53d600640 1 -- 172.21.15.69:0/2044926613 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa53801b070 con 0x7fa540156c60 2024-04-24T08:11:34.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.406+0000 7fa53d600640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52006c8f0 0x7fa52006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.406+0000 7fa53d600640 1 -- 172.21.15.69:0/2044926613 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7fa53808aea0 con 0x7fa540156c60 2024-04-24T08:11:34.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.407+0000 7fa545a00640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52006c8f0 0x7fa52006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.408+0000 7fa545a00640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52006c8f0 0x7fa52006edb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa534034000 tx=0x7fa53402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.410+0000 7fa53d600640 1 -- 172.21.15.69:0/2044926613 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa538059530 con 0x7fa540156c60 2024-04-24T08:11:34.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.649+0000 7fa546e00640 1 -- 172.21.15.69:0/2044926613 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa54006f9e0 con 0x7fa540156c60 2024-04-24T08:11:34.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.650+0000 7fa53d600640 1 -- 172.21.15.69:0/2044926613 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+3869 (secure 0 0 0) 0x7fa538058e30 con 0x7fa540156c60 2024-04-24T08:11:34.651 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:34.651 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":27,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4984,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/376722301","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":376722301},{"type":"v1","addr":"172.21.15.69:6835","nonce":376722301}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":4990,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1327624292","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1327624292},{"type":"v1","addr":"172.21.15.121:6836","nonce":1327624292}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5167,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/424759612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":424759612},{"type":"v1","addr":"172.21.15.121:6837","nonce":424759612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/3244002416","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":3244002416},{"type":"v1","addr":"172.21.15.69:6834","nonce":3244002416}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[]} 2024-04-24T08:11:34.651 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 27 2024-04-24T08:11:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.652+0000 7fa546e00640 1 -- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52006c8f0 msgr2=0x7fa52006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.652+0000 7fa546e00640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52006c8f0 0x7fa52006edb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa534034000 tx=0x7fa53402e000 comp rx=0 tx=0).stop 2024-04-24T08:11:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.652+0000 7fa546e00640 1 -- 172.21.15.69:0/2044926613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 msgr2=0x7fa5400cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.652+0000 7fa546e00640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 0x7fa5400cb3b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fa53800c730 tx=0x7fa53800cbf0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.652+0000 7fa546e00640 1 -- 172.21.15.69:0/2044926613 shutdown_connections 2024-04-24T08:11:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.652+0000 7fa546e00640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa52006c8f0 0x7fa52006edb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.652+0000 7fa546e00640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa540156c60 0x7fa5400cb3b0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.652+0000 7fa546e00640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5400b30f0 0x7fa5401f4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.652+0000 7fa546e00640 1 --2- 172.21.15.69:0/2044926613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5400ad7e0 0x7fa54015dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.653+0000 7fa546e00640 1 -- 172.21.15.69:0/2044926613 >> 172.21.15.69:0/2044926613 conn(0x7fa5400d0160 msgr2=0x7fa5400b4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.653+0000 7fa546e00640 1 -- 172.21.15.69:0/2044926613 shutdown_connections 2024-04-24T08:11:34.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.653+0000 7fa546e00640 1 -- 172.21.15.69:0/2044926613 wait complete. 2024-04-24T08:11:34.671 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T08:11:34.671 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-24T08:11:34.672 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T08:11:34.672 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:11:34.672 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T08:11:34.672 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-24T08:11:34.672 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T08:11:34.672 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:11:34.672 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T08:11:34.672 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-24T08:11:34.673 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T08:11:34.673 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:11:34.673 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T08:11:34.673 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-24T08:11:34.673 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T08:11:34.673 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:11:34.696 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T08:11:34.696 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T08:11:34.697 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T08:11:34.697 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T08:11:34.698 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T08:11:34.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T08:11:34.704 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T08:11:34.704 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T08:11:34.745 INFO:tasks.ceph.mds.c:Started 2024-04-24T08:11:34.745 DEBUG:teuthology.parallel:result is None 2024-04-24T08:11:34.750 INFO:tasks.ceph.mds.a:Started 2024-04-24T08:11:34.750 DEBUG:teuthology.parallel:result is None 2024-04-24T08:11:34.751 INFO:tasks.ceph.mds.d:Started 2024-04-24T08:11:34.751 DEBUG:teuthology.parallel:result is None 2024-04-24T08:11:34.757 INFO:tasks.ceph.mds.b:Started 2024-04-24T08:11:34.758 DEBUG:teuthology.parallel:result is None 2024-04-24T08:11:34.758 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-24T08:11:34.893 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:11:34.894 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:11:34.905 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-24T08:11:34.907 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:11:34.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.929+0000 7f09ac200640 1 Processor -- start 2024-04-24T08:11:34.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.929+0000 7f09ac200640 1 -- start start 2024-04-24T08:11:34.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.930+0000 7f09ac200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a41631a0 0x7f09a40af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09ac200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09a40af7d0 0x7f09a406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09ac200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09a40adab0 con 0x7f09a415ea10 2024-04-24T08:11:34.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09ac200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f09a40afc40 con 0x7f09a40af7d0 2024-04-24T08:11:34.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09ac200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09a40add50 con 0x7f09a41631a0 2024-04-24T08:11:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09aa200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09a40af7d0 0x7f09a406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09a8e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a41631a0 0x7f09a40af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09aa200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09a40af7d0 0x7f09a406dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33128/0 (socket says 172.21.15.69:33128) 2024-04-24T08:11:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09aa200640 1 -- 172.21.15.69:0/770698208 learned_addr learned my addr 172.21.15.69:0/770698208 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09a8e00640 1 --2- 172.21.15.69:0/770698208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a41631a0 0x7f09a40af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:43264/0 (socket says 172.21.15.69:43264) 2024-04-24T08:11:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09a8e00640 1 -- 172.21.15.69:0/770698208 >> v2:172.21.15.69:3300/0 conn(0x7f09a40af7d0 msgr2=0x7f09a406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09a8e00640 1 --2- 172.21.15.69:0/770698208 >> v2:172.21.15.69:3300/0 conn(0x7f09a40af7d0 0x7f09a406dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09a8e00640 1 -- 172.21.15.69:0/770698208 >> v1:172.21.15.69:6789/0 conn(0x7f09a415ea10 legacy=0x7f09a40d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09a8e00640 1 -- 172.21.15.69:0/770698208 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a40ae030 con 0x7f09a41631a0 2024-04-24T08:11:34.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.931+0000 7f09aa200640 1 --2- 172.21.15.69:0/770698208 >> v2:172.21.15.69:3300/0 conn(0x7f09a40af7d0 0x7f09a406dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.932+0000 7f09a8e00640 1 --2- 172.21.15.69:0/770698208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a41631a0 0x7f09a40af290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09900097a0 tx=0x7f099002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8d8b707d451d1b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.933+0000 7f09a3e00640 1 -- 172.21.15.69:0/770698208 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990037070 con 0x7f09a41631a0 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.933+0000 7f09a3e00640 1 -- 172.21.15.69:0/770698208 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0990032440 con 0x7f09a41631a0 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.933+0000 7f09a3e00640 1 -- 172.21.15.69:0/770698208 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990032810 con 0x7f09a41631a0 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.933+0000 7f09ac200640 1 -- 172.21.15.69:0/770698208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a41631a0 msgr2=0x7f09a40af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.933+0000 7f09ac200640 1 --2- 172.21.15.69:0/770698208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a41631a0 0x7f09a40af290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09900097a0 tx=0x7f099002a180 comp rx=0 tx=0).stop 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.933+0000 7f09ac200640 1 -- 172.21.15.69:0/770698208 shutdown_connections 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.933+0000 7f09ac200640 1 --2- 172.21.15.69:0/770698208 >> v2:172.21.15.69:3300/0 conn(0x7f09a40af7d0 0x7f09a406dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.933+0000 7f09ac200640 1 --2- 172.21.15.69:0/770698208 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a41631a0 0x7f09a40af290 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.933+0000 7f09ac200640 1 -- 172.21.15.69:0/770698208 >> 172.21.15.69:0/770698208 conn(0x7f09a4067d90 msgr2=0x7f09a406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.933+0000 7f09ac200640 1 -- 172.21.15.69:0/770698208 shutdown_connections 2024-04-24T08:11:34.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.934+0000 7f09ac200640 1 -- 172.21.15.69:0/770698208 wait complete. 2024-04-24T08:11:34.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.935+0000 7f09ac200640 1 Processor -- start 2024-04-24T08:11:34.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.935+0000 7f09ac200640 1 -- start start 2024-04-24T08:11:34.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.936+0000 7f09ac200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a40af7d0 0x7f09a414fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.936+0000 7f09ac200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a415ea10 0x7f09a41f4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.936+0000 7f09ac200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09a41631a0 0x7f09a4150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.936+0000 7f09ac200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09a40ae8d0 con 0x7f09a415ea10 2024-04-24T08:11:34.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.936+0000 7f09ac200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f09a40ae5f0 con 0x7f09a41631a0 2024-04-24T08:11:34.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.936+0000 7f09ac200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09a40aedf0 con 0x7f09a40af7d0 2024-04-24T08:11:34.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.936+0000 7f09a8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a415ea10 0x7f09a41f4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.936+0000 7f09a8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a415ea10 0x7f09a41f4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33142/0 (socket says 172.21.15.69:33142) 2024-04-24T08:11:34.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.936+0000 7f09a8e00640 1 -- 172.21.15.69:0/2560874746 learned_addr learned my addr 172.21.15.69:0/2560874746 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:34.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09aa200640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09a41631a0 0x7f09a4150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09a9800640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a40af7d0 0x7f09a414fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09a8e00640 1 -- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a40af7d0 msgr2=0x7f09a414fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09a8e00640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a40af7d0 0x7f09a414fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09a8e00640 1 -- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09a41631a0 msgr2=0x7f09a4150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09a8e00640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09a41631a0 0x7f09a4150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09a8e00640 1 -- 172.21.15.69:0/2560874746 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a40cd4f0 con 0x7f09a415ea10 2024-04-24T08:11:34.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09a9800640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a40af7d0 0x7f09a414fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:11:34.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09aa200640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09a41631a0 0x7f09a4150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:11:34.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09a8e00640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a415ea10 0x7f09a41f4930 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0990000c00 tx=0x7f09900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.937+0000 7f09a1600640 1 -- 172.21.15.69:0/2560874746 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990037040 con 0x7f09a415ea10 2024-04-24T08:11:34.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.938+0000 7f09a1600640 1 -- 172.21.15.69:0/2560874746 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0990042070 con 0x7f09a415ea10 2024-04-24T08:11:34.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.938+0000 7f09a1600640 1 -- 172.21.15.69:0/2560874746 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990030920 con 0x7f09a415ea10 2024-04-24T08:11:34.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.938+0000 7f09ac200640 1 -- 172.21.15.69:0/2560874746 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a40cd7b0 con 0x7f09a415ea10 2024-04-24T08:11:34.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.938+0000 7f09ac200640 1 -- 172.21.15.69:0/2560874746 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a40cdd70 con 0x7f09a415ea10 2024-04-24T08:11:34.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.939+0000 7f09ac200640 1 -- 172.21.15.69:0/2560874746 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a4068a80 con 0x7f09a415ea10 2024-04-24T08:11:34.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.940+0000 7f09a1600640 1 -- 172.21.15.69:0/2560874746 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0990030ac0 con 0x7f09a415ea10 2024-04-24T08:11:34.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.941+0000 7f09a1600640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098406c8f0 0x7f098406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.941+0000 7f09a1600640 1 -- 172.21.15.69:0/2560874746 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4943+0+0 (secure 0 0 0) 0x7f09900ac290 con 0x7f09a415ea10 2024-04-24T08:11:34.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.941+0000 7f09a9800640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098406c8f0 0x7f098406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.942+0000 7f09a9800640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098406c8f0 0x7f098406edb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0998001700 tx=0x7f0998009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:34.945+0000 7f09a1600640 1 -- 172.21.15.69:0/2560874746 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f099007a2d0 con 0x7f09a415ea10 2024-04-24T08:11:35.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.156+0000 7f09ac200640 1 -- 172.21.15.69:0/2560874746 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f09a41501c0 con 0x7f09a415ea10 2024-04-24T08:11:35.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.292+0000 7f09a1600640 1 -- 172.21.15.69:0/2560874746 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v51) ==== 97+0+0 (secure 0 0 0) 0x7f099007e5d0 con 0x7f09a415ea10 2024-04-24T08:11:35.294 INFO:teuthology.orchestra.run.smithi069.stderr: removed all blocklist entries 2024-04-24T08:11:35.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.296+0000 7f09ac200640 1 -- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098406c8f0 msgr2=0x7f098406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:35.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.296+0000 7f09ac200640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098406c8f0 0x7f098406edb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0998001700 tx=0x7f0998009040 comp rx=0 tx=0).stop 2024-04-24T08:11:35.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.296+0000 7f09ac200640 1 -- 172.21.15.69:0/2560874746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a415ea10 msgr2=0x7f09a41f4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:35.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.296+0000 7f09ac200640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a415ea10 0x7f09a41f4930 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0990000c00 tx=0x7f09900046c0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.296+0000 7f09ac200640 1 -- 172.21.15.69:0/2560874746 shutdown_connections 2024-04-24T08:11:35.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.296+0000 7f09ac200640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098406c8f0 0x7f098406edb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.296+0000 7f09ac200640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09a415ea10 0x7f09a41f4930 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.296+0000 7f09ac200640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09a41631a0 0x7f09a4150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.296+0000 7f09ac200640 1 --2- 172.21.15.69:0/2560874746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09a40af7d0 0x7f09a414fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.296+0000 7f09ac200640 1 -- 172.21.15.69:0/2560874746 >> 172.21.15.69:0/2560874746 conn(0x7f09a4067d90 msgr2=0x7f09a406eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:35.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.297+0000 7f09ac200640 1 -- 172.21.15.69:0/2560874746 shutdown_connections 2024-04-24T08:11:35.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.297+0000 7f09ac200640 1 -- 172.21.15.69:0/2560874746 wait complete. 2024-04-24T08:11:35.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-24T08:11:35.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.511+0000 7f4edfe00640 1 Processor -- start 2024-04-24T08:11:35.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.511+0000 7f4edfe00640 1 -- start start 2024-04-24T08:11:35.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.512+0000 7f4edfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4ed8071d70 0x7f4ed8072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:35.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.512+0000 7f4edfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed806f730 0x7f4ed806dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:35.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.512+0000 7f4edfe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4ed80b3320 0x7f4ed806e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:35.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.512+0000 7f4edfe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed80b4f80 con 0x7f4ed8071d70 2024-04-24T08:11:35.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.512+0000 7f4edfe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed8072690 con 0x7f4ed80b3320 2024-04-24T08:11:35.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.512+0000 7f4edfe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed80b4370 con 0x7f4ed806f730 2024-04-24T08:11:35.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edd400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4ed8071d70 0x7f4ed8072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:35.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edd400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4ed8071d70 0x7f4ed8072150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33154/0 (socket says 172.21.15.69:33154) 2024-04-24T08:11:35.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edd400640 1 -- 172.21.15.69:0/552880521 learned_addr learned my addr 172.21.15.69:0/552880521 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:35.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edca00640 1 --2- 172.21.15.69:0/552880521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed806f730 0x7f4ed806dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:35.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edd400640 1 -- 172.21.15.69:0/552880521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed806f730 msgr2=0x7f4ed806dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:35.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edd400640 1 --2- 172.21.15.69:0/552880521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed806f730 0x7f4ed806dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edde00640 1 --2- 172.21.15.69:0/552880521 >> v2:172.21.15.121:3300/0 conn(0x7f4ed80b3320 0x7f4ed806e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:35.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edd400640 1 -- 172.21.15.69:0/552880521 >> v2:172.21.15.121:3300/0 conn(0x7f4ed80b3320 msgr2=0x7f4ed806e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:35.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edd400640 1 --2- 172.21.15.69:0/552880521 >> v2:172.21.15.121:3300/0 conn(0x7f4ed80b3320 0x7f4ed806e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edd400640 1 -- 172.21.15.69:0/552880521 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed80b4650 con 0x7f4ed8071d70 2024-04-24T08:11:35.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.513+0000 7f4edca00640 1 --2- 172.21.15.69:0/552880521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed806f730 0x7f4ed806dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:35.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.514+0000 7f4edd400640 1 --2- 172.21.15.69:0/552880521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 0x7f4ed8072150 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80cfeb0 tx=0x7f4ecc02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ec478bb6eccab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:35.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4ed7e00640 1 -- 172.21.15.69:0/552880521 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc036070 con 0x7f4ed8071d70 2024-04-24T08:11:35.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4ed7e00640 1 -- 172.21.15.69:0/552880521 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ecc031600 con 0x7f4ed8071d70 2024-04-24T08:11:35.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4ed7e00640 1 -- 172.21.15.69:0/552880521 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc0319d0 con 0x7f4ed8071d70 2024-04-24T08:11:35.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4edfe00640 1 -- 172.21.15.69:0/552880521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 msgr2=0x7f4ed8072150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:35.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4edfe00640 1 --2- 172.21.15.69:0/552880521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 0x7f4ed8072150 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4ed80cfeb0 tx=0x7f4ecc02a6c0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4edfe00640 1 -- 172.21.15.69:0/552880521 shutdown_connections 2024-04-24T08:11:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4edfe00640 1 --2- 172.21.15.69:0/552880521 >> v2:172.21.15.121:3300/0 conn(0x7f4ed80b3320 0x7f4ed806e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4edfe00640 1 --2- 172.21.15.69:0/552880521 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed806f730 0x7f4ed806dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4edfe00640 1 --2- 172.21.15.69:0/552880521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 0x7f4ed8072150 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4edfe00640 1 -- 172.21.15.69:0/552880521 >> 172.21.15.69:0/552880521 conn(0x7f4ed8067d90 msgr2=0x7f4ed806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.515+0000 7f4edfe00640 1 -- 172.21.15.69:0/552880521 shutdown_connections 2024-04-24T08:11:35.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.516+0000 7f4edfe00640 1 -- 172.21.15.69:0/552880521 wait complete. 2024-04-24T08:11:35.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.517+0000 7f4edfe00640 1 Processor -- start 2024-04-24T08:11:35.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.517+0000 7f4edfe00640 1 -- start start 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.518+0000 7f4edfe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ed806f730 0x7f4ed815b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.518+0000 7f4edfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 0x7f4ed80cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.518+0000 7f4edfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed80b3320 0x7f4ed80c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.518+0000 7f4edfe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed806ee60 con 0x7f4ed8071d70 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.518+0000 7f4edfe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed80b4c10 con 0x7f4ed806f730 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.518+0000 7f4edfe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed815b040 con 0x7f4ed80b3320 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.518+0000 7f4edca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 0x7f4ed80cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.519+0000 7f4edca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 0x7f4ed80cae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33160/0 (socket says 172.21.15.69:33160) 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.519+0000 7f4edca00640 1 -- 172.21.15.69:0/1491109986 learned_addr learned my addr 172.21.15.69:0/1491109986 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.519+0000 7f4edde00640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed80b3320 0x7f4ed80c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:35.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.519+0000 7f4edd400640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ed806f730 0x7f4ed815b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.519+0000 7f4edca00640 1 -- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed80b3320 msgr2=0x7f4ed80c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.519+0000 7f4edca00640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed80b3320 0x7f4ed80c9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.519+0000 7f4edca00640 1 -- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ed806f730 msgr2=0x7f4ed815b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.519+0000 7f4edca00640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ed806f730 0x7f4ed815b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.519+0000 7f4edca00640 1 -- 172.21.15.69:0/1491109986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed815bbc0 con 0x7f4ed8071d70 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.519+0000 7f4edca00640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 0x7f4ed80cae20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400c7f0 tx=0x7f4ec400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.520+0000 7f4ed5600640 1 -- 172.21.15.69:0/1491109986 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4015070 con 0x7f4ed8071d70 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.520+0000 7f4ed5600640 1 -- 172.21.15.69:0/1491109986 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec4007a30 con 0x7f4ed8071d70 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.520+0000 7f4ed5600640 1 -- 172.21.15.69:0/1491109986 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4007e00 con 0x7f4ed8071d70 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.520+0000 7f4edfe00640 1 -- 172.21.15.69:0/1491109986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed815bea0 con 0x7f4ed8071d70 2024-04-24T08:11:35.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.520+0000 7f4edfe00640 1 -- 172.21.15.69:0/1491109986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed815c460 con 0x7f4ed8071d70 2024-04-24T08:11:35.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.521+0000 7f4edfe00640 1 -- 172.21.15.69:0/1491109986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ed8068a80 con 0x7f4ed8071d70 2024-04-24T08:11:35.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.522+0000 7f4ed5600640 1 -- 172.21.15.69:0/1491109986 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ec4018070 con 0x7f4ed8071d70 2024-04-24T08:11:35.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.523+0000 7f4ed5600640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eb806c8f0 0x7f4eb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:35.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.523+0000 7f4ed5600640 1 -- 172.21.15.69:0/1491109986 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4857+0+0 (secure 0 0 0) 0x7f4ec408b7c0 con 0x7f4ed8071d70 2024-04-24T08:11:35.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.524+0000 7f4edd400640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eb806c8f0 0x7f4eb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:35.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.525+0000 7f4edd400640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eb806c8f0 0x7f4eb806edb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc033000 tx=0x7f4ecc009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:35.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.528+0000 7f4ed5600640 1 -- 172.21.15.69:0/1491109986 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ec4059ea0 con 0x7f4ed8071d70 2024-04-24T08:11:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.809+0000 7f4edfe00640 1 -- 172.21.15.69:0/1491109986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f4ed806be30 con 0x7f4ed8071d70 2024-04-24T08:11:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.810+0000 7f4ed5600640 1 -- 172.21.15.69:0/1491109986 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+5304 (secure 0 0 0) 0x7f4ec40597a0 con 0x7f4ed8071d70 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_dump": [ 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.a", 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmsqMeLRAApLPtHjtq78HzlVcsjfqXOg==", 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:11:35.812 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.b", 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmkyd5ChAAGsJGU6GGMvvGzQnABuDx/g==", 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.c", 2024-04-24T08:11:35.813 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmS+yLORAAHPCeBtq99IXEVTpcMT3c4A==", 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.d", 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShm0xmHFRAA3VfYoqbcV8kWF+4fCZMVLA==", 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.814 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.0", 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC5vShmMbvcERAANgogiCC/sXf1y0Ykv7ecNQ==", 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.815 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.1", 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC8vShmDD+uLBAAJHXmxEm1cumJ3wjqe6S0ww==", 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.2", 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC/vShmS63kNhAAXec5HVuwo8THKstXiQOL8Q==", 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:11:35.816 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.3", 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDDvShmxcDJBhAA3V3I4C63EHPRUyhzBCZEzw==", 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.4", 2024-04-24T08:11:35.817 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDJvShm1pX9ERAAL2gjv/+M915Cf5OoqBhZVw==", 2024-04-24T08:11:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:11:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:11:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.5", 2024-04-24T08:11:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDMvShmNIxbHRAA23gxiaGvagddVc2mE5DJJw==", 2024-04-24T08:11:35.818 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.6", 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDPvShmb+x4NhAA6aWmAzU6c0AZomVgWcFwig==", 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.819 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.7", 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDTvShmgcKMBRAA6fkJvNxBjlNhK9CrZ/FMSw==", 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.820 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.0", 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw==", 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow r", 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.1", 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmKAIHMhAAuTuFmTHvBteErsG23LMEyQ==", 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.821 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.2", 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmW0MnKRAASzxCAKuApVvfu9kUfK9BJw==", 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:11:35.822 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.3", 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC2vShmfgoaABAAniS9oubv92R7Qe2EITGKxg==", 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.admin", 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQCzvShmLh6yLxAAoH7NhoEAUKcdXp6n+ai2Ug==", 2024-04-24T08:11:35.823 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.x", 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmv4rwIxAA3rp+yy09qrs7HXip8bZ8xA==", 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.824 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.y", 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShm2eatGxAAFQkL0GBsujypv4VnmSy5bw==", 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-24T08:11:35.825 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:11:35.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 -- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eb806c8f0 msgr2=0x7f4eb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eb806c8f0 0x7f4eb806edb0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc033000 tx=0x7f4ecc009000 comp rx=0 tx=0).stop 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 -- 172.21.15.69:0/1491109986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 msgr2=0x7f4ed80cae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 0x7f4ed80cae20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400c7f0 tx=0x7f4ec400ccb0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 -- 172.21.15.69:0/1491109986 shutdown_connections 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eb806c8f0 0x7f4eb806edb0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4ed8071d70 0x7f4ed80cae20 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4ed80b3320 0x7f4ed80c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 --2- 172.21.15.69:0/1491109986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ed806f730 0x7f4ed815b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 -- 172.21.15.69:0/1491109986 >> 172.21.15.69:0/1491109986 conn(0x7f4ed8067d90 msgr2=0x7f4ed806df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 -- 172.21.15.69:0/1491109986 shutdown_connections 2024-04-24T08:11:35.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:35.813+0000 7f4edfe00640 1 -- 172.21.15.69:0/1491109986 wait complete. 2024-04-24T08:11:35.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:11:36.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.018+0000 7f9e9be00640 1 Processor -- start 2024-04-24T08:11:36.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.018+0000 7f9e9be00640 1 -- start start 2024-04-24T08:11:36.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.019+0000 7f9e9be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9e9c162300 0x7f9e9c143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.019+0000 7f9e9be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9e9c072230 0x7f9e9c143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.019+0000 7f9e9be00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e9c15f540 con 0x7f9e9c071e50 2024-04-24T08:11:36.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.019+0000 7f9e9be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9e9c15f820 con 0x7f9e9c162300 2024-04-24T08:11:36.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.019+0000 7f9e9be00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9e9c15fb00 con 0x7f9e9c072230 2024-04-24T08:11:36.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.020+0000 7f9e92000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9e9c162300 0x7f9e9c143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.020+0000 7f9e92000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9e9c162300 0x7f9e9c143690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33174/0 (socket says 172.21.15.69:33174) 2024-04-24T08:11:36.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.020+0000 7f9e92000640 1 -- 172.21.15.69:0/2047512846 learned_addr learned my addr 172.21.15.69:0/2047512846 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:36.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.020+0000 7f9e92000640 1 -- 172.21.15.69:0/2047512846 >> v2:172.21.15.121:3300/0 conn(0x7f9e9c072230 msgr2=0x7f9e9c143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.020+0000 7f9e92000640 1 --2- 172.21.15.69:0/2047512846 >> v2:172.21.15.121:3300/0 conn(0x7f9e9c072230 0x7f9e9c143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.020+0000 7f9e92000640 1 -- 172.21.15.69:0/2047512846 >> v1:172.21.15.121:6789/0 conn(0x7f9e9c071e50 legacy=0x7f9e9c15ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.020+0000 7f9e92000640 1 -- 172.21.15.69:0/2047512846 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c06dda0 con 0x7f9e9c162300 2024-04-24T08:11:36.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.021+0000 7f9e92000640 1 --2- 172.21.15.69:0/2047512846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 0x7f9e9c143690 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c141de0 tx=0x7f9e8402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e6c5334b9e69a5e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.021+0000 7f9e9a000640 1 -- 172.21.15.69:0/2047512846 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84037070 con 0x7f9e9c162300 2024-04-24T08:11:36.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.021+0000 7f9e9a000640 1 -- 172.21.15.69:0/2047512846 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e84032440 con 0x7f9e9c162300 2024-04-24T08:11:36.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.021+0000 7f9e9a000640 1 -- 172.21.15.69:0/2047512846 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84032810 con 0x7f9e9c162300 2024-04-24T08:11:36.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.022+0000 7f9e9be00640 1 -- 172.21.15.69:0/2047512846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 msgr2=0x7f9e9c143690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.022+0000 7f9e9be00640 1 --2- 172.21.15.69:0/2047512846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 0x7f9e9c143690 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c141de0 tx=0x7f9e8402a230 comp rx=0 tx=0).stop 2024-04-24T08:11:36.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.022+0000 7f9e9be00640 1 -- 172.21.15.69:0/2047512846 shutdown_connections 2024-04-24T08:11:36.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.022+0000 7f9e9be00640 1 --2- 172.21.15.69:0/2047512846 >> v2:172.21.15.121:3300/0 conn(0x7f9e9c072230 0x7f9e9c143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.022+0000 7f9e9be00640 1 --2- 172.21.15.69:0/2047512846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 0x7f9e9c143690 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.022+0000 7f9e9be00640 1 -- 172.21.15.69:0/2047512846 >> 172.21.15.69:0/2047512846 conn(0x7f9e9c0d0160 msgr2=0x7f9e9c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.022+0000 7f9e9be00640 1 -- 172.21.15.69:0/2047512846 shutdown_connections 2024-04-24T08:11:36.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.022+0000 7f9e9be00640 1 -- 172.21.15.69:0/2047512846 wait complete. 2024-04-24T08:11:36.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.024+0000 7f9e9be00640 1 Processor -- start 2024-04-24T08:11:36.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.024+0000 7f9e9be00640 1 -- start start 2024-04-24T08:11:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.025+0000 7f9e9be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e9c071e50 0x7f9e9c099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.025+0000 7f9e9be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e9c072230 0x7f9e9c0ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.025+0000 7f9e9be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 0x7f9e9c0ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.025+0000 7f9e9be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c144110 con 0x7f9e9c162300 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.025+0000 7f9e9be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c144290 con 0x7f9e9c071e50 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.025+0000 7f9e9be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9c144410 con 0x7f9e9c072230 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.025+0000 7f9e9b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 0x7f9e9c0ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.025+0000 7f9e9b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 0x7f9e9c0ae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33182/0 (socket says 172.21.15.69:33182) 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.025+0000 7f9e9b400640 1 -- 172.21.15.69:0/2777026877 learned_addr learned my addr 172.21.15.69:0/2777026877 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.025+0000 7f9e92000640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e9c072230 0x7f9e9c0ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e9b400640 1 -- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e9c072230 msgr2=0x7f9e9c0ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e9b400640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e9c072230 0x7f9e9c0ae120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e9b400640 1 -- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e9c071e50 msgr2=0x7f9e9c099920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e9b400640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e9c071e50 0x7f9e9c099920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e9b400640 1 -- 172.21.15.69:0/2777026877 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c158320 con 0x7f9e9c162300 2024-04-24T08:11:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e9b400640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 0x7f9e9c0ae660 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400b970 tx=0x7f9e9400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e93400640 1 -- 172.21.15.69:0/2777026877 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94019070 con 0x7f9e9c162300 2024-04-24T08:11:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e93400640 1 -- 172.21.15.69:0/2777026877 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e9400cb60 con 0x7f9e9c162300 2024-04-24T08:11:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e93400640 1 -- 172.21.15.69:0/2777026877 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94014440 con 0x7f9e9c162300 2024-04-24T08:11:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e9be00640 1 -- 172.21.15.69:0/2777026877 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9c099e60 con 0x7f9e9c162300 2024-04-24T08:11:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.026+0000 7f9e9be00640 1 -- 172.21.15.69:0/2777026877 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e9c160270 con 0x7f9e9c162300 2024-04-24T08:11:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.027+0000 7f9e9be00640 1 -- 172.21.15.69:0/2777026877 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e9c075d50 con 0x7f9e9c162300 2024-04-24T08:11:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.028+0000 7f9e93400640 1 -- 172.21.15.69:0/2777026877 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9e9401c070 con 0x7f9e9c162300 2024-04-24T08:11:36.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.029+0000 7f9e93400640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9e7806c8f0 0x7f9e7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.029+0000 7f9e93400640 1 -- 172.21.15.69:0/2777026877 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4857+0+0 (secure 0 0 0) 0x7f9e9408b550 con 0x7f9e9c162300 2024-04-24T08:11:36.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.030+0000 7f9e9aa00640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9e7806c8f0 0x7f9e7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.031+0000 7f9e9aa00640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9e7806c8f0 0x7f9e7806edb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c15d760 tx=0x7f9e8c007990 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.033+0000 7f9e93400640 1 -- 172.21.15.69:0/2777026877 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e9c099e60 con 0x7f9e9c162300 2024-04-24T08:11:36.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.216+0000 7f9e9be00640 1 -- 172.21.15.69:0/2777026877 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e9c15ef80 con 0x7f9e9c162300 2024-04-24T08:11:36.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.218+0000 7f9e93400640 1 -- 172.21.15.69:0/2777026877 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) ==== 74+0+12867 (secure 0 0 0) 0x7f9e9c158320 con 0x7f9e9c162300 2024-04-24T08:11:36.219 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:36.219 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":53,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:35.308848+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 -- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9e7806c8f0 msgr2=0x7f9e7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9e7806c8f0 0x7f9e7806edb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c15d760 tx=0x7f9e8c007990 comp rx=0 tx=0).stop 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 -- 172.21.15.69:0/2777026877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 msgr2=0x7f9e9c0ae660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 0x7f9e9c0ae660 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400b970 tx=0x7f9e9400be30 comp rx=0 tx=0).stop 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 -- 172.21.15.69:0/2777026877 shutdown_connections 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9e7806c8f0 0x7f9e7806edb0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e9c162300 0x7f9e9c0ae660 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9e9c072230 0x7f9e9c0ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 --2- 172.21.15.69:0/2777026877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9e9c071e50 0x7f9e9c099920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 -- 172.21.15.69:0/2777026877 >> 172.21.15.69:0/2777026877 conn(0x7f9e9c0d0160 msgr2=0x7f9e9c0b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 -- 172.21.15.69:0/2777026877 shutdown_connections 2024-04-24T08:11:36.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.220+0000 7f9e9be00640 1 -- 172.21.15.69:0/2777026877 wait complete. 2024-04-24T08:11:36.240 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:11:36.240 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:11:36.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.425+0000 7f04b8e00640 1 Processor -- start 2024-04-24T08:11:36.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.425+0000 7f04b8e00640 1 -- start start 2024-04-24T08:11:36.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.426+0000 7f04b8e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04b415ea60 0x7f04b40b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.426+0000 7f04b8e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04b40cdf30 0x7f04b40d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.426+0000 7f04b8e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04b40ce910 con 0x7f04b40ce390 2024-04-24T08:11:36.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.426+0000 7f04b8e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f04b40cebf0 con 0x7f04b40cdf30 2024-04-24T08:11:36.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.426+0000 7f04b8e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04b40ceed0 con 0x7f04b415ea60 2024-04-24T08:11:36.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.427+0000 7f04b3e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f04b40ce390 0x7f04b40b3320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48924/0 (socket says 172.21.15.69:48924) 2024-04-24T08:11:36.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.427+0000 7f04aaa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04b40cdf30 0x7f04b40d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.427+0000 7f04b3e00640 1 -- 172.21.15.69:0/1962163632 learned_addr learned my addr 172.21.15.69:0/1962163632 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:36.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.427+0000 7f04b3400640 1 --2- 172.21.15.69:0/1962163632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04b415ea60 0x7f04b40b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.427+0000 7f04aaa00640 1 -- 172.21.15.69:0/1962163632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04b415ea60 msgr2=0x7f04b40b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.427+0000 7f04aaa00640 1 --2- 172.21.15.69:0/1962163632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04b415ea60 0x7f04b40b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.427+0000 7f04aaa00640 1 -- 172.21.15.69:0/1962163632 >> v1:172.21.15.69:6789/0 conn(0x7f04b40ce390 legacy=0x7f04b40b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.427+0000 7f04aaa00640 1 -- 172.21.15.69:0/1962163632 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b40cf1b0 con 0x7f04b40cdf30 2024-04-24T08:11:36.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.427+0000 7f04b3400640 1 --2- 172.21.15.69:0/1962163632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04b415ea60 0x7f04b40b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:36.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.428+0000 7f04aaa00640 1 --2- 172.21.15.69:0/1962163632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 0x7f04b40d02e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f049c009d90 tx=0x7f049c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682cdb769146c6db server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.428+0000 7f04b2a00640 1 -- 172.21.15.69:0/1962163632 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c031600 con 0x7f04b40cdf30 2024-04-24T08:11:36.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.428+0000 7f04b2a00640 1 -- 172.21.15.69:0/1962163632 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049c0317a0 con 0x7f04b40cdf30 2024-04-24T08:11:36.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.428+0000 7f04b2a00640 1 -- 172.21.15.69:0/1962163632 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049c031b70 con 0x7f04b40cdf30 2024-04-24T08:11:36.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.429+0000 7f04b8e00640 1 -- 172.21.15.69:0/1962163632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 msgr2=0x7f04b40d02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.429+0000 7f04b8e00640 1 --2- 172.21.15.69:0/1962163632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 0x7f04b40d02e0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f049c009d90 tx=0x7f049c02a5d0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.429+0000 7f04b8e00640 1 -- 172.21.15.69:0/1962163632 shutdown_connections 2024-04-24T08:11:36.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.429+0000 7f04b8e00640 1 --2- 172.21.15.69:0/1962163632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 0x7f04b40d02e0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.429+0000 7f04b8e00640 1 --2- 172.21.15.69:0/1962163632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04b415ea60 0x7f04b40b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.429+0000 7f04b8e00640 1 -- 172.21.15.69:0/1962163632 >> 172.21.15.69:0/1962163632 conn(0x7f04b4067b80 msgr2=0x7f04b406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.429+0000 7f04b8e00640 1 -- 172.21.15.69:0/1962163632 shutdown_connections 2024-04-24T08:11:36.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.429+0000 7f04b8e00640 1 -- 172.21.15.69:0/1962163632 wait complete. 2024-04-24T08:11:36.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.431+0000 7f04b8e00640 1 Processor -- start 2024-04-24T08:11:36.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.431+0000 7f04b8e00640 1 -- start start 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.432+0000 7f04b8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 0x7f04b41573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.432+0000 7f04b8e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b40ce390 0x7f04b4159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.432+0000 7f04b8e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04b415ea60 0x7f04b414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.432+0000 7f04b8e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04b406f0d0 con 0x7f04b40cdf30 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.432+0000 7f04b8e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04b406ef50 con 0x7f04b40ce390 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.432+0000 7f04b8e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04b406f250 con 0x7f04b415ea60 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.432+0000 7f04b3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 0x7f04b41573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.432+0000 7f04b3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 0x7f04b41573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33196/0 (socket says 172.21.15.69:33196) 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.432+0000 7f04b3400640 1 -- 172.21.15.69:0/1815753189 learned_addr learned my addr 172.21.15.69:0/1815753189 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04aaa00640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b40ce390 0x7f04b4159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04b3400640 1 -- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04b415ea60 msgr2=0x7f04b414fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04b3400640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04b415ea60 0x7f04b414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04b3400640 1 -- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b40ce390 msgr2=0x7f04b4159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04b3400640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b40ce390 0x7f04b4159ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04b3400640 1 -- 172.21.15.69:0/1815753189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b415a020 con 0x7f04b40cdf30 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04b3400640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 0x7f04b41573d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f04a400c7f0 tx=0x7f04a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04abe00640 1 -- 172.21.15.69:0/1815753189 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4015070 con 0x7f04b40cdf30 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04abe00640 1 -- 172.21.15.69:0/1815753189 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a4007820 con 0x7f04b40cdf30 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04b8e00640 1 -- 172.21.15.69:0/1815753189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b415a400 con 0x7f04b40cdf30 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04abe00640 1 -- 172.21.15.69:0/1815753189 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4007bd0 con 0x7f04b40cdf30 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.433+0000 7f04b8e00640 1 -- 172.21.15.69:0/1815753189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b4152aa0 con 0x7f04b40cdf30 2024-04-24T08:11:36.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.434+0000 7f04b8e00640 1 -- 172.21.15.69:0/1815753189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04b4075d50 con 0x7f04b40cdf30 2024-04-24T08:11:36.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.435+0000 7f04abe00640 1 -- 172.21.15.69:0/1815753189 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04a4018070 con 0x7f04b40cdf30 2024-04-24T08:11:36.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.436+0000 7f04abe00640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f049006c8f0 0x7f049006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.436+0000 7f04abe00640 1 -- 172.21.15.69:0/1815753189 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4857+0+0 (secure 0 0 0) 0x7f04a408b9b0 con 0x7f04b40cdf30 2024-04-24T08:11:36.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.437+0000 7f04aaa00640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f049006c8f0 0x7f049006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.438+0000 7f04aaa00640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f049006c8f0 0x7f049006edb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f049c034000 tx=0x7f049c02f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.440+0000 7f04abe00640 1 -- 172.21.15.69:0/1815753189 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04a405a090 con 0x7f04b40cdf30 2024-04-24T08:11:36.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.623+0000 7f04b8e00640 1 -- 172.21.15.69:0/1815753189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f04b40b35d0 con 0x7f04b40cdf30 2024-04-24T08:11:36.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.624+0000 7f04abe00640 1 -- 172.21.15.69:0/1815753189 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) ==== 93+0+10 (secure 0 0 0) 0x7f04a4059990 con 0x7f04b40cdf30 2024-04-24T08:11:36.625 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 -- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f049006c8f0 msgr2=0x7f049006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f049006c8f0 0x7f049006edb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f049c034000 tx=0x7f049c02f000 comp rx=0 tx=0).stop 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 -- 172.21.15.69:0/1815753189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 msgr2=0x7f04b41573d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 0x7f04b41573d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f04a400c7f0 tx=0x7f04a400ccb0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 -- 172.21.15.69:0/1815753189 shutdown_connections 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f049006c8f0 0x7f049006edb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04b415ea60 0x7f04b414fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04b40ce390 0x7f04b4159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 --2- 172.21.15.69:0/1815753189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04b40cdf30 0x7f04b41573d0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 -- 172.21.15.69:0/1815753189 >> 172.21.15.69:0/1815753189 conn(0x7f04b4067b80 msgr2=0x7f04b4071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 -- 172.21.15.69:0/1815753189 shutdown_connections 2024-04-24T08:11:36.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.626+0000 7f04b8e00640 1 -- 172.21.15.69:0/1815753189 wait complete. 2024-04-24T08:11:36.645 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T08:11:36.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.838+0000 7f9344a00640 1 Processor -- start 2024-04-24T08:11:36.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.838+0000 7f9344a00640 1 -- start start 2024-04-24T08:11:36.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.839+0000 7f9344a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9340156c60 0x7f934015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.840+0000 7f9344a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400ad7e0 0x7f934015d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.840+0000 7f9344a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934015d660 con 0x7f93400b30f0 2024-04-24T08:11:36.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.840+0000 7f9344a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f934006dc30 con 0x7f9340156c60 2024-04-24T08:11:36.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.840+0000 7f9344a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f934006df10 con 0x7f93400ad7e0 2024-04-24T08:11:36.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.840+0000 7f9336a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f93400b30f0 0x7f93400ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48938/0 (socket says 172.21.15.69:48938) 2024-04-24T08:11:36.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.840+0000 7f9336a00640 1 -- 172.21.15.69:0/1784855089 learned_addr learned my addr 172.21.15.69:0/1784855089 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.840+0000 7f933fe00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400ad7e0 0x7f934015d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.840+0000 7f933f400640 1 --2- 172.21.15.69:0/1784855089 >> v2:172.21.15.121:3300/0 conn(0x7f9340156c60 0x7f934015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.840+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4211791124 0 0) 0x7f934015d660 con 0x7f93400b30f0 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.840+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9324003520 con 0x7f93400b30f0 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1866420386 0 0) 0x7f9324003520 con 0x7f93400b30f0 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400ad7e0 msgr2=0x7f934015d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400ad7e0 0x7f934015d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 >> v2:172.21.15.121:3300/0 conn(0x7f9340156c60 msgr2=0x7f934015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 --2- 172.21.15.69:0/1784855089 >> v2:172.21.15.121:3300/0 conn(0x7f9340156c60 0x7f934015cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934006e1f0 con 0x7f93400b30f0 2024-04-24T08:11:36.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f9328001670 con 0x7f93400b30f0 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 >> v1:172.21.15.69:6789/0 conn(0x7f93400b30f0 legacy=0x7f93400ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93240039b0 0x7f9324003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 0x7f9324004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400ad7e0 0x7f93240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f934006dc30 con 0x7f9324004800 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f934006df10 con 0x7f93240039b0 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f932400ba40 con 0x7f93400ad7e0 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.841+0000 7f9336a00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 0x7f9324004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.842+0000 7f933fe00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400ad7e0 0x7f93240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.842+0000 7f9336a00640 1 -- 172.21.15.69:0/1784855089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400ad7e0 msgr2=0x7f93240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.842+0000 7f9336a00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400ad7e0 0x7f93240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.842+0000 7f9336a00640 1 -- 172.21.15.69:0/1784855089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93240039b0 msgr2=0x7f9324003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.842+0000 7f9336a00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93240039b0 0x7f9324003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.842+0000 7f9336a00640 1 -- 172.21.15.69:0/1784855089 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9324003520 con 0x7f9324004800 2024-04-24T08:11:36.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.843+0000 7f9336a00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 0x7f9324004c80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9328001050 tx=0x7f9324005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b36fba800ad1ab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.843+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328007ca0 con 0x7f9324004800 2024-04-24T08:11:36.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.843+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9328005ad0 con 0x7f9324004800 2024-04-24T08:11:36.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.843+0000 7f933ea00640 1 -- 172.21.15.69:0/1784855089 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328007070 con 0x7f9324004800 2024-04-24T08:11:36.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.844+0000 7f9344a00640 1 -- 172.21.15.69:0/1784855089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 msgr2=0x7f9324004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.844+0000 7f9344a00640 1 --2- 172.21.15.69:0/1784855089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 0x7f9324004c80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9328001050 tx=0x7f9324005830 comp rx=0 tx=0).stop 2024-04-24T08:11:36.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.844+0000 7f933fe00640 1 -- 172.21.15.69:0/1784855089 reap_dead start 2024-04-24T08:11:36.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.844+0000 7f9344a00640 1 -- 172.21.15.69:0/1784855089 shutdown_connections 2024-04-24T08:11:36.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.844+0000 7f9344a00640 1 -- 172.21.15.69:0/1784855089 >> 172.21.15.69:0/1784855089 conn(0x7f93400d0160 msgr2=0x7f93400d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.844+0000 7f9344a00640 1 -- 172.21.15.69:0/1784855089 shutdown_connections 2024-04-24T08:11:36.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.844+0000 7f9344a00640 1 -- 172.21.15.69:0/1784855089 wait complete. 2024-04-24T08:11:36.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.846+0000 7f9344a00640 1 Processor -- start 2024-04-24T08:11:36.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.846+0000 7f9344a00640 1 -- start start 2024-04-24T08:11:36.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f9344a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 0x7f93400cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f9344a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400cf9d0 0x7f93400b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f9344a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400b13d0 0x7f9340152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f9344a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f934015d000 con 0x7f9324004800 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f9344a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f934015d940 con 0x7f93400cf9d0 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f9344a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f934015d180 con 0x7f93400b13d0 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f933f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 0x7f93400cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f933f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 0x7f93400cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33214/0 (socket says 172.21.15.69:33214) 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f933f400640 1 -- 172.21.15.69:0/2183594010 learned_addr learned my addr 172.21.15.69:0/2183594010 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f933f400640 1 -- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400b13d0 msgr2=0x7f9340152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f933f400640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400b13d0 0x7f9340152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f933f400640 1 -- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400cf9d0 msgr2=0x7f93400b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f933f400640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400cf9d0 0x7f93400b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.847+0000 7f933f400640 1 -- 172.21.15.69:0/2183594010 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93400ca0d0 con 0x7f9324004800 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.848+0000 7f933f400640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 0x7f93400cb170 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f933000d800 tx=0x7f933000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.848+0000 7f9337e00640 1 -- 172.21.15.69:0/2183594010 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933000b640 con 0x7f9324004800 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.848+0000 7f9337e00640 1 -- 172.21.15.69:0/2183594010 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9330019070 con 0x7f9324004800 2024-04-24T08:11:36.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.848+0000 7f9337e00640 1 -- 172.21.15.69:0/2183594010 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933000ba20 con 0x7f9324004800 2024-04-24T08:11:36.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.848+0000 7f9344a00640 1 -- 172.21.15.69:0/2183594010 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93400ca3b0 con 0x7f9324004800 2024-04-24T08:11:36.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.848+0000 7f9344a00640 1 -- 172.21.15.69:0/2183594010 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93400ca970 con 0x7f9324004800 2024-04-24T08:11:36.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.849+0000 7f9344a00640 1 -- 172.21.15.69:0/2183594010 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93400b17f0 con 0x7f9324004800 2024-04-24T08:11:36.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.850+0000 7f9337e00640 1 -- 172.21.15.69:0/2183594010 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f933000bbc0 con 0x7f9324004800 2024-04-24T08:11:36.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.851+0000 7f9337e00640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 0x7f931c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.851+0000 7f9337e00640 1 -- 172.21.15.69:0/2183594010 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4857+0+0 (secure 0 0 0) 0x7f933008bd20 con 0x7f9324004800 2024-04-24T08:11:36.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.852+0000 7f9336a00640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 0x7f931c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.853+0000 7f9336a00640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 0x7f931c06edb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9328032000 tx=0x7f9328007000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:36.855+0000 7f9337e00640 1 -- 172.21.15.69:0/2183594010 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f933005a400 con 0x7f9324004800 2024-04-24T08:11:37.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.056+0000 7f9344a00640 1 -- 172.21.15.69:0/2183594010 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9340075d50 con 0x7f9324004800 2024-04-24T08:11:37.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.057+0000 7f9337e00640 1 -- 172.21.15.69:0/2183594010 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v53) ==== 84+0+65 (secure 0 0 0) 0x7f9330059d00 con 0x7f9324004800 2024-04-24T08:11:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:37.059 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:11:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: "poolnum": 1, 2024-04-24T08:11:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: "poolname": ".mgr" 2024-04-24T08:11:37.059 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:37.059 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:11:37.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 -- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 msgr2=0x7f931c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:37.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 0x7f931c06edb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9328032000 tx=0x7f9328007000 comp rx=0 tx=0).stop 2024-04-24T08:11:37.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 -- 172.21.15.69:0/2183594010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 msgr2=0x7f93400cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:37.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 0x7f93400cb170 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f933000d800 tx=0x7f933000dcc0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 -- 172.21.15.69:0/2183594010 shutdown_connections 2024-04-24T08:11:37.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 0x7f931c06edb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93400b13d0 0x7f9340152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93400cf9d0 0x7f93400b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 --2- 172.21.15.69:0/2183594010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324004800 0x7f93400cb170 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 -- 172.21.15.69:0/2183594010 >> 172.21.15.69:0/2183594010 conn(0x7f93400d0160 msgr2=0x7f93400afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:37.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 -- 172.21.15.69:0/2183594010 shutdown_connections 2024-04-24T08:11:37.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.060+0000 7f9344a00640 1 -- 172.21.15.69:0/2183594010 wait complete. 2024-04-24T08:11:37.079 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T08:11:37.080 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T08:11:37.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.284+0000 7f2afda00640 1 Processor -- start 2024-04-24T08:11:37.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.284+0000 7f2afda00640 1 -- start start 2024-04-24T08:11:37.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.285+0000 7f2afda00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2af815c590 0x7f2af80b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:37.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2afda00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2af80b4070 0x7f2af80d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:37.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2afda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af80d0d00 0x7f2af80cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2afda00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2af80d1150 con 0x7f2af80b4070 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2afda00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2af80b4840 con 0x7f2af815c590 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2afda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2af80b4560 con 0x7f2af80d0d00 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2aefe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2af80b4070 0x7f2af80d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2aefe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2af80b4070 0x7f2af80d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33216/0 (socket says 172.21.15.69:33216) 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2aefe00640 1 -- 172.21.15.69:0/2433391241 learned_addr learned my addr 172.21.15.69:0/2433391241 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2aefe00640 1 -- 172.21.15.69:0/2433391241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af80d0d00 msgr2=0x7f2af80cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2aefe00640 1 --2- 172.21.15.69:0/2433391241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af80d0d00 0x7f2af80cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2aefe00640 1 -- 172.21.15.69:0/2433391241 >> v2:172.21.15.121:3300/0 conn(0x7f2af815c590 msgr2=0x7f2af80b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2aefe00640 1 --2- 172.21.15.69:0/2433391241 >> v2:172.21.15.121:3300/0 conn(0x7f2af815c590 0x7f2af80b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.286+0000 7f2aefe00640 1 -- 172.21.15.69:0/2433391241 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af80ad2e0 con 0x7f2af80b4070 2024-04-24T08:11:37.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.287+0000 7f2aefe00640 1 --2- 172.21.15.69:0/2433391241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 0x7f2af80d07c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f2af80ceff0 tx=0x7f2ae002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1047bac6e95024e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:37.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2af6000640 1 -- 172.21.15.69:0/2433391241 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0037070 con 0x7f2af80b4070 2024-04-24T08:11:37.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2af6000640 1 -- 172.21.15.69:0/2433391241 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae0032440 con 0x7f2af80b4070 2024-04-24T08:11:37.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2af6000640 1 -- 172.21.15.69:0/2433391241 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae0032810 con 0x7f2af80b4070 2024-04-24T08:11:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2afda00640 1 -- 172.21.15.69:0/2433391241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 msgr2=0x7f2af80d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2afda00640 1 --2- 172.21.15.69:0/2433391241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 0x7f2af80d07c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f2af80ceff0 tx=0x7f2ae002a200 comp rx=0 tx=0).stop 2024-04-24T08:11:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2afda00640 1 -- 172.21.15.69:0/2433391241 shutdown_connections 2024-04-24T08:11:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2afda00640 1 --2- 172.21.15.69:0/2433391241 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af80d0d00 0x7f2af80cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2afda00640 1 --2- 172.21.15.69:0/2433391241 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 0x7f2af80d07c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2afda00640 1 --2- 172.21.15.69:0/2433391241 >> v2:172.21.15.121:3300/0 conn(0x7f2af815c590 0x7f2af80b3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2afda00640 1 -- 172.21.15.69:0/2433391241 >> 172.21.15.69:0/2433391241 conn(0x7f2af806be40 msgr2=0x7f2af806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2afda00640 1 -- 172.21.15.69:0/2433391241 shutdown_connections 2024-04-24T08:11:37.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.288+0000 7f2afda00640 1 -- 172.21.15.69:0/2433391241 wait complete. 2024-04-24T08:11:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.290+0000 7f2afda00640 1 Processor -- start 2024-04-24T08:11:37.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.290+0000 7f2afda00640 1 -- start start 2024-04-24T08:11:37.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.291+0000 7f2afda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 0x7f2af80c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:37.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.291+0000 7f2afda00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af80d0d00 0x7f2af80c79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.291+0000 7f2afda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af815c590 0x7f2af80c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.291+0000 7f2afda00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2af80b5160 con 0x7f2af80b4070 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.291+0000 7f2afda00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2af80ad8a0 con 0x7f2af80d0d00 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.291+0000 7f2afda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2af80b52e0 con 0x7f2af815c590 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.291+0000 7f2af6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 0x7f2af80c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.291+0000 7f2af6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 0x7f2af80c7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33220/0 (socket says 172.21.15.69:33220) 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.291+0000 7f2af6a00640 1 -- 172.21.15.69:0/892409985 learned_addr learned my addr 172.21.15.69:0/892409985 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2af7400640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af815c590 0x7f2af80c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2af6a00640 1 -- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af815c590 msgr2=0x7f2af80c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2aefe00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af80d0d00 0x7f2af80c79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2af6a00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af815c590 0x7f2af80c7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2af6a00640 1 -- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af80d0d00 msgr2=0x7f2af80c79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2af6a00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af80d0d00 0x7f2af80c79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:37.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2af6a00640 1 -- 172.21.15.69:0/892409985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af81f4ad0 con 0x7f2af80b4070 2024-04-24T08:11:37.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2af7400640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af815c590 0x7f2af80c7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:37.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2af6a00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 0x7f2af80c7470 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800b410 tx=0x7f2ae800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:37.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2aeea00640 1 -- 172.21.15.69:0/892409985 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8015020 con 0x7f2af80b4070 2024-04-24T08:11:37.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2aeea00640 1 -- 172.21.15.69:0/892409985 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ae8011830 con 0x7f2af80b4070 2024-04-24T08:11:37.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2aeea00640 1 -- 172.21.15.69:0/892409985 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8011c20 con 0x7f2af80b4070 2024-04-24T08:11:37.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2afda00640 1 -- 172.21.15.69:0/892409985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af8158d80 con 0x7f2af80b4070 2024-04-24T08:11:37.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.292+0000 7f2afda00640 1 -- 172.21.15.69:0/892409985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af80c69f0 con 0x7f2af80b4070 2024-04-24T08:11:37.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.293+0000 7f2afda00640 1 -- 172.21.15.69:0/892409985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2af80ce660 con 0x7f2af80b4070 2024-04-24T08:11:37.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.294+0000 7f2aeea00640 1 -- 172.21.15.69:0/892409985 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2ae8018020 con 0x7f2af80b4070 2024-04-24T08:11:37.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.295+0000 7f2aeea00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad406c920 0x7f2ad406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:37.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.295+0000 7f2aeea00640 1 -- 172.21.15.69:0/892409985 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 4857+0+0 (secure 0 0 0) 0x7f2ae808b700 con 0x7f2af80b4070 2024-04-24T08:11:37.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.296+0000 7f2aefe00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad406c920 0x7f2ad406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:37.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.297+0000 7f2aefe00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad406c920 0x7f2ad406ede0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0034000 tx=0x7f2ae002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:37.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.300+0000 7f2aeea00640 1 -- 172.21.15.69:0/892409985 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2af80ce660 con 0x7f2af80b4070 2024-04-24T08:11:37.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:37.500+0000 7f2afda00640 1 -- 172.21.15.69:0/892409985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f2af8075dc0 con 0x7f2af80b4070 2024-04-24T08:11:38.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.329+0000 7f2aeea00640 1 -- 172.21.15.69:0/892409985 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v54) ==== 138+0+0 (secure 0 0 0) 0x7f2ae801f020 con 0x7f2af80b4070 2024-04-24T08:11:38.331 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' created 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 -- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad406c920 msgr2=0x7f2ad406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad406c920 0x7f2ad406ede0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0034000 tx=0x7f2ae002e000 comp rx=0 tx=0).stop 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 -- 172.21.15.69:0/892409985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 msgr2=0x7f2af80c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 0x7f2af80c7470 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800b410 tx=0x7f2ae800b8d0 comp rx=0 tx=0).stop 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 -- 172.21.15.69:0/892409985 shutdown_connections 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad406c920 0x7f2ad406ede0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af815c590 0x7f2af80c7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af80d0d00 0x7f2af80c79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 --2- 172.21.15.69:0/892409985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af80b4070 0x7f2af80c7470 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 -- 172.21.15.69:0/892409985 >> 172.21.15.69:0/892409985 conn(0x7f2af806be40 msgr2=0x7f2af80cdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 -- 172.21.15.69:0/892409985 shutdown_connections 2024-04-24T08:11:38.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.333+0000 7f2afda00640 1 -- 172.21.15.69:0/892409985 wait complete. 2024-04-24T08:11:38.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T08:11:38.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.554+0000 7f9fd3800640 1 Processor -- start 2024-04-24T08:11:38.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.555+0000 7f9fd3800640 1 -- start start 2024-04-24T08:11:38.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.556+0000 7f9fd3800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9fcc06bc60 0x7f9fcc06c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:38.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.556+0000 7f9fd3800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9fcc0ad0d0 0x7f9fcc0ad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:38.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.556+0000 7f9fd3800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fcc0d0000 con 0x7f9fcc0cfc20 2024-04-24T08:11:38.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.556+0000 7f9fd3800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9fcc15ed90 con 0x7f9fcc06bc60 2024-04-24T08:11:38.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.556+0000 7f9fd3800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9fcc0ce9e0 con 0x7f9fcc0ad0d0 2024-04-24T08:11:38.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.556+0000 7f9fcbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9fcc06bc60 0x7f9fcc06c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:38.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.556+0000 7f9fcbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9fcc06bc60 0x7f9fcc06c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33232/0 (socket says 172.21.15.69:33232) 2024-04-24T08:11:38.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.556+0000 7f9fcbe00640 1 -- 172.21.15.69:0/3841557765 learned_addr learned my addr 172.21.15.69:0/3841557765 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:38.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.557+0000 7f9fcbe00640 1 -- 172.21.15.69:0/3841557765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9fcc0ad0d0 msgr2=0x7f9fcc0ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:38.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.557+0000 7f9fcbe00640 1 --2- 172.21.15.69:0/3841557765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9fcc0ad0d0 0x7f9fcc0ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:38.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.557+0000 7f9fcbe00640 1 -- 172.21.15.69:0/3841557765 >> v1:172.21.15.121:6789/0 conn(0x7f9fcc0cfc20 legacy=0x7f9fcc0b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:38.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.557+0000 7f9fcbe00640 1 -- 172.21.15.69:0/3841557765 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fcc0cee70 con 0x7f9fcc06bc60 2024-04-24T08:11:38.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.558+0000 7f9fcbe00640 1 --2- 172.21.15.69:0/3841557765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 0x7f9fcc06c0f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009870 tx=0x7f9fb802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55782b2d3307a02 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:38.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.558+0000 7f9fcb400640 1 -- 172.21.15.69:0/3841557765 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8037070 con 0x7f9fcc06bc60 2024-04-24T08:11:38.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.558+0000 7f9fcb400640 1 -- 172.21.15.69:0/3841557765 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fb8032440 con 0x7f9fcc06bc60 2024-04-24T08:11:38.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.558+0000 7f9fcb400640 1 -- 172.21.15.69:0/3841557765 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8032810 con 0x7f9fcc06bc60 2024-04-24T08:11:38.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.559+0000 7f9fd3800640 1 -- 172.21.15.69:0/3841557765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 msgr2=0x7f9fcc06c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:38.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.559+0000 7f9fd3800640 1 --2- 172.21.15.69:0/3841557765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 0x7f9fcc06c0f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009870 tx=0x7f9fb802a210 comp rx=0 tx=0).stop 2024-04-24T08:11:38.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.559+0000 7f9fd3800640 1 -- 172.21.15.69:0/3841557765 shutdown_connections 2024-04-24T08:11:38.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.559+0000 7f9fd3800640 1 --2- 172.21.15.69:0/3841557765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9fcc0ad0d0 0x7f9fcc0ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:38.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.559+0000 7f9fd3800640 1 --2- 172.21.15.69:0/3841557765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 0x7f9fcc06c0f0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:38.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.559+0000 7f9fd3800640 1 -- 172.21.15.69:0/3841557765 >> 172.21.15.69:0/3841557765 conn(0x7f9fcc0d0720 msgr2=0x7f9fcc0d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:38.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.559+0000 7f9fd3800640 1 -- 172.21.15.69:0/3841557765 shutdown_connections 2024-04-24T08:11:38.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.559+0000 7f9fd3800640 1 -- 172.21.15.69:0/3841557765 wait complete. 2024-04-24T08:11:38.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.561+0000 7f9fd3800640 1 Processor -- start 2024-04-24T08:11:38.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.561+0000 7f9fd3800640 1 -- start start 2024-04-24T08:11:38.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.562+0000 7f9fd3800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 0x7f9fcc14c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:38.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.562+0000 7f9fd3800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9fcc0ad0d0 0x7f9fcc0aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:38.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.562+0000 7f9fd3800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9fcc0cfc20 0x7f9fcc073440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:38.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.562+0000 7f9fd3800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9fcc0ce670 con 0x7f9fcc06bc60 2024-04-24T08:11:38.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.562+0000 7f9fd3800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9fcc0ce390 con 0x7f9fcc0ad0d0 2024-04-24T08:11:38.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.562+0000 7f9fd3800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9fcc0b4500 con 0x7f9fcc0cfc20 2024-04-24T08:11:38.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.562+0000 7f9fd0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 0x7f9fcc14c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:38.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.562+0000 7f9fd0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 0x7f9fcc14c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33236/0 (socket says 172.21.15.69:33236) 2024-04-24T08:11:38.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.562+0000 7f9fd0e00640 1 -- 172.21.15.69:0/3096667549 learned_addr learned my addr 172.21.15.69:0/3096667549 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:38.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fd1800640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9fcc0cfc20 0x7f9fcc073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:38.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fcbe00640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9fcc0ad0d0 0x7f9fcc0aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:38.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fd0e00640 1 -- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9fcc0cfc20 msgr2=0x7f9fcc073440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:38.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fd0e00640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9fcc0cfc20 0x7f9fcc073440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:38.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fd0e00640 1 -- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9fcc0ad0d0 msgr2=0x7f9fcc0aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:38.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fd0e00640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9fcc0ad0d0 0x7f9fcc0aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:38.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fd0e00640 1 -- 172.21.15.69:0/3096667549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fcc0af440 con 0x7f9fcc06bc60 2024-04-24T08:11:38.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fcbe00640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9fcc0ad0d0 0x7f9fcc0aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:38.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fd0e00640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 0x7f9fcc14c470 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000c410 tx=0x7f9fc000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:38.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fc8c00640 1 -- 172.21.15.69:0/3096667549 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0017070 con 0x7f9fcc06bc60 2024-04-24T08:11:38.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fc8c00640 1 -- 172.21.15.69:0/3096667549 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fc00035f0 con 0x7f9fcc06bc60 2024-04-24T08:11:38.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fc8c00640 1 -- 172.21.15.69:0/3096667549 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc0003a20 con 0x7f9fcc06bc60 2024-04-24T08:11:38.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.563+0000 7f9fd3800640 1 -- 172.21.15.69:0/3096667549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fcc0af6c0 con 0x7f9fcc06bc60 2024-04-24T08:11:38.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.564+0000 7f9fd3800640 1 -- 172.21.15.69:0/3096667549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fcc0c96f0 con 0x7f9fcc06bc60 2024-04-24T08:11:38.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.565+0000 7f9fd3800640 1 -- 172.21.15.69:0/3096667549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fcc068a80 con 0x7f9fcc06bc60 2024-04-24T08:11:38.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.566+0000 7f9fc8c00640 1 -- 172.21.15.69:0/3096667549 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9fc0003bc0 con 0x7f9fcc06bc60 2024-04-24T08:11:38.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.567+0000 7f9fc8c00640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9fac06c8f0 0x7f9fac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:38.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.567+0000 7f9fc8c00640 1 -- 172.21.15.69:0/3096667549 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 5227+0+0 (secure 0 0 0) 0x7f9fc008ae30 con 0x7f9fcc06bc60 2024-04-24T08:11:38.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.568+0000 7f9fcbe00640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9fac06c8f0 0x7f9fac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:38.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.569+0000 7f9fcbe00640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9fac06c8f0 0x7f9fac06edb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8034000 tx=0x7f9fb802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:38.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.571+0000 7f9fc8c00640 1 -- 172.21.15.69:0/3096667549 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9fcc0af440 con 0x7f9fcc06bc60 2024-04-24T08:11:38.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:38.773+0000 7f9fd3800640 1 -- 172.21.15.69:0/3096667549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f9fcc0b4120 con 0x7f9fcc06bc60 2024-04-24T08:11:39.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.343+0000 7f9fc8c00640 1 -- 172.21.15.69:0/3096667549 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v55) ==== 170+0+0 (secure 0 0 0) 0x7f9fc00594b0 con 0x7f9fcc06bc60 2024-04-24T08:11:39.346 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' created 2024-04-24T08:11:39.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.347+0000 7f9fd3800640 1 -- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9fac06c8f0 msgr2=0x7f9fac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:39.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.347+0000 7f9fd3800640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9fac06c8f0 0x7f9fac06edb0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8034000 tx=0x7f9fb802e000 comp rx=0 tx=0).stop 2024-04-24T08:11:39.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.347+0000 7f9fd3800640 1 -- 172.21.15.69:0/3096667549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 msgr2=0x7f9fcc14c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:39.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.347+0000 7f9fd3800640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 0x7f9fcc14c470 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9fc000c410 tx=0x7f9fc000c8d0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.347+0000 7f9fd3800640 1 -- 172.21.15.69:0/3096667549 shutdown_connections 2024-04-24T08:11:39.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.347+0000 7f9fd3800640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9fac06c8f0 0x7f9fac06edb0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.347+0000 7f9fd3800640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9fcc0cfc20 0x7f9fcc073440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.347+0000 7f9fd3800640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9fcc0ad0d0 0x7f9fcc0aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.347+0000 7f9fd3800640 1 --2- 172.21.15.69:0/3096667549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9fcc06bc60 0x7f9fcc14c470 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.347+0000 7f9fd3800640 1 -- 172.21.15.69:0/3096667549 >> 172.21.15.69:0/3096667549 conn(0x7f9fcc0d0720 msgr2=0x7f9fcc15a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:39.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.348+0000 7f9fd3800640 1 -- 172.21.15.69:0/3096667549 shutdown_connections 2024-04-24T08:11:39.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.348+0000 7f9fd3800640 1 -- 172.21.15.69:0/3096667549 wait complete. 2024-04-24T08:11:39.386 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T08:11:39.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.560+0000 7f03b9000640 1 Processor -- start 2024-04-24T08:11:39.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.560+0000 7f03b9000640 1 -- start start 2024-04-24T08:11:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.562+0000 7f03b9000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f03b415c9b0 0x7f03b40b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.562+0000 7f03b9000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b40b3f70 0x7f03b40cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.562+0000 7f03b9000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03b40cd6d0 con 0x7f03b4155f20 2024-04-24T08:11:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.562+0000 7f03b9000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f03b406db90 con 0x7f03b415c9b0 2024-04-24T08:11:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.562+0000 7f03b9000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f03b4072070 con 0x7f03b40b3f70 2024-04-24T08:11:39.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.563+0000 7f03b2800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b40b3f70 0x7f03b40cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:39.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.563+0000 7f03b2800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b40b3f70 0x7f03b40cf360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:46026/0 (socket says 172.21.15.69:46026) 2024-04-24T08:11:39.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.563+0000 7f03b2800640 1 -- 172.21.15.69:0/1144758009 learned_addr learned my addr 172.21.15.69:0/1144758009 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:39.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.563+0000 7f03b1400640 1 --2- 172.21.15.69:0/1144758009 >> v2:172.21.15.121:3300/0 conn(0x7f03b415c9b0 0x7f03b40b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:39.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.563+0000 7f03b2800640 1 -- 172.21.15.69:0/1144758009 >> v2:172.21.15.121:3300/0 conn(0x7f03b415c9b0 msgr2=0x7f03b40b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:39.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.563+0000 7f03b2800640 1 --2- 172.21.15.69:0/1144758009 >> v2:172.21.15.121:3300/0 conn(0x7f03b415c9b0 0x7f03b40b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.563+0000 7f03b2800640 1 -- 172.21.15.69:0/1144758009 >> v1:172.21.15.121:6789/0 conn(0x7f03b4155f20 legacy=0x7f03b40ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:39.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.563+0000 7f03b2800640 1 -- 172.21.15.69:0/1144758009 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b40cd9b0 con 0x7f03b40b3f70 2024-04-24T08:11:39.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.566+0000 7f03b2800640 1 --2- 172.21.15.69:0/1144758009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b40b3f70 0x7f03b40cf360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009870 tx=0x7f03ac02a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9564160d6639ad2f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:39.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.566+0000 7f03b0a00640 1 -- 172.21.15.69:0/1144758009 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac037070 con 0x7f03b40b3f70 2024-04-24T08:11:39.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.567+0000 7f03b0a00640 1 -- 172.21.15.69:0/1144758009 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ac02ae50 con 0x7f03b40b3f70 2024-04-24T08:11:39.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.567+0000 7f03b0a00640 1 -- 172.21.15.69:0/1144758009 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac032860 con 0x7f03b40b3f70 2024-04-24T08:11:39.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.567+0000 7f03b9000640 1 -- 172.21.15.69:0/1144758009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b40b3f70 msgr2=0x7f03b40cf360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:39.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.567+0000 7f03b9000640 1 --2- 172.21.15.69:0/1144758009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b40b3f70 0x7f03b40cf360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009870 tx=0x7f03ac02a300 comp rx=0 tx=0).stop 2024-04-24T08:11:39.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.567+0000 7f03b9000640 1 -- 172.21.15.69:0/1144758009 shutdown_connections 2024-04-24T08:11:39.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.567+0000 7f03b9000640 1 --2- 172.21.15.69:0/1144758009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b40b3f70 0x7f03b40cf360 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.567+0000 7f03b9000640 1 --2- 172.21.15.69:0/1144758009 >> v2:172.21.15.121:3300/0 conn(0x7f03b415c9b0 0x7f03b40b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.567+0000 7f03b9000640 1 -- 172.21.15.69:0/1144758009 >> 172.21.15.69:0/1144758009 conn(0x7f03b40d0350 msgr2=0x7f03b40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:39.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.567+0000 7f03b9000640 1 -- 172.21.15.69:0/1144758009 shutdown_connections 2024-04-24T08:11:39.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.567+0000 7f03b9000640 1 -- 172.21.15.69:0/1144758009 wait complete. 2024-04-24T08:11:39.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.569+0000 7f03b9000640 1 Processor -- start 2024-04-24T08:11:39.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.569+0000 7f03b9000640 1 -- start start 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.570+0000 7f03b9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03b40b3f70 0x7f03b4158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.570+0000 7f03b9000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b4155f20 0x7f03b414e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.570+0000 7f03b9000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b415c9b0 0x7f03b4156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.570+0000 7f03b9000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f03b40cf8a0 con 0x7f03b40b3f70 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.570+0000 7f03b9000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f03b4071d00 con 0x7f03b415c9b0 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.570+0000 7f03b9000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f03b40cfa20 con 0x7f03b4155f20 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.570+0000 7f03b1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03b40b3f70 0x7f03b4158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.570+0000 7f03b1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03b40b3f70 0x7f03b4158ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47294/0 (socket says 172.21.15.69:47294) 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.570+0000 7f03b1e00640 1 -- 172.21.15.69:0/3537582543 learned_addr learned my addr 172.21.15.69:0/3537582543 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.571+0000 7f03b1400640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b4155f20 0x7f03b414e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:39.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.571+0000 7f03b2800640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b415c9b0 0x7f03b4156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.571+0000 7f03b1e00640 1 -- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b4155f20 msgr2=0x7f03b414e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.571+0000 7f03b1e00640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b4155f20 0x7f03b414e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.571+0000 7f03b1e00640 1 -- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b415c9b0 msgr2=0x7f03b4156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.571+0000 7f03b1e00640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b415c9b0 0x7f03b4156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.571+0000 7f03b1e00640 1 -- 172.21.15.69:0/3537582543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b414fd80 con 0x7f03b40b3f70 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.571+0000 7f03b2800640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b415c9b0 0x7f03b4156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.571+0000 7f03b1e00640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03b40b3f70 0x7f03b4158ca0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f03a400c790 tx=0x7f03a400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.572+0000 7f03aa000640 1 -- 172.21.15.69:0/3537582543 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4016020 con 0x7f03b40b3f70 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.572+0000 7f03aa000640 1 -- 172.21.15.69:0/3537582543 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a4006d50 con 0x7f03b40b3f70 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.572+0000 7f03aa000640 1 -- 172.21.15.69:0/3537582543 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4003bb0 con 0x7f03b40b3f70 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.572+0000 7f03b9000640 1 -- 172.21.15.69:0/3537582543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b4150060 con 0x7f03b40b3f70 2024-04-24T08:11:39.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.572+0000 7f03b9000640 1 -- 172.21.15.69:0/3537582543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b4150620 con 0x7f03b40b3f70 2024-04-24T08:11:39.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.573+0000 7f03b9000640 1 -- 172.21.15.69:0/3537582543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b415c510 con 0x7f03b40b3f70 2024-04-24T08:11:39.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.574+0000 7f03aa000640 1 -- 172.21.15.69:0/3537582543 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f03a400a850 con 0x7f03b40b3f70 2024-04-24T08:11:39.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.575+0000 7f03aa000640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f039406c8f0 0x7f039406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:39.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.575+0000 7f03aa000640 1 -- 172.21.15.69:0/3537582543 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5609+0+0 (secure 0 0 0) 0x7f03a408b780 con 0x7f03b40b3f70 2024-04-24T08:11:39.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.576+0000 7f03b1400640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f039406c8f0 0x7f039406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:39.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.577+0000 7f03b1400640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f039406c8f0 0x7f039406edb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f03b415ee70 tx=0x7f03a0009190 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:39.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.580+0000 7f03aa000640 1 -- 172.21.15.69:0/3537582543 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f03b415c510 con 0x7f03b40b3f70 2024-04-24T08:11:39.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:39.822+0000 7f03b9000640 1 -- 172.21.15.69:0/3537582543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f03b40ce530 con 0x7f03b40b3f70 2024-04-24T08:11:40.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.348+0000 7f03aa000640 1 -- 172.21.15.69:0/3537582543 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v30) ==== 172+0+0 (secure 0 0 0) 0x7f03a4020020 con 0x7f03b40b3f70 2024-04-24T08:11:40.350 INFO:teuthology.orchestra.run.smithi069.stderr:new fs with metadata pool 8 and data pool 9 2024-04-24T08:11:40.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.352+0000 7f03b9000640 1 -- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f039406c8f0 msgr2=0x7f039406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:40.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.352+0000 7f03b9000640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f039406c8f0 0x7f039406edb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f03b415ee70 tx=0x7f03a0009190 comp rx=0 tx=0).stop 2024-04-24T08:11:40.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.352+0000 7f03b9000640 1 -- 172.21.15.69:0/3537582543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03b40b3f70 msgr2=0x7f03b4158ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:40.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.352+0000 7f03b9000640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03b40b3f70 0x7f03b4158ca0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f03a400c790 tx=0x7f03a400cc50 comp rx=0 tx=0).stop 2024-04-24T08:11:40.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.352+0000 7f03b9000640 1 -- 172.21.15.69:0/3537582543 shutdown_connections 2024-04-24T08:11:40.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.352+0000 7f03b9000640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f039406c8f0 0x7f039406edb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.352+0000 7f03b9000640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03b415c9b0 0x7f03b4156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.352+0000 7f03b9000640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03b4155f20 0x7f03b414e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.352+0000 7f03b9000640 1 --2- 172.21.15.69:0/3537582543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03b40b3f70 0x7f03b4158ca0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.352+0000 7f03b9000640 1 -- 172.21.15.69:0/3537582543 >> 172.21.15.69:0/3537582543 conn(0x7f03b40d0350 msgr2=0x7f03b40cef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:40.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.353+0000 7f03b9000640 1 -- 172.21.15.69:0/3537582543 shutdown_connections 2024-04-24T08:11:40.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.354+0000 7f03b9000640 1 -- 172.21.15.69:0/3537582543 wait complete. 2024-04-24T08:11:40.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:11:40.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.569+0000 7f1b64400640 1 Processor -- start 2024-04-24T08:11:40.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.569+0000 7f1b64400640 1 -- start start 2024-04-24T08:11:40.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.569+0000 7f1b64400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1b5c145a90 0x7f1b5c06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:40.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.569+0000 7f1b64400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 0x7f1b5c160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:40.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.569+0000 7f1b64400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b5c0cf3d0 con 0x7f1b5c0d0300 2024-04-24T08:11:40.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.569+0000 7f1b64400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1b5c0cf6b0 con 0x7f1b5c145a90 2024-04-24T08:11:40.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.569+0000 7f1b64400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1b5c0cf990 con 0x7f1b5c145fe0 2024-04-24T08:11:40.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.571+0000 7f1b62400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 0x7f1b5c160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:40.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.571+0000 7f1b62400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 0x7f1b5c160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:46032/0 (socket says 172.21.15.69:46032) 2024-04-24T08:11:40.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.571+0000 7f1b62400640 1 -- 172.21.15.69:0/3046273104 learned_addr learned my addr 172.21.15.69:0/3046273104 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:40.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.572+0000 7f1b61000640 1 --2- 172.21.15.69:0/3046273104 >> v2:172.21.15.121:3300/0 conn(0x7f1b5c145a90 0x7f1b5c06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:40.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.572+0000 7f1b62400640 1 -- 172.21.15.69:0/3046273104 >> v2:172.21.15.121:3300/0 conn(0x7f1b5c145a90 msgr2=0x7f1b5c06e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:40.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.572+0000 7f1b62400640 1 --2- 172.21.15.69:0/3046273104 >> v2:172.21.15.121:3300/0 conn(0x7f1b5c145a90 0x7f1b5c06e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.572+0000 7f1b62400640 1 -- 172.21.15.69:0/3046273104 >> v1:172.21.15.121:6789/0 conn(0x7f1b5c0d0300 legacy=0x7f1b5c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:40.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.572+0000 7f1b62400640 1 -- 172.21.15.69:0/3046273104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c0cfc70 con 0x7f1b5c145fe0 2024-04-24T08:11:40.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.573+0000 7f1b62400640 1 --2- 172.21.15.69:0/3046273104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 0x7f1b5c160540 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009e50 tx=0x7f1b5402a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50430570ff29be97 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:40.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b5be00640 1 -- 172.21.15.69:0/3046273104 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b54036070 con 0x7f1b5c145fe0 2024-04-24T08:11:40.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b5be00640 1 -- 172.21.15.69:0/3046273104 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b54031600 con 0x7f1b5c145fe0 2024-04-24T08:11:40.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b5be00640 1 -- 172.21.15.69:0/3046273104 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b540319d0 con 0x7f1b5c145fe0 2024-04-24T08:11:40.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b64400640 1 -- 172.21.15.69:0/3046273104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 msgr2=0x7f1b5c160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:40.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b64400640 1 --2- 172.21.15.69:0/3046273104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 0x7f1b5c160540 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009e50 tx=0x7f1b5402a710 comp rx=0 tx=0).stop 2024-04-24T08:11:40.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b64400640 1 -- 172.21.15.69:0/3046273104 shutdown_connections 2024-04-24T08:11:40.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b64400640 1 --2- 172.21.15.69:0/3046273104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 0x7f1b5c160540 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b64400640 1 --2- 172.21.15.69:0/3046273104 >> v2:172.21.15.121:3300/0 conn(0x7f1b5c145a90 0x7f1b5c06e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b64400640 1 -- 172.21.15.69:0/3046273104 >> 172.21.15.69:0/3046273104 conn(0x7f1b5c067d90 msgr2=0x7f1b5c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:40.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b64400640 1 -- 172.21.15.69:0/3046273104 shutdown_connections 2024-04-24T08:11:40.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.574+0000 7f1b64400640 1 -- 172.21.15.69:0/3046273104 wait complete. 2024-04-24T08:11:40.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.576+0000 7f1b64400640 1 Processor -- start 2024-04-24T08:11:40.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.576+0000 7f1b64400640 1 -- start start 2024-04-24T08:11:40.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.577+0000 7f1b64400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b5c0d0300 0x7f1b5c151260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:40.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.577+0000 7f1b64400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b5c145a90 0x7f1b5c13de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:40.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.577+0000 7f1b64400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 0x7f1b5c15df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:40.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.577+0000 7f1b64400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c1458f0 con 0x7f1b5c145a90 2024-04-24T08:11:40.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.577+0000 7f1b64400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5c145770 con 0x7f1b5c0d0300 2024-04-24T08:11:40.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.577+0000 7f1b64400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1b5c0cdc30 con 0x7f1b5c145fe0 2024-04-24T08:11:40.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b61000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b5c145a90 0x7f1b5c13de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:40.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b61000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b5c145a90 0x7f1b5c13de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47302/0 (socket says 172.21.15.69:47302) 2024-04-24T08:11:40.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b61000640 1 -- 172.21.15.69:0/63490570 learned_addr learned my addr 172.21.15.69:0/63490570 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:40.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b61a00640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b5c0d0300 0x7f1b5c151260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:40.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b62400640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 0x7f1b5c15df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:40.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b61000640 1 -- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 msgr2=0x7f1b5c15df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:40.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b61000640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 0x7f1b5c15df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b61000640 1 -- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b5c0d0300 msgr2=0x7f1b5c151260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b61000640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b5c0d0300 0x7f1b5c151260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b61000640 1 -- 172.21.15.69:0/63490570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b5c1517a0 con 0x7f1b5c145a90 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.578+0000 7f1b61000640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b5c145a90 0x7f1b5c13de90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800e9b0 tx=0x7f1b4800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.579+0000 7f1b59600640 1 -- 172.21.15.69:0/63490570 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b48017070 con 0x7f1b5c145a90 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.579+0000 7f1b59600640 1 -- 172.21.15.69:0/63490570 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b4800c9a0 con 0x7f1b5c145a90 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.579+0000 7f1b64400640 1 -- 172.21.15.69:0/63490570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5c1417c0 con 0x7f1b5c145a90 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.579+0000 7f1b59600640 1 -- 172.21.15.69:0/63490570 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4800cd90 con 0x7f1b5c145a90 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.579+0000 7f1b64400640 1 -- 172.21.15.69:0/63490570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b5c13eb30 con 0x7f1b5c145a90 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.580+0000 7f1b64400640 1 -- 172.21.15.69:0/63490570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b5c0cce60 con 0x7f1b5c145a90 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.581+0000 7f1b59600640 1 -- 172.21.15.69:0/63490570 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1b4801a070 con 0x7f1b5c145a90 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.582+0000 7f1b59600640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3c06c8f0 0x7f1b3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:40.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.582+0000 7f1b59600640 1 -- 172.21.15.69:0/63490570 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f1b4808bc00 con 0x7f1b5c145a90 2024-04-24T08:11:40.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.583+0000 7f1b61a00640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3c06c8f0 0x7f1b3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:40.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.584+0000 7f1b61a00640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3c06c8f0 0x7f1b3c06edb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1b50001640 tx=0x7f1b50009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:40.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.587+0000 7f1b59600640 1 -- 172.21.15.69:0/63490570 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b5c0cce60 con 0x7f1b5c145a90 2024-04-24T08:11:40.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.778+0000 7f1b64400640 1 -- 172.21.15.69:0/63490570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b5c0afcb0 con 0x7f1b5c145a90 2024-04-24T08:11:40.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.779+0000 7f1b59600640 1 -- 172.21.15.69:0/63490570 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16654 (secure 0 0 0) 0x7f1b5c0afcb0 con 0x7f1b5c145a90 2024-04-24T08:11:40.780 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:40.780 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":56,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:39.824891+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:11:37.517598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-24T08:11:38.791285+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:11:40.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 -- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3c06c8f0 msgr2=0x7f1b3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3c06c8f0 0x7f1b3c06edb0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1b50001640 tx=0x7f1b50009040 comp rx=0 tx=0).stop 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 -- 172.21.15.69:0/63490570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b5c145a90 msgr2=0x7f1b5c13de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b5c145a90 0x7f1b5c13de90 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800e9b0 tx=0x7f1b4800ee70 comp rx=0 tx=0).stop 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 -- 172.21.15.69:0/63490570 shutdown_connections 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3c06c8f0 0x7f1b3c06edb0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b5c145a90 0x7f1b5c13de90 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5c145fe0 0x7f1b5c15df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 --2- 172.21.15.69:0/63490570 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b5c0d0300 0x7f1b5c151260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 -- 172.21.15.69:0/63490570 >> 172.21.15.69:0/63490570 conn(0x7f1b5c067d90 msgr2=0x7f1b5c0cc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 -- 172.21.15.69:0/63490570 shutdown_connections 2024-04-24T08:11:40.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.782+0000 7f1b64400640 1 -- 172.21.15.69:0/63490570 wait complete. 2024-04-24T08:11:40.799 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:11:40.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.993+0000 7f1848a00640 1 Processor -- start 2024-04-24T08:11:40.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.993+0000 7f1848a00640 1 -- start start 2024-04-24T08:11:40.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.994+0000 7f1848a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f184415ab10 0x7f1844144090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:40.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.994+0000 7f1848a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1844071be0 0x7f184406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:40.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.994+0000 7f1848a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18440b3190 con 0x7f184406e960 2024-04-24T08:11:40.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.994+0000 7f1848a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f184406edd0 con 0x7f184415ab10 2024-04-24T08:11:40.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.994+0000 7f1848a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f18440ad590 con 0x7f1844071be0 2024-04-24T08:11:40.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.995+0000 7f183be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1844071be0 0x7f184406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:40.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.995+0000 7f183be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1844071be0 0x7f184406e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:46052/0 (socket says 172.21.15.69:46052) 2024-04-24T08:11:40.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.995+0000 7f183be00640 1 -- 172.21.15.69:0/2332589757 learned_addr learned my addr 172.21.15.69:0/2332589757 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:40.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.995+0000 7f1843400640 1 --2- 172.21.15.69:0/2332589757 >> v2:172.21.15.121:3300/0 conn(0x7f184415ab10 0x7f1844144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:40.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.996+0000 7f183be00640 1 -- 172.21.15.69:0/2332589757 >> v2:172.21.15.121:3300/0 conn(0x7f184415ab10 msgr2=0x7f1844144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:40.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.996+0000 7f183be00640 1 --2- 172.21.15.69:0/2332589757 >> v2:172.21.15.121:3300/0 conn(0x7f184415ab10 0x7f1844144090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.996+0000 7f183be00640 1 -- 172.21.15.69:0/2332589757 >> v1:172.21.15.121:6789/0 conn(0x7f184406e960 legacy=0x7f184415e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:40.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.996+0000 7f183be00640 1 -- 172.21.15.69:0/2332589757 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18440b3400 con 0x7f1844071be0 2024-04-24T08:11:40.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.997+0000 7f183be00640 1 --2- 172.21.15.69:0/2332589757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1844071be0 0x7f184406e420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f182c00bb00 tx=0x7f182c02c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e2d4fc1e8e9b4af server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:40.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.998+0000 7f1842a00640 1 -- 172.21.15.69:0/2332589757 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c03a070 con 0x7f1844071be0 2024-04-24T08:11:40.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.998+0000 7f1842a00640 1 -- 172.21.15.69:0/2332589757 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c032480 con 0x7f1844071be0 2024-04-24T08:11:40.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.998+0000 7f1842a00640 1 -- 172.21.15.69:0/2332589757 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c032850 con 0x7f1844071be0 2024-04-24T08:11:40.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.998+0000 7f1848a00640 1 -- 172.21.15.69:0/2332589757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1844071be0 msgr2=0x7f184406e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:40.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.998+0000 7f1848a00640 1 --2- 172.21.15.69:0/2332589757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1844071be0 0x7f184406e420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f182c00bb00 tx=0x7f182c02c6e0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.998+0000 7f1848a00640 1 -- 172.21.15.69:0/2332589757 shutdown_connections 2024-04-24T08:11:40.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.998+0000 7f1848a00640 1 --2- 172.21.15.69:0/2332589757 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1844071be0 0x7f184406e420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.998+0000 7f1848a00640 1 --2- 172.21.15.69:0/2332589757 >> v2:172.21.15.121:3300/0 conn(0x7f184415ab10 0x7f1844144090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:40.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.998+0000 7f1848a00640 1 -- 172.21.15.69:0/2332589757 >> 172.21.15.69:0/2332589757 conn(0x7f18440d0160 msgr2=0x7f18440d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:40.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.998+0000 7f1848a00640 1 -- 172.21.15.69:0/2332589757 shutdown_connections 2024-04-24T08:11:40.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:40.999+0000 7f1848a00640 1 -- 172.21.15.69:0/2332589757 wait complete. 2024-04-24T08:11:41.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.000+0000 7f1848a00640 1 Processor -- start 2024-04-24T08:11:41.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.000+0000 7f1848a00640 1 -- start start 2024-04-24T08:11:41.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.001+0000 7f1848a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f184406e960 0x7f18440adc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.001+0000 7f1848a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1844071be0 0x7f18440ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.001+0000 7f1848a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f184415ab10 0x7f18441f5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.001+0000 7f1848a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1844072150 con 0x7f1844071be0 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.001+0000 7f1848a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f18440b39c0 con 0x7f184406e960 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.001+0000 7f1848a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f184415aef0 con 0x7f184415ab10 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f183be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1844071be0 0x7f18440ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f183be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1844071be0 0x7f18440ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47316/0 (socket says 172.21.15.69:47316) 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f183be00640 1 -- 172.21.15.69:0/2131105876 learned_addr learned my addr 172.21.15.69:0/2131105876 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f1843400640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f184406e960 0x7f18440adc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f183be00640 1 -- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f184415ab10 msgr2=0x7f18441f5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f1843e00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f184415ab10 0x7f18441f5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:41.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f183be00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f184415ab10 0x7f18441f5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f183be00640 1 -- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f184406e960 msgr2=0x7f18440adc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:41.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f183be00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f184406e960 0x7f18440adc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f183be00640 1 -- 172.21.15.69:0/2131105876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184409dd80 con 0x7f1844071be0 2024-04-24T08:11:41.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f1843400640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f184406e960 0x7f18440adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:41.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.002+0000 7f1843e00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f184415ab10 0x7f18441f5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:41.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.003+0000 7f183be00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1844071be0 0x7f18440ae1e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f182c038040 tx=0x7f182c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:41.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.003+0000 7f183b400640 1 -- 172.21.15.69:0/2131105876 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c03a070 con 0x7f1844071be0 2024-04-24T08:11:41.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.003+0000 7f183b400640 1 -- 172.21.15.69:0/2131105876 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c035860 con 0x7f1844071be0 2024-04-24T08:11:41.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.003+0000 7f183b400640 1 -- 172.21.15.69:0/2131105876 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c035c50 con 0x7f1844071be0 2024-04-24T08:11:41.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.003+0000 7f1848a00640 1 -- 172.21.15.69:0/2131105876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f184409e020 con 0x7f1844071be0 2024-04-24T08:11:41.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.003+0000 7f1848a00640 1 -- 172.21.15.69:0/2131105876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18441f5420 con 0x7f1844071be0 2024-04-24T08:11:41.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.004+0000 7f1848a00640 1 -- 172.21.15.69:0/2131105876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18440694b0 con 0x7f1844071be0 2024-04-24T08:11:41.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.005+0000 7f183b400640 1 -- 172.21.15.69:0/2131105876 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f182c047060 con 0x7f1844071be0 2024-04-24T08:11:41.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.006+0000 7f183b400640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f182006c920 0x7f182006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:41.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.006+0000 7f183b400640 1 -- 172.21.15.69:0/2131105876 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5709+0+0 (secure 0 0 0) 0x7f182c0ac150 con 0x7f1844071be0 2024-04-24T08:11:41.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.007+0000 7f1843400640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f182006c920 0x7f182006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:41.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.008+0000 7f1843400640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f182006c920 0x7f182006ede0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f18441e9e80 tx=0x7f1834008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:41.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.010+0000 7f183b400640 1 -- 172.21.15.69:0/2131105876 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f18440694b0 con 0x7f1844071be0 2024-04-24T08:11:41.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.209+0000 7f1848a00640 1 -- 172.21.15.69:0/2131105876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f184406f020 con 0x7f1844071be0 2024-04-24T08:11:41.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.212+0000 7f183b400640 1 -- 172.21.15.69:0/2131105876 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16654 (secure 0 0 0) 0x7f182c07a4e0 con 0x7f1844071be0 2024-04-24T08:11:41.213 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:41.213 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":56,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:39.824891+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:11:37.517598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-24T08:11:38.791285+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:11:41.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.214+0000 7f1848a00640 1 -- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f182006c920 msgr2=0x7f182006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f182006c920 0x7f182006ede0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f18441e9e80 tx=0x7f1834008040 comp rx=0 tx=0).stop 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 -- 172.21.15.69:0/2131105876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1844071be0 msgr2=0x7f18440ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1844071be0 0x7f18440ae1e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f182c038040 tx=0x7f182c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 -- 172.21.15.69:0/2131105876 shutdown_connections 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f182006c920 0x7f182006ede0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1844071be0 0x7f18440ae1e0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f184415ab10 0x7f18441f5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 --2- 172.21.15.69:0/2131105876 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f184406e960 0x7f18440adc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 -- 172.21.15.69:0/2131105876 >> 172.21.15.69:0/2131105876 conn(0x7f18440d0160 msgr2=0x7f18440b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 -- 172.21.15.69:0/2131105876 shutdown_connections 2024-04-24T08:11:41.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.215+0000 7f1848a00640 1 -- 172.21.15.69:0/2131105876 wait complete. 2024-04-24T08:11:41.233 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T08:11:41.305 INFO:tasks.ceph.mds.c.smithi069.stderr:2024-04-24T08:11:41.305+0000 7f0431800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:11:41.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.432+0000 7f20df600640 1 Processor -- start 2024-04-24T08:11:41.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.432+0000 7f20df600640 1 -- start start 2024-04-24T08:11:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.434+0000 7f20df600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20d815c820 0x7f20d80b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.434+0000 7f20df600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f20d80b46a0 0x7f20d806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.434+0000 7f20df600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d815cc00 con 0x7f20d806fd10 2024-04-24T08:11:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.434+0000 7f20df600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f20d806ed30 con 0x7f20d80b46a0 2024-04-24T08:11:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.434+0000 7f20df600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f20d80b4e10 con 0x7f20d815c820 2024-04-24T08:11:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.434+0000 7f20d7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f20d80b46a0 0x7f20d806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.434+0000 7f20d7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f20d80b46a0 0x7f20d806f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47332/0 (socket says 172.21.15.69:47332) 2024-04-24T08:11:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.434+0000 7f20d7e00640 1 -- 172.21.15.69:0/2100172214 learned_addr learned my addr 172.21.15.69:0/2100172214 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.434+0000 7f20dcc00640 1 --2- 172.21.15.69:0/2100172214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20d815c820 0x7f20d80b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:41.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.435+0000 7f20d7e00640 1 -- 172.21.15.69:0/2100172214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20d815c820 msgr2=0x7f20d80b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:41.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.435+0000 7f20d7e00640 1 --2- 172.21.15.69:0/2100172214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20d815c820 0x7f20d80b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.435+0000 7f20d7e00640 1 -- 172.21.15.69:0/2100172214 >> v1:172.21.15.121:6789/0 conn(0x7f20d806fd10 legacy=0x7f20d80ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:41.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.435+0000 7f20d7e00640 1 -- 172.21.15.69:0/2100172214 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d80b50f0 con 0x7f20d80b46a0 2024-04-24T08:11:41.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.436+0000 7f20d7e00640 1 --2- 172.21.15.69:0/2100172214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d80b46a0 0x7f20d806f7d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f20c0009870 tx=0x7f20c002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136047bfde8c5307 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:41.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.436+0000 7f20d7400640 1 -- 172.21.15.69:0/2100172214 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0037070 con 0x7f20d80b46a0 2024-04-24T08:11:41.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.436+0000 7f20d7400640 1 -- 172.21.15.69:0/2100172214 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c0032440 con 0x7f20d80b46a0 2024-04-24T08:11:41.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.436+0000 7f20d7400640 1 -- 172.21.15.69:0/2100172214 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c0032810 con 0x7f20d80b46a0 2024-04-24T08:11:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.436+0000 7f20df600640 1 -- 172.21.15.69:0/2100172214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d80b46a0 msgr2=0x7f20d806f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.436+0000 7f20df600640 1 --2- 172.21.15.69:0/2100172214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d80b46a0 0x7f20d806f7d0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f20c0009870 tx=0x7f20c002a270 comp rx=0 tx=0).stop 2024-04-24T08:11:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.437+0000 7f20df600640 1 -- 172.21.15.69:0/2100172214 shutdown_connections 2024-04-24T08:11:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.437+0000 7f20df600640 1 --2- 172.21.15.69:0/2100172214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d80b46a0 0x7f20d806f7d0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.437+0000 7f20df600640 1 --2- 172.21.15.69:0/2100172214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20d815c820 0x7f20d80b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.437+0000 7f20df600640 1 -- 172.21.15.69:0/2100172214 >> 172.21.15.69:0/2100172214 conn(0x7f20d80d0350 msgr2=0x7f20d80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.437+0000 7f20df600640 1 -- 172.21.15.69:0/2100172214 shutdown_connections 2024-04-24T08:11:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.437+0000 7f20df600640 1 -- 172.21.15.69:0/2100172214 wait complete. 2024-04-24T08:11:41.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.438+0000 7f20df600640 1 Processor -- start 2024-04-24T08:11:41.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.438+0000 7f20df600640 1 -- start start 2024-04-24T08:11:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.439+0000 7f20df600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20d806fd10 0x7f20d80cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.439+0000 7f20df600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d80b46a0 0x7f20d8159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.439+0000 7f20df600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d815c820 0x7f20d8159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.439+0000 7f20df600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20d80b4b00 con 0x7f20d815c820 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.439+0000 7f20df600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f20d806f2b0 con 0x7f20d80b46a0 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.439+0000 7f20df600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f20d80ce960 con 0x7f20d806fd10 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20dd600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d815c820 0x7f20d8159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20dd600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d815c820 0x7f20d8159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47336/0 (socket says 172.21.15.69:47336) 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20dd600640 1 -- 172.21.15.69:0/338680308 learned_addr learned my addr 172.21.15.69:0/338680308 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20d7e00640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d80b46a0 0x7f20d8159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20dcc00640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20d806fd10 0x7f20d80cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20dd600640 1 -- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20d806fd10 msgr2=0x7f20d80cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20dd600640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20d806fd10 0x7f20d80cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20dd600640 1 -- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d80b46a0 msgr2=0x7f20d8159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:41.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20dd600640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d80b46a0 0x7f20d8159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:41.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20dd600640 1 -- 172.21.15.69:0/338680308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d81f8240 con 0x7f20d815c820 2024-04-24T08:11:41.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20d7e00640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d80b46a0 0x7f20d8159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:41.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20dd600640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d815c820 0x7f20d8159540 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f20d000c6b0 tx=0x7f20d000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.440+0000 7f20d4c00640 1 -- 172.21.15.69:0/338680308 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0003b70 con 0x7f20d815c820 2024-04-24T08:11:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.441+0000 7f20d4c00640 1 -- 172.21.15.69:0/338680308 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d0003d30 con 0x7f20d815c820 2024-04-24T08:11:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.441+0000 7f20df600640 1 -- 172.21.15.69:0/338680308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d81f8520 con 0x7f20d815c820 2024-04-24T08:11:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.441+0000 7f20d4c00640 1 -- 172.21.15.69:0/338680308 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0005310 con 0x7f20d815c820 2024-04-24T08:11:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.441+0000 7f20df600640 1 -- 172.21.15.69:0/338680308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d80cb210 con 0x7f20d815c820 2024-04-24T08:11:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.442+0000 7f20df600640 1 -- 172.21.15.69:0/338680308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d8068a80 con 0x7f20d815c820 2024-04-24T08:11:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.443+0000 7f20d4c00640 1 -- 172.21.15.69:0/338680308 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f20d00054b0 con 0x7f20d815c820 2024-04-24T08:11:41.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.443+0000 7f20d4c00640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20b406c920 0x7f20b406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:41.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.443+0000 7f20d4c00640 1 -- 172.21.15.69:0/338680308 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f20d008ba90 con 0x7f20d815c820 2024-04-24T08:11:41.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.444+0000 7f20dcc00640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20b406c920 0x7f20b406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:41.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.445+0000 7f20dcc00640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20b406c920 0x7f20b406ede0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f20c8006fd0 tx=0x7f20c8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:41.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.447+0000 7f20d4c00640 1 -- 172.21.15.69:0/338680308 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f20d0059d90 con 0x7f20d815c820 2024-04-24T08:11:41.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:41.688+0000 7f20df600640 1 -- 172.21.15.69:0/338680308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f20d80cddc0 con 0x7f20d815c820 2024-04-24T08:11:42.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.484+0000 7f20d4c00640 1 -- 172.21.15.69:0/338680308 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) ==== 118+0+0 (secure 0 0 0) 0x7f20d0059690 con 0x7f20d815c820 2024-04-24T08:11:42.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 -- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20b406c920 msgr2=0x7f20b406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:42.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20b406c920 0x7f20b406ede0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f20c8006fd0 tx=0x7f20c8008040 comp rx=0 tx=0).stop 2024-04-24T08:11:42.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 -- 172.21.15.69:0/338680308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d815c820 msgr2=0x7f20d8159540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:42.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d815c820 0x7f20d8159540 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f20d000c6b0 tx=0x7f20d000cb70 comp rx=0 tx=0).stop 2024-04-24T08:11:42.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 -- 172.21.15.69:0/338680308 shutdown_connections 2024-04-24T08:11:42.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20b406c920 0x7f20b406ede0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20d815c820 0x7f20d8159540 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20d80b46a0 0x7f20d8159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 --2- 172.21.15.69:0/338680308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f20d806fd10 0x7f20d80cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 -- 172.21.15.69:0/338680308 >> 172.21.15.69:0/338680308 conn(0x7f20d80d0350 msgr2=0x7f20d80b3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:42.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 -- 172.21.15.69:0/338680308 shutdown_connections 2024-04-24T08:11:42.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.489+0000 7f20df600640 1 -- 172.21.15.69:0/338680308 wait complete. 2024-04-24T08:11:42.511 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-24T08:11:42.511 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T08:11:42.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.709+0000 7fea5fe00640 1 Processor -- start 2024-04-24T08:11:42.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.709+0000 7fea5fe00640 1 -- start start 2024-04-24T08:11:42.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.711+0000 7fea5fe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fea601420a0 0x7fea60071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:42.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.711+0000 7fea5fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fea601606e0 0x7fea600cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:42.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.711+0000 7fea5fe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea60072450 con 0x7fea60142480 2024-04-24T08:11:42.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.711+0000 7fea5fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fea600cf1e0 con 0x7fea601606e0 2024-04-24T08:11:42.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.711+0000 7fea5fe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fea60072730 con 0x7fea601420a0 2024-04-24T08:11:42.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.711+0000 7fea5f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fea601606e0 0x7fea600cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:42.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.711+0000 7fea5f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fea601606e0 0x7fea600cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47340/0 (socket says 172.21.15.69:47340) 2024-04-24T08:11:42.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.711+0000 7fea5f400640 1 -- 172.21.15.69:0/1843858237 learned_addr learned my addr 172.21.15.69:0/1843858237 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:42.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.711+0000 7fea5ea00640 1 --2- 172.21.15.69:0/1843858237 >> v2:172.21.15.121:3300/0 conn(0x7fea601420a0 0x7fea60071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:42.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.711+0000 7fea5f400640 1 -- 172.21.15.69:0/1843858237 >> v2:172.21.15.121:3300/0 conn(0x7fea601420a0 msgr2=0x7fea60071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:42.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.712+0000 7fea5f400640 1 --2- 172.21.15.69:0/1843858237 >> v2:172.21.15.121:3300/0 conn(0x7fea601420a0 0x7fea60071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.712+0000 7fea5f400640 1 -- 172.21.15.69:0/1843858237 >> v1:172.21.15.121:6789/0 conn(0x7fea60142480 legacy=0x7fea6015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:42.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.712+0000 7fea5f400640 1 -- 172.21.15.69:0/1843858237 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea60072a10 con 0x7fea601606e0 2024-04-24T08:11:42.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.712+0000 7fea5ea00640 1 --2- 172.21.15.69:0/1843858237 >> v2:172.21.15.121:3300/0 conn(0x7fea601420a0 0x7fea60071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:42.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.713+0000 7fea5f400640 1 --2- 172.21.15.69:0/1843858237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea601606e0 0x7fea600cf670 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fea600b5770 tx=0x7fea5802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8a7693f8b464c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:42.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.713+0000 7fea5d600640 1 -- 172.21.15.69:0/1843858237 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58036070 con 0x7fea601606e0 2024-04-24T08:11:42.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.713+0000 7fea5d600640 1 -- 172.21.15.69:0/1843858237 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea58031600 con 0x7fea601606e0 2024-04-24T08:11:42.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.713+0000 7fea5d600640 1 -- 172.21.15.69:0/1843858237 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea580319d0 con 0x7fea601606e0 2024-04-24T08:11:42.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.714+0000 7fea5fe00640 1 -- 172.21.15.69:0/1843858237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea601606e0 msgr2=0x7fea600cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:42.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.714+0000 7fea5fe00640 1 --2- 172.21.15.69:0/1843858237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea601606e0 0x7fea600cf670 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fea600b5770 tx=0x7fea5802a6f0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.714+0000 7fea5fe00640 1 -- 172.21.15.69:0/1843858237 shutdown_connections 2024-04-24T08:11:42.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.714+0000 7fea5fe00640 1 --2- 172.21.15.69:0/1843858237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea601606e0 0x7fea600cf670 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.714+0000 7fea5fe00640 1 --2- 172.21.15.69:0/1843858237 >> v2:172.21.15.121:3300/0 conn(0x7fea601420a0 0x7fea60071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.714+0000 7fea5fe00640 1 -- 172.21.15.69:0/1843858237 >> 172.21.15.69:0/1843858237 conn(0x7fea60067b80 msgr2=0x7fea6006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:42.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.714+0000 7fea5fe00640 1 -- 172.21.15.69:0/1843858237 shutdown_connections 2024-04-24T08:11:42.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.714+0000 7fea5fe00640 1 -- 172.21.15.69:0/1843858237 wait complete. 2024-04-24T08:11:42.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.716+0000 7fea5fe00640 1 Processor -- start 2024-04-24T08:11:42.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.716+0000 7fea5fe00640 1 -- start start 2024-04-24T08:11:42.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.717+0000 7fea5fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea601420a0 0x7fea600b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.717+0000 7fea5fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea60142480 0x7fea601f4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.717+0000 7fea5fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea601606e0 0x7fea6015b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.717+0000 7fea5fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fea600cfbb0 con 0x7fea60142480 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.717+0000 7fea5fe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea60142910 con 0x7fea601606e0 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.717+0000 7fea5fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fea600cfd30 con 0x7fea601420a0 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.717+0000 7fea5e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea60142480 0x7fea601f4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.717+0000 7fea5e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea60142480 0x7fea601f4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47350/0 (socket says 172.21.15.69:47350) 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.717+0000 7fea5e000640 1 -- 172.21.15.69:0/4163310216 learned_addr learned my addr 172.21.15.69:0/4163310216 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea5ea00640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea601420a0 0x7fea600b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea5f400640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea601606e0 0x7fea6015b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea5e000640 1 -- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea601420a0 msgr2=0x7fea600b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:42.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea5e000640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea601420a0 0x7fea600b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea5e000640 1 -- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea601606e0 msgr2=0x7fea6015b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:42.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea5e000640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea601606e0 0x7fea6015b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:42.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea5e000640 1 -- 172.21.15.69:0/4163310216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea60154f30 con 0x7fea60142480 2024-04-24T08:11:42.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea5e000640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea60142480 0x7fea601f4d60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00c430 tx=0x7fea4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:42.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea56a00640 1 -- 172.21.15.69:0/4163310216 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c011790 con 0x7fea60142480 2024-04-24T08:11:42.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea56a00640 1 -- 172.21.15.69:0/4163310216 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea4c017070 con 0x7fea60142480 2024-04-24T08:11:42.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea56a00640 1 -- 172.21.15.69:0/4163310216 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4c011b70 con 0x7fea60142480 2024-04-24T08:11:42.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea5fe00640 1 -- 172.21.15.69:0/4163310216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea60155210 con 0x7fea60142480 2024-04-24T08:11:42.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.718+0000 7fea5fe00640 1 -- 172.21.15.69:0/4163310216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea601557d0 con 0x7fea60142480 2024-04-24T08:11:42.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.720+0000 7fea5fe00640 1 -- 172.21.15.69:0/4163310216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea60075d50 con 0x7fea60142480 2024-04-24T08:11:42.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.721+0000 7fea56a00640 1 -- 172.21.15.69:0/4163310216 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea4c011d10 con 0x7fea60142480 2024-04-24T08:11:42.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.722+0000 7fea56a00640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea4006c8f0 0x7fea4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:42.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.722+0000 7fea56a00640 1 -- 172.21.15.69:0/4163310216 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fea4c08bab0 con 0x7fea60142480 2024-04-24T08:11:42.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.722+0000 7fea5ea00640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea4006c8f0 0x7fea4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:42.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.724+0000 7fea5ea00640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea4006c8f0 0x7fea4006edb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fea5000a990 tx=0x7fea50008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:42.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.726+0000 7fea56a00640 1 -- 172.21.15.69:0/4163310216 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fea4c059e40 con 0x7fea60142480 2024-04-24T08:11:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:42.962+0000 7fea5fe00640 1 -- 172.21.15.69:0/4163310216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fea6006c470 con 0x7fea60142480 2024-04-24T08:11:43.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.496+0000 7fea56a00640 1 -- 172.21.15.69:0/4163310216 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) ==== 122+0+0 (secure 0 0 0) 0x7fea4c059740 con 0x7fea60142480 2024-04-24T08:11:43.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.499+0000 7fea5fe00640 1 -- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea4006c8f0 msgr2=0x7fea4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:43.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.499+0000 7fea5fe00640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea4006c8f0 0x7fea4006edb0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fea5000a990 tx=0x7fea50008040 comp rx=0 tx=0).stop 2024-04-24T08:11:43.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.499+0000 7fea5fe00640 1 -- 172.21.15.69:0/4163310216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea60142480 msgr2=0x7fea601f4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:43.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.499+0000 7fea5fe00640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea60142480 0x7fea601f4d60 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fea4c00c430 tx=0x7fea4c00c8f0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.500+0000 7fea5fe00640 1 -- 172.21.15.69:0/4163310216 shutdown_connections 2024-04-24T08:11:43.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.500+0000 7fea5fe00640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea4006c8f0 0x7fea4006edb0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.500+0000 7fea5fe00640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea60142480 0x7fea601f4d60 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.500+0000 7fea5fe00640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea601606e0 0x7fea6015b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.500+0000 7fea5fe00640 1 --2- 172.21.15.69:0/4163310216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea601420a0 0x7fea600b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.500+0000 7fea5fe00640 1 -- 172.21.15.69:0/4163310216 >> 172.21.15.69:0/4163310216 conn(0x7fea60067b80 msgr2=0x7fea60141620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:43.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.500+0000 7fea5fe00640 1 -- 172.21.15.69:0/4163310216 shutdown_connections 2024-04-24T08:11:43.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.500+0000 7fea5fe00640 1 -- 172.21.15.69:0/4163310216 wait complete. 2024-04-24T08:11:43.521 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T08:11:43.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.717+0000 7fdec8600640 1 Processor -- start 2024-04-24T08:11:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.717+0000 7fdec8600640 1 -- start start 2024-04-24T08:11:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.719+0000 7fdec8600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec00d0300 0x7fdec006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.719+0000 7fdec8600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdec0145fe0 0x7fdec0160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.719+0000 7fdec8600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdec00cf3d0 con 0x7fdec0145a90 2024-04-24T08:11:43.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.719+0000 7fdec8600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fdec00cf6b0 con 0x7fdec0145fe0 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.719+0000 7fdec8600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdec00cf990 con 0x7fdec00d0300 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.719+0000 7fdec5200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdec0145a90 0x7fdec00d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:45758/0 (socket says 172.21.15.69:45758) 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.719+0000 7fdec5200640 1 -- 172.21.15.69:0/2014089613 learned_addr learned my addr 172.21.15.69:0/2014089613 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.719+0000 7fdec6600640 1 --2- 172.21.15.69:0/2014089613 >> v2:172.21.15.69:3300/0 conn(0x7fdec0145fe0 0x7fdec0160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.720+0000 7fdec5c00640 1 --2- 172.21.15.69:0/2014089613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec00d0300 0x7fdec006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.720+0000 7fdebfe00640 1 -- 172.21.15.69:0/2014089613 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507677872 0 0) 0x7fdec00cf3d0 con 0x7fdec0145a90 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.720+0000 7fdebfe00640 1 -- 172.21.15.69:0/2014089613 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdea8003520 con 0x7fdec0145a90 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.720+0000 7fdec6600640 1 -- 172.21.15.69:0/2014089613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec00d0300 msgr2=0x7fdec006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.720+0000 7fdec6600640 1 --2- 172.21.15.69:0/2014089613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec00d0300 0x7fdec006e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.720+0000 7fdec6600640 1 -- 172.21.15.69:0/2014089613 >> v1:172.21.15.69:6789/0 conn(0x7fdec0145a90 legacy=0x7fdec00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.720+0000 7fdebfe00640 1 -- 172.21.15.69:0/2014089613 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3712198224 0 0) 0x7fdea8003520 con 0x7fdec0145a90 2024-04-24T08:11:43.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.720+0000 7fdec6600640 1 -- 172.21.15.69:0/2014089613 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec00cfc70 con 0x7fdec0145fe0 2024-04-24T08:11:43.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.721+0000 7fdec6600640 1 --2- 172.21.15.69:0/2014089613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 0x7fdec0160540 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009d90 tx=0x7fdeb802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5444cd3210e5437d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.721+0000 7fdebfe00640 1 -- 172.21.15.69:0/2014089613 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeac001670 con 0x7fdec0145fe0 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.721+0000 7fdebfe00640 1 -- 172.21.15.69:0/2014089613 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb8007540 con 0x7fdec0145fe0 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.721+0000 7fdebfe00640 1 -- 172.21.15.69:0/2014089613 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8007910 con 0x7fdec0145fe0 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.722+0000 7fdec8600640 1 -- 172.21.15.69:0/2014089613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 msgr2=0x7fdec0160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.722+0000 7fdec8600640 1 --2- 172.21.15.69:0/2014089613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 0x7fdec0160540 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009d90 tx=0x7fdeb802a5f0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.722+0000 7fdec8600640 1 -- 172.21.15.69:0/2014089613 shutdown_connections 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.722+0000 7fdec8600640 1 --2- 172.21.15.69:0/2014089613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 0x7fdec0160540 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.722+0000 7fdec8600640 1 --2- 172.21.15.69:0/2014089613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec00d0300 0x7fdec006e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.722+0000 7fdec8600640 1 -- 172.21.15.69:0/2014089613 >> 172.21.15.69:0/2014089613 conn(0x7fdec0067d90 msgr2=0x7fdec006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.722+0000 7fdec8600640 1 -- 172.21.15.69:0/2014089613 shutdown_connections 2024-04-24T08:11:43.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.722+0000 7fdec8600640 1 -- 172.21.15.69:0/2014089613 wait complete. 2024-04-24T08:11:43.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.723+0000 7fdec8600640 1 Processor -- start 2024-04-24T08:11:43.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.724+0000 7fdec8600640 1 -- start start 2024-04-24T08:11:43.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec8600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec00d0300 0x7fdec00c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:43.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec8600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec0145a90 0x7fdec00cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:43.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec8600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 0x7fdec00b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:43.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec8600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdec01458f0 con 0x7fdec0145fe0 2024-04-24T08:11:43.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec8600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdec0145770 con 0x7fdec0145a90 2024-04-24T08:11:43.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec8600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdec00cdc30 con 0x7fdec00d0300 2024-04-24T08:11:43.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec6600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 0x7fdec00b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:43.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec6600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 0x7fdec00b3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47370/0 (socket says 172.21.15.69:47370) 2024-04-24T08:11:43.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec6600640 1 -- 172.21.15.69:0/3637233469 learned_addr learned my addr 172.21.15.69:0/3637233469 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec6600640 1 -- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec00d0300 msgr2=0x7fdec00c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec6600640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec00d0300 0x7fdec00c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec6600640 1 -- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec0145a90 msgr2=0x7fdec00cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec6600640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec0145a90 0x7fdec00cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.725+0000 7fdec6600640 1 -- 172.21.15.69:0/3637233469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec00b43b0 con 0x7fdec0145fe0 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.726+0000 7fdec6600640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 0x7fdec00b3e70 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009eb0 tx=0x7fdeb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.726+0000 7fdebd600640 1 -- 172.21.15.69:0/3637233469 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8040070 con 0x7fdec0145fe0 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.726+0000 7fdebd600640 1 -- 172.21.15.69:0/3637233469 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb8031b40 con 0x7fdec0145fe0 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.726+0000 7fdec8600640 1 -- 172.21.15.69:0/3637233469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec00c7710 con 0x7fdec0145fe0 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.726+0000 7fdebd600640 1 -- 172.21.15.69:0/3637233469 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8007e90 con 0x7fdec0145fe0 2024-04-24T08:11:43.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.726+0000 7fdec8600640 1 -- 172.21.15.69:0/3637233469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec00b3670 con 0x7fdec0145fe0 2024-04-24T08:11:43.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.727+0000 7fdec8600640 1 -- 172.21.15.69:0/3637233469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdec00afcb0 con 0x7fdec0145fe0 2024-04-24T08:11:43.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.728+0000 7fdebd600640 1 -- 172.21.15.69:0/3637233469 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdeb8039040 con 0x7fdec0145fe0 2024-04-24T08:11:43.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.729+0000 7fdebd600640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdea006c8f0 0x7fdea006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:43.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.729+0000 7fdebd600640 1 -- 172.21.15.69:0/3637233469 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fdeb80acf10 con 0x7fdec0145fe0 2024-04-24T08:11:43.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.729+0000 7fdec5c00640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdea006c8f0 0x7fdea006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:43.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.731+0000 7fdec5c00640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdea006c8f0 0x7fdea006edb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4006a90 tx=0x7fdeb4006a20 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:43.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.733+0000 7fdebd600640 1 -- 172.21.15.69:0/3637233469 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdec00afcb0 con 0x7fdec0145fe0 2024-04-24T08:11:43.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.967+0000 7fdec8600640 1 -- 172.21.15.69:0/3637233469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdec0068a80 con 0x7fdec0145fe0 2024-04-24T08:11:43.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.968+0000 7fdebd600640 1 -- 172.21.15.69:0/3637233469 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+5187 (secure 0 0 0) 0x7fdeb804b340 con 0x7fdec0145fe0 2024-04-24T08:11:43.968 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:43.969 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4034001365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4034001365},{"type":"v1","addr":"172.21.15.121:6835","nonce":4034001365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3907163219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3907163219},{"type":"v1","addr":"172.21.15.121:6837","nonce":3907163219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2756395689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2756395689},{"type":"v1","addr":"172.21.15.69:6834","nonce":2756395689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:11:39.824842+0000","modified":"2024-04-24T08:11:42.963374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/2288057943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":2288057943},{"type":"v1","addr":"172.21.15.69:6835","nonce":2288057943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6231,"qdb_cluster":[6231]},"id":4}]} 2024-04-24T08:11:43.969 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 33 2024-04-24T08:11:43.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 -- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdea006c8f0 msgr2=0x7fdea006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:43.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdea006c8f0 0x7fdea006edb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4006a90 tx=0x7fdeb4006a20 comp rx=0 tx=0).stop 2024-04-24T08:11:43.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 -- 172.21.15.69:0/3637233469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 msgr2=0x7fdec00b3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:43.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 0x7fdec00b3e70 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009eb0 tx=0x7fdeb8004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:43.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 -- 172.21.15.69:0/3637233469 shutdown_connections 2024-04-24T08:11:43.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdea006c8f0 0x7fdea006edb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdec0145fe0 0x7fdec00b3e70 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdec0145a90 0x7fdec00cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 --2- 172.21.15.69:0/3637233469 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdec00d0300 0x7fdec00c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:43.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 -- 172.21.15.69:0/3637233469 >> 172.21.15.69:0/3637233469 conn(0x7fdec0067d90 msgr2=0x7fdec006e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:43.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 -- 172.21.15.69:0/3637233469 shutdown_connections 2024-04-24T08:11:43.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:43.970+0000 7fdec8600640 1 -- 172.21.15.69:0/3637233469 wait complete. 2024-04-24T08:11:43.988 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:11:44.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.185+0000 7f64fbe00640 1 Processor -- start 2024-04-24T08:11:44.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.185+0000 7f64fbe00640 1 -- start start 2024-04-24T08:11:44.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.187+0000 7f64fbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f64f4145a90 0x7f64f406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.187+0000 7f64fbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f4145fe0 0x7f64f4160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.187+0000 7f64fbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64f40cf3d0 con 0x7f64f40d0300 2024-04-24T08:11:44.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.187+0000 7f64fbe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f64f40cf6b0 con 0x7f64f4145a90 2024-04-24T08:11:44.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.187+0000 7f64fbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64f40cf990 con 0x7f64f4145fe0 2024-04-24T08:11:44.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.187+0000 7f64f9400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f64f40d0300 0x7f64f40d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:45764/0 (socket says 172.21.15.69:45764) 2024-04-24T08:11:44.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.187+0000 7f64f9400640 1 -- 172.21.15.69:0/2321152334 learned_addr learned my addr 172.21.15.69:0/2321152334 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:44.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.187+0000 7f64f8a00640 1 --2- 172.21.15.69:0/2321152334 >> v2:172.21.15.69:3300/0 conn(0x7f64f4145a90 0x7f64f406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.187+0000 7f64f9e00640 1 --2- 172.21.15.69:0/2321152334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f4145fe0 0x7f64f4160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.188+0000 7f64f3e00640 1 -- 172.21.15.69:0/2321152334 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3356144239 0 0) 0x7f64f40cf3d0 con 0x7f64f40d0300 2024-04-24T08:11:44.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.188+0000 7f64f3e00640 1 -- 172.21.15.69:0/2321152334 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64dc003520 con 0x7f64f40d0300 2024-04-24T08:11:44.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.188+0000 7f64f8a00640 1 -- 172.21.15.69:0/2321152334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f4145fe0 msgr2=0x7f64f4160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:44.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.188+0000 7f64f8a00640 1 --2- 172.21.15.69:0/2321152334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f4145fe0 0x7f64f4160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.188+0000 7f64f8a00640 1 -- 172.21.15.69:0/2321152334 >> v1:172.21.15.69:6789/0 conn(0x7f64f40d0300 legacy=0x7f64f40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.188+0000 7f64f8a00640 1 -- 172.21.15.69:0/2321152334 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f40cfc70 con 0x7f64f4145a90 2024-04-24T08:11:44.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.189+0000 7f64f8a00640 1 --2- 172.21.15.69:0/2321152334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 0x7f64f406e240 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f64e00097a0 tx=0x7f64e002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2cf5b588bb219cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:44.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.189+0000 7f64f3e00640 1 -- 172.21.15.69:0/2321152334 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0037070 con 0x7f64f4145a90 2024-04-24T08:11:44.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.189+0000 7f64f3e00640 1 -- 172.21.15.69:0/2321152334 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e0032440 con 0x7f64f4145a90 2024-04-24T08:11:44.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.189+0000 7f64f3e00640 1 -- 172.21.15.69:0/2321152334 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0032810 con 0x7f64f4145a90 2024-04-24T08:11:44.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.190+0000 7f64fbe00640 1 -- 172.21.15.69:0/2321152334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 msgr2=0x7f64f406e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.190+0000 7f64fbe00640 1 --2- 172.21.15.69:0/2321152334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 0x7f64f406e240 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f64e00097a0 tx=0x7f64e002a180 comp rx=0 tx=0).stop 2024-04-24T08:11:44.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.190+0000 7f64fbe00640 1 -- 172.21.15.69:0/2321152334 shutdown_connections 2024-04-24T08:11:44.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.190+0000 7f64fbe00640 1 --2- 172.21.15.69:0/2321152334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f4145fe0 0x7f64f4160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.190+0000 7f64fbe00640 1 --2- 172.21.15.69:0/2321152334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 0x7f64f406e240 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.190+0000 7f64fbe00640 1 -- 172.21.15.69:0/2321152334 >> 172.21.15.69:0/2321152334 conn(0x7f64f4067d90 msgr2=0x7f64f406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:44.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.190+0000 7f64fbe00640 1 -- 172.21.15.69:0/2321152334 shutdown_connections 2024-04-24T08:11:44.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.190+0000 7f64fbe00640 1 -- 172.21.15.69:0/2321152334 wait complete. 2024-04-24T08:11:44.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.192+0000 7f64fbe00640 1 Processor -- start 2024-04-24T08:11:44.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.192+0000 7f64fbe00640 1 -- start start 2024-04-24T08:11:44.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.193+0000 7f64fbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f40d0300 0x7f64f41f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.193+0000 7f64fbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 0x7f64f41f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.193+0000 7f64fbe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64f4145fe0 0x7f64f4071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.193+0000 7f64fbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f64f41458f0 con 0x7f64f4145a90 2024-04-24T08:11:44.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.193+0000 7f64fbe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f64f4145770 con 0x7f64f4145fe0 2024-04-24T08:11:44.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.193+0000 7f64fbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64f40cdc30 con 0x7f64f40d0300 2024-04-24T08:11:44.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.193+0000 7f64f8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 0x7f64f41f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.193+0000 7f64f8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 0x7f64f41f19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47392/0 (socket says 172.21.15.69:47392) 2024-04-24T08:11:44.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.193+0000 7f64f8a00640 1 -- 172.21.15.69:0/1470194657 learned_addr learned my addr 172.21.15.69:0/1470194657 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:44.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.193+0000 7f64f9400640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f40d0300 0x7f64f41f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64f8a00640 1 -- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f40d0300 msgr2=0x7f64f41f6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64f8a00640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f40d0300 0x7f64f41f6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64f8a00640 1 -- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64f4145fe0 msgr2=0x7f64f4071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64f8a00640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64f4145fe0 0x7f64f4071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64f8a00640 1 -- 172.21.15.69:0/1470194657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64f41f6780 con 0x7f64f4145a90 2024-04-24T08:11:44.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64f9400640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f40d0300 0x7f64f41f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:44.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64f8a00640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 0x7f64f41f19e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f64e0035040 tx=0x7f64e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:44.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64f1600640 1 -- 172.21.15.69:0/1470194657 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0037040 con 0x7f64f4145a90 2024-04-24T08:11:44.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64f1600640 1 -- 172.21.15.69:0/1470194657 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64e0030500 con 0x7f64f4145a90 2024-04-24T08:11:44.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64f1600640 1 -- 172.21.15.69:0/1470194657 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64e0030910 con 0x7f64f4145a90 2024-04-24T08:11:44.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64fbe00640 1 -- 172.21.15.69:0/1470194657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64f40c6010 con 0x7f64f4145a90 2024-04-24T08:11:44.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.194+0000 7f64fbe00640 1 -- 172.21.15.69:0/1470194657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64f40ad4f0 con 0x7f64f4145a90 2024-04-24T08:11:44.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.196+0000 7f64fbe00640 1 -- 172.21.15.69:0/1470194657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64f415cce0 con 0x7f64f4145a90 2024-04-24T08:11:44.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.197+0000 7f64f1600640 1 -- 172.21.15.69:0/1470194657 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f64e0044020 con 0x7f64f4145a90 2024-04-24T08:11:44.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.197+0000 7f64f1600640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f64d406c8f0 0x7f64d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.198+0000 7f64f1600640 1 -- 172.21.15.69:0/1470194657 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f64e00abd40 con 0x7f64f4145a90 2024-04-24T08:11:44.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.198+0000 7f64f9400640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f64d406c8f0 0x7f64d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.199+0000 7f64f9400640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f64d406c8f0 0x7f64d406edb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f64e8005880 tx=0x7f64e8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:44.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.202+0000 7f64f1600640 1 -- 172.21.15.69:0/1470194657 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f64f415cce0 con 0x7f64f4145a90 2024-04-24T08:11:44.256 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:11:44.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:11:44.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.408+0000 7f64fbe00640 1 -- 172.21.15.69:0/1470194657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f64f4068a80 con 0x7f64f4145a90 2024-04-24T08:11:44.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.410+0000 7f64f1600640 1 -- 172.21.15.69:0/1470194657 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16641 (secure 0 0 0) 0x7f64f41f6780 con 0x7f64f4145a90 2024-04-24T08:11:44.411 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:44.411 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":57,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:41.349322+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:11:37.517598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-24T08:11:38.791285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:11:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 -- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f64d406c8f0 msgr2=0x7f64d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f64d406c8f0 0x7f64d406edb0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f64e8005880 tx=0x7f64e8009040 comp rx=0 tx=0).stop 2024-04-24T08:11:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 -- 172.21.15.69:0/1470194657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 msgr2=0x7f64f41f19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 0x7f64f41f19e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f64e0035040 tx=0x7f64e00046c0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 -- 172.21.15.69:0/1470194657 shutdown_connections 2024-04-24T08:11:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f64d406c8f0 0x7f64d406edb0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64f4145a90 0x7f64f41f19e0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64f4145fe0 0x7f64f4071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 --2- 172.21.15.69:0/1470194657 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64f40d0300 0x7f64f41f6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 -- 172.21.15.69:0/1470194657 >> 172.21.15.69:0/1470194657 conn(0x7f64f4067d90 msgr2=0x7f64f406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:44.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 -- 172.21.15.69:0/1470194657 shutdown_connections 2024-04-24T08:11:44.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.413+0000 7f64fbe00640 1 -- 172.21.15.69:0/1470194657 wait complete. 2024-04-24T08:11:44.431 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T08:11:44.431 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:11:44.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.621+0000 7fd805400640 1 Processor -- start 2024-04-24T08:11:44.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.621+0000 7fd805400640 1 -- start start 2024-04-24T08:11:44.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.623+0000 7fd805400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd800156c60 0x7fd80015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.623+0000 7fd805400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd8000ad7e0 0x7fd80015d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.623+0000 7fd805400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd80015d660 con 0x7fd8000b30f0 2024-04-24T08:11:44.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.623+0000 7fd805400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd80006dc30 con 0x7fd8000ad7e0 2024-04-24T08:11:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.623+0000 7fd805400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd80006df10 con 0x7fd800156c60 2024-04-24T08:11:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.623+0000 7fd804a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd8000ad7e0 0x7fd80015d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.623+0000 7fd804a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd8000ad7e0 0x7fd80015d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47396/0 (socket says 172.21.15.69:47396) 2024-04-24T08:11:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.623+0000 7fd804a00640 1 -- 172.21.15.69:0/1474576428 learned_addr learned my addr 172.21.15.69:0/1474576428 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.624+0000 7fd7ffe00640 1 --2- 172.21.15.69:0/1474576428 >> v2:172.21.15.121:3300/0 conn(0x7fd800156c60 0x7fd80015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.624+0000 7fd804a00640 1 -- 172.21.15.69:0/1474576428 >> v2:172.21.15.121:3300/0 conn(0x7fd800156c60 msgr2=0x7fd80015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.624+0000 7fd804a00640 1 --2- 172.21.15.69:0/1474576428 >> v2:172.21.15.121:3300/0 conn(0x7fd800156c60 0x7fd80015cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.624+0000 7fd804a00640 1 -- 172.21.15.69:0/1474576428 >> v1:172.21.15.121:6789/0 conn(0x7fd8000b30f0 legacy=0x7fd8000ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.624+0000 7fd804a00640 1 -- 172.21.15.69:0/1474576428 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80006e1f0 con 0x7fd8000ad7e0 2024-04-24T08:11:44.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.624+0000 7fd7ffe00640 1 --2- 172.21.15.69:0/1474576428 >> v2:172.21.15.121:3300/0 conn(0x7fd800156c60 0x7fd80015cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:44.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.625+0000 7fd804a00640 1 --2- 172.21.15.69:0/1474576428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 0x7fd80015d000 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd8000b5330 tx=0x7fd7f802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569875420c140a80 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.625+0000 7fd7fea00640 1 -- 172.21.15.69:0/1474576428 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f8036070 con 0x7fd8000ad7e0 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.625+0000 7fd7fea00640 1 -- 172.21.15.69:0/1474576428 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7f8031600 con 0x7fd8000ad7e0 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.625+0000 7fd7fea00640 1 -- 172.21.15.69:0/1474576428 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f80319d0 con 0x7fd8000ad7e0 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.626+0000 7fd805400640 1 -- 172.21.15.69:0/1474576428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 msgr2=0x7fd80015d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.626+0000 7fd805400640 1 --2- 172.21.15.69:0/1474576428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 0x7fd80015d000 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fd8000b5330 tx=0x7fd7f802a6f0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.626+0000 7fd805400640 1 -- 172.21.15.69:0/1474576428 shutdown_connections 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.626+0000 7fd805400640 1 --2- 172.21.15.69:0/1474576428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 0x7fd80015d000 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.626+0000 7fd805400640 1 --2- 172.21.15.69:0/1474576428 >> v2:172.21.15.121:3300/0 conn(0x7fd800156c60 0x7fd80015cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.626+0000 7fd805400640 1 -- 172.21.15.69:0/1474576428 >> 172.21.15.69:0/1474576428 conn(0x7fd8000d0160 msgr2=0x7fd8000d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.626+0000 7fd805400640 1 -- 172.21.15.69:0/1474576428 shutdown_connections 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.626+0000 7fd805400640 1 -- 172.21.15.69:0/1474576428 wait complete. 2024-04-24T08:11:44.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.628+0000 7fd805400640 1 Processor -- start 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.628+0000 7fd805400640 1 -- start start 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.629+0000 7fd805400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 0x7fd8000998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.629+0000 7fd805400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8000b30f0 0x7fd8000cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.629+0000 7fd805400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800156c60 0x7fd8000ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.629+0000 7fd805400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd80006e710 con 0x7fd8000ad7e0 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.629+0000 7fd805400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd80015d940 con 0x7fd800156c60 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.629+0000 7fd805400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd80006e890 con 0x7fd8000b30f0 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.629+0000 7fd7ffe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 0x7fd8000998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.629+0000 7fd7ffe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 0x7fd8000998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47412/0 (socket says 172.21.15.69:47412) 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.629+0000 7fd7ffe00640 1 -- 172.21.15.69:0/3631188618 learned_addr learned my addr 172.21.15.69:0/3631188618 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:44.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7ff400640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8000b30f0 0x7fd8000cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7ffe00640 1 -- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8000b30f0 msgr2=0x7fd8000cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7ffe00640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8000b30f0 0x7fd8000cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7ffe00640 1 -- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800156c60 msgr2=0x7fd8000ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd804a00640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800156c60 0x7fd8000ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7ffe00640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800156c60 0x7fd8000ce400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7ffe00640 1 -- 172.21.15.69:0/3631188618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8000ced20 con 0x7fd8000ad7e0 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7ff400640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8000b30f0 0x7fd8000cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7ffe00640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 0x7fd8000998b0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000c7f0 tx=0x7fd7f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7f7e00640 1 -- 172.21.15.69:0/3631188618 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0015070 con 0x7fd8000ad7e0 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7f7e00640 1 -- 172.21.15.69:0/3631188618 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7f0007820 con 0x7fd8000ad7e0 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.630+0000 7fd7f7e00640 1 -- 172.21.15.69:0/3631188618 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0007bd0 con 0x7fd8000ad7e0 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.631+0000 7fd805400640 1 -- 172.21.15.69:0/3631188618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8000cd0f0 con 0x7fd8000ad7e0 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.631+0000 7fd805400640 1 -- 172.21.15.69:0/3631188618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8000cd6b0 con 0x7fd8000ad7e0 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.632+0000 7fd805400640 1 -- 172.21.15.69:0/3631188618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd800075d50 con 0x7fd8000ad7e0 2024-04-24T08:11:44.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.633+0000 7fd7f7e00640 1 -- 172.21.15.69:0/3631188618 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd7f0018070 con 0x7fd8000ad7e0 2024-04-24T08:11:44.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.634+0000 7fd7f7e00640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7e006c8f0 0x7fd7e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:44.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.634+0000 7fd7f7e00640 1 -- 172.21.15.69:0/3631188618 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fd7f008bd00 con 0x7fd8000ad7e0 2024-04-24T08:11:44.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.634+0000 7fd7ff400640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7e006c8f0 0x7fd7e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:44.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.636+0000 7fd7ff400640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7e006c8f0 0x7fd7e006edb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd8001f3cc0 tx=0x7fd7ec009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:44.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.638+0000 7fd7f7e00640 1 -- 172.21.15.69:0/3631188618 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd8000ced20 con 0x7fd8000ad7e0 2024-04-24T08:11:44.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.838+0000 7fd805400640 1 -- 172.21.15.69:0/3631188618 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd80006f9e0 con 0x7fd7e006c8f0 2024-04-24T08:11:44.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.858+0000 7fd7f7e00640 1 -- 172.21.15.69:0/3631188618 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251954 (secure 0 0 0) 0x7fd7ec055850 con 0x7fd7e006c8f0 2024-04-24T08:11:44.862 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:44.864 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:11:44.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 -- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7e006c8f0 msgr2=0x7fd7e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7e006c8f0 0x7fd7e006edb0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd8001f3cc0 tx=0x7fd7ec009040 comp rx=0 tx=0).stop 2024-04-24T08:11:44.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 -- 172.21.15.69:0/3631188618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 msgr2=0x7fd8000998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:44.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 0x7fd8000998b0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000c7f0 tx=0x7fd7f000ccb0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 -- 172.21.15.69:0/3631188618 shutdown_connections 2024-04-24T08:11:44.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7e006c8f0 0x7fd7e006edb0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd800156c60 0x7fd8000ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8000b30f0 0x7fd8000cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 --2- 172.21.15.69:0/3631188618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8000ad7e0 0x7fd8000998b0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:44.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 -- 172.21.15.69:0/3631188618 >> 172.21.15.69:0/3631188618 conn(0x7fd8000d0160 msgr2=0x7fd8000b1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:44.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 -- 172.21.15.69:0/3631188618 shutdown_connections 2024-04-24T08:11:44.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:44.864+0000 7fd805400640 1 -- 172.21.15.69:0/3631188618 wait complete. 2024-04-24T08:11:44.882 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-04-24T08:11:44.735231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239400,"kb_used_data":6568,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735320,"statfs":{"total":773094113280,"available":772848967680,"internally_reserved":0,"allocated":6725632,"data_stored":3468052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001921"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.402811+0000","last_change":"2024-04-24T08:11:40.376215+0000","last_active":"2024-04-24T08:11:42.402811+0000","last_peered":"2024-04-24T08:11:42.402811+0000","last_clean":"2024-04-24T08:11:42.402811+0000","last_became_active":"2024-04-24T08:11:40.375891+0000","last_became_peered":"2024-04-24T08:11:40.375891+0000","last_unstale":"2024-04-24T08:11:42.402811+0000","last_undegraded":"2024-04-24T08:11:42.402811+0000","last_fullsized":"2024-04-24T08:11:42.402811+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:54:55.927367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399894+0000","last_change":"2024-04-24T08:11:40.385512+0000","last_active":"2024-04-24T08:11:42.399894+0000","last_peered":"2024-04-24T08:11:42.399894+0000","last_clean":"2024-04-24T08:11:42.399894+0000","last_became_active":"2024-04-24T08:11:40.384593+0000","last_became_peered":"2024-04-24T08:11:40.384593+0000","last_unstale":"2024-04-24T08:11:42.399894+0000","last_undegraded":"2024-04-24T08:11:42.399894+0000","last_fullsized":"2024-04-24T08:11:42.399894+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:20:09.861597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400150+0000","last_change":"2024-04-24T08:11:40.397826+0000","last_active":"2024-04-24T08:11:42.400150+0000","last_peered":"2024-04-24T08:11:42.400150+0000","last_clean":"2024-04-24T08:11:42.400150+0000","last_became_active":"2024-04-24T08:11:40.397298+0000","last_became_peered":"2024-04-24T08:11:40.397298+0000","last_unstale":"2024-04-24T08:11:42.400150+0000","last_undegraded":"2024-04-24T08:11:42.400150+0000","last_fullsized":"2024-04-24T08:11:42.400150+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:07:16.992916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361377+0000","last_change":"2024-04-24T08:11:40.377612+0000","last_active":"2024-04-24T08:11:41.361377+0000","last_peered":"2024-04-24T08:11:41.361377+0000","last_clean":"2024-04-24T08:11:41.361377+0000","last_became_active":"2024-04-24T08:11:40.377348+0000","last_became_peered":"2024-04-24T08:11:40.377348+0000","last_unstale":"2024-04-24T08:11:41.361377+0000","last_undegraded":"2024-04-24T08:11:41.361377+0000","last_fullsized":"2024-04-24T08:11:41.361377+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:07.648391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401356+0000","last_change":"2024-04-24T08:11:40.384705+0000","last_active":"2024-04-24T08:11:42.401356+0000","last_peered":"2024-04-24T08:11:42.401356+0000","last_clean":"2024-04-24T08:11:42.401356+0000","last_became_active":"2024-04-24T08:11:40.384299+0000","last_became_peered":"2024-04-24T08:11:40.384299+0000","last_unstale":"2024-04-24T08:11:42.401356+0000","last_undegraded":"2024-04-24T08:11:42.401356+0000","last_fullsized":"2024-04-24T08:11:42.401356+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:00:00.237349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369087+0000","last_change":"2024-04-24T08:11:40.364143+0000","last_active":"2024-04-24T08:11:41.369087+0000","last_peered":"2024-04-24T08:11:41.369087+0000","last_clean":"2024-04-24T08:11:41.369087+0000","last_became_active":"2024-04-24T08:11:40.363810+0000","last_became_peered":"2024-04-24T08:11:40.363810+0000","last_unstale":"2024-04-24T08:11:41.369087+0000","last_undegraded":"2024-04-24T08:11:41.369087+0000","last_fullsized":"2024-04-24T08:11:41.369087+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:38:05.022070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.360844+0000","last_change":"2024-04-24T08:11:40.383207+0000","last_active":"2024-04-24T08:11:41.360844+0000","last_peered":"2024-04-24T08:11:41.360844+0000","last_clean":"2024-04-24T08:11:41.360844+0000","last_became_active":"2024-04-24T08:11:40.382145+0000","last_became_peered":"2024-04-24T08:11:40.382145+0000","last_unstale":"2024-04-24T08:11:41.360844+0000","last_undegraded":"2024-04-24T08:11:41.360844+0000","last_fullsized":"2024-04-24T08:11:41.360844+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:38:25.942597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399610+0000","last_change":"2024-04-24T08:11:40.383203+0000","last_active":"2024-04-24T08:11:42.399610+0000","last_peered":"2024-04-24T08:11:42.399610+0000","last_clean":"2024-04-24T08:11:42.399610+0000","last_became_active":"2024-04-24T08:11:40.382913+0000","last_became_peered":"2024-04-24T08:11:40.382913+0000","last_unstale":"2024-04-24T08:11:42.399610+0000","last_undegraded":"2024-04-24T08:11:42.399610+0000","last_fullsized":"2024-04-24T08:11:42.399610+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:14:04.421220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361346+0000","last_change":"2024-04-24T08:11:40.376103+0000","last_active":"2024-04-24T08:11:41.361346+0000","last_peered":"2024-04-24T08:11:41.361346+0000","last_clean":"2024-04-24T08:11:41.361346+0000","last_became_active":"2024-04-24T08:11:40.375660+0000","last_became_peered":"2024-04-24T08:11:40.375660+0000","last_unstale":"2024-04-24T08:11:41.361346+0000","last_undegraded":"2024-04-24T08:11:41.361346+0000","last_fullsized":"2024-04-24T08:11:41.361346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:08:41.295372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.357855+0000","last_change":"2024-04-24T08:11:40.386603+0000","last_active":"2024-04-24T08:11:41.357855+0000","last_peered":"2024-04-24T08:11:41.357855+0000","last_clean":"2024-04-24T08:11:41.357855+0000","last_became_active":"2024-04-24T08:11:40.386291+0000","last_became_peered":"2024-04-24T08:11:40.386291+0000","last_unstale":"2024-04-24T08:11:41.357855+0000","last_undegraded":"2024-04-24T08:11:41.357855+0000","last_fullsized":"2024-04-24T08:11:41.357855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:46.551188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.357531+0000","last_change":"2024-04-24T08:11:40.380790+0000","last_active":"2024-04-24T08:11:41.357531+0000","last_peered":"2024-04-24T08:11:41.357531+0000","last_clean":"2024-04-24T08:11:41.357531+0000","last_became_active":"2024-04-24T08:11:40.380179+0000","last_became_peered":"2024-04-24T08:11:40.380179+0000","last_unstale":"2024-04-24T08:11:41.357531+0000","last_undegraded":"2024-04-24T08:11:41.357531+0000","last_fullsized":"2024-04-24T08:11:41.357531+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:33.253720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362307+0000","last_change":"2024-04-24T08:11:40.387428+0000","last_active":"2024-04-24T08:11:41.362307+0000","last_peered":"2024-04-24T08:11:41.362307+0000","last_clean":"2024-04-24T08:11:41.362307+0000","last_became_active":"2024-04-24T08:11:40.386722+0000","last_became_peered":"2024-04-24T08:11:40.386722+0000","last_unstale":"2024-04-24T08:11:41.362307+0000","last_undegraded":"2024-04-24T08:11:41.362307+0000","last_fullsized":"2024-04-24T08:11:41.362307+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:37:26.171195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376957+0000","last_change":"2024-04-24T08:11:40.398597+0000","last_active":"2024-04-24T08:11:41.376957+0000","last_peered":"2024-04-24T08:11:41.376957+0000","last_clean":"2024-04-24T08:11:41.376957+0000","last_became_active":"2024-04-24T08:11:40.398313+0000","last_became_peered":"2024-04-24T08:11:40.398313+0000","last_unstale":"2024-04-24T08:11:41.376957+0000","last_undegraded":"2024-04-24T08:11:41.376957+0000","last_fullsized":"2024-04-24T08:11:41.376957+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:53:39.597009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361390+0000","last_change":"2024-04-24T08:11:40.389192+0000","last_active":"2024-04-24T08:11:41.361390+0000","last_peered":"2024-04-24T08:11:41.361390+0000","last_clean":"2024-04-24T08:11:41.361390+0000","last_became_active":"2024-04-24T08:11:40.388802+0000","last_became_peered":"2024-04-24T08:11:40.388802+0000","last_unstale":"2024-04-24T08:11:41.361390+0000","last_undegraded":"2024-04-24T08:11:41.361390+0000","last_fullsized":"2024-04-24T08:11:41.361390+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:21:44.729682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361108+0000","last_change":"2024-04-24T08:11:40.378517+0000","last_active":"2024-04-24T08:11:41.361108+0000","last_peered":"2024-04-24T08:11:41.361108+0000","last_clean":"2024-04-24T08:11:41.361108+0000","last_became_active":"2024-04-24T08:11:40.378235+0000","last_became_peered":"2024-04-24T08:11:40.378235+0000","last_unstale":"2024-04-24T08:11:41.361108+0000","last_undegraded":"2024-04-24T08:11:41.361108+0000","last_fullsized":"2024-04-24T08:11:41.361108+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:21:25.512224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369192+0000","last_change":"2024-04-24T08:11:40.390368+0000","last_active":"2024-04-24T08:11:41.369192+0000","last_peered":"2024-04-24T08:11:41.369192+0000","last_clean":"2024-04-24T08:11:41.369192+0000","last_became_active":"2024-04-24T08:11:40.390091+0000","last_became_peered":"2024-04-24T08:11:40.390091+0000","last_unstale":"2024-04-24T08:11:41.369192+0000","last_undegraded":"2024-04-24T08:11:41.369192+0000","last_fullsized":"2024-04-24T08:11:41.369192+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:02.097340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401982+0000","last_change":"2024-04-24T08:11:40.395344+0000","last_active":"2024-04-24T08:11:42.401982+0000","last_peered":"2024-04-24T08:11:42.401982+0000","last_clean":"2024-04-24T08:11:42.401982+0000","last_became_active":"2024-04-24T08:11:40.395047+0000","last_became_peered":"2024-04-24T08:11:40.395047+0000","last_unstale":"2024-04-24T08:11:42.401982+0000","last_undegraded":"2024-04-24T08:11:42.401982+0000","last_fullsized":"2024-04-24T08:11:42.401982+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:35:59.110144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399980+0000","last_change":"2024-04-24T08:11:40.401229+0000","last_active":"2024-04-24T08:11:42.399980+0000","last_peered":"2024-04-24T08:11:42.399980+0000","last_clean":"2024-04-24T08:11:42.399980+0000","last_became_active":"2024-04-24T08:11:40.400954+0000","last_became_peered":"2024-04-24T08:11:40.400954+0000","last_unstale":"2024-04-24T08:11:42.399980+0000","last_undegraded":"2024-04-24T08:11:42.399980+0000","last_fullsized":"2024-04-24T08:11:42.399980+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:07.648391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401946+0000","last_change":"2024-04-24T08:11:40.376761+0000","last_active":"2024-04-24T08:11:42.401946+0000","last_peered":"2024-04-24T08:11:42.401946+0000","last_clean":"2024-04-24T08:11:42.401946+0000","last_became_active":"2024-04-24T08:11:40.375938+0000","last_became_peered":"2024-04-24T08:11:40.375938+0000","last_unstale":"2024-04-24T08:11:42.401946+0000","last_undegraded":"2024-04-24T08:11:42.401946+0000","last_fullsized":"2024-04-24T08:11:42.401946+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:05:12.711112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400397+0000","last_change":"2024-04-24T08:11:40.370776+0000","last_active":"2024-04-24T08:11:42.400397+0000","last_peered":"2024-04-24T08:11:42.400397+0000","last_clean":"2024-04-24T08:11:42.400397+0000","last_became_active":"2024-04-24T08:11:40.369266+0000","last_became_peered":"2024-04-24T08:11:40.369266+0000","last_unstale":"2024-04-24T08:11:42.400397+0000","last_undegraded":"2024-04-24T08:11:42.400397+0000","last_fullsized":"2024-04-24T08:11:42.400397+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:21.582641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400625+0000","last_change":"2024-04-24T08:11:40.396840+0000","last_active":"2024-04-24T08:11:42.400625+0000","last_peered":"2024-04-24T08:11:42.400625+0000","last_clean":"2024-04-24T08:11:42.400625+0000","last_became_active":"2024-04-24T08:11:40.395773+0000","last_became_peered":"2024-04-24T08:11:40.395773+0000","last_unstale":"2024-04-24T08:11:42.400625+0000","last_undegraded":"2024-04-24T08:11:42.400625+0000","last_fullsized":"2024-04-24T08:11:42.400625+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:26.845618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399674+0000","last_change":"2024-04-24T08:11:40.384306+0000","last_active":"2024-04-24T08:11:42.399674+0000","last_peered":"2024-04-24T08:11:42.399674+0000","last_clean":"2024-04-24T08:11:42.399674+0000","last_became_active":"2024-04-24T08:11:40.383983+0000","last_became_peered":"2024-04-24T08:11:40.383983+0000","last_unstale":"2024-04-24T08:11:42.399674+0000","last_undegraded":"2024-04-24T08:11:42.399674+0000","last_fullsized":"2024-04-24T08:11:42.399674+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:42:19.803055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400935+0000","last_change":"2024-04-24T08:11:40.396847+0000","last_active":"2024-04-24T08:11:42.400935+0000","last_peered":"2024-04-24T08:11:42.400935+0000","last_clean":"2024-04-24T08:11:42.400935+0000","last_became_active":"2024-04-24T08:11:40.396550+0000","last_became_peered":"2024-04-24T08:11:40.396550+0000","last_unstale":"2024-04-24T08:11:42.400935+0000","last_undegraded":"2024-04-24T08:11:42.400935+0000","last_fullsized":"2024-04-24T08:11:42.400935+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:03:27.646675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376355+0000","last_change":"2024-04-24T08:11:40.398853+0000","last_active":"2024-04-24T08:11:41.376355+0000","last_peered":"2024-04-24T08:11:41.376355+0000","last_clean":"2024-04-24T08:11:41.376355+0000","last_became_active":"2024-04-24T08:11:40.398590+0000","last_became_peered":"2024-04-24T08:11:40.398590+0000","last_unstale":"2024-04-24T08:11:41.376355+0000","last_undegraded":"2024-04-24T08:11:41.376355+0000","last_fullsized":"2024-04-24T08:11:41.376355+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:15:06.291851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.402093+0000","last_change":"2024-04-24T08:11:40.383822+0000","last_active":"2024-04-24T08:11:42.402093+0000","last_peered":"2024-04-24T08:11:42.402093+0000","last_clean":"2024-04-24T08:11:42.402093+0000","last_became_active":"2024-04-24T08:11:40.382831+0000","last_became_peered":"2024-04-24T08:11:40.382831+0000","last_unstale":"2024-04-24T08:11:42.402093+0000","last_undegraded":"2024-04-24T08:11:42.402093+0000","last_fullsized":"2024-04-24T08:11:42.402093+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:40.021516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362551+0000","last_change":"2024-04-24T08:11:40.382274+0000","last_active":"2024-04-24T08:11:41.362551+0000","last_peered":"2024-04-24T08:11:41.362551+0000","last_clean":"2024-04-24T08:11:41.362551+0000","last_became_active":"2024-04-24T08:11:40.381918+0000","last_became_peered":"2024-04-24T08:11:40.381918+0000","last_unstale":"2024-04-24T08:11:41.362551+0000","last_undegraded":"2024-04-24T08:11:41.362551+0000","last_fullsized":"2024-04-24T08:11:41.362551+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:07:16.992916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370203+0000","last_change":"2024-04-24T08:11:40.389836+0000","last_active":"2024-04-24T08:11:41.370203+0000","last_peered":"2024-04-24T08:11:41.370203+0000","last_clean":"2024-04-24T08:11:41.370203+0000","last_became_active":"2024-04-24T08:11:40.389477+0000","last_became_peered":"2024-04-24T08:11:40.389477+0000","last_unstale":"2024-04-24T08:11:41.370203+0000","last_undegraded":"2024-04-24T08:11:41.370203+0000","last_fullsized":"2024-04-24T08:11:41.370203+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:36.276050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401043+0000","last_change":"2024-04-24T08:11:40.382217+0000","last_active":"2024-04-24T08:11:42.401043+0000","last_peered":"2024-04-24T08:11:42.401043+0000","last_clean":"2024-04-24T08:11:42.401043+0000","last_became_active":"2024-04-24T08:11:40.381675+0000","last_became_peered":"2024-04-24T08:11:40.381675+0000","last_unstale":"2024-04-24T08:11:42.401043+0000","last_undegraded":"2024-04-24T08:11:42.401043+0000","last_fullsized":"2024-04-24T08:11:42.401043+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:11:58.549612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401034+0000","last_change":"2024-04-24T08:11:40.375810+0000","last_active":"2024-04-24T08:11:42.401034+0000","last_peered":"2024-04-24T08:11:42.401034+0000","last_clean":"2024-04-24T08:11:42.401034+0000","last_became_active":"2024-04-24T08:11:40.375509+0000","last_became_peered":"2024-04-24T08:11:40.375509+0000","last_unstale":"2024-04-24T08:11:42.401034+0000","last_undegraded":"2024-04-24T08:11:42.401034+0000","last_fullsized":"2024-04-24T08:11:42.401034+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:38:08.948214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400520+0000","last_change":"2024-04-24T08:11:40.394095+0000","last_active":"2024-04-24T08:11:42.400520+0000","last_peered":"2024-04-24T08:11:42.400520+0000","last_clean":"2024-04-24T08:11:42.400520+0000","last_became_active":"2024-04-24T08:11:40.393767+0000","last_became_peered":"2024-04-24T08:11:40.393767+0000","last_unstale":"2024-04-24T08:11:42.400520+0000","last_undegraded":"2024-04-24T08:11:42.400520+0000","last_fullsized":"2024-04-24T08:11:42.400520+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:56:58.949019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399832+0000","last_change":"2024-04-24T08:11:40.377427+0000","last_active":"2024-04-24T08:11:42.399832+0000","last_peered":"2024-04-24T08:11:42.399832+0000","last_clean":"2024-04-24T08:11:42.399832+0000","last_became_active":"2024-04-24T08:11:40.377117+0000","last_became_peered":"2024-04-24T08:11:40.377117+0000","last_unstale":"2024-04-24T08:11:42.399832+0000","last_undegraded":"2024-04-24T08:11:42.399832+0000","last_fullsized":"2024-04-24T08:11:42.399832+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:41:54.684930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369704+0000","last_change":"2024-04-24T08:11:40.368446+0000","last_active":"2024-04-24T08:11:41.369704+0000","last_peered":"2024-04-24T08:11:41.369704+0000","last_clean":"2024-04-24T08:11:41.369704+0000","last_became_active":"2024-04-24T08:11:40.368060+0000","last_became_peered":"2024-04-24T08:11:40.368060+0000","last_unstale":"2024-04-24T08:11:41.369704+0000","last_undegraded":"2024-04-24T08:11:41.369704+0000","last_fullsized":"2024-04-24T08:11:41.369704+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:05:45.485156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370115+0000","last_change":"2024-04-24T08:11:40.391905+0000","last_active":"2024-04-24T08:11:41.370115+0000","last_peered":"2024-04-24T08:11:41.370115+0000","last_clean":"2024-04-24T08:11:41.370115+0000","last_became_active":"2024-04-24T08:11:40.391600+0000","last_became_peered":"2024-04-24T08:11:40.391600+0000","last_unstale":"2024-04-24T08:11:41.370115+0000","last_undegraded":"2024-04-24T08:11:41.370115+0000","last_fullsized":"2024-04-24T08:11:41.370115+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:33:39.406702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369862+0000","last_change":"2024-04-24T08:11:39.377325+0000","last_active":"2024-04-24T08:11:41.369862+0000","last_peered":"2024-04-24T08:11:41.369862+0000","last_clean":"2024-04-24T08:11:41.369862+0000","last_became_active":"2024-04-24T08:11:39.376998+0000","last_became_peered":"2024-04-24T08:11:39.376998+0000","last_unstale":"2024-04-24T08:11:41.369862+0000","last_undegraded":"2024-04-24T08:11:41.369862+0000","last_fullsized":"2024-04-24T08:11:41.369862+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:38:24.926308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.358013+0000","last_change":"2024-04-24T08:11:40.396290+0000","last_active":"2024-04-24T08:11:41.358013+0000","last_peered":"2024-04-24T08:11:41.358013+0000","last_clean":"2024-04-24T08:11:41.358013+0000","last_became_active":"2024-04-24T08:11:40.396009+0000","last_became_peered":"2024-04-24T08:11:40.396009+0000","last_unstale":"2024-04-24T08:11:41.358013+0000","last_undegraded":"2024-04-24T08:11:41.358013+0000","last_fullsized":"2024-04-24T08:11:41.358013+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:55.221773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400050+0000","last_change":"2024-04-24T08:11:39.369869+0000","last_active":"2024-04-24T08:11:42.400050+0000","last_peered":"2024-04-24T08:11:42.400050+0000","last_clean":"2024-04-24T08:11:42.400050+0000","last_became_active":"2024-04-24T08:11:39.369558+0000","last_became_peered":"2024-04-24T08:11:39.369558+0000","last_unstale":"2024-04-24T08:11:42.400050+0000","last_undegraded":"2024-04-24T08:11:42.400050+0000","last_fullsized":"2024-04-24T08:11:42.400050+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:25.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369761+0000","last_change":"2024-04-24T08:11:40.387704+0000","last_active":"2024-04-24T08:11:41.369761+0000","last_peered":"2024-04-24T08:11:41.369761+0000","last_clean":"2024-04-24T08:11:41.369761+0000","last_became_active":"2024-04-24T08:11:40.387411+0000","last_became_peered":"2024-04-24T08:11:40.387411+0000","last_unstale":"2024-04-24T08:11:41.369761+0000","last_undegraded":"2024-04-24T08:11:41.369761+0000","last_fullsized":"2024-04-24T08:11:41.369761+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:27.911530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370350+0000","last_change":"2024-04-24T08:11:39.369452+0000","last_active":"2024-04-24T08:11:41.370350+0000","last_peered":"2024-04-24T08:11:41.370350+0000","last_clean":"2024-04-24T08:11:41.370350+0000","last_became_active":"2024-04-24T08:11:39.369166+0000","last_became_peered":"2024-04-24T08:11:39.369166+0000","last_unstale":"2024-04-24T08:11:41.370350+0000","last_undegraded":"2024-04-24T08:11:41.370350+0000","last_fullsized":"2024-04-24T08:11:41.370350+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:08:40.279084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370205+0000","last_change":"2024-04-24T08:11:40.375929+0000","last_active":"2024-04-24T08:11:41.370205+0000","last_peered":"2024-04-24T08:11:41.370205+0000","last_clean":"2024-04-24T08:11:41.370205+0000","last_became_active":"2024-04-24T08:11:40.375666+0000","last_became_peered":"2024-04-24T08:11:40.375666+0000","last_unstale":"2024-04-24T08:11:41.370205+0000","last_undegraded":"2024-04-24T08:11:41.370205+0000","last_fullsized":"2024-04-24T08:11:41.370205+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:43.801149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361550+0000","last_change":"2024-04-24T08:11:39.378633+0000","last_active":"2024-04-24T08:11:41.361550+0000","last_peered":"2024-04-24T08:11:41.361550+0000","last_clean":"2024-04-24T08:11:41.361550+0000","last_became_active":"2024-04-24T08:11:39.378214+0000","last_became_peered":"2024-04-24T08:11:39.378214+0000","last_unstale":"2024-04-24T08:11:41.361550+0000","last_undegraded":"2024-04-24T08:11:41.361550+0000","last_fullsized":"2024-04-24T08:11:41.361550+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:55:01.374287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401405+0000","last_change":"2024-04-24T08:11:40.383737+0000","last_active":"2024-04-24T08:11:42.401405+0000","last_peered":"2024-04-24T08:11:42.401405+0000","last_clean":"2024-04-24T08:11:42.401405+0000","last_became_active":"2024-04-24T08:11:40.383395+0000","last_became_peered":"2024-04-24T08:11:40.383395+0000","last_unstale":"2024-04-24T08:11:42.401405+0000","last_undegraded":"2024-04-24T08:11:42.401405+0000","last_fullsized":"2024-04-24T08:11:42.401405+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:32:11.937172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.360904+0000","last_change":"2024-04-24T08:11:39.380088+0000","last_active":"2024-04-24T08:11:41.360904+0000","last_peered":"2024-04-24T08:11:41.360904+0000","last_clean":"2024-04-24T08:11:41.360904+0000","last_became_active":"2024-04-24T08:11:39.379580+0000","last_became_peered":"2024-04-24T08:11:39.379580+0000","last_unstale":"2024-04-24T08:11:41.360904+0000","last_undegraded":"2024-04-24T08:11:41.360904+0000","last_fullsized":"2024-04-24T08:11:41.360904+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:53:38.580721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400599+0000","last_change":"2024-04-24T08:11:40.371654+0000","last_active":"2024-04-24T08:11:42.400599+0000","last_peered":"2024-04-24T08:11:42.400599+0000","last_clean":"2024-04-24T08:11:42.400599+0000","last_became_active":"2024-04-24T08:11:40.371228+0000","last_became_peered":"2024-04-24T08:11:40.371228+0000","last_unstale":"2024-04-24T08:11:42.400599+0000","last_undegraded":"2024-04-24T08:11:42.400599+0000","last_fullsized":"2024-04-24T08:11:42.400599+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:41.726594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369124+0000","last_change":"2024-04-24T08:11:39.382674+0000","last_active":"2024-04-24T08:11:41.369124+0000","last_peered":"2024-04-24T08:11:41.369124+0000","last_clean":"2024-04-24T08:11:41.369124+0000","last_became_active":"2024-04-24T08:11:39.382353+0000","last_became_peered":"2024-04-24T08:11:39.382353+0000","last_unstale":"2024-04-24T08:11:41.369124+0000","last_undegraded":"2024-04-24T08:11:41.369124+0000","last_fullsized":"2024-04-24T08:11:41.369124+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:21:24.495936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376352+0000","last_change":"2024-04-24T08:11:40.394145+0000","last_active":"2024-04-24T08:11:41.376352+0000","last_peered":"2024-04-24T08:11:41.376352+0000","last_clean":"2024-04-24T08:11:41.376352+0000","last_became_active":"2024-04-24T08:11:40.393858+0000","last_became_peered":"2024-04-24T08:11:40.393858+0000","last_unstale":"2024-04-24T08:11:41.376352+0000","last_undegraded":"2024-04-24T08:11:41.376352+0000","last_fullsized":"2024-04-24T08:11:41.376352+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:39:19.795220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.359540+0000","last_change":"2024-04-24T08:11:39.380789+0000","last_active":"2024-04-24T08:11:41.359540+0000","last_peered":"2024-04-24T08:11:41.359540+0000","last_clean":"2024-04-24T08:11:41.359540+0000","last_became_active":"2024-04-24T08:11:39.380460+0000","last_became_peered":"2024-04-24T08:11:39.380460+0000","last_unstale":"2024-04-24T08:11:41.359540+0000","last_undegraded":"2024-04-24T08:11:41.359540+0000","last_fullsized":"2024-04-24T08:11:41.359540+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:45.534899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400301+0000","last_change":"2024-04-24T08:11:40.396741+0000","last_active":"2024-04-24T08:11:42.400301+0000","last_peered":"2024-04-24T08:11:42.400301+0000","last_clean":"2024-04-24T08:11:42.400301+0000","last_became_active":"2024-04-24T08:11:40.394132+0000","last_became_peered":"2024-04-24T08:11:40.394132+0000","last_unstale":"2024-04-24T08:11:42.400301+0000","last_undegraded":"2024-04-24T08:11:42.400301+0000","last_fullsized":"2024-04-24T08:11:42.400301+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:52:25.200562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361253+0000","last_change":"2024-04-24T08:11:39.382605+0000","last_active":"2024-04-24T08:11:41.361253+0000","last_peered":"2024-04-24T08:11:41.361253+0000","last_clean":"2024-04-24T08:11:41.361253+0000","last_became_active":"2024-04-24T08:11:39.381255+0000","last_became_peered":"2024-04-24T08:11:39.381255+0000","last_unstale":"2024-04-24T08:11:41.361253+0000","last_undegraded":"2024-04-24T08:11:41.361253+0000","last_fullsized":"2024-04-24T08:11:41.361253+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:32.237431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362068+0000","last_change":"2024-04-24T08:11:40.381424+0000","last_active":"2024-04-24T08:11:41.362068+0000","last_peered":"2024-04-24T08:11:41.362068+0000","last_clean":"2024-04-24T08:11:41.362068+0000","last_became_active":"2024-04-24T08:11:40.381063+0000","last_became_peered":"2024-04-24T08:11:40.381063+0000","last_unstale":"2024-04-24T08:11:41.362068+0000","last_undegraded":"2024-04-24T08:11:41.362068+0000","last_fullsized":"2024-04-24T08:11:41.362068+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:51.053178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400441+0000","last_change":"2024-04-24T08:11:39.380932+0000","last_active":"2024-04-24T08:11:42.400441+0000","last_peered":"2024-04-24T08:11:42.400441+0000","last_clean":"2024-04-24T08:11:42.400441+0000","last_became_active":"2024-04-24T08:11:39.380608+0000","last_became_peered":"2024-04-24T08:11:39.380608+0000","last_unstale":"2024-04-24T08:11:42.400441+0000","last_undegraded":"2024-04-24T08:11:42.400441+0000","last_fullsized":"2024-04-24T08:11:42.400441+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:19:57.821537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400625+0000","last_change":"2024-04-24T08:11:40.401365+0000","last_active":"2024-04-24T08:11:42.400625+0000","last_peered":"2024-04-24T08:11:42.400625+0000","last_clean":"2024-04-24T08:11:42.400625+0000","last_became_active":"2024-04-24T08:11:40.401056+0000","last_became_peered":"2024-04-24T08:11:40.401056+0000","last_unstale":"2024-04-24T08:11:42.400625+0000","last_undegraded":"2024-04-24T08:11:42.400625+0000","last_fullsized":"2024-04-24T08:11:42.400625+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:35:08.901472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400347+0000","last_change":"2024-04-24T08:11:39.378314+0000","last_active":"2024-04-24T08:11:42.400347+0000","last_peered":"2024-04-24T08:11:42.400347+0000","last_clean":"2024-04-24T08:11:42.400347+0000","last_became_active":"2024-04-24T08:11:39.377954+0000","last_became_peered":"2024-04-24T08:11:39.377954+0000","last_unstale":"2024-04-24T08:11:42.400347+0000","last_undegraded":"2024-04-24T08:11:42.400347+0000","last_fullsized":"2024-04-24T08:11:42.400347+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:38:04.005782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.357132+0000","last_change":"2024-04-24T08:11:40.388856+0000","last_active":"2024-04-24T08:11:41.357132+0000","last_peered":"2024-04-24T08:11:41.357132+0000","last_clean":"2024-04-24T08:11:41.357132+0000","last_became_active":"2024-04-24T08:11:40.388500+0000","last_became_peered":"2024-04-24T08:11:40.388500+0000","last_unstale":"2024-04-24T08:11:41.357132+0000","last_undegraded":"2024-04-24T08:11:41.357132+0000","last_fullsized":"2024-04-24T08:11:41.357132+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:15:06.291851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401061+0000","last_change":"2024-04-24T08:11:39.379805+0000","last_active":"2024-04-24T08:11:42.401061+0000","last_peered":"2024-04-24T08:11:42.401061+0000","last_clean":"2024-04-24T08:11:42.401061+0000","last_became_active":"2024-04-24T08:11:39.379459+0000","last_became_peered":"2024-04-24T08:11:39.379459+0000","last_unstale":"2024-04-24T08:11:42.401061+0000","last_undegraded":"2024-04-24T08:11:42.401061+0000","last_fullsized":"2024-04-24T08:11:42.401061+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:24:44.359232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400923+0000","last_change":"2024-04-24T08:11:40.397553+0000","last_active":"2024-04-24T08:11:42.400923+0000","last_peered":"2024-04-24T08:11:42.400923+0000","last_clean":"2024-04-24T08:11:42.400923+0000","last_became_active":"2024-04-24T08:11:40.396793+0000","last_became_peered":"2024-04-24T08:11:40.396793+0000","last_unstale":"2024-04-24T08:11:42.400923+0000","last_undegraded":"2024-04-24T08:11:42.400923+0000","last_fullsized":"2024-04-24T08:11:42.400923+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:05:45.485156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361712+0000","last_change":"2024-04-24T08:11:39.376800+0000","last_active":"2024-04-24T08:11:41.361712+0000","last_peered":"2024-04-24T08:11:41.361712+0000","last_clean":"2024-04-24T08:11:41.361712+0000","last_became_active":"2024-04-24T08:11:39.376483+0000","last_became_peered":"2024-04-24T08:11:39.376483+0000","last_unstale":"2024-04-24T08:11:41.361712+0000","last_undegraded":"2024-04-24T08:11:41.361712+0000","last_fullsized":"2024-04-24T08:11:41.361712+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:45.534899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.359788+0000","last_change":"2024-04-24T08:11:40.379479+0000","last_active":"2024-04-24T08:11:41.359788+0000","last_peered":"2024-04-24T08:11:41.359788+0000","last_clean":"2024-04-24T08:11:41.359788+0000","last_became_active":"2024-04-24T08:11:40.379135+0000","last_became_peered":"2024-04-24T08:11:40.379135+0000","last_unstale":"2024-04-24T08:11:41.359788+0000","last_undegraded":"2024-04-24T08:11:41.359788+0000","last_fullsized":"2024-04-24T08:11:41.359788+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:14.674330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400741+0000","last_change":"2024-04-24T08:11:39.370114+0000","last_active":"2024-04-24T08:11:42.400741+0000","last_peered":"2024-04-24T08:11:42.400741+0000","last_clean":"2024-04-24T08:11:42.400741+0000","last_became_active":"2024-04-24T08:11:39.369750+0000","last_became_peered":"2024-04-24T08:11:39.369750+0000","last_unstale":"2024-04-24T08:11:42.400741+0000","last_undegraded":"2024-04-24T08:11:42.400741+0000","last_fullsized":"2024-04-24T08:11:42.400741+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:56:57.932731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400379+0000","last_change":"2024-04-24T08:11:40.395703+0000","last_active":"2024-04-24T08:11:42.400379+0000","last_peered":"2024-04-24T08:11:42.400379+0000","last_clean":"2024-04-24T08:11:42.400379+0000","last_became_active":"2024-04-24T08:11:40.395427+0000","last_became_peered":"2024-04-24T08:11:40.395427+0000","last_unstale":"2024-04-24T08:11:42.400379+0000","last_undegraded":"2024-04-24T08:11:42.400379+0000","last_fullsized":"2024-04-24T08:11:42.400379+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:20:09.876703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376244+0000","last_change":"2024-04-24T08:11:39.382845+0000","last_active":"2024-04-24T08:11:41.376244+0000","last_peered":"2024-04-24T08:11:41.376244+0000","last_clean":"2024-04-24T08:11:41.376244+0000","last_became_active":"2024-04-24T08:11:39.381666+0000","last_became_peered":"2024-04-24T08:11:39.381666+0000","last_unstale":"2024-04-24T08:11:41.376244+0000","last_undegraded":"2024-04-24T08:11:41.376244+0000","last_fullsized":"2024-04-24T08:11:41.376244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:01:20.271949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361273+0000","last_change":"2024-04-24T08:11:40.394948+0000","last_active":"2024-04-24T08:11:41.361273+0000","last_peered":"2024-04-24T08:11:41.361273+0000","last_clean":"2024-04-24T08:11:41.361273+0000","last_became_active":"2024-04-24T08:11:40.392461+0000","last_became_peered":"2024-04-24T08:11:40.392461+0000","last_unstale":"2024-04-24T08:11:41.361273+0000","last_undegraded":"2024-04-24T08:11:41.361273+0000","last_fullsized":"2024-04-24T08:11:41.361273+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:14:30.148003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.359596+0000","last_change":"2024-04-24T08:11:39.378417+0000","last_active":"2024-04-24T08:11:41.359596+0000","last_peered":"2024-04-24T08:11:41.359596+0000","last_clean":"2024-04-24T08:11:41.359596+0000","last_became_active":"2024-04-24T08:11:39.378063+0000","last_became_peered":"2024-04-24T08:11:39.378063+0000","last_unstale":"2024-04-24T08:11:41.359596+0000","last_undegraded":"2024-04-24T08:11:41.359596+0000","last_fullsized":"2024-04-24T08:11:41.359596+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:24:44.359232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400073+0000","last_change":"2024-04-24T08:11:40.399681+0000","last_active":"2024-04-24T08:11:42.400073+0000","last_peered":"2024-04-24T08:11:42.400073+0000","last_clean":"2024-04-24T08:11:42.400073+0000","last_became_active":"2024-04-24T08:11:40.399332+0000","last_became_peered":"2024-04-24T08:11:40.399332+0000","last_unstale":"2024-04-24T08:11:42.400073+0000","last_undegraded":"2024-04-24T08:11:42.400073+0000","last_fullsized":"2024-04-24T08:11:42.400073+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:37:26.171195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400280+0000","last_change":"2024-04-24T08:11:39.384034+0000","last_active":"2024-04-24T08:11:42.400280+0000","last_peered":"2024-04-24T08:11:42.400280+0000","last_clean":"2024-04-24T08:11:42.400280+0000","last_became_active":"2024-04-24T08:11:39.383725+0000","last_became_peered":"2024-04-24T08:11:39.383725+0000","last_unstale":"2024-04-24T08:11:42.400280+0000","last_undegraded":"2024-04-24T08:11:42.400280+0000","last_fullsized":"2024-04-24T08:11:42.400280+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:54.205485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400150+0000","last_change":"2024-04-24T08:11:40.399086+0000","last_active":"2024-04-24T08:11:42.400150+0000","last_peered":"2024-04-24T08:11:42.400150+0000","last_clean":"2024-04-24T08:11:42.400150+0000","last_became_active":"2024-04-24T08:11:40.398790+0000","last_became_peered":"2024-04-24T08:11:40.398790+0000","last_unstale":"2024-04-24T08:11:42.400150+0000","last_undegraded":"2024-04-24T08:11:42.400150+0000","last_fullsized":"2024-04-24T08:11:42.400150+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:58:01.370448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401788+0000","last_change":"2024-04-24T08:11:39.372024+0000","last_active":"2024-04-24T08:11:42.401788+0000","last_peered":"2024-04-24T08:11:42.401788+0000","last_clean":"2024-04-24T08:11:42.401788+0000","last_became_active":"2024-04-24T08:11:39.371736+0000","last_became_peered":"2024-04-24T08:11:39.371736+0000","last_unstale":"2024-04-24T08:11:42.401788+0000","last_undegraded":"2024-04-24T08:11:42.401788+0000","last_fullsized":"2024-04-24T08:11:42.401788+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:20:08.860414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362779+0000","last_change":"2024-04-24T08:11:40.381743+0000","last_active":"2024-04-24T08:11:41.362779+0000","last_peered":"2024-04-24T08:11:41.362779+0000","last_clean":"2024-04-24T08:11:41.362779+0000","last_became_active":"2024-04-24T08:11:40.381437+0000","last_became_peered":"2024-04-24T08:11:40.381437+0000","last_unstale":"2024-04-24T08:11:41.362779+0000","last_undegraded":"2024-04-24T08:11:41.362779+0000","last_fullsized":"2024-04-24T08:11:41.362779+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:20:09.876703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369986+0000","last_change":"2024-04-24T08:11:39.380526+0000","last_active":"2024-04-24T08:11:41.369986+0000","last_peered":"2024-04-24T08:11:41.369986+0000","last_clean":"2024-04-24T08:11:41.369986+0000","last_became_active":"2024-04-24T08:11:39.377660+0000","last_became_peered":"2024-04-24T08:11:39.377660+0000","last_unstale":"2024-04-24T08:11:41.369986+0000","last_undegraded":"2024-04-24T08:11:41.369986+0000","last_fullsized":"2024-04-24T08:11:41.369986+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:21:43.713394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400833+0000","last_change":"2024-04-24T08:11:40.383865+0000","last_active":"2024-04-24T08:11:42.400833+0000","last_peered":"2024-04-24T08:11:42.400833+0000","last_clean":"2024-04-24T08:11:42.400833+0000","last_became_active":"2024-04-24T08:11:40.383599+0000","last_became_peered":"2024-04-24T08:11:40.383599+0000","last_unstale":"2024-04-24T08:11:42.400833+0000","last_undegraded":"2024-04-24T08:11:42.400833+0000","last_fullsized":"2024-04-24T08:11:42.400833+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:28.565615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362075+0000","last_change":"2024-04-24T08:11:39.382135+0000","last_active":"2024-04-24T08:11:41.362075+0000","last_peered":"2024-04-24T08:11:41.362075+0000","last_clean":"2024-04-24T08:11:41.362075+0000","last_became_active":"2024-04-24T08:11:39.381856+0000","last_became_peered":"2024-04-24T08:11:39.381856+0000","last_unstale":"2024-04-24T08:11:41.362075+0000","last_undegraded":"2024-04-24T08:11:41.362075+0000","last_fullsized":"2024-04-24T08:11:41.362075+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:19:57.821537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400840+0000","last_change":"2024-04-24T08:11:40.397860+0000","last_active":"2024-04-24T08:11:42.400840+0000","last_peered":"2024-04-24T08:11:42.400840+0000","last_clean":"2024-04-24T08:11:42.400840+0000","last_became_active":"2024-04-24T08:11:40.397580+0000","last_became_peered":"2024-04-24T08:11:40.397580+0000","last_unstale":"2024-04-24T08:11:42.400840+0000","last_undegraded":"2024-04-24T08:11:42.400840+0000","last_fullsized":"2024-04-24T08:11:42.400840+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:38:05.022070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369820+0000","last_change":"2024-04-24T08:11:39.368256+0000","last_active":"2024-04-24T08:11:41.369820+0000","last_peered":"2024-04-24T08:11:41.369820+0000","last_clean":"2024-04-24T08:11:41.369820+0000","last_became_active":"2024-04-24T08:11:39.367733+0000","last_became_peered":"2024-04-24T08:11:39.367733+0000","last_unstale":"2024-04-24T08:11:41.369820+0000","last_undegraded":"2024-04-24T08:11:41.369820+0000","last_fullsized":"2024-04-24T08:11:41.369820+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:14:29.131715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.360408+0000","last_change":"2024-04-24T08:11:40.376808+0000","last_active":"2024-04-24T08:11:41.360408+0000","last_peered":"2024-04-24T08:11:41.360408+0000","last_clean":"2024-04-24T08:11:41.360408+0000","last_became_active":"2024-04-24T08:11:40.376195+0000","last_became_peered":"2024-04-24T08:11:40.376195+0000","last_unstale":"2024-04-24T08:11:41.360408+0000","last_undegraded":"2024-04-24T08:11:41.360408+0000","last_fullsized":"2024-04-24T08:11:41.360408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:01:45.419044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401068+0000","last_change":"2024-04-24T08:11:39.378864+0000","last_active":"2024-04-24T08:11:42.401068+0000","last_peered":"2024-04-24T08:11:42.401068+0000","last_clean":"2024-04-24T08:11:42.401068+0000","last_became_active":"2024-04-24T08:11:39.378185+0000","last_became_peered":"2024-04-24T08:11:39.378185+0000","last_unstale":"2024-04-24T08:11:42.401068+0000","last_undegraded":"2024-04-24T08:11:42.401068+0000","last_fullsized":"2024-04-24T08:11:42.401068+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:35:07.885184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369706+0000","last_change":"2024-04-24T08:11:40.390204+0000","last_active":"2024-04-24T08:11:41.369706+0000","last_peered":"2024-04-24T08:11:41.369706+0000","last_clean":"2024-04-24T08:11:41.369706+0000","last_became_active":"2024-04-24T08:11:40.389588+0000","last_became_peered":"2024-04-24T08:11:40.389588+0000","last_unstale":"2024-04-24T08:11:41.369706+0000","last_undegraded":"2024-04-24T08:11:41.369706+0000","last_fullsized":"2024-04-24T08:11:41.369706+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:14:29.057037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.356827+0000","last_change":"2024-04-24T08:11:39.377997+0000","last_active":"2024-04-24T08:11:41.356827+0000","last_peered":"2024-04-24T08:11:41.356827+0000","last_clean":"2024-04-24T08:11:41.356827+0000","last_became_active":"2024-04-24T08:11:39.376575+0000","last_became_peered":"2024-04-24T08:11:39.376575+0000","last_unstale":"2024-04-24T08:11:41.356827+0000","last_undegraded":"2024-04-24T08:11:41.356827+0000","last_fullsized":"2024-04-24T08:11:41.356827+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:46:49.812730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369275+0000","last_change":"2024-04-24T08:11:40.386876+0000","last_active":"2024-04-24T08:11:41.369275+0000","last_peered":"2024-04-24T08:11:41.369275+0000","last_clean":"2024-04-24T08:11:41.369275+0000","last_became_active":"2024-04-24T08:11:40.386554+0000","last_became_peered":"2024-04-24T08:11:40.386554+0000","last_unstale":"2024-04-24T08:11:41.369275+0000","last_undegraded":"2024-04-24T08:11:41.369275+0000","last_fullsized":"2024-04-24T08:11:41.369275+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:03:27.646675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.359272+0000","last_change":"2024-04-24T08:11:39.375800+0000","last_active":"2024-04-24T08:11:41.359272+0000","last_peered":"2024-04-24T08:11:41.359272+0000","last_clean":"2024-04-24T08:11:41.359272+0000","last_became_active":"2024-04-24T08:11:39.374984+0000","last_became_peered":"2024-04-24T08:11:39.374984+0000","last_unstale":"2024-04-24T08:11:41.359272+0000","last_undegraded":"2024-04-24T08:11:41.359272+0000","last_fullsized":"2024-04-24T08:11:41.359272+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:54.205485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362700+0000","last_change":"2024-04-24T08:11:40.372341+0000","last_active":"2024-04-24T08:11:41.362700+0000","last_peered":"2024-04-24T08:11:41.362700+0000","last_clean":"2024-04-24T08:11:41.362700+0000","last_became_active":"2024-04-24T08:11:40.371277+0000","last_became_peered":"2024-04-24T08:11:40.371277+0000","last_unstale":"2024-04-24T08:11:41.362700+0000","last_undegraded":"2024-04-24T08:11:41.362700+0000","last_fullsized":"2024-04-24T08:11:41.362700+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:58:01.370448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.358015+0000","last_change":"2024-04-24T08:11:39.373873+0000","last_active":"2024-04-24T08:11:41.358015+0000","last_peered":"2024-04-24T08:11:41.358015+0000","last_clean":"2024-04-24T08:11:41.358015+0000","last_became_active":"2024-04-24T08:11:39.373540+0000","last_became_peered":"2024-04-24T08:11:39.373540+0000","last_unstale":"2024-04-24T08:11:41.358015+0000","last_undegraded":"2024-04-24T08:11:41.358015+0000","last_fullsized":"2024-04-24T08:11:41.358015+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:54:22.021374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.363003+0000","last_change":"2024-04-24T08:11:40.393301+0000","last_active":"2024-04-24T08:11:41.363003+0000","last_peered":"2024-04-24T08:11:41.363003+0000","last_clean":"2024-04-24T08:11:41.363003+0000","last_became_active":"2024-04-24T08:11:40.392986+0000","last_became_peered":"2024-04-24T08:11:40.392986+0000","last_unstale":"2024-04-24T08:11:41.363003+0000","last_undegraded":"2024-04-24T08:11:41.363003+0000","last_fullsized":"2024-04-24T08:11:41.363003+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:56:58.949019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"57'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370389+0000","last_change":"2024-04-24T08:11:39.382407+0000","last_active":"2024-04-24T08:11:41.370389+0000","last_peered":"2024-04-24T08:11:41.370389+0000","last_clean":"2024-04-24T08:11:41.370389+0000","last_became_active":"2024-04-24T08:11:39.381816+0000","last_became_peered":"2024-04-24T08:11:39.381816+0000","last_unstale":"2024-04-24T08:11:41.370389+0000","last_undegraded":"2024-04-24T08:11:41.370389+0000","last_fullsized":"2024-04-24T08:11:41.370389+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:50.036889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.357387+0000","last_change":"2024-04-24T08:11:40.388444+0000","last_active":"2024-04-24T08:11:41.357387+0000","last_peered":"2024-04-24T08:11:41.357387+0000","last_clean":"2024-04-24T08:11:41.357387+0000","last_became_active":"2024-04-24T08:11:40.387632+0000","last_became_peered":"2024-04-24T08:11:40.387632+0000","last_unstale":"2024-04-24T08:11:41.357387+0000","last_undegraded":"2024-04-24T08:11:41.357387+0000","last_fullsized":"2024-04-24T08:11:41.357387+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:53:39.597009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376690+0000","last_change":"2024-04-24T08:11:39.381391+0000","last_active":"2024-04-24T08:11:41.376690+0000","last_peered":"2024-04-24T08:11:41.376690+0000","last_clean":"2024-04-24T08:11:41.376690+0000","last_became_active":"2024-04-24T08:11:39.380979+0000","last_became_peered":"2024-04-24T08:11:39.380979+0000","last_unstale":"2024-04-24T08:11:41.376690+0000","last_undegraded":"2024-04-24T08:11:41.376690+0000","last_fullsized":"2024-04-24T08:11:41.376690+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:48:45.846889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370359+0000","last_change":"2024-04-24T08:11:40.375316+0000","last_active":"2024-04-24T08:11:41.370359+0000","last_peered":"2024-04-24T08:11:41.370359+0000","last_clean":"2024-04-24T08:11:41.370359+0000","last_became_active":"2024-04-24T08:11:40.374563+0000","last_became_peered":"2024-04-24T08:11:40.374563+0000","last_unstale":"2024-04-24T08:11:41.370359+0000","last_undegraded":"2024-04-24T08:11:41.370359+0000","last_fullsized":"2024-04-24T08:11:41.370359+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:46:45.450068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362453+0000","last_change":"2024-04-24T08:11:39.381316+0000","last_active":"2024-04-24T08:11:41.362453+0000","last_peered":"2024-04-24T08:11:41.362453+0000","last_clean":"2024-04-24T08:11:41.362453+0000","last_became_active":"2024-04-24T08:11:39.381027+0000","last_became_peered":"2024-04-24T08:11:39.381027+0000","last_unstale":"2024-04-24T08:11:41.362453+0000","last_undegraded":"2024-04-24T08:11:41.362453+0000","last_fullsized":"2024-04-24T08:11:41.362453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:10.676575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400119+0000","last_change":"2024-04-24T08:11:40.376857+0000","last_active":"2024-04-24T08:11:42.400119+0000","last_peered":"2024-04-24T08:11:42.400119+0000","last_clean":"2024-04-24T08:11:42.400119+0000","last_became_active":"2024-04-24T08:11:40.376301+0000","last_became_peered":"2024-04-24T08:11:40.376301+0000","last_unstale":"2024-04-24T08:11:42.400119+0000","last_undegraded":"2024-04-24T08:11:42.400119+0000","last_fullsized":"2024-04-24T08:11:42.400119+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:11:06.384438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.356874+0000","last_change":"2024-04-24T08:11:39.378902+0000","last_active":"2024-04-24T08:11:41.356874+0000","last_peered":"2024-04-24T08:11:41.356874+0000","last_clean":"2024-04-24T08:11:41.356874+0000","last_became_active":"2024-04-24T08:11:39.378628+0000","last_became_peered":"2024-04-24T08:11:39.378628+0000","last_unstale":"2024-04-24T08:11:41.356874+0000","last_undegraded":"2024-04-24T08:11:41.356874+0000","last_fullsized":"2024-04-24T08:11:41.356874+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:09.889758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400921+0000","last_change":"2024-04-24T08:11:40.397358+0000","last_active":"2024-04-24T08:11:42.400921+0000","last_peered":"2024-04-24T08:11:42.400921+0000","last_clean":"2024-04-24T08:11:42.400921+0000","last_became_active":"2024-04-24T08:11:40.396322+0000","last_became_peered":"2024-04-24T08:11:40.396322+0000","last_unstale":"2024-04-24T08:11:42.400921+0000","last_undegraded":"2024-04-24T08:11:42.400921+0000","last_fullsized":"2024-04-24T08:11:42.400921+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:02.097340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361523+0000","last_change":"2024-04-24T08:11:39.376794+0000","last_active":"2024-04-24T08:11:41.361523+0000","last_peered":"2024-04-24T08:11:41.361523+0000","last_clean":"2024-04-24T08:11:41.361523+0000","last_became_active":"2024-04-24T08:11:39.376266+0000","last_became_peered":"2024-04-24T08:11:39.376266+0000","last_unstale":"2024-04-24T08:11:41.361523+0000","last_undegraded":"2024-04-24T08:11:41.361523+0000","last_fullsized":"2024-04-24T08:11:41.361523+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:54.205485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.356856+0000","last_change":"2024-04-24T08:11:40.386891+0000","last_active":"2024-04-24T08:11:41.356856+0000","last_peered":"2024-04-24T08:11:41.356856+0000","last_clean":"2024-04-24T08:11:41.356856+0000","last_became_active":"2024-04-24T08:11:40.386314+0000","last_became_peered":"2024-04-24T08:11:40.386314+0000","last_unstale":"2024-04-24T08:11:41.356856+0000","last_undegraded":"2024-04-24T08:11:41.356856+0000","last_fullsized":"2024-04-24T08:11:41.356856+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:39:19.795220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.356796+0000","last_change":"2024-04-24T08:11:39.380077+0000","last_active":"2024-04-24T08:11:41.356796+0000","last_peered":"2024-04-24T08:11:41.356796+0000","last_clean":"2024-04-24T08:11:41.356796+0000","last_became_active":"2024-04-24T08:11:39.379757+0000","last_became_peered":"2024-04-24T08:11:39.379757+0000","last_unstale":"2024-04-24T08:11:41.356796+0000","last_undegraded":"2024-04-24T08:11:41.356796+0000","last_fullsized":"2024-04-24T08:11:41.356796+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:23:32.037944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400616+0000","last_change":"2024-04-24T08:11:40.390918+0000","last_active":"2024-04-24T08:11:42.400616+0000","last_peered":"2024-04-24T08:11:42.400616+0000","last_clean":"2024-04-24T08:11:42.400616+0000","last_became_active":"2024-04-24T08:11:40.390620+0000","last_became_peered":"2024-04-24T08:11:40.390620+0000","last_unstale":"2024-04-24T08:11:42.400616+0000","last_undegraded":"2024-04-24T08:11:42.400616+0000","last_fullsized":"2024-04-24T08:11:42.400616+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:31.287452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399790+0000","last_change":"2024-04-24T08:11:39.377388+0000","last_active":"2024-04-24T08:11:42.399790+0000","last_peered":"2024-04-24T08:11:42.399790+0000","last_clean":"2024-04-24T08:11:42.399790+0000","last_became_active":"2024-04-24T08:11:39.376962+0000","last_became_peered":"2024-04-24T08:11:39.376962+0000","last_unstale":"2024-04-24T08:11:42.399790+0000","last_undegraded":"2024-04-24T08:11:42.399790+0000","last_fullsized":"2024-04-24T08:11:42.399790+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:01.081051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400827+0000","last_change":"2024-04-24T08:11:40.371825+0000","last_active":"2024-04-24T08:11:42.400827+0000","last_peered":"2024-04-24T08:11:42.400827+0000","last_clean":"2024-04-24T08:11:42.400827+0000","last_became_active":"2024-04-24T08:11:40.371499+0000","last_became_peered":"2024-04-24T08:11:40.371499+0000","last_unstale":"2024-04-24T08:11:42.400827+0000","last_undegraded":"2024-04-24T08:11:42.400827+0000","last_fullsized":"2024-04-24T08:11:42.400827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:49.898857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376456+0000","last_change":"2024-04-24T08:11:39.384160+0000","last_active":"2024-04-24T08:11:41.376456+0000","last_peered":"2024-04-24T08:11:41.376456+0000","last_clean":"2024-04-24T08:11:41.376456+0000","last_became_active":"2024-04-24T08:11:39.383839+0000","last_became_peered":"2024-04-24T08:11:39.383839+0000","last_unstale":"2024-04-24T08:11:41.376456+0000","last_undegraded":"2024-04-24T08:11:41.376456+0000","last_fullsized":"2024-04-24T08:11:41.376456+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:37:55.284801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399625+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:11:42.399625+0000","last_peered":"2024-04-24T08:11:42.399625+0000","last_clean":"2024-04-24T08:11:42.399625+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:11:42.399625+0000","last_undegraded":"2024-04-24T08:11:42.399625+0000","last_fullsized":"2024-04-24T08:11:42.399625+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":875840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":684032,"data_stored":285312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":285964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":286924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574894,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":285368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":285368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":688128,"data_stored":285402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":877874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:11:44.887 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:11:45.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.122+0000 7fb373000640 1 Processor -- start 2024-04-24T08:11:45.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.122+0000 7fb373000640 1 -- start start 2024-04-24T08:11:45.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.124+0000 7fb373000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb36c15d1d0 0x7fb36c0ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.124+0000 7fb373000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb36c0b30f0 0x7fb36c0ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.124+0000 7fb373000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb36c15d660 con 0x7fb36c156c60 2024-04-24T08:11:45.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.124+0000 7fb373000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb36c06dc30 con 0x7fb36c15d1d0 2024-04-24T08:11:45.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.124+0000 7fb373000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb36c06df10 con 0x7fb36c0b30f0 2024-04-24T08:11:45.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.124+0000 7fb371200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb36c15d1d0 0x7fb36c0ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.124+0000 7fb371200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb36c15d1d0 0x7fb36c0ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47424/0 (socket says 172.21.15.69:47424) 2024-04-24T08:11:45.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.124+0000 7fb371200640 1 -- 172.21.15.69:0/3439477858 learned_addr learned my addr 172.21.15.69:0/3439477858 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:45.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.124+0000 7fb371200640 1 -- 172.21.15.69:0/3439477858 >> v2:172.21.15.121:3300/0 conn(0x7fb36c0b30f0 msgr2=0x7fb36c0ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:45.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.124+0000 7fb371200640 1 --2- 172.21.15.69:0/3439477858 >> v2:172.21.15.121:3300/0 conn(0x7fb36c0b30f0 0x7fb36c0ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.125+0000 7fb371200640 1 -- 172.21.15.69:0/3439477858 >> v1:172.21.15.121:6789/0 conn(0x7fb36c156c60 legacy=0x7fb36c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:45.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.125+0000 7fb371200640 1 -- 172.21.15.69:0/3439477858 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c06e1f0 con 0x7fb36c15d1d0 2024-04-24T08:11:45.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.126+0000 7fb371200640 1 --2- 172.21.15.69:0/3439477858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15d1d0 0x7fb36c0ad0d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0b1d80 tx=0x7fb35802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8975bd94fc4aa80 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:45.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.126+0000 7fb36be00640 1 -- 172.21.15.69:0/3439477858 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358037070 con 0x7fb36c15d1d0 2024-04-24T08:11:45.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.126+0000 7fb36be00640 1 -- 172.21.15.69:0/3439477858 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb358032440 con 0x7fb36c15d1d0 2024-04-24T08:11:45.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.126+0000 7fb36be00640 1 -- 172.21.15.69:0/3439477858 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358032810 con 0x7fb36c15d1d0 2024-04-24T08:11:45.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.126+0000 7fb373000640 1 -- 172.21.15.69:0/3439477858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15d1d0 msgr2=0x7fb36c0ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.126+0000 7fb373000640 1 --2- 172.21.15.69:0/3439477858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15d1d0 0x7fb36c0ad0d0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0b1d80 tx=0x7fb35802a250 comp rx=0 tx=0).stop 2024-04-24T08:11:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.127+0000 7fb373000640 1 -- 172.21.15.69:0/3439477858 shutdown_connections 2024-04-24T08:11:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.127+0000 7fb373000640 1 --2- 172.21.15.69:0/3439477858 >> v2:172.21.15.121:3300/0 conn(0x7fb36c0b30f0 0x7fb36c0ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.127+0000 7fb373000640 1 --2- 172.21.15.69:0/3439477858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15d1d0 0x7fb36c0ad0d0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.127+0000 7fb373000640 1 -- 172.21.15.69:0/3439477858 >> 172.21.15.69:0/3439477858 conn(0x7fb36c0d0160 msgr2=0x7fb36c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.127+0000 7fb373000640 1 -- 172.21.15.69:0/3439477858 shutdown_connections 2024-04-24T08:11:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.127+0000 7fb373000640 1 -- 172.21.15.69:0/3439477858 wait complete. 2024-04-24T08:11:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.129+0000 7fb373000640 1 Processor -- start 2024-04-24T08:11:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.129+0000 7fb373000640 1 -- start start 2024-04-24T08:11:45.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb373000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c0b30f0 0x7fb36c1f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb373000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb36c156c60 0x7fb36c1551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb373000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c073930 0x7fb36c155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb373000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c06e710 con 0x7fb36c073930 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb373000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c15d940 con 0x7fb36c0b30f0 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb373000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb36c06e890 con 0x7fb36c156c60 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb372600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c073930 0x7fb36c155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb372600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c073930 0x7fb36c155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47438/0 (socket says 172.21.15.69:47438) 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb372600640 1 -- 172.21.15.69:0/116718859 learned_addr learned my addr 172.21.15.69:0/116718859 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb371c00640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c0b30f0 0x7fb36c1f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb372600640 1 -- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb36c156c60 msgr2=0x7fb36c1551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb371200640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb36c156c60 0x7fb36c1551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb372600640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb36c156c60 0x7fb36c1551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb372600640 1 -- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c0b30f0 msgr2=0x7fb36c1f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb372600640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c0b30f0 0x7fb36c1f4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.130+0000 7fb372600640 1 -- 172.21.15.69:0/116718859 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c0725f0 con 0x7fb36c073930 2024-04-24T08:11:45.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.131+0000 7fb371c00640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c0b30f0 0x7fb36c1f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:45.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.131+0000 7fb372600640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c073930 0x7fb36c155700 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb36400b9e0 tx=0x7fb36400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:45.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.131+0000 7fb369600640 1 -- 172.21.15.69:0/116718859 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364019070 con 0x7fb36c073930 2024-04-24T08:11:45.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.131+0000 7fb369600640 1 -- 172.21.15.69:0/116718859 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb36400cb40 con 0x7fb36c073930 2024-04-24T08:11:45.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.131+0000 7fb369600640 1 -- 172.21.15.69:0/116718859 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364014440 con 0x7fb36c073930 2024-04-24T08:11:45.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.131+0000 7fb373000640 1 -- 172.21.15.69:0/116718859 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36c072890 con 0x7fb36c073930 2024-04-24T08:11:45.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.131+0000 7fb373000640 1 -- 172.21.15.69:0/116718859 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb36c072e50 con 0x7fb36c073930 2024-04-24T08:11:45.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.132+0000 7fb373000640 1 -- 172.21.15.69:0/116718859 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb36c075d50 con 0x7fb36c073930 2024-04-24T08:11:45.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.133+0000 7fb369600640 1 -- 172.21.15.69:0/116718859 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb36401c070 con 0x7fb36c073930 2024-04-24T08:11:45.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.134+0000 7fb369600640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c920 0x7fb34c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.134+0000 7fb369600640 1 -- 172.21.15.69:0/116718859 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fb36408b840 con 0x7fb36c073930 2024-04-24T08:11:45.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.135+0000 7fb371c00640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c920 0x7fb34c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.136+0000 7fb371c00640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c920 0x7fb34c06ede0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb360007fd0 tx=0x7fb360009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:45.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.138+0000 7fb369600640 1 -- 172.21.15.69:0/116718859 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb364059bd0 con 0x7fb36c073930 2024-04-24T08:11:45.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.334+0000 7fb373000640 1 -- 172.21.15.69:0/116718859 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb36c06f9e0 con 0x7fb34c06c920 2024-04-24T08:11:45.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.353+0000 7fb369600640 1 -- 172.21.15.69:0/116718859 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251954 (secure 0 0 0) 0x7fb36c06f9e0 con 0x7fb34c06c920 2024-04-24T08:11:45.357 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:45.359 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 -- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c920 msgr2=0x7fb34c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c920 0x7fb34c06ede0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb360007fd0 tx=0x7fb360009040 comp rx=0 tx=0).stop 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 -- 172.21.15.69:0/116718859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c073930 msgr2=0x7fb36c155700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c073930 0x7fb36c155700 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fb36400b9e0 tx=0x7fb36400bea0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 -- 172.21.15.69:0/116718859 shutdown_connections 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c920 0x7fb34c06ede0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c073930 0x7fb36c155700 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb36c156c60 0x7fb36c1551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 --2- 172.21.15.69:0/116718859 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c0b30f0 0x7fb36c1f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 -- 172.21.15.69:0/116718859 >> 172.21.15.69:0/116718859 conn(0x7fb36c0d0160 msgr2=0x7fb36c15b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 -- 172.21.15.69:0/116718859 shutdown_connections 2024-04-24T08:11:45.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.360+0000 7fb373000640 1 -- 172.21.15.69:0/116718859 wait complete. 2024-04-24T08:11:45.379 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-04-24T08:11:44.735231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239400,"kb_used_data":6568,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754735320,"statfs":{"total":773094113280,"available":772848967680,"internally_reserved":0,"allocated":6725632,"data_stored":3468052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001921"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.402811+0000","last_change":"2024-04-24T08:11:40.376215+0000","last_active":"2024-04-24T08:11:42.402811+0000","last_peered":"2024-04-24T08:11:42.402811+0000","last_clean":"2024-04-24T08:11:42.402811+0000","last_became_active":"2024-04-24T08:11:40.375891+0000","last_became_peered":"2024-04-24T08:11:40.375891+0000","last_unstale":"2024-04-24T08:11:42.402811+0000","last_undegraded":"2024-04-24T08:11:42.402811+0000","last_fullsized":"2024-04-24T08:11:42.402811+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:54:55.927367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399894+0000","last_change":"2024-04-24T08:11:40.385512+0000","last_active":"2024-04-24T08:11:42.399894+0000","last_peered":"2024-04-24T08:11:42.399894+0000","last_clean":"2024-04-24T08:11:42.399894+0000","last_became_active":"2024-04-24T08:11:40.384593+0000","last_became_peered":"2024-04-24T08:11:40.384593+0000","last_unstale":"2024-04-24T08:11:42.399894+0000","last_undegraded":"2024-04-24T08:11:42.399894+0000","last_fullsized":"2024-04-24T08:11:42.399894+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:20:09.861597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400150+0000","last_change":"2024-04-24T08:11:40.397826+0000","last_active":"2024-04-24T08:11:42.400150+0000","last_peered":"2024-04-24T08:11:42.400150+0000","last_clean":"2024-04-24T08:11:42.400150+0000","last_became_active":"2024-04-24T08:11:40.397298+0000","last_became_peered":"2024-04-24T08:11:40.397298+0000","last_unstale":"2024-04-24T08:11:42.400150+0000","last_undegraded":"2024-04-24T08:11:42.400150+0000","last_fullsized":"2024-04-24T08:11:42.400150+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:07:16.992916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361377+0000","last_change":"2024-04-24T08:11:40.377612+0000","last_active":"2024-04-24T08:11:41.361377+0000","last_peered":"2024-04-24T08:11:41.361377+0000","last_clean":"2024-04-24T08:11:41.361377+0000","last_became_active":"2024-04-24T08:11:40.377348+0000","last_became_peered":"2024-04-24T08:11:40.377348+0000","last_unstale":"2024-04-24T08:11:41.361377+0000","last_undegraded":"2024-04-24T08:11:41.361377+0000","last_fullsized":"2024-04-24T08:11:41.361377+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:07.648391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401356+0000","last_change":"2024-04-24T08:11:40.384705+0000","last_active":"2024-04-24T08:11:42.401356+0000","last_peered":"2024-04-24T08:11:42.401356+0000","last_clean":"2024-04-24T08:11:42.401356+0000","last_became_active":"2024-04-24T08:11:40.384299+0000","last_became_peered":"2024-04-24T08:11:40.384299+0000","last_unstale":"2024-04-24T08:11:42.401356+0000","last_undegraded":"2024-04-24T08:11:42.401356+0000","last_fullsized":"2024-04-24T08:11:42.401356+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:00:00.237349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369087+0000","last_change":"2024-04-24T08:11:40.364143+0000","last_active":"2024-04-24T08:11:41.369087+0000","last_peered":"2024-04-24T08:11:41.369087+0000","last_clean":"2024-04-24T08:11:41.369087+0000","last_became_active":"2024-04-24T08:11:40.363810+0000","last_became_peered":"2024-04-24T08:11:40.363810+0000","last_unstale":"2024-04-24T08:11:41.369087+0000","last_undegraded":"2024-04-24T08:11:41.369087+0000","last_fullsized":"2024-04-24T08:11:41.369087+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:38:05.022070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.360844+0000","last_change":"2024-04-24T08:11:40.383207+0000","last_active":"2024-04-24T08:11:41.360844+0000","last_peered":"2024-04-24T08:11:41.360844+0000","last_clean":"2024-04-24T08:11:41.360844+0000","last_became_active":"2024-04-24T08:11:40.382145+0000","last_became_peered":"2024-04-24T08:11:40.382145+0000","last_unstale":"2024-04-24T08:11:41.360844+0000","last_undegraded":"2024-04-24T08:11:41.360844+0000","last_fullsized":"2024-04-24T08:11:41.360844+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:38:25.942597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399610+0000","last_change":"2024-04-24T08:11:40.383203+0000","last_active":"2024-04-24T08:11:42.399610+0000","last_peered":"2024-04-24T08:11:42.399610+0000","last_clean":"2024-04-24T08:11:42.399610+0000","last_became_active":"2024-04-24T08:11:40.382913+0000","last_became_peered":"2024-04-24T08:11:40.382913+0000","last_unstale":"2024-04-24T08:11:42.399610+0000","last_undegraded":"2024-04-24T08:11:42.399610+0000","last_fullsized":"2024-04-24T08:11:42.399610+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:14:04.421220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361346+0000","last_change":"2024-04-24T08:11:40.376103+0000","last_active":"2024-04-24T08:11:41.361346+0000","last_peered":"2024-04-24T08:11:41.361346+0000","last_clean":"2024-04-24T08:11:41.361346+0000","last_became_active":"2024-04-24T08:11:40.375660+0000","last_became_peered":"2024-04-24T08:11:40.375660+0000","last_unstale":"2024-04-24T08:11:41.361346+0000","last_undegraded":"2024-04-24T08:11:41.361346+0000","last_fullsized":"2024-04-24T08:11:41.361346+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:08:41.295372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.357855+0000","last_change":"2024-04-24T08:11:40.386603+0000","last_active":"2024-04-24T08:11:41.357855+0000","last_peered":"2024-04-24T08:11:41.357855+0000","last_clean":"2024-04-24T08:11:41.357855+0000","last_became_active":"2024-04-24T08:11:40.386291+0000","last_became_peered":"2024-04-24T08:11:40.386291+0000","last_unstale":"2024-04-24T08:11:41.357855+0000","last_undegraded":"2024-04-24T08:11:41.357855+0000","last_fullsized":"2024-04-24T08:11:41.357855+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:46.551188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.357531+0000","last_change":"2024-04-24T08:11:40.380790+0000","last_active":"2024-04-24T08:11:41.357531+0000","last_peered":"2024-04-24T08:11:41.357531+0000","last_clean":"2024-04-24T08:11:41.357531+0000","last_became_active":"2024-04-24T08:11:40.380179+0000","last_became_peered":"2024-04-24T08:11:40.380179+0000","last_unstale":"2024-04-24T08:11:41.357531+0000","last_undegraded":"2024-04-24T08:11:41.357531+0000","last_fullsized":"2024-04-24T08:11:41.357531+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:33.253720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362307+0000","last_change":"2024-04-24T08:11:40.387428+0000","last_active":"2024-04-24T08:11:41.362307+0000","last_peered":"2024-04-24T08:11:41.362307+0000","last_clean":"2024-04-24T08:11:41.362307+0000","last_became_active":"2024-04-24T08:11:40.386722+0000","last_became_peered":"2024-04-24T08:11:40.386722+0000","last_unstale":"2024-04-24T08:11:41.362307+0000","last_undegraded":"2024-04-24T08:11:41.362307+0000","last_fullsized":"2024-04-24T08:11:41.362307+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:37:26.171195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376957+0000","last_change":"2024-04-24T08:11:40.398597+0000","last_active":"2024-04-24T08:11:41.376957+0000","last_peered":"2024-04-24T08:11:41.376957+0000","last_clean":"2024-04-24T08:11:41.376957+0000","last_became_active":"2024-04-24T08:11:40.398313+0000","last_became_peered":"2024-04-24T08:11:40.398313+0000","last_unstale":"2024-04-24T08:11:41.376957+0000","last_undegraded":"2024-04-24T08:11:41.376957+0000","last_fullsized":"2024-04-24T08:11:41.376957+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:53:39.597009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361390+0000","last_change":"2024-04-24T08:11:40.389192+0000","last_active":"2024-04-24T08:11:41.361390+0000","last_peered":"2024-04-24T08:11:41.361390+0000","last_clean":"2024-04-24T08:11:41.361390+0000","last_became_active":"2024-04-24T08:11:40.388802+0000","last_became_peered":"2024-04-24T08:11:40.388802+0000","last_unstale":"2024-04-24T08:11:41.361390+0000","last_undegraded":"2024-04-24T08:11:41.361390+0000","last_fullsized":"2024-04-24T08:11:41.361390+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:21:44.729682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361108+0000","last_change":"2024-04-24T08:11:40.378517+0000","last_active":"2024-04-24T08:11:41.361108+0000","last_peered":"2024-04-24T08:11:41.361108+0000","last_clean":"2024-04-24T08:11:41.361108+0000","last_became_active":"2024-04-24T08:11:40.378235+0000","last_became_peered":"2024-04-24T08:11:40.378235+0000","last_unstale":"2024-04-24T08:11:41.361108+0000","last_undegraded":"2024-04-24T08:11:41.361108+0000","last_fullsized":"2024-04-24T08:11:41.361108+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:21:25.512224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369192+0000","last_change":"2024-04-24T08:11:40.390368+0000","last_active":"2024-04-24T08:11:41.369192+0000","last_peered":"2024-04-24T08:11:41.369192+0000","last_clean":"2024-04-24T08:11:41.369192+0000","last_became_active":"2024-04-24T08:11:40.390091+0000","last_became_peered":"2024-04-24T08:11:40.390091+0000","last_unstale":"2024-04-24T08:11:41.369192+0000","last_undegraded":"2024-04-24T08:11:41.369192+0000","last_fullsized":"2024-04-24T08:11:41.369192+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:02.097340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401982+0000","last_change":"2024-04-24T08:11:40.395344+0000","last_active":"2024-04-24T08:11:42.401982+0000","last_peered":"2024-04-24T08:11:42.401982+0000","last_clean":"2024-04-24T08:11:42.401982+0000","last_became_active":"2024-04-24T08:11:40.395047+0000","last_became_peered":"2024-04-24T08:11:40.395047+0000","last_unstale":"2024-04-24T08:11:42.401982+0000","last_undegraded":"2024-04-24T08:11:42.401982+0000","last_fullsized":"2024-04-24T08:11:42.401982+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:35:59.110144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399980+0000","last_change":"2024-04-24T08:11:40.401229+0000","last_active":"2024-04-24T08:11:42.399980+0000","last_peered":"2024-04-24T08:11:42.399980+0000","last_clean":"2024-04-24T08:11:42.399980+0000","last_became_active":"2024-04-24T08:11:40.400954+0000","last_became_peered":"2024-04-24T08:11:40.400954+0000","last_unstale":"2024-04-24T08:11:42.399980+0000","last_undegraded":"2024-04-24T08:11:42.399980+0000","last_fullsized":"2024-04-24T08:11:42.399980+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:07.648391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401946+0000","last_change":"2024-04-24T08:11:40.376761+0000","last_active":"2024-04-24T08:11:42.401946+0000","last_peered":"2024-04-24T08:11:42.401946+0000","last_clean":"2024-04-24T08:11:42.401946+0000","last_became_active":"2024-04-24T08:11:40.375938+0000","last_became_peered":"2024-04-24T08:11:40.375938+0000","last_unstale":"2024-04-24T08:11:42.401946+0000","last_undegraded":"2024-04-24T08:11:42.401946+0000","last_fullsized":"2024-04-24T08:11:42.401946+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:05:12.711112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400397+0000","last_change":"2024-04-24T08:11:40.370776+0000","last_active":"2024-04-24T08:11:42.400397+0000","last_peered":"2024-04-24T08:11:42.400397+0000","last_clean":"2024-04-24T08:11:42.400397+0000","last_became_active":"2024-04-24T08:11:40.369266+0000","last_became_peered":"2024-04-24T08:11:40.369266+0000","last_unstale":"2024-04-24T08:11:42.400397+0000","last_undegraded":"2024-04-24T08:11:42.400397+0000","last_fullsized":"2024-04-24T08:11:42.400397+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:21.582641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400625+0000","last_change":"2024-04-24T08:11:40.396840+0000","last_active":"2024-04-24T08:11:42.400625+0000","last_peered":"2024-04-24T08:11:42.400625+0000","last_clean":"2024-04-24T08:11:42.400625+0000","last_became_active":"2024-04-24T08:11:40.395773+0000","last_became_peered":"2024-04-24T08:11:40.395773+0000","last_unstale":"2024-04-24T08:11:42.400625+0000","last_undegraded":"2024-04-24T08:11:42.400625+0000","last_fullsized":"2024-04-24T08:11:42.400625+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:26.845618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399674+0000","last_change":"2024-04-24T08:11:40.384306+0000","last_active":"2024-04-24T08:11:42.399674+0000","last_peered":"2024-04-24T08:11:42.399674+0000","last_clean":"2024-04-24T08:11:42.399674+0000","last_became_active":"2024-04-24T08:11:40.383983+0000","last_became_peered":"2024-04-24T08:11:40.383983+0000","last_unstale":"2024-04-24T08:11:42.399674+0000","last_undegraded":"2024-04-24T08:11:42.399674+0000","last_fullsized":"2024-04-24T08:11:42.399674+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:42:19.803055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400935+0000","last_change":"2024-04-24T08:11:40.396847+0000","last_active":"2024-04-24T08:11:42.400935+0000","last_peered":"2024-04-24T08:11:42.400935+0000","last_clean":"2024-04-24T08:11:42.400935+0000","last_became_active":"2024-04-24T08:11:40.396550+0000","last_became_peered":"2024-04-24T08:11:40.396550+0000","last_unstale":"2024-04-24T08:11:42.400935+0000","last_undegraded":"2024-04-24T08:11:42.400935+0000","last_fullsized":"2024-04-24T08:11:42.400935+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:03:27.646675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376355+0000","last_change":"2024-04-24T08:11:40.398853+0000","last_active":"2024-04-24T08:11:41.376355+0000","last_peered":"2024-04-24T08:11:41.376355+0000","last_clean":"2024-04-24T08:11:41.376355+0000","last_became_active":"2024-04-24T08:11:40.398590+0000","last_became_peered":"2024-04-24T08:11:40.398590+0000","last_unstale":"2024-04-24T08:11:41.376355+0000","last_undegraded":"2024-04-24T08:11:41.376355+0000","last_fullsized":"2024-04-24T08:11:41.376355+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:15:06.291851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.402093+0000","last_change":"2024-04-24T08:11:40.383822+0000","last_active":"2024-04-24T08:11:42.402093+0000","last_peered":"2024-04-24T08:11:42.402093+0000","last_clean":"2024-04-24T08:11:42.402093+0000","last_became_active":"2024-04-24T08:11:40.382831+0000","last_became_peered":"2024-04-24T08:11:40.382831+0000","last_unstale":"2024-04-24T08:11:42.402093+0000","last_undegraded":"2024-04-24T08:11:42.402093+0000","last_fullsized":"2024-04-24T08:11:42.402093+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:40.021516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362551+0000","last_change":"2024-04-24T08:11:40.382274+0000","last_active":"2024-04-24T08:11:41.362551+0000","last_peered":"2024-04-24T08:11:41.362551+0000","last_clean":"2024-04-24T08:11:41.362551+0000","last_became_active":"2024-04-24T08:11:40.381918+0000","last_became_peered":"2024-04-24T08:11:40.381918+0000","last_unstale":"2024-04-24T08:11:41.362551+0000","last_undegraded":"2024-04-24T08:11:41.362551+0000","last_fullsized":"2024-04-24T08:11:41.362551+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:07:16.992916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370203+0000","last_change":"2024-04-24T08:11:40.389836+0000","last_active":"2024-04-24T08:11:41.370203+0000","last_peered":"2024-04-24T08:11:41.370203+0000","last_clean":"2024-04-24T08:11:41.370203+0000","last_became_active":"2024-04-24T08:11:40.389477+0000","last_became_peered":"2024-04-24T08:11:40.389477+0000","last_unstale":"2024-04-24T08:11:41.370203+0000","last_undegraded":"2024-04-24T08:11:41.370203+0000","last_fullsized":"2024-04-24T08:11:41.370203+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:36.276050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401043+0000","last_change":"2024-04-24T08:11:40.382217+0000","last_active":"2024-04-24T08:11:42.401043+0000","last_peered":"2024-04-24T08:11:42.401043+0000","last_clean":"2024-04-24T08:11:42.401043+0000","last_became_active":"2024-04-24T08:11:40.381675+0000","last_became_peered":"2024-04-24T08:11:40.381675+0000","last_unstale":"2024-04-24T08:11:42.401043+0000","last_undegraded":"2024-04-24T08:11:42.401043+0000","last_fullsized":"2024-04-24T08:11:42.401043+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:11:58.549612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401034+0000","last_change":"2024-04-24T08:11:40.375810+0000","last_active":"2024-04-24T08:11:42.401034+0000","last_peered":"2024-04-24T08:11:42.401034+0000","last_clean":"2024-04-24T08:11:42.401034+0000","last_became_active":"2024-04-24T08:11:40.375509+0000","last_became_peered":"2024-04-24T08:11:40.375509+0000","last_unstale":"2024-04-24T08:11:42.401034+0000","last_undegraded":"2024-04-24T08:11:42.401034+0000","last_fullsized":"2024-04-24T08:11:42.401034+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:38:08.948214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400520+0000","last_change":"2024-04-24T08:11:40.394095+0000","last_active":"2024-04-24T08:11:42.400520+0000","last_peered":"2024-04-24T08:11:42.400520+0000","last_clean":"2024-04-24T08:11:42.400520+0000","last_became_active":"2024-04-24T08:11:40.393767+0000","last_became_peered":"2024-04-24T08:11:40.393767+0000","last_unstale":"2024-04-24T08:11:42.400520+0000","last_undegraded":"2024-04-24T08:11:42.400520+0000","last_fullsized":"2024-04-24T08:11:42.400520+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:56:58.949019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399832+0000","last_change":"2024-04-24T08:11:40.377427+0000","last_active":"2024-04-24T08:11:42.399832+0000","last_peered":"2024-04-24T08:11:42.399832+0000","last_clean":"2024-04-24T08:11:42.399832+0000","last_became_active":"2024-04-24T08:11:40.377117+0000","last_became_peered":"2024-04-24T08:11:40.377117+0000","last_unstale":"2024-04-24T08:11:42.399832+0000","last_undegraded":"2024-04-24T08:11:42.399832+0000","last_fullsized":"2024-04-24T08:11:42.399832+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:41:54.684930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369704+0000","last_change":"2024-04-24T08:11:40.368446+0000","last_active":"2024-04-24T08:11:41.369704+0000","last_peered":"2024-04-24T08:11:41.369704+0000","last_clean":"2024-04-24T08:11:41.369704+0000","last_became_active":"2024-04-24T08:11:40.368060+0000","last_became_peered":"2024-04-24T08:11:40.368060+0000","last_unstale":"2024-04-24T08:11:41.369704+0000","last_undegraded":"2024-04-24T08:11:41.369704+0000","last_fullsized":"2024-04-24T08:11:41.369704+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:05:45.485156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370115+0000","last_change":"2024-04-24T08:11:40.391905+0000","last_active":"2024-04-24T08:11:41.370115+0000","last_peered":"2024-04-24T08:11:41.370115+0000","last_clean":"2024-04-24T08:11:41.370115+0000","last_became_active":"2024-04-24T08:11:40.391600+0000","last_became_peered":"2024-04-24T08:11:40.391600+0000","last_unstale":"2024-04-24T08:11:41.370115+0000","last_undegraded":"2024-04-24T08:11:41.370115+0000","last_fullsized":"2024-04-24T08:11:41.370115+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:33:39.406702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"56'4","reported_seq":18,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369862+0000","last_change":"2024-04-24T08:11:39.377325+0000","last_active":"2024-04-24T08:11:41.369862+0000","last_peered":"2024-04-24T08:11:41.369862+0000","last_clean":"2024-04-24T08:11:41.369862+0000","last_became_active":"2024-04-24T08:11:39.376998+0000","last_became_peered":"2024-04-24T08:11:39.376998+0000","last_unstale":"2024-04-24T08:11:41.369862+0000","last_undegraded":"2024-04-24T08:11:41.369862+0000","last_fullsized":"2024-04-24T08:11:41.369862+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:38:24.926308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.358013+0000","last_change":"2024-04-24T08:11:40.396290+0000","last_active":"2024-04-24T08:11:41.358013+0000","last_peered":"2024-04-24T08:11:41.358013+0000","last_clean":"2024-04-24T08:11:41.358013+0000","last_became_active":"2024-04-24T08:11:40.396009+0000","last_became_peered":"2024-04-24T08:11:40.396009+0000","last_unstale":"2024-04-24T08:11:41.358013+0000","last_undegraded":"2024-04-24T08:11:41.358013+0000","last_fullsized":"2024-04-24T08:11:41.358013+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:55.221773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400050+0000","last_change":"2024-04-24T08:11:39.369869+0000","last_active":"2024-04-24T08:11:42.400050+0000","last_peered":"2024-04-24T08:11:42.400050+0000","last_clean":"2024-04-24T08:11:42.400050+0000","last_became_active":"2024-04-24T08:11:39.369558+0000","last_became_peered":"2024-04-24T08:11:39.369558+0000","last_unstale":"2024-04-24T08:11:42.400050+0000","last_undegraded":"2024-04-24T08:11:42.400050+0000","last_fullsized":"2024-04-24T08:11:42.400050+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:46:25.829329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369761+0000","last_change":"2024-04-24T08:11:40.387704+0000","last_active":"2024-04-24T08:11:41.369761+0000","last_peered":"2024-04-24T08:11:41.369761+0000","last_clean":"2024-04-24T08:11:41.369761+0000","last_became_active":"2024-04-24T08:11:40.387411+0000","last_became_peered":"2024-04-24T08:11:40.387411+0000","last_unstale":"2024-04-24T08:11:41.369761+0000","last_undegraded":"2024-04-24T08:11:41.369761+0000","last_fullsized":"2024-04-24T08:11:41.369761+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:27.911530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370350+0000","last_change":"2024-04-24T08:11:39.369452+0000","last_active":"2024-04-24T08:11:41.370350+0000","last_peered":"2024-04-24T08:11:41.370350+0000","last_clean":"2024-04-24T08:11:41.370350+0000","last_became_active":"2024-04-24T08:11:39.369166+0000","last_became_peered":"2024-04-24T08:11:39.369166+0000","last_unstale":"2024-04-24T08:11:41.370350+0000","last_undegraded":"2024-04-24T08:11:41.370350+0000","last_fullsized":"2024-04-24T08:11:41.370350+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:08:40.279084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370205+0000","last_change":"2024-04-24T08:11:40.375929+0000","last_active":"2024-04-24T08:11:41.370205+0000","last_peered":"2024-04-24T08:11:41.370205+0000","last_clean":"2024-04-24T08:11:41.370205+0000","last_became_active":"2024-04-24T08:11:40.375666+0000","last_became_peered":"2024-04-24T08:11:40.375666+0000","last_unstale":"2024-04-24T08:11:41.370205+0000","last_undegraded":"2024-04-24T08:11:41.370205+0000","last_fullsized":"2024-04-24T08:11:41.370205+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:43.801149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361550+0000","last_change":"2024-04-24T08:11:39.378633+0000","last_active":"2024-04-24T08:11:41.361550+0000","last_peered":"2024-04-24T08:11:41.361550+0000","last_clean":"2024-04-24T08:11:41.361550+0000","last_became_active":"2024-04-24T08:11:39.378214+0000","last_became_peered":"2024-04-24T08:11:39.378214+0000","last_unstale":"2024-04-24T08:11:41.361550+0000","last_undegraded":"2024-04-24T08:11:41.361550+0000","last_fullsized":"2024-04-24T08:11:41.361550+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:55:01.374287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401405+0000","last_change":"2024-04-24T08:11:40.383737+0000","last_active":"2024-04-24T08:11:42.401405+0000","last_peered":"2024-04-24T08:11:42.401405+0000","last_clean":"2024-04-24T08:11:42.401405+0000","last_became_active":"2024-04-24T08:11:40.383395+0000","last_became_peered":"2024-04-24T08:11:40.383395+0000","last_unstale":"2024-04-24T08:11:42.401405+0000","last_undegraded":"2024-04-24T08:11:42.401405+0000","last_fullsized":"2024-04-24T08:11:42.401405+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:32:11.937172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.360904+0000","last_change":"2024-04-24T08:11:39.380088+0000","last_active":"2024-04-24T08:11:41.360904+0000","last_peered":"2024-04-24T08:11:41.360904+0000","last_clean":"2024-04-24T08:11:41.360904+0000","last_became_active":"2024-04-24T08:11:39.379580+0000","last_became_peered":"2024-04-24T08:11:39.379580+0000","last_unstale":"2024-04-24T08:11:41.360904+0000","last_undegraded":"2024-04-24T08:11:41.360904+0000","last_fullsized":"2024-04-24T08:11:41.360904+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:53:38.580721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400599+0000","last_change":"2024-04-24T08:11:40.371654+0000","last_active":"2024-04-24T08:11:42.400599+0000","last_peered":"2024-04-24T08:11:42.400599+0000","last_clean":"2024-04-24T08:11:42.400599+0000","last_became_active":"2024-04-24T08:11:40.371228+0000","last_became_peered":"2024-04-24T08:11:40.371228+0000","last_unstale":"2024-04-24T08:11:42.400599+0000","last_undegraded":"2024-04-24T08:11:42.400599+0000","last_fullsized":"2024-04-24T08:11:42.400599+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:01:41.726594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369124+0000","last_change":"2024-04-24T08:11:39.382674+0000","last_active":"2024-04-24T08:11:41.369124+0000","last_peered":"2024-04-24T08:11:41.369124+0000","last_clean":"2024-04-24T08:11:41.369124+0000","last_became_active":"2024-04-24T08:11:39.382353+0000","last_became_peered":"2024-04-24T08:11:39.382353+0000","last_unstale":"2024-04-24T08:11:41.369124+0000","last_undegraded":"2024-04-24T08:11:41.369124+0000","last_fullsized":"2024-04-24T08:11:41.369124+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:21:24.495936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376352+0000","last_change":"2024-04-24T08:11:40.394145+0000","last_active":"2024-04-24T08:11:41.376352+0000","last_peered":"2024-04-24T08:11:41.376352+0000","last_clean":"2024-04-24T08:11:41.376352+0000","last_became_active":"2024-04-24T08:11:40.393858+0000","last_became_peered":"2024-04-24T08:11:40.393858+0000","last_unstale":"2024-04-24T08:11:41.376352+0000","last_undegraded":"2024-04-24T08:11:41.376352+0000","last_fullsized":"2024-04-24T08:11:41.376352+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:39:19.795220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.359540+0000","last_change":"2024-04-24T08:11:39.380789+0000","last_active":"2024-04-24T08:11:41.359540+0000","last_peered":"2024-04-24T08:11:41.359540+0000","last_clean":"2024-04-24T08:11:41.359540+0000","last_became_active":"2024-04-24T08:11:39.380460+0000","last_became_peered":"2024-04-24T08:11:39.380460+0000","last_unstale":"2024-04-24T08:11:41.359540+0000","last_undegraded":"2024-04-24T08:11:41.359540+0000","last_fullsized":"2024-04-24T08:11:41.359540+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:45.534899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400301+0000","last_change":"2024-04-24T08:11:40.396741+0000","last_active":"2024-04-24T08:11:42.400301+0000","last_peered":"2024-04-24T08:11:42.400301+0000","last_clean":"2024-04-24T08:11:42.400301+0000","last_became_active":"2024-04-24T08:11:40.394132+0000","last_became_peered":"2024-04-24T08:11:40.394132+0000","last_unstale":"2024-04-24T08:11:42.400301+0000","last_undegraded":"2024-04-24T08:11:42.400301+0000","last_fullsized":"2024-04-24T08:11:42.400301+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:52:25.200562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361253+0000","last_change":"2024-04-24T08:11:39.382605+0000","last_active":"2024-04-24T08:11:41.361253+0000","last_peered":"2024-04-24T08:11:41.361253+0000","last_clean":"2024-04-24T08:11:41.361253+0000","last_became_active":"2024-04-24T08:11:39.381255+0000","last_became_peered":"2024-04-24T08:11:39.381255+0000","last_unstale":"2024-04-24T08:11:41.361253+0000","last_undegraded":"2024-04-24T08:11:41.361253+0000","last_fullsized":"2024-04-24T08:11:41.361253+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:39:32.237431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362068+0000","last_change":"2024-04-24T08:11:40.381424+0000","last_active":"2024-04-24T08:11:41.362068+0000","last_peered":"2024-04-24T08:11:41.362068+0000","last_clean":"2024-04-24T08:11:41.362068+0000","last_became_active":"2024-04-24T08:11:40.381063+0000","last_became_peered":"2024-04-24T08:11:40.381063+0000","last_unstale":"2024-04-24T08:11:41.362068+0000","last_undegraded":"2024-04-24T08:11:41.362068+0000","last_fullsized":"2024-04-24T08:11:41.362068+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:51.053178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400441+0000","last_change":"2024-04-24T08:11:39.380932+0000","last_active":"2024-04-24T08:11:42.400441+0000","last_peered":"2024-04-24T08:11:42.400441+0000","last_clean":"2024-04-24T08:11:42.400441+0000","last_became_active":"2024-04-24T08:11:39.380608+0000","last_became_peered":"2024-04-24T08:11:39.380608+0000","last_unstale":"2024-04-24T08:11:42.400441+0000","last_undegraded":"2024-04-24T08:11:42.400441+0000","last_fullsized":"2024-04-24T08:11:42.400441+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:19:57.821537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400625+0000","last_change":"2024-04-24T08:11:40.401365+0000","last_active":"2024-04-24T08:11:42.400625+0000","last_peered":"2024-04-24T08:11:42.400625+0000","last_clean":"2024-04-24T08:11:42.400625+0000","last_became_active":"2024-04-24T08:11:40.401056+0000","last_became_peered":"2024-04-24T08:11:40.401056+0000","last_unstale":"2024-04-24T08:11:42.400625+0000","last_undegraded":"2024-04-24T08:11:42.400625+0000","last_fullsized":"2024-04-24T08:11:42.400625+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:35:08.901472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400347+0000","last_change":"2024-04-24T08:11:39.378314+0000","last_active":"2024-04-24T08:11:42.400347+0000","last_peered":"2024-04-24T08:11:42.400347+0000","last_clean":"2024-04-24T08:11:42.400347+0000","last_became_active":"2024-04-24T08:11:39.377954+0000","last_became_peered":"2024-04-24T08:11:39.377954+0000","last_unstale":"2024-04-24T08:11:42.400347+0000","last_undegraded":"2024-04-24T08:11:42.400347+0000","last_fullsized":"2024-04-24T08:11:42.400347+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:38:04.005782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.357132+0000","last_change":"2024-04-24T08:11:40.388856+0000","last_active":"2024-04-24T08:11:41.357132+0000","last_peered":"2024-04-24T08:11:41.357132+0000","last_clean":"2024-04-24T08:11:41.357132+0000","last_became_active":"2024-04-24T08:11:40.388500+0000","last_became_peered":"2024-04-24T08:11:40.388500+0000","last_unstale":"2024-04-24T08:11:41.357132+0000","last_undegraded":"2024-04-24T08:11:41.357132+0000","last_fullsized":"2024-04-24T08:11:41.357132+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:15:06.291851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"56'3","reported_seq":17,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401061+0000","last_change":"2024-04-24T08:11:39.379805+0000","last_active":"2024-04-24T08:11:42.401061+0000","last_peered":"2024-04-24T08:11:42.401061+0000","last_clean":"2024-04-24T08:11:42.401061+0000","last_became_active":"2024-04-24T08:11:39.379459+0000","last_became_peered":"2024-04-24T08:11:39.379459+0000","last_unstale":"2024-04-24T08:11:42.401061+0000","last_undegraded":"2024-04-24T08:11:42.401061+0000","last_fullsized":"2024-04-24T08:11:42.401061+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:24:44.359232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400923+0000","last_change":"2024-04-24T08:11:40.397553+0000","last_active":"2024-04-24T08:11:42.400923+0000","last_peered":"2024-04-24T08:11:42.400923+0000","last_clean":"2024-04-24T08:11:42.400923+0000","last_became_active":"2024-04-24T08:11:40.396793+0000","last_became_peered":"2024-04-24T08:11:40.396793+0000","last_unstale":"2024-04-24T08:11:42.400923+0000","last_undegraded":"2024-04-24T08:11:42.400923+0000","last_fullsized":"2024-04-24T08:11:42.400923+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:05:45.485156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"56'6","reported_seq":20,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361712+0000","last_change":"2024-04-24T08:11:39.376800+0000","last_active":"2024-04-24T08:11:41.361712+0000","last_peered":"2024-04-24T08:11:41.361712+0000","last_clean":"2024-04-24T08:11:41.361712+0000","last_became_active":"2024-04-24T08:11:39.376483+0000","last_became_peered":"2024-04-24T08:11:39.376483+0000","last_unstale":"2024-04-24T08:11:41.361712+0000","last_undegraded":"2024-04-24T08:11:41.361712+0000","last_fullsized":"2024-04-24T08:11:41.361712+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:45.534899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.359788+0000","last_change":"2024-04-24T08:11:40.379479+0000","last_active":"2024-04-24T08:11:41.359788+0000","last_peered":"2024-04-24T08:11:41.359788+0000","last_clean":"2024-04-24T08:11:41.359788+0000","last_became_active":"2024-04-24T08:11:40.379135+0000","last_became_peered":"2024-04-24T08:11:40.379135+0000","last_unstale":"2024-04-24T08:11:41.359788+0000","last_undegraded":"2024-04-24T08:11:41.359788+0000","last_fullsized":"2024-04-24T08:11:41.359788+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:14.674330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400741+0000","last_change":"2024-04-24T08:11:39.370114+0000","last_active":"2024-04-24T08:11:42.400741+0000","last_peered":"2024-04-24T08:11:42.400741+0000","last_clean":"2024-04-24T08:11:42.400741+0000","last_became_active":"2024-04-24T08:11:39.369750+0000","last_became_peered":"2024-04-24T08:11:39.369750+0000","last_unstale":"2024-04-24T08:11:42.400741+0000","last_undegraded":"2024-04-24T08:11:42.400741+0000","last_fullsized":"2024-04-24T08:11:42.400741+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:56:57.932731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400379+0000","last_change":"2024-04-24T08:11:40.395703+0000","last_active":"2024-04-24T08:11:42.400379+0000","last_peered":"2024-04-24T08:11:42.400379+0000","last_clean":"2024-04-24T08:11:42.400379+0000","last_became_active":"2024-04-24T08:11:40.395427+0000","last_became_peered":"2024-04-24T08:11:40.395427+0000","last_unstale":"2024-04-24T08:11:42.400379+0000","last_undegraded":"2024-04-24T08:11:42.400379+0000","last_fullsized":"2024-04-24T08:11:42.400379+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:20:09.876703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376244+0000","last_change":"2024-04-24T08:11:39.382845+0000","last_active":"2024-04-24T08:11:41.376244+0000","last_peered":"2024-04-24T08:11:41.376244+0000","last_clean":"2024-04-24T08:11:41.376244+0000","last_became_active":"2024-04-24T08:11:39.381666+0000","last_became_peered":"2024-04-24T08:11:39.381666+0000","last_unstale":"2024-04-24T08:11:41.376244+0000","last_undegraded":"2024-04-24T08:11:41.376244+0000","last_fullsized":"2024-04-24T08:11:41.376244+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:01:20.271949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361273+0000","last_change":"2024-04-24T08:11:40.394948+0000","last_active":"2024-04-24T08:11:41.361273+0000","last_peered":"2024-04-24T08:11:41.361273+0000","last_clean":"2024-04-24T08:11:41.361273+0000","last_became_active":"2024-04-24T08:11:40.392461+0000","last_became_peered":"2024-04-24T08:11:40.392461+0000","last_unstale":"2024-04-24T08:11:41.361273+0000","last_undegraded":"2024-04-24T08:11:41.361273+0000","last_fullsized":"2024-04-24T08:11:41.361273+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:14:30.148003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.359596+0000","last_change":"2024-04-24T08:11:39.378417+0000","last_active":"2024-04-24T08:11:41.359596+0000","last_peered":"2024-04-24T08:11:41.359596+0000","last_clean":"2024-04-24T08:11:41.359596+0000","last_became_active":"2024-04-24T08:11:39.378063+0000","last_became_peered":"2024-04-24T08:11:39.378063+0000","last_unstale":"2024-04-24T08:11:41.359596+0000","last_undegraded":"2024-04-24T08:11:41.359596+0000","last_fullsized":"2024-04-24T08:11:41.359596+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:24:44.359232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400073+0000","last_change":"2024-04-24T08:11:40.399681+0000","last_active":"2024-04-24T08:11:42.400073+0000","last_peered":"2024-04-24T08:11:42.400073+0000","last_clean":"2024-04-24T08:11:42.400073+0000","last_became_active":"2024-04-24T08:11:40.399332+0000","last_became_peered":"2024-04-24T08:11:40.399332+0000","last_unstale":"2024-04-24T08:11:42.400073+0000","last_undegraded":"2024-04-24T08:11:42.400073+0000","last_fullsized":"2024-04-24T08:11:42.400073+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:37:26.171195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"56'8","reported_seq":22,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400280+0000","last_change":"2024-04-24T08:11:39.384034+0000","last_active":"2024-04-24T08:11:42.400280+0000","last_peered":"2024-04-24T08:11:42.400280+0000","last_clean":"2024-04-24T08:11:42.400280+0000","last_became_active":"2024-04-24T08:11:39.383725+0000","last_became_peered":"2024-04-24T08:11:39.383725+0000","last_unstale":"2024-04-24T08:11:42.400280+0000","last_undegraded":"2024-04-24T08:11:42.400280+0000","last_fullsized":"2024-04-24T08:11:42.400280+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:54.205485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400150+0000","last_change":"2024-04-24T08:11:40.399086+0000","last_active":"2024-04-24T08:11:42.400150+0000","last_peered":"2024-04-24T08:11:42.400150+0000","last_clean":"2024-04-24T08:11:42.400150+0000","last_became_active":"2024-04-24T08:11:40.398790+0000","last_became_peered":"2024-04-24T08:11:40.398790+0000","last_unstale":"2024-04-24T08:11:42.400150+0000","last_undegraded":"2024-04-24T08:11:42.400150+0000","last_fullsized":"2024-04-24T08:11:42.400150+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:58:01.370448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401788+0000","last_change":"2024-04-24T08:11:39.372024+0000","last_active":"2024-04-24T08:11:42.401788+0000","last_peered":"2024-04-24T08:11:42.401788+0000","last_clean":"2024-04-24T08:11:42.401788+0000","last_became_active":"2024-04-24T08:11:39.371736+0000","last_became_peered":"2024-04-24T08:11:39.371736+0000","last_unstale":"2024-04-24T08:11:42.401788+0000","last_undegraded":"2024-04-24T08:11:42.401788+0000","last_fullsized":"2024-04-24T08:11:42.401788+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:20:08.860414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362779+0000","last_change":"2024-04-24T08:11:40.381743+0000","last_active":"2024-04-24T08:11:41.362779+0000","last_peered":"2024-04-24T08:11:41.362779+0000","last_clean":"2024-04-24T08:11:41.362779+0000","last_became_active":"2024-04-24T08:11:40.381437+0000","last_became_peered":"2024-04-24T08:11:40.381437+0000","last_unstale":"2024-04-24T08:11:41.362779+0000","last_undegraded":"2024-04-24T08:11:41.362779+0000","last_fullsized":"2024-04-24T08:11:41.362779+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:20:09.876703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369986+0000","last_change":"2024-04-24T08:11:39.380526+0000","last_active":"2024-04-24T08:11:41.369986+0000","last_peered":"2024-04-24T08:11:41.369986+0000","last_clean":"2024-04-24T08:11:41.369986+0000","last_became_active":"2024-04-24T08:11:39.377660+0000","last_became_peered":"2024-04-24T08:11:39.377660+0000","last_unstale":"2024-04-24T08:11:41.369986+0000","last_undegraded":"2024-04-24T08:11:41.369986+0000","last_fullsized":"2024-04-24T08:11:41.369986+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:21:43.713394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400833+0000","last_change":"2024-04-24T08:11:40.383865+0000","last_active":"2024-04-24T08:11:42.400833+0000","last_peered":"2024-04-24T08:11:42.400833+0000","last_clean":"2024-04-24T08:11:42.400833+0000","last_became_active":"2024-04-24T08:11:40.383599+0000","last_became_peered":"2024-04-24T08:11:40.383599+0000","last_unstale":"2024-04-24T08:11:42.400833+0000","last_undegraded":"2024-04-24T08:11:42.400833+0000","last_fullsized":"2024-04-24T08:11:42.400833+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:28.565615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"56'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362075+0000","last_change":"2024-04-24T08:11:39.382135+0000","last_active":"2024-04-24T08:11:41.362075+0000","last_peered":"2024-04-24T08:11:41.362075+0000","last_clean":"2024-04-24T08:11:41.362075+0000","last_became_active":"2024-04-24T08:11:39.381856+0000","last_became_peered":"2024-04-24T08:11:39.381856+0000","last_unstale":"2024-04-24T08:11:41.362075+0000","last_undegraded":"2024-04-24T08:11:41.362075+0000","last_fullsized":"2024-04-24T08:11:41.362075+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:19:57.821537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400840+0000","last_change":"2024-04-24T08:11:40.397860+0000","last_active":"2024-04-24T08:11:42.400840+0000","last_peered":"2024-04-24T08:11:42.400840+0000","last_clean":"2024-04-24T08:11:42.400840+0000","last_became_active":"2024-04-24T08:11:40.397580+0000","last_became_peered":"2024-04-24T08:11:40.397580+0000","last_unstale":"2024-04-24T08:11:42.400840+0000","last_undegraded":"2024-04-24T08:11:42.400840+0000","last_fullsized":"2024-04-24T08:11:42.400840+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:38:05.022070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369820+0000","last_change":"2024-04-24T08:11:39.368256+0000","last_active":"2024-04-24T08:11:41.369820+0000","last_peered":"2024-04-24T08:11:41.369820+0000","last_clean":"2024-04-24T08:11:41.369820+0000","last_became_active":"2024-04-24T08:11:39.367733+0000","last_became_peered":"2024-04-24T08:11:39.367733+0000","last_unstale":"2024-04-24T08:11:41.369820+0000","last_undegraded":"2024-04-24T08:11:41.369820+0000","last_fullsized":"2024-04-24T08:11:41.369820+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:14:29.131715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.360408+0000","last_change":"2024-04-24T08:11:40.376808+0000","last_active":"2024-04-24T08:11:41.360408+0000","last_peered":"2024-04-24T08:11:41.360408+0000","last_clean":"2024-04-24T08:11:41.360408+0000","last_became_active":"2024-04-24T08:11:40.376195+0000","last_became_peered":"2024-04-24T08:11:40.376195+0000","last_unstale":"2024-04-24T08:11:41.360408+0000","last_undegraded":"2024-04-24T08:11:41.360408+0000","last_fullsized":"2024-04-24T08:11:41.360408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:01:45.419044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.401068+0000","last_change":"2024-04-24T08:11:39.378864+0000","last_active":"2024-04-24T08:11:42.401068+0000","last_peered":"2024-04-24T08:11:42.401068+0000","last_clean":"2024-04-24T08:11:42.401068+0000","last_became_active":"2024-04-24T08:11:39.378185+0000","last_became_peered":"2024-04-24T08:11:39.378185+0000","last_unstale":"2024-04-24T08:11:42.401068+0000","last_undegraded":"2024-04-24T08:11:42.401068+0000","last_fullsized":"2024-04-24T08:11:42.401068+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:35:07.885184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369706+0000","last_change":"2024-04-24T08:11:40.390204+0000","last_active":"2024-04-24T08:11:41.369706+0000","last_peered":"2024-04-24T08:11:41.369706+0000","last_clean":"2024-04-24T08:11:41.369706+0000","last_became_active":"2024-04-24T08:11:40.389588+0000","last_became_peered":"2024-04-24T08:11:40.389588+0000","last_unstale":"2024-04-24T08:11:41.369706+0000","last_undegraded":"2024-04-24T08:11:41.369706+0000","last_fullsized":"2024-04-24T08:11:41.369706+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:14:29.057037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.356827+0000","last_change":"2024-04-24T08:11:39.377997+0000","last_active":"2024-04-24T08:11:41.356827+0000","last_peered":"2024-04-24T08:11:41.356827+0000","last_clean":"2024-04-24T08:11:41.356827+0000","last_became_active":"2024-04-24T08:11:39.376575+0000","last_became_peered":"2024-04-24T08:11:39.376575+0000","last_unstale":"2024-04-24T08:11:41.356827+0000","last_undegraded":"2024-04-24T08:11:41.356827+0000","last_fullsized":"2024-04-24T08:11:41.356827+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:46:49.812730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.369275+0000","last_change":"2024-04-24T08:11:40.386876+0000","last_active":"2024-04-24T08:11:41.369275+0000","last_peered":"2024-04-24T08:11:41.369275+0000","last_clean":"2024-04-24T08:11:41.369275+0000","last_became_active":"2024-04-24T08:11:40.386554+0000","last_became_peered":"2024-04-24T08:11:40.386554+0000","last_unstale":"2024-04-24T08:11:41.369275+0000","last_undegraded":"2024-04-24T08:11:41.369275+0000","last_fullsized":"2024-04-24T08:11:41.369275+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:03:27.646675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.359272+0000","last_change":"2024-04-24T08:11:39.375800+0000","last_active":"2024-04-24T08:11:41.359272+0000","last_peered":"2024-04-24T08:11:41.359272+0000","last_clean":"2024-04-24T08:11:41.359272+0000","last_became_active":"2024-04-24T08:11:39.374984+0000","last_became_peered":"2024-04-24T08:11:39.374984+0000","last_unstale":"2024-04-24T08:11:41.359272+0000","last_undegraded":"2024-04-24T08:11:41.359272+0000","last_fullsized":"2024-04-24T08:11:41.359272+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:54.205485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362700+0000","last_change":"2024-04-24T08:11:40.372341+0000","last_active":"2024-04-24T08:11:41.362700+0000","last_peered":"2024-04-24T08:11:41.362700+0000","last_clean":"2024-04-24T08:11:41.362700+0000","last_became_active":"2024-04-24T08:11:40.371277+0000","last_became_peered":"2024-04-24T08:11:40.371277+0000","last_unstale":"2024-04-24T08:11:41.362700+0000","last_undegraded":"2024-04-24T08:11:41.362700+0000","last_fullsized":"2024-04-24T08:11:41.362700+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:58:01.370448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"56'2","reported_seq":16,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.358015+0000","last_change":"2024-04-24T08:11:39.373873+0000","last_active":"2024-04-24T08:11:41.358015+0000","last_peered":"2024-04-24T08:11:41.358015+0000","last_clean":"2024-04-24T08:11:41.358015+0000","last_became_active":"2024-04-24T08:11:39.373540+0000","last_became_peered":"2024-04-24T08:11:39.373540+0000","last_unstale":"2024-04-24T08:11:41.358015+0000","last_undegraded":"2024-04-24T08:11:41.358015+0000","last_fullsized":"2024-04-24T08:11:41.358015+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:54:22.021374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.363003+0000","last_change":"2024-04-24T08:11:40.393301+0000","last_active":"2024-04-24T08:11:41.363003+0000","last_peered":"2024-04-24T08:11:41.363003+0000","last_clean":"2024-04-24T08:11:41.363003+0000","last_became_active":"2024-04-24T08:11:40.392986+0000","last_became_peered":"2024-04-24T08:11:40.392986+0000","last_unstale":"2024-04-24T08:11:41.363003+0000","last_undegraded":"2024-04-24T08:11:41.363003+0000","last_fullsized":"2024-04-24T08:11:41.363003+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:56:58.949019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"57'1","reported_seq":15,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370389+0000","last_change":"2024-04-24T08:11:39.382407+0000","last_active":"2024-04-24T08:11:41.370389+0000","last_peered":"2024-04-24T08:11:41.370389+0000","last_clean":"2024-04-24T08:11:41.370389+0000","last_became_active":"2024-04-24T08:11:39.381816+0000","last_became_peered":"2024-04-24T08:11:39.381816+0000","last_unstale":"2024-04-24T08:11:41.370389+0000","last_undegraded":"2024-04-24T08:11:41.370389+0000","last_fullsized":"2024-04-24T08:11:41.370389+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:50.036889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.357387+0000","last_change":"2024-04-24T08:11:40.388444+0000","last_active":"2024-04-24T08:11:41.357387+0000","last_peered":"2024-04-24T08:11:41.357387+0000","last_clean":"2024-04-24T08:11:41.357387+0000","last_became_active":"2024-04-24T08:11:40.387632+0000","last_became_peered":"2024-04-24T08:11:40.387632+0000","last_unstale":"2024-04-24T08:11:41.357387+0000","last_undegraded":"2024-04-24T08:11:41.357387+0000","last_fullsized":"2024-04-24T08:11:41.357387+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:53:39.597009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376690+0000","last_change":"2024-04-24T08:11:39.381391+0000","last_active":"2024-04-24T08:11:41.376690+0000","last_peered":"2024-04-24T08:11:41.376690+0000","last_clean":"2024-04-24T08:11:41.376690+0000","last_became_active":"2024-04-24T08:11:39.380979+0000","last_became_peered":"2024-04-24T08:11:39.380979+0000","last_unstale":"2024-04-24T08:11:41.376690+0000","last_undegraded":"2024-04-24T08:11:41.376690+0000","last_fullsized":"2024-04-24T08:11:41.376690+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:48:45.846889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.370359+0000","last_change":"2024-04-24T08:11:40.375316+0000","last_active":"2024-04-24T08:11:41.370359+0000","last_peered":"2024-04-24T08:11:41.370359+0000","last_clean":"2024-04-24T08:11:41.370359+0000","last_became_active":"2024-04-24T08:11:40.374563+0000","last_became_peered":"2024-04-24T08:11:40.374563+0000","last_unstale":"2024-04-24T08:11:41.370359+0000","last_undegraded":"2024-04-24T08:11:41.370359+0000","last_fullsized":"2024-04-24T08:11:41.370359+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:46:45.450068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.362453+0000","last_change":"2024-04-24T08:11:39.381316+0000","last_active":"2024-04-24T08:11:41.362453+0000","last_peered":"2024-04-24T08:11:41.362453+0000","last_clean":"2024-04-24T08:11:41.362453+0000","last_became_active":"2024-04-24T08:11:39.381027+0000","last_became_peered":"2024-04-24T08:11:39.381027+0000","last_unstale":"2024-04-24T08:11:41.362453+0000","last_undegraded":"2024-04-24T08:11:41.362453+0000","last_fullsized":"2024-04-24T08:11:41.362453+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:12:10.676575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400119+0000","last_change":"2024-04-24T08:11:40.376857+0000","last_active":"2024-04-24T08:11:42.400119+0000","last_peered":"2024-04-24T08:11:42.400119+0000","last_clean":"2024-04-24T08:11:42.400119+0000","last_became_active":"2024-04-24T08:11:40.376301+0000","last_became_peered":"2024-04-24T08:11:40.376301+0000","last_unstale":"2024-04-24T08:11:42.400119+0000","last_undegraded":"2024-04-24T08:11:42.400119+0000","last_fullsized":"2024-04-24T08:11:42.400119+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:11:06.384438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.356874+0000","last_change":"2024-04-24T08:11:39.378902+0000","last_active":"2024-04-24T08:11:41.356874+0000","last_peered":"2024-04-24T08:11:41.356874+0000","last_clean":"2024-04-24T08:11:41.356874+0000","last_became_active":"2024-04-24T08:11:39.378628+0000","last_became_peered":"2024-04-24T08:11:39.378628+0000","last_unstale":"2024-04-24T08:11:41.356874+0000","last_undegraded":"2024-04-24T08:11:41.356874+0000","last_fullsized":"2024-04-24T08:11:41.356874+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:09.889758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400921+0000","last_change":"2024-04-24T08:11:40.397358+0000","last_active":"2024-04-24T08:11:42.400921+0000","last_peered":"2024-04-24T08:11:42.400921+0000","last_clean":"2024-04-24T08:11:42.400921+0000","last_became_active":"2024-04-24T08:11:40.396322+0000","last_became_peered":"2024-04-24T08:11:40.396322+0000","last_unstale":"2024-04-24T08:11:42.400921+0000","last_undegraded":"2024-04-24T08:11:42.400921+0000","last_fullsized":"2024-04-24T08:11:42.400921+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:02.097340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.361523+0000","last_change":"2024-04-24T08:11:39.376794+0000","last_active":"2024-04-24T08:11:41.361523+0000","last_peered":"2024-04-24T08:11:41.361523+0000","last_clean":"2024-04-24T08:11:41.361523+0000","last_became_active":"2024-04-24T08:11:39.376266+0000","last_became_peered":"2024-04-24T08:11:39.376266+0000","last_unstale":"2024-04-24T08:11:41.361523+0000","last_undegraded":"2024-04-24T08:11:41.361523+0000","last_fullsized":"2024-04-24T08:11:41.361523+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:28:54.205485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.356856+0000","last_change":"2024-04-24T08:11:40.386891+0000","last_active":"2024-04-24T08:11:41.356856+0000","last_peered":"2024-04-24T08:11:41.356856+0000","last_clean":"2024-04-24T08:11:41.356856+0000","last_became_active":"2024-04-24T08:11:40.386314+0000","last_became_peered":"2024-04-24T08:11:40.386314+0000","last_unstale":"2024-04-24T08:11:41.356856+0000","last_undegraded":"2024-04-24T08:11:41.356856+0000","last_fullsized":"2024-04-24T08:11:41.356856+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:39:19.795220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.356796+0000","last_change":"2024-04-24T08:11:39.380077+0000","last_active":"2024-04-24T08:11:41.356796+0000","last_peered":"2024-04-24T08:11:41.356796+0000","last_clean":"2024-04-24T08:11:41.356796+0000","last_became_active":"2024-04-24T08:11:39.379757+0000","last_became_peered":"2024-04-24T08:11:39.379757+0000","last_unstale":"2024-04-24T08:11:41.356796+0000","last_undegraded":"2024-04-24T08:11:41.356796+0000","last_fullsized":"2024-04-24T08:11:41.356796+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:23:32.037944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400616+0000","last_change":"2024-04-24T08:11:40.390918+0000","last_active":"2024-04-24T08:11:42.400616+0000","last_peered":"2024-04-24T08:11:42.400616+0000","last_clean":"2024-04-24T08:11:42.400616+0000","last_became_active":"2024-04-24T08:11:40.390620+0000","last_became_peered":"2024-04-24T08:11:40.390620+0000","last_unstale":"2024-04-24T08:11:42.400616+0000","last_undegraded":"2024-04-24T08:11:42.400616+0000","last_fullsized":"2024-04-24T08:11:42.400616+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:31.287452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399790+0000","last_change":"2024-04-24T08:11:39.377388+0000","last_active":"2024-04-24T08:11:42.399790+0000","last_peered":"2024-04-24T08:11:42.399790+0000","last_clean":"2024-04-24T08:11:42.399790+0000","last_became_active":"2024-04-24T08:11:39.376962+0000","last_became_peered":"2024-04-24T08:11:39.376962+0000","last_unstale":"2024-04-24T08:11:42.399790+0000","last_undegraded":"2024-04-24T08:11:42.399790+0000","last_fullsized":"2024-04-24T08:11:42.399790+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:01.081051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":12,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.400827+0000","last_change":"2024-04-24T08:11:40.371825+0000","last_active":"2024-04-24T08:11:42.400827+0000","last_peered":"2024-04-24T08:11:42.400827+0000","last_clean":"2024-04-24T08:11:42.400827+0000","last_became_active":"2024-04-24T08:11:40.371499+0000","last_became_peered":"2024-04-24T08:11:40.371499+0000","last_unstale":"2024-04-24T08:11:42.400827+0000","last_undegraded":"2024-04-24T08:11:42.400827+0000","last_fullsized":"2024-04-24T08:11:42.400827+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:39.332154+0000","last_clean_scrub_stamp":"2024-04-24T08:11:39.332154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:49.898857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":14,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:41.376456+0000","last_change":"2024-04-24T08:11:39.384160+0000","last_active":"2024-04-24T08:11:41.376456+0000","last_peered":"2024-04-24T08:11:41.376456+0000","last_clean":"2024-04-24T08:11:41.376456+0000","last_became_active":"2024-04-24T08:11:39.383839+0000","last_became_peered":"2024-04-24T08:11:39.383839+0000","last_unstale":"2024-04-24T08:11:41.376456+0000","last_undegraded":"2024-04-24T08:11:41.376456+0000","last_fullsized":"2024-04-24T08:11:41.376456+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:11:38.315866+0000","last_clean_scrub_stamp":"2024-04-24T08:11:38.315866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:37:55.284801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":236,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-24T08:11:42.399625+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:11:42.399625+0000","last_peered":"2024-04-24T08:11:42.399625+0000","last_clean":"2024-04-24T08:11:42.399625+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:11:42.399625+0000","last_undegraded":"2024-04-24T08:11:42.399625+0000","last_fullsized":"2024-04-24T08:11:42.399625+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1286144,"data_stored":875840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574896,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":684032,"data_stored":285312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":692224,"data_stored":285964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":696320,"data_stored":286924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574894,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":692224,"data_stored":285368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574895,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27696,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":692224,"data_stored":285368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":688128,"data_stored":285402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574894,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32448,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339392,"statfs":{"total":96636764160,"available":96603537408,"internally_reserved":0,"allocated":1294336,"data_stored":877874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:11:45.384 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T08:11:45.384 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-24T08:11:45.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.629+0000 7f4233a00640 1 Processor -- start 2024-04-24T08:11:45.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.629+0000 7f4233a00640 1 -- start start 2024-04-24T08:11:45.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.630+0000 7f4233a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f422c0cd8e0 0x7f422c06e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.631+0000 7f4233a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f422c0d0a10 0x7f422c145e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.631+0000 7f4233a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f422c0cf610 con 0x7f422c145a20 2024-04-24T08:11:45.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.631+0000 7f4233a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f422c0cf8f0 con 0x7f422c0d0a10 2024-04-24T08:11:45.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.631+0000 7f4233a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f422c0cfbd0 con 0x7f422c0cd8e0 2024-04-24T08:11:45.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.631+0000 7f4231a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f422c0d0a10 0x7f422c145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.631+0000 7f4231a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f422c0d0a10 0x7f422c145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47444/0 (socket says 172.21.15.69:47444) 2024-04-24T08:11:45.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.631+0000 7f4231a00640 1 -- 172.21.15.69:0/4115002028 learned_addr learned my addr 172.21.15.69:0/4115002028 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:45.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.631+0000 7f4231000640 1 --2- 172.21.15.69:0/4115002028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f422c0cd8e0 0x7f422c06e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.631+0000 7f4231a00640 1 -- 172.21.15.69:0/4115002028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f422c0cd8e0 msgr2=0x7f422c06e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:45.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.631+0000 7f4231a00640 1 --2- 172.21.15.69:0/4115002028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f422c0cd8e0 0x7f422c06e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.632+0000 7f4231a00640 1 -- 172.21.15.69:0/4115002028 >> v1:172.21.15.121:6789/0 conn(0x7f422c145a20 legacy=0x7f422c0d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:45.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.632+0000 7f4231a00640 1 -- 172.21.15.69:0/4115002028 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422c0cfeb0 con 0x7f422c0d0a10 2024-04-24T08:11:45.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.633+0000 7f4231a00640 1 --2- 172.21.15.69:0/4115002028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c0d0a10 0x7f422c145e80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4224009e50 tx=0x7f422402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bd149da434a9742 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.633+0000 7f422be00640 1 -- 172.21.15.69:0/4115002028 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4224036070 con 0x7f422c0d0a10 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.633+0000 7f422be00640 1 -- 172.21.15.69:0/4115002028 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4224031600 con 0x7f422c0d0a10 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.633+0000 7f422be00640 1 -- 172.21.15.69:0/4115002028 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42240319d0 con 0x7f422c0d0a10 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.633+0000 7f4233a00640 1 -- 172.21.15.69:0/4115002028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c0d0a10 msgr2=0x7f422c145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.633+0000 7f4233a00640 1 --2- 172.21.15.69:0/4115002028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c0d0a10 0x7f422c145e80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f4224009e50 tx=0x7f422402a6c0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.633+0000 7f4233a00640 1 -- 172.21.15.69:0/4115002028 shutdown_connections 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.633+0000 7f4233a00640 1 --2- 172.21.15.69:0/4115002028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c0d0a10 0x7f422c145e80 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.633+0000 7f4233a00640 1 --2- 172.21.15.69:0/4115002028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f422c0cd8e0 0x7f422c06e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.633+0000 7f4233a00640 1 -- 172.21.15.69:0/4115002028 >> 172.21.15.69:0/4115002028 conn(0x7f422c06ba20 msgr2=0x7f422c06be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.634+0000 7f4233a00640 1 -- 172.21.15.69:0/4115002028 shutdown_connections 2024-04-24T08:11:45.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.634+0000 7f4233a00640 1 -- 172.21.15.69:0/4115002028 wait complete. 2024-04-24T08:11:45.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.635+0000 7f4233a00640 1 Processor -- start 2024-04-24T08:11:45.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.635+0000 7f4233a00640 1 -- start start 2024-04-24T08:11:45.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.636+0000 7f4233a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f422c0cd8e0 0x7f422c0c5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.636+0000 7f4233a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f422c0d0a10 0x7f422c15f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.636+0000 7f4233a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c145a20 0x7f422c0c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.636+0000 7f4233a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f422c0cf230 con 0x7f422c145a20 2024-04-24T08:11:45.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.636+0000 7f4233a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f422c0cdcc0 con 0x7f422c0cd8e0 2024-04-24T08:11:45.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.636+0000 7f4233a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f422c0cf3b0 con 0x7f422c0d0a10 2024-04-24T08:11:45.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.636+0000 7f4231a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c145a20 0x7f422c0c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.636+0000 7f4231a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c145a20 0x7f422c0c6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47448/0 (socket says 172.21.15.69:47448) 2024-04-24T08:11:45.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.637+0000 7f4231a00640 1 -- 172.21.15.69:0/4151814274 learned_addr learned my addr 172.21.15.69:0/4151814274 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:45.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.637+0000 7f4231000640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f422c0cd8e0 0x7f422c0c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.637+0000 7f4223e00640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f422c0d0a10 0x7f422c15f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.637+0000 7f4231a00640 1 -- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f422c0d0a10 msgr2=0x7f422c15f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.637+0000 7f4231a00640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f422c0d0a10 0x7f422c15f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.637+0000 7f4231a00640 1 -- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f422c0cd8e0 msgr2=0x7f422c0c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.637+0000 7f4231a00640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f422c0cd8e0 0x7f422c0c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.637+0000 7f4231a00640 1 -- 172.21.15.69:0/4151814274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422c15fdb0 con 0x7f422c145a20 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.637+0000 7f4231a00640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c145a20 0x7f422c0c6a60 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4224009e50 tx=0x7f4224004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.637+0000 7f4229600640 1 -- 172.21.15.69:0/4151814274 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4224036050 con 0x7f422c145a20 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.638+0000 7f4229600640 1 -- 172.21.15.69:0/4151814274 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4224042070 con 0x7f422c145a20 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.638+0000 7f4229600640 1 -- 172.21.15.69:0/4151814274 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4224009510 con 0x7f422c145a20 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.638+0000 7f4233a00640 1 -- 172.21.15.69:0/4151814274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f422c0c59c0 con 0x7f422c145a20 2024-04-24T08:11:45.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.638+0000 7f4233a00640 1 -- 172.21.15.69:0/4151814274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f422c0c5f80 con 0x7f422c145a20 2024-04-24T08:11:45.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.639+0000 7f4233a00640 1 -- 172.21.15.69:0/4151814274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f422c075e30 con 0x7f422c145a20 2024-04-24T08:11:45.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.640+0000 7f4229600640 1 -- 172.21.15.69:0/4151814274 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4224007b90 con 0x7f422c145a20 2024-04-24T08:11:45.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.641+0000 7f4229600640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f420806c8f0 0x7f420806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:45.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.641+0000 7f4229600640 1 -- 172.21.15.69:0/4151814274 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f42240aca70 con 0x7f422c145a20 2024-04-24T08:11:45.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.642+0000 7f4231000640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f420806c8f0 0x7f420806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:45.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.643+0000 7f4231000640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f420806c8f0 0x7f420806edb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f421c006fd0 tx=0x7f421c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:45.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.646+0000 7f4229600640 1 -- 172.21.15.69:0/4151814274 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f422407ae00 con 0x7f422c145a20 2024-04-24T08:11:45.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.899+0000 7f4233a00640 1 -- 172.21.15.69:0/4151814274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f422c0d1050 con 0x7f422c145a20 2024-04-24T08:11:45.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.903+0000 7f4229600640 1 -- 172.21.15.69:0/4151814274 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7f422407a700 con 0x7f422c145a20 2024-04-24T08:11:45.903 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-24T08:11:45.904 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw== 2024-04-24T08:11:45.904 INFO:teuthology.orchestra.run.smithi069.stdout: caps mds = "allow" 2024-04-24T08:11:45.904 INFO:teuthology.orchestra.run.smithi069.stdout: caps mon = "allow r" 2024-04-24T08:11:45.904 INFO:teuthology.orchestra.run.smithi069.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-24T08:11:45.904 INFO:teuthology.orchestra.run.smithi069.stderr:updated caps for client.0 2024-04-24T08:11:45.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.905+0000 7f4233a00640 1 -- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f420806c8f0 msgr2=0x7f420806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.905+0000 7f4233a00640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f420806c8f0 0x7f420806edb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f421c006fd0 tx=0x7f421c008040 comp rx=0 tx=0).stop 2024-04-24T08:11:45.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.905+0000 7f4233a00640 1 -- 172.21.15.69:0/4151814274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c145a20 msgr2=0x7f422c0c6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:45.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.905+0000 7f4233a00640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c145a20 0x7f422c0c6a60 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4224009e50 tx=0x7f4224004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:45.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.906+0000 7f4233a00640 1 -- 172.21.15.69:0/4151814274 shutdown_connections 2024-04-24T08:11:45.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.906+0000 7f4233a00640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f420806c8f0 0x7f420806edb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.906+0000 7f4233a00640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f422c145a20 0x7f422c0c6a60 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.906+0000 7f4233a00640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f422c0d0a10 0x7f422c15f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.906+0000 7f4233a00640 1 --2- 172.21.15.69:0/4151814274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f422c0cd8e0 0x7f422c0c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.906+0000 7f4233a00640 1 -- 172.21.15.69:0/4151814274 >> 172.21.15.69:0/4151814274 conn(0x7f422c06ba20 msgr2=0x7f422c0aff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.906+0000 7f4233a00640 1 -- 172.21.15.69:0/4151814274 shutdown_connections 2024-04-24T08:11:45.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:45.906+0000 7f4233a00640 1 -- 172.21.15.69:0/4151814274 wait complete. 2024-04-24T08:11:45.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:46.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.121+0000 7f9aab200640 1 Processor -- start 2024-04-24T08:11:46.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.122+0000 7f9aab200640 1 -- start start 2024-04-24T08:11:46.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.123+0000 7f9aab200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aac15e960 0x7f9aac162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:46.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.123+0000 7f9aab200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9aac162980 0x7f9aac06cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:46.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.123+0000 7f9aab200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9aac06d4f0 0x7f9aac0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:46.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.123+0000 7f9aab200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9aac15ed40 con 0x7f9aac06d4f0 2024-04-24T08:11:46.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.123+0000 7f9aab200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9aac0cf050 con 0x7f9aac162980 2024-04-24T08:11:46.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.123+0000 7f9aab200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9aac162e20 con 0x7f9aac15e960 2024-04-24T08:11:46.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.123+0000 7f9aaa800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9aac06d4f0 0x7f9aac0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:46.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.123+0000 7f9aaa800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9aac06d4f0 0x7f9aac0cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47452/0 (socket says 172.21.15.69:47452) 2024-04-24T08:11:46.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.123+0000 7f9aaa800640 1 -- 172.21.15.69:0/3754848105 learned_addr learned my addr 172.21.15.69:0/3754848105 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:46.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.124+0000 7f9aa1400640 1 --2- 172.21.15.69:0/3754848105 >> v2:172.21.15.121:3300/0 conn(0x7f9aac162980 0x7f9aac06cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:46.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.124+0000 7f9aaa800640 1 -- 172.21.15.69:0/3754848105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aac15e960 msgr2=0x7f9aac162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:46.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.124+0000 7f9aaa800640 1 --2- 172.21.15.69:0/3754848105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aac15e960 0x7f9aac162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.124+0000 7f9aaa800640 1 -- 172.21.15.69:0/3754848105 >> v2:172.21.15.121:3300/0 conn(0x7f9aac162980 msgr2=0x7f9aac06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:46.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.124+0000 7f9aaa800640 1 --2- 172.21.15.69:0/3754848105 >> v2:172.21.15.121:3300/0 conn(0x7f9aac162980 0x7f9aac06cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.124+0000 7f9aaa800640 1 -- 172.21.15.69:0/3754848105 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aac163100 con 0x7f9aac06d4f0 2024-04-24T08:11:46.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.125+0000 7f9aaa800640 1 --2- 172.21.15.69:0/3754848105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac06d4f0 0x7f9aac0cfaf0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9aac09e030 tx=0x7f9aa402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b8ad72902468b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.125+0000 7f9aa9400640 1 -- 172.21.15.69:0/3754848105 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4036070 con 0x7f9aac06d4f0 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.125+0000 7f9aa9400640 1 -- 172.21.15.69:0/3754848105 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa4031600 con 0x7f9aac06d4f0 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.125+0000 7f9aa9400640 1 -- 172.21.15.69:0/3754848105 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa40319d0 con 0x7f9aac06d4f0 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.126+0000 7f9aab200640 1 -- 172.21.15.69:0/3754848105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac06d4f0 msgr2=0x7f9aac0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.126+0000 7f9aab200640 1 --2- 172.21.15.69:0/3754848105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac06d4f0 0x7f9aac0cfaf0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9aac09e030 tx=0x7f9aa402a6a0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.126+0000 7f9aab200640 1 -- 172.21.15.69:0/3754848105 shutdown_connections 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.126+0000 7f9aab200640 1 --2- 172.21.15.69:0/3754848105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac06d4f0 0x7f9aac0cfaf0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.126+0000 7f9aab200640 1 --2- 172.21.15.69:0/3754848105 >> v2:172.21.15.121:3300/0 conn(0x7f9aac162980 0x7f9aac06cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.126+0000 7f9aab200640 1 --2- 172.21.15.69:0/3754848105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aac15e960 0x7f9aac162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.126+0000 7f9aab200640 1 -- 172.21.15.69:0/3754848105 >> 172.21.15.69:0/3754848105 conn(0x7f9aac09d900 msgr2=0x7f9aac08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.126+0000 7f9aab200640 1 -- 172.21.15.69:0/3754848105 shutdown_connections 2024-04-24T08:11:46.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.126+0000 7f9aab200640 1 -- 172.21.15.69:0/3754848105 wait complete. 2024-04-24T08:11:46.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.128+0000 7f9aab200640 1 Processor -- start 2024-04-24T08:11:46.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.128+0000 7f9aab200640 1 -- start start 2024-04-24T08:11:46.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.129+0000 7f9aab200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aac06d4f0 0x7f9aac0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:46.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.129+0000 7f9aab200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac15e960 0x7f9aac0b7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:46.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.129+0000 7f9aab200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aac162980 0x7f9aac0cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:46.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.129+0000 7f9aab200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac0b6170 con 0x7f9aac15e960 2024-04-24T08:11:46.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.129+0000 7f9aab200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9aac0cf5d0 con 0x7f9aac162980 2024-04-24T08:11:46.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.129+0000 7f9aab200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9aac0b62f0 con 0x7f9aac06d4f0 2024-04-24T08:11:46.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.129+0000 7f9aa1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac15e960 0x7f9aac0b7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:46.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.129+0000 7f9aa1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac15e960 0x7f9aac0b7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47464/0 (socket says 172.21.15.69:47464) 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.129+0000 7f9aa1400640 1 -- 172.21.15.69:0/3946763824 learned_addr learned my addr 172.21.15.69:0/3946763824 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aaa800640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aac162980 0x7f9aac0cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa1400640 1 -- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aac06d4f0 msgr2=0x7f9aac0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa9e00640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aac06d4f0 0x7f9aac0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa1400640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aac06d4f0 0x7f9aac0b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa1400640 1 -- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aac162980 msgr2=0x7f9aac0cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa1400640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aac162980 0x7f9aac0cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa1400640 1 -- 172.21.15.69:0/3946763824 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aac0cc500 con 0x7f9aac15e960 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aaa800640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aac162980 0x7f9aac0cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa9e00640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aac06d4f0 0x7f9aac0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa1400640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac15e960 0x7f9aac0b7380 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400c820 tx=0x7f9a9400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa2a00640 1 -- 172.21.15.69:0/3946763824 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94015070 con 0x7f9aac15e960 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa2a00640 1 -- 172.21.15.69:0/3946763824 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a94004020 con 0x7f9aac15e960 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.130+0000 7f9aa2a00640 1 -- 172.21.15.69:0/3946763824 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a940043f0 con 0x7f9aac15e960 2024-04-24T08:11:46.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.131+0000 7f9aab200640 1 -- 172.21.15.69:0/3946763824 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aac0ccdd0 con 0x7f9aac15e960 2024-04-24T08:11:46.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.131+0000 7f9aab200640 1 -- 172.21.15.69:0/3946763824 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aac0cd390 con 0x7f9aac15e960 2024-04-24T08:11:46.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.132+0000 7f9aab200640 1 -- 172.21.15.69:0/3946763824 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aac075d50 con 0x7f9aac15e960 2024-04-24T08:11:46.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.133+0000 7f9aa2a00640 1 -- 172.21.15.69:0/3946763824 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a94007830 con 0x7f9aac15e960 2024-04-24T08:11:46.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.134+0000 7f9aa2a00640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a8806c920 0x7f9a8806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:46.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.134+0000 7f9aa2a00640 1 -- 172.21.15.69:0/3946763824 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f9a9408b9b0 con 0x7f9aac15e960 2024-04-24T08:11:46.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.134+0000 7f9aa9e00640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a8806c920 0x7f9a8806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:46.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.136+0000 7f9aa9e00640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a8806c920 0x7f9a8806ede0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c006fd0 tx=0x7f9a9c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:46.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.138+0000 7f9aa2a00640 1 -- 172.21.15.69:0/3946763824 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a94059d40 con 0x7f9aac15e960 2024-04-24T08:11:46.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.382+0000 7f9aab200640 1 -- 172.21.15.69:0/3946763824 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9aac0d1dc0 con 0x7f9aac15e960 2024-04-24T08:11:46.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.383+0000 7f9aa2a00640 1 -- 172.21.15.69:0/3946763824 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+5187 (secure 0 0 0) 0x7f9a94059640 con 0x7f9aac15e960 2024-04-24T08:11:46.384 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:46.384 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4034001365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4034001365},{"type":"v1","addr":"172.21.15.121:6835","nonce":4034001365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3907163219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3907163219},{"type":"v1","addr":"172.21.15.121:6837","nonce":3907163219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2756395689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2756395689},{"type":"v1","addr":"172.21.15.69:6834","nonce":2756395689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:11:39.824842+0000","modified":"2024-04-24T08:11:42.963374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/2288057943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":2288057943},{"type":"v1","addr":"172.21.15.69:6835","nonce":2288057943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6231,"qdb_cluster":[6231]},"id":4}]} 2024-04-24T08:11:46.384 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 33 2024-04-24T08:11:46.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 -- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a8806c920 msgr2=0x7f9a8806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:46.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a8806c920 0x7f9a8806ede0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c006fd0 tx=0x7f9a9c008040 comp rx=0 tx=0).stop 2024-04-24T08:11:46.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 -- 172.21.15.69:0/3946763824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac15e960 msgr2=0x7f9aac0b7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac15e960 0x7f9aac0b7380 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f9a9400c820 tx=0x7f9a9400cce0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 -- 172.21.15.69:0/3946763824 shutdown_connections 2024-04-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a8806c920 0x7f9a8806ede0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9aac15e960 0x7f9aac0b7380 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9aac162980 0x7f9aac0cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 --2- 172.21.15.69:0/3946763824 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9aac06d4f0 0x7f9aac0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 -- 172.21.15.69:0/3946763824 >> 172.21.15.69:0/3946763824 conn(0x7f9aac09d900 msgr2=0x7f9aac0cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 -- 172.21.15.69:0/3946763824 shutdown_connections 2024-04-24T08:11:46.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:46.386+0000 7f9aab200640 1 -- 172.21.15.69:0/3946763824 wait complete. 2024-04-24T08:11:46.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:11:39.824842+0000', 'modified': '2024-04-24T08:11:42.963374+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6231}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6231': {'gid': 6231, 'name': 'c', 'rank': 0, 'incarnation': 30, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.69:6835/2288057943', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6833', 'nonce': 2288057943}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 2288057943}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6231, 'qdb_cluster': [6231]} 2024-04-24T08:11:46.807 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:11:46.807 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-24T08:11:47.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.013+0000 7f006e800640 1 Processor -- start 2024-04-24T08:11:47.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.013+0000 7f006e800640 1 -- start start 2024-04-24T08:11:47.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.014+0000 7f006e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0068145fe0 0x7f006806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.014+0000 7f006e800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00680cf3d0 con 0x7f00680d0300 2024-04-24T08:11:47.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.014+0000 7f006e800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00680cf6b0 con 0x7f0068145a90 2024-04-24T08:11:47.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.014+0000 7f006e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f00680cf990 con 0x7f0068145fe0 2024-04-24T08:11:47.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.015+0000 7f0067400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f00680d0300 0x7f00680d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:45772/0 (socket says 172.21.15.69:45772) 2024-04-24T08:11:47.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.015+0000 7f0067400640 1 -- 172.21.15.69:0/538331441 learned_addr learned my addr 172.21.15.69:0/538331441 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:47.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.015+0000 7f0067e00640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0068145fe0 0x7f006806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.015+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3543247716 0 0) 0x7f00680cf3d0 con 0x7f00680d0300 2024-04-24T08:11:47.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.015+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f004c003520 con 0x7f00680d0300 2024-04-24T08:11:47.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.015+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2300829539 0 0) 0x7f004c003520 con 0x7f00680d0300 2024-04-24T08:11:47.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0068145fe0 msgr2=0x7f006806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0068145fe0 0x7f006806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 >> v1:172.21.15.121:6789/0 conn(0x7f0068145a90 legacy=0x7f0068160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00680cfc70 con 0x7f00680d0300 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f0058001670 con 0x7f00680d0300 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 >> v1:172.21.15.69:6789/0 conn(0x7f00680d0300 legacy=0x7f00680d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f004c0039e0 0x7f004c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f004c004800 0x7f004c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0068145fe0 0x7f004c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00680cf990 con 0x7f004c004800 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f004c00baa0 con 0x7f004c0039e0 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f004c00bca0 con 0x7f0068145fe0 2024-04-24T08:11:47.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f005ea00640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f004c004800 0x7f004c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f0067e00640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0068145fe0 0x7f004c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f005ea00640 1 -- 172.21.15.69:0/538331441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0068145fe0 msgr2=0x7f004c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f005ea00640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0068145fe0 0x7f004c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f005ea00640 1 -- 172.21.15.69:0/538331441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f004c0039e0 msgr2=0x7f004c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f005ea00640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f004c0039e0 0x7f004c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.016+0000 7f005ea00640 1 -- 172.21.15.69:0/538331441 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004c003520 con 0x7f004c004800 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.017+0000 7f005ea00640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f004c004800 0x7f004c004c80 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0050001720 tx=0x7f004c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dbda98f469d039 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.018+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050001370 con 0x7f004c004800 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.018+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00500052d0 con 0x7f004c004800 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.018+0000 7f0066a00640 1 -- 172.21.15.69:0/538331441 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005002ab90 con 0x7f004c004800 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.018+0000 7f006e800640 1 -- 172.21.15.69:0/538331441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f004c004800 msgr2=0x7f004c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.018+0000 7f006e800640 1 --2- 172.21.15.69:0/538331441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f004c004800 0x7f004c004c80 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0050001720 tx=0x7f004c005830 comp rx=0 tx=0).stop 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.018+0000 7f0067e00640 1 -- 172.21.15.69:0/538331441 reap_dead start 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.018+0000 7f006e800640 1 -- 172.21.15.69:0/538331441 shutdown_connections 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.018+0000 7f006e800640 1 -- 172.21.15.69:0/538331441 >> 172.21.15.69:0/538331441 conn(0x7f0068067d90 msgr2=0x7f006806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.018+0000 7f006e800640 1 -- 172.21.15.69:0/538331441 shutdown_connections 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.019+0000 7f006e800640 1 -- 172.21.15.69:0/538331441 wait complete. 2024-04-24T08:11:47.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.020+0000 7f006e800640 1 Processor -- start 2024-04-24T08:11:47.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.020+0000 7f006e800640 1 -- start start 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.021+0000 7f006e800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f004c004800 0x7f006815c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.021+0000 7f006e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00680cb210 0x7f00680b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.021+0000 7f006e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00680cb690 0x7f00680b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.021+0000 7f006e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00681458f0 con 0x7f00680cb210 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.021+0000 7f006e800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0068145770 con 0x7f004c004800 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.021+0000 7f006e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f00680cdc30 con 0x7f00680cb690 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.022+0000 7f005ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00680cb210 0x7f00680b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.022+0000 7f005ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00680cb210 0x7f00680b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47478/0 (socket says 172.21.15.69:47478) 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.022+0000 7f005ea00640 1 -- 172.21.15.69:0/839429006 learned_addr learned my addr 172.21.15.69:0/839429006 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.022+0000 7f0067e00640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00680cb690 0x7f00680b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.022+0000 7f0067400640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f004c004800 0x7f006815c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.022+0000 7f005ea00640 1 -- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00680cb690 msgr2=0x7f00680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.022+0000 7f005ea00640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00680cb690 0x7f00680b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.022+0000 7f005ea00640 1 -- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f004c004800 msgr2=0x7f006815c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.022+0000 7f005ea00640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f004c004800 0x7f006815c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.022+0000 7f005ea00640 1 -- 172.21.15.69:0/839429006 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00680cbc40 con 0x7f00680cb210 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.023+0000 7f005ea00640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00680cb210 0x7f00680b16a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0050001840 tx=0x7f0050001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.023+0000 7f005fe00640 1 -- 172.21.15.69:0/839429006 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050040020 con 0x7f00680cb210 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.023+0000 7f005fe00640 1 -- 172.21.15.69:0/839429006 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0050007680 con 0x7f00680cb210 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.023+0000 7f005fe00640 1 -- 172.21.15.69:0/839429006 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050001370 con 0x7f00680cb210 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.023+0000 7f006e800640 1 -- 172.21.15.69:0/839429006 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00680cbf20 con 0x7f00680cb210 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.023+0000 7f006e800640 1 -- 172.21.15.69:0/839429006 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0068143c70 con 0x7f00680cb210 2024-04-24T08:11:47.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.025+0000 7f005fe00640 1 -- 172.21.15.69:0/839429006 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0050043020 con 0x7f00680cb210 2024-04-24T08:11:47.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.026+0000 7f005fe00640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004406c920 0x7f004406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.026+0000 7f005fe00640 1 -- 172.21.15.69:0/839429006 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f00500abec0 con 0x7f00680cb210 2024-04-24T08:11:47.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.027+0000 7f0067400640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004406c920 0x7f004406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.027+0000 7f0071587740 1 Processor -- start 2024-04-24T08:11:47.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.027+0000 7f0071587740 1 -- start start 2024-04-24T08:11:47.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f0071587740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b54e5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f0067400640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004406c920 0x7f004406ede0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f00580079b0 tx=0x7f0058009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:47.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f0071587740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b55158d0 0x55d8b5514c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f0071587740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b5515190 0x55d8b54e9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f0071587740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d8b544fb10 con 0x55d8b55327b0 2024-04-24T08:11:47.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f0071587740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d8b53f62b0 con 0x55d8b55158d0 2024-04-24T08:11:47.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f0071587740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55d8b544bad0 con 0x55d8b5515190 2024-04-24T08:11:47.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f0067400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b54e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f0067400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b54e5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47482/0 (socket says 172.21.15.69:47482) 2024-04-24T08:11:47.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f0067400640 1 -- 172.21.15.69:0/1681303606 learned_addr learned my addr 172.21.15.69:0/1681303606 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:47.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.028+0000 7f005ea00640 1 --2- 172.21.15.69:0/1681303606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b55158d0 0x55d8b5514c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.029+0000 7f0067400640 1 -- 172.21.15.69:0/1681303606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b5515190 msgr2=0x55d8b54e9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.029+0000 7f0067400640 1 --2- 172.21.15.69:0/1681303606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b5515190 0x55d8b54e9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.029+0000 7f0067400640 1 -- 172.21.15.69:0/1681303606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b55158d0 msgr2=0x55d8b5514c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.029+0000 7f0067400640 1 --2- 172.21.15.69:0/1681303606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b55158d0 0x55d8b5514c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.029+0000 7f0067400640 1 -- 172.21.15.69:0/1681303606 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8b51622d0 con 0x55d8b55327b0 2024-04-24T08:11:47.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.029+0000 7f0067400640 1 --2- 172.21.15.69:0/1681303606 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b54e5140 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f00580074f0 tx=0x7f0058010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ad69f8495c3e54 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:47.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.029+0000 7f0057e00640 1 -- 172.21.15.69:0/1681303606 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0058020450 con 0x55d8b55327b0 2024-04-24T08:11:47.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.029+0000 7f0057e00640 1 -- 172.21.15.69:0/1681303606 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00580205f0 con 0x55d8b55327b0 2024-04-24T08:11:47.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.029+0000 7f0057e00640 1 -- 172.21.15.69:0/1681303606 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00580209a0 con 0x55d8b55327b0 2024-04-24T08:11:47.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.030+0000 7f0071587740 1 -- 172.21.15.69:0/1681303606 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 msgr2=0x55d8b54e5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.030+0000 7f0071587740 1 --2- 172.21.15.69:0/1681303606 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b54e5140 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f00580074f0 tx=0x7f0058010000 comp rx=0 tx=0).stop 2024-04-24T08:11:47.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.030+0000 7f0071587740 1 -- 172.21.15.69:0/1681303606 shutdown_connections 2024-04-24T08:11:47.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.030+0000 7f0071587740 1 --2- 172.21.15.69:0/1681303606 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b5515190 0x55d8b54e9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.030+0000 7f0071587740 1 --2- 172.21.15.69:0/1681303606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b55158d0 0x55d8b5514c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.030+0000 7f0071587740 1 --2- 172.21.15.69:0/1681303606 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b54e5140 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.030+0000 7f0071587740 1 -- 172.21.15.69:0/1681303606 >> 172.21.15.69:0/1681303606 conn(0x55d8b54e73b0 msgr2=0x55d8b552fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.030+0000 7f0071587740 1 -- 172.21.15.69:0/1681303606 shutdown_connections 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.030+0000 7f0071587740 1 -- 172.21.15.69:0/1681303606 wait complete. 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.031+0000 7f0071587740 1 -- start start 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.031+0000 7f0071587740 1 Processor -- start 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.032+0000 7f0071587740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55158d0 0x55d8b5510510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.032+0000 7f0071587740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55327b0 0x55d8b5543cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.032+0000 7f0071587740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b5517ee0 0x55d8b553fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.032+0000 7f0071587740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d8b54e9560 con 0x55d8b55158d0 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.032+0000 7f0071587740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d8b5409590 con 0x55d8b5517ee0 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.032+0000 7f0071587740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55d8b54e8950 con 0x55d8b55327b0 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.032+0000 7f0071587740 10 client.0 _ll_register_callbacks cb 0x55d8b53ec3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.032+0000 7f0067400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55158d0 0x55d8b5510510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.032+0000 7f0067400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55158d0 0x55d8b5510510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47488/0 (socket says 172.21.15.69:47488) 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.032+0000 7f0067400640 1 -- 172.21.15.69:0/731919755 learned_addr learned my addr 172.21.15.69:0/731919755 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0067400640 1 -- 172.21.15.69:0/731919755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55327b0 msgr2=0x55d8b5543cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0067400640 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55327b0 0x55d8b5543cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0067400640 1 -- 172.21.15.69:0/731919755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b5517ee0 msgr2=0x55d8b553fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0067400640 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b5517ee0 0x55d8b553fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0067400640 1 -- 172.21.15.69:0/731919755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8b522c6b0 con 0x55d8b55158d0 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0067400640 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55158d0 0x55d8b5510510 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0058009010 tx=0x7f0058007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bcb49ccefc598f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0054c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0054c00640 1 -- 172.21.15.69:0/731919755 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0058020c70 con 0x55d8b55158d0 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0054c00640 1 -- 172.21.15.69:0/731919755 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f0058012410 con 0x55d8b55158d0 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0071587740 1 -- 172.21.15.69:0/731919755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f00580205f0 con 0x55d8b55158d0 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0054c00640 1 -- 172.21.15.69:0/731919755 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0058033050 con 0x55d8b55158d0 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0071587740 10 client.6369 fetch_fsmap learned FSMap version 33 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.033+0000 7f0071587740 1 -- 172.21.15.69:0/731919755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) -- 0x55d8b55434f0 con 0x55d8b55158d0 2024-04-24T08:11:47.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.034+0000 7f0054c00640 1 -- 172.21.15.69:0/731919755 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00580205f0 con 0x55d8b55158d0 2024-04-24T08:11:47.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.034+0000 7f0054c00640 1 -- 172.21.15.69:0/731919755 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 33) ==== 3494+0+0 (secure 0 0 0) 0x7f0058036020 con 0x55d8b55158d0 2024-04-24T08:11:47.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.034+0000 7f0071587740 10 client.6369 fetch_fsmap finished waiting for FSMap version 33 2024-04-24T08:11:47.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.034+0000 7f0071587740 10 client.6369 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:11:47.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.034+0000 7f0071587740 20 client.6369 populate_metadata read hostname 'smithi069' 2024-04-24T08:11:47.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.034+0000 7f0071587740 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55d8b5511870 0x55d8b54e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.034+0000 7f0071587740 4 client.6369 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:47.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.034+0000 7f0071587740 1 -- 172.21.15.69:0/731919755 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8b545d530 con 0x55d8b5511870 2024-04-24T08:11:47.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.034+0000 7f005ea00640 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55d8b5511870 0x55d8b54e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.035+0000 7f005ea00640 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55d8b5511870 0x55d8b54e9fe0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1a39942f0ba07b6 server_cookie=359fd989cc873a32 in_seq=0 out_seq=0 2024-04-24T08:11:47.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.035+0000 7f0054c00640 10 client.6369 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:47.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0054c00640 1 -- 172.21.15.69:0/731919755 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d8b545d530 con 0x55d8b5511870 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0054c00640 10 client.6369 handle_command_reply: tid=0 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 client.6369 shutdown 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 -- 172.21.15.69:0/731919755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55158d0 msgr2=0x55d8b5510510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55158d0 0x55d8b5510510 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0058009010 tx=0x7f0058007000 comp rx=0 tx=0).stop 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 -- 172.21.15.69:0/731919755 shutdown_connections 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55d8b5511870 0x55d8b54e9fe0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b5517ee0 0x55d8b553fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55327b0 0x55d8b5543cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 --2- 172.21.15.69:0/731919755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55158d0 0x55d8b5510510 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0054c00640 0 client.6369 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 -- 172.21.15.69:0/731919755 >> 172.21.15.69:0/731919755 conn(0x55d8b5515190 msgr2=0x55d8b552fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 -- 172.21.15.69:0/731919755 shutdown_connections 2024-04-24T08:11:47.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 1 -- 172.21.15.69:0/731919755 wait complete. 2024-04-24T08:11:47.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.036+0000 7f0071587740 20 client.6369 trim_cache size 0 max 16384 2024-04-24T08:11:47.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.056+0000 7f0071587740 1 Processor -- start 2024-04-24T08:11:47.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.056+0000 7f0071587740 1 -- start start 2024-04-24T08:11:47.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.056+0000 7f0071587740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55158d0 0x55d8b54e8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0071587740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b5517ee0 0x55d8b54e8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0071587740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b552fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0071587740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00682007c0 con 0x55d8b55327b0 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0071587740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0068138430 con 0x55d8b5517ee0 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0071587740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0068200170 con 0x55d8b55158d0 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0067e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b552fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0067e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b552fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47502/0 (socket says 172.21.15.69:47502) 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0067e00640 1 -- 172.21.15.69:0/3167142371 learned_addr learned my addr 172.21.15.69:0/3167142371 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0067400640 1 --2- 172.21.15.69:0/3167142371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55158d0 0x55d8b54e8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0067e00640 1 -- 172.21.15.69:0/3167142371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55158d0 msgr2=0x55d8b54e8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0067e00640 1 --2- 172.21.15.69:0/3167142371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55158d0 0x55d8b54e8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0067e00640 1 -- 172.21.15.69:0/3167142371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b5517ee0 msgr2=0x55d8b54e8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0067e00640 1 --2- 172.21.15.69:0/3167142371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b5517ee0 0x55d8b54e8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.057+0000 7f0067e00640 1 -- 172.21.15.69:0/3167142371 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8b54eab70 con 0x55d8b55327b0 2024-04-24T08:11:47.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.058+0000 7f0067e00640 1 --2- 172.21.15.69:0/3167142371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b552fa50 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f00680d13f0 tx=0x7f0060009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d607816db6265a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:47.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.058+0000 7f0057400640 1 -- 172.21.15.69:0/3167142371 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f006001b070 con 0x55d8b55327b0 2024-04-24T08:11:47.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.058+0000 7f0057400640 1 -- 172.21.15.69:0/3167142371 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0060016400 con 0x55d8b55327b0 2024-04-24T08:11:47.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.058+0000 7f0057400640 1 -- 172.21.15.69:0/3167142371 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00600167f0 con 0x55d8b55327b0 2024-04-24T08:11:47.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.059+0000 7f0071587740 1 -- 172.21.15.69:0/3167142371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 msgr2=0x55d8b552fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.059+0000 7f0071587740 1 --2- 172.21.15.69:0/3167142371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b552fa50 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f00680d13f0 tx=0x7f0060009f90 comp rx=0 tx=0).stop 2024-04-24T08:11:47.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.059+0000 7f0071587740 1 -- 172.21.15.69:0/3167142371 shutdown_connections 2024-04-24T08:11:47.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.059+0000 7f0071587740 1 --2- 172.21.15.69:0/3167142371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b55327b0 0x55d8b552fa50 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.059+0000 7f0071587740 1 --2- 172.21.15.69:0/3167142371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b5517ee0 0x55d8b54e8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.059+0000 7f0071587740 1 --2- 172.21.15.69:0/3167142371 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55158d0 0x55d8b54e8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.059+0000 7f0071587740 1 -- 172.21.15.69:0/3167142371 >> 172.21.15.69:0/3167142371 conn(0x55d8b5515190 msgr2=0x55d8b5543ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:47.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.059+0000 7f0071587740 1 -- 172.21.15.69:0/3167142371 shutdown_connections 2024-04-24T08:11:47.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.059+0000 7f0071587740 1 -- 172.21.15.69:0/3167142371 wait complete. 2024-04-24T08:11:47.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.060+0000 7f0071587740 1 -- start start 2024-04-24T08:11:47.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.060+0000 7f0071587740 1 Processor -- start 2024-04-24T08:11:47.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.061+0000 7f0071587740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b55158d0 0x55d8b5517630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.061+0000 7f0071587740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b5517ee0 0x55d8b54ea000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.061+0000 7f0071587740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55327b0 0x55d8b55436c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.061+0000 7f0071587740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d8b55407a0 con 0x55d8b5517ee0 2024-04-24T08:11:47.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.061+0000 7f0071587740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d8b54e9560 con 0x55d8b55158d0 2024-04-24T08:11:47.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.061+0000 7f0071587740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55d8b5511c70 con 0x55d8b55327b0 2024-04-24T08:11:47.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.061+0000 7f0071587740 10 client.0 _ll_register_callbacks cb 0x7f00580076a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.061+0000 7f005ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b5517ee0 0x55d8b54ea000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.061+0000 7f005ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b5517ee0 0x55d8b54ea000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47518/0 (socket says 172.21.15.69:47518) 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.061+0000 7f005ea00640 1 -- 172.21.15.69:0/2503505988 learned_addr learned my addr 172.21.15.69:0/2503505988 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f005ea00640 1 -- 172.21.15.69:0/2503505988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55327b0 msgr2=0x55d8b55436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f005ea00640 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55327b0 0x55d8b55436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f005ea00640 1 -- 172.21.15.69:0/2503505988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b55158d0 msgr2=0x55d8b5517630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f005ea00640 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b55158d0 0x55d8b5517630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f005ea00640 1 -- 172.21.15.69:0/2503505988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8b552f8e0 con 0x55d8b5517ee0 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f005ea00640 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b5517ee0 0x55d8b54ea000 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0050001280 tx=0x7f0050076060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9613987768e56b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f0054c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f0054c00640 1 -- 172.21.15.69:0/2503505988 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050079d00 con 0x55d8b5517ee0 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f0054c00640 1 -- 172.21.15.69:0/2503505988 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f0050076370 con 0x55d8b5517ee0 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f0071587740 1 -- 172.21.15.69:0/2503505988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0060016400 con 0x55d8b5517ee0 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f0054c00640 1 -- 172.21.15.69:0/2503505988 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f0050087050 con 0x55d8b5517ee0 2024-04-24T08:11:47.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f0071587740 10 client.6375 fetch_fsmap learned FSMap version 33 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.062+0000 7f0071587740 1 -- 172.21.15.69:0/2503505988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) -- 0x55d8b55b2c90 con 0x55d8b5517ee0 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0054c00640 1 -- 172.21.15.69:0/2503505988 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0060016400 con 0x55d8b5517ee0 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0054c00640 1 -- 172.21.15.69:0/2503505988 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 33) ==== 3494+0+0 (secure 0 0 0) 0x7f00500052d0 con 0x55d8b5517ee0 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0071587740 10 client.6375 fetch_fsmap finished waiting for FSMap version 33 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0071587740 10 client.6375 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0071587740 20 client.6375 populate_metadata read hostname 'smithi069' 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0071587740 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55d8b553ec80 0x55d8b55b6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0071587740 4 client.6375 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0071587740 1 -- 172.21.15.69:0/2503505988 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "status"}) -- 0x55d8b5543240 con 0x55d8b553ec80 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0067400640 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55d8b553ec80 0x55d8b55b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0067400640 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55d8b553ec80 0x55d8b55b6280 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c668837a5580cb45 server_cookie=258aa0ed4a6c7550 in_seq=0 out_seq=0 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.063+0000 7f0054c00640 10 client.6375 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0054c00640 1 -- 172.21.15.69:0/2503505988 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x55d8b5543240 con 0x55d8b553ec80 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0054c00640 10 client.6375 handle_command_reply: tid=0 2024-04-24T08:11:47.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 client.6375 shutdown 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 -- 172.21.15.69:0/2503505988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b5517ee0 msgr2=0x55d8b54ea000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b5517ee0 0x55d8b54ea000 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f0050001280 tx=0x7f0050076060 comp rx=0 tx=0).stop 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 -- 172.21.15.69:0/2503505988 shutdown_connections 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55d8b553ec80 0x55d8b55b6280 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d8b5517ee0 0x55d8b54ea000 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d8b55327b0 0x55d8b55436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0054c00640 0 client.6375 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 --2- 172.21.15.69:0/2503505988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d8b55158d0 0x55d8b5517630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 -- 172.21.15.69:0/2503505988 >> 172.21.15.69:0/2503505988 conn(0x55d8b5515190 msgr2=0x55d8b5529720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 -- 172.21.15.69:0/2503505988 shutdown_connections 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 1 -- 172.21.15.69:0/2503505988 wait complete. 2024-04-24T08:11:47.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.064+0000 7f0071587740 20 client.6375 trim_cache size 0 max 16384 2024-04-24T08:11:47.070 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:11:47.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:11:47.070 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:11:47.070 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 6231, 2024-04-24T08:11:47.070 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:11:47.070 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:11:47.070 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:11:47.070 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 6.7134270349999996, 2024-04-24T08:11:47.071 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 33, 2024-04-24T08:11:47.071 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 57, 2024-04-24T08:11:47.071 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 57, 2024-04-24T08:11:47.071 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 12.166225054 2024-04-24T08:11:47.071 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 -- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004406c920 msgr2=0x7f004406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004406c920 0x7f004406ede0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f00580079b0 tx=0x7f0058009040 comp rx=0 tx=0).stop 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 -- 172.21.15.69:0/839429006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00680cb210 msgr2=0x7f00680b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00680cb210 0x7f00680b16a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0050001840 tx=0x7f0050001120 comp rx=0 tx=0).stop 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 -- 172.21.15.69:0/839429006 shutdown_connections 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f004406c920 0x7f004406ede0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00680cb210 0x7f00680b16a0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00680cb690 0x7f00680b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 --2- 172.21.15.69:0/839429006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f004c004800 0x7f006815c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 -- 172.21.15.69:0/839429006 >> 172.21.15.69:0/839429006 conn(0x7f0068067d90 msgr2=0x7f006815f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 -- 172.21.15.69:0/839429006 shutdown_connections 2024-04-24T08:11:47.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.065+0000 7f006e800640 1 -- 172.21.15.69:0/839429006 wait complete. 2024-04-24T08:11:47.082 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:11:47.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.266+0000 7f4bba000640 1 Processor -- start 2024-04-24T08:11:47.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.267+0000 7f4bba000640 1 -- start start 2024-04-24T08:11:47.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.268+0000 7f4bba000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4bb40d0300 0x7f4bb406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.268+0000 7f4bba000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bb4145a90 0x7f4bb40d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.268+0000 7f4bba000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bb40d0c20 con 0x7f4bb4145fe0 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.268+0000 7f4bba000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4bb40cf3d0 con 0x7f4bb40d0300 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.268+0000 7f4bba000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb40cf6b0 con 0x7f4bb4145a90 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.269+0000 7f4bb3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4bb40d0300 0x7f4bb406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.269+0000 7f4bb3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4bb40d0300 0x7f4bb406e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47532/0 (socket says 172.21.15.69:47532) 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.269+0000 7f4bb3400640 1 -- 172.21.15.69:0/1670628892 learned_addr learned my addr 172.21.15.69:0/1670628892 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.269+0000 7f4baaa00640 1 --2- 172.21.15.69:0/1670628892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bb4145a90 0x7f4bb40d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.269+0000 7f4bb3400640 1 -- 172.21.15.69:0/1670628892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bb4145a90 msgr2=0x7f4bb40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.269+0000 7f4bb3400640 1 --2- 172.21.15.69:0/1670628892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bb4145a90 0x7f4bb40d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.269+0000 7f4bb3400640 1 -- 172.21.15.69:0/1670628892 >> v1:172.21.15.69:6789/0 conn(0x7f4bb4145fe0 legacy=0x7f4bb4160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:47.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.269+0000 7f4bb3400640 1 -- 172.21.15.69:0/1670628892 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb40cf990 con 0x7f4bb40d0300 2024-04-24T08:11:47.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.269+0000 7f4baaa00640 1 --2- 172.21.15.69:0/1670628892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bb4145a90 0x7f4bb40d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:47.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.270+0000 7f4bb3400640 1 --2- 172.21.15.69:0/1670628892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb40d0300 0x7f4bb406e240 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009870 tx=0x7f4ba402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9ce59f44db0e9f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:47.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.270+0000 7f4bb2a00640 1 -- 172.21.15.69:0/1670628892 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4037070 con 0x7f4bb40d0300 2024-04-24T08:11:47.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.270+0000 7f4bb2a00640 1 -- 172.21.15.69:0/1670628892 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba4032440 con 0x7f4bb40d0300 2024-04-24T08:11:47.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.270+0000 7f4bb2a00640 1 -- 172.21.15.69:0/1670628892 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4032810 con 0x7f4bb40d0300 2024-04-24T08:11:47.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.271+0000 7f4bba000640 1 -- 172.21.15.69:0/1670628892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb40d0300 msgr2=0x7f4bb406e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.271+0000 7f4bba000640 1 --2- 172.21.15.69:0/1670628892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb40d0300 0x7f4bb406e240 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009870 tx=0x7f4ba402a250 comp rx=0 tx=0).stop 2024-04-24T08:11:47.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.271+0000 7f4bba000640 1 -- 172.21.15.69:0/1670628892 shutdown_connections 2024-04-24T08:11:47.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.271+0000 7f4bba000640 1 --2- 172.21.15.69:0/1670628892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bb4145a90 0x7f4bb40d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.271+0000 7f4bba000640 1 --2- 172.21.15.69:0/1670628892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb40d0300 0x7f4bb406e240 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.271+0000 7f4bba000640 1 -- 172.21.15.69:0/1670628892 >> 172.21.15.69:0/1670628892 conn(0x7f4bb4067d90 msgr2=0x7f4bb406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:47.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.271+0000 7f4bba000640 1 -- 172.21.15.69:0/1670628892 shutdown_connections 2024-04-24T08:11:47.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.271+0000 7f4bba000640 1 -- 172.21.15.69:0/1670628892 wait complete. 2024-04-24T08:11:47.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.273+0000 7f4bba000640 1 Processor -- start 2024-04-24T08:11:47.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.274+0000 7f4bba000640 1 -- start start 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4bba000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bb40d0300 0x7f4bb415f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4bba000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb4145a90 0x7f4bb415bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4bba000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bb4145fe0 0x7f4bb40c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4bba000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb4145770 con 0x7f4bb4145a90 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4bba000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb40cff50 con 0x7f4bb40d0300 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4bba000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb41458f0 con 0x7f4bb4145fe0 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4baaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb4145a90 0x7f4bb415bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4baaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb4145a90 0x7f4bb415bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47538/0 (socket says 172.21.15.69:47538) 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4baaa00640 1 -- 172.21.15.69:0/3595659510 learned_addr learned my addr 172.21.15.69:0/3595659510 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4bb3400640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bb40d0300 0x7f4bb415f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4baaa00640 1 -- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bb4145fe0 msgr2=0x7f4bb40c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.275+0000 7f4baaa00640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bb4145fe0 0x7f4bb40c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.276+0000 7f4baaa00640 1 -- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bb40d0300 msgr2=0x7f4bb415f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.276+0000 7f4baaa00640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bb40d0300 0x7f4bb415f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.276+0000 7f4baaa00640 1 -- 172.21.15.69:0/3595659510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb415e850 con 0x7f4bb4145a90 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.276+0000 7f4bb3400640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bb40d0300 0x7f4bb415f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.276+0000 7f4baaa00640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb4145a90 0x7f4bb415bf80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c7f0 tx=0x7f4b9c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.276+0000 7f4babe00640 1 -- 172.21.15.69:0/3595659510 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c015070 con 0x7f4bb4145a90 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.276+0000 7f4babe00640 1 -- 172.21.15.69:0/3595659510 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b9c007820 con 0x7f4bb4145a90 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.276+0000 7f4bba000640 1 -- 172.21.15.69:0/3595659510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb415eac0 con 0x7f4bb4145a90 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.276+0000 7f4babe00640 1 -- 172.21.15.69:0/3595659510 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c007bd0 con 0x7f4bb4145a90 2024-04-24T08:11:47.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.276+0000 7f4bba000640 1 -- 172.21.15.69:0/3595659510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb415f360 con 0x7f4bb4145a90 2024-04-24T08:11:47.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.277+0000 7f4bba000640 1 -- 172.21.15.69:0/3595659510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb4068a80 con 0x7f4bb4145a90 2024-04-24T08:11:47.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.279+0000 7f4babe00640 1 -- 172.21.15.69:0/3595659510 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4b9c018070 con 0x7f4bb4145a90 2024-04-24T08:11:47.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.279+0000 7f4babe00640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9006c920 0x7f4b9006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:47.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.280+0000 7f4babe00640 1 -- 172.21.15.69:0/3595659510 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f4b9c08bd20 con 0x7f4bb4145a90 2024-04-24T08:11:47.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.280+0000 7f4bb3400640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9006c920 0x7f4b9006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:47.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.281+0000 7f4bb3400640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9006c920 0x7f4b9006ede0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4034000 tx=0x7f4ba402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:47.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.284+0000 7f4babe00640 1 -- 172.21.15.69:0/3595659510 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4b9c05a020 con 0x7f4bb4145a90 2024-04-24T08:11:47.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.528+0000 7f4bba000640 1 -- 172.21.15.69:0/3595659510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4bb40d1120 con 0x7f4bb4145a90 2024-04-24T08:11:47.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.528+0000 7f4babe00640 1 -- 172.21.15.69:0/3595659510 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) ==== 53+0+73 (secure 0 0 0) 0x7f4b9c059920 con 0x7f4bb4145a90 2024-04-24T08:11:47.529 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:11:47.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 -- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9006c920 msgr2=0x7f4b9006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9006c920 0x7f4b9006ede0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4034000 tx=0x7f4ba402e000 comp rx=0 tx=0).stop 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 -- 172.21.15.69:0/3595659510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb4145a90 msgr2=0x7f4bb415bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb4145a90 0x7f4bb415bf80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c7f0 tx=0x7f4b9c00ccb0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 -- 172.21.15.69:0/3595659510 shutdown_connections 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4b9006c920 0x7f4b9006ede0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bb4145a90 0x7f4bb415bf80 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bb4145fe0 0x7f4bb40c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 --2- 172.21.15.69:0/3595659510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bb40d0300 0x7f4bb415f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 -- 172.21.15.69:0/3595659510 >> 172.21.15.69:0/3595659510 conn(0x7f4bb4067d90 msgr2=0x7f4bb40afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 -- 172.21.15.69:0/3595659510 shutdown_connections 2024-04-24T08:11:47.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47.531+0000 7f4bba000640 1 -- 172.21.15.69:0/3595659510 wait complete. 2024-04-24T08:11:47.550 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:11:47.550 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:11:47.550 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:11:47.550 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:11:47.550 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:11:47.550 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:11:47.550 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:11:47.550 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:11:47.550 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:11:47.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:47.551 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:11:47.598 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:11:47.598 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:11:47.598 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:11:47.598 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:11:47.598 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:11:47.598 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:11:47.598 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42472sec preferred_lft 42472sec 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:11:47.599 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:11:47.600 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:11:47.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:47.602 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:11:47.723 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:11:47.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:47.727 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:11:47.778 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:11:47.778 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:11:47.779 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:11:47.779 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:11:47.779 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:11:47.779 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:11:47.779 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:11:47.779 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:11:47.779 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:11:47.779 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:11:47.779 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:11:47.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:11:47.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:11:47.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:47.986 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:11:48.031 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:11:48.031 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:11:48.032 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:11:48.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:48.032 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:11:48.160 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:11:48.160 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:11:48.160 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:11:48.214 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:11:48.214 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:11:48.215 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:11:48.265 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:11:48.347 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:11:48.402 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:11:48.402 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:11:48.403 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:11:48.404 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:11:48.405 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:11:48.405 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:11:48.405 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:11:48.405 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:11:48.405 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:11:48.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:48.406 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:11:48.453 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:11:48.453 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:11:48.501 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:11:48.574 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:11:48.616 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:11:48.616 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:11:48.616 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:11:48.616 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:11:48.617 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:11:48.618 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:11:48.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:48.619 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:11:48.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:11:48.673+0000 7f35f4379480 -1 init, newargv = 0x55a345576540 newargc=15 2024-04-24T08:11:48.673 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[54050]: starting ceph client 2024-04-24T08:11:48.683 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[54050]: starting fuse 2024-04-24T08:11:48.703 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:11:48.703 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:11:48.703 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:11:48.703 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:11:48.703 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:11:48.704 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:11:48.705 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:11:48.888 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.54050.asok 2024-04-24T08:11:48.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:11:48.904 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54050.asok 2024-04-24T08:11:48.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:48.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54050.asok status 2024-04-24T08:11:49.129 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "54050", 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:11:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 6384, 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 6384 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2507287482 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:11:49.131 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2507287482 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.6384 192.168.0.1:0/2507287482", 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2507287482", 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 33, 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 57, 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:11:49.132 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:11:49.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:49.145 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:11:49.145 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:11:49.196 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:11:49.196 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:11:49.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:49.196 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:11:49.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:49.269 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_test) 2024-04-24T08:11:49.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:49.331 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir_test/file) 2024-04-24T08:11:49.390 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:11:49.390 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:11:49.390 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:11:49.390 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:11:49.390 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_test/file").st_ino) 2024-04-24T08:11:49.391 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:11:49.520 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627777 2024-04-24T08:11:49.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:11:49.533 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:49.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.779+0000 7f3f12e00640 1 Processor -- start 2024-04-24T08:11:49.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.779+0000 7f3f12e00640 1 -- start start 2024-04-24T08:11:49.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.780+0000 7f3f12e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3f0c072230 0x7f3f0c169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:49.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.780+0000 7f3f12e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f0c144db0 con 0x7f3f0c162300 2024-04-24T08:11:49.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.780+0000 7f3f12e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f0c15f540 con 0x7f3f0c071e50 2024-04-24T08:11:49.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.780+0000 7f3f12e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3f0c15f820 con 0x7f3f0c072230 2024-04-24T08:11:49.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.781+0000 7f3f11000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3f0c162300 0x7f3f0c143690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56680/0 (socket says 172.21.15.69:56680) 2024-04-24T08:11:49.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.781+0000 7f3f11000640 1 -- 172.21.15.69:0/1471510613 learned_addr learned my addr 172.21.15.69:0/1471510613 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:49.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.781+0000 7f3f12400640 1 --2- 172.21.15.69:0/1471510613 >> v2:172.21.15.121:3300/0 conn(0x7f3f0c072230 0x7f3f0c169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:49.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.781+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3978373599 0 0) 0x7f3f0c144db0 con 0x7f3f0c162300 2024-04-24T08:11:49.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.781+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ef4003520 con 0x7f3f0c162300 2024-04-24T08:11:49.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.781+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2289876838 0 0) 0x7f3ef4003520 con 0x7f3f0c162300 2024-04-24T08:11:49.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 >> v2:172.21.15.121:3300/0 conn(0x7f3f0c072230 msgr2=0x7f3f0c169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:49.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 --2- 172.21.15.69:0/1471510613 >> v2:172.21.15.121:3300/0 conn(0x7f3f0c072230 0x7f3f0c169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:49.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 >> v1:172.21.15.121:6789/0 conn(0x7f3f0c071e50 legacy=0x7f3f0c15ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:49.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0c15fb00 con 0x7f3f0c162300 2024-04-24T08:11:49.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f3efc001670 con 0x7f3f0c162300 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 >> v1:172.21.15.69:6789/0 conn(0x7f3f0c162300 legacy=0x7f3f0c143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 --2- 172.21.15.69:0/1471510613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ef40039b0 0x7f3ef4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 --2- 172.21.15.69:0/1471510613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef4004800 0x7f3ef4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 --2- 172.21.15.69:0/1471510613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ef40051c0 0x7f3ef400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c15f820 con 0x7f3ef40051c0 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3ef400c060 con 0x7f3ef4004800 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3ef400c260 con 0x7f3ef40039b0 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f12400640 1 --2- 172.21.15.69:0/1471510613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ef40051c0 0x7f3ef400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f12400640 1 -- 172.21.15.69:0/1471510613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ef40039b0 msgr2=0x7f3ef4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f12400640 1 --2- 172.21.15.69:0/1471510613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ef40039b0 0x7f3ef4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f12400640 1 -- 172.21.15.69:0/1471510613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef4004800 msgr2=0x7f3ef4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f12400640 1 --2- 172.21.15.69:0/1471510613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ef4004800 0x7f3ef4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:49.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.782+0000 7f3f12400640 1 -- 172.21.15.69:0/1471510613 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef4003520 con 0x7f3ef40051c0 2024-04-24T08:11:49.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.783+0000 7f3f12400640 1 -- 172.21.15.69:0/1471510613 reap_dead start 2024-04-24T08:11:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.783+0000 7f3f12400640 1 --2- 172.21.15.69:0/1471510613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ef40051c0 0x7f3ef400b9c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009fc0 tx=0x7f3ef4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75dbbbb70294a909 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.784+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04035020 con 0x7f3ef40051c0 2024-04-24T08:11:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.784+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f04003bd0 con 0x7f3ef40051c0 2024-04-24T08:11:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.784+0000 7f3f0be00640 1 -- 172.21.15.69:0/1471510613 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04009c10 con 0x7f3ef40051c0 2024-04-24T08:11:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.784+0000 7f3f12e00640 1 -- 172.21.15.69:0/1471510613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ef40051c0 msgr2=0x7f3ef400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.784+0000 7f3f12e00640 1 --2- 172.21.15.69:0/1471510613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ef40051c0 0x7f3ef400b9c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009fc0 tx=0x7f3ef4005830 comp rx=0 tx=0).stop 2024-04-24T08:11:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.784+0000 7f3f12e00640 1 -- 172.21.15.69:0/1471510613 shutdown_connections 2024-04-24T08:11:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.784+0000 7f3f12e00640 1 --2- 172.21.15.69:0/1471510613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3ef40051c0 0x7f3ef400b9c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.784+0000 7f3f12e00640 1 -- 172.21.15.69:0/1471510613 >> 172.21.15.69:0/1471510613 conn(0x7f3f0c0d0160 msgr2=0x7f3f0c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:49.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.785+0000 7f3f12e00640 1 -- 172.21.15.69:0/1471510613 shutdown_connections 2024-04-24T08:11:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.785+0000 7f3f12e00640 1 -- 172.21.15.69:0/1471510613 wait complete. 2024-04-24T08:11:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.786+0000 7f3f12e00640 1 Processor -- start 2024-04-24T08:11:49.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.786+0000 7f3f12e00640 1 -- start start 2024-04-24T08:11:49.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.787+0000 7f3f12e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ef40051c0 0x7f3f0c147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:49.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.787+0000 7f3f12e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f0c144fe0 0x7f3f0c1520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.787+0000 7f3f12e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f0c152600 0x7f3f0c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.787+0000 7f3f12e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c15ee40 con 0x7f3f0c144fe0 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.787+0000 7f3f12e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c15ecc0 con 0x7f3f0c152600 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.787+0000 7f3f12e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3f0c15efc0 con 0x7f3ef40051c0 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f11000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f0c144fe0 0x7f3f0c1520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f11000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f0c144fe0 0x7f3f0c1520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57220/0 (socket says 172.21.15.69:57220) 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f11000640 1 -- 172.21.15.69:0/725989509 learned_addr learned my addr 172.21.15.69:0/725989509 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f12400640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f0c152600 0x7f3f0c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f11a00640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ef40051c0 0x7f3f0c147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f11000640 1 -- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ef40051c0 msgr2=0x7f3f0c147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f11000640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ef40051c0 0x7f3f0c147a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f11000640 1 -- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f0c152600 msgr2=0x7f3f0c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f11000640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f0c152600 0x7f3f0c0b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f11000640 1 -- 172.21.15.69:0/725989509 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0c151300 con 0x7f3f0c144fe0 2024-04-24T08:11:49.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.788+0000 7f3f11000640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f0c144fe0 0x7f3f0c1520c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00c760 tx=0x7f3efc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.789+0000 7f3f09600640 1 -- 172.21.15.69:0/725989509 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc018070 con 0x7f3f0c144fe0 2024-04-24T08:11:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.789+0000 7f3f09600640 1 -- 172.21.15.69:0/725989509 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3efc0074e0 con 0x7f3f0c144fe0 2024-04-24T08:11:49.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.789+0000 7f3f09600640 1 -- 172.21.15.69:0/725989509 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc004a20 con 0x7f3f0c144fe0 2024-04-24T08:11:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.789+0000 7f3f12e00640 1 -- 172.21.15.69:0/725989509 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0c1515e0 con 0x7f3f0c144fe0 2024-04-24T08:11:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.789+0000 7f3f12e00640 1 -- 172.21.15.69:0/725989509 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0c151ba0 con 0x7f3f0c144fe0 2024-04-24T08:11:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.790+0000 7f3f12e00640 1 -- 172.21.15.69:0/725989509 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f0c075d50 con 0x7f3f0c144fe0 2024-04-24T08:11:49.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.791+0000 7f3f09600640 1 -- 172.21.15.69:0/725989509 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3efc01b070 con 0x7f3f0c144fe0 2024-04-24T08:11:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.792+0000 7f3f09600640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ef006c920 0x7f3ef006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:49.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.792+0000 7f3f09600640 1 -- 172.21.15.69:0/725989509 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f3efc08b090 con 0x7f3f0c144fe0 2024-04-24T08:11:49.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.793+0000 7f3f11a00640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ef006c920 0x7f3ef006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:49.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.794+0000 7f3f11a00640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ef006c920 0x7f3ef006ede0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3f000015d0 tx=0x7f3f00009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:49.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:49.797+0000 7f3f09600640 1 -- 172.21.15.69:0/725989509 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3efc059420 con 0x7f3f0c144fe0 2024-04-24T08:11:50.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.039+0000 7f3f12e00640 1 -- 172.21.15.69:0/725989509 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f0c0d0d90 con 0x7f3f0c144fe0 2024-04-24T08:11:50.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.040+0000 7f3f09600640 1 -- 172.21.15.69:0/725989509 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+5187 (secure 0 0 0) 0x7f3efc058d20 con 0x7f3f0c144fe0 2024-04-24T08:11:50.041 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:50.041 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4034001365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4034001365},{"type":"v1","addr":"172.21.15.121:6835","nonce":4034001365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3907163219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3907163219},{"type":"v1","addr":"172.21.15.121:6837","nonce":3907163219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2756395689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2756395689},{"type":"v1","addr":"172.21.15.69:6834","nonce":2756395689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:11:39.824842+0000","modified":"2024-04-24T08:11:42.963374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/2288057943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":2288057943},{"type":"v1","addr":"172.21.15.69:6835","nonce":2288057943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6231,"qdb_cluster":[6231]},"id":4}]} 2024-04-24T08:11:50.041 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 33 2024-04-24T08:11:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 -- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ef006c920 msgr2=0x7f3ef006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ef006c920 0x7f3ef006ede0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3f000015d0 tx=0x7f3f00009040 comp rx=0 tx=0).stop 2024-04-24T08:11:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 -- 172.21.15.69:0/725989509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f0c144fe0 msgr2=0x7f3f0c1520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f0c144fe0 0x7f3f0c1520c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00c760 tx=0x7f3efc00cc20 comp rx=0 tx=0).stop 2024-04-24T08:11:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 -- 172.21.15.69:0/725989509 shutdown_connections 2024-04-24T08:11:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3ef006c920 0x7f3ef006ede0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f0c144fe0 0x7f3f0c1520c0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3f0c152600 0x7f3f0c0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 --2- 172.21.15.69:0/725989509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ef40051c0 0x7f3f0c147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 -- 172.21.15.69:0/725989509 >> 172.21.15.69:0/725989509 conn(0x7f3f0c0d0160 msgr2=0x7f3f0c143460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 -- 172.21.15.69:0/725989509 shutdown_connections 2024-04-24T08:11:50.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.043+0000 7f3f12e00640 1 -- 172.21.15.69:0/725989509 wait complete. 2024-04-24T08:11:50.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-04-24T08:11:50.311 INFO:teuthology.orchestra.run.smithi069.stdout:{"message":"","return_code":0} 2024-04-24T08:11:50.311 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-24T08:11:50.312 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-04-24T08:11:50.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.410+0000 7fda54de4f40 1 Processor -- start 2024-04-24T08:11:50.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.410+0000 7fda54de4f40 1 -- start start 2024-04-24T08:11:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda54de4f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380028d0 0x55591538fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda54de4f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5559152fc4e0 0x5559152f7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda54de4f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55591539a400 0x555915390b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda54de4f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5559152fa180 con 0x7fda380028d0 2024-04-24T08:11:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda54de4f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555915270cc0 con 0x5559152fc4e0 2024-04-24T08:11:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda54de4f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5559152c99b0 con 0x55591539a400 2024-04-24T08:11:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda52000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380028d0 0x55591538fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda52000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380028d0 0x55591538fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57244/0 (socket says 172.21.15.69:57244) 2024-04-24T08:11:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda52000640 1 -- 172.21.15.69:0/226475279 learned_addr learned my addr 172.21.15.69:0/226475279 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda52000640 1 -- 172.21.15.69:0/226475279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55591539a400 msgr2=0x555915390b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda52000640 1 --2- 172.21.15.69:0/226475279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55591539a400 0x555915390b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda52000640 1 -- 172.21.15.69:0/226475279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5559152fc4e0 msgr2=0x5559152f7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda52000640 1 --2- 172.21.15.69:0/226475279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5559152fc4e0 0x5559152f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.411+0000 7fda52000640 1 -- 172.21.15.69:0/226475279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559153a39b0 con 0x7fda380028d0 2024-04-24T08:11:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.412+0000 7fda52000640 1 --2- 172.21.15.69:0/226475279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380028d0 0x55591538fc50 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fda440095f0 tx=0x7fda440096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff72348047e4c6ce server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.412+0000 7fda4be00640 1 -- 172.21.15.69:0/226475279 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44019070 con 0x7fda380028d0 2024-04-24T08:11:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.412+0000 7fda4be00640 1 -- 172.21.15.69:0/226475279 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda44007800 con 0x7fda380028d0 2024-04-24T08:11:50.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.412+0000 7fda4be00640 1 -- 172.21.15.69:0/226475279 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44007bd0 con 0x7fda380028d0 2024-04-24T08:11:50.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.412+0000 7fda54de4f40 1 -- 172.21.15.69:0/226475279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380028d0 msgr2=0x55591538fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.412+0000 7fda54de4f40 1 --2- 172.21.15.69:0/226475279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380028d0 0x55591538fc50 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fda440095f0 tx=0x7fda440096e0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.412+0000 7fda54de4f40 1 -- 172.21.15.69:0/226475279 shutdown_connections 2024-04-24T08:11:50.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.412+0000 7fda54de4f40 1 --2- 172.21.15.69:0/226475279 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55591539a400 0x555915390b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.413+0000 7fda54de4f40 1 --2- 172.21.15.69:0/226475279 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5559152fc4e0 0x5559152f7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.413+0000 7fda54de4f40 1 --2- 172.21.15.69:0/226475279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda380028d0 0x55591538fc50 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.413+0000 7fda54de4f40 1 -- 172.21.15.69:0/226475279 >> 172.21.15.69:0/226475279 conn(0x5559152f8b50 msgr2=0x5559152f8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.413+0000 7fda54de4f40 1 -- 172.21.15.69:0/226475279 shutdown_connections 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.413+0000 7fda54de4f40 1 -- 172.21.15.69:0/226475279 wait complete. 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.413+0000 7fda54de4f40 1 Processor -- start 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.414+0000 7fda54de4f40 1 -- start start 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.414+0000 7fda54de4f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5559152fc4e0 0x555915386350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.414+0000 7fda54de4f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55591539a400 0x555915390a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.414+0000 7fda54de4f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda380028d0 0x5559152fb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.414+0000 7fda54de4f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5559153a3ed0 con 0x55591539a400 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.414+0000 7fda54de4f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5559153979b0 con 0x7fda380028d0 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.414+0000 7fda54de4f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5559153908c0 con 0x5559152fc4e0 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.415+0000 7fda51600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55591539a400 0x555915390a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.415+0000 7fda51600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55591539a400 0x555915390a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57254/0 (socket says 172.21.15.69:57254) 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.415+0000 7fda51600640 1 -- 172.21.15.69:0/3964998924 learned_addr learned my addr 172.21.15.69:0/3964998924 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.415+0000 7fda52000640 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5559152fc4e0 0x555915386350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.415+0000 7fda51600640 1 -- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5559152fc4e0 msgr2=0x555915386350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.415+0000 7fda51600640 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5559152fc4e0 0x555915386350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.415+0000 7fda51600640 1 -- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda380028d0 msgr2=0x5559152fb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.415+0000 7fda51600640 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda380028d0 0x5559152fb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.415+0000 7fda51600640 1 -- 172.21.15.69:0/3964998924 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559152f8350 con 0x55591539a400 2024-04-24T08:11:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.415+0000 7fda51600640 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55591539a400 0x555915390a40 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fda40025bf0 tx=0x7fda40000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.416+0000 7fda4aa00640 1 -- 172.21.15.69:0/3964998924 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4000a320 con 0x55591539a400 2024-04-24T08:11:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.416+0000 7fda4aa00640 1 -- 172.21.15.69:0/3964998924 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda4005e070 con 0x55591539a400 2024-04-24T08:11:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.416+0000 7fda54de4f40 1 -- 172.21.15.69:0/3964998924 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x555915393070 con 0x55591539a400 2024-04-24T08:11:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.416+0000 7fda4aa00640 1 -- 172.21.15.69:0/3964998924 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4004dd40 con 0x55591539a400 2024-04-24T08:11:50.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.416+0000 7fda54de4f40 1 -- 172.21.15.69:0/3964998924 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555915393b50 con 0x55591539a400 2024-04-24T08:11:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.418+0000 7fda4aa00640 1 -- 172.21.15.69:0/3964998924 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fda40060070 con 0x55591539a400 2024-04-24T08:11:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.419+0000 7fda4aa00640 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda3406b2d0 0x7fda3406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.419+0000 7fda4aa00640 1 -- 172.21.15.69:0/3964998924 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fda400cd6f0 con 0x55591539a400 2024-04-24T08:11:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.419+0000 7fda54de4f40 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x5559153a1320 0x5559153a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.419+0000 7fda54de4f40 1 -- 172.21.15.69:0/3964998924 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:1 9.7 9:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57) -- 0x5559153a1c40 con 0x5559153a1320 2024-04-24T08:11:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.420+0000 7fda52000640 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda3406b2d0 0x7fda3406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.420+0000 7fda52a00640 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x5559153a1320 0x5559153a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.421+0000 7fda52000640 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda3406b2d0 0x7fda3406d790 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x555915398330 tx=0x7fda4400c070 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.421+0000 7fda52a00640 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x5559153a1320 0x5559153a1700 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda52a00640 1 -- 172.21.15.69:0/3964998924 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (94) out=94b] v0'0 uv1 ondisk = 0) ==== 164+0+94 (crc 0 0 0) 0x7fda4c00f450 con 0x5559153a1320 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda54de4f40 1 -- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x5559153a1320 msgr2=0x5559153a1700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda54de4f40 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x5559153a1320 0x5559153a1700 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda54de4f40 1 -- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda3406b2d0 msgr2=0x7fda3406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda54de4f40 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda3406b2d0 0x7fda3406d790 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x555915398330 tx=0x7fda4400c070 comp rx=0 tx=0).stop 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda54de4f40 1 -- 172.21.15.69:0/3964998924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55591539a400 msgr2=0x555915390a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda54de4f40 1 --2- 172.21.15.69:0/3964998924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55591539a400 0x555915390a40 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fda40025bf0 tx=0x7fda40000f40 comp rx=0 tx=0).stop 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda52a00640 1 -- 172.21.15.69:0/3964998924 reap_dead start 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda54de4f40 1 -- 172.21.15.69:0/3964998924 shutdown_connections 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda54de4f40 1 -- 172.21.15.69:0/3964998924 >> 172.21.15.69:0/3964998924 conn(0x5559152f8b50 msgr2=0x5559152f7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda54de4f40 1 -- 172.21.15.69:0/3964998924 shutdown_connections 2024-04-24T08:11:50.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.422+0000 7fda54de4f40 1 -- 172.21.15.69:0/3964998924 wait complete. 2024-04-24T08:11:50.428 INFO:teuthology.orchestra.run.smithi069.stdout:Xfiledir_test 2024-04-24T08:11:50.428 DEBUG:teuthology.orchestra.run.smithi069:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-04-24T08:11:50.526 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:11:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511627777, 2024-04-24T08:11:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "ancestors": [ 2024-04-24T08:11:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "dirino": 1099511627776, 2024-04-24T08:11:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "dname": "file", 2024-04-24T08:11:50.526 INFO:teuthology.orchestra.run.smithi069.stdout: "version": 2 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "dirino": 1, 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "dname": "dir_test", 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "version": 4 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "pool": 9, 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: "old_pools": [] 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:11:50.527 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:11:50.531 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'The backtrace is corrupted' 2024-04-24T08:11:50.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.636+0000 7eff10586f40 1 Processor -- start 2024-04-24T08:11:50.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.636+0000 7eff10586f40 1 -- start start 2024-04-24T08:11:50.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.637+0000 7eff10586f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ba9816950 0x556ba98a1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.637+0000 7eff10586f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba9818570 0x556ba97fbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.637+0000 7eff10586f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ba98a27c0 0x556ba97fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.637+0000 7eff10586f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556ba9816d60 con 0x556ba9818570 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.637+0000 7eff10586f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556ba97fa5b0 con 0x556ba9816950 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.637+0000 7eff10586f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556ba97cf610 con 0x556ba98a27c0 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.638+0000 7eff04e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba9818570 0x556ba97fbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.638+0000 7eff04e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba9818570 0x556ba97fbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57278/0 (socket says 172.21.15.69:57278) 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.638+0000 7eff04e00640 1 -- 172.21.15.69:0/1169171221 learned_addr learned my addr 172.21.15.69:0/1169171221 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.638+0000 7eff0d800640 1 --2- 172.21.15.69:0/1169171221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ba9816950 0x556ba98a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.638+0000 7eff04e00640 1 -- 172.21.15.69:0/1169171221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ba98a27c0 msgr2=0x556ba97fc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.638+0000 7eff04e00640 1 --2- 172.21.15.69:0/1169171221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ba98a27c0 0x556ba97fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.638+0000 7eff04e00640 1 -- 172.21.15.69:0/1169171221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ba9816950 msgr2=0x556ba98a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.638+0000 7eff04e00640 1 --2- 172.21.15.69:0/1169171221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ba9816950 0x556ba98a1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.638+0000 7eff04e00640 1 -- 172.21.15.69:0/1169171221 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ba96f8b30 con 0x556ba9818570 2024-04-24T08:11:50.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.638+0000 7eff04e00640 1 --2- 172.21.15.69:0/1169171221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba9818570 0x556ba97fbdb0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7efef800a5e0 tx=0x7efef8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ccff9e9612c3e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff07e00640 1 -- 172.21.15.69:0/1169171221 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8008a40 con 0x556ba9818570 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff07e00640 1 -- 172.21.15.69:0/1169171221 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef8008c00 con 0x556ba9818570 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff07e00640 1 -- 172.21.15.69:0/1169171221 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8013440 con 0x556ba9818570 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff10586f40 1 -- 172.21.15.69:0/1169171221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba9818570 msgr2=0x556ba97fbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff10586f40 1 --2- 172.21.15.69:0/1169171221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba9818570 0x556ba97fbdb0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7efef800a5e0 tx=0x7efef8005f90 comp rx=0 tx=0).stop 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff10586f40 1 -- 172.21.15.69:0/1169171221 shutdown_connections 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff10586f40 1 --2- 172.21.15.69:0/1169171221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba9818570 0x556ba97fbdb0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff10586f40 1 --2- 172.21.15.69:0/1169171221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ba98a27c0 0x556ba97fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff10586f40 1 --2- 172.21.15.69:0/1169171221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ba9816950 0x556ba98a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff10586f40 1 -- 172.21.15.69:0/1169171221 >> 172.21.15.69:0/1169171221 conn(0x556ba97f9730 msgr2=0x556ba98a0480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff10586f40 1 -- 172.21.15.69:0/1169171221 shutdown_connections 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.639+0000 7eff10586f40 1 -- 172.21.15.69:0/1169171221 wait complete. 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.640+0000 7eff10586f40 1 Processor -- start 2024-04-24T08:11:50.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.640+0000 7eff10586f40 1 -- start start 2024-04-24T08:11:50.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.641+0000 7eff10586f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ba9816950 0x556ba9885830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.641+0000 7eff10586f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ba9818570 0x556ba9899d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.641+0000 7eff10586f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba98a27c0 0x556ba9897f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.641+0000 7eff10586f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556ba989ff80 con 0x556ba98a27c0 2024-04-24T08:11:50.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.641+0000 7eff10586f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556ba9886c10 con 0x556ba9818570 2024-04-24T08:11:50.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.641+0000 7eff10586f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556ba9816590 con 0x556ba9816950 2024-04-24T08:11:50.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.642+0000 7eff0e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba98a27c0 0x556ba9897f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.642+0000 7eff0e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba98a27c0 0x556ba9897f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57294/0 (socket says 172.21.15.69:57294) 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.642+0000 7eff0e200640 1 -- 172.21.15.69:0/4033080455 learned_addr learned my addr 172.21.15.69:0/4033080455 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.642+0000 7eff0e200640 1 -- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ba9816950 msgr2=0x556ba9885830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.642+0000 7eff0e200640 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ba9816950 0x556ba9885830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.642+0000 7eff0e200640 1 -- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ba9818570 msgr2=0x556ba9899d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.642+0000 7eff04e00640 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ba9818570 0x556ba9899d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.642+0000 7eff0e200640 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ba9818570 0x556ba9899d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.642+0000 7eff0e200640 1 -- 172.21.15.69:0/4033080455 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ba989a260 con 0x556ba98a27c0 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.642+0000 7eff0e200640 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba98a27c0 0x556ba9897f10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7eff0800af90 tx=0x7eff08008840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.643+0000 7eff06a00640 1 -- 172.21.15.69:0/4033080455 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff080067a0 con 0x556ba98a27c0 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.643+0000 7eff06a00640 1 -- 172.21.15.69:0/4033080455 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff08006940 con 0x556ba98a27c0 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.643+0000 7eff10586f40 1 -- 172.21.15.69:0/4033080455 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556ba989a7d0 con 0x556ba98a27c0 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.643+0000 7eff06a00640 1 -- 172.21.15.69:0/4033080455 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff08006d30 con 0x556ba98a27c0 2024-04-24T08:11:50.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.643+0000 7eff10586f40 1 -- 172.21.15.69:0/4033080455 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ba989ad90 con 0x556ba98a27c0 2024-04-24T08:11:50.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.645+0000 7eff06a00640 1 -- 172.21.15.69:0/4033080455 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7eff0801c440 con 0x556ba98a27c0 2024-04-24T08:11:50.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.646+0000 7eff06a00640 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efeec06b2d0 0x7efeec06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.646+0000 7eff06a00640 1 -- 172.21.15.69:0/4033080455 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7eff0808a870 con 0x556ba98a27c0 2024-04-24T08:11:50.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.646+0000 7eff10586f40 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556ba97fca60 0x556ba9947870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.646+0000 7eff10586f40 1 -- 172.21.15.69:0/4033080455 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:1 9.7 9:e2e07930:::10000000001.00000000:head [setxattr parent (26) in=32b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57) -- 0x556ba9947db0 con 0x556ba97fca60 2024-04-24T08:11:50.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.647+0000 7eff0d800640 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efeec06b2d0 0x7efeec06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.647+0000 7eff04e00640 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556ba97fca60 0x556ba9947870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.648+0000 7eff0d800640 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efeec06b2d0 0x7efeec06d790 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7eff00056000 tx=0x7eff00050000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.648+0000 7eff04e00640 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556ba97fca60 0x556ba9947870 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.651+0000 7eff04e00640 1 -- 172.21.15.69:0/4033080455 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (26)] v57'2 uv2 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7efef8008840 con 0x556ba97fca60 2024-04-24T08:11:50.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff10586f40 1 -- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556ba97fca60 msgr2=0x556ba9947870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff10586f40 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556ba97fca60 0x556ba9947870 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff10586f40 1 -- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efeec06b2d0 msgr2=0x7efeec06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff10586f40 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efeec06b2d0 0x7efeec06d790 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7eff00056000 tx=0x7eff00050000 comp rx=0 tx=0).stop 2024-04-24T08:11:50.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff10586f40 1 -- 172.21.15.69:0/4033080455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba98a27c0 msgr2=0x556ba9897f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff10586f40 1 --2- 172.21.15.69:0/4033080455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ba98a27c0 0x556ba9897f10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7eff0800af90 tx=0x7eff08008840 comp rx=0 tx=0).stop 2024-04-24T08:11:50.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff0e200640 1 -- 172.21.15.69:0/4033080455 reap_dead start 2024-04-24T08:11:50.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff10586f40 1 -- 172.21.15.69:0/4033080455 shutdown_connections 2024-04-24T08:11:50.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff10586f40 1 -- 172.21.15.69:0/4033080455 >> 172.21.15.69:0/4033080455 conn(0x556ba97f9730 msgr2=0x556ba992bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff10586f40 1 -- 172.21.15.69:0/4033080455 shutdown_connections 2024-04-24T08:11:50.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.652+0000 7eff10586f40 1 -- 172.21.15.69:0/4033080455 wait complete. 2024-04-24T08:11:50.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:50.657 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub start / recursive 2024-04-24T08:11:50.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.854+0000 7fc084000640 1 Processor -- start 2024-04-24T08:11:50.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.855+0000 7fc084000640 1 -- start start 2024-04-24T08:11:50.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.856+0000 7fc084000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c0b3fb0 0x7fc07c15c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.856+0000 7fc084000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc07c15ca40 con 0x7fc07c0aeb00 2024-04-24T08:11:50.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.856+0000 7fc084000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc07c0b31f0 con 0x7fc07c0d0ed0 2024-04-24T08:11:50.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.856+0000 7fc084000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc07c0b34d0 con 0x7fc07c0b3fb0 2024-04-24T08:11:50.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.856+0000 7fc081600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc07c0aeb00 0x7fc07c0d07c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56698/0 (socket says 172.21.15.69:56698) 2024-04-24T08:11:50.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.856+0000 7fc081600640 1 -- 172.21.15.69:0/3498340403 learned_addr learned my addr 172.21.15.69:0/3498340403 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:50.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc082000640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c0b3fb0 0x7fc07c15c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3861946837 0 0) 0x7fc07c15ca40 con 0x7fc07c0aeb00 2024-04-24T08:11:50.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc060003520 con 0x7fc07c0aeb00 2024-04-24T08:11:50.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3376596785 0 0) 0x7fc060003520 con 0x7fc07c0aeb00 2024-04-24T08:11:50.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c0b3fb0 msgr2=0x7fc07c15c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc07be00640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c0b3fb0 0x7fc07c15c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 >> v1:172.21.15.121:6789/0 conn(0x7fc07c0d0ed0 legacy=0x7fc07c15bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c0b37b0 con 0x7fc07c0aeb00 2024-04-24T08:11:50.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fc06c001670 con 0x7fc07c0aeb00 2024-04-24T08:11:50.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 >> v1:172.21.15.69:6789/0 conn(0x7fc07c0aeb00 legacy=0x7fc07c0d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.857+0000 7fc07be00640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c0b3fb0 0x7fc0600039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc07be00640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060003fb0 0x7fc060004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc07be00640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc060004d40 0x7fc0600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c0b34d0 con 0x7fc060004d40 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc06000dd10 con 0x7fc060003fb0 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc06000df10 con 0x7fc07c0b3fb0 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc082000640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc060004d40 0x7fc0600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc081600640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c0b3fb0 0x7fc0600039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc082000640 1 -- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c0b3fb0 msgr2=0x7fc0600039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc082000640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c0b3fb0 0x7fc0600039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc082000640 1 -- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060003fb0 msgr2=0x7fc060004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc080c00640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060003fb0 0x7fc060004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc082000640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060003fb0 0x7fc060004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc082000640 1 -- 172.21.15.69:0/3498340403 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc060003520 con 0x7fc060004d40 2024-04-24T08:11:50.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.858+0000 7fc080c00640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060003fb0 0x7fc060004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.859+0000 7fc082000640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc060004d40 0x7fc0600051c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc074009f90 tx=0x7fc060005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9df892f9e0895b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.860+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074009c10 con 0x7fc060004d40 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.860+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc074036070 con 0x7fc060004d40 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.860+0000 7fc07be00640 1 -- 172.21.15.69:0/3498340403 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074031850 con 0x7fc060004d40 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.860+0000 7fc084000640 1 -- 172.21.15.69:0/3498340403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc060004d40 msgr2=0x7fc0600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.860+0000 7fc084000640 1 --2- 172.21.15.69:0/3498340403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc060004d40 0x7fc0600051c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc074009f90 tx=0x7fc060005830 comp rx=0 tx=0).stop 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.860+0000 7fc082000640 1 -- 172.21.15.69:0/3498340403 reap_dead start 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.860+0000 7fc084000640 1 -- 172.21.15.69:0/3498340403 shutdown_connections 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.860+0000 7fc084000640 1 -- 172.21.15.69:0/3498340403 >> 172.21.15.69:0/3498340403 conn(0x7fc07c06be40 msgr2=0x7fc07c06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.860+0000 7fc084000640 1 -- 172.21.15.69:0/3498340403 shutdown_connections 2024-04-24T08:11:50.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.861+0000 7fc084000640 1 -- 172.21.15.69:0/3498340403 wait complete. 2024-04-24T08:11:50.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.862+0000 7fc084000640 1 Processor -- start 2024-04-24T08:11:50.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.862+0000 7fc084000640 1 -- start start 2024-04-24T08:11:50.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.863+0000 7fc084000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060004d40 0x7fc07c0b19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.863+0000 7fc084000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc07c0b1020 0x7fc07c159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.863+0000 7fc084000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c159860 0x7fc07c072d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.863+0000 7fc084000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c0aeee0 con 0x7fc07c0b1020 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.863+0000 7fc084000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c15c3e0 con 0x7fc060004d40 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.863+0000 7fc084000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc07c0aff80 con 0x7fc07c159860 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc080c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc07c0b1020 0x7fc07c159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc080c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc07c0b1020 0x7fc07c159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57316/0 (socket says 172.21.15.69:57316) 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc080c00640 1 -- 172.21.15.69:0/621596703 learned_addr learned my addr 172.21.15.69:0/621596703 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc082000640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c159860 0x7fc07c072d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc081600640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060004d40 0x7fc07c0b19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc080c00640 1 -- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c159860 msgr2=0x7fc07c072d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc080c00640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c159860 0x7fc07c072d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc080c00640 1 -- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060004d40 msgr2=0x7fc07c0b19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc080c00640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060004d40 0x7fc07c0b19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc080c00640 1 -- 172.21.15.69:0/621596703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c0b0260 con 0x7fc07c0b1020 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc081600640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060004d40 0x7fc07c0b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:50.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc082000640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c159860 0x7fc07c072d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:50.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.864+0000 7fc080c00640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc07c0b1020 0x7fc07c159320 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc068000c00 tx=0x7fc06800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.865+0000 7fc079600640 1 -- 172.21.15.69:0/621596703 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc068019070 con 0x7fc07c0b1020 2024-04-24T08:11:50.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.865+0000 7fc079600640 1 -- 172.21.15.69:0/621596703 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc068005270 con 0x7fc07c0b1020 2024-04-24T08:11:50.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.865+0000 7fc079600640 1 -- 172.21.15.69:0/621596703 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc068007250 con 0x7fc07c0b1020 2024-04-24T08:11:50.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.865+0000 7fc084000640 1 -- 172.21.15.69:0/621596703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc07c0b0540 con 0x7fc07c0b1020 2024-04-24T08:11:50.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.865+0000 7fc084000640 1 -- 172.21.15.69:0/621596703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc07c0b0b00 con 0x7fc07c0b1020 2024-04-24T08:11:50.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.867+0000 7fc079600640 1 -- 172.21.15.69:0/621596703 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc0680073f0 con 0x7fc07c0b1020 2024-04-24T08:11:50.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.869+0000 7fc079600640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc05c06c8f0 0x7fc05c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.869+0000 7fc079600640 1 -- 172.21.15.69:0/621596703 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fc06808b340 con 0x7fc07c0b1020 2024-04-24T08:11:50.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.869+0000 7fc086d6d740 1 Processor -- start 2024-04-24T08:11:50.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.870+0000 7fc081600640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc05c06c8f0 0x7fc05c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.870+0000 7fc086d6d740 1 -- start start 2024-04-24T08:11:50.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.870+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20e4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.870+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20b8410 0x5606b20e0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.870+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20e38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc086d6d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5606b201cf50 con 0x5606b20e1020 2024-04-24T08:11:50.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc086d6d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5606b1fc32c0 con 0x5606b20e0700 2024-04-24T08:11:50.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc086d6d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5606b2018f10 con 0x5606b20b8410 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc081600640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc05c06c8f0 0x7fc05c06edb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0017e0 tx=0x7fc06c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc082000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20e38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc082000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20e38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57320/0 (socket says 172.21.15.69:57320) 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc082000640 1 -- 172.21.15.69:0/4107410793 learned_addr learned my addr 172.21.15.69:0/4107410793 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc081600640 1 --2- 172.21.15.69:0/4107410793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20e4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc082000640 1 -- 172.21.15.69:0/4107410793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20b8410 msgr2=0x5606b20e0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc080c00640 1 --2- 172.21.15.69:0/4107410793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20b8410 0x5606b20e0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc082000640 1 --2- 172.21.15.69:0/4107410793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20b8410 0x5606b20e0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc082000640 1 -- 172.21.15.69:0/4107410793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 msgr2=0x5606b20e4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc082000640 1 --2- 172.21.15.69:0/4107410793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20e4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc082000640 1 -- 172.21.15.69:0/4107410793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606b1dfb7b0 con 0x5606b20e1020 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.871+0000 7fc081600640 1 --2- 172.21.15.69:0/4107410793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20e4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.872+0000 7fc082000640 1 --2- 172.21.15.69:0/4107410793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20e38b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc07400e200 tx=0x7fc07402d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344043e97d54d6e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.872+0000 7fc071c00640 1 -- 172.21.15.69:0/4107410793 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074003b90 con 0x5606b20e1020 2024-04-24T08:11:50.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.872+0000 7fc071c00640 1 -- 172.21.15.69:0/4107410793 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc074040070 con 0x5606b20e1020 2024-04-24T08:11:50.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.872+0000 7fc071c00640 1 -- 172.21.15.69:0/4107410793 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074004030 con 0x5606b20e1020 2024-04-24T08:11:50.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.873+0000 7fc086d6d740 1 -- 172.21.15.69:0/4107410793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 msgr2=0x5606b20e38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.873+0000 7fc086d6d740 1 --2- 172.21.15.69:0/4107410793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20e38b0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc07400e200 tx=0x7fc07402d000 comp rx=0 tx=0).stop 2024-04-24T08:11:50.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.873+0000 7fc086d6d740 1 -- 172.21.15.69:0/4107410793 shutdown_connections 2024-04-24T08:11:50.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.873+0000 7fc086d6d740 1 --2- 172.21.15.69:0/4107410793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20e38b0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.873+0000 7fc086d6d740 1 --2- 172.21.15.69:0/4107410793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20b8410 0x5606b20e0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.873+0000 7fc086d6d740 1 --2- 172.21.15.69:0/4107410793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20e4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.873+0000 7fc086d6d740 1 -- 172.21.15.69:0/4107410793 >> 172.21.15.69:0/4107410793 conn(0x5606b20b6010 msgr2=0x5606b20fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.873+0000 7fc086d6d740 1 -- 172.21.15.69:0/4107410793 shutdown_connections 2024-04-24T08:11:50.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.873+0000 7fc086d6d740 1 -- 172.21.15.69:0/4107410793 wait complete. 2024-04-24T08:11:50.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.874+0000 7fc086d6d740 1 -- start start 2024-04-24T08:11:50.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.874+0000 7fc086d6d740 1 Processor -- start 2024-04-24T08:11:50.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.875+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20d4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.875+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e1020 0x5606b20ded90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.875+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b20e2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.875+0000 7fc086d6d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5606b20b4360 con 0x5606b20d5220 2024-04-24T08:11:50.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.875+0000 7fc086d6d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5606b1fd6680 con 0x5606b20e0700 2024-04-24T08:11:50.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.875+0000 7fc086d6d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5606b20b3da0 con 0x5606b20e1020 2024-04-24T08:11:50.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.875+0000 7fc086d6d740 10 client.0 _ll_register_callbacks cb 0x5606b20134f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.875+0000 7fc082000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b20e2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.875+0000 7fc082000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b20e2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57326/0 (socket says 172.21.15.69:57326) 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.875+0000 7fc082000640 1 -- 172.21.15.69:0/3408656463 learned_addr learned my addr 172.21.15.69:0/3408656463 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc082000640 1 -- 172.21.15.69:0/3408656463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e1020 msgr2=0x5606b20ded90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc082000640 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e1020 0x5606b20ded90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc082000640 1 -- 172.21.15.69:0/3408656463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 msgr2=0x5606b20d4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc081600640 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20d4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc082000640 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20d4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc082000640 1 -- 172.21.15.69:0/3408656463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606b1e5bf40 con 0x5606b20d5220 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc082000640 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b20e2d00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc07400e230 tx=0x7fc074004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4563cdd821305636 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc066000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc066000640 1 -- 172.21.15.69:0/3408656463 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074004240 con 0x5606b20d5220 2024-04-24T08:11:50.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc066000640 1 -- 172.21.15.69:0/3408656463 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fc0740043e0 con 0x5606b20d5220 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc086d6d740 1 -- 172.21.15.69:0/3408656463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc074040070 con 0x5606b20d5220 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc066000640 1 -- 172.21.15.69:0/3408656463 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc074058050 con 0x5606b20d5220 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.876+0000 7fc086d6d740 10 client.6420 fetch_fsmap learned FSMap version 33 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.877+0000 7fc086d6d740 1 -- 172.21.15.69:0/3408656463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) -- 0x5606b1cef0a0 con 0x5606b20d5220 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.877+0000 7fc066000640 1 -- 172.21.15.69:0/3408656463 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc074040070 con 0x5606b20d5220 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.877+0000 7fc066000640 1 -- 172.21.15.69:0/3408656463 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 33) ==== 3494+0+0 (secure 0 0 0) 0x7fc07405b020 con 0x5606b20d5220 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.877+0000 7fc086d6d740 10 client.6420 fetch_fsmap finished waiting for FSMap version 33 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.877+0000 7fc086d6d740 10 client.6420 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.877+0000 7fc086d6d740 20 client.6420 populate_metadata read hostname 'smithi069' 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.877+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606b20dce20 0x5606b20d5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.877+0000 7fc086d6d740 4 client.6420 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.877+0000 7fc086d6d740 1 -- 172.21.15.69:0/3408656463 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606b202a970 con 0x5606b20dce20 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.877+0000 7fc081600640 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606b20dce20 0x5606b20d5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.878+0000 7fc081600640 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606b20dce20 0x5606b20d5e50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a002ef4dc16b735 server_cookie=2451caa35a56a4ab in_seq=0 out_seq=0 2024-04-24T08:11:50.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.878+0000 7fc066000640 10 client.6420 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc066000640 1 -- 172.21.15.69:0/3408656463 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5606b202a970 con 0x5606b20dce20 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc066000640 10 client.6420 handle_command_reply: tid=0 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 client.6420 shutdown 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 -- 172.21.15.69:0/3408656463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 msgr2=0x5606b20e2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b20e2d00 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc07400e230 tx=0x7fc074004ab0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 -- 172.21.15.69:0/3408656463 shutdown_connections 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606b20dce20 0x5606b20d5e50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b20e2d00 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e1020 0x5606b20ded90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3408656463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20d4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 -- 172.21.15.69:0/3408656463 >> 172.21.15.69:0/3408656463 conn(0x5606b20b8410 msgr2=0x5606b20b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc066000640 0 client.6420 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:50.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 -- 172.21.15.69:0/3408656463 shutdown_connections 2024-04-24T08:11:50.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 1 -- 172.21.15.69:0/3408656463 wait complete. 2024-04-24T08:11:50.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.879+0000 7fc086d6d740 20 client.6420 trim_cache size 0 max 16384 2024-04-24T08:11:50.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.899+0000 7fc086d6d740 1 Processor -- start 2024-04-24T08:11:50.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.899+0000 7fc086d6d740 1 -- start start 2024-04-24T08:11:50.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20d5220 0x5606b2101040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e0700 0x5606b20b8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc086d6d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c1fd2b0 con 0x5606b20e1020 2024-04-24T08:11:50.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc086d6d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c1fc380 con 0x5606b20d5220 2024-04-24T08:11:50.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc086d6d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc07c158fd0 con 0x5606b20e0700 2024-04-24T08:11:50.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc082000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc082000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57342/0 (socket says 172.21.15.69:57342) 2024-04-24T08:11:50.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc082000640 1 -- 172.21.15.69:0/3003949193 learned_addr learned my addr 172.21.15.69:0/3003949193 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:50.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc082000640 1 -- 172.21.15.69:0/3003949193 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e0700 msgr2=0x5606b20b8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc082000640 1 --2- 172.21.15.69:0/3003949193 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e0700 0x5606b20b8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc082000640 1 -- 172.21.15.69:0/3003949193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20d5220 msgr2=0x5606b2101040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc082000640 1 --2- 172.21.15.69:0/3003949193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20d5220 0x5606b2101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.900+0000 7fc082000640 1 -- 172.21.15.69:0/3003949193 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606b20e3070 con 0x5606b20e1020 2024-04-24T08:11:50.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.901+0000 7fc082000640 1 --2- 172.21.15.69:0/3003949193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20af6e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fc07400c090 tx=0x7fc07404b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d92029e638c8f5c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.901+0000 7fc071200640 1 -- 172.21.15.69:0/3003949193 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc074049040 con 0x5606b20e1020 2024-04-24T08:11:50.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.901+0000 7fc071200640 1 -- 172.21.15.69:0/3003949193 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc074040040 con 0x5606b20e1020 2024-04-24T08:11:50.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.901+0000 7fc071200640 1 -- 172.21.15.69:0/3003949193 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0740456a0 con 0x5606b20e1020 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.901+0000 7fc086d6d740 1 -- 172.21.15.69:0/3003949193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 msgr2=0x5606b20af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.901+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3003949193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20af6e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fc07400c090 tx=0x7fc07404b000 comp rx=0 tx=0).stop 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.902+0000 7fc086d6d740 1 -- 172.21.15.69:0/3003949193 shutdown_connections 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.902+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3003949193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20e1020 0x5606b20af6e0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.902+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3003949193 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e0700 0x5606b20b8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.902+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3003949193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20d5220 0x5606b2101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.902+0000 7fc086d6d740 1 -- 172.21.15.69:0/3003949193 >> 172.21.15.69:0/3003949193 conn(0x5606b20b8410 msgr2=0x5606b20d4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.902+0000 7fc086d6d740 1 -- 172.21.15.69:0/3003949193 shutdown_connections 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.902+0000 7fc086d6d740 1 -- 172.21.15.69:0/3003949193 wait complete. 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.902+0000 7fc086d6d740 1 -- start start 2024-04-24T08:11:50.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.903+0000 7fc086d6d740 1 Processor -- start 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b217cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20d3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc086d6d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e1020 0x5606b20d3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc086d6d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5606b20e6f10 con 0x5606b20d5220 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc086d6d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c0c54c0 con 0x5606b20e0700 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc086d6d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5606b20bafe0 con 0x5606b20e1020 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc086d6d740 10 client.0 _ll_register_callbacks cb 0x7fc0600045d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc081600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b217cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc081600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b217cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57346/0 (socket says 172.21.15.69:57346) 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc081600640 1 -- 172.21.15.69:0/3317943407 learned_addr learned my addr 172.21.15.69:0/3317943407 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc081600640 1 -- 172.21.15.69:0/3317943407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e1020 msgr2=0x5606b20d3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc081600640 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e1020 0x5606b20d3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc081600640 1 -- 172.21.15.69:0/3317943407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 msgr2=0x5606b20d3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:50.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc081600640 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc081600640 1 -- 172.21.15.69:0/3317943407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606b20b7800 con 0x5606b20d5220 2024-04-24T08:11:50.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc081600640 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b217cc70 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00e590 tx=0x7fc06c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea7072794f353b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:50.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.904+0000 7fc066000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:11:50.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc066000640 1 -- 172.21.15.69:0/3317943407 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc06c0074f0 con 0x5606b20d5220 2024-04-24T08:11:50.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc066000640 1 -- 172.21.15.69:0/3317943407 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fc06c007690 con 0x5606b20d5220 2024-04-24T08:11:50.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc086d6d740 1 -- 172.21.15.69:0/3317943407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc074040040 con 0x5606b20d5220 2024-04-24T08:11:50.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc066000640 1 -- 172.21.15.69:0/3317943407 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fc06c024070 con 0x5606b20d5220 2024-04-24T08:11:50.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc086d6d740 10 client.6426 fetch_fsmap learned FSMap version 33 2024-04-24T08:11:50.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc086d6d740 1 -- 172.21.15.69:0/3317943407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) -- 0x5606b20b3d90 con 0x5606b20d5220 2024-04-24T08:11:50.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc066000640 1 -- 172.21.15.69:0/3317943407 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc074040040 con 0x5606b20d5220 2024-04-24T08:11:50.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc066000640 1 -- 172.21.15.69:0/3317943407 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 33) ==== 3494+0+0 (secure 0 0 0) 0x7fc06c024210 con 0x5606b20d5220 2024-04-24T08:11:50.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc086d6d740 10 client.6426 fetch_fsmap finished waiting for FSMap version 33 2024-04-24T08:11:50.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc086d6d740 10 client.6426 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:11:50.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc086d6d740 20 client.6426 populate_metadata read hostname 'smithi069' 2024-04-24T08:11:50.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606b20d6e00 0x5606b20de8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc086d6d740 4 client.6426 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}] 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.905+0000 7fc086d6d740 1 -- 172.21.15.69:0/3317943407 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}) -- 0x5606b20247e0 con 0x5606b20d6e00 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.906+0000 7fc080c00640 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606b20d6e00 0x5606b20de8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.906+0000 7fc080c00640 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606b20d6e00 0x5606b20de8d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65e8ea8ba55adf9a server_cookie=b4286af4c26d0fcf in_seq=0 out_seq=0 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.906+0000 7fc066000640 10 client.6426 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc066000640 1 -- 172.21.15.69:0/3317943407 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5606b20247e0 con 0x5606b20d6e00 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc066000640 10 client.6426 handle_command_reply: tid=0 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc086d6d740 1 client.6426 shutdown 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc086d6d740 1 -- 172.21.15.69:0/3317943407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 msgr2=0x5606b217cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b217cc70 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00e590 tx=0x7fc06c000ed0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc086d6d740 1 -- 172.21.15.69:0/3317943407 shutdown_connections 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606b20d6e00 0x5606b20de8d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5606b20e1020 0x5606b20d3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606b20e0700 0x5606b20d3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc066000640 0 client.6426 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:50.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc086d6d740 1 --2- 172.21.15.69:0/3317943407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5606b20d5220 0x5606b217cc70 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.907+0000 7fc086d6d740 1 -- 172.21.15.69:0/3317943407 >> 172.21.15.69:0/3317943407 conn(0x5606b20b8410 msgr2=0x5606b20b7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.908+0000 7fc086d6d740 1 -- 172.21.15.69:0/3317943407 shutdown_connections 2024-04-24T08:11:50.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.908+0000 7fc086d6d740 1 -- 172.21.15.69:0/3317943407 wait complete. 2024-04-24T08:11:50.910 INFO:tasks.ceph.mds.c.smithi069.stderr:2024-04-24T08:11:50.908+0000 7f042c800640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={6384=0-4194304@1} caps={6384=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6384 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x563937a40b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-24T08:11:50.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.908+0000 7fc086d6d740 20 client.6426 trim_cache size 0 max 16384 2024-04-24T08:11:50.911 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:11:50.911 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-04-24T08:11:50.911 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "b5212086-1aea-4406-86b2-7af8c80734a4", 2024-04-24T08:11:50.911 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-04-24T08:11:50.911 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.908+0000 7fc084000640 1 -- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc05c06c8f0 msgr2=0x7fc05c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.908+0000 7fc084000640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc05c06c8f0 0x7fc05c06edb0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc06c0017e0 tx=0x7fc06c009040 comp rx=0 tx=0).stop 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.908+0000 7fc084000640 1 -- 172.21.15.69:0/621596703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc07c0b1020 msgr2=0x7fc07c159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.908+0000 7fc084000640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc07c0b1020 0x7fc07c159320 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc068000c00 tx=0x7fc06800dc00 comp rx=0 tx=0).stop 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.909+0000 7fc084000640 1 -- 172.21.15.69:0/621596703 shutdown_connections 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.909+0000 7fc084000640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc05c06c8f0 0x7fc05c06edb0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.909+0000 7fc084000640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc07c0b1020 0x7fc07c159320 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.909+0000 7fc084000640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc07c159860 0x7fc07c072d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.909+0000 7fc084000640 1 --2- 172.21.15.69:0/621596703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc060004d40 0x7fc07c0b19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.909+0000 7fc084000640 1 -- 172.21.15.69:0/621596703 >> 172.21.15.69:0/621596703 conn(0x7fc07c06be40 msgr2=0x7fc07c15d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.909+0000 7fc084000640 1 -- 172.21.15.69:0/621596703 shutdown_connections 2024-04-24T08:11:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:50.909+0000 7fc084000640 1 -- 172.21.15.69:0/621596703 wait complete. 2024-04-24T08:11:50.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:50.926 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-04-24T08:11:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.125+0000 7f8a38600640 1 Processor -- start 2024-04-24T08:11:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.125+0000 7f8a38600640 1 -- start start 2024-04-24T08:11:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.127+0000 7f8a38600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300b3e90 0x7f8a300b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.127+0000 7f8a38600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8a300ce250 0x7f8a300ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.127+0000 7f8a38600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a3015cc00 con 0x7f8a3015c820 2024-04-24T08:11:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.127+0000 7f8a38600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8a300b4be0 con 0x7f8a300ce250 2024-04-24T08:11:51.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.127+0000 7f8a38600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8a300b42f0 con 0x7f8a300b3e90 2024-04-24T08:11:51.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.128+0000 7f8a35200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300b3e90 0x7f8a300b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.128+0000 7f8a35c00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f8a3015c820 0x7f8a3006f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:58758/0 (socket says 172.21.15.69:58758) 2024-04-24T08:11:51.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.128+0000 7f8a35c00640 1 -- 172.21.15.69:0/2291070530 learned_addr learned my addr 172.21.15.69:0/2291070530 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:11:51.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.128+0000 7f8a36600640 1 --2- 172.21.15.69:0/2291070530 >> v2:172.21.15.121:3300/0 conn(0x7f8a300ce250 0x7f8a300ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.128+0000 7f8a2fe00640 1 -- 172.21.15.69:0/2291070530 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2112262187 0 0) 0x7f8a3015cc00 con 0x7f8a3015c820 2024-04-24T08:11:51.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.129+0000 7f8a2fe00640 1 -- 172.21.15.69:0/2291070530 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a18003520 con 0x7f8a3015c820 2024-04-24T08:11:51.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.129+0000 7f8a35200640 1 -- 172.21.15.69:0/2291070530 >> v2:172.21.15.121:3300/0 conn(0x7f8a300ce250 msgr2=0x7f8a300ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:51.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.129+0000 7f8a35200640 1 --2- 172.21.15.69:0/2291070530 >> v2:172.21.15.121:3300/0 conn(0x7f8a300ce250 0x7f8a300ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.129+0000 7f8a35200640 1 -- 172.21.15.69:0/2291070530 >> v1:172.21.15.121:6789/0 conn(0x7f8a3015c820 legacy=0x7f8a3006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:51.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.129+0000 7f8a35200640 1 -- 172.21.15.69:0/2291070530 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3006ed30 con 0x7f8a300b3e90 2024-04-24T08:11:51.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.130+0000 7f8a35200640 1 --2- 172.21.15.69:0/2291070530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300b3e90 0x7f8a300b46a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0097a0 tx=0x7f8a1c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a83a1dce5e0fc14b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:51.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.130+0000 7f8a2fe00640 1 -- 172.21.15.69:0/2291070530 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c037070 con 0x7f8a300b3e90 2024-04-24T08:11:51.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.131+0000 7f8a2fe00640 1 -- 172.21.15.69:0/2291070530 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a1c032440 con 0x7f8a300b3e90 2024-04-24T08:11:51.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.131+0000 7f8a2fe00640 1 -- 172.21.15.69:0/2291070530 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c032810 con 0x7f8a300b3e90 2024-04-24T08:11:51.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.131+0000 7f8a38600640 1 -- 172.21.15.69:0/2291070530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300b3e90 msgr2=0x7f8a300b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:51.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.131+0000 7f8a38600640 1 --2- 172.21.15.69:0/2291070530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300b3e90 0x7f8a300b46a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0097a0 tx=0x7f8a1c02a180 comp rx=0 tx=0).stop 2024-04-24T08:11:51.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.131+0000 7f8a38600640 1 -- 172.21.15.69:0/2291070530 shutdown_connections 2024-04-24T08:11:51.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.131+0000 7f8a38600640 1 --2- 172.21.15.69:0/2291070530 >> v2:172.21.15.121:3300/0 conn(0x7f8a300ce250 0x7f8a300ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.131+0000 7f8a38600640 1 --2- 172.21.15.69:0/2291070530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300b3e90 0x7f8a300b46a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.131+0000 7f8a38600640 1 -- 172.21.15.69:0/2291070530 >> 172.21.15.69:0/2291070530 conn(0x7f8a300d0350 msgr2=0x7f8a300d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:51.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.131+0000 7f8a38600640 1 -- 172.21.15.69:0/2291070530 shutdown_connections 2024-04-24T08:11:51.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.132+0000 7f8a38600640 1 -- 172.21.15.69:0/2291070530 wait complete. 2024-04-24T08:11:51.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.133+0000 7f8a38600640 1 Processor -- start 2024-04-24T08:11:51.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.133+0000 7f8a38600640 1 -- start start 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.134+0000 7f8a38600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300b3e90 0x7f8a3015e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.134+0000 7f8a38600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300ce250 0x7f8a301598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.134+0000 7f8a38600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a3015c820 0x7f8a301f4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.134+0000 7f8a38600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3006efd0 con 0x7f8a300b3e90 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.134+0000 7f8a38600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8a300b50f0 con 0x7f8a3015c820 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.134+0000 7f8a38600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8a3006f2b0 con 0x7f8a300ce250 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.135+0000 7f8a35c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300b3e90 0x7f8a3015e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.135+0000 7f8a35c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300b3e90 0x7f8a3015e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57352/0 (socket says 172.21.15.69:57352) 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.135+0000 7f8a35c00640 1 -- 172.21.15.69:0/3560776374 learned_addr learned my addr 172.21.15.69:0/3560776374 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.135+0000 7f8a35200640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300ce250 0x7f8a301598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.135+0000 7f8a36600640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a3015c820 0x7f8a301f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.135+0000 7f8a35c00640 1 -- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300ce250 msgr2=0x7f8a301598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:51.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.135+0000 7f8a35c00640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300ce250 0x7f8a301598f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.135+0000 7f8a35c00640 1 -- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a3015c820 msgr2=0x7f8a301f4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:51.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.135+0000 7f8a35c00640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a3015c820 0x7f8a301f4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.135+0000 7f8a35c00640 1 -- 172.21.15.69:0/3560776374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a30159e30 con 0x7f8a300b3e90 2024-04-24T08:11:51.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.136+0000 7f8a35c00640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300b3e90 0x7f8a3015e2f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400cbc0 tx=0x7f8a2400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:51.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.136+0000 7f8a2d600640 1 -- 172.21.15.69:0/3560776374 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24017070 con 0x7f8a300b3e90 2024-04-24T08:11:51.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.136+0000 7f8a2d600640 1 -- 172.21.15.69:0/3560776374 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2400ad40 con 0x7f8a300b3e90 2024-04-24T08:11:51.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.136+0000 7f8a38600640 1 -- 172.21.15.69:0/3560776374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a300b0340 con 0x7f8a300b3e90 2024-04-24T08:11:51.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.136+0000 7f8a2d600640 1 -- 172.21.15.69:0/3560776374 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24012620 con 0x7f8a300b3e90 2024-04-24T08:11:51.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.136+0000 7f8a38600640 1 -- 172.21.15.69:0/3560776374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a300b0900 con 0x7f8a300b3e90 2024-04-24T08:11:51.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.139+0000 7f8a2d600640 1 -- 172.21.15.69:0/3560776374 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8a2401a070 con 0x7f8a300b3e90 2024-04-24T08:11:51.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.140+0000 7f8a2d600640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a1006c8f0 0x7f8a1006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.140+0000 7f8a2d600640 1 -- 172.21.15.69:0/3560776374 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f8a2408ba60 con 0x7f8a300b3e90 2024-04-24T08:11:51.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.141+0000 7f8a3b3ac740 1 Processor -- start 2024-04-24T08:11:51.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.141+0000 7f8a35200640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a1006c8f0 0x7f8a1006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.141+0000 7f8a3b3ac740 1 -- start start 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.142+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 0x557bd83075c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.142+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd83370d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.142+0000 7f8a35200640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a1006c8f0 0x7f8a1006edb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c034000 tx=0x7f8a1c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.142+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8337610 0x557bd830c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.142+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557bd82711f0 con 0x557bd8337610 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.142+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557bd8217350 con 0x557bd8354c30 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.142+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557bd826d1b0 con 0x557bd8337d50 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a36600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8337610 0x557bd830c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a36600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8337610 0x557bd830c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57368/0 (socket says 172.21.15.69:57368) 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a36600640 1 -- 172.21.15.69:0/1264000124 learned_addr learned my addr 172.21.15.69:0/1264000124 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a35c00640 1 --2- 172.21.15.69:0/1264000124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 0x557bd83075c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a35200640 1 --2- 172.21.15.69:0/1264000124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd83370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a36600640 1 -- 172.21.15.69:0/1264000124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 msgr2=0x557bd83370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:51.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a36600640 1 --2- 172.21.15.69:0/1264000124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd83370d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a36600640 1 -- 172.21.15.69:0/1264000124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 msgr2=0x557bd83075c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:51.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a36600640 1 --2- 172.21.15.69:0/1264000124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 0x557bd83075c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a36600640 1 -- 172.21.15.69:0/1264000124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bd826bf50 con 0x557bd8337610 2024-04-24T08:11:51.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.143+0000 7f8a36600640 1 --2- 172.21.15.69:0/1264000124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8337610 0x557bd830c420 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009ba0 tx=0x7f8a28009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908338216ba9bdf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:51.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a22000640 1 -- 172.21.15.69:0/1264000124 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28018070 con 0x557bd8337610 2024-04-24T08:11:51.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a22000640 1 -- 172.21.15.69:0/1264000124 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a28010e60 con 0x557bd8337610 2024-04-24T08:11:51.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a22000640 1 -- 172.21.15.69:0/1264000124 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2800f040 con 0x557bd8337610 2024-04-24T08:11:51.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/1264000124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8337610 msgr2=0x557bd830c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:51.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/1264000124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8337610 0x557bd830c420 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009ba0 tx=0x7f8a28009190 comp rx=0 tx=0).stop 2024-04-24T08:11:51.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/1264000124 shutdown_connections 2024-04-24T08:11:51.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/1264000124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8337610 0x557bd830c420 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/1264000124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd83370d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/1264000124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 0x557bd83075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/1264000124 >> 172.21.15.69:0/1264000124 conn(0x557bd8309830 msgr2=0x557bd83520d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:51.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.144+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/1264000124 shutdown_connections 2024-04-24T08:11:51.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.145+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/1264000124 wait complete. 2024-04-24T08:11:51.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.145+0000 7f8a3b3ac740 1 -- start start 2024-04-24T08:11:51.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.145+0000 7f8a3b3ac740 1 Processor -- start 2024-04-24T08:11:51.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.146+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd8332ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.146+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8354c30 0x557bd8353e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.146+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd830c780 0x557bd830cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.146+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557bd830b9e0 con 0x557bd8354c30 2024-04-24T08:11:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557bd822a690 con 0x557bd830c780 2024-04-24T08:11:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557bd830add0 con 0x557bd8337d50 2024-04-24T08:11:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a3b3ac740 10 client.0 _ll_register_callbacks cb 0x557bd8266650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a35200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8354c30 0x557bd8353e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a35200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8354c30 0x557bd8353e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57382/0 (socket says 172.21.15.69:57382) 2024-04-24T08:11:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a35200640 1 -- 172.21.15.69:0/2992580168 learned_addr learned my addr 172.21.15.69:0/2992580168 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a35200640 1 -- 172.21.15.69:0/2992580168 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 msgr2=0x557bd8332ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a35200640 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd8332ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a35200640 1 -- 172.21.15.69:0/2992580168 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd830c780 msgr2=0x557bd830cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a35200640 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd830c780 0x557bd830cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.147+0000 7f8a35200640 1 -- 172.21.15.69:0/2992580168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bd8229920 con 0x557bd8354c30 2024-04-24T08:11:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.148+0000 7f8a35200640 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8354c30 0x557bd8353e30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009890 tx=0x7f8a1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a8dd8d602f7f5d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.148+0000 7f8a16a00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.148+0000 7f8a16a00640 1 -- 172.21.15.69:0/2992580168 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c04e060 con 0x557bd8354c30 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.148+0000 7f8a16a00640 1 -- 172.21.15.69:0/2992580168 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f8a1c03e930 con 0x557bd8354c30 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.148+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/2992580168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a28010e60 con 0x557bd8354c30 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.148+0000 7f8a16a00640 1 -- 172.21.15.69:0/2992580168 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8a1c054050 con 0x557bd8354c30 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.148+0000 7f8a3b3ac740 10 client.6435 fetch_fsmap learned FSMap version 33 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.148+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/2992580168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) -- 0x557bd804d7d0 con 0x557bd8354c30 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.148+0000 7f8a16a00640 1 -- 172.21.15.69:0/2992580168 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a28010e60 con 0x557bd8354c30 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.148+0000 7f8a16a00640 1 -- 172.21.15.69:0/2992580168 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 33) ==== 3494+0+0 (secure 0 0 0) 0x7f8a1c057020 con 0x557bd8354c30 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.149+0000 7f8a3b3ac740 10 client.6435 fetch_fsmap finished waiting for FSMap version 33 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.149+0000 7f8a3b3ac740 10 client.6435 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.149+0000 7f8a3b3ac740 20 client.6435 populate_metadata read hostname 'smithi069' 2024-04-24T08:11:51.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.149+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x557bd8338810 0x557bd8338c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.149+0000 7f8a3b3ac740 4 client.6435 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.149+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/2992580168 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bd827ec10 con 0x557bd8338810 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.149+0000 7f8a35c00640 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x557bd8338810 0x557bd8338c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.149+0000 7f8a35c00640 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x557bd8338810 0x557bd8338c90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d6e03ab0daf9afb server_cookie=7e41ce59ab9f7dc5 in_seq=0 out_seq=0 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.149+0000 7f8a16a00640 10 client.6435 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a16a00640 1 -- 172.21.15.69:0/2992580168 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557bd827ec10 con 0x557bd8338810 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a16a00640 10 client.6435 handle_command_reply: tid=0 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a3b3ac740 1 client.6435 shutdown 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/2992580168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8354c30 msgr2=0x557bd8353e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8354c30 0x557bd8353e30 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009890 tx=0x7f8a1c004730 comp rx=0 tx=0).stop 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/2992580168 shutdown_connections 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x557bd8338810 0x557bd8338c90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd8354c30 0x557bd8353e30 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd830c780 0x557bd830cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/2992580168 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd8332ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a16a00640 0 client.6435 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:51.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.150+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/2992580168 >> 172.21.15.69:0/2992580168 conn(0x557bd8337610 msgr2=0x557bd8306c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:51.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.151+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/2992580168 shutdown_connections 2024-04-24T08:11:51.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.151+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/2992580168 wait complete. 2024-04-24T08:11:51.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.151+0000 7f8a3b3ac740 20 client.6435 trim_cache size 0 max 16384 2024-04-24T08:11:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.170+0000 7f8a3b3ac740 1 Processor -- start 2024-04-24T08:11:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.170+0000 7f8a3b3ac740 1 -- start start 2024-04-24T08:11:51.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd833a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd8338bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 0x557bd8339130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a30200be0 con 0x557bd830c780 2024-04-24T08:11:51.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8a301f8840 con 0x557bd8354c30 2024-04-24T08:11:51.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8a301f7ba0 con 0x557bd8337d50 2024-04-24T08:11:51.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a35c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd833a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a35c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd833a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57390/0 (socket says 172.21.15.69:57390) 2024-04-24T08:11:51.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a35c00640 1 -- 172.21.15.69:0/3178390573 learned_addr learned my addr 172.21.15.69:0/3178390573 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a35c00640 1 -- 172.21.15.69:0/3178390573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 msgr2=0x557bd8338bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a35c00640 1 --2- 172.21.15.69:0/3178390573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd8338bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a35c00640 1 -- 172.21.15.69:0/3178390573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 msgr2=0x557bd8339130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a35c00640 1 --2- 172.21.15.69:0/3178390573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 0x557bd8339130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.171+0000 7f8a35c00640 1 -- 172.21.15.69:0/3178390573 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bd7f833f0 con 0x557bd830c780 2024-04-24T08:11:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.172+0000 7f8a35c00640 1 --2- 172.21.15.69:0/3178390573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd833a760 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400a930 tx=0x7f8a240622e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=269fd44873d52160 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.172+0000 7f8a21600640 1 -- 172.21.15.69:0/3178390573 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24070070 con 0x557bd830c780 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.172+0000 7f8a21600640 1 -- 172.21.15.69:0/3178390573 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a24057a00 con 0x557bd830c780 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.172+0000 7f8a21600640 1 -- 172.21.15.69:0/3178390573 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24061e50 con 0x557bd830c780 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.172+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/3178390573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 msgr2=0x557bd833a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.172+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/3178390573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd833a760 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400a930 tx=0x7f8a240622e0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.173+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/3178390573 shutdown_connections 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.173+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/3178390573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 0x557bd8339130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.173+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/3178390573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd8338bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.173+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/3178390573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd833a760 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.173+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/3178390573 >> 172.21.15.69:0/3178390573 conn(0x557bd8337610 msgr2=0x557bd8351260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.173+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/3178390573 shutdown_connections 2024-04-24T08:11:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.173+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/3178390573 wait complete. 2024-04-24T08:11:51.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.173+0000 7f8a3b3ac740 1 -- start start 2024-04-24T08:11:51.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.174+0000 7f8a3b3ac740 1 Processor -- start 2024-04-24T08:11:51.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.174+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd83cb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.174+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd8306390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a3b3ac740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 0x557bd83068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557bd8216790 con 0x557bd830c780 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8a301f81f0 con 0x557bd8354c30 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a3b3ac740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557bd8308be0 con 0x557bd8337d50 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a3b3ac740 10 client.0 _ll_register_callbacks cb 0x7f8a1c035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a35c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd83cb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a35c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd83cb670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57392/0 (socket says 172.21.15.69:57392) 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a35c00640 1 -- 172.21.15.69:0/896076331 learned_addr learned my addr 172.21.15.69:0/896076331 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a35c00640 1 -- 172.21.15.69:0/896076331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 msgr2=0x557bd8306390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a35c00640 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd8306390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a35c00640 1 -- 172.21.15.69:0/896076331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 msgr2=0x557bd83068d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a35c00640 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 0x557bd83068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.175+0000 7f8a35c00640 1 -- 172.21.15.69:0/896076331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bd7f430f0 con 0x557bd830c780 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a35c00640 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd83cb670 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f8a24059400 tx=0x7f8a24001c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d27859ee6f3b30 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a16a00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a16a00640 1 -- 172.21.15.69:0/896076331 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24070040 con 0x557bd830c780 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a16a00640 1 -- 172.21.15.69:0/896076331 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f8a24062050 con 0x557bd830c780 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/896076331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a24057a00 con 0x557bd830c780 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a16a00640 1 -- 172.21.15.69:0/896076331 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f8a24059700 con 0x557bd830c780 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a3b3ac740 10 client.6441 fetch_fsmap learned FSMap version 33 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/896076331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) -- 0x557bd8236810 con 0x557bd830c780 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a16a00640 1 -- 172.21.15.69:0/896076331 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a24057a00 con 0x557bd830c780 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a16a00640 1 -- 172.21.15.69:0/896076331 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 33) ==== 3494+0+0 (secure 0 0 0) 0x7f8a2407fbb0 con 0x557bd830c780 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a3b3ac740 10 client.6441 fetch_fsmap finished waiting for FSMap version 33 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a3b3ac740 10 client.6441 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a3b3ac740 20 client.6441 populate_metadata read hostname 'smithi069' 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x557bd8334360 0x557bd83d85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a3b3ac740 4 client.6441 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.176+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/896076331 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557bd8349a40 con 0x557bd8334360 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.177+0000 7f8a35200640 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x557bd8334360 0x557bd83d85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.177+0000 7f8a35200640 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x557bd8334360 0x557bd83d85d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf4c046c323f9e47 server_cookie=8ff419bd85bdc87b in_seq=0 out_seq=0 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.177+0000 7f8a16a00640 10 client.6441 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.177+0000 7f8a16a00640 1 -- 172.21.15.69:0/896076331 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+253 (crc 0 0 0) 0x557bd8349a40 con 0x557bd8334360 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.177+0000 7f8a16a00640 10 client.6441 handle_command_reply: tid=0 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.177+0000 7f8a3b3ac740 1 client.6441 shutdown 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/896076331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 msgr2=0x557bd83cb670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd83cb670 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f8a24059400 tx=0x7f8a24001c60 comp rx=0 tx=0).stop 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/896076331 shutdown_connections 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x557bd8334360 0x557bd83d85d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557bd8354c30 0x557bd83068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557bd8337d50 0x557bd8306390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a16a00640 0 client.6441 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:11:51.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 1 --2- 172.21.15.69:0/896076331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557bd830c780 0x557bd83cb670 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/896076331 >> 172.21.15.69:0/896076331 conn(0x557bd8337610 msgr2=0x557bd83554f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/896076331 shutdown_connections 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 1 -- 172.21.15.69:0/896076331 wait complete. 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.178+0000 7f8a3b3ac740 20 client.6441 trim_cache size 0 max 16384 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stdout: "b5212086-1aea-4406-86b2-7af8c80734a4": { 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/", 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "b5212086-1aea-4406-86b2-7af8c80734a4", 2024-04-24T08:11:51.182 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive" 2024-04-24T08:11:51.183 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:51.183 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:11:51.183 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:11:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 -- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a1006c8f0 msgr2=0x7f8a1006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a1006c8f0 0x7f8a1006edb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c034000 tx=0x7f8a1c02e000 comp rx=0 tx=0).stop 2024-04-24T08:11:51.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 -- 172.21.15.69:0/3560776374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300b3e90 msgr2=0x7f8a3015e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:51.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300b3e90 0x7f8a3015e2f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400cbc0 tx=0x7f8a2400a430 comp rx=0 tx=0).stop 2024-04-24T08:11:51.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 -- 172.21.15.69:0/3560776374 shutdown_connections 2024-04-24T08:11:51.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8a1006c8f0 0x7f8a1006edb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a3015c820 0x7f8a301f4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a300ce250 0x7f8a301598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 --2- 172.21.15.69:0/3560776374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a300b3e90 0x7f8a3015e2f0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:51.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 -- 172.21.15.69:0/3560776374 >> 172.21.15.69:0/3560776374 conn(0x7f8a300d0350 msgr2=0x7f8a300ceef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:51.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 -- 172.21.15.69:0/3560776374 shutdown_connections 2024-04-24T08:11:51.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:11:51.179+0000 7f8a38600640 1 -- 172.21.15.69:0/3560776374 wait complete. 2024-04-24T08:11:51.197 INFO:tasks.cephfs.filesystem:scrub status for tag:b5212086-1aea-4406-86b2-7af8c80734a4 - {'path': '/', 'tag': 'b5212086-1aea-4406-86b2-7af8c80734a4', 'options': 'recursive'} 2024-04-24T08:11:56.532 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:11:56.532+0000 7fa5d8600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-24T08:12:14.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:12:14.384 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:12:21.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:21.197 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-04-24T08:12:21.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.409+0000 7fcd4c000640 1 Processor -- start 2024-04-24T08:12:21.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.409+0000 7fcd4c000640 1 -- start start 2024-04-24T08:12:21.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.411+0000 7fcd4c000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcd4415c820 0x7fcd440b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.411+0000 7fcd4c000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcd440b46a0 0x7fcd4406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.411+0000 7fcd4c000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd4415cc00 con 0x7fcd4406fd10 2024-04-24T08:12:21.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.411+0000 7fcd4c000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcd4406ed30 con 0x7fcd4415c820 2024-04-24T08:12:21.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.411+0000 7fcd4c000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcd440b4e10 con 0x7fcd440b46a0 2024-04-24T08:12:21.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.411+0000 7fcd4a000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcd4406fd10 0x7fcd440ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:41886/0 (socket says 172.21.15.69:41886) 2024-04-24T08:12:21.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.411+0000 7fcd4a000640 1 -- 172.21.15.69:0/3684034345 learned_addr learned my addr 172.21.15.69:0/3684034345 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:21.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.411+0000 7fcd49600640 1 --2- 172.21.15.69:0/3684034345 >> v2:172.21.15.69:3300/0 conn(0x7fcd4415c820 0x7fcd440b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.412+0000 7fcd48c00640 1 --2- 172.21.15.69:0/3684034345 >> v2:172.21.15.121:3300/0 conn(0x7fcd440b46a0 0x7fcd4406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.412+0000 7fcd43e00640 1 -- 172.21.15.69:0/3684034345 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2218095948 0 0) 0x7fcd4415cc00 con 0x7fcd4406fd10 2024-04-24T08:12:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.412+0000 7fcd43e00640 1 -- 172.21.15.69:0/3684034345 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd2c003520 con 0x7fcd4406fd10 2024-04-24T08:12:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.412+0000 7fcd49600640 1 -- 172.21.15.69:0/3684034345 >> v2:172.21.15.121:3300/0 conn(0x7fcd440b46a0 msgr2=0x7fcd4406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.412+0000 7fcd49600640 1 --2- 172.21.15.69:0/3684034345 >> v2:172.21.15.121:3300/0 conn(0x7fcd440b46a0 0x7fcd4406f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.412+0000 7fcd49600640 1 -- 172.21.15.69:0/3684034345 >> v1:172.21.15.69:6789/0 conn(0x7fcd4406fd10 legacy=0x7fcd440ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.412+0000 7fcd49600640 1 -- 172.21.15.69:0/3684034345 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd440b50f0 con 0x7fcd4415c820 2024-04-24T08:12:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.413+0000 7fcd49600640 1 --2- 172.21.15.69:0/3684034345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4415c820 0x7fcd440b3e90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fcd4406e200 tx=0x7fcd3802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72bfd7e0d6ee47fd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.413+0000 7fcd43e00640 1 -- 172.21.15.69:0/3684034345 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38037070 con 0x7fcd4415c820 2024-04-24T08:12:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.413+0000 7fcd43e00640 1 -- 172.21.15.69:0/3684034345 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd38032440 con 0x7fcd4415c820 2024-04-24T08:12:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.413+0000 7fcd43e00640 1 -- 172.21.15.69:0/3684034345 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38032810 con 0x7fcd4415c820 2024-04-24T08:12:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.414+0000 7fcd4c000640 1 -- 172.21.15.69:0/3684034345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4415c820 msgr2=0x7fcd440b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.414+0000 7fcd4c000640 1 --2- 172.21.15.69:0/3684034345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4415c820 0x7fcd440b3e90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fcd4406e200 tx=0x7fcd3802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.414+0000 7fcd4c000640 1 -- 172.21.15.69:0/3684034345 shutdown_connections 2024-04-24T08:12:21.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.414+0000 7fcd4c000640 1 --2- 172.21.15.69:0/3684034345 >> v2:172.21.15.121:3300/0 conn(0x7fcd440b46a0 0x7fcd4406f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.414+0000 7fcd4c000640 1 --2- 172.21.15.69:0/3684034345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4415c820 0x7fcd440b3e90 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.414+0000 7fcd4c000640 1 -- 172.21.15.69:0/3684034345 >> 172.21.15.69:0/3684034345 conn(0x7fcd440d0350 msgr2=0x7fcd440d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:21.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.414+0000 7fcd4c000640 1 -- 172.21.15.69:0/3684034345 shutdown_connections 2024-04-24T08:12:21.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.414+0000 7fcd4c000640 1 -- 172.21.15.69:0/3684034345 wait complete. 2024-04-24T08:12:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.416+0000 7fcd4c000640 1 Processor -- start 2024-04-24T08:12:21.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.416+0000 7fcd4c000640 1 -- start start 2024-04-24T08:12:21.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.417+0000 7fcd4c000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406fd10 0x7fcd440c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.417+0000 7fcd4c000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd440b46a0 0x7fcd4415d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.417+0000 7fcd4c000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4415c820 0x7fcd441f4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.417+0000 7fcd4c000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd440b4b30 con 0x7fcd4406fd10 2024-04-24T08:12:21.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.417+0000 7fcd4c000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4406f2b0 con 0x7fcd4415c820 2024-04-24T08:12:21.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.417+0000 7fcd4c000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd440ce960 con 0x7fcd440b46a0 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.417+0000 7fcd49600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406fd10 0x7fcd440c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.417+0000 7fcd49600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406fd10 0x7fcd440c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44184/0 (socket says 172.21.15.69:44184) 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.417+0000 7fcd49600640 1 -- 172.21.15.69:0/3549401109 learned_addr learned my addr 172.21.15.69:0/3549401109 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd49600640 1 -- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd440b46a0 msgr2=0x7fcd4415d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd49600640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd440b46a0 0x7fcd4415d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd49600640 1 -- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4415c820 msgr2=0x7fcd441f4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd4a000640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4415c820 0x7fcd441f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd49600640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4415c820 0x7fcd441f4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd49600640 1 -- 172.21.15.69:0/3549401109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4415e920 con 0x7fcd4406fd10 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd4a000640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4415c820 0x7fcd441f4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:21.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd49600640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406fd10 0x7fcd440c96c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcd38000c00 tx=0x7fcd380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:21.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd41600640 1 -- 172.21.15.69:0/3549401109 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38037040 con 0x7fcd4406fd10 2024-04-24T08:12:21.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd41600640 1 -- 172.21.15.69:0/3549401109 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd38042070 con 0x7fcd4406fd10 2024-04-24T08:12:21.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd4c000640 1 -- 172.21.15.69:0/3549401109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd441579d0 con 0x7fcd4406fd10 2024-04-24T08:12:21.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.418+0000 7fcd41600640 1 -- 172.21.15.69:0/3549401109 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38030960 con 0x7fcd4406fd10 2024-04-24T08:12:21.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.419+0000 7fcd4c000640 1 -- 172.21.15.69:0/3549401109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd4415e360 con 0x7fcd4406fd10 2024-04-24T08:12:21.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.421+0000 7fcd41600640 1 -- 172.21.15.69:0/3549401109 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd38030b00 con 0x7fcd4406fd10 2024-04-24T08:12:21.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.422+0000 7fcd41600640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd2406c8f0 0x7fcd2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.422+0000 7fcd41600640 1 -- 172.21.15.69:0/3549401109 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fcd380ac4c0 con 0x7fcd4406fd10 2024-04-24T08:12:21.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.422+0000 7fcd48c00640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd2406c8f0 0x7fcd2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.423+0000 7fcd4ee34740 1 Processor -- start 2024-04-24T08:12:21.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.423+0000 7fcd4ee34740 1 -- start start 2024-04-24T08:12:21.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd48c00640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd2406c8f0 0x7fcd2406edb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcd300098f0 tx=0x7fcd30008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:21.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29daf45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db240d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24610 0x55a29daf9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a29da5e1f0 con 0x55a29db24610 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a29da04350 con 0x55a29db41c30 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a29da5a1b0 con 0x55a29db24d50 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd4a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24610 0x55a29daf9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd4a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24610 0x55a29daf9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44194/0 (socket says 172.21.15.69:44194) 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd4a000640 1 -- 172.21.15.69:0/3053080281 learned_addr learned my addr 172.21.15.69:0/3053080281 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd49600640 1 --2- 172.21.15.69:0/3053080281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29daf45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.424+0000 7fcd48c00640 1 --2- 172.21.15.69:0/3053080281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db240d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.425+0000 7fcd4a000640 1 -- 172.21.15.69:0/3053080281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 msgr2=0x55a29db240d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.425+0000 7fcd4a000640 1 --2- 172.21.15.69:0/3053080281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db240d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.425+0000 7fcd4a000640 1 -- 172.21.15.69:0/3053080281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 msgr2=0x55a29daf45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.425+0000 7fcd4a000640 1 --2- 172.21.15.69:0/3053080281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29daf45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.425+0000 7fcd4a000640 1 -- 172.21.15.69:0/3053080281 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a29da58f50 con 0x55a29db24610 2024-04-24T08:12:21.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.425+0000 7fcd4a000640 1 --2- 172.21.15.69:0/3053080281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24610 0x55a29daf9420 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c0021d0 tx=0x7fcd3c00d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164b0d61d98b5cd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:21.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.425+0000 7fcd35c00640 1 -- 172.21.15.69:0/3053080281 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c018070 con 0x55a29db24610 2024-04-24T08:12:21.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.425+0000 7fcd35c00640 1 -- 172.21.15.69:0/3053080281 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd3c013670 con 0x55a29db24610 2024-04-24T08:12:21.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.425+0000 7fcd35c00640 1 -- 172.21.15.69:0/3053080281 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c013a20 con 0x55a29db24610 2024-04-24T08:12:21.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.426+0000 7fcd4ee34740 1 -- 172.21.15.69:0/3053080281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24610 msgr2=0x55a29daf9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.426+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/3053080281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24610 0x55a29daf9420 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c0021d0 tx=0x7fcd3c00d5e0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.426+0000 7fcd4ee34740 1 -- 172.21.15.69:0/3053080281 shutdown_connections 2024-04-24T08:12:21.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.426+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/3053080281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24610 0x55a29daf9420 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.426+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/3053080281 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db240d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.426+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/3053080281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29daf45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.426+0000 7fcd4ee34740 1 -- 172.21.15.69:0/3053080281 >> 172.21.15.69:0/3053080281 conn(0x55a29daf6830 msgr2=0x55a29db3f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:21.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.426+0000 7fcd4ee34740 1 -- 172.21.15.69:0/3053080281 shutdown_connections 2024-04-24T08:12:21.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.426+0000 7fcd4ee34740 1 -- 172.21.15.69:0/3053080281 wait complete. 2024-04-24T08:12:21.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.427+0000 7fcd4ee34740 1 -- start start 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.427+0000 7fcd4ee34740 1 Processor -- start 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.428+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db0ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.428+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db41c30 0x55a29db40ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.428+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db0f3d0 0x55a29db41000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.428+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a29daf89e0 con 0x55a29db41c30 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.428+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a29da17690 con 0x55a29db0f3d0 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.428+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a29daf7dd0 con 0x55a29db24d50 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.428+0000 7fcd4ee34740 10 client.0 _ll_register_callbacks cb 0x55a29da53650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.428+0000 7fcd48c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db41c30 0x55a29db40ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.428+0000 7fcd48c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db41c30 0x55a29db40ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44196/0 (socket says 172.21.15.69:44196) 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.428+0000 7fcd48c00640 1 -- 172.21.15.69:0/1664021444 learned_addr learned my addr 172.21.15.69:0/1664021444 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd4a000640 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db0f3d0 0x55a29db41000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd49600640 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db0ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd48c00640 1 -- 172.21.15.69:0/1664021444 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 msgr2=0x55a29db0ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd48c00640 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db0ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd48c00640 1 -- 172.21.15.69:0/1664021444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db0f3d0 msgr2=0x55a29db41000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd48c00640 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db0f3d0 0x55a29db41000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd48c00640 1 -- 172.21.15.69:0/1664021444 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a29da16920 con 0x55a29db41c30 2024-04-24T08:12:21.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd48c00640 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db41c30 0x55a29db40ac0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000dea0 tx=0x7fcd30006e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4c2d47e735dde server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:21.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd2a000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:21.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd2a000640 1 -- 172.21.15.69:0/1664021444 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd30014e40 con 0x55a29db41c30 2024-04-24T08:12:21.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd2a000640 1 -- 172.21.15.69:0/1664021444 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fcd3000f040 con 0x55a29db41c30 2024-04-24T08:12:21.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.429+0000 7fcd4ee34740 1 -- 172.21.15.69:0/1664021444 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd3c013670 con 0x55a29db41c30 2024-04-24T08:12:21.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd2a000640 1 -- 172.21.15.69:0/1664021444 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fcd3002b050 con 0x55a29db41c30 2024-04-24T08:12:21.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd4ee34740 10 client.6456 fetch_fsmap learned FSMap version 34 2024-04-24T08:12:21.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd4ee34740 1 -- 172.21.15.69:0/1664021444 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x55a29d83a7d0 con 0x55a29db41c30 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd2a000640 1 -- 172.21.15.69:0/1664021444 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd3c013670 con 0x55a29db41c30 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd2a000640 1 -- 172.21.15.69:0/1664021444 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7fcd3002e020 con 0x55a29db41c30 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd4ee34740 10 client.6456 fetch_fsmap finished waiting for FSMap version 34 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd4ee34740 10 client.6456 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd4ee34740 20 client.6456 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55a29db22560 0x55a29db22a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd4ee34740 4 client.6456 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd4ee34740 1 -- 172.21.15.69:0/1664021444 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a29da6bc10 con 0x55a29db22560 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.430+0000 7fcd49600640 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55a29db22560 0x55a29db22a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.431+0000 7fcd49600640 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55a29db22560 0x55a29db22a10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc92e1c3ac0063df server_cookie=1700a28379589dc2 in_seq=0 out_seq=0 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.431+0000 7fcd2a000640 10 client.6456 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.431+0000 7fcd2a000640 1 -- 172.21.15.69:0/1664021444 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a29da6bc10 con 0x55a29db22560 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.431+0000 7fcd2a000640 10 client.6456 handle_command_reply: tid=0 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 client.6456 shutdown 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 -- 172.21.15.69:0/1664021444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db41c30 msgr2=0x55a29db40ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db41c30 0x55a29db40ac0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000dea0 tx=0x7fcd30006e10 comp rx=0 tx=0).stop 2024-04-24T08:12:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 -- 172.21.15.69:0/1664021444 shutdown_connections 2024-04-24T08:12:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55a29db22560 0x55a29db22a10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db41c30 0x55a29db40ac0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db0f3d0 0x55a29db41000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd2a000640 0 client.6456 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/1664021444 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db0ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 -- 172.21.15.69:0/1664021444 >> 172.21.15.69:0/1664021444 conn(0x55a29db24610 msgr2=0x55a29daf5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 -- 172.21.15.69:0/1664021444 shutdown_connections 2024-04-24T08:12:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 1 -- 172.21.15.69:0/1664021444 wait complete. 2024-04-24T08:12:21.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.432+0000 7fcd4ee34740 20 client.6456 trim_cache size 0 max 16384 2024-04-24T08:12:21.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.451+0000 7fcd4ee34740 1 Processor -- start 2024-04-24T08:12:21.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.451+0000 7fcd4ee34740 1 -- start start 2024-04-24T08:12:21.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.452+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db0f3d0 0x55a29db3f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.452+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24d50 0x55a29daf4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.452+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29db23320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.452+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd44200e80 con 0x55a29db24d50 2024-04-24T08:12:21.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.452+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcd441ffe80 con 0x55a29db41c30 2024-04-24T08:12:21.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.452+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd441ff1e0 con 0x55a29db0f3d0 2024-04-24T08:12:21.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.452+0000 7fcd48c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24d50 0x55a29daf4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.452+0000 7fcd48c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24d50 0x55a29daf4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44210/0 (socket says 172.21.15.69:44210) 2024-04-24T08:12:21.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.452+0000 7fcd48c00640 1 -- 172.21.15.69:0/3912401249 learned_addr learned my addr 172.21.15.69:0/3912401249 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:21.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.453+0000 7fcd4a000640 1 --2- 172.21.15.69:0/3912401249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29db23320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.453+0000 7fcd48c00640 1 -- 172.21.15.69:0/3912401249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db0f3d0 msgr2=0x55a29db3f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.453+0000 7fcd48c00640 1 --2- 172.21.15.69:0/3912401249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db0f3d0 0x55a29db3f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.453+0000 7fcd48c00640 1 -- 172.21.15.69:0/3912401249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 msgr2=0x55a29db23320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.453+0000 7fcd48c00640 1 --2- 172.21.15.69:0/3912401249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29db23320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.453+0000 7fcd48c00640 1 -- 172.21.15.69:0/3912401249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a29d7703f0 con 0x55a29db24d50 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.453+0000 7fcd48c00640 1 --2- 172.21.15.69:0/3912401249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24d50 0x55a29daf4630 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000d530 tx=0x7fcd3001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf5d49dcadfd339 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.453+0000 7fcd35200640 1 -- 172.21.15.69:0/3912401249 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd30034070 con 0x55a29db24d50 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.453+0000 7fcd35200640 1 -- 172.21.15.69:0/3912401249 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd300125f0 con 0x55a29db24d50 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.453+0000 7fcd35200640 1 -- 172.21.15.69:0/3912401249 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd300127c0 con 0x55a29db24d50 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.454+0000 7fcd4ee34740 1 -- 172.21.15.69:0/3912401249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24d50 msgr2=0x55a29daf4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.454+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/3912401249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24d50 0x55a29daf4630 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fcd3000d530 tx=0x7fcd3001d000 comp rx=0 tx=0).stop 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.454+0000 7fcd4ee34740 1 -- 172.21.15.69:0/3912401249 shutdown_connections 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.454+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/3912401249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db24d50 0x55a29daf4630 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.454+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/3912401249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29db23320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.454+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/3912401249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db0f3d0 0x55a29db3f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.454+0000 7fcd4ee34740 1 -- 172.21.15.69:0/3912401249 >> 172.21.15.69:0/3912401249 conn(0x55a29db24610 msgr2=0x55a29daf3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.454+0000 7fcd4ee34740 1 -- 172.21.15.69:0/3912401249 shutdown_connections 2024-04-24T08:12:21.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.454+0000 7fcd4ee34740 1 -- 172.21.15.69:0/3912401249 wait complete. 2024-04-24T08:12:21.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.455+0000 7fcd4ee34740 1 -- start start 2024-04-24T08:12:21.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.455+0000 7fcd4ee34740 1 Processor -- start 2024-04-24T08:12:21.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.456+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db0f3d0 0x55a29dbb8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.456+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db20670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.456+0000 7fcd4ee34740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29db276e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.456+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a29db62300 con 0x55a29db0f3d0 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.456+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcd441ff830 con 0x55a29db41c30 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.456+0000 7fcd4ee34740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a29da34840 con 0x55a29db24d50 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.456+0000 7fcd4ee34740 10 client.0 _ll_register_callbacks cb 0x7fcd30009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.456+0000 7fcd49600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db0f3d0 0x55a29dbb8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.456+0000 7fcd49600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db0f3d0 0x55a29dbb8b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44216/0 (socket says 172.21.15.69:44216) 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.456+0000 7fcd49600640 1 -- 172.21.15.69:0/4236589218 learned_addr learned my addr 172.21.15.69:0/4236589218 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd4a000640 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29db276e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd48c00640 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db20670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd49600640 1 -- 172.21.15.69:0/4236589218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 msgr2=0x55a29db20670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd49600640 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db20670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd49600640 1 -- 172.21.15.69:0/4236589218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 msgr2=0x55a29db276e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd49600640 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29db276e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd49600640 1 -- 172.21.15.69:0/4236589218 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a29d7300f0 con 0x55a29db0f3d0 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd49600640 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db0f3d0 0x55a29dbb8b10 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fcd380091d0 tx=0x7fcd38009160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330ca46639c8b0cb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd2a000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd2a000640 1 -- 172.21.15.69:0/4236589218 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38097060 con 0x55a29db0f3d0 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd2a000640 1 -- 172.21.15.69:0/4236589218 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fcd380784e0 con 0x55a29db0f3d0 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.457+0000 7fcd4ee34740 1 -- 172.21.15.69:0/4236589218 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd300125f0 con 0x55a29db0f3d0 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd2a000640 1 -- 172.21.15.69:0/4236589218 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fcd3807a1e0 con 0x55a29db0f3d0 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd4ee34740 10 client.6462 fetch_fsmap learned FSMap version 34 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd4ee34740 1 -- 172.21.15.69:0/4236589218 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x55a29da23810 con 0x55a29db0f3d0 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd2a000640 1 -- 172.21.15.69:0/4236589218 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd300125f0 con 0x55a29db0f3d0 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd2a000640 1 -- 172.21.15.69:0/4236589218 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7fcd3803a070 con 0x55a29db0f3d0 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd4ee34740 10 client.6462 fetch_fsmap finished waiting for FSMap version 34 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd4ee34740 10 client.6462 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:12:21.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd4ee34740 20 client.6462 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:21.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55a29db4edc0 0x55a29db42350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd4ee34740 4 client.6462 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:12:21.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd4ee34740 1 -- 172.21.15.69:0/4236589218 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a29da65a80 con 0x55a29db4edc0 2024-04-24T08:12:21.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.458+0000 7fcd48c00640 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55a29db4edc0 0x55a29db42350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd48c00640 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55a29db4edc0 0x55a29db42350 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ad02915c533861c server_cookie=ea77edb82553a1d4 in_seq=0 out_seq=0 2024-04-24T08:12:21.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd2a000640 10 client.6462 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:21.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd2a000640 1 -- 172.21.15.69:0/4236589218 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a29da65a80 con 0x55a29db4edc0 2024-04-24T08:12:21.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd2a000640 10 client.6462 handle_command_reply: tid=0 2024-04-24T08:12:21.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd4ee34740 1 client.6462 shutdown 2024-04-24T08:12:21.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd4ee34740 1 -- 172.21.15.69:0/4236589218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db0f3d0 msgr2=0x55a29dbb8b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db0f3d0 0x55a29dbb8b10 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fcd380091d0 tx=0x7fcd38009160 comp rx=0 tx=0).stop 2024-04-24T08:12:21.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd4ee34740 1 -- 172.21.15.69:0/4236589218 shutdown_connections 2024-04-24T08:12:21.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55a29db4edc0 0x55a29db42350 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a29db41c30 0x55a29db276e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a29db24d50 0x55a29db20670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd4ee34740 1 --2- 172.21.15.69:0/4236589218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a29db0f3d0 0x55a29dbb8b10 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd2a000640 0 client.6462 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd4ee34740 1 -- 172.21.15.69:0/4236589218 >> 172.21.15.69:0/4236589218 conn(0x55a29db24610 msgr2=0x55a29db232d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.459+0000 7fcd4ee34740 1 -- 172.21.15.69:0/4236589218 shutdown_connections 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4ee34740 1 -- 172.21.15.69:0/4236589218 wait complete. 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4ee34740 20 client.6462 trim_cache size 0 max 16384 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-04-24T08:12:21.464 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:12:21.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4c000640 1 -- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd2406c8f0 msgr2=0x7fcd2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4c000640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd2406c8f0 0x7fcd2406edb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcd300098f0 tx=0x7fcd30008040 comp rx=0 tx=0).stop 2024-04-24T08:12:21.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4c000640 1 -- 172.21.15.69:0/3549401109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406fd10 msgr2=0x7fcd440c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4c000640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406fd10 0x7fcd440c96c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcd38000c00 tx=0x7fcd380046c0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4c000640 1 -- 172.21.15.69:0/3549401109 shutdown_connections 2024-04-24T08:12:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4c000640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd2406c8f0 0x7fcd2406edb0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4c000640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd4415c820 0x7fcd441f4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4c000640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd440b46a0 0x7fcd4415d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4c000640 1 --2- 172.21.15.69:0/3549401109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd4406fd10 0x7fcd440c96c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.460+0000 7fcd4c000640 1 -- 172.21.15.69:0/3549401109 >> 172.21.15.69:0/3549401109 conn(0x7fcd440d0350 msgr2=0x7fcd4406bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.461+0000 7fcd4c000640 1 -- 172.21.15.69:0/3549401109 shutdown_connections 2024-04-24T08:12:21.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.461+0000 7fcd4c000640 1 -- 172.21.15.69:0/3549401109 wait complete. 2024-04-24T08:12:21.479 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:12:21.479 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:12:21.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.675+0000 7fe5c6000640 1 Processor -- start 2024-04-24T08:12:21.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.675+0000 7fe5c6000640 1 -- start start 2024-04-24T08:12:21.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.676+0000 7fe5c6000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c00d0300 0x7fe5c006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.676+0000 7fe5c6000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe5c0145a90 0x7fe5c00d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.677+0000 7fe5c6000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c00d0c20 con 0x7fe5c0145fe0 2024-04-24T08:12:21.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.677+0000 7fe5c6000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe5c00cf3d0 con 0x7fe5c0145a90 2024-04-24T08:12:21.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.677+0000 7fe5c6000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c00cf6b0 con 0x7fe5c00d0300 2024-04-24T08:12:21.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.677+0000 7fe5bfe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe5c0145fe0 0x7fe5c0160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:41890/0 (socket says 172.21.15.69:41890) 2024-04-24T08:12:21.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.677+0000 7fe5bfe00640 1 -- 172.21.15.69:0/618819177 learned_addr learned my addr 172.21.15.69:0/618819177 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:21.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.677+0000 7fe5b6a00640 1 --2- 172.21.15.69:0/618819177 >> v2:172.21.15.121:3300/0 conn(0x7fe5c0145a90 0x7fe5c00d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.677+0000 7fe5bf400640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c00d0300 0x7fe5c006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.677+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204446547 0 0) 0x7fe5c00d0c20 con 0x7fe5c0145fe0 2024-04-24T08:12:21.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.677+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5a4003520 con 0x7fe5c0145fe0 2024-04-24T08:12:21.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1783518211 0 0) 0x7fe5a4003520 con 0x7fe5c0145fe0 2024-04-24T08:12:21.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c00d0300 msgr2=0x7fe5c006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c00d0300 0x7fe5c006e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 >> v2:172.21.15.121:3300/0 conn(0x7fe5c0145a90 msgr2=0x7fe5c00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 --2- 172.21.15.69:0/618819177 >> v2:172.21.15.121:3300/0 conn(0x7fe5c0145a90 0x7fe5c00d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c00cf990 con 0x7fe5c0145fe0 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe5b8001670 con 0x7fe5c0145fe0 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 >> v1:172.21.15.69:6789/0 conn(0x7fe5c0145fe0 legacy=0x7fe5c0160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5a40039b0 0x7fe5a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a4004800 0x7fe5a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c00d0300 0x7fe5a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c00cf3d0 con 0x7fe5a4004800 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c00cf6b0 con 0x7fe5a40039b0 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe5a400b9c0 con 0x7fe5c00d0300 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.678+0000 7fe5b6a00640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a4004800 0x7fe5a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.679+0000 7fe5b6a00640 1 -- 172.21.15.69:0/618819177 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c00d0300 msgr2=0x7fe5a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.679+0000 7fe5b6a00640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c00d0300 0x7fe5a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.679+0000 7fe5bf400640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5a40039b0 0x7fe5a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.679+0000 7fe5b6a00640 1 -- 172.21.15.69:0/618819177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5a40039b0 msgr2=0x7fe5a4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.679+0000 7fe5b6a00640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5a40039b0 0x7fe5a4003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.679+0000 7fe5b6a00640 1 -- 172.21.15.69:0/618819177 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a4003520 con 0x7fe5a4004800 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.680+0000 7fe5b6a00640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a4004800 0x7fe5a4004c80 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8009fc0 tx=0x7fe5a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a225888e17831a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.680+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8009c10 con 0x7fe5a4004800 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.680+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5a8036070 con 0x7fe5a4004800 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.680+0000 7fe5bea00640 1 -- 172.21.15.69:0/618819177 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a802d070 con 0x7fe5a4004800 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.681+0000 7fe5c6000640 1 -- 172.21.15.69:0/618819177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a4004800 msgr2=0x7fe5a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.681+0000 7fe5c6000640 1 --2- 172.21.15.69:0/618819177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5a4004800 0x7fe5a4004c80 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8009fc0 tx=0x7fe5a4005830 comp rx=0 tx=0).stop 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.681+0000 7fe5bfe00640 1 -- 172.21.15.69:0/618819177 reap_dead start 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.681+0000 7fe5c6000640 1 -- 172.21.15.69:0/618819177 shutdown_connections 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.681+0000 7fe5c6000640 1 -- 172.21.15.69:0/618819177 >> 172.21.15.69:0/618819177 conn(0x7fe5c0067d90 msgr2=0x7fe5c006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.681+0000 7fe5c6000640 1 -- 172.21.15.69:0/618819177 shutdown_connections 2024-04-24T08:12:21.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.681+0000 7fe5c6000640 1 -- 172.21.15.69:0/618819177 wait complete. 2024-04-24T08:12:21.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.682+0000 7fe5c6000640 1 Processor -- start 2024-04-24T08:12:21.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.683+0000 7fe5c6000640 1 -- start start 2024-04-24T08:12:21.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5c6000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5a4004800 0x7fe5c015f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5c6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015b550 0x7fe5c00ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5c6000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01437d0 0x7fe5c0143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5c6000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c0145770 con 0x7fe5c015b550 2024-04-24T08:12:21.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5c6000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c00cff50 con 0x7fe5c01437d0 2024-04-24T08:12:21.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5c6000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c01458f0 con 0x7fe5a4004800 2024-04-24T08:12:21.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5b6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015b550 0x7fe5c00ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5b6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015b550 0x7fe5c00ccd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44236/0 (socket says 172.21.15.69:44236) 2024-04-24T08:12:21.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5b6a00640 1 -- 172.21.15.69:0/1754714046 learned_addr learned my addr 172.21.15.69:0/1754714046 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5bf400640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5a4004800 0x7fe5c015f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5bfe00640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01437d0 0x7fe5c0143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5b6a00640 1 -- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5a4004800 msgr2=0x7fe5c015f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5b6a00640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5a4004800 0x7fe5c015f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5b6a00640 1 -- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01437d0 msgr2=0x7fe5c0143c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5b6a00640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01437d0 0x7fe5c0143c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.684+0000 7fe5b6a00640 1 -- 172.21.15.69:0/1754714046 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c015e1c0 con 0x7fe5c015b550 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.685+0000 7fe5b6a00640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015b550 0x7fe5c00ccd70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800e230 tx=0x7fe5a8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.685+0000 7fe5b7e00640 1 -- 172.21.15.69:0/1754714046 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8009c10 con 0x7fe5c015b550 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.685+0000 7fe5b7e00640 1 -- 172.21.15.69:0/1754714046 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5a8003bb0 con 0x7fe5c015b550 2024-04-24T08:12:21.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.685+0000 7fe5b7e00640 1 -- 172.21.15.69:0/1754714046 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a8004020 con 0x7fe5c015b550 2024-04-24T08:12:21.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.685+0000 7fe5c6000640 1 -- 172.21.15.69:0/1754714046 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c015e4a0 con 0x7fe5c015b550 2024-04-24T08:12:21.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.685+0000 7fe5c6000640 1 -- 172.21.15.69:0/1754714046 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c015ea60 con 0x7fe5c015b550 2024-04-24T08:12:21.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.686+0000 7fe5c6000640 1 -- 172.21.15.69:0/1754714046 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5c0068a80 con 0x7fe5c015b550 2024-04-24T08:12:21.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.687+0000 7fe5b7e00640 1 -- 172.21.15.69:0/1754714046 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe5a8005d10 con 0x7fe5c015b550 2024-04-24T08:12:21.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.688+0000 7fe5b7e00640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe59c06c8f0 0x7fe59c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:21.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.688+0000 7fe5b7e00640 1 -- 172.21.15.69:0/1754714046 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fe5a80ad160 con 0x7fe5c015b550 2024-04-24T08:12:21.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.689+0000 7fe5bf400640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe59c06c8f0 0x7fe59c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:21.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.690+0000 7fe5bf400640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe59c06c8f0 0x7fe59c06edb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0009930 tx=0x7fe5b0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:21.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.693+0000 7fe5b7e00640 1 -- 172.21.15.69:0/1754714046 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe5a807b4f0 con 0x7fe5c015b550 2024-04-24T08:12:21.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.931+0000 7fe5c6000640 1 -- 172.21.15.69:0/1754714046 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5c00d1120 con 0x7fe5c015b550 2024-04-24T08:12:21.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.933+0000 7fe5b7e00640 1 -- 172.21.15.69:0/1754714046 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+5187 (secure 0 0 0) 0x7fe5a807adf0 con 0x7fe5c015b550 2024-04-24T08:12:21.934 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:12:21.934 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4034001365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4034001365},{"type":"v1","addr":"172.21.15.121:6835","nonce":4034001365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3907163219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3907163219},{"type":"v1","addr":"172.21.15.121:6837","nonce":3907163219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2756395689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2756395689},{"type":"v1","addr":"172.21.15.69:6834","nonce":2756395689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:11:39.824842+0000","modified":"2024-04-24T08:11:56.406430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":7,"addr":"172.21.15.69:6835/2288057943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":2288057943},{"type":"v1","addr":"172.21.15.69:6835","nonce":2288057943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6231,"qdb_cluster":[6231]},"id":4}]} 2024-04-24T08:12:21.935 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 34 2024-04-24T08:12:21.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.935+0000 7fe5c6000640 1 -- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe59c06c8f0 msgr2=0x7fe59c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.935+0000 7fe5c6000640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe59c06c8f0 0x7fe59c06edb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0009930 tx=0x7fe5b0008040 comp rx=0 tx=0).stop 2024-04-24T08:12:21.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.935+0000 7fe5c6000640 1 -- 172.21.15.69:0/1754714046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015b550 msgr2=0x7fe5c00ccd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:21.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.935+0000 7fe5c6000640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015b550 0x7fe5c00ccd70 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fe5a800e230 tx=0x7fe5a8009750 comp rx=0 tx=0).stop 2024-04-24T08:12:21.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.935+0000 7fe5c6000640 1 -- 172.21.15.69:0/1754714046 shutdown_connections 2024-04-24T08:12:21.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.935+0000 7fe5c6000640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe59c06c8f0 0x7fe59c06edb0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.935+0000 7fe5c6000640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c015b550 0x7fe5c00ccd70 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.935+0000 7fe5c6000640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c01437d0 0x7fe5c0143c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.935+0000 7fe5c6000640 1 --2- 172.21.15.69:0/1754714046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5a4004800 0x7fe5c015f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:21.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.935+0000 7fe5c6000640 1 -- 172.21.15.69:0/1754714046 >> 172.21.15.69:0/1754714046 conn(0x7fe5c0067d90 msgr2=0x7fe5c00aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:21.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.936+0000 7fe5c6000640 1 -- 172.21.15.69:0/1754714046 shutdown_connections 2024-04-24T08:12:21.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:21.936+0000 7fe5c6000640 1 -- 172.21.15.69:0/1754714046 wait complete. 2024-04-24T08:12:21.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-04-24T08:12:22.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.157+0000 7f0af2600640 1 Processor -- start 2024-04-24T08:12:22.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.157+0000 7f0af2600640 1 -- start start 2024-04-24T08:12:22.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.159+0000 7f0af2600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0aec15c820 0x7f0aec0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.159+0000 7f0af2600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0aec0b46a0 0x7f0aec06f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.159+0000 7f0af2600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0aec15cc00 con 0x7f0aec06fd10 2024-04-24T08:12:22.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.159+0000 7f0af2600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0aec06ed30 con 0x7f0aec0b46a0 2024-04-24T08:12:22.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.159+0000 7f0af2600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0aec0b4e10 con 0x7f0aec15c820 2024-04-24T08:12:22.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.159+0000 7f0ae2a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0aec0b46a0 0x7f0aec06f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.159+0000 7f0aebe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0aec06fd10 0x7f0aec0ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:41896/0 (socket says 172.21.15.69:41896) 2024-04-24T08:12:22.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.159+0000 7f0ae2a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0aec0b46a0 0x7f0aec06f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44252/0 (socket says 172.21.15.69:44252) 2024-04-24T08:12:22.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.159+0000 7f0aebe00640 1 -- 172.21.15.69:0/2912935616 learned_addr learned my addr 172.21.15.69:0/2912935616 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:22.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.160+0000 7f0ae2a00640 1 -- 172.21.15.69:0/2912935616 >> v2:172.21.15.121:3300/0 conn(0x7f0aec15c820 msgr2=0x7f0aec0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.160+0000 7f0aeb400640 1 --2- 172.21.15.69:0/2912935616 >> v2:172.21.15.121:3300/0 conn(0x7f0aec15c820 0x7f0aec0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.160+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/2912935616 >> v2:172.21.15.121:3300/0 conn(0x7f0aec15c820 0x7f0aec0b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.160+0000 7f0ae2a00640 1 -- 172.21.15.69:0/2912935616 >> v1:172.21.15.69:6789/0 conn(0x7f0aec06fd10 legacy=0x7f0aec0ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.160+0000 7f0ae2a00640 1 -- 172.21.15.69:0/2912935616 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aec0b50f0 con 0x7f0aec0b46a0 2024-04-24T08:12:22.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.160+0000 7f0aeb400640 1 --2- 172.21.15.69:0/2912935616 >> v2:172.21.15.121:3300/0 conn(0x7f0aec15c820 0x7f0aec0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:22.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.161+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/2912935616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec0b46a0 0x7f0aec06f7d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0b2f60 tx=0x7f0ad402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a3c3df6e920b18e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.161+0000 7f0aeaa00640 1 -- 172.21.15.69:0/2912935616 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4037070 con 0x7f0aec0b46a0 2024-04-24T08:12:22.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.161+0000 7f0aeaa00640 1 -- 172.21.15.69:0/2912935616 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad4032440 con 0x7f0aec0b46a0 2024-04-24T08:12:22.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.161+0000 7f0aeaa00640 1 -- 172.21.15.69:0/2912935616 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4032810 con 0x7f0aec0b46a0 2024-04-24T08:12:22.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.162+0000 7f0af2600640 1 -- 172.21.15.69:0/2912935616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec0b46a0 msgr2=0x7f0aec06f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.162+0000 7f0af2600640 1 --2- 172.21.15.69:0/2912935616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec0b46a0 0x7f0aec06f7d0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0b2f60 tx=0x7f0ad402a1a0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.162+0000 7f0af2600640 1 -- 172.21.15.69:0/2912935616 shutdown_connections 2024-04-24T08:12:22.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.162+0000 7f0af2600640 1 --2- 172.21.15.69:0/2912935616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec0b46a0 0x7f0aec06f7d0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.162+0000 7f0af2600640 1 --2- 172.21.15.69:0/2912935616 >> v2:172.21.15.121:3300/0 conn(0x7f0aec15c820 0x7f0aec0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.162+0000 7f0af2600640 1 -- 172.21.15.69:0/2912935616 >> 172.21.15.69:0/2912935616 conn(0x7f0aec0d0350 msgr2=0x7f0aec0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.162+0000 7f0af2600640 1 -- 172.21.15.69:0/2912935616 shutdown_connections 2024-04-24T08:12:22.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.162+0000 7f0af2600640 1 -- 172.21.15.69:0/2912935616 wait complete. 2024-04-24T08:12:22.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.164+0000 7f0af2600640 1 Processor -- start 2024-04-24T08:12:22.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.164+0000 7f0af2600640 1 -- start start 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.165+0000 7f0af2600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec06fd10 0x7f0aec159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.165+0000 7f0af2600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aec0b46a0 0x7f0aec1e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.165+0000 7f0af2600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aec15c820 0x7f0aec150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.165+0000 7f0af2600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec0b4b30 con 0x7f0aec06fd10 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.165+0000 7f0af2600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec06f2b0 con 0x7f0aec15c820 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.165+0000 7f0af2600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0aec0ce960 con 0x7f0aec0b46a0 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.165+0000 7f0aeb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec06fd10 0x7f0aec159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.165+0000 7f0aeb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec06fd10 0x7f0aec159810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44258/0 (socket says 172.21.15.69:44258) 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.165+0000 7f0aeb400640 1 -- 172.21.15.69:0/2469298994 learned_addr learned my addr 172.21.15.69:0/2469298994 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aec0b46a0 0x7f0aec1e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0aeb400640 1 -- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aec0b46a0 msgr2=0x7f0aec1e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0aeb400640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aec0b46a0 0x7f0aec1e6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0aeb400640 1 -- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aec15c820 msgr2=0x7f0aec150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0aeb400640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aec15c820 0x7f0aec150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0aeb400640 1 -- 172.21.15.69:0/2469298994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aec1e75b0 con 0x7f0aec06fd10 2024-04-24T08:12:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aec0b46a0 0x7f0aec1e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0aeb400640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec06fd10 0x7f0aec159810 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00cf60 tx=0x7f0adc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0ae3e00640 1 -- 172.21.15.69:0/2469298994 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc01a070 con 0x7f0aec06fd10 2024-04-24T08:12:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0ae3e00640 1 -- 172.21.15.69:0/2469298994 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0adc00d840 con 0x7f0aec06fd10 2024-04-24T08:12:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0ae3e00640 1 -- 172.21.15.69:0/2469298994 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc00dc10 con 0x7f0aec06fd10 2024-04-24T08:12:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0af2600640 1 -- 172.21.15.69:0/2469298994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aec156b50 con 0x7f0aec06fd10 2024-04-24T08:12:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.166+0000 7f0af2600640 1 -- 172.21.15.69:0/2469298994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aec156520 con 0x7f0aec06fd10 2024-04-24T08:12:22.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.169+0000 7f0ae3e00640 1 -- 172.21.15.69:0/2469298994 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0adc01d070 con 0x7f0aec06fd10 2024-04-24T08:12:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.170+0000 7f0ae3e00640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ac806c8f0 0x7f0ac806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.170+0000 7f0ae3e00640 1 -- 172.21.15.69:0/2469298994 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f0adc08c360 con 0x7f0aec06fd10 2024-04-24T08:12:22.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.170+0000 7f0af5433740 1 Processor -- start 2024-04-24T08:12:22.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.170+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ac806c8f0 0x7f0ac806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.171+0000 7f0af5433740 1 -- start start 2024-04-24T08:12:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.171+0000 7f0af5433740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2c6ff0 0x559aaf2c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.171+0000 7f0af5433740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2e4890 0x559aaf2e4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.171+0000 7f0af5433740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf29ccd0 0x559aaf2e2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0af5433740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559aaf201190 con 0x559aaf29ccd0 2024-04-24T08:12:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0af5433740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x559aaf1a7030 con 0x559aaf2e4890 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0af5433740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x559aaf1fd150 con 0x559aaf2c6ff0 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ac806c8f0 0x7f0ac806edb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4034000 tx=0x7f0ad402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0aebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf29ccd0 0x559aaf2e2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0aebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf29ccd0 0x559aaf2e2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44274/0 (socket says 172.21.15.69:44274) 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0aebe00640 1 -- 172.21.15.69:0/1240270944 learned_addr learned my addr 172.21.15.69:0/1240270944 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/1240270944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2e4890 0x559aaf2e4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0aeb400640 1 --2- 172.21.15.69:0/1240270944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2c6ff0 0x559aaf2c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0aebe00640 1 -- 172.21.15.69:0/1240270944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2c6ff0 msgr2=0x559aaf2c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0aebe00640 1 --2- 172.21.15.69:0/1240270944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2c6ff0 0x559aaf2c73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0aebe00640 1 -- 172.21.15.69:0/1240270944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2e4890 msgr2=0x559aaf2e4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0aebe00640 1 --2- 172.21.15.69:0/1240270944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2e4890 0x559aaf2e4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0aebe00640 1 -- 172.21.15.69:0/1240270944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559aaf18fb60 con 0x559aaf29ccd0 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.172+0000 7f0aeb400640 1 --2- 172.21.15.69:0/1240270944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2c6ff0 0x559aaf2c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:22.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.173+0000 7f0aebe00640 1 --2- 172.21.15.69:0/1240270944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf29ccd0 0x559aaf2e2f50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4001590 tx=0x7f0ae4001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873703c1bcd570ac server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.173+0000 7f0adbe00640 1 -- 172.21.15.69:0/1240270944 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4017070 con 0x559aaf29ccd0 2024-04-24T08:12:22.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.173+0000 7f0adbe00640 1 -- 172.21.15.69:0/1240270944 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae4005ab0 con 0x559aaf29ccd0 2024-04-24T08:12:22.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.173+0000 7f0adbe00640 1 -- 172.21.15.69:0/1240270944 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae40070d0 con 0x559aaf29ccd0 2024-04-24T08:12:22.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.173+0000 7f0af5433740 1 -- 172.21.15.69:0/1240270944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf29ccd0 msgr2=0x559aaf2e2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.173+0000 7f0af5433740 1 --2- 172.21.15.69:0/1240270944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf29ccd0 0x559aaf2e2f50 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4001590 tx=0x7f0ae4001a50 comp rx=0 tx=0).stop 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.174+0000 7f0af5433740 1 -- 172.21.15.69:0/1240270944 shutdown_connections 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.174+0000 7f0af5433740 1 --2- 172.21.15.69:0/1240270944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf29ccd0 0x559aaf2e2f50 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.174+0000 7f0af5433740 1 --2- 172.21.15.69:0/1240270944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2e4890 0x559aaf2e4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.174+0000 7f0af5433740 1 --2- 172.21.15.69:0/1240270944 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2c6ff0 0x559aaf2c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.174+0000 7f0af5433740 1 -- 172.21.15.69:0/1240270944 >> 172.21.15.69:0/1240270944 conn(0x559aaf299600 msgr2=0x559aaf2e1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.174+0000 7f0af5433740 1 -- 172.21.15.69:0/1240270944 shutdown_connections 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.174+0000 7f0af5433740 1 -- 172.21.15.69:0/1240270944 wait complete. 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.174+0000 7f0af5433740 1 -- start start 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.175+0000 7f0af5433740 1 Processor -- start 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0af5433740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2c6ff0 0x559aaf2c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0af5433740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2e4890 0x559aaf2c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0af5433740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2f7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0af5433740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559aaf29da70 con 0x559aaf2f9e30 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0af5433740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x559aaf1ba500 con 0x559aaf2c6ff0 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0af5433740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x559aaf297530 con 0x559aaf2e4890 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0af5433740 10 client.0 _ll_register_callbacks cb 0x559aaf1f7270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0aebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0aebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2f7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44280/0 (socket says 172.21.15.69:44280) 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0aebe00640 1 -- 172.21.15.69:0/4114961321 learned_addr learned my addr 172.21.15.69:0/4114961321 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0aeb400640 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2c6ff0 0x559aaf2c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.176+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2e4890 0x559aaf2c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0aebe00640 1 -- 172.21.15.69:0/4114961321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2e4890 msgr2=0x559aaf2c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0aebe00640 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2e4890 0x559aaf2c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0aebe00640 1 -- 172.21.15.69:0/4114961321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2c6ff0 msgr2=0x559aaf2c38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0aebe00640 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2c6ff0 0x559aaf2c38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0aebe00640 1 -- 172.21.15.69:0/4114961321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aaefdf7c0 con 0x559aaf2f9e30 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2e4890 0x559aaf2c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0aebe00640 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2f7020 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4000c00 tx=0x7f0ae4001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c69fa68b1c7e537 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0ad8c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0ad8c00640 1 -- 172.21.15.69:0/4114961321 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4021020 con 0x559aaf2f9e30 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0ad8c00640 1 -- 172.21.15.69:0/4114961321 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f0ae40074e0 con 0x559aaf2f9e30 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.177+0000 7f0af5433740 1 -- 172.21.15.69:0/4114961321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ae4005ab0 con 0x559aaf2f9e30 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0ad8c00640 1 -- 172.21.15.69:0/4114961321 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0ae402b050 con 0x559aaf2f9e30 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0af5433740 10 client.6483 fetch_fsmap learned FSMap version 34 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0af5433740 1 -- 172.21.15.69:0/4114961321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x559aaf03ff50 con 0x559aaf2f9e30 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0ad8c00640 1 -- 172.21.15.69:0/4114961321 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae4005ab0 con 0x559aaf2f9e30 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0ad8c00640 1 -- 172.21.15.69:0/4114961321 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7f0ae402e020 con 0x559aaf2f9e30 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0af5433740 10 client.6483 fetch_fsmap finished waiting for FSMap version 34 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0af5433740 10 client.6483 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0af5433740 20 client.6483 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0af5433740 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x559aaf2c9e60 0x559aaf2ca240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0af5433740 4 client.6483 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:22.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0af5433740 1 -- 172.21.15.69:0/4114961321 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559aaf20ebb0 con 0x559aaf2c9e60 2024-04-24T08:12:22.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.178+0000 7f0aeb400640 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x559aaf2c9e60 0x559aaf2ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.179+0000 7f0aeb400640 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x559aaf2c9e60 0x559aaf2ca240 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c986f3424fa1cca server_cookie=8704be00f49bc554 in_seq=0 out_seq=0 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.179+0000 7f0ad8c00640 10 client.6483 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0ad8c00640 1 -- 172.21.15.69:0/4114961321 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559aaf20ebb0 con 0x559aaf2c9e60 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0ad8c00640 10 client.6483 handle_command_reply: tid=0 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 client.6483 shutdown 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 -- 172.21.15.69:0/4114961321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 msgr2=0x559aaf2f7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2f7020 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4000c00 tx=0x7f0ae4001080 comp rx=0 tx=0).stop 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 -- 172.21.15.69:0/4114961321 shutdown_connections 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x559aaf2c9e60 0x559aaf2ca240 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2f7020 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2e4890 0x559aaf2c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 --2- 172.21.15.69:0/4114961321 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2c6ff0 0x559aaf2c38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0ad8c00640 0 client.6483 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:22.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 -- 172.21.15.69:0/4114961321 >> 172.21.15.69:0/4114961321 conn(0x559aaf29ccd0 msgr2=0x559aaf2e5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 -- 172.21.15.69:0/4114961321 shutdown_connections 2024-04-24T08:12:22.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 1 -- 172.21.15.69:0/4114961321 wait complete. 2024-04-24T08:12:22.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.180+0000 7f0af5433740 20 client.6483 trim_cache size 0 max 16384 2024-04-24T08:12:22.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.195+0000 7f0af5433740 1 Processor -- start 2024-04-24T08:12:22.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.195+0000 7f0af5433740 1 -- start start 2024-04-24T08:12:22.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.196+0000 7f0af5433740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2c6ff0 0x559aaf29d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.196+0000 7f0af5433740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2e4890 0x559aaf2dfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.196+0000 7f0af5433740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2e02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.196+0000 7f0af5433740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec157350 con 0x559aaf2f9e30 2024-04-24T08:12:22.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.196+0000 7f0af5433740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0aec0732e0 con 0x559aaf2e4890 2024-04-24T08:12:22.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.196+0000 7f0af5433740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0aec073930 con 0x559aaf2c6ff0 2024-04-24T08:12:22.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.196+0000 7f0aebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2e02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.196+0000 7f0aebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2e02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44296/0 (socket says 172.21.15.69:44296) 2024-04-24T08:12:22.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.196+0000 7f0aebe00640 1 -- 172.21.15.69:0/945375392 learned_addr learned my addr 172.21.15.69:0/945375392 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.196+0000 7f0aebe00640 1 -- 172.21.15.69:0/945375392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2c6ff0 msgr2=0x559aaf29d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.197+0000 7f0aebe00640 1 --2- 172.21.15.69:0/945375392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2c6ff0 0x559aaf29d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.197+0000 7f0aebe00640 1 -- 172.21.15.69:0/945375392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2e4890 msgr2=0x559aaf2dfd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.197+0000 7f0aebe00640 1 --2- 172.21.15.69:0/945375392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2e4890 0x559aaf2dfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.197+0000 7f0aebe00640 1 -- 172.21.15.69:0/945375392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559aaeed30b0 con 0x559aaf2f9e30 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.197+0000 7f0aebe00640 1 --2- 172.21.15.69:0/945375392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2e02a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4007b80 tx=0x7f0ae401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edce075b2b55775e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.197+0000 7f0adb400640 1 -- 172.21.15.69:0/945375392 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4021030 con 0x559aaf2f9e30 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.197+0000 7f0adb400640 1 -- 172.21.15.69:0/945375392 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae4018040 con 0x559aaf2f9e30 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.197+0000 7f0adb400640 1 -- 172.21.15.69:0/945375392 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4017070 con 0x559aaf2f9e30 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.198+0000 7f0af5433740 1 -- 172.21.15.69:0/945375392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 msgr2=0x559aaf2e02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.198+0000 7f0af5433740 1 --2- 172.21.15.69:0/945375392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2e02a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4007b80 tx=0x7f0ae401c000 comp rx=0 tx=0).stop 2024-04-24T08:12:22.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.198+0000 7f0af5433740 1 -- 172.21.15.69:0/945375392 shutdown_connections 2024-04-24T08:12:22.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.198+0000 7f0af5433740 1 --2- 172.21.15.69:0/945375392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2f9e30 0x559aaf2e02a0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.198+0000 7f0af5433740 1 --2- 172.21.15.69:0/945375392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2e4890 0x559aaf2dfd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.198+0000 7f0af5433740 1 --2- 172.21.15.69:0/945375392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2c6ff0 0x559aaf29d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.198+0000 7f0af5433740 1 -- 172.21.15.69:0/945375392 >> 172.21.15.69:0/945375392 conn(0x559aaf29ccd0 msgr2=0x559aaf292a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.198+0000 7f0af5433740 1 -- 172.21.15.69:0/945375392 shutdown_connections 2024-04-24T08:12:22.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.198+0000 7f0af5433740 1 -- 172.21.15.69:0/945375392 wait complete. 2024-04-24T08:12:22.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.198+0000 7f0af5433740 1 -- start start 2024-04-24T08:12:22.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.199+0000 7f0af5433740 1 Processor -- start 2024-04-24T08:12:22.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0af5433740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2c6ff0 0x559aaf3096e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0af5433740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2e4890 0x559aaf35ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0af5433740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2f9e30 0x559aaf35b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0af5433740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559aaf2c86f0 con 0x559aaf2e4890 2024-04-24T08:12:22.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0af5433740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x559aaf2fab60 con 0x559aaf2c6ff0 2024-04-24T08:12:22.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0af5433740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x559aaf2965c0 con 0x559aaf2f9e30 2024-04-24T08:12:22.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0af5433740 10 client.0 _ll_register_callbacks cb 0x7f0ae4015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:22.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0ae2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2e4890 0x559aaf35ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0ae2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2e4890 0x559aaf35ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44310/0 (socket says 172.21.15.69:44310) 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0ae2a00640 1 -- 172.21.15.69:0/2863317269 learned_addr learned my addr 172.21.15.69:0/2863317269 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0ae2a00640 1 -- 172.21.15.69:0/2863317269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2f9e30 msgr2=0x559aaf35b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2f9e30 0x559aaf35b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0ae2a00640 1 -- 172.21.15.69:0/2863317269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2c6ff0 msgr2=0x559aaf3096e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2c6ff0 0x559aaf3096e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.200+0000 7f0ae2a00640 1 -- 172.21.15.69:0/2863317269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aaf29aaa0 con 0x559aaf2e4890 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0ae2a00640 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2e4890 0x559aaf35ad40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4009890 tx=0x7f0ad40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6e77227ce66f8c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0ad8c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0ad8c00640 1 -- 172.21.15.69:0/2863317269 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4037040 con 0x559aaf2e4890 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0ad8c00640 1 -- 172.21.15.69:0/2863317269 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f0ad403c870 con 0x559aaf2e4890 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0af5433740 1 -- 172.21.15.69:0/2863317269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ae4018040 con 0x559aaf2e4890 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0ad8c00640 1 -- 172.21.15.69:0/2863317269 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f0ad4055050 con 0x559aaf2e4890 2024-04-24T08:12:22.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0af5433740 10 client.6489 fetch_fsmap learned FSMap version 34 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0af5433740 1 -- 172.21.15.69:0/2863317269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x559aaf29d560 con 0x559aaf2e4890 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0ad8c00640 1 -- 172.21.15.69:0/2863317269 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae4018040 con 0x559aaf2e4890 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0ad8c00640 1 -- 172.21.15.69:0/2863317269 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7f0ad4058020 con 0x559aaf2e4890 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0af5433740 10 client.6489 fetch_fsmap finished waiting for FSMap version 34 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0af5433740 10 client.6489 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0af5433740 20 client.6489 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0af5433740 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x559aaf2e0000 0x559aaf2e03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.201+0000 7f0af5433740 4 client.6489 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.202+0000 7f0af5433740 1 -- 172.21.15.69:0/2863317269 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x559aaf2dfa80 con 0x559aaf2e0000 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.202+0000 7f0aeb400640 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x559aaf2e0000 0x559aaf2e03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.202+0000 7f0aeb400640 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x559aaf2e0000 0x559aaf2e03e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2d823398c8403e2 server_cookie=8dece56b3fe26f0d in_seq=0 out_seq=0 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.202+0000 7f0ad8c00640 10 client.6489 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.202+0000 7f0ad8c00640 1 -- 172.21.15.69:0/2863317269 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+141 (crc 0 0 0) 0x559aaf2dfa80 con 0x559aaf2e0000 2024-04-24T08:12:22.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.202+0000 7f0ad8c00640 10 client.6489 handle_command_reply: tid=0 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.202+0000 7f0af5433740 1 client.6489 shutdown 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 1 -- 172.21.15.69:0/2863317269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2e4890 msgr2=0x559aaf35ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2e4890 0x559aaf35ad40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4009890 tx=0x7f0ad40046c0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 1 -- 172.21.15.69:0/2863317269 shutdown_connections 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x559aaf2e0000 0x559aaf2e03e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559aaf2e4890 0x559aaf35ad40 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x559aaf2f9e30 0x559aaf35b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 1 --2- 172.21.15.69:0/2863317269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559aaf2c6ff0 0x559aaf3096e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0ad8c00640 0 client.6489 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 1 -- 172.21.15.69:0/2863317269 >> 172.21.15.69:0/2863317269 conn(0x559aaf29ccd0 msgr2=0x559aaf292a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 1 -- 172.21.15.69:0/2863317269 shutdown_connections 2024-04-24T08:12:22.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 1 -- 172.21.15.69:0/2863317269 wait complete. 2024-04-24T08:12:22.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.203+0000 7f0af5433740 20 client.6489 trim_cache size 0 max 16384 2024-04-24T08:12:22.209 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:12:22.209 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:12:22.209 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:12:22.209 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-04-24T08:12:22.209 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2099099140, 2024-04-24T08:12:22.209 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511627777, 2024-04-24T08:12:22.209 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/dir_test/file" 2024-04-24T08:12:22.209 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:12:22.209 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:12:22.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 -- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ac806c8f0 msgr2=0x7f0ac806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ac806c8f0 0x7f0ac806edb0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4034000 tx=0x7f0ad402e000 comp rx=0 tx=0).stop 2024-04-24T08:12:22.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 -- 172.21.15.69:0/2469298994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec06fd10 msgr2=0x7f0aec159810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec06fd10 0x7f0aec159810 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00cf60 tx=0x7f0adc00cf90 comp rx=0 tx=0).stop 2024-04-24T08:12:22.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 -- 172.21.15.69:0/2469298994 shutdown_connections 2024-04-24T08:12:22.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ac806c8f0 0x7f0ac806edb0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0aec15c820 0x7f0aec150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0aec0b46a0 0x7f0aec1e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 --2- 172.21.15.69:0/2469298994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0aec06fd10 0x7f0aec159810 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 -- 172.21.15.69:0/2469298994 >> 172.21.15.69:0/2469298994 conn(0x7f0aec0d0350 msgr2=0x7f0aec0ceeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 -- 172.21.15.69:0/2469298994 shutdown_connections 2024-04-24T08:12:22.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.204+0000 7f0af2600640 1 -- 172.21.15.69:0/2469298994 wait complete. 2024-04-24T08:12:22.221 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T08:12:22.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.412+0000 7f406a800640 1 Processor -- start 2024-04-24T08:12:22.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.413+0000 7f406a800640 1 -- start start 2024-04-24T08:12:22.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.414+0000 7f406a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f406415ea10 0x7f40640b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.414+0000 7f406a800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f40640d4d20 0x7f40640b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.414+0000 7f406a800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40640b1930 con 0x7f40640d2ff0 2024-04-24T08:12:22.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.414+0000 7f406a800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f40640b1700 con 0x7f406415ea10 2024-04-24T08:12:22.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.414+0000 7f406a800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f40640b2540 con 0x7f40640d4d20 2024-04-24T08:12:22.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.414+0000 7f4069400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f406415ea10 0x7f40640b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.414+0000 7f4069400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f406415ea10 0x7f40640b11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44318/0 (socket says 172.21.15.69:44318) 2024-04-24T08:12:22.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.414+0000 7f4069400640 1 -- 172.21.15.69:0/3082867658 learned_addr learned my addr 172.21.15.69:0/3082867658 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.415+0000 7f4069400640 1 -- 172.21.15.69:0/3082867658 >> v2:172.21.15.121:3300/0 conn(0x7f40640d4d20 msgr2=0x7f40640b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.415+0000 7f4069400640 1 --2- 172.21.15.69:0/3082867658 >> v2:172.21.15.121:3300/0 conn(0x7f40640d4d20 0x7f40640b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.415+0000 7f4069400640 1 -- 172.21.15.69:0/3082867658 >> v1:172.21.15.121:6789/0 conn(0x7f40640d2ff0 legacy=0x7f40640d4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.415+0000 7f4069400640 1 -- 172.21.15.69:0/3082867658 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40640b1c10 con 0x7f406415ea10 2024-04-24T08:12:22.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.416+0000 7f4069400640 1 --2- 172.21.15.69:0/3082867658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f406415ea10 0x7f40640b11c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f40640cedd0 tx=0x7f405402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c32b3201852ce11 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.416+0000 7f4063e00640 1 -- 172.21.15.69:0/3082867658 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054036070 con 0x7f406415ea10 2024-04-24T08:12:22.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.416+0000 7f4063e00640 1 -- 172.21.15.69:0/3082867658 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4054031600 con 0x7f406415ea10 2024-04-24T08:12:22.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.416+0000 7f4063e00640 1 -- 172.21.15.69:0/3082867658 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40540319d0 con 0x7f406415ea10 2024-04-24T08:12:22.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.417+0000 7f406a800640 1 -- 172.21.15.69:0/3082867658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f406415ea10 msgr2=0x7f40640b11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.417+0000 7f406a800640 1 --2- 172.21.15.69:0/3082867658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f406415ea10 0x7f40640b11c0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f40640cedd0 tx=0x7f405402a6f0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.417+0000 7f406a800640 1 -- 172.21.15.69:0/3082867658 shutdown_connections 2024-04-24T08:12:22.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.417+0000 7f406a800640 1 --2- 172.21.15.69:0/3082867658 >> v2:172.21.15.121:3300/0 conn(0x7f40640d4d20 0x7f40640b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.417+0000 7f406a800640 1 --2- 172.21.15.69:0/3082867658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f406415ea10 0x7f40640b11c0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.417+0000 7f406a800640 1 -- 172.21.15.69:0/3082867658 >> 172.21.15.69:0/3082867658 conn(0x7f406408dfb0 msgr2=0x7f406409d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.417+0000 7f406a800640 1 -- 172.21.15.69:0/3082867658 shutdown_connections 2024-04-24T08:12:22.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.417+0000 7f406a800640 1 -- 172.21.15.69:0/3082867658 wait complete. 2024-04-24T08:12:22.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.418+0000 7f406a800640 1 Processor -- start 2024-04-24T08:12:22.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.419+0000 7f406a800640 1 -- start start 2024-04-24T08:12:22.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.419+0000 7f406a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40640d2ff0 0x7f40640ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.419+0000 7f406a800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40640d4d20 0x7f40641f5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.419+0000 7f406a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f406415ea10 0x7f40640b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.419+0000 7f406a800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f406415edf0 con 0x7f40640d2ff0 2024-04-24T08:12:22.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.419+0000 7f406a800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f40640b21d0 con 0x7f40640d4d20 2024-04-24T08:12:22.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.419+0000 7f406a800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f40640d3450 con 0x7f406415ea10 2024-04-24T08:12:22.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.420+0000 7f4069400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40640d2ff0 0x7f40640ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.420+0000 7f4069400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40640d2ff0 0x7f40640ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44326/0 (socket says 172.21.15.69:44326) 2024-04-24T08:12:22.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.420+0000 7f4069400640 1 -- 172.21.15.69:0/3738600549 learned_addr learned my addr 172.21.15.69:0/3738600549 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.420+0000 7f4069400640 1 -- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f406415ea10 msgr2=0x7f40640b6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.420+0000 7f4069400640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f406415ea10 0x7f40640b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.420+0000 7f4069400640 1 -- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40640d4d20 msgr2=0x7f40641f5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.420+0000 7f4069400640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40640d4d20 0x7f40641f5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.420+0000 7f4069400640 1 -- 172.21.15.69:0/3738600549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4064154e60 con 0x7f40640d2ff0 2024-04-24T08:12:22.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.420+0000 7f4069400640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40640d2ff0 0x7f40640ccf30 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4054034040 tx=0x7f4054004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.421+0000 7f4061600640 1 -- 172.21.15.69:0/3738600549 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054041020 con 0x7f40640d2ff0 2024-04-24T08:12:22.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.421+0000 7f4061600640 1 -- 172.21.15.69:0/3738600549 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4054007820 con 0x7f40640d2ff0 2024-04-24T08:12:22.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.421+0000 7f4061600640 1 -- 172.21.15.69:0/3738600549 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054036040 con 0x7f40640d2ff0 2024-04-24T08:12:22.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.421+0000 7f406a800640 1 -- 172.21.15.69:0/3738600549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40641f5a50 con 0x7f40640d2ff0 2024-04-24T08:12:22.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.421+0000 7f406a800640 1 -- 172.21.15.69:0/3738600549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4064156a20 con 0x7f40640d2ff0 2024-04-24T08:12:22.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.422+0000 7f406a800640 1 -- 172.21.15.69:0/3738600549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4064075d50 con 0x7f40640d2ff0 2024-04-24T08:12:22.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.423+0000 7f4061600640 1 -- 172.21.15.69:0/3738600549 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4054044020 con 0x7f40640d2ff0 2024-04-24T08:12:22.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.424+0000 7f4061600640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f404406c8f0 0x7f404406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.424+0000 7f4061600640 1 -- 172.21.15.69:0/3738600549 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f40540aca40 con 0x7f40640d2ff0 2024-04-24T08:12:22.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.425+0000 7f4068a00640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f404406c8f0 0x7f404406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.426+0000 7f4068a00640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f404406c8f0 0x7f404406edb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f40640cd830 tx=0x7f4050006ba0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.429+0000 7f4061600640 1 -- 172.21.15.69:0/3738600549 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f405404b030 con 0x7f40640d2ff0 2024-04-24T08:12:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.671+0000 7f406a800640 1 -- 172.21.15.69:0/3738600549 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f40640ce990 con 0x7f40640d2ff0 2024-04-24T08:12:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.672+0000 7f4061600640 1 -- 172.21.15.69:0/3738600549 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+164 (secure 0 0 0) 0x7f40641f5a50 con 0x7f40640d2ff0 2024-04-24T08:12:22.673 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:12:22.673 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-04-24T08:12:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.674+0000 7f406a800640 1 -- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f404406c8f0 msgr2=0x7f404406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.674+0000 7f406a800640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f404406c8f0 0x7f404406edb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f40640cd830 tx=0x7f4050006ba0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.675+0000 7f406a800640 1 -- 172.21.15.69:0/3738600549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40640d2ff0 msgr2=0x7f40640ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.675+0000 7f406a800640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40640d2ff0 0x7f40640ccf30 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4054034040 tx=0x7f4054004a40 comp rx=0 tx=0).stop 2024-04-24T08:12:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.675+0000 7f406a800640 1 -- 172.21.15.69:0/3738600549 shutdown_connections 2024-04-24T08:12:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.675+0000 7f406a800640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f404406c8f0 0x7f404406edb0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.675+0000 7f406a800640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f406415ea10 0x7f40640b6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.675+0000 7f406a800640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40640d4d20 0x7f40641f5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.675+0000 7f406a800640 1 --2- 172.21.15.69:0/3738600549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40640d2ff0 0x7f40640ccf30 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.675+0000 7f406a800640 1 -- 172.21.15.69:0/3738600549 >> 172.21.15.69:0/3738600549 conn(0x7f406408dfb0 msgr2=0x7f40640cf020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.675+0000 7f406a800640 1 -- 172.21.15.69:0/3738600549 shutdown_connections 2024-04-24T08:12:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.675+0000 7f406a800640 1 -- 172.21.15.69:0/3738600549 wait complete. 2024-04-24T08:12:22.696 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-24T08:12:22.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:22.696 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub start / repair,recursive,force 2024-04-24T08:12:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.897+0000 7ff1ed600640 1 Processor -- start 2024-04-24T08:12:22.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.897+0000 7ff1ed600640 1 -- start start 2024-04-24T08:12:22.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.898+0000 7ff1ed600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80d07c0 0x7ff1e80d0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.898+0000 7ff1ed600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1e80b4840 con 0x7ff1e80addc0 2024-04-24T08:12:22.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.898+0000 7ff1ed600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1e80b4560 con 0x7ff1e815c590 2024-04-24T08:12:22.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.898+0000 7ff1ed600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e80d1160 con 0x7ff1e80d07c0 2024-04-24T08:12:22.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.898+0000 7ff1e7400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff1e80addc0 0x7ff1e80ae1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:41902/0 (socket says 172.21.15.69:41902) 2024-04-24T08:12:22.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.898+0000 7ff1e7400640 1 -- 172.21.15.69:0/3129691173 learned_addr learned my addr 172.21.15.69:0/3129691173 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e6000640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80d07c0 0x7ff1e80d0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2618527861 0 0) 0x7ff1e80b4840 con 0x7ff1e80addc0 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1cc003520 con 0x7ff1e80addc0 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2499737544 0 0) 0x7ff1cc003520 con 0x7ff1e80addc0 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80d07c0 msgr2=0x7ff1e80d0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80d07c0 0x7ff1e80d0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 >> v1:172.21.15.121:6789/0 conn(0x7ff1e815c590 legacy=0x7ff1e80b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e80ad2e0 con 0x7ff1e80addc0 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7ff1e00019f0 con 0x7ff1e80addc0 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 >> v1:172.21.15.69:6789/0 conn(0x7ff1e80addc0 legacy=0x7ff1e80ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80d07c0 0x7ff1cc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1cc003fb0 0x7ff1cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1cc004d40 0x7ff1cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc00bad0 con 0x7ff1cc003fb0 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc00bc80 con 0x7ff1cc004d40 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.899+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff1cc00be80 con 0x7ff1e80d07c0 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.900+0000 7ff1e6000640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1cc003fb0 0x7ff1cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.900+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80d07c0 0x7ff1cc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.900+0000 7ff1e7400640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1cc004d40 0x7ff1cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.900+0000 7ff1e6000640 1 -- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80d07c0 msgr2=0x7ff1cc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.900+0000 7ff1e6000640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80d07c0 0x7ff1cc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.900+0000 7ff1e6000640 1 -- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1cc004d40 msgr2=0x7ff1cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.900+0000 7ff1e6000640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1cc004d40 0x7ff1cc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.900+0000 7ff1e6000640 1 -- 172.21.15.69:0/3129691173 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc003520 con 0x7ff1cc003fb0 2024-04-24T08:12:22.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.900+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80d07c0 0x7ff1cc0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:22.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.901+0000 7ff1e6000640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1cc003fb0 0x7ff1cc004800 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009ac0 tx=0x7ff1cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1f403f48ec321a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.902+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0036070 con 0x7ff1cc003fb0 2024-04-24T08:12:22.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.902+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d0031400 con 0x7ff1cc003fb0 2024-04-24T08:12:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.902+0000 7ff1e5600640 1 -- 172.21.15.69:0/3129691173 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0009690 con 0x7ff1cc003fb0 2024-04-24T08:12:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.902+0000 7ff1ed600640 1 -- 172.21.15.69:0/3129691173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1cc003fb0 msgr2=0x7ff1cc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.902+0000 7ff1ed600640 1 --2- 172.21.15.69:0/3129691173 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1cc003fb0 0x7ff1cc004800 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009ac0 tx=0x7ff1cc005830 comp rx=0 tx=0).stop 2024-04-24T08:12:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.902+0000 7ff1e7400640 1 -- 172.21.15.69:0/3129691173 reap_dead start 2024-04-24T08:12:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.902+0000 7ff1ed600640 1 -- 172.21.15.69:0/3129691173 shutdown_connections 2024-04-24T08:12:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.902+0000 7ff1ed600640 1 -- 172.21.15.69:0/3129691173 >> 172.21.15.69:0/3129691173 conn(0x7ff1e806be40 msgr2=0x7ff1e806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.902+0000 7ff1ed600640 1 -- 172.21.15.69:0/3129691173 shutdown_connections 2024-04-24T08:12:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.902+0000 7ff1ed600640 1 -- 172.21.15.69:0/3129691173 wait complete. 2024-04-24T08:12:22.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.904+0000 7ff1ed600640 1 Processor -- start 2024-04-24T08:12:22.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.904+0000 7ff1ed600640 1 -- start start 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.905+0000 7ff1ed600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1cc003fb0 0x7ff1e8099930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.905+0000 7ff1ed600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80c75c0 0x7ff1e81579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.905+0000 7ff1ed600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e80b0ca0 0x7ff1e80c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.905+0000 7ff1ed600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e80b4240 con 0x7ff1e80b0ca0 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.905+0000 7ff1ed600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e80ad8a0 con 0x7ff1cc003fb0 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.905+0000 7ff1ed600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e80b5160 con 0x7ff1e80c75c0 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e80b0ca0 0x7ff1e80c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e80b0ca0 0x7ff1e80c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44336/0 (socket says 172.21.15.69:44336) 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e7400640 1 -- 172.21.15.69:0/1887209724 learned_addr learned my addr 172.21.15.69:0/1887209724 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1cc003fb0 0x7ff1e8099930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e6000640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80c75c0 0x7ff1e81579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e7400640 1 -- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80c75c0 msgr2=0x7ff1e81579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e7400640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80c75c0 0x7ff1e81579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e7400640 1 -- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1cc003fb0 msgr2=0x7ff1e8099930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e7400640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1cc003fb0 0x7ff1e8099930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e7400640 1 -- 172.21.15.69:0/1887209724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e806db90 con 0x7ff1e80b0ca0 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.906+0000 7ff1e7400640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e80b0ca0 0x7ff1e80c7e80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80b2f40 tx=0x7ff1e000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.907+0000 7ff1dea00640 1 -- 172.21.15.69:0/1887209724 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e000eb90 con 0x7ff1e80b0ca0 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.907+0000 7ff1dea00640 1 -- 172.21.15.69:0/1887209724 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e0018070 con 0x7ff1e80b0ca0 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.907+0000 7ff1dea00640 1 -- 172.21.15.69:0/1887209724 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0013420 con 0x7ff1e80b0ca0 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.907+0000 7ff1ed600640 1 -- 172.21.15.69:0/1887209724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e80ca440 con 0x7ff1e80b0ca0 2024-04-24T08:12:22.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.907+0000 7ff1ed600640 1 -- 172.21.15.69:0/1887209724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e806de30 con 0x7ff1e80b0ca0 2024-04-24T08:12:22.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.910+0000 7ff1dea00640 1 -- 172.21.15.69:0/1887209724 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1e00135c0 con 0x7ff1e80b0ca0 2024-04-24T08:12:22.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.911+0000 7ff1dea00640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff1c406c8f0 0x7ff1c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.911+0000 7ff1dea00640 1 -- 172.21.15.69:0/1887209724 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7ff1e008be50 con 0x7ff1e80b0ca0 2024-04-24T08:12:22.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.911+0000 7ff1f03df740 1 Processor -- start 2024-04-24T08:12:22.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.912+0000 7ff1f03df740 1 -- start start 2024-04-24T08:12:22.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.912+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff1c406c8f0 0x7ff1c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08de7bf0 0x560e08e0fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08de7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 0x560e08de63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1f03df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560e08d4cff0 con 0x560e08e0ff70 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1f03df740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x560e08cf2f40 con 0x560e08de7bf0 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1f03df740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x560e08d48fb0 con 0x560e08e30860 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff1c406c8f0 0x7ff1c406edb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8007fd0 tx=0x7ff1d8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08de7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08de7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44352/0 (socket says 172.21.15.69:44352) 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e6000640 1 -- 172.21.15.69:0/1133790456 learned_addr learned my addr 172.21.15.69:0/1133790456 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/1133790456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08de7bf0 0x560e08e0fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e7400640 1 --2- 172.21.15.69:0/1133790456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 0x560e08de63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e6000640 1 -- 172.21.15.69:0/1133790456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 msgr2=0x560e08de63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e6000640 1 --2- 172.21.15.69:0/1133790456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 0x560e08de63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e6000640 1 -- 172.21.15.69:0/1133790456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08de7bf0 msgr2=0x560e08e0fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e6000640 1 --2- 172.21.15.69:0/1133790456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08de7bf0 0x560e08e0fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.913+0000 7ff1e6000640 1 -- 172.21.15.69:0/1133790456 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e08b2b940 con 0x560e08e0ff70 2024-04-24T08:12:22.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.914+0000 7ff1e6000640 1 --2- 172.21.15.69:0/1133790456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08de7fd0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0034040 tx=0x7ff1d002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba5ef3e2764e40d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.914+0000 7ff1d7000640 1 -- 172.21.15.69:0/1133790456 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0040070 con 0x560e08e0ff70 2024-04-24T08:12:22.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.914+0000 7ff1d7000640 1 -- 172.21.15.69:0/1133790456 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d00043a0 con 0x560e08e0ff70 2024-04-24T08:12:22.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.914+0000 7ff1d7000640 1 -- 172.21.15.69:0/1133790456 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0036040 con 0x560e08e0ff70 2024-04-24T08:12:22.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.915+0000 7ff1f03df740 1 -- 172.21.15.69:0/1133790456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 msgr2=0x560e08de7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.915+0000 7ff1f03df740 1 --2- 172.21.15.69:0/1133790456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08de7fd0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0034040 tx=0x7ff1d002e000 comp rx=0 tx=0).stop 2024-04-24T08:12:22.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.915+0000 7ff1f03df740 1 -- 172.21.15.69:0/1133790456 shutdown_connections 2024-04-24T08:12:22.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.915+0000 7ff1f03df740 1 --2- 172.21.15.69:0/1133790456 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08de7fd0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.915+0000 7ff1f03df740 1 --2- 172.21.15.69:0/1133790456 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 0x560e08de63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.915+0000 7ff1f03df740 1 --2- 172.21.15.69:0/1133790456 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08de7bf0 0x560e08e0fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.915+0000 7ff1f03df740 1 -- 172.21.15.69:0/1133790456 >> 172.21.15.69:0/1133790456 conn(0x560e08de4e10 msgr2=0x560e08e2d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.915+0000 7ff1f03df740 1 -- 172.21.15.69:0/1133790456 shutdown_connections 2024-04-24T08:12:22.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.915+0000 7ff1f03df740 1 -- 172.21.15.69:0/1133790456 wait complete. 2024-04-24T08:12:22.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.916+0000 7ff1f03df740 1 -- start start 2024-04-24T08:12:22.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.916+0000 7ff1f03df740 1 Processor -- start 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0ff70 0x560e08e074f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e30860 0x560e08e3def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e0d930 0x560e08e3e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1f03df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560e08e0f5a0 con 0x560e08e30860 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1f03df740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x560e08d062c0 con 0x560e08e0ff70 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1f03df740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x560e08e0ef90 con 0x560e08e0d930 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1f03df740 10 client.0 _ll_register_callbacks cb 0x560e08cd4c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e30860 0x560e08e3def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e30860 0x560e08e3def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44356/0 (socket says 172.21.15.69:44356) 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1e6000640 1 -- 172.21.15.69:0/2210849524 learned_addr learned my addr 172.21.15.69:0/2210849524 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0ff70 0x560e08e074f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.917+0000 7ff1e6000640 1 -- 172.21.15.69:0/2210849524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e0d930 msgr2=0x560e08e3e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1e6000640 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e0d930 0x560e08e3e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1e6000640 1 -- 172.21.15.69:0/2210849524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0ff70 msgr2=0x560e08e074f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1e6000640 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0ff70 0x560e08e074f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1e6000640 1 -- 172.21.15.69:0/2210849524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e08b8c0d0 con 0x560e08e30860 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0ff70 0x560e08e074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1e6000640 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e30860 0x560e08e3def0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009b20 tx=0x7ff1d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c994a71c567f015a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1cb400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1cb400640 1 -- 172.21.15.69:0/2210849524 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0040070 con 0x560e08e30860 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1cb400640 1 -- 172.21.15.69:0/2210849524 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7ff1d0003810 con 0x560e08e30860 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1f03df740 1 -- 172.21.15.69:0/2210849524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d00043a0 con 0x560e08e30860 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1cb400640 1 -- 172.21.15.69:0/2210849524 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff1d0036040 con 0x560e08e30860 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1f03df740 10 client.6507 fetch_fsmap learned FSMap version 34 2024-04-24T08:12:22.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.918+0000 7ff1f03df740 1 -- 172.21.15.69:0/2210849524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x560e08a1f230 con 0x560e08e30860 2024-04-24T08:12:22.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.919+0000 7ff1cb400640 1 -- 172.21.15.69:0/2210849524 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d00043a0 con 0x560e08e30860 2024-04-24T08:12:22.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.919+0000 7ff1cb400640 1 -- 172.21.15.69:0/2210849524 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7ff1d0003c20 con 0x560e08e30860 2024-04-24T08:12:22.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.919+0000 7ff1f03df740 10 client.6507 fetch_fsmap finished waiting for FSMap version 34 2024-04-24T08:12:22.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.919+0000 7ff1f03df740 10 client.6507 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:12:22.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.919+0000 7ff1f03df740 20 client.6507 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:22.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.919+0000 7ff1f03df740 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x560e08e4a800 0x560e08e4cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.919+0000 7ff1f03df740 4 client.6507 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:22.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.919+0000 7ff1f03df740 1 -- 172.21.15.69:0/2210849524 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e08d5aa10 con 0x560e08e4a800 2024-04-24T08:12:22.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.919+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x560e08e4a800 0x560e08e4cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.920+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x560e08e4a800 0x560e08e4cbf0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2abc8ef857ab2c78 server_cookie=552f8d1274f09cd3 in_seq=0 out_seq=0 2024-04-24T08:12:22.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.920+0000 7ff1cb400640 10 client.6507 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1cb400640 1 -- 172.21.15.69:0/2210849524 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e08d5aa10 con 0x560e08e4a800 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1cb400640 10 client.6507 handle_command_reply: tid=0 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 client.6507 shutdown 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 -- 172.21.15.69:0/2210849524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e30860 msgr2=0x560e08e3def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e30860 0x560e08e3def0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009b20 tx=0x7ff1d00046c0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 -- 172.21.15.69:0/2210849524 shutdown_connections 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x560e08e4a800 0x560e08e4cbf0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e30860 0x560e08e3def0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e0d930 0x560e08e3e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 --2- 172.21.15.69:0/2210849524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0ff70 0x560e08e074f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1cb400640 0 client.6507 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 -- 172.21.15.69:0/2210849524 >> 172.21.15.69:0/2210849524 conn(0x560e08de7bf0 msgr2=0x560e08de1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 -- 172.21.15.69:0/2210849524 shutdown_connections 2024-04-24T08:12:22.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 1 -- 172.21.15.69:0/2210849524 wait complete. 2024-04-24T08:12:22.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.921+0000 7ff1f03df740 20 client.6507 trim_cache size 0 max 16384 2024-04-24T08:12:22.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.940+0000 7ff1f03df740 1 Processor -- start 2024-04-24T08:12:22.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.940+0000 7ff1f03df740 1 -- start start 2024-04-24T08:12:22.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.941+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e0d930 0x560e08de2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.941+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e14a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.941+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e30860 0x560e08e14fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.941+0000 7ff1f03df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e8201370 con 0x560e08e0ff70 2024-04-24T08:12:22.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.941+0000 7ff1f03df740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e8200390 con 0x560e08e30860 2024-04-24T08:12:22.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.941+0000 7ff1f03df740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e806efc0 con 0x560e08e0d930 2024-04-24T08:12:22.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.941+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e14a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.941+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e14a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44364/0 (socket says 172.21.15.69:44364) 2024-04-24T08:12:22.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.941+0000 7ff1e6000640 1 -- 172.21.15.69:0/3473594729 learned_addr learned my addr 172.21.15.69:0/3473594729 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.942+0000 7ff1e6000640 1 -- 172.21.15.69:0/3473594729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e0d930 msgr2=0x560e08de2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.942+0000 7ff1e6000640 1 --2- 172.21.15.69:0/3473594729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e0d930 0x560e08de2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.942+0000 7ff1e6000640 1 -- 172.21.15.69:0/3473594729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e30860 msgr2=0x560e08e14fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.942+0000 7ff1e6000640 1 --2- 172.21.15.69:0/3473594729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e30860 0x560e08e14fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.942+0000 7ff1e6000640 1 -- 172.21.15.69:0/3473594729 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e08e2f780 con 0x560e08e0ff70 2024-04-24T08:12:22.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.942+0000 7ff1e6000640 1 --2- 172.21.15.69:0/3473594729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e14a70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009160 tx=0x7ff1d0047070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455dfbd9250c257a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.942+0000 7ff1d6600640 1 -- 172.21.15.69:0/3473594729 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0040040 con 0x560e08e0ff70 2024-04-24T08:12:22.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.942+0000 7ff1d6600640 1 -- 172.21.15.69:0/3473594729 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d0049070 con 0x560e08e0ff70 2024-04-24T08:12:22.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.942+0000 7ff1d6600640 1 -- 172.21.15.69:0/3473594729 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0003850 con 0x560e08e0ff70 2024-04-24T08:12:22.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.943+0000 7ff1f03df740 1 -- 172.21.15.69:0/3473594729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 msgr2=0x560e08e14a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.943+0000 7ff1f03df740 1 --2- 172.21.15.69:0/3473594729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e14a70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009160 tx=0x7ff1d0047070 comp rx=0 tx=0).stop 2024-04-24T08:12:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.943+0000 7ff1f03df740 1 -- 172.21.15.69:0/3473594729 shutdown_connections 2024-04-24T08:12:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.943+0000 7ff1f03df740 1 --2- 172.21.15.69:0/3473594729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e14a70 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.943+0000 7ff1f03df740 1 --2- 172.21.15.69:0/3473594729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e30860 0x560e08e14fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.943+0000 7ff1f03df740 1 --2- 172.21.15.69:0/3473594729 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e0d930 0x560e08de2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.943+0000 7ff1f03df740 1 -- 172.21.15.69:0/3473594729 >> 172.21.15.69:0/3473594729 conn(0x560e08de7bf0 msgr2=0x560e08e2d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.943+0000 7ff1f03df740 1 -- 172.21.15.69:0/3473594729 shutdown_connections 2024-04-24T08:12:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.943+0000 7ff1f03df740 1 -- 172.21.15.69:0/3473594729 wait complete. 2024-04-24T08:12:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.944+0000 7ff1f03df740 1 -- start start 2024-04-24T08:12:22.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.944+0000 7ff1f03df740 1 Processor -- start 2024-04-24T08:12:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.945+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0d930 0x560e08e51ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.945+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e52420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.945+0000 7ff1f03df740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 0x560e08e08c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.945+0000 7ff1f03df740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560e08e0fd80 con 0x560e08e0ff70 2024-04-24T08:12:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.945+0000 7ff1f03df740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e81ffdd0 con 0x560e08e0d930 2024-04-24T08:12:22.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.945+0000 7ff1f03df740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x560e08ea9000 con 0x560e08e30860 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.945+0000 7ff1f03df740 10 client.0 _ll_register_callbacks cb 0x7ff1d0009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.945+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e52420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.945+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e52420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44376/0 (socket says 172.21.15.69:44376) 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.945+0000 7ff1e6000640 1 -- 172.21.15.69:0/721922921 learned_addr learned my addr 172.21.15.69:0/721922921 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0d930 0x560e08e51ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1e7400640 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 0x560e08e08c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1e6000640 1 -- 172.21.15.69:0/721922921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 msgr2=0x560e08e08c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1e6000640 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 0x560e08e08c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1e6000640 1 -- 172.21.15.69:0/721922921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0d930 msgr2=0x560e08e51ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1e6000640 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0d930 0x560e08e51ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1e6000640 1 -- 172.21.15.69:0/721922921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e08de9bc0 con 0x560e08e0ff70 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1e7400640 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 0x560e08e08c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1e6000640 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e52420 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009590 tx=0x7ff1d0008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43787216236aec4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:22.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1cb400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1cb400640 1 -- 172.21.15.69:0/721922921 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0040070 con 0x560e08e0ff70 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1cb400640 1 -- 172.21.15.69:0/721922921 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7ff1d004e940 con 0x560e08e0ff70 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.946+0000 7ff1f03df740 1 -- 172.21.15.69:0/721922921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff1d0049070 con 0x560e08e0ff70 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1cb400640 1 -- 172.21.15.69:0/721922921 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7ff1d0064050 con 0x560e08e0ff70 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1f03df740 10 client.6513 fetch_fsmap learned FSMap version 34 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1f03df740 1 -- 172.21.15.69:0/721922921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x560e08e0a450 con 0x560e08e0ff70 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1cb400640 1 -- 172.21.15.69:0/721922921 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d0049070 con 0x560e08e0ff70 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1cb400640 1 -- 172.21.15.69:0/721922921 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7ff1d0067020 con 0x560e08e0ff70 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1f03df740 10 client.6513 fetch_fsmap finished waiting for FSMap version 34 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1f03df740 10 client.6513 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1f03df740 20 client.6513 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1f03df740 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x560e08e3d220 0x560e08e091c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1f03df740 4 client.6513 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}] 2024-04-24T08:12:22.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1f03df740 1 -- 172.21.15.69:0/721922921 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}) -- 0x560e08de5570 con 0x560e08e3d220 2024-04-24T08:12:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.947+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x560e08e3d220 0x560e08e091c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.948+0000 7ff1e6a00640 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x560e08e3d220 0x560e08e091c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e8a12005b20ccce server_cookie=c0c8d5ce7f79cf6e in_seq=0 out_seq=0 2024-04-24T08:12:22.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.948+0000 7ff1cb400640 10 client.6513 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1cb400640 1 -- 172.21.15.69:0/721922921 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560e08de5570 con 0x560e08e3d220 2024-04-24T08:12:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1cb400640 10 client.6513 handle_command_reply: tid=0 2024-04-24T08:12:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 client.6513 shutdown 2024-04-24T08:12:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 -- 172.21.15.69:0/721922921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 msgr2=0x560e08e52420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e52420 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0009590 tx=0x7ff1d0008030 comp rx=0 tx=0).stop 2024-04-24T08:12:22.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 -- 172.21.15.69:0/721922921 shutdown_connections 2024-04-24T08:12:22.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x560e08e3d220 0x560e08e091c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560e08e0ff70 0x560e08e52420 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560e08e30860 0x560e08e08c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 --2- 172.21.15.69:0/721922921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560e08e0d930 0x560e08e51ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 -- 172.21.15.69:0/721922921 >> 172.21.15.69:0/721922921 conn(0x560e08de7bf0 msgr2=0x560e08de1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1cb400640 0 client.6513 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:22.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 -- 172.21.15.69:0/721922921 shutdown_connections 2024-04-24T08:12:22.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 1 -- 172.21.15.69:0/721922921 wait complete. 2024-04-24T08:12:22.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.949+0000 7ff1f03df740 20 client.6513 trim_cache size 0 max 16384 2024-04-24T08:12:22.955 INFO:tasks.ceph.mds.c.smithi069.stderr:2024-04-24T08:12:22.949+0000 7f042c800640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-04-24T08:11:49.389063+0000 1=1+0) caps={6384=pAsLsXsFs/-@2},l=6384(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x563937a40b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-24T08:12:22.955 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:12:22.955 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-04-24T08:12:22.955 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "65ed9faa-f7d1-4316-bca0-9dc10817c488", 2024-04-24T08:12:22.955 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-04-24T08:12:22.955 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:12:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 -- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff1c406c8f0 msgr2=0x7ff1c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff1c406c8f0 0x7ff1c406edb0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8007fd0 tx=0x7ff1d8009040 comp rx=0 tx=0).stop 2024-04-24T08:12:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 -- 172.21.15.69:0/1887209724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e80b0ca0 msgr2=0x7ff1e80c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e80b0ca0 0x7ff1e80c7e80 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff1e80b2f40 tx=0x7ff1e000e530 comp rx=0 tx=0).stop 2024-04-24T08:12:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 -- 172.21.15.69:0/1887209724 shutdown_connections 2024-04-24T08:12:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff1c406c8f0 0x7ff1c406edb0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1e80b0ca0 0x7ff1e80c7e80 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1e80c75c0 0x7ff1e81579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 --2- 172.21.15.69:0/1887209724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1cc003fb0 0x7ff1e8099930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:22.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 -- 172.21.15.69:0/1887209724 >> 172.21.15.69:0/1887209724 conn(0x7ff1e806be40 msgr2=0x7ff1e8159400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:22.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 -- 172.21.15.69:0/1887209724 shutdown_connections 2024-04-24T08:12:22.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:22.950+0000 7ff1ed600640 1 -- 172.21.15.69:0/1887209724 wait complete. 2024-04-24T08:12:22.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:22.968 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-04-24T08:12:23.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.153+0000 7fb966200640 1 Processor -- start 2024-04-24T08:12:23.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.153+0000 7fb966200640 1 -- start start 2024-04-24T08:12:23.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.154+0000 7fb966200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb960162f90 0x7fb9600cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.154+0000 7fb966200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9600d0190 0x7fb96006cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.154+0000 7fb966200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb96015ed60 con 0x7fb96015e980 2024-04-24T08:12:23.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.154+0000 7fb966200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb9601633d0 con 0x7fb960162f90 2024-04-24T08:12:23.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.154+0000 7fb966200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb9600d1580 con 0x7fb9600d0190 2024-04-24T08:12:23.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.154+0000 7fb964e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb96015e980 0x7fb9600d0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:41916/0 (socket says 172.21.15.69:41916) 2024-04-24T08:12:23.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.154+0000 7fb964e00640 1 -- 172.21.15.69:0/2517165195 learned_addr learned my addr 172.21.15.69:0/2517165195 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:23.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.154+0000 7fb95fe00640 1 --2- 172.21.15.69:0/2517165195 >> v2:172.21.15.69:3300/0 conn(0x7fb960162f90 0x7fb9600cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.154+0000 7fb965800640 1 --2- 172.21.15.69:0/2517165195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9600d0190 0x7fb96006cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.154+0000 7fb95f400640 1 -- 172.21.15.69:0/2517165195 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815869024 0 0) 0x7fb96015ed60 con 0x7fb96015e980 2024-04-24T08:12:23.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.155+0000 7fb95f400640 1 -- 172.21.15.69:0/2517165195 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb944003520 con 0x7fb96015e980 2024-04-24T08:12:23.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.155+0000 7fb95fe00640 1 -- 172.21.15.69:0/2517165195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9600d0190 msgr2=0x7fb96006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:23.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.155+0000 7fb95fe00640 1 --2- 172.21.15.69:0/2517165195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9600d0190 0x7fb96006cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.155+0000 7fb95fe00640 1 -- 172.21.15.69:0/2517165195 >> v1:172.21.15.69:6789/0 conn(0x7fb96015e980 legacy=0x7fb9600d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:23.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.155+0000 7fb95fe00640 1 -- 172.21.15.69:0/2517165195 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9600d1860 con 0x7fb960162f90 2024-04-24T08:12:23.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.156+0000 7fb95fe00640 1 --2- 172.21.15.69:0/2517165195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 0x7fb9600cfc50 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb9480097a0 tx=0x7fb94802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8891c4a011c30de1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:23.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.156+0000 7fb95f400640 1 -- 172.21.15.69:0/2517165195 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb948037070 con 0x7fb960162f90 2024-04-24T08:12:23.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.156+0000 7fb95f400640 1 -- 172.21.15.69:0/2517165195 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb948032440 con 0x7fb960162f90 2024-04-24T08:12:23.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.156+0000 7fb95f400640 1 -- 172.21.15.69:0/2517165195 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9480327f0 con 0x7fb960162f90 2024-04-24T08:12:23.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.156+0000 7fb966200640 1 -- 172.21.15.69:0/2517165195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 msgr2=0x7fb9600cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:23.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.156+0000 7fb966200640 1 --2- 172.21.15.69:0/2517165195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 0x7fb9600cfc50 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb9480097a0 tx=0x7fb94802a210 comp rx=0 tx=0).stop 2024-04-24T08:12:23.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.157+0000 7fb966200640 1 -- 172.21.15.69:0/2517165195 shutdown_connections 2024-04-24T08:12:23.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.157+0000 7fb966200640 1 --2- 172.21.15.69:0/2517165195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9600d0190 0x7fb96006cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.157+0000 7fb966200640 1 --2- 172.21.15.69:0/2517165195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 0x7fb9600cfc50 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.157+0000 7fb966200640 1 -- 172.21.15.69:0/2517165195 >> 172.21.15.69:0/2517165195 conn(0x7fb96009d920 msgr2=0x7fb96008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:23.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.157+0000 7fb966200640 1 -- 172.21.15.69:0/2517165195 shutdown_connections 2024-04-24T08:12:23.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.157+0000 7fb966200640 1 -- 172.21.15.69:0/2517165195 wait complete. 2024-04-24T08:12:23.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.158+0000 7fb966200640 1 Processor -- start 2024-04-24T08:12:23.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.158+0000 7fb966200640 1 -- start start 2024-04-24T08:12:23.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.159+0000 7fb966200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9600d0190 0x7fb9601f4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.159+0000 7fb966200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb96015e980 0x7fb960158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.159+0000 7fb966200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 0x7fb9600d0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.159+0000 7fb966200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb9600cf070 con 0x7fb960162f90 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.159+0000 7fb966200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb9600cf730 con 0x7fb9600d0190 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.159+0000 7fb966200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb9600cf1f0 con 0x7fb96015e980 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb965800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 0x7fb9600d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb965800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 0x7fb9600d0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44408/0 (socket says 172.21.15.69:44408) 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb965800640 1 -- 172.21.15.69:0/303345096 learned_addr learned my addr 172.21.15.69:0/303345096 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb964e00640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9600d0190 0x7fb9601f4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb95fe00640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb96015e980 0x7fb960158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb965800640 1 -- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb96015e980 msgr2=0x7fb960158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb965800640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb96015e980 0x7fb960158da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb965800640 1 -- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9600d0190 msgr2=0x7fb9601f4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:23.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb965800640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9600d0190 0x7fb9601f4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb965800640 1 -- 172.21.15.69:0/303345096 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9600cc900 con 0x7fb960162f90 2024-04-24T08:12:23.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.160+0000 7fb965800640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 0x7fb9600d0600 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb95800d830 tx=0x7fb95800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:23.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.161+0000 7fb95cc00640 1 -- 172.21.15.69:0/303345096 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95800b6c0 con 0x7fb960162f90 2024-04-24T08:12:23.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.161+0000 7fb95cc00640 1 -- 172.21.15.69:0/303345096 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb958019070 con 0x7fb960162f90 2024-04-24T08:12:23.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.161+0000 7fb95cc00640 1 -- 172.21.15.69:0/303345096 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95800bac0 con 0x7fb960162f90 2024-04-24T08:12:23.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.161+0000 7fb966200640 1 -- 172.21.15.69:0/303345096 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9600ccbe0 con 0x7fb960162f90 2024-04-24T08:12:23.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.161+0000 7fb966200640 1 -- 172.21.15.69:0/303345096 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9601592e0 con 0x7fb960162f90 2024-04-24T08:12:23.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.164+0000 7fb95cc00640 1 -- 172.21.15.69:0/303345096 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb95801e460 con 0x7fb960162f90 2024-04-24T08:12:23.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.165+0000 7fb95cc00640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb93c06c920 0x7fb93c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.165+0000 7fb95cc00640 1 -- 172.21.15.69:0/303345096 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fb95808c130 con 0x7fb960162f90 2024-04-24T08:12:23.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.165+0000 7fb964e00640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb93c06c920 0x7fb93c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.165+0000 7fb96a605740 1 Processor -- start 2024-04-24T08:12:23.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.166+0000 7fb96a605740 1 -- start start 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb96a605740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c659e00 0x55ee5c60c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb96a605740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c63c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb96a605740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63c7e0 0x55ee5c6115f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb96a605740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c5761f0 con 0x55ee5c63cf20 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb96a605740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c51c350 con 0x55ee5c63c7e0 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb96a605740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ee5c5721b0 con 0x55ee5c659e00 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb964e00640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb93c06c920 0x7fb93c06ede0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb950001580 tx=0x7fb950009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb95fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c63c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb95fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c63c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44420/0 (socket says 172.21.15.69:44420) 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb95fe00640 1 -- 172.21.15.69:0/3243829446 learned_addr learned my addr 172.21.15.69:0/3243829446 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:23.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb964e00640 1 --2- 172.21.15.69:0/3243829446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c659e00 0x55ee5c60c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb95fe00640 1 -- 172.21.15.69:0/3243829446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c659e00 msgr2=0x55ee5c60c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:23.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb95fe00640 1 --2- 172.21.15.69:0/3243829446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c659e00 0x55ee5c60c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb95fe00640 1 -- 172.21.15.69:0/3243829446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63c7e0 msgr2=0x55ee5c6115f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:23.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb95fe00640 1 --2- 172.21.15.69:0/3243829446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63c7e0 0x55ee5c6115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.167+0000 7fb95fe00640 1 -- 172.21.15.69:0/3243829446 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5c570f50 con 0x55ee5c63cf20 2024-04-24T08:12:23.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.168+0000 7fb95fe00640 1 --2- 172.21.15.69:0/3243829446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c63c2a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb9480098c0 tx=0x7fb94802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef110dfb5519848f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:23.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.168+0000 7fb955600640 1 -- 172.21.15.69:0/3243829446 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb948037040 con 0x55ee5c63cf20 2024-04-24T08:12:23.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.168+0000 7fb955600640 1 -- 172.21.15.69:0/3243829446 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb948042070 con 0x55ee5c63cf20 2024-04-24T08:12:23.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.168+0000 7fb955600640 1 -- 172.21.15.69:0/3243829446 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb94803c860 con 0x55ee5c63cf20 2024-04-24T08:12:23.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.169+0000 7fb96a605740 1 -- 172.21.15.69:0/3243829446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 msgr2=0x55ee5c63c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:23.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.169+0000 7fb96a605740 1 --2- 172.21.15.69:0/3243829446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c63c2a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fb9480098c0 tx=0x7fb94802c000 comp rx=0 tx=0).stop 2024-04-24T08:12:23.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.169+0000 7fb96a605740 1 -- 172.21.15.69:0/3243829446 shutdown_connections 2024-04-24T08:12:23.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.169+0000 7fb96a605740 1 --2- 172.21.15.69:0/3243829446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c63c2a0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.169+0000 7fb96a605740 1 --2- 172.21.15.69:0/3243829446 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63c7e0 0x55ee5c6115f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.169+0000 7fb96a605740 1 --2- 172.21.15.69:0/3243829446 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c659e00 0x55ee5c60c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.169+0000 7fb96a605740 1 -- 172.21.15.69:0/3243829446 >> 172.21.15.69:0/3243829446 conn(0x55ee5c60ea00 msgr2=0x55ee5c6572a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:23.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.169+0000 7fb96a605740 1 -- 172.21.15.69:0/3243829446 shutdown_connections 2024-04-24T08:12:23.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.169+0000 7fb96a605740 1 -- 172.21.15.69:0/3243829446 wait complete. 2024-04-24T08:12:23.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.170+0000 7fb96a605740 1 -- start start 2024-04-24T08:12:23.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.170+0000 7fb96a605740 1 Processor -- start 2024-04-24T08:12:23.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.171+0000 7fb96a605740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63cf20 0x55ee5c63bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.171+0000 7fb96a605740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6588b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.171+0000 7fb96a605740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c653c90 0x55ee5c658df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.171+0000 7fb96a605740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c610bb0 con 0x55ee5c659e00 2024-04-24T08:12:23.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.171+0000 7fb96a605740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c52f690 con 0x55ee5c63cf20 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.171+0000 7fb96a605740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ee5c60ffa0 con 0x55ee5c653c90 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.171+0000 7fb96a605740 10 client.0 _ll_register_callbacks cb 0x55ee5c56b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.171+0000 7fb95fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.171+0000 7fb95fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44432/0 (socket says 172.21.15.69:44432) 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.171+0000 7fb95fe00640 1 -- 172.21.15.69:0/861847226 learned_addr learned my addr 172.21.15.69:0/861847226 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb964e00640 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63cf20 0x55ee5c63bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb95fe00640 1 -- 172.21.15.69:0/861847226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c653c90 msgr2=0x55ee5c658df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb95fe00640 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c653c90 0x55ee5c658df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb95fe00640 1 -- 172.21.15.69:0/861847226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63cf20 msgr2=0x55ee5c63bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb95fe00640 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63cf20 0x55ee5c63bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb95fe00640 1 -- 172.21.15.69:0/861847226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5c52e920 con 0x55ee5c659e00 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb964e00640 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63cf20 0x55ee5c63bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb95fe00640 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6588b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb94802a800 tx=0x7fb948004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f9e5342485fccde server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb94e000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:23.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb94e000640 1 -- 172.21.15.69:0/861847226 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb948037040 con 0x55ee5c659e00 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb94e000640 1 -- 172.21.15.69:0/861847226 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fb948003a20 con 0x55ee5c659e00 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.172+0000 7fb96a605740 1 -- 172.21.15.69:0/861847226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb948042070 con 0x55ee5c659e00 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb94e000640 1 -- 172.21.15.69:0/861847226 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb948056050 con 0x55ee5c659e00 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb96a605740 10 client.6528 fetch_fsmap learned FSMap version 34 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb96a605740 1 -- 172.21.15.69:0/861847226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x55ee5c3527d0 con 0x55ee5c659e00 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb94e000640 1 -- 172.21.15.69:0/861847226 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb948042070 con 0x55ee5c659e00 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb94e000640 1 -- 172.21.15.69:0/861847226 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7fb948059020 con 0x55ee5c659e00 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb96a605740 10 client.6528 fetch_fsmap finished waiting for FSMap version 34 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb96a605740 10 client.6528 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb96a605740 20 client.6528 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb96a605740 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55ee5c66ae00 0x55ee5c66b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb96a605740 4 client.6528 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb96a605740 1 -- 172.21.15.69:0/861847226 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee5c583c10 con 0x55ee5c66ae00 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.173+0000 7fb964e00640 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55ee5c66ae00 0x55ee5c66b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.174+0000 7fb964e00640 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55ee5c66ae00 0x55ee5c66b230 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fad4c105606dd9a server_cookie=55c31e1b22cfe35f in_seq=0 out_seq=0 2024-04-24T08:12:23.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.174+0000 7fb94e000640 10 client.6528 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb94e000640 1 -- 172.21.15.69:0/861847226 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee5c583c10 con 0x55ee5c66ae00 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb94e000640 10 client.6528 handle_command_reply: tid=0 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 client.6528 shutdown 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 -- 172.21.15.69:0/861847226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 msgr2=0x55ee5c6588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6588b0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fb94802a800 tx=0x7fb948004730 comp rx=0 tx=0).stop 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 -- 172.21.15.69:0/861847226 shutdown_connections 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55ee5c66ae00 0x55ee5c66b230 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6588b0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c653c90 0x55ee5c658df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 --2- 172.21.15.69:0/861847226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63cf20 0x55ee5c63bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb94e000640 0 client.6528 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 -- 172.21.15.69:0/861847226 >> 172.21.15.69:0/861847226 conn(0x55ee5c63c7e0 msgr2=0x55ee5c657250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 -- 172.21.15.69:0/861847226 shutdown_connections 2024-04-24T08:12:23.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 1 -- 172.21.15.69:0/861847226 wait complete. 2024-04-24T08:12:23.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.175+0000 7fb96a605740 20 client.6528 trim_cache size 0 max 16384 2024-04-24T08:12:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.195+0000 7fb96a605740 1 Processor -- start 2024-04-24T08:12:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.195+0000 7fb96a605740 1 -- start start 2024-04-24T08:12:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.195+0000 7fb96a605740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63cf20 0x55ee5c656510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.195+0000 7fb96a605740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c653c90 0x55ee5c611cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb96a605740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6121f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb96a605740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ee5c64e6e0 con 0x55ee5c659e00 2024-04-24T08:12:23.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb96a605740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb960200b20 con 0x55ee5c63cf20 2024-04-24T08:12:23.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb96a605740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb9600c93e0 con 0x55ee5c653c90 2024-04-24T08:12:23.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb965800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6121f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb965800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6121f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44438/0 (socket says 172.21.15.69:44438) 2024-04-24T08:12:23.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb965800640 1 -- 172.21.15.69:0/3329771724 learned_addr learned my addr 172.21.15.69:0/3329771724 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:23.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb965800640 1 -- 172.21.15.69:0/3329771724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c653c90 msgr2=0x55ee5c611cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:23.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb965800640 1 --2- 172.21.15.69:0/3329771724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c653c90 0x55ee5c611cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb965800640 1 -- 172.21.15.69:0/3329771724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63cf20 msgr2=0x55ee5c656510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb965800640 1 --2- 172.21.15.69:0/3329771724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63cf20 0x55ee5c656510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.196+0000 7fb965800640 1 -- 172.21.15.69:0/3329771724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee5c2883f0 con 0x55ee5c659e00 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb965800640 1 --2- 172.21.15.69:0/3329771724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6121f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fb95800d800 tx=0x7fb95800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2715728345ccce41 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb954c00640 1 -- 172.21.15.69:0/3329771724 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb958072020 con 0x55ee5c659e00 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb954c00640 1 -- 172.21.15.69:0/3329771724 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb958059ef0 con 0x55ee5c659e00 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb954c00640 1 -- 172.21.15.69:0/3329771724 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95805e2a0 con 0x55ee5c659e00 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb96a605740 1 -- 172.21.15.69:0/3329771724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 msgr2=0x55ee5c6121f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb96a605740 1 --2- 172.21.15.69:0/3329771724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6121f0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fb95800d800 tx=0x7fb95800be70 comp rx=0 tx=0).stop 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb96a605740 1 -- 172.21.15.69:0/3329771724 shutdown_connections 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb96a605740 1 --2- 172.21.15.69:0/3329771724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c659e00 0x55ee5c6121f0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb96a605740 1 --2- 172.21.15.69:0/3329771724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c653c90 0x55ee5c611cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb96a605740 1 --2- 172.21.15.69:0/3329771724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c63cf20 0x55ee5c656510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb96a605740 1 -- 172.21.15.69:0/3329771724 >> 172.21.15.69:0/3329771724 conn(0x55ee5c63c7e0 msgr2=0x55ee5c63e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:23.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.197+0000 7fb96a605740 1 -- 172.21.15.69:0/3329771724 shutdown_connections 2024-04-24T08:12:23.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.198+0000 7fb96a605740 1 -- 172.21.15.69:0/3329771724 wait complete. 2024-04-24T08:12:23.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.198+0000 7fb96a605740 1 -- start start 2024-04-24T08:12:23.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.198+0000 7fb96a605740 1 Processor -- start 2024-04-24T08:12:23.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.199+0000 7fb96a605740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c60c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.199+0000 7fb96a605740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c653c90 0x55ee5c6d38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.199+0000 7fb96a605740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c659e00 0x55ee5c669a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.199+0000 7fb96a605740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb9600c8d90 con 0x55ee5c63cf20 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.199+0000 7fb96a605740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb9600c8740 con 0x55ee5c653c90 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.199+0000 7fb96a605740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ee5c636180 con 0x55ee5c659e00 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.199+0000 7fb96a605740 10 client.0 _ll_register_callbacks cb 0x7fb9440036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb964e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c60c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb964e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c60c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44442/0 (socket says 172.21.15.69:44442) 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb964e00640 1 -- 172.21.15.69:0/2275675534 learned_addr learned my addr 172.21.15.69:0/2275675534 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb964e00640 1 -- 172.21.15.69:0/2275675534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c659e00 msgr2=0x55ee5c669a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb964e00640 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c659e00 0x55ee5c669a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb964e00640 1 -- 172.21.15.69:0/2275675534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c653c90 msgr2=0x55ee5c6d38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb964e00640 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c653c90 0x55ee5c6d38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb964e00640 1 -- 172.21.15.69:0/2275675534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee5c2480f0 con 0x55ee5c63cf20 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb964e00640 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c60c010 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x55ee5c63e5a0 tx=0x7fb950000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9aed497638f99b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb94e000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb94e000640 1 -- 172.21.15.69:0/2275675534 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb950007a70 con 0x55ee5c63cf20 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.200+0000 7fb94e000640 1 -- 172.21.15.69:0/2275675534 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fb95001e040 con 0x55ee5c63cf20 2024-04-24T08:12:23.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb96a605740 1 -- 172.21.15.69:0/2275675534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb958059ef0 con 0x55ee5c63cf20 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb94e000640 1 -- 172.21.15.69:0/2275675534 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fb950033050 con 0x55ee5c63cf20 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb96a605740 10 client.6534 fetch_fsmap learned FSMap version 34 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb96a605740 1 -- 172.21.15.69:0/2275675534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=2+}) -- 0x55ee5c53b810 con 0x55ee5c63cf20 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb94e000640 1 -- 172.21.15.69:0/2275675534 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb958059ef0 con 0x55ee5c63cf20 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb94e000640 1 -- 172.21.15.69:0/2275675534 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 34) ==== 3494+0+0 (secure 0 0 0) 0x7fb950036020 con 0x55ee5c63cf20 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb96a605740 10 client.6534 fetch_fsmap finished waiting for FSMap version 34 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb96a605740 10 client.6534 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb96a605740 20 client.6534 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb96a605740 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55ee5c669fc0 0x55ee5c6d41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb96a605740 4 client.6534 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb96a605740 1 -- 172.21.15.69:0/2275675534 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ee5c57da80 con 0x55ee5c669fc0 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.201+0000 7fb95fe00640 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55ee5c669fc0 0x55ee5c6d41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb95fe00640 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55ee5c669fc0 0x55ee5c6d41a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=366181519d5cd123 server_cookie=9fef5d30b7e372c3 in_seq=0 out_seq=0 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb94e000640 10 client.6534 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:23.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb94e000640 1 -- 172.21.15.69:0/2275675534 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x55ee5c57da80 con 0x55ee5c669fc0 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb94e000640 10 client.6534 handle_command_reply: tid=0 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb96a605740 1 client.6534 shutdown 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb96a605740 1 -- 172.21.15.69:0/2275675534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 msgr2=0x55ee5c60c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb96a605740 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c60c010 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x55ee5c63e5a0 tx=0x7fb950000f40 comp rx=0 tx=0).stop 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb96a605740 1 -- 172.21.15.69:0/2275675534 shutdown_connections 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb96a605740 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55ee5c669fc0 0x55ee5c6d41a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb96a605740 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ee5c659e00 0x55ee5c669a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb94e000640 0 client.6534 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb96a605740 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ee5c653c90 0x55ee5c6d38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb96a605740 1 --2- 172.21.15.69:0/2275675534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee5c63cf20 0x55ee5c60c010 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.202+0000 7fb96a605740 1 -- 172.21.15.69:0/2275675534 >> 172.21.15.69:0/2275675534 conn(0x55ee5c63c7e0 msgr2=0x55ee5c6121c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb96a605740 1 -- 172.21.15.69:0/2275675534 shutdown_connections 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb96a605740 1 -- 172.21.15.69:0/2275675534 wait complete. 2024-04-24T08:12:23.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb96a605740 20 client.6534 trim_cache size 0 max 16384 2024-04-24T08:12:23.208 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:12:23.208 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:12:23.208 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-04-24T08:12:23.208 INFO:teuthology.orchestra.run.smithi069.stdout: "65ed9faa-f7d1-4316-bca0-9dc10817c488": { 2024-04-24T08:12:23.208 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/", 2024-04-24T08:12:23.208 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "65ed9faa-f7d1-4316-bca0-9dc10817c488", 2024-04-24T08:12:23.208 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,repair,force" 2024-04-24T08:12:23.208 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:12:23.208 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:12:23.208 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:12:23.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb966200640 1 -- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb93c06c920 msgr2=0x7fb93c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:23.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb966200640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb93c06c920 0x7fb93c06ede0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb950001580 tx=0x7fb950009040 comp rx=0 tx=0).stop 2024-04-24T08:12:23.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb966200640 1 -- 172.21.15.69:0/303345096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 msgr2=0x7fb9600d0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:23.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb966200640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 0x7fb9600d0600 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb95800d830 tx=0x7fb95800dcf0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb966200640 1 -- 172.21.15.69:0/303345096 shutdown_connections 2024-04-24T08:12:23.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb966200640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb93c06c920 0x7fb93c06ede0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb966200640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb960162f90 0x7fb9600d0600 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb966200640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb96015e980 0x7fb960158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.203+0000 7fb966200640 1 --2- 172.21.15.69:0/303345096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb9600d0190 0x7fb9601f4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:23.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.204+0000 7fb966200640 1 -- 172.21.15.69:0/303345096 >> 172.21.15.69:0/303345096 conn(0x7fb96009d920 msgr2=0x7fb9600b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:23.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.204+0000 7fb966200640 1 -- 172.21.15.69:0/303345096 shutdown_connections 2024-04-24T08:12:23.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:23.204+0000 7fb966200640 1 -- 172.21.15.69:0/303345096 wait complete. 2024-04-24T08:12:23.221 INFO:tasks.cephfs.filesystem:scrub status for tag:65ed9faa-f7d1-4316-bca0-9dc10817c488 - {'path': '/', 'tag': '65ed9faa-f7d1-4316-bca0-9dc10817c488', 'options': 'recursive,repair,force'} 2024-04-24T08:12:44.508 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:12:44.516 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:12:53.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:53.222 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.4:0 scrub status 2024-04-24T08:12:53.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.432+0000 7f7249800640 1 Processor -- start 2024-04-24T08:12:53.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.432+0000 7f7249800640 1 -- start start 2024-04-24T08:12:53.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.433+0000 7f7249800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f724415f4b0 0x7f72440b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.433+0000 7f7249800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f724415f890 0x7f72440732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.433+0000 7f7249800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7244073820 0x7f72440b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.433+0000 7f7249800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7244073cc0 con 0x7f724415f890 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.433+0000 7f7249800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7244073fa0 con 0x7f7244073820 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.433+0000 7f7249800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72440b3300 con 0x7f724415f4b0 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f723a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f724415f890 0x7f72440732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f723a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f724415f890 0x7f72440732e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47294/0 (socket says 172.21.15.69:47294) 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f723a000640 1 -- 172.21.15.69:0/2314498728 learned_addr learned my addr 172.21.15.69:0/2314498728 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f7243400640 1 --2- 172.21.15.69:0/2314498728 >> v2:172.21.15.121:3300/0 conn(0x7f7244073820 0x7f72440b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f723a000640 1 -- 172.21.15.69:0/2314498728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f724415f4b0 msgr2=0x7f72440b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f7242a00640 1 --2- 172.21.15.69:0/2314498728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f724415f4b0 0x7f72440b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f723a000640 1 --2- 172.21.15.69:0/2314498728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f724415f4b0 0x7f72440b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f723a000640 1 -- 172.21.15.69:0/2314498728 >> v2:172.21.15.121:3300/0 conn(0x7f7244073820 msgr2=0x7f72440b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f723a000640 1 --2- 172.21.15.69:0/2314498728 >> v2:172.21.15.121:3300/0 conn(0x7f7244073820 0x7f72440b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f723a000640 1 -- 172.21.15.69:0/2314498728 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72440b35e0 con 0x7f724415f890 2024-04-24T08:12:53.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.434+0000 7f7243400640 1 --2- 172.21.15.69:0/2314498728 >> v2:172.21.15.121:3300/0 conn(0x7f7244073820 0x7f72440b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:53.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.435+0000 7f723a000640 1 --2- 172.21.15.69:0/2314498728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 0x7f72440732e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f72440cdd90 tx=0x7f722c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67b84bf54a5dac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:53.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.435+0000 7f7242000640 1 -- 172.21.15.69:0/2314498728 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c036070 con 0x7f724415f890 2024-04-24T08:12:53.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7242000640 1 -- 172.21.15.69:0/2314498728 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f722c031600 con 0x7f724415f890 2024-04-24T08:12:53.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7242000640 1 -- 172.21.15.69:0/2314498728 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c0319d0 con 0x7f724415f890 2024-04-24T08:12:53.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7249800640 1 -- 172.21.15.69:0/2314498728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 msgr2=0x7f72440732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7249800640 1 --2- 172.21.15.69:0/2314498728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 0x7f72440732e0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f72440cdd90 tx=0x7f722c02a6a0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7249800640 1 -- 172.21.15.69:0/2314498728 shutdown_connections 2024-04-24T08:12:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7249800640 1 --2- 172.21.15.69:0/2314498728 >> v2:172.21.15.121:3300/0 conn(0x7f7244073820 0x7f72440b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7249800640 1 --2- 172.21.15.69:0/2314498728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 0x7f72440732e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7249800640 1 --2- 172.21.15.69:0/2314498728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f724415f4b0 0x7f72440b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7249800640 1 -- 172.21.15.69:0/2314498728 >> 172.21.15.69:0/2314498728 conn(0x7f72440d0350 msgr2=0x7f72440d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7249800640 1 -- 172.21.15.69:0/2314498728 shutdown_connections 2024-04-24T08:12:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.436+0000 7f7249800640 1 -- 172.21.15.69:0/2314498728 wait complete. 2024-04-24T08:12:53.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.438+0000 7f7249800640 1 Processor -- start 2024-04-24T08:12:53.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.438+0000 7f7249800640 1 -- start start 2024-04-24T08:12:53.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.439+0000 7f7249800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7244073820 0x7f72440af830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.439+0000 7f7249800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f724415f4b0 0x7f724414c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.439+0000 7f7249800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 0x7f72440cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.439+0000 7f7249800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f724415fe90 con 0x7f724415f890 2024-04-24T08:12:53.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.439+0000 7f7249800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f724415fd10 con 0x7f7244073820 2024-04-24T08:12:53.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.439+0000 7f7249800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72440b4be0 con 0x7f724415f4b0 2024-04-24T08:12:53.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.439+0000 7f7243400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 0x7f72440cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.439+0000 7f7243400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 0x7f72440cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47310/0 (socket says 172.21.15.69:47310) 2024-04-24T08:12:53.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.439+0000 7f7243400640 1 -- 172.21.15.69:0/2197242361 learned_addr learned my addr 172.21.15.69:0/2197242361 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:53.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f7242a00640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7244073820 0x7f72440af830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f723a000640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f724415f4b0 0x7f724414c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f7243400640 1 -- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f724415f4b0 msgr2=0x7f724414c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f7243400640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f724415f4b0 0x7f724414c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f7243400640 1 -- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7244073820 msgr2=0x7f72440af830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f7243400640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7244073820 0x7f72440af830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f7243400640 1 -- 172.21.15.69:0/2197242361 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72440adcf0 con 0x7f724415f890 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f7243400640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 0x7f72440cd0f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f723c00c7f0 tx=0x7f723c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f723b400640 1 -- 172.21.15.69:0/2197242361 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c015020 con 0x7f724415f890 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f723b400640 1 -- 172.21.15.69:0/2197242361 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f723c004230 con 0x7f724415f890 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f723b400640 1 -- 172.21.15.69:0/2197242361 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c007620 con 0x7f724415f890 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.440+0000 7f7249800640 1 -- 172.21.15.69:0/2197242361 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72440adfd0 con 0x7f724415f890 2024-04-24T08:12:53.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.441+0000 7f7249800640 1 -- 172.21.15.69:0/2197242361 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72440ae590 con 0x7f724415f890 2024-04-24T08:12:53.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.443+0000 7f723b400640 1 -- 172.21.15.69:0/2197242361 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f723c018020 con 0x7f724415f890 2024-04-24T08:12:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.444+0000 7f723b400640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f722006c8f0 0x7f722006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.444+0000 7f723b400640 1 -- 172.21.15.69:0/2197242361 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f723c08bea0 con 0x7f724415f890 2024-04-24T08:12:53.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.444+0000 7f7242a00640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f722006c8f0 0x7f722006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.444+0000 7f724c76d740 1 Processor -- start 2024-04-24T08:12:53.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.445+0000 7f724c76d740 1 -- start start 2024-04-24T08:12:53.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.445+0000 7f724c76d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x56063911a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.445+0000 7f724c76d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56063914ad50 0x56063914a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f724c76d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914a610 0x56063911f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f724c76d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5606390841f0 con 0x560639167c30 2024-04-24T08:12:53.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f724c76d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56063902a350 con 0x56063914a610 2024-04-24T08:12:53.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f724c76d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5606390801b0 con 0x56063914ad50 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7242a00640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f722006c8f0 0x7f722006edb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f723400a8f0 tx=0x7f7234008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7242a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x56063911a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7242a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x56063911a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47324/0 (socket says 172.21.15.69:47324) 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7242a00640 1 -- 172.21.15.69:0/1078919145 learned_addr learned my addr 172.21.15.69:0/1078919145 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7242a00640 1 -- 172.21.15.69:0/1078919145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56063914ad50 msgr2=0x56063914a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7242a00640 1 --2- 172.21.15.69:0/1078919145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56063914ad50 0x56063914a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7242a00640 1 -- 172.21.15.69:0/1078919145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914a610 msgr2=0x56063911f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7243400640 1 --2- 172.21.15.69:0/1078919145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914a610 0x56063911f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7242a00640 1 --2- 172.21.15.69:0/1078919145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914a610 0x56063911f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7242a00640 1 -- 172.21.15.69:0/1078919145 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56063907ef50 con 0x560639167c30 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.446+0000 7f7243400640 1 --2- 172.21.15.69:0/1078919145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914a610 0x56063911f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.447+0000 7f7242a00640 1 --2- 172.21.15.69:0/1078919145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x56063911a5c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f723400eba0 tx=0x7f7234011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83aa8e36561865b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.447+0000 7f7233400640 1 -- 172.21.15.69:0/1078919145 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723401a050 con 0x560639167c30 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.447+0000 7f7233400640 1 -- 172.21.15.69:0/1078919145 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7234024070 con 0x560639167c30 2024-04-24T08:12:53.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.447+0000 7f7233400640 1 -- 172.21.15.69:0/1078919145 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723401f440 con 0x560639167c30 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.448+0000 7f724c76d740 1 -- 172.21.15.69:0/1078919145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 msgr2=0x56063911a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.448+0000 7f724c76d740 1 --2- 172.21.15.69:0/1078919145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x56063911a5c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f723400eba0 tx=0x7f7234011000 comp rx=0 tx=0).stop 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.448+0000 7f724c76d740 1 -- 172.21.15.69:0/1078919145 shutdown_connections 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.448+0000 7f724c76d740 1 --2- 172.21.15.69:0/1078919145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914a610 0x56063911f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.448+0000 7f724c76d740 1 --2- 172.21.15.69:0/1078919145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56063914ad50 0x56063914a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.448+0000 7f724c76d740 1 --2- 172.21.15.69:0/1078919145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x56063911a5c0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.448+0000 7f724c76d740 1 -- 172.21.15.69:0/1078919145 >> 172.21.15.69:0/1078919145 conn(0x56063911c830 msgr2=0x5606391650d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.448+0000 7f724c76d740 1 -- 172.21.15.69:0/1078919145 shutdown_connections 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.448+0000 7f724c76d740 1 -- 172.21.15.69:0/1078919145 wait complete. 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.449+0000 7f724c76d740 1 -- start start 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.449+0000 7f724c76d740 1 Processor -- start 2024-04-24T08:12:53.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f724c76d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 0x560639166600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f724c76d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x5606391785a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f724c76d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560639178ae0 0x5606391374b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f724c76d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56063911e9e0 con 0x560639167c30 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f724c76d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56063903d690 con 0x56063914ad50 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f724c76d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56063911ddd0 con 0x560639178ae0 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f724c76d740 10 client.0 _ll_register_callbacks cb 0x560639079650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f723a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x5606391785a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f723a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x5606391785a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47328/0 (socket says 172.21.15.69:47328) 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f723a000640 1 -- 172.21.15.69:0/863477766 learned_addr learned my addr 172.21.15.69:0/863477766 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.450+0000 7f7242a00640 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 0x560639166600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f723a000640 1 -- 172.21.15.69:0/863477766 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560639178ae0 msgr2=0x5606391374b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f723a000640 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560639178ae0 0x5606391374b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f723a000640 1 -- 172.21.15.69:0/863477766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 msgr2=0x560639166600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f723a000640 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 0x560639166600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f723a000640 1 -- 172.21.15.69:0/863477766 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56063903c920 con 0x560639167c30 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f7242a00640 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 0x560639166600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f723a000640 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x5606391785a0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f722c02ac90 tx=0x7f722c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca16b45c090f4fa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f7227e00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f7227e00640 1 -- 172.21.15.69:0/863477766 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c041020 con 0x560639167c30 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f7227e00640 1 -- 172.21.15.69:0/863477766 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f722c031de0 con 0x560639167c30 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f724c76d740 1 -- 172.21.15.69:0/863477766 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7234024070 con 0x560639167c30 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f7227e00640 1 -- 172.21.15.69:0/863477766 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f722c036040 con 0x560639167c30 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f724c76d740 10 client.6546 fetch_fsmap learned FSMap version 35 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.451+0000 7f724c76d740 1 -- 172.21.15.69:0/863477766 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x560638e607d0 con 0x560639167c30 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.452+0000 7f7227e00640 1 -- 172.21.15.69:0/863477766 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7234024070 con 0x560639167c30 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.452+0000 7f7227e00640 1 -- 172.21.15.69:0/863477766 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f722c04e020 con 0x560639167c30 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.452+0000 7f724c76d740 10 client.6546 fetch_fsmap finished waiting for FSMap version 35 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.452+0000 7f724c76d740 10 client.6546 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:12:53.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.452+0000 7f724c76d740 20 client.6546 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:53.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.452+0000 7f724c76d740 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x56063914b710 0x56063914bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.452+0000 7f724c76d740 4 client.6546 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:53.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.452+0000 7f724c76d740 1 -- 172.21.15.69:0/863477766 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560639091c10 con 0x56063914b710 2024-04-24T08:12:53.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.452+0000 7f7242a00640 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x56063914b710 0x56063914bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.453+0000 7f7242a00640 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x56063914b710 0x56063914bb30 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c48b1bbb7f6a50cb server_cookie=6de5cd955ca2a6b0 in_seq=0 out_seq=0 2024-04-24T08:12:53.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.453+0000 7f7227e00640 10 client.6546 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:53.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.453+0000 7f7227e00640 1 -- 172.21.15.69:0/863477766 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560639091c10 con 0x56063914b710 2024-04-24T08:12:53.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f7227e00640 10 client.6546 handle_command_reply: tid=0 2024-04-24T08:12:53.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 client.6546 shutdown 2024-04-24T08:12:53.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 -- 172.21.15.69:0/863477766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 msgr2=0x5606391785a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x5606391785a0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f722c02ac90 tx=0x7f722c004a40 comp rx=0 tx=0).stop 2024-04-24T08:12:53.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 -- 172.21.15.69:0/863477766 shutdown_connections 2024-04-24T08:12:53.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x56063914b710 0x56063914bb30 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x5606391785a0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560639178ae0 0x5606391374b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 --2- 172.21.15.69:0/863477766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 0x560639166600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f7227e00640 0 client.6546 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:53.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 -- 172.21.15.69:0/863477766 >> 172.21.15.69:0/863477766 conn(0x56063914a610 msgr2=0x56063911b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:53.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 -- 172.21.15.69:0/863477766 shutdown_connections 2024-04-24T08:12:53.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 1 -- 172.21.15.69:0/863477766 wait complete. 2024-04-24T08:12:53.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.454+0000 7f724c76d740 20 client.6546 trim_cache size 0 max 16384 2024-04-24T08:12:53.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.473+0000 7f724c76d740 1 Processor -- start 2024-04-24T08:12:53.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.473+0000 7f724c76d740 1 -- start start 2024-04-24T08:12:53.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.474+0000 7f724c76d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56063914ad50 0x5606391475b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.474+0000 7f724c76d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560639167c30 0x56063915eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.474+0000 7f724c76d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639178ae0 0x5606391779a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.474+0000 7f724c76d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7244200dc0 con 0x560639178ae0 2024-04-24T08:12:53.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.474+0000 7f724c76d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f72441ffe60 con 0x560639167c30 2024-04-24T08:12:53.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.474+0000 7f724c76d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72441f88a0 con 0x56063914ad50 2024-04-24T08:12:53.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.474+0000 7f7243400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639178ae0 0x5606391779a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.474+0000 7f7243400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639178ae0 0x5606391779a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47330/0 (socket says 172.21.15.69:47330) 2024-04-24T08:12:53.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f7243400640 1 -- 172.21.15.69:0/2165099454 learned_addr learned my addr 172.21.15.69:0/2165099454 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:53.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f7243400640 1 -- 172.21.15.69:0/2165099454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56063914ad50 msgr2=0x5606391475b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f723a000640 1 --2- 172.21.15.69:0/2165099454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560639167c30 0x56063915eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f7243400640 1 --2- 172.21.15.69:0/2165099454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56063914ad50 0x5606391475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f7243400640 1 -- 172.21.15.69:0/2165099454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560639167c30 msgr2=0x56063915eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f7243400640 1 --2- 172.21.15.69:0/2165099454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560639167c30 0x56063915eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f7243400640 1 -- 172.21.15.69:0/2165099454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560638d963f0 con 0x560639178ae0 2024-04-24T08:12:53.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f7243400640 1 --2- 172.21.15.69:0/2165099454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639178ae0 0x5606391779a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f723c0622a0 tx=0x7f723c05a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fbf07d1ae53e73 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:53.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f7232a00640 1 -- 172.21.15.69:0/2165099454 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c010e00 con 0x560639178ae0 2024-04-24T08:12:53.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f7232a00640 1 -- 172.21.15.69:0/2165099454 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f723c088070 con 0x560639178ae0 2024-04-24T08:12:53.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.475+0000 7f7232a00640 1 -- 172.21.15.69:0/2165099454 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f723c056260 con 0x560639178ae0 2024-04-24T08:12:53.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.476+0000 7f724c76d740 1 -- 172.21.15.69:0/2165099454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639178ae0 msgr2=0x5606391779a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.476+0000 7f724c76d740 1 --2- 172.21.15.69:0/2165099454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639178ae0 0x5606391779a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f723c0622a0 tx=0x7f723c05a430 comp rx=0 tx=0).stop 2024-04-24T08:12:53.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.476+0000 7f724c76d740 1 -- 172.21.15.69:0/2165099454 shutdown_connections 2024-04-24T08:12:53.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.476+0000 7f724c76d740 1 --2- 172.21.15.69:0/2165099454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639178ae0 0x5606391779a0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.476+0000 7f724c76d740 1 --2- 172.21.15.69:0/2165099454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560639167c30 0x56063915eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.476+0000 7f724c76d740 1 --2- 172.21.15.69:0/2165099454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56063914ad50 0x5606391475b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.476+0000 7f724c76d740 1 -- 172.21.15.69:0/2165099454 >> 172.21.15.69:0/2165099454 conn(0x56063914a610 msgr2=0x560639145310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:53.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.476+0000 7f724c76d740 1 -- 172.21.15.69:0/2165099454 shutdown_connections 2024-04-24T08:12:53.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.476+0000 7f724c76d740 1 -- 172.21.15.69:0/2165099454 wait complete. 2024-04-24T08:12:53.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.477+0000 7f724c76d740 1 -- start start 2024-04-24T08:12:53.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.477+0000 7f724c76d740 1 Processor -- start 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.478+0000 7f724c76d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 0x560639135f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.478+0000 7f724c76d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x560639136480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.478+0000 7f724c76d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560639178ae0 0x56063914d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.478+0000 7f724c76d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560639188e60 con 0x560639167c30 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.478+0000 7f724c76d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7244147410 con 0x56063914ad50 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.478+0000 7f724c76d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x560639163500 con 0x560639178ae0 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.478+0000 7f724c76d740 10 client.0 _ll_register_callbacks cb 0x7f722c034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.478+0000 7f723a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x560639136480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.478+0000 7f723a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x560639136480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47340/0 (socket says 172.21.15.69:47340) 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.478+0000 7f723a000640 1 -- 172.21.15.69:0/2126714460 learned_addr learned my addr 172.21.15.69:0/2126714460 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f723a000640 1 -- 172.21.15.69:0/2126714460 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560639178ae0 msgr2=0x56063914d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f723a000640 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560639178ae0 0x56063914d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f723a000640 1 -- 172.21.15.69:0/2126714460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 msgr2=0x560639135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f7242a00640 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 0x560639135f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f723a000640 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 0x560639135f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f723a000640 1 -- 172.21.15.69:0/2126714460 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560638d560f0 con 0x560639167c30 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f7242a00640 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 0x560639135f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f723a000640 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x560639136480 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f722c009c30 tx=0x7f722c009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0687fcfd7b7b22 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f7227e00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f7227e00640 1 -- 172.21.15.69:0/2126714460 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f722c054060 con 0x560639167c30 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f7227e00640 1 -- 172.21.15.69:0/2126714460 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f722c031bd0 con 0x560639167c30 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.479+0000 7f724c76d740 1 -- 172.21.15.69:0/2126714460 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f723c088070 con 0x560639167c30 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f7227e00640 1 -- 172.21.15.69:0/2126714460 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f722c05a050 con 0x560639167c30 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f724c76d740 10 client.6552 fetch_fsmap learned FSMap version 35 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f724c76d740 1 -- 172.21.15.69:0/2126714460 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x560639049810 con 0x560639167c30 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f7227e00640 1 -- 172.21.15.69:0/2126714460 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f723c088070 con 0x560639167c30 2024-04-24T08:12:53.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f7227e00640 1 -- 172.21.15.69:0/2126714460 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7f722c05d020 con 0x560639167c30 2024-04-24T08:12:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f724c76d740 10 client.6552 fetch_fsmap finished waiting for FSMap version 35 2024-04-24T08:12:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f724c76d740 10 client.6552 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-24T08:12:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f724c76d740 20 client.6552 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f724c76d740 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606391debe0 0x56063914d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f724c76d740 4 client.6552 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:12:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f724c76d740 1 -- 172.21.15.69:0/2126714460 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56063908ba80 con 0x5606391debe0 2024-04-24T08:12:53.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.480+0000 7f7242a00640 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606391debe0 0x56063914d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f7242a00640 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606391debe0 0x56063914d590 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abadb94286075203 server_cookie=d89f72f6b5ef0f17 in_seq=0 out_seq=0 2024-04-24T08:12:53.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f7227e00640 10 client.6552 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:53.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f7227e00640 1 -- 172.21.15.69:0/2126714460 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56063908ba80 con 0x5606391debe0 2024-04-24T08:12:53.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f7227e00640 10 client.6552 handle_command_reply: tid=0 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f724c76d740 1 client.6552 shutdown 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f724c76d740 1 -- 172.21.15.69:0/2126714460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 msgr2=0x560639136480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f724c76d740 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x560639136480 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f722c009c30 tx=0x7f722c009000 comp rx=0 tx=0).stop 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f724c76d740 1 -- 172.21.15.69:0/2126714460 shutdown_connections 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f724c76d740 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x5606391debe0 0x56063914d590 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f724c76d740 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560639167c30 0x560639136480 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f724c76d740 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560639178ae0 0x56063914d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f724c76d740 1 --2- 172.21.15.69:0/2126714460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56063914ad50 0x560639135f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f7227e00640 0 client.6552 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.481+0000 7f724c76d740 1 -- 172.21.15.69:0/2126714460 >> 172.21.15.69:0/2126714460 conn(0x56063914a610 msgr2=0x560639119390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f724c76d740 1 -- 172.21.15.69:0/2126714460 shutdown_connections 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f724c76d740 1 -- 172.21.15.69:0/2126714460 wait complete. 2024-04-24T08:12:53.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f724c76d740 20 client.6552 trim_cache size 0 max 16384 2024-04-24T08:12:53.487 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:12:53.487 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-04-24T08:12:53.487 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-04-24T08:12:53.487 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f7249800640 1 -- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f722006c8f0 msgr2=0x7f722006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f7249800640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f722006c8f0 0x7f722006edb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f723400a8f0 tx=0x7f7234008040 comp rx=0 tx=0).stop 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f7249800640 1 -- 172.21.15.69:0/2197242361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 msgr2=0x7f72440cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f7249800640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 0x7f72440cd0f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f723c00c7f0 tx=0x7f723c00ccb0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f7249800640 1 -- 172.21.15.69:0/2197242361 shutdown_connections 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f7249800640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f722006c8f0 0x7f722006edb0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f7249800640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f724415f890 0x7f72440cd0f0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f7249800640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f724415f4b0 0x7f724414c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f7249800640 1 --2- 172.21.15.69:0/2197242361 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7244073820 0x7f72440af830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.482+0000 7f7249800640 1 -- 172.21.15.69:0/2197242361 >> 172.21.15.69:0/2197242361 conn(0x7f72440d0350 msgr2=0x7f724406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.483+0000 7f7249800640 1 -- 172.21.15.69:0/2197242361 shutdown_connections 2024-04-24T08:12:53.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.483+0000 7f7249800640 1 -- 172.21.15.69:0/2197242361 wait complete. 2024-04-24T08:12:53.503 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:12:53.503 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:12:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.692+0000 7f0bd1600640 1 Processor -- start 2024-04-24T08:12:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.692+0000 7f0bd1600640 1 -- start start 2024-04-24T08:12:53.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.694+0000 7f0bd1600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0bcc156c60 0x7f0bcc15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.694+0000 7f0bd1600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bcc15d660 con 0x7f0bcc0b30f0 2024-04-24T08:12:53.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.694+0000 7f0bd1600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bcc06dc30 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.694+0000 7f0bd1600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0bcc06df10 con 0x7f0bcc156c60 2024-04-24T08:12:53.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.694+0000 7f0bcbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0bcc156c60 0x7f0bcc15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.694+0000 7f0bcbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0bcc156c60 0x7f0bcc15cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47352/0 (socket says 172.21.15.69:47352) 2024-04-24T08:12:53.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.694+0000 7f0bcbe00640 1 -- 172.21.15.69:0/1043464910 learned_addr learned my addr 172.21.15.69:0/1043464910 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:53.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.695+0000 7f0bcbe00640 1 -- 172.21.15.69:0/1043464910 >> v1:172.21.15.121:6789/0 conn(0x7f0bcc0ad7e0 legacy=0x7f0bcc06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:53.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.695+0000 7f0bcbe00640 1 -- 172.21.15.69:0/1043464910 >> v1:172.21.15.69:6789/0 conn(0x7f0bcc0b30f0 legacy=0x7f0bcc0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.695+0000 7f0bcbe00640 1 -- 172.21.15.69:0/1043464910 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bcc06e1f0 con 0x7f0bcc156c60 2024-04-24T08:12:53.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.695+0000 7f0bcaa00640 1 -- 172.21.15.69:0/1043464910 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 913162393 0 0) 0x7f0bcc15d660 con 0x7f0bcc0b30f0 2024-04-24T08:12:53.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.696+0000 7f0bcbe00640 1 --2- 172.21.15.69:0/1043464910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc156c60 0x7f0bcc15cac0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc009870 tx=0x7f0bbc02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c170178ec6781199 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:53.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.696+0000 7f0bcaa00640 1 -- 172.21.15.69:0/1043464910 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc037070 con 0x7f0bcc156c60 2024-04-24T08:12:53.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.696+0000 7f0bcaa00640 1 -- 172.21.15.69:0/1043464910 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bbc032440 con 0x7f0bcc156c60 2024-04-24T08:12:53.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.696+0000 7f0bcaa00640 1 -- 172.21.15.69:0/1043464910 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc032810 con 0x7f0bcc156c60 2024-04-24T08:12:53.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.697+0000 7f0bd1600640 1 -- 172.21.15.69:0/1043464910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc156c60 msgr2=0x7f0bcc15cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.697+0000 7f0bd1600640 1 --2- 172.21.15.69:0/1043464910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc156c60 0x7f0bcc15cac0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc009870 tx=0x7f0bbc02a2a0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.697+0000 7f0bd1600640 1 -- 172.21.15.69:0/1043464910 shutdown_connections 2024-04-24T08:12:53.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.697+0000 7f0bd1600640 1 --2- 172.21.15.69:0/1043464910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc156c60 0x7f0bcc15cac0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.697+0000 7f0bd1600640 1 -- 172.21.15.69:0/1043464910 >> 172.21.15.69:0/1043464910 conn(0x7f0bcc0d0160 msgr2=0x7f0bcc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:53.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.697+0000 7f0bd1600640 1 -- 172.21.15.69:0/1043464910 shutdown_connections 2024-04-24T08:12:53.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.697+0000 7f0bd1600640 1 -- 172.21.15.69:0/1043464910 wait complete. 2024-04-24T08:12:53.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.699+0000 7f0bd1600640 1 Processor -- start 2024-04-24T08:12:53.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.699+0000 7f0bd1600640 1 -- start start 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bd1600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc0ad7e0 0x7f0bcc154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bd1600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bcc0b30f0 0x7f0bcc0994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bd1600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bcc156c60 0x7f0bcc160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bd1600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc15d000 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bd1600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc15d940 con 0x7f0bcc156c60 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bd1600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0bcc15d180 con 0x7f0bcc0b30f0 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bcbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc0ad7e0 0x7f0bcc154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bcbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc0ad7e0 0x7f0bcc154110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47364/0 (socket says 172.21.15.69:47364) 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bcbe00640 1 -- 172.21.15.69:0/1496564528 learned_addr learned my addr 172.21.15.69:0/1496564528 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bcbe00640 1 -- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bcc0b30f0 msgr2=0x7f0bcc0994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bcbe00640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bcc0b30f0 0x7f0bcc0994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bcbe00640 1 -- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bcc156c60 msgr2=0x7f0bcc160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bcbe00640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bcc156c60 0x7f0bcc160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.700+0000 7f0bcbe00640 1 -- 172.21.15.69:0/1496564528 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bcc152490 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.701+0000 7f0bcbe00640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc0ad7e0 0x7f0bcc154110 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc009870 tx=0x7f0bbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:53.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.701+0000 7f0bc3e00640 1 -- 172.21.15.69:0/1496564528 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc041070 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.701+0000 7f0bc3e00640 1 -- 172.21.15.69:0/1496564528 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bbc03c870 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.701+0000 7f0bd1600640 1 -- 172.21.15.69:0/1496564528 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bcc1521b0 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.701+0000 7f0bc3e00640 1 -- 172.21.15.69:0/1496564528 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc037040 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.701+0000 7f0bd1600640 1 -- 172.21.15.69:0/1496564528 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bcc159b60 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.702+0000 7f0bd1600640 1 -- 172.21.15.69:0/1496564528 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bcc075d50 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.703+0000 7f0bc3e00640 1 -- 172.21.15.69:0/1496564528 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0bbc044070 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.704+0000 7f0bc3e00640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0bac06c8f0 0x7f0bac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:53.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.704+0000 7f0bc3e00640 1 -- 172.21.15.69:0/1496564528 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f0bbc0ac740 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.705+0000 7f0bcb400640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0bac06c8f0 0x7f0bac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:53.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.706+0000 7f0bcb400640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0bac06c8f0 0x7f0bac06edb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc0d00c0 tx=0x7f0bb8007910 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:53.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.709+0000 7f0bc3e00640 1 -- 172.21.15.69:0/1496564528 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0bbc07ac80 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.953+0000 7f0bd1600640 1 -- 172.21.15.69:0/1496564528 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0bcc06f9e0 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.954+0000 7f0bc3e00640 1 -- 172.21.15.69:0/1496564528 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5188 (secure 0 0 0) 0x7f0bbc07a580 con 0x7f0bcc0ad7e0 2024-04-24T08:12:53.956 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:12:53.956 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4034001365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4034001365},{"type":"v1","addr":"172.21.15.121:6835","nonce":4034001365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3907163219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3907163219},{"type":"v1","addr":"172.21.15.121:6837","nonce":3907163219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2756395689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2756395689},{"type":"v1","addr":"172.21.15.69:6834","nonce":2756395689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:11:39.824842+0000","modified":"2024-04-24T08:12:28.409867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.69:6835/2288057943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":2288057943},{"type":"v1","addr":"172.21.15.69:6835","nonce":2288057943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6231,"qdb_cluster":[6231]},"id":4}]} 2024-04-24T08:12:53.956 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 35 2024-04-24T08:12:53.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 -- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0bac06c8f0 msgr2=0x7f0bac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0bac06c8f0 0x7f0bac06edb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc0d00c0 tx=0x7f0bb8007910 comp rx=0 tx=0).stop 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 -- 172.21.15.69:0/1496564528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc0ad7e0 msgr2=0x7f0bcc154110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc0ad7e0 0x7f0bcc154110 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc009870 tx=0x7f0bbc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 -- 172.21.15.69:0/1496564528 shutdown_connections 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0bac06c8f0 0x7f0bac06edb0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0bcc156c60 0x7f0bcc160610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0bcc0b30f0 0x7f0bcc0994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 --2- 172.21.15.69:0/1496564528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0bcc0ad7e0 0x7f0bcc154110 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 -- 172.21.15.69:0/1496564528 >> 172.21.15.69:0/1496564528 conn(0x7f0bcc0d0160 msgr2=0x7f0bcc0aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 -- 172.21.15.69:0/1496564528 shutdown_connections 2024-04-24T08:12:53.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:53.957+0000 7f0bd1600640 1 -- 172.21.15.69:0/1496564528 wait complete. 2024-04-24T08:12:53.977 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-04-24T08:12:54.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.179+0000 7fe19da00640 1 Processor -- start 2024-04-24T08:12:54.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.179+0000 7fe19da00640 1 -- start start 2024-04-24T08:12:54.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.181+0000 7fe19da00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe1980cf360 0x7fe1980cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.181+0000 7fe19da00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1980cd6d0 con 0x7fe198155f20 2024-04-24T08:12:54.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.181+0000 7fe19da00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1980cd9b0 con 0x7fe19815c9b0 2024-04-24T08:12:54.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.181+0000 7fe19da00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe19806db90 con 0x7fe1980cf360 2024-04-24T08:12:54.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.181+0000 7fe197400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe1980cf360 0x7fe1980cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.181+0000 7fe196a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe198155f20 0x7fe1980ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54558/0 (socket says 172.21.15.69:54558) 2024-04-24T08:12:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.181+0000 7fe196a00640 1 -- 172.21.15.69:0/3093667189 learned_addr learned my addr 172.21.15.69:0/3093667189 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.181+0000 7fe197400640 1 -- 172.21.15.69:0/3093667189 >> v1:172.21.15.121:6789/0 conn(0x7fe19815c9b0 legacy=0x7fe1980b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.182+0000 7fe197400640 1 -- 172.21.15.69:0/3093667189 >> v1:172.21.15.69:6789/0 conn(0x7fe198155f20 legacy=0x7fe1980ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.182+0000 7fe197400640 1 -- 172.21.15.69:0/3093667189 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1980cdc90 con 0x7fe1980cf360 2024-04-24T08:12:54.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.183+0000 7fe197400640 1 --2- 172.21.15.69:0/3093667189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1980cf360 0x7fe1980cf810 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe190009d90 tx=0x7fe19002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ce110388476a49 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:54.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.183+0000 7fe196000640 1 -- 172.21.15.69:0/3093667189 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190007740 con 0x7fe1980cf360 2024-04-24T08:12:54.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.183+0000 7fe196000640 1 -- 172.21.15.69:0/3093667189 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1900078e0 con 0x7fe1980cf360 2024-04-24T08:12:54.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.183+0000 7fe196000640 1 -- 172.21.15.69:0/3093667189 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190007cb0 con 0x7fe1980cf360 2024-04-24T08:12:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.183+0000 7fe19da00640 1 -- 172.21.15.69:0/3093667189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1980cf360 msgr2=0x7fe1980cf810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.183+0000 7fe19da00640 1 --2- 172.21.15.69:0/3093667189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1980cf360 0x7fe1980cf810 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fe190009d90 tx=0x7fe19002a730 comp rx=0 tx=0).stop 2024-04-24T08:12:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.184+0000 7fe19da00640 1 -- 172.21.15.69:0/3093667189 shutdown_connections 2024-04-24T08:12:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.184+0000 7fe19da00640 1 --2- 172.21.15.69:0/3093667189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1980cf360 0x7fe1980cf810 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.184+0000 7fe19da00640 1 -- 172.21.15.69:0/3093667189 >> 172.21.15.69:0/3093667189 conn(0x7fe1980d0350 msgr2=0x7fe1980d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.184+0000 7fe19da00640 1 -- 172.21.15.69:0/3093667189 shutdown_connections 2024-04-24T08:12:54.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.184+0000 7fe19da00640 1 -- 172.21.15.69:0/3093667189 wait complete. 2024-04-24T08:12:54.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.185+0000 7fe19da00640 1 Processor -- start 2024-04-24T08:12:54.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.186+0000 7fe19da00640 1 -- start start 2024-04-24T08:12:54.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.186+0000 7fe19da00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1980cf360 0x7fe1980c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.186+0000 7fe19da00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe198155f20 0x7fe1980ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.186+0000 7fe19da00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe19815c9b0 0x7fe19814a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.186+0000 7fe19da00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe1980cd470 con 0x7fe19815c9b0 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.186+0000 7fe19da00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe198072070 con 0x7fe1980cf360 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.186+0000 7fe19da00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe1980b4140 con 0x7fe198155f20 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.187+0000 7fe197400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe19815c9b0 0x7fe19814a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.187+0000 7fe197400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe19815c9b0 0x7fe19814a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47374/0 (socket says 172.21.15.69:47374) 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.187+0000 7fe197400640 1 -- 172.21.15.69:0/1759455912 learned_addr learned my addr 172.21.15.69:0/1759455912 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.187+0000 7fe196a00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1980cf360 0x7fe1980c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.187+0000 7fe18fe00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe198155f20 0x7fe1980ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.187+0000 7fe197400640 1 -- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe198155f20 msgr2=0x7fe1980ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.187+0000 7fe197400640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe198155f20 0x7fe1980ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.187+0000 7fe197400640 1 -- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1980cf360 msgr2=0x7fe1980c9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.187+0000 7fe197400640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1980cf360 0x7fe1980c9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.187+0000 7fe197400640 1 -- 172.21.15.69:0/1759455912 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1980c9c40 con 0x7fe19815c9b0 2024-04-24T08:12:54.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.188+0000 7fe197400640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe19815c9b0 0x7fe19814a570 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe190009eb0 tx=0x7fe190004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:54.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.188+0000 7fe18ea00640 1 -- 172.21.15.69:0/1759455912 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190040070 con 0x7fe19815c9b0 2024-04-24T08:12:54.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.188+0000 7fe18ea00640 1 -- 172.21.15.69:0/1759455912 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe190031c90 con 0x7fe19815c9b0 2024-04-24T08:12:54.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.188+0000 7fe18ea00640 1 -- 172.21.15.69:0/1759455912 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe190045680 con 0x7fe19815c9b0 2024-04-24T08:12:54.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.188+0000 7fe19da00640 1 -- 172.21.15.69:0/1759455912 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19815d3c0 con 0x7fe19815c9b0 2024-04-24T08:12:54.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.188+0000 7fe19da00640 1 -- 172.21.15.69:0/1759455912 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe19815d980 con 0x7fe19815c9b0 2024-04-24T08:12:54.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.190+0000 7fe18ea00640 1 -- 172.21.15.69:0/1759455912 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe190039040 con 0x7fe19815c9b0 2024-04-24T08:12:54.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.191+0000 7fe18ea00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe17406c8f0 0x7fe17406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.191+0000 7fe18ea00640 1 -- 172.21.15.69:0/1759455912 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fe1900acfc0 con 0x7fe19815c9b0 2024-04-24T08:12:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.192+0000 7fe1a0869740 1 Processor -- start 2024-04-24T08:12:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.192+0000 7fe196a00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe17406c8f0 0x7fe17406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.192+0000 7fe1a0869740 1 -- start start 2024-04-24T08:12:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.193+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f20883d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.193+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f20a5890 0x55c7f20a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.193+0000 7fe196a00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe17406c8f0 0x7fe17406edb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe188000f60 tx=0x7fe188009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.193+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f205dcd0 0x55c7f20a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.193+0000 7fe1a0869740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c7f1fc2190 con 0x55c7f2087ff0 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.193+0000 7fe1a0869740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c7f1f68030 con 0x55c7f205dcd0 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.193+0000 7fe1a0869740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c7f1fbe150 con 0x55c7f20a5890 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.193+0000 7fe196a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f20883d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.193+0000 7fe196a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f20883d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47382/0 (socket says 172.21.15.69:47382) 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.193+0000 7fe196a00640 1 -- 172.21.15.69:0/216456033 learned_addr learned my addr 172.21.15.69:0/216456033 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe18fe00640 1 --2- 172.21.15.69:0/216456033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f20a5890 0x55c7f20a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe197400640 1 --2- 172.21.15.69:0/216456033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f205dcd0 0x55c7f20a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe196a00640 1 -- 172.21.15.69:0/216456033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f20a5890 msgr2=0x55c7f20a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe196a00640 1 --2- 172.21.15.69:0/216456033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f20a5890 0x55c7f20a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe196a00640 1 -- 172.21.15.69:0/216456033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f205dcd0 msgr2=0x55c7f20a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe196a00640 1 --2- 172.21.15.69:0/216456033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f205dcd0 0x55c7f20a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe196a00640 1 -- 172.21.15.69:0/216456033 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7f1f50b60 con 0x55c7f2087ff0 2024-04-24T08:12:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe196a00640 1 --2- 172.21.15.69:0/216456033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f20883d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe188005c60 tx=0x7fe188010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a27e997fc1db15 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe187400640 1 -- 172.21.15.69:0/216456033 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18800daa0 con 0x55c7f2087ff0 2024-04-24T08:12:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe187400640 1 -- 172.21.15.69:0/216456033 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe18800dc40 con 0x55c7f2087ff0 2024-04-24T08:12:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.194+0000 7fe187400640 1 -- 172.21.15.69:0/216456033 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe188020450 con 0x55c7f2087ff0 2024-04-24T08:12:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.195+0000 7fe1a0869740 1 -- 172.21.15.69:0/216456033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 msgr2=0x55c7f20883d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.195+0000 7fe1a0869740 1 --2- 172.21.15.69:0/216456033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f20883d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe188005c60 tx=0x7fe188010000 comp rx=0 tx=0).stop 2024-04-24T08:12:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.195+0000 7fe1a0869740 1 -- 172.21.15.69:0/216456033 shutdown_connections 2024-04-24T08:12:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.195+0000 7fe1a0869740 1 --2- 172.21.15.69:0/216456033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f205dcd0 0x55c7f20a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.195+0000 7fe1a0869740 1 --2- 172.21.15.69:0/216456033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f20a5890 0x55c7f20a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.195+0000 7fe1a0869740 1 --2- 172.21.15.69:0/216456033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f20883d0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.195+0000 7fe1a0869740 1 -- 172.21.15.69:0/216456033 >> 172.21.15.69:0/216456033 conn(0x55c7f205a600 msgr2=0x55c7f20a2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.195+0000 7fe1a0869740 1 -- 172.21.15.69:0/216456033 shutdown_connections 2024-04-24T08:12:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.195+0000 7fe1a0869740 1 -- 172.21.15.69:0/216456033 wait complete. 2024-04-24T08:12:54.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.196+0000 7fe1a0869740 1 -- start start 2024-04-24T08:12:54.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.196+0000 7fe1a0869740 1 Processor -- start 2024-04-24T08:12:54.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.197+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f2087ff0 0x55c7f20b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.197+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f20a5890 0x55c7f2099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.197+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f209a3f0 0x55c7f209f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.197+0000 7fe1a0869740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c7f205ea70 con 0x55c7f209a3f0 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.197+0000 7fe1a0869740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c7f1f7b500 con 0x55c7f2087ff0 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.197+0000 7fe1a0869740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c7f2058530 con 0x55c7f20a5890 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.197+0000 7fe1a0869740 10 client.0 _ll_register_callbacks cb 0x55c7f1fb8270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe197400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f209a3f0 0x55c7f209f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe197400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f209a3f0 0x55c7f209f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47392/0 (socket says 172.21.15.69:47392) 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe197400640 1 -- 172.21.15.69:0/2195569698 learned_addr learned my addr 172.21.15.69:0/2195569698 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe196a00640 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f2087ff0 0x55c7f20b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe18fe00640 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f20a5890 0x55c7f2099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe197400640 1 -- 172.21.15.69:0/2195569698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f20a5890 msgr2=0x55c7f2099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe197400640 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f20a5890 0x55c7f2099eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe197400640 1 -- 172.21.15.69:0/2195569698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f2087ff0 msgr2=0x55c7f20b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe197400640 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f2087ff0 0x55c7f20b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe197400640 1 -- 172.21.15.69:0/2195569698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7f1da07c0 con 0x55c7f209a3f0 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe197400640 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f209a3f0 0x55c7f209f050 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe190009e80 tx=0x7fe190044ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18d3b9c9857140d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe17be00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.198+0000 7fe17be00640 1 -- 172.21.15.69:0/2195569698 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19007f0c0 con 0x55c7f209a3f0 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe17be00640 1 -- 172.21.15.69:0/2195569698 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fe190083460 con 0x55c7f209a3f0 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe1a0869740 1 -- 172.21.15.69:0/2195569698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe18800dc40 con 0x55c7f209a3f0 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe17be00640 1 -- 172.21.15.69:0/2195569698 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe190099050 con 0x55c7f209a3f0 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe1a0869740 10 client.6576 fetch_fsmap learned FSMap version 35 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe1a0869740 1 -- 172.21.15.69:0/2195569698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55c7f1e00f50 con 0x55c7f209a3f0 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe17be00640 1 -- 172.21.15.69:0/2195569698 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe18800dc40 con 0x55c7f209a3f0 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe17be00640 1 -- 172.21.15.69:0/2195569698 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7fe19007a160 con 0x55c7f209a3f0 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe1a0869740 10 client.6576 fetch_fsmap finished waiting for FSMap version 35 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe1a0869740 10 client.6576 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe1a0869740 20 client.6576 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe1a0869740 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55c7f20b72a0 0x55c7f209d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe1a0869740 4 client.6576 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:54.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.199+0000 7fe1a0869740 1 -- 172.21.15.69:0/2195569698 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7f1fcfbb0 con 0x55c7f20b72a0 2024-04-24T08:12:54.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.200+0000 7fe196a00640 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55c7f20b72a0 0x55c7f209d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.200+0000 7fe196a00640 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55c7f20b72a0 0x55c7f209d230 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55bfb73f6431d569 server_cookie=163c36553d35cdad in_seq=0 out_seq=0 2024-04-24T08:12:54.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.200+0000 7fe17be00640 10 client.6576 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:54.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe17be00640 1 -- 172.21.15.69:0/2195569698 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c7f1fcfbb0 con 0x55c7f20b72a0 2024-04-24T08:12:54.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe17be00640 10 client.6576 handle_command_reply: tid=0 2024-04-24T08:12:54.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 client.6576 shutdown 2024-04-24T08:12:54.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 -- 172.21.15.69:0/2195569698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f209a3f0 msgr2=0x55c7f209f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f209a3f0 0x55c7f209f050 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fe190009e80 tx=0x7fe190044ea0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 -- 172.21.15.69:0/2195569698 shutdown_connections 2024-04-24T08:12:54.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55c7f20b72a0 0x55c7f209d230 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f209a3f0 0x55c7f209f050 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f20a5890 0x55c7f2099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 --2- 172.21.15.69:0/2195569698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f2087ff0 0x55c7f20b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 -- 172.21.15.69:0/2195569698 >> 172.21.15.69:0/2195569698 conn(0x55c7f205dcd0 msgr2=0x55c7f20a6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:54.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe17be00640 0 client.6576 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:54.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 -- 172.21.15.69:0/2195569698 shutdown_connections 2024-04-24T08:12:54.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.201+0000 7fe1a0869740 1 -- 172.21.15.69:0/2195569698 wait complete. 2024-04-24T08:12:54.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.202+0000 7fe1a0869740 20 client.6576 trim_cache size 0 max 16384 2024-04-24T08:12:54.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.216+0000 7fe1a0869740 1 Processor -- start 2024-04-24T08:12:54.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.216+0000 7fe1a0869740 1 -- start start 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f205e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f209a3f0 0x55c7f20b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f20a5890 0x55c7f20b2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe1a0869740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe1981f8f00 con 0x55c7f2087ff0 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe1a0869740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe1981f8260 con 0x55c7f20a5890 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe1a0869740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe1981f88b0 con 0x55c7f209a3f0 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe196a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f205e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe196a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f205e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47406/0 (socket says 172.21.15.69:47406) 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe196a00640 1 -- 172.21.15.69:0/3323727362 learned_addr learned my addr 172.21.15.69:0/3323727362 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe196a00640 1 -- 172.21.15.69:0/3323727362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f209a3f0 msgr2=0x55c7f20b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe196a00640 1 --2- 172.21.15.69:0/3323727362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f209a3f0 0x55c7f20b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe196a00640 1 -- 172.21.15.69:0/3323727362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f20a5890 msgr2=0x55c7f20b2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:54.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.217+0000 7fe196a00640 1 --2- 172.21.15.69:0/3323727362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f20a5890 0x55c7f20b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.218+0000 7fe196a00640 1 -- 172.21.15.69:0/3323727362 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7f1c940b0 con 0x55c7f2087ff0 2024-04-24T08:12:54.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.218+0000 7fe196a00640 1 --2- 172.21.15.69:0/3323727362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f205e2a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe18800d8f0 tx=0x7fe188007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a76df493aa0ed9c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:54.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.218+0000 7fe186a00640 1 -- 172.21.15.69:0/3323727362 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18800daa0 con 0x55c7f2087ff0 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.218+0000 7fe186a00640 1 -- 172.21.15.69:0/3323727362 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe18800dc40 con 0x55c7f2087ff0 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.218+0000 7fe186a00640 1 -- 172.21.15.69:0/3323727362 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18802f380 con 0x55c7f2087ff0 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.219+0000 7fe1a0869740 1 -- 172.21.15.69:0/3323727362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 msgr2=0x55c7f205e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.219+0000 7fe1a0869740 1 --2- 172.21.15.69:0/3323727362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f205e2a0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fe18800d8f0 tx=0x7fe188007000 comp rx=0 tx=0).stop 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.219+0000 7fe1a0869740 1 -- 172.21.15.69:0/3323727362 shutdown_connections 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.219+0000 7fe1a0869740 1 --2- 172.21.15.69:0/3323727362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f20a5890 0x55c7f20b2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.219+0000 7fe1a0869740 1 --2- 172.21.15.69:0/3323727362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f209a3f0 0x55c7f20b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.219+0000 7fe1a0869740 1 --2- 172.21.15.69:0/3323727362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f2087ff0 0x55c7f205e2a0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.219+0000 7fe1a0869740 1 -- 172.21.15.69:0/3323727362 >> 172.21.15.69:0/3323727362 conn(0x55c7f205dcd0 msgr2=0x55c7f20b3fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.219+0000 7fe1a0869740 1 -- 172.21.15.69:0/3323727362 shutdown_connections 2024-04-24T08:12:54.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.219+0000 7fe1a0869740 1 -- 172.21.15.69:0/3323727362 wait complete. 2024-04-24T08:12:54.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.219+0000 7fe1a0869740 1 -- start start 2024-04-24T08:12:54.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.220+0000 7fe1a0869740 1 Processor -- start 2024-04-24T08:12:54.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f2087ff0 0x55c7f20bbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f209a3f0 0x55c7f20acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe1a0869740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f20a5890 0x55c7f20b9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe1a0869740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c7f20b7120 con 0x55c7f20a5890 2024-04-24T08:12:54.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe1a0869740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c7f209b760 con 0x55c7f2087ff0 2024-04-24T08:12:54.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe1a0869740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c7f20b7770 con 0x55c7f209a3f0 2024-04-24T08:12:54.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe1a0869740 10 client.0 _ll_register_callbacks cb 0x7fe190076fb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:54.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe197400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f20a5890 0x55c7f20b9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe197400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f20a5890 0x55c7f20b9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47412/0 (socket says 172.21.15.69:47412) 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe197400640 1 -- 172.21.15.69:0/349551688 learned_addr learned my addr 172.21.15.69:0/349551688 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.221+0000 7fe18fe00640 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f209a3f0 0x55c7f20acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe197400640 1 -- 172.21.15.69:0/349551688 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f209a3f0 msgr2=0x55c7f20acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe197400640 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f209a3f0 0x55c7f20acbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe197400640 1 -- 172.21.15.69:0/349551688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f2087ff0 msgr2=0x55c7f20bbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe197400640 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f2087ff0 0x55c7f20bbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe197400640 1 -- 172.21.15.69:0/349551688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7f2099520 con 0x55c7f20a5890 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe18fe00640 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f209a3f0 0x55c7f20acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe197400640 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f20a5890 0x55c7f20b9900 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe19002ad20 tx=0x7fe190009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c0a9876071afa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe17be00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe17be00640 1 -- 172.21.15.69:0/349551688 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1900a0060 con 0x55c7f20a5890 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe17be00640 1 -- 172.21.15.69:0/349551688 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fe190083660 con 0x55c7f20a5890 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe1a0869740 1 -- 172.21.15.69:0/349551688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe18800dc40 con 0x55c7f20a5890 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe17be00640 1 -- 172.21.15.69:0/349551688 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe19007a670 con 0x55c7f20a5890 2024-04-24T08:12:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe1a0869740 10 client.6582 fetch_fsmap learned FSMap version 35 2024-04-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.222+0000 7fe1a0869740 1 -- 172.21.15.69:0/349551688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) -- 0x55c7f2099520 con 0x55c7f20a5890 2024-04-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.223+0000 7fe17be00640 1 -- 172.21.15.69:0/349551688 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe18800dc40 con 0x55c7f20a5890 2024-04-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.223+0000 7fe17be00640 1 -- 172.21.15.69:0/349551688 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 3494+0+0 (secure 0 0 0) 0x7fe190083b40 con 0x55c7f20a5890 2024-04-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.223+0000 7fe1a0869740 10 client.6582 fetch_fsmap finished waiting for FSMap version 35 2024-04-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.223+0000 7fe1a0869740 10 client.6582 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.223+0000 7fe1a0869740 20 client.6582 populate_metadata read hostname 'smithi069' 2024-04-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.223+0000 7fe1a0869740 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55c7f20b9e40 0x55c7f208af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.223+0000 7fe1a0869740 4 client.6582 mds_command: new command op to 6231 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.223+0000 7fe1a0869740 1 -- 172.21.15.69:0/349551688 --> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55c7f20a56e0 con 0x55c7f20b9e40 2024-04-24T08:12:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.223+0000 7fe196a00640 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55c7f20b9e40 0x55c7f208af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe196a00640 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55c7f20b9e40 0x55c7f208af50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8a73a315321a5f1 server_cookie=b5d2e7bd7b8fbcf9 in_seq=0 out_seq=0 2024-04-24T08:12:54.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe17be00640 10 client.6582 ms_handle_connect on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:54.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe17be00640 1 -- 172.21.15.69:0/349551688 <== mds.0 v2:172.21.15.69:6833/2288057943 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7f20a56e0 con 0x55c7f20b9e40 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe17be00640 10 client.6582 handle_command_reply: tid=0 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 client.6582 shutdown 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 -- 172.21.15.69:0/349551688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f20a5890 msgr2=0x55c7f20b9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f20a5890 0x55c7f20b9900 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fe19002ad20 tx=0x7fe190009750 comp rx=0 tx=0).stop 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 -- 172.21.15.69:0/349551688 shutdown_connections 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.69:6833/2288057943,v1:172.21.15.69:6835/2288057943] conn(0x55c7f20b9e40 0x55c7f208af50 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c7f20a5890 0x55c7f20b9900 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c7f209a3f0 0x55c7f20acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe17be00640 0 client.6582 ms_handle_reset on v2:172.21.15.69:6833/2288057943 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 --2- 172.21.15.69:0/349551688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c7f2087ff0 0x55c7f20bbb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 -- 172.21.15.69:0/349551688 >> 172.21.15.69:0/349551688 conn(0x55c7f205dcd0 msgr2=0x55c7f205e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 -- 172.21.15.69:0/349551688 shutdown_connections 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 1 -- 172.21.15.69:0/349551688 wait complete. 2024-04-24T08:12:54.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.224+0000 7fe1a0869740 20 client.6582 trim_cache size 0 max 16384 2024-04-24T08:12:54.229 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:12:54.229 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-04-24T08:12:54.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 -- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe17406c8f0 msgr2=0x7fe17406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe17406c8f0 0x7fe17406edb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe188000f60 tx=0x7fe188009040 comp rx=0 tx=0).stop 2024-04-24T08:12:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 -- 172.21.15.69:0/1759455912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe19815c9b0 msgr2=0x7fe19814a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe19815c9b0 0x7fe19814a570 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fe190009eb0 tx=0x7fe190004a40 comp rx=0 tx=0).stop 2024-04-24T08:12:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 -- 172.21.15.69:0/1759455912 shutdown_connections 2024-04-24T08:12:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe17406c8f0 0x7fe17406edb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe19815c9b0 0x7fe19814a570 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe198155f20 0x7fe1980ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 --2- 172.21.15.69:0/1759455912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1980cf360 0x7fe1980c9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 -- 172.21.15.69:0/1759455912 >> 172.21.15.69:0/1759455912 conn(0x7fe1980d0350 msgr2=0x7fe198072620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 -- 172.21.15.69:0/1759455912 shutdown_connections 2024-04-24T08:12:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.225+0000 7fe19da00640 1 -- 172.21.15.69:0/1759455912 wait complete. 2024-04-24T08:12:54.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T08:12:54.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.420+0000 7fb5db200640 1 Processor -- start 2024-04-24T08:12:54.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.420+0000 7fb5db200640 1 -- start start 2024-04-24T08:12:54.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.421+0000 7fb5db200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5d415ae80 0x7fb5d40ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.421+0000 7fb5db200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb5d40cc5b0 0x7fb5d40cca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.421+0000 7fb5db200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5d40cb140 con 0x7fb5d40ccf60 2024-04-24T08:12:54.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.421+0000 7fb5db200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb5d415b260 con 0x7fb5d415ae80 2024-04-24T08:12:54.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.421+0000 7fb5db200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb5d40cfbc0 con 0x7fb5d40cc5b0 2024-04-24T08:12:54.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.422+0000 7fb5d3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5d415ae80 0x7fb5d40ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.422+0000 7fb5d3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb5d415ae80 0x7fb5d40ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47416/0 (socket says 172.21.15.69:47416) 2024-04-24T08:12:54.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.422+0000 7fb5d3e00640 1 -- 172.21.15.69:0/636295083 learned_addr learned my addr 172.21.15.69:0/636295083 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:54.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.422+0000 7fb5cb400640 1 --2- 172.21.15.69:0/636295083 >> v2:172.21.15.121:3300/0 conn(0x7fb5d40cc5b0 0x7fb5d40cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.422+0000 7fb5d3e00640 1 -- 172.21.15.69:0/636295083 >> v2:172.21.15.121:3300/0 conn(0x7fb5d40cc5b0 msgr2=0x7fb5d40cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:54.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.422+0000 7fb5d3e00640 1 --2- 172.21.15.69:0/636295083 >> v2:172.21.15.121:3300/0 conn(0x7fb5d40cc5b0 0x7fb5d40cca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.422+0000 7fb5d3e00640 1 -- 172.21.15.69:0/636295083 >> v1:172.21.15.69:6789/0 conn(0x7fb5d40ccf60 legacy=0x7fb5d415e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.422+0000 7fb5d3e00640 1 -- 172.21.15.69:0/636295083 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d40b3da0 con 0x7fb5d415ae80 2024-04-24T08:12:54.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.423+0000 7fb5d3e00640 1 --2- 172.21.15.69:0/636295083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d415ae80 0x7fb5d40ca9f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40cfd40 tx=0x7fb5c402b380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dae0aa663c410ca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:54.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.423+0000 7fb5d3400640 1 -- 172.21.15.69:0/636295083 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4038070 con 0x7fb5d415ae80 2024-04-24T08:12:54.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.423+0000 7fb5d3400640 1 -- 172.21.15.69:0/636295083 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5c4033640 con 0x7fb5d415ae80 2024-04-24T08:12:54.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.423+0000 7fb5d3400640 1 -- 172.21.15.69:0/636295083 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4033a10 con 0x7fb5d415ae80 2024-04-24T08:12:54.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.424+0000 7fb5db200640 1 -- 172.21.15.69:0/636295083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d415ae80 msgr2=0x7fb5d40ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.424+0000 7fb5db200640 1 --2- 172.21.15.69:0/636295083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d415ae80 0x7fb5d40ca9f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40cfd40 tx=0x7fb5c402b380 comp rx=0 tx=0).stop 2024-04-24T08:12:54.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.424+0000 7fb5db200640 1 -- 172.21.15.69:0/636295083 shutdown_connections 2024-04-24T08:12:54.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.424+0000 7fb5db200640 1 --2- 172.21.15.69:0/636295083 >> v2:172.21.15.121:3300/0 conn(0x7fb5d40cc5b0 0x7fb5d40cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.424+0000 7fb5db200640 1 --2- 172.21.15.69:0/636295083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d415ae80 0x7fb5d40ca9f0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.424+0000 7fb5db200640 1 -- 172.21.15.69:0/636295083 >> 172.21.15.69:0/636295083 conn(0x7fb5d406be40 msgr2=0x7fb5d406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:54.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.424+0000 7fb5db200640 1 -- 172.21.15.69:0/636295083 shutdown_connections 2024-04-24T08:12:54.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.424+0000 7fb5db200640 1 -- 172.21.15.69:0/636295083 wait complete. 2024-04-24T08:12:54.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.426+0000 7fb5db200640 1 Processor -- start 2024-04-24T08:12:54.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.426+0000 7fb5db200640 1 -- start start 2024-04-24T08:12:54.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.427+0000 7fb5db200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5d40cc5b0 0x7fb5d41f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.427+0000 7fb5db200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d40ccf60 0x7fb5d41e7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.427+0000 7fb5db200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5d415ae80 0x7fb5d41e82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.427+0000 7fb5db200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d40d0050 con 0x7fb5d40ccf60 2024-04-24T08:12:54.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.427+0000 7fb5db200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d40cd3d0 con 0x7fb5d415ae80 2024-04-24T08:12:54.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.427+0000 7fb5db200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb5d40d01d0 con 0x7fb5d40cc5b0 2024-04-24T08:12:54.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.427+0000 7fb5cb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d40ccf60 0x7fb5d41e7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.427+0000 7fb5cb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d40ccf60 0x7fb5d41e7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47428/0 (socket says 172.21.15.69:47428) 2024-04-24T08:12:54.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.427+0000 7fb5cb400640 1 -- 172.21.15.69:0/1385722221 learned_addr learned my addr 172.21.15.69:0/1385722221 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:54.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5d3e00640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5d40cc5b0 0x7fb5d41f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5cb400640 1 -- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5d40cc5b0 msgr2=0x7fb5d41f4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5cb400640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5d40cc5b0 0x7fb5d41f4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5cb400640 1 -- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5d415ae80 msgr2=0x7fb5d41e82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5cb400640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5d415ae80 0x7fb5d41e82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5cb400640 1 -- 172.21.15.69:0/1385722221 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d41e8820 con 0x7fb5d40ccf60 2024-04-24T08:12:54.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5d3e00640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5d40cc5b0 0x7fb5d41f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:54.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5cb400640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d40ccf60 0x7fb5d41e7da0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00d800 tx=0x7fb5bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5d0c00640 1 -- 172.21.15.69:0/1385722221 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc00b660 con 0x7fb5d40ccf60 2024-04-24T08:12:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5d0c00640 1 -- 172.21.15.69:0/1385722221 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5bc019070 con 0x7fb5d40ccf60 2024-04-24T08:12:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5d0c00640 1 -- 172.21.15.69:0/1385722221 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5bc00ba20 con 0x7fb5d40ccf60 2024-04-24T08:12:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5db200640 1 -- 172.21.15.69:0/1385722221 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d40b0eb0 con 0x7fb5d40ccf60 2024-04-24T08:12:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.428+0000 7fb5db200640 1 -- 172.21.15.69:0/1385722221 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d40b1170 con 0x7fb5d40ccf60 2024-04-24T08:12:54.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.430+0000 7fb5db200640 1 -- 172.21.15.69:0/1385722221 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5d4075dc0 con 0x7fb5d40ccf60 2024-04-24T08:12:54.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.430+0000 7fb5d0c00640 1 -- 172.21.15.69:0/1385722221 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb5bc00bbc0 con 0x7fb5d40ccf60 2024-04-24T08:12:54.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.432+0000 7fb5d0c00640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5b006c8f0 0x7fb5b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:54.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.432+0000 7fb5d0c00640 1 -- 172.21.15.69:0/1385722221 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fb5bc08bf10 con 0x7fb5d40ccf60 2024-04-24T08:12:54.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.432+0000 7fb5d3e00640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5b006c8f0 0x7fb5b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:54.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.433+0000 7fb5d3e00640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5b006c8f0 0x7fb5b006edb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4035000 tx=0x7fb5c402f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:54.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.436+0000 7fb5d0c00640 1 -- 172.21.15.69:0/1385722221 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb5bc01c360 con 0x7fb5d40ccf60 2024-04-24T08:12:54.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.672+0000 7fb5db200640 1 -- 172.21.15.69:0/1385722221 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb5d4073de0 con 0x7fb5d40ccf60 2024-04-24T08:12:54.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.673+0000 7fb5d0c00640 1 -- 172.21.15.69:0/1385722221 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb5bc05a3b0 con 0x7fb5d40ccf60 2024-04-24T08:12:54.674 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:12:54.674 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T08:12:54.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 -- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5b006c8f0 msgr2=0x7fb5b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5b006c8f0 0x7fb5b006edb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4035000 tx=0x7fb5c402f000 comp rx=0 tx=0).stop 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 -- 172.21.15.69:0/1385722221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d40ccf60 msgr2=0x7fb5d41e7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d40ccf60 0x7fb5d41e7da0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00d800 tx=0x7fb5bc00dcc0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 -- 172.21.15.69:0/1385722221 shutdown_connections 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5b006c8f0 0x7fb5b006edb0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb5d40ccf60 0x7fb5d41e7da0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5d415ae80 0x7fb5d41e82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 --2- 172.21.15.69:0/1385722221 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb5d40cc5b0 0x7fb5d41f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 -- 172.21.15.69:0/1385722221 >> 172.21.15.69:0/1385722221 conn(0x7fb5d406be40 msgr2=0x7fb5d40736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 -- 172.21.15.69:0/1385722221 shutdown_connections 2024-04-24T08:12:54.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:54.675+0000 7fb5db200640 1 -- 172.21.15.69:0/1385722221 wait complete. 2024-04-24T08:12:54.694 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-24T08:12:54.694 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:12:54.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:12:54.779 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:12:54.892 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:12:54.892 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:12:54.892 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:12:54.893 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:12:54.947 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:12:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.148+0000 7fcecbe00640 1 Processor -- start 2024-04-24T08:12:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.148+0000 7fcecbe00640 1 -- start start 2024-04-24T08:12:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.150+0000 7fcecbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec415ea10 0x7fcec40d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.150+0000 7fcecbe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcec40af9a0 0x7fcec406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.150+0000 7fcecbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec40adab0 con 0x7fcec41631a0 2024-04-24T08:12:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.150+0000 7fcecbe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcec40add50 con 0x7fcec40af9a0 2024-04-24T08:12:55.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.150+0000 7fcecbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcec40ae030 con 0x7fcec415ea10 2024-04-24T08:12:55.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.150+0000 7fcec8a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcec41631a0 0x7fcec40af290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54578/0 (socket says 172.21.15.69:54578) 2024-04-24T08:12:55.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.150+0000 7fcec8a00640 1 -- 172.21.15.69:0/1980395687 learned_addr learned my addr 172.21.15.69:0/1980395687 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:55.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec9e00640 1 --2- 172.21.15.69:0/1980395687 >> v2:172.21.15.121:3300/0 conn(0x7fcec40af9a0 0x7fcec406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec9400640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec415ea10 0x7fcec40d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006338211 0 0) 0x7fcec40adab0 con 0x7fcec41631a0 2024-04-24T08:12:55.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceac003520 con 0x7fcec41631a0 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2843188012 0 0) 0x7fceac003520 con 0x7fcec41631a0 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec415ea10 msgr2=0x7fcec40d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec3e00640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec415ea10 0x7fcec40d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 >> v2:172.21.15.121:3300/0 conn(0x7fcec40af9a0 msgr2=0x7fcec406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec3e00640 1 --2- 172.21.15.69:0/1980395687 >> v2:172.21.15.121:3300/0 conn(0x7fcec40af9a0 0x7fcec406dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec40ae310 con 0x7fcec41631a0 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fceb0001670 con 0x7fcec41631a0 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.151+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 >> v1:172.21.15.69:6789/0 conn(0x7fcec41631a0 legacy=0x7fcec40af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec3e00640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fceac0039b0 0x7fceac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec3e00640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fceac004800 0x7fceac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec3e00640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec415ea10 0x7fceac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40add50 con 0x7fceac004800 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40ae030 con 0x7fceac0039b0 2024-04-24T08:12:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fceac00b9c0 con 0x7fcec415ea10 2024-04-24T08:12:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec8a00640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fceac004800 0x7fceac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec9400640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fceac0039b0 0x7fceac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec8a00640 1 -- 172.21.15.69:0/1980395687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec415ea10 msgr2=0x7fceac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec8a00640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec415ea10 0x7fceac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec8a00640 1 -- 172.21.15.69:0/1980395687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fceac0039b0 msgr2=0x7fceac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec8a00640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fceac0039b0 0x7fceac003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.152+0000 7fcec8a00640 1 -- 172.21.15.69:0/1980395687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceac003520 con 0x7fceac004800 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.153+0000 7fcec8a00640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fceac004800 0x7fceac004c80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fceb000c190 tx=0x7fceac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc81e5e1f631e7e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.154+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0007c80 con 0x7fceac004800 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.154+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb002e070 con 0x7fceac004800 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.154+0000 7fcec3e00640 1 -- 172.21.15.69:0/1980395687 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0005d50 con 0x7fceac004800 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.154+0000 7fcecbe00640 1 -- 172.21.15.69:0/1980395687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fceac004800 msgr2=0x7fceac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.154+0000 7fcecbe00640 1 --2- 172.21.15.69:0/1980395687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fceac004800 0x7fceac004c80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fceb000c190 tx=0x7fceac005830 comp rx=0 tx=0).stop 2024-04-24T08:12:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.154+0000 7fcec9e00640 1 -- 172.21.15.69:0/1980395687 reap_dead start 2024-04-24T08:12:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.154+0000 7fcecbe00640 1 -- 172.21.15.69:0/1980395687 shutdown_connections 2024-04-24T08:12:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.154+0000 7fcecbe00640 1 -- 172.21.15.69:0/1980395687 >> 172.21.15.69:0/1980395687 conn(0x7fcec4067d90 msgr2=0x7fcec406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.154+0000 7fcecbe00640 1 -- 172.21.15.69:0/1980395687 shutdown_connections 2024-04-24T08:12:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.154+0000 7fcecbe00640 1 -- 172.21.15.69:0/1980395687 wait complete. 2024-04-24T08:12:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.156+0000 7fcecbe00640 1 Processor -- start 2024-04-24T08:12:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.156+0000 7fcecbe00640 1 -- start start 2024-04-24T08:12:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.157+0000 7fcecbe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fceac004800 0x7fcec415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.157+0000 7fcecbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec415a7a0 0x7fcec414dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.157+0000 7fcecbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40ce790 0x7fcec40cec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.157+0000 7fcecbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40aedf0 con 0x7fcec415a7a0 2024-04-24T08:12:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.157+0000 7fcecbe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40ae8d0 con 0x7fceac004800 2024-04-24T08:12:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.157+0000 7fcecbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcec40d0830 con 0x7fcec40ce790 2024-04-24T08:12:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.158+0000 7fcec8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec415a7a0 0x7fcec414dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.158+0000 7fcec8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec415a7a0 0x7fcec414dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47440/0 (socket says 172.21.15.69:47440) 2024-04-24T08:12:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.158+0000 7fcec8a00640 1 -- 172.21.15.69:0/3994431098 learned_addr learned my addr 172.21.15.69:0/3994431098 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:55.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.158+0000 7fcec9400640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fceac004800 0x7fcec415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.158+0000 7fcec8a00640 1 -- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40ce790 msgr2=0x7fcec40cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.158+0000 7fcec8a00640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40ce790 0x7fcec40cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.158+0000 7fcec8a00640 1 -- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fceac004800 msgr2=0x7fcec415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.158+0000 7fcec8a00640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fceac004800 0x7fcec415cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.158+0000 7fcec8a00640 1 -- 172.21.15.69:0/3994431098 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec415bdc0 con 0x7fcec415a7a0 2024-04-24T08:12:55.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.158+0000 7fcec8a00640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec415a7a0 0x7fcec414dcd0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fceb0001030 tx=0x7fceb0001970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:55.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.159+0000 7fcec1600640 1 -- 172.21.15.69:0/3994431098 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0044070 con 0x7fcec415a7a0 2024-04-24T08:12:55.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.159+0000 7fcec1600640 1 -- 172.21.15.69:0/3994431098 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb0007480 con 0x7fcec415a7a0 2024-04-24T08:12:55.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.159+0000 7fcec1600640 1 -- 172.21.15.69:0/3994431098 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb0007c80 con 0x7fcec415a7a0 2024-04-24T08:12:55.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.159+0000 7fcecbe00640 1 -- 172.21.15.69:0/3994431098 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec415ad20 con 0x7fcec415a7a0 2024-04-24T08:12:55.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.159+0000 7fcecbe00640 1 -- 172.21.15.69:0/3994431098 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec415b2e0 con 0x7fcec415a7a0 2024-04-24T08:12:55.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.160+0000 7fcecbe00640 1 -- 172.21.15.69:0/3994431098 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec406f2e0 con 0x7fcec415a7a0 2024-04-24T08:12:55.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.161+0000 7fcec1600640 1 -- 172.21.15.69:0/3994431098 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fceb0033470 con 0x7fcec415a7a0 2024-04-24T08:12:55.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.162+0000 7fcec1600640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcea406c920 0x7fcea406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.162+0000 7fcec1600640 1 -- 172.21.15.69:0/3994431098 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fceb00ab950 con 0x7fcec415a7a0 2024-04-24T08:12:55.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.162+0000 7fcec9400640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcea406c920 0x7fcea406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.164+0000 7fcec9400640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcea406c920 0x7fcea406ede0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fceb8003b80 tx=0x7fceb800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:55.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.166+0000 7fcec1600640 1 -- 172.21.15.69:0/3994431098 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fceb0079ce0 con 0x7fcec415a7a0 2024-04-24T08:12:55.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.366+0000 7fcecbe00640 1 -- 172.21.15.69:0/3994431098 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcec4068a80 con 0x7fcec415a7a0 2024-04-24T08:12:55.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.367+0000 7fcec1600640 1 -- 172.21.15.69:0/3994431098 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16641 (secure 0 0 0) 0x7fceb00795e0 con 0x7fcec415a7a0 2024-04-24T08:12:55.369 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:12:55.369 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":57,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:41.349322+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:11:37.517598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-24T08:11:38.791285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 -- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcea406c920 msgr2=0x7fcea406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcea406c920 0x7fcea406ede0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fceb8003b80 tx=0x7fceb800b040 comp rx=0 tx=0).stop 2024-04-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 -- 172.21.15.69:0/3994431098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec415a7a0 msgr2=0x7fcec414dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec415a7a0 0x7fcec414dcd0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fceb0001030 tx=0x7fceb0001970 comp rx=0 tx=0).stop 2024-04-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 -- 172.21.15.69:0/3994431098 shutdown_connections 2024-04-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcea406c920 0x7fcea406ede0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec415a7a0 0x7fcec414dcd0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcec40ce790 0x7fcec40cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 --2- 172.21.15.69:0/3994431098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fceac004800 0x7fcec415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 -- 172.21.15.69:0/3994431098 >> 172.21.15.69:0/3994431098 conn(0x7fcec4067d90 msgr2=0x7fcec4159120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.370+0000 7fcecbe00640 1 -- 172.21.15.69:0/3994431098 shutdown_connections 2024-04-24T08:12:55.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.371+0000 7fcecbe00640 1 -- 172.21.15.69:0/3994431098 wait complete. 2024-04-24T08:12:55.390 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:11:37.517598+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:11:38.791285+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:12:55.390 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.591+0000 7efeb1400640 1 Processor -- start 2024-04-24T08:12:55.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.591+0000 7efeb1400640 1 -- start start 2024-04-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.593+0000 7efeb1400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15ec30 0x7efeac0d1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.593+0000 7efeb1400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeac0d2710 con 0x7efeac0d1690 2024-04-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.593+0000 7efeb1400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efeac0d29f0 con 0x7efeac15e850 2024-04-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.593+0000 7efeb1400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efeac0d37b0 con 0x7efeac15ec30 2024-04-24T08:12:55.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.593+0000 7efeab200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7efeac0d1690 0x7efeac1688f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54582/0 (socket says 172.21.15.69:54582) 2024-04-24T08:12:55.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.593+0000 7efeab200640 1 -- 172.21.15.69:0/2005508137 learned_addr learned my addr 172.21.15.69:0/2005508137 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:55.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.593+0000 7efea3e00640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15ec30 0x7efeac0d1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2823931336 0 0) 0x7efeac0d2710 con 0x7efeac0d1690 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe90003520 con 0x7efeac0d1690 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4236847157 0 0) 0x7efe90003520 con 0x7efeac0d1690 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15ec30 msgr2=0x7efeac0d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15ec30 0x7efeac0d1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 >> v1:172.21.15.121:6789/0 conn(0x7efeac15e850 legacy=0x7efeac0cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeac0d2cd0 con 0x7efeac0d1690 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7efea40019f0 con 0x7efeac0d1690 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 >> v1:172.21.15.69:6789/0 conn(0x7efeac0d1690 legacy=0x7efeac1688f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe900039e0 0x7efe90003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15ec30 0x7efe90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 0x7efe900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efeac0d37b0 con 0x7efe90004d40 2024-04-24T08:12:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efe9000bad0 con 0x7efe900039e0 2024-04-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efe9000bcd0 con 0x7efeac15ec30 2024-04-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.594+0000 7efeab200640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 0x7efe900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.595+0000 7efeaa800640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe900039e0 0x7efe90003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.595+0000 7efea3e00640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15ec30 0x7efe90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.595+0000 7efeab200640 1 -- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15ec30 msgr2=0x7efe90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.595+0000 7efeab200640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15ec30 0x7efe90004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.595+0000 7efeab200640 1 -- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe900039e0 msgr2=0x7efe90003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.595+0000 7efeab200640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe900039e0 0x7efe90003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.595+0000 7efeab200640 1 -- 172.21.15.69:0/2005508137 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe90003520 con 0x7efe90004d40 2024-04-24T08:12:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.595+0000 7efeaa800640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efe900039e0 0x7efe90003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:55.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.596+0000 7efeab200640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 0x7efe900051c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efea400a740 tx=0x7efe90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a056d0194597a3e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:55.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.596+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea400a390 con 0x7efe90004d40 2024-04-24T08:12:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.596+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efea402f070 con 0x7efe90004d40 2024-04-24T08:12:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.596+0000 7efea9e00640 1 -- 172.21.15.69:0/2005508137 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea402bbc0 con 0x7efe90004d40 2024-04-24T08:12:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.597+0000 7efeb1400640 1 -- 172.21.15.69:0/2005508137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 msgr2=0x7efe900051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.597+0000 7efeb1400640 1 --2- 172.21.15.69:0/2005508137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 0x7efe900051c0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efea400a740 tx=0x7efe90005830 comp rx=0 tx=0).stop 2024-04-24T08:12:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.597+0000 7efeab200640 1 -- 172.21.15.69:0/2005508137 reap_dead start 2024-04-24T08:12:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.597+0000 7efeb1400640 1 -- 172.21.15.69:0/2005508137 shutdown_connections 2024-04-24T08:12:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.597+0000 7efeb1400640 1 -- 172.21.15.69:0/2005508137 >> 172.21.15.69:0/2005508137 conn(0x7efeac08bf60 msgr2=0x7efeac0d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.597+0000 7efeb1400640 1 -- 172.21.15.69:0/2005508137 shutdown_connections 2024-04-24T08:12:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.597+0000 7efeb1400640 1 -- 172.21.15.69:0/2005508137 wait complete. 2024-04-24T08:12:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.599+0000 7efeb1400640 1 Processor -- start 2024-04-24T08:12:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.599+0000 7efeb1400640 1 -- start start 2024-04-24T08:12:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeb1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 0x7efeac0cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeb1400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15b820 0x7efeac0cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeb1400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeac1f8b60 0x7efeac0b9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeb1400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efeac0d1ae0 con 0x7efe90004d40 2024-04-24T08:12:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeb1400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efeac06eba0 con 0x7efeac1f8b60 2024-04-24T08:12:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeb1400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efeac0d3290 con 0x7efeac15b820 2024-04-24T08:12:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeaa800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 0x7efeac0cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeaa800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 0x7efeac0cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47456/0 (socket says 172.21.15.69:47456) 2024-04-24T08:12:55.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeaa800640 1 -- 172.21.15.69:0/3626167897 learned_addr learned my addr 172.21.15.69:0/3626167897 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeaa800640 1 -- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15b820 msgr2=0x7efeac0cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeaa800640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15b820 0x7efeac0cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeaa800640 1 -- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeac1f8b60 msgr2=0x7efeac0b9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeaa800640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeac1f8b60 0x7efeac0b9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.600+0000 7efeaa800640 1 -- 172.21.15.69:0/3626167897 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeac15bca0 con 0x7efe90004d40 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.601+0000 7efeaa800640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 0x7efeac0cb220 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00d7a0 tx=0x7efe9c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.601+0000 7efea2a00640 1 -- 172.21.15.69:0/3626167897 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c005070 con 0x7efe90004d40 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.601+0000 7efea2a00640 1 -- 172.21.15.69:0/3626167897 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe9c005210 con 0x7efe90004d40 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.601+0000 7efea2a00640 1 -- 172.21.15.69:0/3626167897 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9c007030 con 0x7efe90004d40 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.601+0000 7efeb1400640 1 -- 172.21.15.69:0/3626167897 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeac0ce0c0 con 0x7efe90004d40 2024-04-24T08:12:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.601+0000 7efeb1400640 1 -- 172.21.15.69:0/3626167897 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeac0ce680 con 0x7efe90004d40 2024-04-24T08:12:55.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.602+0000 7efeb1400640 1 -- 172.21.15.69:0/3626167897 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeac068a80 con 0x7efe90004d40 2024-04-24T08:12:55.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.603+0000 7efea2a00640 1 -- 172.21.15.69:0/3626167897 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efe9c01a020 con 0x7efe90004d40 2024-04-24T08:12:55.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.604+0000 7efea2a00640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe8806c8f0 0x7efe8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:55.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.604+0000 7efea2a00640 1 -- 172.21.15.69:0/3626167897 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7efe9c08b290 con 0x7efe90004d40 2024-04-24T08:12:55.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.605+0000 7efea3e00640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe8806c8f0 0x7efe8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:55.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.606+0000 7efea3e00640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe8806c8f0 0x7efe8806edb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7efeac0cc8d0 tx=0x7efe94008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:55.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.610+0000 7efea2a00640 1 -- 172.21.15.69:0/3626167897 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe9c059750 con 0x7efe90004d40 2024-04-24T08:12:55.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.792+0000 7efeb1400640 1 -- 172.21.15.69:0/3626167897 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efeac0ba390 con 0x7efe90004d40 2024-04-24T08:12:55.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.793+0000 7efea2a00640 1 -- 172.21.15.69:0/3626167897 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7efe9c059050 con 0x7efe90004d40 2024-04-24T08:12:55.795 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 -- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe8806c8f0 msgr2=0x7efe8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe8806c8f0 0x7efe8806edb0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7efeac0cc8d0 tx=0x7efe94008040 comp rx=0 tx=0).stop 2024-04-24T08:12:55.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 -- 172.21.15.69:0/3626167897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 msgr2=0x7efeac0cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 0x7efeac0cb220 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7efe9c00d7a0 tx=0x7efe9c00dc60 comp rx=0 tx=0).stop 2024-04-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 -- 172.21.15.69:0/3626167897 shutdown_connections 2024-04-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efe8806c8f0 0x7efe8806edb0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efeac1f8b60 0x7efeac0b9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efeac15b820 0x7efeac0cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 --2- 172.21.15.69:0/3626167897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe90004d40 0x7efeac0cb220 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 -- 172.21.15.69:0/3626167897 >> 172.21.15.69:0/3626167897 conn(0x7efeac08bf60 msgr2=0x7efeac0b26e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 -- 172.21.15.69:0/3626167897 shutdown_connections 2024-04-24T08:12:55.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:55.796+0000 7efeb1400640 1 -- 172.21.15.69:0/3626167897 wait complete. 2024-04-24T08:12:55.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:12:56.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.014+0000 7fabebe00640 1 Processor -- start 2024-04-24T08:12:56.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.014+0000 7fabebe00640 1 -- start start 2024-04-24T08:12:56.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabebe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fabec071d70 0x7fabec15ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabebe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabec160510 0x7fabec0cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabebe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabec072450 con 0x7fabec1420a0 2024-04-24T08:12:56.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabebe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fabec160960 con 0x7fabec071d70 2024-04-24T08:12:56.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabebe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fabec0cf1e0 con 0x7fabec160510 2024-04-24T08:12:56.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabe3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fabec071d70 0x7fabec15ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabe3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fabec071d70 0x7fabec15ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47472/0 (socket says 172.21.15.69:47472) 2024-04-24T08:12:56.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabe3e00640 1 -- 172.21.15.69:0/2597572880 learned_addr learned my addr 172.21.15.69:0/2597572880 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:56.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabe3e00640 1 -- 172.21.15.69:0/2597572880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabec160510 msgr2=0x7fabec0cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:56.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabe3e00640 1 --2- 172.21.15.69:0/2597572880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabec160510 0x7fabec0cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabe3e00640 1 -- 172.21.15.69:0/2597572880 >> v1:172.21.15.121:6789/0 conn(0x7fabec1420a0 legacy=0x7fabec142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:56.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.016+0000 7fabe3e00640 1 -- 172.21.15.69:0/2597572880 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec072730 con 0x7fabec071d70 2024-04-24T08:12:56.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.017+0000 7fabe3e00640 1 --2- 172.21.15.69:0/2597572880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 0x7fabec15ffd0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009870 tx=0x7fabd402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff1c18731ee2057 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.018+0000 7fabea000640 1 -- 172.21.15.69:0/2597572880 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4037070 con 0x7fabec071d70 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.018+0000 7fabea000640 1 -- 172.21.15.69:0/2597572880 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabd4032440 con 0x7fabec071d70 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.018+0000 7fabea000640 1 -- 172.21.15.69:0/2597572880 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4032810 con 0x7fabec071d70 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.018+0000 7fabebe00640 1 -- 172.21.15.69:0/2597572880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 msgr2=0x7fabec15ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.018+0000 7fabebe00640 1 --2- 172.21.15.69:0/2597572880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 0x7fabec15ffd0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009870 tx=0x7fabd402a210 comp rx=0 tx=0).stop 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.018+0000 7fabebe00640 1 -- 172.21.15.69:0/2597572880 shutdown_connections 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.018+0000 7fabebe00640 1 --2- 172.21.15.69:0/2597572880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabec160510 0x7fabec0cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.018+0000 7fabebe00640 1 --2- 172.21.15.69:0/2597572880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 0x7fabec15ffd0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.018+0000 7fabebe00640 1 -- 172.21.15.69:0/2597572880 >> 172.21.15.69:0/2597572880 conn(0x7fabec067b80 msgr2=0x7fabec06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.019+0000 7fabebe00640 1 -- 172.21.15.69:0/2597572880 shutdown_connections 2024-04-24T08:12:56.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.019+0000 7fabebe00640 1 -- 172.21.15.69:0/2597572880 wait complete. 2024-04-24T08:12:56.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.021+0000 7fabebe00640 1 Processor -- start 2024-04-24T08:12:56.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.021+0000 7fabebe00640 1 -- start start 2024-04-24T08:12:56.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.022+0000 7fabebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 0x7fabec1e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.022+0000 7fabebe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabec1420a0 0x7fabec156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.022+0000 7fabebe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabec160510 0x7fabec15e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.022+0000 7fabebe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fabec072cf0 con 0x7fabec071d70 2024-04-24T08:12:56.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.022+0000 7fabebe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fabec072200 con 0x7fabec1420a0 2024-04-24T08:12:56.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.022+0000 7fabebe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fabec0cfbb0 con 0x7fabec160510 2024-04-24T08:12:56.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.022+0000 7fabeaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 0x7fabec1e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.022+0000 7fabeaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 0x7fabec1e5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47478/0 (socket says 172.21.15.69:47478) 2024-04-24T08:12:56.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.022+0000 7fabeaa00640 1 -- 172.21.15.69:0/1567869988 learned_addr learned my addr 172.21.15.69:0/1567869988 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:56.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabe3e00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabec1420a0 0x7fabec156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabeb400640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabec160510 0x7fabec15e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabeaa00640 1 -- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabec160510 msgr2=0x7fabec15e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabeaa00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabec160510 0x7fabec15e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabeaa00640 1 -- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabec1420a0 msgr2=0x7fabec156310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabeaa00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabec1420a0 0x7fabec156310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabeaa00640 1 -- 172.21.15.69:0/1567869988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec1e6330 con 0x7fabec071d70 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabe3e00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabec1420a0 0x7fabec156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabeb400640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabec160510 0x7fabec15e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabeaa00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 0x7fabec1e5df0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fabdc00c410 tx=0x7fabdc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabe2a00640 1 -- 172.21.15.69:0/1567869988 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc017070 con 0x7fabec071d70 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabe2a00640 1 -- 172.21.15.69:0/1567869988 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabdc0035f0 con 0x7fabec071d70 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabebe00640 1 -- 172.21.15.69:0/1567869988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec153de0 con 0x7fabec071d70 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.023+0000 7fabe2a00640 1 -- 172.21.15.69:0/1567869988 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc003a20 con 0x7fabec071d70 2024-04-24T08:12:56.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.024+0000 7fabebe00640 1 -- 172.21.15.69:0/1567869988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec13fc20 con 0x7fabec071d70 2024-04-24T08:12:56.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.025+0000 7fabebe00640 1 -- 172.21.15.69:0/1567869988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabec075d50 con 0x7fabec071d70 2024-04-24T08:12:56.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.026+0000 7fabe2a00640 1 -- 172.21.15.69:0/1567869988 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fabdc01a070 con 0x7fabec071d70 2024-04-24T08:12:56.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.026+0000 7fabe2a00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fabc806c8f0 0x7fabc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.027+0000 7fabe2a00640 1 -- 172.21.15.69:0/1567869988 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fabdc08b130 con 0x7fabec071d70 2024-04-24T08:12:56.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.027+0000 7fabe3e00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fabc806c8f0 0x7fabc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.028+0000 7fabe3e00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fabc806c8f0 0x7fabc806edb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fabd4034000 tx=0x7fabd402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:56.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.031+0000 7fabe2a00640 1 -- 172.21.15.69:0/1567869988 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fabdc059430 con 0x7fabec071d70 2024-04-24T08:12:56.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.218+0000 7fabebe00640 1 -- 172.21.15.69:0/1567869988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fabec06c470 con 0x7fabec071d70 2024-04-24T08:12:56.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.219+0000 7fabe2a00640 1 -- 172.21.15.69:0/1567869988 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) ==== 104+0+11 (secure 0 0 0) 0x7fabdc058d30 con 0x7fabec071d70 2024-04-24T08:12:56.220 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:12:56.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.221+0000 7fabebe00640 1 -- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fabc806c8f0 msgr2=0x7fabc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.221+0000 7fabebe00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fabc806c8f0 0x7fabc806edb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fabd4034000 tx=0x7fabd402e000 comp rx=0 tx=0).stop 2024-04-24T08:12:56.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.221+0000 7fabebe00640 1 -- 172.21.15.69:0/1567869988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 msgr2=0x7fabec1e5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.221+0000 7fabebe00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 0x7fabec1e5df0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fabdc00c410 tx=0x7fabdc00c8d0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.221+0000 7fabebe00640 1 -- 172.21.15.69:0/1567869988 shutdown_connections 2024-04-24T08:12:56.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.221+0000 7fabebe00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fabc806c8f0 0x7fabc806edb0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.221+0000 7fabebe00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fabec160510 0x7fabec15e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.221+0000 7fabebe00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fabec1420a0 0x7fabec156310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.221+0000 7fabebe00640 1 --2- 172.21.15.69:0/1567869988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec071d70 0x7fabec1e5df0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.222+0000 7fabebe00640 1 -- 172.21.15.69:0/1567869988 >> 172.21.15.69:0/1567869988 conn(0x7fabec067b80 msgr2=0x7fabec153120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:56.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.222+0000 7fabebe00640 1 -- 172.21.15.69:0/1567869988 shutdown_connections 2024-04-24T08:12:56.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.222+0000 7fabebe00640 1 -- 172.21.15.69:0/1567869988 wait complete. 2024-04-24T08:12:56.242 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:12:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.431+0000 7fbca3e00640 1 Processor -- start 2024-04-24T08:12:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.432+0000 7fbca3e00640 1 -- start start 2024-04-24T08:12:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.433+0000 7fbca3e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbca415e980 0x7fbca4162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.433+0000 7fbca3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbca40d0da0 0x7fbca40cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.433+0000 7fbca3e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbca415ed60 con 0x7fbca40d0190 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.433+0000 7fbca3e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbca40d1580 con 0x7fbca40d0da0 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.433+0000 7fbca3e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fbca40d1860 con 0x7fbca415e980 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.433+0000 7fbc9a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbca40d0da0 0x7fbca40cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.433+0000 7fbca3400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbca40d0190 0x7fbca406cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54592/0 (socket says 172.21.15.69:54592) 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.433+0000 7fbca3400640 1 -- 172.21.15.69:0/4034388157 learned_addr learned my addr 172.21.15.69:0/4034388157 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.434+0000 7fbca2a00640 1 --2- 172.21.15.69:0/4034388157 >> v2:172.21.15.121:3300/0 conn(0x7fbca415e980 0x7fbca4162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.434+0000 7fbc9a000640 1 -- 172.21.15.69:0/4034388157 >> v2:172.21.15.121:3300/0 conn(0x7fbca415e980 msgr2=0x7fbca4162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.434+0000 7fbc9a000640 1 --2- 172.21.15.69:0/4034388157 >> v2:172.21.15.121:3300/0 conn(0x7fbca415e980 0x7fbca4162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.434+0000 7fbc9a000640 1 -- 172.21.15.69:0/4034388157 >> v1:172.21.15.69:6789/0 conn(0x7fbca40d0190 legacy=0x7fbca406cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.434+0000 7fbc9a000640 1 -- 172.21.15.69:0/4034388157 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca40cf450 con 0x7fbca40d0da0 2024-04-24T08:12:56.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.434+0000 7fbca2000640 1 -- 172.21.15.69:0/4034388157 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1812428621 0 0) 0x7fbca415ed60 con 0x7fbca40d0190 2024-04-24T08:12:56.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.435+0000 7fbc9a000640 1 --2- 172.21.15.69:0/4034388157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca40d0da0 0x7fbca40cfc50 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fbca409de50 tx=0x7fbc8c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=813b24f41f03daa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:56.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.435+0000 7fbca2000640 1 -- 172.21.15.69:0/4034388157 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c037070 con 0x7fbca40d0da0 2024-04-24T08:12:56.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.435+0000 7fbca2000640 1 -- 172.21.15.69:0/4034388157 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc8c032440 con 0x7fbca40d0da0 2024-04-24T08:12:56.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.435+0000 7fbca2000640 1 -- 172.21.15.69:0/4034388157 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c032810 con 0x7fbca40d0da0 2024-04-24T08:12:56.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.436+0000 7fbca3e00640 1 -- 172.21.15.69:0/4034388157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca40d0da0 msgr2=0x7fbca40cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.436+0000 7fbca3e00640 1 --2- 172.21.15.69:0/4034388157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca40d0da0 0x7fbca40cfc50 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fbca409de50 tx=0x7fbc8c02a230 comp rx=0 tx=0).stop 2024-04-24T08:12:56.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.436+0000 7fbca3e00640 1 -- 172.21.15.69:0/4034388157 shutdown_connections 2024-04-24T08:12:56.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.436+0000 7fbca3e00640 1 --2- 172.21.15.69:0/4034388157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca40d0da0 0x7fbca40cfc50 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.436+0000 7fbca3e00640 1 --2- 172.21.15.69:0/4034388157 >> v2:172.21.15.121:3300/0 conn(0x7fbca415e980 0x7fbca4162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.436+0000 7fbca3e00640 1 -- 172.21.15.69:0/4034388157 >> 172.21.15.69:0/4034388157 conn(0x7fbca409d920 msgr2=0x7fbca408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:56.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.436+0000 7fbca3e00640 1 -- 172.21.15.69:0/4034388157 shutdown_connections 2024-04-24T08:12:56.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.436+0000 7fbca3e00640 1 -- 172.21.15.69:0/4034388157 wait complete. 2024-04-24T08:12:56.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.438+0000 7fbca3e00640 1 Processor -- start 2024-04-24T08:12:56.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.438+0000 7fbca3e00640 1 -- start start 2024-04-24T08:12:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.439+0000 7fbca3e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbca40d0190 0x7fbca40b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.439+0000 7fbca3e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbca40d0da0 0x7fbca40b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.439+0000 7fbca3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca415e980 0x7fbca40b9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.439+0000 7fbca3e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbca40d1360 con 0x7fbca415e980 2024-04-24T08:12:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.439+0000 7fbca3e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbca40d11e0 con 0x7fbca40d0da0 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.439+0000 7fbca3e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbca40cf070 con 0x7fbca40d0190 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.439+0000 7fbca3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca415e980 0x7fbca40b9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.439+0000 7fbca3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca415e980 0x7fbca40b9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47498/0 (socket says 172.21.15.69:47498) 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.439+0000 7fbca3400640 1 -- 172.21.15.69:0/349067960 learned_addr learned my addr 172.21.15.69:0/349067960 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbca2a00640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbca40d0190 0x7fbca40b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbc9a000640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbca40d0da0 0x7fbca40b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbca3400640 1 -- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbca40d0190 msgr2=0x7fbca40b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbca3400640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbca40d0190 0x7fbca40b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbca3400640 1 -- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbca40d0da0 msgr2=0x7fbca40b8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbca3400640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbca40d0da0 0x7fbca40b8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbca3400640 1 -- 172.21.15.69:0/349067960 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca41e8410 con 0x7fbca415e980 2024-04-24T08:12:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbca3400640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca415e980 0x7fbca40b9220 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00c440 tx=0x7fbc9c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbc9b400640 1 -- 172.21.15.69:0/349067960 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c0074e0 con 0x7fbca415e980 2024-04-24T08:12:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbc9b400640 1 -- 172.21.15.69:0/349067960 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc9c0047e0 con 0x7fbca415e980 2024-04-24T08:12:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbc9b400640 1 -- 172.21.15.69:0/349067960 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9c004bd0 con 0x7fbca415e980 2024-04-24T08:12:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.440+0000 7fbca3e00640 1 -- 172.21.15.69:0/349067960 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca40cd600 con 0x7fbca415e980 2024-04-24T08:12:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.441+0000 7fbca3e00640 1 -- 172.21.15.69:0/349067960 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca40cdea0 con 0x7fbca415e980 2024-04-24T08:12:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.442+0000 7fbca3e00640 1 -- 172.21.15.69:0/349067960 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca41e85e0 con 0x7fbca415e980 2024-04-24T08:12:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.443+0000 7fbc9b400640 1 -- 172.21.15.69:0/349067960 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc9c011070 con 0x7fbca415e980 2024-04-24T08:12:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.443+0000 7fbc9b400640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbc8006c8f0 0x7fbc8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.444+0000 7fbc9b400640 1 -- 172.21.15.69:0/349067960 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fbc9c08b6d0 con 0x7fbca415e980 2024-04-24T08:12:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.444+0000 7fbca2a00640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbc8006c8f0 0x7fbc8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.445+0000 7fbca2a00640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbc8006c8f0 0x7fbc8006edb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbc94009a60 tx=0x7fbc94003990 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:56.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.448+0000 7fbc9b400640 1 -- 172.21.15.69:0/349067960 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc9c059a60 con 0x7fbca415e980 2024-04-24T08:12:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.633+0000 7fbca3e00640 1 -- 172.21.15.69:0/349067960 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbca4075d50 con 0x7fbca415e980 2024-04-24T08:12:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.634+0000 7fbc9b400640 1 -- 172.21.15.69:0/349067960 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) ==== 100+0+11 (secure 0 0 0) 0x7fbc9c059360 con 0x7fbca415e980 2024-04-24T08:12:56.635 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:12:56.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.636+0000 7fbca3e00640 1 -- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbc8006c8f0 msgr2=0x7fbc8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.636+0000 7fbca3e00640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbc8006c8f0 0x7fbc8006edb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbc94009a60 tx=0x7fbc94003990 comp rx=0 tx=0).stop 2024-04-24T08:12:56.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.636+0000 7fbca3e00640 1 -- 172.21.15.69:0/349067960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca415e980 msgr2=0x7fbca40b9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.636+0000 7fbca3e00640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca415e980 0x7fbca40b9220 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c00c440 tx=0x7fbc9c00c900 comp rx=0 tx=0).stop 2024-04-24T08:12:56.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.636+0000 7fbca3e00640 1 -- 172.21.15.69:0/349067960 shutdown_connections 2024-04-24T08:12:56.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.636+0000 7fbca3e00640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbc8006c8f0 0x7fbc8006edb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.636+0000 7fbca3e00640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbca415e980 0x7fbca40b9220 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.636+0000 7fbca3e00640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbca40d0da0 0x7fbca40b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.636+0000 7fbca3e00640 1 --2- 172.21.15.69:0/349067960 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbca40d0190 0x7fbca40b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.637+0000 7fbca3e00640 1 -- 172.21.15.69:0/349067960 >> 172.21.15.69:0/349067960 conn(0x7fbca409d920 msgr2=0x7fbca406cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:56.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.637+0000 7fbca3e00640 1 -- 172.21.15.69:0/349067960 shutdown_connections 2024-04-24T08:12:56.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.637+0000 7fbca3e00640 1 -- 172.21.15.69:0/349067960 wait complete. 2024-04-24T08:12:56.656 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:12:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.852+0000 7f8685600640 1 Processor -- start 2024-04-24T08:12:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.852+0000 7f8685600640 1 -- start start 2024-04-24T08:12:56.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.853+0000 7f8685600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f86800d10f0 0x7f86800d1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.853+0000 7f8685600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86800d22d0 con 0x7f8680162150 2024-04-24T08:12:56.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.853+0000 7f8685600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86800d03b0 con 0x7f86800cfe40 2024-04-24T08:12:56.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.853+0000 7f8685600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f86800d1a80 con 0x7f86800d10f0 2024-04-24T08:12:56.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.854+0000 7f867ea00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8680162150 0x7f86800d5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54606/0 (socket says 172.21.15.69:54606) 2024-04-24T08:12:56.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.854+0000 7f867ea00640 1 -- 172.21.15.69:0/1367365880 learned_addr learned my addr 172.21.15.69:0/1367365880 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.854+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1654219673 0 0) 0x7f86800d22d0 con 0x7f8680162150 2024-04-24T08:12:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.854+0000 7f867f400640 1 --2- 172.21.15.69:0/1367365880 >> v2:172.21.15.121:3300/0 conn(0x7f86800d10f0 0x7f86800d1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.854+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8664003520 con 0x7f8680162150 2024-04-24T08:12:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.854+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1646165120 0 0) 0x7f8664003520 con 0x7f8680162150 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.854+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 >> v2:172.21.15.121:3300/0 conn(0x7f86800d10f0 msgr2=0x7f86800d1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.854+0000 7f867e000640 1 --2- 172.21.15.69:0/1367365880 >> v2:172.21.15.121:3300/0 conn(0x7f86800d10f0 0x7f86800d1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.854+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 >> v1:172.21.15.121:6789/0 conn(0x7f86800cfe40 legacy=0x7f86800b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86800d0690 con 0x7f8680162150 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f86700017f0 con 0x7f8680162150 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 >> v1:172.21.15.69:6789/0 conn(0x7f8680162150 legacy=0x7f86800d5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867e000640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86640039b0 0x7f8664003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867e000640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f8664004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867e000640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86640051c0 0x7f866400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f86800d1a80 con 0x7f8664004800 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f866400c110 con 0x7f86640051c0 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f866400c310 con 0x7f86640039b0 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f8676000640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f8664004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867f400640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86640051c0 0x7f866400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f8676000640 1 -- 172.21.15.69:0/1367365880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86640039b0 msgr2=0x7f8664003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.855+0000 7f867ea00640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86640039b0 0x7f8664003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.856+0000 7f8676000640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86640039b0 0x7f8664003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.856+0000 7f8676000640 1 -- 172.21.15.69:0/1367365880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86640051c0 msgr2=0x7f866400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.856+0000 7f8676000640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86640051c0 0x7f866400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.856+0000 7f8676000640 1 -- 172.21.15.69:0/1367365880 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8664003520 con 0x7f8664004800 2024-04-24T08:12:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.856+0000 7f867f400640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86640051c0 0x7f866400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.856+0000 7f867f400640 1 -- 172.21.15.69:0/1367365880 reap_dead start 2024-04-24T08:12:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.856+0000 7f867ea00640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86640039b0 0x7f8664003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:56.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.857+0000 7f8676000640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f8664004c80 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8668001770 tx=0x7f8664005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55506a0fcf8636bc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:56.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.857+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86680013c0 con 0x7f8664004800 2024-04-24T08:12:56.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.857+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86680052d0 con 0x7f8664004800 2024-04-24T08:12:56.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.857+0000 7f867e000640 1 -- 172.21.15.69:0/1367365880 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f866802a8a0 con 0x7f8664004800 2024-04-24T08:12:56.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.858+0000 7f8685600640 1 -- 172.21.15.69:0/1367365880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 msgr2=0x7f8664004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.858+0000 7f8685600640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f8664004c80 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8668001770 tx=0x7f8664005830 comp rx=0 tx=0).stop 2024-04-24T08:12:56.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.858+0000 7f8685600640 1 -- 172.21.15.69:0/1367365880 shutdown_connections 2024-04-24T08:12:56.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.858+0000 7f8685600640 1 --2- 172.21.15.69:0/1367365880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f8664004c80 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.858+0000 7f8685600640 1 -- 172.21.15.69:0/1367365880 >> 172.21.15.69:0/1367365880 conn(0x7f868008bf60 msgr2=0x7f86800d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:56.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.858+0000 7f8685600640 1 -- 172.21.15.69:0/1367365880 shutdown_connections 2024-04-24T08:12:56.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.858+0000 7f8685600640 1 -- 172.21.15.69:0/1367365880 wait complete. 2024-04-24T08:12:56.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.860+0000 7f8685600640 1 Processor -- start 2024-04-24T08:12:56.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.860+0000 7f8685600640 1 -- start start 2024-04-24T08:12:56.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.861+0000 7f8685600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f86800caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.861+0000 7f8685600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86801f8b60 0x7f868006df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.861+0000 7f8685600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86801f8fe0 0x7f868006e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.861+0000 7f8685600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f86800d1540 con 0x7f8664004800 2024-04-24T08:12:56.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.861+0000 7f8685600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f86800d2900 con 0x7f86801f8b60 2024-04-24T08:12:56.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.861+0000 7f8685600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f86800d16c0 con 0x7f86801f8fe0 2024-04-24T08:12:56.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.861+0000 7f867ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f86800caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.861+0000 7f867ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f86800caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47522/0 (socket says 172.21.15.69:47522) 2024-04-24T08:12:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.861+0000 7f867ea00640 1 -- 172.21.15.69:0/2625634345 learned_addr learned my addr 172.21.15.69:0/2625634345 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f8676000640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86801f8b60 0x7f868006df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f867f400640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86801f8fe0 0x7f868006e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f867ea00640 1 -- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86801f8fe0 msgr2=0x7f868006e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f867ea00640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86801f8fe0 0x7f868006e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f867ea00640 1 -- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86801f8b60 msgr2=0x7f868006df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f867ea00640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86801f8b60 0x7f868006df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:56.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f867ea00640 1 -- 172.21.15.69:0/2625634345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86800cb2c0 con 0x7f8664004800 2024-04-24T08:12:56.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f867ea00640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f86800caca0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8670001960 tx=0x7f867000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:56.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f8677400640 1 -- 172.21.15.69:0/2625634345 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8670007270 con 0x7f8664004800 2024-04-24T08:12:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f8677400640 1 -- 172.21.15.69:0/2625634345 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8670017070 con 0x7f8664004800 2024-04-24T08:12:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f8685600640 1 -- 172.21.15.69:0/2625634345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8680147210 con 0x7f8664004800 2024-04-24T08:12:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f8677400640 1 -- 172.21.15.69:0/2625634345 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8670007610 con 0x7f8664004800 2024-04-24T08:12:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.862+0000 7f8685600640 1 -- 172.21.15.69:0/2625634345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8680146730 con 0x7f8664004800 2024-04-24T08:12:56.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.864+0000 7f8685600640 1 -- 172.21.15.69:0/2625634345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8680068a80 con 0x7f8664004800 2024-04-24T08:12:56.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.864+0000 7f8677400640 1 -- 172.21.15.69:0/2625634345 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f86700057a0 con 0x7f8664004800 2024-04-24T08:12:56.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.865+0000 7f8677400640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f865c06cac0 0x7f865c06ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:56.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.866+0000 7f8677400640 1 -- 172.21.15.69:0/2625634345 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f867008bb60 con 0x7f8664004800 2024-04-24T08:12:56.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.866+0000 7f8676000640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f865c06cac0 0x7f865c06ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:56.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.867+0000 7f8676000640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f865c06cac0 0x7f865c06ef80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8668032000 tx=0x7f866802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:56.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:56.870+0000 7f8677400640 1 -- 172.21.15.69:0/2625634345 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8670059ef0 con 0x7f8664004800 2024-04-24T08:12:57.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.080+0000 7f8685600640 1 -- 172.21.15.69:0/2625634345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f86800d22d0 con 0x7f8664004800 2024-04-24T08:12:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.081+0000 7f8677400640 1 -- 172.21.15.69:0/2625634345 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v57) ==== 80+0+0 (secure 0 0 0) 0x7f86700597f0 con 0x7f8664004800 2024-04-24T08:12:57.082 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:12:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.083+0000 7f8685600640 1 -- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f865c06cac0 msgr2=0x7f865c06ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.083+0000 7f8685600640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f865c06cac0 0x7f865c06ef80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f8668032000 tx=0x7f866802e000 comp rx=0 tx=0).stop 2024-04-24T08:12:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.083+0000 7f8685600640 1 -- 172.21.15.69:0/2625634345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 msgr2=0x7f86800caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.083+0000 7f8685600640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f86800caca0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8670001960 tx=0x7f867000dc00 comp rx=0 tx=0).stop 2024-04-24T08:12:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.084+0000 7f8685600640 1 -- 172.21.15.69:0/2625634345 shutdown_connections 2024-04-24T08:12:57.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.084+0000 7f8685600640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f865c06cac0 0x7f865c06ef80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.084+0000 7f8685600640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f86801f8fe0 0x7f868006e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.084+0000 7f8685600640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86801f8b60 0x7f868006df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.084+0000 7f8685600640 1 --2- 172.21.15.69:0/2625634345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8664004800 0x7f86800caca0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.084+0000 7f8685600640 1 -- 172.21.15.69:0/2625634345 >> 172.21.15.69:0/2625634345 conn(0x7f868008bf60 msgr2=0x7f86800d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.084+0000 7f8685600640 1 -- 172.21.15.69:0/2625634345 shutdown_connections 2024-04-24T08:12:57.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.084+0000 7f8685600640 1 -- 172.21.15.69:0/2625634345 wait complete. 2024-04-24T08:12:57.103 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:12:57.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:57.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:12:57.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[54050]: fuse finished with error 0 and tester_r 0 2024-04-24T08:12:57.172 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:12:57.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:57.172 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:12:57.362 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:12:57.362 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:12:57.362 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:12:57.413 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:12:57.414 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:12:57.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:57.414 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-24T08:12:57.509 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-24T08:12:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:12:57.511 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:12:57.511 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:12:57.511 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:12:57.566 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:12:57.566 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:12:57.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:57.569 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-24T08:12:57.615 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-24T08:12:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:12:57.617 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:12:57.617 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:12:57.617 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:12:57.675 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:12:57.675 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:12:57.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:57.675 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-24T08:12:57.725 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-24T08:12:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:12:57.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:12:57.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.932+0000 7fc2e0e00640 1 Processor -- start 2024-04-24T08:12:57.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.933+0000 7fc2e0e00640 1 -- start start 2024-04-24T08:12:57.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.934+0000 7fc2e0e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc2dc0d08c0 0x7fc2dc0d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:57.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.934+0000 7fc2e0e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2dc0cd680 0x7fc2dc06ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:57.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.934+0000 7fc2e0e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2dc0b0e00 0x7fc2dc15f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:57.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.934+0000 7fc2e0e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc2dc0d11e0 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.934+0000 7fc2e0e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc2dc06c270 con 0x7fc2dc0d08c0 2024-04-24T08:12:57.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.934+0000 7fc2e0e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc06c550 con 0x7fc2dc0cd680 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.934+0000 7fc2da800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2dc0b0e00 0x7fc2dc15f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.935+0000 7fc2da800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2dc0b0e00 0x7fc2dc15f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47532/0 (socket says 172.21.15.69:47532) 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.935+0000 7fc2da800640 1 -- 172.21.15.69:0/3901353876 learned_addr learned my addr 172.21.15.69:0/3901353876 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.935+0000 7fc2d9400640 1 --2- 172.21.15.69:0/3901353876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2dc0cd680 0x7fc2dc06ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.935+0000 7fc2d9e00640 1 --2- 172.21.15.69:0/3901353876 >> v2:172.21.15.121:3300/0 conn(0x7fc2dc0d08c0 0x7fc2dc0d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.935+0000 7fc2da800640 1 -- 172.21.15.69:0/3901353876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2dc0cd680 msgr2=0x7fc2dc06ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.935+0000 7fc2da800640 1 --2- 172.21.15.69:0/3901353876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2dc0cd680 0x7fc2dc06ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.935+0000 7fc2da800640 1 -- 172.21.15.69:0/3901353876 >> v2:172.21.15.121:3300/0 conn(0x7fc2dc0d08c0 msgr2=0x7fc2dc0d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.935+0000 7fc2da800640 1 --2- 172.21.15.69:0/3901353876 >> v2:172.21.15.121:3300/0 conn(0x7fc2dc0d08c0 0x7fc2dc0d0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.935+0000 7fc2da800640 1 -- 172.21.15.69:0/3901353876 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc06c830 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.935+0000 7fc2d9e00640 1 --2- 172.21.15.69:0/3901353876 >> v2:172.21.15.121:3300/0 conn(0x7fc2dc0d08c0 0x7fc2dc0d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:57.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.936+0000 7fc2da800640 1 --2- 172.21.15.69:0/3901353876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 0x7fc2dc15f850 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc072490 tx=0x7fc2d402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967e549025cc8789 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.936+0000 7fc2d8a00640 1 -- 172.21.15.69:0/3901353876 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4037070 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.936+0000 7fc2d8a00640 1 -- 172.21.15.69:0/3901353876 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d4032440 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.936+0000 7fc2d8a00640 1 -- 172.21.15.69:0/3901353876 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4032810 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.937+0000 7fc2e0e00640 1 -- 172.21.15.69:0/3901353876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 msgr2=0x7fc2dc15f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.937+0000 7fc2e0e00640 1 --2- 172.21.15.69:0/3901353876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 0x7fc2dc15f850 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc072490 tx=0x7fc2d402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.937+0000 7fc2e0e00640 1 -- 172.21.15.69:0/3901353876 shutdown_connections 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.937+0000 7fc2e0e00640 1 --2- 172.21.15.69:0/3901353876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 0x7fc2dc15f850 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.937+0000 7fc2e0e00640 1 --2- 172.21.15.69:0/3901353876 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2dc0cd680 0x7fc2dc06ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.937+0000 7fc2e0e00640 1 --2- 172.21.15.69:0/3901353876 >> v2:172.21.15.121:3300/0 conn(0x7fc2dc0d08c0 0x7fc2dc0d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.937+0000 7fc2e0e00640 1 -- 172.21.15.69:0/3901353876 >> 172.21.15.69:0/3901353876 conn(0x7fc2dc09a9d0 msgr2=0x7fc2dc0d14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.937+0000 7fc2e0e00640 1 -- 172.21.15.69:0/3901353876 shutdown_connections 2024-04-24T08:12:57.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.937+0000 7fc2e0e00640 1 -- 172.21.15.69:0/3901353876 wait complete. 2024-04-24T08:12:57.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.939+0000 7fc2e0e00640 1 Processor -- start 2024-04-24T08:12:57.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.939+0000 7fc2e0e00640 1 -- start start 2024-04-24T08:12:57.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2e0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 0x7fc2dc1f45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:57.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2e0e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2dc0cd680 0x7fc2dc0b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:57.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2e0e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2dc0d08c0 0x7fc2dc09d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2e0e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc0b2060 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2e0e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc06bf60 con 0x7fc2dc0cd680 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2e0e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc0b21e0 con 0x7fc2dc0d08c0 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2d9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 0x7fc2dc1f45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2d9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 0x7fc2dc1f45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47540/0 (socket says 172.21.15.69:47540) 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2d9e00640 1 -- 172.21.15.69:0/440246488 learned_addr learned my addr 172.21.15.69:0/440246488 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2d9e00640 1 -- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2dc0d08c0 msgr2=0x7fc2dc09d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2d9e00640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2dc0d08c0 0x7fc2dc09d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2d9e00640 1 -- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2dc0cd680 msgr2=0x7fc2dc0b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2d9e00640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2dc0cd680 0x7fc2dc0b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.940+0000 7fc2d9e00640 1 -- 172.21.15.69:0/440246488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc09c720 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.941+0000 7fc2d9e00640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 0x7fc2dc1f45a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00cf60 tx=0x7fc2cc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:57.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.941+0000 7fc2d2000640 1 -- 172.21.15.69:0/440246488 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc01a070 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.941+0000 7fc2d2000640 1 -- 172.21.15.69:0/440246488 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2cc00d840 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.941+0000 7fc2d2000640 1 -- 172.21.15.69:0/440246488 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc00dc10 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.941+0000 7fc2e0e00640 1 -- 172.21.15.69:0/440246488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2dc0b4240 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.941+0000 7fc2e0e00640 1 -- 172.21.15.69:0/440246488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2dc0b7010 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.942+0000 7fc2e0e00640 1 -- 172.21.15.69:0/440246488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2dc068a80 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.943+0000 7fc2d2000640 1 -- 172.21.15.69:0/440246488 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc2cc01d070 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.944+0000 7fc2d2000640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc2bc06c8f0 0x7fc2bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:57.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.944+0000 7fc2d2000640 1 -- 172.21.15.69:0/440246488 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fc2cc08c330 con 0x7fc2dc0b0e00 2024-04-24T08:12:57.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.945+0000 7fc2d9400640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc2bc06c8f0 0x7fc2bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:57.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.946+0000 7fc2d9400640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc2bc06c8f0 0x7fc2bc06edb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0b6a20 tx=0x7fc2c8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:57.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:57.949+0000 7fc2d2000640 1 -- 172.21.15.69:0/440246488 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc2cc05a070 con 0x7fc2dc0b0e00 2024-04-24T08:12:58.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.191+0000 7fc2e0e00640 1 -- 172.21.15.69:0/440246488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2dc09b5f0 con 0x7fc2dc0b0e00 2024-04-24T08:12:58.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.193+0000 7fc2d2000640 1 -- 172.21.15.69:0/440246488 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5188 (secure 0 0 0) 0x7fc2cc05e370 con 0x7fc2dc0b0e00 2024-04-24T08:12:58.194 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:12:58.194 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4034001365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4034001365},{"type":"v1","addr":"172.21.15.121:6835","nonce":4034001365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3907163219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3907163219},{"type":"v1","addr":"172.21.15.121:6837","nonce":3907163219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2756395689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2756395689},{"type":"v1","addr":"172.21.15.69:6834","nonce":2756395689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:11:39.824842+0000","modified":"2024-04-24T08:12:28.409867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.69:6835/2288057943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":2288057943},{"type":"v1","addr":"172.21.15.69:6835","nonce":2288057943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6231,"qdb_cluster":[6231]},"id":4}]} 2024-04-24T08:12:58.194 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 35 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 -- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc2bc06c8f0 msgr2=0x7fc2bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc2bc06c8f0 0x7fc2bc06edb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0b6a20 tx=0x7fc2c8008040 comp rx=0 tx=0).stop 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 -- 172.21.15.69:0/440246488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 msgr2=0x7fc2dc1f45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 0x7fc2dc1f45a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc00cf60 tx=0x7fc2cc00cf90 comp rx=0 tx=0).stop 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 -- 172.21.15.69:0/440246488 shutdown_connections 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc2bc06c8f0 0x7fc2bc06edb0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2dc0d08c0 0x7fc2dc09d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2dc0cd680 0x7fc2dc0b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 --2- 172.21.15.69:0/440246488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2dc0b0e00 0x7fc2dc1f45a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 -- 172.21.15.69:0/440246488 >> 172.21.15.69:0/440246488 conn(0x7fc2dc09a9d0 msgr2=0x7fc2dc15f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:58.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 -- 172.21.15.69:0/440246488 shutdown_connections 2024-04-24T08:12:58.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.195+0000 7fc2e0e00640 1 -- 172.21.15.69:0/440246488 wait complete. 2024-04-24T08:12:58.216 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:12:58.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.413+0000 7fcf09000640 1 Processor -- start 2024-04-24T08:12:58.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.414+0000 7fcf09000640 1 -- start start 2024-04-24T08:12:58.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.415+0000 7fcf09000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415e890 0x7fcf040d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.415+0000 7fcf09000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcf040d05d0 0x7fcf040d16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.415+0000 7fcf09000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf040d26c0 con 0x7fcf0415ec70 2024-04-24T08:12:58.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.415+0000 7fcf09000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcf040d32d0 con 0x7fcf040d05d0 2024-04-24T08:12:58.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.415+0000 7fcf09000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcf040d29a0 con 0x7fcf0415e890 2024-04-24T08:12:58.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.415+0000 7fcf01400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcf0415ec70 0x7fcf040cfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:54620/0 (socket says 172.21.15.69:54620) 2024-04-24T08:12:58.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.415+0000 7fcf01400640 1 -- 172.21.15.69:0/3695950903 learned_addr learned my addr 172.21.15.69:0/3695950903 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:58.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2388777853 0 0) 0x7fcf040d26c0 con 0x7fcf0415ec70 2024-04-24T08:12:58.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcee8003520 con 0x7fcf0415ec70 2024-04-24T08:12:58.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf02800640 1 --2- 172.21.15.69:0/3695950903 >> v2:172.21.15.121:3300/0 conn(0x7fcf040d05d0 0x7fcf040d16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf01e00640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415e890 0x7fcf040d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3807857322 0 0) 0x7fcee8003520 con 0x7fcf0415ec70 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415e890 msgr2=0x7fcf040d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415e890 0x7fcf040d11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 >> v2:172.21.15.121:3300/0 conn(0x7fcf040d05d0 msgr2=0x7fcf040d16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 --2- 172.21.15.69:0/3695950903 >> v2:172.21.15.121:3300/0 conn(0x7fcf040d05d0 0x7fcf040d16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf040d2c80 con 0x7fcf0415ec70 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fceec005090 con 0x7fcf0415ec70 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 >> v1:172.21.15.69:6789/0 conn(0x7fcf0415ec70 legacy=0x7fcf040cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 0x7fcee8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee8004800 0x7fcee8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415e890 0x7fcee8005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcf040d32d0 con 0x7fcee80039b0 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcf040d29a0 con 0x7fcee8004800 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.416+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcee800bc30 con 0x7fcf0415e890 2024-04-24T08:12:58.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.417+0000 7fcf01e00640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 0x7fcee8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.417+0000 7fcf02800640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415e890 0x7fcee8005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.417+0000 7fcf01e00640 1 -- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415e890 msgr2=0x7fcee8005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.417+0000 7fcf01400640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee8004800 0x7fcee8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.417+0000 7fcf01e00640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415e890 0x7fcee8005220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.417+0000 7fcf01e00640 1 -- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee8004800 msgr2=0x7fcee8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.417+0000 7fcf01e00640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcee8004800 0x7fcee8004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.417+0000 7fcf01e00640 1 -- 172.21.15.69:0/3695950903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee8003520 con 0x7fcee80039b0 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.418+0000 7fcf01e00640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 0x7fcee8003dd0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcef400a9d0 tx=0x7fcee8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208c102214beb483 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.418+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4037070 con 0x7fcee80039b0 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.418+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef402ab80 con 0x7fcee80039b0 2024-04-24T08:12:58.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.418+0000 7fcf00a00640 1 -- 172.21.15.69:0/3695950903 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef400a570 con 0x7fcee80039b0 2024-04-24T08:12:58.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.419+0000 7fcf09000640 1 -- 172.21.15.69:0/3695950903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 msgr2=0x7fcee8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.419+0000 7fcf09000640 1 --2- 172.21.15.69:0/3695950903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 0x7fcee8003dd0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fcef400a9d0 tx=0x7fcee8005830 comp rx=0 tx=0).stop 2024-04-24T08:12:58.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.419+0000 7fcf02800640 1 -- 172.21.15.69:0/3695950903 reap_dead start 2024-04-24T08:12:58.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.419+0000 7fcf09000640 1 -- 172.21.15.69:0/3695950903 shutdown_connections 2024-04-24T08:12:58.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.419+0000 7fcf09000640 1 -- 172.21.15.69:0/3695950903 >> 172.21.15.69:0/3695950903 conn(0x7fcf0408bf60 msgr2=0x7fcf040d47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:58.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.419+0000 7fcf09000640 1 -- 172.21.15.69:0/3695950903 shutdown_connections 2024-04-24T08:12:58.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.419+0000 7fcf09000640 1 -- 172.21.15.69:0/3695950903 wait complete. 2024-04-24T08:12:58.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.421+0000 7fcf09000640 1 Processor -- start 2024-04-24T08:12:58.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.421+0000 7fcf09000640 1 -- start start 2024-04-24T08:12:58.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf09000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 0x7fcf0415b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf09000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415bda0 0x7fcf04159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf09000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf0415c200 0x7fcf0414c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf09000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcf040d2370 con 0x7fcee80039b0 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf09000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcf040d3760 con 0x7fcf0415c200 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf09000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcf040d11b0 con 0x7fcf0415bda0 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf01e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 0x7fcf0415b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf01e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 0x7fcf0415b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47544/0 (socket says 172.21.15.69:47544) 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf01e00640 1 -- 172.21.15.69:0/2791437745 learned_addr learned my addr 172.21.15.69:0/2791437745 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf01e00640 1 -- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415bda0 msgr2=0x7fcf04159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf01e00640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415bda0 0x7fcf04159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf01e00640 1 -- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf0415c200 msgr2=0x7fcf0414c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf01e00640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf0415c200 0x7fcf0414c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.422+0000 7fcf01e00640 1 -- 172.21.15.69:0/2791437745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0415a0b0 con 0x7fcee80039b0 2024-04-24T08:12:58.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.423+0000 7fcf01e00640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 0x7fcf0415b150 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fcef400a9d0 tx=0x7fcef4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:58.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.423+0000 7fcefa000640 1 -- 172.21.15.69:0/2791437745 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4041020 con 0x7fcee80039b0 2024-04-24T08:12:58.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.423+0000 7fcefa000640 1 -- 172.21.15.69:0/2791437745 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef403d420 con 0x7fcee80039b0 2024-04-24T08:12:58.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.423+0000 7fcf09000640 1 -- 172.21.15.69:0/2791437745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0415a390 con 0x7fcee80039b0 2024-04-24T08:12:58.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.423+0000 7fcefa000640 1 -- 172.21.15.69:0/2791437745 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef4037040 con 0x7fcee80039b0 2024-04-24T08:12:58.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.423+0000 7fcf09000640 1 -- 172.21.15.69:0/2791437745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf0415a950 con 0x7fcee80039b0 2024-04-24T08:12:58.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.424+0000 7fcf09000640 1 -- 172.21.15.69:0/2791437745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf040d38e0 con 0x7fcee80039b0 2024-04-24T08:12:58.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.425+0000 7fcefa000640 1 -- 172.21.15.69:0/2791437745 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcef4044020 con 0x7fcee80039b0 2024-04-24T08:12:58.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.426+0000 7fcefa000640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcee006c920 0x7fcee006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.426+0000 7fcefa000640 1 -- 172.21.15.69:0/2791437745 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7fcef40ac060 con 0x7fcee80039b0 2024-04-24T08:12:58.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.427+0000 7fcf01400640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcee006c920 0x7fcee006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.428+0000 7fcf01400640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcee006c920 0x7fcee006ede0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fceec004870 tx=0x7fceec009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:58.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.430+0000 7fcefa000640 1 -- 172.21.15.69:0/2791437745 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcf040d38e0 con 0x7fcee80039b0 2024-04-24T08:12:58.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.634+0000 7fcf09000640 1 -- 172.21.15.69:0/2791437745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcf040d03b0 con 0x7fcee80039b0 2024-04-24T08:12:58.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.637+0000 7fcefa000640 1 -- 172.21.15.69:0/2791437745 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16641 (secure 0 0 0) 0x7fcf040d03b0 con 0x7fcee80039b0 2024-04-24T08:12:58.639 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:12:58.639 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":57,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:41.349322+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:11:37.517598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-24T08:11:38.791285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:12:58.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 -- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcee006c920 msgr2=0x7fcee006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcee006c920 0x7fcee006ede0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fceec004870 tx=0x7fceec009040 comp rx=0 tx=0).stop 2024-04-24T08:12:58.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 -- 172.21.15.69:0/2791437745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 msgr2=0x7fcf0415b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 0x7fcf0415b150 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fcef400a9d0 tx=0x7fcef4004730 comp rx=0 tx=0).stop 2024-04-24T08:12:58.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 -- 172.21.15.69:0/2791437745 shutdown_connections 2024-04-24T08:12:58.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcee006c920 0x7fcee006ede0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf0415c200 0x7fcf0414c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf0415bda0 0x7fcf04159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 --2- 172.21.15.69:0/2791437745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcee80039b0 0x7fcf0415b150 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 -- 172.21.15.69:0/2791437745 >> 172.21.15.69:0/2791437745 conn(0x7fcf0408bf60 msgr2=0x7fcf0415f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:58.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 -- 172.21.15.69:0/2791437745 shutdown_connections 2024-04-24T08:12:58.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.640+0000 7fcf09000640 1 -- 172.21.15.69:0/2791437745 wait complete. 2024-04-24T08:12:58.659 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:11:37.517598+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:11:38.791285+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '57', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:12:58.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:12:58.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.859+0000 7f9d98a00640 1 Processor -- start 2024-04-24T08:12:58.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.859+0000 7f9d98a00640 1 -- start start 2024-04-24T08:12:58.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.860+0000 7f9d98a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9d940b46a0 0x7f9d940b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.861+0000 7f9d98a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d940739f0 con 0x7f9d9415f4b0 2024-04-24T08:12:58.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.861+0000 7f9d98a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d94073cc0 con 0x7f9d940b3de0 2024-04-24T08:12:58.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.861+0000 7f9d98a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9d940b5010 con 0x7f9d940b46a0 2024-04-24T08:12:58.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.861+0000 7f9d91e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9d9415f4b0 0x7f9d9415f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56516/0 (socket says 172.21.15.69:56516) 2024-04-24T08:12:58.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.861+0000 7f9d91e00640 1 -- 172.21.15.69:0/111487914 learned_addr learned my addr 172.21.15.69:0/111487914 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:58.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.861+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1362476589 0 0) 0x7f9d940739f0 con 0x7f9d9415f4b0 2024-04-24T08:12:58.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.861+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d78003520 con 0x7f9d9415f4b0 2024-04-24T08:12:58.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.861+0000 7f9d92800640 1 --2- 172.21.15.69:0/111487914 >> v2:172.21.15.121:3300/0 conn(0x7f9d940b46a0 0x7f9d940b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 699291038 0 0) 0x7f9d78003520 con 0x7f9d9415f4b0 2024-04-24T08:12:58.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 >> v2:172.21.15.121:3300/0 conn(0x7f9d940b46a0 msgr2=0x7f9d940b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:58.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 --2- 172.21.15.69:0/111487914 >> v2:172.21.15.121:3300/0 conn(0x7f9d940b46a0 0x7f9d940b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 >> v1:172.21.15.121:6789/0 conn(0x7f9d940b3de0 legacy=0x7f9d940732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d94073fa0 con 0x7f9d9415f4b0 2024-04-24T08:12:58.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f9d84001670 con 0x7f9d9415f4b0 2024-04-24T08:12:58.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 >> v1:172.21.15.69:6789/0 conn(0x7f9d9415f4b0 legacy=0x7f9d9415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 --2- 172.21.15.69:0/111487914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d780039b0 0x7f9d78003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 --2- 172.21.15.69:0/111487914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d78004800 0x7f9d78004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 --2- 172.21.15.69:0/111487914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d780051c0 0x7f9d7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d940b5010 con 0x7f9d780051c0 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7800c110 con 0x7f9d780039b0 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d7800c310 con 0x7f9d78004800 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d92800640 1 --2- 172.21.15.69:0/111487914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d780051c0 0x7f9d7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d92800640 1 -- 172.21.15.69:0/111487914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d78004800 msgr2=0x7f9d78004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d92800640 1 --2- 172.21.15.69:0/111487914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d78004800 0x7f9d78004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d92800640 1 -- 172.21.15.69:0/111487914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d780039b0 msgr2=0x7f9d78003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d92800640 1 --2- 172.21.15.69:0/111487914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d780039b0 0x7f9d78003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.862+0000 7f9d92800640 1 -- 172.21.15.69:0/111487914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d78003520 con 0x7f9d780051c0 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.863+0000 7f9d92800640 1 -- 172.21.15.69:0/111487914 reap_dead start 2024-04-24T08:12:58.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.863+0000 7f9d92800640 1 --2- 172.21.15.69:0/111487914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d780051c0 0x7f9d7800b9c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c007360 tx=0x7f9d78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527de80f46265e32 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:58.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.864+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c009be0 con 0x7f9d780051c0 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.864+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d8c003bd0 con 0x7f9d780051c0 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.864+0000 7f9d91400640 1 -- 172.21.15.69:0/111487914 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c004020 con 0x7f9d780051c0 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.864+0000 7f9d98a00640 1 -- 172.21.15.69:0/111487914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d780051c0 msgr2=0x7f9d7800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.864+0000 7f9d98a00640 1 --2- 172.21.15.69:0/111487914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d780051c0 0x7f9d7800b9c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c007360 tx=0x7f9d78005830 comp rx=0 tx=0).stop 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.865+0000 7f9d98a00640 1 -- 172.21.15.69:0/111487914 shutdown_connections 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.865+0000 7f9d98a00640 1 --2- 172.21.15.69:0/111487914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d780051c0 0x7f9d7800b9c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.865+0000 7f9d98a00640 1 -- 172.21.15.69:0/111487914 >> 172.21.15.69:0/111487914 conn(0x7f9d940d0350 msgr2=0x7f9d940d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.865+0000 7f9d98a00640 1 -- 172.21.15.69:0/111487914 shutdown_connections 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.865+0000 7f9d98a00640 1 -- 172.21.15.69:0/111487914 wait complete. 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.866+0000 7f9d98a00640 1 Processor -- start 2024-04-24T08:12:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.866+0000 7f9d98a00640 1 -- start start 2024-04-24T08:12:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.867+0000 7f9d98a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d780051c0 0x7f9d94147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.867+0000 7f9d98a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d941e7560 0x7f9d940cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.867+0000 7f9d98a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d941608e0 0x7f9d94160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.867+0000 7f9d98a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d940b38c0 con 0x7f9d941e7560 2024-04-24T08:12:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.867+0000 7f9d98a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9d940b35e0 con 0x7f9d780051c0 2024-04-24T08:12:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.867+0000 7f9d98a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d940b4ad0 con 0x7f9d941608e0 2024-04-24T08:12:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.868+0000 7f9d8be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d941e7560 0x7f9d940cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.868+0000 7f9d8be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d941e7560 0x7f9d940cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37384/0 (socket says 172.21.15.69:37384) 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.868+0000 7f9d8be00640 1 -- 172.21.15.69:0/860701891 learned_addr learned my addr 172.21.15.69:0/860701891 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.868+0000 7f9d8be00640 1 -- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d941608e0 msgr2=0x7f9d94160d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.868+0000 7f9d8be00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d941608e0 0x7f9d94160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.868+0000 7f9d8be00640 1 -- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d780051c0 msgr2=0x7f9d94147860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.868+0000 7f9d8be00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d780051c0 0x7f9d94147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.868+0000 7f9d8be00640 1 -- 172.21.15.69:0/860701891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d940cd560 con 0x7f9d941e7560 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.868+0000 7f9d8be00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d941e7560 0x7f9d940cbbe0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c430 tx=0x7f9d7c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.869+0000 7f9d8a000640 1 -- 172.21.15.69:0/860701891 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c017070 con 0x7f9d941e7560 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.869+0000 7f9d8a000640 1 -- 172.21.15.69:0/860701891 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d7c0052f0 con 0x7f9d941e7560 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.869+0000 7f9d8a000640 1 -- 172.21.15.69:0/860701891 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c003830 con 0x7f9d941e7560 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.869+0000 7f9d98a00640 1 -- 172.21.15.69:0/860701891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d94147da0 con 0x7f9d941e7560 2024-04-24T08:12:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.869+0000 7f9d98a00640 1 -- 172.21.15.69:0/860701891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d940cc7a0 con 0x7f9d941e7560 2024-04-24T08:12:58.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.870+0000 7f9d98a00640 1 -- 172.21.15.69:0/860701891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d941612a0 con 0x7f9d941e7560 2024-04-24T08:12:58.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.871+0000 7f9d8a000640 1 -- 172.21.15.69:0/860701891 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9d7c01a070 con 0x7f9d941e7560 2024-04-24T08:12:58.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.872+0000 7f9d8a000640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d7006c8f0 0x7f9d7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:58.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.872+0000 7f9d8a000640 1 -- 172.21.15.69:0/860701891 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f9d7c08afe0 con 0x7f9d941e7560 2024-04-24T08:12:58.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.872+0000 7f9d91e00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d7006c8f0 0x7f9d7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:58.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.874+0000 7f9d91e00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d7006c8f0 0x7f9d7006edb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9d840079b0 tx=0x7f9d84001540 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:58.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:58.876+0000 7f9d8a000640 1 -- 172.21.15.69:0/860701891 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d940cd560 con 0x7f9d941e7560 2024-04-24T08:12:59.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.079+0000 7f9d98a00640 1 -- 172.21.15.69:0/860701891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9d94068a80 con 0x7f9d941e7560 2024-04-24T08:12:59.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.080+0000 7f9d8a000640 1 -- 172.21.15.69:0/860701891 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) ==== 93+0+10 (secure 0 0 0) 0x7f9d7c0593b0 con 0x7f9d941e7560 2024-04-24T08:12:59.082 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:12:59.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.083+0000 7f9d98a00640 1 -- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d7006c8f0 msgr2=0x7f9d7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.083+0000 7f9d98a00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d7006c8f0 0x7f9d7006edb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9d840079b0 tx=0x7f9d84001540 comp rx=0 tx=0).stop 2024-04-24T08:12:59.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.083+0000 7f9d98a00640 1 -- 172.21.15.69:0/860701891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d941e7560 msgr2=0x7f9d940cbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.083+0000 7f9d98a00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d941e7560 0x7f9d940cbbe0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c00c430 tx=0x7f9d7c00c8f0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.083+0000 7f9d98a00640 1 -- 172.21.15.69:0/860701891 shutdown_connections 2024-04-24T08:12:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.083+0000 7f9d98a00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d7006c8f0 0x7f9d7006edb0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.083+0000 7f9d98a00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d941e7560 0x7f9d940cbbe0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.083+0000 7f9d98a00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d941608e0 0x7f9d94160d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.084+0000 7f9d98a00640 1 --2- 172.21.15.69:0/860701891 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d780051c0 0x7f9d94147860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.084+0000 7f9d98a00640 1 -- 172.21.15.69:0/860701891 >> 172.21.15.69:0/860701891 conn(0x7f9d940d0350 msgr2=0x7f9d940d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.084+0000 7f9d98a00640 1 -- 172.21.15.69:0/860701891 shutdown_connections 2024-04-24T08:12:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.084+0000 7f9d98a00640 1 -- 172.21.15.69:0/860701891 wait complete. 2024-04-24T08:12:59.104 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:12:59.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.305+0000 7f74cb000640 1 Processor -- start 2024-04-24T08:12:59.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.306+0000 7f74cb000640 1 -- start start 2024-04-24T08:12:59.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.307+0000 7f74cb000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f74c4155f20 0x7f74c40ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.307+0000 7f74cb000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c415c9b0 0x7f74c40b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.307+0000 7f74cb000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74c40cd6d0 con 0x7f74c40b3f70 2024-04-24T08:12:59.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.307+0000 7f74cb000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f74c406db90 con 0x7f74c4155f20 2024-04-24T08:12:59.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.307+0000 7f74cb000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f74c4072070 con 0x7f74c415c9b0 2024-04-24T08:12:59.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c9000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f74c40b3f70 0x7f74c40cf360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56518/0 (socket says 172.21.15.69:56518) 2024-04-24T08:12:59.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c9000640 1 -- 172.21.15.69:0/273600432 learned_addr learned my addr 172.21.15.69:0/273600432 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74bb400640 1 --2- 172.21.15.69:0/273600432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c415c9b0 0x7f74c40b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330055693 0 0) 0x7f74c40cd6d0 con 0x7f74c40b3f70 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3e00640 1 --2- 172.21.15.69:0/273600432 >> v2:172.21.15.121:3300/0 conn(0x7f74c4155f20 0x7f74c40ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74a8003520 con 0x7f74c40b3f70 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3892717241 0 0) 0x7f74a8003520 con 0x7f74c40b3f70 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c415c9b0 msgr2=0x7f74c40b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3400640 1 --2- 172.21.15.69:0/273600432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c415c9b0 0x7f74c40b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 >> v2:172.21.15.121:3300/0 conn(0x7f74c4155f20 msgr2=0x7f74c40ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3400640 1 --2- 172.21.15.69:0/273600432 >> v2:172.21.15.121:3300/0 conn(0x7f74c4155f20 0x7f74c40ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c40cd9b0 con 0x7f74c40b3f70 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f74bc0019f0 con 0x7f74c40b3f70 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.308+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 >> v1:172.21.15.69:6789/0 conn(0x7f74c40b3f70 legacy=0x7f74c40cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3400640 1 --2- 172.21.15.69:0/273600432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 0x7f74a8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3400640 1 --2- 172.21.15.69:0/273600432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74a8004800 0x7f74a8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3400640 1 --2- 172.21.15.69:0/273600432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c415c9b0 0x7f74a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f74c4072070 con 0x7f74a80039b0 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f74a800dd10 con 0x7f74a8004800 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f74a800dee0 con 0x7f74c415c9b0 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3e00640 1 --2- 172.21.15.69:0/273600432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 0x7f74a8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3e00640 1 -- 172.21.15.69:0/273600432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c415c9b0 msgr2=0x7f74a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3e00640 1 --2- 172.21.15.69:0/273600432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c415c9b0 0x7f74a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3e00640 1 -- 172.21.15.69:0/273600432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74a8004800 msgr2=0x7f74a8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3e00640 1 --2- 172.21.15.69:0/273600432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74a8004800 0x7f74a8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.309+0000 7f74c3e00640 1 -- 172.21.15.69:0/273600432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74a8003520 con 0x7f74a80039b0 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.310+0000 7f74c3e00640 1 --2- 172.21.15.69:0/273600432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 0x7f74a8003dd0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f74b400ba00 tx=0x7f74a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3a6ef62bac4559f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.311+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4030070 con 0x7f74a80039b0 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.311+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74b402adb0 con 0x7f74a80039b0 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.311+0000 7f74c3400640 1 -- 172.21.15.69:0/273600432 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b400b600 con 0x7f74a80039b0 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.311+0000 7f74cb000640 1 -- 172.21.15.69:0/273600432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 msgr2=0x7f74a8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.311+0000 7f74cb000640 1 --2- 172.21.15.69:0/273600432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 0x7f74a8003dd0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f74b400ba00 tx=0x7f74a8005830 comp rx=0 tx=0).stop 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.311+0000 7f74c9000640 1 -- 172.21.15.69:0/273600432 reap_dead start 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.311+0000 7f74cb000640 1 -- 172.21.15.69:0/273600432 shutdown_connections 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.311+0000 7f74cb000640 1 -- 172.21.15.69:0/273600432 >> 172.21.15.69:0/273600432 conn(0x7f74c40d0350 msgr2=0x7f74c40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.311+0000 7f74cb000640 1 -- 172.21.15.69:0/273600432 shutdown_connections 2024-04-24T08:12:59.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.311+0000 7f74cb000640 1 -- 172.21.15.69:0/273600432 wait complete. 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.313+0000 7f74cb000640 1 Processor -- start 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.313+0000 7f74cb000640 1 -- start start 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.314+0000 7f74cb000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 0x7f74c40b0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.314+0000 7f74cb000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c414d0f0 0x7f74c406dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.314+0000 7f74cb000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c406e230 0x7f74c415d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.314+0000 7f74cb000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f74c40cd2a0 con 0x7f74a80039b0 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.314+0000 7f74cb000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f74c4071d00 con 0x7f74c414d0f0 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.314+0000 7f74cb000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f74c40cd420 con 0x7f74c406e230 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.314+0000 7f74c3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 0x7f74c40b0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.314+0000 7f74c3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 0x7f74c40b0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37408/0 (socket says 172.21.15.69:37408) 2024-04-24T08:12:59.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.314+0000 7f74c3e00640 1 -- 172.21.15.69:0/4285487819 learned_addr learned my addr 172.21.15.69:0/4285487819 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:59.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.315+0000 7f74bb400640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c414d0f0 0x7f74c406dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.315+0000 7f74c9000640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c406e230 0x7f74c415d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.315+0000 7f74c3e00640 1 -- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c406e230 msgr2=0x7f74c415d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.315+0000 7f74c3e00640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c406e230 0x7f74c415d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.315+0000 7f74c3e00640 1 -- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c414d0f0 msgr2=0x7f74c406dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.315+0000 7f74c3e00640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c414d0f0 0x7f74c406dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.315+0000 7f74c3e00640 1 -- 172.21.15.69:0/4285487819 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c406fb10 con 0x7f74a80039b0 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.315+0000 7f74c3e00640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 0x7f74c40b0f00 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f74b400bb20 tx=0x7f74b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.315+0000 7f74c0c00640 1 -- 172.21.15.69:0/4285487819 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4043020 con 0x7f74a80039b0 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.316+0000 7f74c0c00640 1 -- 172.21.15.69:0/4285487819 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74b403f680 con 0x7f74a80039b0 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.316+0000 7f74c0c00640 1 -- 172.21.15.69:0/4285487819 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b400b600 con 0x7f74a80039b0 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.316+0000 7f74cb000640 1 -- 172.21.15.69:0/4285487819 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c414d570 con 0x7f74a80039b0 2024-04-24T08:12:59.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.316+0000 7f74cb000640 1 -- 172.21.15.69:0/4285487819 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c406f830 con 0x7f74a80039b0 2024-04-24T08:12:59.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.317+0000 7f74cb000640 1 -- 172.21.15.69:0/4285487819 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74c4068a80 con 0x7f74a80039b0 2024-04-24T08:12:59.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.318+0000 7f74c0c00640 1 -- 172.21.15.69:0/4285487819 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f74b4046020 con 0x7f74a80039b0 2024-04-24T08:12:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.318+0000 7f74c0c00640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74a006c920 0x7f74a006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.319+0000 7f74c0c00640 1 -- 172.21.15.69:0/4285487819 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f74b40ac780 con 0x7f74a80039b0 2024-04-24T08:12:59.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.319+0000 7f74bb400640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74a006c920 0x7f74a006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.321+0000 7f74bb400640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74a006c920 0x7f74a006ede0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f74ac006a90 tx=0x7f74ac008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:59.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.323+0000 7f74c0c00640 1 -- 172.21.15.69:0/4285487819 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74b404d030 con 0x7f74a80039b0 2024-04-24T08:12:59.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.529+0000 7f74cb000640 1 -- 172.21.15.69:0/4285487819 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f74c40cc240 con 0x7f74a80039b0 2024-04-24T08:12:59.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.530+0000 7f74c0c00640 1 -- 172.21.15.69:0/4285487819 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v57) ==== 104+0+11 (secure 0 0 0) 0x7f74b407aa80 con 0x7f74a80039b0 2024-04-24T08:12:59.532 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:12:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.533+0000 7f74cb000640 1 -- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74a006c920 msgr2=0x7f74a006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.533+0000 7f74cb000640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74a006c920 0x7f74a006ede0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f74ac006a90 tx=0x7f74ac008040 comp rx=0 tx=0).stop 2024-04-24T08:12:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.533+0000 7f74cb000640 1 -- 172.21.15.69:0/4285487819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 msgr2=0x7f74c40b0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.533+0000 7f74cb000640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 0x7f74c40b0f00 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f74b400bb20 tx=0x7f74b4004730 comp rx=0 tx=0).stop 2024-04-24T08:12:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.533+0000 7f74cb000640 1 -- 172.21.15.69:0/4285487819 shutdown_connections 2024-04-24T08:12:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.533+0000 7f74cb000640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74a006c920 0x7f74a006ede0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.533+0000 7f74cb000640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74c406e230 0x7f74c415d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.533+0000 7f74cb000640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c414d0f0 0x7f74c406dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.533+0000 7f74cb000640 1 --2- 172.21.15.69:0/4285487819 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74a80039b0 0x7f74c40b0f00 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.533+0000 7f74cb000640 1 -- 172.21.15.69:0/4285487819 >> 172.21.15.69:0/4285487819 conn(0x7f74c40d0350 msgr2=0x7f74c4074b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.534+0000 7f74cb000640 1 -- 172.21.15.69:0/4285487819 shutdown_connections 2024-04-24T08:12:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.534+0000 7f74cb000640 1 -- 172.21.15.69:0/4285487819 wait complete. 2024-04-24T08:12:59.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:12:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.757+0000 7f826f000640 1 Processor -- start 2024-04-24T08:12:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.757+0000 7f826f000640 1 -- start start 2024-04-24T08:12:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.759+0000 7f826f000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8268158f20 0x7f8268162710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.759+0000 7f826f000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8268163190 con 0x7f8268162c50 2024-04-24T08:12:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.759+0000 7f826f000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8268163470 con 0x7f826806d140 2024-04-24T08:12:59.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.759+0000 7f826f000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8268160460 con 0x7f8268158f20 2024-04-24T08:12:59.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.759+0000 7f8267400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8268162c50 0x7f826806ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56532/0 (socket says 172.21.15.69:56532) 2024-04-24T08:12:59.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.759+0000 7f8267e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8268158f20 0x7f8268162710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.759+0000 7f8267400640 1 -- 172.21.15.69:0/1803595603 learned_addr learned my addr 172.21.15.69:0/1803595603 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:12:59.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.760+0000 7f8267e00640 1 -- 172.21.15.69:0/1803595603 >> v1:172.21.15.121:6789/0 conn(0x7f826806d140 legacy=0x7f8268160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.760+0000 7f8267e00640 1 -- 172.21.15.69:0/1803595603 >> v1:172.21.15.69:6789/0 conn(0x7f8268162c50 legacy=0x7f826806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.760+0000 7f8267e00640 1 -- 172.21.15.69:0/1803595603 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8268160740 con 0x7f8268158f20 2024-04-24T08:12:59.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.760+0000 7f8266a00640 1 -- 172.21.15.69:0/1803595603 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2471538105 0 0) 0x7f8268163190 con 0x7f8268162c50 2024-04-24T08:12:59.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.761+0000 7f8267e00640 1 --2- 172.21.15.69:0/1803595603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 0x7f8268162710 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f82580097a0 tx=0x7f825802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5de3fce8cead7f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:59.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.761+0000 7f8266a00640 1 -- 172.21.15.69:0/1803595603 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8258037070 con 0x7f8268158f20 2024-04-24T08:12:59.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.762+0000 7f8266a00640 1 -- 172.21.15.69:0/1803595603 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8258032440 con 0x7f8268158f20 2024-04-24T08:12:59.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.762+0000 7f8266a00640 1 -- 172.21.15.69:0/1803595603 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825802ae60 con 0x7f8268158f20 2024-04-24T08:12:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.762+0000 7f826f000640 1 -- 172.21.15.69:0/1803595603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 msgr2=0x7f8268162710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.762+0000 7f826f000640 1 --2- 172.21.15.69:0/1803595603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 0x7f8268162710 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f82580097a0 tx=0x7f825802a110 comp rx=0 tx=0).stop 2024-04-24T08:12:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.762+0000 7f826f000640 1 -- 172.21.15.69:0/1803595603 shutdown_connections 2024-04-24T08:12:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.762+0000 7f826f000640 1 --2- 172.21.15.69:0/1803595603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 0x7f8268162710 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.762+0000 7f826f000640 1 -- 172.21.15.69:0/1803595603 >> 172.21.15.69:0/1803595603 conn(0x7f8268072510 msgr2=0x7f8268072920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.762+0000 7f826f000640 1 -- 172.21.15.69:0/1803595603 shutdown_connections 2024-04-24T08:12:59.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.762+0000 7f826f000640 1 -- 172.21.15.69:0/1803595603 wait complete. 2024-04-24T08:12:59.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.764+0000 7f826f000640 1 Processor -- start 2024-04-24T08:12:59.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.764+0000 7f826f000640 1 -- start start 2024-04-24T08:12:59.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.765+0000 7f826f000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f826806d140 0x7f826815e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.765+0000 7f826f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 0x7f8268157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.765+0000 7f826f000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8268162c50 0x7f8268157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.765+0000 7f826f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8268159480 con 0x7f8268158f20 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.765+0000 7f826f000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8268159300 con 0x7f8268162c50 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.765+0000 7f826f000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8268159600 con 0x7f826806d140 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.765+0000 7f8267400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 0x7f8268157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.765+0000 7f8267400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 0x7f8268157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37428/0 (socket says 172.21.15.69:37428) 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.765+0000 7f8267400640 1 -- 172.21.15.69:0/2583744085 learned_addr learned my addr 172.21.15.69:0/2583744085 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f826d000640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8268162c50 0x7f8268157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f8267e00640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f826806d140 0x7f826815e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f8267400640 1 -- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f826806d140 msgr2=0x7f826815e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f8267400640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f826806d140 0x7f826815e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f8267400640 1 -- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8268162c50 msgr2=0x7f8268157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f8267400640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8268162c50 0x7f8268157a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f8267400640 1 -- 172.21.15.69:0/2583744085 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826815ea60 con 0x7f8268158f20 2024-04-24T08:12:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f826d000640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8268162c50 0x7f8268157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f8267e00640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f826806d140 0x7f826815e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f8267400640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 0x7f8268157520 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f825400c460 tx=0x7f825400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f825fe00640 1 -- 172.21.15.69:0/2583744085 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254018070 con 0x7f8268158f20 2024-04-24T08:12:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f825fe00640 1 -- 172.21.15.69:0/2583744085 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8254007520 con 0x7f8268158f20 2024-04-24T08:12:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f825fe00640 1 -- 172.21.15.69:0/2583744085 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254004a20 con 0x7f8268158f20 2024-04-24T08:12:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f826f000640 1 -- 172.21.15.69:0/2583744085 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8268143000 con 0x7f8268158f20 2024-04-24T08:12:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.766+0000 7f826f000640 1 -- 172.21.15.69:0/2583744085 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82681435c0 con 0x7f8268158f20 2024-04-24T08:12:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.768+0000 7f826f000640 1 -- 172.21.15.69:0/2583744085 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8268068a80 con 0x7f8268158f20 2024-04-24T08:12:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.768+0000 7f825fe00640 1 -- 172.21.15.69:0/2583744085 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8254004bc0 con 0x7f8268158f20 2024-04-24T08:12:59.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.770+0000 7f825fe00640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824806c8f0 0x7f824806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:59.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.770+0000 7f825fe00640 1 -- 172.21.15.69:0/2583744085 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f825408add0 con 0x7f8268158f20 2024-04-24T08:12:59.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.771+0000 7f8267e00640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824806c8f0 0x7f824806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:59.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.772+0000 7f8267e00640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824806c8f0 0x7f824806edb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8258034000 tx=0x7f825802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:59.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.774+0000 7f825fe00640 1 -- 172.21.15.69:0/2583744085 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8254058ba0 con 0x7f8268158f20 2024-04-24T08:12:59.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.977+0000 7f826f000640 1 -- 172.21.15.69:0/2583744085 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8268067f40 con 0x7f8268158f20 2024-04-24T08:12:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.978+0000 7f825fe00640 1 -- 172.21.15.69:0/2583744085 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v57) ==== 100+0+11 (secure 0 0 0) 0x7f825405cea0 con 0x7f8268158f20 2024-04-24T08:12:59.979 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:12:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.980+0000 7f826f000640 1 -- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824806c8f0 msgr2=0x7f824806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.980+0000 7f826f000640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824806c8f0 0x7f824806edb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f8258034000 tx=0x7f825802e000 comp rx=0 tx=0).stop 2024-04-24T08:12:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.980+0000 7f826f000640 1 -- 172.21.15.69:0/2583744085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 msgr2=0x7f8268157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.980+0000 7f826f000640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 0x7f8268157520 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f825400c460 tx=0x7f825400c920 comp rx=0 tx=0).stop 2024-04-24T08:12:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.980+0000 7f826f000640 1 -- 172.21.15.69:0/2583744085 shutdown_connections 2024-04-24T08:12:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.980+0000 7f826f000640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f824806c8f0 0x7f824806edb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.980+0000 7f826f000640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8268158f20 0x7f8268157520 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.980+0000 7f826f000640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8268162c50 0x7f8268157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.980+0000 7f826f000640 1 --2- 172.21.15.69:0/2583744085 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f826806d140 0x7f826815e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.980+0000 7f826f000640 1 -- 172.21.15.69:0/2583744085 >> 172.21.15.69:0/2583744085 conn(0x7f8268072510 msgr2=0x7f82680b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.981+0000 7f826f000640 1 -- 172.21.15.69:0/2583744085 shutdown_connections 2024-04-24T08:12:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:12:59.981+0000 7f826f000640 1 -- 172.21.15.69:0/2583744085 wait complete. 2024-04-24T08:12:59.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:00.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.189+0000 7f6735a00640 1 Processor -- start 2024-04-24T08:13:00.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.190+0000 7f6735a00640 1 -- start start 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.191+0000 7f6735a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f673006db90 0x7f6730073530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.191+0000 7f6735a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f673015b980 0x7f673015be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.191+0000 7f6735a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6730073a70 con 0x7f67301523f0 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.191+0000 7f6735a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f67300aeac0 con 0x7f673006db90 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.191+0000 7f6735a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f67300aeda0 con 0x7f673015b980 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.191+0000 7f6727400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f673006db90 0x7f6730073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.191+0000 7f6727400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f673006db90 0x7f6730073530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37440/0 (socket says 172.21.15.69:37440) 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.191+0000 7f6727400640 1 -- 172.21.15.69:0/416316436 learned_addr learned my addr 172.21.15.69:0/416316436 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.191+0000 7f6735000640 1 --2- 172.21.15.69:0/416316436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f673015b980 0x7f673015be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.192+0000 7f6727400640 1 -- 172.21.15.69:0/416316436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f673015b980 msgr2=0x7f673015be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:00.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.192+0000 7f6727400640 1 --2- 172.21.15.69:0/416316436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f673015b980 0x7f673015be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.192+0000 7f6727400640 1 -- 172.21.15.69:0/416316436 >> v1:172.21.15.121:6789/0 conn(0x7f67301523f0 legacy=0x7f6730072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:00.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.192+0000 7f6727400640 1 -- 172.21.15.69:0/416316436 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67300af080 con 0x7f673006db90 2024-04-24T08:13:00.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.193+0000 7f6727400640 1 --2- 172.21.15.69:0/416316436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 0x7f6730073530 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f671800bb10 tx=0x7f671802c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c84a6537f7623b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:00.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.193+0000 7f672f400640 1 -- 172.21.15.69:0/416316436 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671803a070 con 0x7f673006db90 2024-04-24T08:13:00.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.193+0000 7f672f400640 1 -- 172.21.15.69:0/416316436 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67180326f0 con 0x7f673006db90 2024-04-24T08:13:00.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.193+0000 7f672f400640 1 -- 172.21.15.69:0/416316436 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718032ac0 con 0x7f673006db90 2024-04-24T08:13:00.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.194+0000 7f6735a00640 1 -- 172.21.15.69:0/416316436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 msgr2=0x7f6730073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.194+0000 7f6735a00640 1 --2- 172.21.15.69:0/416316436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 0x7f6730073530 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f671800bb10 tx=0x7f671802c830 comp rx=0 tx=0).stop 2024-04-24T08:13:00.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.194+0000 7f6735a00640 1 -- 172.21.15.69:0/416316436 shutdown_connections 2024-04-24T08:13:00.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.194+0000 7f6735a00640 1 --2- 172.21.15.69:0/416316436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f673015b980 0x7f673015be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.194+0000 7f6735a00640 1 --2- 172.21.15.69:0/416316436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 0x7f6730073530 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.194+0000 7f6735a00640 1 -- 172.21.15.69:0/416316436 >> 172.21.15.69:0/416316436 conn(0x7f67300d0160 msgr2=0x7f67300d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:00.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.194+0000 7f6735a00640 1 -- 172.21.15.69:0/416316436 shutdown_connections 2024-04-24T08:13:00.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.194+0000 7f6735a00640 1 -- 172.21.15.69:0/416316436 wait complete. 2024-04-24T08:13:00.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.196+0000 7f6735a00640 1 Processor -- start 2024-04-24T08:13:00.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.196+0000 7f6735a00640 1 -- start start 2024-04-24T08:13:00.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f6735a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 0x7f6730155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f6735a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67301523f0 0x7f6730156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f6735a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f673015b980 0x7f6730141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f6735a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f673006dfd0 con 0x7f673006db90 2024-04-24T08:13:00.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f6735a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f67300b3190 con 0x7f673015b980 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f6735a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f67301527d0 con 0x7f67301523f0 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f672fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 0x7f6730155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f672fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 0x7f6730155490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37450/0 (socket says 172.21.15.69:37450) 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f672fe00640 1 -- 172.21.15.69:0/2534875568 learned_addr learned my addr 172.21.15.69:0/2534875568 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f672fe00640 1 -- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67301523f0 msgr2=0x7f6730156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f672fe00640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67301523f0 0x7f6730156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f672fe00640 1 -- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f673015b980 msgr2=0x7f6730141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f672fe00640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f673015b980 0x7f6730141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.197+0000 7f672fe00640 1 -- 172.21.15.69:0/2534875568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6730156580 con 0x7f673006db90 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.198+0000 7f672fe00640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 0x7f6730155490 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f672000c7f0 tx=0x7f672000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:00.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.198+0000 7f672cc00640 1 -- 172.21.15.69:0/2534875568 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6720016020 con 0x7f673006db90 2024-04-24T08:13:00.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.198+0000 7f672cc00640 1 -- 172.21.15.69:0/2534875568 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f672000aa50 con 0x7f673006db90 2024-04-24T08:13:00.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.198+0000 7f6735a00640 1 -- 172.21.15.69:0/2534875568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6730141950 con 0x7f673006db90 2024-04-24T08:13:00.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.198+0000 7f672cc00640 1 -- 172.21.15.69:0/2534875568 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672000ae40 con 0x7f673006db90 2024-04-24T08:13:00.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.198+0000 7f6735a00640 1 -- 172.21.15.69:0/2534875568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6730156820 con 0x7f673006db90 2024-04-24T08:13:00.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.199+0000 7f6735a00640 1 -- 172.21.15.69:0/2534875568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f673014efe0 con 0x7f673006db90 2024-04-24T08:13:00.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.200+0000 7f672cc00640 1 -- 172.21.15.69:0/2534875568 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6720019020 con 0x7f673006db90 2024-04-24T08:13:00.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.201+0000 7f672cc00640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f670c06c8f0 0x7f670c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.201+0000 7f672cc00640 1 -- 172.21.15.69:0/2534875568 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f672008b810 con 0x7f673006db90 2024-04-24T08:13:00.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.202+0000 7f6727400640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f670c06c8f0 0x7f670c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:00.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.203+0000 7f6727400640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f670c06c8f0 0x7f670c06edb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6718037000 tx=0x7f6718008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:00.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.206+0000 7f672cc00640 1 -- 172.21.15.69:0/2534875568 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f673014efe0 con 0x7f673006db90 2024-04-24T08:13:00.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.450+0000 7f6735a00640 1 -- 172.21.15.69:0/2534875568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6730075d50 con 0x7f673006db90 2024-04-24T08:13:00.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.451+0000 7f672cc00640 1 -- 172.21.15.69:0/2534875568 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5188 (secure 0 0 0) 0x7f6720020070 con 0x7f673006db90 2024-04-24T08:13:00.452 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:00.452 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4034001365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4034001365},{"type":"v1","addr":"172.21.15.121:6835","nonce":4034001365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3907163219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3907163219},{"type":"v1","addr":"172.21.15.121:6837","nonce":3907163219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2756395689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2756395689},{"type":"v1","addr":"172.21.15.69:6834","nonce":2756395689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:11:39.824842+0000","modified":"2024-04-24T08:12:28.409867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.69:6835/2288057943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":2288057943},{"type":"v1","addr":"172.21.15.69:6835","nonce":2288057943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6231,"qdb_cluster":[6231]},"id":4}]} 2024-04-24T08:13:00.453 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 35 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.453+0000 7f6735a00640 1 -- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f670c06c8f0 msgr2=0x7f670c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.453+0000 7f6735a00640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f670c06c8f0 0x7f670c06edb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6718037000 tx=0x7f6718008040 comp rx=0 tx=0).stop 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.453+0000 7f6735a00640 1 -- 172.21.15.69:0/2534875568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 msgr2=0x7f6730155490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.453+0000 7f6735a00640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 0x7f6730155490 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f672000c7f0 tx=0x7f672000ccb0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.453+0000 7f6735a00640 1 -- 172.21.15.69:0/2534875568 shutdown_connections 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.453+0000 7f6735a00640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f670c06c8f0 0x7f670c06edb0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.453+0000 7f6735a00640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f673015b980 0x7f6730141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.453+0000 7f6735a00640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f67301523f0 0x7f6730156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.453+0000 7f6735a00640 1 --2- 172.21.15.69:0/2534875568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f673006db90 0x7f6730155490 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.453+0000 7f6735a00640 1 -- 172.21.15.69:0/2534875568 >> 172.21.15.69:0/2534875568 conn(0x7f67300d0160 msgr2=0x7f67300b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.454+0000 7f6735a00640 1 -- 172.21.15.69:0/2534875568 shutdown_connections 2024-04-24T08:13:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.454+0000 7f6735a00640 1 -- 172.21.15.69:0/2534875568 wait complete. 2024-04-24T08:13:00.473 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:00.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.663+0000 7febc3000640 1 Processor -- start 2024-04-24T08:13:00.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.663+0000 7febc3000640 1 -- start start 2024-04-24T08:13:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.664+0000 7febc3000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febbc145a90 0x7febbc06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.664+0000 7febc3000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7febbc145fe0 0x7febbc160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.664+0000 7febc3000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febbc0cf3d0 con 0x7febbc0d0300 2024-04-24T08:13:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.664+0000 7febc3000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7febbc0cf6b0 con 0x7febbc145a90 2024-04-24T08:13:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.664+0000 7febc3000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7febbc0cf990 con 0x7febbc145fe0 2024-04-24T08:13:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.665+0000 7febb3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febbc145a90 0x7febbc06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.665+0000 7febbbe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7febbc0d0300 0x7febbc0d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56542/0 (socket says 172.21.15.69:56542) 2024-04-24T08:13:00.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.665+0000 7febbbe00640 1 -- 172.21.15.69:0/3104642358 learned_addr learned my addr 172.21.15.69:0/3104642358 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.665+0000 7febb3e00640 1 -- 172.21.15.69:0/3104642358 >> v2:172.21.15.121:3300/0 conn(0x7febbc145fe0 msgr2=0x7febbc160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.665+0000 7febc1000640 1 --2- 172.21.15.69:0/3104642358 >> v2:172.21.15.121:3300/0 conn(0x7febbc145fe0 0x7febbc160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.665+0000 7febb3e00640 1 --2- 172.21.15.69:0/3104642358 >> v2:172.21.15.121:3300/0 conn(0x7febbc145fe0 0x7febbc160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.665+0000 7febb3e00640 1 -- 172.21.15.69:0/3104642358 >> v1:172.21.15.69:6789/0 conn(0x7febbc0d0300 legacy=0x7febbc0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.665+0000 7febb3e00640 1 -- 172.21.15.69:0/3104642358 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc0cfc70 con 0x7febbc145a90 2024-04-24T08:13:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.665+0000 7febbb400640 1 -- 172.21.15.69:0/3104642358 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2674665825 0 0) 0x7febbc0cf3d0 con 0x7febbc0d0300 2024-04-24T08:13:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.665+0000 7febc1000640 1 --2- 172.21.15.69:0/3104642358 >> v2:172.21.15.121:3300/0 conn(0x7febbc145fe0 0x7febbc160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:00.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.666+0000 7febb3e00640 1 --2- 172.21.15.69:0/3104642358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc145a90 0x7febbc06e240 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7febbc0d1050 tx=0x7feba402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359f02bbde5db8a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:00.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.666+0000 7febbb400640 1 -- 172.21.15.69:0/3104642358 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba4037070 con 0x7febbc145a90 2024-04-24T08:13:00.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.666+0000 7febbb400640 1 -- 172.21.15.69:0/3104642358 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feba4032440 con 0x7febbc145a90 2024-04-24T08:13:00.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.666+0000 7febbb400640 1 -- 172.21.15.69:0/3104642358 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feba4032810 con 0x7febbc145a90 2024-04-24T08:13:00.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.667+0000 7febc3000640 1 -- 172.21.15.69:0/3104642358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc145a90 msgr2=0x7febbc06e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.667+0000 7febc3000640 1 --2- 172.21.15.69:0/3104642358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc145a90 0x7febbc06e240 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7febbc0d1050 tx=0x7feba402a1a0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.667+0000 7febc3000640 1 -- 172.21.15.69:0/3104642358 shutdown_connections 2024-04-24T08:13:00.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.667+0000 7febc3000640 1 --2- 172.21.15.69:0/3104642358 >> v2:172.21.15.121:3300/0 conn(0x7febbc145fe0 0x7febbc160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.667+0000 7febc3000640 1 --2- 172.21.15.69:0/3104642358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc145a90 0x7febbc06e240 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.667+0000 7febc3000640 1 -- 172.21.15.69:0/3104642358 >> 172.21.15.69:0/3104642358 conn(0x7febbc067d90 msgr2=0x7febbc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:00.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.667+0000 7febc3000640 1 -- 172.21.15.69:0/3104642358 shutdown_connections 2024-04-24T08:13:00.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.667+0000 7febc3000640 1 -- 172.21.15.69:0/3104642358 wait complete. 2024-04-24T08:13:00.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.669+0000 7febc3000640 1 Processor -- start 2024-04-24T08:13:00.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.669+0000 7febc3000640 1 -- start start 2024-04-24T08:13:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febc3000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc0d0300 0x7febbc0ad740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febc3000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febbc145a90 0x7febbc1433c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febc3000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febbc145fe0 0x7febbc1f5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febc3000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7febbc1458f0 con 0x7febbc0d0300 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febc3000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7febbc145770 con 0x7febbc145a90 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febc3000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7febbc0cdc30 con 0x7febbc145fe0 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febbbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc0d0300 0x7febbc0ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febbbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc0d0300 0x7febbc0ad740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37472/0 (socket says 172.21.15.69:37472) 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febbbe00640 1 -- 172.21.15.69:0/1895922826 learned_addr learned my addr 172.21.15.69:0/1895922826 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febbbe00640 1 -- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febbc145fe0 msgr2=0x7febbc1f5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febbbe00640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febbc145fe0 0x7febbc1f5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febbbe00640 1 -- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febbc145a90 msgr2=0x7febbc1433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febbbe00640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febbc145a90 0x7febbc1433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febbbe00640 1 -- 172.21.15.69:0/1895922826 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febbc1f64f0 con 0x7febbc0d0300 2024-04-24T08:13:00.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.670+0000 7febbbe00640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc0d0300 0x7febbc0ad740 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7febac00c430 tx=0x7febac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.671+0000 7febb8c00640 1 -- 172.21.15.69:0/1895922826 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac0074e0 con 0x7febbc0d0300 2024-04-24T08:13:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.671+0000 7febb8c00640 1 -- 172.21.15.69:0/1895922826 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febac0047e0 con 0x7febbc0d0300 2024-04-24T08:13:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.671+0000 7febc3000640 1 -- 172.21.15.69:0/1895922826 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febbc1f67d0 con 0x7febbc0d0300 2024-04-24T08:13:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.671+0000 7febb8c00640 1 -- 172.21.15.69:0/1895922826 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febac004bd0 con 0x7febbc0d0300 2024-04-24T08:13:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.671+0000 7febc3000640 1 -- 172.21.15.69:0/1895922826 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febbc0a9920 con 0x7febbc0d0300 2024-04-24T08:13:00.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.672+0000 7febc3000640 1 -- 172.21.15.69:0/1895922826 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febbc068a80 con 0x7febbc0d0300 2024-04-24T08:13:00.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.673+0000 7febb8c00640 1 -- 172.21.15.69:0/1895922826 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febac011070 con 0x7febbc0d0300 2024-04-24T08:13:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.674+0000 7febb8c00640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feb9806c8f0 0x7feb9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.674+0000 7febb8c00640 1 -- 172.21.15.69:0/1895922826 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7febac08b8d0 con 0x7febbc0d0300 2024-04-24T08:13:00.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.675+0000 7febb3e00640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feb9806c8f0 0x7feb9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:00.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.676+0000 7febb3e00640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feb9806c8f0 0x7feb9806edb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7feba4034000 tx=0x7feba402c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:00.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.678+0000 7febb8c00640 1 -- 172.21.15.69:0/1895922826 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febac059c60 con 0x7febbc0d0300 2024-04-24T08:13:00.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.865+0000 7febc3000640 1 -- 172.21.15.69:0/1895922826 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7febbc0d1120 con 0x7febbc0d0300 2024-04-24T08:13:00.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.868+0000 7febb8c00640 1 -- 172.21.15.69:0/1895922826 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16641 (secure 0 0 0) 0x7febac059560 con 0x7febbc0d0300 2024-04-24T08:13:00.870 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:00.870 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":57,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:41.349322+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:11:37.517598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-24T08:11:38.791285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.870+0000 7febc3000640 1 -- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feb9806c8f0 msgr2=0x7feb9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.870+0000 7febc3000640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feb9806c8f0 0x7feb9806edb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7feba4034000 tx=0x7feba402c000 comp rx=0 tx=0).stop 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.870+0000 7febc3000640 1 -- 172.21.15.69:0/1895922826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc0d0300 msgr2=0x7febbc0ad740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.870+0000 7febc3000640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc0d0300 0x7febbc0ad740 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7febac00c430 tx=0x7febac00c8f0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.870+0000 7febc3000640 1 -- 172.21.15.69:0/1895922826 shutdown_connections 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.870+0000 7febc3000640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feb9806c8f0 0x7feb9806edb0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.870+0000 7febc3000640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febbc145fe0 0x7febbc1f5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.870+0000 7febc3000640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febbc145a90 0x7febbc1433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.870+0000 7febc3000640 1 --2- 172.21.15.69:0/1895922826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febbc0d0300 0x7febbc0ad740 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.870+0000 7febc3000640 1 -- 172.21.15.69:0/1895922826 >> 172.21.15.69:0/1895922826 conn(0x7febbc067d90 msgr2=0x7febbc15f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.871+0000 7febc3000640 1 -- 172.21.15.69:0/1895922826 shutdown_connections 2024-04-24T08:13:00.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:00.871+0000 7febc3000640 1 -- 172.21.15.69:0/1895922826 wait complete. 2024-04-24T08:13:00.890 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:13:00.890 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:01.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.079+0000 7f5d29600640 1 Processor -- start 2024-04-24T08:13:01.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.079+0000 7f5d29600640 1 -- start start 2024-04-24T08:13:01.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.081+0000 7f5d29600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5d240d0300 0x7f5d2406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.081+0000 7f5d29600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145fe0 0x7f5d24160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.081+0000 7f5d29600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d240cf3d0 con 0x7f5d24145a90 2024-04-24T08:13:01.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.081+0000 7f5d29600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5d240cf6b0 con 0x7f5d240d0300 2024-04-24T08:13:01.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.081+0000 7f5d29600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5d240cf990 con 0x7f5d24145fe0 2024-04-24T08:13:01.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.082+0000 7f5d23400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145fe0 0x7f5d24160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.082+0000 7f5d23400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145fe0 0x7f5d24160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:52596/0 (socket says 172.21.15.69:52596) 2024-04-24T08:13:01.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.082+0000 7f5d1be00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f5d24145a90 0x7f5d240d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:58840/0 (socket says 172.21.15.69:58840) 2024-04-24T08:13:01.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.082+0000 7f5d23400640 1 -- 172.21.15.69:0/827672957 learned_addr learned my addr 172.21.15.69:0/827672957 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:01.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.082+0000 7f5d22a00640 1 --2- 172.21.15.69:0/827672957 >> v2:172.21.15.121:3300/0 conn(0x7f5d240d0300 0x7f5d2406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.083+0000 7f5d23400640 1 -- 172.21.15.69:0/827672957 >> v2:172.21.15.121:3300/0 conn(0x7f5d240d0300 msgr2=0x7f5d2406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:01.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.083+0000 7f5d23400640 1 --2- 172.21.15.69:0/827672957 >> v2:172.21.15.121:3300/0 conn(0x7f5d240d0300 0x7f5d2406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.083+0000 7f5d23400640 1 -- 172.21.15.69:0/827672957 >> v1:172.21.15.121:6789/0 conn(0x7f5d24145a90 legacy=0x7f5d240d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.083+0000 7f5d23400640 1 -- 172.21.15.69:0/827672957 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d240cfc70 con 0x7f5d24145fe0 2024-04-24T08:13:01.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.083+0000 7f5d22000640 1 -- 172.21.15.69:0/827672957 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4008752856 0 0) 0x7f5d240cf3d0 con 0x7f5d24145a90 2024-04-24T08:13:01.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.085+0000 7f5d23400640 1 --2- 172.21.15.69:0/827672957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145fe0 0x7f5d24160540 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009e50 tx=0x7f5d1c02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=274b7362f7dbaf39 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:01.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.085+0000 7f5d22000640 1 -- 172.21.15.69:0/827672957 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c036070 con 0x7f5d24145fe0 2024-04-24T08:13:01.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.085+0000 7f5d22000640 1 -- 172.21.15.69:0/827672957 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d1c031600 con 0x7f5d24145fe0 2024-04-24T08:13:01.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.085+0000 7f5d22000640 1 -- 172.21.15.69:0/827672957 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c0319d0 con 0x7f5d24145fe0 2024-04-24T08:13:01.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.086+0000 7f5d29600640 1 -- 172.21.15.69:0/827672957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145fe0 msgr2=0x7f5d24160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.086+0000 7f5d29600640 1 --2- 172.21.15.69:0/827672957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145fe0 0x7f5d24160540 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c009e50 tx=0x7f5d1c02a710 comp rx=0 tx=0).stop 2024-04-24T08:13:01.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.086+0000 7f5d29600640 1 -- 172.21.15.69:0/827672957 shutdown_connections 2024-04-24T08:13:01.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.086+0000 7f5d29600640 1 --2- 172.21.15.69:0/827672957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145fe0 0x7f5d24160540 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.086+0000 7f5d29600640 1 --2- 172.21.15.69:0/827672957 >> v2:172.21.15.121:3300/0 conn(0x7f5d240d0300 0x7f5d2406e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.086+0000 7f5d29600640 1 -- 172.21.15.69:0/827672957 >> 172.21.15.69:0/827672957 conn(0x7f5d24067d90 msgr2=0x7f5d2406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:01.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.086+0000 7f5d29600640 1 -- 172.21.15.69:0/827672957 shutdown_connections 2024-04-24T08:13:01.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.086+0000 7f5d29600640 1 -- 172.21.15.69:0/827672957 wait complete. 2024-04-24T08:13:01.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.088+0000 7f5d29600640 1 Processor -- start 2024-04-24T08:13:01.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.088+0000 7f5d29600640 1 -- start start 2024-04-24T08:13:01.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.089+0000 7f5d29600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d240d0300 0x7f5d241f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.089+0000 7f5d29600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145a90 0x7f5d2413d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.089+0000 7f5d29600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d24145fe0 0x7f5d2413c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.089+0000 7f5d29600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5d241458f0 con 0x7f5d240d0300 2024-04-24T08:13:01.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.089+0000 7f5d29600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5d24145770 con 0x7f5d24145fe0 2024-04-24T08:13:01.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.089+0000 7f5d29600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5d240cdc30 con 0x7f5d24145a90 2024-04-24T08:13:01.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.089+0000 7f5d22a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d240d0300 0x7f5d241f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.089+0000 7f5d22a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d240d0300 0x7f5d241f2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37482/0 (socket says 172.21.15.69:37482) 2024-04-24T08:13:01.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.089+0000 7f5d22a00640 1 -- 172.21.15.69:0/2317746712 learned_addr learned my addr 172.21.15.69:0/2317746712 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d1be00640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145a90 0x7f5d2413d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d23400640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d24145fe0 0x7f5d2413c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d22a00640 1 -- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145a90 msgr2=0x7f5d2413d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d22a00640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145a90 0x7f5d2413d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d22a00640 1 -- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d24145fe0 msgr2=0x7f5d2413c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d22a00640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d24145fe0 0x7f5d2413c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d22a00640 1 -- 172.21.15.69:0/2317746712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2413ce00 con 0x7f5d240d0300 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d23400640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d24145fe0 0x7f5d2413c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d1be00640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145a90 0x7f5d2413d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d22a00640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d240d0300 0x7f5d241f2c40 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400e9b0 tx=0x7f5d1400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d1aa00640 1 -- 172.21.15.69:0/2317746712 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d14017070 con 0x7f5d240d0300 2024-04-24T08:13:01.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.090+0000 7f5d1aa00640 1 -- 172.21.15.69:0/2317746712 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d1400c9a0 con 0x7f5d240d0300 2024-04-24T08:13:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.091+0000 7f5d1aa00640 1 -- 172.21.15.69:0/2317746712 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1400cd90 con 0x7f5d240d0300 2024-04-24T08:13:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.091+0000 7f5d29600640 1 -- 172.21.15.69:0/2317746712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2413dd50 con 0x7f5d240d0300 2024-04-24T08:13:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.091+0000 7f5d29600640 1 -- 172.21.15.69:0/2317746712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2413e310 con 0x7f5d240d0300 2024-04-24T08:13:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.092+0000 7f5d29600640 1 -- 172.21.15.69:0/2317746712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d24068a80 con 0x7f5d240d0300 2024-04-24T08:13:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.093+0000 7f5d1aa00640 1 -- 172.21.15.69:0/2317746712 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d1401a070 con 0x7f5d240d0300 2024-04-24T08:13:01.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.093+0000 7f5d1aa00640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0006c920 0x7f5d0006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.094+0000 7f5d1aa00640 1 -- 172.21.15.69:0/2317746712 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f5d1408bb60 con 0x7f5d240d0300 2024-04-24T08:13:01.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.094+0000 7f5d1be00640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0006c920 0x7f5d0006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.095+0000 7f5d1be00640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0006c920 0x7f5d0006ede0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5d2413b660 tx=0x7f5d0c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:01.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.098+0000 7f5d1aa00640 1 -- 172.21.15.69:0/2317746712 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d14059f80 con 0x7f5d240d0300 2024-04-24T08:13:01.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.337+0000 7f5d29600640 1 -- 172.21.15.69:0/2317746712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d240d1120 con 0x7f5d240d0300 2024-04-24T08:13:01.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.338+0000 7f5d1aa00640 1 -- 172.21.15.69:0/2317746712 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5188 (secure 0 0 0) 0x7f5d14059880 con 0x7f5d240d0300 2024-04-24T08:13:01.340 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:01.340 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4034001365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4034001365},{"type":"v1","addr":"172.21.15.121:6835","nonce":4034001365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3907163219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3907163219},{"type":"v1","addr":"172.21.15.121:6837","nonce":3907163219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2756395689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2756395689},{"type":"v1","addr":"172.21.15.69:6834","nonce":2756395689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:11:39.824842+0000","modified":"2024-04-24T08:12:28.409867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.69:6835/2288057943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":2288057943},{"type":"v1","addr":"172.21.15.69:6835","nonce":2288057943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6231,"qdb_cluster":[6231]},"id":4}]} 2024-04-24T08:13:01.340 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 35 2024-04-24T08:13:01.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.340+0000 7f5d29600640 1 -- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0006c920 msgr2=0x7f5d0006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.340+0000 7f5d29600640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0006c920 0x7f5d0006ede0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5d2413b660 tx=0x7f5d0c009040 comp rx=0 tx=0).stop 2024-04-24T08:13:01.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.340+0000 7f5d29600640 1 -- 172.21.15.69:0/2317746712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d240d0300 msgr2=0x7f5d241f2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.340+0000 7f5d29600640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d240d0300 0x7f5d241f2c40 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f5d1400e9b0 tx=0x7f5d1400ee70 comp rx=0 tx=0).stop 2024-04-24T08:13:01.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.341+0000 7f5d29600640 1 -- 172.21.15.69:0/2317746712 shutdown_connections 2024-04-24T08:13:01.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.341+0000 7f5d29600640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0006c920 0x7f5d0006ede0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.341+0000 7f5d29600640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d24145fe0 0x7f5d2413c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.341+0000 7f5d29600640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d24145a90 0x7f5d2413d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.341+0000 7f5d29600640 1 --2- 172.21.15.69:0/2317746712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d240d0300 0x7f5d241f2c40 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.341+0000 7f5d29600640 1 -- 172.21.15.69:0/2317746712 >> 172.21.15.69:0/2317746712 conn(0x7f5d24067d90 msgr2=0x7f5d2406e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:01.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.341+0000 7f5d29600640 1 -- 172.21.15.69:0/2317746712 shutdown_connections 2024-04-24T08:13:01.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.341+0000 7f5d29600640 1 -- 172.21.15.69:0/2317746712 wait complete. 2024-04-24T08:13:01.361 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:01.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.558+0000 7f16d4a00640 1 Processor -- start 2024-04-24T08:13:01.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.558+0000 7f16d4a00640 1 -- start start 2024-04-24T08:13:01.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.559+0000 7f16d4a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f16d006f2d0 0x7f16d006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.559+0000 7f16d4a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f16d00cfbe0 0x7f16d00d0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.559+0000 7f16d4a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d00b46e0 con 0x7f16d015b490 2024-04-24T08:13:01.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.559+0000 7f16d4a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f16d006fc70 con 0x7f16d00cfbe0 2024-04-24T08:13:01.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.559+0000 7f16d4a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f16d00b5890 con 0x7f16d006f2d0 2024-04-24T08:13:01.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.560+0000 7f16ce800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f16d00cfbe0 0x7f16d00d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.560+0000 7f16ce800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f16d00cfbe0 0x7f16d00d0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37492/0 (socket says 172.21.15.69:37492) 2024-04-24T08:13:01.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.560+0000 7f16ce800640 1 -- 172.21.15.69:0/4185180393 learned_addr learned my addr 172.21.15.69:0/4185180393 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:01.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.560+0000 7f16ce800640 1 -- 172.21.15.69:0/4185180393 >> v2:172.21.15.121:3300/0 conn(0x7f16d006f2d0 msgr2=0x7f16d006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:01.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.560+0000 7f16ce800640 1 --2- 172.21.15.69:0/4185180393 >> v2:172.21.15.121:3300/0 conn(0x7f16d006f2d0 0x7f16d006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.560+0000 7f16ce800640 1 -- 172.21.15.69:0/4185180393 >> v1:172.21.15.121:6789/0 conn(0x7f16d015b490 legacy=0x7f16d00b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:01.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.560+0000 7f16ce800640 1 -- 172.21.15.69:0/4185180393 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d00d0960 con 0x7f16d00cfbe0 2024-04-24T08:13:01.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.561+0000 7f16ce800640 1 --2- 172.21.15.69:0/4185180393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 0x7f16d00d0010 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f16d00afb50 tx=0x7f16c802a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63fd3bdd7d0370e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:01.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16cca00640 1 -- 172.21.15.69:0/4185180393 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8037070 con 0x7f16d00cfbe0 2024-04-24T08:13:01.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16cca00640 1 -- 172.21.15.69:0/4185180393 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c8032440 con 0x7f16d00cfbe0 2024-04-24T08:13:01.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16cca00640 1 -- 172.21.15.69:0/4185180393 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c8032810 con 0x7f16d00cfbe0 2024-04-24T08:13:01.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16d4a00640 1 -- 172.21.15.69:0/4185180393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 msgr2=0x7f16d00d0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16d4a00640 1 --2- 172.21.15.69:0/4185180393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 0x7f16d00d0010 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f16d00afb50 tx=0x7f16c802a1f0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16d4a00640 1 -- 172.21.15.69:0/4185180393 shutdown_connections 2024-04-24T08:13:01.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16d4a00640 1 --2- 172.21.15.69:0/4185180393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 0x7f16d00d0010 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16d4a00640 1 --2- 172.21.15.69:0/4185180393 >> v2:172.21.15.121:3300/0 conn(0x7f16d006f2d0 0x7f16d006f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16d4a00640 1 -- 172.21.15.69:0/4185180393 >> 172.21.15.69:0/4185180393 conn(0x7f16d0067d90 msgr2=0x7f16d006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:01.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16d4a00640 1 -- 172.21.15.69:0/4185180393 shutdown_connections 2024-04-24T08:13:01.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.562+0000 7f16d4a00640 1 -- 172.21.15.69:0/4185180393 wait complete. 2024-04-24T08:13:01.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.564+0000 7f16d4a00640 1 Processor -- start 2024-04-24T08:13:01.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.564+0000 7f16d4a00640 1 -- start start 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.565+0000 7f16d4a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16d006f2d0 0x7f16d01e83a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.565+0000 7f16d4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 0x7f16d0145a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.565+0000 7f16d4a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16d015b490 0x7f16d0145f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.565+0000 7f16d4a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f16d00d0c40 con 0x7f16d00cfbe0 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.565+0000 7f16d4a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f16d00b4c60 con 0x7f16d015b490 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.565+0000 7f16d4a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f16d015b870 con 0x7f16d006f2d0 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.565+0000 7f16cd400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 0x7f16d0145a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.565+0000 7f16cd400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 0x7f16d0145a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37500/0 (socket says 172.21.15.69:37500) 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.565+0000 7f16cd400640 1 -- 172.21.15.69:0/2737353387 learned_addr learned my addr 172.21.15.69:0/2737353387 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.566+0000 7f16cde00640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16d006f2d0 0x7f16d01e83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.566+0000 7f16ce800640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16d015b490 0x7f16d0145f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.566+0000 7f16cd400640 1 -- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16d006f2d0 msgr2=0x7f16d01e83a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.566+0000 7f16cd400640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16d006f2d0 0x7f16d01e83a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.566+0000 7f16cd400640 1 -- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16d015b490 msgr2=0x7f16d0145f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.566+0000 7f16cd400640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16d015b490 0x7f16d0145f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.566+0000 7f16cd400640 1 -- 172.21.15.69:0/2737353387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d01f86d0 con 0x7f16d00cfbe0 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.566+0000 7f16ce800640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16d015b490 0x7f16d0145f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.566+0000 7f16cd400640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 0x7f16d0145a30 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00b4a0 tx=0x7f16bc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.566+0000 7f16c6000640 1 -- 172.21.15.69:0/2737353387 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc014070 con 0x7f16d00cfbe0 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.567+0000 7f16c6000640 1 -- 172.21.15.69:0/2737353387 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16bc010610 con 0x7f16d00cfbe0 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.567+0000 7f16d4a00640 1 -- 172.21.15.69:0/2737353387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d01552f0 con 0x7f16d00cfbe0 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.567+0000 7f16c6000640 1 -- 172.21.15.69:0/2737353387 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc010a00 con 0x7f16d00cfbe0 2024-04-24T08:13:01.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.567+0000 7f16d4a00640 1 -- 172.21.15.69:0/2737353387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d01f8c20 con 0x7f16d00cfbe0 2024-04-24T08:13:01.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.568+0000 7f16d4a00640 1 -- 172.21.15.69:0/2737353387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d00b5480 con 0x7f16d00cfbe0 2024-04-24T08:13:01.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.569+0000 7f16c6000640 1 -- 172.21.15.69:0/2737353387 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f16bc017070 con 0x7f16d00cfbe0 2024-04-24T08:13:01.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.570+0000 7f16c6000640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f16b006c8f0 0x7f16b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:01.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.570+0000 7f16c6000640 1 -- 172.21.15.69:0/2737353387 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f16bc08b640 con 0x7f16d00cfbe0 2024-04-24T08:13:01.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.571+0000 7f16cde00640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f16b006c8f0 0x7f16b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:01.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.572+0000 7f16cde00640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f16b006c8f0 0x7f16b006edb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009f20 tx=0x7f16c00094d0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:01.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.574+0000 7f16c6000640 1 -- 172.21.15.69:0/2737353387 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f16d00b5480 con 0x7f16d00cfbe0 2024-04-24T08:13:01.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.812+0000 7f16d4a00640 1 -- 172.21.15.69:0/2737353387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16d0158900 con 0x7f16d00cfbe0 2024-04-24T08:13:01.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.813+0000 7f16c6000640 1 -- 172.21.15.69:0/2737353387 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5188 (secure 0 0 0) 0x7f16bc0599d0 con 0x7f16d00cfbe0 2024-04-24T08:13:01.815 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:01.815 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4034001365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4034001365},{"type":"v1","addr":"172.21.15.121:6835","nonce":4034001365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3907163219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3907163219},{"type":"v1","addr":"172.21.15.121:6837","nonce":3907163219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2756395689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2756395689},{"type":"v1","addr":"172.21.15.69:6834","nonce":2756395689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:11:39.824842+0000","modified":"2024-04-24T08:12:28.409867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6231},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6231":{"gid":6231,"name":"c","rank":0,"incarnation":30,"state":"up:active","state_seq":15,"addr":"172.21.15.69:6835/2288057943","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":2288057943},{"type":"v1","addr":"172.21.15.69:6835","nonce":2288057943}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6231,"qdb_cluster":[6231]},"id":4}]} 2024-04-24T08:13:01.815 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 35 2024-04-24T08:13:01.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.815+0000 7f16d4a00640 1 -- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f16b006c8f0 msgr2=0x7f16b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.815+0000 7f16d4a00640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f16b006c8f0 0x7f16b006edb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f16c0009f20 tx=0x7f16c00094d0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.816+0000 7f16d4a00640 1 -- 172.21.15.69:0/2737353387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 msgr2=0x7f16d0145a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:01.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.816+0000 7f16d4a00640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 0x7f16d0145a30 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f16bc00b4a0 tx=0x7f16bc00b960 comp rx=0 tx=0).stop 2024-04-24T08:13:01.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.816+0000 7f16d4a00640 1 -- 172.21.15.69:0/2737353387 shutdown_connections 2024-04-24T08:13:01.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.816+0000 7f16d4a00640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f16b006c8f0 0x7f16b006edb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.816+0000 7f16d4a00640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f16d00cfbe0 0x7f16d0145a30 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.816+0000 7f16d4a00640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f16d015b490 0x7f16d0145f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.816+0000 7f16d4a00640 1 --2- 172.21.15.69:0/2737353387 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f16d006f2d0 0x7f16d01e83a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:01.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.816+0000 7f16d4a00640 1 -- 172.21.15.69:0/2737353387 >> 172.21.15.69:0/2737353387 conn(0x7f16d0067d90 msgr2=0x7f16d00b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:01.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.816+0000 7f16d4a00640 1 -- 172.21.15.69:0/2737353387 shutdown_connections 2024-04-24T08:13:01.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:01.816+0000 7f16d4a00640 1 -- 172.21.15.69:0/2737353387 wait complete. 2024-04-24T08:13:01.837 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:02.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.026+0000 7f2debc00640 1 Processor -- start 2024-04-24T08:13:02.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.026+0000 7f2debc00640 1 -- start start 2024-04-24T08:13:02.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.028+0000 7f2debc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2de415b490 0x7f2de40b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.028+0000 7f2debc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de40b46e0 con 0x7f2de40d02f0 2024-04-24T08:13:02.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.028+0000 7f2debc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de40d0960 con 0x7f2de406f2d0 2024-04-24T08:13:02.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.028+0000 7f2debc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2de40d0720 con 0x7f2de415b490 2024-04-24T08:13:02.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.028+0000 7f2de9200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2de415b490 0x7f2de40b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.028+0000 7f2de9c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2de40d02f0 0x7f2de40ce0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56554/0 (socket says 172.21.15.69:56554) 2024-04-24T08:13:02.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.028+0000 7f2de9c00640 1 -- 172.21.15.69:0/2980034522 learned_addr learned my addr 172.21.15.69:0/2980034522 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:02.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.028+0000 7f2de9200640 1 -- 172.21.15.69:0/2980034522 >> v1:172.21.15.121:6789/0 conn(0x7f2de406f2d0 legacy=0x7f2de40cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:02.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.029+0000 7f2de9200640 1 -- 172.21.15.69:0/2980034522 >> v1:172.21.15.69:6789/0 conn(0x7f2de40d02f0 legacy=0x7f2de40ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.029+0000 7f2de9200640 1 -- 172.21.15.69:0/2980034522 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de40b4980 con 0x7f2de415b490 2024-04-24T08:13:02.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.029+0000 7f2de9200640 1 --2- 172.21.15.69:0/2980034522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de415b490 0x7f2de40b5180 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2dd80097a0 tx=0x7f2dd802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4948ab01604c36 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:02.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.030+0000 7f2de3400640 1 -- 172.21.15.69:0/2980034522 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8037070 con 0x7f2de415b490 2024-04-24T08:13:02.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.030+0000 7f2de3400640 1 -- 172.21.15.69:0/2980034522 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd8032440 con 0x7f2de415b490 2024-04-24T08:13:02.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.030+0000 7f2de3400640 1 -- 172.21.15.69:0/2980034522 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8032810 con 0x7f2de415b490 2024-04-24T08:13:02.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.030+0000 7f2debc00640 1 -- 172.21.15.69:0/2980034522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de415b490 msgr2=0x7f2de40b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.030+0000 7f2debc00640 1 --2- 172.21.15.69:0/2980034522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de415b490 0x7f2de40b5180 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f2dd80097a0 tx=0x7f2dd802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.031+0000 7f2debc00640 1 -- 172.21.15.69:0/2980034522 shutdown_connections 2024-04-24T08:13:02.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.031+0000 7f2debc00640 1 --2- 172.21.15.69:0/2980034522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de415b490 0x7f2de40b5180 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.031+0000 7f2debc00640 1 -- 172.21.15.69:0/2980034522 >> 172.21.15.69:0/2980034522 conn(0x7f2de4067d90 msgr2=0x7f2de406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:02.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.031+0000 7f2debc00640 1 -- 172.21.15.69:0/2980034522 shutdown_connections 2024-04-24T08:13:02.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.031+0000 7f2debc00640 1 -- 172.21.15.69:0/2980034522 wait complete. 2024-04-24T08:13:02.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.032+0000 7f2debc00640 1 Processor -- start 2024-04-24T08:13:02.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.033+0000 7f2debc00640 1 -- start start 2024-04-24T08:13:02.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.033+0000 7f2debc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2de406f2d0 0x7f2de414ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.033+0000 7f2debc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de40d02f0 0x7f2de415da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.033+0000 7f2debc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2de415b490 0x7f2de415df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.033+0000 7f2debc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2de415b870 con 0x7f2de40d02f0 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2debc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2de40d0c40 con 0x7f2de415b490 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2debc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2de40b56c0 con 0x7f2de406f2d0 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2de3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de40d02f0 0x7f2de415da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2de3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de40d02f0 0x7f2de415da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37528/0 (socket says 172.21.15.69:37528) 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2de3e00640 1 -- 172.21.15.69:0/3895647011 learned_addr learned my addr 172.21.15.69:0/3895647011 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2de3e00640 1 -- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2de406f2d0 msgr2=0x7f2de414ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2de3e00640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2de406f2d0 0x7f2de414ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2de3e00640 1 -- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2de415b490 msgr2=0x7f2de415df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2de3e00640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2de415b490 0x7f2de415df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2de3e00640 1 -- 172.21.15.69:0/3895647011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de41e6d50 con 0x7f2de40d02f0 2024-04-24T08:13:02.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.034+0000 7f2de3e00640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de40d02f0 0x7f2de415da00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000c430 tx=0x7f2dd000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:02.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.035+0000 7f2de0c00640 1 -- 172.21.15.69:0/3895647011 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0011790 con 0x7f2de40d02f0 2024-04-24T08:13:02.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.035+0000 7f2de0c00640 1 -- 172.21.15.69:0/3895647011 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dd0017070 con 0x7f2de40d02f0 2024-04-24T08:13:02.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.035+0000 7f2de0c00640 1 -- 172.21.15.69:0/3895647011 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0011bb0 con 0x7f2de40d02f0 2024-04-24T08:13:02.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.035+0000 7f2debc00640 1 -- 172.21.15.69:0/3895647011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de41e7030 con 0x7f2de40d02f0 2024-04-24T08:13:02.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.035+0000 7f2debc00640 1 -- 172.21.15.69:0/3895647011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de41e75f0 con 0x7f2de40d02f0 2024-04-24T08:13:02.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.036+0000 7f2debc00640 1 -- 172.21.15.69:0/3895647011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de4068a80 con 0x7f2de40d02f0 2024-04-24T08:13:02.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.037+0000 7f2de0c00640 1 -- 172.21.15.69:0/3895647011 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2dd001c460 con 0x7f2de40d02f0 2024-04-24T08:13:02.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.038+0000 7f2de0c00640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2dc406c920 0x7f2dc406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.038+0000 7f2de0c00640 1 -- 172.21.15.69:0/3895647011 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f2dd008ba70 con 0x7f2de40d02f0 2024-04-24T08:13:02.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.039+0000 7f2de9200640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2dc406c920 0x7f2dc406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.040+0000 7f2de9200640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2dc406c920 0x7f2dc406ede0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8034000 tx=0x7f2dd802c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:02.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.042+0000 7f2de0c00640 1 -- 172.21.15.69:0/3895647011 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2dd0059e00 con 0x7f2de40d02f0 2024-04-24T08:13:02.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.242+0000 7f2debc00640 1 -- 172.21.15.69:0/3895647011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2de40b3b90 con 0x7f2de40d02f0 2024-04-24T08:13:02.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.244+0000 7f2de0c00640 1 -- 172.21.15.69:0/3895647011 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) ==== 74+0+16641 (secure 0 0 0) 0x7f2dd0059700 con 0x7f2de40d02f0 2024-04-24T08:13:02.246 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:02.246 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":57,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:11:41.349322+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:11:37.517598+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-04-24T08:11:38.791285+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 -- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2dc406c920 msgr2=0x7f2dc406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2dc406c920 0x7f2dc406ede0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8034000 tx=0x7f2dd802c000 comp rx=0 tx=0).stop 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 -- 172.21.15.69:0/3895647011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de40d02f0 msgr2=0x7f2de415da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de40d02f0 0x7f2de415da00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000c430 tx=0x7f2dd000c8f0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 -- 172.21.15.69:0/3895647011 shutdown_connections 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2dc406c920 0x7f2dc406ede0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2de40d02f0 0x7f2de415da00 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2de415b490 0x7f2de415df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 --2- 172.21.15.69:0/3895647011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2de406f2d0 0x7f2de414ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 -- 172.21.15.69:0/3895647011 >> 172.21.15.69:0/3895647011 conn(0x7f2de4067d90 msgr2=0x7f2de40b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 -- 172.21.15.69:0/3895647011 shutdown_connections 2024-04-24T08:13:02.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.247+0000 7f2debc00640 1 -- 172.21.15.69:0/3895647011 wait complete. 2024-04-24T08:13:02.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:13:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.461+0000 7f7dbee00640 1 Processor -- start 2024-04-24T08:13:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.461+0000 7f7dbee00640 1 -- start start 2024-04-24T08:13:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.463+0000 7f7dbee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7db80d0300 0x7f7db806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.463+0000 7f7dbee00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7db8145fe0 0x7f7db8160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.463+0000 7f7dbee00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db80cf3d0 con 0x7f7db8145a90 2024-04-24T08:13:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.463+0000 7f7dbee00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7db80cf6b0 con 0x7f7db8145fe0 2024-04-24T08:13:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.463+0000 7f7dbee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7db80cf990 con 0x7f7db80d0300 2024-04-24T08:13:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.464+0000 7f7dbce00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7db8145fe0 0x7f7db8160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.464+0000 7f7dbce00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7db8145fe0 0x7f7db8160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37544/0 (socket says 172.21.15.69:37544) 2024-04-24T08:13:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.464+0000 7f7dbce00640 1 -- 172.21.15.69:0/835037206 learned_addr learned my addr 172.21.15.69:0/835037206 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.464+0000 7f7db7e00640 1 --2- 172.21.15.69:0/835037206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7db80d0300 0x7f7db806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.464+0000 7f7dbce00640 1 -- 172.21.15.69:0/835037206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7db80d0300 msgr2=0x7f7db806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.464+0000 7f7dbce00640 1 --2- 172.21.15.69:0/835037206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7db80d0300 0x7f7db806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.464+0000 7f7dbce00640 1 -- 172.21.15.69:0/835037206 >> v1:172.21.15.121:6789/0 conn(0x7f7db8145a90 legacy=0x7f7db80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.464+0000 7f7dbce00640 1 -- 172.21.15.69:0/835037206 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db80cfc70 con 0x7f7db8145fe0 2024-04-24T08:13:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.465+0000 7f7dbce00640 1 --2- 172.21.15.69:0/835037206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db8145fe0 0x7f7db8160540 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f7db0009870 tx=0x7f7db002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccfa9d737f0700a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7db6a00640 1 -- 172.21.15.69:0/835037206 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0037070 con 0x7f7db8145fe0 2024-04-24T08:13:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7db6a00640 1 -- 172.21.15.69:0/835037206 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7db0032440 con 0x7f7db8145fe0 2024-04-24T08:13:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7db6a00640 1 -- 172.21.15.69:0/835037206 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0032810 con 0x7f7db8145fe0 2024-04-24T08:13:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7dbee00640 1 -- 172.21.15.69:0/835037206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db8145fe0 msgr2=0x7f7db8160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7dbee00640 1 --2- 172.21.15.69:0/835037206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db8145fe0 0x7f7db8160540 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f7db0009870 tx=0x7f7db002a270 comp rx=0 tx=0).stop 2024-04-24T08:13:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7dbee00640 1 -- 172.21.15.69:0/835037206 shutdown_connections 2024-04-24T08:13:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7dbee00640 1 --2- 172.21.15.69:0/835037206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db8145fe0 0x7f7db8160540 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7dbee00640 1 --2- 172.21.15.69:0/835037206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7db80d0300 0x7f7db806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7dbee00640 1 -- 172.21.15.69:0/835037206 >> 172.21.15.69:0/835037206 conn(0x7f7db8067d90 msgr2=0x7f7db806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7dbee00640 1 -- 172.21.15.69:0/835037206 shutdown_connections 2024-04-24T08:13:02.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.466+0000 7f7dbee00640 1 -- 172.21.15.69:0/835037206 wait complete. 2024-04-24T08:13:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.468+0000 7f7dbee00640 1 Processor -- start 2024-04-24T08:13:02.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.468+0000 7f7dbee00640 1 -- start start 2024-04-24T08:13:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.469+0000 7f7dbee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db80d0300 0x7f7db81e8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.469+0000 7f7dbee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7db8145a90 0x7f7db815ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.469+0000 7f7dbee00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db8145fe0 0x7f7db81e5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.469+0000 7f7dbee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7db81458f0 con 0x7f7db80d0300 2024-04-24T08:13:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.469+0000 7f7dbee00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7db8145770 con 0x7f7db8145fe0 2024-04-24T08:13:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.469+0000 7f7dbee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7db80cdc30 con 0x7f7db8145a90 2024-04-24T08:13:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.469+0000 7f7db7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db80d0300 0x7f7db81e8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.469+0000 7f7db7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db80d0300 0x7f7db81e8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37546/0 (socket says 172.21.15.69:37546) 2024-04-24T08:13:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.469+0000 7f7db7e00640 1 -- 172.21.15.69:0/3397515248 learned_addr learned my addr 172.21.15.69:0/3397515248 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:02.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7dbce00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db8145fe0 0x7f7db81e5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7db7400640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7db8145a90 0x7f7db815ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7db7e00640 1 -- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7db8145a90 msgr2=0x7f7db815ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7db7e00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7db8145a90 0x7f7db815ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7db7e00640 1 -- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db8145fe0 msgr2=0x7f7db81e5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7db7e00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db8145fe0 0x7f7db81e5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7db7e00640 1 -- 172.21.15.69:0/3397515248 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db8130eb0 con 0x7f7db80d0300 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7dbce00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db8145fe0 0x7f7db81e5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7db7e00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db80d0300 0x7f7db81e8600 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f7da000d5d0 tx=0x7f7da000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7dafe00640 1 -- 172.21.15.69:0/3397515248 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da00125f0 con 0x7f7db80d0300 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7dafe00640 1 -- 172.21.15.69:0/3397515248 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da0019070 con 0x7f7db80d0300 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7dafe00640 1 -- 172.21.15.69:0/3397515248 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da00129f0 con 0x7f7db80d0300 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.470+0000 7f7dbee00640 1 -- 172.21.15.69:0/3397515248 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db8131190 con 0x7f7db80d0300 2024-04-24T08:13:02.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.471+0000 7f7dbee00640 1 -- 172.21.15.69:0/3397515248 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db8131750 con 0x7f7db80d0300 2024-04-24T08:13:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.472+0000 7f7dbee00640 1 -- 172.21.15.69:0/3397515248 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db8068a80 con 0x7f7db80d0300 2024-04-24T08:13:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.472+0000 7f7dafe00640 1 -- 172.21.15.69:0/3397515248 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7da0012b90 con 0x7f7db80d0300 2024-04-24T08:13:02.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.473+0000 7f7dafe00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d9406c8f0 0x7f7d9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.473+0000 7f7dafe00640 1 -- 172.21.15.69:0/3397515248 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 5709+0+0 (secure 0 0 0) 0x7f7da008bfe0 con 0x7f7db80d0300 2024-04-24T08:13:02.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.474+0000 7f7db7400640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d9406c8f0 0x7f7d9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.475+0000 7f7db7400640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d9406c8f0 0x7f7d9406edb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7da8007fd0 tx=0x7f7da8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:02.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.478+0000 7f7dafe00640 1 -- 172.21.15.69:0/3397515248 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7da005a390 con 0x7f7db80d0300 2024-04-24T08:13:02.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.719+0000 7f7dbee00640 1 -- 172.21.15.69:0/3397515248 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7db80d1120 con 0x7f7db80d0300 2024-04-24T08:13:02.722 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:13:02.720+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:13:02.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.724+0000 7f7dafe00640 1 -- 172.21.15.69:0/3397515248 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) ==== 161+0+0 (secure 0 0 0) 0x7f7da0059c90 con 0x7f7db80d0300 2024-04-24T08:13:02.726 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 -- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d9406c8f0 msgr2=0x7f7d9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d9406c8f0 0x7f7d9406edb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7da8007fd0 tx=0x7f7da8009040 comp rx=0 tx=0).stop 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 -- 172.21.15.69:0/3397515248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db80d0300 msgr2=0x7f7db81e8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db80d0300 0x7f7db81e8600 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f7da000d5d0 tx=0x7f7da000da90 comp rx=0 tx=0).stop 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 -- 172.21.15.69:0/3397515248 shutdown_connections 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d9406c8f0 0x7f7d9406edb0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db8145fe0 0x7f7db81e5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7db8145a90 0x7f7db815ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 --2- 172.21.15.69:0/3397515248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7db80d0300 0x7f7db81e8600 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 -- 172.21.15.69:0/3397515248 >> 172.21.15.69:0/3397515248 conn(0x7f7db8067d90 msgr2=0x7f7db806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:02.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 -- 172.21.15.69:0/3397515248 shutdown_connections 2024-04-24T08:13:02.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.727+0000 7f7dbee00640 1 -- 172.21.15.69:0/3397515248 wait complete. 2024-04-24T08:13:02.745 INFO:tasks.ceph.mds.c.smithi069.stderr: -6855> 2024-04-24T08:11:41.305+0000 7f0431800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:13:02.752 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T08:13:02.752 INFO:tasks.ceph.mds.c.smithi069.stderr: -4944> 2024-04-24T08:11:50.908+0000 7f042c800640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={6384=0-4194304@1} caps={6384=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6384 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x563937a40b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-24T08:13:02.756 INFO:tasks.ceph.mds.c.smithi069.stderr: -2986> 2024-04-24T08:12:22.949+0000 7f042c800640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-04-24T08:11:49.389063+0000 1=1+0) caps={6384=pAsLsXsFs/-@2},l=6384(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x563937a40b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(9)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-24T08:13:02.809 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:13:02.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.930+0000 7fa29e800640 1 Processor -- start 2024-04-24T08:13:02.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.930+0000 7fa29e800640 1 -- start start 2024-04-24T08:13:02.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.931+0000 7fa29e800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa29806f780 0x7fa298162f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.931+0000 7fa29e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa29806ea50 0x7fa298072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.931+0000 7fa29e800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa29806eeb0 con 0x7fa2980c9df0 2024-04-24T08:13:02.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.932+0000 7fa29e800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa2980726c0 con 0x7fa29806f780 2024-04-24T08:13:02.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.932+0000 7fa29e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa2980728f0 con 0x7fa29806ea50 2024-04-24T08:13:02.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.932+0000 7fa29de00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa2980c9df0 0x7fa2980cfc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56578/0 (socket says 172.21.15.69:56578) 2024-04-24T08:13:02.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.932+0000 7fa29de00640 1 -- 172.21.15.69:0/3619376761 learned_addr learned my addr 172.21.15.69:0/3619376761 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:02.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.932+0000 7fa29d400640 1 --2- 172.21.15.69:0/3619376761 >> v2:172.21.15.121:3300/0 conn(0x7fa29806f780 0x7fa298162f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.932+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1191289876 0 0) 0x7fa29806eeb0 con 0x7fa2980c9df0 2024-04-24T08:13:02.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.932+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa280003520 con 0x7fa2980c9df0 2024-04-24T08:13:02.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.932+0000 7fa29ca00640 1 --2- 172.21.15.69:0/3619376761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa29806ea50 0x7fa298072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3024808901 0 0) 0x7fa280003520 con 0x7fa2980c9df0 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa29806ea50 msgr2=0x7fa298072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 --2- 172.21.15.69:0/3619376761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa29806ea50 0x7fa298072180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 >> v2:172.21.15.121:3300/0 conn(0x7fa29806f780 msgr2=0x7fa298162f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 --2- 172.21.15.69:0/3619376761 >> v2:172.21.15.121:3300/0 conn(0x7fa29806f780 0x7fa298162f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa298072bd0 con 0x7fa2980c9df0 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fa290001a10 con 0x7fa2980c9df0 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 >> v1:172.21.15.69:6789/0 conn(0x7fa2980c9df0 legacy=0x7fa2980cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 --2- 172.21.15.69:0/3619376761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2800039b0 0x7fa280003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 --2- 172.21.15.69:0/3619376761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 0x7fa280004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 --2- 172.21.15.69:0/3619376761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa29806ea50 0x7fa2800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa2980728f0 con 0x7fa280004800 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa28000ba20 con 0x7fa2800039b0 2024-04-24T08:13:02.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa28000bc20 con 0x7fa29806ea50 2024-04-24T08:13:02.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa29d400640 1 --2- 172.21.15.69:0/3619376761 >> v2:172.21.15.121:3300/0 conn(0x7fa29806f780 0x7fa298162f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:02.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa29ca00640 1 --2- 172.21.15.69:0/3619376761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 0x7fa280004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa29ca00640 1 -- 172.21.15.69:0/3619376761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa29806ea50 msgr2=0x7fa2800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:02.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.933+0000 7fa29ca00640 1 --2- 172.21.15.69:0/3619376761 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa29806ea50 0x7fa2800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.934+0000 7fa29ca00640 1 -- 172.21.15.69:0/3619376761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2800039b0 msgr2=0x7fa280003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:02.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.934+0000 7fa29ca00640 1 --2- 172.21.15.69:0/3619376761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2800039b0 0x7fa280003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.934+0000 7fa29ca00640 1 -- 172.21.15.69:0/3619376761 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa280003520 con 0x7fa280004800 2024-04-24T08:13:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.935+0000 7fa29ca00640 1 --2- 172.21.15.69:0/3619376761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 0x7fa280004c80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa28400ba00 tx=0x7fa280005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eff74328e4ec5ba server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.935+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa284030070 con 0x7fa280004800 2024-04-24T08:13:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.935+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa28402adb0 con 0x7fa280004800 2024-04-24T08:13:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.935+0000 7fa297e00640 1 -- 172.21.15.69:0/3619376761 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28400b600 con 0x7fa280004800 2024-04-24T08:13:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.935+0000 7fa29e800640 1 -- 172.21.15.69:0/3619376761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 msgr2=0x7fa280004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.935+0000 7fa29e800640 1 --2- 172.21.15.69:0/3619376761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 0x7fa280004c80 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa28400ba00 tx=0x7fa280005830 comp rx=0 tx=0).stop 2024-04-24T08:13:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.935+0000 7fa29de00640 1 -- 172.21.15.69:0/3619376761 reap_dead start 2024-04-24T08:13:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.936+0000 7fa29e800640 1 -- 172.21.15.69:0/3619376761 shutdown_connections 2024-04-24T08:13:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.936+0000 7fa29e800640 1 -- 172.21.15.69:0/3619376761 >> 172.21.15.69:0/3619376761 conn(0x7fa2980cd540 msgr2=0x7fa2980cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.936+0000 7fa29e800640 1 -- 172.21.15.69:0/3619376761 shutdown_connections 2024-04-24T08:13:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.936+0000 7fa29e800640 1 -- 172.21.15.69:0/3619376761 wait complete. 2024-04-24T08:13:02.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.937+0000 7fa29e800640 1 Processor -- start 2024-04-24T08:13:02.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.938+0000 7fa29e800640 1 -- start start 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.938+0000 7fa29e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 0x7fa2981461c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.938+0000 7fa29e800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa29815dc10 0x7fa29815e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.938+0000 7fa29e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2981f6850 0x7fa2981f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa29806fb60 con 0x7fa280004800 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29e800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa298073190 con 0x7fa29815dc10 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa2980cf640 con 0x7fa2981f6850 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 0x7fa2981461c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 0x7fa2981461c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37590/0 (socket says 172.21.15.69:37590) 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29d400640 1 -- 172.21.15.69:0/1327573283 learned_addr learned my addr 172.21.15.69:0/1327573283 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29d400640 1 -- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2981f6850 msgr2=0x7fa2981f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29d400640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2981f6850 0x7fa2981f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29d400640 1 -- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa29815dc10 msgr2=0x7fa29815e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29d400640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa29815dc10 0x7fa29815e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29d400640 1 -- 172.21.15.69:0/1327573283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2981f75c0 con 0x7fa280004800 2024-04-24T08:13:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.939+0000 7fa29d400640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 0x7fa2981461c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00e810 tx=0x7fa28c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:02.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.940+0000 7fa295600640 1 -- 172.21.15.69:0/1327573283 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c00f040 con 0x7fa280004800 2024-04-24T08:13:02.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.940+0000 7fa295600640 1 -- 172.21.15.69:0/1327573283 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa28c004230 con 0x7fa280004800 2024-04-24T08:13:02.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.940+0000 7fa295600640 1 -- 172.21.15.69:0/1327573283 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa28c00c6d0 con 0x7fa280004800 2024-04-24T08:13:02.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.940+0000 7fa29e800640 1 -- 172.21.15.69:0/1327573283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2981f78a0 con 0x7fa280004800 2024-04-24T08:13:02.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.940+0000 7fa29e800640 1 -- 172.21.15.69:0/1327573283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa298141e40 con 0x7fa280004800 2024-04-24T08:13:02.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.941+0000 7fa29e800640 1 -- 172.21.15.69:0/1327573283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2981482c0 con 0x7fa280004800 2024-04-24T08:13:02.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.942+0000 7fa295600640 1 -- 172.21.15.69:0/1327573283 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa28c00c870 con 0x7fa280004800 2024-04-24T08:13:02.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.943+0000 7fa295600640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa27806c8f0 0x7fa27806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:02.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.943+0000 7fa295600640 1 -- 172.21.15.69:0/1327573283 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7fa28c08bab0 con 0x7fa280004800 2024-04-24T08:13:02.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.943+0000 7fa29ca00640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa27806c8f0 0x7fa27806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:02.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.945+0000 7fa29ca00640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa27806c8f0 0x7fa27806edb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa2800057e0 tx=0x7fa28402c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:02.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:02.947+0000 7fa295600640 1 -- 172.21.15.69:0/1327573283 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa2981482c0 con 0x7fa280004800 2024-04-24T08:13:03.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.187+0000 7fa29e800640 1 -- 172.21.15.69:0/1327573283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fa298075d50 con 0x7fa280004800 2024-04-24T08:13:03.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.735+0000 7fa295600640 1 -- 172.21.15.69:0/1327573283 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) ==== 104+0+0 (secure 0 0 0) 0x7fa28c017020 con 0x7fa280004800 2024-04-24T08:13:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 -- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa27806c8f0 msgr2=0x7fa27806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa27806c8f0 0x7fa27806edb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa2800057e0 tx=0x7fa28402c000 comp rx=0 tx=0).stop 2024-04-24T08:13:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 -- 172.21.15.69:0/1327573283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 msgr2=0x7fa2981461c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 0x7fa2981461c0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa28c00e810 tx=0x7fa28c00ecd0 comp rx=0 tx=0).stop 2024-04-24T08:13:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 -- 172.21.15.69:0/1327573283 shutdown_connections 2024-04-24T08:13:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa27806c8f0 0x7fa27806edb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2981f6850 0x7fa2981f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa29815dc10 0x7fa29815e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:03.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 --2- 172.21.15.69:0/1327573283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa280004800 0x7fa2981461c0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:03.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 -- 172.21.15.69:0/1327573283 >> 172.21.15.69:0/1327573283 conn(0x7fa2980cd540 msgr2=0x7fa29813fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:03.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.738+0000 7fa29e800640 1 -- 172.21.15.69:0/1327573283 shutdown_connections 2024-04-24T08:13:03.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.739+0000 7fa29e800640 1 -- 172.21.15.69:0/1327573283 wait complete. 2024-04-24T08:13:03.761 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T08:13:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.961+0000 7f912a600640 1 Processor -- start 2024-04-24T08:13:03.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.961+0000 7f912a600640 1 -- start start 2024-04-24T08:13:03.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.962+0000 7f912a600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91240cd940 0x7f91240b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:03.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.962+0000 7f912a600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91240ccc40 con 0x7f912415c870 2024-04-24T08:13:03.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.962+0000 7f912a600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91240cc960 con 0x7f91240b4980 2024-04-24T08:13:03.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.962+0000 7f912a600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f91240b44f0 con 0x7f91240cd940 2024-04-24T08:13:03.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.962+0000 7f9123400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f912415c870 0x7f91240cdf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56580/0 (socket says 172.21.15.69:56580) 2024-04-24T08:13:03.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.962+0000 7f9123400640 1 -- 172.21.15.69:0/1053256874 learned_addr learned my addr 172.21.15.69:0/1053256874 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:03.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.963+0000 7f9123e00640 1 --2- 172.21.15.69:0/1053256874 >> v2:172.21.15.69:3300/0 conn(0x7f91240cd940 0x7f91240b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:03.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.963+0000 7f9122a00640 1 -- 172.21.15.69:0/1053256874 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 88546951 0 0) 0x7f91240ccc40 con 0x7f912415c870 2024-04-24T08:13:03.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.963+0000 7f9122a00640 1 -- 172.21.15.69:0/1053256874 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9108003520 con 0x7f912415c870 2024-04-24T08:13:03.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.963+0000 7f9123e00640 1 -- 172.21.15.69:0/1053256874 >> v1:172.21.15.121:6789/0 conn(0x7f91240b4980 legacy=0x7f91240cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:03.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.963+0000 7f9123e00640 1 -- 172.21.15.69:0/1053256874 >> v1:172.21.15.69:6789/0 conn(0x7f912415c870 legacy=0x7f91240cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:03.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.963+0000 7f9123e00640 1 -- 172.21.15.69:0/1053256874 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91240b3ea0 con 0x7f91240cd940 2024-04-24T08:13:03.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.964+0000 7f9123e00640 1 --2- 172.21.15.69:0/1053256874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240cd940 0x7f91240b4e10 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f911c009dc0 tx=0x7f911c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ad4e74e9404cbf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:03.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.964+0000 7f9122a00640 1 -- 172.21.15.69:0/1053256874 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911c007740 con 0x7f91240cd940 2024-04-24T08:13:03.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.964+0000 7f9122a00640 1 -- 172.21.15.69:0/1053256874 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f911c0078e0 con 0x7f91240cd940 2024-04-24T08:13:03.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.964+0000 7f9122a00640 1 -- 172.21.15.69:0/1053256874 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911c007cb0 con 0x7f91240cd940 2024-04-24T08:13:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.965+0000 7f912a600640 1 -- 172.21.15.69:0/1053256874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240cd940 msgr2=0x7f91240b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.965+0000 7f912a600640 1 --2- 172.21.15.69:0/1053256874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240cd940 0x7f91240b4e10 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f911c009dc0 tx=0x7f911c02a730 comp rx=0 tx=0).stop 2024-04-24T08:13:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.965+0000 7f912a600640 1 -- 172.21.15.69:0/1053256874 shutdown_connections 2024-04-24T08:13:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.965+0000 7f912a600640 1 --2- 172.21.15.69:0/1053256874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240cd940 0x7f91240b4e10 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.965+0000 7f912a600640 1 -- 172.21.15.69:0/1053256874 >> 172.21.15.69:0/1053256874 conn(0x7f91240b2380 msgr2=0x7f91240b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.965+0000 7f912a600640 1 -- 172.21.15.69:0/1053256874 shutdown_connections 2024-04-24T08:13:03.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.965+0000 7f912a600640 1 -- 172.21.15.69:0/1053256874 wait complete. 2024-04-24T08:13:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.967+0000 7f912a600640 1 Processor -- start 2024-04-24T08:13:03.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.967+0000 7f912a600640 1 -- start start 2024-04-24T08:13:03.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.968+0000 7f912a600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240b4980 0x7f912406dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:03.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.968+0000 7f912a600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91240cd940 0x7f91240c2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:03.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.968+0000 7f912a600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f912415c870 0x7f91240c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:03.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.968+0000 7f912a600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f91240cbbf0 con 0x7f91240b4980 2024-04-24T08:13:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.968+0000 7f912a600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f91240ce670 con 0x7f91240cd940 2024-04-24T08:13:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.968+0000 7f912a600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f91240cbd70 con 0x7f912415c870 2024-04-24T08:13:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.968+0000 7f9123400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240b4980 0x7f912406dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.968+0000 7f9123400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240b4980 0x7f912406dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37612/0 (socket says 172.21.15.69:37612) 2024-04-24T08:13:03.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.968+0000 7f9123400640 1 -- 172.21.15.69:0/4031467174 learned_addr learned my addr 172.21.15.69:0/4031467174 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f9123400640 1 -- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f912415c870 msgr2=0x7f91240c48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f9123400640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f912415c870 0x7f91240c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f9123400640 1 -- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91240cd940 msgr2=0x7f91240c2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f9123400640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91240cd940 0x7f91240c2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f9123400640 1 -- 172.21.15.69:0/4031467174 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9124157c40 con 0x7f91240b4980 2024-04-24T08:13:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f9123400640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240b4980 0x7f912406dac0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f911400c410 tx=0x7f911400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f911b400640 1 -- 172.21.15.69:0/4031467174 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114017070 con 0x7f91240b4980 2024-04-24T08:13:03.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f911b400640 1 -- 172.21.15.69:0/4031467174 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9114005ce0 con 0x7f91240b4980 2024-04-24T08:13:03.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f912a600640 1 -- 172.21.15.69:0/4031467174 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91240c8700 con 0x7f91240b4980 2024-04-24T08:13:03.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f911b400640 1 -- 172.21.15.69:0/4031467174 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91140072f0 con 0x7f91240b4980 2024-04-24T08:13:03.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.969+0000 7f912a600640 1 -- 172.21.15.69:0/4031467174 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91240c8cc0 con 0x7f91240b4980 2024-04-24T08:13:03.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.970+0000 7f912a600640 1 -- 172.21.15.69:0/4031467174 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91240682f0 con 0x7f91240b4980 2024-04-24T08:13:03.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.971+0000 7f911b400640 1 -- 172.21.15.69:0/4031467174 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f911401a070 con 0x7f91240b4980 2024-04-24T08:13:03.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.973+0000 7f911b400640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f910006c8f0 0x7f910006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:03.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.973+0000 7f911b400640 1 -- 172.21.15.69:0/4031467174 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 5795+0+0 (secure 0 0 0) 0x7f911408aff0 con 0x7f91240b4980 2024-04-24T08:13:03.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.973+0000 7f911be00640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f910006c8f0 0x7f910006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:03.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.974+0000 7f911be00640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f910006c8f0 0x7f910006edb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f910c000f60 tx=0x7f910c0079d0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:03.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:03.977+0000 7f911b400640 1 -- 172.21.15.69:0/4031467174 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91240682f0 con 0x7f91240b4980 2024-04-24T08:13:04.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.173+0000 7f912a600640 1 -- 172.21.15.69:0/4031467174 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9124068a80 con 0x7f91240b4980 2024-04-24T08:13:04.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.943+0000 7f911b400640 1 -- 172.21.15.69:0/4031467174 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v59) ==== 181+0+0 (secure 0 0 0) 0x7f9124157c40 con 0x7f91240b4980 2024-04-24T08:13:04.945 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' removed 2024-04-24T08:13:04.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.947+0000 7f912a600640 1 -- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f910006c8f0 msgr2=0x7f910006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:04.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.947+0000 7f912a600640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f910006c8f0 0x7f910006edb0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f910c000f60 tx=0x7f910c0079d0 comp rx=0 tx=0).stop 2024-04-24T08:13:04.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.947+0000 7f912a600640 1 -- 172.21.15.69:0/4031467174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240b4980 msgr2=0x7f912406dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:04.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.947+0000 7f912a600640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240b4980 0x7f912406dac0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f911400c410 tx=0x7f911400c8d0 comp rx=0 tx=0).stop 2024-04-24T08:13:04.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.948+0000 7f912a600640 1 -- 172.21.15.69:0/4031467174 shutdown_connections 2024-04-24T08:13:04.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.948+0000 7f912a600640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f910006c8f0 0x7f910006edb0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:04.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.948+0000 7f912a600640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f912415c870 0x7f91240c48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:04.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.948+0000 7f912a600640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91240cd940 0x7f91240c2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:04.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.948+0000 7f912a600640 1 --2- 172.21.15.69:0/4031467174 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91240b4980 0x7f912406dac0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:04.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.948+0000 7f912a600640 1 -- 172.21.15.69:0/4031467174 >> 172.21.15.69:0/4031467174 conn(0x7f91240b2380 msgr2=0x7f91240b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:04.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.948+0000 7f912a600640 1 -- 172.21.15.69:0/4031467174 shutdown_connections 2024-04-24T08:13:04.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:04.949+0000 7f912a600640 1 -- 172.21.15.69:0/4031467174 wait complete. 2024-04-24T08:13:04.974 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T08:13:05.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.159+0000 7fb622800640 1 Processor -- start 2024-04-24T08:13:05.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.160+0000 7fb622800640 1 -- start start 2024-04-24T08:13:05.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.161+0000 7fb622800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb61c0d1610 0x7fb61c0d2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.161+0000 7fb622800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb61c0d3320 con 0x7fb61c0d4ef0 2024-04-24T08:13:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.161+0000 7fb622800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb61c0b74b0 con 0x7fb61c161dc0 2024-04-24T08:13:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.161+0000 7fb622800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb61c0d30e0 con 0x7fb61c0d1610 2024-04-24T08:13:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.162+0000 7fb620a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb61c0d1610 0x7fb61c0d2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:05.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.162+0000 7fb621e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb61c0d4ef0 0x7fb61c0b7f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56594/0 (socket says 172.21.15.69:56594) 2024-04-24T08:13:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.162+0000 7fb621e00640 1 -- 172.21.15.69:0/2320277032 learned_addr learned my addr 172.21.15.69:0/2320277032 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.162+0000 7fb620a00640 1 -- 172.21.15.69:0/2320277032 >> v1:172.21.15.121:6789/0 conn(0x7fb61c161dc0 legacy=0x7fb61c0d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.162+0000 7fb620a00640 1 -- 172.21.15.69:0/2320277032 >> v1:172.21.15.69:6789/0 conn(0x7fb61c0d4ef0 legacy=0x7fb61c0b7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.162+0000 7fb620a00640 1 -- 172.21.15.69:0/2320277032 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb61c0b7790 con 0x7fb61c0d1610 2024-04-24T08:13:05.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.163+0000 7fb620a00640 1 --2- 172.21.15.69:0/2320277032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c0d1610 0x7fb61c0d2ba0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb608009d90 tx=0x7fb60802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188a3fb2568f1bec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:05.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.163+0000 7fb61be00640 1 -- 172.21.15.69:0/2320277032 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608007740 con 0x7fb61c0d1610 2024-04-24T08:13:05.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.163+0000 7fb61be00640 1 -- 172.21.15.69:0/2320277032 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6080078e0 con 0x7fb61c0d1610 2024-04-24T08:13:05.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.163+0000 7fb61be00640 1 -- 172.21.15.69:0/2320277032 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608007cb0 con 0x7fb61c0d1610 2024-04-24T08:13:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.164+0000 7fb622800640 1 -- 172.21.15.69:0/2320277032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c0d1610 msgr2=0x7fb61c0d2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.164+0000 7fb622800640 1 --2- 172.21.15.69:0/2320277032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c0d1610 0x7fb61c0d2ba0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb608009d90 tx=0x7fb60802a6f0 comp rx=0 tx=0).stop 2024-04-24T08:13:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.164+0000 7fb622800640 1 -- 172.21.15.69:0/2320277032 shutdown_connections 2024-04-24T08:13:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.164+0000 7fb622800640 1 --2- 172.21.15.69:0/2320277032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c0d1610 0x7fb61c0d2ba0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.164+0000 7fb622800640 1 -- 172.21.15.69:0/2320277032 >> 172.21.15.69:0/2320277032 conn(0x7fb61c08bf60 msgr2=0x7fb61c0d45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:05.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.164+0000 7fb622800640 1 -- 172.21.15.69:0/2320277032 shutdown_connections 2024-04-24T08:13:05.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.164+0000 7fb622800640 1 -- 172.21.15.69:0/2320277032 wait complete. 2024-04-24T08:13:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.166+0000 7fb622800640 1 Processor -- start 2024-04-24T08:13:05.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.166+0000 7fb622800640 1 -- start start 2024-04-24T08:13:05.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.167+0000 7fb622800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb61c0d1610 0x7fb61c14d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.167+0000 7fb622800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb61c0d4ef0 0x7fb61c0cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb622800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c161dc0 0x7fb61c0cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb622800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c1621a0 con 0x7fb61c161dc0 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb622800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb61c0d3600 con 0x7fb61c0d4ef0 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb622800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb61c162320 con 0x7fb61c0d1610 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb621e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c161dc0 0x7fb61c0cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb621e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c161dc0 0x7fb61c0cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37634/0 (socket says 172.21.15.69:37634) 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb621e00640 1 -- 172.21.15.69:0/4103424723 learned_addr learned my addr 172.21.15.69:0/4103424723 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb621400640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb61c0d1610 0x7fb61c14d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb620a00640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb61c0d4ef0 0x7fb61c0cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb621e00640 1 -- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb61c0d1610 msgr2=0x7fb61c14d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb621e00640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb61c0d1610 0x7fb61c14d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:05.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb621e00640 1 -- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb61c0d4ef0 msgr2=0x7fb61c0cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:05.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb621e00640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb61c0d4ef0 0x7fb61c0cbb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:05.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb621e00640 1 -- 172.21.15.69:0/4103424723 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb61c1f51a0 con 0x7fb61c161dc0 2024-04-24T08:13:05.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb621400640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb61c0d1610 0x7fb61c14d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:05.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.168+0000 7fb620a00640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb61c0d4ef0 0x7fb61c0cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:05.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.169+0000 7fb621e00640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c161dc0 0x7fb61c0cc070 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c400 tx=0x7fb61400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:05.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.169+0000 7fb619600640 1 -- 172.21.15.69:0/4103424723 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614017070 con 0x7fb61c161dc0 2024-04-24T08:13:05.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.169+0000 7fb619600640 1 -- 172.21.15.69:0/4103424723 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb614005ce0 con 0x7fb61c161dc0 2024-04-24T08:13:05.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.169+0000 7fb622800640 1 -- 172.21.15.69:0/4103424723 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb61c0cc6a0 con 0x7fb61c161dc0 2024-04-24T08:13:05.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.169+0000 7fb619600640 1 -- 172.21.15.69:0/4103424723 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb614007310 con 0x7fb61c161dc0 2024-04-24T08:13:05.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.169+0000 7fb622800640 1 -- 172.21.15.69:0/4103424723 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb61c0ccc60 con 0x7fb61c161dc0 2024-04-24T08:13:05.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.170+0000 7fb622800640 1 -- 172.21.15.69:0/4103424723 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb61c068a80 con 0x7fb61c161dc0 2024-04-24T08:13:05.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.171+0000 7fb619600640 1 -- 172.21.15.69:0/4103424723 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb61401a070 con 0x7fb61c161dc0 2024-04-24T08:13:05.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.172+0000 7fb619600640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5fc06c8f0 0x7fb5fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:05.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.172+0000 7fb619600640 1 -- 172.21.15.69:0/4103424723 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 5357+0+0 (secure 0 0 0) 0x7fb61408ae40 con 0x7fb61c161dc0 2024-04-24T08:13:05.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.173+0000 7fb621400640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5fc06c8f0 0x7fb5fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:05.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.174+0000 7fb621400640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5fc06c8f0 0x7fb5fc06edb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb60c000f60 tx=0x7fb60c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:05.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.177+0000 7fb619600640 1 -- 172.21.15.69:0/4103424723 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb614059330 con 0x7fb61c161dc0 2024-04-24T08:13:05.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.388+0000 7fb622800640 1 -- 172.21.15.69:0/4103424723 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb61c0cd370 con 0x7fb61c161dc0 2024-04-24T08:13:05.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.955+0000 7fb619600640 1 -- 172.21.15.69:0/4103424723 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v60) ==== 169+0+0 (secure 0 0 0) 0x7fb614058c30 con 0x7fb61c161dc0 2024-04-24T08:13:05.958 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' removed 2024-04-24T08:13:05.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.958+0000 7fb622800640 1 -- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5fc06c8f0 msgr2=0x7fb5fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:05.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.958+0000 7fb622800640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5fc06c8f0 0x7fb5fc06edb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb60c000f60 tx=0x7fb60c009040 comp rx=0 tx=0).stop 2024-04-24T08:13:05.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.958+0000 7fb622800640 1 -- 172.21.15.69:0/4103424723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c161dc0 msgr2=0x7fb61c0cc070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:05.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.958+0000 7fb622800640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c161dc0 0x7fb61c0cc070 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c400 tx=0x7fb61400c8c0 comp rx=0 tx=0).stop 2024-04-24T08:13:05.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.959+0000 7fb622800640 1 -- 172.21.15.69:0/4103424723 shutdown_connections 2024-04-24T08:13:05.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.959+0000 7fb622800640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb5fc06c8f0 0x7fb5fc06edb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:05.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.959+0000 7fb622800640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb61c161dc0 0x7fb61c0cc070 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:05.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.959+0000 7fb622800640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb61c0d4ef0 0x7fb61c0cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:05.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.959+0000 7fb622800640 1 --2- 172.21.15.69:0/4103424723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb61c0d1610 0x7fb61c14d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:05.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.959+0000 7fb622800640 1 -- 172.21.15.69:0/4103424723 >> 172.21.15.69:0/4103424723 conn(0x7fb61c08bf60 msgr2=0x7fb61c0d47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:05.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.959+0000 7fb622800640 1 -- 172.21.15.69:0/4103424723 shutdown_connections 2024-04-24T08:13:05.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:05.959+0000 7fb622800640 1 -- 172.21.15.69:0/4103424723 wait complete. 2024-04-24T08:13:05.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-24T08:13:06.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.173+0000 7fea91600640 1 Processor -- start 2024-04-24T08:13:06.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.173+0000 7fea91600640 1 -- start start 2024-04-24T08:13:06.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.174+0000 7fea91600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea8c15e980 0x7fea8c162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.174+0000 7fea91600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fea8c0d0da0 0x7fea8c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.174+0000 7fea91600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea8c15ed60 con 0x7fea8c0d0190 2024-04-24T08:13:06.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.174+0000 7fea91600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fea8c0d1580 con 0x7fea8c0d0da0 2024-04-24T08:13:06.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.174+0000 7fea91600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fea8c0d1860 con 0x7fea8c15e980 2024-04-24T08:13:06.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.174+0000 7fea90c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fea8c0d0190 0x7fea8c06cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56598/0 (socket says 172.21.15.69:56598) 2024-04-24T08:13:06.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.174+0000 7fea90c00640 1 -- 172.21.15.69:0/3176238393 learned_addr learned my addr 172.21.15.69:0/3176238393 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:06.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8be00640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea8c15e980 0x7fea8c162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8b400640 1 --2- 172.21.15.69:0/3176238393 >> v2:172.21.15.121:3300/0 conn(0x7fea8c0d0da0 0x7fea8c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2872290434 0 0) 0x7fea8c15ed60 con 0x7fea8c0d0190 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea70003520 con 0x7fea8c0d0190 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2610104328 0 0) 0x7fea70003520 con 0x7fea8c0d0190 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea8c15e980 msgr2=0x7fea8c162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea8c15e980 0x7fea8c162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 >> v2:172.21.15.121:3300/0 conn(0x7fea8c0d0da0 msgr2=0x7fea8c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 --2- 172.21.15.69:0/3176238393 >> v2:172.21.15.121:3300/0 conn(0x7fea8c0d0da0 0x7fea8c0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c0cf450 con 0x7fea8c0d0190 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fea840019f0 con 0x7fea8c0d0190 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 >> v1:172.21.15.69:6789/0 conn(0x7fea8c0d0190 legacy=0x7fea8c06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea700039b0 0x7fea70003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea70004800 0x7fea70004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea8c15e980 0x7fea700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c0d1580 con 0x7fea70004800 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c0d1860 con 0x7fea700039b0 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.175+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fea7000bb00 con 0x7fea8c15e980 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.176+0000 7fea8b400640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea70004800 0x7fea70004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.176+0000 7fea8be00640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea700039b0 0x7fea70003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.176+0000 7fea8b400640 1 -- 172.21.15.69:0/3176238393 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea8c15e980 msgr2=0x7fea700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.176+0000 7fea8b400640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea8c15e980 0x7fea700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.176+0000 7fea8b400640 1 -- 172.21.15.69:0/3176238393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea700039b0 msgr2=0x7fea70003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.176+0000 7fea8b400640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea700039b0 0x7fea70003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.176+0000 7fea8b400640 1 -- 172.21.15.69:0/3176238393 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea70003520 con 0x7fea70004800 2024-04-24T08:13:06.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.177+0000 7fea8b400640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea70004800 0x7fea70004c80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fea7400ba00 tx=0x7fea70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934d4a24a14a7eb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:06.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.177+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea74030070 con 0x7fea70004800 2024-04-24T08:13:06.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.177+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea7402adb0 con 0x7fea70004800 2024-04-24T08:13:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.177+0000 7fea8aa00640 1 -- 172.21.15.69:0/3176238393 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7400b600 con 0x7fea70004800 2024-04-24T08:13:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.178+0000 7fea91600640 1 -- 172.21.15.69:0/3176238393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea70004800 msgr2=0x7fea70004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.178+0000 7fea91600640 1 --2- 172.21.15.69:0/3176238393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea70004800 0x7fea70004c80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fea7400ba00 tx=0x7fea70005830 comp rx=0 tx=0).stop 2024-04-24T08:13:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.178+0000 7fea90c00640 1 -- 172.21.15.69:0/3176238393 reap_dead start 2024-04-24T08:13:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.178+0000 7fea91600640 1 -- 172.21.15.69:0/3176238393 shutdown_connections 2024-04-24T08:13:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.178+0000 7fea91600640 1 -- 172.21.15.69:0/3176238393 >> 172.21.15.69:0/3176238393 conn(0x7fea8c09d920 msgr2=0x7fea8c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.178+0000 7fea91600640 1 -- 172.21.15.69:0/3176238393 shutdown_connections 2024-04-24T08:13:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.178+0000 7fea91600640 1 -- 172.21.15.69:0/3176238393 wait complete. 2024-04-24T08:13:06.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.180+0000 7fea91600640 1 Processor -- start 2024-04-24T08:13:06.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.180+0000 7fea91600640 1 -- start start 2024-04-24T08:13:06.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.181+0000 7fea91600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea70004800 0x7fea8c1fbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.181+0000 7fea91600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea8c0b7170 0x7fea8c0b75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.181+0000 7fea91600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea8c0b9160 0x7fea8c0b95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.181+0000 7fea91600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c0d1360 con 0x7fea8c0b7170 2024-04-24T08:13:06.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.181+0000 7fea91600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fea8c0d11e0 con 0x7fea8c0b9160 2024-04-24T08:13:06.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.181+0000 7fea91600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fea8c0cf070 con 0x7fea70004800 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.181+0000 7fea8b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea8c0b7170 0x7fea8c0b75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.181+0000 7fea8b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea8c0b7170 0x7fea8c0b75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37646/0 (socket says 172.21.15.69:37646) 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.181+0000 7fea8b400640 1 -- 172.21.15.69:0/3720569245 learned_addr learned my addr 172.21.15.69:0/3720569245 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea8be00640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea70004800 0x7fea8c1fbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea90c00640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea8c0b9160 0x7fea8c0b95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea8b400640 1 -- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea70004800 msgr2=0x7fea8c1fbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea8b400640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea70004800 0x7fea8c1fbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea8b400640 1 -- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea8c0b9160 msgr2=0x7fea8c0b95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea8b400640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea8c0b9160 0x7fea8c0b95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea8b400640 1 -- 172.21.15.69:0/3720569245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea8c0b9f30 con 0x7fea8c0b7170 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea90c00640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea8c0b9160 0x7fea8c0b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:06.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea8b400640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea8c0b7170 0x7fea8c0b75f0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fea7402a8a0 tx=0x7fea74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea83e00640 1 -- 172.21.15.69:0/3720569245 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea74043020 con 0x7fea8c0b7170 2024-04-24T08:13:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea83e00640 1 -- 172.21.15.69:0/3720569245 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea7403f680 con 0x7fea8c0b7170 2024-04-24T08:13:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea83e00640 1 -- 172.21.15.69:0/3720569245 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea7400b600 con 0x7fea8c0b7170 2024-04-24T08:13:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.182+0000 7fea91600640 1 -- 172.21.15.69:0/3720569245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea8c0ba210 con 0x7fea8c0b7170 2024-04-24T08:13:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.183+0000 7fea91600640 1 -- 172.21.15.69:0/3720569245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea8c0c95d0 con 0x7fea8c0b7170 2024-04-24T08:13:06.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.184+0000 7fea91600640 1 -- 172.21.15.69:0/3720569245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea8c0ceb90 con 0x7fea8c0b7170 2024-04-24T08:13:06.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.185+0000 7fea83e00640 1 -- 172.21.15.69:0/3720569245 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea74046020 con 0x7fea8c0b7170 2024-04-24T08:13:06.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.185+0000 7fea83e00640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea6806c8f0 0x7fea6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.186+0000 7fea83e00640 1 -- 172.21.15.69:0/3720569245 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 4943+0+0 (secure 0 0 0) 0x7fea740ac890 con 0x7fea8c0b7170 2024-04-24T08:13:06.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.186+0000 7fea8be00640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea6806c8f0 0x7fea6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.187+0000 7fea8be00640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea6806c8f0 0x7fea6806edb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fea7c006fd0 tx=0x7fea7c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:06.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.190+0000 7fea83e00640 1 -- 172.21.15.69:0/3720569245 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fea8c0ceb90 con 0x7fea8c0b7170 2024-04-24T08:13:06.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.406+0000 7fea91600640 1 -- 172.21.15.69:0/3720569245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fea8c0ba3e0 con 0x7fea8c0b7170 2024-04-24T08:13:06.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.411+0000 7fea83e00640 1 -- 172.21.15.69:0/3720569245 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) ==== 104+0+0 (secure 0 0 0) 0x7fea7404d020 con 0x7fea8c0b7170 2024-04-24T08:13:06.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.413+0000 7fea91600640 1 -- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea6806c8f0 msgr2=0x7fea6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.413+0000 7fea91600640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea6806c8f0 0x7fea6806edb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fea7c006fd0 tx=0x7fea7c008040 comp rx=0 tx=0).stop 2024-04-24T08:13:06.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.413+0000 7fea91600640 1 -- 172.21.15.69:0/3720569245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea8c0b7170 msgr2=0x7fea8c0b75f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.413+0000 7fea91600640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea8c0b7170 0x7fea8c0b75f0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fea7402a8a0 tx=0x7fea74004730 comp rx=0 tx=0).stop 2024-04-24T08:13:06.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.414+0000 7fea91600640 1 -- 172.21.15.69:0/3720569245 shutdown_connections 2024-04-24T08:13:06.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.414+0000 7fea91600640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fea6806c8f0 0x7fea6806edb0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.414+0000 7fea91600640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fea8c0b7170 0x7fea8c0b75f0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.414+0000 7fea91600640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fea8c0b9160 0x7fea8c0b95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.414+0000 7fea91600640 1 --2- 172.21.15.69:0/3720569245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fea70004800 0x7fea8c1fbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.414+0000 7fea91600640 1 -- 172.21.15.69:0/3720569245 >> 172.21.15.69:0/3720569245 conn(0x7fea8c09d920 msgr2=0x7fea8c0d24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:06.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.414+0000 7fea91600640 1 -- 172.21.15.69:0/3720569245 shutdown_connections 2024-04-24T08:13:06.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.414+0000 7fea91600640 1 -- 172.21.15.69:0/3720569245 wait complete. 2024-04-24T08:13:06.434 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-04-24T08:13:06.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.631+0000 7fdc8e000640 1 Processor -- start 2024-04-24T08:13:06.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.631+0000 7fdc8e000640 1 -- start start 2024-04-24T08:13:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.632+0000 7fdc8e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc88145fe0 0x7fdc8806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.632+0000 7fdc8e000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc880cf3d0 con 0x7fdc88145a90 2024-04-24T08:13:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.632+0000 7fdc8e000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc880cf6b0 con 0x7fdc880d0300 2024-04-24T08:13:06.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.632+0000 7fdc8e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdc880cf990 con 0x7fdc88145fe0 2024-04-24T08:13:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.632+0000 7fdc7fe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdc88145a90 0x7fdc88160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56604/0 (socket says 172.21.15.69:56604) 2024-04-24T08:13:06.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.632+0000 7fdc7fe00640 1 -- 172.21.15.69:0/1202960749 learned_addr learned my addr 172.21.15.69:0/1202960749 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.632+0000 7fdc87e00640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc88145fe0 0x7fdc8806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3732214434 0 0) 0x7fdc880cf3d0 con 0x7fdc88145a90 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc6c003520 con 0x7fdc88145a90 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2728950460 0 0) 0x7fdc6c003520 con 0x7fdc88145a90 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc88145fe0 msgr2=0x7fdc8806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc88145fe0 0x7fdc8806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 >> v1:172.21.15.121:6789/0 conn(0x7fdc880d0300 legacy=0x7fdc880d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc880cfc70 con 0x7fdc88145a90 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fdc70001670 con 0x7fdc88145a90 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 >> v1:172.21.15.69:6789/0 conn(0x7fdc88145a90 legacy=0x7fdc88160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 0x7fdc6c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc88145fe0 0x7fdc6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c004d40 0x7fdc6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdc880cf990 con 0x7fdc6c0039e0 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c00bad0 con 0x7fdc6c004d40 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdc6c00bcd0 con 0x7fdc88145fe0 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.633+0000 7fdc87400640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 0x7fdc6c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.634+0000 7fdc7fe00640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc88145fe0 0x7fdc6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.634+0000 7fdc87400640 1 -- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc88145fe0 msgr2=0x7fdc6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.634+0000 7fdc87400640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc88145fe0 0x7fdc6c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.634+0000 7fdc87400640 1 -- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c004d40 msgr2=0x7fdc6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.634+0000 7fdc87400640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc6c004d40 0x7fdc6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.634+0000 7fdc87400640 1 -- 172.21.15.69:0/1202960749 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6c003520 con 0x7fdc6c0039e0 2024-04-24T08:13:06.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.634+0000 7fdc7fe00640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc88145fe0 0x7fdc6c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.635+0000 7fdc87400640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 0x7fdc6c003e00 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fdc78001720 tx=0x7fdc6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0e2235559b0b50 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.635+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc78001370 con 0x7fdc6c0039e0 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.635+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc780052d0 con 0x7fdc6c0039e0 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.635+0000 7fdc86a00640 1 -- 172.21.15.69:0/1202960749 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc7802abc0 con 0x7fdc6c0039e0 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.636+0000 7fdc8e000640 1 -- 172.21.15.69:0/1202960749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 msgr2=0x7fdc6c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.636+0000 7fdc8e000640 1 --2- 172.21.15.69:0/1202960749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 0x7fdc6c003e00 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fdc78001720 tx=0x7fdc6c005830 comp rx=0 tx=0).stop 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.636+0000 7fdc87e00640 1 -- 172.21.15.69:0/1202960749 reap_dead start 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.636+0000 7fdc8e000640 1 -- 172.21.15.69:0/1202960749 shutdown_connections 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.636+0000 7fdc8e000640 1 -- 172.21.15.69:0/1202960749 >> 172.21.15.69:0/1202960749 conn(0x7fdc88067d90 msgr2=0x7fdc8806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.636+0000 7fdc8e000640 1 -- 172.21.15.69:0/1202960749 shutdown_connections 2024-04-24T08:13:06.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.636+0000 7fdc8e000640 1 -- 172.21.15.69:0/1202960749 wait complete. 2024-04-24T08:13:06.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.638+0000 7fdc8e000640 1 Processor -- start 2024-04-24T08:13:06.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.638+0000 7fdc8e000640 1 -- start start 2024-04-24T08:13:06.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc8e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 0x7fdc8815bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc8e000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc8813cf60 0x7fdc8815b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc8e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc8813e9e0 0x7fdc8813ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc8e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdc881458f0 con 0x7fdc6c0039e0 2024-04-24T08:13:06.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc8e000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdc88145770 con 0x7fdc8813cf60 2024-04-24T08:13:06.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc8e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdc880cdc30 con 0x7fdc8813e9e0 2024-04-24T08:13:06.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc87400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 0x7fdc8815bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc87400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 0x7fdc8815bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37670/0 (socket says 172.21.15.69:37670) 2024-04-24T08:13:06.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc87400640 1 -- 172.21.15.69:0/2079827921 learned_addr learned my addr 172.21.15.69:0/2079827921 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:06.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc87400640 1 -- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc8813e9e0 msgr2=0x7fdc8813ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc87400640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc8813e9e0 0x7fdc8813ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc87400640 1 -- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc8813cf60 msgr2=0x7fdc8815b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc87400640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc8813cf60 0x7fdc8815b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.639+0000 7fdc87400640 1 -- 172.21.15.69:0/2079827921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc8813cc00 con 0x7fdc6c0039e0 2024-04-24T08:13:06.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.640+0000 7fdc87400640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 0x7fdc8815bbe0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fdc78000c00 tx=0x7fdc78001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:06.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.640+0000 7fdc7f400640 1 -- 172.21.15.69:0/2079827921 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc78040020 con 0x7fdc6c0039e0 2024-04-24T08:13:06.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.640+0000 7fdc7f400640 1 -- 172.21.15.69:0/2079827921 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc78005480 con 0x7fdc6c0039e0 2024-04-24T08:13:06.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.640+0000 7fdc7f400640 1 -- 172.21.15.69:0/2079827921 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc78007a30 con 0x7fdc6c0039e0 2024-04-24T08:13:06.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.640+0000 7fdc8e000640 1 -- 172.21.15.69:0/2079827921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc8815b9d0 con 0x7fdc6c0039e0 2024-04-24T08:13:06.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.640+0000 7fdc8e000640 1 -- 172.21.15.69:0/2079827921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc8815d950 con 0x7fdc6c0039e0 2024-04-24T08:13:06.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.641+0000 7fdc8e000640 1 -- 172.21.15.69:0/2079827921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc88068a80 con 0x7fdc6c0039e0 2024-04-24T08:13:06.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.642+0000 7fdc7f400640 1 -- 172.21.15.69:0/2079827921 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdc78043020 con 0x7fdc6c0039e0 2024-04-24T08:13:06.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.643+0000 7fdc7f400640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc6406c8f0 0x7fdc6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:06.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.643+0000 7fdc7f400640 1 -- 172.21.15.69:0/2079827921 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 4943+0+0 (secure 0 0 0) 0x7fdc780ab700 con 0x7fdc6c0039e0 2024-04-24T08:13:06.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.644+0000 7fdc7fe00640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc6406c8f0 0x7fdc6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:06.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.645+0000 7fdc7fe00640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc6406c8f0 0x7fdc6406edb0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdc70000ff0 tx=0x7fdc70009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:06.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.649+0000 7fdc7f400640 1 -- 172.21.15.69:0/2079827921 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc7804a030 con 0x7fdc6c0039e0 2024-04-24T08:13:06.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.925+0000 7fdc8e000640 1 -- 172.21.15.69:0/2079827921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) -- 0x7fdc880d1120 con 0x7fdc6c0039e0 2024-04-24T08:13:06.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.960+0000 7fdc7f400640 1 -- 172.21.15.69:0/2079827921 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v246) ==== 169+0+0 (secure 0 0 0) 0x7fdc78079e20 con 0x7fdc6c0039e0 2024-04-24T08:13:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.962+0000 7fdc8e000640 1 -- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc6406c8f0 msgr2=0x7fdc6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.962+0000 7fdc8e000640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc6406c8f0 0x7fdc6406edb0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdc70000ff0 tx=0x7fdc70009040 comp rx=0 tx=0).stop 2024-04-24T08:13:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.962+0000 7fdc8e000640 1 -- 172.21.15.69:0/2079827921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 msgr2=0x7fdc8815bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.962+0000 7fdc8e000640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 0x7fdc8815bbe0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fdc78000c00 tx=0x7fdc78001120 comp rx=0 tx=0).stop 2024-04-24T08:13:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.963+0000 7fdc8e000640 1 -- 172.21.15.69:0/2079827921 shutdown_connections 2024-04-24T08:13:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.963+0000 7fdc8e000640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdc6406c8f0 0x7fdc6406edb0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.963+0000 7fdc8e000640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdc8813e9e0 0x7fdc8813ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.963+0000 7fdc8e000640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdc8813cf60 0x7fdc8815b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.963+0000 7fdc8e000640 1 --2- 172.21.15.69:0/2079827921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdc6c0039e0 0x7fdc8815bbe0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:06.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.963+0000 7fdc8e000640 1 -- 172.21.15.69:0/2079827921 >> 172.21.15.69:0/2079827921 conn(0x7fdc88067d90 msgr2=0x7fdc880aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:06.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.963+0000 7fdc8e000640 1 -- 172.21.15.69:0/2079827921 shutdown_connections 2024-04-24T08:13:06.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:06.963+0000 7fdc8e000640 1 -- 172.21.15.69:0/2079827921 wait complete. 2024-04-24T08:13:06.984 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-04-24T08:13:06.984 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-04-24T08:13:06.984 INFO:tasks.cephfs_test_runner:test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-24T08:13:06.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:07.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.186+0000 7f6601400640 1 Processor -- start 2024-04-24T08:13:07.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.186+0000 7f6601400640 1 -- start start 2024-04-24T08:13:07.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.187+0000 7f6601400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65fc1523f0 0x7f65fc06db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.187+0000 7f6601400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f65fc072e20 0x7f65fc073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.187+0000 7f6601400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc0aeac0 con 0x7f65fc0737a0 2024-04-24T08:13:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.187+0000 7f6601400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f65fc0aeda0 con 0x7f65fc072e20 2024-04-24T08:13:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.187+0000 7f6601400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f65fc0af080 con 0x7f65fc1523f0 2024-04-24T08:13:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f6600a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f65fc0737a0 0x7f65fc15ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56620/0 (socket says 172.21.15.69:56620) 2024-04-24T08:13:07.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f6600a00640 1 -- 172.21.15.69:0/2651771285 learned_addr learned my addr 172.21.15.69:0/2651771285 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65f3400640 1 --2- 172.21.15.69:0/2651771285 >> v2:172.21.15.121:3300/0 conn(0x7f65fc072e20 0x7f65fc073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fbe00640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65fc1523f0 0x7f65fc06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225776076 0 0) 0x7f65fc0aeac0 con 0x7f65fc0737a0 2024-04-24T08:13:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65e0003520 con 0x7f65fc0737a0 2024-04-24T08:13:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1410752790 0 0) 0x7f65e0003520 con 0x7f65fc0737a0 2024-04-24T08:13:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65fc1523f0 msgr2=0x7f65fc06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65fc1523f0 0x7f65fc06db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 >> v2:172.21.15.121:3300/0 conn(0x7f65fc072e20 msgr2=0x7f65fc073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 --2- 172.21.15.69:0/2651771285 >> v2:172.21.15.121:3300/0 conn(0x7f65fc072e20 0x7f65fc073260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc0af360 con 0x7f65fc0737a0 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f65f4001670 con 0x7f65fc0737a0 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 >> v1:172.21.15.69:6789/0 conn(0x7f65fc0737a0 legacy=0x7f65fc15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65e00039b0 0x7f65e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.188+0000 7f65fb400640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e0004800 0x7f65e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f65fb400640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e00051c0 0x7f65e000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc0aeda0 con 0x7f65e00051c0 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc0af080 con 0x7f65e0004800 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f65e000e2a0 con 0x7f65e00039b0 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f65f3400640 1 --2- 172.21.15.69:0/2651771285 >> v2:172.21.15.121:3300/0 conn(0x7f65fc072e20 0x7f65fc073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f65fbe00640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65fc1523f0 0x7f65fc06db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f6600a00640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e00051c0 0x7f65e000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f6600a00640 1 -- 172.21.15.69:0/2651771285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65e00039b0 msgr2=0x7f65e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f6600a00640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65e00039b0 0x7f65e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f6600a00640 1 -- 172.21.15.69:0/2651771285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e0004800 msgr2=0x7f65e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:07.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f6600a00640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e0004800 0x7f65e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.189+0000 7f6600a00640 1 -- 172.21.15.69:0/2651771285 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e0003520 con 0x7f65e00051c0 2024-04-24T08:13:07.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.190+0000 7f6600a00640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e00051c0 0x7f65e000dbd0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f65f400c290 tx=0x7f65e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f317cedff3c2383d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.190+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4007d50 con 0x7f65e00051c0 2024-04-24T08:13:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.190+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f4005af0 con 0x7f65e00051c0 2024-04-24T08:13:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.190+0000 7f65fb400640 1 -- 172.21.15.69:0/2651771285 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4007070 con 0x7f65e00051c0 2024-04-24T08:13:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.191+0000 7f6601400640 1 -- 172.21.15.69:0/2651771285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e00051c0 msgr2=0x7f65e000dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.191+0000 7f6601400640 1 --2- 172.21.15.69:0/2651771285 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65e00051c0 0x7f65e000dbd0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f65f400c290 tx=0x7f65e0005830 comp rx=0 tx=0).stop 2024-04-24T08:13:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.191+0000 7f6600a00640 1 -- 172.21.15.69:0/2651771285 reap_dead start 2024-04-24T08:13:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.191+0000 7f6601400640 1 -- 172.21.15.69:0/2651771285 shutdown_connections 2024-04-24T08:13:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.191+0000 7f6601400640 1 -- 172.21.15.69:0/2651771285 >> 172.21.15.69:0/2651771285 conn(0x7f65fc0d0160 msgr2=0x7f65fc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.191+0000 7f6601400640 1 -- 172.21.15.69:0/2651771285 shutdown_connections 2024-04-24T08:13:07.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.191+0000 7f6601400640 1 -- 172.21.15.69:0/2651771285 wait complete. 2024-04-24T08:13:07.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.193+0000 7f6601400640 1 Processor -- start 2024-04-24T08:13:07.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.193+0000 7f6601400640 1 -- start start 2024-04-24T08:13:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.194+0000 7f6601400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e00051c0 0x7f65fc09ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.194+0000 7f6601400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65fc1e1ca0 0x7f65fc159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.194+0000 7f6601400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc149290 0x7f65fc149710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.194+0000 7f6601400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc06db90 con 0x7f65fc149290 2024-04-24T08:13:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.194+0000 7f6601400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc1527d0 con 0x7f65e00051c0 2024-04-24T08:13:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.194+0000 7f6601400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f65fc06dd10 con 0x7f65fc1e1ca0 2024-04-24T08:13:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.194+0000 7f6600a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc149290 0x7f65fc149710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.194+0000 7f6600a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc149290 0x7f65fc149710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37686/0 (socket says 172.21.15.69:37686) 2024-04-24T08:13:07.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.194+0000 7f6600a00640 1 -- 172.21.15.69:0/1361309353 learned_addr learned my addr 172.21.15.69:0/1361309353 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f65fbe00640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e00051c0 0x7f65fc09ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f65f3400640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65fc1e1ca0 0x7f65fc159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f6600a00640 1 -- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65fc1e1ca0 msgr2=0x7f65fc159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f6600a00640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65fc1e1ca0 0x7f65fc159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f6600a00640 1 -- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e00051c0 msgr2=0x7f65fc09ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f6600a00640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e00051c0 0x7f65fc09ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f6600a00640 1 -- 172.21.15.69:0/1361309353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc09e0a0 con 0x7f65fc149290 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f6600a00640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc149290 0x7f65fc149710 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f65f4034040 tx=0x7f65fc15c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f65f8c00640 1 -- 172.21.15.69:0/1361309353 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4001c80 con 0x7f65fc149290 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f65f8c00640 1 -- 172.21.15.69:0/1361309353 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f4044070 con 0x7f65fc149290 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f65f8c00640 1 -- 172.21.15.69:0/1361309353 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f40076c0 con 0x7f65fc149290 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.195+0000 7f6601400640 1 -- 172.21.15.69:0/1361309353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65fc09d000 con 0x7f65fc149290 2024-04-24T08:13:07.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.196+0000 7f6601400640 1 -- 172.21.15.69:0/1361309353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65fc09d5c0 con 0x7f65fc149290 2024-04-24T08:13:07.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.197+0000 7f6601400640 1 -- 172.21.15.69:0/1361309353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65fc075d50 con 0x7f65fc149290 2024-04-24T08:13:07.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.198+0000 7f65f8c00640 1 -- 172.21.15.69:0/1361309353 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f65f403e480 con 0x7f65fc149290 2024-04-24T08:13:07.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.198+0000 7f65f8c00640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65d806c920 0x7f65d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.199+0000 7f65f8c00640 1 -- 172.21.15.69:0/1361309353 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 4943+0+0 (secure 0 0 0) 0x7f65f40ac210 con 0x7f65fc149290 2024-04-24T08:13:07.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.199+0000 7f65fbe00640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65d806c920 0x7f65d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.200+0000 7f65fbe00640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65d806c920 0x7f65d806ede0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f65fc06cb50 tx=0x7f65ec00b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:07.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.203+0000 7f65f8c00640 1 -- 172.21.15.69:0/1361309353 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f65f407a8b0 con 0x7f65fc149290 2024-04-24T08:13:07.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.401+0000 7f6601400640 1 -- 172.21.15.69:0/1361309353 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65fc15ac70 con 0x7f65fc149290 2024-04-24T08:13:07.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.403+0000 7f65f8c00640 1 -- 172.21.15.69:0/1361309353 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) ==== 74+0+12996 (secure 0 0 0) 0x7f65f407a1b0 con 0x7f65fc149290 2024-04-24T08:13:07.404 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:07.404 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":60,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:13:05.945654+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/2288057943":"2024-04-25T08:13:02.721950+0000","172.21.15.69:6833/2288057943":"2024-04-25T08:13:02.721950+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 -- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65d806c920 msgr2=0x7f65d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65d806c920 0x7f65d806ede0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f65fc06cb50 tx=0x7f65ec00b040 comp rx=0 tx=0).stop 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 -- 172.21.15.69:0/1361309353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc149290 msgr2=0x7f65fc149710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc149290 0x7f65fc149710 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f65f4034040 tx=0x7f65fc15c990 comp rx=0 tx=0).stop 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 -- 172.21.15.69:0/1361309353 shutdown_connections 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65d806c920 0x7f65d806ede0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc149290 0x7f65fc149710 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f65fc1e1ca0 0x7f65fc159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 --2- 172.21.15.69:0/1361309353 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65e00051c0 0x7f65fc09ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 -- 172.21.15.69:0/1361309353 >> 172.21.15.69:0/1361309353 conn(0x7f65fc0d0160 msgr2=0x7f65fc0b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 -- 172.21.15.69:0/1361309353 shutdown_connections 2024-04-24T08:13:07.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.406+0000 7f6601400640 1 -- 172.21.15.69:0/1361309353 wait complete. 2024-04-24T08:13:07.428 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:13:07.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:13:07.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.627+0000 7f91eca00640 1 Processor -- start 2024-04-24T08:13:07.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.627+0000 7f91eca00640 1 -- start start 2024-04-24T08:13:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.629+0000 7f91eca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91e815c820 0x7f91e80b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.629+0000 7f91eca00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e80b46a0 0x7f91e806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.629+0000 7f91eca00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f91e806fd10 0x7f91e80ce250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.629+0000 7f91eca00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f91e815cc00 con 0x7f91e815c820 2024-04-24T08:13:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.629+0000 7f91eca00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f91e806ed30 con 0x7f91e806fd10 2024-04-24T08:13:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.629+0000 7f91eca00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f91e80b4e10 con 0x7f91e80b46a0 2024-04-24T08:13:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.629+0000 7f91e5e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91e815c820 0x7f91e80b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.629+0000 7f91e5e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f91e815c820 0x7f91e80b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37692/0 (socket says 172.21.15.69:37692) 2024-04-24T08:13:07.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.629+0000 7f91e5e00640 1 -- 172.21.15.69:0/1245808728 learned_addr learned my addr 172.21.15.69:0/1245808728 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:07.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.630+0000 7f91dfe00640 1 --2- 172.21.15.69:0/1245808728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e80b46a0 0x7f91e806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.630+0000 7f91e5e00640 1 -- 172.21.15.69:0/1245808728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e80b46a0 msgr2=0x7f91e806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:07.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.630+0000 7f91e5e00640 1 --2- 172.21.15.69:0/1245808728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e80b46a0 0x7f91e806f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.630+0000 7f91e5e00640 1 -- 172.21.15.69:0/1245808728 >> v2:172.21.15.121:3300/0 conn(0x7f91e806fd10 msgr2=0x7f91e80ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:07.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.630+0000 7f91e5e00640 1 --2- 172.21.15.69:0/1245808728 >> v2:172.21.15.121:3300/0 conn(0x7f91e806fd10 0x7f91e80ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.630+0000 7f91e5e00640 1 -- 172.21.15.69:0/1245808728 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e80b50f0 con 0x7f91e815c820 2024-04-24T08:13:07.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.630+0000 7f91dfe00640 1 --2- 172.21.15.69:0/1245808728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e80b46a0 0x7f91e806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:07.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.631+0000 7f91e5e00640 1 --2- 172.21.15.69:0/1245808728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e815c820 0x7f91e80b3e90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f91e80afd00 tx=0x7f91d802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d93f7b80488fe3fa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:07.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.631+0000 7f91e5400640 1 -- 172.21.15.69:0/1245808728 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8037070 con 0x7f91e815c820 2024-04-24T08:13:07.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.631+0000 7f91e5400640 1 -- 172.21.15.69:0/1245808728 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d8032440 con 0x7f91e815c820 2024-04-24T08:13:07.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.631+0000 7f91e5400640 1 -- 172.21.15.69:0/1245808728 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d8032810 con 0x7f91e815c820 2024-04-24T08:13:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.632+0000 7f91eca00640 1 -- 172.21.15.69:0/1245808728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e815c820 msgr2=0x7f91e80b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.632+0000 7f91eca00640 1 --2- 172.21.15.69:0/1245808728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e815c820 0x7f91e80b3e90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f91e80afd00 tx=0x7f91d802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.632+0000 7f91eca00640 1 -- 172.21.15.69:0/1245808728 shutdown_connections 2024-04-24T08:13:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.632+0000 7f91eca00640 1 --2- 172.21.15.69:0/1245808728 >> v2:172.21.15.121:3300/0 conn(0x7f91e806fd10 0x7f91e80ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.632+0000 7f91eca00640 1 --2- 172.21.15.69:0/1245808728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e80b46a0 0x7f91e806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.632+0000 7f91eca00640 1 --2- 172.21.15.69:0/1245808728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e815c820 0x7f91e80b3e90 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.632+0000 7f91eca00640 1 -- 172.21.15.69:0/1245808728 >> 172.21.15.69:0/1245808728 conn(0x7f91e80d0350 msgr2=0x7f91e80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.632+0000 7f91eca00640 1 -- 172.21.15.69:0/1245808728 shutdown_connections 2024-04-24T08:13:07.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.632+0000 7f91eca00640 1 -- 172.21.15.69:0/1245808728 wait complete. 2024-04-24T08:13:07.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.634+0000 7f91eca00640 1 Processor -- start 2024-04-24T08:13:07.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.634+0000 7f91eca00640 1 -- start start 2024-04-24T08:13:07.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91eca00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e806fd10 0x7f91e8151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91eca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b46a0 0x7f91e815cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91eca00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e815c820 0x7f91e80ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91eca00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f91e80b4b00 con 0x7f91e80b46a0 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91eca00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f91e806f2b0 con 0x7f91e806fd10 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91eca00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f91e80ce790 con 0x7f91e815c820 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91dfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b46a0 0x7f91e815cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91dfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b46a0 0x7f91e815cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37708/0 (socket says 172.21.15.69:37708) 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91dfe00640 1 -- 172.21.15.69:0/1723600159 learned_addr learned my addr 172.21.15.69:0/1723600159 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91dfe00640 1 -- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e815c820 msgr2=0x7f91e80ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91e5e00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e806fd10 0x7f91e8151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91e6800640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e815c820 0x7f91e80ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91dfe00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e815c820 0x7f91e80ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91dfe00640 1 -- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e806fd10 msgr2=0x7f91e8151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91dfe00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e806fd10 0x7f91e8151870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91dfe00640 1 -- 172.21.15.69:0/1723600159 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e815e3b0 con 0x7f91e80b46a0 2024-04-24T08:13:07.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.636+0000 7f91e6800640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e815c820 0x7f91e80ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.637+0000 7f91dfe00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b46a0 0x7f91e815cde0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f91d000c7f0 tx=0x7f91d000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.637+0000 7f91de000640 1 -- 172.21.15.69:0/1723600159 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0015070 con 0x7f91e80b46a0 2024-04-24T08:13:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.637+0000 7f91de000640 1 -- 172.21.15.69:0/1723600159 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91d0007a30 con 0x7f91e80b46a0 2024-04-24T08:13:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.637+0000 7f91eca00640 1 -- 172.21.15.69:0/1723600159 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e815d830 con 0x7f91e80b46a0 2024-04-24T08:13:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.637+0000 7f91de000640 1 -- 172.21.15.69:0/1723600159 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91d0007e00 con 0x7f91e80b46a0 2024-04-24T08:13:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.637+0000 7f91eca00640 1 -- 172.21.15.69:0/1723600159 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e815ddf0 con 0x7f91e80b46a0 2024-04-24T08:13:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.638+0000 7f91eca00640 1 -- 172.21.15.69:0/1723600159 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e8068a80 con 0x7f91e80b46a0 2024-04-24T08:13:07.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.639+0000 7f91de000640 1 -- 172.21.15.69:0/1723600159 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f91d0018070 con 0x7f91e80b46a0 2024-04-24T08:13:07.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.640+0000 7f91de000640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f91c406c8f0 0x7f91c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:07.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.640+0000 7f91de000640 1 -- 172.21.15.69:0/1723600159 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 4943+0+0 (secure 0 0 0) 0x7f91d008b8f0 con 0x7f91e80b46a0 2024-04-24T08:13:07.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.641+0000 7f91e5e00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f91c406c8f0 0x7f91c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:07.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.642+0000 7f91e5e00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f91c406c8f0 0x7f91c406edb0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f91d8034000 tx=0x7f91d802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:07.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.645+0000 7f91de000640 1 -- 172.21.15.69:0/1723600159 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91d0059ef0 con 0x7f91e80b46a0 2024-04-24T08:13:07.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.849+0000 7f91eca00640 1 -- 172.21.15.69:0/1723600159 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f91e80cddc0 con 0x7f91e80b46a0 2024-04-24T08:13:07.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.850+0000 7f91de000640 1 -- 172.21.15.69:0/1723600159 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v60) ==== 93+0+10 (secure 0 0 0) 0x7f91d00597f0 con 0x7f91e80b46a0 2024-04-24T08:13:07.851 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:13:07.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 -- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f91c406c8f0 msgr2=0x7f91c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f91c406c8f0 0x7f91c406edb0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f91d8034000 tx=0x7f91d802e000 comp rx=0 tx=0).stop 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 -- 172.21.15.69:0/1723600159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b46a0 msgr2=0x7f91e815cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b46a0 0x7f91e815cde0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f91d000c7f0 tx=0x7f91d000cbf0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 -- 172.21.15.69:0/1723600159 shutdown_connections 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f91c406c8f0 0x7f91c406edb0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f91e80b46a0 0x7f91e815cde0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f91e815c820 0x7f91e80ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 --2- 172.21.15.69:0/1723600159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f91e806fd10 0x7f91e8151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 -- 172.21.15.69:0/1723600159 >> 172.21.15.69:0/1723600159 conn(0x7f91e80d0350 msgr2=0x7f91e80b0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 -- 172.21.15.69:0/1723600159 shutdown_connections 2024-04-24T08:13:07.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:07.853+0000 7f91eca00640 1 -- 172.21.15.69:0/1723600159 wait complete. 2024-04-24T08:13:07.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-04-24T08:13:08.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.076+0000 7f7af1a00640 1 Processor -- start 2024-04-24T08:13:08.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.076+0000 7f7af1a00640 1 -- start start 2024-04-24T08:13:08.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.078+0000 7f7af1a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7aec0cea60 0x7f7aec06d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:08.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.078+0000 7f7af1a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7aec06dd90 0x7f7aec09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:08.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.078+0000 7f7af1a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7aec0d0db0 con 0x7f7aec0d0820 2024-04-24T08:13:08.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.078+0000 7f7af1a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7aec0cee40 con 0x7f7aec0cea60 2024-04-24T08:13:08.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.078+0000 7f7af1a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7aec0d1090 con 0x7f7aec06dd90 2024-04-24T08:13:08.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.078+0000 7f7aeaa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7aec0cea60 0x7f7aec06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:08.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.078+0000 7f7aeb400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7aec0d0820 0x7f7aec15e1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56624/0 (socket says 172.21.15.69:56624) 2024-04-24T08:13:08.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.078+0000 7f7aeb400640 1 -- 172.21.15.69:0/4125694128 learned_addr learned my addr 172.21.15.69:0/4125694128 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:08.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.079+0000 7f7aea000640 1 --2- 172.21.15.69:0/4125694128 >> v2:172.21.15.121:3300/0 conn(0x7f7aec06dd90 0x7f7aec09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:08.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.079+0000 7f7aeaa00640 1 -- 172.21.15.69:0/4125694128 >> v2:172.21.15.121:3300/0 conn(0x7f7aec06dd90 msgr2=0x7f7aec09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:08.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.079+0000 7f7aeaa00640 1 --2- 172.21.15.69:0/4125694128 >> v2:172.21.15.121:3300/0 conn(0x7f7aec06dd90 0x7f7aec09a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.079+0000 7f7aeaa00640 1 -- 172.21.15.69:0/4125694128 >> v1:172.21.15.69:6789/0 conn(0x7f7aec0d0820 legacy=0x7f7aec15e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:08.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.079+0000 7f7ae9600640 1 -- 172.21.15.69:0/4125694128 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 814107945 0 0) 0x7f7aec0d0db0 con 0x7f7aec0d0820 2024-04-24T08:13:08.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.079+0000 7f7aeaa00640 1 -- 172.21.15.69:0/4125694128 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aec0d1370 con 0x7f7aec0cea60 2024-04-24T08:13:08.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.080+0000 7f7aeaa00640 1 --2- 172.21.15.69:0/4125694128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0cea60 0x7f7aec06d850 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0d02b0 tx=0x7f7adc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d6740fb5631f60 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:08.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.080+0000 7f7ae9600640 1 -- 172.21.15.69:0/4125694128 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc037070 con 0x7f7aec0cea60 2024-04-24T08:13:08.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.080+0000 7f7ae9600640 1 -- 172.21.15.69:0/4125694128 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7adc032440 con 0x7f7aec0cea60 2024-04-24T08:13:08.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.080+0000 7f7ae9600640 1 -- 172.21.15.69:0/4125694128 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7adc032810 con 0x7f7aec0cea60 2024-04-24T08:13:08.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.081+0000 7f7af1a00640 1 -- 172.21.15.69:0/4125694128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0cea60 msgr2=0x7f7aec06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:08.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.081+0000 7f7af1a00640 1 --2- 172.21.15.69:0/4125694128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0cea60 0x7f7aec06d850 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f7aec0d02b0 tx=0x7f7adc02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.081+0000 7f7af1a00640 1 -- 172.21.15.69:0/4125694128 shutdown_connections 2024-04-24T08:13:08.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.081+0000 7f7af1a00640 1 --2- 172.21.15.69:0/4125694128 >> v2:172.21.15.121:3300/0 conn(0x7f7aec06dd90 0x7f7aec09a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.081+0000 7f7af1a00640 1 --2- 172.21.15.69:0/4125694128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0cea60 0x7f7aec06d850 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.081+0000 7f7af1a00640 1 -- 172.21.15.69:0/4125694128 >> 172.21.15.69:0/4125694128 conn(0x7f7aec0b4040 msgr2=0x7f7aec0b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:08.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.081+0000 7f7af1a00640 1 -- 172.21.15.69:0/4125694128 shutdown_connections 2024-04-24T08:13:08.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.081+0000 7f7af1a00640 1 -- 172.21.15.69:0/4125694128 wait complete. 2024-04-24T08:13:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.083+0000 7f7af1a00640 1 Processor -- start 2024-04-24T08:13:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.083+0000 7f7af1a00640 1 -- start start 2024-04-24T08:13:08.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7af1a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aec06dd90 0x7f7aec0ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:08.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7af1a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7aec0cea60 0x7f7aec09e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:08.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7af1a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0d0820 0x7f7aec156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:08.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7af1a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec09af10 con 0x7f7aec0d0820 2024-04-24T08:13:08.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7af1a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7aec0d1930 con 0x7f7aec06dd90 2024-04-24T08:13:08.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7af1a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7aec09b090 con 0x7f7aec0cea60 2024-04-24T08:13:08.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7aeb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0d0820 0x7f7aec156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:08.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7aeb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0d0820 0x7f7aec156900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37722/0 (socket says 172.21.15.69:37722) 2024-04-24T08:13:08.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7aeb400640 1 -- 172.21.15.69:0/1666257275 learned_addr learned my addr 172.21.15.69:0/1666257275 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:08.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7aea000640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7aec0cea60 0x7f7aec09e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:08.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7aeaa00640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aec06dd90 0x7f7aec0ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:08.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.084+0000 7f7aeb400640 1 -- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7aec0cea60 msgr2=0x7f7aec09e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:08.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7aeb400640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7aec0cea60 0x7f7aec09e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7aeb400640 1 -- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aec06dd90 msgr2=0x7f7aec0ca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:08.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7aeb400640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aec06dd90 0x7f7aec0ca030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7aeb400640 1 -- 172.21.15.69:0/1666257275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aec09f080 con 0x7f7aec0d0820 2024-04-24T08:13:08.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7aea000640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7aec0cea60 0x7f7aec09e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:08.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7aeb400640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0d0820 0x7f7aec156900 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400c700 tx=0x7f7ae400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:08.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7ae2a00640 1 -- 172.21.15.69:0/1666257275 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4018070 con 0x7f7aec0d0820 2024-04-24T08:13:08.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7ae2a00640 1 -- 172.21.15.69:0/1666257275 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ae400ce10 con 0x7f7aec0d0820 2024-04-24T08:13:08.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7ae2a00640 1 -- 172.21.15.69:0/1666257275 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400a860 con 0x7f7aec0d0820 2024-04-24T08:13:08.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7af1a00640 1 -- 172.21.15.69:0/1666257275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aec09f360 con 0x7f7aec0d0820 2024-04-24T08:13:08.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.085+0000 7f7af1a00640 1 -- 172.21.15.69:0/1666257275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aec09f920 con 0x7f7aec0d0820 2024-04-24T08:13:08.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.086+0000 7f7af1a00640 1 -- 172.21.15.69:0/1666257275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aec071dd0 con 0x7f7aec0d0820 2024-04-24T08:13:08.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.088+0000 7f7ae2a00640 1 -- 172.21.15.69:0/1666257275 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ae400aa00 con 0x7f7aec0d0820 2024-04-24T08:13:08.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.088+0000 7f7ae2a00640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7acc06c8f0 0x7f7acc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:08.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.089+0000 7f7ae2a00640 1 -- 172.21.15.69:0/1666257275 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 4943+0+0 (secure 0 0 0) 0x7f7ae408ae60 con 0x7f7aec0d0820 2024-04-24T08:13:08.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.089+0000 7f7aeaa00640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7acc06c8f0 0x7f7acc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:08.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.090+0000 7f7aeaa00640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7acc06c8f0 0x7f7acc06edb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7adc034000 tx=0x7f7adc02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:08.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.093+0000 7f7ae2a00640 1 -- 172.21.15.69:0/1666257275 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7aec071dd0 con 0x7f7aec0d0820 2024-04-24T08:13:08.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.373+0000 7f7af1a00640 1 -- 172.21.15.69:0/1666257275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) -- 0x7f7aec068a80 con 0x7f7aec0d0820 2024-04-24T08:13:08.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.974+0000 7f7ae2a00640 1 -- 172.21.15.69:0/1666257275 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v248) ==== 169+0+0 (secure 0 0 0) 0x7f7ae4059580 con 0x7f7aec0d0820 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 -- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7acc06c8f0 msgr2=0x7f7acc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7acc06c8f0 0x7f7acc06edb0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7adc034000 tx=0x7f7adc02e000 comp rx=0 tx=0).stop 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 -- 172.21.15.69:0/1666257275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0d0820 msgr2=0x7f7aec156900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0d0820 0x7f7aec156900 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400c700 tx=0x7f7ae400cbc0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 -- 172.21.15.69:0/1666257275 shutdown_connections 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7acc06c8f0 0x7f7acc06edb0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7aec0d0820 0x7f7aec156900 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7aec0cea60 0x7f7aec09e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 --2- 172.21.15.69:0/1666257275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aec06dd90 0x7f7aec0ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 -- 172.21.15.69:0/1666257275 >> 172.21.15.69:0/1666257275 conn(0x7f7aec0b4040 msgr2=0x7f7aec06cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.978+0000 7f7af1a00640 1 -- 172.21.15.69:0/1666257275 shutdown_connections 2024-04-24T08:13:08.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:08.979+0000 7f7af1a00640 1 -- 172.21.15.69:0/1666257275 wait complete. 2024-04-24T08:13:09.001 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-24T08:13:09.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.198+0000 7f8fe0800640 1 Processor -- start 2024-04-24T08:13:09.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.198+0000 7f8fe0800640 1 -- start start 2024-04-24T08:13:09.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.199+0000 7f8fe0800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8fd8155f20 0x7f8fd80ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:09.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.199+0000 7f8fe0800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 0x7f8fd80cf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:09.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.199+0000 7f8fe0800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fd80cd6d0 con 0x7f8fd815c9b0 2024-04-24T08:13:09.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.199+0000 7f8fe0800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8fd806db90 con 0x7f8fd8155f20 2024-04-24T08:13:09.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.199+0000 7f8fe0800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd8072070 con 0x7f8fd80cf360 2024-04-24T08:13:09.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.199+0000 7f8fdde00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8fd8155f20 0x7f8fd80ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:09.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.199+0000 7f8fdde00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8fd8155f20 0x7f8fd80ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35328/0 (socket says 172.21.15.69:35328) 2024-04-24T08:13:09.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.199+0000 7f8fdde00640 1 -- 172.21.15.69:0/2247500058 learned_addr learned my addr 172.21.15.69:0/2247500058 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.200+0000 7f8fdde00640 1 -- 172.21.15.69:0/2247500058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 msgr2=0x7f8fd80cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.200+0000 7f8fde800640 1 --2- 172.21.15.69:0/2247500058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 0x7f8fd80cf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.200+0000 7f8fdde00640 1 --2- 172.21.15.69:0/2247500058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 0x7f8fd80cf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.200+0000 7f8fdde00640 1 -- 172.21.15.69:0/2247500058 >> v1:172.21.15.121:6789/0 conn(0x7f8fd815c9b0 legacy=0x7f8fd80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.200+0000 7f8fdde00640 1 -- 172.21.15.69:0/2247500058 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd80cd9b0 con 0x7f8fd8155f20 2024-04-24T08:13:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.200+0000 7f8fde800640 1 --2- 172.21.15.69:0/2247500058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 0x7f8fd80cf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.201+0000 7f8fdde00640 1 --2- 172.21.15.69:0/2247500058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 0x7f8fd80ccd60 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009870 tx=0x7f8fcc02a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b9b78e869d140dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.201+0000 7f8fdca00640 1 -- 172.21.15.69:0/2247500058 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc037070 con 0x7f8fd8155f20 2024-04-24T08:13:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.202+0000 7f8fdca00640 1 -- 172.21.15.69:0/2247500058 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fcc032440 con 0x7f8fd8155f20 2024-04-24T08:13:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.202+0000 7f8fdca00640 1 -- 172.21.15.69:0/2247500058 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc032810 con 0x7f8fd8155f20 2024-04-24T08:13:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.202+0000 7f8fe0800640 1 -- 172.21.15.69:0/2247500058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 msgr2=0x7f8fd80ccd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.202+0000 7f8fe0800640 1 --2- 172.21.15.69:0/2247500058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 0x7f8fd80ccd60 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009870 tx=0x7f8fcc02a290 comp rx=0 tx=0).stop 2024-04-24T08:13:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.202+0000 7f8fe0800640 1 -- 172.21.15.69:0/2247500058 shutdown_connections 2024-04-24T08:13:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.202+0000 7f8fe0800640 1 --2- 172.21.15.69:0/2247500058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 0x7f8fd80cf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.202+0000 7f8fe0800640 1 --2- 172.21.15.69:0/2247500058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 0x7f8fd80ccd60 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.202+0000 7f8fe0800640 1 -- 172.21.15.69:0/2247500058 >> 172.21.15.69:0/2247500058 conn(0x7f8fd80d0350 msgr2=0x7f8fd80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.202+0000 7f8fe0800640 1 -- 172.21.15.69:0/2247500058 shutdown_connections 2024-04-24T08:13:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.202+0000 7f8fe0800640 1 -- 172.21.15.69:0/2247500058 wait complete. 2024-04-24T08:13:09.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.204+0000 7f8fe0800640 1 Processor -- start 2024-04-24T08:13:09.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.204+0000 7f8fe0800640 1 -- start start 2024-04-24T08:13:09.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.205+0000 7f8fe0800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 0x7f8fd814b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:09.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.205+0000 7f8fe0800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 0x7f8fd813c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:09.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.205+0000 7f8fe0800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd815c9b0 0x7f8fd815f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:09.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.205+0000 7f8fe0800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd80b4140 con 0x7f8fd8155f20 2024-04-24T08:13:09.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.205+0000 7f8fe0800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd8071d00 con 0x7f8fd815c9b0 2024-04-24T08:13:09.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.205+0000 7f8fe0800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd80cd2a0 con 0x7f8fd80cf360 2024-04-24T08:13:09.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.205+0000 7f8fdd400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 0x7f8fd813c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:09.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.205+0000 7f8fdd400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 0x7f8fd813c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35332/0 (socket says 172.21.15.69:35332) 2024-04-24T08:13:09.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.205+0000 7f8fdd400640 1 -- 172.21.15.69:0/315223385 learned_addr learned my addr 172.21.15.69:0/315223385 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:09.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fdde00640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 0x7f8fd814b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:09.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fde800640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd815c9b0 0x7f8fd815f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:09.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fdd400640 1 -- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 msgr2=0x7f8fd814b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:09.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fdd400640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 0x7f8fd814b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:09.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fdd400640 1 -- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd815c9b0 msgr2=0x7f8fd815f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:09.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fdd400640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd815c9b0 0x7f8fd815f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:09.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fdd400640 1 -- 172.21.15.69:0/315223385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd81551d0 con 0x7f8fd8155f20 2024-04-24T08:13:09.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fde800640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd815c9b0 0x7f8fd815f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:09.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fdd400640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 0x7f8fd813c4f0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800c430 tx=0x7f8fc800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:09.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fd2000640 1 -- 172.21.15.69:0/315223385 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8011790 con 0x7f8fd8155f20 2024-04-24T08:13:09.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fd2000640 1 -- 172.21.15.69:0/315223385 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc8017070 con 0x7f8fd8155f20 2024-04-24T08:13:09.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.206+0000 7f8fe0800640 1 -- 172.21.15.69:0/315223385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd8149bb0 con 0x7f8fd8155f20 2024-04-24T08:13:09.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.207+0000 7f8fd2000640 1 -- 172.21.15.69:0/315223385 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8011b70 con 0x7f8fd8155f20 2024-04-24T08:13:09.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.207+0000 7f8fe0800640 1 -- 172.21.15.69:0/315223385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd815e5e0 con 0x7f8fd8155f20 2024-04-24T08:13:09.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.208+0000 7f8fe0800640 1 -- 172.21.15.69:0/315223385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd8068290 con 0x7f8fd8155f20 2024-04-24T08:13:09.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.209+0000 7f8fd2000640 1 -- 172.21.15.69:0/315223385 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8fc8011d10 con 0x7f8fd8155f20 2024-04-24T08:13:09.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.210+0000 7f8fd2000640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8fbc06c8f0 0x7f8fbc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:09.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.210+0000 7f8fd2000640 1 -- 172.21.15.69:0/315223385 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 4943+0+0 (secure 0 0 0) 0x7f8fc808b730 con 0x7f8fd8155f20 2024-04-24T08:13:09.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.210+0000 7f8fdde00640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8fbc06c8f0 0x7f8fbc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:09.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.212+0000 7f8fdde00640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8fbc06c8f0 0x7f8fbc06edb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc034000 tx=0x7f8fcc02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:09.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.214+0000 7f8fd2000640 1 -- 172.21.15.69:0/315223385 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8fd8068290 con 0x7f8fd8155f20 2024-04-24T08:13:09.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.414+0000 7f8fe0800640 1 -- 172.21.15.69:0/315223385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8fd8068a80 con 0x7f8fd8155f20 2024-04-24T08:13:09.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.420+0000 7f8fd2000640 1 -- 172.21.15.69:0/315223385 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f8fc8059e50 con 0x7f8fd8155f20 2024-04-24T08:13:09.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.423+0000 7f8fe0800640 1 -- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8fbc06c8f0 msgr2=0x7f8fbc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:09.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.423+0000 7f8fe0800640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8fbc06c8f0 0x7f8fbc06edb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc034000 tx=0x7f8fcc02e000 comp rx=0 tx=0).stop 2024-04-24T08:13:09.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.423+0000 7f8fe0800640 1 -- 172.21.15.69:0/315223385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 msgr2=0x7f8fd813c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:09.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.423+0000 7f8fe0800640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 0x7f8fd813c4f0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f8fc800c430 tx=0x7f8fc800c8f0 comp rx=0 tx=0).stop 2024-04-24T08:13:09.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.423+0000 7f8fe0800640 1 -- 172.21.15.69:0/315223385 shutdown_connections 2024-04-24T08:13:09.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.423+0000 7f8fe0800640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8fbc06c8f0 0x7f8fbc06edb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:09.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.423+0000 7f8fe0800640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8fd8155f20 0x7f8fd813c4f0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:09.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.423+0000 7f8fe0800640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8fd815c9b0 0x7f8fd815f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:09.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.423+0000 7f8fe0800640 1 --2- 172.21.15.69:0/315223385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8fd80cf360 0x7f8fd814b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:09.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.423+0000 7f8fe0800640 1 -- 172.21.15.69:0/315223385 >> 172.21.15.69:0/315223385 conn(0x7f8fd80d0350 msgr2=0x7f8fd80b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:09.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.424+0000 7f8fe0800640 1 -- 172.21.15.69:0/315223385 shutdown_connections 2024-04-24T08:13:09.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:09.424+0000 7f8fe0800640 1 -- 172.21.15.69:0/315223385 wait complete. 2024-04-24T08:13:09.442 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:13:09.443 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:13:09.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:13:09.663 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:13:09.663 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:13:09.663 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:13:09.723 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:13:09.723 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:13:09.723 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:13:09.783 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:13:09.784 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:13:09.784 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:13:09.840 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:13:09.840 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:13:09.840 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:13:09.896 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:13:09.896 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-24T08:13:09.896 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:10.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.108+0000 7f108fa00640 1 Processor -- start 2024-04-24T08:13:10.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.108+0000 7f108fa00640 1 -- start start 2024-04-24T08:13:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.109+0000 7f108fa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f108815ec40 0x7f108806cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.109+0000 7f108fa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10880b24b0 0x7f108806d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.109+0000 7f108fa00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10880b2c20 con 0x7f108806d980 2024-04-24T08:13:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.109+0000 7f108fa00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f108815f020 con 0x7f10880b24b0 2024-04-24T08:13:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.109+0000 7f108fa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f10880b2f00 con 0x7f108815ec40 2024-04-24T08:13:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.110+0000 7f107fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10880b24b0 0x7f108806d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.110+0000 7f107fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f10880b24b0 0x7f108806d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35346/0 (socket says 172.21.15.69:35346) 2024-04-24T08:13:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.110+0000 7f107fe00640 1 -- 172.21.15.69:0/153975104 learned_addr learned my addr 172.21.15.69:0/153975104 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:10.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.110+0000 7f108da00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f108806d980 0x7f10880cee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58830/0 (socket says 172.21.15.69:58830) 2024-04-24T08:13:10.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.110+0000 7f108d000640 1 --2- 172.21.15.69:0/153975104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f108815ec40 0x7f108806cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.110+0000 7f107fe00640 1 -- 172.21.15.69:0/153975104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f108815ec40 msgr2=0x7f108806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:10.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.110+0000 7f107fe00640 1 --2- 172.21.15.69:0/153975104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f108815ec40 0x7f108806cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.110+0000 7f107fe00640 1 -- 172.21.15.69:0/153975104 >> v1:172.21.15.69:6789/0 conn(0x7f108806d980 legacy=0x7f10880cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:10.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.110+0000 7f107fe00640 1 -- 172.21.15.69:0/153975104 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10880b31e0 con 0x7f10880b24b0 2024-04-24T08:13:10.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.110+0000 7f1087e00640 1 -- 172.21.15.69:0/153975104 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530811763 0 0) 0x7f10880b2c20 con 0x7f108806d980 2024-04-24T08:13:10.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.111+0000 7f107fe00640 1 --2- 172.21.15.69:0/153975104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 0x7f108806d440 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f1070009870 tx=0x7f107002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6c27ffe5bd466f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:10.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f1087e00640 1 -- 172.21.15.69:0/153975104 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070037070 con 0x7f10880b24b0 2024-04-24T08:13:10.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f1087e00640 1 -- 172.21.15.69:0/153975104 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1070032440 con 0x7f10880b24b0 2024-04-24T08:13:10.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f1087e00640 1 -- 172.21.15.69:0/153975104 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070032810 con 0x7f10880b24b0 2024-04-24T08:13:10.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f108fa00640 1 -- 172.21.15.69:0/153975104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 msgr2=0x7f108806d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:10.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f108fa00640 1 --2- 172.21.15.69:0/153975104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 0x7f108806d440 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f1070009870 tx=0x7f107002a250 comp rx=0 tx=0).stop 2024-04-24T08:13:10.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f108fa00640 1 -- 172.21.15.69:0/153975104 shutdown_connections 2024-04-24T08:13:10.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f108fa00640 1 --2- 172.21.15.69:0/153975104 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 0x7f108806d440 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f108fa00640 1 --2- 172.21.15.69:0/153975104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f108815ec40 0x7f108806cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f108fa00640 1 -- 172.21.15.69:0/153975104 >> 172.21.15.69:0/153975104 conn(0x7f108809b8b0 msgr2=0x7f108808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:10.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f108fa00640 1 -- 172.21.15.69:0/153975104 shutdown_connections 2024-04-24T08:13:10.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.112+0000 7f108fa00640 1 -- 172.21.15.69:0/153975104 wait complete. 2024-04-24T08:13:10.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.114+0000 7f108fa00640 1 Processor -- start 2024-04-24T08:13:10.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.114+0000 7f108fa00640 1 -- start start 2024-04-24T08:13:10.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.115+0000 7f108fa00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f108806d980 0x7f1088160370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.115+0000 7f108fa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 0x7f10880c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.115+0000 7f108fa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f108815ec40 0x7f10880aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.115+0000 7f108fa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f10880b2910 con 0x7f10880b24b0 2024-04-24T08:13:10.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.115+0000 7f108fa00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f10880b37a0 con 0x7f108806d980 2024-04-24T08:13:10.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.115+0000 7f108fa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f10880d1c80 con 0x7f108815ec40 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.115+0000 7f107fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 0x7f10880c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.115+0000 7f107fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 0x7f10880c6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35352/0 (socket says 172.21.15.69:35352) 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.115+0000 7f107fe00640 1 -- 172.21.15.69:0/2770054493 learned_addr learned my addr 172.21.15.69:0/2770054493 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f108da00640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f108815ec40 0x7f10880aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f107fe00640 1 -- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f108815ec40 msgr2=0x7f10880aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f108d000640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f108806d980 0x7f1088160370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f107fe00640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f108815ec40 0x7f10880aff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f107fe00640 1 -- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f108806d980 msgr2=0x7f1088160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f107fe00640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f108806d980 0x7f1088160370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f107fe00640 1 -- 172.21.15.69:0/2770054493 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10880c6940 con 0x7f10880b24b0 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f108d000640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f108806d980 0x7f1088160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:10.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f107fe00640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 0x7f10880c6040 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1070009870 tx=0x7f10700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:10.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f1085600640 1 -- 172.21.15.69:0/2770054493 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070037040 con 0x7f10880b24b0 2024-04-24T08:13:10.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f1085600640 1 -- 172.21.15.69:0/2770054493 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1070004270 con 0x7f10880b24b0 2024-04-24T08:13:10.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.116+0000 7f1085600640 1 -- 172.21.15.69:0/2770054493 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107003c670 con 0x7f10880b24b0 2024-04-24T08:13:10.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.117+0000 7f108fa00640 1 -- 172.21.15.69:0/2770054493 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10880c6c20 con 0x7f10880b24b0 2024-04-24T08:13:10.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.117+0000 7f108fa00640 1 -- 172.21.15.69:0/2770054493 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10880c7bf0 con 0x7f10880b24b0 2024-04-24T08:13:10.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.118+0000 7f108fa00640 1 -- 172.21.15.69:0/2770054493 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1088069190 con 0x7f10880b24b0 2024-04-24T08:13:10.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.119+0000 7f1085600640 1 -- 172.21.15.69:0/2770054493 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f107003a070 con 0x7f10880b24b0 2024-04-24T08:13:10.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.120+0000 7f1085600640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f106406c8f0 0x7f106406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.120+0000 7f1085600640 1 -- 172.21.15.69:0/2770054493 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 4943+0+0 (secure 0 0 0) 0x7f10700ac050 con 0x7f10880b24b0 2024-04-24T08:13:10.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.120+0000 7f108d000640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f106406c8f0 0x7f106406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.122+0000 7f108d000640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f106406c8f0 0x7f106406edb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1078009f50 tx=0x7f107800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:10.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.124+0000 7f1085600640 1 -- 172.21.15.69:0/2770054493 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1070041020 con 0x7f10880b24b0 2024-04-24T08:13:10.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.366+0000 7f108fa00640 1 -- 172.21.15.69:0/2770054493 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f108808c760 con 0x7f10880b24b0 2024-04-24T08:13:10.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.367+0000 7f1085600640 1 -- 172.21.15.69:0/2770054493 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+3875 (secure 0 0 0) 0x7f107007a810 con 0x7f10880b24b0 2024-04-24T08:13:10.367 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:10.367 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5218,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/4034001365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":4034001365},{"type":"v1","addr":"172.21.15.121:6835","nonce":4034001365}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":5221,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3907163219","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":3907163219},{"type":"v1","addr":"172.21.15.121:6837","nonce":3907163219}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":29},{"gid":6228,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2756395689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2756395689},{"type":"v1","addr":"172.21.15.69:6834","nonce":2756395689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":28},{"gid":6705,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/4214430603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":4214430603},{"type":"v1","addr":"172.21.15.69:6835","nonce":4214430603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[]} 2024-04-24T08:13:10.368 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 37 2024-04-24T08:13:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.369+0000 7f108fa00640 1 -- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f106406c8f0 msgr2=0x7f106406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.369+0000 7f108fa00640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f106406c8f0 0x7f106406edb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f1078009f50 tx=0x7f107800b040 comp rx=0 tx=0).stop 2024-04-24T08:13:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.369+0000 7f108fa00640 1 -- 172.21.15.69:0/2770054493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 msgr2=0x7f10880c6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.369+0000 7f108fa00640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 0x7f10880c6040 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f1070009870 tx=0x7f10700046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.369+0000 7f108fa00640 1 -- 172.21.15.69:0/2770054493 shutdown_connections 2024-04-24T08:13:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.369+0000 7f108fa00640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f106406c8f0 0x7f106406edb0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.369+0000 7f108fa00640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f10880b24b0 0x7f10880c6040 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.369+0000 7f108fa00640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f108815ec40 0x7f10880aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.369+0000 7f108fa00640 1 --2- 172.21.15.69:0/2770054493 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f108806d980 0x7f1088160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.369+0000 7f108fa00640 1 -- 172.21.15.69:0/2770054493 >> 172.21.15.69:0/2770054493 conn(0x7f108809b8b0 msgr2=0x7f10880d2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:10.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.370+0000 7f108fa00640 1 -- 172.21.15.69:0/2770054493 shutdown_connections 2024-04-24T08:13:10.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.370+0000 7f108fa00640 1 -- 172.21.15.69:0/2770054493 wait complete. 2024-04-24T08:13:10.388 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T08:13:10.388 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-24T08:13:10.388 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T08:13:10.388 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:13:10.388 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T08:13:10.389 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-24T08:13:10.389 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T08:13:10.389 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:13:10.389 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T08:13:10.389 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-24T08:13:10.389 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T08:13:10.389 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:13:10.390 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T08:13:10.390 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-24T08:13:10.390 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T08:13:10.390 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:13:10.404 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T08:13:10.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T08:13:10.406 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T08:13:10.406 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T08:13:10.408 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T08:13:10.408 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T08:13:10.409 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T08:13:10.410 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T08:13:10.450 INFO:tasks.ceph.mds.c:Started 2024-04-24T08:13:10.451 DEBUG:teuthology.parallel:result is None 2024-04-24T08:13:10.455 INFO:tasks.ceph.mds.a:Started 2024-04-24T08:13:10.455 DEBUG:teuthology.parallel:result is None 2024-04-24T08:13:10.455 INFO:tasks.ceph.mds.d:Started 2024-04-24T08:13:10.456 DEBUG:teuthology.parallel:result is None 2024-04-24T08:13:10.462 INFO:tasks.ceph.mds.b:Started 2024-04-24T08:13:10.462 DEBUG:teuthology.parallel:result is None 2024-04-24T08:13:10.462 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-24T08:13:10.588 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:13:10.598 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:13:10.615 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-24T08:13:10.615 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:13:10.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.628+0000 7ff9c2600640 1 Processor -- start 2024-04-24T08:13:10.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.628+0000 7ff9c2600640 1 -- start start 2024-04-24T08:13:10.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.629+0000 7ff9c2600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff9bc162c70 0x7ff9bc0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.629+0000 7ff9c2600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9bc0b3d30 0x7ff9bc0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.629+0000 7ff9c2600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9bc071d70 0x7ff9bc0721a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.629+0000 7ff9c2600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff9bc1669d0 con 0x7ff9bc0b3d30 2024-04-24T08:13:10.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.629+0000 7ff9c2600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff9bc166cb0 con 0x7ff9bc162c70 2024-04-24T08:13:10.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.629+0000 7ff9c2600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff9bc166f90 con 0x7ff9bc071d70 2024-04-24T08:13:10.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.629+0000 7ff9baa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9bc0b3d30 0x7ff9bc0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.629+0000 7ff9baa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9bc0b3d30 0x7ff9bc0b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35392/0 (socket says 172.21.15.69:35392) 2024-04-24T08:13:10.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.629+0000 7ff9baa00640 1 -- 172.21.15.69:0/3490744471 learned_addr learned my addr 172.21.15.69:0/3490744471 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:10.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.630+0000 7ff9bbe00640 1 --2- 172.21.15.69:0/3490744471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9bc071d70 0x7ff9bc0721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.630+0000 7ff9bb400640 1 --2- 172.21.15.69:0/3490744471 >> v2:172.21.15.121:3300/0 conn(0x7ff9bc162c70 0x7ff9bc0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.630+0000 7ff9baa00640 1 -- 172.21.15.69:0/3490744471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9bc071d70 msgr2=0x7ff9bc0721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:10.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.630+0000 7ff9baa00640 1 --2- 172.21.15.69:0/3490744471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9bc071d70 0x7ff9bc0721a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.630+0000 7ff9baa00640 1 -- 172.21.15.69:0/3490744471 >> v2:172.21.15.121:3300/0 conn(0x7ff9bc162c70 msgr2=0x7ff9bc0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:10.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.630+0000 7ff9baa00640 1 --2- 172.21.15.69:0/3490744471 >> v2:172.21.15.121:3300/0 conn(0x7ff9bc162c70 0x7ff9bc0b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.630+0000 7ff9baa00640 1 -- 172.21.15.69:0/3490744471 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc167270 con 0x7ff9bc0b3d30 2024-04-24T08:13:10.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.631+0000 7ff9baa00640 1 --2- 172.21.15.69:0/3490744471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc0b3d30 0x7ff9bc0b41c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc070120 tx=0x7ff9a802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f4310b79afc86a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:10.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.631+0000 7ff9ba000640 1 -- 172.21.15.69:0/3490744471 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8037070 con 0x7ff9bc0b3d30 2024-04-24T08:13:10.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.631+0000 7ff9ba000640 1 -- 172.21.15.69:0/3490744471 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a8032440 con 0x7ff9bc0b3d30 2024-04-24T08:13:10.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.631+0000 7ff9ba000640 1 -- 172.21.15.69:0/3490744471 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8032810 con 0x7ff9bc0b3d30 2024-04-24T08:13:10.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.631+0000 7ff9c2600640 1 -- 172.21.15.69:0/3490744471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc0b3d30 msgr2=0x7ff9bc0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:10.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.631+0000 7ff9c2600640 1 --2- 172.21.15.69:0/3490744471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc0b3d30 0x7ff9bc0b41c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc070120 tx=0x7ff9a802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.632+0000 7ff9c2600640 1 -- 172.21.15.69:0/3490744471 shutdown_connections 2024-04-24T08:13:10.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.632+0000 7ff9c2600640 1 --2- 172.21.15.69:0/3490744471 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9bc071d70 0x7ff9bc0721a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.632+0000 7ff9c2600640 1 --2- 172.21.15.69:0/3490744471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc0b3d30 0x7ff9bc0b41c0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.632+0000 7ff9c2600640 1 --2- 172.21.15.69:0/3490744471 >> v2:172.21.15.121:3300/0 conn(0x7ff9bc162c70 0x7ff9bc0b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.632+0000 7ff9c2600640 1 -- 172.21.15.69:0/3490744471 >> 172.21.15.69:0/3490744471 conn(0x7ff9bc067d90 msgr2=0x7ff9bc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:10.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.632+0000 7ff9c2600640 1 -- 172.21.15.69:0/3490744471 shutdown_connections 2024-04-24T08:13:10.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.632+0000 7ff9c2600640 1 -- 172.21.15.69:0/3490744471 wait complete. 2024-04-24T08:13:10.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.633+0000 7ff9c2600640 1 Processor -- start 2024-04-24T08:13:10.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.633+0000 7ff9c2600640 1 -- start start 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.634+0000 7ff9c2600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc071d70 0x7ff9bc0c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.634+0000 7ff9c2600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9bc0b3d30 0x7ff9bc0c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.634+0000 7ff9c2600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9bc162c70 0x7ff9bc0c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.634+0000 7ff9c2600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc0cfc40 con 0x7ff9bc071d70 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.634+0000 7ff9c2600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc163050 con 0x7ff9bc162c70 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.634+0000 7ff9c2600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff9bc0cfdc0 con 0x7ff9bc0b3d30 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.634+0000 7ff9bb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc071d70 0x7ff9bc0c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9bb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc071d70 0x7ff9bc0c9c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35408/0 (socket says 172.21.15.69:35408) 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9bb400640 1 -- 172.21.15.69:0/904624417 learned_addr learned my addr 172.21.15.69:0/904624417 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9baa00640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9bc0b3d30 0x7ff9bc0c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9bbe00640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9bc162c70 0x7ff9bc0c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9bb400640 1 -- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9bc0b3d30 msgr2=0x7ff9bc0c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:10.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9bb400640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9bc0b3d30 0x7ff9bc0c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9bb400640 1 -- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9bc162c70 msgr2=0x7ff9bc0c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:10.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9bb400640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9bc162c70 0x7ff9bc0c63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:10.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9bb400640 1 -- 172.21.15.69:0/904624417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc0c5330 con 0x7ff9bc071d70 2024-04-24T08:13:10.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9bb400640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc071d70 0x7ff9bc0c9c10 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00d800 tx=0x7ff9ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:10.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9b3400640 1 -- 172.21.15.69:0/904624417 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00b6c0 con 0x7ff9bc071d70 2024-04-24T08:13:10.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9b3400640 1 -- 172.21.15.69:0/904624417 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9ac019070 con 0x7ff9bc071d70 2024-04-24T08:13:10.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.635+0000 7ff9b3400640 1 -- 172.21.15.69:0/904624417 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00ba80 con 0x7ff9bc071d70 2024-04-24T08:13:10.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.636+0000 7ff9c2600640 1 -- 172.21.15.69:0/904624417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9bc0c5670 con 0x7ff9bc071d70 2024-04-24T08:13:10.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.636+0000 7ff9c2600640 1 -- 172.21.15.69:0/904624417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9bc0c5eb0 con 0x7ff9bc071d70 2024-04-24T08:13:10.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.637+0000 7ff9c2600640 1 -- 172.21.15.69:0/904624417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9bc068a80 con 0x7ff9bc071d70 2024-04-24T08:13:10.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.638+0000 7ff9b3400640 1 -- 172.21.15.69:0/904624417 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff9ac00bc20 con 0x7ff9bc071d70 2024-04-24T08:13:10.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.639+0000 7ff9b3400640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff99c06c8f0 0x7ff99c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:10.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.639+0000 7ff9b3400640 1 -- 172.21.15.69:0/904624417 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 4943+0+0 (secure 0 0 0) 0x7ff9ac08bb20 con 0x7ff9bc071d70 2024-04-24T08:13:10.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.640+0000 7ff9baa00640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff99c06c8f0 0x7ff99c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:10.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.641+0000 7ff9baa00640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff99c06c8f0 0x7ff99c06edb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8034000 tx=0x7ff9a802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:10.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.644+0000 7ff9b3400640 1 -- 172.21.15.69:0/904624417 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9bc0c5670 con 0x7ff9bc071d70 2024-04-24T08:13:10.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:10.857+0000 7ff9c2600640 1 -- 172.21.15.69:0/904624417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff9bc06ddd0 con 0x7ff9bc071d70 2024-04-24T08:13:11.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.002+0000 7ff9b3400640 1 -- 172.21.15.69:0/904624417 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v62) ==== 97+0+0 (secure 0 0 0) 0x7ff9ac059bf0 con 0x7ff9bc071d70 2024-04-24T08:13:11.003 INFO:teuthology.orchestra.run.smithi069.stderr: removed all blocklist entries 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.005+0000 7ff9c2600640 1 -- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff99c06c8f0 msgr2=0x7ff99c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.005+0000 7ff9c2600640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff99c06c8f0 0x7ff99c06edb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8034000 tx=0x7ff9a802e000 comp rx=0 tx=0).stop 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.006+0000 7ff9c2600640 1 -- 172.21.15.69:0/904624417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc071d70 msgr2=0x7ff9bc0c9c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.006+0000 7ff9c2600640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc071d70 0x7ff9bc0c9c10 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00d800 tx=0x7ff9ac00dcc0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.006+0000 7ff9c2600640 1 -- 172.21.15.69:0/904624417 shutdown_connections 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.006+0000 7ff9c2600640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff99c06c8f0 0x7ff99c06edb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.006+0000 7ff9c2600640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9bc162c70 0x7ff9bc0c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.006+0000 7ff9c2600640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9bc0b3d30 0x7ff9bc0c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.006+0000 7ff9c2600640 1 --2- 172.21.15.69:0/904624417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9bc071d70 0x7ff9bc0c9c10 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.006+0000 7ff9c2600640 1 -- 172.21.15.69:0/904624417 >> 172.21.15.69:0/904624417 conn(0x7ff9bc067d90 msgr2=0x7ff9bc0cdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.006+0000 7ff9c2600640 1 -- 172.21.15.69:0/904624417 shutdown_connections 2024-04-24T08:13:11.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.006+0000 7ff9c2600640 1 -- 172.21.15.69:0/904624417 wait complete. 2024-04-24T08:13:11.027 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-24T08:13:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.204+0000 7fc8e5200640 1 Processor -- start 2024-04-24T08:13:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.204+0000 7fc8e5200640 1 -- start start 2024-04-24T08:13:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.206+0000 7fc8e5200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e00d0300 0x7fc8e006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.206+0000 7fc8e5200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc8e0145a90 0x7fc8e00d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.206+0000 7fc8e5200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8e00d0c20 con 0x7fc8e0145fe0 2024-04-24T08:13:11.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.206+0000 7fc8e5200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc8e00cf3d0 con 0x7fc8e0145a90 2024-04-24T08:13:11.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.206+0000 7fc8e5200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e00cf6b0 con 0x7fc8e00d0300 2024-04-24T08:13:11.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.206+0000 7fc8de800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc8e0145fe0 0x7fc8e0160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58858/0 (socket says 172.21.15.69:58858) 2024-04-24T08:13:11.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.206+0000 7fc8de800640 1 -- 172.21.15.69:0/2535067831 learned_addr learned my addr 172.21.15.69:0/2535067831 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:11.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dde00640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e00d0300 0x7fc8e006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dd400640 1 --2- 172.21.15.69:0/2535067831 >> v2:172.21.15.121:3300/0 conn(0x7fc8e0145a90 0x7fc8e00d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 278111069 0 0) 0x7fc8e00d0c20 con 0x7fc8e0145fe0 2024-04-24T08:13:11.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8c4003520 con 0x7fc8e0145fe0 2024-04-24T08:13:11.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2831585694 0 0) 0x7fc8c4003520 con 0x7fc8e0145fe0 2024-04-24T08:13:11.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e00d0300 msgr2=0x7fc8e006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:11.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e00d0300 0x7fc8e006e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 >> v2:172.21.15.121:3300/0 conn(0x7fc8e0145a90 msgr2=0x7fc8e00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:11.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 --2- 172.21.15.69:0/2535067831 >> v2:172.21.15.121:3300/0 conn(0x7fc8e0145a90 0x7fc8e00d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e00cf990 con 0x7fc8e0145fe0 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fc8d8001670 con 0x7fc8e0145fe0 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 >> v1:172.21.15.69:6789/0 conn(0x7fc8e0145fe0 legacy=0x7fc8e0160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 0x7fc8c4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e00d0300 0x7fc8c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8c4004d40 0x7fc8c40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e00cf3d0 con 0x7fc8c40039b0 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e00cf6b0 con 0x7fc8c4004d40 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.207+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc8c400b9c0 con 0x7fc8e00d0300 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.208+0000 7fc8dde00640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 0x7fc8c4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.208+0000 7fc8dd400640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e00d0300 0x7fc8c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.208+0000 7fc8de800640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8c4004d40 0x7fc8c40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.208+0000 7fc8dde00640 1 -- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e00d0300 msgr2=0x7fc8c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:11.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.208+0000 7fc8dde00640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e00d0300 0x7fc8c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.208+0000 7fc8dde00640 1 -- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8c4004d40 msgr2=0x7fc8c40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:11.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.208+0000 7fc8dde00640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8c4004d40 0x7fc8c40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.208+0000 7fc8dde00640 1 -- 172.21.15.69:0/2535067831 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c4003520 con 0x7fc8c40039b0 2024-04-24T08:13:11.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.208+0000 7fc8de800640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8c4004d40 0x7fc8c40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:11.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.209+0000 7fc8dde00640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 0x7fc8c4003dd0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0009fc0 tx=0x7fc8c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400eb7f38c640f31 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:11.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.209+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0009c10 con 0x7fc8c40039b0 2024-04-24T08:13:11.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.209+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d0036070 con 0x7fc8c40039b0 2024-04-24T08:13:11.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.209+0000 7fc8dca00640 1 -- 172.21.15.69:0/2535067831 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0031880 con 0x7fc8c40039b0 2024-04-24T08:13:11.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.210+0000 7fc8e5200640 1 -- 172.21.15.69:0/2535067831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 msgr2=0x7fc8c4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.210+0000 7fc8e5200640 1 --2- 172.21.15.69:0/2535067831 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 0x7fc8c4003dd0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc8d0009fc0 tx=0x7fc8c4005830 comp rx=0 tx=0).stop 2024-04-24T08:13:11.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.210+0000 7fc8de800640 1 -- 172.21.15.69:0/2535067831 reap_dead start 2024-04-24T08:13:11.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.210+0000 7fc8e5200640 1 -- 172.21.15.69:0/2535067831 shutdown_connections 2024-04-24T08:13:11.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.210+0000 7fc8e5200640 1 -- 172.21.15.69:0/2535067831 >> 172.21.15.69:0/2535067831 conn(0x7fc8e0067d90 msgr2=0x7fc8e006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:11.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.210+0000 7fc8e5200640 1 -- 172.21.15.69:0/2535067831 shutdown_connections 2024-04-24T08:13:11.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.210+0000 7fc8e5200640 1 -- 172.21.15.69:0/2535067831 wait complete. 2024-04-24T08:13:11.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.212+0000 7fc8e5200640 1 Processor -- start 2024-04-24T08:13:11.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.212+0000 7fc8e5200640 1 -- start start 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8e5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 0x7fc8e015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8e5200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e0140450 0x7fc8e01408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8e5200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8e01f5990 0x7fc8e01f5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8e5200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e0145770 con 0x7fc8c40039b0 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8e5200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc8e00cff50 con 0x7fc8e01f5990 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8e5200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc8e01458f0 con 0x7fc8e0140450 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8dde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 0x7fc8e015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8dde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 0x7fc8e015c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35438/0 (socket says 172.21.15.69:35438) 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8dde00640 1 -- 172.21.15.69:0/1493332452 learned_addr learned my addr 172.21.15.69:0/1493332452 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8dde00640 1 -- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e0140450 msgr2=0x7fc8e01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8dde00640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e0140450 0x7fc8e01408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8dde00640 1 -- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8e01f5990 msgr2=0x7fc8e01f5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8dde00640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8e01f5990 0x7fc8e01f5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.213+0000 7fc8dde00640 1 -- 172.21.15.69:0/1493332452 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8e015c850 con 0x7fc8c40039b0 2024-04-24T08:13:11.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.214+0000 7fc8dde00640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 0x7fc8e015c310 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e200 tx=0x7fc8d0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:11.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.214+0000 7fc8d6000640 1 -- 172.21.15.69:0/1493332452 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0009c10 con 0x7fc8c40039b0 2024-04-24T08:13:11.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.214+0000 7fc8d6000640 1 -- 172.21.15.69:0/1493332452 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d0003990 con 0x7fc8c40039b0 2024-04-24T08:13:11.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.214+0000 7fc8d6000640 1 -- 172.21.15.69:0/1493332452 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0003d60 con 0x7fc8c40039b0 2024-04-24T08:13:11.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.214+0000 7fc8e5200640 1 -- 172.21.15.69:0/1493332452 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8e0140f80 con 0x7fc8c40039b0 2024-04-24T08:13:11.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.214+0000 7fc8e5200640 1 -- 172.21.15.69:0/1493332452 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8e01f36e0 con 0x7fc8c40039b0 2024-04-24T08:13:11.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.215+0000 7fc8e5200640 1 -- 172.21.15.69:0/1493332452 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8e0141440 con 0x7fc8c40039b0 2024-04-24T08:13:11.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.216+0000 7fc8d6000640 1 -- 172.21.15.69:0/1493332452 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc8d0005d10 con 0x7fc8c40039b0 2024-04-24T08:13:11.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.217+0000 7fc8d6000640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc8bc06c920 0x7fc8bc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.217+0000 7fc8d6000640 1 -- 172.21.15.69:0/1493332452 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7fc8d00acba0 con 0x7fc8c40039b0 2024-04-24T08:13:11.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.218+0000 7fc8dd400640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc8bc06c920 0x7fc8bc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.219+0000 7fc8dd400640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc8bc06c920 0x7fc8bc06ede0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00cf000 tx=0x7fc8c8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:11.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.221+0000 7fc8d6000640 1 -- 172.21.15.69:0/1493332452 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8d007b280 con 0x7fc8c40039b0 2024-04-24T08:13:11.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.491+0000 7fc8e5200640 1 -- 172.21.15.69:0/1493332452 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fc8e0068a80 con 0x7fc8c40039b0 2024-04-24T08:13:11.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.493+0000 7fc8d6000640 1 -- 172.21.15.69:0/1493332452 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) ==== 80+0+5304 (secure 0 0 0) 0x7fc8d007ab80 con 0x7fc8c40039b0 2024-04-24T08:13:11.493 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:11.493 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:13:11.493 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_dump": [ 2024-04-24T08:13:11.493 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.a", 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmsqMeLRAApLPtHjtq78HzlVcsjfqXOg==", 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.b", 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmkyd5ChAAGsJGU6GGMvvGzQnABuDx/g==", 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:13:11.494 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.c", 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmS+yLORAAHPCeBtq99IXEVTpcMT3c4A==", 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:13:11.495 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:13:11.496 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.496 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.496 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.496 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.496 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.d", 2024-04-24T08:13:11.496 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShm0xmHFRAA3VfYoqbcV8kWF+4fCZMVLA==", 2024-04-24T08:13:11.496 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.496 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.0", 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC5vShmMbvcERAANgogiCC/sXf1y0Ykv7ecNQ==", 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:13:11.497 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.1", 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC8vShmDD+uLBAAJHXmxEm1cumJ3wjqe6S0ww==", 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.498 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.2", 2024-04-24T08:13:11.499 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC/vShmS63kNhAAXec5HVuwo8THKstXiQOL8Q==", 2024-04-24T08:13:11.499 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.499 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:13:11.499 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:13:11.499 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.499 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.499 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.499 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.500 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.3", 2024-04-24T08:13:11.500 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDDvShmxcDJBhAA3V3I4C63EHPRUyhzBCZEzw==", 2024-04-24T08:13:11.500 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.500 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:13:11.500 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:13:11.500 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.500 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.4", 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDJvShm1pX9ERAAL2gjv/+M915Cf5OoqBhZVw==", 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.5", 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDMvShmNIxbHRAA23gxiaGvagddVc2mE5DJJw==", 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.501 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:13:11.502 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:13:11.502 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.502 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.502 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.502 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.502 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.6", 2024-04-24T08:13:11.502 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDPvShmb+x4NhAA6aWmAzU6c0AZomVgWcFwig==", 2024-04-24T08:13:11.502 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.502 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:13:11.502 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.7", 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDTvShmgcKMBRAA6fkJvNxBjlNhK9CrZ/FMSw==", 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.503 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.0", 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw==", 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow r", 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.1", 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmKAIHMhAAuTuFmTHvBteErsG23LMEyQ==", 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:13:11.504 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.2", 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmW0MnKRAASzxCAKuApVvfu9kUfK9BJw==", 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.3", 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC2vShmfgoaABAAniS9oubv92R7Qe2EITGKxg==", 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.505 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.admin", 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQCzvShmLh6yLxAAoH7NhoEAUKcdXp6n+ai2Ug==", 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.506 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.x", 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmv4rwIxAA3rp+yy09qrs7HXip8bZ8xA==", 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.y", 2024-04-24T08:13:11.507 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShm2eatGxAAFQkL0GBsujypv4VnmSy5bw==", 2024-04-24T08:13:11.508 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:13:11.508 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:13:11.508 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:13:11.508 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:13:11.508 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.508 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:11.508 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-24T08:13:11.508 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.495+0000 7fc8e5200640 1 -- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc8bc06c920 msgr2=0x7fc8bc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.495+0000 7fc8e5200640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc8bc06c920 0x7fc8bc06ede0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc8e00cf000 tx=0x7fc8c8008040 comp rx=0 tx=0).stop 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.495+0000 7fc8e5200640 1 -- 172.21.15.69:0/1493332452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 msgr2=0x7fc8e015c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.495+0000 7fc8e5200640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 0x7fc8e015c310 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e200 tx=0x7fc8d0009750 comp rx=0 tx=0).stop 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.496+0000 7fc8e5200640 1 -- 172.21.15.69:0/1493332452 shutdown_connections 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.496+0000 7fc8e5200640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc8bc06c920 0x7fc8bc06ede0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.496+0000 7fc8e5200640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc8e01f5990 0x7fc8e01f5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.496+0000 7fc8e5200640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc8e0140450 0x7fc8e01408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.496+0000 7fc8e5200640 1 --2- 172.21.15.69:0/1493332452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc8c40039b0 0x7fc8e015c310 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.496+0000 7fc8e5200640 1 -- 172.21.15.69:0/1493332452 >> 172.21.15.69:0/1493332452 conn(0x7fc8e0067d90 msgr2=0x7fc8e006e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:11.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.496+0000 7fc8e5200640 1 -- 172.21.15.69:0/1493332452 shutdown_connections 2024-04-24T08:13:11.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.496+0000 7fc8e5200640 1 -- 172.21.15.69:0/1493332452 wait complete. 2024-04-24T08:13:11.515 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:11.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.712+0000 7fa6fb200640 1 Processor -- start 2024-04-24T08:13:11.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.712+0000 7fa6fb200640 1 -- start start 2024-04-24T08:13:11.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.713+0000 7fa6fb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 0x7fa6fc06db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.713+0000 7fa6fb200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6fc072e20 0x7fa6fc073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.713+0000 7fa6fb200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa6fc0737a0 0x7fa6fc15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.713+0000 7fa6fb200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa6fc0aeac0 con 0x7fa6fc072e20 2024-04-24T08:13:11.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.713+0000 7fa6fb200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa6fc0aeda0 con 0x7fa6fc0737a0 2024-04-24T08:13:11.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.713+0000 7fa6fb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa6fc0af080 con 0x7fa6fc1523f0 2024-04-24T08:13:11.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6f9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6fc072e20 0x7fa6fc073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6f9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa6fc072e20 0x7fa6fc073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35444/0 (socket says 172.21.15.69:35444) 2024-04-24T08:13:11.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6f9400640 1 -- 172.21.15.69:0/1983720934 learned_addr learned my addr 172.21.15.69:0/1983720934 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:11.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6fa800640 1 --2- 172.21.15.69:0/1983720934 >> v2:172.21.15.121:3300/0 conn(0x7fa6fc0737a0 0x7fa6fc15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6f9400640 1 -- 172.21.15.69:0/1983720934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 msgr2=0x7fa6fc06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:11.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6f9e00640 1 --2- 172.21.15.69:0/1983720934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 0x7fa6fc06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6f9400640 1 --2- 172.21.15.69:0/1983720934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 0x7fa6fc06db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6f9400640 1 -- 172.21.15.69:0/1983720934 >> v2:172.21.15.121:3300/0 conn(0x7fa6fc0737a0 msgr2=0x7fa6fc15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:11.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6f9400640 1 --2- 172.21.15.69:0/1983720934 >> v2:172.21.15.121:3300/0 conn(0x7fa6fc0737a0 0x7fa6fc15ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6f9400640 1 -- 172.21.15.69:0/1983720934 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6fc0af360 con 0x7fa6fc072e20 2024-04-24T08:13:11.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6fa800640 1 --2- 172.21.15.69:0/1983720934 >> v2:172.21.15.121:3300/0 conn(0x7fa6fc0737a0 0x7fa6fc15ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:11.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.714+0000 7fa6f9e00640 1 --2- 172.21.15.69:0/1983720934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 0x7fa6fc06db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:11.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.716+0000 7fa6f9400640 1 --2- 172.21.15.69:0/1983720934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 0x7fa6fc073260 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc15ccf0 tx=0x7fa6e802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af2fdab620cd5d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:11.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.716+0000 7fa6f8a00640 1 -- 172.21.15.69:0/1983720934 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8037070 con 0x7fa6fc072e20 2024-04-24T08:13:11.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.716+0000 7fa6f8a00640 1 -- 172.21.15.69:0/1983720934 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6e8032440 con 0x7fa6fc072e20 2024-04-24T08:13:11.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.716+0000 7fa6f8a00640 1 -- 172.21.15.69:0/1983720934 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8032810 con 0x7fa6fc072e20 2024-04-24T08:13:11.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.716+0000 7fa6fb200640 1 -- 172.21.15.69:0/1983720934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 msgr2=0x7fa6fc073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.716+0000 7fa6fb200640 1 --2- 172.21.15.69:0/1983720934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 0x7fa6fc073260 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc15ccf0 tx=0x7fa6e802a140 comp rx=0 tx=0).stop 2024-04-24T08:13:11.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.717+0000 7fa6fb200640 1 -- 172.21.15.69:0/1983720934 shutdown_connections 2024-04-24T08:13:11.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.717+0000 7fa6fb200640 1 --2- 172.21.15.69:0/1983720934 >> v2:172.21.15.121:3300/0 conn(0x7fa6fc0737a0 0x7fa6fc15ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.717+0000 7fa6fb200640 1 --2- 172.21.15.69:0/1983720934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 0x7fa6fc073260 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.717+0000 7fa6fb200640 1 --2- 172.21.15.69:0/1983720934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 0x7fa6fc06db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.717+0000 7fa6fb200640 1 -- 172.21.15.69:0/1983720934 >> 172.21.15.69:0/1983720934 conn(0x7fa6fc0d0160 msgr2=0x7fa6fc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:11.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.717+0000 7fa6fb200640 1 -- 172.21.15.69:0/1983720934 shutdown_connections 2024-04-24T08:13:11.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.717+0000 7fa6fb200640 1 -- 172.21.15.69:0/1983720934 wait complete. 2024-04-24T08:13:11.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.718+0000 7fa6fb200640 1 Processor -- start 2024-04-24T08:13:11.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.718+0000 7fa6fb200640 1 -- start start 2024-04-24T08:13:11.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.719+0000 7fa6fb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 0x7fa6fc152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.719+0000 7fa6fb200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6fc0737a0 0x7fa6fc15a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.719+0000 7fa6fb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 0x7fa6fc099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.719+0000 7fa6fb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa6fc15f4b0 con 0x7fa6fc072e20 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.719+0000 7fa6fb200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa6fc1527d0 con 0x7fa6fc0737a0 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.719+0000 7fa6fb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa6fc15f630 con 0x7fa6fc1523f0 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6f9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 0x7fa6fc152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6f9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 0x7fa6fc152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35450/0 (socket says 172.21.15.69:35450) 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6f9e00640 1 -- 172.21.15.69:0/458028794 learned_addr learned my addr 172.21.15.69:0/458028794 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6f9400640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6fc0737a0 0x7fa6fc15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6fa800640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 0x7fa6fc099510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6f9e00640 1 -- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 msgr2=0x7fa6fc099510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6f9e00640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 0x7fa6fc099510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6f9e00640 1 -- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6fc0737a0 msgr2=0x7fa6fc15a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6f9e00640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6fc0737a0 0x7fa6fc15a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6f9e00640 1 -- 172.21.15.69:0/458028794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6fc14a410 con 0x7fa6fc072e20 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6fa800640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 0x7fa6fc099510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.720+0000 7fa6f9e00640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 0x7fa6fc152950 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c7f0 tx=0x7fa6ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:11.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.721+0000 7fa6f2000640 1 -- 172.21.15.69:0/458028794 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec015070 con 0x7fa6fc072e20 2024-04-24T08:13:11.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.721+0000 7fa6f2000640 1 -- 172.21.15.69:0/458028794 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6ec007820 con 0x7fa6fc072e20 2024-04-24T08:13:11.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.721+0000 7fa6f2000640 1 -- 172.21.15.69:0/458028794 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec007bd0 con 0x7fa6fc072e20 2024-04-24T08:13:11.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.721+0000 7fa6fb200640 1 -- 172.21.15.69:0/458028794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6fc149370 con 0x7fa6fc072e20 2024-04-24T08:13:11.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.721+0000 7fa6fb200640 1 -- 172.21.15.69:0/458028794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6fc149930 con 0x7fa6fc072e20 2024-04-24T08:13:11.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.722+0000 7fa6fb200640 1 -- 172.21.15.69:0/458028794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6fc075d50 con 0x7fa6fc072e20 2024-04-24T08:13:11.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.723+0000 7fa6f2000640 1 -- 172.21.15.69:0/458028794 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa6ec018070 con 0x7fa6fc072e20 2024-04-24T08:13:11.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.724+0000 7fa6f2000640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6dc06c8f0 0x7fa6dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:11.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.724+0000 7fa6f2000640 1 -- 172.21.15.69:0/458028794 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7fa6ec08b6d0 con 0x7fa6fc072e20 2024-04-24T08:13:11.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.724+0000 7fa6f9400640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6dc06c8f0 0x7fa6dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:11.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.726+0000 7fa6f9400640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6dc06c8f0 0x7fa6dc06edb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8034000 tx=0x7fa6e802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:11.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.728+0000 7fa6f2000640 1 -- 172.21.15.69:0/458028794 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa6ec059e50 con 0x7fa6fc072e20 2024-04-24T08:13:11.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.927+0000 7fa6fb200640 1 -- 172.21.15.69:0/458028794 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6fc0b1bd0 con 0x7fa6fc072e20 2024-04-24T08:13:11.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.929+0000 7fa6f2000640 1 -- 172.21.15.69:0/458028794 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) ==== 74+0+12867 (secure 0 0 0) 0x7fa6ec059750 con 0x7fa6fc072e20 2024-04-24T08:13:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:11.930 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":64,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:13:11.019835+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:11.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.931+0000 7fa6fb200640 1 -- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6dc06c8f0 msgr2=0x7fa6dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.931+0000 7fa6fb200640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6dc06c8f0 0x7fa6dc06edb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8034000 tx=0x7fa6e802e000 comp rx=0 tx=0).stop 2024-04-24T08:13:11.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.931+0000 7fa6fb200640 1 -- 172.21.15.69:0/458028794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 msgr2=0x7fa6fc152950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:11.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.931+0000 7fa6fb200640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 0x7fa6fc152950 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00c7f0 tx=0x7fa6ec00ccb0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.932+0000 7fa6fb200640 1 -- 172.21.15.69:0/458028794 shutdown_connections 2024-04-24T08:13:11.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.932+0000 7fa6fb200640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6dc06c8f0 0x7fa6dc06edb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.932+0000 7fa6fb200640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa6fc1523f0 0x7fa6fc099510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.932+0000 7fa6fb200640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6fc0737a0 0x7fa6fc15a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.932+0000 7fa6fb200640 1 --2- 172.21.15.69:0/458028794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6fc072e20 0x7fa6fc152950 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:11.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.932+0000 7fa6fb200640 1 -- 172.21.15.69:0/458028794 >> 172.21.15.69:0/458028794 conn(0x7fa6fc0d0160 msgr2=0x7fa6fc0addf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:11.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.932+0000 7fa6fb200640 1 -- 172.21.15.69:0/458028794 shutdown_connections 2024-04-24T08:13:11.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:11.932+0000 7fa6fb200640 1 -- 172.21.15.69:0/458028794 wait complete. 2024-04-24T08:13:11.950 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:13:11.950 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:13:12.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.136+0000 7f5e0b200640 1 Processor -- start 2024-04-24T08:13:12.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.136+0000 7f5e0b200640 1 -- start start 2024-04-24T08:13:12.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.138+0000 7f5e0b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5e0c071d70 0x7f5e0c072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.138+0000 7f5e0b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e0c06e240 0x7f5e0c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.138+0000 7f5e0b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e0c0b3700 con 0x7f5e0c15d030 2024-04-24T08:13:12.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.138+0000 7f5e0b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5e0c0b39e0 con 0x7f5e0c071d70 2024-04-24T08:13:12.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.138+0000 7f5e0b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5e0c0b3e20 con 0x7f5e0c06e240 2024-04-24T08:13:12.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.138+0000 7f5e09e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5e0c15d030 0x7f5e0c163b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58870/0 (socket says 172.21.15.69:58870) 2024-04-24T08:13:12.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.138+0000 7f5e09e00640 1 -- 172.21.15.69:0/3073625490 learned_addr learned my addr 172.21.15.69:0/3073625490 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:12.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.138+0000 7f5e03e00640 1 --2- 172.21.15.69:0/3073625490 >> v2:172.21.15.69:3300/0 conn(0x7f5e0c071d70 0x7f5e0c072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:12.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.139+0000 7f5e0a800640 1 --2- 172.21.15.69:0/3073625490 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e0c06e240 0x7f5e0c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:12.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.139+0000 7f5e03e00640 1 -- 172.21.15.69:0/3073625490 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e0c06e240 msgr2=0x7f5e0c06e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:12.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.139+0000 7f5e03e00640 1 --2- 172.21.15.69:0/3073625490 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e0c06e240 0x7f5e0c06e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.139+0000 7f5e03e00640 1 -- 172.21.15.69:0/3073625490 >> v1:172.21.15.69:6789/0 conn(0x7f5e0c15d030 legacy=0x7f5e0c163b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.139+0000 7f5e03e00640 1 -- 172.21.15.69:0/3073625490 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0c0b4100 con 0x7f5e0c071d70 2024-04-24T08:13:12.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.139+0000 7f5e09400640 1 -- 172.21.15.69:0/3073625490 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 777394845 0 0) 0x7f5e0c0b3700 con 0x7f5e0c15d030 2024-04-24T08:13:12.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.140+0000 7f5e03e00640 1 --2- 172.21.15.69:0/3073625490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 0x7f5e0c072220 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5df40097a0 tx=0x7f5df402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b087263d2fa506e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:12.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.140+0000 7f5e09400640 1 -- 172.21.15.69:0/3073625490 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4037070 con 0x7f5e0c071d70 2024-04-24T08:13:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.140+0000 7f5e09400640 1 -- 172.21.15.69:0/3073625490 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df4032440 con 0x7f5e0c071d70 2024-04-24T08:13:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.140+0000 7f5e09400640 1 -- 172.21.15.69:0/3073625490 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4032810 con 0x7f5e0c071d70 2024-04-24T08:13:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.141+0000 7f5e0b200640 1 -- 172.21.15.69:0/3073625490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 msgr2=0x7f5e0c072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.141+0000 7f5e0b200640 1 --2- 172.21.15.69:0/3073625490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 0x7f5e0c072220 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f5df40097a0 tx=0x7f5df402a180 comp rx=0 tx=0).stop 2024-04-24T08:13:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.141+0000 7f5e0b200640 1 -- 172.21.15.69:0/3073625490 shutdown_connections 2024-04-24T08:13:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.141+0000 7f5e0b200640 1 --2- 172.21.15.69:0/3073625490 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e0c06e240 0x7f5e0c06e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.141+0000 7f5e0b200640 1 --2- 172.21.15.69:0/3073625490 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 0x7f5e0c072220 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.141+0000 7f5e0b200640 1 -- 172.21.15.69:0/3073625490 >> 172.21.15.69:0/3073625490 conn(0x7f5e0c067b80 msgr2=0x7f5e0c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.141+0000 7f5e0b200640 1 -- 172.21.15.69:0/3073625490 shutdown_connections 2024-04-24T08:13:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.141+0000 7f5e0b200640 1 -- 172.21.15.69:0/3073625490 wait complete. 2024-04-24T08:13:12.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.144+0000 7f5e0b200640 1 Processor -- start 2024-04-24T08:13:12.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.144+0000 7f5e0b200640 1 -- start start 2024-04-24T08:13:12.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.145+0000 7f5e0b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e0c06e240 0x7f5e0c151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.145+0000 7f5e0b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 0x7f5e0c0b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.145+0000 7f5e0b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e0c15d030 0x7f5e0c15e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.145+0000 7f5e0b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c0b3320 con 0x7f5e0c071d70 2024-04-24T08:13:12.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.145+0000 7f5e0b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5e0c06f7b0 con 0x7f5e0c06e240 2024-04-24T08:13:12.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.145+0000 7f5e0b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5e0c0b34a0 con 0x7f5e0c15d030 2024-04-24T08:13:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.145+0000 7f5e03e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 0x7f5e0c0b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.145+0000 7f5e03e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 0x7f5e0c0b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35468/0 (socket says 172.21.15.69:35468) 2024-04-24T08:13:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.145+0000 7f5e03e00640 1 -- 172.21.15.69:0/739243432 learned_addr learned my addr 172.21.15.69:0/739243432 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.146+0000 7f5e03e00640 1 -- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e0c15d030 msgr2=0x7f5e0c15e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.146+0000 7f5e03e00640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e0c15d030 0x7f5e0c15e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.146+0000 7f5e03e00640 1 -- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e0c06e240 msgr2=0x7f5e0c151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.146+0000 7f5e03e00640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e0c06e240 0x7f5e0c151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.146+0000 7f5e03e00640 1 -- 172.21.15.69:0/739243432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e0c153800 con 0x7f5e0c071d70 2024-04-24T08:13:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.146+0000 7f5e03e00640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 0x7f5e0c0b1e80 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5df4035040 tx=0x7f5df40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:12.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.146+0000 7f5e02000640 1 -- 172.21.15.69:0/739243432 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4037040 con 0x7f5e0c071d70 2024-04-24T08:13:12.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.146+0000 7f5e02000640 1 -- 172.21.15.69:0/739243432 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df4030770 con 0x7f5e0c071d70 2024-04-24T08:13:12.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.146+0000 7f5e0b200640 1 -- 172.21.15.69:0/739243432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e0c15d950 con 0x7f5e0c071d70 2024-04-24T08:13:12.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.147+0000 7f5e02000640 1 -- 172.21.15.69:0/739243432 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4030b40 con 0x7f5e0c071d70 2024-04-24T08:13:12.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.147+0000 7f5e0b200640 1 -- 172.21.15.69:0/739243432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e0c15df10 con 0x7f5e0c071d70 2024-04-24T08:13:12.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.148+0000 7f5e0b200640 1 -- 172.21.15.69:0/739243432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e0c075d50 con 0x7f5e0c071d70 2024-04-24T08:13:12.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.149+0000 7f5e02000640 1 -- 172.21.15.69:0/739243432 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5df4044020 con 0x7f5e0c071d70 2024-04-24T08:13:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.150+0000 7f5e02000640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5de806c8f0 0x7f5de806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.150+0000 7f5e02000640 1 -- 172.21.15.69:0/739243432 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7f5df40abd80 con 0x7f5e0c071d70 2024-04-24T08:13:12.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.151+0000 7f5e09e00640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5de806c8f0 0x7f5de806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:12.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.152+0000 7f5e09e00640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5de806c8f0 0x7f5de806edb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc007fd0 tx=0x7f5dfc007960 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:12.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.154+0000 7f5e02000640 1 -- 172.21.15.69:0/739243432 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5df4032e20 con 0x7f5e0c071d70 2024-04-24T08:13:12.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.352+0000 7f5e0b200640 1 -- 172.21.15.69:0/739243432 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5e0c0d0d00 con 0x7f5e0c071d70 2024-04-24T08:13:12.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.353+0000 7f5e02000640 1 -- 172.21.15.69:0/739243432 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) ==== 93+0+10 (secure 0 0 0) 0x7f5df407a460 con 0x7f5e0c071d70 2024-04-24T08:13:12.353 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:13:12.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.355+0000 7f5e0b200640 1 -- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5de806c8f0 msgr2=0x7f5de806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.355+0000 7f5e0b200640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5de806c8f0 0x7f5de806edb0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc007fd0 tx=0x7f5dfc007960 comp rx=0 tx=0).stop 2024-04-24T08:13:12.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.355+0000 7f5e0b200640 1 -- 172.21.15.69:0/739243432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 msgr2=0x7f5e0c0b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.355+0000 7f5e0b200640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 0x7f5e0c0b1e80 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f5df4035040 tx=0x7f5df40046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.355+0000 7f5e0b200640 1 -- 172.21.15.69:0/739243432 shutdown_connections 2024-04-24T08:13:12.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.355+0000 7f5e0b200640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5de806c8f0 0x7f5de806edb0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.355+0000 7f5e0b200640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5e0c071d70 0x7f5e0c0b1e80 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.355+0000 7f5e0b200640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5e0c15d030 0x7f5e0c15e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.355+0000 7f5e0b200640 1 --2- 172.21.15.69:0/739243432 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e0c06e240 0x7f5e0c151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.355+0000 7f5e0b200640 1 -- 172.21.15.69:0/739243432 >> 172.21.15.69:0/739243432 conn(0x7f5e0c067b80 msgr2=0x7f5e0c15b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:12.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.356+0000 7f5e0b200640 1 -- 172.21.15.69:0/739243432 shutdown_connections 2024-04-24T08:13:12.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.356+0000 7f5e0b200640 1 -- 172.21.15.69:0/739243432 wait complete. 2024-04-24T08:13:12.375 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T08:13:12.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.571+0000 7f9de0e00640 1 Processor -- start 2024-04-24T08:13:12.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.571+0000 7f9de0e00640 1 -- start start 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.572+0000 7f9de0e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9ddc15e960 0x7f9ddc162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9de0e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9ddc0cfaf0 0x7f9ddc0cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9de0e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ddc15ed40 con 0x7f9ddc162980 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9de0e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9ddc0cf050 con 0x7f9ddc15e960 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9de0e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9ddc162e20 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9ddb400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9ddc15e960 0x7f9ddc162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9ddaa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9ddc162980 0x7f9ddc06cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58878/0 (socket says 172.21.15.69:58878) 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9ddaa00640 1 -- 172.21.15.69:0/3926544859 learned_addr learned my addr 172.21.15.69:0/3926544859 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9ddbe00640 1 --2- 172.21.15.69:0/3926544859 >> v2:172.21.15.121:3300/0 conn(0x7f9ddc0cfaf0 0x7f9ddc0cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9ddb400640 1 -- 172.21.15.69:0/3926544859 >> v2:172.21.15.121:3300/0 conn(0x7f9ddc0cfaf0 msgr2=0x7f9ddc0cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9ddb400640 1 --2- 172.21.15.69:0/3926544859 >> v2:172.21.15.121:3300/0 conn(0x7f9ddc0cfaf0 0x7f9ddc0cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9ddb400640 1 -- 172.21.15.69:0/3926544859 >> v1:172.21.15.69:6789/0 conn(0x7f9ddc162980 legacy=0x7f9ddc06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9ddb400640 1 -- 172.21.15.69:0/3926544859 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ddc163100 con 0x7f9ddc15e960 2024-04-24T08:13:12.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.573+0000 7f9ddbe00640 1 --2- 172.21.15.69:0/3926544859 >> v2:172.21.15.121:3300/0 conn(0x7f9ddc0cfaf0 0x7f9ddc0cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.574+0000 7f9ddb400640 1 --2- 172.21.15.69:0/3926544859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc15e960 0x7f9ddc162440 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc08de10 tx=0x7f9dcc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90c11176a3c10fc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.575+0000 7f9dda000640 1 -- 172.21.15.69:0/3926544859 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc037070 con 0x7f9ddc15e960 2024-04-24T08:13:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.575+0000 7f9dda000640 1 -- 172.21.15.69:0/3926544859 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dcc032440 con 0x7f9ddc15e960 2024-04-24T08:13:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.575+0000 7f9dda000640 1 -- 172.21.15.69:0/3926544859 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc032810 con 0x7f9ddc15e960 2024-04-24T08:13:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.575+0000 7f9de0e00640 1 -- 172.21.15.69:0/3926544859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc15e960 msgr2=0x7f9ddc162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.575+0000 7f9de0e00640 1 --2- 172.21.15.69:0/3926544859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc15e960 0x7f9ddc162440 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc08de10 tx=0x7f9dcc02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.575+0000 7f9de0e00640 1 -- 172.21.15.69:0/3926544859 shutdown_connections 2024-04-24T08:13:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.575+0000 7f9de0e00640 1 --2- 172.21.15.69:0/3926544859 >> v2:172.21.15.121:3300/0 conn(0x7f9ddc0cfaf0 0x7f9ddc0cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.575+0000 7f9de0e00640 1 --2- 172.21.15.69:0/3926544859 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc15e960 0x7f9ddc162440 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.575+0000 7f9de0e00640 1 -- 172.21.15.69:0/3926544859 >> 172.21.15.69:0/3926544859 conn(0x7f9ddc09d900 msgr2=0x7f9ddc08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:12.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.575+0000 7f9de0e00640 1 -- 172.21.15.69:0/3926544859 shutdown_connections 2024-04-24T08:13:12.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.576+0000 7f9de0e00640 1 -- 172.21.15.69:0/3926544859 wait complete. 2024-04-24T08:13:12.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.577+0000 7f9de0e00640 1 Processor -- start 2024-04-24T08:13:12.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.577+0000 7f9de0e00640 1 -- start start 2024-04-24T08:13:12.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.578+0000 7f9de0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc0cfaf0 0x7f9ddc06df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.578+0000 7f9de0e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ddc15e960 0x7f9ddc15f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.578+0000 7f9de0e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ddc162980 0x7f9ddc161440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.578+0000 7f9de0e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9ddc06d6c0 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.578+0000 7f9de0e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9ddc0cf5d0 con 0x7f9ddc162980 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.578+0000 7f9de0e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9ddc0b6170 con 0x7f9ddc15e960 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.578+0000 7f9ddb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc0cfaf0 0x7f9ddc06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.578+0000 7f9ddb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc0cfaf0 0x7f9ddc06df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35484/0 (socket says 172.21.15.69:35484) 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.578+0000 7f9ddb400640 1 -- 172.21.15.69:0/2594721093 learned_addr learned my addr 172.21.15.69:0/2594721093 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.579+0000 7f9ddb400640 1 -- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ddc15e960 msgr2=0x7f9ddc15f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.579+0000 7f9ddaa00640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ddc15e960 0x7f9ddc15f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.579+0000 7f9ddbe00640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ddc162980 0x7f9ddc161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.579+0000 7f9ddb400640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ddc15e960 0x7f9ddc15f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.579+0000 7f9ddb400640 1 -- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ddc162980 msgr2=0x7f9ddc161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.579+0000 7f9ddb400640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ddc162980 0x7f9ddc161440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.579+0000 7f9ddb400640 1 -- 172.21.15.69:0/2594721093 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ddc0c86c0 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.579+0000 7f9ddb400640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc0cfaf0 0x7f9ddc06df40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc000c00 tx=0x7f9dcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.579+0000 7f9dd3400640 1 -- 172.21.15.69:0/2594721093 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc037040 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.580+0000 7f9dd3400640 1 -- 172.21.15.69:0/2594721093 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dcc042070 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.580+0000 7f9de0e00640 1 -- 172.21.15.69:0/2594721093 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ddc0c89a0 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.580+0000 7f9dd3400640 1 -- 172.21.15.69:0/2594721093 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dcc030a50 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.580+0000 7f9de0e00640 1 -- 172.21.15.69:0/2594721093 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ddc0c8f60 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.581+0000 7f9de0e00640 1 -- 172.21.15.69:0/2594721093 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ddc075d50 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.582+0000 7f9dd3400640 1 -- 172.21.15.69:0/2594721093 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9dcc030bf0 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.583+0000 7f9dd3400640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dbc06c8f0 0x7f9dbc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:12.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.583+0000 7f9dd3400640 1 -- 172.21.15.69:0/2594721093 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7f9dcc0ac2a0 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.583+0000 7f9ddaa00640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dbc06c8f0 0x7f9dbc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:12.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.585+0000 7f9ddaa00640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dbc06c8f0 0x7f9dbc06edb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0ceaf0 tx=0x7f9dc8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:12.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.587+0000 7f9dd3400640 1 -- 172.21.15.69:0/2594721093 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9dcc045360 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.788+0000 7f9de0e00640 1 -- 172.21.15.69:0/2594721093 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f9ddc0d1dc0 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.789+0000 7f9dd3400640 1 -- 172.21.15.69:0/2594721093 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) ==== 84+0+65 (secure 0 0 0) 0x7f9dcc07aa10 con 0x7f9ddc0cfaf0 2024-04-24T08:13:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:12.790 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:13:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:13:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "poolnum": 1, 2024-04-24T08:13:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "poolname": ".mgr" 2024-04-24T08:13:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:12.790 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:13:12.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 -- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dbc06c8f0 msgr2=0x7f9dbc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dbc06c8f0 0x7f9dbc06edb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc0ceaf0 tx=0x7f9dc8009040 comp rx=0 tx=0).stop 2024-04-24T08:13:12.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 -- 172.21.15.69:0/2594721093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc0cfaf0 msgr2=0x7f9ddc06df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc0cfaf0 0x7f9ddc06df40 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc000c00 tx=0x7f9dcc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 -- 172.21.15.69:0/2594721093 shutdown_connections 2024-04-24T08:13:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9dbc06c8f0 0x7f9dbc06edb0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ddc162980 0x7f9ddc161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9ddc15e960 0x7f9ddc15f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 --2- 172.21.15.69:0/2594721093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ddc0cfaf0 0x7f9ddc06df40 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 -- 172.21.15.69:0/2594721093 >> 172.21.15.69:0/2594721093 conn(0x7f9ddc09d900 msgr2=0x7f9ddc0b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 -- 172.21.15.69:0/2594721093 shutdown_connections 2024-04-24T08:13:12.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:12.792+0000 7f9de0e00640 1 -- 172.21.15.69:0/2594721093 wait complete. 2024-04-24T08:13:12.812 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T08:13:12.812 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T08:13:13.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.023+0000 7fb10be00640 1 Processor -- start 2024-04-24T08:13:13.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.024+0000 7fb10be00640 1 -- start start 2024-04-24T08:13:13.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.025+0000 7fb10be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb10c0af5b0 0x7fb10c1623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.025+0000 7fb10be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb10c1632b0 con 0x7fb10c15e930 2024-04-24T08:13:13.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.025+0000 7fb10be00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb10c162a10 con 0x7fb10c0cd000 2024-04-24T08:13:13.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.025+0000 7fb10be00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb10c15ed10 con 0x7fb10c0af5b0 2024-04-24T08:13:13.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.025+0000 7fb10aa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb10c15e930 0x7fb10c0cc8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58880/0 (socket says 172.21.15.69:58880) 2024-04-24T08:13:13.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.025+0000 7fb10aa00640 1 -- 172.21.15.69:0/1457035089 learned_addr learned my addr 172.21.15.69:0/1457035089 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:13.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10b400640 1 --2- 172.21.15.69:0/1457035089 >> v2:172.21.15.121:3300/0 conn(0x7fb10c0af5b0 0x7fb10c1623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 520926416 0 0) 0x7fb10c1632b0 con 0x7fb10c15e930 2024-04-24T08:13:13.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0f0003520 con 0x7fb10c15e930 2024-04-24T08:13:13.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3527901447 0 0) 0x7fb0f0003520 con 0x7fb10c15e930 2024-04-24T08:13:13.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 >> v2:172.21.15.121:3300/0 conn(0x7fb10c0af5b0 msgr2=0x7fb10c1623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:13.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 --2- 172.21.15.69:0/1457035089 >> v2:172.21.15.121:3300/0 conn(0x7fb10c0af5b0 0x7fb10c1623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 >> v1:172.21.15.121:6789/0 conn(0x7fb10c0cd000 legacy=0x7fb10c0cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c162cf0 con 0x7fb10c15e930 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fb0fc001670 con 0x7fb10c15e930 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 >> v1:172.21.15.69:6789/0 conn(0x7fb10c15e930 legacy=0x7fb10c0cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 --2- 172.21.15.69:0/1457035089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f0003a00 0x7fb0f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 --2- 172.21.15.69:0/1457035089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0f0004800 0x7fb0f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 --2- 172.21.15.69:0/1457035089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0f0005210 0x7fb0f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f000c130 con 0x7fb0f0003a00 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f000c2e0 con 0x7fb0f0004800 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.026+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb0f000c4e0 con 0x7fb0f0005210 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.027+0000 7fb10aa00640 1 --2- 172.21.15.69:0/1457035089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f0003a00 0x7fb0f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.027+0000 7fb103e00640 1 --2- 172.21.15.69:0/1457035089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0f0004800 0x7fb0f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.027+0000 7fb10aa00640 1 -- 172.21.15.69:0/1457035089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0f0005210 msgr2=0x7fb0f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:13.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.027+0000 7fb10aa00640 1 --2- 172.21.15.69:0/1457035089 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0f0005210 0x7fb0f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.027+0000 7fb10aa00640 1 -- 172.21.15.69:0/1457035089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0f0004800 msgr2=0x7fb0f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:13.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.027+0000 7fb10aa00640 1 --2- 172.21.15.69:0/1457035089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0f0004800 0x7fb0f0004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.027+0000 7fb10aa00640 1 -- 172.21.15.69:0/1457035089 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f0003520 con 0x7fb0f0003a00 2024-04-24T08:13:13.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.027+0000 7fb10b400640 1 -- 172.21.15.69:0/1457035089 reap_dead start 2024-04-24T08:13:13.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.028+0000 7fb10aa00640 1 --2- 172.21.15.69:0/1457035089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f0003a00 0x7fb0f0003e20 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00c0e0 tx=0x7fb0f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9f9ac50857e6b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.028+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc007c40 con 0x7fb0f0003a00 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.028+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0fc005af0 con 0x7fb0f0003a00 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.028+0000 7fb10a000640 1 -- 172.21.15.69:0/1457035089 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc007070 con 0x7fb0f0003a00 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.029+0000 7fb10be00640 1 -- 172.21.15.69:0/1457035089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f0003a00 msgr2=0x7fb0f0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.029+0000 7fb10be00640 1 --2- 172.21.15.69:0/1457035089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f0003a00 0x7fb0f0003e20 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00c0e0 tx=0x7fb0f0005830 comp rx=0 tx=0).stop 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.029+0000 7fb10be00640 1 -- 172.21.15.69:0/1457035089 shutdown_connections 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.029+0000 7fb10be00640 1 --2- 172.21.15.69:0/1457035089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f0003a00 0x7fb0f0003e20 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.029+0000 7fb10be00640 1 -- 172.21.15.69:0/1457035089 >> 172.21.15.69:0/1457035089 conn(0x7fb10c0b1c50 msgr2=0x7fb10c0b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.029+0000 7fb10be00640 1 -- 172.21.15.69:0/1457035089 shutdown_connections 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.029+0000 7fb10be00640 1 -- 172.21.15.69:0/1457035089 wait complete. 2024-04-24T08:13:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.031+0000 7fb10be00640 1 Processor -- start 2024-04-24T08:13:13.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.031+0000 7fb10be00640 1 -- start start 2024-04-24T08:13:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.032+0000 7fb10be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0f0003a00 0x7fb10c1578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.032+0000 7fb10be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb10c0c5890 0x7fb10c160270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.032+0000 7fb10be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb10c0c5d10 0x7fb10c1607b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.032+0000 7fb10be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c0ca920 con 0x7fb10c0c5890 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.032+0000 7fb10be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c0cacf0 con 0x7fb10c0c5d10 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.032+0000 7fb10be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb10c1623a0 con 0x7fb0f0003a00 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.032+0000 7fb103e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb10c0c5890 0x7fb10c160270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.032+0000 7fb103e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb10c0c5890 0x7fb10c160270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35502/0 (socket says 172.21.15.69:35502) 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.032+0000 7fb103e00640 1 -- 172.21.15.69:0/204627926 learned_addr learned my addr 172.21.15.69:0/204627926 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb10aa00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0f0003a00 0x7fb10c1578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb10b400640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb10c0c5d10 0x7fb10c1607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb103e00640 1 -- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0f0003a00 msgr2=0x7fb10c1578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb103e00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0f0003a00 0x7fb10c1578e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb103e00640 1 -- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb10c0c5d10 msgr2=0x7fb10c1607b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb103e00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb10c0c5d10 0x7fb10c1607b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb103e00640 1 -- 172.21.15.69:0/204627926 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c0c6a60 con 0x7fb10c0c5890 2024-04-24T08:13:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb103e00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb10c0c5890 0x7fb10c160270 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400d4a0 tx=0x7fb0f400d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb102a00640 1 -- 172.21.15.69:0/204627926 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4005480 con 0x7fb10c0c5890 2024-04-24T08:13:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb102a00640 1 -- 172.21.15.69:0/204627926 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f4017070 con 0x7fb10c0c5890 2024-04-24T08:13:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb10be00640 1 -- 172.21.15.69:0/204627926 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10c15f830 con 0x7fb10c0c5890 2024-04-24T08:13:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.033+0000 7fb102a00640 1 -- 172.21.15.69:0/204627926 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4003810 con 0x7fb10c0c5890 2024-04-24T08:13:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.034+0000 7fb10be00640 1 -- 172.21.15.69:0/204627926 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10c15fdf0 con 0x7fb10c0c5890 2024-04-24T08:13:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.035+0000 7fb10be00640 1 -- 172.21.15.69:0/204627926 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb10c075d50 con 0x7fb10c0c5890 2024-04-24T08:13:13.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.036+0000 7fb102a00640 1 -- 172.21.15.69:0/204627926 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb0f40039b0 con 0x7fb10c0c5890 2024-04-24T08:13:13.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.036+0000 7fb102a00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb0e806c920 0x7fb0e806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.037+0000 7fb102a00640 1 -- 172.21.15.69:0/204627926 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 4857+0+0 (secure 0 0 0) 0x7fb0f408b4f0 con 0x7fb10c0c5890 2024-04-24T08:13:13.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.037+0000 7fb10aa00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb0e806c920 0x7fb0e806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.038+0000 7fb10aa00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb0e806c920 0x7fb0e806ede0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc032000 tx=0x7fb0fc02d000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.041+0000 7fb102a00640 1 -- 172.21.15.69:0/204627926 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb0f4059d20 con 0x7fb10c0c5890 2024-04-24T08:13:13.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:13.236+0000 7fb10be00640 1 -- 172.21.15.69:0/204627926 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb10c0cfcf0 con 0x7fb10c0c5890 2024-04-24T08:13:14.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.027+0000 7fb102a00640 1 -- 172.21.15.69:0/204627926 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) ==== 138+0+0 (secure 0 0 0) 0x7fb0f4059620 con 0x7fb10c0c5890 2024-04-24T08:13:14.028 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' created 2024-04-24T08:13:14.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.031+0000 7fb10be00640 1 -- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb0e806c920 msgr2=0x7fb0e806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.031+0000 7fb10be00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb0e806c920 0x7fb0e806ede0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc032000 tx=0x7fb0fc02d000 comp rx=0 tx=0).stop 2024-04-24T08:13:14.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.031+0000 7fb10be00640 1 -- 172.21.15.69:0/204627926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb10c0c5890 msgr2=0x7fb10c160270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.031+0000 7fb10be00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb10c0c5890 0x7fb10c160270 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400d4a0 tx=0x7fb0f400d960 comp rx=0 tx=0).stop 2024-04-24T08:13:14.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.031+0000 7fb10be00640 1 -- 172.21.15.69:0/204627926 shutdown_connections 2024-04-24T08:13:14.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.031+0000 7fb10be00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb0e806c920 0x7fb0e806ede0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.031+0000 7fb10be00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb10c0c5890 0x7fb10c160270 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.031+0000 7fb10be00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb10c0c5d10 0x7fb10c1607b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.031+0000 7fb10be00640 1 --2- 172.21.15.69:0/204627926 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb0f0003a00 0x7fb10c1578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.031+0000 7fb10be00640 1 -- 172.21.15.69:0/204627926 >> 172.21.15.69:0/204627926 conn(0x7fb10c0b1c50 msgr2=0x7fb10c0d00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:14.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.032+0000 7fb10be00640 1 -- 172.21.15.69:0/204627926 shutdown_connections 2024-04-24T08:13:14.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.032+0000 7fb10be00640 1 -- 172.21.15.69:0/204627926 wait complete. 2024-04-24T08:13:14.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T08:13:14.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.253+0000 7fbef3200640 1 Processor -- start 2024-04-24T08:13:14.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.253+0000 7fbef3200640 1 -- start start 2024-04-24T08:13:14.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.255+0000 7fbef3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef40b2cd0 0x7fbef40b0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.255+0000 7fbef3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbef406eb80 0x7fbef406f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.255+0000 7fbef3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbef4072bc0 con 0x7fbef40700e0 2024-04-24T08:13:14.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.255+0000 7fbef3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbef4073480 con 0x7fbef406eb80 2024-04-24T08:13:14.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.255+0000 7fbef3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbef4072e30 con 0x7fbef40b2cd0 2024-04-24T08:13:14.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.255+0000 7fbef2800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbef406eb80 0x7fbef406f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.255+0000 7fbef1400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbef40700e0 0x7fbef40b30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58886/0 (socket says 172.21.15.69:58886) 2024-04-24T08:13:14.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.255+0000 7fbef1400640 1 -- 172.21.15.69:0/2024125841 learned_addr learned my addr 172.21.15.69:0/2024125841 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:14.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.256+0000 7fbef1e00640 1 --2- 172.21.15.69:0/2024125841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef40b2cd0 0x7fbef40b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.256+0000 7fbef2800640 1 -- 172.21.15.69:0/2024125841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef40b2cd0 msgr2=0x7fbef40b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:14.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.256+0000 7fbef2800640 1 --2- 172.21.15.69:0/2024125841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef40b2cd0 0x7fbef40b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.256+0000 7fbef2800640 1 -- 172.21.15.69:0/2024125841 >> v1:172.21.15.69:6789/0 conn(0x7fbef40700e0 legacy=0x7fbef40b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.256+0000 7fbef2800640 1 -- 172.21.15.69:0/2024125841 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef4073110 con 0x7fbef406eb80 2024-04-24T08:13:14.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.257+0000 7fbef2800640 1 --2- 172.21.15.69:0/2024125841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 0x7fbef406f010 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00bfa0 tx=0x7fbeec02c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b380a798b9097c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:14.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.257+0000 7fbef0a00640 1 -- 172.21.15.69:0/2024125841 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec039070 con 0x7fbef406eb80 2024-04-24T08:13:14.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.257+0000 7fbef0a00640 1 -- 172.21.15.69:0/2024125841 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeec009540 con 0x7fbef406eb80 2024-04-24T08:13:14.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.257+0000 7fbef0a00640 1 -- 172.21.15.69:0/2024125841 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeec009910 con 0x7fbef406eb80 2024-04-24T08:13:14.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.258+0000 7fbef3200640 1 -- 172.21.15.69:0/2024125841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 msgr2=0x7fbef406f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.258+0000 7fbef3200640 1 --2- 172.21.15.69:0/2024125841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 0x7fbef406f010 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00bfa0 tx=0x7fbeec02c860 comp rx=0 tx=0).stop 2024-04-24T08:13:14.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.258+0000 7fbef3200640 1 -- 172.21.15.69:0/2024125841 shutdown_connections 2024-04-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.258+0000 7fbef3200640 1 --2- 172.21.15.69:0/2024125841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 0x7fbef406f010 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.258+0000 7fbef3200640 1 --2- 172.21.15.69:0/2024125841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef40b2cd0 0x7fbef40b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.258+0000 7fbef3200640 1 -- 172.21.15.69:0/2024125841 >> 172.21.15.69:0/2024125841 conn(0x7fbef40cdcd0 msgr2=0x7fbef40ce0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.258+0000 7fbef3200640 1 -- 172.21.15.69:0/2024125841 shutdown_connections 2024-04-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.258+0000 7fbef3200640 1 -- 172.21.15.69:0/2024125841 wait complete. 2024-04-24T08:13:14.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.259+0000 7fbef3200640 1 Processor -- start 2024-04-24T08:13:14.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.260+0000 7fbef3200640 1 -- start start 2024-04-24T08:13:14.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.260+0000 7fbef3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 0x7fbef415a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.260+0000 7fbef3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef40700e0 0x7fbef415b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.260+0000 7fbef3200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbef40b2cd0 0x7fbef41f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.260+0000 7fbef3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbef4073910 con 0x7fbef406eb80 2024-04-24T08:13:14.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.260+0000 7fbef3200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbef40b08e0 con 0x7fbef40b2cd0 2024-04-24T08:13:14.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.260+0000 7fbef3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbef414a480 con 0x7fbef40700e0 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.261+0000 7fbef1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 0x7fbef415a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.261+0000 7fbef1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 0x7fbef415a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35520/0 (socket says 172.21.15.69:35520) 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.261+0000 7fbef1e00640 1 -- 172.21.15.69:0/1494743923 learned_addr learned my addr 172.21.15.69:0/1494743923 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.261+0000 7fbef1e00640 1 -- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef40700e0 msgr2=0x7fbef415b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.261+0000 7fbef1e00640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef40700e0 0x7fbef415b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.261+0000 7fbef1e00640 1 -- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbef40b2cd0 msgr2=0x7fbef41f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.261+0000 7fbef1e00640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbef40b2cd0 0x7fbef41f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.261+0000 7fbef1e00640 1 -- 172.21.15.69:0/1494743923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbef4149870 con 0x7fbef406eb80 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.261+0000 7fbef1e00640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 0x7fbef415a6f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fbee000d620 tx=0x7fbee000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.262+0000 7fbeea000640 1 -- 172.21.15.69:0/1494743923 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee000b620 con 0x7fbef406eb80 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.262+0000 7fbeea000640 1 -- 172.21.15.69:0/1494743923 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbee0019070 con 0x7fbef406eb80 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.262+0000 7fbeea000640 1 -- 172.21.15.69:0/1494743923 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbee000ba00 con 0x7fbef406eb80 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.262+0000 7fbef3200640 1 -- 172.21.15.69:0/1494743923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbef4149b50 con 0x7fbef406eb80 2024-04-24T08:13:14.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.262+0000 7fbef3200640 1 -- 172.21.15.69:0/1494743923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbef414a110 con 0x7fbef406eb80 2024-04-24T08:13:14.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.263+0000 7fbef3200640 1 -- 172.21.15.69:0/1494743923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbef4075d50 con 0x7fbef406eb80 2024-04-24T08:13:14.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.264+0000 7fbeea000640 1 -- 172.21.15.69:0/1494743923 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbee000bba0 con 0x7fbef406eb80 2024-04-24T08:13:14.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.265+0000 7fbeea000640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbed406c920 0x7fbed406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.265+0000 7fbeea000640 1 -- 172.21.15.69:0/1494743923 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5227+0+0 (secure 0 0 0) 0x7fbee008be20 con 0x7fbef406eb80 2024-04-24T08:13:14.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.266+0000 7fbef1400640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbed406c920 0x7fbed406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.267+0000 7fbef1400640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbed406c920 0x7fbed406ede0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbee4000f60 tx=0x7fbee4007a80 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:14.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.270+0000 7fbeea000640 1 -- 172.21.15.69:0/1494743923 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbee005a390 con 0x7fbef406eb80 2024-04-24T08:13:14.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:14.454+0000 7fbef3200640 1 -- 172.21.15.69:0/1494743923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbef40ce7c0 con 0x7fbef406eb80 2024-04-24T08:13:14.633 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:13:14.641 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:13:15.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.035+0000 7fbeea000640 1 -- 172.21.15.69:0/1494743923 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) ==== 170+0+0 (secure 0 0 0) 0x7fbee0059c90 con 0x7fbef406eb80 2024-04-24T08:13:15.035 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' created 2024-04-24T08:13:15.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.039+0000 7fbef3200640 1 -- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbed406c920 msgr2=0x7fbed406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbed406c920 0x7fbed406ede0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbee4000f60 tx=0x7fbee4007a80 comp rx=0 tx=0).stop 2024-04-24T08:13:15.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 -- 172.21.15.69:0/1494743923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 msgr2=0x7fbef415a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 0x7fbef415a6f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fbee000d620 tx=0x7fbee000dae0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 -- 172.21.15.69:0/1494743923 shutdown_connections 2024-04-24T08:13:15.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbed406c920 0x7fbed406ede0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbef40b2cd0 0x7fbef41f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbef40700e0 0x7fbef415b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 --2- 172.21.15.69:0/1494743923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbef406eb80 0x7fbef415a6f0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 -- 172.21.15.69:0/1494743923 >> 172.21.15.69:0/1494743923 conn(0x7fbef40cdcd0 msgr2=0x7fbef40b44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 -- 172.21.15.69:0/1494743923 shutdown_connections 2024-04-24T08:13:15.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.040+0000 7fbef3200640 1 -- 172.21.15.69:0/1494743923 wait complete. 2024-04-24T08:13:15.072 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T08:13:15.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.246+0000 7f5086e00640 1 Processor -- start 2024-04-24T08:13:15.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.246+0000 7f5086e00640 1 -- start start 2024-04-24T08:13:15.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.248+0000 7f5086e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f508015f4b0 0x7f50800b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.248+0000 7f5086e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f508015f890 0x7f50800732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.248+0000 7f5086e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5080073cc0 con 0x7f5080073820 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.248+0000 7f5086e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5080073fa0 con 0x7f508015f890 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.248+0000 7f5086e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f50800b3300 con 0x7f508015f4b0 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.248+0000 7f507f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f508015f890 0x7f50800732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.248+0000 7f5084e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5080073820 0x7f50800b46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58894/0 (socket says 172.21.15.69:58894) 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.248+0000 7f5084e00640 1 -- 172.21.15.69:0/3835355263 learned_addr learned my addr 172.21.15.69:0/3835355263 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:15.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.249+0000 7f507fe00640 1 --2- 172.21.15.69:0/3835355263 >> v2:172.21.15.121:3300/0 conn(0x7f508015f4b0 0x7f50800b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.249+0000 7f507f400640 1 -- 172.21.15.69:0/3835355263 >> v2:172.21.15.121:3300/0 conn(0x7f508015f4b0 msgr2=0x7f50800b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.249+0000 7f507f400640 1 --2- 172.21.15.69:0/3835355263 >> v2:172.21.15.121:3300/0 conn(0x7f508015f4b0 0x7f50800b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.249+0000 7f507f400640 1 -- 172.21.15.69:0/3835355263 >> v1:172.21.15.69:6789/0 conn(0x7f5080073820 legacy=0x7f50800b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.249+0000 7f507f400640 1 -- 172.21.15.69:0/3835355263 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50800b35e0 con 0x7f508015f890 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.250+0000 7f507f400640 1 --2- 172.21.15.69:0/3835355263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f508015f890 0x7f50800732e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f50800cde60 tx=0x7f506802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39498a1bb0eefe28 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.250+0000 7f507ea00640 1 -- 172.21.15.69:0/3835355263 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068037070 con 0x7f508015f890 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.250+0000 7f507ea00640 1 -- 172.21.15.69:0/3835355263 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5068032440 con 0x7f508015f890 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.250+0000 7f507ea00640 1 -- 172.21.15.69:0/3835355263 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068032810 con 0x7f508015f890 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.251+0000 7f5086e00640 1 -- 172.21.15.69:0/3835355263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f508015f890 msgr2=0x7f50800732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.251+0000 7f5086e00640 1 --2- 172.21.15.69:0/3835355263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f508015f890 0x7f50800732e0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f50800cde60 tx=0x7f506802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.251+0000 7f5086e00640 1 -- 172.21.15.69:0/3835355263 shutdown_connections 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.251+0000 7f5086e00640 1 --2- 172.21.15.69:0/3835355263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f508015f890 0x7f50800732e0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.251+0000 7f5086e00640 1 --2- 172.21.15.69:0/3835355263 >> v2:172.21.15.121:3300/0 conn(0x7f508015f4b0 0x7f50800b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.251+0000 7f5086e00640 1 -- 172.21.15.69:0/3835355263 >> 172.21.15.69:0/3835355263 conn(0x7f50800d0350 msgr2=0x7f50800d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.251+0000 7f5086e00640 1 -- 172.21.15.69:0/3835355263 shutdown_connections 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.251+0000 7f5086e00640 1 -- 172.21.15.69:0/3835355263 wait complete. 2024-04-24T08:13:15.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.253+0000 7f5086e00640 1 Processor -- start 2024-04-24T08:13:15.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.253+0000 7f5086e00640 1 -- start start 2024-04-24T08:13:15.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.254+0000 7f5086e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5080073820 0x7f50800cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.254+0000 7f5086e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f508015f4b0 0x7f5080156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.254+0000 7f5086e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f508015f890 0x7f50800c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.254+0000 7f5086e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f508015fec0 con 0x7f5080073820 2024-04-24T08:13:15.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.254+0000 7f5086e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f508015fd40 con 0x7f508015f4b0 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.254+0000 7f5086e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f50800b4db0 con 0x7f508015f890 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.254+0000 7f507fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5080073820 0x7f50800cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.254+0000 7f507fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5080073820 0x7f50800cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35546/0 (socket says 172.21.15.69:35546) 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.254+0000 7f507fe00640 1 -- 172.21.15.69:0/1546877058 learned_addr learned my addr 172.21.15.69:0/1546877058 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f507f400640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f508015f4b0 0x7f5080156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f507fe00640 1 -- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f508015f890 msgr2=0x7f50800c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f5084e00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f508015f890 0x7f50800c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f507fe00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f508015f890 0x7f50800c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f507fe00640 1 -- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f508015f4b0 msgr2=0x7f5080156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f507fe00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f508015f4b0 0x7f5080156b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f507fe00640 1 -- 172.21.15.69:0/1546877058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50800ca530 con 0x7f5080073820 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f5084e00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f508015f890 0x7f50800c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f507fe00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5080073820 0x7f50800cc230 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5070007f20 tx=0x7f507000e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f5077e00640 1 -- 172.21.15.69:0/1546877058 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070014620 con 0x7f5080073820 2024-04-24T08:13:15.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.255+0000 7f5077e00640 1 -- 172.21.15.69:0/1546877058 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5070019070 con 0x7f5080073820 2024-04-24T08:13:15.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.256+0000 7f5077e00640 1 -- 172.21.15.69:0/1546877058 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5070014a00 con 0x7f5080073820 2024-04-24T08:13:15.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.256+0000 7f5086e00640 1 -- 172.21.15.69:0/1546877058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50800ca7d0 con 0x7f5080073820 2024-04-24T08:13:15.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.256+0000 7f5086e00640 1 -- 172.21.15.69:0/1546877058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50800cad90 con 0x7f5080073820 2024-04-24T08:13:15.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.257+0000 7f5086e00640 1 -- 172.21.15.69:0/1546877058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5080068a80 con 0x7f5080073820 2024-04-24T08:13:15.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.258+0000 7f5077e00640 1 -- 172.21.15.69:0/1546877058 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5070014ba0 con 0x7f5080073820 2024-04-24T08:13:15.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.259+0000 7f5077e00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f505c06c8f0 0x7f505c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.259+0000 7f5077e00640 1 -- 172.21.15.69:0/1546877058 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 5609+0+0 (secure 0 0 0) 0x7f507008c380 con 0x7f5080073820 2024-04-24T08:13:15.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.259+0000 7f507f400640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f505c06c8f0 0x7f505c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.261+0000 7f507f400640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f505c06c8f0 0x7f505c06edb0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5068034000 tx=0x7f506802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.263+0000 7f5077e00640 1 -- 172.21.15.69:0/1546877058 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f507005a8a0 con 0x7f5080073820 2024-04-24T08:13:15.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:15.497+0000 7f5086e00640 1 -- 172.21.15.69:0/1546877058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f508009e1e0 con 0x7f5080073820 2024-04-24T08:13:16.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.089+0000 7f5077e00640 1 -- 172.21.15.69:0/1546877058 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v40) ==== 174+0+0 (secure 0 0 0) 0x7f507005a1a0 con 0x7f5080073820 2024-04-24T08:13:16.091 INFO:teuthology.orchestra.run.smithi069.stderr:new fs with metadata pool 10 and data pool 11 2024-04-24T08:13:16.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.097+0000 7f5086e00640 1 -- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f505c06c8f0 msgr2=0x7f505c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.097+0000 7f5086e00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f505c06c8f0 0x7f505c06edb0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5068034000 tx=0x7f506802e000 comp rx=0 tx=0).stop 2024-04-24T08:13:16.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.097+0000 7f5086e00640 1 -- 172.21.15.69:0/1546877058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5080073820 msgr2=0x7f50800cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.097+0000 7f5086e00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5080073820 0x7f50800cc230 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5070007f20 tx=0x7f507000e570 comp rx=0 tx=0).stop 2024-04-24T08:13:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.099+0000 7f5086e00640 1 -- 172.21.15.69:0/1546877058 shutdown_connections 2024-04-24T08:13:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.099+0000 7f5086e00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f505c06c8f0 0x7f505c06edb0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.099+0000 7f5086e00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f508015f890 0x7f50800c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.099+0000 7f5086e00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f508015f4b0 0x7f5080156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.099+0000 7f5086e00640 1 --2- 172.21.15.69:0/1546877058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5080073820 0x7f50800cc230 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.099+0000 7f5086e00640 1 -- 172.21.15.69:0/1546877058 >> 172.21.15.69:0/1546877058 conn(0x7f50800d0350 msgr2=0x7f50800ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:16.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.101+0000 7f5086e00640 1 -- 172.21.15.69:0/1546877058 shutdown_connections 2024-04-24T08:13:16.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.101+0000 7f5086e00640 1 -- 172.21.15.69:0/1546877058 wait complete. 2024-04-24T08:13:16.129 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:16.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.300+0000 7f18e2200640 1 Processor -- start 2024-04-24T08:13:16.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.300+0000 7f18e2200640 1 -- start start 2024-04-24T08:13:16.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.302+0000 7f18e2200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18dc0d0300 0x7f18dc06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.302+0000 7f18e2200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18dc145a90 0x7f18dc0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.302+0000 7f18e2200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18dc0d0c20 con 0x7f18dc145fe0 2024-04-24T08:13:16.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.302+0000 7f18e2200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f18dc0cf3d0 con 0x7f18dc145a90 2024-04-24T08:13:16.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.302+0000 7f18e2200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f18dc0cf6b0 con 0x7f18dc0d0300 2024-04-24T08:13:16.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.302+0000 7f18daa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18dc145a90 0x7f18dc0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:16.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.302+0000 7f18daa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f18dc145a90 0x7f18dc0d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35560/0 (socket says 172.21.15.69:35560) 2024-04-24T08:13:16.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.302+0000 7f18daa00640 1 -- 172.21.15.69:0/694522997 learned_addr learned my addr 172.21.15.69:0/694522997 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:16.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.302+0000 7f18db400640 1 --2- 172.21.15.69:0/694522997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18dc0d0300 0x7f18dc06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:16.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.303+0000 7f18daa00640 1 -- 172.21.15.69:0/694522997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18dc0d0300 msgr2=0x7f18dc06e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:16.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.303+0000 7f18daa00640 1 --2- 172.21.15.69:0/694522997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18dc0d0300 0x7f18dc06e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.303+0000 7f18daa00640 1 -- 172.21.15.69:0/694522997 >> v1:172.21.15.121:6789/0 conn(0x7f18dc145fe0 legacy=0x7f18dc160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:16.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.303+0000 7f18daa00640 1 -- 172.21.15.69:0/694522997 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18dc0cf990 con 0x7f18dc145a90 2024-04-24T08:13:16.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.304+0000 7f18daa00640 1 --2- 172.21.15.69:0/694522997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 0x7f18dc0d06e0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009870 tx=0x7f18c402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daebebf681f4c9cd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:16.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.304+0000 7f18da000640 1 -- 172.21.15.69:0/694522997 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4037070 con 0x7f18dc145a90 2024-04-24T08:13:16.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.304+0000 7f18da000640 1 -- 172.21.15.69:0/694522997 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c4032440 con 0x7f18dc145a90 2024-04-24T08:13:16.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.304+0000 7f18da000640 1 -- 172.21.15.69:0/694522997 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4032810 con 0x7f18dc145a90 2024-04-24T08:13:16.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.304+0000 7f18e2200640 1 -- 172.21.15.69:0/694522997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 msgr2=0x7f18dc0d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.304+0000 7f18e2200640 1 --2- 172.21.15.69:0/694522997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 0x7f18dc0d06e0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f18c4009870 tx=0x7f18c402a210 comp rx=0 tx=0).stop 2024-04-24T08:13:16.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.305+0000 7f18e2200640 1 -- 172.21.15.69:0/694522997 shutdown_connections 2024-04-24T08:13:16.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.305+0000 7f18e2200640 1 --2- 172.21.15.69:0/694522997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 0x7f18dc0d06e0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.305+0000 7f18e2200640 1 --2- 172.21.15.69:0/694522997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18dc0d0300 0x7f18dc06e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.305+0000 7f18e2200640 1 -- 172.21.15.69:0/694522997 >> 172.21.15.69:0/694522997 conn(0x7f18dc067d90 msgr2=0x7f18dc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:16.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.305+0000 7f18e2200640 1 -- 172.21.15.69:0/694522997 shutdown_connections 2024-04-24T08:13:16.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.305+0000 7f18e2200640 1 -- 172.21.15.69:0/694522997 wait complete. 2024-04-24T08:13:16.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.306+0000 7f18e2200640 1 Processor -- start 2024-04-24T08:13:16.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.307+0000 7f18e2200640 1 -- start start 2024-04-24T08:13:16.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18e2200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18dc0d0300 0x7f18dc13e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18e2200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 0x7f18dc13ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18e2200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18dc145fe0 0x7f18dc143120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18e2200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc145770 con 0x7f18dc145a90 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18e2200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f18dc0cff50 con 0x7f18dc145fe0 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18e2200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f18dc1458f0 con 0x7f18dc0d0300 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18daa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 0x7f18dc13ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18daa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 0x7f18dc13ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35566/0 (socket says 172.21.15.69:35566) 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18daa00640 1 -- 172.21.15.69:0/2500615385 learned_addr learned my addr 172.21.15.69:0/2500615385 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18db400640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18dc0d0300 0x7f18dc13e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18daa00640 1 -- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18dc0d0300 msgr2=0x7f18dc13e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18daa00640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18dc0d0300 0x7f18dc13e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18daa00640 1 -- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18dc145fe0 msgr2=0x7f18dc143120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18dbe00640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18dc145fe0 0x7f18dc143120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:16.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18daa00640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18dc145fe0 0x7f18dc143120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18daa00640 1 -- 172.21.15.69:0/2500615385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18dc15ec00 con 0x7f18dc145a90 2024-04-24T08:13:16.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.308+0000 7f18dbe00640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18dc145fe0 0x7f18dc143120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:16.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.309+0000 7f18daa00640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 0x7f18dc13ffe0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f18c4000c00 tx=0x7f18c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:16.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.309+0000 7f18d3400640 1 -- 172.21.15.69:0/2500615385 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4037040 con 0x7f18dc145a90 2024-04-24T08:13:16.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.309+0000 7f18d3400640 1 -- 172.21.15.69:0/2500615385 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c4042070 con 0x7f18dc145a90 2024-04-24T08:13:16.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.309+0000 7f18e2200640 1 -- 172.21.15.69:0/2500615385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18dc15eee0 con 0x7f18dc145a90 2024-04-24T08:13:16.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.309+0000 7f18d3400640 1 -- 172.21.15.69:0/2500615385 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4030bd0 con 0x7f18dc145a90 2024-04-24T08:13:16.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.309+0000 7f18e2200640 1 -- 172.21.15.69:0/2500615385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18dc15f4a0 con 0x7f18dc145a90 2024-04-24T08:13:16.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.310+0000 7f18e2200640 1 -- 172.21.15.69:0/2500615385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18dc068a80 con 0x7f18dc145a90 2024-04-24T08:13:16.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.311+0000 7f18d3400640 1 -- 172.21.15.69:0/2500615385 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f18c4047460 con 0x7f18dc145a90 2024-04-24T08:13:16.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.312+0000 7f18d3400640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f18b806c8f0 0x7f18b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.312+0000 7f18d3400640 1 -- 172.21.15.69:0/2500615385 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7f18c40ac820 con 0x7f18dc145a90 2024-04-24T08:13:16.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.313+0000 7f18db400640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f18b806c8f0 0x7f18b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:16.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.314+0000 7f18db400640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f18b806c8f0 0x7f18b806edb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f18d4008e50 tx=0x7f18d40083a0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:16.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.317+0000 7f18d3400640 1 -- 172.21.15.69:0/2500615385 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f18c407abb0 con 0x7f18dc145a90 2024-04-24T08:13:16.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.522+0000 7f18e2200640 1 -- 172.21.15.69:0/2500615385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18dc0d1120 con 0x7f18dc145a90 2024-04-24T08:13:16.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.524+0000 7f18d3400640 1 -- 172.21.15.69:0/2500615385 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16657 (secure 0 0 0) 0x7f18c407a4b0 con 0x7f18dc145a90 2024-04-24T08:13:16.525 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:16.525 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":67,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:13:15.498766+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 -- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f18b806c8f0 msgr2=0x7f18b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f18b806c8f0 0x7f18b806edb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f18d4008e50 tx=0x7f18d40083a0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 -- 172.21.15.69:0/2500615385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 msgr2=0x7f18dc13ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 0x7f18dc13ffe0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f18c4000c00 tx=0x7f18c40046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 -- 172.21.15.69:0/2500615385 shutdown_connections 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f18b806c8f0 0x7f18b806edb0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18dc145a90 0x7f18dc13ffe0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18dc145fe0 0x7f18dc143120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 --2- 172.21.15.69:0/2500615385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18dc0d0300 0x7f18dc13e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 -- 172.21.15.69:0/2500615385 >> 172.21.15.69:0/2500615385 conn(0x7f18dc067d90 msgr2=0x7f18dc0b3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 -- 172.21.15.69:0/2500615385 shutdown_connections 2024-04-24T08:13:16.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.527+0000 7f18e2200640 1 -- 172.21.15.69:0/2500615385 wait complete. 2024-04-24T08:13:16.546 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.743+0000 7ff0a6000640 1 Processor -- start 2024-04-24T08:13:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.743+0000 7ff0a6000640 1 -- start start 2024-04-24T08:13:16.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.745+0000 7ff0a6000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a015e960 0x7ff0a0162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.745+0000 7ff0a6000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff0a00cfaf0 0x7ff0a00cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.745+0000 7ff0a6000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0a015ed40 con 0x7ff0a0162980 2024-04-24T08:13:16.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.745+0000 7ff0a6000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff0a00cf050 con 0x7ff0a00cfaf0 2024-04-24T08:13:16.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.745+0000 7ff0a6000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a0162e20 con 0x7ff0a015e960 2024-04-24T08:13:16.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.745+0000 7ff0a5600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff0a00cfaf0 0x7ff0a00cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:16.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.745+0000 7ff0a5600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff0a00cfaf0 0x7ff0a00cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35574/0 (socket says 172.21.15.69:35574) 2024-04-24T08:13:16.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.745+0000 7ff0a5600640 1 -- 172.21.15.69:0/2279621727 learned_addr learned my addr 172.21.15.69:0/2279621727 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:16.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.746+0000 7ff0a4c00640 1 --2- 172.21.15.69:0/2279621727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a015e960 0x7ff0a0162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:16.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.746+0000 7ff0a5600640 1 -- 172.21.15.69:0/2279621727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a015e960 msgr2=0x7ff0a0162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:16.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.746+0000 7ff0a5600640 1 --2- 172.21.15.69:0/2279621727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a015e960 0x7ff0a0162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.746+0000 7ff0a5600640 1 -- 172.21.15.69:0/2279621727 >> v1:172.21.15.121:6789/0 conn(0x7ff0a0162980 legacy=0x7ff0a006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:16.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.746+0000 7ff0a5600640 1 -- 172.21.15.69:0/2279621727 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a0163100 con 0x7ff0a00cfaf0 2024-04-24T08:13:16.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.747+0000 7ff0a5600640 1 --2- 172.21.15.69:0/2279621727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a00cfaf0 0x7ff0a00cffa0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff098009e50 tx=0x7ff09802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113e6b466cf06938 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:16.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.747+0000 7ff09fe00640 1 -- 172.21.15.69:0/2279621727 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098036070 con 0x7ff0a00cfaf0 2024-04-24T08:13:16.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.747+0000 7ff09fe00640 1 -- 172.21.15.69:0/2279621727 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff098031600 con 0x7ff0a00cfaf0 2024-04-24T08:13:16.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.747+0000 7ff09fe00640 1 -- 172.21.15.69:0/2279621727 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0980319d0 con 0x7ff0a00cfaf0 2024-04-24T08:13:16.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.748+0000 7ff0a6000640 1 -- 172.21.15.69:0/2279621727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a00cfaf0 msgr2=0x7ff0a00cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.748+0000 7ff0a6000640 1 --2- 172.21.15.69:0/2279621727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a00cfaf0 0x7ff0a00cffa0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff098009e50 tx=0x7ff09802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.748+0000 7ff0a6000640 1 -- 172.21.15.69:0/2279621727 shutdown_connections 2024-04-24T08:13:16.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.748+0000 7ff0a6000640 1 --2- 172.21.15.69:0/2279621727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a00cfaf0 0x7ff0a00cffa0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.748+0000 7ff0a6000640 1 --2- 172.21.15.69:0/2279621727 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a015e960 0x7ff0a0162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.748+0000 7ff0a6000640 1 -- 172.21.15.69:0/2279621727 >> 172.21.15.69:0/2279621727 conn(0x7ff0a009d900 msgr2=0x7ff0a008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:16.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.748+0000 7ff0a6000640 1 -- 172.21.15.69:0/2279621727 shutdown_connections 2024-04-24T08:13:16.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.748+0000 7ff0a6000640 1 -- 172.21.15.69:0/2279621727 wait complete. 2024-04-24T08:13:16.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.750+0000 7ff0a6000640 1 Processor -- start 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.750+0000 7ff0a6000640 1 -- start start 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.751+0000 7ff0a6000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a00cfaf0 0x7ff0a014cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.751+0000 7ff0a6000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0a015e960 0x7ff0a015ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.751+0000 7ff0a6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a0162980 0x7ff0a014d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.751+0000 7ff0a6000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a006d6c0 con 0x7ff0a0162980 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.751+0000 7ff0a6000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a00cf5d0 con 0x7ff0a015e960 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.751+0000 7ff0a6000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a00b6170 con 0x7ff0a00cfaf0 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.751+0000 7ff0a5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a0162980 0x7ff0a014d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.751+0000 7ff0a5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a0162980 0x7ff0a014d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35590/0 (socket says 172.21.15.69:35590) 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.751+0000 7ff0a5600640 1 -- 172.21.15.69:0/2579073183 learned_addr learned my addr 172.21.15.69:0/2579073183 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff0a5600640 1 -- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a00cfaf0 msgr2=0x7ff0a014cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff0a5600640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a00cfaf0 0x7ff0a014cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff0a5600640 1 -- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0a015e960 msgr2=0x7ff0a015ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff0a5600640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0a015e960 0x7ff0a015ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff0a5600640 1 -- 172.21.15.69:0/2579073183 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a014d140 con 0x7ff0a0162980 2024-04-24T08:13:16.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff0a5600640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a0162980 0x7ff0a014d650 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff098009f70 tx=0x7ff098004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:16.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff09d600640 1 -- 172.21.15.69:0/2579073183 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09803f020 con 0x7ff0a0162980 2024-04-24T08:13:16.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff09d600640 1 -- 172.21.15.69:0/2579073183 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff098037040 con 0x7ff0a0162980 2024-04-24T08:13:16.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff09d600640 1 -- 172.21.15.69:0/2579073183 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff098036070 con 0x7ff0a0162980 2024-04-24T08:13:16.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff0a6000640 1 -- 172.21.15.69:0/2579073183 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a00c9320 con 0x7ff0a0162980 2024-04-24T08:13:16.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.752+0000 7ff0a6000640 1 -- 172.21.15.69:0/2579073183 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a00c98e0 con 0x7ff0a0162980 2024-04-24T08:13:16.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.754+0000 7ff0a6000640 1 -- 172.21.15.69:0/2579073183 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a0075d50 con 0x7ff0a0162980 2024-04-24T08:13:16.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.755+0000 7ff09d600640 1 -- 172.21.15.69:0/2579073183 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff098042020 con 0x7ff0a0162980 2024-04-24T08:13:16.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.755+0000 7ff09d600640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff07c06c8f0 0x7ff07c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:16.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.756+0000 7ff09d600640 1 -- 172.21.15.69:0/2579073183 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 5709+0+0 (secure 0 0 0) 0x7ff0980ac240 con 0x7ff0a0162980 2024-04-24T08:13:16.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.756+0000 7ff0a4c00640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff07c06c8f0 0x7ff07c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:16.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.757+0000 7ff0a4c00640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff07c06c8f0 0x7ff07c06edb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff090006fd0 tx=0x7ff090008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:16.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.760+0000 7ff09d600640 1 -- 172.21.15.69:0/2579073183 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff098049030 con 0x7ff0a0162980 2024-04-24T08:13:16.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.962+0000 7ff0a6000640 1 -- 172.21.15.69:0/2579073183 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0a00d3770 con 0x7ff0a0162980 2024-04-24T08:13:16.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.964+0000 7ff09d600640 1 -- 172.21.15.69:0/2579073183 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) ==== 74+0+16657 (secure 0 0 0) 0x7ff0a014d140 con 0x7ff0a0162980 2024-04-24T08:13:16.965 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:16.965 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":67,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:13:15.498766+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:16.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.966+0000 7ff0a6000640 1 -- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff07c06c8f0 msgr2=0x7ff07c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.966+0000 7ff0a6000640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff07c06c8f0 0x7ff07c06edb0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff090006fd0 tx=0x7ff090008040 comp rx=0 tx=0).stop 2024-04-24T08:13:16.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.966+0000 7ff0a6000640 1 -- 172.21.15.69:0/2579073183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a0162980 msgr2=0x7ff0a014d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:16.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.967+0000 7ff0a6000640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a0162980 0x7ff0a014d650 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff098009f70 tx=0x7ff098004a40 comp rx=0 tx=0).stop 2024-04-24T08:13:16.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.967+0000 7ff0a6000640 1 -- 172.21.15.69:0/2579073183 shutdown_connections 2024-04-24T08:13:16.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.967+0000 7ff0a6000640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff07c06c8f0 0x7ff07c06edb0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.967+0000 7ff0a6000640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0a0162980 0x7ff0a014d650 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.967+0000 7ff0a6000640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0a015e960 0x7ff0a015ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.967+0000 7ff0a6000640 1 --2- 172.21.15.69:0/2579073183 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0a00cfaf0 0x7ff0a014cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:16.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.967+0000 7ff0a6000640 1 -- 172.21.15.69:0/2579073183 >> 172.21.15.69:0/2579073183 conn(0x7ff0a009d900 msgr2=0x7ff0a00b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:16.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.967+0000 7ff0a6000640 1 -- 172.21.15.69:0/2579073183 shutdown_connections 2024-04-24T08:13:16.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:16.967+0000 7ff0a6000640 1 -- 172.21.15.69:0/2579073183 wait complete. 2024-04-24T08:13:16.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T08:13:17.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.194+0000 7fc77b800640 1 Processor -- start 2024-04-24T08:13:17.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.194+0000 7fc77b800640 1 -- start start 2024-04-24T08:13:17.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.195+0000 7fc77b800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc7740b3e90 0x7fc7740b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.195+0000 7fc77b800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7740ce250 0x7fc7740ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.196+0000 7fc77b800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc77415cc00 con 0x7fc77415c820 2024-04-24T08:13:17.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.196+0000 7fc77b800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc7740b4be0 con 0x7fc7740b3e90 2024-04-24T08:13:17.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.196+0000 7fc77b800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc77406ed30 con 0x7fc7740ce250 2024-04-24T08:13:17.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.196+0000 7fc778e00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fc77415c820 0x7fc77406f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:34996/0 (socket says 172.21.15.69:34996) 2024-04-24T08:13:17.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.196+0000 7fc778e00640 1 -- 172.21.15.69:0/2148498618 learned_addr learned my addr 172.21.15.69:0/2148498618 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:17.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.196+0000 7fc779800640 1 --2- 172.21.15.69:0/2148498618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7740ce250 0x7fc7740ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.196+0000 7fc773e00640 1 --2- 172.21.15.69:0/2148498618 >> v2:172.21.15.121:3300/0 conn(0x7fc7740b3e90 0x7fc7740b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.197+0000 7fc773400640 1 -- 172.21.15.69:0/2148498618 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1140494410 0 0) 0x7fc77415cc00 con 0x7fc77415c820 2024-04-24T08:13:17.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.197+0000 7fc773400640 1 -- 172.21.15.69:0/2148498618 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc758003520 con 0x7fc77415c820 2024-04-24T08:13:17.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.197+0000 7fc779800640 1 -- 172.21.15.69:0/2148498618 >> v2:172.21.15.121:3300/0 conn(0x7fc7740b3e90 msgr2=0x7fc7740b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.197+0000 7fc779800640 1 --2- 172.21.15.69:0/2148498618 >> v2:172.21.15.121:3300/0 conn(0x7fc7740b3e90 0x7fc7740b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.197+0000 7fc779800640 1 -- 172.21.15.69:0/2148498618 >> v1:172.21.15.121:6789/0 conn(0x7fc77415c820 legacy=0x7fc77406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.197+0000 7fc779800640 1 -- 172.21.15.69:0/2148498618 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7740b4e10 con 0x7fc7740ce250 2024-04-24T08:13:17.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.197+0000 7fc773e00640 1 --2- 172.21.15.69:0/2148498618 >> v2:172.21.15.121:3300/0 conn(0x7fc7740b3e90 0x7fc7740b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:13:17.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.199+0000 7fc779800640 1 --2- 172.21.15.69:0/2148498618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7740ce250 0x7fc7740ce6e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc76c0097a0 tx=0x7fc76c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7df93d5b74735869 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.199+0000 7fc773400640 1 -- 172.21.15.69:0/2148498618 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c037070 con 0x7fc7740ce250 2024-04-24T08:13:17.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.199+0000 7fc773400640 1 -- 172.21.15.69:0/2148498618 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc76c032440 con 0x7fc7740ce250 2024-04-24T08:13:17.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.199+0000 7fc773400640 1 -- 172.21.15.69:0/2148498618 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76c0327f0 con 0x7fc7740ce250 2024-04-24T08:13:17.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.200+0000 7fc77b800640 1 -- 172.21.15.69:0/2148498618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7740ce250 msgr2=0x7fc7740ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.200+0000 7fc77b800640 1 --2- 172.21.15.69:0/2148498618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7740ce250 0x7fc7740ce6e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc76c0097a0 tx=0x7fc76c02a180 comp rx=0 tx=0).stop 2024-04-24T08:13:17.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.200+0000 7fc77b800640 1 -- 172.21.15.69:0/2148498618 shutdown_connections 2024-04-24T08:13:17.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.200+0000 7fc77b800640 1 --2- 172.21.15.69:0/2148498618 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7740ce250 0x7fc7740ce6e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.200+0000 7fc77b800640 1 --2- 172.21.15.69:0/2148498618 >> v2:172.21.15.121:3300/0 conn(0x7fc7740b3e90 0x7fc7740b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.200+0000 7fc77b800640 1 -- 172.21.15.69:0/2148498618 >> 172.21.15.69:0/2148498618 conn(0x7fc7740d0350 msgr2=0x7fc7740d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.200+0000 7fc77b800640 1 -- 172.21.15.69:0/2148498618 shutdown_connections 2024-04-24T08:13:17.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.200+0000 7fc77b800640 1 -- 172.21.15.69:0/2148498618 wait complete. 2024-04-24T08:13:17.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.202+0000 7fc77b800640 1 Processor -- start 2024-04-24T08:13:17.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.202+0000 7fc77b800640 1 -- start start 2024-04-24T08:13:17.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc77b800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7740b3e90 0x7fc77415f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc77b800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7740ce250 0x7fc7741f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc77b800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc77415c820 0x7fc77409d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc77b800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc77406f2b0 con 0x7fc7740ce250 2024-04-24T08:13:17.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc77b800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc77406efd0 con 0x7fc7740b3e90 2024-04-24T08:13:17.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc77b800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc77406fee0 con 0x7fc77415c820 2024-04-24T08:13:17.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc773e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7740ce250 0x7fc7741f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc773e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7740ce250 0x7fc7741f1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35604/0 (socket says 172.21.15.69:35604) 2024-04-24T08:13:17.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc773e00640 1 -- 172.21.15.69:0/4103839334 learned_addr learned my addr 172.21.15.69:0/4103839334 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:17.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc778e00640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7740b3e90 0x7fc77415f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc773e00640 1 -- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc77415c820 msgr2=0x7fc77409d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc773e00640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc77415c820 0x7fc77409d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc773e00640 1 -- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7740b3e90 msgr2=0x7fc77415f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc773e00640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7740b3e90 0x7fc77415f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.203+0000 7fc773e00640 1 -- 172.21.15.69:0/4103839334 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc77415e980 con 0x7fc7740ce250 2024-04-24T08:13:17.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.204+0000 7fc773e00640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7740ce250 0x7fc7741f1bf0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00e960 tx=0x7fc75c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.204+0000 7fc770c00640 1 -- 172.21.15.69:0/4103839334 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c017070 con 0x7fc7740ce250 2024-04-24T08:13:17.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.204+0000 7fc770c00640 1 -- 172.21.15.69:0/4103839334 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc75c00ca00 con 0x7fc7740ce250 2024-04-24T08:13:17.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.204+0000 7fc770c00640 1 -- 172.21.15.69:0/4103839334 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c00cdd0 con 0x7fc7740ce250 2024-04-24T08:13:17.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.204+0000 7fc77b800640 1 -- 172.21.15.69:0/4103839334 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc77415ec20 con 0x7fc7740ce250 2024-04-24T08:13:17.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.204+0000 7fc77b800640 1 -- 172.21.15.69:0/4103839334 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc77409e290 con 0x7fc7740ce250 2024-04-24T08:13:17.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.205+0000 7fc77b800640 1 -- 172.21.15.69:0/4103839334 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc774068a80 con 0x7fc7740ce250 2024-04-24T08:13:17.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.206+0000 7fc770c00640 1 -- 172.21.15.69:0/4103839334 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc75c01a070 con 0x7fc7740ce250 2024-04-24T08:13:17.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.207+0000 7fc770c00640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc75006c8f0 0x7fc75006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.207+0000 7fc770c00640 1 -- 172.21.15.69:0/4103839334 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fc75c08baa0 con 0x7fc7740ce250 2024-04-24T08:13:17.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.208+0000 7fc778e00640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc75006c8f0 0x7fc75006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.209+0000 7fc778e00640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc75006c8f0 0x7fc75006edb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc7640012e0 tx=0x7fc764009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.212+0000 7fc770c00640 1 -- 172.21.15.69:0/4103839334 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc75c059e30 con 0x7fc7740ce250 2024-04-24T08:13:17.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:17.457+0000 7fc77b800640 1 -- 172.21.15.69:0/4103839334 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc7740cddc0 con 0x7fc7740ce250 2024-04-24T08:13:18.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.098+0000 7fc770c00640 1 -- 172.21.15.69:0/4103839334 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v43) ==== 118+0+0 (secure 0 0 0) 0x7fc75c059730 con 0x7fc7740ce250 2024-04-24T08:13:18.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.101+0000 7fc77b800640 1 -- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc75006c8f0 msgr2=0x7fc75006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:18.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.101+0000 7fc77b800640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc75006c8f0 0x7fc75006edb0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc7640012e0 tx=0x7fc764009040 comp rx=0 tx=0).stop 2024-04-24T08:13:18.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.101+0000 7fc77b800640 1 -- 172.21.15.69:0/4103839334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7740ce250 msgr2=0x7fc7741f1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:18.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.101+0000 7fc77b800640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7740ce250 0x7fc7741f1bf0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00e960 tx=0x7fc75c00ee20 comp rx=0 tx=0).stop 2024-04-24T08:13:18.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.101+0000 7fc77b800640 1 -- 172.21.15.69:0/4103839334 shutdown_connections 2024-04-24T08:13:18.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.101+0000 7fc77b800640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc75006c8f0 0x7fc75006edb0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:18.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.101+0000 7fc77b800640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7740ce250 0x7fc7741f1bf0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:18.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.101+0000 7fc77b800640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc77415c820 0x7fc77409d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:18.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.101+0000 7fc77b800640 1 --2- 172.21.15.69:0/4103839334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7740b3e90 0x7fc77415f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:18.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.101+0000 7fc77b800640 1 -- 172.21.15.69:0/4103839334 >> 172.21.15.69:0/4103839334 conn(0x7fc7740d0350 msgr2=0x7fc7740b13a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:18.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.102+0000 7fc77b800640 1 -- 172.21.15.69:0/4103839334 shutdown_connections 2024-04-24T08:13:18.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.102+0000 7fc77b800640 1 -- 172.21.15.69:0/4103839334 wait complete. 2024-04-24T08:13:18.125 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-24T08:13:18.126 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T08:13:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.334+0000 7f8188c00640 1 Processor -- start 2024-04-24T08:13:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.335+0000 7f8188c00640 1 -- start start 2024-04-24T08:13:18.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.336+0000 7f8188c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f81840d36a0 0x7f81840d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:18.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.336+0000 7f8188c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81840d2600 con 0x7f81840d16f0 2024-04-24T08:13:18.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.336+0000 7f8188c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81840d28e0 con 0x7f818415ea60 2024-04-24T08:13:18.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.336+0000 7f8188c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f81840d23b0 con 0x7f81840d36a0 2024-04-24T08:13:18.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.337+0000 7f8182800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f81840d16f0 0x7f818406de50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:58906/0 (socket says 172.21.15.69:58906) 2024-04-24T08:13:18.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.337+0000 7f8182800640 1 -- 172.21.15.69:0/3701952638 learned_addr learned my addr 172.21.15.69:0/3701952638 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:18.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.337+0000 7f8179400640 1 --2- 172.21.15.69:0/3701952638 >> v2:172.21.15.121:3300/0 conn(0x7f81840d36a0 0x7f81840d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:18.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.337+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1682662561 0 0) 0x7f81840d2600 con 0x7f81840d16f0 2024-04-24T08:13:18.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.337+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8168003520 con 0x7f81840d16f0 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.337+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4245151438 0 0) 0x7f8168003520 con 0x7f81840d16f0 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.337+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 >> v2:172.21.15.121:3300/0 conn(0x7f81840d36a0 msgr2=0x7f81840d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.337+0000 7f8181400640 1 --2- 172.21.15.69:0/3701952638 >> v2:172.21.15.121:3300/0 conn(0x7f81840d36a0 0x7f81840d11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 >> v1:172.21.15.121:6789/0 conn(0x7f818415ea60 legacy=0x7f81840cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81840d2bc0 con 0x7f81840d16f0 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f817c001670 con 0x7f81840d16f0 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 >> v1:172.21.15.69:6789/0 conn(0x7f81840d16f0 legacy=0x7f818406de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181400640 1 --2- 172.21.15.69:0/3701952638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81680039b0 0x7f8168003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181400640 1 --2- 172.21.15.69:0/3701952638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8168004800 0x7f8168004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181400640 1 --2- 172.21.15.69:0/3701952638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81680051c0 0x7f816800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f81840d23b0 con 0x7f81680051c0 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f816800c060 con 0x7f8168004800 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f816800c260 con 0x7f81680039b0 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8182800640 1 --2- 172.21.15.69:0/3701952638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81680051c0 0x7f816800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8181e00640 1 --2- 172.21.15.69:0/3701952638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81680039b0 0x7f8168003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8182800640 1 -- 172.21.15.69:0/3701952638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81680039b0 msgr2=0x7f8168003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8182800640 1 --2- 172.21.15.69:0/3701952638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81680039b0 0x7f8168003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8182800640 1 -- 172.21.15.69:0/3701952638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8168004800 msgr2=0x7f8168004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8182800640 1 --2- 172.21.15.69:0/3701952638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8168004800 0x7f8168004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.338+0000 7f8182800640 1 -- 172.21.15.69:0/3701952638 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8168003520 con 0x7f81680051c0 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.339+0000 7f8182800640 1 -- 172.21.15.69:0/3701952638 reap_dead start 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.339+0000 7f8182800640 1 --2- 172.21.15.69:0/3701952638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81680051c0 0x7f816800b9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f817c00c240 tx=0x7f8168005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239d693a5859c098 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.340+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c007d10 con 0x7f81680051c0 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.340+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f817c005ad0 con 0x7f81680051c0 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.340+0000 7f8181400640 1 -- 172.21.15.69:0/3701952638 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c007070 con 0x7f81680051c0 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.340+0000 7f8188c00640 1 -- 172.21.15.69:0/3701952638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81680051c0 msgr2=0x7f816800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.340+0000 7f8188c00640 1 --2- 172.21.15.69:0/3701952638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81680051c0 0x7f816800b9c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f817c00c240 tx=0x7f8168005830 comp rx=0 tx=0).stop 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.340+0000 7f8188c00640 1 -- 172.21.15.69:0/3701952638 shutdown_connections 2024-04-24T08:13:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.340+0000 7f8188c00640 1 --2- 172.21.15.69:0/3701952638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81680051c0 0x7f816800b9c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:18.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.340+0000 7f8188c00640 1 -- 172.21.15.69:0/3701952638 >> 172.21.15.69:0/3701952638 conn(0x7f818408bf60 msgr2=0x7f81840d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:18.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.341+0000 7f8188c00640 1 -- 172.21.15.69:0/3701952638 shutdown_connections 2024-04-24T08:13:18.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.341+0000 7f8188c00640 1 -- 172.21.15.69:0/3701952638 wait complete. 2024-04-24T08:13:18.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.342+0000 7f8188c00640 1 Processor -- start 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.342+0000 7f8188c00640 1 -- start start 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.343+0000 7f8188c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81680051c0 0x7f81840ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.343+0000 7f8188c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81840cb8b0 0x7f81841f85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.343+0000 7f8188c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f818414f340 0x7f818406de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.343+0000 7f8188c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f81840d3180 con 0x7f81840cb8b0 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.343+0000 7f8188c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f81840d1b20 con 0x7f818414f340 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.343+0000 7f8188c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f818415ee40 con 0x7f81680051c0 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8179400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81840cb8b0 0x7f81841f85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8179400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81840cb8b0 0x7f81841f85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35626/0 (socket says 172.21.15.69:35626) 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8179400640 1 -- 172.21.15.69:0/3740703121 learned_addr learned my addr 172.21.15.69:0/3740703121 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8181e00640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81680051c0 0x7f81840ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8182800640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f818414f340 0x7f818406de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8179400640 1 -- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81680051c0 msgr2=0x7f81840ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:18.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8179400640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81680051c0 0x7f81840ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:18.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8179400640 1 -- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f818414f340 msgr2=0x7f818406de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:18.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8179400640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f818414f340 0x7f818406de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:18.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8179400640 1 -- 172.21.15.69:0/3740703121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81841f8ae0 con 0x7f81840cb8b0 2024-04-24T08:13:18.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.344+0000 7f8179400640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81840cb8b0 0x7f81841f85a0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f816c00e7a0 tx=0x7f816c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:18.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.345+0000 7f817aa00640 1 -- 172.21.15.69:0/3740703121 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c017070 con 0x7f81840cb8b0 2024-04-24T08:13:18.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.345+0000 7f817aa00640 1 -- 172.21.15.69:0/3740703121 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f816c00c940 con 0x7f81840cb8b0 2024-04-24T08:13:18.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.345+0000 7f8188c00640 1 -- 172.21.15.69:0/3740703121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81840cbde0 con 0x7f81840cb8b0 2024-04-24T08:13:18.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.345+0000 7f817aa00640 1 -- 172.21.15.69:0/3740703121 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816c00cd10 con 0x7f81840cb8b0 2024-04-24T08:13:18.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.345+0000 7f8188c00640 1 -- 172.21.15.69:0/3740703121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81840cdaa0 con 0x7f81840cb8b0 2024-04-24T08:13:18.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.346+0000 7f8188c00640 1 -- 172.21.15.69:0/3740703121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8184068a80 con 0x7f81840cb8b0 2024-04-24T08:13:18.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.347+0000 7f817aa00640 1 -- 172.21.15.69:0/3740703121 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f816c01a070 con 0x7f81840cb8b0 2024-04-24T08:13:18.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.348+0000 7f817aa00640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f816006c920 0x7f816006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:18.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.348+0000 7f817aa00640 1 -- 172.21.15.69:0/3740703121 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f816c08b9f0 con 0x7f81840cb8b0 2024-04-24T08:13:18.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.349+0000 7f8181e00640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f816006c920 0x7f816006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:18.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.350+0000 7f8181e00640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f816006c920 0x7f816006ede0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8174000f80 tx=0x7f8174009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:18.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.352+0000 7f817aa00640 1 -- 172.21.15.69:0/3740703121 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f816c059e20 con 0x7f81840cb8b0 2024-04-24T08:13:18.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:18.587+0000 7f8188c00640 1 -- 172.21.15.69:0/3740703121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f81840d1f70 con 0x7f81840cb8b0 2024-04-24T08:13:19.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.109+0000 7f817aa00640 1 -- 172.21.15.69:0/3740703121 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v44) ==== 122+0+0 (secure 0 0 0) 0x7f816c059720 con 0x7f81840cb8b0 2024-04-24T08:13:19.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.112+0000 7f8188c00640 1 -- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f816006c920 msgr2=0x7f816006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.112+0000 7f8188c00640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f816006c920 0x7f816006ede0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8174000f80 tx=0x7f8174009040 comp rx=0 tx=0).stop 2024-04-24T08:13:19.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.112+0000 7f8188c00640 1 -- 172.21.15.69:0/3740703121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81840cb8b0 msgr2=0x7f81841f85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.112+0000 7f8188c00640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81840cb8b0 0x7f81841f85a0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f816c00e7a0 tx=0x7f816c00ec60 comp rx=0 tx=0).stop 2024-04-24T08:13:19.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.112+0000 7f8188c00640 1 -- 172.21.15.69:0/3740703121 shutdown_connections 2024-04-24T08:13:19.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.112+0000 7f8188c00640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f816006c920 0x7f816006ede0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.113+0000 7f8188c00640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81840cb8b0 0x7f81841f85a0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.113+0000 7f8188c00640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f818414f340 0x7f818406de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.113+0000 7f8188c00640 1 --2- 172.21.15.69:0/3740703121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81680051c0 0x7f81840ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.113+0000 7f8188c00640 1 -- 172.21.15.69:0/3740703121 >> 172.21.15.69:0/3740703121 conn(0x7f818408bf60 msgr2=0x7f81840d0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:19.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.113+0000 7f8188c00640 1 -- 172.21.15.69:0/3740703121 shutdown_connections 2024-04-24T08:13:19.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.113+0000 7f8188c00640 1 -- 172.21.15.69:0/3740703121 wait complete. 2024-04-24T08:13:19.135 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T08:13:19.135 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:19.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.335+0000 7f74d5800640 1 Processor -- start 2024-04-24T08:13:19.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.335+0000 7f74d5800640 1 -- start start 2024-04-24T08:13:19.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.336+0000 7f74d5800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f74d015e960 0x7f74d0162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:19.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.337+0000 7f74d5800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74d015ed40 con 0x7f74d00cfaf0 2024-04-24T08:13:19.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.337+0000 7f74d5800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74d00cf050 con 0x7f74d0162980 2024-04-24T08:13:19.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.337+0000 7f74d5800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f74d0162e20 con 0x7f74d015e960 2024-04-24T08:13:19.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.337+0000 7f74cfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f74d015e960 0x7f74d0162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:19.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.337+0000 7f74d4e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f74d00cfaf0 0x7f74d00d2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39002/0 (socket says 172.21.15.69:39002) 2024-04-24T08:13:19.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.337+0000 7f74d4e00640 1 -- 172.21.15.69:0/4086754449 learned_addr learned my addr 172.21.15.69:0/4086754449 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.337+0000 7f74cfe00640 1 -- 172.21.15.69:0/4086754449 >> v1:172.21.15.121:6789/0 conn(0x7f74d0162980 legacy=0x7f74d006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.337+0000 7f74cfe00640 1 -- 172.21.15.69:0/4086754449 >> v1:172.21.15.69:6789/0 conn(0x7f74d00cfaf0 legacy=0x7f74d00d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.337+0000 7f74cfe00640 1 -- 172.21.15.69:0/4086754449 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d0163100 con 0x7f74d015e960 2024-04-24T08:13:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.337+0000 7f74cea00640 1 -- 172.21.15.69:0/4086754449 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3108896849 0 0) 0x7f74d015ed40 con 0x7f74d00cfaf0 2024-04-24T08:13:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.338+0000 7f74cfe00640 1 --2- 172.21.15.69:0/4086754449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d015e960 0x7f74d0162440 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f74d015ef10 tx=0x7f74c002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2194d78edf91a72 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.339+0000 7f74cea00640 1 -- 172.21.15.69:0/4086754449 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c0037070 con 0x7f74d015e960 2024-04-24T08:13:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.339+0000 7f74cea00640 1 -- 172.21.15.69:0/4086754449 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74c0032440 con 0x7f74d015e960 2024-04-24T08:13:19.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.339+0000 7f74cea00640 1 -- 172.21.15.69:0/4086754449 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c0032810 con 0x7f74d015e960 2024-04-24T08:13:19.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.339+0000 7f74d5800640 1 -- 172.21.15.69:0/4086754449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d015e960 msgr2=0x7f74d0162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.339+0000 7f74d5800640 1 --2- 172.21.15.69:0/4086754449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d015e960 0x7f74d0162440 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f74d015ef10 tx=0x7f74c002a1c0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.339+0000 7f74d5800640 1 -- 172.21.15.69:0/4086754449 shutdown_connections 2024-04-24T08:13:19.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.339+0000 7f74d5800640 1 --2- 172.21.15.69:0/4086754449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d015e960 0x7f74d0162440 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.339+0000 7f74d5800640 1 -- 172.21.15.69:0/4086754449 >> 172.21.15.69:0/4086754449 conn(0x7f74d009d900 msgr2=0x7f74d008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:19.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.339+0000 7f74d5800640 1 -- 172.21.15.69:0/4086754449 shutdown_connections 2024-04-24T08:13:19.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.339+0000 7f74d5800640 1 -- 172.21.15.69:0/4086754449 wait complete. 2024-04-24T08:13:19.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.341+0000 7f74d5800640 1 Processor -- start 2024-04-24T08:13:19.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.341+0000 7f74d5800640 1 -- start start 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.342+0000 7f74d5800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74d00cfaf0 0x7f74d00ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.342+0000 7f74d5800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74d015e960 0x7f74d006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.342+0000 7f74d5800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d0162980 0x7f74d006e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.342+0000 7f74d5800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f74d006d6c0 con 0x7f74d0162980 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.342+0000 7f74d5800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f74d00cf5d0 con 0x7f74d015e960 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.342+0000 7f74d5800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f74d00b6170 con 0x7f74d00cfaf0 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.342+0000 7f74d4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d0162980 0x7f74d006e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.342+0000 7f74d4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d0162980 0x7f74d006e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41480/0 (socket says 172.21.15.69:41480) 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.342+0000 7f74d4e00640 1 -- 172.21.15.69:0/1267350314 learned_addr learned my addr 172.21.15.69:0/1267350314 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74cfe00640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74d00cfaf0 0x7f74d00ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74cf400640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74d015e960 0x7f74d006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74d4e00640 1 -- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74d00cfaf0 msgr2=0x7f74d00ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74d4e00640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74d00cfaf0 0x7f74d00ce0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74d4e00640 1 -- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74d015e960 msgr2=0x7f74d006dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74d4e00640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74d015e960 0x7f74d006dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74d4e00640 1 -- 172.21.15.69:0/1267350314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d00cd000 con 0x7f74d0162980 2024-04-24T08:13:19.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74d4e00640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d0162980 0x7f74d006e210 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f74c800c730 tx=0x7f74c800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:19.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74c7e00640 1 -- 172.21.15.69:0/1267350314 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c8018070 con 0x7f74d0162980 2024-04-24T08:13:19.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74c7e00640 1 -- 172.21.15.69:0/1267350314 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74c800ce40 con 0x7f74d0162980 2024-04-24T08:13:19.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74c7e00640 1 -- 172.21.15.69:0/1267350314 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c800a860 con 0x7f74d0162980 2024-04-24T08:13:19.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74d5800640 1 -- 172.21.15.69:0/1267350314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d00cd2e0 con 0x7f74d0162980 2024-04-24T08:13:19.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.343+0000 7f74d5800640 1 -- 172.21.15.69:0/1267350314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d00cd8a0 con 0x7f74d0162980 2024-04-24T08:13:19.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.344+0000 7f74d5800640 1 -- 172.21.15.69:0/1267350314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d0075d50 con 0x7f74d0162980 2024-04-24T08:13:19.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.346+0000 7f74c7e00640 1 -- 172.21.15.69:0/1267350314 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f74c800aa00 con 0x7f74d0162980 2024-04-24T08:13:19.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.346+0000 7f74c7e00640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74b006c920 0x7f74b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:19.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.347+0000 7f74c7e00640 1 -- 172.21.15.69:0/1267350314 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f74c808b130 con 0x7f74d0162980 2024-04-24T08:13:19.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.347+0000 7f74cfe00640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74b006c920 0x7f74b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:19.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.348+0000 7f74cfe00640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74b006c920 0x7f74b006ede0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f74c0034000 tx=0x7f74c002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:19.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.351+0000 7f74c7e00640 1 -- 172.21.15.69:0/1267350314 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74c8059430 con 0x7f74d0162980 2024-04-24T08:13:19.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.572+0000 7f74d5800640 1 -- 172.21.15.69:0/1267350314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74d00d1dc0 con 0x7f74d0162980 2024-04-24T08:13:19.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.573+0000 7f74c7e00640 1 -- 172.21.15.69:0/1267350314 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5186 (secure 0 0 0) 0x7f74c8058d30 con 0x7f74d0162980 2024-04-24T08:13:19.574 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:19.574 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":6792,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:13:18.588797+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"b","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1135108804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1135108804},{"type":"v1","addr":"172.21.15.121:6837","nonce":1135108804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":5}]} 2024-04-24T08:13:19.574 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 44 2024-04-24T08:13:19.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.575+0000 7f74d5800640 1 -- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74b006c920 msgr2=0x7f74b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.575+0000 7f74d5800640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74b006c920 0x7f74b006ede0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f74c0034000 tx=0x7f74c002e000 comp rx=0 tx=0).stop 2024-04-24T08:13:19.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.575+0000 7f74d5800640 1 -- 172.21.15.69:0/1267350314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d0162980 msgr2=0x7f74d006e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.575+0000 7f74d5800640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d0162980 0x7f74d006e210 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f74c800c730 tx=0x7f74c800cbf0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.576+0000 7f74d5800640 1 -- 172.21.15.69:0/1267350314 shutdown_connections 2024-04-24T08:13:19.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.576+0000 7f74d5800640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74b006c920 0x7f74b006ede0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.576+0000 7f74d5800640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74d0162980 0x7f74d006e210 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.576+0000 7f74d5800640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74d015e960 0x7f74d006dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.576+0000 7f74d5800640 1 --2- 172.21.15.69:0/1267350314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74d00cfaf0 0x7f74d00ce0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.576+0000 7f74d5800640 1 -- 172.21.15.69:0/1267350314 >> 172.21.15.69:0/1267350314 conn(0x7f74d009d900 msgr2=0x7f74d00b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:19.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.576+0000 7f74d5800640 1 -- 172.21.15.69:0/1267350314 shutdown_connections 2024-04-24T08:13:19.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.576+0000 7f74d5800640 1 -- 172.21.15.69:0/1267350314 wait complete. 2024-04-24T08:13:19.594 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:19.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.791+0000 7f6f78a00640 1 Processor -- start 2024-04-24T08:13:19.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.792+0000 7f6f78a00640 1 -- start start 2024-04-24T08:13:19.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.793+0000 7f6f78a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f740d0200 0x7f6f740d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:19.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.793+0000 7f6f78a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f7415ed40 con 0x7f6f7415e960 2024-04-24T08:13:19.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.793+0000 7f6f78a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f740cf050 con 0x7f6f7406cfb0 2024-04-24T08:13:19.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.793+0000 7f6f78a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6f74162b50 con 0x7f6f740d0200 2024-04-24T08:13:19.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.793+0000 7f6f73e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6f740d0200 0x7f6f740d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:19.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.793+0000 7f6f73400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6f7415e960 0x7f6f74162440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39006/0 (socket says 172.21.15.69:39006) 2024-04-24T08:13:19.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.793+0000 7f6f73400640 1 -- 172.21.15.69:0/123935252 learned_addr learned my addr 172.21.15.69:0/123935252 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:19.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.794+0000 7f6f73e00640 1 -- 172.21.15.69:0/123935252 >> v1:172.21.15.121:6789/0 conn(0x7f6f7406cfb0 legacy=0x7f6f740cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:19.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.794+0000 7f6f73e00640 1 -- 172.21.15.69:0/123935252 >> v1:172.21.15.69:6789/0 conn(0x7f6f7415e960 legacy=0x7f6f74162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.794+0000 7f6f73e00640 1 -- 172.21.15.69:0/123935252 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f74162e20 con 0x7f6f740d0200 2024-04-24T08:13:19.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.794+0000 7f6f72a00640 1 -- 172.21.15.69:0/123935252 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4280973128 0 0) 0x7f6f7415ed40 con 0x7f6f7415e960 2024-04-24T08:13:19.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.795+0000 7f6f73e00640 1 --2- 172.21.15.69:0/123935252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f740d0200 0x7f6f740d2980 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f6f740cee40 tx=0x7f6f6c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b7a9fc2d14c873 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:19.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.795+0000 7f6f72a00640 1 -- 172.21.15.69:0/123935252 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c037070 con 0x7f6f740d0200 2024-04-24T08:13:19.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.795+0000 7f6f72a00640 1 -- 172.21.15.69:0/123935252 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6c032440 con 0x7f6f740d0200 2024-04-24T08:13:19.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.795+0000 7f6f72a00640 1 -- 172.21.15.69:0/123935252 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c032810 con 0x7f6f740d0200 2024-04-24T08:13:19.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.796+0000 7f6f78a00640 1 -- 172.21.15.69:0/123935252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f740d0200 msgr2=0x7f6f740d2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.796+0000 7f6f78a00640 1 --2- 172.21.15.69:0/123935252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f740d0200 0x7f6f740d2980 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f6f740cee40 tx=0x7f6f6c02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.796+0000 7f6f78a00640 1 -- 172.21.15.69:0/123935252 shutdown_connections 2024-04-24T08:13:19.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.796+0000 7f6f78a00640 1 --2- 172.21.15.69:0/123935252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f740d0200 0x7f6f740d2980 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.796+0000 7f6f78a00640 1 -- 172.21.15.69:0/123935252 >> 172.21.15.69:0/123935252 conn(0x7f6f7409d900 msgr2=0x7f6f7408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:19.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.796+0000 7f6f78a00640 1 -- 172.21.15.69:0/123935252 shutdown_connections 2024-04-24T08:13:19.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.796+0000 7f6f78a00640 1 -- 172.21.15.69:0/123935252 wait complete. 2024-04-24T08:13:19.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.798+0000 7f6f78a00640 1 Processor -- start 2024-04-24T08:13:19.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.798+0000 7f6f78a00640 1 -- start start 2024-04-24T08:13:19.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.799+0000 7f6f78a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7406cfb0 0x7f6f741f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.799+0000 7f6f78a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f740d0200 0x7f6f74157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.799+0000 7f6f78a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f7415e960 0x7f6f740c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.799+0000 7f6f78a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f740cf5d0 con 0x7f6f7406cfb0 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.799+0000 7f6f78a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6f740cf2f0 con 0x7f6f7415e960 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.799+0000 7f6f78a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6f740b6170 con 0x7f6f740d0200 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.799+0000 7f6f73400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7406cfb0 0x7f6f741f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.799+0000 7f6f73400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7406cfb0 0x7f6f741f4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41508/0 (socket says 172.21.15.69:41508) 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.799+0000 7f6f73400640 1 -- 172.21.15.69:0/2936210792 learned_addr learned my addr 172.21.15.69:0/2936210792 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f73400640 1 -- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f740d0200 msgr2=0x7f6f74157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f73e00640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f7415e960 0x7f6f740c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f73400640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f740d0200 0x7f6f74157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f73400640 1 -- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f7415e960 msgr2=0x7f6f740c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f73400640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f7415e960 0x7f6f740c8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f73400640 1 -- 172.21.15.69:0/2936210792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f740c9e10 con 0x7f6f7406cfb0 2024-04-24T08:13:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f73400640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7406cfb0 0x7f6f741f4bd0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400c7f0 tx=0x7f6f6400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f6be00640 1 -- 172.21.15.69:0/2936210792 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64018070 con 0x7f6f7406cfb0 2024-04-24T08:13:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f6be00640 1 -- 172.21.15.69:0/2936210792 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6400a640 con 0x7f6f7406cfb0 2024-04-24T08:13:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f78a00640 1 -- 172.21.15.69:0/2936210792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f740c8180 con 0x7f6f7406cfb0 2024-04-24T08:13:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f6be00640 1 -- 172.21.15.69:0/2936210792 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6400aa30 con 0x7f6f7406cfb0 2024-04-24T08:13:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.800+0000 7f6f78a00640 1 -- 172.21.15.69:0/2936210792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f740c8740 con 0x7f6f7406cfb0 2024-04-24T08:13:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.801+0000 7f6f78a00640 1 -- 172.21.15.69:0/2936210792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f74075d50 con 0x7f6f7406cfb0 2024-04-24T08:13:19.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.803+0000 7f6f6be00640 1 -- 172.21.15.69:0/2936210792 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f6401b070 con 0x7f6f7406cfb0 2024-04-24T08:13:19.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.804+0000 7f6f6be00640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6f5006c8f0 0x7f6f5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:19.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.804+0000 7f6f6be00640 1 -- 172.21.15.69:0/2936210792 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f6f6408b150 con 0x7f6f7406cfb0 2024-04-24T08:13:19.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.804+0000 7f6f6aa00640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6f5006c8f0 0x7f6f5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:19.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.805+0000 7f6f6aa00640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6f5006c8f0 0x7f6f5006edb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6f740b5f30 tx=0x7f6f5c0079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:19.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:19.808+0000 7f6f6be00640 1 -- 172.21.15.69:0/2936210792 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f64059570 con 0x7f6f7406cfb0 2024-04-24T08:13:20.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.009+0000 7f6f78a00640 1 -- 172.21.15.69:0/2936210792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f740ba450 con 0x7f6f7406cfb0 2024-04-24T08:13:20.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.012+0000 7f6f6be00640 1 -- 172.21.15.69:0/2936210792 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+16644 (secure 0 0 0) 0x7f6f64058e70 con 0x7f6f7406cfb0 2024-04-24T08:13:20.013 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:20.013 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":68,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:13:17.084219+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:20.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.015+0000 7f6f78a00640 1 -- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6f5006c8f0 msgr2=0x7f6f5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.015+0000 7f6f78a00640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6f5006c8f0 0x7f6f5006edb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6f740b5f30 tx=0x7f6f5c0079b0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.015+0000 7f6f78a00640 1 -- 172.21.15.69:0/2936210792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7406cfb0 msgr2=0x7f6f741f4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.015+0000 7f6f78a00640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7406cfb0 0x7f6f741f4bd0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400c7f0 tx=0x7f6f6400ccb0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.015+0000 7f6f78a00640 1 -- 172.21.15.69:0/2936210792 shutdown_connections 2024-04-24T08:13:20.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.015+0000 7f6f78a00640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6f5006c8f0 0x7f6f5006edb0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.015+0000 7f6f78a00640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f7415e960 0x7f6f740c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.015+0000 7f6f78a00640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6f740d0200 0x7f6f74157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.015+0000 7f6f78a00640 1 --2- 172.21.15.69:0/2936210792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f7406cfb0 0x7f6f741f4bd0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.015+0000 7f6f78a00640 1 -- 172.21.15.69:0/2936210792 >> 172.21.15.69:0/2936210792 conn(0x7f6f7409d900 msgr2=0x7f6f740b5200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:20.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.016+0000 7f6f78a00640 1 -- 172.21.15.69:0/2936210792 shutdown_connections 2024-04-24T08:13:20.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.016+0000 7f6f78a00640 1 -- 172.21.15.69:0/2936210792 wait complete. 2024-04-24T08:13:20.036 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T08:13:20.036 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:13:20.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.236+0000 7fcfffe00640 1 Processor -- start 2024-04-24T08:13:20.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.236+0000 7fcfffe00640 1 -- start start 2024-04-24T08:13:20.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.237+0000 7fcfffe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcff806f2d0 0x7fcff806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.238+0000 7fcfffe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcff80b46e0 con 0x7fcff815b490 2024-04-24T08:13:20.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.238+0000 7fcfffe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcff80d0960 con 0x7fcff80cfbe0 2024-04-24T08:13:20.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.238+0000 7fcfffe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcff806fc70 con 0x7fcff806f2d0 2024-04-24T08:13:20.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.238+0000 7fcffd400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcff815b490 0x7fcff80b5180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39020/0 (socket says 172.21.15.69:39020) 2024-04-24T08:13:20.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.238+0000 7fcffd400640 1 -- 172.21.15.69:0/809135793 learned_addr learned my addr 172.21.15.69:0/809135793 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:20.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.238+0000 7fcffca00640 1 --2- 172.21.15.69:0/809135793 >> v2:172.21.15.69:3300/0 conn(0x7fcff806f2d0 0x7fcff806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.238+0000 7fcff7e00640 1 -- 172.21.15.69:0/809135793 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1404112128 0 0) 0x7fcff80b46e0 con 0x7fcff815b490 2024-04-24T08:13:20.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.238+0000 7fcff7e00640 1 -- 172.21.15.69:0/809135793 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfe0003520 con 0x7fcff815b490 2024-04-24T08:13:20.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.239+0000 7fcffca00640 1 -- 172.21.15.69:0/809135793 >> v1:172.21.15.121:6789/0 conn(0x7fcff80cfbe0 legacy=0x7fcff80d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.239+0000 7fcffca00640 1 -- 172.21.15.69:0/809135793 >> v1:172.21.15.69:6789/0 conn(0x7fcff815b490 legacy=0x7fcff80b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.239+0000 7fcff7e00640 1 -- 172.21.15.69:0/809135793 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 17015144 0 0) 0x7fcfe0003520 con 0x7fcff815b490 2024-04-24T08:13:20.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.239+0000 7fcffca00640 1 -- 172.21.15.69:0/809135793 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff80b4980 con 0x7fcff806f2d0 2024-04-24T08:13:20.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.240+0000 7fcffca00640 1 --2- 172.21.15.69:0/809135793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff806f2d0 0x7fcff806f730 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40097a0 tx=0x7fcfe402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5f46ba92bdedf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:20.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.240+0000 7fcff7e00640 1 -- 172.21.15.69:0/809135793 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4037070 con 0x7fcff806f2d0 2024-04-24T08:13:20.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.240+0000 7fcff7e00640 1 -- 172.21.15.69:0/809135793 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfe4032440 con 0x7fcff806f2d0 2024-04-24T08:13:20.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.240+0000 7fcff7e00640 1 -- 172.21.15.69:0/809135793 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4032810 con 0x7fcff806f2d0 2024-04-24T08:13:20.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.241+0000 7fcfffe00640 1 -- 172.21.15.69:0/809135793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff806f2d0 msgr2=0x7fcff806f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.241+0000 7fcfffe00640 1 --2- 172.21.15.69:0/809135793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff806f2d0 0x7fcff806f730 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40097a0 tx=0x7fcfe402a230 comp rx=0 tx=0).stop 2024-04-24T08:13:20.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.241+0000 7fcfffe00640 1 -- 172.21.15.69:0/809135793 shutdown_connections 2024-04-24T08:13:20.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.241+0000 7fcfffe00640 1 --2- 172.21.15.69:0/809135793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff806f2d0 0x7fcff806f730 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.241+0000 7fcfffe00640 1 -- 172.21.15.69:0/809135793 >> 172.21.15.69:0/809135793 conn(0x7fcff8067d90 msgr2=0x7fcff806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:20.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.241+0000 7fcfffe00640 1 -- 172.21.15.69:0/809135793 shutdown_connections 2024-04-24T08:13:20.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.241+0000 7fcfffe00640 1 -- 172.21.15.69:0/809135793 wait complete. 2024-04-24T08:13:20.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.243+0000 7fcfffe00640 1 Processor -- start 2024-04-24T08:13:20.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.243+0000 7fcfffe00640 1 -- start start 2024-04-24T08:13:20.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.244+0000 7fcfffe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcff806f2d0 0x7fcff8150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.244+0000 7fcfffe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff80cfbe0 0x7fcff81f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.244+0000 7fcfffe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcff815b490 0x7fcff8158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.244+0000 7fcfffe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcff80b5890 con 0x7fcff80cfbe0 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.244+0000 7fcfffe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcff80d0720 con 0x7fcff806f2d0 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.244+0000 7fcfffe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcff80d0c40 con 0x7fcff815b490 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.244+0000 7fcffca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff80cfbe0 0x7fcff81f7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.244+0000 7fcffca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff80cfbe0 0x7fcff81f7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41522/0 (socket says 172.21.15.69:41522) 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.244+0000 7fcffca00640 1 -- 172.21.15.69:0/1699243901 learned_addr learned my addr 172.21.15.69:0/1699243901 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.245+0000 7fcffd400640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcff806f2d0 0x7fcff8150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.245+0000 7fcffde00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcff815b490 0x7fcff8158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.245+0000 7fcffca00640 1 -- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcff815b490 msgr2=0x7fcff8158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.245+0000 7fcffca00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcff815b490 0x7fcff8158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.245+0000 7fcffca00640 1 -- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcff806f2d0 msgr2=0x7fcff8150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.245+0000 7fcffca00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcff806f2d0 0x7fcff8150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.245+0000 7fcffca00640 1 -- 172.21.15.69:0/1699243901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff81f87e0 con 0x7fcff80cfbe0 2024-04-24T08:13:20.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.245+0000 7fcffde00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcff815b490 0x7fcff8158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:20.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.245+0000 7fcffca00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff80cfbe0 0x7fcff81f7e20 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40098c0 tx=0x7fcfe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:20.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.246+0000 7fcff5600640 1 -- 172.21.15.69:0/1699243901 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4041020 con 0x7fcff80cfbe0 2024-04-24T08:13:20.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.246+0000 7fcff5600640 1 -- 172.21.15.69:0/1699243901 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfe403d6e0 con 0x7fcff80cfbe0 2024-04-24T08:13:20.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.246+0000 7fcff5600640 1 -- 172.21.15.69:0/1699243901 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4037040 con 0x7fcff80cfbe0 2024-04-24T08:13:20.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.246+0000 7fcfffe00640 1 -- 172.21.15.69:0/1699243901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff81f8ac0 con 0x7fcff80cfbe0 2024-04-24T08:13:20.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.246+0000 7fcfffe00640 1 -- 172.21.15.69:0/1699243901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff8156200 con 0x7fcff80cfbe0 2024-04-24T08:13:20.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.247+0000 7fcfffe00640 1 -- 172.21.15.69:0/1699243901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff8068a80 con 0x7fcff80cfbe0 2024-04-24T08:13:20.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.248+0000 7fcff5600640 1 -- 172.21.15.69:0/1699243901 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcfe4044020 con 0x7fcff80cfbe0 2024-04-24T08:13:20.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.249+0000 7fcff5600640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfd806c920 0x7fcfd806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.249+0000 7fcff5600640 1 -- 172.21.15.69:0/1699243901 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fcfe40ac310 con 0x7fcff80cfbe0 2024-04-24T08:13:20.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.249+0000 7fcffd400640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfd806c920 0x7fcfd806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.250+0000 7fcffd400640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfd806c920 0x7fcfd806ede0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcfec000fb0 tx=0x7fcfec007a90 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:20.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.253+0000 7fcff5600640 1 -- 172.21.15.69:0/1699243901 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcfe404b030 con 0x7fcff80cfbe0 2024-04-24T08:13:20.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.451+0000 7fcfffe00640 1 -- 172.21.15.69:0/1699243901 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcff8156430 con 0x7fcfd806c920 2024-04-24T08:13:20.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.472+0000 7fcff5600640 1 -- 172.21.15.69:0/1699243901 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252069 (secure 0 0 0) 0x7fcff8156430 con 0x7fcfd806c920 2024-04-24T08:13:20.478 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:20.480 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.477+0000 7fcfffe00640 1 -- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfd806c920 msgr2=0x7fcfd806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.477+0000 7fcfffe00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfd806c920 0x7fcfd806ede0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcfec000fb0 tx=0x7fcfec007a90 comp rx=0 tx=0).stop 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.477+0000 7fcfffe00640 1 -- 172.21.15.69:0/1699243901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff80cfbe0 msgr2=0x7fcff81f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.477+0000 7fcfffe00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff80cfbe0 0x7fcff81f7e20 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40098c0 tx=0x7fcfe40046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.478+0000 7fcfffe00640 1 -- 172.21.15.69:0/1699243901 shutdown_connections 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.478+0000 7fcfffe00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfd806c920 0x7fcfd806ede0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.478+0000 7fcfffe00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff80cfbe0 0x7fcff81f7e20 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.478+0000 7fcfffe00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcff815b490 0x7fcff8158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.478+0000 7fcfffe00640 1 --2- 172.21.15.69:0/1699243901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcff806f2d0 0x7fcff8150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.478+0000 7fcfffe00640 1 -- 172.21.15.69:0/1699243901 >> 172.21.15.69:0/1699243901 conn(0x7fcff8067d90 msgr2=0x7fcff8159fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:20.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.478+0000 7fcfffe00640 1 -- 172.21.15.69:0/1699243901 shutdown_connections 2024-04-24T08:13:20.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.478+0000 7fcfffe00640 1 -- 172.21.15.69:0/1699243901 wait complete. 2024-04-24T08:13:20.497 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-04-24T08:13:18.771728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":180,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252992,"kb_used_data":7688,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721728,"statfs":{"total":773094113280,"available":772835049472,"internally_reserved":0,"allocated":7872512,"data_stored":3973652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.741225"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234739+0000","last_change":"2024-04-24T08:13:16.117848+0000","last_active":"2024-04-24T08:13:17.234739+0000","last_peered":"2024-04-24T08:13:17.234739+0000","last_clean":"2024-04-24T08:13:17.234739+0000","last_became_active":"2024-04-24T08:13:16.116700+0000","last_became_peered":"2024-04-24T08:13:16.116700+0000","last_unstale":"2024-04-24T08:13:17.234739+0000","last_undegraded":"2024-04-24T08:13:17.234739+0000","last_fullsized":"2024-04-24T08:13:17.234739+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:10:15.310592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101681+0000","last_change":"2024-04-24T08:13:16.093849+0000","last_active":"2024-04-24T08:13:17.101681+0000","last_peered":"2024-04-24T08:13:17.101681+0000","last_clean":"2024-04-24T08:13:17.101681+0000","last_became_active":"2024-04-24T08:13:16.093479+0000","last_became_peered":"2024-04-24T08:13:16.093479+0000","last_unstale":"2024-04-24T08:13:17.101681+0000","last_undegraded":"2024-04-24T08:13:17.101681+0000","last_fullsized":"2024-04-24T08:13:17.101681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:10.923326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230645+0000","last_change":"2024-04-24T08:13:16.098526+0000","last_active":"2024-04-24T08:13:17.230645+0000","last_peered":"2024-04-24T08:13:17.230645+0000","last_clean":"2024-04-24T08:13:17.230645+0000","last_became_active":"2024-04-24T08:13:16.098166+0000","last_became_peered":"2024-04-24T08:13:16.098166+0000","last_unstale":"2024-04-24T08:13:17.230645+0000","last_undegraded":"2024-04-24T08:13:17.230645+0000","last_fullsized":"2024-04-24T08:13:17.230645+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:21:00.059160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231248+0000","last_change":"2024-04-24T08:13:16.105602+0000","last_active":"2024-04-24T08:13:17.231248+0000","last_peered":"2024-04-24T08:13:17.231248+0000","last_clean":"2024-04-24T08:13:17.231248+0000","last_became_active":"2024-04-24T08:13:16.105103+0000","last_became_peered":"2024-04-24T08:13:16.105103+0000","last_unstale":"2024-04-24T08:13:17.231248+0000","last_undegraded":"2024-04-24T08:13:17.231248+0000","last_fullsized":"2024-04-24T08:13:17.231248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:57:17.202289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.099928+0000","last_change":"2024-04-24T08:13:16.112786+0000","last_active":"2024-04-24T08:13:17.099928+0000","last_peered":"2024-04-24T08:13:17.099928+0000","last_clean":"2024-04-24T08:13:17.099928+0000","last_became_active":"2024-04-24T08:13:16.112508+0000","last_became_peered":"2024-04-24T08:13:16.112508+0000","last_unstale":"2024-04-24T08:13:17.099928+0000","last_undegraded":"2024-04-24T08:13:17.099928+0000","last_fullsized":"2024-04-24T08:13:17.099928+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:34:09.318804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100546+0000","last_change":"2024-04-24T08:13:16.095900+0000","last_active":"2024-04-24T08:13:17.100546+0000","last_peered":"2024-04-24T08:13:17.100546+0000","last_clean":"2024-04-24T08:13:17.100546+0000","last_became_active":"2024-04-24T08:13:16.095589+0000","last_became_peered":"2024-04-24T08:13:16.095589+0000","last_unstale":"2024-04-24T08:13:17.100546+0000","last_undegraded":"2024-04-24T08:13:17.100546+0000","last_fullsized":"2024-04-24T08:13:17.100546+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:03:31.209692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100470+0000","last_change":"2024-04-24T08:13:16.087673+0000","last_active":"2024-04-24T08:13:17.100470+0000","last_peered":"2024-04-24T08:13:17.100470+0000","last_clean":"2024-04-24T08:13:17.100470+0000","last_became_active":"2024-04-24T08:13:16.087370+0000","last_became_peered":"2024-04-24T08:13:16.087370+0000","last_unstale":"2024-04-24T08:13:17.100470+0000","last_undegraded":"2024-04-24T08:13:17.100470+0000","last_fullsized":"2024-04-24T08:13:17.100470+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:37.376610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100396+0000","last_change":"2024-04-24T08:13:16.099155+0000","last_active":"2024-04-24T08:13:17.100396+0000","last_peered":"2024-04-24T08:13:17.100396+0000","last_clean":"2024-04-24T08:13:17.100396+0000","last_became_active":"2024-04-24T08:13:16.098838+0000","last_became_peered":"2024-04-24T08:13:16.098838+0000","last_unstale":"2024-04-24T08:13:17.100396+0000","last_undegraded":"2024-04-24T08:13:17.100396+0000","last_fullsized":"2024-04-24T08:13:17.100396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:42:42.296218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100633+0000","last_change":"2024-04-24T08:13:16.103940+0000","last_active":"2024-04-24T08:13:17.100633+0000","last_peered":"2024-04-24T08:13:17.100633+0000","last_clean":"2024-04-24T08:13:17.100633+0000","last_became_active":"2024-04-24T08:13:16.103628+0000","last_became_peered":"2024-04-24T08:13:16.103628+0000","last_unstale":"2024-04-24T08:13:17.100633+0000","last_undegraded":"2024-04-24T08:13:17.100633+0000","last_fullsized":"2024-04-24T08:13:17.100633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:54:00.896362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234085+0000","last_change":"2024-04-24T08:13:16.096499+0000","last_active":"2024-04-24T08:13:17.234085+0000","last_peered":"2024-04-24T08:13:17.234085+0000","last_clean":"2024-04-24T08:13:17.234085+0000","last_became_active":"2024-04-24T08:13:16.096189+0000","last_became_peered":"2024-04-24T08:13:16.096189+0000","last_unstale":"2024-04-24T08:13:17.234085+0000","last_undegraded":"2024-04-24T08:13:17.234085+0000","last_fullsized":"2024-04-24T08:13:17.234085+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:20:12.916151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235273+0000","last_change":"2024-04-24T08:13:16.113958+0000","last_active":"2024-04-24T08:13:17.235273+0000","last_peered":"2024-04-24T08:13:17.235273+0000","last_clean":"2024-04-24T08:13:17.235273+0000","last_became_active":"2024-04-24T08:13:16.113372+0000","last_became_peered":"2024-04-24T08:13:16.113372+0000","last_unstale":"2024-04-24T08:13:17.235273+0000","last_undegraded":"2024-04-24T08:13:17.235273+0000","last_fullsized":"2024-04-24T08:13:17.235273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:25.287257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231324+0000","last_change":"2024-04-24T08:13:16.095382+0000","last_active":"2024-04-24T08:13:17.231324+0000","last_peered":"2024-04-24T08:13:17.231324+0000","last_clean":"2024-04-24T08:13:17.231324+0000","last_became_active":"2024-04-24T08:13:16.093955+0000","last_became_peered":"2024-04-24T08:13:16.093955+0000","last_unstale":"2024-04-24T08:13:17.231324+0000","last_undegraded":"2024-04-24T08:13:17.231324+0000","last_fullsized":"2024-04-24T08:13:17.231324+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:22.319406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115254+0000","last_change":"2024-04-24T08:13:16.099815+0000","last_active":"2024-04-24T08:13:17.115254+0000","last_peered":"2024-04-24T08:13:17.115254+0000","last_clean":"2024-04-24T08:13:17.115254+0000","last_became_active":"2024-04-24T08:13:16.099418+0000","last_became_peered":"2024-04-24T08:13:16.099418+0000","last_unstale":"2024-04-24T08:13:17.115254+0000","last_undegraded":"2024-04-24T08:13:17.115254+0000","last_fullsized":"2024-04-24T08:13:17.115254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:25.287257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.099950+0000","last_change":"2024-04-24T08:13:16.096718+0000","last_active":"2024-04-24T08:13:17.099950+0000","last_peered":"2024-04-24T08:13:17.099950+0000","last_clean":"2024-04-24T08:13:17.099950+0000","last_became_active":"2024-04-24T08:13:16.096433+0000","last_became_peered":"2024-04-24T08:13:16.096433+0000","last_unstale":"2024-04-24T08:13:17.099950+0000","last_undegraded":"2024-04-24T08:13:17.099950+0000","last_fullsized":"2024-04-24T08:13:17.099950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:18:54.741567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115053+0000","last_change":"2024-04-24T08:13:16.089445+0000","last_active":"2024-04-24T08:13:17.115053+0000","last_peered":"2024-04-24T08:13:17.115053+0000","last_clean":"2024-04-24T08:13:17.115053+0000","last_became_active":"2024-04-24T08:13:16.089177+0000","last_became_peered":"2024-04-24T08:13:16.089177+0000","last_unstale":"2024-04-24T08:13:17.115053+0000","last_undegraded":"2024-04-24T08:13:17.115053+0000","last_fullsized":"2024-04-24T08:13:17.115053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:26:04.544052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233852+0000","last_change":"2024-04-24T08:13:16.105962+0000","last_active":"2024-04-24T08:13:17.233852+0000","last_peered":"2024-04-24T08:13:17.233852+0000","last_clean":"2024-04-24T08:13:17.233852+0000","last_became_active":"2024-04-24T08:13:16.105690+0000","last_became_peered":"2024-04-24T08:13:16.105690+0000","last_unstale":"2024-04-24T08:13:17.233852+0000","last_undegraded":"2024-04-24T08:13:17.233852+0000","last_fullsized":"2024-04-24T08:13:17.233852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:41:17.847511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100767+0000","last_change":"2024-04-24T08:13:16.102963+0000","last_active":"2024-04-24T08:13:17.100767+0000","last_peered":"2024-04-24T08:13:17.100767+0000","last_clean":"2024-04-24T08:13:17.100767+0000","last_became_active":"2024-04-24T08:13:16.102073+0000","last_became_peered":"2024-04-24T08:13:16.102073+0000","last_unstale":"2024-04-24T08:13:17.100767+0000","last_undegraded":"2024-04-24T08:13:17.100767+0000","last_fullsized":"2024-04-24T08:13:17.100767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:57.278440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231359+0000","last_change":"2024-04-24T08:13:16.099981+0000","last_active":"2024-04-24T08:13:17.231359+0000","last_peered":"2024-04-24T08:13:17.231359+0000","last_clean":"2024-04-24T08:13:17.231359+0000","last_became_active":"2024-04-24T08:13:16.099701+0000","last_became_peered":"2024-04-24T08:13:16.099701+0000","last_unstale":"2024-04-24T08:13:17.231359+0000","last_undegraded":"2024-04-24T08:13:17.231359+0000","last_fullsized":"2024-04-24T08:13:17.231359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:51:29.950774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101982+0000","last_change":"2024-04-24T08:13:16.101112+0000","last_active":"2024-04-24T08:13:17.101982+0000","last_peered":"2024-04-24T08:13:17.101982+0000","last_clean":"2024-04-24T08:13:17.101982+0000","last_became_active":"2024-04-24T08:13:16.100574+0000","last_became_peered":"2024-04-24T08:13:16.100574+0000","last_unstale":"2024-04-24T08:13:17.101982+0000","last_undegraded":"2024-04-24T08:13:17.101982+0000","last_fullsized":"2024-04-24T08:13:17.101982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:20:45.863887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115322+0000","last_change":"2024-04-24T08:13:16.092810+0000","last_active":"2024-04-24T08:13:17.115322+0000","last_peered":"2024-04-24T08:13:17.115322+0000","last_clean":"2024-04-24T08:13:17.115322+0000","last_became_active":"2024-04-24T08:13:16.092423+0000","last_became_peered":"2024-04-24T08:13:16.092423+0000","last_unstale":"2024-04-24T08:13:17.115322+0000","last_undegraded":"2024-04-24T08:13:17.115322+0000","last_fullsized":"2024-04-24T08:13:17.115322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:12:06.874957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233248+0000","last_change":"2024-04-24T08:13:16.093298+0000","last_active":"2024-04-24T08:13:17.233248+0000","last_peered":"2024-04-24T08:13:17.233248+0000","last_clean":"2024-04-24T08:13:17.233248+0000","last_became_active":"2024-04-24T08:13:16.092990+0000","last_became_peered":"2024-04-24T08:13:16.092990+0000","last_unstale":"2024-04-24T08:13:17.233248+0000","last_undegraded":"2024-04-24T08:13:17.233248+0000","last_fullsized":"2024-04-24T08:13:17.233248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:37.376610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233452+0000","last_change":"2024-04-24T08:13:16.102034+0000","last_active":"2024-04-24T08:13:17.233452+0000","last_peered":"2024-04-24T08:13:17.233452+0000","last_clean":"2024-04-24T08:13:17.233452+0000","last_became_active":"2024-04-24T08:13:16.101741+0000","last_became_peered":"2024-04-24T08:13:16.101741+0000","last_unstale":"2024-04-24T08:13:17.233452+0000","last_undegraded":"2024-04-24T08:13:17.233452+0000","last_fullsized":"2024-04-24T08:13:17.233452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:20:45.863887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233146+0000","last_change":"2024-04-24T08:13:16.111440+0000","last_active":"2024-04-24T08:13:17.233146+0000","last_peered":"2024-04-24T08:13:17.233146+0000","last_clean":"2024-04-24T08:13:17.233146+0000","last_became_active":"2024-04-24T08:13:16.110858+0000","last_became_peered":"2024-04-24T08:13:16.110858+0000","last_unstale":"2024-04-24T08:13:17.233146+0000","last_undegraded":"2024-04-24T08:13:17.233146+0000","last_fullsized":"2024-04-24T08:13:17.233146+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:41.452749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233360+0000","last_change":"2024-04-24T08:13:16.107244+0000","last_active":"2024-04-24T08:13:17.233360+0000","last_peered":"2024-04-24T08:13:17.233360+0000","last_clean":"2024-04-24T08:13:17.233360+0000","last_became_active":"2024-04-24T08:13:16.105966+0000","last_became_peered":"2024-04-24T08:13:16.105966+0000","last_unstale":"2024-04-24T08:13:17.233360+0000","last_undegraded":"2024-04-24T08:13:17.233360+0000","last_fullsized":"2024-04-24T08:13:17.233360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:20:12.916151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233873+0000","last_change":"2024-04-24T08:13:16.107076+0000","last_active":"2024-04-24T08:13:17.233873+0000","last_peered":"2024-04-24T08:13:17.233873+0000","last_clean":"2024-04-24T08:13:17.233873+0000","last_became_active":"2024-04-24T08:13:16.106751+0000","last_became_peered":"2024-04-24T08:13:16.106751+0000","last_unstale":"2024-04-24T08:13:17.233873+0000","last_undegraded":"2024-04-24T08:13:17.233873+0000","last_fullsized":"2024-04-24T08:13:17.233873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:41.452749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.102687+0000","last_change":"2024-04-24T08:13:16.101998+0000","last_active":"2024-04-24T08:13:17.102687+0000","last_peered":"2024-04-24T08:13:17.102687+0000","last_clean":"2024-04-24T08:13:17.102687+0000","last_became_active":"2024-04-24T08:13:16.101546+0000","last_became_peered":"2024-04-24T08:13:16.101546+0000","last_unstale":"2024-04-24T08:13:17.102687+0000","last_undegraded":"2024-04-24T08:13:17.102687+0000","last_fullsized":"2024-04-24T08:13:17.102687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:17:23.703236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230757+0000","last_change":"2024-04-24T08:13:16.111564+0000","last_active":"2024-04-24T08:13:17.230757+0000","last_peered":"2024-04-24T08:13:17.230757+0000","last_clean":"2024-04-24T08:13:17.230757+0000","last_became_active":"2024-04-24T08:13:16.111154+0000","last_became_peered":"2024-04-24T08:13:16.111154+0000","last_unstale":"2024-04-24T08:13:17.230757+0000","last_undegraded":"2024-04-24T08:13:17.230757+0000","last_fullsized":"2024-04-24T08:13:17.230757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:39:28.319204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233301+0000","last_change":"2024-04-24T08:13:16.101359+0000","last_active":"2024-04-24T08:13:17.233301+0000","last_peered":"2024-04-24T08:13:17.233301+0000","last_clean":"2024-04-24T08:13:17.233301+0000","last_became_active":"2024-04-24T08:13:16.096175+0000","last_became_peered":"2024-04-24T08:13:16.096175+0000","last_unstale":"2024-04-24T08:13:17.233301+0000","last_undegraded":"2024-04-24T08:13:17.233301+0000","last_fullsized":"2024-04-24T08:13:17.233301+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:10.923326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230671+0000","last_change":"2024-04-24T08:13:16.098975+0000","last_active":"2024-04-24T08:13:17.230671+0000","last_peered":"2024-04-24T08:13:17.230671+0000","last_clean":"2024-04-24T08:13:17.230671+0000","last_became_active":"2024-04-24T08:13:16.097915+0000","last_became_peered":"2024-04-24T08:13:16.097915+0000","last_unstale":"2024-04-24T08:13:17.230671+0000","last_undegraded":"2024-04-24T08:13:17.230671+0000","last_fullsized":"2024-04-24T08:13:17.230671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:05:38.937402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101363+0000","last_change":"2024-04-24T08:13:16.112216+0000","last_active":"2024-04-24T08:13:17.101363+0000","last_peered":"2024-04-24T08:13:17.101363+0000","last_clean":"2024-04-24T08:13:17.101363+0000","last_became_active":"2024-04-24T08:13:16.111872+0000","last_became_peered":"2024-04-24T08:13:16.111872+0000","last_unstale":"2024-04-24T08:13:17.101363+0000","last_undegraded":"2024-04-24T08:13:17.101363+0000","last_fullsized":"2024-04-24T08:13:17.101363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:41:17.847511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100757+0000","last_change":"2024-04-24T08:13:16.094867+0000","last_active":"2024-04-24T08:13:17.100757+0000","last_peered":"2024-04-24T08:13:17.100757+0000","last_clean":"2024-04-24T08:13:17.100757+0000","last_became_active":"2024-04-24T08:13:16.090985+0000","last_became_peered":"2024-04-24T08:13:16.090985+0000","last_unstale":"2024-04-24T08:13:17.100757+0000","last_undegraded":"2024-04-24T08:13:17.100757+0000","last_fullsized":"2024-04-24T08:13:17.100757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:10.923326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100732+0000","last_change":"2024-04-24T08:13:16.099179+0000","last_active":"2024-04-24T08:13:17.100732+0000","last_peered":"2024-04-24T08:13:17.100732+0000","last_clean":"2024-04-24T08:13:17.100732+0000","last_became_active":"2024-04-24T08:13:16.098897+0000","last_became_peered":"2024-04-24T08:13:16.098897+0000","last_unstale":"2024-04-24T08:13:17.100732+0000","last_undegraded":"2024-04-24T08:13:17.100732+0000","last_fullsized":"2024-04-24T08:13:17.100732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:18:20.774821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234246+0000","last_change":"2024-04-24T08:13:16.107049+0000","last_active":"2024-04-24T08:13:17.234246+0000","last_peered":"2024-04-24T08:13:17.234246+0000","last_clean":"2024-04-24T08:13:17.234246+0000","last_became_active":"2024-04-24T08:13:16.106755+0000","last_became_peered":"2024-04-24T08:13:16.106755+0000","last_unstale":"2024-04-24T08:13:17.234246+0000","last_undegraded":"2024-04-24T08:13:17.234246+0000","last_fullsized":"2024-04-24T08:13:17.234246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:03.715939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235164+0000","last_change":"2024-04-24T08:13:15.075208+0000","last_active":"2024-04-24T08:13:17.235164+0000","last_peered":"2024-04-24T08:13:17.235164+0000","last_clean":"2024-04-24T08:13:17.235164+0000","last_became_active":"2024-04-24T08:13:15.074904+0000","last_became_peered":"2024-04-24T08:13:15.074904+0000","last_unstale":"2024-04-24T08:13:17.235164+0000","last_undegraded":"2024-04-24T08:13:17.235164+0000","last_fullsized":"2024-04-24T08:13:17.235164+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:28.677994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100972+0000","last_change":"2024-04-24T08:13:16.102681+0000","last_active":"2024-04-24T08:13:17.100972+0000","last_peered":"2024-04-24T08:13:17.100972+0000","last_clean":"2024-04-24T08:13:17.100972+0000","last_became_active":"2024-04-24T08:13:16.101584+0000","last_became_peered":"2024-04-24T08:13:16.101584+0000","last_unstale":"2024-04-24T08:13:17.100972+0000","last_undegraded":"2024-04-24T08:13:17.100972+0000","last_fullsized":"2024-04-24T08:13:17.100972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:03:17.422393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115478+0000","last_change":"2024-04-24T08:13:15.065334+0000","last_active":"2024-04-24T08:13:17.115478+0000","last_peered":"2024-04-24T08:13:17.115478+0000","last_clean":"2024-04-24T08:13:17.115478+0000","last_became_active":"2024-04-24T08:13:15.064990+0000","last_became_peered":"2024-04-24T08:13:15.064990+0000","last_unstale":"2024-04-24T08:13:17.115478+0000","last_undegraded":"2024-04-24T08:13:17.115478+0000","last_fullsized":"2024-04-24T08:13:17.115478+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:20:44.849545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.102512+0000","last_change":"2024-04-24T08:13:16.110937+0000","last_active":"2024-04-24T08:13:17.102512+0000","last_peered":"2024-04-24T08:13:17.102512+0000","last_clean":"2024-04-24T08:13:17.102512+0000","last_became_active":"2024-04-24T08:13:16.109194+0000","last_became_peered":"2024-04-24T08:13:16.109194+0000","last_unstale":"2024-04-24T08:13:17.102512+0000","last_undegraded":"2024-04-24T08:13:17.102512+0000","last_fullsized":"2024-04-24T08:13:17.102512+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:41.452749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233449+0000","last_change":"2024-04-24T08:13:15.075504+0000","last_active":"2024-04-24T08:13:17.233449+0000","last_peered":"2024-04-24T08:13:17.233449+0000","last_clean":"2024-04-24T08:13:17.233449+0000","last_became_active":"2024-04-24T08:13:15.075216+0000","last_became_peered":"2024-04-24T08:13:15.075216+0000","last_unstale":"2024-04-24T08:13:17.233449+0000","last_undegraded":"2024-04-24T08:13:17.233449+0000","last_fullsized":"2024-04-24T08:13:17.233449+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:35.089414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235388+0000","last_change":"2024-04-24T08:13:16.114272+0000","last_active":"2024-04-24T08:13:17.235388+0000","last_peered":"2024-04-24T08:13:17.235388+0000","last_clean":"2024-04-24T08:13:17.235388+0000","last_became_active":"2024-04-24T08:13:16.113383+0000","last_became_peered":"2024-04-24T08:13:16.113383+0000","last_unstale":"2024-04-24T08:13:17.235388+0000","last_undegraded":"2024-04-24T08:13:17.235388+0000","last_fullsized":"2024-04-24T08:13:17.235388+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:12:06.874957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.102382+0000","last_change":"2024-04-24T08:13:15.070163+0000","last_active":"2024-04-24T08:13:17.102382+0000","last_peered":"2024-04-24T08:13:17.102382+0000","last_clean":"2024-04-24T08:13:17.102382+0000","last_became_active":"2024-04-24T08:13:15.069791+0000","last_became_peered":"2024-04-24T08:13:15.069791+0000","last_unstale":"2024-04-24T08:13:17.102382+0000","last_undegraded":"2024-04-24T08:13:17.102382+0000","last_fullsized":"2024-04-24T08:13:17.102382+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:28.677994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101230+0000","last_change":"2024-04-24T08:13:16.096903+0000","last_active":"2024-04-24T08:13:17.101230+0000","last_peered":"2024-04-24T08:13:17.101230+0000","last_clean":"2024-04-24T08:13:17.101230+0000","last_became_active":"2024-04-24T08:13:16.096517+0000","last_became_peered":"2024-04-24T08:13:16.096517+0000","last_unstale":"2024-04-24T08:13:17.101230+0000","last_undegraded":"2024-04-24T08:13:17.101230+0000","last_fullsized":"2024-04-24T08:13:17.101230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:20:45.863887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231011+0000","last_change":"2024-04-24T08:13:15.073598+0000","last_active":"2024-04-24T08:13:17.231011+0000","last_peered":"2024-04-24T08:13:17.231011+0000","last_clean":"2024-04-24T08:13:17.231011+0000","last_became_active":"2024-04-24T08:13:15.073267+0000","last_became_peered":"2024-04-24T08:13:15.073267+0000","last_unstale":"2024-04-24T08:13:17.231011+0000","last_undegraded":"2024-04-24T08:13:17.231011+0000","last_fullsized":"2024-04-24T08:13:17.231011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:50:23.307699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234607+0000","last_change":"2024-04-24T08:13:16.107693+0000","last_active":"2024-04-24T08:13:17.234607+0000","last_peered":"2024-04-24T08:13:17.234607+0000","last_clean":"2024-04-24T08:13:17.234607+0000","last_became_active":"2024-04-24T08:13:16.106228+0000","last_became_peered":"2024-04-24T08:13:16.106228+0000","last_unstale":"2024-04-24T08:13:17.234607+0000","last_undegraded":"2024-04-24T08:13:17.234607+0000","last_fullsized":"2024-04-24T08:13:17.234607+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:17:23.703236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100688+0000","last_change":"2024-04-24T08:13:15.069709+0000","last_active":"2024-04-24T08:13:17.100688+0000","last_peered":"2024-04-24T08:13:17.100688+0000","last_clean":"2024-04-24T08:13:17.100688+0000","last_became_active":"2024-04-24T08:13:15.068812+0000","last_became_peered":"2024-04-24T08:13:15.068812+0000","last_unstale":"2024-04-24T08:13:17.100688+0000","last_undegraded":"2024-04-24T08:13:17.100688+0000","last_fullsized":"2024-04-24T08:13:17.100688+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:25:03.246516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.099465+0000","last_change":"2024-04-24T08:13:16.091291+0000","last_active":"2024-04-24T08:13:17.099465+0000","last_peered":"2024-04-24T08:13:17.099465+0000","last_clean":"2024-04-24T08:13:17.099465+0000","last_became_active":"2024-04-24T08:13:16.091013+0000","last_became_peered":"2024-04-24T08:13:16.091013+0000","last_unstale":"2024-04-24T08:13:17.099465+0000","last_undegraded":"2024-04-24T08:13:17.099465+0000","last_fullsized":"2024-04-24T08:13:17.099465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:48:53.506577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230934+0000","last_change":"2024-04-24T08:13:15.074271+0000","last_active":"2024-04-24T08:13:17.230934+0000","last_peered":"2024-04-24T08:13:17.230934+0000","last_clean":"2024-04-24T08:13:17.230934+0000","last_became_active":"2024-04-24T08:13:15.073770+0000","last_became_peered":"2024-04-24T08:13:15.073770+0000","last_unstale":"2024-04-24T08:13:17.230934+0000","last_undegraded":"2024-04-24T08:13:17.230934+0000","last_fullsized":"2024-04-24T08:13:17.230934+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:56.264098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.114831+0000","last_change":"2024-04-24T08:13:16.108575+0000","last_active":"2024-04-24T08:13:17.114831+0000","last_peered":"2024-04-24T08:13:17.114831+0000","last_clean":"2024-04-24T08:13:17.114831+0000","last_became_active":"2024-04-24T08:13:16.108195+0000","last_became_peered":"2024-04-24T08:13:16.108195+0000","last_unstale":"2024-04-24T08:13:17.114831+0000","last_undegraded":"2024-04-24T08:13:17.114831+0000","last_fullsized":"2024-04-24T08:13:17.114831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:34.269939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.099531+0000","last_change":"2024-04-24T08:13:15.065237+0000","last_active":"2024-04-24T08:13:17.099531+0000","last_peered":"2024-04-24T08:13:17.099531+0000","last_clean":"2024-04-24T08:13:17.099531+0000","last_became_active":"2024-04-24T08:13:15.064337+0000","last_became_peered":"2024-04-24T08:13:15.064337+0000","last_unstale":"2024-04-24T08:13:17.099531+0000","last_undegraded":"2024-04-24T08:13:17.099531+0000","last_fullsized":"2024-04-24T08:13:17.099531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:36:43.582929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101773+0000","last_change":"2024-04-24T08:13:16.109914+0000","last_active":"2024-04-24T08:13:17.101773+0000","last_peered":"2024-04-24T08:13:17.101773+0000","last_clean":"2024-04-24T08:13:17.101773+0000","last_became_active":"2024-04-24T08:13:16.109605+0000","last_became_peered":"2024-04-24T08:13:16.109605+0000","last_unstale":"2024-04-24T08:13:17.101773+0000","last_undegraded":"2024-04-24T08:13:17.101773+0000","last_fullsized":"2024-04-24T08:13:17.101773+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:20:12.916151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234122+0000","last_change":"2024-04-24T08:13:15.064559+0000","last_active":"2024-04-24T08:13:17.234122+0000","last_peered":"2024-04-24T08:13:17.234122+0000","last_clean":"2024-04-24T08:13:17.234122+0000","last_became_active":"2024-04-24T08:13:15.064253+0000","last_became_peered":"2024-04-24T08:13:15.064253+0000","last_unstale":"2024-04-24T08:13:17.234122+0000","last_undegraded":"2024-04-24T08:13:17.234122+0000","last_fullsized":"2024-04-24T08:13:17.234122+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:51:28.936432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234853+0000","last_change":"2024-04-24T08:13:16.118367+0000","last_active":"2024-04-24T08:13:17.234853+0000","last_peered":"2024-04-24T08:13:17.234853+0000","last_clean":"2024-04-24T08:13:17.234853+0000","last_became_active":"2024-04-24T08:13:16.115497+0000","last_became_peered":"2024-04-24T08:13:16.115497+0000","last_unstale":"2024-04-24T08:13:17.234853+0000","last_undegraded":"2024-04-24T08:13:17.234853+0000","last_fullsized":"2024-04-24T08:13:17.234853+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:55.671217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101837+0000","last_change":"2024-04-24T08:13:15.075219+0000","last_active":"2024-04-24T08:13:17.101837+0000","last_peered":"2024-04-24T08:13:17.101837+0000","last_clean":"2024-04-24T08:13:17.101837+0000","last_became_active":"2024-04-24T08:13:15.074952+0000","last_became_peered":"2024-04-24T08:13:15.074952+0000","last_unstale":"2024-04-24T08:13:17.101837+0000","last_undegraded":"2024-04-24T08:13:17.101837+0000","last_fullsized":"2024-04-24T08:13:17.101837+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:35.089414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.102191+0000","last_change":"2024-04-24T08:13:16.104943+0000","last_active":"2024-04-24T08:13:17.102191+0000","last_peered":"2024-04-24T08:13:17.102191+0000","last_clean":"2024-04-24T08:13:17.102191+0000","last_became_active":"2024-04-24T08:13:16.104530+0000","last_became_peered":"2024-04-24T08:13:16.104530+0000","last_unstale":"2024-04-24T08:13:17.102191+0000","last_undegraded":"2024-04-24T08:13:17.102191+0000","last_fullsized":"2024-04-24T08:13:17.102191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:41:17.847511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230768+0000","last_change":"2024-04-24T08:13:15.071218+0000","last_active":"2024-04-24T08:13:17.230768+0000","last_peered":"2024-04-24T08:13:17.230768+0000","last_clean":"2024-04-24T08:13:17.230768+0000","last_became_active":"2024-04-24T08:13:15.070675+0000","last_became_peered":"2024-04-24T08:13:15.070675+0000","last_unstale":"2024-04-24T08:13:17.230768+0000","last_undegraded":"2024-04-24T08:13:17.230768+0000","last_fullsized":"2024-04-24T08:13:17.230768+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:12:05.860614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100202+0000","last_change":"2024-04-24T08:13:16.103051+0000","last_active":"2024-04-24T08:13:17.100202+0000","last_peered":"2024-04-24T08:13:17.100202+0000","last_clean":"2024-04-24T08:13:17.100202+0000","last_became_active":"2024-04-24T08:13:16.102717+0000","last_became_peered":"2024-04-24T08:13:16.102717+0000","last_unstale":"2024-04-24T08:13:17.100202+0000","last_undegraded":"2024-04-24T08:13:17.100202+0000","last_fullsized":"2024-04-24T08:13:17.100202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:43:55.498854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231437+0000","last_change":"2024-04-24T08:13:15.066088+0000","last_active":"2024-04-24T08:13:17.231437+0000","last_peered":"2024-04-24T08:13:17.231437+0000","last_clean":"2024-04-24T08:13:17.231437+0000","last_became_active":"2024-04-24T08:13:15.065822+0000","last_became_peered":"2024-04-24T08:13:15.065822+0000","last_unstale":"2024-04-24T08:13:17.231437+0000","last_undegraded":"2024-04-24T08:13:17.231437+0000","last_fullsized":"2024-04-24T08:13:17.231437+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:24.580314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234418+0000","last_change":"2024-04-24T08:13:16.109852+0000","last_active":"2024-04-24T08:13:17.234418+0000","last_peered":"2024-04-24T08:13:17.234418+0000","last_clean":"2024-04-24T08:13:17.234418+0000","last_became_active":"2024-04-24T08:13:16.103554+0000","last_became_peered":"2024-04-24T08:13:16.103554+0000","last_unstale":"2024-04-24T08:13:17.234418+0000","last_undegraded":"2024-04-24T08:13:17.234418+0000","last_fullsized":"2024-04-24T08:13:17.234418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:17:23.703236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231041+0000","last_change":"2024-04-24T08:13:15.071661+0000","last_active":"2024-04-24T08:13:17.231041+0000","last_peered":"2024-04-24T08:13:17.231041+0000","last_clean":"2024-04-24T08:13:17.231041+0000","last_became_active":"2024-04-24T08:13:15.069876+0000","last_became_peered":"2024-04-24T08:13:15.069876+0000","last_unstale":"2024-04-24T08:13:17.231041+0000","last_undegraded":"2024-04-24T08:13:17.231041+0000","last_fullsized":"2024-04-24T08:13:17.231041+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:05.968909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101377+0000","last_change":"2024-04-24T08:13:16.084092+0000","last_active":"2024-04-24T08:13:17.101377+0000","last_peered":"2024-04-24T08:13:17.101377+0000","last_clean":"2024-04-24T08:13:17.101377+0000","last_became_active":"2024-04-24T08:13:16.083580+0000","last_became_peered":"2024-04-24T08:13:16.083580+0000","last_unstale":"2024-04-24T08:13:17.101377+0000","last_undegraded":"2024-04-24T08:13:17.101377+0000","last_fullsized":"2024-04-24T08:13:17.101377+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:41.452749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100825+0000","last_change":"2024-04-24T08:13:15.069831+0000","last_active":"2024-04-24T08:13:17.100825+0000","last_peered":"2024-04-24T08:13:17.100825+0000","last_clean":"2024-04-24T08:13:17.100825+0000","last_became_active":"2024-04-24T08:13:15.068208+0000","last_became_peered":"2024-04-24T08:13:15.068208+0000","last_unstale":"2024-04-24T08:13:17.100825+0000","last_undegraded":"2024-04-24T08:13:17.100825+0000","last_fullsized":"2024-04-24T08:13:17.100825+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:21.305064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.114810+0000","last_change":"2024-04-24T08:13:16.100173+0000","last_active":"2024-04-24T08:13:17.114810+0000","last_peered":"2024-04-24T08:13:17.114810+0000","last_clean":"2024-04-24T08:13:17.114810+0000","last_became_active":"2024-04-24T08:13:16.099655+0000","last_became_peered":"2024-04-24T08:13:16.099655+0000","last_unstale":"2024-04-24T08:13:17.114810+0000","last_undegraded":"2024-04-24T08:13:17.114810+0000","last_fullsized":"2024-04-24T08:13:17.114810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:55.671217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100668+0000","last_change":"2024-04-24T08:13:15.066869+0000","last_active":"2024-04-24T08:13:17.100668+0000","last_peered":"2024-04-24T08:13:17.100668+0000","last_clean":"2024-04-24T08:13:17.100668+0000","last_became_active":"2024-04-24T08:13:15.066249+0000","last_became_peered":"2024-04-24T08:13:15.066249+0000","last_unstale":"2024-04-24T08:13:17.100668+0000","last_undegraded":"2024-04-24T08:13:17.100668+0000","last_fullsized":"2024-04-24T08:13:17.100668+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:09.619604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.099980+0000","last_change":"2024-04-24T08:13:16.101594+0000","last_active":"2024-04-24T08:13:17.099980+0000","last_peered":"2024-04-24T08:13:17.099980+0000","last_clean":"2024-04-24T08:13:17.099980+0000","last_became_active":"2024-04-24T08:13:16.101278+0000","last_became_peered":"2024-04-24T08:13:16.101278+0000","last_unstale":"2024-04-24T08:13:17.099980+0000","last_undegraded":"2024-04-24T08:13:17.099980+0000","last_fullsized":"2024-04-24T08:13:17.099980+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:15:40.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":22,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234173+0000","last_change":"2024-04-24T08:13:15.071500+0000","last_active":"2024-04-24T08:13:17.234173+0000","last_peered":"2024-04-24T08:13:17.234173+0000","last_clean":"2024-04-24T08:13:17.234173+0000","last_became_active":"2024-04-24T08:13:15.070640+0000","last_became_peered":"2024-04-24T08:13:15.070640+0000","last_unstale":"2024-04-24T08:13:17.234173+0000","last_undegraded":"2024-04-24T08:13:17.234173+0000","last_fullsized":"2024-04-24T08:13:17.234173+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:09.619604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234478+0000","last_change":"2024-04-24T08:13:16.103842+0000","last_active":"2024-04-24T08:13:17.234478+0000","last_peered":"2024-04-24T08:13:17.234478+0000","last_clean":"2024-04-24T08:13:17.234478+0000","last_became_active":"2024-04-24T08:13:16.103563+0000","last_became_peered":"2024-04-24T08:13:16.103563+0000","last_unstale":"2024-04-24T08:13:17.234478+0000","last_undegraded":"2024-04-24T08:13:17.234478+0000","last_fullsized":"2024-04-24T08:13:17.234478+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:03.715939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101810+0000","last_change":"2024-04-24T08:13:15.074428+0000","last_active":"2024-04-24T08:13:17.101810+0000","last_peered":"2024-04-24T08:13:17.101810+0000","last_clean":"2024-04-24T08:13:17.101810+0000","last_became_active":"2024-04-24T08:13:15.074092+0000","last_became_peered":"2024-04-24T08:13:15.074092+0000","last_unstale":"2024-04-24T08:13:17.101810+0000","last_undegraded":"2024-04-24T08:13:17.101810+0000","last_fullsized":"2024-04-24T08:13:17.101810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:38.953112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235271+0000","last_change":"2024-04-24T08:13:16.105275+0000","last_active":"2024-04-24T08:13:17.235271+0000","last_peered":"2024-04-24T08:13:17.235271+0000","last_clean":"2024-04-24T08:13:17.235271+0000","last_became_active":"2024-04-24T08:13:16.102860+0000","last_became_peered":"2024-04-24T08:13:16.102860+0000","last_unstale":"2024-04-24T08:13:17.235271+0000","last_undegraded":"2024-04-24T08:13:17.235271+0000","last_fullsized":"2024-04-24T08:13:17.235271+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:26:04.544052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234752+0000","last_change":"2024-04-24T08:13:15.071576+0000","last_active":"2024-04-24T08:13:17.234752+0000","last_peered":"2024-04-24T08:13:17.234752+0000","last_clean":"2024-04-24T08:13:17.234752+0000","last_became_active":"2024-04-24T08:13:15.071279+0000","last_became_peered":"2024-04-24T08:13:15.071279+0000","last_unstale":"2024-04-24T08:13:17.234752+0000","last_undegraded":"2024-04-24T08:13:17.234752+0000","last_fullsized":"2024-04-24T08:13:17.234752+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:38.953112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231404+0000","last_change":"2024-04-24T08:13:16.096510+0000","last_active":"2024-04-24T08:13:17.231404+0000","last_peered":"2024-04-24T08:13:17.231404+0000","last_clean":"2024-04-24T08:13:17.231404+0000","last_became_active":"2024-04-24T08:13:16.093509+0000","last_became_peered":"2024-04-24T08:13:16.093509+0000","last_unstale":"2024-04-24T08:13:17.231404+0000","last_undegraded":"2024-04-24T08:13:17.231404+0000","last_fullsized":"2024-04-24T08:13:17.231404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:39.967454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.102142+0000","last_change":"2024-04-24T08:13:15.077674+0000","last_active":"2024-04-24T08:13:17.102142+0000","last_peered":"2024-04-24T08:13:17.102142+0000","last_clean":"2024-04-24T08:13:17.102142+0000","last_became_active":"2024-04-24T08:13:15.077040+0000","last_became_peered":"2024-04-24T08:13:15.077040+0000","last_unstale":"2024-04-24T08:13:17.102142+0000","last_undegraded":"2024-04-24T08:13:17.102142+0000","last_fullsized":"2024-04-24T08:13:17.102142+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:07:35.163982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100350+0000","last_change":"2024-04-24T08:13:16.103342+0000","last_active":"2024-04-24T08:13:17.100350+0000","last_peered":"2024-04-24T08:13:17.100350+0000","last_clean":"2024-04-24T08:13:17.100350+0000","last_became_active":"2024-04-24T08:13:16.102755+0000","last_became_peered":"2024-04-24T08:13:16.102755+0000","last_unstale":"2024-04-24T08:13:17.100350+0000","last_undegraded":"2024-04-24T08:13:17.100350+0000","last_fullsized":"2024-04-24T08:13:17.100350+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:21:45.557396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235033+0000","last_change":"2024-04-24T08:13:15.074871+0000","last_active":"2024-04-24T08:13:17.235033+0000","last_peered":"2024-04-24T08:13:17.235033+0000","last_clean":"2024-04-24T08:13:17.235033+0000","last_became_active":"2024-04-24T08:13:15.074448+0000","last_became_peered":"2024-04-24T08:13:15.074448+0000","last_unstale":"2024-04-24T08:13:17.235033+0000","last_undegraded":"2024-04-24T08:13:17.235033+0000","last_fullsized":"2024-04-24T08:13:17.235033+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:37:58.198960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230880+0000","last_change":"2024-04-24T08:13:16.101507+0000","last_active":"2024-04-24T08:13:17.230880+0000","last_peered":"2024-04-24T08:13:17.230880+0000","last_clean":"2024-04-24T08:13:17.230880+0000","last_became_active":"2024-04-24T08:13:16.101222+0000","last_became_peered":"2024-04-24T08:13:16.101222+0000","last_unstale":"2024-04-24T08:13:17.230880+0000","last_undegraded":"2024-04-24T08:13:17.230880+0000","last_fullsized":"2024-04-24T08:13:17.230880+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:16.240899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100932+0000","last_change":"2024-04-24T08:13:15.070239+0000","last_active":"2024-04-24T08:13:17.100932+0000","last_peered":"2024-04-24T08:13:17.100932+0000","last_clean":"2024-04-24T08:13:17.100932+0000","last_became_active":"2024-04-24T08:13:15.069396+0000","last_became_peered":"2024-04-24T08:13:15.069396+0000","last_unstale":"2024-04-24T08:13:17.100932+0000","last_undegraded":"2024-04-24T08:13:17.100932+0000","last_fullsized":"2024-04-24T08:13:17.100932+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:51:28.936432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230462+0000","last_change":"2024-04-24T08:13:16.107811+0000","last_active":"2024-04-24T08:13:17.230462+0000","last_peered":"2024-04-24T08:13:17.230462+0000","last_clean":"2024-04-24T08:13:17.230462+0000","last_became_active":"2024-04-24T08:13:16.106526+0000","last_became_peered":"2024-04-24T08:13:16.106526+0000","last_unstale":"2024-04-24T08:13:17.230462+0000","last_undegraded":"2024-04-24T08:13:17.230462+0000","last_fullsized":"2024-04-24T08:13:17.230462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:05:11.404817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100830+0000","last_change":"2024-04-24T08:13:15.064228+0000","last_active":"2024-04-24T08:13:17.100830+0000","last_peered":"2024-04-24T08:13:17.100830+0000","last_clean":"2024-04-24T08:13:17.100830+0000","last_became_active":"2024-04-24T08:13:15.063916+0000","last_became_peered":"2024-04-24T08:13:15.063916+0000","last_unstale":"2024-04-24T08:13:17.100830+0000","last_undegraded":"2024-04-24T08:13:17.100830+0000","last_fullsized":"2024-04-24T08:13:17.100830+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:35.089414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230498+0000","last_change":"2024-04-24T08:13:16.106128+0000","last_active":"2024-04-24T08:13:17.230498+0000","last_peered":"2024-04-24T08:13:17.230498+0000","last_clean":"2024-04-24T08:13:17.230498+0000","last_became_active":"2024-04-24T08:13:16.105874+0000","last_became_peered":"2024-04-24T08:13:16.105874+0000","last_unstale":"2024-04-24T08:13:17.230498+0000","last_undegraded":"2024-04-24T08:13:17.230498+0000","last_fullsized":"2024-04-24T08:13:17.230498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:36:11.669117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.114589+0000","last_change":"2024-04-24T08:13:15.061960+0000","last_active":"2024-04-24T08:13:17.114589+0000","last_peered":"2024-04-24T08:13:17.114589+0000","last_clean":"2024-04-24T08:13:17.114589+0000","last_became_active":"2024-04-24T08:13:15.061680+0000","last_became_peered":"2024-04-24T08:13:15.061680+0000","last_unstale":"2024-04-24T08:13:17.114589+0000","last_undegraded":"2024-04-24T08:13:17.114589+0000","last_fullsized":"2024-04-24T08:13:17.114589+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:36.362268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115623+0000","last_change":"2024-04-24T08:13:16.098352+0000","last_active":"2024-04-24T08:13:17.115623+0000","last_peered":"2024-04-24T08:13:17.115623+0000","last_clean":"2024-04-24T08:13:17.115623+0000","last_became_active":"2024-04-24T08:13:16.098088+0000","last_became_peered":"2024-04-24T08:13:16.098088+0000","last_unstale":"2024-04-24T08:13:17.115623+0000","last_undegraded":"2024-04-24T08:13:17.115623+0000","last_fullsized":"2024-04-24T08:13:17.115623+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:50:24.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101825+0000","last_change":"2024-04-24T08:13:15.070521+0000","last_active":"2024-04-24T08:13:17.101825+0000","last_peered":"2024-04-24T08:13:17.101825+0000","last_clean":"2024-04-24T08:13:17.101825+0000","last_became_active":"2024-04-24T08:13:15.070214+0000","last_became_peered":"2024-04-24T08:13:15.070214+0000","last_unstale":"2024-04-24T08:13:17.101825+0000","last_undegraded":"2024-04-24T08:13:17.101825+0000","last_fullsized":"2024-04-24T08:13:17.101825+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:09.619604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101944+0000","last_change":"2024-04-24T08:13:16.082145+0000","last_active":"2024-04-24T08:13:17.101944+0000","last_peered":"2024-04-24T08:13:17.101944+0000","last_clean":"2024-04-24T08:13:17.101944+0000","last_became_active":"2024-04-24T08:13:16.080812+0000","last_became_peered":"2024-04-24T08:13:16.080812+0000","last_unstale":"2024-04-24T08:13:17.101944+0000","last_undegraded":"2024-04-24T08:13:17.101944+0000","last_fullsized":"2024-04-24T08:13:17.101944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:03.715939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"68'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.113390+0000","last_change":"2024-04-24T08:13:15.066949+0000","last_active":"2024-04-24T08:13:17.113390+0000","last_peered":"2024-04-24T08:13:17.113390+0000","last_clean":"2024-04-24T08:13:17.113390+0000","last_became_active":"2024-04-24T08:13:15.065150+0000","last_became_peered":"2024-04-24T08:13:15.065150+0000","last_unstale":"2024-04-24T08:13:17.113390+0000","last_undegraded":"2024-04-24T08:13:17.113390+0000","last_fullsized":"2024-04-24T08:13:17.113390+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:28.677994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235219+0000","last_change":"2024-04-24T08:13:16.114504+0000","last_active":"2024-04-24T08:13:17.235219+0000","last_peered":"2024-04-24T08:13:17.235219+0000","last_clean":"2024-04-24T08:13:17.235219+0000","last_became_active":"2024-04-24T08:13:16.114043+0000","last_became_peered":"2024-04-24T08:13:16.114043+0000","last_unstale":"2024-04-24T08:13:17.235219+0000","last_undegraded":"2024-04-24T08:13:17.235219+0000","last_fullsized":"2024-04-24T08:13:17.235219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:34.269939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100401+0000","last_change":"2024-04-24T08:13:15.066972+0000","last_active":"2024-04-24T08:13:17.100401+0000","last_peered":"2024-04-24T08:13:17.100401+0000","last_clean":"2024-04-24T08:13:17.100401+0000","last_became_active":"2024-04-24T08:13:15.065993+0000","last_became_peered":"2024-04-24T08:13:15.065993+0000","last_unstale":"2024-04-24T08:13:17.100401+0000","last_undegraded":"2024-04-24T08:13:17.100401+0000","last_fullsized":"2024-04-24T08:13:17.100401+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:12:36.778796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235536+0000","last_change":"2024-04-24T08:13:16.116076+0000","last_active":"2024-04-24T08:13:17.235536+0000","last_peered":"2024-04-24T08:13:17.235536+0000","last_clean":"2024-04-24T08:13:17.235536+0000","last_became_active":"2024-04-24T08:13:16.113710+0000","last_became_peered":"2024-04-24T08:13:16.113710+0000","last_unstale":"2024-04-24T08:13:17.235536+0000","last_undegraded":"2024-04-24T08:13:17.235536+0000","last_fullsized":"2024-04-24T08:13:17.235536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:50:24.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115226+0000","last_change":"2024-04-24T08:13:15.067169+0000","last_active":"2024-04-24T08:13:17.115226+0000","last_peered":"2024-04-24T08:13:17.115226+0000","last_clean":"2024-04-24T08:13:17.115226+0000","last_became_active":"2024-04-24T08:13:15.066889+0000","last_became_peered":"2024-04-24T08:13:15.066889+0000","last_unstale":"2024-04-24T08:13:17.115226+0000","last_undegraded":"2024-04-24T08:13:17.115226+0000","last_fullsized":"2024-04-24T08:13:17.115226+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:09.908983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100406+0000","last_change":"2024-04-24T08:13:16.102378+0000","last_active":"2024-04-24T08:13:17.100406+0000","last_peered":"2024-04-24T08:13:17.100406+0000","last_clean":"2024-04-24T08:13:17.100406+0000","last_became_active":"2024-04-24T08:13:16.102065+0000","last_became_peered":"2024-04-24T08:13:16.102065+0000","last_unstale":"2024-04-24T08:13:17.100406+0000","last_undegraded":"2024-04-24T08:13:17.100406+0000","last_fullsized":"2024-04-24T08:13:17.100406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:12:42.080237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100087+0000","last_change":"2024-04-24T08:13:15.068237+0000","last_active":"2024-04-24T08:13:17.100087+0000","last_peered":"2024-04-24T08:13:17.100087+0000","last_clean":"2024-04-24T08:13:17.100087+0000","last_became_active":"2024-04-24T08:13:15.067449+0000","last_became_peered":"2024-04-24T08:13:15.067449+0000","last_unstale":"2024-04-24T08:13:17.100087+0000","last_undegraded":"2024-04-24T08:13:17.100087+0000","last_fullsized":"2024-04-24T08:13:17.100087+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:39:39.703527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101127+0000","last_change":"2024-04-24T08:13:16.115536+0000","last_active":"2024-04-24T08:13:17.101127+0000","last_peered":"2024-04-24T08:13:17.101127+0000","last_clean":"2024-04-24T08:13:17.101127+0000","last_became_active":"2024-04-24T08:13:16.115352+0000","last_became_peered":"2024-04-24T08:13:16.115352+0000","last_unstale":"2024-04-24T08:13:17.101127+0000","last_undegraded":"2024-04-24T08:13:17.101127+0000","last_fullsized":"2024-04-24T08:13:17.101127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:37.376610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230930+0000","last_change":"2024-04-24T08:13:15.071531+0000","last_active":"2024-04-24T08:13:17.230930+0000","last_peered":"2024-04-24T08:13:17.230930+0000","last_clean":"2024-04-24T08:13:17.230930+0000","last_became_active":"2024-04-24T08:13:15.071222+0000","last_became_peered":"2024-04-24T08:13:15.071222+0000","last_unstale":"2024-04-24T08:13:17.230930+0000","last_undegraded":"2024-04-24T08:13:17.230930+0000","last_fullsized":"2024-04-24T08:13:17.230930+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:03:16.408051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231500+0000","last_change":"2024-04-24T08:13:16.099257+0000","last_active":"2024-04-24T08:13:17.231500+0000","last_peered":"2024-04-24T08:13:17.231500+0000","last_clean":"2024-04-24T08:13:17.231500+0000","last_became_active":"2024-04-24T08:13:16.098770+0000","last_became_peered":"2024-04-24T08:13:16.098770+0000","last_unstale":"2024-04-24T08:13:17.231500+0000","last_undegraded":"2024-04-24T08:13:17.231500+0000","last_fullsized":"2024-04-24T08:13:17.231500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:10.633947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.232556+0000","last_change":"2024-04-24T08:13:15.073008+0000","last_active":"2024-04-24T08:13:17.232556+0000","last_peered":"2024-04-24T08:13:17.232556+0000","last_clean":"2024-04-24T08:13:17.232556+0000","last_became_active":"2024-04-24T08:13:15.071170+0000","last_became_peered":"2024-04-24T08:13:15.071170+0000","last_unstale":"2024-04-24T08:13:17.232556+0000","last_undegraded":"2024-04-24T08:13:17.232556+0000","last_fullsized":"2024-04-24T08:13:17.232556+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:28.677994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100489+0000","last_change":"2024-04-24T08:13:16.099311+0000","last_active":"2024-04-24T08:13:17.100489+0000","last_peered":"2024-04-24T08:13:17.100489+0000","last_clean":"2024-04-24T08:13:17.100489+0000","last_became_active":"2024-04-24T08:13:16.098961+0000","last_became_peered":"2024-04-24T08:13:16.098961+0000","last_unstale":"2024-04-24T08:13:17.100489+0000","last_undegraded":"2024-04-24T08:13:17.100489+0000","last_fullsized":"2024-04-24T08:13:17.100489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:49:43.085760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100521+0000","last_change":"2024-04-24T08:13:15.069521+0000","last_active":"2024-04-24T08:13:17.100521+0000","last_peered":"2024-04-24T08:13:17.100521+0000","last_clean":"2024-04-24T08:13:17.100521+0000","last_became_active":"2024-04-24T08:13:15.068894+0000","last_became_peered":"2024-04-24T08:13:15.068894+0000","last_unstale":"2024-04-24T08:13:17.100521+0000","last_undegraded":"2024-04-24T08:13:17.100521+0000","last_fullsized":"2024-04-24T08:13:17.100521+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:24.580314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230527+0000","last_change":"2024-04-24T08:13:16.111468+0000","last_active":"2024-04-24T08:13:17.230527+0000","last_peered":"2024-04-24T08:13:17.230527+0000","last_clean":"2024-04-24T08:13:17.230527+0000","last_became_active":"2024-04-24T08:13:16.111194+0000","last_became_peered":"2024-04-24T08:13:16.111194+0000","last_unstale":"2024-04-24T08:13:17.230527+0000","last_undegraded":"2024-04-24T08:13:17.230527+0000","last_fullsized":"2024-04-24T08:13:17.230527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:03:00.261397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101299+0000","last_change":"2024-04-24T08:13:15.075499+0000","last_active":"2024-04-24T08:13:17.101299+0000","last_peered":"2024-04-24T08:13:17.101299+0000","last_clean":"2024-04-24T08:13:17.101299+0000","last_became_active":"2024-04-24T08:13:15.072540+0000","last_became_peered":"2024-04-24T08:13:15.072540+0000","last_unstale":"2024-04-24T08:13:17.101299+0000","last_undegraded":"2024-04-24T08:13:17.101299+0000","last_fullsized":"2024-04-24T08:13:17.101299+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:57:16.187947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":256,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233501+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:13:17.233501+0000","last_peered":"2024-04-24T08:13:17.233501+0000","last_clean":"2024-04-24T08:13:17.233501+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:13:17.233501+0000","last_undegraded":"2024-04-24T08:13:17.233501+0000","last_fullsized":"2024-04-24T08:13:17.233501+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1429504,"data_stored":939496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":835584,"data_stored":348604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":831488,"data_stored":348602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":831488,"data_stored":348582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":835584,"data_stored":348624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":839680,"data_stored":350124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":847872,"data_stored":350740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":938880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:13:20.502 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:13:20.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.741+0000 7f525b200640 1 Processor -- start 2024-04-24T08:13:20.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.741+0000 7f525b200640 1 -- start start 2024-04-24T08:13:20.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.743+0000 7f525b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0d0200 0x7f525c0d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.743+0000 7f525b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f525c15ed40 con 0x7f525c06cfb0 2024-04-24T08:13:20.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.743+0000 7f525b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f525c0cf050 con 0x7f525c15e960 2024-04-24T08:13:20.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.743+0000 7f525b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f525c162b50 con 0x7f525c0d0200 2024-04-24T08:13:20.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.743+0000 7f5253e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f525c06cfb0 0x7f525c0cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39022/0 (socket says 172.21.15.69:39022) 2024-04-24T08:13:20.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.743+0000 7f5253e00640 1 -- 172.21.15.69:0/692796280 learned_addr learned my addr 172.21.15.69:0/692796280 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:20.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.743+0000 7f525a800640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0d0200 0x7f525c0d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.743+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775445511 0 0) 0x7f525c15ed40 con 0x7f525c06cfb0 2024-04-24T08:13:20.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.743+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5240003520 con 0x7f525c06cfb0 2024-04-24T08:13:20.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2387610670 0 0) 0x7f5240003520 con 0x7f525c06cfb0 2024-04-24T08:13:20.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0d0200 msgr2=0x7f525c0d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:20.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0d0200 0x7f525c0d2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 >> v1:172.21.15.121:6789/0 conn(0x7f525c15e960 legacy=0x7f525c162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c162e20 con 0x7f525c06cfb0 2024-04-24T08:13:20.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f5244001670 con 0x7f525c06cfb0 2024-04-24T08:13:20.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 >> v1:172.21.15.69:6789/0 conn(0x7f525c06cfb0 legacy=0x7f525c0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0d0200 0x7f52400039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 0x7f5240004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5240004d40 0x7f52400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f525c162b50 con 0x7f5240003f60 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f524000baa0 con 0x7f5240004d40 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f524000bca0 con 0x7f525c0d0200 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.744+0000 7f5253e00640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 0x7f5240004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.745+0000 7f525a800640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5240004d40 0x7f52400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.745+0000 7f5253e00640 1 -- 172.21.15.69:0/692796280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0d0200 msgr2=0x7f52400039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.745+0000 7f5253e00640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0d0200 0x7f52400039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.745+0000 7f5253e00640 1 -- 172.21.15.69:0/692796280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5240004d40 msgr2=0x7f52400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.745+0000 7f5253e00640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5240004d40 0x7f52400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.745+0000 7f5253e00640 1 -- 172.21.15.69:0/692796280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5240003520 con 0x7f5240003f60 2024-04-24T08:13:20.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.746+0000 7f5253e00640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 0x7f5240004800 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f524400c240 tx=0x7f5240005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f5f11953782676 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.746+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5244007d10 con 0x7f5240003f60 2024-04-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.746+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5244005ad0 con 0x7f5240003f60 2024-04-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.746+0000 7f5259400640 1 -- 172.21.15.69:0/692796280 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5244007070 con 0x7f5240003f60 2024-04-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.747+0000 7f525b200640 1 -- 172.21.15.69:0/692796280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 msgr2=0x7f5240004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.747+0000 7f525b200640 1 --2- 172.21.15.69:0/692796280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 0x7f5240004800 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f524400c240 tx=0x7f5240005830 comp rx=0 tx=0).stop 2024-04-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.747+0000 7f525a800640 1 -- 172.21.15.69:0/692796280 reap_dead start 2024-04-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.747+0000 7f525b200640 1 -- 172.21.15.69:0/692796280 shutdown_connections 2024-04-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.747+0000 7f525b200640 1 -- 172.21.15.69:0/692796280 >> 172.21.15.69:0/692796280 conn(0x7f525c09d900 msgr2=0x7f525c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.747+0000 7f525b200640 1 -- 172.21.15.69:0/692796280 shutdown_connections 2024-04-24T08:13:20.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.747+0000 7f525b200640 1 -- 172.21.15.69:0/692796280 wait complete. 2024-04-24T08:13:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.749+0000 7f525b200640 1 Processor -- start 2024-04-24T08:13:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.749+0000 7f525b200640 1 -- start start 2024-04-24T08:13:20.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f525b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 0x7f525c0b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f525b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0cb610 0x7f525c160f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f525b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f525c161490 0x7f525c155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f525b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f525c0cf5d0 con 0x7f5240003f60 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f525b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f525c0cf2f0 con 0x7f525c161490 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f525b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f525c0b6170 con 0x7f525c0cb610 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f5259e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 0x7f525c0b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f5259e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 0x7f525c0b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41540/0 (socket says 172.21.15.69:41540) 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f5259e00640 1 -- 172.21.15.69:0/3980811514 learned_addr learned my addr 172.21.15.69:0/3980811514 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f5259e00640 1 -- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0cb610 msgr2=0x7f525c160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f5259e00640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0cb610 0x7f525c160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f5259e00640 1 -- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f525c161490 msgr2=0x7f525c155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f525a800640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f525c161490 0x7f525c155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f5259e00640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f525c161490 0x7f525c155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.750+0000 7f5259e00640 1 -- 172.21.15.69:0/3980811514 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c161da0 con 0x7f5240003f60 2024-04-24T08:13:20.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.751+0000 7f5259e00640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 0x7f525c0b28a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f524c00c4b0 tx=0x7f524c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:20.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.751+0000 7f5252000640 1 -- 172.21.15.69:0/3980811514 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c0052d0 con 0x7f5240003f60 2024-04-24T08:13:20.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.751+0000 7f5252000640 1 -- 172.21.15.69:0/3980811514 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f524c005490 con 0x7f5240003f60 2024-04-24T08:13:20.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.751+0000 7f5252000640 1 -- 172.21.15.69:0/3980811514 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f524c012870 con 0x7f5240003f60 2024-04-24T08:13:20.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.751+0000 7f525b200640 1 -- 172.21.15.69:0/3980811514 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525c161ac0 con 0x7f5240003f60 2024-04-24T08:13:20.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.751+0000 7f525b200640 1 -- 172.21.15.69:0/3980811514 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f525c1e84e0 con 0x7f5240003f60 2024-04-24T08:13:20.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.752+0000 7f525b200640 1 -- 172.21.15.69:0/3980811514 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f525c075d50 con 0x7f5240003f60 2024-04-24T08:13:20.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.753+0000 7f5252000640 1 -- 172.21.15.69:0/3980811514 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f524c012a10 con 0x7f5240003f60 2024-04-24T08:13:20.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.754+0000 7f5252000640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f523806c8f0 0x7f523806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:20.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.754+0000 7f5252000640 1 -- 172.21.15.69:0/3980811514 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f524c08b890 con 0x7f5240003f60 2024-04-24T08:13:20.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.755+0000 7f5253e00640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f523806c8f0 0x7f523806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:20.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.756+0000 7f5253e00640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f523806c8f0 0x7f523806edb0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5244033000 tx=0x7f5244009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.759+0000 7f5252000640 1 -- 172.21.15.69:0/3980811514 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f524c059ce0 con 0x7f5240003f60 2024-04-24T08:13:20.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.942+0000 7f525b200640 1 -- 172.21.15.69:0/3980811514 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f525c1eb070 con 0x7f523806c8f0 2024-04-24T08:13:20.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.961+0000 7f5252000640 1 -- 172.21.15.69:0/3980811514 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252069 (secure 0 0 0) 0x7f525c1eb070 con 0x7f523806c8f0 2024-04-24T08:13:20.965 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:20.966 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:13:20.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 -- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f523806c8f0 msgr2=0x7f523806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f523806c8f0 0x7f523806edb0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5244033000 tx=0x7f5244009030 comp rx=0 tx=0).stop 2024-04-24T08:13:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 -- 172.21.15.69:0/3980811514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 msgr2=0x7f525c0b28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 0x7f525c0b28a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f524c00c4b0 tx=0x7f524c00c970 comp rx=0 tx=0).stop 2024-04-24T08:13:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 -- 172.21.15.69:0/3980811514 shutdown_connections 2024-04-24T08:13:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f523806c8f0 0x7f523806edb0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f525c161490 0x7f525c155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f525c0cb610 0x7f525c160f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 --2- 172.21.15.69:0/3980811514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5240003f60 0x7f525c0b28a0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 -- 172.21.15.69:0/3980811514 >> 172.21.15.69:0/3980811514 conn(0x7f525c09d900 msgr2=0x7f525c06d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 -- 172.21.15.69:0/3980811514 shutdown_connections 2024-04-24T08:13:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:20.966+0000 7f525b200640 1 -- 172.21.15.69:0/3980811514 wait complete. 2024-04-24T08:13:20.983 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-04-24T08:13:20.772532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":180,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252992,"kb_used_data":7688,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754721728,"statfs":{"total":773094113280,"available":772835049472,"internally_reserved":0,"allocated":7872512,"data_stored":3973652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002151"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234739+0000","last_change":"2024-04-24T08:13:16.117848+0000","last_active":"2024-04-24T08:13:17.234739+0000","last_peered":"2024-04-24T08:13:17.234739+0000","last_clean":"2024-04-24T08:13:17.234739+0000","last_became_active":"2024-04-24T08:13:16.116700+0000","last_became_peered":"2024-04-24T08:13:16.116700+0000","last_unstale":"2024-04-24T08:13:17.234739+0000","last_undegraded":"2024-04-24T08:13:17.234739+0000","last_fullsized":"2024-04-24T08:13:17.234739+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:10:15.310592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101681+0000","last_change":"2024-04-24T08:13:16.093849+0000","last_active":"2024-04-24T08:13:17.101681+0000","last_peered":"2024-04-24T08:13:17.101681+0000","last_clean":"2024-04-24T08:13:17.101681+0000","last_became_active":"2024-04-24T08:13:16.093479+0000","last_became_peered":"2024-04-24T08:13:16.093479+0000","last_unstale":"2024-04-24T08:13:17.101681+0000","last_undegraded":"2024-04-24T08:13:17.101681+0000","last_fullsized":"2024-04-24T08:13:17.101681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:10.923326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230645+0000","last_change":"2024-04-24T08:13:16.098526+0000","last_active":"2024-04-24T08:13:17.230645+0000","last_peered":"2024-04-24T08:13:17.230645+0000","last_clean":"2024-04-24T08:13:17.230645+0000","last_became_active":"2024-04-24T08:13:16.098166+0000","last_became_peered":"2024-04-24T08:13:16.098166+0000","last_unstale":"2024-04-24T08:13:17.230645+0000","last_undegraded":"2024-04-24T08:13:17.230645+0000","last_fullsized":"2024-04-24T08:13:17.230645+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:21:00.059160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231248+0000","last_change":"2024-04-24T08:13:16.105602+0000","last_active":"2024-04-24T08:13:17.231248+0000","last_peered":"2024-04-24T08:13:17.231248+0000","last_clean":"2024-04-24T08:13:17.231248+0000","last_became_active":"2024-04-24T08:13:16.105103+0000","last_became_peered":"2024-04-24T08:13:16.105103+0000","last_unstale":"2024-04-24T08:13:17.231248+0000","last_undegraded":"2024-04-24T08:13:17.231248+0000","last_fullsized":"2024-04-24T08:13:17.231248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:57:17.202289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.099928+0000","last_change":"2024-04-24T08:13:16.112786+0000","last_active":"2024-04-24T08:13:17.099928+0000","last_peered":"2024-04-24T08:13:17.099928+0000","last_clean":"2024-04-24T08:13:17.099928+0000","last_became_active":"2024-04-24T08:13:16.112508+0000","last_became_peered":"2024-04-24T08:13:16.112508+0000","last_unstale":"2024-04-24T08:13:17.099928+0000","last_undegraded":"2024-04-24T08:13:17.099928+0000","last_fullsized":"2024-04-24T08:13:17.099928+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:34:09.318804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100546+0000","last_change":"2024-04-24T08:13:16.095900+0000","last_active":"2024-04-24T08:13:17.100546+0000","last_peered":"2024-04-24T08:13:17.100546+0000","last_clean":"2024-04-24T08:13:17.100546+0000","last_became_active":"2024-04-24T08:13:16.095589+0000","last_became_peered":"2024-04-24T08:13:16.095589+0000","last_unstale":"2024-04-24T08:13:17.100546+0000","last_undegraded":"2024-04-24T08:13:17.100546+0000","last_fullsized":"2024-04-24T08:13:17.100546+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:03:31.209692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100470+0000","last_change":"2024-04-24T08:13:16.087673+0000","last_active":"2024-04-24T08:13:17.100470+0000","last_peered":"2024-04-24T08:13:17.100470+0000","last_clean":"2024-04-24T08:13:17.100470+0000","last_became_active":"2024-04-24T08:13:16.087370+0000","last_became_peered":"2024-04-24T08:13:16.087370+0000","last_unstale":"2024-04-24T08:13:17.100470+0000","last_undegraded":"2024-04-24T08:13:17.100470+0000","last_fullsized":"2024-04-24T08:13:17.100470+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:37.376610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100396+0000","last_change":"2024-04-24T08:13:16.099155+0000","last_active":"2024-04-24T08:13:17.100396+0000","last_peered":"2024-04-24T08:13:17.100396+0000","last_clean":"2024-04-24T08:13:17.100396+0000","last_became_active":"2024-04-24T08:13:16.098838+0000","last_became_peered":"2024-04-24T08:13:16.098838+0000","last_unstale":"2024-04-24T08:13:17.100396+0000","last_undegraded":"2024-04-24T08:13:17.100396+0000","last_fullsized":"2024-04-24T08:13:17.100396+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:42:42.296218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100633+0000","last_change":"2024-04-24T08:13:16.103940+0000","last_active":"2024-04-24T08:13:17.100633+0000","last_peered":"2024-04-24T08:13:17.100633+0000","last_clean":"2024-04-24T08:13:17.100633+0000","last_became_active":"2024-04-24T08:13:16.103628+0000","last_became_peered":"2024-04-24T08:13:16.103628+0000","last_unstale":"2024-04-24T08:13:17.100633+0000","last_undegraded":"2024-04-24T08:13:17.100633+0000","last_fullsized":"2024-04-24T08:13:17.100633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:54:00.896362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234085+0000","last_change":"2024-04-24T08:13:16.096499+0000","last_active":"2024-04-24T08:13:17.234085+0000","last_peered":"2024-04-24T08:13:17.234085+0000","last_clean":"2024-04-24T08:13:17.234085+0000","last_became_active":"2024-04-24T08:13:16.096189+0000","last_became_peered":"2024-04-24T08:13:16.096189+0000","last_unstale":"2024-04-24T08:13:17.234085+0000","last_undegraded":"2024-04-24T08:13:17.234085+0000","last_fullsized":"2024-04-24T08:13:17.234085+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:20:12.916151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235273+0000","last_change":"2024-04-24T08:13:16.113958+0000","last_active":"2024-04-24T08:13:17.235273+0000","last_peered":"2024-04-24T08:13:17.235273+0000","last_clean":"2024-04-24T08:13:17.235273+0000","last_became_active":"2024-04-24T08:13:16.113372+0000","last_became_peered":"2024-04-24T08:13:16.113372+0000","last_unstale":"2024-04-24T08:13:17.235273+0000","last_undegraded":"2024-04-24T08:13:17.235273+0000","last_fullsized":"2024-04-24T08:13:17.235273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:25.287257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231324+0000","last_change":"2024-04-24T08:13:16.095382+0000","last_active":"2024-04-24T08:13:17.231324+0000","last_peered":"2024-04-24T08:13:17.231324+0000","last_clean":"2024-04-24T08:13:17.231324+0000","last_became_active":"2024-04-24T08:13:16.093955+0000","last_became_peered":"2024-04-24T08:13:16.093955+0000","last_unstale":"2024-04-24T08:13:17.231324+0000","last_undegraded":"2024-04-24T08:13:17.231324+0000","last_fullsized":"2024-04-24T08:13:17.231324+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:22.319406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115254+0000","last_change":"2024-04-24T08:13:16.099815+0000","last_active":"2024-04-24T08:13:17.115254+0000","last_peered":"2024-04-24T08:13:17.115254+0000","last_clean":"2024-04-24T08:13:17.115254+0000","last_became_active":"2024-04-24T08:13:16.099418+0000","last_became_peered":"2024-04-24T08:13:16.099418+0000","last_unstale":"2024-04-24T08:13:17.115254+0000","last_undegraded":"2024-04-24T08:13:17.115254+0000","last_fullsized":"2024-04-24T08:13:17.115254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:25.287257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.099950+0000","last_change":"2024-04-24T08:13:16.096718+0000","last_active":"2024-04-24T08:13:17.099950+0000","last_peered":"2024-04-24T08:13:17.099950+0000","last_clean":"2024-04-24T08:13:17.099950+0000","last_became_active":"2024-04-24T08:13:16.096433+0000","last_became_peered":"2024-04-24T08:13:16.096433+0000","last_unstale":"2024-04-24T08:13:17.099950+0000","last_undegraded":"2024-04-24T08:13:17.099950+0000","last_fullsized":"2024-04-24T08:13:17.099950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:18:54.741567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115053+0000","last_change":"2024-04-24T08:13:16.089445+0000","last_active":"2024-04-24T08:13:17.115053+0000","last_peered":"2024-04-24T08:13:17.115053+0000","last_clean":"2024-04-24T08:13:17.115053+0000","last_became_active":"2024-04-24T08:13:16.089177+0000","last_became_peered":"2024-04-24T08:13:16.089177+0000","last_unstale":"2024-04-24T08:13:17.115053+0000","last_undegraded":"2024-04-24T08:13:17.115053+0000","last_fullsized":"2024-04-24T08:13:17.115053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:26:04.544052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233852+0000","last_change":"2024-04-24T08:13:16.105962+0000","last_active":"2024-04-24T08:13:17.233852+0000","last_peered":"2024-04-24T08:13:17.233852+0000","last_clean":"2024-04-24T08:13:17.233852+0000","last_became_active":"2024-04-24T08:13:16.105690+0000","last_became_peered":"2024-04-24T08:13:16.105690+0000","last_unstale":"2024-04-24T08:13:17.233852+0000","last_undegraded":"2024-04-24T08:13:17.233852+0000","last_fullsized":"2024-04-24T08:13:17.233852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:41:17.847511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100767+0000","last_change":"2024-04-24T08:13:16.102963+0000","last_active":"2024-04-24T08:13:17.100767+0000","last_peered":"2024-04-24T08:13:17.100767+0000","last_clean":"2024-04-24T08:13:17.100767+0000","last_became_active":"2024-04-24T08:13:16.102073+0000","last_became_peered":"2024-04-24T08:13:16.102073+0000","last_unstale":"2024-04-24T08:13:17.100767+0000","last_undegraded":"2024-04-24T08:13:17.100767+0000","last_fullsized":"2024-04-24T08:13:17.100767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:57.278440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231359+0000","last_change":"2024-04-24T08:13:16.099981+0000","last_active":"2024-04-24T08:13:17.231359+0000","last_peered":"2024-04-24T08:13:17.231359+0000","last_clean":"2024-04-24T08:13:17.231359+0000","last_became_active":"2024-04-24T08:13:16.099701+0000","last_became_peered":"2024-04-24T08:13:16.099701+0000","last_unstale":"2024-04-24T08:13:17.231359+0000","last_undegraded":"2024-04-24T08:13:17.231359+0000","last_fullsized":"2024-04-24T08:13:17.231359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:51:29.950774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101982+0000","last_change":"2024-04-24T08:13:16.101112+0000","last_active":"2024-04-24T08:13:17.101982+0000","last_peered":"2024-04-24T08:13:17.101982+0000","last_clean":"2024-04-24T08:13:17.101982+0000","last_became_active":"2024-04-24T08:13:16.100574+0000","last_became_peered":"2024-04-24T08:13:16.100574+0000","last_unstale":"2024-04-24T08:13:17.101982+0000","last_undegraded":"2024-04-24T08:13:17.101982+0000","last_fullsized":"2024-04-24T08:13:17.101982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:20:45.863887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115322+0000","last_change":"2024-04-24T08:13:16.092810+0000","last_active":"2024-04-24T08:13:17.115322+0000","last_peered":"2024-04-24T08:13:17.115322+0000","last_clean":"2024-04-24T08:13:17.115322+0000","last_became_active":"2024-04-24T08:13:16.092423+0000","last_became_peered":"2024-04-24T08:13:16.092423+0000","last_unstale":"2024-04-24T08:13:17.115322+0000","last_undegraded":"2024-04-24T08:13:17.115322+0000","last_fullsized":"2024-04-24T08:13:17.115322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:12:06.874957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233248+0000","last_change":"2024-04-24T08:13:16.093298+0000","last_active":"2024-04-24T08:13:17.233248+0000","last_peered":"2024-04-24T08:13:17.233248+0000","last_clean":"2024-04-24T08:13:17.233248+0000","last_became_active":"2024-04-24T08:13:16.092990+0000","last_became_peered":"2024-04-24T08:13:16.092990+0000","last_unstale":"2024-04-24T08:13:17.233248+0000","last_undegraded":"2024-04-24T08:13:17.233248+0000","last_fullsized":"2024-04-24T08:13:17.233248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:37.376610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233452+0000","last_change":"2024-04-24T08:13:16.102034+0000","last_active":"2024-04-24T08:13:17.233452+0000","last_peered":"2024-04-24T08:13:17.233452+0000","last_clean":"2024-04-24T08:13:17.233452+0000","last_became_active":"2024-04-24T08:13:16.101741+0000","last_became_peered":"2024-04-24T08:13:16.101741+0000","last_unstale":"2024-04-24T08:13:17.233452+0000","last_undegraded":"2024-04-24T08:13:17.233452+0000","last_fullsized":"2024-04-24T08:13:17.233452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:20:45.863887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233146+0000","last_change":"2024-04-24T08:13:16.111440+0000","last_active":"2024-04-24T08:13:17.233146+0000","last_peered":"2024-04-24T08:13:17.233146+0000","last_clean":"2024-04-24T08:13:17.233146+0000","last_became_active":"2024-04-24T08:13:16.110858+0000","last_became_peered":"2024-04-24T08:13:16.110858+0000","last_unstale":"2024-04-24T08:13:17.233146+0000","last_undegraded":"2024-04-24T08:13:17.233146+0000","last_fullsized":"2024-04-24T08:13:17.233146+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:41.452749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233360+0000","last_change":"2024-04-24T08:13:16.107244+0000","last_active":"2024-04-24T08:13:17.233360+0000","last_peered":"2024-04-24T08:13:17.233360+0000","last_clean":"2024-04-24T08:13:17.233360+0000","last_became_active":"2024-04-24T08:13:16.105966+0000","last_became_peered":"2024-04-24T08:13:16.105966+0000","last_unstale":"2024-04-24T08:13:17.233360+0000","last_undegraded":"2024-04-24T08:13:17.233360+0000","last_fullsized":"2024-04-24T08:13:17.233360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:20:12.916151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233873+0000","last_change":"2024-04-24T08:13:16.107076+0000","last_active":"2024-04-24T08:13:17.233873+0000","last_peered":"2024-04-24T08:13:17.233873+0000","last_clean":"2024-04-24T08:13:17.233873+0000","last_became_active":"2024-04-24T08:13:16.106751+0000","last_became_peered":"2024-04-24T08:13:16.106751+0000","last_unstale":"2024-04-24T08:13:17.233873+0000","last_undegraded":"2024-04-24T08:13:17.233873+0000","last_fullsized":"2024-04-24T08:13:17.233873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:41.452749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.102687+0000","last_change":"2024-04-24T08:13:16.101998+0000","last_active":"2024-04-24T08:13:17.102687+0000","last_peered":"2024-04-24T08:13:17.102687+0000","last_clean":"2024-04-24T08:13:17.102687+0000","last_became_active":"2024-04-24T08:13:16.101546+0000","last_became_peered":"2024-04-24T08:13:16.101546+0000","last_unstale":"2024-04-24T08:13:17.102687+0000","last_undegraded":"2024-04-24T08:13:17.102687+0000","last_fullsized":"2024-04-24T08:13:17.102687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:17:23.703236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230757+0000","last_change":"2024-04-24T08:13:16.111564+0000","last_active":"2024-04-24T08:13:17.230757+0000","last_peered":"2024-04-24T08:13:17.230757+0000","last_clean":"2024-04-24T08:13:17.230757+0000","last_became_active":"2024-04-24T08:13:16.111154+0000","last_became_peered":"2024-04-24T08:13:16.111154+0000","last_unstale":"2024-04-24T08:13:17.230757+0000","last_undegraded":"2024-04-24T08:13:17.230757+0000","last_fullsized":"2024-04-24T08:13:17.230757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:39:28.319204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233301+0000","last_change":"2024-04-24T08:13:16.101359+0000","last_active":"2024-04-24T08:13:17.233301+0000","last_peered":"2024-04-24T08:13:17.233301+0000","last_clean":"2024-04-24T08:13:17.233301+0000","last_became_active":"2024-04-24T08:13:16.096175+0000","last_became_peered":"2024-04-24T08:13:16.096175+0000","last_unstale":"2024-04-24T08:13:17.233301+0000","last_undegraded":"2024-04-24T08:13:17.233301+0000","last_fullsized":"2024-04-24T08:13:17.233301+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:10.923326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230671+0000","last_change":"2024-04-24T08:13:16.098975+0000","last_active":"2024-04-24T08:13:17.230671+0000","last_peered":"2024-04-24T08:13:17.230671+0000","last_clean":"2024-04-24T08:13:17.230671+0000","last_became_active":"2024-04-24T08:13:16.097915+0000","last_became_peered":"2024-04-24T08:13:16.097915+0000","last_unstale":"2024-04-24T08:13:17.230671+0000","last_undegraded":"2024-04-24T08:13:17.230671+0000","last_fullsized":"2024-04-24T08:13:17.230671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:05:38.937402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101363+0000","last_change":"2024-04-24T08:13:16.112216+0000","last_active":"2024-04-24T08:13:17.101363+0000","last_peered":"2024-04-24T08:13:17.101363+0000","last_clean":"2024-04-24T08:13:17.101363+0000","last_became_active":"2024-04-24T08:13:16.111872+0000","last_became_peered":"2024-04-24T08:13:16.111872+0000","last_unstale":"2024-04-24T08:13:17.101363+0000","last_undegraded":"2024-04-24T08:13:17.101363+0000","last_fullsized":"2024-04-24T08:13:17.101363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:41:17.847511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100757+0000","last_change":"2024-04-24T08:13:16.094867+0000","last_active":"2024-04-24T08:13:17.100757+0000","last_peered":"2024-04-24T08:13:17.100757+0000","last_clean":"2024-04-24T08:13:17.100757+0000","last_became_active":"2024-04-24T08:13:16.090985+0000","last_became_peered":"2024-04-24T08:13:16.090985+0000","last_unstale":"2024-04-24T08:13:17.100757+0000","last_undegraded":"2024-04-24T08:13:17.100757+0000","last_fullsized":"2024-04-24T08:13:17.100757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:10.923326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100732+0000","last_change":"2024-04-24T08:13:16.099179+0000","last_active":"2024-04-24T08:13:17.100732+0000","last_peered":"2024-04-24T08:13:17.100732+0000","last_clean":"2024-04-24T08:13:17.100732+0000","last_became_active":"2024-04-24T08:13:16.098897+0000","last_became_peered":"2024-04-24T08:13:16.098897+0000","last_unstale":"2024-04-24T08:13:17.100732+0000","last_undegraded":"2024-04-24T08:13:17.100732+0000","last_fullsized":"2024-04-24T08:13:17.100732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:18:20.774821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234246+0000","last_change":"2024-04-24T08:13:16.107049+0000","last_active":"2024-04-24T08:13:17.234246+0000","last_peered":"2024-04-24T08:13:17.234246+0000","last_clean":"2024-04-24T08:13:17.234246+0000","last_became_active":"2024-04-24T08:13:16.106755+0000","last_became_peered":"2024-04-24T08:13:16.106755+0000","last_unstale":"2024-04-24T08:13:17.234246+0000","last_undegraded":"2024-04-24T08:13:17.234246+0000","last_fullsized":"2024-04-24T08:13:17.234246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:03.715939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"67'4","reported_seq":18,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235164+0000","last_change":"2024-04-24T08:13:15.075208+0000","last_active":"2024-04-24T08:13:17.235164+0000","last_peered":"2024-04-24T08:13:17.235164+0000","last_clean":"2024-04-24T08:13:17.235164+0000","last_became_active":"2024-04-24T08:13:15.074904+0000","last_became_peered":"2024-04-24T08:13:15.074904+0000","last_unstale":"2024-04-24T08:13:17.235164+0000","last_undegraded":"2024-04-24T08:13:17.235164+0000","last_fullsized":"2024-04-24T08:13:17.235164+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:28.677994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100972+0000","last_change":"2024-04-24T08:13:16.102681+0000","last_active":"2024-04-24T08:13:17.100972+0000","last_peered":"2024-04-24T08:13:17.100972+0000","last_clean":"2024-04-24T08:13:17.100972+0000","last_became_active":"2024-04-24T08:13:16.101584+0000","last_became_peered":"2024-04-24T08:13:16.101584+0000","last_unstale":"2024-04-24T08:13:17.100972+0000","last_undegraded":"2024-04-24T08:13:17.100972+0000","last_fullsized":"2024-04-24T08:13:17.100972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:03:17.422393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115478+0000","last_change":"2024-04-24T08:13:15.065334+0000","last_active":"2024-04-24T08:13:17.115478+0000","last_peered":"2024-04-24T08:13:17.115478+0000","last_clean":"2024-04-24T08:13:17.115478+0000","last_became_active":"2024-04-24T08:13:15.064990+0000","last_became_peered":"2024-04-24T08:13:15.064990+0000","last_unstale":"2024-04-24T08:13:17.115478+0000","last_undegraded":"2024-04-24T08:13:17.115478+0000","last_fullsized":"2024-04-24T08:13:17.115478+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:20:44.849545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.102512+0000","last_change":"2024-04-24T08:13:16.110937+0000","last_active":"2024-04-24T08:13:17.102512+0000","last_peered":"2024-04-24T08:13:17.102512+0000","last_clean":"2024-04-24T08:13:17.102512+0000","last_became_active":"2024-04-24T08:13:16.109194+0000","last_became_peered":"2024-04-24T08:13:16.109194+0000","last_unstale":"2024-04-24T08:13:17.102512+0000","last_undegraded":"2024-04-24T08:13:17.102512+0000","last_fullsized":"2024-04-24T08:13:17.102512+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:41.452749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233449+0000","last_change":"2024-04-24T08:13:15.075504+0000","last_active":"2024-04-24T08:13:17.233449+0000","last_peered":"2024-04-24T08:13:17.233449+0000","last_clean":"2024-04-24T08:13:17.233449+0000","last_became_active":"2024-04-24T08:13:15.075216+0000","last_became_peered":"2024-04-24T08:13:15.075216+0000","last_unstale":"2024-04-24T08:13:17.233449+0000","last_undegraded":"2024-04-24T08:13:17.233449+0000","last_fullsized":"2024-04-24T08:13:17.233449+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:35.089414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235388+0000","last_change":"2024-04-24T08:13:16.114272+0000","last_active":"2024-04-24T08:13:17.235388+0000","last_peered":"2024-04-24T08:13:17.235388+0000","last_clean":"2024-04-24T08:13:17.235388+0000","last_became_active":"2024-04-24T08:13:16.113383+0000","last_became_peered":"2024-04-24T08:13:16.113383+0000","last_unstale":"2024-04-24T08:13:17.235388+0000","last_undegraded":"2024-04-24T08:13:17.235388+0000","last_fullsized":"2024-04-24T08:13:17.235388+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:12:06.874957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.102382+0000","last_change":"2024-04-24T08:13:15.070163+0000","last_active":"2024-04-24T08:13:17.102382+0000","last_peered":"2024-04-24T08:13:17.102382+0000","last_clean":"2024-04-24T08:13:17.102382+0000","last_became_active":"2024-04-24T08:13:15.069791+0000","last_became_peered":"2024-04-24T08:13:15.069791+0000","last_unstale":"2024-04-24T08:13:17.102382+0000","last_undegraded":"2024-04-24T08:13:17.102382+0000","last_fullsized":"2024-04-24T08:13:17.102382+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:28.677994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101230+0000","last_change":"2024-04-24T08:13:16.096903+0000","last_active":"2024-04-24T08:13:17.101230+0000","last_peered":"2024-04-24T08:13:17.101230+0000","last_clean":"2024-04-24T08:13:17.101230+0000","last_became_active":"2024-04-24T08:13:16.096517+0000","last_became_peered":"2024-04-24T08:13:16.096517+0000","last_unstale":"2024-04-24T08:13:17.101230+0000","last_undegraded":"2024-04-24T08:13:17.101230+0000","last_fullsized":"2024-04-24T08:13:17.101230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:20:45.863887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231011+0000","last_change":"2024-04-24T08:13:15.073598+0000","last_active":"2024-04-24T08:13:17.231011+0000","last_peered":"2024-04-24T08:13:17.231011+0000","last_clean":"2024-04-24T08:13:17.231011+0000","last_became_active":"2024-04-24T08:13:15.073267+0000","last_became_peered":"2024-04-24T08:13:15.073267+0000","last_unstale":"2024-04-24T08:13:17.231011+0000","last_undegraded":"2024-04-24T08:13:17.231011+0000","last_fullsized":"2024-04-24T08:13:17.231011+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:50:23.307699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234607+0000","last_change":"2024-04-24T08:13:16.107693+0000","last_active":"2024-04-24T08:13:17.234607+0000","last_peered":"2024-04-24T08:13:17.234607+0000","last_clean":"2024-04-24T08:13:17.234607+0000","last_became_active":"2024-04-24T08:13:16.106228+0000","last_became_peered":"2024-04-24T08:13:16.106228+0000","last_unstale":"2024-04-24T08:13:17.234607+0000","last_undegraded":"2024-04-24T08:13:17.234607+0000","last_fullsized":"2024-04-24T08:13:17.234607+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:17:23.703236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100688+0000","last_change":"2024-04-24T08:13:15.069709+0000","last_active":"2024-04-24T08:13:17.100688+0000","last_peered":"2024-04-24T08:13:17.100688+0000","last_clean":"2024-04-24T08:13:17.100688+0000","last_became_active":"2024-04-24T08:13:15.068812+0000","last_became_peered":"2024-04-24T08:13:15.068812+0000","last_unstale":"2024-04-24T08:13:17.100688+0000","last_undegraded":"2024-04-24T08:13:17.100688+0000","last_fullsized":"2024-04-24T08:13:17.100688+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:25:03.246516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.099465+0000","last_change":"2024-04-24T08:13:16.091291+0000","last_active":"2024-04-24T08:13:17.099465+0000","last_peered":"2024-04-24T08:13:17.099465+0000","last_clean":"2024-04-24T08:13:17.099465+0000","last_became_active":"2024-04-24T08:13:16.091013+0000","last_became_peered":"2024-04-24T08:13:16.091013+0000","last_unstale":"2024-04-24T08:13:17.099465+0000","last_undegraded":"2024-04-24T08:13:17.099465+0000","last_fullsized":"2024-04-24T08:13:17.099465+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:48:53.506577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230934+0000","last_change":"2024-04-24T08:13:15.074271+0000","last_active":"2024-04-24T08:13:17.230934+0000","last_peered":"2024-04-24T08:13:17.230934+0000","last_clean":"2024-04-24T08:13:17.230934+0000","last_became_active":"2024-04-24T08:13:15.073770+0000","last_became_peered":"2024-04-24T08:13:15.073770+0000","last_unstale":"2024-04-24T08:13:17.230934+0000","last_undegraded":"2024-04-24T08:13:17.230934+0000","last_fullsized":"2024-04-24T08:13:17.230934+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:56.264098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.114831+0000","last_change":"2024-04-24T08:13:16.108575+0000","last_active":"2024-04-24T08:13:17.114831+0000","last_peered":"2024-04-24T08:13:17.114831+0000","last_clean":"2024-04-24T08:13:17.114831+0000","last_became_active":"2024-04-24T08:13:16.108195+0000","last_became_peered":"2024-04-24T08:13:16.108195+0000","last_unstale":"2024-04-24T08:13:17.114831+0000","last_undegraded":"2024-04-24T08:13:17.114831+0000","last_fullsized":"2024-04-24T08:13:17.114831+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:34.269939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.099531+0000","last_change":"2024-04-24T08:13:15.065237+0000","last_active":"2024-04-24T08:13:17.099531+0000","last_peered":"2024-04-24T08:13:17.099531+0000","last_clean":"2024-04-24T08:13:17.099531+0000","last_became_active":"2024-04-24T08:13:15.064337+0000","last_became_peered":"2024-04-24T08:13:15.064337+0000","last_unstale":"2024-04-24T08:13:17.099531+0000","last_undegraded":"2024-04-24T08:13:17.099531+0000","last_fullsized":"2024-04-24T08:13:17.099531+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:36:43.582929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101773+0000","last_change":"2024-04-24T08:13:16.109914+0000","last_active":"2024-04-24T08:13:17.101773+0000","last_peered":"2024-04-24T08:13:17.101773+0000","last_clean":"2024-04-24T08:13:17.101773+0000","last_became_active":"2024-04-24T08:13:16.109605+0000","last_became_peered":"2024-04-24T08:13:16.109605+0000","last_unstale":"2024-04-24T08:13:17.101773+0000","last_undegraded":"2024-04-24T08:13:17.101773+0000","last_fullsized":"2024-04-24T08:13:17.101773+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:20:12.916151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234122+0000","last_change":"2024-04-24T08:13:15.064559+0000","last_active":"2024-04-24T08:13:17.234122+0000","last_peered":"2024-04-24T08:13:17.234122+0000","last_clean":"2024-04-24T08:13:17.234122+0000","last_became_active":"2024-04-24T08:13:15.064253+0000","last_became_peered":"2024-04-24T08:13:15.064253+0000","last_unstale":"2024-04-24T08:13:17.234122+0000","last_undegraded":"2024-04-24T08:13:17.234122+0000","last_fullsized":"2024-04-24T08:13:17.234122+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:51:28.936432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234853+0000","last_change":"2024-04-24T08:13:16.118367+0000","last_active":"2024-04-24T08:13:17.234853+0000","last_peered":"2024-04-24T08:13:17.234853+0000","last_clean":"2024-04-24T08:13:17.234853+0000","last_became_active":"2024-04-24T08:13:16.115497+0000","last_became_peered":"2024-04-24T08:13:16.115497+0000","last_unstale":"2024-04-24T08:13:17.234853+0000","last_undegraded":"2024-04-24T08:13:17.234853+0000","last_fullsized":"2024-04-24T08:13:17.234853+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:55.671217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101837+0000","last_change":"2024-04-24T08:13:15.075219+0000","last_active":"2024-04-24T08:13:17.101837+0000","last_peered":"2024-04-24T08:13:17.101837+0000","last_clean":"2024-04-24T08:13:17.101837+0000","last_became_active":"2024-04-24T08:13:15.074952+0000","last_became_peered":"2024-04-24T08:13:15.074952+0000","last_unstale":"2024-04-24T08:13:17.101837+0000","last_undegraded":"2024-04-24T08:13:17.101837+0000","last_fullsized":"2024-04-24T08:13:17.101837+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:35.089414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.102191+0000","last_change":"2024-04-24T08:13:16.104943+0000","last_active":"2024-04-24T08:13:17.102191+0000","last_peered":"2024-04-24T08:13:17.102191+0000","last_clean":"2024-04-24T08:13:17.102191+0000","last_became_active":"2024-04-24T08:13:16.104530+0000","last_became_peered":"2024-04-24T08:13:16.104530+0000","last_unstale":"2024-04-24T08:13:17.102191+0000","last_undegraded":"2024-04-24T08:13:17.102191+0000","last_fullsized":"2024-04-24T08:13:17.102191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:41:17.847511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"67'3","reported_seq":17,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230768+0000","last_change":"2024-04-24T08:13:15.071218+0000","last_active":"2024-04-24T08:13:17.230768+0000","last_peered":"2024-04-24T08:13:17.230768+0000","last_clean":"2024-04-24T08:13:17.230768+0000","last_became_active":"2024-04-24T08:13:15.070675+0000","last_became_peered":"2024-04-24T08:13:15.070675+0000","last_unstale":"2024-04-24T08:13:17.230768+0000","last_undegraded":"2024-04-24T08:13:17.230768+0000","last_fullsized":"2024-04-24T08:13:17.230768+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:12:05.860614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100202+0000","last_change":"2024-04-24T08:13:16.103051+0000","last_active":"2024-04-24T08:13:17.100202+0000","last_peered":"2024-04-24T08:13:17.100202+0000","last_clean":"2024-04-24T08:13:17.100202+0000","last_became_active":"2024-04-24T08:13:16.102717+0000","last_became_peered":"2024-04-24T08:13:16.102717+0000","last_unstale":"2024-04-24T08:13:17.100202+0000","last_undegraded":"2024-04-24T08:13:17.100202+0000","last_fullsized":"2024-04-24T08:13:17.100202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:43:55.498854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"67'6","reported_seq":20,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231437+0000","last_change":"2024-04-24T08:13:15.066088+0000","last_active":"2024-04-24T08:13:17.231437+0000","last_peered":"2024-04-24T08:13:17.231437+0000","last_clean":"2024-04-24T08:13:17.231437+0000","last_became_active":"2024-04-24T08:13:15.065822+0000","last_became_peered":"2024-04-24T08:13:15.065822+0000","last_unstale":"2024-04-24T08:13:17.231437+0000","last_undegraded":"2024-04-24T08:13:17.231437+0000","last_fullsized":"2024-04-24T08:13:17.231437+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:24.580314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234418+0000","last_change":"2024-04-24T08:13:16.109852+0000","last_active":"2024-04-24T08:13:17.234418+0000","last_peered":"2024-04-24T08:13:17.234418+0000","last_clean":"2024-04-24T08:13:17.234418+0000","last_became_active":"2024-04-24T08:13:16.103554+0000","last_became_peered":"2024-04-24T08:13:16.103554+0000","last_unstale":"2024-04-24T08:13:17.234418+0000","last_undegraded":"2024-04-24T08:13:17.234418+0000","last_fullsized":"2024-04-24T08:13:17.234418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:17:23.703236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231041+0000","last_change":"2024-04-24T08:13:15.071661+0000","last_active":"2024-04-24T08:13:17.231041+0000","last_peered":"2024-04-24T08:13:17.231041+0000","last_clean":"2024-04-24T08:13:17.231041+0000","last_became_active":"2024-04-24T08:13:15.069876+0000","last_became_peered":"2024-04-24T08:13:15.069876+0000","last_unstale":"2024-04-24T08:13:17.231041+0000","last_undegraded":"2024-04-24T08:13:17.231041+0000","last_fullsized":"2024-04-24T08:13:17.231041+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:05.968909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101377+0000","last_change":"2024-04-24T08:13:16.084092+0000","last_active":"2024-04-24T08:13:17.101377+0000","last_peered":"2024-04-24T08:13:17.101377+0000","last_clean":"2024-04-24T08:13:17.101377+0000","last_became_active":"2024-04-24T08:13:16.083580+0000","last_became_peered":"2024-04-24T08:13:16.083580+0000","last_unstale":"2024-04-24T08:13:17.101377+0000","last_undegraded":"2024-04-24T08:13:17.101377+0000","last_fullsized":"2024-04-24T08:13:17.101377+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:41.452749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100825+0000","last_change":"2024-04-24T08:13:15.069831+0000","last_active":"2024-04-24T08:13:17.100825+0000","last_peered":"2024-04-24T08:13:17.100825+0000","last_clean":"2024-04-24T08:13:17.100825+0000","last_became_active":"2024-04-24T08:13:15.068208+0000","last_became_peered":"2024-04-24T08:13:15.068208+0000","last_unstale":"2024-04-24T08:13:17.100825+0000","last_undegraded":"2024-04-24T08:13:17.100825+0000","last_fullsized":"2024-04-24T08:13:17.100825+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:21.305064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.114810+0000","last_change":"2024-04-24T08:13:16.100173+0000","last_active":"2024-04-24T08:13:17.114810+0000","last_peered":"2024-04-24T08:13:17.114810+0000","last_clean":"2024-04-24T08:13:17.114810+0000","last_became_active":"2024-04-24T08:13:16.099655+0000","last_became_peered":"2024-04-24T08:13:16.099655+0000","last_unstale":"2024-04-24T08:13:17.114810+0000","last_undegraded":"2024-04-24T08:13:17.114810+0000","last_fullsized":"2024-04-24T08:13:17.114810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:55.671217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100668+0000","last_change":"2024-04-24T08:13:15.066869+0000","last_active":"2024-04-24T08:13:17.100668+0000","last_peered":"2024-04-24T08:13:17.100668+0000","last_clean":"2024-04-24T08:13:17.100668+0000","last_became_active":"2024-04-24T08:13:15.066249+0000","last_became_peered":"2024-04-24T08:13:15.066249+0000","last_unstale":"2024-04-24T08:13:17.100668+0000","last_undegraded":"2024-04-24T08:13:17.100668+0000","last_fullsized":"2024-04-24T08:13:17.100668+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:09.619604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.099980+0000","last_change":"2024-04-24T08:13:16.101594+0000","last_active":"2024-04-24T08:13:17.099980+0000","last_peered":"2024-04-24T08:13:17.099980+0000","last_clean":"2024-04-24T08:13:17.099980+0000","last_became_active":"2024-04-24T08:13:16.101278+0000","last_became_peered":"2024-04-24T08:13:16.101278+0000","last_unstale":"2024-04-24T08:13:17.099980+0000","last_undegraded":"2024-04-24T08:13:17.099980+0000","last_fullsized":"2024-04-24T08:13:17.099980+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:15:40.117019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"67'8","reported_seq":22,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234173+0000","last_change":"2024-04-24T08:13:15.071500+0000","last_active":"2024-04-24T08:13:17.234173+0000","last_peered":"2024-04-24T08:13:17.234173+0000","last_clean":"2024-04-24T08:13:17.234173+0000","last_became_active":"2024-04-24T08:13:15.070640+0000","last_became_peered":"2024-04-24T08:13:15.070640+0000","last_unstale":"2024-04-24T08:13:17.234173+0000","last_undegraded":"2024-04-24T08:13:17.234173+0000","last_fullsized":"2024-04-24T08:13:17.234173+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:09.619604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234478+0000","last_change":"2024-04-24T08:13:16.103842+0000","last_active":"2024-04-24T08:13:17.234478+0000","last_peered":"2024-04-24T08:13:17.234478+0000","last_clean":"2024-04-24T08:13:17.234478+0000","last_became_active":"2024-04-24T08:13:16.103563+0000","last_became_peered":"2024-04-24T08:13:16.103563+0000","last_unstale":"2024-04-24T08:13:17.234478+0000","last_undegraded":"2024-04-24T08:13:17.234478+0000","last_fullsized":"2024-04-24T08:13:17.234478+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:03.715939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101810+0000","last_change":"2024-04-24T08:13:15.074428+0000","last_active":"2024-04-24T08:13:17.101810+0000","last_peered":"2024-04-24T08:13:17.101810+0000","last_clean":"2024-04-24T08:13:17.101810+0000","last_became_active":"2024-04-24T08:13:15.074092+0000","last_became_peered":"2024-04-24T08:13:15.074092+0000","last_unstale":"2024-04-24T08:13:17.101810+0000","last_undegraded":"2024-04-24T08:13:17.101810+0000","last_fullsized":"2024-04-24T08:13:17.101810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:38.953112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235271+0000","last_change":"2024-04-24T08:13:16.105275+0000","last_active":"2024-04-24T08:13:17.235271+0000","last_peered":"2024-04-24T08:13:17.235271+0000","last_clean":"2024-04-24T08:13:17.235271+0000","last_became_active":"2024-04-24T08:13:16.102860+0000","last_became_peered":"2024-04-24T08:13:16.102860+0000","last_unstale":"2024-04-24T08:13:17.235271+0000","last_undegraded":"2024-04-24T08:13:17.235271+0000","last_fullsized":"2024-04-24T08:13:17.235271+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:26:04.544052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.234752+0000","last_change":"2024-04-24T08:13:15.071576+0000","last_active":"2024-04-24T08:13:17.234752+0000","last_peered":"2024-04-24T08:13:17.234752+0000","last_clean":"2024-04-24T08:13:17.234752+0000","last_became_active":"2024-04-24T08:13:15.071279+0000","last_became_peered":"2024-04-24T08:13:15.071279+0000","last_unstale":"2024-04-24T08:13:17.234752+0000","last_undegraded":"2024-04-24T08:13:17.234752+0000","last_fullsized":"2024-04-24T08:13:17.234752+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:38.953112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231404+0000","last_change":"2024-04-24T08:13:16.096510+0000","last_active":"2024-04-24T08:13:17.231404+0000","last_peered":"2024-04-24T08:13:17.231404+0000","last_clean":"2024-04-24T08:13:17.231404+0000","last_became_active":"2024-04-24T08:13:16.093509+0000","last_became_peered":"2024-04-24T08:13:16.093509+0000","last_unstale":"2024-04-24T08:13:17.231404+0000","last_undegraded":"2024-04-24T08:13:17.231404+0000","last_fullsized":"2024-04-24T08:13:17.231404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:39.967454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"67'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.102142+0000","last_change":"2024-04-24T08:13:15.077674+0000","last_active":"2024-04-24T08:13:17.102142+0000","last_peered":"2024-04-24T08:13:17.102142+0000","last_clean":"2024-04-24T08:13:17.102142+0000","last_became_active":"2024-04-24T08:13:15.077040+0000","last_became_peered":"2024-04-24T08:13:15.077040+0000","last_unstale":"2024-04-24T08:13:17.102142+0000","last_undegraded":"2024-04-24T08:13:17.102142+0000","last_fullsized":"2024-04-24T08:13:17.102142+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:07:35.163982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100350+0000","last_change":"2024-04-24T08:13:16.103342+0000","last_active":"2024-04-24T08:13:17.100350+0000","last_peered":"2024-04-24T08:13:17.100350+0000","last_clean":"2024-04-24T08:13:17.100350+0000","last_became_active":"2024-04-24T08:13:16.102755+0000","last_became_peered":"2024-04-24T08:13:16.102755+0000","last_unstale":"2024-04-24T08:13:17.100350+0000","last_undegraded":"2024-04-24T08:13:17.100350+0000","last_fullsized":"2024-04-24T08:13:17.100350+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:21:45.557396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235033+0000","last_change":"2024-04-24T08:13:15.074871+0000","last_active":"2024-04-24T08:13:17.235033+0000","last_peered":"2024-04-24T08:13:17.235033+0000","last_clean":"2024-04-24T08:13:17.235033+0000","last_became_active":"2024-04-24T08:13:15.074448+0000","last_became_peered":"2024-04-24T08:13:15.074448+0000","last_unstale":"2024-04-24T08:13:17.235033+0000","last_undegraded":"2024-04-24T08:13:17.235033+0000","last_fullsized":"2024-04-24T08:13:17.235033+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:37:58.198960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230880+0000","last_change":"2024-04-24T08:13:16.101507+0000","last_active":"2024-04-24T08:13:17.230880+0000","last_peered":"2024-04-24T08:13:17.230880+0000","last_clean":"2024-04-24T08:13:17.230880+0000","last_became_active":"2024-04-24T08:13:16.101222+0000","last_became_peered":"2024-04-24T08:13:16.101222+0000","last_unstale":"2024-04-24T08:13:17.230880+0000","last_undegraded":"2024-04-24T08:13:17.230880+0000","last_fullsized":"2024-04-24T08:13:17.230880+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:16.240899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100932+0000","last_change":"2024-04-24T08:13:15.070239+0000","last_active":"2024-04-24T08:13:17.100932+0000","last_peered":"2024-04-24T08:13:17.100932+0000","last_clean":"2024-04-24T08:13:17.100932+0000","last_became_active":"2024-04-24T08:13:15.069396+0000","last_became_peered":"2024-04-24T08:13:15.069396+0000","last_unstale":"2024-04-24T08:13:17.100932+0000","last_undegraded":"2024-04-24T08:13:17.100932+0000","last_fullsized":"2024-04-24T08:13:17.100932+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:51:28.936432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230462+0000","last_change":"2024-04-24T08:13:16.107811+0000","last_active":"2024-04-24T08:13:17.230462+0000","last_peered":"2024-04-24T08:13:17.230462+0000","last_clean":"2024-04-24T08:13:17.230462+0000","last_became_active":"2024-04-24T08:13:16.106526+0000","last_became_peered":"2024-04-24T08:13:16.106526+0000","last_unstale":"2024-04-24T08:13:17.230462+0000","last_undegraded":"2024-04-24T08:13:17.230462+0000","last_fullsized":"2024-04-24T08:13:17.230462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:05:11.404817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100830+0000","last_change":"2024-04-24T08:13:15.064228+0000","last_active":"2024-04-24T08:13:17.100830+0000","last_peered":"2024-04-24T08:13:17.100830+0000","last_clean":"2024-04-24T08:13:17.100830+0000","last_became_active":"2024-04-24T08:13:15.063916+0000","last_became_peered":"2024-04-24T08:13:15.063916+0000","last_unstale":"2024-04-24T08:13:17.100830+0000","last_undegraded":"2024-04-24T08:13:17.100830+0000","last_fullsized":"2024-04-24T08:13:17.100830+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:35.089414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230498+0000","last_change":"2024-04-24T08:13:16.106128+0000","last_active":"2024-04-24T08:13:17.230498+0000","last_peered":"2024-04-24T08:13:17.230498+0000","last_clean":"2024-04-24T08:13:17.230498+0000","last_became_active":"2024-04-24T08:13:16.105874+0000","last_became_peered":"2024-04-24T08:13:16.105874+0000","last_unstale":"2024-04-24T08:13:17.230498+0000","last_undegraded":"2024-04-24T08:13:17.230498+0000","last_fullsized":"2024-04-24T08:13:17.230498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:36:11.669117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.114589+0000","last_change":"2024-04-24T08:13:15.061960+0000","last_active":"2024-04-24T08:13:17.114589+0000","last_peered":"2024-04-24T08:13:17.114589+0000","last_clean":"2024-04-24T08:13:17.114589+0000","last_became_active":"2024-04-24T08:13:15.061680+0000","last_became_peered":"2024-04-24T08:13:15.061680+0000","last_unstale":"2024-04-24T08:13:17.114589+0000","last_undegraded":"2024-04-24T08:13:17.114589+0000","last_fullsized":"2024-04-24T08:13:17.114589+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:36.362268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115623+0000","last_change":"2024-04-24T08:13:16.098352+0000","last_active":"2024-04-24T08:13:17.115623+0000","last_peered":"2024-04-24T08:13:17.115623+0000","last_clean":"2024-04-24T08:13:17.115623+0000","last_became_active":"2024-04-24T08:13:16.098088+0000","last_became_peered":"2024-04-24T08:13:16.098088+0000","last_unstale":"2024-04-24T08:13:17.115623+0000","last_undegraded":"2024-04-24T08:13:17.115623+0000","last_fullsized":"2024-04-24T08:13:17.115623+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:50:24.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101825+0000","last_change":"2024-04-24T08:13:15.070521+0000","last_active":"2024-04-24T08:13:17.101825+0000","last_peered":"2024-04-24T08:13:17.101825+0000","last_clean":"2024-04-24T08:13:17.101825+0000","last_became_active":"2024-04-24T08:13:15.070214+0000","last_became_peered":"2024-04-24T08:13:15.070214+0000","last_unstale":"2024-04-24T08:13:17.101825+0000","last_undegraded":"2024-04-24T08:13:17.101825+0000","last_fullsized":"2024-04-24T08:13:17.101825+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:09.619604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101944+0000","last_change":"2024-04-24T08:13:16.082145+0000","last_active":"2024-04-24T08:13:17.101944+0000","last_peered":"2024-04-24T08:13:17.101944+0000","last_clean":"2024-04-24T08:13:17.101944+0000","last_became_active":"2024-04-24T08:13:16.080812+0000","last_became_peered":"2024-04-24T08:13:16.080812+0000","last_unstale":"2024-04-24T08:13:17.101944+0000","last_undegraded":"2024-04-24T08:13:17.101944+0000","last_fullsized":"2024-04-24T08:13:17.101944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:22:03.715939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"68'1","reported_seq":15,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.113390+0000","last_change":"2024-04-24T08:13:15.066949+0000","last_active":"2024-04-24T08:13:17.113390+0000","last_peered":"2024-04-24T08:13:17.113390+0000","last_clean":"2024-04-24T08:13:17.113390+0000","last_became_active":"2024-04-24T08:13:15.065150+0000","last_became_peered":"2024-04-24T08:13:15.065150+0000","last_unstale":"2024-04-24T08:13:17.113390+0000","last_undegraded":"2024-04-24T08:13:17.113390+0000","last_fullsized":"2024-04-24T08:13:17.113390+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:28.677994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235219+0000","last_change":"2024-04-24T08:13:16.114504+0000","last_active":"2024-04-24T08:13:17.235219+0000","last_peered":"2024-04-24T08:13:17.235219+0000","last_clean":"2024-04-24T08:13:17.235219+0000","last_became_active":"2024-04-24T08:13:16.114043+0000","last_became_peered":"2024-04-24T08:13:16.114043+0000","last_unstale":"2024-04-24T08:13:17.235219+0000","last_undegraded":"2024-04-24T08:13:17.235219+0000","last_fullsized":"2024-04-24T08:13:17.235219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:51:34.269939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100401+0000","last_change":"2024-04-24T08:13:15.066972+0000","last_active":"2024-04-24T08:13:17.100401+0000","last_peered":"2024-04-24T08:13:17.100401+0000","last_clean":"2024-04-24T08:13:17.100401+0000","last_became_active":"2024-04-24T08:13:15.065993+0000","last_became_peered":"2024-04-24T08:13:15.065993+0000","last_unstale":"2024-04-24T08:13:17.100401+0000","last_undegraded":"2024-04-24T08:13:17.100401+0000","last_fullsized":"2024-04-24T08:13:17.100401+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:12:36.778796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.235536+0000","last_change":"2024-04-24T08:13:16.116076+0000","last_active":"2024-04-24T08:13:17.235536+0000","last_peered":"2024-04-24T08:13:17.235536+0000","last_clean":"2024-04-24T08:13:17.235536+0000","last_became_active":"2024-04-24T08:13:16.113710+0000","last_became_peered":"2024-04-24T08:13:16.113710+0000","last_unstale":"2024-04-24T08:13:17.235536+0000","last_undegraded":"2024-04-24T08:13:17.235536+0000","last_fullsized":"2024-04-24T08:13:17.235536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:50:24.322042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.115226+0000","last_change":"2024-04-24T08:13:15.067169+0000","last_active":"2024-04-24T08:13:17.115226+0000","last_peered":"2024-04-24T08:13:17.115226+0000","last_clean":"2024-04-24T08:13:17.115226+0000","last_became_active":"2024-04-24T08:13:15.066889+0000","last_became_peered":"2024-04-24T08:13:15.066889+0000","last_unstale":"2024-04-24T08:13:17.115226+0000","last_undegraded":"2024-04-24T08:13:17.115226+0000","last_fullsized":"2024-04-24T08:13:17.115226+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:09.908983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100406+0000","last_change":"2024-04-24T08:13:16.102378+0000","last_active":"2024-04-24T08:13:17.100406+0000","last_peered":"2024-04-24T08:13:17.100406+0000","last_clean":"2024-04-24T08:13:17.100406+0000","last_became_active":"2024-04-24T08:13:16.102065+0000","last_became_peered":"2024-04-24T08:13:16.102065+0000","last_unstale":"2024-04-24T08:13:17.100406+0000","last_undegraded":"2024-04-24T08:13:17.100406+0000","last_fullsized":"2024-04-24T08:13:17.100406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:12:42.080237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100087+0000","last_change":"2024-04-24T08:13:15.068237+0000","last_active":"2024-04-24T08:13:17.100087+0000","last_peered":"2024-04-24T08:13:17.100087+0000","last_clean":"2024-04-24T08:13:17.100087+0000","last_became_active":"2024-04-24T08:13:15.067449+0000","last_became_peered":"2024-04-24T08:13:15.067449+0000","last_unstale":"2024-04-24T08:13:17.100087+0000","last_undegraded":"2024-04-24T08:13:17.100087+0000","last_fullsized":"2024-04-24T08:13:17.100087+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:39:39.703527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101127+0000","last_change":"2024-04-24T08:13:16.115536+0000","last_active":"2024-04-24T08:13:17.101127+0000","last_peered":"2024-04-24T08:13:17.101127+0000","last_clean":"2024-04-24T08:13:17.101127+0000","last_became_active":"2024-04-24T08:13:16.115352+0000","last_became_peered":"2024-04-24T08:13:16.115352+0000","last_unstale":"2024-04-24T08:13:17.101127+0000","last_undegraded":"2024-04-24T08:13:17.101127+0000","last_fullsized":"2024-04-24T08:13:17.101127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:37.376610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230930+0000","last_change":"2024-04-24T08:13:15.071531+0000","last_active":"2024-04-24T08:13:17.230930+0000","last_peered":"2024-04-24T08:13:17.230930+0000","last_clean":"2024-04-24T08:13:17.230930+0000","last_became_active":"2024-04-24T08:13:15.071222+0000","last_became_peered":"2024-04-24T08:13:15.071222+0000","last_unstale":"2024-04-24T08:13:17.230930+0000","last_undegraded":"2024-04-24T08:13:17.230930+0000","last_fullsized":"2024-04-24T08:13:17.230930+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:03:16.408051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.231500+0000","last_change":"2024-04-24T08:13:16.099257+0000","last_active":"2024-04-24T08:13:17.231500+0000","last_peered":"2024-04-24T08:13:17.231500+0000","last_clean":"2024-04-24T08:13:17.231500+0000","last_became_active":"2024-04-24T08:13:16.098770+0000","last_became_peered":"2024-04-24T08:13:16.098770+0000","last_unstale":"2024-04-24T08:13:17.231500+0000","last_undegraded":"2024-04-24T08:13:17.231500+0000","last_fullsized":"2024-04-24T08:13:17.231500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:10.633947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.232556+0000","last_change":"2024-04-24T08:13:15.073008+0000","last_active":"2024-04-24T08:13:17.232556+0000","last_peered":"2024-04-24T08:13:17.232556+0000","last_clean":"2024-04-24T08:13:17.232556+0000","last_became_active":"2024-04-24T08:13:15.071170+0000","last_became_peered":"2024-04-24T08:13:15.071170+0000","last_unstale":"2024-04-24T08:13:17.232556+0000","last_undegraded":"2024-04-24T08:13:17.232556+0000","last_fullsized":"2024-04-24T08:13:17.232556+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:59:28.677994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100489+0000","last_change":"2024-04-24T08:13:16.099311+0000","last_active":"2024-04-24T08:13:17.100489+0000","last_peered":"2024-04-24T08:13:17.100489+0000","last_clean":"2024-04-24T08:13:17.100489+0000","last_became_active":"2024-04-24T08:13:16.098961+0000","last_became_peered":"2024-04-24T08:13:16.098961+0000","last_unstale":"2024-04-24T08:13:17.100489+0000","last_undegraded":"2024-04-24T08:13:17.100489+0000","last_fullsized":"2024-04-24T08:13:17.100489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:49:43.085760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.100521+0000","last_change":"2024-04-24T08:13:15.069521+0000","last_active":"2024-04-24T08:13:17.100521+0000","last_peered":"2024-04-24T08:13:17.100521+0000","last_clean":"2024-04-24T08:13:17.100521+0000","last_became_active":"2024-04-24T08:13:15.068894+0000","last_became_peered":"2024-04-24T08:13:15.068894+0000","last_unstale":"2024-04-24T08:13:17.100521+0000","last_undegraded":"2024-04-24T08:13:17.100521+0000","last_fullsized":"2024-04-24T08:13:17.100521+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:47:24.580314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.230527+0000","last_change":"2024-04-24T08:13:16.111468+0000","last_active":"2024-04-24T08:13:17.230527+0000","last_peered":"2024-04-24T08:13:17.230527+0000","last_clean":"2024-04-24T08:13:17.230527+0000","last_became_active":"2024-04-24T08:13:16.111194+0000","last_became_peered":"2024-04-24T08:13:16.111194+0000","last_unstale":"2024-04-24T08:13:17.230527+0000","last_undegraded":"2024-04-24T08:13:17.230527+0000","last_fullsized":"2024-04-24T08:13:17.230527+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:15.027953+0000","last_clean_scrub_stamp":"2024-04-24T08:13:15.027953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:03:00.261397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"67'2","reported_seq":16,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.101299+0000","last_change":"2024-04-24T08:13:15.075499+0000","last_active":"2024-04-24T08:13:17.101299+0000","last_peered":"2024-04-24T08:13:17.101299+0000","last_clean":"2024-04-24T08:13:17.101299+0000","last_became_active":"2024-04-24T08:13:15.072540+0000","last_became_peered":"2024-04-24T08:13:15.072540+0000","last_unstale":"2024-04-24T08:13:17.101299+0000","last_undegraded":"2024-04-24T08:13:17.101299+0000","last_fullsized":"2024-04-24T08:13:17.101299+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:13:14.013611+0000","last_clean_scrub_stamp":"2024-04-24T08:13:14.013611+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:57:16.187947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":256,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-04-24T08:13:17.233501+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:13:17.233501+0000","last_peered":"2024-04-24T08:13:17.233501+0000","last_clean":"2024-04-24T08:13:17.233501+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:13:17.233501+0000","last_undegraded":"2024-04-24T08:13:17.233501+0000","last_fullsized":"2024-04-24T08:13:17.233501+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574915,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1429504,"data_stored":939496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27840,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344000,"statfs":{"total":96636764160,"available":96608256000,"internally_reserved":0,"allocated":835584,"data_stored":348604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339844,"statfs":{"total":96636764160,"available":96604000256,"internally_reserved":0,"allocated":831488,"data_stored":348602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":831488,"data_stored":348582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32000,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339840,"statfs":{"total":96636764160,"available":96603996160,"internally_reserved":0,"allocated":835584,"data_stored":348624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32004,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339836,"statfs":{"total":96636764160,"available":96603992064,"internally_reserved":0,"allocated":839680,"data_stored":350124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":847872,"data_stored":350740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":938880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:13:20.987 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T08:13:20.987 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.230+0000 7f39c3c00640 1 Processor -- start 2024-04-24T08:13:21.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.231+0000 7f39c3c00640 1 -- start start 2024-04-24T08:13:21.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.232+0000 7f39c3c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f39bc145a20 0x7f39bc145eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.232+0000 7f39c3c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39bc0d0300 0x7f39bc0d0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.232+0000 7f39c3c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39bc0cf610 con 0x7f39bc0cd8e0 2024-04-24T08:13:21.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.232+0000 7f39c3c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f39bc0cf8f0 con 0x7f39bc145a20 2024-04-24T08:13:21.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.232+0000 7f39c3c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f39bc0cfbd0 con 0x7f39bc0d0300 2024-04-24T08:13:21.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.232+0000 7f39c1200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f39bc0cd8e0 0x7f39bc06e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39030/0 (socket says 172.21.15.69:39030) 2024-04-24T08:13:21.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.232+0000 7f39c1200640 1 -- 172.21.15.69:0/3258740482 learned_addr learned my addr 172.21.15.69:0/3258740482 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:21.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bbe00640 1 --2- 172.21.15.69:0/3258740482 >> v2:172.21.15.121:3300/0 conn(0x7f39bc145a20 0x7f39bc145eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39c1c00640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39bc0d0300 0x7f39bc0d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 953765080 0 0) 0x7f39bc0cf610 con 0x7f39bc0cd8e0 2024-04-24T08:13:21.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f39a4003520 con 0x7f39bc0cd8e0 2024-04-24T08:13:21.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2162939879 0 0) 0x7f39a4003520 con 0x7f39bc0cd8e0 2024-04-24T08:13:21.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39bc0d0300 msgr2=0x7f39bc0d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:21.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39bc0d0300 0x7f39bc0d0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 >> v2:172.21.15.121:3300/0 conn(0x7f39bc145a20 msgr2=0x7f39bc145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:21.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 --2- 172.21.15.69:0/3258740482 >> v2:172.21.15.121:3300/0 conn(0x7f39bc145a20 0x7f39bc145eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39bc0cfeb0 con 0x7f39bc0cd8e0 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f39ac001670 con 0x7f39bc0cd8e0 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 >> v1:172.21.15.69:6789/0 conn(0x7f39bc0cd8e0 legacy=0x7f39bc06e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39a40039b0 0x7f39a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.233+0000 7f39bb400640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a4004800 0x7f39a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39bb400640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39a40051c0 0x7f39a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc0cf8f0 con 0x7f39a4004800 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc0cfbd0 con 0x7f39a40039b0 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f39a400bf30 con 0x7f39a40051c0 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39c1c00640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39bc0d0300 0x7f39bc0d0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39bbe00640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a4004800 0x7f39a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39bbe00640 1 -- 172.21.15.69:0/3258740482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39a40051c0 msgr2=0x7f39a400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39bbe00640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39a40051c0 0x7f39a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39bbe00640 1 -- 172.21.15.69:0/3258740482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39a40039b0 msgr2=0x7f39a4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39bbe00640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39a40039b0 0x7f39a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.234+0000 7f39bbe00640 1 -- 172.21.15.69:0/3258740482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a4003520 con 0x7f39a4004800 2024-04-24T08:13:21.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.235+0000 7f39bbe00640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a4004800 0x7f39a4004c80 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009ac0 tx=0x7f39a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3b60091f500681 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:21.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.235+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0036070 con 0x7f39a4004800 2024-04-24T08:13:21.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.235+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b0031400 con 0x7f39a4004800 2024-04-24T08:13:21.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.235+0000 7f39bb400640 1 -- 172.21.15.69:0/3258740482 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b0009690 con 0x7f39a4004800 2024-04-24T08:13:21.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.236+0000 7f39c3c00640 1 -- 172.21.15.69:0/3258740482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a4004800 msgr2=0x7f39a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.236+0000 7f39c3c00640 1 --2- 172.21.15.69:0/3258740482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39a4004800 0x7f39a4004c80 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009ac0 tx=0x7f39a4005830 comp rx=0 tx=0).stop 2024-04-24T08:13:21.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.236+0000 7f39c1c00640 1 -- 172.21.15.69:0/3258740482 reap_dead start 2024-04-24T08:13:21.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.236+0000 7f39c3c00640 1 -- 172.21.15.69:0/3258740482 shutdown_connections 2024-04-24T08:13:21.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.236+0000 7f39c3c00640 1 -- 172.21.15.69:0/3258740482 >> 172.21.15.69:0/3258740482 conn(0x7f39bc06ba20 msgr2=0x7f39bc06be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:21.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.236+0000 7f39c3c00640 1 -- 172.21.15.69:0/3258740482 shutdown_connections 2024-04-24T08:13:21.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.236+0000 7f39c3c00640 1 -- 172.21.15.69:0/3258740482 wait complete. 2024-04-24T08:13:21.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.238+0000 7f39c3c00640 1 Processor -- start 2024-04-24T08:13:21.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.238+0000 7f39c3c00640 1 -- start start 2024-04-24T08:13:21.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c3c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39a4004800 0x7f39bc15eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c3c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39bc140810 0x7f39bc0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c3c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39bc0a94c0 0x7f39bc0a9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c3c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc0cf3b0 con 0x7f39bc0a94c0 2024-04-24T08:13:21.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c3c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc0cf230 con 0x7f39a4004800 2024-04-24T08:13:21.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c3c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f39bc0d0790 con 0x7f39bc140810 2024-04-24T08:13:21.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c1c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39bc0a94c0 0x7f39bc0a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c1c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39bc0a94c0 0x7f39bc0a9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41558/0 (socket says 172.21.15.69:41558) 2024-04-24T08:13:21.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c1c00640 1 -- 172.21.15.69:0/2734033087 learned_addr learned my addr 172.21.15.69:0/2734033087 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:21.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c1200640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39a4004800 0x7f39bc15eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c1c00640 1 -- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39bc140810 msgr2=0x7f39bc0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c1c00640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39bc140810 0x7f39bc0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c1c00640 1 -- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39a4004800 msgr2=0x7f39bc15eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c1c00640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39a4004800 0x7f39bc15eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.239+0000 7f39c1c00640 1 -- 172.21.15.69:0/2734033087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39bc15f3f0 con 0x7f39bc0a94c0 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.240+0000 7f39c1200640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39a4004800 0x7f39bc15eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.240+0000 7f39c1c00640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39bc0a94c0 0x7f39bc0a9940 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f39b4007c40 tx=0x7f39b400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.240+0000 7f39b8c00640 1 -- 172.21.15.69:0/2734033087 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b401a070 con 0x7f39bc0a94c0 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.240+0000 7f39b8c00640 1 -- 172.21.15.69:0/2734033087 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b4007e50 con 0x7f39bc0a94c0 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.240+0000 7f39c3c00640 1 -- 172.21.15.69:0/2734033087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39bc0b4880 con 0x7f39bc0a94c0 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.240+0000 7f39b8c00640 1 -- 172.21.15.69:0/2734033087 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4003bd0 con 0x7f39bc0a94c0 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.240+0000 7f39c3c00640 1 -- 172.21.15.69:0/2734033087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39bc0b3fe0 con 0x7f39bc0a94c0 2024-04-24T08:13:21.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.241+0000 7f39c3c00640 1 -- 172.21.15.69:0/2734033087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39bc0a9e80 con 0x7f39bc0a94c0 2024-04-24T08:13:21.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.242+0000 7f39b8c00640 1 -- 172.21.15.69:0/2734033087 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f39b401d070 con 0x7f39bc0a94c0 2024-04-24T08:13:21.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.243+0000 7f39b8c00640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a006c8f0 0x7f39a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.243+0000 7f39b8c00640 1 -- 172.21.15.69:0/2734033087 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f39b408b980 con 0x7f39bc0a94c0 2024-04-24T08:13:21.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.244+0000 7f39c1200640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a006c8f0 0x7f39a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.245+0000 7f39c1200640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a006c8f0 0x7f39a006edb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f39ac001eb0 tx=0x7f39ac009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:21.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.248+0000 7f39b8c00640 1 -- 172.21.15.69:0/2734033087 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f39b4010070 con 0x7f39bc0a94c0 2024-04-24T08:13:21.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.524+0000 7f39c3c00640 1 -- 172.21.15.69:0/2734033087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f39bc0af400 con 0x7f39bc0a94c0 2024-04-24T08:13:21.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.529+0000 7f39b8c00640 1 -- 172.21.15.69:0/2734033087 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) ==== 190+0+147 (secure 0 0 0) 0x7f39b4059d10 con 0x7f39bc0a94c0 2024-04-24T08:13:21.530 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-24T08:13:21.530 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw== 2024-04-24T08:13:21.530 INFO:teuthology.orchestra.run.smithi069.stdout: caps mds = "allow" 2024-04-24T08:13:21.530 INFO:teuthology.orchestra.run.smithi069.stdout: caps mon = "allow r" 2024-04-24T08:13:21.530 INFO:teuthology.orchestra.run.smithi069.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-24T08:13:21.531 INFO:teuthology.orchestra.run.smithi069.stderr:updated caps for client.0 2024-04-24T08:13:21.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.532+0000 7f39c3c00640 1 -- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a006c8f0 msgr2=0x7f39a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.532+0000 7f39c3c00640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a006c8f0 0x7f39a006edb0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f39ac001eb0 tx=0x7f39ac009040 comp rx=0 tx=0).stop 2024-04-24T08:13:21.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.532+0000 7f39c3c00640 1 -- 172.21.15.69:0/2734033087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39bc0a94c0 msgr2=0x7f39bc0a9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.532+0000 7f39c3c00640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39bc0a94c0 0x7f39bc0a9940 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f39b4007c40 tx=0x7f39b400d4a0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.533+0000 7f39c3c00640 1 -- 172.21.15.69:0/2734033087 shutdown_connections 2024-04-24T08:13:21.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.533+0000 7f39c3c00640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f39a006c8f0 0x7f39a006edb0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.533+0000 7f39c3c00640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39bc0a94c0 0x7f39bc0a9940 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.533+0000 7f39c3c00640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39bc140810 0x7f39bc0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.533+0000 7f39c3c00640 1 --2- 172.21.15.69:0/2734033087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39a4004800 0x7f39bc15eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.533+0000 7f39c3c00640 1 -- 172.21.15.69:0/2734033087 >> 172.21.15.69:0/2734033087 conn(0x7f39bc06ba20 msgr2=0x7f39bc072100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:21.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.533+0000 7f39c3c00640 1 -- 172.21.15.69:0/2734033087 shutdown_connections 2024-04-24T08:13:21.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.533+0000 7f39c3c00640 1 -- 172.21.15.69:0/2734033087 wait complete. 2024-04-24T08:13:21.552 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:21.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.736+0000 7f3308000640 1 Processor -- start 2024-04-24T08:13:21.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.736+0000 7f3308000640 1 -- start start 2024-04-24T08:13:21.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.737+0000 7f3308000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f33000d0300 0x7f330006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.737+0000 7f3308000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 0x7f33000d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.737+0000 7f3308000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33000d0c20 con 0x7f3300145fe0 2024-04-24T08:13:21.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.737+0000 7f3308000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f33000cf3d0 con 0x7f33000d0300 2024-04-24T08:13:21.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.737+0000 7f3308000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f33000cf6b0 con 0x7f3300145a90 2024-04-24T08:13:21.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.738+0000 7f3305600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f33000d0300 0x7f330006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.738+0000 7f3305600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f33000d0300 0x7f330006e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:50626/0 (socket says 172.21.15.69:50626) 2024-04-24T08:13:21.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.738+0000 7f3305600640 1 -- 172.21.15.69:0/824319890 learned_addr learned my addr 172.21.15.69:0/824319890 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:21.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.738+0000 7f3304c00640 1 --2- 172.21.15.69:0/824319890 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 0x7f33000d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.739+0000 7f3305600640 1 -- 172.21.15.69:0/824319890 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 msgr2=0x7f33000d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:21.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.739+0000 7f3305600640 1 --2- 172.21.15.69:0/824319890 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 0x7f33000d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.739+0000 7f3305600640 1 -- 172.21.15.69:0/824319890 >> v1:172.21.15.121:6789/0 conn(0x7f3300145fe0 legacy=0x7f3300160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.739+0000 7f3305600640 1 -- 172.21.15.69:0/824319890 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33000cf990 con 0x7f33000d0300 2024-04-24T08:13:21.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.740+0000 7f3305600640 1 --2- 172.21.15.69:0/824319890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33000d0300 0x7f330006e240 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009870 tx=0x7f32f402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfa183d8ab2b4b8c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:21.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.741+0000 7f32ffe00640 1 -- 172.21.15.69:0/824319890 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f4037070 con 0x7f33000d0300 2024-04-24T08:13:21.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.741+0000 7f32ffe00640 1 -- 172.21.15.69:0/824319890 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32f4032440 con 0x7f33000d0300 2024-04-24T08:13:21.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.741+0000 7f32ffe00640 1 -- 172.21.15.69:0/824319890 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f4032810 con 0x7f33000d0300 2024-04-24T08:13:21.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.741+0000 7f3308000640 1 -- 172.21.15.69:0/824319890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33000d0300 msgr2=0x7f330006e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.741+0000 7f3308000640 1 --2- 172.21.15.69:0/824319890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33000d0300 0x7f330006e240 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009870 tx=0x7f32f402a270 comp rx=0 tx=0).stop 2024-04-24T08:13:21.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.742+0000 7f3308000640 1 -- 172.21.15.69:0/824319890 shutdown_connections 2024-04-24T08:13:21.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.742+0000 7f3308000640 1 --2- 172.21.15.69:0/824319890 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 0x7f33000d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.742+0000 7f3308000640 1 --2- 172.21.15.69:0/824319890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33000d0300 0x7f330006e240 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.742+0000 7f3308000640 1 -- 172.21.15.69:0/824319890 >> 172.21.15.69:0/824319890 conn(0x7f3300067d90 msgr2=0x7f330006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:21.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.742+0000 7f3308000640 1 -- 172.21.15.69:0/824319890 shutdown_connections 2024-04-24T08:13:21.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.742+0000 7f3308000640 1 -- 172.21.15.69:0/824319890 wait complete. 2024-04-24T08:13:21.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.743+0000 7f3308000640 1 Processor -- start 2024-04-24T08:13:21.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.743+0000 7f3308000640 1 -- start start 2024-04-24T08:13:21.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.744+0000 7f3308000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33000d0300 0x7f330015e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.744+0000 7f3308000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 0x7f330015bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.744+0000 7f3308000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3300145fe0 0x7f33001428d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.744+0000 7f3308000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3300145770 con 0x7f3300145fe0 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.744+0000 7f3308000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f33000cff50 con 0x7f33000d0300 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.744+0000 7f3308000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f33001458f0 con 0x7f3300145a90 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3306000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3300145fe0 0x7f33001428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3306000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3300145fe0 0x7f33001428d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41568/0 (socket says 172.21.15.69:41568) 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3306000640 1 -- 172.21.15.69:0/1687894443 learned_addr learned my addr 172.21.15.69:0/1687894443 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3305600640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33000d0300 0x7f330015e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3304c00640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 0x7f330015bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3306000640 1 -- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 msgr2=0x7f330015bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3306000640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 0x7f330015bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3306000640 1 -- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33000d0300 msgr2=0x7f330015e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3306000640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33000d0300 0x7f330015e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3306000640 1 -- 172.21.15.69:0/1687894443 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f330015d350 con 0x7f3300145fe0 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3305600640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33000d0300 0x7f330015e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3304c00640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 0x7f330015bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:21.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f3306000640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3300145fe0 0x7f33001428d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f32f800c7a0 tx=0x7f32f800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:21.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.745+0000 7f32fd600640 1 -- 172.21.15.69:0/1687894443 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8003b70 con 0x7f3300145fe0 2024-04-24T08:13:21.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.746+0000 7f32fd600640 1 -- 172.21.15.69:0/1687894443 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32f8003d30 con 0x7f3300145fe0 2024-04-24T08:13:21.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.746+0000 7f32fd600640 1 -- 172.21.15.69:0/1687894443 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f800aa90 con 0x7f3300145fe0 2024-04-24T08:13:21.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.746+0000 7f3308000640 1 -- 172.21.15.69:0/1687894443 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f330015d630 con 0x7f3300145fe0 2024-04-24T08:13:21.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.746+0000 7f3308000640 1 -- 172.21.15.69:0/1687894443 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f330015dbf0 con 0x7f3300145fe0 2024-04-24T08:13:21.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.747+0000 7f3308000640 1 -- 172.21.15.69:0/1687894443 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3300068a80 con 0x7f3300145fe0 2024-04-24T08:13:21.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.748+0000 7f32fd600640 1 -- 172.21.15.69:0/1687894443 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f32f800ac30 con 0x7f3300145fe0 2024-04-24T08:13:21.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.749+0000 7f32fd600640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f32e006c920 0x7f32e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:21.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.749+0000 7f32fd600640 1 -- 172.21.15.69:0/1687894443 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f32f808ba90 con 0x7f3300145fe0 2024-04-24T08:13:21.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.749+0000 7f3305600640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f32e006c920 0x7f32e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:21.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.751+0000 7f3305600640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f32e006c920 0x7f32e006ede0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f32f4034000 tx=0x7f32f402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:21.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.753+0000 7f32fd600640 1 -- 172.21.15.69:0/1687894443 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f32f8059eb0 con 0x7f3300145fe0 2024-04-24T08:13:21.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.986+0000 7f3308000640 1 -- 172.21.15.69:0/1687894443 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f330006cd50 con 0x7f3300145fe0 2024-04-24T08:13:21.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.988+0000 7f32fd600640 1 -- 172.21.15.69:0/1687894443 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5186 (secure 0 0 0) 0x7f32f8060080 con 0x7f3300145fe0 2024-04-24T08:13:21.989 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:21.989 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":6792,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:13:18.588797+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"b","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1135108804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1135108804},{"type":"v1","addr":"172.21.15.121:6837","nonce":1135108804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":5}]} 2024-04-24T08:13:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 44 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 -- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f32e006c920 msgr2=0x7f32e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f32e006c920 0x7f32e006ede0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f32f4034000 tx=0x7f32f402e000 comp rx=0 tx=0).stop 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 -- 172.21.15.69:0/1687894443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3300145fe0 msgr2=0x7f33001428d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3300145fe0 0x7f33001428d0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f32f800c7a0 tx=0x7f32f800cc60 comp rx=0 tx=0).stop 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 -- 172.21.15.69:0/1687894443 shutdown_connections 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f32e006c920 0x7f32e006ede0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3300145fe0 0x7f33001428d0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3300145a90 0x7f330015bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 --2- 172.21.15.69:0/1687894443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33000d0300 0x7f330015e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 -- 172.21.15.69:0/1687894443 >> 172.21.15.69:0/1687894443 conn(0x7f3300067d90 msgr2=0x7f330006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 -- 172.21.15.69:0/1687894443 shutdown_connections 2024-04-24T08:13:21.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:21.990+0000 7f3308000640 1 -- 172.21.15.69:0/1687894443 wait complete. 2024-04-24T08:13:22.016 INFO:tasks.ceph.mds.b.smithi121.stderr:2024-04-24T08:13:22.015+0000 7f4f32600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:13:22.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:13:15.498700+0000', 'modified': '2024-04-24T08:13:18.588797+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6804}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6804': {'gid': 6804, 'name': 'b', 'rank': 0, 'incarnation': 40, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6837/1135108804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6835', 'nonce': 1135108804}, {'type': 'v1', 'addr': '172.21.15.121:6837', 'nonce': 1135108804}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6804, 'qdb_cluster': [6804]} 2024-04-24T08:13:22.411 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:13:22.412 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T08:13:22.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.618+0000 7f66f5e00640 1 Processor -- start 2024-04-24T08:13:22.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.618+0000 7f66f5e00640 1 -- start start 2024-04-24T08:13:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.619+0000 7f66f5e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66f0156c60 0x7f66f015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.619+0000 7f66f5e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00ad7e0 0x7f66f015d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.619+0000 7f66f5e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66f015d660 con 0x7f66f00b30f0 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.619+0000 7f66f5e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f66f006dc30 con 0x7f66f0156c60 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.619+0000 7f66f5e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f66f006df10 con 0x7f66f00ad7e0 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.620+0000 7f66f4a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66f0156c60 0x7f66f015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.620+0000 7f66f4a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66f0156c60 0x7f66f015cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41582/0 (socket says 172.21.15.69:41582) 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.620+0000 7f66f4a00640 1 -- 172.21.15.69:0/419738408 learned_addr learned my addr 172.21.15.69:0/419738408 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.620+0000 7f66f4a00640 1 -- 172.21.15.69:0/419738408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00ad7e0 msgr2=0x7f66f015d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.620+0000 7f66f4a00640 1 --2- 172.21.15.69:0/419738408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00ad7e0 0x7f66f015d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.620+0000 7f66f4a00640 1 -- 172.21.15.69:0/419738408 >> v1:172.21.15.121:6789/0 conn(0x7f66f00b30f0 legacy=0x7f66f00ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.620+0000 7f66f4a00640 1 -- 172.21.15.69:0/419738408 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f006e1f0 con 0x7f66f0156c60 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.621+0000 7f66f4a00640 1 --2- 172.21.15.69:0/419738408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 0x7f66f015cac0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009870 tx=0x7f66e402a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f438bb0fa024579 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:22.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.621+0000 7f66ef400640 1 -- 172.21.15.69:0/419738408 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4037070 con 0x7f66f0156c60 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.621+0000 7f66ef400640 1 -- 172.21.15.69:0/419738408 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e4032440 con 0x7f66f0156c60 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.621+0000 7f66ef400640 1 -- 172.21.15.69:0/419738408 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4032810 con 0x7f66f0156c60 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.622+0000 7f66f5e00640 1 -- 172.21.15.69:0/419738408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 msgr2=0x7f66f015cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.622+0000 7f66f5e00640 1 --2- 172.21.15.69:0/419738408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 0x7f66f015cac0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009870 tx=0x7f66e402a200 comp rx=0 tx=0).stop 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.622+0000 7f66f5e00640 1 -- 172.21.15.69:0/419738408 shutdown_connections 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.622+0000 7f66f5e00640 1 --2- 172.21.15.69:0/419738408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00ad7e0 0x7f66f015d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.622+0000 7f66f5e00640 1 --2- 172.21.15.69:0/419738408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 0x7f66f015cac0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.622+0000 7f66f5e00640 1 -- 172.21.15.69:0/419738408 >> 172.21.15.69:0/419738408 conn(0x7f66f00d0160 msgr2=0x7f66f00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.622+0000 7f66f5e00640 1 -- 172.21.15.69:0/419738408 shutdown_connections 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.622+0000 7f66f5e00640 1 -- 172.21.15.69:0/419738408 wait complete. 2024-04-24T08:13:22.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.624+0000 7f66f5e00640 1 Processor -- start 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.624+0000 7f66f5e00640 1 -- start start 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66f00ad7e0 0x7f66f014bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00b30f0 0x7f66f01449a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 0x7f66f0144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f66f006e710 con 0x7f66f0156c60 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f66f015d940 con 0x7f66f00ad7e0 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f66f006e890 con 0x7f66f00b30f0 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 0x7f66f0144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 0x7f66f0144ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41584/0 (socket says 172.21.15.69:41584) 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5400640 1 -- 172.21.15.69:0/2655242641 learned_addr learned my addr 172.21.15.69:0/2655242641 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f4a00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66f00ad7e0 0x7f66f014bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66efe00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00b30f0 0x7f66f01449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5400640 1 -- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00b30f0 msgr2=0x7f66f01449a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5400640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00b30f0 0x7f66f01449a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5400640 1 -- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66f00ad7e0 msgr2=0x7f66f014bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5400640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66f00ad7e0 0x7f66f014bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66f5400640 1 -- 172.21.15.69:0/2655242641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f014c070 con 0x7f66f0156c60 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.625+0000 7f66efe00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00b30f0 0x7f66f01449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.626+0000 7f66f5400640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 0x7f66f0144ee0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f66e800b9e0 tx=0x7f66e800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.626+0000 7f66ecc00640 1 -- 172.21.15.69:0/2655242641 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8019070 con 0x7f66f0156c60 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.626+0000 7f66ecc00640 1 -- 172.21.15.69:0/2655242641 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e800cb40 con 0x7f66f0156c60 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.626+0000 7f66ecc00640 1 -- 172.21.15.69:0/2655242641 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8014440 con 0x7f66f0156c60 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.626+0000 7f66f5e00640 1 -- 172.21.15.69:0/2655242641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f0159d50 con 0x7f66f0156c60 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.626+0000 7f66f5e00640 1 -- 172.21.15.69:0/2655242641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f00cd220 con 0x7f66f0156c60 2024-04-24T08:13:22.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.628+0000 7f66ecc00640 1 -- 172.21.15.69:0/2655242641 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f66e801c070 con 0x7f66f0156c60 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.629+0000 7f66ecc00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66d006c8f0 0x7f66d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.629+0000 7f66ecc00640 1 -- 172.21.15.69:0/2655242641 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f66e808b8c0 con 0x7f66f0156c60 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.630+0000 7f66f4a00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66d006c8f0 0x7f66d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.630+0000 7f66fa3d3740 1 Processor -- start 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.630+0000 7f66fa3d3740 1 -- start start 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d513310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d543aa0 0x56434d542e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543360 0x56434d518170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56434d47db10 con 0x56434d560980 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56434d4242b0 con 0x56434d543360 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56434d479ad0 con 0x56434d543aa0 2024-04-24T08:13:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66f4a00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66d006c8f0 0x7f66d006edb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f66e4034000 tx=0x7f66e402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66f4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d513310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66f4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d513310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41590/0 (socket says 172.21.15.69:41590) 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66f4a00640 1 -- 172.21.15.69:0/3280511443 learned_addr learned my addr 172.21.15.69:0/3280511443 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66f5400640 1 --2- 172.21.15.69:0/3280511443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543360 0x56434d518170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66efe00640 1 --2- 172.21.15.69:0/3280511443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d543aa0 0x56434d542e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66f4a00640 1 -- 172.21.15.69:0/3280511443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d543aa0 msgr2=0x56434d542e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.631+0000 7f66f4a00640 1 --2- 172.21.15.69:0/3280511443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d543aa0 0x56434d542e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.632+0000 7f66f4a00640 1 -- 172.21.15.69:0/3280511443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543360 msgr2=0x56434d518170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.632+0000 7f66f4a00640 1 --2- 172.21.15.69:0/3280511443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543360 0x56434d518170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.632+0000 7f66f4a00640 1 -- 172.21.15.69:0/3280511443 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56434d1902d0 con 0x56434d560980 2024-04-24T08:13:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.632+0000 7f66f5400640 1 --2- 172.21.15.69:0/3280511443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543360 0x56434d518170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.632+0000 7f66f4a00640 1 --2- 172.21.15.69:0/3280511443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d513310 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009960 tx=0x7f66e402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3fa254987673024 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.632+0000 7f66e1200640 1 -- 172.21.15.69:0/3280511443 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4037040 con 0x56434d560980 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.632+0000 7f66e1200640 1 -- 172.21.15.69:0/3280511443 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e4032de0 con 0x56434d560980 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.632+0000 7f66e1200640 1 -- 172.21.15.69:0/3280511443 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e403e660 con 0x56434d560980 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.633+0000 7f66fa3d3740 1 -- 172.21.15.69:0/3280511443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 msgr2=0x56434d513310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.633+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/3280511443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d513310 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009960 tx=0x7f66e402c000 comp rx=0 tx=0).stop 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.633+0000 7f66fa3d3740 1 -- 172.21.15.69:0/3280511443 shutdown_connections 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.633+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/3280511443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543360 0x56434d518170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.633+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/3280511443 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d543aa0 0x56434d542e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.633+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/3280511443 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d513310 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.633+0000 7f66fa3d3740 1 -- 172.21.15.69:0/3280511443 >> 172.21.15.69:0/3280511443 conn(0x56434d515580 msgr2=0x56434d55de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.633+0000 7f66fa3d3740 1 -- 172.21.15.69:0/3280511443 shutdown_connections 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.633+0000 7f66fa3d3740 1 -- 172.21.15.69:0/3280511443 wait complete. 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.633+0000 7f66fa3d3740 1 -- start start 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.634+0000 7f66fa3d3740 1 Processor -- start 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d543aa0 0x56434d5563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d560980 0x56434d546d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d56d0f0 0x56434d554ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56434d517730 con 0x56434d56d0f0 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56434d437590 con 0x56434d560980 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56434d516b20 con 0x56434d543aa0 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66fa3d3740 10 client.0 _ll_register_callbacks cb 0x56434d41a3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66f5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d56d0f0 0x56434d554ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66f5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d56d0f0 0x56434d554ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41604/0 (socket says 172.21.15.69:41604) 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66f5400640 1 -- 172.21.15.69:0/1782598463 learned_addr learned my addr 172.21.15.69:0/1782598463 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66f5400640 1 -- 172.21.15.69:0/1782598463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d543aa0 msgr2=0x56434d5563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66f5400640 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d543aa0 0x56434d5563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66f5400640 1 -- 172.21.15.69:0/1782598463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d560980 msgr2=0x56434d546d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66f5400640 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d560980 0x56434d546d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.635+0000 7f66f5400640 1 -- 172.21.15.69:0/1782598463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56434d25a6b0 con 0x56434d56d0f0 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66f5400640 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d56d0f0 0x56434d554ae0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f66e800c620 tx=0x7f66e805d9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529042932fc5d2c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66d5600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66d5600640 1 -- 172.21.15.69:0/1782598463 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8057910 con 0x56434d56d0f0 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66fa3d3740 1 -- 172.21.15.69:0/1782598463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66e4032de0 con 0x56434d56d0f0 2024-04-24T08:13:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66d5600640 1 -- 172.21.15.69:0/1782598463 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f66e80589a0 con 0x56434d56d0f0 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66d5600640 1 -- 172.21.15.69:0/1782598463 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f66e8058ba0 con 0x56434d56d0f0 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66fa3d3740 10 client.6939 fetch_fsmap learned FSMap version 44 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66fa3d3740 1 -- 172.21.15.69:0/1782598463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x56434d53dd30 con 0x56434d56d0f0 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66d5600640 1 -- 172.21.15.69:0/1782598463 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e4032de0 con 0x56434d56d0f0 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66d5600640 1 -- 172.21.15.69:0/1782598463 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 44) ==== 3494+0+0 (secure 0 0 0) 0x7f66e80552a0 con 0x56434d56d0f0 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66fa3d3740 10 client.6939 fetch_fsmap finished waiting for FSMap version 44 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66fa3d3740 10 client.6939 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.636+0000 7f66fa3d3740 20 client.6939 populate_metadata read hostname 'smithi069' 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.637+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.121:6835/1135108804,v1:172.21.15.121:6837/1135108804] conn(0x56434d5447d0 0x56434d580ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.637+0000 7f66fa3d3740 4 client.6939 mds_command: new command op to 6804 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.637+0000 7f66fa3d3740 1 -- 172.21.15.69:0/1782598463 --> [v2:172.21.15.121:6835/1135108804,v1:172.21.15.121:6837/1135108804] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56434d48b530 con 0x56434d5447d0 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.637+0000 7f66f4a00640 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.121:6835/1135108804,v1:172.21.15.121:6837/1135108804] conn(0x56434d5447d0 0x56434d580ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.638+0000 7f66f4a00640 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.121:6835/1135108804,v1:172.21.15.121:6837/1135108804] conn(0x56434d5447d0 0x56434d580ef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3ed6905df60e16d server_cookie=37fdfe6b289d4a4f in_seq=0 out_seq=0 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.638+0000 7f66d5600640 10 client.6939 ms_handle_connect on v2:172.21.15.121:6835/1135108804 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.640+0000 7f66d5600640 1 -- 172.21.15.69:0/1782598463 <== mds.0 v2:172.21.15.121:6835/1135108804 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56434d48b530 con 0x56434d5447d0 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.640+0000 7f66d5600640 10 client.6939 handle_command_reply: tid=0 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.640+0000 7f66fa3d3740 1 client.6939 shutdown 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.640+0000 7f66fa3d3740 1 -- 172.21.15.69:0/1782598463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d56d0f0 msgr2=0x56434d554ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.640+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d56d0f0 0x56434d554ae0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f66e800c620 tx=0x7f66e805d9c0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.641+0000 7f66fa3d3740 1 -- 172.21.15.69:0/1782598463 shutdown_connections 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.641+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.121:6835/1135108804,v1:172.21.15.121:6837/1135108804] conn(0x56434d5447d0 0x56434d580ef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.641+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d56d0f0 0x56434d554ae0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.641+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d560980 0x56434d546d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.641+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/1782598463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d543aa0 0x56434d5563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.641+0000 7f66fa3d3740 1 -- 172.21.15.69:0/1782598463 >> 172.21.15.69:0/1782598463 conn(0x56434d543360 msgr2=0x56434d514050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.641+0000 7f66d5600640 0 client.6939 ms_handle_reset on v2:172.21.15.121:6835/1135108804 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.641+0000 7f66fa3d3740 1 -- 172.21.15.69:0/1782598463 shutdown_connections 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.641+0000 7f66fa3d3740 1 -- 172.21.15.69:0/1782598463 wait complete. 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.641+0000 7f66fa3d3740 20 client.6939 trim_cache size 0 max 16384 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.660+0000 7f66fa3d3740 1 Processor -- start 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.660+0000 7f66fa3d3740 1 -- start start 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.661+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d543aa0 0x56434d55f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.661+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d560980 0x56434d514240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.661+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d56d0f0 0x56434d517980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.661+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f66f0159960 con 0x56434d543aa0 2024-04-24T08:13:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.661+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f66f00ced00 con 0x56434d560980 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.661+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f66f0159310 con 0x56434d56d0f0 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66f4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d543aa0 0x56434d55f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66f5400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d56d0f0 0x56434d517980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66f4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d543aa0 0x56434d55f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41614/0 (socket says 172.21.15.69:41614) 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66f4a00640 1 -- 172.21.15.69:0/184515417 learned_addr learned my addr 172.21.15.69:0/184515417 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66efe00640 1 --2- 172.21.15.69:0/184515417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d560980 0x56434d514240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66f4a00640 1 -- 172.21.15.69:0/184515417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d56d0f0 msgr2=0x56434d517980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66f4a00640 1 --2- 172.21.15.69:0/184515417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d56d0f0 0x56434d517980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66f4a00640 1 -- 172.21.15.69:0/184515417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d560980 msgr2=0x56434d514240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66f4a00640 1 --2- 172.21.15.69:0/184515417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d560980 0x56434d514240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66f4a00640 1 -- 172.21.15.69:0/184515417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56434d545160 con 0x56434d543aa0 2024-04-24T08:13:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.662+0000 7f66f4a00640 1 --2- 172.21.15.69:0/184515417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d543aa0 0x56434d55f230 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f66e4000c00 tx=0x7f66e402c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e109dcb098cbd57d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66d7e00640 1 -- 172.21.15.69:0/184515417 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4037070 con 0x56434d543aa0 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66d7e00640 1 -- 172.21.15.69:0/184515417 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e40044c0 con 0x56434d543aa0 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66d7e00640 1 -- 172.21.15.69:0/184515417 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4054120 con 0x56434d543aa0 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66fa3d3740 1 -- 172.21.15.69:0/184515417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d543aa0 msgr2=0x56434d55f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/184515417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d543aa0 0x56434d55f230 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f66e4000c00 tx=0x7f66e402c070 comp rx=0 tx=0).stop 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66fa3d3740 1 -- 172.21.15.69:0/184515417 shutdown_connections 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/184515417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d56d0f0 0x56434d517980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/184515417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d560980 0x56434d514240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/184515417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d543aa0 0x56434d55f230 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66fa3d3740 1 -- 172.21.15.69:0/184515417 >> 172.21.15.69:0/184515417 conn(0x56434d543360 msgr2=0x56434d5170a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.663+0000 7f66fa3d3740 1 -- 172.21.15.69:0/184515417 shutdown_connections 2024-04-24T08:13:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.664+0000 7f66fa3d3740 1 -- 172.21.15.69:0/184515417 wait complete. 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.664+0000 7f66fa3d3740 1 -- start start 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.664+0000 7f66fa3d3740 1 Processor -- start 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.665+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543aa0 0x56434d5c27a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.665+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d556af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.665+0000 7f66fa3d3740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d56d0f0 0x56434d539160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.665+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56434d555360 con 0x56434d560980 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.665+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56434d53e060 con 0x56434d543aa0 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.665+0000 7f66fa3d3740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56434d560120 con 0x56434d56d0f0 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.665+0000 7f66fa3d3740 10 client.0 _ll_register_callbacks cb 0x7f66e8059320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.665+0000 7f66efe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d556af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66efe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d556af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41626/0 (socket says 172.21.15.69:41626) 2024-04-24T08:13:22.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66efe00640 1 -- 172.21.15.69:0/717709021 learned_addr learned my addr 172.21.15.69:0/717709021 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66f4a00640 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543aa0 0x56434d5c27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66f5400640 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d56d0f0 0x56434d539160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66efe00640 1 -- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d56d0f0 msgr2=0x56434d539160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66efe00640 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d56d0f0 0x56434d539160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66efe00640 1 -- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543aa0 msgr2=0x56434d5c27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66efe00640 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543aa0 0x56434d5c27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66efe00640 1 -- 172.21.15.69:0/717709021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56434d5561a0 con 0x56434d560980 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66efe00640 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d556af0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f66f00b34d0 tx=0x7f66dc001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32016ea0b006d27c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66d5600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66d5600640 1 -- 172.21.15.69:0/717709021 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc01b060 con 0x56434d560980 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66d5600640 1 -- 172.21.15.69:0/717709021 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f66dc016b20 con 0x56434d560980 2024-04-24T08:13:22.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.666+0000 7f66fa3d3740 1 -- 172.21.15.69:0/717709021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56434d4853a0 con 0x56434d560980 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66d5600640 1 -- 172.21.15.69:0/717709021 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f66dc016d20 con 0x56434d560980 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66fa3d3740 10 client.6945 fetch_fsmap learned FSMap version 44 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66fa3d3740 1 -- 172.21.15.69:0/717709021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=2+}) -- 0x56434d557ab0 con 0x56434d560980 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66d5600640 1 -- 172.21.15.69:0/717709021 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x56434d4853a0 con 0x56434d560980 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66d5600640 1 -- 172.21.15.69:0/717709021 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 44) ==== 3494+0+0 (secure 0 0 0) 0x7f66dc013540 con 0x56434d560980 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66fa3d3740 10 client.6945 fetch_fsmap finished waiting for FSMap version 44 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66fa3d3740 10 client.6945 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66fa3d3740 20 client.6945 populate_metadata read hostname 'smithi069' 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:6835/1135108804,v1:172.21.15.121:6837/1135108804] conn(0x56434d57c750 0x56434d53c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66fa3d3740 4 client.6945 mds_command: new command op to 6804 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:13:22.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.667+0000 7f66fa3d3740 1 -- 172.21.15.69:0/717709021 --> [v2:172.21.15.121:6835/1135108804,v1:172.21.15.121:6837/1135108804] -- command(tid 0: {"prefix": "status"}) -- 0x56434d401860 con 0x56434d57c750 2024-04-24T08:13:22.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.668+0000 7f66f4a00640 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:6835/1135108804,v1:172.21.15.121:6837/1135108804] conn(0x56434d57c750 0x56434d53c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.669+0000 7f66f4a00640 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:6835/1135108804,v1:172.21.15.121:6837/1135108804] conn(0x56434d57c750 0x56434d53c720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51caafc95c8c473e server_cookie=e273f13d49f9ad0d in_seq=0 out_seq=0 2024-04-24T08:13:22.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.669+0000 7f66d5600640 10 client.6945 ms_handle_connect on v2:172.21.15.121:6835/1135108804 2024-04-24T08:13:22.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.669+0000 7f66d5600640 1 -- 172.21.15.69:0/717709021 <== mds.0 v2:172.21.15.121:6835/1135108804 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x56434d401860 con 0x56434d57c750 2024-04-24T08:13:22.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.669+0000 7f66d5600640 10 client.6945 handle_command_reply: tid=0 2024-04-24T08:13:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.669+0000 7f66fa3d3740 1 client.6945 shutdown 2024-04-24T08:13:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 1 -- 172.21.15.69:0/717709021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 msgr2=0x56434d556af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d556af0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f66f00b34d0 tx=0x7f66dc001510 comp rx=0 tx=0).stop 2024-04-24T08:13:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 1 -- 172.21.15.69:0/717709021 shutdown_connections 2024-04-24T08:13:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:6835/1135108804,v1:172.21.15.121:6837/1135108804] conn(0x56434d57c750 0x56434d53c720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56434d560980 0x56434d556af0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56434d56d0f0 0x56434d539160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66d5600640 0 client.6945 ms_handle_reset on v2:172.21.15.121:6835/1135108804 2024-04-24T08:13:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 1 --2- 172.21.15.69:0/717709021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56434d543aa0 0x56434d5c27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 1 -- 172.21.15.69:0/717709021 >> 172.21.15.69:0/717709021 conn(0x56434d543360 msgr2=0x56434d55ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:22.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 1 -- 172.21.15.69:0/717709021 shutdown_connections 2024-04-24T08:13:22.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 1 -- 172.21.15.69:0/717709021 wait complete. 2024-04-24T08:13:22.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.670+0000 7f66fa3d3740 20 client.6945 trim_cache size 0 max 16384 2024-04-24T08:13:22.689 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:13:22.689 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:13:22.689 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:13:22.689 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 6804, 2024-04-24T08:13:22.689 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:13:22.689 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:13:22.690 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:13:22.690 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 6.5844379809999998, 2024-04-24T08:13:22.690 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 44, 2024-04-24T08:13:22.690 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 68, 2024-04-24T08:13:22.690 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 68, 2024-04-24T08:13:22.690 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 12.050652044 2024-04-24T08:13:22.690 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 -- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66d006c8f0 msgr2=0x7f66d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66d006c8f0 0x7f66d006edb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f66e4034000 tx=0x7f66e402e000 comp rx=0 tx=0).stop 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 -- 172.21.15.69:0/2655242641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 msgr2=0x7f66f0144ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 0x7f66f0144ee0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f66e800b9e0 tx=0x7f66e800bea0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 -- 172.21.15.69:0/2655242641 shutdown_connections 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66d006c8f0 0x7f66d006edb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f0156c60 0x7f66f0144ee0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00b30f0 0x7f66f01449a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 --2- 172.21.15.69:0/2655242641 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66f00ad7e0 0x7f66f014bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 -- 172.21.15.69:0/2655242641 >> 172.21.15.69:0/2655242641 conn(0x7f66f00d0160 msgr2=0x7f66f006ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:22.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 -- 172.21.15.69:0/2655242641 shutdown_connections 2024-04-24T08:13:22.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.671+0000 7f66f5e00640 1 -- 172.21.15.69:0/2655242641 wait complete. 2024-04-24T08:13:22.693 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:13:22.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.873+0000 7fc92be00640 1 Processor -- start 2024-04-24T08:13:22.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.873+0000 7fc92be00640 1 -- start start 2024-04-24T08:13:22.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.874+0000 7fc92be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc9240d0300 0x7fc92406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.874+0000 7fc92be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc924145a90 0x7fc9240d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.874+0000 7fc92be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 0x7fc924160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.874+0000 7fc92be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc9240d0c20 con 0x7fc924145a90 2024-04-24T08:13:22.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.874+0000 7fc92be00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc9240cf3d0 con 0x7fc9240d0300 2024-04-24T08:13:22.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc92be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc9240cf6b0 con 0x7fc924145fe0 2024-04-24T08:13:22.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc928a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc924145a90 0x7fc9240d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc928a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc924145a90 0x7fc9240d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41630/0 (socket says 172.21.15.69:41630) 2024-04-24T08:13:22.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc928a00640 1 -- 172.21.15.69:0/1127637484 learned_addr learned my addr 172.21.15.69:0/1127637484 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:22.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc929e00640 1 --2- 172.21.15.69:0/1127637484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 0x7fc924160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc929400640 1 --2- 172.21.15.69:0/1127637484 >> v2:172.21.15.121:3300/0 conn(0x7fc9240d0300 0x7fc92406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc928a00640 1 -- 172.21.15.69:0/1127637484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 msgr2=0x7fc924160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc928a00640 1 --2- 172.21.15.69:0/1127637484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 0x7fc924160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc928a00640 1 -- 172.21.15.69:0/1127637484 >> v2:172.21.15.121:3300/0 conn(0x7fc9240d0300 msgr2=0x7fc92406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc928a00640 1 --2- 172.21.15.69:0/1127637484 >> v2:172.21.15.121:3300/0 conn(0x7fc9240d0300 0x7fc92406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc928a00640 1 -- 172.21.15.69:0/1127637484 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9240cf990 con 0x7fc924145a90 2024-04-24T08:13:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc929e00640 1 --2- 172.21.15.69:0/1127637484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 0x7fc924160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:22.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.875+0000 7fc929400640 1 --2- 172.21.15.69:0/1127637484 >> v2:172.21.15.121:3300/0 conn(0x7fc9240d0300 0x7fc92406e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:22.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.877+0000 7fc928a00640 1 --2- 172.21.15.69:0/1127637484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc924145a90 0x7fc9240d06e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc9240b59e0 tx=0x7fc91002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7b99817188c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:22.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.877+0000 7fc923e00640 1 -- 172.21.15.69:0/1127637484 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc910037070 con 0x7fc924145a90 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.877+0000 7fc923e00640 1 -- 172.21.15.69:0/1127637484 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc910032440 con 0x7fc924145a90 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.877+0000 7fc923e00640 1 -- 172.21.15.69:0/1127637484 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc910032810 con 0x7fc924145a90 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.878+0000 7fc92be00640 1 -- 172.21.15.69:0/1127637484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc924145a90 msgr2=0x7fc9240d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.878+0000 7fc92be00640 1 --2- 172.21.15.69:0/1127637484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc924145a90 0x7fc9240d06e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc9240b59e0 tx=0x7fc91002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.878+0000 7fc92be00640 1 -- 172.21.15.69:0/1127637484 shutdown_connections 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.878+0000 7fc92be00640 1 --2- 172.21.15.69:0/1127637484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 0x7fc924160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.878+0000 7fc92be00640 1 --2- 172.21.15.69:0/1127637484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc924145a90 0x7fc9240d06e0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.878+0000 7fc92be00640 1 --2- 172.21.15.69:0/1127637484 >> v2:172.21.15.121:3300/0 conn(0x7fc9240d0300 0x7fc92406e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.878+0000 7fc92be00640 1 -- 172.21.15.69:0/1127637484 >> 172.21.15.69:0/1127637484 conn(0x7fc924067d90 msgr2=0x7fc92406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.878+0000 7fc92be00640 1 -- 172.21.15.69:0/1127637484 shutdown_connections 2024-04-24T08:13:22.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.878+0000 7fc92be00640 1 -- 172.21.15.69:0/1127637484 wait complete. 2024-04-24T08:13:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.880+0000 7fc92be00640 1 Processor -- start 2024-04-24T08:13:22.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.880+0000 7fc92be00640 1 -- start start 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.881+0000 7fc92be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9240d0300 0x7fc924143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.881+0000 7fc92be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc924145a90 0x7fc9241e83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.881+0000 7fc92be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 0x7fc9241e8900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.881+0000 7fc92be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc924145770 con 0x7fc9240d0300 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.881+0000 7fc92be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc9240cff50 con 0x7fc924145a90 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.881+0000 7fc92be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc9241458f0 con 0x7fc924145fe0 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.881+0000 7fc929400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9240d0300 0x7fc924143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.881+0000 7fc929400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9240d0300 0x7fc924143730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41644/0 (socket says 172.21.15.69:41644) 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.881+0000 7fc929400640 1 -- 172.21.15.69:0/3638584305 learned_addr learned my addr 172.21.15.69:0/3638584305 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc929e00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 0x7fc9241e8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc929400640 1 -- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 msgr2=0x7fc9241e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc928a00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc924145a90 0x7fc9241e83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc929400640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 0x7fc9241e8900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc929400640 1 -- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc924145a90 msgr2=0x7fc9241e83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc929400640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc924145a90 0x7fc9241e83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc929400640 1 -- 172.21.15.69:0/3638584305 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92415e660 con 0x7fc9240d0300 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc929e00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 0x7fc9241e8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc929400640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9240d0300 0x7fc924143730 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc91800c7f0 tx=0x7fc91800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc921600640 1 -- 172.21.15.69:0/3638584305 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918015070 con 0x7fc9240d0300 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc921600640 1 -- 172.21.15.69:0/3638584305 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc918007820 con 0x7fc9240d0300 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc921600640 1 -- 172.21.15.69:0/3638584305 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918007c10 con 0x7fc9240d0300 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.882+0000 7fc92be00640 1 -- 172.21.15.69:0/3638584305 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc92415e940 con 0x7fc9240d0300 2024-04-24T08:13:22.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.883+0000 7fc92be00640 1 -- 172.21.15.69:0/3638584305 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92415ef00 con 0x7fc9240d0300 2024-04-24T08:13:22.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.884+0000 7fc92be00640 1 -- 172.21.15.69:0/3638584305 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc924068a80 con 0x7fc9240d0300 2024-04-24T08:13:22.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.885+0000 7fc921600640 1 -- 172.21.15.69:0/3638584305 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc918018070 con 0x7fc9240d0300 2024-04-24T08:13:22.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.885+0000 7fc921600640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90406c920 0x7fc90406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:22.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.886+0000 7fc921600640 1 -- 172.21.15.69:0/3638584305 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fc91808bd00 con 0x7fc9240d0300 2024-04-24T08:13:22.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.886+0000 7fc928a00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90406c920 0x7fc90406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:22.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.887+0000 7fc928a00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90406c920 0x7fc90406ede0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc910034000 tx=0x7fc91002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:22.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:22.890+0000 7fc921600640 1 -- 172.21.15.69:0/3638584305 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc91805a010 con 0x7fc9240d0300 2024-04-24T08:13:23.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.142+0000 7fc92be00640 1 -- 172.21.15.69:0/3638584305 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc9240d1120 con 0x7fc9240d0300 2024-04-24T08:13:23.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.143+0000 7fc921600640 1 -- 172.21.15.69:0/3638584305 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v44) ==== 53+0+73 (secure 0 0 0) 0x7fc918059910 con 0x7fc9240d0300 2024-04-24T08:13:23.143 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 -- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90406c920 msgr2=0x7fc90406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90406c920 0x7fc90406ede0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc910034000 tx=0x7fc91002e000 comp rx=0 tx=0).stop 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 -- 172.21.15.69:0/3638584305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9240d0300 msgr2=0x7fc924143730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9240d0300 0x7fc924143730 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fc91800c7f0 tx=0x7fc91800ccb0 comp rx=0 tx=0).stop 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 -- 172.21.15.69:0/3638584305 shutdown_connections 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90406c920 0x7fc90406ede0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc924145fe0 0x7fc9241e8900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc924145a90 0x7fc9241e83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 --2- 172.21.15.69:0/3638584305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9240d0300 0x7fc924143730 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 -- 172.21.15.69:0/3638584305 >> 172.21.15.69:0/3638584305 conn(0x7fc924067d90 msgr2=0x7fc92406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 -- 172.21.15.69:0/3638584305 shutdown_connections 2024-04-24T08:13:23.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23.146+0000 7fc92be00640 1 -- 172.21.15.69:0/3638584305 wait complete. 2024-04-24T08:13:23.165 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:13:23.165 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:13:23.165 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:13:23.165 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:13:23.165 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:23.165 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:13:23.165 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:13:23.165 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:13:23.165 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:13:23.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:23.166 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:13:23.213 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:13:23.213 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:13:23.214 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42376sec preferred_lft 42376sec 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:13:23.215 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:23.216 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:13:23.216 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:13:23.216 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:13:23.216 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:23.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:23.216 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:13:23.337 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:13:23.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:23.341 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:13:23.392 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:13:23.392 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:13:23.392 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:13:23.392 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:13:23.392 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:13:23.392 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:13:23.393 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:13:23.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:13:23.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:13:23.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:13:23.393 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:13:23.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:13:23.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:13:23.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:23.596 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:13:23.642 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:13:23.642 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:13:23.643 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:13:23.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:23.643 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:13:23.723 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:13:23.723 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:13:23.723 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:23.775 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:13:23.775 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:13:23.776 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:23.826 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:13:23.910 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:13:23.965 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:13:23.966 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:13:23.967 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:13:23.967 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:13:23.967 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:13:23.967 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:13:23.967 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:13:23.967 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:13:23.967 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:13:23.968 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:13:23.968 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:13:23.968 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:13:23.968 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:13:23.968 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:13:23.968 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:13:23.968 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:13:23.968 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:13:23.969 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:13:23.969 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:13:23.969 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:13:23.969 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:13:23.969 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:13:23.969 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:13:23.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:23.970 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:13:24.020 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:13:24.020 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:13:24.025 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:13:24.091 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:13:24.133 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:13:24.134 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:13:24.134 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:13:24.134 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:13:24.134 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:13:24.134 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:13:24.135 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:13:24.136 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:13:24.136 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:13:24.136 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:13:24.136 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:13:24.136 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:13:24.136 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:13:24.136 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:13:24.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:24.137 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:13:24.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:13:24.190+0000 7fbcd7a0b480 -1 init, newargv = 0x55ae5a67b310 newargc=15 2024-04-24T08:13:24.190 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[57597]: starting ceph client 2024-04-24T08:13:24.199 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[57597]: starting fuse 2024-04-24T08:13:24.220 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:13:24.220 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:13:24.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:13:24.220 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:13:24.221 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:13:24.222 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:13:24.222 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:13:24.222 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:13:24.222 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:13:24.222 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:13:24.222 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:13:24.222 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:24.222 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:13:24.222 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:13:24.410 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.57597.asok 2024-04-24T08:13:24.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:13:24.427 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.57597.asok 2024-04-24T08:13:24.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:24.427 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.57597.asok status 2024-04-24T08:13:24.654 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:13:24.654 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:13:24.655 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:13:24.655 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:13:24.655 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:13:24.655 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:13:24.655 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:13:24.655 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "57597", 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 6960, 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 6960 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1911078551 2024-04-24T08:13:24.656 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1911078551 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.6960 192.168.0.1:0/1911078551", 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/1911078551", 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 44, 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 68, 2024-04-24T08:13:24.657 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:13:24.658 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:13:24.658 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:13:24.658 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:13:24.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:24.669 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:13:24.669 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:24.722 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:13:24.722 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:24.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:24.722 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:24.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:24.804 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/) 2024-04-24T08:13:24.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:24.873 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_undamaged) 2024-04-24T08:13:24.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:24.932 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_to_be_damaged) 2024-04-24T08:13:24.986 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:13:24.986 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:13:24.987 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:13:24.987 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:24.987 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-04-24T08:13:24.987 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:13:25.125 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627776 2024-04-24T08:13:25.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:13:25.139 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:13:25.140 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:13:25.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:25.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.433+0000 7fe68b200640 1 Processor -- start 2024-04-24T08:13:25.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.434+0000 7fe68b200640 1 -- start start 2024-04-24T08:13:25.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.435+0000 7fe68b200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe68c156c60 0x7fe68c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.435+0000 7fe68b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68c15d660 con 0x7fe68c0b30f0 2024-04-24T08:13:25.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.435+0000 7fe68b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe68c06dc30 con 0x7fe68c0ad7e0 2024-04-24T08:13:25.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.435+0000 7fe68b200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe68c06df10 con 0x7fe68c156c60 2024-04-24T08:13:25.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.436+0000 7fe689400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe68c0b30f0 0x7fe68c0ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39042/0 (socket says 172.21.15.69:39042) 2024-04-24T08:13:25.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.436+0000 7fe689400640 1 -- 172.21.15.69:0/3740119470 learned_addr learned my addr 172.21.15.69:0/3740119470 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.436+0000 7fe689e00640 1 --2- 172.21.15.69:0/3740119470 >> v2:172.21.15.121:3300/0 conn(0x7fe68c156c60 0x7fe68c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.436+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2692153089 0 0) 0x7fe68c15d660 con 0x7fe68c0b30f0 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.436+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe670003520 con 0x7fe68c0b30f0 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.436+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2330926336 0 0) 0x7fe670003520 con 0x7fe68c0b30f0 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.436+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 >> v2:172.21.15.121:3300/0 conn(0x7fe68c156c60 msgr2=0x7fe68c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.436+0000 7fe688a00640 1 --2- 172.21.15.69:0/3740119470 >> v2:172.21.15.121:3300/0 conn(0x7fe68c156c60 0x7fe68c15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.436+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 >> v1:172.21.15.121:6789/0 conn(0x7fe68c0ad7e0 legacy=0x7fe68c06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.436+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c06e1f0 con 0x7fe68c0b30f0 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe674001670 con 0x7fe68c0b30f0 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 >> v1:172.21.15.69:6789/0 conn(0x7fe68c0b30f0 legacy=0x7fe68c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe688a00640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6700039b0 0x7fe670003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe688a00640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe670004800 0x7fe670004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe688a00640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700051c0 0x7fe67000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c06df10 con 0x7fe670004800 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe67000c0f0 con 0x7fe6700051c0 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe67000c2f0 con 0x7fe6700039b0 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe689400640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe670004800 0x7fe670004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe68a800640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700051c0 0x7fe67000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe689400640 1 -- 172.21.15.69:0/3740119470 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6700039b0 msgr2=0x7fe670003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe689400640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6700039b0 0x7fe670003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe689400640 1 -- 172.21.15.69:0/3740119470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700051c0 msgr2=0x7fe67000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe689400640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700051c0 0x7fe67000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe689400640 1 -- 172.21.15.69:0/3740119470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe670003520 con 0x7fe670004800 2024-04-24T08:13:25.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe68a800640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700051c0 0x7fe67000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:25.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.437+0000 7fe68a800640 1 -- 172.21.15.69:0/3740119470 reap_dead start 2024-04-24T08:13:25.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.438+0000 7fe689400640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe670004800 0x7fe670004c80 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0b34d0 tx=0x7fe670005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0100396f6b53cd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:25.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.439+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674007d30 con 0x7fe670004800 2024-04-24T08:13:25.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.439+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe674005af0 con 0x7fe670004800 2024-04-24T08:13:25.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.439+0000 7fe688a00640 1 -- 172.21.15.69:0/3740119470 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674007070 con 0x7fe670004800 2024-04-24T08:13:25.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.439+0000 7fe68b200640 1 -- 172.21.15.69:0/3740119470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe670004800 msgr2=0x7fe670004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.439+0000 7fe68b200640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe670004800 0x7fe670004c80 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fe68c0b34d0 tx=0x7fe670005830 comp rx=0 tx=0).stop 2024-04-24T08:13:25.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.439+0000 7fe68b200640 1 -- 172.21.15.69:0/3740119470 shutdown_connections 2024-04-24T08:13:25.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.439+0000 7fe68b200640 1 --2- 172.21.15.69:0/3740119470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe670004800 0x7fe670004c80 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.439+0000 7fe68b200640 1 -- 172.21.15.69:0/3740119470 >> 172.21.15.69:0/3740119470 conn(0x7fe68c0d0160 msgr2=0x7fe68c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:25.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.439+0000 7fe68b200640 1 -- 172.21.15.69:0/3740119470 shutdown_connections 2024-04-24T08:13:25.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.440+0000 7fe68b200640 1 -- 172.21.15.69:0/3740119470 wait complete. 2024-04-24T08:13:25.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.441+0000 7fe68b200640 1 Processor -- start 2024-04-24T08:13:25.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.441+0000 7fe68b200640 1 -- start start 2024-04-24T08:13:25.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.442+0000 7fe68b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe670004800 0x7fe68c1594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.442+0000 7fe68b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe68c1432c0 0x7fe68c152c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.442+0000 7fe68b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe68c15ec40 0x7fe68c15f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.442+0000 7fe68b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c0ad0d0 con 0x7fe68c1432c0 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.442+0000 7fe68b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe68c15d940 con 0x7fe670004800 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.442+0000 7fe68b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe68c0ad250 con 0x7fe68c15ec40 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.443+0000 7fe689400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe68c1432c0 0x7fe68c152c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.443+0000 7fe689400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe68c1432c0 0x7fe68c152c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41666/0 (socket says 172.21.15.69:41666) 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.443+0000 7fe689400640 1 -- 172.21.15.69:0/3433380540 learned_addr learned my addr 172.21.15.69:0/3433380540 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.443+0000 7fe689e00640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe670004800 0x7fe68c1594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.443+0000 7fe689400640 1 -- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe68c15ec40 msgr2=0x7fe68c15f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.443+0000 7fe689400640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe68c15ec40 0x7fe68c15f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.443+0000 7fe689400640 1 -- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe670004800 msgr2=0x7fe68c1594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.443+0000 7fe689400640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe670004800 0x7fe68c1594b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.443+0000 7fe689400640 1 -- 172.21.15.69:0/3433380540 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe68c072bb0 con 0x7fe68c1432c0 2024-04-24T08:13:25.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.443+0000 7fe689400640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe68c1432c0 0x7fe68c152c90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe67402add0 tx=0x7fe68c15d830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:25.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.444+0000 7fe682000640 1 -- 172.21.15.69:0/3433380540 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674037040 con 0x7fe68c1432c0 2024-04-24T08:13:25.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.444+0000 7fe682000640 1 -- 172.21.15.69:0/3433380540 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe674001430 con 0x7fe68c1432c0 2024-04-24T08:13:25.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.444+0000 7fe682000640 1 -- 172.21.15.69:0/3433380540 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe674007d30 con 0x7fe68c1432c0 2024-04-24T08:13:25.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.444+0000 7fe68b200640 1 -- 172.21.15.69:0/3433380540 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe68c071df0 con 0x7fe68c1432c0 2024-04-24T08:13:25.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.444+0000 7fe68b200640 1 -- 172.21.15.69:0/3433380540 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe68c0723b0 con 0x7fe68c1432c0 2024-04-24T08:13:25.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.445+0000 7fe68b200640 1 -- 172.21.15.69:0/3433380540 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe68c075d50 con 0x7fe68c1432c0 2024-04-24T08:13:25.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.446+0000 7fe682000640 1 -- 172.21.15.69:0/3433380540 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe674043020 con 0x7fe68c1432c0 2024-04-24T08:13:25.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.447+0000 7fe682000640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe66806c830 0x7fe66806ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.447+0000 7fe682000640 1 -- 172.21.15.69:0/3433380540 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fe6740ab430 con 0x7fe68c1432c0 2024-04-24T08:13:25.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.448+0000 7fe689e00640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe66806c830 0x7fe66806ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.449+0000 7fe689e00640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe66806c830 0x7fe66806ecf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe67c006a90 tx=0x7fe67c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:25.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.451+0000 7fe682000640 1 -- 172.21.15.69:0/3433380540 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe674079730 con 0x7fe68c1432c0 2024-04-24T08:13:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.651+0000 7fe68b200640 1 -- 172.21.15.69:0/3433380540 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe68c06f9e0 con 0x7fe68c1432c0 2024-04-24T08:13:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.654+0000 7fe682000640 1 -- 172.21.15.69:0/3433380540 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+16644 (secure 0 0 0) 0x7fe674079030 con 0x7fe68c1432c0 2024-04-24T08:13:25.656 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:25.656 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":68,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:13:17.084219+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:25.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.657+0000 7fe68b200640 1 -- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe66806c830 msgr2=0x7fe66806ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.657+0000 7fe68b200640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe66806c830 0x7fe66806ecf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe67c006a90 tx=0x7fe67c008040 comp rx=0 tx=0).stop 2024-04-24T08:13:25.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.657+0000 7fe68b200640 1 -- 172.21.15.69:0/3433380540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe68c1432c0 msgr2=0x7fe68c152c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.657+0000 7fe68b200640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe68c1432c0 0x7fe68c152c90 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe67402add0 tx=0x7fe68c15d830 comp rx=0 tx=0).stop 2024-04-24T08:13:25.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.658+0000 7fe68b200640 1 -- 172.21.15.69:0/3433380540 shutdown_connections 2024-04-24T08:13:25.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.658+0000 7fe68b200640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe66806c830 0x7fe66806ecf0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.658+0000 7fe68b200640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe68c1432c0 0x7fe68c152c90 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.658+0000 7fe68b200640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe68c15ec40 0x7fe68c15f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.658+0000 7fe68b200640 1 --2- 172.21.15.69:0/3433380540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe670004800 0x7fe68c1594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.658+0000 7fe68b200640 1 -- 172.21.15.69:0/3433380540 >> 172.21.15.69:0/3433380540 conn(0x7fe68c0d0160 msgr2=0x7fe68c06ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:25.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.658+0000 7fe68b200640 1 -- 172.21.15.69:0/3433380540 shutdown_connections 2024-04-24T08:13:25.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.658+0000 7fe68b200640 1 -- 172.21.15.69:0/3433380540 wait complete. 2024-04-24T08:13:25.677 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:13:13.250209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:13:14.468238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:13:25.677 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:13:25.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.872+0000 7f6030a00640 1 Processor -- start 2024-04-24T08:13:25.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.872+0000 7f6030a00640 1 -- start start 2024-04-24T08:13:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.874+0000 7f6030a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f602c0d09f0 0x7f602c166020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.874+0000 7f6030a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f602c0ce640 con 0x7f602c0b5300 2024-04-24T08:13:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.874+0000 7f6030a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f602c0ce910 con 0x7f602c15ea60 2024-04-24T08:13:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.874+0000 7f6030a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f602c0cebf0 con 0x7f602c0d09f0 2024-04-24T08:13:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.874+0000 7f6029e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f602c0b5300 0x7f602c0d02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39044/0 (socket says 172.21.15.69:39044) 2024-04-24T08:13:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.874+0000 7f6029e00640 1 -- 172.21.15.69:0/326683442 learned_addr learned my addr 172.21.15.69:0/326683442 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.874+0000 7f602b200640 1 --2- 172.21.15.69:0/326683442 >> v2:172.21.15.121:3300/0 conn(0x7f602c0d09f0 0x7f602c166020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.874+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2180481057 0 0) 0x7f602c0ce640 con 0x7f602c0b5300 2024-04-24T08:13:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6010003520 con 0x7f602c0b5300 2024-04-24T08:13:25.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1168842339 0 0) 0x7f6010003520 con 0x7f602c0b5300 2024-04-24T08:13:25.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 >> v2:172.21.15.121:3300/0 conn(0x7f602c0d09f0 msgr2=0x7f602c166020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:25.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 --2- 172.21.15.69:0/326683442 >> v2:172.21.15.121:3300/0 conn(0x7f602c0d09f0 0x7f602c166020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 >> v1:172.21.15.121:6789/0 conn(0x7f602c15ea60 legacy=0x7f602c0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602c0ceed0 con 0x7f602c0b5300 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f6018001670 con 0x7f602c0b5300 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 >> v1:172.21.15.69:6789/0 conn(0x7f602c0b5300 legacy=0x7f602c0d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60100039b0 0x7f6010003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6010004800 0x7f6010004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f601000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f602c0cebf0 con 0x7f60100051c0 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f601000e270 con 0x7f60100039b0 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f601000e470 con 0x7f6010004800 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.875+0000 7f602b200640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f601000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.876+0000 7f6029e00640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6010004800 0x7f6010004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.876+0000 7f602a800640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60100039b0 0x7f6010003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.876+0000 7f602b200640 1 -- 172.21.15.69:0/326683442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6010004800 msgr2=0x7f6010004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.876+0000 7f602b200640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6010004800 0x7f6010004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.876+0000 7f602b200640 1 -- 172.21.15.69:0/326683442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60100039b0 msgr2=0x7f6010003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.876+0000 7f602b200640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60100039b0 0x7f6010003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.876+0000 7f602b200640 1 -- 172.21.15.69:0/326683442 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6010003520 con 0x7f60100051c0 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.876+0000 7f602a800640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60100039b0 0x7f6010003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.877+0000 7f602b200640 1 -- 172.21.15.69:0/326683442 reap_dead start 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.877+0000 7f602b200640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f601000dbd0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f6024009fc0 tx=0x7f6010005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53f0a9b26a928e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.877+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024035020 con 0x7f60100051c0 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.877+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6024003bd0 con 0x7f60100051c0 2024-04-24T08:13:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.877+0000 7f6029400640 1 -- 172.21.15.69:0/326683442 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6024009c10 con 0x7f60100051c0 2024-04-24T08:13:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.878+0000 7f6030a00640 1 -- 172.21.15.69:0/326683442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 msgr2=0x7f601000dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.878+0000 7f6030a00640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f601000dbd0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f6024009fc0 tx=0x7f6010005830 comp rx=0 tx=0).stop 2024-04-24T08:13:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.878+0000 7f6030a00640 1 -- 172.21.15.69:0/326683442 shutdown_connections 2024-04-24T08:13:25.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.878+0000 7f6030a00640 1 --2- 172.21.15.69:0/326683442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f601000dbd0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.878+0000 7f6030a00640 1 -- 172.21.15.69:0/326683442 >> 172.21.15.69:0/326683442 conn(0x7f602c067b80 msgr2=0x7f602c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:25.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.878+0000 7f6030a00640 1 -- 172.21.15.69:0/326683442 shutdown_connections 2024-04-24T08:13:25.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.878+0000 7f6030a00640 1 -- 172.21.15.69:0/326683442 wait complete. 2024-04-24T08:13:25.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.880+0000 7f6030a00640 1 Processor -- start 2024-04-24T08:13:25.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.880+0000 7f6030a00640 1 -- start start 2024-04-24T08:13:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f6030a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f602c1e3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f6030a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f602c1ede40 0x7f602c1ee2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f6030a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f602c15c0e0 0x7f602c15c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f6030a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f602c0d02e0 con 0x7f60100051c0 2024-04-24T08:13:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f6030a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f602c0cf490 con 0x7f602c15c0e0 2024-04-24T08:13:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f6030a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f602c0d0460 con 0x7f602c1ede40 2024-04-24T08:13:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f602a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f602c1e3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f602a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f602c1e3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41680/0 (socket says 172.21.15.69:41680) 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f602a800640 1 -- 172.21.15.69:0/1439170193 learned_addr learned my addr 172.21.15.69:0/1439170193 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f602a800640 1 -- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f602c1ede40 msgr2=0x7f602c1ee2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f602a800640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f602c1ede40 0x7f602c1ee2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f602a800640 1 -- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f602c15c0e0 msgr2=0x7f602c15c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.881+0000 7f602a800640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f602c15c0e0 0x7f602c15c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.882+0000 7f602a800640 1 -- 172.21.15.69:0/1439170193 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f602c1ee800 con 0x7f60100051c0 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.882+0000 7f602a800640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f602c1e3dc0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f601c000f60 tx=0x7f601c00d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.882+0000 7f6022a00640 1 -- 172.21.15.69:0/1439170193 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c005480 con 0x7f60100051c0 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.882+0000 7f6022a00640 1 -- 172.21.15.69:0/1439170193 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f601c017070 con 0x7f60100051c0 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.882+0000 7f6022a00640 1 -- 172.21.15.69:0/1439170193 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c003810 con 0x7f60100051c0 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.882+0000 7f6030a00640 1 -- 172.21.15.69:0/1439170193 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f602c15ce10 con 0x7f60100051c0 2024-04-24T08:13:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.882+0000 7f6030a00640 1 -- 172.21.15.69:0/1439170193 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f602c15d3d0 con 0x7f60100051c0 2024-04-24T08:13:25.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.883+0000 7f6030a00640 1 -- 172.21.15.69:0/1439170193 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f602c1455b0 con 0x7f60100051c0 2024-04-24T08:13:25.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.884+0000 7f6022a00640 1 -- 172.21.15.69:0/1439170193 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f601c0039b0 con 0x7f60100051c0 2024-04-24T08:13:25.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.885+0000 7f6022a00640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f600c06c8f0 0x7f600c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:25.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.885+0000 7f6022a00640 1 -- 172.21.15.69:0/1439170193 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f601c08ba00 con 0x7f60100051c0 2024-04-24T08:13:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.886+0000 7f6029e00640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f600c06c8f0 0x7f600c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:25.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.887+0000 7f6029e00640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f600c06c8f0 0x7f600c06edb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f60180017e0 tx=0x7f6018009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:25.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:25.890+0000 7f6022a00640 1 -- 172.21.15.69:0/1439170193 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f601c059d90 con 0x7f60100051c0 2024-04-24T08:13:26.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.089+0000 7f6030a00640 1 -- 172.21.15.69:0/1439170193 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f602c075d50 con 0x7f60100051c0 2024-04-24T08:13:26.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.090+0000 7f6022a00640 1 -- 172.21.15.69:0/1439170193 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) ==== 93+0+10 (secure 0 0 0) 0x7f601c059690 con 0x7f60100051c0 2024-04-24T08:13:26.090 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:13:26.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.092+0000 7f6030a00640 1 -- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f600c06c8f0 msgr2=0x7f600c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.092+0000 7f6030a00640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f600c06c8f0 0x7f600c06edb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f60180017e0 tx=0x7f6018009040 comp rx=0 tx=0).stop 2024-04-24T08:13:26.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.092+0000 7f6030a00640 1 -- 172.21.15.69:0/1439170193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 msgr2=0x7f602c1e3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.092+0000 7f6030a00640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f602c1e3dc0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f601c000f60 tx=0x7f601c00d9f0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.092+0000 7f6030a00640 1 -- 172.21.15.69:0/1439170193 shutdown_connections 2024-04-24T08:13:26.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.092+0000 7f6030a00640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f600c06c8f0 0x7f600c06edb0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.092+0000 7f6030a00640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f602c15c0e0 0x7f602c15c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.092+0000 7f6030a00640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f602c1ede40 0x7f602c1ee2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.092+0000 7f6030a00640 1 --2- 172.21.15.69:0/1439170193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60100051c0 0x7f602c1e3dc0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.092+0000 7f6030a00640 1 -- 172.21.15.69:0/1439170193 >> 172.21.15.69:0/1439170193 conn(0x7f602c067b80 msgr2=0x7f602c0ad840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:26.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.093+0000 7f6030a00640 1 -- 172.21.15.69:0/1439170193 shutdown_connections 2024-04-24T08:13:26.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.093+0000 7f6030a00640 1 -- 172.21.15.69:0/1439170193 wait complete. 2024-04-24T08:13:26.111 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:13:26.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.309+0000 7f5c1a000640 1 Processor -- start 2024-04-24T08:13:26.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.309+0000 7f5c1a000640 1 -- start start 2024-04-24T08:13:26.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.310+0000 7f5c1a000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5c141420a0 0x7f5c14071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.311+0000 7f5c1a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5c141606e0 0x7f5c140cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.311+0000 7f5c1a000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c14072450 con 0x7f5c14142480 2024-04-24T08:13:26.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.311+0000 7f5c1a000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5c140cf1e0 con 0x7f5c141606e0 2024-04-24T08:13:26.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.311+0000 7f5c1a000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5c14072730 con 0x7f5c141420a0 2024-04-24T08:13:26.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.311+0000 7f5c19600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5c141606e0 0x7f5c140cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:26.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.311+0000 7f5c19600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5c141606e0 0x7f5c140cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41682/0 (socket says 172.21.15.69:41682) 2024-04-24T08:13:26.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.311+0000 7f5c13e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5c14142480 0x7f5c1415ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39052/0 (socket says 172.21.15.69:39052) 2024-04-24T08:13:26.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.311+0000 7f5c19600640 1 -- 172.21.15.69:0/416675980 learned_addr learned my addr 172.21.15.69:0/416675980 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:26.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.311+0000 7f5c18c00640 1 --2- 172.21.15.69:0/416675980 >> v2:172.21.15.121:3300/0 conn(0x7f5c141420a0 0x7f5c14071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:26.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.312+0000 7f5c19600640 1 -- 172.21.15.69:0/416675980 >> v2:172.21.15.121:3300/0 conn(0x7f5c141420a0 msgr2=0x7f5c14071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:26.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.312+0000 7f5c19600640 1 --2- 172.21.15.69:0/416675980 >> v2:172.21.15.121:3300/0 conn(0x7f5c141420a0 0x7f5c14071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.312+0000 7f5c19600640 1 -- 172.21.15.69:0/416675980 >> v1:172.21.15.69:6789/0 conn(0x7f5c14142480 legacy=0x7f5c1415ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.312+0000 7f5c19600640 1 -- 172.21.15.69:0/416675980 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c14072a10 con 0x7f5c141606e0 2024-04-24T08:13:26.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.312+0000 7f5c18c00640 1 --2- 172.21.15.69:0/416675980 >> v2:172.21.15.121:3300/0 conn(0x7f5c141420a0 0x7f5c14071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:26.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.312+0000 7f5c13400640 1 -- 172.21.15.69:0/416675980 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085403096 0 0) 0x7f5c14072450 con 0x7f5c14142480 2024-04-24T08:13:26.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.313+0000 7f5c19600640 1 --2- 172.21.15.69:0/416675980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 0x7f5c140cf670 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5c140b5770 tx=0x7f5c0c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d26cde7e8ba58d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:26.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.313+0000 7f5c13400640 1 -- 172.21.15.69:0/416675980 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c036070 con 0x7f5c141606e0 2024-04-24T08:13:26.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.313+0000 7f5c13400640 1 -- 172.21.15.69:0/416675980 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c0c031600 con 0x7f5c141606e0 2024-04-24T08:13:26.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.313+0000 7f5c13400640 1 -- 172.21.15.69:0/416675980 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c0319d0 con 0x7f5c141606e0 2024-04-24T08:13:26.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.313+0000 7f5c1a000640 1 -- 172.21.15.69:0/416675980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 msgr2=0x7f5c140cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.313+0000 7f5c1a000640 1 --2- 172.21.15.69:0/416675980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 0x7f5c140cf670 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5c140b5770 tx=0x7f5c0c02a730 comp rx=0 tx=0).stop 2024-04-24T08:13:26.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.314+0000 7f5c1a000640 1 -- 172.21.15.69:0/416675980 shutdown_connections 2024-04-24T08:13:26.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.314+0000 7f5c1a000640 1 --2- 172.21.15.69:0/416675980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 0x7f5c140cf670 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.314+0000 7f5c1a000640 1 --2- 172.21.15.69:0/416675980 >> v2:172.21.15.121:3300/0 conn(0x7f5c141420a0 0x7f5c14071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.314+0000 7f5c1a000640 1 -- 172.21.15.69:0/416675980 >> 172.21.15.69:0/416675980 conn(0x7f5c14067b80 msgr2=0x7f5c1406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:26.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.314+0000 7f5c1a000640 1 -- 172.21.15.69:0/416675980 shutdown_connections 2024-04-24T08:13:26.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.314+0000 7f5c1a000640 1 -- 172.21.15.69:0/416675980 wait complete. 2024-04-24T08:13:26.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.315+0000 7f5c1a000640 1 Processor -- start 2024-04-24T08:13:26.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.316+0000 7f5c1a000640 1 -- start start 2024-04-24T08:13:26.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.316+0000 7f5c1a000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c141420a0 0x7f5c1413a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.316+0000 7f5c1a000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c14142480 0x7f5c140b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.316+0000 7f5c1a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 0x7f5c1415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.316+0000 7f5c1a000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c140cfbb0 con 0x7f5c141606e0 2024-04-24T08:13:26.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.316+0000 7f5c1a000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5c14142910 con 0x7f5c14142480 2024-04-24T08:13:26.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.316+0000 7f5c1a000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5c140cfd30 con 0x7f5c141420a0 2024-04-24T08:13:26.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.317+0000 7f5c19600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 0x7f5c1415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:26.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.317+0000 7f5c19600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 0x7f5c1415b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41692/0 (socket says 172.21.15.69:41692) 2024-04-24T08:13:26.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.317+0000 7f5c19600640 1 -- 172.21.15.69:0/3404189563 learned_addr learned my addr 172.21.15.69:0/3404189563 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.317+0000 7f5c19600640 1 -- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c141420a0 msgr2=0x7f5c1413a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.317+0000 7f5c19600640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c141420a0 0x7f5c1413a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.317+0000 7f5c13e00640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c14142480 0x7f5c140b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.317+0000 7f5c19600640 1 -- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c14142480 msgr2=0x7f5c140b4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.317+0000 7f5c19600640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c14142480 0x7f5c140b4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.317+0000 7f5c19600640 1 -- 172.21.15.69:0/3404189563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c141f3f40 con 0x7f5c141606e0 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.318+0000 7f5c19600640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 0x7f5c1415b3f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009f40 tx=0x7f5c0c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.318+0000 7f5c10c00640 1 -- 172.21.15.69:0/3404189563 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c041020 con 0x7f5c141606e0 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.318+0000 7f5c10c00640 1 -- 172.21.15.69:0/3404189563 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c0c037040 con 0x7f5c141606e0 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.318+0000 7f5c1a000640 1 -- 172.21.15.69:0/3404189563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c140b3550 con 0x7f5c141606e0 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.318+0000 7f5c10c00640 1 -- 172.21.15.69:0/3404189563 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c036050 con 0x7f5c141606e0 2024-04-24T08:13:26.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.318+0000 7f5c1a000640 1 -- 172.21.15.69:0/3404189563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c140b3b10 con 0x7f5c141606e0 2024-04-24T08:13:26.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.319+0000 7f5c1a000640 1 -- 172.21.15.69:0/3404189563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c14075d50 con 0x7f5c141606e0 2024-04-24T08:13:26.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.320+0000 7f5c10c00640 1 -- 172.21.15.69:0/3404189563 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5c0c007c50 con 0x7f5c141606e0 2024-04-24T08:13:26.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.321+0000 7f5c10c00640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5bf406c8f0 0x7f5bf406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.321+0000 7f5c10c00640 1 -- 172.21.15.69:0/3404189563 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f5c0c0ac470 con 0x7f5c141606e0 2024-04-24T08:13:26.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.322+0000 7f5c18c00640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5bf406c8f0 0x7f5bf406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:26.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.323+0000 7f5c18c00640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5bf406c8f0 0x7f5bf406edb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5c08006fd0 tx=0x7f5c08008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:26.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.326+0000 7f5c10c00640 1 -- 172.21.15.69:0/3404189563 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c0c04b060 con 0x7f5c141606e0 2024-04-24T08:13:26.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.524+0000 7f5c1a000640 1 -- 172.21.15.69:0/3404189563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5c140b1d90 con 0x7f5c141606e0 2024-04-24T08:13:26.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.525+0000 7f5c10c00640 1 -- 172.21.15.69:0/3404189563 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v68) ==== 104+0+11 (secure 0 0 0) 0x7f5c0c07a890 con 0x7f5c141606e0 2024-04-24T08:13:26.525 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 -- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5bf406c8f0 msgr2=0x7f5bf406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5bf406c8f0 0x7f5bf406edb0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5c08006fd0 tx=0x7f5c08008040 comp rx=0 tx=0).stop 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 -- 172.21.15.69:0/3404189563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 msgr2=0x7f5c1415b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 0x7f5c1415b3f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009f40 tx=0x7f5c0c004a40 comp rx=0 tx=0).stop 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 -- 172.21.15.69:0/3404189563 shutdown_connections 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5bf406c8f0 0x7f5bf406edb0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c141606e0 0x7f5c1415b3f0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c14142480 0x7f5c140b4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 --2- 172.21.15.69:0/3404189563 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c141420a0 0x7f5c1413a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 -- 172.21.15.69:0/3404189563 >> 172.21.15.69:0/3404189563 conn(0x7f5c14067b80 msgr2=0x7f5c1415e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 -- 172.21.15.69:0/3404189563 shutdown_connections 2024-04-24T08:13:26.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.527+0000 7f5c1a000640 1 -- 172.21.15.69:0/3404189563 wait complete. 2024-04-24T08:13:26.546 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:13:26.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.751+0000 7f3950000640 1 Processor -- start 2024-04-24T08:13:26.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.752+0000 7f3950000640 1 -- start start 2024-04-24T08:13:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.753+0000 7f3950000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f394815c9b0 0x7f39480b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.753+0000 7f3950000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f39480b3f70 0x7f39480cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.753+0000 7f3950000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39480cd6d0 con 0x7f3948155f20 2024-04-24T08:13:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.753+0000 7f3950000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f394806db90 con 0x7f39480b3f70 2024-04-24T08:13:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.753+0000 7f3950000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3948072070 con 0x7f394815c9b0 2024-04-24T08:13:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.753+0000 7f394d600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3948155f20 0x7f39480ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39068/0 (socket says 172.21.15.69:39068) 2024-04-24T08:13:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.753+0000 7f394d600640 1 -- 172.21.15.69:0/430198075 learned_addr learned my addr 172.21.15.69:0/430198075 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:26.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.753+0000 7f394e000640 1 --2- 172.21.15.69:0/430198075 >> v2:172.21.15.69:3300/0 conn(0x7f39480b3f70 0x7f39480cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.754+0000 7f394cc00640 1 --2- 172.21.15.69:0/430198075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f394815c9b0 0x7f39480b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.754+0000 7f3947e00640 1 -- 172.21.15.69:0/430198075 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4255280136 0 0) 0x7f39480cd6d0 con 0x7f3948155f20 2024-04-24T08:13:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.754+0000 7f3947e00640 1 -- 172.21.15.69:0/430198075 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f392c005730 con 0x7f3948155f20 2024-04-24T08:13:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.754+0000 7f394e000640 1 -- 172.21.15.69:0/430198075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f394815c9b0 msgr2=0x7f39480b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.754+0000 7f394e000640 1 --2- 172.21.15.69:0/430198075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f394815c9b0 0x7f39480b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.754+0000 7f394e000640 1 -- 172.21.15.69:0/430198075 >> v1:172.21.15.69:6789/0 conn(0x7f3948155f20 legacy=0x7f39480ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.754+0000 7f394e000640 1 -- 172.21.15.69:0/430198075 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39480cd9b0 con 0x7f39480b3f70 2024-04-24T08:13:26.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.755+0000 7f394e000640 1 --2- 172.21.15.69:0/430198075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39480b3f70 0x7f39480cf360 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f39400097a0 tx=0x7f394002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bc7089741242a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:26.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.755+0000 7f3947e00640 1 -- 172.21.15.69:0/430198075 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940037070 con 0x7f39480b3f70 2024-04-24T08:13:26.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.755+0000 7f3947e00640 1 -- 172.21.15.69:0/430198075 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3940032440 con 0x7f39480b3f70 2024-04-24T08:13:26.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.755+0000 7f3947e00640 1 -- 172.21.15.69:0/430198075 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39400327f0 con 0x7f39480b3f70 2024-04-24T08:13:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.756+0000 7f3950000640 1 -- 172.21.15.69:0/430198075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39480b3f70 msgr2=0x7f39480cf360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.756+0000 7f3950000640 1 --2- 172.21.15.69:0/430198075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39480b3f70 0x7f39480cf360 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f39400097a0 tx=0x7f394002a210 comp rx=0 tx=0).stop 2024-04-24T08:13:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.756+0000 7f3950000640 1 -- 172.21.15.69:0/430198075 shutdown_connections 2024-04-24T08:13:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.756+0000 7f3950000640 1 --2- 172.21.15.69:0/430198075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39480b3f70 0x7f39480cf360 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.756+0000 7f3950000640 1 --2- 172.21.15.69:0/430198075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f394815c9b0 0x7f39480b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.756+0000 7f3950000640 1 -- 172.21.15.69:0/430198075 >> 172.21.15.69:0/430198075 conn(0x7f39480d0350 msgr2=0x7f39480d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:26.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.756+0000 7f3950000640 1 -- 172.21.15.69:0/430198075 shutdown_connections 2024-04-24T08:13:26.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.756+0000 7f3950000640 1 -- 172.21.15.69:0/430198075 wait complete. 2024-04-24T08:13:26.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.758+0000 7f3950000640 1 Processor -- start 2024-04-24T08:13:26.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.758+0000 7f3950000640 1 -- start start 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.759+0000 7f3950000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39480b3f70 0x7f394814e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.759+0000 7f3950000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3948155f20 0x7f39480adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.759+0000 7f3950000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f394815c9b0 0x7f39480ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.759+0000 7f3950000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39480cf8a0 con 0x7f3948155f20 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.759+0000 7f3950000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3948071d00 con 0x7f39480b3f70 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.759+0000 7f3950000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f39480cfa20 con 0x7f394815c9b0 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.759+0000 7f394cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3948155f20 0x7f39480adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.759+0000 7f394cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3948155f20 0x7f39480adbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41704/0 (socket says 172.21.15.69:41704) 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.759+0000 7f394cc00640 1 -- 172.21.15.69:0/3247005693 learned_addr learned my addr 172.21.15.69:0/3247005693 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f394d600640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39480b3f70 0x7f394814e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f394cc00640 1 -- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f394815c9b0 msgr2=0x7f39480ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f394cc00640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f394815c9b0 0x7f39480ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f394cc00640 1 -- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39480b3f70 msgr2=0x7f394814e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f394cc00640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39480b3f70 0x7f394814e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f394cc00640 1 -- 172.21.15.69:0/3247005693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f394814ee60 con 0x7f3948155f20 2024-04-24T08:13:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f394cc00640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3948155f20 0x7f39480adbc0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f3934007e60 tx=0x7f393400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f3945600640 1 -- 172.21.15.69:0/3247005693 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934014620 con 0x7f3948155f20 2024-04-24T08:13:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f3945600640 1 -- 172.21.15.69:0/3247005693 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3934019070 con 0x7f3948155f20 2024-04-24T08:13:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f3945600640 1 -- 172.21.15.69:0/3247005693 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3934014a00 con 0x7f3948155f20 2024-04-24T08:13:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.760+0000 7f3950000640 1 -- 172.21.15.69:0/3247005693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39481f3f20 con 0x7f3948155f20 2024-04-24T08:13:26.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.761+0000 7f3950000640 1 -- 172.21.15.69:0/3247005693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39481f44a0 con 0x7f3948155f20 2024-04-24T08:13:26.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.762+0000 7f3950000640 1 -- 172.21.15.69:0/3247005693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3948068a80 con 0x7f3948155f20 2024-04-24T08:13:26.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.763+0000 7f3945600640 1 -- 172.21.15.69:0/3247005693 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3934014ba0 con 0x7f3948155f20 2024-04-24T08:13:26.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.763+0000 7f3945600640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f392806c8f0 0x7f392806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:26.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.764+0000 7f3945600640 1 -- 172.21.15.69:0/3247005693 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f393408c470 con 0x7f3948155f20 2024-04-24T08:13:26.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.764+0000 7f394d600640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f392806c8f0 0x7f392806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:26.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.765+0000 7f394d600640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f392806c8f0 0x7f392806edb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3938000f60 tx=0x7f3938009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:26.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.768+0000 7f3945600640 1 -- 172.21.15.69:0/3247005693 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f394814ee60 con 0x7f3948155f20 2024-04-24T08:13:26.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.970+0000 7f3950000640 1 -- 172.21.15.69:0/3247005693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f39480cf680 con 0x7f3948155f20 2024-04-24T08:13:26.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.972+0000 7f3945600640 1 -- 172.21.15.69:0/3247005693 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v68) ==== 100+0+11 (secure 0 0 0) 0x7f393405a800 con 0x7f3948155f20 2024-04-24T08:13:26.972 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 -- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f392806c8f0 msgr2=0x7f392806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f392806c8f0 0x7f392806edb0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3938000f60 tx=0x7f3938009040 comp rx=0 tx=0).stop 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 -- 172.21.15.69:0/3247005693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3948155f20 msgr2=0x7f39480adbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3948155f20 0x7f39480adbc0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f3934007e60 tx=0x7f393400e4b0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 -- 172.21.15.69:0/3247005693 shutdown_connections 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f392806c8f0 0x7f392806edb0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3948155f20 0x7f39480adbc0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f394815c9b0 0x7f39480ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 --2- 172.21.15.69:0/3247005693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39480b3f70 0x7f394814e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 -- 172.21.15.69:0/3247005693 >> 172.21.15.69:0/3247005693 conn(0x7f39480d0350 msgr2=0x7f3948154ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:26.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.974+0000 7f3950000640 1 -- 172.21.15.69:0/3247005693 shutdown_connections 2024-04-24T08:13:26.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:26.975+0000 7f3950000640 1 -- 172.21.15.69:0/3247005693 wait complete. 2024-04-24T08:13:26.994 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:13:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.192+0000 7f9061200640 1 Processor -- start 2024-04-24T08:13:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.193+0000 7f9061200640 1 -- start start 2024-04-24T08:13:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.194+0000 7f9061200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f905c0d0200 0x7f905c0d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.194+0000 7f9061200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f905c15ed40 con 0x7f905c15e960 2024-04-24T08:13:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.194+0000 7f9061200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f905c0cf050 con 0x7f905c06cfb0 2024-04-24T08:13:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.194+0000 7f9061200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f905c162b50 con 0x7f905c0d0200 2024-04-24T08:13:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.194+0000 7f905b400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f905c15e960 0x7f905c162440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39078/0 (socket says 172.21.15.69:39078) 2024-04-24T08:13:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.194+0000 7f905b400640 1 -- 172.21.15.69:0/4227316099 learned_addr learned my addr 172.21.15.69:0/4227316099 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905be00640 1 --2- 172.21.15.69:0/4227316099 >> v2:172.21.15.121:3300/0 conn(0x7f905c0d0200 0x7f905c0d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3140580721 0 0) 0x7f905c15ed40 con 0x7f905c15e960 2024-04-24T08:13:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f903c003520 con 0x7f905c15e960 2024-04-24T08:13:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4132142002 0 0) 0x7f903c003520 con 0x7f905c15e960 2024-04-24T08:13:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 >> v2:172.21.15.121:3300/0 conn(0x7f905c0d0200 msgr2=0x7f905c0d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 --2- 172.21.15.69:0/4227316099 >> v2:172.21.15.121:3300/0 conn(0x7f905c0d0200 0x7f905c0d2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 >> v1:172.21.15.121:6789/0 conn(0x7f905c06cfb0 legacy=0x7f905c0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c162e20 con 0x7f905c15e960 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f904c001670 con 0x7f905c15e960 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 >> v1:172.21.15.69:6789/0 conn(0x7f905c15e960 legacy=0x7f905c162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f903c0039b0 0x7f903c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f903c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f903c0051c0 0x7f903c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f905c162b50 con 0x7f903c004800 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f903c00c060 con 0x7f903c0039b0 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.195+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f903c00c260 con 0x7f903c0051c0 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.196+0000 7f905aa00640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f903c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.196+0000 7f905b400640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f903c0039b0 0x7f903c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.196+0000 7f905be00640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f903c0051c0 0x7f903c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.196+0000 7f905aa00640 1 -- 172.21.15.69:0/4227316099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f903c0051c0 msgr2=0x7f903c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:27.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.196+0000 7f905aa00640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f903c0051c0 0x7f903c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.196+0000 7f905aa00640 1 -- 172.21.15.69:0/4227316099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f903c0039b0 msgr2=0x7f903c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.196+0000 7f905aa00640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f903c0039b0 0x7f903c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.196+0000 7f905aa00640 1 -- 172.21.15.69:0/4227316099 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903c003520 con 0x7f903c004800 2024-04-24T08:13:27.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.196+0000 7f905be00640 1 -- 172.21.15.69:0/4227316099 reap_dead start 2024-04-24T08:13:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.197+0000 7f905aa00640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f903c004c80 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9044001720 tx=0x7f903c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04d138b674bd173 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.197+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9044001370 con 0x7f903c004800 2024-04-24T08:13:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.198+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90440052d0 con 0x7f903c004800 2024-04-24T08:13:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.198+0000 7f905a000640 1 -- 172.21.15.69:0/4227316099 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904402ab90 con 0x7f903c004800 2024-04-24T08:13:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.198+0000 7f9061200640 1 -- 172.21.15.69:0/4227316099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 msgr2=0x7f903c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.198+0000 7f9061200640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f903c004c80 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f9044001720 tx=0x7f903c005830 comp rx=0 tx=0).stop 2024-04-24T08:13:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.198+0000 7f9061200640 1 -- 172.21.15.69:0/4227316099 shutdown_connections 2024-04-24T08:13:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.198+0000 7f9061200640 1 --2- 172.21.15.69:0/4227316099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f903c004c80 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.198+0000 7f9061200640 1 -- 172.21.15.69:0/4227316099 >> 172.21.15.69:0/4227316099 conn(0x7f905c09d900 msgr2=0x7f905c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.198+0000 7f9061200640 1 -- 172.21.15.69:0/4227316099 shutdown_connections 2024-04-24T08:13:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.198+0000 7f9061200640 1 -- 172.21.15.69:0/4227316099 wait complete. 2024-04-24T08:13:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.200+0000 7f9061200640 1 Processor -- start 2024-04-24T08:13:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.200+0000 7f9061200640 1 -- start start 2024-04-24T08:13:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.201+0000 7f9061200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f905c159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.201+0000 7f9061200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f905c0b2410 0x7f905c0b2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.201+0000 7f9061200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f905c15ef40 0x7f905c15f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.201+0000 7f9061200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f905c0cf5d0 con 0x7f903c004800 2024-04-24T08:13:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.201+0000 7f9061200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f905c0cf2f0 con 0x7f905c15ef40 2024-04-24T08:13:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.201+0000 7f9061200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f905c0b6170 con 0x7f905c0b2410 2024-04-24T08:13:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.201+0000 7f905b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f905c159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.201+0000 7f905b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f905c159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41726/0 (socket says 172.21.15.69:41726) 2024-04-24T08:13:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.201+0000 7f905b400640 1 -- 172.21.15.69:0/4209719501 learned_addr learned my addr 172.21.15.69:0/4209719501 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.202+0000 7f905be00640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f905c15ef40 0x7f905c15f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.202+0000 7f905aa00640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f905c0b2410 0x7f905c0b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.202+0000 7f905b400640 1 -- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f905c0b2410 msgr2=0x7f905c0b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.202+0000 7f905b400640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f905c0b2410 0x7f905c0b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.202+0000 7f905b400640 1 -- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f905c15ef40 msgr2=0x7f905c15f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.202+0000 7f905b400640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f905c15ef40 0x7f905c15f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.202+0000 7f905b400640 1 -- 172.21.15.69:0/4209719501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f905c1f7f20 con 0x7f903c004800 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.202+0000 7f905b400640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f905c159880 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e730 tx=0x7f904c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.202+0000 7f9053400640 1 -- 172.21.15.69:0/4209719501 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c00c660 con 0x7f903c004800 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.203+0000 7f9053400640 1 -- 172.21.15.69:0/4209719501 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f904c018070 con 0x7f903c004800 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.203+0000 7f9053400640 1 -- 172.21.15.69:0/4209719501 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f904c00ca40 con 0x7f903c004800 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.203+0000 7f9061200640 1 -- 172.21.15.69:0/4209719501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f905c1f8200 con 0x7f903c004800 2024-04-24T08:13:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.203+0000 7f9061200640 1 -- 172.21.15.69:0/4209719501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f905c1f87c0 con 0x7f903c004800 2024-04-24T08:13:27.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.204+0000 7f9061200640 1 -- 172.21.15.69:0/4209719501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f905c075d50 con 0x7f903c004800 2024-04-24T08:13:27.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.205+0000 7f9053400640 1 -- 172.21.15.69:0/4209719501 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f904c00cbe0 con 0x7f903c004800 2024-04-24T08:13:27.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.205+0000 7f9053400640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f903806c8f0 0x7f903806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.206+0000 7f9053400640 1 -- 172.21.15.69:0/4209719501 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f904c08b830 con 0x7f903c004800 2024-04-24T08:13:27.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.206+0000 7f905aa00640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f903806c8f0 0x7f903806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.207+0000 7f905aa00640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f903806c8f0 0x7f903806edb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9044032000 tx=0x7f904402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:27.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.210+0000 7f9053400640 1 -- 172.21.15.69:0/4209719501 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f904c059cf0 con 0x7f903c004800 2024-04-24T08:13:27.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.422+0000 7f9061200640 1 -- 172.21.15.69:0/4209719501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f905c0d1dc0 con 0x7f903c004800 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.423+0000 7f9053400640 1 -- 172.21.15.69:0/4209719501 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v68) ==== 80+0+0 (secure 0 0 0) 0x7f904c0595f0 con 0x7f903c004800 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.425+0000 7f9061200640 1 -- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f903806c8f0 msgr2=0x7f903806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.425+0000 7f9061200640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f903806c8f0 0x7f903806edb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9044032000 tx=0x7f904402e000 comp rx=0 tx=0).stop 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.425+0000 7f9061200640 1 -- 172.21.15.69:0/4209719501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 msgr2=0x7f905c159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.426+0000 7f9061200640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f905c159880 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f904c00e730 tx=0x7f904c00ebf0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.426+0000 7f9061200640 1 -- 172.21.15.69:0/4209719501 shutdown_connections 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.426+0000 7f9061200640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f903806c8f0 0x7f903806edb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.426+0000 7f9061200640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f905c15ef40 0x7f905c15f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.426+0000 7f9061200640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f905c0b2410 0x7f905c0b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.426+0000 7f9061200640 1 --2- 172.21.15.69:0/4209719501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f903c004800 0x7f905c159880 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.426+0000 7f9061200640 1 -- 172.21.15.69:0/4209719501 >> 172.21.15.69:0/4209719501 conn(0x7f905c09d900 msgr2=0x7f905c0b48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:27.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.426+0000 7f9061200640 1 -- 172.21.15.69:0/4209719501 shutdown_connections 2024-04-24T08:13:27.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.426+0000 7f9061200640 1 -- 172.21.15.69:0/4209719501 wait complete. 2024-04-24T08:13:27.445 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:13:27.445 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:13:27.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:27.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.746+0000 7f7298600640 1 Processor -- start 2024-04-24T08:13:27.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.746+0000 7f7298600640 1 -- start start 2024-04-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.747+0000 7f7298600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f729015ea10 0x7f72900d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.747+0000 7f7298600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f72901631a0 0x7f72900af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.747+0000 7f7298600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72900adab0 con 0x7f72900af7d0 2024-04-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.747+0000 7f7298600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f72900afbb0 con 0x7f729015ea10 2024-04-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.747+0000 7f7298600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f72900add50 con 0x7f72901631a0 2024-04-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.748+0000 7f7295c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f729015ea10 0x7f72900d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.748+0000 7f7295c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f729015ea10 0x7f72900d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41734/0 (socket says 172.21.15.69:41734) 2024-04-24T08:13:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.748+0000 7f7296600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f72900af7d0 0x7f7290071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39082/0 (socket says 172.21.15.69:39082) 2024-04-24T08:13:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.748+0000 7f7295c00640 1 -- 172.21.15.69:0/2387785113 learned_addr learned my addr 172.21.15.69:0/2387785113 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.748+0000 7f7295200640 1 --2- 172.21.15.69:0/2387785113 >> v2:172.21.15.121:3300/0 conn(0x7f72901631a0 0x7f72900af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.748+0000 7f7295c00640 1 -- 172.21.15.69:0/2387785113 >> v2:172.21.15.121:3300/0 conn(0x7f72901631a0 msgr2=0x7f72900af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.748+0000 7f7295c00640 1 --2- 172.21.15.69:0/2387785113 >> v2:172.21.15.121:3300/0 conn(0x7f72901631a0 0x7f72900af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.748+0000 7f7295c00640 1 -- 172.21.15.69:0/2387785113 >> v1:172.21.15.69:6789/0 conn(0x7f72900af7d0 legacy=0x7f7290071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.748+0000 7f7295c00640 1 -- 172.21.15.69:0/2387785113 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72900ae030 con 0x7f729015ea10 2024-04-24T08:13:27.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.748+0000 7f7295200640 1 --2- 172.21.15.69:0/2387785113 >> v2:172.21.15.121:3300/0 conn(0x7f72901631a0 0x7f72900af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:27.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.749+0000 7f7295c00640 1 --2- 172.21.15.69:0/2387785113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 0x7f72900d0830 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f729006f5d0 tx=0x7f728402a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b49c7d5a49d0cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:27.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f728fe00640 1 -- 172.21.15.69:0/2387785113 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284037070 con 0x7f729015ea10 2024-04-24T08:13:27.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f728fe00640 1 -- 172.21.15.69:0/2387785113 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7284032440 con 0x7f729015ea10 2024-04-24T08:13:27.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f728fe00640 1 -- 172.21.15.69:0/2387785113 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7284032810 con 0x7f729015ea10 2024-04-24T08:13:27.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f7298600640 1 -- 172.21.15.69:0/2387785113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 msgr2=0x7f72900d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f7298600640 1 --2- 172.21.15.69:0/2387785113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 0x7f72900d0830 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f729006f5d0 tx=0x7f728402a280 comp rx=0 tx=0).stop 2024-04-24T08:13:27.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f7298600640 1 -- 172.21.15.69:0/2387785113 shutdown_connections 2024-04-24T08:13:27.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f7298600640 1 --2- 172.21.15.69:0/2387785113 >> v2:172.21.15.121:3300/0 conn(0x7f72901631a0 0x7f72900af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f7298600640 1 --2- 172.21.15.69:0/2387785113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 0x7f72900d0830 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f7298600640 1 -- 172.21.15.69:0/2387785113 >> 172.21.15.69:0/2387785113 conn(0x7f7290067d90 msgr2=0x7f729006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:27.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f7298600640 1 -- 172.21.15.69:0/2387785113 shutdown_connections 2024-04-24T08:13:27.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.750+0000 7f7298600640 1 -- 172.21.15.69:0/2387785113 wait complete. 2024-04-24T08:13:27.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.752+0000 7f7298600640 1 Processor -- start 2024-04-24T08:13:27.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.752+0000 7f7298600640 1 -- start start 2024-04-24T08:13:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.753+0000 7f7298600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72900af7d0 0x7f7290161900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.753+0000 7f7298600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 0x7f7290153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.753+0000 7f7298600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72901631a0 0x7f7290150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.753+0000 7f7298600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72900ae8d0 con 0x7f729015ea10 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.753+0000 7f7298600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f72900ae5f0 con 0x7f72900af7d0 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.753+0000 7f7298600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f72900aedf0 con 0x7f72901631a0 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.753+0000 7f7295200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 0x7f7290153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.753+0000 7f7295200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 0x7f7290153190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41748/0 (socket says 172.21.15.69:41748) 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.753+0000 7f7295200640 1 -- 172.21.15.69:0/803272397 learned_addr learned my addr 172.21.15.69:0/803272397 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.754+0000 7f7295c00640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72900af7d0 0x7f7290161900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.754+0000 7f7296600640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72901631a0 0x7f7290150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.754+0000 7f7295200640 1 -- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72901631a0 msgr2=0x7f7290150d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.754+0000 7f7295200640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72901631a0 0x7f7290150d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.754+0000 7f7295200640 1 -- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72900af7d0 msgr2=0x7f7290161900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.754+0000 7f7295200640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72900af7d0 0x7f7290161900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.754+0000 7f7295200640 1 -- 172.21.15.69:0/803272397 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72901536d0 con 0x7f729015ea10 2024-04-24T08:13:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.754+0000 7f7295200640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 0x7f7290153190 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f727c00c820 tx=0x7f727c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:27.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.754+0000 7f728d600640 1 -- 172.21.15.69:0/803272397 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c015070 con 0x7f729015ea10 2024-04-24T08:13:27.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.755+0000 7f728d600640 1 -- 172.21.15.69:0/803272397 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f727c007a30 con 0x7f729015ea10 2024-04-24T08:13:27.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.755+0000 7f7298600640 1 -- 172.21.15.69:0/803272397 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7290156570 con 0x7f729015ea10 2024-04-24T08:13:27.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.755+0000 7f728d600640 1 -- 172.21.15.69:0/803272397 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c007de0 con 0x7f729015ea10 2024-04-24T08:13:27.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.755+0000 7f7298600640 1 -- 172.21.15.69:0/803272397 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7290152990 con 0x7f729015ea10 2024-04-24T08:13:27.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.756+0000 7f7298600640 1 -- 172.21.15.69:0/803272397 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f729015a1e0 con 0x7f729015ea10 2024-04-24T08:13:27.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.757+0000 7f728d600640 1 -- 172.21.15.69:0/803272397 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f727c018070 con 0x7f729015ea10 2024-04-24T08:13:27.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.758+0000 7f728d600640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f727006c8f0 0x7f727006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:27.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.758+0000 7f728d600640 1 -- 172.21.15.69:0/803272397 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f727c08ba90 con 0x7f729015ea10 2024-04-24T08:13:27.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.758+0000 7f7295c00640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f727006c8f0 0x7f727006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:27.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.759+0000 7f7295c00640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f727006c8f0 0x7f727006edb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7284034000 tx=0x7f728402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:27.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.762+0000 7f728d600640 1 -- 172.21.15.69:0/803272397 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f729015a1e0 con 0x7f729015ea10 2024-04-24T08:13:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.968+0000 7f7298600640 1 -- 172.21.15.69:0/803272397 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7290068a80 con 0x7f729015ea10 2024-04-24T08:13:27.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.970+0000 7f728d600640 1 -- 172.21.15.69:0/803272397 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) ==== 74+0+16644 (secure 0 0 0) 0x7f727c059e20 con 0x7f729015ea10 2024-04-24T08:13:27.972 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:27.972 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":68,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:13:17.084219+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:27.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 -- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f727006c8f0 msgr2=0x7f727006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f727006c8f0 0x7f727006edb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7284034000 tx=0x7f728402e000 comp rx=0 tx=0).stop 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 -- 172.21.15.69:0/803272397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 msgr2=0x7f7290153190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 0x7f7290153190 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f727c00c820 tx=0x7f727c00cce0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 -- 172.21.15.69:0/803272397 shutdown_connections 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f727006c8f0 0x7f727006edb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f729015ea10 0x7f7290153190 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f72901631a0 0x7f7290150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 --2- 172.21.15.69:0/803272397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72900af7d0 0x7f7290161900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 -- 172.21.15.69:0/803272397 >> 172.21.15.69:0/803272397 conn(0x7f7290067d90 msgr2=0x7f72900b0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 -- 172.21.15.69:0/803272397 shutdown_connections 2024-04-24T08:13:27.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:27.973+0000 7f7298600640 1 -- 172.21.15.69:0/803272397 wait complete. 2024-04-24T08:13:27.994 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:13:13.250209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:13:14.468238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:13:27.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:13:28.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.190+0000 7effe5800640 1 Processor -- start 2024-04-24T08:13:28.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.190+0000 7effe5800640 1 -- start start 2024-04-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.191+0000 7effe5800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7effe015ea70 0x7effe00cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.191+0000 7effe5800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe006ef40 0x7effe006ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.191+0000 7effe5800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe015ee50 con 0x7effe00cfd20 2024-04-24T08:13:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.191+0000 7effe5800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7effe00cdcb0 con 0x7effe015ea70 2024-04-24T08:13:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.191+0000 7effe5800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7effe006f4b0 con 0x7effe006ef40 2024-04-24T08:13:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.191+0000 7effe4e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7effe00cfd20 0x7effe00ce140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39090/0 (socket says 172.21.15.69:39090) 2024-04-24T08:13:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.191+0000 7effe4e00640 1 -- 172.21.15.69:0/1017826040 learned_addr learned my addr 172.21.15.69:0/1017826040 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdfe00640 1 --2- 172.21.15.69:0/1017826040 >> v2:172.21.15.121:3300/0 conn(0x7effe015ea70 0x7effe00cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdf400640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe006ef40 0x7effe006ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3921674390 0 0) 0x7effe015ee50 con 0x7effe00cfd20 2024-04-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effc4003520 con 0x7effe00cfd20 2024-04-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3427988250 0 0) 0x7effc4003520 con 0x7effe00cfd20 2024-04-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe006ef40 msgr2=0x7effe006ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe006ef40 0x7effe006ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 >> v2:172.21.15.121:3300/0 conn(0x7effe015ea70 msgr2=0x7effe00cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 --2- 172.21.15.69:0/1017826040 >> v2:172.21.15.121:3300/0 conn(0x7effe015ea70 0x7effe00cf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe006f790 con 0x7effe00cfd20 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7effd8001670 con 0x7effe00cfd20 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 >> v1:172.21.15.69:6789/0 conn(0x7effe00cfd20 legacy=0x7effe00ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe006ef40 0x7effc40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effc4003f30 0x7effc4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effc4004d40 0x7effc40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7effe00cdcb0 con 0x7effc4003f30 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7effe006f4b0 con 0x7effc4004d40 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.192+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7effc400ba70 con 0x7effe006ef40 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.193+0000 7effdf400640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effc4003f30 0x7effc4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.193+0000 7effe4e00640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effc4004d40 0x7effc40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.193+0000 7effdfe00640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe006ef40 0x7effc40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.193+0000 7effdf400640 1 -- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe006ef40 msgr2=0x7effc40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:28.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.193+0000 7effdf400640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe006ef40 0x7effc40039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.193+0000 7effdf400640 1 -- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effc4004d40 msgr2=0x7effc40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:28.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.193+0000 7effdf400640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effc4004d40 0x7effc40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.193+0000 7effdf400640 1 -- 172.21.15.69:0/1017826040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe006f790 con 0x7effc4003f30 2024-04-24T08:13:28.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.193+0000 7effe4e00640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effc4004d40 0x7effc40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:28.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.194+0000 7effdf400640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effc4003f30 0x7effc4004800 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7effc800ba30 tx=0x7effc4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e4673cd6db6a42 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:28.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.194+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8030070 con 0x7effc4003f30 2024-04-24T08:13:28.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.194+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc802ade0 con 0x7effc4003f30 2024-04-24T08:13:28.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.194+0000 7effdea00640 1 -- 172.21.15.69:0/1017826040 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc800b600 con 0x7effc4003f30 2024-04-24T08:13:28.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.195+0000 7effe5800640 1 -- 172.21.15.69:0/1017826040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effc4003f30 msgr2=0x7effc4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.195+0000 7effe5800640 1 --2- 172.21.15.69:0/1017826040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effc4003f30 0x7effc4004800 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7effc800ba30 tx=0x7effc4005830 comp rx=0 tx=0).stop 2024-04-24T08:13:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.195+0000 7effe4e00640 1 -- 172.21.15.69:0/1017826040 reap_dead start 2024-04-24T08:13:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.195+0000 7effe5800640 1 -- 172.21.15.69:0/1017826040 shutdown_connections 2024-04-24T08:13:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.195+0000 7effe5800640 1 -- 172.21.15.69:0/1017826040 >> 172.21.15.69:0/1017826040 conn(0x7effe006cb30 msgr2=0x7effe006cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.195+0000 7effe5800640 1 -- 172.21.15.69:0/1017826040 shutdown_connections 2024-04-24T08:13:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.195+0000 7effe5800640 1 -- 172.21.15.69:0/1017826040 wait complete. 2024-04-24T08:13:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.197+0000 7effe5800640 1 Processor -- start 2024-04-24T08:13:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.197+0000 7effe5800640 1 -- start start 2024-04-24T08:13:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.198+0000 7effe5800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effc4003f30 0x7effe01608b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.198+0000 7effe5800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe00b0e20 0x7effe01f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.198+0000 7effe5800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe01f8530 0x7effe01f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.198+0000 7effe5800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7effe00b0290 con 0x7effe00b0e20 2024-04-24T08:13:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.198+0000 7effe5800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7effe006fd50 con 0x7effc4003f30 2024-04-24T08:13:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.198+0000 7effe5800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7effe00ce850 con 0x7effe01f8530 2024-04-24T08:13:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.198+0000 7effdf400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe00b0e20 0x7effe01f7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.198+0000 7effdf400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe00b0e20 0x7effe01f7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41764/0 (socket says 172.21.15.69:41764) 2024-04-24T08:13:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.198+0000 7effdf400640 1 -- 172.21.15.69:0/3491548162 learned_addr learned my addr 172.21.15.69:0/3491548162 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effdf400640 1 -- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe01f8530 msgr2=0x7effe01f89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effe4e00640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe01f8530 0x7effe01f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effdf400640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe01f8530 0x7effe01f89b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effdf400640 1 -- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effc4003f30 msgr2=0x7effe01608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effdfe00640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effc4003f30 0x7effe01608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effdf400640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effc4003f30 0x7effe01608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effdf400640 1 -- 172.21.15.69:0/3491548162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe00b13e0 con 0x7effe00b0e20 2024-04-24T08:13:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effdf400640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe00b0e20 0x7effe01f7ff0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7effc800bb50 tx=0x7effc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effd7e00640 1 -- 172.21.15.69:0/3491548162 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8043030 con 0x7effe00b0e20 2024-04-24T08:13:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effd7e00640 1 -- 172.21.15.69:0/3491548162 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc802abe0 con 0x7effe00b0e20 2024-04-24T08:13:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.199+0000 7effe5800640 1 -- 172.21.15.69:0/3491548162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe00b16c0 con 0x7effe00b0e20 2024-04-24T08:13:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.200+0000 7effd7e00640 1 -- 172.21.15.69:0/3491548162 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc800b600 con 0x7effe00b0e20 2024-04-24T08:13:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.200+0000 7effe5800640 1 -- 172.21.15.69:0/3491548162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effe00b1c80 con 0x7effe00b0e20 2024-04-24T08:13:28.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.201+0000 7effe5800640 1 -- 172.21.15.69:0/3491548162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effe0075d50 con 0x7effe00b0e20 2024-04-24T08:13:28.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.202+0000 7effd7e00640 1 -- 172.21.15.69:0/3491548162 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7effc8046070 con 0x7effe00b0e20 2024-04-24T08:13:28.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.202+0000 7effd7e00640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effbc06c8f0 0x7effbc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.203+0000 7effd7e00640 1 -- 172.21.15.69:0/3491548162 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7effc80ac5c0 con 0x7effe00b0e20 2024-04-24T08:13:28.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.203+0000 7effdfe00640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effbc06c8f0 0x7effbc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.204+0000 7effdfe00640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effbc06c8f0 0x7effbc06edb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7effd0003a50 tx=0x7effd000b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:28.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.207+0000 7effd7e00640 1 -- 172.21.15.69:0/3491548162 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7effc807a8c0 con 0x7effe00b0e20 2024-04-24T08:13:28.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.408+0000 7effe5800640 1 -- 172.21.15.69:0/3491548162 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7effe00af5b0 con 0x7effe00b0e20 2024-04-24T08:13:28.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.409+0000 7effd7e00640 1 -- 172.21.15.69:0/3491548162 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) ==== 93+0+10 (secure 0 0 0) 0x7effc807a1c0 con 0x7effe00b0e20 2024-04-24T08:13:28.411 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:13:28.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 -- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effbc06c8f0 msgr2=0x7effbc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effbc06c8f0 0x7effbc06edb0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7effd0003a50 tx=0x7effd000b040 comp rx=0 tx=0).stop 2024-04-24T08:13:28.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 -- 172.21.15.69:0/3491548162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe00b0e20 msgr2=0x7effe01f7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe00b0e20 0x7effe01f7ff0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7effc800bb50 tx=0x7effc80046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 -- 172.21.15.69:0/3491548162 shutdown_connections 2024-04-24T08:13:28.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effbc06c8f0 0x7effbc06edb0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe00b0e20 0x7effe01f7ff0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effe01f8530 0x7effe01f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 --2- 172.21.15.69:0/3491548162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effc4003f30 0x7effe01608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 -- 172.21.15.69:0/3491548162 >> 172.21.15.69:0/3491548162 conn(0x7effe006cb30 msgr2=0x7effe006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:28.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.412+0000 7effe5800640 1 -- 172.21.15.69:0/3491548162 shutdown_connections 2024-04-24T08:13:28.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.413+0000 7effe5800640 1 -- 172.21.15.69:0/3491548162 wait complete. 2024-04-24T08:13:28.433 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:13:28.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.634+0000 7f279e600640 1 Processor -- start 2024-04-24T08:13:28.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.634+0000 7f279e600640 1 -- start start 2024-04-24T08:13:28.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.635+0000 7f279e600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f27980d18c0 0x7f279806de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.635+0000 7f279e600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27980d2600 con 0x7f279815ea60 2024-04-24T08:13:28.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.635+0000 7f279e600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27980d28e0 con 0x7f27980d36a0 2024-04-24T08:13:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.635+0000 7f279e600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f27980d23b0 con 0x7f27980d18c0 2024-04-24T08:13:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.636+0000 7f2797400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f279815ea60 0x7f27980cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39092/0 (socket says 172.21.15.69:39092) 2024-04-24T08:13:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.636+0000 7f2797400640 1 -- 172.21.15.69:0/1709354893 learned_addr learned my addr 172.21.15.69:0/1709354893 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.636+0000 7f2797e00640 1 --2- 172.21.15.69:0/1709354893 >> v2:172.21.15.121:3300/0 conn(0x7f27980d18c0 0x7f279806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.636+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288813071 0 0) 0x7f27980d2600 con 0x7f279815ea60 2024-04-24T08:13:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.636+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f277c003520 con 0x7f279815ea60 2024-04-24T08:13:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 695900340 0 0) 0x7f277c003520 con 0x7f279815ea60 2024-04-24T08:13:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 >> v2:172.21.15.121:3300/0 conn(0x7f27980d18c0 msgr2=0x7f279806de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 --2- 172.21.15.69:0/1709354893 >> v2:172.21.15.121:3300/0 conn(0x7f27980d18c0 0x7f279806de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 >> v1:172.21.15.121:6789/0 conn(0x7f27980d36a0 legacy=0x7f27980d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27980d2bc0 con 0x7f279815ea60 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f2788001670 con 0x7f279815ea60 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 >> v1:172.21.15.69:6789/0 conn(0x7f279815ea60 legacy=0x7f27980cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 --2- 172.21.15.69:0/1709354893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f277c0039b0 0x7f277c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 --2- 172.21.15.69:0/1709354893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f277c004800 0x7f277c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 --2- 172.21.15.69:0/1709354893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f277c0051c0 0x7f277c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f27980d23b0 con 0x7f277c004800 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f277c00c060 con 0x7f277c0051c0 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f277c00c260 con 0x7f277c0039b0 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f278ea00640 1 --2- 172.21.15.69:0/1709354893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f277c004800 0x7f277c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f278ea00640 1 -- 172.21.15.69:0/1709354893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f277c0039b0 msgr2=0x7f277c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f278ea00640 1 --2- 172.21.15.69:0/1709354893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f277c0039b0 0x7f277c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f278ea00640 1 -- 172.21.15.69:0/1709354893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f277c0051c0 msgr2=0x7f277c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f278ea00640 1 --2- 172.21.15.69:0/1709354893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f277c0051c0 0x7f277c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f278ea00640 1 -- 172.21.15.69:0/1709354893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f277c003520 con 0x7f277c004800 2024-04-24T08:13:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.637+0000 7f2797e00640 1 -- 172.21.15.69:0/1709354893 reap_dead start 2024-04-24T08:13:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.638+0000 7f278ea00640 1 --2- 172.21.15.69:0/1709354893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f277c004800 0x7f277c004c80 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f2780001e50 tx=0x7f277c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b144ed49842a66 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.639+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780001a70 con 0x7f277c004800 2024-04-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.639+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2780003970 con 0x7f277c004800 2024-04-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.639+0000 7f2796a00640 1 -- 172.21.15.69:0/1709354893 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2780003d60 con 0x7f277c004800 2024-04-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.639+0000 7f279e600640 1 -- 172.21.15.69:0/1709354893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f277c004800 msgr2=0x7f277c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.639+0000 7f279e600640 1 --2- 172.21.15.69:0/1709354893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f277c004800 0x7f277c004c80 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f2780001e50 tx=0x7f277c005830 comp rx=0 tx=0).stop 2024-04-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.639+0000 7f279e600640 1 -- 172.21.15.69:0/1709354893 shutdown_connections 2024-04-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.639+0000 7f279e600640 1 --2- 172.21.15.69:0/1709354893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f277c004800 0x7f277c004c80 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.640+0000 7f279e600640 1 -- 172.21.15.69:0/1709354893 >> 172.21.15.69:0/1709354893 conn(0x7f279808bf60 msgr2=0x7f27980d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:28.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.640+0000 7f279e600640 1 -- 172.21.15.69:0/1709354893 shutdown_connections 2024-04-24T08:13:28.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.640+0000 7f279e600640 1 -- 172.21.15.69:0/1709354893 wait complete. 2024-04-24T08:13:28.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.641+0000 7f279e600640 1 Processor -- start 2024-04-24T08:13:28.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.641+0000 7f279e600640 1 -- start start 2024-04-24T08:13:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.642+0000 7f279e600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f277c004800 0x7f2798157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.642+0000 7f279e600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27980cc1c0 0x7f27981f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.642+0000 7f279e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27981f8200 0x7f27981f8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.642+0000 7f279e600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f279815ee40 con 0x7f27981f8200 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.642+0000 7f279e600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f27980d3180 con 0x7f277c004800 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.642+0000 7f279e600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f27980cc780 con 0x7f27980cc1c0 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.643+0000 7f2797e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27981f8200 0x7f27981f8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.643+0000 7f2797e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27981f8200 0x7f27981f8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41780/0 (socket says 172.21.15.69:41780) 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.643+0000 7f2797e00640 1 -- 172.21.15.69:0/3807298982 learned_addr learned my addr 172.21.15.69:0/3807298982 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.643+0000 7f2797400640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f277c004800 0x7f2798157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.643+0000 7f278ea00640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27980cc1c0 0x7f27981f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.643+0000 7f2797e00640 1 -- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27980cc1c0 msgr2=0x7f27981f7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.643+0000 7f2797e00640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27980cc1c0 0x7f27981f7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.643+0000 7f2797e00640 1 -- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f277c004800 msgr2=0x7f2798157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.643+0000 7f2797e00640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f277c004800 0x7f2798157ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.643+0000 7f2797e00640 1 -- 172.21.15.69:0/3807298982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2798160d30 con 0x7f27981f8200 2024-04-24T08:13:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.644+0000 7f2797e00640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27981f8200 0x7f27981f8680 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f27980d58c0 tx=0x7f279000e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.644+0000 7f278fe00640 1 -- 172.21.15.69:0/3807298982 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2790017020 con 0x7f27981f8200 2024-04-24T08:13:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.644+0000 7f278fe00640 1 -- 172.21.15.69:0/3807298982 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2790004240 con 0x7f27981f8200 2024-04-24T08:13:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.644+0000 7f278fe00640 1 -- 172.21.15.69:0/3807298982 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2790003610 con 0x7f27981f8200 2024-04-24T08:13:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.644+0000 7f279e600640 1 -- 172.21.15.69:0/3807298982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2798161010 con 0x7f27981f8200 2024-04-24T08:13:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.644+0000 7f279e600640 1 -- 172.21.15.69:0/3807298982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27980a2050 con 0x7f27981f8200 2024-04-24T08:13:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.645+0000 7f279e600640 1 -- 172.21.15.69:0/3807298982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2798068a80 con 0x7f27981f8200 2024-04-24T08:13:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.646+0000 7f278fe00640 1 -- 172.21.15.69:0/3807298982 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f279001a020 con 0x7f27981f8200 2024-04-24T08:13:28.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.647+0000 7f278fe00640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f277406c8f0 0x7f277406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:28.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.647+0000 7f278fe00640 1 -- 172.21.15.69:0/3807298982 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f279008b880 con 0x7f27981f8200 2024-04-24T08:13:28.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.648+0000 7f2797400640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f277406c8f0 0x7f277406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:28.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.649+0000 7f2797400640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f277406c8f0 0x7f277406edb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2788001860 tx=0x7f27880015f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:28.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.651+0000 7f278fe00640 1 -- 172.21.15.69:0/3807298982 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2790021030 con 0x7f27981f8200 2024-04-24T08:13:28.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.849+0000 7f279e600640 1 -- 172.21.15.69:0/3807298982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f27980d1f70 con 0x7f27981f8200 2024-04-24T08:13:28.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.850+0000 7f278fe00640 1 -- 172.21.15.69:0/3807298982 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v68) ==== 104+0+11 (secure 0 0 0) 0x7f2790059d40 con 0x7f27981f8200 2024-04-24T08:13:28.851 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:13:28.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 -- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f277406c8f0 msgr2=0x7f277406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f277406c8f0 0x7f277406edb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2788001860 tx=0x7f27880015f0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 -- 172.21.15.69:0/3807298982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27981f8200 msgr2=0x7f27981f8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:28.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27981f8200 0x7f27981f8680 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f27980d58c0 tx=0x7f279000e9d0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 -- 172.21.15.69:0/3807298982 shutdown_connections 2024-04-24T08:13:28.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f277406c8f0 0x7f277406edb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27981f8200 0x7f27981f8680 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27980cc1c0 0x7f27981f7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 --2- 172.21.15.69:0/3807298982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f277c004800 0x7f2798157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:28.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 -- 172.21.15.69:0/3807298982 >> 172.21.15.69:0/3807298982 conn(0x7f279808bf60 msgr2=0x7f279806de90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:28.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 -- 172.21.15.69:0/3807298982 shutdown_connections 2024-04-24T08:13:28.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:28.852+0000 7f279e600640 1 -- 172.21.15.69:0/3807298982 wait complete. 2024-04-24T08:13:28.872 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:13:29.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.068+0000 7fcfba200640 1 Processor -- start 2024-04-24T08:13:29.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.068+0000 7fcfba200640 1 -- start start 2024-04-24T08:13:29.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.069+0000 7fcfba200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcfb40cd3d0 0x7fcfb40b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.069+0000 7fcfba200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfb40b4780 0x7fcfb40b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.069+0000 7fcfba200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfb41631c0 con 0x7fcfb4073db0 2024-04-24T08:13:29.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.069+0000 7fcfba200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcfb40cf4d0 con 0x7fcfb40cd3d0 2024-04-24T08:13:29.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.069+0000 7fcfba200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb40cf7b0 con 0x7fcfb40b4780 2024-04-24T08:13:29.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.069+0000 7fcfb3400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcfb4073db0 0x7fcfb40cccc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35674/0 (socket says 172.21.15.69:35674) 2024-04-24T08:13:29.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.070+0000 7fcfb3400640 1 -- 172.21.15.69:0/2773798584 learned_addr learned my addr 172.21.15.69:0/2773798584 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:29.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.070+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2000176744 0 0) 0x7fcfb41631c0 con 0x7fcfb4073db0 2024-04-24T08:13:29.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.070+0000 7fcfaaa00640 1 --2- 172.21.15.69:0/2773798584 >> v2:172.21.15.121:3300/0 conn(0x7fcfb40cd3d0 0x7fcfb40b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.070+0000 7fcfb3e00640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfb40b4780 0x7fcfb40b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.070+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf98003520 con 0x7fcfb4073db0 2024-04-24T08:13:29.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.070+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3327550072 0 0) 0x7fcf98003520 con 0x7fcfb4073db0 2024-04-24T08:13:29.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.070+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfb40b4780 msgr2=0x7fcfb40b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfb40b4780 0x7fcfb40b4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 >> v2:172.21.15.121:3300/0 conn(0x7fcfb40cd3d0 msgr2=0x7fcfb40b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 --2- 172.21.15.69:0/2773798584 >> v2:172.21.15.121:3300/0 conn(0x7fcfb40cd3d0 0x7fcfb40b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb40cfa90 con 0x7fcfb4073db0 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fcfa4001670 con 0x7fcfb4073db0 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 >> v1:172.21.15.69:6789/0 conn(0x7fcfb4073db0 legacy=0x7fcfb40cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf980039b0 0x7fcf98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf98004800 0x7fcf98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfb40b4780 0x7fcf980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb40cf4d0 con 0x7fcf980039b0 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb40cf7b0 con 0x7fcf98004800 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcf9800b9c0 con 0x7fcfb40b4780 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb3400640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf980039b0 0x7fcf98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfaaa00640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf98004800 0x7fcf98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb3400640 1 -- 172.21.15.69:0/2773798584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfb40b4780 msgr2=0x7fcf980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:29.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb3400640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcfb40b4780 0x7fcf980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb3400640 1 -- 172.21.15.69:0/2773798584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf98004800 msgr2=0x7fcf98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:29.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb3400640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf98004800 0x7fcf98004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.071+0000 7fcfb3400640 1 -- 172.21.15.69:0/2773798584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf98003520 con 0x7fcf980039b0 2024-04-24T08:13:29.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.072+0000 7fcfb3400640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf980039b0 0x7fcf98003dd0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4007fd0 tx=0x7fcf98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f217845e038c27a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:29.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.073+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4007d50 con 0x7fcf980039b0 2024-04-24T08:13:29.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.073+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa4005ad0 con 0x7fcf980039b0 2024-04-24T08:13:29.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.073+0000 7fcfb2a00640 1 -- 172.21.15.69:0/2773798584 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4007070 con 0x7fcf980039b0 2024-04-24T08:13:29.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.073+0000 7fcfba200640 1 -- 172.21.15.69:0/2773798584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf980039b0 msgr2=0x7fcf98003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.073+0000 7fcfba200640 1 --2- 172.21.15.69:0/2773798584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf980039b0 0x7fcf98003dd0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4007fd0 tx=0x7fcf98005830 comp rx=0 tx=0).stop 2024-04-24T08:13:29.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.073+0000 7fcfb3e00640 1 -- 172.21.15.69:0/2773798584 reap_dead start 2024-04-24T08:13:29.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.074+0000 7fcfba200640 1 -- 172.21.15.69:0/2773798584 shutdown_connections 2024-04-24T08:13:29.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.074+0000 7fcfba200640 1 -- 172.21.15.69:0/2773798584 >> 172.21.15.69:0/2773798584 conn(0x7fcfb40d0350 msgr2=0x7fcfb40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:29.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.074+0000 7fcfba200640 1 -- 172.21.15.69:0/2773798584 shutdown_connections 2024-04-24T08:13:29.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.074+0000 7fcfba200640 1 -- 172.21.15.69:0/2773798584 wait complete. 2024-04-24T08:13:29.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.075+0000 7fcfba200640 1 Processor -- start 2024-04-24T08:13:29.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.076+0000 7fcfba200640 1 -- start start 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.076+0000 7fcfba200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf980039b0 0x7fcfb415b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.076+0000 7fcfba200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfb41507f0 0x7fcfb4165b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.076+0000 7fcfba200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfb406e6b0 0x7fcfb406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.076+0000 7fcfba200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb40b4c30 con 0x7fcfb41507f0 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.076+0000 7fcfba200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb40d0050 con 0x7fcfb406e6b0 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.076+0000 7fcfba200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb40b4db0 con 0x7fcf980039b0 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfaaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfb41507f0 0x7fcfb4165b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfaaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfb41507f0 0x7fcfb4165b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47380/0 (socket says 172.21.15.69:47380) 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfaaa00640 1 -- 172.21.15.69:0/2160575028 learned_addr learned my addr 172.21.15.69:0/2160575028 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfb3400640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf980039b0 0x7fcfb415b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfaaa00640 1 -- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf980039b0 msgr2=0x7fcfb415b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfaaa00640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf980039b0 0x7fcfb415b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfaaa00640 1 -- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfb406e6b0 msgr2=0x7fcfb406eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfaaa00640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfb406e6b0 0x7fcfb406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfaaa00640 1 -- 172.21.15.69:0/2160575028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb4150c70 con 0x7fcfb41507f0 2024-04-24T08:13:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfb3400640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf980039b0 0x7fcfb415b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.077+0000 7fcfaaa00640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfb41507f0 0x7fcfb4165b60 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0108c0 tx=0x7fcf9c010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.078+0000 7fcfabe00640 1 -- 172.21.15.69:0/2160575028 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c00f040 con 0x7fcfb41507f0 2024-04-24T08:13:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.078+0000 7fcfabe00640 1 -- 172.21.15.69:0/2160575028 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf9c00ea80 con 0x7fcfb41507f0 2024-04-24T08:13:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.078+0000 7fcfabe00640 1 -- 172.21.15.69:0/2160575028 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c00ee50 con 0x7fcfb41507f0 2024-04-24T08:13:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.078+0000 7fcfba200640 1 -- 172.21.15.69:0/2160575028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb415d7e0 con 0x7fcfb41507f0 2024-04-24T08:13:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.078+0000 7fcfba200640 1 -- 172.21.15.69:0/2160575028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb415dda0 con 0x7fcfb41507f0 2024-04-24T08:13:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.079+0000 7fcfba200640 1 -- 172.21.15.69:0/2160575028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfb4068a80 con 0x7fcfb41507f0 2024-04-24T08:13:29.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.080+0000 7fcfabe00640 1 -- 172.21.15.69:0/2160575028 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcf9c01c400 con 0x7fcfb41507f0 2024-04-24T08:13:29.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.081+0000 7fcfabe00640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf9006c8f0 0x7fcf9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.081+0000 7fcfabe00640 1 -- 172.21.15.69:0/2160575028 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7fcf9c08c0b0 con 0x7fcfb41507f0 2024-04-24T08:13:29.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.082+0000 7fcfb3400640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf9006c8f0 0x7fcf9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.083+0000 7fcfb3400640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf9006c8f0 0x7fcf9006edb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4033000 tx=0x7fcfa4009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:29.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.085+0000 7fcfabe00640 1 -- 172.21.15.69:0/2160575028 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcfb4150c70 con 0x7fcfb41507f0 2024-04-24T08:13:29.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.287+0000 7fcfba200640 1 -- 172.21.15.69:0/2160575028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcfb40b41d0 con 0x7fcfb41507f0 2024-04-24T08:13:29.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.288+0000 7fcfabe00640 1 -- 172.21.15.69:0/2160575028 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v68) ==== 100+0+11 (secure 0 0 0) 0x7fcf9c018030 con 0x7fcfb41507f0 2024-04-24T08:13:29.290 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:13:29.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 -- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf9006c8f0 msgr2=0x7fcf9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf9006c8f0 0x7fcf9006edb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4033000 tx=0x7fcfa4009030 comp rx=0 tx=0).stop 2024-04-24T08:13:29.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 -- 172.21.15.69:0/2160575028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfb41507f0 msgr2=0x7fcfb4165b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfb41507f0 0x7fcfb4165b60 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c0108c0 tx=0x7fcf9c010d80 comp rx=0 tx=0).stop 2024-04-24T08:13:29.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 -- 172.21.15.69:0/2160575028 shutdown_connections 2024-04-24T08:13:29.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf9006c8f0 0x7fcf9006edb0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfb41507f0 0x7fcfb4165b60 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfb406e6b0 0x7fcfb406eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 --2- 172.21.15.69:0/2160575028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf980039b0 0x7fcfb415b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 -- 172.21.15.69:0/2160575028 >> 172.21.15.69:0/2160575028 conn(0x7fcfb40d0350 msgr2=0x7fcfb4073410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:29.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.291+0000 7fcfba200640 1 -- 172.21.15.69:0/2160575028 shutdown_connections 2024-04-24T08:13:29.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.292+0000 7fcfba200640 1 -- 172.21.15.69:0/2160575028 wait complete. 2024-04-24T08:13:29.310 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:13:29.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.508+0000 7f789aa00640 1 Processor -- start 2024-04-24T08:13:29.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.508+0000 7f789aa00640 1 -- start start 2024-04-24T08:13:29.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.510+0000 7f789aa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 0x7f789406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.510+0000 7f789aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7894145a90 0x7f78940d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.510+0000 7f789aa00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7894145fe0 0x7f7894160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.510+0000 7f789aa00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f78940d0c20 con 0x7f7894145a90 2024-04-24T08:13:29.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.510+0000 7f789aa00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f78940cf3d0 con 0x7f7894145fe0 2024-04-24T08:13:29.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.510+0000 7f789aa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f78940cf6b0 con 0x7f78940d0300 2024-04-24T08:13:29.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.510+0000 7f7893400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7894145a90 0x7f78940d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.510+0000 7f7893400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7894145a90 0x7f78940d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47386/0 (socket says 172.21.15.69:47386) 2024-04-24T08:13:29.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.510+0000 7f7893400640 1 -- 172.21.15.69:0/2445830069 learned_addr learned my addr 172.21.15.69:0/2445830069 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.511+0000 7f7893400640 1 -- 172.21.15.69:0/2445830069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 msgr2=0x7f789406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.511+0000 7f7893e00640 1 --2- 172.21.15.69:0/2445830069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 0x7f789406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.511+0000 7f7898a00640 1 --2- 172.21.15.69:0/2445830069 >> v2:172.21.15.121:3300/0 conn(0x7f7894145fe0 0x7f7894160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.511+0000 7f7893400640 1 --2- 172.21.15.69:0/2445830069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 0x7f789406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.511+0000 7f7893400640 1 -- 172.21.15.69:0/2445830069 >> v2:172.21.15.121:3300/0 conn(0x7f7894145fe0 msgr2=0x7f7894160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.511+0000 7f7893400640 1 --2- 172.21.15.69:0/2445830069 >> v2:172.21.15.121:3300/0 conn(0x7f7894145fe0 0x7f7894160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.511+0000 7f7893400640 1 -- 172.21.15.69:0/2445830069 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78940cf990 con 0x7f7894145a90 2024-04-24T08:13:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.511+0000 7f7893e00640 1 --2- 172.21.15.69:0/2445830069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 0x7f789406e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.511+0000 7f7898a00640 1 --2- 172.21.15.69:0/2445830069 >> v2:172.21.15.121:3300/0 conn(0x7f7894145fe0 0x7f7894160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:29.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.512+0000 7f7893400640 1 --2- 172.21.15.69:0/2445830069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 0x7f78940d06e0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7894071b30 tx=0x7f788002a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5e779f3689ca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:29.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.512+0000 7f7892a00640 1 -- 172.21.15.69:0/2445830069 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880037070 con 0x7f7894145a90 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.512+0000 7f7892a00640 1 -- 172.21.15.69:0/2445830069 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7880032440 con 0x7f7894145a90 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.512+0000 7f7892a00640 1 -- 172.21.15.69:0/2445830069 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880032810 con 0x7f7894145a90 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.513+0000 7f789aa00640 1 -- 172.21.15.69:0/2445830069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 msgr2=0x7f78940d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.513+0000 7f789aa00640 1 --2- 172.21.15.69:0/2445830069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 0x7f78940d06e0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f7894071b30 tx=0x7f788002a1d0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.513+0000 7f789aa00640 1 -- 172.21.15.69:0/2445830069 shutdown_connections 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.513+0000 7f789aa00640 1 --2- 172.21.15.69:0/2445830069 >> v2:172.21.15.121:3300/0 conn(0x7f7894145fe0 0x7f7894160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.513+0000 7f789aa00640 1 --2- 172.21.15.69:0/2445830069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 0x7f78940d06e0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.513+0000 7f789aa00640 1 --2- 172.21.15.69:0/2445830069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 0x7f789406e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.513+0000 7f789aa00640 1 -- 172.21.15.69:0/2445830069 >> 172.21.15.69:0/2445830069 conn(0x7f7894067d90 msgr2=0x7f789406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.513+0000 7f789aa00640 1 -- 172.21.15.69:0/2445830069 shutdown_connections 2024-04-24T08:13:29.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.513+0000 7f789aa00640 1 -- 172.21.15.69:0/2445830069 wait complete. 2024-04-24T08:13:29.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.515+0000 7f789aa00640 1 Processor -- start 2024-04-24T08:13:29.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.515+0000 7f789aa00640 1 -- start start 2024-04-24T08:13:29.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.516+0000 7f789aa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 0x7f789415f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.516+0000 7f789aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 0x7f78941f4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.516+0000 7f789aa00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7894145fe0 0x7f789415d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.516+0000 7f789aa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7894145770 con 0x7f7894145a90 2024-04-24T08:13:29.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.516+0000 7f789aa00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f78940cff50 con 0x7f7894145fe0 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.516+0000 7f789aa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f78941458f0 con 0x7f78940d0300 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.516+0000 7f7893400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 0x7f78941f4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.516+0000 7f7893400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 0x7f78941f4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47400/0 (socket says 172.21.15.69:47400) 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.516+0000 7f7893400640 1 -- 172.21.15.69:0/2119492314 learned_addr learned my addr 172.21.15.69:0/2119492314 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f7898a00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7894145fe0 0x7f789415d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f7893e00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 0x7f789415f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f7893400640 1 -- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 msgr2=0x7f789415f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f7893400640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 0x7f789415f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f7893400640 1 -- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7894145fe0 msgr2=0x7f789415d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f7893400640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7894145fe0 0x7f789415d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f7893400640 1 -- 172.21.15.69:0/2119492314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78941f57c0 con 0x7f7894145a90 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f7893e00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 0x7f789415f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f7898a00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7894145fe0 0x7f789415d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:29.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f7893400640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 0x7f78941f4820 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f788002a210 tx=0x7f78800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:29.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.517+0000 7f788be00640 1 -- 172.21.15.69:0/2119492314 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880041070 con 0x7f7894145a90 2024-04-24T08:13:29.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.518+0000 7f788be00640 1 -- 172.21.15.69:0/2119492314 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f788003c870 con 0x7f7894145a90 2024-04-24T08:13:29.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.518+0000 7f789aa00640 1 -- 172.21.15.69:0/2119492314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78941f4f20 con 0x7f7894145a90 2024-04-24T08:13:29.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.518+0000 7f788be00640 1 -- 172.21.15.69:0/2119492314 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880037040 con 0x7f7894145a90 2024-04-24T08:13:29.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.518+0000 7f789aa00640 1 -- 172.21.15.69:0/2119492314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78941f54e0 con 0x7f7894145a90 2024-04-24T08:13:29.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.519+0000 7f789aa00640 1 -- 172.21.15.69:0/2119492314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78940afcb0 con 0x7f7894145a90 2024-04-24T08:13:29.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.520+0000 7f788be00640 1 -- 172.21.15.69:0/2119492314 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7880044070 con 0x7f7894145a90 2024-04-24T08:13:29.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.521+0000 7f788be00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f787406c8f0 0x7f787406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:29.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.521+0000 7f788be00640 1 -- 172.21.15.69:0/2119492314 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f78800ac6f0 con 0x7f7894145a90 2024-04-24T08:13:29.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.521+0000 7f7893e00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f787406c8f0 0x7f787406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:29.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.523+0000 7f7893e00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f787406c8f0 0x7f787406edb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f788400b840 tx=0x7f7884008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:29.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.525+0000 7f788be00640 1 -- 172.21.15.69:0/2119492314 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78940afcb0 con 0x7f7894145a90 2024-04-24T08:13:29.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.735+0000 7f789aa00640 1 -- 172.21.15.69:0/2119492314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7894068a80 con 0x7f7894145a90 2024-04-24T08:13:29.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.736+0000 7f788be00640 1 -- 172.21.15.69:0/2119492314 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v68) ==== 80+0+0 (secure 0 0 0) 0x7f788007ab10 con 0x7f7894145a90 2024-04-24T08:13:29.738 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:13:29.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 -- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f787406c8f0 msgr2=0x7f787406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f787406c8f0 0x7f787406edb0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f788400b840 tx=0x7f7884008040 comp rx=0 tx=0).stop 2024-04-24T08:13:29.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 -- 172.21.15.69:0/2119492314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 msgr2=0x7f78941f4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:29.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 0x7f78941f4820 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f788002a210 tx=0x7f78800046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 -- 172.21.15.69:0/2119492314 shutdown_connections 2024-04-24T08:13:29.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f787406c8f0 0x7f787406edb0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7894145a90 0x7f78941f4820 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7894145fe0 0x7f789415d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 --2- 172.21.15.69:0/2119492314 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78940d0300 0x7f789415f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:29.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 -- 172.21.15.69:0/2119492314 >> 172.21.15.69:0/2119492314 conn(0x7f7894067d90 msgr2=0x7f789406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:29.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 -- 172.21.15.69:0/2119492314 shutdown_connections 2024-04-24T08:13:29.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:29.739+0000 7f789aa00640 1 -- 172.21.15.69:0/2119492314 wait complete. 2024-04-24T08:13:29.759 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:13:29.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:29.759 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:29.827 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[57597]: fuse finished with error 0 and tester_r 0 2024-04-24T08:13:29.833 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:13:35.934 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:13:35.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:35.935 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:13:35.993 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:36.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.195+0000 7f297ce00640 1 Processor -- start 2024-04-24T08:13:36.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.195+0000 7f297ce00640 1 -- start start 2024-04-24T08:13:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.196+0000 7f297ce00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f297806f2d0 0x7f297806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.196+0000 7f297ce00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29780cfbe0 0x7f29780d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.196+0000 7f297ce00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29780b46e0 con 0x7f297815b490 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.197+0000 7f297ce00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f297806fc70 con 0x7f297806f2d0 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.197+0000 7f297ce00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f29780b5890 con 0x7f29780cfbe0 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.197+0000 7f2975400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f297806f2d0 0x7f297806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.197+0000 7f2975400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f297806f2d0 0x7f297806f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47402/0 (socket says 172.21.15.69:47402) 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.197+0000 7f2975400640 1 -- 172.21.15.69:0/2219023431 learned_addr learned my addr 172.21.15.69:0/2219023431 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.197+0000 7f2975400640 1 -- 172.21.15.69:0/2219023431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29780cfbe0 msgr2=0x7f29780d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.197+0000 7f2976800640 1 --2- 172.21.15.69:0/2219023431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29780cfbe0 0x7f29780d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.197+0000 7f2975400640 1 --2- 172.21.15.69:0/2219023431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29780cfbe0 0x7f29780d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.197+0000 7f2975400640 1 -- 172.21.15.69:0/2219023431 >> v1:172.21.15.121:6789/0 conn(0x7f297815b490 legacy=0x7f29780b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.197+0000 7f2975400640 1 -- 172.21.15.69:0/2219023431 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29780d0960 con 0x7f297806f2d0 2024-04-24T08:13:36.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.199+0000 7f2975400640 1 --2- 172.21.15.69:0/2219023431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297806f2d0 0x7f297806f730 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2964009870 tx=0x7f296402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2d98c5e45c2ce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:36.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.199+0000 7f2974a00640 1 -- 172.21.15.69:0/2219023431 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964037070 con 0x7f297806f2d0 2024-04-24T08:13:36.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.199+0000 7f2974a00640 1 -- 172.21.15.69:0/2219023431 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2964032440 con 0x7f297806f2d0 2024-04-24T08:13:36.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.199+0000 7f2974a00640 1 -- 172.21.15.69:0/2219023431 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964032810 con 0x7f297806f2d0 2024-04-24T08:13:36.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.200+0000 7f297ce00640 1 -- 172.21.15.69:0/2219023431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297806f2d0 msgr2=0x7f297806f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.200+0000 7f297ce00640 1 --2- 172.21.15.69:0/2219023431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297806f2d0 0x7f297806f730 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2964009870 tx=0x7f296402a210 comp rx=0 tx=0).stop 2024-04-24T08:13:36.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.200+0000 7f297ce00640 1 -- 172.21.15.69:0/2219023431 shutdown_connections 2024-04-24T08:13:36.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.200+0000 7f297ce00640 1 --2- 172.21.15.69:0/2219023431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29780cfbe0 0x7f29780d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.200+0000 7f297ce00640 1 --2- 172.21.15.69:0/2219023431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297806f2d0 0x7f297806f730 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.200+0000 7f297ce00640 1 -- 172.21.15.69:0/2219023431 >> 172.21.15.69:0/2219023431 conn(0x7f2978067d90 msgr2=0x7f297806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:36.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.200+0000 7f297ce00640 1 -- 172.21.15.69:0/2219023431 shutdown_connections 2024-04-24T08:13:36.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.200+0000 7f297ce00640 1 -- 172.21.15.69:0/2219023431 wait complete. 2024-04-24T08:13:36.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.201+0000 7f297ce00640 1 Processor -- start 2024-04-24T08:13:36.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.202+0000 7f297ce00640 1 -- start start 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.202+0000 7f297ce00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f297806f2d0 0x7f297814e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.202+0000 7f297ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29780cfbe0 0x7f297815a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.202+0000 7f297ce00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297815b490 0x7f297815a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f297ce00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29780d0c40 con 0x7f29780cfbe0 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f297ce00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f29780b4c60 con 0x7f297806f2d0 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f297ce00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f297815b870 con 0x7f297815b490 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2975400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29780cfbe0 0x7f297815a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2975400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29780cfbe0 0x7f297815a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47408/0 (socket says 172.21.15.69:47408) 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2975400640 1 -- 172.21.15.69:0/2207339267 learned_addr learned my addr 172.21.15.69:0/2207339267 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2975e00640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f297806f2d0 0x7f297814e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2975400640 1 -- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297815b490 msgr2=0x7f297815a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2976800640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297815b490 0x7f297815a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2975400640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297815b490 0x7f297815a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2975400640 1 -- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f297806f2d0 msgr2=0x7f297814e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2975400640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f297806f2d0 0x7f297814e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2975400640 1 -- 172.21.15.69:0/2207339267 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297815d460 con 0x7f29780cfbe0 2024-04-24T08:13:36.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.203+0000 7f2975e00640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f297806f2d0 0x7f297814e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:36.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.204+0000 7f2975400640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29780cfbe0 0x7f297815a1d0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f2964009870 tx=0x7f29640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:36.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.204+0000 7f296e000640 1 -- 172.21.15.69:0/2207339267 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964041020 con 0x7f29780cfbe0 2024-04-24T08:13:36.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.204+0000 7f296e000640 1 -- 172.21.15.69:0/2207339267 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2964030970 con 0x7f29780cfbe0 2024-04-24T08:13:36.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.204+0000 7f296e000640 1 -- 172.21.15.69:0/2207339267 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964037040 con 0x7f29780cfbe0 2024-04-24T08:13:36.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.204+0000 7f297ce00640 1 -- 172.21.15.69:0/2207339267 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2978152cb0 con 0x7f29780cfbe0 2024-04-24T08:13:36.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.204+0000 7f297ce00640 1 -- 172.21.15.69:0/2207339267 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29781f48e0 con 0x7f29780cfbe0 2024-04-24T08:13:36.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.205+0000 7f297ce00640 1 -- 172.21.15.69:0/2207339267 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2978068a80 con 0x7f29780cfbe0 2024-04-24T08:13:36.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.206+0000 7f296e000640 1 -- 172.21.15.69:0/2207339267 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2964044020 con 0x7f29780cfbe0 2024-04-24T08:13:36.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.207+0000 7f296e000640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 0x7f295806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.207+0000 7f296e000640 1 -- 172.21.15.69:0/2207339267 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f29640ac5c0 con 0x7f29780cfbe0 2024-04-24T08:13:36.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.208+0000 7f2975e00640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 0x7f295806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.209+0000 7f2975e00640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 0x7f295806edb0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2968003eb0 tx=0x7f2968009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:36.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.212+0000 7f296e000640 1 -- 172.21.15.69:0/2207339267 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f296404b030 con 0x7f29780cfbe0 2024-04-24T08:13:36.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.448+0000 7f297ce00640 1 -- 172.21.15.69:0/2207339267 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f297806cd50 con 0x7f29780cfbe0 2024-04-24T08:13:36.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.450+0000 7f296e000640 1 -- 172.21.15.69:0/2207339267 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+5186 (secure 0 0 0) 0x7f297815d460 con 0x7f29780cfbe0 2024-04-24T08:13:36.451 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:36.451 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":6792,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:13:18.588797+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6804},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6804":{"gid":6804,"name":"b","rank":0,"incarnation":40,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1135108804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1135108804},{"type":"v1","addr":"172.21.15.121:6837","nonce":1135108804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6804,"qdb_cluster":[6804]},"id":5}]} 2024-04-24T08:13:36.451 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 44 2024-04-24T08:13:36.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 -- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 msgr2=0x7f295806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 0x7f295806edb0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2968003eb0 tx=0x7f2968009040 comp rx=0 tx=0).stop 2024-04-24T08:13:36.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 -- 172.21.15.69:0/2207339267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29780cfbe0 msgr2=0x7f297815a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29780cfbe0 0x7f297815a1d0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f2964009870 tx=0x7f29640046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 -- 172.21.15.69:0/2207339267 shutdown_connections 2024-04-24T08:13:36.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 0x7f295806edb0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29780cfbe0 0x7f297815a1d0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297815b490 0x7f297815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 --2- 172.21.15.69:0/2207339267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f297806f2d0 0x7f297814e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 -- 172.21.15.69:0/2207339267 >> 172.21.15.69:0/2207339267 conn(0x7f2978067d90 msgr2=0x7f29781583e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:36.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 -- 172.21.15.69:0/2207339267 shutdown_connections 2024-04-24T08:13:36.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.452+0000 7f297ce00640 1 -- 172.21.15.69:0/2207339267 wait complete. 2024-04-24T08:13:36.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-04-24T08:13:36.725 INFO:teuthology.orchestra.run.smithi121.stdout:{"message":"","return_code":0} 2024-04-24T08:13:36.725 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-24T08:13:36.725 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:13:36.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.931+0000 7f933f000640 1 Processor -- start 2024-04-24T08:13:36.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.932+0000 7f933f000640 1 -- start start 2024-04-24T08:13:36.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.933+0000 7f933f000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f933806cfb0 0x7f93380cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.933+0000 7f933f000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933815ed40 con 0x7f933815e960 2024-04-24T08:13:36.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.933+0000 7f933f000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93380cf050 con 0x7f933806d410 2024-04-24T08:13:36.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.933+0000 7f933f000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9338162b50 con 0x7f933806cfb0 2024-04-24T08:13:36.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.933+0000 7f933dc00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f933815e960 0x7f9338162440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35688/0 (socket says 172.21.15.69:35688) 2024-04-24T08:13:36.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.933+0000 7f933dc00640 1 -- 172.21.15.69:0/1478509230 learned_addr learned my addr 172.21.15.69:0/1478509230 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2468801828 0 0) 0x7f933815ed40 con 0x7f933815e960 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9320003520 con 0x7f933815e960 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f933d200640 1 --2- 172.21.15.69:0/1478509230 >> v2:172.21.15.121:3300/0 conn(0x7f933806cfb0 0x7f93380cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3789450599 0 0) 0x7f9320003520 con 0x7f933815e960 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 >> v2:172.21.15.121:3300/0 conn(0x7f933806cfb0 msgr2=0x7f93380cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 --2- 172.21.15.69:0/1478509230 >> v2:172.21.15.121:3300/0 conn(0x7f933806cfb0 0x7f93380cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 >> v1:172.21.15.121:6789/0 conn(0x7f933806d410 legacy=0x7f93380d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9338162e20 con 0x7f933815e960 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f932c001670 con 0x7f933815e960 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 >> v1:172.21.15.69:6789/0 conn(0x7f933815e960 legacy=0x7f9338162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9320003a00 0x7f9320003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9320004800 0x7f9320004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9320005210 0x7f932000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9338162b50 con 0x7f9320003a00 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f932000c140 con 0x7f9320005210 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.934+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f932000c340 con 0x7f9320004800 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.935+0000 7f933dc00640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9320003a00 0x7f9320003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.935+0000 7f933d200640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9320004800 0x7f9320004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.935+0000 7f933e600640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9320005210 0x7f932000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.935+0000 7f933dc00640 1 -- 172.21.15.69:0/1478509230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9320004800 msgr2=0x7f9320004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.935+0000 7f933dc00640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9320004800 0x7f9320004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.935+0000 7f933dc00640 1 -- 172.21.15.69:0/1478509230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9320005210 msgr2=0x7f932000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.936+0000 7f933dc00640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9320005210 0x7f932000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.936+0000 7f933dc00640 1 -- 172.21.15.69:0/1478509230 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9320003520 con 0x7f9320003a00 2024-04-24T08:13:36.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.936+0000 7f933e600640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9320005210 0x7f932000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:36.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.936+0000 7f933e600640 1 -- 172.21.15.69:0/1478509230 reap_dead start 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.937+0000 7f933dc00640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9320003a00 0x7f9320003e00 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f932c00c160 tx=0x7f9320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=368176fa814f4f1e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.937+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c007c60 con 0x7f9320003a00 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.937+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f932c005ad0 con 0x7f9320003a00 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.937+0000 7f9337e00640 1 -- 172.21.15.69:0/1478509230 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f932c007070 con 0x7f9320003a00 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.937+0000 7f933f000640 1 -- 172.21.15.69:0/1478509230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9320003a00 msgr2=0x7f9320003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.937+0000 7f933f000640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9320003a00 0x7f9320003e00 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f932c00c160 tx=0x7f9320005830 comp rx=0 tx=0).stop 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.938+0000 7f933f000640 1 -- 172.21.15.69:0/1478509230 shutdown_connections 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.938+0000 7f933f000640 1 --2- 172.21.15.69:0/1478509230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9320003a00 0x7f9320003e00 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.938+0000 7f933f000640 1 -- 172.21.15.69:0/1478509230 >> 172.21.15.69:0/1478509230 conn(0x7f933809d900 msgr2=0x7f933808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.938+0000 7f933f000640 1 -- 172.21.15.69:0/1478509230 shutdown_connections 2024-04-24T08:13:36.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.938+0000 7f933f000640 1 -- 172.21.15.69:0/1478509230 wait complete. 2024-04-24T08:13:36.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.939+0000 7f933f000640 1 Processor -- start 2024-04-24T08:13:36.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.939+0000 7f933f000640 1 -- start start 2024-04-24T08:13:36.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.940+0000 7f933f000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9320003a00 0x7f93381f8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.940+0000 7f933f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9338152370 0x7f93380b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.940+0000 7f933f000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93381527f0 0x7f93380b2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.940+0000 7f933f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f93380cf5d0 con 0x7f9338152370 2024-04-24T08:13:36.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.940+0000 7f933f000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f93380cf2f0 con 0x7f93381527f0 2024-04-24T08:13:36.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.940+0000 7f933f000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f93380b6170 con 0x7f9320003a00 2024-04-24T08:13:36.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9338152370 0x7f93380b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9338152370 0x7f93380b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47430/0 (socket says 172.21.15.69:47430) 2024-04-24T08:13:36.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933d200640 1 -- 172.21.15.69:0/1173443617 learned_addr learned my addr 172.21.15.69:0/1173443617 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:36.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933e600640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93381527f0 0x7f93380b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933d200640 1 -- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9320003a00 msgr2=0x7f93381f8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933d200640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9320003a00 0x7f93381f8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933d200640 1 -- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93381527f0 msgr2=0x7f93380b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933d200640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93381527f0 0x7f93380b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933d200640 1 -- 172.21.15.69:0/1173443617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93381f8dc0 con 0x7f9338152370 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933e600640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93381527f0 0x7f93380b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.941+0000 7f933d200640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9338152370 0x7f93380b22e0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f932800f630 tx=0x7f932800faf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.942+0000 7f9335600640 1 -- 172.21.15.69:0/1173443617 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328019020 con 0x7f9338152370 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.942+0000 7f9335600640 1 -- 172.21.15.69:0/1173443617 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9328014870 con 0x7f9338152370 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.942+0000 7f9335600640 1 -- 172.21.15.69:0/1173443617 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9328014c20 con 0x7f9338152370 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.942+0000 7f933f000640 1 -- 172.21.15.69:0/1173443617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9338161690 con 0x7f9338152370 2024-04-24T08:13:36.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.942+0000 7f933f000640 1 -- 172.21.15.69:0/1173443617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9338161c10 con 0x7f9338152370 2024-04-24T08:13:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.943+0000 7f933f000640 1 -- 172.21.15.69:0/1173443617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9338075d50 con 0x7f9338152370 2024-04-24T08:13:36.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.944+0000 7f9335600640 1 -- 172.21.15.69:0/1173443617 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f932801c020 con 0x7f9338152370 2024-04-24T08:13:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.945+0000 7f9335600640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 0x7f931c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.945+0000 7f9335600640 1 -- 172.21.15.69:0/1173443617 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 5709+0+0 (secure 0 0 0) 0x7f932808c380 con 0x7f9338152370 2024-04-24T08:13:36.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.945+0000 7f933dc00640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 0x7f931c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:36.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.947+0000 7f933dc00640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 0x7f931c06edb0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f932c032000 tx=0x7f932c009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:36.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:36.949+0000 7f9335600640 1 -- 172.21.15.69:0/1173443617 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9328023020 con 0x7f9338152370 2024-04-24T08:13:37.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.183+0000 7f933f000640 1 -- 172.21.15.69:0/1173443617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f93380d1dc0 con 0x7f9338152370 2024-04-24T08:13:37.186 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:13:37.185+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:13:37.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.192+0000 7f9335600640 1 -- 172.21.15.69:0/1173443617 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v45) ==== 161+0+0 (secure 0 0 0) 0x7f93381f8dc0 con 0x7f9338152370 2024-04-24T08:13:37.194 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:13:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.195+0000 7f933f000640 1 -- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 msgr2=0x7f931c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.195+0000 7f933f000640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 0x7f931c06edb0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f932c032000 tx=0x7f932c009030 comp rx=0 tx=0).stop 2024-04-24T08:13:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.195+0000 7f933f000640 1 -- 172.21.15.69:0/1173443617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9338152370 msgr2=0x7f93380b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.195+0000 7f933f000640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9338152370 0x7f93380b22e0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f932800f630 tx=0x7f932800faf0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.195+0000 7f933f000640 1 -- 172.21.15.69:0/1173443617 shutdown_connections 2024-04-24T08:13:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.195+0000 7f933f000640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f931c06c8f0 0x7f931c06edb0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.195+0000 7f933f000640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9338152370 0x7f93380b22e0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.195+0000 7f933f000640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93381527f0 0x7f93380b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.195+0000 7f933f000640 1 --2- 172.21.15.69:0/1173443617 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9320003a00 0x7f93381f8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.195+0000 7f933f000640 1 -- 172.21.15.69:0/1173443617 >> 172.21.15.69:0/1173443617 conn(0x7f933809d900 msgr2=0x7f93380d1510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.196+0000 7f933f000640 1 -- 172.21.15.69:0/1173443617 shutdown_connections 2024-04-24T08:13:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.196+0000 7f933f000640 1 -- 172.21.15.69:0/1173443617 wait complete. 2024-04-24T08:13:37.216 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-04-24T08:13:37.226 INFO:tasks.ceph.mds.b.smithi121.stderr: -3607> 2024-04-24T08:13:22.015+0000 7f4f32600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:13:37.312 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:13:37.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.318+0000 7f046a51bf40 1 Processor -- start 2024-04-24T08:13:37.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.318+0000 7f046a51bf40 1 -- start start 2024-04-24T08:13:37.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f046a51bf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae45abded0 0x55ae45a74b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f046a51bf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae45abe2b0 0x55ae45a763b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f046a51bf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45b48ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f046a51bf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ae45b4bc10 con 0x55ae45b4b5f0 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f046a51bf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ae45a1ccc0 con 0x55ae45abded0 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f046a51bf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ae45aa23b0 con 0x55ae45abe2b0 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f0467800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45b48ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f0466e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae45abded0 0x55ae45a74b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f0467800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45b48ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47446/0 (socket says 172.21.15.69:47446) 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f0466400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae45abe2b0 0x55ae45a763b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.319+0000 7f0467800640 1 -- 172.21.15.69:0/2024400847 learned_addr learned my addr 172.21.15.69:0/2024400847 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.320+0000 7f0467800640 1 -- 172.21.15.69:0/2024400847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae45abe2b0 msgr2=0x55ae45a763b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:37.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.320+0000 7f0467800640 1 --2- 172.21.15.69:0/2024400847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae45abe2b0 0x55ae45a763b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.320+0000 7f0467800640 1 -- 172.21.15.69:0/2024400847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae45abded0 msgr2=0x55ae45a74b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:37.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.320+0000 7f0467800640 1 --2- 172.21.15.69:0/2024400847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae45abded0 0x55ae45a74b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.320+0000 7f0467800640 1 -- 172.21.15.69:0/2024400847 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae45b4a820 con 0x55ae45b4b5f0 2024-04-24T08:13:37.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.320+0000 7f0466400640 1 --2- 172.21.15.69:0/2024400847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae45abe2b0 0x55ae45a763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:13:37.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.320+0000 7f0467800640 1 --2- 172.21.15.69:0/2024400847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45b48ad0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f046000a810 tx=0x7f0460005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7673fe6418b676a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:37.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.320+0000 7f0465000640 1 -- 172.21.15.69:0/2024400847 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046000d9c0 con 0x55ae45b4b5f0 2024-04-24T08:13:37.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.320+0000 7f0465000640 1 -- 172.21.15.69:0/2024400847 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f046000db60 con 0x55ae45b4b5f0 2024-04-24T08:13:37.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.320+0000 7f0465000640 1 -- 172.21.15.69:0/2024400847 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460012450 con 0x55ae45b4b5f0 2024-04-24T08:13:37.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.321+0000 7f046a51bf40 1 -- 172.21.15.69:0/2024400847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 msgr2=0x55ae45b48ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.321+0000 7f046a51bf40 1 --2- 172.21.15.69:0/2024400847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45b48ad0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f046000a810 tx=0x7f0460005d80 comp rx=0 tx=0).stop 2024-04-24T08:13:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.321+0000 7f046a51bf40 1 -- 172.21.15.69:0/2024400847 shutdown_connections 2024-04-24T08:13:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.321+0000 7f046a51bf40 1 --2- 172.21.15.69:0/2024400847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45b48ad0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.321+0000 7f046a51bf40 1 --2- 172.21.15.69:0/2024400847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae45abe2b0 0x55ae45a763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.321+0000 7f046a51bf40 1 --2- 172.21.15.69:0/2024400847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae45abded0 0x55ae45a74b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.321+0000 7f046a51bf40 1 -- 172.21.15.69:0/2024400847 >> 172.21.15.69:0/2024400847 conn(0x55ae45aa2e40 msgr2=0x55ae45b47a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.321+0000 7f046a51bf40 1 -- 172.21.15.69:0/2024400847 shutdown_connections 2024-04-24T08:13:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.321+0000 7f046a51bf40 1 -- 172.21.15.69:0/2024400847 wait complete. 2024-04-24T08:13:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.321+0000 7f046a51bf40 1 Processor -- start 2024-04-24T08:13:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.322+0000 7f046a51bf40 1 -- start start 2024-04-24T08:13:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.322+0000 7f046a51bf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae45abded0 0x55ae45b49080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.322+0000 7f046a51bf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae45abe2b0 0x55ae45be2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.322+0000 7f046a51bf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45a74b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.322+0000 7f046a51bf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ae45b476d0 con 0x55ae45b4b5f0 2024-04-24T08:13:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.322+0000 7f046a51bf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ae45aa06e0 con 0x55ae45abded0 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.323+0000 7f046a51bf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ae45b43de0 con 0x55ae45abe2b0 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.323+0000 7f0467800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45a74b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.323+0000 7f0467800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45a74b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47460/0 (socket says 172.21.15.69:47460) 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.323+0000 7f0467800640 1 -- 172.21.15.69:0/1965491568 learned_addr learned my addr 172.21.15.69:0/1965491568 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.323+0000 7f0467800640 1 -- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae45abe2b0 msgr2=0x55ae45be2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.323+0000 7f0467800640 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ae45abe2b0 0x55ae45be2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.323+0000 7f0467800640 1 -- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae45abded0 msgr2=0x55ae45b49080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.323+0000 7f0467800640 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ae45abded0 0x55ae45b49080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.323+0000 7f0467800640 1 -- 172.21.15.69:0/1965491568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae45a78120 con 0x55ae45b4b5f0 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.323+0000 7f0467800640 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45a74b70 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f0460003000 tx=0x7f04600033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.324+0000 7f045f400640 1 -- 172.21.15.69:0/1965491568 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460021070 con 0x55ae45b4b5f0 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.324+0000 7f045f400640 1 -- 172.21.15.69:0/1965491568 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0460026680 con 0x55ae45b4b5f0 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.324+0000 7f046a51bf40 1 -- 172.21.15.69:0/1965491568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ae45a78400 con 0x55ae45b4b5f0 2024-04-24T08:13:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.324+0000 7f045f400640 1 -- 172.21.15.69:0/1965491568 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046000d9c0 con 0x55ae45b4b5f0 2024-04-24T08:13:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.324+0000 7f046a51bf40 1 -- 172.21.15.69:0/1965491568 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae45b3c220 con 0x55ae45b4b5f0 2024-04-24T08:13:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.326+0000 7f045f400640 1 -- 172.21.15.69:0/1965491568 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f046000dd10 con 0x55ae45b4b5f0 2024-04-24T08:13:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.327+0000 7f045f400640 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f044806b2d0 0x7f044806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.327+0000 7f045f400640 1 -- 172.21.15.69:0/1965491568 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f046008cf90 con 0x55ae45b4b5f0 2024-04-24T08:13:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.327+0000 7f046a51bf40 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55ae45b3c740 0x55ae45b3cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.327+0000 7f046a51bf40 1 -- 172.21.15.69:0/1965491568 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:1 10.10 10:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69) -- 0x55ae45b3d100 con 0x55ae45b3c740 2024-04-24T08:13:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.327+0000 7f0466e00640 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f044806b2d0 0x7f044806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.327+0000 7f0466400640 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55ae45b3c740 0x55ae45b3cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.328+0000 7f0466e00640 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f044806b2d0 0x7f044806d790 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0458004170 tx=0x7f04580016f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:37.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.328+0000 7f0466400640 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55ae45b3c740 0x55ae45b3cbc0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:37.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f0466400640 1 -- 172.21.15.69:0/1965491568 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v69'11 uv11 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f045405c040 con 0x55ae45b3c740 2024-04-24T08:13:37.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f046a51bf40 1 -- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55ae45b3c740 msgr2=0x55ae45b3cbc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f046a51bf40 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55ae45b3c740 0x55ae45b3cbc0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f046a51bf40 1 -- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f044806b2d0 msgr2=0x7f044806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f046a51bf40 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f044806b2d0 0x7f044806d790 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0458004170 tx=0x7f04580016f0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f046a51bf40 1 -- 172.21.15.69:0/1965491568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 msgr2=0x55ae45a74b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f046a51bf40 1 --2- 172.21.15.69:0/1965491568 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae45b4b5f0 0x55ae45a74b70 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f0460003000 tx=0x7f04600033a0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f0467800640 1 -- 172.21.15.69:0/1965491568 reap_dead start 2024-04-24T08:13:37.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f046a51bf40 1 -- 172.21.15.69:0/1965491568 shutdown_connections 2024-04-24T08:13:37.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f046a51bf40 1 -- 172.21.15.69:0/1965491568 >> 172.21.15.69:0/1965491568 conn(0x55ae45aa2e40 msgr2=0x55ae45b49c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:37.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f046a51bf40 1 -- 172.21.15.69:0/1965491568 shutdown_connections 2024-04-24T08:13:37.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.330+0000 7f046a51bf40 1 -- 172.21.15.69:0/1965491568 wait complete. 2024-04-24T08:13:37.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-24T08:13:37.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.524+0000 7f2faac00640 1 Processor -- start 2024-04-24T08:13:37.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.525+0000 7f2faac00640 1 -- start start 2024-04-24T08:13:37.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.526+0000 7f2faac00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa415e980 0x7f2fa4162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.526+0000 7f2faac00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2fa406cfb0 0x7f2fa406d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.526+0000 7f2faac00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fa415ed60 con 0x7f2fa40d0da0 2024-04-24T08:13:37.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.526+0000 7f2faac00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2fa40d1580 con 0x7f2fa406cfb0 2024-04-24T08:13:37.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.526+0000 7f2faac00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa40d1860 con 0x7f2fa415e980 2024-04-24T08:13:37.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.527+0000 7f2fa8e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2fa40d0da0 0x7f2fa40cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35698/0 (socket says 172.21.15.69:35698) 2024-04-24T08:13:37.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.527+0000 7f2fa8e00640 1 -- 172.21.15.69:0/4142674122 learned_addr learned my addr 172.21.15.69:0/4142674122 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:37.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.527+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1983864437 0 0) 0x7f2fa415ed60 con 0x7f2fa40d0da0 2024-04-24T08:13:37.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.527+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f8c003520 con 0x7f2fa40d0da0 2024-04-24T08:13:37.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.527+0000 7f2faa200640 1 --2- 172.21.15.69:0/4142674122 >> v2:172.21.15.121:3300/0 conn(0x7f2fa406cfb0 0x7f2fa406d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.527+0000 7f2fa9800640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa415e980 0x7f2fa4162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2156282213 0 0) 0x7f2f8c003520 con 0x7f2fa40d0da0 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa415e980 msgr2=0x7f2fa4162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa415e980 0x7f2fa4162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 >> v2:172.21.15.121:3300/0 conn(0x7f2fa406cfb0 msgr2=0x7f2fa406d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 --2- 172.21.15.69:0/4142674122 >> v2:172.21.15.121:3300/0 conn(0x7f2fa406cfb0 0x7f2fa406d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa40cf450 con 0x7f2fa40d0da0 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f2f90001670 con 0x7f2fa40d0da0 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 >> v1:172.21.15.69:6789/0 conn(0x7f2fa40d0da0 legacy=0x7f2fa40cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c0039b0 0x7f2f8c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa415e980 0x7f2f8c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 0x7f2f8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa40d1580 con 0x7f2f8c004d40 2024-04-24T08:13:37.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa40d1860 con 0x7f2f8c0039b0 2024-04-24T08:13:37.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2f8c00bb80 con 0x7f2fa415e980 2024-04-24T08:13:37.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2faa200640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 0x7f2f8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa9800640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c0039b0 0x7f2f8c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.528+0000 7f2fa8e00640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa415e980 0x7f2f8c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.529+0000 7f2faa200640 1 -- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa415e980 msgr2=0x7f2f8c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:37.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.529+0000 7f2faa200640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa415e980 0x7f2f8c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.529+0000 7f2faa200640 1 -- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c0039b0 msgr2=0x7f2f8c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:37.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.529+0000 7f2faa200640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f8c0039b0 0x7f2f8c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.529+0000 7f2faa200640 1 -- 172.21.15.69:0/4142674122 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8c003520 con 0x7f2f8c004d40 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.530+0000 7f2faa200640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 0x7f2f8c0051c0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c007360 tx=0x7f2f8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9e9aca8f1222e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.530+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c009be0 con 0x7f2f8c004d40 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.530+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f9c02e620 con 0x7f2f8c004d40 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.530+0000 7f2fa3e00640 1 -- 172.21.15.69:0/4142674122 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c02e9f0 con 0x7f2f8c004d40 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.530+0000 7f2faac00640 1 -- 172.21.15.69:0/4142674122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 msgr2=0x7f2f8c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.530+0000 7f2faac00640 1 --2- 172.21.15.69:0/4142674122 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 0x7f2f8c0051c0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c007360 tx=0x7f2f8c005830 comp rx=0 tx=0).stop 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.530+0000 7f2faa200640 1 -- 172.21.15.69:0/4142674122 reap_dead start 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.531+0000 7f2faac00640 1 -- 172.21.15.69:0/4142674122 shutdown_connections 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.531+0000 7f2faac00640 1 -- 172.21.15.69:0/4142674122 >> 172.21.15.69:0/4142674122 conn(0x7f2fa409d920 msgr2=0x7f2fa408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.531+0000 7f2faac00640 1 -- 172.21.15.69:0/4142674122 shutdown_connections 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.531+0000 7f2faac00640 1 -- 172.21.15.69:0/4142674122 wait complete. 2024-04-24T08:13:37.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.532+0000 7f2faac00640 1 Processor -- start 2024-04-24T08:13:37.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.533+0000 7f2faac00640 1 -- start start 2024-04-24T08:13:37.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.533+0000 7f2faac00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 0x7f2fa41f88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2faac00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa414b2b0 0x7f2fa41f41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2faac00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fa415ed60 0x7f2fa40b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2faac00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa40d11e0 con 0x7f2f8c004d40 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2faac00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2fa40d0360 con 0x7f2fa415ed60 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2faac00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2fa40d1360 con 0x7f2fa414b2b0 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2fa9800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 0x7f2fa41f88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2fa9800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 0x7f2fa41f88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47484/0 (socket says 172.21.15.69:47484) 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2fa9800640 1 -- 172.21.15.69:0/2889986283 learned_addr learned my addr 172.21.15.69:0/2889986283 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2fa9800640 1 -- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa414b2b0 msgr2=0x7f2fa41f41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2fa9800640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa414b2b0 0x7f2fa41f41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2fa9800640 1 -- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fa415ed60 msgr2=0x7f2fa40b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2fa9800640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fa415ed60 0x7f2fa40b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2fa9800640 1 -- 172.21.15.69:0/2889986283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fa4161be0 con 0x7f2f8c004d40 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.534+0000 7f2fa9800640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 0x7f2fa41f88c0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800f580 tx=0x7f2f9800fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.535+0000 7f2fa1600640 1 -- 172.21.15.69:0/2889986283 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98013810 con 0x7f2f8c004d40 2024-04-24T08:13:37.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.535+0000 7f2fa1600640 1 -- 172.21.15.69:0/2889986283 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f9801a070 con 0x7f2f8c004d40 2024-04-24T08:13:37.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.535+0000 7f2fa1600640 1 -- 172.21.15.69:0/2889986283 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f98013bb0 con 0x7f2f8c004d40 2024-04-24T08:13:37.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.535+0000 7f2faac00640 1 -- 172.21.15.69:0/2889986283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fa4161900 con 0x7f2f8c004d40 2024-04-24T08:13:37.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.535+0000 7f2faac00640 1 -- 172.21.15.69:0/2889986283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fa40b8e00 con 0x7f2f8c004d40 2024-04-24T08:13:37.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.536+0000 7f2faac00640 1 -- 172.21.15.69:0/2889986283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fa40cf070 con 0x7f2f8c004d40 2024-04-24T08:13:37.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.537+0000 7f2fa1600640 1 -- 172.21.15.69:0/2889986283 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2f98012070 con 0x7f2f8c004d40 2024-04-24T08:13:37.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.538+0000 7f2fa1600640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2f8406c920 0x7f2f8406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:37.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.538+0000 7f2fa1600640 1 -- 172.21.15.69:0/2889986283 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f2f9808bbd0 con 0x7f2f8c004d40 2024-04-24T08:13:37.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.538+0000 7f2fa8e00640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2f8406c920 0x7f2f8406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:37.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.539+0000 7f2fa8e00640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2f8406c920 0x7f2f8406ede0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2fa414b840 tx=0x7f2f90000f50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:37.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.542+0000 7f2fa1600640 1 -- 172.21.15.69:0/2889986283 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2fa40cf070 con 0x7f2f8c004d40 2024-04-24T08:13:37.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:37.763+0000 7f2faac00640 1 -- 172.21.15.69:0/2889986283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f2fa4075d50 con 0x7f2f8c004d40 2024-04-24T08:13:38.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.206+0000 7f2fa1600640 1 -- 172.21.15.69:0/2889986283 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v46) ==== 162+0+0 (secure 0 0 0) 0x7f2f9801c310 con 0x7f2f8c004d40 2024-04-24T08:13:38.208 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-24T08:13:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.209+0000 7f2faac00640 1 -- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2f8406c920 msgr2=0x7f2f8406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.209+0000 7f2faac00640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2f8406c920 0x7f2f8406ede0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2fa414b840 tx=0x7f2f90000f50 comp rx=0 tx=0).stop 2024-04-24T08:13:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.209+0000 7f2faac00640 1 -- 172.21.15.69:0/2889986283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 msgr2=0x7f2fa41f88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.209+0000 7f2faac00640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 0x7f2fa41f88c0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f2f9800f580 tx=0x7f2f9800fa40 comp rx=0 tx=0).stop 2024-04-24T08:13:38.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.209+0000 7f2faac00640 1 -- 172.21.15.69:0/2889986283 shutdown_connections 2024-04-24T08:13:38.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.210+0000 7f2faac00640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2f8406c920 0x7f2f8406ede0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.210+0000 7f2faac00640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fa415ed60 0x7f2fa40b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.210+0000 7f2faac00640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fa414b2b0 0x7f2fa41f41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.210+0000 7f2faac00640 1 --2- 172.21.15.69:0/2889986283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2f8c004d40 0x7f2fa41f88c0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.210+0000 7f2faac00640 1 -- 172.21.15.69:0/2889986283 >> 172.21.15.69:0/2889986283 conn(0x7f2fa409d920 msgr2=0x7f2fa40b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:38.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.210+0000 7f2faac00640 1 -- 172.21.15.69:0/2889986283 shutdown_connections 2024-04-24T08:13:38.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.210+0000 7f2faac00640 1 -- 172.21.15.69:0/2889986283 wait complete. 2024-04-24T08:13:38.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:38.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.412+0000 7fcaad000640 1 Processor -- start 2024-04-24T08:13:38.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.412+0000 7fcaad000640 1 -- start start 2024-04-24T08:13:38.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.413+0000 7fcaad000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcaa80b36b0 0x7fcaa80b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:38.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.413+0000 7fcaad000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa80aeac0 con 0x7fcaa81523f0 2024-04-24T08:13:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.413+0000 7fcaad000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaa80aeda0 con 0x7fcaa806db90 2024-04-24T08:13:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.414+0000 7fcaad000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcaa80af080 con 0x7fcaa80b36b0 2024-04-24T08:13:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.414+0000 7fcaa7400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcaa81523f0 0x7fcaa8072e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35700/0 (socket says 172.21.15.69:35700) 2024-04-24T08:13:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.414+0000 7fcaa7400640 1 -- 172.21.15.69:0/2664131760 learned_addr learned my addr 172.21.15.69:0/2664131760 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.414+0000 7fcaa7e00640 1 --2- 172.21.15.69:0/2664131760 >> v2:172.21.15.69:3300/0 conn(0x7fcaa80b36b0 0x7fcaa80b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.414+0000 7fcaa6000640 1 -- 172.21.15.69:0/2664131760 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1122241569 0 0) 0x7fcaa80aeac0 con 0x7fcaa81523f0 2024-04-24T08:13:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.414+0000 7fcaa7e00640 1 -- 172.21.15.69:0/2664131760 >> v1:172.21.15.121:6789/0 conn(0x7fcaa806db90 legacy=0x7fcaa8073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.414+0000 7fcaa7e00640 1 -- 172.21.15.69:0/2664131760 >> v1:172.21.15.69:6789/0 conn(0x7fcaa81523f0 legacy=0x7fcaa8072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.414+0000 7fcaa7e00640 1 -- 172.21.15.69:0/2664131760 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa80af360 con 0x7fcaa80b36b0 2024-04-24T08:13:38.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.415+0000 7fcaa7e00640 1 --2- 172.21.15.69:0/2664131760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa80b36b0 0x7fcaa80b3b40 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0009d90 tx=0x7fcaa002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae95b6e314063e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:38.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.416+0000 7fcaa6000640 1 -- 172.21.15.69:0/2664131760 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0031600 con 0x7fcaa80b36b0 2024-04-24T08:13:38.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.416+0000 7fcaa6000640 1 -- 172.21.15.69:0/2664131760 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaa00317a0 con 0x7fcaa80b36b0 2024-04-24T08:13:38.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.416+0000 7fcaa6000640 1 -- 172.21.15.69:0/2664131760 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0031b70 con 0x7fcaa80b36b0 2024-04-24T08:13:38.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.416+0000 7fcaad000640 1 -- 172.21.15.69:0/2664131760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa80b36b0 msgr2=0x7fcaa80b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:38.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.416+0000 7fcaad000640 1 --2- 172.21.15.69:0/2664131760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa80b36b0 0x7fcaa80b3b40 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0009d90 tx=0x7fcaa002a610 comp rx=0 tx=0).stop 2024-04-24T08:13:38.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.416+0000 7fcaad000640 1 -- 172.21.15.69:0/2664131760 shutdown_connections 2024-04-24T08:13:38.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.416+0000 7fcaad000640 1 --2- 172.21.15.69:0/2664131760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa80b36b0 0x7fcaa80b3b40 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.416+0000 7fcaad000640 1 -- 172.21.15.69:0/2664131760 >> 172.21.15.69:0/2664131760 conn(0x7fcaa80d0160 msgr2=0x7fcaa80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:38.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.416+0000 7fcaad000640 1 -- 172.21.15.69:0/2664131760 shutdown_connections 2024-04-24T08:13:38.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.416+0000 7fcaad000640 1 -- 172.21.15.69:0/2664131760 wait complete. 2024-04-24T08:13:38.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.418+0000 7fcaad000640 1 Processor -- start 2024-04-24T08:13:38.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.418+0000 7fcaad000640 1 -- start start 2024-04-24T08:13:38.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.419+0000 7fcaad000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaa806db90 0x7fcaa81f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.419+0000 7fcaad000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaa80b36b0 0x7fcaa8153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.419+0000 7fcaad000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa81523f0 0x7fcaa8149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.419+0000 7fcaad000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa81527d0 con 0x7fcaa81523f0 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.419+0000 7fcaad000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcaa806dfd0 con 0x7fcaa806db90 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.419+0000 7fcaad000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcaa80b4080 con 0x7fcaa80b36b0 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa81523f0 0x7fcaa8149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa81523f0 0x7fcaa8149fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47504/0 (socket says 172.21.15.69:47504) 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7e00640 1 -- 172.21.15.69:0/802321994 learned_addr learned my addr 172.21.15.69:0/802321994 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7400640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaa806db90 0x7fcaa81f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa6a00640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaa80b36b0 0x7fcaa8153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7e00640 1 -- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaa80b36b0 msgr2=0x7fcaa8153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7e00640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaa80b36b0 0x7fcaa8153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7e00640 1 -- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaa806db90 msgr2=0x7fcaa81f3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:38.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7e00640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaa806db90 0x7fcaa81f3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7e00640 1 -- 172.21.15.69:0/802321994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaa8152af0 con 0x7fcaa81523f0 2024-04-24T08:13:38.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7400640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaa806db90 0x7fcaa81f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:38.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.420+0000 7fcaa7e00640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa81523f0 0x7fcaa8149fe0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0009d90 tx=0x7fcaa0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:38.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.421+0000 7fca9f400640 1 -- 172.21.15.69:0/802321994 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0040070 con 0x7fcaa81523f0 2024-04-24T08:13:38.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.421+0000 7fca9f400640 1 -- 172.21.15.69:0/802321994 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaa0030be0 con 0x7fcaa81523f0 2024-04-24T08:13:38.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.421+0000 7fca9f400640 1 -- 172.21.15.69:0/802321994 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaa0009450 con 0x7fcaa81523f0 2024-04-24T08:13:38.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.421+0000 7fcaad000640 1 -- 172.21.15.69:0/802321994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaa8152dd0 con 0x7fcaa81523f0 2024-04-24T08:13:38.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.421+0000 7fcaad000640 1 -- 172.21.15.69:0/802321994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaa8153390 con 0x7fcaa81523f0 2024-04-24T08:13:38.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.422+0000 7fcaad000640 1 -- 172.21.15.69:0/802321994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaa8075d50 con 0x7fcaa81523f0 2024-04-24T08:13:38.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.423+0000 7fca9f400640 1 -- 172.21.15.69:0/802321994 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcaa0039040 con 0x7fcaa81523f0 2024-04-24T08:13:38.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.424+0000 7fca9f400640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 0x7fca8806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:38.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.424+0000 7fca9f400640 1 -- 172.21.15.69:0/802321994 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7fcaa00acd40 con 0x7fcaa81523f0 2024-04-24T08:13:38.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.425+0000 7fcaa7400640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 0x7fca8806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:38.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.426+0000 7fcaa7400640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 0x7fca8806ede0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fca98000f60 tx=0x7fca98009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:38.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.429+0000 7fca9f400640 1 -- 172.21.15.69:0/802321994 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcaa007b070 con 0x7fcaa81523f0 2024-04-24T08:13:38.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.668+0000 7fcaad000640 1 -- 172.21.15.69:0/802321994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcaa80ad410 con 0x7fcaa81523f0 2024-04-24T08:13:38.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.669+0000 7fca9f400640 1 -- 172.21.15.69:0/802321994 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+5180 (secure 0 0 0) 0x7fcaa007a970 con 0x7fcaa81523f0 2024-04-24T08:13:38.670 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:38.671 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3504199143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3504199143},{"type":"v1","addr":"172.21.15.121:6837","nonce":3504199143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:13:38.212068+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6792},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6792":{"gid":6792,"name":"a","rank":0,"incarnation":47,"state":"up:replay","state_seq":1,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-04-24T08:13:38.671 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 47 2024-04-24T08:13:38.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 -- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 msgr2=0x7fca8806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:38.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 0x7fca8806ede0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fca98000f60 tx=0x7fca98009040 comp rx=0 tx=0).stop 2024-04-24T08:13:38.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 -- 172.21.15.69:0/802321994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa81523f0 msgr2=0x7fcaa8149fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:38.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa81523f0 0x7fcaa8149fe0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0009d90 tx=0x7fcaa0004a40 comp rx=0 tx=0).stop 2024-04-24T08:13:38.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 -- 172.21.15.69:0/802321994 shutdown_connections 2024-04-24T08:13:38.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fca8806c920 0x7fca8806ede0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcaa81523f0 0x7fcaa8149fe0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcaa80b36b0 0x7fcaa8153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 --2- 172.21.15.69:0/802321994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcaa806db90 0x7fcaa81f3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:38.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 -- 172.21.15.69:0/802321994 >> 172.21.15.69:0/802321994 conn(0x7fcaa80d0160 msgr2=0x7fcaa80b43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:38.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 -- 172.21.15.69:0/802321994 shutdown_connections 2024-04-24T08:13:38.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:38.672+0000 7fcaad000640 1 -- 172.21.15.69:0/802321994 wait complete. 2024-04-24T08:13:39.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:13:15.498700+0000', 'modified': '2024-04-24T08:13:38.212068+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 69, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6792}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6792': {'gid': 6792, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.69:6835/2999903126', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 2999903126}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 2999903126}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:13:39.095 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6792:up:replay 2024-04-24T08:13:40.095 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:40.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.298+0000 7f7421c00640 1 Processor -- start 2024-04-24T08:13:40.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.298+0000 7f7421c00640 1 -- start start 2024-04-24T08:13:40.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.300+0000 7f7421c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c156c60 0x7f741c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:40.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.300+0000 7f7421c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f741c0ad7e0 0x7f741c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:40.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.300+0000 7f7421c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f741c15d660 con 0x7f741c0b30f0 2024-04-24T08:13:40.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.300+0000 7f7421c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f741c06dc30 con 0x7f741c0ad7e0 2024-04-24T08:13:40.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.300+0000 7f7421c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f741c06df10 con 0x7f741c156c60 2024-04-24T08:13:40.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.300+0000 7f7413400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f741c0b30f0 0x7f741c0ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33580/0 (socket says 172.21.15.69:33580) 2024-04-24T08:13:40.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.300+0000 7f7413400640 1 -- 172.21.15.69:0/863320936 learned_addr learned my addr 172.21.15.69:0/863320936 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:40.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.300+0000 7f7421200640 1 --2- 172.21.15.69:0/863320936 >> v2:172.21.15.121:3300/0 conn(0x7f741c0ad7e0 0x7f741c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:40.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.300+0000 7f741be00640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c156c60 0x7f741c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.300+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148519750 0 0) 0x7f741c15d660 con 0x7f741c0b30f0 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7400003520 con 0x7f741c0b30f0 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1180768559 0 0) 0x7f7400003520 con 0x7f741c0b30f0 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c156c60 msgr2=0x7f741c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c156c60 0x7f741c15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 >> v2:172.21.15.121:3300/0 conn(0x7f741c0ad7e0 msgr2=0x7f741c15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 --2- 172.21.15.69:0/863320936 >> v2:172.21.15.121:3300/0 conn(0x7f741c0ad7e0 0x7f741c15d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741c06e1f0 con 0x7f741c0b30f0 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f7404001670 con 0x7f741c0b30f0 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 >> v1:172.21.15.69:6789/0 conn(0x7f741c0b30f0 legacy=0x7f741c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c156c60 0x7f74000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7400003fe0 0x7f7400004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:40.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7400004d40 0x7f74000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f741c06dc30 con 0x7f7400003fe0 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f741c06df10 con 0x7f7400004d40 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f740000bac0 con 0x7f741c156c60 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f7421200640 1 --2- 172.21.15.69:0/863320936 >> v2:172.21.15.121:3300/0 conn(0x7f741c0ad7e0 0x7f741c15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.301+0000 7f7413400640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7400003fe0 0x7f7400004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.302+0000 7f741be00640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c156c60 0x7f74000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.302+0000 7f7413400640 1 -- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c156c60 msgr2=0x7f74000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.302+0000 7f7413400640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c156c60 0x7f74000039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.302+0000 7f7413400640 1 -- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7400004d40 msgr2=0x7f74000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.302+0000 7f7413400640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7400004d40 0x7f74000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.302+0000 7f7413400640 1 -- 172.21.15.69:0/863320936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7400003520 con 0x7f7400003fe0 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.302+0000 7f741be00640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c156c60 0x7f74000039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:40.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.303+0000 7f7413400640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7400003fe0 0x7f7400004800 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7404007fd0 tx=0x7f7400005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ecc844dced3f2db server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:40.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.303+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404007d50 con 0x7f7400003fe0 2024-04-24T08:13:40.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.303+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7404005af0 con 0x7f7400003fe0 2024-04-24T08:13:40.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.303+0000 7f741b400640 1 -- 172.21.15.69:0/863320936 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404007070 con 0x7f7400003fe0 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.303+0000 7f7421c00640 1 -- 172.21.15.69:0/863320936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7400003fe0 msgr2=0x7f7400004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.304+0000 7f7421c00640 1 --2- 172.21.15.69:0/863320936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7400003fe0 0x7f7400004800 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7404007fd0 tx=0x7f7400005830 comp rx=0 tx=0).stop 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.304+0000 7f7421200640 1 -- 172.21.15.69:0/863320936 reap_dead start 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.304+0000 7f7421c00640 1 -- 172.21.15.69:0/863320936 shutdown_connections 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.304+0000 7f7421c00640 1 -- 172.21.15.69:0/863320936 >> 172.21.15.69:0/863320936 conn(0x7f741c0d0160 msgr2=0x7f741c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.304+0000 7f7421c00640 1 -- 172.21.15.69:0/863320936 shutdown_connections 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.304+0000 7f7421c00640 1 -- 172.21.15.69:0/863320936 wait complete. 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.305+0000 7f7421c00640 1 Processor -- start 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.306+0000 7f7421c00640 1 -- start start 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.306+0000 7f7421c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7400003fe0 0x7f741c1521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7421c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f741c15b1c0 0x7f741c15ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:40.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7421c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c15dac0 0x7f741c15df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7421c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f741c15cac0 con 0x7f741c15b1c0 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7421c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f741c15d940 con 0x7f7400003fe0 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7421c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f741c15cc40 con 0x7f741c15dac0 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7413400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f741c15b1c0 0x7f741c15ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7413400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f741c15b1c0 0x7f741c15ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60708/0 (socket says 172.21.15.69:60708) 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7413400640 1 -- 172.21.15.69:0/2536454084 learned_addr learned my addr 172.21.15.69:0/2536454084 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f741be00640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7400003fe0 0x7f741c1521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7421200640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c15dac0 0x7f741c15df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7413400640 1 -- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c15dac0 msgr2=0x7f741c15df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7413400640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c15dac0 0x7f741c15df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7413400640 1 -- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7400003fe0 msgr2=0x7f741c1521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7413400640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7400003fe0 0x7f741c1521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7413400640 1 -- 172.21.15.69:0/2536454084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741c15a720 con 0x7f741c15b1c0 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.307+0000 7f7421200640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c15dac0 0x7f741c15df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:40.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.308+0000 7f7413400640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f741c15b1c0 0x7f741c15ef10 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f740402ae20 tx=0x7f7404001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:40.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.308+0000 7f7418c00640 1 -- 172.21.15.69:0/2536454084 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404042020 con 0x7f741c15b1c0 2024-04-24T08:13:40.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.308+0000 7f7418c00640 1 -- 172.21.15.69:0/2536454084 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7404004bf0 con 0x7f741c15b1c0 2024-04-24T08:13:40.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.308+0000 7f7418c00640 1 -- 172.21.15.69:0/2536454084 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7404001c90 con 0x7f741c15b1c0 2024-04-24T08:13:40.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.308+0000 7f7421c00640 1 -- 172.21.15.69:0/2536454084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741c153e30 con 0x7f741c15b1c0 2024-04-24T08:13:40.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.308+0000 7f7421c00640 1 -- 172.21.15.69:0/2536454084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f741c15f450 con 0x7f741c15b1c0 2024-04-24T08:13:40.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.309+0000 7f7421c00640 1 -- 172.21.15.69:0/2536454084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f741c075d50 con 0x7f741c15b1c0 2024-04-24T08:13:40.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.310+0000 7f7418c00640 1 -- 172.21.15.69:0/2536454084 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7404045020 con 0x7f741c15b1c0 2024-04-24T08:13:40.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.311+0000 7f7418c00640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f73f806c860 0x7f73f806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:40.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.311+0000 7f7418c00640 1 -- 172.21.15.69:0/2536454084 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f74040ac7b0 con 0x7f741c15b1c0 2024-04-24T08:13:40.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.312+0000 7f741be00640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f73f806c860 0x7f73f806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:40.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.313+0000 7f741be00640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f73f806c860 0x7f73f806ed20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f741c06db40 tx=0x7f740c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:40.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.315+0000 7f7418c00640 1 -- 172.21.15.69:0/2536454084 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f740404c020 con 0x7f741c15b1c0 2024-04-24T08:13:40.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.554+0000 7f7421c00640 1 -- 172.21.15.69:0/2536454084 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f741c0b45b0 con 0x7f741c15b1c0 2024-04-24T08:13:40.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.555+0000 7f7418c00640 1 -- 172.21.15.69:0/2536454084 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+5180 (secure 0 0 0) 0x7f740407ab70 con 0x7f741c15b1c0 2024-04-24T08:13:40.557 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 49 2024-04-24T08:13:40.557 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:40.557 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":49,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3504199143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3504199143},{"type":"v1","addr":"172.21.15.121:6837","nonce":3504199143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:13:39.233775+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6792},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6792":{"gid":6792,"name":"a","rank":0,"incarnation":47,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":5}]} 2024-04-24T08:13:40.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 -- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f73f806c860 msgr2=0x7f73f806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:40.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f73f806c860 0x7f73f806ed20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f741c06db40 tx=0x7f740c008040 comp rx=0 tx=0).stop 2024-04-24T08:13:40.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 -- 172.21.15.69:0/2536454084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f741c15b1c0 msgr2=0x7f741c15ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:40.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f741c15b1c0 0x7f741c15ef10 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f740402ae20 tx=0x7f7404001a40 comp rx=0 tx=0).stop 2024-04-24T08:13:40.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 -- 172.21.15.69:0/2536454084 shutdown_connections 2024-04-24T08:13:40.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f73f806c860 0x7f73f806ed20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:40.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f741c15b1c0 0x7f741c15ef10 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:40.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f741c15dac0 0x7f741c15df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:40.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 --2- 172.21.15.69:0/2536454084 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7400003fe0 0x7f741c1521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:40.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 -- 172.21.15.69:0/2536454084 >> 172.21.15.69:0/2536454084 conn(0x7f741c0d0160 msgr2=0x7f741c153160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:40.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 -- 172.21.15.69:0/2536454084 shutdown_connections 2024-04-24T08:13:40.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:40.558+0000 7f7421c00640 1 -- 172.21.15.69:0/2536454084 wait complete. 2024-04-24T08:13:40.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:13:15.498700+0000', 'modified': '2024-04-24T08:13:39.233775+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 69, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6792}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6792': {'gid': 6792, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.69:6835/2999903126', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 2999903126}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 2999903126}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:13:40.980 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6792:up:rejoin 2024-04-24T08:13:41.981 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:42.015 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:13:42.014+0000 7f980ba00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:13:42.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.189+0000 7f333e000640 1 Processor -- start 2024-04-24T08:13:42.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.189+0000 7f333e000640 1 -- start start 2024-04-24T08:13:42.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.191+0000 7f333e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33380b30f0 0x7f3338167470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:42.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.191+0000 7f333e000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f333815d660 con 0x7f3338156c60 2024-04-24T08:13:42.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.191+0000 7f333e000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f333806dc30 con 0x7f333815d1d0 2024-04-24T08:13:42.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.191+0000 7f333e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f333806df10 con 0x7f33380b30f0 2024-04-24T08:13:42.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.191+0000 7f333cc00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3338156c60 0x7f333815cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33590/0 (socket says 172.21.15.69:33590) 2024-04-24T08:13:42.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.191+0000 7f333cc00640 1 -- 172.21.15.69:0/2604165178 learned_addr learned my addr 172.21.15.69:0/2604165178 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.191+0000 7f333d600640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33380b30f0 0x7f3338167470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.191+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477866422 0 0) 0x7f333815d660 con 0x7f3338156c60 2024-04-24T08:13:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3320003520 con 0x7f3338156c60 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2031755056 0 0) 0x7f3320003520 con 0x7f3338156c60 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33380b30f0 msgr2=0x7f3338167470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33380b30f0 0x7f3338167470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 >> v1:172.21.15.121:6789/0 conn(0x7f333815d1d0 legacy=0x7f33380ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333806e1f0 con 0x7f3338156c60 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f332c001670 con 0x7f3338156c60 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 >> v1:172.21.15.69:6789/0 conn(0x7f3338156c60 legacy=0x7f333815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33200039e0 0x7f3320003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3320004800 0x7f3320004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33380b30f0 0x7f33200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f333806df10 con 0x7f33200039e0 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f332000ba70 con 0x7f3320004800 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f332000bc70 con 0x7f33380b30f0 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.192+0000 7f333cc00640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33200039e0 0x7f3320003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.193+0000 7f3337e00640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3320004800 0x7f3320004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:42.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.193+0000 7f333d600640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33380b30f0 0x7f33200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:42.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.193+0000 7f333cc00640 1 -- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33380b30f0 msgr2=0x7f33200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:42.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.193+0000 7f333cc00640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33380b30f0 0x7f33200051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:42.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.193+0000 7f333cc00640 1 -- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3320004800 msgr2=0x7f3320004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:42.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.193+0000 7f333cc00640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3320004800 0x7f3320004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:42.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.193+0000 7f333cc00640 1 -- 172.21.15.69:0/2604165178 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3320003520 con 0x7f33200039e0 2024-04-24T08:13:42.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.194+0000 7f333cc00640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33200039e0 0x7f3320003e00 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f332c001eb0 tx=0x7f3320005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1602499588511df1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:42.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.194+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c007d30 con 0x7f33200039e0 2024-04-24T08:13:42.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.194+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f332c005af0 con 0x7f33200039e0 2024-04-24T08:13:42.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.194+0000 7f3337400640 1 -- 172.21.15.69:0/2604165178 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c007070 con 0x7f33200039e0 2024-04-24T08:13:42.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.195+0000 7f333e000640 1 -- 172.21.15.69:0/2604165178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33200039e0 msgr2=0x7f3320003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:42.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.195+0000 7f333e000640 1 --2- 172.21.15.69:0/2604165178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33200039e0 0x7f3320003e00 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f332c001eb0 tx=0x7f3320005830 comp rx=0 tx=0).stop 2024-04-24T08:13:42.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.195+0000 7f333d600640 1 -- 172.21.15.69:0/2604165178 reap_dead start 2024-04-24T08:13:42.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.195+0000 7f333e000640 1 -- 172.21.15.69:0/2604165178 shutdown_connections 2024-04-24T08:13:42.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.195+0000 7f333e000640 1 -- 172.21.15.69:0/2604165178 >> 172.21.15.69:0/2604165178 conn(0x7f33380d0160 msgr2=0x7f33380d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:42.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.195+0000 7f333e000640 1 -- 172.21.15.69:0/2604165178 shutdown_connections 2024-04-24T08:13:42.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.195+0000 7f333e000640 1 -- 172.21.15.69:0/2604165178 wait complete. 2024-04-24T08:13:42.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.196+0000 7f333e000640 1 Processor -- start 2024-04-24T08:13:42.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.197+0000 7f333e000640 1 -- start start 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.197+0000 7f333e000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33200039e0 0x7f33381e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.197+0000 7f333e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33380cf650 0x7f33380c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.197+0000 7f333e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33381f46c0 0x7f33380cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f333e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f33380ad7e0 con 0x7f33380cf650 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f333e000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f333815d940 con 0x7f33200039e0 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f333e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f33380ad960 con 0x7f33381f46c0 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f3337e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33380cf650 0x7f33380c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f3337e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33380cf650 0x7f33380c9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60716/0 (socket says 172.21.15.69:60716) 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f3337e00640 1 -- 172.21.15.69:0/129101372 learned_addr learned my addr 172.21.15.69:0/129101372 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f3337e00640 1 -- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33381f46c0 msgr2=0x7f33380cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:42.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f333d600640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33381f46c0 0x7f33380cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f333cc00640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33200039e0 0x7f33381e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f3337e00640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33381f46c0 0x7f33380cb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f3337e00640 1 -- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33200039e0 msgr2=0x7f33381e85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f3337e00640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33200039e0 0x7f33381e85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.198+0000 7f3337e00640 1 -- 172.21.15.69:0/129101372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33380c9ae0 con 0x7f33380cf650 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.199+0000 7f3337e00640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33380cf650 0x7f33380c9470 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f332400e4d0 tx=0x7f332400e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.199+0000 7f3334c00640 1 -- 172.21.15.69:0/129101372 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33240052e0 con 0x7f33380cf650 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.199+0000 7f3334c00640 1 -- 172.21.15.69:0/129101372 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3324018070 con 0x7f33380cf650 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.199+0000 7f3334c00640 1 -- 172.21.15.69:0/129101372 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3324003810 con 0x7f33380cf650 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.199+0000 7f333e000640 1 -- 172.21.15.69:0/129101372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33381e7b00 con 0x7f33380cf650 2024-04-24T08:13:42.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.199+0000 7f333e000640 1 -- 172.21.15.69:0/129101372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33381e80c0 con 0x7f33380cf650 2024-04-24T08:13:42.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.200+0000 7f333e000640 1 -- 172.21.15.69:0/129101372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f333815cff0 con 0x7f33380cf650 2024-04-24T08:13:42.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.201+0000 7f3334c00640 1 -- 172.21.15.69:0/129101372 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f33240039b0 con 0x7f33380cf650 2024-04-24T08:13:42.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.202+0000 7f3334c00640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f331806c920 0x7f331806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:42.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.202+0000 7f3334c00640 1 -- 172.21.15.69:0/129101372 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f332408b870 con 0x7f33380cf650 2024-04-24T08:13:42.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.203+0000 7f333cc00640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f331806c920 0x7f331806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:42.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.204+0000 7f333cc00640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f331806c920 0x7f331806ede0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f332c032000 tx=0x7f332c02d000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:42.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.207+0000 7f3334c00640 1 -- 172.21.15.69:0/129101372 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f333815cff0 con 0x7f33380cf650 2024-04-24T08:13:42.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.441+0000 7f333e000640 1 -- 172.21.15.69:0/129101372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3338075d50 con 0x7f33380cf650 2024-04-24T08:13:42.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.442+0000 7f3334c00640 1 -- 172.21.15.69:0/129101372 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5188 (secure 0 0 0) 0x7f3324059c30 con 0x7f33380cf650 2024-04-24T08:13:42.443 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:42.444 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":50,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3504199143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3504199143},{"type":"v1","addr":"172.21.15.121:6837","nonce":3504199143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:13:41.243566+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6792},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6792":{"gid":6792,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6792,"qdb_cluster":[6792]},"id":5}]} 2024-04-24T08:13:42.444 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 50 2024-04-24T08:13:42.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.444+0000 7f333e000640 1 -- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f331806c920 msgr2=0x7f331806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.444+0000 7f333e000640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f331806c920 0x7f331806ede0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f332c032000 tx=0x7f332c02d000 comp rx=0 tx=0).stop 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.444+0000 7f333e000640 1 -- 172.21.15.69:0/129101372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33380cf650 msgr2=0x7f33380c9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.444+0000 7f333e000640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33380cf650 0x7f33380c9470 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f332400e4d0 tx=0x7f332400e990 comp rx=0 tx=0).stop 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.445+0000 7f333e000640 1 -- 172.21.15.69:0/129101372 shutdown_connections 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.445+0000 7f333e000640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f331806c920 0x7f331806ede0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.445+0000 7f333e000640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33380cf650 0x7f33380c9470 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.445+0000 7f333e000640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f33381f46c0 0x7f33380cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.445+0000 7f333e000640 1 --2- 172.21.15.69:0/129101372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33200039e0 0x7f33381e85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.445+0000 7f333e000640 1 -- 172.21.15.69:0/129101372 >> 172.21.15.69:0/129101372 conn(0x7f33380d0160 msgr2=0x7f33380b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.445+0000 7f333e000640 1 -- 172.21.15.69:0/129101372 shutdown_connections 2024-04-24T08:13:42.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:42.445+0000 7f333e000640 1 -- 172.21.15.69:0/129101372 wait complete. 2024-04-24T08:13:42.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:13:15.498700+0000', 'modified': '2024-04-24T08:13:41.243566+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 69, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6792}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6792': {'gid': 6792, 'name': 'a', 'rank': 0, 'incarnation': 47, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.69:6835/2999903126', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 2999903126}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 2999903126}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6792, 'qdb_cluster': [6792]} 2024-04-24T08:13:42.865 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:13:42.866 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-24T08:13:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.064+0000 7f9d53200640 1 Processor -- start 2024-04-24T08:13:43.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.064+0000 7f9d53200640 1 -- start start 2024-04-24T08:13:43.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.066+0000 7f9d53200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9d5415d1d0 0x7f9d540ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.066+0000 7f9d53200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d540b30f0 0x7f9d540ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.066+0000 7f9d53200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d5415d660 con 0x7f9d54156c60 2024-04-24T08:13:43.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.066+0000 7f9d53200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9d5406dc30 con 0x7f9d5415d1d0 2024-04-24T08:13:43.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.066+0000 7f9d53200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d5406df10 con 0x7f9d540b30f0 2024-04-24T08:13:43.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.066+0000 7f9d51e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9d54156c60 0x7f9d5415cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33606/0 (socket says 172.21.15.69:33606) 2024-04-24T08:13:43.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.066+0000 7f9d51400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9d5415d1d0 0x7f9d540ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.066+0000 7f9d51e00640 1 -- 172.21.15.69:0/575787567 learned_addr learned my addr 172.21.15.69:0/575787567 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.066+0000 7f9d52800640 1 --2- 172.21.15.69:0/575787567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d540b30f0 0x7f9d540ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.067+0000 7f9d50a00640 1 -- 172.21.15.69:0/575787567 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633290999 0 0) 0x7f9d5415d660 con 0x7f9d54156c60 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.067+0000 7f9d50a00640 1 -- 172.21.15.69:0/575787567 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d38003520 con 0x7f9d54156c60 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.067+0000 7f9d51400640 1 -- 172.21.15.69:0/575787567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d540b30f0 msgr2=0x7f9d540ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.067+0000 7f9d51400640 1 --2- 172.21.15.69:0/575787567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d540b30f0 0x7f9d540ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.067+0000 7f9d51400640 1 -- 172.21.15.69:0/575787567 >> v1:172.21.15.69:6789/0 conn(0x7f9d54156c60 legacy=0x7f9d5415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.067+0000 7f9d51400640 1 -- 172.21.15.69:0/575787567 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d5406e1f0 con 0x7f9d5415d1d0 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.068+0000 7f9d51400640 1 --2- 172.21.15.69:0/575787567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d5415d1d0 0x7f9d540ad0d0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9d400097a0 tx=0x7f9d4002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7b645fcc0a8dc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.068+0000 7f9d50a00640 1 -- 172.21.15.69:0/575787567 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40037070 con 0x7f9d5415d1d0 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.068+0000 7f9d50a00640 1 -- 172.21.15.69:0/575787567 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d40032440 con 0x7f9d5415d1d0 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.068+0000 7f9d50a00640 1 -- 172.21.15.69:0/575787567 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40032810 con 0x7f9d5415d1d0 2024-04-24T08:13:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.068+0000 7f9d53200640 1 -- 172.21.15.69:0/575787567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d5415d1d0 msgr2=0x7f9d540ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.068+0000 7f9d53200640 1 --2- 172.21.15.69:0/575787567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d5415d1d0 0x7f9d540ad0d0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9d400097a0 tx=0x7f9d4002a180 comp rx=0 tx=0).stop 2024-04-24T08:13:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.069+0000 7f9d53200640 1 -- 172.21.15.69:0/575787567 shutdown_connections 2024-04-24T08:13:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.069+0000 7f9d53200640 1 --2- 172.21.15.69:0/575787567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d540b30f0 0x7f9d540ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.069+0000 7f9d53200640 1 --2- 172.21.15.69:0/575787567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d5415d1d0 0x7f9d540ad0d0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.069+0000 7f9d53200640 1 -- 172.21.15.69:0/575787567 >> 172.21.15.69:0/575787567 conn(0x7f9d540d0160 msgr2=0x7f9d540d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.069+0000 7f9d53200640 1 -- 172.21.15.69:0/575787567 shutdown_connections 2024-04-24T08:13:43.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.069+0000 7f9d53200640 1 -- 172.21.15.69:0/575787567 wait complete. 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.070+0000 7f9d53200640 1 Processor -- start 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.071+0000 7f9d53200640 1 -- start start 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.071+0000 7f9d53200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d540b30f0 0x7f9d54144380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.071+0000 7f9d53200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d54156c60 0x7f9d541448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.071+0000 7f9d53200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d54158360 0x7f9d541587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.071+0000 7f9d53200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5406e710 con 0x7f9d54156c60 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.071+0000 7f9d53200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5415d940 con 0x7f9d540b30f0 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.071+0000 7f9d53200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d5406e890 con 0x7f9d54158360 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.072+0000 7f9d51400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d54156c60 0x7f9d541448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.072+0000 7f9d51400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d54156c60 0x7f9d541448c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60736/0 (socket says 172.21.15.69:60736) 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.072+0000 7f9d51400640 1 -- 172.21.15.69:0/1171017148 learned_addr learned my addr 172.21.15.69:0/1171017148 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.072+0000 7f9d51400640 1 -- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d54158360 msgr2=0x7f9d541587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.072+0000 7f9d51e00640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d540b30f0 0x7f9d54144380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.072+0000 7f9d51400640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d54158360 0x7f9d541587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.072+0000 7f9d51400640 1 -- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d540b30f0 msgr2=0x7f9d54144380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.072+0000 7f9d51400640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d540b30f0 0x7f9d54144380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.072+0000 7f9d51400640 1 -- 172.21.15.69:0/1171017148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d540cc4f0 con 0x7f9d54156c60 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.072+0000 7f9d51400640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d54156c60 0x7f9d541448c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9d400098c0 tx=0x7f9d400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.073+0000 7f9d4a000640 1 -- 172.21.15.69:0/1171017148 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40037040 con 0x7f9d54156c60 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.073+0000 7f9d4a000640 1 -- 172.21.15.69:0/1171017148 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d40030500 con 0x7f9d54156c60 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.073+0000 7f9d53200640 1 -- 172.21.15.69:0/1171017148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d540cc7d0 con 0x7f9d54156c60 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.073+0000 7f9d4a000640 1 -- 172.21.15.69:0/1171017148 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d400308d0 con 0x7f9d54156c60 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.073+0000 7f9d53200640 1 -- 172.21.15.69:0/1171017148 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d540ccd90 con 0x7f9d54156c60 2024-04-24T08:13:43.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.075+0000 7f9d4a000640 1 -- 172.21.15.69:0/1171017148 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9d40044020 con 0x7f9d54156c60 2024-04-24T08:13:43.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.076+0000 7f9d4a000640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d3406c920 0x7f9d3406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.076+0000 7f9d4a000640 1 -- 172.21.15.69:0/1171017148 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f9d400abf10 con 0x7f9d54156c60 2024-04-24T08:13:43.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.077+0000 7f9d5bbeb740 1 Processor -- start 2024-04-24T08:13:43.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.077+0000 7f9d51e00640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d3406c920 0x7f9d3406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.077+0000 7f9d5bbeb740 1 -- start start 2024-04-24T08:13:43.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf3c70 0x55940ecf7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940eccb980 0x55940ecf4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 0x55940ecf6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55940ec31b10 con 0x55940eccb980 2024-04-24T08:13:43.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55940ebd82b0 con 0x55940ecf3c70 2024-04-24T08:13:43.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55940ec2dad0 con 0x55940ecf4590 2024-04-24T08:13:43.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d51e00640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d3406c920 0x7f9d3406ede0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9d540d00a0 tx=0x7f9d44009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d51400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940eccb980 0x55940ecf4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d51400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940eccb980 0x55940ecf4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60746/0 (socket says 172.21.15.69:60746) 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d51400640 1 -- 172.21.15.69:0/1811043650 learned_addr learned my addr 172.21.15.69:0/1811043650 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d51e00640 1 --2- 172.21.15.69:0/1811043650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf3c70 0x55940ecf7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d51400640 1 -- 172.21.15.69:0/1811043650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 msgr2=0x55940ecf6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d51400640 1 --2- 172.21.15.69:0/1811043650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 0x55940ecf6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d51400640 1 -- 172.21.15.69:0/1811043650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf3c70 msgr2=0x55940ecf7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d51400640 1 --2- 172.21.15.69:0/1811043650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf3c70 0x55940ecf7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.078+0000 7f9d51400640 1 -- 172.21.15.69:0/1811043650 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55940e9442d0 con 0x55940eccb980 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.079+0000 7f9d51400640 1 --2- 172.21.15.69:0/1811043650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940eccb980 0x55940ecf4050 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9d40035040 tx=0x7f9d40075f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242a30e08b35aaca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.079+0000 7f9d3ea00640 1 -- 172.21.15.69:0/1811043650 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40079b40 con 0x55940eccb980 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.079+0000 7f9d3ea00640 1 -- 172.21.15.69:0/1811043650 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4007def0 con 0x55940eccb980 2024-04-24T08:13:43.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.079+0000 7f9d3ea00640 1 -- 172.21.15.69:0/1811043650 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d40082720 con 0x55940eccb980 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.079+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1811043650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940eccb980 msgr2=0x55940ecf4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.079+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/1811043650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940eccb980 0x55940ecf4050 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f9d40035040 tx=0x7f9d40075f10 comp rx=0 tx=0).stop 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.080+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1811043650 shutdown_connections 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.080+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/1811043650 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940eccb980 0x55940ecf4050 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.080+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/1811043650 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 0x55940ecf6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.080+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/1811043650 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf3c70 0x55940ecf7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.080+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1811043650 >> 172.21.15.69:0/1811043650 conn(0x55940ecc9580 msgr2=0x55940ed11e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.080+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1811043650 shutdown_connections 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.080+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1811043650 wait complete. 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.080+0000 7f9d5bbeb740 1 -- start start 2024-04-24T08:13:43.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.081+0000 7f9d5bbeb740 1 Processor -- start 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.081+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf4590 0x55940ece9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecea020 0x55940eceb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55940ec31b10 con 0x55940ecf3c70 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55940ebeb590 con 0x55940ecf4590 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55940ecc78d0 con 0x55940ecea020 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d5bbeb740 10 client.0 _ll_register_callbacks cb 0x55940ebce3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d51e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d51e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60750/0 (socket says 172.21.15.69:60750) 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d51e00640 1 -- 172.21.15.69:0/1966405963 learned_addr learned my addr 172.21.15.69:0/1966405963 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d51e00640 1 -- 172.21.15.69:0/1966405963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecea020 msgr2=0x55940eceb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d51e00640 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecea020 0x55940eceb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d51e00640 1 -- 172.21.15.69:0/1966405963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf4590 msgr2=0x55940ece9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:43.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d51e00640 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf4590 0x55940ece9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d51e00640 1 -- 172.21.15.69:0/1966405963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55940ea0e6b0 con 0x55940ecf3c70 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d51e00640 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece8990 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f9d44007680 tx=0x7f9d44000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45967c489f1a546f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.082+0000 7f9d33400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d33400640 1 -- 172.21.15.69:0/1966405963 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44021430 con 0x55940ecf3c70 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d33400640 1 -- 172.21.15.69:0/1966405963 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f9d44020ae0 con 0x55940ecf3c70 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1966405963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d4007def0 con 0x55940ecf3c70 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d33400640 1 -- 172.21.15.69:0/1966405963 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f9d44020ce0 con 0x55940ecf3c70 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d5bbeb740 10 client.7098 fetch_fsmap learned FSMap version 50 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1966405963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x55940eceb100 con 0x55940ecf3c70 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d33400640 1 -- 172.21.15.69:0/1966405963 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d4007def0 con 0x55940ecf3c70 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d33400640 1 -- 172.21.15.69:0/1966405963 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7f9d44020420 con 0x55940ecf3c70 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d5bbeb740 10 client.7098 fetch_fsmap finished waiting for FSMap version 50 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d5bbeb740 10 client.7098 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d5bbeb740 20 client.7098 populate_metadata read hostname 'smithi069' 2024-04-24T08:13:43.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55940ed135d0 0x55940ecf5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d5bbeb740 4 client.7098 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.083+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1966405963 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55940ec3f530 con 0x55940ed135d0 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.084+0000 7f9d51400640 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55940ed135d0 0x55940ecf5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.084+0000 7f9d51400640 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55940ed135d0 0x55940ecf5c70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d20faf3cbf4cca8 server_cookie=db8962c5c34550ea in_seq=0 out_seq=0 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.084+0000 7f9d33400640 10 client.7098 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d33400640 1 -- 172.21.15.69:0/1966405963 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55940ec3f530 con 0x55940ed135d0 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d33400640 10 client.7098 handle_command_reply: tid=0 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 client.7098 shutdown 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1966405963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 msgr2=0x55940ece8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece8990 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f9d44007680 tx=0x7f9d44000bc0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1966405963 shutdown_connections 2024-04-24T08:13:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55940ed135d0 0x55940ecf5c70 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecea020 0x55940eceb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf4590 0x55940ece9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/1966405963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece8990 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1966405963 >> 172.21.15.69:0/1966405963 conn(0x55940eccb980 msgr2=0x55940eccc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d33400640 0 client.7098 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1966405963 shutdown_connections 2024-04-24T08:13:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.085+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/1966405963 wait complete. 2024-04-24T08:13:43.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.086+0000 7f9d5bbeb740 20 client.7098 trim_cache size 0 max 16384 2024-04-24T08:13:43.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.104+0000 7f9d5bbeb740 1 Processor -- start 2024-04-24T08:13:43.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.104+0000 7f9d5bbeb740 1 -- start start 2024-04-24T08:13:43.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.105+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecea020 0x55940eccbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.105+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf3c70 0x55940ed127d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.105+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 0x55940ecc60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.105+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d54072df0 con 0x55940ecea020 2024-04-24T08:13:43.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.105+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9d541564e0 con 0x55940ecf3c70 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.105+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9d540ce350 con 0x55940ecf4590 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.105+0000 7f9d51e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecea020 0x55940eccbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.105+0000 7f9d51e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecea020 0x55940eccbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60762/0 (socket says 172.21.15.69:60762) 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.105+0000 7f9d51e00640 1 -- 172.21.15.69:0/4090461901 learned_addr learned my addr 172.21.15.69:0/4090461901 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.106+0000 7f9d52800640 1 --2- 172.21.15.69:0/4090461901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 0x55940ecc60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.106+0000 7f9d51400640 1 --2- 172.21.15.69:0/4090461901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf3c70 0x55940ed127d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.106+0000 7f9d51e00640 1 -- 172.21.15.69:0/4090461901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 msgr2=0x55940ecc60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.106+0000 7f9d51e00640 1 --2- 172.21.15.69:0/4090461901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 0x55940ecc60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.106+0000 7f9d51e00640 1 -- 172.21.15.69:0/4090461901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf3c70 msgr2=0x55940ed127d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.106+0000 7f9d51e00640 1 --2- 172.21.15.69:0/4090461901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf3c70 0x55940ed127d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.106+0000 7f9d51e00640 1 -- 172.21.15.69:0/4090461901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55940ed12d10 con 0x55940ecea020 2024-04-24T08:13:43.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.106+0000 7f9d51e00640 1 --2- 172.21.15.69:0/4090461901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecea020 0x55940eccbfa0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f9d44020ee0 tx=0x7f9d4401d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a264305e35b8c8ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.106+0000 7f9d3e000640 1 -- 172.21.15.69:0/4090461901 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4402f040 con 0x55940ecea020 2024-04-24T08:13:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.106+0000 7f9d3e000640 1 -- 172.21.15.69:0/4090461901 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d44005e50 con 0x55940ecea020 2024-04-24T08:13:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.107+0000 7f9d3e000640 1 -- 172.21.15.69:0/4090461901 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d44034440 con 0x55940ecea020 2024-04-24T08:13:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.107+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/4090461901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecea020 msgr2=0x55940eccbfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.107+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/4090461901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecea020 0x55940eccbfa0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f9d44020ee0 tx=0x7f9d4401d000 comp rx=0 tx=0).stop 2024-04-24T08:13:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.107+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/4090461901 shutdown_connections 2024-04-24T08:13:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.107+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/4090461901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 0x55940ecc60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.107+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/4090461901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecf3c70 0x55940ed127d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.107+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/4090461901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecea020 0x55940eccbfa0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.107+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/4090461901 >> 172.21.15.69:0/4090461901 conn(0x55940eccb980 msgr2=0x55940ecf5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:43.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.107+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/4090461901 shutdown_connections 2024-04-24T08:13:43.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.107+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/4090461901 wait complete. 2024-04-24T08:13:43.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.108+0000 7f9d5bbeb740 1 -- start start 2024-04-24T08:13:43.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.108+0000 7f9d5bbeb740 1 Processor -- start 2024-04-24T08:13:43.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.109+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecea020 0x55940ece6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.109+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.109+0000 7f9d5bbeb740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 0x55940ed157b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.109+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55940ed13450 con 0x55940ecf3c70 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.109+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55940ecf4f10 con 0x55940ecea020 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.109+0000 7f9d5bbeb740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55940ed13aa0 con 0x55940ecf4590 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.109+0000 7f9d5bbeb740 10 client.0 _ll_register_callbacks cb 0x7f9d44007300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.109+0000 7f9d51400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.109+0000 7f9d51400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60766/0 (socket says 172.21.15.69:60766) 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.109+0000 7f9d51400640 1 -- 172.21.15.69:0/2930879552 learned_addr learned my addr 172.21.15.69:0/2930879552 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d51e00640 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecea020 0x55940ece6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d51400640 1 -- 172.21.15.69:0/2930879552 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 msgr2=0x55940ed157b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d51400640 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 0x55940ed157b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d51400640 1 -- 172.21.15.69:0/2930879552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecea020 msgr2=0x55940ece6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d51400640 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecea020 0x55940ece6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d51400640 1 -- 172.21.15.69:0/2930879552 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55940ecc6680 con 0x55940ecf3c70 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d51e00640 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecea020 0x55940ece6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d51400640 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece7340 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f9d40008010 tx=0x7f9d40009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a1bfc7c5e895e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d33400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:13:43.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d33400640 1 -- 172.21.15.69:0/2930879552 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4009e060 con 0x55940ecf3c70 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d33400640 1 -- 172.21.15.69:0/2930879552 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f9d40077e40 con 0x55940ecf3c70 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/2930879552 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d44005e50 con 0x55940ecf3c70 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d33400640 1 -- 172.21.15.69:0/2930879552 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=50) ==== 24+0+0 (secure 0 0 0) 0x7f9d4007def0 con 0x55940ecf3c70 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d5bbeb740 10 client.7104 fetch_fsmap learned FSMap version 50 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/2930879552 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=50,monmap=2+}) -- 0x55940ed39970 con 0x55940ecf3c70 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.110+0000 7f9d33400640 1 -- 172.21.15.69:0/2930879552 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d44005e50 con 0x55940ecf3c70 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.111+0000 7f9d33400640 1 -- 172.21.15.69:0/2930879552 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 50) ==== 3494+0+0 (secure 0 0 0) 0x7f9d4003a070 con 0x55940ecf3c70 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.111+0000 7f9d5bbeb740 10 client.7104 fetch_fsmap finished waiting for FSMap version 50 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.111+0000 7f9d5bbeb740 10 client.7104 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.111+0000 7f9d5bbeb740 20 client.7104 populate_metadata read hostname 'smithi069' 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.111+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55940ed15cf0 0x55940ecc7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.111+0000 7f9d5bbeb740 4 client.7104 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.111+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/2930879552 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "status"}) -- 0x55940ec393a0 con 0x55940ed15cf0 2024-04-24T08:13:43.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.111+0000 7f9d51e00640 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55940ed15cf0 0x55940ecc7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.111+0000 7f9d51e00640 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55940ed15cf0 0x55940ecc7510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36bceca1ffb40846 server_cookie=cad4330637763246 in_seq=0 out_seq=0 2024-04-24T08:13:43.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.111+0000 7f9d33400640 10 client.7104 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:43.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d33400640 1 -- 172.21.15.69:0/2930879552 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55940ec393a0 con 0x55940ed15cf0 2024-04-24T08:13:43.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d33400640 10 client.7104 handle_command_reply: tid=0 2024-04-24T08:13:43.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 client.7104 shutdown 2024-04-24T08:13:43.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/2930879552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 msgr2=0x55940ece7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece7340 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f9d40008010 tx=0x7f9d40009190 comp rx=0 tx=0).stop 2024-04-24T08:13:43.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/2930879552 shutdown_connections 2024-04-24T08:13:43.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55940ed15cf0 0x55940ecc7510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55940ecf3c70 0x55940ece7340 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d33400640 0 client.7104 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:43.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55940ecf4590 0x55940ed157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 --2- 172.21.15.69:0/2930879552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55940ecea020 0x55940ece6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/2930879552 >> 172.21.15.69:0/2930879552 conn(0x55940eccb980 msgr2=0x55940ed39f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:43.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/2930879552 shutdown_connections 2024-04-24T08:13:43.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 1 -- 172.21.15.69:0/2930879552 wait complete. 2024-04-24T08:13:43.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.112+0000 7f9d5bbeb740 20 client.7104 trim_cache size 0 max 16384 2024-04-24T08:13:43.120 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 6792, 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 4.8965908220000003, 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 50, 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 69, 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 69, 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 32.511640868999997 2024-04-24T08:13:43.121 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:13:43.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 -- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d3406c920 msgr2=0x7f9d3406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d3406c920 0x7f9d3406ede0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9d540d00a0 tx=0x7f9d44009040 comp rx=0 tx=0).stop 2024-04-24T08:13:43.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 -- 172.21.15.69:0/1171017148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d54156c60 msgr2=0x7f9d541448c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d54156c60 0x7f9d541448c0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f9d400098c0 tx=0x7f9d400046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 -- 172.21.15.69:0/1171017148 shutdown_connections 2024-04-24T08:13:43.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9d3406c920 0x7f9d3406ede0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d54156c60 0x7f9d541448c0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9d54158360 0x7f9d541587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 --2- 172.21.15.69:0/1171017148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d540b30f0 0x7f9d54144380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 -- 172.21.15.69:0/1171017148 >> 172.21.15.69:0/1171017148 conn(0x7f9d540d0160 msgr2=0x7f9d540afdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:43.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 -- 172.21.15.69:0/1171017148 shutdown_connections 2024-04-24T08:13:43.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.113+0000 7f9d53200640 1 -- 172.21.15.69:0/1171017148 wait complete. 2024-04-24T08:13:43.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:13:43.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.316+0000 7f442be00640 1 Processor -- start 2024-04-24T08:13:43.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.316+0000 7f442be00640 1 -- start start 2024-04-24T08:13:43.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.318+0000 7f442be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f442c15d1d0 0x7f442c0ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.318+0000 7f442be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f442c0b30f0 0x7f442c0ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.318+0000 7f442be00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f442c15d660 con 0x7f442c156c60 2024-04-24T08:13:43.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.318+0000 7f442be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f442c06dc30 con 0x7f442c0b30f0 2024-04-24T08:13:43.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.318+0000 7f442be00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f442c06df10 con 0x7f442c15d1d0 2024-04-24T08:13:43.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.318+0000 7f442b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f442c0b30f0 0x7f442c0ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.318+0000 7f442b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f442c0b30f0 0x7f442c0ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60774/0 (socket says 172.21.15.69:60774) 2024-04-24T08:13:43.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.318+0000 7f442b400640 1 -- 172.21.15.69:0/1724746251 learned_addr learned my addr 172.21.15.69:0/1724746251 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:43.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.318+0000 7f442a000640 1 --2- 172.21.15.69:0/1724746251 >> v2:172.21.15.121:3300/0 conn(0x7f442c15d1d0 0x7f442c0ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.319+0000 7f442b400640 1 -- 172.21.15.69:0/1724746251 >> v2:172.21.15.121:3300/0 conn(0x7f442c15d1d0 msgr2=0x7f442c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.319+0000 7f442b400640 1 --2- 172.21.15.69:0/1724746251 >> v2:172.21.15.121:3300/0 conn(0x7f442c15d1d0 0x7f442c0ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.319+0000 7f442b400640 1 -- 172.21.15.69:0/1724746251 >> v1:172.21.15.121:6789/0 conn(0x7f442c156c60 legacy=0x7f442c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.319+0000 7f442b400640 1 -- 172.21.15.69:0/1724746251 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c06e1f0 con 0x7f442c0b30f0 2024-04-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.319+0000 7f442a000640 1 --2- 172.21.15.69:0/1724746251 >> v2:172.21.15.121:3300/0 conn(0x7f442c15d1d0 0x7f442c0ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:43.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.320+0000 7f442b400640 1 --2- 172.21.15.69:0/1724746251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0b30f0 0x7f442c0ad610 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f442c06d900 tx=0x7f442402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a7beb9ed1a1fef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f4429600640 1 -- 172.21.15.69:0/1724746251 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424036070 con 0x7f442c0b30f0 2024-04-24T08:13:43.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f4429600640 1 -- 172.21.15.69:0/1724746251 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4424031600 con 0x7f442c0b30f0 2024-04-24T08:13:43.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f4429600640 1 -- 172.21.15.69:0/1724746251 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44240319d0 con 0x7f442c0b30f0 2024-04-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f442be00640 1 -- 172.21.15.69:0/1724746251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0b30f0 msgr2=0x7f442c0ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f442be00640 1 --2- 172.21.15.69:0/1724746251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0b30f0 0x7f442c0ad610 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f442c06d900 tx=0x7f442402a730 comp rx=0 tx=0).stop 2024-04-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f442be00640 1 -- 172.21.15.69:0/1724746251 shutdown_connections 2024-04-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f442be00640 1 --2- 172.21.15.69:0/1724746251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0b30f0 0x7f442c0ad610 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f442be00640 1 --2- 172.21.15.69:0/1724746251 >> v2:172.21.15.121:3300/0 conn(0x7f442c15d1d0 0x7f442c0ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f442be00640 1 -- 172.21.15.69:0/1724746251 >> 172.21.15.69:0/1724746251 conn(0x7f442c0d0160 msgr2=0x7f442c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f442be00640 1 -- 172.21.15.69:0/1724746251 shutdown_connections 2024-04-24T08:13:43.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.321+0000 7f442be00640 1 -- 172.21.15.69:0/1724746251 wait complete. 2024-04-24T08:13:43.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.323+0000 7f442be00640 1 Processor -- start 2024-04-24T08:13:43.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.323+0000 7f442be00640 1 -- start start 2024-04-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.324+0000 7f442be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f442c0b30f0 0x7f442c1f4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.324+0000 7f442be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f442c156c60 0x7f442c155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.324+0000 7f442be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0cb150 0x7f442c0cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.324+0000 7f442be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f442c06e710 con 0x7f442c0cb150 2024-04-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.324+0000 7f442be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f442c15d940 con 0x7f442c0b30f0 2024-04-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.324+0000 7f442be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f442c06e890 con 0x7f442c156c60 2024-04-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.324+0000 7f442b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0cb150 0x7f442c0cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f442b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0cb150 0x7f442c0cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60782/0 (socket says 172.21.15.69:60782) 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f442b400640 1 -- 172.21.15.69:0/2486568565 learned_addr learned my addr 172.21.15.69:0/2486568565 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f442aa00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f442c0b30f0 0x7f442c1f4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f442b400640 1 -- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f442c156c60 msgr2=0x7f442c155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f442b400640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f442c156c60 0x7f442c155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f442b400640 1 -- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f442c0b30f0 msgr2=0x7f442c1f4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f442b400640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f442c0b30f0 0x7f442c1f4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f442b400640 1 -- 172.21.15.69:0/2486568565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c143ee0 con 0x7f442c0cb150 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f442aa00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f442c0b30f0 0x7f442c1f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f442b400640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0cb150 0x7f442c0cb600 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f4424034040 tx=0x7f4424004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.325+0000 7f4422a00640 1 -- 172.21.15.69:0/2486568565 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424036050 con 0x7f442c0cb150 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.326+0000 7f4422a00640 1 -- 172.21.15.69:0/2486568565 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4424042070 con 0x7f442c0cb150 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.326+0000 7f4422a00640 1 -- 172.21.15.69:0/2486568565 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424009510 con 0x7f442c0cb150 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.326+0000 7f442be00640 1 -- 172.21.15.69:0/2486568565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f442c0cbb40 con 0x7f442c0cb150 2024-04-24T08:13:43.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.326+0000 7f442be00640 1 -- 172.21.15.69:0/2486568565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f442c144150 con 0x7f442c0cb150 2024-04-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.327+0000 7f442be00640 1 -- 172.21.15.69:0/2486568565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f442c075d50 con 0x7f442c0cb150 2024-04-24T08:13:43.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.328+0000 7f4422a00640 1 -- 172.21.15.69:0/2486568565 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4424007bf0 con 0x7f442c0cb150 2024-04-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.329+0000 7f4422a00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f440c06c8f0 0x7f440c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:43.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.329+0000 7f4422a00640 1 -- 172.21.15.69:0/2486568565 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f44240ac980 con 0x7f442c0cb150 2024-04-24T08:13:43.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.330+0000 7f442aa00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f440c06c8f0 0x7f440c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:43.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.331+0000 7f442aa00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f440c06c8f0 0x7f440c06edb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f441c001680 tx=0x7f441c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:43.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.333+0000 7f4422a00640 1 -- 172.21.15.69:0/2486568565 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f442407ad50 con 0x7f442c0cb150 2024-04-24T08:13:43.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.575+0000 7f442be00640 1 -- 172.21.15.69:0/2486568565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f442c0b3cc0 con 0x7f442c0cb150 2024-04-24T08:13:43.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.576+0000 7f4422a00640 1 -- 172.21.15.69:0/2486568565 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v50) ==== 53+0+73 (secure 0 0 0) 0x7f442407a650 con 0x7f442c0cb150 2024-04-24T08:13:43.578 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:13:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 -- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f440c06c8f0 msgr2=0x7f440c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f440c06c8f0 0x7f440c06edb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f441c001680 tx=0x7f441c009040 comp rx=0 tx=0).stop 2024-04-24T08:13:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 -- 172.21.15.69:0/2486568565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0cb150 msgr2=0x7f442c0cb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0cb150 0x7f442c0cb600 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f4424034040 tx=0x7f4424004a40 comp rx=0 tx=0).stop 2024-04-24T08:13:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 -- 172.21.15.69:0/2486568565 shutdown_connections 2024-04-24T08:13:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f440c06c8f0 0x7f440c06edb0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f442c0cb150 0x7f442c0cb600 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f442c156c60 0x7f442c155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 --2- 172.21.15.69:0/2486568565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f442c0b30f0 0x7f442c1f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:43.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 -- 172.21.15.69:0/2486568565 >> 172.21.15.69:0/2486568565 conn(0x7f442c0d0160 msgr2=0x7f442c06ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:43.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 -- 172.21.15.69:0/2486568565 shutdown_connections 2024-04-24T08:13:43.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43.579+0000 7f442be00640 1 -- 172.21.15.69:0/2486568565 wait complete. 2024-04-24T08:13:43.600 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:13:43.600 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:13:43.600 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:13:43.600 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:13:43.600 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:43.600 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:13:43.601 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:13:43.601 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:13:43.601 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:13:43.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:43.601 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:13:43.650 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:13:43.651 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42356sec preferred_lft 42356sec 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:43.652 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:13:43.653 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:13:43.653 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:13:43.653 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:43.653 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:13:43.653 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:13:43.653 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:13:43.653 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:13:43.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:43.654 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:13:43.728 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:13:43.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:43.732 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:13:43.780 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:13:43.781 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:13:43.781 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:13:43.781 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:13:43.781 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:13:43.782 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:13:43.782 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:13:43.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:13:43.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:13:43.782 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:13:43.782 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:13:43.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:13:43.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:13:43.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:43.985 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:13:44.029 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:13:44.029 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:13:44.030 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:13:44.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:44.030 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:13:44.154 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:13:44.155 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:13:44.155 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:44.207 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:13:44.207 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:13:44.207 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:44.254 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:13:44.333 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:13:44.387 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:13:44.388 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:13:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:13:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:13:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:13:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:13:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:13:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:13:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:13:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:13:44.389 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:13:44.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:44.390 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:13:44.437 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:13:44.437 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:13:44.442 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:13:44.509 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:13:44.553 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:13:44.554 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:13:44.555 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:13:44.555 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:13:44.555 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:13:44.555 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:13:44.555 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:13:44.555 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:13:44.555 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:13:44.555 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:13:44.555 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:13:44.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:44.556 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:13:44.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:13:44.614+0000 7f9e77979480 -1 init, newargv = 0x5650c6827e80 newargc=15 2024-04-24T08:13:44.615 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[58941]: starting ceph client 2024-04-24T08:13:44.623 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[58941]: starting fuse 2024-04-24T08:13:44.640 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:13:44.641 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:13:44.641 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:13:44.641 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:13:44.641 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:13:44.641 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:13:44.641 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:13:44.641 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:44.641 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:13:44.641 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:13:44.641 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:13:44.642 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:13:44.643 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:13:44.643 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:44.643 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:13:44.643 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:13:44.779 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:13:44.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:13:44.826 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.58941.asok 2024-04-24T08:13:44.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:13:44.843 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58941.asok 2024-04-24T08:13:44.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:44.843 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58941.asok status 2024-04-24T08:13:45.107 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:13:45.107 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:13:45.107 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:13:45.107 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:13:45.107 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:13:45.107 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "58941", 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 7116, 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 7116 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:45.108 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 4012555405 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 4012555405 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.7116 192.168.0.1:0/4012555405", 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/4012555405", 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:13:45.109 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 50, 2024-04-24T08:13:45.110 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 69, 2024-04-24T08:13:45.110 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:13:45.110 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:13:45.110 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:13:45.110 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:13:45.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:45.121 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:13:45.121 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:45.169 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:13:45.170 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:45.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:45.170 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:45.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:45.247 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir/) 2024-04-24T08:13:45.305 INFO:teuthology.orchestra.run.smithi069.stdout:file_undamaged 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> import json 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> import sys 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> try: 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> except OSError as e: 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> sys.exit(e.errno) 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> print(json.dumps( 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> dict([(a, getattr(s, a)) for a in attrs]), 2024-04-24T08:13:45.307 DEBUG:teuthology.orchestra.run.smithi069:> indent=2)) 2024-04-24T08:13:45.308 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:13:45.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45 stdin-killer INFO: command exited with status 2: exiting normally with same code! 2024-04-24T08:13:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-24T08:13:45.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:45.461 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-04-24T08:13:45.559 INFO:teuthology.orchestra.run.smithi069.stdout:2 2024-04-24T08:13:45.559 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:13:45.559 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:13:45.609 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:45.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.809+0000 7f9560a00640 1 Processor -- start 2024-04-24T08:13:45.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.809+0000 7f9560a00640 1 -- start start 2024-04-24T08:13:45.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.811+0000 7f9560a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f955c0b30f0 0x7f955c167470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:45.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.811+0000 7f9560a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f955c15d660 con 0x7f955c156c60 2024-04-24T08:13:45.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.811+0000 7f9560a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f955c06dc30 con 0x7f955c15d1d0 2024-04-24T08:13:45.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.811+0000 7f9560a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f955c06df10 con 0x7f955c0b30f0 2024-04-24T08:13:45.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.811+0000 7f955a800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f955c156c60 0x7f955c15cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33620/0 (socket says 172.21.15.69:33620) 2024-04-24T08:13:45.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.811+0000 7f955a800640 1 -- 172.21.15.69:0/446495344 learned_addr learned my addr 172.21.15.69:0/446495344 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:45.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.812+0000 7f955b200640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f955c0b30f0 0x7f955c167470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:45.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.812+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 164398072 0 0) 0x7f955c15d660 con 0x7f955c156c60 2024-04-24T08:13:45.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.812+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9540003520 con 0x7f955c156c60 2024-04-24T08:13:45.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.812+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4175747819 0 0) 0x7f9540003520 con 0x7f955c156c60 2024-04-24T08:13:45.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.812+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f955c0b30f0 msgr2=0x7f955c167470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:45.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.812+0000 7f9559e00640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f955c0b30f0 0x7f955c167470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.812+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 >> v1:172.21.15.121:6789/0 conn(0x7f955c15d1d0 legacy=0x7f955c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.812+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955c06e1f0 con 0x7f955c156c60 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.812+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f954c001670 con 0x7f955c156c60 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 >> v1:172.21.15.69:6789/0 conn(0x7f955c156c60 legacy=0x7f955c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9559e00640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f955c0b30f0 0x7f95400039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9559e00640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9540003f60 0x7f9540004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9559e00640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9540004d40 0x7f95400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f955c06df10 con 0x7f9540003f60 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f954000baa0 con 0x7f9540004d40 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f954000bca0 con 0x7f955c0b30f0 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9553e00640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9540003f60 0x7f9540004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f955b200640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9540004d40 0x7f95400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f955a800640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f955c0b30f0 0x7f95400039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:45.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9553e00640 1 -- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f955c0b30f0 msgr2=0x7f95400039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:45.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9553e00640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f955c0b30f0 0x7f95400039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:45.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9553e00640 1 -- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9540004d40 msgr2=0x7f95400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:45.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9553e00640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9540004d40 0x7f95400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:45.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.813+0000 7f9553e00640 1 -- 172.21.15.69:0/446495344 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9540003520 con 0x7f9540003f60 2024-04-24T08:13:45.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.815+0000 7f9553e00640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9540003f60 0x7f9540004800 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f9544001750 tx=0x7f9540005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acec8b7b87c342c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:45.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.815+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544001370 con 0x7f9540003f60 2024-04-24T08:13:45.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.815+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95440052d0 con 0x7f9540003f60 2024-04-24T08:13:45.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.815+0000 7f9559e00640 1 -- 172.21.15.69:0/446495344 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954402abe0 con 0x7f9540003f60 2024-04-24T08:13:45.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.816+0000 7f9560a00640 1 -- 172.21.15.69:0/446495344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9540003f60 msgr2=0x7f9540004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:45.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.816+0000 7f9560a00640 1 --2- 172.21.15.69:0/446495344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9540003f60 0x7f9540004800 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f9544001750 tx=0x7f9540005830 comp rx=0 tx=0).stop 2024-04-24T08:13:45.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.816+0000 7f955b200640 1 -- 172.21.15.69:0/446495344 reap_dead start 2024-04-24T08:13:45.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.816+0000 7f9560a00640 1 -- 172.21.15.69:0/446495344 shutdown_connections 2024-04-24T08:13:45.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.816+0000 7f9560a00640 1 -- 172.21.15.69:0/446495344 >> 172.21.15.69:0/446495344 conn(0x7f955c0d0160 msgr2=0x7f955c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:45.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.816+0000 7f9560a00640 1 -- 172.21.15.69:0/446495344 shutdown_connections 2024-04-24T08:13:45.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.816+0000 7f9560a00640 1 -- 172.21.15.69:0/446495344 wait complete. 2024-04-24T08:13:45.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.818+0000 7f9560a00640 1 Processor -- start 2024-04-24T08:13:45.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.818+0000 7f9560a00640 1 -- start start 2024-04-24T08:13:45.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f9560a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9540003f60 0x7f955c154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:45.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f9560a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f955c145b40 0x7f955c1385a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:45.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f9560a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f955c158a80 0x7f955c158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:45.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f9560a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f955c0ad7e0 con 0x7f955c145b40 2024-04-24T08:13:45.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f9560a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f955c15d940 con 0x7f955c158a80 2024-04-24T08:13:45.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f9560a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f955c0ad960 con 0x7f9540003f60 2024-04-24T08:13:45.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f9553e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f955c145b40 0x7f955c1385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:45.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f9553e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f955c145b40 0x7f955c1385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60802/0 (socket says 172.21.15.69:60802) 2024-04-24T08:13:45.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f9553e00640 1 -- 172.21.15.69:0/4025977136 learned_addr learned my addr 172.21.15.69:0/4025977136 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:45.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f955a800640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9540003f60 0x7f955c154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:45.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.819+0000 7f955b200640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f955c158a80 0x7f955c158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:45.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.820+0000 7f9553e00640 1 -- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9540003f60 msgr2=0x7f955c154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:45.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.820+0000 7f9553e00640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9540003f60 0x7f955c154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:45.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.820+0000 7f9553e00640 1 -- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f955c158a80 msgr2=0x7f955c158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:45.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.820+0000 7f9553e00640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f955c158a80 0x7f955c158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:45.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.820+0000 7f9553e00640 1 -- 172.21.15.69:0/4025977136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f955c138ae0 con 0x7f955c145b40 2024-04-24T08:13:45.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.820+0000 7f9553e00640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f955c145b40 0x7f955c1385a0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9544001870 tx=0x7f9544001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:45.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.820+0000 7f9552a00640 1 -- 172.21.15.69:0/4025977136 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544040020 con 0x7f955c145b40 2024-04-24T08:13:45.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.820+0000 7f9552a00640 1 -- 172.21.15.69:0/4025977136 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9544007680 con 0x7f955c145b40 2024-04-24T08:13:45.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.820+0000 7f9552a00640 1 -- 172.21.15.69:0/4025977136 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544007a30 con 0x7f955c145b40 2024-04-24T08:13:45.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.820+0000 7f9560a00640 1 -- 172.21.15.69:0/4025977136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f955c138d80 con 0x7f955c145b40 2024-04-24T08:13:45.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.821+0000 7f9560a00640 1 -- 172.21.15.69:0/4025977136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f955c139340 con 0x7f955c145b40 2024-04-24T08:13:45.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.822+0000 7f9560a00640 1 -- 172.21.15.69:0/4025977136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f955c075d50 con 0x7f955c145b40 2024-04-24T08:13:45.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.823+0000 7f9552a00640 1 -- 172.21.15.69:0/4025977136 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9544043020 con 0x7f955c145b40 2024-04-24T08:13:45.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.824+0000 7f9552a00640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f953806c8f0 0x7f953806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:45.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.824+0000 7f9552a00640 1 -- 172.21.15.69:0/4025977136 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f95440abbe0 con 0x7f955c145b40 2024-04-24T08:13:45.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.824+0000 7f955a800640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f953806c8f0 0x7f953806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:45.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.826+0000 7f955a800640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f953806c8f0 0x7f953806edb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f954c001cd0 tx=0x7f954c007040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:45.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:45.829+0000 7f9552a00640 1 -- 172.21.15.69:0/4025977136 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f954404a030 con 0x7f955c145b40 2024-04-24T08:13:46.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.030+0000 7f9560a00640 1 -- 172.21.15.69:0/4025977136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f955c06f9e0 con 0x7f955c145b40 2024-04-24T08:13:46.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.033+0000 7f9552a00640 1 -- 172.21.15.69:0/4025977136 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+16775 (secure 0 0 0) 0x7f9544079f10 con 0x7f955c145b40 2024-04-24T08:13:46.038 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:46.038 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":69,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:13:37.186047+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6837/1135108804":"2024-04-25T08:13:37.186005+0000","172.21.15.121:6835/1135108804":"2024-04-25T08:13:37.186005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 -- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f953806c8f0 msgr2=0x7f953806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f953806c8f0 0x7f953806edb0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f954c001cd0 tx=0x7f954c007040 comp rx=0 tx=0).stop 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 -- 172.21.15.69:0/4025977136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f955c145b40 msgr2=0x7f955c1385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f955c145b40 0x7f955c1385a0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f9544001870 tx=0x7f9544001120 comp rx=0 tx=0).stop 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 -- 172.21.15.69:0/4025977136 shutdown_connections 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f953806c8f0 0x7f953806edb0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f955c145b40 0x7f955c1385a0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f955c158a80 0x7f955c158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 --2- 172.21.15.69:0/4025977136 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9540003f60 0x7f955c154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 -- 172.21.15.69:0/4025977136 >> 172.21.15.69:0/4025977136 conn(0x7f955c0d0160 msgr2=0x7f955c0b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.036+0000 7f9560a00640 1 -- 172.21.15.69:0/4025977136 shutdown_connections 2024-04-24T08:13:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.037+0000 7f9560a00640 1 -- 172.21.15.69:0/4025977136 wait complete. 2024-04-24T08:13:46.057 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:13:13.250209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:13:14.468238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:13:46.058 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:13:46.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.257+0000 7f686d600640 1 Processor -- start 2024-04-24T08:13:46.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.257+0000 7f686d600640 1 -- start start 2024-04-24T08:13:46.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.259+0000 7f686d600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f68680b3de0 0x7f68680732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.259+0000 7f686d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 0x7f68680b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.259+0000 7f686d600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6868073cc0 con 0x7f686815f4b0 2024-04-24T08:13:46.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.259+0000 7f686d600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6868073fa0 con 0x7f68680b3de0 2024-04-24T08:13:46.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.259+0000 7f686d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f68680b3300 con 0x7f6868073820 2024-04-24T08:13:46.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.259+0000 7f6866000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f68680b3de0 0x7f68680732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.259+0000 7f6866a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f686815f4b0 0x7f686815f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33628/0 (socket says 172.21.15.69:33628) 2024-04-24T08:13:46.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.259+0000 7f6866a00640 1 -- 172.21.15.69:0/3176588272 learned_addr learned my addr 172.21.15.69:0/3176588272 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:46.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.259+0000 7f6867400640 1 --2- 172.21.15.69:0/3176588272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 0x7f68680b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.259+0000 7f6866000640 1 -- 172.21.15.69:0/3176588272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 msgr2=0x7f68680b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:46.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.260+0000 7f6866000640 1 --2- 172.21.15.69:0/3176588272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 0x7f68680b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.260+0000 7f6866000640 1 -- 172.21.15.69:0/3176588272 >> v1:172.21.15.69:6789/0 conn(0x7f686815f4b0 legacy=0x7f686815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.260+0000 7f6866000640 1 -- 172.21.15.69:0/3176588272 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68680b35e0 con 0x7f68680b3de0 2024-04-24T08:13:46.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.260+0000 7f6867400640 1 --2- 172.21.15.69:0/3176588272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 0x7f68680b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:46.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.261+0000 7f6866000640 1 --2- 172.21.15.69:0/3176588272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68680b3de0 0x7f68680732e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f68500097a0 tx=0x7f685002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1887d3fc98bdd74d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.261+0000 7f6865600640 1 -- 172.21.15.69:0/3176588272 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850037070 con 0x7f68680b3de0 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.261+0000 7f6865600640 1 -- 172.21.15.69:0/3176588272 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6850032440 con 0x7f68680b3de0 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.261+0000 7f6865600640 1 -- 172.21.15.69:0/3176588272 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6850032810 con 0x7f68680b3de0 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.262+0000 7f686d600640 1 -- 172.21.15.69:0/3176588272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68680b3de0 msgr2=0x7f68680732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.262+0000 7f686d600640 1 --2- 172.21.15.69:0/3176588272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68680b3de0 0x7f68680732e0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f68500097a0 tx=0x7f685002a180 comp rx=0 tx=0).stop 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.262+0000 7f686d600640 1 -- 172.21.15.69:0/3176588272 shutdown_connections 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.262+0000 7f686d600640 1 --2- 172.21.15.69:0/3176588272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 0x7f68680b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.262+0000 7f686d600640 1 --2- 172.21.15.69:0/3176588272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68680b3de0 0x7f68680732e0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.262+0000 7f686d600640 1 -- 172.21.15.69:0/3176588272 >> 172.21.15.69:0/3176588272 conn(0x7f68680d0350 msgr2=0x7f68680d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.262+0000 7f686d600640 1 -- 172.21.15.69:0/3176588272 shutdown_connections 2024-04-24T08:13:46.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.262+0000 7f686d600640 1 -- 172.21.15.69:0/3176588272 wait complete. 2024-04-24T08:13:46.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.263+0000 7f686d600640 1 Processor -- start 2024-04-24T08:13:46.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.264+0000 7f686d600640 1 -- start start 2024-04-24T08:13:46.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f686d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 0x7f686814a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f686d600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680b3de0 0x7f68681f4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f686d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686815f4b0 0x7f68680ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f686d600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f68680b4d60 con 0x7f686815f4b0 2024-04-24T08:13:46.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f686d600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f68680b4be0 con 0x7f68680b3de0 2024-04-24T08:13:46.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f686d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f68680b4ee0 con 0x7f6868073820 2024-04-24T08:13:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f6867400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686815f4b0 0x7f68680ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f6867400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686815f4b0 0x7f68680ae3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60814/0 (socket says 172.21.15.69:60814) 2024-04-24T08:13:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f6867400640 1 -- 172.21.15.69:0/2877554473 learned_addr learned my addr 172.21.15.69:0/2877554473 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f6866000640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680b3de0 0x7f68681f4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.265+0000 7f6866a00640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 0x7f686814a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.266+0000 7f6867400640 1 -- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 msgr2=0x7f686814a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.266+0000 7f6867400640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 0x7f686814a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.266+0000 7f6867400640 1 -- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680b3de0 msgr2=0x7f68681f4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.266+0000 7f6867400640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680b3de0 0x7f68681f4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.266+0000 7f6867400640 1 -- 172.21.15.69:0/2877554473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68681f49a0 con 0x7f686815f4b0 2024-04-24T08:13:46.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.266+0000 7f6866000640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680b3de0 0x7f68681f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:46.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.266+0000 7f6867400640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686815f4b0 0x7f68680ae3c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f685800c5e0 tx=0x7f685800caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:46.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.267+0000 7f685ea00640 1 -- 172.21.15.69:0/2877554473 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6858015070 con 0x7f686815f4b0 2024-04-24T08:13:46.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.267+0000 7f685ea00640 1 -- 172.21.15.69:0/2877554473 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6858004240 con 0x7f686815f4b0 2024-04-24T08:13:46.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.267+0000 7f685ea00640 1 -- 172.21.15.69:0/2877554473 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6858003610 con 0x7f686815f4b0 2024-04-24T08:13:46.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.267+0000 7f686d600640 1 -- 172.21.15.69:0/2877554473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68681ea7a0 con 0x7f686815f4b0 2024-04-24T08:13:46.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.267+0000 7f686d600640 1 -- 172.21.15.69:0/2877554473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68680cbf60 con 0x7f686815f4b0 2024-04-24T08:13:46.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.269+0000 7f686d600640 1 -- 172.21.15.69:0/2877554473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6868068a80 con 0x7f686815f4b0 2024-04-24T08:13:46.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.270+0000 7f685ea00640 1 -- 172.21.15.69:0/2877554473 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f68580037b0 con 0x7f686815f4b0 2024-04-24T08:13:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.271+0000 7f685ea00640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f684406c8f0 0x7f684406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.271+0000 7f685ea00640 1 -- 172.21.15.69:0/2877554473 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f685808be20 con 0x7f686815f4b0 2024-04-24T08:13:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.271+0000 7f6866a00640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f684406c8f0 0x7f684406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.272+0000 7f6866a00640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f684406c8f0 0x7f684406edb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f68600012e0 tx=0x7f6860009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:46.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.275+0000 7f685ea00640 1 -- 172.21.15.69:0/2877554473 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f685805a150 con 0x7f686815f4b0 2024-04-24T08:13:46.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.477+0000 7f686d600640 1 -- 172.21.15.69:0/2877554473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f686806faa0 con 0x7f686815f4b0 2024-04-24T08:13:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.478+0000 7f685ea00640 1 -- 172.21.15.69:0/2877554473 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) ==== 93+0+10 (secure 0 0 0) 0x7f6858059a50 con 0x7f686815f4b0 2024-04-24T08:13:46.479 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 -- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f684406c8f0 msgr2=0x7f684406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f684406c8f0 0x7f684406edb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f68600012e0 tx=0x7f6860009040 comp rx=0 tx=0).stop 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 -- 172.21.15.69:0/2877554473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686815f4b0 msgr2=0x7f68680ae3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686815f4b0 0x7f68680ae3c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f685800c5e0 tx=0x7f685800caa0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 -- 172.21.15.69:0/2877554473 shutdown_connections 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f684406c8f0 0x7f684406edb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f686815f4b0 0x7f68680ae3c0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68680b3de0 0x7f68681f4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 --2- 172.21.15.69:0/2877554473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6868073820 0x7f686814a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 -- 172.21.15.69:0/2877554473 >> 172.21.15.69:0/2877554473 conn(0x7f68680d0350 msgr2=0x7f68680d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 -- 172.21.15.69:0/2877554473 shutdown_connections 2024-04-24T08:13:46.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.481+0000 7f686d600640 1 -- 172.21.15.69:0/2877554473 wait complete. 2024-04-24T08:13:46.501 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:13:46.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.703+0000 7fa115a00640 1 Processor -- start 2024-04-24T08:13:46.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.703+0000 7fa115a00640 1 -- start start 2024-04-24T08:13:46.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa115a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa110155f20 0x7fa1100ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa115a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa11015c9b0 0x7fa1100b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa115a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1100b3f70 0x7fa1100cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa115a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa11006db90 con 0x7fa11015c9b0 2024-04-24T08:13:46.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa115a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa110072070 con 0x7fa110155f20 2024-04-24T08:13:46.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa115a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa1100cd6d0 con 0x7fa1100b3f70 2024-04-24T08:13:46.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa107e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa11015c9b0 0x7fa1100b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa107e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa11015c9b0 0x7fa1100b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60830/0 (socket says 172.21.15.69:60830) 2024-04-24T08:13:46.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa107e00640 1 -- 172.21.15.69:0/3277048762 learned_addr learned my addr 172.21.15.69:0/3277048762 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:46.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa10ea00640 1 --2- 172.21.15.69:0/3277048762 >> v2:172.21.15.121:3300/0 conn(0x7fa110155f20 0x7fa1100ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.705+0000 7fa10f400640 1 --2- 172.21.15.69:0/3277048762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1100b3f70 0x7fa1100cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.706+0000 7fa107e00640 1 -- 172.21.15.69:0/3277048762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1100b3f70 msgr2=0x7fa1100cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:46.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.706+0000 7fa107e00640 1 --2- 172.21.15.69:0/3277048762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1100b3f70 0x7fa1100cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.706+0000 7fa107e00640 1 -- 172.21.15.69:0/3277048762 >> v2:172.21.15.121:3300/0 conn(0x7fa110155f20 msgr2=0x7fa1100ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:46.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.706+0000 7fa107e00640 1 --2- 172.21.15.69:0/3277048762 >> v2:172.21.15.121:3300/0 conn(0x7fa110155f20 0x7fa1100ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.706+0000 7fa107e00640 1 -- 172.21.15.69:0/3277048762 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1100cd9b0 con 0x7fa11015c9b0 2024-04-24T08:13:46.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.707+0000 7fa107e00640 1 --2- 172.21.15.69:0/3277048762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 0x7fa1100b3a30 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fa110159870 tx=0x7fa0f802c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f0e55f1b020c3ff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:46.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.707+0000 7fa10e000640 1 -- 172.21.15.69:0/3277048762 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8036020 con 0x7fa11015c9b0 2024-04-24T08:13:46.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.707+0000 7fa10e000640 1 -- 172.21.15.69:0/3277048762 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f8032820 con 0x7fa11015c9b0 2024-04-24T08:13:46.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.707+0000 7fa10e000640 1 -- 172.21.15.69:0/3277048762 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8032bf0 con 0x7fa11015c9b0 2024-04-24T08:13:46.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.708+0000 7fa115a00640 1 -- 172.21.15.69:0/3277048762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 msgr2=0x7fa1100b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.708+0000 7fa115a00640 1 --2- 172.21.15.69:0/3277048762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 0x7fa1100b3a30 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fa110159870 tx=0x7fa0f802c920 comp rx=0 tx=0).stop 2024-04-24T08:13:46.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.708+0000 7fa115a00640 1 -- 172.21.15.69:0/3277048762 shutdown_connections 2024-04-24T08:13:46.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.708+0000 7fa115a00640 1 --2- 172.21.15.69:0/3277048762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa1100b3f70 0x7fa1100cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.708+0000 7fa115a00640 1 --2- 172.21.15.69:0/3277048762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 0x7fa1100b3a30 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.708+0000 7fa115a00640 1 --2- 172.21.15.69:0/3277048762 >> v2:172.21.15.121:3300/0 conn(0x7fa110155f20 0x7fa1100ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.708+0000 7fa115a00640 1 -- 172.21.15.69:0/3277048762 >> 172.21.15.69:0/3277048762 conn(0x7fa1100d0350 msgr2=0x7fa1100d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:46.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.708+0000 7fa115a00640 1 -- 172.21.15.69:0/3277048762 shutdown_connections 2024-04-24T08:13:46.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.708+0000 7fa115a00640 1 -- 172.21.15.69:0/3277048762 wait complete. 2024-04-24T08:13:46.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.710+0000 7fa115a00640 1 Processor -- start 2024-04-24T08:13:46.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.710+0000 7fa115a00640 1 -- start start 2024-04-24T08:13:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.711+0000 7fa115a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1100b3f70 0x7fa1101f4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.711+0000 7fa115a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa110155f20 0x7fa110154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.711+0000 7fa115a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 0x7fa11015de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.711+0000 7fa115a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa1100cd2a0 con 0x7fa11015c9b0 2024-04-24T08:13:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.711+0000 7fa115a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa110071d00 con 0x7fa1100b3f70 2024-04-24T08:13:46.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.711+0000 7fa115a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa1100cd420 con 0x7fa110155f20 2024-04-24T08:13:46.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.712+0000 7fa10ea00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1100b3f70 0x7fa1101f4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.712+0000 7fa10f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 0x7fa11015de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.712+0000 7fa10f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 0x7fa11015de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60838/0 (socket says 172.21.15.69:60838) 2024-04-24T08:13:46.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.712+0000 7fa10f400640 1 -- 172.21.15.69:0/2895207854 learned_addr learned my addr 172.21.15.69:0/2895207854 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:46.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.712+0000 7fa107e00640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa110155f20 0x7fa110154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.712+0000 7fa10f400640 1 -- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa110155f20 msgr2=0x7fa110154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.712+0000 7fa10f400640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa110155f20 0x7fa110154c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.712+0000 7fa10f400640 1 -- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1100b3f70 msgr2=0x7fa1101f4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.712+0000 7fa10f400640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1100b3f70 0x7fa1101f4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.712+0000 7fa10f400640 1 -- 172.21.15.69:0/2895207854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa11015e7a0 con 0x7fa11015c9b0 2024-04-24T08:13:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.713+0000 7fa10f400640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 0x7fa11015de20 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa108007e30 tx=0x7fa10800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.713+0000 7fa106a00640 1 -- 172.21.15.69:0/2895207854 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa108014420 con 0x7fa11015c9b0 2024-04-24T08:13:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.713+0000 7fa106a00640 1 -- 172.21.15.69:0/2895207854 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa108019070 con 0x7fa11015c9b0 2024-04-24T08:13:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.713+0000 7fa115a00640 1 -- 172.21.15.69:0/2895207854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1100ae4d0 con 0x7fa11015c9b0 2024-04-24T08:13:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.713+0000 7fa106a00640 1 -- 172.21.15.69:0/2895207854 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa108014800 con 0x7fa11015c9b0 2024-04-24T08:13:46.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.713+0000 7fa115a00640 1 -- 172.21.15.69:0/2895207854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1101f4cb0 con 0x7fa11015c9b0 2024-04-24T08:13:46.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.714+0000 7fa115a00640 1 -- 172.21.15.69:0/2895207854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1100ae8c0 con 0x7fa11015c9b0 2024-04-24T08:13:46.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.715+0000 7fa106a00640 1 -- 172.21.15.69:0/2895207854 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa1080149a0 con 0x7fa11015c9b0 2024-04-24T08:13:46.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.716+0000 7fa106a00640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0ec06c8f0 0x7fa0ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:46.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.716+0000 7fa106a00640 1 -- 172.21.15.69:0/2895207854 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7fa10808c340 con 0x7fa11015c9b0 2024-04-24T08:13:46.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.717+0000 7fa10ea00640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0ec06c8f0 0x7fa0ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:46.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.718+0000 7fa10ea00640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0ec06c8f0 0x7fa0ec06edb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa100006fd0 tx=0x7fa100008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:46.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.721+0000 7fa106a00640 1 -- 172.21.15.69:0/2895207854 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa10805a700 con 0x7fa11015c9b0 2024-04-24T08:13:46.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.915+0000 7fa115a00640 1 -- 172.21.15.69:0/2895207854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa110068a80 con 0x7fa11015c9b0 2024-04-24T08:13:46.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.916+0000 7fa106a00640 1 -- 172.21.15.69:0/2895207854 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) ==== 104+0+11 (secure 0 0 0) 0x7fa10805a000 con 0x7fa11015c9b0 2024-04-24T08:13:46.917 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:13:46.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.918+0000 7fa115a00640 1 -- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0ec06c8f0 msgr2=0x7fa0ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.918+0000 7fa115a00640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0ec06c8f0 0x7fa0ec06edb0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa100006fd0 tx=0x7fa100008040 comp rx=0 tx=0).stop 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.918+0000 7fa115a00640 1 -- 172.21.15.69:0/2895207854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 msgr2=0x7fa11015de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.918+0000 7fa115a00640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 0x7fa11015de20 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa108007e30 tx=0x7fa10800e4b0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.919+0000 7fa115a00640 1 -- 172.21.15.69:0/2895207854 shutdown_connections 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.919+0000 7fa115a00640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa0ec06c8f0 0x7fa0ec06edb0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.919+0000 7fa115a00640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa11015c9b0 0x7fa11015de20 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.919+0000 7fa115a00640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa110155f20 0x7fa110154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.919+0000 7fa115a00640 1 --2- 172.21.15.69:0/2895207854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1100b3f70 0x7fa1101f4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.919+0000 7fa115a00640 1 -- 172.21.15.69:0/2895207854 >> 172.21.15.69:0/2895207854 conn(0x7fa1100d0350 msgr2=0x7fa1100ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.919+0000 7fa115a00640 1 -- 172.21.15.69:0/2895207854 shutdown_connections 2024-04-24T08:13:46.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:46.919+0000 7fa115a00640 1 -- 172.21.15.69:0/2895207854 wait complete. 2024-04-24T08:13:46.939 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:13:47.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.141+0000 7f0f04e00640 1 Processor -- start 2024-04-24T08:13:47.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.141+0000 7f0f04e00640 1 -- start start 2024-04-24T08:13:47.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.142+0000 7f0f04e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00155f20 0x7f0f000ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.142+0000 7f0f04e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0f0015c9b0 0x7f0f000b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.142+0000 7f0f04e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f000cd6d0 con 0x7f0f000b3f70 2024-04-24T08:13:47.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.142+0000 7f0f04e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0f0006db90 con 0x7f0f0015c9b0 2024-04-24T08:13:47.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.142+0000 7f0f04e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0f00072070 con 0x7f0f00155f20 2024-04-24T08:13:47.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.143+0000 7f0efd400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0f0015c9b0 0x7f0f000b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.143+0000 7f0efe800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0f000b3f70 0x7f0f000cf360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33640/0 (socket says 172.21.15.69:33640) 2024-04-24T08:13:47.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.143+0000 7f0efe800640 1 -- 172.21.15.69:0/2986675182 learned_addr learned my addr 172.21.15.69:0/2986675182 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:47.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.143+0000 7f0efde00640 1 --2- 172.21.15.69:0/2986675182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00155f20 0x7f0f000ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.143+0000 7f0efd400640 1 -- 172.21.15.69:0/2986675182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00155f20 msgr2=0x7f0f000ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:47.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.143+0000 7f0efd400640 1 --2- 172.21.15.69:0/2986675182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00155f20 0x7f0f000ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.143+0000 7f0efd400640 1 -- 172.21.15.69:0/2986675182 >> v1:172.21.15.69:6789/0 conn(0x7f0f000b3f70 legacy=0x7f0f000cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.143+0000 7f0efd400640 1 -- 172.21.15.69:0/2986675182 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f000cd9b0 con 0x7f0f0015c9b0 2024-04-24T08:13:47.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.143+0000 7f0efca00640 1 -- 172.21.15.69:0/2986675182 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3009296219 0 0) 0x7f0f000cd6d0 con 0x7f0f000b3f70 2024-04-24T08:13:47.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.144+0000 7f0efd400640 1 --2- 172.21.15.69:0/2986675182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f0015c9b0 0x7f0f000b3a30 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0f000721f0 tx=0x7f0eec02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2246de1fac26fa10 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:47.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.145+0000 7f0efca00640 1 -- 172.21.15.69:0/2986675182 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec037070 con 0x7f0f0015c9b0 2024-04-24T08:13:47.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.145+0000 7f0efca00640 1 -- 172.21.15.69:0/2986675182 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec032440 con 0x7f0f0015c9b0 2024-04-24T08:13:47.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.145+0000 7f0efca00640 1 -- 172.21.15.69:0/2986675182 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec032810 con 0x7f0f0015c9b0 2024-04-24T08:13:47.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.145+0000 7f0f04e00640 1 -- 172.21.15.69:0/2986675182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f0015c9b0 msgr2=0x7f0f000b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.145+0000 7f0f04e00640 1 --2- 172.21.15.69:0/2986675182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f0015c9b0 0x7f0f000b3a30 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f0f000721f0 tx=0x7f0eec02a210 comp rx=0 tx=0).stop 2024-04-24T08:13:47.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.145+0000 7f0f04e00640 1 -- 172.21.15.69:0/2986675182 shutdown_connections 2024-04-24T08:13:47.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.145+0000 7f0f04e00640 1 --2- 172.21.15.69:0/2986675182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f0015c9b0 0x7f0f000b3a30 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.145+0000 7f0f04e00640 1 --2- 172.21.15.69:0/2986675182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f00155f20 0x7f0f000ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.145+0000 7f0f04e00640 1 -- 172.21.15.69:0/2986675182 >> 172.21.15.69:0/2986675182 conn(0x7f0f000d0350 msgr2=0x7f0f000d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:47.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.145+0000 7f0f04e00640 1 -- 172.21.15.69:0/2986675182 shutdown_connections 2024-04-24T08:13:47.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.146+0000 7f0f04e00640 1 -- 172.21.15.69:0/2986675182 wait complete. 2024-04-24T08:13:47.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.147+0000 7f0f04e00640 1 Processor -- start 2024-04-24T08:13:47.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.147+0000 7f0f04e00640 1 -- start start 2024-04-24T08:13:47.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.148+0000 7f0f04e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f000b3f70 0x7f0f00151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.148+0000 7f0f04e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00155f20 0x7f0f00158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.148+0000 7f0f04e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f0015c9b0 0x7f0f0015a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.148+0000 7f0f04e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0f000cd2a0 con 0x7f0f00155f20 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.148+0000 7f0f04e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0f00071d80 con 0x7f0f000b3f70 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.148+0000 7f0f04e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0f000cd420 con 0x7f0f0015c9b0 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efd400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00155f20 0x7f0f00158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efd400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00155f20 0x7f0f00158d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60848/0 (socket says 172.21.15.69:60848) 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efd400640 1 -- 172.21.15.69:0/350739923 learned_addr learned my addr 172.21.15.69:0/350739923 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efde00640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f000b3f70 0x7f0f00151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efe800640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f0015c9b0 0x7f0f0015a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efd400640 1 -- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f0015c9b0 msgr2=0x7f0f0015a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efd400640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f0015c9b0 0x7f0f0015a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efd400640 1 -- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f000b3f70 msgr2=0x7f0f00151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efd400640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f000b3f70 0x7f0f00151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efd400640 1 -- 172.21.15.69:0/350739923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0006ed70 con 0x7f0f00155f20 2024-04-24T08:13:47.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efde00640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f000b3f70 0x7f0f00151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.149+0000 7f0efd400640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00155f20 0x7f0f00158d00 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009890 tx=0x7f0eec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.150+0000 7f0ef6000640 1 -- 172.21.15.69:0/350739923 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec037040 con 0x7f0f00155f20 2024-04-24T08:13:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.150+0000 7f0ef6000640 1 -- 172.21.15.69:0/350739923 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eec042070 con 0x7f0f00155f20 2024-04-24T08:13:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.150+0000 7f0ef6000640 1 -- 172.21.15.69:0/350739923 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eec030bd0 con 0x7f0f00155f20 2024-04-24T08:13:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.150+0000 7f0f04e00640 1 -- 172.21.15.69:0/350739923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f000ad490 con 0x7f0f00155f20 2024-04-24T08:13:47.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.150+0000 7f0f04e00640 1 -- 172.21.15.69:0/350739923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f000ada50 con 0x7f0f00155f20 2024-04-24T08:13:47.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.151+0000 7f0f04e00640 1 -- 172.21.15.69:0/350739923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f00068a80 con 0x7f0f00155f20 2024-04-24T08:13:47.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.152+0000 7f0ef6000640 1 -- 172.21.15.69:0/350739923 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0eec047460 con 0x7f0f00155f20 2024-04-24T08:13:47.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.153+0000 7f0ef6000640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 0x7f0ee006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.153+0000 7f0ef6000640 1 -- 172.21.15.69:0/350739923 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f0eec0ac840 con 0x7f0f00155f20 2024-04-24T08:13:47.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.153+0000 7f0efde00640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 0x7f0ee006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.154+0000 7f0efde00640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 0x7f0ee006edb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0008890 tx=0x7f0ef0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:47.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.157+0000 7f0ef6000640 1 -- 172.21.15.69:0/350739923 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0eec032e20 con 0x7f0f00155f20 2024-04-24T08:13:47.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.362+0000 7f0f04e00640 1 -- 172.21.15.69:0/350739923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0f000ce940 con 0x7f0f00155f20 2024-04-24T08:13:47.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.363+0000 7f0ef6000640 1 -- 172.21.15.69:0/350739923 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) ==== 100+0+11 (secure 0 0 0) 0x7f0eec07ac00 con 0x7f0f00155f20 2024-04-24T08:13:47.365 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 -- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 msgr2=0x7f0ee006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 0x7f0ee006edb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0008890 tx=0x7f0ef0008040 comp rx=0 tx=0).stop 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 -- 172.21.15.69:0/350739923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00155f20 msgr2=0x7f0f00158d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00155f20 0x7f0f00158d00 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f0eec009890 tx=0x7f0eec0046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 -- 172.21.15.69:0/350739923 shutdown_connections 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ee006c8f0 0x7f0ee006edb0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f00155f20 0x7f0f00158d00 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0f0015c9b0 0x7f0f0015a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 --2- 172.21.15.69:0/350739923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f000b3f70 0x7f0f00151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 -- 172.21.15.69:0/350739923 >> 172.21.15.69:0/350739923 conn(0x7f0f000d0350 msgr2=0x7f0f0006bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 -- 172.21.15.69:0/350739923 shutdown_connections 2024-04-24T08:13:47.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.366+0000 7f0f04e00640 1 -- 172.21.15.69:0/350739923 wait complete. 2024-04-24T08:13:47.386 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:13:47.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.585+0000 7f604fe00640 1 Processor -- start 2024-04-24T08:13:47.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.585+0000 7f604fe00640 1 -- start start 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.587+0000 7f604fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 0x7f6050143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.587+0000 7f604fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6050072230 0x7f6050143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.587+0000 7f604fe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f605015f540 con 0x7f6050071e50 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.587+0000 7f604fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f605015f820 con 0x7f6050072230 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.587+0000 7f604fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f605015fb00 con 0x7f6050162300 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.587+0000 7f604ea00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6050071e50 0x7f605015ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33652/0 (socket says 172.21.15.69:33652) 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.587+0000 7f604ea00640 1 -- 172.21.15.69:0/2600947548 learned_addr learned my addr 172.21.15.69:0/2600947548 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.587+0000 7f604f400640 1 --2- 172.21.15.69:0/2600947548 >> v2:172.21.15.69:3300/0 conn(0x7f6050072230 0x7f6050143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.588+0000 7f6046000640 1 --2- 172.21.15.69:0/2600947548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 0x7f6050143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.588+0000 7f604e000640 1 -- 172.21.15.69:0/2600947548 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 528321208 0 0) 0x7f605015f540 con 0x7f6050071e50 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.588+0000 7f604e000640 1 -- 172.21.15.69:0/2600947548 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6034003520 con 0x7f6050071e50 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.588+0000 7f604f400640 1 -- 172.21.15.69:0/2600947548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 msgr2=0x7f6050143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:47.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.588+0000 7f604f400640 1 --2- 172.21.15.69:0/2600947548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 0x7f6050143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.588+0000 7f604f400640 1 -- 172.21.15.69:0/2600947548 >> v1:172.21.15.69:6789/0 conn(0x7f6050071e50 legacy=0x7f605015ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.588+0000 7f604f400640 1 -- 172.21.15.69:0/2600947548 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f605006dda0 con 0x7f6050072230 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.588+0000 7f6046000640 1 --2- 172.21.15.69:0/2600947548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 0x7f6050143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.589+0000 7f604f400640 1 --2- 172.21.15.69:0/2600947548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050072230 0x7f6050143bd0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f6048009d90 tx=0x7f604802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48be73d07c87924 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.589+0000 7f604e000640 1 -- 172.21.15.69:0/2600947548 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048007740 con 0x7f6050072230 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.589+0000 7f604e000640 1 -- 172.21.15.69:0/2600947548 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60480078e0 con 0x7f6050072230 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.589+0000 7f604e000640 1 -- 172.21.15.69:0/2600947548 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048007cb0 con 0x7f6050072230 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.590+0000 7f604fe00640 1 -- 172.21.15.69:0/2600947548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050072230 msgr2=0x7f6050143bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.590+0000 7f604fe00640 1 --2- 172.21.15.69:0/2600947548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050072230 0x7f6050143bd0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f6048009d90 tx=0x7f604802a660 comp rx=0 tx=0).stop 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.590+0000 7f604fe00640 1 -- 172.21.15.69:0/2600947548 shutdown_connections 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.590+0000 7f604fe00640 1 --2- 172.21.15.69:0/2600947548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050072230 0x7f6050143bd0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.590+0000 7f604fe00640 1 --2- 172.21.15.69:0/2600947548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 0x7f6050143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.590+0000 7f604fe00640 1 -- 172.21.15.69:0/2600947548 >> 172.21.15.69:0/2600947548 conn(0x7f60500d0160 msgr2=0x7f60500d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.590+0000 7f604fe00640 1 -- 172.21.15.69:0/2600947548 shutdown_connections 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.590+0000 7f604fe00640 1 -- 172.21.15.69:0/2600947548 wait complete. 2024-04-24T08:13:47.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.592+0000 7f604fe00640 1 Processor -- start 2024-04-24T08:13:47.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.592+0000 7f604fe00640 1 -- start start 2024-04-24T08:13:47.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050071e50 0x7f605009a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6050072230 0x7f605009c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 0x7f60500b3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6050144290 con 0x7f6050071e50 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604fe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6050144110 con 0x7f6050072230 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6050144410 con 0x7f6050162300 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050071e50 0x7f605009a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050071e50 0x7f605009a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60860/0 (socket says 172.21.15.69:60860) 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604ea00640 1 -- 172.21.15.69:0/3237784745 learned_addr learned my addr 172.21.15.69:0/3237784745 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604f400640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 0x7f60500b3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604ea00640 1 -- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 msgr2=0x7f60500b3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f6046000640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6050072230 0x7f605009c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604ea00640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 0x7f60500b3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604ea00640 1 -- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6050072230 msgr2=0x7f605009c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604ea00640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6050072230 0x7f605009c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604ea00640 1 -- 172.21.15.69:0/3237784745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60500b31d0 con 0x7f6050071e50 2024-04-24T08:13:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.593+0000 7f604f400640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 0x7f60500b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.594+0000 7f604ea00640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050071e50 0x7f605009a4c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f604000c430 tx=0x7f604000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.594+0000 7f6047400640 1 -- 172.21.15.69:0/3237784745 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040017070 con 0x7f6050071e50 2024-04-24T08:13:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.594+0000 7f6047400640 1 -- 172.21.15.69:0/3237784745 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6040005ce0 con 0x7f6050071e50 2024-04-24T08:13:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.594+0000 7f604fe00640 1 -- 172.21.15.69:0/3237784745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f605015fec0 con 0x7f6050071e50 2024-04-24T08:13:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.594+0000 7f6047400640 1 -- 172.21.15.69:0/3237784745 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60400072f0 con 0x7f6050071e50 2024-04-24T08:13:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.594+0000 7f604fe00640 1 -- 172.21.15.69:0/3237784745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6050160760 con 0x7f6050071e50 2024-04-24T08:13:47.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.595+0000 7f604fe00640 1 -- 172.21.15.69:0/3237784745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6050075d50 con 0x7f6050071e50 2024-04-24T08:13:47.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.596+0000 7f6047400640 1 -- 172.21.15.69:0/3237784745 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f604001a070 con 0x7f6050071e50 2024-04-24T08:13:47.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.598+0000 7f6047400640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f602c06c8f0 0x7f602c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:47.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.598+0000 7f6047400640 1 -- 172.21.15.69:0/3237784745 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f604008b130 con 0x7f6050071e50 2024-04-24T08:13:47.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.599+0000 7f6046000640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f602c06c8f0 0x7f602c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:47.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.600+0000 7f6046000640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f602c06c8f0 0x7f602c06edb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6050143ef0 tx=0x7f6038008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:47.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.602+0000 7f6047400640 1 -- 172.21.15.69:0/3237784745 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f60500b31d0 con 0x7f6050071e50 2024-04-24T08:13:47.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.823+0000 7f604fe00640 1 -- 172.21.15.69:0/3237784745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f605015dc90 con 0x7f6050071e50 2024-04-24T08:13:47.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.824+0000 7f6047400640 1 -- 172.21.15.69:0/3237784745 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v69) ==== 80+0+124 (secure 0 0 0) 0x7f6040059460 con 0x7f6050071e50 2024-04-24T08:13:47.825 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6837/1135108804 2024-04-25T08:13:37.186005+0000 2024-04-24T08:13:47.826 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6835/1135108804 2024-04-25T08:13:37.186005+0000 2024-04-24T08:13:47.826 INFO:teuthology.orchestra.run.smithi069.stderr:listed 2 entries 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 -- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f602c06c8f0 msgr2=0x7f602c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f602c06c8f0 0x7f602c06edb0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6050143ef0 tx=0x7f6038008040 comp rx=0 tx=0).stop 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 -- 172.21.15.69:0/3237784745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050071e50 msgr2=0x7f605009a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050071e50 0x7f605009a4c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f604000c430 tx=0x7f604000c8f0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 -- 172.21.15.69:0/3237784745 shutdown_connections 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f602c06c8f0 0x7f602c06edb0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6050162300 0x7f60500b3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6050072230 0x7f605009c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 --2- 172.21.15.69:0/3237784745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6050071e50 0x7f605009a4c0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 -- 172.21.15.69:0/3237784745 >> 172.21.15.69:0/3237784745 conn(0x7f60500d0160 msgr2=0x7f6050072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 -- 172.21.15.69:0/3237784745 shutdown_connections 2024-04-24T08:13:47.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:47.827+0000 7f604fe00640 1 -- 172.21.15.69:0/3237784745 wait complete. 2024-04-24T08:13:47.846 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:13:47.846 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:13:47.896 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:13:48.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.143+0000 7f7596e00640 1 Processor -- start 2024-04-24T08:13:48.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.143+0000 7f7596e00640 1 -- start start 2024-04-24T08:13:48.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f759015d1d0 0x7f75900ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75900b30f0 0x7f75900ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f759015d660 con 0x7f7590156c60 2024-04-24T08:13:48.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f759006dc30 con 0x7f75900b30f0 2024-04-24T08:13:48.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f759006df10 con 0x7f759015d1d0 2024-04-24T08:13:48.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75900b30f0 0x7f75900ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75900b30f0 0x7f75900ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60870/0 (socket says 172.21.15.69:60870) 2024-04-24T08:13:48.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596400640 1 -- 172.21.15.69:0/1406731907 learned_addr learned my addr 172.21.15.69:0/1406731907 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:48.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596400640 1 -- 172.21.15.69:0/1406731907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f759015d1d0 msgr2=0x7f75900ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7595000640 1 --2- 172.21.15.69:0/1406731907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f759015d1d0 0x7f75900ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596400640 1 --2- 172.21.15.69:0/1406731907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f759015d1d0 0x7f75900ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596400640 1 -- 172.21.15.69:0/1406731907 >> v1:172.21.15.121:6789/0 conn(0x7f7590156c60 legacy=0x7f759015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.145+0000 7f7596400640 1 -- 172.21.15.69:0/1406731907 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759006e1f0 con 0x7f75900b30f0 2024-04-24T08:13:48.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.146+0000 7f7596400640 1 --2- 172.21.15.69:0/1406731907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75900b30f0 0x7f75900ad610 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f7588009e50 tx=0x7f758802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabc260801dc872b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.147+0000 7f758fe00640 1 -- 172.21.15.69:0/1406731907 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7588036070 con 0x7f75900b30f0 2024-04-24T08:13:48.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.147+0000 7f758fe00640 1 -- 172.21.15.69:0/1406731907 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7588031600 con 0x7f75900b30f0 2024-04-24T08:13:48.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.147+0000 7f758fe00640 1 -- 172.21.15.69:0/1406731907 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75880319d0 con 0x7f75900b30f0 2024-04-24T08:13:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.147+0000 7f7596e00640 1 -- 172.21.15.69:0/1406731907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75900b30f0 msgr2=0x7f75900ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.147+0000 7f7596e00640 1 --2- 172.21.15.69:0/1406731907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75900b30f0 0x7f75900ad610 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f7588009e50 tx=0x7f758802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.147+0000 7f7596e00640 1 -- 172.21.15.69:0/1406731907 shutdown_connections 2024-04-24T08:13:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.147+0000 7f7596e00640 1 --2- 172.21.15.69:0/1406731907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75900b30f0 0x7f75900ad610 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.147+0000 7f7596e00640 1 --2- 172.21.15.69:0/1406731907 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f759015d1d0 0x7f75900ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.147+0000 7f7596e00640 1 -- 172.21.15.69:0/1406731907 >> 172.21.15.69:0/1406731907 conn(0x7f75900d0160 msgr2=0x7f75900d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.148+0000 7f7596e00640 1 -- 172.21.15.69:0/1406731907 shutdown_connections 2024-04-24T08:13:48.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.148+0000 7f7596e00640 1 -- 172.21.15.69:0/1406731907 wait complete. 2024-04-24T08:13:48.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.149+0000 7f7596e00640 1 Processor -- start 2024-04-24T08:13:48.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.149+0000 7f7596e00640 1 -- start start 2024-04-24T08:13:48.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.150+0000 7f7596e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75900b30f0 0x7f7590155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.150+0000 7f7596e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7590156c60 0x7f759015eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.150+0000 7f7596e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759015bf80 0x7f759015f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.150+0000 7f7596e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f759006e710 con 0x7f759015bf80 2024-04-24T08:13:48.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.150+0000 7f7596e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f759015d940 con 0x7f7590156c60 2024-04-24T08:13:48.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.150+0000 7f7596e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f759006e890 con 0x7f75900b30f0 2024-04-24T08:13:48.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.151+0000 7f7596400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759015bf80 0x7f759015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.151+0000 7f7596400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759015bf80 0x7f759015f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60886/0 (socket says 172.21.15.69:60886) 2024-04-24T08:13:48.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.151+0000 7f7596400640 1 -- 172.21.15.69:0/589946619 learned_addr learned my addr 172.21.15.69:0/589946619 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.151+0000 7f7595a00640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75900b30f0 0x7f7590155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.151+0000 7f7595000640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7590156c60 0x7f759015eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.151+0000 7f7596400640 1 -- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75900b30f0 msgr2=0x7f7590155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.151+0000 7f7596400640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75900b30f0 0x7f7590155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.151+0000 7f7596400640 1 -- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7590156c60 msgr2=0x7f759015eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.151+0000 7f7596400640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7590156c60 0x7f759015eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.151+0000 7f7596400640 1 -- 172.21.15.69:0/589946619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75900cabf0 con 0x7f759015bf80 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.152+0000 7f7596400640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759015bf80 0x7f759015f050 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f7588034040 tx=0x7f7588004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.152+0000 7f758d600640 1 -- 172.21.15.69:0/589946619 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7588036050 con 0x7f759015bf80 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.152+0000 7f758d600640 1 -- 172.21.15.69:0/589946619 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7588042070 con 0x7f759015bf80 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.152+0000 7f758d600640 1 -- 172.21.15.69:0/589946619 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7588009510 con 0x7f759015bf80 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.152+0000 7f7596e00640 1 -- 172.21.15.69:0/589946619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75900caed0 con 0x7f759015bf80 2024-04-24T08:13:48.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.152+0000 7f7596e00640 1 -- 172.21.15.69:0/589946619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7590143230 con 0x7f759015bf80 2024-04-24T08:13:48.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.153+0000 7f7596e00640 1 -- 172.21.15.69:0/589946619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7590075d50 con 0x7f759015bf80 2024-04-24T08:13:48.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.154+0000 7f758d600640 1 -- 172.21.15.69:0/589946619 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7588007bf0 con 0x7f759015bf80 2024-04-24T08:13:48.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.155+0000 7f758d600640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f757006c8f0 0x7f757006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.155+0000 7f758d600640 1 -- 172.21.15.69:0/589946619 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f75880acb40 con 0x7f759015bf80 2024-04-24T08:13:48.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.156+0000 7f7595a00640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f757006c8f0 0x7f757006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.157+0000 7f7595a00640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f757006c8f0 0x7f757006edb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7584007fd0 tx=0x7f7584009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.159+0000 7f758d600640 1 -- 172.21.15.69:0/589946619 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7588045360 con 0x7f759015bf80 2024-04-24T08:13:48.277 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:13:48.275+0000 7fa5d8600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-24T08:13:48.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.365+0000 7f7596e00640 1 -- 172.21.15.69:0/589946619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75900ae600 con 0x7f759015bf80 2024-04-24T08:13:48.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.367+0000 7f758d600640 1 -- 172.21.15.69:0/589946619 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) ==== 74+0+16775 (secure 0 0 0) 0x7f758807af10 con 0x7f759015bf80 2024-04-24T08:13:48.369 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:13:48.369 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":69,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:13:37.186047+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6837/1135108804":"2024-04-25T08:13:37.186005+0000","172.21.15.121:6835/1135108804":"2024-04-25T08:13:37.186005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 -- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f757006c8f0 msgr2=0x7f757006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f757006c8f0 0x7f757006edb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7584007fd0 tx=0x7f7584009040 comp rx=0 tx=0).stop 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 -- 172.21.15.69:0/589946619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759015bf80 msgr2=0x7f759015f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759015bf80 0x7f759015f050 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f7588034040 tx=0x7f7588004a40 comp rx=0 tx=0).stop 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 -- 172.21.15.69:0/589946619 shutdown_connections 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f757006c8f0 0x7f757006edb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f759015bf80 0x7f759015f050 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7590156c60 0x7f759015eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 --2- 172.21.15.69:0/589946619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75900b30f0 0x7f7590155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 -- 172.21.15.69:0/589946619 >> 172.21.15.69:0/589946619 conn(0x7f75900d0160 msgr2=0x7f759006ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.369+0000 7f7596e00640 1 -- 172.21.15.69:0/589946619 shutdown_connections 2024-04-24T08:13:48.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.370+0000 7f7596e00640 1 -- 172.21.15.69:0/589946619 wait complete. 2024-04-24T08:13:48.392 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:13:13.250209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:13:14.468238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:13:48.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:13:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.597+0000 7fcd08c00640 1 Processor -- start 2024-04-24T08:13:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.597+0000 7fcd08c00640 1 -- start start 2024-04-24T08:13:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.599+0000 7fcd08c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd040b3e90 0x7fcd040b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.599+0000 7fcd08c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd0415cc00 con 0x7fcd040ce250 2024-04-24T08:13:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.599+0000 7fcd08c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd0406ed30 con 0x7fcd0415c820 2024-04-24T08:13:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.599+0000 7fcd08c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd040b4be0 con 0x7fcd040b3e90 2024-04-24T08:13:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.599+0000 7fcd02800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcd040ce250 0x7fcd040ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33662/0 (socket says 172.21.15.69:33662) 2024-04-24T08:13:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.599+0000 7fcd02800640 1 -- 172.21.15.69:0/1812014111 learned_addr learned my addr 172.21.15.69:0/1812014111 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd01400640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd040b3e90 0x7fcd040b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1322356092 0 0) 0x7fcd0415cc00 con 0x7fcd040ce250 2024-04-24T08:13:48.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcce8003520 con 0x7fcd040ce250 2024-04-24T08:13:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3536899152 0 0) 0x7fcce8003520 con 0x7fcd040ce250 2024-04-24T08:13:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd040b3e90 msgr2=0x7fcd040b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd040b3e90 0x7fcd040b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 >> v1:172.21.15.121:6789/0 conn(0x7fcd0415c820 legacy=0x7fcd0406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd040b4e10 con 0x7fcd040ce250 2024-04-24T08:13:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fccfc0019f0 con 0x7fcd040ce250 2024-04-24T08:13:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 >> v1:172.21.15.69:6789/0 conn(0x7fcd040ce250 legacy=0x7fcd040ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcce80039e0 0x7fcce8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcce8004800 0x7fcce8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.600+0000 7fcd00a00640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd040b3e90 0x7fcce80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcce800ba70 con 0x7fcce8004800 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcce800bc20 con 0x7fcce80039e0 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcce800be20 con 0x7fcd040b3e90 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd01400640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcce8004800 0x7fcce8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd01e00640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcce80039e0 0x7fcce8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd02800640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd040b3e90 0x7fcce80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd01400640 1 -- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd040b3e90 msgr2=0x7fcce80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd01400640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd040b3e90 0x7fcce80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd01400640 1 -- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcce80039e0 msgr2=0x7fcce8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd01400640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcce80039e0 0x7fcce8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd01400640 1 -- 172.21.15.69:0/1812014111 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce8003520 con 0x7fcce8004800 2024-04-24T08:13:48.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.601+0000 7fcd01e00640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcce80039e0 0x7fcce8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:48.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.602+0000 7fcd01400640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcce8004800 0x7fcce8004c80 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009a90 tx=0x7fcce8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7b9fc6d281d51a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.603+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4036070 con 0x7fcce8004800 2024-04-24T08:13:48.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.603+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf4031400 con 0x7fcce8004800 2024-04-24T08:13:48.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.603+0000 7fcd00a00640 1 -- 172.21.15.69:0/1812014111 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4009690 con 0x7fcce8004800 2024-04-24T08:13:48.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.603+0000 7fcd08c00640 1 -- 172.21.15.69:0/1812014111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcce8004800 msgr2=0x7fcce8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.603+0000 7fcd08c00640 1 --2- 172.21.15.69:0/1812014111 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcce8004800 0x7fcce8004c80 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fccf4009a90 tx=0x7fcce8005830 comp rx=0 tx=0).stop 2024-04-24T08:13:48.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.603+0000 7fcd02800640 1 -- 172.21.15.69:0/1812014111 reap_dead start 2024-04-24T08:13:48.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.603+0000 7fcd08c00640 1 -- 172.21.15.69:0/1812014111 shutdown_connections 2024-04-24T08:13:48.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.603+0000 7fcd08c00640 1 -- 172.21.15.69:0/1812014111 >> 172.21.15.69:0/1812014111 conn(0x7fcd040d0350 msgr2=0x7fcd040d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.603+0000 7fcd08c00640 1 -- 172.21.15.69:0/1812014111 shutdown_connections 2024-04-24T08:13:48.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.603+0000 7fcd08c00640 1 -- 172.21.15.69:0/1812014111 wait complete. 2024-04-24T08:13:48.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.605+0000 7fcd08c00640 1 Processor -- start 2024-04-24T08:13:48.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.605+0000 7fcd08c00640 1 -- start start 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.606+0000 7fcd08c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcce8004800 0x7fcd0415ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.606+0000 7fcd08c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd040af960 0x7fcd040ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.606+0000 7fcd08c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0415ce00 0x7fcd040994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.606+0000 7fcd08c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0406efd0 con 0x7fcd040af960 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.606+0000 7fcd08c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcd040b42f0 con 0x7fcd0415ce00 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.606+0000 7fcd08c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd0406f2b0 con 0x7fcce8004800 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.606+0000 7fcd01400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd040af960 0x7fcd040ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.607+0000 7fcd01400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd040af960 0x7fcd040ae500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60896/0 (socket says 172.21.15.69:60896) 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.607+0000 7fcd01400640 1 -- 172.21.15.69:0/3024722950 learned_addr learned my addr 172.21.15.69:0/3024722950 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.607+0000 7fcd01e00640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcce8004800 0x7fcd0415ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.607+0000 7fcd01400640 1 -- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcce8004800 msgr2=0x7fcd0415ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.607+0000 7fcd01400640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcce8004800 0x7fcd0415ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.607+0000 7fcd01400640 1 -- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0415ce00 msgr2=0x7fcd040994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.607+0000 7fcd02800640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0415ce00 0x7fcd040994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.607+0000 7fcd01400640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0415ce00 0x7fcd040994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.607+0000 7fcd01400640 1 -- 172.21.15.69:0/3024722950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0415d290 con 0x7fcd040af960 2024-04-24T08:13:48.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.607+0000 7fcd01400640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd040af960 0x7fcd040ae500 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fccf4034040 tx=0x7fccf4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.608+0000 7fccfa000640 1 -- 172.21.15.69:0/3024722950 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4040020 con 0x7fcd040af960 2024-04-24T08:13:48.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.608+0000 7fccfa000640 1 -- 172.21.15.69:0/3024722950 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf4003940 con 0x7fcd040af960 2024-04-24T08:13:48.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.608+0000 7fccfa000640 1 -- 172.21.15.69:0/3024722950 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4036040 con 0x7fcd040af960 2024-04-24T08:13:48.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.608+0000 7fcd08c00640 1 -- 172.21.15.69:0/3024722950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd041583b0 con 0x7fcd040af960 2024-04-24T08:13:48.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.608+0000 7fcd08c00640 1 -- 172.21.15.69:0/3024722950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd04158c50 con 0x7fcd040af960 2024-04-24T08:13:48.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.609+0000 7fcd08c00640 1 -- 172.21.15.69:0/3024722950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd04158580 con 0x7fcd040af960 2024-04-24T08:13:48.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.610+0000 7fccfa000640 1 -- 172.21.15.69:0/3024722950 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fccf4043020 con 0x7fcd040af960 2024-04-24T08:13:48.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.611+0000 7fccfa000640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcce006c8f0 0x7fcce006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.611+0000 7fccfa000640 1 -- 172.21.15.69:0/3024722950 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7fccf40abea0 con 0x7fcd040af960 2024-04-24T08:13:48.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.612+0000 7fcd01e00640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcce006c8f0 0x7fcce006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.613+0000 7fcd01e00640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcce006c8f0 0x7fcce006edb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcd041f38a0 tx=0x7fccec009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.615+0000 7fccfa000640 1 -- 172.21.15.69:0/3024722950 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fccf404a070 con 0x7fcd040af960 2024-04-24T08:13:48.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.816+0000 7fcd08c00640 1 -- 172.21.15.69:0/3024722950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcd04068a80 con 0x7fcd040af960 2024-04-24T08:13:48.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.817+0000 7fccfa000640 1 -- 172.21.15.69:0/3024722950 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) ==== 93+0+10 (secure 0 0 0) 0x7fccf407a1d0 con 0x7fcd040af960 2024-04-24T08:13:48.819 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:13:48.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 -- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcce006c8f0 msgr2=0x7fcce006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcce006c8f0 0x7fcce006edb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fcd041f38a0 tx=0x7fccec009040 comp rx=0 tx=0).stop 2024-04-24T08:13:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 -- 172.21.15.69:0/3024722950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd040af960 msgr2=0x7fcd040ae500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd040af960 0x7fcd040ae500 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fccf4034040 tx=0x7fccf4004730 comp rx=0 tx=0).stop 2024-04-24T08:13:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 -- 172.21.15.69:0/3024722950 shutdown_connections 2024-04-24T08:13:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcce006c8f0 0x7fcce006edb0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd040af960 0x7fcd040ae500 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd0415ce00 0x7fcd040994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 --2- 172.21.15.69:0/3024722950 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcce8004800 0x7fcd0415ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 -- 172.21.15.69:0/3024722950 >> 172.21.15.69:0/3024722950 conn(0x7fcd040d0350 msgr2=0x7fcd0406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 -- 172.21.15.69:0/3024722950 shutdown_connections 2024-04-24T08:13:48.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:48.820+0000 7fcd08c00640 1 -- 172.21.15.69:0/3024722950 wait complete. 2024-04-24T08:13:48.841 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:13:49.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.047+0000 7f7d16c00640 1 Processor -- start 2024-04-24T08:13:49.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.047+0000 7f7d16c00640 1 -- start start 2024-04-24T08:13:49.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d16c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d1015e980 0x7f7d10162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d16c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d100d0da0 0x7f7d100cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d16c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1015ed60 con 0x7f7d100d0190 2024-04-24T08:13:49.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d16c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7d100d1580 con 0x7f7d1015e980 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d16c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d100d1860 con 0x7f7d100d0da0 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d15800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d1015e980 0x7f7d10162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d15800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d1015e980 0x7f7d10162f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54074/0 (socket says 172.21.15.69:54074) 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d16200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7d100d0190 0x7f7d1006cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59706/0 (socket says 172.21.15.69:59706) 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d15800640 1 -- 172.21.15.69:0/2803774309 learned_addr learned my addr 172.21.15.69:0/2803774309 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d14e00640 1 --2- 172.21.15.69:0/2803774309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d100d0da0 0x7f7d100cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d15800640 1 -- 172.21.15.69:0/2803774309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d100d0da0 msgr2=0x7f7d100cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d15800640 1 --2- 172.21.15.69:0/2803774309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d100d0da0 0x7f7d100cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d15800640 1 -- 172.21.15.69:0/2803774309 >> v1:172.21.15.69:6789/0 conn(0x7f7d100d0190 legacy=0x7f7d1006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d0fe00640 1 -- 172.21.15.69:0/2803774309 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1118276024 0 0) 0x7f7d1015ed60 con 0x7f7d100d0190 2024-04-24T08:13:49.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.049+0000 7f7d15800640 1 -- 172.21.15.69:0/2803774309 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d100cf450 con 0x7f7d1015e980 2024-04-24T08:13:49.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.050+0000 7f7d15800640 1 --2- 172.21.15.69:0/2803774309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d1015e980 0x7f7d10162f90 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f7d000098a0 tx=0x7f7d0002a320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b6e8a653ee88bc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.050+0000 7f7d0fe00640 1 -- 172.21.15.69:0/2803774309 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00037070 con 0x7f7d1015e980 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.050+0000 7f7d0fe00640 1 -- 172.21.15.69:0/2803774309 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d00032640 con 0x7f7d1015e980 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.050+0000 7f7d0fe00640 1 -- 172.21.15.69:0/2803774309 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00032a10 con 0x7f7d1015e980 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.051+0000 7f7d16c00640 1 -- 172.21.15.69:0/2803774309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d1015e980 msgr2=0x7f7d10162f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.051+0000 7f7d16c00640 1 --2- 172.21.15.69:0/2803774309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d1015e980 0x7f7d10162f90 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f7d000098a0 tx=0x7f7d0002a320 comp rx=0 tx=0).stop 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.051+0000 7f7d16c00640 1 -- 172.21.15.69:0/2803774309 shutdown_connections 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.051+0000 7f7d16c00640 1 --2- 172.21.15.69:0/2803774309 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d100d0da0 0x7f7d100cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.051+0000 7f7d16c00640 1 --2- 172.21.15.69:0/2803774309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d1015e980 0x7f7d10162f90 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.051+0000 7f7d16c00640 1 -- 172.21.15.69:0/2803774309 >> 172.21.15.69:0/2803774309 conn(0x7f7d1009d920 msgr2=0x7f7d1008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.051+0000 7f7d16c00640 1 -- 172.21.15.69:0/2803774309 shutdown_connections 2024-04-24T08:13:49.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.051+0000 7f7d16c00640 1 -- 172.21.15.69:0/2803774309 wait complete. 2024-04-24T08:13:49.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.053+0000 7f7d16c00640 1 Processor -- start 2024-04-24T08:13:49.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.053+0000 7f7d16c00640 1 -- start start 2024-04-24T08:13:49.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d16c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d100d0190 0x7f7d101525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d16c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d100d0da0 0x7f7d100ce020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d16c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d1015e980 0x7f7d100c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d16c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d100d1360 con 0x7f7d100d0190 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d16c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7d100d11e0 con 0x7f7d100d0da0 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d16c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d100cf070 con 0x7f7d1015e980 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d15800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d100d0190 0x7f7d101525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d15800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d100d0190 0x7f7d101525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54090/0 (socket says 172.21.15.69:54090) 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d15800640 1 -- 172.21.15.69:0/1150250524 learned_addr learned my addr 172.21.15.69:0/1150250524 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d15800640 1 -- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d1015e980 msgr2=0x7f7d100c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d15800640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d1015e980 0x7f7d100c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d15800640 1 -- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d100d0da0 msgr2=0x7f7d100ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d14e00640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d100d0da0 0x7f7d100ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d15800640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d100d0da0 0x7f7d100ce020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.054+0000 7f7d15800640 1 -- 172.21.15.69:0/1150250524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d100ce560 con 0x7f7d100d0190 2024-04-24T08:13:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.055+0000 7f7d15800640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d100d0190 0x7f7d101525c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7d00000c00 tx=0x7f7d000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.055+0000 7f7d0d600640 1 -- 172.21.15.69:0/1150250524 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00037040 con 0x7f7d100d0190 2024-04-24T08:13:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.055+0000 7f7d0d600640 1 -- 172.21.15.69:0/1150250524 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d00042070 con 0x7f7d100d0190 2024-04-24T08:13:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.055+0000 7f7d0d600640 1 -- 172.21.15.69:0/1150250524 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d000309b0 con 0x7f7d100d0190 2024-04-24T08:13:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.055+0000 7f7d16c00640 1 -- 172.21.15.69:0/1150250524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d100c8b90 con 0x7f7d100d0190 2024-04-24T08:13:49.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.055+0000 7f7d16c00640 1 -- 172.21.15.69:0/1150250524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d100ccf80 con 0x7f7d100d0190 2024-04-24T08:13:49.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.056+0000 7f7d16c00640 1 -- 172.21.15.69:0/1150250524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d10075d50 con 0x7f7d100d0190 2024-04-24T08:13:49.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.057+0000 7f7d0d600640 1 -- 172.21.15.69:0/1150250524 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d00030b50 con 0x7f7d100d0190 2024-04-24T08:13:49.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.059+0000 7f7d0d600640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7cf006c8f0 0x7f7cf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.059+0000 7f7d0d600640 1 -- 172.21.15.69:0/1150250524 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f7d000ac4c0 con 0x7f7d100d0190 2024-04-24T08:13:49.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.059+0000 7f7d14e00640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7cf006c8f0 0x7f7cf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.060+0000 7f7d14e00640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7cf006c8f0 0x7f7cf006edb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7d10163250 tx=0x7f7cfc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.064+0000 7f7d0d600640 1 -- 172.21.15.69:0/1150250524 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d00045360 con 0x7f7d100d0190 2024-04-24T08:13:49.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.254+0000 7f7d16c00640 1 -- 172.21.15.69:0/1150250524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7d100d1e40 con 0x7f7d100d0190 2024-04-24T08:13:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.254+0000 7f7d0d600640 1 -- 172.21.15.69:0/1150250524 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) ==== 104+0+11 (secure 0 0 0) 0x7f7d0007a890 con 0x7f7d100d0190 2024-04-24T08:13:49.256 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:13:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 -- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7cf006c8f0 msgr2=0x7f7cf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7cf006c8f0 0x7f7cf006edb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7d10163250 tx=0x7f7cfc008040 comp rx=0 tx=0).stop 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 -- 172.21.15.69:0/1150250524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d100d0190 msgr2=0x7f7d101525c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d100d0190 0x7f7d101525c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f7d00000c00 tx=0x7f7d000046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 -- 172.21.15.69:0/1150250524 shutdown_connections 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7cf006c8f0 0x7f7cf006edb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d1015e980 0x7f7d100c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d100d0da0 0x7f7d100ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 --2- 172.21.15.69:0/1150250524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d100d0190 0x7f7d101525c0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 -- 172.21.15.69:0/1150250524 >> 172.21.15.69:0/1150250524 conn(0x7f7d1009d920 msgr2=0x7f7d1008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 -- 172.21.15.69:0/1150250524 shutdown_connections 2024-04-24T08:13:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.257+0000 7f7d16c00640 1 -- 172.21.15.69:0/1150250524 wait complete. 2024-04-24T08:13:49.276 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:13:49.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.474+0000 7fb904e00640 1 Processor -- start 2024-04-24T08:13:49.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.474+0000 7fb904e00640 1 -- start start 2024-04-24T08:13:49.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.476+0000 7fb904e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb9000b3e90 0x7fb9000b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.476+0000 7fb904e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000ce250 0x7fb9000ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.476+0000 7fb904e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb90015cc00 con 0x7fb90015c820 2024-04-24T08:13:49.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.476+0000 7fb904e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb9000b4be0 con 0x7fb9000b3e90 2024-04-24T08:13:49.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.476+0000 7fb904e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb90006ed30 con 0x7fb9000ce250 2024-04-24T08:13:49.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.476+0000 7fb8f5400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb9000b3e90 0x7fb9000b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.476+0000 7fb8fde00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb90015c820 0x7fb90006f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59718/0 (socket says 172.21.15.69:59718) 2024-04-24T08:13:49.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.476+0000 7fb8fde00640 1 -- 172.21.15.69:0/2759193397 learned_addr learned my addr 172.21.15.69:0/2759193397 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:49.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.476+0000 7fb8f5400640 1 -- 172.21.15.69:0/2759193397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000ce250 msgr2=0x7fb9000ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.476+0000 7fb8f5400640 1 --2- 172.21.15.69:0/2759193397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000ce250 0x7fb9000ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.477+0000 7fb8f5400640 1 -- 172.21.15.69:0/2759193397 >> v1:172.21.15.69:6789/0 conn(0x7fb90015c820 legacy=0x7fb90006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.477+0000 7fb8f5400640 1 -- 172.21.15.69:0/2759193397 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9000b4e10 con 0x7fb9000b3e90 2024-04-24T08:13:49.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.478+0000 7fb8f5400640 1 --2- 172.21.15.69:0/2759193397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000b3e90 0x7fb9000b46a0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fb8e80097a0 tx=0x7fb8e802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a502b25e14d75f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.478+0000 7fb8fd400640 1 -- 172.21.15.69:0/2759193397 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8037070 con 0x7fb9000b3e90 2024-04-24T08:13:49.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.478+0000 7fb8fd400640 1 -- 172.21.15.69:0/2759193397 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8e8032440 con 0x7fb9000b3e90 2024-04-24T08:13:49.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.478+0000 7fb8fd400640 1 -- 172.21.15.69:0/2759193397 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8032810 con 0x7fb9000b3e90 2024-04-24T08:13:49.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.478+0000 7fb904e00640 1 -- 172.21.15.69:0/2759193397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000b3e90 msgr2=0x7fb9000b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.478+0000 7fb904e00640 1 --2- 172.21.15.69:0/2759193397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000b3e90 0x7fb9000b46a0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fb8e80097a0 tx=0x7fb8e802a180 comp rx=0 tx=0).stop 2024-04-24T08:13:49.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.478+0000 7fb904e00640 1 -- 172.21.15.69:0/2759193397 shutdown_connections 2024-04-24T08:13:49.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.478+0000 7fb904e00640 1 --2- 172.21.15.69:0/2759193397 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000ce250 0x7fb9000ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.478+0000 7fb904e00640 1 --2- 172.21.15.69:0/2759193397 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000b3e90 0x7fb9000b46a0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.478+0000 7fb904e00640 1 -- 172.21.15.69:0/2759193397 >> 172.21.15.69:0/2759193397 conn(0x7fb9000d0350 msgr2=0x7fb9000d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.479+0000 7fb904e00640 1 -- 172.21.15.69:0/2759193397 shutdown_connections 2024-04-24T08:13:49.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.479+0000 7fb904e00640 1 -- 172.21.15.69:0/2759193397 wait complete. 2024-04-24T08:13:49.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.480+0000 7fb904e00640 1 Processor -- start 2024-04-24T08:13:49.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.480+0000 7fb904e00640 1 -- start start 2024-04-24T08:13:49.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.481+0000 7fb904e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000b3e90 0x7fb900073670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.481+0000 7fb904e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000ce250 0x7fb9001f8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.481+0000 7fb904e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90015c820 0x7fb9000ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.481+0000 7fb904e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb90006f2b0 con 0x7fb9000ce250 2024-04-24T08:13:49.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.481+0000 7fb904e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb90006efd0 con 0x7fb90015c820 2024-04-24T08:13:49.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.481+0000 7fb904e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb90006fee0 con 0x7fb9000b3e90 2024-04-24T08:13:49.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.481+0000 7fb8f5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000ce250 0x7fb9001f8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.482+0000 7fb8f5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000ce250 0x7fb9001f8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54108/0 (socket says 172.21.15.69:54108) 2024-04-24T08:13:49.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.482+0000 7fb8f5400640 1 -- 172.21.15.69:0/1013133953 learned_addr learned my addr 172.21.15.69:0/1013133953 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:49.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.482+0000 7fb8fde00640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000b3e90 0x7fb900073670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.482+0000 7fb8fe800640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90015c820 0x7fb9000ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.482+0000 7fb8f5400640 1 -- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000b3e90 msgr2=0x7fb900073670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.482+0000 7fb8f5400640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000b3e90 0x7fb900073670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.482+0000 7fb8f5400640 1 -- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90015c820 msgr2=0x7fb9000ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.482+0000 7fb8f5400640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90015c820 0x7fb9000ca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.482+0000 7fb8f5400640 1 -- 172.21.15.69:0/1013133953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb900073d00 con 0x7fb9000ce250 2024-04-24T08:13:49.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.482+0000 7fb8f5400640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000ce250 0x7fb9001f8be0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8000c00 tx=0x7fb8e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.483+0000 7fb8f6a00640 1 -- 172.21.15.69:0/1013133953 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8037040 con 0x7fb9000ce250 2024-04-24T08:13:49.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.483+0000 7fb8f6a00640 1 -- 172.21.15.69:0/1013133953 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8e8042070 con 0x7fb9000ce250 2024-04-24T08:13:49.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.483+0000 7fb8f6a00640 1 -- 172.21.15.69:0/1013133953 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8030a70 con 0x7fb9000ce250 2024-04-24T08:13:49.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.483+0000 7fb904e00640 1 -- 172.21.15.69:0/1013133953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9000cb300 con 0x7fb9000ce250 2024-04-24T08:13:49.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.483+0000 7fb904e00640 1 -- 172.21.15.69:0/1013133953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9000cb8c0 con 0x7fb9000ce250 2024-04-24T08:13:49.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.484+0000 7fb904e00640 1 -- 172.21.15.69:0/1013133953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9000cbbf0 con 0x7fb9000ce250 2024-04-24T08:13:49.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.485+0000 7fb8f6a00640 1 -- 172.21.15.69:0/1013133953 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb8e8030c10 con 0x7fb9000ce250 2024-04-24T08:13:49.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.486+0000 7fb8f6a00640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8dc06c8f0 0x7fb8dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.486+0000 7fb8f6a00640 1 -- 172.21.15.69:0/1013133953 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7fb8e80ac680 con 0x7fb9000ce250 2024-04-24T08:13:49.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.486+0000 7fb8fde00640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8dc06c8f0 0x7fb8dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.488+0000 7fb8fde00640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8dc06c8f0 0x7fb8dc06edb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0000f60 tx=0x7fb8f0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.491+0000 7fb8f6a00640 1 -- 172.21.15.69:0/1013133953 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb8e8045360 con 0x7fb9000ce250 2024-04-24T08:13:49.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.692+0000 7fb904e00640 1 -- 172.21.15.69:0/1013133953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb900068a80 con 0x7fb9000ce250 2024-04-24T08:13:49.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.693+0000 7fb8f6a00640 1 -- 172.21.15.69:0/1013133953 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) ==== 100+0+11 (secure 0 0 0) 0x7fb8e807aa40 con 0x7fb9000ce250 2024-04-24T08:13:49.694 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 -- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8dc06c8f0 msgr2=0x7fb8dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8dc06c8f0 0x7fb8dc06edb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0000f60 tx=0x7fb8f0009040 comp rx=0 tx=0).stop 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 -- 172.21.15.69:0/1013133953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000ce250 msgr2=0x7fb9001f8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000ce250 0x7fb9001f8be0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8000c00 tx=0x7fb8e80046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 -- 172.21.15.69:0/1013133953 shutdown_connections 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8dc06c8f0 0x7fb8dc06edb0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb9000ce250 0x7fb9001f8be0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb90015c820 0x7fb9000ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 --2- 172.21.15.69:0/1013133953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb9000b3e90 0x7fb900073670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 -- 172.21.15.69:0/1013133953 >> 172.21.15.69:0/1013133953 conn(0x7fb9000d0350 msgr2=0x7fb9000cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 -- 172.21.15.69:0/1013133953 shutdown_connections 2024-04-24T08:13:49.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.695+0000 7fb904e00640 1 -- 172.21.15.69:0/1013133953 wait complete. 2024-04-24T08:13:49.714 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:13:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.908+0000 7f1ec5000640 1 Processor -- start 2024-04-24T08:13:49.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.909+0000 7f1ec5000640 1 -- start start 2024-04-24T08:13:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.910+0000 7f1ec5000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1ec015d1d0 0x7f1ec00ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.910+0000 7f1ec5000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec015d660 con 0x7f1ec00b30f0 2024-04-24T08:13:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.910+0000 7f1ec5000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ec006dc30 con 0x7f1ec0156c60 2024-04-24T08:13:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.910+0000 7f1ec5000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1ec006df10 con 0x7f1ec015d1d0 2024-04-24T08:13:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.910+0000 7f1ebea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1ec015d1d0 0x7f1ec00ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.911+0000 7f1ebea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1ec015d1d0 0x7f1ec00ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54116/0 (socket says 172.21.15.69:54116) 2024-04-24T08:13:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.911+0000 7f1ebea00640 1 -- 172.21.15.69:0/1725281686 learned_addr learned my addr 172.21.15.69:0/1725281686 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.911+0000 7f1ebfe00640 1 --1- 172.21.15.69:0/1725281686 >> v1:172.21.15.69:6789/0 conn(0x7f1ec00b30f0 0x7f1ec006e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59728/0 (socket says 172.21.15.69:59728) 2024-04-24T08:13:49.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.911+0000 7f1ebea00640 1 -- 172.21.15.69:0/1725281686 >> v1:172.21.15.121:6789/0 conn(0x7f1ec0156c60 legacy=0x7f1ec015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.911+0000 7f1ebea00640 1 -- 172.21.15.69:0/1725281686 >> v1:172.21.15.69:6789/0 conn(0x7f1ec00b30f0 legacy=0x7f1ec006e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.911+0000 7f1ebea00640 1 -- 172.21.15.69:0/1725281686 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec006e1f0 con 0x7f1ec015d1d0 2024-04-24T08:13:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.912+0000 7f1ebea00640 1 --2- 172.21.15.69:0/1725281686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec015d1d0 0x7f1ec00ad0d0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009e50 tx=0x7f1ea802a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c294f1b0f6bec83f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.912+0000 7f1ebe000640 1 -- 172.21.15.69:0/1725281686 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8036070 con 0x7f1ec015d1d0 2024-04-24T08:13:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.912+0000 7f1ebe000640 1 -- 172.21.15.69:0/1725281686 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea8031600 con 0x7f1ec015d1d0 2024-04-24T08:13:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.912+0000 7f1ebe000640 1 -- 172.21.15.69:0/1725281686 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea80319d0 con 0x7f1ec015d1d0 2024-04-24T08:13:49.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.913+0000 7f1ec5000640 1 -- 172.21.15.69:0/1725281686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec015d1d0 msgr2=0x7f1ec00ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.913+0000 7f1ec5000640 1 --2- 172.21.15.69:0/1725281686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec015d1d0 0x7f1ec00ad0d0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009e50 tx=0x7f1ea802a760 comp rx=0 tx=0).stop 2024-04-24T08:13:49.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.913+0000 7f1ec5000640 1 -- 172.21.15.69:0/1725281686 shutdown_connections 2024-04-24T08:13:49.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.913+0000 7f1ec5000640 1 --2- 172.21.15.69:0/1725281686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec015d1d0 0x7f1ec00ad0d0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.913+0000 7f1ec5000640 1 -- 172.21.15.69:0/1725281686 >> 172.21.15.69:0/1725281686 conn(0x7f1ec00d0160 msgr2=0x7f1ec00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.913+0000 7f1ec5000640 1 -- 172.21.15.69:0/1725281686 shutdown_connections 2024-04-24T08:13:49.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.913+0000 7f1ec5000640 1 -- 172.21.15.69:0/1725281686 wait complete. 2024-04-24T08:13:49.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.915+0000 7f1ec5000640 1 Processor -- start 2024-04-24T08:13:49.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.915+0000 7f1ec5000640 1 -- start start 2024-04-24T08:13:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.916+0000 7f1ec5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec00b30f0 0x7f1ec0155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.916+0000 7f1ec5000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ec0156c60 0x7f1ec015e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.916+0000 7f1ec5000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ec0157040 0x7f1ec01f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.916+0000 7f1ec5000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec00ad610 con 0x7f1ec00b30f0 2024-04-24T08:13:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.916+0000 7f1ec5000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ec015d940 con 0x7f1ec0156c60 2024-04-24T08:13:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.916+0000 7f1ec5000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1ec00ad790 con 0x7f1ec0157040 2024-04-24T08:13:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.916+0000 7f1ebf400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec00b30f0 0x7f1ec0155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.916+0000 7f1ebf400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec00b30f0 0x7f1ec0155de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54124/0 (socket says 172.21.15.69:54124) 2024-04-24T08:13:49.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.916+0000 7f1ebf400640 1 -- 172.21.15.69:0/3685102356 learned_addr learned my addr 172.21.15.69:0/3685102356 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1ebea00640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ec0156c60 0x7f1ec015e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1ebfe00640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ec0157040 0x7f1ec01f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1ebf400640 1 -- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ec0157040 msgr2=0x7f1ec01f43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1ebf400640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ec0157040 0x7f1ec01f43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1ebf400640 1 -- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ec0156c60 msgr2=0x7f1ec015e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1ebf400640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ec0156c60 0x7f1ec015e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1ebf400640 1 -- 172.21.15.69:0/3685102356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ec0156320 con 0x7f1ec00b30f0 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1ebea00640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ec0156c60 0x7f1ec015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1ebf400640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec00b30f0 0x7f1ec0155de0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000c460 tx=0x7f1eb000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1eb7400640 1 -- 172.21.15.69:0/3685102356 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0016020 con 0x7f1ec00b30f0 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1eb7400640 1 -- 172.21.15.69:0/3685102356 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eb0012870 con 0x7f1ec00b30f0 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1eb7400640 1 -- 172.21.15.69:0/3685102356 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0012c60 con 0x7f1ec00b30f0 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.917+0000 7f1ec5000640 1 -- 172.21.15.69:0/3685102356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ec01575b0 con 0x7f1ec00b30f0 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.918+0000 7f1ec5000640 1 -- 172.21.15.69:0/3685102356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ec0157b70 con 0x7f1ec00b30f0 2024-04-24T08:13:49.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.919+0000 7f1ec5000640 1 -- 172.21.15.69:0/3685102356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ec0075d50 con 0x7f1ec00b30f0 2024-04-24T08:13:49.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.919+0000 7f1eb7400640 1 -- 172.21.15.69:0/3685102356 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1eb0019020 con 0x7f1ec00b30f0 2024-04-24T08:13:49.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.920+0000 7f1eb7400640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1e9c06c8f0 0x7f1e9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.920+0000 7f1eb7400640 1 -- 172.21.15.69:0/3685102356 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f1eb008b5a0 con 0x7f1ec00b30f0 2024-04-24T08:13:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.921+0000 7f1ebea00640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1e9c06c8f0 0x7f1e9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.922+0000 7f1ebea00640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1e9c06c8f0 0x7f1e9c06edb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8033000 tx=0x7f1ea8009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:49.925+0000 7f1eb7400640 1 -- 172.21.15.69:0/3685102356 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1eb0020070 con 0x7f1ec00b30f0 2024-04-24T08:13:50.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.137+0000 7f1ec5000640 1 -- 172.21.15.69:0/3685102356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1ec015c440 con 0x7f1ec00b30f0 2024-04-24T08:13:50.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.138+0000 7f1eb7400640 1 -- 172.21.15.69:0/3685102356 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v69) ==== 80+0+124 (secure 0 0 0) 0x7f1eb0059a00 con 0x7f1ec00b30f0 2024-04-24T08:13:50.138 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6837/1135108804 2024-04-25T08:13:37.186005+0000 2024-04-24T08:13:50.138 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6835/1135108804 2024-04-25T08:13:37.186005+0000 2024-04-24T08:13:50.138 INFO:teuthology.orchestra.run.smithi069.stderr:listed 2 entries 2024-04-24T08:13:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.140+0000 7f1ec5000640 1 -- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1e9c06c8f0 msgr2=0x7f1e9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.140+0000 7f1ec5000640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1e9c06c8f0 0x7f1e9c06edb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8033000 tx=0x7f1ea8009000 comp rx=0 tx=0).stop 2024-04-24T08:13:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.140+0000 7f1ec5000640 1 -- 172.21.15.69:0/3685102356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec00b30f0 msgr2=0x7f1ec0155de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.140+0000 7f1ec5000640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec00b30f0 0x7f1ec0155de0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1eb000c460 tx=0x7f1eb000c920 comp rx=0 tx=0).stop 2024-04-24T08:13:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.140+0000 7f1ec5000640 1 -- 172.21.15.69:0/3685102356 shutdown_connections 2024-04-24T08:13:50.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.140+0000 7f1ec5000640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1e9c06c8f0 0x7f1e9c06edb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.140+0000 7f1ec5000640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ec0157040 0x7f1ec01f43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.140+0000 7f1ec5000640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ec0156c60 0x7f1ec015e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.140+0000 7f1ec5000640 1 --2- 172.21.15.69:0/3685102356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ec00b30f0 0x7f1ec0155de0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.140+0000 7f1ec5000640 1 -- 172.21.15.69:0/3685102356 >> 172.21.15.69:0/3685102356 conn(0x7f1ec00d0160 msgr2=0x7f1ec00b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.141+0000 7f1ec5000640 1 -- 172.21.15.69:0/3685102356 shutdown_connections 2024-04-24T08:13:50.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:50.141+0000 7f1ec5000640 1 -- 172.21.15.69:0/3685102356 wait complete. 2024-04-24T08:13:50.159 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:13:50.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:50.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:13:50.236 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[58941]: fuse finished with error 0 and tester_r 0 2024-04-24T08:13:50.242 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:13:56.343 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:13:56.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:56.343 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:13:56.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:56.401 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub start /subdir recursive 2024-04-24T08:13:56.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.612+0000 7fbab3800640 1 Processor -- start 2024-04-24T08:13:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.612+0000 7fbab3800640 1 -- start start 2024-04-24T08:13:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.613+0000 7fbab3800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbaac0aeb00 0x7fbaac0d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.613+0000 7fbab3800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbaac15c3e0 0x7fbaac0d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.613+0000 7fbab3800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaac15ca40 con 0x7fbaac0b3fb0 2024-04-24T08:13:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.613+0000 7fbab3800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbaac0b31f0 con 0x7fbaac0aeb00 2024-04-24T08:13:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.613+0000 7fbab3800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbaac0b34d0 con 0x7fbaac15c3e0 2024-04-24T08:13:56.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.614+0000 7fbaa3e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbaac0b3fb0 0x7fbaac15bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59736/0 (socket says 172.21.15.69:59736) 2024-04-24T08:13:56.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.614+0000 7fbab0e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbaac0aeb00 0x7fbaac0d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.614+0000 7fbaa3e00640 1 -- 172.21.15.69:0/3578695706 learned_addr learned my addr 172.21.15.69:0/3578695706 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:56.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.614+0000 7fbab1800640 1 --2- 172.21.15.69:0/3578695706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbaac15c3e0 0x7fbaac0d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.614+0000 7fbab0e00640 1 -- 172.21.15.69:0/3578695706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbaac15c3e0 msgr2=0x7fbaac0d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.614+0000 7fbab0e00640 1 --2- 172.21.15.69:0/3578695706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbaac15c3e0 0x7fbaac0d0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.614+0000 7fbab0e00640 1 -- 172.21.15.69:0/3578695706 >> v1:172.21.15.69:6789/0 conn(0x7fbaac0b3fb0 legacy=0x7fbaac15bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.615+0000 7fbab0e00640 1 -- 172.21.15.69:0/3578695706 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaac0b37b0 con 0x7fbaac0aeb00 2024-04-24T08:13:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.615+0000 7fbaabe00640 1 -- 172.21.15.69:0/3578695706 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3598408098 0 0) 0x7fbaac15ca40 con 0x7fbaac0b3fb0 2024-04-24T08:13:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.615+0000 7fbab1800640 1 --2- 172.21.15.69:0/3578695706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbaac15c3e0 0x7fbaac0d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.616+0000 7fbab0e00640 1 --2- 172.21.15.69:0/3578695706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 0x7fbaac0d07c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0097a0 tx=0x7fba9c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c644f8829dcd7d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.616+0000 7fbaabe00640 1 -- 172.21.15.69:0/3578695706 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c037070 con 0x7fbaac0aeb00 2024-04-24T08:13:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.616+0000 7fbaabe00640 1 -- 172.21.15.69:0/3578695706 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9c032440 con 0x7fbaac0aeb00 2024-04-24T08:13:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.616+0000 7fbaabe00640 1 -- 172.21.15.69:0/3578695706 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c032810 con 0x7fbaac0aeb00 2024-04-24T08:13:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.617+0000 7fbab3800640 1 -- 172.21.15.69:0/3578695706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 msgr2=0x7fbaac0d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.617+0000 7fbab3800640 1 --2- 172.21.15.69:0/3578695706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 0x7fbaac0d07c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fba9c0097a0 tx=0x7fba9c02a180 comp rx=0 tx=0).stop 2024-04-24T08:13:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.617+0000 7fbab3800640 1 -- 172.21.15.69:0/3578695706 shutdown_connections 2024-04-24T08:13:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.617+0000 7fbab3800640 1 --2- 172.21.15.69:0/3578695706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbaac15c3e0 0x7fbaac0d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.617+0000 7fbab3800640 1 --2- 172.21.15.69:0/3578695706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 0x7fbaac0d07c0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.617+0000 7fbab3800640 1 -- 172.21.15.69:0/3578695706 >> 172.21.15.69:0/3578695706 conn(0x7fbaac06be40 msgr2=0x7fbaac06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.617+0000 7fbab3800640 1 -- 172.21.15.69:0/3578695706 shutdown_connections 2024-04-24T08:13:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.617+0000 7fbab3800640 1 -- 172.21.15.69:0/3578695706 wait complete. 2024-04-24T08:13:56.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.619+0000 7fbab3800640 1 Processor -- start 2024-04-24T08:13:56.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.619+0000 7fbab3800640 1 -- start start 2024-04-24T08:13:56.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.620+0000 7fbab3800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 0x7fbaac158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.620+0000 7fbab3800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbaac0b3fb0 0x7fbaac0c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.620+0000 7fbab3800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaac15c3e0 0x7fbaac0c5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.620+0000 7fbab3800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac0aeee0 con 0x7fbaac0aeb00 2024-04-24T08:13:56.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.620+0000 7fbab3800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac0ad1d0 con 0x7fbaac15c3e0 2024-04-24T08:13:56.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.620+0000 7fbab3800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbaac158b70 con 0x7fbaac0b3fb0 2024-04-24T08:13:56.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.620+0000 7fbab0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 0x7fbaac158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.621+0000 7fbab0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 0x7fbaac158630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54142/0 (socket says 172.21.15.69:54142) 2024-04-24T08:13:56.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.621+0000 7fbab0e00640 1 -- 172.21.15.69:0/1788647689 learned_addr learned my addr 172.21.15.69:0/1788647689 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:56.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.621+0000 7fbab0e00640 1 -- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbaac0b3fb0 msgr2=0x7fbaac0c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.621+0000 7fbab0e00640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbaac0b3fb0 0x7fbaac0c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.621+0000 7fbab0e00640 1 -- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaac15c3e0 msgr2=0x7fbaac0c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.621+0000 7fbab0e00640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaac15c3e0 0x7fbaac0c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.621+0000 7fbab0e00640 1 -- 172.21.15.69:0/1788647689 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaac15b0c0 con 0x7fbaac0aeb00 2024-04-24T08:13:56.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.621+0000 7fbab0e00640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 0x7fbaac158630 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fba9c000c00 tx=0x7fba9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.621+0000 7fbaa9600640 1 -- 172.21.15.69:0/1788647689 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c037040 con 0x7fbaac0aeb00 2024-04-24T08:13:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.622+0000 7fbaa9600640 1 -- 172.21.15.69:0/1788647689 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9c042070 con 0x7fbaac0aeb00 2024-04-24T08:13:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.622+0000 7fbaa9600640 1 -- 172.21.15.69:0/1788647689 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c030a70 con 0x7fbaac0aeb00 2024-04-24T08:13:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.622+0000 7fbab3800640 1 -- 172.21.15.69:0/1788647689 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaac159ce0 con 0x7fbaac0aeb00 2024-04-24T08:13:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.622+0000 7fbab3800640 1 -- 172.21.15.69:0/1788647689 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaac159200 con 0x7fbaac0aeb00 2024-04-24T08:13:56.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.624+0000 7fbaa9600640 1 -- 172.21.15.69:0/1788647689 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fba9c030c10 con 0x7fbaac0aeb00 2024-04-24T08:13:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.625+0000 7fbaa9600640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba8806c920 0x7fba8806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.625+0000 7fbaa9600640 1 -- 172.21.15.69:0/1788647689 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7fba9c0ac6e0 con 0x7fbaac0aeb00 2024-04-24T08:13:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.625+0000 7fbab67e4740 1 Processor -- start 2024-04-24T08:13:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.625+0000 7fbaa3e00640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba8806c920 0x7fba8806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.626+0000 7fbab67e4740 1 -- start start 2024-04-24T08:13:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.626+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c5c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c79ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c31d90 0x55b109c78010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab67e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b109b953a0 con 0x55b109c31d90 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab67e4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b109b3b240 con 0x55b109c5c0b0 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab67e4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55b109b91360 con 0x55b109c79950 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbaa3e00640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba8806c920 0x7fba8806ede0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fba94007fd0 tx=0x7fba94007920 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c31d90 0x55b109c78010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c31d90 0x55b109c78010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54148/0 (socket says 172.21.15.69:54148) 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab1800640 1 -- 172.21.15.69:0/3984596166 learned_addr learned my addr 172.21.15.69:0/3984596166 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab0e00640 1 --2- 172.21.15.69:0/3984596166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c5c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbaa3e00640 1 --2- 172.21.15.69:0/3984596166 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c79ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab1800640 1 -- 172.21.15.69:0/3984596166 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 msgr2=0x55b109c79ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab1800640 1 --2- 172.21.15.69:0/3984596166 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c79ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab1800640 1 -- 172.21.15.69:0/3984596166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 msgr2=0x55b109c5c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab1800640 1 --2- 172.21.15.69:0/3984596166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c5c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.627+0000 7fbab1800640 1 -- 172.21.15.69:0/3984596166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b109973a20 con 0x55b109c31d90 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.628+0000 7fbab1800640 1 --2- 172.21.15.69:0/3984596166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c31d90 0x55b109c78010 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4008e60 tx=0x7fbaa400d6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193267781384ac2f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.628+0000 7fbaa1600640 1 -- 172.21.15.69:0/3984596166 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa400f800 con 0x55b109c31d90 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.628+0000 7fbaa1600640 1 -- 172.21.15.69:0/3984596166 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaa400f9a0 con 0x55b109c31d90 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.628+0000 7fbaa1600640 1 -- 172.21.15.69:0/3984596166 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa400fd70 con 0x55b109c31d90 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.629+0000 7fbab67e4740 1 -- 172.21.15.69:0/3984596166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c31d90 msgr2=0x55b109c78010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.629+0000 7fbab67e4740 1 --2- 172.21.15.69:0/3984596166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c31d90 0x55b109c78010 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4008e60 tx=0x7fbaa400d6e0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.629+0000 7fbab67e4740 1 -- 172.21.15.69:0/3984596166 shutdown_connections 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.629+0000 7fbab67e4740 1 --2- 172.21.15.69:0/3984596166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c31d90 0x55b109c78010 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.629+0000 7fbab67e4740 1 --2- 172.21.15.69:0/3984596166 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c79ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.629+0000 7fbab67e4740 1 --2- 172.21.15.69:0/3984596166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c5c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.629+0000 7fbab67e4740 1 -- 172.21.15.69:0/3984596166 >> 172.21.15.69:0/3984596166 conn(0x55b109c2e6c0 msgr2=0x55b109c76ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.629+0000 7fbab67e4740 1 -- 172.21.15.69:0/3984596166 shutdown_connections 2024-04-24T08:13:56.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.629+0000 7fbab67e4740 1 -- 172.21.15.69:0/3984596166 wait complete. 2024-04-24T08:13:56.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.630+0000 7fbab67e4740 1 -- start start 2024-04-24T08:13:56.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.630+0000 7fbab67e4740 1 Processor -- start 2024-04-24T08:13:56.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.631+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c6e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.631+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c4d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.631+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c7a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.631+0000 7fbab67e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b109c32b30 con 0x55b109c5f9b0 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.631+0000 7fbab67e4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b109b4e7e0 con 0x55b109c5c0b0 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.631+0000 7fbab67e4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55b109c2c5f0 con 0x55b109c79950 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.631+0000 7fbab67e4740 10 client.0 _ll_register_callbacks cb 0x55b109b8bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.631+0000 7fbab1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c7a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.631+0000 7fbab1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c7a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54162/0 (socket says 172.21.15.69:54162) 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.631+0000 7fbab1800640 1 -- 172.21.15.69:0/1728945130 learned_addr learned my addr 172.21.15.69:0/1728945130 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fbab0e00640 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c6e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fbab1800640 1 -- 172.21.15.69:0/1728945130 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 msgr2=0x55b109c4d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fbaa3e00640 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c4d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fbab1800640 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c4d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fbab1800640 1 -- 172.21.15.69:0/1728945130 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 msgr2=0x55b109c6e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fbab1800640 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c6e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fbab1800640 1 -- 172.21.15.69:0/1728945130 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b109867310 con 0x55b109c5f9b0 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fbab0e00640 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c6e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fbab1800640 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c7a440 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4000f80 tx=0x7fbaa4005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd4d32cb398e676 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fba9a000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fba9a000640 1 -- 172.21.15.69:0/1728945130 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4028060 con 0x55b109c5f9b0 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fba9a000640 1 -- 172.21.15.69:0/1728945130 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7fbaa4018480 con 0x55b109c5f9b0 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.632+0000 7fbab67e4740 1 -- 172.21.15.69:0/1728945130 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbaa400f9a0 con 0x55b109c5f9b0 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fba9a000640 1 -- 172.21.15.69:0/1728945130 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fbaa4018680 con 0x55b109c5f9b0 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fbab67e4740 10 client.7209 fetch_fsmap learned FSMap version 51 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fbab67e4740 1 -- 172.21.15.69:0/1728945130 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55b1099d41b0 con 0x55b109c5f9b0 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fba9a000640 1 -- 172.21.15.69:0/1728945130 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaa400f9a0 con 0x55b109c5f9b0 2024-04-24T08:13:56.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fba9a000640 1 -- 172.21.15.69:0/1728945130 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7fbaa4018820 con 0x55b109c5f9b0 2024-04-24T08:13:56.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fbab67e4740 10 client.7209 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:13:56.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fbab67e4740 10 client.7209 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:13:56.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fbab67e4740 20 client.7209 populate_metadata read hostname 'smithi069' 2024-04-24T08:13:56.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fbab67e4740 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55b109c78310 0x55b109c78760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fbab67e4740 4 client.7209 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:56.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fbab67e4740 1 -- 172.21.15.69:0/1728945130 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b109ba2dc0 con 0x55b109c78310 2024-04-24T08:13:56.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.633+0000 7fbab0e00640 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55b109c78310 0x55b109c78760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.634+0000 7fbab0e00640 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55b109c78310 0x55b109c78760 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=213400ddb57a3de2 server_cookie=98de89aa90c87322 in_seq=0 out_seq=0 2024-04-24T08:13:56.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.634+0000 7fba9a000640 10 client.7209 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:56.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.634+0000 7fba9a000640 1 -- 172.21.15.69:0/1728945130 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b109ba2dc0 con 0x55b109c78310 2024-04-24T08:13:56.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.634+0000 7fba9a000640 10 client.7209 handle_command_reply: tid=0 2024-04-24T08:13:56.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 client.7209 shutdown 2024-04-24T08:13:56.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 -- 172.21.15.69:0/1728945130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 msgr2=0x55b109c7a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c7a440 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4000f80 tx=0x7fbaa4005d10 comp rx=0 tx=0).stop 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 -- 172.21.15.69:0/1728945130 shutdown_connections 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55b109c78310 0x55b109c78760 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c7a440 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c4d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 --2- 172.21.15.69:0/1728945130 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c6e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fba9a000640 0 client.7209 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 -- 172.21.15.69:0/1728945130 >> 172.21.15.69:0/1728945130 conn(0x55b109c31d90 msgr2=0x55b109c76760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 -- 172.21.15.69:0/1728945130 shutdown_connections 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 1 -- 172.21.15.69:0/1728945130 wait complete. 2024-04-24T08:13:56.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.635+0000 7fbab67e4740 20 client.7209 trim_cache size 0 max 16384 2024-04-24T08:13:56.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.654+0000 7fbab67e4740 1 Processor -- start 2024-04-24T08:13:56.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.655+0000 7fbab67e4740 1 -- start start 2024-04-24T08:13:56.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.655+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.655+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c5f9b0 0x55b109c5b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.655+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c79950 0x55b109c6df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.655+0000 7fbab67e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac200e70 con 0x55b109c5c0b0 2024-04-24T08:13:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.655+0000 7fbab67e4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac1ffec0 con 0x55b109c79950 2024-04-24T08:13:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.655+0000 7fbab67e4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbaac0c4b70 con 0x55b109c5f9b0 2024-04-24T08:13:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.656+0000 7fbab0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.656+0000 7fbab0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54170/0 (socket says 172.21.15.69:54170) 2024-04-24T08:13:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.656+0000 7fbab0e00640 1 -- 172.21.15.69:0/3896581442 learned_addr learned my addr 172.21.15.69:0/3896581442 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.656+0000 7fbab0e00640 1 -- 172.21.15.69:0/3896581442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c5f9b0 msgr2=0x55b109c5b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.656+0000 7fbab0e00640 1 --2- 172.21.15.69:0/3896581442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c5f9b0 0x55b109c5b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.656+0000 7fbab0e00640 1 -- 172.21.15.69:0/3896581442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c79950 msgr2=0x55b109c6df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.656+0000 7fbab0e00640 1 --2- 172.21.15.69:0/3896581442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c79950 0x55b109c6df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.656+0000 7fbab0e00640 1 -- 172.21.15.69:0/3896581442 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b109c711e0 con 0x55b109c5c0b0 2024-04-24T08:13:56.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.657+0000 7fbab0e00640 1 --2- 172.21.15.69:0/3896581442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fba9c07a010 tx=0x7fba9c0aa9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bda4e30431fd6ee server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.657+0000 7fbaa0c00640 1 -- 172.21.15.69:0/3896581442 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c0a9070 con 0x55b109c5c0b0 2024-04-24T08:13:56.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.657+0000 7fbaa0c00640 1 -- 172.21.15.69:0/3896581442 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9c07aa10 con 0x55b109c5c0b0 2024-04-24T08:13:56.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.657+0000 7fbaa0c00640 1 -- 172.21.15.69:0/3896581442 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c07a310 con 0x55b109c5c0b0 2024-04-24T08:13:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.657+0000 7fbab67e4740 1 -- 172.21.15.69:0/3896581442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5c0b0 msgr2=0x55b109c5b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.657+0000 7fbab67e4740 1 --2- 172.21.15.69:0/3896581442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fba9c07a010 tx=0x7fba9c0aa9e0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.658+0000 7fbab67e4740 1 -- 172.21.15.69:0/3896581442 shutdown_connections 2024-04-24T08:13:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.658+0000 7fbab67e4740 1 --2- 172.21.15.69:0/3896581442 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c79950 0x55b109c6df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.658+0000 7fbab67e4740 1 --2- 172.21.15.69:0/3896581442 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c5f9b0 0x55b109c5b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.658+0000 7fbab67e4740 1 --2- 172.21.15.69:0/3896581442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.658+0000 7fbab67e4740 1 -- 172.21.15.69:0/3896581442 >> 172.21.15.69:0/3896581442 conn(0x55b109c31d90 msgr2=0x55b109c79e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.658+0000 7fbab67e4740 1 -- 172.21.15.69:0/3896581442 shutdown_connections 2024-04-24T08:13:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.658+0000 7fbab67e4740 1 -- 172.21.15.69:0/3896581442 wait complete. 2024-04-24T08:13:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.658+0000 7fbab67e4740 1 -- start start 2024-04-24T08:13:56.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.659+0000 7fbab67e4740 1 Processor -- start 2024-04-24T08:13:56.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.659+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.659+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c5b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.659+0000 7fbab67e4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c2fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.659+0000 7fbab67e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b109c32990 con 0x55b109c5f9b0 2024-04-24T08:13:56.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbab67e4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbaac1ff870 con 0x55b109c5c0b0 2024-04-24T08:13:56.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbab67e4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55b109c700e0 con 0x55b109c79950 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbab67e4740 10 client.0 _ll_register_callbacks cb 0x7fbaa4008cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbaa3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c5b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbaa3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c5b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54178/0 (socket says 172.21.15.69:54178) 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbaa3e00640 1 -- 172.21.15.69:0/4025100142 learned_addr learned my addr 172.21.15.69:0/4025100142 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbab0e00640 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbab1800640 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c2fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbaa3e00640 1 -- 172.21.15.69:0/4025100142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 msgr2=0x55b109c2fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbaa3e00640 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c2fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbaa3e00640 1 -- 172.21.15.69:0/4025100142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 msgr2=0x55b109c5b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbaa3e00640 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbaa3e00640 1 -- 172.21.15.69:0/4025100142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b109840490 con 0x55b109c5f9b0 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.660+0000 7fbab0e00640 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.661+0000 7fbaa3e00640 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c5b630 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fbaac15b360 tx=0x7fba94005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18bc140d64ce319 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.661+0000 7fba9a000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:13:56.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.661+0000 7fba9a000640 1 -- 172.21.15.69:0/4025100142 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba94022420 con 0x55b109c5f9b0 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.661+0000 7fba9a000640 1 -- 172.21.15.69:0/4025100142 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7fba94023cd0 con 0x55b109c5f9b0 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.661+0000 7fbab67e4740 1 -- 172.21.15.69:0/4025100142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba9c07aa10 con 0x55b109c5f9b0 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.661+0000 7fba9a000640 1 -- 172.21.15.69:0/4025100142 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fba94012420 con 0x55b109c5f9b0 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.661+0000 7fbab67e4740 10 client.7215 fetch_fsmap learned FSMap version 51 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.661+0000 7fbab67e4740 1 -- 172.21.15.69:0/4025100142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55b109c57f50 con 0x55b109c5f9b0 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.661+0000 7fba9a000640 1 -- 172.21.15.69:0/4025100142 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9c07aa10 con 0x55b109c5f9b0 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.662+0000 7fba9a000640 1 -- 172.21.15.69:0/4025100142 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7fba940125c0 con 0x55b109c5f9b0 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.662+0000 7fbab67e4740 10 client.7215 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.662+0000 7fbab67e4740 10 client.7215 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.662+0000 7fbab67e4740 20 client.7215 populate_metadata read hostname 'smithi069' 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.662+0000 7fbab67e4740 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55b109cfbfb0 0x55b109c301a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.662+0000 7fbab67e4740 4 client.7215 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}] 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.662+0000 7fbab67e4740 1 -- 172.21.15.69:0/4025100142 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}) -- 0x55b109b9cc30 con 0x55b109cfbfb0 2024-04-24T08:13:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.662+0000 7fbab0e00640 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55b109cfbfb0 0x55b109c301a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.662+0000 7fbab0e00640 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55b109cfbfb0 0x55b109c301a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5869948395addca server_cookie=b807e052ba141af4 in_seq=0 out_seq=0 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.662+0000 7fba9a000640 10 client.7215 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fba9a000640 1 -- 172.21.15.69:0/4025100142 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b109b9cc30 con 0x55b109cfbfb0 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fba9a000640 10 client.7215 handle_command_reply: tid=0 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fbab67e4740 1 client.7215 shutdown 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fbab67e4740 1 -- 172.21.15.69:0/4025100142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 msgr2=0x55b109c5b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fbab67e4740 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c5b630 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fbaac15b360 tx=0x7fba94005f10 comp rx=0 tx=0).stop 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fbab67e4740 1 -- 172.21.15.69:0/4025100142 shutdown_connections 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fbab67e4740 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55b109cfbfb0 0x55b109c301a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fbab67e4740 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b109c5f9b0 0x55b109c5b630 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fbab67e4740 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b109c79950 0x55b109c2fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fbab67e4740 1 --2- 172.21.15.69:0/4025100142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b109c5c0b0 0x55b109c5b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fba9a000640 0 client.7215 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.663+0000 7fbab67e4740 1 -- 172.21.15.69:0/4025100142 >> 172.21.15.69:0/4025100142 conn(0x55b109c31d90 msgr2=0x55b109c76f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.664+0000 7fbab67e4740 1 -- 172.21.15.69:0/4025100142 shutdown_connections 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.664+0000 7fbab67e4740 1 -- 172.21.15.69:0/4025100142 wait complete. 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.664+0000 7fbab67e4740 20 client.7215 trim_cache size 0 max 16384 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "9abc36ec-a16f-414d-a503-178bbb22a2a6", 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-04-24T08:13:56.667 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:13:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.664+0000 7fbab3800640 1 -- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba8806c920 msgr2=0x7fba8806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.664+0000 7fbab3800640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba8806c920 0x7fba8806ede0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fba94007fd0 tx=0x7fba94007920 comp rx=0 tx=0).stop 2024-04-24T08:13:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.664+0000 7fbab3800640 1 -- 172.21.15.69:0/1788647689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 msgr2=0x7fbaac158630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.664+0000 7fbab3800640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 0x7fbaac158630 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fba9c000c00 tx=0x7fba9c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.664+0000 7fbab3800640 1 -- 172.21.15.69:0/1788647689 shutdown_connections 2024-04-24T08:13:56.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.664+0000 7fbab3800640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba8806c920 0x7fba8806ede0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.665+0000 7fbab3800640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbaac15c3e0 0x7fbaac0c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.665+0000 7fbab3800640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbaac0b3fb0 0x7fbaac0c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.665+0000 7fbab3800640 1 --2- 172.21.15.69:0/1788647689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbaac0aeb00 0x7fbaac158630 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.665+0000 7fbab3800640 1 -- 172.21.15.69:0/1788647689 >> 172.21.15.69:0/1788647689 conn(0x7fbaac06be40 msgr2=0x7fbaac0ad5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.665+0000 7fbab3800640 1 -- 172.21.15.69:0/1788647689 shutdown_connections 2024-04-24T08:13:56.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.665+0000 7fbab3800640 1 -- 172.21.15.69:0/1788647689 wait complete. 2024-04-24T08:13:56.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:56.683 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-04-24T08:13:56.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.888+0000 7f7a27200640 1 Processor -- start 2024-04-24T08:13:56.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.888+0000 7f7a27200640 1 -- start start 2024-04-24T08:13:56.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.890+0000 7f7a27200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a20073db0 0x7f7a200b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.890+0000 7f7a27200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200b4e90 0x7f7a200cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.890+0000 7f7a27200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a201631c0 con 0x7f7a200cccc0 2024-04-24T08:13:56.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.890+0000 7f7a27200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7a200cd690 con 0x7f7a20073db0 2024-04-24T08:13:56.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.890+0000 7f7a27200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a200cf4d0 con 0x7f7a200b4e90 2024-04-24T08:13:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.890+0000 7f7a1f400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7a200cccc0 0x7f7a200b4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:59748/0 (socket says 172.21.15.69:59748) 2024-04-24T08:13:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.890+0000 7f7a1f400640 1 -- 172.21.15.69:0/1465642840 learned_addr learned my addr 172.21.15.69:0/1465642840 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:13:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.890+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/1465642840 >> v2:172.21.15.69:3300/0 conn(0x7f7a20073db0 0x7f7a200b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.890+0000 7f7a25200640 1 --2- 172.21.15.69:0/1465642840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200b4e90 0x7f7a200cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.891+0000 7f7a1ea00640 1 -- 172.21.15.69:0/1465642840 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2635846682 0 0) 0x7f7a201631c0 con 0x7f7a200cccc0 2024-04-24T08:13:56.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.891+0000 7f7a1ea00640 1 -- 172.21.15.69:0/1465642840 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a04003520 con 0x7f7a200cccc0 2024-04-24T08:13:56.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.891+0000 7f7a1fe00640 1 -- 172.21.15.69:0/1465642840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200b4e90 msgr2=0x7f7a200cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.891+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/1465642840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200b4e90 0x7f7a200cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.891+0000 7f7a1fe00640 1 -- 172.21.15.69:0/1465642840 >> v1:172.21.15.69:6789/0 conn(0x7f7a200cccc0 legacy=0x7f7a200b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.891+0000 7f7a1fe00640 1 -- 172.21.15.69:0/1465642840 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a200cf7b0 con 0x7f7a20073db0 2024-04-24T08:13:56.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.891+0000 7f7a25200640 1 --2- 172.21.15.69:0/1465642840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200b4e90 0x7f7a200cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:56.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.892+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/1465642840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a20073db0 0x7f7a200b3260 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f7a100097a0 tx=0x7f7a1002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4dcc6a8217c1adc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.892+0000 7f7a1ea00640 1 -- 172.21.15.69:0/1465642840 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a10037070 con 0x7f7a20073db0 2024-04-24T08:13:56.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.892+0000 7f7a1ea00640 1 -- 172.21.15.69:0/1465642840 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a10032440 con 0x7f7a20073db0 2024-04-24T08:13:56.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.892+0000 7f7a1ea00640 1 -- 172.21.15.69:0/1465642840 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a100327f0 con 0x7f7a20073db0 2024-04-24T08:13:56.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.893+0000 7f7a27200640 1 -- 172.21.15.69:0/1465642840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a20073db0 msgr2=0x7f7a200b3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.893+0000 7f7a27200640 1 --2- 172.21.15.69:0/1465642840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a20073db0 0x7f7a200b3260 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f7a100097a0 tx=0x7f7a1002a180 comp rx=0 tx=0).stop 2024-04-24T08:13:56.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.893+0000 7f7a27200640 1 -- 172.21.15.69:0/1465642840 shutdown_connections 2024-04-24T08:13:56.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.893+0000 7f7a27200640 1 --2- 172.21.15.69:0/1465642840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200b4e90 0x7f7a200cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.893+0000 7f7a27200640 1 --2- 172.21.15.69:0/1465642840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a20073db0 0x7f7a200b3260 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.893+0000 7f7a27200640 1 -- 172.21.15.69:0/1465642840 >> 172.21.15.69:0/1465642840 conn(0x7f7a200d0350 msgr2=0x7f7a200d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.893+0000 7f7a27200640 1 -- 172.21.15.69:0/1465642840 shutdown_connections 2024-04-24T08:13:56.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.893+0000 7f7a27200640 1 -- 172.21.15.69:0/1465642840 wait complete. 2024-04-24T08:13:56.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.895+0000 7f7a27200640 1 Processor -- start 2024-04-24T08:13:56.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.895+0000 7f7a27200640 1 -- start start 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.896+0000 7f7a27200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a20073db0 0x7f7a20163f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.896+0000 7f7a27200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a200b4e90 0x7f7a201506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.896+0000 7f7a27200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200cccc0 0x7f7a200ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.896+0000 7f7a27200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a200d0050 con 0x7f7a200b4e90 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.896+0000 7f7a27200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a200cfd70 con 0x7f7a20073db0 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.896+0000 7f7a27200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a200ce470 con 0x7f7a200cccc0 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.896+0000 7f7a1f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a200b4e90 0x7f7a201506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.896+0000 7f7a1f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a200b4e90 0x7f7a201506c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54192/0 (socket says 172.21.15.69:54192) 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.896+0000 7f7a1f400640 1 -- 172.21.15.69:0/3785088155 learned_addr learned my addr 172.21.15.69:0/3785088155 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a20073db0 0x7f7a20163f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a25200640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200cccc0 0x7f7a200ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a1f400640 1 -- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200cccc0 msgr2=0x7f7a200ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a1f400640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200cccc0 0x7f7a200ce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a1f400640 1 -- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a20073db0 msgr2=0x7f7a20163f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a1f400640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a20073db0 0x7f7a20163f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a1f400640 1 -- 172.21.15.69:0/3785088155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a201644c0 con 0x7f7a200b4e90 2024-04-24T08:13:56.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a20073db0 0x7f7a20163f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:56.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a1f400640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a200b4e90 0x7f7a201506c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c390 tx=0x7f7a0c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a17e00640 1 -- 172.21.15.69:0/3785088155 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c017070 con 0x7f7a200b4e90 2024-04-24T08:13:56.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a17e00640 1 -- 172.21.15.69:0/3785088155 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0c005ce0 con 0x7f7a200b4e90 2024-04-24T08:13:56.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a17e00640 1 -- 172.21.15.69:0/3785088155 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0072f0 con 0x7f7a200b4e90 2024-04-24T08:13:56.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.897+0000 7f7a27200640 1 -- 172.21.15.69:0/3785088155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a20160480 con 0x7f7a200b4e90 2024-04-24T08:13:56.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.898+0000 7f7a27200640 1 -- 172.21.15.69:0/3785088155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a20163780 con 0x7f7a200b4e90 2024-04-24T08:13:56.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.900+0000 7f7a17e00640 1 -- 172.21.15.69:0/3785088155 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7a0c01a070 con 0x7f7a200b4e90 2024-04-24T08:13:56.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.901+0000 7f7a17e00640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a0006c920 0x7f7a0006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.901+0000 7f7a17e00640 1 -- 172.21.15.69:0/3785088155 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f7a0c08aff0 con 0x7f7a200b4e90 2024-04-24T08:13:56.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.901+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a0006c920 0x7f7a0006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.901+0000 7f7a2a050740 1 Processor -- start 2024-04-24T08:13:56.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.902+0000 7f7a2a050740 1 -- start start 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.902+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e1fe25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.902+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e20120d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.902+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e2012610 0x55a0e1fe7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.902+0000 7f7a2a050740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a0e1f4c1f0 con 0x55a0e202fc30 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.902+0000 7f7a2a050740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a0e1ef2350 con 0x55a0e2012d50 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.902+0000 7f7a2a050740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a0e1f481b0 con 0x55a0e2012610 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a0006c920 0x7f7a0006ede0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7a10034000 tx=0x7f7a1002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e1fe25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e1fe25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54204/0 (socket says 172.21.15.69:54204) 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1fe00640 1 -- 172.21.15.69:0/456783560 learned_addr learned my addr 172.21.15.69:0/456783560 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1f400640 1 --2- 172.21.15.69:0/456783560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e20120d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1fe00640 1 -- 172.21.15.69:0/456783560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e2012610 msgr2=0x55a0e1fe7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/456783560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e2012610 0x55a0e1fe7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1fe00640 1 -- 172.21.15.69:0/456783560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 msgr2=0x55a0e20120d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/456783560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e20120d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1fe00640 1 -- 172.21.15.69:0/456783560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0e1f46f50 con 0x55a0e202fc30 2024-04-24T08:13:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.903+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/456783560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e1fe25c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f7a10032780 tx=0x7f7a1002c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69cacdf55c05de3a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.904+0000 7f7a14c00640 1 -- 172.21.15.69:0/456783560 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1004c070 con 0x55a0e202fc30 2024-04-24T08:13:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.904+0000 7f7a14c00640 1 -- 172.21.15.69:0/456783560 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a1003c660 con 0x55a0e202fc30 2024-04-24T08:13:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.904+0000 7f7a14c00640 1 -- 172.21.15.69:0/456783560 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a10037040 con 0x55a0e202fc30 2024-04-24T08:13:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.904+0000 7f7a2a050740 1 -- 172.21.15.69:0/456783560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 msgr2=0x55a0e1fe25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.904+0000 7f7a2a050740 1 --2- 172.21.15.69:0/456783560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e1fe25c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f7a10032780 tx=0x7f7a1002c000 comp rx=0 tx=0).stop 2024-04-24T08:13:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.904+0000 7f7a2a050740 1 -- 172.21.15.69:0/456783560 shutdown_connections 2024-04-24T08:13:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.904+0000 7f7a2a050740 1 --2- 172.21.15.69:0/456783560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e2012610 0x55a0e1fe7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.904+0000 7f7a2a050740 1 --2- 172.21.15.69:0/456783560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e20120d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.904+0000 7f7a2a050740 1 --2- 172.21.15.69:0/456783560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e1fe25c0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.904+0000 7f7a2a050740 1 -- 172.21.15.69:0/456783560 >> 172.21.15.69:0/456783560 conn(0x55a0e1fe4830 msgr2=0x55a0e202d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.905+0000 7f7a2a050740 1 -- 172.21.15.69:0/456783560 shutdown_connections 2024-04-24T08:13:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.905+0000 7f7a2a050740 1 -- 172.21.15.69:0/456783560 wait complete. 2024-04-24T08:13:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.905+0000 7f7a2a050740 1 -- start start 2024-04-24T08:13:56.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.905+0000 7f7a2a050740 1 Processor -- start 2024-04-24T08:13:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.906+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e202e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.906+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e200d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.906+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202bfd0 0x55a0e202d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.906+0000 7f7a2a050740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a0e1fe69e0 con 0x55a0e202fc30 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.906+0000 7f7a2a050740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a0e1f05690 con 0x55a0e2012d50 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.906+0000 7f7a2a050740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a0e1fe5dd0 con 0x55a0e202bfd0 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a2a050740 10 client.0 _ll_register_callbacks cb 0x55a0e1f41650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a1f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e200d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a1f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e200d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54214/0 (socket says 172.21.15.69:54214) 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a1f400640 1 -- 172.21.15.69:0/3353490519 learned_addr learned my addr 172.21.15.69:0/3353490519 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a25200640 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202bfd0 0x55a0e202d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e202e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a1f400640 1 -- 172.21.15.69:0/3353490519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202bfd0 msgr2=0x55a0e202d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a1f400640 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202bfd0 0x55a0e202d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a1f400640 1 -- 172.21.15.69:0/3353490519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 msgr2=0x55a0e202e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a1f400640 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e202e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a1f400640 1 -- 172.21.15.69:0/3353490519 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0e1f04920 con 0x55a0e202fc30 2024-04-24T08:13:56.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.907+0000 7f7a1f400640 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e200d4d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0612d0 tx=0x7f7a0c058ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83198918768b7c9a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a09600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a09600640 1 -- 172.21.15.69:0/3353490519 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c007490 con 0x55a0e202fc30 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a09600640 1 -- 172.21.15.69:0/3353490519 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f7a0c007630 con 0x55a0e202fc30 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a2a050740 1 -- 172.21.15.69:0/3353490519 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a1003c660 con 0x55a0e202fc30 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a09600640 1 -- 172.21.15.69:0/3353490519 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f7a0c089120 con 0x55a0e202fc30 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a2a050740 10 client.7230 fetch_fsmap learned FSMap version 51 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a2a050740 1 -- 172.21.15.69:0/3353490519 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55a0e1d287d0 con 0x55a0e202fc30 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a09600640 1 -- 172.21.15.69:0/3353490519 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a1003c660 con 0x55a0e202fc30 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a09600640 1 -- 172.21.15.69:0/3353490519 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f7a0c058070 con 0x55a0e202fc30 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a2a050740 10 client.7230 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a2a050740 10 client.7230 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a2a050740 20 client.7230 populate_metadata read hostname 'smithi069' 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55a0e2024260 0x55a0e2024670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a2a050740 4 client.7230 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.908+0000 7f7a2a050740 1 -- 172.21.15.69:0/3353490519 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a0e1f59c10 con 0x55a0e2024260 2024-04-24T08:13:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.909+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55a0e2024260 0x55a0e2024670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.909+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55a0e2024260 0x55a0e2024670 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b83207f4d8963b3 server_cookie=52a699c424d91a08 in_seq=0 out_seq=0 2024-04-24T08:13:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.909+0000 7f7a09600640 10 client.7230 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a09600640 1 -- 172.21.15.69:0/3353490519 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a0e1f59c10 con 0x55a0e2024260 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a09600640 10 client.7230 handle_command_reply: tid=0 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a2a050740 1 client.7230 shutdown 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a2a050740 1 -- 172.21.15.69:0/3353490519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 msgr2=0x55a0e200d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e200d4d0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c0612d0 tx=0x7f7a0c058ce0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a2a050740 1 -- 172.21.15.69:0/3353490519 shutdown_connections 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55a0e2024260 0x55a0e2024670 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e200d4d0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202bfd0 0x55a0e202d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a09600640 0 client.7230 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3353490519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e202e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a2a050740 1 -- 172.21.15.69:0/3353490519 >> 172.21.15.69:0/3353490519 conn(0x55a0e2012610 msgr2=0x55a0e1fe1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.910+0000 7f7a2a050740 1 -- 172.21.15.69:0/3353490519 shutdown_connections 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.911+0000 7f7a2a050740 1 -- 172.21.15.69:0/3353490519 wait complete. 2024-04-24T08:13:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.911+0000 7f7a2a050740 20 client.7230 trim_cache size 0 max 16384 2024-04-24T08:13:56.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.930+0000 7f7a2a050740 1 Processor -- start 2024-04-24T08:13:56.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.930+0000 7f7a2a050740 1 -- start start 2024-04-24T08:13:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.930+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e2012d50 0x55a0e2014730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.930+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e202bfd0 0x55a0e1fe6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.930+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202fc30 0x55a0e2050f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.930+0000 7f7a2a050740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a202051b0 con 0x55a0e2012d50 2024-04-24T08:13:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.930+0000 7f7a2a050740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a200af180 con 0x55a0e202bfd0 2024-04-24T08:13:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.930+0000 7f7a2a050740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a200ae4e0 con 0x55a0e202fc30 2024-04-24T08:13:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.931+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e2012d50 0x55a0e2014730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.931+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e2012d50 0x55a0e2014730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54226/0 (socket says 172.21.15.69:54226) 2024-04-24T08:13:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.931+0000 7f7a1fe00640 1 -- 172.21.15.69:0/219740686 learned_addr learned my addr 172.21.15.69:0/219740686 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.931+0000 7f7a1fe00640 1 -- 172.21.15.69:0/219740686 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202fc30 msgr2=0x55a0e2050f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.931+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/219740686 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202fc30 0x55a0e2050f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.931+0000 7f7a1fe00640 1 -- 172.21.15.69:0/219740686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e202bfd0 msgr2=0x55a0e1fe6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.931+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/219740686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e202bfd0 0x55a0e1fe6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.931+0000 7f7a1fe00640 1 -- 172.21.15.69:0/219740686 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a0e202f360 con 0x55a0e2012d50 2024-04-24T08:13:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.931+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/219740686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e2012d50 0x55a0e2014730 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f7a1002a770 tx=0x7f7a1003a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e97db0e6d001b80 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.932+0000 7f7a0be00640 1 -- 172.21.15.69:0/219740686 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a10056070 con 0x55a0e2012d50 2024-04-24T08:13:56.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.932+0000 7f7a0be00640 1 -- 172.21.15.69:0/219740686 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a10052470 con 0x55a0e2012d50 2024-04-24T08:13:56.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.932+0000 7f7a0be00640 1 -- 172.21.15.69:0/219740686 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1004c040 con 0x55a0e2012d50 2024-04-24T08:13:56.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.932+0000 7f7a2a050740 1 -- 172.21.15.69:0/219740686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e2012d50 msgr2=0x55a0e2014730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.932+0000 7f7a2a050740 1 --2- 172.21.15.69:0/219740686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e2012d50 0x55a0e2014730 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f7a1002a770 tx=0x7f7a1003a000 comp rx=0 tx=0).stop 2024-04-24T08:13:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.932+0000 7f7a2a050740 1 -- 172.21.15.69:0/219740686 shutdown_connections 2024-04-24T08:13:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.932+0000 7f7a2a050740 1 --2- 172.21.15.69:0/219740686 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202fc30 0x55a0e2050f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.932+0000 7f7a2a050740 1 --2- 172.21.15.69:0/219740686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e202bfd0 0x55a0e1fe6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.932+0000 7f7a2a050740 1 --2- 172.21.15.69:0/219740686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e2012d50 0x55a0e2014730 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.932+0000 7f7a2a050740 1 -- 172.21.15.69:0/219740686 >> 172.21.15.69:0/219740686 conn(0x55a0e2012610 msgr2=0x55a0e200d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.933+0000 7f7a2a050740 1 -- 172.21.15.69:0/219740686 shutdown_connections 2024-04-24T08:13:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.933+0000 7f7a2a050740 1 -- 172.21.15.69:0/219740686 wait complete. 2024-04-24T08:13:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.933+0000 7f7a2a050740 1 -- start start 2024-04-24T08:13:56.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.933+0000 7f7a2a050740 1 Processor -- start 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.934+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e20aa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.934+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202bfd0 0x55a0e2022a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.934+0000 7f7a2a050740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e2022fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.934+0000 7f7a2a050740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a0e200f7f0 con 0x55a0e202fc30 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.934+0000 7f7a2a050740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a200aeb30 con 0x55a0e2012d50 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.934+0000 7f7a2a050740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a0e1fe2970 con 0x55a0e202bfd0 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.934+0000 7f7a2a050740 10 client.0 _ll_register_callbacks cb 0x7f7a0c054ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a25200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e2022fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a25200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e2022fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54236/0 (socket says 172.21.15.69:54236) 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a25200640 1 -- 172.21.15.69:0/3216724922 learned_addr learned my addr 172.21.15.69:0/3216724922 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a1f400640 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202bfd0 0x55a0e2022a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e20aa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a25200640 1 -- 172.21.15.69:0/3216724922 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202bfd0 msgr2=0x55a0e2022a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a25200640 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202bfd0 0x55a0e2022a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a25200640 1 -- 172.21.15.69:0/3216724922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 msgr2=0x55a0e20aa290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a25200640 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e20aa290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a25200640 1 -- 172.21.15.69:0/3216724922 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a0e1c5e3f0 con 0x55a0e202fc30 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a25200640 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e2022fa0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7a18004320 tx=0x7f7a180047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1464d5237e71953 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.935+0000 7f7a09600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a09600640 1 -- 172.21.15.69:0/3216724922 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18007950 con 0x55a0e202fc30 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a09600640 1 -- 172.21.15.69:0/3216724922 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(69..69 src has 1..69) ==== 5795+0+0 (secure 0 0 0) 0x7f7a18007af0 con 0x55a0e202fc30 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a2a050740 1 -- 172.21.15.69:0/3216724922 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a10052470 con 0x55a0e202fc30 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a09600640 1 -- 172.21.15.69:0/3216724922 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f7a18021050 con 0x55a0e202fc30 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a2a050740 10 client.7236 fetch_fsmap learned FSMap version 51 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a2a050740 1 -- 172.21.15.69:0/3216724922 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x55a0e1c1e0f0 con 0x55a0e202fc30 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a09600640 1 -- 172.21.15.69:0/3216724922 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a10052470 con 0x55a0e202fc30 2024-04-24T08:13:56.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a09600640 1 -- 172.21.15.69:0/3216724922 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f7a18024020 con 0x55a0e202fc30 2024-04-24T08:13:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a2a050740 10 client.7236 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:13:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a2a050740 10 client.7236 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:13:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a2a050740 20 client.7236 populate_metadata read hostname 'smithi069' 2024-04-24T08:13:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55a0e20120a0 0x55a0e203bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a2a050740 4 client.7236 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:13:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a2a050740 1 -- 172.21.15.69:0/3216724922 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a0e1fe3c80 con 0x55a0e20120a0 2024-04-24T08:13:56.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.936+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55a0e20120a0 0x55a0e203bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a1fe00640 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55a0e20120a0 0x55a0e203bf40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5b4227e53bbcdb4 server_cookie=2986edf58cc6175c in_seq=0 out_seq=0 2024-04-24T08:13:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a09600640 10 client.7236 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a09600640 1 -- 172.21.15.69:0/3216724922 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a0e1fe3c80 con 0x55a0e20120a0 2024-04-24T08:13:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a09600640 10 client.7236 handle_command_reply: tid=0 2024-04-24T08:13:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a2a050740 1 client.7236 shutdown 2024-04-24T08:13:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a2a050740 1 -- 172.21.15.69:0/3216724922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 msgr2=0x55a0e2022fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e2022fa0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7a18004320 tx=0x7f7a180047e0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a2a050740 1 -- 172.21.15.69:0/3216724922 shutdown_connections 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55a0e20120a0 0x55a0e203bf40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a0e202fc30 0x55a0e2022fa0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a0e202bfd0 0x55a0e2022a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a2a050740 1 --2- 172.21.15.69:0/3216724922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a0e2012d50 0x55a0e20aa290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.937+0000 7f7a09600640 0 client.7236 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.938+0000 7f7a2a050740 1 -- 172.21.15.69:0/3216724922 >> 172.21.15.69:0/3216724922 conn(0x55a0e2012610 msgr2=0x55a0e202cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.938+0000 7f7a2a050740 1 -- 172.21.15.69:0/3216724922 shutdown_connections 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.938+0000 7f7a2a050740 1 -- 172.21.15.69:0/3216724922 wait complete. 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.938+0000 7f7a2a050740 20 client.7236 trim_cache size 0 max 16384 2024-04-24T08:13:56.941 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:13:56.942 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:13:56.942 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-04-24T08:13:56.942 INFO:teuthology.orchestra.run.smithi069.stdout: "9abc36ec-a16f-414d-a503-178bbb22a2a6": { 2024-04-24T08:13:56.942 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/subdir", 2024-04-24T08:13:56.942 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "9abc36ec-a16f-414d-a503-178bbb22a2a6", 2024-04-24T08:13:56.942 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive" 2024-04-24T08:13:56.942 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:56.942 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:13:56.942 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:13:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.938+0000 7f7a27200640 1 -- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a0006c920 msgr2=0x7f7a0006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.938+0000 7f7a27200640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a0006c920 0x7f7a0006ede0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7a10034000 tx=0x7f7a1002e000 comp rx=0 tx=0).stop 2024-04-24T08:13:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.938+0000 7f7a27200640 1 -- 172.21.15.69:0/3785088155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a200b4e90 msgr2=0x7f7a201506c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.938+0000 7f7a27200640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a200b4e90 0x7f7a201506c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c390 tx=0x7f7a0c00c850 comp rx=0 tx=0).stop 2024-04-24T08:13:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.939+0000 7f7a27200640 1 -- 172.21.15.69:0/3785088155 shutdown_connections 2024-04-24T08:13:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.939+0000 7f7a27200640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a0006c920 0x7f7a0006ede0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.939+0000 7f7a27200640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a200b4e90 0x7f7a201506c0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.939+0000 7f7a27200640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a200cccc0 0x7f7a200ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.939+0000 7f7a27200640 1 --2- 172.21.15.69:0/3785088155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a20073db0 0x7f7a20163f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:56.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.939+0000 7f7a27200640 1 -- 172.21.15.69:0/3785088155 >> 172.21.15.69:0/3785088155 conn(0x7f7a200d0350 msgr2=0x7f7a20162890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:56.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.939+0000 7f7a27200640 1 -- 172.21.15.69:0/3785088155 shutdown_connections 2024-04-24T08:13:56.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:13:56.939+0000 7f7a27200640 1 -- 172.21.15.69:0/3785088155 wait complete. 2024-04-24T08:13:56.956 INFO:tasks.cephfs.filesystem:scrub status for tag:9abc36ec-a16f-414d-a503-178bbb22a2a6 - {'path': '/subdir', 'tag': '9abc36ec-a16f-414d-a503-178bbb22a2a6', 'options': 'recursive'} 2024-04-24T08:14:14.916 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:14:14.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:14:26.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:14:26.958 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-04-24T08:14:27.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.179+0000 7ff17e000640 1 Processor -- start 2024-04-24T08:14:27.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.179+0000 7ff17e000640 1 -- start start 2024-04-24T08:14:27.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.181+0000 7ff17e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff178071d70 0x7ff17815ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.181+0000 7ff17e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff178160510 0x7ff1780cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.181+0000 7ff17e000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff178072450 con 0x7ff1781420a0 2024-04-24T08:14:27.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.181+0000 7ff17e000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff178160960 con 0x7ff178071d70 2024-04-24T08:14:27.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.181+0000 7ff17e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff1780cf1e0 con 0x7ff178160510 2024-04-24T08:14:27.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.181+0000 7ff177e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff178071d70 0x7ff17815ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.181+0000 7ff177e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff178071d70 0x7ff17815ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50808/0 (socket says 172.21.15.69:50808) 2024-04-24T08:14:27.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.181+0000 7ff177e00640 1 -- 172.21.15.69:0/3667090661 learned_addr learned my addr 172.21.15.69:0/3667090661 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.181+0000 7ff17d600640 1 --2- 172.21.15.69:0/3667090661 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff178160510 0x7ff1780cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.182+0000 7ff177e00640 1 -- 172.21.15.69:0/3667090661 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff178160510 msgr2=0x7ff1780cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.182+0000 7ff177e00640 1 --2- 172.21.15.69:0/3667090661 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff178160510 0x7ff1780cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.182+0000 7ff177e00640 1 -- 172.21.15.69:0/3667090661 >> v1:172.21.15.121:6789/0 conn(0x7ff1781420a0 legacy=0x7ff178142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.182+0000 7ff177e00640 1 -- 172.21.15.69:0/3667090661 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff178072730 con 0x7ff178071d70 2024-04-24T08:14:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.183+0000 7ff177e00640 1 --2- 172.21.15.69:0/3667090661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff178071d70 0x7ff17815ffd0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009870 tx=0x7ff16c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fddd48419df4c15 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.183+0000 7ff177400640 1 -- 172.21.15.69:0/3667090661 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c037070 con 0x7ff178071d70 2024-04-24T08:14:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.183+0000 7ff177400640 1 -- 172.21.15.69:0/3667090661 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff16c032440 con 0x7ff178071d70 2024-04-24T08:14:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.183+0000 7ff177400640 1 -- 172.21.15.69:0/3667090661 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c032810 con 0x7ff178071d70 2024-04-24T08:14:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.184+0000 7ff17e000640 1 -- 172.21.15.69:0/3667090661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff178071d70 msgr2=0x7ff17815ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.184+0000 7ff17e000640 1 --2- 172.21.15.69:0/3667090661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff178071d70 0x7ff17815ffd0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009870 tx=0x7ff16c02a210 comp rx=0 tx=0).stop 2024-04-24T08:14:27.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.184+0000 7ff17e000640 1 -- 172.21.15.69:0/3667090661 shutdown_connections 2024-04-24T08:14:27.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.184+0000 7ff17e000640 1 --2- 172.21.15.69:0/3667090661 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff178160510 0x7ff1780cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.184+0000 7ff17e000640 1 --2- 172.21.15.69:0/3667090661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff178071d70 0x7ff17815ffd0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.184+0000 7ff17e000640 1 -- 172.21.15.69:0/3667090661 >> 172.21.15.69:0/3667090661 conn(0x7ff178067b80 msgr2=0x7ff17806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.184+0000 7ff17e000640 1 -- 172.21.15.69:0/3667090661 shutdown_connections 2024-04-24T08:14:27.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.184+0000 7ff17e000640 1 -- 172.21.15.69:0/3667090661 wait complete. 2024-04-24T08:14:27.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.186+0000 7ff17e000640 1 Processor -- start 2024-04-24T08:14:27.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.186+0000 7ff17e000640 1 -- start start 2024-04-24T08:14:27.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff17e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff178071d70 0x7ff1780ae940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff17e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1781420a0 0x7ff17813ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff17e000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff178160510 0x7ff1780a9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff17e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff178072cf0 con 0x7ff1781420a0 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff17e000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff178072200 con 0x7ff178160510 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff17e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff1780cfbb0 con 0x7ff178071d70 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff177e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1781420a0 0x7ff17813ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff177e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1781420a0 0x7ff17813ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50814/0 (socket says 172.21.15.69:50814) 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff177e00640 1 -- 172.21.15.69:0/3008369216 learned_addr learned my addr 172.21.15.69:0/3008369216 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff177e00640 1 -- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff178071d70 msgr2=0x7ff1780ae940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff177e00640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff178071d70 0x7ff1780ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff177e00640 1 -- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff178160510 msgr2=0x7ff1780a9430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff177e00640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff178160510 0x7ff1780a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.187+0000 7ff177e00640 1 -- 172.21.15.69:0/3008369216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1780b53e0 con 0x7ff1781420a0 2024-04-24T08:14:27.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.188+0000 7ff177e00640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1781420a0 0x7ff17813ed60 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009870 tx=0x7ff16c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.188+0000 7ff174c00640 1 -- 172.21.15.69:0/3008369216 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c041020 con 0x7ff1781420a0 2024-04-24T08:14:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.188+0000 7ff174c00640 1 -- 172.21.15.69:0/3008369216 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff16c0306c0 con 0x7ff1781420a0 2024-04-24T08:14:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.188+0000 7ff174c00640 1 -- 172.21.15.69:0/3008369216 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c037040 con 0x7ff1781420a0 2024-04-24T08:14:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.188+0000 7ff17e000640 1 -- 172.21.15.69:0/3008369216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1780b4860 con 0x7ff1781420a0 2024-04-24T08:14:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.188+0000 7ff17e000640 1 -- 172.21.15.69:0/3008369216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1780b4e20 con 0x7ff1781420a0 2024-04-24T08:14:27.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.190+0000 7ff174c00640 1 -- 172.21.15.69:0/3008369216 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff16c044020 con 0x7ff1781420a0 2024-04-24T08:14:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.191+0000 7ff174c00640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff15c06c920 0x7ff15c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.191+0000 7ff174c00640 1 -- 172.21.15.69:0/3008369216 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7ff16c0ac3f0 con 0x7ff1781420a0 2024-04-24T08:14:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.192+0000 7ff182468740 1 Processor -- start 2024-04-24T08:14:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.192+0000 7ff17cc00640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff15c06c920 0x7ff15c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.192+0000 7ff182468740 1 -- start start 2024-04-24T08:14:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.193+0000 7ff182468740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 0x5585bf57b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.193+0000 7ff182468740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.193+0000 7ff182468740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5ab7e0 0x5585bf5805f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.193+0000 7ff182468740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5585bf4e51f0 con 0x5585bf5ab7e0 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.193+0000 7ff182468740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5585bf48b350 con 0x5585bf5abf20 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.193+0000 7ff182468740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5585bf4e11b0 con 0x5585bf5c8e00 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.193+0000 7ff17cc00640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff15c06c920 0x7ff15c06ede0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff17815f320 tx=0x7ff1680079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.193+0000 7ff17d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5ab7e0 0x5585bf5805f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.193+0000 7ff17d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5ab7e0 0x5585bf5805f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50828/0 (socket says 172.21.15.69:50828) 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.193+0000 7ff17d600640 1 -- 172.21.15.69:0/2281359126 learned_addr learned my addr 172.21.15.69:0/2281359126 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff17cc00640 1 --2- 172.21.15.69:0/2281359126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 0x5585bf57b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff177e00640 1 --2- 172.21.15.69:0/2281359126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff17d600640 1 -- 172.21.15.69:0/2281359126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 msgr2=0x5585bf57b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff17d600640 1 --2- 172.21.15.69:0/2281359126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 0x5585bf57b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff17d600640 1 -- 172.21.15.69:0/2281359126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 msgr2=0x5585bf5ab2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff17d600640 1 --2- 172.21.15.69:0/2281359126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5ab2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff17d600640 1 -- 172.21.15.69:0/2281359126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585bf4dff50 con 0x5585bf5ab7e0 2024-04-24T08:14:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff17cc00640 1 --2- 172.21.15.69:0/2281359126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 0x5585bf57b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff17d600640 1 --2- 172.21.15.69:0/2281359126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5ab7e0 0x5585bf5805f0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff170009f70 tx=0x7ff1700077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc7049728e44d3f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff165200640 1 -- 172.21.15.69:0/2281359126 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170019070 con 0x5585bf5ab7e0 2024-04-24T08:14:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff165200640 1 -- 172.21.15.69:0/2281359126 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff170012710 con 0x5585bf5ab7e0 2024-04-24T08:14:27.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.194+0000 7ff165200640 1 -- 172.21.15.69:0/2281359126 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170012b00 con 0x5585bf5ab7e0 2024-04-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.195+0000 7ff182468740 1 -- 172.21.15.69:0/2281359126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5ab7e0 msgr2=0x5585bf5805f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.195+0000 7ff182468740 1 --2- 172.21.15.69:0/2281359126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5ab7e0 0x5585bf5805f0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ff170009f70 tx=0x7ff1700077c0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.195+0000 7ff182468740 1 -- 172.21.15.69:0/2281359126 shutdown_connections 2024-04-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.195+0000 7ff182468740 1 --2- 172.21.15.69:0/2281359126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5ab7e0 0x5585bf5805f0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.195+0000 7ff182468740 1 --2- 172.21.15.69:0/2281359126 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5ab2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.195+0000 7ff182468740 1 --2- 172.21.15.69:0/2281359126 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 0x5585bf57b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.195+0000 7ff182468740 1 -- 172.21.15.69:0/2281359126 >> 172.21.15.69:0/2281359126 conn(0x5585bf57da00 msgr2=0x5585bf5c62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.195+0000 7ff182468740 1 -- 172.21.15.69:0/2281359126 shutdown_connections 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.195+0000 7ff182468740 1 -- 172.21.15.69:0/2281359126 wait complete. 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.196+0000 7ff182468740 1 -- start start 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.196+0000 7ff182468740 1 Processor -- start 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.197+0000 7ff182468740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5aec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.197+0000 7ff182468740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf5d60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.197+0000 7ff182468740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5a7f70 0x5585bf5a83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.197+0000 7ff182468740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5585bf57fbb0 con 0x5585bf5c8e00 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.197+0000 7ff182468740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5585bf49e690 con 0x5585bf5abf20 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.197+0000 7ff182468740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5585bf57efa0 con 0x5585bf5a7f70 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.197+0000 7ff182468740 10 client.0 _ll_register_callbacks cb 0x5585bf4da650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.197+0000 7ff177e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf5d60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.197+0000 7ff177e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf5d60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50838/0 (socket says 172.21.15.69:50838) 2024-04-24T08:14:27.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.197+0000 7ff177e00640 1 -- 172.21.15.69:0/4053925547 learned_addr learned my addr 172.21.15.69:0/4053925547 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff17cc00640 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff177e00640 1 -- 172.21.15.69:0/4053925547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5a7f70 msgr2=0x5585bf5a83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff177e00640 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5a7f70 0x5585bf5a83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff177e00640 1 -- 172.21.15.69:0/4053925547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 msgr2=0x5585bf5aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff177e00640 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff177e00640 1 -- 172.21.15.69:0/4053925547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585bf49d920 con 0x5585bf5c8e00 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff17cc00640 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff177e00640 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf5d60f0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ff16c0826f0 tx=0x7ff16c07a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89fa1ec6e04949d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff159600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff159600640 1 -- 172.21.15.69:0/4053925547 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c07e3f0 con 0x5585bf5c8e00 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff159600640 1 -- 172.21.15.69:0/4053925547 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7ff16c0ac0c0 con 0x5585bf5c8e00 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.198+0000 7ff182468740 1 -- 172.21.15.69:0/4053925547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff170012710 con 0x5585bf5c8e00 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff159600640 1 -- 172.21.15.69:0/4053925547 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7ff16c082c20 con 0x5585bf5c8e00 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff182468740 10 client.7248 fetch_fsmap learned FSMap version 51 2024-04-24T08:14:27.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff182468740 1 -- 172.21.15.69:0/4053925547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x5585bf2c17d0 con 0x5585bf5c8e00 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff159600640 1 -- 172.21.15.69:0/4053925547 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff170012710 con 0x5585bf5c8e00 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff159600640 1 -- 172.21.15.69:0/4053925547 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7ff16c03a030 con 0x5585bf5c8e00 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff182468740 10 client.7248 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff182468740 10 client.7248 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff182468740 20 client.7248 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff182468740 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5585bf5ac8e0 0x5585bf5acd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff182468740 4 client.7248 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff182468740 1 -- 172.21.15.69:0/4053925547 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585bf4f2c10 con 0x5585bf5ac8e0 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.199+0000 7ff17cc00640 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5585bf5ac8e0 0x5585bf5acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.200+0000 7ff159600640 10 client.7248 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:27.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.200+0000 7ff17cc00640 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5585bf5ac8e0 0x5585bf5acd20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2418e9852eb337c5 server_cookie=5d79d6705df7c6b in_seq=0 out_seq=0 2024-04-24T08:14:27.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff159600640 1 -- 172.21.15.69:0/4053925547 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5585bf4f2c10 con 0x5585bf5ac8e0 2024-04-24T08:14:27.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff159600640 10 client.7248 handle_command_reply: tid=0 2024-04-24T08:14:27.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 client.7248 shutdown 2024-04-24T08:14:27.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 -- 172.21.15.69:0/4053925547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 msgr2=0x5585bf5d60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf5d60f0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ff16c0826f0 tx=0x7ff16c07a800 comp rx=0 tx=0).stop 2024-04-24T08:14:27.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 -- 172.21.15.69:0/4053925547 shutdown_connections 2024-04-24T08:14:27.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5585bf5ac8e0 0x5585bf5acd20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf5d60f0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5a7f70 0x5585bf5a83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff159600640 0 client.7248 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:27.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 --2- 172.21.15.69:0/4053925547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 -- 172.21.15.69:0/4053925547 >> 172.21.15.69:0/4053925547 conn(0x5585bf5ab7e0 msgr2=0x5585bf5c62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 -- 172.21.15.69:0/4053925547 shutdown_connections 2024-04-24T08:14:27.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 1 -- 172.21.15.69:0/4053925547 wait complete. 2024-04-24T08:14:27.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.201+0000 7ff182468740 20 client.7248 trim_cache size 0 max 16384 2024-04-24T08:14:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.221+0000 7ff182468740 1 Processor -- start 2024-04-24T08:14:27.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.221+0000 7ff182468740 1 -- start start 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.221+0000 7ff182468740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5a7f70 0x5585bf5c6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.221+0000 7ff182468740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.221+0000 7ff182468740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf57a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.221+0000 7ff182468740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff178201220 con 0x5585bf5c8e00 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.221+0000 7ff182468740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff1780b4160 con 0x5585bf5abf20 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.221+0000 7ff182468740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff1780b34c0 con 0x5585bf5a7f70 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.222+0000 7ff17d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf57a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.222+0000 7ff17d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf57a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50848/0 (socket says 172.21.15.69:50848) 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.222+0000 7ff17d600640 1 -- 172.21.15.69:0/3782860066 learned_addr learned my addr 172.21.15.69:0/3782860066 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.222+0000 7ff17cc00640 1 --2- 172.21.15.69:0/3782860066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5a7f70 0x5585bf5c6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.222+0000 7ff17d600640 1 -- 172.21.15.69:0/3782860066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5a7f70 msgr2=0x5585bf5c6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.222+0000 7ff17d600640 1 --2- 172.21.15.69:0/3782860066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5a7f70 0x5585bf5c6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.222+0000 7ff17d600640 1 -- 172.21.15.69:0/3782860066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 msgr2=0x5585bf5c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.222+0000 7ff17d600640 1 --2- 172.21.15.69:0/3782860066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.222+0000 7ff17d600640 1 -- 172.21.15.69:0/3782860066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585bf1f73f0 con 0x5585bf5c8e00 2024-04-24T08:14:27.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.223+0000 7ff17d600640 1 --2- 172.21.15.69:0/3782860066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf57a740 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff170000f80 tx=0x7ff170011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f6b3af5f42b4efd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.223+0000 7ff15be00640 1 -- 172.21.15.69:0/3782860066 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170019040 con 0x5585bf5c8e00 2024-04-24T08:14:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.223+0000 7ff15be00640 1 -- 172.21.15.69:0/3782860066 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff170025070 con 0x5585bf5c8e00 2024-04-24T08:14:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.223+0000 7ff15be00640 1 -- 172.21.15.69:0/3782860066 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170003dd0 con 0x5585bf5c8e00 2024-04-24T08:14:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.223+0000 7ff182468740 1 -- 172.21.15.69:0/3782860066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 msgr2=0x5585bf57a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.223+0000 7ff182468740 1 --2- 172.21.15.69:0/3782860066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf57a740 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff170000f80 tx=0x7ff170011000 comp rx=0 tx=0).stop 2024-04-24T08:14:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.224+0000 7ff182468740 1 -- 172.21.15.69:0/3782860066 shutdown_connections 2024-04-24T08:14:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.224+0000 7ff182468740 1 --2- 172.21.15.69:0/3782860066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5c8e00 0x5585bf57a740 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.224+0000 7ff182468740 1 --2- 172.21.15.69:0/3782860066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5abf20 0x5585bf5c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.224+0000 7ff182468740 1 --2- 172.21.15.69:0/3782860066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5a7f70 0x5585bf5c6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.224+0000 7ff182468740 1 -- 172.21.15.69:0/3782860066 >> 172.21.15.69:0/3782860066 conn(0x5585bf5ab7e0 msgr2=0x5585bf576e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.224+0000 7ff182468740 1 -- 172.21.15.69:0/3782860066 shutdown_connections 2024-04-24T08:14:27.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.224+0000 7ff182468740 1 -- 172.21.15.69:0/3782860066 wait complete. 2024-04-24T08:14:27.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.224+0000 7ff182468740 1 -- start start 2024-04-24T08:14:27.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.224+0000 7ff182468740 1 Processor -- start 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.225+0000 7ff182468740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5a7f70 0x5585bf57f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.225+0000 7ff182468740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5abf20 0x5585bf647c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.225+0000 7ff182468740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 0x5585bf580070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.225+0000 7ff182468740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff1780b3b10 con 0x5585bf5abf20 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.225+0000 7ff182468740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5585bf57c1a0 con 0x5585bf5a7f70 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.225+0000 7ff182468740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5585bf5e9830 con 0x5585bf5c8e00 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff182468740 10 client.0 _ll_register_callbacks cb 0x7ff16c0091c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff177e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5abf20 0x5585bf647c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff177e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5abf20 0x5585bf647c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50860/0 (socket says 172.21.15.69:50860) 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff177e00640 1 -- 172.21.15.69:0/3192082574 learned_addr learned my addr 172.21.15.69:0/3192082574 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff17cc00640 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5a7f70 0x5585bf57f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff17d600640 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 0x5585bf580070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff177e00640 1 -- 172.21.15.69:0/3192082574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 msgr2=0x5585bf580070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff177e00640 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 0x5585bf580070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff177e00640 1 -- 172.21.15.69:0/3192082574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5a7f70 msgr2=0x5585bf57f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff177e00640 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5a7f70 0x5585bf57f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff177e00640 1 -- 172.21.15.69:0/3192082574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585bf1b70f0 con 0x5585bf5abf20 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff17cc00640 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5a7f70 0x5585bf57f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff177e00640 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5abf20 0x5585bf647c10 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff16c000c00 tx=0x7ff16c0aa710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2339f993597904ec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.226+0000 7ff159600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff159600640 1 -- 172.21.15.69:0/3192082574 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16c07e3f0 con 0x5585bf5abf20 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff159600640 1 -- 172.21.15.69:0/3192082574 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7ff16c079690 con 0x5585bf5abf20 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff182468740 1 -- 172.21.15.69:0/3192082574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff170025070 con 0x5585bf5abf20 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff159600640 1 -- 172.21.15.69:0/3192082574 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7ff16c0a1050 con 0x5585bf5abf20 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff182468740 10 client.7254 fetch_fsmap learned FSMap version 51 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff182468740 1 -- 172.21.15.69:0/3192082574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x5585bf4aa810 con 0x5585bf5abf20 2024-04-24T08:14:27.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff159600640 1 -- 172.21.15.69:0/3192082574 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff170025070 con 0x5585bf5abf20 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff159600640 1 -- 172.21.15.69:0/3192082574 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7ff16c075d90 con 0x5585bf5abf20 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff182468740 10 client.7254 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff182468740 10 client.7254 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff182468740 20 client.7254 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff182468740 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5585bf5c82e0 0x5585bf5ae7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff182468740 4 client.7254 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.227+0000 7ff182468740 1 -- 172.21.15.69:0/3192082574 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5585bf5a7240 con 0x5585bf5c82e0 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.228+0000 7ff17cc00640 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5585bf5c82e0 0x5585bf5ae7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.228+0000 7ff17cc00640 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5585bf5c82e0 0x5585bf5ae7b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8176a160a0e157bc server_cookie=eaec8ee875860bd1 in_seq=0 out_seq=0 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.228+0000 7ff159600640 10 client.7254 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.228+0000 7ff159600640 1 -- 172.21.15.69:0/3192082574 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5585bf5a7240 con 0x5585bf5c82e0 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.228+0000 7ff159600640 10 client.7254 handle_command_reply: tid=0 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.228+0000 7ff182468740 1 client.7254 shutdown 2024-04-24T08:14:27.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 1 -- 172.21.15.69:0/3192082574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5abf20 msgr2=0x5585bf647c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5abf20 0x5585bf647c10 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7ff16c000c00 tx=0x7ff16c0aa710 comp rx=0 tx=0).stop 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 1 -- 172.21.15.69:0/3192082574 shutdown_connections 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5585bf5c82e0 0x5585bf5ae7b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5585bf5abf20 0x5585bf647c10 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5585bf5c8e00 0x5585bf580070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff159600640 0 client.7254 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 1 --2- 172.21.15.69:0/3192082574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5585bf5a7f70 0x5585bf57f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 1 -- 172.21.15.69:0/3192082574 >> 172.21.15.69:0/3192082574 conn(0x5585bf5ab7e0 msgr2=0x5585bf5c6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 1 -- 172.21.15.69:0/3192082574 shutdown_connections 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 1 -- 172.21.15.69:0/3192082574 wait complete. 2024-04-24T08:14:27.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.229+0000 7ff182468740 20 client.7254 trim_cache size 0 max 16384 2024-04-24T08:14:27.235 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:14:27.235 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-04-24T08:14:27.235 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-04-24T08:14:27.235 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 -- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff15c06c920 msgr2=0x7ff15c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff15c06c920 0x7ff15c06ede0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff17815f320 tx=0x7ff1680079b0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 -- 172.21.15.69:0/3008369216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1781420a0 msgr2=0x7ff17813ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1781420a0 0x7ff17813ed60 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009870 tx=0x7ff16c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 -- 172.21.15.69:0/3008369216 shutdown_connections 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff15c06c920 0x7ff15c06ede0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1781420a0 0x7ff17813ed60 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff178160510 0x7ff1780a9430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 --2- 172.21.15.69:0/3008369216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff178071d70 0x7ff1780ae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 -- 172.21.15.69:0/3008369216 >> 172.21.15.69:0/3008369216 conn(0x7ff178067b80 msgr2=0x7ff17815ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 -- 172.21.15.69:0/3008369216 shutdown_connections 2024-04-24T08:14:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.230+0000 7ff17e000640 1 -- 172.21.15.69:0/3008369216 wait complete. 2024-04-24T08:14:27.250 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:14:27.250 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:14:27.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.448+0000 7f0497200640 1 Processor -- start 2024-04-24T08:14:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.448+0000 7f0497200640 1 -- start start 2024-04-24T08:14:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.449+0000 7f0497200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0498071e50 0x7f049815ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.449+0000 7f0497200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0498162300 0x7f049806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.450+0000 7f0497200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f049815f540 con 0x7f0498072230 2024-04-24T08:14:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.450+0000 7f0497200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f049815f820 con 0x7f0498162300 2024-04-24T08:14:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.450+0000 7f0497200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f049815fb00 con 0x7f0498071e50 2024-04-24T08:14:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.450+0000 7f0495400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0498162300 0x7f049806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.450+0000 7f0495400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0498162300 0x7f049806e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50864/0 (socket says 172.21.15.69:50864) 2024-04-24T08:14:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.450+0000 7f0495400640 1 -- 172.21.15.69:0/2401507635 learned_addr learned my addr 172.21.15.69:0/2401507635 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.450+0000 7f0495e00640 1 --2- 172.21.15.69:0/2401507635 >> v2:172.21.15.121:3300/0 conn(0x7f0498071e50 0x7f049815ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.450+0000 7f0495400640 1 -- 172.21.15.69:0/2401507635 >> v2:172.21.15.121:3300/0 conn(0x7f0498071e50 msgr2=0x7f049815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.450+0000 7f0495400640 1 --2- 172.21.15.69:0/2401507635 >> v2:172.21.15.121:3300/0 conn(0x7f0498071e50 0x7f049815ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.451+0000 7f0495400640 1 -- 172.21.15.69:0/2401507635 >> v1:172.21.15.121:6789/0 conn(0x7f0498072230 legacy=0x7f049806e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.451+0000 7f0495400640 1 -- 172.21.15.69:0/2401507635 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f049806dda0 con 0x7f0498162300 2024-04-24T08:14:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.451+0000 7f0495e00640 1 --2- 172.21.15.69:0/2401507635 >> v2:172.21.15.121:3300/0 conn(0x7f0498071e50 0x7f049815ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.452+0000 7f0495400640 1 --2- 172.21.15.69:0/2401507635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 0x7f049806e2c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f04980b2d00 tx=0x7f048402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233991a8a42dfba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.452+0000 7f0494a00640 1 -- 172.21.15.69:0/2401507635 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484037070 con 0x7f0498162300 2024-04-24T08:14:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.452+0000 7f0494a00640 1 -- 172.21.15.69:0/2401507635 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0484032440 con 0x7f0498162300 2024-04-24T08:14:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.452+0000 7f0494a00640 1 -- 172.21.15.69:0/2401507635 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0484032810 con 0x7f0498162300 2024-04-24T08:14:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.453+0000 7f0497200640 1 -- 172.21.15.69:0/2401507635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 msgr2=0x7f049806e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.453+0000 7f0497200640 1 --2- 172.21.15.69:0/2401507635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 0x7f049806e2c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f04980b2d00 tx=0x7f048402a230 comp rx=0 tx=0).stop 2024-04-24T08:14:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.453+0000 7f0497200640 1 -- 172.21.15.69:0/2401507635 shutdown_connections 2024-04-24T08:14:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.453+0000 7f0497200640 1 --2- 172.21.15.69:0/2401507635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 0x7f049806e2c0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.453+0000 7f0497200640 1 --2- 172.21.15.69:0/2401507635 >> v2:172.21.15.121:3300/0 conn(0x7f0498071e50 0x7f049815ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.453+0000 7f0497200640 1 -- 172.21.15.69:0/2401507635 >> 172.21.15.69:0/2401507635 conn(0x7f04980d0160 msgr2=0x7f04980d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.453+0000 7f0497200640 1 -- 172.21.15.69:0/2401507635 shutdown_connections 2024-04-24T08:14:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.453+0000 7f0497200640 1 -- 172.21.15.69:0/2401507635 wait complete. 2024-04-24T08:14:27.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.455+0000 7f0497200640 1 Processor -- start 2024-04-24T08:14:27.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.455+0000 7f0497200640 1 -- start start 2024-04-24T08:14:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0497200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0498071e50 0x7f0498157630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0497200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0498072230 0x7f049814d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0497200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 0x7f04980b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0497200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f049806f090 con 0x7f0498162300 2024-04-24T08:14:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0497200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f049806ef10 con 0x7f0498072230 2024-04-24T08:14:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0497200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f049806f210 con 0x7f0498071e50 2024-04-24T08:14:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0496800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 0x7f04980b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0496800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 0x7f04980b4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50866/0 (socket says 172.21.15.69:50866) 2024-04-24T08:14:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0496800640 1 -- 172.21.15.69:0/4217502025 learned_addr learned my addr 172.21.15.69:0/4217502025 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0495400640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0498072230 0x7f049814d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0495e00640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0498071e50 0x7f0498157630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0496800640 1 -- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0498071e50 msgr2=0x7f0498157630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0496800640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0498071e50 0x7f0498157630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0496800640 1 -- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0498072230 msgr2=0x7f049814d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0496800640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0498072230 0x7f049814d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0496800640 1 -- 172.21.15.69:0/4217502025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0498157ea0 con 0x7f0498162300 2024-04-24T08:14:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.456+0000 7f0495400640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0498072230 0x7f049814d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.457+0000 7f0496800640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 0x7f04980b4020 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f049000c780 tx=0x7f049000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.457+0000 7f048e000640 1 -- 172.21.15.69:0/4217502025 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490004b60 con 0x7f0498162300 2024-04-24T08:14:27.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.457+0000 7f048e000640 1 -- 172.21.15.69:0/4217502025 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0490004d00 con 0x7f0498162300 2024-04-24T08:14:27.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.457+0000 7f0497200640 1 -- 172.21.15.69:0/4217502025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0498158180 con 0x7f0498162300 2024-04-24T08:14:27.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.457+0000 7f048e000640 1 -- 172.21.15.69:0/4217502025 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049000aa90 con 0x7f0498162300 2024-04-24T08:14:27.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.457+0000 7f0497200640 1 -- 172.21.15.69:0/4217502025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0498158740 con 0x7f0498162300 2024-04-24T08:14:27.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.458+0000 7f0497200640 1 -- 172.21.15.69:0/4217502025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0498075d50 con 0x7f0498162300 2024-04-24T08:14:27.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.459+0000 7f048e000640 1 -- 172.21.15.69:0/4217502025 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f049000ac30 con 0x7f0498162300 2024-04-24T08:14:27.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.460+0000 7f048e000640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f047806c8f0 0x7f047806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.460+0000 7f048e000640 1 -- 172.21.15.69:0/4217502025 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f049008bfe0 con 0x7f0498162300 2024-04-24T08:14:27.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.461+0000 7f0495e00640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f047806c8f0 0x7f047806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.462+0000 7f0495e00640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f047806c8f0 0x7f047806edb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0488008850 tx=0x7f0488008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.465+0000 7f048e000640 1 -- 172.21.15.69:0/4217502025 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0490020070 con 0x7f0498162300 2024-04-24T08:14:27.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.704+0000 7f0497200640 1 -- 172.21.15.69:0/4217502025 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0498073720 con 0x7f0498162300 2024-04-24T08:14:27.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.705+0000 7f048e000640 1 -- 172.21.15.69:0/4217502025 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+5188 (secure 0 0 0) 0x7f049005a2a0 con 0x7f0498162300 2024-04-24T08:14:27.707 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:14:27.707 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":51,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3504199143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3504199143},{"type":"v1","addr":"172.21.15.121:6837","nonce":3504199143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:13:48.244688+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6792},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6792":{"gid":6792,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":12,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6792,"qdb_cluster":[6792]},"id":5}]} 2024-04-24T08:14:27.707 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 51 2024-04-24T08:14:27.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 -- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f047806c8f0 msgr2=0x7f047806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f047806c8f0 0x7f047806edb0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f0488008850 tx=0x7f0488008040 comp rx=0 tx=0).stop 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 -- 172.21.15.69:0/4217502025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 msgr2=0x7f04980b4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 0x7f04980b4020 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f049000c780 tx=0x7f049000cc40 comp rx=0 tx=0).stop 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 -- 172.21.15.69:0/4217502025 shutdown_connections 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f047806c8f0 0x7f047806edb0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0498162300 0x7f04980b4020 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0498072230 0x7f049814d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 --2- 172.21.15.69:0/4217502025 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0498071e50 0x7f0498157630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 -- 172.21.15.69:0/4217502025 >> 172.21.15.69:0/4217502025 conn(0x7f04980d0160 msgr2=0x7f04980afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 -- 172.21.15.69:0/4217502025 shutdown_connections 2024-04-24T08:14:27.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.708+0000 7f0497200640 1 -- 172.21.15.69:0/4217502025 wait complete. 2024-04-24T08:14:27.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-24T08:14:27.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.930+0000 7f25be400640 1 Processor -- start 2024-04-24T08:14:27.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.930+0000 7f25be400640 1 -- start start 2024-04-24T08:14:27.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.931+0000 7f25be400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f25b81420a0 0x7f25b8071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.931+0000 7f25be400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25b81606e0 0x7f25b80cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.931+0000 7f25be400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25b8072450 con 0x7f25b8142480 2024-04-24T08:14:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.931+0000 7f25be400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f25b80cf1e0 con 0x7f25b81420a0 2024-04-24T08:14:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.931+0000 7f25be400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f25b8072730 con 0x7f25b81606e0 2024-04-24T08:14:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.932+0000 7f25bd000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f25b81420a0 0x7f25b8071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.932+0000 7f25bd000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f25b81420a0 0x7f25b8071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:45096/0 (socket says 172.21.15.69:45096) 2024-04-24T08:14:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.932+0000 7f25bd000640 1 -- 172.21.15.69:0/3502918583 learned_addr learned my addr 172.21.15.69:0/3502918583 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.932+0000 7f25bda00640 1 --2- 172.21.15.69:0/3502918583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25b81606e0 0x7f25b80cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.933+0000 7f25bd000640 1 -- 172.21.15.69:0/3502918583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25b81606e0 msgr2=0x7f25b80cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.933+0000 7f25bd000640 1 --2- 172.21.15.69:0/3502918583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25b81606e0 0x7f25b80cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.933+0000 7f25bd000640 1 -- 172.21.15.69:0/3502918583 >> v1:172.21.15.121:6789/0 conn(0x7f25b8142480 legacy=0x7f25b815ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.933+0000 7f25bd000640 1 -- 172.21.15.69:0/3502918583 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b8072a10 con 0x7f25b81420a0 2024-04-24T08:14:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.934+0000 7f25bd000640 1 --2- 172.21.15.69:0/3502918583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25b81420a0 0x7f25b8071d70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009870 tx=0x7f25a802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=151026cfda87ff62 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.934+0000 7f25b7e00640 1 -- 172.21.15.69:0/3502918583 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8037070 con 0x7f25b81420a0 2024-04-24T08:14:27.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.935+0000 7f25b7e00640 1 -- 172.21.15.69:0/3502918583 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a8032440 con 0x7f25b81420a0 2024-04-24T08:14:27.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.935+0000 7f25b7e00640 1 -- 172.21.15.69:0/3502918583 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8032810 con 0x7f25b81420a0 2024-04-24T08:14:27.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.935+0000 7f25be400640 1 -- 172.21.15.69:0/3502918583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25b81420a0 msgr2=0x7f25b8071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.935+0000 7f25be400640 1 --2- 172.21.15.69:0/3502918583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25b81420a0 0x7f25b8071d70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009870 tx=0x7f25a802a250 comp rx=0 tx=0).stop 2024-04-24T08:14:27.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.935+0000 7f25be400640 1 -- 172.21.15.69:0/3502918583 shutdown_connections 2024-04-24T08:14:27.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.935+0000 7f25be400640 1 --2- 172.21.15.69:0/3502918583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25b81606e0 0x7f25b80cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.935+0000 7f25be400640 1 --2- 172.21.15.69:0/3502918583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25b81420a0 0x7f25b8071d70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.935+0000 7f25be400640 1 -- 172.21.15.69:0/3502918583 >> 172.21.15.69:0/3502918583 conn(0x7f25b8067b80 msgr2=0x7f25b806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.935+0000 7f25be400640 1 -- 172.21.15.69:0/3502918583 shutdown_connections 2024-04-24T08:14:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.935+0000 7f25be400640 1 -- 172.21.15.69:0/3502918583 wait complete. 2024-04-24T08:14:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.937+0000 7f25be400640 1 Processor -- start 2024-04-24T08:14:27.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.937+0000 7f25be400640 1 -- start start 2024-04-24T08:14:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.938+0000 7f25be400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25b81420a0 0x7f25b813d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.938+0000 7f25be400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25b8142480 0x7f25b813f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.938+0000 7f25be400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b81606e0 0x7f25b8157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.938+0000 7f25be400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8142a10 con 0x7f25b81606e0 2024-04-24T08:14:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.938+0000 7f25be400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8142890 con 0x7f25b8142480 2024-04-24T08:14:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.938+0000 7f25be400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f25b80cfbb0 con 0x7f25b81420a0 2024-04-24T08:14:27.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25bda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b81606e0 0x7f25b8157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25bda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b81606e0 0x7f25b8157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50878/0 (socket says 172.21.15.69:50878) 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25bda00640 1 -- 172.21.15.69:0/422204755 learned_addr learned my addr 172.21.15.69:0/422204755 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25bd000640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25b81420a0 0x7f25b813d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25afe00640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25b8142480 0x7f25b813f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25bda00640 1 -- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25b81420a0 msgr2=0x7f25b813d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25bda00640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25b81420a0 0x7f25b813d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25bda00640 1 -- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25b8142480 msgr2=0x7f25b813f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25bda00640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25b8142480 0x7f25b813f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25bda00640 1 -- 172.21.15.69:0/422204755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b8140060 con 0x7f25b81606e0 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.939+0000 7f25bda00640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b81606e0 0x7f25b8157010 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f25b000efc0 tx=0x7f25b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.940+0000 7f25b5600640 1 -- 172.21.15.69:0/422204755 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0018020 con 0x7f25b81606e0 2024-04-24T08:14:27.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.940+0000 7f25b5600640 1 -- 172.21.15.69:0/422204755 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25b0014480 con 0x7f25b81606e0 2024-04-24T08:14:27.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.940+0000 7f25be400640 1 -- 172.21.15.69:0/422204755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b8140340 con 0x7f25b81606e0 2024-04-24T08:14:27.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.940+0000 7f25b5600640 1 -- 172.21.15.69:0/422204755 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0009e60 con 0x7f25b81606e0 2024-04-24T08:14:27.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.940+0000 7f25be400640 1 -- 172.21.15.69:0/422204755 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b8140900 con 0x7f25b81606e0 2024-04-24T08:14:27.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.942+0000 7f25b5600640 1 -- 172.21.15.69:0/422204755 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f25b0005d10 con 0x7f25b81606e0 2024-04-24T08:14:27.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.943+0000 7f25b5600640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f259406c8f0 0x7f259406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.943+0000 7f25b5600640 1 -- 172.21.15.69:0/422204755 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f25b008caa0 con 0x7f25b81606e0 2024-04-24T08:14:27.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.944+0000 7f25bd000640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f259406c8f0 0x7f259406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.944+0000 7f25c2992740 1 Processor -- start 2024-04-24T08:14:27.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.944+0000 7f25c2992740 1 -- start start 2024-04-24T08:14:27.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.945+0000 7f25bd000640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f259406c8f0 0x7f259406edb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f25a8034000 tx=0x7f25a802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.945+0000 7f25c2992740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232237220 0x557232237600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.945+0000 7f25c2992740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232254ac0 0x557232255010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.945+0000 7f25c2992740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55723220cf00 0x557232253180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.945+0000 7f25c2992740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557232171190 con 0x557232237220 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.945+0000 7f25c2992740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557232117030 con 0x55723220cf00 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.945+0000 7f25c2992740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55723216d150 con 0x557232254ac0 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.945+0000 7f25bd000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232237220 0x557232237600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.945+0000 7f25bd000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232237220 0x557232237600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50882/0 (socket says 172.21.15.69:50882) 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.945+0000 7f25bd000640 1 -- 172.21.15.69:0/2582192266 learned_addr learned my addr 172.21.15.69:0/2582192266 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25afe00640 1 --2- 172.21.15.69:0/2582192266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232254ac0 0x557232255010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25bda00640 1 --2- 172.21.15.69:0/2582192266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55723220cf00 0x557232253180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25bd000640 1 -- 172.21.15.69:0/2582192266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232254ac0 msgr2=0x557232255010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25bd000640 1 --2- 172.21.15.69:0/2582192266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232254ac0 0x557232255010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25bd000640 1 -- 172.21.15.69:0/2582192266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55723220cf00 msgr2=0x557232253180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25bd000640 1 --2- 172.21.15.69:0/2582192266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55723220cf00 0x557232253180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25bd000640 1 -- 172.21.15.69:0/2582192266 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572320ffb60 con 0x557232237220 2024-04-24T08:14:27.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25bd000640 1 --2- 172.21.15.69:0/2582192266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232237220 0x557232237600 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f25a802a290 tx=0x7f25a802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8932e1f9325a1540 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25ad600640 1 -- 172.21.15.69:0/2582192266 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a8037040 con 0x557232237220 2024-04-24T08:14:27.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25ad600640 1 -- 172.21.15.69:0/2582192266 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a803e660 con 0x557232237220 2024-04-24T08:14:27.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.946+0000 7f25ad600640 1 -- 172.21.15.69:0/2582192266 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a803ea10 con 0x557232237220 2024-04-24T08:14:27.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.947+0000 7f25c2992740 1 -- 172.21.15.69:0/2582192266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232237220 msgr2=0x557232237600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.947+0000 7f25c2992740 1 --2- 172.21.15.69:0/2582192266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232237220 0x557232237600 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f25a802a290 tx=0x7f25a802c000 comp rx=0 tx=0).stop 2024-04-24T08:14:27.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.947+0000 7f25c2992740 1 -- 172.21.15.69:0/2582192266 shutdown_connections 2024-04-24T08:14:27.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.947+0000 7f25c2992740 1 --2- 172.21.15.69:0/2582192266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55723220cf00 0x557232253180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.947+0000 7f25c2992740 1 --2- 172.21.15.69:0/2582192266 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232254ac0 0x557232255010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.947+0000 7f25c2992740 1 --2- 172.21.15.69:0/2582192266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232237220 0x557232237600 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.947+0000 7f25c2992740 1 -- 172.21.15.69:0/2582192266 >> 172.21.15.69:0/2582192266 conn(0x557232209830 msgr2=0x557232252030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.947+0000 7f25c2992740 1 -- 172.21.15.69:0/2582192266 shutdown_connections 2024-04-24T08:14:27.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.947+0000 7f25c2992740 1 -- 172.21.15.69:0/2582192266 wait complete. 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.948+0000 7f25c2992740 1 -- start start 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.948+0000 7f25c2992740 1 Processor -- start 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.949+0000 7f25c2992740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232237220 0x55723222dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.949+0000 7f25c2992740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 0x55723224a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.949+0000 7f25c2992740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723224eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.949+0000 7f25c2992740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55723220dca0 con 0x55723224aa40 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.949+0000 7f25c2992740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55723212a500 con 0x557232254ac0 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.949+0000 7f25c2992740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557232207760 con 0x557232237220 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.949+0000 7f25c2992740 10 client.0 _ll_register_callbacks cb 0x557232167270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25bda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723224eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25bda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723224eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50896/0 (socket says 172.21.15.69:50896) 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25bda00640 1 -- 172.21.15.69:0/1308337874 learned_addr learned my addr 172.21.15.69:0/1308337874 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25afe00640 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 0x55723224a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25bd000640 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232237220 0x55723222dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25bda00640 1 -- 172.21.15.69:0/1308337874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232237220 msgr2=0x55723222dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25bda00640 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232237220 0x55723222dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25bda00640 1 -- 172.21.15.69:0/1308337874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 msgr2=0x55723224a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25bda00640 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 0x55723224a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25bda00640 1 -- 172.21.15.69:0/1308337874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557231f4f7c0 con 0x55723224aa40 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25afe00640 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 0x55723224a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25bda00640 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723224eb50 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f25b0056a30 tx=0x7f25b005ae10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2945697273090f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25a6000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.950+0000 7f25a6000640 1 -- 172.21.15.69:0/1308337874 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0059a20 con 0x55723224aa40 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25c2992740 1 -- 172.21.15.69:0/1308337874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25a803e660 con 0x55723224aa40 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25a6000640 1 -- 172.21.15.69:0/1308337874 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f25b0059bc0 con 0x55723224aa40 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25a6000640 1 -- 172.21.15.69:0/1308337874 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f25b0077050 con 0x55723224aa40 2024-04-24T08:14:27.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25c2992740 10 client.7269 fetch_fsmap learned FSMap version 51 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25c2992740 1 -- 172.21.15.69:0/1308337874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x557231faff50 con 0x55723224aa40 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25a6000640 1 -- 172.21.15.69:0/1308337874 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a803e660 con 0x55723224aa40 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25a6000640 1 -- 172.21.15.69:0/1308337874 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f25b0056320 con 0x55723224aa40 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25c2992740 10 client.7269 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25c2992740 10 client.7269 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25c2992740 20 client.7269 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25c2992740 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5572322515d0 0x5572322519b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25c2992740 4 client.7269 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25c2992740 1 -- 172.21.15.69:0/1308337874 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55723217ebb0 con 0x5572322515d0 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.951+0000 7f25bd000640 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5572322515d0 0x5572322519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.952+0000 7f25bd000640 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5572322515d0 0x5572322519b0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7cf13eef911c968 server_cookie=3703a2254306cb12 in_seq=0 out_seq=0 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.952+0000 7f25a6000640 10 client.7269 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:27.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25a6000640 1 -- 172.21.15.69:0/1308337874 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55723217ebb0 con 0x5572322515d0 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25a6000640 10 client.7269 handle_command_reply: tid=0 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 client.7269 shutdown 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 -- 172.21.15.69:0/1308337874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 msgr2=0x55723224eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723224eb50 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f25b0056a30 tx=0x7f25b005ae10 comp rx=0 tx=0).stop 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 -- 172.21.15.69:0/1308337874 shutdown_connections 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5572322515d0 0x5572322519b0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723224eb50 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 0x55723224a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25a6000640 0 client.7269 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 --2- 172.21.15.69:0/1308337874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232237220 0x55723222dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 -- 172.21.15.69:0/1308337874 >> 172.21.15.69:0/1308337874 conn(0x55723220cf00 msgr2=0x5572322555b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 -- 172.21.15.69:0/1308337874 shutdown_connections 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.953+0000 7f25c2992740 1 -- 172.21.15.69:0/1308337874 wait complete. 2024-04-24T08:14:27.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.954+0000 7f25c2992740 20 client.7269 trim_cache size 0 max 16384 2024-04-24T08:14:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.968+0000 7f25c2992740 1 Processor -- start 2024-04-24T08:14:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.968+0000 7f25c2992740 1 -- start start 2024-04-24T08:14:27.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.969+0000 7f25c2992740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232237220 0x55723222fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.969+0000 7f25c2992740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55723224aa40 0x55723224b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.969+0000 7f25c2992740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232254ac0 0x5572322519b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.969+0000 7f25c2992740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25b81f63e0 con 0x557232254ac0 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.969+0000 7f25c2992740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f25b81f5290 con 0x557232237220 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.969+0000 7f25c2992740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f25b81f5d90 con 0x55723224aa40 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.969+0000 7f25bda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232254ac0 0x5572322519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.969+0000 7f25bda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232254ac0 0x5572322519b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50908/0 (socket says 172.21.15.69:50908) 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.969+0000 7f25bda00640 1 -- 172.21.15.69:0/429332461 learned_addr learned my addr 172.21.15.69:0/429332461 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.970+0000 7f25bd000640 1 --2- 172.21.15.69:0/429332461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232237220 0x55723222fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.970+0000 7f25bda00640 1 -- 172.21.15.69:0/429332461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55723224aa40 msgr2=0x55723224b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.970+0000 7f25bda00640 1 --2- 172.21.15.69:0/429332461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55723224aa40 0x55723224b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.970+0000 7f25bda00640 1 -- 172.21.15.69:0/429332461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232237220 msgr2=0x55723222fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.970+0000 7f25bda00640 1 --2- 172.21.15.69:0/429332461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232237220 0x55723222fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.970+0000 7f25bda00640 1 -- 172.21.15.69:0/429332461 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557231e430b0 con 0x557232254ac0 2024-04-24T08:14:27.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.970+0000 7f25bda00640 1 --2- 172.21.15.69:0/429332461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232254ac0 0x5572322519b0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f25b000c990 tx=0x7f25b0016070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbe3fa2a7925486 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.970+0000 7f25acc00640 1 -- 172.21.15.69:0/429332461 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b007d070 con 0x557232254ac0 2024-04-24T08:14:27.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.970+0000 7f25acc00640 1 -- 172.21.15.69:0/429332461 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25b0059a20 con 0x557232254ac0 2024-04-24T08:14:27.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.970+0000 7f25acc00640 1 -- 172.21.15.69:0/429332461 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0059490 con 0x557232254ac0 2024-04-24T08:14:27.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.971+0000 7f25c2992740 1 -- 172.21.15.69:0/429332461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232254ac0 msgr2=0x5572322519b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.971+0000 7f25c2992740 1 --2- 172.21.15.69:0/429332461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232254ac0 0x5572322519b0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f25b000c990 tx=0x7f25b0016070 comp rx=0 tx=0).stop 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.971+0000 7f25c2992740 1 -- 172.21.15.69:0/429332461 shutdown_connections 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.971+0000 7f25c2992740 1 --2- 172.21.15.69:0/429332461 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557232254ac0 0x5572322519b0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.971+0000 7f25c2992740 1 --2- 172.21.15.69:0/429332461 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55723224aa40 0x55723224b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.971+0000 7f25c2992740 1 --2- 172.21.15.69:0/429332461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232237220 0x55723222fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.971+0000 7f25c2992740 1 -- 172.21.15.69:0/429332461 >> 172.21.15.69:0/429332461 conn(0x55723220cf00 msgr2=0x55723224a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.971+0000 7f25c2992740 1 -- 172.21.15.69:0/429332461 shutdown_connections 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.971+0000 7f25c2992740 1 -- 172.21.15.69:0/429332461 wait complete. 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.972+0000 7f25c2992740 1 -- start start 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.972+0000 7f25c2992740 1 Processor -- start 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.973+0000 7f25c2992740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232237220 0x5572322cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.973+0000 7f25c2992740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723222e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.973+0000 7f25c2992740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 0x55723222ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.973+0000 7f25c2992740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55723222e280 con 0x55723224aa40 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.973+0000 7f25c2992740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55723224cec0 con 0x557232254ac0 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.973+0000 7f25c2992740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55723224db70 con 0x557232237220 2024-04-24T08:14:27.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.973+0000 7f25c2992740 10 client.0 _ll_register_callbacks cb 0x7f25b0056870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.973+0000 7f25afe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723222e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.973+0000 7f25afe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723222e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50924/0 (socket says 172.21.15.69:50924) 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.973+0000 7f25afe00640 1 -- 172.21.15.69:0/823415562 learned_addr learned my addr 172.21.15.69:0/823415562 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25bda00640 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 0x55723222ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25bd000640 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232237220 0x5572322cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25afe00640 1 -- 172.21.15.69:0/823415562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232237220 msgr2=0x5572322cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25afe00640 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232237220 0x5572322cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25afe00640 1 -- 172.21.15.69:0/823415562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 msgr2=0x55723222ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25afe00640 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 0x55723222ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25afe00640 1 -- 172.21.15.69:0/823415562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557232247a60 con 0x55723224aa40 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25afe00640 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723222e7c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f25b806f640 tx=0x7f25a00014f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deb612c78c42c4b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:27.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25a6000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25a6000640 1 -- 172.21.15.69:0/823415562 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a0010ea0 con 0x55723224aa40 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25a6000640 1 -- 172.21.15.69:0/823415562 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f25a00052d0 con 0x55723224aa40 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25c2992740 1 -- 172.21.15.69:0/823415562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25b0059a20 con 0x55723224aa40 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25a6000640 1 -- 172.21.15.69:0/823415562 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f25a0021050 con 0x55723224aa40 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.974+0000 7f25c2992740 10 client.7275 fetch_fsmap learned FSMap version 51 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.975+0000 7f25c2992740 1 -- 172.21.15.69:0/823415562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x5572322cc6b0 con 0x55723224aa40 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.975+0000 7f25a6000640 1 -- 172.21.15.69:0/823415562 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25b0059a20 con 0x55723224aa40 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.975+0000 7f25a6000640 1 -- 172.21.15.69:0/823415562 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7f25a0007020 con 0x55723224aa40 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.975+0000 7f25c2992740 10 client.7275 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.975+0000 7f25c2992740 10 client.7275 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.975+0000 7f25c2992740 20 client.7275 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.975+0000 7f25c2992740 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55723220ac40 0x55723220b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.975+0000 7f25c2992740 4 client.7275 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.975+0000 7f25c2992740 1 -- 172.21.15.69:0/823415562 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x557232178a20 con 0x55723220ac40 2024-04-24T08:14:27.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.975+0000 7f25bd000640 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55723220ac40 0x55723220b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25bd000640 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55723220ac40 0x55723220b020 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=417048995f9c90f2 server_cookie=561d7bb8fad6cd68 in_seq=0 out_seq=0 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25a6000640 10 client.7275 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25a6000640 1 -- 172.21.15.69:0/823415562 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+237 (crc 0 0 0) 0x557232178a20 con 0x55723220ac40 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25a6000640 10 client.7275 handle_command_reply: tid=0 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 client.7275 shutdown 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 -- 172.21.15.69:0/823415562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 msgr2=0x55723222e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723222e7c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f25b806f640 tx=0x7f25a00014f0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 -- 172.21.15.69:0/823415562 shutdown_connections 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55723220ac40 0x55723220b020 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55723224aa40 0x55723222e7c0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557232254ac0 0x55723222ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 --2- 172.21.15.69:0/823415562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557232237220 0x5572322cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 -- 172.21.15.69:0/823415562 >> 172.21.15.69:0/823415562 conn(0x55723220cf00 msgr2=0x55723220bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25a6000640 0 client.7275 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 -- 172.21.15.69:0/823415562 shutdown_connections 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 1 -- 172.21.15.69:0/823415562 wait complete. 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.976+0000 7f25c2992740 20 client.7275 trim_cache size 0 max 16384 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "dentry", 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 3506197013, 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511627776, 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stdout: "frag": "*", 2024-04-24T08:14:27.982 INFO:teuthology.orchestra.run.smithi069.stdout: "dname": "file_to_be_damaged", 2024-04-24T08:14:27.983 INFO:teuthology.orchestra.run.smithi069.stdout: "snap_id": "head", 2024-04-24T08:14:27.983 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/subdir/file_to_be_damaged" 2024-04-24T08:14:27.983 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:14:27.983 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:14:27.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 -- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f259406c8f0 msgr2=0x7f259406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f259406c8f0 0x7f259406edb0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f25a8034000 tx=0x7f25a802e000 comp rx=0 tx=0).stop 2024-04-24T08:14:27.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 -- 172.21.15.69:0/422204755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b81606e0 msgr2=0x7f25b8157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:27.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b81606e0 0x7f25b8157010 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f25b000efc0 tx=0x7f25b000c490 comp rx=0 tx=0).stop 2024-04-24T08:14:27.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 -- 172.21.15.69:0/422204755 shutdown_connections 2024-04-24T08:14:27.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f259406c8f0 0x7f259406edb0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25b81606e0 0x7f25b8157010 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25b8142480 0x7f25b813f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 --2- 172.21.15.69:0/422204755 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f25b81420a0 0x7f25b813d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:27.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 -- 172.21.15.69:0/422204755 >> 172.21.15.69:0/422204755 conn(0x7f25b8067b80 msgr2=0x7f25b80cff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:27.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 -- 172.21.15.69:0/422204755 shutdown_connections 2024-04-24T08:14:27.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:27.977+0000 7f25be400640 1 -- 172.21.15.69:0/422204755 wait complete. 2024-04-24T08:14:27.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T08:14:28.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.192+0000 7fa2e9c00640 1 Processor -- start 2024-04-24T08:14:28.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.192+0000 7fa2e9c00640 1 -- start start 2024-04-24T08:14:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.194+0000 7fa2e9c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e40d0cf0 0x7fa2e406e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.194+0000 7fa2e9c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa2e415ca30 0x7fa2e415cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.194+0000 7fa2e9c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2e40ad400 con 0x7fa2e4156cc0 2024-04-24T08:14:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.194+0000 7fa2e9c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa2e415d420 con 0x7fa2e415ca30 2024-04-24T08:14:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.194+0000 7fa2e9c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e415d680 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.194+0000 7fa2e3e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa2e4156cc0 0x7fa2e40d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60560/0 (socket says 172.21.15.69:60560) 2024-04-24T08:14:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.194+0000 7fa2e3e00640 1 -- 172.21.15.69:0/3335775996 learned_addr learned my addr 172.21.15.69:0/3335775996 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.194+0000 7fa2e9200640 1 --2- 172.21.15.69:0/3335775996 >> v2:172.21.15.69:3300/0 conn(0x7fa2e415ca30 0x7fa2e415cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.195+0000 7fa2e3400640 1 --2- 172.21.15.69:0/3335775996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e40d0cf0 0x7fa2e406e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.195+0000 7fa2e2a00640 1 -- 172.21.15.69:0/3335775996 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778132550 0 0) 0x7fa2d4001110 con 0x7fa2e4156cc0 2024-04-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.195+0000 7fa2e2a00640 1 -- 172.21.15.69:0/3335775996 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2c8003520 con 0x7fa2e4156cc0 2024-04-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.195+0000 7fa2e9200640 1 -- 172.21.15.69:0/3335775996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e40d0cf0 msgr2=0x7fa2e406e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.195+0000 7fa2e9200640 1 --2- 172.21.15.69:0/3335775996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e40d0cf0 0x7fa2e406e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.195+0000 7fa2e9200640 1 -- 172.21.15.69:0/3335775996 >> v1:172.21.15.69:6789/0 conn(0x7fa2e4156cc0 legacy=0x7fa2e40d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.195+0000 7fa2e9200640 1 -- 172.21.15.69:0/3335775996 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e415d960 con 0x7fa2e415ca30 2024-04-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.196+0000 7fa2e9200640 1 --2- 172.21.15.69:0/3335775996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e415ca30 0x7fa2e415cee0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009d90 tx=0x7fa2dc02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7069bd0e9a85a0e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.196+0000 7fa2e2a00640 1 -- 172.21.15.69:0/3335775996 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc031600 con 0x7fa2e415ca30 2024-04-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.196+0000 7fa2e2a00640 1 -- 172.21.15.69:0/3335775996 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2dc0317a0 con 0x7fa2e415ca30 2024-04-24T08:14:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.196+0000 7fa2e2a00640 1 -- 172.21.15.69:0/3335775996 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc031b50 con 0x7fa2e415ca30 2024-04-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.197+0000 7fa2e9c00640 1 -- 172.21.15.69:0/3335775996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e415ca30 msgr2=0x7fa2e415cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.197+0000 7fa2e9c00640 1 --2- 172.21.15.69:0/3335775996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e415ca30 0x7fa2e415cee0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009d90 tx=0x7fa2dc02a5d0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.197+0000 7fa2e9c00640 1 -- 172.21.15.69:0/3335775996 shutdown_connections 2024-04-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.197+0000 7fa2e9c00640 1 --2- 172.21.15.69:0/3335775996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e415ca30 0x7fa2e415cee0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.197+0000 7fa2e9c00640 1 --2- 172.21.15.69:0/3335775996 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e40d0cf0 0x7fa2e406e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.197+0000 7fa2e9c00640 1 -- 172.21.15.69:0/3335775996 >> 172.21.15.69:0/3335775996 conn(0x7fa2e406be40 msgr2=0x7fa2e406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.197+0000 7fa2e9c00640 1 -- 172.21.15.69:0/3335775996 shutdown_connections 2024-04-24T08:14:28.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.197+0000 7fa2e9c00640 1 -- 172.21.15.69:0/3335775996 wait complete. 2024-04-24T08:14:28.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.199+0000 7fa2e9c00640 1 Processor -- start 2024-04-24T08:14:28.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.199+0000 7fa2e9c00640 1 -- start start 2024-04-24T08:14:28.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.200+0000 7fa2e9c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e40d0cf0 0x7fa2e408bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.200+0000 7fa2e9c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e4156cc0 0x7fa2e41590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.200+0000 7fa2e9c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e415ca30 0x7fa2e415f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.200+0000 7fa2e9c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e406e160 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.200+0000 7fa2e9c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa2e406de80 con 0x7fa2e4156cc0 2024-04-24T08:14:28.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.200+0000 7fa2e9c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa2e4071c70 con 0x7fa2e415ca30 2024-04-24T08:14:28.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.200+0000 7fa2e3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e40d0cf0 0x7fa2e408bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.200+0000 7fa2e3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e40d0cf0 0x7fa2e408bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50944/0 (socket says 172.21.15.69:50944) 2024-04-24T08:14:28.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.200+0000 7fa2e3e00640 1 -- 172.21.15.69:0/615436026 learned_addr learned my addr 172.21.15.69:0/615436026 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e9200640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e415ca30 0x7fa2e415f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e3400640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e4156cc0 0x7fa2e41590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e3e00640 1 -- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e415ca30 msgr2=0x7fa2e415f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e3e00640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e415ca30 0x7fa2e415f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e3e00640 1 -- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e4156cc0 msgr2=0x7fa2e41590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e3e00640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e4156cc0 0x7fa2e41590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e3e00640 1 -- 172.21.15.69:0/615436026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2e415efe0 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e3400640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e4156cc0 0x7fa2e41590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:28.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e3e00640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e40d0cf0 0x7fa2e408bf60 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400c730 tx=0x7fa2d400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2dbe00640 1 -- 172.21.15.69:0/615436026 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4017070 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2dbe00640 1 -- 172.21.15.69:0/615436026 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2d4005ad0 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2dbe00640 1 -- 172.21.15.69:0/615436026 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2d4007e20 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e9c00640 1 -- 172.21.15.69:0/615436026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2e41f2840 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.201+0000 7fa2e9c00640 1 -- 172.21.15.69:0/615436026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2e41f2e00 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.202+0000 7fa2e9c00640 1 -- 172.21.15.69:0/615436026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2e4075d50 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.203+0000 7fa2dbe00640 1 -- 172.21.15.69:0/615436026 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa2d401a070 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.204+0000 7fa2dbe00640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2c406c8f0 0x7fa2c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.204+0000 7fa2dbe00640 1 -- 172.21.15.69:0/615436026 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fa2d408aed0 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.205+0000 7fa2e3400640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2c406c8f0 0x7fa2c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.206+0000 7fa2e3400640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2c406c8f0 0x7fa2c406edb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40b0010 tx=0x7fa2d0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.209+0000 7fa2dbe00640 1 -- 172.21.15.69:0/615436026 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa2e415efe0 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.444+0000 7fa2e9c00640 1 -- 172.21.15.69:0/615436026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa2e406f5b0 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.447+0000 7fa2dbe00640 1 -- 172.21.15.69:0/615436026 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+164 (secure 0 0 0) 0x7fa2e40ad400 con 0x7fa2e40d0cf0 2024-04-24T08:14:28.448 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:14:28.448 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-04-24T08:14:28.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.449+0000 7fa2e9c00640 1 -- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2c406c8f0 msgr2=0x7fa2c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.449+0000 7fa2e9c00640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2c406c8f0 0x7fa2c406edb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40b0010 tx=0x7fa2d0008040 comp rx=0 tx=0).stop 2024-04-24T08:14:28.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.450+0000 7fa2e9c00640 1 -- 172.21.15.69:0/615436026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e40d0cf0 msgr2=0x7fa2e408bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.450+0000 7fa2e9c00640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e40d0cf0 0x7fa2e408bf60 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fa2d400c730 tx=0x7fa2d400cbf0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.450+0000 7fa2e9c00640 1 -- 172.21.15.69:0/615436026 shutdown_connections 2024-04-24T08:14:28.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.450+0000 7fa2e9c00640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2c406c8f0 0x7fa2c406edb0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.450+0000 7fa2e9c00640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa2e415ca30 0x7fa2e415f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.450+0000 7fa2e9c00640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2e4156cc0 0x7fa2e41590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.450+0000 7fa2e9c00640 1 --2- 172.21.15.69:0/615436026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2e40d0cf0 0x7fa2e408bf60 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.450+0000 7fa2e9c00640 1 -- 172.21.15.69:0/615436026 >> 172.21.15.69:0/615436026 conn(0x7fa2e406be40 msgr2=0x7fa2e40aee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.450+0000 7fa2e9c00640 1 -- 172.21.15.69:0/615436026 shutdown_connections 2024-04-24T08:14:28.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.450+0000 7fa2e9c00640 1 -- 172.21.15.69:0/615436026 wait complete. 2024-04-24T08:14:28.467 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-24T08:14:28.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:14:28.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub start /subdir repair,recursive 2024-04-24T08:14:28.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.668+0000 7fc26f400640 1 Processor -- start 2024-04-24T08:14:28.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.668+0000 7fc26f400640 1 -- start start 2024-04-24T08:14:28.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.669+0000 7fc26f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2680b32d0 0x7fc26815f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.669+0000 7fc26f400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc26815fad0 0x7fc2680b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.669+0000 7fc26f400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc26815edb0 con 0x7fc26806f5e0 2024-04-24T08:14:28.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.669+0000 7fc26f400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc26806f9c0 con 0x7fc2680b32d0 2024-04-24T08:14:28.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.669+0000 7fc26f400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc268162bb0 con 0x7fc26815fad0 2024-04-24T08:14:28.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.670+0000 7fc267e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2680b32d0 0x7fc26815f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.670+0000 7fc267e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc2680b32d0 0x7fc26815f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50946/0 (socket says 172.21.15.69:50946) 2024-04-24T08:14:28.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.670+0000 7fc267e00640 1 -- 172.21.15.69:0/1172876387 learned_addr learned my addr 172.21.15.69:0/1172876387 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.670+0000 7fc26d400640 1 --2- 172.21.15.69:0/1172876387 >> v2:172.21.15.121:3300/0 conn(0x7fc26815fad0 0x7fc2680b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.670+0000 7fc267e00640 1 -- 172.21.15.69:0/1172876387 >> v2:172.21.15.121:3300/0 conn(0x7fc26815fad0 msgr2=0x7fc2680b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.670+0000 7fc267e00640 1 --2- 172.21.15.69:0/1172876387 >> v2:172.21.15.121:3300/0 conn(0x7fc26815fad0 0x7fc2680b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.670+0000 7fc267e00640 1 -- 172.21.15.69:0/1172876387 >> v1:172.21.15.121:6789/0 conn(0x7fc26806f5e0 legacy=0x7fc2681623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.670+0000 7fc267e00640 1 -- 172.21.15.69:0/1172876387 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc268162e90 con 0x7fc2680b32d0 2024-04-24T08:14:28.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.672+0000 7fc267e00640 1 --2- 172.21.15.69:0/1172876387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2680b32d0 0x7fc26815f590 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc26806fb40 tx=0x7fc25402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ef9b8413da794 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.672+0000 7fc267400640 1 -- 172.21.15.69:0/1172876387 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254037070 con 0x7fc2680b32d0 2024-04-24T08:14:28.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.672+0000 7fc267400640 1 -- 172.21.15.69:0/1172876387 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc254032440 con 0x7fc2680b32d0 2024-04-24T08:14:28.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.672+0000 7fc267400640 1 -- 172.21.15.69:0/1172876387 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254032810 con 0x7fc2680b32d0 2024-04-24T08:14:28.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.673+0000 7fc26f400640 1 -- 172.21.15.69:0/1172876387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2680b32d0 msgr2=0x7fc26815f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.673+0000 7fc26f400640 1 --2- 172.21.15.69:0/1172876387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2680b32d0 0x7fc26815f590 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc26806fb40 tx=0x7fc25402a210 comp rx=0 tx=0).stop 2024-04-24T08:14:28.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.673+0000 7fc26f400640 1 -- 172.21.15.69:0/1172876387 shutdown_connections 2024-04-24T08:14:28.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.673+0000 7fc26f400640 1 --2- 172.21.15.69:0/1172876387 >> v2:172.21.15.121:3300/0 conn(0x7fc26815fad0 0x7fc2680b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.673+0000 7fc26f400640 1 --2- 172.21.15.69:0/1172876387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc2680b32d0 0x7fc26815f590 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.673+0000 7fc26f400640 1 -- 172.21.15.69:0/1172876387 >> 172.21.15.69:0/1172876387 conn(0x7fc2680b2380 msgr2=0x7fc2680b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.673+0000 7fc26f400640 1 -- 172.21.15.69:0/1172876387 shutdown_connections 2024-04-24T08:14:28.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.673+0000 7fc26f400640 1 -- 172.21.15.69:0/1172876387 wait complete. 2024-04-24T08:14:28.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.675+0000 7fc26f400640 1 Processor -- start 2024-04-24T08:14:28.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.675+0000 7fc26f400640 1 -- start start 2024-04-24T08:14:28.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc26f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc26806f5e0 0x7fc26815a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc26f400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680b32d0 0x7fc2681e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc26f400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2681e8720 0x7fc26815ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc26f400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc2680b4b30 con 0x7fc26806f5e0 2024-04-24T08:14:28.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc26f400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc26815f070 con 0x7fc2680b32d0 2024-04-24T08:14:28.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc26f400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc2680b4cb0 con 0x7fc2681e8720 2024-04-24T08:14:28.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc26ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc26806f5e0 0x7fc26815a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc26ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc26806f5e0 0x7fc26815a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50952/0 (socket says 172.21.15.69:50952) 2024-04-24T08:14:28.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc26ca00640 1 -- 172.21.15.69:0/1536754280 learned_addr learned my addr 172.21.15.69:0/1536754280 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc26d400640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2681e8720 0x7fc26815ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.676+0000 7fc267e00640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680b32d0 0x7fc2681e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc26ca00640 1 -- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2681e8720 msgr2=0x7fc26815ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc26ca00640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2681e8720 0x7fc26815ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc26ca00640 1 -- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680b32d0 msgr2=0x7fc2681e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc26ca00640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680b32d0 0x7fc2681e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc26ca00640 1 -- 172.21.15.69:0/1536754280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2681e8bd0 con 0x7fc26806f5e0 2024-04-24T08:14:28.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc267e00640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680b32d0 0x7fc2681e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:28.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc26ca00640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc26806f5e0 0x7fc26815a240 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc25c00c460 tx=0x7fc25c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc264c00640 1 -- 172.21.15.69:0/1536754280 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c016020 con 0x7fc26806f5e0 2024-04-24T08:14:28.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc264c00640 1 -- 172.21.15.69:0/1536754280 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc25c012870 con 0x7fc26806f5e0 2024-04-24T08:14:28.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc264c00640 1 -- 172.21.15.69:0/1536754280 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c012c60 con 0x7fc26806f5e0 2024-04-24T08:14:28.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc26f400640 1 -- 172.21.15.69:0/1536754280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc26815ba70 con 0x7fc26806f5e0 2024-04-24T08:14:28.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.677+0000 7fc26f400640 1 -- 172.21.15.69:0/1536754280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc26815c030 con 0x7fc26806f5e0 2024-04-24T08:14:28.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.680+0000 7fc264c00640 1 -- 172.21.15.69:0/1536754280 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc25c019020 con 0x7fc26806f5e0 2024-04-24T08:14:28.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.681+0000 7fc264c00640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc24806c920 0x7fc24806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.681+0000 7fc264c00640 1 -- 172.21.15.69:0/1536754280 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fc25c08b920 con 0x7fc26806f5e0 2024-04-24T08:14:28.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.681+0000 7fc2723dd740 1 Processor -- start 2024-04-24T08:14:28.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.681+0000 7fc2723dd740 1 -- start start 2024-04-24T08:14:28.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.681+0000 7fc267e00640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc24806c920 0x7fc24806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.682+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880f3c10 0x5653880a65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.682+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880d6d30 0x5653880d60b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.682+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880d65f0 0x5653880ab400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.682+0000 7fc2723dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x565388010070 con 0x5653880f3c10 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.682+0000 7fc2723dd740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x565387fb6580 con 0x5653880d65f0 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.682+0000 7fc2723dd740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56538800c030 con 0x5653880d6d30 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.682+0000 7fc26ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880f3c10 0x5653880a65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.682+0000 7fc26ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880f3c10 0x5653880a65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50954/0 (socket says 172.21.15.69:50954) 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.682+0000 7fc26ca00640 1 -- 172.21.15.69:0/574541722 learned_addr learned my addr 172.21.15.69:0/574541722 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.683+0000 7fc267e00640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc24806c920 0x7fc24806ede0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc254034000 tx=0x7fc25402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.683+0000 7fc26ca00640 1 -- 172.21.15.69:0/574541722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880d6d30 msgr2=0x5653880d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.683+0000 7fc26ca00640 1 --2- 172.21.15.69:0/574541722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880d6d30 0x5653880d60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.683+0000 7fc26ca00640 1 -- 172.21.15.69:0/574541722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880d65f0 msgr2=0x5653880ab400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.683+0000 7fc26ca00640 1 --2- 172.21.15.69:0/574541722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880d65f0 0x5653880ab400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.683+0000 7fc26ca00640 1 -- 172.21.15.69:0/574541722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565387deaa50 con 0x5653880f3c10 2024-04-24T08:14:28.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.683+0000 7fc26ca00640 1 --2- 172.21.15.69:0/574541722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880f3c10 0x5653880a65a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0559d0 tx=0x7fc25c059000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f926d498e4b196 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.683+0000 7fc259200640 1 -- 172.21.15.69:0/574541722 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c088070 con 0x5653880f3c10 2024-04-24T08:14:28.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.683+0000 7fc259200640 1 -- 172.21.15.69:0/574541722 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc25c057870 con 0x5653880f3c10 2024-04-24T08:14:28.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.683+0000 7fc259200640 1 -- 172.21.15.69:0/574541722 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c071070 con 0x5653880f3c10 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.684+0000 7fc2723dd740 1 -- 172.21.15.69:0/574541722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880f3c10 msgr2=0x5653880a65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.684+0000 7fc2723dd740 1 --2- 172.21.15.69:0/574541722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880f3c10 0x5653880a65a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc25c0559d0 tx=0x7fc25c059000 comp rx=0 tx=0).stop 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.684+0000 7fc2723dd740 1 -- 172.21.15.69:0/574541722 shutdown_connections 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.684+0000 7fc2723dd740 1 --2- 172.21.15.69:0/574541722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880d65f0 0x5653880ab400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.684+0000 7fc2723dd740 1 --2- 172.21.15.69:0/574541722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880d6d30 0x5653880d60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.684+0000 7fc2723dd740 1 --2- 172.21.15.69:0/574541722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880f3c10 0x5653880a65a0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.684+0000 7fc2723dd740 1 -- 172.21.15.69:0/574541722 >> 172.21.15.69:0/574541722 conn(0x5653880a8810 msgr2=0x5653880f10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.684+0000 7fc2723dd740 1 -- 172.21.15.69:0/574541722 shutdown_connections 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.684+0000 7fc2723dd740 1 -- 172.21.15.69:0/574541722 wait complete. 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.685+0000 7fc2723dd740 1 -- start start 2024-04-24T08:14:28.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.685+0000 7fc2723dd740 1 Processor -- start 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880d9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 0x5653880f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880c4310 0x5653880f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc2723dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5653880aa9c0 con 0x5653880d6d30 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc2723dd740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x565387fc9860 con 0x5653880c4310 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc2723dd740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5653880a9db0 con 0x5653880f3c10 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc2723dd740 10 client.0 _ll_register_callbacks cb 0x565387fb7ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc26ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880d9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc26ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880d9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50970/0 (socket says 172.21.15.69:50970) 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc26ca00640 1 -- 172.21.15.69:0/452461685 learned_addr learned my addr 172.21.15.69:0/452461685 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc26ca00640 1 -- 172.21.15.69:0/452461685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 msgr2=0x5653880f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc26ca00640 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 0x5653880f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc26ca00640 1 -- 172.21.15.69:0/452461685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880c4310 msgr2=0x5653880f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc26ca00640 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880c4310 0x5653880f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.686+0000 7fc26ca00640 1 -- 172.21.15.69:0/452461685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565387ce23e0 con 0x5653880d6d30 2024-04-24T08:14:28.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.687+0000 7fc26ca00640 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880d9610 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc25c05dae0 tx=0x7fc25c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248a8a37f4c744f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.687+0000 7fc24d600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:28.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.687+0000 7fc24d600640 1 -- 172.21.15.69:0/452461685 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c088030 con 0x5653880d6d30 2024-04-24T08:14:28.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.687+0000 7fc24d600640 1 -- 172.21.15.69:0/452461685 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fc25c055be0 con 0x5653880d6d30 2024-04-24T08:14:28.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.687+0000 7fc2723dd740 1 -- 172.21.15.69:0/452461685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc25c057870 con 0x5653880d6d30 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.687+0000 7fc24d600640 1 -- 172.21.15.69:0/452461685 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fc25c07f050 con 0x5653880d6d30 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.687+0000 7fc2723dd740 10 client.7296 fetch_fsmap learned FSMap version 51 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.687+0000 7fc2723dd740 1 -- 172.21.15.69:0/452461685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x565387e4b1e0 con 0x5653880d6d30 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.687+0000 7fc24d600640 1 -- 172.21.15.69:0/452461685 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc25c057870 con 0x5653880d6d30 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.688+0000 7fc24d600640 1 -- 172.21.15.69:0/452461685 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7fc25c082020 con 0x5653880d6d30 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.688+0000 7fc2723dd740 10 client.7296 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.688+0000 7fc2723dd740 10 client.7296 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.688+0000 7fc2723dd740 20 client.7296 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.688+0000 7fc2723dd740 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5653880c3be0 0x5653880e96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.688+0000 7fc2723dd740 4 client.7296 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.688+0000 7fc2723dd740 1 -- 172.21.15.69:0/452461685 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56538801da90 con 0x5653880c3be0 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.688+0000 7fc267e00640 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5653880c3be0 0x5653880e96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.689+0000 7fc267e00640 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5653880c3be0 0x5653880e96c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c08fddc7482feb6 server_cookie=e1de34b8ff39a24c in_seq=0 out_seq=0 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.689+0000 7fc24d600640 10 client.7296 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.689+0000 7fc24d600640 1 -- 172.21.15.69:0/452461685 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56538801da90 con 0x5653880c3be0 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.689+0000 7fc24d600640 10 client.7296 handle_command_reply: tid=0 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.689+0000 7fc2723dd740 1 client.7296 shutdown 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.689+0000 7fc2723dd740 1 -- 172.21.15.69:0/452461685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 msgr2=0x5653880d9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.689+0000 7fc2723dd740 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880d9610 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc25c05dae0 tx=0x7fc25c0036f0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.690+0000 7fc2723dd740 1 -- 172.21.15.69:0/452461685 shutdown_connections 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.690+0000 7fc2723dd740 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5653880c3be0 0x5653880e96c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.690+0000 7fc2723dd740 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880c4310 0x5653880f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.690+0000 7fc2723dd740 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 0x5653880f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.690+0000 7fc2723dd740 1 --2- 172.21.15.69:0/452461685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880d9610 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.690+0000 7fc24d600640 0 client.7296 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.690+0000 7fc2723dd740 1 -- 172.21.15.69:0/452461685 >> 172.21.15.69:0/452461685 conn(0x5653880d65f0 msgr2=0x5653880f10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.690+0000 7fc2723dd740 1 -- 172.21.15.69:0/452461685 shutdown_connections 2024-04-24T08:14:28.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.690+0000 7fc2723dd740 1 -- 172.21.15.69:0/452461685 wait complete. 2024-04-24T08:14:28.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.690+0000 7fc2723dd740 20 client.7296 trim_cache size 0 max 16384 2024-04-24T08:14:28.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.709+0000 7fc2723dd740 1 Processor -- start 2024-04-24T08:14:28.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.709+0000 7fc2723dd740 1 -- start start 2024-04-24T08:14:28.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880c4310 0x5653880efb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880ac160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 0x5653880f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc2723dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc268200fd0 con 0x5653880d6d30 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc2723dd740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc2682000a0 con 0x5653880c4310 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc2723dd740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc2680af0c0 con 0x5653880f3c10 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc267e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc267e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880ac160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50980/0 (socket says 172.21.15.69:50980) 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc267e00640 1 -- 172.21.15.69:0/3740273782 learned_addr learned my addr 172.21.15.69:0/3740273782 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc267e00640 1 -- 172.21.15.69:0/3740273782 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 msgr2=0x5653880f15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc267e00640 1 --2- 172.21.15.69:0/3740273782 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 0x5653880f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc267e00640 1 -- 172.21.15.69:0/3740273782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880c4310 msgr2=0x5653880efb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc267e00640 1 --2- 172.21.15.69:0/3740273782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880c4310 0x5653880efb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.710+0000 7fc267e00640 1 -- 172.21.15.69:0/3740273782 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565387cbb4a0 con 0x5653880d6d30 2024-04-24T08:14:28.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.711+0000 7fc267e00640 1 --2- 172.21.15.69:0/3740273782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880ac160 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fc254035040 tx=0x7fc25402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7887b695c9ab4d3d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.711+0000 7fc24fe00640 1 -- 172.21.15.69:0/3740273782 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254050070 con 0x5653880d6d30 2024-04-24T08:14:28.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.711+0000 7fc24fe00640 1 -- 172.21.15.69:0/3740273782 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc25404b250 con 0x5653880d6d30 2024-04-24T08:14:28.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.711+0000 7fc24fe00640 1 -- 172.21.15.69:0/3740273782 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc254037040 con 0x5653880d6d30 2024-04-24T08:14:28.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.712+0000 7fc2723dd740 1 -- 172.21.15.69:0/3740273782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 msgr2=0x5653880ac160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.712+0000 7fc2723dd740 1 --2- 172.21.15.69:0/3740273782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880ac160 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fc254035040 tx=0x7fc25402c000 comp rx=0 tx=0).stop 2024-04-24T08:14:28.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.712+0000 7fc2723dd740 1 -- 172.21.15.69:0/3740273782 shutdown_connections 2024-04-24T08:14:28.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.712+0000 7fc2723dd740 1 --2- 172.21.15.69:0/3740273782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880d6d30 0x5653880ac160 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.712+0000 7fc2723dd740 1 --2- 172.21.15.69:0/3740273782 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 0x5653880f15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.712+0000 7fc2723dd740 1 --2- 172.21.15.69:0/3740273782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880c4310 0x5653880efb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.712+0000 7fc2723dd740 1 -- 172.21.15.69:0/3740273782 >> 172.21.15.69:0/3740273782 conn(0x5653880d65f0 msgr2=0x5653880d5d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.712+0000 7fc2723dd740 1 -- 172.21.15.69:0/3740273782 shutdown_connections 2024-04-24T08:14:28.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.712+0000 7fc2723dd740 1 -- 172.21.15.69:0/3740273782 wait complete. 2024-04-24T08:14:28.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.712+0000 7fc2723dd740 1 -- start start 2024-04-24T08:14:28.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.713+0000 7fc2723dd740 1 Processor -- start 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880c4310 0x5653881139d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880d6d30 0x565388113f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc2723dd740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 0x5653880d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc2723dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56538801da90 con 0x5653880c4310 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc2723dd740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc26809b7a0 con 0x5653880d6d30 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc2723dd740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5653880d3b80 con 0x5653880f3c10 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc2723dd740 10 client.0 _ll_register_callbacks cb 0x7fc25c086070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc26ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880c4310 0x5653881139d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc26ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880c4310 0x5653881139d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50992/0 (socket says 172.21.15.69:50992) 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc26ca00640 1 -- 172.21.15.69:0/1305883121 learned_addr learned my addr 172.21.15.69:0/1305883121 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc26ca00640 1 -- 172.21.15.69:0/1305883121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 msgr2=0x5653880d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc26ca00640 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 0x5653880d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc26ca00640 1 -- 172.21.15.69:0/1305883121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880d6d30 msgr2=0x565388113f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc26ca00640 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880d6d30 0x565388113f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.714+0000 7fc26ca00640 1 -- 172.21.15.69:0/1305883121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653880d8f40 con 0x5653880c4310 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc26ca00640 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880c4310 0x5653881139d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fc25c06dfd0 tx=0x7fc25c089c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746c9b7d3af7ea8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc24d600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc24d600640 1 -- 172.21.15.69:0/1305883121 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc25c088050 con 0x5653880c4310 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc24d600640 1 -- 172.21.15.69:0/1305883121 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fc25c055be0 con 0x5653880c4310 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc2723dd740 1 -- 172.21.15.69:0/1305883121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc25404b250 con 0x5653880c4310 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc24d600640 1 -- 172.21.15.69:0/1305883121 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fc25c071040 con 0x5653880c4310 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc2723dd740 10 client.7302 fetch_fsmap learned FSMap version 51 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc2723dd740 1 -- 172.21.15.69:0/1305883121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x5653880d1ff0 con 0x5653880c4310 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc24d600640 1 -- 172.21.15.69:0/1305883121 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc25404b250 con 0x5653880c4310 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc24d600640 1 -- 172.21.15.69:0/1305883121 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7fc25c082040 con 0x5653880c4310 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc2723dd740 10 client.7302 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc2723dd740 10 client.7302 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:14:28.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc2723dd740 20 client.7302 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc2723dd740 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5653880e8cd0 0x5653880d18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc2723dd740 4 client.7302 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}] 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.715+0000 7fc2723dd740 1 -- 172.21.15.69:0/1305883121 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}) -- 0x5653880eae00 con 0x5653880e8cd0 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.716+0000 7fc267e00640 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5653880e8cd0 0x5653880d18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.716+0000 7fc24d600640 10 client.7302 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.716+0000 7fc267e00640 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5653880e8cd0 0x5653880d18d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebce7af9d2a13e48 server_cookie=4fa2f7b8e9e09ec3 in_seq=0 out_seq=0 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc24d600640 1 -- 172.21.15.69:0/1305883121 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5653880eae00 con 0x5653880e8cd0 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc24d600640 10 client.7302 handle_command_reply: tid=0 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 client.7302 shutdown 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 -- 172.21.15.69:0/1305883121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880c4310 msgr2=0x5653881139d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880c4310 0x5653881139d0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fc25c06dfd0 tx=0x7fc25c089c40 comp rx=0 tx=0).stop 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 -- 172.21.15.69:0/1305883121 shutdown_connections 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5653880e8cd0 0x5653880d18d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5653880f3c10 0x5653880d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5653880d6d30 0x565388113f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc24d600640 0 client.7302 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:28.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 --2- 172.21.15.69:0/1305883121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653880c4310 0x5653881139d0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 -- 172.21.15.69:0/1305883121 >> 172.21.15.69:0/1305883121 conn(0x5653880d65f0 msgr2=0x5653880ef630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 -- 172.21.15.69:0/1305883121 shutdown_connections 2024-04-24T08:14:28.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 1 -- 172.21.15.69:0/1305883121 wait complete. 2024-04-24T08:14:28.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.717+0000 7fc2723dd740 20 client.7302 trim_cache size 0 max 16384 2024-04-24T08:14:28.722 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:14:28.722 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-04-24T08:14:28.723 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "d9472f8e-54b7-49d2-b23e-8006561bcb8e", 2024-04-24T08:14:28.723 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-04-24T08:14:28.723 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:14:28.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 -- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc24806c920 msgr2=0x7fc24806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc24806c920 0x7fc24806ede0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc254034000 tx=0x7fc25402e000 comp rx=0 tx=0).stop 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 -- 172.21.15.69:0/1536754280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc26806f5e0 msgr2=0x7fc26815a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc26806f5e0 0x7fc26815a240 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc25c00c460 tx=0x7fc25c00c920 comp rx=0 tx=0).stop 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 -- 172.21.15.69:0/1536754280 shutdown_connections 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc24806c920 0x7fc24806ede0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc2681e8720 0x7fc26815ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680b32d0 0x7fc2681e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 --2- 172.21.15.69:0/1536754280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc26806f5e0 0x7fc26815a240 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 -- 172.21.15.69:0/1536754280 >> 172.21.15.69:0/1536754280 conn(0x7fc2680b2380 msgr2=0x7fc2680d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 -- 172.21.15.69:0/1536754280 shutdown_connections 2024-04-24T08:14:28.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.718+0000 7fc26f400640 1 -- 172.21.15.69:0/1536754280 wait complete. 2024-04-24T08:14:28.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:14:28.736 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-04-24T08:14:28.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.919+0000 7fb21e200640 1 Processor -- start 2024-04-24T08:14:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.919+0000 7fb21e200640 1 -- start start 2024-04-24T08:14:28.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.920+0000 7fb21e200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb2180b46a0 0x7fb2180ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.921+0000 7fb21e200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb21815cc00 con 0x7fb2180b3e90 2024-04-24T08:14:28.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.921+0000 7fb21e200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb21806ed30 con 0x7fb21815c820 2024-04-24T08:14:28.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.921+0000 7fb21e200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb2180b4e10 con 0x7fb2180b46a0 2024-04-24T08:14:28.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.921+0000 7fb216a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb2180b3e90 0x7fb2180ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56656/0 (socket says 172.21.15.69:56656) 2024-04-24T08:14:28.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.921+0000 7fb216a00640 1 -- 172.21.15.69:0/1460696799 learned_addr learned my addr 172.21.15.69:0/1460696799 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:14:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.921+0000 7fb217e00640 1 --2- 172.21.15.69:0/1460696799 >> v2:172.21.15.121:3300/0 conn(0x7fb2180b46a0 0x7fb2180ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.921+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3443217402 0 0) 0x7fb21815cc00 con 0x7fb2180b3e90 2024-04-24T08:14:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.921+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1fc003520 con 0x7fb2180b3e90 2024-04-24T08:14:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3399599519 0 0) 0x7fb1fc003520 con 0x7fb2180b3e90 2024-04-24T08:14:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 >> v2:172.21.15.121:3300/0 conn(0x7fb2180b46a0 msgr2=0x7fb2180ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 --2- 172.21.15.69:0/1460696799 >> v2:172.21.15.121:3300/0 conn(0x7fb2180b46a0 0x7fb2180ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 >> v1:172.21.15.121:6789/0 conn(0x7fb21815c820 legacy=0x7fb21806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2180b50f0 con 0x7fb2180b3e90 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fb204001670 con 0x7fb2180b3e90 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 >> v1:172.21.15.69:6789/0 conn(0x7fb2180b3e90 legacy=0x7fb2180ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 --2- 172.21.15.69:0/1460696799 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1fc003a00 0x7fb1fc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 --2- 172.21.15.69:0/1460696799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc004800 0x7fb1fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 --2- 172.21.15.69:0/1460696799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1fc0051c0 0x7fb1fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2180b4e10 con 0x7fb1fc004800 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb1fc00c0f0 con 0x7fb1fc0051c0 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb1fc00c2f0 con 0x7fb1fc003a00 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.922+0000 7fb216a00640 1 --2- 172.21.15.69:0/1460696799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc004800 0x7fb1fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.923+0000 7fb216a00640 1 -- 172.21.15.69:0/1460696799 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1fc003a00 msgr2=0x7fb1fc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.923+0000 7fb216a00640 1 --2- 172.21.15.69:0/1460696799 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1fc003a00 0x7fb1fc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.923+0000 7fb216a00640 1 -- 172.21.15.69:0/1460696799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1fc0051c0 msgr2=0x7fb1fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.923+0000 7fb216a00640 1 --2- 172.21.15.69:0/1460696799 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1fc0051c0 0x7fb1fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.923+0000 7fb216a00640 1 -- 172.21.15.69:0/1460696799 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1fc003520 con 0x7fb1fc004800 2024-04-24T08:14:28.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.923+0000 7fb217e00640 1 -- 172.21.15.69:0/1460696799 reap_dead start 2024-04-24T08:14:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.923+0000 7fb216a00640 1 --2- 172.21.15.69:0/1460696799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc004800 0x7fb1fc004c80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fb20400c110 tx=0x7fb1fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdc88ef87d5e7cd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.924+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb204007c40 con 0x7fb1fc004800 2024-04-24T08:14:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.924+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb204005af0 con 0x7fb1fc004800 2024-04-24T08:14:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.924+0000 7fb216000640 1 -- 172.21.15.69:0/1460696799 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb204007070 con 0x7fb1fc004800 2024-04-24T08:14:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.924+0000 7fb21e200640 1 -- 172.21.15.69:0/1460696799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc004800 msgr2=0x7fb1fc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.924+0000 7fb21e200640 1 --2- 172.21.15.69:0/1460696799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc004800 0x7fb1fc004c80 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fb20400c110 tx=0x7fb1fc005830 comp rx=0 tx=0).stop 2024-04-24T08:14:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.924+0000 7fb21e200640 1 -- 172.21.15.69:0/1460696799 shutdown_connections 2024-04-24T08:14:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.924+0000 7fb21e200640 1 --2- 172.21.15.69:0/1460696799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1fc004800 0x7fb1fc004c80 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.924+0000 7fb21e200640 1 -- 172.21.15.69:0/1460696799 >> 172.21.15.69:0/1460696799 conn(0x7fb2180d0350 msgr2=0x7fb2180d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.925+0000 7fb21e200640 1 -- 172.21.15.69:0/1460696799 shutdown_connections 2024-04-24T08:14:28.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.925+0000 7fb21e200640 1 -- 172.21.15.69:0/1460696799 wait complete. 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.926+0000 7fb21e200640 1 Processor -- start 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.926+0000 7fb21e200640 1 -- start start 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.927+0000 7fb21e200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1fc004800 0x7fb21815e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.927+0000 7fb21e200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb218148ca0 0x7fb21809aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.927+0000 7fb21e200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21809d430 0x7fb21809d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.927+0000 7fb21e200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2180b4b10 con 0x7fb218148ca0 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.927+0000 7fb21e200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb21806f2b0 con 0x7fb1fc004800 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.927+0000 7fb21e200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb21806f7d0 con 0x7fb21809d430 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.927+0000 7fb216a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb218148ca0 0x7fb21809aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.928+0000 7fb216a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb218148ca0 0x7fb21809aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52338/0 (socket says 172.21.15.69:52338) 2024-04-24T08:14:28.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.928+0000 7fb216a00640 1 -- 172.21.15.69:0/1989510561 learned_addr learned my addr 172.21.15.69:0/1989510561 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.928+0000 7fb216a00640 1 -- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21809d430 msgr2=0x7fb21809d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.928+0000 7fb217e00640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21809d430 0x7fb21809d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.928+0000 7fb217400640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1fc004800 0x7fb21815e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.928+0000 7fb216a00640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21809d430 0x7fb21809d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.928+0000 7fb216a00640 1 -- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1fc004800 msgr2=0x7fb21815e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.928+0000 7fb216a00640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1fc004800 0x7fb21815e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.928+0000 7fb216a00640 1 -- 172.21.15.69:0/1989510561 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21815f570 con 0x7fb218148ca0 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.928+0000 7fb216a00640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb218148ca0 0x7fb21809aab0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb20400c230 tx=0x7fb2180b52c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.929+0000 7fb20f400640 1 -- 172.21.15.69:0/1989510561 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb204037040 con 0x7fb218148ca0 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.929+0000 7fb20f400640 1 -- 172.21.15.69:0/1989510561 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb204041070 con 0x7fb218148ca0 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.929+0000 7fb20f400640 1 -- 172.21.15.69:0/1989510561 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2040047e0 con 0x7fb218148ca0 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.929+0000 7fb21e200640 1 -- 172.21.15.69:0/1989510561 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb21815ecd0 con 0x7fb218148ca0 2024-04-24T08:14:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.929+0000 7fb21e200640 1 -- 172.21.15.69:0/1989510561 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb21815f290 con 0x7fb218148ca0 2024-04-24T08:14:28.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.931+0000 7fb20f400640 1 -- 172.21.15.69:0/1989510561 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb2040049a0 con 0x7fb218148ca0 2024-04-24T08:14:28.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.932+0000 7fb20f400640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f806c8f0 0x7fb1f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.933+0000 7fb20f400640 1 -- 172.21.15.69:0/1989510561 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fb2040abaa0 con 0x7fb218148ca0 2024-04-24T08:14:28.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.933+0000 7fb220fc9740 1 Processor -- start 2024-04-24T08:14:28.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.933+0000 7fb217400640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f806c8f0 0x7fb1f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.933+0000 7fb220fc9740 1 -- start start 2024-04-24T08:14:28.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.934+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fb4c30 0x556db6f675c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.934+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6f97d50 0x556db6f970d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.934+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6f97610 0x556db6f6c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.934+0000 7fb220fc9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556db6ed11f0 con 0x556db6fb4c30 2024-04-24T08:14:28.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.934+0000 7fb220fc9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556db6e77350 con 0x556db6f97d50 2024-04-24T08:14:28.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.934+0000 7fb220fc9740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556db6ecd1b0 con 0x556db6f97610 2024-04-24T08:14:28.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.934+0000 7fb217400640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f806c8f0 0x7fb1f806edb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb218051740 tx=0x7fb208009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.934+0000 7fb217400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fb4c30 0x556db6f675c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.934+0000 7fb217400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fb4c30 0x556db6f675c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52340/0 (socket says 172.21.15.69:52340) 2024-04-24T08:14:28.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.934+0000 7fb217400640 1 -- 172.21.15.69:0/290622079 learned_addr learned my addr 172.21.15.69:0/290622079 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.935+0000 7fb216a00640 1 --2- 172.21.15.69:0/290622079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6f97d50 0x556db6f970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.935+0000 7fb217400640 1 -- 172.21.15.69:0/290622079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6f97610 msgr2=0x556db6f6c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.935+0000 7fb217400640 1 --2- 172.21.15.69:0/290622079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6f97610 0x556db6f6c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.935+0000 7fb217400640 1 -- 172.21.15.69:0/290622079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6f97d50 msgr2=0x556db6f970d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.935+0000 7fb217400640 1 --2- 172.21.15.69:0/290622079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6f97d50 0x556db6f970d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.935+0000 7fb217400640 1 -- 172.21.15.69:0/290622079 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556db6ecbf50 con 0x556db6fb4c30 2024-04-24T08:14:28.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.935+0000 7fb217400640 1 --2- 172.21.15.69:0/290622079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fb4c30 0x556db6f675c0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb208014820 tx=0x7fb208010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a7b8381dc34786 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.935+0000 7fb203400640 1 -- 172.21.15.69:0/290622079 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208024070 con 0x556db6fb4c30 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.935+0000 7fb203400640 1 -- 172.21.15.69:0/290622079 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb208012ad0 con 0x556db6fb4c30 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.935+0000 7fb203400640 1 -- 172.21.15.69:0/290622079 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20801f400 con 0x556db6fb4c30 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.936+0000 7fb220fc9740 1 -- 172.21.15.69:0/290622079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fb4c30 msgr2=0x556db6f675c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.936+0000 7fb220fc9740 1 --2- 172.21.15.69:0/290622079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fb4c30 0x556db6f675c0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fb208014820 tx=0x7fb208010000 comp rx=0 tx=0).stop 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.936+0000 7fb220fc9740 1 -- 172.21.15.69:0/290622079 shutdown_connections 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.936+0000 7fb220fc9740 1 --2- 172.21.15.69:0/290622079 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6f97610 0x556db6f6c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.936+0000 7fb220fc9740 1 --2- 172.21.15.69:0/290622079 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6f97d50 0x556db6f970d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.936+0000 7fb220fc9740 1 --2- 172.21.15.69:0/290622079 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fb4c30 0x556db6f675c0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.936+0000 7fb220fc9740 1 -- 172.21.15.69:0/290622079 >> 172.21.15.69:0/290622079 conn(0x556db6f69830 msgr2=0x556db6fb20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.936+0000 7fb220fc9740 1 -- 172.21.15.69:0/290622079 shutdown_connections 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.936+0000 7fb220fc9740 1 -- 172.21.15.69:0/290622079 wait complete. 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.937+0000 7fb220fc9740 1 -- start start 2024-04-24T08:14:28.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.937+0000 7fb220fc9740 1 Processor -- start 2024-04-24T08:14:28.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.938+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6f8cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.938+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6fb4c30 0x556db6fb0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.938+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fc6b60 0x556db6fb20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.938+0000 7fb220fc9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556db6f6b9e0 con 0x556db6f97d50 2024-04-24T08:14:28.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.938+0000 7fb220fc9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556db6e8a690 con 0x556db6fc6b60 2024-04-24T08:14:28.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.938+0000 7fb220fc9740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556db6f6add0 con 0x556db6fb4c30 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.938+0000 7fb220fc9740 10 client.0 _ll_register_callbacks cb 0x556db6ec6650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.938+0000 7fb217400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6f8cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.938+0000 7fb217400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6f8cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52350/0 (socket says 172.21.15.69:52350) 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.938+0000 7fb217400640 1 -- 172.21.15.69:0/2411552097 learned_addr learned my addr 172.21.15.69:0/2411552097 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.939+0000 7fb217400640 1 -- 172.21.15.69:0/2411552097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6fb4c30 msgr2=0x556db6fb0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.939+0000 7fb217400640 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6fb4c30 0x556db6fb0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.939+0000 7fb217400640 1 -- 172.21.15.69:0/2411552097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fc6b60 msgr2=0x556db6fb20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.939+0000 7fb217400640 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fc6b60 0x556db6fb20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.939+0000 7fb217400640 1 -- 172.21.15.69:0/2411552097 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556db6e89920 con 0x556db6f97d50 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.939+0000 7fb217400640 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6f8cc80 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb208022040 tx=0x7fb208001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7815e78dec05576 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.939+0000 7fb1f7e00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.939+0000 7fb1f7e00640 1 -- 172.21.15.69:0/2411552097 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20802f070 con 0x556db6f97d50 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.939+0000 7fb1f7e00640 1 -- 172.21.15.69:0/2411552097 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fb20802b660 con 0x556db6f97d50 2024-04-24T08:14:28.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.939+0000 7fb220fc9740 1 -- 172.21.15.69:0/2411552097 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb208012ad0 con 0x556db6f97d50 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb1f7e00640 1 -- 172.21.15.69:0/2411552097 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fb208039050 con 0x556db6f97d50 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb220fc9740 10 client.7314 fetch_fsmap learned FSMap version 51 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb220fc9740 1 -- 172.21.15.69:0/2411552097 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x556db6cad7d0 con 0x556db6f97d50 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb1f7e00640 1 -- 172.21.15.69:0/2411552097 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb208012ad0 con 0x556db6f97d50 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb1f7e00640 1 -- 172.21.15.69:0/2411552097 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7fb20803c020 con 0x556db6f97d50 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb220fc9740 10 client.7314 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb220fc9740 10 client.7314 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb220fc9740 20 client.7314 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb220fc9740 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x556db6f9aac0 0x556db6fad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb220fc9740 4 client.7314 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb220fc9740 1 -- 172.21.15.69:0/2411552097 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556db6edec10 con 0x556db6f9aac0 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.940+0000 7fb216a00640 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x556db6f9aac0 0x556db6fad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.941+0000 7fb216a00640 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x556db6f9aac0 0x556db6fad430 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36e7b710f1a95988 server_cookie=b110fc1adfed739f in_seq=0 out_seq=0 2024-04-24T08:14:28.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.941+0000 7fb1f7e00640 10 client.7314 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.941+0000 7fb1f7e00640 1 -- 172.21.15.69:0/2411552097 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556db6edec10 con 0x556db6f9aac0 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.941+0000 7fb1f7e00640 10 client.7314 handle_command_reply: tid=0 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.941+0000 7fb220fc9740 1 client.7314 shutdown 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 1 -- 172.21.15.69:0/2411552097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 msgr2=0x556db6f8cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6f8cc80 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb208022040 tx=0x7fb208001120 comp rx=0 tx=0).stop 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 1 -- 172.21.15.69:0/2411552097 shutdown_connections 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x556db6f9aac0 0x556db6fad430 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fc6b60 0x556db6fb20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6fb4c30 0x556db6fb0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb1f7e00640 0 client.7314 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 1 --2- 172.21.15.69:0/2411552097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6f8cc80 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 1 -- 172.21.15.69:0/2411552097 >> 172.21.15.69:0/2411552097 conn(0x556db6f97610 msgr2=0x556db6f66c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 1 -- 172.21.15.69:0/2411552097 shutdown_connections 2024-04-24T08:14:28.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 1 -- 172.21.15.69:0/2411552097 wait complete. 2024-04-24T08:14:28.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.942+0000 7fb220fc9740 20 client.7314 trim_cache size 0 max 16384 2024-04-24T08:14:28.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.960+0000 7fb220fc9740 1 Processor -- start 2024-04-24T08:14:28.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.960+0000 7fb220fc9740 1 -- start start 2024-04-24T08:14:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6f97d50 0x556db6f68300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fb4c30 0x556db6f93a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fc6b60 0x556db6f6add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb220fc9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2181fd630 con 0x556db6fc6b60 2024-04-24T08:14:28.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb220fc9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb218073a00 con 0x556db6fb4c30 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb220fc9740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb21814ad50 con 0x556db6f97d50 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb217e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fc6b60 0x556db6f6add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb217e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fc6b60 0x556db6f6add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52360/0 (socket says 172.21.15.69:52360) 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb217e00640 1 -- 172.21.15.69:0/3672542975 learned_addr learned my addr 172.21.15.69:0/3672542975 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb217e00640 1 -- 172.21.15.69:0/3672542975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6f97d50 msgr2=0x556db6f68300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.961+0000 7fb217400640 1 --2- 172.21.15.69:0/3672542975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6f97d50 0x556db6f68300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.962+0000 7fb216a00640 1 --2- 172.21.15.69:0/3672542975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fb4c30 0x556db6f93a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.962+0000 7fb217e00640 1 --2- 172.21.15.69:0/3672542975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6f97d50 0x556db6f68300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.962+0000 7fb217e00640 1 -- 172.21.15.69:0/3672542975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fb4c30 msgr2=0x556db6f93a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.962+0000 7fb217e00640 1 --2- 172.21.15.69:0/3672542975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fb4c30 0x556db6f93a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.962+0000 7fb217e00640 1 -- 172.21.15.69:0/3672542975 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556db6be33f0 con 0x556db6fc6b60 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.962+0000 7fb217e00640 1 --2- 172.21.15.69:0/3672542975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fc6b60 0x556db6f6add0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fb2180ce120 tx=0x7fb210007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2ff7b2719fb8ba server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.962+0000 7fb202a00640 1 -- 172.21.15.69:0/3672542975 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210003b90 con 0x556db6fc6b60 2024-04-24T08:14:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.962+0000 7fb202a00640 1 -- 172.21.15.69:0/3672542975 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb210003d50 con 0x556db6fc6b60 2024-04-24T08:14:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.962+0000 7fb202a00640 1 -- 172.21.15.69:0/3672542975 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210004270 con 0x556db6fc6b60 2024-04-24T08:14:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.963+0000 7fb220fc9740 1 -- 172.21.15.69:0/3672542975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fc6b60 msgr2=0x556db6f6add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.963+0000 7fb220fc9740 1 --2- 172.21.15.69:0/3672542975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fc6b60 0x556db6f6add0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fb2180ce120 tx=0x7fb210007700 comp rx=0 tx=0).stop 2024-04-24T08:14:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.963+0000 7fb220fc9740 1 -- 172.21.15.69:0/3672542975 shutdown_connections 2024-04-24T08:14:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.963+0000 7fb220fc9740 1 --2- 172.21.15.69:0/3672542975 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6fc6b60 0x556db6f6add0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.963+0000 7fb220fc9740 1 --2- 172.21.15.69:0/3672542975 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fb4c30 0x556db6f93a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.963+0000 7fb220fc9740 1 --2- 172.21.15.69:0/3672542975 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6f97d50 0x556db6f68300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.963+0000 7fb220fc9740 1 -- 172.21.15.69:0/3672542975 >> 172.21.15.69:0/3672542975 conn(0x556db6f97610 msgr2=0x556db6fb1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.963+0000 7fb220fc9740 1 -- 172.21.15.69:0/3672542975 shutdown_connections 2024-04-24T08:14:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.963+0000 7fb220fc9740 1 -- 172.21.15.69:0/3672542975 wait complete. 2024-04-24T08:14:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.964+0000 7fb220fc9740 1 -- start start 2024-04-24T08:14:28.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.964+0000 7fb220fc9740 1 Processor -- start 2024-04-24T08:14:28.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6fa7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fb4c30 0x556db7033c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb220fc9740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6fc6b60 0x556db6f6acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb220fc9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556db6faf220 con 0x556db6f97d50 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb220fc9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb2180733b0 con 0x556db6fb4c30 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb220fc9740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556db6f93830 con 0x556db6fc6b60 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb220fc9740 10 client.0 _ll_register_callbacks cb 0x7fb208022070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb217400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6fa7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb217400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6fa7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52370/0 (socket says 172.21.15.69:52370) 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb217400640 1 -- 172.21.15.69:0/1359711261 learned_addr learned my addr 172.21.15.69:0/1359711261 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb217400640 1 -- 172.21.15.69:0/1359711261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6fc6b60 msgr2=0x556db6f6acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb217400640 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6fc6b60 0x556db6f6acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb217400640 1 -- 172.21.15.69:0/1359711261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fb4c30 msgr2=0x556db7033c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb217400640 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fb4c30 0x556db7033c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.965+0000 7fb217400640 1 -- 172.21.15.69:0/1359711261 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556db6ba30f0 con 0x556db6f97d50 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb217400640 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6fa7a60 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb208007d80 tx=0x7fb208013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a428acc3442c6ef1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb1f7e00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb1f7e00640 1 -- 172.21.15.69:0/1359711261 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208042060 con 0x556db6f97d50 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb1f7e00640 1 -- 172.21.15.69:0/1359711261 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fb20802bca0 con 0x556db6f97d50 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb220fc9740 1 -- 172.21.15.69:0/1359711261 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb210003d50 con 0x556db6f97d50 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb1f7e00640 1 -- 172.21.15.69:0/1359711261 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fb20802f050 con 0x556db6f97d50 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb220fc9740 10 client.7320 fetch_fsmap learned FSMap version 51 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb220fc9740 1 -- 172.21.15.69:0/1359711261 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=2+}) -- 0x556db6e96810 con 0x556db6f97d50 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb1f7e00640 1 -- 172.21.15.69:0/1359711261 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb210003d50 con 0x556db6f97d50 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb1f7e00640 1 -- 172.21.15.69:0/1359711261 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 51) ==== 3494+0+0 (secure 0 0 0) 0x7fb20804b080 con 0x556db6f97d50 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb220fc9740 10 client.7320 fetch_fsmap finished waiting for FSMap version 51 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb220fc9740 10 client.7320 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb220fc9740 20 client.7320 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb220fc9740 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x556db6fda1f0 0x556db6f6b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:28.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb220fc9740 4 client.7320 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.966+0000 7fb220fc9740 1 -- 172.21.15.69:0/1359711261 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556db6fb0b00 con 0x556db6fda1f0 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.967+0000 7fb216a00640 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x556db6fda1f0 0x556db6f6b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.967+0000 7fb216a00640 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x556db6fda1f0 0x556db6f6b210 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80c4d8f7f3efadab server_cookie=60be5cbfe07e7955 in_seq=0 out_seq=0 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.967+0000 7fb1f7e00640 10 client.7320 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.967+0000 7fb1f7e00640 1 -- 172.21.15.69:0/1359711261 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x556db6fb0b00 con 0x556db6fda1f0 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.967+0000 7fb1f7e00640 10 client.7320 handle_command_reply: tid=0 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.967+0000 7fb220fc9740 1 client.7320 shutdown 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.967+0000 7fb220fc9740 1 -- 172.21.15.69:0/1359711261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 msgr2=0x556db6fa7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.967+0000 7fb220fc9740 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6fa7a60 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb208007d80 tx=0x7fb208013000 comp rx=0 tx=0).stop 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb220fc9740 1 -- 172.21.15.69:0/1359711261 shutdown_connections 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb220fc9740 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x556db6fda1f0 0x556db6f6b210 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb220fc9740 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556db6fc6b60 0x556db6f6acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb220fc9740 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556db6fb4c30 0x556db7033c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb1f7e00640 0 client.7320 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:28.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb220fc9740 1 --2- 172.21.15.69:0/1359711261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556db6f97d50 0x556db6fa7a60 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb220fc9740 1 -- 172.21.15.69:0/1359711261 >> 172.21.15.69:0/1359711261 conn(0x556db6f97610 msgr2=0x556db6fb3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb220fc9740 1 -- 172.21.15.69:0/1359711261 shutdown_connections 2024-04-24T08:14:28.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb220fc9740 1 -- 172.21.15.69:0/1359711261 wait complete. 2024-04-24T08:14:28.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb220fc9740 20 client.7320 trim_cache size 0 max 16384 2024-04-24T08:14:28.973 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:14:28.973 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:14:28.973 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-04-24T08:14:28.973 INFO:teuthology.orchestra.run.smithi069.stdout: "d9472f8e-54b7-49d2-b23e-8006561bcb8e": { 2024-04-24T08:14:28.973 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/subdir", 2024-04-24T08:14:28.974 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "d9472f8e-54b7-49d2-b23e-8006561bcb8e", 2024-04-24T08:14:28.974 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,repair" 2024-04-24T08:14:28.974 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:14:28.974 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:14:28.974 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:14:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb21e200640 1 -- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f806c8f0 msgr2=0x7fb1f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb21e200640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f806c8f0 0x7fb1f806edb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb218051740 tx=0x7fb208009040 comp rx=0 tx=0).stop 2024-04-24T08:14:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb21e200640 1 -- 172.21.15.69:0/1989510561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb218148ca0 msgr2=0x7fb21809aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb21e200640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb218148ca0 0x7fb21809aab0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fb20400c230 tx=0x7fb2180b52c0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb21e200640 1 -- 172.21.15.69:0/1989510561 shutdown_connections 2024-04-24T08:14:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb21e200640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1f806c8f0 0x7fb1f806edb0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.968+0000 7fb21e200640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb218148ca0 0x7fb21809aab0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.969+0000 7fb21e200640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb21809d430 0x7fb21809d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.969+0000 7fb21e200640 1 --2- 172.21.15.69:0/1989510561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1fc004800 0x7fb21815e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:28.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.969+0000 7fb21e200640 1 -- 172.21.15.69:0/1989510561 >> 172.21.15.69:0/1989510561 conn(0x7fb2180d0350 msgr2=0x7fb2180b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:28.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.969+0000 7fb21e200640 1 -- 172.21.15.69:0/1989510561 shutdown_connections 2024-04-24T08:14:28.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:28.969+0000 7fb21e200640 1 -- 172.21.15.69:0/1989510561 wait complete. 2024-04-24T08:14:28.987 INFO:tasks.cephfs.filesystem:scrub status for tag:d9472f8e-54b7-49d2-b23e-8006561bcb8e - {'path': '/subdir', 'tag': 'd9472f8e-54b7-49d2-b23e-8006561bcb8e', 'options': 'recursive,repair'} 2024-04-24T08:14:45.046 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:14:45.054 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:14:58.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:14:58.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-04-24T08:14:59.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.206+0000 7f5f35600640 1 Processor -- start 2024-04-24T08:14:59.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.206+0000 7f5f35600640 1 -- start start 2024-04-24T08:14:59.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.207+0000 7f5f35600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5f300b3e90 0x7f5f300b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.207+0000 7f5f35600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5f300ce250 0x7f5f300ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.207+0000 7f5f35600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f3015cc00 con 0x7f5f3015c820 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.207+0000 7f5f35600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5f300b4be0 con 0x7f5f300ce250 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.207+0000 7f5f35600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5f3006ed30 con 0x7f5f300b3e90 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.208+0000 7f5f2f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5f300ce250 0x7f5f300ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.208+0000 7f5f2f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5f300ce250 0x7f5f300ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34348/0 (socket says 172.21.15.69:34348) 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.208+0000 7f5f2f400640 1 -- 172.21.15.69:0/3290436751 learned_addr learned my addr 172.21.15.69:0/3290436751 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.208+0000 7f5f2e000640 1 --2- 172.21.15.69:0/3290436751 >> v2:172.21.15.121:3300/0 conn(0x7f5f300b3e90 0x7f5f300b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.208+0000 7f5f2f400640 1 -- 172.21.15.69:0/3290436751 >> v2:172.21.15.121:3300/0 conn(0x7f5f300b3e90 msgr2=0x7f5f300b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.208+0000 7f5f2f400640 1 --2- 172.21.15.69:0/3290436751 >> v2:172.21.15.121:3300/0 conn(0x7f5f300b3e90 0x7f5f300b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.208+0000 7f5f2f400640 1 -- 172.21.15.69:0/3290436751 >> v1:172.21.15.121:6789/0 conn(0x7f5f3015c820 legacy=0x7f5f3006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.208+0000 7f5f2f400640 1 -- 172.21.15.69:0/3290436751 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f300b4e10 con 0x7f5f300ce250 2024-04-24T08:14:59.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.209+0000 7f5f2f400640 1 --2- 172.21.15.69:0/3290436751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f300ce250 0x7f5f300ce6c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f5f3006e200 tx=0x7f5f2802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bbb1627206bb08d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.209+0000 7f5f2d600640 1 -- 172.21.15.69:0/3290436751 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28036070 con 0x7f5f300ce250 2024-04-24T08:14:59.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.210+0000 7f5f2d600640 1 -- 172.21.15.69:0/3290436751 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f28031600 con 0x7f5f300ce250 2024-04-24T08:14:59.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.210+0000 7f5f2d600640 1 -- 172.21.15.69:0/3290436751 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f280319d0 con 0x7f5f300ce250 2024-04-24T08:14:59.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.210+0000 7f5f35600640 1 -- 172.21.15.69:0/3290436751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f300ce250 msgr2=0x7f5f300ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.210+0000 7f5f35600640 1 --2- 172.21.15.69:0/3290436751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f300ce250 0x7f5f300ce6c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f5f3006e200 tx=0x7f5f2802a6f0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.210+0000 7f5f35600640 1 -- 172.21.15.69:0/3290436751 shutdown_connections 2024-04-24T08:14:59.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.210+0000 7f5f35600640 1 --2- 172.21.15.69:0/3290436751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f300ce250 0x7f5f300ce6c0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.210+0000 7f5f35600640 1 --2- 172.21.15.69:0/3290436751 >> v2:172.21.15.121:3300/0 conn(0x7f5f300b3e90 0x7f5f300b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.210+0000 7f5f35600640 1 -- 172.21.15.69:0/3290436751 >> 172.21.15.69:0/3290436751 conn(0x7f5f300d0350 msgr2=0x7f5f300d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.210+0000 7f5f35600640 1 -- 172.21.15.69:0/3290436751 shutdown_connections 2024-04-24T08:14:59.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.210+0000 7f5f35600640 1 -- 172.21.15.69:0/3290436751 wait complete. 2024-04-24T08:14:59.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.212+0000 7f5f35600640 1 Processor -- start 2024-04-24T08:14:59.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.212+0000 7f5f35600640 1 -- start start 2024-04-24T08:14:59.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.213+0000 7f5f35600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f300b3e90 0x7f5f301576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.213+0000 7f5f35600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f300ce250 0x7f5f301e61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.213+0000 7f5f35600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f3015c820 0x7f5f300ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.213+0000 7f5f35600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3006f2b0 con 0x7f5f3015c820 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.213+0000 7f5f35600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3006efd0 con 0x7f5f300ce250 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.213+0000 7f5f35600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5f3006fee0 con 0x7f5f300b3e90 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f3015c820 0x7f5f300ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f3015c820 0x7f5f300ae7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34358/0 (socket says 172.21.15.69:34358) 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2f400640 1 -- 172.21.15.69:0/3963567841 learned_addr learned my addr 172.21.15.69:0/3963567841 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2e000640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f300ce250 0x7f5f301e61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f300b3e90 0x7f5f301576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2f400640 1 -- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f300b3e90 msgr2=0x7f5f301576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2f400640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f300b3e90 0x7f5f301576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2f400640 1 -- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f300ce250 msgr2=0x7f5f301e61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2f400640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f300ce250 0x7f5f301e61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2f400640 1 -- 172.21.15.69:0/3963567841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f301e6700 con 0x7f5f3015c820 2024-04-24T08:14:59.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2e000640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f300ce250 0x7f5f301e61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:59.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.214+0000 7f5f2f400640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f3015c820 0x7f5f300ae7a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5f2802ace0 tx=0x7f5f28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.215+0000 7f5f26a00640 1 -- 172.21.15.69:0/3963567841 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28036050 con 0x7f5f3015c820 2024-04-24T08:14:59.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.215+0000 7f5f26a00640 1 -- 172.21.15.69:0/3963567841 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f28042070 con 0x7f5f3015c820 2024-04-24T08:14:59.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.215+0000 7f5f26a00640 1 -- 172.21.15.69:0/3963567841 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28009510 con 0x7f5f3015c820 2024-04-24T08:14:59.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.215+0000 7f5f35600640 1 -- 172.21.15.69:0/3963567841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f301e6980 con 0x7f5f3015c820 2024-04-24T08:14:59.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.215+0000 7f5f35600640 1 -- 172.21.15.69:0/3963567841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f301e7220 con 0x7f5f3015c820 2024-04-24T08:14:59.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.217+0000 7f5f26a00640 1 -- 172.21.15.69:0/3963567841 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5f28007bd0 con 0x7f5f3015c820 2024-04-24T08:14:59.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.218+0000 7f5f26a00640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f1006c8f0 0x7f5f1006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.218+0000 7f5f26a00640 1 -- 172.21.15.69:0/3963567841 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f5f280acc80 con 0x7f5f3015c820 2024-04-24T08:14:59.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.219+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f1006c8f0 0x7f5f1006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.219+0000 7f5f383af740 1 Processor -- start 2024-04-24T08:14:59.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.219+0000 7f5f383af740 1 -- start start 2024-04-24T08:14:59.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.220+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f1006c8f0 0x7f5f1006edb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5f300af760 tx=0x7f5f20009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.220+0000 7f5f383af740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756826c30 0x55f7567d95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.220+0000 7f5f383af740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756809d50 0x55f7568090d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.220+0000 7f5f383af740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809610 0x55f7567de420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.220+0000 7f5f383af740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f7567431f0 con 0x55f756809d50 2024-04-24T08:14:59.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.220+0000 7f5f383af740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f7566e9350 con 0x55f756826c30 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.220+0000 7f5f383af740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f75673f1b0 con 0x55f756809610 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756809d50 0x55f7568090d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756809d50 0x55f7568090d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34374/0 (socket says 172.21.15.69:34374) 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2e000640 1 -- 172.21.15.69:0/1233592115 learned_addr learned my addr 172.21.15.69:0/1233592115 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756826c30 0x55f7567d95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2f400640 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809610 0x55f7567de420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2e000640 1 -- 172.21.15.69:0/1233592115 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809610 msgr2=0x55f7567de420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2e000640 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809610 0x55f7567de420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2e000640 1 -- 172.21.15.69:0/1233592115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756826c30 msgr2=0x55f7567d95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2e000640 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756826c30 0x55f7567d95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2e000640 1 -- 172.21.15.69:0/1233592115 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f75673df50 con 0x55f756809d50 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756826c30 0x55f7567d95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2f400640 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809610 0x55f7567de420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:59.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.221+0000 7f5f2e000640 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756809d50 0x55f7568090d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00b9b0 tx=0x7f5f1c00be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1ae6a8efd0dcef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f1b000640 1 -- 172.21.15.69:0/1233592115 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c016070 con 0x55f756809d50 2024-04-24T08:14:59.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f1b000640 1 -- 172.21.15.69:0/1233592115 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f1c00dc70 con 0x55f756809d50 2024-04-24T08:14:59.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f1b000640 1 -- 172.21.15.69:0/1233592115 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c012460 con 0x55f756809d50 2024-04-24T08:14:59.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f383af740 1 -- 172.21.15.69:0/1233592115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756809d50 msgr2=0x55f7568090d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f383af740 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756809d50 0x55f7568090d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c00b9b0 tx=0x7f5f1c00be70 comp rx=0 tx=0).stop 2024-04-24T08:14:59.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f383af740 1 -- 172.21.15.69:0/1233592115 shutdown_connections 2024-04-24T08:14:59.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f383af740 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756809d50 0x55f7568090d0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f383af740 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809610 0x55f7567de420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f383af740 1 --2- 172.21.15.69:0/1233592115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756826c30 0x55f7567d95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f383af740 1 -- 172.21.15.69:0/1233592115 >> 172.21.15.69:0/1233592115 conn(0x55f7567db830 msgr2=0x55f7568240d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f383af740 1 -- 172.21.15.69:0/1233592115 shutdown_connections 2024-04-24T08:14:59.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.222+0000 7f5f383af740 1 -- 172.21.15.69:0/1233592115 wait complete. 2024-04-24T08:14:59.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.223+0000 7f5f383af740 1 -- start start 2024-04-24T08:14:59.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.223+0000 7f5f383af740 1 Processor -- start 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.224+0000 7f5f383af740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809d50 0x55f756804700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.224+0000 7f5f383af740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f75680c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.224+0000 7f5f383af740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f75681ad90 0x55f75680cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.224+0000 7f5f383af740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f7567dd9e0 con 0x55f756826c30 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.224+0000 7f5f383af740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f7566fc690 con 0x55f75681ad90 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.224+0000 7f5f383af740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f7567dcdd0 con 0x55f756809d50 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.224+0000 7f5f383af740 10 client.0 _ll_register_callbacks cb 0x55f756738650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f75680c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f75680c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34382/0 (socket says 172.21.15.69:34382) 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2e000640 1 -- 172.21.15.69:0/3771458375 learned_addr learned my addr 172.21.15.69:0/3771458375 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809d50 0x55f756804700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2f400640 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f75681ad90 0x55f75680cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2e000640 1 -- 172.21.15.69:0/3771458375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809d50 msgr2=0x55f756804700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2e000640 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809d50 0x55f756804700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2e000640 1 -- 172.21.15.69:0/3771458375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f75681ad90 msgr2=0x55f75680cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2e000640 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f75681ad90 0x55f75680cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2e000640 1 -- 172.21.15.69:0/3771458375 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7566fb920 con 0x55f756826c30 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f2e000640 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f75680c6c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c000c00 tx=0x7f5f1c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2d20fc12ef7b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.225+0000 7f5f0f400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.226+0000 7f5f0f400640 1 -- 172.21.15.69:0/3771458375 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c02a060 con 0x55f756826c30 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.226+0000 7f5f0f400640 1 -- 172.21.15.69:0/3771458375 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f5f1c00b460 con 0x55f756826c30 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.226+0000 7f5f383af740 1 -- 172.21.15.69:0/3771458375 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f1c00dc70 con 0x55f756826c30 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.226+0000 7f5f0f400640 1 -- 172.21.15.69:0/3771458375 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f5f1c030050 con 0x55f756826c30 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.226+0000 7f5f383af740 10 client.7332 fetch_fsmap learned FSMap version 52 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.226+0000 7f5f383af740 1 -- 172.21.15.69:0/3771458375 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x55f75651f7d0 con 0x55f756826c30 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.226+0000 7f5f0f400640 1 -- 172.21.15.69:0/3771458375 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f1c00dc70 con 0x55f756826c30 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.226+0000 7f5f0f400640 1 -- 172.21.15.69:0/3771458375 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7f5f1c033020 con 0x55f756826c30 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.227+0000 7f5f383af740 10 client.7332 fetch_fsmap finished waiting for FSMap version 52 2024-04-24T08:14:59.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.227+0000 7f5f383af740 10 client.7332 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:14:59.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.227+0000 7f5f383af740 20 client.7332 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:59.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.227+0000 7f5f383af740 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55f756837ef0 0x55f7568382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.227+0000 7f5f383af740 4 client.7332 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:14:59.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.227+0000 7f5f383af740 1 -- 172.21.15.69:0/3771458375 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f756750c10 con 0x55f756837ef0 2024-04-24T08:14:59.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.227+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55f756837ef0 0x55f7568382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.227+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55f756837ef0 0x55f7568382f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd4bce50a28037d7 server_cookie=ff7690c884608d48 in_seq=0 out_seq=0 2024-04-24T08:14:59.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.227+0000 7f5f0f400640 10 client.7332 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:59.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f0f400640 1 -- 172.21.15.69:0/3771458375 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f756750c10 con 0x55f756837ef0 2024-04-24T08:14:59.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f0f400640 10 client.7332 handle_command_reply: tid=0 2024-04-24T08:14:59.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f383af740 1 client.7332 shutdown 2024-04-24T08:14:59.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f383af740 1 -- 172.21.15.69:0/3771458375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 msgr2=0x55f75680c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f383af740 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f75680c6c0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c000c00 tx=0x7f5f1c010000 comp rx=0 tx=0).stop 2024-04-24T08:14:59.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f383af740 1 -- 172.21.15.69:0/3771458375 shutdown_connections 2024-04-24T08:14:59.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f383af740 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55f756837ef0 0x55f7568382f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f383af740 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f75680c6c0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f383af740 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f75681ad90 0x55f75680cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f0f400640 0 client.7332 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:59.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f383af740 1 --2- 172.21.15.69:0/3771458375 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756809d50 0x55f756804700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.228+0000 7f5f383af740 1 -- 172.21.15.69:0/3771458375 >> 172.21.15.69:0/3771458375 conn(0x55f756809610 msgr2=0x55f7568274f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.229+0000 7f5f383af740 1 -- 172.21.15.69:0/3771458375 shutdown_connections 2024-04-24T08:14:59.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.229+0000 7f5f383af740 1 -- 172.21.15.69:0/3771458375 wait complete. 2024-04-24T08:14:59.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.229+0000 7f5f383af740 20 client.7332 trim_cache size 0 max 16384 2024-04-24T08:14:59.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.248+0000 7f5f383af740 1 Processor -- start 2024-04-24T08:14:59.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.248+0000 7f5f383af740 1 -- start start 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.249+0000 7f5f383af740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756809d50 0x55f7567df0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.249+0000 7f5f383af740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f75681ad90 0x55f756808a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.249+0000 7f5f383af740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f756823000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.249+0000 7f5f383af740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5f302012d0 con 0x55f756826c30 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.249+0000 7f5f383af740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3015ce50 con 0x55f756809d50 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.249+0000 7f5f383af740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5f30073290 con 0x55f75681ad90 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f2f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f756823000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f2f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f756823000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34388/0 (socket says 172.21.15.69:34388) 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f2f400640 1 -- 172.21.15.69:0/432934935 learned_addr learned my addr 172.21.15.69:0/432934935 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f2f400640 1 -- 172.21.15.69:0/432934935 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f75681ad90 msgr2=0x55f756808a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f2f400640 1 --2- 172.21.15.69:0/432934935 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f75681ad90 0x55f756808a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f2f400640 1 -- 172.21.15.69:0/432934935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756809d50 msgr2=0x55f7567df0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/432934935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756809d50 0x55f7567df0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f2f400640 1 --2- 172.21.15.69:0/432934935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756809d50 0x55f7567df0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f2f400640 1 -- 172.21.15.69:0/432934935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7564553f0 con 0x55f756826c30 2024-04-24T08:14:59.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f2f400640 1 --2- 172.21.15.69:0/432934935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f756823000 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5f280aafe0 tx=0x7f5f28076c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a5e9c44698f17d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.250+0000 7f5f1a600640 1 -- 172.21.15.69:0/432934935 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f280a9070 con 0x55f756826c30 2024-04-24T08:14:59.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f1a600640 1 -- 172.21.15.69:0/432934935 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f28037040 con 0x55f756826c30 2024-04-24T08:14:59.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f1a600640 1 -- 172.21.15.69:0/432934935 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f28031600 con 0x55f756826c30 2024-04-24T08:14:59.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f383af740 1 -- 172.21.15.69:0/432934935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 msgr2=0x55f756823000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f383af740 1 --2- 172.21.15.69:0/432934935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f756823000 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5f280aafe0 tx=0x7f5f28076c10 comp rx=0 tx=0).stop 2024-04-24T08:14:59.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f383af740 1 -- 172.21.15.69:0/432934935 shutdown_connections 2024-04-24T08:14:59.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f383af740 1 --2- 172.21.15.69:0/432934935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f756826c30 0x55f756823000 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f383af740 1 --2- 172.21.15.69:0/432934935 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f75681ad90 0x55f756808a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f383af740 1 --2- 172.21.15.69:0/432934935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756809d50 0x55f7567df0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f383af740 1 -- 172.21.15.69:0/432934935 >> 172.21.15.69:0/432934935 conn(0x55f756809610 msgr2=0x55f75680bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f383af740 1 -- 172.21.15.69:0/432934935 shutdown_connections 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.251+0000 7f5f383af740 1 -- 172.21.15.69:0/432934935 wait complete. 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.252+0000 7f5f383af740 1 -- start start 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.252+0000 7f5f383af740 1 Processor -- start 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.253+0000 7f5f383af740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756809d50 0x55f7568947f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.253+0000 7f5f383af740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f75681ad90 0x55f756824640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.253+0000 7f5f383af740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756826c30 0x55f756819a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.253+0000 7f5f383af740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5f300738e0 con 0x55f75681ad90 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.253+0000 7f5f383af740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f756807f40 con 0x55f756809d50 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.253+0000 7f5f383af740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f75681b490 con 0x55f756826c30 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.253+0000 7f5f383af740 10 client.0 _ll_register_callbacks cb 0x7f5f1c00b7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.253+0000 7f5f2e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f75681ad90 0x55f756824640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.254+0000 7f5f2e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f75681ad90 0x55f756824640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34392/0 (socket says 172.21.15.69:34392) 2024-04-24T08:14:59.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.254+0000 7f5f2e000640 1 -- 172.21.15.69:0/3208571537 learned_addr learned my addr 172.21.15.69:0/3208571537 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.254+0000 7f5f2e000640 1 -- 172.21.15.69:0/3208571537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756826c30 msgr2=0x55f756819a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.254+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756809d50 0x55f7568947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.254+0000 7f5f2e000640 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756826c30 0x55f756819a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.254+0000 7f5f2e000640 1 -- 172.21.15.69:0/3208571537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756809d50 msgr2=0x55f7568947f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.254+0000 7f5f2e000640 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756809d50 0x55f7568947f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.254+0000 7f5f2e000640 1 -- 172.21.15.69:0/3208571537 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7564150f0 con 0x55f75681ad90 2024-04-24T08:14:59.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.254+0000 7f5f2e000640 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f75681ad90 0x55f756824640 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c01bf80 tx=0x7f5f1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31fdf7c7e935ae64 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.254+0000 7f5f0f400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f0f400640 1 -- 172.21.15.69:0/3208571537 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c038070 con 0x55f75681ad90 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f0f400640 1 -- 172.21.15.69:0/3208571537 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f5f1c0343f0 con 0x55f75681ad90 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f383af740 1 -- 172.21.15.69:0/3208571537 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f28037040 con 0x55f75681ad90 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f0f400640 1 -- 172.21.15.69:0/3208571537 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f5f1c042050 con 0x55f75681ad90 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f383af740 10 client.7338 fetch_fsmap learned FSMap version 52 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f383af740 1 -- 172.21.15.69:0/3208571537 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x55f756708810 con 0x55f75681ad90 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f0f400640 1 -- 172.21.15.69:0/3208571537 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f28037040 con 0x55f75681ad90 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f0f400640 1 -- 172.21.15.69:0/3208571537 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7f5f1c045020 con 0x55f75681ad90 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f383af740 10 client.7338 fetch_fsmap finished waiting for FSMap version 52 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f383af740 10 client.7338 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f383af740 20 client.7338 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:59.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f383af740 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55f7568252d0 0x55f756819fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f383af740 4 client.7338 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.255+0000 7f5f383af740 1 -- 172.21.15.69:0/3208571537 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f75674aa80 con 0x55f7568252d0 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.256+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55f7568252d0 0x55f756819fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.256+0000 7f5f2ea00640 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55f7568252d0 0x55f756819fa0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e214ad0beeb0129 server_cookie=e9707e1b0b100664 in_seq=0 out_seq=0 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.256+0000 7f5f0f400640 10 client.7338 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.256+0000 7f5f0f400640 1 -- 172.21.15.69:0/3208571537 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f75674aa80 con 0x55f7568252d0 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.256+0000 7f5f0f400640 10 client.7338 handle_command_reply: tid=0 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.256+0000 7f5f383af740 1 client.7338 shutdown 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.256+0000 7f5f383af740 1 -- 172.21.15.69:0/3208571537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f75681ad90 msgr2=0x55f756824640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.256+0000 7f5f383af740 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f75681ad90 0x55f756824640 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c01bf80 tx=0x7f5f1c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.256+0000 7f5f383af740 1 -- 172.21.15.69:0/3208571537 shutdown_connections 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f383af740 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x55f7568252d0 0x55f756819fa0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f383af740 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f75681ad90 0x55f756824640 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f383af740 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f756826c30 0x55f756819a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f383af740 1 --2- 172.21.15.69:0/3208571537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f756809d50 0x55f7568947f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f383af740 1 -- 172.21.15.69:0/3208571537 >> 172.21.15.69:0/3208571537 conn(0x55f756809610 msgr2=0x55f7568327f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f0f400640 0 client.7338 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:59.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f383af740 1 -- 172.21.15.69:0/3208571537 shutdown_connections 2024-04-24T08:14:59.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f383af740 1 -- 172.21.15.69:0/3208571537 wait complete. 2024-04-24T08:14:59.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f383af740 20 client.7338 trim_cache size 0 max 16384 2024-04-24T08:14:59.262 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:14:59.262 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-04-24T08:14:59.262 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-04-24T08:14:59.262 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:14:59.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f35600640 1 -- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f1006c8f0 msgr2=0x7f5f1006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f35600640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f1006c8f0 0x7f5f1006edb0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5f300af760 tx=0x7f5f20009040 comp rx=0 tx=0).stop 2024-04-24T08:14:59.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f35600640 1 -- 172.21.15.69:0/3963567841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f3015c820 msgr2=0x7f5f300ae7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.257+0000 7f5f35600640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f3015c820 0x7f5f300ae7a0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5f2802ace0 tx=0x7f5f28004a40 comp rx=0 tx=0).stop 2024-04-24T08:14:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.258+0000 7f5f35600640 1 -- 172.21.15.69:0/3963567841 shutdown_connections 2024-04-24T08:14:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.258+0000 7f5f35600640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f1006c8f0 0x7f5f1006edb0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.258+0000 7f5f35600640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f3015c820 0x7f5f300ae7a0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.258+0000 7f5f35600640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f300ce250 0x7f5f301e61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.258+0000 7f5f35600640 1 --2- 172.21.15.69:0/3963567841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f300b3e90 0x7f5f301576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.258+0000 7f5f35600640 1 -- 172.21.15.69:0/3963567841 >> 172.21.15.69:0/3963567841 conn(0x7f5f300d0350 msgr2=0x7f5f300b0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.258+0000 7f5f35600640 1 -- 172.21.15.69:0/3963567841 shutdown_connections 2024-04-24T08:14:59.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.258+0000 7f5f35600640 1 -- 172.21.15.69:0/3963567841 wait complete. 2024-04-24T08:14:59.277 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:14:59.278 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:14:59.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.471+0000 7f5f4b200640 1 Processor -- start 2024-04-24T08:14:59.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.471+0000 7f5f4b200640 1 -- start start 2024-04-24T08:14:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f4b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f4c156c60 0x7f5f4c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f4b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5f4c0b30f0 0x7f5f4c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f4b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f4c15d660 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f4b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5f4c06dc30 con 0x7f5f4c0b30f0 2024-04-24T08:14:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f4b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5f4c06df10 con 0x7f5f4c156c60 2024-04-24T08:14:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f49400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5f4c0b30f0 0x7f5f4c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f49400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5f4c0b30f0 0x7f5f4c15d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34404/0 (socket says 172.21.15.69:34404) 2024-04-24T08:14:59.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f49400640 1 -- 172.21.15.69:0/4185784793 learned_addr learned my addr 172.21.15.69:0/4185784793 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f49400640 1 -- 172.21.15.69:0/4185784793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f4c156c60 msgr2=0x7f5f4c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f49400640 1 --2- 172.21.15.69:0/4185784793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f4c156c60 0x7f5f4c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f49400640 1 -- 172.21.15.69:0/4185784793 >> v1:172.21.15.121:6789/0 conn(0x7f5f4c0ad0d0 legacy=0x7f5f4c1600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.473+0000 7f5f49400640 1 -- 172.21.15.69:0/4185784793 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4c06e1f0 con 0x7f5f4c0b30f0 2024-04-24T08:14:59.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.474+0000 7f5f49400640 1 --2- 172.21.15.69:0/4185784793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0b30f0 0x7f5f4c15d000 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009870 tx=0x7f5f3802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cdb0798cc349103 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.475+0000 7f5f48a00640 1 -- 172.21.15.69:0/4185784793 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f38037070 con 0x7f5f4c0b30f0 2024-04-24T08:14:59.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.475+0000 7f5f48a00640 1 -- 172.21.15.69:0/4185784793 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f38032440 con 0x7f5f4c0b30f0 2024-04-24T08:14:59.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.475+0000 7f5f48a00640 1 -- 172.21.15.69:0/4185784793 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f38032810 con 0x7f5f4c0b30f0 2024-04-24T08:14:59.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.475+0000 7f5f4b200640 1 -- 172.21.15.69:0/4185784793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0b30f0 msgr2=0x7f5f4c15d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.475+0000 7f5f4b200640 1 --2- 172.21.15.69:0/4185784793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0b30f0 0x7f5f4c15d000 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f5f38009870 tx=0x7f5f3802a200 comp rx=0 tx=0).stop 2024-04-24T08:14:59.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.476+0000 7f5f4b200640 1 -- 172.21.15.69:0/4185784793 shutdown_connections 2024-04-24T08:14:59.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.476+0000 7f5f4b200640 1 --2- 172.21.15.69:0/4185784793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0b30f0 0x7f5f4c15d000 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.476+0000 7f5f4b200640 1 --2- 172.21.15.69:0/4185784793 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f4c156c60 0x7f5f4c15cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.476+0000 7f5f4b200640 1 -- 172.21.15.69:0/4185784793 >> 172.21.15.69:0/4185784793 conn(0x7f5f4c0d0160 msgr2=0x7f5f4c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.476+0000 7f5f4b200640 1 -- 172.21.15.69:0/4185784793 shutdown_connections 2024-04-24T08:14:59.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.476+0000 7f5f4b200640 1 -- 172.21.15.69:0/4185784793 wait complete. 2024-04-24T08:14:59.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.477+0000 7f5f4b200640 1 Processor -- start 2024-04-24T08:14:59.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.478+0000 7f5f4b200640 1 -- start start 2024-04-24T08:14:59.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.478+0000 7f5f4b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0ad0d0 0x7f5f4c159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f4b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f4c0b30f0 0x7f5f4c1f7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f4b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f4c156c60 0x7f5f4c1555a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f4b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c0ad530 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f4b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c15d940 con 0x7f5f4c0b30f0 2024-04-24T08:14:59.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f4b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5f4c0ad6b0 con 0x7f5f4c156c60 2024-04-24T08:14:59.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f49e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0ad0d0 0x7f5f4c159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f49e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0ad0d0 0x7f5f4c159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34416/0 (socket says 172.21.15.69:34416) 2024-04-24T08:14:59.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f49e00640 1 -- 172.21.15.69:0/2285245385 learned_addr learned my addr 172.21.15.69:0/2285245385 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f4a800640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f4c156c60 0x7f5f4c1555a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f49400640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f4c0b30f0 0x7f5f4c1f7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f49e00640 1 -- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f4c156c60 msgr2=0x7f5f4c1555a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f49e00640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f4c156c60 0x7f5f4c1555a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f49e00640 1 -- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f4c0b30f0 msgr2=0x7f5f4c1f7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f49e00640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f4c0b30f0 0x7f5f4c1f7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.479+0000 7f5f49e00640 1 -- 172.21.15.69:0/2285245385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4c1f8390 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.480+0000 7f5f49e00640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0ad0d0 0x7f5f4c159070 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00b410 tx=0x7f5f3c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.480+0000 7f5f42000640 1 -- 172.21.15.69:0/2285245385 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c015020 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.480+0000 7f5f42000640 1 -- 172.21.15.69:0/2285245385 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f3c011850 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.480+0000 7f5f42000640 1 -- 172.21.15.69:0/2285245385 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c011c40 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.480+0000 7f5f4b200640 1 -- 172.21.15.69:0/2285245385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4c1f94b0 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.480+0000 7f5f4b200640 1 -- 172.21.15.69:0/2285245385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f4c158590 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.482+0000 7f5f4b200640 1 -- 172.21.15.69:0/2285245385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f4c075d50 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.482+0000 7f5f42000640 1 -- 172.21.15.69:0/2285245385 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5f3c018020 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.483+0000 7f5f42000640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f2c06c8f0 0x7f5f2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.483+0000 7f5f42000640 1 -- 172.21.15.69:0/2285245385 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f5f3c08ba70 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.484+0000 7f5f49400640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f2c06c8f0 0x7f5f2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.485+0000 7f5f49400640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f2c06c8f0 0x7f5f2c06edb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5f38034000 tx=0x7f5f3802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.488+0000 7f5f42000640 1 -- 172.21.15.69:0/2285245385 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5f3c01f020 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.730+0000 7f5f4b200640 1 -- 172.21.15.69:0/2285245385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f4c0cfed0 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.731+0000 7f5f42000640 1 -- 172.21.15.69:0/2285245385 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5188 (secure 0 0 0) 0x7f5f3c0172c0 con 0x7f5f4c0ad0d0 2024-04-24T08:14:59.733 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:14:59.733 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3504199143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3504199143},{"type":"v1","addr":"172.21.15.121:6837","nonce":3504199143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:14:32.246239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6792},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6792":{"gid":6792,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":23,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6792,"qdb_cluster":[6792]},"id":5}]} 2024-04-24T08:14:59.733 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 52 2024-04-24T08:14:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 -- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f2c06c8f0 msgr2=0x7f5f2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f2c06c8f0 0x7f5f2c06edb0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f5f38034000 tx=0x7f5f3802e000 comp rx=0 tx=0).stop 2024-04-24T08:14:59.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 -- 172.21.15.69:0/2285245385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0ad0d0 msgr2=0x7f5f4c159070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0ad0d0 0x7f5f4c159070 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c00b410 tx=0x7f5f3c00b8d0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 -- 172.21.15.69:0/2285245385 shutdown_connections 2024-04-24T08:14:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5f2c06c8f0 0x7f5f2c06edb0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5f4c156c60 0x7f5f4c1555a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f4c0b30f0 0x7f5f4c1f7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 --2- 172.21.15.69:0/2285245385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f4c0ad0d0 0x7f5f4c159070 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 -- 172.21.15.69:0/2285245385 >> 172.21.15.69:0/2285245385 conn(0x7f5f4c0d0160 msgr2=0x7f5f4c160070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 -- 172.21.15.69:0/2285245385 shutdown_connections 2024-04-24T08:14:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.734+0000 7f5f4b200640 1 -- 172.21.15.69:0/2285245385 wait complete. 2024-04-24T08:14:59.755 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-24T08:14:59.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.955+0000 7f7a54200640 1 Processor -- start 2024-04-24T08:14:59.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.955+0000 7f7a54200640 1 -- start start 2024-04-24T08:14:59.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.956+0000 7f7a54200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15c820 0x7f7a4c0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.956+0000 7f7a54200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a4c15cc00 con 0x7f7a4c0b46a0 2024-04-24T08:14:59.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.956+0000 7f7a54200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a4c06ed30 con 0x7f7a4c0ce250 2024-04-24T08:14:59.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.956+0000 7f7a54200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4c0b4e10 con 0x7f7a4c15c820 2024-04-24T08:14:59.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.957+0000 7f7a50e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7a4c0b46a0 0x7f7a4c06f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53120/0 (socket says 172.21.15.69:53120) 2024-04-24T08:14:59.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.957+0000 7f7a50e00640 1 -- 172.21.15.69:0/2687284423 learned_addr learned my addr 172.21.15.69:0/2687284423 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:14:59.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.957+0000 7f7a51800640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15c820 0x7f7a4c0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.957+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925843697 0 0) 0x7f7a4c15cc00 con 0x7f7a4c0b46a0 2024-04-24T08:14:59.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.957+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a34003520 con 0x7f7a4c0b46a0 2024-04-24T08:14:59.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 793209934 0 0) 0x7f7a34003520 con 0x7f7a4c0b46a0 2024-04-24T08:14:59.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15c820 msgr2=0x7f7a4c0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15c820 0x7f7a4c0b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 >> v1:172.21.15.121:6789/0 conn(0x7f7a4c0ce250 legacy=0x7f7a4c0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4c0b50f0 con 0x7f7a4c0b46a0 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f7a38001670 con 0x7f7a4c0b46a0 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 >> v1:172.21.15.69:6789/0 conn(0x7f7a4c0b46a0 legacy=0x7f7a4c06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15c820 0x7f7a340039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 0x7f7a34004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a34004d40 0x7f7a340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c0b4e10 con 0x7f7a34003fb0 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3400bb00 con 0x7f7a34004d40 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a3400bd00 con 0x7f7a4c15c820 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.958+0000 7f7a50e00640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 0x7f7a34004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.959+0000 7f7a52200640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a34004d40 0x7f7a340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.959+0000 7f7a50e00640 1 -- 172.21.15.69:0/2687284423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15c820 msgr2=0x7f7a340039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.959+0000 7f7a50e00640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15c820 0x7f7a340039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.959+0000 7f7a50e00640 1 -- 172.21.15.69:0/2687284423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a34004d40 msgr2=0x7f7a340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.959+0000 7f7a50e00640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a34004d40 0x7f7a340051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.959+0000 7f7a50e00640 1 -- 172.21.15.69:0/2687284423 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a34003520 con 0x7f7a34003fb0 2024-04-24T08:14:59.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.960+0000 7f7a50e00640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 0x7f7a34004800 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c110 tx=0x7f7a34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971b0bf684f6132 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.960+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38007c40 con 0x7f7a34003fb0 2024-04-24T08:14:59.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.960+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a38005af0 con 0x7f7a34003fb0 2024-04-24T08:14:59.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.960+0000 7f7a4be00640 1 -- 172.21.15.69:0/2687284423 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38007070 con 0x7f7a34003fb0 2024-04-24T08:14:59.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.961+0000 7f7a54200640 1 -- 172.21.15.69:0/2687284423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 msgr2=0x7f7a34004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.961+0000 7f7a54200640 1 --2- 172.21.15.69:0/2687284423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 0x7f7a34004800 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c110 tx=0x7f7a34005830 comp rx=0 tx=0).stop 2024-04-24T08:14:59.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.961+0000 7f7a52200640 1 -- 172.21.15.69:0/2687284423 reap_dead start 2024-04-24T08:14:59.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.961+0000 7f7a54200640 1 -- 172.21.15.69:0/2687284423 shutdown_connections 2024-04-24T08:14:59.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.961+0000 7f7a54200640 1 -- 172.21.15.69:0/2687284423 >> 172.21.15.69:0/2687284423 conn(0x7f7a4c0d0350 msgr2=0x7f7a4c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.961+0000 7f7a54200640 1 -- 172.21.15.69:0/2687284423 shutdown_connections 2024-04-24T08:14:59.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.961+0000 7f7a54200640 1 -- 172.21.15.69:0/2687284423 wait complete. 2024-04-24T08:14:59.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.963+0000 7f7a54200640 1 Processor -- start 2024-04-24T08:14:59.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.963+0000 7f7a54200640 1 -- start start 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a54200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 0x7f7a4c15f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a54200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15ce00 0x7f7a4c0b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a54200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c150a80 0x7f7a4c0b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a54200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c0b4b00 con 0x7f7a34003fb0 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a54200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c06f2b0 con 0x7f7a4c150a80 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a54200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4c06fee0 con 0x7f7a4c15ce00 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a51800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 0x7f7a4c15f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a51800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 0x7f7a4c15f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34432/0 (socket says 172.21.15.69:34432) 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a51800640 1 -- 172.21.15.69:0/410076212 learned_addr learned my addr 172.21.15.69:0/410076212 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a51800640 1 -- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15ce00 msgr2=0x7f7a4c0b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a51800640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15ce00 0x7f7a4c0b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a51800640 1 -- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c150a80 msgr2=0x7f7a4c0b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a51800640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c150a80 0x7f7a4c0b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.964+0000 7f7a51800640 1 -- 172.21.15.69:0/410076212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4c15d350 con 0x7f7a34003fb0 2024-04-24T08:14:59.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.965+0000 7f7a51800640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 0x7f7a4c15f490 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000e820 tx=0x7f7a4000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.965+0000 7f7a49600640 1 -- 172.21.15.69:0/410076212 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40017070 con 0x7f7a34003fb0 2024-04-24T08:14:59.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.965+0000 7f7a49600640 1 -- 172.21.15.69:0/410076212 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a4000c8e0 con 0x7f7a34003fb0 2024-04-24T08:14:59.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.965+0000 7f7a54200640 1 -- 172.21.15.69:0/410076212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a4c15d630 con 0x7f7a34003fb0 2024-04-24T08:14:59.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.965+0000 7f7a49600640 1 -- 172.21.15.69:0/410076212 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a4000ccb0 con 0x7f7a34003fb0 2024-04-24T08:14:59.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.965+0000 7f7a54200640 1 -- 172.21.15.69:0/410076212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a4c15dbf0 con 0x7f7a34003fb0 2024-04-24T08:14:59.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.967+0000 7f7a49600640 1 -- 172.21.15.69:0/410076212 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7a4001a070 con 0x7f7a34003fb0 2024-04-24T08:14:59.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.968+0000 7f7a49600640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a2c06c8f0 0x7f7a2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.968+0000 7f7a49600640 1 -- 172.21.15.69:0/410076212 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f7a4008be00 con 0x7f7a34003fb0 2024-04-24T08:14:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.969+0000 7f7a57034740 1 Processor -- start 2024-04-24T08:14:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.969+0000 7f7a57034740 1 -- start start 2024-04-24T08:14:59.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.969+0000 7f7a50e00640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a2c06c8f0 0x7f7a2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a57034740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db63fff0 0x5579db65b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a57034740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db6403d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a57034740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db6511b0 0x5579db651630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a57034740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5579db57a190 con 0x5579db63fff0 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a57034740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5579db520030 con 0x5579db65de30 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a57034740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5579db576150 con 0x5579db6511b0 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a51800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db63fff0 0x5579db65b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a51800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db63fff0 0x5579db65b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34444/0 (socket says 172.21.15.69:34444) 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a51800640 1 -- 172.21.15.69:0/969159195 learned_addr learned my addr 172.21.15.69:0/969159195 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a50e00640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a2c06c8f0 0x7f7a2c06edb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7a38032000 tx=0x7f7a38009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a51800640 1 -- 172.21.15.69:0/969159195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db6511b0 msgr2=0x5579db651630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a51800640 1 --2- 172.21.15.69:0/969159195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db6511b0 0x5579db651630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a51800640 1 -- 172.21.15.69:0/969159195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 msgr2=0x5579db6403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a51800640 1 --2- 172.21.15.69:0/969159195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db6403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.970+0000 7f7a51800640 1 -- 172.21.15.69:0/969159195 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579db508b60 con 0x5579db63fff0 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.971+0000 7f7a51800640 1 --2- 172.21.15.69:0/969159195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db63fff0 0x5579db65b9d0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000f010 tx=0x7f7a40009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678c53503d33b795 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.971+0000 7f7a3e000640 1 -- 172.21.15.69:0/969159195 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40021070 con 0x5579db63fff0 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.971+0000 7f7a3e000640 1 -- 172.21.15.69:0/969159195 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a4000ce50 con 0x5579db63fff0 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.971+0000 7f7a3e000640 1 -- 172.21.15.69:0/969159195 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a400560c0 con 0x5579db63fff0 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.972+0000 7f7a57034740 1 -- 172.21.15.69:0/969159195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db63fff0 msgr2=0x5579db65b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.972+0000 7f7a57034740 1 --2- 172.21.15.69:0/969159195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db63fff0 0x5579db65b9d0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000f010 tx=0x7f7a40009190 comp rx=0 tx=0).stop 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.972+0000 7f7a57034740 1 -- 172.21.15.69:0/969159195 shutdown_connections 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.972+0000 7f7a57034740 1 --2- 172.21.15.69:0/969159195 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db6511b0 0x5579db651630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.972+0000 7f7a57034740 1 --2- 172.21.15.69:0/969159195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db6403d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.972+0000 7f7a57034740 1 --2- 172.21.15.69:0/969159195 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db63fff0 0x5579db65b9d0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.972+0000 7f7a57034740 1 -- 172.21.15.69:0/969159195 >> 172.21.15.69:0/969159195 conn(0x5579db612600 msgr2=0x5579db65ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.972+0000 7f7a57034740 1 -- 172.21.15.69:0/969159195 shutdown_connections 2024-04-24T08:14:59.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.972+0000 7f7a57034740 1 -- 172.21.15.69:0/969159195 wait complete. 2024-04-24T08:14:59.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.972+0000 7f7a57034740 1 -- start start 2024-04-24T08:14:59.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.973+0000 7f7a57034740 1 Processor -- start 2024-04-24T08:14:59.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a57034740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db6511b0 0x5579db670ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a57034740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db674310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a57034740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db643810 0x5579db653650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a57034740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5579db63dc20 con 0x5579db643810 2024-04-24T08:14:59.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a57034740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5579db533500 con 0x5579db65de30 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a57034740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5579db610430 con 0x5579db6511b0 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a57034740 10 client.0 _ll_register_callbacks cb 0x5579db570270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a52200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db643810 0x5579db653650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a52200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db643810 0x5579db653650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34450/0 (socket says 172.21.15.69:34450) 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a52200640 1 -- 172.21.15.69:0/3284443250 learned_addr learned my addr 172.21.15.69:0/3284443250 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a51800640 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db6511b0 0x5579db670ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a50e00640 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db674310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a52200640 1 -- 172.21.15.69:0/3284443250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db6511b0 msgr2=0x5579db670ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a52200640 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db6511b0 0x5579db670ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a52200640 1 -- 172.21.15.69:0/3284443250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 msgr2=0x5579db674310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a52200640 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db674310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a52200640 1 -- 172.21.15.69:0/3284443250 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579db3587c0 con 0x5579db643810 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a51800640 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db6511b0 0x5579db670ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.974+0000 7f7a50e00640 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db674310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:59.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.975+0000 7f7a52200640 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db643810 0x5579db653650 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c06d4c0 tx=0x7f7a4400d5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e791133c2ccb0302 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.975+0000 7f7a32a00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.975+0000 7f7a32a00640 1 -- 172.21.15.69:0/3284443250 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a44018060 con 0x5579db643810 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.975+0000 7f7a32a00640 1 -- 172.21.15.69:0/3284443250 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f7a44003b70 con 0x5579db643810 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.975+0000 7f7a57034740 1 -- 172.21.15.69:0/3284443250 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a4000ce50 con 0x5579db643810 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.975+0000 7f7a32a00640 1 -- 172.21.15.69:0/3284443250 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f7a4401e050 con 0x5579db643810 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.975+0000 7f7a57034740 10 client.7356 fetch_fsmap learned FSMap version 52 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.975+0000 7f7a57034740 1 -- 172.21.15.69:0/3284443250 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x5579db3b8f50 con 0x5579db643810 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.976+0000 7f7a32a00640 1 -- 172.21.15.69:0/3284443250 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a4000ce50 con 0x5579db643810 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.976+0000 7f7a32a00640 1 -- 172.21.15.69:0/3284443250 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7f7a44021020 con 0x5579db643810 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.976+0000 7f7a57034740 10 client.7356 fetch_fsmap finished waiting for FSMap version 52 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.976+0000 7f7a57034740 10 client.7356 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.976+0000 7f7a57034740 20 client.7356 populate_metadata read hostname 'smithi069' 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.976+0000 7f7a57034740 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5579db641120 0x5579db677590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.976+0000 7f7a57034740 4 client.7356 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:14:59.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.976+0000 7f7a57034740 1 -- 172.21.15.69:0/3284443250 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579db587bb0 con 0x5579db641120 2024-04-24T08:14:59.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.976+0000 7f7a51800640 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5579db641120 0x5579db677590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.977+0000 7f7a51800640 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5579db641120 0x5579db677590 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c2563db44ea9ba9 server_cookie=7eb7983193f55eae in_seq=0 out_seq=0 2024-04-24T08:14:59.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.977+0000 7f7a32a00640 10 client.7356 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.977+0000 7f7a32a00640 1 -- 172.21.15.69:0/3284443250 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5579db587bb0 con 0x5579db641120 2024-04-24T08:14:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.977+0000 7f7a32a00640 10 client.7356 handle_command_reply: tid=0 2024-04-24T08:14:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.977+0000 7f7a57034740 1 client.7356 shutdown 2024-04-24T08:14:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 1 -- 172.21.15.69:0/3284443250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db643810 msgr2=0x5579db653650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db643810 0x5579db653650 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c06d4c0 tx=0x7f7a4400d5e0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 1 -- 172.21.15.69:0/3284443250 shutdown_connections 2024-04-24T08:14:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5579db641120 0x5579db677590 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db643810 0x5579db653650 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db674310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 1 --2- 172.21.15.69:0/3284443250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db6511b0 0x5579db670ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 1 -- 172.21.15.69:0/3284443250 >> 172.21.15.69:0/3284443250 conn(0x5579db63fff0 msgr2=0x5579db63dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a32a00640 0 client.7356 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:14:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 1 -- 172.21.15.69:0/3284443250 shutdown_connections 2024-04-24T08:14:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 1 -- 172.21.15.69:0/3284443250 wait complete. 2024-04-24T08:14:59.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.978+0000 7f7a57034740 20 client.7356 trim_cache size 0 max 16384 2024-04-24T08:14:59.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.993+0000 7f7a57034740 1 Processor -- start 2024-04-24T08:14:59.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.993+0000 7f7a57034740 1 -- start start 2024-04-24T08:14:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.993+0000 7f7a57034740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db643810 0x5579db63d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.993+0000 7f7a57034740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db67d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.993+0000 7f7a57034740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db652c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.993+0000 7f7a57034740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c0c6a30 con 0x5579db6511b0 2024-04-24T08:14:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.993+0000 7f7a57034740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c1536d0 con 0x5579db65de30 2024-04-24T08:14:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.993+0000 7f7a57034740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4c0c63e0 con 0x5579db643810 2024-04-24T08:14:59.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a50e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db67d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a50e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db67d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34464/0 (socket says 172.21.15.69:34464) 2024-04-24T08:14:59.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a50e00640 1 -- 172.21.15.69:0/1025227525 learned_addr learned my addr 172.21.15.69:0/1025227525 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:14:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a51800640 1 --2- 172.21.15.69:0/1025227525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db643810 0x5579db63d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a50e00640 1 -- 172.21.15.69:0/1025227525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db643810 msgr2=0x5579db63d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a52200640 1 --2- 172.21.15.69:0/1025227525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db652c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a50e00640 1 --2- 172.21.15.69:0/1025227525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db643810 0x5579db63d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a50e00640 1 -- 172.21.15.69:0/1025227525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 msgr2=0x5579db652c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a50e00640 1 --2- 172.21.15.69:0/1025227525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db652c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a50e00640 1 -- 172.21.15.69:0/1025227525 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5579db24c0b0 con 0x5579db6511b0 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.994+0000 7f7a51800640 1 --2- 172.21.15.69:0/1025227525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db643810 0x5579db63d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.995+0000 7f7a50e00640 1 --2- 172.21.15.69:0/1025227525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db67d8a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7a38001490 tx=0x7f7a3802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7e6f61c18b6e60b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.995+0000 7f7a3d600640 1 -- 172.21.15.69:0/1025227525 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3804f070 con 0x5579db6511b0 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.995+0000 7f7a3d600640 1 -- 172.21.15.69:0/1025227525 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a38001640 con 0x5579db6511b0 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.995+0000 7f7a3d600640 1 -- 172.21.15.69:0/1025227525 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38037040 con 0x5579db6511b0 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.996+0000 7f7a57034740 1 -- 172.21.15.69:0/1025227525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 msgr2=0x5579db67d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.996+0000 7f7a57034740 1 --2- 172.21.15.69:0/1025227525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db67d8a0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f7a38001490 tx=0x7f7a3802e000 comp rx=0 tx=0).stop 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.996+0000 7f7a57034740 1 -- 172.21.15.69:0/1025227525 shutdown_connections 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.996+0000 7f7a57034740 1 --2- 172.21.15.69:0/1025227525 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db67d8a0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.996+0000 7f7a57034740 1 --2- 172.21.15.69:0/1025227525 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db65de30 0x5579db652c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.996+0000 7f7a57034740 1 --2- 172.21.15.69:0/1025227525 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db643810 0x5579db63d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:59.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.996+0000 7f7a57034740 1 -- 172.21.15.69:0/1025227525 >> 172.21.15.69:0/1025227525 conn(0x5579db63fff0 msgr2=0x5579db670cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:59.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.996+0000 7f7a57034740 1 -- 172.21.15.69:0/1025227525 shutdown_connections 2024-04-24T08:14:59.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.996+0000 7f7a57034740 1 -- 172.21.15.69:0/1025227525 wait complete. 2024-04-24T08:15:00.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.996+0000 7f7a57034740 1 -- start start 2024-04-24T08:15:00.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.997+0000 7f7a57034740 1 Processor -- start 2024-04-24T08:15:00.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.997+0000 7f7a57034740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db643810 0x5579db642080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.997+0000 7f7a57034740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db65e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a57034740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db65de30 0x5579db652c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a57034740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5579db640fa0 con 0x5579db6511b0 2024-04-24T08:15:00.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a57034740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5579db6520e0 con 0x5579db643810 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a57034740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5579db6415f0 con 0x5579db65de30 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a57034740 10 client.0 _ll_register_callbacks cb 0x7f7a44007de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a50e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db65e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a50e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db65e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34472/0 (socket says 172.21.15.69:34472) 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a50e00640 1 -- 172.21.15.69:0/1181961222 learned_addr learned my addr 172.21.15.69:0/1181961222 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a50e00640 1 -- 172.21.15.69:0/1181961222 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db65de30 msgr2=0x5579db652c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a50e00640 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db65de30 0x5579db652c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a51800640 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db643810 0x5579db642080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a50e00640 1 -- 172.21.15.69:0/1181961222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db643810 msgr2=0x5579db642080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a50e00640 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db643810 0x5579db642080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.998+0000 7f7a50e00640 1 -- 172.21.15.69:0/1181961222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5579db6508d0 con 0x5579db6511b0 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a50e00640 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db65e210 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c140 tx=0x7f7a3802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a147976a41474cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a32a00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:15:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a32a00640 1 -- 172.21.15.69:0/1181961222 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3804f040 con 0x5579db6511b0 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a32a00640 1 -- 172.21.15.69:0/1181961222 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f7a38055bc0 con 0x5579db6511b0 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a57034740 1 -- 172.21.15.69:0/1181961222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a38001640 con 0x5579db6511b0 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a32a00640 1 -- 172.21.15.69:0/1181961222 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f7a38007730 con 0x5579db6511b0 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a57034740 10 client.7362 fetch_fsmap learned FSMap version 52 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a57034740 1 -- 172.21.15.69:0/1181961222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) -- 0x5579db6de910 con 0x5579db6511b0 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a32a00640 1 -- 172.21.15.69:0/1181961222 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a38001640 con 0x5579db6511b0 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a32a00640 1 -- 172.21.15.69:0/1181961222 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 52) ==== 3494+0+0 (secure 0 0 0) 0x7f7a38052070 con 0x5579db6511b0 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:14:59.999+0000 7f7a57034740 10 client.7362 fetch_fsmap finished waiting for FSMap version 52 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.000+0000 7f7a57034740 10 client.7362 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.000+0000 7f7a57034740 20 client.7362 populate_metadata read hostname 'smithi069' 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.000+0000 7f7a57034740 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5579db6531b0 0x5579db63bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.000+0000 7f7a57034740 4 client.7362 mds_command: new command op to 6792 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-24T08:15:00.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.000+0000 7f7a57034740 1 -- 172.21.15.69:0/1181961222 --> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5579db581a20 con 0x5579db6531b0 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.000+0000 7f7a51800640 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5579db6531b0 0x5579db63bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.000+0000 7f7a51800640 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5579db6531b0 0x5579db63bcf0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3240a3781a565f88 server_cookie=ae9812f349aad52b in_seq=0 out_seq=0 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.000+0000 7f7a32a00640 10 client.7362 ms_handle_connect on v2:172.21.15.69:6834/2999903126 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a32a00640 1 -- 172.21.15.69:0/1181961222 <== mds.0 v2:172.21.15.69:6834/2999903126 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5579db581a20 con 0x5579db6531b0 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a32a00640 10 client.7362 handle_command_reply: tid=0 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 client.7362 shutdown 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 -- 172.21.15.69:0/1181961222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 msgr2=0x5579db65e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db65e210 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c140 tx=0x7f7a3802c000 comp rx=0 tx=0).stop 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 -- 172.21.15.69:0/1181961222 shutdown_connections 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.69:6834/2999903126,v1:172.21.15.69:6835/2999903126] conn(0x5579db6531b0 0x5579db63bcf0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5579db6511b0 0x5579db65e210 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5579db65de30 0x5579db652c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a32a00640 0 client.7362 ms_handle_reset on v2:172.21.15.69:6834/2999903126 2024-04-24T08:15:00.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 --2- 172.21.15.69:0/1181961222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579db643810 0x5579db642080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 -- 172.21.15.69:0/1181961222 >> 172.21.15.69:0/1181961222 conn(0x5579db63fff0 msgr2=0x5579db63d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:00.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 -- 172.21.15.69:0/1181961222 shutdown_connections 2024-04-24T08:15:00.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 1 -- 172.21.15.69:0/1181961222 wait complete. 2024-04-24T08:15:00.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.001+0000 7f7a57034740 20 client.7362 trim_cache size 0 max 16384 2024-04-24T08:15:00.007 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:00.007 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-04-24T08:15:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 -- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a2c06c8f0 msgr2=0x7f7a2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a2c06c8f0 0x7f7a2c06edb0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7a38032000 tx=0x7f7a38009030 comp rx=0 tx=0).stop 2024-04-24T08:15:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 -- 172.21.15.69:0/410076212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 msgr2=0x7f7a4c15f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 0x7f7a4c15f490 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000e820 tx=0x7f7a4000ece0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 -- 172.21.15.69:0/410076212 shutdown_connections 2024-04-24T08:15:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a2c06c8f0 0x7f7a2c06edb0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c150a80 0x7f7a4c0b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a4c15ce00 0x7f7a4c0b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 --2- 172.21.15.69:0/410076212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a34003fb0 0x7f7a4c15f490 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 -- 172.21.15.69:0/410076212 >> 172.21.15.69:0/410076212 conn(0x7f7a4c0d0350 msgr2=0x7f7a4c160c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 -- 172.21.15.69:0/410076212 shutdown_connections 2024-04-24T08:15:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.002+0000 7f7a54200640 1 -- 172.21.15.69:0/410076212 wait complete. 2024-04-24T08:15:00.020 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T08:15:00.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.214+0000 7f93f1e00640 1 Processor -- start 2024-04-24T08:15:00.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.214+0000 7f93f1e00640 1 -- start start 2024-04-24T08:15:00.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.216+0000 7f93f1e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f93ec0d4d20 0x7f93ec0b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.216+0000 7f93f1e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93ec0d2ff0 0x7f93ec0b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.216+0000 7f93f1e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93ec0b1930 con 0x7f93ec15ea10 2024-04-24T08:15:00.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.216+0000 7f93f1e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f93ec0b1700 con 0x7f93ec0d4d20 2024-04-24T08:15:00.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.216+0000 7f93f1e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f93ec0b2540 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.216+0000 7f93ebe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f93ec0d4d20 0x7f93ec0b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.216+0000 7f93ebe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f93ec0d4d20 0x7f93ec0b11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34482/0 (socket says 172.21.15.69:34482) 2024-04-24T08:15:00.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.216+0000 7f93ebe00640 1 -- 172.21.15.69:0/3792664724 learned_addr learned my addr 172.21.15.69:0/3792664724 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:00.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.217+0000 7f93f1400640 1 --2- 172.21.15.69:0/3792664724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93ec0d2ff0 0x7f93ec0b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.217+0000 7f93ebe00640 1 -- 172.21.15.69:0/3792664724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93ec0d2ff0 msgr2=0x7f93ec0b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:00.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.217+0000 7f93ebe00640 1 --2- 172.21.15.69:0/3792664724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93ec0d2ff0 0x7f93ec0b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.217+0000 7f93ebe00640 1 -- 172.21.15.69:0/3792664724 >> v1:172.21.15.121:6789/0 conn(0x7f93ec15ea10 legacy=0x7f93ec0d4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:00.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.217+0000 7f93ebe00640 1 -- 172.21.15.69:0/3792664724 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ec0b1c10 con 0x7f93ec0d4d20 2024-04-24T08:15:00.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.218+0000 7f93ebe00640 1 --2- 172.21.15.69:0/3792664724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d4d20 0x7f93ec0b11c0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009e50 tx=0x7f93d402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7215204649a12525 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:00.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.218+0000 7f93eb400640 1 -- 172.21.15.69:0/3792664724 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d4036070 con 0x7f93ec0d4d20 2024-04-24T08:15:00.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.218+0000 7f93eb400640 1 -- 172.21.15.69:0/3792664724 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d4031600 con 0x7f93ec0d4d20 2024-04-24T08:15:00.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.218+0000 7f93eb400640 1 -- 172.21.15.69:0/3792664724 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40319d0 con 0x7f93ec0d4d20 2024-04-24T08:15:00.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.219+0000 7f93f1e00640 1 -- 172.21.15.69:0/3792664724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d4d20 msgr2=0x7f93ec0b11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.219+0000 7f93f1e00640 1 --2- 172.21.15.69:0/3792664724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d4d20 0x7f93ec0b11c0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f93d4009e50 tx=0x7f93d402a6c0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.219+0000 7f93f1e00640 1 -- 172.21.15.69:0/3792664724 shutdown_connections 2024-04-24T08:15:00.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.219+0000 7f93f1e00640 1 --2- 172.21.15.69:0/3792664724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93ec0d2ff0 0x7f93ec0b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.219+0000 7f93f1e00640 1 --2- 172.21.15.69:0/3792664724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d4d20 0x7f93ec0b11c0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.219+0000 7f93f1e00640 1 -- 172.21.15.69:0/3792664724 >> 172.21.15.69:0/3792664724 conn(0x7f93ec08dfb0 msgr2=0x7f93ec09d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:00.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.219+0000 7f93f1e00640 1 -- 172.21.15.69:0/3792664724 shutdown_connections 2024-04-24T08:15:00.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.219+0000 7f93f1e00640 1 -- 172.21.15.69:0/3792664724 wait complete. 2024-04-24T08:15:00.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.221+0000 7f93f1e00640 1 Processor -- start 2024-04-24T08:15:00.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.221+0000 7f93f1e00640 1 -- start start 2024-04-24T08:15:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.222+0000 7f93f1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d2ff0 0x7f93ec0c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.222+0000 7f93f1e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93ec0d4d20 0x7f93ec0b98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.222+0000 7f93f1e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93ec15ea10 0x7f93ec14de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f1e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f93ec0d3460 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f1e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f93ec0b21d0 con 0x7f93ec0d4d20 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f1e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f93ec15edf0 con 0x7f93ec15ea10 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d2ff0 0x7f93ec0c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d2ff0 0x7f93ec0c7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34486/0 (socket says 172.21.15.69:34486) 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f0a00640 1 -- 172.21.15.69:0/1478527256 learned_addr learned my addr 172.21.15.69:0/1478527256 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f0a00640 1 -- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93ec15ea10 msgr2=0x7f93ec14de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f0a00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93ec15ea10 0x7f93ec14de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f0a00640 1 -- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93ec0d4d20 msgr2=0x7f93ec0b98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f0a00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93ec0d4d20 0x7f93ec0b98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.223+0000 7f93f0a00640 1 -- 172.21.15.69:0/1478527256 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93ec160c60 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.224+0000 7f93f0a00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d2ff0 0x7f93ec0c7ee0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00b420 tx=0x7f93dc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:00.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.224+0000 7f93e8c00640 1 -- 172.21.15.69:0/1478527256 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc011830 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.224+0000 7f93e8c00640 1 -- 172.21.15.69:0/1478527256 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93dc016070 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.224+0000 7f93e8c00640 1 -- 172.21.15.69:0/1478527256 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc011c10 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.224+0000 7f93f1e00640 1 -- 172.21.15.69:0/1478527256 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93ec160f40 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.224+0000 7f93f1e00640 1 -- 172.21.15.69:0/1478527256 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93ec161500 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.225+0000 7f93f1e00640 1 -- 172.21.15.69:0/1478527256 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93ec075d50 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.226+0000 7f93e8c00640 1 -- 172.21.15.69:0/1478527256 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f93dc01b460 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.227+0000 7f93e8c00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f93c806c8f0 0x7f93c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.227+0000 7f93e8c00640 1 -- 172.21.15.69:0/1478527256 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f93dc08b7f0 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.228+0000 7f93ebe00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f93c806c8f0 0x7f93c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.229+0000 7f93ebe00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f93c806c8f0 0x7f93c806edb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f93d4033000 tx=0x7f93d402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:00.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.231+0000 7f93e8c00640 1 -- 172.21.15.69:0/1478527256 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f93dc020020 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.470+0000 7f93f1e00640 1 -- 172.21.15.69:0/1478527256 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f93ec0b3c90 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.471+0000 7f93e8c00640 1 -- 172.21.15.69:0/1478527256 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f93dc059ab0 con 0x7f93ec0d2ff0 2024-04-24T08:15:00.473 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:00.473 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T08:15:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.474+0000 7f93f1e00640 1 -- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f93c806c8f0 msgr2=0x7f93c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.474+0000 7f93f1e00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f93c806c8f0 0x7f93c806edb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f93d4033000 tx=0x7f93d402e000 comp rx=0 tx=0).stop 2024-04-24T08:15:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.474+0000 7f93f1e00640 1 -- 172.21.15.69:0/1478527256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d2ff0 msgr2=0x7f93ec0c7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.474+0000 7f93f1e00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d2ff0 0x7f93ec0c7ee0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00b420 tx=0x7f93dc00b8e0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.474+0000 7f93f1e00640 1 -- 172.21.15.69:0/1478527256 shutdown_connections 2024-04-24T08:15:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.474+0000 7f93f1e00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f93c806c8f0 0x7f93c806edb0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.474+0000 7f93f1e00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93ec15ea10 0x7f93ec14de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.474+0000 7f93f1e00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93ec0d4d20 0x7f93ec0b98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.474+0000 7f93f1e00640 1 --2- 172.21.15.69:0/1478527256 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f93ec0d2ff0 0x7f93ec0c7ee0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.474+0000 7f93f1e00640 1 -- 172.21.15.69:0/1478527256 >> 172.21.15.69:0/1478527256 conn(0x7f93ec08dfb0 msgr2=0x7f93ec0d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.475+0000 7f93f1e00640 1 -- 172.21.15.69:0/1478527256 shutdown_connections 2024-04-24T08:15:00.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.475+0000 7f93f1e00640 1 -- 172.21.15.69:0/1478527256 wait complete. 2024-04-24T08:15:00.495 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-24T08:15:00.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:15:00.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.693+0000 7f62abe00640 1 Processor -- start 2024-04-24T08:15:00.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.693+0000 7f62abe00640 1 -- start start 2024-04-24T08:15:00.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.695+0000 7f62abe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f62ac15e960 0x7f62ac162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.695+0000 7f62abe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac162980 0x7f62ac06cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.695+0000 7f62abe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f62ac06d4f0 0x7f62ac0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.695+0000 7f62abe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f62ac15ed40 con 0x7f62ac15e960 2024-04-24T08:15:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.695+0000 7f62abe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f62ac0cf050 con 0x7f62ac06d4f0 2024-04-24T08:15:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.695+0000 7f62abe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62ac162e20 con 0x7f62ac162980 2024-04-24T08:15:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.695+0000 7f62aaa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f62ac15e960 0x7f62ac162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.695+0000 7f62aaa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f62ac15e960 0x7f62ac162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34500/0 (socket says 172.21.15.69:34500) 2024-04-24T08:15:00.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.695+0000 7f62aaa00640 1 -- 172.21.15.69:0/442923101 learned_addr learned my addr 172.21.15.69:0/442923101 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.696+0000 7f62ab400640 1 --2- 172.21.15.69:0/442923101 >> v2:172.21.15.121:3300/0 conn(0x7f62ac06d4f0 0x7f62ac0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.696+0000 7f62aa000640 1 --2- 172.21.15.69:0/442923101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac162980 0x7f62ac06cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.696+0000 7f62aaa00640 1 -- 172.21.15.69:0/442923101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac162980 msgr2=0x7f62ac06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.696+0000 7f62aaa00640 1 --2- 172.21.15.69:0/442923101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac162980 0x7f62ac06cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.696+0000 7f62aaa00640 1 -- 172.21.15.69:0/442923101 >> v2:172.21.15.121:3300/0 conn(0x7f62ac06d4f0 msgr2=0x7f62ac0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.696+0000 7f62aaa00640 1 --2- 172.21.15.69:0/442923101 >> v2:172.21.15.121:3300/0 conn(0x7f62ac06d4f0 0x7f62ac0cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.696+0000 7f62aaa00640 1 -- 172.21.15.69:0/442923101 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ac163100 con 0x7f62ac15e960 2024-04-24T08:15:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.696+0000 7f62ab400640 1 --2- 172.21.15.69:0/442923101 >> v2:172.21.15.121:3300/0 conn(0x7f62ac06d4f0 0x7f62ac0cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.696+0000 7f62aa000640 1 --2- 172.21.15.69:0/442923101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac162980 0x7f62ac06cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.697+0000 7f62aaa00640 1 --2- 172.21.15.69:0/442923101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac15e960 0x7f62ac162440 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0c7fc0 tx=0x7f629402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7275b3c0510334cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.697+0000 7f62a9600640 1 -- 172.21.15.69:0/442923101 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294037070 con 0x7f62ac15e960 2024-04-24T08:15:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.697+0000 7f62a9600640 1 -- 172.21.15.69:0/442923101 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6294032440 con 0x7f62ac15e960 2024-04-24T08:15:00.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.697+0000 7f62a9600640 1 -- 172.21.15.69:0/442923101 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6294032810 con 0x7f62ac15e960 2024-04-24T08:15:00.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.698+0000 7f62abe00640 1 -- 172.21.15.69:0/442923101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac15e960 msgr2=0x7f62ac162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.698+0000 7f62abe00640 1 --2- 172.21.15.69:0/442923101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac15e960 0x7f62ac162440 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0c7fc0 tx=0x7f629402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.698+0000 7f62abe00640 1 -- 172.21.15.69:0/442923101 shutdown_connections 2024-04-24T08:15:00.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.698+0000 7f62abe00640 1 --2- 172.21.15.69:0/442923101 >> v2:172.21.15.121:3300/0 conn(0x7f62ac06d4f0 0x7f62ac0cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.698+0000 7f62abe00640 1 --2- 172.21.15.69:0/442923101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac162980 0x7f62ac06cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.698+0000 7f62abe00640 1 --2- 172.21.15.69:0/442923101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac15e960 0x7f62ac162440 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.698+0000 7f62abe00640 1 -- 172.21.15.69:0/442923101 >> 172.21.15.69:0/442923101 conn(0x7f62ac09d900 msgr2=0x7f62ac08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:00.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.698+0000 7f62abe00640 1 -- 172.21.15.69:0/442923101 shutdown_connections 2024-04-24T08:15:00.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.698+0000 7f62abe00640 1 -- 172.21.15.69:0/442923101 wait complete. 2024-04-24T08:15:00.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.700+0000 7f62abe00640 1 Processor -- start 2024-04-24T08:15:00.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.700+0000 7f62abe00640 1 -- start start 2024-04-24T08:15:00.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62abe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ac06d4f0 0x7f62ac15c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62abe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac15e960 0x7f62ac0cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62abe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac162980 0x7f62ac0cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62abe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac0b6170 con 0x7f62ac162980 2024-04-24T08:15:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62abe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f62ac0cf5d0 con 0x7f62ac06d4f0 2024-04-24T08:15:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62abe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62ac0b62f0 con 0x7f62ac15e960 2024-04-24T08:15:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62ab400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac162980 0x7f62ac0cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62ab400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac162980 0x7f62ac0cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34516/0 (socket says 172.21.15.69:34516) 2024-04-24T08:15:00.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62ab400640 1 -- 172.21.15.69:0/2468513115 learned_addr learned my addr 172.21.15.69:0/2468513115 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:00.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62aa000640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac15e960 0x7f62ac0cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62aaa00640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ac06d4f0 0x7f62ac15c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62ab400640 1 -- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac15e960 msgr2=0x7f62ac0cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62ab400640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac15e960 0x7f62ac0cd7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.701+0000 7f62ab400640 1 -- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ac06d4f0 msgr2=0x7f62ac15c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.702+0000 7f62ab400640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ac06d4f0 0x7f62ac15c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.702+0000 7f62ab400640 1 -- 172.21.15.69:0/2468513115 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ac15ce80 con 0x7f62ac162980 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.702+0000 7f62ab400640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac162980 0x7f62ac0cdd20 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c820 tx=0x7f62a400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.702+0000 7f62a2a00640 1 -- 172.21.15.69:0/2468513115 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4015070 con 0x7f62ac162980 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.702+0000 7f62a2a00640 1 -- 172.21.15.69:0/2468513115 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a4007830 con 0x7f62ac162980 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.702+0000 7f62abe00640 1 -- 172.21.15.69:0/2468513115 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ac0b7d60 con 0x7f62ac162980 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.702+0000 7f62a2a00640 1 -- 172.21.15.69:0/2468513115 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4007c00 con 0x7f62ac162980 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.702+0000 7f62abe00640 1 -- 172.21.15.69:0/2468513115 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ac0b8320 con 0x7f62ac162980 2024-04-24T08:15:00.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.703+0000 7f62abe00640 1 -- 172.21.15.69:0/2468513115 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ac075d50 con 0x7f62ac162980 2024-04-24T08:15:00.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.704+0000 7f62a2a00640 1 -- 172.21.15.69:0/2468513115 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f62a4018070 con 0x7f62ac162980 2024-04-24T08:15:00.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.705+0000 7f62a2a00640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f628806c8f0 0x7f628806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:00.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.705+0000 7f62a2a00640 1 -- 172.21.15.69:0/2468513115 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f62a408bb60 con 0x7f62ac162980 2024-04-24T08:15:00.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.706+0000 7f62aaa00640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f628806c8f0 0x7f628806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:00.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.707+0000 7f62aaa00640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f628806c8f0 0x7f628806edb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6294034000 tx=0x7f629402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:00.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.710+0000 7f62a2a00640 1 -- 172.21.15.69:0/2468513115 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62a4059e20 con 0x7f62ac162980 2024-04-24T08:15:00.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.953+0000 7f62abe00640 1 -- 172.21.15.69:0/2468513115 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f62ac0d1dc0 con 0x7f62ac162980 2024-04-24T08:15:00.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.954+0000 7f62a2a00640 1 -- 172.21.15.69:0/2468513115 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v52) ==== 53+0+73 (secure 0 0 0) 0x7f62a4059720 con 0x7f62ac162980 2024-04-24T08:15:00.955 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:15:00.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 -- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f628806c8f0 msgr2=0x7f628806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f628806c8f0 0x7f628806edb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f6294034000 tx=0x7f629402e000 comp rx=0 tx=0).stop 2024-04-24T08:15:00.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 -- 172.21.15.69:0/2468513115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac162980 msgr2=0x7f62ac0cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:00.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac162980 0x7f62ac0cdd20 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f62a400c820 tx=0x7f62a400cce0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 -- 172.21.15.69:0/2468513115 shutdown_connections 2024-04-24T08:15:00.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f628806c8f0 0x7f628806edb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62ac162980 0x7f62ac0cdd20 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62ac15e960 0x7f62ac0cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 --2- 172.21.15.69:0/2468513115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62ac06d4f0 0x7f62ac15c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:00.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 -- 172.21.15.69:0/2468513115 >> 172.21.15.69:0/2468513115 conn(0x7f62ac09d900 msgr2=0x7f62ac0b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:00.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 -- 172.21.15.69:0/2468513115 shutdown_connections 2024-04-24T08:15:00.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:00.956+0000 7f62abe00640 1 -- 172.21.15.69:0/2468513115 wait complete. 2024-04-24T08:15:00.977 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:15:00.977 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:15:00.977 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:15:00.977 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:15:00.977 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:00.977 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:15:00.977 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:15:00.977 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:15:00.978 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:15:00.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:00.978 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:15:01.026 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:15:01.027 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42279sec preferred_lft 42279sec 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:15:01.028 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:15:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:01.029 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:15:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:15:01.029 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:01.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:01.029 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:15:01.107 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:15:01.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:01.111 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:15:01.163 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:15:01.164 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:15:01.164 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:15:01.164 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:15:01.164 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:15:01.164 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:15:01.164 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:15:01.164 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:15:01.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:15:01.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:15:01.165 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:15:01.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:15:01.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:15:01.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:01.376 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:15:01.426 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:15:01.426 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:15:01.427 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:15:01.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:01.427 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:15:01.507 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:15:01.508 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:15:01.508 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:01.560 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:15:01.560 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:15:01.561 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:01.612 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:15:01.695 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:15:01.748 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:15:01.748 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:15:01.748 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:15:01.748 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:15:01.748 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:15:01.748 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:15:01.748 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:15:01.749 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:15:01.750 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:15:01.751 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:01.751 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:01.751 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:01.751 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:01.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:01.752 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:15:01.799 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:15:01.800 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:15:01.806 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:15:01.874 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:15:01.916 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:15:01.917 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:15:01.918 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:15:01.918 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:15:01.918 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:15:01.918 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:15:01.918 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:15:01.918 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:15:01.918 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:15:01.918 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:15:01.918 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:15:01.919 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:15:01.919 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:15:01.919 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:15:01.919 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:15:01.919 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:15:01.919 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:01.919 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:01.920 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:01.920 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:01.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:01.922 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:15:01.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:15:01.971+0000 7f4950327480 -1 init, newargv = 0x55ee9abd1290 newargc=15 2024-04-24T08:15:01.973 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[60665]: starting ceph client 2024-04-24T08:15:01.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[60665]: starting fuse 2024-04-24T08:15:02.005 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:15:02.006 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:15:02.006 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:15:02.006 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:15:02.006 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:15:02.006 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:15:02.006 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:15:02.006 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:02.006 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:15:02.006 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:15:02.007 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:15:02.008 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:15:02.008 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:15:02.008 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:02.008 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:15:02.008 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:15:02.194 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.60665.asok 2024-04-24T08:15:02.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:15:02.211 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.60665.asok 2024-04-24T08:15:02.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:02.211 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.60665.asok status 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "60665", 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:15:02.439 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 7380, 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 7380 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2760243889 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:15:02.440 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2760243889 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.7380 192.168.0.1:0/2760243889", 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2760243889", 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 52, 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 71, 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:15:02.441 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:15:02.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:02.454 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:15:02.454 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:02.507 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:15:02.507 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:02.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:02.507 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:02.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:02.593 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-04-24T08:15:02.647 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:15:02.647 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-04-24T08:15:02.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.753+0000 7fd9f15def40 1 Processor -- start 2024-04-24T08:15:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.753+0000 7fd9f15def40 1 -- start start 2024-04-24T08:15:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.754+0000 7fd9f15def40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9d4003ab0 0x556a68e18aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:02.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.754+0000 7fd9f15def40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a68f839f0 0x556a68f83e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.754+0000 7fd9f15def40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a68f84390 0x556a68e659d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.754+0000 7fd9f15def40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556a68dc1cc0 con 0x556a68f839f0 2024-04-24T08:15:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.754+0000 7fd9f15def40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556a68dc5d00 con 0x556a68f84390 2024-04-24T08:15:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.754+0000 7fd9f15def40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556a68ef28a0 con 0x7fd9d4003ab0 2024-04-24T08:15:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.754+0000 7fd9ede00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a68f839f0 0x556a68f83e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.754+0000 7fd9ede00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a68f839f0 0x556a68f83e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34530/0 (socket says 172.21.15.69:34530) 2024-04-24T08:15:02.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.754+0000 7fd9ede00640 1 -- 172.21.15.69:0/2116389473 learned_addr learned my addr 172.21.15.69:0/2116389473 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9ee800640 1 --2- 172.21.15.69:0/2116389473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9d4003ab0 0x556a68e18aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9ef200640 1 --2- 172.21.15.69:0/2116389473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a68f84390 0x556a68e659d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9ede00640 1 -- 172.21.15.69:0/2116389473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9d4003ab0 msgr2=0x556a68e18aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9ede00640 1 --2- 172.21.15.69:0/2116389473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9d4003ab0 0x556a68e18aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9ede00640 1 -- 172.21.15.69:0/2116389473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a68f84390 msgr2=0x556a68e659d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9ede00640 1 --2- 172.21.15.69:0/2116389473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a68f84390 0x556a68e659d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9ede00640 1 -- 172.21.15.69:0/2116389473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a68eef2a0 con 0x556a68f839f0 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9ede00640 1 --2- 172.21.15.69:0/2116389473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a68f839f0 0x556a68f83e50 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c3b0 tx=0x7fd9dc001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995ffa47f2d399ae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9eca00640 1 -- 172.21.15.69:0/2116389473 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc00e540 con 0x556a68f839f0 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9eca00640 1 -- 172.21.15.69:0/2116389473 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9dc00e6e0 con 0x556a68f839f0 2024-04-24T08:15:02.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.755+0000 7fd9eca00640 1 -- 172.21.15.69:0/2116389473 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc00e8d0 con 0x556a68f839f0 2024-04-24T08:15:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.756+0000 7fd9f15def40 1 -- 172.21.15.69:0/2116389473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a68f839f0 msgr2=0x556a68f83e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.756+0000 7fd9f15def40 1 --2- 172.21.15.69:0/2116389473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a68f839f0 0x556a68f83e50 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00c3b0 tx=0x7fd9dc001e70 comp rx=0 tx=0).stop 2024-04-24T08:15:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.756+0000 7fd9f15def40 1 -- 172.21.15.69:0/2116389473 shutdown_connections 2024-04-24T08:15:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.756+0000 7fd9f15def40 1 --2- 172.21.15.69:0/2116389473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556a68f839f0 0x556a68f83e50 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.756+0000 7fd9f15def40 1 --2- 172.21.15.69:0/2116389473 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a68f84390 0x556a68e659d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.756+0000 7fd9f15def40 1 --2- 172.21.15.69:0/2116389473 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9d4003ab0 0x556a68e18aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.756+0000 7fd9f15def40 1 -- 172.21.15.69:0/2116389473 >> 172.21.15.69:0/2116389473 conn(0x556a68e49be0 msgr2=0x556a68e1f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.756+0000 7fd9f15def40 1 -- 172.21.15.69:0/2116389473 shutdown_connections 2024-04-24T08:15:02.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.756+0000 7fd9f15def40 1 -- 172.21.15.69:0/2116389473 wait complete. 2024-04-24T08:15:02.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.757+0000 7fd9f15def40 1 Processor -- start 2024-04-24T08:15:02.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.757+0000 7fd9f15def40 1 -- start start 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.758+0000 7fd9f15def40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a68f839f0 0x556a68edde70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.758+0000 7fd9f15def40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556a68f84390 0x556a68f80d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.758+0000 7fd9f15def40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9d4003ab0 0x556a68f81240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.758+0000 7fd9f15def40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556a68f84810 con 0x7fd9d4003ab0 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.758+0000 7fd9f15def40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556a68eef4b0 con 0x556a68f839f0 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.758+0000 7fd9f15def40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556a68eef000 con 0x556a68f84390 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.758+0000 7fd9ef200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9d4003ab0 0x556a68f81240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.758+0000 7fd9ef200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9d4003ab0 0x556a68f81240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34544/0 (socket says 172.21.15.69:34544) 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.758+0000 7fd9ef200640 1 -- 172.21.15.69:0/697251500 learned_addr learned my addr 172.21.15.69:0/697251500 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.759+0000 7fd9ee800640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a68f839f0 0x556a68edde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.759+0000 7fd9ede00640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556a68f84390 0x556a68f80d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.759+0000 7fd9ef200640 1 -- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556a68f84390 msgr2=0x556a68f80d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.759+0000 7fd9ef200640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556a68f84390 0x556a68f80d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:02.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.759+0000 7fd9ef200640 1 -- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a68f839f0 msgr2=0x556a68edde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:02.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.759+0000 7fd9ef200640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556a68f839f0 0x556a68edde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:02.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.759+0000 7fd9ef200640 1 -- 172.21.15.69:0/697251500 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a68eed100 con 0x7fd9d4003ab0 2024-04-24T08:15:02.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.759+0000 7fd9ede00640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556a68f84390 0x556a68f80d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:02.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.759+0000 7fd9ef200640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9d4003ab0 0x556a68f81240 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8046b40 tx=0x556a68e4c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:02.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.759+0000 7fd9e7400640 1 -- 172.21.15.69:0/697251500 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8061620 con 0x7fd9d4003ab0 2024-04-24T08:15:02.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.760+0000 7fd9e7400640 1 -- 172.21.15.69:0/697251500 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9e8000ed0 con 0x7fd9d4003ab0 2024-04-24T08:15:02.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.760+0000 7fd9e7400640 1 -- 172.21.15.69:0/697251500 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8061a20 con 0x7fd9d4003ab0 2024-04-24T08:15:02.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.760+0000 7fd9f15def40 1 -- 172.21.15.69:0/697251500 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556a68e18d60 con 0x7fd9d4003ab0 2024-04-24T08:15:02.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.760+0000 7fd9f15def40 1 -- 172.21.15.69:0/697251500 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a68ede3b0 con 0x7fd9d4003ab0 2024-04-24T08:15:02.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.762+0000 7fd9e7400640 1 -- 172.21.15.69:0/697251500 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9e805e040 con 0x7fd9d4003ab0 2024-04-24T08:15:02.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.763+0000 7fd9e7400640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d006b2d0 0x7fd9d006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:02.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.763+0000 7fd9e7400640 1 -- 172.21.15.69:0/697251500 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fd9e80cf2a0 con 0x7fd9d4003ab0 2024-04-24T08:15:02.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.763+0000 7fd9f15def40 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x556a68ef4850 0x556a68eedaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:02.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.763+0000 7fd9f15def40 1 -- 172.21.15.69:0/697251500 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:1 10.10 10:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71) -- 0x556a68eee030 con 0x556a68ef4850 2024-04-24T08:15:02.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.763+0000 7fd9ee800640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d006b2d0 0x7fd9d006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:02.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.764+0000 7fd9ede00640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x556a68ef4850 0x556a68eedaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.764+0000 7fd9ee800640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d006b2d0 0x7fd9d006d790 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x556a68e67090 tx=0x7fd9e0007040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:02.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.765+0000 7fd9ede00640 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x556a68ef4850 0x556a68eedaf0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:02.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.769+0000 7fd9ede00640 1 -- 172.21.15.69:0/697251500 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v71'12 uv12 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7fd9dc00e890 con 0x556a68ef4850 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.769+0000 7fd9f15def40 1 -- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x556a68ef4850 msgr2=0x556a68eedaf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.770+0000 7fd9f15def40 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x556a68ef4850 0x556a68eedaf0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.770+0000 7fd9f15def40 1 -- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d006b2d0 msgr2=0x7fd9d006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.770+0000 7fd9f15def40 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9d006b2d0 0x7fd9d006d790 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x556a68e67090 tx=0x7fd9e0007040 comp rx=0 tx=0).stop 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.770+0000 7fd9f15def40 1 -- 172.21.15.69:0/697251500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9d4003ab0 msgr2=0x556a68f81240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.770+0000 7fd9f15def40 1 --2- 172.21.15.69:0/697251500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9d4003ab0 0x556a68f81240 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8046b40 tx=0x556a68e4c950 comp rx=0 tx=0).stop 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.770+0000 7fd9ef200640 1 -- 172.21.15.69:0/697251500 reap_dead start 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.770+0000 7fd9f15def40 1 -- 172.21.15.69:0/697251500 shutdown_connections 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.770+0000 7fd9f15def40 1 -- 172.21.15.69:0/697251500 >> 172.21.15.69:0/697251500 conn(0x556a68e49be0 msgr2=0x556a68e1f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.770+0000 7fd9f15def40 1 -- 172.21.15.69:0/697251500 shutdown_connections 2024-04-24T08:15:02.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:02.770+0000 7fd9f15def40 1 -- 172.21.15.69:0/697251500 wait complete. 2024-04-24T08:15:02.777 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:15:02.777 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:15:02.879 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:15:02.985 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:15:02.985 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:15:02.985 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:15:02.985 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:15:03.043 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:03.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.237+0000 7fa930000640 1 Processor -- start 2024-04-24T08:15:03.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.237+0000 7fa930000640 1 -- start start 2024-04-24T08:15:03.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.238+0000 7fa930000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa9280d0300 0x7fa92806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.238+0000 7fa930000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa928145fe0 0x7fa928160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.238+0000 7fa930000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9280cf3d0 con 0x7fa928145a90 2024-04-24T08:15:03.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.238+0000 7fa930000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa9280cf6b0 con 0x7fa9280d0300 2024-04-24T08:15:03.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.239+0000 7fa930000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa9280cf990 con 0x7fa928145fe0 2024-04-24T08:15:03.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.239+0000 7fa92cc00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa928145a90 0x7fa9280d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53140/0 (socket says 172.21.15.69:53140) 2024-04-24T08:15:03.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.239+0000 7fa92cc00640 1 -- 172.21.15.69:0/1305425374 learned_addr learned my addr 172.21.15.69:0/1305425374 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:03.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.239+0000 7fa92e000640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa928145fe0 0x7fa928160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.239+0000 7fa92d600640 1 --2- 172.21.15.69:0/1305425374 >> v2:172.21.15.121:3300/0 conn(0x7fa9280d0300 0x7fa92806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.239+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2369551579 0 0) 0x7fa9280cf3d0 con 0x7fa928145a90 2024-04-24T08:15:03.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.239+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa910003520 con 0x7fa928145a90 2024-04-24T08:15:03.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1002586972 0 0) 0x7fa910003520 con 0x7fa928145a90 2024-04-24T08:15:03.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa928145fe0 msgr2=0x7fa928160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:03.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa928145fe0 0x7fa928160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 >> v2:172.21.15.121:3300/0 conn(0x7fa9280d0300 msgr2=0x7fa92806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:03.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 --2- 172.21.15.69:0/1305425374 >> v2:172.21.15.121:3300/0 conn(0x7fa9280d0300 0x7fa92806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9280cfc70 con 0x7fa928145a90 2024-04-24T08:15:03.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fa914001670 con 0x7fa928145a90 2024-04-24T08:15:03.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 >> v1:172.21.15.69:6789/0 conn(0x7fa928145a90 legacy=0x7fa9280d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9100039b0 0x7fa910003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa928145fe0 0x7fa910004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa910004d40 0x7fa9100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9280cf6b0 con 0x7fa910004d40 2024-04-24T08:15:03.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa9280cf990 con 0x7fa9100039b0 2024-04-24T08:15:03.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa91000b9c0 con 0x7fa928145fe0 2024-04-24T08:15:03.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.240+0000 7fa92e000640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa910004d40 0x7fa9100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.241+0000 7fa92d600640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9100039b0 0x7fa910003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.241+0000 7fa92e000640 1 -- 172.21.15.69:0/1305425374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa928145fe0 msgr2=0x7fa910004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:03.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.241+0000 7fa92e000640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa928145fe0 0x7fa910004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.241+0000 7fa92e000640 1 -- 172.21.15.69:0/1305425374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9100039b0 msgr2=0x7fa910003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:03.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.241+0000 7fa92e000640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9100039b0 0x7fa910003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.241+0000 7fa92e000640 1 -- 172.21.15.69:0/1305425374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa910003520 con 0x7fa910004d40 2024-04-24T08:15:03.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.242+0000 7fa92e000640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa910004d40 0x7fa9100051c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fa92000e170 tx=0x7fa910005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1152140802260fa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:03.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.242+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa920034020 con 0x7fa910004d40 2024-04-24T08:15:03.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.242+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa920030430 con 0x7fa910004d40 2024-04-24T08:15:03.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.242+0000 7fa927e00640 1 -- 172.21.15.69:0/1305425374 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa920009c10 con 0x7fa910004d40 2024-04-24T08:15:03.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.243+0000 7fa930000640 1 -- 172.21.15.69:0/1305425374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa910004d40 msgr2=0x7fa9100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.243+0000 7fa930000640 1 --2- 172.21.15.69:0/1305425374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa910004d40 0x7fa9100051c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fa92000e170 tx=0x7fa910005830 comp rx=0 tx=0).stop 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.243+0000 7fa92e000640 1 -- 172.21.15.69:0/1305425374 reap_dead start 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.243+0000 7fa930000640 1 -- 172.21.15.69:0/1305425374 shutdown_connections 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.243+0000 7fa930000640 1 -- 172.21.15.69:0/1305425374 >> 172.21.15.69:0/1305425374 conn(0x7fa928067d90 msgr2=0x7fa92806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.243+0000 7fa930000640 1 -- 172.21.15.69:0/1305425374 shutdown_connections 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.243+0000 7fa930000640 1 -- 172.21.15.69:0/1305425374 wait complete. 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.245+0000 7fa930000640 1 Processor -- start 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.245+0000 7fa930000640 1 -- start start 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa930000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa910004d40 0x7fa92815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa930000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa92815b9b0 0x7fa92815e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa930000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa92815e550 0x7fa9281394d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa930000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9281458f0 con 0x7fa92815b9b0 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa930000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa928145770 con 0x7fa92815e550 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa930000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa9280cdc30 con 0x7fa910004d40 2024-04-24T08:15:03.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa92cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa92815b9b0 0x7fa92815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa92cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa92815b9b0 0x7fa92815e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34564/0 (socket says 172.21.15.69:34564) 2024-04-24T08:15:03.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa92cc00640 1 -- 172.21.15.69:0/3794149482 learned_addr learned my addr 172.21.15.69:0/3794149482 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:03.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa92d600640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa910004d40 0x7fa92815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa92cc00640 1 -- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa910004d40 msgr2=0x7fa92815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa92cc00640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa910004d40 0x7fa92815dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa92cc00640 1 -- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa92815e550 msgr2=0x7fa9281394d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa92cc00640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa92815e550 0x7fa9281394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.246+0000 7fa92cc00640 1 -- 172.21.15.69:0/3794149482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92813f020 con 0x7fa92815b9b0 2024-04-24T08:15:03.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.247+0000 7fa92cc00640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa92815b9b0 0x7fa92815e010 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fa91400c7a0 tx=0x7fa91400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:03.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.247+0000 7fa925600640 1 -- 172.21.15.69:0/3794149482 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914018070 con 0x7fa92815b9b0 2024-04-24T08:15:03.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.247+0000 7fa925600640 1 -- 172.21.15.69:0/3794149482 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9140074e0 con 0x7fa92815b9b0 2024-04-24T08:15:03.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.247+0000 7fa925600640 1 -- 172.21.15.69:0/3794149482 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140049e0 con 0x7fa92815b9b0 2024-04-24T08:15:03.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.247+0000 7fa930000640 1 -- 172.21.15.69:0/3794149482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa92813f300 con 0x7fa92815b9b0 2024-04-24T08:15:03.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.247+0000 7fa930000640 1 -- 172.21.15.69:0/3794149482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa92815ecc0 con 0x7fa92815b9b0 2024-04-24T08:15:03.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.248+0000 7fa930000640 1 -- 172.21.15.69:0/3794149482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa928068a80 con 0x7fa92815b9b0 2024-04-24T08:15:03.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.249+0000 7fa925600640 1 -- 172.21.15.69:0/3794149482 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa914004b80 con 0x7fa92815b9b0 2024-04-24T08:15:03.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.250+0000 7fa925600640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa90806c8f0 0x7fa90806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.250+0000 7fa925600640 1 -- 172.21.15.69:0/3794149482 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fa91408b2b0 con 0x7fa92815b9b0 2024-04-24T08:15:03.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.251+0000 7fa92d600640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa90806c8f0 0x7fa90806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.252+0000 7fa92d600640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa90806c8f0 0x7fa90806edb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0099d0 tx=0x7fa91c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:03.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.254+0000 7fa925600640 1 -- 172.21.15.69:0/3794149482 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9140594e0 con 0x7fa92815b9b0 2024-04-24T08:15:03.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.454+0000 7fa930000640 1 -- 172.21.15.69:0/3794149482 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9280af7d0 con 0x7fa92815b9b0 2024-04-24T08:15:03.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.458+0000 7fa925600640 1 -- 172.21.15.69:0/3794149482 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16969 (secure 0 0 0) 0x7fa914058de0 con 0x7fa92815b9b0 2024-04-24T08:15:03.459 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:03.460 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":71,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:14:03.353108+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.17","mappings":[{"from":6,"to":5}]},{"pgid":"11.22","mappings":[{"from":6,"to":0}]},{"pgid":"11.2e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6837/1135108804":"2024-04-25T08:13:37.186005+0000","172.21.15.121:6835/1135108804":"2024-04-25T08:13:37.186005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:03.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.460+0000 7fa930000640 1 -- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa90806c8f0 msgr2=0x7fa90806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.460+0000 7fa930000640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa90806c8f0 0x7fa90806edb0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0099d0 tx=0x7fa91c008040 comp rx=0 tx=0).stop 2024-04-24T08:15:03.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.460+0000 7fa930000640 1 -- 172.21.15.69:0/3794149482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa92815b9b0 msgr2=0x7fa92815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.460+0000 7fa930000640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa92815b9b0 0x7fa92815e010 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fa91400c7a0 tx=0x7fa91400cc60 comp rx=0 tx=0).stop 2024-04-24T08:15:03.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.460+0000 7fa930000640 1 -- 172.21.15.69:0/3794149482 shutdown_connections 2024-04-24T08:15:03.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.460+0000 7fa930000640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa90806c8f0 0x7fa90806edb0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.460+0000 7fa930000640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa92815b9b0 0x7fa92815e010 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.460+0000 7fa930000640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa92815e550 0x7fa9281394d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.460+0000 7fa930000640 1 --2- 172.21.15.69:0/3794149482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa910004d40 0x7fa92815dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.460+0000 7fa930000640 1 -- 172.21.15.69:0/3794149482 >> 172.21.15.69:0/3794149482 conn(0x7fa928067d90 msgr2=0x7fa9280b4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:03.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.461+0000 7fa930000640 1 -- 172.21.15.69:0/3794149482 shutdown_connections 2024-04-24T08:15:03.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.461+0000 7fa930000640 1 -- 172.21.15.69:0/3794149482 wait complete. 2024-04-24T08:15:03.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:13:13.250209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:13:14.468238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:15:03.481 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:15:03.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.679+0000 7f7e58a00640 1 Processor -- start 2024-04-24T08:15:03.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.679+0000 7f7e58a00640 1 -- start start 2024-04-24T08:15:03.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.681+0000 7f7e58a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5415c820 0x7f7e540b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.681+0000 7f7e58a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e540b46a0 0x7f7e5406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.681+0000 7f7e58a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e5415cc00 con 0x7f7e5406fd10 2024-04-24T08:15:03.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.681+0000 7f7e58a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7e5406ed30 con 0x7f7e540b46a0 2024-04-24T08:15:03.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.681+0000 7f7e58a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7e540b4e10 con 0x7f7e5415c820 2024-04-24T08:15:03.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.681+0000 7f7e49400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e540b46a0 0x7f7e5406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.681+0000 7f7e49400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e540b46a0 0x7f7e5406f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34574/0 (socket says 172.21.15.69:34574) 2024-04-24T08:15:03.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.681+0000 7f7e52800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7e5406fd10 0x7f7e540ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53144/0 (socket says 172.21.15.69:53144) 2024-04-24T08:15:03.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.681+0000 7f7e49400640 1 -- 172.21.15.69:0/2150060489 learned_addr learned my addr 172.21.15.69:0/2150060489 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:03.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.682+0000 7f7e51e00640 1 --2- 172.21.15.69:0/2150060489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5415c820 0x7f7e540b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.682+0000 7f7e49400640 1 -- 172.21.15.69:0/2150060489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5415c820 msgr2=0x7f7e540b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:03.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.682+0000 7f7e49400640 1 --2- 172.21.15.69:0/2150060489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5415c820 0x7f7e540b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.682+0000 7f7e49400640 1 -- 172.21.15.69:0/2150060489 >> v1:172.21.15.69:6789/0 conn(0x7f7e5406fd10 legacy=0x7f7e540ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.682+0000 7f7e49400640 1 -- 172.21.15.69:0/2150060489 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e540b50f0 con 0x7f7e540b46a0 2024-04-24T08:15:03.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.682+0000 7f7e51400640 1 -- 172.21.15.69:0/2150060489 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3228706501 0 0) 0x7f7e5415cc00 con 0x7f7e5406fd10 2024-04-24T08:15:03.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.683+0000 7f7e49400640 1 --2- 172.21.15.69:0/2150060489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e540b46a0 0x7f7e5406f7d0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009870 tx=0x7f7e3c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac5f04c572eb2f97 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:03.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.683+0000 7f7e51400640 1 -- 172.21.15.69:0/2150060489 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c037070 con 0x7f7e540b46a0 2024-04-24T08:15:03.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.683+0000 7f7e51400640 1 -- 172.21.15.69:0/2150060489 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e3c032440 con 0x7f7e540b46a0 2024-04-24T08:15:03.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.683+0000 7f7e51400640 1 -- 172.21.15.69:0/2150060489 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c032810 con 0x7f7e540b46a0 2024-04-24T08:15:03.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.684+0000 7f7e58a00640 1 -- 172.21.15.69:0/2150060489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e540b46a0 msgr2=0x7f7e5406f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.684+0000 7f7e58a00640 1 --2- 172.21.15.69:0/2150060489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e540b46a0 0x7f7e5406f7d0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009870 tx=0x7f7e3c02a270 comp rx=0 tx=0).stop 2024-04-24T08:15:03.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.684+0000 7f7e58a00640 1 -- 172.21.15.69:0/2150060489 shutdown_connections 2024-04-24T08:15:03.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.684+0000 7f7e58a00640 1 --2- 172.21.15.69:0/2150060489 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e540b46a0 0x7f7e5406f7d0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.684+0000 7f7e58a00640 1 --2- 172.21.15.69:0/2150060489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5415c820 0x7f7e540b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.684+0000 7f7e58a00640 1 -- 172.21.15.69:0/2150060489 >> 172.21.15.69:0/2150060489 conn(0x7f7e540d0350 msgr2=0x7f7e540d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:03.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.684+0000 7f7e58a00640 1 -- 172.21.15.69:0/2150060489 shutdown_connections 2024-04-24T08:15:03.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.684+0000 7f7e58a00640 1 -- 172.21.15.69:0/2150060489 wait complete. 2024-04-24T08:15:03.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.686+0000 7f7e58a00640 1 Processor -- start 2024-04-24T08:15:03.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.686+0000 7f7e58a00640 1 -- start start 2024-04-24T08:15:03.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e58a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5406fd10 0x7f7e540b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e58a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e540b46a0 0x7f7e54157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e58a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415c820 0x7f7e540ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e58a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e540b4b00 con 0x7f7e5415c820 2024-04-24T08:15:03.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e58a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5406f2b0 con 0x7f7e540b46a0 2024-04-24T08:15:03.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e58a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7e540ce960 con 0x7f7e5406fd10 2024-04-24T08:15:03.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e52800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415c820 0x7f7e540ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e52800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415c820 0x7f7e540ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34578/0 (socket says 172.21.15.69:34578) 2024-04-24T08:15:03.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e52800640 1 -- 172.21.15.69:0/2187094092 learned_addr learned my addr 172.21.15.69:0/2187094092 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:03.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e49400640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e540b46a0 0x7f7e54157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.687+0000 7f7e51e00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5406fd10 0x7f7e540b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e52800640 1 -- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5406fd10 msgr2=0x7f7e540b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e52800640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5406fd10 0x7f7e540b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e52800640 1 -- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e540b46a0 msgr2=0x7f7e54157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e52800640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e540b46a0 0x7f7e54157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e52800640 1 -- 172.21.15.69:0/2187094092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e541f8120 con 0x7f7e5415c820 2024-04-24T08:15:03.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e51e00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5406fd10 0x7f7e540b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:03.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e52800640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415c820 0x7f7e540ca180 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00c410 tx=0x7f7e4c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:03.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e4aa00640 1 -- 172.21.15.69:0/2187094092 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c0074e0 con 0x7f7e5415c820 2024-04-24T08:15:03.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e4aa00640 1 -- 172.21.15.69:0/2187094092 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e4c0047e0 con 0x7f7e5415c820 2024-04-24T08:15:03.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e4aa00640 1 -- 172.21.15.69:0/2187094092 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c004bd0 con 0x7f7e5415c820 2024-04-24T08:15:03.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e58a00640 1 -- 172.21.15.69:0/2187094092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e541f8400 con 0x7f7e5415c820 2024-04-24T08:15:03.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.688+0000 7f7e58a00640 1 -- 172.21.15.69:0/2187094092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e541f89c0 con 0x7f7e5415c820 2024-04-24T08:15:03.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.689+0000 7f7e58a00640 1 -- 172.21.15.69:0/2187094092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e54068a80 con 0x7f7e5415c820 2024-04-24T08:15:03.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.690+0000 7f7e4aa00640 1 -- 172.21.15.69:0/2187094092 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e4c011070 con 0x7f7e5415c820 2024-04-24T08:15:03.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.691+0000 7f7e4aa00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3006c8f0 0x7f7e3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:03.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.692+0000 7f7e4aa00640 1 -- 172.21.15.69:0/2187094092 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f7e4c08b810 con 0x7f7e5415c820 2024-04-24T08:15:03.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.692+0000 7f7e51e00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3006c8f0 0x7f7e3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:03.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.693+0000 7f7e51e00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3006c8f0 0x7f7e3006edb0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7e44007330 tx=0x7f7e4400b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:03.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.696+0000 7f7e4aa00640 1 -- 172.21.15.69:0/2187094092 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7e4c01a2b0 con 0x7f7e5415c820 2024-04-24T08:15:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.900+0000 7f7e58a00640 1 -- 172.21.15.69:0/2187094092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7e541581c0 con 0x7f7e5415c820 2024-04-24T08:15:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.901+0000 7f7e4aa00640 1 -- 172.21.15.69:0/2187094092 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) ==== 93+0+10 (secure 0 0 0) 0x7f7e4c059b60 con 0x7f7e5415c820 2024-04-24T08:15:03.902 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:15:03.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.903+0000 7f7e58a00640 1 -- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3006c8f0 msgr2=0x7f7e3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.903+0000 7f7e58a00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3006c8f0 0x7f7e3006edb0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7e44007330 tx=0x7f7e4400b040 comp rx=0 tx=0).stop 2024-04-24T08:15:03.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.903+0000 7f7e58a00640 1 -- 172.21.15.69:0/2187094092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415c820 msgr2=0x7f7e540ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:03.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.903+0000 7f7e58a00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415c820 0x7f7e540ca180 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c00c410 tx=0x7f7e4c00c8d0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.903+0000 7f7e58a00640 1 -- 172.21.15.69:0/2187094092 shutdown_connections 2024-04-24T08:15:03.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.903+0000 7f7e58a00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3006c8f0 0x7f7e3006edb0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.903+0000 7f7e58a00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e5415c820 0x7f7e540ca180 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.903+0000 7f7e58a00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e540b46a0 0x7f7e54157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.903+0000 7f7e58a00640 1 --2- 172.21.15.69:0/2187094092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e5406fd10 0x7f7e540b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:03.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.903+0000 7f7e58a00640 1 -- 172.21.15.69:0/2187094092 >> 172.21.15.69:0/2187094092 conn(0x7f7e540d0350 msgr2=0x7f7e5415bae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:03.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.904+0000 7f7e58a00640 1 -- 172.21.15.69:0/2187094092 shutdown_connections 2024-04-24T08:15:03.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:03.904+0000 7f7e58a00640 1 -- 172.21.15.69:0/2187094092 wait complete. 2024-04-24T08:15:03.925 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:15:04.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.128+0000 7f58b8000640 1 Processor -- start 2024-04-24T08:15:04.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.128+0000 7f58b8000640 1 -- start start 2024-04-24T08:15:04.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.129+0000 7f58b8000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58b00b3e90 0x7f58b00b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.129+0000 7f58b8000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f58b00ce250 0x7f58b00ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.129+0000 7f58b8000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58b015cc00 con 0x7f58b015c820 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.129+0000 7f58b8000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f58b00b4be0 con 0x7f58b00b3e90 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.129+0000 7f58b8000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f58b006ed30 con 0x7f58b00ce250 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.130+0000 7f58b4c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58b00b3e90 0x7f58b00b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.130+0000 7f58b5600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f58b015c820 0x7f58b006f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53148/0 (socket says 172.21.15.69:53148) 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.130+0000 7f58b4c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f58b00b3e90 0x7f58b00b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34590/0 (socket says 172.21.15.69:34590) 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.130+0000 7f58b5600640 1 -- 172.21.15.69:0/2111506229 learned_addr learned my addr 172.21.15.69:0/2111506229 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.130+0000 7f58b4c00640 1 -- 172.21.15.69:0/2111506229 >> v2:172.21.15.121:3300/0 conn(0x7f58b00ce250 msgr2=0x7f58b00ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.130+0000 7f58b4c00640 1 --2- 172.21.15.69:0/2111506229 >> v2:172.21.15.121:3300/0 conn(0x7f58b00ce250 0x7f58b00ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.130+0000 7f58b4c00640 1 -- 172.21.15.69:0/2111506229 >> v1:172.21.15.69:6789/0 conn(0x7f58b015c820 legacy=0x7f58b006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.130+0000 7f58b4c00640 1 -- 172.21.15.69:0/2111506229 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b00b4e10 con 0x7f58b00b3e90 2024-04-24T08:15:04.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.131+0000 7f58b4c00640 1 --2- 172.21.15.69:0/2111506229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00b3e90 0x7f58b00b46a0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f58b0161420 tx=0x7f589c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a70f53c5225ea4c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:04.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.131+0000 7f58afe00640 1 -- 172.21.15.69:0/2111506229 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c037070 con 0x7f58b00b3e90 2024-04-24T08:15:04.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.132+0000 7f58afe00640 1 -- 172.21.15.69:0/2111506229 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589c032440 con 0x7f58b00b3e90 2024-04-24T08:15:04.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.132+0000 7f58afe00640 1 -- 172.21.15.69:0/2111506229 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c032810 con 0x7f58b00b3e90 2024-04-24T08:15:04.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.132+0000 7f58b8000640 1 -- 172.21.15.69:0/2111506229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00b3e90 msgr2=0x7f58b00b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.132+0000 7f58b8000640 1 --2- 172.21.15.69:0/2111506229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00b3e90 0x7f58b00b46a0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f58b0161420 tx=0x7f589c02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.132+0000 7f58b8000640 1 -- 172.21.15.69:0/2111506229 shutdown_connections 2024-04-24T08:15:04.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.132+0000 7f58b8000640 1 --2- 172.21.15.69:0/2111506229 >> v2:172.21.15.121:3300/0 conn(0x7f58b00ce250 0x7f58b00ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.132+0000 7f58b8000640 1 --2- 172.21.15.69:0/2111506229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00b3e90 0x7f58b00b46a0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.132+0000 7f58b8000640 1 -- 172.21.15.69:0/2111506229 >> 172.21.15.69:0/2111506229 conn(0x7f58b00d0350 msgr2=0x7f58b00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:04.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.132+0000 7f58b8000640 1 -- 172.21.15.69:0/2111506229 shutdown_connections 2024-04-24T08:15:04.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.132+0000 7f58b8000640 1 -- 172.21.15.69:0/2111506229 wait complete. 2024-04-24T08:15:04.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.134+0000 7f58b8000640 1 Processor -- start 2024-04-24T08:15:04.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.134+0000 7f58b8000640 1 -- start start 2024-04-24T08:15:04.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.135+0000 7f58b8000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00b3e90 0x7f58b00aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.135+0000 7f58b8000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00ce250 0x7f58b00c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.135+0000 7f58b8000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b015c820 0x7f58b0146620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.135+0000 7f58b8000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58b006f2b0 con 0x7f58b00ce250 2024-04-24T08:15:04.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.135+0000 7f58b8000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f58b006efd0 con 0x7f58b015c820 2024-04-24T08:15:04.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.135+0000 7f58b8000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f58b006fee0 con 0x7f58b00b3e90 2024-04-24T08:15:04.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.135+0000 7f58b4c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00ce250 0x7f58b00c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.135+0000 7f58b4c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00ce250 0x7f58b00c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34602/0 (socket says 172.21.15.69:34602) 2024-04-24T08:15:04.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.135+0000 7f58b4c00640 1 -- 172.21.15.69:0/2476620889 learned_addr learned my addr 172.21.15.69:0/2476620889 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:04.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.136+0000 7f58b6000640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b015c820 0x7f58b0146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.136+0000 7f58b5600640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00b3e90 0x7f58b00aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.136+0000 7f58b4c00640 1 -- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00b3e90 msgr2=0x7f58b00aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.136+0000 7f58b4c00640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00b3e90 0x7f58b00aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.136+0000 7f58b4c00640 1 -- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b015c820 msgr2=0x7f58b0146620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.136+0000 7f58b4c00640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b015c820 0x7f58b0146620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.136+0000 7f58b4c00640 1 -- 172.21.15.69:0/2476620889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58b00caf40 con 0x7f58b00ce250 2024-04-24T08:15:04.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.136+0000 7f58b5600640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00b3e90 0x7f58b00aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:04.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.136+0000 7f58b4c00640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00ce250 0x7f58b00c95e0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f589c000c00 tx=0x7f589c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:04.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.136+0000 7f58ad600640 1 -- 172.21.15.69:0/2476620889 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c037040 con 0x7f58b00ce250 2024-04-24T08:15:04.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.137+0000 7f58ad600640 1 -- 172.21.15.69:0/2476620889 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589c042070 con 0x7f58b00ce250 2024-04-24T08:15:04.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.137+0000 7f58b8000640 1 -- 172.21.15.69:0/2476620889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58b00cb1c0 con 0x7f58b00ce250 2024-04-24T08:15:04.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.137+0000 7f58ad600640 1 -- 172.21.15.69:0/2476620889 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c030bd0 con 0x7f58b00ce250 2024-04-24T08:15:04.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.137+0000 7f58b8000640 1 -- 172.21.15.69:0/2476620889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58b00cb780 con 0x7f58b00ce250 2024-04-24T08:15:04.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.138+0000 7f58b8000640 1 -- 172.21.15.69:0/2476620889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58b0068a80 con 0x7f58b00ce250 2024-04-24T08:15:04.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.139+0000 7f58ad600640 1 -- 172.21.15.69:0/2476620889 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f589c047460 con 0x7f58b00ce250 2024-04-24T08:15:04.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.140+0000 7f58ad600640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 0x7f589006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.140+0000 7f58ad600640 1 -- 172.21.15.69:0/2476620889 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f589c0ac9d0 con 0x7f58b00ce250 2024-04-24T08:15:04.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.140+0000 7f58b5600640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 0x7f589006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.142+0000 7f58b5600640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 0x7f589006edb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f58a4000f60 tx=0x7f58a4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:04.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.144+0000 7f58ad600640 1 -- 172.21.15.69:0/2476620889 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58b00caf40 con 0x7f58b00ce250 2024-04-24T08:15:04.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.343+0000 7f58b8000640 1 -- 172.21.15.69:0/2476620889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f58b00cddc0 con 0x7f58b00ce250 2024-04-24T08:15:04.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.344+0000 7f58ad600640 1 -- 172.21.15.69:0/2476620889 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) ==== 104+0+11 (secure 0 0 0) 0x7f589c0ac020 con 0x7f58b00ce250 2024-04-24T08:15:04.345 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:15:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.346+0000 7f58b8000640 1 -- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 msgr2=0x7f589006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.346+0000 7f58b8000640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 0x7f589006edb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f58a4000f60 tx=0x7f58a4009040 comp rx=0 tx=0).stop 2024-04-24T08:15:04.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.346+0000 7f58b8000640 1 -- 172.21.15.69:0/2476620889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00ce250 msgr2=0x7f58b00c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.346+0000 7f58b8000640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00ce250 0x7f58b00c95e0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f589c000c00 tx=0x7f589c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.347+0000 7f58b8000640 1 -- 172.21.15.69:0/2476620889 shutdown_connections 2024-04-24T08:15:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.347+0000 7f58b8000640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f589006c8f0 0x7f589006edb0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.347+0000 7f58b8000640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58b00ce250 0x7f58b00c95e0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.347+0000 7f58b8000640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58b015c820 0x7f58b0146620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.347+0000 7f58b8000640 1 --2- 172.21.15.69:0/2476620889 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f58b00b3e90 0x7f58b00aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.347+0000 7f58b8000640 1 -- 172.21.15.69:0/2476620889 >> 172.21.15.69:0/2476620889 conn(0x7f58b00d0350 msgr2=0x7f58b0160c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.347+0000 7f58b8000640 1 -- 172.21.15.69:0/2476620889 shutdown_connections 2024-04-24T08:15:04.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.347+0000 7f58b8000640 1 -- 172.21.15.69:0/2476620889 wait complete. 2024-04-24T08:15:04.368 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:15:04.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.563+0000 7f8d56600640 1 Processor -- start 2024-04-24T08:15:04.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.563+0000 7f8d56600640 1 -- start start 2024-04-24T08:15:04.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.565+0000 7f8d56600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8d50071d70 0x7f8d50072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.565+0000 7f8d56600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 0x7f8d5015f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.565+0000 7f8d56600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d50072690 con 0x7f8d50162270 2024-04-24T08:15:04.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.565+0000 7f8d56600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8d500d0a00 con 0x7f8d50071d70 2024-04-24T08:15:04.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.565+0000 7f8d56600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8d5006f010 con 0x7f8d5015edf0 2024-04-24T08:15:04.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.566+0000 7f8d47e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 0x7f8d5015f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.566+0000 7f8d55200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8d50071d70 0x7f8d50072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.566+0000 7f8d55200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8d50071d70 0x7f8d50072150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:60976/0 (socket says 172.21.15.69:60976) 2024-04-24T08:15:04.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.566+0000 7f8d47e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 0x7f8d5015f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:53914/0 (socket says 172.21.15.69:53914) 2024-04-24T08:15:04.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.566+0000 7f8d55200640 1 -- 172.21.15.69:0/2500121691 learned_addr learned my addr 172.21.15.69:0/2500121691 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:04.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.567+0000 7f8d55200640 1 -- 172.21.15.69:0/2500121691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 msgr2=0x7f8d5015f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:04.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.567+0000 7f8d55200640 1 --2- 172.21.15.69:0/2500121691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 0x7f8d5015f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.567+0000 7f8d55200640 1 -- 172.21.15.69:0/2500121691 >> v1:172.21.15.121:6789/0 conn(0x7f8d50162270 legacy=0x7f8d50162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:04.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.567+0000 7f8d55200640 1 -- 172.21.15.69:0/2500121691 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d5015fbb0 con 0x7f8d50071d70 2024-04-24T08:15:04.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.568+0000 7f8d55200640 1 --2- 172.21.15.69:0/2500121691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50071d70 0x7f8d50072150 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009870 tx=0x7f8d4002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6eba1f7d3febcaf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:04.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.568+0000 7f8d4fe00640 1 -- 172.21.15.69:0/2500121691 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40037070 con 0x7f8d50071d70 2024-04-24T08:15:04.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.569+0000 7f8d4fe00640 1 -- 172.21.15.69:0/2500121691 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d40032440 con 0x7f8d50071d70 2024-04-24T08:15:04.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.569+0000 7f8d4fe00640 1 -- 172.21.15.69:0/2500121691 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40032810 con 0x7f8d50071d70 2024-04-24T08:15:04.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.569+0000 7f8d56600640 1 -- 172.21.15.69:0/2500121691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50071d70 msgr2=0x7f8d50072150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.569+0000 7f8d56600640 1 --2- 172.21.15.69:0/2500121691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50071d70 0x7f8d50072150 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8d40009870 tx=0x7f8d4002a250 comp rx=0 tx=0).stop 2024-04-24T08:15:04.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.569+0000 7f8d56600640 1 -- 172.21.15.69:0/2500121691 shutdown_connections 2024-04-24T08:15:04.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.569+0000 7f8d56600640 1 --2- 172.21.15.69:0/2500121691 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 0x7f8d5015f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.569+0000 7f8d56600640 1 --2- 172.21.15.69:0/2500121691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50071d70 0x7f8d50072150 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.569+0000 7f8d56600640 1 -- 172.21.15.69:0/2500121691 >> 172.21.15.69:0/2500121691 conn(0x7f8d50067b80 msgr2=0x7f8d5006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:04.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.569+0000 7f8d56600640 1 -- 172.21.15.69:0/2500121691 shutdown_connections 2024-04-24T08:15:04.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.569+0000 7f8d56600640 1 -- 172.21.15.69:0/2500121691 wait complete. 2024-04-24T08:15:04.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.571+0000 7f8d56600640 1 Processor -- start 2024-04-24T08:15:04.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.571+0000 7f8d56600640 1 -- start start 2024-04-24T08:15:04.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.572+0000 7f8d56600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d50071d70 0x7f8d50142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.572+0000 7f8d56600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 0x7f8d5014a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.572+0000 7f8d56600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50162270 0x7f8d5014ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.572+0000 7f8d56600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d500d0690 con 0x7f8d50071d70 2024-04-24T08:15:04.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.572+0000 7f8d56600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8d500d03b0 con 0x7f8d50162270 2024-04-24T08:15:04.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.572+0000 7f8d56600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8d500d0e90 con 0x7f8d5015edf0 2024-04-24T08:15:04.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d55200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d50071d70 0x7f8d50142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d55200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d50071d70 0x7f8d50142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34612/0 (socket says 172.21.15.69:34612) 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d55200640 1 -- 172.21.15.69:0/950855297 learned_addr learned my addr 172.21.15.69:0/950855297 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d55c00640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50162270 0x7f8d5014ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d47e00640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 0x7f8d5014a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d55200640 1 -- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 msgr2=0x7f8d5014a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d55200640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 0x7f8d5014a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d55200640 1 -- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50162270 msgr2=0x7f8d5014ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d55200640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50162270 0x7f8d5014ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d55200640 1 -- 172.21.15.69:0/950855297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d5014b3d0 con 0x7f8d50071d70 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d47e00640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 0x7f8d5014a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.573+0000 7f8d55200640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d50071d70 0x7f8d50142fd0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f8d40000c00 tx=0x7f8d400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.574+0000 7f8d4d600640 1 -- 172.21.15.69:0/950855297 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40041020 con 0x7f8d50071d70 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.574+0000 7f8d4d600640 1 -- 172.21.15.69:0/950855297 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d4003d620 con 0x7f8d50071d70 2024-04-24T08:15:04.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.574+0000 7f8d4d600640 1 -- 172.21.15.69:0/950855297 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40037040 con 0x7f8d50071d70 2024-04-24T08:15:04.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.574+0000 7f8d56600640 1 -- 172.21.15.69:0/950855297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d5014b6b0 con 0x7f8d50071d70 2024-04-24T08:15:04.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.574+0000 7f8d56600640 1 -- 172.21.15.69:0/950855297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d5014bc70 con 0x7f8d50071d70 2024-04-24T08:15:04.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.575+0000 7f8d56600640 1 -- 172.21.15.69:0/950855297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d50075d50 con 0x7f8d50071d70 2024-04-24T08:15:04.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.576+0000 7f8d4d600640 1 -- 172.21.15.69:0/950855297 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8d40044020 con 0x7f8d50071d70 2024-04-24T08:15:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.577+0000 7f8d4d600640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2c06c8f0 0x7f8d2c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:04.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.577+0000 7f8d4d600640 1 -- 172.21.15.69:0/950855297 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f8d400ac420 con 0x7f8d50071d70 2024-04-24T08:15:04.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.578+0000 7f8d47e00640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2c06c8f0 0x7f8d2c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:04.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.579+0000 7f8d47e00640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2c06c8f0 0x7f8d2c06edb0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8d5006bf80 tx=0x7f8d38008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:04.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.581+0000 7f8d4d600640 1 -- 172.21.15.69:0/950855297 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d4004b030 con 0x7f8d50071d70 2024-04-24T08:15:04.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.784+0000 7f8d56600640 1 -- 172.21.15.69:0/950855297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8d5015e6c0 con 0x7f8d50071d70 2024-04-24T08:15:04.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.785+0000 7f8d4d600640 1 -- 172.21.15.69:0/950855297 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) ==== 100+0+11 (secure 0 0 0) 0x7f8d4007a6f0 con 0x7f8d50071d70 2024-04-24T08:15:04.786 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:15:04.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.787+0000 7f8d56600640 1 -- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2c06c8f0 msgr2=0x7f8d2c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.787+0000 7f8d56600640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2c06c8f0 0x7f8d2c06edb0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8d5006bf80 tx=0x7f8d38008040 comp rx=0 tx=0).stop 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.787+0000 7f8d56600640 1 -- 172.21.15.69:0/950855297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d50071d70 msgr2=0x7f8d50142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.787+0000 7f8d56600640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d50071d70 0x7f8d50142fd0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f8d40000c00 tx=0x7f8d400046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.787+0000 7f8d56600640 1 -- 172.21.15.69:0/950855297 shutdown_connections 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.787+0000 7f8d56600640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2c06c8f0 0x7f8d2c06edb0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.787+0000 7f8d56600640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d50162270 0x7f8d5014ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.787+0000 7f8d56600640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5015edf0 0x7f8d5014a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.787+0000 7f8d56600640 1 --2- 172.21.15.69:0/950855297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d50071d70 0x7f8d50142fd0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.787+0000 7f8d56600640 1 -- 172.21.15.69:0/950855297 >> 172.21.15.69:0/950855297 conn(0x7f8d50067b80 msgr2=0x7f8d50146420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.788+0000 7f8d56600640 1 -- 172.21.15.69:0/950855297 shutdown_connections 2024-04-24T08:15:04.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:04.788+0000 7f8d56600640 1 -- 172.21.15.69:0/950855297 wait complete. 2024-04-24T08:15:04.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:15:05.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.006+0000 7f045b000640 1 Processor -- start 2024-04-24T08:15:05.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.007+0000 7f045b000640 1 -- start start 2024-04-24T08:15:05.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.008+0000 7f045b000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454156c60 0x7f045415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:05.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.008+0000 7f045b000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04540ad7e0 0x7f045415d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:05.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.008+0000 7f045b000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045415d660 con 0x7f04540b30f0 2024-04-24T08:15:05.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.008+0000 7f045b000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f045406dc30 con 0x7f04540ad7e0 2024-04-24T08:15:05.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.008+0000 7f045b000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f045406df10 con 0x7f0454156c60 2024-04-24T08:15:05.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.008+0000 7f045a600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04540ad7e0 0x7f045415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:05.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.008+0000 7f045a600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04540ad7e0 0x7f045415d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34622/0 (socket says 172.21.15.69:34622) 2024-04-24T08:15:05.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.008+0000 7f045a600640 1 -- 172.21.15.69:0/3078210842 learned_addr learned my addr 172.21.15.69:0/3078210842 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:05.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.009+0000 7f0459c00640 1 --2- 172.21.15.69:0/3078210842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454156c60 0x7f045415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:05.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.009+0000 7f045a600640 1 -- 172.21.15.69:0/3078210842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454156c60 msgr2=0x7f045415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:05.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.009+0000 7f045a600640 1 --2- 172.21.15.69:0/3078210842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454156c60 0x7f045415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:05.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.009+0000 7f045a600640 1 -- 172.21.15.69:0/3078210842 >> v1:172.21.15.69:6789/0 conn(0x7f04540b30f0 legacy=0x7f04540ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:05.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.009+0000 7f045a600640 1 -- 172.21.15.69:0/3078210842 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045406e1f0 con 0x7f04540ad7e0 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.010+0000 7f045a600640 1 --2- 172.21.15.69:0/3078210842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 0x7f045415d000 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f044c009e50 tx=0x7f044c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c501a4a1c9bb95c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.010+0000 7f0453e00640 1 -- 172.21.15.69:0/3078210842 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c036070 con 0x7f04540ad7e0 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.010+0000 7f0453e00640 1 -- 172.21.15.69:0/3078210842 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f044c031600 con 0x7f04540ad7e0 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.010+0000 7f0453e00640 1 -- 172.21.15.69:0/3078210842 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c0319d0 con 0x7f04540ad7e0 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.011+0000 7f045b000640 1 -- 172.21.15.69:0/3078210842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 msgr2=0x7f045415d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.011+0000 7f045b000640 1 --2- 172.21.15.69:0/3078210842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 0x7f045415d000 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f044c009e50 tx=0x7f044c02a710 comp rx=0 tx=0).stop 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.011+0000 7f045b000640 1 -- 172.21.15.69:0/3078210842 shutdown_connections 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.011+0000 7f045b000640 1 --2- 172.21.15.69:0/3078210842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 0x7f045415d000 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.011+0000 7f045b000640 1 --2- 172.21.15.69:0/3078210842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0454156c60 0x7f045415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.011+0000 7f045b000640 1 -- 172.21.15.69:0/3078210842 >> 172.21.15.69:0/3078210842 conn(0x7f04540d0160 msgr2=0x7f04540d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.011+0000 7f045b000640 1 -- 172.21.15.69:0/3078210842 shutdown_connections 2024-04-24T08:15:05.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.011+0000 7f045b000640 1 -- 172.21.15.69:0/3078210842 wait complete. 2024-04-24T08:15:05.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.013+0000 7f045b000640 1 Processor -- start 2024-04-24T08:15:05.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.013+0000 7f045b000640 1 -- start start 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.014+0000 7f045b000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 0x7f0454155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.014+0000 7f045b000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04540b30f0 0x7f0454157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.014+0000 7f045b000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0454156c60 0x7f0454157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.014+0000 7f045b000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f045406e710 con 0x7f04540ad7e0 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.014+0000 7f045b000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f045415d940 con 0x7f0454156c60 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.014+0000 7f045b000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f045406e890 con 0x7f04540b30f0 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.014+0000 7f0459c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 0x7f0454155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.014+0000 7f0459c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 0x7f0454155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34636/0 (socket says 172.21.15.69:34636) 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.014+0000 7f0459c00640 1 -- 172.21.15.69:0/2214556507 learned_addr learned my addr 172.21.15.69:0/2214556507 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0459200640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04540b30f0 0x7f0454157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0459c00640 1 -- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04540b30f0 msgr2=0x7f0454157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:05.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f045a600640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0454156c60 0x7f0454157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0459c00640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04540b30f0 0x7f0454157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0459c00640 1 -- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0454156c60 msgr2=0x7f0454157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0459c00640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0454156c60 0x7f0454157f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0459c00640 1 -- 172.21.15.69:0/2214556507 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04541432c0 con 0x7f04540ad7e0 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0459200640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04540b30f0 0x7f0454157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0459c00640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 0x7f0454155bd0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f044800d5f0 tx=0x7f044800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0451600640 1 -- 172.21.15.69:0/2214556507 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04480125f0 con 0x7f04540ad7e0 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0451600640 1 -- 172.21.15.69:0/2214556507 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0448019070 con 0x7f04540ad7e0 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f045b000640 1 -- 172.21.15.69:0/2214556507 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0454159120 con 0x7f04540ad7e0 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.015+0000 7f0451600640 1 -- 172.21.15.69:0/2214556507 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04480129d0 con 0x7f04540ad7e0 2024-04-24T08:15:05.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.016+0000 7f045b000640 1 -- 172.21.15.69:0/2214556507 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0454158880 con 0x7f04540ad7e0 2024-04-24T08:15:05.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.017+0000 7f045b000640 1 -- 172.21.15.69:0/2214556507 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0454075d50 con 0x7f04540ad7e0 2024-04-24T08:15:05.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.018+0000 7f0451600640 1 -- 172.21.15.69:0/2214556507 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0448012b70 con 0x7f04540ad7e0 2024-04-24T08:15:05.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.019+0000 7f0451600640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f043406c920 0x7f043406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:05.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.019+0000 7f0451600640 1 -- 172.21.15.69:0/2214556507 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f044808c080 con 0x7f04540ad7e0 2024-04-24T08:15:05.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.019+0000 7f0459200640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f043406c920 0x7f043406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:05.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.020+0000 7f0459200640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f043406c920 0x7f043406ede0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f04540b34d0 tx=0x7f0440009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:05.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.023+0000 7f0451600640 1 -- 172.21.15.69:0/2214556507 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f044805a340 con 0x7f04540ad7e0 2024-04-24T08:15:05.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.237+0000 7f045b000640 1 -- 172.21.15.69:0/2214556507 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f045406f9e0 con 0x7f04540ad7e0 2024-04-24T08:15:05.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.238+0000 7f0451600640 1 -- 172.21.15.69:0/2214556507 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v71) ==== 80+0+124 (secure 0 0 0) 0x7f0448059c40 con 0x7f04540ad7e0 2024-04-24T08:15:05.239 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6837/1135108804 2024-04-25T08:13:37.186005+0000 2024-04-24T08:15:05.240 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6835/1135108804 2024-04-25T08:13:37.186005+0000 2024-04-24T08:15:05.240 INFO:teuthology.orchestra.run.smithi069.stderr:listed 2 entries 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 -- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f043406c920 msgr2=0x7f043406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f043406c920 0x7f043406ede0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f04540b34d0 tx=0x7f0440009040 comp rx=0 tx=0).stop 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 -- 172.21.15.69:0/2214556507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 msgr2=0x7f0454155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 0x7f0454155bd0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f044800d5f0 tx=0x7f044800dab0 comp rx=0 tx=0).stop 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 -- 172.21.15.69:0/2214556507 shutdown_connections 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f043406c920 0x7f043406ede0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0454156c60 0x7f0454157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04540b30f0 0x7f0454157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 --2- 172.21.15.69:0/2214556507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04540ad7e0 0x7f0454155bd0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 -- 172.21.15.69:0/2214556507 >> 172.21.15.69:0/2214556507 conn(0x7f04540d0160 msgr2=0x7f04540b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 -- 172.21.15.69:0/2214556507 shutdown_connections 2024-04-24T08:15:05.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:05.241+0000 7f045b000640 1 -- 172.21.15.69:0/2214556507 wait complete. 2024-04-24T08:15:05.261 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:15:05.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:05.261 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:05.332 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[60665]: fuse finished with error 0 and tester_r 0 2024-04-24T08:15:05.336 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:15:05.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:05.336 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:15:05.511 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:15:05.511 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:15:05.511 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:15:05.569 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:15:05.569 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:15:05.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:05.569 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-24T08:15:05.619 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-24T08:15:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:15:05.620 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:15:05.621 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:15:05.621 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:15:05.679 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:15:05.679 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:15:05.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:05.679 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-24T08:15:05.732 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-24T08:15:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:15:05.733 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:15:05.733 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:15:05.733 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:15:05.792 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:15:05.792 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:15:05.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:05.793 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-24T08:15:05.847 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-24T08:15:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:15:05.849 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:06.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.060+0000 7fe06e000640 1 Processor -- start 2024-04-24T08:15:06.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.060+0000 7fe06e000640 1 -- start start 2024-04-24T08:15:06.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.061+0000 7fe06e000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe06815d1d0 0x7fe0680ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.061+0000 7fe06e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 0x7fe0680ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.061+0000 7fe06e000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe06815d660 con 0x7fe068156c60 2024-04-24T08:15:06.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.061+0000 7fe06e000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe06806dc30 con 0x7fe06815d1d0 2024-04-24T08:15:06.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.061+0000 7fe06e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe06806df10 con 0x7fe0680b30f0 2024-04-24T08:15:06.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.062+0000 7fe06d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 0x7fe0680ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.062+0000 7fe067e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe06815d1d0 0x7fe0680ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.062+0000 7fe06d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 0x7fe0680ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:53950/0 (socket says 172.21.15.69:53950) 2024-04-24T08:15:06.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.062+0000 7fe06d600640 1 -- 172.21.15.69:0/4294514750 learned_addr learned my addr 172.21.15.69:0/4294514750 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:06.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.063+0000 7fe06d600640 1 -- 172.21.15.69:0/4294514750 >> v2:172.21.15.121:3300/0 conn(0x7fe06815d1d0 msgr2=0x7fe0680ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:06.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.063+0000 7fe06d600640 1 --2- 172.21.15.69:0/4294514750 >> v2:172.21.15.121:3300/0 conn(0x7fe06815d1d0 0x7fe0680ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.063+0000 7fe06d600640 1 -- 172.21.15.69:0/4294514750 >> v1:172.21.15.121:6789/0 conn(0x7fe068156c60 legacy=0x7fe06815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.063+0000 7fe06d600640 1 -- 172.21.15.69:0/4294514750 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe06806e1f0 con 0x7fe0680b30f0 2024-04-24T08:15:06.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.063+0000 7fe067e00640 1 --2- 172.21.15.69:0/4294514750 >> v2:172.21.15.121:3300/0 conn(0x7fe06815d1d0 0x7fe0680ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.064+0000 7fe06d600640 1 --2- 172.21.15.69:0/4294514750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 0x7fe0680ad610 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe060009e50 tx=0x7fe06002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dde529bc689cfa16 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.065+0000 7fe067400640 1 -- 172.21.15.69:0/4294514750 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe060036070 con 0x7fe0680b30f0 2024-04-24T08:15:06.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.065+0000 7fe067400640 1 -- 172.21.15.69:0/4294514750 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe060031600 con 0x7fe0680b30f0 2024-04-24T08:15:06.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.065+0000 7fe067400640 1 -- 172.21.15.69:0/4294514750 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0600319d0 con 0x7fe0680b30f0 2024-04-24T08:15:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.066+0000 7fe06e000640 1 -- 172.21.15.69:0/4294514750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 msgr2=0x7fe0680ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.066+0000 7fe06e000640 1 --2- 172.21.15.69:0/4294514750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 0x7fe0680ad610 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe060009e50 tx=0x7fe06002a710 comp rx=0 tx=0).stop 2024-04-24T08:15:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.066+0000 7fe06e000640 1 -- 172.21.15.69:0/4294514750 shutdown_connections 2024-04-24T08:15:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.066+0000 7fe06e000640 1 --2- 172.21.15.69:0/4294514750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 0x7fe0680ad610 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.066+0000 7fe06e000640 1 --2- 172.21.15.69:0/4294514750 >> v2:172.21.15.121:3300/0 conn(0x7fe06815d1d0 0x7fe0680ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.066+0000 7fe06e000640 1 -- 172.21.15.69:0/4294514750 >> 172.21.15.69:0/4294514750 conn(0x7fe0680d0160 msgr2=0x7fe0680d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.066+0000 7fe06e000640 1 -- 172.21.15.69:0/4294514750 shutdown_connections 2024-04-24T08:15:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.066+0000 7fe06e000640 1 -- 172.21.15.69:0/4294514750 wait complete. 2024-04-24T08:15:06.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.068+0000 7fe06e000640 1 Processor -- start 2024-04-24T08:15:06.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.068+0000 7fe06e000640 1 -- start start 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.069+0000 7fe06e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 0x7fe0680ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.069+0000 7fe06e000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe068156c60 0x7fe0681f4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.069+0000 7fe06e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe068155630 0x7fe0681f4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.069+0000 7fe06e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe06806e710 con 0x7fe068155630 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.069+0000 7fe06e000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe06815d940 con 0x7fe068156c60 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.069+0000 7fe06e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe06806e890 con 0x7fe0680b30f0 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.069+0000 7fe06d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe068155630 0x7fe0681f4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.069+0000 7fe06d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe068155630 0x7fe0681f4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34642/0 (socket says 172.21.15.69:34642) 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.069+0000 7fe06d600640 1 -- 172.21.15.69:0/2689172662 learned_addr learned my addr 172.21.15.69:0/2689172662 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe067e00640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe068156c60 0x7fe0681f4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe06d600640 1 -- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 msgr2=0x7fe0680ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe06cc00640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 0x7fe0680ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe06d600640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 0x7fe0680ca020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe06d600640 1 -- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe068156c60 msgr2=0x7fe0681f4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe06d600640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe068156c60 0x7fe0681f4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe06d600640 1 -- 172.21.15.69:0/2689172662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe068155b80 con 0x7fe068155630 2024-04-24T08:15:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe067e00640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe068156c60 0x7fe0681f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe06d600640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe068155630 0x7fe0681f4b70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe060034040 tx=0x7fe060004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe064c00640 1 -- 172.21.15.69:0/2689172662 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06003f020 con 0x7fe068155630 2024-04-24T08:15:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe064c00640 1 -- 172.21.15.69:0/2689172662 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe060037040 con 0x7fe068155630 2024-04-24T08:15:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe064c00640 1 -- 172.21.15.69:0/2689172662 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe060036070 con 0x7fe068155630 2024-04-24T08:15:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.070+0000 7fe06e000640 1 -- 172.21.15.69:0/2689172662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe068071d10 con 0x7fe068155630 2024-04-24T08:15:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.071+0000 7fe06e000640 1 -- 172.21.15.69:0/2689172662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0680722d0 con 0x7fe068155630 2024-04-24T08:15:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.072+0000 7fe06e000640 1 -- 172.21.15.69:0/2689172662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0680724d0 con 0x7fe068155630 2024-04-24T08:15:06.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.073+0000 7fe064c00640 1 -- 172.21.15.69:0/2689172662 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe060042020 con 0x7fe068155630 2024-04-24T08:15:06.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.074+0000 7fe064c00640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe04806c920 0x7fe04806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.074+0000 7fe064c00640 1 -- 172.21.15.69:0/2689172662 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fe0600ac2e0 con 0x7fe068155630 2024-04-24T08:15:06.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.074+0000 7fe06cc00640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe04806c920 0x7fe04806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.075+0000 7fe06cc00640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe04806c920 0x7fe04806ede0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0015e0 tx=0x7fe05c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:06.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.078+0000 7fe064c00640 1 -- 172.21.15.69:0/2689172662 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe0680724d0 con 0x7fe068155630 2024-04-24T08:15:06.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.316+0000 7fe06e000640 1 -- 172.21.15.69:0/2689172662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe068075d50 con 0x7fe068155630 2024-04-24T08:15:06.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.317+0000 7fe064c00640 1 -- 172.21.15.69:0/2689172662 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5188 (secure 0 0 0) 0x7fe060049020 con 0x7fe068155630 2024-04-24T08:15:06.319 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:06.319 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3504199143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3504199143},{"type":"v1","addr":"172.21.15.121:6837","nonce":3504199143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:14:32.246239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6792},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6792":{"gid":6792,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":23,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6792,"qdb_cluster":[6792]},"id":5}]} 2024-04-24T08:15:06.319 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 52 2024-04-24T08:15:06.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 -- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe04806c920 msgr2=0x7fe04806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe04806c920 0x7fe04806ede0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0015e0 tx=0x7fe05c009040 comp rx=0 tx=0).stop 2024-04-24T08:15:06.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 -- 172.21.15.69:0/2689172662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe068155630 msgr2=0x7fe0681f4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe068155630 0x7fe0681f4b70 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fe060034040 tx=0x7fe060004a40 comp rx=0 tx=0).stop 2024-04-24T08:15:06.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 -- 172.21.15.69:0/2689172662 shutdown_connections 2024-04-24T08:15:06.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe04806c920 0x7fe04806ede0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe068155630 0x7fe0681f4b70 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe068156c60 0x7fe0681f4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 --2- 172.21.15.69:0/2689172662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0680b30f0 0x7fe0680ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 -- 172.21.15.69:0/2689172662 >> 172.21.15.69:0/2689172662 conn(0x7fe0680d0160 msgr2=0x7fe0680b3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:06.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 -- 172.21.15.69:0/2689172662 shutdown_connections 2024-04-24T08:15:06.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.320+0000 7fe06e000640 1 -- 172.21.15.69:0/2689172662 wait complete. 2024-04-24T08:15:06.342 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:06.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.542+0000 7f0ca0400640 1 Processor -- start 2024-04-24T08:15:06.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.542+0000 7f0ca0400640 1 -- start start 2024-04-24T08:15:06.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.543+0000 7f0ca0400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 0x7f0c980b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.543+0000 7f0ca0400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0c9806f2d0 0x7f0c9806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.543+0000 7f0ca0400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c980b46e0 con 0x7f0c980cfbe0 2024-04-24T08:15:06.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.543+0000 7f0ca0400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0c9806fc70 con 0x7f0c9806f2d0 2024-04-24T08:15:06.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.543+0000 7f0ca0400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0c980d0960 con 0x7f0c9815b490 2024-04-24T08:15:06.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.544+0000 7f0c9d000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0c9806f2d0 0x7f0c9806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.544+0000 7f0c9d000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0c9806f2d0 0x7f0c9806f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34650/0 (socket says 172.21.15.69:34650) 2024-04-24T08:15:06.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.544+0000 7f0c9d000640 1 -- 172.21.15.69:0/2714655277 learned_addr learned my addr 172.21.15.69:0/2714655277 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:06.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.544+0000 7f0c9da00640 1 --2- 172.21.15.69:0/2714655277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 0x7f0c980b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.544+0000 7f0c9d000640 1 -- 172.21.15.69:0/2714655277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 msgr2=0x7f0c980b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:06.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.544+0000 7f0c9d000640 1 --2- 172.21.15.69:0/2714655277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 0x7f0c980b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.544+0000 7f0c9d000640 1 -- 172.21.15.69:0/2714655277 >> v1:172.21.15.121:6789/0 conn(0x7f0c980cfbe0 legacy=0x7f0c980d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:06.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.544+0000 7f0c9d000640 1 -- 172.21.15.69:0/2714655277 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c980b4980 con 0x7f0c9806f2d0 2024-04-24T08:15:06.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.544+0000 7f0c9da00640 1 --2- 172.21.15.69:0/2714655277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 0x7f0c980b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:06.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.545+0000 7f0c9d000640 1 --2- 172.21.15.69:0/2714655277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c9806f2d0 0x7f0c9806f730 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009870 tx=0x7f0c8402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e9dbe49722d806 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:06.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.545+0000 7f0c97e00640 1 -- 172.21.15.69:0/2714655277 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84037070 con 0x7f0c9806f2d0 2024-04-24T08:15:06.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.545+0000 7f0c97e00640 1 -- 172.21.15.69:0/2714655277 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c84032440 con 0x7f0c9806f2d0 2024-04-24T08:15:06.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.546+0000 7f0c97e00640 1 -- 172.21.15.69:0/2714655277 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84032810 con 0x7f0c9806f2d0 2024-04-24T08:15:06.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.546+0000 7f0ca0400640 1 -- 172.21.15.69:0/2714655277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c9806f2d0 msgr2=0x7f0c9806f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.546+0000 7f0ca0400640 1 --2- 172.21.15.69:0/2714655277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c9806f2d0 0x7f0c9806f730 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009870 tx=0x7f0c8402a210 comp rx=0 tx=0).stop 2024-04-24T08:15:06.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.546+0000 7f0ca0400640 1 -- 172.21.15.69:0/2714655277 shutdown_connections 2024-04-24T08:15:06.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.546+0000 7f0ca0400640 1 --2- 172.21.15.69:0/2714655277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c9806f2d0 0x7f0c9806f730 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.546+0000 7f0ca0400640 1 --2- 172.21.15.69:0/2714655277 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 0x7f0c980b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.546+0000 7f0ca0400640 1 -- 172.21.15.69:0/2714655277 >> 172.21.15.69:0/2714655277 conn(0x7f0c98067d90 msgr2=0x7f0c9806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:06.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.546+0000 7f0ca0400640 1 -- 172.21.15.69:0/2714655277 shutdown_connections 2024-04-24T08:15:06.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.546+0000 7f0ca0400640 1 -- 172.21.15.69:0/2714655277 wait complete. 2024-04-24T08:15:06.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.548+0000 7f0ca0400640 1 Processor -- start 2024-04-24T08:15:06.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.548+0000 7f0ca0400640 1 -- start start 2024-04-24T08:15:06.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.549+0000 7f0ca0400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c9806f2d0 0x7f0c98147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.549+0000 7f0ca0400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c980cfbe0 0x7f0c98145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.549+0000 7f0ca0400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 0x7f0c981e43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.549+0000 7f0ca0400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9815b870 con 0x7f0c980cfbe0 2024-04-24T08:15:06.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.549+0000 7f0ca0400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0c980d0c40 con 0x7f0c9806f2d0 2024-04-24T08:15:06.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.549+0000 7f0ca0400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0c980b56c0 con 0x7f0c9815b490 2024-04-24T08:15:06.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c980cfbe0 0x7f0c98145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9da00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c9806f2d0 0x7f0c98147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c980cfbe0 0x7f0c98145500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34658/0 (socket says 172.21.15.69:34658) 2024-04-24T08:15:06.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9d000640 1 -- 172.21.15.69:0/3634401847 learned_addr learned my addr 172.21.15.69:0/3634401847 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:06.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9e400640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 0x7f0c981e43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9d000640 1 -- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 msgr2=0x7f0c981e43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9d000640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 0x7f0c981e43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9d000640 1 -- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c9806f2d0 msgr2=0x7f0c98147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9d000640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c9806f2d0 0x7f0c98147ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9d000640 1 -- 172.21.15.69:0/3634401847 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c981e4930 con 0x7f0c980cfbe0 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.550+0000 7f0c9da00640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c9806f2d0 0x7f0c98147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.551+0000 7f0c9d000640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c980cfbe0 0x7f0c98145500 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f0c84000c00 tx=0x7f0c840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.551+0000 7f0c95600640 1 -- 172.21.15.69:0/3634401847 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84037040 con 0x7f0c980cfbe0 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.551+0000 7f0c95600640 1 -- 172.21.15.69:0/3634401847 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c84042070 con 0x7f0c980cfbe0 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.551+0000 7f0ca0400640 1 -- 172.21.15.69:0/3634401847 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c981482d0 con 0x7f0c980cfbe0 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.551+0000 7f0c95600640 1 -- 172.21.15.69:0/3634401847 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84030bd0 con 0x7f0c980cfbe0 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.551+0000 7f0ca0400640 1 -- 172.21.15.69:0/3634401847 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c98148890 con 0x7f0c980cfbe0 2024-04-24T08:15:06.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.552+0000 7f0ca0400640 1 -- 172.21.15.69:0/3634401847 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c98068a80 con 0x7f0c980cfbe0 2024-04-24T08:15:06.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.553+0000 7f0c95600640 1 -- 172.21.15.69:0/3634401847 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c84047460 con 0x7f0c980cfbe0 2024-04-24T08:15:06.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.554+0000 7f0c95600640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0c7806c8f0 0x7f0c7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.554+0000 7f0c95600640 1 -- 172.21.15.69:0/3634401847 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f0c840ac860 con 0x7f0c980cfbe0 2024-04-24T08:15:06.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.555+0000 7f0c9da00640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0c7806c8f0 0x7f0c7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.556+0000 7f0c9da00640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0c7806c8f0 0x7f0c7806edb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00aa30 tx=0x7f0c8c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:06.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.559+0000 7f0c95600640 1 -- 172.21.15.69:0/3634401847 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c8407abb0 con 0x7f0c980cfbe0 2024-04-24T08:15:06.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.762+0000 7f0ca0400640 1 -- 172.21.15.69:0/3634401847 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c981e4b60 con 0x7f0c980cfbe0 2024-04-24T08:15:06.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.765+0000 7f0c95600640 1 -- 172.21.15.69:0/3634401847 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16969 (secure 0 0 0) 0x7f0c8407a4b0 con 0x7f0c980cfbe0 2024-04-24T08:15:06.767 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:06.768 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":71,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:14:03.353108+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.17","mappings":[{"from":6,"to":5}]},{"pgid":"11.22","mappings":[{"from":6,"to":0}]},{"pgid":"11.2e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6837/1135108804":"2024-04-25T08:13:37.186005+0000","172.21.15.121:6835/1135108804":"2024-04-25T08:13:37.186005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:06.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 -- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0c7806c8f0 msgr2=0x7f0c7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0c7806c8f0 0x7f0c7806edb0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00aa30 tx=0x7f0c8c008040 comp rx=0 tx=0).stop 2024-04-24T08:15:06.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 -- 172.21.15.69:0/3634401847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c980cfbe0 msgr2=0x7f0c98145500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c980cfbe0 0x7f0c98145500 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f0c84000c00 tx=0x7f0c840046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 -- 172.21.15.69:0/3634401847 shutdown_connections 2024-04-24T08:15:06.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0c7806c8f0 0x7f0c7806edb0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0c980cfbe0 0x7f0c98145500 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0c9815b490 0x7f0c981e43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 --2- 172.21.15.69:0/3634401847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0c9806f2d0 0x7f0c98147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 -- 172.21.15.69:0/3634401847 >> 172.21.15.69:0/3634401847 conn(0x7f0c98067d90 msgr2=0x7f0c980cf340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:06.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 -- 172.21.15.69:0/3634401847 shutdown_connections 2024-04-24T08:15:06.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.768+0000 7f0ca0400640 1 -- 172.21.15.69:0/3634401847 wait complete. 2024-04-24T08:15:06.789 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:13:13.250209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:13:14.468238+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:15:06.789 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:15:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.990+0000 7f1f74600640 1 Processor -- start 2024-04-24T08:15:06.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.990+0000 7f1f74600640 1 -- start start 2024-04-24T08:15:06.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.992+0000 7f1f74600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f6c15c820 0x7f1f6c0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.992+0000 7f1f74600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1f6c0ce250 0x7f1f6c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:06.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.992+0000 7f1f74600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f6c15cc00 con 0x7f1f6c0b46a0 2024-04-24T08:15:06.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.992+0000 7f1f74600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1f6c06ed30 con 0x7f1f6c15c820 2024-04-24T08:15:06.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.992+0000 7f1f74600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1f6c0b4e10 con 0x7f1f6c0ce250 2024-04-24T08:15:06.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.992+0000 7f1f71c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f6c15c820 0x7f1f6c0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.992+0000 7f1f71c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f6c15c820 0x7f1f6c0b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34668/0 (socket says 172.21.15.69:34668) 2024-04-24T08:15:06.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.992+0000 7f1f71c00640 1 -- 172.21.15.69:0/1348059468 learned_addr learned my addr 172.21.15.69:0/1348059468 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:06.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.992+0000 7f1f72600640 1 --2- 172.21.15.69:0/1348059468 >> v2:172.21.15.121:3300/0 conn(0x7f1f6c0ce250 0x7f1f6c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:06.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.993+0000 7f1f71c00640 1 -- 172.21.15.69:0/1348059468 >> v2:172.21.15.121:3300/0 conn(0x7f1f6c0ce250 msgr2=0x7f1f6c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:06.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.993+0000 7f1f71c00640 1 --2- 172.21.15.69:0/1348059468 >> v2:172.21.15.121:3300/0 conn(0x7f1f6c0ce250 0x7f1f6c0ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.993+0000 7f1f71c00640 1 -- 172.21.15.69:0/1348059468 >> v1:172.21.15.121:6789/0 conn(0x7f1f6c0b46a0 legacy=0x7f1f6c06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:06.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.993+0000 7f1f71c00640 1 -- 172.21.15.69:0/1348059468 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6c0b50f0 con 0x7f1f6c15c820 2024-04-24T08:15:06.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.994+0000 7f1f71c00640 1 --2- 172.21.15.69:0/1348059468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 0x7f1f6c0b3e90 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c161420 tx=0x7f1f6002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943f8b0a5e86f425 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:06.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.994+0000 7f1f6be00640 1 -- 172.21.15.69:0/1348059468 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60037070 con 0x7f1f6c15c820 2024-04-24T08:15:06.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.994+0000 7f1f6be00640 1 -- 172.21.15.69:0/1348059468 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f60032440 con 0x7f1f6c15c820 2024-04-24T08:15:06.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.994+0000 7f1f6be00640 1 -- 172.21.15.69:0/1348059468 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60032810 con 0x7f1f6c15c820 2024-04-24T08:15:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.995+0000 7f1f74600640 1 -- 172.21.15.69:0/1348059468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 msgr2=0x7f1f6c0b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.995+0000 7f1f74600640 1 --2- 172.21.15.69:0/1348059468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 0x7f1f6c0b3e90 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c161420 tx=0x7f1f6002a230 comp rx=0 tx=0).stop 2024-04-24T08:15:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.995+0000 7f1f74600640 1 -- 172.21.15.69:0/1348059468 shutdown_connections 2024-04-24T08:15:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.995+0000 7f1f74600640 1 --2- 172.21.15.69:0/1348059468 >> v2:172.21.15.121:3300/0 conn(0x7f1f6c0ce250 0x7f1f6c0ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.995+0000 7f1f74600640 1 --2- 172.21.15.69:0/1348059468 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 0x7f1f6c0b3e90 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.995+0000 7f1f74600640 1 -- 172.21.15.69:0/1348059468 >> 172.21.15.69:0/1348059468 conn(0x7f1f6c0d0350 msgr2=0x7f1f6c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.995+0000 7f1f74600640 1 -- 172.21.15.69:0/1348059468 shutdown_connections 2024-04-24T08:15:06.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.995+0000 7f1f74600640 1 -- 172.21.15.69:0/1348059468 wait complete. 2024-04-24T08:15:06.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.997+0000 7f1f74600640 1 Processor -- start 2024-04-24T08:15:06.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.997+0000 7f1f74600640 1 -- start start 2024-04-24T08:15:07.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.998+0000 7f1f74600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f6c0b46a0 0x7f1f6c146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.998+0000 7f1f74600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f6c0ce250 0x7f1f6c0cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.998+0000 7f1f74600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 0x7f1f6c1e64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.998+0000 7f1f74600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c0b4b30 con 0x7f1f6c15c820 2024-04-24T08:15:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.998+0000 7f1f74600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f6c06f2b0 con 0x7f1f6c0b46a0 2024-04-24T08:15:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.998+0000 7f1f74600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f6c06fee0 con 0x7f1f6c0ce250 2024-04-24T08:15:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f72600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 0x7f1f6c1e64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f72600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 0x7f1f6c1e64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34680/0 (socket says 172.21.15.69:34680) 2024-04-24T08:15:07.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f72600640 1 -- 172.21.15.69:0/2362558144 learned_addr learned my addr 172.21.15.69:0/2362558144 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f71200640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f6c0ce250 0x7f1f6c0cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f71c00640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f6c0b46a0 0x7f1f6c146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f72600640 1 -- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f6c0ce250 msgr2=0x7f1f6c0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f72600640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f6c0ce250 0x7f1f6c0cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f72600640 1 -- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f6c0b46a0 msgr2=0x7f1f6c146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f72600640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f6c0b46a0 0x7f1f6c146700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f72600640 1 -- 172.21.15.69:0/2362558144 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6c146ce0 con 0x7f1f6c15c820 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:06.999+0000 7f1f71200640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f6c0ce250 0x7f1f6c0cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.000+0000 7f1f72600640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 0x7f1f6c1e64f0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f1f64007e40 tx=0x7f1f6400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.000+0000 7f1f69600640 1 -- 172.21.15.69:0/2362558144 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64014620 con 0x7f1f6c15c820 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.000+0000 7f1f69600640 1 -- 172.21.15.69:0/2362558144 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f64019070 con 0x7f1f6c15c820 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.000+0000 7f1f69600640 1 -- 172.21.15.69:0/2362558144 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f64014a20 con 0x7f1f6c15c820 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.000+0000 7f1f74600640 1 -- 172.21.15.69:0/2362558144 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6c146fc0 con 0x7f1f6c15c820 2024-04-24T08:15:07.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.000+0000 7f1f74600640 1 -- 172.21.15.69:0/2362558144 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f6c147580 con 0x7f1f6c15c820 2024-04-24T08:15:07.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.001+0000 7f1f74600640 1 -- 172.21.15.69:0/2362558144 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f6c068a80 con 0x7f1f6c15c820 2024-04-24T08:15:07.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.002+0000 7f1f69600640 1 -- 172.21.15.69:0/2362558144 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f64014bc0 con 0x7f1f6c15c820 2024-04-24T08:15:07.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.003+0000 7f1f69600640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4c06c8f0 0x7f1f4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.003+0000 7f1f69600640 1 -- 172.21.15.69:0/2362558144 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f1f6408c540 con 0x7f1f6c15c820 2024-04-24T08:15:07.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.004+0000 7f1f71c00640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4c06c8f0 0x7f1f4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.005+0000 7f1f71c00640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4c06c8f0 0x7f1f4c06edb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1f60034000 tx=0x7f1f6002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.007+0000 7f1f69600640 1 -- 172.21.15.69:0/2362558144 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f6405a890 con 0x7f1f6c15c820 2024-04-24T08:15:07.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.212+0000 7f1f74600640 1 -- 172.21.15.69:0/2362558144 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1f6c160f80 con 0x7f1f6c15c820 2024-04-24T08:15:07.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.212+0000 7f1f69600640 1 -- 172.21.15.69:0/2362558144 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) ==== 93+0+10 (secure 0 0 0) 0x7f1f6405a190 con 0x7f1f6c15c820 2024-04-24T08:15:07.214 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:15:07.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.215+0000 7f1f74600640 1 -- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4c06c8f0 msgr2=0x7f1f4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.215+0000 7f1f74600640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4c06c8f0 0x7f1f4c06edb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1f60034000 tx=0x7f1f6002e000 comp rx=0 tx=0).stop 2024-04-24T08:15:07.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.215+0000 7f1f74600640 1 -- 172.21.15.69:0/2362558144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 msgr2=0x7f1f6c1e64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.215+0000 7f1f74600640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 0x7f1f6c1e64f0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f1f64007e40 tx=0x7f1f6400e4b0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.215+0000 7f1f74600640 1 -- 172.21.15.69:0/2362558144 shutdown_connections 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.215+0000 7f1f74600640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f4c06c8f0 0x7f1f4c06edb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.215+0000 7f1f74600640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f6c15c820 0x7f1f6c1e64f0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.215+0000 7f1f74600640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f6c0ce250 0x7f1f6c0cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.215+0000 7f1f74600640 1 --2- 172.21.15.69:0/2362558144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f6c0b46a0 0x7f1f6c146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.216+0000 7f1f74600640 1 -- 172.21.15.69:0/2362558144 >> 172.21.15.69:0/2362558144 conn(0x7f1f6c0d0350 msgr2=0x7f1f6c0b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.216+0000 7f1f74600640 1 -- 172.21.15.69:0/2362558144 shutdown_connections 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.216+0000 7f1f74600640 1 -- 172.21.15.69:0/2362558144 wait complete. 2024-04-24T08:15:07.237 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:15:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.441+0000 7f35df200640 1 Processor -- start 2024-04-24T08:15:07.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.441+0000 7f35df200640 1 -- start start 2024-04-24T08:15:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.443+0000 7f35df200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f35e006cfb0 0x7f35e006d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.443+0000 7f35df200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e015ed60 con 0x7f35e0162f90 2024-04-24T08:15:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.443+0000 7f35df200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e00d1580 con 0x7f35e015e980 2024-04-24T08:15:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.443+0000 7f35df200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f35e01633d0 con 0x7f35e006cfb0 2024-04-24T08:15:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.443+0000 7f35d5400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f35e0162f90 0x7f35e00cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53168/0 (socket says 172.21.15.69:53168) 2024-04-24T08:15:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.443+0000 7f35d5400640 1 -- 172.21.15.69:0/1546191039 learned_addr learned my addr 172.21.15.69:0/1546191039 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.443+0000 7f35de800640 1 --2- 172.21.15.69:0/1546191039 >> v2:172.21.15.69:3300/0 conn(0x7f35e006cfb0 0x7f35e006d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.444+0000 7f35dd400640 1 -- 172.21.15.69:0/1546191039 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3338333074 0 0) 0x7f35e015ed60 con 0x7f35e0162f90 2024-04-24T08:15:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.444+0000 7f35dd400640 1 -- 172.21.15.69:0/1546191039 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35c4003520 con 0x7f35e0162f90 2024-04-24T08:15:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.444+0000 7f35de800640 1 -- 172.21.15.69:0/1546191039 >> v1:172.21.15.121:6789/0 conn(0x7f35e015e980 legacy=0x7f35e00d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.444+0000 7f35de800640 1 -- 172.21.15.69:0/1546191039 >> v1:172.21.15.69:6789/0 conn(0x7f35e0162f90 legacy=0x7f35e00cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.444+0000 7f35de800640 1 -- 172.21.15.69:0/1546191039 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e00d1860 con 0x7f35e006cfb0 2024-04-24T08:15:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.445+0000 7f35de800640 1 --2- 172.21.15.69:0/1546191039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 0x7f35e006d440 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f35d8009d90 tx=0x7f35d802a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ff0f43dc8a318cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.445+0000 7f35dd400640 1 -- 172.21.15.69:0/1546191039 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8037070 con 0x7f35e006cfb0 2024-04-24T08:15:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.445+0000 7f35dd400640 1 -- 172.21.15.69:0/1546191039 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d8007740 con 0x7f35e006cfb0 2024-04-24T08:15:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.445+0000 7f35dd400640 1 -- 172.21.15.69:0/1546191039 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8007b10 con 0x7f35e006cfb0 2024-04-24T08:15:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.446+0000 7f35df200640 1 -- 172.21.15.69:0/1546191039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 msgr2=0x7f35e006d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.446+0000 7f35df200640 1 --2- 172.21.15.69:0/1546191039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 0x7f35e006d440 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f35d8009d90 tx=0x7f35d802a780 comp rx=0 tx=0).stop 2024-04-24T08:15:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.446+0000 7f35df200640 1 -- 172.21.15.69:0/1546191039 shutdown_connections 2024-04-24T08:15:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.446+0000 7f35df200640 1 --2- 172.21.15.69:0/1546191039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 0x7f35e006d440 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.446+0000 7f35df200640 1 -- 172.21.15.69:0/1546191039 >> 172.21.15.69:0/1546191039 conn(0x7f35e009d920 msgr2=0x7f35e008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.446+0000 7f35df200640 1 -- 172.21.15.69:0/1546191039 shutdown_connections 2024-04-24T08:15:07.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.446+0000 7f35df200640 1 -- 172.21.15.69:0/1546191039 wait complete. 2024-04-24T08:15:07.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.448+0000 7f35df200640 1 Processor -- start 2024-04-24T08:15:07.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.448+0000 7f35df200640 1 -- start start 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.449+0000 7f35df200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 0x7f35e00c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.449+0000 7f35df200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e015e980 0x7f35e0159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.449+0000 7f35df200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35e0162f90 0x7f35e015ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.449+0000 7f35df200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f35e00d0360 con 0x7f35e006cfb0 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.449+0000 7f35df200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f35e00cf730 con 0x7f35e015e980 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.449+0000 7f35df200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f35e00cf070 con 0x7f35e0162f90 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.449+0000 7f35dde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 0x7f35e00c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.449+0000 7f35dde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 0x7f35e00c9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34690/0 (socket says 172.21.15.69:34690) 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.449+0000 7f35dde00640 1 -- 172.21.15.69:0/3359902378 learned_addr learned my addr 172.21.15.69:0/3359902378 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35d5400640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e015e980 0x7f35e0159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35de800640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35e0162f90 0x7f35e015ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35dde00640 1 -- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35e0162f90 msgr2=0x7f35e015ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35dde00640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35e0162f90 0x7f35e015ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35dde00640 1 -- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e015e980 msgr2=0x7f35e0159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35dde00640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e015e980 0x7f35e0159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35dde00640 1 -- 172.21.15.69:0/3359902378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e00ca4a0 con 0x7f35e006cfb0 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35de800640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35e0162f90 0x7f35e015ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35dde00640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 0x7f35e00c9c50 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f35d000c430 tx=0x7f35d000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35d6a00640 1 -- 172.21.15.69:0/3359902378 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d00052d0 con 0x7f35e006cfb0 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35d6a00640 1 -- 172.21.15.69:0/3359902378 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35d0005490 con 0x7f35e006cfb0 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35df200640 1 -- 172.21.15.69:0/3359902378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e00ca780 con 0x7f35e006cfb0 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.450+0000 7f35d6a00640 1 -- 172.21.15.69:0/3359902378 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d0007260 con 0x7f35e006cfb0 2024-04-24T08:15:07.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.451+0000 7f35df200640 1 -- 172.21.15.69:0/3359902378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e00cad40 con 0x7f35e006cfb0 2024-04-24T08:15:07.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.452+0000 7f35df200640 1 -- 172.21.15.69:0/3359902378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e00caf80 con 0x7f35e006cfb0 2024-04-24T08:15:07.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.453+0000 7f35d6a00640 1 -- 172.21.15.69:0/3359902378 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f35d0007400 con 0x7f35e006cfb0 2024-04-24T08:15:07.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.454+0000 7f35d6a00640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35bc06c8f0 0x7f35bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.454+0000 7f35d6a00640 1 -- 172.21.15.69:0/3359902378 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f35d008b220 con 0x7f35e006cfb0 2024-04-24T08:15:07.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.455+0000 7f35d5400640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35bc06c8f0 0x7f35bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.456+0000 7f35d5400640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35bc06c8f0 0x7f35bc06edb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f35e00c7b40 tx=0x7f35c8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.459+0000 7f35d6a00640 1 -- 172.21.15.69:0/3359902378 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f35e00caf80 con 0x7f35e006cfb0 2024-04-24T08:15:07.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.655+0000 7f35df200640 1 -- 172.21.15.69:0/3359902378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f35e0075d50 con 0x7f35e006cfb0 2024-04-24T08:15:07.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.656+0000 7f35d6a00640 1 -- 172.21.15.69:0/3359902378 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) ==== 104+0+11 (secure 0 0 0) 0x7f35d00594e0 con 0x7f35e006cfb0 2024-04-24T08:15:07.658 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 -- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35bc06c8f0 msgr2=0x7f35bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35bc06c8f0 0x7f35bc06edb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f35e00c7b40 tx=0x7f35c8009040 comp rx=0 tx=0).stop 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 -- 172.21.15.69:0/3359902378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 msgr2=0x7f35e00c9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 0x7f35e00c9c50 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f35d000c430 tx=0x7f35d000c8f0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 -- 172.21.15.69:0/3359902378 shutdown_connections 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35bc06c8f0 0x7f35bc06edb0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f35e0162f90 0x7f35e015ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e015e980 0x7f35e0159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 --2- 172.21.15.69:0/3359902378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e006cfb0 0x7f35e00c9c50 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 -- 172.21.15.69:0/3359902378 >> 172.21.15.69:0/3359902378 conn(0x7f35e009d920 msgr2=0x7f35e00b1f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 -- 172.21.15.69:0/3359902378 shutdown_connections 2024-04-24T08:15:07.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.659+0000 7f35df200640 1 -- 172.21.15.69:0/3359902378 wait complete. 2024-04-24T08:15:07.678 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:15:07.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.880+0000 7f7c8be00640 1 Processor -- start 2024-04-24T08:15:07.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.881+0000 7f7c8be00640 1 -- start start 2024-04-24T08:15:07.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.882+0000 7f7c8be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7c8c15d030 0x7f7c8c163b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.882+0000 7f7c8be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7c8c071d70 0x7f7c8c072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.882+0000 7f7c8be00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c8c1640a0 con 0x7f7c8c1666d0 2024-04-24T08:15:07.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.882+0000 7f7c8be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7c8c0b39e0 con 0x7f7c8c071d70 2024-04-24T08:15:07.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.882+0000 7f7c8be00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7c8c0b3700 con 0x7f7c8c15d030 2024-04-24T08:15:07.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.882+0000 7f7c83e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7c8c071d70 0x7f7c8c072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.883+0000 7f7c83e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7c8c071d70 0x7f7c8c072220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34698/0 (socket says 172.21.15.69:34698) 2024-04-24T08:15:07.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.883+0000 7f7c83e00640 1 -- 172.21.15.69:0/4215307721 learned_addr learned my addr 172.21.15.69:0/4215307721 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.883+0000 7f7c8aa00640 1 --2- 172.21.15.69:0/4215307721 >> v2:172.21.15.121:3300/0 conn(0x7f7c8c15d030 0x7f7c8c163b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.883+0000 7f7c83e00640 1 -- 172.21.15.69:0/4215307721 >> v2:172.21.15.121:3300/0 conn(0x7f7c8c15d030 msgr2=0x7f7c8c163b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.883+0000 7f7c83e00640 1 --2- 172.21.15.69:0/4215307721 >> v2:172.21.15.121:3300/0 conn(0x7f7c8c15d030 0x7f7c8c163b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.883+0000 7f7c83e00640 1 -- 172.21.15.69:0/4215307721 >> v1:172.21.15.121:6789/0 conn(0x7f7c8c1666d0 legacy=0x7f7c8c166b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:07.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.883+0000 7f7c83e00640 1 -- 172.21.15.69:0/4215307721 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8c0b3e20 con 0x7f7c8c071d70 2024-04-24T08:15:07.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.885+0000 7f7c83e00640 1 --2- 172.21.15.69:0/4215307721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c071d70 0x7f7c8c072220 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c06ffa0 tx=0x7f7c7402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7fcd14aee37451 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.885+0000 7f7c8a000640 1 -- 172.21.15.69:0/4215307721 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74037070 con 0x7f7c8c071d70 2024-04-24T08:15:07.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.885+0000 7f7c8a000640 1 -- 172.21.15.69:0/4215307721 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c74032440 con 0x7f7c8c071d70 2024-04-24T08:15:07.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.885+0000 7f7c8a000640 1 -- 172.21.15.69:0/4215307721 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c74032810 con 0x7f7c8c071d70 2024-04-24T08:15:07.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.885+0000 7f7c8be00640 1 -- 172.21.15.69:0/4215307721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c071d70 msgr2=0x7f7c8c072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.885+0000 7f7c8be00640 1 --2- 172.21.15.69:0/4215307721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c071d70 0x7f7c8c072220 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c06ffa0 tx=0x7f7c7402a230 comp rx=0 tx=0).stop 2024-04-24T08:15:07.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.886+0000 7f7c8be00640 1 -- 172.21.15.69:0/4215307721 shutdown_connections 2024-04-24T08:15:07.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.886+0000 7f7c8be00640 1 --2- 172.21.15.69:0/4215307721 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c071d70 0x7f7c8c072220 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.886+0000 7f7c8be00640 1 --2- 172.21.15.69:0/4215307721 >> v2:172.21.15.121:3300/0 conn(0x7f7c8c15d030 0x7f7c8c163b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.886+0000 7f7c8be00640 1 -- 172.21.15.69:0/4215307721 >> 172.21.15.69:0/4215307721 conn(0x7f7c8c067b80 msgr2=0x7f7c8c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.886+0000 7f7c8be00640 1 -- 172.21.15.69:0/4215307721 shutdown_connections 2024-04-24T08:15:07.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.886+0000 7f7c8be00640 1 -- 172.21.15.69:0/4215307721 wait complete. 2024-04-24T08:15:07.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.887+0000 7f7c8be00640 1 Processor -- start 2024-04-24T08:15:07.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.888+0000 7f7c8be00640 1 -- start start 2024-04-24T08:15:07.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c8c071d70 0x7f7c8c1620e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c8c15d030 0x7f7c8c15fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c1666d0 0x7f7c8c146080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c06f7b0 con 0x7f7c8c1666d0 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8c0b43e0 con 0x7f7c8c15d030 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7c8c0b3320 con 0x7f7c8c071d70 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c1666d0 0x7f7c8c146080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c1666d0 0x7f7c8c146080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34704/0 (socket says 172.21.15.69:34704) 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8b400640 1 -- 172.21.15.69:0/3457775874 learned_addr learned my addr 172.21.15.69:0/3457775874 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8aa00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c8c071d70 0x7f7c8c1620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8b400640 1 -- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c8c071d70 msgr2=0x7f7c8c1620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8b400640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c8c071d70 0x7f7c8c1620e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c83e00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c8c15d030 0x7f7c8c15fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8b400640 1 -- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c8c15d030 msgr2=0x7f7c8c15fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8b400640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c8c15d030 0x7f7c8c15fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.889+0000 7f7c8b400640 1 -- 172.21.15.69:0/3457775874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8c15f320 con 0x7f7c8c1666d0 2024-04-24T08:15:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.890+0000 7f7c8b400640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c1666d0 0x7f7c8c146080 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400c780 tx=0x7f7c8400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.890+0000 7f7c82a00640 1 -- 172.21.15.69:0/3457775874 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c84017070 con 0x7f7c8c1666d0 2024-04-24T08:15:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.890+0000 7f7c82a00640 1 -- 172.21.15.69:0/3457775874 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c84003da0 con 0x7f7c8c1666d0 2024-04-24T08:15:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.890+0000 7f7c82a00640 1 -- 172.21.15.69:0/3457775874 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c840052b0 con 0x7f7c8c1666d0 2024-04-24T08:15:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.890+0000 7f7c8be00640 1 -- 172.21.15.69:0/3457775874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c8c162620 con 0x7f7c8c1666d0 2024-04-24T08:15:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.890+0000 7f7c8be00640 1 -- 172.21.15.69:0/3457775874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c8c146d20 con 0x7f7c8c1666d0 2024-04-24T08:15:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.891+0000 7f7c8be00640 1 -- 172.21.15.69:0/3457775874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c8c075d50 con 0x7f7c8c1666d0 2024-04-24T08:15:07.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.892+0000 7f7c82a00640 1 -- 172.21.15.69:0/3457775874 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7c84005450 con 0x7f7c8c1666d0 2024-04-24T08:15:07.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.893+0000 7f7c82a00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7c6806c8f0 0x7f7c6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.893+0000 7f7c82a00640 1 -- 172.21.15.69:0/3457775874 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f7c8408b2b0 con 0x7f7c8c1666d0 2024-04-24T08:15:07.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.894+0000 7f7c8aa00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7c6806c8f0 0x7f7c6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.895+0000 7f7c8aa00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7c6806c8f0 0x7f7c6806edb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c006fd0 tx=0x7f7c7c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:07.897+0000 7f7c82a00640 1 -- 172.21.15.69:0/3457775874 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c8c15f320 con 0x7f7c8c1666d0 2024-04-24T08:15:08.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.078+0000 7f7c8be00640 1 -- 172.21.15.69:0/3457775874 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7c8c157140 con 0x7f7c8c1666d0 2024-04-24T08:15:08.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.079+0000 7f7c82a00640 1 -- 172.21.15.69:0/3457775874 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) ==== 100+0+11 (secure 0 0 0) 0x7f7c84059570 con 0x7f7c8c1666d0 2024-04-24T08:15:08.081 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:15:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 -- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7c6806c8f0 msgr2=0x7f7c6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7c6806c8f0 0x7f7c6806edb0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c006fd0 tx=0x7f7c7c008040 comp rx=0 tx=0).stop 2024-04-24T08:15:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 -- 172.21.15.69:0/3457775874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c1666d0 msgr2=0x7f7c8c146080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c1666d0 0x7f7c8c146080 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f7c8400c780 tx=0x7f7c8400cc40 comp rx=0 tx=0).stop 2024-04-24T08:15:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 -- 172.21.15.69:0/3457775874 shutdown_connections 2024-04-24T08:15:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7c6806c8f0 0x7f7c6806edb0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c8c1666d0 0x7f7c8c146080 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c8c15d030 0x7f7c8c15fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 --2- 172.21.15.69:0/3457775874 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7c8c071d70 0x7f7c8c1620e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 -- 172.21.15.69:0/3457775874 >> 172.21.15.69:0/3457775874 conn(0x7f7c8c067b80 msgr2=0x7f7c8c155700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:08.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 -- 172.21.15.69:0/3457775874 shutdown_connections 2024-04-24T08:15:08.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.081+0000 7f7c8be00640 1 -- 172.21.15.69:0/3457775874 wait complete. 2024-04-24T08:15:08.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:08.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.294+0000 7f40c2000640 1 Processor -- start 2024-04-24T08:15:08.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.294+0000 7f40c2000640 1 -- start start 2024-04-24T08:15:08.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.295+0000 7f40c2000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f40bc156c60 0x7f40bc15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.295+0000 7f40c2000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40bc15d660 con 0x7f40bc0b30f0 2024-04-24T08:15:08.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.295+0000 7f40c2000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40bc06dc30 con 0x7f40bc0ad7e0 2024-04-24T08:15:08.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.295+0000 7f40c2000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f40bc06df10 con 0x7f40bc156c60 2024-04-24T08:15:08.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.295+0000 7f40bbe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f40bc0b30f0 0x7f40bc0ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53176/0 (socket says 172.21.15.69:53176) 2024-04-24T08:15:08.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.295+0000 7f40bbe00640 1 -- 172.21.15.69:0/3533803228 learned_addr learned my addr 172.21.15.69:0/3533803228 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40c0c00640 1 --2- 172.21.15.69:0/3533803228 >> v2:172.21.15.121:3300/0 conn(0x7f40bc156c60 0x7f40bc15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3193627369 0 0) 0x7f40bc15d660 con 0x7f40bc0b30f0 2024-04-24T08:15:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40a4003520 con 0x7f40bc0b30f0 2024-04-24T08:15:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1009490831 0 0) 0x7f40a4003520 con 0x7f40bc0b30f0 2024-04-24T08:15:08.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 >> v2:172.21.15.121:3300/0 conn(0x7f40bc156c60 msgr2=0x7f40bc15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 --2- 172.21.15.69:0/3533803228 >> v2:172.21.15.121:3300/0 conn(0x7f40bc156c60 0x7f40bc15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 >> v1:172.21.15.121:6789/0 conn(0x7f40bc0ad7e0 legacy=0x7f40bc06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40bc06e1f0 con 0x7f40bc0b30f0 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f40a8001670 con 0x7f40bc0b30f0 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 >> v1:172.21.15.69:6789/0 conn(0x7f40bc0b30f0 legacy=0x7f40bc0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a40039b0 0x7f40a4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4004800 0x7f40a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.296+0000 7f40bb400640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a40051c0 0x7f40a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f40bc06df10 con 0x7f40a40051c0 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f40a400c040 con 0x7f40a40039b0 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f40a400c240 con 0x7f40a4004800 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40c1600640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a40051c0 0x7f40a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40c0c00640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a40039b0 0x7f40a4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40bbe00640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4004800 0x7f40a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40c1600640 1 -- 172.21.15.69:0/3533803228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4004800 msgr2=0x7f40a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40c1600640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a4004800 0x7f40a4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40c1600640 1 -- 172.21.15.69:0/3533803228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a40039b0 msgr2=0x7f40a4003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40c1600640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40a40039b0 0x7f40a4003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.297+0000 7f40c1600640 1 -- 172.21.15.69:0/3533803228 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a4003520 con 0x7f40a40051c0 2024-04-24T08:15:08.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.298+0000 7f40c1600640 1 -- 172.21.15.69:0/3533803228 reap_dead start 2024-04-24T08:15:08.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.298+0000 7f40c1600640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a40051c0 0x7f40a400b9c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f40bc06d590 tx=0x7f40a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7385beb2d57923d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:08.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.299+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b40040a0 con 0x7f40a40051c0 2024-04-24T08:15:08.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.299+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b4006d70 con 0x7f40a40051c0 2024-04-24T08:15:08.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.299+0000 7f40bb400640 1 -- 172.21.15.69:0/3533803228 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b402ad80 con 0x7f40a40051c0 2024-04-24T08:15:08.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.299+0000 7f40c2000640 1 -- 172.21.15.69:0/3533803228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a40051c0 msgr2=0x7f40a400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.299+0000 7f40c2000640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a40051c0 0x7f40a400b9c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f40bc06d590 tx=0x7f40a4005830 comp rx=0 tx=0).stop 2024-04-24T08:15:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.299+0000 7f40c2000640 1 -- 172.21.15.69:0/3533803228 shutdown_connections 2024-04-24T08:15:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.299+0000 7f40c2000640 1 --2- 172.21.15.69:0/3533803228 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40a40051c0 0x7f40a400b9c0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.299+0000 7f40c2000640 1 -- 172.21.15.69:0/3533803228 >> 172.21.15.69:0/3533803228 conn(0x7f40bc0d0160 msgr2=0x7f40bc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.299+0000 7f40c2000640 1 -- 172.21.15.69:0/3533803228 shutdown_connections 2024-04-24T08:15:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.299+0000 7f40c2000640 1 -- 172.21.15.69:0/3533803228 wait complete. 2024-04-24T08:15:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.301+0000 7f40c2000640 1 Processor -- start 2024-04-24T08:15:08.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.301+0000 7f40c2000640 1 -- start start 2024-04-24T08:15:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.302+0000 7f40c2000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a40051c0 0x7f40bc153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.302+0000 7f40c2000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40bc0cb630 0x7f40bc0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.302+0000 7f40c2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40bc072710 0x7f40bc072bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.302+0000 7f40c2000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f40bc0ad0d0 con 0x7f40bc072710 2024-04-24T08:15:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.302+0000 7f40c2000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f40bc15d940 con 0x7f40bc0cb630 2024-04-24T08:15:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.302+0000 7f40c2000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f40bc0ad250 con 0x7f40a40051c0 2024-04-24T08:15:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.302+0000 7f40c1600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40bc072710 0x7f40bc072bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.302+0000 7f40c1600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40bc072710 0x7f40bc072bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34708/0 (socket says 172.21.15.69:34708) 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.302+0000 7f40c1600640 1 -- 172.21.15.69:0/25970783 learned_addr learned my addr 172.21.15.69:0/25970783 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40bbe00640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40bc0cb630 0x7f40bc0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40c0c00640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a40051c0 0x7f40bc153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40c1600640 1 -- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a40051c0 msgr2=0x7f40bc153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40c1600640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a40051c0 0x7f40bc153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40c1600640 1 -- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40bc0cb630 msgr2=0x7f40bc0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40c1600640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40bc0cb630 0x7f40bc0cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40c1600640 1 -- 172.21.15.69:0/25970783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40bc15b300 con 0x7f40bc072710 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40c1600640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40bc072710 0x7f40bc072bc0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f40b402a7f0 tx=0x7f40b4003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40b8c00640 1 -- 172.21.15.69:0/25970783 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b40040a0 con 0x7f40bc072710 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40b8c00640 1 -- 172.21.15.69:0/25970783 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b4042070 con 0x7f40bc072710 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40c2000640 1 -- 172.21.15.69:0/25970783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40bc15b5e0 con 0x7f40bc072710 2024-04-24T08:15:08.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.303+0000 7f40b8c00640 1 -- 172.21.15.69:0/25970783 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4007a20 con 0x7f40bc072710 2024-04-24T08:15:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.304+0000 7f40c2000640 1 -- 172.21.15.69:0/25970783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40bc15bba0 con 0x7f40bc072710 2024-04-24T08:15:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.305+0000 7f40c2000640 1 -- 172.21.15.69:0/25970783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40bc075d50 con 0x7f40bc072710 2024-04-24T08:15:08.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.306+0000 7f40b8c00640 1 -- 172.21.15.69:0/25970783 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f40b4047460 con 0x7f40bc072710 2024-04-24T08:15:08.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.307+0000 7f40b8c00640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f409c06c920 0x7f409c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.307+0000 7f40b8c00640 1 -- 172.21.15.69:0/25970783 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f40b40ac140 con 0x7f40bc072710 2024-04-24T08:15:08.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.308+0000 7f40c0c00640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f409c06c920 0x7f409c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.309+0000 7f40c0c00640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f409c06c920 0x7f409c06ede0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f40bc15a0f0 tx=0x7f40b0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:08.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.311+0000 7f40b8c00640 1 -- 172.21.15.69:0/25970783 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40b407a400 con 0x7f40bc072710 2024-04-24T08:15:08.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.537+0000 7f40c2000640 1 -- 172.21.15.69:0/25970783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40bc0adf80 con 0x7f40bc072710 2024-04-24T08:15:08.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.538+0000 7f40b8c00640 1 -- 172.21.15.69:0/25970783 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5188 (secure 0 0 0) 0x7f40b4079d00 con 0x7f40bc072710 2024-04-24T08:15:08.540 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:08.540 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3504199143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3504199143},{"type":"v1","addr":"172.21.15.121:6837","nonce":3504199143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:14:32.246239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6792},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6792":{"gid":6792,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":23,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6792,"qdb_cluster":[6792]},"id":5}]} 2024-04-24T08:15:08.540 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 52 2024-04-24T08:15:08.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.540+0000 7f40c2000640 1 -- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f409c06c920 msgr2=0x7f409c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.540+0000 7f40c2000640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f409c06c920 0x7f409c06ede0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f40bc15a0f0 tx=0x7f40b0008040 comp rx=0 tx=0).stop 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.540+0000 7f40c2000640 1 -- 172.21.15.69:0/25970783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40bc072710 msgr2=0x7f40bc072bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.540+0000 7f40c2000640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40bc072710 0x7f40bc072bc0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f40b402a7f0 tx=0x7f40b4003a70 comp rx=0 tx=0).stop 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.541+0000 7f40c2000640 1 -- 172.21.15.69:0/25970783 shutdown_connections 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.541+0000 7f40c2000640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f409c06c920 0x7f409c06ede0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.541+0000 7f40c2000640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40bc072710 0x7f40bc072bc0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.541+0000 7f40c2000640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40bc0cb630 0x7f40bc0cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.541+0000 7f40c2000640 1 --2- 172.21.15.69:0/25970783 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f40a40051c0 0x7f40bc153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.541+0000 7f40c2000640 1 -- 172.21.15.69:0/25970783 >> 172.21.15.69:0/25970783 conn(0x7f40bc0d0160 msgr2=0x7f40bc1549e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.541+0000 7f40c2000640 1 -- 172.21.15.69:0/25970783 shutdown_connections 2024-04-24T08:15:08.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.541+0000 7f40c2000640 1 -- 172.21.15.69:0/25970783 wait complete. 2024-04-24T08:15:08.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:08.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.750+0000 7f418e600640 1 Processor -- start 2024-04-24T08:15:08.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.750+0000 7f418e600640 1 -- start start 2024-04-24T08:15:08.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.751+0000 7f418e600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418814c650 0x7f4188071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.752+0000 7f418e600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f418806db90 0x7f41880721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.752+0000 7f418e600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4188163fb0 con 0x7f41881666e0 2024-04-24T08:15:08.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.752+0000 7f418e600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f418806e130 con 0x7f418806db90 2024-04-24T08:15:08.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.752+0000 7f418e600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f418806e410 con 0x7f418814c650 2024-04-24T08:15:08.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.752+0000 7f4187e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f41881666e0 0x7f41880b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:53186/0 (socket says 172.21.15.69:53186) 2024-04-24T08:15:08.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.752+0000 7f4187e00640 1 -- 172.21.15.69:0/225096952 learned_addr learned my addr 172.21.15.69:0/225096952 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:08.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.752+0000 7f418dc00640 1 --2- 172.21.15.69:0/225096952 >> v2:172.21.15.121:3300/0 conn(0x7f418806db90 0x7f41880721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.752+0000 7f418d200640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418814c650 0x7f4188071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.752+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2333277421 0 0) 0x7f4188163fb0 con 0x7f41881666e0 2024-04-24T08:15:08.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.752+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4170003520 con 0x7f41881666e0 2024-04-24T08:15:08.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1470992007 0 0) 0x7f4170003520 con 0x7f41881666e0 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418814c650 msgr2=0x7f4188071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418814c650 0x7f4188071c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 >> v2:172.21.15.121:3300/0 conn(0x7f418806db90 msgr2=0x7f41880721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 --2- 172.21.15.69:0/225096952 >> v2:172.21.15.121:3300/0 conn(0x7f418806db90 0x7f41880721d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418806e6f0 con 0x7f41881666e0 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f4174001690 con 0x7f41881666e0 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 >> v1:172.21.15.69:6789/0 conn(0x7f41881666e0 legacy=0x7f41880b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41700039b0 0x7f4170003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418814c650 0x7f4170004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4170004d40 0x7f41700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f418806e130 con 0x7f4170004d40 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f418806e410 con 0x7f41700039b0 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f417000bb50 con 0x7f418814c650 2024-04-24T08:15:08.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f418dc00640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4170004d40 0x7f41700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f418d200640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41700039b0 0x7f4170003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f418dc00640 1 -- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418814c650 msgr2=0x7f4170004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:08.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f418dc00640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f418814c650 0x7f4170004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f418dc00640 1 -- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41700039b0 msgr2=0x7f4170003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:08.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f418dc00640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41700039b0 0x7f4170003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f418dc00640 1 -- 172.21.15.69:0/225096952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4170003520 con 0x7f4170004d40 2024-04-24T08:15:08.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.753+0000 7f418d200640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41700039b0 0x7f4170003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.754+0000 7f418dc00640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4170004d40 0x7f41700051c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f4180007360 tx=0x7f4170005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c497e65290ea1c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.755+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4180009be0 con 0x7f4170004d40 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.755+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f418002e620 con 0x7f4170004d40 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.755+0000 7f4187400640 1 -- 172.21.15.69:0/225096952 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418002e9f0 con 0x7f4170004d40 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.755+0000 7f418e600640 1 -- 172.21.15.69:0/225096952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4170004d40 msgr2=0x7f41700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.755+0000 7f418e600640 1 --2- 172.21.15.69:0/225096952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4170004d40 0x7f41700051c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f4180007360 tx=0x7f4170005830 comp rx=0 tx=0).stop 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.755+0000 7f418dc00640 1 -- 172.21.15.69:0/225096952 reap_dead start 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.755+0000 7f418e600640 1 -- 172.21.15.69:0/225096952 shutdown_connections 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.755+0000 7f418e600640 1 -- 172.21.15.69:0/225096952 >> 172.21.15.69:0/225096952 conn(0x7f41880d0160 msgr2=0x7f41880d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.755+0000 7f418e600640 1 -- 172.21.15.69:0/225096952 shutdown_connections 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.755+0000 7f418e600640 1 -- 172.21.15.69:0/225096952 wait complete. 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.757+0000 7f418e600640 1 Processor -- start 2024-04-24T08:15:08.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.757+0000 7f418e600640 1 -- start start 2024-04-24T08:15:08.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.758+0000 7f418e600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4170004d40 0x7f41881ee740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.758+0000 7f418e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41881fc3f0 0x7f41881fc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.758+0000 7f418e600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f418814acc0 0x7f418814b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.758+0000 7f418e600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f418814ca30 con 0x7f41881fc3f0 2024-04-24T08:15:08.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.758+0000 7f418e600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f418806ecb0 con 0x7f418814acc0 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.758+0000 7f418e600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4188072830 con 0x7f4170004d40 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.758+0000 7f4187e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41881fc3f0 0x7f41881fc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.758+0000 7f4187e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41881fc3f0 0x7f41881fc870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34740/0 (socket says 172.21.15.69:34740) 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.758+0000 7f4187e00640 1 -- 172.21.15.69:0/732217994 learned_addr learned my addr 172.21.15.69:0/732217994 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f418d200640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4170004d40 0x7f41881ee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f418dc00640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f418814acc0 0x7f418814b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f4187e00640 1 -- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4170004d40 msgr2=0x7f41881ee740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f4187e00640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4170004d40 0x7f41881ee740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f4187e00640 1 -- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f418814acc0 msgr2=0x7f418814b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f4187e00640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f418814acc0 0x7f418814b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f4187e00640 1 -- 172.21.15.69:0/732217994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41881eddf0 con 0x7f41881fc3f0 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f4187e00640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41881fc3f0 0x7f41881fc870 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f417400c7a0 tx=0x7f417400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f4184c00640 1 -- 172.21.15.69:0/732217994 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174018070 con 0x7f41881fc3f0 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f4184c00640 1 -- 172.21.15.69:0/732217994 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41740074e0 con 0x7f41881fc3f0 2024-04-24T08:15:08.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f4184c00640 1 -- 172.21.15.69:0/732217994 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174004a00 con 0x7f41881fc3f0 2024-04-24T08:15:08.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.759+0000 7f418e600640 1 -- 172.21.15.69:0/732217994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f418809a5c0 con 0x7f41881fc3f0 2024-04-24T08:15:08.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.760+0000 7f418e600640 1 -- 172.21.15.69:0/732217994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41881ed5f0 con 0x7f41881fc3f0 2024-04-24T08:15:08.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.761+0000 7f418e600640 1 -- 172.21.15.69:0/732217994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4188075d50 con 0x7f41881fc3f0 2024-04-24T08:15:08.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.761+0000 7f4184c00640 1 -- 172.21.15.69:0/732217994 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4174004ba0 con 0x7f41881fc3f0 2024-04-24T08:15:08.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.762+0000 7f4184c00640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f416806c8f0 0x7f416806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:08.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.763+0000 7f4184c00640 1 -- 172.21.15.69:0/732217994 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f417408b1a0 con 0x7f41881fc3f0 2024-04-24T08:15:08.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.763+0000 7f418d200640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f416806c8f0 0x7f416806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:08.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.764+0000 7f418d200640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f416806c8f0 0x7f416806edb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f417c006fd0 tx=0x7f417c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:08.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.767+0000 7f4184c00640 1 -- 172.21.15.69:0/732217994 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4174059460 con 0x7f41881fc3f0 2024-04-24T08:15:08.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.971+0000 7f418e600640 1 -- 172.21.15.69:0/732217994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f41880b0010 con 0x7f41881fc3f0 2024-04-24T08:15:08.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.973+0000 7f4184c00640 1 -- 172.21.15.69:0/732217994 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16969 (secure 0 0 0) 0x7f4174058d60 con 0x7f41881fc3f0 2024-04-24T08:15:08.975 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:08.975 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":71,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:14:03.353108+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.17","mappings":[{"from":6,"to":5}]},{"pgid":"11.22","mappings":[{"from":6,"to":0}]},{"pgid":"11.2e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6837/1135108804":"2024-04-25T08:13:37.186005+0000","172.21.15.121:6835/1135108804":"2024-04-25T08:13:37.186005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:08.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 -- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f416806c8f0 msgr2=0x7f416806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f416806c8f0 0x7f416806edb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f417c006fd0 tx=0x7f417c008040 comp rx=0 tx=0).stop 2024-04-24T08:15:08.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 -- 172.21.15.69:0/732217994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41881fc3f0 msgr2=0x7f41881fc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41881fc3f0 0x7f41881fc870 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f417400c7a0 tx=0x7f417400cc60 comp rx=0 tx=0).stop 2024-04-24T08:15:08.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 -- 172.21.15.69:0/732217994 shutdown_connections 2024-04-24T08:15:08.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f416806c8f0 0x7f416806edb0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41881fc3f0 0x7f41881fc870 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f418814acc0 0x7f418814b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 --2- 172.21.15.69:0/732217994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4170004d40 0x7f41881ee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 -- 172.21.15.69:0/732217994 >> 172.21.15.69:0/732217994 conn(0x7f41880d0160 msgr2=0x7f4188166b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:08.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 -- 172.21.15.69:0/732217994 shutdown_connections 2024-04-24T08:15:08.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:08.976+0000 7f418e600640 1 -- 172.21.15.69:0/732217994 wait complete. 2024-04-24T08:15:08.998 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:15:08.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:09.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.189+0000 7fd2c3200640 1 Processor -- start 2024-04-24T08:15:09.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.189+0000 7fd2c3200640 1 -- start start 2024-04-24T08:15:09.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.191+0000 7fd2c3200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd2bc15d1d0 0x7fd2bc0ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.191+0000 7fd2c3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 0x7fd2bc0ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.191+0000 7fd2c3200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2bc15d660 con 0x7fd2bc156c60 2024-04-24T08:15:09.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.191+0000 7fd2c3200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd2bc06dc30 con 0x7fd2bc15d1d0 2024-04-24T08:15:09.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.191+0000 7fd2c3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2bc06df10 con 0x7fd2bc0b30f0 2024-04-24T08:15:09.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.191+0000 7fd2c2800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 0x7fd2bc0ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.191+0000 7fd2c2800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 0x7fd2bc0ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:45948/0 (socket says 172.21.15.69:45948) 2024-04-24T08:15:09.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.191+0000 7fd2c2800640 1 -- 172.21.15.69:0/1912473798 learned_addr learned my addr 172.21.15.69:0/1912473798 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:09.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.191+0000 7fd2c1400640 1 --2- 172.21.15.69:0/1912473798 >> v2:172.21.15.121:3300/0 conn(0x7fd2bc15d1d0 0x7fd2bc0ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.192+0000 7fd2c2800640 1 -- 172.21.15.69:0/1912473798 >> v2:172.21.15.121:3300/0 conn(0x7fd2bc15d1d0 msgr2=0x7fd2bc0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:09.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.192+0000 7fd2c2800640 1 --2- 172.21.15.69:0/1912473798 >> v2:172.21.15.121:3300/0 conn(0x7fd2bc15d1d0 0x7fd2bc0ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.192+0000 7fd2c2800640 1 -- 172.21.15.69:0/1912473798 >> v1:172.21.15.121:6789/0 conn(0x7fd2bc156c60 legacy=0x7fd2bc15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.192+0000 7fd2c2800640 1 -- 172.21.15.69:0/1912473798 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2bc06e1f0 con 0x7fd2bc0b30f0 2024-04-24T08:15:09.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.193+0000 7fd2c2800640 1 --2- 172.21.15.69:0/1912473798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 0x7fd2bc0ad610 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009e50 tx=0x7fd2b802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2875d2a27cd8b746 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:09.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.194+0000 7fd2c0a00640 1 -- 172.21.15.69:0/1912473798 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8036070 con 0x7fd2bc0b30f0 2024-04-24T08:15:09.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.194+0000 7fd2c0a00640 1 -- 172.21.15.69:0/1912473798 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b8031600 con 0x7fd2bc0b30f0 2024-04-24T08:15:09.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.194+0000 7fd2c0a00640 1 -- 172.21.15.69:0/1912473798 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b80319d0 con 0x7fd2bc0b30f0 2024-04-24T08:15:09.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.195+0000 7fd2c3200640 1 -- 172.21.15.69:0/1912473798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 msgr2=0x7fd2bc0ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.195+0000 7fd2c3200640 1 --2- 172.21.15.69:0/1912473798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 0x7fd2bc0ad610 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009e50 tx=0x7fd2b802a710 comp rx=0 tx=0).stop 2024-04-24T08:15:09.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.195+0000 7fd2c3200640 1 -- 172.21.15.69:0/1912473798 shutdown_connections 2024-04-24T08:15:09.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.195+0000 7fd2c3200640 1 --2- 172.21.15.69:0/1912473798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 0x7fd2bc0ad610 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.195+0000 7fd2c3200640 1 --2- 172.21.15.69:0/1912473798 >> v2:172.21.15.121:3300/0 conn(0x7fd2bc15d1d0 0x7fd2bc0ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.195+0000 7fd2c3200640 1 -- 172.21.15.69:0/1912473798 >> 172.21.15.69:0/1912473798 conn(0x7fd2bc0d0160 msgr2=0x7fd2bc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:09.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.195+0000 7fd2c3200640 1 -- 172.21.15.69:0/1912473798 shutdown_connections 2024-04-24T08:15:09.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.195+0000 7fd2c3200640 1 -- 172.21.15.69:0/1912473798 wait complete. 2024-04-24T08:15:09.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.197+0000 7fd2c3200640 1 Processor -- start 2024-04-24T08:15:09.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.198+0000 7fd2c3200640 1 -- start start 2024-04-24T08:15:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 0x7fd2bc1f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c3200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc156c60 0x7fd2bc155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2bc0725f0 0x7fd2bc155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc06e710 con 0x7fd2bc0725f0 2024-04-24T08:15:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c3200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd2bc15d940 con 0x7fd2bc156c60 2024-04-24T08:15:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd2bc06e890 con 0x7fd2bc0b30f0 2024-04-24T08:15:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2bc0725f0 0x7fd2bc155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2bc0725f0 0x7fd2bc155a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34268/0 (socket says 172.21.15.69:34268) 2024-04-24T08:15:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c2800640 1 -- 172.21.15.69:0/1752643553 learned_addr learned my addr 172.21.15.69:0/1752643553 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c1e00640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 0x7fd2bc1f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.199+0000 7fd2c1400640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc156c60 0x7fd2bc155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.200+0000 7fd2c2800640 1 -- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 msgr2=0x7fd2bc1f3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.200+0000 7fd2c2800640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 0x7fd2bc1f3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.200+0000 7fd2c2800640 1 -- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc156c60 msgr2=0x7fd2bc155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.200+0000 7fd2c2800640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc156c60 0x7fd2bc155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.200+0000 7fd2c2800640 1 -- 172.21.15.69:0/1752643553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2bc072a80 con 0x7fd2bc0725f0 2024-04-24T08:15:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.200+0000 7fd2c2800640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2bc0725f0 0x7fd2bc155a50 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009f70 tx=0x7fd2b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.200+0000 7fd2b6000640 1 -- 172.21.15.69:0/1752643553 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8041020 con 0x7fd2bc0725f0 2024-04-24T08:15:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.200+0000 7fd2b6000640 1 -- 172.21.15.69:0/1752643553 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2b8008730 con 0x7fd2bc0725f0 2024-04-24T08:15:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.200+0000 7fd2c3200640 1 -- 172.21.15.69:0/1752643553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2bc072d60 con 0x7fd2bc0725f0 2024-04-24T08:15:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.200+0000 7fd2b6000640 1 -- 172.21.15.69:0/1752643553 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2b8036040 con 0x7fd2bc0725f0 2024-04-24T08:15:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.201+0000 7fd2c3200640 1 -- 172.21.15.69:0/1752643553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2bc073320 con 0x7fd2bc0725f0 2024-04-24T08:15:09.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.202+0000 7fd2c3200640 1 -- 172.21.15.69:0/1752643553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2bc0b1c10 con 0x7fd2bc0725f0 2024-04-24T08:15:09.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.203+0000 7fd2b6000640 1 -- 172.21.15.69:0/1752643553 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd2b8044020 con 0x7fd2bc0725f0 2024-04-24T08:15:09.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.203+0000 7fd2b6000640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2a006c920 0x7fd2a006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.204+0000 7fd2b6000640 1 -- 172.21.15.69:0/1752643553 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fd2b80ac900 con 0x7fd2bc0725f0 2024-04-24T08:15:09.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.204+0000 7fd2c1e00640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2a006c920 0x7fd2a006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.205+0000 7fd2c1e00640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2a006c920 0x7fd2a006ede0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0001640 tx=0x7fd2b0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:09.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.208+0000 7fd2b6000640 1 -- 172.21.15.69:0/1752643553 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd2bc0b1c10 con 0x7fd2bc0725f0 2024-04-24T08:15:09.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.447+0000 7fd2c3200640 1 -- 172.21.15.69:0/1752643553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2bc06f9e0 con 0x7fd2bc0725f0 2024-04-24T08:15:09.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.448+0000 7fd2b6000640 1 -- 172.21.15.69:0/1752643553 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5188 (secure 0 0 0) 0x7fd2b804b030 con 0x7fd2bc0725f0 2024-04-24T08:15:09.450 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:09.450 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3504199143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3504199143},{"type":"v1","addr":"172.21.15.121:6837","nonce":3504199143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:14:32.246239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6792},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6792":{"gid":6792,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":23,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6792,"qdb_cluster":[6792]},"id":5}]} 2024-04-24T08:15:09.450 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 52 2024-04-24T08:15:09.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 -- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2a006c920 msgr2=0x7fd2a006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2a006c920 0x7fd2a006ede0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd2b0001640 tx=0x7fd2b0009040 comp rx=0 tx=0).stop 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 -- 172.21.15.69:0/1752643553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2bc0725f0 msgr2=0x7fd2bc155a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2bc0725f0 0x7fd2bc155a50 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8009f70 tx=0x7fd2b8004a40 comp rx=0 tx=0).stop 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 -- 172.21.15.69:0/1752643553 shutdown_connections 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd2a006c920 0x7fd2a006ede0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2bc0725f0 0x7fd2bc155a50 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd2bc156c60 0x7fd2bc155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 --2- 172.21.15.69:0/1752643553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd2bc0b30f0 0x7fd2bc1f3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 -- 172.21.15.69:0/1752643553 >> 172.21.15.69:0/1752643553 conn(0x7fd2bc0d0160 msgr2=0x7fd2bc0ad0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 -- 172.21.15.69:0/1752643553 shutdown_connections 2024-04-24T08:15:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.451+0000 7fd2c3200640 1 -- 172.21.15.69:0/1752643553 wait complete. 2024-04-24T08:15:09.472 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:09.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.672+0000 7f0899400640 1 Processor -- start 2024-04-24T08:15:09.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.672+0000 7f0899400640 1 -- start start 2024-04-24T08:15:09.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.673+0000 7f0899400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0894156c60 0x7f089415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.673+0000 7f0899400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089415d660 con 0x7f08940ad7e0 2024-04-24T08:15:09.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.673+0000 7f0899400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089406dc30 con 0x7f08940b30f0 2024-04-24T08:15:09.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.673+0000 7f0899400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f089406df10 con 0x7f0894156c60 2024-04-24T08:15:09.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.674+0000 7f0898a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f08940ad7e0 0x7f089406e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35864/0 (socket says 172.21.15.69:35864) 2024-04-24T08:15:09.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.674+0000 7f0898a00640 1 -- 172.21.15.69:0/2695330651 learned_addr learned my addr 172.21.15.69:0/2695330651 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:09.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.674+0000 7f0893e00640 1 --2- 172.21.15.69:0/2695330651 >> v2:172.21.15.121:3300/0 conn(0x7f0894156c60 0x7f089415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 401382810 0 0) 0x7f089415d660 con 0x7f08940ad7e0 2024-04-24T08:15:09.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0878003520 con 0x7f08940ad7e0 2024-04-24T08:15:09.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 527030006 0 0) 0x7f0878003520 con 0x7f08940ad7e0 2024-04-24T08:15:09.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 >> v2:172.21.15.121:3300/0 conn(0x7f0894156c60 msgr2=0x7f089415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 --2- 172.21.15.69:0/2695330651 >> v2:172.21.15.121:3300/0 conn(0x7f0894156c60 0x7f089415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 >> v1:172.21.15.121:6789/0 conn(0x7f08940b30f0 legacy=0x7f08940ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089406e1f0 con 0x7f08940ad7e0 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f088c001670 con 0x7f08940ad7e0 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 >> v1:172.21.15.69:6789/0 conn(0x7f08940ad7e0 legacy=0x7f089406e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 --2- 172.21.15.69:0/2695330651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0878003a00 0x7f0878003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 --2- 172.21.15.69:0/2695330651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0878004800 0x7f0878004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 --2- 172.21.15.69:0/2695330651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08780051c0 0x7f087800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f089406df10 con 0x7f0878004800 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f087800c090 con 0x7f08780051c0 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f087800c290 con 0x7f0878003a00 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f0893e00640 1 --2- 172.21.15.69:0/2695330651 >> v2:172.21.15.121:3300/0 conn(0x7f0894156c60 0x7f089415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.675+0000 7f088b400640 1 --2- 172.21.15.69:0/2695330651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0878004800 0x7f0878004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.676+0000 7f0898a00640 1 --2- 172.21.15.69:0/2695330651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08780051c0 0x7f087800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.676+0000 7f088b400640 1 -- 172.21.15.69:0/2695330651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0878003a00 msgr2=0x7f0878003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:09.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.676+0000 7f088b400640 1 --2- 172.21.15.69:0/2695330651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0878003a00 0x7f0878003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.676+0000 7f088b400640 1 -- 172.21.15.69:0/2695330651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08780051c0 msgr2=0x7f087800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:09.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.676+0000 7f088b400640 1 --2- 172.21.15.69:0/2695330651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08780051c0 0x7f087800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.676+0000 7f088b400640 1 -- 172.21.15.69:0/2695330651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0878003520 con 0x7f0878004800 2024-04-24T08:15:09.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.676+0000 7f0898a00640 1 -- 172.21.15.69:0/2695330651 reap_dead start 2024-04-24T08:15:09.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.677+0000 7f088b400640 1 --2- 172.21.15.69:0/2695330651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0878004800 0x7f0878004c80 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f087c003d80 tx=0x7f0878005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624376a0ed3618ee server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:09.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.677+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c037070 con 0x7f0878004800 2024-04-24T08:15:09.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.677+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f087c02a9c0 con 0x7f0878004800 2024-04-24T08:15:09.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.677+0000 7f0893400640 1 -- 172.21.15.69:0/2695330651 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c003980 con 0x7f0878004800 2024-04-24T08:15:09.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.678+0000 7f0899400640 1 -- 172.21.15.69:0/2695330651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0878004800 msgr2=0x7f0878004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.678+0000 7f0899400640 1 --2- 172.21.15.69:0/2695330651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0878004800 0x7f0878004c80 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f087c003d80 tx=0x7f0878005830 comp rx=0 tx=0).stop 2024-04-24T08:15:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.678+0000 7f0899400640 1 -- 172.21.15.69:0/2695330651 shutdown_connections 2024-04-24T08:15:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.678+0000 7f0899400640 1 --2- 172.21.15.69:0/2695330651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0878004800 0x7f0878004c80 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.678+0000 7f0899400640 1 -- 172.21.15.69:0/2695330651 >> 172.21.15.69:0/2695330651 conn(0x7f08940d0160 msgr2=0x7f08940d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.678+0000 7f0899400640 1 -- 172.21.15.69:0/2695330651 shutdown_connections 2024-04-24T08:15:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.678+0000 7f0899400640 1 -- 172.21.15.69:0/2695330651 wait complete. 2024-04-24T08:15:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.680+0000 7f0899400640 1 Processor -- start 2024-04-24T08:15:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.680+0000 7f0899400640 1 -- start start 2024-04-24T08:15:09.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f0899400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0878004800 0x7f0894159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f0899400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08941485b0 0x7f0894146e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f0899400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0894148a10 0x7f0894142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f0899400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f08940ad0d0 con 0x7f08941485b0 2024-04-24T08:15:09.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f0899400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f089415d940 con 0x7f0894148a10 2024-04-24T08:15:09.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f0899400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f08940ad250 con 0x7f0878004800 2024-04-24T08:15:09.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f088b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08941485b0 0x7f0894146e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f088b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08941485b0 0x7f0894146e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34296/0 (socket says 172.21.15.69:34296) 2024-04-24T08:15:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f088b400640 1 -- 172.21.15.69:0/2990433573 learned_addr learned my addr 172.21.15.69:0/2990433573 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f0893e00640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0878004800 0x7f0894159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.681+0000 7f0898a00640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0894148a10 0x7f0894142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f088b400640 1 -- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0878004800 msgr2=0x7f0894159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f088b400640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0878004800 0x7f0894159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f088b400640 1 -- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0894148a10 msgr2=0x7f0894142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f088b400640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0894148a10 0x7f0894142e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f088b400640 1 -- 172.21.15.69:0/2990433573 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0894147530 con 0x7f08941485b0 2024-04-24T08:15:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f088b400640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08941485b0 0x7f0894146e30 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f087c035040 tx=0x7f087c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:09.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f0890c00640 1 -- 172.21.15.69:0/2990433573 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c037050 con 0x7f08941485b0 2024-04-24T08:15:09.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f0890c00640 1 -- 172.21.15.69:0/2990433573 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f087c038040 con 0x7f08941485b0 2024-04-24T08:15:09.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f0890c00640 1 -- 172.21.15.69:0/2990433573 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f087c007a70 con 0x7f08941485b0 2024-04-24T08:15:09.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f0899400640 1 -- 172.21.15.69:0/2990433573 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0894147810 con 0x7f08941485b0 2024-04-24T08:15:09.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.682+0000 7f0899400640 1 -- 172.21.15.69:0/2990433573 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0894147dd0 con 0x7f08941485b0 2024-04-24T08:15:09.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.683+0000 7f0899400640 1 -- 172.21.15.69:0/2990433573 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08940b47e0 con 0x7f08941485b0 2024-04-24T08:15:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.684+0000 7f0890c00640 1 -- 172.21.15.69:0/2990433573 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f087c045020 con 0x7f08941485b0 2024-04-24T08:15:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.685+0000 7f0890c00640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f087006c8f0 0x7f087006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.685+0000 7f0890c00640 1 -- 172.21.15.69:0/2990433573 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f087c0ab930 con 0x7f08941485b0 2024-04-24T08:15:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.686+0000 7f0893e00640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f087006c8f0 0x7f087006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:09.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.687+0000 7f0893e00640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f087006c8f0 0x7f087006edb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0894069130 tx=0x7f088400b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.691+0000 7f0890c00640 1 -- 172.21.15.69:0/2990433573 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f08940b47e0 con 0x7f08941485b0 2024-04-24T08:15:09.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.934+0000 7f0899400640 1 -- 172.21.15.69:0/2990433573 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0894149290 con 0x7f08941485b0 2024-04-24T08:15:09.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.935+0000 7f0890c00640 1 -- 172.21.15.69:0/2990433573 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+5188 (secure 0 0 0) 0x7f0894149290 con 0x7f08941485b0 2024-04-24T08:15:09.936 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:09.936 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":52,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3504199143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3504199143},{"type":"v1","addr":"172.21.15.121:6837","nonce":3504199143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:13:15.498700+0000","modified":"2024-04-24T08:14:32.246239+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6792},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6792":{"gid":6792,"name":"a","rank":0,"incarnation":47,"state":"up:active","state_seq":23,"addr":"172.21.15.69:6835/2999903126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2999903126},{"type":"v1","addr":"172.21.15.69:6835","nonce":2999903126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6792,"qdb_cluster":[6792]},"id":5}]} 2024-04-24T08:15:09.936 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 52 2024-04-24T08:15:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.937+0000 7f0899400640 1 -- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f087006c8f0 msgr2=0x7f087006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.937+0000 7f0899400640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f087006c8f0 0x7f087006edb0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f0894069130 tx=0x7f088400b040 comp rx=0 tx=0).stop 2024-04-24T08:15:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.937+0000 7f0899400640 1 -- 172.21.15.69:0/2990433573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08941485b0 msgr2=0x7f0894146e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.937+0000 7f0899400640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08941485b0 0x7f0894146e30 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f087c035040 tx=0x7f087c0036f0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.937+0000 7f0899400640 1 -- 172.21.15.69:0/2990433573 shutdown_connections 2024-04-24T08:15:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.937+0000 7f0899400640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f087006c8f0 0x7f087006edb0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.937+0000 7f0899400640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08941485b0 0x7f0894146e30 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.937+0000 7f0899400640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0894148a10 0x7f0894142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.937+0000 7f0899400640 1 --2- 172.21.15.69:0/2990433573 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0878004800 0x7f0894159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.937+0000 7f0899400640 1 -- 172.21.15.69:0/2990433573 >> 172.21.15.69:0/2990433573 conn(0x7f08940d0160 msgr2=0x7f089415b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.938+0000 7f0899400640 1 -- 172.21.15.69:0/2990433573 shutdown_connections 2024-04-24T08:15:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:09.938+0000 7f0899400640 1 -- 172.21.15.69:0/2990433573 wait complete. 2024-04-24T08:15:09.960 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:10.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.151+0000 7fc74ec00640 1 Processor -- start 2024-04-24T08:15:10.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.151+0000 7fc74ec00640 1 -- start start 2024-04-24T08:15:10.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc74ec00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc7481631a0 0x7fc7480af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc74ec00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc7480af7d0 0x7fc74806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc74ec00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7480adab0 con 0x7fc74815ea10 2024-04-24T08:15:10.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc74ec00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc7480afbb0 con 0x7fc7480af7d0 2024-04-24T08:15:10.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc74ec00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc7480add50 con 0x7fc7481631a0 2024-04-24T08:15:10.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc747e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc74815ea10 0x7fc7480d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35870/0 (socket says 172.21.15.69:35870) 2024-04-24T08:15:10.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc747e00640 1 -- 172.21.15.69:0/1621863885 learned_addr learned my addr 172.21.15.69:0/1621863885 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:10.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc74cc00640 1 --2- 172.21.15.69:0/1621863885 >> v2:172.21.15.69:3300/0 conn(0x7fc7480af7d0 0x7fc74806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc73f400640 1 --2- 172.21.15.69:0/1621863885 >> v2:172.21.15.121:3300/0 conn(0x7fc7481631a0 0x7fc7480af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc747400640 1 -- 172.21.15.69:0/1621863885 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3839970884 0 0) 0x7fc7480adab0 con 0x7fc74815ea10 2024-04-24T08:15:10.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc74cc00640 1 -- 172.21.15.69:0/1621863885 >> v2:172.21.15.121:3300/0 conn(0x7fc7481631a0 msgr2=0x7fc7480af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:10.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc74cc00640 1 --2- 172.21.15.69:0/1621863885 >> v2:172.21.15.121:3300/0 conn(0x7fc7481631a0 0x7fc7480af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.153+0000 7fc74cc00640 1 -- 172.21.15.69:0/1621863885 >> v1:172.21.15.69:6789/0 conn(0x7fc74815ea10 legacy=0x7fc7480d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.154+0000 7fc74cc00640 1 -- 172.21.15.69:0/1621863885 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7480ae030 con 0x7fc7480af7d0 2024-04-24T08:15:10.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.154+0000 7fc73f400640 1 --2- 172.21.15.69:0/1621863885 >> v2:172.21.15.121:3300/0 conn(0x7fc7481631a0 0x7fc7480af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:10.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.155+0000 7fc74cc00640 1 --2- 172.21.15.69:0/1621863885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 0x7fc74806dff0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc74806f5d0 tx=0x7fc74002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6225865592984222 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:10.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.155+0000 7fc747400640 1 -- 172.21.15.69:0/1621863885 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740007740 con 0x7fc7480af7d0 2024-04-24T08:15:10.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.155+0000 7fc747400640 1 -- 172.21.15.69:0/1621863885 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7400078e0 con 0x7fc7480af7d0 2024-04-24T08:15:10.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.155+0000 7fc747400640 1 -- 172.21.15.69:0/1621863885 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc740007cb0 con 0x7fc7480af7d0 2024-04-24T08:15:10.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.155+0000 7fc74ec00640 1 -- 172.21.15.69:0/1621863885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 msgr2=0x7fc74806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.155+0000 7fc74ec00640 1 --2- 172.21.15.69:0/1621863885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 0x7fc74806dff0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7fc74806f5d0 tx=0x7fc74002a680 comp rx=0 tx=0).stop 2024-04-24T08:15:10.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.156+0000 7fc74ec00640 1 -- 172.21.15.69:0/1621863885 shutdown_connections 2024-04-24T08:15:10.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.156+0000 7fc74ec00640 1 --2- 172.21.15.69:0/1621863885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 0x7fc74806dff0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.156+0000 7fc74ec00640 1 --2- 172.21.15.69:0/1621863885 >> v2:172.21.15.121:3300/0 conn(0x7fc7481631a0 0x7fc7480af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.156+0000 7fc74ec00640 1 -- 172.21.15.69:0/1621863885 >> 172.21.15.69:0/1621863885 conn(0x7fc748067d90 msgr2=0x7fc74806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:10.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.156+0000 7fc74ec00640 1 -- 172.21.15.69:0/1621863885 shutdown_connections 2024-04-24T08:15:10.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.156+0000 7fc74ec00640 1 -- 172.21.15.69:0/1621863885 wait complete. 2024-04-24T08:15:10.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.158+0000 7fc74ec00640 1 Processor -- start 2024-04-24T08:15:10.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.158+0000 7fc74ec00640 1 -- start start 2024-04-24T08:15:10.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.159+0000 7fc74ec00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 0x7fc74814ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.159+0000 7fc74ec00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc74815ea10 0x7fc74815f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.159+0000 7fc74ec00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7481631a0 0x7fc7481613d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.159+0000 7fc74ec00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc7480ae8d0 con 0x7fc7480af7d0 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.159+0000 7fc74ec00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc7480ae5f0 con 0x7fc7481631a0 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.159+0000 7fc74ec00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc7480aedf0 con 0x7fc74815ea10 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.159+0000 7fc747e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 0x7fc74814ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.159+0000 7fc747e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 0x7fc74814ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34318/0 (socket says 172.21.15.69:34318) 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.159+0000 7fc747e00640 1 -- 172.21.15.69:0/1725371706 learned_addr learned my addr 172.21.15.69:0/1725371706 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.159+0000 7fc73f400640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc74815ea10 0x7fc74815f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc747e00640 1 -- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc74815ea10 msgr2=0x7fc74815f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc74cc00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7481631a0 0x7fc7481613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc747e00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc74815ea10 0x7fc74815f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc747e00640 1 -- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7481631a0 msgr2=0x7fc7481613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc747e00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7481631a0 0x7fc7481613d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc747e00640 1 -- 172.21.15.69:0/1725371706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc748160530 con 0x7fc7480af7d0 2024-04-24T08:15:10.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc747e00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 0x7fc74814ca10 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc73800c4a0 tx=0x7fc73800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:10.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc744c00640 1 -- 172.21.15.69:0/1725371706 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738017070 con 0x7fc7480af7d0 2024-04-24T08:15:10.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc744c00640 1 -- 172.21.15.69:0/1725371706 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc738005ce0 con 0x7fc7480af7d0 2024-04-24T08:15:10.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc744c00640 1 -- 172.21.15.69:0/1725371706 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7380072b0 con 0x7fc7480af7d0 2024-04-24T08:15:10.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc74ec00640 1 -- 172.21.15.69:0/1725371706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc748160810 con 0x7fc7480af7d0 2024-04-24T08:15:10.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.160+0000 7fc74ec00640 1 -- 172.21.15.69:0/1725371706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc748160dd0 con 0x7fc7480af7d0 2024-04-24T08:15:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.161+0000 7fc74ec00640 1 -- 172.21.15.69:0/1725371706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc748068a80 con 0x7fc7480af7d0 2024-04-24T08:15:10.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.162+0000 7fc744c00640 1 -- 172.21.15.69:0/1725371706 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc73801a070 con 0x7fc7480af7d0 2024-04-24T08:15:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.163+0000 7fc744c00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc72406c8f0 0x7fc72406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.163+0000 7fc744c00640 1 -- 172.21.15.69:0/1725371706 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7fc73808aff0 con 0x7fc7480af7d0 2024-04-24T08:15:10.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.164+0000 7fc73f400640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc72406c8f0 0x7fc72406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.165+0000 7fc73f400640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc72406c8f0 0x7fc72406edb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc7480cf3d0 tx=0x7fc730008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:10.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.168+0000 7fc744c00640 1 -- 172.21.15.69:0/1725371706 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc738059340 con 0x7fc7480af7d0 2024-04-24T08:15:10.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.370+0000 7fc74ec00640 1 -- 172.21.15.69:0/1725371706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc74806ef50 con 0x7fc7480af7d0 2024-04-24T08:15:10.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.373+0000 7fc744c00640 1 -- 172.21.15.69:0/1725371706 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+16969 (secure 0 0 0) 0x7fc738058c40 con 0x7fc7480af7d0 2024-04-24T08:15:10.375 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:10.376 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":71,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:14:03.353108+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:13:13.250209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-04-24T08:13:14.468238+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"11.10","mappings":[{"from":6,"to":1}]},{"pgid":"11.17","mappings":[{"from":6,"to":5}]},{"pgid":"11.22","mappings":[{"from":6,"to":0}]},{"pgid":"11.2e","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6837/1135108804":"2024-04-25T08:13:37.186005+0000","172.21.15.121:6835/1135108804":"2024-04-25T08:13:37.186005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:10.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 -- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc72406c8f0 msgr2=0x7fc72406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc72406c8f0 0x7fc72406edb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc7480cf3d0 tx=0x7fc730008040 comp rx=0 tx=0).stop 2024-04-24T08:15:10.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 -- 172.21.15.69:0/1725371706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 msgr2=0x7fc74814ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 0x7fc74814ca10 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fc73800c4a0 tx=0x7fc73800c960 comp rx=0 tx=0).stop 2024-04-24T08:15:10.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 -- 172.21.15.69:0/1725371706 shutdown_connections 2024-04-24T08:15:10.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc72406c8f0 0x7fc72406edb0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7481631a0 0x7fc7481613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc74815ea10 0x7fc74815f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 --2- 172.21.15.69:0/1725371706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7480af7d0 0x7fc74814ca10 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 -- 172.21.15.69:0/1725371706 >> 172.21.15.69:0/1725371706 conn(0x7fc748067d90 msgr2=0x7fc74806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:10.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 -- 172.21.15.69:0/1725371706 shutdown_connections 2024-04-24T08:15:10.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.376+0000 7fc74ec00640 1 -- 172.21.15.69:0/1725371706 wait complete. 2024-04-24T08:15:10.397 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:15:10.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.596+0000 7f3ce7200640 1 Processor -- start 2024-04-24T08:15:10.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.596+0000 7f3ce7200640 1 -- start start 2024-04-24T08:15:10.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.598+0000 7f3ce7200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3ce015d1d0 0x7f3ce00ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.598+0000 7f3ce7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00b30f0 0x7f3ce00ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.598+0000 7f3ce7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce015d660 con 0x7f3ce0156c60 2024-04-24T08:15:10.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.598+0000 7f3ce7200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce006dc30 con 0x7f3ce015d1d0 2024-04-24T08:15:10.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.598+0000 7f3ce7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce006df10 con 0x7f3ce00b30f0 2024-04-24T08:15:10.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.598+0000 7f3ce5e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3ce0156c60 0x7f3ce015cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35874/0 (socket says 172.21.15.69:35874) 2024-04-24T08:15:10.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.598+0000 7f3ce5e00640 1 -- 172.21.15.69:0/2211130598 learned_addr learned my addr 172.21.15.69:0/2211130598 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:10.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.598+0000 7f3ce6800640 1 --2- 172.21.15.69:0/2211130598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00b30f0 0x7f3ce00ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.598+0000 7f3ce5400640 1 --2- 172.21.15.69:0/2211130598 >> v2:172.21.15.121:3300/0 conn(0x7f3ce015d1d0 0x7f3ce00ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3989765301 0 0) 0x7f3ce015d660 con 0x7f3ce0156c60 2024-04-24T08:15:10.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc8003520 con 0x7f3ce0156c60 2024-04-24T08:15:10.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1546376589 0 0) 0x7f3cc8003520 con 0x7f3ce0156c60 2024-04-24T08:15:10.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00b30f0 msgr2=0x7f3ce00ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:10.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 --2- 172.21.15.69:0/2211130598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00b30f0 0x7f3ce00ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 >> v2:172.21.15.121:3300/0 conn(0x7f3ce015d1d0 msgr2=0x7f3ce00ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:10.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 --2- 172.21.15.69:0/2211130598 >> v2:172.21.15.121:3300/0 conn(0x7f3ce015d1d0 0x7f3ce00ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce006e1f0 con 0x7f3ce0156c60 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f3cd4001670 con 0x7f3ce0156c60 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 >> v1:172.21.15.69:6789/0 conn(0x7f3ce0156c60 legacy=0x7f3ce015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 --2- 172.21.15.69:0/2211130598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 0x7f3cc8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 --2- 172.21.15.69:0/2211130598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cc8004800 0x7f3cc8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 --2- 172.21.15.69:0/2211130598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00b30f0 0x7f3cc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce006dc30 con 0x7f3cc80039b0 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce006df10 con 0x7f3cc8004800 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3cc800b9c0 con 0x7f3ce00b30f0 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.599+0000 7f3ce5e00640 1 --2- 172.21.15.69:0/2211130598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 0x7f3cc8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.600+0000 7f3ce5e00640 1 -- 172.21.15.69:0/2211130598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00b30f0 msgr2=0x7f3cc80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.600+0000 7f3ce5e00640 1 --2- 172.21.15.69:0/2211130598 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce00b30f0 0x7f3cc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.600+0000 7f3ce5e00640 1 -- 172.21.15.69:0/2211130598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cc8004800 msgr2=0x7f3cc8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.600+0000 7f3ce5e00640 1 --2- 172.21.15.69:0/2211130598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cc8004800 0x7f3cc8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.600+0000 7f3ce5e00640 1 -- 172.21.15.69:0/2211130598 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc8003520 con 0x7f3cc80039b0 2024-04-24T08:15:10.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.601+0000 7f3ce5e00640 1 --2- 172.21.15.69:0/2211130598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 0x7f3cc8003dd0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c210 tx=0x7f3cc8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a22753bd006395 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:10.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.601+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4007d50 con 0x7f3cc80039b0 2024-04-24T08:15:10.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.601+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd4005af0 con 0x7f3cc80039b0 2024-04-24T08:15:10.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.601+0000 7f3ce4a00640 1 -- 172.21.15.69:0/2211130598 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4007070 con 0x7f3cc80039b0 2024-04-24T08:15:10.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.602+0000 7f3ce7200640 1 -- 172.21.15.69:0/2211130598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 msgr2=0x7f3cc8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.602+0000 7f3ce7200640 1 --2- 172.21.15.69:0/2211130598 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 0x7f3cc8003dd0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c210 tx=0x7f3cc8005830 comp rx=0 tx=0).stop 2024-04-24T08:15:10.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.602+0000 7f3ce6800640 1 -- 172.21.15.69:0/2211130598 reap_dead start 2024-04-24T08:15:10.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.602+0000 7f3ce7200640 1 -- 172.21.15.69:0/2211130598 shutdown_connections 2024-04-24T08:15:10.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.602+0000 7f3ce7200640 1 -- 172.21.15.69:0/2211130598 >> 172.21.15.69:0/2211130598 conn(0x7f3ce00d0160 msgr2=0x7f3ce00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:10.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.602+0000 7f3ce7200640 1 -- 172.21.15.69:0/2211130598 shutdown_connections 2024-04-24T08:15:10.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.602+0000 7f3ce7200640 1 -- 172.21.15.69:0/2211130598 wait complete. 2024-04-24T08:15:10.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.603+0000 7f3ce7200640 1 Processor -- start 2024-04-24T08:15:10.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.604+0000 7f3ce7200640 1 -- start start 2024-04-24T08:15:10.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 0x7f3ce0155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce01454c0 0x7f3ce00b1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce7200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ce0145920 0x7f3ce00b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00ad610 con 0x7f3cc80039b0 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce7200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce015d940 con 0x7f3ce0145920 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce00ad790 con 0x7f3ce01454c0 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce5e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 0x7f3ce0155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce5e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 0x7f3ce0155d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34334/0 (socket says 172.21.15.69:34334) 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce5e00640 1 -- 172.21.15.69:0/727163698 learned_addr learned my addr 172.21.15.69:0/727163698 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce5e00640 1 -- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce01454c0 msgr2=0x7f3ce00b1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce5e00640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce01454c0 0x7f3ce00b1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce5e00640 1 -- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ce0145920 msgr2=0x7f3ce00b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce5e00640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ce0145920 0x7f3ce00b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.605+0000 7f3ce5e00640 1 -- 172.21.15.69:0/727163698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00729d0 con 0x7f3cc80039b0 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.606+0000 7f3ce5e00640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 0x7f3ce0155d90 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c210 tx=0x7f3cd4001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.606+0000 7f3cda000640 1 -- 172.21.15.69:0/727163698 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4042070 con 0x7f3cc80039b0 2024-04-24T08:15:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.606+0000 7f3cda000640 1 -- 172.21.15.69:0/727163698 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd4007480 con 0x7f3cc80039b0 2024-04-24T08:15:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.606+0000 7f3cda000640 1 -- 172.21.15.69:0/727163698 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd403e450 con 0x7f3cc80039b0 2024-04-24T08:15:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.606+0000 7f3ce7200640 1 -- 172.21.15.69:0/727163698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce0072cb0 con 0x7f3cc80039b0 2024-04-24T08:15:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.606+0000 7f3ce7200640 1 -- 172.21.15.69:0/727163698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce0073270 con 0x7f3cc80039b0 2024-04-24T08:15:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.607+0000 7f3ce7200640 1 -- 172.21.15.69:0/727163698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce0075d50 con 0x7f3cc80039b0 2024-04-24T08:15:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.608+0000 7f3cda000640 1 -- 172.21.15.69:0/727163698 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3cd4045070 con 0x7f3cc80039b0 2024-04-24T08:15:10.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.609+0000 7f3cda000640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cc006c8f0 0x7f3cc006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:10.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.609+0000 7f3cda000640 1 -- 172.21.15.69:0/727163698 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 5911+0+0 (secure 0 0 0) 0x7f3cd40ac870 con 0x7f3cc80039b0 2024-04-24T08:15:10.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.610+0000 7f3ce5400640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cc006c8f0 0x7f3cc006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:10.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.611+0000 7f3ce5400640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cc006c8f0 0x7f3cc006edb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc006e80 tx=0x7f3ccc006dd0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:10.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.614+0000 7f3cda000640 1 -- 172.21.15.69:0/727163698 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3cd407a4e0 con 0x7f3cc80039b0 2024-04-24T08:15:10.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.857+0000 7f3ce7200640 1 -- 172.21.15.69:0/727163698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3ce006f9e0 con 0x7f3cc80039b0 2024-04-24T08:15:10.859 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:15:10.857+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:15:10.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.861+0000 7f3cda000640 1 -- 172.21.15.69:0/727163698 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v53) ==== 161+0+0 (secure 0 0 0) 0x7f3cd407e7e0 con 0x7f3cc80039b0 2024-04-24T08:15:10.863 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:15:10.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.865+0000 7f3ce7200640 1 -- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cc006c8f0 msgr2=0x7f3cc006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.865+0000 7f3ce7200640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cc006c8f0 0x7f3cc006edb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc006e80 tx=0x7f3ccc006dd0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.865+0000 7f3ce7200640 1 -- 172.21.15.69:0/727163698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 msgr2=0x7f3ce0155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:10.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.865+0000 7f3ce7200640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 0x7f3ce0155d90 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400c210 tx=0x7f3cd4001a40 comp rx=0 tx=0).stop 2024-04-24T08:15:10.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.865+0000 7f3ce7200640 1 -- 172.21.15.69:0/727163698 shutdown_connections 2024-04-24T08:15:10.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.865+0000 7f3ce7200640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cc006c8f0 0x7f3cc006edb0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.865+0000 7f3ce7200640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3ce0145920 0x7f3ce00b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.865+0000 7f3ce7200640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3ce01454c0 0x7f3ce00b1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.865+0000 7f3ce7200640 1 --2- 172.21.15.69:0/727163698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cc80039b0 0x7f3ce0155d90 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:10.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.865+0000 7f3ce7200640 1 -- 172.21.15.69:0/727163698 >> 172.21.15.69:0/727163698 conn(0x7f3ce00d0160 msgr2=0x7f3ce015a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:10.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.866+0000 7f3ce7200640 1 -- 172.21.15.69:0/727163698 shutdown_connections 2024-04-24T08:15:10.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:10.866+0000 7f3ce7200640 1 -- 172.21.15.69:0/727163698 wait complete. 2024-04-24T08:15:10.882 INFO:tasks.ceph.mds.a.smithi069.stderr: -4802> 2024-04-24T08:13:42.014+0000 7f980ba00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:15:10.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T08:15:10.939 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:15:11.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.072+0000 7f48a6a00640 1 Processor -- start 2024-04-24T08:15:11.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.072+0000 7f48a6a00640 1 -- start start 2024-04-24T08:15:11.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.074+0000 7f48a6a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f48a0155f20 0x7f48a00ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:11.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.074+0000 7f48a6a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48a015c9b0 0x7f48a00b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:11.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.074+0000 7f48a6a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a00cd6d0 con 0x7f48a00b3f70 2024-04-24T08:15:11.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.074+0000 7f48a6a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f48a006db90 con 0x7f48a015c9b0 2024-04-24T08:15:11.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.074+0000 7f48a6a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f48a0072070 con 0x7f48a0155f20 2024-04-24T08:15:11.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.074+0000 7f4897400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48a015c9b0 0x7f48a00b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:11.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.074+0000 7f4897400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f48a015c9b0 0x7f48a00b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34362/0 (socket says 172.21.15.69:34362) 2024-04-24T08:15:11.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.074+0000 7f4897400640 1 -- 172.21.15.69:0/2082243238 learned_addr learned my addr 172.21.15.69:0/2082243238 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:11.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.075+0000 7f489fe00640 1 --2- 172.21.15.69:0/2082243238 >> v2:172.21.15.121:3300/0 conn(0x7f48a0155f20 0x7f48a00ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:11.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.075+0000 7f4897400640 1 -- 172.21.15.69:0/2082243238 >> v2:172.21.15.121:3300/0 conn(0x7f48a0155f20 msgr2=0x7f48a00ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:11.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.075+0000 7f4897400640 1 --2- 172.21.15.69:0/2082243238 >> v2:172.21.15.121:3300/0 conn(0x7f48a0155f20 0x7f48a00ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:11.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.075+0000 7f4897400640 1 -- 172.21.15.69:0/2082243238 >> v1:172.21.15.121:6789/0 conn(0x7f48a00b3f70 legacy=0x7f48a00cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:11.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.075+0000 7f4897400640 1 -- 172.21.15.69:0/2082243238 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a00cd9b0 con 0x7f48a015c9b0 2024-04-24T08:15:11.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.076+0000 7f4897400640 1 --2- 172.21.15.69:0/2082243238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a015c9b0 0x7f48a00b3a30 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f48a0159870 tx=0x7f488802c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebfe89a77809cf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:11.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.076+0000 7f489f400640 1 -- 172.21.15.69:0/2082243238 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888036020 con 0x7f48a015c9b0 2024-04-24T08:15:11.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.076+0000 7f489f400640 1 -- 172.21.15.69:0/2082243238 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4888032830 con 0x7f48a015c9b0 2024-04-24T08:15:11.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.076+0000 7f489f400640 1 -- 172.21.15.69:0/2082243238 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888032c00 con 0x7f48a015c9b0 2024-04-24T08:15:11.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.077+0000 7f48a6a00640 1 -- 172.21.15.69:0/2082243238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a015c9b0 msgr2=0x7f48a00b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:11.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.077+0000 7f48a6a00640 1 --2- 172.21.15.69:0/2082243238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a015c9b0 0x7f48a00b3a30 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f48a0159870 tx=0x7f488802c970 comp rx=0 tx=0).stop 2024-04-24T08:15:11.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.077+0000 7f48a6a00640 1 -- 172.21.15.69:0/2082243238 shutdown_connections 2024-04-24T08:15:11.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.077+0000 7f48a6a00640 1 --2- 172.21.15.69:0/2082243238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a015c9b0 0x7f48a00b3a30 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:11.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.077+0000 7f48a6a00640 1 --2- 172.21.15.69:0/2082243238 >> v2:172.21.15.121:3300/0 conn(0x7f48a0155f20 0x7f48a00ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:11.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.077+0000 7f48a6a00640 1 -- 172.21.15.69:0/2082243238 >> 172.21.15.69:0/2082243238 conn(0x7f48a00d0350 msgr2=0x7f48a00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:11.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.077+0000 7f48a6a00640 1 -- 172.21.15.69:0/2082243238 shutdown_connections 2024-04-24T08:15:11.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.077+0000 7f48a6a00640 1 -- 172.21.15.69:0/2082243238 wait complete. 2024-04-24T08:15:11.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.079+0000 7f48a6a00640 1 Processor -- start 2024-04-24T08:15:11.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.079+0000 7f48a6a00640 1 -- start start 2024-04-24T08:15:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.080+0000 7f48a6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a00b3f70 0x7f48a014aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.080+0000 7f48a6a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a0155f20 0x7f48a015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.080+0000 7f48a6a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48a015c9b0 0x7f48a01e5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.080+0000 7f48a6a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f48a00cd2a0 con 0x7f48a00b3f70 2024-04-24T08:15:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.080+0000 7f48a6a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f48a0071d00 con 0x7f48a0155f20 2024-04-24T08:15:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.080+0000 7f48a6a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f48a00cd420 con 0x7f48a015c9b0 2024-04-24T08:15:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.080+0000 7f489fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a00b3f70 0x7f48a014aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:11.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f489fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a00b3f70 0x7f48a014aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34378/0 (socket says 172.21.15.69:34378) 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f489fe00640 1 -- 172.21.15.69:0/996208019 learned_addr learned my addr 172.21.15.69:0/996208019 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f4897400640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a0155f20 0x7f48a015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f48a4a00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48a015c9b0 0x7f48a01e5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f489fe00640 1 -- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48a015c9b0 msgr2=0x7f48a01e5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f489fe00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48a015c9b0 0x7f48a01e5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f489fe00640 1 -- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a0155f20 msgr2=0x7f48a015cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f489fe00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a0155f20 0x7f48a015cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f489fe00640 1 -- 172.21.15.69:0/996208019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a0152b30 con 0x7f48a00b3f70 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f4897400640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a0155f20 0x7f48a015cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.081+0000 7f489fe00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a00b3f70 0x7f48a014aab0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f489000d800 tx=0x7f489000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.082+0000 7f489cc00640 1 -- 172.21.15.69:0/996208019 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489000b6c0 con 0x7f48a00b3f70 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.082+0000 7f489cc00640 1 -- 172.21.15.69:0/996208019 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4890019070 con 0x7f48a00b3f70 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.082+0000 7f48a6a00640 1 -- 172.21.15.69:0/996208019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a0152e10 con 0x7f48a00b3f70 2024-04-24T08:15:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.082+0000 7f489cc00640 1 -- 172.21.15.69:0/996208019 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f489000ba80 con 0x7f48a00b3f70 2024-04-24T08:15:11.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.082+0000 7f48a6a00640 1 -- 172.21.15.69:0/996208019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a01527e0 con 0x7f48a00b3f70 2024-04-24T08:15:11.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.083+0000 7f48a6a00640 1 -- 172.21.15.69:0/996208019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a0068a80 con 0x7f48a00b3f70 2024-04-24T08:15:11.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.084+0000 7f489cc00640 1 -- 172.21.15.69:0/996208019 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f489000bc20 con 0x7f48a00b3f70 2024-04-24T08:15:11.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.085+0000 7f489cc00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f487c06c8f0 0x7f487c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:11.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.085+0000 7f489cc00640 1 -- 172.21.15.69:0/996208019 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5997+0+0 (secure 0 0 0) 0x7f489008c150 con 0x7f48a00b3f70 2024-04-24T08:15:11.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.085+0000 7f4897400640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f487c06c8f0 0x7f487c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.086+0000 7f4897400640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f487c06c8f0 0x7f487c06edb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4888037000 tx=0x7f4888008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:11.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.089+0000 7f489cc00640 1 -- 172.21.15.69:0/996208019 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f489005a450 con 0x7f48a00b3f70 2024-04-24T08:15:11.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.318+0000 7f48a6a00640 1 -- 172.21.15.69:0/996208019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f48a0155380 con 0x7f48a00b3f70 2024-04-24T08:15:11.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.864+0000 7f489cc00640 1 -- 172.21.15.69:0/996208019 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v54) ==== 104+0+0 (secure 0 0 0) 0x7f4890059d50 con 0x7f48a00b3f70 2024-04-24T08:15:11.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.868+0000 7f48a6a00640 1 -- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f487c06c8f0 msgr2=0x7f487c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:11.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.868+0000 7f48a6a00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f487c06c8f0 0x7f487c06edb0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f4888037000 tx=0x7f4888008040 comp rx=0 tx=0).stop 2024-04-24T08:15:11.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.868+0000 7f48a6a00640 1 -- 172.21.15.69:0/996208019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a00b3f70 msgr2=0x7f48a014aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:11.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.868+0000 7f48a6a00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a00b3f70 0x7f48a014aab0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f489000d800 tx=0x7f489000dcc0 comp rx=0 tx=0).stop 2024-04-24T08:15:11.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.868+0000 7f48a6a00640 1 -- 172.21.15.69:0/996208019 shutdown_connections 2024-04-24T08:15:11.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.868+0000 7f48a6a00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f487c06c8f0 0x7f487c06edb0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:11.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.868+0000 7f48a6a00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48a015c9b0 0x7f48a01e5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:11.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.868+0000 7f48a6a00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a0155f20 0x7f48a015cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:11.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.868+0000 7f48a6a00640 1 --2- 172.21.15.69:0/996208019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f48a00b3f70 0x7f48a014aab0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:11.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.868+0000 7f48a6a00640 1 -- 172.21.15.69:0/996208019 >> 172.21.15.69:0/996208019 conn(0x7f48a00d0350 msgr2=0x7f48a006bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:11.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.869+0000 7f48a6a00640 1 -- 172.21.15.69:0/996208019 shutdown_connections 2024-04-24T08:15:11.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:11.869+0000 7f48a6a00640 1 -- 172.21.15.69:0/996208019 wait complete. 2024-04-24T08:15:11.894 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T08:15:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.140+0000 7f62dc400640 1 Processor -- start 2024-04-24T08:15:12.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.140+0000 7f62dc400640 1 -- start start 2024-04-24T08:15:12.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.142+0000 7f62dc400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d415c690 0x7f62d40d3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:12.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.142+0000 7f62dc400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f62d41625b0 0x7f62d4162a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:12.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.142+0000 7f62dc400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62d415f0d0 con 0x7f62d40d1070 2024-04-24T08:15:12.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.142+0000 7f62dc400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f62d415fa00 con 0x7f62d41625b0 2024-04-24T08:15:12.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.142+0000 7f62dc400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62d415f3b0 con 0x7f62d415c690 2024-04-24T08:15:12.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.142+0000 7f62d9000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f62d40d1070 0x7f62d40d14f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35894/0 (socket says 172.21.15.69:35894) 2024-04-24T08:15:12.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.142+0000 7f62d9000640 1 -- 172.21.15.69:0/1531659476 learned_addr learned my addr 172.21.15.69:0/1531659476 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:12.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.142+0000 7f62da400640 1 --2- 172.21.15.69:0/1531659476 >> v2:172.21.15.69:3300/0 conn(0x7f62d41625b0 0x7f62d4162a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:12.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.143+0000 7f62d9a00640 1 --2- 172.21.15.69:0/1531659476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d415c690 0x7f62d40d3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:12.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.143+0000 7f62d3e00640 1 -- 172.21.15.69:0/1531659476 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 511944186 0 0) 0x7f62d415f0d0 con 0x7f62d40d1070 2024-04-24T08:15:12.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.143+0000 7f62d3e00640 1 -- 172.21.15.69:0/1531659476 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f62bc003520 con 0x7f62d40d1070 2024-04-24T08:15:12.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.143+0000 7f62da400640 1 -- 172.21.15.69:0/1531659476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d415c690 msgr2=0x7f62d40d3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:12.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.143+0000 7f62da400640 1 --2- 172.21.15.69:0/1531659476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d415c690 0x7f62d40d3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:12.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.143+0000 7f62da400640 1 -- 172.21.15.69:0/1531659476 >> v1:172.21.15.69:6789/0 conn(0x7f62d40d1070 legacy=0x7f62d40d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:12.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.143+0000 7f62da400640 1 -- 172.21.15.69:0/1531659476 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d415f690 con 0x7f62d41625b0 2024-04-24T08:15:12.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.144+0000 7f62da400640 1 --2- 172.21.15.69:0/1531659476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d41625b0 0x7f62d4162a40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009d90 tx=0x7f62cc02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba6f5bbae02fcd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:12.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.144+0000 7f62d3e00640 1 -- 172.21.15.69:0/1531659476 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc007740 con 0x7f62d41625b0 2024-04-24T08:15:12.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.144+0000 7f62d3e00640 1 -- 172.21.15.69:0/1531659476 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62cc0078e0 con 0x7f62d41625b0 2024-04-24T08:15:12.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.144+0000 7f62d3e00640 1 -- 172.21.15.69:0/1531659476 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62cc007cb0 con 0x7f62d41625b0 2024-04-24T08:15:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.145+0000 7f62dc400640 1 -- 172.21.15.69:0/1531659476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d41625b0 msgr2=0x7f62d4162a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.145+0000 7f62dc400640 1 --2- 172.21.15.69:0/1531659476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d41625b0 0x7f62d4162a40 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f62cc009d90 tx=0x7f62cc02a660 comp rx=0 tx=0).stop 2024-04-24T08:15:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.145+0000 7f62dc400640 1 -- 172.21.15.69:0/1531659476 shutdown_connections 2024-04-24T08:15:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.145+0000 7f62dc400640 1 --2- 172.21.15.69:0/1531659476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d41625b0 0x7f62d4162a40 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.145+0000 7f62dc400640 1 --2- 172.21.15.69:0/1531659476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d415c690 0x7f62d40d3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.145+0000 7f62dc400640 1 -- 172.21.15.69:0/1531659476 >> 172.21.15.69:0/1531659476 conn(0x7f62d4067760 msgr2=0x7f62d4067b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.145+0000 7f62dc400640 1 -- 172.21.15.69:0/1531659476 shutdown_connections 2024-04-24T08:15:12.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.145+0000 7f62dc400640 1 -- 172.21.15.69:0/1531659476 wait complete. 2024-04-24T08:15:12.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.147+0000 7f62dc400640 1 Processor -- start 2024-04-24T08:15:12.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.147+0000 7f62dc400640 1 -- start start 2024-04-24T08:15:12.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.147+0000 7f62dc400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d40d1070 0x7f62d40cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:12.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.147+0000 7f62dc400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d415c690 0x7f62d40c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:12.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.147+0000 7f62dc400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d41625b0 0x7f62d4158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62dc400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62d40b6da0 con 0x7f62d415c690 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62dc400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f62d40b7140 con 0x7f62d41625b0 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62dc400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62d40b6f20 con 0x7f62d40d1070 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62d9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d415c690 0x7f62d40c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62d9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d415c690 0x7f62d40c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34396/0 (socket says 172.21.15.69:34396) 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62d9000640 1 -- 172.21.15.69:0/2743584852 learned_addr learned my addr 172.21.15.69:0/2743584852 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62d9000640 1 -- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d40d1070 msgr2=0x7f62d40cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62d9000640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d40d1070 0x7f62d40cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62d9000640 1 -- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d41625b0 msgr2=0x7f62d4158690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62d9000640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d41625b0 0x7f62d4158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:12.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.148+0000 7f62d9000640 1 -- 172.21.15.69:0/2743584852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d40ccc80 con 0x7f62d415c690 2024-04-24T08:15:12.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.149+0000 7f62d9000640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d415c690 0x7f62d40c7b80 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f62c000c460 tx=0x7f62c000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:12.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.149+0000 7f62d1600640 1 -- 172.21.15.69:0/2743584852 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c0017070 con 0x7f62d415c690 2024-04-24T08:15:12.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.149+0000 7f62d1600640 1 -- 172.21.15.69:0/2743584852 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62c0005ce0 con 0x7f62d415c690 2024-04-24T08:15:12.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.149+0000 7f62d1600640 1 -- 172.21.15.69:0/2743584852 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c0007290 con 0x7f62d415c690 2024-04-24T08:15:12.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.149+0000 7f62dc400640 1 -- 172.21.15.69:0/2743584852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d40ccf60 con 0x7f62d415c690 2024-04-24T08:15:12.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.149+0000 7f62dc400640 1 -- 172.21.15.69:0/2743584852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d40cd520 con 0x7f62d415c690 2024-04-24T08:15:12.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.150+0000 7f62dc400640 1 -- 172.21.15.69:0/2743584852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d4068a80 con 0x7f62d415c690 2024-04-24T08:15:12.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.151+0000 7f62d1600640 1 -- 172.21.15.69:0/2743584852 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f62c001a070 con 0x7f62d415c690 2024-04-24T08:15:12.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.152+0000 7f62d1600640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f62b406c8f0 0x7f62b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:12.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.152+0000 7f62d1600640 1 -- 172.21.15.69:0/2743584852 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 5997+0+0 (secure 0 0 0) 0x7f62c008b080 con 0x7f62d415c690 2024-04-24T08:15:12.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.152+0000 7f62d9a00640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f62b406c8f0 0x7f62b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:12.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.154+0000 7f62d9a00640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f62b406c8f0 0x7f62b406edb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f62c8006a90 tx=0x7f62c8006a20 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:12.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.156+0000 7f62d1600640 1 -- 172.21.15.69:0/2743584852 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62c00592f0 con 0x7f62d415c690 2024-04-24T08:15:12.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.354+0000 7f62dc400640 1 -- 172.21.15.69:0/2743584852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f62d40d41b0 con 0x7f62d415c690 2024-04-24T08:15:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.868+0000 7f62d1600640 1 -- 172.21.15.69:0/2743584852 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) ==== 181+0+0 (secure 0 0 0) 0x7f62c0058bf0 con 0x7f62d415c690 2024-04-24T08:15:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' removed 2024-04-24T08:15:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.874+0000 7f62dc400640 1 -- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f62b406c8f0 msgr2=0x7f62b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.874+0000 7f62dc400640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f62b406c8f0 0x7f62b406edb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f62c8006a90 tx=0x7f62c8006a20 comp rx=0 tx=0).stop 2024-04-24T08:15:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.874+0000 7f62dc400640 1 -- 172.21.15.69:0/2743584852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d415c690 msgr2=0x7f62d40c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.874+0000 7f62dc400640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d415c690 0x7f62d40c7b80 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f62c000c460 tx=0x7f62c000c920 comp rx=0 tx=0).stop 2024-04-24T08:15:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.875+0000 7f62dc400640 1 -- 172.21.15.69:0/2743584852 shutdown_connections 2024-04-24T08:15:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.875+0000 7f62dc400640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f62b406c8f0 0x7f62b406edb0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.875+0000 7f62dc400640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62d415c690 0x7f62d40c7b80 secure :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f62c000c460 tx=0x7f62c000c920 comp rx=0 tx=0).stop 2024-04-24T08:15:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.875+0000 7f62dc400640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d41625b0 0x7f62d4158690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.875+0000 7f62dc400640 1 --2- 172.21.15.69:0/2743584852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62d40d1070 0x7f62d40cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.875+0000 7f62dc400640 1 -- 172.21.15.69:0/2743584852 >> 172.21.15.69:0/2743584852 conn(0x7f62d4067760 msgr2=0x7f62d4067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.876+0000 7f62dc400640 1 -- 172.21.15.69:0/2743584852 shutdown_connections 2024-04-24T08:15:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:12.876+0000 7f62dc400640 1 -- 172.21.15.69:0/2743584852 wait complete. 2024-04-24T08:15:12.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T08:15:13.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.103+0000 7f8406200640 1 Processor -- start 2024-04-24T08:15:13.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.103+0000 7f8406200640 1 -- start start 2024-04-24T08:15:13.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.104+0000 7f8406200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f840015e8d0 0x7f84000d1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.104+0000 7f8406200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f84000cfd00 0x7f84000d45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.104+0000 7f8406200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84000d0470 con 0x7f840015e4f0 2024-04-24T08:15:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.104+0000 7f8406200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f84000d2240 con 0x7f840015e8d0 2024-04-24T08:15:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.104+0000 7f8406200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f84000d2ae0 con 0x7f84000cfd00 2024-04-24T08:15:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.104+0000 7f8404e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f840015e4f0 0x7f84000d2f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35902/0 (socket says 172.21.15.69:35902) 2024-04-24T08:15:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.104+0000 7f8404e00640 1 -- 172.21.15.69:0/3590743428 learned_addr learned my addr 172.21.15.69:0/3590743428 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.104+0000 7f83f7e00640 1 --2- 172.21.15.69:0/3590743428 >> v2:172.21.15.69:3300/0 conn(0x7f840015e8d0 0x7f84000d1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.105+0000 7f83ffe00640 1 -- 172.21.15.69:0/3590743428 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3072379088 0 0) 0x7f84000d0470 con 0x7f840015e4f0 2024-04-24T08:15:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.105+0000 7f83ffe00640 1 -- 172.21.15.69:0/3590743428 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83e4003520 con 0x7f840015e4f0 2024-04-24T08:15:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.105+0000 7f8405800640 1 --2- 172.21.15.69:0/3590743428 >> v2:172.21.15.121:3300/0 conn(0x7f84000cfd00 0x7f84000d45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.105+0000 7f83f7e00640 1 -- 172.21.15.69:0/3590743428 >> v2:172.21.15.121:3300/0 conn(0x7f84000cfd00 msgr2=0x7f84000d45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.105+0000 7f83f7e00640 1 --2- 172.21.15.69:0/3590743428 >> v2:172.21.15.121:3300/0 conn(0x7f84000cfd00 0x7f84000d45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.105+0000 7f83f7e00640 1 -- 172.21.15.69:0/3590743428 >> v1:172.21.15.69:6789/0 conn(0x7f840015e4f0 legacy=0x7f84000d2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.105+0000 7f83f7e00640 1 -- 172.21.15.69:0/3590743428 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84000d2490 con 0x7f840015e8d0 2024-04-24T08:15:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.105+0000 7f8405800640 1 --2- 172.21.15.69:0/3590743428 >> v2:172.21.15.121:3300/0 conn(0x7f84000cfd00 0x7f84000d45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.106+0000 7f83f7e00640 1 --2- 172.21.15.69:0/3590743428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f840015e8d0 0x7f84000d1ac0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f84000d44f0 tx=0x7f83e802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4402417d502b317d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.106+0000 7f83ffe00640 1 -- 172.21.15.69:0/3590743428 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e8037070 con 0x7f840015e8d0 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.106+0000 7f83ffe00640 1 -- 172.21.15.69:0/3590743428 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e8032440 con 0x7f840015e8d0 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.106+0000 7f83ffe00640 1 -- 172.21.15.69:0/3590743428 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e80327f0 con 0x7f840015e8d0 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.107+0000 7f8406200640 1 -- 172.21.15.69:0/3590743428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f840015e8d0 msgr2=0x7f84000d1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.107+0000 7f8406200640 1 --2- 172.21.15.69:0/3590743428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f840015e8d0 0x7f84000d1ac0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f84000d44f0 tx=0x7f83e802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.107+0000 7f8406200640 1 -- 172.21.15.69:0/3590743428 shutdown_connections 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.107+0000 7f8406200640 1 --2- 172.21.15.69:0/3590743428 >> v2:172.21.15.121:3300/0 conn(0x7f84000cfd00 0x7f84000d45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.107+0000 7f8406200640 1 --2- 172.21.15.69:0/3590743428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f840015e8d0 0x7f84000d1ac0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.107+0000 7f8406200640 1 -- 172.21.15.69:0/3590743428 >> 172.21.15.69:0/3590743428 conn(0x7f840009d940 msgr2=0x7f840008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.107+0000 7f8406200640 1 -- 172.21.15.69:0/3590743428 shutdown_connections 2024-04-24T08:15:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.107+0000 7f8406200640 1 -- 172.21.15.69:0/3590743428 wait complete. 2024-04-24T08:15:13.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.109+0000 7f8406200640 1 Processor -- start 2024-04-24T08:15:13.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.109+0000 7f8406200640 1 -- start start 2024-04-24T08:15:13.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.110+0000 7f8406200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84000cfd00 0x7f8400160d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.110+0000 7f8406200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840015e4f0 0x7f840015cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.110+0000 7f8406200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f840015e8d0 0x7f84000cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.110+0000 7f8406200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84000d0170 con 0x7f84000cfd00 2024-04-24T08:15:13.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.110+0000 7f8406200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f84000d3680 con 0x7f840015e4f0 2024-04-24T08:15:13.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.110+0000 7f8406200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f84000d4b20 con 0x7f840015e8d0 2024-04-24T08:15:13.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.110+0000 7f8404e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84000cfd00 0x7f8400160d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.110+0000 7f8404e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84000cfd00 0x7f8400160d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34406/0 (socket says 172.21.15.69:34406) 2024-04-24T08:15:13.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.110+0000 7f8404e00640 1 -- 172.21.15.69:0/3220276855 learned_addr learned my addr 172.21.15.69:0/3220276855 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:13.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f8404e00640 1 -- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f840015e8d0 msgr2=0x7f84000cbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f8404e00640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f840015e8d0 0x7f84000cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f8404e00640 1 -- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840015e4f0 msgr2=0x7f840015cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f83f7e00640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840015e4f0 0x7f840015cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f8404e00640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840015e4f0 0x7f840015cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f8404e00640 1 -- 172.21.15.69:0/3220276855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84000b52a0 con 0x7f84000cfd00 2024-04-24T08:15:13.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f8404e00640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84000cfd00 0x7f8400160d10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f83f000c460 tx=0x7f83f000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f83fd600640 1 -- 172.21.15.69:0/3220276855 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0017070 con 0x7f84000cfd00 2024-04-24T08:15:13.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f83fd600640 1 -- 172.21.15.69:0/3220276855 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f0005ce0 con 0x7f84000cfd00 2024-04-24T08:15:13.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f83fd600640 1 -- 172.21.15.69:0/3220276855 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0007290 con 0x7f84000cfd00 2024-04-24T08:15:13.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f8406200640 1 -- 172.21.15.69:0/3220276855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8400161c40 con 0x7f84000cfd00 2024-04-24T08:15:13.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.111+0000 7f8406200640 1 -- 172.21.15.69:0/3220276855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8400161680 con 0x7f84000cfd00 2024-04-24T08:15:13.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.113+0000 7f8406200640 1 -- 172.21.15.69:0/3220276855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84000cb0d0 con 0x7f84000cfd00 2024-04-24T08:15:13.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.115+0000 7f83fd600640 1 -- 172.21.15.69:0/3220276855 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f83f001a070 con 0x7f84000cfd00 2024-04-24T08:15:13.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.115+0000 7f83fd600640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f83dc06c8f0 0x7f83dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.115+0000 7f83fd600640 1 -- 172.21.15.69:0/3220276855 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 5559+0+0 (secure 0 0 0) 0x7f83f008ae40 con 0x7f84000cfd00 2024-04-24T08:15:13.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.115+0000 7f83f7e00640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f83dc06c8f0 0x7f83dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.116+0000 7f83f7e00640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f83dc06c8f0 0x7f83dc06edb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f83e8034000 tx=0x7f83e802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.119+0000 7f83fd600640 1 -- 172.21.15.69:0/3220276855 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f84000cb0d0 con 0x7f84000cfd00 2024-04-24T08:15:13.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.320+0000 7f8406200640 1 -- 172.21.15.69:0/3220276855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8400068a80 con 0x7f84000cfd00 2024-04-24T08:15:13.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.883+0000 7f83fd600640 1 -- 172.21.15.69:0/3220276855 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) ==== 169+0+0 (secure 0 0 0) 0x7f83f00592f0 con 0x7f84000cfd00 2024-04-24T08:15:13.884 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' removed 2024-04-24T08:15:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.885+0000 7f8406200640 1 -- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f83dc06c8f0 msgr2=0x7f83dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.885+0000 7f8406200640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f83dc06c8f0 0x7f83dc06edb0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f83e8034000 tx=0x7f83e802e000 comp rx=0 tx=0).stop 2024-04-24T08:15:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.885+0000 7f8406200640 1 -- 172.21.15.69:0/3220276855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84000cfd00 msgr2=0x7f8400160d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.885+0000 7f8406200640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84000cfd00 0x7f8400160d10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f83f000c460 tx=0x7f83f000c920 comp rx=0 tx=0).stop 2024-04-24T08:15:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.885+0000 7f8406200640 1 -- 172.21.15.69:0/3220276855 shutdown_connections 2024-04-24T08:15:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.885+0000 7f8406200640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f83dc06c8f0 0x7f83dc06edb0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.886+0000 7f8406200640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f840015e8d0 0x7f84000cbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.886+0000 7f8406200640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f840015e4f0 0x7f840015cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.886+0000 7f8406200640 1 --2- 172.21.15.69:0/3220276855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84000cfd00 0x7f8400160d10 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.886+0000 7f8406200640 1 -- 172.21.15.69:0/3220276855 >> 172.21.15.69:0/3220276855 conn(0x7f840009d940 msgr2=0x7f8400074cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.886+0000 7f8406200640 1 -- 172.21.15.69:0/3220276855 shutdown_connections 2024-04-24T08:15:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:13.886+0000 7f8406200640 1 -- 172.21.15.69:0/3220276855 wait complete. 2024-04-24T08:15:13.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-24T08:15:14.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.107+0000 7f35a1200640 1 Processor -- start 2024-04-24T08:15:14.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.107+0000 7f35a1200640 1 -- start start 2024-04-24T08:15:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.108+0000 7f35a1200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f359c15d030 0x7f359c163b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.108+0000 7f35a1200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f359c071d70 0x7f359c0721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.108+0000 7f35a1200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f359c1666d0 0x7f359c166b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.108+0000 7f35a1200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f359c0b39e0 con 0x7f359c1666d0 2024-04-24T08:15:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.108+0000 7f35a1200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f359c1640a0 con 0x7f359c071d70 2024-04-24T08:15:14.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.108+0000 7f35a1200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f359c0b3700 con 0x7f359c15d030 2024-04-24T08:15:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f359be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f359c1666d0 0x7f359c166b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f359be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f359c1666d0 0x7f359c166b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34420/0 (socket says 172.21.15.69:34420) 2024-04-24T08:15:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f359be00640 1 -- 172.21.15.69:0/430062465 learned_addr learned my addr 172.21.15.69:0/430062465 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f359b400640 1 --2- 172.21.15.69:0/430062465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f359c15d030 0x7f359c163b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f3592a00640 1 --2- 172.21.15.69:0/430062465 >> v2:172.21.15.121:3300/0 conn(0x7f359c071d70 0x7f359c0721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f359be00640 1 -- 172.21.15.69:0/430062465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f359c15d030 msgr2=0x7f359c163b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f359be00640 1 --2- 172.21.15.69:0/430062465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f359c15d030 0x7f359c163b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f359be00640 1 -- 172.21.15.69:0/430062465 >> v2:172.21.15.121:3300/0 conn(0x7f359c071d70 msgr2=0x7f359c0721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:14.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f359be00640 1 --2- 172.21.15.69:0/430062465 >> v2:172.21.15.121:3300/0 conn(0x7f359c071d70 0x7f359c0721f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f359be00640 1 -- 172.21.15.69:0/430062465 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359c0b3e20 con 0x7f359c1666d0 2024-04-24T08:15:14.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.109+0000 7f3592a00640 1 --2- 172.21.15.69:0/430062465 >> v2:172.21.15.121:3300/0 conn(0x7f359c071d70 0x7f359c0721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:14.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.110+0000 7f359be00640 1 --2- 172.21.15.69:0/430062465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c1666d0 0x7f359c166b40 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f359c06ffd0 tx=0x7f359402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1bcb238cfc120ec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.111+0000 7f359aa00640 1 -- 172.21.15.69:0/430062465 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594036070 con 0x7f359c1666d0 2024-04-24T08:15:14.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.111+0000 7f359aa00640 1 -- 172.21.15.69:0/430062465 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3594031600 con 0x7f359c1666d0 2024-04-24T08:15:14.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.111+0000 7f359aa00640 1 -- 172.21.15.69:0/430062465 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35940319d0 con 0x7f359c1666d0 2024-04-24T08:15:14.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.111+0000 7f35a1200640 1 -- 172.21.15.69:0/430062465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c1666d0 msgr2=0x7f359c166b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.111+0000 7f35a1200640 1 --2- 172.21.15.69:0/430062465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c1666d0 0x7f359c166b40 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f359c06ffd0 tx=0x7f359402a6a0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.112+0000 7f35a1200640 1 -- 172.21.15.69:0/430062465 shutdown_connections 2024-04-24T08:15:14.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.112+0000 7f35a1200640 1 --2- 172.21.15.69:0/430062465 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c1666d0 0x7f359c166b40 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.112+0000 7f35a1200640 1 --2- 172.21.15.69:0/430062465 >> v2:172.21.15.121:3300/0 conn(0x7f359c071d70 0x7f359c0721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.112+0000 7f35a1200640 1 --2- 172.21.15.69:0/430062465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f359c15d030 0x7f359c163b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.112+0000 7f35a1200640 1 -- 172.21.15.69:0/430062465 >> 172.21.15.69:0/430062465 conn(0x7f359c067b80 msgr2=0x7f359c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:14.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.112+0000 7f35a1200640 1 -- 172.21.15.69:0/430062465 shutdown_connections 2024-04-24T08:15:14.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.112+0000 7f35a1200640 1 -- 172.21.15.69:0/430062465 wait complete. 2024-04-24T08:15:14.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.113+0000 7f35a1200640 1 Processor -- start 2024-04-24T08:15:14.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.114+0000 7f35a1200640 1 -- start start 2024-04-24T08:15:14.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.114+0000 7f35a1200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f359c071d70 0x7f359c1547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.114+0000 7f35a1200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c15d030 0x7f359c15dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f35a1200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359c1666d0 0x7f359c0b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f35a1200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f359c06f7b0 con 0x7f359c15d030 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f35a1200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f359c0b43e0 con 0x7f359c1666d0 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f35a1200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f359c0b3320 con 0x7f359c071d70 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f3592a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c15d030 0x7f359c15dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f3592a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c15d030 0x7f359c15dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34434/0 (socket says 172.21.15.69:34434) 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f3592a00640 1 -- 172.21.15.69:0/4024946956 learned_addr learned my addr 172.21.15.69:0/4024946956 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f359b400640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f359c071d70 0x7f359c1547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f359be00640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359c1666d0 0x7f359c0b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f3592a00640 1 -- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f359c071d70 msgr2=0x7f359c1547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f3592a00640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f359c071d70 0x7f359c1547d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f3592a00640 1 -- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359c1666d0 msgr2=0x7f359c0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f3592a00640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359c1666d0 0x7f359c0b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.115+0000 7f3592a00640 1 -- 172.21.15.69:0/4024946956 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359c1f8860 con 0x7f359c15d030 2024-04-24T08:15:14.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.116+0000 7f3592a00640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c15d030 0x7f359c15dde0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f358400c7c0 tx=0x7f358400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.116+0000 7f3593e00640 1 -- 172.21.15.69:0/4024946956 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584015070 con 0x7f359c15d030 2024-04-24T08:15:14.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.116+0000 7f3593e00640 1 -- 172.21.15.69:0/4024946956 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3584007a30 con 0x7f359c15d030 2024-04-24T08:15:14.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.116+0000 7f35a1200640 1 -- 172.21.15.69:0/4024946956 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f359c15a660 con 0x7f359c15d030 2024-04-24T08:15:14.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.116+0000 7f3593e00640 1 -- 172.21.15.69:0/4024946956 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584007e00 con 0x7f359c15d030 2024-04-24T08:15:14.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.116+0000 7f35a1200640 1 -- 172.21.15.69:0/4024946956 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f359c15ac20 con 0x7f359c15d030 2024-04-24T08:15:14.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.117+0000 7f35a1200640 1 -- 172.21.15.69:0/4024946956 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f359c165e70 con 0x7f359c15d030 2024-04-24T08:15:14.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.118+0000 7f3593e00640 1 -- 172.21.15.69:0/4024946956 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3584018070 con 0x7f359c15d030 2024-04-24T08:15:14.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.119+0000 7f3593e00640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f357806c8f0 0x7f357806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.119+0000 7f3593e00640 1 -- 172.21.15.69:0/4024946956 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f358408b7b0 con 0x7f359c15d030 2024-04-24T08:15:14.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.120+0000 7f359b400640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f357806c8f0 0x7f357806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.121+0000 7f359b400640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f357806c8f0 0x7f357806edb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f358c006fd0 tx=0x7f358c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.124+0000 7f3593e00640 1 -- 172.21.15.69:0/4024946956 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f359c165e70 con 0x7f359c15d030 2024-04-24T08:15:14.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.323+0000 7f35a1200640 1 -- 172.21.15.69:0/4024946956 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f359c1610a0 con 0x7f359c15d030 2024-04-24T08:15:14.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.329+0000 7f3593e00640 1 -- 172.21.15.69:0/4024946956 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) ==== 104+0+0 (secure 0 0 0) 0x7f3584059e70 con 0x7f359c15d030 2024-04-24T08:15:14.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.331+0000 7f35a1200640 1 -- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f357806c8f0 msgr2=0x7f357806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.331+0000 7f35a1200640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f357806c8f0 0x7f357806edb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f358c006fd0 tx=0x7f358c008040 comp rx=0 tx=0).stop 2024-04-24T08:15:14.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.331+0000 7f35a1200640 1 -- 172.21.15.69:0/4024946956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c15d030 msgr2=0x7f359c15dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.331+0000 7f35a1200640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c15d030 0x7f359c15dde0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f358400c7c0 tx=0x7f358400cc80 comp rx=0 tx=0).stop 2024-04-24T08:15:14.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.331+0000 7f35a1200640 1 -- 172.21.15.69:0/4024946956 shutdown_connections 2024-04-24T08:15:14.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.332+0000 7f35a1200640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f357806c8f0 0x7f357806edb0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.332+0000 7f35a1200640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f359c15d030 0x7f359c15dde0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.332+0000 7f35a1200640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f359c1666d0 0x7f359c0b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.332+0000 7f35a1200640 1 --2- 172.21.15.69:0/4024946956 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f359c071d70 0x7f359c1547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.332+0000 7f35a1200640 1 -- 172.21.15.69:0/4024946956 >> 172.21.15.69:0/4024946956 conn(0x7f359c067b80 msgr2=0x7f359c06e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:14.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.332+0000 7f35a1200640 1 -- 172.21.15.69:0/4024946956 shutdown_connections 2024-04-24T08:15:14.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.332+0000 7f35a1200640 1 -- 172.21.15.69:0/4024946956 wait complete. 2024-04-24T08:15:14.351 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-04-24T08:15:14.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.555+0000 7fa51f200640 1 Processor -- start 2024-04-24T08:15:14.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.555+0000 7fa51f200640 1 -- start start 2024-04-24T08:15:14.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.557+0000 7fa51f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5200d0200 0x7fa5200d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.557+0000 7fa51f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa52015ed40 con 0x7fa52006cfb0 2024-04-24T08:15:14.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.557+0000 7fa51f200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5200cf050 con 0x7fa52015e960 2024-04-24T08:15:14.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.557+0000 7fa51f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa520162b50 con 0x7fa5200d0200 2024-04-24T08:15:14.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.557+0000 7fa51e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa5200d0200 0x7fa5200d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.557+0000 7fa517e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa52006cfb0 0x7fa5200cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35906/0 (socket says 172.21.15.69:35906) 2024-04-24T08:15:14.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.557+0000 7fa517e00640 1 -- 172.21.15.69:0/1023945841 learned_addr learned my addr 172.21.15.69:0/1023945841 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:14.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.557+0000 7fa51e800640 1 -- 172.21.15.69:0/1023945841 >> v1:172.21.15.121:6789/0 conn(0x7fa52015e960 legacy=0x7fa520162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:14.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.557+0000 7fa51e800640 1 -- 172.21.15.69:0/1023945841 >> v1:172.21.15.69:6789/0 conn(0x7fa52006cfb0 legacy=0x7fa5200cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.558+0000 7fa51e800640 1 -- 172.21.15.69:0/1023945841 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa520162e20 con 0x7fa5200d0200 2024-04-24T08:15:14.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.559+0000 7fa51e800640 1 --2- 172.21.15.69:0/1023945841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 0x7fa5200d2980 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fa5200cee40 tx=0x7fa51802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6299fbe8cc6c7817 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.559+0000 7fa51d400640 1 -- 172.21.15.69:0/1023945841 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa51802ae60 con 0x7fa5200d0200 2024-04-24T08:15:14.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.559+0000 7fa51d400640 1 -- 172.21.15.69:0/1023945841 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa518007740 con 0x7fa5200d0200 2024-04-24T08:15:14.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.559+0000 7fa51d400640 1 -- 172.21.15.69:0/1023945841 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518007b10 con 0x7fa5200d0200 2024-04-24T08:15:14.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.560+0000 7fa51f200640 1 -- 172.21.15.69:0/1023945841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 msgr2=0x7fa5200d2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.560+0000 7fa51f200640 1 --2- 172.21.15.69:0/1023945841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 0x7fa5200d2980 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fa5200cee40 tx=0x7fa51802a680 comp rx=0 tx=0).stop 2024-04-24T08:15:14.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.560+0000 7fa51f200640 1 -- 172.21.15.69:0/1023945841 shutdown_connections 2024-04-24T08:15:14.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.560+0000 7fa51f200640 1 --2- 172.21.15.69:0/1023945841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 0x7fa5200d2980 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.560+0000 7fa51f200640 1 -- 172.21.15.69:0/1023945841 >> 172.21.15.69:0/1023945841 conn(0x7fa52009d900 msgr2=0x7fa52008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:14.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.560+0000 7fa51f200640 1 -- 172.21.15.69:0/1023945841 shutdown_connections 2024-04-24T08:15:14.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.560+0000 7fa51f200640 1 -- 172.21.15.69:0/1023945841 wait complete. 2024-04-24T08:15:14.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.561+0000 7fa51f200640 1 Processor -- start 2024-04-24T08:15:14.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.562+0000 7fa51f200640 1 -- start start 2024-04-24T08:15:14.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa51f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa52006cfb0 0x7fa5200ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa51f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 0x7fa5200ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa51f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa52015e960 0x7fa5200cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa51f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa5200cf5d0 con 0x7fa5200d0200 2024-04-24T08:15:14.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa51f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa5200cf2f0 con 0x7fa52006cfb0 2024-04-24T08:15:14.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa51f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa5200b6170 con 0x7fa52015e960 2024-04-24T08:15:14.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa517e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 0x7fa5200ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa517e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 0x7fa5200ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34446/0 (socket says 172.21.15.69:34446) 2024-04-24T08:15:14.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa517e00640 1 -- 172.21.15.69:0/1262416778 learned_addr learned my addr 172.21.15.69:0/1262416778 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:14.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa517e00640 1 -- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa52015e960 msgr2=0x7fa5200cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa517e00640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa52015e960 0x7fa5200cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa517e00640 1 -- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa52006cfb0 msgr2=0x7fa5200ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa517e00640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa52006cfb0 0x7fa5200ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.563+0000 7fa517e00640 1 -- 172.21.15.69:0/1262416778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5200cd6a0 con 0x7fa5200d0200 2024-04-24T08:15:14.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.564+0000 7fa517e00640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 0x7fa5200ccb00 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fa50800c430 tx=0x7fa50800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.564+0000 7fa516000640 1 -- 172.21.15.69:0/1262416778 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa508017070 con 0x7fa5200d0200 2024-04-24T08:15:14.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.564+0000 7fa516000640 1 -- 172.21.15.69:0/1262416778 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa508005ce0 con 0x7fa5200d0200 2024-04-24T08:15:14.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.564+0000 7fa516000640 1 -- 172.21.15.69:0/1262416778 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5080072f0 con 0x7fa5200d0200 2024-04-24T08:15:14.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.564+0000 7fa51f200640 1 -- 172.21.15.69:0/1262416778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5200cd980 con 0x7fa5200d0200 2024-04-24T08:15:14.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.564+0000 7fa51f200640 1 -- 172.21.15.69:0/1262416778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa52015f3a0 con 0x7fa5200d0200 2024-04-24T08:15:14.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.565+0000 7fa51f200640 1 -- 172.21.15.69:0/1262416778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa520075d50 con 0x7fa5200d0200 2024-04-24T08:15:14.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.566+0000 7fa516000640 1 -- 172.21.15.69:0/1262416778 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa50801a070 con 0x7fa5200d0200 2024-04-24T08:15:14.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.568+0000 7fa516000640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4fc06c8f0 0x7fa4fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.568+0000 7fa516000640 1 -- 172.21.15.69:0/1262416778 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7fa50808ad10 con 0x7fa5200d0200 2024-04-24T08:15:14.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.568+0000 7fa51de00640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4fc06c8f0 0x7fa4fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.569+0000 7fa51de00640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4fc06c8f0 0x7fa4fc06edb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa510006fd0 tx=0x7fa510006bc0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.572+0000 7fa516000640 1 -- 172.21.15.69:0/1262416778 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa508090020 con 0x7fa5200d0200 2024-04-24T08:15:14.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.847+0000 7fa51f200640 1 -- 172.21.15.69:0/1262416778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) -- 0x7fa5200cdb50 con 0x7fa5200d0200 2024-04-24T08:15:14.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.886+0000 7fa516000640 1 -- 172.21.15.69:0/1262416778 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v336) ==== 166+0+0 (secure 0 0 0) 0x7fa508059340 con 0x7fa5200d0200 2024-04-24T08:15:14.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.889+0000 7fa51f200640 1 -- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4fc06c8f0 msgr2=0x7fa4fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.889+0000 7fa51f200640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4fc06c8f0 0x7fa4fc06edb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa510006fd0 tx=0x7fa510006bc0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.889+0000 7fa51f200640 1 -- 172.21.15.69:0/1262416778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 msgr2=0x7fa5200ccb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.889+0000 7fa51f200640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 0x7fa5200ccb00 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fa50800c430 tx=0x7fa50800c8f0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.889+0000 7fa51f200640 1 -- 172.21.15.69:0/1262416778 shutdown_connections 2024-04-24T08:15:14.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.889+0000 7fa51f200640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4fc06c8f0 0x7fa4fc06edb0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.889+0000 7fa51f200640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5200d0200 0x7fa5200ccb00 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.889+0000 7fa51f200640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa52015e960 0x7fa5200cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.889+0000 7fa51f200640 1 --2- 172.21.15.69:0/1262416778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa52006cfb0 0x7fa5200ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.889+0000 7fa51f200640 1 -- 172.21.15.69:0/1262416778 >> 172.21.15.69:0/1262416778 conn(0x7fa52009d900 msgr2=0x7fa52006db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:14.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.890+0000 7fa51f200640 1 -- 172.21.15.69:0/1262416778 shutdown_connections 2024-04-24T08:15:14.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:14.890+0000 7fa51f200640 1 -- 172.21.15.69:0/1262416778 wait complete. 2024-04-24T08:15:14.910 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-04-24T08:15:14.911 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-04-24T08:15:14.911 INFO:tasks.cephfs_test_runner:test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-24T08:15:14.911 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:15.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.112+0000 7f09cee00640 1 Processor -- start 2024-04-24T08:15:15.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.112+0000 7f09cee00640 1 -- start start 2024-04-24T08:15:15.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.113+0000 7f09cee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09c815d1d0 0x7f09c80ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.113+0000 7f09cee00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09c80b30f0 0x7f09c80ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.113+0000 7f09cee00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c815d660 con 0x7f09c8156c60 2024-04-24T08:15:15.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.113+0000 7f09cee00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f09c806dc30 con 0x7f09c80b30f0 2024-04-24T08:15:15.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.113+0000 7f09cee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09c806df10 con 0x7f09c815d1d0 2024-04-24T08:15:15.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.114+0000 7f09ce400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09c80b30f0 0x7f09c80ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.114+0000 7f09ce400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09c80b30f0 0x7f09c80ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34452/0 (socket says 172.21.15.69:34452) 2024-04-24T08:15:15.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.114+0000 7f09ce400640 1 -- 172.21.15.69:0/4291687257 learned_addr learned my addr 172.21.15.69:0/4291687257 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:15.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.114+0000 7f09ce400640 1 -- 172.21.15.69:0/4291687257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09c815d1d0 msgr2=0x7f09c80ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.114+0000 7f09ce400640 1 --2- 172.21.15.69:0/4291687257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09c815d1d0 0x7f09c80ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.114+0000 7f09ce400640 1 -- 172.21.15.69:0/4291687257 >> v1:172.21.15.121:6789/0 conn(0x7f09c8156c60 legacy=0x7f09c815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.114+0000 7f09ce400640 1 -- 172.21.15.69:0/4291687257 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c806e1f0 con 0x7f09c80b30f0 2024-04-24T08:15:15.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.115+0000 7f09ce400640 1 --2- 172.21.15.69:0/4291687257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c80b30f0 0x7f09c80ad610 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009870 tx=0x7f09c002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d3b918fcef0915 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.115+0000 7f09c7e00640 1 -- 172.21.15.69:0/4291687257 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0037070 con 0x7f09c80b30f0 2024-04-24T08:15:15.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.115+0000 7f09c7e00640 1 -- 172.21.15.69:0/4291687257 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09c0032440 con 0x7f09c80b30f0 2024-04-24T08:15:15.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.115+0000 7f09c7e00640 1 -- 172.21.15.69:0/4291687257 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0032810 con 0x7f09c80b30f0 2024-04-24T08:15:15.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.116+0000 7f09cee00640 1 -- 172.21.15.69:0/4291687257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c80b30f0 msgr2=0x7f09c80ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.116+0000 7f09cee00640 1 --2- 172.21.15.69:0/4291687257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c80b30f0 0x7f09c80ad610 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f09c0009870 tx=0x7f09c002a210 comp rx=0 tx=0).stop 2024-04-24T08:15:15.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.116+0000 7f09cee00640 1 -- 172.21.15.69:0/4291687257 shutdown_connections 2024-04-24T08:15:15.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.116+0000 7f09cee00640 1 --2- 172.21.15.69:0/4291687257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c80b30f0 0x7f09c80ad610 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.116+0000 7f09cee00640 1 --2- 172.21.15.69:0/4291687257 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09c815d1d0 0x7f09c80ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.116+0000 7f09cee00640 1 -- 172.21.15.69:0/4291687257 >> 172.21.15.69:0/4291687257 conn(0x7f09c80d0160 msgr2=0x7f09c80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:15.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.116+0000 7f09cee00640 1 -- 172.21.15.69:0/4291687257 shutdown_connections 2024-04-24T08:15:15.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.116+0000 7f09cee00640 1 -- 172.21.15.69:0/4291687257 wait complete. 2024-04-24T08:15:15.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.118+0000 7f09cee00640 1 Processor -- start 2024-04-24T08:15:15.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.118+0000 7f09cee00640 1 -- start start 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.119+0000 7f09cee00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c80b30f0 0x7f09c81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.119+0000 7f09cee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09c8156c60 0x7f09c8158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.119+0000 7f09cee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c815c3c0 0x7f09c8159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.119+0000 7f09cee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09c806e710 con 0x7f09c815c3c0 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.119+0000 7f09cee00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f09c815d940 con 0x7f09c80b30f0 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.119+0000 7f09cee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09c806e890 con 0x7f09c8156c60 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.119+0000 7f09ce400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c815c3c0 0x7f09c8159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.119+0000 7f09ce400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c815c3c0 0x7f09c8159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34460/0 (socket says 172.21.15.69:34460) 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.119+0000 7f09ce400640 1 -- 172.21.15.69:0/1161704469 learned_addr learned my addr 172.21.15.69:0/1161704469 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09cda00640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c80b30f0 0x7f09c81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09ce400640 1 -- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09c8156c60 msgr2=0x7f09c8158be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09ce400640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09c8156c60 0x7f09c8158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09ce400640 1 -- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c80b30f0 msgr2=0x7f09c81559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09ce400640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c80b30f0 0x7f09c81559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09ce400640 1 -- 172.21.15.69:0/1161704469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c80cbdc0 con 0x7f09c815c3c0 2024-04-24T08:15:15.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09ce400640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c815c3c0 0x7f09c8159120 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f09c002a250 tx=0x7f09c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09c5600640 1 -- 172.21.15.69:0/1161704469 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0041070 con 0x7f09c815c3c0 2024-04-24T08:15:15.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09c5600640 1 -- 172.21.15.69:0/1161704469 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09c003c950 con 0x7f09c815c3c0 2024-04-24T08:15:15.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09c5600640 1 -- 172.21.15.69:0/1161704469 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c0037040 con 0x7f09c815c3c0 2024-04-24T08:15:15.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.120+0000 7f09cee00640 1 -- 172.21.15.69:0/1161704469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c8073600 con 0x7f09c815c3c0 2024-04-24T08:15:15.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.121+0000 7f09cee00640 1 -- 172.21.15.69:0/1161704469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c8159840 con 0x7f09c815c3c0 2024-04-24T08:15:15.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.121+0000 7f09cee00640 1 -- 172.21.15.69:0/1161704469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c8159660 con 0x7f09c815c3c0 2024-04-24T08:15:15.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.123+0000 7f09c5600640 1 -- 172.21.15.69:0/1161704469 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f09c0044070 con 0x7f09c815c3c0 2024-04-24T08:15:15.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.124+0000 7f09c5600640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f09a806c8f0 0x7f09a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.124+0000 7f09c5600640 1 -- 172.21.15.69:0/1161704469 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f09c00ac580 con 0x7f09c815c3c0 2024-04-24T08:15:15.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.124+0000 7f09cda00640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f09a806c8f0 0x7f09a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.126+0000 7f09cda00640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f09a806c8f0 0x7f09a806edb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f09bc007fd0 tx=0x7f09bc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.128+0000 7f09c5600640 1 -- 172.21.15.69:0/1161704469 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f09c8159660 con 0x7f09c815c3c0 2024-04-24T08:15:15.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:15:15.205 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:15:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.329+0000 7f09cee00640 1 -- 172.21.15.69:0/1161704469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09c8075d50 con 0x7f09c815c3c0 2024-04-24T08:15:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.333+0000 7f09c5600640 1 -- 172.21.15.69:0/1161704469 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+13129 (secure 0 0 0) 0x7f09c007acd0 con 0x7f09c815c3c0 2024-04-24T08:15:15.334 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:15.334 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":74,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:15:13.870212+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/2999903126":"2024-04-25T08:15:10.859113+0000","172.21.15.121:6837/1135108804":"2024-04-25T08:13:37.186005+0000","172.21.15.69:6834/2999903126":"2024-04-25T08:15:10.859113+0000","172.21.15.121:6835/1135108804":"2024-04-25T08:13:37.186005+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:15.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.336+0000 7f09cee00640 1 -- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f09a806c8f0 msgr2=0x7f09a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.336+0000 7f09cee00640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f09a806c8f0 0x7f09a806edb0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f09bc007fd0 tx=0x7f09bc009040 comp rx=0 tx=0).stop 2024-04-24T08:15:15.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.336+0000 7f09cee00640 1 -- 172.21.15.69:0/1161704469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c815c3c0 msgr2=0x7f09c8159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.336+0000 7f09cee00640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c815c3c0 0x7f09c8159120 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f09c002a250 tx=0x7f09c00046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.336+0000 7f09cee00640 1 -- 172.21.15.69:0/1161704469 shutdown_connections 2024-04-24T08:15:15.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.336+0000 7f09cee00640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f09a806c8f0 0x7f09a806edb0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.336+0000 7f09cee00640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c815c3c0 0x7f09c8159120 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.336+0000 7f09cee00640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09c8156c60 0x7f09c8158be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.336+0000 7f09cee00640 1 --2- 172.21.15.69:0/1161704469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09c80b30f0 0x7f09c81559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.336+0000 7f09cee00640 1 -- 172.21.15.69:0/1161704469 >> 172.21.15.69:0/1161704469 conn(0x7f09c80d0160 msgr2=0x7f09c80b34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:15.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.337+0000 7f09cee00640 1 -- 172.21.15.69:0/1161704469 shutdown_connections 2024-04-24T08:15:15.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.337+0000 7f09cee00640 1 -- 172.21.15.69:0/1161704469 wait complete. 2024-04-24T08:15:15.355 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:15:15.355 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:15:15.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.596+0000 7f00d7200640 1 Processor -- start 2024-04-24T08:15:15.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.596+0000 7f00d7200640 1 -- start start 2024-04-24T08:15:15.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.598+0000 7f00d7200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f00d015c9b0 0x7f00d00b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.598+0000 7f00d7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d00b3f70 0x7f00d00cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.598+0000 7f00d7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d00cd6d0 con 0x7f00d0155f20 2024-04-24T08:15:15.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.598+0000 7f00d7200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f00d006db90 con 0x7f00d015c9b0 2024-04-24T08:15:15.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.598+0000 7f00d7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f00d0072070 con 0x7f00d00b3f70 2024-04-24T08:15:15.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.598+0000 7f00cfe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f00d0155f20 0x7f00d00ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35908/0 (socket says 172.21.15.69:35908) 2024-04-24T08:15:15.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.598+0000 7f00cfe00640 1 -- 172.21.15.69:0/2388130102 learned_addr learned my addr 172.21.15.69:0/2388130102 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2711639259 0 0) 0x7f00d00cd6d0 con 0x7f00d0155f20 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00b4005730 con 0x7f00d0155f20 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00d5200640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d00b3f70 0x7f00d00cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00c7400640 1 --2- 172.21.15.69:0/2388130102 >> v2:172.21.15.121:3300/0 conn(0x7f00d015c9b0 0x7f00d00b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 573898755 0 0) 0x7f00b4005730 con 0x7f00d0155f20 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d00b3f70 msgr2=0x7f00d00cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d00b3f70 0x7f00d00cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 >> v2:172.21.15.121:3300/0 conn(0x7f00d015c9b0 msgr2=0x7f00d00b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 --2- 172.21.15.69:0/2388130102 >> v2:172.21.15.121:3300/0 conn(0x7f00d015c9b0 0x7f00d00b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d00cd9b0 con 0x7f00d0155f20 2024-04-24T08:15:15.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f00c00019f0 con 0x7f00d0155f20 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 >> v1:172.21.15.69:6789/0 conn(0x7f00d0155f20 legacy=0x7f00d00ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d00b3f70 0x7f00b4005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b40061f0 0x7f00b4006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b4006f50 0x7f00b40073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00d0072070 con 0x7f00b40061f0 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f00b400dce0 con 0x7f00b4006f50 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f00b400dee0 con 0x7f00d00b3f70 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.599+0000 7f00c7400640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b40061f0 0x7f00b4006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.600+0000 7f00cfe00640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d00b3f70 0x7f00b4005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.600+0000 7f00d5200640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b4006f50 0x7f00b40073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.600+0000 7f00c7400640 1 -- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d00b3f70 msgr2=0x7f00b4005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.600+0000 7f00c7400640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d00b3f70 0x7f00b4005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.600+0000 7f00c7400640 1 -- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b4006f50 msgr2=0x7f00b40073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.600+0000 7f00c7400640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b4006f50 0x7f00b40073d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.600+0000 7f00c7400640 1 -- 172.21.15.69:0/2388130102 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b4005730 con 0x7f00b40061f0 2024-04-24T08:15:15.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.600+0000 7f00cfe00640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d00b3f70 0x7f00b4005bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:15.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.601+0000 7f00c7400640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b40061f0 0x7f00b4006a10 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f00b800ba00 tx=0x7f00b4007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80eb130f8eb6c0c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.601+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b8030070 con 0x7f00b40061f0 2024-04-24T08:15:15.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.601+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b802ade0 con 0x7f00b40061f0 2024-04-24T08:15:15.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.601+0000 7f00cf400640 1 -- 172.21.15.69:0/2388130102 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b800b600 con 0x7f00b40061f0 2024-04-24T08:15:15.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.602+0000 7f00d7200640 1 -- 172.21.15.69:0/2388130102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b40061f0 msgr2=0x7f00b4006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.602+0000 7f00d7200640 1 --2- 172.21.15.69:0/2388130102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00b40061f0 0x7f00b4006a10 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f00b800ba00 tx=0x7f00b4007a40 comp rx=0 tx=0).stop 2024-04-24T08:15:15.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.602+0000 7f00d5200640 1 -- 172.21.15.69:0/2388130102 reap_dead start 2024-04-24T08:15:15.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.602+0000 7f00d7200640 1 -- 172.21.15.69:0/2388130102 shutdown_connections 2024-04-24T08:15:15.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.602+0000 7f00d7200640 1 -- 172.21.15.69:0/2388130102 >> 172.21.15.69:0/2388130102 conn(0x7f00d00d0350 msgr2=0x7f00d00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:15.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.602+0000 7f00d7200640 1 -- 172.21.15.69:0/2388130102 shutdown_connections 2024-04-24T08:15:15.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.602+0000 7f00d7200640 1 -- 172.21.15.69:0/2388130102 wait complete. 2024-04-24T08:15:15.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.604+0000 7f00d7200640 1 Processor -- start 2024-04-24T08:15:15.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.604+0000 7f00d7200640 1 -- start start 2024-04-24T08:15:15.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.605+0000 7f00d7200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b40061f0 0x7f00d01502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.605+0000 7f00d7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00d006ea90 0x7f00d00b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.605+0000 7f00d7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d014e410 0x7f00d014e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.605+0000 7f00d7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00d0071d00 con 0x7f00d006ea90 2024-04-24T08:15:15.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.605+0000 7f00d7200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f00d00cd470 con 0x7f00b40061f0 2024-04-24T08:15:15.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.605+0000 7f00d7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f00d0072500 con 0x7f00d014e410 2024-04-24T08:15:15.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.605+0000 7f00c7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00d006ea90 0x7f00d00b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.605+0000 7f00c7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00d006ea90 0x7f00d00b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34478/0 (socket says 172.21.15.69:34478) 2024-04-24T08:15:15.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.605+0000 7f00c7400640 1 -- 172.21.15.69:0/1833326175 learned_addr learned my addr 172.21.15.69:0/1833326175 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:15.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00d5200640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d014e410 0x7f00d014e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00c7400640 1 -- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d014e410 msgr2=0x7f00d014e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00cfe00640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b40061f0 0x7f00d01502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00c7400640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d014e410 0x7f00d014e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00c7400640 1 -- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b40061f0 msgr2=0x7f00d01502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00c7400640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b40061f0 0x7f00d01502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00c7400640 1 -- 172.21.15.69:0/1833326175 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d006ef90 con 0x7f00d006ea90 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00c7400640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00d006ea90 0x7f00d00b4610 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f00b802a8d0 tx=0x7f00b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00ccc00640 1 -- 172.21.15.69:0/1833326175 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b8043020 con 0x7f00d006ea90 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00ccc00640 1 -- 172.21.15.69:0/1833326175 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b8035e90 con 0x7f00d006ea90 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00d7200640 1 -- 172.21.15.69:0/1833326175 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d006f270 con 0x7f00d006ea90 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00ccc00640 1 -- 172.21.15.69:0/1833326175 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b803f8c0 con 0x7f00d006ea90 2024-04-24T08:15:15.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.606+0000 7f00d7200640 1 -- 172.21.15.69:0/1833326175 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d006f830 con 0x7f00d006ea90 2024-04-24T08:15:15.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.607+0000 7f00d7200640 1 -- 172.21.15.69:0/1833326175 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00d0068a80 con 0x7f00d006ea90 2024-04-24T08:15:15.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.608+0000 7f00ccc00640 1 -- 172.21.15.69:0/1833326175 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f00b8046020 con 0x7f00d006ea90 2024-04-24T08:15:15.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.609+0000 7f00ccc00640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00ac06c920 0x7f00ac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.609+0000 7f00ccc00640 1 -- 172.21.15.69:0/1833326175 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f00b80ac5c0 con 0x7f00d006ea90 2024-04-24T08:15:15.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.610+0000 7f00cfe00640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00ac06c920 0x7f00ac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.611+0000 7f00cfe00640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00ac06c920 0x7f00ac06ede0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f00c0007c40 tx=0x7f00c0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.614+0000 7f00ccc00640 1 -- 172.21.15.69:0/1833326175 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00b804d030 con 0x7f00d006ea90 2024-04-24T08:15:15.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.813+0000 7f00d7200640 1 -- 172.21.15.69:0/1833326175 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f00d0072990 con 0x7f00d006ea90 2024-04-24T08:15:15.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.814+0000 7f00ccc00640 1 -- 172.21.15.69:0/1833326175 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7f00b807ab60 con 0x7f00d006ea90 2024-04-24T08:15:15.814 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:15:15.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.816+0000 7f00d7200640 1 -- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00ac06c920 msgr2=0x7f00ac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.816+0000 7f00d7200640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00ac06c920 0x7f00ac06ede0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f00c0007c40 tx=0x7f00c0009040 comp rx=0 tx=0).stop 2024-04-24T08:15:15.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.816+0000 7f00d7200640 1 -- 172.21.15.69:0/1833326175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00d006ea90 msgr2=0x7f00d00b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.816+0000 7f00d7200640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00d006ea90 0x7f00d00b4610 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f00b802a8d0 tx=0x7f00b80046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.816+0000 7f00d7200640 1 -- 172.21.15.69:0/1833326175 shutdown_connections 2024-04-24T08:15:15.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.816+0000 7f00d7200640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00ac06c920 0x7f00ac06ede0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.816+0000 7f00d7200640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00d006ea90 0x7f00d00b4610 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.816+0000 7f00d7200640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00d014e410 0x7f00d014e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.816+0000 7f00d7200640 1 --2- 172.21.15.69:0/1833326175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00b40061f0 0x7f00d01502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.816+0000 7f00d7200640 1 -- 172.21.15.69:0/1833326175 >> 172.21.15.69:0/1833326175 conn(0x7f00d00d0350 msgr2=0x7f00d015f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:15.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.817+0000 7f00d7200640 1 -- 172.21.15.69:0/1833326175 shutdown_connections 2024-04-24T08:15:15.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:15.817+0000 7f00d7200640 1 -- 172.21.15.69:0/1833326175 wait complete. 2024-04-24T08:15:15.836 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-04-24T08:15:16.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.031+0000 7f2290c00640 1 Processor -- start 2024-04-24T08:15:16.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.031+0000 7f2290c00640 1 -- start start 2024-04-24T08:15:16.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.033+0000 7f2290c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f228c15ea10 0x7f228c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:16.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.033+0000 7f2290c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f228c1631a0 0x7f228c0af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:16.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.033+0000 7f2290c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f228c0af7d0 0x7f228c071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:16.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.033+0000 7f2290c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f228c0afc40 con 0x7f228c15ea10 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.033+0000 7f2290c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f228c0adab0 con 0x7f228c0af7d0 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.033+0000 7f2290c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f228c0add50 con 0x7f228c1631a0 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.033+0000 7f2289e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f228c15ea10 0x7f228c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.033+0000 7f2289e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f228c15ea10 0x7f228c0d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34484/0 (socket says 172.21.15.69:34484) 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.034+0000 7f2289e00640 1 -- 172.21.15.69:0/1988977328 learned_addr learned my addr 172.21.15.69:0/1988977328 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.034+0000 7f2281400640 1 --2- 172.21.15.69:0/1988977328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f228c1631a0 0x7f228c0af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.034+0000 7f228a800640 1 --2- 172.21.15.69:0/1988977328 >> v2:172.21.15.121:3300/0 conn(0x7f228c0af7d0 0x7f228c071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.034+0000 7f2289e00640 1 -- 172.21.15.69:0/1988977328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f228c1631a0 msgr2=0x7f228c0af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.034+0000 7f2289e00640 1 --2- 172.21.15.69:0/1988977328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f228c1631a0 0x7f228c0af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.034+0000 7f2289e00640 1 -- 172.21.15.69:0/1988977328 >> v2:172.21.15.121:3300/0 conn(0x7f228c0af7d0 msgr2=0x7f228c071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.034+0000 7f2289e00640 1 --2- 172.21.15.69:0/1988977328 >> v2:172.21.15.121:3300/0 conn(0x7f228c0af7d0 0x7f228c071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.034+0000 7f2289e00640 1 -- 172.21.15.69:0/1988977328 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228c0ae030 con 0x7f228c15ea10 2024-04-24T08:15:16.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.035+0000 7f2289e00640 1 --2- 172.21.15.69:0/1988977328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c15ea10 0x7f228c0d0830 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f228c071d00 tx=0x7f227c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86cbdbff48836a85 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:16.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.036+0000 7f2289400640 1 -- 172.21.15.69:0/1988977328 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c037070 con 0x7f228c15ea10 2024-04-24T08:15:16.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.036+0000 7f2289400640 1 -- 172.21.15.69:0/1988977328 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f227c032440 con 0x7f228c15ea10 2024-04-24T08:15:16.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.036+0000 7f2289400640 1 -- 172.21.15.69:0/1988977328 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c032810 con 0x7f228c15ea10 2024-04-24T08:15:16.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.036+0000 7f2290c00640 1 -- 172.21.15.69:0/1988977328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c15ea10 msgr2=0x7f228c0d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:16.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.036+0000 7f2290c00640 1 --2- 172.21.15.69:0/1988977328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c15ea10 0x7f228c0d0830 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f228c071d00 tx=0x7f227c02a1e0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.037+0000 7f2290c00640 1 -- 172.21.15.69:0/1988977328 shutdown_connections 2024-04-24T08:15:16.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.037+0000 7f2290c00640 1 --2- 172.21.15.69:0/1988977328 >> v2:172.21.15.121:3300/0 conn(0x7f228c0af7d0 0x7f228c071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.037+0000 7f2290c00640 1 --2- 172.21.15.69:0/1988977328 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f228c1631a0 0x7f228c0af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.037+0000 7f2290c00640 1 --2- 172.21.15.69:0/1988977328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c15ea10 0x7f228c0d0830 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.037+0000 7f2290c00640 1 -- 172.21.15.69:0/1988977328 >> 172.21.15.69:0/1988977328 conn(0x7f228c067d90 msgr2=0x7f228c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:16.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.037+0000 7f2290c00640 1 -- 172.21.15.69:0/1988977328 shutdown_connections 2024-04-24T08:15:16.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.037+0000 7f2290c00640 1 -- 172.21.15.69:0/1988977328 wait complete. 2024-04-24T08:15:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.038+0000 7f2290c00640 1 Processor -- start 2024-04-24T08:15:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.039+0000 7f2290c00640 1 -- start start 2024-04-24T08:15:16.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f2290c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f228c0af7d0 0x7f228c1f4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f2290c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f228c15ea10 0x7f228c160850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f2290c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c1631a0 0x7f228c0cc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f2290c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f228c0ae8d0 con 0x7f228c1631a0 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f2290c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f228c0ae5f0 con 0x7f228c15ea10 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f2290c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f228c0aedf0 con 0x7f228c0af7d0 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f228a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c1631a0 0x7f228c0cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f228a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c1631a0 0x7f228c0cc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34488/0 (socket says 172.21.15.69:34488) 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f228a800640 1 -- 172.21.15.69:0/2638021219 learned_addr learned my addr 172.21.15.69:0/2638021219 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f228a800640 1 -- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f228c0af7d0 msgr2=0x7f228c1f4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f228a800640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f228c0af7d0 0x7f228c1f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f228a800640 1 -- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f228c15ea10 msgr2=0x7f228c160850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f228a800640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f228c15ea10 0x7f228c160850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.040+0000 7f228a800640 1 -- 172.21.15.69:0/2638021219 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228c161310 con 0x7f228c1631a0 2024-04-24T08:15:16.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.041+0000 7f228a800640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c1631a0 0x7f228c0cc780 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f2284007e60 tx=0x7f228400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:16.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.041+0000 7f2282a00640 1 -- 172.21.15.69:0/2638021219 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284014620 con 0x7f228c1631a0 2024-04-24T08:15:16.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.041+0000 7f2282a00640 1 -- 172.21.15.69:0/2638021219 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2284019070 con 0x7f228c1631a0 2024-04-24T08:15:16.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.041+0000 7f2282a00640 1 -- 172.21.15.69:0/2638021219 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2284014a20 con 0x7f228c1631a0 2024-04-24T08:15:16.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.041+0000 7f2290c00640 1 -- 172.21.15.69:0/2638021219 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f228c1615f0 con 0x7f228c1631a0 2024-04-24T08:15:16.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.041+0000 7f2290c00640 1 -- 172.21.15.69:0/2638021219 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f228c153cf0 con 0x7f228c1631a0 2024-04-24T08:15:16.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.042+0000 7f2290c00640 1 -- 172.21.15.69:0/2638021219 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f228c153ef0 con 0x7f228c1631a0 2024-04-24T08:15:16.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.043+0000 7f2282a00640 1 -- 172.21.15.69:0/2638021219 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2284014bc0 con 0x7f228c1631a0 2024-04-24T08:15:16.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.044+0000 7f2282a00640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f226806c8f0 0x7f226806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:16.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.044+0000 7f2282a00640 1 -- 172.21.15.69:0/2638021219 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7f228408c1d0 con 0x7f228c1631a0 2024-04-24T08:15:16.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.045+0000 7f2289e00640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f226806c8f0 0x7f226806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:16.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.046+0000 7f2289e00640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f226806c8f0 0x7f226806edb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f227c034000 tx=0x7f227c02c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:16.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.048+0000 7f2282a00640 1 -- 172.21.15.69:0/2638021219 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f228c153ef0 con 0x7f228c1631a0 2024-04-24T08:15:16.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.330+0000 7f2290c00640 1 -- 172.21.15.69:0/2638021219 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) -- 0x7f228c068a80 con 0x7f228c1631a0 2024-04-24T08:15:16.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.892+0000 7f2282a00640 1 -- 172.21.15.69:0/2638021219 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v338) ==== 170+0+0 (secure 0 0 0) 0x7f228405a800 con 0x7f228c1631a0 2024-04-24T08:15:16.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 -- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f226806c8f0 msgr2=0x7f226806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:16.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f226806c8f0 0x7f226806edb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f227c034000 tx=0x7f227c02c000 comp rx=0 tx=0).stop 2024-04-24T08:15:16.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 -- 172.21.15.69:0/2638021219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c1631a0 msgr2=0x7f228c0cc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:16.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c1631a0 0x7f228c0cc780 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f2284007e60 tx=0x7f228400e4b0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 -- 172.21.15.69:0/2638021219 shutdown_connections 2024-04-24T08:15:16.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f226806c8f0 0x7f226806edb0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f228c1631a0 0x7f228c0cc780 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f228c15ea10 0x7f228c160850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 --2- 172.21.15.69:0/2638021219 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f228c0af7d0 0x7f228c1f4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:16.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 -- 172.21.15.69:0/2638021219 >> 172.21.15.69:0/2638021219 conn(0x7f228c067d90 msgr2=0x7f228c162110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:16.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 -- 172.21.15.69:0/2638021219 shutdown_connections 2024-04-24T08:15:16.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:16.897+0000 7f2290c00640 1 -- 172.21.15.69:0/2638021219 wait complete. 2024-04-24T08:15:16.921 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-24T08:15:17.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.125+0000 7fcf61c00640 1 Processor -- start 2024-04-24T08:15:17.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.126+0000 7fcf61c00640 1 -- start start 2024-04-24T08:15:17.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.127+0000 7fcf61c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcf5c0d0850 0x7fcf5c0cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:17.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.127+0000 7fcf61c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcf5c0b07a0 0x7fcf5c0cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:17.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.127+0000 7fcf61c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c06bb50 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.127+0000 7fcf61c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcf5c0cdc00 con 0x7fcf5c0b07a0 2024-04-24T08:15:17.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.127+0000 7fcf61c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcf5c0cf320 con 0x7fcf5c0d0850 2024-04-24T08:15:17.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.128+0000 7fcf5b400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcf5c0d0c30 0x7fcf5c0ce110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35920/0 (socket says 172.21.15.69:35920) 2024-04-24T08:15:17.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.128+0000 7fcf5b400640 1 -- 172.21.15.69:0/4065789872 learned_addr learned my addr 172.21.15.69:0/4065789872 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:17.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.128+0000 7fcf5a000640 1 --2- 172.21.15.69:0/4065789872 >> v2:172.21.15.69:3300/0 conn(0x7fcf5c0b07a0 0x7fcf5c0cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:17.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.128+0000 7fcf5aa00640 1 --2- 172.21.15.69:0/4065789872 >> v2:172.21.15.121:3300/0 conn(0x7fcf5c0d0850 0x7fcf5c0cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:17.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.128+0000 7fcf59600640 1 -- 172.21.15.69:0/4065789872 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129547851 0 0) 0x7fcf5c06bb50 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.128+0000 7fcf5a000640 1 -- 172.21.15.69:0/4065789872 >> v2:172.21.15.121:3300/0 conn(0x7fcf5c0d0850 msgr2=0x7fcf5c0cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:17.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.128+0000 7fcf5a000640 1 --2- 172.21.15.69:0/4065789872 >> v2:172.21.15.121:3300/0 conn(0x7fcf5c0d0850 0x7fcf5c0cbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:17.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.128+0000 7fcf5a000640 1 -- 172.21.15.69:0/4065789872 >> v1:172.21.15.69:6789/0 conn(0x7fcf5c0d0c30 legacy=0x7fcf5c0ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:17.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.128+0000 7fcf5a000640 1 -- 172.21.15.69:0/4065789872 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c06be30 con 0x7fcf5c0b07a0 2024-04-24T08:15:17.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.129+0000 7fcf5a000640 1 --2- 172.21.15.69:0/4065789872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0b07a0 0x7fcf5c0cd6c0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0d1420 tx=0x7fcf4402a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d030865102ccdaa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:17.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.129+0000 7fcf59600640 1 -- 172.21.15.69:0/4065789872 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44007740 con 0x7fcf5c0b07a0 2024-04-24T08:15:17.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.129+0000 7fcf59600640 1 -- 172.21.15.69:0/4065789872 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf440078e0 con 0x7fcf5c0b07a0 2024-04-24T08:15:17.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.130+0000 7fcf59600640 1 -- 172.21.15.69:0/4065789872 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44007cb0 con 0x7fcf5c0b07a0 2024-04-24T08:15:17.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.130+0000 7fcf61c00640 1 -- 172.21.15.69:0/4065789872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0b07a0 msgr2=0x7fcf5c0cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:17.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.130+0000 7fcf61c00640 1 --2- 172.21.15.69:0/4065789872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0b07a0 0x7fcf5c0cd6c0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c0d1420 tx=0x7fcf4402a560 comp rx=0 tx=0).stop 2024-04-24T08:15:17.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.130+0000 7fcf61c00640 1 -- 172.21.15.69:0/4065789872 shutdown_connections 2024-04-24T08:15:17.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.130+0000 7fcf61c00640 1 --2- 172.21.15.69:0/4065789872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0b07a0 0x7fcf5c0cd6c0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:17.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.130+0000 7fcf61c00640 1 --2- 172.21.15.69:0/4065789872 >> v2:172.21.15.121:3300/0 conn(0x7fcf5c0d0850 0x7fcf5c0cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:17.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.130+0000 7fcf61c00640 1 -- 172.21.15.69:0/4065789872 >> 172.21.15.69:0/4065789872 conn(0x7fcf5c0b3050 msgr2=0x7fcf5c0b3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:17.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.130+0000 7fcf61c00640 1 -- 172.21.15.69:0/4065789872 shutdown_connections 2024-04-24T08:15:17.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.130+0000 7fcf61c00640 1 -- 172.21.15.69:0/4065789872 wait complete. 2024-04-24T08:15:17.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.132+0000 7fcf61c00640 1 Processor -- start 2024-04-24T08:15:17.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.132+0000 7fcf61c00640 1 -- start start 2024-04-24T08:15:17.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.133+0000 7fcf61c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf5c0b07a0 0x7fcf5c09f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:17.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.133+0000 7fcf61c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf5c0d0850 0x7fcf5c15bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:17.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.133+0000 7fcf61c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0d0c30 0x7fcf5c09b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:17.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.133+0000 7fcf61c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c06c3f0 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.133+0000 7fcf61c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c06c6d0 con 0x7fcf5c0d0850 2024-04-24T08:15:17.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.133+0000 7fcf61c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcf5c0d1080 con 0x7fcf5c0b07a0 2024-04-24T08:15:17.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.133+0000 7fcf5b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0d0c30 0x7fcf5c09b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0d0c30 0x7fcf5c09b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34502/0 (socket says 172.21.15.69:34502) 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5b400640 1 -- 172.21.15.69:0/2025177879 learned_addr learned my addr 172.21.15.69:0/2025177879 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5aa00640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf5c0b07a0 0x7fcf5c09f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5a000640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf5c0d0850 0x7fcf5c15bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5b400640 1 -- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf5c0b07a0 msgr2=0x7fcf5c09f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5b400640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf5c0b07a0 0x7fcf5c09f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5b400640 1 -- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf5c0d0850 msgr2=0x7fcf5c15bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5b400640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf5c0d0850 0x7fcf5c15bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5b400640 1 -- 172.21.15.69:0/2025177879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c1f4960 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5a000640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf5c0d0850 0x7fcf5c15bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.134+0000 7fcf5b400640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0d0c30 0x7fcf5c09b440 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400c450 tx=0x7fcf5400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.135+0000 7fcf52a00640 1 -- 172.21.15.69:0/2025177879 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54018070 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.135+0000 7fcf52a00640 1 -- 172.21.15.69:0/2025177879 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf54007500 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.135+0000 7fcf52a00640 1 -- 172.21.15.69:0/2025177879 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54004a20 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.135+0000 7fcf61c00640 1 -- 172.21.15.69:0/2025177879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5c1f4c40 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.135+0000 7fcf61c00640 1 -- 172.21.15.69:0/2025177879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5c1f5200 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.136+0000 7fcf61c00640 1 -- 172.21.15.69:0/2025177879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf5c068a80 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.137+0000 7fcf52a00640 1 -- 172.21.15.69:0/2025177879 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcf54004bc0 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.138+0000 7fcf52a00640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf3806c8f0 0x7fcf3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:17.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.138+0000 7fcf52a00640 1 -- 172.21.15.69:0/2025177879 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7fcf5408ac30 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.139+0000 7fcf5aa00640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf3806c8f0 0x7fcf3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:17.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.141+0000 7fcf5aa00640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf3806c8f0 0x7fcf3806edb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c006fd0 tx=0x7fcf4c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:17.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.143+0000 7fcf52a00640 1 -- 172.21.15.69:0/2025177879 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcf540591d0 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.348+0000 7fcf61c00640 1 -- 172.21.15.69:0/2025177879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fcf5c15c520 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.354+0000 7fcf52a00640 1 -- 172.21.15.69:0/2025177879 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7fcf54058ad0 con 0x7fcf5c0d0c30 2024-04-24T08:15:17.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.356+0000 7fcf61c00640 1 -- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf3806c8f0 msgr2=0x7fcf3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:17.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.356+0000 7fcf61c00640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf3806c8f0 0x7fcf3806edb0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c006fd0 tx=0x7fcf4c008040 comp rx=0 tx=0).stop 2024-04-24T08:15:17.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.356+0000 7fcf61c00640 1 -- 172.21.15.69:0/2025177879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0d0c30 msgr2=0x7fcf5c09b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:17.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.356+0000 7fcf61c00640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0d0c30 0x7fcf5c09b440 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400c450 tx=0x7fcf5400c910 comp rx=0 tx=0).stop 2024-04-24T08:15:17.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.357+0000 7fcf61c00640 1 -- 172.21.15.69:0/2025177879 shutdown_connections 2024-04-24T08:15:17.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.357+0000 7fcf61c00640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcf3806c8f0 0x7fcf3806edb0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:17.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.357+0000 7fcf61c00640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcf5c0d0c30 0x7fcf5c09b440 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:17.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.357+0000 7fcf61c00640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf5c0d0850 0x7fcf5c15bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:17.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.357+0000 7fcf61c00640 1 --2- 172.21.15.69:0/2025177879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcf5c0b07a0 0x7fcf5c09f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:17.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.357+0000 7fcf61c00640 1 -- 172.21.15.69:0/2025177879 >> 172.21.15.69:0/2025177879 conn(0x7fcf5c0b3050 msgr2=0x7fcf5c0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:17.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.357+0000 7fcf61c00640 1 -- 172.21.15.69:0/2025177879 shutdown_connections 2024-04-24T08:15:17.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:17.357+0000 7fcf61c00640 1 -- 172.21.15.69:0/2025177879 wait complete. 2024-04-24T08:15:17.376 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:15:17.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:15:17.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:15:17.592 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:15:17.592 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:15:17.592 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:15:17.651 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:15:17.652 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:15:17.652 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:15:17.710 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:15:17.710 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:15:17.710 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:15:17.765 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:15:17.765 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:15:17.765 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:15:17.823 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:15:17.823 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-24T08:15:17.823 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:18.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.026+0000 7fa9d7e00640 1 Processor -- start 2024-04-24T08:15:18.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.026+0000 7fa9d7e00640 1 -- start start 2024-04-24T08:15:18.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.028+0000 7fa9d7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d81523f0 0x7fa9d806db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.028+0000 7fa9d7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa9d815b980 0x7fa9d815be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.028+0000 7fa9d7e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9d80aeac0 con 0x7fa9d8072e20 2024-04-24T08:15:18.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.028+0000 7fa9d7e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa9d80aeda0 con 0x7fa9d815b980 2024-04-24T08:15:18.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.028+0000 7fa9d7e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d80af080 con 0x7fa9d81523f0 2024-04-24T08:15:18.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.028+0000 7fa9d7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa9d815b980 0x7fa9d815be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.028+0000 7fa9ce000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa9d8072e20 0x7fa9d8073260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:35928/0 (socket says 172.21.15.69:35928) 2024-04-24T08:15:18.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.028+0000 7fa9ce000640 1 -- 172.21.15.69:0/580374143 learned_addr learned my addr 172.21.15.69:0/580374143 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:18.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.029+0000 7fa9d6a00640 1 --2- 172.21.15.69:0/580374143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d81523f0 0x7fa9d806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.029+0000 7fa9d7400640 1 -- 172.21.15.69:0/580374143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d81523f0 msgr2=0x7fa9d806db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:18.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.029+0000 7fa9d7400640 1 --2- 172.21.15.69:0/580374143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d81523f0 0x7fa9d806db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.029+0000 7fa9d7400640 1 -- 172.21.15.69:0/580374143 >> v1:172.21.15.69:6789/0 conn(0x7fa9d8072e20 legacy=0x7fa9d8073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.029+0000 7fa9d7400640 1 -- 172.21.15.69:0/580374143 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d80af360 con 0x7fa9d815b980 2024-04-24T08:15:18.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.029+0000 7fa9d6000640 1 -- 172.21.15.69:0/580374143 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 692388813 0 0) 0x7fa9d80aeac0 con 0x7fa9d8072e20 2024-04-24T08:15:18.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.030+0000 7fa9d7400640 1 --2- 172.21.15.69:0/580374143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d815b980 0x7fa9d815be10 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00097a0 tx=0x7fa9d002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137bb1b7edb2dcd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:18.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.030+0000 7fa9d6000640 1 -- 172.21.15.69:0/580374143 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0037070 con 0x7fa9d815b980 2024-04-24T08:15:18.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.030+0000 7fa9d6000640 1 -- 172.21.15.69:0/580374143 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d002ae50 con 0x7fa9d815b980 2024-04-24T08:15:18.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.030+0000 7fa9d6000640 1 -- 172.21.15.69:0/580374143 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0032660 con 0x7fa9d815b980 2024-04-24T08:15:18.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.031+0000 7fa9d7e00640 1 -- 172.21.15.69:0/580374143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d815b980 msgr2=0x7fa9d815be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.031+0000 7fa9d7e00640 1 --2- 172.21.15.69:0/580374143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d815b980 0x7fa9d815be10 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00097a0 tx=0x7fa9d002a100 comp rx=0 tx=0).stop 2024-04-24T08:15:18.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.031+0000 7fa9d7e00640 1 -- 172.21.15.69:0/580374143 shutdown_connections 2024-04-24T08:15:18.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.031+0000 7fa9d7e00640 1 --2- 172.21.15.69:0/580374143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d815b980 0x7fa9d815be10 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.031+0000 7fa9d7e00640 1 --2- 172.21.15.69:0/580374143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d81523f0 0x7fa9d806db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.031+0000 7fa9d7e00640 1 -- 172.21.15.69:0/580374143 >> 172.21.15.69:0/580374143 conn(0x7fa9d80d0160 msgr2=0x7fa9d80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:18.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.031+0000 7fa9d7e00640 1 -- 172.21.15.69:0/580374143 shutdown_connections 2024-04-24T08:15:18.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.031+0000 7fa9d7e00640 1 -- 172.21.15.69:0/580374143 wait complete. 2024-04-24T08:15:18.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.033+0000 7fa9d7e00640 1 Processor -- start 2024-04-24T08:15:18.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.033+0000 7fa9d7e00640 1 -- start start 2024-04-24T08:15:18.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.033+0000 7fa9d7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d8072e20 0x7fa9d80b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d81523f0 0x7fa9d80af7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d7e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9d815b980 0x7fa9d80afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d8073970 con 0x7fa9d8072e20 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d7e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d81527d0 con 0x7fa9d815b980 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d7e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d8073af0 con 0x7fa9d81523f0 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d8072e20 0x7fa9d80b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d8072e20 0x7fa9d80b10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34510/0 (socket says 172.21.15.69:34510) 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d6a00640 1 -- 172.21.15.69:0/1630215119 learned_addr learned my addr 172.21.15.69:0/1630215119 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d6a00640 1 -- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d81523f0 msgr2=0x7fa9d80af7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d6a00640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d81523f0 0x7fa9d80af7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d6a00640 1 -- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9d815b980 msgr2=0x7fa9d80afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d6a00640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9d815b980 0x7fa9d80afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.034+0000 7fa9d6a00640 1 -- 172.21.15.69:0/1630215119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d8158900 con 0x7fa9d8072e20 2024-04-24T08:15:18.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.035+0000 7fa9d6a00640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d8072e20 0x7fa9d80b10c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800ce10 tx=0x7fa9c800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:18.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.035+0000 7fa9cf400640 1 -- 172.21.15.69:0/1630215119 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8003b90 con 0x7fa9d8072e20 2024-04-24T08:15:18.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.035+0000 7fa9cf400640 1 -- 172.21.15.69:0/1630215119 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9c8003d30 con 0x7fa9d8072e20 2024-04-24T08:15:18.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.035+0000 7fa9cf400640 1 -- 172.21.15.69:0/1630215119 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c8004240 con 0x7fa9d8072e20 2024-04-24T08:15:18.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.035+0000 7fa9d7e00640 1 -- 172.21.15.69:0/1630215119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d8158be0 con 0x7fa9d8072e20 2024-04-24T08:15:18.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.035+0000 7fa9d7e00640 1 -- 172.21.15.69:0/1630215119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d813f680 con 0x7fa9d8072e20 2024-04-24T08:15:18.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.036+0000 7fa9d7e00640 1 -- 172.21.15.69:0/1630215119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d813f880 con 0x7fa9d8072e20 2024-04-24T08:15:18.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.037+0000 7fa9cf400640 1 -- 172.21.15.69:0/1630215119 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa9c80043e0 con 0x7fa9d8072e20 2024-04-24T08:15:18.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.038+0000 7fa9cf400640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9b406c8f0 0x7fa9b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.038+0000 7fa9cf400640 1 -- 172.21.15.69:0/1630215119 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7fa9c808c460 con 0x7fa9d8072e20 2024-04-24T08:15:18.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.039+0000 7fa9ce000640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9b406c8f0 0x7fa9b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.040+0000 7fa9ce000640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9b406c8f0 0x7fa9b406edb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0007fd0 tx=0x7fa9c0007920 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:18.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.042+0000 7fa9cf400640 1 -- 172.21.15.69:0/1630215119 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9d813f880 con 0x7fa9d8072e20 2024-04-24T08:15:18.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.280+0000 7fa9d7e00640 1 -- 172.21.15.69:0/1630215119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9d8075d50 con 0x7fa9d8072e20 2024-04-24T08:15:18.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.281+0000 7fa9cf400640 1 -- 172.21.15.69:0/1630215119 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+3869 (secure 0 0 0) 0x7fa9c801d340 con 0x7fa9d8072e20 2024-04-24T08:15:18.282 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:18.282 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":54,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3504199143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3504199143},{"type":"v1","addr":"172.21.15.121:6837","nonce":3504199143}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46},{"gid":5401,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/1624482680","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1624482680},{"type":"v1","addr":"172.21.15.121:6836","nonce":1624482680}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39},{"gid":6789,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/417784626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":417784626},{"type":"v1","addr":"172.21.15.69:6833","nonce":417784626}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":38},{"gid":7497,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/163576202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":163576202},{"type":"v1","addr":"172.21.15.69:6835","nonce":163576202}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":54}],"filesystems":[]} 2024-04-24T08:15:18.282 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 54 2024-04-24T08:15:18.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.283+0000 7fa9d7e00640 1 -- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9b406c8f0 msgr2=0x7fa9b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.283+0000 7fa9d7e00640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9b406c8f0 0x7fa9b406edb0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0007fd0 tx=0x7fa9c0007920 comp rx=0 tx=0).stop 2024-04-24T08:15:18.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.283+0000 7fa9d7e00640 1 -- 172.21.15.69:0/1630215119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d8072e20 msgr2=0x7fa9d80b10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.283+0000 7fa9d7e00640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d8072e20 0x7fa9d80b10c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800ce10 tx=0x7fa9c800a660 comp rx=0 tx=0).stop 2024-04-24T08:15:18.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.283+0000 7fa9d7e00640 1 -- 172.21.15.69:0/1630215119 shutdown_connections 2024-04-24T08:15:18.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.283+0000 7fa9d7e00640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa9b406c8f0 0x7fa9b406edb0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.284+0000 7fa9d7e00640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9d815b980 0x7fa9d80afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.284+0000 7fa9d7e00640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9d81523f0 0x7fa9d80af7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.284+0000 7fa9d7e00640 1 --2- 172.21.15.69:0/1630215119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9d8072e20 0x7fa9d80b10c0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.284+0000 7fa9d7e00640 1 -- 172.21.15.69:0/1630215119 >> 172.21.15.69:0/1630215119 conn(0x7fa9d80d0160 msgr2=0x7fa9d80b3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:18.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.284+0000 7fa9d7e00640 1 -- 172.21.15.69:0/1630215119 shutdown_connections 2024-04-24T08:15:18.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.284+0000 7fa9d7e00640 1 -- 172.21.15.69:0/1630215119 wait complete. 2024-04-24T08:15:18.306 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T08:15:18.306 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-24T08:15:18.307 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T08:15:18.307 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:15:18.307 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T08:15:18.307 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-24T08:15:18.307 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T08:15:18.307 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:15:18.307 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T08:15:18.308 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-24T08:15:18.308 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T08:15:18.308 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:15:18.308 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T08:15:18.308 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-24T08:15:18.308 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T08:15:18.308 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:15:18.326 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T08:15:18.326 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T08:15:18.327 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T08:15:18.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T08:15:18.331 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T08:15:18.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T08:15:18.333 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T08:15:18.333 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T08:15:18.374 INFO:tasks.ceph.mds.a:Started 2024-04-24T08:15:18.374 DEBUG:teuthology.parallel:result is None 2024-04-24T08:15:18.379 INFO:tasks.ceph.mds.b:Started 2024-04-24T08:15:18.379 DEBUG:teuthology.parallel:result is None 2024-04-24T08:15:18.381 INFO:tasks.ceph.mds.c:Started 2024-04-24T08:15:18.381 DEBUG:teuthology.parallel:result is None 2024-04-24T08:15:18.385 INFO:tasks.ceph.mds.d:Started 2024-04-24T08:15:18.385 DEBUG:teuthology.parallel:result is None 2024-04-24T08:15:18.385 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-24T08:15:18.519 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:15:18.525 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:15:18.533 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-24T08:15:18.533 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:15:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.561+0000 7ff284400640 1 Processor -- start 2024-04-24T08:15:18.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.561+0000 7ff284400640 1 -- start start 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.562+0000 7ff284400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c162c70 0x7ff27c0b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff284400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff27c0b3d30 0x7ff27c0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff284400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff27c071d70 0x7ff27c0721a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff284400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff27c0b46d0 con 0x7ff27c071d70 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff284400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff27c1669d0 con 0x7ff27c0b3d30 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff284400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff27c166cb0 con 0x7ff27c162c70 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff282400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff27c071d70 0x7ff27c0721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff282400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff27c071d70 0x7ff27c0721a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34552/0 (socket says 172.21.15.69:34552) 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff282400640 1 -- 172.21.15.69:0/233860452 learned_addr learned my addr 172.21.15.69:0/233860452 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff281000640 1 --2- 172.21.15.69:0/233860452 >> v2:172.21.15.121:3300/0 conn(0x7ff27c0b3d30 0x7ff27c0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff281a00640 1 --2- 172.21.15.69:0/233860452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c162c70 0x7ff27c0b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff282400640 1 -- 172.21.15.69:0/233860452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c162c70 msgr2=0x7ff27c0b3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff282400640 1 --2- 172.21.15.69:0/233860452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c162c70 0x7ff27c0b3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff282400640 1 -- 172.21.15.69:0/233860452 >> v2:172.21.15.121:3300/0 conn(0x7ff27c0b3d30 msgr2=0x7ff27c0b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff282400640 1 --2- 172.21.15.69:0/233860452 >> v2:172.21.15.121:3300/0 conn(0x7ff27c0b3d30 0x7ff27c0b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff282400640 1 -- 172.21.15.69:0/233860452 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27c166f90 con 0x7ff27c071d70 2024-04-24T08:15:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff281a00640 1 --2- 172.21.15.69:0/233860452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c162c70 0x7ff27c0b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.563+0000 7ff281000640 1 --2- 172.21.15.69:0/233860452 >> v2:172.21.15.121:3300/0 conn(0x7ff27c0b3d30 0x7ff27c0b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.564+0000 7ff282400640 1 --2- 172.21.15.69:0/233860452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c071d70 0x7ff27c0721a0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7ff27c073210 tx=0x7ff27402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b64fb0cf7a8612ad server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff27be00640 1 -- 172.21.15.69:0/233860452 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff274036070 con 0x7ff27c071d70 2024-04-24T08:15:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff27be00640 1 -- 172.21.15.69:0/233860452 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff274031600 con 0x7ff27c071d70 2024-04-24T08:15:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff27be00640 1 -- 172.21.15.69:0/233860452 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2740319d0 con 0x7ff27c071d70 2024-04-24T08:15:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff284400640 1 -- 172.21.15.69:0/233860452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c071d70 msgr2=0x7ff27c0721a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff284400640 1 --2- 172.21.15.69:0/233860452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c071d70 0x7ff27c0721a0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7ff27c073210 tx=0x7ff27402a6a0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff284400640 1 -- 172.21.15.69:0/233860452 shutdown_connections 2024-04-24T08:15:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff284400640 1 --2- 172.21.15.69:0/233860452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c071d70 0x7ff27c0721a0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff284400640 1 --2- 172.21.15.69:0/233860452 >> v2:172.21.15.121:3300/0 conn(0x7ff27c0b3d30 0x7ff27c0b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff284400640 1 --2- 172.21.15.69:0/233860452 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c162c70 0x7ff27c0b3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff284400640 1 -- 172.21.15.69:0/233860452 >> 172.21.15.69:0/233860452 conn(0x7ff27c067d90 msgr2=0x7ff27c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff284400640 1 -- 172.21.15.69:0/233860452 shutdown_connections 2024-04-24T08:15:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.565+0000 7ff284400640 1 -- 172.21.15.69:0/233860452 wait complete. 2024-04-24T08:15:18.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.567+0000 7ff284400640 1 Processor -- start 2024-04-24T08:15:18.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.567+0000 7ff284400640 1 -- start start 2024-04-24T08:15:18.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff284400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27c071d70 0x7ff27c0b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff284400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c0b3d30 0x7ff27c14ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff284400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c162c70 0x7ff27c14f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff284400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c163050 con 0x7ff27c162c70 2024-04-24T08:15:18.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff284400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff27c167550 con 0x7ff27c071d70 2024-04-24T08:15:18.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff284400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff27c0cfc40 con 0x7ff27c0b3d30 2024-04-24T08:15:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff282400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c162c70 0x7ff27c14f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff282400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c162c70 0x7ff27c14f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34560/0 (socket says 172.21.15.69:34560) 2024-04-24T08:15:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff282400640 1 -- 172.21.15.69:0/2967152832 learned_addr learned my addr 172.21.15.69:0/2967152832 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff281a00640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27c071d70 0x7ff27c0b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.568+0000 7ff281000640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c0b3d30 0x7ff27c14ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.569+0000 7ff282400640 1 -- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c0b3d30 msgr2=0x7ff27c14ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.569+0000 7ff282400640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c0b3d30 0x7ff27c14ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.569+0000 7ff282400640 1 -- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27c071d70 msgr2=0x7ff27c0b1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.569+0000 7ff282400640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27c071d70 0x7ff27c0b1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.569+0000 7ff282400640 1 -- 172.21.15.69:0/2967152832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff27c163370 con 0x7ff27c162c70 2024-04-24T08:15:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.569+0000 7ff282400640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c162c70 0x7ff27c14f240 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff27402ac90 tx=0x7ff274004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.569+0000 7ff279600640 1 -- 172.21.15.69:0/2967152832 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27403f020 con 0x7ff27c162c70 2024-04-24T08:15:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.570+0000 7ff279600640 1 -- 172.21.15.69:0/2967152832 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff274037040 con 0x7ff27c162c70 2024-04-24T08:15:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.570+0000 7ff279600640 1 -- 172.21.15.69:0/2967152832 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff274036070 con 0x7ff27c162c70 2024-04-24T08:15:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.570+0000 7ff284400640 1 -- 172.21.15.69:0/2967152832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff27c163650 con 0x7ff27c162c70 2024-04-24T08:15:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.570+0000 7ff284400640 1 -- 172.21.15.69:0/2967152832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff27c163c10 con 0x7ff27c162c70 2024-04-24T08:15:18.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.571+0000 7ff284400640 1 -- 172.21.15.69:0/2967152832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff27c068a80 con 0x7ff27c162c70 2024-04-24T08:15:18.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.572+0000 7ff279600640 1 -- 172.21.15.69:0/2967152832 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff274042020 con 0x7ff27c162c70 2024-04-24T08:15:18.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.573+0000 7ff279600640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff25c06c8f0 0x7ff25c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:18.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.573+0000 7ff279600640 1 -- 172.21.15.69:0/2967152832 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 5029+0+0 (secure 0 0 0) 0x7ff2740abdb0 con 0x7ff27c162c70 2024-04-24T08:15:18.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.573+0000 7ff281a00640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff25c06c8f0 0x7ff25c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:18.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.574+0000 7ff281a00640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff25c06c8f0 0x7ff25c06edb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff27000aae0 tx=0x7ff270008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.577+0000 7ff279600640 1 -- 172.21.15.69:0/2967152832 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff2740b0070 con 0x7ff27c162c70 2024-04-24T08:15:18.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.788+0000 7ff284400640 1 -- 172.21.15.69:0/2967152832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7ff27c06ddd0 con 0x7ff27c162c70 2024-04-24T08:15:18.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.920+0000 7ff279600640 1 -- 172.21.15.69:0/2967152832 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v76) ==== 97+0+0 (secure 0 0 0) 0x7ff274049030 con 0x7ff27c162c70 2024-04-24T08:15:18.921 INFO:teuthology.orchestra.run.smithi069.stderr: removed all blocklist entries 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.924+0000 7ff284400640 1 -- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff25c06c8f0 msgr2=0x7ff25c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.924+0000 7ff284400640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff25c06c8f0 0x7ff25c06edb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff27000aae0 tx=0x7ff270008040 comp rx=0 tx=0).stop 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.924+0000 7ff284400640 1 -- 172.21.15.69:0/2967152832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c162c70 msgr2=0x7ff27c14f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.924+0000 7ff284400640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c162c70 0x7ff27c14f240 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7ff27402ac90 tx=0x7ff274004a40 comp rx=0 tx=0).stop 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.925+0000 7ff284400640 1 -- 172.21.15.69:0/2967152832 shutdown_connections 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.925+0000 7ff284400640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff25c06c8f0 0x7ff25c06edb0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.925+0000 7ff284400640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff27c162c70 0x7ff27c14f240 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.925+0000 7ff284400640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff27c0b3d30 0x7ff27c14ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.925+0000 7ff284400640 1 --2- 172.21.15.69:0/2967152832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff27c071d70 0x7ff27c0b1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.925+0000 7ff284400640 1 -- 172.21.15.69:0/2967152832 >> 172.21.15.69:0/2967152832 conn(0x7ff27c067d90 msgr2=0x7ff27c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.925+0000 7ff284400640 1 -- 172.21.15.69:0/2967152832 shutdown_connections 2024-04-24T08:15:18.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:18.925+0000 7ff284400640 1 -- 172.21.15.69:0/2967152832 wait complete. 2024-04-24T08:15:18.946 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-24T08:15:19.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.136+0000 7f1f36800640 1 Processor -- start 2024-04-24T08:15:19.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.137+0000 7f1f36800640 1 -- start start 2024-04-24T08:15:19.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.138+0000 7f1f36800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1f300d0860 0x7f1f300cd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.138+0000 7f1f36800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f300cd740 0x7f1f300c6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.138+0000 7f1f36800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f300cdf10 con 0x7f1f3015cac0 2024-04-24T08:15:19.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.138+0000 7f1f36800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1f300c6170 con 0x7f1f300cd740 2024-04-24T08:15:19.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.138+0000 7f1f36800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1f300ce1b0 con 0x7f1f300d0860 2024-04-24T08:15:19.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.138+0000 7f1f2fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1f300cd740 0x7f1f300c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:19.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.138+0000 7f1f2f400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1f3015cac0 0x7f1f300d0150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47648/0 (socket says 172.21.15.69:47648) 2024-04-24T08:15:19.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.138+0000 7f1f2f400640 1 -- 172.21.15.69:0/1802458943 learned_addr learned my addr 172.21.15.69:0/1802458943 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.139+0000 7f1f2fe00640 1 -- 172.21.15.69:0/1802458943 >> v2:172.21.15.121:3300/0 conn(0x7f1f300d0860 msgr2=0x7f1f300cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.139+0000 7f1f2ea00640 1 --2- 172.21.15.69:0/1802458943 >> v2:172.21.15.121:3300/0 conn(0x7f1f300d0860 0x7f1f300cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.139+0000 7f1f2fe00640 1 --2- 172.21.15.69:0/1802458943 >> v2:172.21.15.121:3300/0 conn(0x7f1f300d0860 0x7f1f300cd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.139+0000 7f1f2fe00640 1 -- 172.21.15.69:0/1802458943 >> v1:172.21.15.69:6789/0 conn(0x7f1f3015cac0 legacy=0x7f1f300d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.139+0000 7f1f2fe00640 1 -- 172.21.15.69:0/1802458943 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f300c6410 con 0x7f1f300cd740 2024-04-24T08:15:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.139+0000 7f1f2e000640 1 -- 172.21.15.69:0/1802458943 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2968023693 0 0) 0x7f1f300cdf10 con 0x7f1f3015cac0 2024-04-24T08:15:19.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.140+0000 7f1f2fe00640 1 --2- 172.21.15.69:0/1802458943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f300cd740 0x7f1f300c6930 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f1f300b4e30 tx=0x7f1f2802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb176b67b0be633a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:19.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.140+0000 7f1f2e000640 1 -- 172.21.15.69:0/1802458943 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28037070 con 0x7f1f300cd740 2024-04-24T08:15:19.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.140+0000 7f1f2e000640 1 -- 172.21.15.69:0/1802458943 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f28032440 con 0x7f1f300cd740 2024-04-24T08:15:19.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.140+0000 7f1f2e000640 1 -- 172.21.15.69:0/1802458943 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28032810 con 0x7f1f300cd740 2024-04-24T08:15:19.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.141+0000 7f1f36800640 1 -- 172.21.15.69:0/1802458943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f300cd740 msgr2=0x7f1f300c6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.141+0000 7f1f36800640 1 --2- 172.21.15.69:0/1802458943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f300cd740 0x7f1f300c6930 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f1f300b4e30 tx=0x7f1f2802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.141+0000 7f1f36800640 1 -- 172.21.15.69:0/1802458943 shutdown_connections 2024-04-24T08:15:19.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.141+0000 7f1f36800640 1 --2- 172.21.15.69:0/1802458943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f300cd740 0x7f1f300c6930 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.141+0000 7f1f36800640 1 --2- 172.21.15.69:0/1802458943 >> v2:172.21.15.121:3300/0 conn(0x7f1f300d0860 0x7f1f300cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.141+0000 7f1f36800640 1 -- 172.21.15.69:0/1802458943 >> 172.21.15.69:0/1802458943 conn(0x7f1f300b4070 msgr2=0x7f1f300b4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:19.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.141+0000 7f1f36800640 1 -- 172.21.15.69:0/1802458943 shutdown_connections 2024-04-24T08:15:19.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.141+0000 7f1f36800640 1 -- 172.21.15.69:0/1802458943 wait complete. 2024-04-24T08:15:19.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.142+0000 7f1f36800640 1 Processor -- start 2024-04-24T08:15:19.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.143+0000 7f1f36800640 1 -- start start 2024-04-24T08:15:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.143+0000 7f1f36800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f300cd740 0x7f1f300b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f36800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f300d0860 0x7f1f301508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f36800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f3015cac0 0x7f1f300cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f36800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1f300c6e70 con 0x7f1f3015cac0 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f36800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f300cdb90 con 0x7f1f300cd740 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f36800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f300c6ff0 con 0x7f1f300d0860 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f2fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f3015cac0 0x7f1f300cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f2fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f3015cac0 0x7f1f300cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53650/0 (socket says 172.21.15.69:53650) 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f2fe00640 1 -- 172.21.15.69:0/74634073 learned_addr learned my addr 172.21.15.69:0/74634073 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f2fe00640 1 -- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f300d0860 msgr2=0x7f1f301508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f2fe00640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f300d0860 0x7f1f301508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f2fe00640 1 -- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f300cd740 msgr2=0x7f1f300b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f2fe00640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f300cd740 0x7f1f300b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.144+0000 7f1f2fe00640 1 -- 172.21.15.69:0/74634073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f300b56e0 con 0x7f1f3015cac0 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.145+0000 7f1f2fe00640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f3015cac0 0x7f1f300cbdf0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f1f28035040 tx=0x7f1f280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:19.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.145+0000 7f1f27400640 1 -- 172.21.15.69:0/74634073 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28037040 con 0x7f1f3015cac0 2024-04-24T08:15:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.145+0000 7f1f27400640 1 -- 172.21.15.69:0/74634073 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f28042070 con 0x7f1f3015cac0 2024-04-24T08:15:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.145+0000 7f1f36800640 1 -- 172.21.15.69:0/74634073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f30157d30 con 0x7f1f3015cac0 2024-04-24T08:15:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.145+0000 7f1f27400640 1 -- 172.21.15.69:0/74634073 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f28030b50 con 0x7f1f3015cac0 2024-04-24T08:15:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.145+0000 7f1f36800640 1 -- 172.21.15.69:0/74634073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f3015e430 con 0x7f1f3015cac0 2024-04-24T08:15:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.146+0000 7f1f36800640 1 -- 172.21.15.69:0/74634073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f30069190 con 0x7f1f3015cac0 2024-04-24T08:15:19.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.147+0000 7f1f27400640 1 -- 172.21.15.69:0/74634073 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f28030cf0 con 0x7f1f3015cac0 2024-04-24T08:15:19.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.148+0000 7f1f27400640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f0c06c8f0 0x7f1f0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.148+0000 7f1f27400640 1 -- 172.21.15.69:0/74634073 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 4857+0+0 (secure 0 0 0) 0x7f1f280ac4f0 con 0x7f1f3015cac0 2024-04-24T08:15:19.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.149+0000 7f1f2f400640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f0c06c8f0 0x7f1f0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:19.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.150+0000 7f1f2f400640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f0c06c8f0 0x7f1f0c06edb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1f20007fd0 tx=0x7f1f20007960 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:19.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.152+0000 7f1f27400640 1 -- 172.21.15.69:0/74634073 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f2807abd0 con 0x7f1f3015cac0 2024-04-24T08:15:19.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.421+0000 7f1f36800640 1 -- 172.21.15.69:0/74634073 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f1f300ce900 con 0x7f1f3015cac0 2024-04-24T08:15:19.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.423+0000 7f1f27400640 1 -- 172.21.15.69:0/74634073 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) ==== 80+0+5304 (secure 0 0 0) 0x7f1f2807a4d0 con 0x7f1f3015cac0 2024-04-24T08:15:19.423 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:19.423 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:15:19.423 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_dump": [ 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.a", 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmsqMeLRAApLPtHjtq78HzlVcsjfqXOg==", 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.b", 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmkyd5ChAAGsJGU6GGMvvGzQnABuDx/g==", 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.424 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.c", 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmS+yLORAAHPCeBtq99IXEVTpcMT3c4A==", 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.425 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.d", 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShm0xmHFRAA3VfYoqbcV8kWF+4fCZMVLA==", 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.426 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.0", 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC5vShmMbvcERAANgogiCC/sXf1y0Ykv7ecNQ==", 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.1", 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC8vShmDD+uLBAAJHXmxEm1cumJ3wjqe6S0ww==", 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.427 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.2", 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC/vShmS63kNhAAXec5HVuwo8THKstXiQOL8Q==", 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.428 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.429 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.3", 2024-04-24T08:15:19.429 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDDvShmxcDJBhAA3V3I4C63EHPRUyhzBCZEzw==", 2024-04-24T08:15:19.429 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.429 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:15:19.429 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:15:19.429 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.430 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.430 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.430 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.430 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.4", 2024-04-24T08:15:19.430 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDJvShm1pX9ERAAL2gjv/+M915Cf5OoqBhZVw==", 2024-04-24T08:15:19.430 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.430 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:15:19.430 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:15:19.430 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.430 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.5", 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDMvShmNIxbHRAA23gxiaGvagddVc2mE5DJJw==", 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.6", 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDPvShmb+x4NhAA6aWmAzU6c0AZomVgWcFwig==", 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.431 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.7", 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDTvShmgcKMBRAA6fkJvNxBjlNhK9CrZ/FMSw==", 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.432 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.0", 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw==", 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow r", 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.1", 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmKAIHMhAAuTuFmTHvBteErsG23LMEyQ==", 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:15:19.433 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.2", 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmW0MnKRAASzxCAKuApVvfu9kUfK9BJw==", 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.434 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.3", 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC2vShmfgoaABAAniS9oubv92R7Qe2EITGKxg==", 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.admin", 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQCzvShmLh6yLxAAoH7NhoEAUKcdXp6n+ai2Ug==", 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.435 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.x", 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmv4rwIxAA3rp+yy09qrs7HXip8bZ8xA==", 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.436 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.y", 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShm2eatGxAAFQkL0GBsujypv4VnmSy5bw==", 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-24T08:15:19.437 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:15:19.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.425+0000 7f1f36800640 1 -- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f0c06c8f0 msgr2=0x7f1f0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.425+0000 7f1f36800640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f0c06c8f0 0x7f1f0c06edb0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1f20007fd0 tx=0x7f1f20007960 comp rx=0 tx=0).stop 2024-04-24T08:15:19.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.425+0000 7f1f36800640 1 -- 172.21.15.69:0/74634073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f3015cac0 msgr2=0x7f1f300cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.425+0000 7f1f36800640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f3015cac0 0x7f1f300cbdf0 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f1f28035040 tx=0x7f1f280046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.425+0000 7f1f36800640 1 -- 172.21.15.69:0/74634073 shutdown_connections 2024-04-24T08:15:19.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.425+0000 7f1f36800640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1f0c06c8f0 0x7f1f0c06edb0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.425+0000 7f1f36800640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f3015cac0 0x7f1f300cbdf0 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.425+0000 7f1f36800640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f300d0860 0x7f1f301508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.425+0000 7f1f36800640 1 --2- 172.21.15.69:0/74634073 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f300cd740 0x7f1f300b51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.425+0000 7f1f36800640 1 -- 172.21.15.69:0/74634073 >> 172.21.15.69:0/74634073 conn(0x7f1f300b4070 msgr2=0x7f1f3009a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:19.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.426+0000 7f1f36800640 1 -- 172.21.15.69:0/74634073 shutdown_connections 2024-04-24T08:15:19.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.426+0000 7f1f36800640 1 -- 172.21.15.69:0/74634073 wait complete. 2024-04-24T08:15:19.444 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:19.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.626+0000 7f4e8fe00640 1 Processor -- start 2024-04-24T08:15:19.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.627+0000 7f4e8fe00640 1 -- start start 2024-04-24T08:15:19.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.628+0000 7f4e8fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e90071e50 0x7f4e9015ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.628+0000 7f4e8fe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4e90162300 0x7f4e9006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.628+0000 7f4e8fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4e90072230 0x7f4e9006e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.628+0000 7f4e8fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9015f540 con 0x7f4e90072230 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.628+0000 7f4e8fe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9015f820 con 0x7f4e90162300 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.628+0000 7f4e8fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9015fb00 con 0x7f4e90071e50 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.628+0000 7f4e8f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4e90072230 0x7f4e9006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.628+0000 7f4e8f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4e90072230 0x7f4e9006e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53658/0 (socket says 172.21.15.69:53658) 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.628+0000 7f4e8f400640 1 -- 172.21.15.69:0/3701734439 learned_addr learned my addr 172.21.15.69:0/3701734439 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.629+0000 7f4e8ea00640 1 --2- 172.21.15.69:0/3701734439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e90071e50 0x7f4e9015ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.629+0000 7f4e87e00640 1 --2- 172.21.15.69:0/3701734439 >> v2:172.21.15.121:3300/0 conn(0x7f4e90162300 0x7f4e9006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.629+0000 7f4e8f400640 1 -- 172.21.15.69:0/3701734439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e90071e50 msgr2=0x7f4e9015ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.629+0000 7f4e8f400640 1 --2- 172.21.15.69:0/3701734439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e90071e50 0x7f4e9015ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.629+0000 7f4e8f400640 1 -- 172.21.15.69:0/3701734439 >> v2:172.21.15.121:3300/0 conn(0x7f4e90162300 msgr2=0x7f4e9006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.629+0000 7f4e8f400640 1 --2- 172.21.15.69:0/3701734439 >> v2:172.21.15.121:3300/0 conn(0x7f4e90162300 0x7f4e9006e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.629+0000 7f4e8f400640 1 -- 172.21.15.69:0/3701734439 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9006dda0 con 0x7f4e90072230 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.630+0000 7f4e8f400640 1 --2- 172.21.15.69:0/3701734439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90072230 0x7f4e9006e800 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f4e9006d930 tx=0x7f4e8802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa7337263aa2c3e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.630+0000 7f4e8e000640 1 -- 172.21.15.69:0/3701734439 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88036070 con 0x7f4e90072230 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.630+0000 7f4e8e000640 1 -- 172.21.15.69:0/3701734439 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e88031600 con 0x7f4e90072230 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.630+0000 7f4e8e000640 1 -- 172.21.15.69:0/3701734439 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e880319d0 con 0x7f4e90072230 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.631+0000 7f4e8fe00640 1 -- 172.21.15.69:0/3701734439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90072230 msgr2=0x7f4e9006e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.631+0000 7f4e8fe00640 1 --2- 172.21.15.69:0/3701734439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90072230 0x7f4e9006e800 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f4e9006d930 tx=0x7f4e8802a6a0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.631+0000 7f4e8fe00640 1 -- 172.21.15.69:0/3701734439 shutdown_connections 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.631+0000 7f4e8fe00640 1 --2- 172.21.15.69:0/3701734439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90072230 0x7f4e9006e800 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.631+0000 7f4e8fe00640 1 --2- 172.21.15.69:0/3701734439 >> v2:172.21.15.121:3300/0 conn(0x7f4e90162300 0x7f4e9006e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.631+0000 7f4e8fe00640 1 --2- 172.21.15.69:0/3701734439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e90071e50 0x7f4e9015ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.631+0000 7f4e8fe00640 1 -- 172.21.15.69:0/3701734439 >> 172.21.15.69:0/3701734439 conn(0x7f4e900d0160 msgr2=0x7f4e900d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:19.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.631+0000 7f4e8fe00640 1 -- 172.21.15.69:0/3701734439 shutdown_connections 2024-04-24T08:15:19.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.631+0000 7f4e8fe00640 1 -- 172.21.15.69:0/3701734439 wait complete. 2024-04-24T08:15:19.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.633+0000 7f4e8fe00640 1 Processor -- start 2024-04-24T08:15:19.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.633+0000 7f4e8fe00640 1 -- start start 2024-04-24T08:15:19.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.634+0000 7f4e8fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90071e50 0x7f4e901f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.634+0000 7f4e8fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e90072230 0x7f4e901f1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e90162300 0x7f4e9014a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9006eec0 con 0x7f4e90071e50 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8fe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9006ed40 con 0x7f4e90072230 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9006f040 con 0x7f4e90162300 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90071e50 0x7f4e901f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90071e50 0x7f4e901f42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53666/0 (socket says 172.21.15.69:53666) 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8ea00640 1 -- 172.21.15.69:0/845639779 learned_addr learned my addr 172.21.15.69:0/845639779 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8ea00640 1 -- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e90162300 msgr2=0x7f4e9014a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8ea00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e90162300 0x7f4e9014a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8ea00640 1 -- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e90072230 msgr2=0x7f4e901f1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8ea00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e90072230 0x7f4e901f1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8ea00640 1 -- 172.21.15.69:0/845639779 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9014ac00 con 0x7f4e90071e50 2024-04-24T08:15:19.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.635+0000 7f4e8ea00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90071e50 0x7f4e901f42f0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000d800 tx=0x7f4e8000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:19.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.636+0000 7f4e86a00640 1 -- 172.21.15.69:0/845639779 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8000b6c0 con 0x7f4e90071e50 2024-04-24T08:15:19.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.636+0000 7f4e86a00640 1 -- 172.21.15.69:0/845639779 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e80019070 con 0x7f4e90071e50 2024-04-24T08:15:19.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.636+0000 7f4e86a00640 1 -- 172.21.15.69:0/845639779 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8000ba80 con 0x7f4e90071e50 2024-04-24T08:15:19.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.636+0000 7f4e8fe00640 1 -- 172.21.15.69:0/845639779 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e901f4920 con 0x7f4e90071e50 2024-04-24T08:15:19.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.636+0000 7f4e8fe00640 1 -- 172.21.15.69:0/845639779 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e901f4ee0 con 0x7f4e90071e50 2024-04-24T08:15:19.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.637+0000 7f4e8fe00640 1 -- 172.21.15.69:0/845639779 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e90075d50 con 0x7f4e90071e50 2024-04-24T08:15:19.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.638+0000 7f4e86a00640 1 -- 172.21.15.69:0/845639779 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4e8000bc20 con 0x7f4e90071e50 2024-04-24T08:15:19.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.639+0000 7f4e86a00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e6c06c8f0 0x7f4e6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:19.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.640+0000 7f4e86a00640 1 -- 172.21.15.69:0/845639779 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 4857+0+0 (secure 0 0 0) 0x7f4e8008bd90 con 0x7f4e90071e50 2024-04-24T08:15:19.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.640+0000 7f4e87e00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e6c06c8f0 0x7f4e6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:19.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.641+0000 7f4e87e00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e6c06c8f0 0x7f4e6c06edb0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4e90147af0 tx=0x7f4e78008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:19.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.646+0000 7f4e86a00640 1 -- 172.21.15.69:0/845639779 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e9014ac00 con 0x7f4e90071e50 2024-04-24T08:15:19.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.837+0000 7f4e8fe00640 1 -- 172.21.15.69:0/845639779 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e900d0d90 con 0x7f4e90071e50 2024-04-24T08:15:19.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.839+0000 7f4e86a00640 1 -- 172.21.15.69:0/845639779 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) ==== 74+0+12868 (secure 0 0 0) 0x7f4e8001c360 con 0x7f4e90071e50 2024-04-24T08:15:19.839 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:19.839 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":78,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:15:18.934652+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.841+0000 7f4e8fe00640 1 -- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e6c06c8f0 msgr2=0x7f4e6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.841+0000 7f4e8fe00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e6c06c8f0 0x7f4e6c06edb0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4e90147af0 tx=0x7f4e78008040 comp rx=0 tx=0).stop 2024-04-24T08:15:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.841+0000 7f4e8fe00640 1 -- 172.21.15.69:0/845639779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90071e50 msgr2=0x7f4e901f42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:19.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.841+0000 7f4e8fe00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90071e50 0x7f4e901f42f0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000d800 tx=0x7f4e8000dcc0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.842+0000 7f4e8fe00640 1 -- 172.21.15.69:0/845639779 shutdown_connections 2024-04-24T08:15:19.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.842+0000 7f4e8fe00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e6c06c8f0 0x7f4e6c06edb0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.842+0000 7f4e8fe00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e90162300 0x7f4e9014a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.842+0000 7f4e8fe00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e90072230 0x7f4e901f1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.842+0000 7f4e8fe00640 1 --2- 172.21.15.69:0/845639779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e90071e50 0x7f4e901f42f0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:19.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.842+0000 7f4e8fe00640 1 -- 172.21.15.69:0/845639779 >> 172.21.15.69:0/845639779 conn(0x7f4e900d0160 msgr2=0x7f4e900b3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:19.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.842+0000 7f4e8fe00640 1 -- 172.21.15.69:0/845639779 shutdown_connections 2024-04-24T08:15:19.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:19.842+0000 7f4e8fe00640 1 -- 172.21.15.69:0/845639779 wait complete. 2024-04-24T08:15:19.860 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:15:19.860 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:15:20.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.035+0000 7f9576400640 1 Processor -- start 2024-04-24T08:15:20.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.035+0000 7f9576400640 1 -- start start 2024-04-24T08:15:20.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.037+0000 7f9576400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f957015ea60 0x7f95700b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.037+0000 7f9576400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95700ce910 con 0x7f95700cdf30 2024-04-24T08:15:20.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.037+0000 7f9576400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95700cebf0 con 0x7f95700d09f0 2024-04-24T08:15:20.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.037+0000 7f9576400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f95700ceed0 con 0x7f957015ea60 2024-04-24T08:15:20.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.037+0000 7f956fe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f95700cdf30 0x7f95700d02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47658/0 (socket says 172.21.15.69:47658) 2024-04-24T08:15:20.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.037+0000 7f956fe00640 1 -- 172.21.15.69:0/2451378520 learned_addr learned my addr 172.21.15.69:0/2451378520 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:20.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.037+0000 7f9575000640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f957015ea60 0x7f95700b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.037+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3584929949 0 0) 0x7f95700ce910 con 0x7f95700cdf30 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9558005730 con 0x7f95700cdf30 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 617051824 0 0) 0x7f9558005730 con 0x7f95700cdf30 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f957015ea60 msgr2=0x7f95700b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f957015ea60 0x7f95700b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 >> v1:172.21.15.121:6789/0 conn(0x7f95700d09f0 legacy=0x7f957006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95700cf1b0 con 0x7f95700cdf30 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f955c001670 con 0x7f95700cdf30 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 >> v1:172.21.15.69:6789/0 conn(0x7f95700cdf30 legacy=0x7f95700d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f957015ea60 0x7f9558005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558006220 0x7f9558006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9558006f50 0x7f95580073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95700ceed0 con 0x7f9558006f50 2024-04-24T08:15:20.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f955800de10 con 0x7f9558006220 2024-04-24T08:15:20.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f955800e010 con 0x7f957015ea60 2024-04-24T08:15:20.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.038+0000 7f9575a00640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9558006f50 0x7f95580073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.039+0000 7f9575000640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f957015ea60 0x7f9558005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.039+0000 7f9575a00640 1 -- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f957015ea60 msgr2=0x7f9558005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.039+0000 7f9575a00640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f957015ea60 0x7f9558005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.039+0000 7f9575a00640 1 -- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558006220 msgr2=0x7f9558006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.039+0000 7f9575a00640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558006220 0x7f9558006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.039+0000 7f9575a00640 1 -- 172.21.15.69:0/2451378520 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9558005730 con 0x7f9558006f50 2024-04-24T08:15:20.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.039+0000 7f9575000640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f957015ea60 0x7f9558005bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.040+0000 7f9575a00640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9558006f50 0x7f95580073d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f9568001e20 tx=0x7f9558007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1112aa057c9eb3f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.040+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9568001a70 con 0x7f9558006f50 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.040+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f956802ae40 con 0x7f9558006f50 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.040+0000 7f956f400640 1 -- 172.21.15.69:0/2451378520 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9568003bb0 con 0x7f9558006f50 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.040+0000 7f9576400640 1 -- 172.21.15.69:0/2451378520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9558006f50 msgr2=0x7f95580073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.040+0000 7f9576400640 1 --2- 172.21.15.69:0/2451378520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9558006f50 0x7f95580073d0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f9568001e20 tx=0x7f9558007a40 comp rx=0 tx=0).stop 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.040+0000 7f9575a00640 1 -- 172.21.15.69:0/2451378520 reap_dead start 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.040+0000 7f9576400640 1 -- 172.21.15.69:0/2451378520 shutdown_connections 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.040+0000 7f9576400640 1 -- 172.21.15.69:0/2451378520 >> 172.21.15.69:0/2451378520 conn(0x7f9570067b80 msgr2=0x7f957006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.041+0000 7f9576400640 1 -- 172.21.15.69:0/2451378520 shutdown_connections 2024-04-24T08:15:20.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.041+0000 7f9576400640 1 -- 172.21.15.69:0/2451378520 wait complete. 2024-04-24T08:15:20.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.042+0000 7f9576400640 1 Processor -- start 2024-04-24T08:15:20.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.042+0000 7f9576400640 1 -- start start 2024-04-24T08:15:20.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.043+0000 7f9576400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558006f50 0x7f957014dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.043+0000 7f9576400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95700b4160 0x7f95700b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.043+0000 7f9576400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9570151ef0 0x7f9570152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.043+0000 7f9576400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95700ce510 con 0x7f95700b4160 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.043+0000 7f9576400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f95700ce390 con 0x7f9558006f50 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.043+0000 7f9576400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f95700ce690 con 0x7f9570151ef0 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.043+0000 7f956fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95700b4160 0x7f95700b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.043+0000 7f956fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95700b4160 0x7f95700b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53682/0 (socket says 172.21.15.69:53682) 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.043+0000 7f956fe00640 1 -- 172.21.15.69:0/2490582264 learned_addr learned my addr 172.21.15.69:0/2490582264 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f9575000640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558006f50 0x7f957014dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f956fe00640 1 -- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9570151ef0 msgr2=0x7f9570152370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f956fe00640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9570151ef0 0x7f9570152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f956fe00640 1 -- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558006f50 msgr2=0x7f957014dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f956fe00640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558006f50 0x7f957014dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f956fe00640 1 -- 172.21.15.69:0/2490582264 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95701528b0 con 0x7f95700b4160 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f9575000640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558006f50 0x7f957014dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:20.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f956fe00640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95700b4160 0x7f95700b45c0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f955c00c7a0 tx=0x7f955c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f956cc00640 1 -- 172.21.15.69:0/2490582264 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c018070 con 0x7f95700b4160 2024-04-24T08:15:20.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f956cc00640 1 -- 172.21.15.69:0/2490582264 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f955c0074e0 con 0x7f95700b4160 2024-04-24T08:15:20.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f956cc00640 1 -- 172.21.15.69:0/2490582264 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c004a20 con 0x7f95700b4160 2024-04-24T08:15:20.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.044+0000 7f9576400640 1 -- 172.21.15.69:0/2490582264 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95700b0110 con 0x7f95700b4160 2024-04-24T08:15:20.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.045+0000 7f9576400640 1 -- 172.21.15.69:0/2490582264 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95700b06d0 con 0x7f95700b4160 2024-04-24T08:15:20.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.046+0000 7f9576400640 1 -- 172.21.15.69:0/2490582264 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9570075d50 con 0x7f95700b4160 2024-04-24T08:15:20.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.047+0000 7f956cc00640 1 -- 172.21.15.69:0/2490582264 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f955c004bc0 con 0x7f95700b4160 2024-04-24T08:15:20.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.048+0000 7f956cc00640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f955006c920 0x7f955006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.048+0000 7f956cc00640 1 -- 172.21.15.69:0/2490582264 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 4857+0+0 (secure 0 0 0) 0x7f955c08abe0 con 0x7f95700b4160 2024-04-24T08:15:20.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.049+0000 7f9575000640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f955006c920 0x7f955006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.050+0000 7f9575000640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f955006c920 0x7f955006ede0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9564006fd0 tx=0x7f9564008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.052+0000 7f956cc00640 1 -- 172.21.15.69:0/2490582264 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f955c059360 con 0x7f95700b4160 2024-04-24T08:15:20.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.251+0000 7f9576400640 1 -- 172.21.15.69:0/2490582264 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f95700ccc80 con 0x7f95700b4160 2024-04-24T08:15:20.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.252+0000 7f956cc00640 1 -- 172.21.15.69:0/2490582264 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) ==== 93+0+10 (secure 0 0 0) 0x7f955c058c60 con 0x7f95700b4160 2024-04-24T08:15:20.253 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:15:20.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.254+0000 7f9576400640 1 -- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f955006c920 msgr2=0x7f955006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.254+0000 7f9576400640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f955006c920 0x7f955006ede0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9564006fd0 tx=0x7f9564008040 comp rx=0 tx=0).stop 2024-04-24T08:15:20.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.254+0000 7f9576400640 1 -- 172.21.15.69:0/2490582264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95700b4160 msgr2=0x7f95700b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.254+0000 7f9576400640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95700b4160 0x7f95700b45c0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f955c00c7a0 tx=0x7f955c00cc60 comp rx=0 tx=0).stop 2024-04-24T08:15:20.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.255+0000 7f9576400640 1 -- 172.21.15.69:0/2490582264 shutdown_connections 2024-04-24T08:15:20.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.255+0000 7f9576400640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f955006c920 0x7f955006ede0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.255+0000 7f9576400640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95700b4160 0x7f95700b45c0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.255+0000 7f9576400640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9570151ef0 0x7f9570152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.255+0000 7f9576400640 1 --2- 172.21.15.69:0/2490582264 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9558006f50 0x7f957014dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.255+0000 7f9576400640 1 -- 172.21.15.69:0/2490582264 >> 172.21.15.69:0/2490582264 conn(0x7f9570067b80 msgr2=0x7f957015cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.255+0000 7f9576400640 1 -- 172.21.15.69:0/2490582264 shutdown_connections 2024-04-24T08:15:20.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.255+0000 7f9576400640 1 -- 172.21.15.69:0/2490582264 wait complete. 2024-04-24T08:15:20.274 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T08:15:20.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.462+0000 7ff931600640 1 Processor -- start 2024-04-24T08:15:20.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.463+0000 7ff931600640 1 -- start start 2024-04-24T08:15:20.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff931600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff92c145a90 0x7ff92c06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff931600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff92c145fe0 0x7ff92c160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff931600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff92c0cf3d0 con 0x7ff92c0d0300 2024-04-24T08:15:20.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff931600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff92c0cf6b0 con 0x7ff92c145a90 2024-04-24T08:15:20.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff931600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff92c0cf990 con 0x7ff92c145fe0 2024-04-24T08:15:20.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff922000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff92c145a90 0x7ff92c06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff922000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff92c145a90 0x7ff92c06e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53696/0 (socket says 172.21.15.69:53696) 2024-04-24T08:15:20.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff922000640 1 -- 172.21.15.69:0/3788746735 learned_addr learned my addr 172.21.15.69:0/3788746735 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:20.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff922000640 1 -- 172.21.15.69:0/3788746735 >> v2:172.21.15.121:3300/0 conn(0x7ff92c145fe0 msgr2=0x7ff92c160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff922000640 1 --2- 172.21.15.69:0/3788746735 >> v2:172.21.15.121:3300/0 conn(0x7ff92c145fe0 0x7ff92c160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff922000640 1 -- 172.21.15.69:0/3788746735 >> v1:172.21.15.121:6789/0 conn(0x7ff92c0d0300 legacy=0x7ff92c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.465+0000 7ff922000640 1 -- 172.21.15.69:0/3788746735 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c0cfc70 con 0x7ff92c145a90 2024-04-24T08:15:20.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.466+0000 7ff922000640 1 --2- 172.21.15.69:0/3788746735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 0x7ff92c06e240 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0d1050 tx=0x7ff91402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c8feb2d0714ce5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff92a000640 1 -- 172.21.15.69:0/3788746735 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff914037070 con 0x7ff92c145a90 2024-04-24T08:15:20.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff92a000640 1 -- 172.21.15.69:0/3788746735 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff914032440 con 0x7ff92c145a90 2024-04-24T08:15:20.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff92a000640 1 -- 172.21.15.69:0/3788746735 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff914032810 con 0x7ff92c145a90 2024-04-24T08:15:20.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff931600640 1 -- 172.21.15.69:0/3788746735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 msgr2=0x7ff92c06e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff931600640 1 --2- 172.21.15.69:0/3788746735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 0x7ff92c06e240 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0d1050 tx=0x7ff91402a230 comp rx=0 tx=0).stop 2024-04-24T08:15:20.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff931600640 1 -- 172.21.15.69:0/3788746735 shutdown_connections 2024-04-24T08:15:20.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff931600640 1 --2- 172.21.15.69:0/3788746735 >> v2:172.21.15.121:3300/0 conn(0x7ff92c145fe0 0x7ff92c160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff931600640 1 --2- 172.21.15.69:0/3788746735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 0x7ff92c06e240 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff931600640 1 -- 172.21.15.69:0/3788746735 >> 172.21.15.69:0/3788746735 conn(0x7ff92c067d90 msgr2=0x7ff92c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff931600640 1 -- 172.21.15.69:0/3788746735 shutdown_connections 2024-04-24T08:15:20.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.467+0000 7ff931600640 1 -- 172.21.15.69:0/3788746735 wait complete. 2024-04-24T08:15:20.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.469+0000 7ff931600640 1 Processor -- start 2024-04-24T08:15:20.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.469+0000 7ff931600640 1 -- start start 2024-04-24T08:15:20.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.470+0000 7ff931600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff92c0d0300 0x7ff92c13d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.470+0000 7ff931600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 0x7ff92c15c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.470+0000 7ff931600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c145fe0 0x7ff92c15e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.470+0000 7ff931600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c1458f0 con 0x7ff92c145a90 2024-04-24T08:15:20.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.470+0000 7ff931600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c145770 con 0x7ff92c145fe0 2024-04-24T08:15:20.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.470+0000 7ff931600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff92c0cdc30 con 0x7ff92c0d0300 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.470+0000 7ff922000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 0x7ff92c15c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff922000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 0x7ff92c15c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53704/0 (socket says 172.21.15.69:53704) 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff922000640 1 -- 172.21.15.69:0/843174855 learned_addr learned my addr 172.21.15.69:0/843174855 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff92b400640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c145fe0 0x7ff92c15e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff92aa00640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff92c0d0300 0x7ff92c13d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff922000640 1 -- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff92c0d0300 msgr2=0x7ff92c13d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff922000640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff92c0d0300 0x7ff92c13d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff922000640 1 -- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c145fe0 msgr2=0x7ff92c15e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff922000640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c145fe0 0x7ff92c15e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff922000640 1 -- 172.21.15.69:0/843174855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c0b5320 con 0x7ff92c145a90 2024-04-24T08:15:20.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff92b400640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c145fe0 0x7ff92c15e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:20.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.471+0000 7ff922000640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 0x7ff92c15c080 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff914000c00 tx=0x7ff9140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.472+0000 7ff923400640 1 -- 172.21.15.69:0/843174855 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff914041070 con 0x7ff92c145a90 2024-04-24T08:15:20.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.472+0000 7ff923400640 1 -- 172.21.15.69:0/843174855 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff91403c870 con 0x7ff92c145a90 2024-04-24T08:15:20.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.472+0000 7ff931600640 1 -- 172.21.15.69:0/843174855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff92c1f8130 con 0x7ff92c145a90 2024-04-24T08:15:20.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.472+0000 7ff923400640 1 -- 172.21.15.69:0/843174855 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff914037040 con 0x7ff92c145a90 2024-04-24T08:15:20.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.472+0000 7ff931600640 1 -- 172.21.15.69:0/843174855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff92c1f86f0 con 0x7ff92c145a90 2024-04-24T08:15:20.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.473+0000 7ff931600640 1 -- 172.21.15.69:0/843174855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff92c068a80 con 0x7ff92c145a90 2024-04-24T08:15:20.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.474+0000 7ff923400640 1 -- 172.21.15.69:0/843174855 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff914044070 con 0x7ff92c145a90 2024-04-24T08:15:20.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.475+0000 7ff923400640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff90806c8f0 0x7ff90806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.475+0000 7ff923400640 1 -- 172.21.15.69:0/843174855 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 4857+0+0 (secure 0 0 0) 0x7ff9140ac490 con 0x7ff92c145a90 2024-04-24T08:15:20.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.475+0000 7ff92aa00640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff90806c8f0 0x7ff90806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.476+0000 7ff92aa00640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff90806c8f0 0x7ff90806edb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009f60 tx=0x7ff91c0094f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.479+0000 7ff923400640 1 -- 172.21.15.69:0/843174855 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff91407ad20 con 0x7ff92c145a90 2024-04-24T08:15:20.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.681+0000 7ff931600640 1 -- 172.21.15.69:0/843174855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff92c0d1120 con 0x7ff92c145a90 2024-04-24T08:15:20.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.682+0000 7ff923400640 1 -- 172.21.15.69:0/843174855 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) ==== 84+0+65 (secure 0 0 0) 0x7ff91407a620 con 0x7ff92c145a90 2024-04-24T08:15:20.683 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:20.683 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:15:20.683 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:15:20.683 INFO:teuthology.orchestra.run.smithi069.stdout: "poolnum": 1, 2024-04-24T08:15:20.683 INFO:teuthology.orchestra.run.smithi069.stdout: "poolname": ".mgr" 2024-04-24T08:15:20.683 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:20.683 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 -- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff90806c8f0 msgr2=0x7ff90806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff90806c8f0 0x7ff90806edb0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009f60 tx=0x7ff91c0094f0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 -- 172.21.15.69:0/843174855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 msgr2=0x7ff92c15c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 0x7ff92c15c080 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7ff914000c00 tx=0x7ff9140046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 -- 172.21.15.69:0/843174855 shutdown_connections 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff90806c8f0 0x7ff90806edb0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff92c145a90 0x7ff92c15c080 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff92c145fe0 0x7ff92c15e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 --2- 172.21.15.69:0/843174855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff92c0d0300 0x7ff92c13d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 -- 172.21.15.69:0/843174855 >> 172.21.15.69:0/843174855 conn(0x7ff92c067d90 msgr2=0x7ff92c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 -- 172.21.15.69:0/843174855 shutdown_connections 2024-04-24T08:15:20.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.685+0000 7ff931600640 1 -- 172.21.15.69:0/843174855 wait complete. 2024-04-24T08:15:20.703 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T08:15:20.703 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T08:15:20.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.902+0000 7f8c7a200640 1 Processor -- start 2024-04-24T08:15:20.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.902+0000 7f8c7a200640 1 -- start start 2024-04-24T08:15:20.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.903+0000 7f8c7a200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8c7415c590 0x7f8c740b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.903+0000 7f8c7a200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8c740b4070 0x7f8c740d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.904+0000 7f8c7a200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c740b4840 con 0x7f8c740d0d00 2024-04-24T08:15:20.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.904+0000 7f8c7a200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8c740d1130 con 0x7f8c740b4070 2024-04-24T08:15:20.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.904+0000 7f8c7a200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8c740b4560 con 0x7f8c7415c590 2024-04-24T08:15:20.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.904+0000 7f8c72a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8c740b4070 0x7f8c740d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.904+0000 7f8c72a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8c740b4070 0x7f8c740d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53710/0 (socket says 172.21.15.69:53710) 2024-04-24T08:15:20.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.904+0000 7f8c72a00640 1 -- 172.21.15.69:0/3711006222 learned_addr learned my addr 172.21.15.69:0/3711006222 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:20.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.904+0000 7f8c73400640 1 --2- 172.21.15.69:0/3711006222 >> v2:172.21.15.121:3300/0 conn(0x7f8c7415c590 0x7f8c740b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.905+0000 7f8c72a00640 1 -- 172.21.15.69:0/3711006222 >> v2:172.21.15.121:3300/0 conn(0x7f8c7415c590 msgr2=0x7f8c740b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.905+0000 7f8c72a00640 1 --2- 172.21.15.69:0/3711006222 >> v2:172.21.15.121:3300/0 conn(0x7f8c7415c590 0x7f8c740b3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.905+0000 7f8c72a00640 1 -- 172.21.15.69:0/3711006222 >> v1:172.21.15.121:6789/0 conn(0x7f8c740d0d00 legacy=0x7f8c740cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.905+0000 7f8c72a00640 1 -- 172.21.15.69:0/3711006222 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c740ad2e0 con 0x7f8c740b4070 2024-04-24T08:15:20.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.906+0000 7f8c72a00640 1 --2- 172.21.15.69:0/3711006222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 0x7f8c740d07c0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f8c740ceff0 tx=0x7f8c6002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43305e4a2c3c6ae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.907+0000 7f8c72000640 1 -- 172.21.15.69:0/3711006222 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c60037070 con 0x7f8c740b4070 2024-04-24T08:15:20.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.907+0000 7f8c72000640 1 -- 172.21.15.69:0/3711006222 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c60032440 con 0x7f8c740b4070 2024-04-24T08:15:20.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.907+0000 7f8c72000640 1 -- 172.21.15.69:0/3711006222 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c60032810 con 0x7f8c740b4070 2024-04-24T08:15:20.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.907+0000 7f8c7a200640 1 -- 172.21.15.69:0/3711006222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 msgr2=0x7f8c740d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.907+0000 7f8c7a200640 1 --2- 172.21.15.69:0/3711006222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 0x7f8c740d07c0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f8c740ceff0 tx=0x7f8c6002a270 comp rx=0 tx=0).stop 2024-04-24T08:15:20.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.907+0000 7f8c7a200640 1 -- 172.21.15.69:0/3711006222 shutdown_connections 2024-04-24T08:15:20.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.907+0000 7f8c7a200640 1 --2- 172.21.15.69:0/3711006222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 0x7f8c740d07c0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.907+0000 7f8c7a200640 1 --2- 172.21.15.69:0/3711006222 >> v2:172.21.15.121:3300/0 conn(0x7f8c7415c590 0x7f8c740b3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.907+0000 7f8c7a200640 1 -- 172.21.15.69:0/3711006222 >> 172.21.15.69:0/3711006222 conn(0x7f8c7406be40 msgr2=0x7f8c7406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.908+0000 7f8c7a200640 1 -- 172.21.15.69:0/3711006222 shutdown_connections 2024-04-24T08:15:20.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.908+0000 7f8c7a200640 1 -- 172.21.15.69:0/3711006222 wait complete. 2024-04-24T08:15:20.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.909+0000 7f8c7a200640 1 Processor -- start 2024-04-24T08:15:20.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.909+0000 7f8c7a200640 1 -- start start 2024-04-24T08:15:20.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.910+0000 7f8c7a200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 0x7f8c741f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.910+0000 7f8c7a200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740d0d00 0x7f8c74072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.910+0000 7f8c7a200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c7415c590 0x7f8c74073510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.910+0000 7f8c7a200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8c740b5160 con 0x7f8c740b4070 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.910+0000 7f8c7a200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8c740ad8a0 con 0x7f8c740d0d00 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.910+0000 7f8c7a200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8c740b52e0 con 0x7f8c7415c590 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.911+0000 7f8c73400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 0x7f8c741f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.911+0000 7f8c73400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 0x7f8c741f7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53722/0 (socket says 172.21.15.69:53722) 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.911+0000 7f8c73400640 1 -- 172.21.15.69:0/3667908881 learned_addr learned my addr 172.21.15.69:0/3667908881 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.911+0000 7f8c72a00640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740d0d00 0x7f8c74072fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.911+0000 7f8c73400640 1 -- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c7415c590 msgr2=0x7f8c74073510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.911+0000 7f8c73400640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c7415c590 0x7f8c74073510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.911+0000 7f8c73400640 1 -- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740d0d00 msgr2=0x7f8c74072fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.911+0000 7f8c73400640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740d0d00 0x7f8c74072fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.911+0000 7f8c73400640 1 -- 172.21.15.69:0/3667908881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7406e950 con 0x7f8c740b4070 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.911+0000 7f8c73400640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 0x7f8c741f7740 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400d600 tx=0x7f8c6400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.912+0000 7f8c6b400640 1 -- 172.21.15.69:0/3667908881 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c640125f0 con 0x7f8c740b4070 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.912+0000 7f8c6b400640 1 -- 172.21.15.69:0/3667908881 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c64019070 con 0x7f8c740b4070 2024-04-24T08:15:20.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.912+0000 7f8c7a200640 1 -- 172.21.15.69:0/3667908881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c7406ec30 con 0x7f8c740b4070 2024-04-24T08:15:20.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.912+0000 7f8c6b400640 1 -- 172.21.15.69:0/3667908881 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c640129f0 con 0x7f8c740b4070 2024-04-24T08:15:20.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.912+0000 7f8c7a200640 1 -- 172.21.15.69:0/3667908881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c740c7cb0 con 0x7f8c740b4070 2024-04-24T08:15:20.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.913+0000 7f8c7a200640 1 -- 172.21.15.69:0/3667908881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c7406f150 con 0x7f8c740b4070 2024-04-24T08:15:20.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.914+0000 7f8c6b400640 1 -- 172.21.15.69:0/3667908881 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8c64012b90 con 0x7f8c740b4070 2024-04-24T08:15:20.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.915+0000 7f8c6b400640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c5406c8f0 0x7f8c5406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.915+0000 7f8c6b400640 1 -- 172.21.15.69:0/3667908881 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 4857+0+0 (secure 0 0 0) 0x7f8c6408bc60 con 0x7f8c740b4070 2024-04-24T08:15:20.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.916+0000 7f8c72a00640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c5406c8f0 0x7f8c5406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.917+0000 7f8c72a00640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c5406c8f0 0x7f8c5406edb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8c60034000 tx=0x7f8c6002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:20.920+0000 7f8c6b400640 1 -- 172.21.15.69:0/3667908881 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c7406f150 con 0x7f8c740b4070 2024-04-24T08:15:21.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.113+0000 7f8c7a200640 1 -- 172.21.15.69:0/3667908881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8c74075dc0 con 0x7f8c740b4070 2024-04-24T08:15:21.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.953+0000 7f8c6b400640 1 -- 172.21.15.69:0/3667908881 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) ==== 138+0+0 (secure 0 0 0) 0x7f8c6405a3d0 con 0x7f8c740b4070 2024-04-24T08:15:21.954 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' created 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 -- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c5406c8f0 msgr2=0x7f8c5406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c5406c8f0 0x7f8c5406edb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8c60034000 tx=0x7f8c6002e000 comp rx=0 tx=0).stop 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 -- 172.21.15.69:0/3667908881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 msgr2=0x7f8c741f7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 0x7f8c741f7740 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400d600 tx=0x7f8c6400dac0 comp rx=0 tx=0).stop 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 -- 172.21.15.69:0/3667908881 shutdown_connections 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c5406c8f0 0x7f8c5406edb0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c7415c590 0x7f8c74073510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c740d0d00 0x7f8c74072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 --2- 172.21.15.69:0/3667908881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c740b4070 0x7f8c741f7740 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 -- 172.21.15.69:0/3667908881 >> 172.21.15.69:0/3667908881 conn(0x7f8c7406be40 msgr2=0x7f8c7415a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 -- 172.21.15.69:0/3667908881 shutdown_connections 2024-04-24T08:15:21.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:21.957+0000 7f8c7a200640 1 -- 172.21.15.69:0/3667908881 wait complete. 2024-04-24T08:15:21.983 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T08:15:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.168+0000 7f2af3e00640 1 Processor -- start 2024-04-24T08:15:22.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.169+0000 7f2af3e00640 1 -- start start 2024-04-24T08:15:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.170+0000 7f2af3e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2af40722f0 0x7f2af415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.170+0000 7f2af3e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af40cc4a0 0x7f2af406e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.170+0000 7f2af3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2af406e5a0 0x7f2af40cab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.170+0000 7f2af3e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2af4073cd0 con 0x7f2af406e5a0 2024-04-24T08:15:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.170+0000 7f2af3e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2af415d1c0 con 0x7f2af40722f0 2024-04-24T08:15:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.170+0000 7f2af3e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2af415d4a0 con 0x7f2af40cc4a0 2024-04-24T08:15:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.170+0000 7f2af3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2af406e5a0 0x7f2af40cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.171+0000 7f2af3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2af406e5a0 0x7f2af40cab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53736/0 (socket says 172.21.15.69:53736) 2024-04-24T08:15:22.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.171+0000 7f2af3400640 1 -- 172.21.15.69:0/3414976703 learned_addr learned my addr 172.21.15.69:0/3414976703 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:22.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.171+0000 7f2af2a00640 1 --2- 172.21.15.69:0/3414976703 >> v2:172.21.15.121:3300/0 conn(0x7f2af40722f0 0x7f2af415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:22.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.171+0000 7f2aea000640 1 --2- 172.21.15.69:0/3414976703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af40cc4a0 0x7f2af406e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:22.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.171+0000 7f2af3400640 1 -- 172.21.15.69:0/3414976703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af40cc4a0 msgr2=0x7f2af406e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:22.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.171+0000 7f2af3400640 1 --2- 172.21.15.69:0/3414976703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af40cc4a0 0x7f2af406e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.171+0000 7f2af3400640 1 -- 172.21.15.69:0/3414976703 >> v2:172.21.15.121:3300/0 conn(0x7f2af40722f0 msgr2=0x7f2af415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:22.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.171+0000 7f2af3400640 1 --2- 172.21.15.69:0/3414976703 >> v2:172.21.15.121:3300/0 conn(0x7f2af40722f0 0x7f2af415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.171+0000 7f2af3400640 1 -- 172.21.15.69:0/3414976703 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af415d780 con 0x7f2af406e5a0 2024-04-24T08:15:22.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.172+0000 7f2af3400640 1 --2- 172.21.15.69:0/3414976703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af406e5a0 0x7f2af40cab90 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f2af40cda30 tx=0x7f2aec02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daf2698cd6191ad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:22.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.173+0000 7f2af2000640 1 -- 172.21.15.69:0/3414976703 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec036070 con 0x7f2af406e5a0 2024-04-24T08:15:22.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.173+0000 7f2af2000640 1 -- 172.21.15.69:0/3414976703 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aec031600 con 0x7f2af406e5a0 2024-04-24T08:15:22.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.173+0000 7f2af2000640 1 -- 172.21.15.69:0/3414976703 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec0319d0 con 0x7f2af406e5a0 2024-04-24T08:15:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.173+0000 7f2af3e00640 1 -- 172.21.15.69:0/3414976703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af406e5a0 msgr2=0x7f2af40cab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.173+0000 7f2af3e00640 1 --2- 172.21.15.69:0/3414976703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af406e5a0 0x7f2af40cab90 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f2af40cda30 tx=0x7f2aec02a6a0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.173+0000 7f2af3e00640 1 -- 172.21.15.69:0/3414976703 shutdown_connections 2024-04-24T08:15:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.173+0000 7f2af3e00640 1 --2- 172.21.15.69:0/3414976703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af406e5a0 0x7f2af40cab90 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.173+0000 7f2af3e00640 1 --2- 172.21.15.69:0/3414976703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af40cc4a0 0x7f2af406e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.173+0000 7f2af3e00640 1 --2- 172.21.15.69:0/3414976703 >> v2:172.21.15.121:3300/0 conn(0x7f2af40722f0 0x7f2af415cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.173+0000 7f2af3e00640 1 -- 172.21.15.69:0/3414976703 >> 172.21.15.69:0/3414976703 conn(0x7f2af40b1da0 msgr2=0x7f2af40b2180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.174+0000 7f2af3e00640 1 -- 172.21.15.69:0/3414976703 shutdown_connections 2024-04-24T08:15:22.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.174+0000 7f2af3e00640 1 -- 172.21.15.69:0/3414976703 wait complete. 2024-04-24T08:15:22.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.175+0000 7f2af3e00640 1 Processor -- start 2024-04-24T08:15:22.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.175+0000 7f2af3e00640 1 -- start start 2024-04-24T08:15:22.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.176+0000 7f2af3e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af406e5a0 0x7f2af414f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:22.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.176+0000 7f2af3e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af40722f0 0x7f2af4160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:22.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.176+0000 7f2af3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af40cc4a0 0x7f2af4160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.176+0000 7f2af3e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2af40cb250 con 0x7f2af40cc4a0 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.176+0000 7f2af3e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2af40cb0d0 con 0x7f2af406e5a0 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.176+0000 7f2af3e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2af40cb3d0 con 0x7f2af40722f0 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2af3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af40cc4a0 0x7f2af4160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2af3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af40cc4a0 0x7f2af4160560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53742/0 (socket says 172.21.15.69:53742) 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2af3400640 1 -- 172.21.15.69:0/2010959546 learned_addr learned my addr 172.21.15.69:0/2010959546 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2af2a00640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af406e5a0 0x7f2af414f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2aea000640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af40722f0 0x7f2af4160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2af3400640 1 -- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af40722f0 msgr2=0x7f2af4160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2af3400640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af40722f0 0x7f2af4160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2af3400640 1 -- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af406e5a0 msgr2=0x7f2af414f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2af3400640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af406e5a0 0x7f2af414f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2af3400640 1 -- 172.21.15.69:0/2010959546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2af414fff0 con 0x7f2af40cc4a0 2024-04-24T08:15:22.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.177+0000 7f2af2a00640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af406e5a0 0x7f2af414f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:22.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.178+0000 7f2af3400640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af40cc4a0 0x7f2af4160560 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009f40 tx=0x7f2aec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:22.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.178+0000 7f2aeb400640 1 -- 172.21.15.69:0/2010959546 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec041020 con 0x7f2af40cc4a0 2024-04-24T08:15:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.178+0000 7f2aeb400640 1 -- 172.21.15.69:0/2010959546 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aec0077a0 con 0x7f2af40cc4a0 2024-04-24T08:15:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.178+0000 7f2aeb400640 1 -- 172.21.15.69:0/2010959546 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aec036040 con 0x7f2af40cc4a0 2024-04-24T08:15:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.178+0000 7f2af3e00640 1 -- 172.21.15.69:0/2010959546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2af41502d0 con 0x7f2af40cc4a0 2024-04-24T08:15:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.178+0000 7f2af3e00640 1 -- 172.21.15.69:0/2010959546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2af4150890 con 0x7f2af40cc4a0 2024-04-24T08:15:22.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.179+0000 7f2af3e00640 1 -- 172.21.15.69:0/2010959546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2af4075d50 con 0x7f2af40cc4a0 2024-04-24T08:15:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.180+0000 7f2aeb400640 1 -- 172.21.15.69:0/2010959546 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2aec044020 con 0x7f2af40cc4a0 2024-04-24T08:15:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.181+0000 7f2aeb400640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad006c8f0 0x7f2ad006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:22.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.181+0000 7f2aeb400640 1 -- 172.21.15.69:0/2010959546 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 5227+0+0 (secure 0 0 0) 0x7f2aec0ace10 con 0x7f2af40cc4a0 2024-04-24T08:15:22.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.182+0000 7f2af2a00640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad006c8f0 0x7f2ad006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:22.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.183+0000 7f2af2a00640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad006c8f0 0x7f2ad006edb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4006fd0 tx=0x7f2ae4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:22.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.185+0000 7f2aeb400640 1 -- 172.21.15.69:0/2010959546 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2aec04b020 con 0x7f2af40cc4a0 2024-04-24T08:15:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.385+0000 7f2af3e00640 1 -- 172.21.15.69:0/2010959546 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f2af415c590 con 0x7f2af40cc4a0 2024-04-24T08:15:22.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.969+0000 7f2aeb400640 1 -- 172.21.15.69:0/2010959546 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) ==== 170+0+0 (secure 0 0 0) 0x7f2aec07b6b0 con 0x7f2af40cc4a0 2024-04-24T08:15:22.970 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' created 2024-04-24T08:15:22.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.977+0000 7f2af3e00640 1 -- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad006c8f0 msgr2=0x7f2ad006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:22.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.977+0000 7f2af3e00640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad006c8f0 0x7f2ad006edb0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2ae4006fd0 tx=0x7f2ae4008040 comp rx=0 tx=0).stop 2024-04-24T08:15:22.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.977+0000 7f2af3e00640 1 -- 172.21.15.69:0/2010959546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af40cc4a0 msgr2=0x7f2af4160560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:22.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.977+0000 7f2af3e00640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af40cc4a0 0x7f2af4160560 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f2aec009f40 tx=0x7f2aec004a40 comp rx=0 tx=0).stop 2024-04-24T08:15:22.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.978+0000 7f2af3e00640 1 -- 172.21.15.69:0/2010959546 shutdown_connections 2024-04-24T08:15:22.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.978+0000 7f2af3e00640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ad006c8f0 0x7f2ad006edb0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.978+0000 7f2af3e00640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2af40cc4a0 0x7f2af4160560 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.978+0000 7f2af3e00640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2af40722f0 0x7f2af4160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.978+0000 7f2af3e00640 1 --2- 172.21.15.69:0/2010959546 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2af406e5a0 0x7f2af414f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:22.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.978+0000 7f2af3e00640 1 -- 172.21.15.69:0/2010959546 >> 172.21.15.69:0/2010959546 conn(0x7f2af40b1da0 msgr2=0x7f2af4156710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:22.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.979+0000 7f2af3e00640 1 -- 172.21.15.69:0/2010959546 shutdown_connections 2024-04-24T08:15:22.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:22.981+0000 7f2af3e00640 1 -- 172.21.15.69:0/2010959546 wait complete. 2024-04-24T08:15:23.012 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T08:15:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.184+0000 7f4562400640 1 Processor -- start 2024-04-24T08:15:23.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.185+0000 7f4562400640 1 -- start start 2024-04-24T08:15:23.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.186+0000 7f4562400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f455c15c820 0x7f455c0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:23.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.186+0000 7f4562400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 0x7f455c06f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:23.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.186+0000 7f4562400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f455c15cc00 con 0x7f455c06fd10 2024-04-24T08:15:23.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.186+0000 7f4562400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f455c06ed30 con 0x7f455c15c820 2024-04-24T08:15:23.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.186+0000 7f4562400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f455c0b4e10 con 0x7f455c0b46a0 2024-04-24T08:15:23.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.187+0000 7f4553e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 0x7f455c06f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:23.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.187+0000 7f4553e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 0x7f455c06f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:35240/0 (socket says 172.21.15.69:35240) 2024-04-24T08:15:23.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.187+0000 7f455b400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f455c15c820 0x7f455c0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:23.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.187+0000 7f4553e00640 1 -- 172.21.15.69:0/1895695489 learned_addr learned my addr 172.21.15.69:0/1895695489 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:23.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.188+0000 7f455b400640 1 -- 172.21.15.69:0/1895695489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 msgr2=0x7f455c06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:23.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.188+0000 7f455b400640 1 --2- 172.21.15.69:0/1895695489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 0x7f455c06f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:23.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.188+0000 7f455b400640 1 -- 172.21.15.69:0/1895695489 >> v1:172.21.15.121:6789/0 conn(0x7f455c06fd10 legacy=0x7f455c0ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:23.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.188+0000 7f455b400640 1 -- 172.21.15.69:0/1895695489 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455c0b50f0 con 0x7f455c15c820 2024-04-24T08:15:23.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.188+0000 7f4553e00640 1 --2- 172.21.15.69:0/1895695489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 0x7f455c06f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:23.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.189+0000 7f455b400640 1 --2- 172.21.15.69:0/1895695489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f455c15c820 0x7f455c0b3e90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f454c0097a0 tx=0x7f454c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5b4132f2be2a866 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:23.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.190+0000 7f455aa00640 1 -- 172.21.15.69:0/1895695489 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c037070 con 0x7f455c15c820 2024-04-24T08:15:23.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.190+0000 7f455aa00640 1 -- 172.21.15.69:0/1895695489 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f454c032440 con 0x7f455c15c820 2024-04-24T08:15:23.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.190+0000 7f455aa00640 1 -- 172.21.15.69:0/1895695489 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c032810 con 0x7f455c15c820 2024-04-24T08:15:23.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.191+0000 7f4562400640 1 -- 172.21.15.69:0/1895695489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f455c15c820 msgr2=0x7f455c0b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:23.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.191+0000 7f4562400640 1 --2- 172.21.15.69:0/1895695489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f455c15c820 0x7f455c0b3e90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f454c0097a0 tx=0x7f454c02a180 comp rx=0 tx=0).stop 2024-04-24T08:15:23.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.191+0000 7f4562400640 1 -- 172.21.15.69:0/1895695489 shutdown_connections 2024-04-24T08:15:23.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.191+0000 7f4562400640 1 --2- 172.21.15.69:0/1895695489 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 0x7f455c06f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:23.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.191+0000 7f4562400640 1 --2- 172.21.15.69:0/1895695489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f455c15c820 0x7f455c0b3e90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:23.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.191+0000 7f4562400640 1 -- 172.21.15.69:0/1895695489 >> 172.21.15.69:0/1895695489 conn(0x7f455c0d0350 msgr2=0x7f455c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:23.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.191+0000 7f4562400640 1 -- 172.21.15.69:0/1895695489 shutdown_connections 2024-04-24T08:15:23.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.191+0000 7f4562400640 1 -- 172.21.15.69:0/1895695489 wait complete. 2024-04-24T08:15:23.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.193+0000 7f4562400640 1 Processor -- start 2024-04-24T08:15:23.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.193+0000 7f4562400640 1 -- start start 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f4562400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f455c06fd10 0x7f455c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f4562400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 0x7f455c1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f4562400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f455c15c820 0x7f455c150a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f4562400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f455c0b4b00 con 0x7f455c06fd10 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f4562400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f455c06f2b0 con 0x7f455c15c820 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f4562400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f455c0ce960 con 0x7f455c0b46a0 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f455b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f455c06fd10 0x7f455c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f455b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f455c06fd10 0x7f455c15f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53758/0 (socket says 172.21.15.69:53758) 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f455b400640 1 -- 172.21.15.69:0/3039626182 learned_addr learned my addr 172.21.15.69:0/3039626182 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f4553e00640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 0x7f455c1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:23.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f455b400640 1 -- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 msgr2=0x7f455c1550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.194+0000 7f455b400640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 0x7f455c1550c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f455b400640 1 -- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f455c15c820 msgr2=0x7f455c150a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f455be00640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f455c15c820 0x7f455c150a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f455b400640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f455c15c820 0x7f455c150a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f455b400640 1 -- 172.21.15.69:0/3039626182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f455c150f40 con 0x7f455c06fd10 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f455be00640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f455c15c820 0x7f455c150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f455b400640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f455c06fd10 0x7f455c15f050 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f454c0098c0 tx=0x7f454c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f4553400640 1 -- 172.21.15.69:0/3039626182 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c041070 con 0x7f455c06fd10 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f4553400640 1 -- 172.21.15.69:0/3039626182 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f454c03c870 con 0x7f455c06fd10 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f4553400640 1 -- 172.21.15.69:0/3039626182 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f454c037040 con 0x7f455c06fd10 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f4562400640 1 -- 172.21.15.69:0/3039626182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f455c1557f0 con 0x7f455c06fd10 2024-04-24T08:15:23.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.195+0000 7f4562400640 1 -- 172.21.15.69:0/3039626182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f455c155db0 con 0x7f455c06fd10 2024-04-24T08:15:23.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.197+0000 7f4562400640 1 -- 172.21.15.69:0/3039626182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f455c068a80 con 0x7f455c06fd10 2024-04-24T08:15:23.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.198+0000 7f4553400640 1 -- 172.21.15.69:0/3039626182 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f454c044070 con 0x7f455c06fd10 2024-04-24T08:15:23.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.198+0000 7f4553400640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f453806c8f0 0x7f453806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:23.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.199+0000 7f4553400640 1 -- 172.21.15.69:0/3039626182 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 5609+0+0 (secure 0 0 0) 0x7f454c0ac590 con 0x7f455c06fd10 2024-04-24T08:15:23.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.199+0000 7f4553e00640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f453806c8f0 0x7f453806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:23.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.201+0000 7f4553e00640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f453806c8f0 0x7f453806edb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f455c069390 tx=0x7f4544008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:23.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.204+0000 7f4553400640 1 -- 172.21.15.69:0/3039626182 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f454c07a8f0 con 0x7f455c06fd10 2024-04-24T08:15:23.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.441+0000 7f4562400640 1 -- 172.21.15.69:0/3039626182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f455c0cddc0 con 0x7f455c06fd10 2024-04-24T08:15:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.984+0000 7f4553400640 1 -- 172.21.15.69:0/3039626182 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v57) ==== 174+0+0 (secure 0 0 0) 0x7f454c07a1f0 con 0x7f455c06fd10 2024-04-24T08:15:23.989 INFO:teuthology.orchestra.run.smithi069.stderr:new fs with metadata pool 12 and data pool 13 2024-04-24T08:15:23.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.993+0000 7f4562400640 1 -- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f453806c8f0 msgr2=0x7f453806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:23.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.993+0000 7f4562400640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f453806c8f0 0x7f453806edb0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f455c069390 tx=0x7f4544008040 comp rx=0 tx=0).stop 2024-04-24T08:15:23.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.993+0000 7f4562400640 1 -- 172.21.15.69:0/3039626182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f455c06fd10 msgr2=0x7f455c15f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:23.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.993+0000 7f4562400640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f455c06fd10 0x7f455c15f050 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f454c0098c0 tx=0x7f454c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:23.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.994+0000 7f4562400640 1 -- 172.21.15.69:0/3039626182 shutdown_connections 2024-04-24T08:15:23.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.994+0000 7f4562400640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f453806c8f0 0x7f453806edb0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:23.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.994+0000 7f4562400640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f455c15c820 0x7f455c150a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:23.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.994+0000 7f4562400640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f455c0b46a0 0x7f455c1550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:23.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.994+0000 7f4562400640 1 --2- 172.21.15.69:0/3039626182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f455c06fd10 0x7f455c15f050 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:23.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.994+0000 7f4562400640 1 -- 172.21.15.69:0/3039626182 >> 172.21.15.69:0/3039626182 conn(0x7f455c0d0350 msgr2=0x7f455c0b0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:23.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.996+0000 7f4562400640 1 -- 172.21.15.69:0/3039626182 shutdown_connections 2024-04-24T08:15:23.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:23.996+0000 7f4562400640 1 -- 172.21.15.69:0/3039626182 wait complete. 2024-04-24T08:15:24.028 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:24.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.193+0000 7fb40c200640 1 Processor -- start 2024-04-24T08:15:24.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.193+0000 7fb40c200640 1 -- start start 2024-04-24T08:15:24.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.195+0000 7fb40c200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb4040d0300 0x7fb40406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.195+0000 7fb40c200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4040cf3d0 con 0x7fb404145fe0 2024-04-24T08:15:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.195+0000 7fb40c200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4040cf6b0 con 0x7fb404145a90 2024-04-24T08:15:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.195+0000 7fb40c200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb4040cf990 con 0x7fb4040d0300 2024-04-24T08:15:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.195+0000 7fb409800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb4040d0300 0x7fb40406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.195+0000 7fb409800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb4040d0300 0x7fb40406e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53760/0 (socket says 172.21.15.69:53760) 2024-04-24T08:15:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.195+0000 7fb409800640 1 -- 172.21.15.69:0/3830343708 learned_addr learned my addr 172.21.15.69:0/3830343708 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.196+0000 7fb409800640 1 -- 172.21.15.69:0/3830343708 >> v1:172.21.15.121:6789/0 conn(0x7fb404145a90 legacy=0x7fb4040d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:24.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.196+0000 7fb409800640 1 -- 172.21.15.69:0/3830343708 >> v1:172.21.15.69:6789/0 conn(0x7fb404145fe0 legacy=0x7fb404160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.196+0000 7fb409800640 1 -- 172.21.15.69:0/3830343708 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4040cfc70 con 0x7fb4040d0300 2024-04-24T08:15:24.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.197+0000 7fb409800640 1 --2- 172.21.15.69:0/3830343708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4040d0300 0x7fb40406e240 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009870 tx=0x7fb3f802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77ba5e00fa30045 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:24.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.197+0000 7fb403e00640 1 -- 172.21.15.69:0/3830343708 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8037070 con 0x7fb4040d0300 2024-04-24T08:15:24.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.197+0000 7fb403e00640 1 -- 172.21.15.69:0/3830343708 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3f8032440 con 0x7fb4040d0300 2024-04-24T08:15:24.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.197+0000 7fb403e00640 1 -- 172.21.15.69:0/3830343708 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8032810 con 0x7fb4040d0300 2024-04-24T08:15:24.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.198+0000 7fb40c200640 1 -- 172.21.15.69:0/3830343708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4040d0300 msgr2=0x7fb40406e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.198+0000 7fb40c200640 1 --2- 172.21.15.69:0/3830343708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4040d0300 0x7fb40406e240 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009870 tx=0x7fb3f802a2a0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.198+0000 7fb40c200640 1 -- 172.21.15.69:0/3830343708 shutdown_connections 2024-04-24T08:15:24.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.198+0000 7fb40c200640 1 --2- 172.21.15.69:0/3830343708 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4040d0300 0x7fb40406e240 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.198+0000 7fb40c200640 1 -- 172.21.15.69:0/3830343708 >> 172.21.15.69:0/3830343708 conn(0x7fb404067d90 msgr2=0x7fb40406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:24.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.198+0000 7fb40c200640 1 -- 172.21.15.69:0/3830343708 shutdown_connections 2024-04-24T08:15:24.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.198+0000 7fb40c200640 1 -- 172.21.15.69:0/3830343708 wait complete. 2024-04-24T08:15:24.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.199+0000 7fb40c200640 1 Processor -- start 2024-04-24T08:15:24.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.199+0000 7fb40c200640 1 -- start start 2024-04-24T08:15:24.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.200+0000 7fb40c200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4040d0300 0x7fb404141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.200+0000 7fb40c200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb404145a90 0x7fb4040abf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.200+0000 7fb40c200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb404145fe0 0x7fb4040aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.200+0000 7fb40c200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb4041458f0 con 0x7fb404145a90 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.200+0000 7fb40c200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb404145770 con 0x7fb4040d0300 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.200+0000 7fb40c200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb4040cdc30 con 0x7fb404145fe0 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.200+0000 7fb408e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb404145a90 0x7fb4040abf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.200+0000 7fb408e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb404145a90 0x7fb4040abf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53764/0 (socket says 172.21.15.69:53764) 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb408e00640 1 -- 172.21.15.69:0/3341947760 learned_addr learned my addr 172.21.15.69:0/3341947760 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb409800640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4040d0300 0x7fb404141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb408e00640 1 -- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb404145fe0 msgr2=0x7fb4040aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb408e00640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb404145fe0 0x7fb4040aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb408e00640 1 -- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4040d0300 msgr2=0x7fb404141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb408e00640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4040d0300 0x7fb404141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb408e00640 1 -- 172.21.15.69:0/3341947760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb404140e40 con 0x7fb404145a90 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb409800640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4040d0300 0x7fb404141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:24.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb408e00640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb404145a90 0x7fb4040abf00 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000c430 tx=0x7fb3f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:24.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb401600640 1 -- 172.21.15.69:0/3341947760 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0011790 con 0x7fb404145a90 2024-04-24T08:15:24.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb401600640 1 -- 172.21.15.69:0/3341947760 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3f0017070 con 0x7fb404145a90 2024-04-24T08:15:24.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.201+0000 7fb40c200640 1 -- 172.21.15.69:0/3341947760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb404141120 con 0x7fb404145a90 2024-04-24T08:15:24.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.202+0000 7fb401600640 1 -- 172.21.15.69:0/3341947760 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0011bb0 con 0x7fb404145a90 2024-04-24T08:15:24.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.202+0000 7fb40c200640 1 -- 172.21.15.69:0/3341947760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4041419c0 con 0x7fb404145a90 2024-04-24T08:15:24.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.203+0000 7fb40c200640 1 -- 172.21.15.69:0/3341947760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4041412f0 con 0x7fb404145a90 2024-04-24T08:15:24.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.204+0000 7fb401600640 1 -- 172.21.15.69:0/3341947760 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb3f001c460 con 0x7fb404145a90 2024-04-24T08:15:24.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.205+0000 7fb401600640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3e406c8f0 0x7fb3e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.205+0000 7fb401600640 1 -- 172.21.15.69:0/3341947760 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7fb3f008b9f0 con 0x7fb404145a90 2024-04-24T08:15:24.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.205+0000 7fb409800640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3e406c8f0 0x7fb3e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:24.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.207+0000 7fb409800640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3e406c8f0 0x7fb3e406edb0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8034000 tx=0x7fb3f802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:24.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.209+0000 7fb401600640 1 -- 172.21.15.69:0/3341947760 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3f0059e20 con 0x7fb404145a90 2024-04-24T08:15:24.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.409+0000 7fb40c200640 1 -- 172.21.15.69:0/3341947760 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb404141b90 con 0x7fb404145a90 2024-04-24T08:15:24.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.411+0000 7fb401600640 1 -- 172.21.15.69:0/3341947760 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16649 (secure 0 0 0) 0x7fb3f0059720 con 0x7fb404145a90 2024-04-24T08:15:24.412 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:24.412 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":81,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:15:23.442760+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:24.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.413+0000 7fb40c200640 1 -- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3e406c8f0 msgr2=0x7fb3e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.413+0000 7fb40c200640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3e406c8f0 0x7fb3e406edb0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8034000 tx=0x7fb3f802e000 comp rx=0 tx=0).stop 2024-04-24T08:15:24.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.413+0000 7fb40c200640 1 -- 172.21.15.69:0/3341947760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb404145a90 msgr2=0x7fb4040abf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.413+0000 7fb40c200640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb404145a90 0x7fb4040abf00 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000c430 tx=0x7fb3f000c8f0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.413+0000 7fb40c200640 1 -- 172.21.15.69:0/3341947760 shutdown_connections 2024-04-24T08:15:24.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.413+0000 7fb40c200640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3e406c8f0 0x7fb3e406edb0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.413+0000 7fb40c200640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb404145a90 0x7fb4040abf00 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.413+0000 7fb40c200640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb404145fe0 0x7fb4040aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.413+0000 7fb40c200640 1 --2- 172.21.15.69:0/3341947760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4040d0300 0x7fb404141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.413+0000 7fb40c200640 1 -- 172.21.15.69:0/3341947760 >> 172.21.15.69:0/3341947760 conn(0x7fb404067d90 msgr2=0x7fb4040ccdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:24.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.414+0000 7fb40c200640 1 -- 172.21.15.69:0/3341947760 shutdown_connections 2024-04-24T08:15:24.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.414+0000 7fb40c200640 1 -- 172.21.15.69:0/3341947760 wait complete. 2024-04-24T08:15:24.432 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:24.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.624+0000 7f3d1d400640 1 Processor -- start 2024-04-24T08:15:24.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.624+0000 7f3d1d400640 1 -- start start 2024-04-24T08:15:24.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.626+0000 7f3d1d400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d180d0200 0x7f3d180d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.626+0000 7f3d1d400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d1815ed40 con 0x7f3d1815e960 2024-04-24T08:15:24.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.626+0000 7f3d1d400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d180cf050 con 0x7f3d1806cfb0 2024-04-24T08:15:24.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.626+0000 7f3d1d400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d18162b50 con 0x7f3d180d0200 2024-04-24T08:15:24.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.626+0000 7f3d17e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3d1815e960 0x7f3d18162440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47678/0 (socket says 172.21.15.69:47678) 2024-04-24T08:15:24.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.626+0000 7f3d17e00640 1 -- 172.21.15.69:0/537093719 learned_addr learned my addr 172.21.15.69:0/537093719 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.626+0000 7f3d1ca00640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d180d0200 0x7f3d180d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 350544867 0 0) 0x7f3d1815ed40 con 0x7f3d1815e960 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cfc003520 con 0x7f3d1815e960 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2594007060 0 0) 0x7f3cfc003520 con 0x7f3d1815e960 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d180d0200 msgr2=0x7f3d180d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d180d0200 0x7f3d180d2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 >> v1:172.21.15.121:6789/0 conn(0x7f3d1806cfb0 legacy=0x7f3d180cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d18162e20 con 0x7f3d1815e960 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f3d08001670 con 0x7f3d1815e960 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 >> v1:172.21.15.69:6789/0 conn(0x7f3d1815e960 legacy=0x7f3d18162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d180d0200 0x7f3cfc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cfc003fb0 0x7f3cfc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cfc004d40 0x7f3cfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3d18162b50 con 0x7f3cfc004d40 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3cfc00bb00 con 0x7f3cfc003fb0 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.627+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3cfc00bd00 con 0x7f3d180d0200 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.628+0000 7f3d1ca00640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cfc004d40 0x7f3cfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.628+0000 7f3d17e00640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d180d0200 0x7f3cfc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.628+0000 7f3d1ca00640 1 -- 172.21.15.69:0/537093719 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d180d0200 msgr2=0x7f3cfc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.628+0000 7f3d1ca00640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d180d0200 0x7f3cfc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.628+0000 7f3d1ca00640 1 -- 172.21.15.69:0/537093719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cfc003fb0 msgr2=0x7f3cfc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.628+0000 7f3d1ca00640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cfc003fb0 0x7f3cfc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.628+0000 7f3d1ca00640 1 -- 172.21.15.69:0/537093719 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cfc003520 con 0x7f3cfc004d40 2024-04-24T08:15:24.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.629+0000 7f3d1ca00640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cfc004d40 0x7f3cfc0051c0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009fc0 tx=0x7f3cfc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129ab44a0afa5020 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.629+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10009c10 con 0x7f3cfc004d40 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.629+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d10036070 con 0x7f3cfc004d40 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.629+0000 7f3d16a00640 1 -- 172.21.15.69:0/537093719 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10031880 con 0x7f3cfc004d40 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.630+0000 7f3d1d400640 1 -- 172.21.15.69:0/537093719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cfc004d40 msgr2=0x7f3cfc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.630+0000 7f3d1d400640 1 --2- 172.21.15.69:0/537093719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cfc004d40 0x7f3cfc0051c0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009fc0 tx=0x7f3cfc005830 comp rx=0 tx=0).stop 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.630+0000 7f3d1ca00640 1 -- 172.21.15.69:0/537093719 reap_dead start 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.630+0000 7f3d1d400640 1 -- 172.21.15.69:0/537093719 shutdown_connections 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.630+0000 7f3d1d400640 1 -- 172.21.15.69:0/537093719 >> 172.21.15.69:0/537093719 conn(0x7f3d1809d900 msgr2=0x7f3d1808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.630+0000 7f3d1d400640 1 -- 172.21.15.69:0/537093719 shutdown_connections 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.630+0000 7f3d1d400640 1 -- 172.21.15.69:0/537093719 wait complete. 2024-04-24T08:15:24.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.631+0000 7f3d1d400640 1 Processor -- start 2024-04-24T08:15:24.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.632+0000 7f3d1d400640 1 -- start start 2024-04-24T08:15:24.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d1d400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cfc004d40 0x7f3d180ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d1d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d18160520 0x7f3d1814e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d1d400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d18161a60 0x7f3d1806d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d1d400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3d180cf5d0 con 0x7f3d18160520 2024-04-24T08:15:24.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d1d400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d180cf2f0 con 0x7f3cfc004d40 2024-04-24T08:15:24.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d1d400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d180b6170 con 0x7f3d18161a60 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d17400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d18160520 0x7f3d1814e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d17400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d18160520 0x7f3d1814e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53794/0 (socket says 172.21.15.69:53794) 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d17400640 1 -- 172.21.15.69:0/1655826830 learned_addr learned my addr 172.21.15.69:0/1655826830 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d17400640 1 -- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d18161a60 msgr2=0x7f3d1806d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d17400640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d18161a60 0x7f3d1806d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d17400640 1 -- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cfc004d40 msgr2=0x7f3d180ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d17400640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cfc004d40 0x7f3d180ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d17400640 1 -- 172.21.15.69:0/1655826830 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1814e9d0 con 0x7f3d18160520 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.633+0000 7f3d17400640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d18160520 0x7f3d1814e490 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400c390 tx=0x7f3d0400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.634+0000 7f3d0fe00640 1 -- 172.21.15.69:0/1655826830 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d040052d0 con 0x7f3d18160520 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.634+0000 7f3d0fe00640 1 -- 172.21.15.69:0/1655826830 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d04005470 con 0x7f3d18160520 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.634+0000 7f3d0fe00640 1 -- 172.21.15.69:0/1655826830 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04007260 con 0x7f3d18160520 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.634+0000 7f3d1d400640 1 -- 172.21.15.69:0/1655826830 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d181609c0 con 0x7f3d18160520 2024-04-24T08:15:24.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.634+0000 7f3d1d400640 1 -- 172.21.15.69:0/1655826830 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d18160f80 con 0x7f3d18160520 2024-04-24T08:15:24.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.635+0000 7f3d1d400640 1 -- 172.21.15.69:0/1655826830 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d18075d50 con 0x7f3d18160520 2024-04-24T08:15:24.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.636+0000 7f3d0fe00640 1 -- 172.21.15.69:0/1655826830 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d04007400 con 0x7f3d18160520 2024-04-24T08:15:24.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.637+0000 7f3d0fe00640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf806c8f0 0x7f3cf806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:24.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.637+0000 7f3d0fe00640 1 -- 172.21.15.69:0/1655826830 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 5709+0+0 (secure 0 0 0) 0x7f3d0408b0a0 con 0x7f3d18160520 2024-04-24T08:15:24.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.638+0000 7f3d17e00640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf806c8f0 0x7f3cf806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:24.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.639+0000 7f3d17e00640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf806c8f0 0x7f3cf806edb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3d08001050 tx=0x7f3d08000f60 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:24.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.641+0000 7f3d0fe00640 1 -- 172.21.15.69:0/1655826830 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d04059440 con 0x7f3d18160520 2024-04-24T08:15:24.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.842+0000 7f3d1d400640 1 -- 172.21.15.69:0/1655826830 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d180d1dc0 con 0x7f3d18160520 2024-04-24T08:15:24.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.844+0000 7f3d0fe00640 1 -- 172.21.15.69:0/1655826830 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+16649 (secure 0 0 0) 0x7f3d04058d40 con 0x7f3d18160520 2024-04-24T08:15:24.845 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:24.845 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":81,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:15:23.442760+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 -- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf806c8f0 msgr2=0x7f3cf806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf806c8f0 0x7f3cf806edb0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3d08001050 tx=0x7f3d08000f60 comp rx=0 tx=0).stop 2024-04-24T08:15:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 -- 172.21.15.69:0/1655826830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d18160520 msgr2=0x7f3d1814e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d18160520 0x7f3d1814e490 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400c390 tx=0x7f3d0400c850 comp rx=0 tx=0).stop 2024-04-24T08:15:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 -- 172.21.15.69:0/1655826830 shutdown_connections 2024-04-24T08:15:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf806c8f0 0x7f3cf806edb0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d18160520 0x7f3d1814e490 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d18161a60 0x7f3d1806d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 --2- 172.21.15.69:0/1655826830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cfc004d40 0x7f3d180ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 -- 172.21.15.69:0/1655826830 >> 172.21.15.69:0/1655826830 conn(0x7f3d1809d900 msgr2=0x7f3d1815d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 -- 172.21.15.69:0/1655826830 shutdown_connections 2024-04-24T08:15:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:24.847+0000 7f3d1d400640 1 -- 172.21.15.69:0/1655826830 wait complete. 2024-04-24T08:15:24.866 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T08:15:25.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.073+0000 7fdd3bc00640 1 Processor -- start 2024-04-24T08:15:25.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.073+0000 7fdd3bc00640 1 -- start start 2024-04-24T08:15:25.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.074+0000 7fdd3bc00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd3415c820 0x7fdd340b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:25.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.074+0000 7fdd3bc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd340b46a0 0x7fdd3406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:25.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.074+0000 7fdd3bc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3415cc00 con 0x7fdd3406fd10 2024-04-24T08:15:25.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.074+0000 7fdd3bc00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fdd3406ed30 con 0x7fdd3415c820 2024-04-24T08:15:25.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.074+0000 7fdd3bc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdd340b4e10 con 0x7fdd340b46a0 2024-04-24T08:15:25.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd39c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdd3406fd10 0x7fdd340ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47682/0 (socket says 172.21.15.69:47682) 2024-04-24T08:15:25.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd39c00640 1 -- 172.21.15.69:0/4030535981 learned_addr learned my addr 172.21.15.69:0/4030535981 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:25.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd39200640 1 --2- 172.21.15.69:0/4030535981 >> v2:172.21.15.121:3300/0 conn(0x7fdd3415c820 0x7fdd340b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:25.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd2be00640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd340b46a0 0x7fdd3406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:25.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1749904364 0 0) 0x7fdd3415cc00 con 0x7fdd3406fd10 2024-04-24T08:15:25.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd14003520 con 0x7fdd3406fd10 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2817092084 0 0) 0x7fdd14003520 con 0x7fdd3406fd10 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd340b46a0 msgr2=0x7fdd3406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd33e00640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd340b46a0 0x7fdd3406f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 >> v2:172.21.15.121:3300/0 conn(0x7fdd3415c820 msgr2=0x7fdd340b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd33e00640 1 --2- 172.21.15.69:0/4030535981 >> v2:172.21.15.121:3300/0 conn(0x7fdd3415c820 0x7fdd340b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd340b50f0 con 0x7fdd3406fd10 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.075+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fdd2c0019f0 con 0x7fdd3406fd10 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 >> v1:172.21.15.69:6789/0 conn(0x7fdd3406fd10 legacy=0x7fdd340ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd33e00640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd140039b0 0x7fdd14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd33e00640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd340b46a0 0x7fdd14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:25.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd33e00640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd14004d40 0x7fdd140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3406ed30 con 0x7fdd140039b0 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdd340b4e10 con 0x7fdd14004d40 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1400ba70 con 0x7fdd340b46a0 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd39200640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd140039b0 0x7fdd14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd39200640 1 -- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd340b46a0 msgr2=0x7fdd14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd2be00640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd340b46a0 0x7fdd14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd39c00640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd14004d40 0x7fdd140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd39200640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd340b46a0 0x7fdd14004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd39200640 1 -- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd14004d40 msgr2=0x7fdd140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd39200640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd14004d40 0x7fdd140051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:25.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.076+0000 7fdd39200640 1 -- 172.21.15.69:0/4030535981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd14003520 con 0x7fdd140039b0 2024-04-24T08:15:25.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.077+0000 7fdd39200640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd140039b0 0x7fdd14003dd0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400ba60 tx=0x7fdd14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102ff18bb6768714 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:25.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.078+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd24030070 con 0x7fdd140039b0 2024-04-24T08:15:25.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.078+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd2402ae40 con 0x7fdd140039b0 2024-04-24T08:15:25.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.078+0000 7fdd33e00640 1 -- 172.21.15.69:0/4030535981 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd2400b600 con 0x7fdd140039b0 2024-04-24T08:15:25.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.078+0000 7fdd3bc00640 1 -- 172.21.15.69:0/4030535981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd140039b0 msgr2=0x7fdd14003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:25.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.078+0000 7fdd3bc00640 1 --2- 172.21.15.69:0/4030535981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd140039b0 0x7fdd14003dd0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400ba60 tx=0x7fdd14005830 comp rx=0 tx=0).stop 2024-04-24T08:15:25.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.078+0000 7fdd39c00640 1 -- 172.21.15.69:0/4030535981 reap_dead start 2024-04-24T08:15:25.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.078+0000 7fdd3bc00640 1 -- 172.21.15.69:0/4030535981 shutdown_connections 2024-04-24T08:15:25.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.078+0000 7fdd3bc00640 1 -- 172.21.15.69:0/4030535981 >> 172.21.15.69:0/4030535981 conn(0x7fdd340d0350 msgr2=0x7fdd340d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:25.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.078+0000 7fdd3bc00640 1 -- 172.21.15.69:0/4030535981 shutdown_connections 2024-04-24T08:15:25.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.078+0000 7fdd3bc00640 1 -- 172.21.15.69:0/4030535981 wait complete. 2024-04-24T08:15:25.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.080+0000 7fdd3bc00640 1 Processor -- start 2024-04-24T08:15:25.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.080+0000 7fdd3bc00640 1 -- start start 2024-04-24T08:15:25.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.081+0000 7fdd3bc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd140039b0 0x7fdd3415f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:25.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.081+0000 7fdd3bc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3415faa0 0x7fdd341f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:25.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.081+0000 7fdd3bc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd341f2d30 0x7fdd341f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:25.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.081+0000 7fdd3bc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdd340b4b00 con 0x7fdd3415faa0 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.081+0000 7fdd3bc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3406f2b0 con 0x7fdd140039b0 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.081+0000 7fdd3bc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdd340ce960 con 0x7fdd341f2d30 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.081+0000 7fdd2be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3415faa0 0x7fdd341f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.082+0000 7fdd2be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3415faa0 0x7fdd341f56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53818/0 (socket says 172.21.15.69:53818) 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.082+0000 7fdd2be00640 1 -- 172.21.15.69:0/1068022531 learned_addr learned my addr 172.21.15.69:0/1068022531 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.082+0000 7fdd2be00640 1 -- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd341f2d30 msgr2=0x7fdd341f5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.082+0000 7fdd2be00640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd341f2d30 0x7fdd341f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.082+0000 7fdd2be00640 1 -- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd140039b0 msgr2=0x7fdd3415f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.082+0000 7fdd2be00640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd140039b0 0x7fdd3415f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.082+0000 7fdd2be00640 1 -- 172.21.15.69:0/1068022531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd34157b40 con 0x7fdd3415faa0 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.082+0000 7fdd2be00640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3415faa0 0x7fdd341f56d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00e7e0 tx=0x7fdd1c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.082+0000 7fdd31600640 1 -- 172.21.15.69:0/1068022531 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c017070 con 0x7fdd3415faa0 2024-04-24T08:15:25.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.083+0000 7fdd31600640 1 -- 172.21.15.69:0/1068022531 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd1c00c6d0 con 0x7fdd3415faa0 2024-04-24T08:15:25.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.083+0000 7fdd31600640 1 -- 172.21.15.69:0/1068022531 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c00f040 con 0x7fdd3415faa0 2024-04-24T08:15:25.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.083+0000 7fdd3bc00640 1 -- 172.21.15.69:0/1068022531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd34158370 con 0x7fdd3415faa0 2024-04-24T08:15:25.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.083+0000 7fdd3bc00640 1 -- 172.21.15.69:0/1068022531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd34158090 con 0x7fdd3415faa0 2024-04-24T08:15:25.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.084+0000 7fdd3bc00640 1 -- 172.21.15.69:0/1068022531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd34068a80 con 0x7fdd3415faa0 2024-04-24T08:15:25.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.085+0000 7fdd31600640 1 -- 172.21.15.69:0/1068022531 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdd1c01a070 con 0x7fdd3415faa0 2024-04-24T08:15:25.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.085+0000 7fdd31600640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd1006c8f0 0x7fdd1006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:25.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.086+0000 7fdd31600640 1 -- 172.21.15.69:0/1068022531 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fdd1c08b7a0 con 0x7fdd3415faa0 2024-04-24T08:15:25.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.086+0000 7fdd39200640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd1006c8f0 0x7fdd1006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:25.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.087+0000 7fdd39200640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd1006c8f0 0x7fdd1006edb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400ba30 tx=0x7fdd2402c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:25.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.090+0000 7fdd31600640 1 -- 172.21.15.69:0/1068022531 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd1c059bc0 con 0x7fdd3415faa0 2024-04-24T08:15:25.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.331+0000 7fdd3bc00640 1 -- 172.21.15.69:0/1068022531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fdd340cddc0 con 0x7fdd3415faa0 2024-04-24T08:15:25.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.996+0000 7fdd31600640 1 -- 172.21.15.69:0/1068022531 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v60) ==== 118+0+0 (secure 0 0 0) 0x7fdd1c0594c0 con 0x7fdd3415faa0 2024-04-24T08:15:26.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.999+0000 7fdd3bc00640 1 -- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd1006c8f0 msgr2=0x7fdd1006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:26.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.999+0000 7fdd3bc00640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd1006c8f0 0x7fdd1006edb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400ba30 tx=0x7fdd2402c000 comp rx=0 tx=0).stop 2024-04-24T08:15:26.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.999+0000 7fdd3bc00640 1 -- 172.21.15.69:0/1068022531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3415faa0 msgr2=0x7fdd341f56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:26.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:25.999+0000 7fdd3bc00640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3415faa0 0x7fdd341f56d0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00e7e0 tx=0x7fdd1c00eca0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.000+0000 7fdd3bc00640 1 -- 172.21.15.69:0/1068022531 shutdown_connections 2024-04-24T08:15:26.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.000+0000 7fdd3bc00640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd1006c8f0 0x7fdd1006edb0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.000+0000 7fdd3bc00640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd3415faa0 0x7fdd341f56d0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.000+0000 7fdd3bc00640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd341f2d30 0x7fdd341f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.000+0000 7fdd3bc00640 1 --2- 172.21.15.69:0/1068022531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd140039b0 0x7fdd3415f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.000+0000 7fdd3bc00640 1 -- 172.21.15.69:0/1068022531 >> 172.21.15.69:0/1068022531 conn(0x7fdd340d0350 msgr2=0x7fdd340b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:26.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.000+0000 7fdd3bc00640 1 -- 172.21.15.69:0/1068022531 shutdown_connections 2024-04-24T08:15:26.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.000+0000 7fdd3bc00640 1 -- 172.21.15.69:0/1068022531 wait complete. 2024-04-24T08:15:26.022 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-24T08:15:26.022 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T08:15:26.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.225+0000 7fed71e00640 1 Processor -- start 2024-04-24T08:15:26.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.225+0000 7fed71e00640 1 -- start start 2024-04-24T08:15:26.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.226+0000 7fed71e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fed6c144850 0x7fed6c06fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:26.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.226+0000 7fed71e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed6c15b350 con 0x7fed6c06dbf0 2024-04-24T08:15:26.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.226+0000 7fed71e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed6c0cf040 con 0x7fed6c071d00 2024-04-24T08:15:26.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.226+0000 7fed71e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fed6c0cfc50 con 0x7fed6c144850 2024-04-24T08:15:26.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.227+0000 7fed6b400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fed6c06dbf0 0x7fed6c06e020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47684/0 (socket says 172.21.15.69:47684) 2024-04-24T08:15:26.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.227+0000 7fed6b400640 1 -- 172.21.15.69:0/2136803517 learned_addr learned my addr 172.21.15.69:0/2136803517 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:26.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.227+0000 7fed6aa00640 1 --2- 172.21.15.69:0/2136803517 >> v2:172.21.15.121:3300/0 conn(0x7fed6c144850 0x7fed6c06fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:26.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.227+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 190797407 0 0) 0x7fed6c15b350 con 0x7fed6c06dbf0 2024-04-24T08:15:26.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.227+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed50003520 con 0x7fed6c06dbf0 2024-04-24T08:15:26.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 182971239 0 0) 0x7fed50003520 con 0x7fed6c06dbf0 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 >> v2:172.21.15.121:3300/0 conn(0x7fed6c144850 msgr2=0x7fed6c06fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 --2- 172.21.15.69:0/2136803517 >> v2:172.21.15.121:3300/0 conn(0x7fed6c144850 0x7fed6c06fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 >> v1:172.21.15.121:6789/0 conn(0x7fed6c071d00 legacy=0x7fed6c072160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed6c0cf320 con 0x7fed6c06dbf0 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fed640019f0 con 0x7fed6c06dbf0 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 >> v1:172.21.15.69:6789/0 conn(0x7fed6c06dbf0 legacy=0x7fed6c06e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed500039b0 0x7fed50003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed50004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed500052e0 0x7fed5000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c0cfc50 con 0x7fed50004800 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fed5000c1b0 con 0x7fed500039b0 2024-04-24T08:15:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed5000c3b0 con 0x7fed500052e0 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.228+0000 7fed62000640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed50004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.229+0000 7fed62000640 1 -- 172.21.15.69:0/2136803517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed500052e0 msgr2=0x7fed5000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.229+0000 7fed62000640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed500052e0 0x7fed5000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.229+0000 7fed62000640 1 -- 172.21.15.69:0/2136803517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed500039b0 msgr2=0x7fed50003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.229+0000 7fed6aa00640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed500039b0 0x7fed50003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.229+0000 7fed62000640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed500039b0 0x7fed50003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.229+0000 7fed62000640 1 -- 172.21.15.69:0/2136803517 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed50003520 con 0x7fed50004800 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.229+0000 7fed6b400640 1 -- 172.21.15.69:0/2136803517 reap_dead start 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.229+0000 7fed6aa00640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed500039b0 0x7fed50003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.230+0000 7fed62000640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed50004c80 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fed54001720 tx=0x7fed50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f557f652ee7d393 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.230+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed54001370 con 0x7fed50004800 2024-04-24T08:15:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.230+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed540052d0 con 0x7fed50004800 2024-04-24T08:15:26.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.230+0000 7fed6a000640 1 -- 172.21.15.69:0/2136803517 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5402ab70 con 0x7fed50004800 2024-04-24T08:15:26.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.231+0000 7fed71e00640 1 -- 172.21.15.69:0/2136803517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 msgr2=0x7fed50004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:26.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.231+0000 7fed71e00640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed50004c80 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fed54001720 tx=0x7fed50005830 comp rx=0 tx=0).stop 2024-04-24T08:15:26.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.231+0000 7fed71e00640 1 -- 172.21.15.69:0/2136803517 shutdown_connections 2024-04-24T08:15:26.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.231+0000 7fed71e00640 1 --2- 172.21.15.69:0/2136803517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed50004c80 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.231+0000 7fed71e00640 1 -- 172.21.15.69:0/2136803517 >> 172.21.15.69:0/2136803517 conn(0x7fed6c0d0350 msgr2=0x7fed6c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:26.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.231+0000 7fed71e00640 1 -- 172.21.15.69:0/2136803517 shutdown_connections 2024-04-24T08:15:26.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.231+0000 7fed71e00640 1 -- 172.21.15.69:0/2136803517 wait complete. 2024-04-24T08:15:26.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.233+0000 7fed71e00640 1 Processor -- start 2024-04-24T08:15:26.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.233+0000 7fed71e00640 1 -- start start 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed71e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed6c0aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed71e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6c1f4a40 0x7fed6c099890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed71e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed6c0c95e0 0x7fed6c0c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed71e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c073e10 con 0x7fed50004800 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed71e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fed6c0cf8e0 con 0x7fed6c0c95e0 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed71e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed6c073f90 con 0x7fed6c1f4a40 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed6aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed6c0aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed6aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed6c0aeca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53832/0 (socket says 172.21.15.69:53832) 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed6aa00640 1 -- 172.21.15.69:0/1965648712 learned_addr learned my addr 172.21.15.69:0/1965648712 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed6aa00640 1 -- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6c1f4a40 msgr2=0x7fed6c099890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed6aa00640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6c1f4a40 0x7fed6c099890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed6aa00640 1 -- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed6c0c95e0 msgr2=0x7fed6c0c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed6aa00640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed6c0c95e0 0x7fed6c0c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:26.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.234+0000 7fed6aa00640 1 -- 172.21.15.69:0/1965648712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed6c1f4f70 con 0x7fed50004800 2024-04-24T08:15:26.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.235+0000 7fed6aa00640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed6c0aeca0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00f8b0 tx=0x7fed5c00fd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:26.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.235+0000 7fed63e00640 1 -- 172.21.15.69:0/1965648712 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c00b040 con 0x7fed50004800 2024-04-24T08:15:26.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.235+0000 7fed63e00640 1 -- 172.21.15.69:0/1965648712 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed5c00dc50 con 0x7fed50004800 2024-04-24T08:15:26.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.235+0000 7fed63e00640 1 -- 172.21.15.69:0/1965648712 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed5c00fea0 con 0x7fed50004800 2024-04-24T08:15:26.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.235+0000 7fed71e00640 1 -- 172.21.15.69:0/1965648712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed6c1f5250 con 0x7fed50004800 2024-04-24T08:15:26.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.235+0000 7fed71e00640 1 -- 172.21.15.69:0/1965648712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed6c1f5810 con 0x7fed50004800 2024-04-24T08:15:26.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.236+0000 7fed71e00640 1 -- 172.21.15.69:0/1965648712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed6c068a80 con 0x7fed50004800 2024-04-24T08:15:26.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.237+0000 7fed63e00640 1 -- 172.21.15.69:0/1965648712 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fed5c012460 con 0x7fed50004800 2024-04-24T08:15:26.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.238+0000 7fed63e00640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed4806c920 0x7fed4806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:26.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.238+0000 7fed63e00640 1 -- 172.21.15.69:0/1965648712 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fed5c08c150 con 0x7fed50004800 2024-04-24T08:15:26.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.239+0000 7fed62000640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed4806c920 0x7fed4806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:26.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.240+0000 7fed62000640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed4806c920 0x7fed4806ede0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fed54032000 tx=0x7fed5402c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:26.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.243+0000 7fed63e00640 1 -- 172.21.15.69:0/1965648712 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fed5c017070 con 0x7fed50004800 2024-04-24T08:15:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:26.486+0000 7fed71e00640 1 -- 172.21.15.69:0/1965648712 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fed6c15c880 con 0x7fed50004800 2024-04-24T08:15:27.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.007+0000 7fed63e00640 1 -- 172.21.15.69:0/1965648712 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v61) ==== 122+0+0 (secure 0 0 0) 0x7fed5c05a4e0 con 0x7fed50004800 2024-04-24T08:15:27.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 -- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed4806c920 msgr2=0x7fed4806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed4806c920 0x7fed4806ede0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fed54032000 tx=0x7fed5402c000 comp rx=0 tx=0).stop 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 -- 172.21.15.69:0/1965648712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 msgr2=0x7fed6c0aeca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed6c0aeca0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00f8b0 tx=0x7fed5c00fd70 comp rx=0 tx=0).stop 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 -- 172.21.15.69:0/1965648712 shutdown_connections 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed4806c920 0x7fed4806ede0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed6c0c95e0 0x7fed6c0c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed6c1f4a40 0x7fed6c099890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 --2- 172.21.15.69:0/1965648712 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed50004800 0x7fed6c0aeca0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 -- 172.21.15.69:0/1965648712 >> 172.21.15.69:0/1965648712 conn(0x7fed6c0d0350 msgr2=0x7fed6c0afdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 -- 172.21.15.69:0/1965648712 shutdown_connections 2024-04-24T08:15:27.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.011+0000 7fed71e00640 1 -- 172.21.15.69:0/1965648712 wait complete. 2024-04-24T08:15:27.033 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T08:15:27.033 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.236+0000 7f8d4b200640 1 Processor -- start 2024-04-24T08:15:27.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.236+0000 7f8d4b200640 1 -- start start 2024-04-24T08:15:27.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.237+0000 7f8d4b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d44071e50 0x7f8d4415ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:27.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.237+0000 7f8d4b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d44144db0 con 0x7f8d44162300 2024-04-24T08:15:27.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.237+0000 7f8d4b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d4415f540 con 0x7f8d44072230 2024-04-24T08:15:27.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.237+0000 7f8d4b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8d4415f820 con 0x7f8d44071e50 2024-04-24T08:15:27.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.238+0000 7f8d49e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d44071e50 0x7f8d4415ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:27.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.238+0000 7f8d49400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8d44162300 0x7f8d4406e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47700/0 (socket says 172.21.15.69:47700) 2024-04-24T08:15:27.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.238+0000 7f8d49400640 1 -- 172.21.15.69:0/1778259639 learned_addr learned my addr 172.21.15.69:0/1778259639 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:27.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.238+0000 7f8d49e00640 1 -- 172.21.15.69:0/1778259639 >> v1:172.21.15.121:6789/0 conn(0x7f8d44072230 legacy=0x7f8d441446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:27.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.238+0000 7f8d49e00640 1 -- 172.21.15.69:0/1778259639 >> v1:172.21.15.69:6789/0 conn(0x7f8d44162300 legacy=0x7f8d4406e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.238+0000 7f8d49e00640 1 -- 172.21.15.69:0/1778259639 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d4415fb00 con 0x7f8d44071e50 2024-04-24T08:15:27.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.240+0000 7f8d49e00640 1 --2- 172.21.15.69:0/1778259639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 0x7f8d4415ecc0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f8d380097a0 tx=0x7f8d3802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6a8678059ca036 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:27.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.240+0000 7f8d48a00640 1 -- 172.21.15.69:0/1778259639 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d38037070 con 0x7f8d44071e50 2024-04-24T08:15:27.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.240+0000 7f8d48a00640 1 -- 172.21.15.69:0/1778259639 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d38032440 con 0x7f8d44071e50 2024-04-24T08:15:27.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.240+0000 7f8d48a00640 1 -- 172.21.15.69:0/1778259639 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d38032810 con 0x7f8d44071e50 2024-04-24T08:15:27.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.240+0000 7f8d4b200640 1 -- 172.21.15.69:0/1778259639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 msgr2=0x7f8d4415ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.240+0000 7f8d4b200640 1 --2- 172.21.15.69:0/1778259639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 0x7f8d4415ecc0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f8d380097a0 tx=0x7f8d3802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.241+0000 7f8d4b200640 1 -- 172.21.15.69:0/1778259639 shutdown_connections 2024-04-24T08:15:27.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.241+0000 7f8d4b200640 1 --2- 172.21.15.69:0/1778259639 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 0x7f8d4415ecc0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.241+0000 7f8d4b200640 1 -- 172.21.15.69:0/1778259639 >> 172.21.15.69:0/1778259639 conn(0x7f8d440d0160 msgr2=0x7f8d440d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:27.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.241+0000 7f8d4b200640 1 -- 172.21.15.69:0/1778259639 shutdown_connections 2024-04-24T08:15:27.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.241+0000 7f8d4b200640 1 -- 172.21.15.69:0/1778259639 wait complete. 2024-04-24T08:15:27.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.242+0000 7f8d4b200640 1 Processor -- start 2024-04-24T08:15:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.243+0000 7f8d4b200640 1 -- start start 2024-04-24T08:15:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.243+0000 7f8d4b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 0x7f8d44156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.243+0000 7f8d4b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d44072230 0x7f8d440b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.243+0000 7f8d4b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d44162300 0x7f8d440b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.243+0000 7f8d4b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4406e9d0 con 0x7f8d44071e50 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.243+0000 7f8d4b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4415f200 con 0x7f8d44072230 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.243+0000 7f8d4b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8d4406eb50 con 0x7f8d44162300 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.244+0000 7f8d49e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 0x7f8d44156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.244+0000 7f8d49e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 0x7f8d44156c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53856/0 (socket says 172.21.15.69:53856) 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.244+0000 7f8d49e00640 1 -- 172.21.15.69:0/346517817 learned_addr learned my addr 172.21.15.69:0/346517817 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.244+0000 7f8d49e00640 1 -- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d44162300 msgr2=0x7f8d440b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.244+0000 7f8d49e00640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d44162300 0x7f8d440b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.244+0000 7f8d49e00640 1 -- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d44072230 msgr2=0x7f8d440b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.244+0000 7f8d49e00640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d44072230 0x7f8d440b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.244+0000 7f8d49e00640 1 -- 172.21.15.69:0/346517817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d441e81e0 con 0x7f8d44071e50 2024-04-24T08:15:27.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.245+0000 7f8d49e00640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 0x7f8d44156c70 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f8d38000c00 tx=0x7f8d380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:27.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.245+0000 7f8d3e000640 1 -- 172.21.15.69:0/346517817 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d38037040 con 0x7f8d44071e50 2024-04-24T08:15:27.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.245+0000 7f8d3e000640 1 -- 172.21.15.69:0/346517817 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d38042070 con 0x7f8d44071e50 2024-04-24T08:15:27.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.245+0000 7f8d3e000640 1 -- 172.21.15.69:0/346517817 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d38030a70 con 0x7f8d44071e50 2024-04-24T08:15:27.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.245+0000 7f8d4b200640 1 -- 172.21.15.69:0/346517817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d4415c250 con 0x7f8d44071e50 2024-04-24T08:15:27.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.245+0000 7f8d4b200640 1 -- 172.21.15.69:0/346517817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d441e8730 con 0x7f8d44071e50 2024-04-24T08:15:27.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.246+0000 7f8d4b200640 1 -- 172.21.15.69:0/346517817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d440ad5b0 con 0x7f8d44071e50 2024-04-24T08:15:27.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.247+0000 7f8d3e000640 1 -- 172.21.15.69:0/346517817 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8d38030c10 con 0x7f8d44071e50 2024-04-24T08:15:27.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.248+0000 7f8d3e000640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2806c8f0 0x7f8d2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:27.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.248+0000 7f8d3e000640 1 -- 172.21.15.69:0/346517817 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f8d380ac640 con 0x7f8d44071e50 2024-04-24T08:15:27.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.249+0000 7f8d49400640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2806c8f0 0x7f8d2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:27.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.250+0000 7f8d49400640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2806c8f0 0x7f8d2806edb0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8d34000f60 tx=0x7f8d34007a80 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:27.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.253+0000 7f8d3e000640 1 -- 172.21.15.69:0/346517817 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d440ad5b0 con 0x7f8d44071e50 2024-04-24T08:15:27.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.494+0000 7f8d4b200640 1 -- 172.21.15.69:0/346517817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d440d0d90 con 0x7f8d44071e50 2024-04-24T08:15:27.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.495+0000 7f8d3e000640 1 -- 172.21.15.69:0/346517817 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5180 (secure 0 0 0) 0x7f8d38045360 con 0x7f8d44071e50 2024-04-24T08:15:27.496 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:27.496 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7581,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/2529876410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2529876410},{"type":"v1","addr":"172.21.15.69:6833","nonce":2529876410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:15:26.487664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7593":{"gid":7593,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1179683245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1179683245},{"type":"v1","addr":"172.21.15.121:6837","nonce":1179683245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7593,"qdb_cluster":[7593]},"id":6}]} 2024-04-24T08:15:27.496 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 61 2024-04-24T08:15:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.498+0000 7f8d4b200640 1 -- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2806c8f0 msgr2=0x7f8d2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.498+0000 7f8d4b200640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2806c8f0 0x7f8d2806edb0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8d34000f60 tx=0x7f8d34007a80 comp rx=0 tx=0).stop 2024-04-24T08:15:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.498+0000 7f8d4b200640 1 -- 172.21.15.69:0/346517817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 msgr2=0x7f8d44156c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.498+0000 7f8d4b200640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 0x7f8d44156c70 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f8d38000c00 tx=0x7f8d380046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.498+0000 7f8d4b200640 1 -- 172.21.15.69:0/346517817 shutdown_connections 2024-04-24T08:15:27.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.498+0000 7f8d4b200640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d2806c8f0 0x7f8d2806edb0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.498+0000 7f8d4b200640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d44162300 0x7f8d440b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.498+0000 7f8d4b200640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d44072230 0x7f8d440b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.498+0000 7f8d4b200640 1 --2- 172.21.15.69:0/346517817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d44071e50 0x7f8d44156c70 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.498+0000 7f8d4b200640 1 -- 172.21.15.69:0/346517817 >> 172.21.15.69:0/346517817 conn(0x7f8d440d0160 msgr2=0x7f8d440af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.499+0000 7f8d4b200640 1 -- 172.21.15.69:0/346517817 shutdown_connections 2024-04-24T08:15:27.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.499+0000 7f8d4b200640 1 -- 172.21.15.69:0/346517817 wait complete. 2024-04-24T08:15:27.517 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:27.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.717+0000 7f1d09800640 1 Processor -- start 2024-04-24T08:15:27.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.717+0000 7f1d09800640 1 -- start start 2024-04-24T08:15:27.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.718+0000 7f1d09800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d040b30f0 0x7f1d04167470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:27.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.718+0000 7f1d09800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d0415d660 con 0x7f1d0415d1d0 2024-04-24T08:15:27.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.719+0000 7f1d09800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d0406dc30 con 0x7f1d04156c60 2024-04-24T08:15:27.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.719+0000 7f1d09800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1d0406df10 con 0x7f1d040b30f0 2024-04-24T08:15:27.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.719+0000 7f1cfb400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1d0415d1d0 0x7f1d040ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47702/0 (socket says 172.21.15.69:47702) 2024-04-24T08:15:27.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.719+0000 7f1cfb400640 1 -- 172.21.15.69:0/832927934 learned_addr learned my addr 172.21.15.69:0/832927934 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:27.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.719+0000 7f1d08e00640 1 --2- 172.21.15.69:0/832927934 >> v2:172.21.15.69:3300/0 conn(0x7f1d040b30f0 0x7f1d04167470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:27.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.719+0000 7f1d08e00640 1 -- 172.21.15.69:0/832927934 >> v1:172.21.15.121:6789/0 conn(0x7f1d04156c60 legacy=0x7f1d0415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:27.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.719+0000 7f1d08e00640 1 -- 172.21.15.69:0/832927934 >> v1:172.21.15.69:6789/0 conn(0x7f1d0415d1d0 legacy=0x7f1d040ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.719+0000 7f1d08e00640 1 -- 172.21.15.69:0/832927934 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0406e1f0 con 0x7f1d040b30f0 2024-04-24T08:15:27.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.720+0000 7f1d08e00640 1 --2- 172.21.15.69:0/832927934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d040b30f0 0x7f1d04167470 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009d90 tx=0x7f1cfc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875abb948e398ccf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:27.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.721+0000 7f1d03400640 1 -- 172.21.15.69:0/832927934 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc007740 con 0x7f1d040b30f0 2024-04-24T08:15:27.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.721+0000 7f1d03400640 1 -- 172.21.15.69:0/832927934 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cfc0078e0 con 0x7f1d040b30f0 2024-04-24T08:15:27.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.721+0000 7f1d03400640 1 -- 172.21.15.69:0/832927934 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc007cb0 con 0x7f1d040b30f0 2024-04-24T08:15:27.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.721+0000 7f1d09800640 1 -- 172.21.15.69:0/832927934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d040b30f0 msgr2=0x7f1d04167470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.721+0000 7f1d09800640 1 --2- 172.21.15.69:0/832927934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d040b30f0 0x7f1d04167470 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009d90 tx=0x7f1cfc02a6a0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.721+0000 7f1d09800640 1 -- 172.21.15.69:0/832927934 shutdown_connections 2024-04-24T08:15:27.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.721+0000 7f1d09800640 1 --2- 172.21.15.69:0/832927934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d040b30f0 0x7f1d04167470 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.721+0000 7f1d09800640 1 -- 172.21.15.69:0/832927934 >> 172.21.15.69:0/832927934 conn(0x7f1d040d0160 msgr2=0x7f1d040d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:27.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.721+0000 7f1d09800640 1 -- 172.21.15.69:0/832927934 shutdown_connections 2024-04-24T08:15:27.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.721+0000 7f1d09800640 1 -- 172.21.15.69:0/832927934 wait complete. 2024-04-24T08:15:27.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.723+0000 7f1d09800640 1 Processor -- start 2024-04-24T08:15:27.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.723+0000 7f1d09800640 1 -- start start 2024-04-24T08:15:27.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.724+0000 7f1d09800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d040b30f0 0x7f1d040997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:27.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.724+0000 7f1d09800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d04156c60 0x7f1d04158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:27.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.724+0000 7f1d09800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d04155b50 0x7f1d041585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:27.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.724+0000 7f1d09800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d040ad7e0 con 0x7f1d04155b50 2024-04-24T08:15:27.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.724+0000 7f1d09800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0415d940 con 0x7f1d04156c60 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.724+0000 7f1d09800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1d040ad960 con 0x7f1d040b30f0 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.724+0000 7f1d08e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d04155b50 0x7f1d041585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.724+0000 7f1d08e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d04155b50 0x7f1d041585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53882/0 (socket says 172.21.15.69:53882) 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.724+0000 7f1d08e00640 1 -- 172.21.15.69:0/766305626 learned_addr learned my addr 172.21.15.69:0/766305626 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.725+0000 7f1d03e00640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d040b30f0 0x7f1d040997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.725+0000 7f1cfb400640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d04156c60 0x7f1d04158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.725+0000 7f1d08e00640 1 -- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d040b30f0 msgr2=0x7f1d040997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.725+0000 7f1d08e00640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d040b30f0 0x7f1d040997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.725+0000 7f1d08e00640 1 -- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d04156c60 msgr2=0x7f1d04158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.725+0000 7f1d08e00640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d04156c60 0x7f1d04158090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.725+0000 7f1d08e00640 1 -- 172.21.15.69:0/766305626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d040ced90 con 0x7f1d04155b50 2024-04-24T08:15:27.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.725+0000 7f1d08e00640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d04155b50 0x7f1d041585d0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009d90 tx=0x7f1cfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:27.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.726+0000 7f1d00c00640 1 -- 172.21.15.69:0/766305626 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc040070 con 0x7f1d04155b50 2024-04-24T08:15:27.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.726+0000 7f1d00c00640 1 -- 172.21.15.69:0/766305626 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cfc031c90 con 0x7f1d04155b50 2024-04-24T08:15:27.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.726+0000 7f1d00c00640 1 -- 172.21.15.69:0/766305626 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc03c640 con 0x7f1d04155b50 2024-04-24T08:15:27.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.726+0000 7f1d09800640 1 -- 172.21.15.69:0/766305626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d040cf070 con 0x7f1d04155b50 2024-04-24T08:15:27.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.726+0000 7f1d09800640 1 -- 172.21.15.69:0/766305626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d041e6380 con 0x7f1d04155b50 2024-04-24T08:15:27.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.727+0000 7f1d09800640 1 -- 172.21.15.69:0/766305626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d04075d50 con 0x7f1d04155b50 2024-04-24T08:15:27.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.728+0000 7f1d00c00640 1 -- 172.21.15.69:0/766305626 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1cfc039040 con 0x7f1d04155b50 2024-04-24T08:15:27.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.729+0000 7f1d00c00640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ce006c8f0 0x7f1ce006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:27.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.730+0000 7f1d00c00640 1 -- 172.21.15.69:0/766305626 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f1cfc0acdf0 con 0x7f1d04155b50 2024-04-24T08:15:27.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.730+0000 7f1d03e00640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ce006c8f0 0x7f1ce006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:27.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.731+0000 7f1d03e00640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ce006c8f0 0x7f1ce006edb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4004ea0 tx=0x7f1cf4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:27.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.734+0000 7f1d00c00640 1 -- 172.21.15.69:0/766305626 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1cfc039330 con 0x7f1d04155b50 2024-04-24T08:15:27.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.935+0000 7f1d09800640 1 -- 172.21.15.69:0/766305626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d0406f9e0 con 0x7f1d04155b50 2024-04-24T08:15:27.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.936+0000 7f1d00c00640 1 -- 172.21.15.69:0/766305626 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+16636 (secure 0 0 0) 0x7f1cfc07b240 con 0x7f1d04155b50 2024-04-24T08:15:27.937 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:27.937 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":82,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:15:24.982737+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:27.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 -- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ce006c8f0 msgr2=0x7f1ce006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ce006c8f0 0x7f1ce006edb0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4004ea0 tx=0x7f1cf4009040 comp rx=0 tx=0).stop 2024-04-24T08:15:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 -- 172.21.15.69:0/766305626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d04155b50 msgr2=0x7f1d041585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d04155b50 0x7f1d041585d0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009d90 tx=0x7f1cfc004a40 comp rx=0 tx=0).stop 2024-04-24T08:15:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 -- 172.21.15.69:0/766305626 shutdown_connections 2024-04-24T08:15:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ce006c8f0 0x7f1ce006edb0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d04155b50 0x7f1d041585d0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d04156c60 0x7f1d04158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 --2- 172.21.15.69:0/766305626 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d040b30f0 0x7f1d040997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 -- 172.21.15.69:0/766305626 >> 172.21.15.69:0/766305626 conn(0x7f1d040d0160 msgr2=0x7f1d0415a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 -- 172.21.15.69:0/766305626 shutdown_connections 2024-04-24T08:15:27.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:27.939+0000 7f1d09800640 1 -- 172.21.15.69:0/766305626 wait complete. 2024-04-24T08:15:27.959 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T08:15:27.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:15:28.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.155+0000 7f3b1b600640 1 Processor -- start 2024-04-24T08:15:28.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.155+0000 7f3b1b600640 1 -- start start 2024-04-24T08:15:28.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.157+0000 7f3b1b600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b14145fe0 0x7f3b1406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.157+0000 7f3b1b600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b140cf3d0 con 0x7f3b140d0300 2024-04-24T08:15:28.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.157+0000 7f3b1b600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b140cf6b0 con 0x7f3b14145a90 2024-04-24T08:15:28.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.157+0000 7f3b1b600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3b140cf990 con 0x7f3b14145fe0 2024-04-24T08:15:28.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.157+0000 7f3b18c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3b140d0300 0x7f3b140d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47710/0 (socket says 172.21.15.69:47710) 2024-04-24T08:15:28.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.157+0000 7f3b18c00640 1 -- 172.21.15.69:0/707910666 learned_addr learned my addr 172.21.15.69:0/707910666 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:28.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.157+0000 7f3b19600640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b14145fe0 0x7f3b1406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522922553 0 0) 0x7f3b140cf3d0 con 0x7f3b140d0300 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3af8003520 con 0x7f3b140d0300 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 513857149 0 0) 0x7f3af8003520 con 0x7f3b140d0300 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b14145fe0 msgr2=0x7f3b1406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b14145fe0 0x7f3b1406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 >> v1:172.21.15.121:6789/0 conn(0x7f3b14145a90 legacy=0x7f3b14160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b140cfc70 con 0x7f3b140d0300 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f3b04001670 con 0x7f3b140d0300 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 >> v1:172.21.15.69:6789/0 conn(0x7f3b140d0300 legacy=0x7f3b140d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3af80039e0 0x7f3af8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af8004800 0x7f3af8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b14145fe0 0x7f3af80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3b140cf990 con 0x7f3af80039e0 2024-04-24T08:15:28.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3af800baa0 con 0x7f3af8004800 2024-04-24T08:15:28.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3af800bca0 con 0x7f3b14145fe0 2024-04-24T08:15:28.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.158+0000 7f3b18c00640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3af80039e0 0x7f3af8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.159+0000 7f3b0be00640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af8004800 0x7f3af8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.159+0000 7f3b19600640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b14145fe0 0x7f3af80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.159+0000 7f3b18c00640 1 -- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b14145fe0 msgr2=0x7f3af80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:28.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.159+0000 7f3b18c00640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b14145fe0 0x7f3af80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.159+0000 7f3b18c00640 1 -- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af8004800 msgr2=0x7f3af8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:28.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.159+0000 7f3b18c00640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af8004800 0x7f3af8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.159+0000 7f3b18c00640 1 -- 172.21.15.69:0/707910666 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af8003520 con 0x7f3af80039e0 2024-04-24T08:15:28.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.160+0000 7f3b18c00640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3af80039e0 0x7f3af8003e00 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400c210 tx=0x7f3af8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b8572439e4af50 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:28.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.160+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b04007d10 con 0x7f3af80039e0 2024-04-24T08:15:28.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.160+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b04005ad0 con 0x7f3af80039e0 2024-04-24T08:15:28.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.160+0000 7f3b13e00640 1 -- 172.21.15.69:0/707910666 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b04007070 con 0x7f3af80039e0 2024-04-24T08:15:28.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.161+0000 7f3b1b600640 1 -- 172.21.15.69:0/707910666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3af80039e0 msgr2=0x7f3af8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.161+0000 7f3b1b600640 1 --2- 172.21.15.69:0/707910666 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3af80039e0 0x7f3af8003e00 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f3b0400c210 tx=0x7f3af8005830 comp rx=0 tx=0).stop 2024-04-24T08:15:28.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.161+0000 7f3b19600640 1 -- 172.21.15.69:0/707910666 reap_dead start 2024-04-24T08:15:28.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.161+0000 7f3b1b600640 1 -- 172.21.15.69:0/707910666 shutdown_connections 2024-04-24T08:15:28.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.161+0000 7f3b1b600640 1 -- 172.21.15.69:0/707910666 >> 172.21.15.69:0/707910666 conn(0x7f3b14067d90 msgr2=0x7f3b1406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:28.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.161+0000 7f3b1b600640 1 -- 172.21.15.69:0/707910666 shutdown_connections 2024-04-24T08:15:28.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.161+0000 7f3b1b600640 1 -- 172.21.15.69:0/707910666 wait complete. 2024-04-24T08:15:28.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.163+0000 7f3b1b600640 1 Processor -- start 2024-04-24T08:15:28.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.163+0000 7f3b1b600640 1 -- start start 2024-04-24T08:15:28.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.164+0000 7f3b1b600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af80039e0 0x7f3b140c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.164+0000 7f3b1b600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b140ae7e0 0x7f3b141f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.164+0000 7f3b1b600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b140c6d90 0x7f3b141439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.164+0000 7f3b1b600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3b141458f0 con 0x7f3b140ae7e0 2024-04-24T08:15:28.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.164+0000 7f3b1b600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3b14145770 con 0x7f3af80039e0 2024-04-24T08:15:28.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.164+0000 7f3b1b600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3b140cdc30 con 0x7f3b140c6d90 2024-04-24T08:15:28.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.164+0000 7f3b0be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b140ae7e0 0x7f3b141f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.164+0000 7f3b0be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b140ae7e0 0x7f3b141f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53894/0 (socket says 172.21.15.69:53894) 2024-04-24T08:15:28.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.164+0000 7f3b0be00640 1 -- 172.21.15.69:0/2866028724 learned_addr learned my addr 172.21.15.69:0/2866028724 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:28.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.164+0000 7f3b18c00640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af80039e0 0x7f3b140c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b19600640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b140c6d90 0x7f3b141439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b0be00640 1 -- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b140c6d90 msgr2=0x7f3b141439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b0be00640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b140c6d90 0x7f3b141439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b0be00640 1 -- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af80039e0 msgr2=0x7f3b140c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b0be00640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af80039e0 0x7f3b140c5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b0be00640 1 -- 172.21.15.69:0/2866028724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b140ac530 con 0x7f3b140ae7e0 2024-04-24T08:15:28.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b19600640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b140c6d90 0x7f3b141439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:28.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b0be00640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b140ae7e0 0x7f3b141f2840 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00e4d0 tx=0x7f3afc00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:28.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b11600640 1 -- 172.21.15.69:0/2866028724 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0052e0 con 0x7f3b140ae7e0 2024-04-24T08:15:28.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b11600640 1 -- 172.21.15.69:0/2866028724 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3afc018070 con 0x7f3b140ae7e0 2024-04-24T08:15:28.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b11600640 1 -- 172.21.15.69:0/2866028724 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc003810 con 0x7f3b140ae7e0 2024-04-24T08:15:28.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b1b600640 1 -- 172.21.15.69:0/2866028724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b141f2d80 con 0x7f3b140ae7e0 2024-04-24T08:15:28.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.165+0000 7f3b1b600640 1 -- 172.21.15.69:0/2866028724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b140ac7a0 con 0x7f3b140ae7e0 2024-04-24T08:15:28.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.167+0000 7f3b1b600640 1 -- 172.21.15.69:0/2866028724 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b14068a80 con 0x7f3b140ae7e0 2024-04-24T08:15:28.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.167+0000 7f3b11600640 1 -- 172.21.15.69:0/2866028724 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3afc0039b0 con 0x7f3b140ae7e0 2024-04-24T08:15:28.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.168+0000 7f3b11600640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3af006c8f0 0x7f3af006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.168+0000 7f3b11600640 1 -- 172.21.15.69:0/2866028724 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f3afc08b800 con 0x7f3b140ae7e0 2024-04-24T08:15:28.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.169+0000 7f3b18c00640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3af006c8f0 0x7f3af006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.170+0000 7f3b18c00640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3af006c8f0 0x7f3af006edb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3b04032000 tx=0x7f3b0402d000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:28.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.173+0000 7f3b11600640 1 -- 172.21.15.69:0/2866028724 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3afc059c30 con 0x7f3b140ae7e0 2024-04-24T08:15:28.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.363+0000 7f3b1b600640 1 -- 172.21.15.69:0/2866028724 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3b140d1120 con 0x7f3af006c8f0 2024-04-24T08:15:28.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.383+0000 7f3b11600640 1 -- 172.21.15.69:0/2866028724 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+246754 (secure 0 0 0) 0x7f3b140d1120 con 0x7f3af006c8f0 2024-04-24T08:15:28.387 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:28.389 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:15:28.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 -- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3af006c8f0 msgr2=0x7f3af006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3af006c8f0 0x7f3af006edb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3b04032000 tx=0x7f3b0402d000 comp rx=0 tx=0).stop 2024-04-24T08:15:28.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 -- 172.21.15.69:0/2866028724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b140ae7e0 msgr2=0x7f3b141f2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b140ae7e0 0x7f3b141f2840 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00e4d0 tx=0x7f3afc00e990 comp rx=0 tx=0).stop 2024-04-24T08:15:28.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 -- 172.21.15.69:0/2866028724 shutdown_connections 2024-04-24T08:15:28.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3af006c8f0 0x7f3af006edb0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b140ae7e0 0x7f3b141f2840 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3b140c6d90 0x7f3b141439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 --2- 172.21.15.69:0/2866028724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3af80039e0 0x7f3b140c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 -- 172.21.15.69:0/2866028724 >> 172.21.15.69:0/2866028724 conn(0x7f3b14067d90 msgr2=0x7f3b140ccdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:28.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 -- 172.21.15.69:0/2866028724 shutdown_connections 2024-04-24T08:15:28.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.389+0000 7f3b1b600640 1 -- 172.21.15.69:0/2866028724 wait complete. 2024-04-24T08:15:28.407 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-04-24T08:15:26.820341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258296,"kb_used_data":8840,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716424,"statfs":{"total":773094113280,"available":772829618176,"internally_reserved":0,"allocated":9052160,"data_stored":4523152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.864130"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:23.002656+0000","last_change":"2024-04-24T08:15:23.002656+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:23.002656+0000","last_undegraded":"2024-04-24T08:15:23.002656+0000","last_fullsized":"2024-04-24T08:15:23.002656+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.978967+0000","last_change":"2024-04-24T08:15:22.978967+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.978967+0000","last_undegraded":"2024-04-24T08:15:22.978967+0000","last_fullsized":"2024-04-24T08:15:22.978967+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.999700+0000","last_change":"2024-04-24T08:15:22.999700+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.999700+0000","last_undegraded":"2024-04-24T08:15:22.999700+0000","last_fullsized":"2024-04-24T08:15:22.999700+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.990510+0000","last_change":"2024-04-24T08:15:22.990510+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.990510+0000","last_undegraded":"2024-04-24T08:15:22.990510+0000","last_fullsized":"2024-04-24T08:15:22.990510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.991068+0000","last_change":"2024-04-24T08:15:22.991068+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.991068+0000","last_undegraded":"2024-04-24T08:15:22.991068+0000","last_fullsized":"2024-04-24T08:15:22.991068+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.982994+0000","last_change":"2024-04-24T08:15:22.982994+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.982994+0000","last_undegraded":"2024-04-24T08:15:22.982994+0000","last_fullsized":"2024-04-24T08:15:22.982994+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.988156+0000","last_change":"2024-04-24T08:15:22.988156+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.988156+0000","last_undegraded":"2024-04-24T08:15:22.988156+0000","last_fullsized":"2024-04-24T08:15:22.988156+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.981094+0000","last_change":"2024-04-24T08:15:22.981094+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.981094+0000","last_undegraded":"2024-04-24T08:15:22.981094+0000","last_fullsized":"2024-04-24T08:15:22.981094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.983917+0000","last_change":"2024-04-24T08:15:22.983917+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.983917+0000","last_undegraded":"2024-04-24T08:15:22.983917+0000","last_fullsized":"2024-04-24T08:15:22.983917+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.994770+0000","last_change":"2024-04-24T08:15:22.994770+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.994770+0000","last_undegraded":"2024-04-24T08:15:22.994770+0000","last_fullsized":"2024-04-24T08:15:22.994770+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.978654+0000","last_change":"2024-04-24T08:15:22.978654+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.978654+0000","last_undegraded":"2024-04-24T08:15:22.978654+0000","last_fullsized":"2024-04-24T08:15:22.978654+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:23.002892+0000","last_change":"2024-04-24T08:15:23.002892+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:23.002892+0000","last_undegraded":"2024-04-24T08:15:23.002892+0000","last_fullsized":"2024-04-24T08:15:23.002892+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.990392+0000","last_change":"2024-04-24T08:15:22.990392+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.990392+0000","last_undegraded":"2024-04-24T08:15:22.990392+0000","last_fullsized":"2024-04-24T08:15:22.990392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.990091+0000","last_change":"2024-04-24T08:15:22.990091+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.990091+0000","last_undegraded":"2024-04-24T08:15:22.990091+0000","last_fullsized":"2024-04-24T08:15:22.990091+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.986255+0000","last_change":"2024-04-24T08:15:22.986255+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.986255+0000","last_undegraded":"2024-04-24T08:15:22.986255+0000","last_fullsized":"2024-04-24T08:15:22.986255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.989454+0000","last_change":"2024-04-24T08:15:22.989454+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.989454+0000","last_undegraded":"2024-04-24T08:15:22.989454+0000","last_fullsized":"2024-04-24T08:15:22.989454+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.990809+0000","last_change":"2024-04-24T08:15:22.990809+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.990809+0000","last_undegraded":"2024-04-24T08:15:22.990809+0000","last_fullsized":"2024-04-24T08:15:22.990809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.981633+0000","last_change":"2024-04-24T08:15:22.981633+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.981633+0000","last_undegraded":"2024-04-24T08:15:22.981633+0000","last_fullsized":"2024-04-24T08:15:22.981633+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.985250+0000","last_change":"2024-04-24T08:15:22.985250+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.985250+0000","last_undegraded":"2024-04-24T08:15:22.985250+0000","last_fullsized":"2024-04-24T08:15:22.985250+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.982780+0000","last_change":"2024-04-24T08:15:22.982780+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.982780+0000","last_undegraded":"2024-04-24T08:15:22.982780+0000","last_fullsized":"2024-04-24T08:15:22.982780+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.986345+0000","last_change":"2024-04-24T08:15:22.986345+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.986345+0000","last_undegraded":"2024-04-24T08:15:22.986345+0000","last_fullsized":"2024-04-24T08:15:22.986345+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.978818+0000","last_change":"2024-04-24T08:15:22.978818+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.978818+0000","last_undegraded":"2024-04-24T08:15:22.978818+0000","last_fullsized":"2024-04-24T08:15:22.978818+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.997477+0000","last_change":"2024-04-24T08:15:22.997477+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.997477+0000","last_undegraded":"2024-04-24T08:15:22.997477+0000","last_fullsized":"2024-04-24T08:15:22.997477+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.992637+0000","last_change":"2024-04-24T08:15:22.992637+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.992637+0000","last_undegraded":"2024-04-24T08:15:22.992637+0000","last_fullsized":"2024-04-24T08:15:22.992637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.978100+0000","last_change":"2024-04-24T08:15:22.978100+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.978100+0000","last_undegraded":"2024-04-24T08:15:22.978100+0000","last_fullsized":"2024-04-24T08:15:22.978100+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.997382+0000","last_change":"2024-04-24T08:15:22.997382+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.997382+0000","last_undegraded":"2024-04-24T08:15:22.997382+0000","last_fullsized":"2024-04-24T08:15:22.997382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.993136+0000","last_change":"2024-04-24T08:15:22.993136+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.993136+0000","last_undegraded":"2024-04-24T08:15:22.993136+0000","last_fullsized":"2024-04-24T08:15:22.993136+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:23.001508+0000","last_change":"2024-04-24T08:15:23.001508+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:23.001508+0000","last_undegraded":"2024-04-24T08:15:23.001508+0000","last_fullsized":"2024-04-24T08:15:23.001508+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.977390+0000","last_change":"2024-04-24T08:15:22.977390+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.977390+0000","last_undegraded":"2024-04-24T08:15:22.977390+0000","last_fullsized":"2024-04-24T08:15:22.977390+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.977881+0000","last_change":"2024-04-24T08:15:22.977881+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.977881+0000","last_undegraded":"2024-04-24T08:15:22.977881+0000","last_fullsized":"2024-04-24T08:15:22.977881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.987522+0000","last_change":"2024-04-24T08:15:22.987522+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.987522+0000","last_undegraded":"2024-04-24T08:15:22.987522+0000","last_fullsized":"2024-04-24T08:15:22.987522+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.987362+0000","last_change":"2024-04-24T08:15:22.987362+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.987362+0000","last_undegraded":"2024-04-24T08:15:22.987362+0000","last_fullsized":"2024-04-24T08:15:22.987362+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.992216+0000","last_change":"2024-04-24T08:15:22.992216+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.992216+0000","last_undegraded":"2024-04-24T08:15:22.992216+0000","last_fullsized":"2024-04-24T08:15:22.992216+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.010923+0000","last_change":"2024-04-24T08:15:23.008526+0000","last_active":"2024-04-24T08:15:23.010923+0000","last_peered":"2024-04-24T08:15:23.010923+0000","last_clean":"2024-04-24T08:15:23.010923+0000","last_became_active":"2024-04-24T08:15:23.008215+0000","last_became_peered":"2024-04-24T08:15:23.008215+0000","last_unstale":"2024-04-24T08:15:23.010923+0000","last_undegraded":"2024-04-24T08:15:23.010923+0000","last_fullsized":"2024-04-24T08:15:23.010923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:04:18.218365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:23.002184+0000","last_change":"2024-04-24T08:15:23.002184+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:23.002184+0000","last_undegraded":"2024-04-24T08:15:23.002184+0000","last_fullsized":"2024-04-24T08:15:23.002184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:22.997761+0000","last_change":"2024-04-24T08:15:22.994471+0000","last_active":"2024-04-24T08:15:22.997761+0000","last_peered":"2024-04-24T08:15:22.997761+0000","last_clean":"2024-04-24T08:15:22.997761+0000","last_became_active":"2024-04-24T08:15:22.993954+0000","last_became_peered":"2024-04-24T08:15:22.993954+0000","last_unstale":"2024-04-24T08:15:22.997761+0000","last_undegraded":"2024-04-24T08:15:22.997761+0000","last_fullsized":"2024-04-24T08:15:22.997761+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:37:59.257564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:23.001066+0000","last_change":"2024-04-24T08:15:23.001066+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:23.001066+0000","last_undegraded":"2024-04-24T08:15:23.001066+0000","last_fullsized":"2024-04-24T08:15:23.001066+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.003053+0000","last_change":"2024-04-24T08:15:23.000155+0000","last_active":"2024-04-24T08:15:23.003053+0000","last_peered":"2024-04-24T08:15:23.003053+0000","last_clean":"2024-04-24T08:15:23.003053+0000","last_became_active":"2024-04-24T08:15:22.998678+0000","last_became_peered":"2024-04-24T08:15:22.998678+0000","last_unstale":"2024-04-24T08:15:23.003053+0000","last_undegraded":"2024-04-24T08:15:23.003053+0000","last_fullsized":"2024-04-24T08:15:23.003053+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:37.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.994155+0000","last_change":"2024-04-24T08:15:22.994155+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.994155+0000","last_undegraded":"2024-04-24T08:15:22.994155+0000","last_fullsized":"2024-04-24T08:15:22.994155+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.001007+0000","last_change":"2024-04-24T08:15:22.996657+0000","last_active":"2024-04-24T08:15:23.001007+0000","last_peered":"2024-04-24T08:15:23.001007+0000","last_clean":"2024-04-24T08:15:23.001007+0000","last_became_active":"2024-04-24T08:15:22.996342+0000","last_became_peered":"2024-04-24T08:15:22.996342+0000","last_unstale":"2024-04-24T08:15:23.001007+0000","last_undegraded":"2024-04-24T08:15:23.001007+0000","last_fullsized":"2024-04-24T08:15:23.001007+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:09:44.565995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.977453+0000","last_change":"2024-04-24T08:15:22.977453+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.977453+0000","last_undegraded":"2024-04-24T08:15:22.977453+0000","last_fullsized":"2024-04-24T08:15:22.977453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.003603+0000","last_change":"2024-04-24T08:15:22.999054+0000","last_active":"2024-04-24T08:15:23.003603+0000","last_peered":"2024-04-24T08:15:23.003603+0000","last_clean":"2024-04-24T08:15:23.003603+0000","last_became_active":"2024-04-24T08:15:22.998710+0000","last_became_peered":"2024-04-24T08:15:22.998710+0000","last_unstale":"2024-04-24T08:15:23.003603+0000","last_undegraded":"2024-04-24T08:15:23.003603+0000","last_fullsized":"2024-04-24T08:15:23.003603+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:15:53.896120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.984838+0000","last_change":"2024-04-24T08:15:22.984838+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.984838+0000","last_undegraded":"2024-04-24T08:15:22.984838+0000","last_fullsized":"2024-04-24T08:15:22.984838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.003645+0000","last_change":"2024-04-24T08:15:22.997874+0000","last_active":"2024-04-24T08:15:23.003645+0000","last_peered":"2024-04-24T08:15:23.003645+0000","last_clean":"2024-04-24T08:15:23.003645+0000","last_became_active":"2024-04-24T08:15:22.994836+0000","last_became_peered":"2024-04-24T08:15:22.994836+0000","last_unstale":"2024-04-24T08:15:23.003645+0000","last_undegraded":"2024-04-24T08:15:23.003645+0000","last_fullsized":"2024-04-24T08:15:23.003645+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:50:35.997657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.986824+0000","last_change":"2024-04-24T08:15:22.986824+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.986824+0000","last_undegraded":"2024-04-24T08:15:22.986824+0000","last_fullsized":"2024-04-24T08:15:22.986824+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.003975+0000","last_change":"2024-04-24T08:15:23.000628+0000","last_active":"2024-04-24T08:15:23.003975+0000","last_peered":"2024-04-24T08:15:23.003975+0000","last_clean":"2024-04-24T08:15:23.003975+0000","last_became_active":"2024-04-24T08:15:23.000351+0000","last_became_peered":"2024-04-24T08:15:23.000351+0000","last_unstale":"2024-04-24T08:15:23.003975+0000","last_undegraded":"2024-04-24T08:15:23.003975+0000","last_fullsized":"2024-04-24T08:15:23.003975+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:29.597202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.983796+0000","last_change":"2024-04-24T08:15:22.983796+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.983796+0000","last_undegraded":"2024-04-24T08:15:22.983796+0000","last_fullsized":"2024-04-24T08:15:22.983796+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:22.997976+0000","last_change":"2024-04-24T08:15:22.996673+0000","last_active":"2024-04-24T08:15:22.997976+0000","last_peered":"2024-04-24T08:15:22.997976+0000","last_clean":"2024-04-24T08:15:22.997976+0000","last_became_active":"2024-04-24T08:15:22.996387+0000","last_became_peered":"2024-04-24T08:15:22.996387+0000","last_unstale":"2024-04-24T08:15:22.997976+0000","last_undegraded":"2024-04-24T08:15:22.997976+0000","last_fullsized":"2024-04-24T08:15:22.997976+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:51.595217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.985722+0000","last_change":"2024-04-24T08:15:22.985722+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.985722+0000","last_undegraded":"2024-04-24T08:15:22.985722+0000","last_fullsized":"2024-04-24T08:15:22.985722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.010899+0000","last_change":"2024-04-24T08:15:23.008510+0000","last_active":"2024-04-24T08:15:23.010899+0000","last_peered":"2024-04-24T08:15:23.010899+0000","last_clean":"2024-04-24T08:15:23.010899+0000","last_became_active":"2024-04-24T08:15:23.008190+0000","last_became_peered":"2024-04-24T08:15:23.008190+0000","last_unstale":"2024-04-24T08:15:23.010899+0000","last_undegraded":"2024-04-24T08:15:23.010899+0000","last_fullsized":"2024-04-24T08:15:23.010899+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:15:53.896120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.982239+0000","last_change":"2024-04-24T08:15:22.982239+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.982239+0000","last_undegraded":"2024-04-24T08:15:22.982239+0000","last_fullsized":"2024-04-24T08:15:22.982239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.011503+0000","last_change":"2024-04-24T08:15:23.009900+0000","last_active":"2024-04-24T08:15:23.011503+0000","last_peered":"2024-04-24T08:15:23.011503+0000","last_clean":"2024-04-24T08:15:23.011503+0000","last_became_active":"2024-04-24T08:15:23.007568+0000","last_became_peered":"2024-04-24T08:15:23.007568+0000","last_unstale":"2024-04-24T08:15:23.011503+0000","last_undegraded":"2024-04-24T08:15:23.011503+0000","last_fullsized":"2024-04-24T08:15:23.011503+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:16:51.843549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.981319+0000","last_change":"2024-04-24T08:15:22.981319+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.981319+0000","last_undegraded":"2024-04-24T08:15:22.981319+0000","last_fullsized":"2024-04-24T08:15:22.981319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.011487+0000","last_change":"2024-04-24T08:15:23.009798+0000","last_active":"2024-04-24T08:15:23.011487+0000","last_peered":"2024-04-24T08:15:23.011487+0000","last_clean":"2024-04-24T08:15:23.011487+0000","last_became_active":"2024-04-24T08:15:23.009498+0000","last_became_peered":"2024-04-24T08:15:23.009498+0000","last_unstale":"2024-04-24T08:15:23.011487+0000","last_undegraded":"2024-04-24T08:15:23.011487+0000","last_fullsized":"2024-04-24T08:15:23.011487+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:44:29.386933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.989757+0000","last_change":"2024-04-24T08:15:22.989757+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.989757+0000","last_undegraded":"2024-04-24T08:15:22.989757+0000","last_fullsized":"2024-04-24T08:15:22.989757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:22.992951+0000","last_change":"2024-04-24T08:15:22.991044+0000","last_active":"2024-04-24T08:15:22.992951+0000","last_peered":"2024-04-24T08:15:22.992951+0000","last_clean":"2024-04-24T08:15:22.992951+0000","last_became_active":"2024-04-24T08:15:22.989979+0000","last_became_peered":"2024-04-24T08:15:22.989979+0000","last_unstale":"2024-04-24T08:15:22.992951+0000","last_undegraded":"2024-04-24T08:15:22.992951+0000","last_fullsized":"2024-04-24T08:15:22.992951+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:15:53.896120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.983816+0000","last_change":"2024-04-24T08:15:22.983816+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.983816+0000","last_undegraded":"2024-04-24T08:15:22.983816+0000","last_fullsized":"2024-04-24T08:15:22.983816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:22.998863+0000","last_change":"2024-04-24T08:15:22.997073+0000","last_active":"2024-04-24T08:15:22.998863+0000","last_peered":"2024-04-24T08:15:22.998863+0000","last_clean":"2024-04-24T08:15:22.998863+0000","last_became_active":"2024-04-24T08:15:22.996795+0000","last_became_peered":"2024-04-24T08:15:22.996795+0000","last_unstale":"2024-04-24T08:15:22.998863+0000","last_undegraded":"2024-04-24T08:15:22.998863+0000","last_fullsized":"2024-04-24T08:15:22.998863+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:38:57.578808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.989407+0000","last_change":"2024-04-24T08:15:22.989407+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.989407+0000","last_undegraded":"2024-04-24T08:15:22.989407+0000","last_fullsized":"2024-04-24T08:15:22.989407+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:22.993115+0000","last_change":"2024-04-24T08:15:22.989631+0000","last_active":"2024-04-24T08:15:22.993115+0000","last_peered":"2024-04-24T08:15:22.993115+0000","last_clean":"2024-04-24T08:15:22.993115+0000","last_became_active":"2024-04-24T08:15:22.989332+0000","last_became_peered":"2024-04-24T08:15:22.989332+0000","last_unstale":"2024-04-24T08:15:22.993115+0000","last_undegraded":"2024-04-24T08:15:22.993115+0000","last_fullsized":"2024-04-24T08:15:22.993115+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:48:04.467577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.976982+0000","last_change":"2024-04-24T08:15:22.976982+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.976982+0000","last_undegraded":"2024-04-24T08:15:22.976982+0000","last_fullsized":"2024-04-24T08:15:22.976982+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.003883+0000","last_change":"2024-04-24T08:15:23.000665+0000","last_active":"2024-04-24T08:15:23.003883+0000","last_peered":"2024-04-24T08:15:23.003883+0000","last_clean":"2024-04-24T08:15:23.003883+0000","last_became_active":"2024-04-24T08:15:23.000363+0000","last_became_peered":"2024-04-24T08:15:23.000363+0000","last_unstale":"2024-04-24T08:15:23.003883+0000","last_undegraded":"2024-04-24T08:15:23.003883+0000","last_fullsized":"2024-04-24T08:15:23.003883+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:04:18.218365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.976978+0000","last_change":"2024-04-24T08:15:22.976978+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.976978+0000","last_undegraded":"2024-04-24T08:15:22.976978+0000","last_fullsized":"2024-04-24T08:15:22.976978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.006694+0000","last_change":"2024-04-24T08:15:23.005385+0000","last_active":"2024-04-24T08:15:23.006694+0000","last_peered":"2024-04-24T08:15:23.006694+0000","last_clean":"2024-04-24T08:15:23.006694+0000","last_became_active":"2024-04-24T08:15:23.004176+0000","last_became_peered":"2024-04-24T08:15:23.004176+0000","last_unstale":"2024-04-24T08:15:23.006694+0000","last_undegraded":"2024-04-24T08:15:23.006694+0000","last_fullsized":"2024-04-24T08:15:23.006694+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:15:30.810147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.989575+0000","last_change":"2024-04-24T08:15:22.989575+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.989575+0000","last_undegraded":"2024-04-24T08:15:22.989575+0000","last_fullsized":"2024-04-24T08:15:22.989575+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.010848+0000","last_change":"2024-04-24T08:15:23.007867+0000","last_active":"2024-04-24T08:15:23.010848+0000","last_peered":"2024-04-24T08:15:23.010848+0000","last_clean":"2024-04-24T08:15:23.010848+0000","last_became_active":"2024-04-24T08:15:23.007560+0000","last_became_peered":"2024-04-24T08:15:23.007560+0000","last_unstale":"2024-04-24T08:15:23.010848+0000","last_undegraded":"2024-04-24T08:15:23.010848+0000","last_fullsized":"2024-04-24T08:15:23.010848+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:50:35.997657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:23.000045+0000","last_change":"2024-04-24T08:15:23.000045+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:23.000045+0000","last_undegraded":"2024-04-24T08:15:23.000045+0000","last_fullsized":"2024-04-24T08:15:23.000045+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.004714+0000","last_change":"2024-04-24T08:15:23.002922+0000","last_active":"2024-04-24T08:15:23.004714+0000","last_peered":"2024-04-24T08:15:23.004714+0000","last_clean":"2024-04-24T08:15:23.004714+0000","last_became_active":"2024-04-24T08:15:23.002433+0000","last_became_peered":"2024-04-24T08:15:23.002433+0000","last_unstale":"2024-04-24T08:15:23.004714+0000","last_undegraded":"2024-04-24T08:15:23.004714+0000","last_fullsized":"2024-04-24T08:15:23.004714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:59:49.937546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.988848+0000","last_change":"2024-04-24T08:15:22.988848+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.988848+0000","last_undegraded":"2024-04-24T08:15:22.988848+0000","last_fullsized":"2024-04-24T08:15:22.988848+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.002927+0000","last_change":"2024-04-24T08:15:22.997691+0000","last_active":"2024-04-24T08:15:23.002927+0000","last_peered":"2024-04-24T08:15:23.002927+0000","last_clean":"2024-04-24T08:15:23.002927+0000","last_became_active":"2024-04-24T08:15:22.997384+0000","last_became_peered":"2024-04-24T08:15:22.997384+0000","last_unstale":"2024-04-24T08:15:23.002927+0000","last_undegraded":"2024-04-24T08:15:23.002927+0000","last_fullsized":"2024-04-24T08:15:23.002927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:07:26.028117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.986905+0000","last_change":"2024-04-24T08:15:22.986905+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.986905+0000","last_undegraded":"2024-04-24T08:15:22.986905+0000","last_fullsized":"2024-04-24T08:15:22.986905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.007426+0000","last_change":"2024-04-24T08:15:23.002900+0000","last_active":"2024-04-24T08:15:23.007426+0000","last_peered":"2024-04-24T08:15:23.007426+0000","last_clean":"2024-04-24T08:15:23.007426+0000","last_became_active":"2024-04-24T08:15:23.002447+0000","last_became_peered":"2024-04-24T08:15:23.002447+0000","last_unstale":"2024-04-24T08:15:23.007426+0000","last_undegraded":"2024-04-24T08:15:23.007426+0000","last_fullsized":"2024-04-24T08:15:23.007426+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:07:51.451539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.981286+0000","last_change":"2024-04-24T08:15:22.981286+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.981286+0000","last_undegraded":"2024-04-24T08:15:22.981286+0000","last_fullsized":"2024-04-24T08:15:22.981286+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.005300+0000","last_change":"2024-04-24T08:15:22.998536+0000","last_active":"2024-04-24T08:15:23.005300+0000","last_peered":"2024-04-24T08:15:23.005300+0000","last_clean":"2024-04-24T08:15:23.005300+0000","last_became_active":"2024-04-24T08:15:22.998231+0000","last_became_peered":"2024-04-24T08:15:22.998231+0000","last_unstale":"2024-04-24T08:15:23.005300+0000","last_undegraded":"2024-04-24T08:15:23.005300+0000","last_fullsized":"2024-04-24T08:15:23.005300+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:33:21.431109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.983816+0000","last_change":"2024-04-24T08:15:22.983816+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.983816+0000","last_undegraded":"2024-04-24T08:15:22.983816+0000","last_fullsized":"2024-04-24T08:15:22.983816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.015221+0000","last_change":"2024-04-24T08:15:23.008190+0000","last_active":"2024-04-24T08:15:23.015221+0000","last_peered":"2024-04-24T08:15:23.015221+0000","last_clean":"2024-04-24T08:15:23.015221+0000","last_became_active":"2024-04-24T08:15:23.007876+0000","last_became_peered":"2024-04-24T08:15:23.007876+0000","last_unstale":"2024-04-24T08:15:23.015221+0000","last_undegraded":"2024-04-24T08:15:23.015221+0000","last_fullsized":"2024-04-24T08:15:23.015221+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:58:15.960219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.981315+0000","last_change":"2024-04-24T08:15:22.981315+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.981315+0000","last_undegraded":"2024-04-24T08:15:22.981315+0000","last_fullsized":"2024-04-24T08:15:22.981315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.007464+0000","last_change":"2024-04-24T08:15:23.003814+0000","last_active":"2024-04-24T08:15:23.007464+0000","last_peered":"2024-04-24T08:15:23.007464+0000","last_clean":"2024-04-24T08:15:23.007464+0000","last_became_active":"2024-04-24T08:15:23.003549+0000","last_became_peered":"2024-04-24T08:15:23.003549+0000","last_unstale":"2024-04-24T08:15:23.007464+0000","last_undegraded":"2024-04-24T08:15:23.007464+0000","last_fullsized":"2024-04-24T08:15:23.007464+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:14:18.466524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.986914+0000","last_change":"2024-04-24T08:15:22.986914+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.986914+0000","last_undegraded":"2024-04-24T08:15:22.986914+0000","last_fullsized":"2024-04-24T08:15:22.986914+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.005077+0000","last_change":"2024-04-24T08:15:23.003384+0000","last_active":"2024-04-24T08:15:23.005077+0000","last_peered":"2024-04-24T08:15:23.005077+0000","last_clean":"2024-04-24T08:15:23.005077+0000","last_became_active":"2024-04-24T08:15:23.003108+0000","last_became_peered":"2024-04-24T08:15:23.003108+0000","last_unstale":"2024-04-24T08:15:23.005077+0000","last_undegraded":"2024-04-24T08:15:23.005077+0000","last_fullsized":"2024-04-24T08:15:23.005077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:20:22.431131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.998657+0000","last_change":"2024-04-24T08:15:22.998657+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.998657+0000","last_undegraded":"2024-04-24T08:15:22.998657+0000","last_fullsized":"2024-04-24T08:15:22.998657+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.002061+0000","last_change":"2024-04-24T08:15:22.999739+0000","last_active":"2024-04-24T08:15:23.002061+0000","last_peered":"2024-04-24T08:15:23.002061+0000","last_clean":"2024-04-24T08:15:23.002061+0000","last_became_active":"2024-04-24T08:15:22.998302+0000","last_became_peered":"2024-04-24T08:15:22.998302+0000","last_unstale":"2024-04-24T08:15:23.002061+0000","last_undegraded":"2024-04-24T08:15:23.002061+0000","last_fullsized":"2024-04-24T08:15:23.002061+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:19.047198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.996117+0000","last_change":"2024-04-24T08:15:22.996117+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.996117+0000","last_undegraded":"2024-04-24T08:15:22.996117+0000","last_fullsized":"2024-04-24T08:15:22.996117+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.009940+0000","last_change":"2024-04-24T08:15:23.006451+0000","last_active":"2024-04-24T08:15:23.009940+0000","last_peered":"2024-04-24T08:15:23.009940+0000","last_clean":"2024-04-24T08:15:23.009940+0000","last_became_active":"2024-04-24T08:15:23.005935+0000","last_became_peered":"2024-04-24T08:15:23.005935+0000","last_unstale":"2024-04-24T08:15:23.009940+0000","last_undegraded":"2024-04-24T08:15:23.009940+0000","last_fullsized":"2024-04-24T08:15:23.009940+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:37:59.257564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.988105+0000","last_change":"2024-04-24T08:15:22.988105+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.988105+0000","last_undegraded":"2024-04-24T08:15:22.988105+0000","last_fullsized":"2024-04-24T08:15:22.988105+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.009524+0000","last_change":"2024-04-24T08:15:23.004865+0000","last_active":"2024-04-24T08:15:23.009524+0000","last_peered":"2024-04-24T08:15:23.009524+0000","last_clean":"2024-04-24T08:15:23.009524+0000","last_became_active":"2024-04-24T08:15:23.004355+0000","last_became_peered":"2024-04-24T08:15:23.004355+0000","last_unstale":"2024-04-24T08:15:23.009524+0000","last_undegraded":"2024-04-24T08:15:23.009524+0000","last_fullsized":"2024-04-24T08:15:23.009524+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:19:39.481109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.981532+0000","last_change":"2024-04-24T08:15:22.981532+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.981532+0000","last_undegraded":"2024-04-24T08:15:22.981532+0000","last_fullsized":"2024-04-24T08:15:22.981532+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.006568+0000","last_change":"2024-04-24T08:15:23.005219+0000","last_active":"2024-04-24T08:15:23.006568+0000","last_peered":"2024-04-24T08:15:23.006568+0000","last_clean":"2024-04-24T08:15:23.006568+0000","last_became_active":"2024-04-24T08:15:23.004788+0000","last_became_peered":"2024-04-24T08:15:23.004788+0000","last_unstale":"2024-04-24T08:15:23.006568+0000","last_undegraded":"2024-04-24T08:15:23.006568+0000","last_fullsized":"2024-04-24T08:15:23.006568+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:50.368224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.990150+0000","last_change":"2024-04-24T08:15:22.990150+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.990150+0000","last_undegraded":"2024-04-24T08:15:22.990150+0000","last_fullsized":"2024-04-24T08:15:22.990150+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:22.998540+0000","last_change":"2024-04-24T08:15:22.994398+0000","last_active":"2024-04-24T08:15:22.998540+0000","last_peered":"2024-04-24T08:15:22.998540+0000","last_clean":"2024-04-24T08:15:22.998540+0000","last_became_active":"2024-04-24T08:15:22.993888+0000","last_became_peered":"2024-04-24T08:15:22.993888+0000","last_unstale":"2024-04-24T08:15:22.998540+0000","last_undegraded":"2024-04-24T08:15:22.998540+0000","last_fullsized":"2024-04-24T08:15:22.998540+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:19:39.481109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.985417+0000","last_change":"2024-04-24T08:15:22.985417+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.985417+0000","last_undegraded":"2024-04-24T08:15:22.985417+0000","last_fullsized":"2024-04-24T08:15:22.985417+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.010963+0000","last_change":"2024-04-24T08:15:23.008814+0000","last_active":"2024-04-24T08:15:23.010963+0000","last_peered":"2024-04-24T08:15:23.010963+0000","last_clean":"2024-04-24T08:15:23.010963+0000","last_became_active":"2024-04-24T08:15:23.008582+0000","last_became_peered":"2024-04-24T08:15:23.008582+0000","last_unstale":"2024-04-24T08:15:23.010963+0000","last_undegraded":"2024-04-24T08:15:23.010963+0000","last_fullsized":"2024-04-24T08:15:23.010963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:48.189131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.996831+0000","last_change":"2024-04-24T08:15:22.996831+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.996831+0000","last_undegraded":"2024-04-24T08:15:22.996831+0000","last_fullsized":"2024-04-24T08:15:22.996831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.011073+0000","last_change":"2024-04-24T08:15:23.008676+0000","last_active":"2024-04-24T08:15:23.011073+0000","last_peered":"2024-04-24T08:15:23.011073+0000","last_clean":"2024-04-24T08:15:23.011073+0000","last_became_active":"2024-04-24T08:15:23.007815+0000","last_became_peered":"2024-04-24T08:15:23.007815+0000","last_unstale":"2024-04-24T08:15:23.011073+0000","last_undegraded":"2024-04-24T08:15:23.011073+0000","last_fullsized":"2024-04-24T08:15:23.011073+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:04:18.218365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-24T08:15:22.977217+0000","last_change":"2024-04-24T08:15:22.977217+0000","last_active":"2024-04-24T08:15:22.954537+0000","last_peered":"2024-04-24T08:15:22.954537+0000","last_clean":"2024-04-24T08:15:22.954537+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:15:22.977217+0000","last_undegraded":"2024-04-24T08:15:22.977217+0000","last_fullsized":"2024-04-24T08:15:22.977217+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.010451+0000","last_change":"2024-04-24T08:15:23.007108+0000","last_active":"2024-04-24T08:15:23.010451+0000","last_peered":"2024-04-24T08:15:23.010451+0000","last_clean":"2024-04-24T08:15:23.010451+0000","last_became_active":"2024-04-24T08:15:23.006755+0000","last_became_peered":"2024-04-24T08:15:23.006755+0000","last_unstale":"2024-04-24T08:15:23.010451+0000","last_undegraded":"2024-04-24T08:15:23.010451+0000","last_fullsized":"2024-04-24T08:15:23.010451+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:09:44.565995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":280,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-24T08:15:23.000356+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:15:23.000356+0000","last_peered":"2024-04-24T08:15:23.000356+0000","last_clean":"2024-04-24T08:15:23.000356+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:15:23.000356+0000","last_undegraded":"2024-04-24T08:15:23.000356+0000","last_fullsized":"2024-04-24T08:15:23.000356+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574940,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1008170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574940,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":417802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574938,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":417802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574937,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32140,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339700,"statfs":{"total":96636764160,"available":96603852800,"internally_reserved":0,"allocated":983040,"data_stored":417802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574938,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":417802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":417802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":983040,"data_stored":417802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32720,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339120,"statfs":{"total":96636764160,"available":96603258880,"internally_reserved":0,"allocated":1576960,"data_stored":1008170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:15:28.412 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:15:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.647+0000 7f04de400640 1 Processor -- start 2024-04-24T08:15:28.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.647+0000 7f04de400640 1 -- start start 2024-04-24T08:15:28.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.648+0000 7f04de400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04d815ab10 0x7f04d8144090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04de400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04d8071be0 0x7f04d806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04de400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04d806e960 0x7f04d815e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04de400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f04d806edd0 con 0x7f04d8071be0 2024-04-24T08:15:28.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04de400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f04d80b3190 con 0x7f04d806e960 2024-04-24T08:15:28.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04de400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04d80ad590 con 0x7f04d815ab10 2024-04-24T08:15:28.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04cfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04d8071be0 0x7f04d806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04cfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f04d8071be0 0x7f04d806e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53910/0 (socket says 172.21.15.69:53910) 2024-04-24T08:15:28.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04cfe00640 1 -- 172.21.15.69:0/2043232268 learned_addr learned my addr 172.21.15.69:0/2043232268 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:28.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04cfe00640 1 -- 172.21.15.69:0/2043232268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04d815ab10 msgr2=0x7f04d8144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:28.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04cfe00640 1 --2- 172.21.15.69:0/2043232268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04d815ab10 0x7f04d8144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04cfe00640 1 -- 172.21.15.69:0/2043232268 >> v2:172.21.15.121:3300/0 conn(0x7f04d806e960 msgr2=0x7f04d815e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:28.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04cfe00640 1 --2- 172.21.15.69:0/2043232268 >> v2:172.21.15.121:3300/0 conn(0x7f04d806e960 0x7f04d815e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.649+0000 7f04cfe00640 1 -- 172.21.15.69:0/2043232268 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d80b3400 con 0x7f04d8071be0 2024-04-24T08:15:28.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.650+0000 7f04cfe00640 1 --2- 172.21.15.69:0/2043232268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 0x7f04d806e420 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f04d80ad740 tx=0x7f04c802c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9092ec1af989baa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:28.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04d7e00640 1 -- 172.21.15.69:0/2043232268 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8038070 con 0x7f04d8071be0 2024-04-24T08:15:28.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04d7e00640 1 -- 172.21.15.69:0/2043232268 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04c8031710 con 0x7f04d8071be0 2024-04-24T08:15:28.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04d7e00640 1 -- 172.21.15.69:0/2043232268 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8031ae0 con 0x7f04d8071be0 2024-04-24T08:15:28.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04de400640 1 -- 172.21.15.69:0/2043232268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 msgr2=0x7f04d806e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04de400640 1 --2- 172.21.15.69:0/2043232268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 0x7f04d806e420 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f04d80ad740 tx=0x7f04c802c3d0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04de400640 1 -- 172.21.15.69:0/2043232268 shutdown_connections 2024-04-24T08:15:28.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04de400640 1 --2- 172.21.15.69:0/2043232268 >> v2:172.21.15.121:3300/0 conn(0x7f04d806e960 0x7f04d815e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04de400640 1 --2- 172.21.15.69:0/2043232268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 0x7f04d806e420 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04de400640 1 --2- 172.21.15.69:0/2043232268 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04d815ab10 0x7f04d8144090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04de400640 1 -- 172.21.15.69:0/2043232268 >> 172.21.15.69:0/2043232268 conn(0x7f04d80d0160 msgr2=0x7f04d80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:28.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.651+0000 7f04de400640 1 -- 172.21.15.69:0/2043232268 shutdown_connections 2024-04-24T08:15:28.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.652+0000 7f04de400640 1 -- 172.21.15.69:0/2043232268 wait complete. 2024-04-24T08:15:28.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.653+0000 7f04de400640 1 Processor -- start 2024-04-24T08:15:28.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.653+0000 7f04de400640 1 -- start start 2024-04-24T08:15:28.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.654+0000 7f04de400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04d806e960 0x7f04d81f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.654+0000 7f04de400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 0x7f04d8154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.654+0000 7f04de400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04d815ab10 0x7f04d8155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.654+0000 7f04de400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04d8072150 con 0x7f04d8071be0 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.654+0000 7f04de400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04d80b39c0 con 0x7f04d806e960 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.654+0000 7f04de400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04d815aef0 con 0x7f04d815ab10 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.655+0000 7f04cfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 0x7f04d8154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.655+0000 7f04cfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 0x7f04d8154fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53918/0 (socket says 172.21.15.69:53918) 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.655+0000 7f04cfe00640 1 -- 172.21.15.69:0/1439281854 learned_addr learned my addr 172.21.15.69:0/1439281854 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.655+0000 7f04cfe00640 1 -- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04d815ab10 msgr2=0x7f04d8155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.655+0000 7f04cfe00640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04d815ab10 0x7f04d8155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.655+0000 7f04cfe00640 1 -- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04d806e960 msgr2=0x7f04d81f4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.655+0000 7f04cfe00640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04d806e960 0x7f04d81f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.655+0000 7f04cfe00640 1 -- 172.21.15.69:0/1439281854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d81f4590 con 0x7f04d8071be0 2024-04-24T08:15:28.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.655+0000 7f04cfe00640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 0x7f04d8154fd0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f04c8000c00 tx=0x7f04c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:28.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.656+0000 7f04d5600640 1 -- 172.21.15.69:0/1439281854 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8042020 con 0x7f04d8071be0 2024-04-24T08:15:28.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.656+0000 7f04d5600640 1 -- 172.21.15.69:0/1439281854 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04c8033830 con 0x7f04d8071be0 2024-04-24T08:15:28.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.656+0000 7f04d5600640 1 -- 172.21.15.69:0/1439281854 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8038040 con 0x7f04d8071be0 2024-04-24T08:15:28.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.656+0000 7f04de400640 1 -- 172.21.15.69:0/1439281854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d815ce00 con 0x7f04d8071be0 2024-04-24T08:15:28.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.656+0000 7f04de400640 1 -- 172.21.15.69:0/1439281854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d815d3c0 con 0x7f04d8071be0 2024-04-24T08:15:28.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.657+0000 7f04de400640 1 -- 172.21.15.69:0/1439281854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d81f4760 con 0x7f04d8071be0 2024-04-24T08:15:28.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.658+0000 7f04d5600640 1 -- 172.21.15.69:0/1439281854 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04c8045020 con 0x7f04d8071be0 2024-04-24T08:15:28.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.659+0000 7f04d5600640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04b406c8f0 0x7f04b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:28.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.659+0000 7f04d5600640 1 -- 172.21.15.69:0/1439281854 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f04c80ac900 con 0x7f04d8071be0 2024-04-24T08:15:28.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.660+0000 7f04dd000640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04b406c8f0 0x7f04b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:28.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.661+0000 7f04dd000640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04b406c8f0 0x7f04b406edb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f04c0009670 tx=0x7f04c0006bf0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:28.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.663+0000 7f04d5600640 1 -- 172.21.15.69:0/1439281854 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04c804c030 con 0x7f04d8071be0 2024-04-24T08:15:28.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.858+0000 7f04de400640 1 -- 172.21.15.69:0/1439281854 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f04d806f020 con 0x7f04b406c8f0 2024-04-24T08:15:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.878+0000 7f04d5600640 1 -- 172.21.15.69:0/1439281854 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252081 (secure 0 0 0) 0x7f04d806f020 con 0x7f04b406c8f0 2024-04-24T08:15:28.882 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:28.883 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 -- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04b406c8f0 msgr2=0x7f04b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04b406c8f0 0x7f04b406edb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f04c0009670 tx=0x7f04c0006bf0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 -- 172.21.15.69:0/1439281854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 msgr2=0x7f04d8154fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 0x7f04d8154fd0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f04c8000c00 tx=0x7f04c80046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 -- 172.21.15.69:0/1439281854 shutdown_connections 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04b406c8f0 0x7f04b406edb0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04d8071be0 0x7f04d8154fd0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04d815ab10 0x7f04d8155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 --2- 172.21.15.69:0/1439281854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04d806e960 0x7f04d81f4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 -- 172.21.15.69:0/1439281854 >> 172.21.15.69:0/1439281854 conn(0x7f04d80d0160 msgr2=0x7f04d8099730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:28.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 -- 172.21.15.69:0/1439281854 shutdown_connections 2024-04-24T08:15:28.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:28.884+0000 7f04de400640 1 -- 172.21.15.69:0/1439281854 wait complete. 2024-04-24T08:15:28.904 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-04-24T08:15:28.820948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":258568,"kb_used_data":9096,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754716152,"statfs":{"total":773094113280,"available":772829339648,"internally_reserved":0,"allocated":9314304,"data_stored":4632572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001852"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.387228+0000","last_change":"2024-04-24T08:15:24.004834+0000","last_active":"2024-04-24T08:15:25.387228+0000","last_peered":"2024-04-24T08:15:25.387228+0000","last_clean":"2024-04-24T08:15:25.387228+0000","last_became_active":"2024-04-24T08:15:24.004562+0000","last_became_peered":"2024-04-24T08:15:24.004562+0000","last_unstale":"2024-04-24T08:15:25.387228+0000","last_undegraded":"2024-04-24T08:15:25.387228+0000","last_fullsized":"2024-04-24T08:15:25.387228+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:42:33.021408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.004380+0000","last_change":"2024-04-24T08:15:23.999034+0000","last_active":"2024-04-24T08:15:25.004380+0000","last_peered":"2024-04-24T08:15:25.004380+0000","last_clean":"2024-04-24T08:15:25.004380+0000","last_became_active":"2024-04-24T08:15:23.998627+0000","last_became_peered":"2024-04-24T08:15:23.998627+0000","last_unstale":"2024-04-24T08:15:25.004380+0000","last_undegraded":"2024-04-24T08:15:25.004380+0000","last_fullsized":"2024-04-24T08:15:25.004380+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:59:50.953183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.386626+0000","last_change":"2024-04-24T08:15:24.000454+0000","last_active":"2024-04-24T08:15:25.386626+0000","last_peered":"2024-04-24T08:15:25.386626+0000","last_clean":"2024-04-24T08:15:25.386626+0000","last_became_active":"2024-04-24T08:15:24.000185+0000","last_became_peered":"2024-04-24T08:15:24.000185+0000","last_unstale":"2024-04-24T08:15:25.386626+0000","last_undegraded":"2024-04-24T08:15:25.386626+0000","last_fullsized":"2024-04-24T08:15:25.386626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:06.324376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385317+0000","last_change":"2024-04-24T08:15:24.016075+0000","last_active":"2024-04-24T08:15:25.385317+0000","last_peered":"2024-04-24T08:15:25.385317+0000","last_clean":"2024-04-24T08:15:25.385317+0000","last_became_active":"2024-04-24T08:15:24.012741+0000","last_became_peered":"2024-04-24T08:15:24.012741+0000","last_unstale":"2024-04-24T08:15:25.385317+0000","last_undegraded":"2024-04-24T08:15:25.385317+0000","last_fullsized":"2024-04-24T08:15:25.385317+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:13:46.487651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.999286+0000","last_change":"2024-04-24T08:15:23.996093+0000","last_active":"2024-04-24T08:15:24.999286+0000","last_peered":"2024-04-24T08:15:24.999286+0000","last_clean":"2024-04-24T08:15:24.999286+0000","last_became_active":"2024-04-24T08:15:23.995846+0000","last_became_peered":"2024-04-24T08:15:23.995846+0000","last_unstale":"2024-04-24T08:15:24.999286+0000","last_undegraded":"2024-04-24T08:15:24.999286+0000","last_fullsized":"2024-04-24T08:15:24.999286+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:06.324376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.001246+0000","last_change":"2024-04-24T08:15:24.009453+0000","last_active":"2024-04-24T08:15:25.001246+0000","last_peered":"2024-04-24T08:15:25.001246+0000","last_clean":"2024-04-24T08:15:25.001246+0000","last_became_active":"2024-04-24T08:15:24.007088+0000","last_became_peered":"2024-04-24T08:15:24.007088+0000","last_unstale":"2024-04-24T08:15:25.001246+0000","last_undegraded":"2024-04-24T08:15:25.001246+0000","last_fullsized":"2024-04-24T08:15:25.001246+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:57:20.609480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.383750+0000","last_change":"2024-04-24T08:15:24.011083+0000","last_active":"2024-04-24T08:15:25.383750+0000","last_peered":"2024-04-24T08:15:25.383750+0000","last_clean":"2024-04-24T08:15:25.383750+0000","last_became_active":"2024-04-24T08:15:24.010722+0000","last_became_peered":"2024-04-24T08:15:24.010722+0000","last_unstale":"2024-04-24T08:15:25.383750+0000","last_undegraded":"2024-04-24T08:15:25.383750+0000","last_fullsized":"2024-04-24T08:15:25.383750+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:41:36.245788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.001461+0000","last_change":"2024-04-24T08:15:23.980125+0000","last_active":"2024-04-24T08:15:25.001461+0000","last_peered":"2024-04-24T08:15:25.001461+0000","last_clean":"2024-04-24T08:15:25.001461+0000","last_became_active":"2024-04-24T08:15:23.978889+0000","last_became_peered":"2024-04-24T08:15:23.978889+0000","last_unstale":"2024-04-24T08:15:25.001461+0000","last_undegraded":"2024-04-24T08:15:25.001461+0000","last_fullsized":"2024-04-24T08:15:25.001461+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:21.101731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.001564+0000","last_change":"2024-04-24T08:15:23.996595+0000","last_active":"2024-04-24T08:15:25.001564+0000","last_peered":"2024-04-24T08:15:25.001564+0000","last_clean":"2024-04-24T08:15:25.001564+0000","last_became_active":"2024-04-24T08:15:23.996282+0000","last_became_peered":"2024-04-24T08:15:23.996282+0000","last_unstale":"2024-04-24T08:15:25.001564+0000","last_undegraded":"2024-04-24T08:15:25.001564+0000","last_fullsized":"2024-04-24T08:15:25.001564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:48:05.483214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.387200+0000","last_change":"2024-04-24T08:15:23.997918+0000","last_active":"2024-04-24T08:15:25.387200+0000","last_peered":"2024-04-24T08:15:25.387200+0000","last_clean":"2024-04-24T08:15:25.387200+0000","last_became_active":"2024-04-24T08:15:23.997434+0000","last_became_peered":"2024-04-24T08:15:23.997434+0000","last_unstale":"2024-04-24T08:15:25.387200+0000","last_undegraded":"2024-04-24T08:15:25.387200+0000","last_fullsized":"2024-04-24T08:15:25.387200+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:19:47.970752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.003974+0000","last_change":"2024-04-24T08:15:23.999398+0000","last_active":"2024-04-24T08:15:25.003974+0000","last_peered":"2024-04-24T08:15:25.003974+0000","last_clean":"2024-04-24T08:15:25.003974+0000","last_became_active":"2024-04-24T08:15:23.997906+0000","last_became_peered":"2024-04-24T08:15:23.997906+0000","last_unstale":"2024-04-24T08:15:25.003974+0000","last_undegraded":"2024-04-24T08:15:25.003974+0000","last_fullsized":"2024-04-24T08:15:25.003974+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:30.612838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385356+0000","last_change":"2024-04-24T08:15:24.004979+0000","last_active":"2024-04-24T08:15:25.385356+0000","last_peered":"2024-04-24T08:15:25.385356+0000","last_clean":"2024-04-24T08:15:25.385356+0000","last_became_active":"2024-04-24T08:15:24.004714+0000","last_became_peered":"2024-04-24T08:15:24.004714+0000","last_unstale":"2024-04-24T08:15:25.385356+0000","last_undegraded":"2024-04-24T08:15:25.385356+0000","last_fullsized":"2024-04-24T08:15:25.385356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:14:58.471323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.386336+0000","last_change":"2024-04-24T08:15:24.002034+0000","last_active":"2024-04-24T08:15:25.386336+0000","last_peered":"2024-04-24T08:15:25.386336+0000","last_clean":"2024-04-24T08:15:25.386336+0000","last_became_active":"2024-04-24T08:15:24.001753+0000","last_became_peered":"2024-04-24T08:15:24.001753+0000","last_unstale":"2024-04-24T08:15:25.386336+0000","last_undegraded":"2024-04-24T08:15:25.386336+0000","last_fullsized":"2024-04-24T08:15:25.386336+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:51:16.125934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384349+0000","last_change":"2024-04-24T08:15:24.008331+0000","last_active":"2024-04-24T08:15:25.384349+0000","last_peered":"2024-04-24T08:15:25.384349+0000","last_clean":"2024-04-24T08:15:25.384349+0000","last_became_active":"2024-04-24T08:15:24.007840+0000","last_became_peered":"2024-04-24T08:15:24.007840+0000","last_unstale":"2024-04-24T08:15:25.384349+0000","last_undegraded":"2024-04-24T08:15:25.384349+0000","last_fullsized":"2024-04-24T08:15:25.384349+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:38:58.594445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384949+0000","last_change":"2024-04-24T08:15:23.998000+0000","last_active":"2024-04-24T08:15:25.384949+0000","last_peered":"2024-04-24T08:15:25.384949+0000","last_clean":"2024-04-24T08:15:25.384949+0000","last_became_active":"2024-04-24T08:15:23.997685+0000","last_became_peered":"2024-04-24T08:15:23.997685+0000","last_unstale":"2024-04-24T08:15:25.384949+0000","last_undegraded":"2024-04-24T08:15:25.384949+0000","last_fullsized":"2024-04-24T08:15:25.384949+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:30:59.423299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998517+0000","last_change":"2024-04-24T08:15:23.994893+0000","last_active":"2024-04-24T08:15:24.998517+0000","last_peered":"2024-04-24T08:15:24.998517+0000","last_clean":"2024-04-24T08:15:24.998517+0000","last_became_active":"2024-04-24T08:15:23.992150+0000","last_became_peered":"2024-04-24T08:15:23.992150+0000","last_unstale":"2024-04-24T08:15:24.998517+0000","last_undegraded":"2024-04-24T08:15:24.998517+0000","last_fullsized":"2024-04-24T08:15:24.998517+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:48:32.630626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.999310+0000","last_change":"2024-04-24T08:15:24.004660+0000","last_active":"2024-04-24T08:15:24.999310+0000","last_peered":"2024-04-24T08:15:24.999310+0000","last_clean":"2024-04-24T08:15:24.999310+0000","last_became_active":"2024-04-24T08:15:24.004373+0000","last_became_peered":"2024-04-24T08:15:24.004373+0000","last_unstale":"2024-04-24T08:15:24.999310+0000","last_undegraded":"2024-04-24T08:15:24.999310+0000","last_fullsized":"2024-04-24T08:15:24.999310+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:51:16.125934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384463+0000","last_change":"2024-04-24T08:15:24.005099+0000","last_active":"2024-04-24T08:15:25.384463+0000","last_peered":"2024-04-24T08:15:25.384463+0000","last_clean":"2024-04-24T08:15:25.384463+0000","last_became_active":"2024-04-24T08:15:24.004734+0000","last_became_peered":"2024-04-24T08:15:24.004734+0000","last_unstale":"2024-04-24T08:15:25.384463+0000","last_undegraded":"2024-04-24T08:15:25.384463+0000","last_fullsized":"2024-04-24T08:15:25.384463+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:20:23.446768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998966+0000","last_change":"2024-04-24T08:15:23.994256+0000","last_active":"2024-04-24T08:15:24.998966+0000","last_peered":"2024-04-24T08:15:24.998966+0000","last_clean":"2024-04-24T08:15:24.998966+0000","last_became_active":"2024-04-24T08:15:23.993922+0000","last_became_peered":"2024-04-24T08:15:23.993922+0000","last_unstale":"2024-04-24T08:15:24.998966+0000","last_undegraded":"2024-04-24T08:15:24.998966+0000","last_fullsized":"2024-04-24T08:15:24.998966+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:30:59.423299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385419+0000","last_change":"2024-04-24T08:15:24.016102+0000","last_active":"2024-04-24T08:15:25.385419+0000","last_peered":"2024-04-24T08:15:25.385419+0000","last_clean":"2024-04-24T08:15:25.385419+0000","last_became_active":"2024-04-24T08:15:24.015884+0000","last_became_peered":"2024-04-24T08:15:24.015884+0000","last_unstale":"2024-04-24T08:15:25.385419+0000","last_undegraded":"2024-04-24T08:15:25.385419+0000","last_fullsized":"2024-04-24T08:15:25.385419+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:27:21.334966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384308+0000","last_change":"2024-04-24T08:15:24.007396+0000","last_active":"2024-04-24T08:15:25.384308+0000","last_peered":"2024-04-24T08:15:25.384308+0000","last_clean":"2024-04-24T08:15:25.384308+0000","last_became_active":"2024-04-24T08:15:24.006854+0000","last_became_peered":"2024-04-24T08:15:24.006854+0000","last_unstale":"2024-04-24T08:15:25.384308+0000","last_undegraded":"2024-04-24T08:15:25.384308+0000","last_fullsized":"2024-04-24T08:15:25.384308+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:12:52.610854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.003156+0000","last_change":"2024-04-24T08:15:23.998583+0000","last_active":"2024-04-24T08:15:25.003156+0000","last_peered":"2024-04-24T08:15:25.003156+0000","last_clean":"2024-04-24T08:15:25.003156+0000","last_became_active":"2024-04-24T08:15:23.998310+0000","last_became_peered":"2024-04-24T08:15:23.998310+0000","last_unstale":"2024-04-24T08:15:25.003156+0000","last_undegraded":"2024-04-24T08:15:25.003156+0000","last_fullsized":"2024-04-24T08:15:25.003156+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:49.204768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385274+0000","last_change":"2024-04-24T08:15:24.013298+0000","last_active":"2024-04-24T08:15:25.385274+0000","last_peered":"2024-04-24T08:15:25.385274+0000","last_clean":"2024-04-24T08:15:25.385274+0000","last_became_active":"2024-04-24T08:15:24.012953+0000","last_became_peered":"2024-04-24T08:15:24.012953+0000","last_unstale":"2024-04-24T08:15:25.385274+0000","last_undegraded":"2024-04-24T08:15:25.385274+0000","last_fullsized":"2024-04-24T08:15:25.385274+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:06.324376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.383936+0000","last_change":"2024-04-24T08:15:24.016285+0000","last_active":"2024-04-24T08:15:25.383936+0000","last_peered":"2024-04-24T08:15:25.383936+0000","last_clean":"2024-04-24T08:15:25.383936+0000","last_became_active":"2024-04-24T08:15:24.015935+0000","last_became_peered":"2024-04-24T08:15:24.015935+0000","last_unstale":"2024-04-24T08:15:25.383936+0000","last_undegraded":"2024-04-24T08:15:25.383936+0000","last_fullsized":"2024-04-24T08:15:25.383936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:13:24.167482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.003841+0000","last_change":"2024-04-24T08:15:23.997575+0000","last_active":"2024-04-24T08:15:25.003841+0000","last_peered":"2024-04-24T08:15:25.003841+0000","last_clean":"2024-04-24T08:15:25.003841+0000","last_became_active":"2024-04-24T08:15:23.997279+0000","last_became_peered":"2024-04-24T08:15:23.997279+0000","last_unstale":"2024-04-24T08:15:25.003841+0000","last_undegraded":"2024-04-24T08:15:25.003841+0000","last_fullsized":"2024-04-24T08:15:25.003841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:38.092609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385767+0000","last_change":"2024-04-24T08:15:24.003245+0000","last_active":"2024-04-24T08:15:25.385767+0000","last_peered":"2024-04-24T08:15:25.385767+0000","last_clean":"2024-04-24T08:15:25.385767+0000","last_became_active":"2024-04-24T08:15:24.002916+0000","last_became_peered":"2024-04-24T08:15:24.002916+0000","last_unstale":"2024-04-24T08:15:25.385767+0000","last_undegraded":"2024-04-24T08:15:25.385767+0000","last_fullsized":"2024-04-24T08:15:25.385767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:24.675572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998974+0000","last_change":"2024-04-24T08:15:23.993767+0000","last_active":"2024-04-24T08:15:24.998974+0000","last_peered":"2024-04-24T08:15:24.998974+0000","last_clean":"2024-04-24T08:15:24.998974+0000","last_became_active":"2024-04-24T08:15:23.993447+0000","last_became_peered":"2024-04-24T08:15:23.993447+0000","last_unstale":"2024-04-24T08:15:24.998974+0000","last_undegraded":"2024-04-24T08:15:24.998974+0000","last_fullsized":"2024-04-24T08:15:24.998974+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:23:18.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385787+0000","last_change":"2024-04-24T08:15:24.004285+0000","last_active":"2024-04-24T08:15:25.385787+0000","last_peered":"2024-04-24T08:15:25.385787+0000","last_clean":"2024-04-24T08:15:25.385787+0000","last_became_active":"2024-04-24T08:15:24.002037+0000","last_became_peered":"2024-04-24T08:15:24.002037+0000","last_unstale":"2024-04-24T08:15:25.385787+0000","last_undegraded":"2024-04-24T08:15:25.385787+0000","last_fullsized":"2024-04-24T08:15:25.385787+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:03.797611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.003409+0000","last_change":"2024-04-24T08:15:23.994753+0000","last_active":"2024-04-24T08:15:25.003409+0000","last_peered":"2024-04-24T08:15:25.003409+0000","last_clean":"2024-04-24T08:15:25.003409+0000","last_became_active":"2024-04-24T08:15:23.994415+0000","last_became_peered":"2024-04-24T08:15:23.994415+0000","last_unstale":"2024-04-24T08:15:25.003409+0000","last_undegraded":"2024-04-24T08:15:25.003409+0000","last_fullsized":"2024-04-24T08:15:25.003409+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:16:52.859186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.003336+0000","last_change":"2024-04-24T08:15:24.008656+0000","last_active":"2024-04-24T08:15:25.003336+0000","last_peered":"2024-04-24T08:15:25.003336+0000","last_clean":"2024-04-24T08:15:25.003336+0000","last_became_active":"2024-04-24T08:15:24.008120+0000","last_became_peered":"2024-04-24T08:15:24.008120+0000","last_unstale":"2024-04-24T08:15:25.003336+0000","last_undegraded":"2024-04-24T08:15:25.003336+0000","last_fullsized":"2024-04-24T08:15:25.003336+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:44:30.402570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998792+0000","last_change":"2024-04-24T08:15:24.001637+0000","last_active":"2024-04-24T08:15:24.998792+0000","last_peered":"2024-04-24T08:15:24.998792+0000","last_clean":"2024-04-24T08:15:24.998792+0000","last_became_active":"2024-04-24T08:15:24.001321+0000","last_became_peered":"2024-04-24T08:15:24.001321+0000","last_unstale":"2024-04-24T08:15:24.998792+0000","last_undegraded":"2024-04-24T08:15:24.998792+0000","last_fullsized":"2024-04-24T08:15:24.998792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:24.675572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385529+0000","last_change":"2024-04-24T08:15:23.997825+0000","last_active":"2024-04-24T08:15:25.385529+0000","last_peered":"2024-04-24T08:15:25.385529+0000","last_clean":"2024-04-24T08:15:25.385529+0000","last_became_active":"2024-04-24T08:15:23.997299+0000","last_became_peered":"2024-04-24T08:15:23.997299+0000","last_unstale":"2024-04-24T08:15:25.385529+0000","last_undegraded":"2024-04-24T08:15:25.385529+0000","last_fullsized":"2024-04-24T08:15:25.385529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:30:59.423299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.999251+0000","last_change":"2024-04-24T08:15:23.992220+0000","last_active":"2024-04-24T08:15:24.999251+0000","last_peered":"2024-04-24T08:15:24.999251+0000","last_clean":"2024-04-24T08:15:24.999251+0000","last_became_active":"2024-04-24T08:15:23.991898+0000","last_became_peered":"2024-04-24T08:15:23.991898+0000","last_unstale":"2024-04-24T08:15:24.999251+0000","last_undegraded":"2024-04-24T08:15:24.999251+0000","last_fullsized":"2024-04-24T08:15:24.999251+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:28:40.421322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"81'4","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.387517+0000","last_change":"2024-04-24T08:15:23.008526+0000","last_active":"2024-04-24T08:15:25.387517+0000","last_peered":"2024-04-24T08:15:25.387517+0000","last_clean":"2024-04-24T08:15:25.387517+0000","last_became_active":"2024-04-24T08:15:23.008215+0000","last_became_peered":"2024-04-24T08:15:23.008215+0000","last_unstale":"2024-04-24T08:15:25.387517+0000","last_undegraded":"2024-04-24T08:15:25.387517+0000","last_fullsized":"2024-04-24T08:15:25.387517+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:59:49.937546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.387290+0000","last_change":"2024-04-24T08:15:24.011001+0000","last_active":"2024-04-24T08:15:25.387290+0000","last_peered":"2024-04-24T08:15:25.387290+0000","last_clean":"2024-04-24T08:15:25.387290+0000","last_became_active":"2024-04-24T08:15:24.010450+0000","last_became_peered":"2024-04-24T08:15:24.010450+0000","last_unstale":"2024-04-24T08:15:25.387290+0000","last_undegraded":"2024-04-24T08:15:25.387290+0000","last_fullsized":"2024-04-24T08:15:25.387290+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:21:44.106578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.000178+0000","last_change":"2024-04-24T08:15:22.994471+0000","last_active":"2024-04-24T08:15:25.000178+0000","last_peered":"2024-04-24T08:15:25.000178+0000","last_clean":"2024-04-24T08:15:25.000178+0000","last_became_active":"2024-04-24T08:15:22.993954+0000","last_became_peered":"2024-04-24T08:15:22.993954+0000","last_unstale":"2024-04-24T08:15:25.000178+0000","last_undegraded":"2024-04-24T08:15:25.000178+0000","last_fullsized":"2024-04-24T08:15:25.000178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:07:26.028117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.386216+0000","last_change":"2024-04-24T08:15:24.008981+0000","last_active":"2024-04-24T08:15:25.386216+0000","last_peered":"2024-04-24T08:15:25.386216+0000","last_clean":"2024-04-24T08:15:25.386216+0000","last_became_active":"2024-04-24T08:15:24.007738+0000","last_became_peered":"2024-04-24T08:15:24.007738+0000","last_unstale":"2024-04-24T08:15:25.386216+0000","last_undegraded":"2024-04-24T08:15:25.386216+0000","last_fullsized":"2024-04-24T08:15:25.386216+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:53:36.741944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.999019+0000","last_change":"2024-04-24T08:15:23.000155+0000","last_active":"2024-04-24T08:15:24.999019+0000","last_peered":"2024-04-24T08:15:24.999019+0000","last_clean":"2024-04-24T08:15:24.999019+0000","last_became_active":"2024-04-24T08:15:22.998678+0000","last_became_peered":"2024-04-24T08:15:22.998678+0000","last_unstale":"2024-04-24T08:15:24.999019+0000","last_undegraded":"2024-04-24T08:15:24.999019+0000","last_fullsized":"2024-04-24T08:15:24.999019+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:48:04.467577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385731+0000","last_change":"2024-04-24T08:15:24.007050+0000","last_active":"2024-04-24T08:15:25.385731+0000","last_peered":"2024-04-24T08:15:25.385731+0000","last_clean":"2024-04-24T08:15:25.385731+0000","last_became_active":"2024-04-24T08:15:24.006726+0000","last_became_peered":"2024-04-24T08:15:24.006726+0000","last_unstale":"2024-04-24T08:15:25.385731+0000","last_undegraded":"2024-04-24T08:15:25.385731+0000","last_fullsized":"2024-04-24T08:15:25.385731+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:48:32.630626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998387+0000","last_change":"2024-04-24T08:15:22.996657+0000","last_active":"2024-04-24T08:15:24.998387+0000","last_peered":"2024-04-24T08:15:24.998387+0000","last_clean":"2024-04-24T08:15:24.998387+0000","last_became_active":"2024-04-24T08:15:22.996342+0000","last_became_peered":"2024-04-24T08:15:22.996342+0000","last_unstale":"2024-04-24T08:15:24.998387+0000","last_undegraded":"2024-04-24T08:15:24.998387+0000","last_fullsized":"2024-04-24T08:15:24.998387+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:14:18.466524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.003664+0000","last_change":"2024-04-24T08:15:24.008422+0000","last_active":"2024-04-24T08:15:25.003664+0000","last_peered":"2024-04-24T08:15:25.003664+0000","last_clean":"2024-04-24T08:15:25.003664+0000","last_became_active":"2024-04-24T08:15:24.007689+0000","last_became_peered":"2024-04-24T08:15:24.007689+0000","last_unstale":"2024-04-24T08:15:25.003664+0000","last_undegraded":"2024-04-24T08:15:25.003664+0000","last_fullsized":"2024-04-24T08:15:25.003664+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:07:27.043754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.999779+0000","last_change":"2024-04-24T08:15:22.999054+0000","last_active":"2024-04-24T08:15:24.999779+0000","last_peered":"2024-04-24T08:15:24.999779+0000","last_clean":"2024-04-24T08:15:24.999779+0000","last_became_active":"2024-04-24T08:15:22.998710+0000","last_became_peered":"2024-04-24T08:15:22.998710+0000","last_unstale":"2024-04-24T08:15:24.999779+0000","last_undegraded":"2024-04-24T08:15:24.999779+0000","last_fullsized":"2024-04-24T08:15:24.999779+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:09:44.565995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.999491+0000","last_change":"2024-04-24T08:15:24.005067+0000","last_active":"2024-04-24T08:15:24.999491+0000","last_peered":"2024-04-24T08:15:24.999491+0000","last_clean":"2024-04-24T08:15:24.999491+0000","last_became_active":"2024-04-24T08:15:24.004733+0000","last_became_peered":"2024-04-24T08:15:24.004733+0000","last_unstale":"2024-04-24T08:15:24.999491+0000","last_undegraded":"2024-04-24T08:15:24.999491+0000","last_fullsized":"2024-04-24T08:15:24.999491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:14:58.471323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998793+0000","last_change":"2024-04-24T08:15:22.997874+0000","last_active":"2024-04-24T08:15:24.998793+0000","last_peered":"2024-04-24T08:15:24.998793+0000","last_clean":"2024-04-24T08:15:24.998793+0000","last_became_active":"2024-04-24T08:15:22.994836+0000","last_became_peered":"2024-04-24T08:15:22.994836+0000","last_unstale":"2024-04-24T08:15:24.998793+0000","last_undegraded":"2024-04-24T08:15:24.998793+0000","last_fullsized":"2024-04-24T08:15:24.998793+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:44:29.386933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384576+0000","last_change":"2024-04-24T08:15:24.010854+0000","last_active":"2024-04-24T08:15:25.384576+0000","last_peered":"2024-04-24T08:15:25.384576+0000","last_clean":"2024-04-24T08:15:25.384576+0000","last_became_active":"2024-04-24T08:15:24.009688+0000","last_became_peered":"2024-04-24T08:15:24.009688+0000","last_unstale":"2024-04-24T08:15:25.384576+0000","last_undegraded":"2024-04-24T08:15:25.384576+0000","last_fullsized":"2024-04-24T08:15:25.384576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:48:05.483214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.997229+0000","last_change":"2024-04-24T08:15:23.000628+0000","last_active":"2024-04-24T08:15:24.997229+0000","last_peered":"2024-04-24T08:15:24.997229+0000","last_clean":"2024-04-24T08:15:24.997229+0000","last_became_active":"2024-04-24T08:15:23.000351+0000","last_became_peered":"2024-04-24T08:15:23.000351+0000","last_unstale":"2024-04-24T08:15:24.997229+0000","last_undegraded":"2024-04-24T08:15:24.997229+0000","last_fullsized":"2024-04-24T08:15:24.997229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:57:19.593843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385043+0000","last_change":"2024-04-24T08:15:23.998723+0000","last_active":"2024-04-24T08:15:25.385043+0000","last_peered":"2024-04-24T08:15:25.385043+0000","last_clean":"2024-04-24T08:15:25.385043+0000","last_became_active":"2024-04-24T08:15:23.998385+0000","last_became_peered":"2024-04-24T08:15:23.998385+0000","last_unstale":"2024-04-24T08:15:25.385043+0000","last_undegraded":"2024-04-24T08:15:25.385043+0000","last_fullsized":"2024-04-24T08:15:25.385043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:03.797611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.002942+0000","last_change":"2024-04-24T08:15:22.996673+0000","last_active":"2024-04-24T08:15:25.002942+0000","last_peered":"2024-04-24T08:15:25.002942+0000","last_clean":"2024-04-24T08:15:25.002942+0000","last_became_active":"2024-04-24T08:15:22.996387+0000","last_became_peered":"2024-04-24T08:15:22.996387+0000","last_unstale":"2024-04-24T08:15:25.002942+0000","last_undegraded":"2024-04-24T08:15:25.002942+0000","last_fullsized":"2024-04-24T08:15:25.002942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:07:00.398346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.386339+0000","last_change":"2024-04-24T08:15:24.002697+0000","last_active":"2024-04-24T08:15:25.386339+0000","last_peered":"2024-04-24T08:15:25.386339+0000","last_clean":"2024-04-24T08:15:25.386339+0000","last_became_active":"2024-04-24T08:15:24.000071+0000","last_became_peered":"2024-04-24T08:15:24.000071+0000","last_unstale":"2024-04-24T08:15:25.386339+0000","last_undegraded":"2024-04-24T08:15:25.386339+0000","last_fullsized":"2024-04-24T08:15:25.386339+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:19:16.013532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.387643+0000","last_change":"2024-04-24T08:15:23.008510+0000","last_active":"2024-04-24T08:15:25.387643+0000","last_peered":"2024-04-24T08:15:25.387643+0000","last_clean":"2024-04-24T08:15:25.387643+0000","last_became_active":"2024-04-24T08:15:23.008190+0000","last_became_peered":"2024-04-24T08:15:23.008190+0000","last_unstale":"2024-04-24T08:15:25.387643+0000","last_undegraded":"2024-04-24T08:15:25.387643+0000","last_fullsized":"2024-04-24T08:15:25.387643+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:50.368224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.001681+0000","last_change":"2024-04-24T08:15:24.003223+0000","last_active":"2024-04-24T08:15:25.001681+0000","last_peered":"2024-04-24T08:15:25.001681+0000","last_clean":"2024-04-24T08:15:25.001681+0000","last_became_active":"2024-04-24T08:15:24.002878+0000","last_became_peered":"2024-04-24T08:15:24.002878+0000","last_unstale":"2024-04-24T08:15:25.001681+0000","last_undegraded":"2024-04-24T08:15:25.001681+0000","last_fullsized":"2024-04-24T08:15:25.001681+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:07:52.467176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.387142+0000","last_change":"2024-04-24T08:15:23.009900+0000","last_active":"2024-04-24T08:15:25.387142+0000","last_peered":"2024-04-24T08:15:25.387142+0000","last_clean":"2024-04-24T08:15:25.387142+0000","last_became_active":"2024-04-24T08:15:23.007568+0000","last_became_peered":"2024-04-24T08:15:23.007568+0000","last_unstale":"2024-04-24T08:15:25.387142+0000","last_undegraded":"2024-04-24T08:15:25.387142+0000","last_fullsized":"2024-04-24T08:15:25.387142+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:57:19.593843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998291+0000","last_change":"2024-04-24T08:15:23.998356+0000","last_active":"2024-04-24T08:15:24.998291+0000","last_peered":"2024-04-24T08:15:24.998291+0000","last_clean":"2024-04-24T08:15:24.998291+0000","last_became_active":"2024-04-24T08:15:23.998048+0000","last_became_peered":"2024-04-24T08:15:23.998048+0000","last_unstale":"2024-04-24T08:15:24.998291+0000","last_undegraded":"2024-04-24T08:15:24.998291+0000","last_fullsized":"2024-04-24T08:15:24.998291+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:14:58.471323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"81'3","reported_seq":17,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385149+0000","last_change":"2024-04-24T08:15:23.009798+0000","last_active":"2024-04-24T08:15:25.385149+0000","last_peered":"2024-04-24T08:15:25.385149+0000","last_clean":"2024-04-24T08:15:25.385149+0000","last_became_active":"2024-04-24T08:15:23.009498+0000","last_became_peered":"2024-04-24T08:15:23.009498+0000","last_unstale":"2024-04-24T08:15:25.385149+0000","last_undegraded":"2024-04-24T08:15:25.385149+0000","last_fullsized":"2024-04-24T08:15:25.385149+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:37.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.999609+0000","last_change":"2024-04-24T08:15:24.003879+0000","last_active":"2024-04-24T08:15:24.999609+0000","last_peered":"2024-04-24T08:15:24.999609+0000","last_clean":"2024-04-24T08:15:24.999609+0000","last_became_active":"2024-04-24T08:15:24.003545+0000","last_became_peered":"2024-04-24T08:15:24.003545+0000","last_unstale":"2024-04-24T08:15:24.999609+0000","last_undegraded":"2024-04-24T08:15:24.999609+0000","last_fullsized":"2024-04-24T08:15:24.999609+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:30:59.423299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"81'6","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.001073+0000","last_change":"2024-04-24T08:15:22.991044+0000","last_active":"2024-04-24T08:15:25.001073+0000","last_peered":"2024-04-24T08:15:25.001073+0000","last_clean":"2024-04-24T08:15:25.001073+0000","last_became_active":"2024-04-24T08:15:22.989979+0000","last_became_peered":"2024-04-24T08:15:22.989979+0000","last_unstale":"2024-04-24T08:15:25.001073+0000","last_undegraded":"2024-04-24T08:15:25.001073+0000","last_fullsized":"2024-04-24T08:15:25.001073+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:16:51.843549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.997713+0000","last_change":"2024-04-24T08:15:24.002697+0000","last_active":"2024-04-24T08:15:24.997713+0000","last_peered":"2024-04-24T08:15:24.997713+0000","last_clean":"2024-04-24T08:15:24.997713+0000","last_became_active":"2024-04-24T08:15:24.000908+0000","last_became_peered":"2024-04-24T08:15:24.000908+0000","last_unstale":"2024-04-24T08:15:24.997713+0000","last_undegraded":"2024-04-24T08:15:24.997713+0000","last_fullsized":"2024-04-24T08:15:24.997713+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:51:16.125934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.003313+0000","last_change":"2024-04-24T08:15:22.997073+0000","last_active":"2024-04-24T08:15:25.003313+0000","last_peered":"2024-04-24T08:15:25.003313+0000","last_clean":"2024-04-24T08:15:25.003313+0000","last_became_active":"2024-04-24T08:15:22.996795+0000","last_became_peered":"2024-04-24T08:15:22.996795+0000","last_unstale":"2024-04-24T08:15:25.003313+0000","last_undegraded":"2024-04-24T08:15:25.003313+0000","last_fullsized":"2024-04-24T08:15:25.003313+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:13:45.472014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385260+0000","last_change":"2024-04-24T08:15:24.007769+0000","last_active":"2024-04-24T08:15:25.385260+0000","last_peered":"2024-04-24T08:15:25.385260+0000","last_clean":"2024-04-24T08:15:25.385260+0000","last_became_active":"2024-04-24T08:15:24.007360+0000","last_became_peered":"2024-04-24T08:15:24.007360+0000","last_unstale":"2024-04-24T08:15:25.385260+0000","last_undegraded":"2024-04-24T08:15:25.385260+0000","last_fullsized":"2024-04-24T08:15:25.385260+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:14:19.482161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385555+0000","last_change":"2024-04-24T08:15:22.989631+0000","last_active":"2024-04-24T08:15:25.385555+0000","last_peered":"2024-04-24T08:15:25.385555+0000","last_clean":"2024-04-24T08:15:25.385555+0000","last_became_active":"2024-04-24T08:15:22.989332+0000","last_became_peered":"2024-04-24T08:15:22.989332+0000","last_unstale":"2024-04-24T08:15:25.385555+0000","last_undegraded":"2024-04-24T08:15:25.385555+0000","last_fullsized":"2024-04-24T08:15:25.385555+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:23:17.129592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.002987+0000","last_change":"2024-04-24T08:15:24.004313+0000","last_active":"2024-04-24T08:15:25.002987+0000","last_peered":"2024-04-24T08:15:25.002987+0000","last_clean":"2024-04-24T08:15:25.002987+0000","last_became_active":"2024-04-24T08:15:24.004027+0000","last_became_peered":"2024-04-24T08:15:24.004027+0000","last_unstale":"2024-04-24T08:15:25.002987+0000","last_undegraded":"2024-04-24T08:15:25.002987+0000","last_fullsized":"2024-04-24T08:15:25.002987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:04:19.234002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.000948+0000","last_change":"2024-04-24T08:15:23.000665+0000","last_active":"2024-04-24T08:15:25.000948+0000","last_peered":"2024-04-24T08:15:25.000948+0000","last_clean":"2024-04-24T08:15:25.000948+0000","last_became_active":"2024-04-24T08:15:23.000363+0000","last_became_peered":"2024-04-24T08:15:23.000363+0000","last_unstale":"2024-04-24T08:15:25.000948+0000","last_undegraded":"2024-04-24T08:15:25.000948+0000","last_fullsized":"2024-04-24T08:15:25.000948+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:17:48.189131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.002861+0000","last_change":"2024-04-24T08:15:23.995876+0000","last_active":"2024-04-24T08:15:25.002861+0000","last_peered":"2024-04-24T08:15:25.002861+0000","last_clean":"2024-04-24T08:15:25.002861+0000","last_became_active":"2024-04-24T08:15:23.995623+0000","last_became_peered":"2024-04-24T08:15:23.995623+0000","last_unstale":"2024-04-24T08:15:25.002861+0000","last_undegraded":"2024-04-24T08:15:25.002861+0000","last_fullsized":"2024-04-24T08:15:25.002861+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:15:54.911756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"81'8","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.997471+0000","last_change":"2024-04-24T08:15:23.005385+0000","last_active":"2024-04-24T08:15:24.997471+0000","last_peered":"2024-04-24T08:15:24.997471+0000","last_clean":"2024-04-24T08:15:24.997471+0000","last_became_active":"2024-04-24T08:15:23.004176+0000","last_became_peered":"2024-04-24T08:15:23.004176+0000","last_unstale":"2024-04-24T08:15:24.997471+0000","last_undegraded":"2024-04-24T08:15:24.997471+0000","last_fullsized":"2024-04-24T08:15:24.997471+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:56:26.008112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.386208+0000","last_change":"2024-04-24T08:15:23.999806+0000","last_active":"2024-04-24T08:15:25.386208+0000","last_peered":"2024-04-24T08:15:25.386208+0000","last_clean":"2024-04-24T08:15:25.386208+0000","last_became_active":"2024-04-24T08:15:23.996642+0000","last_became_peered":"2024-04-24T08:15:23.996642+0000","last_unstale":"2024-04-24T08:15:25.386208+0000","last_undegraded":"2024-04-24T08:15:25.386208+0000","last_fullsized":"2024-04-24T08:15:25.386208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:23:18.145229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.387781+0000","last_change":"2024-04-24T08:15:23.007867+0000","last_active":"2024-04-24T08:15:25.387781+0000","last_peered":"2024-04-24T08:15:25.387781+0000","last_clean":"2024-04-24T08:15:25.387781+0000","last_became_active":"2024-04-24T08:15:23.007560+0000","last_became_peered":"2024-04-24T08:15:23.007560+0000","last_unstale":"2024-04-24T08:15:25.387781+0000","last_undegraded":"2024-04-24T08:15:25.387781+0000","last_fullsized":"2024-04-24T08:15:25.387781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:15:30.810147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.387519+0000","last_change":"2024-04-24T08:15:24.010748+0000","last_active":"2024-04-24T08:15:25.387519+0000","last_peered":"2024-04-24T08:15:25.387519+0000","last_clean":"2024-04-24T08:15:25.387519+0000","last_became_active":"2024-04-24T08:15:24.009872+0000","last_became_peered":"2024-04-24T08:15:24.009872+0000","last_unstale":"2024-04-24T08:15:25.387519+0000","last_undegraded":"2024-04-24T08:15:25.387519+0000","last_fullsized":"2024-04-24T08:15:25.387519+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:48:55.418785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.999104+0000","last_change":"2024-04-24T08:15:23.002922+0000","last_active":"2024-04-24T08:15:24.999104+0000","last_peered":"2024-04-24T08:15:24.999104+0000","last_clean":"2024-04-24T08:15:24.999104+0000","last_became_active":"2024-04-24T08:15:23.002433+0000","last_became_peered":"2024-04-24T08:15:23.002433+0000","last_unstale":"2024-04-24T08:15:24.999104+0000","last_undegraded":"2024-04-24T08:15:24.999104+0000","last_fullsized":"2024-04-24T08:15:24.999104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:02:20.086095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.999779+0000","last_change":"2024-04-24T08:15:23.995587+0000","last_active":"2024-04-24T08:15:24.999779+0000","last_peered":"2024-04-24T08:15:24.999779+0000","last_clean":"2024-04-24T08:15:24.999779+0000","last_became_active":"2024-04-24T08:15:23.995249+0000","last_became_peered":"2024-04-24T08:15:23.995249+0000","last_unstale":"2024-04-24T08:15:24.999779+0000","last_undegraded":"2024-04-24T08:15:24.999779+0000","last_fullsized":"2024-04-24T08:15:24.999779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:03.797611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998509+0000","last_change":"2024-04-24T08:15:22.997691+0000","last_active":"2024-04-24T08:15:24.998509+0000","last_peered":"2024-04-24T08:15:24.998509+0000","last_clean":"2024-04-24T08:15:24.998509+0000","last_became_active":"2024-04-24T08:15:22.997384+0000","last_became_peered":"2024-04-24T08:15:22.997384+0000","last_unstale":"2024-04-24T08:15:24.998509+0000","last_undegraded":"2024-04-24T08:15:24.998509+0000","last_fullsized":"2024-04-24T08:15:24.998509+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:48:31.614989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384359+0000","last_change":"2024-04-24T08:15:24.002316+0000","last_active":"2024-04-24T08:15:25.384359+0000","last_peered":"2024-04-24T08:15:25.384359+0000","last_clean":"2024-04-24T08:15:25.384359+0000","last_became_active":"2024-04-24T08:15:24.002022+0000","last_became_peered":"2024-04-24T08:15:24.002022+0000","last_unstale":"2024-04-24T08:15:25.384359+0000","last_undegraded":"2024-04-24T08:15:25.384359+0000","last_fullsized":"2024-04-24T08:15:25.384359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:20.062835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385729+0000","last_change":"2024-04-24T08:15:23.002900+0000","last_active":"2024-04-24T08:15:25.385729+0000","last_peered":"2024-04-24T08:15:25.385729+0000","last_clean":"2024-04-24T08:15:25.385729+0000","last_became_active":"2024-04-24T08:15:23.002447+0000","last_became_peered":"2024-04-24T08:15:23.002447+0000","last_unstale":"2024-04-24T08:15:25.385729+0000","last_undegraded":"2024-04-24T08:15:25.385729+0000","last_fullsized":"2024-04-24T08:15:25.385729+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:10:23.659935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.001212+0000","last_change":"2024-04-24T08:15:23.993279+0000","last_active":"2024-04-24T08:15:25.001212+0000","last_peered":"2024-04-24T08:15:25.001212+0000","last_clean":"2024-04-24T08:15:25.001212+0000","last_became_active":"2024-04-24T08:15:23.992977+0000","last_became_peered":"2024-04-24T08:15:23.992977+0000","last_unstale":"2024-04-24T08:15:25.001212+0000","last_undegraded":"2024-04-24T08:15:25.001212+0000","last_fullsized":"2024-04-24T08:15:25.001212+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:56:27.023748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384695+0000","last_change":"2024-04-24T08:15:22.998536+0000","last_active":"2024-04-24T08:15:25.384695+0000","last_peered":"2024-04-24T08:15:25.384695+0000","last_clean":"2024-04-24T08:15:25.384695+0000","last_became_active":"2024-04-24T08:15:22.998231+0000","last_became_peered":"2024-04-24T08:15:22.998231+0000","last_unstale":"2024-04-24T08:15:25.384695+0000","last_undegraded":"2024-04-24T08:15:25.384695+0000","last_fullsized":"2024-04-24T08:15:25.384695+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:36.353930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384740+0000","last_change":"2024-04-24T08:15:24.002709+0000","last_active":"2024-04-24T08:15:25.384740+0000","last_peered":"2024-04-24T08:15:25.384740+0000","last_clean":"2024-04-24T08:15:25.384740+0000","last_became_active":"2024-04-24T08:15:24.002215+0000","last_became_peered":"2024-04-24T08:15:24.002215+0000","last_unstale":"2024-04-24T08:15:25.384740+0000","last_undegraded":"2024-04-24T08:15:25.384740+0000","last_fullsized":"2024-04-24T08:15:25.384740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:07:01.413983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.383909+0000","last_change":"2024-04-24T08:15:23.008190+0000","last_active":"2024-04-24T08:15:25.383909+0000","last_peered":"2024-04-24T08:15:25.383909+0000","last_clean":"2024-04-24T08:15:25.383909+0000","last_became_active":"2024-04-24T08:15:23.007876+0000","last_became_peered":"2024-04-24T08:15:23.007876+0000","last_unstale":"2024-04-24T08:15:25.383909+0000","last_undegraded":"2024-04-24T08:15:25.383909+0000","last_fullsized":"2024-04-24T08:15:25.383909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:51:38.524421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.000798+0000","last_change":"2024-04-24T08:15:23.995318+0000","last_active":"2024-04-24T08:15:25.000798+0000","last_peered":"2024-04-24T08:15:25.000798+0000","last_clean":"2024-04-24T08:15:25.000798+0000","last_became_active":"2024-04-24T08:15:23.995002+0000","last_became_peered":"2024-04-24T08:15:23.995002+0000","last_unstale":"2024-04-24T08:15:25.000798+0000","last_undegraded":"2024-04-24T08:15:25.000798+0000","last_fullsized":"2024-04-24T08:15:25.000798+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:15:31.825784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384476+0000","last_change":"2024-04-24T08:15:23.003814+0000","last_active":"2024-04-24T08:15:25.384476+0000","last_peered":"2024-04-24T08:15:25.384476+0000","last_clean":"2024-04-24T08:15:25.384476+0000","last_became_active":"2024-04-24T08:15:23.003549+0000","last_became_peered":"2024-04-24T08:15:23.003549+0000","last_unstale":"2024-04-24T08:15:25.384476+0000","last_undegraded":"2024-04-24T08:15:25.384476+0000","last_fullsized":"2024-04-24T08:15:25.384476+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:48:31.614989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.000282+0000","last_change":"2024-04-24T08:15:23.992964+0000","last_active":"2024-04-24T08:15:25.000282+0000","last_peered":"2024-04-24T08:15:25.000282+0000","last_clean":"2024-04-24T08:15:25.000282+0000","last_became_active":"2024-04-24T08:15:23.992547+0000","last_became_peered":"2024-04-24T08:15:23.992547+0000","last_unstale":"2024-04-24T08:15:25.000282+0000","last_undegraded":"2024-04-24T08:15:25.000282+0000","last_fullsized":"2024-04-24T08:15:25.000282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:53:36.741944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"81'2","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.003734+0000","last_change":"2024-04-24T08:15:23.003384+0000","last_active":"2024-04-24T08:15:25.003734+0000","last_peered":"2024-04-24T08:15:25.003734+0000","last_clean":"2024-04-24T08:15:25.003734+0000","last_became_active":"2024-04-24T08:15:23.003108+0000","last_became_peered":"2024-04-24T08:15:23.003108+0000","last_unstale":"2024-04-24T08:15:25.003734+0000","last_undegraded":"2024-04-24T08:15:25.003734+0000","last_fullsized":"2024-04-24T08:15:25.003734+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:15:49.086332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.386272+0000","last_change":"2024-04-24T08:15:24.004666+0000","last_active":"2024-04-24T08:15:25.386272+0000","last_peered":"2024-04-24T08:15:25.386272+0000","last_clean":"2024-04-24T08:15:25.386272+0000","last_became_active":"2024-04-24T08:15:24.001496+0000","last_became_peered":"2024-04-24T08:15:24.001496+0000","last_unstale":"2024-04-24T08:15:25.386272+0000","last_undegraded":"2024-04-24T08:15:25.386272+0000","last_fullsized":"2024-04-24T08:15:25.386272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:28:40.421322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"82'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.004398+0000","last_change":"2024-04-24T08:15:22.999739+0000","last_active":"2024-04-24T08:15:25.004398+0000","last_peered":"2024-04-24T08:15:25.004398+0000","last_clean":"2024-04-24T08:15:25.004398+0000","last_became_active":"2024-04-24T08:15:22.998302+0000","last_became_peered":"2024-04-24T08:15:22.998302+0000","last_unstale":"2024-04-24T08:15:25.004398+0000","last_undegraded":"2024-04-24T08:15:25.004398+0000","last_fullsized":"2024-04-24T08:15:25.004398+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:27:20.319330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.386621+0000","last_change":"2024-04-24T08:15:24.007637+0000","last_active":"2024-04-24T08:15:25.386621+0000","last_peered":"2024-04-24T08:15:25.386621+0000","last_clean":"2024-04-24T08:15:25.386621+0000","last_became_active":"2024-04-24T08:15:24.007364+0000","last_became_peered":"2024-04-24T08:15:24.007364+0000","last_unstale":"2024-04-24T08:15:25.386621+0000","last_undegraded":"2024-04-24T08:15:25.386621+0000","last_fullsized":"2024-04-24T08:15:25.386621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:19:17.856347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385709+0000","last_change":"2024-04-24T08:15:23.006451+0000","last_active":"2024-04-24T08:15:25.385709+0000","last_peered":"2024-04-24T08:15:25.385709+0000","last_clean":"2024-04-24T08:15:25.385709+0000","last_became_active":"2024-04-24T08:15:23.005935+0000","last_became_peered":"2024-04-24T08:15:23.005935+0000","last_unstale":"2024-04-24T08:15:25.385709+0000","last_undegraded":"2024-04-24T08:15:25.385709+0000","last_fullsized":"2024-04-24T08:15:25.385709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:17:29.597202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.999941+0000","last_change":"2024-04-24T08:15:24.001657+0000","last_active":"2024-04-24T08:15:24.999941+0000","last_peered":"2024-04-24T08:15:24.999941+0000","last_clean":"2024-04-24T08:15:24.999941+0000","last_became_active":"2024-04-24T08:15:24.001245+0000","last_became_peered":"2024-04-24T08:15:24.001245+0000","last_unstale":"2024-04-24T08:15:24.999941+0000","last_undegraded":"2024-04-24T08:15:24.999941+0000","last_fullsized":"2024-04-24T08:15:24.999941+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:19:16.013532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.386600+0000","last_change":"2024-04-24T08:15:23.004865+0000","last_active":"2024-04-24T08:15:25.386600+0000","last_peered":"2024-04-24T08:15:25.386600+0000","last_clean":"2024-04-24T08:15:25.386600+0000","last_became_active":"2024-04-24T08:15:23.004355+0000","last_became_peered":"2024-04-24T08:15:23.004355+0000","last_unstale":"2024-04-24T08:15:25.386600+0000","last_undegraded":"2024-04-24T08:15:25.386600+0000","last_fullsized":"2024-04-24T08:15:25.386600+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:07:26.028117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.001427+0000","last_change":"2024-04-24T08:15:23.993180+0000","last_active":"2024-04-24T08:15:25.001427+0000","last_peered":"2024-04-24T08:15:25.001427+0000","last_clean":"2024-04-24T08:15:25.001427+0000","last_became_active":"2024-04-24T08:15:23.992517+0000","last_became_peered":"2024-04-24T08:15:23.992517+0000","last_unstale":"2024-04-24T08:15:25.001427+0000","last_undegraded":"2024-04-24T08:15:25.001427+0000","last_fullsized":"2024-04-24T08:15:25.001427+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:09:28.934082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998100+0000","last_change":"2024-04-24T08:15:23.005219+0000","last_active":"2024-04-24T08:15:24.998100+0000","last_peered":"2024-04-24T08:15:24.998100+0000","last_clean":"2024-04-24T08:15:24.998100+0000","last_became_active":"2024-04-24T08:15:23.004788+0000","last_became_peered":"2024-04-24T08:15:23.004788+0000","last_unstale":"2024-04-24T08:15:24.998100+0000","last_undegraded":"2024-04-24T08:15:24.998100+0000","last_fullsized":"2024-04-24T08:15:24.998100+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:09:27.918445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.997851+0000","last_change":"2024-04-24T08:15:24.003083+0000","last_active":"2024-04-24T08:15:24.997851+0000","last_peered":"2024-04-24T08:15:24.997851+0000","last_clean":"2024-04-24T08:15:24.997851+0000","last_became_active":"2024-04-24T08:15:24.002741+0000","last_became_peered":"2024-04-24T08:15:24.002741+0000","last_unstale":"2024-04-24T08:15:24.997851+0000","last_undegraded":"2024-04-24T08:15:24.997851+0000","last_fullsized":"2024-04-24T08:15:24.997851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:45:06.324376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998888+0000","last_change":"2024-04-24T08:15:22.994398+0000","last_active":"2024-04-24T08:15:24.998888+0000","last_peered":"2024-04-24T08:15:24.998888+0000","last_clean":"2024-04-24T08:15:24.998888+0000","last_became_active":"2024-04-24T08:15:22.993888+0000","last_became_peered":"2024-04-24T08:15:22.993888+0000","last_unstale":"2024-04-24T08:15:24.998888+0000","last_undegraded":"2024-04-24T08:15:24.998888+0000","last_fullsized":"2024-04-24T08:15:24.998888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:16:51.843549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384780+0000","last_change":"2024-04-24T08:15:24.003169+0000","last_active":"2024-04-24T08:15:25.384780+0000","last_peered":"2024-04-24T08:15:25.384780+0000","last_clean":"2024-04-24T08:15:25.384780+0000","last_became_active":"2024-04-24T08:15:24.002846+0000","last_became_peered":"2024-04-24T08:15:24.002846+0000","last_unstale":"2024-04-24T08:15:25.384780+0000","last_undegraded":"2024-04-24T08:15:25.384780+0000","last_fullsized":"2024-04-24T08:15:25.384780+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:07:52.467176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.386172+0000","last_change":"2024-04-24T08:15:23.008814+0000","last_active":"2024-04-24T08:15:25.386172+0000","last_peered":"2024-04-24T08:15:25.386172+0000","last_clean":"2024-04-24T08:15:25.386172+0000","last_became_active":"2024-04-24T08:15:23.008582+0000","last_became_peered":"2024-04-24T08:15:23.008582+0000","last_unstale":"2024-04-24T08:15:25.386172+0000","last_undegraded":"2024-04-24T08:15:25.386172+0000","last_fullsized":"2024-04-24T08:15:25.386172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:09:44.565995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385907+0000","last_change":"2024-04-24T08:15:24.006124+0000","last_active":"2024-04-24T08:15:25.385907+0000","last_peered":"2024-04-24T08:15:25.385907+0000","last_clean":"2024-04-24T08:15:25.385907+0000","last_became_active":"2024-04-24T08:15:24.005796+0000","last_became_peered":"2024-04-24T08:15:24.005796+0000","last_unstale":"2024-04-24T08:15:25.385907+0000","last_undegraded":"2024-04-24T08:15:25.385907+0000","last_fullsized":"2024-04-24T08:15:25.385907+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:19:16.013532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:24.998553+0000","last_change":"2024-04-24T08:15:23.008676+0000","last_active":"2024-04-24T08:15:24.998553+0000","last_peered":"2024-04-24T08:15:24.998553+0000","last_clean":"2024-04-24T08:15:24.998553+0000","last_became_active":"2024-04-24T08:15:23.007815+0000","last_became_peered":"2024-04-24T08:15:23.007815+0000","last_unstale":"2024-04-24T08:15:24.998553+0000","last_undegraded":"2024-04-24T08:15:24.998553+0000","last_fullsized":"2024-04-24T08:15:24.998553+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:06:37.076973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.003623+0000","last_change":"2024-04-24T08:15:23.996454+0000","last_active":"2024-04-24T08:15:25.003623+0000","last_peered":"2024-04-24T08:15:25.003623+0000","last_clean":"2024-04-24T08:15:25.003623+0000","last_became_active":"2024-04-24T08:15:23.996145+0000","last_became_peered":"2024-04-24T08:15:23.996145+0000","last_unstale":"2024-04-24T08:15:25.003623+0000","last_undegraded":"2024-04-24T08:15:25.003623+0000","last_fullsized":"2024-04-24T08:15:25.003623+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:22.954537+0000","last_clean_scrub_stamp":"2024-04-24T08:15:22.954537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:09:45.581631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"81'1","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.385999+0000","last_change":"2024-04-24T08:15:23.007108+0000","last_active":"2024-04-24T08:15:25.385999+0000","last_peered":"2024-04-24T08:15:25.385999+0000","last_clean":"2024-04-24T08:15:25.385999+0000","last_became_active":"2024-04-24T08:15:23.006755+0000","last_became_peered":"2024-04-24T08:15:23.006755+0000","last_unstale":"2024-04-24T08:15:25.385999+0000","last_undegraded":"2024-04-24T08:15:25.385999+0000","last_fullsized":"2024-04-24T08:15:25.385999+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:15:21.938900+0000","last_clean_scrub_stamp":"2024-04-24T08:15:21.938900+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:07:26.028117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":284,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-24T08:15:25.384920+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:15:25.384920+0000","last_peered":"2024-04-24T08:15:25.384920+0000","last_clean":"2024-04-24T08:15:25.384920+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:15:25.384920+0000","last_undegraded":"2024-04-24T08:15:25.384920+0000","last_fullsized":"2024-04-24T08:15:25.384920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32756,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339084,"statfs":{"total":96636764160,"available":96603222016,"internally_reserved":0,"allocated":1605632,"data_stored":1021315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574941,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":431571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574939,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32188,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339652,"statfs":{"total":96636764160,"available":96603803648,"internally_reserved":0,"allocated":1024000,"data_stored":432511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574938,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":1007616,"data_stored":430877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574939,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":432489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574940,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339672,"statfs":{"total":96636764160,"available":96603824128,"internally_reserved":0,"allocated":1011712,"data_stored":431459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":431071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574939,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32756,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339084,"statfs":{"total":96636764160,"available":96603222016,"internally_reserved":0,"allocated":1605632,"data_stored":1021279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:15:28.908 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T08:15:28.908 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-24T08:15:29.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.161+0000 7f134b200640 1 Processor -- start 2024-04-24T08:15:29.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.162+0000 7f134b200640 1 -- start start 2024-04-24T08:15:29.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.163+0000 7f134b200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f134c0d0a60 0x7f134c06f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.163+0000 7f134b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0b4d90 0x7f134c0b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.163+0000 7f134b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f134c0b3cf0 con 0x7f134c0aeb10 2024-04-24T08:15:29.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.163+0000 7f134b200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f134c0b3fd0 con 0x7f134c0d0a60 2024-04-24T08:15:29.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.163+0000 7f134b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f134c0b42b0 con 0x7f134c0b4d90 2024-04-24T08:15:29.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.164+0000 7f134a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0b4d90 0x7f134c0b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:29.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.164+0000 7f1349400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f134c0d0a60 0x7f134c06f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:29.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.164+0000 7f134a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0b4d90 0x7f134c0b35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:51816/0 (socket says 172.21.15.69:51816) 2024-04-24T08:15:29.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.164+0000 7f134a800640 1 -- 172.21.15.69:0/3169219746 learned_addr learned my addr 172.21.15.69:0/3169219746 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:29.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.164+0000 7f134a800640 1 -- 172.21.15.69:0/3169219746 >> v2:172.21.15.121:3300/0 conn(0x7f134c0d0a60 msgr2=0x7f134c06f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:29.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.164+0000 7f134a800640 1 --2- 172.21.15.69:0/3169219746 >> v2:172.21.15.121:3300/0 conn(0x7f134c0d0a60 0x7f134c06f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.164+0000 7f134a800640 1 -- 172.21.15.69:0/3169219746 >> v1:172.21.15.121:6789/0 conn(0x7f134c0aeb10 legacy=0x7f134c0d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.164+0000 7f134a800640 1 -- 172.21.15.69:0/3169219746 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134c0b4590 con 0x7f134c0b4d90 2024-04-24T08:15:29.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.166+0000 7f134a800640 1 --2- 172.21.15.69:0/3169219746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0b4d90 0x7f134c0b35f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1344009e50 tx=0x7f134402a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d26d52c5c4f8a792 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:29.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.166+0000 7f1348a00640 1 -- 172.21.15.69:0/3169219746 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344036070 con 0x7f134c0b4d90 2024-04-24T08:15:29.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.166+0000 7f1348a00640 1 -- 172.21.15.69:0/3169219746 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1344031600 con 0x7f134c0b4d90 2024-04-24T08:15:29.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.166+0000 7f1348a00640 1 -- 172.21.15.69:0/3169219746 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13440319d0 con 0x7f134c0b4d90 2024-04-24T08:15:29.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.167+0000 7f134b200640 1 -- 172.21.15.69:0/3169219746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0b4d90 msgr2=0x7f134c0b35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.167+0000 7f134b200640 1 --2- 172.21.15.69:0/3169219746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0b4d90 0x7f134c0b35f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1344009e50 tx=0x7f134402a710 comp rx=0 tx=0).stop 2024-04-24T08:15:29.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.167+0000 7f134b200640 1 -- 172.21.15.69:0/3169219746 shutdown_connections 2024-04-24T08:15:29.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.167+0000 7f134b200640 1 --2- 172.21.15.69:0/3169219746 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0b4d90 0x7f134c0b35f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.167+0000 7f134b200640 1 --2- 172.21.15.69:0/3169219746 >> v2:172.21.15.121:3300/0 conn(0x7f134c0d0a60 0x7f134c06f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.167+0000 7f134b200640 1 -- 172.21.15.69:0/3169219746 >> 172.21.15.69:0/3169219746 conn(0x7f134c06bc00 msgr2=0x7f134c06c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:29.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.167+0000 7f134b200640 1 -- 172.21.15.69:0/3169219746 shutdown_connections 2024-04-24T08:15:29.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.167+0000 7f134b200640 1 -- 172.21.15.69:0/3169219746 wait complete. 2024-04-24T08:15:29.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.169+0000 7f134b200640 1 Processor -- start 2024-04-24T08:15:29.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.169+0000 7f134b200640 1 -- start start 2024-04-24T08:15:29.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.170+0000 7f134b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0aeb10 0x7f134c0c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.170+0000 7f134b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f134c0b4d90 0x7f134c0c6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.170+0000 7f134b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f134c0d0a60 0x7f134c14db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.170+0000 7f134b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f134c0ad140 con 0x7f134c0b4d90 2024-04-24T08:15:29.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.170+0000 7f134b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f134c0acfc0 con 0x7f134c0d0a60 2024-04-24T08:15:29.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.170+0000 7f134b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f134c0ad2c0 con 0x7f134c0aeb10 2024-04-24T08:15:29.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.170+0000 7f1349400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f134c0b4d90 0x7f134c0c6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:29.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.170+0000 7f1349400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f134c0b4d90 0x7f134c0c6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33088/0 (socket says 172.21.15.69:33088) 2024-04-24T08:15:29.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.170+0000 7f1349400640 1 -- 172.21.15.69:0/653352227 learned_addr learned my addr 172.21.15.69:0/653352227 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:29.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.170+0000 7f1349e00640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0aeb10 0x7f134c0c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:29.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f134a800640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f134c0d0a60 0x7f134c14db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:29.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f1349400640 1 -- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0aeb10 msgr2=0x7f134c0c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f1349400640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0aeb10 0x7f134c0c4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f1349400640 1 -- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f134c0d0a60 msgr2=0x7f134c14db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f1349400640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f134c0d0a60 0x7f134c14db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f1349400640 1 -- 172.21.15.69:0/653352227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134c0c55e0 con 0x7f134c0b4d90 2024-04-24T08:15:29.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f1349e00640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0aeb10 0x7f134c0c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:29.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f1349400640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f134c0b4d90 0x7f134c0c6680 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f133400e870 tx=0x7f133400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:29.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f1342000640 1 -- 172.21.15.69:0/653352227 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1334017070 con 0x7f134c0b4d90 2024-04-24T08:15:29.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f1342000640 1 -- 172.21.15.69:0/653352227 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f133400c940 con 0x7f134c0b4d90 2024-04-24T08:15:29.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f1342000640 1 -- 172.21.15.69:0/653352227 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133400cd30 con 0x7f134c0b4d90 2024-04-24T08:15:29.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f134b200640 1 -- 172.21.15.69:0/653352227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134c0c58c0 con 0x7f134c0b4d90 2024-04-24T08:15:29.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.171+0000 7f134b200640 1 -- 172.21.15.69:0/653352227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134c0c5e80 con 0x7f134c0b4d90 2024-04-24T08:15:29.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.172+0000 7f134b200640 1 -- 172.21.15.69:0/653352227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134c075d50 con 0x7f134c0b4d90 2024-04-24T08:15:29.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.175+0000 7f1342000640 1 -- 172.21.15.69:0/653352227 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f133401a070 con 0x7f134c0b4d90 2024-04-24T08:15:29.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.176+0000 7f1342000640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f132806c8f0 0x7f132806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.176+0000 7f1342000640 1 -- 172.21.15.69:0/653352227 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f133408bf00 con 0x7f134c0b4d90 2024-04-24T08:15:29.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.177+0000 7f1349e00640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f132806c8f0 0x7f132806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:29.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.178+0000 7f1349e00640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f132806c8f0 0x7f132806edb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f133c001680 tx=0x7f133c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.181+0000 7f1342000640 1 -- 172.21.15.69:0/653352227 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f133405a210 con 0x7f134c0b4d90 2024-04-24T08:15:29.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.459+0000 7f134b200640 1 -- 172.21.15.69:0/653352227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f134c0cc370 con 0x7f134c0b4d90 2024-04-24T08:15:29.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.464+0000 7f1342000640 1 -- 172.21.15.69:0/653352227 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) ==== 190+0+147 (secure 0 0 0) 0x7f1334059b10 con 0x7f134c0b4d90 2024-04-24T08:15:29.465 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-24T08:15:29.465 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw== 2024-04-24T08:15:29.465 INFO:teuthology.orchestra.run.smithi069.stdout: caps mds = "allow" 2024-04-24T08:15:29.465 INFO:teuthology.orchestra.run.smithi069.stdout: caps mon = "allow r" 2024-04-24T08:15:29.465 INFO:teuthology.orchestra.run.smithi069.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-24T08:15:29.465 INFO:teuthology.orchestra.run.smithi069.stderr:updated caps for client.0 2024-04-24T08:15:29.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.467+0000 7f134b200640 1 -- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f132806c8f0 msgr2=0x7f132806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.467+0000 7f134b200640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f132806c8f0 0x7f132806edb0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f133c001680 tx=0x7f133c009040 comp rx=0 tx=0).stop 2024-04-24T08:15:29.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.467+0000 7f134b200640 1 -- 172.21.15.69:0/653352227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f134c0b4d90 msgr2=0x7f134c0c6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.467+0000 7f134b200640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f134c0b4d90 0x7f134c0c6680 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f133400e870 tx=0x7f133400ed30 comp rx=0 tx=0).stop 2024-04-24T08:15:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.468+0000 7f134b200640 1 -- 172.21.15.69:0/653352227 shutdown_connections 2024-04-24T08:15:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.468+0000 7f134b200640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f132806c8f0 0x7f132806edb0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.468+0000 7f134b200640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f134c0b4d90 0x7f134c0c6680 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.468+0000 7f134b200640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f134c0d0a60 0x7f134c14db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.468+0000 7f134b200640 1 --2- 172.21.15.69:0/653352227 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f134c0aeb10 0x7f134c0c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.468+0000 7f134b200640 1 -- 172.21.15.69:0/653352227 >> 172.21.15.69:0/653352227 conn(0x7f134c06bc00 msgr2=0x7f134c158180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.468+0000 7f134b200640 1 -- 172.21.15.69:0/653352227 shutdown_connections 2024-04-24T08:15:29.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.468+0000 7f134b200640 1 -- 172.21.15.69:0/653352227 wait complete. 2024-04-24T08:15:29.489 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:29.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.683+0000 7fd6db000640 1 Processor -- start 2024-04-24T08:15:29.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.684+0000 7fd6db000640 1 -- start start 2024-04-24T08:15:29.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.685+0000 7fd6db000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd6d40d0300 0x7fd6d406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.685+0000 7fd6db000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6d4145fe0 0x7fd6d4160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.685+0000 7fd6db000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d40cf3d0 con 0x7fd6d4145a90 2024-04-24T08:15:29.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.685+0000 7fd6db000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd6d40cf6b0 con 0x7fd6d40d0300 2024-04-24T08:15:29.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.685+0000 7fd6db000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d40cf990 con 0x7fd6d4145fe0 2024-04-24T08:15:29.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.685+0000 7fd6d3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd6d40d0300 0x7fd6d406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:29.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.686+0000 7fd6d3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd6d40d0300 0x7fd6d406e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33094/0 (socket says 172.21.15.69:33094) 2024-04-24T08:15:29.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.686+0000 7fd6d3e00640 1 -- 172.21.15.69:0/2929047688 learned_addr learned my addr 172.21.15.69:0/2929047688 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:29.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.686+0000 7fd6d9000640 1 --2- 172.21.15.69:0/2929047688 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6d4145fe0 0x7fd6d4160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:29.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.686+0000 7fd6d3e00640 1 -- 172.21.15.69:0/2929047688 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6d4145fe0 msgr2=0x7fd6d4160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:29.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.686+0000 7fd6d3e00640 1 --2- 172.21.15.69:0/2929047688 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6d4145fe0 0x7fd6d4160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.686+0000 7fd6d3e00640 1 -- 172.21.15.69:0/2929047688 >> v1:172.21.15.69:6789/0 conn(0x7fd6d4145a90 legacy=0x7fd6d40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.686+0000 7fd6d3e00640 1 -- 172.21.15.69:0/2929047688 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d40cfc70 con 0x7fd6d40d0300 2024-04-24T08:15:29.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.686+0000 7fd6d9000640 1 --2- 172.21.15.69:0/2929047688 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6d4145fe0 0x7fd6d4160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:29.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.687+0000 7fd6d3e00640 1 --2- 172.21.15.69:0/2929047688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 0x7fd6d406e240 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009870 tx=0x7fd6c402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92bde5081719ae60 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:29.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.687+0000 7fd6d2a00640 1 -- 172.21.15.69:0/2929047688 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4037070 con 0x7fd6d40d0300 2024-04-24T08:15:29.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.687+0000 7fd6d2a00640 1 -- 172.21.15.69:0/2929047688 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c4032440 con 0x7fd6d40d0300 2024-04-24T08:15:29.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.687+0000 7fd6d2a00640 1 -- 172.21.15.69:0/2929047688 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4032810 con 0x7fd6d40d0300 2024-04-24T08:15:29.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.688+0000 7fd6db000640 1 -- 172.21.15.69:0/2929047688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 msgr2=0x7fd6d406e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.688+0000 7fd6db000640 1 --2- 172.21.15.69:0/2929047688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 0x7fd6d406e240 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009870 tx=0x7fd6c402a250 comp rx=0 tx=0).stop 2024-04-24T08:15:29.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.688+0000 7fd6db000640 1 -- 172.21.15.69:0/2929047688 shutdown_connections 2024-04-24T08:15:29.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.688+0000 7fd6db000640 1 --2- 172.21.15.69:0/2929047688 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6d4145fe0 0x7fd6d4160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.688+0000 7fd6db000640 1 --2- 172.21.15.69:0/2929047688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 0x7fd6d406e240 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.688+0000 7fd6db000640 1 -- 172.21.15.69:0/2929047688 >> 172.21.15.69:0/2929047688 conn(0x7fd6d4067d90 msgr2=0x7fd6d406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:29.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.688+0000 7fd6db000640 1 -- 172.21.15.69:0/2929047688 shutdown_connections 2024-04-24T08:15:29.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.688+0000 7fd6db000640 1 -- 172.21.15.69:0/2929047688 wait complete. 2024-04-24T08:15:29.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.690+0000 7fd6db000640 1 Processor -- start 2024-04-24T08:15:29.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.691+0000 7fd6db000640 1 -- start start 2024-04-24T08:15:29.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.692+0000 7fd6db000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 0x7fd6d4137b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.692+0000 7fd6db000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6d4145a90 0x7fd6d4144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.692+0000 7fd6db000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6d4145fe0 0x7fd6d40a96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.692+0000 7fd6db000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41458f0 con 0x7fd6d40d0300 2024-04-24T08:15:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.692+0000 7fd6db000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d4145770 con 0x7fd6d4145a90 2024-04-24T08:15:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.692+0000 7fd6db000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d40cdc30 con 0x7fd6d4145fe0 2024-04-24T08:15:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.692+0000 7fd6d3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 0x7fd6d4137b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:29.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.692+0000 7fd6d3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 0x7fd6d4137b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33096/0 (socket says 172.21.15.69:33096) 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.692+0000 7fd6d3e00640 1 -- 172.21.15.69:0/3543817585 learned_addr learned my addr 172.21.15.69:0/3543817585 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6d3e00640 1 -- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6d4145fe0 msgr2=0x7fd6d40a96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6d3e00640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6d4145fe0 0x7fd6d40a96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6d3e00640 1 -- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6d4145a90 msgr2=0x7fd6d4144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6d3e00640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6d4145a90 0x7fd6d4144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6d3e00640 1 -- 172.21.15.69:0/3543817585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d4136a80 con 0x7fd6d40d0300 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6d3e00640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 0x7fd6d4137b20 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4000c00 tx=0x7fd6c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6cbe00640 1 -- 172.21.15.69:0/3543817585 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4037040 con 0x7fd6d40d0300 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6cbe00640 1 -- 172.21.15.69:0/3543817585 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c4042070 con 0x7fd6d40d0300 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6cbe00640 1 -- 172.21.15.69:0/3543817585 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4030bd0 con 0x7fd6d40d0300 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6db000640 1 -- 172.21.15.69:0/3543817585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d4136dc0 con 0x7fd6d40d0300 2024-04-24T08:15:29.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.693+0000 7fd6db000640 1 -- 172.21.15.69:0/3543817585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d4137600 con 0x7fd6d40d0300 2024-04-24T08:15:29.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.695+0000 7fd6db000640 1 -- 172.21.15.69:0/3543817585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d4068a80 con 0x7fd6d40d0300 2024-04-24T08:15:29.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.695+0000 7fd6cbe00640 1 -- 172.21.15.69:0/3543817585 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd6c4047460 con 0x7fd6d40d0300 2024-04-24T08:15:29.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.696+0000 7fd6cbe00640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6b006c920 0x7fd6b006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:29.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.696+0000 7fd6cbe00640 1 -- 172.21.15.69:0/3543817585 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fd6c40ac620 con 0x7fd6d40d0300 2024-04-24T08:15:29.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.697+0000 7fd6d3400640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6b006c920 0x7fd6b006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:29.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.698+0000 7fd6d3400640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6b006c920 0x7fd6b006ede0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc000f60 tx=0x7fd6bc0079d0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:29.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.701+0000 7fd6cbe00640 1 -- 172.21.15.69:0/3543817585 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd6d4136dc0 con 0x7fd6d40d0300 2024-04-24T08:15:29.935 INFO:tasks.ceph.mds.b.smithi121.stderr:2024-04-24T08:15:29.934+0000 7f1f79e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:15:29.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.946+0000 7fd6db000640 1 -- 172.21.15.69:0/3543817585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6d40d1120 con 0x7fd6d40d0300 2024-04-24T08:15:29.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.947+0000 7fd6cbe00640 1 -- 172.21.15.69:0/3543817585 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5180 (secure 0 0 0) 0x7fd6c4045360 con 0x7fd6d40d0300 2024-04-24T08:15:29.948 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:29.948 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7581,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/2529876410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2529876410},{"type":"v1","addr":"172.21.15.69:6833","nonce":2529876410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:15:26.487664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7593":{"gid":7593,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1179683245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1179683245},{"type":"v1","addr":"172.21.15.121:6837","nonce":1179683245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7593,"qdb_cluster":[7593]},"id":6}]} 2024-04-24T08:15:29.948 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 61 2024-04-24T08:15:29.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 -- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6b006c920 msgr2=0x7fd6b006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6b006c920 0x7fd6b006ede0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc000f60 tx=0x7fd6bc0079d0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 -- 172.21.15.69:0/3543817585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 msgr2=0x7fd6d4137b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:29.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 0x7fd6d4137b20 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4000c00 tx=0x7fd6c40046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 -- 172.21.15.69:0/3543817585 shutdown_connections 2024-04-24T08:15:29.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd6b006c920 0x7fd6b006ede0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd6d4145fe0 0x7fd6d40a96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6d4145a90 0x7fd6d4144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 --2- 172.21.15.69:0/3543817585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd6d40d0300 0x7fd6d4137b20 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:29.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 -- 172.21.15.69:0/3543817585 >> 172.21.15.69:0/3543817585 conn(0x7fd6d4067d90 msgr2=0x7fd6d40b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:29.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 -- 172.21.15.69:0/3543817585 shutdown_connections 2024-04-24T08:15:29.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:29.950+0000 7fd6db000640 1 -- 172.21.15.69:0/3543817585 wait complete. 2024-04-24T08:15:30.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:15:23.442689+0000', 'modified': '2024-04-24T08:15:26.487664+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7593}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7593': {'gid': 7593, 'name': 'b', 'rank': 0, 'incarnation': 57, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6837/1179683245', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6835', 'nonce': 1179683245}, {'type': 'v1', 'addr': '172.21.15.121:6837', 'nonce': 1179683245}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7593, 'qdb_cluster': [7593]} 2024-04-24T08:15:30.372 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:15:30.373 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T08:15:30.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.582+0000 7fde59000640 1 Processor -- start 2024-04-24T08:15:30.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.582+0000 7fde59000640 1 -- start start 2024-04-24T08:15:30.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.584+0000 7fde59000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fde540723a0 0x7fde540b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.584+0000 7fde59000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 0x7fde540b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.584+0000 7fde59000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde54163fb0 con 0x7fde5414c650 2024-04-24T08:15:30.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.584+0000 7fde59000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fde5406e130 con 0x7fde540723a0 2024-04-24T08:15:30.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.584+0000 7fde59000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fde5406e410 con 0x7fde541666e0 2024-04-24T08:15:30.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.585+0000 7fde53e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 0x7fde540b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.585+0000 7fde4aa00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fde540723a0 0x7fde540b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.585+0000 7fde53e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 0x7fde540b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:51850/0 (socket says 172.21.15.69:51850) 2024-04-24T08:15:30.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.585+0000 7fde53e00640 1 -- 172.21.15.69:0/2945747365 learned_addr learned my addr 172.21.15.69:0/2945747365 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:30.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.585+0000 7fde4aa00640 1 -- 172.21.15.69:0/2945747365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 msgr2=0x7fde540b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:30.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.585+0000 7fde4aa00640 1 --2- 172.21.15.69:0/2945747365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 0x7fde540b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.585+0000 7fde4aa00640 1 -- 172.21.15.69:0/2945747365 >> v1:172.21.15.121:6789/0 conn(0x7fde5414c650 legacy=0x7fde54071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.586+0000 7fde4aa00640 1 -- 172.21.15.69:0/2945747365 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5406e6f0 con 0x7fde540723a0 2024-04-24T08:15:30.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.586+0000 7fde53e00640 1 --2- 172.21.15.69:0/2945747365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 0x7fde540b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:30.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.586+0000 7fde52a00640 1 -- 172.21.15.69:0/2945747365 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3410099788 0 0) 0x7fde54163fb0 con 0x7fde5414c650 2024-04-24T08:15:30.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.587+0000 7fde4aa00640 1 --2- 172.21.15.69:0/2945747365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde540723a0 0x7fde540b30f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0097a0 tx=0x7fde3c02a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25190d8c8b2d7977 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:30.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.588+0000 7fde52a00640 1 -- 172.21.15.69:0/2945747365 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c037070 con 0x7fde540723a0 2024-04-24T08:15:30.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.588+0000 7fde52a00640 1 -- 172.21.15.69:0/2945747365 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde3c02ae50 con 0x7fde540723a0 2024-04-24T08:15:30.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.588+0000 7fde52a00640 1 -- 172.21.15.69:0/2945747365 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c032660 con 0x7fde540723a0 2024-04-24T08:15:30.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.588+0000 7fde59000640 1 -- 172.21.15.69:0/2945747365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde540723a0 msgr2=0x7fde540b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.588+0000 7fde59000640 1 --2- 172.21.15.69:0/2945747365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde540723a0 0x7fde540b30f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0097a0 tx=0x7fde3c02a100 comp rx=0 tx=0).stop 2024-04-24T08:15:30.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.588+0000 7fde59000640 1 -- 172.21.15.69:0/2945747365 shutdown_connections 2024-04-24T08:15:30.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.588+0000 7fde59000640 1 --2- 172.21.15.69:0/2945747365 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 0x7fde540b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.588+0000 7fde59000640 1 --2- 172.21.15.69:0/2945747365 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde540723a0 0x7fde540b30f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.588+0000 7fde59000640 1 -- 172.21.15.69:0/2945747365 >> 172.21.15.69:0/2945747365 conn(0x7fde540d0160 msgr2=0x7fde540d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:30.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.588+0000 7fde59000640 1 -- 172.21.15.69:0/2945747365 shutdown_connections 2024-04-24T08:15:30.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.589+0000 7fde59000640 1 -- 172.21.15.69:0/2945747365 wait complete. 2024-04-24T08:15:30.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.590+0000 7fde59000640 1 Processor -- start 2024-04-24T08:15:30.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.590+0000 7fde59000640 1 -- start start 2024-04-24T08:15:30.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.591+0000 7fde59000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde540723a0 0x7fde5414b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.591+0000 7fde59000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde5414c650 0x7fde5409b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.591+0000 7fde59000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 0x7fde540cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.591+0000 7fde59000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde5414ca30 con 0x7fde540723a0 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.591+0000 7fde59000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fde5406ecb0 con 0x7fde5414c650 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.591+0000 7fde59000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fde54072830 con 0x7fde541666e0 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.591+0000 7fde53400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde540723a0 0x7fde5414b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.591+0000 7fde53400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde540723a0 0x7fde5414b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33102/0 (socket says 172.21.15.69:33102) 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.591+0000 7fde53400640 1 -- 172.21.15.69:0/697469407 learned_addr learned my addr 172.21.15.69:0/697469407 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde4aa00640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde5414c650 0x7fde5409b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde53400640 1 -- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 msgr2=0x7fde540cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde53e00640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 0x7fde540cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde53400640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 0x7fde540cc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde53400640 1 -- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde5414c650 msgr2=0x7fde5409b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde53400640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde5414c650 0x7fde5409b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde53400640 1 -- 172.21.15.69:0/697469407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde5409a2f0 con 0x7fde540723a0 2024-04-24T08:15:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde53400640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde540723a0 0x7fde5414b5b0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fde4400c520 tx=0x7fde4400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde4be00640 1 -- 172.21.15.69:0/697469407 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde44017070 con 0x7fde540723a0 2024-04-24T08:15:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde4be00640 1 -- 172.21.15.69:0/697469407 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde440052d0 con 0x7fde540723a0 2024-04-24T08:15:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde4be00640 1 -- 172.21.15.69:0/697469407 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde44012870 con 0x7fde540723a0 2024-04-24T08:15:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde59000640 1 -- 172.21.15.69:0/697469407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde5409a590 con 0x7fde540723a0 2024-04-24T08:15:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.592+0000 7fde59000640 1 -- 172.21.15.69:0/697469407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde5409ab50 con 0x7fde540723a0 2024-04-24T08:15:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.594+0000 7fde4be00640 1 -- 172.21.15.69:0/697469407 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fde4401a070 con 0x7fde540723a0 2024-04-24T08:15:30.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.595+0000 7fde4be00640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fde3006c8f0 0x7fde3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.596+0000 7fde4be00640 1 -- 172.21.15.69:0/697469407 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fde4408af50 con 0x7fde540723a0 2024-04-24T08:15:30.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.596+0000 7fde5d384740 1 Processor -- start 2024-04-24T08:15:30.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.596+0000 7fde4aa00640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fde3006c8f0 0x7fde3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.596+0000 7fde5d384740 1 -- start start 2024-04-24T08:15:30.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde5d384740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b0f9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde5d384740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b128e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde5d384740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129360 0x55c58b0fe170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde5d384740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c58b063b10 con 0x55c58b129aa0 2024-04-24T08:15:30.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde5d384740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c58b00a2b0 con 0x55c58b146980 2024-04-24T08:15:30.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde5d384740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c58b05fad0 con 0x55c58b129360 2024-04-24T08:15:30.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde4aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b128e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde4aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b128e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33110/0 (socket says 172.21.15.69:33110) 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde4aa00640 1 -- 172.21.15.69:0/2067565740 learned_addr learned my addr 172.21.15.69:0/2067565740 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde4aa00640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fde3006c8f0 0x7fde3006edb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fde3c034000 tx=0x7fde3c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.597+0000 7fde53400640 1 --2- 172.21.15.69:0/2067565740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b0f9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.598+0000 7fde53e00640 1 --2- 172.21.15.69:0/2067565740 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129360 0x55c58b0fe170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.598+0000 7fde4aa00640 1 -- 172.21.15.69:0/2067565740 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129360 msgr2=0x55c58b0fe170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.598+0000 7fde4aa00640 1 --2- 172.21.15.69:0/2067565740 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129360 0x55c58b0fe170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.598+0000 7fde4aa00640 1 -- 172.21.15.69:0/2067565740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 msgr2=0x55c58b0f9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.598+0000 7fde4aa00640 1 --2- 172.21.15.69:0/2067565740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b0f9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.598+0000 7fde4aa00640 1 -- 172.21.15.69:0/2067565740 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c58ad762d0 con 0x55c58b129aa0 2024-04-24T08:15:30.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.598+0000 7fde4aa00640 1 --2- 172.21.15.69:0/2067565740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b128e20 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0098c0 tx=0x7fde3c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248d8d17001e3671 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:30.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.598+0000 7fde43e00640 1 -- 172.21.15.69:0/2067565740 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c04c070 con 0x55c58b129aa0 2024-04-24T08:15:30.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.598+0000 7fde43e00640 1 -- 172.21.15.69:0/2067565740 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde3c03c460 con 0x55c58b129aa0 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.598+0000 7fde43e00640 1 -- 172.21.15.69:0/2067565740 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c037040 con 0x55c58b129aa0 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.599+0000 7fde5d384740 1 -- 172.21.15.69:0/2067565740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 msgr2=0x55c58b128e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.599+0000 7fde5d384740 1 --2- 172.21.15.69:0/2067565740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b128e20 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fde3c0098c0 tx=0x7fde3c008030 comp rx=0 tx=0).stop 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.599+0000 7fde5d384740 1 -- 172.21.15.69:0/2067565740 shutdown_connections 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.599+0000 7fde5d384740 1 --2- 172.21.15.69:0/2067565740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b128e20 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.599+0000 7fde5d384740 1 --2- 172.21.15.69:0/2067565740 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129360 0x55c58b0fe170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.599+0000 7fde5d384740 1 --2- 172.21.15.69:0/2067565740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b0f9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.599+0000 7fde5d384740 1 -- 172.21.15.69:0/2067565740 >> 172.21.15.69:0/2067565740 conn(0x55c58b0fb580 msgr2=0x55c58b143e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.599+0000 7fde5d384740 1 -- 172.21.15.69:0/2067565740 shutdown_connections 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.599+0000 7fde5d384740 1 -- 172.21.15.69:0/2067565740 wait complete. 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.600+0000 7fde5d384740 1 -- start start 2024-04-24T08:15:30.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.600+0000 7fde5d384740 1 Processor -- start 2024-04-24T08:15:30.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.601+0000 7fde5d384740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129aa0 0x55c58b12a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.601+0000 7fde5d384740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b0fec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.601+0000 7fde5d384740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b1283d0 0x55c58b0ff150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.601+0000 7fde5d384740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c58b0fd730 con 0x55c58b1283d0 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.601+0000 7fde5d384740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c58b01d590 con 0x55c58b146980 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.601+0000 7fde5d384740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c58b0fcb20 con 0x55c58b129aa0 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.601+0000 7fde5d384740 10 client.0 _ll_register_callbacks cb 0x55c58b0003a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.601+0000 7fde53e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b1283d0 0x55c58b0ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.601+0000 7fde53e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b1283d0 0x55c58b0ff150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33126/0 (socket says 172.21.15.69:33126) 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.601+0000 7fde53e00640 1 -- 172.21.15.69:0/1333376155 learned_addr learned my addr 172.21.15.69:0/1333376155 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde53400640 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129aa0 0x55c58b12a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde53e00640 1 -- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129aa0 msgr2=0x55c58b12a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde53e00640 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129aa0 0x55c58b12a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde4aa00640 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b0fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde53e00640 1 -- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 msgr2=0x55c58b0fec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde53e00640 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b0fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde53e00640 1 -- 172.21.15.69:0/1333376155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c58ae406b0 con 0x55c58b1283d0 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde53400640 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129aa0 0x55c58b12a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde53e00640 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b1283d0 0x55c58b0ff150 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fde4c0079e0 tx=0x7fde4c007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ced2e897b740f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde40c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde40c00640 1 -- 172.21.15.69:0/1333376155 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde4c01a020 con 0x55c58b1283d0 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde40c00640 1 -- 172.21.15.69:0/1333376155 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fde4c0118b0 con 0x55c58b1283d0 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde5d384740 1 -- 172.21.15.69:0/1333376155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde3c03c460 con 0x55c58b1283d0 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.602+0000 7fde40c00640 1 -- 172.21.15.69:0/1333376155 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fde4c024050 con 0x55c58b1283d0 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.603+0000 7fde5d384740 10 client.7716 fetch_fsmap learned FSMap version 61 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.603+0000 7fde5d384740 1 -- 172.21.15.69:0/1333376155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) -- 0x55c58b12bb90 con 0x55c58b1283d0 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.603+0000 7fde40c00640 1 -- 172.21.15.69:0/1333376155 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde3c03c460 con 0x55c58b1283d0 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.603+0000 7fde40c00640 1 -- 172.21.15.69:0/1333376155 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 61) ==== 3494+0+0 (secure 0 0 0) 0x7fde4c027020 con 0x55c58b1283d0 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.603+0000 7fde5d384740 10 client.7716 fetch_fsmap finished waiting for FSMap version 61 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.603+0000 7fde5d384740 10 client.7716 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:15:30.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.603+0000 7fde5d384740 20 client.7716 populate_metadata read hostname 'smithi069' 2024-04-24T08:15:30.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.603+0000 7fde5d384740 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:6835/1179683245,v1:172.21.15.121:6837/1179683245] conn(0x55c58b127690 0x55c58b154780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.603+0000 7fde5d384740 4 client.7716 mds_command: new command op to 7593 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:30.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.603+0000 7fde5d384740 1 -- 172.21.15.69:0/1333376155 --> [v2:172.21.15.121:6835/1179683245,v1:172.21.15.121:6837/1179683245] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c58b071530 con 0x55c58b127690 2024-04-24T08:15:30.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.604+0000 7fde53400640 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:6835/1179683245,v1:172.21.15.121:6837/1179683245] conn(0x55c58b127690 0x55c58b154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.604+0000 7fde53400640 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:6835/1179683245,v1:172.21.15.121:6837/1179683245] conn(0x55c58b127690 0x55c58b154780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2ca674ec1631504 server_cookie=bb5f3ebae6a4a071 in_seq=0 out_seq=0 2024-04-24T08:15:30.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.604+0000 7fde40c00640 10 client.7716 ms_handle_connect on v2:172.21.15.121:6835/1179683245 2024-04-24T08:15:30.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.606+0000 7fde40c00640 1 -- 172.21.15.69:0/1333376155 <== mds.0 v2:172.21.15.121:6835/1179683245 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c58b071530 con 0x55c58b127690 2024-04-24T08:15:30.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.606+0000 7fde40c00640 10 client.7716 handle_command_reply: tid=0 2024-04-24T08:15:30.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.606+0000 7fde5d384740 1 client.7716 shutdown 2024-04-24T08:15:30.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 1 -- 172.21.15.69:0/1333376155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b1283d0 msgr2=0x55c58b0ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b1283d0 0x55c58b0ff150 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fde4c0079e0 tx=0x7fde4c007ea0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 1 -- 172.21.15.69:0/1333376155 shutdown_connections 2024-04-24T08:15:30.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:6835/1179683245,v1:172.21.15.121:6837/1179683245] conn(0x55c58b127690 0x55c58b154780 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b1283d0 0x55c58b0ff150 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b0fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde40c00640 0 client.7716 ms_handle_reset on v2:172.21.15.121:6835/1179683245 2024-04-24T08:15:30.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 1 --2- 172.21.15.69:0/1333376155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b129aa0 0x55c58b12a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 1 -- 172.21.15.69:0/1333376155 >> 172.21.15.69:0/1333376155 conn(0x55c58b129360 msgr2=0x55c58b0f8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:30.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 1 -- 172.21.15.69:0/1333376155 shutdown_connections 2024-04-24T08:15:30.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 1 -- 172.21.15.69:0/1333376155 wait complete. 2024-04-24T08:15:30.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.607+0000 7fde5d384740 20 client.7716 trim_cache size 0 max 16384 2024-04-24T08:15:30.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.626+0000 7fde5d384740 1 Processor -- start 2024-04-24T08:15:30.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.626+0000 7fde5d384740 1 -- start start 2024-04-24T08:15:30.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.627+0000 7fde5d384740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b1283d0 0x55c58b156410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.627+0000 7fde5d384740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b0f9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.627+0000 7fde5d384740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b146980 0x55c58b167bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.627+0000 7fde5d384740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fde541fca70 con 0x55c58b129aa0 2024-04-24T08:15:30.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.627+0000 7fde5d384740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fde541fbdd0 con 0x55c58b1283d0 2024-04-24T08:15:30.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.627+0000 7fde5d384740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fde541fc420 con 0x55c58b146980 2024-04-24T08:15:30.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.627+0000 7fde4aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b0f9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.627+0000 7fde4aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b0f9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33132/0 (socket says 172.21.15.69:33132) 2024-04-24T08:15:30.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.627+0000 7fde4aa00640 1 -- 172.21.15.69:0/3591274777 learned_addr learned my addr 172.21.15.69:0/3591274777 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:30.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde53e00640 1 --2- 172.21.15.69:0/3591274777 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b146980 0x55c58b167bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde4aa00640 1 -- 172.21.15.69:0/3591274777 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b146980 msgr2=0x55c58b167bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:30.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde53400640 1 --2- 172.21.15.69:0/3591274777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b1283d0 0x55c58b156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde4aa00640 1 --2- 172.21.15.69:0/3591274777 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b146980 0x55c58b167bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde4aa00640 1 -- 172.21.15.69:0/3591274777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b1283d0 msgr2=0x55c58b156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde4aa00640 1 --2- 172.21.15.69:0/3591274777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b1283d0 0x55c58b156410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde4aa00640 1 -- 172.21.15.69:0/3591274777 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c58b1446d0 con 0x55c58b129aa0 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde4aa00640 1 --2- 172.21.15.69:0/3591274777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b0f9760 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fde3c02a6f0 tx=0x7fde3c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb058ae525e61d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde43400640 1 -- 172.21.15.69:0/3591274777 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c04c040 con 0x55c58b129aa0 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde43400640 1 -- 172.21.15.69:0/3591274777 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde3c0042c0 con 0x55c58b129aa0 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.628+0000 7fde43400640 1 -- 172.21.15.69:0/3591274777 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c037040 con 0x55c58b129aa0 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.629+0000 7fde5d384740 1 -- 172.21.15.69:0/3591274777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 msgr2=0x55c58b0f9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.629+0000 7fde5d384740 1 --2- 172.21.15.69:0/3591274777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b0f9760 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fde3c02a6f0 tx=0x7fde3c02c000 comp rx=0 tx=0).stop 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.629+0000 7fde5d384740 1 -- 172.21.15.69:0/3591274777 shutdown_connections 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.629+0000 7fde5d384740 1 --2- 172.21.15.69:0/3591274777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b0f9760 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.629+0000 7fde5d384740 1 --2- 172.21.15.69:0/3591274777 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b146980 0x55c58b167bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.629+0000 7fde5d384740 1 --2- 172.21.15.69:0/3591274777 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b1283d0 0x55c58b156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.629+0000 7fde5d384740 1 -- 172.21.15.69:0/3591274777 >> 172.21.15.69:0/3591274777 conn(0x55c58b129360 msgr2=0x55c58b143f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:30.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.629+0000 7fde5d384740 1 -- 172.21.15.69:0/3591274777 shutdown_connections 2024-04-24T08:15:30.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.629+0000 7fde5d384740 1 -- 172.21.15.69:0/3591274777 wait complete. 2024-04-24T08:15:30.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.630+0000 7fde5d384740 1 -- start start 2024-04-24T08:15:30.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.630+0000 7fde5d384740 1 Processor -- start 2024-04-24T08:15:30.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde5d384740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b1283d0 0x55c58b1c26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde5d384740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b1be7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde5d384740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b129f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde5d384740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c58b127510 con 0x55c58b129aa0 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde5d384740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c58b154e30 con 0x55c58b146980 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde5d384740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c58b127b60 con 0x55c58b1283d0 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde5d384740 10 client.0 _ll_register_callbacks cb 0x7fde4c007600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde4aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b1be7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde4aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b1be7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33136/0 (socket says 172.21.15.69:33136) 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde4aa00640 1 -- 172.21.15.69:0/1405147003 learned_addr learned my addr 172.21.15.69:0/1405147003 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde4aa00640 1 -- 172.21.15.69:0/1405147003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b1283d0 msgr2=0x55c58b1c26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde4aa00640 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b1283d0 0x55c58b1c26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde4aa00640 1 -- 172.21.15.69:0/1405147003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 msgr2=0x55c58b129f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde4aa00640 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b129f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.631+0000 7fde4aa00640 1 -- 172.21.15.69:0/1405147003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c58b145900 con 0x55c58b129aa0 2024-04-24T08:15:30.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde4aa00640 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b1be7a0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fde3c02a140 tx=0x7fde3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d99d221a53283 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde40c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde40c00640 1 -- 172.21.15.69:0/1405147003 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c037040 con 0x55c58b129aa0 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde40c00640 1 -- 172.21.15.69:0/1405147003 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fde3c004470 con 0x55c58b129aa0 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde5d384740 1 -- 172.21.15.69:0/1405147003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde3c0042c0 con 0x55c58b129aa0 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde40c00640 1 -- 172.21.15.69:0/1405147003 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=61) ==== 24+0+0 (secure 0 0 0) 0x7fde3c05d050 con 0x55c58b129aa0 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde5d384740 10 client.7722 fetch_fsmap learned FSMap version 61 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde5d384740 1 -- 172.21.15.69:0/1405147003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=61,monmap=2+}) -- 0x55c58b145be0 con 0x55c58b129aa0 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde40c00640 1 -- 172.21.15.69:0/1405147003 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde3c0042c0 con 0x55c58b129aa0 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde40c00640 1 -- 172.21.15.69:0/1405147003 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 61) ==== 3494+0+0 (secure 0 0 0) 0x7fde3c060020 con 0x55c58b129aa0 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde5d384740 10 client.7722 fetch_fsmap finished waiting for FSMap version 61 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde5d384740 10 client.7722 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde5d384740 20 client.7722 populate_metadata read hostname 'smithi069' 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde5d384740 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.121:6835/1179683245,v1:172.21.15.121:6837/1179683245] conn(0x55c58b12a750 0x55c58b166660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde5d384740 4 client.7722 mds_command: new command op to 7593 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:15:30.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.632+0000 7fde5d384740 1 -- 172.21.15.69:0/1405147003 --> [v2:172.21.15.121:6835/1179683245,v1:172.21.15.121:6837/1179683245] -- command(tid 0: {"prefix": "status"}) -- 0x55c58b0fa9d0 con 0x55c58b12a750 2024-04-24T08:15:30.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.633+0000 7fde53400640 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.121:6835/1179683245,v1:172.21.15.121:6837/1179683245] conn(0x55c58b12a750 0x55c58b166660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.634+0000 7fde53400640 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.121:6835/1179683245,v1:172.21.15.121:6837/1179683245] conn(0x55c58b12a750 0x55c58b166660 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb2a95f7451081c1 server_cookie=de9ac6638c45f9bc in_seq=0 out_seq=0 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.634+0000 7fde40c00640 10 client.7722 ms_handle_connect on v2:172.21.15.121:6835/1179683245 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde40c00640 1 -- 172.21.15.69:0/1405147003 <== mds.0 v2:172.21.15.121:6835/1179683245 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55c58b0fa9d0 con 0x55c58b12a750 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde40c00640 10 client.7722 handle_command_reply: tid=0 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 client.7722 shutdown 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 -- 172.21.15.69:0/1405147003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 msgr2=0x55c58b1be7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b1be7a0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fde3c02a140 tx=0x7fde3c004730 comp rx=0 tx=0).stop 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 -- 172.21.15.69:0/1405147003 shutdown_connections 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.121:6835/1179683245,v1:172.21.15.121:6837/1179683245] conn(0x55c58b12a750 0x55c58b166660 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c58b129aa0 0x55c58b1be7a0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c58b146980 0x55c58b129f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde40c00640 0 client.7722 ms_handle_reset on v2:172.21.15.121:6835/1179683245 2024-04-24T08:15:30.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 --2- 172.21.15.69:0/1405147003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c58b1283d0 0x55c58b1c26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 -- 172.21.15.69:0/1405147003 >> 172.21.15.69:0/1405147003 conn(0x55c58b129360 msgr2=0x55c58b146250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:30.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 -- 172.21.15.69:0/1405147003 shutdown_connections 2024-04-24T08:15:30.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 1 -- 172.21.15.69:0/1405147003 wait complete. 2024-04-24T08:15:30.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.635+0000 7fde5d384740 20 client.7722 trim_cache size 0 max 16384 2024-04-24T08:15:30.638 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:15:30.638 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:15:30.638 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:15:30.638 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 7593, 2024-04-24T08:15:30.639 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:15:30.639 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:15:30.639 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:15:30.639 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 6.6508598750000001, 2024-04-24T08:15:30.639 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 61, 2024-04-24T08:15:30.639 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 82, 2024-04-24T08:15:30.639 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 82, 2024-04-24T08:15:30.639 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 12.09780089 2024-04-24T08:15:30.639 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 -- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fde3006c8f0 msgr2=0x7fde3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fde3006c8f0 0x7fde3006edb0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fde3c034000 tx=0x7fde3c02e000 comp rx=0 tx=0).stop 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 -- 172.21.15.69:0/697469407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde540723a0 msgr2=0x7fde5414b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde540723a0 0x7fde5414b5b0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fde4400c520 tx=0x7fde4400c9e0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 -- 172.21.15.69:0/697469407 shutdown_connections 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fde3006c8f0 0x7fde3006edb0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fde541666e0 0x7fde540cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde5414c650 0x7fde5409b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 --2- 172.21.15.69:0/697469407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fde540723a0 0x7fde5414b5b0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 -- 172.21.15.69:0/697469407 >> 172.21.15.69:0/697469407 conn(0x7fde540d0160 msgr2=0x7fde54166bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:30.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 -- 172.21.15.69:0/697469407 shutdown_connections 2024-04-24T08:15:30.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.636+0000 7fde59000640 1 -- 172.21.15.69:0/697469407 wait complete. 2024-04-24T08:15:30.654 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:15:30.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.844+0000 7fdb83e00640 1 Processor -- start 2024-04-24T08:15:30.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.844+0000 7fdb83e00640 1 -- start start 2024-04-24T08:15:30.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.845+0000 7fdb83e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdb8415d1d0 0x7fdb840ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.845+0000 7fdb83e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb840b30f0 0x7fdb840ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.845+0000 7fdb83e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb8415d660 con 0x7fdb84156c60 2024-04-24T08:15:30.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.845+0000 7fdb83e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fdb8406dc30 con 0x7fdb8415d1d0 2024-04-24T08:15:30.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.845+0000 7fdb83e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdb8406df10 con 0x7fdb840b30f0 2024-04-24T08:15:30.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.846+0000 7fdb82000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdb8415d1d0 0x7fdb840ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.846+0000 7fdb82000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdb8415d1d0 0x7fdb840ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33150/0 (socket says 172.21.15.69:33150) 2024-04-24T08:15:30.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.846+0000 7fdb82000640 1 -- 172.21.15.69:0/3979189884 learned_addr learned my addr 172.21.15.69:0/3979189884 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:30.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.846+0000 7fdb83400640 1 --2- 172.21.15.69:0/3979189884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb840b30f0 0x7fdb840ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.846+0000 7fdb82000640 1 -- 172.21.15.69:0/3979189884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb840b30f0 msgr2=0x7fdb840ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:30.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.846+0000 7fdb82000640 1 --2- 172.21.15.69:0/3979189884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb840b30f0 0x7fdb840ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.846+0000 7fdb82000640 1 -- 172.21.15.69:0/3979189884 >> v1:172.21.15.121:6789/0 conn(0x7fdb84156c60 legacy=0x7fdb8415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.846+0000 7fdb82000640 1 -- 172.21.15.69:0/3979189884 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb8406e1f0 con 0x7fdb8415d1d0 2024-04-24T08:15:30.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.847+0000 7fdb82000640 1 --2- 172.21.15.69:0/3979189884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb8415d1d0 0x7fdb840ad0d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009870 tx=0x7fdb7002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ecc73aca652139 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:30.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.848+0000 7fdb81600640 1 -- 172.21.15.69:0/3979189884 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb70037070 con 0x7fdb8415d1d0 2024-04-24T08:15:30.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.848+0000 7fdb81600640 1 -- 172.21.15.69:0/3979189884 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb70032440 con 0x7fdb8415d1d0 2024-04-24T08:15:30.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.848+0000 7fdb81600640 1 -- 172.21.15.69:0/3979189884 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb70032810 con 0x7fdb8415d1d0 2024-04-24T08:15:30.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.848+0000 7fdb83e00640 1 -- 172.21.15.69:0/3979189884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb8415d1d0 msgr2=0x7fdb840ad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.848+0000 7fdb83e00640 1 --2- 172.21.15.69:0/3979189884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb8415d1d0 0x7fdb840ad0d0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009870 tx=0x7fdb7002a250 comp rx=0 tx=0).stop 2024-04-24T08:15:30.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.848+0000 7fdb83e00640 1 -- 172.21.15.69:0/3979189884 shutdown_connections 2024-04-24T08:15:30.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.848+0000 7fdb83e00640 1 --2- 172.21.15.69:0/3979189884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb840b30f0 0x7fdb840ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.848+0000 7fdb83e00640 1 --2- 172.21.15.69:0/3979189884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb8415d1d0 0x7fdb840ad0d0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.848+0000 7fdb83e00640 1 -- 172.21.15.69:0/3979189884 >> 172.21.15.69:0/3979189884 conn(0x7fdb840d0160 msgr2=0x7fdb840d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:30.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.848+0000 7fdb83e00640 1 -- 172.21.15.69:0/3979189884 shutdown_connections 2024-04-24T08:15:30.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.849+0000 7fdb83e00640 1 -- 172.21.15.69:0/3979189884 wait complete. 2024-04-24T08:15:30.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.850+0000 7fdb83e00640 1 Processor -- start 2024-04-24T08:15:30.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.850+0000 7fdb83e00640 1 -- start start 2024-04-24T08:15:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.851+0000 7fdb83e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb840b30f0 0x7fdb840c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.851+0000 7fdb83e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb84156c60 0x7fdb8415e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.851+0000 7fdb83e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb841e7660 0x7fdb84145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.851+0000 7fdb83e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8406e710 con 0x7fdb840b30f0 2024-04-24T08:15:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.851+0000 7fdb83e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8415d940 con 0x7fdb84156c60 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.851+0000 7fdb83e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdb8406e890 con 0x7fdb841e7660 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb82a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb840b30f0 0x7fdb840c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb82a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb840b30f0 0x7fdb840c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33152/0 (socket says 172.21.15.69:33152) 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb82a00640 1 -- 172.21.15.69:0/660937556 learned_addr learned my addr 172.21.15.69:0/660937556 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb82000640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb84156c60 0x7fdb8415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb83400640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb841e7660 0x7fdb84145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb82a00640 1 -- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb841e7660 msgr2=0x7fdb84145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb82a00640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb841e7660 0x7fdb84145370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb82a00640 1 -- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb84156c60 msgr2=0x7fdb8415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb82a00640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb84156c60 0x7fdb8415e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:30.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb82a00640 1 -- 172.21.15.69:0/660937556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb840ca7d0 con 0x7fdb840b30f0 2024-04-24T08:15:30.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.852+0000 7fdb82a00640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb840b30f0 0x7fdb840c9e80 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400c430 tx=0x7fdb7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:30.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.853+0000 7fdb7aa00640 1 -- 172.21.15.69:0/660937556 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb740052d0 con 0x7fdb840b30f0 2024-04-24T08:15:30.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.853+0000 7fdb7aa00640 1 -- 172.21.15.69:0/660937556 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb74005490 con 0x7fdb840b30f0 2024-04-24T08:15:30.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.853+0000 7fdb7aa00640 1 -- 172.21.15.69:0/660937556 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb74007260 con 0x7fdb840b30f0 2024-04-24T08:15:30.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.853+0000 7fdb83e00640 1 -- 172.21.15.69:0/660937556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb840caab0 con 0x7fdb840b30f0 2024-04-24T08:15:30.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.853+0000 7fdb83e00640 1 -- 172.21.15.69:0/660937556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb840cb070 con 0x7fdb840b30f0 2024-04-24T08:15:30.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.854+0000 7fdb83e00640 1 -- 172.21.15.69:0/660937556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb84075d50 con 0x7fdb840b30f0 2024-04-24T08:15:30.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.855+0000 7fdb7aa00640 1 -- 172.21.15.69:0/660937556 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb74007400 con 0x7fdb840b30f0 2024-04-24T08:15:30.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.857+0000 7fdb7aa00640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb6406c8f0 0x7fdb6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:30.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.857+0000 7fdb7aa00640 1 -- 172.21.15.69:0/660937556 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fdb7408b150 con 0x7fdb840b30f0 2024-04-24T08:15:30.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.857+0000 7fdb82000640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb6406c8f0 0x7fdb6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:30.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.858+0000 7fdb82000640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb6406c8f0 0x7fdb6406edb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdb70034000 tx=0x7fdb7002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:30.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:30.862+0000 7fdb7aa00640 1 -- 172.21.15.69:0/660937556 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb740594e0 con 0x7fdb840b30f0 2024-04-24T08:15:31.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.110+0000 7fdb83e00640 1 -- 172.21.15.69:0/660937556 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdb8406f9e0 con 0x7fdb840b30f0 2024-04-24T08:15:31.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.111+0000 7fdb7aa00640 1 -- 172.21.15.69:0/660937556 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v61) ==== 53+0+73 (secure 0 0 0) 0x7fdb74058de0 con 0x7fdb840b30f0 2024-04-24T08:15:31.111 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 -- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb6406c8f0 msgr2=0x7fdb6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb6406c8f0 0x7fdb6406edb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdb70034000 tx=0x7fdb7002e000 comp rx=0 tx=0).stop 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 -- 172.21.15.69:0/660937556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb840b30f0 msgr2=0x7fdb840c9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb840b30f0 0x7fdb840c9e80 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400c430 tx=0x7fdb7400c8f0 comp rx=0 tx=0).stop 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 -- 172.21.15.69:0/660937556 shutdown_connections 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb6406c8f0 0x7fdb6406edb0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb841e7660 0x7fdb84145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb84156c60 0x7fdb8415e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 --2- 172.21.15.69:0/660937556 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb840b30f0 0x7fdb840c9e80 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 -- 172.21.15.69:0/660937556 >> 172.21.15.69:0/660937556 conn(0x7fdb840d0160 msgr2=0x7fdb840b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 -- 172.21.15.69:0/660937556 shutdown_connections 2024-04-24T08:15:31.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31.114+0000 7fdb83e00640 1 -- 172.21.15.69:0/660937556 wait complete. 2024-04-24T08:15:31.134 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:15:31.134 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:15:31.134 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:15:31.134 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:15:31.134 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:31.134 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:15:31.134 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:15:31.134 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:15:31.134 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:15:31.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:31.135 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:15:31.186 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:15:31.186 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:15:31.186 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:15:31.186 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:31.186 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:15:31.186 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:31.186 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42249sec preferred_lft 42249sec 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:31.187 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:15:31.188 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:31.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:31.189 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:15:31.265 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:15:31.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:31.269 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:15:31.322 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:15:31.322 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:15:31.323 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:15:31.323 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:15:31.323 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:15:31.323 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:15:31.323 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:15:31.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:15:31.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:15:31.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:15:31.323 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:15:31.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:15:31.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:15:31.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:31.528 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:15:31.576 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:15:31.576 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:15:31.577 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:15:31.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:31.577 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:15:31.705 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:15:31.705 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:15:31.705 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:31.754 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:15:31.754 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:15:31.755 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:31.804 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:15:31.886 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:15:31.941 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:15:31.941 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:15:31.941 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:15:31.941 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:15:31.941 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:15:31.941 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:15:31.941 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:15:31.942 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:31.943 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:31.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:31.944 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:15:32.000 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:15:32.000 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:15:32.005 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:15:32.075 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:15:32.116 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:15:32.116 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:15:32.116 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:15:32.116 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:15:32.116 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:15:32.116 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:15:32.116 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:15:32.117 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:32.118 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:32.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:32.119 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:15:32.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:15:32.174+0000 7fe24280d480 -1 init, newargv = 0x56349f7164e0 newargc=15 2024-04-24T08:15:32.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[63328]: starting ceph client 2024-04-24T08:15:32.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[63328]: starting fuse 2024-04-24T08:15:32.202 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:15:32.202 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:15:32.202 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:15:32.202 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:15:32.202 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:15:32.202 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:15:32.203 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:15:32.204 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:15:32.204 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:15:32.204 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:15:32.204 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:32.204 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:15:32.204 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:15:32.387 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.63328.asok 2024-04-24T08:15:32.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:15:32.403 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.63328.asok 2024-04-24T08:15:32.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:32.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.63328.asok status 2024-04-24T08:15:32.629 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:15:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:15:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:15:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "63328", 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 7731, 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:15:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 7731 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2511136021 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2511136021 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.7731 192.168.0.1:0/2511136021", 2024-04-24T08:15:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2511136021", 2024-04-24T08:15:32.632 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:15:32.632 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 61, 2024-04-24T08:15:32.632 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 82, 2024-04-24T08:15:32.632 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:15:32.632 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:15:32.632 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:15:32.632 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:15:32.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:32.644 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:15:32.644 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:32.696 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:15:32.696 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:32.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:32.696 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:32.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:32.773 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir) 2024-04-24T08:15:32.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:32.845 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir/file) 2024-04-24T08:15:32.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:32.902 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-04-24T08:15:32.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:32.956 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir/file testdir/hardlink) 2024-04-24T08:15:33.027 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:15:33.027 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:15:33.027 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:15:33.027 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:33.027 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir").st_ino) 2024-04-24T08:15:33.027 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:15:33.170 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627776 2024-04-24T08:15:33.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:15:33.183 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:15:33.183 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:15:33.279 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:33.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.484+0000 7fb6daa00640 1 Processor -- start 2024-04-24T08:15:33.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.484+0000 7fb6daa00640 1 -- start start 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.486+0000 7fb6daa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb6d40d0300 0x7fb6d406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.486+0000 7fb6daa00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb6d4145a90 0x7fb6d40d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.486+0000 7fb6daa00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d40d0c20 con 0x7fb6d4145fe0 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.486+0000 7fb6daa00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb6d40cf3d0 con 0x7fb6d40d0300 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.486+0000 7fb6daa00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb6d40cf6b0 con 0x7fb6d4145a90 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.486+0000 7fb6d3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb6d40d0300 0x7fb6d406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.486+0000 7fb6d3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb6d40d0300 0x7fb6d406e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33166/0 (socket says 172.21.15.69:33166) 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.486+0000 7fb6d8a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb6d4145fe0 0x7fb6d4160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42992/0 (socket says 172.21.15.69:42992) 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.486+0000 7fb6d3e00640 1 -- 172.21.15.69:0/1272213061 learned_addr learned my addr 172.21.15.69:0/1272213061 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.486+0000 7fb6cb400640 1 --2- 172.21.15.69:0/1272213061 >> v2:172.21.15.121:3300/0 conn(0x7fb6d4145a90 0x7fb6d40d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.487+0000 7fb6d3e00640 1 -- 172.21.15.69:0/1272213061 >> v2:172.21.15.121:3300/0 conn(0x7fb6d4145a90 msgr2=0x7fb6d40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.487+0000 7fb6d3e00640 1 --2- 172.21.15.69:0/1272213061 >> v2:172.21.15.121:3300/0 conn(0x7fb6d4145a90 0x7fb6d40d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.487+0000 7fb6d3e00640 1 -- 172.21.15.69:0/1272213061 >> v1:172.21.15.69:6789/0 conn(0x7fb6d4145fe0 legacy=0x7fb6d4160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.487+0000 7fb6d3e00640 1 -- 172.21.15.69:0/1272213061 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d40cf990 con 0x7fb6d40d0300 2024-04-24T08:15:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.487+0000 7fb6d3400640 1 -- 172.21.15.69:0/1272213061 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1067034706 0 0) 0x7fb6d40d0c20 con 0x7fb6d4145fe0 2024-04-24T08:15:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.487+0000 7fb6cb400640 1 --2- 172.21.15.69:0/1272213061 >> v2:172.21.15.121:3300/0 conn(0x7fb6d4145a90 0x7fb6d40d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.488+0000 7fb6d3e00640 1 --2- 172.21.15.69:0/1272213061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d40d0300 0x7fb6d406e240 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fb6d40b5780 tx=0x7fb6c402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54bc07c4b655eed server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.488+0000 7fb6d3400640 1 -- 172.21.15.69:0/1272213061 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4037070 con 0x7fb6d40d0300 2024-04-24T08:15:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.488+0000 7fb6d3400640 1 -- 172.21.15.69:0/1272213061 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6c4032440 con 0x7fb6d40d0300 2024-04-24T08:15:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.488+0000 7fb6d3400640 1 -- 172.21.15.69:0/1272213061 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c4032810 con 0x7fb6d40d0300 2024-04-24T08:15:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.488+0000 7fb6daa00640 1 -- 172.21.15.69:0/1272213061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d40d0300 msgr2=0x7fb6d406e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.488+0000 7fb6daa00640 1 --2- 172.21.15.69:0/1272213061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d40d0300 0x7fb6d406e240 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fb6d40b5780 tx=0x7fb6c402a270 comp rx=0 tx=0).stop 2024-04-24T08:15:33.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.489+0000 7fb6daa00640 1 -- 172.21.15.69:0/1272213061 shutdown_connections 2024-04-24T08:15:33.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.489+0000 7fb6daa00640 1 --2- 172.21.15.69:0/1272213061 >> v2:172.21.15.121:3300/0 conn(0x7fb6d4145a90 0x7fb6d40d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.489+0000 7fb6daa00640 1 --2- 172.21.15.69:0/1272213061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d40d0300 0x7fb6d406e240 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.489+0000 7fb6daa00640 1 -- 172.21.15.69:0/1272213061 >> 172.21.15.69:0/1272213061 conn(0x7fb6d4067d90 msgr2=0x7fb6d406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:33.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.489+0000 7fb6daa00640 1 -- 172.21.15.69:0/1272213061 shutdown_connections 2024-04-24T08:15:33.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.489+0000 7fb6daa00640 1 -- 172.21.15.69:0/1272213061 wait complete. 2024-04-24T08:15:33.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.490+0000 7fb6daa00640 1 Processor -- start 2024-04-24T08:15:33.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.491+0000 7fb6daa00640 1 -- start start 2024-04-24T08:15:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.491+0000 7fb6daa00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6d40d0300 0x7fb6d413ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.491+0000 7fb6daa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6d4145a90 0x7fb6d415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.491+0000 7fb6daa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d4145fe0 0x7fb6d40c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.491+0000 7fb6daa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d4145770 con 0x7fb6d4145fe0 2024-04-24T08:15:33.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.491+0000 7fb6daa00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d40cff50 con 0x7fb6d40d0300 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.491+0000 7fb6daa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d41458f0 con 0x7fb6d4145a90 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.492+0000 7fb6d8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d4145fe0 0x7fb6d40c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.492+0000 7fb6d8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d4145fe0 0x7fb6d40c7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33168/0 (socket says 172.21.15.69:33168) 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.492+0000 7fb6d8a00640 1 -- 172.21.15.69:0/2843273792 learned_addr learned my addr 172.21.15.69:0/2843273792 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.492+0000 7fb6d8a00640 1 -- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6d4145a90 msgr2=0x7fb6d415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.492+0000 7fb6d8a00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6d4145a90 0x7fb6d415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.492+0000 7fb6d8a00640 1 -- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6d40d0300 msgr2=0x7fb6d413ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.492+0000 7fb6d8a00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6d40d0300 0x7fb6d413ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.492+0000 7fb6d8a00640 1 -- 172.21.15.69:0/2843273792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d41404d0 con 0x7fb6d4145fe0 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.492+0000 7fb6d8a00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d4145fe0 0x7fb6d40c7f80 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00c7d0 tx=0x7fb6cc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.493+0000 7fb6d0c00640 1 -- 172.21.15.69:0/2843273792 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc018070 con 0x7fb6d4145fe0 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.493+0000 7fb6d0c00640 1 -- 172.21.15.69:0/2843273792 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6cc00a640 con 0x7fb6d4145fe0 2024-04-24T08:15:33.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.493+0000 7fb6daa00640 1 -- 172.21.15.69:0/2843273792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d415f270 con 0x7fb6d4145fe0 2024-04-24T08:15:33.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.493+0000 7fb6d0c00640 1 -- 172.21.15.69:0/2843273792 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc00aa70 con 0x7fb6d4145fe0 2024-04-24T08:15:33.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.493+0000 7fb6daa00640 1 -- 172.21.15.69:0/2843273792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d40c57b0 con 0x7fb6d4145fe0 2024-04-24T08:15:33.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.494+0000 7fb6daa00640 1 -- 172.21.15.69:0/2843273792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d415f790 con 0x7fb6d4145fe0 2024-04-24T08:15:33.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.495+0000 7fb6d0c00640 1 -- 172.21.15.69:0/2843273792 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb6cc01b070 con 0x7fb6d4145fe0 2024-04-24T08:15:33.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.496+0000 7fb6d0c00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb6b006c8f0 0x7fb6b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.496+0000 7fb6d0c00640 1 -- 172.21.15.69:0/2843273792 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fb6cc08b1c0 con 0x7fb6d4145fe0 2024-04-24T08:15:33.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.497+0000 7fb6d3e00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb6b006c8f0 0x7fb6b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:33.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.498+0000 7fb6d3e00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb6b006c8f0 0x7fb6b006edb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4034000 tx=0x7fb6c402c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:33.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.501+0000 7fb6d0c00640 1 -- 172.21.15.69:0/2843273792 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6d415f790 con 0x7fb6d4145fe0 2024-04-24T08:15:33.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.706+0000 7fb6daa00640 1 -- 172.21.15.69:0/2843273792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6d4068a80 con 0x7fb6d4145fe0 2024-04-24T08:15:33.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.709+0000 7fb6d0c00640 1 -- 172.21.15.69:0/2843273792 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+16636 (secure 0 0 0) 0x7fb6cc0595e0 con 0x7fb6d4145fe0 2024-04-24T08:15:33.709 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:33.710 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":82,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:15:24.982737+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:33.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 -- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb6b006c8f0 msgr2=0x7fb6b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb6b006c8f0 0x7fb6b006edb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4034000 tx=0x7fb6c402c000 comp rx=0 tx=0).stop 2024-04-24T08:15:33.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 -- 172.21.15.69:0/2843273792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d4145fe0 msgr2=0x7fb6d40c7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d4145fe0 0x7fb6d40c7f80 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc00c7d0 tx=0x7fb6cc00cc90 comp rx=0 tx=0).stop 2024-04-24T08:15:33.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 -- 172.21.15.69:0/2843273792 shutdown_connections 2024-04-24T08:15:33.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb6b006c8f0 0x7fb6b006edb0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb6d4145fe0 0x7fb6d40c7f80 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6d4145a90 0x7fb6d415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 --2- 172.21.15.69:0/2843273792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6d40d0300 0x7fb6d413ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 -- 172.21.15.69:0/2843273792 >> 172.21.15.69:0/2843273792 conn(0x7fb6d4067d90 msgr2=0x7fb6d40ccdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:33.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 -- 172.21.15.69:0/2843273792 shutdown_connections 2024-04-24T08:15:33.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.711+0000 7fb6daa00640 1 -- 172.21.15.69:0/2843273792 wait complete. 2024-04-24T08:15:33.732 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:15:21.126081+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:15:22.402310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:15:33.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:15:33.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.928+0000 7fbccfe00640 1 Processor -- start 2024-04-24T08:15:33.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.928+0000 7fbccfe00640 1 -- start start 2024-04-24T08:15:33.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.929+0000 7fbccfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbcd00b5300 0x7fbcd00d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.930+0000 7fbccfe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd00d0c40 con 0x7fbcd00d0820 2024-04-24T08:15:33.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.930+0000 7fbccfe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd00ce640 con 0x7fbcd015ea60 2024-04-24T08:15:33.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.930+0000 7fbccfe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbcd00ce910 con 0x7fbcd00b5300 2024-04-24T08:15:33.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.930+0000 7fbcc6000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbcd00b5300 0x7fbcd00d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:33.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.930+0000 7fbccf400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbcd00d0820 0x7fbcd006df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42996/0 (socket says 172.21.15.69:42996) 2024-04-24T08:15:33.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.930+0000 7fbccf400640 1 -- 172.21.15.69:0/2813368998 learned_addr learned my addr 172.21.15.69:0/2813368998 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:33.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.930+0000 7fbcce000640 1 -- 172.21.15.69:0/2813368998 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2794466119 0 0) 0x7fbcd00d0c40 con 0x7fbcd00d0820 2024-04-24T08:15:33.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.930+0000 7fbcce000640 1 -- 172.21.15.69:0/2813368998 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcb4003520 con 0x7fbcd00d0820 2024-04-24T08:15:33.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.931+0000 7fbcc6000640 1 -- 172.21.15.69:0/2813368998 >> v1:172.21.15.121:6789/0 conn(0x7fbcd015ea60 legacy=0x7fbcd00cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.931+0000 7fbcc6000640 1 -- 172.21.15.69:0/2813368998 >> v1:172.21.15.69:6789/0 conn(0x7fbcd00d0820 legacy=0x7fbcd006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.931+0000 7fbcc6000640 1 -- 172.21.15.69:0/2813368998 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd00cebf0 con 0x7fbcd00b5300 2024-04-24T08:15:33.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.932+0000 7fbcc6000640 1 --2- 172.21.15.69:0/2813368998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 0x7fbcd00d02e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80097a0 tx=0x7fbcb802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9cb63e16943a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:33.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.932+0000 7fbcce000640 1 -- 172.21.15.69:0/2813368998 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8037070 con 0x7fbcd00b5300 2024-04-24T08:15:33.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.932+0000 7fbcce000640 1 -- 172.21.15.69:0/2813368998 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcb8032440 con 0x7fbcd00b5300 2024-04-24T08:15:33.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.932+0000 7fbcce000640 1 -- 172.21.15.69:0/2813368998 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb8032810 con 0x7fbcd00b5300 2024-04-24T08:15:33.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.933+0000 7fbccfe00640 1 -- 172.21.15.69:0/2813368998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 msgr2=0x7fbcd00d02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.933+0000 7fbccfe00640 1 --2- 172.21.15.69:0/2813368998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 0x7fbcd00d02e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80097a0 tx=0x7fbcb802a230 comp rx=0 tx=0).stop 2024-04-24T08:15:33.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.933+0000 7fbccfe00640 1 -- 172.21.15.69:0/2813368998 shutdown_connections 2024-04-24T08:15:33.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.933+0000 7fbccfe00640 1 --2- 172.21.15.69:0/2813368998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 0x7fbcd00d02e0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.933+0000 7fbccfe00640 1 -- 172.21.15.69:0/2813368998 >> 172.21.15.69:0/2813368998 conn(0x7fbcd0067b80 msgr2=0x7fbcd006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:33.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.933+0000 7fbccfe00640 1 -- 172.21.15.69:0/2813368998 shutdown_connections 2024-04-24T08:15:33.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.933+0000 7fbccfe00640 1 -- 172.21.15.69:0/2813368998 wait complete. 2024-04-24T08:15:33.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.935+0000 7fbccfe00640 1 Processor -- start 2024-04-24T08:15:33.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.935+0000 7fbccfe00640 1 -- start start 2024-04-24T08:15:33.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 0x7fbcd01f80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccfe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd00d0820 0x7fbcd014dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd015ea60 0x7fbcd00b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccfe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd00cf490 con 0x7fbcd00b5300 2024-04-24T08:15:33.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccfe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd00cf1b0 con 0x7fbcd00d0820 2024-04-24T08:15:33.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccfe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd006e650 con 0x7fbcd015ea60 2024-04-24T08:15:33.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 0x7fbcd01f80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:33.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 0x7fbcd01f80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33194/0 (socket says 172.21.15.69:33194) 2024-04-24T08:15:33.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccea00640 1 -- 172.21.15.69:0/3599391223 learned_addr learned my addr 172.21.15.69:0/3599391223 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:33.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccf400640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd015ea60 0x7fbcd00b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:33.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbcc6000640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd00d0820 0x7fbcd014dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:33.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccea00640 1 -- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd015ea60 msgr2=0x7fbcd00b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccea00640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd015ea60 0x7fbcd00b3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccea00640 1 -- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd00d0820 msgr2=0x7fbcd014dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:33.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccea00640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd00d0820 0x7fbcd014dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:33.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.936+0000 7fbccea00640 1 -- 172.21.15.69:0/3599391223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd01f8620 con 0x7fbcd00b5300 2024-04-24T08:15:33.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.937+0000 7fbccea00640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 0x7fbcd01f80e0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000c730 tx=0x7fbcc000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:33.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.937+0000 7fbcc7400640 1 -- 172.21.15.69:0/3599391223 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc0003b70 con 0x7fbcd00b5300 2024-04-24T08:15:33.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.937+0000 7fbcc7400640 1 -- 172.21.15.69:0/3599391223 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc0003d30 con 0x7fbcd00b5300 2024-04-24T08:15:33.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.937+0000 7fbccfe00640 1 -- 172.21.15.69:0/3599391223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd00b4220 con 0x7fbcd00b5300 2024-04-24T08:15:33.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.937+0000 7fbcc7400640 1 -- 172.21.15.69:0/3599391223 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc000aa90 con 0x7fbcd00b5300 2024-04-24T08:15:33.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.937+0000 7fbccfe00640 1 -- 172.21.15.69:0/3599391223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd0154c00 con 0x7fbcd00b5300 2024-04-24T08:15:33.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.939+0000 7fbccfe00640 1 -- 172.21.15.69:0/3599391223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd0075d50 con 0x7fbcd00b5300 2024-04-24T08:15:33.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.940+0000 7fbcc7400640 1 -- 172.21.15.69:0/3599391223 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbcc000ac30 con 0x7fbcd00b5300 2024-04-24T08:15:33.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.941+0000 7fbcc7400640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbcac06c8f0 0x7fbcac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:33.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.941+0000 7fbcc7400640 1 -- 172.21.15.69:0/3599391223 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fbcc008ba90 con 0x7fbcd00b5300 2024-04-24T08:15:33.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.941+0000 7fbcc6000640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbcac06c8f0 0x7fbcac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:33.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.942+0000 7fbcc6000640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbcac06c8f0 0x7fbcac06edb0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8034000 tx=0x7fbcb802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:33.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:33.945+0000 7fbcc7400640 1 -- 172.21.15.69:0/3599391223 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbcc0059eb0 con 0x7fbcd00b5300 2024-04-24T08:15:34.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.149+0000 7fbccfe00640 1 -- 172.21.15.69:0/3599391223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbcd006bcc0 con 0x7fbcd00b5300 2024-04-24T08:15:34.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.150+0000 7fbcc7400640 1 -- 172.21.15.69:0/3599391223 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7fbcc0060080 con 0x7fbcd00b5300 2024-04-24T08:15:34.150 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:15:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.152+0000 7fbccfe00640 1 -- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbcac06c8f0 msgr2=0x7fbcac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.152+0000 7fbccfe00640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbcac06c8f0 0x7fbcac06edb0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbcb8034000 tx=0x7fbcb802e000 comp rx=0 tx=0).stop 2024-04-24T08:15:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.152+0000 7fbccfe00640 1 -- 172.21.15.69:0/3599391223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 msgr2=0x7fbcd01f80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.152+0000 7fbccfe00640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 0x7fbcd01f80e0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fbcc000c730 tx=0x7fbcc000cbf0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.153+0000 7fbccfe00640 1 -- 172.21.15.69:0/3599391223 shutdown_connections 2024-04-24T08:15:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.153+0000 7fbccfe00640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbcac06c8f0 0x7fbcac06edb0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.153+0000 7fbccfe00640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbcd015ea60 0x7fbcd00b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.153+0000 7fbccfe00640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbcd00d0820 0x7fbcd014dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.153+0000 7fbccfe00640 1 --2- 172.21.15.69:0/3599391223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbcd00b5300 0x7fbcd01f80e0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.153+0000 7fbccfe00640 1 -- 172.21.15.69:0/3599391223 >> 172.21.15.69:0/3599391223 conn(0x7fbcd0067b80 msgr2=0x7fbcd0158930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.153+0000 7fbccfe00640 1 -- 172.21.15.69:0/3599391223 shutdown_connections 2024-04-24T08:15:34.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.153+0000 7fbccfe00640 1 -- 172.21.15.69:0/3599391223 wait complete. 2024-04-24T08:15:34.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:15:34.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.375+0000 7fe819200640 1 Processor -- start 2024-04-24T08:15:34.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.376+0000 7fe819200640 1 -- start start 2024-04-24T08:15:34.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.377+0000 7fe819200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe81415c820 0x7fe8140b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.377+0000 7fe819200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 0x7fe81406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.377+0000 7fe819200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe81415cc00 con 0x7fe81406fd10 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.377+0000 7fe819200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe81406ed30 con 0x7fe81415c820 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.377+0000 7fe819200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe8140b4e10 con 0x7fe8140b46a0 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.377+0000 7fe811e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe81415c820 0x7fe8140b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.377+0000 7fe811e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe81415c820 0x7fe8140b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33202/0 (socket says 172.21.15.69:33202) 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.377+0000 7fe811e00640 1 -- 172.21.15.69:0/1128724509 learned_addr learned my addr 172.21.15.69:0/1128724509 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.378+0000 7fe809400640 1 --2- 172.21.15.69:0/1128724509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 0x7fe81406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.378+0000 7fe811e00640 1 -- 172.21.15.69:0/1128724509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 msgr2=0x7fe81406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.378+0000 7fe811e00640 1 --2- 172.21.15.69:0/1128724509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 0x7fe81406f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.378+0000 7fe811e00640 1 -- 172.21.15.69:0/1128724509 >> v1:172.21.15.69:6789/0 conn(0x7fe81406fd10 legacy=0x7fe8140ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.378+0000 7fe811e00640 1 -- 172.21.15.69:0/1128724509 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8140b50f0 con 0x7fe81415c820 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.378+0000 7fe811400640 1 -- 172.21.15.69:0/1128724509 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804293525 0 0) 0x7fe81415cc00 con 0x7fe81406fd10 2024-04-24T08:15:34.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.378+0000 7fe809400640 1 --2- 172.21.15.69:0/1128724509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 0x7fe81406f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:34.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.379+0000 7fe811e00640 1 --2- 172.21.15.69:0/1128724509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81415c820 0x7fe8140b3e90 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fe804009870 tx=0x7fe80402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef93265b8d69b29 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.379+0000 7fe811400640 1 -- 172.21.15.69:0/1128724509 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804037070 con 0x7fe81415c820 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.379+0000 7fe811400640 1 -- 172.21.15.69:0/1128724509 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe804032440 con 0x7fe81415c820 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.379+0000 7fe811400640 1 -- 172.21.15.69:0/1128724509 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804032810 con 0x7fe81415c820 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.380+0000 7fe819200640 1 -- 172.21.15.69:0/1128724509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81415c820 msgr2=0x7fe8140b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.380+0000 7fe819200640 1 --2- 172.21.15.69:0/1128724509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81415c820 0x7fe8140b3e90 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7fe804009870 tx=0x7fe80402a250 comp rx=0 tx=0).stop 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.380+0000 7fe819200640 1 -- 172.21.15.69:0/1128724509 shutdown_connections 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.380+0000 7fe819200640 1 --2- 172.21.15.69:0/1128724509 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 0x7fe81406f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.380+0000 7fe819200640 1 --2- 172.21.15.69:0/1128724509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81415c820 0x7fe8140b3e90 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.380+0000 7fe819200640 1 -- 172.21.15.69:0/1128724509 >> 172.21.15.69:0/1128724509 conn(0x7fe8140d0350 msgr2=0x7fe8140d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.380+0000 7fe819200640 1 -- 172.21.15.69:0/1128724509 shutdown_connections 2024-04-24T08:15:34.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.380+0000 7fe819200640 1 -- 172.21.15.69:0/1128724509 wait complete. 2024-04-24T08:15:34.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.382+0000 7fe819200640 1 Processor -- start 2024-04-24T08:15:34.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.382+0000 7fe819200640 1 -- start start 2024-04-24T08:15:34.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.383+0000 7fe819200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81406fd10 0x7fe814159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.383+0000 7fe819200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 0x7fe81415de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.383+0000 7fe819200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81415c820 0x7fe8141601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.383+0000 7fe819200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe8140b4b00 con 0x7fe81406fd10 2024-04-24T08:15:34.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.383+0000 7fe819200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe81406f2b0 con 0x7fe81415c820 2024-04-24T08:15:34.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.383+0000 7fe819200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe8140ce960 con 0x7fe8140b46a0 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.383+0000 7fe811e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81406fd10 0x7fe814159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.383+0000 7fe811e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81406fd10 0x7fe814159860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33208/0 (socket says 172.21.15.69:33208) 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.383+0000 7fe811e00640 1 -- 172.21.15.69:0/4253557677 learned_addr learned my addr 172.21.15.69:0/4253557677 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe811e00640 1 -- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 msgr2=0x7fe81415de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe809400640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 0x7fe81415de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe812800640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81415c820 0x7fe8141601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe811e00640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 0x7fe81415de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe811e00640 1 -- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81415c820 msgr2=0x7fe8141601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe811e00640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81415c820 0x7fe8141601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe811e00640 1 -- 172.21.15.69:0/4253557677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81415e910 con 0x7fe81406fd10 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe812800640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81415c820 0x7fe8141601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe809400640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 0x7fe81415de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe811e00640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81406fd10 0x7fe814159860 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fe804009870 tx=0x7fe8040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe80aa00640 1 -- 172.21.15.69:0/4253557677 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe804037040 con 0x7fe81406fd10 2024-04-24T08:15:34.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe80aa00640 1 -- 172.21.15.69:0/4253557677 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe804004250 con 0x7fe81406fd10 2024-04-24T08:15:34.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.384+0000 7fe80aa00640 1 -- 172.21.15.69:0/4253557677 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80403c670 con 0x7fe81406fd10 2024-04-24T08:15:34.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.385+0000 7fe819200640 1 -- 172.21.15.69:0/4253557677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81415ebf0 con 0x7fe81406fd10 2024-04-24T08:15:34.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.385+0000 7fe819200640 1 -- 172.21.15.69:0/4253557677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81415f1b0 con 0x7fe81406fd10 2024-04-24T08:15:34.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.386+0000 7fe819200640 1 -- 172.21.15.69:0/4253557677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8140b3110 con 0x7fe81406fd10 2024-04-24T08:15:34.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.387+0000 7fe80aa00640 1 -- 172.21.15.69:0/4253557677 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe80403a070 con 0x7fe81406fd10 2024-04-24T08:15:34.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.388+0000 7fe80aa00640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7f006c8f0 0x7fe7f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.388+0000 7fe80aa00640 1 -- 172.21.15.69:0/4253557677 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fe8040ac4e0 con 0x7fe81406fd10 2024-04-24T08:15:34.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.389+0000 7fe809400640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7f006c8f0 0x7fe7f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:34.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.390+0000 7fe809400640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7f006c8f0 0x7fe7f006edb0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc006fd0 tx=0x7fe7fc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:34.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.392+0000 7fe80aa00640 1 -- 172.21.15.69:0/4253557677 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8140b3110 con 0x7fe81406fd10 2024-04-24T08:15:34.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.591+0000 7fe819200640 1 -- 172.21.15.69:0/4253557677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe814068a80 con 0x7fe81406fd10 2024-04-24T08:15:34.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.592+0000 7fe80aa00640 1 -- 172.21.15.69:0/4253557677 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) ==== 104+0+11 (secure 0 0 0) 0x7fe804041020 con 0x7fe81406fd10 2024-04-24T08:15:34.593 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:15:34.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.594+0000 7fe819200640 1 -- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7f006c8f0 msgr2=0x7fe7f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.594+0000 7fe819200640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7f006c8f0 0x7fe7f006edb0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc006fd0 tx=0x7fe7fc008040 comp rx=0 tx=0).stop 2024-04-24T08:15:34.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.594+0000 7fe819200640 1 -- 172.21.15.69:0/4253557677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81406fd10 msgr2=0x7fe814159860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.594+0000 7fe819200640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81406fd10 0x7fe814159860 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fe804009870 tx=0x7fe8040046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.595+0000 7fe819200640 1 -- 172.21.15.69:0/4253557677 shutdown_connections 2024-04-24T08:15:34.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.595+0000 7fe819200640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe7f006c8f0 0x7fe7f006edb0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.595+0000 7fe819200640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe81415c820 0x7fe8141601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.595+0000 7fe819200640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe8140b46a0 0x7fe81415de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.595+0000 7fe819200640 1 --2- 172.21.15.69:0/4253557677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe81406fd10 0x7fe814159860 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.595+0000 7fe819200640 1 -- 172.21.15.69:0/4253557677 >> 172.21.15.69:0/4253557677 conn(0x7fe8140d0350 msgr2=0x7fe8140b0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:34.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.595+0000 7fe819200640 1 -- 172.21.15.69:0/4253557677 shutdown_connections 2024-04-24T08:15:34.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.595+0000 7fe819200640 1 -- 172.21.15.69:0/4253557677 wait complete. 2024-04-24T08:15:34.616 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:15:34.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.817+0000 7fd403800640 1 Processor -- start 2024-04-24T08:15:34.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.817+0000 7fd403800640 1 -- start start 2024-04-24T08:15:34.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.819+0000 7fd403800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd3fc06f140 0x7fd3fc06f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.819+0000 7fd403800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd3fc0cfaf0 0x7fd3fc15eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.819+0000 7fd403800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3fc06fa60 con 0x7fd3fc15f0e0 2024-04-24T08:15:34.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.819+0000 7fd403800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd3fc15e710 con 0x7fd3fc06f140 2024-04-24T08:15:34.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.819+0000 7fd403800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd3fc0b4a80 con 0x7fd3fc0cfaf0 2024-04-24T08:15:34.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.819+0000 7fd400e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd3fc06f140 0x7fd3fc06f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:34.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.819+0000 7fd401800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd3fc15f0e0 0x7fd3fc0b3260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43018/0 (socket says 172.21.15.69:43018) 2024-04-24T08:15:34.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.819+0000 7fd401800640 1 -- 172.21.15.69:0/1958621683 learned_addr learned my addr 172.21.15.69:0/1958621683 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:34.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.820+0000 7fd3f3e00640 1 --2- 172.21.15.69:0/1958621683 >> v2:172.21.15.121:3300/0 conn(0x7fd3fc0cfaf0 0x7fd3fc15eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:34.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.820+0000 7fd400e00640 1 -- 172.21.15.69:0/1958621683 >> v2:172.21.15.121:3300/0 conn(0x7fd3fc0cfaf0 msgr2=0x7fd3fc15eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:34.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.820+0000 7fd3fbe00640 1 -- 172.21.15.69:0/1958621683 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3819246322 0 0) 0x7fd3fc06fa60 con 0x7fd3fc15f0e0 2024-04-24T08:15:34.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.820+0000 7fd400e00640 1 --2- 172.21.15.69:0/1958621683 >> v2:172.21.15.121:3300/0 conn(0x7fd3fc0cfaf0 0x7fd3fc15eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.820+0000 7fd400e00640 1 -- 172.21.15.69:0/1958621683 >> v1:172.21.15.69:6789/0 conn(0x7fd3fc15f0e0 legacy=0x7fd3fc0b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.820+0000 7fd400e00640 1 -- 172.21.15.69:0/1958621683 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3fc0b4d60 con 0x7fd3fc06f140 2024-04-24T08:15:34.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.822+0000 7fd400e00640 1 --2- 172.21.15.69:0/1958621683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 0x7fd3fc06f520 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0afa40 tx=0x7fd3ec02b2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b12590a36e10a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:34.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.822+0000 7fd3fbe00640 1 -- 172.21.15.69:0/1958621683 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec038070 con 0x7fd3fc06f140 2024-04-24T08:15:34.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.822+0000 7fd3fbe00640 1 -- 172.21.15.69:0/1958621683 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3ec033440 con 0x7fd3fc06f140 2024-04-24T08:15:34.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.822+0000 7fd3fbe00640 1 -- 172.21.15.69:0/1958621683 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec033810 con 0x7fd3fc06f140 2024-04-24T08:15:34.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.823+0000 7fd403800640 1 -- 172.21.15.69:0/1958621683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 msgr2=0x7fd3fc06f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.823+0000 7fd403800640 1 --2- 172.21.15.69:0/1958621683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 0x7fd3fc06f520 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0afa40 tx=0x7fd3ec02b2b0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.823+0000 7fd403800640 1 -- 172.21.15.69:0/1958621683 shutdown_connections 2024-04-24T08:15:34.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.823+0000 7fd403800640 1 --2- 172.21.15.69:0/1958621683 >> v2:172.21.15.121:3300/0 conn(0x7fd3fc0cfaf0 0x7fd3fc15eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.823+0000 7fd403800640 1 --2- 172.21.15.69:0/1958621683 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 0x7fd3fc06f520 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.823+0000 7fd403800640 1 -- 172.21.15.69:0/1958621683 >> 172.21.15.69:0/1958621683 conn(0x7fd3fc0d0350 msgr2=0x7fd3fc0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:34.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.823+0000 7fd403800640 1 -- 172.21.15.69:0/1958621683 shutdown_connections 2024-04-24T08:15:34.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.823+0000 7fd403800640 1 -- 172.21.15.69:0/1958621683 wait complete. 2024-04-24T08:15:34.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.825+0000 7fd403800640 1 Processor -- start 2024-04-24T08:15:34.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.826+0000 7fd403800640 1 -- start start 2024-04-24T08:15:34.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd403800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 0x7fd3fc15a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd403800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3fc0cfaf0 0x7fd3fc0ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd403800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3fc15f0e0 0x7fd3fc0ad780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd403800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc0b4820 con 0x7fd3fc06f140 2024-04-24T08:15:34.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd403800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd3fc0b46a0 con 0x7fd3fc15f0e0 2024-04-24T08:15:34.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd403800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd3fc0cd6c0 con 0x7fd3fc0cfaf0 2024-04-24T08:15:34.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd400e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 0x7fd3fc15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:34.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd400e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 0x7fd3fc15a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33220/0 (socket says 172.21.15.69:33220) 2024-04-24T08:15:34.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd400e00640 1 -- 172.21.15.69:0/2808396009 learned_addr learned my addr 172.21.15.69:0/2808396009 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:34.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd400e00640 1 -- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3fc0cfaf0 msgr2=0x7fd3fc0ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd400e00640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3fc0cfaf0 0x7fd3fc0ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd400e00640 1 -- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3fc15f0e0 msgr2=0x7fd3fc0ad780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:34.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd400e00640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3fc15f0e0 0x7fd3fc0ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:34.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.827+0000 7fd400e00640 1 -- 172.21.15.69:0/2808396009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3fc0c99d0 con 0x7fd3fc06f140 2024-04-24T08:15:34.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.828+0000 7fd400e00640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 0x7fd3fc15a970 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009890 tx=0x7fd3ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:34.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.828+0000 7fd3f9600640 1 -- 172.21.15.69:0/2808396009 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec038040 con 0x7fd3fc06f140 2024-04-24T08:15:34.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.828+0000 7fd3f9600640 1 -- 172.21.15.69:0/2808396009 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3ec031770 con 0x7fd3fc06f140 2024-04-24T08:15:34.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.828+0000 7fd3f9600640 1 -- 172.21.15.69:0/2808396009 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ec031b60 con 0x7fd3fc06f140 2024-04-24T08:15:34.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.828+0000 7fd403800640 1 -- 172.21.15.69:0/2808396009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3fc0c9cb0 con 0x7fd3fc06f140 2024-04-24T08:15:34.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.828+0000 7fd403800640 1 -- 172.21.15.69:0/2808396009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3fc0ca270 con 0x7fd3fc06f140 2024-04-24T08:15:34.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.830+0000 7fd403800640 1 -- 172.21.15.69:0/2808396009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3fc068a80 con 0x7fd3fc06f140 2024-04-24T08:15:34.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.830+0000 7fd3f9600640 1 -- 172.21.15.69:0/2808396009 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd3ec031d00 con 0x7fd3fc06f140 2024-04-24T08:15:34.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.831+0000 7fd3f9600640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd3d806c8f0 0x7fd3d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:34.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.832+0000 7fd3f9600640 1 -- 172.21.15.69:0/2808396009 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fd3ec0ac400 con 0x7fd3fc06f140 2024-04-24T08:15:34.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.832+0000 7fd3f3e00640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd3d806c8f0 0x7fd3d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:34.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.833+0000 7fd3f3e00640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd3d806c8f0 0x7fd3d806edb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0ccae0 tx=0x7fd3e4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:34.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:34.836+0000 7fd3f9600640 1 -- 172.21.15.69:0/2808396009 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd3ec033e20 con 0x7fd3fc06f140 2024-04-24T08:15:35.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.035+0000 7fd403800640 1 -- 172.21.15.69:0/2808396009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd3fc15cd50 con 0x7fd3fc06f140 2024-04-24T08:15:35.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.036+0000 7fd3f9600640 1 -- 172.21.15.69:0/2808396009 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) ==== 100+0+11 (secure 0 0 0) 0x7fd3ec0323a0 con 0x7fd3fc06f140 2024-04-24T08:15:35.038 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:15:35.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.039+0000 7fd403800640 1 -- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd3d806c8f0 msgr2=0x7fd3d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.039+0000 7fd403800640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd3d806c8f0 0x7fd3d806edb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc0ccae0 tx=0x7fd3e4008040 comp rx=0 tx=0).stop 2024-04-24T08:15:35.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.039+0000 7fd403800640 1 -- 172.21.15.69:0/2808396009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 msgr2=0x7fd3fc15a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.039+0000 7fd403800640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 0x7fd3fc15a970 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec009890 tx=0x7fd3ec0046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.039+0000 7fd403800640 1 -- 172.21.15.69:0/2808396009 shutdown_connections 2024-04-24T08:15:35.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.039+0000 7fd403800640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd3d806c8f0 0x7fd3d806edb0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.039+0000 7fd403800640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3fc15f0e0 0x7fd3fc0ad780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.039+0000 7fd403800640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd3fc0cfaf0 0x7fd3fc0ad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.039+0000 7fd403800640 1 --2- 172.21.15.69:0/2808396009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd3fc06f140 0x7fd3fc15a970 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.039+0000 7fd403800640 1 -- 172.21.15.69:0/2808396009 >> 172.21.15.69:0/2808396009 conn(0x7fd3fc0d0350 msgr2=0x7fd3fc15b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:35.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.040+0000 7fd403800640 1 -- 172.21.15.69:0/2808396009 shutdown_connections 2024-04-24T08:15:35.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.040+0000 7fd403800640 1 -- 172.21.15.69:0/2808396009 wait complete. 2024-04-24T08:15:35.061 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:15:35.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.253+0000 7f30e1000640 1 Processor -- start 2024-04-24T08:15:35.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.253+0000 7f30e1000640 1 -- start start 2024-04-24T08:15:35.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.255+0000 7f30e1000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc145a90 0x7f30dc06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.255+0000 7f30e1000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f30dc145fe0 0x7f30dc160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.255+0000 7f30e1000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30dc0cf3d0 con 0x7f30dc0d0300 2024-04-24T08:15:35.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.255+0000 7f30e1000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f30dc0cf6b0 con 0x7f30dc145fe0 2024-04-24T08:15:35.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.255+0000 7f30e1000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f30dc0cf990 con 0x7f30dc145a90 2024-04-24T08:15:35.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.255+0000 7f30da800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f30dc145fe0 0x7f30dc160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.255+0000 7f30d9400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc145a90 0x7f30dc06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.256+0000 7f30d9400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc145a90 0x7f30dc06e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:51960/0 (socket says 172.21.15.69:51960) 2024-04-24T08:15:35.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.256+0000 7f30d9400640 1 -- 172.21.15.69:0/1090785144 learned_addr learned my addr 172.21.15.69:0/1090785144 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:35.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.256+0000 7f30da800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f30dc145fe0 0x7f30dc160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:46884/0 (socket says 172.21.15.69:46884) 2024-04-24T08:15:35.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.256+0000 7f30d9e00640 1 --1- 172.21.15.69:0/1090785144 >> v1:172.21.15.121:6789/0 conn(0x7f30dc0d0300 0x7f30dc0d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:48220/0 (socket says 172.21.15.69:48220) 2024-04-24T08:15:35.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.256+0000 7f30d9400640 1 -- 172.21.15.69:0/1090785144 >> v2:172.21.15.121:3300/0 conn(0x7f30dc145fe0 msgr2=0x7f30dc160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:35.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.256+0000 7f30d9400640 1 --2- 172.21.15.69:0/1090785144 >> v2:172.21.15.121:3300/0 conn(0x7f30dc145fe0 0x7f30dc160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.256+0000 7f30d9400640 1 -- 172.21.15.69:0/1090785144 >> v1:172.21.15.121:6789/0 conn(0x7f30dc0d0300 legacy=0x7f30dc0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.256+0000 7f30d9400640 1 -- 172.21.15.69:0/1090785144 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30dc0cfc70 con 0x7f30dc145a90 2024-04-24T08:15:35.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.257+0000 7f30d9400640 1 --2- 172.21.15.69:0/1090785144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc145a90 0x7f30dc06e240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009870 tx=0x7f30c802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c660231989c38dd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:35.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.258+0000 7f30d8a00640 1 -- 172.21.15.69:0/1090785144 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8037070 con 0x7f30dc145a90 2024-04-24T08:15:35.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.258+0000 7f30d8a00640 1 -- 172.21.15.69:0/1090785144 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c8032440 con 0x7f30dc145a90 2024-04-24T08:15:35.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.258+0000 7f30d8a00640 1 -- 172.21.15.69:0/1090785144 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8032810 con 0x7f30dc145a90 2024-04-24T08:15:35.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.259+0000 7f30e1000640 1 -- 172.21.15.69:0/1090785144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc145a90 msgr2=0x7f30dc06e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.259+0000 7f30e1000640 1 --2- 172.21.15.69:0/1090785144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc145a90 0x7f30dc06e240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009870 tx=0x7f30c802a250 comp rx=0 tx=0).stop 2024-04-24T08:15:35.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.259+0000 7f30e1000640 1 -- 172.21.15.69:0/1090785144 shutdown_connections 2024-04-24T08:15:35.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.259+0000 7f30e1000640 1 --2- 172.21.15.69:0/1090785144 >> v2:172.21.15.121:3300/0 conn(0x7f30dc145fe0 0x7f30dc160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.259+0000 7f30e1000640 1 --2- 172.21.15.69:0/1090785144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc145a90 0x7f30dc06e240 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.259+0000 7f30e1000640 1 -- 172.21.15.69:0/1090785144 >> 172.21.15.69:0/1090785144 conn(0x7f30dc067d90 msgr2=0x7f30dc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:35.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.259+0000 7f30e1000640 1 -- 172.21.15.69:0/1090785144 shutdown_connections 2024-04-24T08:15:35.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.259+0000 7f30e1000640 1 -- 172.21.15.69:0/1090785144 wait complete. 2024-04-24T08:15:35.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.260+0000 7f30e1000640 1 Processor -- start 2024-04-24T08:15:35.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.261+0000 7f30e1000640 1 -- start start 2024-04-24T08:15:35.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.261+0000 7f30e1000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc0d0300 0x7f30dc15ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.261+0000 7f30e1000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30dc145a90 0x7f30dc15f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.261+0000 7f30e1000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30dc145fe0 0x7f30dc14a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.261+0000 7f30e1000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc1458f0 con 0x7f30dc145a90 2024-04-24T08:15:35.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.261+0000 7f30e1000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f30dc145770 con 0x7f30dc145fe0 2024-04-24T08:15:35.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.261+0000 7f30e1000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f30dc0cdc30 con 0x7f30dc0d0300 2024-04-24T08:15:35.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30d9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30dc145a90 0x7f30dc15f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30d9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30dc145a90 0x7f30dc15f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33234/0 (socket says 172.21.15.69:33234) 2024-04-24T08:15:35.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30d9400640 1 -- 172.21.15.69:0/3434258440 learned_addr learned my addr 172.21.15.69:0/3434258440 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:35.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30d9e00640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc0d0300 0x7f30dc15ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30da800640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30dc145fe0 0x7f30dc14a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30d9400640 1 -- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc0d0300 msgr2=0x7f30dc15ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30d9400640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc0d0300 0x7f30dc15ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30d9400640 1 -- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30dc145fe0 msgr2=0x7f30dc14a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30d9400640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30dc145fe0 0x7f30dc14a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30d9400640 1 -- 172.21.15.69:0/3434258440 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30dc142810 con 0x7f30dc145a90 2024-04-24T08:15:35.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.262+0000 7f30da800640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30dc145fe0 0x7f30dc14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:35.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.263+0000 7f30d9400640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30dc145a90 0x7f30dc15f740 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f30c8000c00 tx=0x7f30c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:35.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.263+0000 7f30d2000640 1 -- 172.21.15.69:0/3434258440 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8037040 con 0x7f30dc145a90 2024-04-24T08:15:35.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.263+0000 7f30d2000640 1 -- 172.21.15.69:0/3434258440 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c8042070 con 0x7f30dc145a90 2024-04-24T08:15:35.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.263+0000 7f30d2000640 1 -- 172.21.15.69:0/3434258440 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c8030880 con 0x7f30dc145a90 2024-04-24T08:15:35.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.263+0000 7f30e1000640 1 -- 172.21.15.69:0/3434258440 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30dc142af0 con 0x7f30dc145a90 2024-04-24T08:15:35.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.263+0000 7f30e1000640 1 -- 172.21.15.69:0/3434258440 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30dc1430b0 con 0x7f30dc145a90 2024-04-24T08:15:35.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.264+0000 7f30e1000640 1 -- 172.21.15.69:0/3434258440 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30dc068a80 con 0x7f30dc145a90 2024-04-24T08:15:35.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.265+0000 7f30d2000640 1 -- 172.21.15.69:0/3434258440 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f30c8030a20 con 0x7f30dc145a90 2024-04-24T08:15:35.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.266+0000 7f30d2000640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f30bc06c8f0 0x7f30bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.266+0000 7f30d2000640 1 -- 172.21.15.69:0/3434258440 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f30c80ac4d0 con 0x7f30dc145a90 2024-04-24T08:15:35.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.267+0000 7f30d9e00640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f30bc06c8f0 0x7f30bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.268+0000 7f30d9e00640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f30bc06c8f0 0x7f30bc06edb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f30dc142180 tx=0x7f30cc007ce0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:35.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.271+0000 7f30d2000640 1 -- 172.21.15.69:0/3434258440 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f30c8045360 con 0x7f30dc145a90 2024-04-24T08:15:35.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.486+0000 7f30e1000640 1 -- 172.21.15.69:0/3434258440 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f30dc0d1120 con 0x7f30dc145a90 2024-04-24T08:15:35.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.486+0000 7f30d2000640 1 -- 172.21.15.69:0/3434258440 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) ==== 80+0+0 (secure 0 0 0) 0x7f30c807a860 con 0x7f30dc145a90 2024-04-24T08:15:35.488 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 -- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f30bc06c8f0 msgr2=0x7f30bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f30bc06c8f0 0x7f30bc06edb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f30dc142180 tx=0x7f30cc007ce0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 -- 172.21.15.69:0/3434258440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30dc145a90 msgr2=0x7f30dc15f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30dc145a90 0x7f30dc15f740 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f30c8000c00 tx=0x7f30c80046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 -- 172.21.15.69:0/3434258440 shutdown_connections 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f30bc06c8f0 0x7f30bc06edb0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f30dc145a90 0x7f30dc15f740 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30dc145fe0 0x7f30dc14a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 --2- 172.21.15.69:0/3434258440 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f30dc0d0300 0x7f30dc15ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 -- 172.21.15.69:0/3434258440 >> 172.21.15.69:0/3434258440 conn(0x7f30dc067d90 msgr2=0x7f30dc071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 -- 172.21.15.69:0/3434258440 shutdown_connections 2024-04-24T08:15:35.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.489+0000 7f30e1000640 1 -- 172.21.15.69:0/3434258440 wait complete. 2024-04-24T08:15:35.510 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:15:35.510 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:15:35.560 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:15:35.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.804+0000 7ff0f7000640 1 Processor -- start 2024-04-24T08:15:35.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.804+0000 7ff0f7000640 1 -- start start 2024-04-24T08:15:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.805+0000 7ff0f7000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0f00d0300 0x7ff0f006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.805+0000 7ff0f7000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f00cf3d0 con 0x7ff0f0145a90 2024-04-24T08:15:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.805+0000 7ff0f7000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0f00cf6b0 con 0x7ff0f0145fe0 2024-04-24T08:15:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.805+0000 7ff0f7000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff0f00cf990 con 0x7ff0f00d0300 2024-04-24T08:15:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.806+0000 7ff0e7e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff0f0145a90 0x7ff0f00d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43032/0 (socket says 172.21.15.69:43032) 2024-04-24T08:15:35.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.806+0000 7ff0e7e00640 1 -- 172.21.15.69:0/3789691574 learned_addr learned my addr 172.21.15.69:0/3789691574 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:35.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.806+0000 7ff0efe00640 1 --2- 172.21.15.69:0/3789691574 >> v2:172.21.15.121:3300/0 conn(0x7ff0f00d0300 0x7ff0f006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.806+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3969359411 0 0) 0x7ff0f00cf3d0 con 0x7ff0f0145a90 2024-04-24T08:15:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.806+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff0d4003520 con 0x7ff0f0145a90 2024-04-24T08:15:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3281679754 0 0) 0x7ff0d4003520 con 0x7ff0f0145a90 2024-04-24T08:15:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 >> v2:172.21.15.121:3300/0 conn(0x7ff0f00d0300 msgr2=0x7ff0f006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 --2- 172.21.15.69:0/3789691574 >> v2:172.21.15.121:3300/0 conn(0x7ff0f00d0300 0x7ff0f006e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 >> v1:172.21.15.121:6789/0 conn(0x7ff0f0145fe0 legacy=0x7ff0f0160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f00cfc70 con 0x7ff0f0145a90 2024-04-24T08:15:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7ff0d8001670 con 0x7ff0f0145a90 2024-04-24T08:15:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 >> v1:172.21.15.69:6789/0 conn(0x7ff0f0145a90 legacy=0x7ff0f00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d4003a00 0x7ff0d4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d4004800 0x7ff0d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d40051c0 0x7ff0d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f00cf990 con 0x7ff0d4003a00 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d400c0c0 con 0x7ff0d40051c0 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d400c2c0 con 0x7ff0d4004800 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0efe00640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d4003a00 0x7ff0d4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.807+0000 7ff0f5000640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d40051c0 0x7ff0d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.808+0000 7ff0efe00640 1 -- 172.21.15.69:0/3789691574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d4004800 msgr2=0x7ff0d4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.808+0000 7ff0efe00640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0d4004800 0x7ff0d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.808+0000 7ff0efe00640 1 -- 172.21.15.69:0/3789691574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d40051c0 msgr2=0x7ff0d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.808+0000 7ff0efe00640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d40051c0 0x7ff0d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.808+0000 7ff0efe00640 1 -- 172.21.15.69:0/3789691574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d4003520 con 0x7ff0d4003a00 2024-04-24T08:15:35.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.808+0000 7ff0f5000640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d40051c0 0x7ff0d400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:35.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.808+0000 7ff0f5000640 1 -- 172.21.15.69:0/3789691574 reap_dead start 2024-04-24T08:15:35.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.809+0000 7ff0efe00640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d4003a00 0x7ff0d4003e20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009ac0 tx=0x7ff0d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce48d94ce8449b58 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:35.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.809+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0036070 con 0x7ff0d4003a00 2024-04-24T08:15:35.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.809+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e0031400 con 0x7ff0d4003a00 2024-04-24T08:15:35.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.809+0000 7ff0ef400640 1 -- 172.21.15.69:0/3789691574 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e0009690 con 0x7ff0d4003a00 2024-04-24T08:15:35.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.810+0000 7ff0f7000640 1 -- 172.21.15.69:0/3789691574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d4003a00 msgr2=0x7ff0d4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.810+0000 7ff0f7000640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d4003a00 0x7ff0d4003e20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009ac0 tx=0x7ff0d4005830 comp rx=0 tx=0).stop 2024-04-24T08:15:35.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.810+0000 7ff0f7000640 1 -- 172.21.15.69:0/3789691574 shutdown_connections 2024-04-24T08:15:35.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.810+0000 7ff0f7000640 1 --2- 172.21.15.69:0/3789691574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0d4003a00 0x7ff0d4003e20 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.810+0000 7ff0f7000640 1 -- 172.21.15.69:0/3789691574 >> 172.21.15.69:0/3789691574 conn(0x7ff0f0067d90 msgr2=0x7ff0f006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:35.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.810+0000 7ff0f7000640 1 -- 172.21.15.69:0/3789691574 shutdown_connections 2024-04-24T08:15:35.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.810+0000 7ff0f7000640 1 -- 172.21.15.69:0/3789691574 wait complete. 2024-04-24T08:15:35.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.811+0000 7ff0f7000640 1 Processor -- start 2024-04-24T08:15:35.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.812+0000 7ff0f7000640 1 -- start start 2024-04-24T08:15:35.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.812+0000 7ff0f7000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d4003a00 0x7ff0f015c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.812+0000 7ff0f7000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0f0140810 0x7ff0f0141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.812+0000 7ff0f7000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0f00af220 0x7ff0f00abf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.812+0000 7ff0f7000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f01458f0 con 0x7ff0f00af220 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.812+0000 7ff0f7000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f0145770 con 0x7ff0d4003a00 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.812+0000 7ff0f7000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff0f00cdc30 con 0x7ff0f0140810 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.813+0000 7ff0f5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0f00af220 0x7ff0f00abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.813+0000 7ff0f5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0f00af220 0x7ff0f00abf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33250/0 (socket says 172.21.15.69:33250) 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.813+0000 7ff0f5000640 1 -- 172.21.15.69:0/3703998574 learned_addr learned my addr 172.21.15.69:0/3703998574 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.813+0000 7ff0efe00640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d4003a00 0x7ff0f015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.813+0000 7ff0f5000640 1 -- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0f0140810 msgr2=0x7ff0f0141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.813+0000 7ff0f5000640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0f0140810 0x7ff0f0141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.813+0000 7ff0f5000640 1 -- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d4003a00 msgr2=0x7ff0f015c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.813+0000 7ff0f5000640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d4003a00 0x7ff0f015c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:35.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.813+0000 7ff0f5000640 1 -- 172.21.15.69:0/3703998574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f015c870 con 0x7ff0f00af220 2024-04-24T08:15:35.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.813+0000 7ff0efe00640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d4003a00 0x7ff0f015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:35.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.814+0000 7ff0f5000640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0f00af220 0x7ff0f00abf10 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0071b30 tx=0x7ff0e800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:35.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.814+0000 7ff0ecc00640 1 -- 172.21.15.69:0/3703998574 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8018070 con 0x7ff0f00af220 2024-04-24T08:15:35.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.814+0000 7ff0ecc00640 1 -- 172.21.15.69:0/3703998574 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e8003da0 con 0x7ff0f00af220 2024-04-24T08:15:35.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.814+0000 7ff0ecc00640 1 -- 172.21.15.69:0/3703998574 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80052b0 con 0x7ff0f00af220 2024-04-24T08:15:35.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.814+0000 7ff0f7000640 1 -- 172.21.15.69:0/3703998574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f0142360 con 0x7ff0f00af220 2024-04-24T08:15:35.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.814+0000 7ff0f7000640 1 -- 172.21.15.69:0/3703998574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f0141060 con 0x7ff0f00af220 2024-04-24T08:15:35.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.815+0000 7ff0f7000640 1 -- 172.21.15.69:0/3703998574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0f006edf0 con 0x7ff0f00af220 2024-04-24T08:15:35.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.816+0000 7ff0ecc00640 1 -- 172.21.15.69:0/3703998574 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0e801b070 con 0x7ff0f00af220 2024-04-24T08:15:35.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.817+0000 7ff0ecc00640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0cc06c920 0x7ff0cc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:35.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.817+0000 7ff0ecc00640 1 -- 172.21.15.69:0/3703998574 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7ff0e808b120 con 0x7ff0f00af220 2024-04-24T08:15:35.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.818+0000 7ff0efe00640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0cc06c920 0x7ff0cc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:35.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.819+0000 7ff0efe00640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0cc06c920 0x7ff0cc06ede0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0033000 tx=0x7ff0e0008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:35.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:35.822+0000 7ff0ecc00640 1 -- 172.21.15.69:0/3703998574 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff0f006edf0 con 0x7ff0f00af220 2024-04-24T08:15:36.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.021+0000 7ff0f7000640 1 -- 172.21.15.69:0/3703998574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0f00d1120 con 0x7ff0f00af220 2024-04-24T08:15:36.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.023+0000 7ff0ecc00640 1 -- 172.21.15.69:0/3703998574 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+16636 (secure 0 0 0) 0x7ff0f015c870 con 0x7ff0f00af220 2024-04-24T08:15:36.025 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:36.025 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":82,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:15:24.982737+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:15:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.026+0000 7ff0f7000640 1 -- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0cc06c920 msgr2=0x7ff0cc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.026+0000 7ff0f7000640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0cc06c920 0x7ff0cc06ede0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0033000 tx=0x7ff0e0008030 comp rx=0 tx=0).stop 2024-04-24T08:15:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.026+0000 7ff0f7000640 1 -- 172.21.15.69:0/3703998574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0f00af220 msgr2=0x7ff0f00abf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.026+0000 7ff0f7000640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0f00af220 0x7ff0f00abf10 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0071b30 tx=0x7ff0e800d9d0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.026+0000 7ff0f7000640 1 -- 172.21.15.69:0/3703998574 shutdown_connections 2024-04-24T08:15:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.026+0000 7ff0f7000640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff0cc06c920 0x7ff0cc06ede0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.026+0000 7ff0f7000640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff0f00af220 0x7ff0f00abf10 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.026+0000 7ff0f7000640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0f0140810 0x7ff0f0141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.026+0000 7ff0f7000640 1 --2- 172.21.15.69:0/3703998574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0d4003a00 0x7ff0f015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.026+0000 7ff0f7000640 1 -- 172.21.15.69:0/3703998574 >> 172.21.15.69:0/3703998574 conn(0x7ff0f0067d90 msgr2=0x7ff0f00b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.027+0000 7ff0f7000640 1 -- 172.21.15.69:0/3703998574 shutdown_connections 2024-04-24T08:15:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.027+0000 7ff0f7000640 1 -- 172.21.15.69:0/3703998574 wait complete. 2024-04-24T08:15:36.048 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:15:21.126081+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:15:22.402310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:15:36.048 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:15:36.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.251+0000 7f8d58c00640 1 Processor -- start 2024-04-24T08:15:36.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.252+0000 7f8d58c00640 1 -- start start 2024-04-24T08:15:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.253+0000 7f8d58c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8d54162f90 0x7f8d540cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.253+0000 7f8d58c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d540d0190 0x7f8d5406cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.253+0000 7f8d58c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d5415ed60 con 0x7f8d5415e980 2024-04-24T08:15:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.253+0000 7f8d58c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8d541633d0 con 0x7f8d540d0190 2024-04-24T08:15:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.253+0000 7f8d58c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8d540d1580 con 0x7f8d54162f90 2024-04-24T08:15:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.253+0000 7f8d53400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8d5415e980 0x7f8d540d0da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43034/0 (socket says 172.21.15.69:43034) 2024-04-24T08:15:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.253+0000 7f8d53400640 1 -- 172.21.15.69:0/1033632431 learned_addr learned my addr 172.21.15.69:0/1033632431 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.253+0000 7f8d53e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d540d0190 0x7f8d5406cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.254+0000 7f8d52a00640 1 --2- 172.21.15.69:0/1033632431 >> v2:172.21.15.121:3300/0 conn(0x7f8d54162f90 0x7f8d540cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.254+0000 7f8d53e00640 1 -- 172.21.15.69:0/1033632431 >> v2:172.21.15.121:3300/0 conn(0x7f8d54162f90 msgr2=0x7f8d540cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.254+0000 7f8d53e00640 1 --2- 172.21.15.69:0/1033632431 >> v2:172.21.15.121:3300/0 conn(0x7f8d54162f90 0x7f8d540cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.254+0000 7f8d53e00640 1 -- 172.21.15.69:0/1033632431 >> v1:172.21.15.69:6789/0 conn(0x7f8d5415e980 legacy=0x7f8d540d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.254+0000 7f8d52000640 1 -- 172.21.15.69:0/1033632431 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3907251334 0 0) 0x7f8d5415ed60 con 0x7f8d5415e980 2024-04-24T08:15:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.254+0000 7f8d53e00640 1 -- 172.21.15.69:0/1033632431 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d540d1860 con 0x7f8d540d0190 2024-04-24T08:15:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.254+0000 7f8d52a00640 1 --2- 172.21.15.69:0/1033632431 >> v2:172.21.15.121:3300/0 conn(0x7f8d54162f90 0x7f8d540cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.255+0000 7f8d53e00640 1 --2- 172.21.15.69:0/1033632431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 0x7f8d5406cfb0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f8d5408da70 tx=0x7f8d4c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=582ecc8b7033e3e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:36.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.255+0000 7f8d52000640 1 -- 172.21.15.69:0/1033632431 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c007740 con 0x7f8d540d0190 2024-04-24T08:15:36.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.255+0000 7f8d52000640 1 -- 172.21.15.69:0/1033632431 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d4c0078e0 con 0x7f8d540d0190 2024-04-24T08:15:36.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.255+0000 7f8d52000640 1 -- 172.21.15.69:0/1033632431 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c007cb0 con 0x7f8d540d0190 2024-04-24T08:15:36.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.256+0000 7f8d58c00640 1 -- 172.21.15.69:0/1033632431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 msgr2=0x7f8d5406cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.256+0000 7f8d58c00640 1 --2- 172.21.15.69:0/1033632431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 0x7f8d5406cfb0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f8d5408da70 tx=0x7f8d4c02a710 comp rx=0 tx=0).stop 2024-04-24T08:15:36.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.256+0000 7f8d58c00640 1 -- 172.21.15.69:0/1033632431 shutdown_connections 2024-04-24T08:15:36.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.256+0000 7f8d58c00640 1 --2- 172.21.15.69:0/1033632431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 0x7f8d5406cfb0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.256+0000 7f8d58c00640 1 --2- 172.21.15.69:0/1033632431 >> v2:172.21.15.121:3300/0 conn(0x7f8d54162f90 0x7f8d540cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.256+0000 7f8d58c00640 1 -- 172.21.15.69:0/1033632431 >> 172.21.15.69:0/1033632431 conn(0x7f8d5409d920 msgr2=0x7f8d5408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:36.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.256+0000 7f8d58c00640 1 -- 172.21.15.69:0/1033632431 shutdown_connections 2024-04-24T08:15:36.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.256+0000 7f8d58c00640 1 -- 172.21.15.69:0/1033632431 wait complete. 2024-04-24T08:15:36.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.258+0000 7f8d58c00640 1 Processor -- start 2024-04-24T08:15:36.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.258+0000 7f8d58c00640 1 -- start start 2024-04-24T08:15:36.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.259+0000 7f8d58c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 0x7f8d5415ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.259+0000 7f8d58c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5415e980 0x7f8d5406d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.259+0000 7f8d58c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d54162f90 0x7f8d541496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.259+0000 7f8d58c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d540cf070 con 0x7f8d540d0190 2024-04-24T08:15:36.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.259+0000 7f8d58c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8d540cf730 con 0x7f8d54162f90 2024-04-24T08:15:36.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.259+0000 7f8d58c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8d540cf1f0 con 0x7f8d5415e980 2024-04-24T08:15:36.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.259+0000 7f8d53400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 0x7f8d5415ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.259+0000 7f8d53400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 0x7f8d5415ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33256/0 (socket says 172.21.15.69:33256) 2024-04-24T08:15:36.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.259+0000 7f8d53400640 1 -- 172.21.15.69:0/647219645 learned_addr learned my addr 172.21.15.69:0/647219645 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:36.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.260+0000 7f8d53e00640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d54162f90 0x7f8d541496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.260+0000 7f8d52a00640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5415e980 0x7f8d5406d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.260+0000 7f8d53400640 1 -- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5415e980 msgr2=0x7f8d5406d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.260+0000 7f8d53400640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5415e980 0x7f8d5406d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.260+0000 7f8d53400640 1 -- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d54162f90 msgr2=0x7f8d541496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.260+0000 7f8d53400640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d54162f90 0x7f8d541496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.260+0000 7f8d53400640 1 -- 172.21.15.69:0/647219645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d540b3d50 con 0x7f8d540d0190 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.260+0000 7f8d53400640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 0x7f8d5415ef40 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400c500 tx=0x7f8d4400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.260+0000 7f8d4b400640 1 -- 172.21.15.69:0/647219645 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d44018070 con 0x7f8d540d0190 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.261+0000 7f8d4b400640 1 -- 172.21.15.69:0/647219645 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d44007580 con 0x7f8d540d0190 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.261+0000 7f8d4b400640 1 -- 172.21.15.69:0/647219645 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d44004a20 con 0x7f8d540d0190 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.261+0000 7f8d58c00640 1 -- 172.21.15.69:0/647219645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d540b4030 con 0x7f8d540d0190 2024-04-24T08:15:36.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.261+0000 7f8d58c00640 1 -- 172.21.15.69:0/647219645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d540b45f0 con 0x7f8d540d0190 2024-04-24T08:15:36.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.262+0000 7f8d58c00640 1 -- 172.21.15.69:0/647219645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d54075d50 con 0x7f8d540d0190 2024-04-24T08:15:36.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.263+0000 7f8d4b400640 1 -- 172.21.15.69:0/647219645 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8d44004bc0 con 0x7f8d540d0190 2024-04-24T08:15:36.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.264+0000 7f8d4b400640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d3006c8f0 0x7f8d3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.264+0000 7f8d4b400640 1 -- 172.21.15.69:0/647219645 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f8d4408ae70 con 0x7f8d540d0190 2024-04-24T08:15:36.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.264+0000 7f8d52a00640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d3006c8f0 0x7f8d3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.265+0000 7f8d52a00640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d3006c8f0 0x7f8d3006edb0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8d54161970 tx=0x7f8d3c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:36.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.268+0000 7f8d4b400640 1 -- 172.21.15.69:0/647219645 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d44059290 con 0x7f8d540d0190 2024-04-24T08:15:36.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.466+0000 7f8d58c00640 1 -- 172.21.15.69:0/647219645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d540d1e40 con 0x7f8d540d0190 2024-04-24T08:15:36.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.467+0000 7f8d4b400640 1 -- 172.21.15.69:0/647219645 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f8d44058b90 con 0x7f8d540d0190 2024-04-24T08:15:36.469 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:15:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.469+0000 7f8d58c00640 1 -- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d3006c8f0 msgr2=0x7f8d3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.469+0000 7f8d58c00640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d3006c8f0 0x7f8d3006edb0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8d54161970 tx=0x7f8d3c008040 comp rx=0 tx=0).stop 2024-04-24T08:15:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.469+0000 7f8d58c00640 1 -- 172.21.15.69:0/647219645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 msgr2=0x7f8d5415ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.469+0000 7f8d58c00640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 0x7f8d5415ef40 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f8d4400c500 tx=0x7f8d4400c9c0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.469+0000 7f8d58c00640 1 -- 172.21.15.69:0/647219645 shutdown_connections 2024-04-24T08:15:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.469+0000 7f8d58c00640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d3006c8f0 0x7f8d3006edb0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.469+0000 7f8d58c00640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d54162f90 0x7f8d541496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.469+0000 7f8d58c00640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d5415e980 0x7f8d5406d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.469+0000 7f8d58c00640 1 --2- 172.21.15.69:0/647219645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d540d0190 0x7f8d5415ef40 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.469+0000 7f8d58c00640 1 -- 172.21.15.69:0/647219645 >> 172.21.15.69:0/647219645 conn(0x7f8d5409d920 msgr2=0x7f8d540b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.470+0000 7f8d58c00640 1 -- 172.21.15.69:0/647219645 shutdown_connections 2024-04-24T08:15:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.470+0000 7f8d58c00640 1 -- 172.21.15.69:0/647219645 wait complete. 2024-04-24T08:15:36.491 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:15:36.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.693+0000 7f99eb200640 1 Processor -- start 2024-04-24T08:15:36.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.694+0000 7f99eb200640 1 -- start start 2024-04-24T08:15:36.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.695+0000 7f99eb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec15edf0 0x7f99ec15f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.695+0000 7f99eb200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99ec06e000 0x7f99ec06e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.695+0000 7f99eb200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99ec15fbb0 con 0x7f99ec071d70 2024-04-24T08:15:36.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.695+0000 7f99eb200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f99ec0d0a00 con 0x7f99ec06e000 2024-04-24T08:15:36.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.695+0000 7f99eb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99ec0d00d0 con 0x7f99ec15edf0 2024-04-24T08:15:36.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.696+0000 7f99ea800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99ec06e000 0x7f99ec06e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.696+0000 7f99ea800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f99ec06e000 0x7f99ec06e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33264/0 (socket says 172.21.15.69:33264) 2024-04-24T08:15:36.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.696+0000 7f99ea800640 1 -- 172.21.15.69:0/1106425188 learned_addr learned my addr 172.21.15.69:0/1106425188 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:36.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.696+0000 7f99e3e00640 1 --2- 172.21.15.69:0/1106425188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec15edf0 0x7f99ec15f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.696+0000 7f99ea800640 1 -- 172.21.15.69:0/1106425188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec15edf0 msgr2=0x7f99ec15f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:36.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.696+0000 7f99ea800640 1 --2- 172.21.15.69:0/1106425188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec15edf0 0x7f99ec15f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.696+0000 7f99ea800640 1 -- 172.21.15.69:0/1106425188 >> v1:172.21.15.121:6789/0 conn(0x7f99ec071d70 legacy=0x7f99ec072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:36.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.696+0000 7f99ea800640 1 -- 172.21.15.69:0/1106425188 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec0d03b0 con 0x7f99ec06e000 2024-04-24T08:15:36.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.697+0000 7f99ea800640 1 --2- 172.21.15.69:0/1106425188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec06e000 0x7f99ec06e490 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009e50 tx=0x7f99e402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f2166cf7769a86 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:36.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.697+0000 7f99e9400640 1 -- 172.21.15.69:0/1106425188 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e4036070 con 0x7f99ec06e000 2024-04-24T08:15:36.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.698+0000 7f99e9400640 1 -- 172.21.15.69:0/1106425188 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99e4031600 con 0x7f99ec06e000 2024-04-24T08:15:36.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.698+0000 7f99e9400640 1 -- 172.21.15.69:0/1106425188 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e40319d0 con 0x7f99ec06e000 2024-04-24T08:15:36.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.698+0000 7f99eb200640 1 -- 172.21.15.69:0/1106425188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec06e000 msgr2=0x7f99ec06e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.698+0000 7f99eb200640 1 --2- 172.21.15.69:0/1106425188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec06e000 0x7f99ec06e490 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009e50 tx=0x7f99e402a6c0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.699+0000 7f99eb200640 1 -- 172.21.15.69:0/1106425188 shutdown_connections 2024-04-24T08:15:36.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.699+0000 7f99eb200640 1 --2- 172.21.15.69:0/1106425188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec06e000 0x7f99ec06e490 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.699+0000 7f99eb200640 1 --2- 172.21.15.69:0/1106425188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec15edf0 0x7f99ec15f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.699+0000 7f99eb200640 1 -- 172.21.15.69:0/1106425188 >> 172.21.15.69:0/1106425188 conn(0x7f99ec067b80 msgr2=0x7f99ec06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:36.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.699+0000 7f99eb200640 1 -- 172.21.15.69:0/1106425188 shutdown_connections 2024-04-24T08:15:36.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.699+0000 7f99eb200640 1 -- 172.21.15.69:0/1106425188 wait complete. 2024-04-24T08:15:36.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.701+0000 7f99eb200640 1 Processor -- start 2024-04-24T08:15:36.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.701+0000 7f99eb200640 1 -- start start 2024-04-24T08:15:36.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.702+0000 7f99eb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec06e000 0x7f99ec1f5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.702+0000 7f99eb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec071d70 0x7f99ec0b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.702+0000 7f99eb200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99ec15edf0 0x7f99ec0c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.702+0000 7f99eb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec0d1010 con 0x7f99ec071d70 2024-04-24T08:15:36.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.702+0000 7f99eb200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec0d0e90 con 0x7f99ec15edf0 2024-04-24T08:15:36.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.702+0000 7f99eb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99ec1481a0 con 0x7f99ec06e000 2024-04-24T08:15:36.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.702+0000 7f99e3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec071d70 0x7f99ec0b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.702+0000 7f99e3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec071d70 0x7f99ec0b0a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33268/0 (socket says 172.21.15.69:33268) 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.702+0000 7f99e3e00640 1 -- 172.21.15.69:0/2329223698 learned_addr learned my addr 172.21.15.69:0/2329223698 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99e9e00640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec06e000 0x7f99ec1f5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99ea800640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99ec15edf0 0x7f99ec0c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99e3e00640 1 -- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec06e000 msgr2=0x7f99ec1f5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99e3e00640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec06e000 0x7f99ec1f5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99e3e00640 1 -- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99ec15edf0 msgr2=0x7f99ec0c76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99e3e00640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99ec15edf0 0x7f99ec0c76d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99e3e00640 1 -- 172.21.15.69:0/2329223698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec1f5940 con 0x7f99ec071d70 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99e9e00640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec06e000 0x7f99ec1f5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99e3e00640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec071d70 0x7f99ec0b0a90 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f99d400d5f0 tx=0x7f99d400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:36.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99e2000640 1 -- 172.21.15.69:0/2329223698 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d40125f0 con 0x7f99ec071d70 2024-04-24T08:15:36.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99e2000640 1 -- 172.21.15.69:0/2329223698 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99d4019070 con 0x7f99ec071d70 2024-04-24T08:15:36.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.703+0000 7f99eb200640 1 -- 172.21.15.69:0/2329223698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99ec0af9f0 con 0x7f99ec071d70 2024-04-24T08:15:36.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.704+0000 7f99e2000640 1 -- 172.21.15.69:0/2329223698 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d40129d0 con 0x7f99ec071d70 2024-04-24T08:15:36.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.704+0000 7f99eb200640 1 -- 172.21.15.69:0/2329223698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99ec0affb0 con 0x7f99ec071d70 2024-04-24T08:15:36.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.705+0000 7f99eb200640 1 -- 172.21.15.69:0/2329223698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99ec075d50 con 0x7f99ec071d70 2024-04-24T08:15:36.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.706+0000 7f99e2000640 1 -- 172.21.15.69:0/2329223698 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f99d4012b70 con 0x7f99ec071d70 2024-04-24T08:15:36.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.707+0000 7f99e2000640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f99c806c8f0 0x7f99c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:36.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.707+0000 7f99e2000640 1 -- 172.21.15.69:0/2329223698 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f99d408bef0 con 0x7f99ec071d70 2024-04-24T08:15:36.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.707+0000 7f99e9e00640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f99c806c8f0 0x7f99c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:36.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.708+0000 7f99e9e00640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f99c806c8f0 0x7f99c806edb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f99ec1478c0 tx=0x7f99dc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:36.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.711+0000 7f99e2000640 1 -- 172.21.15.69:0/2329223698 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f99ec1f5940 con 0x7f99ec071d70 2024-04-24T08:15:36.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.909+0000 7f99eb200640 1 -- 172.21.15.69:0/2329223698 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f99ec0cfeb0 con 0x7f99ec071d70 2024-04-24T08:15:36.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.910+0000 7f99e2000640 1 -- 172.21.15.69:0/2329223698 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) ==== 104+0+11 (secure 0 0 0) 0x7f99d405a390 con 0x7f99ec071d70 2024-04-24T08:15:36.912 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 -- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f99c806c8f0 msgr2=0x7f99c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f99c806c8f0 0x7f99c806edb0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f99ec1478c0 tx=0x7f99dc009040 comp rx=0 tx=0).stop 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 -- 172.21.15.69:0/2329223698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec071d70 msgr2=0x7f99ec0b0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec071d70 0x7f99ec0b0a90 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f99d400d5f0 tx=0x7f99d400dab0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 -- 172.21.15.69:0/2329223698 shutdown_connections 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f99c806c8f0 0x7f99c806edb0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99ec071d70 0x7f99ec0b0a90 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99ec15edf0 0x7f99ec0c76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 --2- 172.21.15.69:0/2329223698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99ec06e000 0x7f99ec1f5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 -- 172.21.15.69:0/2329223698 >> 172.21.15.69:0/2329223698 conn(0x7f99ec067b80 msgr2=0x7f99ec144830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 -- 172.21.15.69:0/2329223698 shutdown_connections 2024-04-24T08:15:36.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:36.913+0000 7f99eb200640 1 -- 172.21.15.69:0/2329223698 wait complete. 2024-04-24T08:15:36.934 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:15:37.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.137+0000 7f3d17e00640 1 Processor -- start 2024-04-24T08:15:37.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.137+0000 7f3d17e00640 1 -- start start 2024-04-24T08:15:37.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.138+0000 7f3d17e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3d1015c820 0x7f3d100b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.138+0000 7f3d17e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3d100ce250 0x7f3d100ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.138+0000 7f3d17e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d1015cc00 con 0x7f3d100b46a0 2024-04-24T08:15:37.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.138+0000 7f3d17e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f3d1006ed30 con 0x7f3d100ce250 2024-04-24T08:15:37.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.138+0000 7f3d17e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3d100b4e10 con 0x7f3d1015c820 2024-04-24T08:15:37.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.139+0000 7f3d15e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3d100ce250 0x7f3d100ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:37.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.139+0000 7f3d15e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f3d100ce250 0x7f3d100ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33278/0 (socket says 172.21.15.69:33278) 2024-04-24T08:15:37.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.139+0000 7f3d15e00640 1 -- 172.21.15.69:0/1202774705 learned_addr learned my addr 172.21.15.69:0/1202774705 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:37.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.139+0000 7f3d15400640 1 --2- 172.21.15.69:0/1202774705 >> v2:172.21.15.121:3300/0 conn(0x7f3d1015c820 0x7f3d100b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:37.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.139+0000 7f3d15e00640 1 -- 172.21.15.69:0/1202774705 >> v2:172.21.15.121:3300/0 conn(0x7f3d1015c820 msgr2=0x7f3d100b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:37.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.139+0000 7f3d15e00640 1 --2- 172.21.15.69:0/1202774705 >> v2:172.21.15.121:3300/0 conn(0x7f3d1015c820 0x7f3d100b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.139+0000 7f3d15e00640 1 -- 172.21.15.69:0/1202774705 >> v1:172.21.15.121:6789/0 conn(0x7f3d100b46a0 legacy=0x7f3d1006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:37.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.139+0000 7f3d15e00640 1 -- 172.21.15.69:0/1202774705 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d100b50f0 con 0x7f3d100ce250 2024-04-24T08:15:37.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.141+0000 7f3d15e00640 1 --2- 172.21.15.69:0/1202774705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100ce250 0x7f3d100ce6c0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f3d100b2f60 tx=0x7f3d0802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c29000dbf139e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:37.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.141+0000 7f3d0fe00640 1 -- 172.21.15.69:0/1202774705 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d08036070 con 0x7f3d100ce250 2024-04-24T08:15:37.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.141+0000 7f3d0fe00640 1 -- 172.21.15.69:0/1202774705 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d08031600 con 0x7f3d100ce250 2024-04-24T08:15:37.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.141+0000 7f3d0fe00640 1 -- 172.21.15.69:0/1202774705 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d080319d0 con 0x7f3d100ce250 2024-04-24T08:15:37.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.142+0000 7f3d17e00640 1 -- 172.21.15.69:0/1202774705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100ce250 msgr2=0x7f3d100ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.142+0000 7f3d17e00640 1 --2- 172.21.15.69:0/1202774705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100ce250 0x7f3d100ce6c0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f3d100b2f60 tx=0x7f3d0802a6f0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.142+0000 7f3d17e00640 1 -- 172.21.15.69:0/1202774705 shutdown_connections 2024-04-24T08:15:37.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.142+0000 7f3d17e00640 1 --2- 172.21.15.69:0/1202774705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100ce250 0x7f3d100ce6c0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.142+0000 7f3d17e00640 1 --2- 172.21.15.69:0/1202774705 >> v2:172.21.15.121:3300/0 conn(0x7f3d1015c820 0x7f3d100b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.142+0000 7f3d17e00640 1 -- 172.21.15.69:0/1202774705 >> 172.21.15.69:0/1202774705 conn(0x7f3d100d0350 msgr2=0x7f3d100d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:37.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.142+0000 7f3d17e00640 1 -- 172.21.15.69:0/1202774705 shutdown_connections 2024-04-24T08:15:37.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.142+0000 7f3d17e00640 1 -- 172.21.15.69:0/1202774705 wait complete. 2024-04-24T08:15:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.144+0000 7f3d17e00640 1 Processor -- start 2024-04-24T08:15:37.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.144+0000 7f3d17e00640 1 -- start start 2024-04-24T08:15:37.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.145+0000 7f3d17e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100b46a0 0x7f3d101e61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.145+0000 7f3d17e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d100ce250 0x7f3d100aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.145+0000 7f3d17e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d1015c820 0x7f3d1015f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.145+0000 7f3d17e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3d100b4b30 con 0x7f3d100b46a0 2024-04-24T08:15:37.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.145+0000 7f3d17e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1006f2b0 con 0x7f3d1015c820 2024-04-24T08:15:37.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.145+0000 7f3d17e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1006fee0 con 0x7f3d100ce250 2024-04-24T08:15:37.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.145+0000 7f3d15400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100b46a0 0x7f3d101e61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.145+0000 7f3d15400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100b46a0 0x7f3d101e61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33282/0 (socket says 172.21.15.69:33282) 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.145+0000 7f3d15400640 1 -- 172.21.15.69:0/2026167630 learned_addr learned my addr 172.21.15.69:0/2026167630 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.145+0000 7f3d15400640 1 -- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d100ce250 msgr2=0x7f3d100aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.146+0000 7f3d15400640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d100ce250 0x7f3d100aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.146+0000 7f3d15400640 1 -- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d1015c820 msgr2=0x7f3d1015f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.146+0000 7f3d15400640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d1015c820 0x7f3d1015f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.146+0000 7f3d15400640 1 -- 172.21.15.69:0/2026167630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1015e410 con 0x7f3d100b46a0 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.146+0000 7f3d15400640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100b46a0 0x7f3d101e61c0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400d830 tx=0x7f3d0400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.146+0000 7f3d0d600640 1 -- 172.21.15.69:0/2026167630 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0400b6c0 con 0x7f3d100b46a0 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.146+0000 7f3d0d600640 1 -- 172.21.15.69:0/2026167630 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d04019070 con 0x7f3d100b46a0 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.146+0000 7f3d17e00640 1 -- 172.21.15.69:0/2026167630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d1015e6f0 con 0x7f3d100b46a0 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.146+0000 7f3d0d600640 1 -- 172.21.15.69:0/2026167630 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d0400ba80 con 0x7f3d100b46a0 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.146+0000 7f3d17e00640 1 -- 172.21.15.69:0/2026167630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d1015ecb0 con 0x7f3d100b46a0 2024-04-24T08:15:37.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.147+0000 7f3d17e00640 1 -- 172.21.15.69:0/2026167630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d10068a80 con 0x7f3d100b46a0 2024-04-24T08:15:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.148+0000 7f3d0d600640 1 -- 172.21.15.69:0/2026167630 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d0401e460 con 0x7f3d100b46a0 2024-04-24T08:15:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.149+0000 7f3d0d600640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf006c8f0 0x7f3cf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.149+0000 7f3d0d600640 1 -- 172.21.15.69:0/2026167630 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f3d0408c2b0 con 0x7f3d100b46a0 2024-04-24T08:15:37.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.150+0000 7f3d14a00640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf006c8f0 0x7f3cf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:37.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.151+0000 7f3d14a00640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf006c8f0 0x7f3cf006edb0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc006fd0 tx=0x7f3cfc006bc0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:37.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.153+0000 7f3d0d600640 1 -- 172.21.15.69:0/2026167630 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d0405a6d0 con 0x7f3d100b46a0 2024-04-24T08:15:37.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.352+0000 7f3d17e00640 1 -- 172.21.15.69:0/2026167630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3d1015f0b0 con 0x7f3d100b46a0 2024-04-24T08:15:37.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.353+0000 7f3d0d600640 1 -- 172.21.15.69:0/2026167630 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) ==== 100+0+11 (secure 0 0 0) 0x7f3d04059fd0 con 0x7f3d100b46a0 2024-04-24T08:15:37.354 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:15:37.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 -- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf006c8f0 msgr2=0x7f3cf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf006c8f0 0x7f3cf006edb0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc006fd0 tx=0x7f3cfc006bc0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 -- 172.21.15.69:0/2026167630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100b46a0 msgr2=0x7f3d101e61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100b46a0 0x7f3d101e61c0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f3d0400d830 tx=0x7f3d0400dcf0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 -- 172.21.15.69:0/2026167630 shutdown_connections 2024-04-24T08:15:37.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3cf006c8f0 0x7f3cf006edb0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d1015c820 0x7f3d1015f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3d100ce250 0x7f3d100aea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 --2- 172.21.15.69:0/2026167630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d100b46a0 0x7f3d101e61c0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 -- 172.21.15.69:0/2026167630 >> 172.21.15.69:0/2026167630 conn(0x7f3d100d0350 msgr2=0x7f3d100afdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:37.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 -- 172.21.15.69:0/2026167630 shutdown_connections 2024-04-24T08:15:37.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.356+0000 7f3d17e00640 1 -- 172.21.15.69:0/2026167630 wait complete. 2024-04-24T08:15:37.376 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:15:37.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.573+0000 7f8e79a00640 1 Processor -- start 2024-04-24T08:15:37.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.574+0000 7f8e79a00640 1 -- start start 2024-04-24T08:15:37.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.575+0000 7f8e79a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8e7406f2d0 0x7f8e7406f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.575+0000 7f8e79a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e740b46e0 con 0x7f8e7415b490 2024-04-24T08:15:37.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.575+0000 7f8e79a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e740d0960 con 0x7f8e740cfbe0 2024-04-24T08:15:37.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.575+0000 7f8e79a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8e7406fc70 con 0x7f8e7406f2d0 2024-04-24T08:15:37.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.576+0000 7f8e6a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8e7406f2d0 0x7f8e7406f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:37.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.576+0000 7f8e6a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8e7406f2d0 0x7f8e7406f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33284/0 (socket says 172.21.15.69:33284) 2024-04-24T08:15:37.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.576+0000 7f8e6a000640 1 -- 172.21.15.69:0/1074134738 learned_addr learned my addr 172.21.15.69:0/1074134738 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:37.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.576+0000 7f8e72a00640 1 --1- 172.21.15.69:0/1074134738 >> v1:172.21.15.69:6789/0 conn(0x7f8e7415b490 0x7f8e740b5180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43042/0 (socket says 172.21.15.69:43042) 2024-04-24T08:15:37.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.576+0000 7f8e6a000640 1 -- 172.21.15.69:0/1074134738 >> v1:172.21.15.121:6789/0 conn(0x7f8e740cfbe0 legacy=0x7f8e740d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:37.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.576+0000 7f8e6a000640 1 -- 172.21.15.69:0/1074134738 >> v1:172.21.15.69:6789/0 conn(0x7f8e7415b490 legacy=0x7f8e740b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.576+0000 7f8e6a000640 1 -- 172.21.15.69:0/1074134738 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e740b4980 con 0x7f8e7406f2d0 2024-04-24T08:15:37.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.576+0000 7f8e72000640 1 -- 172.21.15.69:0/1074134738 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3386087015 0 0) 0x7f8e740b46e0 con 0x7f8e7415b490 2024-04-24T08:15:37.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.578+0000 7f8e6a000640 1 --2- 172.21.15.69:0/1074134738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7406f2d0 0x7f8e7406f730 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009870 tx=0x7f8e5c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b98664296634d55 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:37.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.578+0000 7f8e72000640 1 -- 172.21.15.69:0/1074134738 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c037070 con 0x7f8e7406f2d0 2024-04-24T08:15:37.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.578+0000 7f8e72000640 1 -- 172.21.15.69:0/1074134738 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e5c032440 con 0x7f8e7406f2d0 2024-04-24T08:15:37.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.578+0000 7f8e72000640 1 -- 172.21.15.69:0/1074134738 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c032810 con 0x7f8e7406f2d0 2024-04-24T08:15:37.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.578+0000 7f8e79a00640 1 -- 172.21.15.69:0/1074134738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7406f2d0 msgr2=0x7f8e7406f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.578+0000 7f8e79a00640 1 --2- 172.21.15.69:0/1074134738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7406f2d0 0x7f8e7406f730 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c009870 tx=0x7f8e5c02a2a0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.579+0000 7f8e79a00640 1 -- 172.21.15.69:0/1074134738 shutdown_connections 2024-04-24T08:15:37.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.579+0000 7f8e79a00640 1 --2- 172.21.15.69:0/1074134738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7406f2d0 0x7f8e7406f730 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.579+0000 7f8e79a00640 1 -- 172.21.15.69:0/1074134738 >> 172.21.15.69:0/1074134738 conn(0x7f8e74067d90 msgr2=0x7f8e7406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:37.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.579+0000 7f8e79a00640 1 -- 172.21.15.69:0/1074134738 shutdown_connections 2024-04-24T08:15:37.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.579+0000 7f8e79a00640 1 -- 172.21.15.69:0/1074134738 wait complete. 2024-04-24T08:15:37.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.580+0000 7f8e79a00640 1 Processor -- start 2024-04-24T08:15:37.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.581+0000 7f8e79a00640 1 -- start start 2024-04-24T08:15:37.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.581+0000 7f8e79a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e7406f2d0 0x7f8e741e58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.581+0000 7f8e79a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e740cfbe0 0x7f8e741f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e79a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7415b490 0x7f8e740a9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e79a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8e740b5890 con 0x7f8e7415b490 2024-04-24T08:15:37.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e79a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8e740d0720 con 0x7f8e7406f2d0 2024-04-24T08:15:37.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e79a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8e740d0c40 con 0x7f8e740cfbe0 2024-04-24T08:15:37.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e73400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7415b490 0x7f8e740a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:37.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e73400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7415b490 0x7f8e740a9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33288/0 (socket says 172.21.15.69:33288) 2024-04-24T08:15:37.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e73400640 1 -- 172.21.15.69:0/2692376857 learned_addr learned my addr 172.21.15.69:0/2692376857 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:37.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e72a00640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e7406f2d0 0x7f8e741e58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:37.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e6a000640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e740cfbe0 0x7f8e741f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:37.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e73400640 1 -- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e740cfbe0 msgr2=0x7f8e741f2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e73400640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e740cfbe0 0x7f8e741f2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e73400640 1 -- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e7406f2d0 msgr2=0x7f8e741e58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e73400640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e7406f2d0 0x7f8e741e58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e73400640 1 -- 172.21.15.69:0/2692376857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e740a9970 con 0x7f8e7415b490 2024-04-24T08:15:37.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.582+0000 7f8e6a000640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e740cfbe0 0x7f8e741f2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:37.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.583+0000 7f8e73400640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7415b490 0x7f8e740a9430 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00c780 tx=0x7f8e6c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:37.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.583+0000 7f8e6b400640 1 -- 172.21.15.69:0/2692376857 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c004b60 con 0x7f8e7415b490 2024-04-24T08:15:37.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.583+0000 7f8e6b400640 1 -- 172.21.15.69:0/2692376857 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e6c004d00 con 0x7f8e7415b490 2024-04-24T08:15:37.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.583+0000 7f8e6b400640 1 -- 172.21.15.69:0/2692376857 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c00aa90 con 0x7f8e7415b490 2024-04-24T08:15:37.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.583+0000 7f8e79a00640 1 -- 172.21.15.69:0/2692376857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e741e5f20 con 0x7f8e7415b490 2024-04-24T08:15:37.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.583+0000 7f8e79a00640 1 -- 172.21.15.69:0/2692376857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e741e64e0 con 0x7f8e7415b490 2024-04-24T08:15:37.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.584+0000 7f8e79a00640 1 -- 172.21.15.69:0/2692376857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e7406bf50 con 0x7f8e7415b490 2024-04-24T08:15:37.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.586+0000 7f8e6b400640 1 -- 172.21.15.69:0/2692376857 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8e6c00ac30 con 0x7f8e7415b490 2024-04-24T08:15:37.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.587+0000 7f8e6b400640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8e5006c8f0 0x7f8e5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:37.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.587+0000 7f8e6b400640 1 -- 172.21.15.69:0/2692376857 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f8e6c08bd80 con 0x7f8e7415b490 2024-04-24T08:15:37.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.587+0000 7f8e72a00640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8e5006c8f0 0x7f8e5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:37.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.589+0000 7f8e72a00640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8e5006c8f0 0x7f8e5006edb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8e64007fd0 tx=0x7f8e64007c20 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:37.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.591+0000 7f8e6b400640 1 -- 172.21.15.69:0/2692376857 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8e7406bf50 con 0x7f8e7415b490 2024-04-24T08:15:37.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.804+0000 7f8e79a00640 1 -- 172.21.15.69:0/2692376857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8e7406bc70 con 0x7f8e7415b490 2024-04-24T08:15:37.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.804+0000 7f8e6b400640 1 -- 172.21.15.69:0/2692376857 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) ==== 80+0+0 (secure 0 0 0) 0x7f8e6c020070 con 0x7f8e7415b490 2024-04-24T08:15:37.806 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:15:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 -- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8e5006c8f0 msgr2=0x7f8e5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8e5006c8f0 0x7f8e5006edb0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8e64007fd0 tx=0x7f8e64007c20 comp rx=0 tx=0).stop 2024-04-24T08:15:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 -- 172.21.15.69:0/2692376857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7415b490 msgr2=0x7f8e740a9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7415b490 0x7f8e740a9430 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00c780 tx=0x7f8e6c00cc40 comp rx=0 tx=0).stop 2024-04-24T08:15:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 -- 172.21.15.69:0/2692376857 shutdown_connections 2024-04-24T08:15:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8e5006c8f0 0x7f8e5006edb0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e7415b490 0x7f8e740a9430 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8e740cfbe0 0x7f8e741f2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 --2- 172.21.15.69:0/2692376857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8e7406f2d0 0x7f8e741e58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 -- 172.21.15.69:0/2692376857 >> 172.21.15.69:0/2692376857 conn(0x7f8e74067d90 msgr2=0x7f8e7415e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 -- 172.21.15.69:0/2692376857 shutdown_connections 2024-04-24T08:15:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:37.807+0000 7f8e79a00640 1 -- 172.21.15.69:0/2692376857 wait complete. 2024-04-24T08:15:37.828 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:15:37.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:37.828 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:37.903 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[63328]: fuse finished with error 0 and tester_r 0 2024-04-24T08:15:37.907 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:15:44.009 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:15:44.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:44.010 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:15:44.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:44.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.261+0000 7fae7ec00640 1 Processor -- start 2024-04-24T08:15:44.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.261+0000 7fae7ec00640 1 -- start start 2024-04-24T08:15:44.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.262+0000 7fae7ec00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae7815d1d0 0x7fae780ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:44.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.262+0000 7fae7ec00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fae780b30f0 0x7fae780ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:44.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.262+0000 7fae7ec00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae7815d660 con 0x7fae78156c60 2024-04-24T08:15:44.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.262+0000 7fae7ec00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fae7806dc30 con 0x7fae780b30f0 2024-04-24T08:15:44.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.262+0000 7fae7ec00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fae7806df10 con 0x7fae7815d1d0 2024-04-24T08:15:44.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.263+0000 7fae7d800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fae78156c60 0x7fae7815cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32920/0 (socket says 172.21.15.69:32920) 2024-04-24T08:15:44.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.263+0000 7fae7d800640 1 -- 172.21.15.69:0/383671670 learned_addr learned my addr 172.21.15.69:0/383671670 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:44.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.263+0000 7fae7ce00640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae7815d1d0 0x7fae780ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:44.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.263+0000 7fae7e200640 1 --2- 172.21.15.69:0/383671670 >> v2:172.21.15.121:3300/0 conn(0x7fae780b30f0 0x7fae780ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.263+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2072339060 0 0) 0x7fae7815d660 con 0x7fae78156c60 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.263+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae5c003520 con 0x7fae78156c60 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 403984449 0 0) 0x7fae5c003520 con 0x7fae78156c60 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae7815d1d0 msgr2=0x7fae780ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae7815d1d0 0x7fae780ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 >> v2:172.21.15.121:3300/0 conn(0x7fae780b30f0 msgr2=0x7fae780ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 --2- 172.21.15.69:0/383671670 >> v2:172.21.15.121:3300/0 conn(0x7fae780b30f0 0x7fae780ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7806e1f0 con 0x7fae78156c60 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fae680019f0 con 0x7fae78156c60 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 >> v1:172.21.15.69:6789/0 conn(0x7fae78156c60 legacy=0x7fae7815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae5c0039b0 0x7fae5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae5c004800 0x7fae5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae5c005180 0x7fae5c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fae7806dc30 con 0x7fae5c005180 2024-04-24T08:15:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fae7806df10 con 0x7fae5c0039b0 2024-04-24T08:15:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fae5c004600 con 0x7fae5c004800 2024-04-24T08:15:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae7e200640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae5c005180 0x7fae5c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae7d800640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae5c0039b0 0x7fae5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae7e200640 1 -- 172.21.15.69:0/383671670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae5c004800 msgr2=0x7fae5c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.264+0000 7fae7e200640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae5c004800 0x7fae5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.265+0000 7fae7e200640 1 -- 172.21.15.69:0/383671670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae5c0039b0 msgr2=0x7fae5c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.265+0000 7fae7e200640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae5c0039b0 0x7fae5c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.265+0000 7fae7e200640 1 -- 172.21.15.69:0/383671670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5c003520 con 0x7fae5c005180 2024-04-24T08:15:44.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.266+0000 7fae7e200640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae5c005180 0x7fae5c00b9c0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fae780b34d0 tx=0x7fae5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7473d32cee8cd1b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:44.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.266+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae70036070 con 0x7fae5c005180 2024-04-24T08:15:44.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.266+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae70031400 con 0x7fae5c005180 2024-04-24T08:15:44.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.266+0000 7fae77e00640 1 -- 172.21.15.69:0/383671670 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae70009690 con 0x7fae5c005180 2024-04-24T08:15:44.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.266+0000 7fae7ec00640 1 -- 172.21.15.69:0/383671670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae5c005180 msgr2=0x7fae5c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:44.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.266+0000 7fae7ec00640 1 --2- 172.21.15.69:0/383671670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae5c005180 0x7fae5c00b9c0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fae780b34d0 tx=0x7fae5c005830 comp rx=0 tx=0).stop 2024-04-24T08:15:44.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.266+0000 7fae7e200640 1 -- 172.21.15.69:0/383671670 reap_dead start 2024-04-24T08:15:44.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.267+0000 7fae7ec00640 1 -- 172.21.15.69:0/383671670 shutdown_connections 2024-04-24T08:15:44.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.267+0000 7fae7ec00640 1 -- 172.21.15.69:0/383671670 >> 172.21.15.69:0/383671670 conn(0x7fae780d0160 msgr2=0x7fae780d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:44.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.267+0000 7fae7ec00640 1 -- 172.21.15.69:0/383671670 shutdown_connections 2024-04-24T08:15:44.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.267+0000 7fae7ec00640 1 -- 172.21.15.69:0/383671670 wait complete. 2024-04-24T08:15:44.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.268+0000 7fae7ec00640 1 Processor -- start 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.268+0000 7fae7ec00640 1 -- start start 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.269+0000 7fae7ec00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae780996f0 0x7fae780b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.269+0000 7fae7ec00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae78157a50 0x7fae78159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.269+0000 7fae7ec00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae780cca80 0x7fae780ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.269+0000 7fae7ec00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fae780ad0d0 con 0x7fae780996f0 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.269+0000 7fae7ec00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fae7815d940 con 0x7fae78157a50 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.269+0000 7fae7ec00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fae780ad250 con 0x7fae780cca80 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.270+0000 7fae7d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae780996f0 0x7fae780b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.270+0000 7fae7d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae780996f0 0x7fae780b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41540/0 (socket says 172.21.15.69:41540) 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.270+0000 7fae7d800640 1 -- 172.21.15.69:0/630983384 learned_addr learned my addr 172.21.15.69:0/630983384 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.270+0000 7fae7ce00640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae78157a50 0x7fae78159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.270+0000 7fae7d800640 1 -- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae780cca80 msgr2=0x7fae780ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.270+0000 7fae7d800640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae780cca80 0x7fae780ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.270+0000 7fae7d800640 1 -- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae78157a50 msgr2=0x7fae78159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:44.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.270+0000 7fae7d800640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae78157a50 0x7fae78159060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.270+0000 7fae7d800640 1 -- 172.21.15.69:0/630983384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae78099ad0 con 0x7fae780996f0 2024-04-24T08:15:44.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.270+0000 7fae7d800640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae780996f0 0x7fae780b41e0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fae6800b100 tx=0x7fae6800b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:44.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.271+0000 7fae75600640 1 -- 172.21.15.69:0/630983384 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68019070 con 0x7fae780996f0 2024-04-24T08:15:44.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.271+0000 7fae75600640 1 -- 172.21.15.69:0/630983384 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae680072d0 con 0x7fae780996f0 2024-04-24T08:15:44.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.271+0000 7fae75600640 1 -- 172.21.15.69:0/630983384 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae680076a0 con 0x7fae780996f0 2024-04-24T08:15:44.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.271+0000 7fae7ec00640 1 -- 172.21.15.69:0/630983384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae78157fc0 con 0x7fae780996f0 2024-04-24T08:15:44.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.271+0000 7fae7ec00640 1 -- 172.21.15.69:0/630983384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae78158580 con 0x7fae780996f0 2024-04-24T08:15:44.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.272+0000 7fae7ec00640 1 -- 172.21.15.69:0/630983384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae780694b0 con 0x7fae780996f0 2024-04-24T08:15:44.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.273+0000 7fae75600640 1 -- 172.21.15.69:0/630983384 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fae68007840 con 0x7fae780996f0 2024-04-24T08:15:44.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.274+0000 7fae75600640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae5806c920 0x7fae5806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:44.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.274+0000 7fae75600640 1 -- 172.21.15.69:0/630983384 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7fae6808c190 con 0x7fae780996f0 2024-04-24T08:15:44.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.274+0000 7fae7ce00640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae5806c920 0x7fae5806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:44.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.276+0000 7fae7ce00640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae5806c920 0x7fae5806ede0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fae64004770 tx=0x7fae64008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:44.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.279+0000 7fae75600640 1 -- 172.21.15.69:0/630983384 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae780694b0 con 0x7fae780996f0 2024-04-24T08:15:44.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.514+0000 7fae7ec00640 1 -- 172.21.15.69:0/630983384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fae7806f9e0 con 0x7fae780996f0 2024-04-24T08:15:44.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.516+0000 7fae75600640 1 -- 172.21.15.69:0/630983384 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+5180 (secure 0 0 0) 0x7fae6805a520 con 0x7fae780996f0 2024-04-24T08:15:44.517 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:44.517 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":7581,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/2529876410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2529876410},{"type":"v1","addr":"172.21.15.69:6833","nonce":2529876410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":55}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:15:26.487664+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7593},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7593":{"gid":7593,"name":"b","rank":0,"incarnation":57,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6837/1179683245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1179683245},{"type":"v1","addr":"172.21.15.121:6837","nonce":1179683245}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7593,"qdb_cluster":[7593]},"id":6}]} 2024-04-24T08:15:44.518 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 61 2024-04-24T08:15:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.519+0000 7fae7ec00640 1 -- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae5806c920 msgr2=0x7fae5806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.519+0000 7fae7ec00640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae5806c920 0x7fae5806ede0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fae64004770 tx=0x7fae64008040 comp rx=0 tx=0).stop 2024-04-24T08:15:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.519+0000 7fae7ec00640 1 -- 172.21.15.69:0/630983384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae780996f0 msgr2=0x7fae780b41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.519+0000 7fae7ec00640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae780996f0 0x7fae780b41e0 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fae6800b100 tx=0x7fae6800b5c0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.519+0000 7fae7ec00640 1 -- 172.21.15.69:0/630983384 shutdown_connections 2024-04-24T08:15:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.519+0000 7fae7ec00640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fae5806c920 0x7fae5806ede0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.519+0000 7fae7ec00640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fae780cca80 0x7fae780ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.519+0000 7fae7ec00640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae78157a50 0x7fae78159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.519+0000 7fae7ec00640 1 --2- 172.21.15.69:0/630983384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fae780996f0 0x7fae780b41e0 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:44.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.519+0000 7fae7ec00640 1 -- 172.21.15.69:0/630983384 >> 172.21.15.69:0/630983384 conn(0x7fae780d0160 msgr2=0x7fae780b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:44.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.520+0000 7fae7ec00640 1 -- 172.21.15.69:0/630983384 shutdown_connections 2024-04-24T08:15:44.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.520+0000 7fae7ec00640 1 -- 172.21.15.69:0/630983384 wait complete. 2024-04-24T08:15:44.539 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-04-24T08:15:44.792 INFO:teuthology.orchestra.run.smithi121.stdout:{"message":"","return_code":0} 2024-04-24T08:15:44.793 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-24T08:15:44.793 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:15:45.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.999+0000 7f6366000640 1 Processor -- start 2024-04-24T08:15:45.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:44.999+0000 7f6366000640 1 -- start start 2024-04-24T08:15:45.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.001+0000 7f6366000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6360156c60 0x7f636015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.001+0000 7f6366000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63600ad7e0 0x7f636015d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.001+0000 7f6366000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f636015d660 con 0x7f63600b30f0 2024-04-24T08:15:45.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.001+0000 7f6366000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f636006dc30 con 0x7f63600ad7e0 2024-04-24T08:15:45.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.001+0000 7f6366000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f636006df10 con 0x7f6360156c60 2024-04-24T08:15:45.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.001+0000 7f6365600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63600ad7e0 0x7f636015d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.001+0000 7f6365600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f63600ad7e0 0x7f636015d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41552/0 (socket says 172.21.15.69:41552) 2024-04-24T08:15:45.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.001+0000 7f6365600640 1 -- 172.21.15.69:0/2076773045 learned_addr learned my addr 172.21.15.69:0/2076773045 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:45.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.001+0000 7f6364c00640 1 --2- 172.21.15.69:0/2076773045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6360156c60 0x7f636015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.001+0000 7f6365600640 1 -- 172.21.15.69:0/2076773045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6360156c60 msgr2=0x7f636015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.002+0000 7f6365600640 1 --2- 172.21.15.69:0/2076773045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6360156c60 0x7f636015cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.002+0000 7f6365600640 1 -- 172.21.15.69:0/2076773045 >> v1:172.21.15.121:6789/0 conn(0x7f63600b30f0 legacy=0x7f63600ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.002+0000 7f6365600640 1 -- 172.21.15.69:0/2076773045 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636006e1f0 con 0x7f63600ad7e0 2024-04-24T08:15:45.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.002+0000 7f6365600640 1 --2- 172.21.15.69:0/2076773045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 0x7f636015d000 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f6358009e50 tx=0x7f635802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5faa12aaaeb84f28 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.003+0000 7f635f400640 1 -- 172.21.15.69:0/2076773045 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358036070 con 0x7f63600ad7e0 2024-04-24T08:15:45.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.003+0000 7f635f400640 1 -- 172.21.15.69:0/2076773045 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6358031600 con 0x7f63600ad7e0 2024-04-24T08:15:45.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.003+0000 7f635f400640 1 -- 172.21.15.69:0/2076773045 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63580319d0 con 0x7f63600ad7e0 2024-04-24T08:15:45.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.003+0000 7f6366000640 1 -- 172.21.15.69:0/2076773045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 msgr2=0x7f636015d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.003+0000 7f6366000640 1 --2- 172.21.15.69:0/2076773045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 0x7f636015d000 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f6358009e50 tx=0x7f635802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.004+0000 7f6366000640 1 -- 172.21.15.69:0/2076773045 shutdown_connections 2024-04-24T08:15:45.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.004+0000 7f6366000640 1 --2- 172.21.15.69:0/2076773045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 0x7f636015d000 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.004+0000 7f6366000640 1 --2- 172.21.15.69:0/2076773045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6360156c60 0x7f636015cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.004+0000 7f6366000640 1 -- 172.21.15.69:0/2076773045 >> 172.21.15.69:0/2076773045 conn(0x7f63600d0160 msgr2=0x7f63600d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:45.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.004+0000 7f6366000640 1 -- 172.21.15.69:0/2076773045 shutdown_connections 2024-04-24T08:15:45.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.004+0000 7f6366000640 1 -- 172.21.15.69:0/2076773045 wait complete. 2024-04-24T08:15:45.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.005+0000 7f6366000640 1 Processor -- start 2024-04-24T08:15:45.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.005+0000 7f6366000640 1 -- start start 2024-04-24T08:15:45.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.006+0000 7f6366000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 0x7f6360099820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.006+0000 7f6366000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63600b30f0 0x7f63601f4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.006+0000 7f6366000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6360156c60 0x7f6360072560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.006+0000 7f6366000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f636006e710 con 0x7f63600ad7e0 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.006+0000 7f6366000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f636015d940 con 0x7f6360156c60 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.006+0000 7f6366000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f636006e890 con 0x7f63600b30f0 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.007+0000 7f6364c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 0x7f6360099820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.007+0000 7f6364c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 0x7f6360099820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41562/0 (socket says 172.21.15.69:41562) 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.007+0000 7f6364c00640 1 -- 172.21.15.69:0/1034070717 learned_addr learned my addr 172.21.15.69:0/1034070717 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.007+0000 7f6364c00640 1 -- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63600b30f0 msgr2=0x7f63601f4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.007+0000 7f6364c00640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63600b30f0 0x7f63601f4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.007+0000 7f6364c00640 1 -- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6360156c60 msgr2=0x7f6360072560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.007+0000 7f6364c00640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6360156c60 0x7f6360072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.007+0000 7f6364c00640 1 -- 172.21.15.69:0/1034070717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6360099d60 con 0x7f63600ad7e0 2024-04-24T08:15:45.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.007+0000 7f6364c00640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 0x7f6360099820 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f635400c7f0 tx=0x7f635400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.008+0000 7f635cc00640 1 -- 172.21.15.69:0/1034070717 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354015070 con 0x7f63600ad7e0 2024-04-24T08:15:45.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.008+0000 7f635cc00640 1 -- 172.21.15.69:0/1034070717 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6354004220 con 0x7f63600ad7e0 2024-04-24T08:15:45.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.008+0000 7f635cc00640 1 -- 172.21.15.69:0/1034070717 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354007620 con 0x7f63600ad7e0 2024-04-24T08:15:45.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.008+0000 7f6366000640 1 -- 172.21.15.69:0/1034070717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63601f52c0 con 0x7f63600ad7e0 2024-04-24T08:15:45.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.008+0000 7f6366000640 1 -- 172.21.15.69:0/1034070717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6360071d60 con 0x7f63600ad7e0 2024-04-24T08:15:45.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.009+0000 7f6366000640 1 -- 172.21.15.69:0/1034070717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6360075d50 con 0x7f63600ad7e0 2024-04-24T08:15:45.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.010+0000 7f635cc00640 1 -- 172.21.15.69:0/1034070717 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f63540079d0 con 0x7f63600ad7e0 2024-04-24T08:15:45.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.011+0000 7f635cc00640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f634006c8f0 0x7f634006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.011+0000 7f635cc00640 1 -- 172.21.15.69:0/1034070717 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 5709+0+0 (secure 0 0 0) 0x7f635408bba0 con 0x7f63600ad7e0 2024-04-24T08:15:45.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.012+0000 7f635fe00640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f634006c8f0 0x7f634006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.013+0000 7f635fe00640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f634006c8f0 0x7f634006edb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6360153b20 tx=0x7f634c007990 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.016+0000 7f635cc00640 1 -- 172.21.15.69:0/1034070717 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6354059f30 con 0x7f63600ad7e0 2024-04-24T08:15:45.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.257+0000 7f6366000640 1 -- 172.21.15.69:0/1034070717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6360159f30 con 0x7f63600ad7e0 2024-04-24T08:15:45.260 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:15:45.258+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:15:45.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.262+0000 7f635cc00640 1 -- 172.21.15.69:0/1034070717 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v62) ==== 161+0+0 (secure 0 0 0) 0x7f6354059830 con 0x7f63600ad7e0 2024-04-24T08:15:45.264 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:15:45.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.265+0000 7f6366000640 1 -- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f634006c8f0 msgr2=0x7f634006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.265+0000 7f6366000640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f634006c8f0 0x7f634006edb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6360153b20 tx=0x7f634c007990 comp rx=0 tx=0).stop 2024-04-24T08:15:45.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.265+0000 7f6366000640 1 -- 172.21.15.69:0/1034070717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 msgr2=0x7f6360099820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.265+0000 7f6366000640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 0x7f6360099820 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f635400c7f0 tx=0x7f635400ccb0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.265+0000 7f6366000640 1 -- 172.21.15.69:0/1034070717 shutdown_connections 2024-04-24T08:15:45.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.265+0000 7f6366000640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f634006c8f0 0x7f634006edb0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.266+0000 7f6366000640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6360156c60 0x7f6360072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.266+0000 7f6366000640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f63600b30f0 0x7f63601f4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.266+0000 7f6366000640 1 --2- 172.21.15.69:0/1034070717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63600ad7e0 0x7f6360099820 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.266+0000 7f6366000640 1 -- 172.21.15.69:0/1034070717 >> 172.21.15.69:0/1034070717 conn(0x7f63600d0160 msgr2=0x7f63600afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:45.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.266+0000 7f6366000640 1 -- 172.21.15.69:0/1034070717 shutdown_connections 2024-04-24T08:15:45.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.266+0000 7f6366000640 1 -- 172.21.15.69:0/1034070717 wait complete. 2024-04-24T08:15:45.286 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 10000000000.00000000 2024-04-24T08:15:45.296 INFO:tasks.ceph.mds.b.smithi121.stderr: -4054> 2024-04-24T08:15:29.934+0000 7f1f79e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:15:45.326 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:15:45.338 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:15:45.377 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:15:45.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.392+0000 7f71ae249f40 1 Processor -- start 2024-04-24T08:15:45.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.392+0000 7f71ae249f40 1 -- start start 2024-04-24T08:15:45.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71ae249f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f6dbc33170 0x55f6dbcf40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71ae249f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc34cb0 0x55f6dbcf0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71ae249f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f6dbc676c0 0x55f6dbcf1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71ae249f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f6dbd0c020 con 0x55f6dbc34cb0 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71ae249f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f6dbbdbcc0 con 0x55f6dbc33170 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71ae249f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f6dbc80520 con 0x55f6dbc676c0 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71aaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc34cb0 0x55f6dbcf0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71aaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc34cb0 0x55f6dbcf0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41580/0 (socket says 172.21.15.69:41580) 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71aaa00640 1 -- 172.21.15.69:0/1075711923 learned_addr learned my addr 172.21.15.69:0/1075711923 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71abe00640 1 --2- 172.21.15.69:0/1075711923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f6dbc676c0 0x55f6dbcf1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.393+0000 7f71ab400640 1 --2- 172.21.15.69:0/1075711923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f6dbc33170 0x55f6dbcf40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.394+0000 7f71aaa00640 1 -- 172.21.15.69:0/1075711923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f6dbc676c0 msgr2=0x55f6dbcf1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.394+0000 7f71aaa00640 1 --2- 172.21.15.69:0/1075711923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f6dbc676c0 0x55f6dbcf1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.394+0000 7f71aaa00640 1 -- 172.21.15.69:0/1075711923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f6dbc33170 msgr2=0x55f6dbcf40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.394+0000 7f71aaa00640 1 --2- 172.21.15.69:0/1075711923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f6dbc33170 0x55f6dbcf40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.394+0000 7f71aaa00640 1 -- 172.21.15.69:0/1075711923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6dbcf53b0 con 0x55f6dbc34cb0 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.394+0000 7f71ab400640 1 --2- 172.21.15.69:0/1075711923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f6dbc33170 0x55f6dbcf40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:45.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.394+0000 7f71aaa00640 1 --2- 172.21.15.69:0/1075711923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc34cb0 0x55f6dbcf0c50 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f7198001d80 tx=0x7f7198001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc6c774c788e5d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.394+0000 7f71a9600640 1 -- 172.21.15.69:0/1075711923 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719800bd70 con 0x55f6dbc34cb0 2024-04-24T08:15:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.394+0000 7f71a9600640 1 -- 172.21.15.69:0/1075711923 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f719800f400 con 0x55f6dbc34cb0 2024-04-24T08:15:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.394+0000 7f71a9600640 1 -- 172.21.15.69:0/1075711923 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719800f7f0 con 0x55f6dbc34cb0 2024-04-24T08:15:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.395+0000 7f71ae249f40 1 -- 172.21.15.69:0/1075711923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc34cb0 msgr2=0x55f6dbcf0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.395+0000 7f71ae249f40 1 --2- 172.21.15.69:0/1075711923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc34cb0 0x55f6dbcf0c50 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f7198001d80 tx=0x7f7198001e70 comp rx=0 tx=0).stop 2024-04-24T08:15:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.395+0000 7f71ae249f40 1 -- 172.21.15.69:0/1075711923 shutdown_connections 2024-04-24T08:15:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.395+0000 7f71ae249f40 1 --2- 172.21.15.69:0/1075711923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc34cb0 0x55f6dbcf0c50 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.395+0000 7f71ae249f40 1 --2- 172.21.15.69:0/1075711923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f6dbc676c0 0x55f6dbcf1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.395+0000 7f71ae249f40 1 --2- 172.21.15.69:0/1075711923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f6dbc33170 0x55f6dbcf40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.395+0000 7f71ae249f40 1 -- 172.21.15.69:0/1075711923 >> 172.21.15.69:0/1075711923 conn(0x55f6dbc82100 msgr2=0x55f6dbcef230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:45.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.395+0000 7f71ae249f40 1 -- 172.21.15.69:0/1075711923 shutdown_connections 2024-04-24T08:15:45.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.396+0000 7f71ae249f40 1 -- 172.21.15.69:0/1075711923 wait complete. 2024-04-24T08:15:45.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.396+0000 7f71ae249f40 1 Processor -- start 2024-04-24T08:15:45.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.396+0000 7f71ae249f40 1 -- start start 2024-04-24T08:15:45.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.397+0000 7f71ae249f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f6dbc33170 0x55f6dbcfef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.397+0000 7f71ae249f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f6dbc34cb0 0x55f6dbd08ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.397+0000 7f71ae249f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc676c0 0x55f6dbd05840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.397+0000 7f71ae249f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f6dbcf5150 con 0x55f6dbc676c0 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.397+0000 7f71ae249f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f6dbcf0350 con 0x55f6dbc33170 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.397+0000 7f71ae249f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f6dbcf05a0 con 0x55f6dbc34cb0 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71abe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc676c0 0x55f6dbd05840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71ab400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f6dbc33170 0x55f6dbcfef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71abe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc676c0 0x55f6dbd05840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41588/0 (socket says 172.21.15.69:41588) 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71abe00640 1 -- 172.21.15.69:0/2983772119 learned_addr learned my addr 172.21.15.69:0/2983772119 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71aaa00640 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f6dbc34cb0 0x55f6dbd08ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71abe00640 1 -- 172.21.15.69:0/2983772119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f6dbc34cb0 msgr2=0x55f6dbd08ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71abe00640 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f6dbc34cb0 0x55f6dbd08ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71abe00640 1 -- 172.21.15.69:0/2983772119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f6dbc33170 msgr2=0x55f6dbcfef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71abe00640 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f6dbc33170 0x55f6dbcfef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71abe00640 1 -- 172.21.15.69:0/2983772119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6dbcff8f0 con 0x55f6dbc676c0 2024-04-24T08:15:45.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.398+0000 7f71abe00640 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc676c0 0x55f6dbd05840 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f71a40108e0 tx=0x7f71a4010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.399+0000 7f71a3e00640 1 -- 172.21.15.69:0/2983772119 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a4002e60 con 0x55f6dbc676c0 2024-04-24T08:15:45.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.399+0000 7f71a3e00640 1 -- 172.21.15.69:0/2983772119 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71a400e9b0 con 0x55f6dbc676c0 2024-04-24T08:15:45.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.399+0000 7f71a3e00640 1 -- 172.21.15.69:0/2983772119 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a400eda0 con 0x55f6dbc676c0 2024-04-24T08:15:45.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.399+0000 7f71ae249f40 1 -- 172.21.15.69:0/2983772119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f6dbcffbd0 con 0x55f6dbc676c0 2024-04-24T08:15:45.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.399+0000 7f71ae249f40 1 -- 172.21.15.69:0/2983772119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6dbd07790 con 0x55f6dbc676c0 2024-04-24T08:15:45.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.401+0000 7f71a3e00640 1 -- 172.21.15.69:0/2983772119 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f71a4011040 con 0x55f6dbc676c0 2024-04-24T08:15:45.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.402+0000 7f71a3e00640 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f718c06b300 0x7f718c06d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.402+0000 7f71a3e00640 1 -- 172.21.15.69:0/2983772119 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f71a408ae20 con 0x55f6dbc676c0 2024-04-24T08:15:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.402+0000 7f71ae249f40 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55f6dbd09410 0x55f6dbda59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.402+0000 7f71ae249f40 1 -- 172.21.15.69:0/2983772119 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:1 12.10 12:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e83) -- 0x55f6dbc66a00 con 0x55f6dbd09410 2024-04-24T08:15:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.402+0000 7f71ab400640 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f718c06b300 0x7f718c06d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.403+0000 7f71aaa00640 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55f6dbd09410 0x55f6dbda59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.403+0000 7f71ab400640 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f718c06b300 0x7f718c06d7c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f719c057000 tx=0x7f719c061040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.403+0000 7f71aaa00640 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55f6dbd09410 0x55f6dbda59f0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.407+0000 7f71aaa00640 1 -- 172.21.15.69:0/2983772119 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== osd_op_reply(1 10000000000.00000000 [delete] v83'11 uv11 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f719800f5b0 con 0x55f6dbd09410 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.407+0000 7f71ae249f40 1 -- 172.21.15.69:0/2983772119 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55f6dbd09410 msgr2=0x55f6dbda59f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.407+0000 7f71ae249f40 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55f6dbd09410 0x55f6dbda59f0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.408+0000 7f71ae249f40 1 -- 172.21.15.69:0/2983772119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f718c06b300 msgr2=0x7f718c06d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.408+0000 7f71ae249f40 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f718c06b300 0x7f718c06d7c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f719c057000 tx=0x7f719c061040 comp rx=0 tx=0).stop 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.408+0000 7f71ae249f40 1 -- 172.21.15.69:0/2983772119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc676c0 msgr2=0x55f6dbd05840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.408+0000 7f71ae249f40 1 --2- 172.21.15.69:0/2983772119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f6dbc676c0 0x55f6dbd05840 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f71a40108e0 tx=0x7f71a4010da0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.408+0000 7f71abe00640 1 -- 172.21.15.69:0/2983772119 reap_dead start 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.408+0000 7f71ae249f40 1 -- 172.21.15.69:0/2983772119 shutdown_connections 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.408+0000 7f71ae249f40 1 -- 172.21.15.69:0/2983772119 >> 172.21.15.69:0/2983772119 conn(0x55f6dbc82100 msgr2=0x55f6dbc824e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.408+0000 7f71ae249f40 1 -- 172.21.15.69:0/2983772119 shutdown_connections 2024-04-24T08:15:45.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.408+0000 7f71ae249f40 1 -- 172.21.15.69:0/2983772119 wait complete. 2024-04-24T08:15:45.414 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset 2024-04-24T08:15:45.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.495+0000 7fb8a580c280 1 -- start start 2024-04-24T08:15:45.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.495+0000 7fb8a580c280 1 Processor -- start 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.496+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4154000 0x564bd352d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.496+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4154800 0x564bd352cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.496+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4154400 0x564bd352db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.496+0000 7fb8a580c280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564bd41ac480 con 0x564bd4154800 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.496+0000 7fb8a580c280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564bd41ac600 con 0x564bd4154400 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.496+0000 7fb8a580c280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564bd41ac780 con 0x564bd4154000 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.496+0000 7fb8a2200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4154800 0x564bd352cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.496+0000 7fb8a2200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4154800 0x564bd352cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41598/0 (socket says 172.21.15.69:41598) 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.496+0000 7fb8a2200640 1 -- 172.21.15.69:0/2058960258 learned_addr learned my addr 172.21.15.69:0/2058960258 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a2c00640 1 --2- 172.21.15.69:0/2058960258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4154000 0x564bd352d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a3600640 1 --2- 172.21.15.69:0/2058960258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4154400 0x564bd352db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a2200640 1 -- 172.21.15.69:0/2058960258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4154000 msgr2=0x564bd352d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a2200640 1 --2- 172.21.15.69:0/2058960258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4154000 0x564bd352d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a2200640 1 -- 172.21.15.69:0/2058960258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4154400 msgr2=0x564bd352db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a2200640 1 --2- 172.21.15.69:0/2058960258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4154400 0x564bd352db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a2200640 1 -- 172.21.15.69:0/2058960258 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bd33f0780 con 0x564bd4154800 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a3600640 1 --2- 172.21.15.69:0/2058960258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4154400 0x564bd352db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a2200640 1 --2- 172.21.15.69:0/2058960258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4154800 0x564bd352cb00 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x564bd41e7350 tx=0x564bd4158120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b3c5356be3cb2b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb89fa00640 1 -- 172.21.15.69:0/2058960258 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564bd33e2d00 con 0x564bd4154800 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb89fa00640 1 -- 172.21.15.69:0/2058960258 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x564bd33e8700 con 0x564bd4154800 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb89fa00640 1 -- 172.21.15.69:0/2058960258 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564bd33e2680 con 0x564bd4154800 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a580c280 1 -- 172.21.15.69:0/2058960258 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bd33f0000 con 0x564bd4154800 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a580c280 1 -- 172.21.15.69:0/2058960258 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x564bd33f1a40 con 0x564bd4154800 2024-04-24T08:15:45.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.497+0000 7fb8a580c280 4 waiting for MDS map... 2024-04-24T08:15:45.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.498+0000 7fb89fa00640 1 -- 172.21.15.69:0/2058960258 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x564bd428a600 con 0x564bd4154800 2024-04-24T08:15:45.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.498+0000 7fb89fa00640 1 -- 172.21.15.69:0/2058960258 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 62) ==== 2885+0+0 (secure 0 0 0) 0x564bd3417180 con 0x564bd4154800 2024-04-24T08:15:45.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.498+0000 7fb8a580c280 4 Got MDS map 62 2024-04-24T08:15:45.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.498+0000 7fb8a580c280 10 main: JournalTool::main 2024-04-24T08:15:45.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.498+0000 7fb8a580c280 4 main: JournalTool: connecting to RADOS... 2024-04-24T08:15:45.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.499+0000 7fb8a580c280 1 Processor -- start 2024-04-24T08:15:45.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.499+0000 7fb8a580c280 1 -- start start 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4155c00 0x564bd352d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4287c00 0x564bd352c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd42ee000 0x564bd352e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a580c280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564bd341e180 con 0x564bd4155c00 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a580c280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564bd41acf00 con 0x564bd42ee000 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a580c280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564bd41ad080 con 0x564bd4287c00 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a3600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4155c00 0x564bd352d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a3600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4155c00 0x564bd352d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41602/0 (socket says 172.21.15.69:41602) 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a3600640 1 -- 172.21.15.69:0/3959146997 learned_addr learned my addr 172.21.15.69:0/3959146997 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a2200640 1 --2- 172.21.15.69:0/3959146997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd42ee000 0x564bd352e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a3600640 1 -- 172.21.15.69:0/3959146997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4287c00 msgr2=0x564bd352c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a3600640 1 --2- 172.21.15.69:0/3959146997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4287c00 0x564bd352c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a3600640 1 -- 172.21.15.69:0/3959146997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd42ee000 msgr2=0x564bd352e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a3600640 1 --2- 172.21.15.69:0/3959146997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd42ee000 0x564bd352e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.500+0000 7fb8a3600640 1 -- 172.21.15.69:0/3959146997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bd33f1860 con 0x564bd4155c00 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.501+0000 7fb8a3600640 1 --2- 172.21.15.69:0/3959146997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4155c00 0x564bd352d600 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x564bd429c690 tx=0x564bd429ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba84175f825af80 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.501+0000 7fb89d200640 1 -- 172.21.15.69:0/3959146997 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564bd33e2d00 con 0x564bd4155c00 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.501+0000 7fb89d200640 1 -- 172.21.15.69:0/3959146997 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x564bd33e8fc0 con 0x564bd4155c00 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.501+0000 7fb89d200640 1 -- 172.21.15.69:0/3959146997 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564bd33e2680 con 0x564bd4155c00 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.501+0000 7fb8a580c280 1 -- 172.21.15.69:0/3959146997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4155c00 msgr2=0x564bd352d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.501+0000 7fb8a580c280 1 --2- 172.21.15.69:0/3959146997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4155c00 0x564bd352d600 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x564bd429c690 tx=0x564bd429ecc0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.501+0000 7fb8a580c280 1 -- 172.21.15.69:0/3959146997 shutdown_connections 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.501+0000 7fb8a580c280 1 --2- 172.21.15.69:0/3959146997 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd42ee000 0x564bd352e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.501+0000 7fb8a580c280 1 --2- 172.21.15.69:0/3959146997 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4287c00 0x564bd352c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.501+0000 7fb8a580c280 1 --2- 172.21.15.69:0/3959146997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4155c00 0x564bd352d600 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.502+0000 7fb8a580c280 1 -- 172.21.15.69:0/3959146997 >> 172.21.15.69:0/3959146997 conn(0x564bd4155800 msgr2=0x564bd4143800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.502+0000 7fb8a580c280 1 -- 172.21.15.69:0/3959146997 shutdown_connections 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.502+0000 7fb8a580c280 1 -- 172.21.15.69:0/3959146997 wait complete. 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.502+0000 7fb8a580c280 1 Processor -- start 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.502+0000 7fb8a580c280 1 -- start start 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.503+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4155c00 0x564bd352f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.503+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4287c00 0x564bd352ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.503+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42fe000 0x564bd352e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.503+0000 7fb8a580c280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564bd41ad380 con 0x564bd42fe000 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.503+0000 7fb8a580c280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564bd41ad200 con 0x564bd4287c00 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.503+0000 7fb8a580c280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564bd41ad500 con 0x564bd4155c00 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.503+0000 7fb8a2200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42fe000 0x564bd352e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb8a2200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42fe000 0x564bd352e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41604/0 (socket says 172.21.15.69:41604) 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb8a2200640 1 -- 172.21.15.69:0/2664685066 learned_addr learned my addr 172.21.15.69:0/2664685066 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:45.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb8a3600640 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4155c00 0x564bd352f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb8a2c00640 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4287c00 0x564bd352ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb8a2200640 1 -- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4155c00 msgr2=0x564bd352f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb8a2200640 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4155c00 0x564bd352f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb8a2200640 1 -- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4287c00 msgr2=0x564bd352ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb8a2200640 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4287c00 0x564bd352ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb8a2200640 1 -- 172.21.15.69:0/2664685066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bd33f0780 con 0x564bd42fe000 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb8a2200640 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42fe000 0x564bd352e680 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x564bd429d3e0 tx=0x564bd4157f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.504+0000 7fb89b400640 1 -- 172.21.15.69:0/2664685066 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564bd33e2820 con 0x564bd42fe000 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.505+0000 7fb89b400640 1 -- 172.21.15.69:0/2664685066 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x564bd33e8c40 con 0x564bd42fe000 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.505+0000 7fb89b400640 1 -- 172.21.15.69:0/2664685066 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564bd33e2d00 con 0x564bd42fe000 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.505+0000 7fb8a580c280 1 -- 172.21.15.69:0/2664685066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564bd33f0000 con 0x564bd42fe000 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.505+0000 7fb8a580c280 1 -- 172.21.15.69:0/2664685066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bd33f1680 con 0x564bd42fe000 2024-04-24T08:15:45.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.505+0000 7fb8a580c280 4 main: JournalTool: resolving pool 12 2024-04-24T08:15:45.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.506+0000 7fb89b400640 1 -- 172.21.15.69:0/2664685066 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x564bd42eec00 con 0x564bd42fe000 2024-04-24T08:15:45.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.507+0000 7fb89b400640 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x564bd4305000 0x564bd352f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.507+0000 7fb89b400640 1 -- 172.21.15.69:0/2664685066 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x564bd41c0a00 con 0x564bd42fe000 2024-04-24T08:15:45.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.507+0000 7fb8a580c280 4 main: JournalTool: creating IoCtx.. 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.507+0000 7fb8a580c280 4 main: Executing for rank 0 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.508+0000 7fb8a3600640 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x564bd4305000 0x564bd352f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.508+0000 7fb8a580c280 1 -- start start 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.508+0000 7fb8a580c280 1 Processor -- start 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.509+0000 7fb8a3600640 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x564bd4305000 0x564bd352f700 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x564bd429c690 tx=0x564bd429ec00 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.509+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd42fe800 0x564bd352e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.509+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd42fec00 0x564bd352c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.509+0000 7fb8a580c280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42ff000 0x564bd352d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.509+0000 7fb8a580c280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564bd41ad980 con 0x564bd42ff000 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.509+0000 7fb8a580c280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564bd41adb00 con 0x564bd42fe800 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.509+0000 7fb8a580c280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564bd41adc80 con 0x564bd42fec00 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42ff000 0x564bd352d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42ff000 0x564bd352d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41620/0 (socket says 172.21.15.69:41620) 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2200640 1 -- 172.21.15.69:0/568543880 learned_addr learned my addr 172.21.15.69:0/568543880 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:45.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a3600640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd42fe800 0x564bd352e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2c00640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd42fec00 0x564bd352c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2200640 1 -- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd42fec00 msgr2=0x564bd352c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2200640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd42fec00 0x564bd352c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2200640 1 -- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd42fe800 msgr2=0x564bd352e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2200640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd42fe800 0x564bd352e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2200640 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bd33f03c0 con 0x564bd42ff000 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2c00640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd42fec00 0x564bd352c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.510+0000 7fb8a2200640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42ff000 0x564bd352d600 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x564bd433b440 tx=0x564bd373b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ec5e17567931027 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb897800640 1 -- 172.21.15.69:0/568543880 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564bd36c3520 con 0x564bd42ff000 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb897800640 1 -- 172.21.15.69:0/568543880 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x564bd33e81c0 con 0x564bd42ff000 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb897800640 1 -- 172.21.15.69:0/568543880 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x564bd36c36c0 con 0x564bd42ff000 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bd33f1a40 con 0x564bd42ff000 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x564bd33f0780 con 0x564bd42ff000 2024-04-24T08:15:45.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb8a580c280 4 waiting for MDS map... 2024-04-24T08:15:45.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb897800640 1 -- 172.21.15.69:0/568543880 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x564bd3794600 con 0x564bd42ff000 2024-04-24T08:15:45.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb897800640 1 -- 172.21.15.69:0/568543880 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 62) ==== 2885+0+0 (secure 0 0 0) 0x564bd3417500 con 0x564bd42ff000 2024-04-24T08:15:45.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb8a580c280 4 Got MDS map 62 2024-04-24T08:15:45.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x564bd42ff400 0x564bd37b2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.511+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:1 12.f 12:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564bd42ff800 con 0x564bd42ff400 2024-04-24T08:15:45.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.512+0000 7fb8a3600640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x564bd42ff400 0x564bd37b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.513+0000 7fb8a3600640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x564bd42ff400 0x564bd37b2000 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=b66d93ebce614e89 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.520+0000 7fb8a3600640 1 -- 172.21.15.69:0/568543880 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x564bd4146000 con 0x564bd42ff400 2024-04-24T08:15:45.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.521+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x564bd42ffc00 0x564bd37b2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.521+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:2 12.14 12:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e83) -- 0x564bd37ac400 con 0x564bd42ffc00 2024-04-24T08:15:45.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.522+0000 7fb8a2c00640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x564bd42ffc00 0x564bd37b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.523+0000 7fb8a2c00640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x564bd42ffc00 0x564bd37b2580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=d6c8a3e441e0ba5b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.523+0000 7fb8a2c00640 1 -- 172.21.15.69:0/568543880 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv20 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x564bd4146500 con 0x564bd42ffc00 2024-04-24T08:15:45.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.523+0000 7fb895a00640 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:3 12.14 12:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83) -- 0x564bd37ba000 con 0x564bd42ffc00 2024-04-24T08:15:45.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.524+0000 7fb895a00640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x564bd37ba400 0x564bd37b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.524+0000 7fb895a00640 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:4 12.1b 12:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e83) -- 0x564bd37ba800 con 0x564bd37ba400 2024-04-24T08:15:45.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.524+0000 7fb8a2c00640 1 -- 172.21.15.69:0/568543880 <== osd.5 v2:172.21.15.121:6822/2187246263 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv15 ondisk = 0) ==== 156+0+16 (crc 0 0 0) 0x564bd4146500 con 0x564bd42ffc00 2024-04-24T08:15:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.524+0000 7fb8a3600640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x564bd37ba400 0x564bd37b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.525+0000 7fb8a3600640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x564bd37ba400 0x564bd37b4680 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=4caf9c6487cbf7cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.533+0000 7fb8a3600640 1 -- 172.21.15.69:0/568543880 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564bd4146000 con 0x564bd37ba400 2024-04-24T08:15:45.534 INFO:teuthology.orchestra.run.smithi069.stdout:old journal was 4209834~892 2024-04-24T08:15:45.534 INFO:teuthology.orchestra.run.smithi069.stdout:new journal start will be 8388608 (4177882 bytes past old end) 2024-04-24T08:15:45.534 INFO:teuthology.orchestra.run.smithi069.stdout:writing journal head 2024-04-24T08:15:45.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.533+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:5 12.14 12:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x564bd37ac800 con 0x564bd42ffc00 2024-04-24T08:15:45.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a2c00640 1 -- 172.21.15.69:0/568543880 <== osd.5 v2:172.21.15.121:6822/2187246263 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v83'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x564bd4146500 con 0x564bd42ffc00 2024-04-24T08:15:45.537 INFO:teuthology.orchestra.run.smithi069.stdout:writing EResetJournal entry 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:6 12.1b 12:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x564bd37acc00 con 0x564bd37ba400 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x564bd37ad000 0x564bd37b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:7 12.b 12:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x564bd37ad400 con 0x564bd37ad000 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:8 12.1f 12:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x564bd37ad800 con 0x564bd42ff400 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:9 12.1a 12:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x564bd37adc00 con 0x564bd37ba400 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:10 12.1e 12:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x564bd37ba800 con 0x564bd37ba400 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x564bd37ba000 0x564bd37b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:11 12.1d 12:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x564bd37bac00 con 0x564bd37ba000 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a2c00640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x564bd37ad000 0x564bd37b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.536+0000 7fb8a2200640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x564bd37ba000 0x564bd37b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.537+0000 7fb8a2c00640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x564bd37ad000 0x564bd37b2b00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=bc6902bc32c40e71 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.537+0000 7fb8a2200640 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x564bd37ba000 0x564bd37b3080 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=745bb83511c81deb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.539+0000 7fb8a3600640 1 -- 172.21.15.69:0/568543880 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== osd_op_reply(6 200.00000002 [delete] v83'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564bd4146000 con 0x564bd37ba400 2024-04-24T08:15:45.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.540+0000 7fb8a3600640 1 -- 172.21.15.69:0/568543880 <== osd.6 v2:172.21.15.121:6807/3068198325 2 ==== osd_op_reply(8 200.00000004 [delete] v83'7 uv6 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564bd4146000 con 0x564bd42ff400 2024-04-24T08:15:45.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.540+0000 7fb8a3600640 1 -- 172.21.15.69:0/568543880 <== osd.4 v2:172.21.15.121:6800/2274643300 3 ==== osd_op_reply(10 200.00000006 [delete] v83'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564bd4146000 con 0x564bd37ba400 2024-04-24T08:15:45.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.540+0000 7fb8a3600640 1 -- 172.21.15.69:0/568543880 <== osd.4 v2:172.21.15.121:6800/2274643300 4 ==== osd_op_reply(9 200.00000005 [delete] v83'2 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564bd4146000 con 0x564bd37ba400 2024-04-24T08:15:45.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.541+0000 7fb8a2c00640 1 -- 172.21.15.69:0/568543880 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(7 200.00000003 [delete] v83'4 uv3 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564bd4146500 con 0x564bd37ad000 2024-04-24T08:15:45.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.541+0000 7fb8a2200640 1 -- 172.21.15.69:0/568543880 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== osd_op_reply(11 200.00000007 [delete] v83'3 uv2 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x564bd4146780 con 0x564bd37ba000 2024-04-24T08:15:45.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.541+0000 7fb895a00640 1 -- 172.21.15.69:0/568543880 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:12 12.1b 12:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) -- 0x564bd37bbc00 con 0x564bd37ba400 2024-04-24T08:15:45.545 INFO:teuthology.orchestra.run.smithi069.stdout:done 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a3600640 1 -- 172.21.15.69:0/568543880 <== osd.4 v2:172.21.15.121:6800/2274643300 5 ==== osd_op_reply(12 200.00000002 [write 0~48] v83'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x564bd4146000 con 0x564bd37ba400 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x564bd37ba000 msgr2=0x564bd37b3080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x564bd37ba000 0x564bd37b3080 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x564bd37ad000 msgr2=0x564bd37b2b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x564bd37ad000 0x564bd37b2b00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x564bd37ba400 msgr2=0x564bd37b4680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x564bd37ba400 0x564bd37b4680 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x564bd42ffc00 msgr2=0x564bd37b2580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x564bd42ffc00 0x564bd37b2580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a2c00640 1 -- 172.21.15.69:0/568543880 reap_dead start 2024-04-24T08:15:45.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x564bd42ff400 msgr2=0x564bd37b2000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x564bd42ff400 0x564bd37b2000 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42ff000 msgr2=0x564bd352d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42ff000 0x564bd352d600 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x564bd433b440 tx=0x564bd373b8c0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 shutdown_connections 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x564bd42ff400 0x564bd37b2000 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x564bd42ffc00 0x564bd37b2580 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 --2- 172.21.15.69:0/568543880 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42ff000 0x564bd352d600 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 >> 172.21.15.69:0/568543880 conn(0x564bd42fe400 msgr2=0x564bd3782000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 shutdown_connections 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.544+0000 7fb8a580c280 1 -- 172.21.15.69:0/568543880 wait complete. 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 -- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x564bd4305000 msgr2=0x564bd352f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x564bd4305000 0x564bd352f700 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x564bd429c690 tx=0x564bd429ec00 comp rx=0 tx=0).stop 2024-04-24T08:15:45.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 -- 172.21.15.69:0/2664685066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42fe000 msgr2=0x564bd352e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42fe000 0x564bd352e680 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x564bd429d3e0 tx=0x564bd4157f80 comp rx=0 tx=0).stop 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 -- 172.21.15.69:0/2664685066 shutdown_connections 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x564bd4305000 0x564bd352f700 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd42fe000 0x564bd352e680 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4287c00 0x564bd352ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 --2- 172.21.15.69:0/2664685066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4155c00 0x564bd352f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 -- 172.21.15.69:0/2664685066 >> 172.21.15.69:0/2664685066 conn(0x564bd4155800 msgr2=0x564bd4143800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 -- 172.21.15.69:0/2664685066 shutdown_connections 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.545+0000 7fb8a580c280 1 -- 172.21.15.69:0/2664685066 wait complete. 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.546+0000 7fb8a580c280 1 -- 172.21.15.69:0/2058960258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4154800 msgr2=0x564bd352cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.546+0000 7fb8a580c280 1 --2- 172.21.15.69:0/2058960258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4154800 0x564bd352cb00 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x564bd41e7350 tx=0x564bd4158120 comp rx=0 tx=0).stop 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.546+0000 7fb8a580c280 1 -- 172.21.15.69:0/2058960258 shutdown_connections 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.546+0000 7fb8a580c280 1 --2- 172.21.15.69:0/2058960258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564bd4154800 0x564bd352cb00 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.546+0000 7fb8a580c280 1 --2- 172.21.15.69:0/2058960258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564bd4154400 0x564bd352db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.546+0000 7fb8a580c280 1 --2- 172.21.15.69:0/2058960258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564bd4154000 0x564bd352d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.546+0000 7fb8a580c280 1 -- 172.21.15.69:0/2058960258 >> 172.21.15.69:0/2058960258 conn(0x564bd33bd000 msgr2=0x564bd4143000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.546+0000 7fb8a580c280 1 -- 172.21.15.69:0/2058960258 shutdown_connections 2024-04-24T08:15:45.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.546+0000 7fb8a580c280 1 -- 172.21.15.69:0/2058960258 wait complete. 2024-04-24T08:15:45.553 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset'] in time 0:00:00.139206, result: old journal was 4209834~892 new journal start will be 8388608 (4177882 bytes past old end) writing journal head writing EResetJournal entry done 2024-04-24T08:15:45.553 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-24T08:15:45.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.746+0000 7f6259a00640 1 Processor -- start 2024-04-24T08:15:45.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.746+0000 7f6259a00640 1 -- start start 2024-04-24T08:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.747+0000 7f6259a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f625415f4b0 0x7f62540b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.747+0000 7f6259a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f625415f890 0x7f62540732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.747+0000 7f6259a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6254073820 0x7f62540b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.747+0000 7f6259a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6254073cc0 con 0x7f625415f890 2024-04-24T08:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.747+0000 7f6259a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6254073fa0 con 0x7f625415f4b0 2024-04-24T08:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.747+0000 7f6259a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62540b3300 con 0x7f6254073820 2024-04-24T08:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.748+0000 7f624a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f625415f890 0x7f62540732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.748+0000 7f624a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f625415f890 0x7f62540732e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41628/0 (socket says 172.21.15.69:41628) 2024-04-24T08:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.748+0000 7f624a000640 1 -- 172.21.15.69:0/3737038067 learned_addr learned my addr 172.21.15.69:0/3737038067 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.748+0000 7f6252a00640 1 --2- 172.21.15.69:0/3737038067 >> v2:172.21.15.121:3300/0 conn(0x7f625415f4b0 0x7f62540b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.748+0000 7f6253400640 1 --2- 172.21.15.69:0/3737038067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6254073820 0x7f62540b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.748+0000 7f624a000640 1 -- 172.21.15.69:0/3737038067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6254073820 msgr2=0x7f62540b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.748+0000 7f624a000640 1 --2- 172.21.15.69:0/3737038067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6254073820 0x7f62540b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.748+0000 7f624a000640 1 -- 172.21.15.69:0/3737038067 >> v2:172.21.15.121:3300/0 conn(0x7f625415f4b0 msgr2=0x7f62540b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:45.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.748+0000 7f624a000640 1 --2- 172.21.15.69:0/3737038067 >> v2:172.21.15.121:3300/0 conn(0x7f625415f4b0 0x7f62540b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.748+0000 7f624a000640 1 -- 172.21.15.69:0/3737038067 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62540b35e0 con 0x7f625415f890 2024-04-24T08:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.749+0000 7f624a000640 1 --2- 172.21.15.69:0/3737038067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f890 0x7f62540732e0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f62540cde60 tx=0x7f623c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6120a51bc7ee7852 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6252000640 1 -- 172.21.15.69:0/3737038067 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c037070 con 0x7f625415f890 2024-04-24T08:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6252000640 1 -- 172.21.15.69:0/3737038067 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f623c032440 con 0x7f625415f890 2024-04-24T08:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6252000640 1 -- 172.21.15.69:0/3737038067 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c032810 con 0x7f625415f890 2024-04-24T08:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6259a00640 1 -- 172.21.15.69:0/3737038067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f890 msgr2=0x7f62540732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6259a00640 1 --2- 172.21.15.69:0/3737038067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f890 0x7f62540732e0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f62540cde60 tx=0x7f623c02a1e0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6259a00640 1 -- 172.21.15.69:0/3737038067 shutdown_connections 2024-04-24T08:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6259a00640 1 --2- 172.21.15.69:0/3737038067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6254073820 0x7f62540b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6259a00640 1 --2- 172.21.15.69:0/3737038067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f890 0x7f62540732e0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6259a00640 1 --2- 172.21.15.69:0/3737038067 >> v2:172.21.15.121:3300/0 conn(0x7f625415f4b0 0x7f62540b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6259a00640 1 -- 172.21.15.69:0/3737038067 >> 172.21.15.69:0/3737038067 conn(0x7f62540d0350 msgr2=0x7f62540d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:45.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.750+0000 7f6259a00640 1 -- 172.21.15.69:0/3737038067 shutdown_connections 2024-04-24T08:15:45.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.751+0000 7f6259a00640 1 -- 172.21.15.69:0/3737038067 wait complete. 2024-04-24T08:15:45.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.752+0000 7f6259a00640 1 Processor -- start 2024-04-24T08:15:45.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.752+0000 7f6259a00640 1 -- start start 2024-04-24T08:15:45.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.753+0000 7f6259a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6254073820 0x7f625413fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.753+0000 7f6259a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f4b0 0x7f625406ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.753+0000 7f6259a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f625415f890 0x7f62540adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.753+0000 7f6259a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f625415fec0 con 0x7f625415f4b0 2024-04-24T08:15:45.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.753+0000 7f6259a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f625415fd40 con 0x7f625415f890 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.753+0000 7f6259a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62540b4be0 con 0x7f6254073820 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.753+0000 7f624a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f4b0 0x7f625406ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.754+0000 7f624a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f4b0 0x7f625406ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41634/0 (socket says 172.21.15.69:41634) 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.754+0000 7f624a000640 1 -- 172.21.15.69:0/770905026 learned_addr learned my addr 172.21.15.69:0/770905026 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.754+0000 7f6252a00640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6254073820 0x7f625413fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.754+0000 7f6253400640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f625415f890 0x7f62540adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.754+0000 7f624a000640 1 -- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6254073820 msgr2=0x7f625413fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.754+0000 7f624a000640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6254073820 0x7f625413fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.754+0000 7f624a000640 1 -- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f625415f890 msgr2=0x7f62540adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.754+0000 7f624a000640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f625415f890 0x7f62540adf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:45.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.754+0000 7f624a000640 1 -- 172.21.15.69:0/770905026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6254143c90 con 0x7f625415f4b0 2024-04-24T08:15:45.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.754+0000 7f624a000640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f4b0 0x7f625406ec50 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f623c000c00 tx=0x7f623c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.755+0000 7f624b400640 1 -- 172.21.15.69:0/770905026 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c041020 con 0x7f625415f4b0 2024-04-24T08:15:45.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.755+0000 7f624b400640 1 -- 172.21.15.69:0/770905026 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f623c0306c0 con 0x7f625415f4b0 2024-04-24T08:15:45.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.755+0000 7f624b400640 1 -- 172.21.15.69:0/770905026 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c037040 con 0x7f625415f4b0 2024-04-24T08:15:45.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.755+0000 7f6259a00640 1 -- 172.21.15.69:0/770905026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6254140d80 con 0x7f625415f4b0 2024-04-24T08:15:45.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.755+0000 7f6259a00640 1 -- 172.21.15.69:0/770905026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62541407c0 con 0x7f625415f4b0 2024-04-24T08:15:45.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.756+0000 7f6259a00640 1 -- 172.21.15.69:0/770905026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6254068a80 con 0x7f625415f4b0 2024-04-24T08:15:45.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.757+0000 7f624b400640 1 -- 172.21.15.69:0/770905026 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f623c044020 con 0x7f625415f4b0 2024-04-24T08:15:45.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.758+0000 7f624b400640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f623006c8f0 0x7f623006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:45.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.758+0000 7f624b400640 1 -- 172.21.15.69:0/770905026 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f623c0ac370 con 0x7f625415f4b0 2024-04-24T08:15:45.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.759+0000 7f6252a00640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f623006c8f0 0x7f623006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:45.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.760+0000 7f6252a00640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f623006c8f0 0x7f623006edb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6244006fd0 tx=0x7f6244008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:45.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:45.764+0000 7f624b400640 1 -- 172.21.15.69:0/770905026 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f623c04b030 con 0x7f625415f4b0 2024-04-24T08:15:46.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.009+0000 7f6259a00640 1 -- 172.21.15.69:0/770905026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f62540afb20 con 0x7f625415f4b0 2024-04-24T08:15:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.266+0000 7f624b400640 1 -- 172.21.15.69:0/770905026 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v63) ==== 162+0+0 (secure 0 0 0) 0x7f623c07a7d0 con 0x7f625415f4b0 2024-04-24T08:15:46.268 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-24T08:15:46.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 -- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f623006c8f0 msgr2=0x7f623006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f623006c8f0 0x7f623006edb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6244006fd0 tx=0x7f6244008040 comp rx=0 tx=0).stop 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 -- 172.21.15.69:0/770905026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f4b0 msgr2=0x7f625406ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f4b0 0x7f625406ec50 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f623c000c00 tx=0x7f623c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 -- 172.21.15.69:0/770905026 shutdown_connections 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f623006c8f0 0x7f623006edb0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f625415f4b0 0x7f625406ec50 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f625415f890 0x7f62540adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 --2- 172.21.15.69:0/770905026 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6254073820 0x7f625413fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 -- 172.21.15.69:0/770905026 >> 172.21.15.69:0/770905026 conn(0x7f62540d0350 msgr2=0x7f62540b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 -- 172.21.15.69:0/770905026 shutdown_connections 2024-04-24T08:15:46.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.270+0000 7f6259a00640 1 -- 172.21.15.69:0/770905026 wait complete. 2024-04-24T08:15:46.296 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.479+0000 7f41ed400640 1 Processor -- start 2024-04-24T08:15:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.479+0000 7f41ed400640 1 -- start start 2024-04-24T08:15:46.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.480+0000 7f41ed400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f41e815d1d0 0x7f41e80ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:46.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.480+0000 7f41ed400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41e815d660 con 0x7f41e80b30f0 2024-04-24T08:15:46.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.480+0000 7f41ed400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41e806dc30 con 0x7f41e8156c60 2024-04-24T08:15:46.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.480+0000 7f41ed400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f41e806df10 con 0x7f41e815d1d0 2024-04-24T08:15:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.481+0000 7f41eca00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f41e80b30f0 0x7f41e806e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32942/0 (socket says 172.21.15.69:32942) 2024-04-24T08:15:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.481+0000 7f41eca00640 1 -- 172.21.15.69:0/2092553458 learned_addr learned my addr 172.21.15.69:0/2092553458 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.481+0000 7f41df400640 1 --2- 172.21.15.69:0/2092553458 >> v2:172.21.15.121:3300/0 conn(0x7f41e815d1d0 0x7f41e80ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.481+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3052728762 0 0) 0x7f41e815d660 con 0x7f41e80b30f0 2024-04-24T08:15:46.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.481+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41cc003520 con 0x7f41e80b30f0 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.481+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1737586528 0 0) 0x7f41cc003520 con 0x7f41e80b30f0 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 >> v2:172.21.15.121:3300/0 conn(0x7f41e815d1d0 msgr2=0x7f41e80ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 --2- 172.21.15.69:0/2092553458 >> v2:172.21.15.121:3300/0 conn(0x7f41e815d1d0 0x7f41e80ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 >> v1:172.21.15.121:6789/0 conn(0x7f41e8156c60 legacy=0x7f41e815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e806e1f0 con 0x7f41e80b30f0 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f41e0001670 con 0x7f41e80b30f0 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 >> v1:172.21.15.69:6789/0 conn(0x7f41e80b30f0 legacy=0x7f41e806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 --2- 172.21.15.69:0/2092553458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41cc0039b0 0x7f41cc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 --2- 172.21.15.69:0/2092553458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41cc004800 0x7f41cc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 --2- 172.21.15.69:0/2092553458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc0051c0 0x7f41cc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f41e806df10 con 0x7f41cc004800 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc00c060 con 0x7f41cc0051c0 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f41cc00c260 con 0x7f41cc0039b0 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41df400640 1 --2- 172.21.15.69:0/2092553458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41cc004800 0x7f41cc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:46.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41df400640 1 -- 172.21.15.69:0/2092553458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41cc0039b0 msgr2=0x7f41cc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41eca00640 1 --2- 172.21.15.69:0/2092553458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc0051c0 0x7f41cc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41df400640 1 --2- 172.21.15.69:0/2092553458 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41cc0039b0 0x7f41cc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41df400640 1 -- 172.21.15.69:0/2092553458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc0051c0 msgr2=0x7f41cc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.482+0000 7f41df400640 1 --2- 172.21.15.69:0/2092553458 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc0051c0 0x7f41cc00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.483+0000 7f41eca00640 1 -- 172.21.15.69:0/2092553458 reap_dead start 2024-04-24T08:15:46.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.483+0000 7f41df400640 1 -- 172.21.15.69:0/2092553458 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc003520 con 0x7f41cc004800 2024-04-24T08:15:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.483+0000 7f41df400640 1 --2- 172.21.15.69:0/2092553458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41cc004800 0x7f41cc004c80 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009ac0 tx=0x7f41cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f846fba01763466 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.484+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d0036070 con 0x7f41cc004800 2024-04-24T08:15:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.484+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d0031400 con 0x7f41cc004800 2024-04-24T08:15:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.484+0000 7f41e7400640 1 -- 172.21.15.69:0/2092553458 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d0009690 con 0x7f41cc004800 2024-04-24T08:15:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.484+0000 7f41ed400640 1 -- 172.21.15.69:0/2092553458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41cc004800 msgr2=0x7f41cc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.484+0000 7f41ed400640 1 --2- 172.21.15.69:0/2092553458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41cc004800 0x7f41cc004c80 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009ac0 tx=0x7f41cc005830 comp rx=0 tx=0).stop 2024-04-24T08:15:46.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.484+0000 7f41ed400640 1 -- 172.21.15.69:0/2092553458 shutdown_connections 2024-04-24T08:15:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.484+0000 7f41ed400640 1 --2- 172.21.15.69:0/2092553458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41cc004800 0x7f41cc004c80 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.484+0000 7f41ed400640 1 -- 172.21.15.69:0/2092553458 >> 172.21.15.69:0/2092553458 conn(0x7f41e80d0160 msgr2=0x7f41e80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.485+0000 7f41ed400640 1 -- 172.21.15.69:0/2092553458 shutdown_connections 2024-04-24T08:15:46.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.485+0000 7f41ed400640 1 -- 172.21.15.69:0/2092553458 wait complete. 2024-04-24T08:15:46.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.486+0000 7f41ed400640 1 Processor -- start 2024-04-24T08:15:46.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.486+0000 7f41ed400640 1 -- start start 2024-04-24T08:15:46.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.487+0000 7f41ed400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc004800 0x7f41e80cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:46.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.487+0000 7f41ed400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8155b80 0x7f41e8145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.487+0000 7f41ed400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41e8155340 0x7f41e8072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.487+0000 7f41ed400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f41e815cac0 con 0x7f41e8155b80 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.487+0000 7f41ed400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f41e815d940 con 0x7f41cc004800 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.487+0000 7f41ed400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f41e815cc40 con 0x7f41e8155340 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41df400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8155b80 0x7f41e8145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41df400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8155b80 0x7f41e8145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41646/0 (socket says 172.21.15.69:41646) 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41df400640 1 -- 172.21.15.69:0/3383585905 learned_addr learned my addr 172.21.15.69:0/3383585905 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41eca00640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41e8155340 0x7f41e8072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41e7e00640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc004800 0x7f41e80cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41df400640 1 -- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41e8155340 msgr2=0x7f41e8072f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41df400640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41e8155340 0x7f41e8072f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41df400640 1 -- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc004800 msgr2=0x7f41e80cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41df400640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc004800 0x7f41e80cba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41df400640 1 -- 172.21.15.69:0/3383585905 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e8144f50 con 0x7f41e8155b80 2024-04-24T08:15:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.488+0000 7f41df400640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8155b80 0x7f41e8145710 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009be0 tx=0x7f41d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.489+0000 7f41e4c00640 1 -- 172.21.15.69:0/3383585905 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d003f020 con 0x7f41e8155b80 2024-04-24T08:15:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.489+0000 7f41e4c00640 1 -- 172.21.15.69:0/3383585905 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41d0037040 con 0x7f41e8155b80 2024-04-24T08:15:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.489+0000 7f41ed400640 1 -- 172.21.15.69:0/3383585905 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e8145c50 con 0x7f41e8155b80 2024-04-24T08:15:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.489+0000 7f41e4c00640 1 -- 172.21.15.69:0/3383585905 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d0036070 con 0x7f41e8155b80 2024-04-24T08:15:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.489+0000 7f41ed400640 1 -- 172.21.15.69:0/3383585905 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e80ca270 con 0x7f41e8155b80 2024-04-24T08:15:46.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.490+0000 7f41ed400640 1 -- 172.21.15.69:0/3383585905 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41e80694b0 con 0x7f41e8155b80 2024-04-24T08:15:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.491+0000 7f41e4c00640 1 -- 172.21.15.69:0/3383585905 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f41d0042020 con 0x7f41e8155b80 2024-04-24T08:15:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.492+0000 7f41e4c00640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f41c406c920 0x7f41c406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:46.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.492+0000 7f41e4c00640 1 -- 172.21.15.69:0/3383585905 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f41d00abce0 con 0x7f41e8155b80 2024-04-24T08:15:46.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.492+0000 7f41e7e00640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f41c406c920 0x7f41c406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:46.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.494+0000 7f41e7e00640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f41c406c920 0x7f41c406ede0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f41d80012a0 tx=0x7f41d8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:46.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.496+0000 7f41e4c00640 1 -- 172.21.15.69:0/3383585905 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41e80694b0 con 0x7f41e8155b80 2024-04-24T08:15:46.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.736+0000 7f41ed400640 1 -- 172.21.15.69:0/3383585905 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41e8075d50 con 0x7f41e8155b80 2024-04-24T08:15:46.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.737+0000 7f41e4c00640 1 -- 172.21.15.69:0/3383585905 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+5174 (secure 0 0 0) 0x7f41d0049030 con 0x7f41e8155b80 2024-04-24T08:15:46.739 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:46.739 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:15:46.271970+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7581":{"gid":7581,"name":"a","rank":0,"incarnation":64,"state":"up:replay","state_seq":1,"addr":"172.21.15.69:6833/2529876410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2529876410},{"type":"v1","addr":"172.21.15.69:6833","nonce":2529876410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-04-24T08:15:46.739 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 64 2024-04-24T08:15:46.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 -- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f41c406c920 msgr2=0x7f41c406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f41c406c920 0x7f41c406ede0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f41d80012a0 tx=0x7f41d8009040 comp rx=0 tx=0).stop 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 -- 172.21.15.69:0/3383585905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8155b80 msgr2=0x7f41e8145710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8155b80 0x7f41e8145710 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f41d0009be0 tx=0x7f41d0004730 comp rx=0 tx=0).stop 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 -- 172.21.15.69:0/3383585905 shutdown_connections 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f41c406c920 0x7f41c406ede0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8155b80 0x7f41e8145710 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f41e8155340 0x7f41e8072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 --2- 172.21.15.69:0/3383585905 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41cc004800 0x7f41e80cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 -- 172.21.15.69:0/3383585905 >> 172.21.15.69:0/3383585905 conn(0x7f41e80d0160 msgr2=0x7f41e806e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 -- 172.21.15.69:0/3383585905 shutdown_connections 2024-04-24T08:15:46.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:46.741+0000 7f41ed400640 1 -- 172.21.15.69:0/3383585905 wait complete. 2024-04-24T08:15:47.165 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:15:23.442689+0000', 'modified': '2024-04-24T08:15:46.271970+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 83, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7581}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7581': {'gid': 7581, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.69:6833/2529876410', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 2529876410}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 2529876410}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:15:47.165 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7581:up:replay 2024-04-24T08:15:48.166 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:48.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.374+0000 7f189bc00640 1 Processor -- start 2024-04-24T08:15:48.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.374+0000 7f189bc00640 1 -- start start 2024-04-24T08:15:48.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.375+0000 7f189bc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18940d02f0 0x7f189406f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:48.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.375+0000 7f189bc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18940b46e0 con 0x7f189415b490 2024-04-24T08:15:48.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.375+0000 7f189bc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18940d0960 con 0x7f189406f2d0 2024-04-24T08:15:48.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.375+0000 7f189bc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f189406fc70 con 0x7f18940d02f0 2024-04-24T08:15:48.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.376+0000 7f1899200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f189415b490 0x7f18940b5180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:32952/0 (socket says 172.21.15.69:32952) 2024-04-24T08:15:48.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.376+0000 7f1899200640 1 -- 172.21.15.69:0/2357927412 learned_addr learned my addr 172.21.15.69:0/2357927412 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:48.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.376+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420251443 0 0) 0x7f18940b46e0 con 0x7f189415b490 2024-04-24T08:15:48.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.376+0000 7f1899c00640 1 --2- 172.21.15.69:0/2357927412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18940d02f0 0x7f189406f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:48.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.376+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f187c005730 con 0x7f189415b490 2024-04-24T08:15:48.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 70721099 0 0) 0x7f187c005730 con 0x7f189415b490 2024-04-24T08:15:48.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18940d02f0 msgr2=0x7f189406f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 --2- 172.21.15.69:0/2357927412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18940d02f0 0x7f189406f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 >> v1:172.21.15.121:6789/0 conn(0x7f189406f2d0 legacy=0x7f18940cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18940b4980 con 0x7f189415b490 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f1888001670 con 0x7f189415b490 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 >> v1:172.21.15.69:6789/0 conn(0x7f189415b490 legacy=0x7f18940b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 --2- 172.21.15.69:0/2357927412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18940d02f0 0x7f187c005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 --2- 172.21.15.69:0/2357927412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f187c0061c0 0x7f187c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 --2- 172.21.15.69:0/2357927412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f187c006f50 0x7f187c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f189406fc70 con 0x7f187c0061c0 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f187c00dd10 con 0x7f187c006f50 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f187c00df10 con 0x7f18940d02f0 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893e00640 1 --2- 172.21.15.69:0/2357927412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f187c0061c0 0x7f187c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893e00640 1 -- 172.21.15.69:0/2357927412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18940d02f0 msgr2=0x7f187c005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893e00640 1 --2- 172.21.15.69:0/2357927412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f18940d02f0 0x7f187c005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:48.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893e00640 1 -- 172.21.15.69:0/2357927412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f187c006f50 msgr2=0x7f187c0073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:48.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893e00640 1 --2- 172.21.15.69:0/2357927412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f187c006f50 0x7f187c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:48.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.377+0000 7f1893e00640 1 -- 172.21.15.69:0/2357927412 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187c005730 con 0x7f187c0061c0 2024-04-24T08:15:48.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.378+0000 7f1893e00640 1 --2- 172.21.15.69:0/2357927412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f187c0061c0 0x7f187c006a10 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f1880001750 tx=0x7f187c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=febbcfffbdbdf76e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:48.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.379+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1880001370 con 0x7f187c0061c0 2024-04-24T08:15:48.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.379+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18800052d0 con 0x7f187c0061c0 2024-04-24T08:15:48.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.379+0000 7f1893400640 1 -- 172.21.15.69:0/2357927412 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188002ab20 con 0x7f187c0061c0 2024-04-24T08:15:48.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.379+0000 7f189bc00640 1 -- 172.21.15.69:0/2357927412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f187c0061c0 msgr2=0x7f187c006a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:48.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.379+0000 7f189bc00640 1 --2- 172.21.15.69:0/2357927412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f187c0061c0 0x7f187c006a10 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f1880001750 tx=0x7f187c007a40 comp rx=0 tx=0).stop 2024-04-24T08:15:48.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.379+0000 7f1899c00640 1 -- 172.21.15.69:0/2357927412 reap_dead start 2024-04-24T08:15:48.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.379+0000 7f189bc00640 1 -- 172.21.15.69:0/2357927412 shutdown_connections 2024-04-24T08:15:48.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.379+0000 7f189bc00640 1 -- 172.21.15.69:0/2357927412 >> 172.21.15.69:0/2357927412 conn(0x7f1894067d90 msgr2=0x7f189406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:48.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.380+0000 7f189bc00640 1 -- 172.21.15.69:0/2357927412 shutdown_connections 2024-04-24T08:15:48.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.380+0000 7f189bc00640 1 -- 172.21.15.69:0/2357927412 wait complete. 2024-04-24T08:15:48.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.381+0000 7f189bc00640 1 Processor -- start 2024-04-24T08:15:48.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.381+0000 7f189bc00640 1 -- start start 2024-04-24T08:15:48.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.382+0000 7f189bc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f187c0061c0 0x7f18941528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:48.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.382+0000 7f189bc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1894152e30 0x7f1894147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:48.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.382+0000 7f189bc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18941484e0 0x7f1894148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:48.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.382+0000 7f189bc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f18940d0c40 con 0x7f18941484e0 2024-04-24T08:15:48.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.382+0000 7f189bc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f18940b5890 con 0x7f187c0061c0 2024-04-24T08:15:48.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.382+0000 7f189bc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f189415b870 con 0x7f1894152e30 2024-04-24T08:15:48.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1899c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18941484e0 0x7f1894148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:48.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1899c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18941484e0 0x7f1894148960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41678/0 (socket says 172.21.15.69:41678) 2024-04-24T08:15:48.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1899c00640 1 -- 172.21.15.69:0/1170598308 learned_addr learned my addr 172.21.15.69:0/1170598308 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:48.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1899200640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f187c0061c0 0x7f18941528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:48.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1893e00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1894152e30 0x7f1894147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:48.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1899c00640 1 -- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1894152e30 msgr2=0x7f1894147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:48.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1899c00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1894152e30 0x7f1894147fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:48.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1899c00640 1 -- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f187c0061c0 msgr2=0x7f18941528f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:48.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1899c00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f187c0061c0 0x7f18941528f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:48.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1899c00640 1 -- 172.21.15.69:0/1170598308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18941522f0 con 0x7f18941484e0 2024-04-24T08:15:48.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.383+0000 7f1893e00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1894152e30 0x7f1894147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:48.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.384+0000 7f1899c00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18941484e0 0x7f1894148960 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f18940ccfd0 tx=0x7f188c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:48.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.384+0000 7f1890c00640 1 -- 172.21.15.69:0/1170598308 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c00bab0 con 0x7f18941484e0 2024-04-24T08:15:48.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.384+0000 7f1890c00640 1 -- 172.21.15.69:0/1170598308 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f188c01a070 con 0x7f18941484e0 2024-04-24T08:15:48.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.384+0000 7f189bc00640 1 -- 172.21.15.69:0/1170598308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1894155330 con 0x7f18941484e0 2024-04-24T08:15:48.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.384+0000 7f1890c00640 1 -- 172.21.15.69:0/1170598308 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c014400 con 0x7f18941484e0 2024-04-24T08:15:48.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.384+0000 7f189bc00640 1 -- 172.21.15.69:0/1170598308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18941558f0 con 0x7f18941484e0 2024-04-24T08:15:48.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.385+0000 7f189bc00640 1 -- 172.21.15.69:0/1170598308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1894068a80 con 0x7f18941484e0 2024-04-24T08:15:48.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.386+0000 7f1890c00640 1 -- 172.21.15.69:0/1170598308 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f188c005d10 con 0x7f18941484e0 2024-04-24T08:15:48.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.387+0000 7f1890c00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f187406c8f0 0x7f187406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:48.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.387+0000 7f1890c00640 1 -- 172.21.15.69:0/1170598308 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f188c08bd80 con 0x7f18941484e0 2024-04-24T08:15:48.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.388+0000 7f1899200640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f187406c8f0 0x7f187406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:48.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.389+0000 7f1899200640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f187406c8f0 0x7f187406edb0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1888000ff0 tx=0x7f1888001720 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:48.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.393+0000 7f1890c00640 1 -- 172.21.15.69:0/1170598308 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f188c024020 con 0x7f18941484e0 2024-04-24T08:15:48.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.632+0000 7f189bc00640 1 -- 172.21.15.69:0/1170598308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f189406bce0 con 0x7f18941484e0 2024-04-24T08:15:48.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.633+0000 7f1890c00640 1 -- 172.21.15.69:0/1170598308 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5174 (secure 0 0 0) 0x7f18941522f0 con 0x7f18941484e0 2024-04-24T08:15:48.635 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:48.635 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":66,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:15:47.294311+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7581":{"gid":7581,"name":"a","rank":0,"incarnation":64,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.69:6833/2529876410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2529876410},{"type":"v1","addr":"172.21.15.69:6833","nonce":2529876410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-04-24T08:15:48.635 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 66 2024-04-24T08:15:48.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.636+0000 7f189bc00640 1 -- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f187406c8f0 msgr2=0x7f187406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:48.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.636+0000 7f189bc00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f187406c8f0 0x7f187406edb0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f1888000ff0 tx=0x7f1888001720 comp rx=0 tx=0).stop 2024-04-24T08:15:48.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.636+0000 7f189bc00640 1 -- 172.21.15.69:0/1170598308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18941484e0 msgr2=0x7f1894148960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:48.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.636+0000 7f189bc00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18941484e0 0x7f1894148960 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f18940ccfd0 tx=0x7f188c00b480 comp rx=0 tx=0).stop 2024-04-24T08:15:48.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.636+0000 7f189bc00640 1 -- 172.21.15.69:0/1170598308 shutdown_connections 2024-04-24T08:15:48.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.636+0000 7f189bc00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f187406c8f0 0x7f187406edb0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:48.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.636+0000 7f189bc00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18941484e0 0x7f1894148960 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:48.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.636+0000 7f189bc00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1894152e30 0x7f1894147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:48.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.636+0000 7f189bc00640 1 --2- 172.21.15.69:0/1170598308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f187c0061c0 0x7f18941528f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:48.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.636+0000 7f189bc00640 1 -- 172.21.15.69:0/1170598308 >> 172.21.15.69:0/1170598308 conn(0x7f1894067d90 msgr2=0x7f1894072600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:48.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.637+0000 7f189bc00640 1 -- 172.21.15.69:0/1170598308 shutdown_connections 2024-04-24T08:15:48.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:48.637+0000 7f189bc00640 1 -- 172.21.15.69:0/1170598308 wait complete. 2024-04-24T08:15:49.063 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:15:23.442689+0000', 'modified': '2024-04-24T08:15:47.294311+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 83, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7581}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7581': {'gid': 7581, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:rejoin', 'state_seq': 9, 'addr': '172.21.15.69:6833/2529876410', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 2529876410}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 2529876410}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:15:49.063 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7581:up:rejoin 2024-04-24T08:15:50.064 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:50.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.273+0000 7f656a200640 1 Processor -- start 2024-04-24T08:15:50.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.274+0000 7f656a200640 1 -- start start 2024-04-24T08:15:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.275+0000 7f656a200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f656415e960 0x7f6564162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.275+0000 7f656a200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6564162980 0x7f656406cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.275+0000 7f656a200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f656415ed40 con 0x7f656406d4f0 2024-04-24T08:15:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.275+0000 7f656a200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f65640cf050 con 0x7f656415e960 2024-04-24T08:15:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.275+0000 7f656a200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6564162e20 con 0x7f6564162980 2024-04-24T08:15:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.275+0000 7f6568e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f656415e960 0x7f6564162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.275+0000 7f6569800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f656406d4f0 0x7f65640cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46246/0 (socket says 172.21.15.69:46246) 2024-04-24T08:15:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.275+0000 7f6569800640 1 -- 172.21.15.69:0/272749860 learned_addr learned my addr 172.21.15.69:0/272749860 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:50.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.275+0000 7f6568e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f656415e960 0x7f6564162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57982/0 (socket says 172.21.15.69:57982) 2024-04-24T08:15:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.276+0000 7f6563e00640 1 --2- 172.21.15.69:0/272749860 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6564162980 0x7f656406cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.276+0000 7f6568e00640 1 -- 172.21.15.69:0/272749860 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6564162980 msgr2=0x7f656406cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.276+0000 7f6568e00640 1 --2- 172.21.15.69:0/272749860 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6564162980 0x7f656406cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.276+0000 7f6568e00640 1 -- 172.21.15.69:0/272749860 >> v1:172.21.15.69:6789/0 conn(0x7f656406d4f0 legacy=0x7f65640cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.276+0000 7f6568e00640 1 -- 172.21.15.69:0/272749860 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6564163100 con 0x7f656415e960 2024-04-24T08:15:50.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.276+0000 7f6563400640 1 -- 172.21.15.69:0/272749860 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2161061541 0 0) 0x7f656415ed40 con 0x7f656406d4f0 2024-04-24T08:15:50.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.277+0000 7f6568e00640 1 --2- 172.21.15.69:0/272749860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f656415e960 0x7f6564162440 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f654c0097a0 tx=0x7f654c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96210aa5ce026b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.277+0000 7f6563400640 1 -- 172.21.15.69:0/272749860 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c037070 con 0x7f656415e960 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.277+0000 7f6563400640 1 -- 172.21.15.69:0/272749860 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f654c032440 con 0x7f656415e960 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.277+0000 7f6563400640 1 -- 172.21.15.69:0/272749860 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654c032810 con 0x7f656415e960 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.278+0000 7f656a200640 1 -- 172.21.15.69:0/272749860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f656415e960 msgr2=0x7f6564162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.278+0000 7f656a200640 1 --2- 172.21.15.69:0/272749860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f656415e960 0x7f6564162440 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f654c0097a0 tx=0x7f654c02a180 comp rx=0 tx=0).stop 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.278+0000 7f656a200640 1 -- 172.21.15.69:0/272749860 shutdown_connections 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.278+0000 7f656a200640 1 --2- 172.21.15.69:0/272749860 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6564162980 0x7f656406cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.278+0000 7f656a200640 1 --2- 172.21.15.69:0/272749860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f656415e960 0x7f6564162440 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.278+0000 7f656a200640 1 -- 172.21.15.69:0/272749860 >> 172.21.15.69:0/272749860 conn(0x7f656409d900 msgr2=0x7f656408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.278+0000 7f656a200640 1 -- 172.21.15.69:0/272749860 shutdown_connections 2024-04-24T08:15:50.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.278+0000 7f656a200640 1 -- 172.21.15.69:0/272749860 wait complete. 2024-04-24T08:15:50.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.280+0000 7f656a200640 1 Processor -- start 2024-04-24T08:15:50.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.280+0000 7f656a200640 1 -- start start 2024-04-24T08:15:50.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.281+0000 7f656a200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656406d4f0 0x7f65641e8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:50.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.281+0000 7f656a200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f656415e960 0x7f65641609b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:50.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.281+0000 7f656a200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564162980 0x7f6564160ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:50.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.281+0000 7f656a200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f65640b6170 con 0x7f6564162980 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.281+0000 7f656a200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f65640cf5d0 con 0x7f656406d4f0 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.281+0000 7f656a200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f65640b62f0 con 0x7f656415e960 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.281+0000 7f6569800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564162980 0x7f6564160ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.282+0000 7f6569800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564162980 0x7f6564160ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57994/0 (socket says 172.21.15.69:57994) 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.282+0000 7f6569800640 1 -- 172.21.15.69:0/2289053494 learned_addr learned my addr 172.21.15.69:0/2289053494 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.282+0000 7f6563e00640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f656415e960 0x7f65641609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.282+0000 7f6568e00640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656406d4f0 0x7f65641e8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.282+0000 7f6569800640 1 -- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f656415e960 msgr2=0x7f65641609b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.282+0000 7f6569800640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f656415e960 0x7f65641609b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.282+0000 7f6569800640 1 -- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656406d4f0 msgr2=0x7f65641e8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.282+0000 7f6569800640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656406d4f0 0x7f65641e8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:50.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.282+0000 7f6569800640 1 -- 172.21.15.69:0/2289053494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65640b73f0 con 0x7f6564162980 2024-04-24T08:15:50.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.282+0000 7f6569800640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564162980 0x7f6564160ef0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f655c00c480 tx=0x7f655c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:50.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.283+0000 7f6560c00640 1 -- 172.21.15.69:0/2289053494 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c018070 con 0x7f6564162980 2024-04-24T08:15:50.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.283+0000 7f6560c00640 1 -- 172.21.15.69:0/2289053494 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f655c0074e0 con 0x7f6564162980 2024-04-24T08:15:50.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.283+0000 7f6560c00640 1 -- 172.21.15.69:0/2289053494 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c004a20 con 0x7f6564162980 2024-04-24T08:15:50.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.283+0000 7f656a200640 1 -- 172.21.15.69:0/2289053494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f656415c240 con 0x7f6564162980 2024-04-24T08:15:50.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.283+0000 7f656a200640 1 -- 172.21.15.69:0/2289053494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65641e9ba0 con 0x7f6564162980 2024-04-24T08:15:50.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.284+0000 7f656a200640 1 -- 172.21.15.69:0/2289053494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65640b7890 con 0x7f6564162980 2024-04-24T08:15:50.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.285+0000 7f6560c00640 1 -- 172.21.15.69:0/2289053494 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f655c01b070 con 0x7f6564162980 2024-04-24T08:15:50.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.286+0000 7f6560c00640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f654006c8f0 0x7f654006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:50.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.286+0000 7f6560c00640 1 -- 172.21.15.69:0/2289053494 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f655c08ae30 con 0x7f6564162980 2024-04-24T08:15:50.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.286+0000 7f6568e00640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f654006c8f0 0x7f654006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:50.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.288+0000 7f6568e00640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f654006c8f0 0x7f654006edb0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f654c034000 tx=0x7f654c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:50.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.290+0000 7f6560c00640 1 -- 172.21.15.69:0/2289053494 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f65640b7890 con 0x7f6564162980 2024-04-24T08:15:50.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.529+0000 7f656a200640 1 -- 172.21.15.69:0/2289053494 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6564075d50 con 0x7f6564162980 2024-04-24T08:15:50.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.529+0000 7f6560c00640 1 -- 172.21.15.69:0/2289053494 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+5182 (secure 0 0 0) 0x7f655c059290 con 0x7f6564162980 2024-04-24T08:15:50.531 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:15:50.531 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":67,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:15:49.302320+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7581":{"gid":7581,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":10,"addr":"172.21.15.69:6833/2529876410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2529876410},{"type":"v1","addr":"172.21.15.69:6833","nonce":2529876410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7581,"qdb_cluster":[7581]},"id":6}]} 2024-04-24T08:15:50.531 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 67 2024-04-24T08:15:50.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 -- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f654006c8f0 msgr2=0x7f654006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:50.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f654006c8f0 0x7f654006edb0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f654c034000 tx=0x7f654c02e000 comp rx=0 tx=0).stop 2024-04-24T08:15:50.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 -- 172.21.15.69:0/2289053494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564162980 msgr2=0x7f6564160ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:50.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564162980 0x7f6564160ef0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f655c00c480 tx=0x7f655c00c940 comp rx=0 tx=0).stop 2024-04-24T08:15:50.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 -- 172.21.15.69:0/2289053494 shutdown_connections 2024-04-24T08:15:50.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f654006c8f0 0x7f654006edb0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:50.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6564162980 0x7f6564160ef0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:50.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f656415e960 0x7f65641609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:50.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 --2- 172.21.15.69:0/2289053494 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f656406d4f0 0x7f65641e8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:50.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 -- 172.21.15.69:0/2289053494 >> 172.21.15.69:0/2289053494 conn(0x7f656409d900 msgr2=0x7f65640b4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:50.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 -- 172.21.15.69:0/2289053494 shutdown_connections 2024-04-24T08:15:50.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:50.532+0000 7f656a200640 1 -- 172.21.15.69:0/2289053494 wait complete. 2024-04-24T08:15:50.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:15:23.442689+0000', 'modified': '2024-04-24T08:15:49.302320+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 83, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7581}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7581': {'gid': 7581, 'name': 'a', 'rank': 0, 'incarnation': 64, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.69:6833/2529876410', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 2529876410}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 2529876410}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7581, 'qdb_cluster': [7581]} 2024-04-24T08:15:50.955 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:15:50.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-24T08:15:51.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.161+0000 7f9b88c00640 1 Processor -- start 2024-04-24T08:15:51.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.162+0000 7f9b88c00640 1 -- start start 2024-04-24T08:15:51.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.163+0000 7f9b88c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9b840d0200 0x7f9b840d2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.163+0000 7f9b88c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b8415ed40 con 0x7f9b8406cfb0 2024-04-24T08:15:51.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.163+0000 7f9b88c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b840cf050 con 0x7f9b8415e960 2024-04-24T08:15:51.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.163+0000 7f9b88c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9b84162b50 con 0x7f9b840d0200 2024-04-24T08:15:51.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.163+0000 7f9b82a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9b8406cfb0 0x7f9b840cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46258/0 (socket says 172.21.15.69:46258) 2024-04-24T08:15:51.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.163+0000 7f9b82a00640 1 -- 172.21.15.69:0/1065521301 learned_addr learned my addr 172.21.15.69:0/1065521301 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:51.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b83e00640 1 --2- 172.21.15.69:0/1065521301 >> v2:172.21.15.121:3300/0 conn(0x7f9b840d0200 0x7f9b840d2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4024893619 0 0) 0x7f9b8415ed40 con 0x7f9b8406cfb0 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b68003520 con 0x7f9b8406cfb0 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4066008139 0 0) 0x7f9b68003520 con 0x7f9b8406cfb0 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 >> v2:172.21.15.121:3300/0 conn(0x7f9b840d0200 msgr2=0x7f9b840d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 --2- 172.21.15.69:0/1065521301 >> v2:172.21.15.121:3300/0 conn(0x7f9b840d0200 0x7f9b840d2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 >> v1:172.21.15.121:6789/0 conn(0x7f9b8415e960 legacy=0x7f9b84162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b84162e20 con 0x7f9b8406cfb0 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f9b700019f0 con 0x7f9b8406cfb0 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 >> v1:172.21.15.69:6789/0 conn(0x7f9b8406cfb0 legacy=0x7f9b840cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b680039b0 0x7f9b68003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b68004800 0x7f9b68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b84162b50 con 0x7f9b680051c0 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6800c060 con 0x7f9b680039b0 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.164+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6800c260 con 0x7f9b68004800 2024-04-24T08:15:51.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.165+0000 7f9b83e00640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.165+0000 7f9b83400640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b680039b0 0x7f9b68003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.165+0000 7f9b83e00640 1 -- 172.21.15.69:0/1065521301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b68004800 msgr2=0x7f9b68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.165+0000 7f9b83e00640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b68004800 0x7f9b68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.165+0000 7f9b83e00640 1 -- 172.21.15.69:0/1065521301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b680039b0 msgr2=0x7f9b68003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.165+0000 7f9b83e00640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b680039b0 0x7f9b68003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.165+0000 7f9b83e00640 1 -- 172.21.15.69:0/1065521301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b68003520 con 0x7f9b680051c0 2024-04-24T08:15:51.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.165+0000 7f9b83400640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b680039b0 0x7f9b68003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.166+0000 7f9b83e00640 1 -- 172.21.15.69:0/1065521301 reap_dead start 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.166+0000 7f9b83e00640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b6800b9c0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f9b8408de30 tx=0x7f9b68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4349b5022aefa432 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.167+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c009690 con 0x7f9b680051c0 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.167+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7c004020 con 0x7f9b680051c0 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.167+0000 7f9b82000640 1 -- 172.21.15.69:0/1065521301 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c004410 con 0x7f9b680051c0 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.167+0000 7f9b88c00640 1 -- 172.21.15.69:0/1065521301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 msgr2=0x7f9b6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.167+0000 7f9b88c00640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b6800b9c0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f9b8408de30 tx=0x7f9b68005830 comp rx=0 tx=0).stop 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.167+0000 7f9b88c00640 1 -- 172.21.15.69:0/1065521301 shutdown_connections 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.167+0000 7f9b88c00640 1 --2- 172.21.15.69:0/1065521301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b6800b9c0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.167+0000 7f9b88c00640 1 -- 172.21.15.69:0/1065521301 >> 172.21.15.69:0/1065521301 conn(0x7f9b8409d900 msgr2=0x7f9b8408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.167+0000 7f9b88c00640 1 -- 172.21.15.69:0/1065521301 shutdown_connections 2024-04-24T08:15:51.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.167+0000 7f9b88c00640 1 -- 172.21.15.69:0/1065521301 wait complete. 2024-04-24T08:15:51.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.169+0000 7f9b88c00640 1 Processor -- start 2024-04-24T08:15:51.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.169+0000 7f9b88c00640 1 -- start start 2024-04-24T08:15:51.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.170+0000 7f9b88c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b841558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.170+0000 7f9b88c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b840b7150 0x7f9b840b75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.170+0000 7f9b88c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b840b7b10 0x7f9b840b8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.170+0000 7f9b88c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b840cf5d0 con 0x7f9b680051c0 2024-04-24T08:15:51.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.170+0000 7f9b88c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9b840cf2f0 con 0x7f9b840b7150 2024-04-24T08:15:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.170+0000 7f9b88c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9b840b6170 con 0x7f9b840b7b10 2024-04-24T08:15:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.170+0000 7f9b83400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b841558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.170+0000 7f9b83400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b841558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58022/0 (socket says 172.21.15.69:58022) 2024-04-24T08:15:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.170+0000 7f9b83400640 1 -- 172.21.15.69:0/1872725886 learned_addr learned my addr 172.21.15.69:0/1872725886 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.171+0000 7f9b83400640 1 -- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b840b7b10 msgr2=0x7f9b840b8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.171+0000 7f9b83400640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b840b7b10 0x7f9b840b8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.171+0000 7f9b83400640 1 -- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b840b7150 msgr2=0x7f9b840b75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.171+0000 7f9b83400640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b840b7150 0x7f9b840b75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.171+0000 7f9b83400640 1 -- 172.21.15.69:0/1872725886 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b840b7f90 con 0x7f9b680051c0 2024-04-24T08:15:51.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.171+0000 7f9b83400640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b841558b0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f9b74001eb0 tx=0x7f9b7400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.171+0000 7f9b7b400640 1 -- 172.21.15.69:0/1872725886 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74016070 con 0x7f9b680051c0 2024-04-24T08:15:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.171+0000 7f9b7b400640 1 -- 172.21.15.69:0/1872725886 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b74012800 con 0x7f9b680051c0 2024-04-24T08:15:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.171+0000 7f9b88c00640 1 -- 172.21.15.69:0/1872725886 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b840cd800 con 0x7f9b680051c0 2024-04-24T08:15:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.172+0000 7f9b7b400640 1 -- 172.21.15.69:0/1872725886 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74012bd0 con 0x7f9b680051c0 2024-04-24T08:15:51.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.172+0000 7f9b88c00640 1 -- 172.21.15.69:0/1872725886 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b840cddc0 con 0x7f9b680051c0 2024-04-24T08:15:51.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.174+0000 7f9b7b400640 1 -- 172.21.15.69:0/1872725886 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b74019070 con 0x7f9b680051c0 2024-04-24T08:15:51.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.175+0000 7f9b7b400640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9b6406c920 0x7f9b6406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.175+0000 7f9b7b400640 1 -- 172.21.15.69:0/1872725886 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f9b7408ba10 con 0x7f9b680051c0 2024-04-24T08:15:51.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.175+0000 7f9b8d00d740 1 Processor -- start 2024-04-24T08:15:51.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.175+0000 7f9b82a00640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9b6406c920 0x7f9b6406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.175+0000 7f9b8d00d740 1 -- start start 2024-04-24T08:15:51.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.176+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 0x555a53f0e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.176+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f3eaa0 0x555a53f3de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.176+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f3e360 0x555a53f13170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.176+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555a53e78b10 con 0x555a53f3e360 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.176+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555a53e1f2b0 con 0x555a53f3eaa0 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.176+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x555a53e74ad0 con 0x555a53f5b980 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f3e360 0x555a53f13170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f3e360 0x555a53f13170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58034/0 (socket says 172.21.15.69:58034) 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83e00640 1 -- 172.21.15.69:0/702774587 learned_addr learned my addr 172.21.15.69:0/702774587 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b82a00640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9b6406c920 0x7f9b6406ede0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9b70001370 tx=0x7f9b70007bf0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83400640 1 --2- 172.21.15.69:0/702774587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 0x555a53f0e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b82a00640 1 --2- 172.21.15.69:0/702774587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f3eaa0 0x555a53f3de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83e00640 1 -- 172.21.15.69:0/702774587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 msgr2=0x555a53f0e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83e00640 1 --2- 172.21.15.69:0/702774587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 0x555a53f0e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83e00640 1 -- 172.21.15.69:0/702774587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f3eaa0 msgr2=0x555a53f3de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83e00640 1 --2- 172.21.15.69:0/702774587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f3eaa0 0x555a53f3de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83e00640 1 -- 172.21.15.69:0/702774587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a53b8b2d0 con 0x555a53f3e360 2024-04-24T08:15:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83400640 1 --2- 172.21.15.69:0/702774587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 0x555a53f0e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b83e00640 1 --2- 172.21.15.69:0/702774587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f3e360 0x555a53f13170 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c006f40 tx=0x7f9b7c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc0aab3470c4611 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.177+0000 7f9b6f400640 1 -- 172.21.15.69:0/702774587 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c009750 con 0x555a53f3e360 2024-04-24T08:15:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b6f400640 1 -- 172.21.15.69:0/702774587 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7c004070 con 0x555a53f3e360 2024-04-24T08:15:51.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b6f400640 1 -- 172.21.15.69:0/702774587 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c004420 con 0x555a53f3e360 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b8d00d740 1 -- 172.21.15.69:0/702774587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f3e360 msgr2=0x555a53f13170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/702774587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f3e360 0x555a53f13170 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c006f40 tx=0x7f9b7c02c000 comp rx=0 tx=0).stop 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b8d00d740 1 -- 172.21.15.69:0/702774587 shutdown_connections 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/702774587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f3e360 0x555a53f13170 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/702774587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f3eaa0 0x555a53f3de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/702774587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 0x555a53f0e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b8d00d740 1 -- 172.21.15.69:0/702774587 >> 172.21.15.69:0/702774587 conn(0x555a53f10580 msgr2=0x555a53f58e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b8d00d740 1 -- 172.21.15.69:0/702774587 shutdown_connections 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.178+0000 7f9b8d00d740 1 -- 172.21.15.69:0/702774587 wait complete. 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.179+0000 7f9b8d00d740 1 -- start start 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.179+0000 7f9b8d00d740 1 Processor -- start 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.180+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f3eaa0 0x555a53f2e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.180+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 0x555a53f13940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.180+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f40eb0 0x555a53f41330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.180+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555a53f12730 con 0x555a53f40eb0 2024-04-24T08:15:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.180+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555a53e32590 con 0x555a53f3eaa0 2024-04-24T08:15:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.180+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x555a53f11b20 con 0x555a53f5b980 2024-04-24T08:15:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.180+0000 7f9b8d00d740 10 client.0 _ll_register_callbacks cb 0x555a53e153a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.180+0000 7f9b83e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f40eb0 0x555a53f41330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.180+0000 7f9b83e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f40eb0 0x555a53f41330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58038/0 (socket says 172.21.15.69:58038) 2024-04-24T08:15:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.180+0000 7f9b83e00640 1 -- 172.21.15.69:0/3569422562 learned_addr learned my addr 172.21.15.69:0/3569422562 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:51.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b82a00640 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 0x555a53f13940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b83e00640 1 -- 172.21.15.69:0/3569422562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 msgr2=0x555a53f13940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b83400640 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f3eaa0 0x555a53f2e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b83e00640 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 0x555a53f13940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b83e00640 1 -- 172.21.15.69:0/3569422562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f3eaa0 msgr2=0x555a53f2e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b83e00640 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f3eaa0 0x555a53f2e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b83e00640 1 -- 172.21.15.69:0/3569422562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a53c556b0 con 0x555a53f40eb0 2024-04-24T08:15:51.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b83e00640 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f40eb0 0x555a53f41330 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c004450 tx=0x7f9b7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4fa493ce3db08d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b63e00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b63e00640 1 -- 172.21.15.69:0/3569422562 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c048060 con 0x555a53f40eb0 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b63e00640 1 -- 172.21.15.69:0/3569422562 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f9b7c0044b0 con 0x555a53f40eb0 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.181+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3569422562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b7c004070 con 0x555a53f40eb0 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b63e00640 1 -- 172.21.15.69:0/3569422562 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f9b7c005990 con 0x555a53f40eb0 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b8d00d740 10 client.7890 fetch_fsmap learned FSMap version 67 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3569422562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x555a53f51420 con 0x555a53f40eb0 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b63e00640 1 -- 172.21.15.69:0/3569422562 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7c004070 con 0x555a53f40eb0 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b63e00640 1 -- 172.21.15.69:0/3569422562 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f9b7c051080 con 0x555a53f40eb0 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b8d00d740 10 client.7890 fetch_fsmap finished waiting for FSMap version 67 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b8d00d740 10 client.7890 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b8d00d740 20 client.7890 populate_metadata read hostname 'smithi069' 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x555a53f69040 0x555a53f3f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b8d00d740 4 client.7890 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:51.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3569422562 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a53e86530 con 0x555a53f69040 2024-04-24T08:15:51.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.182+0000 7f9b83400640 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x555a53f69040 0x555a53f3f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.183+0000 7f9b63e00640 10 client.7890 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:51.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.183+0000 7f9b83400640 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x555a53f69040 0x555a53f3f3a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f44d159590578a71 server_cookie=539344eb04d78941 in_seq=0 out_seq=0 2024-04-24T08:15:51.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b63e00640 1 -- 172.21.15.69:0/3569422562 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a53e86530 con 0x555a53f69040 2024-04-24T08:15:51.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b63e00640 10 client.7890 handle_command_reply: tid=0 2024-04-24T08:15:51.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b8d00d740 1 client.7890 shutdown 2024-04-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3569422562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f40eb0 msgr2=0x555a53f41330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f40eb0 0x555a53f41330 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c004450 tx=0x7f9b7c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3569422562 shutdown_connections 2024-04-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x555a53f69040 0x555a53f3f3a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f40eb0 0x555a53f41330 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f5b980 0x555a53f13940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b63e00640 0 client.7890 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/3569422562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f3eaa0 0x555a53f2e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3569422562 >> 172.21.15.69:0/3569422562 conn(0x555a53f3e360 msgr2=0x555a53f0d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:51.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.184+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3569422562 shutdown_connections 2024-04-24T08:15:51.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.185+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3569422562 wait complete. 2024-04-24T08:15:51.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.185+0000 7f9b8d00d740 20 client.7890 trim_cache size 0 max 16384 2024-04-24T08:15:51.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.203+0000 7f9b8d00d740 1 Processor -- start 2024-04-24T08:15:51.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.203+0000 7f9b8d00d740 1 -- start start 2024-04-24T08:15:51.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.204+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 0x555a53f58c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.204+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f40eb0 0x555a53f11120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.204+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f7c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.204+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b84200650 con 0x555a53f5b980 2024-04-24T08:15:51.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.204+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9b840b1f00 con 0x555a53f40eb0 2024-04-24T08:15:51.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.204+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9b840b2550 con 0x555a53f3eaa0 2024-04-24T08:15:51.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f7c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f7c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58050/0 (socket says 172.21.15.69:58050) 2024-04-24T08:15:51.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83e00640 1 -- 172.21.15.69:0/3599507159 learned_addr learned my addr 172.21.15.69:0/3599507159 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:51.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83400640 1 --2- 172.21.15.69:0/3599507159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 0x555a53f58c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83e00640 1 -- 172.21.15.69:0/3599507159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 msgr2=0x555a53f58c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83e00640 1 --2- 172.21.15.69:0/3599507159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 0x555a53f58c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83e00640 1 -- 172.21.15.69:0/3599507159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f40eb0 msgr2=0x555a53f11120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83e00640 1 --2- 172.21.15.69:0/3599507159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f40eb0 0x555a53f11120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83e00640 1 -- 172.21.15.69:0/3599507159 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a53f502f0 con 0x555a53f5b980 2024-04-24T08:15:51.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83400640 1 --2- 172.21.15.69:0/3599507159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 0x555a53f58c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:51.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.205+0000 7f9b83e00640 1 --2- 172.21.15.69:0/3599507159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f7c480 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c005cf0 tx=0x7f9b7c03e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b5224174e51f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:51.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b6ea00640 1 -- 172.21.15.69:0/3599507159 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c004480 con 0x555a53f5b980 2024-04-24T08:15:51.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b6ea00640 1 -- 172.21.15.69:0/3599507159 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7c00f000 con 0x555a53f5b980 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b6ea00640 1 -- 172.21.15.69:0/3599507159 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c006280 con 0x555a53f5b980 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3599507159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 msgr2=0x555a53f7c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/3599507159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f7c480 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c005cf0 tx=0x7f9b7c03e000 comp rx=0 tx=0).stop 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3599507159 shutdown_connections 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/3599507159 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f7c480 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/3599507159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f40eb0 0x555a53f11120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/3599507159 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 0x555a53f58c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3599507159 >> 172.21.15.69:0/3599507159 conn(0x555a53f3e360 msgr2=0x555a53f3daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3599507159 shutdown_connections 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.206+0000 7f9b8d00d740 1 -- 172.21.15.69:0/3599507159 wait complete. 2024-04-24T08:15:51.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.207+0000 7f9b8d00d740 1 -- start start 2024-04-24T08:15:51.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.207+0000 7f9b8d00d740 1 Processor -- start 2024-04-24T08:15:51.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.208+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 0x555a53fd1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.208+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f40eb0 0x555a53f727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.208+0000 7f9b8d00d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f59f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.208+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555a53f12730 con 0x555a53f5b980 2024-04-24T08:15:51.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.208+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555a53f11b20 con 0x555a53f40eb0 2024-04-24T08:15:51.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.208+0000 7f9b8d00d740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x555a53e32590 con 0x555a53f3eaa0 2024-04-24T08:15:51.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.208+0000 7f9b8d00d740 10 client.0 _ll_register_callbacks cb 0x7f9b680045e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.208+0000 7f9b83e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f59f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.208+0000 7f9b83e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f59f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58052/0 (socket says 172.21.15.69:58052) 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b83e00640 1 -- 172.21.15.69:0/1185968955 learned_addr learned my addr 172.21.15.69:0/1185968955 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b82a00640 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f40eb0 0x555a53f727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b83400640 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 0x555a53fd1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b83e00640 1 -- 172.21.15.69:0/1185968955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 msgr2=0x555a53fd1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b83e00640 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 0x555a53fd1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b83e00640 1 -- 172.21.15.69:0/1185968955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f40eb0 msgr2=0x555a53f727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b83e00640 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f40eb0 0x555a53f727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b83e00640 1 -- 172.21.15.69:0/1185968955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a53f38cc0 con 0x555a53f5b980 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b82a00640 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f40eb0 0x555a53f727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b83e00640 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f59f50 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c005d20 tx=0x7f9b7c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d572b57d685fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b63e00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b63e00640 1 -- 172.21.15.69:0/1185968955 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c048070 con 0x555a53f5b980 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b63e00640 1 -- 172.21.15.69:0/1185968955 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f9b7c053440 con 0x555a53f5b980 2024-04-24T08:15:51.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.209+0000 7f9b8d00d740 1 -- 172.21.15.69:0/1185968955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b7c00f000 con 0x555a53f5b980 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b63e00640 1 -- 172.21.15.69:0/1185968955 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f9b7c05b050 con 0x555a53f5b980 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b8d00d740 10 client.7896 fetch_fsmap learned FSMap version 67 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b8d00d740 1 -- 172.21.15.69:0/1185968955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x555a53f72d80 con 0x555a53f5b980 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b63e00640 1 -- 172.21.15.69:0/1185968955 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b7c00f000 con 0x555a53f5b980 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b63e00640 1 -- 172.21.15.69:0/1185968955 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f9b7c05e020 con 0x555a53f5b980 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b8d00d740 10 client.7896 fetch_fsmap finished waiting for FSMap version 67 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b8d00d740 10 client.7896 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b8d00d740 20 client.7896 populate_metadata read hostname 'smithi069' 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x555a53f40060 0x555a53f40440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b8d00d740 4 client.7896 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b8d00d740 1 -- 172.21.15.69:0/1185968955 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "status"}) -- 0x555a53f52d30 con 0x555a53f40060 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.210+0000 7f9b83400640 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x555a53f40060 0x555a53f40440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b63e00640 10 client.7896 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b83400640 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x555a53f40060 0x555a53f40440 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a40d4afe395401c server_cookie=2eb609fe27bc90a2 in_seq=0 out_seq=0 2024-04-24T08:15:51.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b63e00640 1 -- 172.21.15.69:0/1185968955 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x555a53f52d30 con 0x555a53f40060 2024-04-24T08:15:51.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b63e00640 10 client.7896 handle_command_reply: tid=0 2024-04-24T08:15:51.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 client.7896 shutdown 2024-04-24T08:15:51.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 -- 172.21.15.69:0/1185968955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 msgr2=0x555a53f59f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f59f50 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c005d20 tx=0x7f9b7c02e000 comp rx=0 tx=0).stop 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 -- 172.21.15.69:0/1185968955 shutdown_connections 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x555a53f40060 0x555a53f40440 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a53f5b980 0x555a53f59f50 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555a53f40eb0 0x555a53f727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 --2- 172.21.15.69:0/1185968955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555a53f3eaa0 0x555a53fd1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 -- 172.21.15.69:0/1185968955 >> 172.21.15.69:0/1185968955 conn(0x555a53f3e360 msgr2=0x555a53f09890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b63e00640 0 client.7896 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 -- 172.21.15.69:0/1185968955 shutdown_connections 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.211+0000 7f9b8d00d740 1 -- 172.21.15.69:0/1185968955 wait complete. 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b8d00d740 20 client.7896 trim_cache size 0 max 16384 2024-04-24T08:15:51.216 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 7581, 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 4.9363290620000004, 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 67, 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 83, 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 83, 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 32.688971981999998 2024-04-24T08:15:51.217 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:15:51.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 -- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9b6406c920 msgr2=0x7f9b6406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9b6406c920 0x7f9b6406ede0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f9b70001370 tx=0x7f9b70007bf0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 -- 172.21.15.69:0/1872725886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 msgr2=0x7f9b841558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b841558b0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f9b74001eb0 tx=0x7f9b7400d9d0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 -- 172.21.15.69:0/1872725886 shutdown_connections 2024-04-24T08:15:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9b6406c920 0x7f9b6406ede0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9b840b7b10 0x7f9b840b8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b840b7150 0x7f9b840b75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 --2- 172.21.15.69:0/1872725886 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b680051c0 0x7f9b841558b0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 -- 172.21.15.69:0/1872725886 >> 172.21.15.69:0/1872725886 conn(0x7f9b8409d900 msgr2=0x7f9b8406d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 -- 172.21.15.69:0/1872725886 shutdown_connections 2024-04-24T08:15:51.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.212+0000 7f9b88c00640 1 -- 172.21.15.69:0/1872725886 wait complete. 2024-04-24T08:15:51.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:15:51.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.417+0000 7f0b15200640 1 Processor -- start 2024-04-24T08:15:51.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.417+0000 7f0b15200640 1 -- start start 2024-04-24T08:15:51.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.418+0000 7f0b15200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b1015e960 0x7f0b10162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.418+0000 7f0b15200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b100cfaf0 0x7f0b100cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.418+0000 7f0b15200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b1015ed40 con 0x7f0b10162980 2024-04-24T08:15:51.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.418+0000 7f0b15200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0b100cf050 con 0x7f0b1015e960 2024-04-24T08:15:51.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.418+0000 7f0b15200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0b10162e20 con 0x7f0b100cfaf0 2024-04-24T08:15:51.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.418+0000 7f0b0f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b1015e960 0x7f0b10162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.419+0000 7f0b0f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b1015e960 0x7f0b10162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58066/0 (socket says 172.21.15.69:58066) 2024-04-24T08:15:51.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.419+0000 7f0b0f400640 1 -- 172.21.15.69:0/1087968564 learned_addr learned my addr 172.21.15.69:0/1087968564 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:51.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.419+0000 7f0b0f400640 1 -- 172.21.15.69:0/1087968564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b100cfaf0 msgr2=0x7f0b100cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.419+0000 7f0b0f400640 1 --2- 172.21.15.69:0/1087968564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b100cfaf0 0x7f0b100cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.419+0000 7f0b0f400640 1 -- 172.21.15.69:0/1087968564 >> v1:172.21.15.121:6789/0 conn(0x7f0b10162980 legacy=0x7f0b1006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:51.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.419+0000 7f0b0f400640 1 -- 172.21.15.69:0/1087968564 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b10163100 con 0x7f0b1015e960 2024-04-24T08:15:51.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.420+0000 7f0b0f400640 1 --2- 172.21.15.69:0/1087968564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b1015e960 0x7f0b10162440 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f0b00009e50 tx=0x7f0b0002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87290ca113565fb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:51.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.420+0000 7f0b0e000640 1 -- 172.21.15.69:0/1087968564 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00036070 con 0x7f0b1015e960 2024-04-24T08:15:51.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.420+0000 7f0b0e000640 1 -- 172.21.15.69:0/1087968564 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b00031600 con 0x7f0b1015e960 2024-04-24T08:15:51.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.420+0000 7f0b0e000640 1 -- 172.21.15.69:0/1087968564 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b000319d0 con 0x7f0b1015e960 2024-04-24T08:15:51.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.421+0000 7f0b15200640 1 -- 172.21.15.69:0/1087968564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b1015e960 msgr2=0x7f0b10162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.421+0000 7f0b15200640 1 --2- 172.21.15.69:0/1087968564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b1015e960 0x7f0b10162440 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f0b00009e50 tx=0x7f0b0002a6d0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.421+0000 7f0b15200640 1 -- 172.21.15.69:0/1087968564 shutdown_connections 2024-04-24T08:15:51.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.421+0000 7f0b15200640 1 --2- 172.21.15.69:0/1087968564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b100cfaf0 0x7f0b100cffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.421+0000 7f0b15200640 1 --2- 172.21.15.69:0/1087968564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b1015e960 0x7f0b10162440 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.421+0000 7f0b15200640 1 -- 172.21.15.69:0/1087968564 >> 172.21.15.69:0/1087968564 conn(0x7f0b1009d900 msgr2=0x7f0b1008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:51.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.421+0000 7f0b15200640 1 -- 172.21.15.69:0/1087968564 shutdown_connections 2024-04-24T08:15:51.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.421+0000 7f0b15200640 1 -- 172.21.15.69:0/1087968564 wait complete. 2024-04-24T08:15:51.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.423+0000 7f0b15200640 1 Processor -- start 2024-04-24T08:15:51.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.423+0000 7f0b15200640 1 -- start start 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.424+0000 7f0b15200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b100cfaf0 0x7f0b1015f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.424+0000 7f0b15200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b1015e960 0x7f0b101f4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.424+0000 7f0b15200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b10162980 0x7f0b100b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.424+0000 7f0b15200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1006d6c0 con 0x7f0b10162980 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.424+0000 7f0b15200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0b100cf5d0 con 0x7f0b100cfaf0 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.424+0000 7f0b15200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0b100b6170 con 0x7f0b1015e960 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.424+0000 7f0b0fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b10162980 0x7f0b100b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.424+0000 7f0b0fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b10162980 0x7f0b100b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58082/0 (socket says 172.21.15.69:58082) 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.424+0000 7f0b0fe00640 1 -- 172.21.15.69:0/1681983086 learned_addr learned my addr 172.21.15.69:0/1681983086 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b0f400640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b100cfaf0 0x7f0b1015f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b0ea00640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b1015e960 0x7f0b101f4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b0fe00640 1 -- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b1015e960 msgr2=0x7f0b101f4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b0fe00640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b1015e960 0x7f0b101f4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b0fe00640 1 -- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b100cfaf0 msgr2=0x7f0b1015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b0fe00640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b100cfaf0 0x7f0b1015f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b0fe00640 1 -- 172.21.15.69:0/1681983086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b10160a60 con 0x7f0b10162980 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b0fe00640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b10162980 0x7f0b100b21b0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800b450 tx=0x7f0b0800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b07400640 1 -- 172.21.15.69:0/1681983086 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08015020 con 0x7f0b10162980 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b07400640 1 -- 172.21.15.69:0/1681983086 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b08011890 con 0x7f0b10162980 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b15200640 1 -- 172.21.15.69:0/1681983086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b10160d40 con 0x7f0b10162980 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b07400640 1 -- 172.21.15.69:0/1681983086 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b08011c60 con 0x7f0b10162980 2024-04-24T08:15:51.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.425+0000 7f0b15200640 1 -- 172.21.15.69:0/1681983086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b100c81d0 con 0x7f0b10162980 2024-04-24T08:15:51.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.426+0000 7f0b15200640 1 -- 172.21.15.69:0/1681983086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b10075d50 con 0x7f0b10162980 2024-04-24T08:15:51.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.427+0000 7f0b07400640 1 -- 172.21.15.69:0/1681983086 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b08018020 con 0x7f0b10162980 2024-04-24T08:15:51.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.428+0000 7f0b07400640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0af006c8f0 0x7f0af006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:51.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.428+0000 7f0b07400640 1 -- 172.21.15.69:0/1681983086 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f0b0808be00 con 0x7f0b10162980 2024-04-24T08:15:51.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.429+0000 7f0b0f400640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0af006c8f0 0x7f0af006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:51.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.430+0000 7f0b0f400640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0af006c8f0 0x7f0af006edb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0b00033000 tx=0x7f0b00009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:51.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.433+0000 7f0b07400640 1 -- 172.21.15.69:0/1681983086 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b0801f020 con 0x7f0b10162980 2024-04-24T08:15:51.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.660+0000 7f0b15200640 1 -- 172.21.15.69:0/1681983086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0b100d1dc0 con 0x7f0b10162980 2024-04-24T08:15:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.661+0000 7f0b07400640 1 -- 172.21.15.69:0/1681983086 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v67) ==== 53+0+73 (secure 0 0 0) 0x7f0b0800d040 con 0x7f0b10162980 2024-04-24T08:15:51.662 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:15:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.663+0000 7f0b15200640 1 -- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0af006c8f0 msgr2=0x7f0af006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.663+0000 7f0b15200640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0af006c8f0 0x7f0af006edb0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0b00033000 tx=0x7f0b00009000 comp rx=0 tx=0).stop 2024-04-24T08:15:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.663+0000 7f0b15200640 1 -- 172.21.15.69:0/1681983086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b10162980 msgr2=0x7f0b100b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.663+0000 7f0b15200640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b10162980 0x7f0b100b21b0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f0b0800b450 tx=0x7f0b0800b910 comp rx=0 tx=0).stop 2024-04-24T08:15:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.663+0000 7f0b15200640 1 -- 172.21.15.69:0/1681983086 shutdown_connections 2024-04-24T08:15:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.663+0000 7f0b15200640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0af006c8f0 0x7f0af006edb0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.663+0000 7f0b15200640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b10162980 0x7f0b100b21b0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.663+0000 7f0b15200640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b1015e960 0x7f0b101f4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.663+0000 7f0b15200640 1 --2- 172.21.15.69:0/1681983086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b100cfaf0 0x7f0b1015f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.663+0000 7f0b15200640 1 -- 172.21.15.69:0/1681983086 >> 172.21.15.69:0/1681983086 conn(0x7f0b1009d900 msgr2=0x7f0b100b4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.664+0000 7f0b15200640 1 -- 172.21.15.69:0/1681983086 shutdown_connections 2024-04-24T08:15:51.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51.664+0000 7f0b15200640 1 -- 172.21.15.69:0/1681983086 wait complete. 2024-04-24T08:15:51.683 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:15:51.683 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:15:51.683 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:15:51.683 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:15:51.683 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:51.683 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:15:51.683 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:15:51.683 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:15:51.683 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:15:51.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:51.684 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:15:51.734 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:15:51.734 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:15:51.734 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:15:51.734 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:51.734 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:15:51.734 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:51.734 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:15:51.734 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42228sec preferred_lft 42228sec 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:51.735 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:15:51.736 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:15:51.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:51.738 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:15:51.813 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:15:51.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:51.817 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:15:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:15:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:15:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:15:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:15:51.868 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:15:51.869 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:15:51.869 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:15:51.869 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:15:51.869 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:15:51.869 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:15:51.869 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:15:51.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:15:52.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:15:52.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:52.072 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:15:52.118 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:15:52.119 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:15:52.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:52.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:15:52.196 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:15:52.196 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:15:52.196 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:52.247 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:15:52.248 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:15:52.248 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:52.300 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:15:52.374 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:15:52.426 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:15:52.426 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:15:52.426 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:15:52.426 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:15:52.426 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:15:52.426 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:15:52.426 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:15:52.426 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:15:52.426 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:15:52.427 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:15:52.428 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:15:52.428 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:15:52.428 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:15:52.428 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:15:52.428 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:15:52.428 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:52.428 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:52.428 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:52.428 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:52.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:52.429 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:15:52.474 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:15:52.475 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:15:52.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:15:52.584 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:15:52.623 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:15:52.623 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:15:52.623 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:15:52.623 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:15:52.623 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:15:52.623 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:15:52.623 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:15:52.623 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:15:52.624 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:52.625 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:15:52.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:52.626 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:15:52.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:15:52.683+0000 7f3123bb4480 -1 init, newargv = 0x5604cdfed4d0 newargc=15 2024-04-24T08:15:52.685 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[64754]: starting ceph client 2024-04-24T08:15:52.694 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[64754]: starting fuse 2024-04-24T08:15:52.710 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:15:52.710 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:15:52.710 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:15:52.711 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:15:52.712 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:15:52.898 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.64754.asok 2024-04-24T08:15:52.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:15:52.915 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.64754.asok 2024-04-24T08:15:52.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:52.916 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.64754.asok status 2024-04-24T08:15:53.135 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "64754", 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 7911, 2024-04-24T08:15:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 7911 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1314166299 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1314166299 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:15:53.137 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.7911 192.168.0.1:0/1314166299", 2024-04-24T08:15:53.138 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/1314166299", 2024-04-24T08:15:53.138 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:15:53.138 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 67, 2024-04-24T08:15:53.138 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 83, 2024-04-24T08:15:53.138 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:15:53.138 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:15:53.138 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:15:53.138 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:15:53.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:53.150 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:15:53.150 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:53.201 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:15:53.201 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:53.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:53.202 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:15:53.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:53.287 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat testdir/hardlink) 2024-04-24T08:15:53.345 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:15:53.344+0000 7faf1f600640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-04-24T08:15:53.347 INFO:teuthology.orchestra.run.smithi069.stderr:stat: cannot statx 'testdir/hardlink': Input/output error 2024-04-24T08:15:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:15:53.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:53.349 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive 2024-04-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.554+0000 7f7d6de00640 1 Processor -- start 2024-04-24T08:15:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.555+0000 7f7d6de00640 1 -- start start 2024-04-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.556+0000 7f7d6de00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d6815e930 0x7f7d681623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.556+0000 7f7d6de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680af5b0 0x7f7d680cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.556+0000 7f7d6de00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d680cce30 0x7f7d680ce020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.556+0000 7f7d6de00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7d6815ed10 con 0x7f7d6815e930 2024-04-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.556+0000 7f7d6de00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7d680cacf0 con 0x7f7d680cce30 2024-04-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.556+0000 7f7d6de00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d680cd2c0 con 0x7f7d680af5b0 2024-04-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.556+0000 7f7d6ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d6815e930 0x7f7d681623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.557+0000 7f7d6ca00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7d6815e930 0x7f7d681623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58098/0 (socket says 172.21.15.69:58098) 2024-04-24T08:15:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.557+0000 7f7d6ca00640 1 -- 172.21.15.69:0/1961509959 learned_addr learned my addr 172.21.15.69:0/1961509959 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.557+0000 7f7d6ca00640 1 -- 172.21.15.69:0/1961509959 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680af5b0 msgr2=0x7f7d680cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.557+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/1961509959 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680af5b0 0x7f7d680cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.557+0000 7f7d6ca00640 1 -- 172.21.15.69:0/1961509959 >> v2:172.21.15.121:3300/0 conn(0x7f7d680cce30 msgr2=0x7f7d680ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.557+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/1961509959 >> v2:172.21.15.121:3300/0 conn(0x7f7d680cce30 0x7f7d680ce020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.557+0000 7f7d6ca00640 1 -- 172.21.15.69:0/1961509959 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d681632b0 con 0x7f7d6815e930 2024-04-24T08:15:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.558+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/1961509959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d6815e930 0x7f7d681623a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7d6809deb0 tx=0x7f7d5802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a29139994e9a08b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.558+0000 7f7d67e00640 1 -- 172.21.15.69:0/1961509959 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58037070 con 0x7f7d6815e930 2024-04-24T08:15:53.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.558+0000 7f7d67e00640 1 -- 172.21.15.69:0/1961509959 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d58032440 con 0x7f7d6815e930 2024-04-24T08:15:53.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.558+0000 7f7d67e00640 1 -- 172.21.15.69:0/1961509959 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58032810 con 0x7f7d6815e930 2024-04-24T08:15:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.559+0000 7f7d6de00640 1 -- 172.21.15.69:0/1961509959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d6815e930 msgr2=0x7f7d681623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.559+0000 7f7d6de00640 1 --2- 172.21.15.69:0/1961509959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d6815e930 0x7f7d681623a0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f7d6809deb0 tx=0x7f7d5802a180 comp rx=0 tx=0).stop 2024-04-24T08:15:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.559+0000 7f7d6de00640 1 -- 172.21.15.69:0/1961509959 shutdown_connections 2024-04-24T08:15:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.559+0000 7f7d6de00640 1 --2- 172.21.15.69:0/1961509959 >> v2:172.21.15.121:3300/0 conn(0x7f7d680cce30 0x7f7d680ce020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.559+0000 7f7d6de00640 1 --2- 172.21.15.69:0/1961509959 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d680af5b0 0x7f7d680cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.559+0000 7f7d6de00640 1 --2- 172.21.15.69:0/1961509959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d6815e930 0x7f7d681623a0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.559+0000 7f7d6de00640 1 -- 172.21.15.69:0/1961509959 >> 172.21.15.69:0/1961509959 conn(0x7f7d680b1c50 msgr2=0x7f7d680b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.559+0000 7f7d6de00640 1 -- 172.21.15.69:0/1961509959 shutdown_connections 2024-04-24T08:15:53.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.559+0000 7f7d6de00640 1 -- 172.21.15.69:0/1961509959 wait complete. 2024-04-24T08:15:53.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.561+0000 7f7d6de00640 1 Processor -- start 2024-04-24T08:15:53.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.561+0000 7f7d6de00640 1 -- start start 2024-04-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.562+0000 7f7d6de00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680af5b0 0x7f7d6815d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.562+0000 7f7d6de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d680cce30 0x7f7d680b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.562+0000 7f7d6de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d6815e930 0x7f7d680b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.562+0000 7f7d6de00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d68162fd0 con 0x7f7d680cce30 2024-04-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.562+0000 7f7d6de00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7d68162cf0 con 0x7f7d680af5b0 2024-04-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.562+0000 7f7d6de00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d680ca920 con 0x7f7d6815e930 2024-04-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.562+0000 7f7d5fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d680cce30 0x7f7d680b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.562+0000 7f7d5fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d680cce30 0x7f7d680b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58102/0 (socket says 172.21.15.69:58102) 2024-04-24T08:15:53.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.562+0000 7f7d5fe00640 1 -- 172.21.15.69:0/4253063245 learned_addr learned my addr 172.21.15.69:0/4253063245 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.562+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680af5b0 0x7f7d6815d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d6d400640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d6815e930 0x7f7d680b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d5fe00640 1 -- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d6815e930 msgr2=0x7f7d680b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d5fe00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d6815e930 0x7f7d680b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d5fe00640 1 -- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680af5b0 msgr2=0x7f7d6815d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d5fe00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680af5b0 0x7f7d6815d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d5fe00640 1 -- 172.21.15.69:0/4253063245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d680b49f0 con 0x7f7d680cce30 2024-04-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d6d400640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d6815e930 0x7f7d680b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d5fe00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d680cce30 0x7f7d680b3f30 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000b4d0 tx=0x7f7d5000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d65600640 1 -- 172.21.15.69:0/4253063245 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50014070 con 0x7f7d680cce30 2024-04-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d65600640 1 -- 172.21.15.69:0/4253063245 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d50010870 con 0x7f7d680cce30 2024-04-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d65600640 1 -- 172.21.15.69:0/4253063245 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50010c60 con 0x7f7d680cce30 2024-04-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d6de00640 1 -- 172.21.15.69:0/4253063245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d680b4cd0 con 0x7f7d680cce30 2024-04-24T08:15:53.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.563+0000 7f7d6de00640 1 -- 172.21.15.69:0/4253063245 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d681e8430 con 0x7f7d680cce30 2024-04-24T08:15:53.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.566+0000 7f7d65600640 1 -- 172.21.15.69:0/4253063245 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d50017070 con 0x7f7d680cce30 2024-04-24T08:15:53.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.566+0000 7f7d65600640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d4406c8f0 0x7f7d4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.567+0000 7f7d65600640 1 -- 172.21.15.69:0/4253063245 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f7d5008ba10 con 0x7f7d680cce30 2024-04-24T08:15:53.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.567+0000 7f7d721f6740 1 Processor -- start 2024-04-24T08:15:53.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.567+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d4406c8f0 0x7f7d4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.567+0000 7f7d721f6740 1 -- start start 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.568+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.568+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125d4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.568+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125d4fc0 0x5605125a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.568+0000 7f7d721f6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56051250df50 con 0x5605125f25e0 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.568+0000 7f7d721f6740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5605124b42c0 con 0x5605125d4fc0 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.568+0000 7f7d721f6740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x560512509f10 con 0x5605125d5700 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.568+0000 7f7d6ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.568+0000 7f7d6ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125a4f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58118/0 (socket says 172.21.15.69:58118) 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.568+0000 7f7d6ca00640 1 -- 172.21.15.69:0/244998561 learned_addr learned my addr 172.21.15.69:0/244998561 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d4406c8f0 0x7f7d4406edb0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7d58034000 tx=0x7f7d5802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d6d400640 1 --2- 172.21.15.69:0/244998561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125d4fc0 0x5605125a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d5fe00640 1 --2- 172.21.15.69:0/244998561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125d4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d6ca00640 1 -- 172.21.15.69:0/244998561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 msgr2=0x5605125d4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/244998561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125d4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d6ca00640 1 -- 172.21.15.69:0/244998561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125d4fc0 msgr2=0x5605125a9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/244998561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125d4fc0 0x5605125a9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d6ca00640 1 -- 172.21.15.69:0/244998561 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605122ec7b0 con 0x5605125f25e0 2024-04-24T08:15:53.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/244998561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125a4f70 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f7d58009960 tx=0x7f7d58008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88a6d736e092c7e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d5d200640 1 -- 172.21.15.69:0/244998561 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58037040 con 0x5605125f25e0 2024-04-24T08:15:53.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d5d200640 1 -- 172.21.15.69:0/244998561 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5804c070 con 0x5605125f25e0 2024-04-24T08:15:53.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.569+0000 7f7d5d200640 1 -- 172.21.15.69:0/244998561 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5803c440 con 0x5605125f25e0 2024-04-24T08:15:53.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.570+0000 7f7d721f6740 1 -- 172.21.15.69:0/244998561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 msgr2=0x5605125a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.570+0000 7f7d721f6740 1 --2- 172.21.15.69:0/244998561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125a4f70 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f7d58009960 tx=0x7f7d58008030 comp rx=0 tx=0).stop 2024-04-24T08:15:53.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.570+0000 7f7d721f6740 1 -- 172.21.15.69:0/244998561 shutdown_connections 2024-04-24T08:15:53.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.570+0000 7f7d721f6740 1 --2- 172.21.15.69:0/244998561 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125d4fc0 0x5605125a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.570+0000 7f7d721f6740 1 --2- 172.21.15.69:0/244998561 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125d4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.570+0000 7f7d721f6740 1 --2- 172.21.15.69:0/244998561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125a4f70 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.570+0000 7f7d721f6740 1 -- 172.21.15.69:0/244998561 >> 172.21.15.69:0/244998561 conn(0x5605125a71e0 msgr2=0x5605125efa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.570+0000 7f7d721f6740 1 -- 172.21.15.69:0/244998561 shutdown_connections 2024-04-24T08:15:53.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.570+0000 7f7d721f6740 1 -- 172.21.15.69:0/244998561 wait complete. 2024-04-24T08:15:53.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.571+0000 7f7d721f6740 1 -- start start 2024-04-24T08:15:53.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.571+0000 7f7d721f6740 1 Processor -- start 2024-04-24T08:15:53.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.572+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.572+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.572+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x5605125e6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.572+0000 7f7d721f6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5605125a9390 con 0x5605125f25e0 2024-04-24T08:15:53.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.572+0000 7f7d721f6740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5605124c7680 con 0x5605125c2ab0 2024-04-24T08:15:53.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.572+0000 7f7d721f6740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5605125a8780 con 0x5605125d5700 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.572+0000 7f7d721f6740 10 client.0 _ll_register_callbacks cb 0x5605125044f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.572+0000 7f7d5fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.572+0000 7f7d5fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125e84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58126/0 (socket says 172.21.15.69:58126) 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.572+0000 7f7d5fe00640 1 -- 172.21.15.69:0/2240362298 learned_addr learned my addr 172.21.15.69:0/2240362298 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d6d400640 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x5605125e6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d5fe00640 1 -- 172.21.15.69:0/2240362298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 msgr2=0x5605125f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d5fe00640 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d5fe00640 1 -- 172.21.15.69:0/2240362298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 msgr2=0x5605125e6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d5fe00640 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x5605125e6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d5fe00640 1 -- 172.21.15.69:0/2240362298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56051234cf40 con 0x5605125f25e0 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d5fe00640 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125e84f0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f7d50089de0 tx=0x7f7d50059700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b80631b2861d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d55600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d55600640 1 -- 172.21.15.69:0/2240362298 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50088060 con 0x5605125f25e0 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d55600640 1 -- 172.21.15.69:0/2240362298 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f7d50089910 con 0x5605125f25e0 2024-04-24T08:15:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.573+0000 7f7d721f6740 1 -- 172.21.15.69:0/2240362298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d5804c070 con 0x5605125f25e0 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d55600640 1 -- 172.21.15.69:0/2240362298 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f7d50076050 con 0x5605125f25e0 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d721f6740 10 client.7923 fetch_fsmap learned FSMap version 67 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d721f6740 1 -- 172.21.15.69:0/2240362298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x5605121e00a0 con 0x5605125f25e0 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d55600640 1 -- 172.21.15.69:0/2240362298 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5804c070 con 0x5605125f25e0 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d55600640 1 -- 172.21.15.69:0/2240362298 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f7d5005eb80 con 0x5605125f25e0 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d721f6740 10 client.7923 fetch_fsmap finished waiting for FSMap version 67 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d721f6740 10 client.7923 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d721f6740 20 client.7923 populate_metadata read hostname 'smithi069' 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d721f6740 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5605125c1e80 0x5605125c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d721f6740 4 client.7923 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d721f6740 1 -- 172.21.15.69:0/2240362298 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56051251b970 con 0x5605125c1e80 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.574+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5605125c1e80 0x5605125c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.575+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5605125c1e80 0x5605125c22f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3a5cd8ca61f052a server_cookie=298d146e343cdfb7 in_seq=0 out_seq=0 2024-04-24T08:15:53.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.575+0000 7f7d55600640 10 client.7923 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:53.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d55600640 1 -- 172.21.15.69:0/2240362298 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56051251b970 con 0x5605125c1e80 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d55600640 10 client.7923 handle_command_reply: tid=0 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 client.7923 shutdown 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 -- 172.21.15.69:0/2240362298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 msgr2=0x5605125e84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125e84f0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f7d50089de0 tx=0x7f7d50059700 comp rx=0 tx=0).stop 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 -- 172.21.15.69:0/2240362298 shutdown_connections 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5605125c1e80 0x5605125c22f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125e84f0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x5605125e6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 --2- 172.21.15.69:0/2240362298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d55600640 0 client.7923 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 -- 172.21.15.69:0/2240362298 >> 172.21.15.69:0/2240362298 conn(0x5605125d4fc0 msgr2=0x5605125a5ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 -- 172.21.15.69:0/2240362298 shutdown_connections 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 1 -- 172.21.15.69:0/2240362298 wait complete. 2024-04-24T08:15:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.576+0000 7f7d721f6740 20 client.7923 trim_cache size 0 max 16384 2024-04-24T08:15:53.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.596+0000 7f7d721f6740 1 Processor -- start 2024-04-24T08:15:53.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.596+0000 7f7d721f6740 1 -- start start 2024-04-24T08:15:53.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.597+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x5605125ef880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.597+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125d5700 0x5605125efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.597+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125f25e0 0x5605125f1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.597+0000 7f7d721f6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d68200c00 con 0x5605125d5700 2024-04-24T08:15:53.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.597+0000 7f7d721f6740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7d681e99d0 con 0x5605125c2ab0 2024-04-24T08:15:53.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.597+0000 7f7d721f6740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d681f8890 con 0x5605125f25e0 2024-04-24T08:15:53.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.597+0000 7f7d5fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125d5700 0x5605125efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.597+0000 7f7d5fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125d5700 0x5605125efdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58136/0 (socket says 172.21.15.69:58136) 2024-04-24T08:15:53.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.597+0000 7f7d5fe00640 1 -- 172.21.15.69:0/1079506537 learned_addr learned my addr 172.21.15.69:0/1079506537 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d6d400640 1 --2- 172.21.15.69:0/1079506537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125f25e0 0x5605125f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/1079506537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x5605125ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d5fe00640 1 -- 172.21.15.69:0/1079506537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125f25e0 msgr2=0x5605125f1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d5fe00640 1 --2- 172.21.15.69:0/1079506537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125f25e0 0x5605125f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d5fe00640 1 -- 172.21.15.69:0/1079506537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 msgr2=0x5605125ef880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d5fe00640 1 --2- 172.21.15.69:0/1079506537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x5605125ef880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d5fe00640 1 -- 172.21.15.69:0/1079506537 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5605125c28d0 con 0x5605125d5700 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d5fe00640 1 --2- 172.21.15.69:0/1079506537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125d5700 0x5605125efdc0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f7d500595d0 tx=0x7f7d5006f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a510adbbbfbbe134 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d57e00640 1 -- 172.21.15.69:0/1079506537 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5001e040 con 0x5605125d5700 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d57e00640 1 -- 172.21.15.69:0/1079506537 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5005eb80 con 0x5605125d5700 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.598+0000 7f7d57e00640 1 -- 172.21.15.69:0/1079506537 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50058cb0 con 0x5605125d5700 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.599+0000 7f7d721f6740 1 -- 172.21.15.69:0/1079506537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125d5700 msgr2=0x5605125efdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.599+0000 7f7d721f6740 1 --2- 172.21.15.69:0/1079506537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125d5700 0x5605125efdc0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f7d500595d0 tx=0x7f7d5006f000 comp rx=0 tx=0).stop 2024-04-24T08:15:53.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.599+0000 7f7d721f6740 1 -- 172.21.15.69:0/1079506537 shutdown_connections 2024-04-24T08:15:53.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.599+0000 7f7d721f6740 1 --2- 172.21.15.69:0/1079506537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125d5700 0x5605125efdc0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.599+0000 7f7d721f6740 1 --2- 172.21.15.69:0/1079506537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125f25e0 0x5605125f1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.599+0000 7f7d721f6740 1 --2- 172.21.15.69:0/1079506537 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x5605125ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.599+0000 7f7d721f6740 1 -- 172.21.15.69:0/1079506537 >> 172.21.15.69:0/1079506537 conn(0x5605125d4fc0 msgr2=0x5605125d84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.599+0000 7f7d721f6740 1 -- 172.21.15.69:0/1079506537 shutdown_connections 2024-04-24T08:15:53.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.599+0000 7f7d721f6740 1 -- 172.21.15.69:0/1079506537 wait complete. 2024-04-24T08:15:53.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.600+0000 7f7d721f6740 1 -- start start 2024-04-24T08:15:53.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.600+0000 7f7d721f6740 1 Processor -- start 2024-04-24T08:15:53.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.601+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x560512675970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.601+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.601+0000 7f7d721f6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.601+0000 7f7d721f6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d681e9380 con 0x5605125f25e0 2024-04-24T08:15:53.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.601+0000 7f7d721f6740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5605125a6ac0 con 0x5605125c2ab0 2024-04-24T08:15:53.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.601+0000 7f7d721f6740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5605125e5800 con 0x5605125d5700 2024-04-24T08:15:53.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.601+0000 7f7d721f6740 10 client.0 _ll_register_callbacks cb 0x7f7d50059410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:53.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.601+0000 7f7d6d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.601+0000 7f7d6d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125f2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58148/0 (socket says 172.21.15.69:58148) 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.601+0000 7f7d6d400640 1 -- 172.21.15.69:0/1624564888 learned_addr learned my addr 172.21.15.69:0/1624564888 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x560512675970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d6d400640 1 -- 172.21.15.69:0/1624564888 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 msgr2=0x5605125aaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d6d400640 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d6d400640 1 -- 172.21.15.69:0/1624564888 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 msgr2=0x560512675970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d6d400640 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x560512675970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d6d400640 1 -- 172.21.15.69:0/1624564888 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5605125ab3f0 con 0x5605125f25e0 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x560512675970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d6d400640 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125f2ea0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f7d60010a30 tx=0x7f7d60010ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b52e5db11af226 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d55600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d55600640 1 -- 172.21.15.69:0/1624564888 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6001e060 con 0x5605125f25e0 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d55600640 1 -- 172.21.15.69:0/1624564888 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7f7d6000ed30 con 0x5605125f25e0 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d721f6740 1 -- 172.21.15.69:0/1624564888 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d5005eb80 con 0x5605125f25e0 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.602+0000 7f7d55600640 1 -- 172.21.15.69:0/1624564888 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7f7d60024050 con 0x5605125f25e0 2024-04-24T08:15:53.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d721f6740 10 client.7929 fetch_fsmap learned FSMap version 67 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d721f6740 1 -- 172.21.15.69:0/1624564888 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x5605125ef040 con 0x5605125f25e0 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d55600640 1 -- 172.21.15.69:0/1624564888 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5005eb80 con 0x5605125f25e0 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d55600640 1 -- 172.21.15.69:0/1624564888 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7f7d60027020 con 0x5605125f25e0 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d721f6740 10 client.7929 fetch_fsmap finished waiting for FSMap version 67 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d721f6740 10 client.7929 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d721f6740 20 client.7929 populate_metadata read hostname 'smithi069' 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d721f6740 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5605125f33e0 0x5605126126b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d721f6740 4 client.7929 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}] 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d721f6740 1 -- 172.21.15.69:0/1624564888 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}) -- 0x5605125157e0 con 0x5605125f33e0 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.603+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5605125f33e0 0x5605126126b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.604+0000 7f7d6ca00640 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5605125f33e0 0x5605126126b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b9b03ae8db322f9 server_cookie=9d0f5534bf0b2cf8 in_seq=0 out_seq=0 2024-04-24T08:15:53.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.604+0000 7f7d55600640 10 client.7929 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:53.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.604+0000 7f7d55600640 1 -- 172.21.15.69:0/1624564888 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5605125157e0 con 0x5605125f33e0 2024-04-24T08:15:53.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.604+0000 7f7d55600640 10 client.7929 handle_command_reply: tid=0 2024-04-24T08:15:53.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.604+0000 7f7d721f6740 1 client.7929 shutdown 2024-04-24T08:15:53.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.604+0000 7f7d721f6740 1 -- 172.21.15.69:0/1624564888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 msgr2=0x5605125f2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.604+0000 7f7d721f6740 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125f2ea0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f7d60010a30 tx=0x7f7d60010ef0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.604+0000 7f7d721f6740 1 -- 172.21.15.69:0/1624564888 shutdown_connections 2024-04-24T08:15:53.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d721f6740 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5605125f33e0 0x5605126126b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d721f6740 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5605125f25e0 0x5605125f2ea0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d721f6740 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5605125d5700 0x5605125aaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d721f6740 1 --2- 172.21.15.69:0/1624564888 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5605125c2ab0 0x560512675970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d55600640 0 client.7929 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:53.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d721f6740 1 -- 172.21.15.69:0/1624564888 >> 172.21.15.69:0/1624564888 conn(0x5605125d4fc0 msgr2=0x5605125f1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.609 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:15:53.605+0000 7faf1f600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 ap=1 f(v0 m2024-04-24T08:15:32.898213+0000 1=1+0) n(v0 rc2024-04-24T08:15:33.020316+0000 2=1+1) | dirfrag=0 authpin=1 scrubqueue=0 0x55f9f868c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-04-24T08:15:32.898213+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-04-24T08:15:33.020316+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-04-24T08:15:53.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d721f6740 1 -- 172.21.15.69:0/1624564888 shutdown_connections 2024-04-24T08:15:53.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d721f6740 1 -- 172.21.15.69:0/1624564888 wait complete. 2024-04-24T08:15:53.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d721f6740 20 client.7929 trim_cache size 0 max 16384 2024-04-24T08:15:53.610 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:15:53.610 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-04-24T08:15:53.610 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "3e23f37a-33f3-4048-870d-6ba0d1fe4443", 2024-04-24T08:15:53.610 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-04-24T08:15:53.610 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:15:53.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d6de00640 1 -- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d4406c8f0 msgr2=0x7f7d4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d6de00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d4406c8f0 0x7f7d4406edb0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7d58034000 tx=0x7f7d5802e000 comp rx=0 tx=0).stop 2024-04-24T08:15:53.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d6de00640 1 -- 172.21.15.69:0/4253063245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d680cce30 msgr2=0x7f7d680b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.605+0000 7f7d6de00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d680cce30 0x7f7d680b3f30 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000b4d0 tx=0x7f7d5000b990 comp rx=0 tx=0).stop 2024-04-24T08:15:53.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.606+0000 7f7d6de00640 1 -- 172.21.15.69:0/4253063245 shutdown_connections 2024-04-24T08:15:53.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.606+0000 7f7d6de00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d4406c8f0 0x7f7d4406edb0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.606+0000 7f7d6de00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d680cce30 0x7f7d680b3f30 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.606+0000 7f7d6de00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d6815e930 0x7f7d680b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.606+0000 7f7d6de00640 1 --2- 172.21.15.69:0/4253063245 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d680af5b0 0x7f7d6815d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.606+0000 7f7d6de00640 1 -- 172.21.15.69:0/4253063245 >> 172.21.15.69:0/4253063245 conn(0x7f7d680b1c50 msgr2=0x7f7d68071f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.606+0000 7f7d6de00640 1 -- 172.21.15.69:0/4253063245 shutdown_connections 2024-04-24T08:15:53.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.606+0000 7f7d6de00640 1 -- 172.21.15.69:0/4253063245 wait complete. 2024-04-24T08:15:53.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:53.624 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-24T08:15:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.813+0000 7ff82f400640 1 Processor -- start 2024-04-24T08:15:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.813+0000 7ff82f400640 1 -- start start 2024-04-24T08:15:53.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.815+0000 7ff82f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8280b3e90 0x7ff8280b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.815+0000 7ff82f400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff8280ce250 0x7ff8280ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.815+0000 7ff82f400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff82815cc00 con 0x7ff82815c820 2024-04-24T08:15:53.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.815+0000 7ff82f400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff8280b4be0 con 0x7ff8280b3e90 2024-04-24T08:15:53.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.815+0000 7ff82f400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff82806ed30 con 0x7ff8280ce250 2024-04-24T08:15:53.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.815+0000 7ff827e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8280b3e90 0x7ff8280b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.815+0000 7ff82ca00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff82815c820 0x7ff82806f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46266/0 (socket says 172.21.15.69:46266) 2024-04-24T08:15:53.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.815+0000 7ff82ca00640 1 -- 172.21.15.69:0/2564776678 learned_addr learned my addr 172.21.15.69:0/2564776678 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:15:53.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.816+0000 7ff827e00640 1 -- 172.21.15.69:0/2564776678 >> v2:172.21.15.121:3300/0 conn(0x7ff8280ce250 msgr2=0x7ff8280ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.816+0000 7ff82d400640 1 --2- 172.21.15.69:0/2564776678 >> v2:172.21.15.121:3300/0 conn(0x7ff8280ce250 0x7ff8280ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.816+0000 7ff827400640 1 -- 172.21.15.69:0/2564776678 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3605890129 0 0) 0x7ff82815cc00 con 0x7ff82815c820 2024-04-24T08:15:53.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.816+0000 7ff827e00640 1 --2- 172.21.15.69:0/2564776678 >> v2:172.21.15.121:3300/0 conn(0x7ff8280ce250 0x7ff8280ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.816+0000 7ff827e00640 1 -- 172.21.15.69:0/2564776678 >> v1:172.21.15.69:6789/0 conn(0x7ff82815c820 legacy=0x7ff82806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.816+0000 7ff827e00640 1 -- 172.21.15.69:0/2564776678 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8280b4e10 con 0x7ff8280b3e90 2024-04-24T08:15:53.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.816+0000 7ff82d400640 1 --2- 172.21.15.69:0/2564776678 >> v2:172.21.15.121:3300/0 conn(0x7ff8280ce250 0x7ff8280ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:53.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.817+0000 7ff827e00640 1 --2- 172.21.15.69:0/2564776678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 0x7ff8280b46a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff828161420 tx=0x7ff81402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddcac2855e29ff15 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.817+0000 7ff827400640 1 -- 172.21.15.69:0/2564776678 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814037070 con 0x7ff8280b3e90 2024-04-24T08:15:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.817+0000 7ff827400640 1 -- 172.21.15.69:0/2564776678 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff814032440 con 0x7ff8280b3e90 2024-04-24T08:15:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.817+0000 7ff827400640 1 -- 172.21.15.69:0/2564776678 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814032810 con 0x7ff8280b3e90 2024-04-24T08:15:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.818+0000 7ff82f400640 1 -- 172.21.15.69:0/2564776678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 msgr2=0x7ff8280b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.818+0000 7ff82f400640 1 --2- 172.21.15.69:0/2564776678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 0x7ff8280b46a0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7ff828161420 tx=0x7ff81402a1a0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.818+0000 7ff82f400640 1 -- 172.21.15.69:0/2564776678 shutdown_connections 2024-04-24T08:15:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.818+0000 7ff82f400640 1 --2- 172.21.15.69:0/2564776678 >> v2:172.21.15.121:3300/0 conn(0x7ff8280ce250 0x7ff8280ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.818+0000 7ff82f400640 1 --2- 172.21.15.69:0/2564776678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 0x7ff8280b46a0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.818+0000 7ff82f400640 1 -- 172.21.15.69:0/2564776678 >> 172.21.15.69:0/2564776678 conn(0x7ff8280d0350 msgr2=0x7ff8280d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.818+0000 7ff82f400640 1 -- 172.21.15.69:0/2564776678 shutdown_connections 2024-04-24T08:15:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.818+0000 7ff82f400640 1 -- 172.21.15.69:0/2564776678 wait complete. 2024-04-24T08:15:53.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.820+0000 7ff82f400640 1 Processor -- start 2024-04-24T08:15:53.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.820+0000 7ff82f400640 1 -- start start 2024-04-24T08:15:53.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.821+0000 7ff82f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 0x7ff82809dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.821+0000 7ff82f400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8280ce250 0x7ff828072f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.821+0000 7ff82f400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff82815c820 0x7ff828071d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.821+0000 7ff82f400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff82806f2b0 con 0x7ff8280b3e90 2024-04-24T08:15:53.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.821+0000 7ff82f400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff82806efd0 con 0x7ff8280ce250 2024-04-24T08:15:53.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.821+0000 7ff82f400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff82806fee0 con 0x7ff82815c820 2024-04-24T08:15:53.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.821+0000 7ff82ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 0x7ff82809dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.821+0000 7ff82ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 0x7ff82809dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58154/0 (socket says 172.21.15.69:58154) 2024-04-24T08:15:53.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.821+0000 7ff82ca00640 1 -- 172.21.15.69:0/3409881981 learned_addr learned my addr 172.21.15.69:0/3409881981 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff82ca00640 1 -- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff82815c820 msgr2=0x7ff828071d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff82ca00640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff82815c820 0x7ff828071d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff82ca00640 1 -- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8280ce250 msgr2=0x7ff828072f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff82ca00640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8280ce250 0x7ff828072f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff82ca00640 1 -- 172.21.15.69:0/3409881981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff82815ef10 con 0x7ff8280b3e90 2024-04-24T08:15:53.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff82ca00640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 0x7ff82809dc00 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00c500 tx=0x7ff81c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff824c00640 1 -- 172.21.15.69:0/3409881981 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c018070 con 0x7ff8280b3e90 2024-04-24T08:15:53.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff824c00640 1 -- 172.21.15.69:0/3409881981 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff81c007560 con 0x7ff8280b3e90 2024-04-24T08:15:53.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff82f400640 1 -- 172.21.15.69:0/3409881981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff82815f1b0 con 0x7ff8280b3e90 2024-04-24T08:15:53.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff824c00640 1 -- 172.21.15.69:0/3409881981 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c004a20 con 0x7ff8280b3e90 2024-04-24T08:15:53.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.822+0000 7ff82f400640 1 -- 172.21.15.69:0/3409881981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff82815f770 con 0x7ff8280b3e90 2024-04-24T08:15:53.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.825+0000 7ff824c00640 1 -- 172.21.15.69:0/3409881981 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff81c01b070 con 0x7ff8280b3e90 2024-04-24T08:15:53.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.826+0000 7ff824c00640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff80806c8f0 0x7ff80806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.826+0000 7ff824c00640 1 -- 172.21.15.69:0/3409881981 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7ff81c08aee0 con 0x7ff8280b3e90 2024-04-24T08:15:53.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.826+0000 7ff83222c740 1 Processor -- start 2024-04-24T08:15:53.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.826+0000 7ff827e00640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff80806c8f0 0x7ff80806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.826+0000 7ff83222c740 1 -- start start 2024-04-24T08:15:53.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.827+0000 7ff83222c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7632c30 0x5563f75e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.827+0000 7ff83222c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7615d50 0x5563f76150d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.827+0000 7ff83222c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615610 0x5563f75ea420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.827+0000 7ff83222c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5563f754f1f0 con 0x5563f7632c30 2024-04-24T08:15:53.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.827+0000 7ff83222c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5563f74f5350 con 0x5563f7615d50 2024-04-24T08:15:53.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.827+0000 7ff83222c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5563f754b1b0 con 0x5563f7615610 2024-04-24T08:15:53.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.827+0000 7ff827e00640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff80806c8f0 0x7ff80806edb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff814034000 tx=0x7ff81402c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.827+0000 7ff82ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7632c30 0x5563f75e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff82ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7632c30 0x5563f75e55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58168/0 (socket says 172.21.15.69:58168) 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff82ca00640 1 -- 172.21.15.69:0/73842275 learned_addr learned my addr 172.21.15.69:0/73842275 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff82d400640 1 --2- 172.21.15.69:0/73842275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615610 0x5563f75ea420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff827e00640 1 --2- 172.21.15.69:0/73842275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7615d50 0x5563f76150d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff82ca00640 1 -- 172.21.15.69:0/73842275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615610 msgr2=0x5563f75ea420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff82ca00640 1 --2- 172.21.15.69:0/73842275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615610 0x5563f75ea420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff82ca00640 1 -- 172.21.15.69:0/73842275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7615d50 msgr2=0x5563f76150d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff82ca00640 1 --2- 172.21.15.69:0/73842275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7615d50 0x5563f76150d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff82ca00640 1 -- 172.21.15.69:0/73842275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563f7549f50 con 0x5563f7632c30 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff82d400640 1 --2- 172.21.15.69:0/73842275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615610 0x5563f75ea420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff82ca00640 1 --2- 172.21.15.69:0/73842275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7632c30 0x5563f75e55c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7ff81c061250 tx=0x7ff81c058ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82d4cee6b7a1925 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff819200640 1 -- 172.21.15.69:0/73842275 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c056ea0 con 0x5563f7632c30 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff819200640 1 -- 172.21.15.69:0/73842275 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff81c055210 con 0x5563f7632c30 2024-04-24T08:15:53.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.828+0000 7ff819200640 1 -- 172.21.15.69:0/73842275 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c004dc0 con 0x5563f7632c30 2024-04-24T08:15:53.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.829+0000 7ff83222c740 1 -- 172.21.15.69:0/73842275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7632c30 msgr2=0x5563f75e55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.829+0000 7ff83222c740 1 --2- 172.21.15.69:0/73842275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7632c30 0x5563f75e55c0 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7ff81c061250 tx=0x7ff81c058ce0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.829+0000 7ff83222c740 1 -- 172.21.15.69:0/73842275 shutdown_connections 2024-04-24T08:15:53.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.829+0000 7ff83222c740 1 --2- 172.21.15.69:0/73842275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615610 0x5563f75ea420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.829+0000 7ff83222c740 1 --2- 172.21.15.69:0/73842275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7615d50 0x5563f76150d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.829+0000 7ff83222c740 1 --2- 172.21.15.69:0/73842275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7632c30 0x5563f75e55c0 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.829+0000 7ff83222c740 1 -- 172.21.15.69:0/73842275 >> 172.21.15.69:0/73842275 conn(0x5563f75e7830 msgr2=0x5563f76300d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.829+0000 7ff83222c740 1 -- 172.21.15.69:0/73842275 shutdown_connections 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.829+0000 7ff83222c740 1 -- 172.21.15.69:0/73842275 wait complete. 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.830+0000 7ff83222c740 1 -- start start 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.830+0000 7ff83222c740 1 Processor -- start 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.831+0000 7ff83222c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7615d50 0x5563f760cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.831+0000 7ff83222c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f762f540 0x5563f76300d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.831+0000 7ff83222c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7630610 0x5563f7616240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.831+0000 7ff83222c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5563f75e99e0 con 0x5563f7615d50 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.831+0000 7ff83222c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5563f7508690 con 0x5563f762f540 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.831+0000 7ff83222c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5563f75e8dd0 con 0x5563f7630610 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.831+0000 7ff83222c740 10 client.0 _ll_register_callbacks cb 0x5563f7544650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.831+0000 7ff82ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7615d50 0x5563f760cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff82ca00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7615d50 0x5563f760cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58174/0 (socket says 172.21.15.69:58174) 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff82ca00640 1 -- 172.21.15.69:0/1998277105 learned_addr learned my addr 172.21.15.69:0/1998277105 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff82ca00640 1 -- 172.21.15.69:0/1998277105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7630610 msgr2=0x5563f7616240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff82ca00640 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7630610 0x5563f7616240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff82ca00640 1 -- 172.21.15.69:0/1998277105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f762f540 msgr2=0x5563f76300d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff82ca00640 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f762f540 0x5563f76300d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff82ca00640 1 -- 172.21.15.69:0/1998277105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563f7507920 con 0x5563f7615d50 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff82ca00640 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7615d50 0x5563f760cf30 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7ff81c058850 tx=0x7ff81c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b65723834c5f987 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff80d600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff80d600640 1 -- 172.21.15.69:0/1998277105 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff81c058200 con 0x5563f7615d50 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff80d600640 1 -- 172.21.15.69:0/1998277105 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7ff81c05f150 con 0x5563f7615d50 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.832+0000 7ff83222c740 1 -- 172.21.15.69:0/1998277105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff81c055210 con 0x5563f7615d50 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff80d600640 1 -- 172.21.15.69:0/1998277105 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ff81c07f050 con 0x5563f7615d50 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff83222c740 10 client.7944 fetch_fsmap learned FSMap version 67 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff83222c740 1 -- 172.21.15.69:0/1998277105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x5563f732b7d0 con 0x5563f7615d50 2024-04-24T08:15:53.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff80d600640 1 -- 172.21.15.69:0/1998277105 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff81c055210 con 0x5563f7615d50 2024-04-24T08:15:53.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff80d600640 1 -- 172.21.15.69:0/1998277105 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7ff81c082020 con 0x5563f7615d50 2024-04-24T08:15:53.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff83222c740 10 client.7944 fetch_fsmap finished waiting for FSMap version 67 2024-04-24T08:15:53.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff83222c740 10 client.7944 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:15:53.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff83222c740 20 client.7944 populate_metadata read hostname 'smithi069' 2024-04-24T08:15:53.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff83222c740 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5563f7616c50 0x5563f7626f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff83222c740 4 client.7944 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:53.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff83222c740 1 -- 172.21.15.69:0/1998277105 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563f755cc10 con 0x5563f7616c50 2024-04-24T08:15:53.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.833+0000 7ff827e00640 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5563f7616c50 0x5563f7626f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.834+0000 7ff827e00640 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5563f7616c50 0x5563f7626f40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a42bcaad5d61d298 server_cookie=c15192cc1fb629c1 in_seq=0 out_seq=0 2024-04-24T08:15:53.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.834+0000 7ff80d600640 10 client.7944 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:53.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff80d600640 1 -- 172.21.15.69:0/1998277105 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563f755cc10 con 0x5563f7616c50 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff80d600640 10 client.7944 handle_command_reply: tid=0 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 client.7944 shutdown 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 -- 172.21.15.69:0/1998277105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7615d50 msgr2=0x5563f760cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7615d50 0x5563f760cf30 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7ff81c058850 tx=0x7ff81c000ed0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 -- 172.21.15.69:0/1998277105 shutdown_connections 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5563f7616c50 0x5563f7626f40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7630610 0x5563f7616240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f762f540 0x5563f76300d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 --2- 172.21.15.69:0/1998277105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7615d50 0x5563f760cf30 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff80d600640 0 client.7944 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 -- 172.21.15.69:0/1998277105 >> 172.21.15.69:0/1998277105 conn(0x5563f7615610 msgr2=0x5563f75e4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 -- 172.21.15.69:0/1998277105 shutdown_connections 2024-04-24T08:15:53.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 1 -- 172.21.15.69:0/1998277105 wait complete. 2024-04-24T08:15:53.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.835+0000 7ff83222c740 20 client.7944 trim_cache size 0 max 16384 2024-04-24T08:15:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.854+0000 7ff83222c740 1 Processor -- start 2024-04-24T08:15:53.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.855+0000 7ff83222c740 1 -- start start 2024-04-24T08:15:53.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.855+0000 7ff83222c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615d50 0x5563f760c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.855+0000 7ff83222c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f762f540 0x5563f7626f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.855+0000 7ff83222c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7630610 0x5563f7627480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.855+0000 7ff83222c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff828200c30 con 0x5563f762f540 2024-04-24T08:15:53.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.855+0000 7ff83222c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff828146a60 con 0x5563f7630610 2024-04-24T08:15:53.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.855+0000 7ff83222c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff828148170 con 0x5563f7615d50 2024-04-24T08:15:53.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff827e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f762f540 0x5563f7626f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff827e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f762f540 0x5563f7626f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58188/0 (socket says 172.21.15.69:58188) 2024-04-24T08:15:53.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff827e00640 1 -- 172.21.15.69:0/3155181747 learned_addr learned my addr 172.21.15.69:0/3155181747 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff82ca00640 1 --2- 172.21.15.69:0/3155181747 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615d50 0x5563f760c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff82d400640 1 --2- 172.21.15.69:0/3155181747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7630610 0x5563f7627480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff827e00640 1 -- 172.21.15.69:0/3155181747 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615d50 msgr2=0x5563f760c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff827e00640 1 --2- 172.21.15.69:0/3155181747 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615d50 0x5563f760c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff827e00640 1 -- 172.21.15.69:0/3155181747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7630610 msgr2=0x5563f7627480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff827e00640 1 --2- 172.21.15.69:0/3155181747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7630610 0x5563f7627480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff827e00640 1 -- 172.21.15.69:0/3155181747 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563f72613f0 con 0x5563f762f540 2024-04-24T08:15:53.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.856+0000 7ff827e00640 1 --2- 172.21.15.69:0/3155181747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f762f540 0x5563f7626f40 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ff81402a1e0 tx=0x7ff814008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce973453178f66f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff80fe00640 1 -- 172.21.15.69:0/3155181747 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814050070 con 0x5563f762f540 2024-04-24T08:15:53.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff80fe00640 1 -- 172.21.15.69:0/3155181747 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff814032df0 con 0x5563f762f540 2024-04-24T08:15:53.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff80fe00640 1 -- 172.21.15.69:0/3155181747 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814037040 con 0x5563f762f540 2024-04-24T08:15:53.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff83222c740 1 -- 172.21.15.69:0/3155181747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f762f540 msgr2=0x5563f7626f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff83222c740 1 --2- 172.21.15.69:0/3155181747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f762f540 0x5563f7626f40 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7ff81402a1e0 tx=0x7ff814008030 comp rx=0 tx=0).stop 2024-04-24T08:15:53.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff83222c740 1 -- 172.21.15.69:0/3155181747 shutdown_connections 2024-04-24T08:15:53.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff83222c740 1 --2- 172.21.15.69:0/3155181747 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f762f540 0x5563f7626f40 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff83222c740 1 --2- 172.21.15.69:0/3155181747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7630610 0x5563f7627480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff83222c740 1 --2- 172.21.15.69:0/3155181747 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f7615d50 0x5563f760c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff83222c740 1 -- 172.21.15.69:0/3155181747 >> 172.21.15.69:0/3155181747 conn(0x5563f7615610 msgr2=0x5563f76427f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff83222c740 1 -- 172.21.15.69:0/3155181747 shutdown_connections 2024-04-24T08:15:53.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.857+0000 7ff83222c740 1 -- 172.21.15.69:0/3155181747 wait complete. 2024-04-24T08:15:53.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.858+0000 7ff83222c740 1 -- start start 2024-04-24T08:15:53.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.858+0000 7ff83222c740 1 Processor -- start 2024-04-24T08:15:53.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.859+0000 7ff83222c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7615d50 0x5563f7628c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.859+0000 7ff83222c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f762f540 0x5563f7610080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.859+0000 7ff83222c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7630610 0x5563f75e4390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.859+0000 7ff83222c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5563f74f4790 con 0x5563f7630610 2024-04-24T08:15:53.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.859+0000 7ff83222c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff828146410 con 0x5563f7615d50 2024-04-24T08:15:53.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.859+0000 7ff83222c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5563f75e6a50 con 0x5563f762f540 2024-04-24T08:15:53.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.859+0000 7ff83222c740 10 client.0 _ll_register_callbacks cb 0x7ff81c059330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:53.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.859+0000 7ff82d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7630610 0x5563f75e4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff82d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7630610 0x5563f75e4390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58192/0 (socket says 172.21.15.69:58192) 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff82d400640 1 -- 172.21.15.69:0/936598501 learned_addr learned my addr 172.21.15.69:0/936598501 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff82ca00640 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7615d50 0x5563f7628c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff827e00640 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f762f540 0x5563f7610080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff82d400640 1 -- 172.21.15.69:0/936598501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f762f540 msgr2=0x5563f7610080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff82d400640 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f762f540 0x5563f7610080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff82d400640 1 -- 172.21.15.69:0/936598501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7615d50 msgr2=0x5563f7628c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff82d400640 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7615d50 0x5563f7628c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff82d400640 1 -- 172.21.15.69:0/936598501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563f72210f0 con 0x5563f7630610 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff827e00640 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f762f540 0x5563f7610080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff82d400640 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7630610 0x5563f75e4390 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7ff82000e720 tx=0x7ff82000ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b98ea55ca1dd2f69 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.860+0000 7ff80d600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff80d600640 1 -- 172.21.15.69:0/936598501 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82001b060 con 0x5563f7630610 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff80d600640 1 -- 172.21.15.69:0/936598501 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 5795+0+0 (secure 0 0 0) 0x7ff82000cd30 con 0x5563f7630610 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff83222c740 1 -- 172.21.15.69:0/936598501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff814032df0 con 0x5563f7630610 2024-04-24T08:15:53.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff80d600640 1 -- 172.21.15.69:0/936598501 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) ==== 24+0+0 (secure 0 0 0) 0x7ff820021050 con 0x5563f7630610 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff83222c740 10 client.7950 fetch_fsmap learned FSMap version 67 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff83222c740 1 -- 172.21.15.69:0/936598501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) -- 0x5563f7514810 con 0x5563f7630610 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff80d600640 1 -- 172.21.15.69:0/936598501 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff814032df0 con 0x5563f7630610 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff80d600640 1 -- 172.21.15.69:0/936598501 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 67) ==== 3494+0+0 (secure 0 0 0) 0x7ff820024020 con 0x5563f7630610 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff83222c740 10 client.7950 fetch_fsmap finished waiting for FSMap version 67 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff83222c740 10 client.7950 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff83222c740 20 client.7950 populate_metadata read hostname 'smithi069' 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff83222c740 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5563f76105c0 0x5563f75e48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff83222c740 4 client.7950 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.861+0000 7ff83222c740 1 -- 172.21.15.69:0/936598501 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5563f75e6c80 con 0x5563f76105c0 2024-04-24T08:15:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.862+0000 7ff82ca00640 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5563f76105c0 0x5563f75e48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:53.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.862+0000 7ff80d600640 10 client.7950 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.862+0000 7ff82ca00640 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5563f76105c0 0x5563f75e48d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f19776ad5786e09a server_cookie=924374ef4c58bd4e in_seq=0 out_seq=0 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.862+0000 7ff80d600640 1 -- 172.21.15.69:0/936598501 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+256 (crc 0 0 0) 0x5563f75e6c80 con 0x5563f76105c0 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.862+0000 7ff80d600640 10 client.7950 handle_command_reply: tid=0 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.862+0000 7ff83222c740 1 client.7950 shutdown 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 1 -- 172.21.15.69:0/936598501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7630610 msgr2=0x5563f75e4390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7630610 0x5563f75e4390 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7ff82000e720 tx=0x7ff82000ebe0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 1 -- 172.21.15.69:0/936598501 shutdown_connections 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5563f76105c0 0x5563f75e48d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563f7630610 0x5563f75e4390 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5563f762f540 0x5563f7610080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 1 --2- 172.21.15.69:0/936598501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5563f7615d50 0x5563f7628c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 1 -- 172.21.15.69:0/936598501 >> 172.21.15.69:0/936598501 conn(0x5563f7615610 msgr2=0x5563f760ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff80d600640 0 client.7950 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 1 -- 172.21.15.69:0/936598501 shutdown_connections 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 1 -- 172.21.15.69:0/936598501 wait complete. 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.863+0000 7ff83222c740 20 client.7950 trim_cache size 0 max 16384 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stdout: "3e23f37a-33f3-4048-870d-6ba0d1fe4443": { 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/dir", 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "3e23f37a-33f3-4048-870d-6ba0d1fe4443", 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive" 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:53.868 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:15:53.869 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:15:53.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 -- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff80806c8f0 msgr2=0x7ff80806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff80806c8f0 0x7ff80806edb0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff814034000 tx=0x7ff81402c000 comp rx=0 tx=0).stop 2024-04-24T08:15:53.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 -- 172.21.15.69:0/3409881981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 msgr2=0x7ff82809dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 0x7ff82809dc00 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7ff81c00c500 tx=0x7ff81c00c9c0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 -- 172.21.15.69:0/3409881981 shutdown_connections 2024-04-24T08:15:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff80806c8f0 0x7ff80806edb0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff82815c820 0x7ff828071d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8280ce250 0x7ff828072f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 --2- 172.21.15.69:0/3409881981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8280b3e90 0x7ff82809dc00 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 -- 172.21.15.69:0/3409881981 >> 172.21.15.69:0/3409881981 conn(0x7ff8280d0350 msgr2=0x7ff8280b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 -- 172.21.15.69:0/3409881981 shutdown_connections 2024-04-24T08:15:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:15:53.864+0000 7ff82f400640 1 -- 172.21.15.69:0/3409881981 wait complete. 2024-04-24T08:15:53.882 INFO:tasks.cephfs.filesystem:scrub status for tag:3e23f37a-33f3-4048-870d-6ba0d1fe4443 - {'path': '/dir', 'tag': '3e23f37a-33f3-4048-870d-6ba0d1fe4443', 'options': 'recursive'} 2024-04-24T08:15:53.918 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:15:53.917+0000 7faf24600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:15:56.544 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:15:56.543+0000 7fa5d8600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-24T08:16:15.445 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:16:15.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:16:23.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:16:23.884 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-24T08:16:24.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.097+0000 7f4bd9200640 1 Processor -- start 2024-04-24T08:16:24.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.097+0000 7f4bd9200640 1 -- start start 2024-04-24T08:16:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.098+0000 7f4bd9200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd41420a0 0x7f4bd4071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.098+0000 7f4bd9200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4bd4142480 0x7f4bd415ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.098+0000 7f4bd9200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bd4072450 con 0x7f4bd4160510 2024-04-24T08:16:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.098+0000 7f4bd9200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4bd4160960 con 0x7f4bd4142480 2024-04-24T08:16:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.098+0000 7f4bd9200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd40cf1e0 con 0x7f4bd41420a0 2024-04-24T08:16:24.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.099+0000 7f4bcaa00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4bd4142480 0x7f4bd415ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.099+0000 7f4bd3400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd41420a0 0x7f4bd4071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.099+0000 7f4bd3400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd41420a0 0x7f4bd4071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:38410/0 (socket says 172.21.15.69:38410) 2024-04-24T08:16:24.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.099+0000 7f4bd3400640 1 -- 172.21.15.69:0/2457842044 learned_addr learned my addr 172.21.15.69:0/2457842044 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.100+0000 7f4bd3400640 1 -- 172.21.15.69:0/2457842044 >> v2:172.21.15.121:3300/0 conn(0x7f4bd4142480 msgr2=0x7f4bd415ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.100+0000 7f4bd3400640 1 --2- 172.21.15.69:0/2457842044 >> v2:172.21.15.121:3300/0 conn(0x7f4bd4142480 0x7f4bd415ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.100+0000 7f4bd3400640 1 -- 172.21.15.69:0/2457842044 >> v1:172.21.15.121:6789/0 conn(0x7f4bd4160510 legacy=0x7f4bd415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.100+0000 7f4bd3400640 1 -- 172.21.15.69:0/2457842044 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd4072730 con 0x7f4bd41420a0 2024-04-24T08:16:24.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.101+0000 7f4bd3400640 1 --2- 172.21.15.69:0/2457842044 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd41420a0 0x7f4bd4071d70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009e50 tx=0x7f4bc402a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddca335694bbd024 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.102+0000 7f4bd2a00640 1 -- 172.21.15.69:0/2457842044 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4036070 con 0x7f4bd41420a0 2024-04-24T08:16:24.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.102+0000 7f4bd2a00640 1 -- 172.21.15.69:0/2457842044 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc4031600 con 0x7f4bd41420a0 2024-04-24T08:16:24.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.102+0000 7f4bd2a00640 1 -- 172.21.15.69:0/2457842044 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc40319d0 con 0x7f4bd41420a0 2024-04-24T08:16:24.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.102+0000 7f4bd9200640 1 -- 172.21.15.69:0/2457842044 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd41420a0 msgr2=0x7f4bd4071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.102+0000 7f4bd9200640 1 --2- 172.21.15.69:0/2457842044 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd41420a0 0x7f4bd4071d70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4009e50 tx=0x7f4bc402a710 comp rx=0 tx=0).stop 2024-04-24T08:16:24.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.103+0000 7f4bd9200640 1 -- 172.21.15.69:0/2457842044 shutdown_connections 2024-04-24T08:16:24.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.103+0000 7f4bd9200640 1 --2- 172.21.15.69:0/2457842044 >> v2:172.21.15.121:3300/0 conn(0x7f4bd4142480 0x7f4bd415ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.103+0000 7f4bd9200640 1 --2- 172.21.15.69:0/2457842044 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd41420a0 0x7f4bd4071d70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.103+0000 7f4bd9200640 1 -- 172.21.15.69:0/2457842044 >> 172.21.15.69:0/2457842044 conn(0x7f4bd4067b80 msgr2=0x7f4bd406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.103+0000 7f4bd9200640 1 -- 172.21.15.69:0/2457842044 shutdown_connections 2024-04-24T08:16:24.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.103+0000 7f4bd9200640 1 -- 172.21.15.69:0/2457842044 wait complete. 2024-04-24T08:16:24.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.105+0000 7f4bd9200640 1 Processor -- start 2024-04-24T08:16:24.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.105+0000 7f4bd9200640 1 -- start start 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.106+0000 7f4bd9200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bd41420a0 0x7f4bd4141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.106+0000 7f4bd9200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd4142480 0x7f4bd4153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.106+0000 7f4bd9200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd4160510 0x7f4bd413e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.106+0000 7f4bd9200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd41428e0 con 0x7f4bd4142480 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.106+0000 7f4bd9200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd4072cf0 con 0x7f4bd41420a0 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.106+0000 7f4bd9200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd413f820 con 0x7f4bd4160510 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.106+0000 7f4bcaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd4142480 0x7f4bd4153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.106+0000 7f4bcaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd4142480 0x7f4bd4153b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40452/0 (socket says 172.21.15.69:40452) 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.106+0000 7f4bcaa00640 1 -- 172.21.15.69:0/632011029 learned_addr learned my addr 172.21.15.69:0/632011029 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.107+0000 7f4bd3e00640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd4160510 0x7f4bd413e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.107+0000 7f4bd3400640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bd41420a0 0x7f4bd4141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.107+0000 7f4bcaa00640 1 -- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd4160510 msgr2=0x7f4bd413e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.107+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd4160510 0x7f4bd413e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.107+0000 7f4bcaa00640 1 -- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bd41420a0 msgr2=0x7f4bd4141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.107+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bd41420a0 0x7f4bd4141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.107+0000 7f4bcaa00640 1 -- 172.21.15.69:0/632011029 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd413fb00 con 0x7f4bd4142480 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.107+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd4142480 0x7f4bd4153b00 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00e9b0 tx=0x7f4bbc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.107+0000 7f4bcbe00640 1 -- 172.21.15.69:0/632011029 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc017070 con 0x7f4bd4142480 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.108+0000 7f4bcbe00640 1 -- 172.21.15.69:0/632011029 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bbc00c9a0 con 0x7f4bd4142480 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.108+0000 7f4bd9200640 1 -- 172.21.15.69:0/632011029 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd413fde0 con 0x7f4bd4142480 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.108+0000 7f4bcbe00640 1 -- 172.21.15.69:0/632011029 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc00cd90 con 0x7f4bd4142480 2024-04-24T08:16:24.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.108+0000 7f4bd9200640 1 -- 172.21.15.69:0/632011029 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd41407c0 con 0x7f4bd4142480 2024-04-24T08:16:24.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.110+0000 7f4bcbe00640 1 -- 172.21.15.69:0/632011029 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4bbc01a070 con 0x7f4bd4142480 2024-04-24T08:16:24.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.111+0000 7f4bcbe00640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4bb006c8f0 0x7f4bb006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.111+0000 7f4bcbe00640 1 -- 172.21.15.69:0/632011029 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f4bbc08bfd0 con 0x7f4bd4142480 2024-04-24T08:16:24.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.111+0000 7f4bdd5eb740 1 Processor -- start 2024-04-24T08:16:24.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.112+0000 7f4bd3400640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4bb006c8f0 0x7f4bb006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.112+0000 7f4bdd5eb740 1 -- start start 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.112+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d77e00 0x55a6c6d2a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.112+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d5a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.112+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d5a7e0 0x55a6c6d2f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.112+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a6c6c941f0 con 0x55a6c6d5af20 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.112+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a6c6c3a350 con 0x55a6c6d77e00 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.112+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a6c6c901b0 con 0x55a6c6d5a7e0 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bcaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d5a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bcaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d5a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40462/0 (socket says 172.21.15.69:40462) 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bd3400640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4bb006c8f0 0x7f4bb006edb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4033000 tx=0x7f4bc4009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bcaa00640 1 -- 172.21.15.69:0/1412201342 learned_addr learned my addr 172.21.15.69:0/1412201342 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bd3e00640 1 --2- 172.21.15.69:0/1412201342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d5a7e0 0x55a6c6d2f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bcaa00640 1 -- 172.21.15.69:0/1412201342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d5a7e0 msgr2=0x55a6c6d2f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/1412201342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d5a7e0 0x55a6c6d2f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bd3400640 1 --2- 172.21.15.69:0/1412201342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d77e00 0x55a6c6d2a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bcaa00640 1 -- 172.21.15.69:0/1412201342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d77e00 msgr2=0x55a6c6d2a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/1412201342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d77e00 0x55a6c6d2a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.113+0000 7f4bcaa00640 1 -- 172.21.15.69:0/1412201342 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6c6c8ef50 con 0x55a6c6d5af20 2024-04-24T08:16:24.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.114+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/1412201342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d5a2a0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0622c0 tx=0x7f4bbc05a450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8d26bd85a0edb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.114+0000 7f4bc3e00640 1 -- 172.21.15.69:0/1412201342 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc021040 con 0x55a6c6d5af20 2024-04-24T08:16:24.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.114+0000 7f4bc3e00640 1 -- 172.21.15.69:0/1412201342 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bbc0627f0 con 0x55a6c6d5af20 2024-04-24T08:16:24.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.114+0000 7f4bc3e00640 1 -- 172.21.15.69:0/1412201342 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc057f10 con 0x55a6c6d5af20 2024-04-24T08:16:24.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.114+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1412201342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 msgr2=0x55a6c6d5a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.114+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1412201342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d5a2a0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0622c0 tx=0x7f4bbc05a450 comp rx=0 tx=0).stop 2024-04-24T08:16:24.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.114+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1412201342 shutdown_connections 2024-04-24T08:16:24.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.114+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1412201342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d5a2a0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.114+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1412201342 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d5a7e0 0x55a6c6d2f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.114+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1412201342 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d77e00 0x55a6c6d2a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.115+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1412201342 >> 172.21.15.69:0/1412201342 conn(0x55a6c6d2ca00 msgr2=0x55a6c6d752a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.115+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1412201342 shutdown_connections 2024-04-24T08:16:24.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.115+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1412201342 wait complete. 2024-04-24T08:16:24.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.115+0000 7f4bdd5eb740 1 -- start start 2024-04-24T08:16:24.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.115+0000 7f4bdd5eb740 1 Processor -- start 2024-04-24T08:16:24.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.116+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d73ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.116+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d77e00 0x55a6c6d76dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.116+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d4f040 0x55a6c6d77310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.116+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a6c6c941f0 con 0x55a6c6d5af20 2024-04-24T08:16:24.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.116+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a6c6c4d690 con 0x55a6c6d4f040 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.116+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a6c6d2ebb0 con 0x55a6c6d77e00 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bdd5eb740 10 client.0 _ll_register_callbacks cb 0x55a6c6c89650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bd3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d73ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bd3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d73ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40468/0 (socket says 172.21.15.69:40468) 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bd3400640 1 -- 172.21.15.69:0/2680363893 learned_addr learned my addr 172.21.15.69:0/2680363893 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d77e00 0x55a6c6d76dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bd3e00640 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d4f040 0x55a6c6d77310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bd3400640 1 -- 172.21.15.69:0/2680363893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d77e00 msgr2=0x55a6c6d76dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bd3400640 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d77e00 0x55a6c6d76dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bd3400640 1 -- 172.21.15.69:0/2680363893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d4f040 msgr2=0x55a6c6d77310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bd3400640 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d4f040 0x55a6c6d77310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.117+0000 7f4bd3400640 1 -- 172.21.15.69:0/2680363893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6c6c4c920 con 0x55a6c6d5af20 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.118+0000 7f4bd3400640 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d73ff0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4034040 tx=0x7f4bc4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6c93a8d2f4a435 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.118+0000 7f4bc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.118+0000 7f4bc0c00640 1 -- 172.21.15.69:0/2680363893 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4036040 con 0x55a6c6d5af20 2024-04-24T08:16:24.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.118+0000 7f4bc0c00640 1 -- 172.21.15.69:0/2680363893 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f4bc403cda0 con 0x55a6c6d5af20 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.118+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/2680363893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bbc0627f0 con 0x55a6c6d5af20 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.118+0000 7f4bc0c00640 1 -- 172.21.15.69:0/2680363893 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f4bc4047400 con 0x55a6c6d5af20 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.118+0000 7f4bdd5eb740 10 client.7959 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.118+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/2680363893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55a6c6a707d0 con 0x55a6c6d5af20 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.118+0000 7f4bc0c00640 1 -- 172.21.15.69:0/2680363893 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bbc0627f0 con 0x55a6c6d5af20 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.118+0000 7f4bc0c00640 1 -- 172.21.15.69:0/2680363893 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f4bc40475a0 con 0x55a6c6d5af20 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.119+0000 7f4bdd5eb740 10 client.7959 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.119+0000 7f4bdd5eb740 10 client.7959 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.119+0000 7f4bdd5eb740 20 client.7959 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.119+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a6c6d5b930 0x55a6c6d5bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.119+0000 7f4bdd5eb740 4 client.7959 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.119+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/2680363893 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a6c6ca1c10 con 0x55a6c6d5b930 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.119+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a6c6d5b930 0x55a6c6d5bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.119+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a6c6d5b930 0x55a6c6d5bd50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=135d8574745386b1 server_cookie=35fadfa47af68db8 in_seq=0 out_seq=0 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.119+0000 7f4bc0c00640 10 client.7959 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.120+0000 7f4bc0c00640 1 -- 172.21.15.69:0/2680363893 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a6c6ca1c10 con 0x55a6c6d5b930 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.120+0000 7f4bc0c00640 10 client.7959 handle_command_reply: tid=0 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.120+0000 7f4bdd5eb740 1 client.7959 shutdown 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.120+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/2680363893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 msgr2=0x55a6c6d73ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.120+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d73ff0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4034040 tx=0x7f4bc4004ab0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.120+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/2680363893 shutdown_connections 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.120+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a6c6d5b930 0x55a6c6d5bd50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.121+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d4f040 0x55a6c6d77310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.121+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d77e00 0x55a6c6d76dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.121+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/2680363893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d73ff0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.121+0000 7f4bc0c00640 0 client.7959 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.121+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/2680363893 >> 172.21.15.69:0/2680363893 conn(0x55a6c6d5a7e0 msgr2=0x55a6c6d2dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.121+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/2680363893 shutdown_connections 2024-04-24T08:16:24.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.121+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/2680363893 wait complete. 2024-04-24T08:16:24.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.121+0000 7f4bdd5eb740 20 client.7959 trim_cache size 0 max 16384 2024-04-24T08:16:24.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.140+0000 7f4bdd5eb740 1 Processor -- start 2024-04-24T08:16:24.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.140+0000 7f4bdd5eb740 1 -- start start 2024-04-24T08:16:24.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.141+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d4f040 0x55a6c6d78ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.141+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d5af20 0x55a6c6d790e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.141+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d77e00 0x55a6c6d849d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.141+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd4200ea0 con 0x55a6c6d4f040 2024-04-24T08:16:24.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.141+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd41ffed0 con 0x55a6c6d77e00 2024-04-24T08:16:24.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.141+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd4155f50 con 0x55a6c6d5af20 2024-04-24T08:16:24.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.141+0000 7f4bd3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d4f040 0x55a6c6d78ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.141+0000 7f4bd3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d4f040 0x55a6c6d78ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40478/0 (socket says 172.21.15.69:40478) 2024-04-24T08:16:24.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.141+0000 7f4bd3400640 1 -- 172.21.15.69:0/1462353280 learned_addr learned my addr 172.21.15.69:0/1462353280 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.142+0000 7f4bd3400640 1 -- 172.21.15.69:0/1462353280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d5af20 msgr2=0x55a6c6d790e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.142+0000 7f4bd3400640 1 --2- 172.21.15.69:0/1462353280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d5af20 0x55a6c6d790e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.142+0000 7f4bd3e00640 1 --2- 172.21.15.69:0/1462353280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d77e00 0x55a6c6d849d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.142+0000 7f4bd3400640 1 -- 172.21.15.69:0/1462353280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d77e00 msgr2=0x55a6c6d849d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.142+0000 7f4bd3400640 1 --2- 172.21.15.69:0/1462353280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d77e00 0x55a6c6d849d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.142+0000 7f4bd3400640 1 -- 172.21.15.69:0/1462353280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6c69a63f0 con 0x55a6c6d4f040 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.142+0000 7f4bd3400640 1 --2- 172.21.15.69:0/1462353280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d4f040 0x55a6c6d78ba0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4047840 tx=0x7f4bc4044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f02471a611fa01 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.142+0000 7f4bc3400640 1 -- 172.21.15.69:0/1462353280 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4036070 con 0x55a6c6d4f040 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.142+0000 7f4bc3400640 1 -- 172.21.15.69:0/1462353280 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc404c070 con 0x55a6c6d4f040 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.142+0000 7f4bc3400640 1 -- 172.21.15.69:0/1462353280 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc4047b90 con 0x55a6c6d4f040 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.143+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1462353280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d4f040 msgr2=0x55a6c6d78ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.143+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1462353280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d4f040 0x55a6c6d78ba0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4047840 tx=0x7f4bc4044000 comp rx=0 tx=0).stop 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.143+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1462353280 shutdown_connections 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.143+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1462353280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d77e00 0x55a6c6d849d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.143+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1462353280 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d5af20 0x55a6c6d790e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.143+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1462353280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d4f040 0x55a6c6d78ba0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.143+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1462353280 >> 172.21.15.69:0/1462353280 conn(0x55a6c6d5a7e0 msgr2=0x55a6c6d5c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.143+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1462353280 shutdown_connections 2024-04-24T08:16:24.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.143+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1462353280 wait complete. 2024-04-24T08:16:24.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.144+0000 7f4bdd5eb740 1 -- start start 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.144+0000 7f4bdd5eb740 1 Processor -- start 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.145+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d4f040 0x55a6c6d578f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.145+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d839c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.145+0000 7f4bdd5eb740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d77e00 0x55a6c6df78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.145+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a6c6d74d10 con 0x55a6c6d5af20 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.145+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd41ffa80 con 0x55a6c6d4f040 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.145+0000 7f4bdd5eb740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a6c6d765a0 con 0x55a6c6d77e00 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.145+0000 7f4bdd5eb740 10 client.0 _ll_register_callbacks cb 0x7f4bc4034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.145+0000 7f4bcaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d839c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.145+0000 7f4bcaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d839c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40480/0 (socket says 172.21.15.69:40480) 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.145+0000 7f4bcaa00640 1 -- 172.21.15.69:0/1163417138 learned_addr learned my addr 172.21.15.69:0/1163417138 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bd3400640 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d4f040 0x55a6c6d578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bcaa00640 1 -- 172.21.15.69:0/1163417138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d77e00 msgr2=0x55a6c6df78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d77e00 0x55a6c6df78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bcaa00640 1 -- 172.21.15.69:0/1163417138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d4f040 msgr2=0x55a6c6d578f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d4f040 0x55a6c6d578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bcaa00640 1 -- 172.21.15.69:0/1163417138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6c69660f0 con 0x55a6c6d5af20 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bcaa00640 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d839c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0622c0 tx=0x7f4bbc08a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b1640fcb7de430 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bc0c00640 1 -- 172.21.15.69:0/1163417138 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc021040 con 0x55a6c6d5af20 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bc0c00640 1 -- 172.21.15.69:0/1163417138 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f4bbc056280 con 0x55a6c6d5af20 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1163417138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc404c070 con 0x55a6c6d5af20 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bc0c00640 1 -- 172.21.15.69:0/1163417138 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f4bbc0627f0 con 0x55a6c6d5af20 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bdd5eb740 10 client.7965 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.146+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1163417138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55a6c6c59810 con 0x55a6c6d5af20 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.147+0000 7f4bc0c00640 1 -- 172.21.15.69:0/1163417138 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc404c070 con 0x55a6c6d5af20 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.147+0000 7f4bc0c00640 1 -- 172.21.15.69:0/1163417138 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f4bbc059060 con 0x55a6c6d5af20 2024-04-24T08:16:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.147+0000 7f4bdd5eb740 10 client.7965 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:24.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.147+0000 7f4bdd5eb740 10 client.7965 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:16:24.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.147+0000 7f4bdd5eb740 20 client.7965 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:24.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.147+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a6c6d83f00 0x55a6c6d728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.147+0000 7f4bdd5eb740 4 client.7965 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:16:24.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.147+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1163417138 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a6c6c9ba80 con 0x55a6c6d83f00 2024-04-24T08:16:24.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.147+0000 7f4bd3400640 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a6c6d83f00 0x55a6c6d728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bd3400640 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a6c6d83f00 0x55a6c6d728f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd098de4c862a66a server_cookie=60b49ec561765f62 in_seq=0 out_seq=0 2024-04-24T08:16:24.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bc0c00640 10 client.7965 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:24.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bc0c00640 1 -- 172.21.15.69:0/1163417138 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a6c6c9ba80 con 0x55a6c6d83f00 2024-04-24T08:16:24.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bc0c00640 10 client.7965 handle_command_reply: tid=0 2024-04-24T08:16:24.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 client.7965 shutdown 2024-04-24T08:16:24.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1163417138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 msgr2=0x55a6c6d839c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d839c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0622c0 tx=0x7f4bbc08a2e0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1163417138 shutdown_connections 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a6c6d83f00 0x55a6c6d728f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6c6d5af20 0x55a6c6d839c0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6c6d77e00 0x55a6c6df78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 --2- 172.21.15.69:0/1163417138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6c6d4f040 0x55a6c6d578f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1163417138 >> 172.21.15.69:0/1163417138 conn(0x55a6c6d5a7e0 msgr2=0x55a6c6d84d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bc0c00640 0 client.7965 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1163417138 shutdown_connections 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.148+0000 7f4bdd5eb740 1 -- 172.21.15.69:0/1163417138 wait complete. 2024-04-24T08:16:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bdd5eb740 20 client.7965 trim_cache size 0 max 16384 2024-04-24T08:16:24.154 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:16:24.154 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-04-24T08:16:24.154 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-04-24T08:16:24.154 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 -- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4bb006c8f0 msgr2=0x7f4bb006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4bb006c8f0 0x7f4bb006edb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f4bc4033000 tx=0x7f4bc4009000 comp rx=0 tx=0).stop 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 -- 172.21.15.69:0/632011029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd4142480 msgr2=0x7f4bd4153b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd4142480 0x7f4bd4153b00 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00e9b0 tx=0x7f4bbc00ee70 comp rx=0 tx=0).stop 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 -- 172.21.15.69:0/632011029 shutdown_connections 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4bb006c8f0 0x7f4bb006edb0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd4142480 0x7f4bd4153b00 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4bd4160510 0x7f4bd413e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 --2- 172.21.15.69:0/632011029 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4bd41420a0 0x7f4bd4141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 -- 172.21.15.69:0/632011029 >> 172.21.15.69:0/632011029 conn(0x7f4bd4067b80 msgr2=0x7f4bd406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 -- 172.21.15.69:0/632011029 shutdown_connections 2024-04-24T08:16:24.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.149+0000 7f4bd9200640 1 -- 172.21.15.69:0/632011029 wait complete. 2024-04-24T08:16:24.168 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:16:24.168 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:16:24.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.357+0000 7f9593400640 1 Processor -- start 2024-04-24T08:16:24.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.357+0000 7f9593400640 1 -- start start 2024-04-24T08:16:24.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.359+0000 7f9593400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f958c145fe0 0x7f958c06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.359+0000 7f9593400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f958c0cf3d0 con 0x7f958c145a90 2024-04-24T08:16:24.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.360+0000 7f9593400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f958c0cf6b0 con 0x7f958c0d0300 2024-04-24T08:16:24.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.360+0000 7f9593400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f958c0cf990 con 0x7f958c145fe0 2024-04-24T08:16:24.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.360+0000 7f9583e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f958c145a90 0x7f958c160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36434/0 (socket says 172.21.15.69:36434) 2024-04-24T08:16:24.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.360+0000 7f9583e00640 1 -- 172.21.15.69:0/1448000249 learned_addr learned my addr 172.21.15.69:0/1448000249 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:16:24.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.360+0000 7f9591400640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f958c145fe0 0x7f958c06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.360+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1455240959 0 0) 0x7f958c0cf3d0 con 0x7f958c145a90 2024-04-24T08:16:24.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.360+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9570003520 con 0x7f958c145a90 2024-04-24T08:16:24.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2456082160 0 0) 0x7f9570003520 con 0x7f958c145a90 2024-04-24T08:16:24.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f958c145fe0 msgr2=0x7f958c06e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f958c145fe0 0x7f958c06e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 >> v1:172.21.15.121:6789/0 conn(0x7f958c0d0300 legacy=0x7f958c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c0cfc70 con 0x7f958c145a90 2024-04-24T08:16:24.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f9574001670 con 0x7f958c145a90 2024-04-24T08:16:24.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 >> v1:172.21.15.69:6789/0 conn(0x7f958c145a90 legacy=0x7f958c160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f958c145fe0 0x7f95700039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9570003f60 0x7f9570004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9570004d40 0x7f95700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f958c0cf990 con 0x7f9570004d40 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f957000baa0 con 0x7f9570003f60 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f957000bca0 con 0x7f958c145fe0 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.361+0000 7f9591400640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9570004d40 0x7f95700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.362+0000 7f9591400640 1 -- 172.21.15.69:0/1448000249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f958c145fe0 msgr2=0x7f95700039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.362+0000 7f9591400640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f958c145fe0 0x7f95700039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.362+0000 7f9591400640 1 -- 172.21.15.69:0/1448000249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9570003f60 msgr2=0x7f9570004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.362+0000 7f9583e00640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9570003f60 0x7f9570004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.362+0000 7f9591400640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9570003f60 0x7f9570004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.362+0000 7f9591400640 1 -- 172.21.15.69:0/1448000249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9570003520 con 0x7f9570004d40 2024-04-24T08:16:24.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.363+0000 7f9591400640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9570004d40 0x7f95700051c0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f9584009fc0 tx=0x7f9570005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e4424583e57017 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.363+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584009c10 con 0x7f9570004d40 2024-04-24T08:16:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.363+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9584036070 con 0x7f9570004d40 2024-04-24T08:16:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.363+0000 7f958be00640 1 -- 172.21.15.69:0/1448000249 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958402d070 con 0x7f9570004d40 2024-04-24T08:16:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.363+0000 7f9593400640 1 -- 172.21.15.69:0/1448000249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9570004d40 msgr2=0x7f95700051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.363+0000 7f9593400640 1 --2- 172.21.15.69:0/1448000249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9570004d40 0x7f95700051c0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f9584009fc0 tx=0x7f9570005830 comp rx=0 tx=0).stop 2024-04-24T08:16:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.363+0000 7f9591400640 1 -- 172.21.15.69:0/1448000249 reap_dead start 2024-04-24T08:16:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.364+0000 7f9593400640 1 -- 172.21.15.69:0/1448000249 shutdown_connections 2024-04-24T08:16:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.364+0000 7f9593400640 1 -- 172.21.15.69:0/1448000249 >> 172.21.15.69:0/1448000249 conn(0x7f958c067d90 msgr2=0x7f958c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.364+0000 7f9593400640 1 -- 172.21.15.69:0/1448000249 shutdown_connections 2024-04-24T08:16:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.364+0000 7f9593400640 1 -- 172.21.15.69:0/1448000249 wait complete. 2024-04-24T08:16:24.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.365+0000 7f9593400640 1 Processor -- start 2024-04-24T08:16:24.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.366+0000 7f9593400640 1 -- start start 2024-04-24T08:16:24.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.366+0000 7f9593400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9570004d40 0x7f958c0b2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.366+0000 7f9593400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f958c1f4670 0x7f958c0b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.366+0000 7f9593400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f958c142d70 0x7f958c1431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.366+0000 7f9593400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f958c1458f0 con 0x7f958c142d70 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.366+0000 7f9593400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f958c145770 con 0x7f958c1f4670 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.366+0000 7f9593400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f958c0cdc30 con 0x7f9570004d40 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.367+0000 7f9591400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f958c142d70 0x7f958c1431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.367+0000 7f9591400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f958c142d70 0x7f958c1431d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40504/0 (socket says 172.21.15.69:40504) 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.367+0000 7f9591400640 1 -- 172.21.15.69:0/2422533611 learned_addr learned my addr 172.21.15.69:0/2422533611 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.367+0000 7f9590a00640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9570004d40 0x7f958c0b2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.367+0000 7f9583e00640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f958c1f4670 0x7f958c0b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.367+0000 7f9591400640 1 -- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9570004d40 msgr2=0x7f958c0b2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.367+0000 7f9591400640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9570004d40 0x7f958c0b2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.367+0000 7f9591400640 1 -- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f958c1f4670 msgr2=0x7f958c0b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.367+0000 7f9591400640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f958c1f4670 0x7f958c0b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.367+0000 7f9591400640 1 -- 172.21.15.69:0/2422533611 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f958c0b20d0 con 0x7f958c142d70 2024-04-24T08:16:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.368+0000 7f9591400640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f958c142d70 0x7f958c1431d0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f9584033040 tx=0x7f9584009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.368+0000 7f9589600640 1 -- 172.21.15.69:0/2422533611 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584009c10 con 0x7f958c142d70 2024-04-24T08:16:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.368+0000 7f9589600640 1 -- 172.21.15.69:0/2422533611 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9584003b90 con 0x7f958c142d70 2024-04-24T08:16:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.368+0000 7f9593400640 1 -- 172.21.15.69:0/2422533611 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f958c1f82b0 con 0x7f958c142d70 2024-04-24T08:16:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.368+0000 7f9589600640 1 -- 172.21.15.69:0/2422533611 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584031e10 con 0x7f958c142d70 2024-04-24T08:16:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.368+0000 7f9593400640 1 -- 172.21.15.69:0/2422533611 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f958c1f8870 con 0x7f958c142d70 2024-04-24T08:16:24.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.369+0000 7f9593400640 1 -- 172.21.15.69:0/2422533611 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f958c068a80 con 0x7f958c142d70 2024-04-24T08:16:24.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.370+0000 7f9589600640 1 -- 172.21.15.69:0/2422533611 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9584005d10 con 0x7f958c142d70 2024-04-24T08:16:24.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.371+0000 7f9589600640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f956806c8f0 0x7f956806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.371+0000 7f9589600640 1 -- 172.21.15.69:0/2422533611 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f95840ad4b0 con 0x7f958c142d70 2024-04-24T08:16:24.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.372+0000 7f9590a00640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f956806c8f0 0x7f956806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.373+0000 7f9590a00640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f956806c8f0 0x7f956806edb0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f957c001680 tx=0x7f957c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.376+0000 7f9589600640 1 -- 172.21.15.69:0/2422533611 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f958404a060 con 0x7f958c142d70 2024-04-24T08:16:24.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.614+0000 7f9593400640 1 -- 172.21.15.69:0/2422533611 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f958c0b2300 con 0x7f958c142d70 2024-04-24T08:16:24.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.615+0000 7f9589600640 1 -- 172.21.15.69:0/2422533611 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5182 (secure 0 0 0) 0x7f958c0b20d0 con 0x7f958c142d70 2024-04-24T08:16:24.617 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:16:24.617 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":68,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:15:56.304080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7581":{"gid":7581,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":12,"addr":"172.21.15.69:6833/2529876410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2529876410},{"type":"v1","addr":"172.21.15.69:6833","nonce":2529876410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7581,"qdb_cluster":[7581]},"id":6}]} 2024-04-24T08:16:24.617 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 68 2024-04-24T08:16:24.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 -- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f956806c8f0 msgr2=0x7f956806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f956806c8f0 0x7f956806edb0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f957c001680 tx=0x7f957c009040 comp rx=0 tx=0).stop 2024-04-24T08:16:24.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 -- 172.21.15.69:0/2422533611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f958c142d70 msgr2=0x7f958c1431d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f958c142d70 0x7f958c1431d0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f9584033040 tx=0x7f9584009750 comp rx=0 tx=0).stop 2024-04-24T08:16:24.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 -- 172.21.15.69:0/2422533611 shutdown_connections 2024-04-24T08:16:24.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f956806c8f0 0x7f956806edb0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f958c142d70 0x7f958c1431d0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f958c1f4670 0x7f958c0b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 --2- 172.21.15.69:0/2422533611 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9570004d40 0x7f958c0b2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 -- 172.21.15.69:0/2422533611 >> 172.21.15.69:0/2422533611 conn(0x7f958c067d90 msgr2=0x7f958c15d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 -- 172.21.15.69:0/2422533611 shutdown_connections 2024-04-24T08:16:24.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.618+0000 7f9593400640 1 -- 172.21.15.69:0/2422533611 wait complete. 2024-04-24T08:16:24.639 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-24T08:16:24.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.844+0000 7fe0aa200640 1 Processor -- start 2024-04-24T08:16:24.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.844+0000 7fe0aa200640 1 -- start start 2024-04-24T08:16:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.845+0000 7fe0aa200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0a40cd3d0 0x7fe0a40b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.845+0000 7fe0aa200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe0a40b4780 0x7fe0a40b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.845+0000 7fe0aa200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a41631c0 con 0x7fe0a4073db0 2024-04-24T08:16:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.845+0000 7fe0aa200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe0a40cf4d0 con 0x7fe0a40b4780 2024-04-24T08:16:24.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.845+0000 7fe0aa200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a40cf7b0 con 0x7fe0a40cd3d0 2024-04-24T08:16:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.846+0000 7fe0a3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe0a40b4780 0x7fe0a40b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.846+0000 7fe0a3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe0a40b4780 0x7fe0a40b4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40508/0 (socket says 172.21.15.69:40508) 2024-04-24T08:16:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.846+0000 7fe0a3e00640 1 -- 172.21.15.69:0/2537472378 learned_addr learned my addr 172.21.15.69:0/2537472378 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.846+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/2537472378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0a40cd3d0 0x7fe0a40b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.847+0000 7fe0a3e00640 1 -- 172.21.15.69:0/2537472378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0a40cd3d0 msgr2=0x7fe0a40b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.847+0000 7fe0a3e00640 1 --2- 172.21.15.69:0/2537472378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0a40cd3d0 0x7fe0a40b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.847+0000 7fe0a3e00640 1 -- 172.21.15.69:0/2537472378 >> v1:172.21.15.121:6789/0 conn(0x7fe0a4073db0 legacy=0x7fe0a40cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.847+0000 7fe0a3e00640 1 -- 172.21.15.69:0/2537472378 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a40cfa90 con 0x7fe0a40b4780 2024-04-24T08:16:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.848+0000 7fe0a3e00640 1 --2- 172.21.15.69:0/2537472378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a40b4780 0x7fe0a40b4c30 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009e50 tx=0x7fe09c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21521d18a6a0980f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.848+0000 7fe0a2000640 1 -- 172.21.15.69:0/2537472378 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c036070 con 0x7fe0a40b4780 2024-04-24T08:16:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.848+0000 7fe0a2000640 1 -- 172.21.15.69:0/2537472378 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe09c031600 con 0x7fe0a40b4780 2024-04-24T08:16:24.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.848+0000 7fe0a2000640 1 -- 172.21.15.69:0/2537472378 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0319d0 con 0x7fe0a40b4780 2024-04-24T08:16:24.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.848+0000 7fe0aa200640 1 -- 172.21.15.69:0/2537472378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a40b4780 msgr2=0x7fe0a40b4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.848+0000 7fe0aa200640 1 --2- 172.21.15.69:0/2537472378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a40b4780 0x7fe0a40b4c30 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009e50 tx=0x7fe09c02a710 comp rx=0 tx=0).stop 2024-04-24T08:16:24.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.849+0000 7fe0aa200640 1 -- 172.21.15.69:0/2537472378 shutdown_connections 2024-04-24T08:16:24.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.849+0000 7fe0aa200640 1 --2- 172.21.15.69:0/2537472378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a40b4780 0x7fe0a40b4c30 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.849+0000 7fe0aa200640 1 --2- 172.21.15.69:0/2537472378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0a40cd3d0 0x7fe0a40b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.849+0000 7fe0aa200640 1 -- 172.21.15.69:0/2537472378 >> 172.21.15.69:0/2537472378 conn(0x7fe0a40d0350 msgr2=0x7fe0a40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.849+0000 7fe0aa200640 1 -- 172.21.15.69:0/2537472378 shutdown_connections 2024-04-24T08:16:24.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.849+0000 7fe0aa200640 1 -- 172.21.15.69:0/2537472378 wait complete. 2024-04-24T08:16:24.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.850+0000 7fe0aa200640 1 Processor -- start 2024-04-24T08:16:24.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.851+0000 7fe0aa200640 1 -- start start 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.851+0000 7fe0aa200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a4073db0 0x7fe0a4157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.851+0000 7fe0aa200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0a40b4780 0x7fe0a4164da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.851+0000 7fe0aa200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0a40cd3d0 0x7fe0a41652e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.851+0000 7fe0aa200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a40b5170 con 0x7fe0a4073db0 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.851+0000 7fe0aa200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a40d0050 con 0x7fe0a40b4780 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.851+0000 7fe0aa200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a40ce470 con 0x7fe0a40cd3d0 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.852+0000 7fe0a3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a4073db0 0x7fe0a4157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.852+0000 7fe0a3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a4073db0 0x7fe0a4157730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40512/0 (socket says 172.21.15.69:40512) 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.852+0000 7fe0a3400640 1 -- 172.21.15.69:0/871035066 learned_addr learned my addr 172.21.15.69:0/871035066 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.852+0000 7fe0a3400640 1 -- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0a40cd3d0 msgr2=0x7fe0a41652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.852+0000 7fe0a3400640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0a40cd3d0 0x7fe0a41652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.852+0000 7fe0a3400640 1 -- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0a40b4780 msgr2=0x7fe0a4164da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.852+0000 7fe0a3400640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0a40b4780 0x7fe0a4164da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.852+0000 7fe0a3400640 1 -- 172.21.15.69:0/871035066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a41619a0 con 0x7fe0a4073db0 2024-04-24T08:16:24.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.852+0000 7fe0a3400640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a4073db0 0x7fe0a4157730 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fe09400c430 tx=0x7fe09400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.853+0000 7fe09b400640 1 -- 172.21.15.69:0/871035066 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0940052d0 con 0x7fe0a4073db0 2024-04-24T08:16:24.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.853+0000 7fe09b400640 1 -- 172.21.15.69:0/871035066 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe094005470 con 0x7fe0a4073db0 2024-04-24T08:16:24.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.853+0000 7fe0aa200640 1 -- 172.21.15.69:0/871035066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a4157c70 con 0x7fe0a4073db0 2024-04-24T08:16:24.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.853+0000 7fe09b400640 1 -- 172.21.15.69:0/871035066 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe094007260 con 0x7fe0a4073db0 2024-04-24T08:16:24.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.853+0000 7fe0aa200640 1 -- 172.21.15.69:0/871035066 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a4165c40 con 0x7fe0a4073db0 2024-04-24T08:16:24.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.855+0000 7fe09b400640 1 -- 172.21.15.69:0/871035066 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe094007400 con 0x7fe0a4073db0 2024-04-24T08:16:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.856+0000 7fe09b400640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe08406c8f0 0x7fe08406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.856+0000 7fe09b400640 1 -- 172.21.15.69:0/871035066 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7fe09408b070 con 0x7fe0a4073db0 2024-04-24T08:16:24.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.857+0000 7fe0acf93740 1 Processor -- start 2024-04-24T08:16:24.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.857+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe08406c8f0 0x7fe08406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.857+0000 7fe0acf93740 1 -- start start 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b695e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b697b890 0x5578b697bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6933cd0 0x5578b6979f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0acf93740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5578b6898190 con 0x5578b695dff0 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0acf93740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5578b683e030 con 0x5578b697b890 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0acf93740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5578b6894150 con 0x5578b6933cd0 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe08406c8f0 0x7fe08406edb0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe090006fd0 tx=0x7fe090008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0a3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b695e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0a3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b695e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40518/0 (socket says 172.21.15.69:40518) 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0a3400640 1 -- 172.21.15.69:0/3516891212 learned_addr learned my addr 172.21.15.69:0/3516891212 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/3516891212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b697b890 0x5578b697bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0a3400640 1 -- 172.21.15.69:0/3516891212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6933cd0 msgr2=0x5578b6979f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0a3400640 1 --2- 172.21.15.69:0/3516891212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6933cd0 0x5578b6979f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0a3400640 1 -- 172.21.15.69:0/3516891212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b697b890 msgr2=0x5578b697bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0a3400640 1 --2- 172.21.15.69:0/3516891212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b697b890 0x5578b697bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.858+0000 7fe0a3400640 1 -- 172.21.15.69:0/3516891212 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578b6826b60 con 0x5578b695dff0 2024-04-24T08:16:24.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.859+0000 7fe0a3400640 1 --2- 172.21.15.69:0/3516891212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b695e3d0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fe0940077f0 tx=0x7fe094000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78271b06be58ba8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.859+0000 7fe08f400640 1 -- 172.21.15.69:0/3516891212 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe094061920 con 0x5578b695dff0 2024-04-24T08:16:24.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.859+0000 7fe08f400640 1 -- 172.21.15.69:0/3516891212 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe094057040 con 0x5578b695dff0 2024-04-24T08:16:24.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.859+0000 7fe08f400640 1 -- 172.21.15.69:0/3516891212 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe094071180 con 0x5578b695dff0 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.859+0000 7fe0acf93740 1 -- 172.21.15.69:0/3516891212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 msgr2=0x5578b695e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.859+0000 7fe0acf93740 1 --2- 172.21.15.69:0/3516891212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b695e3d0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fe0940077f0 tx=0x7fe094000ed0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.860+0000 7fe0acf93740 1 -- 172.21.15.69:0/3516891212 shutdown_connections 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.860+0000 7fe0acf93740 1 --2- 172.21.15.69:0/3516891212 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6933cd0 0x5578b6979f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.860+0000 7fe0acf93740 1 --2- 172.21.15.69:0/3516891212 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b697b890 0x5578b697bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.860+0000 7fe0acf93740 1 --2- 172.21.15.69:0/3516891212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b695e3d0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.860+0000 7fe0acf93740 1 -- 172.21.15.69:0/3516891212 >> 172.21.15.69:0/3516891212 conn(0x5578b6930600 msgr2=0x5578b6978e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.860+0000 7fe0acf93740 1 -- 172.21.15.69:0/3516891212 shutdown_connections 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.860+0000 7fe0acf93740 1 -- 172.21.15.69:0/3516891212 wait complete. 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.860+0000 7fe0acf93740 1 -- start start 2024-04-24T08:16:24.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.861+0000 7fe0acf93740 1 Processor -- start 2024-04-24T08:16:24.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b695dff0 0x5578b694e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b696feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b6975360 0x5578b69757e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0acf93740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5578b6934a70 con 0x5578b697b890 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0acf93740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5578b6851500 con 0x5578b6975360 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0acf93740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5578b692e530 con 0x5578b695dff0 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0acf93740 10 client.0 _ll_register_callbacks cb 0x5578b688e270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0a2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b696feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0a2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b696feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40532/0 (socket says 172.21.15.69:40532) 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0a2a00640 1 -- 172.21.15.69:0/2220629261 learned_addr learned my addr 172.21.15.69:0/2220629261 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0a2a00640 1 -- 172.21.15.69:0/2220629261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b695dff0 msgr2=0x5578b694e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0a3400640 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b695dff0 0x5578b694e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0a3e00640 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b6975360 0x5578b69757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b695dff0 0x5578b694e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0a2a00640 1 -- 172.21.15.69:0/2220629261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b6975360 msgr2=0x5578b69757e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b6975360 0x5578b69757e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.862+0000 7fe0a2a00640 1 -- 172.21.15.69:0/2220629261 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578b66767c0 con 0x5578b697b890 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.863+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b696feb0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fe09000dd90 tx=0x7fe0900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3de394955acfc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.863+0000 7fe083e00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.863+0000 7fe083e00640 1 -- 172.21.15.69:0/2220629261 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09001a050 con 0x5578b697b890 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.863+0000 7fe083e00640 1 -- 172.21.15.69:0/2220629261 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7fe09001f440 con 0x5578b697b890 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.863+0000 7fe0acf93740 1 -- 172.21.15.69:0/2220629261 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe094057040 con 0x5578b697b890 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.863+0000 7fe083e00640 1 -- 172.21.15.69:0/2220629261 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7fe090005990 con 0x5578b697b890 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.863+0000 7fe0acf93740 10 client.7983 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.863+0000 7fe0acf93740 1 -- 172.21.15.69:0/2220629261 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x5578b66d6f50 con 0x5578b697b890 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.863+0000 7fe083e00640 1 -- 172.21.15.69:0/2220629261 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe094057040 con 0x5578b697b890 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.864+0000 7fe083e00640 1 -- 172.21.15.69:0/2220629261 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7fe09002f080 con 0x5578b697b890 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.864+0000 7fe0acf93740 10 client.7983 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:24.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.864+0000 7fe0acf93740 10 client.7983 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.864+0000 7fe0acf93740 20 client.7983 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.864+0000 7fe0acf93740 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5578b6989d10 0x5578b698a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.864+0000 7fe0acf93740 4 client.7983 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.864+0000 7fe0acf93740 1 -- 172.21.15.69:0/2220629261 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578b68a5bb0 con 0x5578b6989d10 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.864+0000 7fe0a3400640 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5578b6989d10 0x5578b698a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.864+0000 7fe0a3400640 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5578b6989d10 0x5578b698a0f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b176f405ab441d50 server_cookie=b67bc3fa15125ca3 in_seq=0 out_seq=0 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.864+0000 7fe083e00640 10 client.7983 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe083e00640 1 -- 172.21.15.69:0/2220629261 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5578b68a5bb0 con 0x5578b6989d10 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe083e00640 10 client.7983 handle_command_reply: tid=0 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe0acf93740 1 client.7983 shutdown 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe0acf93740 1 -- 172.21.15.69:0/2220629261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 msgr2=0x5578b696feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe0acf93740 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b696feb0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fe09000dd90 tx=0x7fe0900046c0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe0acf93740 1 -- 172.21.15.69:0/2220629261 shutdown_connections 2024-04-24T08:16:24.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe0acf93740 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5578b6989d10 0x5578b698a0f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe0acf93740 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b696feb0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe0acf93740 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b6975360 0x5578b69757e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe0acf93740 1 --2- 172.21.15.69:0/2220629261 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b695dff0 0x5578b694e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.865+0000 7fe083e00640 0 client.7983 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:24.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.866+0000 7fe0acf93740 1 -- 172.21.15.69:0/2220629261 >> 172.21.15.69:0/2220629261 conn(0x5578b6933cd0 msgr2=0x5578b697cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.866+0000 7fe0acf93740 1 -- 172.21.15.69:0/2220629261 shutdown_connections 2024-04-24T08:16:24.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.866+0000 7fe0acf93740 1 -- 172.21.15.69:0/2220629261 wait complete. 2024-04-24T08:16:24.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.866+0000 7fe0acf93740 20 client.7983 trim_cache size 0 max 16384 2024-04-24T08:16:24.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.880+0000 7fe0acf93740 1 Processor -- start 2024-04-24T08:16:24.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.881+0000 7fe0acf93740 1 -- start start 2024-04-24T08:16:24.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.881+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b695dff0 0x5578b695d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.881+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 0x5578b692d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.881+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b692d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.881+0000 7fe0acf93740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a4205650 con 0x5578b697b890 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.881+0000 7fe0acf93740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a4204650 con 0x5578b695dff0 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.881+0000 7fe0acf93740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a409c9d0 con 0x5578b6975360 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b692d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b692d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40548/0 (socket says 172.21.15.69:40548) 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a3e00640 1 -- 172.21.15.69:0/1931755392 learned_addr learned my addr 172.21.15.69:0/1931755392 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/1931755392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 0x5578b692d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a3400640 1 --2- 172.21.15.69:0/1931755392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b695dff0 0x5578b695d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a3e00640 1 -- 172.21.15.69:0/1931755392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 msgr2=0x5578b692d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a3e00640 1 --2- 172.21.15.69:0/1931755392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 0x5578b692d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a3e00640 1 -- 172.21.15.69:0/1931755392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b695dff0 msgr2=0x5578b695d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a3e00640 1 --2- 172.21.15.69:0/1931755392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b695dff0 0x5578b695d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a3e00640 1 -- 172.21.15.69:0/1931755392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578b656a0b0 con 0x5578b697b890 2024-04-24T08:16:24.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/1931755392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 0x5578b692d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:24.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.882+0000 7fe0a3e00640 1 --2- 172.21.15.69:0/1931755392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b692d820 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fe09c02a750 tx=0x7fe09c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d3321557ae3433 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe08ea00640 1 -- 172.21.15.69:0/1931755392 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c041020 con 0x5578b697b890 2024-04-24T08:16:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe08ea00640 1 -- 172.21.15.69:0/1931755392 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe09c037040 con 0x5578b697b890 2024-04-24T08:16:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe08ea00640 1 -- 172.21.15.69:0/1931755392 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c036070 con 0x5578b697b890 2024-04-24T08:16:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe0acf93740 1 -- 172.21.15.69:0/1931755392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 msgr2=0x5578b692d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe0acf93740 1 --2- 172.21.15.69:0/1931755392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b692d820 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fe09c02a750 tx=0x7fe09c004ab0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe0acf93740 1 -- 172.21.15.69:0/1931755392 shutdown_connections 2024-04-24T08:16:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe0acf93740 1 --2- 172.21.15.69:0/1931755392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b697b890 0x5578b692d820 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe0acf93740 1 --2- 172.21.15.69:0/1931755392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 0x5578b692d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe0acf93740 1 --2- 172.21.15.69:0/1931755392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b695dff0 0x5578b695d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe0acf93740 1 -- 172.21.15.69:0/1931755392 >> 172.21.15.69:0/1931755392 conn(0x5578b6933cd0 msgr2=0x5578b69340d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe0acf93740 1 -- 172.21.15.69:0/1931755392 shutdown_connections 2024-04-24T08:16:24.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.883+0000 7fe0acf93740 1 -- 172.21.15.69:0/1931755392 wait complete. 2024-04-24T08:16:24.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.884+0000 7fe0acf93740 1 -- start start 2024-04-24T08:16:24.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.884+0000 7fe0acf93740 1 Processor -- start 2024-04-24T08:16:24.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.885+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b69f2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.885+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 0x5578b695ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.885+0000 7fe0acf93740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b697b890 0x5578b6932270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.885+0000 7fe0acf93740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5578b6971580 con 0x5578b695dff0 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.885+0000 7fe0acf93740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a4204000 con 0x5578b697b890 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.885+0000 7fe0acf93740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5578b6971210 con 0x5578b6975360 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.885+0000 7fe0acf93740 10 client.0 _ll_register_callbacks cb 0x7fe09000b580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b69f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b69f2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40552/0 (socket says 172.21.15.69:40552) 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a3400640 1 -- 172.21.15.69:0/1805004643 learned_addr learned my addr 172.21.15.69:0/1805004643 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a3400640 1 -- 172.21.15.69:0/1805004643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 msgr2=0x5578b695ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 0x5578b695ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a3e00640 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b697b890 0x5578b6932270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a3400640 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 0x5578b695ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a3400640 1 -- 172.21.15.69:0/1805004643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b697b890 msgr2=0x5578b6932270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a3400640 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b697b890 0x5578b6932270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a3400640 1 -- 172.21.15.69:0/1805004643 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578b69327b0 con 0x5578b695dff0 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe0a3400640 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b69f2290 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fe0940589f0 tx=0x7fe094000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630d8a198f96a5b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe083e00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.886+0000 7fe083e00640 1 -- 172.21.15.69:0/1805004643 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe094056de0 con 0x5578b695dff0 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe083e00640 1 -- 172.21.15.69:0/1805004643 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7fe094057040 con 0x5578b695dff0 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe0acf93740 1 -- 172.21.15.69:0/1805004643 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe09c037040 con 0x5578b695dff0 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe083e00640 1 -- 172.21.15.69:0/1805004643 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7fe0940553b0 con 0x5578b695dff0 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe0acf93740 10 client.7989 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe0acf93740 1 -- 172.21.15.69:0/1805004643 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x5578b6974490 con 0x5578b695dff0 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe083e00640 1 -- 172.21.15.69:0/1805004643 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe09c037040 con 0x5578b695dff0 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe083e00640 1 -- 172.21.15.69:0/1805004643 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7fe094071180 con 0x5578b695dff0 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe0acf93740 10 client.7989 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe0acf93740 10 client.7989 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe0acf93740 20 client.7989 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:24.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe0acf93740 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5578b692d5b0 0x5578b696e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe0acf93740 4 client.7989 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe0acf93740 1 -- 172.21.15.69:0/1805004643 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x5578b69770b0 con 0x5578b692d5b0 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.887+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5578b692d5b0 0x5578b696e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.888+0000 7fe0a2a00640 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5578b692d5b0 0x5578b696e790 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48ce85eeba3cd22 server_cookie=bb0c78a5f35a9f0a in_seq=0 out_seq=0 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.888+0000 7fe083e00640 10 client.7989 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.888+0000 7fe083e00640 1 -- 172.21.15.69:0/1805004643 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+418 (crc 0 0 0) 0x5578b69770b0 con 0x5578b692d5b0 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.888+0000 7fe083e00640 10 client.7989 handle_command_reply: tid=0 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.888+0000 7fe0acf93740 1 client.7989 shutdown 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.888+0000 7fe0acf93740 1 -- 172.21.15.69:0/1805004643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 msgr2=0x5578b69f2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.888+0000 7fe0acf93740 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b69f2290 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fe0940589f0 tx=0x7fe094000bc0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.889+0000 7fe0acf93740 1 -- 172.21.15.69:0/1805004643 shutdown_connections 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.889+0000 7fe0acf93740 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x5578b692d5b0 0x5578b696e790 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.889+0000 7fe0acf93740 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5578b697b890 0x5578b6932270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.889+0000 7fe0acf93740 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5578b6975360 0x5578b695ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.889+0000 7fe0acf93740 1 --2- 172.21.15.69:0/1805004643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5578b695dff0 0x5578b69f2290 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.889+0000 7fe083e00640 0 client.7989 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:24.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.889+0000 7fe0acf93740 1 -- 172.21.15.69:0/1805004643 >> 172.21.15.69:0/1805004643 conn(0x5578b6933cd0 msgr2=0x5578b69340d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.889+0000 7fe0acf93740 1 -- 172.21.15.69:0/1805004643 shutdown_connections 2024-04-24T08:16:24.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.889+0000 7fe0acf93740 1 -- 172.21.15.69:0/1805004643 wait complete. 2024-04-24T08:16:24.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.889+0000 7fe0acf93740 20 client.7989 trim_cache size 0 max 16384 2024-04-24T08:16:24.894 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:16:24.894 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:16:24.894 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "dir_frag", 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 62634788, 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511627776, 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "frag": "*", 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/dir" 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 1301104333, 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511627777, 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/testdir/hardlink" 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-04-24T08:16:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 3082222339, 2024-04-24T08:16:24.896 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511627776, 2024-04-24T08:16:24.896 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/dir" 2024-04-24T08:16:24.896 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:16:24.896 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:16:24.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 -- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe08406c8f0 msgr2=0x7fe08406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe08406c8f0 0x7fe08406edb0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe090006fd0 tx=0x7fe090008040 comp rx=0 tx=0).stop 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 -- 172.21.15.69:0/871035066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a4073db0 msgr2=0x7fe0a4157730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a4073db0 0x7fe0a4157730 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fe09400c430 tx=0x7fe09400c8f0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 -- 172.21.15.69:0/871035066 shutdown_connections 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe08406c8f0 0x7fe08406edb0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe0a40cd3d0 0x7fe0a41652e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0a40b4780 0x7fe0a4164da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 --2- 172.21.15.69:0/871035066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a4073db0 0x7fe0a4157730 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 -- 172.21.15.69:0/871035066 >> 172.21.15.69:0/871035066 conn(0x7fe0a40d0350 msgr2=0x7fe0a40ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 -- 172.21.15.69:0/871035066 shutdown_connections 2024-04-24T08:16:24.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:24.890+0000 7fe0aa200640 1 -- 172.21.15.69:0/871035066 wait complete. 2024-04-24T08:16:24.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T08:16:25.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.109+0000 7ff015400640 1 Processor -- start 2024-04-24T08:16:25.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.109+0000 7ff015400640 1 -- start start 2024-04-24T08:16:25.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.110+0000 7ff015400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 0x7ff0100d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.110+0000 7ff015400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0100adb00 0x7ff0100adf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.110+0000 7ff015400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff01015cb80 con 0x7ff01015c7a0 2024-04-24T08:16:25.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.110+0000 7ff015400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff0100d1130 con 0x7ff0100adb00 2024-04-24T08:16:25.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.110+0000 7ff015400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff010074000 con 0x7ff0100d0790 2024-04-24T08:16:25.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.111+0000 7ff00f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff0100adb00 0x7ff0100adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.111+0000 7ff00de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 0x7ff0100d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.111+0000 7ff00de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 0x7ff0100d0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:38520/0 (socket says 172.21.15.69:38520) 2024-04-24T08:16:25.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.111+0000 7ff00de00640 1 -- 172.21.15.69:0/1185933731 learned_addr learned my addr 172.21.15.69:0/1185933731 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.112+0000 7ff00de00640 1 -- 172.21.15.69:0/1185933731 >> v2:172.21.15.121:3300/0 conn(0x7ff0100adb00 msgr2=0x7ff0100adf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.112+0000 7ff00de00640 1 --2- 172.21.15.69:0/1185933731 >> v2:172.21.15.121:3300/0 conn(0x7ff0100adb00 0x7ff0100adf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.112+0000 7ff00de00640 1 -- 172.21.15.69:0/1185933731 >> v1:172.21.15.121:6789/0 conn(0x7ff01015c7a0 legacy=0x7ff010072ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.112+0000 7ff00de00640 1 -- 172.21.15.69:0/1185933731 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff010073a40 con 0x7ff0100d0790 2024-04-24T08:16:25.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.112+0000 7ff00f200640 1 --2- 172.21.15.69:0/1185933731 >> v2:172.21.15.121:3300/0 conn(0x7ff0100adb00 0x7ff0100adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:16:25.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.113+0000 7ff00de00640 1 --2- 172.21.15.69:0/1185933731 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 0x7ff0100d0bf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feffc009870 tx=0x7feffc02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f75d4ba2beb015fd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.113+0000 7ff00d400640 1 -- 172.21.15.69:0/1185933731 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc037070 con 0x7ff0100d0790 2024-04-24T08:16:25.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.114+0000 7ff00d400640 1 -- 172.21.15.69:0/1185933731 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feffc032440 con 0x7ff0100d0790 2024-04-24T08:16:25.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.114+0000 7ff00d400640 1 -- 172.21.15.69:0/1185933731 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc032810 con 0x7ff0100d0790 2024-04-24T08:16:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.114+0000 7ff015400640 1 -- 172.21.15.69:0/1185933731 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 msgr2=0x7ff0100d0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.114+0000 7ff015400640 1 --2- 172.21.15.69:0/1185933731 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 0x7ff0100d0bf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7feffc009870 tx=0x7feffc02a1c0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.114+0000 7ff015400640 1 -- 172.21.15.69:0/1185933731 shutdown_connections 2024-04-24T08:16:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.114+0000 7ff015400640 1 --2- 172.21.15.69:0/1185933731 >> v2:172.21.15.121:3300/0 conn(0x7ff0100adb00 0x7ff0100adf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.114+0000 7ff015400640 1 --2- 172.21.15.69:0/1185933731 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 0x7ff0100d0bf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.114+0000 7ff015400640 1 -- 172.21.15.69:0/1185933731 >> 172.21.15.69:0/1185933731 conn(0x7ff01006be40 msgr2=0x7ff01006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.114+0000 7ff015400640 1 -- 172.21.15.69:0/1185933731 shutdown_connections 2024-04-24T08:16:25.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.114+0000 7ff015400640 1 -- 172.21.15.69:0/1185933731 wait complete. 2024-04-24T08:16:25.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.116+0000 7ff015400640 1 Processor -- start 2024-04-24T08:16:25.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.116+0000 7ff015400640 1 -- start start 2024-04-24T08:16:25.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.117+0000 7ff015400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0100adb00 0x7ff0100c9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.117+0000 7ff015400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 0x7ff0101f7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.117+0000 7ff015400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff01015c7a0 0x7ff0100c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.117+0000 7ff015400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff0100ad300 con 0x7ff01015c7a0 2024-04-24T08:16:25.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.117+0000 7ff015400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff0100ad670 con 0x7ff0100adb00 2024-04-24T08:16:25.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.117+0000 7ff015400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff0100b4cc0 con 0x7ff0100d0790 2024-04-24T08:16:25.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.117+0000 7ff00f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff01015c7a0 0x7ff0100c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.117+0000 7ff00f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff01015c7a0 0x7ff0100c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40556/0 (socket says 172.21.15.69:40556) 2024-04-24T08:16:25.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.117+0000 7ff00f200640 1 -- 172.21.15.69:0/769141986 learned_addr learned my addr 172.21.15.69:0/769141986 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff00e800640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0100adb00 0x7ff0100c9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff00de00640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 0x7ff0101f7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff00f200640 1 -- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 msgr2=0x7ff0101f7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff00f200640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 0x7ff0101f7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff00f200640 1 -- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0100adb00 msgr2=0x7ff0100c9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff00f200640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0100adb00 0x7ff0100c9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff00f200640 1 -- 172.21.15.69:0/769141986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0100cea20 con 0x7ff01015c7a0 2024-04-24T08:16:25.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff00e800640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0100adb00 0x7ff0100c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:25.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff00f200640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff01015c7a0 0x7ff0100c7680 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ff00800efc0 tx=0x7ff00800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff006a00640 1 -- 172.21.15.69:0/769141986 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff008018030 con 0x7ff01015c7a0 2024-04-24T08:16:25.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff006a00640 1 -- 172.21.15.69:0/769141986 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff00800cdf0 con 0x7ff01015c7a0 2024-04-24T08:16:25.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff006a00640 1 -- 172.21.15.69:0/769141986 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff008003bb0 con 0x7ff01015c7a0 2024-04-24T08:16:25.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.118+0000 7ff015400640 1 -- 172.21.15.69:0/769141986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0100cecc0 con 0x7ff01015c7a0 2024-04-24T08:16:25.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.119+0000 7ff015400640 1 -- 172.21.15.69:0/769141986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff010155bf0 con 0x7ff01015c7a0 2024-04-24T08:16:25.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.120+0000 7ff015400640 1 -- 172.21.15.69:0/769141986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff010075dc0 con 0x7ff01015c7a0 2024-04-24T08:16:25.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.121+0000 7ff006a00640 1 -- 172.21.15.69:0/769141986 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff008005d10 con 0x7ff01015c7a0 2024-04-24T08:16:25.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.122+0000 7ff006a00640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feff006c920 0x7feff006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.122+0000 7ff006a00640 1 -- 172.21.15.69:0/769141986 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7ff00808c5e0 con 0x7ff01015c7a0 2024-04-24T08:16:25.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.122+0000 7ff00e800640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feff006c920 0x7feff006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.123+0000 7ff00e800640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feff006c920 0x7feff006ede0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff000001680 tx=0x7ff000009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.126+0000 7ff006a00640 1 -- 172.21.15.69:0/769141986 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff008022070 con 0x7ff01015c7a0 2024-04-24T08:16:25.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.366+0000 7ff015400640 1 -- 172.21.15.69:0/769141986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff010159a00 con 0x7ff01015c7a0 2024-04-24T08:16:25.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.367+0000 7ff006a00640 1 -- 172.21.15.69:0/769141986 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+164 (secure 0 0 0) 0x7ff008010040 con 0x7ff01015c7a0 2024-04-24T08:16:25.369 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:16:25.369 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-04-24T08:16:25.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.370+0000 7ff015400640 1 -- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feff006c920 msgr2=0x7feff006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.370+0000 7ff015400640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feff006c920 0x7feff006ede0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff000001680 tx=0x7ff000009040 comp rx=0 tx=0).stop 2024-04-24T08:16:25.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.370+0000 7ff015400640 1 -- 172.21.15.69:0/769141986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff01015c7a0 msgr2=0x7ff0100c7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.370+0000 7ff015400640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff01015c7a0 0x7ff0100c7680 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7ff00800efc0 tx=0x7ff00800c490 comp rx=0 tx=0).stop 2024-04-24T08:16:25.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.370+0000 7ff015400640 1 -- 172.21.15.69:0/769141986 shutdown_connections 2024-04-24T08:16:25.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.370+0000 7ff015400640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7feff006c920 0x7feff006ede0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.370+0000 7ff015400640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff01015c7a0 0x7ff0100c7680 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.370+0000 7ff015400640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff0100d0790 0x7ff0101f7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.370+0000 7ff015400640 1 --2- 172.21.15.69:0/769141986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff0100adb00 0x7ff0100c9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.370+0000 7ff015400640 1 -- 172.21.15.69:0/769141986 >> 172.21.15.69:0/769141986 conn(0x7ff01006be40 msgr2=0x7ff0100af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.371+0000 7ff015400640 1 -- 172.21.15.69:0/769141986 shutdown_connections 2024-04-24T08:16:25.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.371+0000 7ff015400640 1 -- 172.21.15.69:0/769141986 wait complete. 2024-04-24T08:16:25.392 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-24T08:16:25.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:16:25.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-04-24T08:16:25.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.596+0000 7f2763000640 1 Processor -- start 2024-04-24T08:16:25.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.596+0000 7f2763000640 1 -- start start 2024-04-24T08:16:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.597+0000 7f2763000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f275c0d07c0 0x7f275c0d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.597+0000 7f2763000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f275c0b4840 con 0x7f275c0addc0 2024-04-24T08:16:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.597+0000 7f2763000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f275c0b4560 con 0x7f275c15c590 2024-04-24T08:16:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.597+0000 7f2763000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f275c0ad2e0 con 0x7f275c0d07c0 2024-04-24T08:16:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.598+0000 7f2761000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f275c0addc0 0x7f275c0ae1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:36448/0 (socket says 172.21.15.69:36448) 2024-04-24T08:16:25.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.598+0000 7f2761000640 1 -- 172.21.15.69:0/3441908553 learned_addr learned my addr 172.21.15.69:0/3441908553 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.598+0000 7f275b400640 1 --2- 172.21.15.69:0/3441908553 >> v2:172.21.15.121:3300/0 conn(0x7f275c0d07c0 0x7f275c0d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.598+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3069848022 0 0) 0x7f275c0b4840 con 0x7f275c0addc0 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.598+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2740003520 con 0x7f275c0addc0 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.598+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2591631614 0 0) 0x7f2740003520 con 0x7f275c0addc0 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 >> v2:172.21.15.121:3300/0 conn(0x7f275c0d07c0 msgr2=0x7f275c0d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 --2- 172.21.15.69:0/3441908553 >> v2:172.21.15.121:3300/0 conn(0x7f275c0d07c0 0x7f275c0d0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 >> v1:172.21.15.121:6789/0 conn(0x7f275c15c590 legacy=0x7f275c0b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275c0ad5c0 con 0x7f275c0addc0 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f27540019f0 con 0x7f275c0addc0 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 >> v1:172.21.15.69:6789/0 conn(0x7f275c0addc0 legacy=0x7f275c0ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 --2- 172.21.15.69:0/3441908553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f2740003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 --2- 172.21.15.69:0/3441908553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2740004800 0x7f2740004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 --2- 172.21.15.69:0/3441908553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27400051c0 0x7f274000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f275c0ad2e0 con 0x7f27400039b0 2024-04-24T08:16:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f274000c040 con 0x7f2740004800 2024-04-24T08:16:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f274000c240 con 0x7f27400051c0 2024-04-24T08:16:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275be00640 1 --2- 172.21.15.69:0/3441908553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f2740003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275be00640 1 -- 172.21.15.69:0/3441908553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27400051c0 msgr2=0x7f274000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275be00640 1 --2- 172.21.15.69:0/3441908553 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27400051c0 0x7f274000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275be00640 1 -- 172.21.15.69:0/3441908553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2740004800 msgr2=0x7f2740004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275be00640 1 --2- 172.21.15.69:0/3441908553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2740004800 0x7f2740004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.599+0000 7f275be00640 1 -- 172.21.15.69:0/3441908553 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2740003520 con 0x7f27400039b0 2024-04-24T08:16:25.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.600+0000 7f2761000640 1 -- 172.21.15.69:0/3441908553 reap_dead start 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.601+0000 7f275be00640 1 --2- 172.21.15.69:0/3441908553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f2740003db0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f274c003db0 tx=0x7f2740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8edf0d2772cf90 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.601+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c037070 con 0x7f27400039b0 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.601+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f274c02aa00 con 0x7f27400039b0 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.601+0000 7f275aa00640 1 -- 172.21.15.69:0/3441908553 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c003980 con 0x7f27400039b0 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.602+0000 7f2763000640 1 -- 172.21.15.69:0/3441908553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 msgr2=0x7f2740003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.602+0000 7f2763000640 1 --2- 172.21.15.69:0/3441908553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f2740003db0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f274c003db0 tx=0x7f2740005830 comp rx=0 tx=0).stop 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.602+0000 7f2763000640 1 -- 172.21.15.69:0/3441908553 shutdown_connections 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.602+0000 7f2763000640 1 --2- 172.21.15.69:0/3441908553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f2740003db0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.602+0000 7f2763000640 1 -- 172.21.15.69:0/3441908553 >> 172.21.15.69:0/3441908553 conn(0x7f275c06be40 msgr2=0x7f275c06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.602+0000 7f2763000640 1 -- 172.21.15.69:0/3441908553 shutdown_connections 2024-04-24T08:16:25.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.602+0000 7f2763000640 1 -- 172.21.15.69:0/3441908553 wait complete. 2024-04-24T08:16:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.603+0000 7f2763000640 1 Processor -- start 2024-04-24T08:16:25.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.604+0000 7f2763000640 1 -- start start 2024-04-24T08:16:25.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.604+0000 7f2763000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f275c0c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.604+0000 7f2763000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f275c0c5860 0x7f275c1f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f2763000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f275c1f6960 0x7f275c099030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f2763000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f275c0b52e0 con 0x7f27400039b0 2024-04-24T08:16:25.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f2763000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f275c0b5160 con 0x7f275c1f6960 2024-04-24T08:16:25.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f2763000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f275c15c970 con 0x7f275c0c5860 2024-04-24T08:16:25.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f275be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f275c0c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f275be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f275c0c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40566/0 (socket says 172.21.15.69:40566) 2024-04-24T08:16:25.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f275be00640 1 -- 172.21.15.69:0/2891988782 learned_addr learned my addr 172.21.15.69:0/2891988782 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f275b400640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f275c0c5860 0x7f275c1f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f275be00640 1 -- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f275c0c5860 msgr2=0x7f275c1f6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f275be00640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f275c0c5860 0x7f275c1f6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f275be00640 1 -- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f275c1f6960 msgr2=0x7f275c099030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f275be00640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f275c1f6960 0x7f275c099030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f275be00640 1 -- 172.21.15.69:0/2891988782 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f275c0c5ce0 con 0x7f27400039b0 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.605+0000 7f275be00640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f275c0c9fc0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f274c035040 tx=0x7f274c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.606+0000 7f2753e00640 1 -- 172.21.15.69:0/2891988782 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c037050 con 0x7f27400039b0 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.606+0000 7f2753e00640 1 -- 172.21.15.69:0/2891988782 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f274c038040 con 0x7f27400039b0 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.606+0000 7f2763000640 1 -- 172.21.15.69:0/2891988782 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f275c0c5f90 con 0x7f27400039b0 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.606+0000 7f2753e00640 1 -- 172.21.15.69:0/2891988782 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c006e50 con 0x7f27400039b0 2024-04-24T08:16:25.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.606+0000 7f2763000640 1 -- 172.21.15.69:0/2891988782 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f275c0c6550 con 0x7f27400039b0 2024-04-24T08:16:25.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.608+0000 7f2753e00640 1 -- 172.21.15.69:0/2891988782 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f274c045020 con 0x7f27400039b0 2024-04-24T08:16:25.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.610+0000 7f2753e00640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f273c06cac0 0x7f273c06ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.610+0000 7f2753e00640 1 -- 172.21.15.69:0/2891988782 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f274c0ab910 con 0x7f27400039b0 2024-04-24T08:16:25.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.610+0000 7f275b400640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f273c06cac0 0x7f273c06ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.611+0000 7f2765b22740 1 Processor -- start 2024-04-24T08:16:25.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.611+0000 7f2765b22740 1 -- start start 2024-04-24T08:16:25.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f275b400640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f273c06cac0 0x7f273c06ef80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f275c0b4c70 tx=0x7f2748008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f2765b22740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a414486df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f2765b22740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144b7580 0x55a4144b6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f2765b22740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b6e40 0x55a41448bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f2765b22740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a4143f0e70 con 0x55a4144d4460 2024-04-24T08:16:25.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f2765b22740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a414397380 con 0x55a4144b7580 2024-04-24T08:16:25.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f2765b22740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a4143ece30 con 0x55a4144b6e40 2024-04-24T08:16:25.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f275be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a414486df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f275be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a414486df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40570/0 (socket says 172.21.15.69:40570) 2024-04-24T08:16:25.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f275be00640 1 -- 172.21.15.69:0/653389737 learned_addr learned my addr 172.21.15.69:0/653389737 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f275b400640 1 --2- 172.21.15.69:0/653389737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144b7580 0x55a4144b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.612+0000 7f2761000640 1 --2- 172.21.15.69:0/653389737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b6e40 0x55a41448bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.613+0000 7f275be00640 1 -- 172.21.15.69:0/653389737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b6e40 msgr2=0x55a41448bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.613+0000 7f275be00640 1 --2- 172.21.15.69:0/653389737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b6e40 0x55a41448bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.613+0000 7f275be00640 1 -- 172.21.15.69:0/653389737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144b7580 msgr2=0x55a4144b6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.613+0000 7f275be00640 1 --2- 172.21.15.69:0/653389737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144b7580 0x55a4144b6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.613+0000 7f275be00640 1 -- 172.21.15.69:0/653389737 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4141cb7e0 con 0x55a4144d4460 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.613+0000 7f275be00640 1 --2- 172.21.15.69:0/653389737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a414486df0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f274c0a9d80 tx=0x7f274c079250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1582d5baab30d0c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.613+0000 7f2750c00640 1 -- 172.21.15.69:0/653389737 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c0a8070 con 0x55a4144d4460 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.613+0000 7f2750c00640 1 -- 172.21.15.69:0/653389737 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f274c07d990 con 0x55a4144d4460 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.613+0000 7f2750c00640 1 -- 172.21.15.69:0/653389737 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c0821c0 con 0x55a4144d4460 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.614+0000 7f2765b22740 1 -- 172.21.15.69:0/653389737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 msgr2=0x55a414486df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.614+0000 7f2765b22740 1 --2- 172.21.15.69:0/653389737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a414486df0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f274c0a9d80 tx=0x7f274c079250 comp rx=0 tx=0).stop 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.614+0000 7f2765b22740 1 -- 172.21.15.69:0/653389737 shutdown_connections 2024-04-24T08:16:25.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.614+0000 7f2765b22740 1 --2- 172.21.15.69:0/653389737 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b6e40 0x55a41448bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.614+0000 7f2765b22740 1 --2- 172.21.15.69:0/653389737 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144b7580 0x55a4144b6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.614+0000 7f2765b22740 1 --2- 172.21.15.69:0/653389737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a414486df0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.614+0000 7f2765b22740 1 -- 172.21.15.69:0/653389737 >> 172.21.15.69:0/653389737 conn(0x55a414489060 msgr2=0x55a4144d1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.614+0000 7f2765b22740 1 -- 172.21.15.69:0/653389737 shutdown_connections 2024-04-24T08:16:25.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.614+0000 7f2765b22740 1 -- 172.21.15.69:0/653389737 wait complete. 2024-04-24T08:16:25.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.615+0000 7f2765b22740 1 -- start start 2024-04-24T08:16:25.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.615+0000 7f2765b22740 1 Processor -- start 2024-04-24T08:16:25.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f2765b22740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144b7580 0x55a4144a72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f2765b22740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a4144ba700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f2765b22740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144cb1a0 0x55a4144d2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f2765b22740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a414397380 con 0x55a4144d4460 2024-04-24T08:16:25.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f2765b22740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a4143aa660 con 0x55a4144b7580 2024-04-24T08:16:25.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f2765b22740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a4143f0e70 con 0x55a4144cb1a0 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f2765b22740 10 client.0 _ll_register_callbacks cb 0x55a414398aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f275b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a4144ba700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f275b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a4144ba700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40586/0 (socket says 172.21.15.69:40586) 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f275b400640 1 -- 172.21.15.69:0/1026490864 learned_addr learned my addr 172.21.15.69:0/1026490864 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.616+0000 7f2761000640 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144cb1a0 0x55a4144d2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f275be00640 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144b7580 0x55a4144a72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f275b400640 1 -- 172.21.15.69:0/1026490864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144cb1a0 msgr2=0x55a4144d2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f275b400640 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144cb1a0 0x55a4144d2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f275b400640 1 -- 172.21.15.69:0/1026490864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144b7580 msgr2=0x55a4144a72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f275b400640 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144b7580 0x55a4144a72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f275b400640 1 -- 172.21.15.69:0/1026490864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a41422bf70 con 0x55a4144d4460 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f2761000640 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144cb1a0 0x55a4144d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f275b400640 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a4144ba700 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f2748015700 tx=0x7f2748004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f707fd7c7f13066 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f2745600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f2745600640 1 -- 172.21.15.69:0/1026490864 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274801c070 con 0x55a4144d4460 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f2745600640 1 -- 172.21.15.69:0/1026490864 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f2748016970 con 0x55a4144d4460 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f2765b22740 1 -- 172.21.15.69:0/1026490864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f274c07d990 con 0x55a4144d4460 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.617+0000 7f2745600640 1 -- 172.21.15.69:0/1026490864 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f274802f050 con 0x55a4144d4460 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f2765b22740 10 client.8004 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f2765b22740 1 -- 172.21.15.69:0/1026490864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55a4140c3170 con 0x55a4144d4460 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f2745600640 1 -- 172.21.15.69:0/1026490864 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f274c07d990 con 0x55a4144d4460 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f2745600640 1 -- 172.21.15.69:0/1026490864 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f2748032020 con 0x55a4144d4460 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f2765b22740 10 client.8004 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f2765b22740 10 client.8004 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f2765b22740 20 client.8004 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f2765b22740 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a4144b9a20 0x55a4144eed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f2765b22740 4 client.8004 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:25.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f2765b22740 1 -- 172.21.15.69:0/1026490864 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a4143fe890 con 0x55a4144b9a20 2024-04-24T08:16:25.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.618+0000 7f275be00640 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a4144b9a20 0x55a4144eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.619+0000 7f275be00640 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a4144b9a20 0x55a4144eed00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccaa45564ab8ffd server_cookie=3d804303695eebe5 in_seq=0 out_seq=0 2024-04-24T08:16:25.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.619+0000 7f2745600640 10 client.8004 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.619+0000 7f2745600640 1 -- 172.21.15.69:0/1026490864 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a4143fe890 con 0x55a4144b9a20 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.619+0000 7f2745600640 10 client.8004 handle_command_reply: tid=0 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.619+0000 7f2765b22740 1 client.8004 shutdown 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 1 -- 172.21.15.69:0/1026490864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 msgr2=0x55a4144ba700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a4144ba700 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f2748015700 tx=0x7f2748004730 comp rx=0 tx=0).stop 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 1 -- 172.21.15.69:0/1026490864 shutdown_connections 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a4144b9a20 0x55a4144eed00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144d4460 0x55a4144ba700 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144cb1a0 0x55a4144d2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 1 --2- 172.21.15.69:0/1026490864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144b7580 0x55a4144a72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2745600640 0 client.8004 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:25.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 1 -- 172.21.15.69:0/1026490864 >> 172.21.15.69:0/1026490864 conn(0x55a4144b6e40 msgr2=0x55a414486510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 1 -- 172.21.15.69:0/1026490864 shutdown_connections 2024-04-24T08:16:25.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 1 -- 172.21.15.69:0/1026490864 wait complete. 2024-04-24T08:16:25.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.620+0000 7f2765b22740 20 client.8004 trim_cache size 0 max 16384 2024-04-24T08:16:25.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.639+0000 7f2765b22740 1 Processor -- start 2024-04-24T08:16:25.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.640+0000 7f2765b22740 1 -- start start 2024-04-24T08:16:25.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.640+0000 7f2765b22740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144b7580 0x55a4144b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.640+0000 7f2765b22740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144cb1a0 0x55a4144f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.640+0000 7f2765b22740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144d4460 0x55a4144a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.640+0000 7f2765b22740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f275c1ff3b0 con 0x55a4144b7580 2024-04-24T08:16:25.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.640+0000 7f2765b22740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f275c1fe430 con 0x55a4144d4460 2024-04-24T08:16:25.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.640+0000 7f2765b22740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f275c0c9730 con 0x55a4144cb1a0 2024-04-24T08:16:25.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.641+0000 7f275be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144b7580 0x55a4144b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.641+0000 7f275be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144b7580 0x55a4144b7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40588/0 (socket says 172.21.15.69:40588) 2024-04-24T08:16:25.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.641+0000 7f275be00640 1 -- 172.21.15.69:0/1099012110 learned_addr learned my addr 172.21.15.69:0/1099012110 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.641+0000 7f275be00640 1 -- 172.21.15.69:0/1099012110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144cb1a0 msgr2=0x55a4144f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.641+0000 7f275be00640 1 --2- 172.21.15.69:0/1099012110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144cb1a0 0x55a4144f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.641+0000 7f275be00640 1 -- 172.21.15.69:0/1099012110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144d4460 msgr2=0x55a4144a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.641+0000 7f275be00640 1 --2- 172.21.15.69:0/1099012110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144d4460 0x55a4144a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.641+0000 7f275be00640 1 -- 172.21.15.69:0/1099012110 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4144b7c40 con 0x55a4144b7580 2024-04-24T08:16:25.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.641+0000 7f275be00640 1 --2- 172.21.15.69:0/1099012110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144b7580 0x55a4144b7ea0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f274c0792e0 tx=0x7f274c091000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe731d3ff5420ba3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.641+0000 7f2747e00640 1 -- 172.21.15.69:0/1099012110 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c0a8070 con 0x55a4144b7580 2024-04-24T08:16:25.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2747e00640 1 -- 172.21.15.69:0/1099012110 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f274c0821c0 con 0x55a4144b7580 2024-04-24T08:16:25.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2747e00640 1 -- 172.21.15.69:0/1099012110 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c07d990 con 0x55a4144b7580 2024-04-24T08:16:25.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2765b22740 1 -- 172.21.15.69:0/1099012110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144b7580 msgr2=0x55a4144b7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2765b22740 1 --2- 172.21.15.69:0/1099012110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144b7580 0x55a4144b7ea0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f274c0792e0 tx=0x7f274c091000 comp rx=0 tx=0).stop 2024-04-24T08:16:25.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2765b22740 1 -- 172.21.15.69:0/1099012110 shutdown_connections 2024-04-24T08:16:25.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2765b22740 1 --2- 172.21.15.69:0/1099012110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144d4460 0x55a4144a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2765b22740 1 --2- 172.21.15.69:0/1099012110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144cb1a0 0x55a4144f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2765b22740 1 --2- 172.21.15.69:0/1099012110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144b7580 0x55a4144b7ea0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2765b22740 1 -- 172.21.15.69:0/1099012110 >> 172.21.15.69:0/1099012110 conn(0x55a4144b6e40 msgr2=0x55a4144d3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2765b22740 1 -- 172.21.15.69:0/1099012110 shutdown_connections 2024-04-24T08:16:25.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.642+0000 7f2765b22740 1 -- 172.21.15.69:0/1099012110 wait complete. 2024-04-24T08:16:25.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.643+0000 7f2765b22740 1 -- start start 2024-04-24T08:16:25.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.643+0000 7f2765b22740 1 Processor -- start 2024-04-24T08:16:25.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.644+0000 7f2765b22740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b7580 0x55a4144866d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.644+0000 7f2765b22740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144cb1a0 0x55a414533f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.644+0000 7f2765b22740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144d4460 0x55a4145344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.644+0000 7f2765b22740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a4144d1140 con 0x55a4144cb1a0 2024-04-24T08:16:25.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.644+0000 7f2765b22740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a4144b0cf0 con 0x55a4144d4460 2024-04-24T08:16:25.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.644+0000 7f2765b22740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a414534d10 con 0x55a4144b7580 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.644+0000 7f2765b22740 10 client.0 _ll_register_callbacks cb 0x7f2748006bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.644+0000 7f275b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144cb1a0 0x55a414533f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.644+0000 7f275b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144cb1a0 0x55a414533f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40596/0 (socket says 172.21.15.69:40596) 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.644+0000 7f275b400640 1 -- 172.21.15.69:0/1968684337 learned_addr learned my addr 172.21.15.69:0/1968684337 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f2761000640 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144d4460 0x55a4145344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f275be00640 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b7580 0x55a4144866d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f275b400640 1 -- 172.21.15.69:0/1968684337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b7580 msgr2=0x55a4144866d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f275b400640 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b7580 0x55a4144866d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f275b400640 1 -- 172.21.15.69:0/1968684337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144d4460 msgr2=0x55a4145344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f275b400640 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144d4460 0x55a4145344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f275b400640 1 -- 172.21.15.69:0/1968684337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a41448d030 con 0x55a4144cb1a0 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f275be00640 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b7580 0x55a4144866d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f275b400640 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144cb1a0 0x55a414533f80 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f2748010630 tx=0x7f2748009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=296ef32bc0823adf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f2745600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:25.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f2745600640 1 -- 172.21.15.69:0/1968684337 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274801c070 con 0x55a4144cb1a0 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f2745600640 1 -- 172.21.15.69:0/1968684337 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f2748016970 con 0x55a4144cb1a0 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.645+0000 7f2765b22740 1 -- 172.21.15.69:0/1968684337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f274c0821c0 con 0x55a4144cb1a0 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2745600640 1 -- 172.21.15.69:0/1968684337 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f274800f040 con 0x55a4144cb1a0 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2765b22740 10 client.8010 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2765b22740 1 -- 172.21.15.69:0/1968684337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55a4144b7e10 con 0x55a4144cb1a0 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2745600640 1 -- 172.21.15.69:0/1968684337 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f274c0821c0 con 0x55a4144cb1a0 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2745600640 1 -- 172.21.15.69:0/1968684337 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f2748045020 con 0x55a4144cb1a0 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2765b22740 10 client.8010 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2765b22740 10 client.8010 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2765b22740 20 client.8010 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2765b22740 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a4145316e0 0x55a4144b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2765b22740 4 client.8010 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f2765b22740 1 -- 172.21.15.69:0/1968684337 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) -- 0x55a4143f8700 con 0x55a4145316e0 2024-04-24T08:16:25.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.646+0000 7f275be00640 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a4145316e0 0x55a4144b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.647+0000 7f275be00640 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a4145316e0 0x55a4144b2de0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4430780a83715f14 server_cookie=156fc77fe56682a4 in_seq=0 out_seq=0 2024-04-24T08:16:25.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.647+0000 7f2745600640 10 client.8010 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2745600640 1 -- 172.21.15.69:0/1968684337 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a4143f8700 con 0x55a4145316e0 2024-04-24T08:16:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2745600640 10 client.8010 handle_command_reply: tid=0 2024-04-24T08:16:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 client.8010 shutdown 2024-04-24T08:16:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 -- 172.21.15.69:0/1968684337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144cb1a0 msgr2=0x55a414533f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144cb1a0 0x55a414533f80 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f2748010630 tx=0x7f2748009190 comp rx=0 tx=0).stop 2024-04-24T08:16:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 -- 172.21.15.69:0/1968684337 shutdown_connections 2024-04-24T08:16:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55a4145316e0 0x55a4144b2de0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4144cb1a0 0x55a414533f80 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a4144d4460 0x55a4145344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 --2- 172.21.15.69:0/1968684337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a4144b7580 0x55a4144866d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 -- 172.21.15.69:0/1968684337 >> 172.21.15.69:0/1968684337 conn(0x55a4144b6e40 msgr2=0x55a4144eb420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2745600640 0 client.8010 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:25.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 -- 172.21.15.69:0/1968684337 shutdown_connections 2024-04-24T08:16:25.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 1 -- 172.21.15.69:0/1968684337 wait complete. 2024-04-24T08:16:25.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.648+0000 7f2765b22740 20 client.8010 trim_cache size 0 max 16384 2024-04-24T08:16:25.653 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:16:25.653 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-04-24T08:16:25.653 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "c17ba41f-461a-4b1d-aa97-d779ca376add", 2024-04-24T08:16:25.653 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-04-24T08:16:25.653 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:16:25.654 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:16:25.649+0000 7faf1f600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 pv17 ap=3 DIRTYPARENT f(v0 m2024-04-24T08:15:32.898213+0000 1=1+0)->f(v1 m2024-04-24T08:15:32.898213+0000) n(v0 rc2024-04-24T08:15:33.020316+0000 2=1+1)->n(v0 rc2024-04-24T08:15:33.020316+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55f9f868c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-04-24T08:15:32.898213+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-04-24T08:15:33.020316+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-04-24T08:16:25.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 -- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f273c06cac0 msgr2=0x7f273c06ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f273c06cac0 0x7f273c06ef80 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f275c0b4c70 tx=0x7f2748008040 comp rx=0 tx=0).stop 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 -- 172.21.15.69:0/2891988782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 msgr2=0x7f275c0c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f275c0c9fc0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f274c035040 tx=0x7f274c0036f0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 -- 172.21.15.69:0/2891988782 shutdown_connections 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f273c06cac0 0x7f273c06ef80 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f275c1f6960 0x7f275c099030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f275c0c5860 0x7f275c1f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 --2- 172.21.15.69:0/2891988782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f27400039b0 0x7f275c0c9fc0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 -- 172.21.15.69:0/2891988782 >> 172.21.15.69:0/2891988782 conn(0x7f275c06be40 msgr2=0x7f275c15ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 -- 172.21.15.69:0/2891988782 shutdown_connections 2024-04-24T08:16:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.649+0000 7f2763000640 1 -- 172.21.15.69:0/2891988782 wait complete. 2024-04-24T08:16:25.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:16:25.671 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-24T08:16:25.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.872+0000 7f7015400640 1 Processor -- start 2024-04-24T08:16:25.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.872+0000 7f7015400640 1 -- start start 2024-04-24T08:16:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.873+0000 7f7015400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100b3e90 0x7f70100b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.873+0000 7f7015400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70100ce250 0x7f70100ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.874+0000 7f7015400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f701015cc00 con 0x7f701015c820 2024-04-24T08:16:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.874+0000 7f7015400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f70100b4be0 con 0x7f70100ce250 2024-04-24T08:16:25.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.874+0000 7f7015400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f70100b42f0 con 0x7f70100b3e90 2024-04-24T08:16:25.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.874+0000 7f7005e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100b3e90 0x7f70100b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.874+0000 7f700f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f70100ce250 0x7f70100ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.874+0000 7f700e800640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f701015c820 0x7f701006f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:39506/0 (socket says 172.21.15.69:39506) 2024-04-24T08:16:25.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.874+0000 7f700e800640 1 -- 172.21.15.69:0/496274029 learned_addr learned my addr 172.21.15.69:0/496274029 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:16:25.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.874+0000 7f7005e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100b3e90 0x7f70100b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:38580/0 (socket says 172.21.15.69:38580) 2024-04-24T08:16:25.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.875+0000 7f700de00640 1 -- 172.21.15.69:0/496274029 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1847241513 0 0) 0x7f701015cc00 con 0x7f701015c820 2024-04-24T08:16:25.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.875+0000 7f7005e00640 1 -- 172.21.15.69:0/496274029 >> v2:172.21.15.121:3300/0 conn(0x7f70100ce250 msgr2=0x7f70100ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.875+0000 7f7005e00640 1 --2- 172.21.15.69:0/496274029 >> v2:172.21.15.121:3300/0 conn(0x7f70100ce250 0x7f70100ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.875+0000 7f7005e00640 1 -- 172.21.15.69:0/496274029 >> v1:172.21.15.121:6789/0 conn(0x7f701015c820 legacy=0x7f701006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.875+0000 7f7005e00640 1 -- 172.21.15.69:0/496274029 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701006ed30 con 0x7f70100b3e90 2024-04-24T08:16:25.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.877+0000 7f7005e00640 1 --2- 172.21.15.69:0/496274029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100b3e90 0x7f70100b46a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80097a0 tx=0x7f6ff802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3e52437e1c837f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.877+0000 7f700de00640 1 -- 172.21.15.69:0/496274029 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8037070 con 0x7f70100b3e90 2024-04-24T08:16:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.877+0000 7f700de00640 1 -- 172.21.15.69:0/496274029 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ff8032440 con 0x7f70100b3e90 2024-04-24T08:16:25.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.877+0000 7f700de00640 1 -- 172.21.15.69:0/496274029 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8032810 con 0x7f70100b3e90 2024-04-24T08:16:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.878+0000 7f7015400640 1 -- 172.21.15.69:0/496274029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100b3e90 msgr2=0x7f70100b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.878+0000 7f7015400640 1 --2- 172.21.15.69:0/496274029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100b3e90 0x7f70100b46a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6ff80097a0 tx=0x7f6ff802a180 comp rx=0 tx=0).stop 2024-04-24T08:16:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.878+0000 7f7015400640 1 -- 172.21.15.69:0/496274029 shutdown_connections 2024-04-24T08:16:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.878+0000 7f7015400640 1 --2- 172.21.15.69:0/496274029 >> v2:172.21.15.121:3300/0 conn(0x7f70100ce250 0x7f70100ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.878+0000 7f7015400640 1 --2- 172.21.15.69:0/496274029 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100b3e90 0x7f70100b46a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.878+0000 7f7015400640 1 -- 172.21.15.69:0/496274029 >> 172.21.15.69:0/496274029 conn(0x7f70100d0350 msgr2=0x7f70100d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.878+0000 7f7015400640 1 -- 172.21.15.69:0/496274029 shutdown_connections 2024-04-24T08:16:25.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.878+0000 7f7015400640 1 -- 172.21.15.69:0/496274029 wait complete. 2024-04-24T08:16:25.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.880+0000 7f7015400640 1 Processor -- start 2024-04-24T08:16:25.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.880+0000 7f7015400640 1 -- start start 2024-04-24T08:16:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.881+0000 7f7015400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70100b3e90 0x7f7010141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.881+0000 7f7015400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100ce250 0x7f701015f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.881+0000 7f7015400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f701015c820 0x7f70101f1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.881+0000 7f7015400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f701006efd0 con 0x7f701015c820 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.881+0000 7f7015400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f70100b50f0 con 0x7f70100b3e90 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.881+0000 7f7015400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f701006f2b0 con 0x7f70100ce250 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f700f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f701015c820 0x7f70101f1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f700f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f701015c820 0x7f70101f1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40598/0 (socket says 172.21.15.69:40598) 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f700f200640 1 -- 172.21.15.69:0/3916939191 learned_addr learned my addr 172.21.15.69:0/3916939191 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f700e800640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70100b3e90 0x7f7010141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f7005e00640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100ce250 0x7f701015f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f700f200640 1 -- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100ce250 msgr2=0x7f701015f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f700f200640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100ce250 0x7f701015f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f700f200640 1 -- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70100b3e90 msgr2=0x7f7010141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f700f200640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70100b3e90 0x7f7010141260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f700f200640 1 -- 172.21.15.69:0/3916939191 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701015fc50 con 0x7f701015c820 2024-04-24T08:16:25.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.882+0000 7f700f200640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f701015c820 0x7f70101f1830 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f700800efc0 tx=0x7f700800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.883+0000 7f7007400640 1 -- 172.21.15.69:0/3916939191 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008018020 con 0x7f701015c820 2024-04-24T08:16:25.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.883+0000 7f7007400640 1 -- 172.21.15.69:0/3916939191 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7008014480 con 0x7f701015c820 2024-04-24T08:16:25.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.883+0000 7f7007400640 1 -- 172.21.15.69:0/3916939191 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008014870 con 0x7f701015c820 2024-04-24T08:16:25.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.883+0000 7f7015400640 1 -- 172.21.15.69:0/3916939191 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f701015cea0 con 0x7f701015c820 2024-04-24T08:16:25.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.883+0000 7f7015400640 1 -- 172.21.15.69:0/3916939191 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f701015d460 con 0x7f701015c820 2024-04-24T08:16:25.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.885+0000 7f7007400640 1 -- 172.21.15.69:0/3916939191 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7008005d10 con 0x7f701015c820 2024-04-24T08:16:25.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.886+0000 7f7007400640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fec06c8f0 0x7f6fec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.886+0000 7f7007400640 1 -- 172.21.15.69:0/3916939191 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f700808c6b0 con 0x7f701015c820 2024-04-24T08:16:25.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.887+0000 7f701820f740 1 Processor -- start 2024-04-24T08:16:25.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.887+0000 7f701820f740 1 -- start start 2024-04-24T08:16:25.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.887+0000 7f700e800640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fec06c8f0 0x7f6fec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f701820f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c968c30 0x55e77c91b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f701820f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c94bd50 0x55e77c94b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f701820f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94b610 0x55e77c920420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f701820f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e77c8851f0 con 0x55e77c94b610 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f701820f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e77c82b350 con 0x55e77c94bd50 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f701820f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55e77c8811b0 con 0x55e77c968c30 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f700f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94b610 0x55e77c920420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f700f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94b610 0x55e77c920420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40612/0 (socket says 172.21.15.69:40612) 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f700f200640 1 -- 172.21.15.69:0/1935249171 learned_addr learned my addr 172.21.15.69:0/1935249171 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f700e800640 1 --2- 172.21.15.69:0/1935249171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c968c30 0x55e77c91b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f700f200640 1 -- 172.21.15.69:0/1935249171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c968c30 msgr2=0x55e77c91b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f700f200640 1 --2- 172.21.15.69:0/1935249171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c968c30 0x55e77c91b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f700f200640 1 -- 172.21.15.69:0/1935249171 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c94bd50 msgr2=0x55e77c94b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f700f200640 1 --2- 172.21.15.69:0/1935249171 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c94bd50 0x55e77c94b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f700e800640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fec06c8f0 0x7f6fec06edb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7000000f60 tx=0x7f7000007d20 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.888+0000 7f700f200640 1 -- 172.21.15.69:0/1935249171 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e77c87ff50 con 0x55e77c94b610 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.889+0000 7f700f200640 1 --2- 172.21.15.69:0/1935249171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94b610 0x55e77c920420 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f700805a310 tx=0x7f7008059e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189b02226ee6a05d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.889+0000 7f6fff000640 1 -- 172.21.15.69:0/1935249171 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008072020 con 0x55e77c94b610 2024-04-24T08:16:25.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.889+0000 7f6fff000640 1 -- 172.21.15.69:0/1935249171 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7008062f60 con 0x55e77c94b610 2024-04-24T08:16:25.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.889+0000 7f6fff000640 1 -- 172.21.15.69:0/1935249171 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70080569f0 con 0x55e77c94b610 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.890+0000 7f701820f740 1 -- 172.21.15.69:0/1935249171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94b610 msgr2=0x55e77c920420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.890+0000 7f701820f740 1 --2- 172.21.15.69:0/1935249171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94b610 0x55e77c920420 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f700805a310 tx=0x7f7008059e10 comp rx=0 tx=0).stop 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.890+0000 7f701820f740 1 -- 172.21.15.69:0/1935249171 shutdown_connections 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.890+0000 7f701820f740 1 --2- 172.21.15.69:0/1935249171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94b610 0x55e77c920420 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.890+0000 7f701820f740 1 --2- 172.21.15.69:0/1935249171 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c94bd50 0x55e77c94b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.890+0000 7f701820f740 1 --2- 172.21.15.69:0/1935249171 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c968c30 0x55e77c91b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.890+0000 7f701820f740 1 -- 172.21.15.69:0/1935249171 >> 172.21.15.69:0/1935249171 conn(0x55e77c91d830 msgr2=0x55e77c9660d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.890+0000 7f701820f740 1 -- 172.21.15.69:0/1935249171 shutdown_connections 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.890+0000 7f701820f740 1 -- 172.21.15.69:0/1935249171 wait complete. 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.891+0000 7f701820f740 1 -- start start 2024-04-24T08:16:25.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.891+0000 7f701820f740 1 Processor -- start 2024-04-24T08:16:25.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f701820f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c94bd50 0x55e77c97a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f701820f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c968c30 0x55e77c946330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f701820f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c962d70 0x55e77c9207a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f701820f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e77c91f9e0 con 0x55e77c962d70 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f701820f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e77c83e690 con 0x55e77c968c30 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f701820f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55e77c91edd0 con 0x55e77c94bd50 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f701820f740 10 client.0 _ll_register_callbacks cb 0x55e77c87a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f700f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c962d70 0x55e77c9207a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f700f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c962d70 0x55e77c9207a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40620/0 (socket says 172.21.15.69:40620) 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f700f200640 1 -- 172.21.15.69:0/2392962488 learned_addr learned my addr 172.21.15.69:0/2392962488 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f700f200640 1 -- 172.21.15.69:0/2392962488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c94bd50 msgr2=0x55e77c97a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f700f200640 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c94bd50 0x55e77c97a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.892+0000 7f700f200640 1 -- 172.21.15.69:0/2392962488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c968c30 msgr2=0x55e77c946330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f7005e00640 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c968c30 0x55e77c946330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f700f200640 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c968c30 0x55e77c946330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f700f200640 1 -- 172.21.15.69:0/2392962488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e77c83d920 con 0x55e77c962d70 2024-04-24T08:16:25.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f700f200640 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c962d70 0x55e77c9207a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7008062a30 tx=0x7f700801f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f20b31f3b1e4dc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f6ff3400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f6ff3400640 1 -- 172.21.15.69:0/2392962488 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008072040 con 0x55e77c962d70 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f6ff3400640 1 -- 172.21.15.69:0/2392962488 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f7008072d60 con 0x55e77c962d70 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f701820f740 1 -- 172.21.15.69:0/2392962488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7008062f60 con 0x55e77c962d70 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f6ff3400640 1 -- 172.21.15.69:0/2392962488 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f70080597d0 con 0x55e77c962d70 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f701820f740 10 client.8019 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f701820f740 1 -- 172.21.15.69:0/2392962488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55e77c6617d0 con 0x55e77c962d70 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.893+0000 7f6ff3400640 1 -- 172.21.15.69:0/2392962488 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7008062f60 con 0x55e77c962d70 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.894+0000 7f6ff3400640 1 -- 172.21.15.69:0/2392962488 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f7008059970 con 0x55e77c962d70 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.894+0000 7f701820f740 10 client.8019 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.894+0000 7f701820f740 10 client.8019 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.894+0000 7f701820f740 20 client.8019 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.894+0000 7f701820f740 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55e77c966a30 0x55e77c94c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.894+0000 7f701820f740 4 client.8019 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:25.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.894+0000 7f701820f740 1 -- 172.21.15.69:0/2392962488 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e77c892c10 con 0x55e77c966a30 2024-04-24T08:16:25.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.894+0000 7f700e800640 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55e77c966a30 0x55e77c94c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.895+0000 7f700e800640 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55e77c966a30 0x55e77c94c240 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c33b79635e52185d server_cookie=62d88a4be15e49f5 in_seq=0 out_seq=0 2024-04-24T08:16:25.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.895+0000 7f6ff3400640 10 client.8019 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:25.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.895+0000 7f6ff3400640 1 -- 172.21.15.69:0/2392962488 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e77c892c10 con 0x55e77c966a30 2024-04-24T08:16:25.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.895+0000 7f6ff3400640 10 client.8019 handle_command_reply: tid=0 2024-04-24T08:16:25.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.895+0000 7f701820f740 1 client.8019 shutdown 2024-04-24T08:16:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 1 -- 172.21.15.69:0/2392962488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c962d70 msgr2=0x55e77c9207a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c962d70 0x55e77c9207a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f7008062a30 tx=0x7f700801f040 comp rx=0 tx=0).stop 2024-04-24T08:16:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 1 -- 172.21.15.69:0/2392962488 shutdown_connections 2024-04-24T08:16:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55e77c966a30 0x55e77c94c240 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c962d70 0x55e77c9207a0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c968c30 0x55e77c946330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 1 --2- 172.21.15.69:0/2392962488 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c94bd50 0x55e77c97a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f6ff3400640 0 client.8019 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 1 -- 172.21.15.69:0/2392962488 >> 172.21.15.69:0/2392962488 conn(0x55e77c94b610 msgr2=0x55e77c91fe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 1 -- 172.21.15.69:0/2392962488 shutdown_connections 2024-04-24T08:16:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 1 -- 172.21.15.69:0/2392962488 wait complete. 2024-04-24T08:16:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.896+0000 7f701820f740 20 client.8019 trim_cache size 0 max 16384 2024-04-24T08:16:25.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.915+0000 7f701820f740 1 Processor -- start 2024-04-24T08:16:25.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.916+0000 7f701820f740 1 -- start start 2024-04-24T08:16:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.916+0000 7f701820f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c94bd50 0x55e77c9748f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.916+0000 7f701820f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c962d70 0x55e77c91e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.916+0000 7f701820f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c968c30 0x55e77c91fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.916+0000 7f701820f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f70101fcb10 con 0x55e77c968c30 2024-04-24T08:16:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.916+0000 7f701820f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7010073310 con 0x55e77c962d70 2024-04-24T08:16:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.916+0000 7f701820f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f701015e800 con 0x55e77c94bd50 2024-04-24T08:16:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.917+0000 7f700f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c968c30 0x55e77c91fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.917+0000 7f700f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c968c30 0x55e77c91fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40628/0 (socket says 172.21.15.69:40628) 2024-04-24T08:16:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.917+0000 7f700f200640 1 -- 172.21.15.69:0/359475496 learned_addr learned my addr 172.21.15.69:0/359475496 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.917+0000 7f700f200640 1 -- 172.21.15.69:0/359475496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c94bd50 msgr2=0x55e77c9748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.917+0000 7f700f200640 1 --2- 172.21.15.69:0/359475496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c94bd50 0x55e77c9748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.917+0000 7f700f200640 1 -- 172.21.15.69:0/359475496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c962d70 msgr2=0x55e77c91e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.917+0000 7f700f200640 1 --2- 172.21.15.69:0/359475496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c962d70 0x55e77c91e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.917+0000 7f700f200640 1 -- 172.21.15.69:0/359475496 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e77c5973f0 con 0x55e77c968c30 2024-04-24T08:16:25.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.917+0000 7f700f200640 1 --2- 172.21.15.69:0/359475496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c968c30 0x55e77c91fc70 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f700805a310 tx=0x7f700808a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb3b879795ae448 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f6ffe600640 1 -- 172.21.15.69:0/359475496 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008072040 con 0x55e77c968c30 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f6ffe600640 1 -- 172.21.15.69:0/359475496 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7008059970 con 0x55e77c968c30 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f6ffe600640 1 -- 172.21.15.69:0/359475496 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7008072da0 con 0x55e77c968c30 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f701820f740 1 -- 172.21.15.69:0/359475496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c968c30 msgr2=0x55e77c91fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f701820f740 1 --2- 172.21.15.69:0/359475496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c968c30 0x55e77c91fc70 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f700805a310 tx=0x7f700808a000 comp rx=0 tx=0).stop 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f701820f740 1 -- 172.21.15.69:0/359475496 shutdown_connections 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f701820f740 1 --2- 172.21.15.69:0/359475496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c968c30 0x55e77c91fc70 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f701820f740 1 --2- 172.21.15.69:0/359475496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c962d70 0x55e77c91e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f701820f740 1 --2- 172.21.15.69:0/359475496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c94bd50 0x55e77c9748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f701820f740 1 -- 172.21.15.69:0/359475496 >> 172.21.15.69:0/359475496 conn(0x55e77c94b610 msgr2=0x55e77c974eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f701820f740 1 -- 172.21.15.69:0/359475496 shutdown_connections 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.918+0000 7f701820f740 1 -- 172.21.15.69:0/359475496 wait complete. 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.919+0000 7f701820f740 1 -- start start 2024-04-24T08:16:25.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.919+0000 7f701820f740 1 Processor -- start 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.920+0000 7f701820f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94bd50 0x55e77c962340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.920+0000 7f701820f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c962d70 0x55e77c9e80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.920+0000 7f701820f740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c968c30 0x55e77c95ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.920+0000 7f701820f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e77c94e370 con 0x55e77c94bd50 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.920+0000 7f701820f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7010144070 con 0x55e77c962d70 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.920+0000 7f701820f740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55e77c95c780 con 0x55e77c968c30 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.920+0000 7f701820f740 10 client.0 _ll_register_callbacks cb 0x7f7008059ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.920+0000 7f700e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94bd50 0x55e77c962340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.920+0000 7f700e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94bd50 0x55e77c962340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40632/0 (socket says 172.21.15.69:40632) 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.920+0000 7f700e800640 1 -- 172.21.15.69:0/473936881 learned_addr learned my addr 172.21.15.69:0/473936881 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:25.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f700e800640 1 -- 172.21.15.69:0/473936881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c968c30 msgr2=0x55e77c95ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f700e800640 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c968c30 0x55e77c95ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f7005e00640 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c962d70 0x55e77c9e80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f700e800640 1 -- 172.21.15.69:0/473936881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c962d70 msgr2=0x55e77c9e80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f700e800640 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c962d70 0x55e77c9e80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f700e800640 1 -- 172.21.15.69:0/473936881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e77c5570f0 con 0x55e77c94bd50 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f700e800640 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94bd50 0x55e77c962340 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f70100afac0 tx=0x7f7000000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf08e04463574628 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f6ff3400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f6ff3400640 1 -- 172.21.15.69:0/473936881 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7000001b40 con 0x55e77c94bd50 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f6ff3400640 1 -- 172.21.15.69:0/473936881 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f7000001ce0 con 0x55e77c94bd50 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.921+0000 7f701820f740 1 -- 172.21.15.69:0/473936881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7008059970 con 0x55e77c94bd50 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f6ff3400640 1 -- 172.21.15.69:0/473936881 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f7000032050 con 0x55e77c94bd50 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f701820f740 10 client.8025 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f701820f740 1 -- 172.21.15.69:0/473936881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55e77c84a810 con 0x55e77c94bd50 2024-04-24T08:16:25.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f6ff3400640 1 -- 172.21.15.69:0/473936881 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7008059970 con 0x55e77c94bd50 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f6ff3400640 1 -- 172.21.15.69:0/473936881 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f7000035020 con 0x55e77c94bd50 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f701820f740 10 client.8025 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f701820f740 10 client.8025 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f701820f740 20 client.8025 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f701820f740 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55e77c94c240 0x55e77c9694f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f701820f740 4 client.8025 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f701820f740 1 -- 172.21.15.69:0/473936881 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e77c88ca80 con 0x55e77c94c240 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.922+0000 7f7005e00640 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55e77c94c240 0x55e77c9694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f7005e00640 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55e77c94c240 0x55e77c9694f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2cd8d2976a711798 server_cookie=eeebe2e1c54d028a in_seq=0 out_seq=0 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f6ff3400640 10 client.8025 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f6ff3400640 1 -- 172.21.15.69:0/473936881 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+263 (crc 0 0 0) 0x55e77c88ca80 con 0x55e77c94c240 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f6ff3400640 10 client.8025 handle_command_reply: tid=0 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f701820f740 1 client.8025 shutdown 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f701820f740 1 -- 172.21.15.69:0/473936881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94bd50 msgr2=0x55e77c962340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f701820f740 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94bd50 0x55e77c962340 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f70100afac0 tx=0x7f7000000ed0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f701820f740 1 -- 172.21.15.69:0/473936881 shutdown_connections 2024-04-24T08:16:25.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f701820f740 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55e77c94c240 0x55e77c9694f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f701820f740 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e77c968c30 0x55e77c95ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f701820f740 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e77c962d70 0x55e77c9e80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f701820f740 1 --2- 172.21.15.69:0/473936881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e77c94bd50 0x55e77c962340 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f701820f740 1 -- 172.21.15.69:0/473936881 >> 172.21.15.69:0/473936881 conn(0x55e77c94b610 msgr2=0x55e77c97a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.923+0000 7f6ff3400640 0 client.8025 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:25.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.924+0000 7f701820f740 1 -- 172.21.15.69:0/473936881 shutdown_connections 2024-04-24T08:16:25.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.924+0000 7f701820f740 1 -- 172.21.15.69:0/473936881 wait complete. 2024-04-24T08:16:25.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.924+0000 7f701820f740 20 client.8025 trim_cache size 0 max 16384 2024-04-24T08:16:25.929 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:16:25.929 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:16:25.929 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-04-24T08:16:25.929 INFO:teuthology.orchestra.run.smithi069.stdout: "c17ba41f-461a-4b1d-aa97-d779ca376add": { 2024-04-24T08:16:25.929 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/dir", 2024-04-24T08:16:25.929 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "c17ba41f-461a-4b1d-aa97-d779ca376add", 2024-04-24T08:16:25.929 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,repair" 2024-04-24T08:16:25.929 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:16:25.929 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:16:25.929 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:16:25.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.924+0000 7f7015400640 1 -- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fec06c8f0 msgr2=0x7f6fec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.924+0000 7f7015400640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fec06c8f0 0x7f6fec06edb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f7000000f60 tx=0x7f7000007d20 comp rx=0 tx=0).stop 2024-04-24T08:16:25.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.924+0000 7f7015400640 1 -- 172.21.15.69:0/3916939191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f701015c820 msgr2=0x7f70101f1830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:25.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.924+0000 7f7015400640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f701015c820 0x7f70101f1830 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f700800efc0 tx=0x7f700800c490 comp rx=0 tx=0).stop 2024-04-24T08:16:25.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.925+0000 7f7015400640 1 -- 172.21.15.69:0/3916939191 shutdown_connections 2024-04-24T08:16:25.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.925+0000 7f7015400640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fec06c8f0 0x7f6fec06edb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.925+0000 7f7015400640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f701015c820 0x7f70101f1830 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.925+0000 7f7015400640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70100ce250 0x7f701015f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.925+0000 7f7015400640 1 --2- 172.21.15.69:0/3916939191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70100b3e90 0x7f7010141260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:25.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.925+0000 7f7015400640 1 -- 172.21.15.69:0/3916939191 >> 172.21.15.69:0/3916939191 conn(0x7f70100d0350 msgr2=0x7f701006f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:25.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.925+0000 7f7015400640 1 -- 172.21.15.69:0/3916939191 shutdown_connections 2024-04-24T08:16:25.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:25.925+0000 7f7015400640 1 -- 172.21.15.69:0/3916939191 wait complete. 2024-04-24T08:16:25.943 INFO:tasks.cephfs.filesystem:scrub status for tag:c17ba41f-461a-4b1d-aa97-d779ca376add - {'path': '/dir', 'tag': 'c17ba41f-461a-4b1d-aa97-d779ca376add', 'options': 'recursive,repair'} 2024-04-24T08:16:45.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:16:45.584 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:16:55.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:16:55.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-24T08:16:56.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.157+0000 7f019d400640 1 Processor -- start 2024-04-24T08:16:56.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.157+0000 7f019d400640 1 -- start start 2024-04-24T08:16:56.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.159+0000 7f019d400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f01980cbb10 0x7f01980cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.159+0000 7f019d400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01980cbf80 con 0x7f01980d0850 2024-04-24T08:16:56.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.159+0000 7f019d400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f019806bb50 con 0x7f01980b07a0 2024-04-24T08:16:56.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.159+0000 7f019d400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f01980cdc00 con 0x7f01980cbb10 2024-04-24T08:16:56.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.159+0000 7f0196800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f01980d0850 0x7f01980d0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57956/0 (socket says 172.21.15.69:57956) 2024-04-24T08:16:56.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.159+0000 7f0196800640 1 -- 172.21.15.69:0/403297424 learned_addr learned my addr 172.21.15.69:0/403297424 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:16:56.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.159+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500928328 0 0) 0x7f01980cbf80 con 0x7f01980d0850 2024-04-24T08:16:56.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.159+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f017c003520 con 0x7f01980d0850 2024-04-24T08:16:56.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.159+0000 7f0197200640 1 --2- 172.21.15.69:0/403297424 >> v2:172.21.15.121:3300/0 conn(0x7f01980cbb10 0x7f01980cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4106584958 0 0) 0x7f017c003520 con 0x7f01980d0850 2024-04-24T08:16:56.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 >> v2:172.21.15.121:3300/0 conn(0x7f01980cbb10 msgr2=0x7f01980cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 --2- 172.21.15.69:0/403297424 >> v2:172.21.15.121:3300/0 conn(0x7f01980cbb10 0x7f01980cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 >> v1:172.21.15.121:6789/0 conn(0x7f01980b07a0 legacy=0x7f01980ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019806be30 con 0x7f01980d0850 2024-04-24T08:16:56.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f0188001670 con 0x7f01980d0850 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 >> v1:172.21.15.69:6789/0 conn(0x7f01980d0850 legacy=0x7f01980d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f017c003a00 0x7f017c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f017c004800 0x7f017c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f017c0052e0 0x7f017c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01980cdc00 con 0x7f017c0052e0 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f017c00c1e0 con 0x7f017c004800 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.160+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f017c00c3e0 con 0x7f017c003a00 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.161+0000 7f0197200640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f017c0052e0 0x7f017c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.161+0000 7f0195e00640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f017c004800 0x7f017c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.161+0000 7f0196800640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f017c003a00 0x7f017c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.161+0000 7f0197200640 1 -- 172.21.15.69:0/403297424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f017c003a00 msgr2=0x7f017c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.161+0000 7f0197200640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f017c003a00 0x7f017c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.161+0000 7f0197200640 1 -- 172.21.15.69:0/403297424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f017c004800 msgr2=0x7f017c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.161+0000 7f0197200640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f017c004800 0x7f017c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.161+0000 7f0197200640 1 -- 172.21.15.69:0/403297424 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017c003520 con 0x7f017c0052e0 2024-04-24T08:16:56.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.161+0000 7f0196800640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f017c003a00 0x7f017c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:56.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.162+0000 7f0197200640 1 -- 172.21.15.69:0/403297424 reap_dead start 2024-04-24T08:16:56.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.162+0000 7f0197200640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f017c0052e0 0x7f017c00b9c0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0190009fc0 tx=0x7f017c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f956a19867bf917a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.162+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190035020 con 0x7f017c0052e0 2024-04-24T08:16:56.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.162+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0190003bd0 con 0x7f017c0052e0 2024-04-24T08:16:56.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.162+0000 7f0195400640 1 -- 172.21.15.69:0/403297424 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190009c10 con 0x7f017c0052e0 2024-04-24T08:16:56.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.163+0000 7f019d400640 1 -- 172.21.15.69:0/403297424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f017c0052e0 msgr2=0x7f017c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.163+0000 7f019d400640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f017c0052e0 0x7f017c00b9c0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0190009fc0 tx=0x7f017c005830 comp rx=0 tx=0).stop 2024-04-24T08:16:56.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.163+0000 7f019d400640 1 -- 172.21.15.69:0/403297424 shutdown_connections 2024-04-24T08:16:56.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.163+0000 7f019d400640 1 --2- 172.21.15.69:0/403297424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f017c0052e0 0x7f017c00b9c0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.163+0000 7f019d400640 1 -- 172.21.15.69:0/403297424 >> 172.21.15.69:0/403297424 conn(0x7f01980b3050 msgr2=0x7f01980b3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.163+0000 7f019d400640 1 -- 172.21.15.69:0/403297424 shutdown_connections 2024-04-24T08:16:56.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.163+0000 7f019d400640 1 -- 172.21.15.69:0/403297424 wait complete. 2024-04-24T08:16:56.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.165+0000 7f019d400640 1 Processor -- start 2024-04-24T08:16:56.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.165+0000 7f019d400640 1 -- start start 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.166+0000 7f019d400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f017c0052e0 0x7f01980ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.166+0000 7f019d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01980b5e40 0x7f01980caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.166+0000 7f019d400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f019814e330 0x7f019814e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.166+0000 7f019d400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f019806c6d0 con 0x7f01980b5e40 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.166+0000 7f019d400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f01980cf320 con 0x7f019814e330 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.166+0000 7f019d400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f019806c3f0 con 0x7f017c0052e0 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.166+0000 7f0195e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01980b5e40 0x7f01980caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.166+0000 7f0195e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01980b5e40 0x7f01980caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43602/0 (socket says 172.21.15.69:43602) 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.166+0000 7f0195e00640 1 -- 172.21.15.69:0/2676498215 learned_addr learned my addr 172.21.15.69:0/2676498215 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f0195e00640 1 -- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f017c0052e0 msgr2=0x7f01980ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f0195e00640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f017c0052e0 0x7f01980ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f0197200640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f019814e330 0x7f019814e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f0195e00640 1 -- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f019814e330 msgr2=0x7f019814e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f0195e00640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f019814e330 0x7f019814e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f0195e00640 1 -- 172.21.15.69:0/2676498215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01980b63e0 con 0x7f01980b5e40 2024-04-24T08:16:56.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f0195e00640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01980b5e40 0x7f01980caca0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f018400e440 tx=0x7f018400e900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f018ea00640 1 -- 172.21.15.69:0/2676498215 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01840052e0 con 0x7f01980b5e40 2024-04-24T08:16:56.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f018ea00640 1 -- 172.21.15.69:0/2676498215 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0184018070 con 0x7f01980b5e40 2024-04-24T08:16:56.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f018ea00640 1 -- 172.21.15.69:0/2676498215 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184003810 con 0x7f01980b5e40 2024-04-24T08:16:56.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f019d400640 1 -- 172.21.15.69:0/2676498215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01980b66c0 con 0x7f01980b5e40 2024-04-24T08:16:56.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.167+0000 7f019d400640 1 -- 172.21.15.69:0/2676498215 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01980b6c80 con 0x7f01980b5e40 2024-04-24T08:16:56.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.170+0000 7f018ea00640 1 -- 172.21.15.69:0/2676498215 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f01840039b0 con 0x7f01980b5e40 2024-04-24T08:16:56.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.170+0000 7f018ea00640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f017806c8f0 0x7f017806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.171+0000 7f018ea00640 1 -- 172.21.15.69:0/2676498215 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f018408b860 con 0x7f01980b5e40 2024-04-24T08:16:56.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.171+0000 7f01a0231740 1 Processor -- start 2024-04-24T08:16:56.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.171+0000 7f0196800640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f017806c8f0 0x7f017806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.171+0000 7f01a0231740 1 -- start start 2024-04-24T08:16:56.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.172+0000 7f01a0231740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9460c30 0x562eb94135c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.172+0000 7f01a0231740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94430d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.172+0000 7f01a0231740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9443610 0x562eb9418420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.172+0000 7f01a0231740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562eb937d1f0 con 0x562eb9443d50 2024-04-24T08:16:56.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.172+0000 7f01a0231740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562eb9323350 con 0x562eb9443610 2024-04-24T08:16:56.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.172+0000 7f01a0231740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x562eb93791b0 con 0x562eb9460c30 2024-04-24T08:16:56.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.172+0000 7f0196800640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f017806c8f0 0x7f017806edb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f01880017e0 tx=0x7f0188009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0195e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0195e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94430d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43616/0 (socket says 172.21.15.69:43616) 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0195e00640 1 -- 172.21.15.69:0/486634216 learned_addr learned my addr 172.21.15.69:0/486634216 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0196800640 1 --2- 172.21.15.69:0/486634216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9460c30 0x562eb94135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0197200640 1 --2- 172.21.15.69:0/486634216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9443610 0x562eb9418420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0195e00640 1 -- 172.21.15.69:0/486634216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9460c30 msgr2=0x562eb94135c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0195e00640 1 --2- 172.21.15.69:0/486634216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9460c30 0x562eb94135c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0195e00640 1 -- 172.21.15.69:0/486634216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9443610 msgr2=0x562eb9418420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0195e00640 1 --2- 172.21.15.69:0/486634216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9443610 0x562eb9418420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0195e00640 1 -- 172.21.15.69:0/486634216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562eb9377f50 con 0x562eb9443d50 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0195e00640 1 --2- 172.21.15.69:0/486634216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94430d0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0184089c10 tx=0x7f01840055c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff1b8d4cd75c52c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.173+0000 7f0183000640 1 -- 172.21.15.69:0/486634216 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f018405d850 con 0x562eb9443d50 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f0183000640 1 -- 172.21.15.69:0/486634216 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0184062080 con 0x562eb9443d50 2024-04-24T08:16:56.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f0183000640 1 -- 172.21.15.69:0/486634216 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184010de0 con 0x562eb9443d50 2024-04-24T08:16:56.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f01a0231740 1 -- 172.21.15.69:0/486634216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 msgr2=0x562eb94430d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f01a0231740 1 --2- 172.21.15.69:0/486634216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94430d0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0184089c10 tx=0x7f01840055c0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f01a0231740 1 -- 172.21.15.69:0/486634216 shutdown_connections 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f01a0231740 1 --2- 172.21.15.69:0/486634216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94430d0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f01a0231740 1 --2- 172.21.15.69:0/486634216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9443610 0x562eb9418420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f01a0231740 1 --2- 172.21.15.69:0/486634216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9460c30 0x562eb94135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f01a0231740 1 -- 172.21.15.69:0/486634216 >> 172.21.15.69:0/486634216 conn(0x562eb9415830 msgr2=0x562eb945e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f01a0231740 1 -- 172.21.15.69:0/486634216 shutdown_connections 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.174+0000 7f01a0231740 1 -- 172.21.15.69:0/486634216 wait complete. 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.175+0000 7f01a0231740 1 -- start start 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.175+0000 7f01a0231740 1 Processor -- start 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.176+0000 7f01a0231740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.176+0000 7f01a0231740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9460c30 0x562eb9430a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.176+0000 7f01a0231740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9454d90 0x562eb945fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.176+0000 7f01a0231740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562eb94179e0 con 0x562eb9443d50 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.176+0000 7f01a0231740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562eb9336690 con 0x562eb9460c30 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.176+0000 7f01a0231740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x562eb9416dd0 con 0x562eb9454d90 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f01a0231740 10 client.0 _ll_register_callbacks cb 0x562eb9372650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0196800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0196800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94469d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43632/0 (socket says 172.21.15.69:43632) 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0196800640 1 -- 172.21.15.69:0/764052362 learned_addr learned my addr 172.21.15.69:0/764052362 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0197200640 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9454d90 0x562eb945fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0196800640 1 -- 172.21.15.69:0/764052362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9454d90 msgr2=0x562eb945fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0195e00640 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9460c30 0x562eb9430a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0196800640 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9454d90 0x562eb945fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0196800640 1 -- 172.21.15.69:0/764052362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9460c30 msgr2=0x562eb9430a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0196800640 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9460c30 0x562eb9430a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0196800640 1 -- 172.21.15.69:0/764052362 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562eb9335920 con 0x562eb9443d50 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0196800640 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94469d0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f0188005b00 tx=0x7f0188001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d896e3b65227a4e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.177+0000 7f0177400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:56.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.178+0000 7f0177400640 1 -- 172.21.15.69:0/764052362 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188026060 con 0x562eb9443d50 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.178+0000 7f0177400640 1 -- 172.21.15.69:0/764052362 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f0188021b00 con 0x562eb9443d50 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.178+0000 7f01a0231740 1 -- 172.21.15.69:0/764052362 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0184062080 con 0x562eb9443d50 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.178+0000 7f0177400640 1 -- 172.21.15.69:0/764052362 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f0188021d00 con 0x562eb9443d50 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.178+0000 7f01a0231740 10 client.8037 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.178+0000 7f01a0231740 1 -- 172.21.15.69:0/764052362 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x562eb91597d0 con 0x562eb9443d50 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.178+0000 7f0177400640 1 -- 172.21.15.69:0/764052362 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0184062080 con 0x562eb9443d50 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.178+0000 7f0177400640 1 -- 172.21.15.69:0/764052362 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f0188012620 con 0x562eb9443d50 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.179+0000 7f01a0231740 10 client.8037 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.179+0000 7f01a0231740 10 client.8037 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.179+0000 7f01a0231740 20 client.8037 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.179+0000 7f01a0231740 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x562eb9456580 0x562eb94569f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.179+0000 7f01a0231740 4 client.8037 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:56.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.179+0000 7f01a0231740 1 -- 172.21.15.69:0/764052362 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562eb938ac10 con 0x562eb9456580 2024-04-24T08:16:56.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.179+0000 7f0195e00640 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x562eb9456580 0x562eb94569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.179+0000 7f0195e00640 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x562eb9456580 0x562eb94569f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a2f246a51437335 server_cookie=ba6b224535844500 in_seq=0 out_seq=0 2024-04-24T08:16:56.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.179+0000 7f0177400640 10 client.8037 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.180+0000 7f0177400640 1 -- 172.21.15.69:0/764052362 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562eb938ac10 con 0x562eb9456580 2024-04-24T08:16:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.180+0000 7f0177400640 10 client.8037 handle_command_reply: tid=0 2024-04-24T08:16:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.180+0000 7f01a0231740 1 client.8037 shutdown 2024-04-24T08:16:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.180+0000 7f01a0231740 1 -- 172.21.15.69:0/764052362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 msgr2=0x562eb94469d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.180+0000 7f01a0231740 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94469d0 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f0188005b00 tx=0x7f0188001020 comp rx=0 tx=0).stop 2024-04-24T08:16:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.180+0000 7f01a0231740 1 -- 172.21.15.69:0/764052362 shutdown_connections 2024-04-24T08:16:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.180+0000 7f01a0231740 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x562eb9456580 0x562eb94569f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.181+0000 7f01a0231740 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9454d90 0x562eb945fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.181+0000 7f01a0231740 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9460c30 0x562eb9430a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.181+0000 7f0177400640 0 client.8037 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:56.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.181+0000 7f01a0231740 1 --2- 172.21.15.69:0/764052362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9443d50 0x562eb94469d0 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.181+0000 7f01a0231740 1 -- 172.21.15.69:0/764052362 >> 172.21.15.69:0/764052362 conn(0x562eb9443610 msgr2=0x562eb94614f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.181+0000 7f01a0231740 1 -- 172.21.15.69:0/764052362 shutdown_connections 2024-04-24T08:16:56.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.181+0000 7f01a0231740 1 -- 172.21.15.69:0/764052362 wait complete. 2024-04-24T08:16:56.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.181+0000 7f01a0231740 20 client.8037 trim_cache size 0 max 16384 2024-04-24T08:16:56.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.200+0000 7f01a0231740 1 Processor -- start 2024-04-24T08:16:56.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.200+0000 7f01a0231740 1 -- start start 2024-04-24T08:16:56.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.201+0000 7f01a0231740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9443d50 0x562eb9442f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.201+0000 7f01a0231740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9454d90 0x562eb9430a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.201+0000 7f01a0231740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9460c30 0x562eb94616e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.201+0000 7f01a0231740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01982017b0 con 0x562eb9454d90 2024-04-24T08:16:56.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.201+0000 7f01a0231740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f01982008b0 con 0x562eb9460c30 2024-04-24T08:16:56.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.201+0000 7f01a0231740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f01980c6280 con 0x562eb9443d50 2024-04-24T08:16:56.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.201+0000 7f0195e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9454d90 0x562eb9430a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.201+0000 7f0195e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9454d90 0x562eb9430a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43642/0 (socket says 172.21.15.69:43642) 2024-04-24T08:16:56.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.201+0000 7f0195e00640 1 -- 172.21.15.69:0/1255014470 learned_addr learned my addr 172.21.15.69:0/1255014470 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.202+0000 7f0195e00640 1 -- 172.21.15.69:0/1255014470 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9443d50 msgr2=0x562eb9442f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.202+0000 7f0197200640 1 --2- 172.21.15.69:0/1255014470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9460c30 0x562eb94616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.202+0000 7f0195e00640 1 --2- 172.21.15.69:0/1255014470 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9443d50 0x562eb9442f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.202+0000 7f0195e00640 1 -- 172.21.15.69:0/1255014470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9460c30 msgr2=0x562eb94616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.202+0000 7f0195e00640 1 --2- 172.21.15.69:0/1255014470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9460c30 0x562eb94616e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.202+0000 7f0195e00640 1 -- 172.21.15.69:0/1255014470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562eb9416970 con 0x562eb9454d90 2024-04-24T08:16:56.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.202+0000 7f0195e00640 1 --2- 172.21.15.69:0/1255014470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9454d90 0x562eb9430a50 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f0184059660 tx=0x7f018405b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19623da29c1ae5d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.202+0000 7f0182600640 1 -- 172.21.15.69:0/1255014470 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184077070 con 0x562eb9454d90 2024-04-24T08:16:56.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.202+0000 7f0182600640 1 -- 172.21.15.69:0/1255014470 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0184062080 con 0x562eb9454d90 2024-04-24T08:16:56.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.202+0000 7f0182600640 1 -- 172.21.15.69:0/1255014470 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0184055740 con 0x562eb9454d90 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.203+0000 7f01a0231740 1 -- 172.21.15.69:0/1255014470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9454d90 msgr2=0x562eb9430a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.203+0000 7f01a0231740 1 --2- 172.21.15.69:0/1255014470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9454d90 0x562eb9430a50 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f0184059660 tx=0x7f018405b000 comp rx=0 tx=0).stop 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.203+0000 7f01a0231740 1 -- 172.21.15.69:0/1255014470 shutdown_connections 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.203+0000 7f01a0231740 1 --2- 172.21.15.69:0/1255014470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9454d90 0x562eb9430a50 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.203+0000 7f01a0231740 1 --2- 172.21.15.69:0/1255014470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9460c30 0x562eb94616e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.203+0000 7f01a0231740 1 --2- 172.21.15.69:0/1255014470 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9443d50 0x562eb9442f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.203+0000 7f01a0231740 1 -- 172.21.15.69:0/1255014470 >> 172.21.15.69:0/1255014470 conn(0x562eb9443610 msgr2=0x562eb9416fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.203+0000 7f01a0231740 1 -- 172.21.15.69:0/1255014470 shutdown_connections 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.203+0000 7f01a0231740 1 -- 172.21.15.69:0/1255014470 wait complete. 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.204+0000 7f01a0231740 1 -- start start 2024-04-24T08:16:56.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.204+0000 7f01a0231740 1 Processor -- start 2024-04-24T08:16:56.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f01a0231740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9443d50 0x562eb9453a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f01a0231740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9454d90 0x562eb9453fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f01a0231740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9460c30 0x562eb942ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f01a0231740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562eb9481dd0 con 0x562eb9460c30 2024-04-24T08:16:56.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f01a0231740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0198200260 con 0x562eb9454d90 2024-04-24T08:16:56.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f01a0231740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x562eb9353840 con 0x562eb9443d50 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f01a0231740 10 client.0 _ll_register_callbacks cb 0x7f0188007e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f0197200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9460c30 0x562eb942ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f0197200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9460c30 0x562eb942ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43654/0 (socket says 172.21.15.69:43654) 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f0197200640 1 -- 172.21.15.69:0/3071217154 learned_addr learned my addr 172.21.15.69:0/3071217154 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f0196800640 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9443d50 0x562eb9453a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.205+0000 7f0195e00640 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9454d90 0x562eb9453fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0197200640 1 -- 172.21.15.69:0/3071217154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9443d50 msgr2=0x562eb9453a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0197200640 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9443d50 0x562eb9453a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0197200640 1 -- 172.21.15.69:0/3071217154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9454d90 msgr2=0x562eb9453fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0197200640 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9454d90 0x562eb9453fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0197200640 1 -- 172.21.15.69:0/3071217154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562eb908f3f0 con 0x562eb9460c30 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0195e00640 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9454d90 0x562eb9453fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0196800640 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9443d50 0x562eb9453a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:56.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0197200640 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9460c30 0x562eb942ef40 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f019000e230 tx=0x7f0190004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ddb1e2501c56f09 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0177400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0177400640 1 -- 172.21.15.69:0/3071217154 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019002d070 con 0x562eb9460c30 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0177400640 1 -- 172.21.15.69:0/3071217154 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f0190047b60 con 0x562eb9460c30 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f01a0231740 1 -- 172.21.15.69:0/3071217154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0184062080 con 0x562eb9460c30 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f0177400640 1 -- 172.21.15.69:0/3071217154 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f019004e050 con 0x562eb9460c30 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f01a0231740 10 client.8043 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.206+0000 7f01a0231740 1 -- 172.21.15.69:0/3071217154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x562eb904f0f0 con 0x562eb9460c30 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.207+0000 7f0177400640 1 -- 172.21.15.69:0/3071217154 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0184062080 con 0x562eb9460c30 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.207+0000 7f0177400640 1 -- 172.21.15.69:0/3071217154 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f0190051020 con 0x562eb9460c30 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.207+0000 7f01a0231740 10 client.8043 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.207+0000 7f01a0231740 10 client.8043 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.207+0000 7f01a0231740 20 client.8043 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:56.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.207+0000 7f01a0231740 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x562eb946d8c0 0x562eb942f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.207+0000 7f01a0231740 4 client.8043 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.207+0000 7f01a0231740 1 -- 172.21.15.69:0/3071217154 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562eb9384a80 con 0x562eb946d8c0 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.207+0000 7f0196800640 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x562eb946d8c0 0x562eb942f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f0196800640 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x562eb946d8c0 0x562eb942f480 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37997aef9c46d40f server_cookie=9f3fcce7ae4e61f5 in_seq=0 out_seq=0 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f0177400640 10 client.8043 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f0177400640 1 -- 172.21.15.69:0/3071217154 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x562eb9384a80 con 0x562eb946d8c0 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f0177400640 10 client.8043 handle_command_reply: tid=0 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 client.8043 shutdown 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 -- 172.21.15.69:0/3071217154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9460c30 msgr2=0x562eb942ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9460c30 0x562eb942ef40 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f019000e230 tx=0x7f0190004ab0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 -- 172.21.15.69:0/3071217154 shutdown_connections 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x562eb946d8c0 0x562eb942f480 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562eb9460c30 0x562eb942ef40 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562eb9454d90 0x562eb9453fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 --2- 172.21.15.69:0/3071217154 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562eb9443d50 0x562eb9453a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f0177400640 0 client.8043 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:56.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 -- 172.21.15.69:0/3071217154 >> 172.21.15.69:0/3071217154 conn(0x562eb9443610 msgr2=0x562eb94616e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 -- 172.21.15.69:0/3071217154 shutdown_connections 2024-04-24T08:16:56.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 1 -- 172.21.15.69:0/3071217154 wait complete. 2024-04-24T08:16:56.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.208+0000 7f01a0231740 20 client.8043 trim_cache size 0 max 16384 2024-04-24T08:16:56.213 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:16:56.213 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-04-24T08:16:56.213 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-04-24T08:16:56.213 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:16:56.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 -- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f017806c8f0 msgr2=0x7f017806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f017806c8f0 0x7f017806edb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f01880017e0 tx=0x7f0188009040 comp rx=0 tx=0).stop 2024-04-24T08:16:56.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 -- 172.21.15.69:0/2676498215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01980b5e40 msgr2=0x7f01980caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01980b5e40 0x7f01980caca0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f018400e440 tx=0x7f018400e900 comp rx=0 tx=0).stop 2024-04-24T08:16:56.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 -- 172.21.15.69:0/2676498215 shutdown_connections 2024-04-24T08:16:56.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f017806c8f0 0x7f017806edb0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01980b5e40 0x7f01980caca0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f019814e330 0x7f019814e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 --2- 172.21.15.69:0/2676498215 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f017c0052e0 0x7f01980ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 -- 172.21.15.69:0/2676498215 >> 172.21.15.69:0/2676498215 conn(0x7f01980b3050 msgr2=0x7f01980b1270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 -- 172.21.15.69:0/2676498215 shutdown_connections 2024-04-24T08:16:56.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.209+0000 7f019d400640 1 -- 172.21.15.69:0/2676498215 wait complete. 2024-04-24T08:16:56.227 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:16:56.227 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:16:56.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.421+0000 7f007d800640 1 Processor -- start 2024-04-24T08:16:56.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.421+0000 7f007d800640 1 -- start start 2024-04-24T08:16:56.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.422+0000 7f007d800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0078156c60 0x7f007815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.422+0000 7f007d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00780b30f0 0x7f007815d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.422+0000 7f007d800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f007815d660 con 0x7f00780ad0d0 2024-04-24T08:16:56.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.422+0000 7f007d800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f007806dc30 con 0x7f0078156c60 2024-04-24T08:16:56.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.422+0000 7f007d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f007806df10 con 0x7f00780b30f0 2024-04-24T08:16:56.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.423+0000 7f007ce00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f00780ad0d0 0x7f00781600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57964/0 (socket says 172.21.15.69:57964) 2024-04-24T08:16:56.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.423+0000 7f007ce00640 1 -- 172.21.15.69:0/418631200 learned_addr learned my addr 172.21.15.69:0/418631200 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.423+0000 7f0077e00640 1 --2- 172.21.15.69:0/418631200 >> v2:172.21.15.121:3300/0 conn(0x7f0078156c60 0x7f007815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.423+0000 7f006fe00640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00780b30f0 0x7f007815d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.423+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724180995 0 0) 0x7f007815d660 con 0x7f00780ad0d0 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.423+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f005c003520 con 0x7f00780ad0d0 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2023840161 0 0) 0x7f005c003520 con 0x7f00780ad0d0 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00780b30f0 msgr2=0x7f007815d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00780b30f0 0x7f007815d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 >> v2:172.21.15.121:3300/0 conn(0x7f0078156c60 msgr2=0x7f007815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 --2- 172.21.15.69:0/418631200 >> v2:172.21.15.121:3300/0 conn(0x7f0078156c60 0x7f007815cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007806e1f0 con 0x7f00780ad0d0 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f00700019f0 con 0x7f00780ad0d0 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 >> v1:172.21.15.69:6789/0 conn(0x7f00780ad0d0 legacy=0x7f00781600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f005c0039b0 0x7f005c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c004800 0x7f005c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00780b30f0 0x7f005c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f007806dc30 con 0x7f005c004800 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f007806df10 con 0x7f005c0039b0 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f005c00b9c0 con 0x7f00780b30f0 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.424+0000 7f006fe00640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c004800 0x7f005c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.425+0000 7f0077e00640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f005c0039b0 0x7f005c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.425+0000 7f007ce00640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00780b30f0 0x7f005c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.425+0000 7f006fe00640 1 -- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00780b30f0 msgr2=0x7f005c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.425+0000 7f006fe00640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00780b30f0 0x7f005c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.425+0000 7f006fe00640 1 -- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f005c0039b0 msgr2=0x7f005c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.425+0000 7f006fe00640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f005c0039b0 0x7f005c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.425+0000 7f006fe00640 1 -- 172.21.15.69:0/418631200 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005c003520 con 0x7f005c004800 2024-04-24T08:16:56.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.426+0000 7f006fe00640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c004800 0x7f005c004c80 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f0060009a90 tx=0x7f005c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35d4401eac2b207a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.426+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060036070 con 0x7f005c004800 2024-04-24T08:16:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.426+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0060031400 con 0x7f005c004800 2024-04-24T08:16:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.426+0000 7f0077400640 1 -- 172.21.15.69:0/418631200 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060009690 con 0x7f005c004800 2024-04-24T08:16:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.427+0000 7f007d800640 1 -- 172.21.15.69:0/418631200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c004800 msgr2=0x7f005c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.427+0000 7f007d800640 1 --2- 172.21.15.69:0/418631200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f005c004800 0x7f005c004c80 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f0060009a90 tx=0x7f005c005830 comp rx=0 tx=0).stop 2024-04-24T08:16:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.427+0000 7f007ce00640 1 -- 172.21.15.69:0/418631200 reap_dead start 2024-04-24T08:16:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.427+0000 7f007d800640 1 -- 172.21.15.69:0/418631200 shutdown_connections 2024-04-24T08:16:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.427+0000 7f007d800640 1 -- 172.21.15.69:0/418631200 >> 172.21.15.69:0/418631200 conn(0x7f00780d0160 msgr2=0x7f00780d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.427+0000 7f007d800640 1 -- 172.21.15.69:0/418631200 shutdown_connections 2024-04-24T08:16:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.427+0000 7f007d800640 1 -- 172.21.15.69:0/418631200 wait complete. 2024-04-24T08:16:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.429+0000 7f007d800640 1 Processor -- start 2024-04-24T08:16:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.429+0000 7f007d800640 1 -- start start 2024-04-24T08:16:56.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.430+0000 7f007d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f005c004800 0x7f0078096320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.430+0000 7f007d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0078151d50 0x7f0078097c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.430+0000 7f007d800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00780981b0 0x7f0078155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.430+0000 7f007d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f007815d000 con 0x7f0078151d50 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.430+0000 7f007d800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f007815d940 con 0x7f00780981b0 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.430+0000 7f007d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f007815d180 con 0x7f005c004800 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.430+0000 7f006fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0078151d50 0x7f0078097c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f006fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0078151d50 0x7f0078097c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43668/0 (socket says 172.21.15.69:43668) 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f006fe00640 1 -- 172.21.15.69:0/4249584067 learned_addr learned my addr 172.21.15.69:0/4249584067 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f0077e00640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f005c004800 0x7f0078096320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f007ce00640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00780981b0 0x7f0078155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f006fe00640 1 -- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f005c004800 msgr2=0x7f0078096320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f006fe00640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f005c004800 0x7f0078096320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f006fe00640 1 -- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00780981b0 msgr2=0x7f0078155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f006fe00640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00780981b0 0x7f0078155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f006fe00640 1 -- 172.21.15.69:0/4249584067 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0078095280 con 0x7f0078151d50 2024-04-24T08:16:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f007ce00640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00780981b0 0x7f0078155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.431+0000 7f006fe00640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0078151d50 0x7f0078097c70 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f0060000c00 tx=0x7f0060004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.432+0000 7f0074c00640 1 -- 172.21.15.69:0/4249584067 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060040020 con 0x7f0078151d50 2024-04-24T08:16:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.432+0000 7f0074c00640 1 -- 172.21.15.69:0/4249584067 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0060003a40 con 0x7f0078151d50 2024-04-24T08:16:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.432+0000 7f0074c00640 1 -- 172.21.15.69:0/4249584067 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060036040 con 0x7f0078151d50 2024-04-24T08:16:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.432+0000 7f007d800640 1 -- 172.21.15.69:0/4249584067 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0078095560 con 0x7f0078151d50 2024-04-24T08:16:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.432+0000 7f007d800640 1 -- 172.21.15.69:0/4249584067 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0078095b20 con 0x7f0078151d50 2024-04-24T08:16:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.433+0000 7f007d800640 1 -- 172.21.15.69:0/4249584067 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00780985c0 con 0x7f0078151d50 2024-04-24T08:16:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.434+0000 7f0074c00640 1 -- 172.21.15.69:0/4249584067 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0060043020 con 0x7f0078151d50 2024-04-24T08:16:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.435+0000 7f0074c00640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f005406c920 0x7f005406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.435+0000 7f0074c00640 1 -- 172.21.15.69:0/4249584067 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f00600abc00 con 0x7f0078151d50 2024-04-24T08:16:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.435+0000 7f0077e00640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f005406c920 0x7f005406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.437+0000 7f0077e00640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f005406c920 0x7f005406ede0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0078099e20 tx=0x7f0068008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.440+0000 7f0074c00640 1 -- 172.21.15.69:0/4249584067 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00780985c0 con 0x7f0078151d50 2024-04-24T08:16:56.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.677+0000 7f007d800640 1 -- 172.21.15.69:0/4249584067 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f007806f9e0 con 0x7f0078151d50 2024-04-24T08:16:56.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.678+0000 7f0074c00640 1 -- 172.21.15.69:0/4249584067 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5182 (secure 0 0 0) 0x7f006004a070 con 0x7f0078151d50 2024-04-24T08:16:56.679 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:16:56.679 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":68,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:15:56.304080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7581":{"gid":7581,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":12,"addr":"172.21.15.69:6833/2529876410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2529876410},{"type":"v1","addr":"172.21.15.69:6833","nonce":2529876410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7581,"qdb_cluster":[7581]},"id":6}]} 2024-04-24T08:16:56.679 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 68 2024-04-24T08:16:56.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 -- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f005406c920 msgr2=0x7f005406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f005406c920 0x7f005406ede0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f0078099e20 tx=0x7f0068008040 comp rx=0 tx=0).stop 2024-04-24T08:16:56.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 -- 172.21.15.69:0/4249584067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0078151d50 msgr2=0x7f0078097c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0078151d50 0x7f0078097c70 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f0060000c00 tx=0x7f0060004730 comp rx=0 tx=0).stop 2024-04-24T08:16:56.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 -- 172.21.15.69:0/4249584067 shutdown_connections 2024-04-24T08:16:56.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f005406c920 0x7f005406ede0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0078151d50 0x7f0078097c70 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00780981b0 0x7f0078155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 --2- 172.21.15.69:0/4249584067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f005c004800 0x7f0078096320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 -- 172.21.15.69:0/4249584067 >> 172.21.15.69:0/4249584067 conn(0x7f00780d0160 msgr2=0x7f007815bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 -- 172.21.15.69:0/4249584067 shutdown_connections 2024-04-24T08:16:56.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.681+0000 7f007d800640 1 -- 172.21.15.69:0/4249584067 wait complete. 2024-04-24T08:16:56.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-24T08:16:56.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.900+0000 7f9f59e00640 1 Processor -- start 2024-04-24T08:16:56.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.900+0000 7f9f59e00640 1 -- start start 2024-04-24T08:16:56.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.902+0000 7f9f59e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9f541420a0 0x7f9f54071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.902+0000 7f9f59e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f541606e0 0x7f9f540cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.902+0000 7f9f59e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f54072450 con 0x7f9f54142480 2024-04-24T08:16:56.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.902+0000 7f9f59e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9f540cf1e0 con 0x7f9f541420a0 2024-04-24T08:16:56.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.902+0000 7f9f59e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9f54072730 con 0x7f9f541606e0 2024-04-24T08:16:56.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.902+0000 7f9f4be00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9f54142480 0x7f9f5415ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57968/0 (socket says 172.21.15.69:57968) 2024-04-24T08:16:56.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.902+0000 7f9f4be00640 1 -- 172.21.15.69:0/1928320303 learned_addr learned my addr 172.21.15.69:0/1928320303 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:16:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f59400640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f541606e0 0x7f9f540cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f58a00640 1 --2- 172.21.15.69:0/1928320303 >> v2:172.21.15.121:3300/0 conn(0x7f9f541420a0 0x7f9f54071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251753586 0 0) 0x7f9f54072450 con 0x7f9f54142480 2024-04-24T08:16:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f38003520 con 0x7f9f54142480 2024-04-24T08:16:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2778367649 0 0) 0x7f9f38003520 con 0x7f9f54142480 2024-04-24T08:16:56.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f541606e0 msgr2=0x7f9f540cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f541606e0 0x7f9f540cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 >> v2:172.21.15.121:3300/0 conn(0x7f9f541420a0 msgr2=0x7f9f54071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 --2- 172.21.15.69:0/1928320303 >> v2:172.21.15.121:3300/0 conn(0x7f9f541420a0 0x7f9f54071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f54072a10 con 0x7f9f54142480 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f9f3c001670 con 0x7f9f54142480 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 >> v1:172.21.15.69:6789/0 conn(0x7f9f54142480 legacy=0x7f9f5415ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f541606e0 0x7f9f380039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.903+0000 7f9f53e00640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f38003f30 0x7f9f38004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f53e00640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 0x7f9f380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9f540cf1e0 con 0x7f9f38004d40 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9f54072730 con 0x7f9f38003f30 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9f3800b9c0 con 0x7f9f541606e0 2024-04-24T08:16:56.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f59400640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 0x7f9f380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f4be00640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f38003f30 0x7f9f38004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f58a00640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f541606e0 0x7f9f380039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f59400640 1 -- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f541606e0 msgr2=0x7f9f380039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f59400640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f541606e0 0x7f9f380039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f59400640 1 -- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f38003f30 msgr2=0x7f9f38004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f59400640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f38003f30 0x7f9f38004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.904+0000 7f9f59400640 1 -- 172.21.15.69:0/1928320303 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f38003520 con 0x7f9f38004d40 2024-04-24T08:16:56.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.905+0000 7f9f59400640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 0x7f9f380051c0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009f90 tx=0x7f9f38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78635b2ca46225be server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.906+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c009c10 con 0x7f9f38004d40 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.906+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f4c036070 con 0x7f9f38004d40 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.906+0000 7f9f53e00640 1 -- 172.21.15.69:0/1928320303 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c031670 con 0x7f9f38004d40 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.906+0000 7f9f59e00640 1 -- 172.21.15.69:0/1928320303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 msgr2=0x7f9f380051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.906+0000 7f9f59e00640 1 --2- 172.21.15.69:0/1928320303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 0x7f9f380051c0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c009f90 tx=0x7f9f38005830 comp rx=0 tx=0).stop 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.906+0000 7f9f59400640 1 -- 172.21.15.69:0/1928320303 reap_dead start 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.906+0000 7f9f59e00640 1 -- 172.21.15.69:0/1928320303 shutdown_connections 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.906+0000 7f9f59e00640 1 -- 172.21.15.69:0/1928320303 >> 172.21.15.69:0/1928320303 conn(0x7f9f54067b80 msgr2=0x7f9f5406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.906+0000 7f9f59e00640 1 -- 172.21.15.69:0/1928320303 shutdown_connections 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.906+0000 7f9f59e00640 1 -- 172.21.15.69:0/1928320303 wait complete. 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.908+0000 7f9f59e00640 1 Processor -- start 2024-04-24T08:16:56.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.908+0000 7f9f59e00640 1 -- start start 2024-04-24T08:16:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.909+0000 7f9f59e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 0x7f9f54152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.909+0000 7f9f59e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f540b3420 0x7f9f5415b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.909+0000 7f9f59e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f540b3880 0x7f9f5415b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.909+0000 7f9f59e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9f540cf670 con 0x7f9f38004d40 2024-04-24T08:16:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.909+0000 7f9f59e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9f54142910 con 0x7f9f540b3880 2024-04-24T08:16:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.909+0000 7f9f59e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9f540cf7f0 con 0x7f9f540b3420 2024-04-24T08:16:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f58a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 0x7f9f54152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f58a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 0x7f9f54152400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43674/0 (socket says 172.21.15.69:43674) 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f58a00640 1 -- 172.21.15.69:0/2716705527 learned_addr learned my addr 172.21.15.69:0/2716705527 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f58a00640 1 -- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f540b3420 msgr2=0x7f9f5415b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f59400640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f540b3880 0x7f9f5415b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f4be00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f540b3420 0x7f9f5415b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f58a00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f540b3420 0x7f9f5415b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f58a00640 1 -- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f540b3880 msgr2=0x7f9f5415b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f58a00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f540b3880 0x7f9f5415b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f58a00640 1 -- 172.21.15.69:0/2716705527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f540b3e00 con 0x7f9f38004d40 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f58a00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 0x7f9f54152400 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f9f44011730 tx=0x7f9f44011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.910+0000 7f9f51600640 1 -- 172.21.15.69:0/2716705527 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4400f9c0 con 0x7f9f38004d40 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.911+0000 7f9f51600640 1 -- 172.21.15.69:0/2716705527 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f4400fb60 con 0x7f9f38004d40 2024-04-24T08:16:56.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.911+0000 7f9f59e00640 1 -- 172.21.15.69:0/2716705527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f540b40e0 con 0x7f9f38004d40 2024-04-24T08:16:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.911+0000 7f9f51600640 1 -- 172.21.15.69:0/2716705527 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f44015460 con 0x7f9f38004d40 2024-04-24T08:16:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.911+0000 7f9f59e00640 1 -- 172.21.15.69:0/2716705527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f540b46a0 con 0x7f9f38004d40 2024-04-24T08:16:56.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.913+0000 7f9f51600640 1 -- 172.21.15.69:0/2716705527 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9f44015600 con 0x7f9f38004d40 2024-04-24T08:16:56.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.914+0000 7f9f51600640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f3006c8f0 0x7f9f3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.914+0000 7f9f51600640 1 -- 172.21.15.69:0/2716705527 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f9f4408c760 con 0x7f9f38004d40 2024-04-24T08:16:56.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.914+0000 7f9f5e226740 1 Processor -- start 2024-04-24T08:16:56.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.915+0000 7f9f4be00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f3006c8f0 0x7f9f3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.915+0000 7f9f5e226740 1 -- start start 2024-04-24T08:16:56.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07acf7220 0x55f07acf7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad14ac0 0x55f07ad15010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07acccf00 0x55f07ad13180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f5e226740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f07ac31190 con 0x55f07ad14ac0 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f5e226740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f07abd7030 con 0x55f07acf7220 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f5e226740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f07ac2d150 con 0x55f07acccf00 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f4be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad14ac0 0x55f07ad15010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f4be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad14ac0 0x55f07ad15010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43690/0 (socket says 172.21.15.69:43690) 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f4be00640 1 -- 172.21.15.69:0/1404643979 learned_addr learned my addr 172.21.15.69:0/1404643979 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f4be00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f3006c8f0 0x7f9f3006edb0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c001eb0 tx=0x7f9f3c000f70 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f4be00640 1 -- 172.21.15.69:0/1404643979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07acccf00 msgr2=0x55f07ad13180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f4be00640 1 --2- 172.21.15.69:0/1404643979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07acccf00 0x55f07ad13180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f4be00640 1 -- 172.21.15.69:0/1404643979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07acf7220 msgr2=0x55f07acf7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f4be00640 1 --2- 172.21.15.69:0/1404643979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07acf7220 0x55f07acf7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.916+0000 7f9f4be00640 1 -- 172.21.15.69:0/1404643979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f07abbfb60 con 0x55f07ad14ac0 2024-04-24T08:16:56.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.917+0000 7f9f4be00640 1 --2- 172.21.15.69:0/1404643979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad14ac0 0x55f07ad15010 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c00fb10 tx=0x7f9f3c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81115da1b1f1051e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.918+0000 7f9f49200640 1 -- 172.21.15.69:0/1404643979 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c026070 con 0x55f07ad14ac0 2024-04-24T08:16:56.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.918+0000 7f9f49200640 1 -- 172.21.15.69:0/1404643979 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f3c019070 con 0x55f07ad14ac0 2024-04-24T08:16:56.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.918+0000 7f9f49200640 1 -- 172.21.15.69:0/1404643979 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c01b070 con 0x55f07ad14ac0 2024-04-24T08:16:56.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.919+0000 7f9f5e226740 1 -- 172.21.15.69:0/1404643979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad14ac0 msgr2=0x55f07ad15010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.919+0000 7f9f5e226740 1 --2- 172.21.15.69:0/1404643979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad14ac0 0x55f07ad15010 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c00fb10 tx=0x7f9f3c012000 comp rx=0 tx=0).stop 2024-04-24T08:16:56.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.919+0000 7f9f5e226740 1 -- 172.21.15.69:0/1404643979 shutdown_connections 2024-04-24T08:16:56.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.919+0000 7f9f5e226740 1 --2- 172.21.15.69:0/1404643979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad14ac0 0x55f07ad15010 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.919+0000 7f9f5e226740 1 --2- 172.21.15.69:0/1404643979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07acccf00 0x55f07ad13180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.919+0000 7f9f5e226740 1 --2- 172.21.15.69:0/1404643979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07acf7220 0x55f07acf7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.919+0000 7f9f5e226740 1 -- 172.21.15.69:0/1404643979 >> 172.21.15.69:0/1404643979 conn(0x55f07acc9830 msgr2=0x55f07ad12030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.919+0000 7f9f5e226740 1 -- 172.21.15.69:0/1404643979 shutdown_connections 2024-04-24T08:16:56.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.919+0000 7f9f5e226740 1 -- 172.21.15.69:0/1404643979 wait complete. 2024-04-24T08:16:56.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.920+0000 7f9f5e226740 1 -- start start 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.920+0000 7f9f5e226740 1 Processor -- start 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.921+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07acf3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.921+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad14ac0 0x55f07ad0b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.921+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07ad0b8c0 0x55f07acc6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.921+0000 7f9f5e226740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f07accdca0 con 0x55f07acf7220 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.921+0000 7f9f5e226740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f07abea500 con 0x55f07ad0b8c0 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.921+0000 7f9f5e226740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f07acc7760 con 0x55f07ad14ac0 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.921+0000 7f9f5e226740 10 client.0 _ll_register_callbacks cb 0x55f07ac27270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f58a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07acf3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f58a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07acf3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43692/0 (socket says 172.21.15.69:43692) 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f58a00640 1 -- 172.21.15.69:0/1740316124 learned_addr learned my addr 172.21.15.69:0/1740316124 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f58a00640 1 -- 172.21.15.69:0/1740316124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad14ac0 msgr2=0x55f07ad0b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f58a00640 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad14ac0 0x55f07ad0b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f58a00640 1 -- 172.21.15.69:0/1740316124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07ad0b8c0 msgr2=0x55f07acc6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f58a00640 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07ad0b8c0 0x55f07acc6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f58a00640 1 -- 172.21.15.69:0/1740316124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f07aa0f7c0 con 0x55f07acf7220 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f58a00640 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07acf3ae0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f9f44012010 tx=0x7f9f4405a450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e35c080bd40dc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f41600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f41600640 1 -- 172.21.15.69:0/1740316124 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f44062cf0 con 0x55f07acf7220 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f41600640 1 -- 172.21.15.69:0/1740316124 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f9f44085c20 con 0x55f07acf7220 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.922+0000 7f9f5e226740 1 -- 172.21.15.69:0/1740316124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f3c019070 con 0x55f07acf7220 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f41600640 1 -- 172.21.15.69:0/1740316124 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f9f44085e20 con 0x55f07acf7220 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f5e226740 10 client.8061 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f5e226740 1 -- 172.21.15.69:0/1740316124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55f07aa6ff50 con 0x55f07acf7220 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f41600640 1 -- 172.21.15.69:0/1740316124 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f3c019070 con 0x55f07acf7220 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f41600640 1 -- 172.21.15.69:0/1740316124 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f9f44072b80 con 0x55f07acf7220 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f5e226740 10 client.8061 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f5e226740 10 client.8061 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:16:56.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f5e226740 20 client.8061 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:56.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f5e226740 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55f07ad261b0 0x55f07acf9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f5e226740 4 client.8061 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:56.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f5e226740 1 -- 172.21.15.69:0/1740316124 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f07ac3ebb0 con 0x55f07ad261b0 2024-04-24T08:16:56.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.923+0000 7f9f4be00640 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55f07ad261b0 0x55f07acf9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.924+0000 7f9f4be00640 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55f07ad261b0 0x55f07acf9a00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=451c78350bbf2bcf server_cookie=95cda7ab8a419f47 in_seq=0 out_seq=0 2024-04-24T08:16:56.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.924+0000 7f9f41600640 10 client.8061 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:56.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f41600640 1 -- 172.21.15.69:0/1740316124 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f07ac3ebb0 con 0x55f07ad261b0 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f41600640 10 client.8061 handle_command_reply: tid=0 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 client.8061 shutdown 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 -- 172.21.15.69:0/1740316124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 msgr2=0x55f07acf3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07acf3ae0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f9f44012010 tx=0x7f9f4405a450 comp rx=0 tx=0).stop 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 -- 172.21.15.69:0/1740316124 shutdown_connections 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55f07ad261b0 0x55f07acf9a00 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07ad0b8c0 0x55f07acc6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad14ac0 0x55f07ad0b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 --2- 172.21.15.69:0/1740316124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07acf3ae0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 -- 172.21.15.69:0/1740316124 >> 172.21.15.69:0/1740316124 conn(0x55f07acccf00 msgr2=0x55f07ad155b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f41600640 0 client.8061 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 -- 172.21.15.69:0/1740316124 shutdown_connections 2024-04-24T08:16:56.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.925+0000 7f9f5e226740 1 -- 172.21.15.69:0/1740316124 wait complete. 2024-04-24T08:16:56.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.926+0000 7f9f5e226740 20 client.8061 trim_cache size 0 max 16384 2024-04-24T08:16:56.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.940+0000 7f9f5e226740 1 Processor -- start 2024-04-24T08:16:56.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.940+0000 7f9f5e226740 1 -- start start 2024-04-24T08:16:56.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.941+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07acf7220 0x55f07acfacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.941+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad0b8c0 0x55f07ad86b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.941+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad14ac0 0x55f07ad10620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.941+0000 7f9f5e226740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9f542004f0 con 0x55f07ad0b8c0 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.941+0000 7f9f5e226740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9f54155900 con 0x55f07acf7220 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.941+0000 7f9f5e226740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9f541ffed0 con 0x55f07ad14ac0 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.941+0000 7f9f4be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad0b8c0 0x55f07ad86b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.941+0000 7f9f4be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad0b8c0 0x55f07ad86b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43706/0 (socket says 172.21.15.69:43706) 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.941+0000 7f9f4be00640 1 -- 172.21.15.69:0/611115564 learned_addr learned my addr 172.21.15.69:0/611115564 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f4be00640 1 -- 172.21.15.69:0/611115564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad14ac0 msgr2=0x55f07ad10620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f58a00640 1 --2- 172.21.15.69:0/611115564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07acf7220 0x55f07acfacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f59400640 1 --2- 172.21.15.69:0/611115564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad14ac0 0x55f07ad10620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f4be00640 1 --2- 172.21.15.69:0/611115564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad14ac0 0x55f07ad10620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f4be00640 1 -- 172.21.15.69:0/611115564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07acf7220 msgr2=0x55f07acfacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f4be00640 1 --2- 172.21.15.69:0/611115564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07acf7220 0x55f07acfacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f4be00640 1 -- 172.21.15.69:0/611115564 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f07acc66e0 con 0x55f07ad0b8c0 2024-04-24T08:16:56.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f4be00640 1 --2- 172.21.15.69:0/611115564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad0b8c0 0x55f07ad86b30 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c00fab0 tx=0x7f9f3c009070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49700ab18fdf8ca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f43e00640 1 -- 172.21.15.69:0/611115564 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c026040 con 0x55f07ad0b8c0 2024-04-24T08:16:56.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f43e00640 1 -- 172.21.15.69:0/611115564 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f3c019070 con 0x55f07ad0b8c0 2024-04-24T08:16:56.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.942+0000 7f9f43e00640 1 -- 172.21.15.69:0/611115564 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c01b040 con 0x55f07ad0b8c0 2024-04-24T08:16:56.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.943+0000 7f9f5e226740 1 -- 172.21.15.69:0/611115564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad0b8c0 msgr2=0x55f07ad86b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.943+0000 7f9f5e226740 1 --2- 172.21.15.69:0/611115564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad0b8c0 0x55f07ad86b30 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c00fab0 tx=0x7f9f3c009070 comp rx=0 tx=0).stop 2024-04-24T08:16:56.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.943+0000 7f9f5e226740 1 -- 172.21.15.69:0/611115564 shutdown_connections 2024-04-24T08:16:56.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.943+0000 7f9f5e226740 1 --2- 172.21.15.69:0/611115564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07ad0b8c0 0x55f07ad86b30 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.943+0000 7f9f5e226740 1 --2- 172.21.15.69:0/611115564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad14ac0 0x55f07ad10620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.943+0000 7f9f5e226740 1 --2- 172.21.15.69:0/611115564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07acf7220 0x55f07acfacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.943+0000 7f9f5e226740 1 -- 172.21.15.69:0/611115564 >> 172.21.15.69:0/611115564 conn(0x55f07acccf00 msgr2=0x55f07acf38f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.943+0000 7f9f5e226740 1 -- 172.21.15.69:0/611115564 shutdown_connections 2024-04-24T08:16:56.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.943+0000 7f9f5e226740 1 -- 172.21.15.69:0/611115564 wait complete. 2024-04-24T08:16:56.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.944+0000 7f9f5e226740 1 -- start start 2024-04-24T08:16:56.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.944+0000 7f9f5e226740 1 Processor -- start 2024-04-24T08:16:56.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.945+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07ad28810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.945+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad0b8c0 0x55f07ad75280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.945+0000 7f9f5e226740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07ad14ac0 0x55f07ad10620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.945+0000 7f9f5e226740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f07ad26030 con 0x55f07acf7220 2024-04-24T08:16:56.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.945+0000 7f9f5e226740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f07ad22fe0 con 0x55f07ad14ac0 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.945+0000 7f9f5e226740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f07ad26680 con 0x55f07ad0b8c0 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.945+0000 7f9f5e226740 10 client.0 _ll_register_callbacks cb 0x7f9f440565b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.945+0000 7f9f58a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07ad28810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.945+0000 7f9f58a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07ad28810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43710/0 (socket says 172.21.15.69:43710) 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.945+0000 7f9f58a00640 1 -- 172.21.15.69:0/2268053986 learned_addr learned my addr 172.21.15.69:0/2268053986 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f59400640 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07ad14ac0 0x55f07ad10620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f4be00640 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad0b8c0 0x55f07ad75280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f58a00640 1 -- 172.21.15.69:0/2268053986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad0b8c0 msgr2=0x55f07ad75280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f58a00640 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad0b8c0 0x55f07ad75280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f58a00640 1 -- 172.21.15.69:0/2268053986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07ad14ac0 msgr2=0x55f07ad10620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f58a00640 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07ad14ac0 0x55f07ad10620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f58a00640 1 -- 172.21.15.69:0/2268053986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f07a9030b0 con 0x55f07acf7220 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f58a00640 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07ad28810 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f9f4408bfd0 tx=0x7f9f4401f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a5512e9a54fd44 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f41600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f41600640 1 -- 172.21.15.69:0/2268053986 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f4407e020 con 0x55f07acf7220 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f41600640 1 -- 172.21.15.69:0/2268053986 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f9f44059760 con 0x55f07acf7220 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f5e226740 1 -- 172.21.15.69:0/2268053986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f3c019070 con 0x55f07acf7220 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f41600640 1 -- 172.21.15.69:0/2268053986 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) ==== 24+0+0 (secure 0 0 0) 0x7f9f44059e50 con 0x55f07acf7220 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f5e226740 10 client.8067 fetch_fsmap learned FSMap version 68 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.946+0000 7f9f5e226740 1 -- 172.21.15.69:0/2268053986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=2+}) -- 0x55f07ad08350 con 0x55f07acf7220 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.947+0000 7f9f41600640 1 -- 172.21.15.69:0/2268053986 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f3c019070 con 0x55f07acf7220 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.947+0000 7f9f41600640 1 -- 172.21.15.69:0/2268053986 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 68) ==== 3494+0+0 (secure 0 0 0) 0x7f9f44084050 con 0x55f07acf7220 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.947+0000 7f9f5e226740 10 client.8067 fetch_fsmap finished waiting for FSMap version 68 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.947+0000 7f9f5e226740 10 client.8067 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:16:56.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.947+0000 7f9f5e226740 20 client.8067 populate_metadata read hostname 'smithi069' 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.947+0000 7f9f5e226740 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55f07ad35890 0x55f07ad2a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.947+0000 7f9f5e226740 4 client.8067 mds_command: new command op to 7581 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.947+0000 7f9f5e226740 1 -- 172.21.15.69:0/2268053986 --> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55f07ac38a20 con 0x55f07ad35890 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.947+0000 7f9f4be00640 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55f07ad35890 0x55f07ad2a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f41600640 10 client.8067 ms_handle_connect on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f4be00640 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55f07ad35890 0x55f07ad2a340 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=668321f2bc4898fe server_cookie=5746eb6884f7e5d4 in_seq=0 out_seq=0 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f41600640 1 -- 172.21.15.69:0/2268053986 <== mds.0 v2:172.21.15.69:6832/2529876410 1 ==== command_reply(tid 0: 0 ) ==== 8+0+144 (crc 0 0 0) 0x55f07ac38a20 con 0x55f07ad35890 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f41600640 10 client.8067 handle_command_reply: tid=0 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f5e226740 1 client.8067 shutdown 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f5e226740 1 -- 172.21.15.69:0/2268053986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 msgr2=0x55f07ad28810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f5e226740 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07ad28810 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f9f4408bfd0 tx=0x7f9f4401f040 comp rx=0 tx=0).stop 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f5e226740 1 -- 172.21.15.69:0/2268053986 shutdown_connections 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f5e226740 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.69:6832/2529876410,v1:172.21.15.69:6833/2529876410] conn(0x55f07ad35890 0x55f07ad2a340 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f5e226740 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f07ad14ac0 0x55f07ad10620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f5e226740 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f07ad0b8c0 0x55f07ad75280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f41600640 0 client.8067 ms_handle_reset on v2:172.21.15.69:6832/2529876410 2024-04-24T08:16:56.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f5e226740 1 --2- 172.21.15.69:0/2268053986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f07acf7220 0x55f07ad28810 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f5e226740 1 -- 172.21.15.69:0/2268053986 >> 172.21.15.69:0/2268053986 conn(0x55f07acccf00 msgr2=0x55f07ad0fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.948+0000 7f9f5e226740 1 -- 172.21.15.69:0/2268053986 shutdown_connections 2024-04-24T08:16:56.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f5e226740 1 -- 172.21.15.69:0/2268053986 wait complete. 2024-04-24T08:16:56.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f5e226740 20 client.8067 trim_cache size 0 max 16384 2024-04-24T08:16:56.953 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:16:56.953 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:16:56.953 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:16:56.954 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-04-24T08:16:56.954 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 1301104333, 2024-04-24T08:16:56.954 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511627777, 2024-04-24T08:16:56.954 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/testdir/hardlink" 2024-04-24T08:16:56.954 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:16:56.954 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 -- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f3006c8f0 msgr2=0x7f9f3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f3006c8f0 0x7f9f3006edb0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c001eb0 tx=0x7f9f3c000f70 comp rx=0 tx=0).stop 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 -- 172.21.15.69:0/2716705527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 msgr2=0x7f9f54152400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 0x7f9f54152400 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f9f44011730 tx=0x7f9f44011bf0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 -- 172.21.15.69:0/2716705527 shutdown_connections 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f3006c8f0 0x7f9f3006edb0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f540b3880 0x7f9f5415b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f540b3420 0x7f9f5415b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 --2- 172.21.15.69:0/2716705527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f38004d40 0x7f9f54152400 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 -- 172.21.15.69:0/2716705527 >> 172.21.15.69:0/2716705527 conn(0x7f9f54067b80 msgr2=0x7f9f5415c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:56.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 -- 172.21.15.69:0/2716705527 shutdown_connections 2024-04-24T08:16:56.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:56.949+0000 7f9f59e00640 1 -- 172.21.15.69:0/2716705527 wait complete. 2024-04-24T08:16:56.967 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:16:56.967 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:16:57.014 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:16:57.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.256+0000 7fda4d800640 1 Processor -- start 2024-04-24T08:16:57.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.256+0000 7fda4d800640 1 -- start start 2024-04-24T08:16:57.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.258+0000 7fda4d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda480b36b0 0x7fda480b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.258+0000 7fda4d800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda480aeac0 con 0x7fda481523f0 2024-04-24T08:16:57.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.258+0000 7fda4d800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda480aeda0 con 0x7fda4806db90 2024-04-24T08:16:57.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.258+0000 7fda4d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fda480af080 con 0x7fda480b36b0 2024-04-24T08:16:57.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.258+0000 7fda47e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fda481523f0 0x7fda48072e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57974/0 (socket says 172.21.15.69:57974) 2024-04-24T08:16:57.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.258+0000 7fda47e00640 1 -- 172.21.15.69:0/824784832 learned_addr learned my addr 172.21.15.69:0/824784832 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:16:57.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.258+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3941135942 0 0) 0x7fda480aeac0 con 0x7fda481523f0 2024-04-24T08:16:57.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.258+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda2c003520 con 0x7fda481523f0 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.258+0000 7fda4ce00640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda480b36b0 0x7fda480b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3117916289 0 0) 0x7fda2c003520 con 0x7fda481523f0 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda480b36b0 msgr2=0x7fda480b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda480b36b0 0x7fda480b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 >> v1:172.21.15.121:6789/0 conn(0x7fda4806db90 legacy=0x7fda48073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda480af360 con 0x7fda481523f0 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fda38001670 con 0x7fda481523f0 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 >> v1:172.21.15.69:6789/0 conn(0x7fda481523f0 legacy=0x7fda48072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda480b36b0 0x7fda2c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c003f20 0x7fda2c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda2c004d40 0x7fda2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fda480af080 con 0x7fda2c004d40 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fda2c010000 con 0x7fda2c003f20 2024-04-24T08:16:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fda2c010200 con 0x7fda480b36b0 2024-04-24T08:16:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda4ce00640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda2c004d40 0x7fda2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.259+0000 7fda3f400640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c003f20 0x7fda2c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.260+0000 7fda47e00640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda480b36b0 0x7fda2c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.260+0000 7fda4ce00640 1 -- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda480b36b0 msgr2=0x7fda2c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.260+0000 7fda4ce00640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda480b36b0 0x7fda2c0039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.260+0000 7fda4ce00640 1 -- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c003f20 msgr2=0x7fda2c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.260+0000 7fda4ce00640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c003f20 0x7fda2c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.260+0000 7fda4ce00640 1 -- 172.21.15.69:0/824784832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2c003520 con 0x7fda2c004d40 2024-04-24T08:16:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.260+0000 7fda3f400640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c003f20 0x7fda2c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:57.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.261+0000 7fda4ce00640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda2c004d40 0x7fda2c0051c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fda40007360 tx=0x7fda2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8e5ed60fb04309 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.261+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40009be0 con 0x7fda2c004d40 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.261+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda40036600 con 0x7fda2c004d40 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.261+0000 7fda47400640 1 -- 172.21.15.69:0/824784832 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda400369d0 con 0x7fda2c004d40 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.261+0000 7fda4d800640 1 -- 172.21.15.69:0/824784832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda2c004d40 msgr2=0x7fda2c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.261+0000 7fda4d800640 1 --2- 172.21.15.69:0/824784832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda2c004d40 0x7fda2c0051c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fda40007360 tx=0x7fda2c005830 comp rx=0 tx=0).stop 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.261+0000 7fda4ce00640 1 -- 172.21.15.69:0/824784832 reap_dead start 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.262+0000 7fda4d800640 1 -- 172.21.15.69:0/824784832 shutdown_connections 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.262+0000 7fda4d800640 1 -- 172.21.15.69:0/824784832 >> 172.21.15.69:0/824784832 conn(0x7fda480d0160 msgr2=0x7fda480d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.262+0000 7fda4d800640 1 -- 172.21.15.69:0/824784832 shutdown_connections 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.262+0000 7fda4d800640 1 -- 172.21.15.69:0/824784832 wait complete. 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.263+0000 7fda4d800640 1 Processor -- start 2024-04-24T08:16:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.263+0000 7fda4d800640 1 -- start start 2024-04-24T08:16:57.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.264+0000 7fda4d800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda481efc60 0x7fda48072880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.264+0000 7fda4d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda481f0040 0x7fda48072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.264+0000 7fda4d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda481488b0 0x7fda4813e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.264+0000 7fda4d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fda481527d0 con 0x7fda481488b0 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.264+0000 7fda4d800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fda4806dfd0 con 0x7fda481efc60 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.264+0000 7fda4d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fda480b3b40 con 0x7fda481f0040 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda4ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda481488b0 0x7fda4813e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda4ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda481488b0 0x7fda4813e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43734/0 (socket says 172.21.15.69:43734) 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda4ce00640 1 -- 172.21.15.69:0/3131289685 learned_addr learned my addr 172.21.15.69:0/3131289685 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda3f400640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda481f0040 0x7fda48072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda4ce00640 1 -- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda481f0040 msgr2=0x7fda48072dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda47e00640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda481efc60 0x7fda48072880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda4ce00640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda481f0040 0x7fda48072dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda4ce00640 1 -- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda481efc60 msgr2=0x7fda48072880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda4ce00640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda481efc60 0x7fda48072880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda4ce00640 1 -- 172.21.15.69:0/3131289685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4809db50 con 0x7fda481488b0 2024-04-24T08:16:57.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda47e00640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda481efc60 0x7fda48072880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:57.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.265+0000 7fda4ce00640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda481488b0 0x7fda4813e4f0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fda40007480 tx=0x7fda40004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:57.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.266+0000 7fda44c00640 1 -- 172.21.15.69:0/3131289685 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40009be0 con 0x7fda481488b0 2024-04-24T08:16:57.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.266+0000 7fda44c00640 1 -- 172.21.15.69:0/3131289685 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda40036400 con 0x7fda481488b0 2024-04-24T08:16:57.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.266+0000 7fda44c00640 1 -- 172.21.15.69:0/3131289685 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40038520 con 0x7fda481488b0 2024-04-24T08:16:57.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.266+0000 7fda4d800640 1 -- 172.21.15.69:0/3131289685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda4809de30 con 0x7fda481488b0 2024-04-24T08:16:57.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.266+0000 7fda4d800640 1 -- 172.21.15.69:0/3131289685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4813dcf0 con 0x7fda481488b0 2024-04-24T08:16:57.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.267+0000 7fda4d800640 1 -- 172.21.15.69:0/3131289685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda480b3ed0 con 0x7fda481488b0 2024-04-24T08:16:57.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.268+0000 7fda44c00640 1 -- 172.21.15.69:0/3131289685 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fda4002e620 con 0x7fda481488b0 2024-04-24T08:16:57.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.269+0000 7fda44c00640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda2406c8f0 0x7fda2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.269+0000 7fda44c00640 1 -- 172.21.15.69:0/3131289685 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7fda400ac180 con 0x7fda481488b0 2024-04-24T08:16:57.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.269+0000 7fda47e00640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda2406c8f0 0x7fda2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.270+0000 7fda47e00640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda2406c8f0 0x7fda2406edb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fda38001200 tx=0x7fda38009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:57.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.273+0000 7fda44c00640 1 -- 172.21.15.69:0/3131289685 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda480b3ed0 con 0x7fda481488b0 2024-04-24T08:16:57.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.477+0000 7fda4d800640 1 -- 172.21.15.69:0/3131289685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda480afb40 con 0x7fda481488b0 2024-04-24T08:16:57.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.479+0000 7fda44c00640 1 -- 172.21.15.69:0/3131289685 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+16817 (secure 0 0 0) 0x7fda480afb40 con 0x7fda481488b0 2024-04-24T08:16:57.480 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:16:57.480 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":85,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:16:03.587918+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6837/1179683245":"2024-04-25T08:15:45.259762+0000","172.21.15.121:6835/1179683245":"2024-04-25T08:15:45.259762+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 -- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda2406c8f0 msgr2=0x7fda2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda2406c8f0 0x7fda2406edb0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fda38001200 tx=0x7fda38009040 comp rx=0 tx=0).stop 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 -- 172.21.15.69:0/3131289685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda481488b0 msgr2=0x7fda4813e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda481488b0 0x7fda4813e4f0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fda40007480 tx=0x7fda40004ab0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 -- 172.21.15.69:0/3131289685 shutdown_connections 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda2406c8f0 0x7fda2406edb0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda481488b0 0x7fda4813e4f0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda481f0040 0x7fda48072dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 --2- 172.21.15.69:0/3131289685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda481efc60 0x7fda48072880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 -- 172.21.15.69:0/3131289685 >> 172.21.15.69:0/3131289685 conn(0x7fda480d0160 msgr2=0x7fda4806e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:57.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 -- 172.21.15.69:0/3131289685 shutdown_connections 2024-04-24T08:16:57.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.482+0000 7fda4d800640 1 -- 172.21.15.69:0/3131289685 wait complete. 2024-04-24T08:16:57.502 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:15:21.126081+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:15:22.402310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:16:57.502 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:16:57.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.705+0000 7f313ec00640 1 Processor -- start 2024-04-24T08:16:57.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.705+0000 7f313ec00640 1 -- start start 2024-04-24T08:16:57.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.706+0000 7f313ec00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f31380cd110 0x7f3138160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.706+0000 7f313ec00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31381607c0 0x7f313815c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.706+0000 7f313ec00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31380d0310 con 0x7f313815c4a0 2024-04-24T08:16:57.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.706+0000 7f313ec00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f31380d2b20 con 0x7f31380cd110 2024-04-24T08:16:57.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.706+0000 7f313ec00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f31380d05f0 con 0x7f31381607c0 2024-04-24T08:16:57.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.707+0000 7f313ce00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f31380cd110 0x7f3138160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.707+0000 7f313e200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31381607c0 0x7f313815c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.707+0000 7f313ce00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f31380cd110 0x7f3138160280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:52250/0 (socket says 172.21.15.69:52250) 2024-04-24T08:16:57.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.707+0000 7f313ce00640 1 -- 172.21.15.69:0/2514390778 learned_addr learned my addr 172.21.15.69:0/2514390778 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:57.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.708+0000 7f313ce00640 1 -- 172.21.15.69:0/2514390778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31381607c0 msgr2=0x7f313815c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:57.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.708+0000 7f313ce00640 1 --2- 172.21.15.69:0/2514390778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31381607c0 0x7f313815c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.708+0000 7f313ce00640 1 -- 172.21.15.69:0/2514390778 >> v1:172.21.15.121:6789/0 conn(0x7f313815c4a0 legacy=0x7f31380cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.708+0000 7f313ce00640 1 -- 172.21.15.69:0/2514390778 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31380d08d0 con 0x7f31380cd110 2024-04-24T08:16:57.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.708+0000 7f313e200640 1 --2- 172.21.15.69:0/2514390778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31381607c0 0x7f313815c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:16:57.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.709+0000 7f313ce00640 1 --2- 172.21.15.69:0/2514390778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31380cd110 0x7f3138160280 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3124009870 tx=0x7f312402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=212359b12b1e2574 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:57.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.709+0000 7f3137e00640 1 -- 172.21.15.69:0/2514390778 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3124037070 con 0x7f31380cd110 2024-04-24T08:16:57.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.710+0000 7f3137e00640 1 -- 172.21.15.69:0/2514390778 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3124032440 con 0x7f31380cd110 2024-04-24T08:16:57.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.710+0000 7f3137e00640 1 -- 172.21.15.69:0/2514390778 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3124032810 con 0x7f31380cd110 2024-04-24T08:16:57.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.710+0000 7f313ec00640 1 -- 172.21.15.69:0/2514390778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31380cd110 msgr2=0x7f3138160280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.710+0000 7f313ec00640 1 --2- 172.21.15.69:0/2514390778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31380cd110 0x7f3138160280 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3124009870 tx=0x7f312402a250 comp rx=0 tx=0).stop 2024-04-24T08:16:57.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.710+0000 7f313ec00640 1 -- 172.21.15.69:0/2514390778 shutdown_connections 2024-04-24T08:16:57.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.710+0000 7f313ec00640 1 --2- 172.21.15.69:0/2514390778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31381607c0 0x7f313815c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.710+0000 7f313ec00640 1 --2- 172.21.15.69:0/2514390778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31380cd110 0x7f3138160280 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.710+0000 7f313ec00640 1 -- 172.21.15.69:0/2514390778 >> 172.21.15.69:0/2514390778 conn(0x7f3138067760 msgr2=0x7f3138067b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:57.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.710+0000 7f313ec00640 1 -- 172.21.15.69:0/2514390778 shutdown_connections 2024-04-24T08:16:57.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.710+0000 7f313ec00640 1 -- 172.21.15.69:0/2514390778 wait complete. 2024-04-24T08:16:57.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.712+0000 7f313ec00640 1 Processor -- start 2024-04-24T08:16:57.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.712+0000 7f313ec00640 1 -- start start 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.713+0000 7f313ec00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31380cd110 0x7f3138157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.713+0000 7f313ec00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f313815c4a0 0x7f313815ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.713+0000 7f313ec00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31381607c0 0x7f313815b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.713+0000 7f313ec00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31380d10d0 con 0x7f313815c4a0 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.713+0000 7f313ec00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3138160c00 con 0x7f31381607c0 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.713+0000 7f313ec00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f31380c5ea0 con 0x7f31380cd110 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.713+0000 7f313ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f313815c4a0 0x7f313815ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.713+0000 7f313ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f313815c4a0 0x7f313815ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43742/0 (socket says 172.21.15.69:43742) 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.713+0000 7f313ce00640 1 -- 172.21.15.69:0/3582205240 learned_addr learned my addr 172.21.15.69:0/3582205240 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.714+0000 7f313d800640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31380cd110 0x7f3138157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.714+0000 7f313e200640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31381607c0 0x7f313815b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.714+0000 7f313ce00640 1 -- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31380cd110 msgr2=0x7f3138157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.714+0000 7f313ce00640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31380cd110 0x7f3138157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.714+0000 7f313ce00640 1 -- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31381607c0 msgr2=0x7f313815b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.714+0000 7f313ce00640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31381607c0 0x7f313815b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.714+0000 7f313ce00640 1 -- 172.21.15.69:0/3582205240 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31380cbb90 con 0x7f313815c4a0 2024-04-24T08:16:57.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.714+0000 7f313ce00640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f313815c4a0 0x7f313815ace0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f3124000c00 tx=0x7f31240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:57.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.715+0000 7f3135600640 1 -- 172.21.15.69:0/3582205240 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3124037040 con 0x7f313815c4a0 2024-04-24T08:16:57.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.715+0000 7f3135600640 1 -- 172.21.15.69:0/3582205240 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3124042070 con 0x7f313815c4a0 2024-04-24T08:16:57.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.715+0000 7f313ec00640 1 -- 172.21.15.69:0/3582205240 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31380cbe00 con 0x7f313815c4a0 2024-04-24T08:16:57.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.715+0000 7f3135600640 1 -- 172.21.15.69:0/3582205240 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3124030880 con 0x7f313815c4a0 2024-04-24T08:16:57.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.715+0000 7f313ec00640 1 -- 172.21.15.69:0/3582205240 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31380cc3c0 con 0x7f313815c4a0 2024-04-24T08:16:57.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.716+0000 7f313ec00640 1 -- 172.21.15.69:0/3582205240 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31380b41f0 con 0x7f313815c4a0 2024-04-24T08:16:57.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.717+0000 7f3135600640 1 -- 172.21.15.69:0/3582205240 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3124030a20 con 0x7f313815c4a0 2024-04-24T08:16:57.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.718+0000 7f3135600640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f311806c8f0 0x7f311806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:57.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.718+0000 7f3135600640 1 -- 172.21.15.69:0/3582205240 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f31240ac400 con 0x7f313815c4a0 2024-04-24T08:16:57.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.719+0000 7f313d800640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f311806c8f0 0x7f311806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:57.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.720+0000 7f313d800640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f311806c8f0 0x7f311806edb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f312c001640 tx=0x7f312c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:57.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.723+0000 7f3135600640 1 -- 172.21.15.69:0/3582205240 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f31380b41f0 con 0x7f313815c4a0 2024-04-24T08:16:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.923+0000 7f313ec00640 1 -- 172.21.15.69:0/3582205240 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3138075d50 con 0x7f313815c4a0 2024-04-24T08:16:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.924+0000 7f3135600640 1 -- 172.21.15.69:0/3582205240 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f312407a7d0 con 0x7f313815c4a0 2024-04-24T08:16:57.925 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:16:57.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.927+0000 7f313ec00640 1 -- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f311806c8f0 msgr2=0x7f311806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.927+0000 7f313ec00640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f311806c8f0 0x7f311806edb0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f312c001640 tx=0x7f312c009040 comp rx=0 tx=0).stop 2024-04-24T08:16:57.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.927+0000 7f313ec00640 1 -- 172.21.15.69:0/3582205240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f313815c4a0 msgr2=0x7f313815ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:57.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.927+0000 7f313ec00640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f313815c4a0 0x7f313815ace0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f3124000c00 tx=0x7f31240046c0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.927+0000 7f313ec00640 1 -- 172.21.15.69:0/3582205240 shutdown_connections 2024-04-24T08:16:57.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.927+0000 7f313ec00640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f311806c8f0 0x7f311806edb0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.927+0000 7f313ec00640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f313815c4a0 0x7f313815ace0 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.927+0000 7f313ec00640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31381607c0 0x7f313815b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.927+0000 7f313ec00640 1 --2- 172.21.15.69:0/3582205240 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31380cd110 0x7f3138157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.927+0000 7f313ec00640 1 -- 172.21.15.69:0/3582205240 >> 172.21.15.69:0/3582205240 conn(0x7f3138067760 msgr2=0x7f313815d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.928+0000 7f313ec00640 1 -- 172.21.15.69:0/3582205240 shutdown_connections 2024-04-24T08:16:57.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:57.928+0000 7f313ec00640 1 -- 172.21.15.69:0/3582205240 wait complete. 2024-04-24T08:16:57.947 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:16:58.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.142+0000 7fd9b9200640 1 Processor -- start 2024-04-24T08:16:58.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.143+0000 7fd9b9200640 1 -- start start 2024-04-24T08:16:58.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.144+0000 7fd9b9200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9b40b3de0 0x7fd9b40732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.144+0000 7fd9b9200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd9b4073820 0x7fd9b40b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.144+0000 7fd9b9200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9b4073cc0 con 0x7fd9b415f4b0 2024-04-24T08:16:58.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.144+0000 7fd9b9200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd9b4073fa0 con 0x7fd9b40b3de0 2024-04-24T08:16:58.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.144+0000 7fd9b9200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd9b40b3300 con 0x7fd9b4073820 2024-04-24T08:16:58.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.144+0000 7fd9b1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9b40b3de0 0x7fd9b40732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.145+0000 7fd9b1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9b40b3de0 0x7fd9b40732e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43748/0 (socket says 172.21.15.69:43748) 2024-04-24T08:16:58.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.145+0000 7fd9b1400640 1 -- 172.21.15.69:0/2657593641 learned_addr learned my addr 172.21.15.69:0/2657593641 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:58.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.145+0000 7fd9b1e00640 1 --1- 172.21.15.69:0/2657593641 >> v1:172.21.15.69:6789/0 conn(0x7fd9b415f4b0 0x7fd9b415f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57980/0 (socket says 172.21.15.69:57980) 2024-04-24T08:16:58.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.145+0000 7fd9b2800640 1 --2- 172.21.15.69:0/2657593641 >> v2:172.21.15.121:3300/0 conn(0x7fd9b4073820 0x7fd9b40b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.145+0000 7fd9b1400640 1 -- 172.21.15.69:0/2657593641 >> v2:172.21.15.121:3300/0 conn(0x7fd9b4073820 msgr2=0x7fd9b40b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:58.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.145+0000 7fd9b1400640 1 --2- 172.21.15.69:0/2657593641 >> v2:172.21.15.121:3300/0 conn(0x7fd9b4073820 0x7fd9b40b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.145+0000 7fd9b1400640 1 -- 172.21.15.69:0/2657593641 >> v1:172.21.15.69:6789/0 conn(0x7fd9b415f4b0 legacy=0x7fd9b415f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.145+0000 7fd9b1400640 1 -- 172.21.15.69:0/2657593641 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b40b35e0 con 0x7fd9b40b3de0 2024-04-24T08:16:58.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.145+0000 7fd9b2800640 1 --2- 172.21.15.69:0/2657593641 >> v2:172.21.15.121:3300/0 conn(0x7fd9b4073820 0x7fd9b40b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:58.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.146+0000 7fd9b1400640 1 --2- 172.21.15.69:0/2657593641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b40b3de0 0x7fd9b40732e0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40cda60 tx=0x7fd9a402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31d9cb1adc188d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.146+0000 7fd9b0a00640 1 -- 172.21.15.69:0/2657593641 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4037070 con 0x7fd9b40b3de0 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.146+0000 7fd9b0a00640 1 -- 172.21.15.69:0/2657593641 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9a4032440 con 0x7fd9b40b3de0 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.146+0000 7fd9b0a00640 1 -- 172.21.15.69:0/2657593641 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4032810 con 0x7fd9b40b3de0 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.147+0000 7fd9b9200640 1 -- 172.21.15.69:0/2657593641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b40b3de0 msgr2=0x7fd9b40732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.147+0000 7fd9b9200640 1 --2- 172.21.15.69:0/2657593641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b40b3de0 0x7fd9b40732e0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fd9b40cda60 tx=0x7fd9a402a270 comp rx=0 tx=0).stop 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.147+0000 7fd9b9200640 1 -- 172.21.15.69:0/2657593641 shutdown_connections 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.147+0000 7fd9b9200640 1 --2- 172.21.15.69:0/2657593641 >> v2:172.21.15.121:3300/0 conn(0x7fd9b4073820 0x7fd9b40b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.147+0000 7fd9b9200640 1 --2- 172.21.15.69:0/2657593641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b40b3de0 0x7fd9b40732e0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.147+0000 7fd9b9200640 1 -- 172.21.15.69:0/2657593641 >> 172.21.15.69:0/2657593641 conn(0x7fd9b40d0350 msgr2=0x7fd9b40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.147+0000 7fd9b9200640 1 -- 172.21.15.69:0/2657593641 shutdown_connections 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.147+0000 7fd9b9200640 1 -- 172.21.15.69:0/2657593641 wait complete. 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.149+0000 7fd9b9200640 1 Processor -- start 2024-04-24T08:16:58.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.149+0000 7fd9b9200640 1 -- start start 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.150+0000 7fd9b9200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b4073820 0x7fd9b41f4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.150+0000 7fd9b9200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9b40b3de0 0x7fd9b40ad240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.150+0000 7fd9b9200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b415f4b0 0x7fd9b40ad780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.150+0000 7fd9b9200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b40b4d60 con 0x7fd9b4073820 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.150+0000 7fd9b9200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b40b4be0 con 0x7fd9b40b3de0 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.150+0000 7fd9b9200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9b40b4ee0 con 0x7fd9b415f4b0 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.150+0000 7fd9b1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b4073820 0x7fd9b41f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.151+0000 7fd9b1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b4073820 0x7fd9b41f4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43762/0 (socket says 172.21.15.69:43762) 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.151+0000 7fd9b1e00640 1 -- 172.21.15.69:0/1702529250 learned_addr learned my addr 172.21.15.69:0/1702529250 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.151+0000 7fd9b2800640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b415f4b0 0x7fd9b40ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.151+0000 7fd9b1400640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9b40b3de0 0x7fd9b40ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.151+0000 7fd9b1e00640 1 -- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b415f4b0 msgr2=0x7fd9b40ad780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.151+0000 7fd9b1e00640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b415f4b0 0x7fd9b40ad780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.151+0000 7fd9b1e00640 1 -- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9b40b3de0 msgr2=0x7fd9b40ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.151+0000 7fd9b1e00640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9b40b3de0 0x7fd9b40ad240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.151+0000 7fd9b1e00640 1 -- 172.21.15.69:0/1702529250 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b41f50a0 con 0x7fd9b4073820 2024-04-24T08:16:58.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.151+0000 7fd9b1e00640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b4073820 0x7fd9b41f4850 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c400 tx=0x7fd99c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:58.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.152+0000 7fd9aa000640 1 -- 172.21.15.69:0/1702529250 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c0074e0 con 0x7fd9b4073820 2024-04-24T08:16:58.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.152+0000 7fd9aa000640 1 -- 172.21.15.69:0/1702529250 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd99c007680 con 0x7fd9b4073820 2024-04-24T08:16:58.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.152+0000 7fd9aa000640 1 -- 172.21.15.69:0/1702529250 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99c004a00 con 0x7fd9b4073820 2024-04-24T08:16:58.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.152+0000 7fd9b9200640 1 -- 172.21.15.69:0/1702529250 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b41f5380 con 0x7fd9b4073820 2024-04-24T08:16:58.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.152+0000 7fd9b9200640 1 -- 172.21.15.69:0/1702529250 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b41e8510 con 0x7fd9b4073820 2024-04-24T08:16:58.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.153+0000 7fd9b9200640 1 -- 172.21.15.69:0/1702529250 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b41e8300 con 0x7fd9b4073820 2024-04-24T08:16:58.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.154+0000 7fd9aa000640 1 -- 172.21.15.69:0/1702529250 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd99c01a020 con 0x7fd9b4073820 2024-04-24T08:16:58.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.155+0000 7fd9aa000640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99006c8f0 0x7fd99006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.155+0000 7fd9aa000640 1 -- 172.21.15.69:0/1702529250 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7fd99c08b390 con 0x7fd9b4073820 2024-04-24T08:16:58.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.156+0000 7fd9b1400640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99006c8f0 0x7fd99006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.157+0000 7fd9b1400640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99006c8f0 0x7fd99006edb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4034000 tx=0x7fd9a402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:58.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.159+0000 7fd9aa000640 1 -- 172.21.15.69:0/1702529250 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd99c0597e0 con 0x7fd9b4073820 2024-04-24T08:16:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.357+0000 7fd9b9200640 1 -- 172.21.15.69:0/1702529250 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd9b41609d0 con 0x7fd9b4073820 2024-04-24T08:16:58.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.358+0000 7fd9aa000640 1 -- 172.21.15.69:0/1702529250 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7fd99c0590e0 con 0x7fd9b4073820 2024-04-24T08:16:58.359 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:16:58.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 -- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99006c8f0 msgr2=0x7fd99006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99006c8f0 0x7fd99006edb0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4034000 tx=0x7fd9a402e000 comp rx=0 tx=0).stop 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 -- 172.21.15.69:0/1702529250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b4073820 msgr2=0x7fd9b41f4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b4073820 0x7fd9b41f4850 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00c400 tx=0x7fd99c00c8c0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 -- 172.21.15.69:0/1702529250 shutdown_connections 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd99006c8f0 0x7fd99006edb0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b415f4b0 0x7fd9b40ad780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9b40b3de0 0x7fd9b40ad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 --2- 172.21.15.69:0/1702529250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b4073820 0x7fd9b41f4850 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 -- 172.21.15.69:0/1702529250 >> 172.21.15.69:0/1702529250 conn(0x7fd9b40d0350 msgr2=0x7fd9b415f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 -- 172.21.15.69:0/1702529250 shutdown_connections 2024-04-24T08:16:58.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.361+0000 7fd9b9200640 1 -- 172.21.15.69:0/1702529250 wait complete. 2024-04-24T08:16:58.381 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:16:58.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.582+0000 7fb37b200640 1 Processor -- start 2024-04-24T08:16:58.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.582+0000 7fb37b200640 1 -- start start 2024-04-24T08:16:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.583+0000 7fb37b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb37c0d09f0 0x7fb37c166020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.583+0000 7fb37b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb37c0ce640 con 0x7fb37c15ea60 2024-04-24T08:16:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.584+0000 7fb37b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb37c0ce910 con 0x7fb37c0b5300 2024-04-24T08:16:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.584+0000 7fb37b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb37c0cebf0 con 0x7fb37c0d09f0 2024-04-24T08:16:58.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.584+0000 7fb379e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb37c15ea60 0x7fb37c0cdf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:57986/0 (socket says 172.21.15.69:57986) 2024-04-24T08:16:58.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.584+0000 7fb379e00640 1 -- 172.21.15.69:0/1434287836 learned_addr learned my addr 172.21.15.69:0/1434287836 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:16:58.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.584+0000 7fb37a800640 1 --2- 172.21.15.69:0/1434287836 >> v2:172.21.15.69:3300/0 conn(0x7fb37c0d09f0 0x7fb37c166020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.584+0000 7fb379400640 1 -- 172.21.15.69:0/1434287836 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2502384019 0 0) 0x7fb37c0ce640 con 0x7fb37c15ea60 2024-04-24T08:16:58.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.584+0000 7fb379400640 1 -- 172.21.15.69:0/1434287836 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb360003520 con 0x7fb37c15ea60 2024-04-24T08:16:58.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.584+0000 7fb37a800640 1 -- 172.21.15.69:0/1434287836 >> v1:172.21.15.121:6789/0 conn(0x7fb37c0b5300 legacy=0x7fb37c0d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:58.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.584+0000 7fb37a800640 1 -- 172.21.15.69:0/1434287836 >> v1:172.21.15.69:6789/0 conn(0x7fb37c15ea60 legacy=0x7fb37c0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.584+0000 7fb37a800640 1 -- 172.21.15.69:0/1434287836 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37c0ceed0 con 0x7fb37c0d09f0 2024-04-24T08:16:58.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.586+0000 7fb37a800640 1 --2- 172.21.15.69:0/1434287836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 0x7fb37c166020 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fb3740097a0 tx=0x7fb37402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c6ee710f8772cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:58.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.586+0000 7fb379400640 1 -- 172.21.15.69:0/1434287836 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb374037070 con 0x7fb37c0d09f0 2024-04-24T08:16:58.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.586+0000 7fb379400640 1 -- 172.21.15.69:0/1434287836 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb374032440 con 0x7fb37c0d09f0 2024-04-24T08:16:58.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.586+0000 7fb379400640 1 -- 172.21.15.69:0/1434287836 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb374032810 con 0x7fb37c0d09f0 2024-04-24T08:16:58.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.586+0000 7fb37b200640 1 -- 172.21.15.69:0/1434287836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 msgr2=0x7fb37c166020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.586+0000 7fb37b200640 1 --2- 172.21.15.69:0/1434287836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 0x7fb37c166020 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fb3740097a0 tx=0x7fb37402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.587+0000 7fb37b200640 1 -- 172.21.15.69:0/1434287836 shutdown_connections 2024-04-24T08:16:58.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.587+0000 7fb37b200640 1 --2- 172.21.15.69:0/1434287836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 0x7fb37c166020 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.587+0000 7fb37b200640 1 -- 172.21.15.69:0/1434287836 >> 172.21.15.69:0/1434287836 conn(0x7fb37c067b80 msgr2=0x7fb37c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:58.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.587+0000 7fb37b200640 1 -- 172.21.15.69:0/1434287836 shutdown_connections 2024-04-24T08:16:58.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.587+0000 7fb37b200640 1 -- 172.21.15.69:0/1434287836 wait complete. 2024-04-24T08:16:58.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.588+0000 7fb37b200640 1 Processor -- start 2024-04-24T08:16:58.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.588+0000 7fb37b200640 1 -- start start 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.589+0000 7fb37b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb37c0b5300 0x7fb37c140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.589+0000 7fb37b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 0x7fb37c15d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.589+0000 7fb37b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb37c15ea60 0x7fb37c15db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.589+0000 7fb37b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c06ef50 con 0x7fb37c0d09f0 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.589+0000 7fb37b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c0cf490 con 0x7fb37c15ea60 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.589+0000 7fb37b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb37c06f0d0 con 0x7fb37c0b5300 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.590+0000 7fb371400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 0x7fb37c15d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.590+0000 7fb371400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 0x7fb37c15d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43766/0 (socket says 172.21.15.69:43766) 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.590+0000 7fb371400640 1 -- 172.21.15.69:0/3571631166 learned_addr learned my addr 172.21.15.69:0/3571631166 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.590+0000 7fb379e00640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb37c0b5300 0x7fb37c140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.590+0000 7fb37a800640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb37c15ea60 0x7fb37c15db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.590+0000 7fb371400640 1 -- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb37c0b5300 msgr2=0x7fb37c140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.590+0000 7fb371400640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb37c0b5300 0x7fb37c140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.590+0000 7fb371400640 1 -- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb37c15ea60 msgr2=0x7fb37c15db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.590+0000 7fb371400640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb37c15ea60 0x7fb37c15db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.590+0000 7fb371400640 1 -- 172.21.15.69:0/3571631166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37c1f3b70 con 0x7fb37c0d09f0 2024-04-24T08:16:58.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.591+0000 7fb371400640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 0x7fb37c15d620 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fb36400b3f0 tx=0x7fb36400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:58.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.591+0000 7fb372a00640 1 -- 172.21.15.69:0/3571631166 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364015020 con 0x7fb37c0d09f0 2024-04-24T08:16:58.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.591+0000 7fb372a00640 1 -- 172.21.15.69:0/3571631166 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb364011830 con 0x7fb37c0d09f0 2024-04-24T08:16:58.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.591+0000 7fb372a00640 1 -- 172.21.15.69:0/3571631166 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364011c20 con 0x7fb37c0d09f0 2024-04-24T08:16:58.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.591+0000 7fb37b200640 1 -- 172.21.15.69:0/3571631166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb37c1f3e50 con 0x7fb37c0d09f0 2024-04-24T08:16:58.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.591+0000 7fb37b200640 1 -- 172.21.15.69:0/3571631166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb37c1f4410 con 0x7fb37c0d09f0 2024-04-24T08:16:58.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.592+0000 7fb37b200640 1 -- 172.21.15.69:0/3571631166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb37c075d50 con 0x7fb37c0d09f0 2024-04-24T08:16:58.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.593+0000 7fb372a00640 1 -- 172.21.15.69:0/3571631166 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb364018020 con 0x7fb37c0d09f0 2024-04-24T08:16:58.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.594+0000 7fb372a00640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb35806c8f0 0x7fb35806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:58.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.594+0000 7fb372a00640 1 -- 172.21.15.69:0/3571631166 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7fb36408b880 con 0x7fb37c0d09f0 2024-04-24T08:16:58.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.595+0000 7fb379e00640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb35806c8f0 0x7fb35806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:58.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.596+0000 7fb379e00640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb35806c8f0 0x7fb35806edb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb36c005c00 tx=0x7fb36c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:58.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.598+0000 7fb372a00640 1 -- 172.21.15.69:0/3571631166 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb36401f030 con 0x7fb37c0d09f0 2024-04-24T08:16:58.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.792+0000 7fb37b200640 1 -- 172.21.15.69:0/3571631166 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb37c159000 con 0x7fb37c0d09f0 2024-04-24T08:16:58.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.793+0000 7fb372a00640 1 -- 172.21.15.69:0/3571631166 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7fb3640172c0 con 0x7fb37c0d09f0 2024-04-24T08:16:58.793 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 -- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb35806c8f0 msgr2=0x7fb35806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb35806c8f0 0x7fb35806edb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fb36c005c00 tx=0x7fb36c009040 comp rx=0 tx=0).stop 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 -- 172.21.15.69:0/3571631166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 msgr2=0x7fb37c15d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 0x7fb37c15d620 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fb36400b3f0 tx=0x7fb36400b8b0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 -- 172.21.15.69:0/3571631166 shutdown_connections 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb35806c8f0 0x7fb35806edb0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb37c0d09f0 0x7fb37c15d620 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb37c15ea60 0x7fb37c15db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 --2- 172.21.15.69:0/3571631166 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb37c0b5300 0x7fb37c140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 -- 172.21.15.69:0/3571631166 >> 172.21.15.69:0/3571631166 conn(0x7fb37c067b80 msgr2=0x7fb37c157ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 -- 172.21.15.69:0/3571631166 shutdown_connections 2024-04-24T08:16:58.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:58.795+0000 7fb37b200640 1 -- 172.21.15.69:0/3571631166 wait complete. 2024-04-24T08:16:58.815 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:16:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.009+0000 7fa30fe00640 1 Processor -- start 2024-04-24T08:16:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.009+0000 7fa30fe00640 1 -- start start 2024-04-24T08:16:59.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.011+0000 7fa30fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa310070e90 0x7fa31006eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.011+0000 7fa30fe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa310160020 0x7fa31015d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.011+0000 7fa30fe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3101605c0 con 0x7fa3100cb910 2024-04-24T08:16:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.011+0000 7fa30fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa31015cb80 con 0x7fa310070e90 2024-04-24T08:16:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.011+0000 7fa30fe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa3101608a0 con 0x7fa310160020 2024-04-24T08:16:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.011+0000 7fa30e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa310070e90 0x7fa31006eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.011+0000 7fa30e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa310070e90 0x7fa31006eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39500/0 (socket says 172.21.15.69:39500) 2024-04-24T08:16:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.011+0000 7fa30e000640 1 -- 172.21.15.69:0/2052204933 learned_addr learned my addr 172.21.15.69:0/2052204933 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.012+0000 7fa30e000640 1 -- 172.21.15.69:0/2052204933 >> v2:172.21.15.121:3300/0 conn(0x7fa310160020 msgr2=0x7fa31015d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.012+0000 7fa30e000640 1 --2- 172.21.15.69:0/2052204933 >> v2:172.21.15.121:3300/0 conn(0x7fa310160020 0x7fa31015d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.012+0000 7fa30e000640 1 -- 172.21.15.69:0/2052204933 >> v1:172.21.15.121:6789/0 conn(0x7fa3100cb910 legacy=0x7fa3100cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.012+0000 7fa30e000640 1 -- 172.21.15.69:0/2052204933 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa310160b80 con 0x7fa310070e90 2024-04-24T08:16:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.013+0000 7fa30e000640 1 --2- 172.21.15.69:0/2052204933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa310070e90 0x7fa31006eb20 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fa3100b0d60 tx=0x7fa2fc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da89bd06278e6809 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:59.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.013+0000 7fa30d600640 1 -- 172.21.15.69:0/2052204933 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc037070 con 0x7fa310070e90 2024-04-24T08:16:59.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.013+0000 7fa30d600640 1 -- 172.21.15.69:0/2052204933 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2fc032440 con 0x7fa310070e90 2024-04-24T08:16:59.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.013+0000 7fa30d600640 1 -- 172.21.15.69:0/2052204933 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc032810 con 0x7fa310070e90 2024-04-24T08:16:59.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.014+0000 7fa30fe00640 1 -- 172.21.15.69:0/2052204933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa310070e90 msgr2=0x7fa31006eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.014+0000 7fa30fe00640 1 --2- 172.21.15.69:0/2052204933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa310070e90 0x7fa31006eb20 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fa3100b0d60 tx=0x7fa2fc02a230 comp rx=0 tx=0).stop 2024-04-24T08:16:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.014+0000 7fa30fe00640 1 -- 172.21.15.69:0/2052204933 shutdown_connections 2024-04-24T08:16:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.014+0000 7fa30fe00640 1 --2- 172.21.15.69:0/2052204933 >> v2:172.21.15.121:3300/0 conn(0x7fa310160020 0x7fa31015d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.014+0000 7fa30fe00640 1 --2- 172.21.15.69:0/2052204933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa310070e90 0x7fa31006eb20 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.014+0000 7fa30fe00640 1 -- 172.21.15.69:0/2052204933 >> 172.21.15.69:0/2052204933 conn(0x7fa3100d12c0 msgr2=0x7fa3100d16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.014+0000 7fa30fe00640 1 -- 172.21.15.69:0/2052204933 shutdown_connections 2024-04-24T08:16:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.014+0000 7fa30fe00640 1 -- 172.21.15.69:0/2052204933 wait complete. 2024-04-24T08:16:59.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.015+0000 7fa30fe00640 1 Processor -- start 2024-04-24T08:16:59.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.016+0000 7fa30fe00640 1 -- start start 2024-04-24T08:16:59.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.016+0000 7fa30fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa310070e90 0x7fa3100b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.016+0000 7fa30fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3100cb910 0x7fa3100b6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa310160020 0x7fa3100729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3100cda90 con 0x7fa3100cb910 2024-04-24T08:16:59.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30fe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa3100cd910 con 0x7fa310160020 2024-04-24T08:16:59.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa31006f060 con 0x7fa310070e90 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3100cb910 0x7fa3100b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3100cb910 0x7fa3100b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39508/0 (socket says 172.21.15.69:39508) 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30e000640 1 -- 172.21.15.69:0/343026413 learned_addr learned my addr 172.21.15.69:0/343026413 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30e000640 1 -- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa310070e90 msgr2=0x7fa3100b4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30e000640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa310070e90 0x7fa3100b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30e000640 1 -- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa310160020 msgr2=0x7fa3100729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30e000640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa310160020 0x7fa3100729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.017+0000 7fa30e000640 1 -- 172.21.15.69:0/343026413 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3100b5e70 con 0x7fa3100cb910 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.018+0000 7fa30e000640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3100cb910 0x7fa3100b6670 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009960 tx=0x7fa2fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.018+0000 7fa306a00640 1 -- 172.21.15.69:0/343026413 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc041070 con 0x7fa3100cb910 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.018+0000 7fa306a00640 1 -- 172.21.15.69:0/343026413 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2fc03c950 con 0x7fa3100cb910 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.018+0000 7fa306a00640 1 -- 172.21.15.69:0/343026413 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc037040 con 0x7fa3100cb910 2024-04-24T08:16:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.018+0000 7fa30fe00640 1 -- 172.21.15.69:0/343026413 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3100b6150 con 0x7fa3100cb910 2024-04-24T08:16:59.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.018+0000 7fa30fe00640 1 -- 172.21.15.69:0/343026413 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3100b4220 con 0x7fa3100cb910 2024-04-24T08:16:59.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.019+0000 7fa30fe00640 1 -- 172.21.15.69:0/343026413 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa310075d50 con 0x7fa3100cb910 2024-04-24T08:16:59.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.020+0000 7fa306a00640 1 -- 172.21.15.69:0/343026413 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa2fc044070 con 0x7fa3100cb910 2024-04-24T08:16:59.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.021+0000 7fa306a00640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2f006c8f0 0x7fa2f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.021+0000 7fa306a00640 1 -- 172.21.15.69:0/343026413 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7fa2fc0ac820 con 0x7fa3100cb910 2024-04-24T08:16:59.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.022+0000 7fa30ea00640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2f006c8f0 0x7fa2f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.023+0000 7fa30ea00640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2f006c8f0 0x7fa2f006edb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa300009650 tx=0x7fa300006ba0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:59.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.025+0000 7fa306a00640 1 -- 172.21.15.69:0/343026413 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa2fc07ad10 con 0x7fa3100cb910 2024-04-24T08:16:59.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.239+0000 7fa30fe00640 1 -- 172.21.15.69:0/343026413 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa3100c9650 con 0x7fa3100cb910 2024-04-24T08:16:59.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.240+0000 7fa306a00640 1 -- 172.21.15.69:0/343026413 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v85) ==== 80+0+124 (secure 0 0 0) 0x7fa2fc07a610 con 0x7fa3100cb910 2024-04-24T08:16:59.242 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6837/1179683245 2024-04-25T08:15:45.259762+0000 2024-04-24T08:16:59.242 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6835/1179683245 2024-04-25T08:15:45.259762+0000 2024-04-24T08:16:59.242 INFO:teuthology.orchestra.run.smithi069.stderr:listed 2 entries 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 -- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2f006c8f0 msgr2=0x7fa2f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2f006c8f0 0x7fa2f006edb0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa300009650 tx=0x7fa300006ba0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 -- 172.21.15.69:0/343026413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3100cb910 msgr2=0x7fa3100b6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3100cb910 0x7fa3100b6670 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc009960 tx=0x7fa2fc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 -- 172.21.15.69:0/343026413 shutdown_connections 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa2f006c8f0 0x7fa2f006edb0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3100cb910 0x7fa3100b6670 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa310160020 0x7fa3100729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 --2- 172.21.15.69:0/343026413 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa310070e90 0x7fa3100b4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 -- 172.21.15.69:0/343026413 >> 172.21.15.69:0/343026413 conn(0x7fa3100d12c0 msgr2=0x7fa3100cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 -- 172.21.15.69:0/343026413 shutdown_connections 2024-04-24T08:16:59.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.243+0000 7fa30fe00640 1 -- 172.21.15.69:0/343026413 wait complete. 2024-04-24T08:16:59.264 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:16:59.264 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:16:59.312 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:16:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.558+0000 7f04a7200640 1 Processor -- start 2024-04-24T08:16:59.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.558+0000 7f04a7200640 1 -- start start 2024-04-24T08:16:59.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.560+0000 7f04a7200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04a815d1d0 0x7f04a80ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.560+0000 7f04a7200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04a815d660 con 0x7f04a8156c60 2024-04-24T08:16:59.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.560+0000 7f04a7200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04a806dc30 con 0x7f04a80b30f0 2024-04-24T08:16:59.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.560+0000 7f04a7200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f04a806df10 con 0x7f04a815d1d0 2024-04-24T08:16:59.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.560+0000 7f04a5e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f04a8156c60 0x7f04a815cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55938/0 (socket says 172.21.15.69:55938) 2024-04-24T08:16:59.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.560+0000 7f04a5e00640 1 -- 172.21.15.69:0/3907166624 learned_addr learned my addr 172.21.15.69:0/3907166624 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:16:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f049d400640 1 --2- 172.21.15.69:0/3907166624 >> v2:172.21.15.121:3300/0 conn(0x7f04a815d1d0 0x7f04a80ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2804409341 0 0) 0x7f04a815d660 con 0x7f04a8156c60 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f048c003520 con 0x7f04a8156c60 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1228972146 0 0) 0x7f048c003520 con 0x7f04a8156c60 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 >> v2:172.21.15.121:3300/0 conn(0x7f04a815d1d0 msgr2=0x7f04a80ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 --2- 172.21.15.69:0/3907166624 >> v2:172.21.15.121:3300/0 conn(0x7f04a815d1d0 0x7f04a80ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 >> v1:172.21.15.121:6789/0 conn(0x7f04a80b30f0 legacy=0x7f04a806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a806e1f0 con 0x7f04a8156c60 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f0498001670 con 0x7f04a8156c60 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 >> v1:172.21.15.69:6789/0 conn(0x7f04a8156c60 legacy=0x7f04a815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f048c0039b0 0x7f048c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048c004800 0x7f048c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c0051c0 0x7f048c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04a806df10 con 0x7f048c0051c0 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f048c00c060 con 0x7f048c0039b0 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.561+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f048c00c260 con 0x7f048c004800 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.562+0000 7f04a6800640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c0051c0 0x7f048c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.562+0000 7f04a5e00640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f048c0039b0 0x7f048c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.562+0000 7f049d400640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048c004800 0x7f048c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.562+0000 7f04a6800640 1 -- 172.21.15.69:0/3907166624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048c004800 msgr2=0x7f048c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.562+0000 7f04a6800640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048c004800 0x7f048c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.562+0000 7f04a6800640 1 -- 172.21.15.69:0/3907166624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f048c0039b0 msgr2=0x7f048c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.562+0000 7f04a6800640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f048c0039b0 0x7f048c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.563+0000 7f04a6800640 1 -- 172.21.15.69:0/3907166624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f048c003520 con 0x7f048c0051c0 2024-04-24T08:16:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.564+0000 7f04a6800640 1 -- 172.21.15.69:0/3907166624 reap_dead start 2024-04-24T08:16:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.564+0000 7f04a6800640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c0051c0 0x7f048c00b9c0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f04a80d11e0 tx=0x7f048c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9b14a7fe685fd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.564+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0001a70 con 0x7f048c0051c0 2024-04-24T08:16:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.564+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a002ae10 con 0x7f048c0051c0 2024-04-24T08:16:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.564+0000 7f04a5400640 1 -- 172.21.15.69:0/3907166624 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0003bb0 con 0x7f048c0051c0 2024-04-24T08:16:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.565+0000 7f04a7200640 1 -- 172.21.15.69:0/3907166624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c0051c0 msgr2=0x7f048c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.565+0000 7f04a7200640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c0051c0 0x7f048c00b9c0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f04a80d11e0 tx=0x7f048c005830 comp rx=0 tx=0).stop 2024-04-24T08:16:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.565+0000 7f04a7200640 1 -- 172.21.15.69:0/3907166624 shutdown_connections 2024-04-24T08:16:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.565+0000 7f04a7200640 1 --2- 172.21.15.69:0/3907166624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f048c0051c0 0x7f048c00b9c0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.565+0000 7f04a7200640 1 -- 172.21.15.69:0/3907166624 >> 172.21.15.69:0/3907166624 conn(0x7f04a80d0160 msgr2=0x7f04a80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.565+0000 7f04a7200640 1 -- 172.21.15.69:0/3907166624 shutdown_connections 2024-04-24T08:16:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.565+0000 7f04a7200640 1 -- 172.21.15.69:0/3907166624 wait complete. 2024-04-24T08:16:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.567+0000 7f04a7200640 1 Processor -- start 2024-04-24T08:16:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.567+0000 7f04a7200640 1 -- start start 2024-04-24T08:16:59.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f04a7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048c0051c0 0x7f04a81f7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f04a7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a80ca640 0x7f04a8154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f04a7200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a8154f60 0x7f04a80cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f04a7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04a815cac0 con 0x7f04a80ca640 2024-04-24T08:16:59.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f04a7200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04a815d940 con 0x7f04a8154f60 2024-04-24T08:16:59.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f04a7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04a815cc40 con 0x7f048c0051c0 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f049d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a80ca640 0x7f04a8154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f049d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a80ca640 0x7f04a8154a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39538/0 (socket says 172.21.15.69:39538) 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f049d400640 1 -- 172.21.15.69:0/3763159439 learned_addr learned my addr 172.21.15.69:0/3763159439 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f04a5e00640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048c0051c0 0x7f04a81f7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f049d400640 1 -- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048c0051c0 msgr2=0x7f04a81f7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f049d400640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048c0051c0 0x7f04a81f7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f04a6800640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a8154f60 0x7f04a80cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f049d400640 1 -- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a8154f60 msgr2=0x7f04a80cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f049d400640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a8154f60 0x7f04a80cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.568+0000 7f049d400640 1 -- 172.21.15.69:0/3763159439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a80c95a0 con 0x7f04a80ca640 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.569+0000 7f049d400640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a80ca640 0x7f04a8154a20 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f0490011910 tx=0x7f0490011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:59.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.569+0000 7f049ea00640 1 -- 172.21.15.69:0/3763159439 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049000faf0 con 0x7f04a80ca640 2024-04-24T08:16:59.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.569+0000 7f049ea00640 1 -- 172.21.15.69:0/3763159439 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049000fc90 con 0x7f04a80ca640 2024-04-24T08:16:59.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.569+0000 7f049ea00640 1 -- 172.21.15.69:0/3763159439 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490014440 con 0x7f04a80ca640 2024-04-24T08:16:59.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.569+0000 7f04a7200640 1 -- 172.21.15.69:0/3763159439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a80c9880 con 0x7f04a80ca640 2024-04-24T08:16:59.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.569+0000 7f04a7200640 1 -- 172.21.15.69:0/3763159439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a80c9e40 con 0x7f04a80ca640 2024-04-24T08:16:59.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.570+0000 7f04a7200640 1 -- 172.21.15.69:0/3763159439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a80ad0d0 con 0x7f04a80ca640 2024-04-24T08:16:59.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.571+0000 7f049ea00640 1 -- 172.21.15.69:0/3763159439 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0490012040 con 0x7f04a80ca640 2024-04-24T08:16:59.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.572+0000 7f049ea00640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 0x7f048406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:59.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.572+0000 7f049ea00640 1 -- 172.21.15.69:0/3763159439 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f049008c220 con 0x7f04a80ca640 2024-04-24T08:16:59.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.573+0000 7f04a5e00640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 0x7f048406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:59.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.574+0000 7f04a5e00640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 0x7f048406edb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0498007cb0 tx=0x7f0498009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:59.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.577+0000 7f049ea00640 1 -- 172.21.15.69:0/3763159439 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04a80ad0d0 con 0x7f04a80ca640 2024-04-24T08:16:59.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.780+0000 7f04a7200640 1 -- 172.21.15.69:0/3763159439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04a80aeba0 con 0x7f04a80ca640 2024-04-24T08:16:59.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.781+0000 7f049ea00640 1 -- 172.21.15.69:0/3763159439 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+16817 (secure 0 0 0) 0x7f04a80aeba0 con 0x7f04a80ca640 2024-04-24T08:16:59.784 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:16:59.784 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":85,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:16:03.587918+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6837/1179683245":"2024-04-25T08:15:45.259762+0000","172.21.15.121:6835/1179683245":"2024-04-25T08:15:45.259762+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:16:59.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.784+0000 7f04a7200640 1 -- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 msgr2=0x7f048406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.784+0000 7f04a7200640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 0x7f048406edb0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0498007cb0 tx=0x7f0498009040 comp rx=0 tx=0).stop 2024-04-24T08:16:59.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.784+0000 7f04a7200640 1 -- 172.21.15.69:0/3763159439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a80ca640 msgr2=0x7f04a8154a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:59.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.784+0000 7f04a7200640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a80ca640 0x7f04a8154a20 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f0490011910 tx=0x7f0490011dd0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.784+0000 7f04a7200640 1 -- 172.21.15.69:0/3763159439 shutdown_connections 2024-04-24T08:16:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.784+0000 7f04a7200640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 0x7f048406edb0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.784+0000 7f04a7200640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04a80ca640 0x7f04a8154a20 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.784+0000 7f04a7200640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a8154f60 0x7f04a80cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.784+0000 7f04a7200640 1 --2- 172.21.15.69:0/3763159439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f048c0051c0 0x7f04a81f7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.784+0000 7f04a7200640 1 -- 172.21.15.69:0/3763159439 >> 172.21.15.69:0/3763159439 conn(0x7f04a80d0160 msgr2=0x7f04a80b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.785+0000 7f04a7200640 1 -- 172.21.15.69:0/3763159439 shutdown_connections 2024-04-24T08:16:59.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:16:59.785+0000 7f04a7200640 1 -- 172.21.15.69:0/3763159439 wait complete. 2024-04-24T08:16:59.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:15:21.126081+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:15:22.402310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:16:59.806 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:17:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.002+0000 7f60f3800640 1 Processor -- start 2024-04-24T08:17:00.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.003+0000 7f60f3800640 1 -- start start 2024-04-24T08:17:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.004+0000 7f60f3800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f60ec15c820 0x7f60ec0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.004+0000 7f60f3800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60ec15cc00 con 0x7f60ec0ce250 2024-04-24T08:17:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.004+0000 7f60f3800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60ec06ed30 con 0x7f60ec0b46a0 2024-04-24T08:17:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.004+0000 7f60f3800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f60ec0b4e10 con 0x7f60ec15c820 2024-04-24T08:17:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.005+0000 7f60f1800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f60ec0ce250 0x7f60ec0ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55952/0 (socket says 172.21.15.69:55952) 2024-04-24T08:17:00.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.005+0000 7f60f1800640 1 -- 172.21.15.69:0/1836801906 learned_addr learned my addr 172.21.15.69:0/1836801906 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.005+0000 7f60f0e00640 1 --2- 172.21.15.69:0/1836801906 >> v2:172.21.15.121:3300/0 conn(0x7f60ec15c820 0x7f60ec0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.005+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893596319 0 0) 0x7f60ec15cc00 con 0x7f60ec0ce250 2024-04-24T08:17:00.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.005+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60d4003520 con 0x7f60ec0ce250 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.005+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4107043859 0 0) 0x7f60d4003520 con 0x7f60ec0ce250 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.005+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 >> v2:172.21.15.121:3300/0 conn(0x7f60ec15c820 msgr2=0x7f60ec0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.005+0000 7f60eb400640 1 --2- 172.21.15.69:0/1836801906 >> v2:172.21.15.121:3300/0 conn(0x7f60ec15c820 0x7f60ec0b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.005+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 >> v1:172.21.15.121:6789/0 conn(0x7f60ec0b46a0 legacy=0x7f60ec06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.006+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ec0b50f0 con 0x7f60ec0ce250 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.006+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f60e40019f0 con 0x7f60ec0ce250 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.006+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 >> v1:172.21.15.69:6789/0 conn(0x7f60ec0ce250 legacy=0x7f60ec0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.006+0000 7f60eb400640 1 --2- 172.21.15.69:0/1836801906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60d40039b0 0x7f60d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.006+0000 7f60eb400640 1 --2- 172.21.15.69:0/1836801906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.006+0000 7f60eb400640 1 --2- 172.21.15.69:0/1836801906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60d40051c0 0x7f60d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.006+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec0b4e10 con 0x7f60d4004800 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.006+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f60d400c110 con 0x7f60d40039b0 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.006+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f60d400c310 con 0x7f60d40051c0 2024-04-24T08:17:00.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.006+0000 7f60ebe00640 1 --2- 172.21.15.69:0/1836801906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.007+0000 7f60ebe00640 1 -- 172.21.15.69:0/1836801906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60d40051c0 msgr2=0x7f60d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:00.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.007+0000 7f60f1800640 1 --2- 172.21.15.69:0/1836801906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60d40051c0 0x7f60d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.007+0000 7f60ebe00640 1 --2- 172.21.15.69:0/1836801906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60d40051c0 0x7f60d400b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.007+0000 7f60ebe00640 1 -- 172.21.15.69:0/1836801906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60d40039b0 msgr2=0x7f60d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:00.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.007+0000 7f60ebe00640 1 --2- 172.21.15.69:0/1836801906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60d40039b0 0x7f60d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.007+0000 7f60ebe00640 1 -- 172.21.15.69:0/1836801906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d4003520 con 0x7f60d4004800 2024-04-24T08:17:00.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.007+0000 7f60f1800640 1 -- 172.21.15.69:0/1836801906 reap_dead start 2024-04-24T08:17:00.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.008+0000 7f60ebe00640 1 --2- 172.21.15.69:0/1836801906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60d4004c80 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f60d8003db0 tx=0x7f60d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c942e83f6e1def server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:00.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.008+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d8037070 con 0x7f60d4004800 2024-04-24T08:17:00.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.008+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d802a9f0 con 0x7f60d4004800 2024-04-24T08:17:00.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.008+0000 7f60eb400640 1 -- 172.21.15.69:0/1836801906 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d8003980 con 0x7f60d4004800 2024-04-24T08:17:00.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.009+0000 7f60f3800640 1 -- 172.21.15.69:0/1836801906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 msgr2=0x7f60d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.009+0000 7f60f3800640 1 --2- 172.21.15.69:0/1836801906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60d4004c80 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f60d8003db0 tx=0x7f60d4005830 comp rx=0 tx=0).stop 2024-04-24T08:17:00.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.009+0000 7f60f3800640 1 -- 172.21.15.69:0/1836801906 shutdown_connections 2024-04-24T08:17:00.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.009+0000 7f60f3800640 1 --2- 172.21.15.69:0/1836801906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60d4004c80 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.009+0000 7f60f3800640 1 -- 172.21.15.69:0/1836801906 >> 172.21.15.69:0/1836801906 conn(0x7f60ec0d0350 msgr2=0x7f60ec0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:00.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.009+0000 7f60f3800640 1 -- 172.21.15.69:0/1836801906 shutdown_connections 2024-04-24T08:17:00.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.009+0000 7f60f3800640 1 -- 172.21.15.69:0/1836801906 wait complete. 2024-04-24T08:17:00.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.011+0000 7f60f3800640 1 Processor -- start 2024-04-24T08:17:00.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.011+0000 7f60f3800640 1 -- start start 2024-04-24T08:17:00.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.012+0000 7f60f3800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60ec1f8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.012+0000 7f60f3800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60ec1505c0 0x7f60ec0ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.012+0000 7f60f3800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60ec0ae900 0x7f60ec0c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.012+0000 7f60f3800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec0b4b30 con 0x7f60d4004800 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.012+0000 7f60f3800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f60ec06f2b0 con 0x7f60ec1505c0 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.012+0000 7f60f3800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f60ec0b3e90 con 0x7f60ec0ae900 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.013+0000 7f60f0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60ec1f8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.013+0000 7f60f0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60ec1f8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39566/0 (socket says 172.21.15.69:39566) 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.013+0000 7f60f0e00640 1 -- 172.21.15.69:0/3034623383 learned_addr learned my addr 172.21.15.69:0/3034623383 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.013+0000 7f60ebe00640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60ec1505c0 0x7f60ec0ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.013+0000 7f60f0e00640 1 -- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60ec0ae900 msgr2=0x7f60ec0c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.013+0000 7f60f0e00640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60ec0ae900 0x7f60ec0c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.013+0000 7f60f0e00640 1 -- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60ec1505c0 msgr2=0x7f60ec0ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.013+0000 7f60f0e00640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60ec1505c0 0x7f60ec0ae3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.013+0000 7f60f0e00640 1 -- 172.21.15.69:0/3034623383 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60ec150a40 con 0x7f60d4004800 2024-04-24T08:17:00.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.013+0000 7f60f0e00640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60ec1f8dd0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f60e000d590 tx=0x7f60e000da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:00.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.014+0000 7f60e8c00640 1 -- 172.21.15.69:0/3034623383 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e00125f0 con 0x7f60d4004800 2024-04-24T08:17:00.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.014+0000 7f60e8c00640 1 -- 172.21.15.69:0/3034623383 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60e0019070 con 0x7f60d4004800 2024-04-24T08:17:00.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.014+0000 7f60e8c00640 1 -- 172.21.15.69:0/3034623383 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e00129d0 con 0x7f60d4004800 2024-04-24T08:17:00.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.014+0000 7f60f3800640 1 -- 172.21.15.69:0/3034623383 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60ec0720c0 con 0x7f60d4004800 2024-04-24T08:17:00.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.014+0000 7f60f3800640 1 -- 172.21.15.69:0/3034623383 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60ec072680 con 0x7f60d4004800 2024-04-24T08:17:00.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.015+0000 7f60f3800640 1 -- 172.21.15.69:0/3034623383 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60ec0aed20 con 0x7f60d4004800 2024-04-24T08:17:00.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.016+0000 7f60e8c00640 1 -- 172.21.15.69:0/3034623383 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f60e0012b70 con 0x7f60d4004800 2024-04-24T08:17:00.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.017+0000 7f60e8c00640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f60cc06c8f0 0x7f60cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.017+0000 7f60e8c00640 1 -- 172.21.15.69:0/3034623383 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f60e008bf50 con 0x7f60d4004800 2024-04-24T08:17:00.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.018+0000 7f60ebe00640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f60cc06c8f0 0x7f60cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.019+0000 7f60ebe00640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f60cc06c8f0 0x7f60cc06edb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f60d8034000 tx=0x7f60d802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:00.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.022+0000 7f60e8c00640 1 -- 172.21.15.69:0/3034623383 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f60e005a290 con 0x7f60d4004800 2024-04-24T08:17:00.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.219+0000 7f60f3800640 1 -- 172.21.15.69:0/3034623383 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f60ec068a80 con 0x7f60d4004800 2024-04-24T08:17:00.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.220+0000 7f60e8c00640 1 -- 172.21.15.69:0/3034623383 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f60e0059b90 con 0x7f60d4004800 2024-04-24T08:17:00.222 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 -- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f60cc06c8f0 msgr2=0x7f60cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f60cc06c8f0 0x7f60cc06edb0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f60d8034000 tx=0x7f60d802e000 comp rx=0 tx=0).stop 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 -- 172.21.15.69:0/3034623383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 msgr2=0x7f60ec1f8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60ec1f8dd0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f60e000d590 tx=0x7f60e000da50 comp rx=0 tx=0).stop 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 -- 172.21.15.69:0/3034623383 shutdown_connections 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f60cc06c8f0 0x7f60cc06edb0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f60ec0ae900 0x7f60ec0c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60ec1505c0 0x7f60ec0ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 --2- 172.21.15.69:0/3034623383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60d4004800 0x7f60ec1f8dd0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 -- 172.21.15.69:0/3034623383 >> 172.21.15.69:0/3034623383 conn(0x7f60ec0d0350 msgr2=0x7f60ec0b0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:00.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 -- 172.21.15.69:0/3034623383 shutdown_connections 2024-04-24T08:17:00.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.223+0000 7f60f3800640 1 -- 172.21.15.69:0/3034623383 wait complete. 2024-04-24T08:17:00.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:17:00.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.447+0000 7f5232e00640 1 Processor -- start 2024-04-24T08:17:00.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.448+0000 7f5232e00640 1 -- start start 2024-04-24T08:17:00.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.449+0000 7f5232e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522c06cfb0 0x7f522c06d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.449+0000 7f5232e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f522c15ed60 con 0x7f522c162f90 2024-04-24T08:17:00.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.449+0000 7f5232e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f522c0d1580 con 0x7f522c15e980 2024-04-24T08:17:00.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.449+0000 7f5232e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f522c1633d0 con 0x7f522c06cfb0 2024-04-24T08:17:00.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.449+0000 7f5231000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f522c162f90 0x7f522c0cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55958/0 (socket says 172.21.15.69:55958) 2024-04-24T08:17:00.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.449+0000 7f5231000640 1 -- 172.21.15.69:0/3170975487 learned_addr learned my addr 172.21.15.69:0/3170975487 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:00.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f5232400640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522c06cfb0 0x7f522c06d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 911387579 0 0) 0x7f522c15ed60 con 0x7f522c162f90 2024-04-24T08:17:00.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5210003520 con 0x7f522c162f90 2024-04-24T08:17:00.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2052832599 0 0) 0x7f5210003520 con 0x7f522c162f90 2024-04-24T08:17:00.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522c06cfb0 msgr2=0x7f522c06d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:00.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f522be00640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522c06cfb0 0x7f522c06d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 >> v1:172.21.15.121:6789/0 conn(0x7f522c15e980 legacy=0x7f522c0d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522c0d1860 con 0x7f522c162f90 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f52180019f0 con 0x7f522c162f90 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 >> v1:172.21.15.69:6789/0 conn(0x7f522c162f90 legacy=0x7f522c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.450+0000 7f522be00640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522c06cfb0 0x7f52100039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f522be00640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5210003f60 0x7f5210004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f522be00640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5210004d40 0x7f52100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f522c1633d0 con 0x7f5210003f60 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f521000bb30 con 0x7f5210004d40 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f521000bd30 con 0x7f522c06cfb0 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f5231000640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5210003f60 0x7f5210004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f5231a00640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522c06cfb0 0x7f52100039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f5231000640 1 -- 172.21.15.69:0/3170975487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522c06cfb0 msgr2=0x7f52100039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f5231000640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f522c06cfb0 0x7f52100039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f5231000640 1 -- 172.21.15.69:0/3170975487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5210004d40 msgr2=0x7f52100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:00.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f5231000640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5210004d40 0x7f52100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.451+0000 7f5231000640 1 -- 172.21.15.69:0/3170975487 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5210003520 con 0x7f5210003f60 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.452+0000 7f5231000640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5210003f60 0x7f5210004800 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f521800a850 tx=0x7f5210005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b8e3222bafcfb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.453+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521800a4a0 con 0x7f5210003f60 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.453+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f521802f070 con 0x7f5210003f60 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.453+0000 7f522be00640 1 -- 172.21.15.69:0/3170975487 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521802bd20 con 0x7f5210003f60 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.453+0000 7f5232e00640 1 -- 172.21.15.69:0/3170975487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5210003f60 msgr2=0x7f5210004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.453+0000 7f5232e00640 1 --2- 172.21.15.69:0/3170975487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5210003f60 0x7f5210004800 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f521800a850 tx=0x7f5210005830 comp rx=0 tx=0).stop 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.453+0000 7f5232400640 1 -- 172.21.15.69:0/3170975487 reap_dead start 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.453+0000 7f5232e00640 1 -- 172.21.15.69:0/3170975487 shutdown_connections 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.453+0000 7f5232e00640 1 -- 172.21.15.69:0/3170975487 >> 172.21.15.69:0/3170975487 conn(0x7f522c09d920 msgr2=0x7f522c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.453+0000 7f5232e00640 1 -- 172.21.15.69:0/3170975487 shutdown_connections 2024-04-24T08:17:00.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.453+0000 7f5232e00640 1 -- 172.21.15.69:0/3170975487 wait complete. 2024-04-24T08:17:00.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.455+0000 7f5232e00640 1 Processor -- start 2024-04-24T08:17:00.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.455+0000 7f5232e00640 1 -- start start 2024-04-24T08:17:00.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.456+0000 7f5232e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5210003f60 0x7f522c152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.456+0000 7f5232e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f522c0b9e00 0x7f522c0b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.456+0000 7f5232e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c0b44b0 0x7f522c0b4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.456+0000 7f5232e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f522c0d0360 con 0x7f522c0b44b0 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.456+0000 7f5232e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f522c0cf730 con 0x7f522c0b9e00 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.456+0000 7f5232e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f522c0cf070 con 0x7f5210003f60 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.456+0000 7f5232400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c0b44b0 0x7f522c0b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.457+0000 7f5232400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c0b44b0 0x7f522c0b4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39580/0 (socket says 172.21.15.69:39580) 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.457+0000 7f5232400640 1 -- 172.21.15.69:0/4201569202 learned_addr learned my addr 172.21.15.69:0/4201569202 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.457+0000 7f5231000640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f522c0b9e00 0x7f522c0b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.457+0000 7f5231a00640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5210003f60 0x7f522c152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.457+0000 7f5232400640 1 -- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5210003f60 msgr2=0x7f522c152b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.457+0000 7f5232400640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5210003f60 0x7f522c152b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.457+0000 7f5232400640 1 -- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f522c0b9e00 msgr2=0x7f522c0b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.457+0000 7f5232400640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f522c0b9e00 0x7f522c0b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.457+0000 7f5232400640 1 -- 172.21.15.69:0/4201569202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522c15ed60 con 0x7f522c0b44b0 2024-04-24T08:17:00.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.457+0000 7f5232400640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c0b44b0 0x7f522c0b4940 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f522c06c480 tx=0x7f522400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:00.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.458+0000 7f5229600640 1 -- 172.21.15.69:0/4201569202 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522400c680 con 0x7f522c0b44b0 2024-04-24T08:17:00.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.458+0000 7f5229600640 1 -- 172.21.15.69:0/4201569202 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f522401a070 con 0x7f522c0b44b0 2024-04-24T08:17:00.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.458+0000 7f5229600640 1 -- 172.21.15.69:0/4201569202 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522400ca60 con 0x7f522c0b44b0 2024-04-24T08:17:00.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.458+0000 7f5232e00640 1 -- 172.21.15.69:0/4201569202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522c0cdb90 con 0x7f522c0b44b0 2024-04-24T08:17:00.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.458+0000 7f5232e00640 1 -- 172.21.15.69:0/4201569202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f522c0b80a0 con 0x7f522c0b44b0 2024-04-24T08:17:00.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.459+0000 7f5232e00640 1 -- 172.21.15.69:0/4201569202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f522c075d50 con 0x7f522c0b44b0 2024-04-24T08:17:00.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.460+0000 7f5229600640 1 -- 172.21.15.69:0/4201569202 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f522400cc00 con 0x7f522c0b44b0 2024-04-24T08:17:00.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.461+0000 7f5229600640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f520c06c920 0x7f520c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.461+0000 7f5229600640 1 -- 172.21.15.69:0/4201569202 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f522408b440 con 0x7f522c0b44b0 2024-04-24T08:17:00.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.462+0000 7f5231a00640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f520c06c920 0x7f520c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.463+0000 7f5231a00640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f520c06c920 0x7f520c06ede0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f522c0b7710 tx=0x7f521c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:00.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.465+0000 7f5229600640 1 -- 172.21.15.69:0/4201569202 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5224023020 con 0x7f522c0b44b0 2024-04-24T08:17:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.665+0000 7f5232e00640 1 -- 172.21.15.69:0/4201569202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f522c15ca90 con 0x7f522c0b44b0 2024-04-24T08:17:00.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.666+0000 7f5229600640 1 -- 172.21.15.69:0/4201569202 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f5224059780 con 0x7f522c0b44b0 2024-04-24T08:17:00.667 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:17:00.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 -- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f520c06c920 msgr2=0x7f520c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f520c06c920 0x7f520c06ede0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f522c0b7710 tx=0x7f521c009040 comp rx=0 tx=0).stop 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 -- 172.21.15.69:0/4201569202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c0b44b0 msgr2=0x7f522c0b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c0b44b0 0x7f522c0b4940 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f522c06c480 tx=0x7f522400ec00 comp rx=0 tx=0).stop 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 -- 172.21.15.69:0/4201569202 shutdown_connections 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f520c06c920 0x7f520c06ede0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f522c0b44b0 0x7f522c0b4940 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f522c0b9e00 0x7f522c0b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 --2- 172.21.15.69:0/4201569202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5210003f60 0x7f522c152b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 -- 172.21.15.69:0/4201569202 >> 172.21.15.69:0/4201569202 conn(0x7f522c09d920 msgr2=0x7f522c0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 -- 172.21.15.69:0/4201569202 shutdown_connections 2024-04-24T08:17:00.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.669+0000 7f5232e00640 1 -- 172.21.15.69:0/4201569202 wait complete. 2024-04-24T08:17:00.690 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:17:00.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.886+0000 7fe742600640 1 Processor -- start 2024-04-24T08:17:00.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.886+0000 7fe742600640 1 -- start start 2024-04-24T08:17:00.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.888+0000 7fe742600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe73c15e980 0x7fe73c162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.888+0000 7fe742600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe73c15ed60 con 0x7fe73c06cfb0 2024-04-24T08:17:00.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.888+0000 7fe742600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe73c0d1580 con 0x7fe73c0d0da0 2024-04-24T08:17:00.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.888+0000 7fe742600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe73c0d1860 con 0x7fe73c15e980 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.888+0000 7fe741c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe73c06cfb0 0x7fe73c0d29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55974/0 (socket says 172.21.15.69:55974) 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.888+0000 7fe741c00640 1 -- 172.21.15.69:0/1610830730 learned_addr learned my addr 172.21.15.69:0/1610830730 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe741200640 1 --2- 172.21.15.69:0/1610830730 >> v2:172.21.15.121:3300/0 conn(0x7fe73c15e980 0x7fe73c162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1057228706 0 0) 0x7fe73c15ed60 con 0x7fe73c06cfb0 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe724003520 con 0x7fe73c06cfb0 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 665518954 0 0) 0x7fe724003520 con 0x7fe73c06cfb0 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 >> v2:172.21.15.121:3300/0 conn(0x7fe73c15e980 msgr2=0x7fe73c162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 --2- 172.21.15.69:0/1610830730 >> v2:172.21.15.121:3300/0 conn(0x7fe73c15e980 0x7fe73c162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 >> v1:172.21.15.121:6789/0 conn(0x7fe73c0d0da0 legacy=0x7fe73c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73c0cf450 con 0x7fe73c06cfb0 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe7340019f0 con 0x7fe73c06cfb0 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 >> v1:172.21.15.69:6789/0 conn(0x7fe73c06cfb0 legacy=0x7fe73c0d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7240039b0 0x7fe724003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe724004800 0x7fe724004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7240051c0 0x7fe72400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c0d1860 con 0x7fe7240051c0 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe72400c070 con 0x7fe724004800 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.889+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe72400c270 con 0x7fe7240039b0 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.890+0000 7fe741c00640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7240051c0 0x7fe72400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.890+0000 7fe741200640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7240039b0 0x7fe724003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.890+0000 7fe73be00640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe724004800 0x7fe724004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.890+0000 7fe741c00640 1 -- 172.21.15.69:0/1610830730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7240039b0 msgr2=0x7fe724003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.890+0000 7fe741c00640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7240039b0 0x7fe724003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.890+0000 7fe741c00640 1 -- 172.21.15.69:0/1610830730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe724004800 msgr2=0x7fe724004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.890+0000 7fe741c00640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe724004800 0x7fe724004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.890+0000 7fe741c00640 1 -- 172.21.15.69:0/1610830730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe724003520 con 0x7fe7240051c0 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.891+0000 7fe741c00640 1 -- 172.21.15.69:0/1610830730 reap_dead start 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.891+0000 7fe741c00640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7240051c0 0x7fe72400b9c0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fe73400a860 tx=0x7fe724005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80069d8e648b83e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.891+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73400a480 con 0x7fe7240051c0 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.891+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe73402f070 con 0x7fe7240051c0 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.891+0000 7fe73b400640 1 -- 172.21.15.69:0/1610830730 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73402bca0 con 0x7fe7240051c0 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.892+0000 7fe742600640 1 -- 172.21.15.69:0/1610830730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7240051c0 msgr2=0x7fe72400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.892+0000 7fe742600640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7240051c0 0x7fe72400b9c0 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fe73400a860 tx=0x7fe724005830 comp rx=0 tx=0).stop 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.892+0000 7fe742600640 1 -- 172.21.15.69:0/1610830730 shutdown_connections 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.892+0000 7fe742600640 1 --2- 172.21.15.69:0/1610830730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7240051c0 0x7fe72400b9c0 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.892+0000 7fe742600640 1 -- 172.21.15.69:0/1610830730 >> 172.21.15.69:0/1610830730 conn(0x7fe73c09d920 msgr2=0x7fe73c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:00.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.892+0000 7fe742600640 1 -- 172.21.15.69:0/1610830730 shutdown_connections 2024-04-24T08:17:00.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.892+0000 7fe742600640 1 -- 172.21.15.69:0/1610830730 wait complete. 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.894+0000 7fe742600640 1 Processor -- start 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.894+0000 7fe742600640 1 -- start start 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.895+0000 7fe742600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7240051c0 0x7fe73c0c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.895+0000 7fe742600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe73c161500 0x7fe73c0b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.895+0000 7fe742600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe73c161980 0x7fe73c0b5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.895+0000 7fe742600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c0d1360 con 0x7fe73c161500 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.895+0000 7fe742600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe73c0d11e0 con 0x7fe73c161980 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.895+0000 7fe742600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe73c0cf070 con 0x7fe7240051c0 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.895+0000 7fe73be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe73c161500 0x7fe73c0b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.895+0000 7fe73be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe73c161500 0x7fe73c0b51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39600/0 (socket says 172.21.15.69:39600) 2024-04-24T08:17:00.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.895+0000 7fe73be00640 1 -- 172.21.15.69:0/1640944731 learned_addr learned my addr 172.21.15.69:0/1640944731 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:00.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe741200640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7240051c0 0x7fe73c0c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe741c00640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe73c161980 0x7fe73c0b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe73be00640 1 -- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7240051c0 msgr2=0x7fe73c0c9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe73be00640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7240051c0 0x7fe73c0c9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe73be00640 1 -- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe73c161980 msgr2=0x7fe73c0b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:00.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe73be00640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe73c161980 0x7fe73c0b5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:00.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe73be00640 1 -- 172.21.15.69:0/1640944731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73c160740 con 0x7fe73c161500 2024-04-24T08:17:00.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe741200640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7240051c0 0x7fe73c0c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:00.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe73be00640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe73c161500 0x7fe73c0b51e0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fe72800e4e0 tx=0x7fe72800e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:00.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe738c00640 1 -- 172.21.15.69:0/1640944731 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7280052e0 con 0x7fe73c161500 2024-04-24T08:17:00.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe738c00640 1 -- 172.21.15.69:0/1640944731 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe728018070 con 0x7fe73c161500 2024-04-24T08:17:00.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe742600640 1 -- 172.21.15.69:0/1640944731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe73c160a20 con 0x7fe73c161500 2024-04-24T08:17:00.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe738c00640 1 -- 172.21.15.69:0/1640944731 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe728003830 con 0x7fe73c161500 2024-04-24T08:17:00.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.896+0000 7fe742600640 1 -- 172.21.15.69:0/1640944731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe73c0ca0d0 con 0x7fe73c161500 2024-04-24T08:17:00.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.898+0000 7fe742600640 1 -- 172.21.15.69:0/1640944731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe73c075d50 con 0x7fe73c161500 2024-04-24T08:17:00.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.898+0000 7fe738c00640 1 -- 172.21.15.69:0/1640944731 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe7280039d0 con 0x7fe73c161500 2024-04-24T08:17:00.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.899+0000 7fe738c00640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe71c06c920 0x7fe71c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:00.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.899+0000 7fe738c00640 1 -- 172.21.15.69:0/1640944731 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7fe72808b750 con 0x7fe73c161500 2024-04-24T08:17:00.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.900+0000 7fe741200640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe71c06c920 0x7fe71c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:00.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.901+0000 7fe741200640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe71c06c920 0x7fe71c06ede0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe73c1f80e0 tx=0x7fe730008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:00.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:00.904+0000 7fe738c00640 1 -- 172.21.15.69:0/1640944731 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe728059b20 con 0x7fe73c161500 2024-04-24T08:17:01.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.089+0000 7fe742600640 1 -- 172.21.15.69:0/1640944731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe73c0d1e40 con 0x7fe73c161500 2024-04-24T08:17:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.089+0000 7fe738c00640 1 -- 172.21.15.69:0/1640944731 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7fe728059420 con 0x7fe73c161500 2024-04-24T08:17:01.091 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:17:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 -- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe71c06c920 msgr2=0x7fe71c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe71c06c920 0x7fe71c06ede0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe73c1f80e0 tx=0x7fe730008040 comp rx=0 tx=0).stop 2024-04-24T08:17:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 -- 172.21.15.69:0/1640944731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe73c161500 msgr2=0x7fe73c0b51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe73c161500 0x7fe73c0b51e0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fe72800e4e0 tx=0x7fe72800e9a0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 -- 172.21.15.69:0/1640944731 shutdown_connections 2024-04-24T08:17:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe71c06c920 0x7fe71c06ede0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe73c161500 0x7fe73c0b51e0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe73c161980 0x7fe73c0b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 --2- 172.21.15.69:0/1640944731 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe7240051c0 0x7fe73c0c9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 -- 172.21.15.69:0/1640944731 >> 172.21.15.69:0/1640944731 conn(0x7fe73c09d920 msgr2=0x7fe73c1621a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:01.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 -- 172.21.15.69:0/1640944731 shutdown_connections 2024-04-24T08:17:01.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.092+0000 7fe742600640 1 -- 172.21.15.69:0/1640944731 wait complete. 2024-04-24T08:17:01.111 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:17:01.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.296+0000 7f5274a00640 1 Processor -- start 2024-04-24T08:17:01.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.296+0000 7f5274a00640 1 -- start start 2024-04-24T08:17:01.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.298+0000 7f5274a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f527015d1d0 0x7f52700ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:01.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.298+0000 7f5274a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f52700b30f0 0x7f52700ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:01.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.298+0000 7f5274a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f527015d660 con 0x7f5270156c60 2024-04-24T08:17:01.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.298+0000 7f5274a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f527006dc30 con 0x7f52700b30f0 2024-04-24T08:17:01.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.298+0000 7f5274a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f527006df10 con 0x7f527015d1d0 2024-04-24T08:17:01.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.298+0000 7f526f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f52700b30f0 0x7f52700ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:01.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.298+0000 7f526e800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5270156c60 0x7f527015cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:55978/0 (socket says 172.21.15.69:55978) 2024-04-24T08:17:01.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.298+0000 7f526e800640 1 -- 172.21.15.69:0/3092278716 learned_addr learned my addr 172.21.15.69:0/3092278716 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:01.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.298+0000 7f526de00640 1 --2- 172.21.15.69:0/3092278716 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f527015d1d0 0x7f52700ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:01.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.299+0000 7f526f200640 1 -- 172.21.15.69:0/3092278716 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f527015d1d0 msgr2=0x7f52700ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:01.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.299+0000 7f526f200640 1 --2- 172.21.15.69:0/3092278716 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f527015d1d0 0x7f52700ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.299+0000 7f526f200640 1 -- 172.21.15.69:0/3092278716 >> v1:172.21.15.69:6789/0 conn(0x7f5270156c60 legacy=0x7f527015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:01.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.299+0000 7f526d400640 1 -- 172.21.15.69:0/3092278716 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 536607186 0 0) 0x7f527015d660 con 0x7f5270156c60 2024-04-24T08:17:01.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.299+0000 7f526f200640 1 -- 172.21.15.69:0/3092278716 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f527006e1f0 con 0x7f52700b30f0 2024-04-24T08:17:01.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.300+0000 7f526f200640 1 --2- 172.21.15.69:0/3092278716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700b30f0 0x7f52700ad610 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f5268009d90 tx=0x7f526802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6063e85cfcf6ca3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:01.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.300+0000 7f526d400640 1 -- 172.21.15.69:0/3092278716 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5268007740 con 0x7f52700b30f0 2024-04-24T08:17:01.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.300+0000 7f526d400640 1 -- 172.21.15.69:0/3092278716 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52680078e0 con 0x7f52700b30f0 2024-04-24T08:17:01.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.300+0000 7f526d400640 1 -- 172.21.15.69:0/3092278716 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5268007cb0 con 0x7f52700b30f0 2024-04-24T08:17:01.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.301+0000 7f5274a00640 1 -- 172.21.15.69:0/3092278716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700b30f0 msgr2=0x7f52700ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:01.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.301+0000 7f5274a00640 1 --2- 172.21.15.69:0/3092278716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700b30f0 0x7f52700ad610 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f5268009d90 tx=0x7f526802a660 comp rx=0 tx=0).stop 2024-04-24T08:17:01.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.301+0000 7f5274a00640 1 -- 172.21.15.69:0/3092278716 shutdown_connections 2024-04-24T08:17:01.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.301+0000 7f5274a00640 1 --2- 172.21.15.69:0/3092278716 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700b30f0 0x7f52700ad610 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.301+0000 7f5274a00640 1 --2- 172.21.15.69:0/3092278716 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f527015d1d0 0x7f52700ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.301+0000 7f5274a00640 1 -- 172.21.15.69:0/3092278716 >> 172.21.15.69:0/3092278716 conn(0x7f52700d0160 msgr2=0x7f52700d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:01.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.301+0000 7f5274a00640 1 -- 172.21.15.69:0/3092278716 shutdown_connections 2024-04-24T08:17:01.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.301+0000 7f5274a00640 1 -- 172.21.15.69:0/3092278716 wait complete. 2024-04-24T08:17:01.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.303+0000 7f5274a00640 1 Processor -- start 2024-04-24T08:17:01.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.303+0000 7f5274a00640 1 -- start start 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.304+0000 7f5274a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52700b30f0 0x7f52701f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.304+0000 7f5274a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5270156c60 0x7f5270073600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.304+0000 7f5274a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700afd60 0x7f5270154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.304+0000 7f5274a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f527006e710 con 0x7f52700afd60 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.304+0000 7f5274a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f527015d940 con 0x7f5270156c60 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.304+0000 7f5274a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f527006e890 con 0x7f52700b30f0 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.304+0000 7f526f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700afd60 0x7f5270154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.304+0000 7f526f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700afd60 0x7f5270154f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39618/0 (socket says 172.21.15.69:39618) 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.304+0000 7f526f200640 1 -- 172.21.15.69:0/3071628530 learned_addr learned my addr 172.21.15.69:0/3071628530 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.304+0000 7f526e800640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52700b30f0 0x7f52701f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f526f200640 1 -- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52700b30f0 msgr2=0x7f52701f3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:01.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f526f200640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52700b30f0 0x7f52701f3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f526de00640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5270156c60 0x7f5270073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f526f200640 1 -- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5270156c60 msgr2=0x7f5270073600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f526f200640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5270156c60 0x7f5270073600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f526f200640 1 -- 172.21.15.69:0/3071628530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52700c9e00 con 0x7f52700afd60 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f526e800640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52700b30f0 0x7f52701f3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f526f200640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700afd60 0x7f5270154f30 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f5268009eb0 tx=0x7f5268004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f5266a00640 1 -- 172.21.15.69:0/3071628530 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5268040070 con 0x7f52700afd60 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f5266a00640 1 -- 172.21.15.69:0/3071628530 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5268031bc0 con 0x7f52700afd60 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f5266a00640 1 -- 172.21.15.69:0/3071628530 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5268045460 con 0x7f52700afd60 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f5274a00640 1 -- 172.21.15.69:0/3071628530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52700ca0e0 con 0x7f52700afd60 2024-04-24T08:17:01.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.305+0000 7f5274a00640 1 -- 172.21.15.69:0/3071628530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52700ca6a0 con 0x7f52700afd60 2024-04-24T08:17:01.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.306+0000 7f5274a00640 1 -- 172.21.15.69:0/3071628530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52700cf380 con 0x7f52700afd60 2024-04-24T08:17:01.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.307+0000 7f5266a00640 1 -- 172.21.15.69:0/3071628530 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5268039040 con 0x7f52700afd60 2024-04-24T08:17:01.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.308+0000 7f5266a00640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f525006c8f0 0x7f525006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:01.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.308+0000 7f5266a00640 1 -- 172.21.15.69:0/3071628530 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f52680ad630 con 0x7f52700afd60 2024-04-24T08:17:01.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.309+0000 7f526e800640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f525006c8f0 0x7f525006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:01.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.310+0000 7f526e800640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f525006c8f0 0x7f525006edb0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5260000f60 tx=0x7f5260009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:01.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.313+0000 7f5266a00640 1 -- 172.21.15.69:0/3071628530 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52700cf380 con 0x7f52700afd60 2024-04-24T08:17:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.516+0000 7f5274a00640 1 -- 172.21.15.69:0/3071628530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f527006f9e0 con 0x7f52700afd60 2024-04-24T08:17:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.517+0000 7f5266a00640 1 -- 172.21.15.69:0/3071628530 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v85) ==== 80+0+124 (secure 0 0 0) 0x7f5268043360 con 0x7f52700afd60 2024-04-24T08:17:01.519 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6837/1179683245 2024-04-25T08:15:45.259762+0000 2024-04-24T08:17:01.519 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6835/1179683245 2024-04-25T08:15:45.259762+0000 2024-04-24T08:17:01.519 INFO:teuthology.orchestra.run.smithi069.stderr:listed 2 entries 2024-04-24T08:17:01.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.519+0000 7f5274a00640 1 -- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f525006c8f0 msgr2=0x7f525006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:01.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.519+0000 7f5274a00640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f525006c8f0 0x7f525006edb0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f5260000f60 tx=0x7f5260009040 comp rx=0 tx=0).stop 2024-04-24T08:17:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.519+0000 7f5274a00640 1 -- 172.21.15.69:0/3071628530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700afd60 msgr2=0x7f5270154f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.519+0000 7f5274a00640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700afd60 0x7f5270154f30 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f5268009eb0 tx=0x7f5268004a40 comp rx=0 tx=0).stop 2024-04-24T08:17:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.520+0000 7f5274a00640 1 -- 172.21.15.69:0/3071628530 shutdown_connections 2024-04-24T08:17:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.520+0000 7f5274a00640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f525006c8f0 0x7f525006edb0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.520+0000 7f5274a00640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f52700afd60 0x7f5270154f30 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.520+0000 7f5274a00640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5270156c60 0x7f5270073600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.520+0000 7f5274a00640 1 --2- 172.21.15.69:0/3071628530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f52700b30f0 0x7f52701f3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.520+0000 7f5274a00640 1 -- 172.21.15.69:0/3071628530 >> 172.21.15.69:0/3071628530 conn(0x7f52700d0160 msgr2=0x7f52700b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.520+0000 7f5274a00640 1 -- 172.21.15.69:0/3071628530 shutdown_connections 2024-04-24T08:17:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:01.520+0000 7f5274a00640 1 -- 172.21.15.69:0/3071628530 wait complete. 2024-04-24T08:17:01.539 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:17:01.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:01.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:17:01.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[64754]: fuse finished with error 0 and tester_r 0 2024-04-24T08:17:01.618 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:17:07.720 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:17:07.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:07.721 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:17:07.779 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:07.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.979+0000 7f6fe1e00640 1 Processor -- start 2024-04-24T08:17:07.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.979+0000 7f6fe1e00640 1 -- start start 2024-04-24T08:17:07.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.981+0000 7f6fe1e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6fdc0afde0 0x7f6fdc06ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.981+0000 7f6fe1e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fdc15cbb0 con 0x7f6fdc15fb30 2024-04-24T08:17:07.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.981+0000 7f6fe1e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fdc15ce90 con 0x7f6fdc06cf70 2024-04-24T08:17:07.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.981+0000 7f6fe1e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f6fdc15d970 con 0x7f6fdc0afde0 2024-04-24T08:17:07.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.981+0000 7f6fdbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6fdc0afde0 0x7f6fdc06ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.981+0000 7f6fdbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6fdc0afde0 0x7f6fdc06ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39628/0 (socket says 172.21.15.69:39628) 2024-04-24T08:17:07.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.981+0000 7f6fdbe00640 1 -- 172.21.15.69:0/3015767146 learned_addr learned my addr 172.21.15.69:0/3015767146 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:07.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.982+0000 7f6fdbe00640 1 -- 172.21.15.69:0/3015767146 >> v1:172.21.15.121:6789/0 conn(0x7f6fdc06cf70 legacy=0x7f6fdc152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.982+0000 7f6fdbe00640 1 -- 172.21.15.69:0/3015767146 >> v1:172.21.15.69:6789/0 conn(0x7f6fdc15fb30 legacy=0x7f6fdc073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.982+0000 7f6fdbe00640 1 -- 172.21.15.69:0/3015767146 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fdc15d170 con 0x7f6fdc0afde0 2024-04-24T08:17:07.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.983+0000 7f6fdbe00640 1 --2- 172.21.15.69:0/3015767146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc0afde0 0x7f6fdc06ca30 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009870 tx=0x7f6fc802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c662734f397a1dd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.983+0000 7f6fdb400640 1 -- 172.21.15.69:0/3015767146 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8037070 con 0x7f6fdc0afde0 2024-04-24T08:17:07.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.983+0000 7f6fdb400640 1 -- 172.21.15.69:0/3015767146 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fc8032440 con 0x7f6fdc0afde0 2024-04-24T08:17:07.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.983+0000 7f6fdb400640 1 -- 172.21.15.69:0/3015767146 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc8032810 con 0x7f6fdc0afde0 2024-04-24T08:17:07.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.984+0000 7f6fe1e00640 1 -- 172.21.15.69:0/3015767146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc0afde0 msgr2=0x7f6fdc06ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.984+0000 7f6fe1e00640 1 --2- 172.21.15.69:0/3015767146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc0afde0 0x7f6fdc06ca30 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8009870 tx=0x7f6fc802a210 comp rx=0 tx=0).stop 2024-04-24T08:17:07.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.984+0000 7f6fe1e00640 1 -- 172.21.15.69:0/3015767146 shutdown_connections 2024-04-24T08:17:07.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.984+0000 7f6fe1e00640 1 --2- 172.21.15.69:0/3015767146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc0afde0 0x7f6fdc06ca30 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.984+0000 7f6fe1e00640 1 -- 172.21.15.69:0/3015767146 >> 172.21.15.69:0/3015767146 conn(0x7f6fdc0722b0 msgr2=0x7f6fdc0726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.984+0000 7f6fe1e00640 1 -- 172.21.15.69:0/3015767146 shutdown_connections 2024-04-24T08:17:07.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.984+0000 7f6fe1e00640 1 -- 172.21.15.69:0/3015767146 wait complete. 2024-04-24T08:17:07.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.986+0000 7f6fe1e00640 1 Processor -- start 2024-04-24T08:17:07.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.986+0000 7f6fe1e00640 1 -- start start 2024-04-24T08:17:07.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc06cf70 0x7f6fdc0b59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe1e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fdc0afde0 0x7f6fdc151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe1e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6fdc15fb30 0x7f6fdc0aaa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe1e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6fdc06d3c0 con 0x7f6fdc06cf70 2024-04-24T08:17:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe1e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6fdc0ce3c0 con 0x7f6fdc0afde0 2024-04-24T08:17:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe1e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6fdc073850 con 0x7f6fdc15fb30 2024-04-24T08:17:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc06cf70 0x7f6fdc0b59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc06cf70 0x7f6fdc0b59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39640/0 (socket says 172.21.15.69:39640) 2024-04-24T08:17:07.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe0a00640 1 -- 172.21.15.69:0/4190303121 learned_addr learned my addr 172.21.15.69:0/4190303121 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:07.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe0a00640 1 -- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6fdc15fb30 msgr2=0x7f6fdc0aaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe0a00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6fdc15fb30 0x7f6fdc0aaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.987+0000 7f6fe0a00640 1 -- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fdc0afde0 msgr2=0x7f6fdc151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.988+0000 7f6fe0a00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fdc0afde0 0x7f6fdc151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.988+0000 7f6fe0a00640 1 -- 172.21.15.69:0/4190303121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fdc152180 con 0x7f6fdc06cf70 2024-04-24T08:17:07.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.988+0000 7f6fe0a00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc06cf70 0x7f6fdc0b59a0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000c760 tx=0x7f6fd000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.988+0000 7f6fd8c00640 1 -- 172.21.15.69:0/4190303121 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0016020 con 0x7f6fdc06cf70 2024-04-24T08:17:07.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.988+0000 7f6fd8c00640 1 -- 172.21.15.69:0/4190303121 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd0006d50 con 0x7f6fdc06cf70 2024-04-24T08:17:07.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.988+0000 7f6fe1e00640 1 -- 172.21.15.69:0/4190303121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fdc0a9980 con 0x7f6fdc06cf70 2024-04-24T08:17:07.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.988+0000 7f6fd8c00640 1 -- 172.21.15.69:0/4190303121 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0003bb0 con 0x7f6fdc06cf70 2024-04-24T08:17:07.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.988+0000 7f6fe1e00640 1 -- 172.21.15.69:0/4190303121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fdc0a9f40 con 0x7f6fdc06cf70 2024-04-24T08:17:07.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.989+0000 7f6fe1e00640 1 -- 172.21.15.69:0/4190303121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fdc0aa650 con 0x7f6fdc06cf70 2024-04-24T08:17:07.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.990+0000 7f6fd8c00640 1 -- 172.21.15.69:0/4190303121 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6fd000a850 con 0x7f6fdc06cf70 2024-04-24T08:17:07.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.992+0000 7f6fd8c00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fbc06c920 0x7f6fbc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.992+0000 7f6fd8c00640 1 -- 172.21.15.69:0/4190303121 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f6fd008b960 con 0x7f6fdc06cf70 2024-04-24T08:17:07.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.992+0000 7f6fdbe00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fbc06c920 0x7f6fbc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.994+0000 7f6fdbe00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fbc06c920 0x7f6fbc06ede0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8034000 tx=0x7f6fc802c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:07.997+0000 7f6fd8c00640 1 -- 172.21.15.69:0/4190303121 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6fd0020020 con 0x7f6fdc06cf70 2024-04-24T08:17:08.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.235+0000 7f6fe1e00640 1 -- 172.21.15.69:0/4190303121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fdc06cd50 con 0x7f6fdc06cf70 2024-04-24T08:17:08.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.237+0000 7f6fd8c00640 1 -- 172.21.15.69:0/4190303121 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+5182 (secure 0 0 0) 0x7f6fdc152180 con 0x7f6fdc06cf70 2024-04-24T08:17:08.239 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:08.239 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":68,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:15:56.304080+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7581},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7581":{"gid":7581,"name":"a","rank":0,"incarnation":64,"state":"up:active","state_seq":12,"addr":"172.21.15.69:6833/2529876410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2529876410},{"type":"v1","addr":"172.21.15.69:6833","nonce":2529876410}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7581,"qdb_cluster":[7581]},"id":6}]} 2024-04-24T08:17:08.239 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 68 2024-04-24T08:17:08.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.239+0000 7f6fe1e00640 1 -- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fbc06c920 msgr2=0x7f6fbc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:08.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.239+0000 7f6fe1e00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fbc06c920 0x7f6fbc06ede0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8034000 tx=0x7f6fc802c000 comp rx=0 tx=0).stop 2024-04-24T08:17:08.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.239+0000 7f6fe1e00640 1 -- 172.21.15.69:0/4190303121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc06cf70 msgr2=0x7f6fdc0b59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:08.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.239+0000 7f6fe1e00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc06cf70 0x7f6fdc0b59a0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f6fd000c760 tx=0x7f6fd000cc20 comp rx=0 tx=0).stop 2024-04-24T08:17:08.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.239+0000 7f6fe1e00640 1 -- 172.21.15.69:0/4190303121 shutdown_connections 2024-04-24T08:17:08.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.239+0000 7f6fe1e00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6fbc06c920 0x7f6fbc06ede0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.239+0000 7f6fe1e00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6fdc15fb30 0x7f6fdc0aaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.239+0000 7f6fe1e00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fdc0afde0 0x7f6fdc151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.239+0000 7f6fe1e00640 1 --2- 172.21.15.69:0/4190303121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6fdc06cf70 0x7f6fdc0b59a0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.239+0000 7f6fe1e00640 1 -- 172.21.15.69:0/4190303121 >> 172.21.15.69:0/4190303121 conn(0x7f6fdc0722b0 msgr2=0x7f6fdc160800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:08.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.240+0000 7f6fe1e00640 1 -- 172.21.15.69:0/4190303121 shutdown_connections 2024-04-24T08:17:08.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.240+0000 7f6fe1e00640 1 -- 172.21.15.69:0/4190303121 wait complete. 2024-04-24T08:17:08.259 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-04-24T08:17:08.495 INFO:teuthology.orchestra.run.smithi069.stdout:{"message":"","return_code":0} 2024-04-24T08:17:08.495 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-24T08:17:08.495 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:17:08.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.672+0000 7f05c7400640 1 Processor -- start 2024-04-24T08:17:08.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.672+0000 7f05c7400640 1 -- start start 2024-04-24T08:17:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.674+0000 7f05c7400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f05c015b490 0x7f05c00b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.674+0000 7f05c7400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c006f2d0 0x7f05c006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.674+0000 7f05c7400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05c00b46e0 con 0x7f05c00cfbe0 2024-04-24T08:17:08.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.674+0000 7f05c7400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f05c006fc70 con 0x7f05c015b490 2024-04-24T08:17:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.674+0000 7f05c7400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f05c00d0960 con 0x7f05c006f2d0 2024-04-24T08:17:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.675+0000 7f05c4a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f05c015b490 0x7f05c00b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.675+0000 7f05bfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c006f2d0 0x7f05c006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.675+0000 7f05c4a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f05c015b490 0x7f05c00b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:38966/0 (socket says 172.21.15.69:38966) 2024-04-24T08:17:08.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.675+0000 7f05c4a00640 1 -- 172.21.15.69:0/974775067 learned_addr learned my addr 172.21.15.69:0/974775067 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:08.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.675+0000 7f05bfe00640 1 -- 172.21.15.69:0/974775067 >> v2:172.21.15.121:3300/0 conn(0x7f05c015b490 msgr2=0x7f05c00b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:08.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.675+0000 7f05bfe00640 1 --2- 172.21.15.69:0/974775067 >> v2:172.21.15.121:3300/0 conn(0x7f05c015b490 0x7f05c00b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.675+0000 7f05bfe00640 1 -- 172.21.15.69:0/974775067 >> v1:172.21.15.121:6789/0 conn(0x7f05c00cfbe0 legacy=0x7f05c00d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:08.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.675+0000 7f05bfe00640 1 -- 172.21.15.69:0/974775067 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c00b4980 con 0x7f05c006f2d0 2024-04-24T08:17:08.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.675+0000 7f05c4a00640 1 --2- 172.21.15.69:0/974775067 >> v2:172.21.15.121:3300/0 conn(0x7f05c015b490 0x7f05c00b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.677+0000 7f05bfe00640 1 --2- 172.21.15.69:0/974775067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c006f2d0 0x7f05c006f730 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0097a0 tx=0x7f05ac02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c1af854ace00886 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:08.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.677+0000 7f05bf400640 1 -- 172.21.15.69:0/974775067 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ac037070 con 0x7f05c006f2d0 2024-04-24T08:17:08.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.677+0000 7f05bf400640 1 -- 172.21.15.69:0/974775067 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05ac032440 con 0x7f05c006f2d0 2024-04-24T08:17:08.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.677+0000 7f05bf400640 1 -- 172.21.15.69:0/974775067 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ac032810 con 0x7f05c006f2d0 2024-04-24T08:17:08.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.678+0000 7f05c7400640 1 -- 172.21.15.69:0/974775067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c006f2d0 msgr2=0x7f05c006f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:08.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.678+0000 7f05c7400640 1 --2- 172.21.15.69:0/974775067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c006f2d0 0x7f05c006f730 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0097a0 tx=0x7f05ac02a180 comp rx=0 tx=0).stop 2024-04-24T08:17:08.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.678+0000 7f05c7400640 1 -- 172.21.15.69:0/974775067 shutdown_connections 2024-04-24T08:17:08.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.678+0000 7f05c7400640 1 --2- 172.21.15.69:0/974775067 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c006f2d0 0x7f05c006f730 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.678+0000 7f05c7400640 1 --2- 172.21.15.69:0/974775067 >> v2:172.21.15.121:3300/0 conn(0x7f05c015b490 0x7f05c00b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.678+0000 7f05c7400640 1 -- 172.21.15.69:0/974775067 >> 172.21.15.69:0/974775067 conn(0x7f05c0067d90 msgr2=0x7f05c006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:08.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.678+0000 7f05c7400640 1 -- 172.21.15.69:0/974775067 shutdown_connections 2024-04-24T08:17:08.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.678+0000 7f05c7400640 1 -- 172.21.15.69:0/974775067 wait complete. 2024-04-24T08:17:08.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.679+0000 7f05c7400640 1 Processor -- start 2024-04-24T08:17:08.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.680+0000 7f05c7400640 1 -- start start 2024-04-24T08:17:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.680+0000 7f05c7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c006f2d0 0x7f05c014b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.680+0000 7f05c7400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c00cfbe0 0x7f05c0150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.680+0000 7f05c7400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05c015b490 0x7f05c00b3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.680+0000 7f05c7400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f05c015b870 con 0x7f05c006f2d0 2024-04-24T08:17:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.680+0000 7f05c7400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f05c00d0c40 con 0x7f05c015b490 2024-04-24T08:17:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.680+0000 7f05c7400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f05c00b56c0 con 0x7f05c00cfbe0 2024-04-24T08:17:08.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05c4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c006f2d0 0x7f05c014b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05c4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c006f2d0 0x7f05c014b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39654/0 (socket says 172.21.15.69:39654) 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05c4a00640 1 -- 172.21.15.69:0/2269910534 learned_addr learned my addr 172.21.15.69:0/2269910534 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05bfe00640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c00cfbe0 0x7f05c0150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05c4a00640 1 -- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c00cfbe0 msgr2=0x7f05c0150440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05c4a00640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c00cfbe0 0x7f05c0150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05c5400640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05c015b490 0x7f05c00b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05c4a00640 1 -- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05c015b490 msgr2=0x7f05c00b3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05c4a00640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05c015b490 0x7f05c00b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05c4a00640 1 -- 172.21.15.69:0/2269910534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c0150e70 con 0x7f05c006f2d0 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.681+0000 7f05c4a00640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c006f2d0 0x7f05c014b0d0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f05b400ea00 tx=0x7f05b400eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.682+0000 7f05bcc00640 1 -- 172.21.15.69:0/2269910534 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b4017070 con 0x7f05c006f2d0 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.682+0000 7f05bcc00640 1 -- 172.21.15.69:0/2269910534 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b400ca60 con 0x7f05c006f2d0 2024-04-24T08:17:08.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.682+0000 7f05bcc00640 1 -- 172.21.15.69:0/2269910534 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b400ce10 con 0x7f05c006f2d0 2024-04-24T08:17:08.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.682+0000 7f05c7400640 1 -- 172.21.15.69:0/2269910534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c0151150 con 0x7f05c006f2d0 2024-04-24T08:17:08.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.682+0000 7f05c7400640 1 -- 172.21.15.69:0/2269910534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c0151710 con 0x7f05c006f2d0 2024-04-24T08:17:08.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.683+0000 7f05c7400640 1 -- 172.21.15.69:0/2269910534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05c0151950 con 0x7f05c006f2d0 2024-04-24T08:17:08.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.684+0000 7f05bcc00640 1 -- 172.21.15.69:0/2269910534 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f05b401a070 con 0x7f05c006f2d0 2024-04-24T08:17:08.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.685+0000 7f05bcc00640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05a006c8f0 0x7f05a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:08.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.685+0000 7f05bcc00640 1 -- 172.21.15.69:0/2269910534 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 5824+0+0 (secure 0 0 0) 0x7f05b408bf20 con 0x7f05c006f2d0 2024-04-24T08:17:08.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.685+0000 7f05bfe00640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05a006c8f0 0x7f05a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:08.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.687+0000 7f05bfe00640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05a006c8f0 0x7f05a006edb0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f05ac034000 tx=0x7f05ac02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:08.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.689+0000 7f05bcc00640 1 -- 172.21.15.69:0/2269910534 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05c0151950 con 0x7f05c006f2d0 2024-04-24T08:17:08.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.929+0000 7f05c7400640 1 -- 172.21.15.69:0/2269910534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f05c00b1a00 con 0x7f05c006f2d0 2024-04-24T08:17:08.932 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:17:08.930+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:17:08.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.934+0000 7f05bcc00640 1 -- 172.21.15.69:0/2269910534 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v69) ==== 161+0+0 (secure 0 0 0) 0x7f05c00b1a00 con 0x7f05c006f2d0 2024-04-24T08:17:08.936 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 -- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05a006c8f0 msgr2=0x7f05a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05a006c8f0 0x7f05a006edb0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f05ac034000 tx=0x7f05ac02e000 comp rx=0 tx=0).stop 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 -- 172.21.15.69:0/2269910534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c006f2d0 msgr2=0x7f05c014b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c006f2d0 0x7f05c014b0d0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f05b400ea00 tx=0x7f05b400eec0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 -- 172.21.15.69:0/2269910534 shutdown_connections 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f05a006c8f0 0x7f05a006edb0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f05c015b490 0x7f05c00b3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f05c00cfbe0 0x7f05c0150440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 --2- 172.21.15.69:0/2269910534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05c006f2d0 0x7f05c014b0d0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 -- 172.21.15.69:0/2269910534 >> 172.21.15.69:0/2269910534 conn(0x7f05c0067d90 msgr2=0x7f05c00b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:08.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.938+0000 7f05c7400640 1 -- 172.21.15.69:0/2269910534 shutdown_connections 2024-04-24T08:17:08.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:08.939+0000 7f05c7400640 1 -- 172.21.15.69:0/2269910534 wait complete. 2024-04-24T08:17:08.951 INFO:tasks.ceph.mds.a.smithi069.stderr: -4564> 2024-04-24T08:15:53.344+0000 7faf1f600640 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-04-24T08:17:08.952 INFO:tasks.ceph.mds.a.smithi069.stderr: -4379> 2024-04-24T08:15:53.605+0000 7faf1f600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 ap=1 f(v0 m2024-04-24T08:15:32.898213+0000 1=1+0) n(v0 rc2024-04-24T08:15:33.020316+0000 2=1+1) | dirfrag=0 authpin=1 scrubqueue=0 0x55f9f868c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-04-24T08:15:32.898213+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-04-24T08:15:33.020316+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-04-24T08:17:08.952 INFO:tasks.ceph.mds.a.smithi069.stderr: -4309> 2024-04-24T08:15:53.917+0000 7faf24600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:17:08.955 INFO:tasks.ceph.mds.a.smithi069.stderr: -2578> 2024-04-24T08:16:25.649+0000 7faf1f600640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 pv17 ap=3 DIRTYPARENT f(v0 m2024-04-24T08:15:32.898213+0000 1=1+0)->f(v1 m2024-04-24T08:15:32.898213+0000) n(v0 rc2024-04-24T08:15:33.020316+0000 2=1+1)->n(v0 rc2024-04-24T08:15:33.020316+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55f9f868c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(12)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-04-24T08:15:32.898213+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-04-24T08:15:33.020316+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-04-24T08:17:08.961 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-04-24T08:17:09.005 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:17:09.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.047+0000 7fe45e4ba740 1 -- start start 2024-04-24T08:17:09.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.048+0000 7fe45e4ba740 1 Processor -- start 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.048+0000 7fe45e4ba740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6aae96c00 0x55c6abc6a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.048+0000 7fe45e4ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6aae97800 0x55c6abc6a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.048+0000 7fe45e4ba740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6aae97c00 0x55c6abc6ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.048+0000 7fe45e4ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c6abc06180 con 0x55c6aae97800 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.048+0000 7fe45e4ba740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c6abc06300 con 0x55c6aae96c00 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.048+0000 7fe45e4ba740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c6abc06480 con 0x55c6aae97c00 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45bc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6aae97800 0x55c6abc6a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45bc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6aae97800 0x55c6abc6a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46514/0 (socket says 172.21.15.69:46514) 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45bc00640 1 -- 172.21.15.69:0/3753673765 learned_addr learned my addr 172.21.15.69:0/3753673765 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45d000640 1 --2- 172.21.15.69:0/3753673765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6aae97c00 0x55c6abc6ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45c600640 1 --2- 172.21.15.69:0/3753673765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6aae96c00 0x55c6abc6a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45bc00640 1 -- 172.21.15.69:0/3753673765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6aae97c00 msgr2=0x55c6abc6ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45bc00640 1 --2- 172.21.15.69:0/3753673765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6aae97c00 0x55c6abc6ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45bc00640 1 -- 172.21.15.69:0/3753673765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6aae96c00 msgr2=0x55c6abc6a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45bc00640 1 --2- 172.21.15.69:0/3753673765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6aae96c00 0x55c6abc6a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45bc00640 1 -- 172.21.15.69:0/3753673765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6aaec3680 con 0x55c6aae97800 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45c600640 1 --2- 172.21.15.69:0/3753673765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6aae96c00 0x55c6abc6a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.049+0000 7fe45bc00640 1 --2- 172.21.15.69:0/3753673765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6aae97800 0x55c6abc6a580 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x55c6abcafc50 tx=0x55c6abc3a120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67a05bcb0382800 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe459400640 1 -- 172.21.15.69:0/3753673765 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c6aaebc680 con 0x55c6aae97800 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe459400640 1 -- 172.21.15.69:0/3753673765 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55c6aaee0e00 con 0x55c6aae97800 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe459400640 1 -- 172.21.15.69:0/3753673765 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c6aaebcb60 con 0x55c6aae97800 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe45e4ba740 1 -- 172.21.15.69:0/3753673765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6aaec32c0 con 0x55c6aae97800 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe45e4ba740 1 -- 172.21.15.69:0/3753673765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55c6aaec34a0 con 0x55c6aae97800 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe45e4ba740 4 waiting for MDS map... 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe459400640 1 -- 172.21.15.69:0/3753673765 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x55c6abd66600 con 0x55c6aae97800 2024-04-24T08:17:09.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe459400640 1 -- 172.21.15.69:0/3753673765 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 69) ==== 2885+0+0 (secure 0 0 0) 0x55c6aaeef180 con 0x55c6aae97800 2024-04-24T08:17:09.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe45e4ba740 4 Got MDS map 69 2024-04-24T08:17:09.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe45e4ba740 4 datascan.main: Using metadata pool output 2024-04-24T08:17:09.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.050+0000 7fe45e4ba740 4 datascan.main: connecting to RADOS... 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.051+0000 7fe45e4ba740 1 Processor -- start 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.051+0000 7fe45e4ba740 1 -- start start 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45e4ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45e4ba740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6abd69800 0x55c6abc6b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45e4ba740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6abd69c00 0x55c6abc6bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45e4ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c6abc06600 con 0x55c6abd69400 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45e4ba740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c6abc06780 con 0x55c6abd69c00 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45e4ba740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c6abc06900 con 0x55c6abd69800 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46526/0 (socket says 172.21.15.69:46526) 2024-04-24T08:17:09.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45d000640 1 -- 172.21.15.69:0/2328380175 learned_addr learned my addr 172.21.15.69:0/2328380175 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45d000640 1 -- 172.21.15.69:0/2328380175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6abd69800 msgr2=0x55c6abc6b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45d000640 1 --2- 172.21.15.69:0/2328380175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6abd69800 0x55c6abc6b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45bc00640 1 --2- 172.21.15.69:0/2328380175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6abd69c00 0x55c6abc6bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45d000640 1 -- 172.21.15.69:0/2328380175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6abd69c00 msgr2=0x55c6abc6bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45d000640 1 --2- 172.21.15.69:0/2328380175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6abd69c00 0x55c6abc6bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.052+0000 7fe45d000640 1 -- 172.21.15.69:0/2328380175 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6aaec30e0 con 0x55c6abd69400 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe45d000640 1 --2- 172.21.15.69:0/2328380175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x55c6abcaf200 tx=0x55c6abc369c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6738b040d895a644 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe456c00640 1 -- 172.21.15.69:0/2328380175 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c6aaebc9c0 con 0x55c6abd69400 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe456c00640 1 -- 172.21.15.69:0/2328380175 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55c6aaee0c40 con 0x55c6abd69400 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe456c00640 1 -- 172.21.15.69:0/2328380175 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c6aaebc680 con 0x55c6abd69400 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe45e4ba740 1 -- 172.21.15.69:0/2328380175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 msgr2=0x55c6abc6b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/2328380175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x55c6abcaf200 tx=0x55c6abc369c0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe45e4ba740 1 -- 172.21.15.69:0/2328380175 shutdown_connections 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/2328380175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6abd69c00 0x55c6abc6bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/2328380175 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6abd69800 0x55c6abc6b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/2328380175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe45e4ba740 1 -- 172.21.15.69:0/2328380175 >> 172.21.15.69:0/2328380175 conn(0x55c6abd69000 msgr2=0x55c6aaf65000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.053+0000 7fe45e4ba740 1 -- 172.21.15.69:0/2328380175 shutdown_connections 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.054+0000 7fe45e4ba740 1 -- 172.21.15.69:0/2328380175 wait complete. 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.054+0000 7fe45e4ba740 1 Processor -- start 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.054+0000 7fe45e4ba740 1 -- start start 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.055+0000 7fe45e4ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.055+0000 7fe45e4ba740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6abd69800 0x55c6abc6b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.055+0000 7fe45e4ba740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6abd69c00 0x55c6abc6bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.055+0000 7fe45e4ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c6abc07b00 con 0x55c6abd69400 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.055+0000 7fe45e4ba740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c6abc07980 con 0x55c6abd69c00 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.055+0000 7fe45e4ba740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c6abc07800 con 0x55c6abd69800 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.055+0000 7fe45d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.055+0000 7fe45d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46542/0 (socket says 172.21.15.69:46542) 2024-04-24T08:17:09.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.055+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 learned_addr learned my addr 172.21.15.69:0/25220706 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6abd69800 msgr2=0x55c6abc6b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe45d000640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6abd69800 0x55c6abc6b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6abd69c00 msgr2=0x55c6abc6bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe45d000640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6abd69c00 0x55c6abc6bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6aaec3680 con 0x55c6abd69400 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe45d000640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x55c6abd7cab0 tx=0x55c6abc36420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe454e00640 1 -- 172.21.15.69:0/25220706 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c6aaebcb60 con 0x55c6abd69400 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe454e00640 1 -- 172.21.15.69:0/25220706 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55c6aaee0a80 con 0x55c6abd69400 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c6aaec32c0 con 0x55c6abd69400 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe454e00640 1 -- 172.21.15.69:0/25220706 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c6aaebc4e0 con 0x55c6abd69400 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6aaec3860 con 0x55c6abd69400 2024-04-24T08:17:09.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.056+0000 7fe45e4ba740 4 datascan.init: resolving metadata pool 12 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.058+0000 7fe454e00640 1 -- 172.21.15.69:0/25220706 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55c6abdcec00 con 0x55c6abd69400 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.058+0000 7fe454e00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55c6ab202000 0x55c6abc6c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.058+0000 7fe454e00640 1 -- 172.21.15.69:0/25220706 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x55c6abc90600 con 0x55c6abd69400 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.059+0000 7fe45e4ba740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.059+0000 7fe45e4ba740 4 datascan.main: data pool 'cephfs_data' has ID 13 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.059+0000 7fe45e4ba740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.059+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55c6ab202400 0x55c6abc6c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.059+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:1 13.0 13:00000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab202800 con 0x55c6ab202400 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.059+0000 7fe45c600640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55c6ab202000 0x55c6abc6c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.059+0000 7fe45bc00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55c6ab202400 0x55c6abc6c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.060+0000 7fe45bc00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55c6ab202400 0x55c6abc6c680 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.061+0000 7fe45c600640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55c6ab202000 0x55c6abc6c100 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x55c6abc39020 tx=0x55c6abd80ea0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.066+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== osd_op_reply(1 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.066+0000 7fe456200640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55c6ab21a000 0x55c6ab220000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.066+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:2 13.20 13:04000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21a400 con 0x55c6ab21a000 2024-04-24T08:17:09.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.067+0000 7fe45d000640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55c6ab21a000 0x55c6ab220000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.068+0000 7fe45d000640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55c6ab21a000 0x55c6ab220000 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.069+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(2 [pgnls start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.069+0000 7fe456c00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55c6ab21ac00 0x55c6ab220580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.069+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:3 13.10 13:08000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21b000 con 0x55c6ab21ac00 2024-04-24T08:17:09.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.069+0000 7fe45c600640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55c6ab21ac00 0x55c6ab220580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.070+0000 7fe45c600640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55c6ab21ac00 0x55c6ab220580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.072+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(3 [pgnls start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab21ac00 2024-04-24T08:17:09.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.072+0000 7fe456200640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55c6ab21a800 0x55c6ab220b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.072+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:4 13.30 13:0c000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21b800 con 0x55c6ab21a800 2024-04-24T08:17:09.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.073+0000 7fe45bc00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55c6ab21a800 0x55c6ab220b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.074+0000 7fe45bc00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55c6ab21a800 0x55c6ab220b00 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.079+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(4 [pgnls start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.080+0000 7fe456c00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55c6ab21b400 0x55c6ab221600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.080+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:5 13.8 13:10000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23a000 con 0x55c6ab21b400 2024-04-24T08:17:09.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.080+0000 7fe45d000640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55c6ab21b400 0x55c6ab221600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.080+0000 7fe45d000640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55c6ab21b400 0x55c6ab221600 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.081+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== osd_op_reply(5 [pgnls start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21b400 2024-04-24T08:17:09.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.081+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:6 13.28 13:14000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21bc00 con 0x55c6ab21b400 2024-04-24T08:17:09.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.081+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.2 v2:172.21.15.69:6808/2630808717 2 ==== osd_op_reply(6 [pgnls start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21b400 2024-04-24T08:17:09.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.081+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:7 13.18 13:18000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23a400 con 0x55c6ab202400 2024-04-24T08:17:09.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.082+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 2 ==== osd_op_reply(7 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.082+0000 7fe456200640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55c6ab23ac00 0x55c6ab221080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.082+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:8 13.38 13:1c000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b000 con 0x55c6ab23ac00 2024-04-24T08:17:09.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.083+0000 7fe45c600640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55c6ab23ac00 0x55c6ab221080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.084+0000 7fe45c600640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55c6ab23ac00 0x55c6ab221080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.085+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(8 [pgnls start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab23ac00 2024-04-24T08:17:09.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.085+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:9 13.4 13:20000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23a800 con 0x55c6ab21a000 2024-04-24T08:17:09.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.086+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 2 ==== osd_op_reply(9 [pgnls start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.086+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:10 13.24 13:24000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b400 con 0x55c6ab21a000 2024-04-24T08:17:09.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.087+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 3 ==== osd_op_reply(10 [pgnls start_epoch 86 out=49b] v82'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.087+0000 7fe456c00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55c6ab23a000 0x55c6ab221b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.087+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:11 13.14 13:28000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21a400 con 0x55c6ab23a000 2024-04-24T08:17:09.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.088+0000 7fe45bc00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55c6ab23a000 0x55c6ab221b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.088+0000 7fe45bc00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55c6ab23a000 0x55c6ab221b80 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.092+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== osd_op_reply(11 [pgnls start_epoch 86 out=49b] v82'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab23a000 2024-04-24T08:17:09.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.092+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:12 13.34 13:2c000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b800 con 0x55c6ab21a000 2024-04-24T08:17:09.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.093+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 4 ==== osd_op_reply(12 [pgnls start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.093+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:13 13.c 13:30000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21bc00 con 0x55c6ab21ac00 2024-04-24T08:17:09.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.093+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.1 v2:172.21.15.69:6815/153736297 2 ==== osd_op_reply(13 [pgnls start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab21ac00 2024-04-24T08:17:09.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.093+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:14 13.2c 13:34000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23a400 con 0x55c6ab21b400 2024-04-24T08:17:09.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.093+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.2 v2:172.21.15.69:6808/2630808717 3 ==== osd_op_reply(14 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21b400 2024-04-24T08:17:09.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.093+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:15 13.1c 13:38000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6abc35400 con 0x55c6ab21a000 2024-04-24T08:17:09.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.094+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 5 ==== osd_op_reply(15 [pgnls start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.094+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:16 13.3c 13:3c000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21b800 con 0x55c6ab21b400 2024-04-24T08:17:09.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.095+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.2 v2:172.21.15.69:6808/2630808717 4 ==== osd_op_reply(16 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21b400 2024-04-24T08:17:09.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.095+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:17 13.2 13:40000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23a800 con 0x55c6ab23ac00 2024-04-24T08:17:09.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.096+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.5 v2:172.21.15.121:6822/2187246263 2 ==== osd_op_reply(17 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab23ac00 2024-04-24T08:17:09.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.096+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:18 13.22 13:44000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21a400 con 0x55c6ab21ac00 2024-04-24T08:17:09.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.097+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.1 v2:172.21.15.69:6815/153736297 3 ==== osd_op_reply(18 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab21ac00 2024-04-24T08:17:09.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.097+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:19 13.12 13:48000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23bc00 con 0x55c6ab23a000 2024-04-24T08:17:09.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.097+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.0 v2:172.21.15.69:6800/146737935 2 ==== osd_op_reply(19 [pgnls start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab23a000 2024-04-24T08:17:09.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.097+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:20 13.32 13:4c000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab202800 con 0x55c6ab21b400 2024-04-24T08:17:09.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.098+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.2 v2:172.21.15.69:6808/2630808717 5 ==== osd_op_reply(20 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21b400 2024-04-24T08:17:09.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.098+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:21 13.a 13:50000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b400 con 0x55c6ab21b400 2024-04-24T08:17:09.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.098+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.2 v2:172.21.15.69:6808/2630808717 6 ==== osd_op_reply(21 [pgnls start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21b400 2024-04-24T08:17:09.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.098+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:22 13.2a 13:54000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6abd68800 con 0x55c6ab21ac00 2024-04-24T08:17:09.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.098+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.1 v2:172.21.15.69:6815/153736297 4 ==== osd_op_reply(22 [pgnls start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab21ac00 2024-04-24T08:17:09.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.099+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:23 13.1a 13:58000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21b800 con 0x55c6ab21ac00 2024-04-24T08:17:09.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.099+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.1 v2:172.21.15.69:6815/153736297 5 ==== osd_op_reply(23 [pgnls start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab21ac00 2024-04-24T08:17:09.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.099+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:24 13.3a 13:5c000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b800 con 0x55c6ab23ac00 2024-04-24T08:17:09.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.100+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.5 v2:172.21.15.121:6822/2187246263 3 ==== osd_op_reply(24 [pgnls start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab23ac00 2024-04-24T08:17:09.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.100+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:25 13.6 13:60000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21b000 con 0x55c6ab21a800 2024-04-24T08:17:09.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.101+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== osd_op_reply(25 [pgnls start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.101+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:26 13.26 13:64000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23a400 con 0x55c6ab202400 2024-04-24T08:17:09.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.102+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 3 ==== osd_op_reply(26 [pgnls start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.102+0000 7fe456c00640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55c6ab21bc00 0x55c6ab222100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.102+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:27 13.16 13:68000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab202800 con 0x55c6ab21bc00 2024-04-24T08:17:09.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.103+0000 7fe45d000640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55c6ab21bc00 0x55c6ab222100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.103+0000 7fe45d000640 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55c6ab21bc00 0x55c6ab222100 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.108+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(27 [pgnls start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21bc00 2024-04-24T08:17:09.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.108+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:28 13.36 13:6c000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b000 con 0x55c6ab21a000 2024-04-24T08:17:09.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.109+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 6 ==== osd_op_reply(28 [pgnls start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.109+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:29 13.e 13:70000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6abc35400 con 0x55c6ab202400 2024-04-24T08:17:09.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.110+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 4 ==== osd_op_reply(29 [pgnls start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.110+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:30 13.2e 13:74000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23a800 con 0x55c6ab21b400 2024-04-24T08:17:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.111+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.2 v2:172.21.15.69:6808/2630808717 7 ==== osd_op_reply(30 [pgnls start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21b400 2024-04-24T08:17:09.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.111+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:31 13.1e 13:78000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24e000 con 0x55c6ab21a800 2024-04-24T08:17:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.112+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 3 ==== osd_op_reply(31 [pgnls start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.112+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:32 13.3e 13:7c000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24fc00 con 0x55c6ab21ac00 2024-04-24T08:17:09.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.112+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.1 v2:172.21.15.69:6815/153736297 6 ==== osd_op_reply(32 [pgnls start_epoch 86 out=49b] v82'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab21ac00 2024-04-24T08:17:09.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.113+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:33 13.1 13:80000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24e400 con 0x55c6ab21a000 2024-04-24T08:17:09.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.114+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 7 ==== osd_op_reply(33 [pgnls start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.114+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:34 13.21 13:84000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24fc00 con 0x55c6ab21a800 2024-04-24T08:17:09.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.115+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 4 ==== osd_op_reply(34 [pgnls start_epoch 86 out=49b] v82'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.115+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:35 13.11 13:88000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24e800 con 0x55c6ab21bc00 2024-04-24T08:17:09.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.115+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.3 v2:172.21.15.69:6811/1875112303 2 ==== osd_op_reply(35 [pgnls start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21bc00 2024-04-24T08:17:09.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.116+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:36 13.31 13:8c000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b800 con 0x55c6ab21bc00 2024-04-24T08:17:09.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.116+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.3 v2:172.21.15.69:6811/1875112303 3 ==== osd_op_reply(36 [pgnls start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21bc00 2024-04-24T08:17:09.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.116+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:37 13.9 13:90000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24ec00 con 0x55c6ab23ac00 2024-04-24T08:17:09.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.117+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.5 v2:172.21.15.121:6822/2187246263 4 ==== osd_op_reply(37 [pgnls start_epoch 86 out=49b] v82'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab23ac00 2024-04-24T08:17:09.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.117+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:38 13.29 13:94000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21a400 con 0x55c6ab21a000 2024-04-24T08:17:09.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.118+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 8 ==== osd_op_reply(38 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.118+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:39 13.19 13:98000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24f000 con 0x55c6ab21bc00 2024-04-24T08:17:09.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.119+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.3 v2:172.21.15.69:6811/1875112303 4 ==== osd_op_reply(39 [pgnls start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21bc00 2024-04-24T08:17:09.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.119+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:40 13.39 13:9c000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21b800 con 0x55c6ab21b400 2024-04-24T08:17:09.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.120+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.2 v2:172.21.15.69:6808/2630808717 8 ==== osd_op_reply(40 [pgnls start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21b400 2024-04-24T08:17:09.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.120+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:41 13.5 13:a0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24f400 con 0x55c6ab21a800 2024-04-24T08:17:09.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.121+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 5 ==== osd_op_reply(41 [pgnls start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.121+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:42 13.25 13:a4000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6abd68800 con 0x55c6ab21a800 2024-04-24T08:17:09.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.122+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 6 ==== osd_op_reply(42 [pgnls start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.122+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:43 13.15 13:a8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24f800 con 0x55c6ab21a800 2024-04-24T08:17:09.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.123+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 7 ==== osd_op_reply(43 [pgnls start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.124+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:44 13.35 13:ac000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6abc35400 con 0x55c6ab21ac00 2024-04-24T08:17:09.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.124+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.1 v2:172.21.15.69:6815/153736297 7 ==== osd_op_reply(44 [pgnls start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab21ac00 2024-04-24T08:17:09.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.124+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:45 13.d 13:b0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24e400 con 0x55c6ab21a800 2024-04-24T08:17:09.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.125+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 8 ==== osd_op_reply(45 [pgnls start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.126+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:46 13.2d 13:b4000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b400 con 0x55c6ab23a000 2024-04-24T08:17:09.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.126+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.0 v2:172.21.15.69:6800/146737935 3 ==== osd_op_reply(46 [pgnls start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab23a000 2024-04-24T08:17:09.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.126+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:47 13.1d 13:b8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24e800 con 0x55c6ab202400 2024-04-24T08:17:09.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.128+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 5 ==== osd_op_reply(47 [pgnls start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.128+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:48 13.3d 13:bc000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b000 con 0x55c6ab202400 2024-04-24T08:17:09.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.129+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 6 ==== osd_op_reply(48 [pgnls start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.129+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:49 13.3 13:c0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24e000 con 0x55c6ab23a000 2024-04-24T08:17:09.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.130+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.0 v2:172.21.15.69:6800/146737935 4 ==== osd_op_reply(49 [pgnls start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab23a000 2024-04-24T08:17:09.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.130+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:50 13.23 13:c4000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23a800 con 0x55c6ab21ac00 2024-04-24T08:17:09.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.130+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.1 v2:172.21.15.69:6815/153736297 8 ==== osd_op_reply(50 [pgnls start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab21ac00 2024-04-24T08:17:09.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.131+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:51 13.13 13:c8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24fc00 con 0x55c6ab202400 2024-04-24T08:17:09.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.132+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 7 ==== osd_op_reply(51 [pgnls start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.132+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:52 13.33 13:cc000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b800 con 0x55c6ab202400 2024-04-24T08:17:09.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.133+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 8 ==== osd_op_reply(52 [pgnls start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.133+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:53 13.b 13:d0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b400 con 0x55c6ab23ac00 2024-04-24T08:17:09.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.134+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.5 v2:172.21.15.121:6822/2187246263 5 ==== osd_op_reply(53 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab23ac00 2024-04-24T08:17:09.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.134+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:54 13.2b 13:d4000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23bc00 con 0x55c6ab21b400 2024-04-24T08:17:09.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.135+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.2 v2:172.21.15.69:6808/2630808717 9 ==== osd_op_reply(54 [pgnls start_epoch 86 out=49b] v82'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21b400 2024-04-24T08:17:09.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.135+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:55 13.1b 13:d8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23b000 con 0x55c6ab21a800 2024-04-24T08:17:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.136+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 9 ==== osd_op_reply(55 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.137+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:56 13.3b 13:dc000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab23a400 con 0x55c6ab21a800 2024-04-24T08:17:09.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.138+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 10 ==== osd_op_reply(56 [pgnls start_epoch 86 out=49b] v82'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.138+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:57 13.7 13:e0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24f400 con 0x55c6ab202400 2024-04-24T08:17:09.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.140+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 9 ==== osd_op_reply(57 [pgnls start_epoch 86 out=81b] v82'6 uv6 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.140+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:58 13.7 13:e2e07930:::10000000001.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab202c00 con 0x55c6ab202400 2024-04-24T08:17:09.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.141+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 10 ==== osd_op_reply(58 10000000001.00000000 [stat out=16b] v0'0 uv6 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.141+0000 7fe45e4ba740 10 datascan.operator(): handling object 1099511627777.0 2024-04-24T08:17:09.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.141+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:59 13.7 13:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab203000 con 0x55c6ab202400 2024-04-24T08:17:09.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.145+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.7 v2:172.21.15.121:6811/176144805 11 ==== osd_op_reply(59 10000000001.00000000 [call] v86'7 uv7 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab202400 2024-04-24T08:17:09.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.145+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:60 13.27 13:e4000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab203400 con 0x55c6ab21ac00 2024-04-24T08:17:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.146+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.1 v2:172.21.15.69:6815/153736297 9 ==== osd_op_reply(60 [pgnls start_epoch 86 out=49b] v82'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab21ac00 2024-04-24T08:17:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.146+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:61 13.17 13:e8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24f800 con 0x55c6ab23ac00 2024-04-24T08:17:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.148+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.5 v2:172.21.15.121:6822/2187246263 6 ==== osd_op_reply(61 [pgnls start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab23ac00 2024-04-24T08:17:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.148+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:62 13.37 13:ec000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab21b000 con 0x55c6ab23ac00 2024-04-24T08:17:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.149+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 <== osd.5 v2:172.21.15.121:6822/2187246263 7 ==== osd_op_reply(62 [pgnls start_epoch 86 out=49b] v82'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24780 con 0x55c6ab23ac00 2024-04-24T08:17:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.149+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:63 13.f 13:f0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24e400 con 0x55c6ab21a000 2024-04-24T08:17:09.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.151+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 9 ==== osd_op_reply(63 [pgnls start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.151+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:64 13.2f 13:f4000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab202800 con 0x55c6ab21a800 2024-04-24T08:17:09.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.152+0000 7fe45bc00640 1 -- 172.21.15.69:0/25220706 <== osd.4 v2:172.21.15.121:6800/2274643300 11 ==== osd_op_reply(64 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24000 con 0x55c6ab21a800 2024-04-24T08:17:09.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.152+0000 7fe456c00640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:65 13.1f 13:f8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24e800 con 0x55c6ab21a000 2024-04-24T08:17:09.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.153+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 10 ==== osd_op_reply(65 [pgnls start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.153+0000 7fe456200640 1 -- 172.21.15.69:0/25220706 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:66 13.3f 13:fc000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55c6ab24ec00 con 0x55c6ab21a000 2024-04-24T08:17:09.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45d000640 1 -- 172.21.15.69:0/25220706 <== osd.6 v2:172.21.15.121:6807/3068198325 11 ==== osd_op_reply(66 [pgnls start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c6abc24500 con 0x55c6ab21a000 2024-04-24T08:17:09.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55c6ab23a000 msgr2=0x55c6ab221b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55c6ab23a000 0x55c6ab221b80 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55c6ab21ac00 msgr2=0x55c6ab220580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55c6ab21ac00 0x55c6ab220580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55c6ab21b400 msgr2=0x55c6ab221600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55c6ab21b400 0x55c6ab221600 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55c6ab21bc00 msgr2=0x55c6ab222100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55c6ab21bc00 0x55c6ab222100 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55c6ab21a800 msgr2=0x55c6ab220b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55c6ab21a800 0x55c6ab220b00 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55c6ab23ac00 msgr2=0x55c6ab221080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55c6ab23ac00 0x55c6ab221080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55c6ab21a000 msgr2=0x55c6ab220000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55c6ab21a000 0x55c6ab220000 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55c6ab202400 msgr2=0x55c6abc6c680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55c6ab202400 0x55c6abc6c680 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55c6ab202000 msgr2=0x55c6abc6c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55c6ab202000 0x55c6abc6c100 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x55c6abc39020 tx=0x55c6abd80ea0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 msgr2=0x55c6abc6b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/25220706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6abd69400 0x55c6abc6b080 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x55c6abd7cab0 tx=0x55c6abc36420 comp rx=0 tx=0).stop 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 reap_dead start 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 reap_dead start 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 reap_dead start 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 reap_dead start 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 shutdown_connections 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 >> 172.21.15.69:0/25220706 conn(0x55c6abd69000 msgr2=0x55c6aaf65000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.155+0000 7fe45c600640 1 -- 172.21.15.69:0/25220706 reap_dead start 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.156+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 shutdown_connections 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.156+0000 7fe45e4ba740 1 -- 172.21.15.69:0/25220706 wait complete. 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.157+0000 7fe45e4ba740 1 -- 172.21.15.69:0/3753673765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6aae97800 msgr2=0x55c6abc6a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.157+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/3753673765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6aae97800 0x55c6abc6a580 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x55c6abcafc50 tx=0x55c6abc3a120 comp rx=0 tx=0).stop 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.157+0000 7fe45e4ba740 1 -- 172.21.15.69:0/3753673765 shutdown_connections 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.157+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/3753673765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c6aae97800 0x55c6abc6a580 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.157+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/3753673765 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c6aae97c00 0x55c6abc6ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.157+0000 7fe45e4ba740 1 --2- 172.21.15.69:0/3753673765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c6aae96c00 0x55c6abc6a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.157+0000 7fe45e4ba740 1 -- 172.21.15.69:0/3753673765 >> 172.21.15.69:0/3753673765 conn(0x55c6aae97000 msgr2=0x55c6aaf64800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.157+0000 7fe45e4ba740 1 -- 172.21.15.69:0/3753673765 shutdown_connections 2024-04-24T08:17:09.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.157+0000 7fe45e4ba740 1 -- 172.21.15.69:0/3753673765 wait complete. 2024-04-24T08:17:09.170 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.209164, result: 2024-04-24T08:17:09.170 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes cephfs_data 2024-04-24T08:17:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.313+0000 7f0d6511b740 1 -- start start 2024-04-24T08:17:09.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.313+0000 7f0d6511b740 1 Processor -- start 2024-04-24T08:17:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.314+0000 7f0d6511b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faacf2cc00 0x55faadd00000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.314+0000 7f0d6511b740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faacf2d800 0x55faadd00580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.314+0000 7f0d6511b740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faacf2dc00 0x55faadd00b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.314+0000 7f0d6511b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55faadc9c600 con 0x55faacf2cc00 2024-04-24T08:17:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.314+0000 7f0d6511b740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55faadc9c480 con 0x55faacf2dc00 2024-04-24T08:17:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.314+0000 7f0d6511b740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55faadc9c300 con 0x55faacf2d800 2024-04-24T08:17:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d62800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faacf2cc00 0x55faadd00000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d62800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faacf2cc00 0x55faadd00000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46556/0 (socket says 172.21.15.69:46556) 2024-04-24T08:17:09.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d62800640 1 -- 172.21.15.69:0/509357628 learned_addr learned my addr 172.21.15.69:0/509357628 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:09.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d63200640 1 --2- 172.21.15.69:0/509357628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faacf2dc00 0x55faadd00b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d61e00640 1 --2- 172.21.15.69:0/509357628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faacf2d800 0x55faadd00580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d62800640 1 -- 172.21.15.69:0/509357628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faacf2d800 msgr2=0x55faadd00580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d62800640 1 --2- 172.21.15.69:0/509357628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faacf2d800 0x55faadd00580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d62800640 1 -- 172.21.15.69:0/509357628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faacf2dc00 msgr2=0x55faadd00b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d62800640 1 --2- 172.21.15.69:0/509357628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faacf2dc00 0x55faadd00b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d62800640 1 -- 172.21.15.69:0/509357628 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faacf59680 con 0x55faacf2cc00 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.315+0000 7f0d63200640 1 --2- 172.21.15.69:0/509357628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faacf2dc00 0x55faadd00b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.316+0000 7f0d62800640 1 --2- 172.21.15.69:0/509357628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faacf2cc00 0x55faadd00000 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x55faadd458f0 tx=0x55faadcd00c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899db8107cc9fef9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.316+0000 7f0d5f600640 1 -- 172.21.15.69:0/509357628 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55faacf52680 con 0x55faacf2cc00 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.316+0000 7f0d5f600640 1 -- 172.21.15.69:0/509357628 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55faacf76540 con 0x55faacf2cc00 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.316+0000 7f0d5f600640 1 -- 172.21.15.69:0/509357628 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55faacf52b60 con 0x55faacf2cc00 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.316+0000 7f0d6511b740 1 -- 172.21.15.69:0/509357628 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faacf594a0 con 0x55faacf2cc00 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.316+0000 7f0d6511b740 1 -- 172.21.15.69:0/509357628 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55faacf59a40 con 0x55faacf2cc00 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.316+0000 7f0d6511b740 4 waiting for MDS map... 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.316+0000 7f0d5f600640 1 -- 172.21.15.69:0/509357628 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x55faade02600 con 0x55faacf2cc00 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.316+0000 7f0d5f600640 1 -- 172.21.15.69:0/509357628 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 69) ==== 2885+0+0 (secure 0 0 0) 0x55faacf85180 con 0x55faacf2cc00 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.316+0000 7f0d6511b740 4 Got MDS map 69 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.317+0000 7f0d6511b740 4 datascan.main: Using metadata pool output 2024-04-24T08:17:09.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.317+0000 7f0d6511b740 4 datascan.main: connecting to RADOS... 2024-04-24T08:17:09.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.317+0000 7f0d6511b740 1 Processor -- start 2024-04-24T08:17:09.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.318+0000 7f0d6511b740 1 -- start start 2024-04-24T08:17:09.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.318+0000 7f0d6511b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05400 0x55faadd01080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.318+0000 7f0d6511b740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faade05800 0x55faadd01600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.318+0000 7f0d6511b740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faade05c00 0x55faadd01b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.318+0000 7f0d6511b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55faadc9c180 con 0x55faade05400 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.318+0000 7f0d6511b740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55faacf8c480 con 0x55faade05c00 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.318+0000 7f0d6511b740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55faacf8c900 con 0x55faade05800 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d63200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05400 0x55faadd01080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d63200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05400 0x55faadd01080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46566/0 (socket says 172.21.15.69:46566) 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d63200640 1 -- 172.21.15.69:0/1911170072 learned_addr learned my addr 172.21.15.69:0/1911170072 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d63200640 1 -- 172.21.15.69:0/1911170072 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faade05800 msgr2=0x55faadd01600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d62800640 1 --2- 172.21.15.69:0/1911170072 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faade05800 0x55faadd01600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d63200640 1 --2- 172.21.15.69:0/1911170072 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faade05800 0x55faadd01600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d63200640 1 -- 172.21.15.69:0/1911170072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faade05c00 msgr2=0x55faadd01b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d61e00640 1 --2- 172.21.15.69:0/1911170072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faade05c00 0x55faadd01b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d63200640 1 --2- 172.21.15.69:0/1911170072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faade05c00 0x55faadd01b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d63200640 1 -- 172.21.15.69:0/1911170072 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faacf59860 con 0x55faade05400 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.319+0000 7f0d63200640 1 --2- 172.21.15.69:0/1911170072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05400 0x55faadd01080 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x55faadd451d0 tx=0x55faadccc7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7baf2fa0663980a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d5ce00640 1 -- 172.21.15.69:0/1911170072 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55faacf529c0 con 0x55faade05400 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d5ce00640 1 -- 172.21.15.69:0/1911170072 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55faacf76c40 con 0x55faade05400 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d5ce00640 1 -- 172.21.15.69:0/1911170072 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55faacf52680 con 0x55faade05400 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d6511b740 1 -- 172.21.15.69:0/1911170072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05400 msgr2=0x55faadd01080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d6511b740 1 --2- 172.21.15.69:0/1911170072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05400 0x55faadd01080 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x55faadd451d0 tx=0x55faadccc7e0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d6511b740 1 -- 172.21.15.69:0/1911170072 shutdown_connections 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d6511b740 1 --2- 172.21.15.69:0/1911170072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faade05c00 0x55faadd01b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d6511b740 1 --2- 172.21.15.69:0/1911170072 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faade05800 0x55faadd01600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d6511b740 1 --2- 172.21.15.69:0/1911170072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05400 0x55faadd01080 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d6511b740 1 -- 172.21.15.69:0/1911170072 >> 172.21.15.69:0/1911170072 conn(0x55faade05000 msgr2=0x55faacff8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d6511b740 1 -- 172.21.15.69:0/1911170072 shutdown_connections 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.320+0000 7f0d6511b740 1 -- 172.21.15.69:0/1911170072 wait complete. 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.321+0000 7f0d6511b740 1 Processor -- start 2024-04-24T08:17:09.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.321+0000 7f0d6511b740 1 -- start start 2024-04-24T08:17:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.322+0000 7f0d6511b740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faade05400 0x55faadd01080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.322+0000 7f0d6511b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05800 0x55faadd01600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.322+0000 7f0d6511b740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faade05c00 0x55faadd01b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.322+0000 7f0d6511b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55faadc9d200 con 0x55faade05800 2024-04-24T08:17:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.322+0000 7f0d6511b740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55faadc9c180 con 0x55faade05c00 2024-04-24T08:17:09.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.322+0000 7f0d6511b740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55faadc9d380 con 0x55faade05400 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.322+0000 7f0d62800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05800 0x55faadd01600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.322+0000 7f0d62800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05800 0x55faadd01600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46574/0 (socket says 172.21.15.69:46574) 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.322+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 learned_addr learned my addr 172.21.15.69:0/3675217069 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d63200640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faade05400 0x55faadd01080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d61e00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faade05c00 0x55faadd01b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faade05400 msgr2=0x55faadd01080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d62800640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faade05400 0x55faadd01080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faade05c00 msgr2=0x55faadd01b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d62800640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faade05c00 0x55faadd01b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55faacf59680 con 0x55faade05800 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d63200640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faade05400 0x55faadd01080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d61e00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faade05c00 0x55faadd01b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d62800640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05800 0x55faadd01600 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x55faade13b60 tx=0x55faadccd980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d5b000640 1 -- 172.21.15.69:0/3675217069 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55faacf52820 con 0x55faade05800 2024-04-24T08:17:09.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d5b000640 1 -- 172.21.15.69:0/3675217069 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55faacf76a80 con 0x55faade05800 2024-04-24T08:17:09.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d5b000640 1 -- 172.21.15.69:0/3675217069 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55faacf529c0 con 0x55faade05800 2024-04-24T08:17:09.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.323+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55faacf594a0 con 0x55faade05800 2024-04-24T08:17:09.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.324+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55faacf59a40 con 0x55faade05800 2024-04-24T08:17:09.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.324+0000 7f0d6511b740 4 datascan.init: resolving metadata pool 12 2024-04-24T08:17:09.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.325+0000 7f0d5b000640 1 -- 172.21.15.69:0/3675217069 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55faadd24c00 con 0x55faade05800 2024-04-24T08:17:09.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.326+0000 7f0d5b000640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55faade6d000 0x55faadd02100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.326+0000 7f0d5b000640 1 -- 172.21.15.69:0/3675217069 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x55faadd2c600 con 0x55faade05800 2024-04-24T08:17:09.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.326+0000 7f0d6511b740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-24T08:17:09.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.326+0000 7f0d6511b740 4 datascan.main: data pool 'cephfs_data' has ID 13 2024-04-24T08:17:09.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.326+0000 7f0d6511b740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-24T08:17:09.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.326+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55faade6d400 0x55faadd02680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.326+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:1 12.14 12:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55faade6d800 con 0x55faade6d400 2024-04-24T08:17:09.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.327+0000 7f0d63200640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55faade6d000 0x55faadd02100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.327+0000 7f0d61e00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55faade6d400 0x55faadd02680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.328+0000 7f0d61e00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55faade6d400 0x55faadd02680 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.328+0000 7f0d63200640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55faade6d000 0x55faadd02100 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x55faadd451d0 tx=0x55faadccc9c0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.328+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv25 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.328+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55faade6dc00 0x55faadd02c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.328+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:2 12.1f 12:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55faad2aa000 con 0x55faade6dc00 2024-04-24T08:17:09.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.329+0000 7f0d63200640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55faade6dc00 0x55faadd02c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.330+0000 7f0d63200640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55faade6dc00 0x55faadd02c00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.330+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.331+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55faad2aa400 0x55faadd03180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.331+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:3 13.0 13:00000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2aa800 con 0x55faad2aa400 2024-04-24T08:17:09.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.331+0000 7f0d62800640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55faad2aa400 0x55faadd03180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.332+0000 7f0d62800640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55faad2aa400 0x55faadd03180 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.333+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.333+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:4 13.20 13:04000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2ab800 con 0x55faade6dc00 2024-04-24T08:17:09.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.334+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 2 ==== osd_op_reply(4 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.335+0000 7f0d5ce00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55faad2abc00 0x55faad2c4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.335+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:5 13.10 13:08000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8000 con 0x55faad2abc00 2024-04-24T08:17:09.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.335+0000 7f0d61e00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55faad2abc00 0x55faad2c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.336+0000 7f0d61e00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55faad2abc00 0x55faad2c4000 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.336+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(5 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.336+0000 7f0d5c400640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55faad2c8000 0x55faad2c4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.336+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:6 13.30 13:0c000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6d800 con 0x55faad2c8000 2024-04-24T08:17:09.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.337+0000 7f0d63200640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55faad2c8000 0x55faad2c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.338+0000 7f0d63200640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55faad2c8000 0x55faad2c4580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.339+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(6 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.339+0000 7f0d5ce00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55faad2c8400 0x55faad2c4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.339+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:7 13.8 13:10000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2aa000 con 0x55faad2c8400 2024-04-24T08:17:09.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.339+0000 7f0d62800640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55faad2c8400 0x55faad2c4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.340+0000 7f0d62800640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55faad2c8400 0x55faad2c4b00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.340+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== osd_op_reply(7 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.340+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:8 13.28 13:14000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8800 con 0x55faad2c8400 2024-04-24T08:17:09.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.340+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 2 ==== osd_op_reply(8 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.340+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:9 13.18 13:18000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faadccb400 con 0x55faad2aa400 2024-04-24T08:17:09.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.341+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 2 ==== osd_op_reply(9 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.341+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:10 13.38 13:1c000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6d800 con 0x55faade6d400 2024-04-24T08:17:09.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.342+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 2 ==== osd_op_reply(10 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.342+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:11 13.4 13:20000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faadccb000 con 0x55faade6dc00 2024-04-24T08:17:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.344+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 3 ==== osd_op_reply(11 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.344+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:12 13.24 13:24000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2ab800 con 0x55faade6dc00 2024-04-24T08:17:09.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.345+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 4 ==== osd_op_reply(12 [pgnls-filter start_epoch 86 out=49b] v82'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.345+0000 7f0d5ce00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55faad2aa800 0x55faad2c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.345+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:13 13.14 13:28000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2aa000 con 0x55faad2aa800 2024-04-24T08:17:09.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.345+0000 7f0d61e00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55faad2aa800 0x55faad2c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.346+0000 7f0d61e00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55faad2aa800 0x55faad2c5080 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.346+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== osd_op_reply(13 [pgnls-filter start_epoch 86 out=49b] v82'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.346+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:14 13.34 13:2c000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8800 con 0x55faade6dc00 2024-04-24T08:17:09.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.347+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 5 ==== osd_op_reply(14 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.347+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:15 13.c 13:30000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6cc00 con 0x55faad2abc00 2024-04-24T08:17:09.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.348+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 2 ==== osd_op_reply(15 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.348+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:16 13.2c 13:34000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8c00 con 0x55faad2c8400 2024-04-24T08:17:09.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.348+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 3 ==== osd_op_reply(16 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.348+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:17 13.1c 13:38000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6c800 con 0x55faade6dc00 2024-04-24T08:17:09.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.349+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 6 ==== osd_op_reply(17 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.350+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:18 13.3c 13:3c000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9000 con 0x55faad2c8400 2024-04-24T08:17:09.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.350+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 4 ==== osd_op_reply(18 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.350+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:19 13.2 13:40000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9400 con 0x55faade6d400 2024-04-24T08:17:09.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.351+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 3 ==== osd_op_reply(19 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.351+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:20 13.22 13:44000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faadccb400 con 0x55faad2abc00 2024-04-24T08:17:09.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.352+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 3 ==== osd_op_reply(20 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.352+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:21 13.12 13:48000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9800 con 0x55faad2aa800 2024-04-24T08:17:09.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.352+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 2 ==== osd_op_reply(21 [pgnls-filter start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.352+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:22 13.32 13:4c000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9c00 con 0x55faad2c8400 2024-04-24T08:17:09.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.353+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 5 ==== osd_op_reply(22 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.353+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:23 13.a 13:50000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6d800 con 0x55faad2c8400 2024-04-24T08:17:09.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.354+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 6 ==== osd_op_reply(23 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.354+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:24 13.2a 13:54000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8000 con 0x55faad2abc00 2024-04-24T08:17:09.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.354+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 4 ==== osd_op_reply(24 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.354+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:25 13.1a 13:58000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6cc00 con 0x55faad2abc00 2024-04-24T08:17:09.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.355+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 5 ==== osd_op_reply(25 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.355+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:26 13.3a 13:5c000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8400 con 0x55faade6d400 2024-04-24T08:17:09.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.356+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 4 ==== osd_op_reply(26 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.356+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:27 13.6 13:60000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2aa000 con 0x55faad2c8000 2024-04-24T08:17:09.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.357+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== osd_op_reply(27 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.357+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:28 13.26 13:64000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8800 con 0x55faad2aa400 2024-04-24T08:17:09.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.358+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 3 ==== osd_op_reply(28 [pgnls-filter start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.358+0000 7f0d5ce00640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55faad2d8000 0x55faad2c5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.358+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:29 13.16 13:68000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9800 con 0x55faad2d8000 2024-04-24T08:17:09.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.358+0000 7f0d63200640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55faad2d8000 0x55faad2c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.359+0000 7f0d63200640 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55faad2d8000 0x55faad2c5600 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.360+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(29 [pgnls-filter start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2d8000 2024-04-24T08:17:09.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.360+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:30 13.36 13:6c000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8c00 con 0x55faade6dc00 2024-04-24T08:17:09.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.361+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 7 ==== osd_op_reply(30 [pgnls-filter start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.361+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:31 13.e 13:70000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9c00 con 0x55faad2aa400 2024-04-24T08:17:09.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.362+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 4 ==== osd_op_reply(31 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.362+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:32 13.2e 13:74000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9000 con 0x55faad2c8400 2024-04-24T08:17:09.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.363+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 7 ==== osd_op_reply(32 [pgnls-filter start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.363+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:33 13.1e 13:78000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9400 con 0x55faad2c8000 2024-04-24T08:17:09.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.364+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 3 ==== osd_op_reply(33 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.364+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:34 13.3e 13:7c000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9400 con 0x55faad2abc00 2024-04-24T08:17:09.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.365+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 6 ==== osd_op_reply(34 [pgnls-filter start_epoch 86 out=49b] v82'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.365+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:35 13.1 13:80000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9800 con 0x55faade6dc00 2024-04-24T08:17:09.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.366+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 8 ==== osd_op_reply(35 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.366+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:36 13.21 13:84000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6d800 con 0x55faad2c8000 2024-04-24T08:17:09.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.367+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 4 ==== osd_op_reply(36 [pgnls-filter start_epoch 86 out=49b] v82'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.367+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:37 13.11 13:88000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faadccb400 con 0x55faad2d8000 2024-04-24T08:17:09.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.368+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.3 v2:172.21.15.69:6811/1875112303 2 ==== osd_op_reply(37 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2d8000 2024-04-24T08:17:09.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.368+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:38 13.31 13:8c000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6c800 con 0x55faad2d8000 2024-04-24T08:17:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.369+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.3 v2:172.21.15.69:6811/1875112303 3 ==== osd_op_reply(38 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2d8000 2024-04-24T08:17:09.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.369+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:39 13.9 13:90000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8800 con 0x55faade6d400 2024-04-24T08:17:09.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.370+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 5 ==== osd_op_reply(39 [pgnls-filter start_epoch 86 out=49b] v82'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.370+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:40 13.29 13:94000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8c00 con 0x55faade6dc00 2024-04-24T08:17:09.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.371+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 9 ==== osd_op_reply(40 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.371+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:41 13.19 13:98000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9c00 con 0x55faad2d8000 2024-04-24T08:17:09.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.372+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.3 v2:172.21.15.69:6811/1875112303 4 ==== osd_op_reply(41 [pgnls-filter start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2d8000 2024-04-24T08:17:09.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.372+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:42 13.39 13:9c000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9000 con 0x55faad2c8400 2024-04-24T08:17:09.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.373+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 8 ==== osd_op_reply(42 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.373+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:43 13.5 13:a0000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9000 con 0x55faad2c8000 2024-04-24T08:17:09.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.374+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 5 ==== osd_op_reply(43 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.374+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:44 13.25 13:a4000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9c00 con 0x55faad2c8000 2024-04-24T08:17:09.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.375+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 6 ==== osd_op_reply(44 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.375+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:45 13.15 13:a8000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9800 con 0x55faad2c8000 2024-04-24T08:17:09.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.376+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 7 ==== osd_op_reply(45 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.376+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:46 13.35 13:ac000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8800 con 0x55faad2abc00 2024-04-24T08:17:09.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.377+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 7 ==== osd_op_reply(46 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.377+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:47 13.d 13:b0000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2ab800 con 0x55faad2c8000 2024-04-24T08:17:09.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.378+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 8 ==== osd_op_reply(47 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.379+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:48 13.2d 13:b4000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faadccb000 con 0x55faad2aa800 2024-04-24T08:17:09.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.379+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 3 ==== osd_op_reply(48 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.379+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:49 13.1d 13:b8000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2aa000 con 0x55faad2aa400 2024-04-24T08:17:09.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.381+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 5 ==== osd_op_reply(49 [pgnls-filter start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.381+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:50 13.3d 13:bc000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6cc00 con 0x55faad2aa400 2024-04-24T08:17:09.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.382+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 6 ==== osd_op_reply(50 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.382+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:51 13.3 13:c0000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8800 con 0x55faad2aa800 2024-04-24T08:17:09.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.382+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 4 ==== osd_op_reply(51 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.383+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:52 13.23 13:c4000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6d800 con 0x55faad2abc00 2024-04-24T08:17:09.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.383+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 8 ==== osd_op_reply(52 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.383+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:53 13.13 13:c8000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9400 con 0x55faad2aa400 2024-04-24T08:17:09.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.385+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 7 ==== osd_op_reply(53 [pgnls-filter start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.385+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:54 13.33 13:cc000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6c800 con 0x55faad2aa400 2024-04-24T08:17:09.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.386+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 8 ==== osd_op_reply(54 [pgnls-filter start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.386+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:55 13.b 13:d0000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9800 con 0x55faade6d400 2024-04-24T08:17:09.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.387+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 6 ==== osd_op_reply(55 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.387+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:56 13.2b 13:d4000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faadccb400 con 0x55faad2c8400 2024-04-24T08:17:09.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.388+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 9 ==== osd_op_reply(56 [pgnls-filter start_epoch 86 out=49b] v82'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.388+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:57 13.1b 13:d8000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8c00 con 0x55faad2c8000 2024-04-24T08:17:09.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.389+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 9 ==== osd_op_reply(57 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.390+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:58 13.3b 13:dc000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8400 con 0x55faad2c8000 2024-04-24T08:17:09.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.391+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 10 ==== osd_op_reply(58 [pgnls-filter start_epoch 86 out=49b] v82'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.391+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:59 13.7 13:e0000000::::head [pgnls-filter start_epoch 86 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8000 con 0x55faad2aa400 2024-04-24T08:17:09.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.392+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 9 ==== osd_op_reply(59 [pgnls-filter start_epoch 86 out=81b] v86'7 uv7 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.392+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:60 13.0 13:00000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2aac00 con 0x55faad2aa400 2024-04-24T08:17:09.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.394+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 10 ==== osd_op_reply(60 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.394+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:61 13.20 13:04000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8400 con 0x55faade6dc00 2024-04-24T08:17:09.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.395+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 10 ==== osd_op_reply(61 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.395+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:62 13.10 13:08000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9400 con 0x55faad2abc00 2024-04-24T08:17:09.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.396+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 9 ==== osd_op_reply(62 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.396+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:63 13.30 13:0c000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8800 con 0x55faad2c8000 2024-04-24T08:17:09.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.397+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 11 ==== osd_op_reply(63 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.397+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:64 13.8 13:10000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8800 con 0x55faad2c8400 2024-04-24T08:17:09.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.398+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 10 ==== osd_op_reply(64 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.398+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:65 13.28 13:14000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8c00 con 0x55faad2c8400 2024-04-24T08:17:09.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.398+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 11 ==== osd_op_reply(65 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.398+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:66 13.18 13:18000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8c00 con 0x55faad2aa400 2024-04-24T08:17:09.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.399+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 11 ==== osd_op_reply(66 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.399+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:67 13.38 13:1c000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e9000 con 0x55faade6d400 2024-04-24T08:17:09.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.401+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 7 ==== osd_op_reply(67 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.401+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:68 13.4 13:20000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8800 con 0x55faade6dc00 2024-04-24T08:17:09.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.402+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 11 ==== osd_op_reply(68 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.402+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:69 13.24 13:24000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e9400 con 0x55faade6dc00 2024-04-24T08:17:09.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.404+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 12 ==== osd_op_reply(69 [pgnls-filter start_epoch 86 out=49b] v82'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.404+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:70 13.14 13:28000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9800 con 0x55faad2aa800 2024-04-24T08:17:09.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.404+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 5 ==== osd_op_reply(70 [pgnls-filter start_epoch 86 out=49b] v82'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.404+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:71 13.34 13:2c000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e9800 con 0x55faade6dc00 2024-04-24T08:17:09.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.406+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 13 ==== osd_op_reply(71 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.406+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:72 13.c 13:30000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9c00 con 0x55faad2abc00 2024-04-24T08:17:09.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.407+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 10 ==== osd_op_reply(72 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.407+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:73 13.2c 13:34000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e9c00 con 0x55faad2c8400 2024-04-24T08:17:09.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.407+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 12 ==== osd_op_reply(73 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.408+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:74 13.1c 13:38000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8c00 con 0x55faade6dc00 2024-04-24T08:17:09.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.409+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 14 ==== osd_op_reply(74 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.409+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:75 13.3c 13:3c000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e9000 con 0x55faad2c8400 2024-04-24T08:17:09.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.410+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 13 ==== osd_op_reply(75 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.410+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:76 13.2 13:40000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9400 con 0x55faade6d400 2024-04-24T08:17:09.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.411+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 8 ==== osd_op_reply(76 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.411+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:77 13.22 13:44000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8400 con 0x55faad2abc00 2024-04-24T08:17:09.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.412+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 11 ==== osd_op_reply(77 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.412+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:78 13.12 13:48000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9c00 con 0x55faad2aa800 2024-04-24T08:17:09.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.413+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 6 ==== osd_op_reply(78 [pgnls-filter start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.413+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:79 13.32 13:4c000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8000 con 0x55faad2c8400 2024-04-24T08:17:09.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.414+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 14 ==== osd_op_reply(79 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.414+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:80 13.a 13:50000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9000 con 0x55faad2c8400 2024-04-24T08:17:09.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.415+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 15 ==== osd_op_reply(80 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.415+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:81 13.2a 13:54000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faadccb000 con 0x55faad2abc00 2024-04-24T08:17:09.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.416+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 12 ==== osd_op_reply(81 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.416+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:82 13.1a 13:58000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9800 con 0x55faad2abc00 2024-04-24T08:17:09.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.417+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 13 ==== osd_op_reply(82 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.417+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:83 13.3a 13:5c000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faadccb400 con 0x55faade6d400 2024-04-24T08:17:09.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.418+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 9 ==== osd_op_reply(83 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.419+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:84 13.6 13:60000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8400 con 0x55faad2c8000 2024-04-24T08:17:09.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.420+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 12 ==== osd_op_reply(84 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.420+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:85 13.26 13:64000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6d800 con 0x55faad2aa400 2024-04-24T08:17:09.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.421+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 12 ==== osd_op_reply(85 [pgnls-filter start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.421+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:86 13.16 13:68000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2ab800 con 0x55faad2d8000 2024-04-24T08:17:09.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.422+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.3 v2:172.21.15.69:6811/1875112303 5 ==== osd_op_reply(86 [pgnls-filter start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2d8000 2024-04-24T08:17:09.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.422+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:87 13.36 13:6c000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6cc00 con 0x55faade6dc00 2024-04-24T08:17:09.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.424+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 15 ==== osd_op_reply(87 [pgnls-filter start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.424+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:88 13.e 13:70000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2aa000 con 0x55faad2aa400 2024-04-24T08:17:09.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.425+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 13 ==== osd_op_reply(88 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.425+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:89 13.2e 13:74000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6c800 con 0x55faad2c8400 2024-04-24T08:17:09.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.426+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 16 ==== osd_op_reply(89 [pgnls-filter start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.426+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:90 13.1e 13:78000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2aac00 con 0x55faad2c8000 2024-04-24T08:17:09.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.427+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 13 ==== osd_op_reply(90 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.427+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:91 13.3e 13:7c000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9000 con 0x55faad2abc00 2024-04-24T08:17:09.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.428+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 14 ==== osd_op_reply(91 [pgnls-filter start_epoch 86 out=49b] v82'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.428+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:92 13.1 13:80000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9400 con 0x55faade6dc00 2024-04-24T08:17:09.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.429+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 16 ==== osd_op_reply(92 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.429+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:93 13.21 13:84000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9400 con 0x55faad2c8000 2024-04-24T08:17:09.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.430+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 14 ==== osd_op_reply(93 [pgnls-filter start_epoch 86 out=49b] v82'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.431+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:94 13.11 13:88000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9c00 con 0x55faad2d8000 2024-04-24T08:17:09.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.431+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.3 v2:172.21.15.69:6811/1875112303 6 ==== osd_op_reply(94 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2d8000 2024-04-24T08:17:09.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.431+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:95 13.31 13:8c000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8400 con 0x55faad2d8000 2024-04-24T08:17:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.432+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.3 v2:172.21.15.69:6811/1875112303 7 ==== osd_op_reply(95 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2d8000 2024-04-24T08:17:09.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.432+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:96 13.9 13:90000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9800 con 0x55faade6d400 2024-04-24T08:17:09.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.434+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 10 ==== osd_op_reply(96 [pgnls-filter start_epoch 86 out=49b] v82'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.434+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:97 13.29 13:94000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8800 con 0x55faade6dc00 2024-04-24T08:17:09.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.435+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 17 ==== osd_op_reply(97 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.435+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:98 13.19 13:98000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faadccb000 con 0x55faad2d8000 2024-04-24T08:17:09.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.436+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.3 v2:172.21.15.69:6811/1875112303 8 ==== osd_op_reply(98 [pgnls-filter start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2d8000 2024-04-24T08:17:09.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.436+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:99 13.39 13:9c000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8800 con 0x55faad2c8400 2024-04-24T08:17:09.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.437+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 17 ==== osd_op_reply(99 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.437+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:100 13.5 13:a0000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8400 con 0x55faad2c8000 2024-04-24T08:17:09.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.438+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 15 ==== osd_op_reply(100 [pgnls-filter start_epoch 86 out=49b] v82'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.438+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:101 13.25 13:a4000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e9400 con 0x55faad2c8000 2024-04-24T08:17:09.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.439+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 16 ==== osd_op_reply(101 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.439+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:102 13.15 13:a8000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9800 con 0x55faad2c8000 2024-04-24T08:17:09.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.440+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 17 ==== osd_op_reply(102 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.441+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:103 13.35 13:ac000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e9800 con 0x55faad2abc00 2024-04-24T08:17:09.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.442+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 15 ==== osd_op_reply(103 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.442+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:104 13.d 13:b0000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9c00 con 0x55faad2c8000 2024-04-24T08:17:09.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.442+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 18 ==== osd_op_reply(104 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.443+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:105 13.2d 13:b4000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8c00 con 0x55faad2aa800 2024-04-24T08:17:09.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.444+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 7 ==== osd_op_reply(105 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.444+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:106 13.1d 13:b8000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9400 con 0x55faad2aa400 2024-04-24T08:17:09.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.445+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 14 ==== osd_op_reply(106 [pgnls-filter start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.445+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:107 13.3d 13:bc000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e9c00 con 0x55faad2aa400 2024-04-24T08:17:09.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.446+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 15 ==== osd_op_reply(107 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.446+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:108 13.3 13:c0000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d9c00 con 0x55faad2aa800 2024-04-24T08:17:09.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.447+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 8 ==== osd_op_reply(108 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.447+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:109 13.23 13:c4000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e9000 con 0x55faad2abc00 2024-04-24T08:17:09.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.448+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 16 ==== osd_op_reply(109 [pgnls-filter start_epoch 86 out=49b] v82'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.448+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:110 13.13 13:c8000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8400 con 0x55faad2aa400 2024-04-24T08:17:09.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.450+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 16 ==== osd_op_reply(110 [pgnls-filter start_epoch 86 out=49b] v82'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.450+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:111 13.33 13:cc000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8000 con 0x55faad2aa400 2024-04-24T08:17:09.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.451+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 17 ==== osd_op_reply(111 [pgnls-filter start_epoch 86 out=49b] v82'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.451+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:112 13.b 13:d0000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8800 con 0x55faade6d400 2024-04-24T08:17:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.452+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 11 ==== osd_op_reply(112 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.452+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:113 13.2b 13:d4000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8800 con 0x55faad2c8400 2024-04-24T08:17:09.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.453+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.2 v2:172.21.15.69:6808/2630808717 18 ==== osd_op_reply(113 [pgnls-filter start_epoch 86 out=49b] v82'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba780 con 0x55faad2c8400 2024-04-24T08:17:09.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.453+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:114 13.1b 13:d8000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2d8c00 con 0x55faad2c8000 2024-04-24T08:17:09.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.454+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 19 ==== osd_op_reply(114 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.454+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:115 13.3b 13:dc000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8c00 con 0x55faad2c8000 2024-04-24T08:17:09.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.456+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 20 ==== osd_op_reply(115 [pgnls-filter start_epoch 86 out=49b] v82'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.456+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:116 13.7 13:e0000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9400 con 0x55faad2aa400 2024-04-24T08:17:09.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.458+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 18 ==== osd_op_reply(116 [pgnls-filter start_epoch 86 out=81b] v86'7 uv7 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.458+0000 7f0d6511b740 20 datascan.forall_objects: OSD matched oid 10000000001.00000000 2024-04-24T08:17:09.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.458+0000 7f0d6511b740 10 datascan.operator(): handling object 10000000001.0 2024-04-24T08:17:09.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.458+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:117 13.7 13:e2e07930:::10000000001.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55faad2ab400 con 0x55faad2aa400 2024-04-24T08:17:09.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.459+0000 7f0d62800640 1 -- 172.21.15.69:0/3675217069 <== osd.7 v2:172.21.15.121:6811/176144805 19 ==== osd_op_reply(117 10000000001.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (89) out=89b,getxattr (30) out=30b,getxattr] v0'0 uv7 ondisk = 0) ==== 416+0+157 (crc 0 0 0) 0x55faadcba780 con 0x55faad2aa400 2024-04-24T08:17:09.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.459+0000 7f0d6511b740 10 datascan.inject_with_backtrace: inode: 0x0x10000000001 2024-04-24T08:17:09.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.459+0000 7f0d6511b740 10 datascan.inject_with_backtrace: backptr: 0x0x10000000000/file 2024-04-24T08:17:09.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.459+0000 7f0d6511b740 20 datascan.get_frag_of: dirino=0x10000000000 target_dname=file 2024-04-24T08:17:09.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.459+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:118 12.10 12:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8800 con 0x55faad2aa800 2024-04-24T08:17:09.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.460+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 9 ==== osd_op_reply(118 10000000000.00000000 [getxattr (59) out=59b] v0'0 uv17 ondisk = 0) ==== 164+0+59 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.460+0000 7f0d6511b740 20 datascan.get_frag_of: got backtrace parent 0x1/dir 2024-04-24T08:17:09.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.461+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:119 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e9c00 con 0x55faade6dc00 2024-04-24T08:17:09.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.462+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 18 ==== osd_op_reply(119 1.00000000 [omap-get-vals-by-keys out=520b] v0'0 uv9 ondisk = 0) ==== 154+0+520 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.462+0000 7f0d6511b740 20 datascan.read_dentry: oid=1.00000000 dname=dir frag=*, r=0 2024-04-24T08:17:09.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.462+0000 7f0d6511b740 20 datascan.get_frag_of: fast path, fragtree is fragtree_t(*) 2024-04-24T08:17:09.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.462+0000 7f0d6511b740 20 datascan.get_frag_of: frag is * 2024-04-24T08:17:09.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.462+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:120 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8000 con 0x55faad2aa800 2024-04-24T08:17:09.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.463+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 10 ==== osd_op_reply(120 10000000000.00000000 [omap-get-header out=274b] v0'0 uv17 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.463+0000 7f0d6511b740 10 datascan.find_or_create_dirfrag: read_version = 17 2024-04-24T08:17:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.463+0000 7f0d6511b740 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000000 * 2024-04-24T08:17:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.463+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:121 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c8800 con 0x55faad2aa800 2024-04-24T08:17:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.464+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 11 ==== osd_op_reply(121 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv17 ondisk = 0) ==== 164+0+4 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.464+0000 7f0d6511b740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-04-24T08:17:09.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.464+0000 7f0d6511b740 20 datascan.read_dentry: file_head not found in result 2024-04-24T08:17:09.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.464+0000 7f0d6511b740 10 datascan.inject_with_backtrace: Linking inode 0x0x10000000001 at 0x0x10000000000/file with size=0 bytes 2024-04-24T08:17:09.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.464+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:122 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9400 con 0x55faad2aa800 2024-04-24T08:17:09.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.468+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.0 v2:172.21.15.69:6800/146737935 12 ==== osd_op_reply(122 10000000000.00000000 [omap-set-vals] v86'18 uv18 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55faadcba000 con 0x55faad2aa800 2024-04-24T08:17:09.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.468+0000 7f0d6511b740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-04-24T08:17:09.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.468+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:123 13.27 13:e4000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9000 con 0x55faad2abc00 2024-04-24T08:17:09.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.469+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.1 v2:172.21.15.69:6815/153736297 17 ==== osd_op_reply(123 [pgnls-filter start_epoch 86 out=49b] v82'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faad2abc00 2024-04-24T08:17:09.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.469+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:124 13.17 13:e8000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9800 con 0x55faade6d400 2024-04-24T08:17:09.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.471+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 12 ==== osd_op_reply(124 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.471+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:125 13.37 13:ec000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6c800 con 0x55faade6d400 2024-04-24T08:17:09.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.472+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 <== osd.5 v2:172.21.15.121:6822/2187246263 13 ==== osd_op_reply(125 [pgnls-filter start_epoch 86 out=49b] v82'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba000 con 0x55faade6d400 2024-04-24T08:17:09.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.472+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:126 13.f 13:f0000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2c9c00 con 0x55faade6dc00 2024-04-24T08:17:09.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.474+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 19 ==== osd_op_reply(126 [pgnls-filter start_epoch 86 out=49b] v82'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.474+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:127 13.2f 13:f4000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6cc00 con 0x55faad2c8000 2024-04-24T08:17:09.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.474+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.4 v2:172.21.15.121:6800/2274643300 21 ==== osd_op_reply(127 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faad2c8000 2024-04-24T08:17:09.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.475+0000 7f0d5c400640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:128 13.1f 13:f8000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faad2e8400 con 0x55faade6dc00 2024-04-24T08:17:09.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.476+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 20 ==== osd_op_reply(128 [pgnls-filter start_epoch 86 out=49b] v82'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.476+0000 7f0d5ce00640 1 -- 172.21.15.69:0/3675217069 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:129 13.3f 13:fc000000::::head [pgnls-filter start_epoch 86 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55faade6d800 con 0x55faade6dc00 2024-04-24T08:17:09.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d63200640 1 -- 172.21.15.69:0/3675217069 <== osd.6 v2:172.21.15.121:6807/3068198325 21 ==== osd_op_reply(129 [pgnls-filter start_epoch 86 out=49b] v82'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55faadcba500 con 0x55faade6dc00 2024-04-24T08:17:09.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55faad2aa800 msgr2=0x55faad2c5080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55faad2aa800 0x55faad2c5080 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55faad2abc00 msgr2=0x55faad2c4000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55faad2abc00 0x55faad2c4000 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55faad2c8400 msgr2=0x55faad2c4b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55faad2c8400 0x55faad2c4b00 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55faad2d8000 msgr2=0x55faad2c5600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55faad2d8000 0x55faad2c5600 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55faad2c8000 msgr2=0x55faad2c4580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55faad2c8000 0x55faad2c4580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55faade6d400 msgr2=0x55faadd02680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55faade6d400 0x55faadd02680 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55faade6dc00 msgr2=0x55faadd02c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55faade6dc00 0x55faadd02c00 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55faad2aa400 msgr2=0x55faadd03180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55faad2aa400 0x55faadd03180 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 reap_dead start 2024-04-24T08:17:09.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.477+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 reap_dead start 2024-04-24T08:17:09.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.478+0000 7f0d61e00640 1 -- 172.21.15.69:0/3675217069 reap_dead start 2024-04-24T08:17:09.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.478+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55faade6d000 msgr2=0x55faadd02100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.478+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55faade6d000 0x55faadd02100 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x55faadd451d0 tx=0x55faadccc9c0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.478+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05800 msgr2=0x55faadd01600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.478+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05800 0x55faadd01600 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x55faade13b60 tx=0x55faadccd980 comp rx=0 tx=0).stop 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.478+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 shutdown_connections 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.478+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55faade6d000 0x55faadd02100 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.478+0000 7f0d6511b740 1 --2- 172.21.15.69:0/3675217069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faade05800 0x55faadd01600 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.478+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 >> 172.21.15.69:0/3675217069 conn(0x55faade05000 msgr2=0x55faacff8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.478+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 shutdown_connections 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.479+0000 7f0d6511b740 1 -- 172.21.15.69:0/3675217069 wait complete. 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.479+0000 7f0d6511b740 1 -- 172.21.15.69:0/509357628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faacf2cc00 msgr2=0x55faadd00000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.479+0000 7f0d6511b740 1 --2- 172.21.15.69:0/509357628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faacf2cc00 0x55faadd00000 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x55faadd458f0 tx=0x55faadcd00c0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.480+0000 7f0d6511b740 1 -- 172.21.15.69:0/509357628 shutdown_connections 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.480+0000 7f0d6511b740 1 --2- 172.21.15.69:0/509357628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55faacf2dc00 0x55faadd00b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.480+0000 7f0d6511b740 1 --2- 172.21.15.69:0/509357628 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55faacf2d800 0x55faadd00580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.480+0000 7f0d6511b740 1 --2- 172.21.15.69:0/509357628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55faacf2cc00 0x55faadd00000 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.480+0000 7f0d6511b740 1 -- 172.21.15.69:0/509357628 >> 172.21.15.69:0/509357628 conn(0x55faacf2d000 msgr2=0x55faacff9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.480+0000 7f0d6511b740 1 -- 172.21.15.69:0/509357628 shutdown_connections 2024-04-24T08:17:09.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.480+0000 7f0d6511b740 1 -- 172.21.15.69:0/509357628 wait complete. 2024-04-24T08:17:09.496 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', 'cephfs_data'] in time 0:00:00.325526, result: 2024-04-24T08:17:09.496 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-04-24T08:17:09.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.589+0000 7f9a6b00e740 1 -- start start 2024-04-24T08:17:09.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.590+0000 7f9a6b00e740 1 Processor -- start 2024-04-24T08:17:09.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.590+0000 7f9a6b00e740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda68ccc00 0x55fda7638000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.590+0000 7f9a6b00e740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda68cd800 0x55fda7638580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.590+0000 7f9a6b00e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda68cdc00 0x55fda7638b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.590+0000 7f9a6b00e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fda763c180 con 0x55fda68cdc00 2024-04-24T08:17:09.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.590+0000 7f9a6b00e740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fda763c300 con 0x55fda68cd800 2024-04-24T08:17:09.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.590+0000 7f9a6b00e740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55fda763c480 con 0x55fda68ccc00 2024-04-24T08:17:09.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a69000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda68cdc00 0x55fda7638b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a69000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda68cdc00 0x55fda7638b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46594/0 (socket says 172.21.15.69:46594) 2024-04-24T08:17:09.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a69000640 1 -- 172.21.15.69:0/2979246952 learned_addr learned my addr 172.21.15.69:0/2979246952 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:09.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda68cd800 0x55fda7638580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a68600640 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda68ccc00 0x55fda7638000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a69000640 1 -- 172.21.15.69:0/2979246952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda68ccc00 msgr2=0x55fda7638000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a69000640 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda68ccc00 0x55fda7638000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a69000640 1 -- 172.21.15.69:0/2979246952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda68cd800 msgr2=0x55fda7638580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a69000640 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda68cd800 0x55fda7638580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a69000640 1 -- 172.21.15.69:0/2979246952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda68f9680 con 0x55fda68cdc00 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a68600640 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda68ccc00 0x55fda7638000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.591+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda68cd800 0x55fda7638580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.592+0000 7f9a69000640 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda68cdc00 0x55fda7638b00 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x55fda76e9cb0 tx=0x55fda7670120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520d22cd709c92aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.592+0000 7f9a65400640 1 -- 172.21.15.69:0/2979246952 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fda68f2680 con 0x55fda68cdc00 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.592+0000 7f9a65400640 1 -- 172.21.15.69:0/2979246952 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55fda69161c0 con 0x55fda68cdc00 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.592+0000 7f9a65400640 1 -- 172.21.15.69:0/2979246952 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fda68f2b60 con 0x55fda68cdc00 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.592+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2979246952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda68f9860 con 0x55fda68cdc00 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.592+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2979246952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55fda68f92c0 con 0x55fda68cdc00 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.592+0000 7f9a6b00e740 4 waiting for MDS map... 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.592+0000 7f9a65400640 1 -- 172.21.15.69:0/2979246952 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x55fda779e600 con 0x55fda68cdc00 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.593+0000 7f9a65400640 1 -- 172.21.15.69:0/2979246952 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 69) ==== 2885+0+0 (secure 0 0 0) 0x55fda6925180 con 0x55fda68cdc00 2024-04-24T08:17:09.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.593+0000 7f9a6b00e740 4 Got MDS map 69 2024-04-24T08:17:09.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.593+0000 7f9a6b00e740 4 datascan.main: Using metadata pool output 2024-04-24T08:17:09.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.593+0000 7f9a6b00e740 4 datascan.main: connecting to RADOS... 2024-04-24T08:17:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.593+0000 7f9a6b00e740 1 Processor -- start 2024-04-24T08:17:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.594+0000 7f9a6b00e740 1 -- start start 2024-04-24T08:17:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.594+0000 7f9a6b00e740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda77a1400 0x55fda7639080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.594+0000 7f9a6b00e740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda77a1800 0x55fda7639600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.594+0000 7f9a6b00e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1c00 0x55fda7639b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.594+0000 7f9a6b00e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fda763c600 con 0x55fda77a1c00 2024-04-24T08:17:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.594+0000 7f9a6b00e740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fda763c780 con 0x55fda77a1400 2024-04-24T08:17:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.594+0000 7f9a6b00e740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55fda763c900 con 0x55fda77a1800 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.595+0000 7f9a67c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1c00 0x55fda7639b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.595+0000 7f9a67c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1c00 0x55fda7639b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46598/0 (socket says 172.21.15.69:46598) 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.595+0000 7f9a67c00640 1 -- 172.21.15.69:0/1140096412 learned_addr learned my addr 172.21.15.69:0/1140096412 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.595+0000 7f9a67c00640 1 --2- 172.21.15.69:0/1140096412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda77a1400 0x55fda7639080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.595+0000 7f9a68600640 1 --2- 172.21.15.69:0/1140096412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda77a1800 0x55fda7639600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.595+0000 7f9a67c00640 1 -- 172.21.15.69:0/1140096412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda77a1800 msgr2=0x55fda7639600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.595+0000 7f9a67c00640 1 --2- 172.21.15.69:0/1140096412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda77a1800 0x55fda7639600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.595+0000 7f9a67c00640 1 -- 172.21.15.69:0/1140096412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda77a1400 msgr2=0x55fda7639080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.595+0000 7f9a67c00640 1 --2- 172.21.15.69:0/1140096412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda77a1400 0x55fda7639080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.595+0000 7f9a67c00640 1 -- 172.21.15.69:0/1140096412 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda68f94a0 con 0x55fda77a1c00 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a67c00640 1 --2- 172.21.15.69:0/1140096412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1c00 0x55fda7639b80 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x55fda77ba960 tx=0x55fda77bcd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d492f46ca70633 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a62c00640 1 -- 172.21.15.69:0/1140096412 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fda68f2680 con 0x55fda77a1c00 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a62c00640 1 -- 172.21.15.69:0/1140096412 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55fda6916e00 con 0x55fda77a1c00 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a62c00640 1 -- 172.21.15.69:0/1140096412 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fda68f2b60 con 0x55fda77a1c00 2024-04-24T08:17:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a6b00e740 1 -- 172.21.15.69:0/1140096412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1c00 msgr2=0x55fda7639b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/1140096412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1c00 0x55fda7639b80 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x55fda77ba960 tx=0x55fda77bcd20 comp rx=0 tx=0).stop 2024-04-24T08:17:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a6b00e740 1 -- 172.21.15.69:0/1140096412 shutdown_connections 2024-04-24T08:17:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/1140096412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1c00 0x55fda7639b80 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/1140096412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda77a1800 0x55fda7639600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/1140096412 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda77a1400 0x55fda7639080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.596+0000 7f9a6b00e740 1 -- 172.21.15.69:0/1140096412 >> 172.21.15.69:0/1140096412 conn(0x55fda77a1000 msgr2=0x55fda699b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.597+0000 7f9a6b00e740 1 -- 172.21.15.69:0/1140096412 shutdown_connections 2024-04-24T08:17:09.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.597+0000 7f9a6b00e740 1 -- 172.21.15.69:0/1140096412 wait complete. 2024-04-24T08:17:09.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.597+0000 7f9a6b00e740 1 Processor -- start 2024-04-24T08:17:09.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.597+0000 7f9a6b00e740 1 -- start start 2024-04-24T08:17:09.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.598+0000 7f9a6b00e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1400 0x55fda7639080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.598+0000 7f9a6b00e740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda77a1800 0x55fda7639600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.598+0000 7f9a6b00e740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda77a1c00 0x55fda7639b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.598+0000 7f9a6b00e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fda763dc80 con 0x55fda77a1400 2024-04-24T08:17:09.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.598+0000 7f9a6b00e740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55fda763db00 con 0x55fda77a1800 2024-04-24T08:17:09.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.598+0000 7f9a6b00e740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55fda763d980 con 0x55fda77a1c00 2024-04-24T08:17:09.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.598+0000 7f9a67c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1400 0x55fda7639080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a67c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1400 0x55fda7639080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46606/0 (socket says 172.21.15.69:46606) 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 learned_addr learned my addr 172.21.15.69:0/2505271884 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a68600640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda77a1800 0x55fda7639600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda77a1c00 msgr2=0x55fda7639b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda77a1c00 0x55fda7639b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda77a1800 msgr2=0x55fda7639600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda77a1800 0x55fda7639600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fda68f9680 con 0x55fda77a1400 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1400 0x55fda7639080 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x55fda77bbc20 tx=0x55fda77bcc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a60e00640 1 -- 172.21.15.69:0/2505271884 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fda68f24e0 con 0x55fda77a1400 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a60e00640 1 -- 172.21.15.69:0/2505271884 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55fda6916c40 con 0x55fda77a1400 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a60e00640 1 -- 172.21.15.69:0/2505271884 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fda68f2680 con 0x55fda77a1400 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.599+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fda68f9860 con 0x55fda77a1400 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.600+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fda68f90e0 con 0x55fda77a1400 2024-04-24T08:17:09.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.600+0000 7f9a6b00e740 4 datascan.init: resolving metadata pool 12 2024-04-24T08:17:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.601+0000 7f9a60e00640 1 -- 172.21.15.69:0/2505271884 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55fda7820c00 con 0x55fda77a1400 2024-04-24T08:17:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.602+0000 7f9a60e00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55fda781cc00 0x55fda763a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.602+0000 7f9a60e00640 1 -- 172.21.15.69:0/2505271884 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x55fda781ea00 con 0x55fda77a1400 2024-04-24T08:17:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.602+0000 7f9a6b00e740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-24T08:17:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.602+0000 7f9a6b00e740 4 datascan.main: resolving metadata pool 12 2024-04-24T08:17:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.602+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55fda781d000 0x55fda763a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.602+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:1 12.0 12:00000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d400 con 0x55fda781d000 2024-04-24T08:17:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.603+0000 7f9a68600640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55fda781cc00 0x55fda763a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.603+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55fda781d000 0x55fda763a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.604+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55fda781d000 0x55fda763a680 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.604+0000 7f9a68600640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55fda781cc00 0x55fda763a100 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x55fda766e690 tx=0x55fda766c9c0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.605+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(1 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda781d000 2024-04-24T08:17:09.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.605+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55fda781d400 0x55fda6c5c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.605+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:2 12.10 12:08000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be4000 con 0x55fda781d400 2024-04-24T08:17:09.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.605+0000 7f9a69000640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55fda781d400 0x55fda6c5c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.606+0000 7f9a69000640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55fda781d400 0x55fda6c5c000 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.606+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== osd_op_reply(2 [pgnls start_epoch 86 out=177b] v86'18 uv18 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.606+0000 7f9a69000640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55fda6be4000 0x55fda6c5c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.607+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:3 12.8 12:10000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda77a0800 con 0x55fda6be4000 2024-04-24T08:17:09.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.607+0000 7f9a68600640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55fda6be4000 0x55fda6c5c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.607+0000 7f9a68600640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55fda6be4000 0x55fda6c5c580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.608+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(3 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4000 2024-04-24T08:17:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.608+0000 7f9a68600640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55fda77a0800 0x55fda6c5cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.608+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:4 12.18 12:18000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b000 con 0x55fda77a0800 2024-04-24T08:17:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.608+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55fda77a0800 0x55fda6c5cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.609+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55fda77a0800 0x55fda6c5cb00 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.609+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(4 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.609+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55fda766b000 0x55fda6c5d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.609+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:5 12.4 12:20000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be4400 con 0x55fda766b000 2024-04-24T08:17:09.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.610+0000 7f9a69000640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55fda766b000 0x55fda6c5d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.611+0000 7f9a69000640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55fda766b000 0x55fda6c5d080 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.611+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(5 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.611+0000 7f9a69000640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55fda6be4400 0x55fda6c5db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.612+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:6 12.14 12:28000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be4c00 con 0x55fda6be4400 2024-04-24T08:17:09.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.612+0000 7f9a68600640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55fda6be4400 0x55fda6c5db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.613+0000 7f9a68600640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55fda6be4400 0x55fda6c5db80 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.614+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(6 [pgnls start_epoch 86 out=149b] v85'28 uv28 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4400 2024-04-24T08:17:09.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.614+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:7 12.c 12:30000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be4c00 con 0x55fda781d400 2024-04-24T08:17:09.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.615+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 2 ==== osd_op_reply(7 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.615+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:8 12.1c 12:38000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be4c00 con 0x55fda781d400 2024-04-24T08:17:09.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.615+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 3 ==== osd_op_reply(8 [pgnls start_epoch 86 out=73b] v85'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.615+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:9 12.2 12:40000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be4c00 con 0x55fda781d000 2024-04-24T08:17:09.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.616+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== osd_op_reply(9 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda781d000 2024-04-24T08:17:09.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.616+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:10 12.12 12:48000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be4c00 con 0x55fda6be4000 2024-04-24T08:17:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.617+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.3 v2:172.21.15.69:6811/1875112303 2 ==== osd_op_reply(10 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4000 2024-04-24T08:17:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.617+0000 7f9a68600640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55fda6be4c00 0x55fda6c5e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.617+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:11 12.a 12:50000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda6be4c00 2024-04-24T08:17:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.617+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55fda6be4c00 0x55fda6c5e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.618+0000 7f9a67c00640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55fda6be4c00 0x55fda6c5e680 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.618+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== osd_op_reply(11 [pgnls start_epoch 86 out=81b] v85'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55fda765a000 con 0x55fda6be4c00 2024-04-24T08:17:09.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.618+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:12 12.1a 12:58000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda781d000 2024-04-24T08:17:09.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.619+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.4 v2:172.21.15.121:6800/2274643300 3 ==== osd_op_reply(12 [pgnls start_epoch 86 out=49b] v83'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda781d000 2024-04-24T08:17:09.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.619+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:13 12.6 12:60000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda77a0800 2024-04-24T08:17:09.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.619+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 2 ==== osd_op_reply(13 [pgnls start_epoch 86 out=77b] v85'9 uv9 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.619+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:14 12.16 12:68000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda766b000 2024-04-24T08:17:09.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.620+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 2 ==== osd_op_reply(14 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.620+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:15 12.e 12:70000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda781d400 2024-04-24T08:17:09.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.621+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 4 ==== osd_op_reply(15 [pgnls start_epoch 86 out=73b] v85'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.621+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:16 12.1e 12:78000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda781d000 2024-04-24T08:17:09.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.622+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.4 v2:172.21.15.121:6800/2274643300 4 ==== osd_op_reply(16 [pgnls start_epoch 86 out=49b] v83'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda781d000 2024-04-24T08:17:09.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.622+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:17 12.1 12:80000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda766b000 2024-04-24T08:17:09.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.623+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 3 ==== osd_op_reply(17 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.623+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:18 12.11 12:88000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda6be4400 2024-04-24T08:17:09.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.624+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.5 v2:172.21.15.121:6822/2187246263 2 ==== osd_op_reply(18 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4400 2024-04-24T08:17:09.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.624+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:19 12.9 12:90000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda6be4c00 2024-04-24T08:17:09.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.625+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.2 v2:172.21.15.69:6808/2630808717 2 ==== osd_op_reply(19 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda6be4c00 2024-04-24T08:17:09.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.625+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:20 12.19 12:98000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda781d400 2024-04-24T08:17:09.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.625+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 5 ==== osd_op_reply(20 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.625+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:21 12.5 12:a0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda766b000 2024-04-24T08:17:09.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.626+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 4 ==== osd_op_reply(21 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.626+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:22 12.15 12:a8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda766b400 con 0x55fda766b000 2024-04-24T08:17:09.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.627+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 5 ==== osd_op_reply(22 [pgnls start_epoch 86 out=97b] v85'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.627+0000 7f9a69000640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55fda766b400 0x55fda6c5e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.627+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:23 12.d 12:b0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda766b400 2024-04-24T08:17:09.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.628+0000 7f9a69000640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55fda766b400 0x55fda6c5e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.629+0000 7f9a69000640 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55fda766b400 0x55fda6c5e100 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.630+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== osd_op_reply(23 [pgnls start_epoch 86 out=74b] v81'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b400 2024-04-24T08:17:09.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.630+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:24 12.1d 12:b8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda781d400 2024-04-24T08:17:09.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.631+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 6 ==== osd_op_reply(24 [pgnls start_epoch 86 out=73b] v85'5 uv5 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.631+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:25 12.3 12:c0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda781d400 2024-04-24T08:17:09.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.632+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 7 ==== osd_op_reply(25 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.632+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:26 12.13 12:c8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda77a0800 2024-04-24T08:17:09.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.632+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 3 ==== osd_op_reply(26 [pgnls start_epoch 86 out=98b] v85'4 uv4 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.632+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:27 12.b 12:d0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda6be4000 2024-04-24T08:17:09.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.633+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.3 v2:172.21.15.69:6811/1875112303 3 ==== osd_op_reply(27 [pgnls start_epoch 86 out=76b] v85'6 uv6 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4000 2024-04-24T08:17:09.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.633+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:28 12.1b 12:d8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda781d000 2024-04-24T08:17:09.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.634+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.4 v2:172.21.15.121:6800/2274643300 5 ==== osd_op_reply(28 [pgnls start_epoch 86 out=73b] v85'10 uv10 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a000 con 0x55fda781d000 2024-04-24T08:17:09.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.634+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:29 12.7 12:e0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda77a0800 2024-04-24T08:17:09.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.635+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 4 ==== osd_op_reply(29 [pgnls start_epoch 86 out=73b] v85'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.635+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:30 12.17 12:e8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda766b000 2024-04-24T08:17:09.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.636+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 6 ==== osd_op_reply(30 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.636+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:31 12.f 12:f0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda766b000 2024-04-24T08:17:09.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.637+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 7 ==== osd_op_reply(31 [pgnls start_epoch 86 out=73b] v81'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.637+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:32 12.1f 12:f8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda766b000 2024-04-24T08:17:09.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.638+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 8 ==== osd_op_reply(32 [pgnls start_epoch 86 out=101b] v85'9 uv9 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.638+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 601.00000000 2024-04-24T08:17:09.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.638+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:33 12.10 12:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d400 2024-04-24T08:17:09.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.638+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 8 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.638+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 602.00000000 2024-04-24T08:17:09.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.638+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:34 12.10 12:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda781dc00 con 0x55fda781d400 2024-04-24T08:17:09.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.639+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 9 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.639+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-04-24T08:17:09.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.639+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:35 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5c00 con 0x55fda781d400 2024-04-24T08:17:09.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.639+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 10 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) ==== 164+0+506 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.639+0000 7f9a6b00e740 20 datascan.scan_links: injected ino detected 2024-04-24T08:17:09.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.639+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 600.00000000 2024-04-24T08:17:09.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.639+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:36 12.10 12:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5800 con 0x55fda781d400 2024-04-24T08:17:09.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 11 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 603.00000000 2024-04-24T08:17:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:37 12.10 12:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5400 con 0x55fda781d400 2024-04-24T08:17:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 12 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-04-24T08:17:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-24T08:17:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 200.00000000 2024-04-24T08:17:09.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-04-24T08:17:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 200.00000001 2024-04-24T08:17:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-04-24T08:17:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 606.00000000 2024-04-24T08:17:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.640+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:38 12.14 12:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5000 con 0x55fda6be4400 2024-04-24T08:17:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.641+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.5 v2:172.21.15.121:6822/2187246263 3 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv26 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4400 2024-04-24T08:17:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.641+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 607.00000000 2024-04-24T08:17:09.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.641+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:39 12.1c 12:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5c00 con 0x55fda781d400 2024-04-24T08:17:09.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.642+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 13 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.642+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 100000001f6.00000000 2024-04-24T08:17:09.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.642+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:40 12.a 12:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda6be4c00 2024-04-24T08:17:09.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.643+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.2 v2:172.21.15.69:6808/2630808717 3 ==== osd_op_reply(40 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) ==== 164+0+54 (crc 0 0 0) 0x55fda765a000 con 0x55fda6be4c00 2024-04-24T08:17:09.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.643+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-04-24T08:17:09.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.643+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-04-24T08:17:09.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.643+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 608.00000000 2024-04-24T08:17:09.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.643+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:41 12.e 12:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda781dc00 con 0x55fda781d400 2024-04-24T08:17:09.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.643+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 14 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.643+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 604.00000000 2024-04-24T08:17:09.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.643+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:42 12.15 12:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7c000 con 0x55fda766b000 2024-04-24T08:17:09.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.644+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 9 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.644+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 500.00000000 2024-04-24T08:17:09.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.644+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-04-24T08:17:09.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.644+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-04-24T08:17:09.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.644+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-24T08:17:09.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.644+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 605.00000000 2024-04-24T08:17:09.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.644+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:43 12.1d 12:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7c400 con 0x55fda781d400 2024-04-24T08:17:09.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.645+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 15 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.645+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-04-24T08:17:09.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.645+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-24T08:17:09.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.645+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 100.00000000 2024-04-24T08:17:09.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.645+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:44 12.13 12:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7c800 con 0x55fda77a0800 2024-04-24T08:17:09.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.646+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 5 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.646+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-04-24T08:17:09.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.646+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-24T08:17:09.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.646+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 200.00000002 2024-04-24T08:17:09.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.646+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-04-24T08:17:09.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.646+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 609.00000000 2024-04-24T08:17:09.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.646+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:45 12.7 12:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7cc00 con 0x55fda77a0800 2024-04-24T08:17:09.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.647+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 6 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.647+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 400.00000000 2024-04-24T08:17:09.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.647+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-04-24T08:17:09.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.647+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-04-24T08:17:09.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.647+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-24T08:17:09.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.647+0000 7f9a6b00e740 10 datascan.scan_links: step 1: handling object 1.00000000 2024-04-24T08:17:09.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.647+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:46 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7d000 con 0x55fda766b000 2024-04-24T08:17:09.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.648+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 10 ==== osd_op_reply(46 1.00000000 [omap-get-vals out=1041b] v0'0 uv9 ondisk = 0) ==== 154+0+1041 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.648+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:47 12.0 12:00000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7d400 con 0x55fda781d000 2024-04-24T08:17:09.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.649+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.4 v2:172.21.15.121:6800/2274643300 6 ==== osd_op_reply(47 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda781d000 2024-04-24T08:17:09.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.649+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:48 12.10 12:08000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d400 2024-04-24T08:17:09.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.650+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 16 ==== osd_op_reply(48 [pgnls start_epoch 86 out=177b] v86'18 uv18 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.650+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:49 12.8 12:10000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda6be4000 2024-04-24T08:17:09.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.651+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.3 v2:172.21.15.69:6811/1875112303 4 ==== osd_op_reply(49 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4000 2024-04-24T08:17:09.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.651+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:50 12.18 12:18000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda77a0800 2024-04-24T08:17:09.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.651+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 7 ==== osd_op_reply(50 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.651+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:51 12.4 12:20000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda766b000 2024-04-24T08:17:09.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.652+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 11 ==== osd_op_reply(51 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.653+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:52 12.14 12:28000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda6be4400 2024-04-24T08:17:09.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.654+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.5 v2:172.21.15.121:6822/2187246263 4 ==== osd_op_reply(52 [pgnls start_epoch 86 out=149b] v85'28 uv28 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4400 2024-04-24T08:17:09.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.654+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:53 12.c 12:30000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d400 2024-04-24T08:17:09.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.654+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 17 ==== osd_op_reply(53 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.654+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:54 12.1c 12:38000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d400 2024-04-24T08:17:09.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.655+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 18 ==== osd_op_reply(54 [pgnls start_epoch 86 out=73b] v85'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.655+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:55 12.2 12:40000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d000 2024-04-24T08:17:09.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.656+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.4 v2:172.21.15.121:6800/2274643300 7 ==== osd_op_reply(55 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda781d000 2024-04-24T08:17:09.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.656+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:56 12.12 12:48000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda6be4000 2024-04-24T08:17:09.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.657+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.3 v2:172.21.15.69:6811/1875112303 5 ==== osd_op_reply(56 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4000 2024-04-24T08:17:09.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.657+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:57 12.a 12:50000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda6be4c00 2024-04-24T08:17:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.658+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.2 v2:172.21.15.69:6808/2630808717 4 ==== osd_op_reply(57 [pgnls start_epoch 86 out=81b] v85'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55fda765a000 con 0x55fda6be4c00 2024-04-24T08:17:09.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.658+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:58 12.1a 12:58000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d000 2024-04-24T08:17:09.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.659+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.4 v2:172.21.15.121:6800/2274643300 8 ==== osd_op_reply(58 [pgnls start_epoch 86 out=49b] v83'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda781d000 2024-04-24T08:17:09.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.659+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:59 12.6 12:60000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda77a0800 2024-04-24T08:17:09.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.659+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 8 ==== osd_op_reply(59 [pgnls start_epoch 86 out=77b] v85'9 uv9 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.659+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:60 12.16 12:68000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda766b000 2024-04-24T08:17:09.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.661+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 12 ==== osd_op_reply(60 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.661+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:61 12.e 12:70000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d400 2024-04-24T08:17:09.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.661+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 19 ==== osd_op_reply(61 [pgnls start_epoch 86 out=73b] v85'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.661+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:62 12.1e 12:78000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d000 2024-04-24T08:17:09.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.663+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.4 v2:172.21.15.121:6800/2274643300 9 ==== osd_op_reply(62 [pgnls start_epoch 86 out=49b] v83'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda781d000 2024-04-24T08:17:09.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.663+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:63 12.1 12:80000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda766b000 2024-04-24T08:17:09.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.664+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 13 ==== osd_op_reply(63 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.664+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:64 12.11 12:88000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda6be4400 2024-04-24T08:17:09.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.665+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.5 v2:172.21.15.121:6822/2187246263 5 ==== osd_op_reply(64 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4400 2024-04-24T08:17:09.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.665+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:65 12.9 12:90000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda6be4c00 2024-04-24T08:17:09.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.666+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.2 v2:172.21.15.69:6808/2630808717 5 ==== osd_op_reply(65 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a000 con 0x55fda6be4c00 2024-04-24T08:17:09.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.666+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:66 12.19 12:98000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d400 2024-04-24T08:17:09.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.667+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 20 ==== osd_op_reply(66 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.667+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:67 12.5 12:a0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda766b000 2024-04-24T08:17:09.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.668+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 14 ==== osd_op_reply(67 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.668+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:68 12.15 12:a8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda766b000 2024-04-24T08:17:09.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.669+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 15 ==== osd_op_reply(68 [pgnls start_epoch 86 out=97b] v85'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.669+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:69 12.d 12:b0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda766b400 2024-04-24T08:17:09.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.670+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.7 v2:172.21.15.121:6811/176144805 2 ==== osd_op_reply(69 [pgnls start_epoch 86 out=74b] v81'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b400 2024-04-24T08:17:09.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.670+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:70 12.1d 12:b8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d400 2024-04-24T08:17:09.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.671+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 21 ==== osd_op_reply(70 [pgnls start_epoch 86 out=73b] v85'5 uv5 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.671+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:71 12.3 12:c0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d400 2024-04-24T08:17:09.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.672+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 22 ==== osd_op_reply(71 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.672+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:72 12.13 12:c8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda77a0800 2024-04-24T08:17:09.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.673+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 9 ==== osd_op_reply(72 [pgnls start_epoch 86 out=98b] v85'4 uv4 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.673+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:73 12.b 12:d0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda6be4000 2024-04-24T08:17:09.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.674+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.3 v2:172.21.15.69:6811/1875112303 6 ==== osd_op_reply(73 [pgnls start_epoch 86 out=76b] v85'6 uv6 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4000 2024-04-24T08:17:09.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.674+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:74 12.1b 12:d8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda781d000 2024-04-24T08:17:09.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.675+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.4 v2:172.21.15.121:6800/2274643300 10 ==== osd_op_reply(74 [pgnls start_epoch 86 out=73b] v85'10 uv10 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a000 con 0x55fda781d000 2024-04-24T08:17:09.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.675+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:75 12.7 12:e0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda77a0800 2024-04-24T08:17:09.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.676+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 10 ==== osd_op_reply(75 [pgnls start_epoch 86 out=73b] v85'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.676+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:76 12.17 12:e8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda766b000 2024-04-24T08:17:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.677+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 16 ==== osd_op_reply(76 [pgnls start_epoch 86 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.677+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:77 12.f 12:f0000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda766b000 2024-04-24T08:17:09.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.678+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 17 ==== osd_op_reply(77 [pgnls start_epoch 86 out=73b] v81'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.678+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:78 12.1f 12:f8000000::::head [pgnls start_epoch 86 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda766b000 2024-04-24T08:17:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.679+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 18 ==== osd_op_reply(78 [pgnls start_epoch 86 out=101b] v85'9 uv9 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.679+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 601.00000000 2024-04-24T08:17:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.679+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:79 12.10 12:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7d800 con 0x55fda781d400 2024-04-24T08:17:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.680+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 23 ==== osd_op_reply(79 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.680+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 602.00000000 2024-04-24T08:17:09.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.680+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:80 12.10 12:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7dc00 con 0x55fda781d400 2024-04-24T08:17:09.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.681+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 24 ==== osd_op_reply(80 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.681+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-04-24T08:17:09.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.681+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:81 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7c800 con 0x55fda781d400 2024-04-24T08:17:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.682+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 25 ==== osd_op_reply(81 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) ==== 164+0+506 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.682+0000 7f9a6b00e740 20 datascan.scan_links: injected ino detected 2024-04-24T08:17:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.682+0000 7f9a6b00e740 -1 datascan.scan_links: Bad nlink on 0x10000000001 expected 2 has 1 2024-04-24T08:17:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.682+0000 7f9a6b00e740 20 datascan.scan_links: adding 0x10000000001 for future processing to fix dnfirst 2024-04-24T08:17:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.682+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 600.00000000 2024-04-24T08:17:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.682+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:82 12.10 12:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7cc00 con 0x55fda781d400 2024-04-24T08:17:09.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.683+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 26 ==== osd_op_reply(82 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.683+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 603.00000000 2024-04-24T08:17:09.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.683+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:83 12.10 12:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7d400 con 0x55fda781d400 2024-04-24T08:17:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.684+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 27 ==== osd_op_reply(83 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.684+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-04-24T08:17:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.684+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-24T08:17:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.684+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 200.00000000 2024-04-24T08:17:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.684+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-04-24T08:17:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.684+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 200.00000001 2024-04-24T08:17:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.684+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-04-24T08:17:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.684+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 606.00000000 2024-04-24T08:17:09.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.684+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:84 12.14 12:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7c000 con 0x55fda6be4400 2024-04-24T08:17:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.685+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 <== osd.5 v2:172.21.15.121:6822/2187246263 6 ==== osd_op_reply(84 606.00000000 [omap-get-vals out=5b] v0'0 uv26 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a780 con 0x55fda6be4400 2024-04-24T08:17:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.685+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 607.00000000 2024-04-24T08:17:09.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.685+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:85 12.1c 12:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7c400 con 0x55fda781d400 2024-04-24T08:17:09.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.686+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 28 ==== osd_op_reply(85 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.686+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 100000001f6.00000000 2024-04-24T08:17:09.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.686+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:86 12.a 12:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7d000 con 0x55fda6be4c00 2024-04-24T08:17:09.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.687+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.2 v2:172.21.15.69:6808/2630808717 6 ==== osd_op_reply(86 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) ==== 164+0+54 (crc 0 0 0) 0x55fda765a000 con 0x55fda6be4c00 2024-04-24T08:17:09.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.688+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-04-24T08:17:09.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.688+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-04-24T08:17:09.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.688+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 608.00000000 2024-04-24T08:17:09.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.688+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:87 12.e 12:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda781dc00 con 0x55fda781d400 2024-04-24T08:17:09.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.689+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 29 ==== osd_op_reply(87 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.689+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 604.00000000 2024-04-24T08:17:09.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.689+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:88 12.15 12:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda781d800 con 0x55fda766b000 2024-04-24T08:17:09.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.690+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 19 ==== osd_op_reply(88 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.690+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 500.00000000 2024-04-24T08:17:09.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.690+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-04-24T08:17:09.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.690+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-04-24T08:17:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.690+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-24T08:17:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.690+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 605.00000000 2024-04-24T08:17:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.690+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:89 12.1d 12:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda781dc00 con 0x55fda781d400 2024-04-24T08:17:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.691+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 30 ==== osd_op_reply(89 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.691+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-04-24T08:17:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.691+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-24T08:17:09.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.691+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 100.00000000 2024-04-24T08:17:09.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.691+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:90 12.13 12:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7d800 con 0x55fda77a0800 2024-04-24T08:17:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.692+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 11 ==== osd_op_reply(90 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.693+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-04-24T08:17:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.693+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-24T08:17:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.693+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 200.00000002 2024-04-24T08:17:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.693+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-04-24T08:17:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.693+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 609.00000000 2024-04-24T08:17:09.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.693+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:91 12.7 12:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7dc00 con 0x55fda77a0800 2024-04-24T08:17:09.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.694+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 12 ==== osd_op_reply(91 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.694+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 400.00000000 2024-04-24T08:17:09.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.694+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-04-24T08:17:09.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.694+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-04-24T08:17:09.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.694+0000 7f9a6b00e740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-24T08:17:09.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.694+0000 7f9a6b00e740 10 datascan.scan_links: step 2: handling object 1.00000000 2024-04-24T08:17:09.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.694+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:92 12.1f 12:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7c800 con 0x55fda766b000 2024-04-24T08:17:09.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.695+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.6 v2:172.21.15.121:6807/3068198325 20 ==== osd_op_reply(92 1.00000000 [omap-get-vals out=1041b] v0'0 uv9 ondisk = 0) ==== 154+0+1041 (crc 0 0 0) 0x55fda765a500 con 0x55fda766b000 2024-04-24T08:17:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.695+0000 7f9a6b00e740 10 datascan.scan_links: processing 0 dup_primaries, 1 remote_links 2024-04-24T08:17:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.695+0000 7f9a6b00e740 10 datascan.scan_links: removing dup dentries from 0 objects 2024-04-24T08:17:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.695+0000 7f9a6b00e740 10 datascan.scan_links: processing 1 bad_nlink_inos 2024-04-24T08:17:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.695+0000 7f9a6b00e740 10 datascan.scan_links: handling bad_nlink_ino 0x10000000001 2024-04-24T08:17:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.695+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:93 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7cc00 con 0x55fda781d400 2024-04-24T08:17:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.696+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 31 ==== osd_op_reply(93 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv18 ondisk = 0) ==== 164+0+505 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.696+0000 7f9a6b00e740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-04-24T08:17:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.696+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:94 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7d400 con 0x55fda781d400 2024-04-24T08:17:09.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.700+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 32 ==== osd_op_reply(94 10000000000.00000000 [omap-set-vals] v86'19 uv19 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.700+0000 7f9a6b00e740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-04-24T08:17:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.700+0000 7f9a6b00e740 10 datascan.scan_links: processing 1 injected_inos 2024-04-24T08:17:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.700+0000 7f9a6b00e740 10 datascan.scan_links: handling injected_ino 0x10000000001 2024-04-24T08:17:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.700+0000 7f9a6b00e740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000000:file 2024-04-24T08:17:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.700+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:95 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7c400 con 0x55fda781d400 2024-04-24T08:17:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.701+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 33 ==== osd_op_reply(95 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv19 ondisk = 0) ==== 164+0+505 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.701+0000 7f9a6b00e740 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-04-24T08:17:09.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.701+0000 7f9a6b00e740 20 datascan.scan_links: first is now 2 2024-04-24T08:17:09.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.701+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:96 12.10 12:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7d000 con 0x55fda781d400 2024-04-24T08:17:09.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.705+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.0 v2:172.21.15.69:6800/146737935 34 ==== osd_op_reply(96 10000000000.00000000 [omap-set-vals] v86'20 uv20 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55fda765a500 con 0x55fda781d400 2024-04-24T08:17:09.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.705+0000 7f9a6b00e740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-04-24T08:17:09.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.705+0000 7f9a6b00e740 10 datascan.scan_links: updating inotable 2024-04-24T08:17:09.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.705+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:97 12.13 12:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7d800 con 0x55fda77a0800 2024-04-24T08:17:09.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.707+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 13 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv3 ondisk = 0) ==== 157+0+50 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.707+0000 7f9a6b00e740 10 mds.0.inotable: skip_inos was [0x10000000002~0x1f4,0x100000001f7~0xfffffffe09] 2024-04-24T08:17:09.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.707+0000 7f9a6b00e740 10 mds.0.inotable: skip_inos now [0x100000001f7~0xfffffffe09] 2024-04-24T08:17:09.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.707+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:98 12.13 12:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e86) -- 0x55fda6c7dc00 con 0x55fda77a0800 2024-04-24T08:17:09.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.711+0000 7f9a67c00640 1 -- 172.21.15.69:0/2505271884 <== osd.1 v2:172.21.15.69:6815/153736297 14 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v86'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x55fda765a000 con 0x55fda77a0800 2024-04-24T08:17:09.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.711+0000 7f9a6b00e740 10 datascan.scan_links: updating snaptable 2024-04-24T08:17:09.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.711+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:99 12.d 12:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86) -- 0x55fda6be5c00 con 0x55fda766b400 2024-04-24T08:17:09.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a69000640 1 -- 172.21.15.69:0/2505271884 <== osd.7 v2:172.21.15.121:6811/176144805 3 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x55fda765a280 con 0x55fda766b400 2024-04-24T08:17:09.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55fda781d400 msgr2=0x55fda6c5c000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55fda781d400 0x55fda6c5c000 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55fda77a0800 msgr2=0x55fda6c5cb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55fda77a0800 0x55fda6c5cb00 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55fda6be4c00 msgr2=0x55fda6c5e680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55fda6be4c00 0x55fda6c5e680 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55fda6be4000 msgr2=0x55fda6c5c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55fda6be4000 0x55fda6c5c580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55fda781d000 msgr2=0x55fda763a680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55fda781d000 0x55fda763a680 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55fda6be4400 msgr2=0x55fda6c5db80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 reap_dead start 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55fda6be4400 0x55fda6c5db80 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 reap_dead start 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55fda766b000 msgr2=0x55fda6c5d080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55fda766b000 0x55fda6c5d080 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55fda766b400 msgr2=0x55fda6c5e100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.713+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55fda766b400 0x55fda6c5e100 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a68600640 1 -- 172.21.15.69:0/2505271884 reap_dead start 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55fda781cc00 msgr2=0x55fda763a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55fda781cc00 0x55fda763a100 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x55fda766e690 tx=0x55fda766c9c0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1400 msgr2=0x55fda7639080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1400 0x55fda7639080 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x55fda77bbc20 tx=0x55fda77bcc00 comp rx=0 tx=0).stop 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 shutdown_connections 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55fda781cc00 0x55fda763a100 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2505271884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda77a1400 0x55fda7639080 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 >> 172.21.15.69:0/2505271884 conn(0x55fda77a1000 msgr2=0x55fda699b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 shutdown_connections 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.714+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2505271884 wait complete. 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.715+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2979246952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda68cdc00 msgr2=0x55fda7638b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.715+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda68cdc00 0x55fda7638b00 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x55fda76e9cb0 tx=0x55fda7670120 comp rx=0 tx=0).stop 2024-04-24T08:17:09.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.716+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2979246952 shutdown_connections 2024-04-24T08:17:09.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.716+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fda68cdc00 0x55fda7638b00 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.716+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fda68cd800 0x55fda7638580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.716+0000 7f9a6b00e740 1 --2- 172.21.15.69:0/2979246952 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55fda68ccc00 0x55fda7638000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.716+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2979246952 >> 172.21.15.69:0/2979246952 conn(0x55fda68cd000 msgr2=0x55fda699a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.716+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2979246952 shutdown_connections 2024-04-24T08:17:09.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.716+0000 7f9a6b00e740 1 -- 172.21.15.69:0/2979246952 wait complete. 2024-04-24T08:17:09.725 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.229196, result: 2024-04-24T08:17:09.726 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-24T08:17:09.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.936+0000 7ff9a6a00640 1 Processor -- start 2024-04-24T08:17:09.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.936+0000 7ff9a6a00640 1 -- start start 2024-04-24T08:17:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.938+0000 7ff9a6a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff9a0071d70 0x7ff9a015ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.938+0000 7ff9a6a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff9a0160510 0x7ff9a00cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.938+0000 7ff9a6a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a0160920 con 0x7ff9a01420a0 2024-04-24T08:17:09.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.938+0000 7ff9a6a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a00cf1e0 con 0x7ff9a0160510 2024-04-24T08:17:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.938+0000 7ff9a6a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a0072200 con 0x7ff9a0071d70 2024-04-24T08:17:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.938+0000 7ff9a5600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff9a01420a0 0x7ff9a0142480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39060/0 (socket says 172.21.15.69:39060) 2024-04-24T08:17:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.938+0000 7ff9a5600640 1 -- 172.21.15.69:0/3579925110 learned_addr learned my addr 172.21.15.69:0/3579925110 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.938+0000 7ff9a6000640 1 --2- 172.21.15.69:0/3579925110 >> v2:172.21.15.69:3300/0 conn(0x7ff9a0160510 0x7ff9a00cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.938+0000 7ff9a4c00640 1 --2- 172.21.15.69:0/3579925110 >> v2:172.21.15.121:3300/0 conn(0x7ff9a0071d70 0x7ff9a015ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.938+0000 7ff99fe00640 1 -- 172.21.15.69:0/3579925110 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1451228790 0 0) 0x7ff994001110 con 0x7ff9a01420a0 2024-04-24T08:17:09.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.939+0000 7ff99fe00640 1 -- 172.21.15.69:0/3579925110 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff988003520 con 0x7ff9a01420a0 2024-04-24T08:17:09.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.939+0000 7ff9a6000640 1 -- 172.21.15.69:0/3579925110 >> v2:172.21.15.121:3300/0 conn(0x7ff9a0071d70 msgr2=0x7ff9a015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.939+0000 7ff9a6000640 1 --2- 172.21.15.69:0/3579925110 >> v2:172.21.15.121:3300/0 conn(0x7ff9a0071d70 0x7ff9a015ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.939+0000 7ff9a6000640 1 -- 172.21.15.69:0/3579925110 >> v1:172.21.15.69:6789/0 conn(0x7ff9a01420a0 legacy=0x7ff9a0142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.939+0000 7ff9a6000640 1 -- 172.21.15.69:0/3579925110 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a0072450 con 0x7ff9a0160510 2024-04-24T08:17:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.940+0000 7ff9a6000640 1 --2- 172.21.15.69:0/3579925110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0160510 0x7ff9a00cf670 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00d13d0 tx=0x7ff99802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecacb1d548ef2b79 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.940+0000 7ff99fe00640 1 -- 172.21.15.69:0/3579925110 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998007740 con 0x7ff9a0160510 2024-04-24T08:17:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.940+0000 7ff99fe00640 1 -- 172.21.15.69:0/3579925110 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9980078e0 con 0x7ff9a0160510 2024-04-24T08:17:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.940+0000 7ff99fe00640 1 -- 172.21.15.69:0/3579925110 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998007cb0 con 0x7ff9a0160510 2024-04-24T08:17:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.940+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3579925110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0160510 msgr2=0x7ff9a00cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.940+0000 7ff9a6a00640 1 --2- 172.21.15.69:0/3579925110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0160510 0x7ff9a00cf670 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00d13d0 tx=0x7ff99802a680 comp rx=0 tx=0).stop 2024-04-24T08:17:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.941+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3579925110 shutdown_connections 2024-04-24T08:17:09.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.941+0000 7ff9a6a00640 1 --2- 172.21.15.69:0/3579925110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0160510 0x7ff9a00cf670 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.941+0000 7ff9a6a00640 1 --2- 172.21.15.69:0/3579925110 >> v2:172.21.15.121:3300/0 conn(0x7ff9a0071d70 0x7ff9a015ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.941+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3579925110 >> 172.21.15.69:0/3579925110 conn(0x7ff9a0067b80 msgr2=0x7ff9a006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.941+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3579925110 shutdown_connections 2024-04-24T08:17:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.941+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3579925110 wait complete. 2024-04-24T08:17:09.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.942+0000 7ff9a6a00640 1 Processor -- start 2024-04-24T08:17:09.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.943+0000 7ff9a6a00640 1 -- start start 2024-04-24T08:17:09.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.943+0000 7ff9a6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0071d70 0x7ff9a0138e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.943+0000 7ff9a6a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9a01420a0 0x7ff9a0155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.943+0000 7ff9a6a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9a0160510 0x7ff9a01561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.943+0000 7ff9a6a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a0072cf0 con 0x7ff9a0071d70 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.943+0000 7ff9a6a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a0072a10 con 0x7ff9a01420a0 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.943+0000 7ff9a6a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a00cfbb0 con 0x7ff9a0160510 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.944+0000 7ff9a5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0071d70 0x7ff9a0138e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.944+0000 7ff9a5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0071d70 0x7ff9a0138e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46620/0 (socket says 172.21.15.69:46620) 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.944+0000 7ff9a5600640 1 -- 172.21.15.69:0/3574334331 learned_addr learned my addr 172.21.15.69:0/3574334331 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.944+0000 7ff9a4c00640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9a01420a0 0x7ff9a0155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.944+0000 7ff9a5600640 1 -- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9a0160510 msgr2=0x7ff9a01561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.944+0000 7ff9a5600640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9a0160510 0x7ff9a01561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.944+0000 7ff9a5600640 1 -- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9a01420a0 msgr2=0x7ff9a0155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.944+0000 7ff9a5600640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9a01420a0 0x7ff9a0155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.944+0000 7ff9a5600640 1 -- 172.21.15.69:0/3574334331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a0156820 con 0x7ff9a0071d70 2024-04-24T08:17:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.944+0000 7ff9a5600640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0071d70 0x7ff9a0138e00 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7ff99400c730 tx=0x7ff99400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.945+0000 7ff99d600640 1 -- 172.21.15.69:0/3574334331 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994017070 con 0x7ff9a0071d70 2024-04-24T08:17:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.945+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3574334331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a01f25f0 con 0x7ff9a0071d70 2024-04-24T08:17:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.945+0000 7ff99d600640 1 -- 172.21.15.69:0/3574334331 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff994005ad0 con 0x7ff9a0071d70 2024-04-24T08:17:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.945+0000 7ff99d600640 1 -- 172.21.15.69:0/3574334331 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff994007e20 con 0x7ff9a0071d70 2024-04-24T08:17:09.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.945+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3574334331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a01f2bb0 con 0x7ff9a0071d70 2024-04-24T08:17:09.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.946+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3574334331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a0075d50 con 0x7ff9a0071d70 2024-04-24T08:17:09.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.947+0000 7ff99d600640 1 -- 172.21.15.69:0/3574334331 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff99401a070 con 0x7ff9a0071d70 2024-04-24T08:17:09.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.948+0000 7ff99d600640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff98006c8f0 0x7ff98006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.948+0000 7ff99d600640 1 -- 172.21.15.69:0/3574334331 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7ff99408b010 con 0x7ff9a0071d70 2024-04-24T08:17:09.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.949+0000 7ff9a4c00640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff98006c8f0 0x7ff98006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.950+0000 7ff9a4c00640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff98006c8f0 0x7ff98006edb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff9a01556c0 tx=0x7ff98c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:09.952+0000 7ff99d600640 1 -- 172.21.15.69:0/3574334331 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9a0160920 con 0x7ff9a0071d70 2024-04-24T08:17:10.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.195+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3574334331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7ff9a0160060 con 0x7ff9a0071d70 2024-04-24T08:17:10.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.569+0000 7ff99d600640 1 -- 172.21.15.69:0/3574334331 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v71) ==== 162+0+0 (secure 0 0 0) 0x7ff994059360 con 0x7ff9a0071d70 2024-04-24T08:17:10.571 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-24T08:17:10.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.572+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff98006c8f0 msgr2=0x7ff98006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:10.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.572+0000 7ff9a6a00640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff98006c8f0 0x7ff98006edb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff9a01556c0 tx=0x7ff98c008040 comp rx=0 tx=0).stop 2024-04-24T08:17:10.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.572+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3574334331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0071d70 msgr2=0x7ff9a0138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:10.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.572+0000 7ff9a6a00640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0071d70 0x7ff9a0138e00 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7ff99400c730 tx=0x7ff99400cbf0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.573+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3574334331 shutdown_connections 2024-04-24T08:17:10.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.573+0000 7ff9a6a00640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff98006c8f0 0x7ff98006edb0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.573+0000 7ff9a6a00640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9a0160510 0x7ff9a01561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.573+0000 7ff9a6a00640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9a01420a0 0x7ff9a0155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.573+0000 7ff9a6a00640 1 --2- 172.21.15.69:0/3574334331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9a0071d70 0x7ff9a0138e00 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.573+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3574334331 >> 172.21.15.69:0/3574334331 conn(0x7ff9a0067b80 msgr2=0x7ff9a0142480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:10.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.573+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3574334331 shutdown_connections 2024-04-24T08:17:10.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.573+0000 7ff9a6a00640 1 -- 172.21.15.69:0/3574334331 wait complete. 2024-04-24T08:17:10.598 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:10.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.774+0000 7f4177e00640 1 Processor -- start 2024-04-24T08:17:10.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.774+0000 7f4177e00640 1 -- start start 2024-04-24T08:17:10.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.775+0000 7f4177e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4178156c60 0x7f417815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:10.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.775+0000 7f4177e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f41780ad7e0 0x7f417815d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:10.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.775+0000 7f4177e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f417815d660 con 0x7f41780b30f0 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.775+0000 7f4177e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f417806dc30 con 0x7f41780ad7e0 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.775+0000 7f4177e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f417806df10 con 0x7f4178156c60 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.776+0000 7f4177400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f41780ad7e0 0x7f417815d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.776+0000 7f4177400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f41780ad7e0 0x7f417815d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46626/0 (socket says 172.21.15.69:46626) 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.776+0000 7f4177400640 1 -- 172.21.15.69:0/977025497 learned_addr learned my addr 172.21.15.69:0/977025497 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.776+0000 7f4176a00640 1 --2- 172.21.15.69:0/977025497 >> v2:172.21.15.121:3300/0 conn(0x7f4178156c60 0x7f417815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.776+0000 7f4177400640 1 -- 172.21.15.69:0/977025497 >> v2:172.21.15.121:3300/0 conn(0x7f4178156c60 msgr2=0x7f417815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.776+0000 7f4177400640 1 --2- 172.21.15.69:0/977025497 >> v2:172.21.15.121:3300/0 conn(0x7f4178156c60 0x7f417815cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.776+0000 7f4177400640 1 -- 172.21.15.69:0/977025497 >> v1:172.21.15.69:6789/0 conn(0x7f41780b30f0 legacy=0x7f41780ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.776+0000 7f4177400640 1 -- 172.21.15.69:0/977025497 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417806e1f0 con 0x7f41780ad7e0 2024-04-24T08:17:10.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.776+0000 7f4176000640 1 -- 172.21.15.69:0/977025497 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728392815 0 0) 0x7f417815d660 con 0x7f41780b30f0 2024-04-24T08:17:10.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.776+0000 7f4176a00640 1 --2- 172.21.15.69:0/977025497 >> v2:172.21.15.121:3300/0 conn(0x7f4178156c60 0x7f417815cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:10.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.777+0000 7f4177400640 1 --2- 172.21.15.69:0/977025497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 0x7f417815d000 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f41780b5330 tx=0x7f417002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcd94369c44f898b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:10.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.778+0000 7f4176000640 1 -- 172.21.15.69:0/977025497 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170036070 con 0x7f41780ad7e0 2024-04-24T08:17:10.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.778+0000 7f4176000640 1 -- 172.21.15.69:0/977025497 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4170031600 con 0x7f41780ad7e0 2024-04-24T08:17:10.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.778+0000 7f4176000640 1 -- 172.21.15.69:0/977025497 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41700319d0 con 0x7f41780ad7e0 2024-04-24T08:17:10.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.778+0000 7f4177e00640 1 -- 172.21.15.69:0/977025497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 msgr2=0x7f417815d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:10.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.778+0000 7f4177e00640 1 --2- 172.21.15.69:0/977025497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 0x7f417815d000 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f41780b5330 tx=0x7f417002a730 comp rx=0 tx=0).stop 2024-04-24T08:17:10.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.778+0000 7f4177e00640 1 -- 172.21.15.69:0/977025497 shutdown_connections 2024-04-24T08:17:10.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.778+0000 7f4177e00640 1 --2- 172.21.15.69:0/977025497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 0x7f417815d000 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.778+0000 7f4177e00640 1 --2- 172.21.15.69:0/977025497 >> v2:172.21.15.121:3300/0 conn(0x7f4178156c60 0x7f417815cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.778+0000 7f4177e00640 1 -- 172.21.15.69:0/977025497 >> 172.21.15.69:0/977025497 conn(0x7f41780d0160 msgr2=0x7f41780d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:10.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.779+0000 7f4177e00640 1 -- 172.21.15.69:0/977025497 shutdown_connections 2024-04-24T08:17:10.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.779+0000 7f4177e00640 1 -- 172.21.15.69:0/977025497 wait complete. 2024-04-24T08:17:10.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.780+0000 7f4177e00640 1 Processor -- start 2024-04-24T08:17:10.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.780+0000 7f4177e00640 1 -- start start 2024-04-24T08:17:10.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.781+0000 7f4177e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 0x7f41781523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:10.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.781+0000 7f4177e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41780b30f0 0x7f41780996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:10.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.781+0000 7f4177e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4178156c60 0x7f41780b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:10.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.781+0000 7f4177e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f417806e710 con 0x7f41780ad7e0 2024-04-24T08:17:10.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.781+0000 7f4177e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f417815d940 con 0x7f41780b30f0 2024-04-24T08:17:10.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.781+0000 7f4177e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f417806e890 con 0x7f4178156c60 2024-04-24T08:17:10.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.781+0000 7f4176a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 0x7f41781523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:10.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f4176a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 0x7f41781523c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46630/0 (socket says 172.21.15.69:46630) 2024-04-24T08:17:10.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f4176a00640 1 -- 172.21.15.69:0/352638763 learned_addr learned my addr 172.21.15.69:0/352638763 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f416fe00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41780b30f0 0x7f41780996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f4177400640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4178156c60 0x7f41780b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f4176a00640 1 -- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4178156c60 msgr2=0x7f41780b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f4176a00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4178156c60 0x7f41780b0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f4176a00640 1 -- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41780b30f0 msgr2=0x7f41780996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f4176a00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41780b30f0 0x7f41780996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f4176a00640 1 -- 172.21.15.69:0/352638763 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41780b12f0 con 0x7f41780ad7e0 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f416fe00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41780b30f0 0x7f41780996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f4176a00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 0x7f41781523c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f416800c7f0 tx=0x7f416800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.782+0000 7f416ea00640 1 -- 172.21.15.69:0/352638763 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168015070 con 0x7f41780ad7e0 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.783+0000 7f416ea00640 1 -- 172.21.15.69:0/352638763 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4168007820 con 0x7f41780ad7e0 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.783+0000 7f416ea00640 1 -- 172.21.15.69:0/352638763 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168007bd0 con 0x7f41780ad7e0 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.783+0000 7f4177e00640 1 -- 172.21.15.69:0/352638763 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41781529a0 con 0x7f41780ad7e0 2024-04-24T08:17:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.783+0000 7f4177e00640 1 -- 172.21.15.69:0/352638763 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4178152f60 con 0x7f41780ad7e0 2024-04-24T08:17:10.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.784+0000 7f4177e00640 1 -- 172.21.15.69:0/352638763 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41780b5040 con 0x7f41780ad7e0 2024-04-24T08:17:10.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.785+0000 7f416ea00640 1 -- 172.21.15.69:0/352638763 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4168018070 con 0x7f41780ad7e0 2024-04-24T08:17:10.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.786+0000 7f416ea00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f415406c8f0 0x7f415406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:10.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.786+0000 7f416ea00640 1 -- 172.21.15.69:0/352638763 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f416808bcd0 con 0x7f41780ad7e0 2024-04-24T08:17:10.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.786+0000 7f416fe00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f415406c8f0 0x7f415406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:10.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.787+0000 7f416fe00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f415406c8f0 0x7f415406edb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4178154670 tx=0x7f4160009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:10.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:10.791+0000 7f416ea00640 1 -- 172.21.15.69:0/352638763 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41780b5040 con 0x7f41780ad7e0 2024-04-24T08:17:11.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.030+0000 7f4177e00640 1 -- 172.21.15.69:0/352638763 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41780aeba0 con 0x7f41780ad7e0 2024-04-24T08:17:11.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.030+0000 7f416ea00640 1 -- 172.21.15.69:0/352638763 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5174 (secure 0 0 0) 0x7f41780aeba0 con 0x7f41780ad7e0 2024-04-24T08:17:11.032 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:11.032 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":71,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:17:10.566261+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8151":{"gid":8151,"name":"a","rank":0,"incarnation":71,"state":"up:replay","state_seq":1,"addr":"172.21.15.69:6833/4254348458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4254348458},{"type":"v1","addr":"172.21.15.69:6833","nonce":4254348458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-04-24T08:17:11.032 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 71 2024-04-24T08:17:11.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.033+0000 7f4177e00640 1 -- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f415406c8f0 msgr2=0x7f415406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:11.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.033+0000 7f4177e00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f415406c8f0 0x7f415406edb0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4178154670 tx=0x7f4160009040 comp rx=0 tx=0).stop 2024-04-24T08:17:11.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.033+0000 7f4177e00640 1 -- 172.21.15.69:0/352638763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 msgr2=0x7f41781523c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:11.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.033+0000 7f4177e00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 0x7f41781523c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f416800c7f0 tx=0x7f416800ccb0 comp rx=0 tx=0).stop 2024-04-24T08:17:11.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.033+0000 7f4177e00640 1 -- 172.21.15.69:0/352638763 shutdown_connections 2024-04-24T08:17:11.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.033+0000 7f4177e00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f415406c8f0 0x7f415406edb0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:11.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.033+0000 7f4177e00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4178156c60 0x7f41780b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:11.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.033+0000 7f4177e00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41780b30f0 0x7f41780996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:11.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.033+0000 7f4177e00640 1 --2- 172.21.15.69:0/352638763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41780ad7e0 0x7f41781523c0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:11.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.033+0000 7f4177e00640 1 -- 172.21.15.69:0/352638763 >> 172.21.15.69:0/352638763 conn(0x7f41780d0160 msgr2=0x7f417806ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:11.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.034+0000 7f4177e00640 1 -- 172.21.15.69:0/352638763 shutdown_connections 2024-04-24T08:17:11.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:11.034+0000 7f4177e00640 1 -- 172.21.15.69:0/352638763 wait complete. 2024-04-24T08:17:11.455 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 71, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:15:23.442689+0000', 'modified': '2024-04-24T08:17:10.566261+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 86, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8151}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8151': {'gid': 8151, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.69:6833/4254348458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 4254348458}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 4254348458}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:17:11.455 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8151:up:replay 2024-04-24T08:17:12.456 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.667+0000 7f6695c00640 1 Processor -- start 2024-04-24T08:17:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.667+0000 7f6695c00640 1 -- start start 2024-04-24T08:17:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.668+0000 7f6695c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f669015ea10 0x7f66900d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.668+0000 7f6695c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f66901631a0 0x7f66900af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.668+0000 7f6695c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66900af7d0 0x7f6690071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.668+0000 7f6695c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f66900afc40 con 0x7f66900af7d0 2024-04-24T08:17:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.668+0000 7f6695c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f66900adab0 con 0x7f66901631a0 2024-04-24T08:17:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.668+0000 7f6695c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f66900add50 con 0x7f669015ea10 2024-04-24T08:17:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.669+0000 7f668f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66900af7d0 0x7f6690071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.669+0000 7f668f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f66900af7d0 0x7f6690071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46642/0 (socket says 172.21.15.69:46642) 2024-04-24T08:17:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.669+0000 7f668f400640 1 -- 172.21.15.69:0/2795771723 learned_addr learned my addr 172.21.15.69:0/2795771723 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.669+0000 7f668f400640 1 -- 172.21.15.69:0/2795771723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f669015ea10 msgr2=0x7f66900d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.669+0000 7f668f400640 1 --2- 172.21.15.69:0/2795771723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f669015ea10 0x7f66900d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.669+0000 7f668f400640 1 -- 172.21.15.69:0/2795771723 >> v2:172.21.15.121:3300/0 conn(0x7f66901631a0 msgr2=0x7f66900af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.669+0000 7f6687e00640 1 --2- 172.21.15.69:0/2795771723 >> v2:172.21.15.121:3300/0 conn(0x7f66901631a0 0x7f66900af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.669+0000 7f668f400640 1 --2- 172.21.15.69:0/2795771723 >> v2:172.21.15.121:3300/0 conn(0x7f66901631a0 0x7f66900af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.669+0000 7f668f400640 1 -- 172.21.15.69:0/2795771723 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66900ae030 con 0x7f66900af7d0 2024-04-24T08:17:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.669+0000 7f6687e00640 1 --2- 172.21.15.69:0/2795771723 >> v2:172.21.15.121:3300/0 conn(0x7f66901631a0 0x7f66900af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.670+0000 7f668f400640 1 --2- 172.21.15.69:0/2795771723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900af7d0 0x7f6690071d70 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f6690073210 tx=0x7f668802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14ea636dfcda87f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.671+0000 7f668e000640 1 -- 172.21.15.69:0/2795771723 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6688036070 con 0x7f66900af7d0 2024-04-24T08:17:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.671+0000 7f668e000640 1 -- 172.21.15.69:0/2795771723 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6688031600 con 0x7f66900af7d0 2024-04-24T08:17:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.671+0000 7f668e000640 1 -- 172.21.15.69:0/2795771723 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66880319d0 con 0x7f66900af7d0 2024-04-24T08:17:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.671+0000 7f6695c00640 1 -- 172.21.15.69:0/2795771723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900af7d0 msgr2=0x7f6690071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.671+0000 7f6695c00640 1 --2- 172.21.15.69:0/2795771723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900af7d0 0x7f6690071d70 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f6690073210 tx=0x7f668802a6a0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.672+0000 7f6695c00640 1 -- 172.21.15.69:0/2795771723 shutdown_connections 2024-04-24T08:17:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.672+0000 7f6695c00640 1 --2- 172.21.15.69:0/2795771723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66900af7d0 0x7f6690071d70 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.672+0000 7f6695c00640 1 --2- 172.21.15.69:0/2795771723 >> v2:172.21.15.121:3300/0 conn(0x7f66901631a0 0x7f66900af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.672+0000 7f6695c00640 1 --2- 172.21.15.69:0/2795771723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f669015ea10 0x7f66900d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.672+0000 7f6695c00640 1 -- 172.21.15.69:0/2795771723 >> 172.21.15.69:0/2795771723 conn(0x7f6690067d90 msgr2=0x7f669006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.672+0000 7f6695c00640 1 -- 172.21.15.69:0/2795771723 shutdown_connections 2024-04-24T08:17:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.672+0000 7f6695c00640 1 -- 172.21.15.69:0/2795771723 wait complete. 2024-04-24T08:17:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.673+0000 7f6695c00640 1 Processor -- start 2024-04-24T08:17:12.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.673+0000 7f6695c00640 1 -- start start 2024-04-24T08:17:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.674+0000 7f6695c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66900af7d0 0x7f66900cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.674+0000 7f6695c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f669015ea10 0x7f669014d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.674+0000 7f6695c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66901631a0 0x7f669014ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.674+0000 7f6695c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f66900ae8d0 con 0x7f669015ea10 2024-04-24T08:17:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.674+0000 7f6695c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f66900ae5f0 con 0x7f66900af7d0 2024-04-24T08:17:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.674+0000 7f6695c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f66900aedf0 con 0x7f66901631a0 2024-04-24T08:17:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f6687e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f669015ea10 0x7f669014d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f6687e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f669015ea10 0x7f669014d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46652/0 (socket says 172.21.15.69:46652) 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f6687e00640 1 -- 172.21.15.69:0/2736672041 learned_addr learned my addr 172.21.15.69:0/2736672041 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f668f400640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66901631a0 0x7f669014ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f6687e00640 1 -- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66901631a0 msgr2=0x7f669014ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f6687e00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66901631a0 0x7f669014ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f668ea00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66900af7d0 0x7f66900cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f6687e00640 1 -- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66900af7d0 msgr2=0x7f66900cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f6687e00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66900af7d0 0x7f66900cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f6687e00640 1 -- 172.21.15.69:0/2736672041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66900b0110 con 0x7f669015ea10 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f668ea00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66900af7d0 0x7f66900cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.675+0000 7f6687e00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f669015ea10 0x7f669014d180 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f667800c820 tx=0x7f667800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.676+0000 7f6686a00640 1 -- 172.21.15.69:0/2736672041 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678015070 con 0x7f669015ea10 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.676+0000 7f6686a00640 1 -- 172.21.15.69:0/2736672041 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6678007a30 con 0x7f669015ea10 2024-04-24T08:17:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.676+0000 7f6686a00640 1 -- 172.21.15.69:0/2736672041 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678007de0 con 0x7f669015ea10 2024-04-24T08:17:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.676+0000 7f6695c00640 1 -- 172.21.15.69:0/2736672041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66900b03f0 con 0x7f669015ea10 2024-04-24T08:17:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.676+0000 7f6695c00640 1 -- 172.21.15.69:0/2736672041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66900b09b0 con 0x7f669015ea10 2024-04-24T08:17:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.677+0000 7f6695c00640 1 -- 172.21.15.69:0/2736672041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6690068a80 con 0x7f669015ea10 2024-04-24T08:17:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.678+0000 7f6686a00640 1 -- 172.21.15.69:0/2736672041 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6678018070 con 0x7f669015ea10 2024-04-24T08:17:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.679+0000 7f6686a00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 0x7f666c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.679+0000 7f6686a00640 1 -- 172.21.15.69:0/2736672041 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f667808bb20 con 0x7f669015ea10 2024-04-24T08:17:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.679+0000 7f668ea00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 0x7f666c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:12.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.681+0000 7f668ea00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 0x7f666c06edb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6680006fd0 tx=0x7f6680008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:12.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.683+0000 7f6686a00640 1 -- 172.21.15.69:0/2736672041 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6678059e70 con 0x7f669015ea10 2024-04-24T08:17:12.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.932+0000 7f6695c00640 1 -- 172.21.15.69:0/2736672041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f669006ddd0 con 0x7f669015ea10 2024-04-24T08:17:12.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.933+0000 7f6686a00640 1 -- 172.21.15.69:0/2736672041 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) ==== 94+0+5174 (secure 0 0 0) 0x7f6678059770 con 0x7f669015ea10 2024-04-24T08:17:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:12.935 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":73,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:17:11.585035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8151":{"gid":8151,"name":"a","rank":0,"incarnation":71,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.69:6833/4254348458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4254348458},{"type":"v1","addr":"172.21.15.69:6833","nonce":4254348458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":6}]} 2024-04-24T08:17:12.935 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 73 2024-04-24T08:17:12.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.936+0000 7f6695c00640 1 -- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 msgr2=0x7f666c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:12.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.936+0000 7f6695c00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 0x7f666c06edb0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6680006fd0 tx=0x7f6680008040 comp rx=0 tx=0).stop 2024-04-24T08:17:12.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.936+0000 7f6695c00640 1 -- 172.21.15.69:0/2736672041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f669015ea10 msgr2=0x7f669014d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:12.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.936+0000 7f6695c00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f669015ea10 0x7f669014d180 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f667800c820 tx=0x7f667800cce0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.936+0000 7f6695c00640 1 -- 172.21.15.69:0/2736672041 shutdown_connections 2024-04-24T08:17:12.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.936+0000 7f6695c00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 0x7f666c06edb0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.937+0000 7f6695c00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f669015ea10 0x7f669014d180 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.937+0000 7f6695c00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66901631a0 0x7f669014ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.937+0000 7f6695c00640 1 --2- 172.21.15.69:0/2736672041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66900af7d0 0x7f66900cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:12.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.937+0000 7f6695c00640 1 -- 172.21.15.69:0/2736672041 >> 172.21.15.69:0/2736672041 conn(0x7f6690067d90 msgr2=0x7f6690071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:12.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.937+0000 7f6695c00640 1 -- 172.21.15.69:0/2736672041 shutdown_connections 2024-04-24T08:17:12.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:12.937+0000 7f6695c00640 1 -- 172.21.15.69:0/2736672041 wait complete. 2024-04-24T08:17:13.360 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:15:23.442689+0000', 'modified': '2024-04-24T08:17:11.585035+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 86, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8151}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8151': {'gid': 8151, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.69:6833/4254348458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 4254348458}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 4254348458}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:17:13.360 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8151:up:rejoin 2024-04-24T08:17:14.361 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:14.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.568+0000 7f1f19200640 1 Processor -- start 2024-04-24T08:17:14.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.568+0000 7f1f19200640 1 -- start start 2024-04-24T08:17:14.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.569+0000 7f1f19200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f14145a90 0x7f1f1406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:14.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.570+0000 7f1f19200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f140cf3d0 con 0x7f1f140d0300 2024-04-24T08:17:14.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.570+0000 7f1f19200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f140cf6b0 con 0x7f1f14145fe0 2024-04-24T08:17:14.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.570+0000 7f1f19200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f140cf990 con 0x7f1f14145a90 2024-04-24T08:17:14.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.570+0000 7f1f11e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1f140d0300 0x7f1f140d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39082/0 (socket says 172.21.15.69:39082) 2024-04-24T08:17:14.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.570+0000 7f1f11e00640 1 -- 172.21.15.69:0/1758106723 learned_addr learned my addr 172.21.15.69:0/1758106723 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:14.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.570+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565222377 0 0) 0x7f1f140cf3d0 con 0x7f1f140d0300 2024-04-24T08:17:14.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.570+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ef8003520 con 0x7f1f140d0300 2024-04-24T08:17:14.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.570+0000 7f1f09400640 1 --2- 172.21.15.69:0/1758106723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f14145a90 0x7f1f1406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:14.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2754926971 0 0) 0x7f1ef8003520 con 0x7f1f140d0300 2024-04-24T08:17:14.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f14145a90 msgr2=0x7f1f1406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:14.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 --2- 172.21.15.69:0/1758106723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f14145a90 0x7f1f1406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:14.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 >> v1:172.21.15.121:6789/0 conn(0x7f1f14145fe0 legacy=0x7f1f14160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:14.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f140cfc70 con 0x7f1f140d0300 2024-04-24T08:17:14.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f1f04001670 con 0x7f1f140d0300 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 >> v1:172.21.15.69:6789/0 conn(0x7f1f140d0300 legacy=0x7f1f140d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 --2- 172.21.15.69:0/1758106723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ef80039e0 0x7f1ef8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 --2- 172.21.15.69:0/1758106723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f14145a90 0x7f1ef8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 --2- 172.21.15.69:0/1758106723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ef8004d40 0x7f1ef80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1f140cf990 con 0x7f1ef80039e0 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef800bb50 con 0x7f1ef8004d40 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1ef800bd20 con 0x7f1f14145a90 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11e00640 1 --2- 172.21.15.69:0/1758106723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ef80039e0 0x7f1ef8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.571+0000 7f1f11e00640 1 -- 172.21.15.69:0/1758106723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f14145a90 msgr2=0x7f1ef8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.572+0000 7f1f11e00640 1 --2- 172.21.15.69:0/1758106723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1f14145a90 0x7f1ef8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.572+0000 7f1f11e00640 1 -- 172.21.15.69:0/1758106723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ef8004d40 msgr2=0x7f1ef80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.572+0000 7f1f11e00640 1 --2- 172.21.15.69:0/1758106723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ef8004d40 0x7f1ef80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.572+0000 7f1f11e00640 1 -- 172.21.15.69:0/1758106723 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef8003520 con 0x7f1ef80039e0 2024-04-24T08:17:14.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.572+0000 7f1f11e00640 1 --2- 172.21.15.69:0/1758106723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ef80039e0 0x7f1ef8003e00 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400c270 tx=0x7f1ef8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efda05db6608cc1d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:14.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.573+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04007d10 con 0x7f1ef80039e0 2024-04-24T08:17:14.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.573+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f04005ad0 con 0x7f1ef80039e0 2024-04-24T08:17:14.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.573+0000 7f1f11400640 1 -- 172.21.15.69:0/1758106723 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04007070 con 0x7f1ef80039e0 2024-04-24T08:17:14.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.573+0000 7f1f19200640 1 -- 172.21.15.69:0/1758106723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ef80039e0 msgr2=0x7f1ef8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:14.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.573+0000 7f1f19200640 1 --2- 172.21.15.69:0/1758106723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1ef80039e0 0x7f1ef8003e00 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400c270 tx=0x7f1ef8005830 comp rx=0 tx=0).stop 2024-04-24T08:17:14.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.573+0000 7f1f12800640 1 -- 172.21.15.69:0/1758106723 reap_dead start 2024-04-24T08:17:14.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.573+0000 7f1f19200640 1 -- 172.21.15.69:0/1758106723 shutdown_connections 2024-04-24T08:17:14.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.573+0000 7f1f19200640 1 -- 172.21.15.69:0/1758106723 >> 172.21.15.69:0/1758106723 conn(0x7f1f14067d90 msgr2=0x7f1f1406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:14.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.574+0000 7f1f19200640 1 -- 172.21.15.69:0/1758106723 shutdown_connections 2024-04-24T08:17:14.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.574+0000 7f1f19200640 1 -- 172.21.15.69:0/1758106723 wait complete. 2024-04-24T08:17:14.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.575+0000 7f1f19200640 1 Processor -- start 2024-04-24T08:17:14.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.576+0000 7f1f19200640 1 -- start start 2024-04-24T08:17:14.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f19200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ef80039e0 0x7f1f140ac570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:14.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f19200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f1415bd80 0x7f1f1415c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:14.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f19200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f140b10f0 0x7f1f140b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f19200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1f141458f0 con 0x7f1f140b10f0 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f19200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1f14145770 con 0x7f1f1415bd80 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f19200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1f140cdc30 con 0x7f1ef80039e0 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f12800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f140b10f0 0x7f1f140b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f12800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f140b10f0 0x7f1f140b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46680/0 (socket says 172.21.15.69:46680) 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f12800640 1 -- 172.21.15.69:0/1280827492 learned_addr learned my addr 172.21.15.69:0/1280827492 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f12800640 1 -- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ef80039e0 msgr2=0x7f1f140ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f11e00640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ef80039e0 0x7f1f140ac570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f09400640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f1415bd80 0x7f1f1415c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f12800640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ef80039e0 0x7f1f140ac570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f12800640 1 -- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f1415bd80 msgr2=0x7f1f1415c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f12800640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f1415bd80 0x7f1f1415c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.577+0000 7f1f12800640 1 -- 172.21.15.69:0/1280827492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f140b1e20 con 0x7f1f140b10f0 2024-04-24T08:17:14.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.578+0000 7f1f12800640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f140b10f0 0x7f1f140b1570 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f1f140b33d0 tx=0x7f1f0c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:14.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.578+0000 7f1f0aa00640 1 -- 172.21.15.69:0/1280827492 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c017070 con 0x7f1f140b10f0 2024-04-24T08:17:14.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.578+0000 7f1f0aa00640 1 -- 172.21.15.69:0/1280827492 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f0c005d60 con 0x7f1f140b10f0 2024-04-24T08:17:14.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.578+0000 7f1f0aa00640 1 -- 172.21.15.69:0/1280827492 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c00a850 con 0x7f1f140b10f0 2024-04-24T08:17:14.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.578+0000 7f1f19200640 1 -- 172.21.15.69:0/1280827492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f140b2100 con 0x7f1f140b10f0 2024-04-24T08:17:14.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.578+0000 7f1f19200640 1 -- 172.21.15.69:0/1280827492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f140b26c0 con 0x7f1f140b10f0 2024-04-24T08:17:14.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.579+0000 7f1f19200640 1 -- 172.21.15.69:0/1280827492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f14068a80 con 0x7f1f140b10f0 2024-04-24T08:17:14.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.580+0000 7f1f0aa00640 1 -- 172.21.15.69:0/1280827492 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f0c00a9f0 con 0x7f1f140b10f0 2024-04-24T08:17:14.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.582+0000 7f1f0aa00640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ef006c920 0x7f1ef006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:14.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.582+0000 7f1f0aa00640 1 -- 172.21.15.69:0/1280827492 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f1f0c08aea0 con 0x7f1f140b10f0 2024-04-24T08:17:14.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.582+0000 7f1f11e00640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ef006c920 0x7f1ef006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:14.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.584+0000 7f1f11e00640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ef006c920 0x7f1ef006ede0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1f04033000 tx=0x7f1f04035040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:14.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.587+0000 7f1f0aa00640 1 -- 172.21.15.69:0/1280827492 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f0c0591f0 con 0x7f1f140b10f0 2024-04-24T08:17:14.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.829+0000 7f1f19200640 1 -- 172.21.15.69:0/1280827492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f1406f600 con 0x7f1f140b10f0 2024-04-24T08:17:14.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.831+0000 7f1f0aa00640 1 -- 172.21.15.69:0/1280827492 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7f1f0c058af0 con 0x7f1f140b10f0 2024-04-24T08:17:14.833 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:14.833 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:17:13.584835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8151":{"gid":8151,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6833/4254348458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4254348458},{"type":"v1","addr":"172.21.15.69:6833","nonce":4254348458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8151,"qdb_cluster":[8151]},"id":6}]} 2024-04-24T08:17:14.833 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 74 2024-04-24T08:17:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 -- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ef006c920 msgr2=0x7f1ef006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ef006c920 0x7f1ef006ede0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f1f04033000 tx=0x7f1f04035040 comp rx=0 tx=0).stop 2024-04-24T08:17:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 -- 172.21.15.69:0/1280827492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f140b10f0 msgr2=0x7f1f140b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f140b10f0 0x7f1f140b1570 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f1f140b33d0 tx=0x7f1f0c00cb90 comp rx=0 tx=0).stop 2024-04-24T08:17:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 -- 172.21.15.69:0/1280827492 shutdown_connections 2024-04-24T08:17:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1ef006c920 0x7f1ef006ede0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1f140b10f0 0x7f1f140b1570 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1f1415bd80 0x7f1f1415c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 --2- 172.21.15.69:0/1280827492 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1ef80039e0 0x7f1f140ac570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 -- 172.21.15.69:0/1280827492 >> 172.21.15.69:0/1280827492 conn(0x7f1f14067d90 msgr2=0x7f1f140b0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 -- 172.21.15.69:0/1280827492 shutdown_connections 2024-04-24T08:17:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:14.834+0000 7f1f19200640 1 -- 172.21.15.69:0/1280827492 wait complete. 2024-04-24T08:17:15.257 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:15:23.442689+0000', 'modified': '2024-04-24T08:17:13.584835+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 86, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8151}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8151': {'gid': 8151, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.69:6833/4254348458', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6832', 'nonce': 4254348458}, {'type': 'v1', 'addr': '172.21.15.69:6833', 'nonce': 4254348458}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8151, 'qdb_cluster': [8151]} 2024-04-24T08:17:15.257 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:17:15.257 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-24T08:17:15.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.468+0000 7f51f0e00640 1 Processor -- start 2024-04-24T08:17:15.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.469+0000 7f51f0e00640 1 -- start start 2024-04-24T08:17:15.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.470+0000 7f51f0e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f51ec145a90 0x7f51ec06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.470+0000 7f51f0e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f51ec145fe0 0x7f51ec160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.470+0000 7f51f0e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51ec0cf3d0 con 0x7f51ec0d0300 2024-04-24T08:17:15.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.470+0000 7f51f0e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f51ec0cf6b0 con 0x7f51ec145a90 2024-04-24T08:17:15.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.470+0000 7f51f0e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f51ec0cf990 con 0x7f51ec145fe0 2024-04-24T08:17:15.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.470+0000 7f51e9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f51ec145a90 0x7f51ec06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.470+0000 7f51e9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f51ec145a90 0x7f51ec06e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46686/0 (socket says 172.21.15.69:46686) 2024-04-24T08:17:15.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.470+0000 7f51e9400640 1 -- 172.21.15.69:0/906418240 learned_addr learned my addr 172.21.15.69:0/906418240 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:15.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.471+0000 7f51ea800640 1 --2- 172.21.15.69:0/906418240 >> v2:172.21.15.121:3300/0 conn(0x7f51ec145fe0 0x7f51ec160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.471+0000 7f51e9400640 1 -- 172.21.15.69:0/906418240 >> v2:172.21.15.121:3300/0 conn(0x7f51ec145fe0 msgr2=0x7f51ec160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:15.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.471+0000 7f51e9400640 1 --2- 172.21.15.69:0/906418240 >> v2:172.21.15.121:3300/0 conn(0x7f51ec145fe0 0x7f51ec160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.471+0000 7f51e9400640 1 -- 172.21.15.69:0/906418240 >> v1:172.21.15.121:6789/0 conn(0x7f51ec0d0300 legacy=0x7f51ec0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:15.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.471+0000 7f51e9400640 1 -- 172.21.15.69:0/906418240 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec0cfc70 con 0x7f51ec145a90 2024-04-24T08:17:15.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.472+0000 7f51e9400640 1 --2- 172.21.15.69:0/906418240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145a90 0x7f51ec06e240 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0d1050 tx=0x7f51d402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1426cbc3ae08cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:15.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.472+0000 7f51e8a00640 1 -- 172.21.15.69:0/906418240 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4036070 con 0x7f51ec145a90 2024-04-24T08:17:15.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.472+0000 7f51e8a00640 1 -- 172.21.15.69:0/906418240 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d4031600 con 0x7f51ec145a90 2024-04-24T08:17:15.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.472+0000 7f51e8a00640 1 -- 172.21.15.69:0/906418240 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d40319d0 con 0x7f51ec145a90 2024-04-24T08:17:15.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.473+0000 7f51f0e00640 1 -- 172.21.15.69:0/906418240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145a90 msgr2=0x7f51ec06e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.473+0000 7f51f0e00640 1 --2- 172.21.15.69:0/906418240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145a90 0x7f51ec06e240 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0d1050 tx=0x7f51d402a6f0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.473+0000 7f51f0e00640 1 -- 172.21.15.69:0/906418240 shutdown_connections 2024-04-24T08:17:15.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.473+0000 7f51f0e00640 1 --2- 172.21.15.69:0/906418240 >> v2:172.21.15.121:3300/0 conn(0x7f51ec145fe0 0x7f51ec160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.473+0000 7f51f0e00640 1 --2- 172.21.15.69:0/906418240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145a90 0x7f51ec06e240 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.473+0000 7f51f0e00640 1 -- 172.21.15.69:0/906418240 >> 172.21.15.69:0/906418240 conn(0x7f51ec067d90 msgr2=0x7f51ec06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:15.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.473+0000 7f51f0e00640 1 -- 172.21.15.69:0/906418240 shutdown_connections 2024-04-24T08:17:15.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.473+0000 7f51f0e00640 1 -- 172.21.15.69:0/906418240 wait complete. 2024-04-24T08:17:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.475+0000 7f51f0e00640 1 Processor -- start 2024-04-24T08:17:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.475+0000 7f51f0e00640 1 -- start start 2024-04-24T08:17:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51f0e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ec0d0300 0x7f51ec1e5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51f0e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ec145a90 0x7f51ec13d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51f0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145fe0 0x7f51ec15ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51f0e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec1458f0 con 0x7f51ec145fe0 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51f0e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec145770 con 0x7f51ec145a90 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51f0e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f51ec0cdc30 con 0x7f51ec0d0300 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51ea800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145fe0 0x7f51ec15ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51ea800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145fe0 0x7f51ec15ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46698/0 (socket says 172.21.15.69:46698) 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51ea800640 1 -- 172.21.15.69:0/3651109711 learned_addr learned my addr 172.21.15.69:0/3651109711 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51e9e00640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ec0d0300 0x7f51ec1e5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51e9400640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ec145a90 0x7f51ec13d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51ea800640 1 -- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ec0d0300 msgr2=0x7f51ec1e5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51ea800640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ec0d0300 0x7f51ec1e5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51ea800640 1 -- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ec145a90 msgr2=0x7f51ec13d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51ea800640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ec145a90 0x7f51ec13d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51ea800640 1 -- 172.21.15.69:0/3651109711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec13d5a0 con 0x7f51ec145fe0 2024-04-24T08:17:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.476+0000 7f51e9400640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ec145a90 0x7f51ec13d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.477+0000 7f51ea800640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145fe0 0x7f51ec15ead0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f51e400c7f0 tx=0x7f51e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.477+0000 7f51e2000640 1 -- 172.21.15.69:0/3651109711 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e4015070 con 0x7f51ec145fe0 2024-04-24T08:17:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.477+0000 7f51e2000640 1 -- 172.21.15.69:0/3651109711 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e4004220 con 0x7f51ec145fe0 2024-04-24T08:17:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.477+0000 7f51e2000640 1 -- 172.21.15.69:0/3651109711 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e4007620 con 0x7f51ec145fe0 2024-04-24T08:17:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.477+0000 7f51f0e00640 1 -- 172.21.15.69:0/3651109711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ec14dca0 con 0x7f51ec145fe0 2024-04-24T08:17:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.477+0000 7f51f0e00640 1 -- 172.21.15.69:0/3651109711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ec14e260 con 0x7f51ec145fe0 2024-04-24T08:17:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.480+0000 7f51e2000640 1 -- 172.21.15.69:0/3651109711 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f51e40079d0 con 0x7f51ec145fe0 2024-04-24T08:17:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.481+0000 7f51e2000640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f51c806c920 0x7f51c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.481+0000 7f51e2000640 1 -- 172.21.15.69:0/3651109711 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f51e408bc10 con 0x7f51ec145fe0 2024-04-24T08:17:15.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.482+0000 7f51e9e00640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f51c806c920 0x7f51c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.482+0000 7f51f3b65740 1 Processor -- start 2024-04-24T08:17:15.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.482+0000 7f51f3b65740 1 -- start start 2024-04-24T08:17:15.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.483+0000 7f51e9e00640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f51c806c920 0x7f51c806ede0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00a730 tx=0x7f51dc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:15.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.483+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f280787b0 0x564f2802b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.483+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805b8d0 0x564f2805ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.483+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805b190 0x564f2802ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.483+0000 7f51f3b65740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564f27f95b10 con 0x564f2805b8d0 2024-04-24T08:17:15.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.483+0000 7f51f3b65740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564f27f3c2b0 con 0x564f2805b190 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.483+0000 7f51f3b65740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564f27f91ad0 con 0x564f280787b0 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.483+0000 7f51e9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805b8d0 0x564f2805ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.483+0000 7f51e9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805b8d0 0x564f2805ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46714/0 (socket says 172.21.15.69:46714) 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.483+0000 7f51e9400640 1 -- 172.21.15.69:0/323898194 learned_addr learned my addr 172.21.15.69:0/323898194 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.484+0000 7f51e9e00640 1 --2- 172.21.15.69:0/323898194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f280787b0 0x564f2802b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.484+0000 7f51ea800640 1 --2- 172.21.15.69:0/323898194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805b190 0x564f2802ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.484+0000 7f51e9400640 1 -- 172.21.15.69:0/323898194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f280787b0 msgr2=0x564f2802b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.484+0000 7f51e9400640 1 --2- 172.21.15.69:0/323898194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f280787b0 0x564f2802b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.484+0000 7f51e9400640 1 -- 172.21.15.69:0/323898194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805b190 msgr2=0x564f2802ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.484+0000 7f51e9400640 1 --2- 172.21.15.69:0/323898194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805b190 0x564f2802ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.484+0000 7f51e9400640 1 -- 172.21.15.69:0/323898194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f27ca82d0 con 0x564f2805b8d0 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.484+0000 7f51e9400640 1 --2- 172.21.15.69:0/323898194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805b8d0 0x564f2805ac50 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f51d402ace0 tx=0x7f51d402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1022bcf270e492e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51daa00640 1 -- 172.21.15.69:0/323898194 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4036040 con 0x564f2805b8d0 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51daa00640 1 -- 172.21.15.69:0/323898194 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d4042070 con 0x564f2805b8d0 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51daa00640 1 -- 172.21.15.69:0/323898194 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4031940 con 0x564f2805b8d0 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51f3b65740 1 -- 172.21.15.69:0/323898194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805b8d0 msgr2=0x564f2805ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51f3b65740 1 --2- 172.21.15.69:0/323898194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805b8d0 0x564f2805ac50 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f51d402ace0 tx=0x7f51d402e000 comp rx=0 tx=0).stop 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51f3b65740 1 -- 172.21.15.69:0/323898194 shutdown_connections 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51f3b65740 1 --2- 172.21.15.69:0/323898194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805b8d0 0x564f2805ac50 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51f3b65740 1 --2- 172.21.15.69:0/323898194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805b190 0x564f2802ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51f3b65740 1 --2- 172.21.15.69:0/323898194 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f280787b0 0x564f2802b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51f3b65740 1 -- 172.21.15.69:0/323898194 >> 172.21.15.69:0/323898194 conn(0x564f2802d3b0 msgr2=0x564f28075c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51f3b65740 1 -- 172.21.15.69:0/323898194 shutdown_connections 2024-04-24T08:17:15.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.485+0000 7f51f3b65740 1 -- 172.21.15.69:0/323898194 wait complete. 2024-04-24T08:17:15.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.486+0000 7f51f3b65740 1 -- start start 2024-04-24T08:17:15.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.486+0000 7f51f3b65740 1 Processor -- start 2024-04-24T08:17:15.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.487+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f2805b8d0 0x564f2806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.487+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f280787b0 0x564f2805d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.487+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f28079070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.487+0000 7f51f3b65740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564f2802f560 con 0x564f2805d7d0 2024-04-24T08:17:15.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.487+0000 7f51f3b65740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564f27f4f590 con 0x564f280787b0 2024-04-24T08:17:15.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.487+0000 7f51f3b65740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564f2802e950 con 0x564f2805b8d0 2024-04-24T08:17:15.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.487+0000 7f51f3b65740 10 client.0 _ll_register_callbacks cb 0x564f27f323a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:15.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51ea800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f28079070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51ea800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f28079070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46728/0 (socket says 172.21.15.69:46728) 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51ea800640 1 -- 172.21.15.69:0/3130457290 learned_addr learned my addr 172.21.15.69:0/3130457290 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51e9400640 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f280787b0 0x564f2805d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51e9e00640 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f2805b8d0 0x564f2806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51ea800640 1 -- 172.21.15.69:0/3130457290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f2805b8d0 msgr2=0x564f2806dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51ea800640 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f2805b8d0 0x564f2806dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51ea800640 1 -- 172.21.15.69:0/3130457290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f280787b0 msgr2=0x564f2805d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51ea800640 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f280787b0 0x564f2805d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51ea800640 1 -- 172.21.15.69:0/3130457290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f27d726b0 con 0x564f2805d7d0 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51e9400640 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f280787b0 0x564f2805d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51ea800640 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f28079070 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f51e408a060 tx=0x7f51e40599a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3079552919a74ddf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.488+0000 7f51d3400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51d3400640 1 -- 172.21.15.69:0/3130457290 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e401e040 con 0x564f2805d7d0 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51d3400640 1 -- 172.21.15.69:0/3130457290 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f51e405ab90 con 0x564f2805d7d0 2024-04-24T08:17:15.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51f3b65740 1 -- 172.21.15.69:0/3130457290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d4042070 con 0x564f2805d7d0 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51d3400640 1 -- 172.21.15.69:0/3130457290 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f51e405ad90 con 0x564f2805d7d0 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51f3b65740 10 client.8235 fetch_fsmap learned FSMap version 74 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51f3b65740 1 -- 172.21.15.69:0/3130457290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x564f2805e560 con 0x564f2805d7d0 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51d3400640 1 -- 172.21.15.69:0/3130457290 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d4042070 con 0x564f2805d7d0 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51d3400640 1 -- 172.21.15.69:0/3130457290 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f51e4058cb0 con 0x564f2805d7d0 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51f3b65740 10 client.8235 fetch_fsmap finished waiting for FSMap version 74 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51f3b65740 10 client.8235 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51f3b65740 20 client.8235 populate_metadata read hostname 'smithi069' 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51f3b65740 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x564f28049500 0x564f2805c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51f3b65740 4 client.8235 mds_command: new command op to 8151 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.489+0000 7f51f3b65740 1 -- 172.21.15.69:0/3130457290 --> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f27fa3530 con 0x564f28049500 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.490+0000 7f51e9e00640 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x564f28049500 0x564f2805c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.490+0000 7f51e9e00640 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x564f28049500 0x564f2805c6f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d080516feec9d14 server_cookie=c819256c356e72f5 in_seq=0 out_seq=0 2024-04-24T08:17:15.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.490+0000 7f51d3400640 10 client.8235 ms_handle_connect on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51d3400640 1 -- 172.21.15.69:0/3130457290 <== mds.0 v2:172.21.15.69:6832/4254348458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564f27fa3530 con 0x564f28049500 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51d3400640 10 client.8235 handle_command_reply: tid=0 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51f3b65740 1 client.8235 shutdown 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51f3b65740 1 -- 172.21.15.69:0/3130457290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 msgr2=0x564f28079070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51f3b65740 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f28079070 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f51e408a060 tx=0x7f51e40599a0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51f3b65740 1 -- 172.21.15.69:0/3130457290 shutdown_connections 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51f3b65740 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x564f28049500 0x564f2805c6f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51f3b65740 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f28079070 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51f3b65740 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f280787b0 0x564f2805d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51f3b65740 1 --2- 172.21.15.69:0/3130457290 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f2805b8d0 0x564f2806dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.491+0000 7f51d3400640 0 client.8235 ms_handle_reset on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.492+0000 7f51f3b65740 1 -- 172.21.15.69:0/3130457290 >> 172.21.15.69:0/3130457290 conn(0x564f2805b190 msgr2=0x564f28075a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:15.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.492+0000 7f51f3b65740 1 -- 172.21.15.69:0/3130457290 shutdown_connections 2024-04-24T08:17:15.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.492+0000 7f51f3b65740 1 -- 172.21.15.69:0/3130457290 wait complete. 2024-04-24T08:17:15.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.492+0000 7f51f3b65740 20 client.8235 trim_cache size 0 max 16384 2024-04-24T08:17:15.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.511+0000 7f51f3b65740 1 Processor -- start 2024-04-24T08:17:15.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.511+0000 7f51f3b65740 1 -- start start 2024-04-24T08:17:15.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.512+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f2805b8d0 0x564f2806bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.512+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805d7d0 0x564f2806e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.512+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f280787b0 0x564f28074e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.512+0000 7f51f3b65740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec0b48f0 con 0x564f280787b0 2024-04-24T08:17:15.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.512+0000 7f51f3b65740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec13e7d0 con 0x564f2805d7d0 2024-04-24T08:17:15.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.512+0000 7f51f3b65740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f51ec13ee20 con 0x564f2805b8d0 2024-04-24T08:17:15.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.513+0000 7f51ea800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f280787b0 0x564f28074e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.513+0000 7f51ea800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f280787b0 0x564f28074e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46738/0 (socket says 172.21.15.69:46738) 2024-04-24T08:17:15.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.513+0000 7f51ea800640 1 -- 172.21.15.69:0/1968344913 learned_addr learned my addr 172.21.15.69:0/1968344913 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:15.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.513+0000 7f51e9e00640 1 --2- 172.21.15.69:0/1968344913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f2805b8d0 0x564f2806bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.513+0000 7f51e9400640 1 --2- 172.21.15.69:0/1968344913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805d7d0 0x564f2806e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.513+0000 7f51ea800640 1 -- 172.21.15.69:0/1968344913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f2805b8d0 msgr2=0x564f2806bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:15.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.513+0000 7f51ea800640 1 --2- 172.21.15.69:0/1968344913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f2805b8d0 0x564f2806bff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.513+0000 7f51ea800640 1 -- 172.21.15.69:0/1968344913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805d7d0 msgr2=0x564f2806e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.513+0000 7f51ea800640 1 --2- 172.21.15.69:0/1968344913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805d7d0 0x564f2806e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.513+0000 7f51ea800640 1 -- 172.21.15.69:0/1968344913 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f280793e0 con 0x564f280787b0 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.514+0000 7f51ea800640 1 --2- 172.21.15.69:0/1968344913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f280787b0 0x564f28074e80 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f51e4055cc0 tx=0x7f51e4060050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0a2fe3917444cb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.514+0000 7f51da000640 1 -- 172.21.15.69:0/1968344913 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e401e040 con 0x564f280787b0 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.514+0000 7f51da000640 1 -- 172.21.15.69:0/1968344913 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e4058cb0 con 0x564f280787b0 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.514+0000 7f51da000640 1 -- 172.21.15.69:0/1968344913 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e40557b0 con 0x564f280787b0 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.514+0000 7f51f3b65740 1 -- 172.21.15.69:0/1968344913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f280787b0 msgr2=0x564f28074e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.514+0000 7f51f3b65740 1 --2- 172.21.15.69:0/1968344913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f280787b0 0x564f28074e80 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f51e4055cc0 tx=0x7f51e4060050 comp rx=0 tx=0).stop 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.515+0000 7f51f3b65740 1 -- 172.21.15.69:0/1968344913 shutdown_connections 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.515+0000 7f51f3b65740 1 --2- 172.21.15.69:0/1968344913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f280787b0 0x564f28074e80 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.515+0000 7f51f3b65740 1 --2- 172.21.15.69:0/1968344913 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805d7d0 0x564f2806e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.515+0000 7f51f3b65740 1 --2- 172.21.15.69:0/1968344913 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f2805b8d0 0x564f2806bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.515+0000 7f51f3b65740 1 -- 172.21.15.69:0/1968344913 >> 172.21.15.69:0/1968344913 conn(0x564f2805b190 msgr2=0x564f28026750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:15.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.515+0000 7f51f3b65740 1 -- 172.21.15.69:0/1968344913 shutdown_connections 2024-04-24T08:17:15.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.515+0000 7f51f3b65740 1 -- 172.21.15.69:0/1968344913 wait complete. 2024-04-24T08:17:15.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.515+0000 7f51f3b65740 1 -- start start 2024-04-24T08:17:15.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.516+0000 7f51f3b65740 1 Processor -- start 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.516+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805b8d0 0x564f28056090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.516+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f2802f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51f3b65740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f280787b0 0x564f2802fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51f3b65740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564f28049380 con 0x564f2805d7d0 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51f3b65740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564f2805e840 con 0x564f2805b8d0 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51f3b65740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564f280499d0 con 0x564f280787b0 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51f3b65740 10 client.0 _ll_register_callbacks cb 0x7f51e4059860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51e9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f2802f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51e9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f2802f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46748/0 (socket says 172.21.15.69:46748) 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51e9400640 1 -- 172.21.15.69:0/1384456324 learned_addr learned my addr 172.21.15.69:0/1384456324 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51e9400640 1 -- 172.21.15.69:0/1384456324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f280787b0 msgr2=0x564f2802fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51ea800640 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f280787b0 0x564f2802fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51e9e00640 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805b8d0 0x564f28056090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51e9400640 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f280787b0 0x564f2802fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51e9400640 1 -- 172.21.15.69:0/1384456324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805b8d0 msgr2=0x564f28056090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51e9400640 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805b8d0 0x564f28056090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.517+0000 7f51e9400640 1 -- 172.21.15.69:0/1384456324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f28074f20 con 0x564f2805d7d0 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.518+0000 7f51e9400640 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f2802f6e0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f51d402a730 tx=0x7f51d4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96be763c09914783 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.518+0000 7f51d3400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.518+0000 7f51d3400640 1 -- 172.21.15.69:0/1384456324 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4036040 con 0x564f2805d7d0 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.518+0000 7f51d3400640 1 -- 172.21.15.69:0/1384456324 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f51d4004450 con 0x564f2805d7d0 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.518+0000 7f51f3b65740 1 -- 172.21.15.69:0/1384456324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51e4058cb0 con 0x564f2805d7d0 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.518+0000 7f51d3400640 1 -- 172.21.15.69:0/1384456324 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f51d4057050 con 0x564f2805d7d0 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.518+0000 7f51f3b65740 10 client.8241 fetch_fsmap learned FSMap version 74 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.518+0000 7f51f3b65740 1 -- 172.21.15.69:0/1384456324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x564f28030220 con 0x564f2805d7d0 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.519+0000 7f51d3400640 1 -- 172.21.15.69:0/1384456324 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e4058cb0 con 0x564f2805d7d0 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.519+0000 7f51d3400640 1 -- 172.21.15.69:0/1384456324 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f51d405a020 con 0x564f2805d7d0 2024-04-24T08:17:15.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.519+0000 7f51f3b65740 10 client.8241 fetch_fsmap finished waiting for FSMap version 74 2024-04-24T08:17:15.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.519+0000 7f51f3b65740 10 client.8241 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:17:15.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.519+0000 7f51f3b65740 20 client.8241 populate_metadata read hostname 'smithi069' 2024-04-24T08:17:15.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.519+0000 7f51f3b65740 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x564f28084a50 0x564f28057120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.519+0000 7f51f3b65740 4 client.8241 mds_command: new command op to 8151 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:17:15.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.519+0000 7f51f3b65740 1 -- 172.21.15.69:0/1384456324 --> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] -- command(tid 0: {"prefix": "status"}) -- 0x564f2802c800 con 0x564f28084a50 2024-04-24T08:17:15.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.519+0000 7f51e9e00640 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x564f28084a50 0x564f28057120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51d3400640 10 client.8241 ms_handle_connect on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51e9e00640 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x564f28084a50 0x564f28057120 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88f44764172726ee server_cookie=3d71a705eb531c0f in_seq=0 out_seq=0 2024-04-24T08:17:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51d3400640 1 -- 172.21.15.69:0/1384456324 <== mds.0 v2:172.21.15.69:6832/4254348458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x564f2802c800 con 0x564f28084a50 2024-04-24T08:17:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51d3400640 10 client.8241 handle_command_reply: tid=0 2024-04-24T08:17:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 client.8241 shutdown 2024-04-24T08:17:15.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 -- 172.21.15.69:0/1384456324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 msgr2=0x564f2802f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f2802f6e0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f51d402a730 tx=0x7f51d4004ab0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 -- 172.21.15.69:0/1384456324 shutdown_connections 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x564f28084a50 0x564f28057120 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564f2805d7d0 0x564f2802f6e0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564f280787b0 0x564f2802fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51d3400640 0 client.8241 ms_handle_reset on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 --2- 172.21.15.69:0/1384456324 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564f2805b8d0 0x564f28056090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 -- 172.21.15.69:0/1384456324 >> 172.21.15.69:0/1384456324 conn(0x564f2805b190 msgr2=0x564f28099530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 -- 172.21.15.69:0/1384456324 shutdown_connections 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 1 -- 172.21.15.69:0/1384456324 wait complete. 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.520+0000 7f51f3b65740 20 client.8241 trim_cache size 0 max 16384 2024-04-24T08:17:15.525 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 8151, 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 4.9493709350000001, 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 74, 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 86, 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 86, 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 6.512759548 2024-04-24T08:17:15.526 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:17:15.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 -- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f51c806c920 msgr2=0x7f51c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f51c806c920 0x7f51c806ede0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00a730 tx=0x7f51dc008040 comp rx=0 tx=0).stop 2024-04-24T08:17:15.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 -- 172.21.15.69:0/3651109711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145fe0 msgr2=0x7f51ec15ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145fe0 0x7f51ec15ead0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f51e400c7f0 tx=0x7f51e400ccb0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 -- 172.21.15.69:0/3651109711 shutdown_connections 2024-04-24T08:17:15.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f51c806c920 0x7f51c806ede0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f51ec145fe0 0x7f51ec15ead0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f51ec145a90 0x7f51ec13d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 --2- 172.21.15.69:0/3651109711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f51ec0d0300 0x7f51ec1e5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 -- 172.21.15.69:0/3651109711 >> 172.21.15.69:0/3651109711 conn(0x7f51ec067d90 msgr2=0x7f51ec06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:15.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 -- 172.21.15.69:0/3651109711 shutdown_connections 2024-04-24T08:17:15.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.521+0000 7f51f0e00640 1 -- 172.21.15.69:0/3651109711 wait complete. 2024-04-24T08:17:15.539 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:17:15.719 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:17:15.727 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:17:15.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.734+0000 7fc3c1400640 1 Processor -- start 2024-04-24T08:17:15.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.734+0000 7fc3c1400640 1 -- start start 2024-04-24T08:17:15.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.735+0000 7fc3c1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc3bc15e960 0x7fc3bc162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.735+0000 7fc3c1400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3bc15ed40 con 0x7fc3bc0cfaf0 2024-04-24T08:17:15.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.735+0000 7fc3c1400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3bc0cf050 con 0x7fc3bc162980 2024-04-24T08:17:15.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.735+0000 7fc3c1400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc3bc162e20 con 0x7fc3bc15e960 2024-04-24T08:17:15.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.735+0000 7fc3bbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc3bc15e960 0x7fc3bc162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.735+0000 7fc3bbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc3bc15e960 0x7fc3bc162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46758/0 (socket says 172.21.15.69:46758) 2024-04-24T08:17:15.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.735+0000 7fc3bbe00640 1 -- 172.21.15.69:0/796653612 learned_addr learned my addr 172.21.15.69:0/796653612 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:15.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.735+0000 7fc3c0a00640 1 --1- 172.21.15.69:0/796653612 >> v1:172.21.15.69:6789/0 conn(0x7fc3bc0cfaf0 0x7fc3bc0d2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39098/0 (socket says 172.21.15.69:39098) 2024-04-24T08:17:15.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.736+0000 7fc3bbe00640 1 -- 172.21.15.69:0/796653612 >> v1:172.21.15.121:6789/0 conn(0x7fc3bc162980 legacy=0x7fc3bc06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.736+0000 7fc3bbe00640 1 -- 172.21.15.69:0/796653612 >> v1:172.21.15.69:6789/0 conn(0x7fc3bc0cfaf0 legacy=0x7fc3bc0d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.736+0000 7fc3bbe00640 1 -- 172.21.15.69:0/796653612 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc163100 con 0x7fc3bc15e960 2024-04-24T08:17:15.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.736+0000 7fc3bb400640 1 -- 172.21.15.69:0/796653612 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526110828 0 0) 0x7fc3bc15ed40 con 0x7fc3bc0cfaf0 2024-04-24T08:17:15.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.737+0000 7fc3bbe00640 1 --2- 172.21.15.69:0/796653612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 0x7fc3bc162440 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc15ef10 tx=0x7fc3ac02a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d6b1c60b0916f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:15.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.737+0000 7fc3bb400640 1 -- 172.21.15.69:0/796653612 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac037070 con 0x7fc3bc15e960 2024-04-24T08:17:15.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.737+0000 7fc3bb400640 1 -- 172.21.15.69:0/796653612 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ac032640 con 0x7fc3bc15e960 2024-04-24T08:17:15.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.737+0000 7fc3bb400640 1 -- 172.21.15.69:0/796653612 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac032a10 con 0x7fc3bc15e960 2024-04-24T08:17:15.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.738+0000 7fc3c1400640 1 -- 172.21.15.69:0/796653612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 msgr2=0x7fc3bc162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.738+0000 7fc3c1400640 1 --2- 172.21.15.69:0/796653612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 0x7fc3bc162440 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc15ef10 tx=0x7fc3ac02a2e0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.738+0000 7fc3c1400640 1 -- 172.21.15.69:0/796653612 shutdown_connections 2024-04-24T08:17:15.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.738+0000 7fc3c1400640 1 --2- 172.21.15.69:0/796653612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 0x7fc3bc162440 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.738+0000 7fc3c1400640 1 -- 172.21.15.69:0/796653612 >> 172.21.15.69:0/796653612 conn(0x7fc3bc09d900 msgr2=0x7fc3bc08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:15.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.738+0000 7fc3c1400640 1 -- 172.21.15.69:0/796653612 shutdown_connections 2024-04-24T08:17:15.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.738+0000 7fc3c1400640 1 -- 172.21.15.69:0/796653612 wait complete. 2024-04-24T08:17:15.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.740+0000 7fc3c1400640 1 Processor -- start 2024-04-24T08:17:15.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.740+0000 7fc3c1400640 1 -- start start 2024-04-24T08:17:15.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3c1400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3bc0cfaf0 0x7fc3bc156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3c1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 0x7fc3bc160a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3c1400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3bc162980 0x7fc3bc0d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3c1400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc06d6c0 con 0x7fc3bc15e960 2024-04-24T08:17:15.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3c1400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc0cf5d0 con 0x7fc3bc162980 2024-04-24T08:17:15.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3c1400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc3bc0b6170 con 0x7fc3bc0cfaf0 2024-04-24T08:17:15.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3b3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 0x7fc3bc160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3b3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 0x7fc3bc160a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46764/0 (socket says 172.21.15.69:46764) 2024-04-24T08:17:15.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3b3400640 1 -- 172.21.15.69:0/807088692 learned_addr learned my addr 172.21.15.69:0/807088692 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:15.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3c0a00640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3bc162980 0x7fc3bc0d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.741+0000 7fc3bbe00640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3bc0cfaf0 0x7fc3bc156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3b3400640 1 -- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3bc0cfaf0 msgr2=0x7fc3bc156f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3b3400640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3bc0cfaf0 0x7fc3bc156f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3b3400640 1 -- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3bc162980 msgr2=0x7fc3bc0d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:15.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3b3400640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3bc162980 0x7fc3bc0d2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:15.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3b3400640 1 -- 172.21.15.69:0/807088692 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc1e8d50 con 0x7fc3bc15e960 2024-04-24T08:17:15.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3bbe00640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3bc0cfaf0 0x7fc3bc156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:17:15.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3b3400640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 0x7fc3bc160a00 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400c730 tx=0x7fc3a400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:15.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3b8c00640 1 -- 172.21.15.69:0/807088692 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4003b70 con 0x7fc3bc15e960 2024-04-24T08:17:15.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3b8c00640 1 -- 172.21.15.69:0/807088692 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a4003d30 con 0x7fc3bc15e960 2024-04-24T08:17:15.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3b8c00640 1 -- 172.21.15.69:0/807088692 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a400aa90 con 0x7fc3bc15e960 2024-04-24T08:17:15.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.742+0000 7fc3c1400640 1 -- 172.21.15.69:0/807088692 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3bc1e9030 con 0x7fc3bc15e960 2024-04-24T08:17:15.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.743+0000 7fc3c1400640 1 -- 172.21.15.69:0/807088692 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3bc1e95f0 con 0x7fc3bc15e960 2024-04-24T08:17:15.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.744+0000 7fc3c1400640 1 -- 172.21.15.69:0/807088692 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3bc075d50 con 0x7fc3bc15e960 2024-04-24T08:17:15.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.745+0000 7fc3b8c00640 1 -- 172.21.15.69:0/807088692 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc3a400ac30 con 0x7fc3bc15e960 2024-04-24T08:17:15.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.746+0000 7fc3b8c00640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc39806c8f0 0x7fc39806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:15.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.746+0000 7fc3b8c00640 1 -- 172.21.15.69:0/807088692 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fc3a408bc30 con 0x7fc3bc15e960 2024-04-24T08:17:15.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.746+0000 7fc3bbe00640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc39806c8f0 0x7fc39806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:15.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.748+0000 7fc3bbe00640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc39806c8f0 0x7fc39806edb0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac034000 tx=0x7fc3ac02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:15.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.750+0000 7fc3b8c00640 1 -- 172.21.15.69:0/807088692 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3a4059e60 con 0x7fc3bc15e960 2024-04-24T08:17:15.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.997+0000 7fc3c1400640 1 -- 172.21.15.69:0/807088692 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc3bc0d1dc0 con 0x7fc3bc15e960 2024-04-24T08:17:15.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:15.998+0000 7fc3b8c00640 1 -- 172.21.15.69:0/807088692 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) ==== 53+0+73 (secure 0 0 0) 0x7fc3a4059760 con 0x7fc3bc15e960 2024-04-24T08:17:16.000 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:17:16.001 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:17:16.000+0000 7f5193400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:17:16.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.001+0000 7fc3c1400640 1 -- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc39806c8f0 msgr2=0x7fc39806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:16.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.001+0000 7fc3c1400640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc39806c8f0 0x7fc39806edb0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac034000 tx=0x7fc3ac02e000 comp rx=0 tx=0).stop 2024-04-24T08:17:16.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.001+0000 7fc3c1400640 1 -- 172.21.15.69:0/807088692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 msgr2=0x7fc3bc160a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:16.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.001+0000 7fc3c1400640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 0x7fc3bc160a00 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400c730 tx=0x7fc3a400cbf0 comp rx=0 tx=0).stop 2024-04-24T08:17:16.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.002+0000 7fc3c1400640 1 -- 172.21.15.69:0/807088692 shutdown_connections 2024-04-24T08:17:16.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.002+0000 7fc3c1400640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc39806c8f0 0x7fc39806edb0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:16.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.002+0000 7fc3c1400640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3bc15e960 0x7fc3bc160a00 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:16.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.002+0000 7fc3c1400640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3bc162980 0x7fc3bc0d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:16.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.002+0000 7fc3c1400640 1 --2- 172.21.15.69:0/807088692 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3bc0cfaf0 0x7fc3bc156f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:16.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.002+0000 7fc3c1400640 1 -- 172.21.15.69:0/807088692 >> 172.21.15.69:0/807088692 conn(0x7fc3bc09d900 msgr2=0x7fc3bc0b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:16.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.002+0000 7fc3c1400640 1 -- 172.21.15.69:0/807088692 shutdown_connections 2024-04-24T08:17:16.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16.002+0000 7fc3c1400640 1 -- 172.21.15.69:0/807088692 wait complete. 2024-04-24T08:17:16.022 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:17:16.022 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:17:16.022 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:17:16.022 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:17:16.023 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:17:16.023 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:17:16.023 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:17:16.023 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:17:16.023 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:17:16.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:16.023 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:17:16.067 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:17:16.068 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42144sec preferred_lft 42144sec 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:17:16.069 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:17:16.070 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:17:16.070 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:17:16.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:16.071 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:17:16.190 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:17:16.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:16.194 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:17:16.241 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:17:16.241 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:17:16.241 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:17:16.241 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:17:16.241 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:17:16.241 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:17:16.241 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:17:16.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:17:16.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:17:16.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:17:16.242 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:17:16.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:17:16.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:17:16.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:16.500 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:17:16.550 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:17:16.550 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:17:16.551 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:17:16.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:16.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:17:16.635 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:17:16.635 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:17:16.635 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:17:16.686 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:17:16.686 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:17:16.686 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:17:16.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:17:16.819 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:17:16.873 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:17:16.873 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:17:16.874 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:17:16.875 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:17:16.876 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:17:16.876 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:17:16.876 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:17:16.876 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:17:16.876 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:17:16.876 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:17:16.876 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:17:16.876 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:17:16.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:16.877 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:17:16.924 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:17:16.924 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:17:16.929 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:17:16.993 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:17:17.033 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:17:17.034 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:17:17.035 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:17:17.035 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:17:17.035 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:17:17.035 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:17:17.035 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:17:17.035 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:17:17.035 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:17:17.035 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:17:17.035 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:17:17.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:17.036 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:17:17.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:17:17.092+0000 7f1aa8b17480 -1 init, newargv = 0x55d7de5dfdc0 newargc=15 2024-04-24T08:17:17.094 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[66883]: starting ceph client 2024-04-24T08:17:17.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[66883]: starting fuse 2024-04-24T08:17:17.121 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:17:17.121 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:17:17.121 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:17:17.121 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:17:17.121 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:17:17.121 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:17:17.121 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:17:17.121 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:17:17.122 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:17:17.123 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:17:17.123 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:17:17.123 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:17:17.123 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:17:17.123 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:17:17.123 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:17:17.307 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.66883.asok 2024-04-24T08:17:17.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:17:17.323 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.66883.asok 2024-04-24T08:17:17.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:17.324 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.66883.asok status 2024-04-24T08:17:17.550 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:17:17.550 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:17:17.550 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:17:17.550 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "66883", 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 8259, 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:17:17.551 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 8259 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2744475957 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:17:17.552 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2744475957 2024-04-24T08:17:17.553 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:17:17.553 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.8259 192.168.0.1:0/2744475957", 2024-04-24T08:17:17.553 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2744475957", 2024-04-24T08:17:17.553 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:17:17.553 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 74, 2024-04-24T08:17:17.553 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 86, 2024-04-24T08:17:17.553 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:17:17.553 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:17:17.553 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:17:17.553 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:17:17.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:17.565 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:17:17.565 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:17:17.615 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:17:17.616 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:17:17.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:17.616 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:17:17.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:17.692 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-04-24T08:17:17.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.894+0000 7fc92be00640 1 Processor -- start 2024-04-24T08:17:17.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.894+0000 7fc92be00640 1 -- start start 2024-04-24T08:17:17.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.895+0000 7fc92be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc92c073b40 0x7fc92c071b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.895+0000 7fc92be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c072050 0x7fc92c0ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.895+0000 7fc92be00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc92c0724e0 con 0x7fc92c15c5f0 2024-04-24T08:17:17.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.895+0000 7fc92be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc92c0acf30 con 0x7fc92c073b40 2024-04-24T08:17:17.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.895+0000 7fc92be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc92c15c9d0 con 0x7fc92c072050 2024-04-24T08:17:17.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.896+0000 7fc92a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc92c073b40 0x7fc92c071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.896+0000 7fc92a000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc92c073b40 0x7fc92c071b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46770/0 (socket says 172.21.15.69:46770) 2024-04-24T08:17:17.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.896+0000 7fc92a000640 1 -- 172.21.15.69:0/570806728 learned_addr learned my addr 172.21.15.69:0/570806728 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:17.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.896+0000 7fc92b400640 1 --2- 172.21.15.69:0/570806728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c072050 0x7fc92c0ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.896+0000 7fc92a000640 1 -- 172.21.15.69:0/570806728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c072050 msgr2=0x7fc92c0ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:17.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.896+0000 7fc92a000640 1 --2- 172.21.15.69:0/570806728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c072050 0x7fc92c0ad6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.896+0000 7fc92a000640 1 -- 172.21.15.69:0/570806728 >> v1:172.21.15.121:6789/0 conn(0x7fc92c15c5f0 legacy=0x7fc92c0ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:17.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.896+0000 7fc92a000640 1 -- 172.21.15.69:0/570806728 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c0ce8c0 con 0x7fc92c073b40 2024-04-24T08:17:17.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.896+0000 7fc92b400640 1 --2- 172.21.15.69:0/570806728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c072050 0x7fc92c0ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:17.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.898+0000 7fc92a000640 1 --2- 172.21.15.69:0/570806728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c073b40 0x7fc92c071b10 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fc9180098a0 tx=0x7fc91802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dedd53662dab0439 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:17.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.898+0000 7fc929600640 1 -- 172.21.15.69:0/570806728 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918037070 con 0x7fc92c073b40 2024-04-24T08:17:17.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.898+0000 7fc929600640 1 -- 172.21.15.69:0/570806728 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc918032440 con 0x7fc92c073b40 2024-04-24T08:17:17.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.898+0000 7fc929600640 1 -- 172.21.15.69:0/570806728 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc918032810 con 0x7fc92c073b40 2024-04-24T08:17:17.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.899+0000 7fc92be00640 1 -- 172.21.15.69:0/570806728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c073b40 msgr2=0x7fc92c071b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:17.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.899+0000 7fc92be00640 1 --2- 172.21.15.69:0/570806728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c073b40 0x7fc92c071b10 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fc9180098a0 tx=0x7fc91802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.899+0000 7fc92be00640 1 -- 172.21.15.69:0/570806728 shutdown_connections 2024-04-24T08:17:17.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.899+0000 7fc92be00640 1 --2- 172.21.15.69:0/570806728 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c072050 0x7fc92c0ad6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.899+0000 7fc92be00640 1 --2- 172.21.15.69:0/570806728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c073b40 0x7fc92c071b10 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.899+0000 7fc92be00640 1 -- 172.21.15.69:0/570806728 >> 172.21.15.69:0/570806728 conn(0x7fc92c0b1c50 msgr2=0x7fc92c0b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:17.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.899+0000 7fc92be00640 1 -- 172.21.15.69:0/570806728 shutdown_connections 2024-04-24T08:17:17.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.899+0000 7fc92be00640 1 -- 172.21.15.69:0/570806728 wait complete. 2024-04-24T08:17:17.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.901+0000 7fc92be00640 1 Processor -- start 2024-04-24T08:17:17.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.901+0000 7fc92be00640 1 -- start start 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc92c072050 0x7fc92c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c073b40 0x7fc92c15cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c15c5f0 0x7fc92c14cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c0ad180 con 0x7fc92c15c5f0 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c0cee80 con 0x7fc92c072050 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc92c0adbe0 con 0x7fc92c073b40 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c15c5f0 0x7fc92c14cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c15c5f0 0x7fc92c14cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46774/0 (socket says 172.21.15.69:46774) 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92b400640 1 -- 172.21.15.69:0/92466750 learned_addr learned my addr 172.21.15.69:0/92466750 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92a000640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c073b40 0x7fc92c15cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92aa00640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc92c072050 0x7fc92c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92b400640 1 -- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c073b40 msgr2=0x7fc92c15cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:17.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92b400640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c073b40 0x7fc92c15cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92b400640 1 -- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc92c072050 msgr2=0x7fc92c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:17.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92b400640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc92c072050 0x7fc92c15e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.902+0000 7fc92b400640 1 -- 172.21.15.69:0/92466750 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92c15eeb0 con 0x7fc92c15c5f0 2024-04-24T08:17:17.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.903+0000 7fc92b400640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c15c5f0 0x7fc92c14cca0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc92400cef0 tx=0x7fc92400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:17.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.903+0000 7fc922a00640 1 -- 172.21.15.69:0/92466750 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92401a070 con 0x7fc92c15c5f0 2024-04-24T08:17:17.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.903+0000 7fc922a00640 1 -- 172.21.15.69:0/92466750 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc92400d860 con 0x7fc92c15c5f0 2024-04-24T08:17:17.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.903+0000 7fc922a00640 1 -- 172.21.15.69:0/92466750 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92400dc50 con 0x7fc92c15c5f0 2024-04-24T08:17:17.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.903+0000 7fc92be00640 1 -- 172.21.15.69:0/92466750 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc92c15f440 con 0x7fc92c15c5f0 2024-04-24T08:17:17.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.903+0000 7fc92be00640 1 -- 172.21.15.69:0/92466750 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92c15fa00 con 0x7fc92c15c5f0 2024-04-24T08:17:17.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.906+0000 7fc922a00640 1 -- 172.21.15.69:0/92466750 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc92401d070 con 0x7fc92c15c5f0 2024-04-24T08:17:17.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.907+0000 7fc922a00640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90c06c920 0x7fc90c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.907+0000 7fc922a00640 1 -- 172.21.15.69:0/92466750 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fc92408c460 con 0x7fc92c15c5f0 2024-04-24T08:17:17.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.908+0000 7fc92aa00640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90c06c920 0x7fc90c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.908+0000 7fc934a15740 1 Processor -- start 2024-04-24T08:17:17.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.908+0000 7fc934a15740 1 -- start start 2024-04-24T08:17:17.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.909+0000 7fc934a15740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781ca1640 0x555781c53fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.909+0000 7fc934a15740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c84760 0x555781c83ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.909+0000 7fc934a15740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c84020 0x555781c58e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.909+0000 7fc934a15740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555781bbde70 con 0x555781ca1640 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.909+0000 7fc934a15740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555781b64380 con 0x555781c84020 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.909+0000 7fc934a15740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x555781bb9e30 con 0x555781c84760 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.909+0000 7fc92aa00640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90c06c920 0x7fc90c06ede0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc91c006fd0 tx=0x7fc91c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.909+0000 7fc92aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781ca1640 0x555781c53fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.909+0000 7fc92aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781ca1640 0x555781c53fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46776/0 (socket says 172.21.15.69:46776) 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.909+0000 7fc92aa00640 1 -- 172.21.15.69:0/1422520406 learned_addr learned my addr 172.21.15.69:0/1422520406 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc92a000640 1 --2- 172.21.15.69:0/1422520406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c84760 0x555781c83ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc92aa00640 1 -- 172.21.15.69:0/1422520406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c84760 msgr2=0x555781c83ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc92aa00640 1 --2- 172.21.15.69:0/1422520406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c84760 0x555781c83ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc92b400640 1 --2- 172.21.15.69:0/1422520406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c84020 0x555781c58e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc92aa00640 1 -- 172.21.15.69:0/1422520406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c84020 msgr2=0x555781c58e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc92aa00640 1 --2- 172.21.15.69:0/1422520406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c84020 0x555781c58e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc92aa00640 1 -- 172.21.15.69:0/1422520406 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557819987e0 con 0x555781ca1640 2024-04-24T08:17:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc92a000640 1 --2- 172.21.15.69:0/1422520406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c84760 0x555781c83ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc92aa00640 1 --2- 172.21.15.69:0/1422520406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781ca1640 0x555781c53fd0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00d8c0 tx=0x7fc91c010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af59c1d19e3ad57 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc917400640 1 -- 172.21.15.69:0/1422520406 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c024070 con 0x555781ca1640 2024-04-24T08:17:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc917400640 1 -- 172.21.15.69:0/1422520406 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc91c00de40 con 0x555781ca1640 2024-04-24T08:17:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.910+0000 7fc917400640 1 -- 172.21.15.69:0/1422520406 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c014de0 con 0x555781ca1640 2024-04-24T08:17:17.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.911+0000 7fc934a15740 1 -- 172.21.15.69:0/1422520406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781ca1640 msgr2=0x555781c53fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:17.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.911+0000 7fc934a15740 1 --2- 172.21.15.69:0/1422520406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781ca1640 0x555781c53fd0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00d8c0 tx=0x7fc91c010000 comp rx=0 tx=0).stop 2024-04-24T08:17:17.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.911+0000 7fc934a15740 1 -- 172.21.15.69:0/1422520406 shutdown_connections 2024-04-24T08:17:17.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.911+0000 7fc934a15740 1 --2- 172.21.15.69:0/1422520406 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c84020 0x555781c58e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.911+0000 7fc934a15740 1 --2- 172.21.15.69:0/1422520406 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c84760 0x555781c83ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.911+0000 7fc934a15740 1 --2- 172.21.15.69:0/1422520406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781ca1640 0x555781c53fd0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.911+0000 7fc934a15740 1 -- 172.21.15.69:0/1422520406 >> 172.21.15.69:0/1422520406 conn(0x555781c56240 msgr2=0x555781c9eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:17.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.911+0000 7fc934a15740 1 -- 172.21.15.69:0/1422520406 shutdown_connections 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.911+0000 7fc934a15740 1 -- 172.21.15.69:0/1422520406 wait complete. 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.912+0000 7fc934a15740 1 -- start start 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.912+0000 7fc934a15740 1 Processor -- start 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.913+0000 7fc934a15740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c84760 0x555781c71410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.913+0000 7fc934a15740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781ca1640 0x555781c71b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.913+0000 7fc934a15740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c95870 0x555781c598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.913+0000 7fc934a15740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555781c583f0 con 0x555781c95870 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.913+0000 7fc934a15740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555781b77660 con 0x555781c84760 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.913+0000 7fc934a15740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x555781c577e0 con 0x555781ca1640 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.913+0000 7fc934a15740 10 client.0 _ll_register_callbacks cb 0x555781b65aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.913+0000 7fc92b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c95870 0x555781c598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.913+0000 7fc92b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c95870 0x555781c598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46786/0 (socket says 172.21.15.69:46786) 2024-04-24T08:17:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.913+0000 7fc92b400640 1 -- 172.21.15.69:0/2562868909 learned_addr learned my addr 172.21.15.69:0/2562868909 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc92aa00640 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c84760 0x555781c71410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc92b400640 1 -- 172.21.15.69:0/2562868909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781ca1640 msgr2=0x555781c71b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc92b400640 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781ca1640 0x555781c71b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc92b400640 1 -- 172.21.15.69:0/2562868909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c84760 msgr2=0x555781c71410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc92b400640 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c84760 0x555781c71410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc92b400640 1 -- 172.21.15.69:0/2562868909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557819f8f70 con 0x555781c95870 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc92b400640 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c95870 0x555781c598d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fc924062760 tx=0x7fc92405a170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98a2203b155dea29 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc90be00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc90be00640 1 -- 172.21.15.69:0/2562868909 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc924062c90 con 0x555781c95870 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc90be00640 1 -- 172.21.15.69:0/2562868909 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fc92405e0d0 con 0x555781c95870 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc934a15740 1 -- 172.21.15.69:0/2562868909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555781bcb890 con 0x555781c95870 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc90be00640 1 -- 172.21.15.69:0/2562868909 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7fc92408c070 con 0x555781c95870 2024-04-24T08:17:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc934a15740 10 client.8271 fetch_fsmap learned FSMap version 74 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.914+0000 7fc934a15740 1 -- 172.21.15.69:0/2562868909 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x555781890170 con 0x555781c95870 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.915+0000 7fc90be00640 1 -- 172.21.15.69:0/2562868909 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x555781bcb890 con 0x555781c95870 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.915+0000 7fc90be00640 1 -- 172.21.15.69:0/2562868909 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7fc924062800 con 0x555781c95870 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.915+0000 7fc934a15740 10 client.8271 fetch_fsmap finished waiting for FSMap version 74 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.915+0000 7fc934a15740 10 client.8271 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.915+0000 7fc934a15740 20 client.8271 populate_metadata read hostname 'smithi069' 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.915+0000 7fc934a15740 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x555781c87700 0x555781c84c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.915+0000 7fc934a15740 4 client.8271 mds_command: new command op to 8151 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.915+0000 7fc934a15740 1 -- 172.21.15.69:0/2562868909 --> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555781bc5700 con 0x555781c87700 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.915+0000 7fc92aa00640 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x555781c87700 0x555781c84c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.916+0000 7fc92aa00640 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x555781c87700 0x555781c84c50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa6cb15ae22f914f server_cookie=b8b7b5b3e7186cb2 in_seq=0 out_seq=0 2024-04-24T08:17:17.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.916+0000 7fc90be00640 10 client.8271 ms_handle_connect on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc90be00640 1 -- 172.21.15.69:0/2562868909 <== mds.0 v2:172.21.15.69:6832/4254348458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555781bc5700 con 0x555781c87700 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc90be00640 10 client.8271 handle_command_reply: tid=0 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 client.8271 shutdown 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 -- 172.21.15.69:0/2562868909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c95870 msgr2=0x555781c598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c95870 0x555781c598d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fc924062760 tx=0x7fc92405a170 comp rx=0 tx=0).stop 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 -- 172.21.15.69:0/2562868909 shutdown_connections 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x555781c87700 0x555781c84c50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c95870 0x555781c598d0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781ca1640 0x555781c71b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 --2- 172.21.15.69:0/2562868909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c84760 0x555781c71410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc90be00640 0 client.8271 ms_handle_reset on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:17.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 -- 172.21.15.69:0/2562868909 >> 172.21.15.69:0/2562868909 conn(0x555781c84020 msgr2=0x555781c9eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:17.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 -- 172.21.15.69:0/2562868909 shutdown_connections 2024-04-24T08:17:17.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 1 -- 172.21.15.69:0/2562868909 wait complete. 2024-04-24T08:17:17.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.917+0000 7fc934a15740 20 client.8271 trim_cache size 0 max 16384 2024-04-24T08:17:17.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.936+0000 7fc934a15740 1 Processor -- start 2024-04-24T08:17:17.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.937+0000 7fc934a15740 1 -- start start 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.937+0000 7fc934a15740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781ca26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.937+0000 7fc934a15740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c95870 0x555781c9c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.937+0000 7fc934a15740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781ca1640 0x555781c9cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.937+0000 7fc934a15740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c200eb0 con 0x555781c84760 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.937+0000 7fc934a15740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c1fffc0 con 0x555781ca1640 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.937+0000 7fc934a15740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc92c14e100 con 0x555781c95870 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781ca26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781ca26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46798/0 (socket says 172.21.15.69:46798) 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92aa00640 1 -- 172.21.15.69:0/418644124 learned_addr learned my addr 172.21.15.69:0/418644124 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92a000640 1 --2- 172.21.15.69:0/418644124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c95870 0x555781c9c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92b400640 1 --2- 172.21.15.69:0/418644124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781ca1640 0x555781c9cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92aa00640 1 -- 172.21.15.69:0/418644124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c95870 msgr2=0x555781c9c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:17.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92aa00640 1 --2- 172.21.15.69:0/418644124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c95870 0x555781c9c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92aa00640 1 -- 172.21.15.69:0/418644124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781ca1640 msgr2=0x555781c9cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:17.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92aa00640 1 --2- 172.21.15.69:0/418644124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781ca1640 0x555781c9cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92aa00640 1 -- 172.21.15.69:0/418644124 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555781c53c70 con 0x555781c84760 2024-04-24T08:17:17.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc92aa00640 1 --2- 172.21.15.69:0/418644124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781ca26d0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00d890 tx=0x7fc91c01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25805f6787ee342f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:17.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.938+0000 7fc916a00640 1 -- 172.21.15.69:0/418644124 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c018070 con 0x555781c84760 2024-04-24T08:17:17.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc916a00640 1 -- 172.21.15.69:0/418644124 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc91c02fe50 con 0x555781c84760 2024-04-24T08:17:17.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc916a00640 1 -- 172.21.15.69:0/418644124 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c024040 con 0x555781c84760 2024-04-24T08:17:17.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc934a15740 1 -- 172.21.15.69:0/418644124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 msgr2=0x555781ca26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:17.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc934a15740 1 --2- 172.21.15.69:0/418644124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781ca26d0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fc91c00d890 tx=0x7fc91c01c000 comp rx=0 tx=0).stop 2024-04-24T08:17:17.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc934a15740 1 -- 172.21.15.69:0/418644124 shutdown_connections 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc934a15740 1 --2- 172.21.15.69:0/418644124 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781ca1640 0x555781c9cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc934a15740 1 --2- 172.21.15.69:0/418644124 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781c95870 0x555781c9c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc934a15740 1 --2- 172.21.15.69:0/418644124 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781ca26d0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc934a15740 1 -- 172.21.15.69:0/418644124 >> 172.21.15.69:0/418644124 conn(0x555781c84020 msgr2=0x555781c97b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc934a15740 1 -- 172.21.15.69:0/418644124 shutdown_connections 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.939+0000 7fc934a15740 1 -- 172.21.15.69:0/418644124 wait complete. 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.940+0000 7fc934a15740 1 -- start start 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.940+0000 7fc934a15740 1 Processor -- start 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.941+0000 7fc934a15740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781d04e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.941+0000 7fc934a15740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c95870 0x555781c94e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.941+0000 7fc934a15740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781ca1640 0x555781d05500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.941+0000 7fc934a15740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555781c815b0 con 0x555781c84760 2024-04-24T08:17:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.941+0000 7fc934a15740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc92c14e750 con 0x555781c95870 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.941+0000 7fc934a15740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x555781c53e40 con 0x555781ca1640 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.941+0000 7fc934a15740 10 client.0 _ll_register_callbacks cb 0x7fc924007a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.941+0000 7fc92aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781d04e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.941+0000 7fc92aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781d04e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46806/0 (socket says 172.21.15.69:46806) 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.941+0000 7fc92aa00640 1 -- 172.21.15.69:0/3998103554 learned_addr learned my addr 172.21.15.69:0/3998103554 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc92a000640 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c95870 0x555781c94e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc92aa00640 1 -- 172.21.15.69:0/3998103554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781ca1640 msgr2=0x555781d05500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc92aa00640 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781ca1640 0x555781d05500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc92aa00640 1 -- 172.21.15.69:0/3998103554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c95870 msgr2=0x555781c94e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc92aa00640 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c95870 0x555781c94e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc92aa00640 1 -- 172.21.15.69:0/3998103554 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555781c70030 con 0x555781c84760 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc92aa00640 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781d04e70 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc91c02e040 tx=0x7fc91c004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0329d9a154ac290 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc90be00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc90be00640 1 -- 172.21.15.69:0/3998103554 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc91c018070 con 0x555781c84760 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc90be00640 1 -- 172.21.15.69:0/3998103554 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fc91c004240 con 0x555781c84760 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc934a15740 1 -- 172.21.15.69:0/3998103554 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc91c02fe50 con 0x555781c84760 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.942+0000 7fc90be00640 1 -- 172.21.15.69:0/3998103554 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7fc91c03d050 con 0x555781c84760 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc934a15740 10 client.8277 fetch_fsmap learned FSMap version 74 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc934a15740 1 -- 172.21.15.69:0/3998103554 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x555781d05a40 con 0x555781c84760 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc90be00640 1 -- 172.21.15.69:0/3998103554 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc91c02fe50 con 0x555781c84760 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc90be00640 1 -- 172.21.15.69:0/3998103554 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7fc91c040020 con 0x555781c84760 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc934a15740 10 client.8277 fetch_fsmap finished waiting for FSMap version 74 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc934a15740 10 client.8277 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc934a15740 20 client.8277 populate_metadata read hostname 'smithi069' 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc934a15740 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x555781c6f900 0x555781cc1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc934a15740 4 client.8277 mds_command: new command op to 8151 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-04-24T08:17:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc934a15740 1 -- 172.21.15.69:0/3998103554 --> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) -- 0x555781b41980 con 0x555781c6f900 2024-04-24T08:17:17.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.943+0000 7fc92a000640 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x555781c6f900 0x555781cc1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:17.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.944+0000 7fc90be00640 10 client.8277 ms_handle_connect on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:17.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.944+0000 7fc92a000640 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x555781c6f900 0x555781cc1e70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37e63c10794a8ae8 server_cookie=33eb81578543e259 in_seq=0 out_seq=0 2024-04-24T08:17:17.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.944+0000 7fc90be00640 1 -- 172.21.15.69:0/3998103554 <== mds.0 v2:172.21.15.69:6832/4254348458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555781b41980 con 0x555781c6f900 2024-04-24T08:17:17.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.944+0000 7fc90be00640 10 client.8277 handle_command_reply: tid=0 2024-04-24T08:17:17.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.944+0000 7fc934a15740 1 client.8277 shutdown 2024-04-24T08:17:17.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 1 -- 172.21.15.69:0/3998103554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 msgr2=0x555781d04e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:17.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781d04e70 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fc91c02e040 tx=0x7fc91c004680 comp rx=0 tx=0).stop 2024-04-24T08:17:17.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 1 -- 172.21.15.69:0/3998103554 shutdown_connections 2024-04-24T08:17:17.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x555781c6f900 0x555781cc1e70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555781ca1640 0x555781d05500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555781c95870 0x555781c94e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 1 --2- 172.21.15.69:0/3998103554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555781c84760 0x555781d04e70 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc90be00640 0 client.8277 ms_handle_reset on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:17.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 1 -- 172.21.15.69:0/3998103554 >> 172.21.15.69:0/3998103554 conn(0x555781c84020 msgr2=0x555781c53fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:17.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 1 -- 172.21.15.69:0/3998103554 shutdown_connections 2024-04-24T08:17:17.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 1 -- 172.21.15.69:0/3998103554 wait complete. 2024-04-24T08:17:17.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.945+0000 7fc934a15740 20 client.8277 trim_cache size 0 max 16384 2024-04-24T08:17:17.950 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:17:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-04-24T08:17:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "dc351a0e-20c3-4182-a654-98f0fc7ad7ec", 2024-04-24T08:17:17.951 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-04-24T08:17:17.951 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 -- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90c06c920 msgr2=0x7fc90c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90c06c920 0x7fc90c06ede0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc91c006fd0 tx=0x7fc91c008040 comp rx=0 tx=0).stop 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 -- 172.21.15.69:0/92466750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c15c5f0 msgr2=0x7fc92c14cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c15c5f0 0x7fc92c14cca0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7fc92400cef0 tx=0x7fc92400a710 comp rx=0 tx=0).stop 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 -- 172.21.15.69:0/92466750 shutdown_connections 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc90c06c920 0x7fc90c06ede0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc92c15c5f0 0x7fc92c14cca0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc92c073b40 0x7fc92c15cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 --2- 172.21.15.69:0/92466750 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc92c072050 0x7fc92c15e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 -- 172.21.15.69:0/92466750 >> 172.21.15.69:0/92466750 conn(0x7fc92c0b1c50 msgr2=0x7fc92c074b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 -- 172.21.15.69:0/92466750 shutdown_connections 2024-04-24T08:17:17.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:17.946+0000 7fc92be00640 1 -- 172.21.15.69:0/92466750 wait complete. 2024-04-24T08:17:17.953 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:17:17.947+0000 7f518e400640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() | dirtyparent=1 authpin=1 scrubqueue=0 0x56135fd28680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(13)0x10000000001:[<0x10000000000/file v6>,<0x1/dir v13>]//[]","memoryvalue":"(13)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-24T08:17:17.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:17.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-24T08:17:18.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.160+0000 7ffb3b400640 1 Processor -- start 2024-04-24T08:17:18.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.160+0000 7ffb3b400640 1 -- start start 2024-04-24T08:17:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb3b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffb34073db0 0x7ffb340b3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb3b400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ffb340b4e90 0x7ffb340cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb3b400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb341631c0 con 0x7ffb340cccc0 2024-04-24T08:17:18.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb3b400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ffb340cd690 con 0x7ffb34073db0 2024-04-24T08:17:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb3b400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ffb340cf4d0 con 0x7ffb340b4e90 2024-04-24T08:17:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb38a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffb34073db0 0x7ffb340b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb38a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ffb34073db0 0x7ffb340b3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46816/0 (socket says 172.21.15.69:46816) 2024-04-24T08:17:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb38a00640 1 -- 172.21.15.69:0/3844142434 learned_addr learned my addr 172.21.15.69:0/3844142434 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb38a00640 1 -- 172.21.15.69:0/3844142434 >> v2:172.21.15.121:3300/0 conn(0x7ffb340b4e90 msgr2=0x7ffb340cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb38a00640 1 --2- 172.21.15.69:0/3844142434 >> v2:172.21.15.121:3300/0 conn(0x7ffb340b4e90 0x7ffb340cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb38a00640 1 -- 172.21.15.69:0/3844142434 >> v1:172.21.15.121:6789/0 conn(0x7ffb340cccc0 legacy=0x7ffb340b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.162+0000 7ffb38a00640 1 -- 172.21.15.69:0/3844142434 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb340cf7b0 con 0x7ffb34073db0 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.163+0000 7ffb38a00640 1 --2- 172.21.15.69:0/3844142434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb34073db0 0x7ffb340b3260 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7ffb340cdd90 tx=0x7ffb2802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12d881bfbcfb167c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.164+0000 7ffb33400640 1 -- 172.21.15.69:0/3844142434 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb28037070 con 0x7ffb34073db0 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.164+0000 7ffb33400640 1 -- 172.21.15.69:0/3844142434 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb28032440 con 0x7ffb34073db0 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.164+0000 7ffb33400640 1 -- 172.21.15.69:0/3844142434 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb28032810 con 0x7ffb34073db0 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.164+0000 7ffb3b400640 1 -- 172.21.15.69:0/3844142434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb34073db0 msgr2=0x7ffb340b3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.164+0000 7ffb3b400640 1 --2- 172.21.15.69:0/3844142434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb34073db0 0x7ffb340b3260 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7ffb340cdd90 tx=0x7ffb2802a230 comp rx=0 tx=0).stop 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.165+0000 7ffb3b400640 1 -- 172.21.15.69:0/3844142434 shutdown_connections 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.165+0000 7ffb3b400640 1 --2- 172.21.15.69:0/3844142434 >> v2:172.21.15.121:3300/0 conn(0x7ffb340b4e90 0x7ffb340cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.165+0000 7ffb3b400640 1 --2- 172.21.15.69:0/3844142434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb34073db0 0x7ffb340b3260 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.165+0000 7ffb3b400640 1 -- 172.21.15.69:0/3844142434 >> 172.21.15.69:0/3844142434 conn(0x7ffb340d0350 msgr2=0x7ffb340d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.165+0000 7ffb3b400640 1 -- 172.21.15.69:0/3844142434 shutdown_connections 2024-04-24T08:17:18.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.165+0000 7ffb3b400640 1 -- 172.21.15.69:0/3844142434 wait complete. 2024-04-24T08:17:18.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.166+0000 7ffb3b400640 1 Processor -- start 2024-04-24T08:17:18.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.167+0000 7ffb3b400640 1 -- start start 2024-04-24T08:17:18.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.167+0000 7ffb3b400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb34073db0 0x7ffb340aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.167+0000 7ffb3b400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffb340b4e90 0x7ffb34146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.167+0000 7ffb3b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb340cccc0 0x7ffb34165820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.167+0000 7ffb3b400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffb340d0050 con 0x7ffb340cccc0 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.167+0000 7ffb3b400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffb340cfd70 con 0x7ffb34073db0 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.167+0000 7ffb3b400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffb340ce470 con 0x7ffb340b4e90 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.168+0000 7ffb39400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb340cccc0 0x7ffb34165820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.168+0000 7ffb39400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb340cccc0 0x7ffb34165820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46824/0 (socket says 172.21.15.69:46824) 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.168+0000 7ffb39400640 1 -- 172.21.15.69:0/2350443863 learned_addr learned my addr 172.21.15.69:0/2350443863 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.168+0000 7ffb33e00640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffb340b4e90 0x7ffb34146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.168+0000 7ffb38a00640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb34073db0 0x7ffb340aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.168+0000 7ffb39400640 1 -- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffb340b4e90 msgr2=0x7ffb34146ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.168+0000 7ffb39400640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffb340b4e90 0x7ffb34146ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.168+0000 7ffb39400640 1 -- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb34073db0 msgr2=0x7ffb340aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.168+0000 7ffb39400640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb34073db0 0x7ffb340aed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.168+0000 7ffb39400640 1 -- 172.21.15.69:0/2350443863 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb341477e0 con 0x7ffb340cccc0 2024-04-24T08:17:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.169+0000 7ffb39400640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb340cccc0 0x7ffb34165820 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c00b9a0 tx=0x7ffb2c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:18.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.169+0000 7ffb30c00640 1 -- 172.21.15.69:0/2350443863 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c019070 con 0x7ffb340cccc0 2024-04-24T08:17:18.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.169+0000 7ffb30c00640 1 -- 172.21.15.69:0/2350443863 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb2c00cbc0 con 0x7ffb340cccc0 2024-04-24T08:17:18.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.169+0000 7ffb30c00640 1 -- 172.21.15.69:0/2350443863 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb2c014440 con 0x7ffb340cccc0 2024-04-24T08:17:18.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.169+0000 7ffb3b400640 1 -- 172.21.15.69:0/2350443863 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb34147ac0 con 0x7ffb340cccc0 2024-04-24T08:17:18.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.169+0000 7ffb3b400640 1 -- 172.21.15.69:0/2350443863 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb34163470 con 0x7ffb340cccc0 2024-04-24T08:17:18.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.171+0000 7ffb30c00640 1 -- 172.21.15.69:0/2350443863 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffb2c01c070 con 0x7ffb340cccc0 2024-04-24T08:17:18.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.172+0000 7ffb30c00640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffb1406c8f0 0x7ffb1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.172+0000 7ffb30c00640 1 -- 172.21.15.69:0/2350443863 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7ffb2c08ba70 con 0x7ffb340cccc0 2024-04-24T08:17:18.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.173+0000 7ffb38a00640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffb1406c8f0 0x7ffb1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.173+0000 7ffb3e1ba740 1 Processor -- start 2024-04-24T08:17:18.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.173+0000 7ffb3e1ba740 1 -- start start 2024-04-24T08:17:18.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb38a00640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffb1406c8f0 0x7ffb1406edb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ffb28034000 tx=0x7ffb2802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:18.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f5832345c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f5832640d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583264610 0x55f583239420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f58319e1f0 con 0x55f583281c30 2024-04-24T08:17:18.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f583144350 con 0x55f583264610 2024-04-24T08:17:18.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f58319a1b0 con 0x55f583264d50 2024-04-24T08:17:18.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb38a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f5832345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb38a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f5832345c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46832/0 (socket says 172.21.15.69:46832) 2024-04-24T08:17:18.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb38a00640 1 -- 172.21.15.69:0/2238665519 learned_addr learned my addr 172.21.15.69:0/2238665519 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb33e00640 1 --2- 172.21.15.69:0/2238665519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f5832640d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb38a00640 1 -- 172.21.15.69:0/2238665519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 msgr2=0x55f5832640d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb38a00640 1 --2- 172.21.15.69:0/2238665519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f5832640d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb38a00640 1 -- 172.21.15.69:0/2238665519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583264610 msgr2=0x55f583239420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb39400640 1 --2- 172.21.15.69:0/2238665519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583264610 0x55f583239420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb38a00640 1 --2- 172.21.15.69:0/2238665519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583264610 0x55f583239420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.174+0000 7ffb38a00640 1 -- 172.21.15.69:0/2238665519 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f583198f50 con 0x55f583281c30 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.175+0000 7ffb38a00640 1 --2- 172.21.15.69:0/2238665519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f5832345c0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7ffb28006da0 tx=0x7ffb2802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7446aa1ec57cc97a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.175+0000 7ffb25200640 1 -- 172.21.15.69:0/2238665519 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb2804c070 con 0x55f583281c30 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.175+0000 7ffb25200640 1 -- 172.21.15.69:0/2238665519 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb2803e460 con 0x55f583281c30 2024-04-24T08:17:18.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.175+0000 7ffb25200640 1 -- 172.21.15.69:0/2238665519 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb28037040 con 0x55f583281c30 2024-04-24T08:17:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.176+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2238665519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 msgr2=0x55f5832345c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.176+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2238665519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f5832345c0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7ffb28006da0 tx=0x7ffb2802c000 comp rx=0 tx=0).stop 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.176+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2238665519 shutdown_connections 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.176+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2238665519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583264610 0x55f583239420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.176+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2238665519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f5832640d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.176+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2238665519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f5832345c0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.176+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2238665519 >> 172.21.15.69:0/2238665519 conn(0x55f583236830 msgr2=0x55f58327f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.176+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2238665519 shutdown_connections 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.176+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2238665519 wait complete. 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.177+0000 7ffb3e1ba740 1 -- start start 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.177+0000 7ffb3e1ba740 1 Processor -- start 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.178+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f583254f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.178+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f583276750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.178+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583267a00 0x55f583260d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.178+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f5832389e0 con 0x55f583281c30 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.178+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f583157690 con 0x55f583267a00 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.178+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f583237dd0 con 0x55f583264d50 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.178+0000 7ffb3e1ba740 10 client.0 _ll_register_callbacks cb 0x55f583193650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.178+0000 7ffb33e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f583276750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.178+0000 7ffb33e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f583276750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46840/0 (socket says 172.21.15.69:46840) 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.178+0000 7ffb33e00640 1 -- 172.21.15.69:0/2657026431 learned_addr learned my addr 172.21.15.69:0/2657026431 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb39400640 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583267a00 0x55f583260d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb38a00640 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f583254f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb33e00640 1 -- 172.21.15.69:0/2657026431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 msgr2=0x55f583254f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb33e00640 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f583254f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb33e00640 1 -- 172.21.15.69:0/2657026431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583267a00 msgr2=0x55f583260d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb33e00640 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583267a00 0x55f583260d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb33e00640 1 -- 172.21.15.69:0/2657026431 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f583156920 con 0x55f583281c30 2024-04-24T08:17:18.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb33e00640 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f583276750 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000de70 tx=0x7ffb2000b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63eebf500eeaebe0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb19600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb19600640 1 -- 172.21.15.69:0/2657026431 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20016020 con 0x55f583281c30 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb19600640 1 -- 172.21.15.69:0/2657026431 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7ffb20012680 con 0x55f583281c30 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.179+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2657026431 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb2803e460 con 0x55f583281c30 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb19600640 1 -- 172.21.15.69:0/2657026431 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7ffb20020050 con 0x55f583281c30 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb3e1ba740 10 client.8289 fetch_fsmap learned FSMap version 74 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2657026431 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55f582f7a7d0 con 0x55f583281c30 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb19600640 1 -- 172.21.15.69:0/2657026431 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb2803e460 con 0x55f583281c30 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb19600640 1 -- 172.21.15.69:0/2657026431 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7ffb20023020 con 0x55f583281c30 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb3e1ba740 10 client.8289 fetch_fsmap finished waiting for FSMap version 74 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb3e1ba740 10 client.8289 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb3e1ba740 20 client.8289 populate_metadata read hostname 'smithi069' 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55f583265710 0x55f583265b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb3e1ba740 4 client.8289 mds_command: new command op to 8151 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2657026431 --> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f5831abc10 con 0x55f583265710 2024-04-24T08:17:18.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.180+0000 7ffb38a00640 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55f583265710 0x55f583265b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.181+0000 7ffb38a00640 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55f583265710 0x55f583265b40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50031208331ea6fb server_cookie=cb06bbd022111ade in_seq=0 out_seq=0 2024-04-24T08:17:18.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.181+0000 7ffb19600640 10 client.8289 ms_handle_connect on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb19600640 1 -- 172.21.15.69:0/2657026431 <== mds.0 v2:172.21.15.69:6832/4254348458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f5831abc10 con 0x55f583265710 2024-04-24T08:17:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb19600640 10 client.8289 handle_command_reply: tid=0 2024-04-24T08:17:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 client.8289 shutdown 2024-04-24T08:17:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2657026431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 msgr2=0x55f583276750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f583276750 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000de70 tx=0x7ffb2000b570 comp rx=0 tx=0).stop 2024-04-24T08:17:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2657026431 shutdown_connections 2024-04-24T08:17:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55f583265710 0x55f583265b40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f583276750 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583267a00 0x55f583260d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb19600640 0 client.8289 ms_handle_reset on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2657026431 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f583254f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2657026431 >> 172.21.15.69:0/2657026431 conn(0x55f583264610 msgr2=0x55f58327f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2657026431 shutdown_connections 2024-04-24T08:17:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2657026431 wait complete. 2024-04-24T08:17:18.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.182+0000 7ffb3e1ba740 20 client.8289 trim_cache size 0 max 16384 2024-04-24T08:17:18.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.201+0000 7ffb3e1ba740 1 Processor -- start 2024-04-24T08:17:18.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.201+0000 7ffb3e1ba740 1 -- start start 2024-04-24T08:17:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.202+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583264d50 0x55f5832341e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.202+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583267a00 0x55f583234720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.202+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583281c30 0x55f583238010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.202+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffb34205500 con 0x55f583267a00 2024-04-24T08:17:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.202+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffb34204550 con 0x55f583264d50 2024-04-24T08:17:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.202+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffb34164580 con 0x55f583281c30 2024-04-24T08:17:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb33e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583267a00 0x55f583234720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb33e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583267a00 0x55f583234720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46854/0 (socket says 172.21.15.69:46854) 2024-04-24T08:17:18.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb33e00640 1 -- 172.21.15.69:0/2549571977 learned_addr learned my addr 172.21.15.69:0/2549571977 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb39400640 1 --2- 172.21.15.69:0/2549571977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583281c30 0x55f583238010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb33e00640 1 -- 172.21.15.69:0/2549571977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583281c30 msgr2=0x55f583238010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb33e00640 1 --2- 172.21.15.69:0/2549571977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583281c30 0x55f583238010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb38a00640 1 --2- 172.21.15.69:0/2549571977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583264d50 0x55f5832341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb33e00640 1 -- 172.21.15.69:0/2549571977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583264d50 msgr2=0x55f5832341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb33e00640 1 --2- 172.21.15.69:0/2549571977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583264d50 0x55f5832341e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb33e00640 1 -- 172.21.15.69:0/2549571977 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f582eb03f0 con 0x55f583267a00 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.203+0000 7ffb33e00640 1 --2- 172.21.15.69:0/2549571977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583267a00 0x55f583234720 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7ffb20001eb0 tx=0x7ffb20011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9424d467ab93bb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.204+0000 7ffb1be00640 1 -- 172.21.15.69:0/2549571977 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20016040 con 0x55f583267a00 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.204+0000 7ffb1be00640 1 -- 172.21.15.69:0/2549571977 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb20012940 con 0x55f583267a00 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.204+0000 7ffb1be00640 1 -- 172.21.15.69:0/2549571977 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb2001b5f0 con 0x55f583267a00 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.204+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2549571977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583267a00 msgr2=0x55f583234720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.204+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2549571977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583267a00 0x55f583234720 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7ffb20001eb0 tx=0x7ffb20011000 comp rx=0 tx=0).stop 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.204+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2549571977 shutdown_connections 2024-04-24T08:17:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.204+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2549571977 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583267a00 0x55f583234720 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.204+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2549571977 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583281c30 0x55f583238010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.204+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/2549571977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f583264d50 0x55f5832341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.204+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2549571977 >> 172.21.15.69:0/2549571977 conn(0x55f583264610 msgr2=0x55f583254d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:18.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.205+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2549571977 shutdown_connections 2024-04-24T08:17:18.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.205+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/2549571977 wait complete. 2024-04-24T08:17:18.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.205+0000 7ffb3e1ba740 1 -- start start 2024-04-24T08:17:18.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.205+0000 7ffb3e1ba740 1 Processor -- start 2024-04-24T08:17:18.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.206+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f58325a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.206+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f58327ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.206+0000 7ffb3e1ba740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f5832fbee0 0x55f583253390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.206+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffb34164bd0 con 0x55f583281c30 2024-04-24T08:17:18.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.206+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55f583237830 con 0x55f5832fbee0 2024-04-24T08:17:18.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.206+0000 7ffb3e1ba740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55f5832610d0 con 0x55f583264d50 2024-04-24T08:17:18.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb3e1ba740 10 client.0 _ll_register_callbacks cb 0x7ffb20006d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:18.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb33e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f58327ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb33e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f58327ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46856/0 (socket says 172.21.15.69:46856) 2024-04-24T08:17:18.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb33e00640 1 -- 172.21.15.69:0/1828100417 learned_addr learned my addr 172.21.15.69:0/1828100417 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb38a00640 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f58325a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb39400640 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f5832fbee0 0x55f583253390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb33e00640 1 -- 172.21.15.69:0/1828100417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 msgr2=0x55f58325a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb33e00640 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f58325a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb33e00640 1 -- 172.21.15.69:0/1828100417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f5832fbee0 msgr2=0x55f583253390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb33e00640 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f5832fbee0 0x55f583253390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.207+0000 7ffb33e00640 1 -- 172.21.15.69:0/1828100417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f582e700f0 con 0x55f583281c30 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.208+0000 7ffb33e00640 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f58327ecc0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000dae0 tx=0x7ffb20010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab2ed648736dfd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.208+0000 7ffb19600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.208+0000 7ffb19600640 1 -- 172.21.15.69:0/1828100417 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20016040 con 0x55f583281c30 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.208+0000 7ffb19600640 1 -- 172.21.15.69:0/1828100417 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7ffb2002c410 con 0x55f583281c30 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.208+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/1828100417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb20012940 con 0x55f583281c30 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.208+0000 7ffb19600640 1 -- 172.21.15.69:0/1828100417 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7ffb2003b050 con 0x55f583281c30 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.208+0000 7ffb3e1ba740 10 client.8295 fetch_fsmap learned FSMap version 74 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.208+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/1828100417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55f583163810 con 0x55f583281c30 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.208+0000 7ffb19600640 1 -- 172.21.15.69:0/1828100417 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb20012940 con 0x55f583281c30 2024-04-24T08:17:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.209+0000 7ffb19600640 1 -- 172.21.15.69:0/1828100417 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7ffb2003e020 con 0x55f583281c30 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.209+0000 7ffb3e1ba740 10 client.8295 fetch_fsmap finished waiting for FSMap version 74 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.209+0000 7ffb3e1ba740 10 client.8295 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.209+0000 7ffb3e1ba740 20 client.8295 populate_metadata read hostname 'smithi069' 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.209+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55f583254060 0x55f5832538d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.209+0000 7ffb3e1ba740 4 client.8295 mds_command: new command op to 8151 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.209+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/1828100417 --> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f5831a5a80 con 0x55f583254060 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.209+0000 7ffb38a00640 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55f583254060 0x55f5832538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.209+0000 7ffb38a00640 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55f583254060 0x55f5832538d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45420714ceb79c1 server_cookie=5d93b634a2f130e8 in_seq=0 out_seq=0 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.209+0000 7ffb19600640 10 client.8295 ms_handle_connect on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb19600640 1 -- 172.21.15.69:0/1828100417 <== mds.0 v2:172.21.15.69:6832/4254348458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+263 (crc 0 0 0) 0x55f5831a5a80 con 0x55f583254060 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb19600640 10 client.8295 handle_command_reply: tid=0 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 client.8295 shutdown 2024-04-24T08:17:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/1828100417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 msgr2=0x55f58327ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f58327ecc0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000dae0 tx=0x7ffb20010000 comp rx=0 tx=0).stop 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/1828100417 shutdown_connections 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55f583254060 0x55f5832538d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f583281c30 0x55f58327ecc0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55f5832fbee0 0x55f583253390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 --2- 172.21.15.69:0/1828100417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55f583264d50 0x55f58325a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb19600640 0 client.8295 ms_handle_reset on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/1828100417 >> 172.21.15.69:0/1828100417 conn(0x55f583264610 msgr2=0x55f58323a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/1828100417 shutdown_connections 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 1 -- 172.21.15.69:0/1828100417 wait complete. 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.210+0000 7ffb3e1ba740 20 client.8295 trim_cache size 0 max 16384 2024-04-24T08:17:18.215 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:17:18.216 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:17:18.216 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-04-24T08:17:18.216 INFO:teuthology.orchestra.run.smithi069.stdout: "dc351a0e-20c3-4182-a654-98f0fc7ad7ec": { 2024-04-24T08:17:18.216 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/dir", 2024-04-24T08:17:18.216 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "dc351a0e-20c3-4182-a654-98f0fc7ad7ec", 2024-04-24T08:17:18.216 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,repair" 2024-04-24T08:17:18.216 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:17:18.216 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:17:18.216 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:17:18.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 -- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffb1406c8f0 msgr2=0x7ffb1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:18.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffb1406c8f0 0x7ffb1406edb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7ffb28034000 tx=0x7ffb2802e000 comp rx=0 tx=0).stop 2024-04-24T08:17:18.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 -- 172.21.15.69:0/2350443863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb340cccc0 msgr2=0x7ffb34165820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:18.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb340cccc0 0x7ffb34165820 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c00b9a0 tx=0x7ffb2c00be60 comp rx=0 tx=0).stop 2024-04-24T08:17:18.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 -- 172.21.15.69:0/2350443863 shutdown_connections 2024-04-24T08:17:18.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffb1406c8f0 0x7ffb1406edb0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffb340cccc0 0x7ffb34165820 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffb340b4e90 0x7ffb34146ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 --2- 172.21.15.69:0/2350443863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb34073db0 0x7ffb340aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:18.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 -- 172.21.15.69:0/2350443863 >> 172.21.15.69:0/2350443863 conn(0x7ffb340d0350 msgr2=0x7ffb3406e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:18.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 -- 172.21.15.69:0/2350443863 shutdown_connections 2024-04-24T08:17:18.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:18.211+0000 7ffb3b400640 1 -- 172.21.15.69:0/2350443863 wait complete. 2024-04-24T08:17:18.229 INFO:tasks.cephfs.filesystem:scrub status for tag:dc351a0e-20c3-4182-a654-98f0fc7ad7ec - {'path': '/dir', 'tag': 'dc351a0e-20c3-4182-a654-98f0fc7ad7ec', 'options': 'recursive,repair'} 2024-04-24T08:17:45.865 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:17:45.874 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:17:48.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:48.230 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-24T08:17:48.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.439+0000 7fe2b4e00640 1 Processor -- start 2024-04-24T08:17:48.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.440+0000 7fe2b4e00640 1 -- start start 2024-04-24T08:17:48.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.441+0000 7fe2b4e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b00b46a0 0x7fe2b00ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.441+0000 7fe2b4e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2b015cc00 con 0x7fe2b00b3e90 2024-04-24T08:17:48.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.441+0000 7fe2b4e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2b006ed30 con 0x7fe2b015c820 2024-04-24T08:17:48.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.441+0000 7fe2b4e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe2b00b4e10 con 0x7fe2b00b46a0 2024-04-24T08:17:48.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.441+0000 7fe2a5400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe2b00b3e90 0x7fe2b00ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33220/0 (socket says 172.21.15.69:33220) 2024-04-24T08:17:48.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.441+0000 7fe2a5400640 1 -- 172.21.15.69:0/2624254966 learned_addr learned my addr 172.21.15.69:0/2624254966 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:48.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ae800640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b00b46a0 0x7fe2b00ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4093976600 0 0) 0x7fe2b015cc00 con 0x7fe2b00b3e90 2024-04-24T08:17:48.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe294003520 con 0x7fe2b00b3e90 2024-04-24T08:17:48.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 718871462 0 0) 0x7fe294003520 con 0x7fe2b00b3e90 2024-04-24T08:17:48.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b00b46a0 msgr2=0x7fe2b00ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b00b46a0 0x7fe2b00ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 >> v1:172.21.15.121:6789/0 conn(0x7fe2b015c820 legacy=0x7fe2b006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b00b50f0 con 0x7fe2b00b3e90 2024-04-24T08:17:48.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe298001670 con 0x7fe2b00b3e90 2024-04-24T08:17:48.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 >> v1:172.21.15.69:6789/0 conn(0x7fe2b00b3e90 legacy=0x7fe2b00ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b00b46a0 0x7fe2940039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe294004060 0x7fe294004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 0x7fe2940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b00b4e10 con 0x7fe294004d40 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe29400bc00 con 0x7fe294004060 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.442+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe29400be00 con 0x7fe2b00b46a0 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.443+0000 7fe2ae800640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 0x7fe2940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.443+0000 7fe2ade00640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b00b46a0 0x7fe2940039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.443+0000 7fe2a5400640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe294004060 0x7fe294004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.443+0000 7fe2ae800640 1 -- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b00b46a0 msgr2=0x7fe2940039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.443+0000 7fe2ae800640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b00b46a0 0x7fe2940039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.443+0000 7fe2ae800640 1 -- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe294004060 msgr2=0x7fe294004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.443+0000 7fe2ae800640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe294004060 0x7fe294004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.443+0000 7fe2ae800640 1 -- 172.21.15.69:0/2624254966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe294003520 con 0x7fe294004d40 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.443+0000 7fe2ade00640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b00b46a0 0x7fe2940039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.444+0000 7fe2ae800640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 0x7fe2940051c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8007360 tx=0x7fe294005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8784692df05c04 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.445+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a8009be0 con 0x7fe294004d40 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.445+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a8036600 con 0x7fe294004d40 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.445+0000 7fe2ad400640 1 -- 172.21.15.69:0/2624254966 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a80369d0 con 0x7fe294004d40 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.445+0000 7fe2b4e00640 1 -- 172.21.15.69:0/2624254966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 msgr2=0x7fe2940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.445+0000 7fe2b4e00640 1 --2- 172.21.15.69:0/2624254966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 0x7fe2940051c0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8007360 tx=0x7fe294005830 comp rx=0 tx=0).stop 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.445+0000 7fe2ae800640 1 -- 172.21.15.69:0/2624254966 reap_dead start 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.445+0000 7fe2b4e00640 1 -- 172.21.15.69:0/2624254966 shutdown_connections 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.445+0000 7fe2b4e00640 1 -- 172.21.15.69:0/2624254966 >> 172.21.15.69:0/2624254966 conn(0x7fe2b00d0350 msgr2=0x7fe2b00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.445+0000 7fe2b4e00640 1 -- 172.21.15.69:0/2624254966 shutdown_connections 2024-04-24T08:17:48.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.445+0000 7fe2b4e00640 1 -- 172.21.15.69:0/2624254966 wait complete. 2024-04-24T08:17:48.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.447+0000 7fe2b4e00640 1 Processor -- start 2024-04-24T08:17:48.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.447+0000 7fe2b4e00640 1 -- start start 2024-04-24T08:17:48.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.448+0000 7fe2b4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 0x7fe2b0154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.448+0000 7fe2b4e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b014a920 0x7fe2b015f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.448+0000 7fe2b4e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b015fa00 0x7fe2b00b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.448+0000 7fe2b4e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b00b4b30 con 0x7fe294004d40 2024-04-24T08:17:48.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.448+0000 7fe2b4e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b006f2b0 con 0x7fe2b015fa00 2024-04-24T08:17:48.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.448+0000 7fe2b4e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe2b006fee0 con 0x7fe2b014a920 2024-04-24T08:17:48.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.448+0000 7fe2ade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 0x7fe2b0154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.448+0000 7fe2ade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 0x7fe2b0154b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41844/0 (socket says 172.21.15.69:41844) 2024-04-24T08:17:48.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.448+0000 7fe2ade00640 1 -- 172.21.15.69:0/1754727609 learned_addr learned my addr 172.21.15.69:0/1754727609 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:48.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.449+0000 7fe2ae800640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b015fa00 0x7fe2b00b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.449+0000 7fe2a5400640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b014a920 0x7fe2b015f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.449+0000 7fe2ade00640 1 -- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b014a920 msgr2=0x7fe2b015f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.449+0000 7fe2ade00640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b014a920 0x7fe2b015f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.449+0000 7fe2ade00640 1 -- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b015fa00 msgr2=0x7fe2b00b0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.449+0000 7fe2ade00640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b015fa00 0x7fe2b00b0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.449+0000 7fe2ade00640 1 -- 172.21.15.69:0/1754727609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2b0153ab0 con 0x7fe294004d40 2024-04-24T08:17:48.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.449+0000 7fe2ae800640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b015fa00 0x7fe2b00b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:48.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.449+0000 7fe2ade00640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 0x7fe2b0154b50 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000d560 tx=0x7fe2a000da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:48.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.449+0000 7fe2a6a00640 1 -- 172.21.15.69:0/1754727609 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0005480 con 0x7fe294004d40 2024-04-24T08:17:48.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.450+0000 7fe2a6a00640 1 -- 172.21.15.69:0/1754727609 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a0017070 con 0x7fe294004d40 2024-04-24T08:17:48.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.450+0000 7fe2a6a00640 1 -- 172.21.15.69:0/1754727609 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0003810 con 0x7fe294004d40 2024-04-24T08:17:48.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.450+0000 7fe2b4e00640 1 -- 172.21.15.69:0/1754727609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2b0153d90 con 0x7fe294004d40 2024-04-24T08:17:48.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.450+0000 7fe2b4e00640 1 -- 172.21.15.69:0/1754727609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2b0154350 con 0x7fe294004d40 2024-04-24T08:17:48.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.453+0000 7fe2a6a00640 1 -- 172.21.15.69:0/1754727609 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2a00039b0 con 0x7fe294004d40 2024-04-24T08:17:48.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.454+0000 7fe2a6a00640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe28c06c8f0 0x7fe28c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.454+0000 7fe2a6a00640 1 -- 172.21.15.69:0/1754727609 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fe2a008b9f0 con 0x7fe294004d40 2024-04-24T08:17:48.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.454+0000 7fe2b7c12740 1 Processor -- start 2024-04-24T08:17:48.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.454+0000 7fe2b7c12740 1 -- start start 2024-04-24T08:17:48.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.454+0000 7fe2a5400640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe28c06c8f0 0x7fe28c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.455+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcad765c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.455+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.455+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6610 0x55cbcad7b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.455+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cbcace01f0 con 0x55cbcada6d50 2024-04-24T08:17:48.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.455+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55cbcac86350 con 0x55cbcada6610 2024-04-24T08:17:48.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.455+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55cbcacdc1b0 con 0x55cbcadc3c30 2024-04-24T08:17:48.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2a5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2a5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41856/0 (socket says 172.21.15.69:41856) 2024-04-24T08:17:48.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2a5400640 1 -- 172.21.15.69:0/4038468077 learned_addr learned my addr 172.21.15.69:0/4038468077 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:48.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2ade00640 1 --2- 172.21.15.69:0/4038468077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcad765c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2ae800640 1 --2- 172.21.15.69:0/4038468077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6610 0x55cbcad7b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2a5400640 1 -- 172.21.15.69:0/4038468077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 msgr2=0x55cbcad765c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2a5400640 1 --2- 172.21.15.69:0/4038468077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcad765c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2a5400640 1 -- 172.21.15.69:0/4038468077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6610 msgr2=0x55cbcad7b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2a5400640 1 --2- 172.21.15.69:0/4038468077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6610 0x55cbcad7b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2a5400640 1 -- 172.21.15.69:0/4038468077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbcacdaf50 con 0x55cbcada6d50 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2a5400640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe28c06c8f0 0x7fe28c06edb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe298001bf0 tx=0x7fe298000f70 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.456+0000 7fe2a5400640 1 --2- 172.21.15.69:0/4038468077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada60d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fe29800fb10 tx=0x7fe29801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92a454abfad6dfae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.457+0000 7fe29ea00640 1 -- 172.21.15.69:0/4038468077 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29801e070 con 0x55cbcada6d50 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.457+0000 7fe29ea00640 1 -- 172.21.15.69:0/4038468077 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe298018ac0 con 0x55cbcada6d50 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.457+0000 7fe29ea00640 1 -- 172.21.15.69:0/4038468077 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298010040 con 0x55cbcada6d50 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.457+0000 7fe2b7c12740 1 -- 172.21.15.69:0/4038468077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 msgr2=0x55cbcada60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.457+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/4038468077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada60d0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fe29800fb10 tx=0x7fe29801c000 comp rx=0 tx=0).stop 2024-04-24T08:17:48.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.457+0000 7fe2b7c12740 1 -- 172.21.15.69:0/4038468077 shutdown_connections 2024-04-24T08:17:48.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.457+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/4038468077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada60d0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.457+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/4038468077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6610 0x55cbcad7b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.457+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/4038468077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcad765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.457+0000 7fe2b7c12740 1 -- 172.21.15.69:0/4038468077 >> 172.21.15.69:0/4038468077 conn(0x55cbcad78830 msgr2=0x55cbcadc10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:48.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.458+0000 7fe2b7c12740 1 -- 172.21.15.69:0/4038468077 shutdown_connections 2024-04-24T08:17:48.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.458+0000 7fe2b7c12740 1 -- 172.21.15.69:0/4038468077 wait complete. 2024-04-24T08:17:48.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.458+0000 7fe2b7c12740 1 -- start start 2024-04-24T08:17:48.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.458+0000 7fe2b7c12740 1 Processor -- start 2024-04-24T08:17:48.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.459+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.459+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcadc3c30 0x55cbcadd1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.459+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc44f0 0x55cbcadc4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.459+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cbcad7a9e0 con 0x55cbcada6d50 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.459+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55cbcac99690 con 0x55cbcadc3c30 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.459+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55cbcad79dd0 con 0x55cbcadc44f0 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2b7c12740 10 client.0 _ll_register_callbacks cb 0x55cbcacd5650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2ade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2ade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41862/0 (socket says 172.21.15.69:41862) 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2ade00640 1 -- 172.21.15.69:0/872428343 learned_addr learned my addr 172.21.15.69:0/872428343 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2ade00640 1 -- 172.21.15.69:0/872428343 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc44f0 msgr2=0x55cbcadc4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2ade00640 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc44f0 0x55cbcadc4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2ade00640 1 -- 172.21.15.69:0/872428343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcadc3c30 msgr2=0x55cbcadd1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2a5400640 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcadc3c30 0x55cbcadd1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2ade00640 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcadc3c30 0x55cbcadd1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2ade00640 1 -- 172.21.15.69:0/872428343 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbcac98920 con 0x55cbcada6d50 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe2ade00640 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada1bc0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0059340 tx=0x7fe2a0059640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea9bf1379dc0d7e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:48.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.460+0000 7fe293400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe293400640 1 -- 172.21.15.69:0/872428343 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0003ca0 con 0x55cbcada6d50 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe293400640 1 -- 172.21.15.69:0/872428343 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fe2a005eb80 con 0x55cbcada6d50 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe2b7c12740 1 -- 172.21.15.69:0/872428343 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe298018ac0 con 0x55cbcada6d50 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe293400640 1 -- 172.21.15.69:0/872428343 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7fe2a0078050 con 0x55cbcada6d50 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe2b7c12740 10 client.8307 fetch_fsmap learned FSMap version 74 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe2b7c12740 1 -- 172.21.15.69:0/872428343 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55cbcaabc7d0 con 0x55cbcada6d50 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe293400640 1 -- 172.21.15.69:0/872428343 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe298018ac0 con 0x55cbcada6d50 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe293400640 1 -- 172.21.15.69:0/872428343 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7fe2a0058a90 con 0x55cbcada6d50 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe2b7c12740 10 client.8307 fetch_fsmap finished waiting for FSMap version 74 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe2b7c12740 10 client.8307 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe2b7c12740 20 client.8307 populate_metadata read hostname 'smithi069' 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55cbcadb9000 0x55cbcadb9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe2b7c12740 4 client.8307 mds_command: new command op to 8151 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.461+0000 7fe2b7c12740 1 -- 172.21.15.69:0/872428343 --> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbcacedc10 con 0x55cbcadb9000 2024-04-24T08:17:48.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.462+0000 7fe2a5400640 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55cbcadb9000 0x55cbcadb9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.462+0000 7fe2a5400640 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55cbcadb9000 0x55cbcadb9460 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be73df38d8bb8206 server_cookie=b8a878cf1687a370 in_seq=0 out_seq=0 2024-04-24T08:17:48.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.462+0000 7fe293400640 10 client.8307 ms_handle_connect on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:48.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe293400640 1 -- 172.21.15.69:0/872428343 <== mds.0 v2:172.21.15.69:6832/4254348458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbcacedc10 con 0x55cbcadb9000 2024-04-24T08:17:48.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe293400640 10 client.8307 handle_command_reply: tid=0 2024-04-24T08:17:48.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 client.8307 shutdown 2024-04-24T08:17:48.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 -- 172.21.15.69:0/872428343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 msgr2=0x55cbcada1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada1bc0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0059340 tx=0x7fe2a0059640 comp rx=0 tx=0).stop 2024-04-24T08:17:48.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 -- 172.21.15.69:0/872428343 shutdown_connections 2024-04-24T08:17:48.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55cbcadb9000 0x55cbcadb9460 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc44f0 0x55cbcadc4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcadc3c30 0x55cbcadd1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe293400640 0 client.8307 ms_handle_reset on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:48.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/872428343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcada1bc0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 -- 172.21.15.69:0/872428343 >> 172.21.15.69:0/872428343 conn(0x55cbcada6610 msgr2=0x55cbcadc18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:48.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 -- 172.21.15.69:0/872428343 shutdown_connections 2024-04-24T08:17:48.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.463+0000 7fe2b7c12740 1 -- 172.21.15.69:0/872428343 wait complete. 2024-04-24T08:17:48.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.464+0000 7fe2b7c12740 20 client.8307 trim_cache size 0 max 16384 2024-04-24T08:17:48.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.483+0000 7fe2b7c12740 1 Processor -- start 2024-04-24T08:17:48.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.483+0000 7fe2b7c12740 1 -- start start 2024-04-24T08:17:48.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.483+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcadc2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.483+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcadc2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.483+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcadc44f0 0x55cbcadd07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.483+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0201850 con 0x55cbcada6d50 2024-04-24T08:17:48.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.483+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b02008e0 con 0x55cbcadc44f0 2024-04-24T08:17:48.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.483+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe2b015ebc0 con 0x55cbcadc3c30 2024-04-24T08:17:48.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.484+0000 7fe2ade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcadc2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.484+0000 7fe2ade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcadc2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41868/0 (socket says 172.21.15.69:41868) 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.484+0000 7fe2ade00640 1 -- 172.21.15.69:0/1392730634 learned_addr learned my addr 172.21.15.69:0/1392730634 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.484+0000 7fe2ae800640 1 --2- 172.21.15.69:0/1392730634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcadc44f0 0x55cbcadd07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.484+0000 7fe2ade00640 1 -- 172.21.15.69:0/1392730634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 msgr2=0x55cbcadc2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.484+0000 7fe2ade00640 1 --2- 172.21.15.69:0/1392730634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcadc2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.484+0000 7fe2ade00640 1 -- 172.21.15.69:0/1392730634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcadc44f0 msgr2=0x55cbcadd07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.484+0000 7fe2ade00640 1 --2- 172.21.15.69:0/1392730634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcadc44f0 0x55cbcadd07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.484+0000 7fe2ade00640 1 -- 172.21.15.69:0/1392730634 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbca9f23f0 con 0x55cbcada6d50 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.484+0000 7fe2ade00640 1 --2- 172.21.15.69:0/1392730634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcadc2880 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00578f0 tx=0x7fe2a005d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efb15a8a8a9cd389 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe29e000640 1 -- 172.21.15.69:0/1392730634 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0003ca0 con 0x55cbcada6d50 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe29e000640 1 -- 172.21.15.69:0/1392730634 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a0088070 con 0x55cbcada6d50 2024-04-24T08:17:48.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe29e000640 1 -- 172.21.15.69:0/1392730634 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a005eb80 con 0x55cbcada6d50 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1392730634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 msgr2=0x55cbcadc2880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/1392730634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcadc2880 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fe2a00578f0 tx=0x7fe2a005d000 comp rx=0 tx=0).stop 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1392730634 shutdown_connections 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/1392730634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcadc44f0 0x55cbcadd07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/1392730634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcadc2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/1392730634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcada6d50 0x55cbcadc2880 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1392730634 >> 172.21.15.69:0/1392730634 conn(0x55cbcada6610 msgr2=0x55cbcad7d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1392730634 shutdown_connections 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.485+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1392730634 wait complete. 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.486+0000 7fe2b7c12740 1 -- start start 2024-04-24T08:17:48.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.486+0000 7fe2b7c12740 1 Processor -- start 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.487+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6d50 0x55cbcae2bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.487+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcae46900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.487+0000 7fe2b7c12740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcadc44f0 0x55cbcada0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.487+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cbcadc55a0 con 0x55cbcadc44f0 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.487+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2b0200350 con 0x55cbcada6d50 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.487+0000 7fe2b7c12740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55cbcae473f0 con 0x55cbcadc3c30 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.487+0000 7fe2b7c12740 10 client.0 _ll_register_callbacks cb 0x7fe2a0059050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ae800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcadc44f0 0x55cbcada0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ae800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcadc44f0 0x55cbcada0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41876/0 (socket says 172.21.15.69:41876) 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ae800640 1 -- 172.21.15.69:0/1982669131 learned_addr learned my addr 172.21.15.69:0/1982669131 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ade00640 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6d50 0x55cbcae2bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2a5400640 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcae46900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ae800640 1 -- 172.21.15.69:0/1982669131 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 msgr2=0x55cbcae46900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ae800640 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcae46900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ae800640 1 -- 172.21.15.69:0/1982669131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6d50 msgr2=0x55cbcae2bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ae800640 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6d50 0x55cbcae2bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ae800640 1 -- 172.21.15.69:0/1982669131 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbca9b20f0 con 0x55cbcadc44f0 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ade00640 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6d50 0x55cbcae2bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe2ae800640 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcadc44f0 0x55cbcada0810 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8007450 tx=0x7fe2a8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbbf78b1c05c948 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.488+0000 7fe293400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe293400640 1 -- 172.21.15.69:0/1982669131 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a8047060 con 0x55cbcadc44f0 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1982669131 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2a0088070 con 0x55cbcadc44f0 2024-04-24T08:17:48.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe293400640 1 -- 172.21.15.69:0/1982669131 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fe2a8036d40 con 0x55cbcadc44f0 2024-04-24T08:17:48.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe293400640 1 -- 172.21.15.69:0/1982669131 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7fe2a804d050 con 0x55cbcadc44f0 2024-04-24T08:17:48.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe2b7c12740 10 client.8313 fetch_fsmap learned FSMap version 74 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1982669131 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55cbcaca5810 con 0x55cbcadc44f0 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe293400640 1 -- 172.21.15.69:0/1982669131 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a0088070 con 0x55cbcadc44f0 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe293400640 1 -- 172.21.15.69:0/1982669131 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7fe2a8050020 con 0x55cbcadc44f0 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe2b7c12740 10 client.8313 fetch_fsmap finished waiting for FSMap version 74 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe2b7c12740 10 client.8313 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe2b7c12740 20 client.8313 populate_metadata read hostname 'smithi069' 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55cbcadc1a00 0x55cbcadb6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe2b7c12740 4 client.8313 mds_command: new command op to 8151 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.489+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1982669131 --> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cbcace7a80 con 0x55cbcadc1a00 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.490+0000 7fe2ade00640 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55cbcadc1a00 0x55cbcadb6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.490+0000 7fe2ade00640 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55cbcadc1a00 0x55cbcadb6a60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c83b7b335140139e server_cookie=25883b1f8fa6b83 in_seq=0 out_seq=0 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.490+0000 7fe293400640 10 client.8313 ms_handle_connect on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:48.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.490+0000 7fe293400640 1 -- 172.21.15.69:0/1982669131 <== mds.0 v2:172.21.15.69:6832/4254348458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cbcace7a80 con 0x55cbcadc1a00 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.490+0000 7fe293400640 10 client.8313 handle_command_reply: tid=0 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.490+0000 7fe2b7c12740 1 client.8313 shutdown 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1982669131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcadc44f0 msgr2=0x55cbcada0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcadc44f0 0x55cbcada0810 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fe2a8007450 tx=0x7fe2a8004ab0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1982669131 shutdown_connections 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55cbcadc1a00 0x55cbcadb6a60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cbcadc44f0 0x55cbcada0810 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55cbcadc3c30 0x55cbcae46900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 1 --2- 172.21.15.69:0/1982669131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cbcada6d50 0x55cbcae2bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe293400640 0 client.8313 ms_handle_reset on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1982669131 >> 172.21.15.69:0/1982669131 conn(0x55cbcada6610 msgr2=0x55cbcada8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1982669131 shutdown_connections 2024-04-24T08:17:48.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 1 -- 172.21.15.69:0/1982669131 wait complete. 2024-04-24T08:17:48.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.491+0000 7fe2b7c12740 20 client.8313 trim_cache size 0 max 16384 2024-04-24T08:17:48.497 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:17:48.497 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-04-24T08:17:48.497 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-04-24T08:17:48.497 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:17:48.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 -- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe28c06c8f0 msgr2=0x7fe28c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe28c06c8f0 0x7fe28c06edb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fe298001bf0 tx=0x7fe298000f70 comp rx=0 tx=0).stop 2024-04-24T08:17:48.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 -- 172.21.15.69:0/1754727609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 msgr2=0x7fe2b0154b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 0x7fe2b0154b50 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fe2a000d560 tx=0x7fe2a000da20 comp rx=0 tx=0).stop 2024-04-24T08:17:48.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 -- 172.21.15.69:0/1754727609 shutdown_connections 2024-04-24T08:17:48.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe28c06c8f0 0x7fe28c06edb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2b015fa00 0x7fe2b00b0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2b014a920 0x7fe2b015f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 --2- 172.21.15.69:0/1754727609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe294004d40 0x7fe2b0154b50 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 -- 172.21.15.69:0/1754727609 >> 172.21.15.69:0/1754727609 conn(0x7fe2b00d0350 msgr2=0x7fe2b0071d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:48.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 -- 172.21.15.69:0/1754727609 shutdown_connections 2024-04-24T08:17:48.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.492+0000 7fe2b4e00640 1 -- 172.21.15.69:0/1754727609 wait complete. 2024-04-24T08:17:48.511 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:17:48.511 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:48.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.706+0000 7fc963600640 1 Processor -- start 2024-04-24T08:17:48.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.706+0000 7fc963600640 1 -- start start 2024-04-24T08:17:48.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.709+0000 7fc963600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc95c0d02f0 0x7fc95c06f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.709+0000 7fc963600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc95c0b46e0 con 0x7fc95c15b490 2024-04-24T08:17:48.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.709+0000 7fc963600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc95c0d0960 con 0x7fc95c06f2d0 2024-04-24T08:17:48.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.709+0000 7fc963600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc95c0d0720 con 0x7fc95c0d02f0 2024-04-24T08:17:48.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.709+0000 7fc960c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc95c15b490 0x7fc95c0b5180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33222/0 (socket says 172.21.15.69:33222) 2024-04-24T08:17:48.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.709+0000 7fc960c00640 1 -- 172.21.15.69:0/615495855 learned_addr learned my addr 172.21.15.69:0/615495855 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:48.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.709+0000 7fc961600640 1 --2- 172.21.15.69:0/615495855 >> v2:172.21.15.121:3300/0 conn(0x7fc95c0d02f0 0x7fc95c06f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 22811925 0 0) 0x7fc95c0b46e0 con 0x7fc95c15b490 2024-04-24T08:17:48.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc944003520 con 0x7fc95c15b490 2024-04-24T08:17:48.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 392099911 0 0) 0x7fc944003520 con 0x7fc95c15b490 2024-04-24T08:17:48.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 >> v2:172.21.15.121:3300/0 conn(0x7fc95c0d02f0 msgr2=0x7fc95c06f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 --2- 172.21.15.69:0/615495855 >> v2:172.21.15.121:3300/0 conn(0x7fc95c0d02f0 0x7fc95c06f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 >> v1:172.21.15.121:6789/0 conn(0x7fc95c06f2d0 legacy=0x7fc95c0cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc95c0b4980 con 0x7fc95c15b490 2024-04-24T08:17:48.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fc950001a10 con 0x7fc95c15b490 2024-04-24T08:17:48.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 >> v1:172.21.15.69:6789/0 conn(0x7fc95c15b490 legacy=0x7fc95c0b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc944003a00 0x7fc944003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc944004800 0x7fc944004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9440051c0 0x7fc94400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c0d0720 con 0x7fc9440051c0 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc94400e2d0 con 0x7fc944003a00 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.710+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc94400e4d0 con 0x7fc944004800 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.711+0000 7fc961600640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9440051c0 0x7fc94400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.711+0000 7fc960c00640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc944003a00 0x7fc944003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.711+0000 7fc95be00640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc944004800 0x7fc944004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.711+0000 7fc961600640 1 -- 172.21.15.69:0/615495855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc944004800 msgr2=0x7fc944004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.711+0000 7fc961600640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc944004800 0x7fc944004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.711+0000 7fc961600640 1 -- 172.21.15.69:0/615495855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc944003a00 msgr2=0x7fc944003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:48.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.711+0000 7fc961600640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc944003a00 0x7fc944003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.711+0000 7fc961600640 1 -- 172.21.15.69:0/615495855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc944003520 con 0x7fc9440051c0 2024-04-24T08:17:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.712+0000 7fc961600640 1 -- 172.21.15.69:0/615495855 reap_dead start 2024-04-24T08:17:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.713+0000 7fc961600640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9440051c0 0x7fc94400dbd0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fc954009ac0 tx=0x7fc944005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ed9157d20f88cd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.713+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc954009690 con 0x7fc9440051c0 2024-04-24T08:17:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.713+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc954004020 con 0x7fc9440051c0 2024-04-24T08:17:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.713+0000 7fc95b400640 1 -- 172.21.15.69:0/615495855 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc954004410 con 0x7fc9440051c0 2024-04-24T08:17:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.714+0000 7fc963600640 1 -- 172.21.15.69:0/615495855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9440051c0 msgr2=0x7fc94400dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.714+0000 7fc963600640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9440051c0 0x7fc94400dbd0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fc954009ac0 tx=0x7fc944005830 comp rx=0 tx=0).stop 2024-04-24T08:17:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.714+0000 7fc963600640 1 -- 172.21.15.69:0/615495855 shutdown_connections 2024-04-24T08:17:48.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.714+0000 7fc963600640 1 --2- 172.21.15.69:0/615495855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9440051c0 0x7fc94400dbd0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.714+0000 7fc963600640 1 -- 172.21.15.69:0/615495855 >> 172.21.15.69:0/615495855 conn(0x7fc95c067d90 msgr2=0x7fc95c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:48.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.714+0000 7fc963600640 1 -- 172.21.15.69:0/615495855 shutdown_connections 2024-04-24T08:17:48.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.714+0000 7fc963600640 1 -- 172.21.15.69:0/615495855 wait complete. 2024-04-24T08:17:48.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.716+0000 7fc963600640 1 Processor -- start 2024-04-24T08:17:48.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.716+0000 7fc963600640 1 -- start start 2024-04-24T08:17:48.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.717+0000 7fc963600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9440051c0 0x7fc95c0b1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.717+0000 7fc963600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc95c143570 0x7fc95c144d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.717+0000 7fc963600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c1572e0 0x7fc95c142890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.717+0000 7fc963600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c15b870 con 0x7fc95c143570 2024-04-24T08:17:48.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.717+0000 7fc963600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c0d0c40 con 0x7fc95c1572e0 2024-04-24T08:17:48.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.717+0000 7fc963600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc95c06f730 con 0x7fc9440051c0 2024-04-24T08:17:48.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.717+0000 7fc95be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc95c143570 0x7fc95c144d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.717+0000 7fc95be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc95c143570 0x7fc95c144d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41892/0 (socket says 172.21.15.69:41892) 2024-04-24T08:17:48.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.717+0000 7fc95be00640 1 -- 172.21.15.69:0/629965454 learned_addr learned my addr 172.21.15.69:0/629965454 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc960c00640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9440051c0 0x7fc95c0b1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc961600640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c1572e0 0x7fc95c142890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc95be00640 1 -- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9440051c0 msgr2=0x7fc95c0b1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc95be00640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9440051c0 0x7fc95c0b1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc95be00640 1 -- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c1572e0 msgr2=0x7fc95c142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc95be00640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c1572e0 0x7fc95c142890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc95be00640 1 -- 172.21.15.69:0/629965454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc95c1439d0 con 0x7fc95c143570 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc961600640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c1572e0 0x7fc95c142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc95be00640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc95c143570 0x7fc95c144d40 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fc94800e410 tx=0x7fc94800e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc958c00640 1 -- 172.21.15.69:0/629965454 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948019070 con 0x7fc95c143570 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc958c00640 1 -- 172.21.15.69:0/629965454 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9480052e0 con 0x7fc95c143570 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc963600640 1 -- 172.21.15.69:0/629965454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc95c143ca0 con 0x7fc95c143570 2024-04-24T08:17:48.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.718+0000 7fc958c00640 1 -- 172.21.15.69:0/629965454 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc948003810 con 0x7fc95c143570 2024-04-24T08:17:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.719+0000 7fc963600640 1 -- 172.21.15.69:0/629965454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc95c144260 con 0x7fc95c143570 2024-04-24T08:17:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.720+0000 7fc963600640 1 -- 172.21.15.69:0/629965454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc95c068a80 con 0x7fc95c143570 2024-04-24T08:17:48.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.721+0000 7fc958c00640 1 -- 172.21.15.69:0/629965454 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc94801c070 con 0x7fc95c143570 2024-04-24T08:17:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.722+0000 7fc958c00640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93c06c8f0 0x7fc93c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.722+0000 7fc958c00640 1 -- 172.21.15.69:0/629965454 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fc94808b220 con 0x7fc95c143570 2024-04-24T08:17:48.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.722+0000 7fc960c00640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93c06c8f0 0x7fc93c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:48.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.723+0000 7fc960c00640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93c06c8f0 0x7fc93c06edb0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc9500013d0 tx=0x7fc950009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:48.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.726+0000 7fc958c00640 1 -- 172.21.15.69:0/629965454 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc948058e90 con 0x7fc95c143570 2024-04-24T08:17:48.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.965+0000 7fc963600640 1 -- 172.21.15.69:0/629965454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc95c0cc6e0 con 0x7fc95c143570 2024-04-24T08:17:48.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.966+0000 7fc958c00640 1 -- 172.21.15.69:0/629965454 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7fc94805d190 con 0x7fc95c143570 2024-04-24T08:17:48.968 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:48.968 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:17:13.584835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8151":{"gid":8151,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6833/4254348458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4254348458},{"type":"v1","addr":"172.21.15.69:6833","nonce":4254348458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8151,"qdb_cluster":[8151]},"id":6}]} 2024-04-24T08:17:48.968 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 74 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 -- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93c06c8f0 msgr2=0x7fc93c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93c06c8f0 0x7fc93c06edb0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fc9500013d0 tx=0x7fc950009040 comp rx=0 tx=0).stop 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 -- 172.21.15.69:0/629965454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc95c143570 msgr2=0x7fc95c144d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc95c143570 0x7fc95c144d40 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fc94800e410 tx=0x7fc94800e8d0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 -- 172.21.15.69:0/629965454 shutdown_connections 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93c06c8f0 0x7fc93c06edb0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc95c143570 0x7fc95c144d40 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc95c1572e0 0x7fc95c142890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 --2- 172.21.15.69:0/629965454 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9440051c0 0x7fc95c0b1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 -- 172.21.15.69:0/629965454 >> 172.21.15.69:0/629965454 conn(0x7fc95c067d90 msgr2=0x7fc95c0b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:48.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.969+0000 7fc963600640 1 -- 172.21.15.69:0/629965454 shutdown_connections 2024-04-24T08:17:48.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:48.970+0000 7fc963600640 1 -- 172.21.15.69:0/629965454 wait complete. 2024-04-24T08:17:48.990 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a damage ls --format=json-pretty 2024-04-24T08:17:49.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.192+0000 7f6712000640 1 Processor -- start 2024-04-24T08:17:49.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.192+0000 7f6712000640 1 -- start start 2024-04-24T08:17:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.193+0000 7f6712000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f670c0b46a0 0x7f670c0b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.194+0000 7f6712000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f670c0739f0 con 0x7f670c0b3de0 2024-04-24T08:17:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.194+0000 7f6712000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f670c073cc0 con 0x7f670c15f4b0 2024-04-24T08:17:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.194+0000 7f6712000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f670c0b5010 con 0x7f670c0b46a0 2024-04-24T08:17:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.194+0000 7f670aa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f670c0b3de0 0x7f670c0732e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39516/0 (socket says 172.21.15.69:39516) 2024-04-24T08:17:49.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.194+0000 7f670aa00640 1 -- 172.21.15.69:0/1970331740 learned_addr learned my addr 172.21.15.69:0/1970331740 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:49.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.194+0000 7f670be00640 1 --2- 172.21.15.69:0/1970331740 >> v2:172.21.15.121:3300/0 conn(0x7f670c0b46a0 0x7f670c0b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.194+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3002141024 0 0) 0x7f670c0739f0 con 0x7f670c0b3de0 2024-04-24T08:17:49.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.194+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66f0003520 con 0x7f670c0b3de0 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1041851735 0 0) 0x7f66f0003520 con 0x7f670c0b3de0 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 >> v2:172.21.15.121:3300/0 conn(0x7f670c0b46a0 msgr2=0x7f670c0b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 --2- 172.21.15.69:0/1970331740 >> v2:172.21.15.121:3300/0 conn(0x7f670c0b46a0 0x7f670c0b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 >> v1:172.21.15.121:6789/0 conn(0x7f670c15f4b0 legacy=0x7f670c15f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670c073fa0 con 0x7f670c0b3de0 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f66f8001670 con 0x7f670c0b3de0 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 >> v1:172.21.15.69:6789/0 conn(0x7f670c0b3de0 legacy=0x7f670c0732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 --2- 172.21.15.69:0/1970331740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f00039b0 0x7f66f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 --2- 172.21.15.69:0/1970331740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66f0004800 0x7f66f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 --2- 172.21.15.69:0/1970331740 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00051c0 0x7f66f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f670c0b5010 con 0x7f66f00039b0 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f66f000c060 con 0x7f66f0004800 2024-04-24T08:17:49.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f66f000c260 con 0x7f66f00051c0 2024-04-24T08:17:49.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.195+0000 7f670b400640 1 --2- 172.21.15.69:0/1970331740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f00039b0 0x7f66f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.196+0000 7f670b400640 1 -- 172.21.15.69:0/1970331740 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00051c0 msgr2=0x7f66f000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.196+0000 7f670b400640 1 --2- 172.21.15.69:0/1970331740 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00051c0 0x7f66f000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.196+0000 7f670b400640 1 -- 172.21.15.69:0/1970331740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66f0004800 msgr2=0x7f66f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.196+0000 7f670b400640 1 --2- 172.21.15.69:0/1970331740 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66f0004800 0x7f66f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.196+0000 7f670b400640 1 -- 172.21.15.69:0/1970331740 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f0003520 con 0x7f66f00039b0 2024-04-24T08:17:49.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.196+0000 7f670be00640 1 -- 172.21.15.69:0/1970331740 reap_dead start 2024-04-24T08:17:49.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.197+0000 7f670b400640 1 --2- 172.21.15.69:0/1970331740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f00039b0 0x7f66f0003dd0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f66fc001750 tx=0x7f66f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97726fb356714b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:49.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.197+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc001370 con 0x7f66f00039b0 2024-04-24T08:17:49.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.197+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66fc0052d0 con 0x7f66f00039b0 2024-04-24T08:17:49.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.197+0000 7f670a000640 1 -- 172.21.15.69:0/1970331740 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc02abe0 con 0x7f66f00039b0 2024-04-24T08:17:49.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.197+0000 7f6712000640 1 -- 172.21.15.69:0/1970331740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f00039b0 msgr2=0x7f66f0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.197+0000 7f6712000640 1 --2- 172.21.15.69:0/1970331740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f00039b0 0x7f66f0003dd0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f66fc001750 tx=0x7f66f0005830 comp rx=0 tx=0).stop 2024-04-24T08:17:49.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.198+0000 7f6712000640 1 -- 172.21.15.69:0/1970331740 shutdown_connections 2024-04-24T08:17:49.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.198+0000 7f6712000640 1 --2- 172.21.15.69:0/1970331740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66f00039b0 0x7f66f0003dd0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.198+0000 7f6712000640 1 -- 172.21.15.69:0/1970331740 >> 172.21.15.69:0/1970331740 conn(0x7f670c0d0350 msgr2=0x7f670c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:49.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.198+0000 7f6712000640 1 -- 172.21.15.69:0/1970331740 shutdown_connections 2024-04-24T08:17:49.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.198+0000 7f6712000640 1 -- 172.21.15.69:0/1970331740 wait complete. 2024-04-24T08:17:49.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.199+0000 7f6712000640 1 Processor -- start 2024-04-24T08:17:49.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.199+0000 7f6712000640 1 -- start start 2024-04-24T08:17:49.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.200+0000 7f6712000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00039b0 0x7f670c1f7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.200+0000 7f6712000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670c1609f0 0x7f670c160e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.200+0000 7f6712000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c09c630 0x7f670c09cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.200+0000 7f6712000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f670c0b38c0 con 0x7f670c09c630 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.200+0000 7f6712000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f670c0b35e0 con 0x7f670c1609f0 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.200+0000 7f6712000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f670c0b4ad0 con 0x7f66f00039b0 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c09c630 0x7f670c09cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c09c630 0x7f670c09cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38810/0 (socket says 172.21.15.69:38810) 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670be00640 1 -- 172.21.15.69:0/4175917638 learned_addr learned my addr 172.21.15.69:0/4175917638 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670b400640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00039b0 0x7f670c1f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670aa00640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670c1609f0 0x7f670c160e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670be00640 1 -- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00039b0 msgr2=0x7f670c1f7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670be00640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00039b0 0x7f670c1f7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670be00640 1 -- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670c1609f0 msgr2=0x7f670c160e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670be00640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670c1609f0 0x7f670c160e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670be00640 1 -- 172.21.15.69:0/4175917638 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670c09afb0 con 0x7f670c09c630 2024-04-24T08:17:49.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.201+0000 7f670be00640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c09c630 0x7f670c09cab0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f670c0cdd90 tx=0x7f670400ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:49.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.202+0000 7f6703400640 1 -- 172.21.15.69:0/4175917638 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670400bb40 con 0x7f670c09c630 2024-04-24T08:17:49.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.202+0000 7f6703400640 1 -- 172.21.15.69:0/4175917638 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f670401a070 con 0x7f670c09c630 2024-04-24T08:17:49.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.202+0000 7f6703400640 1 -- 172.21.15.69:0/4175917638 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6704014400 con 0x7f670c09c630 2024-04-24T08:17:49.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.202+0000 7f6712000640 1 -- 172.21.15.69:0/4175917638 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f670c09b290 con 0x7f670c09c630 2024-04-24T08:17:49.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.202+0000 7f6712000640 1 -- 172.21.15.69:0/4175917638 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f670c09b850 con 0x7f670c09c630 2024-04-24T08:17:49.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.205+0000 7f6703400640 1 -- 172.21.15.69:0/4175917638 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6704005d10 con 0x7f670c09c630 2024-04-24T08:17:49.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.206+0000 7f6703400640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66ec06c8f0 0x7f66ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.206+0000 7f6703400640 1 -- 172.21.15.69:0/4175917638 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f670408be00 con 0x7f670c09c630 2024-04-24T08:17:49.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.206+0000 7f6714fc4740 1 Processor -- start 2024-04-24T08:17:49.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.207+0000 7f6714fc4740 1 -- start start 2024-04-24T08:17:49.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.207+0000 7f670b400640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66ec06c8f0 0x7f66ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b57eff0 0x55c89b57f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b59c890 0x55c89b59cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b554cd0 0x55c89b59af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f6714fc4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c89b4b9190 con 0x55c89b57eff0 2024-04-24T08:17:49.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f6714fc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c89b45f030 con 0x55c89b59c890 2024-04-24T08:17:49.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f6714fc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c89b4b5150 con 0x55c89b554cd0 2024-04-24T08:17:49.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f670b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b57eff0 0x55c89b57f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f670b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b57eff0 0x55c89b57f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38816/0 (socket says 172.21.15.69:38816) 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f670b400640 1 -- 172.21.15.69:0/1187736966 learned_addr learned my addr 172.21.15.69:0/1187736966 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f670b400640 1 -- 172.21.15.69:0/1187736966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b554cd0 msgr2=0x55c89b59af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f670b400640 1 --2- 172.21.15.69:0/1187736966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b554cd0 0x55c89b59af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f670b400640 1 -- 172.21.15.69:0/1187736966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b59c890 msgr2=0x55c89b59cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f670aa00640 1 --2- 172.21.15.69:0/1187736966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b59c890 0x55c89b59cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f670b400640 1 --2- 172.21.15.69:0/1187736966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b59c890 0x55c89b59cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.208+0000 7f670b400640 1 -- 172.21.15.69:0/1187736966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c89b447b60 con 0x55c89b57eff0 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.209+0000 7f670b400640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66ec06c8f0 0x7f66ec06edb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f66fc032000 tx=0x7f66fc02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.209+0000 7f670b400640 1 --2- 172.21.15.69:0/1187736966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b57eff0 0x55c89b57f3d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f66fc02a590 tx=0x7f66fc043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b269527277a530cb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.209+0000 7f66f7400640 1 -- 172.21.15.69:0/1187736966 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc04d070 con 0x55c89b57eff0 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.209+0000 7f66f7400640 1 -- 172.21.15.69:0/1187736966 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66fc005290 con 0x55c89b57eff0 2024-04-24T08:17:49.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.209+0000 7f66f7400640 1 -- 172.21.15.69:0/1187736966 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66fc0039c0 con 0x55c89b57eff0 2024-04-24T08:17:49.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.209+0000 7f6714fc4740 1 -- 172.21.15.69:0/1187736966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b57eff0 msgr2=0x55c89b57f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.210+0000 7f6714fc4740 1 --2- 172.21.15.69:0/1187736966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b57eff0 0x55c89b57f3d0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f66fc02a590 tx=0x7f66fc043000 comp rx=0 tx=0).stop 2024-04-24T08:17:49.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.210+0000 7f6714fc4740 1 -- 172.21.15.69:0/1187736966 shutdown_connections 2024-04-24T08:17:49.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.210+0000 7f6714fc4740 1 --2- 172.21.15.69:0/1187736966 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b554cd0 0x55c89b59af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.210+0000 7f6714fc4740 1 --2- 172.21.15.69:0/1187736966 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b59c890 0x55c89b59cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.210+0000 7f6714fc4740 1 --2- 172.21.15.69:0/1187736966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b57eff0 0x55c89b57f3d0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.210+0000 7f6714fc4740 1 -- 172.21.15.69:0/1187736966 >> 172.21.15.69:0/1187736966 conn(0x55c89b551600 msgr2=0x55c89b599e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:49.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.210+0000 7f6714fc4740 1 -- 172.21.15.69:0/1187736966 shutdown_connections 2024-04-24T08:17:49.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.210+0000 7f6714fc4740 1 -- 172.21.15.69:0/1187736966 wait complete. 2024-04-24T08:17:49.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.210+0000 7f6714fc4740 1 -- start start 2024-04-24T08:17:49.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.211+0000 7f6714fc4740 1 Processor -- start 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b57eff0 0x55c89b552fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b57a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b598ba0 0x55c89b57a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f6714fc4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c89b555a70 con 0x55c89b59c890 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f6714fc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c89b472500 con 0x55c89b57eff0 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f6714fc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c89b54f530 con 0x55c89b598ba0 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f6714fc4740 10 client.0 _ll_register_callbacks cb 0x55c89b4af270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f670aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b57a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f670aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b57a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38820/0 (socket says 172.21.15.69:38820) 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f670aa00640 1 -- 172.21.15.69:0/738195989 learned_addr learned my addr 172.21.15.69:0/738195989 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f670aa00640 1 -- 172.21.15.69:0/738195989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b598ba0 msgr2=0x55c89b57a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f670aa00640 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b598ba0 0x55c89b57a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f670aa00640 1 -- 172.21.15.69:0/738195989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b57eff0 msgr2=0x55c89b552fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f670aa00640 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b57eff0 0x55c89b552fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.212+0000 7f670aa00640 1 -- 172.21.15.69:0/738195989 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c89b2977c0 con 0x55c89b59c890 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.213+0000 7f670aa00640 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b57a0e0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f66f8007f50 tx=0x7f66f800e650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2f8f7f062e39c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.213+0000 7f66ebe00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.213+0000 7f66ebe00640 1 -- 172.21.15.69:0/738195989 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f8019060 con 0x55c89b59c890 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.213+0000 7f66ebe00640 1 -- 172.21.15.69:0/738195989 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f66f8014bc0 con 0x55c89b59c890 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.213+0000 7f6714fc4740 1 -- 172.21.15.69:0/738195989 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66fc005290 con 0x55c89b59c890 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.213+0000 7f66ebe00640 1 -- 172.21.15.69:0/738195989 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f66f8014dc0 con 0x55c89b59c890 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.213+0000 7f6714fc4740 10 client.8331 fetch_fsmap learned FSMap version 74 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.213+0000 7f6714fc4740 1 -- 172.21.15.69:0/738195989 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55c89b2f7f50 con 0x55c89b59c890 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.213+0000 7f66ebe00640 1 -- 172.21.15.69:0/738195989 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66fc005290 con 0x55c89b59c890 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.214+0000 7f66ebe00640 1 -- 172.21.15.69:0/738195989 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f66f80072e0 con 0x55c89b59c890 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.214+0000 7f6714fc4740 10 client.8331 fetch_fsmap finished waiting for FSMap version 74 2024-04-24T08:17:49.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.214+0000 7f6714fc4740 10 client.8331 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:17:49.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.214+0000 7f6714fc4740 20 client.8331 populate_metadata read hostname 'smithi069' 2024-04-24T08:17:49.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.214+0000 7f6714fc4740 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55c89b582b60 0x55c89b54e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.214+0000 7f6714fc4740 4 client.8331 mds_command: new command op to 8151 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:49.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.214+0000 7f6714fc4740 1 -- 172.21.15.69:0/738195989 --> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c89b4c6bb0 con 0x55c89b582b60 2024-04-24T08:17:49.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.214+0000 7f670b400640 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55c89b582b60 0x55c89b54e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.215+0000 7f66ebe00640 10 client.8331 ms_handle_connect on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:49.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.215+0000 7f670b400640 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55c89b582b60 0x55c89b54e9e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=283305cb3ab73a36 server_cookie=be2d0cb00688986 in_seq=0 out_seq=0 2024-04-24T08:17:49.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.215+0000 7f66ebe00640 1 -- 172.21.15.69:0/738195989 <== mds.0 v2:172.21.15.69:6832/4254348458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c89b4c6bb0 con 0x55c89b582b60 2024-04-24T08:17:49.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.215+0000 7f66ebe00640 10 client.8331 handle_command_reply: tid=0 2024-04-24T08:17:49.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.215+0000 7f6714fc4740 1 client.8331 shutdown 2024-04-24T08:17:49.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.215+0000 7f6714fc4740 1 -- 172.21.15.69:0/738195989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 msgr2=0x55c89b57a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.215+0000 7f6714fc4740 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b57a0e0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f66f8007f50 tx=0x7f66f800e650 comp rx=0 tx=0).stop 2024-04-24T08:17:49.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.216+0000 7f6714fc4740 1 -- 172.21.15.69:0/738195989 shutdown_connections 2024-04-24T08:17:49.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.216+0000 7f6714fc4740 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55c89b582b60 0x55c89b54e9e0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.216+0000 7f6714fc4740 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b57a0e0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.216+0000 7f6714fc4740 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b598ba0 0x55c89b57a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.216+0000 7f66ebe00640 0 client.8331 ms_handle_reset on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:49.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.216+0000 7f6714fc4740 1 --2- 172.21.15.69:0/738195989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b57eff0 0x55c89b552fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.216+0000 7f6714fc4740 1 -- 172.21.15.69:0/738195989 >> 172.21.15.69:0/738195989 conn(0x55c89b554cd0 msgr2=0x55c89b59d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:49.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.216+0000 7f6714fc4740 1 -- 172.21.15.69:0/738195989 shutdown_connections 2024-04-24T08:17:49.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.216+0000 7f6714fc4740 1 -- 172.21.15.69:0/738195989 wait complete. 2024-04-24T08:17:49.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.216+0000 7f6714fc4740 20 client.8331 trim_cache size 0 max 16384 2024-04-24T08:17:49.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.230+0000 7f6714fc4740 1 Processor -- start 2024-04-24T08:17:49.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.230+0000 7f6714fc4740 1 -- start start 2024-04-24T08:17:49.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.231+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b57eff0 0x55c89b599670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.231+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b598ba0 0x55c89b59d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.231+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b59d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.231+0000 7f6714fc4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f670c200a60 con 0x55c89b59c890 2024-04-24T08:17:49.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.231+0000 7f6714fc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f670c1f5af0 con 0x55c89b57eff0 2024-04-24T08:17:49.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.231+0000 7f6714fc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f670c200410 con 0x55c89b598ba0 2024-04-24T08:17:49.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.231+0000 7f670be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b59d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.231+0000 7f670be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b59d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38826/0 (socket says 172.21.15.69:38826) 2024-04-24T08:17:49.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.231+0000 7f670be00640 1 -- 172.21.15.69:0/715970185 learned_addr learned my addr 172.21.15.69:0/715970185 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:49.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.232+0000 7f670be00640 1 -- 172.21.15.69:0/715970185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b598ba0 msgr2=0x55c89b59d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.232+0000 7f670be00640 1 --2- 172.21.15.69:0/715970185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b598ba0 0x55c89b59d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.232+0000 7f670b400640 1 --2- 172.21.15.69:0/715970185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b57eff0 0x55c89b599670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.232+0000 7f670be00640 1 -- 172.21.15.69:0/715970185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b57eff0 msgr2=0x55c89b599670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.232+0000 7f670be00640 1 --2- 172.21.15.69:0/715970185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b57eff0 0x55c89b599670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.232+0000 7f670be00640 1 -- 172.21.15.69:0/715970185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c89b18b0b0 con 0x55c89b59c890 2024-04-24T08:17:49.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.232+0000 7f670be00640 1 --2- 172.21.15.69:0/715970185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b59d860 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f67040596c0 tx=0x7f670405a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2af41d09592ccb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:49.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.232+0000 7f66f6a00640 1 -- 172.21.15.69:0/715970185 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67040625a0 con 0x55c89b59c890 2024-04-24T08:17:49.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.232+0000 7f66f6a00640 1 -- 172.21.15.69:0/715970185 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6704057cc0 con 0x55c89b59c890 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.232+0000 7f66f6a00640 1 -- 172.21.15.69:0/715970185 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6704059c20 con 0x55c89b59c890 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.233+0000 7f6714fc4740 1 -- 172.21.15.69:0/715970185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 msgr2=0x55c89b59d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.233+0000 7f6714fc4740 1 --2- 172.21.15.69:0/715970185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b59d860 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f67040596c0 tx=0x7f670405a180 comp rx=0 tx=0).stop 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.233+0000 7f6714fc4740 1 -- 172.21.15.69:0/715970185 shutdown_connections 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.233+0000 7f6714fc4740 1 --2- 172.21.15.69:0/715970185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b59c890 0x55c89b59d860 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.233+0000 7f6714fc4740 1 --2- 172.21.15.69:0/715970185 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b598ba0 0x55c89b59d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.233+0000 7f6714fc4740 1 --2- 172.21.15.69:0/715970185 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b57eff0 0x55c89b599670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.233+0000 7f6714fc4740 1 -- 172.21.15.69:0/715970185 >> 172.21.15.69:0/715970185 conn(0x55c89b554cd0 msgr2=0x55c89b5983e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.233+0000 7f6714fc4740 1 -- 172.21.15.69:0/715970185 shutdown_connections 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.233+0000 7f6714fc4740 1 -- 172.21.15.69:0/715970185 wait complete. 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.234+0000 7f6714fc4740 1 -- start start 2024-04-24T08:17:49.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.234+0000 7f6714fc4740 1 Processor -- start 2024-04-24T08:17:49.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b57eff0 0x55c89b59d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b598ba0 0x55c89b59d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f6714fc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b59c890 0x55c89b578de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f6714fc4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c89b5829e0 con 0x55c89b598ba0 2024-04-24T08:17:49.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f6714fc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55c89b591c80 con 0x55c89b59c890 2024-04-24T08:17:49.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f6714fc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55c89b583020 con 0x55c89b57eff0 2024-04-24T08:17:49.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f6714fc4740 10 client.0 _ll_register_callbacks cb 0x7f66f8001510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:49.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f670aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b598ba0 0x55c89b59d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f670aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b598ba0 0x55c89b59d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38832/0 (socket says 172.21.15.69:38832) 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f670aa00640 1 -- 172.21.15.69:0/2163168882 learned_addr learned my addr 172.21.15.69:0/2163168882 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f670be00640 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b59c890 0x55c89b578de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f670b400640 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b57eff0 0x55c89b59d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.235+0000 7f670aa00640 1 -- 172.21.15.69:0/2163168882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b57eff0 msgr2=0x55c89b59d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f670aa00640 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b57eff0 0x55c89b59d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f670aa00640 1 -- 172.21.15.69:0/2163168882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b59c890 msgr2=0x55c89b578de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f670aa00640 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b59c890 0x55c89b578de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f670aa00640 1 -- 172.21.15.69:0/2163168882 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c89b555190 con 0x55c89b598ba0 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f670aa00640 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b598ba0 0x55c89b59d860 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f66f80070a0 tx=0x7f670c073e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c0ca2bc532c4fd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f66ebe00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f66ebe00640 1 -- 172.21.15.69:0/2163168882 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f802c060 con 0x55c89b598ba0 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f66ebe00640 1 -- 172.21.15.69:0/2163168882 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f66f8004c30 con 0x55c89b598ba0 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f6714fc4740 1 -- 172.21.15.69:0/2163168882 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6704057cc0 con 0x55c89b598ba0 2024-04-24T08:17:49.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f66ebe00640 1 -- 172.21.15.69:0/2163168882 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f66f8032050 con 0x55c89b598ba0 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f6714fc4740 10 client.8337 fetch_fsmap learned FSMap version 74 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.236+0000 7f6714fc4740 1 -- 172.21.15.69:0/2163168882 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) -- 0x55c89b613840 con 0x55c89b598ba0 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.237+0000 7f66ebe00640 1 -- 172.21.15.69:0/2163168882 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6704057cc0 con 0x55c89b598ba0 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.237+0000 7f66ebe00640 1 -- 172.21.15.69:0/2163168882 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 74) ==== 3494+0+0 (secure 0 0 0) 0x7f66f8035020 con 0x55c89b598ba0 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.237+0000 7f6714fc4740 10 client.8337 fetch_fsmap finished waiting for FSMap version 74 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.237+0000 7f6714fc4740 10 client.8337 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.237+0000 7f6714fc4740 20 client.8337 populate_metadata read hostname 'smithi069' 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.237+0000 7f6714fc4740 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55c89b5799f0 0x55c89b57fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.237+0000 7f6714fc4740 4 client.8337 mds_command: new command op to 8151 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.237+0000 7f6714fc4740 1 -- 172.21.15.69:0/2163168882 --> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) -- 0x55c89b4c0a20 con 0x55c89b5799f0 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.237+0000 7f670b400640 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55c89b5799f0 0x55c89b57fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f66ebe00640 10 client.8337 ms_handle_connect on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f670b400640 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55c89b5799f0 0x55c89b57fcb0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e58ed7f473d4623 server_cookie=51ff6168e2ac3e0f in_seq=0 out_seq=0 2024-04-24T08:17:49.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f66ebe00640 1 -- 172.21.15.69:0/2163168882 <== mds.0 v2:172.21.15.69:6832/4254348458 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c89b4c0a20 con 0x55c89b5799f0 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f66ebe00640 10 client.8337 handle_command_reply: tid=0 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 client.8337 shutdown 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 -- 172.21.15.69:0/2163168882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b598ba0 msgr2=0x55c89b59d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b598ba0 0x55c89b59d860 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f66f80070a0 tx=0x7f670c073e90 comp rx=0 tx=0).stop 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 -- 172.21.15.69:0/2163168882 shutdown_connections 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.69:6832/4254348458,v1:172.21.15.69:6833/4254348458] conn(0x55c89b5799f0 0x55c89b57fcb0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89b598ba0 0x55c89b59d860 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c89b59c890 0x55c89b578de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 --2- 172.21.15.69:0/2163168882 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55c89b57eff0 0x55c89b59d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f66ebe00640 0 client.8337 ms_handle_reset on v2:172.21.15.69:6832/4254348458 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 -- 172.21.15.69:0/2163168882 >> 172.21.15.69:0/2163168882 conn(0x55c89b554cd0 msgr2=0x55c89b599650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 -- 172.21.15.69:0/2163168882 shutdown_connections 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 1 -- 172.21.15.69:0/2163168882 wait complete. 2024-04-24T08:17:49.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.238+0000 7f6714fc4740 20 client.8337 trim_cache size 0 max 16384 2024-04-24T08:17:49.244 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:49.244 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 -- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66ec06c8f0 msgr2=0x7f66ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66ec06c8f0 0x7f66ec06edb0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f66fc032000 tx=0x7f66fc02e000 comp rx=0 tx=0).stop 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 -- 172.21.15.69:0/4175917638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c09c630 msgr2=0x7f670c09cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c09c630 0x7f670c09cab0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f670c0cdd90 tx=0x7f670400ef30 comp rx=0 tx=0).stop 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 -- 172.21.15.69:0/4175917638 shutdown_connections 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f66ec06c8f0 0x7f66ec06edb0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f670c09c630 0x7f670c09cab0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f670c1609f0 0x7f670c160e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 --2- 172.21.15.69:0/4175917638 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f66f00039b0 0x7f670c1f7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 -- 172.21.15.69:0/4175917638 >> 172.21.15.69:0/4175917638 conn(0x7f670c0d0350 msgr2=0x7f670c0ae830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:49.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 -- 172.21.15.69:0/4175917638 shutdown_connections 2024-04-24T08:17:49.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.239+0000 7f6712000640 1 -- 172.21.15.69:0/4175917638 wait complete. 2024-04-24T08:17:49.257 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T08:17:49.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.438+0000 7fb1e7e00640 1 Processor -- start 2024-04-24T08:17:49.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.439+0000 7fb1e7e00640 1 -- start start 2024-04-24T08:17:49.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.440+0000 7fb1e7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80d0cf0 0x7fb1e806e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.440+0000 7fb1e7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1e815ca30 0x7fb1e815cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.440+0000 7fb1e7e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e80ad400 con 0x7fb1e8156cc0 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.440+0000 7fb1e7e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb1e815d420 con 0x7fb1e815ca30 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.440+0000 7fb1e7e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e815d680 con 0x7fb1e80d0cf0 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.440+0000 7fb1e6a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb1e8156cc0 0x7fb1e80d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39526/0 (socket says 172.21.15.69:39526) 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.440+0000 7fb1e6a00640 1 -- 172.21.15.69:0/391683057 learned_addr learned my addr 172.21.15.69:0/391683057 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.440+0000 7fb1e7400640 1 --2- 172.21.15.69:0/391683057 >> v2:172.21.15.69:3300/0 conn(0x7fb1e815ca30 0x7fb1e815cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.441+0000 7fb1de000640 1 --2- 172.21.15.69:0/391683057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80d0cf0 0x7fb1e806e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.441+0000 7fb1e6000640 1 -- 172.21.15.69:0/391683057 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2943597667 0 0) 0x7fb1d8001110 con 0x7fb1e8156cc0 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.441+0000 7fb1e6000640 1 -- 172.21.15.69:0/391683057 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1cc003520 con 0x7fb1e8156cc0 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.441+0000 7fb1e7400640 1 -- 172.21.15.69:0/391683057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80d0cf0 msgr2=0x7fb1e806e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.441+0000 7fb1e7400640 1 --2- 172.21.15.69:0/391683057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80d0cf0 0x7fb1e806e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.441+0000 7fb1e7400640 1 -- 172.21.15.69:0/391683057 >> v1:172.21.15.69:6789/0 conn(0x7fb1e8156cc0 legacy=0x7fb1e80d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.441+0000 7fb1e7400640 1 -- 172.21.15.69:0/391683057 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e815d960 con 0x7fb1e815ca30 2024-04-24T08:17:49.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.442+0000 7fb1e7400640 1 --2- 172.21.15.69:0/391683057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 0x7fb1e815cee0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0009d90 tx=0x7fb1e002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbad5614f32b755d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.442+0000 7fb1e6000640 1 -- 172.21.15.69:0/391683057 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0031600 con 0x7fb1e815ca30 2024-04-24T08:17:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.442+0000 7fb1e6000640 1 -- 172.21.15.69:0/391683057 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e00317a0 con 0x7fb1e815ca30 2024-04-24T08:17:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.442+0000 7fb1e6000640 1 -- 172.21.15.69:0/391683057 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0031b70 con 0x7fb1e815ca30 2024-04-24T08:17:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.443+0000 7fb1e7e00640 1 -- 172.21.15.69:0/391683057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 msgr2=0x7fb1e815cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.443+0000 7fb1e7e00640 1 --2- 172.21.15.69:0/391683057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 0x7fb1e815cee0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0009d90 tx=0x7fb1e002a5d0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.443+0000 7fb1e7e00640 1 -- 172.21.15.69:0/391683057 shutdown_connections 2024-04-24T08:17:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.443+0000 7fb1e7e00640 1 --2- 172.21.15.69:0/391683057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 0x7fb1e815cee0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.443+0000 7fb1e7e00640 1 --2- 172.21.15.69:0/391683057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e80d0cf0 0x7fb1e806e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.443+0000 7fb1e7e00640 1 -- 172.21.15.69:0/391683057 >> 172.21.15.69:0/391683057 conn(0x7fb1e806be40 msgr2=0x7fb1e806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:49.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.443+0000 7fb1e7e00640 1 -- 172.21.15.69:0/391683057 shutdown_connections 2024-04-24T08:17:49.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.443+0000 7fb1e7e00640 1 -- 172.21.15.69:0/391683057 wait complete. 2024-04-24T08:17:49.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.445+0000 7fb1e7e00640 1 Processor -- start 2024-04-24T08:17:49.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.445+0000 7fb1e7e00640 1 -- start start 2024-04-24T08:17:49.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1e7e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80d0cf0 0x7fb1e81f41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1e7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e8156cc0 0x7fb1e8159fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1e7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 0x7fb1e81f5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1e7e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e806e160 con 0x7fb1e815ca30 2024-04-24T08:17:49.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1e7e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e806de80 con 0x7fb1e80d0cf0 2024-04-24T08:17:49.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1e7e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e8071c70 con 0x7fb1e8156cc0 2024-04-24T08:17:49.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1e7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 0x7fb1e81f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1e7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 0x7fb1e81f5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38846/0 (socket says 172.21.15.69:38846) 2024-04-24T08:17:49.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1e7400640 1 -- 172.21.15.69:0/3709166495 learned_addr learned my addr 172.21.15.69:0/3709166495 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:49.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1e6a00640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80d0cf0 0x7fb1e81f41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.446+0000 7fb1de000640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e8156cc0 0x7fb1e8159fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1e7400640 1 -- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e8156cc0 msgr2=0x7fb1e8159fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1e7400640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e8156cc0 0x7fb1e8159fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1e7400640 1 -- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80d0cf0 msgr2=0x7fb1e81f41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1e7400640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80d0cf0 0x7fb1e81f41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1e7400640 1 -- 172.21.15.69:0/3709166495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e8157ad0 con 0x7fb1e815ca30 2024-04-24T08:17:49.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1e6a00640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80d0cf0 0x7fb1e81f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:49.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1e7400640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 0x7fb1e81f5240 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fb1e002abc0 tx=0x7fb1e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:49.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1df400640 1 -- 172.21.15.69:0/3709166495 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e003f070 con 0x7fb1e815ca30 2024-04-24T08:17:49.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1df400640 1 -- 172.21.15.69:0/3709166495 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e0030b60 con 0x7fb1e815ca30 2024-04-24T08:17:49.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1e7e00640 1 -- 172.21.15.69:0/3709166495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e81594e0 con 0x7fb1e815ca30 2024-04-24T08:17:49.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1df400640 1 -- 172.21.15.69:0/3709166495 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0009450 con 0x7fb1e815ca30 2024-04-24T08:17:49.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.447+0000 7fb1e7e00640 1 -- 172.21.15.69:0/3709166495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e8159190 con 0x7fb1e815ca30 2024-04-24T08:17:49.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.448+0000 7fb1e7e00640 1 -- 172.21.15.69:0/3709166495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1e81564e0 con 0x7fb1e815ca30 2024-04-24T08:17:49.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.449+0000 7fb1df400640 1 -- 172.21.15.69:0/3709166495 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb1e003b410 con 0x7fb1e815ca30 2024-04-24T08:17:49.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.451+0000 7fb1df400640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c406c8f0 0x7fb1c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:49.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.451+0000 7fb1df400640 1 -- 172.21.15.69:0/3709166495 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fb1e00ad1b0 con 0x7fb1e815ca30 2024-04-24T08:17:49.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.452+0000 7fb1e6a00640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c406c8f0 0x7fb1c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:49.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.453+0000 7fb1e6a00640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c406c8f0 0x7fb1c406edb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80012b0 tx=0x7fb1d8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:49.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.456+0000 7fb1df400640 1 -- 172.21.15.69:0/3709166495 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb1e81564e0 con 0x7fb1e815ca30 2024-04-24T08:17:49.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.676+0000 7fb1e7e00640 1 -- 172.21.15.69:0/3709166495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb1e80cf7a0 con 0x7fb1e815ca30 2024-04-24T08:17:49.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.677+0000 7fb1df400640 1 -- 172.21.15.69:0/3709166495 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb1e007aeb0 con 0x7fb1e815ca30 2024-04-24T08:17:49.678 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:49.678 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T08:17:49.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.679+0000 7fb1e7e00640 1 -- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c406c8f0 msgr2=0x7fb1c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.679+0000 7fb1e7e00640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c406c8f0 0x7fb1c406edb0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80012b0 tx=0x7fb1d8009040 comp rx=0 tx=0).stop 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.679+0000 7fb1e7e00640 1 -- 172.21.15.69:0/3709166495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 msgr2=0x7fb1e81f5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.679+0000 7fb1e7e00640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 0x7fb1e81f5240 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7fb1e002abc0 tx=0x7fb1e0004a40 comp rx=0 tx=0).stop 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.680+0000 7fb1e7e00640 1 -- 172.21.15.69:0/3709166495 shutdown_connections 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.680+0000 7fb1e7e00640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c406c8f0 0x7fb1c406edb0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.680+0000 7fb1e7e00640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e815ca30 0x7fb1e81f5240 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.680+0000 7fb1e7e00640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e8156cc0 0x7fb1e8159fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.680+0000 7fb1e7e00640 1 --2- 172.21.15.69:0/3709166495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e80d0cf0 0x7fb1e81f41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.680+0000 7fb1e7e00640 1 -- 172.21.15.69:0/3709166495 >> 172.21.15.69:0/3709166495 conn(0x7fb1e806be40 msgr2=0x7fb1e8072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.680+0000 7fb1e7e00640 1 -- 172.21.15.69:0/3709166495 shutdown_connections 2024-04-24T08:17:49.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:49.680+0000 7fb1e7e00640 1 -- 172.21.15.69:0/3709166495 wait complete. 2024-04-24T08:17:49.699 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-24T08:17:49.700 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:17:49.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:17:49.797 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:17:49.907 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:17:49.907 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:17:49.908 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:17:49.908 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:17:49.963 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:17:50.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.165+0000 7f713b200640 1 Processor -- start 2024-04-24T08:17:50.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.166+0000 7f713b200640 1 -- start start 2024-04-24T08:17:50.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.167+0000 7f713b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0cf480 0x7f713c0d39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.167+0000 7f713b200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f713c0d3f30 0x7f713c0d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.167+0000 7f713b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f713c0d2950 con 0x7f713c15e530 2024-04-24T08:17:50.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.167+0000 7f713b200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f713c0d2c30 con 0x7f713c0d3f30 2024-04-24T08:17:50.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.167+0000 7f713b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f713c0d2f10 con 0x7f713c0cf480 2024-04-24T08:17:50.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.167+0000 7f7139e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f713c15e530 0x7f713c0cf9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39536/0 (socket says 172.21.15.69:39536) 2024-04-24T08:17:50.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.167+0000 7f7139e00640 1 -- 172.21.15.69:0/2508413372 learned_addr learned my addr 172.21.15.69:0/2508413372 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:50.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 946258244 0 0) 0x7f713c0d2950 con 0x7f713c15e530 2024-04-24T08:17:50.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7120003520 con 0x7f713c15e530 2024-04-24T08:17:50.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7131400640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0cf480 0x7f713c0d39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f713a800640 1 --2- 172.21.15.69:0/2508413372 >> v2:172.21.15.121:3300/0 conn(0x7f713c0d3f30 0x7f713c0d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3371858929 0 0) 0x7f7120003520 con 0x7f713c15e530 2024-04-24T08:17:50.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0cf480 msgr2=0x7f713c0d39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:50.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0cf480 0x7f713c0d39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 >> v2:172.21.15.121:3300/0 conn(0x7f713c0d3f30 msgr2=0x7f713c0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:50.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 --2- 172.21.15.69:0/2508413372 >> v2:172.21.15.121:3300/0 conn(0x7f713c0d3f30 0x7f713c0d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c0d31f0 con 0x7f713c15e530 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f712c001670 con 0x7f713c15e530 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 >> v1:172.21.15.69:6789/0 conn(0x7f713c15e530 legacy=0x7f713c0cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0cf480 0x7f71200039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7120004030 0x7f7120004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7120004d40 0x7f71200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f713c0d2c30 con 0x7f7120004d40 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f713c0d2f10 con 0x7f7120004030 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.168+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f712000bbd0 con 0x7f713c0cf480 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.169+0000 7f713a800640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7120004d40 0x7f71200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.169+0000 7f7139e00640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0cf480 0x7f71200039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.169+0000 7f7131400640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7120004030 0x7f7120004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.169+0000 7f713a800640 1 -- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0cf480 msgr2=0x7f71200039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.169+0000 7f713a800640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0cf480 0x7f71200039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.169+0000 7f713a800640 1 -- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7120004030 msgr2=0x7f7120004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:50.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.169+0000 7f713a800640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7120004030 0x7f7120004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.169+0000 7f713a800640 1 -- 172.21.15.69:0/2508413372 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7120003520 con 0x7f7120004d40 2024-04-24T08:17:50.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.170+0000 7f713a800640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7120004d40 0x7f71200051c0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f7134007330 tx=0x7f7120005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958a5b8e06da85fb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:50.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.170+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7134009be0 con 0x7f7120004d40 2024-04-24T08:17:50.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.170+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7134036600 con 0x7f7120004d40 2024-04-24T08:17:50.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.170+0000 7f7139400640 1 -- 172.21.15.69:0/2508413372 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71340369d0 con 0x7f7120004d40 2024-04-24T08:17:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.171+0000 7f713b200640 1 -- 172.21.15.69:0/2508413372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7120004d40 msgr2=0x7f71200051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.171+0000 7f713b200640 1 --2- 172.21.15.69:0/2508413372 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7120004d40 0x7f71200051c0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f7134007330 tx=0x7f7120005830 comp rx=0 tx=0).stop 2024-04-24T08:17:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.171+0000 7f713a800640 1 -- 172.21.15.69:0/2508413372 reap_dead start 2024-04-24T08:17:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.171+0000 7f713b200640 1 -- 172.21.15.69:0/2508413372 shutdown_connections 2024-04-24T08:17:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.171+0000 7f713b200640 1 -- 172.21.15.69:0/2508413372 >> 172.21.15.69:0/2508413372 conn(0x7f713c09d8e0 msgr2=0x7f713c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.171+0000 7f713b200640 1 -- 172.21.15.69:0/2508413372 shutdown_connections 2024-04-24T08:17:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.171+0000 7f713b200640 1 -- 172.21.15.69:0/2508413372 wait complete. 2024-04-24T08:17:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.173+0000 7f713b200640 1 Processor -- start 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.173+0000 7f713b200640 1 -- start start 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.174+0000 7f713b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f713c159440 0x7f713c1f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.174+0000 7f713b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f713c0b8980 0x7f713c0b8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.174+0000 7f713b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0b9320 0x7f713c0ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.174+0000 7f713b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f713c15ea90 con 0x7f713c0b8980 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.174+0000 7f713b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f713c15e910 con 0x7f713c159440 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.174+0000 7f713b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f713c0d39f0 con 0x7f713c0b9320 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.174+0000 7f7131400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f713c0b8980 0x7f713c0b8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.174+0000 7f7131400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f713c0b8980 0x7f713c0b8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38860/0 (socket says 172.21.15.69:38860) 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.174+0000 7f7131400640 1 -- 172.21.15.69:0/1845015923 learned_addr learned my addr 172.21.15.69:0/1845015923 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f7131400640 1 -- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0b9320 msgr2=0x7f713c0ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f7131400640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0b9320 0x7f713c0ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f7131400640 1 -- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f713c159440 msgr2=0x7f713c1f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f7131400640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f713c159440 0x7f713c1f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f7131400640 1 -- 172.21.15.69:0/1845015923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f713c0b7bc0 con 0x7f713c0b8980 2024-04-24T08:17:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f7131400640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f713c0b8980 0x7f713c0b8de0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f712400f510 tx=0x7f712400f9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f7132a00640 1 -- 172.21.15.69:0/1845015923 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7124019030 con 0x7f713c0b8980 2024-04-24T08:17:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f7132a00640 1 -- 172.21.15.69:0/1845015923 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7124014670 con 0x7f713c0b8980 2024-04-24T08:17:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f7132a00640 1 -- 172.21.15.69:0/1845015923 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7124014a40 con 0x7f713c0b8980 2024-04-24T08:17:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f713b200640 1 -- 172.21.15.69:0/1845015923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f713c0b7ea0 con 0x7f713c0b8980 2024-04-24T08:17:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.175+0000 7f713b200640 1 -- 172.21.15.69:0/1845015923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f713c0b8460 con 0x7f713c0b8980 2024-04-24T08:17:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.176+0000 7f713b200640 1 -- 172.21.15.69:0/1845015923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f713c075d50 con 0x7f713c0b8980 2024-04-24T08:17:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.177+0000 7f7132a00640 1 -- 172.21.15.69:0/1845015923 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f712401c070 con 0x7f713c0b8980 2024-04-24T08:17:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.178+0000 7f7132a00640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f711806c940 0x7f711806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.178+0000 7f7132a00640 1 -- 172.21.15.69:0/1845015923 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f712408c200 con 0x7f713c0b8980 2024-04-24T08:17:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.179+0000 7f7139e00640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f711806c940 0x7f711806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.180+0000 7f7139e00640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f711806c940 0x7f711806ee00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f712c005a50 tx=0x7f712c000f50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.183+0000 7f7132a00640 1 -- 172.21.15.69:0/1845015923 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f712401c360 con 0x7f713c0b8980 2024-04-24T08:17:50.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.368+0000 7f713b200640 1 -- 172.21.15.69:0/1845015923 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f713c0d2520 con 0x7f713c0b8980 2024-04-24T08:17:50.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.371+0000 7f7132a00640 1 -- 172.21.15.69:0/1845015923 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+16947 (secure 0 0 0) 0x7f712405a4b0 con 0x7f713c0b8980 2024-04-24T08:17:50.372 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:50.373 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":86,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:17:08.931744+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6832/2529876410":"2024-04-25T08:17:08.931681+0000","172.21.15.121:6837/1179683245":"2024-04-25T08:15:45.259762+0000","172.21.15.69:6833/2529876410":"2024-04-25T08:17:08.931681+0000","172.21.15.121:6835/1179683245":"2024-04-25T08:15:45.259762+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:17:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 -- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f711806c940 msgr2=0x7f711806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f711806c940 0x7f711806ee00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f712c005a50 tx=0x7f712c000f50 comp rx=0 tx=0).stop 2024-04-24T08:17:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 -- 172.21.15.69:0/1845015923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f713c0b8980 msgr2=0x7f713c0b8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f713c0b8980 0x7f713c0b8de0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f712400f510 tx=0x7f712400f9d0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 -- 172.21.15.69:0/1845015923 shutdown_connections 2024-04-24T08:17:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f711806c940 0x7f711806ee00 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f713c0b8980 0x7f713c0b8de0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f713c0b9320 0x7f713c0ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 --2- 172.21.15.69:0/1845015923 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f713c159440 0x7f713c1f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 -- 172.21.15.69:0/1845015923 >> 172.21.15.69:0/1845015923 conn(0x7f713c09d8e0 msgr2=0x7f713c0b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 -- 172.21.15.69:0/1845015923 shutdown_connections 2024-04-24T08:17:50.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.373+0000 7f713b200640 1 -- 172.21.15.69:0/1845015923 wait complete. 2024-04-24T08:17:50.392 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:15:21.126081+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:15:22.402310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:17:50.392 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:17:50.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.592+0000 7fbf0e800640 1 Processor -- start 2024-04-24T08:17:50.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.592+0000 7fbf0e800640 1 -- start start 2024-04-24T08:17:50.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.593+0000 7fbf0e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf0815c9b0 0x7fbf080b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.594+0000 7fbf0e800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf080cd6d0 con 0x7fbf080b3f70 2024-04-24T08:17:50.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.594+0000 7fbf0e800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf080cd9b0 con 0x7fbf08155f20 2024-04-24T08:17:50.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.594+0000 7fbf0e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbf0806db90 con 0x7fbf0815c9b0 2024-04-24T08:17:50.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.594+0000 7fbf07e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbf080b3f70 0x7fbf080cf360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39542/0 (socket says 172.21.15.69:39542) 2024-04-24T08:17:50.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.594+0000 7fbf07e00640 1 -- 172.21.15.69:0/3036113217 learned_addr learned my addr 172.21.15.69:0/3036113217 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:50.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.594+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2187532440 0 0) 0x7fbf080cd6d0 con 0x7fbf080b3f70 2024-04-24T08:17:50.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.594+0000 7fbefea00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf0815c9b0 0x7fbf080b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.594+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeec003520 con 0x7fbf080b3f70 2024-04-24T08:17:50.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4233457450 0 0) 0x7fbeec003520 con 0x7fbf080b3f70 2024-04-24T08:17:50.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf0815c9b0 msgr2=0x7fbf080b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:50.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf0815c9b0 0x7fbf080b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 >> v1:172.21.15.121:6789/0 conn(0x7fbf08155f20 legacy=0x7fbf080ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf080cdc90 con 0x7fbf080b3f70 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fbf000019f0 con 0x7fbf080b3f70 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 >> v1:172.21.15.69:6789/0 conn(0x7fbf080b3f70 legacy=0x7fbf080cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf0815c9b0 0x7fbeec0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbeec003fb0 0x7fbeec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbeec004d40 0x7fbeec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec00bb60 con 0x7fbeec004d40 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec00bd10 con 0x7fbeec003fb0 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbeec00e120 con 0x7fbf0815c9b0 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf07e00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbeec004d40 0x7fbeec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.595+0000 7fbf07400640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf0815c9b0 0x7fbeec0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.596+0000 7fbefea00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbeec003fb0 0x7fbeec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.596+0000 7fbf07e00640 1 -- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf0815c9b0 msgr2=0x7fbeec0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.596+0000 7fbf07e00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbf0815c9b0 0x7fbeec0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.596+0000 7fbf07e00640 1 -- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbeec003fb0 msgr2=0x7fbeec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:50.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.596+0000 7fbf07e00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbeec003fb0 0x7fbeec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.596+0000 7fbf07e00640 1 -- 172.21.15.69:0/3036113217 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeec003520 con 0x7fbeec004d40 2024-04-24T08:17:50.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.596+0000 7fbefea00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbeec003fb0 0x7fbeec004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.597+0000 7fbf07e00640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbeec004d40 0x7fbeec0051c0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000a760 tx=0x7fbeec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332e63dd0883acae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.597+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0000a3b0 con 0x7fbeec004d40 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.597+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf0002f070 con 0x7fbeec004d40 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.597+0000 7fbf06a00640 1 -- 172.21.15.69:0/3036113217 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0002bc20 con 0x7fbeec004d40 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.597+0000 7fbf0e800640 1 -- 172.21.15.69:0/3036113217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbeec004d40 msgr2=0x7fbeec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.597+0000 7fbf0e800640 1 --2- 172.21.15.69:0/3036113217 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbeec004d40 0x7fbeec0051c0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000a760 tx=0x7fbeec005830 comp rx=0 tx=0).stop 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.597+0000 7fbf07e00640 1 -- 172.21.15.69:0/3036113217 reap_dead start 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.598+0000 7fbf0e800640 1 -- 172.21.15.69:0/3036113217 shutdown_connections 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.598+0000 7fbf0e800640 1 -- 172.21.15.69:0/3036113217 >> 172.21.15.69:0/3036113217 conn(0x7fbf080d0350 msgr2=0x7fbf080d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.598+0000 7fbf0e800640 1 -- 172.21.15.69:0/3036113217 shutdown_connections 2024-04-24T08:17:50.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.598+0000 7fbf0e800640 1 -- 172.21.15.69:0/3036113217 wait complete. 2024-04-24T08:17:50.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.599+0000 7fbf0e800640 1 Processor -- start 2024-04-24T08:17:50.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.600+0000 7fbf0e800640 1 -- start start 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbf0e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbeec004d40 0x7fbf080afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbf0e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf08145ed0 0x7fbf0806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbf0e800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf080c98a0 0x7fbf081f39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbf0e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbf080cd470 con 0x7fbf08145ed0 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbf0e800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbf08072070 con 0x7fbf080c98a0 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbf0e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbf080cfa70 con 0x7fbeec004d40 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbefea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf08145ed0 0x7fbf0806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbefea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf08145ed0 0x7fbf0806ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38874/0 (socket says 172.21.15.69:38874) 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbefea00640 1 -- 172.21.15.69:0/612178583 learned_addr learned my addr 172.21.15.69:0/612178583 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbf07400640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbeec004d40 0x7fbf080afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbefea00640 1 -- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbeec004d40 msgr2=0x7fbf080afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbefea00640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbeec004d40 0x7fbf080afeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbefea00640 1 -- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf080c98a0 msgr2=0x7fbf081f39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbefea00640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf080c98a0 0x7fbf081f39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.601+0000 7fbefea00640 1 -- 172.21.15.69:0/612178583 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf080b03f0 con 0x7fbf08145ed0 2024-04-24T08:17:50.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.602+0000 7fbefea00640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf08145ed0 0x7fbf0806ede0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fbef0010880 tx=0x7fbef0010d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:50.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.602+0000 7fbeffe00640 1 -- 172.21.15.69:0/612178583 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef000f040 con 0x7fbf08145ed0 2024-04-24T08:17:50.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.602+0000 7fbeffe00640 1 -- 172.21.15.69:0/612178583 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbef000eae0 con 0x7fbf08145ed0 2024-04-24T08:17:50.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.602+0000 7fbeffe00640 1 -- 172.21.15.69:0/612178583 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbef001c400 con 0x7fbf08145ed0 2024-04-24T08:17:50.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.602+0000 7fbf0e800640 1 -- 172.21.15.69:0/612178583 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf08146360 con 0x7fbf08145ed0 2024-04-24T08:17:50.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.602+0000 7fbf0e800640 1 -- 172.21.15.69:0/612178583 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf08146920 con 0x7fbf08145ed0 2024-04-24T08:17:50.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.603+0000 7fbf0e800640 1 -- 172.21.15.69:0/612178583 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf080ad300 con 0x7fbf08145ed0 2024-04-24T08:17:50.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.604+0000 7fbeffe00640 1 -- 172.21.15.69:0/612178583 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbef001c5a0 con 0x7fbf08145ed0 2024-04-24T08:17:50.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.605+0000 7fbeffe00640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbee406c920 0x7fbee406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:50.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.605+0000 7fbeffe00640 1 -- 172.21.15.69:0/612178583 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fbef008c2b0 con 0x7fbf08145ed0 2024-04-24T08:17:50.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.606+0000 7fbf07400640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbee406c920 0x7fbee406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:50.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.607+0000 7fbf07400640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbee406c920 0x7fbee406ede0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fbef8007fd0 tx=0x7fbef8009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:50.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.610+0000 7fbeffe00640 1 -- 172.21.15.69:0/612178583 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf080ad300 con 0x7fbf08145ed0 2024-04-24T08:17:50.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.813+0000 7fbf0e800640 1 -- 172.21.15.69:0/612178583 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbf080ccf70 con 0x7fbf08145ed0 2024-04-24T08:17:50.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.814+0000 7fbeffe00640 1 -- 172.21.15.69:0/612178583 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) ==== 93+0+10 (secure 0 0 0) 0x7fbef0018030 con 0x7fbf08145ed0 2024-04-24T08:17:50.816 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 -- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbee406c920 msgr2=0x7fbee406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbee406c920 0x7fbee406ede0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fbef8007fd0 tx=0x7fbef8009040 comp rx=0 tx=0).stop 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 -- 172.21.15.69:0/612178583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf08145ed0 msgr2=0x7fbf0806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf08145ed0 0x7fbf0806ede0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fbef0010880 tx=0x7fbef0010d40 comp rx=0 tx=0).stop 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 -- 172.21.15.69:0/612178583 shutdown_connections 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbee406c920 0x7fbee406ede0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf08145ed0 0x7fbf0806ede0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbf080c98a0 0x7fbf081f39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 --2- 172.21.15.69:0/612178583 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbeec004d40 0x7fbf080afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 -- 172.21.15.69:0/612178583 >> 172.21.15.69:0/612178583 conn(0x7fbf080d0350 msgr2=0x7fbf080d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 -- 172.21.15.69:0/612178583 shutdown_connections 2024-04-24T08:17:50.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:50.817+0000 7fbf0e800640 1 -- 172.21.15.69:0/612178583 wait complete. 2024-04-24T08:17:50.837 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:17:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.040+0000 7fb64dc00640 1 Processor -- start 2024-04-24T08:17:51.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.040+0000 7fb64dc00640 1 -- start start 2024-04-24T08:17:51.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.042+0000 7fb64dc00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb6480b3e90 0x7fb6480b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.042+0000 7fb64dc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64815cc00 con 0x7fb64815c820 2024-04-24T08:17:51.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.042+0000 7fb64dc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64806ed30 con 0x7fb6480ce250 2024-04-24T08:17:51.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.042+0000 7fb64dc00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb6480b4be0 con 0x7fb6480b3e90 2024-04-24T08:17:51.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.042+0000 7fb646a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb64815c820 0x7fb64806f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39556/0 (socket says 172.21.15.69:39556) 2024-04-24T08:17:51.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.042+0000 7fb646a00640 1 -- 172.21.15.69:0/3727610045 learned_addr learned my addr 172.21.15.69:0/3727610045 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:51.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.042+0000 7fb646000640 1 --2- 172.21.15.69:0/3727610045 >> v2:172.21.15.121:3300/0 conn(0x7fb6480b3e90 0x7fb6480b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 206731281 0 0) 0x7fb64815cc00 con 0x7fb64815c820 2024-04-24T08:17:51.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb62c003520 con 0x7fb64815c820 2024-04-24T08:17:51.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1068724535 0 0) 0x7fb62c003520 con 0x7fb64815c820 2024-04-24T08:17:51.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 >> v2:172.21.15.121:3300/0 conn(0x7fb6480b3e90 msgr2=0x7fb6480b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:51.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 --2- 172.21.15.69:0/3727610045 >> v2:172.21.15.121:3300/0 conn(0x7fb6480b3e90 0x7fb6480b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 >> v1:172.21.15.121:6789/0 conn(0x7fb6480ce250 legacy=0x7fb6480ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6480b4e10 con 0x7fb64815c820 2024-04-24T08:17:51.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fb638001670 con 0x7fb64815c820 2024-04-24T08:17:51.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 >> v1:172.21.15.69:6789/0 conn(0x7fb64815c820 legacy=0x7fb64806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62c003a00 0x7fb62c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c004800 0x7fb62c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb62c0051c0 0x7fb62c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c00c0c0 con 0x7fb62c003a00 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb62c00c270 con 0x7fb62c004800 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.043+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb62c00c470 con 0x7fb62c0051c0 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.044+0000 7fb646a00640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62c003a00 0x7fb62c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.044+0000 7fb646000640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c004800 0x7fb62c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.044+0000 7fb646a00640 1 -- 172.21.15.69:0/3727610045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb62c0051c0 msgr2=0x7fb62c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.044+0000 7fb647400640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb62c0051c0 0x7fb62c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.044+0000 7fb646a00640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb62c0051c0 0x7fb62c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.044+0000 7fb646a00640 1 -- 172.21.15.69:0/3727610045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c004800 msgr2=0x7fb62c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.044+0000 7fb646a00640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c004800 0x7fb62c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.044+0000 7fb646a00640 1 -- 172.21.15.69:0/3727610045 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb62c003520 con 0x7fb62c003a00 2024-04-24T08:17:51.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.044+0000 7fb647400640 1 -- 172.21.15.69:0/3727610045 reap_dead start 2024-04-24T08:17:51.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.044+0000 7fb646000640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c004800 0x7fb62c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.046+0000 7fb646a00640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62c003a00 0x7fb62c003e40 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb63800c110 tx=0x7fb62c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703e114ab4cbda69 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.046+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638007c40 con 0x7fb62c003a00 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.046+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb638005af0 con 0x7fb62c003a00 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.046+0000 7fb645600640 1 -- 172.21.15.69:0/3727610045 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638007070 con 0x7fb62c003a00 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.047+0000 7fb64dc00640 1 -- 172.21.15.69:0/3727610045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62c003a00 msgr2=0x7fb62c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.047+0000 7fb64dc00640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62c003a00 0x7fb62c003e40 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb63800c110 tx=0x7fb62c005830 comp rx=0 tx=0).stop 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.047+0000 7fb64dc00640 1 -- 172.21.15.69:0/3727610045 shutdown_connections 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.047+0000 7fb64dc00640 1 --2- 172.21.15.69:0/3727610045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb62c003a00 0x7fb62c003e40 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.047+0000 7fb64dc00640 1 -- 172.21.15.69:0/3727610045 >> 172.21.15.69:0/3727610045 conn(0x7fb6480d0350 msgr2=0x7fb6480d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.047+0000 7fb64dc00640 1 -- 172.21.15.69:0/3727610045 shutdown_connections 2024-04-24T08:17:51.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.047+0000 7fb64dc00640 1 -- 172.21.15.69:0/3727610045 wait complete. 2024-04-24T08:17:51.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.048+0000 7fb64dc00640 1 Processor -- start 2024-04-24T08:17:51.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.049+0000 7fb64dc00640 1 -- start start 2024-04-24T08:17:51.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.049+0000 7fb64dc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c003a00 0x7fb6481542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.049+0000 7fb64dc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb648150790 0x7fb6481466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb64dc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6481f5360 0x7fb6481f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb64dc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb64806f2b0 con 0x7fb648150790 2024-04-24T08:17:51.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb64dc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb64806efd0 con 0x7fb62c003a00 2024-04-24T08:17:51.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb64dc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb6480b46a0 con 0x7fb6481f5360 2024-04-24T08:17:51.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb646000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb648150790 0x7fb6481466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb646000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb648150790 0x7fb6481466e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38892/0 (socket says 172.21.15.69:38892) 2024-04-24T08:17:51.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb646000640 1 -- 172.21.15.69:0/3871593786 learned_addr learned my addr 172.21.15.69:0/3871593786 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:51.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb646000640 1 -- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6481f5360 msgr2=0x7fb6481f57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb647400640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6481f5360 0x7fb6481f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb646a00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c003a00 0x7fb6481542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb646000640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6481f5360 0x7fb6481f57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb646000640 1 -- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c003a00 msgr2=0x7fb6481542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb646000640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c003a00 0x7fb6481542f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb646000640 1 -- 172.21.15.69:0/3871593786 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6481f4320 con 0x7fb648150790 2024-04-24T08:17:51.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.050+0000 7fb646a00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c003a00 0x7fb6481542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:51.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.051+0000 7fb646000640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb648150790 0x7fb6481466e0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fb634010840 tx=0x7fb634010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:51.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.051+0000 7fb63ea00640 1 -- 172.21.15.69:0/3871593786 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63400f040 con 0x7fb648150790 2024-04-24T08:17:51.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.051+0000 7fb63ea00640 1 -- 172.21.15.69:0/3871593786 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb63400e950 con 0x7fb648150790 2024-04-24T08:17:51.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.051+0000 7fb63ea00640 1 -- 172.21.15.69:0/3871593786 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63400ed20 con 0x7fb648150790 2024-04-24T08:17:51.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.051+0000 7fb64dc00640 1 -- 172.21.15.69:0/3871593786 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6480b1330 con 0x7fb648150790 2024-04-24T08:17:51.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.051+0000 7fb64dc00640 1 -- 172.21.15.69:0/3871593786 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6481f45a0 con 0x7fb648150790 2024-04-24T08:17:51.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.052+0000 7fb64dc00640 1 -- 172.21.15.69:0/3871593786 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb648068a80 con 0x7fb648150790 2024-04-24T08:17:51.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.053+0000 7fb63ea00640 1 -- 172.21.15.69:0/3871593786 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb63401c400 con 0x7fb648150790 2024-04-24T08:17:51.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.054+0000 7fb63ea00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb62806c8f0 0x7fb62806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.054+0000 7fb63ea00640 1 -- 172.21.15.69:0/3871593786 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fb63408c0a0 con 0x7fb648150790 2024-04-24T08:17:51.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.055+0000 7fb646a00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb62806c8f0 0x7fb62806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.056+0000 7fb646a00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb62806c8f0 0x7fb62806edb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb638032000 tx=0x7fb638009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:51.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.059+0000 7fb63ea00640 1 -- 172.21.15.69:0/3871593786 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6481f4320 con 0x7fb648150790 2024-04-24T08:17:51.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.257+0000 7fb64dc00640 1 -- 172.21.15.69:0/3871593786 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb6480cddc0 con 0x7fb648150790 2024-04-24T08:17:51.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.258+0000 7fb63ea00640 1 -- 172.21.15.69:0/3871593786 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) ==== 104+0+11 (secure 0 0 0) 0x7fb634018030 con 0x7fb648150790 2024-04-24T08:17:51.260 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:17:51.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 -- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb62806c8f0 msgr2=0x7fb62806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb62806c8f0 0x7fb62806edb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb638032000 tx=0x7fb638009030 comp rx=0 tx=0).stop 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 -- 172.21.15.69:0/3871593786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb648150790 msgr2=0x7fb6481466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb648150790 0x7fb6481466e0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7fb634010840 tx=0x7fb634010d00 comp rx=0 tx=0).stop 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 -- 172.21.15.69:0/3871593786 shutdown_connections 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb62806c8f0 0x7fb62806edb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb648150790 0x7fb6481466e0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb6481f5360 0x7fb6481f57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 --2- 172.21.15.69:0/3871593786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb62c003a00 0x7fb6481542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 -- 172.21.15.69:0/3871593786 >> 172.21.15.69:0/3871593786 conn(0x7fb6480d0350 msgr2=0x7fb6480b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 -- 172.21.15.69:0/3871593786 shutdown_connections 2024-04-24T08:17:51.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.261+0000 7fb64dc00640 1 -- 172.21.15.69:0/3871593786 wait complete. 2024-04-24T08:17:51.281 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:17:51.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.483+0000 7fc9a3200640 1 Processor -- start 2024-04-24T08:17:51.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.483+0000 7fc9a3200640 1 -- start start 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.484+0000 7fc9a3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9a415edf0 0x7fc9a415f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.484+0000 7fc9a3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc9a406e000 0x7fc9a406e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.484+0000 7fc9a3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9a415fbb0 con 0x7fc9a4071d70 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.484+0000 7fc9a3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc9a40d0a00 con 0x7fc9a406e000 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.484+0000 7fc9a3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a40d00d0 con 0x7fc9a415edf0 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.485+0000 7fc9a1e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc9a4071d70 0x7fc9a4072150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39560/0 (socket says 172.21.15.69:39560) 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.485+0000 7fc9a1e00640 1 -- 172.21.15.69:0/2596981002 learned_addr learned my addr 172.21.15.69:0/2596981002 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.485+0000 7fc9a2800640 1 --2- 172.21.15.69:0/2596981002 >> v2:172.21.15.69:3300/0 conn(0x7fc9a406e000 0x7fc9a406e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.485+0000 7fc999400640 1 --2- 172.21.15.69:0/2596981002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9a415edf0 0x7fc9a415f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.485+0000 7fc9a1400640 1 -- 172.21.15.69:0/2596981002 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1427255062 0 0) 0x7fc9a415fbb0 con 0x7fc9a4071d70 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.485+0000 7fc9a1400640 1 -- 172.21.15.69:0/2596981002 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc988003520 con 0x7fc9a4071d70 2024-04-24T08:17:51.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.485+0000 7fc9a2800640 1 -- 172.21.15.69:0/2596981002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9a415edf0 msgr2=0x7fc9a415f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:51.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.485+0000 7fc9a2800640 1 --2- 172.21.15.69:0/2596981002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9a415edf0 0x7fc9a415f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.485+0000 7fc9a2800640 1 -- 172.21.15.69:0/2596981002 >> v1:172.21.15.69:6789/0 conn(0x7fc9a4071d70 legacy=0x7fc9a4072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.485+0000 7fc9a2800640 1 -- 172.21.15.69:0/2596981002 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a40d03b0 con 0x7fc9a406e000 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.486+0000 7fc9a2800640 1 --2- 172.21.15.69:0/2596981002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a406e000 0x7fc9a406e490 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009d90 tx=0x7fc99c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc942ca7f5b242e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.487+0000 7fc9a1400640 1 -- 172.21.15.69:0/2596981002 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c007740 con 0x7fc9a406e000 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.487+0000 7fc9a1400640 1 -- 172.21.15.69:0/2596981002 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc99c0078e0 con 0x7fc9a406e000 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.487+0000 7fc9a1400640 1 -- 172.21.15.69:0/2596981002 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c007c90 con 0x7fc9a406e000 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.487+0000 7fc9a3200640 1 -- 172.21.15.69:0/2596981002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a406e000 msgr2=0x7fc9a406e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.487+0000 7fc9a3200640 1 --2- 172.21.15.69:0/2596981002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a406e000 0x7fc9a406e490 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009d90 tx=0x7fc99c02a660 comp rx=0 tx=0).stop 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.487+0000 7fc9a3200640 1 -- 172.21.15.69:0/2596981002 shutdown_connections 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.487+0000 7fc9a3200640 1 --2- 172.21.15.69:0/2596981002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a406e000 0x7fc9a406e490 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.487+0000 7fc9a3200640 1 --2- 172.21.15.69:0/2596981002 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9a415edf0 0x7fc9a415f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.487+0000 7fc9a3200640 1 -- 172.21.15.69:0/2596981002 >> 172.21.15.69:0/2596981002 conn(0x7fc9a4067b80 msgr2=0x7fc9a406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.488+0000 7fc9a3200640 1 -- 172.21.15.69:0/2596981002 shutdown_connections 2024-04-24T08:17:51.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.488+0000 7fc9a3200640 1 -- 172.21.15.69:0/2596981002 wait complete. 2024-04-24T08:17:51.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.489+0000 7fc9a3200640 1 Processor -- start 2024-04-24T08:17:51.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.489+0000 7fc9a3200640 1 -- start start 2024-04-24T08:17:51.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.490+0000 7fc9a3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9a406e000 0x7fc9a413a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.490+0000 7fc9a3200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9a4071d70 0x7fc9a413fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.490+0000 7fc9a3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a415edf0 0x7fc9a41464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.490+0000 7fc9a3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a40d1010 con 0x7fc9a415edf0 2024-04-24T08:17:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.490+0000 7fc9a3200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a40d0e90 con 0x7fc9a4071d70 2024-04-24T08:17:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.490+0000 7fc9a3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a41481a0 con 0x7fc9a406e000 2024-04-24T08:17:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc9a2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a415edf0 0x7fc9a41464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc9a2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a415edf0 0x7fc9a41464f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38896/0 (socket says 172.21.15.69:38896) 2024-04-24T08:17:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc9a2800640 1 -- 172.21.15.69:0/990747979 learned_addr learned my addr 172.21.15.69:0/990747979 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc9a1e00640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9a406e000 0x7fc9a413a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc999400640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9a4071d70 0x7fc9a413fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc9a2800640 1 -- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9a406e000 msgr2=0x7fc9a413a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc9a2800640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9a406e000 0x7fc9a413a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc9a2800640 1 -- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9a4071d70 msgr2=0x7fc9a413fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc9a2800640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9a4071d70 0x7fc9a413fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc9a2800640 1 -- 172.21.15.69:0/990747979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a413ff40 con 0x7fc9a415edf0 2024-04-24T08:17:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc999400640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9a4071d70 0x7fc9a413fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.491+0000 7fc9a2800640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a415edf0 0x7fc9a41464f0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fc99c02ac50 tx=0x7fc99c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.492+0000 7fc99aa00640 1 -- 172.21.15.69:0/990747979 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c040070 con 0x7fc9a415edf0 2024-04-24T08:17:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.492+0000 7fc99aa00640 1 -- 172.21.15.69:0/990747979 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc99c031c40 con 0x7fc9a415edf0 2024-04-24T08:17:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.492+0000 7fc99aa00640 1 -- 172.21.15.69:0/990747979 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c009450 con 0x7fc9a415edf0 2024-04-24T08:17:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.492+0000 7fc9a3200640 1 -- 172.21.15.69:0/990747979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a41f4a40 con 0x7fc9a415edf0 2024-04-24T08:17:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.492+0000 7fc9a3200640 1 -- 172.21.15.69:0/990747979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a41396e0 con 0x7fc9a415edf0 2024-04-24T08:17:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.493+0000 7fc9a3200640 1 -- 172.21.15.69:0/990747979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a4075d50 con 0x7fc9a415edf0 2024-04-24T08:17:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.494+0000 7fc99aa00640 1 -- 172.21.15.69:0/990747979 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc99c039040 con 0x7fc9a415edf0 2024-04-24T08:17:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.495+0000 7fc99aa00640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc98006c8f0 0x7fc98006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.495+0000 7fc99aa00640 1 -- 172.21.15.69:0/990747979 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fc99c0ad4f0 con 0x7fc9a415edf0 2024-04-24T08:17:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.496+0000 7fc9a1e00640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc98006c8f0 0x7fc98006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.497+0000 7fc9a1e00640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc98006c8f0 0x7fc98006edb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc994007fd0 tx=0x7fc994009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:51.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.499+0000 7fc99aa00640 1 -- 172.21.15.69:0/990747979 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc9a413ff40 con 0x7fc9a415edf0 2024-04-24T08:17:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.700+0000 7fc9a3200640 1 -- 172.21.15.69:0/990747979 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc9a41431f0 con 0x7fc9a415edf0 2024-04-24T08:17:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.701+0000 7fc99aa00640 1 -- 172.21.15.69:0/990747979 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) ==== 100+0+11 (secure 0 0 0) 0x7fc99c07b960 con 0x7fc9a415edf0 2024-04-24T08:17:51.703 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:17:51.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 -- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc98006c8f0 msgr2=0x7fc98006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc98006c8f0 0x7fc98006edb0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc994007fd0 tx=0x7fc994009040 comp rx=0 tx=0).stop 2024-04-24T08:17:51.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 -- 172.21.15.69:0/990747979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a415edf0 msgr2=0x7fc9a41464f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a415edf0 0x7fc9a41464f0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fc99c02ac50 tx=0x7fc99c004a40 comp rx=0 tx=0).stop 2024-04-24T08:17:51.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 -- 172.21.15.69:0/990747979 shutdown_connections 2024-04-24T08:17:51.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc98006c8f0 0x7fc98006edb0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9a415edf0 0x7fc9a41464f0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9a4071d70 0x7fc9a413fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 --2- 172.21.15.69:0/990747979 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9a406e000 0x7fc9a413a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 -- 172.21.15.69:0/990747979 >> 172.21.15.69:0/990747979 conn(0x7fc9a4067b80 msgr2=0x7fc9a4142590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:51.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 -- 172.21.15.69:0/990747979 shutdown_connections 2024-04-24T08:17:51.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.704+0000 7fc9a3200640 1 -- 172.21.15.69:0/990747979 wait complete. 2024-04-24T08:17:51.723 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:17:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.919+0000 7f8da7200640 1 Processor -- start 2024-04-24T08:17:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.920+0000 7f8da7200640 1 -- start start 2024-04-24T08:17:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.921+0000 7f8da7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8da815e960 0x7f8da8162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.921+0000 7f8da7200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8da8162980 0x7f8da806cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.921+0000 7f8da7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da806d4f0 0x7f8da80cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.921+0000 7f8da7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8da815ed40 con 0x7f8da815e960 2024-04-24T08:17:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.921+0000 7f8da7200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8da80cf050 con 0x7f8da8162980 2024-04-24T08:17:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.921+0000 7f8da7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8da8162e20 con 0x7f8da806d4f0 2024-04-24T08:17:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.921+0000 7f8da5e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8da815e960 0x7f8da8162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.921+0000 7f8da5e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8da815e960 0x7f8da8162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38904/0 (socket says 172.21.15.69:38904) 2024-04-24T08:17:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.921+0000 7f8da5e00640 1 -- 172.21.15.69:0/2813288861 learned_addr learned my addr 172.21.15.69:0/2813288861 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.922+0000 7f8da5400640 1 --2- 172.21.15.69:0/2813288861 >> v2:172.21.15.121:3300/0 conn(0x7f8da8162980 0x7f8da806cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.922+0000 7f8da5e00640 1 -- 172.21.15.69:0/2813288861 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da806d4f0 msgr2=0x7f8da80cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.922+0000 7f8da6800640 1 --2- 172.21.15.69:0/2813288861 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da806d4f0 0x7f8da80cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.922+0000 7f8da5e00640 1 --2- 172.21.15.69:0/2813288861 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da806d4f0 0x7f8da80cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.922+0000 7f8da5e00640 1 -- 172.21.15.69:0/2813288861 >> v2:172.21.15.121:3300/0 conn(0x7f8da8162980 msgr2=0x7f8da806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.922+0000 7f8da5e00640 1 --2- 172.21.15.69:0/2813288861 >> v2:172.21.15.121:3300/0 conn(0x7f8da8162980 0x7f8da806cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.922+0000 7f8da5e00640 1 -- 172.21.15.69:0/2813288861 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da8163100 con 0x7f8da815e960 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.922+0000 7f8da5400640 1 --2- 172.21.15.69:0/2813288861 >> v2:172.21.15.121:3300/0 conn(0x7f8da8162980 0x7f8da806cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.922+0000 7f8da6800640 1 --2- 172.21.15.69:0/2813288861 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da806d4f0 0x7f8da80cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.923+0000 7f8da5e00640 1 --2- 172.21.15.69:0/2813288861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da815e960 0x7f8da8162440 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f8da8073c50 tx=0x7f8d9802a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735a2a547bbb387e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.923+0000 7f8da4a00640 1 -- 172.21.15.69:0/2813288861 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98037070 con 0x7f8da815e960 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.923+0000 7f8da4a00640 1 -- 172.21.15.69:0/2813288861 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d98032440 con 0x7f8da815e960 2024-04-24T08:17:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.923+0000 7f8da4a00640 1 -- 172.21.15.69:0/2813288861 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98032810 con 0x7f8da815e960 2024-04-24T08:17:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.924+0000 7f8da7200640 1 -- 172.21.15.69:0/2813288861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da815e960 msgr2=0x7f8da8162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.924+0000 7f8da7200640 1 --2- 172.21.15.69:0/2813288861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da815e960 0x7f8da8162440 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f8da8073c50 tx=0x7f8d9802a1d0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.924+0000 7f8da7200640 1 -- 172.21.15.69:0/2813288861 shutdown_connections 2024-04-24T08:17:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.924+0000 7f8da7200640 1 --2- 172.21.15.69:0/2813288861 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da806d4f0 0x7f8da80cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.924+0000 7f8da7200640 1 --2- 172.21.15.69:0/2813288861 >> v2:172.21.15.121:3300/0 conn(0x7f8da8162980 0x7f8da806cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.924+0000 7f8da7200640 1 --2- 172.21.15.69:0/2813288861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da815e960 0x7f8da8162440 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.924+0000 7f8da7200640 1 -- 172.21.15.69:0/2813288861 >> 172.21.15.69:0/2813288861 conn(0x7f8da809d900 msgr2=0x7f8da808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.924+0000 7f8da7200640 1 -- 172.21.15.69:0/2813288861 shutdown_connections 2024-04-24T08:17:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.924+0000 7f8da7200640 1 -- 172.21.15.69:0/2813288861 wait complete. 2024-04-24T08:17:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.926+0000 7f8da7200640 1 Processor -- start 2024-04-24T08:17:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.926+0000 7f8da7200640 1 -- start start 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.927+0000 7f8da7200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da806d4f0 0x7f8da80b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.927+0000 7f8da7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da815e960 0x7f8da8160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.927+0000 7f8da7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da8162980 0x7f8da815ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.927+0000 7f8da7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8da80b6170 con 0x7f8da8162980 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.927+0000 7f8da7200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8da80cf5d0 con 0x7f8da806d4f0 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.927+0000 7f8da7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8da80b62f0 con 0x7f8da815e960 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.927+0000 7f8da6800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da8162980 0x7f8da815ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.927+0000 7f8da6800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da8162980 0x7f8da815ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38920/0 (socket says 172.21.15.69:38920) 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.927+0000 7f8da6800640 1 -- 172.21.15.69:0/3489875584 learned_addr learned my addr 172.21.15.69:0/3489875584 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8da5e00640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da806d4f0 0x7f8da80b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8da5400640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da815e960 0x7f8da8160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8da6800640 1 -- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da815e960 msgr2=0x7f8da8160d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8da6800640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da815e960 0x7f8da8160d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8da6800640 1 -- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da806d4f0 msgr2=0x7f8da80b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8da6800640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da806d4f0 0x7f8da80b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8da6800640 1 -- 172.21.15.69:0/3489875584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da815f050 con 0x7f8da8162980 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8da6800640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da8162980 0x7f8da815ca90 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f8da000c5c0 tx=0x7f8da000ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8d9e000640 1 -- 172.21.15.69:0/3489875584 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0015070 con 0x7f8da8162980 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8d9e000640 1 -- 172.21.15.69:0/3489875584 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da0004240 con 0x7f8da8162980 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8d9e000640 1 -- 172.21.15.69:0/3489875584 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0003610 con 0x7f8da8162980 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.928+0000 7f8da7200640 1 -- 172.21.15.69:0/3489875584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da815f5d0 con 0x7f8da8162980 2024-04-24T08:17:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.929+0000 7f8da7200640 1 -- 172.21.15.69:0/3489875584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da815fb90 con 0x7f8da8162980 2024-04-24T08:17:51.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.930+0000 7f8da7200640 1 -- 172.21.15.69:0/3489875584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da8075d50 con 0x7f8da8162980 2024-04-24T08:17:51.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.931+0000 7f8d9e000640 1 -- 172.21.15.69:0/3489875584 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8da00037b0 con 0x7f8da8162980 2024-04-24T08:17:51.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.932+0000 7f8d9e000640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d8406c8f0 0x7f8d8406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:51.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.932+0000 7f8d9e000640 1 -- 172.21.15.69:0/3489875584 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f8da008ba10 con 0x7f8da8162980 2024-04-24T08:17:51.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.933+0000 7f8da5e00640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d8406c8f0 0x7f8d8406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:51.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.934+0000 7f8da5e00640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d8406c8f0 0x7f8d8406edb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8d98034000 tx=0x7f8d9802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:51.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:51.937+0000 7f8d9e000640 1 -- 172.21.15.69:0/3489875584 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8da0059d60 con 0x7f8da8162980 2024-04-24T08:17:52.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.154+0000 7f8da7200640 1 -- 172.21.15.69:0/3489875584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8da80d1dc0 con 0x7f8da8162980 2024-04-24T08:17:52.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.155+0000 7f8d9e000640 1 -- 172.21.15.69:0/3489875584 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v86) ==== 80+0+246 (secure 0 0 0) 0x7f8da0059660 con 0x7f8da8162980 2024-04-24T08:17:52.158 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6832/2529876410 2024-04-25T08:17:08.931681+0000 2024-04-24T08:17:52.158 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6837/1179683245 2024-04-25T08:15:45.259762+0000 2024-04-24T08:17:52.159 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6833/2529876410 2024-04-25T08:17:08.931681+0000 2024-04-24T08:17:52.159 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6835/1179683245 2024-04-25T08:15:45.259762+0000 2024-04-24T08:17:52.160 INFO:teuthology.orchestra.run.smithi069.stderr:listed 4 entries 2024-04-24T08:17:52.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.157+0000 7f8da7200640 1 -- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d8406c8f0 msgr2=0x7f8d8406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.157+0000 7f8da7200640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d8406c8f0 0x7f8d8406edb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f8d98034000 tx=0x7f8d9802e000 comp rx=0 tx=0).stop 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.157+0000 7f8da7200640 1 -- 172.21.15.69:0/3489875584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da8162980 msgr2=0x7f8da815ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.157+0000 7f8da7200640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da8162980 0x7f8da815ca90 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f8da000c5c0 tx=0x7f8da000ca80 comp rx=0 tx=0).stop 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.158+0000 7f8da7200640 1 -- 172.21.15.69:0/3489875584 shutdown_connections 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.158+0000 7f8da7200640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d8406c8f0 0x7f8d8406edb0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.158+0000 7f8da7200640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8da8162980 0x7f8da815ca90 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.158+0000 7f8da7200640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8da815e960 0x7f8da8160d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.158+0000 7f8da7200640 1 --2- 172.21.15.69:0/3489875584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8da806d4f0 0x7f8da80b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.158+0000 7f8da7200640 1 -- 172.21.15.69:0/3489875584 >> 172.21.15.69:0/3489875584 conn(0x7f8da809d900 msgr2=0x7f8da80d1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.158+0000 7f8da7200640 1 -- 172.21.15.69:0/3489875584 shutdown_connections 2024-04-24T08:17:52.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.158+0000 7f8da7200640 1 -- 172.21.15.69:0/3489875584 wait complete. 2024-04-24T08:17:52.179 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:17:52.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:52.179 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:17:52.256 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[66883]: fuse finished with error 0 and tester_r 0 2024-04-24T08:17:52.260 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:17:52.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:52.261 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:17:52.326 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:17:52.327 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:17:52.327 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:17:52.385 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:17:52.385 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:17:52.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:52.385 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-24T08:17:52.438 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-24T08:17:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:17:52.440 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:17:52.440 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:17:52.440 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:17:52.496 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:17:52.496 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:17:52.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:52.496 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-24T08:17:52.549 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-24T08:17:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:17:52.551 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:17:52.551 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:17:52.551 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:17:52.607 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:17:52.608 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:17:52.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:52.608 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-24T08:17:52.663 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-24T08:17:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:17:52.665 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:52.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.872+0000 7fec98a00640 1 Processor -- start 2024-04-24T08:17:52.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.872+0000 7fec98a00640 1 -- start start 2024-04-24T08:17:52.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.873+0000 7fec98a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec941523f0 0x7fec9406db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:52.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.873+0000 7fec98a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec9415b980 0x7fec9415be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:52.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.873+0000 7fec98a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec940aeac0 con 0x7fec94072e20 2024-04-24T08:17:52.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.873+0000 7fec98a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fec940aeda0 con 0x7fec941523f0 2024-04-24T08:17:52.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.874+0000 7fec98a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec940af080 con 0x7fec9415b980 2024-04-24T08:17:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.874+0000 7fec93400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec941523f0 0x7fec9406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.874+0000 7fec92a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fec94072e20 0x7fec94073260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39564/0 (socket says 172.21.15.69:39564) 2024-04-24T08:17:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.874+0000 7fec92a00640 1 -- 172.21.15.69:0/3136642635 learned_addr learned my addr 172.21.15.69:0/3136642635 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.874+0000 7fec93400640 1 -- 172.21.15.69:0/3136642635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec9415b980 msgr2=0x7fec9415be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.874+0000 7fec93400640 1 --2- 172.21.15.69:0/3136642635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec9415b980 0x7fec9415be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.874+0000 7fec93400640 1 -- 172.21.15.69:0/3136642635 >> v1:172.21.15.69:6789/0 conn(0x7fec94072e20 legacy=0x7fec94073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.874+0000 7fec92000640 1 -- 172.21.15.69:0/3136642635 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 734362207 0 0) 0x7fec940aeac0 con 0x7fec94072e20 2024-04-24T08:17:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.874+0000 7fec93400640 1 -- 172.21.15.69:0/3136642635 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec940af360 con 0x7fec941523f0 2024-04-24T08:17:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.875+0000 7fec93400640 1 --2- 172.21.15.69:0/3136642635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 0x7fec9406db90 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fec840097a0 tx=0x7fec8402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ede0dd3ed5c28d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec92000640 1 -- 172.21.15.69:0/3136642635 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84037070 con 0x7fec941523f0 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec92000640 1 -- 172.21.15.69:0/3136642635 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8402ae50 con 0x7fec941523f0 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec92000640 1 -- 172.21.15.69:0/3136642635 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84032660 con 0x7fec941523f0 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec98a00640 1 -- 172.21.15.69:0/3136642635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 msgr2=0x7fec9406db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec98a00640 1 --2- 172.21.15.69:0/3136642635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 0x7fec9406db90 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fec840097a0 tx=0x7fec8402a100 comp rx=0 tx=0).stop 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec98a00640 1 -- 172.21.15.69:0/3136642635 shutdown_connections 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec98a00640 1 --2- 172.21.15.69:0/3136642635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec9415b980 0x7fec9415be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec98a00640 1 --2- 172.21.15.69:0/3136642635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 0x7fec9406db90 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec98a00640 1 -- 172.21.15.69:0/3136642635 >> 172.21.15.69:0/3136642635 conn(0x7fec940d0160 msgr2=0x7fec940d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec98a00640 1 -- 172.21.15.69:0/3136642635 shutdown_connections 2024-04-24T08:17:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.876+0000 7fec98a00640 1 -- 172.21.15.69:0/3136642635 wait complete. 2024-04-24T08:17:52.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.878+0000 7fec98a00640 1 Processor -- start 2024-04-24T08:17:52.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.878+0000 7fec98a00640 1 -- start start 2024-04-24T08:17:52.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.879+0000 7fec98a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec94072e20 0x7fec941f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:52.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.879+0000 7fec98a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 0x7fec941496d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:52.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.879+0000 7fec98a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec9415b980 0x7fec94154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:52.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.879+0000 7fec98a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fec94073970 con 0x7fec941523f0 2024-04-24T08:17:52.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.879+0000 7fec98a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fec941527d0 con 0x7fec94072e20 2024-04-24T08:17:52.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.879+0000 7fec98a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec94073af0 con 0x7fec9415b980 2024-04-24T08:17:52.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.879+0000 7fec92a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 0x7fec941496d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.879+0000 7fec92a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 0x7fec941496d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38932/0 (socket says 172.21.15.69:38932) 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.880+0000 7fec92a00640 1 -- 172.21.15.69:0/292141846 learned_addr learned my addr 172.21.15.69:0/292141846 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.880+0000 7fec93400640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec94072e20 0x7fec941f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.880+0000 7fec92a00640 1 -- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec9415b980 msgr2=0x7fec94154d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.880+0000 7fec92a00640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec9415b980 0x7fec94154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.880+0000 7fec92a00640 1 -- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec94072e20 msgr2=0x7fec941f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.880+0000 7fec92a00640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec94072e20 0x7fec941f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.880+0000 7fec92a00640 1 -- 172.21.15.69:0/292141846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec94158740 con 0x7fec941523f0 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.880+0000 7fec92a00640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 0x7fec941496d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fec8000c4d0 tx=0x7fec8000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.880+0000 7fec8b400640 1 -- 172.21.15.69:0/292141846 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80018070 con 0x7fec941523f0 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.881+0000 7fec8b400640 1 -- 172.21.15.69:0/292141846 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec80007500 con 0x7fec941523f0 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.881+0000 7fec8b400640 1 -- 172.21.15.69:0/292141846 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80004a20 con 0x7fec941523f0 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.881+0000 7fec98a00640 1 -- 172.21.15.69:0/292141846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec94149c10 con 0x7fec941523f0 2024-04-24T08:17:52.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.881+0000 7fec98a00640 1 -- 172.21.15.69:0/292141846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec94158cf0 con 0x7fec941523f0 2024-04-24T08:17:52.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.882+0000 7fec98a00640 1 -- 172.21.15.69:0/292141846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec94075d50 con 0x7fec941523f0 2024-04-24T08:17:52.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.883+0000 7fec8b400640 1 -- 172.21.15.69:0/292141846 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fec80004bc0 con 0x7fec941523f0 2024-04-24T08:17:52.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.884+0000 7fec8b400640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec7406c8f0 0x7fec7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:52.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.884+0000 7fec8b400640 1 -- 172.21.15.69:0/292141846 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fec8008ae70 con 0x7fec941523f0 2024-04-24T08:17:52.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.884+0000 7fec93400640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec7406c8f0 0x7fec7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:52.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.886+0000 7fec93400640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec7406c8f0 0x7fec7406edb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fec84034000 tx=0x7fec8402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:52.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:52.888+0000 7fec8b400640 1 -- 172.21.15.69:0/292141846 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec800591c0 con 0x7fec941523f0 2024-04-24T08:17:53.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.133+0000 7fec98a00640 1 -- 172.21.15.69:0/292141846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec940ad130 con 0x7fec941523f0 2024-04-24T08:17:53.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.134+0000 7fec8b400640 1 -- 172.21.15.69:0/292141846 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7fec80058ac0 con 0x7fec941523f0 2024-04-24T08:17:53.136 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:53.136 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:17:13.584835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8151":{"gid":8151,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6833/4254348458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4254348458},{"type":"v1","addr":"172.21.15.69:6833","nonce":4254348458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8151,"qdb_cluster":[8151]},"id":6}]} 2024-04-24T08:17:53.136 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 74 2024-04-24T08:17:53.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 -- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec7406c8f0 msgr2=0x7fec7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec7406c8f0 0x7fec7406edb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fec84034000 tx=0x7fec8402e000 comp rx=0 tx=0).stop 2024-04-24T08:17:53.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 -- 172.21.15.69:0/292141846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 msgr2=0x7fec941496d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 0x7fec941496d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fec8000c4d0 tx=0x7fec8000c990 comp rx=0 tx=0).stop 2024-04-24T08:17:53.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 -- 172.21.15.69:0/292141846 shutdown_connections 2024-04-24T08:17:53.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec7406c8f0 0x7fec7406edb0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec941523f0 0x7fec941496d0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec9415b980 0x7fec94154d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 --2- 172.21.15.69:0/292141846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec94072e20 0x7fec941f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 -- 172.21.15.69:0/292141846 >> 172.21.15.69:0/292141846 conn(0x7fec940d0160 msgr2=0x7fec94071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:53.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 -- 172.21.15.69:0/292141846 shutdown_connections 2024-04-24T08:17:53.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.137+0000 7fec98a00640 1 -- 172.21.15.69:0/292141846 wait complete. 2024-04-24T08:17:53.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:17:53.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.356+0000 7febd3000640 1 Processor -- start 2024-04-24T08:17:53.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.356+0000 7febd3000640 1 -- start start 2024-04-24T08:17:53.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.358+0000 7febd3000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc162300 0x7febcc143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.358+0000 7febd3000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febcc15f540 con 0x7febcc072230 2024-04-24T08:17:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.358+0000 7febd3000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febcc15f820 con 0x7febcc071e50 2024-04-24T08:17:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.358+0000 7febd3000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7febcc15fb00 con 0x7febcc162300 2024-04-24T08:17:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.358+0000 7febd2600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7febcc072230 0x7febcc143bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39574/0 (socket says 172.21.15.69:39574) 2024-04-24T08:17:53.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.358+0000 7febd2600640 1 -- 172.21.15.69:0/1957599921 learned_addr learned my addr 172.21.15.69:0/1957599921 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:53.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febd1200640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc162300 0x7febcc143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 125976413 0 0) 0x7febcc15f540 con 0x7febcc072230 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febb4003520 con 0x7febcc072230 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3929164790 0 0) 0x7febb4003520 con 0x7febcc072230 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc162300 msgr2=0x7febcc143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc162300 0x7febcc143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 >> v1:172.21.15.121:6789/0 conn(0x7febcc071e50 legacy=0x7febcc15ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc06dda0 con 0x7febcc072230 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7febc40019f0 con 0x7febcc072230 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 >> v1:172.21.15.69:6789/0 conn(0x7febcc072230 legacy=0x7febcc143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febb40039e0 0x7febb4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb4004800 0x7febb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc162300 0x7febb40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7febcc15fb00 con 0x7febb40039e0 2024-04-24T08:17:53.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7febb400baa0 con 0x7febb4004800 2024-04-24T08:17:53.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.359+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7febb400bca0 con 0x7febcc162300 2024-04-24T08:17:53.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.360+0000 7febd1c00640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febb40039e0 0x7febb4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.360+0000 7febd1200640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb4004800 0x7febb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.360+0000 7febd1c00640 1 -- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc162300 msgr2=0x7febb40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:53.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.360+0000 7febd1c00640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc162300 0x7febb40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.360+0000 7febd1c00640 1 -- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb4004800 msgr2=0x7febb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:53.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.360+0000 7febd1c00640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb4004800 0x7febb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.360+0000 7febd1c00640 1 -- 172.21.15.69:0/1957599921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febb4003520 con 0x7febb40039e0 2024-04-24T08:17:53.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.360+0000 7febd1200640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febb4004800 0x7febb4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:53.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.361+0000 7febd1c00640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febb40039e0 0x7febb4003e00 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7febc0001720 tx=0x7febb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce85875c06e098ec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:53.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.361+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc0001370 con 0x7febb40039e0 2024-04-24T08:17:53.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.361+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febc00052d0 con 0x7febb40039e0 2024-04-24T08:17:53.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.361+0000 7febcbe00640 1 -- 172.21.15.69:0/1957599921 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc002ab90 con 0x7febb40039e0 2024-04-24T08:17:53.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.362+0000 7febd3000640 1 -- 172.21.15.69:0/1957599921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febb40039e0 msgr2=0x7febb4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.362+0000 7febd3000640 1 --2- 172.21.15.69:0/1957599921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febb40039e0 0x7febb4003e00 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7febc0001720 tx=0x7febb4005830 comp rx=0 tx=0).stop 2024-04-24T08:17:53.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.362+0000 7febd2600640 1 -- 172.21.15.69:0/1957599921 reap_dead start 2024-04-24T08:17:53.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.362+0000 7febd3000640 1 -- 172.21.15.69:0/1957599921 shutdown_connections 2024-04-24T08:17:53.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.362+0000 7febd3000640 1 -- 172.21.15.69:0/1957599921 >> 172.21.15.69:0/1957599921 conn(0x7febcc0d0160 msgr2=0x7febcc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:53.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.362+0000 7febd3000640 1 -- 172.21.15.69:0/1957599921 shutdown_connections 2024-04-24T08:17:53.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.362+0000 7febd3000640 1 -- 172.21.15.69:0/1957599921 wait complete. 2024-04-24T08:17:53.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.364+0000 7febd3000640 1 Processor -- start 2024-04-24T08:17:53.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.364+0000 7febd3000640 1 -- start start 2024-04-24T08:17:53.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.365+0000 7febd3000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febb40039e0 0x7febcc15db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.365+0000 7febd3000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febcc1ea530 0x7febcc1e8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.365+0000 7febd3000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1ea9b0 0x7febcc15cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.365+0000 7febd3000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7febcc143810 con 0x7febcc1ea9b0 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.365+0000 7febd3000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7febcc143690 con 0x7febcc1ea530 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.365+0000 7febd3000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7febcc143990 con 0x7febb40039e0 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.365+0000 7febd2600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1ea9b0 0x7febcc15cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.365+0000 7febd2600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1ea9b0 0x7febcc15cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38946/0 (socket says 172.21.15.69:38946) 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.365+0000 7febd2600640 1 -- 172.21.15.69:0/1555890662 learned_addr learned my addr 172.21.15.69:0/1555890662 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febd1200640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febcc1ea530 0x7febcc1e8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febd1c00640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febb40039e0 0x7febcc15db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febd2600640 1 -- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febb40039e0 msgr2=0x7febcc15db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febd2600640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febb40039e0 0x7febcc15db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febd2600640 1 -- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febcc1ea530 msgr2=0x7febcc1e8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febd2600640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febcc1ea530 0x7febcc1e8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febd2600640 1 -- 172.21.15.69:0/1555890662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc15c160 con 0x7febcc1ea9b0 2024-04-24T08:17:53.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febd2600640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1ea9b0 0x7febcc15cf20 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7febc4001350 tx=0x7febc400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:53.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febc9600640 1 -- 172.21.15.69:0/1555890662 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc400a5b0 con 0x7febcc1ea9b0 2024-04-24T08:17:53.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febc9600640 1 -- 172.21.15.69:0/1555890662 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febc400a750 con 0x7febcc1ea9b0 2024-04-24T08:17:53.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febd3000640 1 -- 172.21.15.69:0/1555890662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febcc15c440 con 0x7febcc1ea9b0 2024-04-24T08:17:53.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febc9600640 1 -- 172.21.15.69:0/1555890662 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc400ab40 con 0x7febcc1ea9b0 2024-04-24T08:17:53.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.366+0000 7febd3000640 1 -- 172.21.15.69:0/1555890662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febcc15ca00 con 0x7febcc1ea9b0 2024-04-24T08:17:53.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.367+0000 7febd3000640 1 -- 172.21.15.69:0/1555890662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febcc075d50 con 0x7febcc1ea9b0 2024-04-24T08:17:53.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.369+0000 7febc9600640 1 -- 172.21.15.69:0/1555890662 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febc4011070 con 0x7febcc1ea9b0 2024-04-24T08:17:53.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.369+0000 7febc9600640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 0x7febac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.370+0000 7febc9600640 1 -- 172.21.15.69:0/1555890662 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7febc408b6c0 con 0x7febcc1ea9b0 2024-04-24T08:17:53.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.370+0000 7febd1c00640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 0x7febac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.371+0000 7febd1c00640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 0x7febac06edb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7febc0032000 tx=0x7febc002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:53.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.374+0000 7febc9600640 1 -- 172.21.15.69:0/1555890662 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febc401a060 con 0x7febcc1ea9b0 2024-04-24T08:17:53.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.575+0000 7febd3000640 1 -- 172.21.15.69:0/1555890662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7febcc0d0d90 con 0x7febcc1ea9b0 2024-04-24T08:17:53.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.578+0000 7febc9600640 1 -- 172.21.15.69:0/1555890662 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+16947 (secure 0 0 0) 0x7febc400f260 con 0x7febcc1ea9b0 2024-04-24T08:17:53.580 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:53.580 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":86,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:17:08.931744+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6832/2529876410":"2024-04-25T08:17:08.931681+0000","172.21.15.121:6837/1179683245":"2024-04-25T08:15:45.259762+0000","172.21.15.69:6833/2529876410":"2024-04-25T08:17:08.931681+0000","172.21.15.121:6835/1179683245":"2024-04-25T08:15:45.259762+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 -- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 msgr2=0x7febac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 0x7febac06edb0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7febc0032000 tx=0x7febc002e000 comp rx=0 tx=0).stop 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 -- 172.21.15.69:0/1555890662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1ea9b0 msgr2=0x7febcc15cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1ea9b0 0x7febcc15cf20 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7febc4001350 tx=0x7febc400cc60 comp rx=0 tx=0).stop 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 -- 172.21.15.69:0/1555890662 shutdown_connections 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 0x7febac06edb0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1ea9b0 0x7febcc15cf20 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febcc1ea530 0x7febcc1e8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 --2- 172.21.15.69:0/1555890662 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febb40039e0 0x7febcc15db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 -- 172.21.15.69:0/1555890662 >> 172.21.15.69:0/1555890662 conn(0x7febcc0d0160 msgr2=0x7febcc0ad0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 -- 172.21.15.69:0/1555890662 shutdown_connections 2024-04-24T08:17:53.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.581+0000 7febd3000640 1 -- 172.21.15.69:0/1555890662 wait complete. 2024-04-24T08:17:53.601 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:15:21.126081+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:15:22.402310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:17:53.601 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:17:53.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.808+0000 7f68ef000640 1 Processor -- start 2024-04-24T08:17:53.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.809+0000 7f68ef000640 1 -- start start 2024-04-24T08:17:53.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.810+0000 7f68ef000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f68e815ea60 0x7f68e80cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.810+0000 7f68ef000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80d18c0 0x7f68e80d04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.810+0000 7f68ef000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68e80d2600 con 0x7f68e80d36a0 2024-04-24T08:17:53.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.810+0000 7f68ef000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f68e80d23b0 con 0x7f68e815ea60 2024-04-24T08:17:53.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.810+0000 7f68ef000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f68e806dba0 con 0x7f68e80d18c0 2024-04-24T08:17:53.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.810+0000 7f68e7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f68e815ea60 0x7f68e80cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.811+0000 7f68e7e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f68e815ea60 0x7f68e80cff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38962/0 (socket says 172.21.15.69:38962) 2024-04-24T08:17:53.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.811+0000 7f68e7e00640 1 -- 172.21.15.69:0/1314110613 learned_addr learned my addr 172.21.15.69:0/1314110613 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:53.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.811+0000 7f68ed000640 1 --2- 172.21.15.69:0/1314110613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80d18c0 0x7f68e80d04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.811+0000 7f68e7e00640 1 -- 172.21.15.69:0/1314110613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80d18c0 msgr2=0x7f68e80d04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:53.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.811+0000 7f68e7e00640 1 --2- 172.21.15.69:0/1314110613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80d18c0 0x7f68e80d04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.811+0000 7f68e7e00640 1 -- 172.21.15.69:0/1314110613 >> v1:172.21.15.121:6789/0 conn(0x7f68e80d36a0 legacy=0x7f68e80d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:53.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.811+0000 7f68e7e00640 1 -- 172.21.15.69:0/1314110613 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e80d28e0 con 0x7f68e815ea60 2024-04-24T08:17:53.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.812+0000 7f68e7e00640 1 --2- 172.21.15.69:0/1314110613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e815ea60 0x7f68e80cff70 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f68d8009e50 tx=0x7f68d802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9243b6b4ee13d0e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:53.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.812+0000 7f68e6a00640 1 -- 172.21.15.69:0/1314110613 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d8036070 con 0x7f68e815ea60 2024-04-24T08:17:53.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.812+0000 7f68e6a00640 1 -- 172.21.15.69:0/1314110613 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68d8031600 con 0x7f68e815ea60 2024-04-24T08:17:53.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.812+0000 7f68e6a00640 1 -- 172.21.15.69:0/1314110613 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d80319d0 con 0x7f68e815ea60 2024-04-24T08:17:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.813+0000 7f68ef000640 1 -- 172.21.15.69:0/1314110613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e815ea60 msgr2=0x7f68e80cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.813+0000 7f68ef000640 1 --2- 172.21.15.69:0/1314110613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e815ea60 0x7f68e80cff70 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f68d8009e50 tx=0x7f68d802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.813+0000 7f68ef000640 1 -- 172.21.15.69:0/1314110613 shutdown_connections 2024-04-24T08:17:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.813+0000 7f68ef000640 1 --2- 172.21.15.69:0/1314110613 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e80d18c0 0x7f68e80d04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.813+0000 7f68ef000640 1 --2- 172.21.15.69:0/1314110613 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e815ea60 0x7f68e80cff70 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.813+0000 7f68ef000640 1 -- 172.21.15.69:0/1314110613 >> 172.21.15.69:0/1314110613 conn(0x7f68e808bf60 msgr2=0x7f68e80d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.813+0000 7f68ef000640 1 -- 172.21.15.69:0/1314110613 shutdown_connections 2024-04-24T08:17:53.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.813+0000 7f68ef000640 1 -- 172.21.15.69:0/1314110613 wait complete. 2024-04-24T08:17:53.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.815+0000 7f68ef000640 1 Processor -- start 2024-04-24T08:17:53.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.815+0000 7f68ef000640 1 -- start start 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.816+0000 7f68ef000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80d18c0 0x7f68e80caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.816+0000 7f68ef000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e80d36a0 0x7f68e814cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.816+0000 7f68ef000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e815ea60 0x7f68e80c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.816+0000 7f68ef000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f68e80d3180 con 0x7f68e80d36a0 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.816+0000 7f68ef000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f68e80d2ea0 con 0x7f68e80d18c0 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.816+0000 7f68ef000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f68e815ee40 con 0x7f68e815ea60 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.816+0000 7f68e7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e80d36a0 0x7f68e814cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.816+0000 7f68e7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e80d36a0 0x7f68e814cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38966/0 (socket says 172.21.15.69:38966) 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.816+0000 7f68e7400640 1 -- 172.21.15.69:0/2819449388 learned_addr learned my addr 172.21.15.69:0/2819449388 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.816+0000 7f68e7e00640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80d18c0 0x7f68e80caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68e7400640 1 -- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e815ea60 msgr2=0x7f68e80c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68ed000640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e815ea60 0x7f68e80c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68e7400640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e815ea60 0x7f68e80c8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68e7400640 1 -- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80d18c0 msgr2=0x7f68e80caad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68e7400640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80d18c0 0x7f68e80caad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68e7400640 1 -- 172.21.15.69:0/2819449388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e80c9a30 con 0x7f68e80d36a0 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68e7e00640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80d18c0 0x7f68e80caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68e7400640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e80d36a0 0x7f68e814cc70 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f68d000c460 tx=0x7f68d000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68dfe00640 1 -- 172.21.15.69:0/2819449388 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0016020 con 0x7f68e80d36a0 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68dfe00640 1 -- 172.21.15.69:0/2819449388 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68d0012870 con 0x7f68e80d36a0 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68dfe00640 1 -- 172.21.15.69:0/2819449388 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d0012c60 con 0x7f68e80d36a0 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68ef000640 1 -- 172.21.15.69:0/2819449388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e80c9d10 con 0x7f68e80d36a0 2024-04-24T08:17:53.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.817+0000 7f68ef000640 1 -- 172.21.15.69:0/2819449388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e80ca2d0 con 0x7f68e80d36a0 2024-04-24T08:17:53.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.818+0000 7f68ef000640 1 -- 172.21.15.69:0/2819449388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e8068a80 con 0x7f68e80d36a0 2024-04-24T08:17:53.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.819+0000 7f68dfe00640 1 -- 172.21.15.69:0/2819449388 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f68d0019020 con 0x7f68e80d36a0 2024-04-24T08:17:53.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.820+0000 7f68dfe00640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f68c406c8f0 0x7f68c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:53.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.820+0000 7f68dfe00640 1 -- 172.21.15.69:0/2819449388 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f68d008bcd0 con 0x7f68e80d36a0 2024-04-24T08:17:53.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.821+0000 7f68e7e00640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f68c406c8f0 0x7f68c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:53.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.822+0000 7f68e7e00640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f68c406c8f0 0x7f68c406edb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f68d8033000 tx=0x7f68d8009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:53.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:53.825+0000 7f68dfe00640 1 -- 172.21.15.69:0/2819449388 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68d0020020 con 0x7f68e80d36a0 2024-04-24T08:17:54.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.023+0000 7f68ef000640 1 -- 172.21.15.69:0/2819449388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f68e80d1f70 con 0x7f68e80d36a0 2024-04-24T08:17:54.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.024+0000 7f68dfe00640 1 -- 172.21.15.69:0/2819449388 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) ==== 93+0+10 (secure 0 0 0) 0x7f68d00182c0 con 0x7f68e80d36a0 2024-04-24T08:17:54.025 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:17:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.026+0000 7f68ef000640 1 -- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f68c406c8f0 msgr2=0x7f68c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.026+0000 7f68ef000640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f68c406c8f0 0x7f68c406edb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f68d8033000 tx=0x7f68d8009000 comp rx=0 tx=0).stop 2024-04-24T08:17:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.027+0000 7f68ef000640 1 -- 172.21.15.69:0/2819449388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e80d36a0 msgr2=0x7f68e814cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.027+0000 7f68ef000640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e80d36a0 0x7f68e814cc70 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f68d000c460 tx=0x7f68d000c920 comp rx=0 tx=0).stop 2024-04-24T08:17:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.027+0000 7f68ef000640 1 -- 172.21.15.69:0/2819449388 shutdown_connections 2024-04-24T08:17:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.027+0000 7f68ef000640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f68c406c8f0 0x7f68c406edb0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.027+0000 7f68ef000640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68e80d36a0 0x7f68e814cc70 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.027+0000 7f68ef000640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68e815ea60 0x7f68e80c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.027+0000 7f68ef000640 1 --2- 172.21.15.69:0/2819449388 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e80d18c0 0x7f68e80caad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.027+0000 7f68ef000640 1 -- 172.21.15.69:0/2819449388 >> 172.21.15.69:0/2819449388 conn(0x7f68e808bf60 msgr2=0x7f68e80b4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.027+0000 7f68ef000640 1 -- 172.21.15.69:0/2819449388 shutdown_connections 2024-04-24T08:17:54.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.027+0000 7f68ef000640 1 -- 172.21.15.69:0/2819449388 wait complete. 2024-04-24T08:17:54.047 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:17:54.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.241+0000 7f7a5d800640 1 Processor -- start 2024-04-24T08:17:54.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.241+0000 7f7a5d800640 1 -- start start 2024-04-24T08:17:54.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.242+0000 7f7a5d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a58155f20 0x7f7a580ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.242+0000 7f7a5d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a580cf360 0x7f7a580cf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.242+0000 7f7a5d800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a580cd6d0 con 0x7f7a5815c9b0 2024-04-24T08:17:54.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.242+0000 7f7a5d800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7a5806db90 con 0x7f7a580cf360 2024-04-24T08:17:54.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.242+0000 7f7a5d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a58072070 con 0x7f7a58155f20 2024-04-24T08:17:54.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.243+0000 7f7a57400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a580cf360 0x7f7a580cf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.243+0000 7f7a56000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7a5815c9b0 0x7f7a580b3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39584/0 (socket says 172.21.15.69:39584) 2024-04-24T08:17:54.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.243+0000 7f7a57400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7a580cf360 0x7f7a580cf7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38972/0 (socket says 172.21.15.69:38972) 2024-04-24T08:17:54.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.243+0000 7f7a56000640 1 -- 172.21.15.69:0/1742861677 learned_addr learned my addr 172.21.15.69:0/1742861677 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:54.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.243+0000 7f7a56a00640 1 --2- 172.21.15.69:0/1742861677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a58155f20 0x7f7a580ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.243+0000 7f7a57400640 1 -- 172.21.15.69:0/1742861677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a58155f20 msgr2=0x7f7a580ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.243+0000 7f7a57400640 1 --2- 172.21.15.69:0/1742861677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a58155f20 0x7f7a580ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.243+0000 7f7a57400640 1 -- 172.21.15.69:0/1742861677 >> v1:172.21.15.69:6789/0 conn(0x7f7a5815c9b0 legacy=0x7f7a580b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.243+0000 7f7a57400640 1 -- 172.21.15.69:0/1742861677 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a580cd9b0 con 0x7f7a580cf360 2024-04-24T08:17:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.243+0000 7f7a55600640 1 -- 172.21.15.69:0/1742861677 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1430876893 0 0) 0x7f7a580cd6d0 con 0x7f7a5815c9b0 2024-04-24T08:17:54.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.244+0000 7f7a57400640 1 --2- 172.21.15.69:0/1742861677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a580cf360 0x7f7a580cf7e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7a500097a0 tx=0x7f7a5002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20b9a4e2753ec7d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:54.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.245+0000 7f7a55600640 1 -- 172.21.15.69:0/1742861677 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50037070 con 0x7f7a580cf360 2024-04-24T08:17:54.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.245+0000 7f7a55600640 1 -- 172.21.15.69:0/1742861677 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a50032440 con 0x7f7a580cf360 2024-04-24T08:17:54.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.245+0000 7f7a55600640 1 -- 172.21.15.69:0/1742861677 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a50032810 con 0x7f7a580cf360 2024-04-24T08:17:54.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.245+0000 7f7a5d800640 1 -- 172.21.15.69:0/1742861677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a580cf360 msgr2=0x7f7a580cf7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.245+0000 7f7a5d800640 1 --2- 172.21.15.69:0/1742861677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a580cf360 0x7f7a580cf7e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7a500097a0 tx=0x7f7a5002a210 comp rx=0 tx=0).stop 2024-04-24T08:17:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.245+0000 7f7a5d800640 1 -- 172.21.15.69:0/1742861677 shutdown_connections 2024-04-24T08:17:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.245+0000 7f7a5d800640 1 --2- 172.21.15.69:0/1742861677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a580cf360 0x7f7a580cf7e0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.245+0000 7f7a5d800640 1 --2- 172.21.15.69:0/1742861677 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a58155f20 0x7f7a580ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.245+0000 7f7a5d800640 1 -- 172.21.15.69:0/1742861677 >> 172.21.15.69:0/1742861677 conn(0x7f7a580d0350 msgr2=0x7f7a580d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.246+0000 7f7a5d800640 1 -- 172.21.15.69:0/1742861677 shutdown_connections 2024-04-24T08:17:54.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.246+0000 7f7a5d800640 1 -- 172.21.15.69:0/1742861677 wait complete. 2024-04-24T08:17:54.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.247+0000 7f7a5d800640 1 Processor -- start 2024-04-24T08:17:54.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.247+0000 7f7a5d800640 1 -- start start 2024-04-24T08:17:54.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.248+0000 7f7a5d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a580cf360 0x7f7a58150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.248+0000 7f7a5d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a58155f20 0x7f7a5814d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.248+0000 7f7a5d800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a5815c9b0 0x7f7a58154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.248+0000 7f7a5d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a580b4140 con 0x7f7a58155f20 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.248+0000 7f7a5d800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58071d00 con 0x7f7a5815c9b0 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.248+0000 7f7a5d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7a580cd2a0 con 0x7f7a580cf360 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a58155f20 0x7f7a5814d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a58155f20 0x7f7a5814d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38974/0 (socket says 172.21.15.69:38974) 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56000640 1 -- 172.21.15.69:0/1506940120 learned_addr learned my addr 172.21.15.69:0/1506940120 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56a00640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a580cf360 0x7f7a58150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a57400640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a5815c9b0 0x7f7a58154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56000640 1 -- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a580cf360 msgr2=0x7f7a58150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56000640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a580cf360 0x7f7a58150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56000640 1 -- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a5815c9b0 msgr2=0x7f7a58154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56000640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a5815c9b0 0x7f7a58154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56000640 1 -- 172.21.15.69:0/1506940120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a5815cfa0 con 0x7f7a58155f20 2024-04-24T08:17:54.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a57400640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a5815c9b0 0x7f7a58154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:54.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56a00640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a580cf360 0x7f7a58150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:54.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.249+0000 7f7a56000640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a58155f20 0x7f7a5814d7c0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400c440 tx=0x7f7a4400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:54.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.250+0000 7f7a4ea00640 1 -- 172.21.15.69:0/1506940120 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a44018070 con 0x7f7a58155f20 2024-04-24T08:17:54.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.250+0000 7f7a4ea00640 1 -- 172.21.15.69:0/1506940120 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a440074e0 con 0x7f7a58155f20 2024-04-24T08:17:54.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.250+0000 7f7a4ea00640 1 -- 172.21.15.69:0/1506940120 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a44004a20 con 0x7f7a58155f20 2024-04-24T08:17:54.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.250+0000 7f7a5d800640 1 -- 172.21.15.69:0/1506940120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a5815d280 con 0x7f7a58155f20 2024-04-24T08:17:54.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.250+0000 7f7a5d800640 1 -- 172.21.15.69:0/1506940120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a5815d840 con 0x7f7a58155f20 2024-04-24T08:17:54.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.251+0000 7f7a5d800640 1 -- 172.21.15.69:0/1506940120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a58068a80 con 0x7f7a58155f20 2024-04-24T08:17:54.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.252+0000 7f7a4ea00640 1 -- 172.21.15.69:0/1506940120 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7a44004bc0 con 0x7f7a58155f20 2024-04-24T08:17:54.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.253+0000 7f7a4ea00640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a3806c920 0x7f7a3806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.253+0000 7f7a4ea00640 1 -- 172.21.15.69:0/1506940120 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f7a4408aeb0 con 0x7f7a58155f20 2024-04-24T08:17:54.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.254+0000 7f7a56a00640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a3806c920 0x7f7a3806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.255+0000 7f7a56a00640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a3806c920 0x7f7a3806ede0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7a48007330 tx=0x7f7a4800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:54.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.258+0000 7f7a4ea00640 1 -- 172.21.15.69:0/1506940120 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7a44059200 con 0x7f7a58155f20 2024-04-24T08:17:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.455+0000 7f7a5d800640 1 -- 172.21.15.69:0/1506940120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7a580cd030 con 0x7f7a58155f20 2024-04-24T08:17:54.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.456+0000 7f7a4ea00640 1 -- 172.21.15.69:0/1506940120 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) ==== 104+0+11 (secure 0 0 0) 0x7f7a44058b00 con 0x7f7a58155f20 2024-04-24T08:17:54.457 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:17:54.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.458+0000 7f7a5d800640 1 -- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a3806c920 msgr2=0x7f7a3806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.458+0000 7f7a5d800640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a3806c920 0x7f7a3806ede0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7a48007330 tx=0x7f7a4800b040 comp rx=0 tx=0).stop 2024-04-24T08:17:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.458+0000 7f7a5d800640 1 -- 172.21.15.69:0/1506940120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a58155f20 msgr2=0x7f7a5814d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.458+0000 7f7a5d800640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a58155f20 0x7f7a5814d7c0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7a4400c440 tx=0x7f7a4400c900 comp rx=0 tx=0).stop 2024-04-24T08:17:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.458+0000 7f7a5d800640 1 -- 172.21.15.69:0/1506940120 shutdown_connections 2024-04-24T08:17:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.458+0000 7f7a5d800640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7a3806c920 0x7f7a3806ede0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.458+0000 7f7a5d800640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a58155f20 0x7f7a5814d7c0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.459+0000 7f7a5d800640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a5815c9b0 0x7f7a58154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.459+0000 7f7a5d800640 1 --2- 172.21.15.69:0/1506940120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7a580cf360 0x7f7a58150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.459+0000 7f7a5d800640 1 -- 172.21.15.69:0/1506940120 >> 172.21.15.69:0/1506940120 conn(0x7f7a580d0350 msgr2=0x7f7a5806f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.459+0000 7f7a5d800640 1 -- 172.21.15.69:0/1506940120 shutdown_connections 2024-04-24T08:17:54.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.459+0000 7f7a5d800640 1 -- 172.21.15.69:0/1506940120 wait complete. 2024-04-24T08:17:54.479 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:17:54.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.677+0000 7f1949200640 1 Processor -- start 2024-04-24T08:17:54.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.677+0000 7f1949200640 1 -- start start 2024-04-24T08:17:54.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.679+0000 7f1949200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19440b3e90 0x7f19440b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.679+0000 7f1949200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f194415cc00 con 0x7f194415c820 2024-04-24T08:17:54.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.679+0000 7f1949200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f194406ed30 con 0x7f19440ce250 2024-04-24T08:17:54.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.679+0000 7f1949200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f19440b4be0 con 0x7f19440b3e90 2024-04-24T08:17:54.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.679+0000 7f193be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19440b3e90 0x7f19440b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.679+0000 7f193be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19440b3e90 0x7f19440b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38990/0 (socket says 172.21.15.69:38990) 2024-04-24T08:17:54.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.679+0000 7f193be00640 1 -- 172.21.15.69:0/1341979225 learned_addr learned my addr 172.21.15.69:0/1341979225 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:54.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.680+0000 7f193be00640 1 -- 172.21.15.69:0/1341979225 >> v1:172.21.15.121:6789/0 conn(0x7f19440ce250 legacy=0x7f19440ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:54.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.680+0000 7f193be00640 1 -- 172.21.15.69:0/1341979225 >> v1:172.21.15.69:6789/0 conn(0x7f194415c820 legacy=0x7f194406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.680+0000 7f193be00640 1 -- 172.21.15.69:0/1341979225 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19440b4e10 con 0x7f19440b3e90 2024-04-24T08:17:54.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.681+0000 7f193be00640 1 --2- 172.21.15.69:0/1341979225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 0x7f19440b46a0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f194415cdd0 tx=0x7f192c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b55f9784586aac server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:54.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.681+0000 7f1941400640 1 -- 172.21.15.69:0/1341979225 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c037070 con 0x7f19440b3e90 2024-04-24T08:17:54.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.681+0000 7f1941400640 1 -- 172.21.15.69:0/1341979225 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f192c032440 con 0x7f19440b3e90 2024-04-24T08:17:54.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.681+0000 7f1941400640 1 -- 172.21.15.69:0/1341979225 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c032810 con 0x7f19440b3e90 2024-04-24T08:17:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.681+0000 7f1949200640 1 -- 172.21.15.69:0/1341979225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 msgr2=0x7f19440b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.681+0000 7f1949200640 1 --2- 172.21.15.69:0/1341979225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 0x7f19440b46a0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f194415cdd0 tx=0x7f192c02a2a0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.682+0000 7f1949200640 1 -- 172.21.15.69:0/1341979225 shutdown_connections 2024-04-24T08:17:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.682+0000 7f1949200640 1 --2- 172.21.15.69:0/1341979225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 0x7f19440b46a0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.682+0000 7f1949200640 1 -- 172.21.15.69:0/1341979225 >> 172.21.15.69:0/1341979225 conn(0x7f19440d0350 msgr2=0x7f19440d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.682+0000 7f1949200640 1 -- 172.21.15.69:0/1341979225 shutdown_connections 2024-04-24T08:17:54.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.682+0000 7f1949200640 1 -- 172.21.15.69:0/1341979225 wait complete. 2024-04-24T08:17:54.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.683+0000 7f1949200640 1 Processor -- start 2024-04-24T08:17:54.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.683+0000 7f1949200640 1 -- start start 2024-04-24T08:17:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.684+0000 7f1949200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 0x7f1944155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.684+0000 7f1949200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19440ce250 0x7f19441e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.684+0000 7f1949200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f194415c820 0x7f19441601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.684+0000 7f1949200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f194406f2b0 con 0x7f19440b3e90 2024-04-24T08:17:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.684+0000 7f1949200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f194406efd0 con 0x7f194415c820 2024-04-24T08:17:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.684+0000 7f1949200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f194406fee0 con 0x7f19440ce250 2024-04-24T08:17:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1941e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 0x7f1944155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1941e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 0x7f1944155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38996/0 (socket says 172.21.15.69:38996) 2024-04-24T08:17:54.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1941e00640 1 -- 172.21.15.69:0/229118207 learned_addr learned my addr 172.21.15.69:0/229118207 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:54.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1942800640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f194415c820 0x7f19441601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f193be00640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19440ce250 0x7f19441e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1941e00640 1 -- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19440ce250 msgr2=0x7f19441e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1941e00640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19440ce250 0x7f19441e6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1941e00640 1 -- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f194415c820 msgr2=0x7f19441601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1941e00640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f194415c820 0x7f19441601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1941e00640 1 -- 172.21.15.69:0/229118207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19441e76d0 con 0x7f19440b3e90 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1942800640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f194415c820 0x7f19441601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.685+0000 7f1941e00640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 0x7f1944155240 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f193400c3c0 tx=0x7f193400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.686+0000 7f193a000640 1 -- 172.21.15.69:0/229118207 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19340052d0 con 0x7f19440b3e90 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.686+0000 7f193a000640 1 -- 172.21.15.69:0/229118207 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1934005490 con 0x7f19440b3e90 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.686+0000 7f193a000640 1 -- 172.21.15.69:0/229118207 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1934007260 con 0x7f19440b3e90 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.686+0000 7f1949200640 1 -- 172.21.15.69:0/229118207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19441e79b0 con 0x7f19440b3e90 2024-04-24T08:17:54.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.686+0000 7f1949200640 1 -- 172.21.15.69:0/229118207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1944155c80 con 0x7f19440b3e90 2024-04-24T08:17:54.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.687+0000 7f1949200640 1 -- 172.21.15.69:0/229118207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1944068a80 con 0x7f19440b3e90 2024-04-24T08:17:54.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.688+0000 7f193a000640 1 -- 172.21.15.69:0/229118207 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1934007400 con 0x7f19440b3e90 2024-04-24T08:17:54.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.689+0000 7f193a000640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192006c8f0 0x7f192006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:54.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.689+0000 7f193a000640 1 -- 172.21.15.69:0/229118207 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f193408b1e0 con 0x7f19440b3e90 2024-04-24T08:17:54.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.689+0000 7f193be00640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192006c8f0 0x7f192006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:54.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.691+0000 7f193be00640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192006c8f0 0x7f192006edb0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f192c034000 tx=0x7f192c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:54.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.693+0000 7f193a000640 1 -- 172.21.15.69:0/229118207 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f19340594a0 con 0x7f19440b3e90 2024-04-24T08:17:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.893+0000 7f1949200640 1 -- 172.21.15.69:0/229118207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f19440cddc0 con 0x7f19440b3e90 2024-04-24T08:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.894+0000 7f193a000640 1 -- 172.21.15.69:0/229118207 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) ==== 100+0+11 (secure 0 0 0) 0x7f1934058da0 con 0x7f19440b3e90 2024-04-24T08:17:54.895 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:17:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.896+0000 7f1949200640 1 -- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192006c8f0 msgr2=0x7f192006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.896+0000 7f1949200640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192006c8f0 0x7f192006edb0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f192c034000 tx=0x7f192c02e000 comp rx=0 tx=0).stop 2024-04-24T08:17:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.896+0000 7f1949200640 1 -- 172.21.15.69:0/229118207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 msgr2=0x7f1944155240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.896+0000 7f1949200640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 0x7f1944155240 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f193400c3c0 tx=0x7f193400c880 comp rx=0 tx=0).stop 2024-04-24T08:17:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.896+0000 7f1949200640 1 -- 172.21.15.69:0/229118207 shutdown_connections 2024-04-24T08:17:54.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.896+0000 7f1949200640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f192006c8f0 0x7f192006edb0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.896+0000 7f1949200640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f194415c820 0x7f19441601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.896+0000 7f1949200640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19440ce250 0x7f19441e6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.896+0000 7f1949200640 1 --2- 172.21.15.69:0/229118207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19440b3e90 0x7f1944155240 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.896+0000 7f1949200640 1 -- 172.21.15.69:0/229118207 >> 172.21.15.69:0/229118207 conn(0x7f19440d0350 msgr2=0x7f19440b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.897+0000 7f1949200640 1 -- 172.21.15.69:0/229118207 shutdown_connections 2024-04-24T08:17:54.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:54.897+0000 7f1949200640 1 -- 172.21.15.69:0/229118207 wait complete. 2024-04-24T08:17:54.918 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.120+0000 7ff73cc00640 1 Processor -- start 2024-04-24T08:17:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.120+0000 7ff73cc00640 1 -- start start 2024-04-24T08:17:55.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.121+0000 7ff73cc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff7380b30f0 0x7ff738167470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.121+0000 7ff73cc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff73815d660 con 0x7ff73815d1d0 2024-04-24T08:17:55.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.122+0000 7ff73cc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff73806dc30 con 0x7ff738156c60 2024-04-24T08:17:55.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.122+0000 7ff73cc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff73806df10 con 0x7ff7380b30f0 2024-04-24T08:17:55.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.122+0000 7ff736a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff73815d1d0 0x7ff7380ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39606/0 (socket says 172.21.15.69:39606) 2024-04-24T08:17:55.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.122+0000 7ff736a00640 1 -- 172.21.15.69:0/455170729 learned_addr learned my addr 172.21.15.69:0/455170729 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.122+0000 7ff737e00640 1 --2- 172.21.15.69:0/455170729 >> v2:172.21.15.69:3300/0 conn(0x7ff7380b30f0 0x7ff738167470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.122+0000 7ff736000640 1 -- 172.21.15.69:0/455170729 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2836062630 0 0) 0x7ff73815d660 con 0x7ff73815d1d0 2024-04-24T08:17:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.122+0000 7ff736000640 1 -- 172.21.15.69:0/455170729 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff718003520 con 0x7ff73815d1d0 2024-04-24T08:17:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.122+0000 7ff737e00640 1 -- 172.21.15.69:0/455170729 >> v1:172.21.15.121:6789/0 conn(0x7ff738156c60 legacy=0x7ff73815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.122+0000 7ff737e00640 1 -- 172.21.15.69:0/455170729 >> v1:172.21.15.69:6789/0 conn(0x7ff73815d1d0 legacy=0x7ff7380ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.123+0000 7ff737e00640 1 -- 172.21.15.69:0/455170729 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73806e1f0 con 0x7ff7380b30f0 2024-04-24T08:17:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.124+0000 7ff737e00640 1 --2- 172.21.15.69:0/455170729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7380b30f0 0x7ff738167470 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7ff7280097a0 tx=0x7ff72802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be76c742564f7148 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.124+0000 7ff736000640 1 -- 172.21.15.69:0/455170729 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff728037070 con 0x7ff7380b30f0 2024-04-24T08:17:55.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.124+0000 7ff736000640 1 -- 172.21.15.69:0/455170729 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff728032440 con 0x7ff7380b30f0 2024-04-24T08:17:55.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.124+0000 7ff736000640 1 -- 172.21.15.69:0/455170729 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7280327f0 con 0x7ff7380b30f0 2024-04-24T08:17:55.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.125+0000 7ff73cc00640 1 -- 172.21.15.69:0/455170729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7380b30f0 msgr2=0x7ff738167470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.125+0000 7ff73cc00640 1 --2- 172.21.15.69:0/455170729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7380b30f0 0x7ff738167470 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7ff7280097a0 tx=0x7ff72802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.125+0000 7ff73cc00640 1 -- 172.21.15.69:0/455170729 shutdown_connections 2024-04-24T08:17:55.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.125+0000 7ff73cc00640 1 --2- 172.21.15.69:0/455170729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7380b30f0 0x7ff738167470 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.125+0000 7ff73cc00640 1 -- 172.21.15.69:0/455170729 >> 172.21.15.69:0/455170729 conn(0x7ff7380d0160 msgr2=0x7ff7380d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:55.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.125+0000 7ff73cc00640 1 -- 172.21.15.69:0/455170729 shutdown_connections 2024-04-24T08:17:55.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.125+0000 7ff73cc00640 1 -- 172.21.15.69:0/455170729 wait complete. 2024-04-24T08:17:55.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.126+0000 7ff73cc00640 1 Processor -- start 2024-04-24T08:17:55.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.127+0000 7ff73cc00640 1 -- start start 2024-04-24T08:17:55.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.127+0000 7ff73cc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7380b30f0 0x7ff7380c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff73cc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff738156c60 0x7ff73814b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff73cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7381f41c0 0x7ff738073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff73cc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff7380ad7e0 con 0x7ff7381f41c0 2024-04-24T08:17:55.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff73cc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff73815d940 con 0x7ff738156c60 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff73cc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff7380ad960 con 0x7ff7380b30f0 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff737e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7381f41c0 0x7ff738073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff737e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7381f41c0 0x7ff738073060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39008/0 (socket says 172.21.15.69:39008) 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff737e00640 1 -- 172.21.15.69:0/1634919346 learned_addr learned my addr 172.21.15.69:0/1634919346 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff737e00640 1 -- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7380b30f0 msgr2=0x7ff7380c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff737e00640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7380b30f0 0x7ff7380c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff737e00640 1 -- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff738156c60 msgr2=0x7ff73814b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff737e00640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff738156c60 0x7ff73814b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff737e00640 1 -- 172.21.15.69:0/1634919346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff73814bac0 con 0x7ff7381f41c0 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.128+0000 7ff737e00640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7381f41c0 0x7ff738073060 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7ff7280097a0 tx=0x7ff7280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.129+0000 7ff72f400640 1 -- 172.21.15.69:0/1634919346 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff728037040 con 0x7ff7381f41c0 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.129+0000 7ff72f400640 1 -- 172.21.15.69:0/1634919346 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff728042070 con 0x7ff7381f41c0 2024-04-24T08:17:55.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.129+0000 7ff72f400640 1 -- 172.21.15.69:0/1634919346 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff728030940 con 0x7ff7381f41c0 2024-04-24T08:17:55.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.129+0000 7ff73cc00640 1 -- 172.21.15.69:0/1634919346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff738071fc0 con 0x7ff7381f41c0 2024-04-24T08:17:55.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.129+0000 7ff73cc00640 1 -- 172.21.15.69:0/1634919346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff738072580 con 0x7ff7381f41c0 2024-04-24T08:17:55.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.130+0000 7ff73cc00640 1 -- 172.21.15.69:0/1634919346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff738075d50 con 0x7ff7381f41c0 2024-04-24T08:17:55.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.131+0000 7ff72f400640 1 -- 172.21.15.69:0/1634919346 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff728030ae0 con 0x7ff7381f41c0 2024-04-24T08:17:55.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.132+0000 7ff72f400640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff71406c8f0 0x7ff71406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.132+0000 7ff72f400640 1 -- 172.21.15.69:0/1634919346 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7ff7280ac570 con 0x7ff7381f41c0 2024-04-24T08:17:55.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.132+0000 7ff737400640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff71406c8f0 0x7ff71406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:55.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.134+0000 7ff737400640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff71406c8f0 0x7ff71406edb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff7300043c0 tx=0x7ff730007d30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:55.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.136+0000 7ff72f400640 1 -- 172.21.15.69:0/1634919346 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff728045360 con 0x7ff7381f41c0 2024-04-24T08:17:55.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.386+0000 7ff73cc00640 1 -- 172.21.15.69:0/1634919346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7380b3f40 con 0x7ff7381f41c0 2024-04-24T08:17:55.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.387+0000 7ff72f400640 1 -- 172.21.15.69:0/1634919346 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7ff72807a8c0 con 0x7ff7381f41c0 2024-04-24T08:17:55.389 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:55.389 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:17:13.584835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8151":{"gid":8151,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6833/4254348458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4254348458},{"type":"v1","addr":"172.21.15.69:6833","nonce":4254348458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8151,"qdb_cluster":[8151]},"id":6}]} 2024-04-24T08:17:55.389 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 74 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 -- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff71406c8f0 msgr2=0x7ff71406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff71406c8f0 0x7ff71406edb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff7300043c0 tx=0x7ff730007d30 comp rx=0 tx=0).stop 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 -- 172.21.15.69:0/1634919346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7381f41c0 msgr2=0x7ff738073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7381f41c0 0x7ff738073060 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7ff7280097a0 tx=0x7ff7280046c0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 -- 172.21.15.69:0/1634919346 shutdown_connections 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff71406c8f0 0x7ff71406edb0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7381f41c0 0x7ff738073060 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff738156c60 0x7ff73814b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 --2- 172.21.15.69:0/1634919346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7380b30f0 0x7ff7380c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 -- 172.21.15.69:0/1634919346 >> 172.21.15.69:0/1634919346 conn(0x7ff7380d0160 msgr2=0x7ff73815c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 -- 172.21.15.69:0/1634919346 shutdown_connections 2024-04-24T08:17:55.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.390+0000 7ff73cc00640 1 -- 172.21.15.69:0/1634919346 wait complete. 2024-04-24T08:17:55.411 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:17:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.595+0000 7fccf9000640 1 Processor -- start 2024-04-24T08:17:55.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.595+0000 7fccf9000640 1 -- start start 2024-04-24T08:17:55.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.597+0000 7fccf9000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145a90 0x7fccf406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.597+0000 7fccf9000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fccf4145fe0 0x7fccf4160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.597+0000 7fccf9000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf40cf3d0 con 0x7fccf40d0300 2024-04-24T08:17:55.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.597+0000 7fccf9000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fccf40cf6b0 con 0x7fccf4145fe0 2024-04-24T08:17:55.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.597+0000 7fccf9000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fccf40cf990 con 0x7fccf4145a90 2024-04-24T08:17:55.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.598+0000 7fccf2800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fccf4145fe0 0x7fccf4160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:55.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.598+0000 7fcce9400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145a90 0x7fccf406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.598+0000 7fccf2800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fccf4145fe0 0x7fccf4160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:54244/0 (socket says 172.21.15.69:54244) 2024-04-24T08:17:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.598+0000 7fccf2800640 1 -- 172.21.15.69:0/3330479852 learned_addr learned my addr 172.21.15.69:0/3330479852 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.599+0000 7fccf2800640 1 -- 172.21.15.69:0/3330479852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145a90 msgr2=0x7fccf406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.599+0000 7fccf2800640 1 --2- 172.21.15.69:0/3330479852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145a90 0x7fccf406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.599+0000 7fccf2800640 1 -- 172.21.15.69:0/3330479852 >> v1:172.21.15.121:6789/0 conn(0x7fccf40d0300 legacy=0x7fccf40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.599+0000 7fccf2800640 1 -- 172.21.15.69:0/3330479852 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf40cfc70 con 0x7fccf4145fe0 2024-04-24T08:17:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.599+0000 7fcce9400640 1 --2- 172.21.15.69:0/3330479852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145a90 0x7fccf406e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:55.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.600+0000 7fccf2800640 1 --2- 172.21.15.69:0/3330479852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf4145fe0 0x7fccf4160540 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fccec009e50 tx=0x7fccec02a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18cc6254c16646a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:55.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.600+0000 7fccf1400640 1 -- 172.21.15.69:0/3330479852 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec036070 con 0x7fccf4145fe0 2024-04-24T08:17:55.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.601+0000 7fccf1400640 1 -- 172.21.15.69:0/3330479852 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccec031600 con 0x7fccf4145fe0 2024-04-24T08:17:55.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.601+0000 7fccf1400640 1 -- 172.21.15.69:0/3330479852 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec0319d0 con 0x7fccf4145fe0 2024-04-24T08:17:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.601+0000 7fccf9000640 1 -- 172.21.15.69:0/3330479852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf4145fe0 msgr2=0x7fccf4160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.601+0000 7fccf9000640 1 --2- 172.21.15.69:0/3330479852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf4145fe0 0x7fccf4160540 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fccec009e50 tx=0x7fccec02a730 comp rx=0 tx=0).stop 2024-04-24T08:17:55.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.601+0000 7fccf9000640 1 -- 172.21.15.69:0/3330479852 shutdown_connections 2024-04-24T08:17:55.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.601+0000 7fccf9000640 1 --2- 172.21.15.69:0/3330479852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf4145fe0 0x7fccf4160540 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.601+0000 7fccf9000640 1 --2- 172.21.15.69:0/3330479852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145a90 0x7fccf406e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.601+0000 7fccf9000640 1 -- 172.21.15.69:0/3330479852 >> 172.21.15.69:0/3330479852 conn(0x7fccf4067d90 msgr2=0x7fccf406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:55.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.601+0000 7fccf9000640 1 -- 172.21.15.69:0/3330479852 shutdown_connections 2024-04-24T08:17:55.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.602+0000 7fccf9000640 1 -- 172.21.15.69:0/3330479852 wait complete. 2024-04-24T08:17:55.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.603+0000 7fccf9000640 1 Processor -- start 2024-04-24T08:17:55.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.604+0000 7fccf9000640 1 -- start start 2024-04-24T08:17:55.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fccf9000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf40d0300 0x7fccf40ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fccf9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccf4145a90 0x7fccf40add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fccf9000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145fe0 0x7fccf41f4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fccf9000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fccf41458f0 con 0x7fccf4145a90 2024-04-24T08:17:55.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fccf9000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fccf4145770 con 0x7fccf40d0300 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fccf9000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fccf40cdc30 con 0x7fccf4145fe0 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fcce9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccf4145a90 0x7fccf40add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fcce9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccf4145a90 0x7fccf40add80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39010/0 (socket says 172.21.15.69:39010) 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fcce9400640 1 -- 172.21.15.69:0/2180601394 learned_addr learned my addr 172.21.15.69:0/2180601394 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fccf2800640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145fe0 0x7fccf41f4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.605+0000 7fccf1e00640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf40d0300 0x7fccf40ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fcce9400640 1 -- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145fe0 msgr2=0x7fccf41f4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fcce9400640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145fe0 0x7fccf41f4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fcce9400640 1 -- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf40d0300 msgr2=0x7fccf40ad840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fcce9400640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf40d0300 0x7fccf40ad840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fcce9400640 1 -- 172.21.15.69:0/2180601394 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf40ae2c0 con 0x7fccf4145a90 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fccf2800640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145fe0 0x7fccf41f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:55.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fcce9400640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccf4145a90 0x7fccf40add80 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00e960 tx=0x7fccdc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:55.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fcceaa00640 1 -- 172.21.15.69:0/2180601394 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc017070 con 0x7fccf4145a90 2024-04-24T08:17:55.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fcceaa00640 1 -- 172.21.15.69:0/2180601394 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccdc00ca00 con 0x7fccf4145a90 2024-04-24T08:17:55.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fcceaa00640 1 -- 172.21.15.69:0/2180601394 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccdc00cdd0 con 0x7fccf4145a90 2024-04-24T08:17:55.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fccf9000640 1 -- 172.21.15.69:0/2180601394 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf40ae570 con 0x7fccf4145a90 2024-04-24T08:17:55.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.606+0000 7fccf9000640 1 -- 172.21.15.69:0/2180601394 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf40aeb30 con 0x7fccf4145a90 2024-04-24T08:17:55.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.608+0000 7fccf9000640 1 -- 172.21.15.69:0/2180601394 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf4068a80 con 0x7fccf4145a90 2024-04-24T08:17:55.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.608+0000 7fcceaa00640 1 -- 172.21.15.69:0/2180601394 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fccdc01a070 con 0x7fccf4145a90 2024-04-24T08:17:55.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.609+0000 7fcceaa00640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fccd006c8f0 0x7fccd006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:55.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.609+0000 7fcceaa00640 1 -- 172.21.15.69:0/2180601394 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7fccdc08bba0 con 0x7fccf4145a90 2024-04-24T08:17:55.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.610+0000 7fccf1e00640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fccd006c8f0 0x7fccd006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:55.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.611+0000 7fccf1e00640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fccd006c8f0 0x7fccd006edb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fcce4007fd0 tx=0x7fcce4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:55.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.614+0000 7fcceaa00640 1 -- 172.21.15.69:0/2180601394 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fccdc059e60 con 0x7fccf4145a90 2024-04-24T08:17:55.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.818+0000 7fccf9000640 1 -- 172.21.15.69:0/2180601394 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccf40d1120 con 0x7fccf4145a90 2024-04-24T08:17:55.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.821+0000 7fcceaa00640 1 -- 172.21.15.69:0/2180601394 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+16947 (secure 0 0 0) 0x7fccdc059760 con 0x7fccf4145a90 2024-04-24T08:17:55.823 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:55.823 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":86,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:17:08.931744+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6832/2529876410":"2024-04-25T08:17:08.931681+0000","172.21.15.121:6837/1179683245":"2024-04-25T08:15:45.259762+0000","172.21.15.69:6833/2529876410":"2024-04-25T08:17:08.931681+0000","172.21.15.121:6835/1179683245":"2024-04-25T08:15:45.259762+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:17:55.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.823+0000 7fccf9000640 1 -- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fccd006c8f0 msgr2=0x7fccd006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.823+0000 7fccf9000640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fccd006c8f0 0x7fccd006edb0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fcce4007fd0 tx=0x7fcce4009040 comp rx=0 tx=0).stop 2024-04-24T08:17:55.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.823+0000 7fccf9000640 1 -- 172.21.15.69:0/2180601394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccf4145a90 msgr2=0x7fccf40add80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:55.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.823+0000 7fccf9000640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccf4145a90 0x7fccf40add80 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00e960 tx=0x7fccdc00ee20 comp rx=0 tx=0).stop 2024-04-24T08:17:55.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.823+0000 7fccf9000640 1 -- 172.21.15.69:0/2180601394 shutdown_connections 2024-04-24T08:17:55.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.823+0000 7fccf9000640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fccd006c8f0 0x7fccd006edb0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.824+0000 7fccf9000640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccf4145a90 0x7fccf40add80 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.824+0000 7fccf9000640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fccf4145fe0 0x7fccf41f4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.824+0000 7fccf9000640 1 --2- 172.21.15.69:0/2180601394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf40d0300 0x7fccf40ad840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:55.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.824+0000 7fccf9000640 1 -- 172.21.15.69:0/2180601394 >> 172.21.15.69:0/2180601394 conn(0x7fccf4067d90 msgr2=0x7fccf40afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:55.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.824+0000 7fccf9000640 1 -- 172.21.15.69:0/2180601394 shutdown_connections 2024-04-24T08:17:55.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:55.824+0000 7fccf9000640 1 -- 172.21.15.69:0/2180601394 wait complete. 2024-04-24T08:17:55.844 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:17:55.844 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:56.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.044+0000 7f92b3000640 1 Processor -- start 2024-04-24T08:17:56.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.044+0000 7f92b3000640 1 -- start start 2024-04-24T08:17:56.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.045+0000 7f92b3000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92ac0d0300 0x7f92ac06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.045+0000 7f92b3000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f92ac145a90 0x7f92ac0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.045+0000 7f92b3000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92ac0d0c20 con 0x7f92ac145fe0 2024-04-24T08:17:56.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.045+0000 7f92b3000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f92ac0cf3d0 con 0x7f92ac145a90 2024-04-24T08:17:56.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.046+0000 7f92b3000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f92ac0cf6b0 con 0x7f92ac0d0300 2024-04-24T08:17:56.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.046+0000 7f92b1000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f92ac145fe0 0x7f92ac160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39608/0 (socket says 172.21.15.69:39608) 2024-04-24T08:17:56.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.046+0000 7f92b1000640 1 -- 172.21.15.69:0/2697751176 learned_addr learned my addr 172.21.15.69:0/2697751176 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:56.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.046+0000 7f92ab400640 1 --2- 172.21.15.69:0/2697751176 >> v2:172.21.15.121:3300/0 conn(0x7f92ac145a90 0x7f92ac0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.046+0000 7f92abe00640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92ac0d0300 0x7f92ac06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.046+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1410094033 0 0) 0x7f92ac0d0c20 con 0x7f92ac145fe0 2024-04-24T08:17:56.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.046+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9290003520 con 0x7f92ac145fe0 2024-04-24T08:17:56.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1713505657 0 0) 0x7f9290003520 con 0x7f92ac145fe0 2024-04-24T08:17:56.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92ac0d0300 msgr2=0x7f92ac06e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:56.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92ac0d0300 0x7f92ac06e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 >> v2:172.21.15.121:3300/0 conn(0x7f92ac145a90 msgr2=0x7f92ac0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:56.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 --2- 172.21.15.69:0/2697751176 >> v2:172.21.15.121:3300/0 conn(0x7f92ac145a90 0x7f92ac0d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ac0cf990 con 0x7f92ac145fe0 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f92a40019f0 con 0x7f92ac145fe0 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 >> v1:172.21.15.69:6789/0 conn(0x7f92ac145fe0 legacy=0x7f92ac160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92900039b0 0x7f9290003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9290004800 0x7f9290004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92ac0d0300 0x7f92900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac0cf3d0 con 0x7f92900039b0 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac0cf6b0 con 0x7f9290004800 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f929000b9c0 con 0x7f92ac0d0300 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.047+0000 7f92abe00640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92900039b0 0x7f9290003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.048+0000 7f92ab400640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9290004800 0x7f9290004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.048+0000 7f92b1000640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92ac0d0300 0x7f92900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.048+0000 7f92abe00640 1 -- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92ac0d0300 msgr2=0x7f92900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:56.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.048+0000 7f92abe00640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92ac0d0300 0x7f92900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.048+0000 7f92abe00640 1 -- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9290004800 msgr2=0x7f9290004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:56.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.048+0000 7f92abe00640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9290004800 0x7f9290004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.048+0000 7f92abe00640 1 -- 172.21.15.69:0/2697751176 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9290003520 con 0x7f92900039b0 2024-04-24T08:17:56.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.049+0000 7f92abe00640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92900039b0 0x7f9290003dd0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f929c009ac0 tx=0x7f9290005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac1b4927b66df99 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.049+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c036070 con 0x7f92900039b0 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.049+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f929c031400 con 0x7f92900039b0 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.049+0000 7f92aaa00640 1 -- 172.21.15.69:0/2697751176 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929c009690 con 0x7f92900039b0 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.050+0000 7f92b3000640 1 -- 172.21.15.69:0/2697751176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92900039b0 msgr2=0x7f9290003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.050+0000 7f92b3000640 1 --2- 172.21.15.69:0/2697751176 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92900039b0 0x7f9290003dd0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f929c009ac0 tx=0x7f9290005830 comp rx=0 tx=0).stop 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.050+0000 7f92b1000640 1 -- 172.21.15.69:0/2697751176 reap_dead start 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.050+0000 7f92b3000640 1 -- 172.21.15.69:0/2697751176 shutdown_connections 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.050+0000 7f92b3000640 1 -- 172.21.15.69:0/2697751176 >> 172.21.15.69:0/2697751176 conn(0x7f92ac067d90 msgr2=0x7f92ac06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.050+0000 7f92b3000640 1 -- 172.21.15.69:0/2697751176 shutdown_connections 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.050+0000 7f92b3000640 1 -- 172.21.15.69:0/2697751176 wait complete. 2024-04-24T08:17:56.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.052+0000 7f92b3000640 1 Processor -- start 2024-04-24T08:17:56.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.052+0000 7f92b3000640 1 -- start start 2024-04-24T08:17:56.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.053+0000 7f92b3000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92900039b0 0x7f92ac143630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.053+0000 7f92b3000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92ac0ca100 0x7f92ac0c7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.053+0000 7f92b3000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92ac0ca580 0x7f92ac0c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.053+0000 7f92b3000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac145770 con 0x7f92ac0ca100 2024-04-24T08:17:56.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.053+0000 7f92b3000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f92ac0cff50 con 0x7f92ac0ca580 2024-04-24T08:17:56.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.053+0000 7f92b3000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f92ac1458f0 con 0x7f92900039b0 2024-04-24T08:17:56.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.053+0000 7f92ab400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92ac0ca100 0x7f92ac0c7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.053+0000 7f92ab400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92ac0ca100 0x7f92ac0c7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39020/0 (socket says 172.21.15.69:39020) 2024-04-24T08:17:56.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.053+0000 7f92ab400640 1 -- 172.21.15.69:0/1070879687 learned_addr learned my addr 172.21.15.69:0/1070879687 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92abe00640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92900039b0 0x7f92ac143630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92b1000640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92ac0ca580 0x7f92ac0c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92ab400640 1 -- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92900039b0 msgr2=0x7f92ac143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92ab400640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92900039b0 0x7f92ac143630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92ab400640 1 -- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92ac0ca580 msgr2=0x7f92ac0c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92ab400640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92ac0ca580 0x7f92ac0c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92ab400640 1 -- 172.21.15.69:0/1070879687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92ac0c5680 con 0x7f92ac0ca100 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92ab400640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92ac0ca100 0x7f92ac0c7c10 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f9298011700 tx=0x7f9298011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92a3e00640 1 -- 172.21.15.69:0/1070879687 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f929800f960 con 0x7f92ac0ca100 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92a3e00640 1 -- 172.21.15.69:0/1070879687 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f929800fb00 con 0x7f92ac0ca100 2024-04-24T08:17:56.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.054+0000 7f92a3e00640 1 -- 172.21.15.69:0/1070879687 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9298015460 con 0x7f92ac0ca100 2024-04-24T08:17:56.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.055+0000 7f92b3000640 1 -- 172.21.15.69:0/1070879687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92ac0c5960 con 0x7f92ac0ca100 2024-04-24T08:17:56.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.055+0000 7f92b3000640 1 -- 172.21.15.69:0/1070879687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92ac0c5f20 con 0x7f92ac0ca100 2024-04-24T08:17:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.056+0000 7f92b3000640 1 -- 172.21.15.69:0/1070879687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92ac068a80 con 0x7f92ac0ca100 2024-04-24T08:17:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.057+0000 7f92a3e00640 1 -- 172.21.15.69:0/1070879687 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9298015600 con 0x7f92ac0ca100 2024-04-24T08:17:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.057+0000 7f92a3e00640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f928c06c8f0 0x7f928c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.058+0000 7f92a3e00640 1 -- 172.21.15.69:0/1070879687 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f929808c5c0 con 0x7f92ac0ca100 2024-04-24T08:17:56.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.058+0000 7f92abe00640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f928c06c8f0 0x7f928c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.060+0000 7f92abe00640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f928c06c8f0 0x7f928c06edb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f929c033000 tx=0x7f929c008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:56.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.062+0000 7f92a3e00640 1 -- 172.21.15.69:0/1070879687 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9298019030 con 0x7f92ac0ca100 2024-04-24T08:17:56.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.298+0000 7f92b3000640 1 -- 172.21.15.69:0/1070879687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92ac0cdef0 con 0x7f92ac0ca100 2024-04-24T08:17:56.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.300+0000 7f92a3e00640 1 -- 172.21.15.69:0/1070879687 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7f929805a880 con 0x7f92ac0ca100 2024-04-24T08:17:56.302 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:56.302 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:17:13.584835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8151":{"gid":8151,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6833/4254348458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4254348458},{"type":"v1","addr":"172.21.15.69:6833","nonce":4254348458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8151,"qdb_cluster":[8151]},"id":6}]} 2024-04-24T08:17:56.302 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 74 2024-04-24T08:17:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 -- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f928c06c8f0 msgr2=0x7f928c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f928c06c8f0 0x7f928c06edb0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f929c033000 tx=0x7f929c008030 comp rx=0 tx=0).stop 2024-04-24T08:17:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 -- 172.21.15.69:0/1070879687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92ac0ca100 msgr2=0x7f92ac0c7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92ac0ca100 0x7f92ac0c7c10 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f9298011700 tx=0x7f9298011bc0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 -- 172.21.15.69:0/1070879687 shutdown_connections 2024-04-24T08:17:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f928c06c8f0 0x7f928c06edb0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f92ac0ca100 0x7f92ac0c7c10 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92ac0ca580 0x7f92ac0c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 --2- 172.21.15.69:0/1070879687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f92900039b0 0x7f92ac143630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 -- 172.21.15.69:0/1070879687 >> 172.21.15.69:0/1070879687 conn(0x7f92ac067d90 msgr2=0x7f92ac0cd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:56.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 -- 172.21.15.69:0/1070879687 shutdown_connections 2024-04-24T08:17:56.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.302+0000 7f92b3000640 1 -- 172.21.15.69:0/1070879687 wait complete. 2024-04-24T08:17:56.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:56.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.518+0000 7f14bb200640 1 Processor -- start 2024-04-24T08:17:56.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.519+0000 7f14bb200640 1 -- start start 2024-04-24T08:17:56.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.520+0000 7f14bb200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f14bc1523f0 0x7f14bc06db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.520+0000 7f14bb200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14bc0aeac0 con 0x7f14bc15b980 2024-04-24T08:17:56.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.520+0000 7f14bb200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14bc0aeda0 con 0x7f14bc072e20 2024-04-24T08:17:56.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.520+0000 7f14bb200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f14bc0af080 con 0x7f14bc1523f0 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.520+0000 7f14ba800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f14bc15b980 0x7f14bc15be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39618/0 (socket says 172.21.15.69:39618) 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.520+0000 7f14ba800640 1 -- 172.21.15.69:0/2947538226 learned_addr learned my addr 172.21.15.69:0/2947538226 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9e00640 1 --2- 172.21.15.69:0/2947538226 >> v2:172.21.15.121:3300/0 conn(0x7f14bc1523f0 0x7f14bc06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1182817135 0 0) 0x7f14bc0aeac0 con 0x7f14bc15b980 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14a0005730 con 0x7f14bc15b980 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3526687139 0 0) 0x7f14a0005730 con 0x7f14bc15b980 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 >> v2:172.21.15.121:3300/0 conn(0x7f14bc1523f0 msgr2=0x7f14bc06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 --2- 172.21.15.69:0/2947538226 >> v2:172.21.15.121:3300/0 conn(0x7f14bc1523f0 0x7f14bc06db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 >> v1:172.21.15.121:6789/0 conn(0x7f14bc072e20 legacy=0x7f14bc073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc0af360 con 0x7f14bc15b980 2024-04-24T08:17:56.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f14b40019f0 con 0x7f14bc15b980 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 >> v1:172.21.15.69:6789/0 conn(0x7f14bc15b980 legacy=0x7f14bc15be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14a0005c10 0x7f14a0006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.521+0000 7f14b9400640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14a0006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b9400640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14a0007420 0x7f14a000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc0af080 con 0x7f14a0006a10 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f14a000e370 con 0x7f14a0007420 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f14a000e570 con 0x7f14a0005c10 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b1400640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14a0006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b9e00640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14a0005c10 0x7f14a0006030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14ba800640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14a0007420 0x7f14a000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b1400640 1 -- 172.21.15.69:0/2947538226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14a0005c10 msgr2=0x7f14a0006030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b1400640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14a0005c10 0x7f14a0006030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b1400640 1 -- 172.21.15.69:0/2947538226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14a0007420 msgr2=0x7f14a000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:56.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b1400640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14a0007420 0x7f14a000dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14b1400640 1 -- 172.21.15.69:0/2947538226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a0005730 con 0x7f14a0006a10 2024-04-24T08:17:56.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.522+0000 7f14ba800640 1 -- 172.21.15.69:0/2947538226 reap_dead start 2024-04-24T08:17:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.523+0000 7f14b1400640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14a0006e90 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f14a4001720 tx=0x7f14a0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76835c497a340b8e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.524+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a4001370 con 0x7f14a0006a10 2024-04-24T08:17:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.524+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14a40052d0 con 0x7f14a0006a10 2024-04-24T08:17:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.524+0000 7f14b9400640 1 -- 172.21.15.69:0/2947538226 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a402ab90 con 0x7f14a0006a10 2024-04-24T08:17:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.524+0000 7f14bb200640 1 -- 172.21.15.69:0/2947538226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 msgr2=0x7f14a0006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.524+0000 7f14bb200640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14a0006e90 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f14a4001720 tx=0x7f14a0007a40 comp rx=0 tx=0).stop 2024-04-24T08:17:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.524+0000 7f14bb200640 1 -- 172.21.15.69:0/2947538226 shutdown_connections 2024-04-24T08:17:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.524+0000 7f14bb200640 1 --2- 172.21.15.69:0/2947538226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14a0006e90 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.524+0000 7f14bb200640 1 -- 172.21.15.69:0/2947538226 >> 172.21.15.69:0/2947538226 conn(0x7f14bc0d0160 msgr2=0x7f14bc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:56.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.524+0000 7f14bb200640 1 -- 172.21.15.69:0/2947538226 shutdown_connections 2024-04-24T08:17:56.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.524+0000 7f14bb200640 1 -- 172.21.15.69:0/2947538226 wait complete. 2024-04-24T08:17:56.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.526+0000 7f14bb200640 1 Processor -- start 2024-04-24T08:17:56.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.526+0000 7f14bb200640 1 -- start start 2024-04-24T08:17:56.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.527+0000 7f14bb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14bc146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.527+0000 7f14bb200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14bc09c9a0 0x7f14bc09ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.527+0000 7f14bb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14bc09b2d0 0x7f14bc1e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.527+0000 7f14bb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc06db90 con 0x7f14a0006a10 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.527+0000 7f14bb200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f14bc1527d0 con 0x7f14bc09c9a0 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.527+0000 7f14bb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f14bc06dd10 con 0x7f14bc09b2d0 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.527+0000 7f14b9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14bc146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14b9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14bc146110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39028/0 (socket says 172.21.15.69:39028) 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14b9e00640 1 -- 172.21.15.69:0/2403209028 learned_addr learned my addr 172.21.15.69:0/2403209028 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14ba800640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14bc09b2d0 0x7f14bc1e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14b1400640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14bc09c9a0 0x7f14bc09ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14b9e00640 1 -- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14bc09b2d0 msgr2=0x7f14bc1e5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14b9e00640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14bc09b2d0 0x7f14bc1e5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14b9e00640 1 -- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14bc09c9a0 msgr2=0x7f14bc09ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14b9e00640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14bc09c9a0 0x7f14bc09ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14b9e00640 1 -- 172.21.15.69:0/2403209028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14bc146b00 con 0x7f14a0006a10 2024-04-24T08:17:56.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14b1400640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14bc09c9a0 0x7f14bc09ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:56.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.528+0000 7f14b9e00640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14bc146110 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f14ac010910 tx=0x7f14ac010dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:56.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.529+0000 7f14b2a00640 1 -- 172.21.15.69:0/2403209028 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac00f040 con 0x7f14a0006a10 2024-04-24T08:17:56.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.529+0000 7f14b2a00640 1 -- 172.21.15.69:0/2403209028 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14ac00ec20 con 0x7f14a0006a10 2024-04-24T08:17:56.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.529+0000 7f14b2a00640 1 -- 172.21.15.69:0/2403209028 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac01c400 con 0x7f14a0006a10 2024-04-24T08:17:56.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.529+0000 7f14bb200640 1 -- 172.21.15.69:0/2403209028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14bc09b900 con 0x7f14a0006a10 2024-04-24T08:17:56.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.529+0000 7f14bb200640 1 -- 172.21.15.69:0/2403209028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14bc09bec0 con 0x7f14a0006a10 2024-04-24T08:17:56.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.530+0000 7f14bb200640 1 -- 172.21.15.69:0/2403209028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14bc075d50 con 0x7f14a0006a10 2024-04-24T08:17:56.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.531+0000 7f14b2a00640 1 -- 172.21.15.69:0/2403209028 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f14ac01c5a0 con 0x7f14a0006a10 2024-04-24T08:17:56.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.532+0000 7f14b2a00640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f149806c8f0 0x7f149806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.532+0000 7f14b2a00640 1 -- 172.21.15.69:0/2403209028 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f14ac08c320 con 0x7f14a0006a10 2024-04-24T08:17:56.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.532+0000 7f14b1400640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f149806c8f0 0x7f149806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:56.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.534+0000 7f14b1400640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f149806c8f0 0x7f149806edb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f14a4032000 tx=0x7f14a402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:56.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.536+0000 7f14b2a00640 1 -- 172.21.15.69:0/2403209028 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14ac018030 con 0x7f14a0006a10 2024-04-24T08:17:56.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.778+0000 7f14bb200640 1 -- 172.21.15.69:0/2403209028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14bc15b3c0 con 0x7f14a0006a10 2024-04-24T08:17:56.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.779+0000 7f14b2a00640 1 -- 172.21.15.69:0/2403209028 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5181 (secure 0 0 0) 0x7f14ac05a030 con 0x7f14a0006a10 2024-04-24T08:17:56.781 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:56.781 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":74,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:15:23.442689+0000","modified":"2024-04-24T08:17:13.584835+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":86,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8151},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8151":{"gid":8151,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6833/4254348458","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4254348458},{"type":"v1","addr":"172.21.15.69:6833","nonce":4254348458}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8151,"qdb_cluster":[8151]},"id":6}]} 2024-04-24T08:17:56.782 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 74 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.782+0000 7f14bb200640 1 -- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f149806c8f0 msgr2=0x7f149806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.782+0000 7f14bb200640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f149806c8f0 0x7f149806edb0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f14a4032000 tx=0x7f14a402e000 comp rx=0 tx=0).stop 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.782+0000 7f14bb200640 1 -- 172.21.15.69:0/2403209028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 msgr2=0x7f14bc146110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.782+0000 7f14bb200640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14bc146110 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f14ac010910 tx=0x7f14ac010dd0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.783+0000 7f14bb200640 1 -- 172.21.15.69:0/2403209028 shutdown_connections 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.783+0000 7f14bb200640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f149806c8f0 0x7f149806edb0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.783+0000 7f14bb200640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f14bc09b2d0 0x7f14bc1e5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.783+0000 7f14bb200640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14bc09c9a0 0x7f14bc09ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.783+0000 7f14bb200640 1 --2- 172.21.15.69:0/2403209028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f14a0006a10 0x7f14bc146110 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.783+0000 7f14bb200640 1 -- 172.21.15.69:0/2403209028 >> 172.21.15.69:0/2403209028 conn(0x7f14bc0d0160 msgr2=0x7f14bc06ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.783+0000 7f14bb200640 1 -- 172.21.15.69:0/2403209028 shutdown_connections 2024-04-24T08:17:56.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.783+0000 7f14bb200640 1 -- 172.21.15.69:0/2403209028 wait complete. 2024-04-24T08:17:56.803 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:17:56.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.996+0000 7f55ea400640 1 Processor -- start 2024-04-24T08:17:56.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.996+0000 7f55ea400640 1 -- start start 2024-04-24T08:17:56.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.997+0000 7f55ea400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4156c60 0x7f55e415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:56.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.998+0000 7f55ea400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55e415d660 con 0x7f55e40b30f0 2024-04-24T08:17:56.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.998+0000 7f55ea400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55e406dc30 con 0x7f55e40ad7e0 2024-04-24T08:17:56.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.998+0000 7f55ea400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f55e406df10 con 0x7f55e4156c60 2024-04-24T08:17:57.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.998+0000 7f55e3e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f55e40b30f0 0x7f55e40ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39632/0 (socket says 172.21.15.69:39632) 2024-04-24T08:17:57.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.998+0000 7f55e3e00640 1 -- 172.21.15.69:0/3814001883 learned_addr learned my addr 172.21.15.69:0/3814001883 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:57.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.998+0000 7f55e9000640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4156c60 0x7f55e415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837749995 0 0) 0x7f55e415d660 con 0x7f55e40b30f0 2024-04-24T08:17:57.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55cc003520 con 0x7f55e40b30f0 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1956696404 0 0) 0x7f55cc003520 con 0x7f55e40b30f0 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4156c60 msgr2=0x7f55e415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4156c60 0x7f55e415cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 >> v1:172.21.15.121:6789/0 conn(0x7f55e40ad7e0 legacy=0x7f55e406e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e406e1f0 con 0x7f55e40b30f0 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f55d0001670 con 0x7f55e40b30f0 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 >> v1:172.21.15.69:6789/0 conn(0x7f55e40b30f0 legacy=0x7f55e40ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4156c60 0x7f55cc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 0x7f55cc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55cc004d40 0x7f55cc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55e406df10 con 0x7f55cc003f60 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f55cc00baa0 con 0x7f55cc004d40 2024-04-24T08:17:57.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f55cc00bca0 con 0x7f55e4156c60 2024-04-24T08:17:57.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:56.999+0000 7f55e3e00640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 0x7f55cc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.000+0000 7f55e9000640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4156c60 0x7f55cc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.000+0000 7f55e9a00640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55cc004d40 0x7f55cc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.000+0000 7f55e3e00640 1 -- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4156c60 msgr2=0x7f55cc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:57.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.000+0000 7f55e3e00640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4156c60 0x7f55cc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.000+0000 7f55e3e00640 1 -- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55cc004d40 msgr2=0x7f55cc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:57.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.000+0000 7f55e3e00640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55cc004d40 0x7f55cc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.000+0000 7f55e3e00640 1 -- 172.21.15.69:0/3814001883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55cc003520 con 0x7f55cc003f60 2024-04-24T08:17:57.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.001+0000 7f55e3e00640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 0x7f55cc004800 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f55e40b34d0 tx=0x7f55cc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c1f7e157ed2353 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:57.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.001+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d0007d30 con 0x7f55cc003f60 2024-04-24T08:17:57.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.001+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d00058d0 con 0x7f55cc003f60 2024-04-24T08:17:57.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.001+0000 7f55e3400640 1 -- 172.21.15.69:0/3814001883 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d0005d20 con 0x7f55cc003f60 2024-04-24T08:17:57.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.002+0000 7f55ea400640 1 -- 172.21.15.69:0/3814001883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 msgr2=0x7f55cc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.002+0000 7f55ea400640 1 --2- 172.21.15.69:0/3814001883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 0x7f55cc004800 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f55e40b34d0 tx=0x7f55cc005830 comp rx=0 tx=0).stop 2024-04-24T08:17:57.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.002+0000 7f55e9a00640 1 -- 172.21.15.69:0/3814001883 reap_dead start 2024-04-24T08:17:57.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.002+0000 7f55ea400640 1 -- 172.21.15.69:0/3814001883 shutdown_connections 2024-04-24T08:17:57.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.002+0000 7f55ea400640 1 -- 172.21.15.69:0/3814001883 >> 172.21.15.69:0/3814001883 conn(0x7f55e40d0160 msgr2=0x7f55e40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:57.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.002+0000 7f55ea400640 1 -- 172.21.15.69:0/3814001883 shutdown_connections 2024-04-24T08:17:57.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.002+0000 7f55ea400640 1 -- 172.21.15.69:0/3814001883 wait complete. 2024-04-24T08:17:57.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.004+0000 7f55ea400640 1 Processor -- start 2024-04-24T08:17:57.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.004+0000 7f55ea400640 1 -- start start 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55ea400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 0x7f55e4098800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55ea400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55e41e6020 0x7f55e41f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55ea400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4096d80 0x7f55e4098dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55ea400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55e406ee20 con 0x7f55cc003f60 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55ea400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f55e415d940 con 0x7f55e41e6020 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55ea400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f55e406efa0 con 0x7f55e4096d80 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55e9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 0x7f55e4098800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55e9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 0x7f55e4098800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39046/0 (socket says 172.21.15.69:39046) 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55e9000640 1 -- 172.21.15.69:0/2888408207 learned_addr learned my addr 172.21.15.69:0/2888408207 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55e3e00640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55e41e6020 0x7f55e41f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55e9a00640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4096d80 0x7f55e4098dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.005+0000 7f55e9000640 1 -- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4096d80 msgr2=0x7f55e4098dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55e9000640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4096d80 0x7f55e4098dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55e9000640 1 -- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55e41e6020 msgr2=0x7f55e41f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55e9000640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55e41e6020 0x7f55e41f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55e9000640 1 -- 172.21.15.69:0/2888408207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e41f2620 con 0x7f55cc003f60 2024-04-24T08:17:57.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55e3e00640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55e41e6020 0x7f55e41f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:57.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55e9000640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 0x7f55e4098800 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f55d8011630 tx=0x7f55d8011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:57.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55e0c00640 1 -- 172.21.15.69:0/2888408207 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d800f920 con 0x7f55cc003f60 2024-04-24T08:17:57.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55e0c00640 1 -- 172.21.15.69:0/2888408207 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55d800fac0 con 0x7f55cc003f60 2024-04-24T08:17:57.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55ea400640 1 -- 172.21.15.69:0/2888408207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55e4097760 con 0x7f55cc003f60 2024-04-24T08:17:57.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55e0c00640 1 -- 172.21.15.69:0/2888408207 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55d800fe70 con 0x7f55cc003f60 2024-04-24T08:17:57.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.006+0000 7f55ea400640 1 -- 172.21.15.69:0/2888408207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55e4097d20 con 0x7f55cc003f60 2024-04-24T08:17:57.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.007+0000 7f55ea400640 1 -- 172.21.15.69:0/2888408207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55e4075d50 con 0x7f55cc003f60 2024-04-24T08:17:57.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.008+0000 7f55e0c00640 1 -- 172.21.15.69:0/2888408207 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f55d8015460 con 0x7f55cc003f60 2024-04-24T08:17:57.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.009+0000 7f55e0c00640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55c406c8f0 0x7f55c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.009+0000 7f55e0c00640 1 -- 172.21.15.69:0/2888408207 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f55d808c5c0 con 0x7f55cc003f60 2024-04-24T08:17:57.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.010+0000 7f55e3e00640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55c406c8f0 0x7f55c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.011+0000 7f55e3e00640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55c406c8f0 0x7f55c406edb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f55d0033000 tx=0x7f55d002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:57.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.013+0000 7f55e0c00640 1 -- 172.21.15.69:0/2888408207 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f55d8019030 con 0x7f55cc003f60 2024-04-24T08:17:57.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.210+0000 7f55ea400640 1 -- 172.21.15.69:0/2888408207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55e415c380 con 0x7f55cc003f60 2024-04-24T08:17:57.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.213+0000 7f55e0c00640 1 -- 172.21.15.69:0/2888408207 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+16947 (secure 0 0 0) 0x7f55d805a910 con 0x7f55cc003f60 2024-04-24T08:17:57.215 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:17:57.215 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":86,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:17:08.931744+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:15:21.126081+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-04-24T08:15:22.402310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.11","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6832/2529876410":"2024-04-25T08:17:08.931681+0000","172.21.15.121:6837/1179683245":"2024-04-25T08:15:45.259762+0000","172.21.15.69:6833/2529876410":"2024-04-25T08:17:08.931681+0000","172.21.15.121:6835/1179683245":"2024-04-25T08:15:45.259762+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:17:57.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 -- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55c406c8f0 msgr2=0x7f55c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55c406c8f0 0x7f55c406edb0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f55d0033000 tx=0x7f55d002e000 comp rx=0 tx=0).stop 2024-04-24T08:17:57.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 -- 172.21.15.69:0/2888408207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 msgr2=0x7f55e4098800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 0x7f55e4098800 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f55d8011630 tx=0x7f55d8011af0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 -- 172.21.15.69:0/2888408207 shutdown_connections 2024-04-24T08:17:57.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55c406c8f0 0x7f55c406edb0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55e4096d80 0x7f55e4098dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55e41e6020 0x7f55e41f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 --2- 172.21.15.69:0/2888408207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55cc003f60 0x7f55e4098800 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 -- 172.21.15.69:0/2888408207 >> 172.21.15.69:0/2888408207 conn(0x7f55e40d0160 msgr2=0x7f55e415b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:57.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 -- 172.21.15.69:0/2888408207 shutdown_connections 2024-04-24T08:17:57.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.216+0000 7f55ea400640 1 -- 172.21.15.69:0/2888408207 wait complete. 2024-04-24T08:17:57.235 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:17:57.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.427+0000 7f0d46e00640 1 Processor -- start 2024-04-24T08:17:57.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.428+0000 7f0d46e00640 1 -- start start 2024-04-24T08:17:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.429+0000 7f0d46e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d400d18c0 0x7f0d400cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.429+0000 7f0d46e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d400d26c0 con 0x7f0d4015ec70 2024-04-24T08:17:57.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.429+0000 7f0d46e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d400d29a0 con 0x7f0d4015e890 2024-04-24T08:17:57.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.429+0000 7f0d46e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0d400d32d0 con 0x7f0d400d18c0 2024-04-24T08:17:57.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.429+0000 7f0d3f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d400d18c0 0x7f0d400cfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.429+0000 7f0d44e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0d4015ec70 0x7f0d400b3b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39638/0 (socket says 172.21.15.69:39638) 2024-04-24T08:17:57.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.429+0000 7f0d44e00640 1 -- 172.21.15.69:0/3084660993 learned_addr learned my addr 172.21.15.69:0/3084660993 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:57.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.430+0000 7f0d3f400640 1 -- 172.21.15.69:0/3084660993 >> v1:172.21.15.121:6789/0 conn(0x7f0d4015e890 legacy=0x7f0d400d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:57.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.430+0000 7f0d3f400640 1 -- 172.21.15.69:0/3084660993 >> v1:172.21.15.69:6789/0 conn(0x7f0d4015ec70 legacy=0x7f0d400b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.430+0000 7f0d3f400640 1 -- 172.21.15.69:0/3084660993 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d400d2c80 con 0x7f0d400d18c0 2024-04-24T08:17:57.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.431+0000 7f0d3f400640 1 --2- 172.21.15.69:0/3084660993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d400d18c0 0x7f0d400cfec0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f0d400d3450 tx=0x7f0d2802a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886b1de33f442417 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:57.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.431+0000 7f0d3ea00640 1 -- 172.21.15.69:0/3084660993 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28007740 con 0x7f0d400d18c0 2024-04-24T08:17:57.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.431+0000 7f0d3ea00640 1 -- 172.21.15.69:0/3084660993 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d280078e0 con 0x7f0d400d18c0 2024-04-24T08:17:57.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.431+0000 7f0d3ea00640 1 -- 172.21.15.69:0/3084660993 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28007cb0 con 0x7f0d400d18c0 2024-04-24T08:17:57.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.432+0000 7f0d46e00640 1 -- 172.21.15.69:0/3084660993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d400d18c0 msgr2=0x7f0d400cfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.432+0000 7f0d46e00640 1 --2- 172.21.15.69:0/3084660993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d400d18c0 0x7f0d400cfec0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f0d400d3450 tx=0x7f0d2802a780 comp rx=0 tx=0).stop 2024-04-24T08:17:57.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.432+0000 7f0d46e00640 1 -- 172.21.15.69:0/3084660993 shutdown_connections 2024-04-24T08:17:57.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.432+0000 7f0d46e00640 1 --2- 172.21.15.69:0/3084660993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d400d18c0 0x7f0d400cfec0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.432+0000 7f0d46e00640 1 -- 172.21.15.69:0/3084660993 >> 172.21.15.69:0/3084660993 conn(0x7f0d4008bf60 msgr2=0x7f0d400d47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:57.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.432+0000 7f0d46e00640 1 -- 172.21.15.69:0/3084660993 shutdown_connections 2024-04-24T08:17:57.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.432+0000 7f0d46e00640 1 -- 172.21.15.69:0/3084660993 wait complete. 2024-04-24T08:17:57.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.434+0000 7f0d46e00640 1 Processor -- start 2024-04-24T08:17:57.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.434+0000 7f0d46e00640 1 -- start start 2024-04-24T08:17:57.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d46e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d400d18c0 0x7f0d400a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d46e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4015e890 0x7f0d400cbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d46e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4015ec70 0x7f0d400b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d46e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0d400d2370 con 0x7f0d4015e890 2024-04-24T08:17:57.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d46e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0d400d3760 con 0x7f0d4015ec70 2024-04-24T08:17:57.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d46e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0d400d0400 con 0x7f0d400d18c0 2024-04-24T08:17:57.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d3f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4015e890 0x7f0d400cbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d3f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4015e890 0x7f0d400cbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39072/0 (socket says 172.21.15.69:39072) 2024-04-24T08:17:57.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d3f400640 1 -- 172.21.15.69:0/3029996925 learned_addr learned my addr 172.21.15.69:0/3029996925 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:57.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d3f400640 1 -- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d400d18c0 msgr2=0x7f0d400a1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.435+0000 7f0d3fe00640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d400d18c0 0x7f0d400a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.436+0000 7f0d3f400640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d400d18c0 0x7f0d400a1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.436+0000 7f0d3f400640 1 -- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4015ec70 msgr2=0x7f0d400b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.436+0000 7f0d3f400640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4015ec70 0x7f0d400b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.436+0000 7f0d3f400640 1 -- 172.21.15.69:0/3029996925 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d400cc590 con 0x7f0d4015e890 2024-04-24T08:17:57.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.436+0000 7f0d3f400640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4015e890 0x7f0d400cbd40 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009ea0 tx=0x7f0d28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:57.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.436+0000 7f0d37e00640 1 -- 172.21.15.69:0/3029996925 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28041030 con 0x7f0d4015e890 2024-04-24T08:17:57.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.436+0000 7f0d37e00640 1 -- 172.21.15.69:0/3029996925 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d28031b00 con 0x7f0d4015e890 2024-04-24T08:17:57.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.436+0000 7f0d37e00640 1 -- 172.21.15.69:0/3029996925 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2803c420 con 0x7f0d4015e890 2024-04-24T08:17:57.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.436+0000 7f0d46e00640 1 -- 172.21.15.69:0/3029996925 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d400cc870 con 0x7f0d4015e890 2024-04-24T08:17:57.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.436+0000 7f0d46e00640 1 -- 172.21.15.69:0/3029996925 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d400cce30 con 0x7f0d4015e890 2024-04-24T08:17:57.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.438+0000 7f0d46e00640 1 -- 172.21.15.69:0/3029996925 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d40068a80 con 0x7f0d4015e890 2024-04-24T08:17:57.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.438+0000 7f0d37e00640 1 -- 172.21.15.69:0/3029996925 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d28044030 con 0x7f0d4015e890 2024-04-24T08:17:57.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.439+0000 7f0d37e00640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0d1c06c8f0 0x7f0d1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.440+0000 7f0d37e00640 1 -- 172.21.15.69:0/3029996925 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 5910+0+0 (secure 0 0 0) 0x7f0d280ad030 con 0x7f0d4015e890 2024-04-24T08:17:57.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.440+0000 7f0d3fe00640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0d1c06c8f0 0x7f0d1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.441+0000 7f0d3fe00640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0d1c06c8f0 0x7f0d1c06edb0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0d30003d90 tx=0x7f0d30009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:57.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.444+0000 7f0d37e00640 1 -- 172.21.15.69:0/3029996925 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d2807b390 con 0x7f0d4015e890 2024-04-24T08:17:57.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.693+0000 7f0d46e00640 1 -- 172.21.15.69:0/3029996925 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f0d400cbb20 con 0x7f0d4015e890 2024-04-24T08:17:57.696 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:17:57.694+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:17:57.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.698+0000 7f0d37e00640 1 -- 172.21.15.69:0/3029996925 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v75) ==== 161+0+0 (secure 0 0 0) 0x7f0d2807ac90 con 0x7f0d4015e890 2024-04-24T08:17:57.699 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:17:57.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 -- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0d1c06c8f0 msgr2=0x7f0d1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0d1c06c8f0 0x7f0d1c06edb0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0d30003d90 tx=0x7f0d30009040 comp rx=0 tx=0).stop 2024-04-24T08:17:57.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 -- 172.21.15.69:0/3029996925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4015e890 msgr2=0x7f0d400cbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4015e890 0x7f0d400cbd40 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009ea0 tx=0x7f0d28004a40 comp rx=0 tx=0).stop 2024-04-24T08:17:57.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 -- 172.21.15.69:0/3029996925 shutdown_connections 2024-04-24T08:17:57.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0d1c06c8f0 0x7f0d1c06edb0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d4015e890 0x7f0d400cbd40 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d4015ec70 0x7f0d400b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 --2- 172.21.15.69:0/3029996925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d400d18c0 0x7f0d400a1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 -- 172.21.15.69:0/3029996925 >> 172.21.15.69:0/3029996925 conn(0x7f0d4008bf60 msgr2=0x7f0d400d47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:57.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 -- 172.21.15.69:0/3029996925 shutdown_connections 2024-04-24T08:17:57.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.701+0000 7f0d46e00640 1 -- 172.21.15.69:0/3029996925 wait complete. 2024-04-24T08:17:57.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T08:17:57.725 INFO:tasks.ceph.mds.a.smithi069.stderr: -2424> 2024-04-24T08:17:16.000+0000 7f5193400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:17:57.735 INFO:tasks.ceph.mds.a.smithi069.stderr: -2037> 2024-04-24T08:17:17.947+0000 7f518e400640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() | dirtyparent=1 authpin=1 scrubqueue=0 0x56135fd28680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(13)0x10000000001:[<0x10000000000/file v6>,<0x1/dir v13>]//[]","memoryvalue":"(13)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v21>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-24T08:17:57.798 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:17:57.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.913+0000 7fc9fc200640 1 Processor -- start 2024-04-24T08:17:57.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.913+0000 7fc9fc200640 1 -- start start 2024-04-24T08:17:57.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.914+0000 7fc9fc200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f40b3e90 0x7fc9f40b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.914+0000 7fc9fc200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc9f40ce250 0x7fc9f40ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.914+0000 7fc9fc200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9f415cc00 con 0x7fc9f415c820 2024-04-24T08:17:57.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.914+0000 7fc9fc200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc9f40b4be0 con 0x7fc9f40ce250 2024-04-24T08:17:57.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.914+0000 7fc9fc200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f40b42f0 con 0x7fc9f40b3e90 2024-04-24T08:17:57.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f9800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc9f415c820 0x7fc9f406f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39656/0 (socket says 172.21.15.69:39656) 2024-04-24T08:17:57.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f9800640 1 -- 172.21.15.69:0/84259968 learned_addr learned my addr 172.21.15.69:0/84259968 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:57.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3194057188 0 0) 0x7fc9f415cc00 con 0x7fc9f415c820 2024-04-24T08:17:57.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9dc003520 con 0x7fc9f415c820 2024-04-24T08:17:57.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9fa200640 1 --2- 172.21.15.69:0/84259968 >> v2:172.21.15.121:3300/0 conn(0x7fc9f40ce250 0x7fc9f40ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f8e00640 1 --2- 172.21.15.69:0/84259968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f40b3e90 0x7fc9f40b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 931672585 0 0) 0x7fc9dc003520 con 0x7fc9f415c820 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f40b3e90 msgr2=0x7fc9f40b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f3e00640 1 --2- 172.21.15.69:0/84259968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f40b3e90 0x7fc9f40b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 >> v2:172.21.15.121:3300/0 conn(0x7fc9f40ce250 msgr2=0x7fc9f40ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f3e00640 1 --2- 172.21.15.69:0/84259968 >> v2:172.21.15.121:3300/0 conn(0x7fc9f40ce250 0x7fc9f40ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f406ed30 con 0x7fc9f415c820 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fc9e8001670 con 0x7fc9f415c820 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.915+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 >> v1:172.21.15.69:6789/0 conn(0x7fc9f415c820 legacy=0x7fc9f406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f3e00640 1 --2- 172.21.15.69:0/84259968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9dc0039b0 0x7fc9dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f3e00640 1 --2- 172.21.15.69:0/84259968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9dc004800 0x7fc9dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f3e00640 1 --2- 172.21.15.69:0/84259968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f40b3e90 0x7fc9dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f40b42f0 con 0x7fc9dc004800 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc9dc00b9f0 con 0x7fc9dc0039b0 2024-04-24T08:17:57.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc9dc00bbf0 con 0x7fc9f40b3e90 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f8e00640 1 --2- 172.21.15.69:0/84259968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9dc004800 0x7fc9dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f8e00640 1 -- 172.21.15.69:0/84259968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f40b3e90 msgr2=0x7fc9dc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f8e00640 1 --2- 172.21.15.69:0/84259968 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f40b3e90 0x7fc9dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f8e00640 1 -- 172.21.15.69:0/84259968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9dc0039b0 msgr2=0x7fc9dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f8e00640 1 --2- 172.21.15.69:0/84259968 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9dc0039b0 0x7fc9dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.916+0000 7fc9f8e00640 1 -- 172.21.15.69:0/84259968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9dc003520 con 0x7fc9dc004800 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.917+0000 7fc9f8e00640 1 --2- 172.21.15.69:0/84259968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9dc004800 0x7fc9dc004c80 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400ba00 tx=0x7fc9dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed879c9d0e74c56 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.917+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4030070 con 0x7fc9dc004800 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.917+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e402adb0 con 0x7fc9dc004800 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.917+0000 7fc9f3e00640 1 -- 172.21.15.69:0/84259968 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e400b600 con 0x7fc9dc004800 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.918+0000 7fc9fc200640 1 -- 172.21.15.69:0/84259968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9dc004800 msgr2=0x7fc9dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.918+0000 7fc9fc200640 1 --2- 172.21.15.69:0/84259968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9dc004800 0x7fc9dc004c80 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400ba00 tx=0x7fc9dc005830 comp rx=0 tx=0).stop 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.918+0000 7fc9fa200640 1 -- 172.21.15.69:0/84259968 reap_dead start 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.918+0000 7fc9fc200640 1 -- 172.21.15.69:0/84259968 shutdown_connections 2024-04-24T08:17:57.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.918+0000 7fc9fc200640 1 -- 172.21.15.69:0/84259968 >> 172.21.15.69:0/84259968 conn(0x7fc9f40d0350 msgr2=0x7fc9f40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:57.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.918+0000 7fc9fc200640 1 -- 172.21.15.69:0/84259968 shutdown_connections 2024-04-24T08:17:57.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.918+0000 7fc9fc200640 1 -- 172.21.15.69:0/84259968 wait complete. 2024-04-24T08:17:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.920+0000 7fc9fc200640 1 Processor -- start 2024-04-24T08:17:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.920+0000 7fc9fc200640 1 -- start start 2024-04-24T08:17:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9fc200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9dc004800 0x7fc9f40b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9fc200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9f415d8e0 0x7fc9f415dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9fc200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f415e360 0x7fc9f415e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9fc200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f406efd0 con 0x7fc9f415d8e0 2024-04-24T08:17:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9fc200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f40b50f0 con 0x7fc9dc004800 2024-04-24T08:17:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9fc200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f406f2b0 con 0x7fc9f415e360 2024-04-24T08:17:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9f8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9f415d8e0 0x7fc9f415dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9f8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9f415d8e0 0x7fc9f415dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39102/0 (socket says 172.21.15.69:39102) 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9f8e00640 1 -- 172.21.15.69:0/3629910258 learned_addr learned my addr 172.21.15.69:0/3629910258 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9f8e00640 1 -- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f415e360 msgr2=0x7fc9f415e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9f8e00640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f415e360 0x7fc9f415e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9f8e00640 1 -- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9dc004800 msgr2=0x7fc9f40b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9f8e00640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9dc004800 0x7fc9f40b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.921+0000 7fc9f8e00640 1 -- 172.21.15.69:0/3629910258 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f41f8430 con 0x7fc9f415d8e0 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.922+0000 7fc9f8e00640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9f415d8e0 0x7fc9f415dd60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400bb20 tx=0x7fc9e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.922+0000 7fc9f1600640 1 -- 172.21.15.69:0/3629910258 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4043020 con 0x7fc9f415d8e0 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.922+0000 7fc9f1600640 1 -- 172.21.15.69:0/3629910258 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e4035ea0 con 0x7fc9f415d8e0 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.922+0000 7fc9f1600640 1 -- 172.21.15.69:0/3629910258 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e403f8a0 con 0x7fc9f415d8e0 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.922+0000 7fc9fc200640 1 -- 172.21.15.69:0/3629910258 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f41f8710 con 0x7fc9f415d8e0 2024-04-24T08:17:57.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.922+0000 7fc9fc200640 1 -- 172.21.15.69:0/3629910258 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f415d0e0 con 0x7fc9f415d8e0 2024-04-24T08:17:57.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.923+0000 7fc9fc200640 1 -- 172.21.15.69:0/3629910258 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9f4068a80 con 0x7fc9f415d8e0 2024-04-24T08:17:57.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.924+0000 7fc9f1600640 1 -- 172.21.15.69:0/3629910258 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc9e4046020 con 0x7fc9f415d8e0 2024-04-24T08:17:57.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.925+0000 7fc9f1600640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc9d806c920 0x7fc9d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:57.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.925+0000 7fc9f1600640 1 -- 172.21.15.69:0/3629910258 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5996+0+0 (secure 0 0 0) 0x7fc9e40ac7f0 con 0x7fc9f415d8e0 2024-04-24T08:17:57.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.926+0000 7fc9f9800640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc9d806c920 0x7fc9d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:57.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.927+0000 7fc9f9800640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc9d806c920 0x7fc9d806ede0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80079f0 tx=0x7fc9e8001500 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:57.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:57.930+0000 7fc9f1600640 1 -- 172.21.15.69:0/3629910258 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc9e404d030 con 0x7fc9f415d8e0 2024-04-24T08:17:58.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.155+0000 7fc9fc200640 1 -- 172.21.15.69:0/3629910258 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fc9f40cddc0 con 0x7fc9f415d8e0 2024-04-24T08:17:58.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.777+0000 7fc9f1600640 1 -- 172.21.15.69:0/3629910258 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v76) ==== 104+0+0 (secure 0 0 0) 0x7fc9e407aa60 con 0x7fc9f415d8e0 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 -- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc9d806c920 msgr2=0x7fc9d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc9d806c920 0x7fc9d806ede0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fc9e80079f0 tx=0x7fc9e8001500 comp rx=0 tx=0).stop 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 -- 172.21.15.69:0/3629910258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9f415d8e0 msgr2=0x7fc9f415dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9f415d8e0 0x7fc9f415dd60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fc9e400bb20 tx=0x7fc9e4004730 comp rx=0 tx=0).stop 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 -- 172.21.15.69:0/3629910258 shutdown_connections 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc9d806c920 0x7fc9d806ede0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9f415d8e0 0x7fc9f415dd60 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc9f415e360 0x7fc9f415e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 --2- 172.21.15.69:0/3629910258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc9dc004800 0x7fc9f40b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 -- 172.21.15.69:0/3629910258 >> 172.21.15.69:0/3629910258 conn(0x7fc9f40d0350 msgr2=0x7fc9f40ceef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:58.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 -- 172.21.15.69:0/3629910258 shutdown_connections 2024-04-24T08:17:58.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:58.781+0000 7fc9fc200640 1 -- 172.21.15.69:0/3629910258 wait complete. 2024-04-24T08:17:58.808 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T08:17:59.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.007+0000 7f4b1f200640 1 Processor -- start 2024-04-24T08:17:59.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.007+0000 7f4b1f200640 1 -- start start 2024-04-24T08:17:59.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.008+0000 7f4b1f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4b2015e4f0 0x7f4b2015e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:59.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.008+0000 7f4b1f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4b200d1ac0 0x7f4b200cfd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:59.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.008+0000 7f4b1f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b200d0470 con 0x7f4b200d2f70 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.008+0000 7f4b1f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4b200d1f70 con 0x7f4b200d1ac0 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.008+0000 7f4b1f200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4b200d0240 con 0x7f4b2015e4f0 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.008+0000 7f4b17e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4b200d1ac0 0x7f4b200cfd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.008+0000 7f4b1e800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4b200d2f70 0x7f4b200d45e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47346/0 (socket says 172.21.15.69:47346) 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.008+0000 7f4b17e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4b200d1ac0 0x7f4b200cfd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33850/0 (socket says 172.21.15.69:33850) 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.008+0000 7f4b1e800640 1 -- 172.21.15.69:0/1482666538 learned_addr learned my addr 172.21.15.69:0/1482666538 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.009+0000 7f4b17e00640 1 -- 172.21.15.69:0/1482666538 >> v2:172.21.15.121:3300/0 conn(0x7f4b2015e4f0 msgr2=0x7f4b2015e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.009+0000 7f4b1de00640 1 --2- 172.21.15.69:0/1482666538 >> v2:172.21.15.121:3300/0 conn(0x7f4b2015e4f0 0x7f4b2015e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.009+0000 7f4b1d400640 1 -- 172.21.15.69:0/1482666538 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1569850540 0 0) 0x7f4b200d0470 con 0x7f4b200d2f70 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.009+0000 7f4b17e00640 1 --2- 172.21.15.69:0/1482666538 >> v2:172.21.15.121:3300/0 conn(0x7f4b2015e4f0 0x7f4b2015e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.009+0000 7f4b17e00640 1 -- 172.21.15.69:0/1482666538 >> v1:172.21.15.69:6789/0 conn(0x7f4b200d2f70 legacy=0x7f4b200d45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.009+0000 7f4b17e00640 1 -- 172.21.15.69:0/1482666538 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b200d2490 con 0x7f4b200d1ac0 2024-04-24T08:17:59.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.009+0000 7f4b1de00640 1 --2- 172.21.15.69:0/1482666538 >> v2:172.21.15.121:3300/0 conn(0x7f4b2015e4f0 0x7f4b2015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:59.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.010+0000 7f4b17e00640 1 --2- 172.21.15.69:0/1482666538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b200d1ac0 0x7f4b200cfd00 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f4b2009de00 tx=0x7f4b0802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7412b2ea9ef77697 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.010+0000 7f4b1d400640 1 -- 172.21.15.69:0/1482666538 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b08037070 con 0x7f4b200d1ac0 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.010+0000 7f4b1d400640 1 -- 172.21.15.69:0/1482666538 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b08032440 con 0x7f4b200d1ac0 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.010+0000 7f4b1d400640 1 -- 172.21.15.69:0/1482666538 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b08032810 con 0x7f4b200d1ac0 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.011+0000 7f4b1f200640 1 -- 172.21.15.69:0/1482666538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b200d1ac0 msgr2=0x7f4b200cfd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.011+0000 7f4b1f200640 1 --2- 172.21.15.69:0/1482666538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b200d1ac0 0x7f4b200cfd00 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f4b2009de00 tx=0x7f4b0802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.011+0000 7f4b1f200640 1 -- 172.21.15.69:0/1482666538 shutdown_connections 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.011+0000 7f4b1f200640 1 --2- 172.21.15.69:0/1482666538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b200d1ac0 0x7f4b200cfd00 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.011+0000 7f4b1f200640 1 --2- 172.21.15.69:0/1482666538 >> v2:172.21.15.121:3300/0 conn(0x7f4b2015e4f0 0x7f4b2015e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.011+0000 7f4b1f200640 1 -- 172.21.15.69:0/1482666538 >> 172.21.15.69:0/1482666538 conn(0x7f4b2009d940 msgr2=0x7f4b2008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.011+0000 7f4b1f200640 1 -- 172.21.15.69:0/1482666538 shutdown_connections 2024-04-24T08:17:59.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.011+0000 7f4b1f200640 1 -- 172.21.15.69:0/1482666538 wait complete. 2024-04-24T08:17:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.013+0000 7f4b1f200640 1 Processor -- start 2024-04-24T08:17:59.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.013+0000 7f4b1f200640 1 -- start start 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b200d1ac0 0x7f4b20157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b200d2f70 0x7f4b200b9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b2015e4f0 0x7f4b2015afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4b200d2ae0 con 0x7f4b2015e4f0 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4b200d2240 con 0x7f4b200d2f70 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4b200d4cf0 con 0x7f4b200d1ac0 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b2015e4f0 0x7f4b2015afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b2015e4f0 0x7f4b2015afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33862/0 (socket says 172.21.15.69:33862) 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1e800640 1 -- 172.21.15.69:0/1282544354 learned_addr learned my addr 172.21.15.69:0/1282544354 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1e800640 1 -- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b200d1ac0 msgr2=0x7f4b20157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1e800640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b200d1ac0 0x7f4b20157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1e800640 1 -- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b200d2f70 msgr2=0x7f4b200b9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1e800640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b200d2f70 0x7f4b200b9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.014+0000 7f4b1e800640 1 -- 172.21.15.69:0/1282544354 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b201580d0 con 0x7f4b2015e4f0 2024-04-24T08:17:59.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.015+0000 7f4b1e800640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b2015e4f0 0x7f4b2015afd0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800c7a0 tx=0x7f4b1800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:59.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.015+0000 7f4b16000640 1 -- 172.21.15.69:0/1282544354 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b18018070 con 0x7f4b2015e4f0 2024-04-24T08:17:59.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.015+0000 7f4b16000640 1 -- 172.21.15.69:0/1282544354 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b1800a640 con 0x7f4b2015e4f0 2024-04-24T08:17:59.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.015+0000 7f4b16000640 1 -- 172.21.15.69:0/1282544354 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1800aab0 con 0x7f4b2015e4f0 2024-04-24T08:17:59.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.015+0000 7f4b1f200640 1 -- 172.21.15.69:0/1282544354 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b20159f30 con 0x7f4b2015e4f0 2024-04-24T08:17:59.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.015+0000 7f4b1f200640 1 -- 172.21.15.69:0/1282544354 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b2015a4f0 con 0x7f4b2015e4f0 2024-04-24T08:17:59.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.016+0000 7f4b1f200640 1 -- 172.21.15.69:0/1282544354 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b200d3910 con 0x7f4b2015e4f0 2024-04-24T08:17:59.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.017+0000 7f4b16000640 1 -- 172.21.15.69:0/1282544354 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4b1800ac50 con 0x7f4b2015e4f0 2024-04-24T08:17:59.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.018+0000 7f4b16000640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4afc06c8f0 0x7f4afc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:59.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.018+0000 7f4b16000640 1 -- 172.21.15.69:0/1282544354 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 5996+0+0 (secure 0 0 0) 0x7f4b1808b300 con 0x7f4b2015e4f0 2024-04-24T08:17:59.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.019+0000 7f4b1de00640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4afc06c8f0 0x7f4afc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:59.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.020+0000 7f4b1de00640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4afc06c8f0 0x7f4afc06edb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4b10006fd0 tx=0x7f4b10008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:59.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.023+0000 7f4b16000640 1 -- 172.21.15.69:0/1282544354 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4b200d3910 con 0x7f4b2015e4f0 2024-04-24T08:17:59.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.221+0000 7f4b1f200640 1 -- 172.21.15.69:0/1282544354 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4b2006ccb0 con 0x7f4b2015e4f0 2024-04-24T08:17:59.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.791+0000 7f4b16000640 1 -- 172.21.15.69:0/1282544354 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v88) ==== 181+0+0 (secure 0 0 0) 0x7f4b18059600 con 0x7f4b2015e4f0 2024-04-24T08:17:59.792 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' removed 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 -- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4afc06c8f0 msgr2=0x7f4afc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4afc06c8f0 0x7f4afc06edb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4b10006fd0 tx=0x7f4b10008040 comp rx=0 tx=0).stop 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 -- 172.21.15.69:0/1282544354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b2015e4f0 msgr2=0x7f4b2015afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b2015e4f0 0x7f4b2015afd0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800c7a0 tx=0x7f4b1800cc60 comp rx=0 tx=0).stop 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 -- 172.21.15.69:0/1282544354 shutdown_connections 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4afc06c8f0 0x7f4afc06edb0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b2015e4f0 0x7f4b2015afd0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b200d2f70 0x7f4b200b9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 --2- 172.21.15.69:0/1282544354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4b200d1ac0 0x7f4b20157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 -- 172.21.15.69:0/1282544354 >> 172.21.15.69:0/1282544354 conn(0x7f4b2009d940 msgr2=0x7f4b20162d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 -- 172.21.15.69:0/1282544354 shutdown_connections 2024-04-24T08:17:59.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:17:59.794+0000 7f4b1f200640 1 -- 172.21.15.69:0/1282544354 wait complete. 2024-04-24T08:17:59.822 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T08:18:00.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.024+0000 7fa725800640 1 Processor -- start 2024-04-24T08:18:00.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.024+0000 7fa725800640 1 -- start start 2024-04-24T08:18:00.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.026+0000 7fa725800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa7200d1b30 0x7fa7200b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.026+0000 7fa725800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7200cdea0 con 0x7fa7200cec60 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.026+0000 7fa725800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7200ce180 con 0x7fa7200b15d0 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.026+0000 7fa725800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa7200d2210 con 0x7fa7200d1b30 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.026+0000 7fa71f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa7200d1b30 0x7fa7200b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.027+0000 7fa71f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa7200d1b30 0x7fa7200b0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33876/0 (socket says 172.21.15.69:33876) 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.027+0000 7fa71f400640 1 -- 172.21.15.69:0/1550122678 learned_addr learned my addr 172.21.15.69:0/1550122678 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.027+0000 7fa716000640 1 --1- 172.21.15.69:0/1550122678 >> v1:172.21.15.69:6789/0 conn(0x7fa7200cec60 0x7fa7200cfa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47356/0 (socket says 172.21.15.69:47356) 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.027+0000 7fa71f400640 1 -- 172.21.15.69:0/1550122678 >> v1:172.21.15.121:6789/0 conn(0x7fa7200b15d0 legacy=0x7fa7200b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.027+0000 7fa71f400640 1 -- 172.21.15.69:0/1550122678 >> v1:172.21.15.69:6789/0 conn(0x7fa7200cec60 legacy=0x7fa7200cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.027+0000 7fa71e000640 1 -- 172.21.15.69:0/1550122678 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 666766346 0 0) 0x7fa7200cdea0 con 0x7fa7200cec60 2024-04-24T08:18:00.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.027+0000 7fa71f400640 1 -- 172.21.15.69:0/1550122678 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7200ce460 con 0x7fa7200d1b30 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.028+0000 7fa71f400640 1 --2- 172.21.15.69:0/1550122678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200d1b30 0x7fa7200b0bd0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fa718009870 tx=0x7fa71802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19d3d770deb10e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.028+0000 7fa71e000640 1 -- 172.21.15.69:0/1550122678 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718037070 con 0x7fa7200d1b30 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.029+0000 7fa71e000640 1 -- 172.21.15.69:0/1550122678 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa718032440 con 0x7fa7200d1b30 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.029+0000 7fa71e000640 1 -- 172.21.15.69:0/1550122678 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718032810 con 0x7fa7200d1b30 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.029+0000 7fa725800640 1 -- 172.21.15.69:0/1550122678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200d1b30 msgr2=0x7fa7200b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.029+0000 7fa725800640 1 --2- 172.21.15.69:0/1550122678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200d1b30 0x7fa7200b0bd0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fa718009870 tx=0x7fa71802a2a0 comp rx=0 tx=0).stop 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.029+0000 7fa725800640 1 -- 172.21.15.69:0/1550122678 shutdown_connections 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.029+0000 7fa725800640 1 --2- 172.21.15.69:0/1550122678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200d1b30 0x7fa7200b0bd0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.029+0000 7fa725800640 1 -- 172.21.15.69:0/1550122678 >> 172.21.15.69:0/1550122678 conn(0x7fa7200b6800 msgr2=0x7fa7200b6c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.029+0000 7fa725800640 1 -- 172.21.15.69:0/1550122678 shutdown_connections 2024-04-24T08:18:00.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.029+0000 7fa725800640 1 -- 172.21.15.69:0/1550122678 wait complete. 2024-04-24T08:18:00.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.031+0000 7fa725800640 1 Processor -- start 2024-04-24T08:18:00.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.031+0000 7fa725800640 1 -- start start 2024-04-24T08:18:00.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.032+0000 7fa725800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200b15d0 0x7fa72014e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:00.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.032+0000 7fa725800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7200cec60 0x7fa7200ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:00.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.032+0000 7fa725800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7200d1b30 0x7fa72015a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:00.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.032+0000 7fa725800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa7200d0140 con 0x7fa7200b15d0 2024-04-24T08:18:00.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.032+0000 7fa725800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa7200d1f30 con 0x7fa7200d1b30 2024-04-24T08:18:00.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.032+0000 7fa725800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa7200d02c0 con 0x7fa7200cec60 2024-04-24T08:18:00.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.032+0000 7fa71ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200b15d0 0x7fa72014e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:00.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa71ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200b15d0 0x7fa72014e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33878/0 (socket says 172.21.15.69:33878) 2024-04-24T08:18:00.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa71ea00640 1 -- 172.21.15.69:0/4238151828 learned_addr learned my addr 172.21.15.69:0/4238151828 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:00.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa71f400640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7200d1b30 0x7fa72015a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:00.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa716000640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7200cec60 0x7fa7200ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:00.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa71ea00640 1 -- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7200cec60 msgr2=0x7fa7200ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:00.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa71ea00640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7200cec60 0x7fa7200ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:00.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa71ea00640 1 -- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7200d1b30 msgr2=0x7fa72015a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:00.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa71ea00640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7200d1b30 0x7fa72015a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:00.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa71ea00640 1 -- 172.21.15.69:0/4238151828 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7200b4d70 con 0x7fa7200b15d0 2024-04-24T08:18:00.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa716000640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7200cec60 0x7fa7200ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:00.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.033+0000 7fa71ea00640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200b15d0 0x7fa72014e030 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fa71000c7c0 tx=0x7fa71000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:00.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.034+0000 7fa717400640 1 -- 172.21.15.69:0/4238151828 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa710016020 con 0x7fa7200b15d0 2024-04-24T08:18:00.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.034+0000 7fa717400640 1 -- 172.21.15.69:0/4238151828 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa71000a9f0 con 0x7fa7200b15d0 2024-04-24T08:18:00.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.034+0000 7fa725800640 1 -- 172.21.15.69:0/4238151828 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7201543a0 con 0x7fa7200b15d0 2024-04-24T08:18:00.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.034+0000 7fa717400640 1 -- 172.21.15.69:0/4238151828 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71000ae20 con 0x7fa7200b15d0 2024-04-24T08:18:00.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.034+0000 7fa725800640 1 -- 172.21.15.69:0/4238151828 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7200b52f0 con 0x7fa7200b15d0 2024-04-24T08:18:00.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.035+0000 7fa725800640 1 -- 172.21.15.69:0/4238151828 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa720068a80 con 0x7fa7200b15d0 2024-04-24T08:18:00.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.036+0000 7fa717400640 1 -- 172.21.15.69:0/4238151828 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa710019020 con 0x7fa7200b15d0 2024-04-24T08:18:00.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.037+0000 7fa717400640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6fc06c8f0 0x7fa6fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:00.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.037+0000 7fa717400640 1 -- 172.21.15.69:0/4238151828 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 5558+0+0 (secure 0 0 0) 0x7fa71008b850 con 0x7fa7200b15d0 2024-04-24T08:18:00.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.037+0000 7fa716000640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6fc06c8f0 0x7fa6fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:00.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.039+0000 7fa716000640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6fc06c8f0 0x7fa6fc06edb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa7200cd6f0 tx=0x7fa708009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:00.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.041+0000 7fa717400640 1 -- 172.21.15.69:0/4238151828 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa710020070 con 0x7fa7200b15d0 2024-04-24T08:18:00.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.245+0000 7fa725800640 1 -- 172.21.15.69:0/4238151828 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa72006f7b0 con 0x7fa7200b15d0 2024-04-24T08:18:00.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.805+0000 7fa717400640 1 -- 172.21.15.69:0/4238151828 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v89) ==== 169+0+0 (secure 0 0 0) 0x7fa710059c70 con 0x7fa7200b15d0 2024-04-24T08:18:00.808 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' removed 2024-04-24T08:18:00.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.809+0000 7fa725800640 1 -- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6fc06c8f0 msgr2=0x7fa6fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:00.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.809+0000 7fa725800640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6fc06c8f0 0x7fa6fc06edb0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa7200cd6f0 tx=0x7fa708009040 comp rx=0 tx=0).stop 2024-04-24T08:18:00.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.809+0000 7fa725800640 1 -- 172.21.15.69:0/4238151828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200b15d0 msgr2=0x7fa72014e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:00.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.809+0000 7fa725800640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200b15d0 0x7fa72014e030 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7fa71000c7c0 tx=0x7fa71000cc80 comp rx=0 tx=0).stop 2024-04-24T08:18:00.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.809+0000 7fa725800640 1 -- 172.21.15.69:0/4238151828 shutdown_connections 2024-04-24T08:18:00.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.809+0000 7fa725800640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa6fc06c8f0 0x7fa6fc06edb0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:00.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.809+0000 7fa725800640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7200d1b30 0x7fa72015a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:00.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.809+0000 7fa725800640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa7200cec60 0x7fa7200ca660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:00.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.809+0000 7fa725800640 1 --2- 172.21.15.69:0/4238151828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7200b15d0 0x7fa72014e030 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:00.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.809+0000 7fa725800640 1 -- 172.21.15.69:0/4238151828 >> 172.21.15.69:0/4238151828 conn(0x7fa7200b6800 msgr2=0x7fa72006ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:00.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.810+0000 7fa725800640 1 -- 172.21.15.69:0/4238151828 shutdown_connections 2024-04-24T08:18:00.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:00.810+0000 7fa725800640 1 -- 172.21.15.69:0/4238151828 wait complete. 2024-04-24T08:18:00.843 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-24T08:18:01.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.032+0000 7f7fc2800640 1 Processor -- start 2024-04-24T08:18:01.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.032+0000 7f7fc2800640 1 -- start start 2024-04-24T08:18:01.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc2800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7fbc15e980 0x7f7fbc162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc2800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fbc0d0da0 0x7f7fbc0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc2800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7fbc15ed60 con 0x7f7fbc0d0190 2024-04-24T08:18:01.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc2800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7fbc0d1580 con 0x7f7fbc0d0da0 2024-04-24T08:18:01.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc2800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7fbc0d1860 con 0x7f7fbc15e980 2024-04-24T08:18:01.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc0a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fbc0d0da0 0x7f7fbc0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:01.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc0a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7fbc0d0da0 0x7f7fbc0cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33892/0 (socket says 172.21.15.69:33892) 2024-04-24T08:18:01.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc0a00640 1 -- 172.21.15.69:0/1087549870 learned_addr learned my addr 172.21.15.69:0/1087549870 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:01.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc1400640 1 --2- 172.21.15.69:0/1087549870 >> v2:172.21.15.121:3300/0 conn(0x7f7fbc15e980 0x7f7fbc162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:01.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc0a00640 1 -- 172.21.15.69:0/1087549870 >> v2:172.21.15.121:3300/0 conn(0x7f7fbc15e980 msgr2=0x7f7fbc162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:01.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.034+0000 7f7fc0a00640 1 --2- 172.21.15.69:0/1087549870 >> v2:172.21.15.121:3300/0 conn(0x7f7fbc15e980 0x7f7fbc162f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.035+0000 7f7fc0a00640 1 -- 172.21.15.69:0/1087549870 >> v1:172.21.15.69:6789/0 conn(0x7f7fbc0d0190 legacy=0x7f7fbc06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.035+0000 7f7fc0a00640 1 -- 172.21.15.69:0/1087549870 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc0cf450 con 0x7f7fbc0d0da0 2024-04-24T08:18:01.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.036+0000 7f7fc0a00640 1 --2- 172.21.15.69:0/1087549870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0da0 0x7f7fbc0cfc50 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc09de50 tx=0x7f7fa802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca7fa70cdb6550f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:01.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.036+0000 7f7fbbe00640 1 -- 172.21.15.69:0/1087549870 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8037070 con 0x7f7fbc0d0da0 2024-04-24T08:18:01.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.036+0000 7f7fbbe00640 1 -- 172.21.15.69:0/1087549870 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fa802ae50 con 0x7f7fbc0d0da0 2024-04-24T08:18:01.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.036+0000 7f7fbbe00640 1 -- 172.21.15.69:0/1087549870 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fa8032860 con 0x7f7fbc0d0da0 2024-04-24T08:18:01.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.037+0000 7f7fc2800640 1 -- 172.21.15.69:0/1087549870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0da0 msgr2=0x7f7fbc0cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.037+0000 7f7fc2800640 1 --2- 172.21.15.69:0/1087549870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0da0 0x7f7fbc0cfc50 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc09de50 tx=0x7f7fa802a300 comp rx=0 tx=0).stop 2024-04-24T08:18:01.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.037+0000 7f7fc2800640 1 -- 172.21.15.69:0/1087549870 shutdown_connections 2024-04-24T08:18:01.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.037+0000 7f7fc2800640 1 --2- 172.21.15.69:0/1087549870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0da0 0x7f7fbc0cfc50 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.037+0000 7f7fc2800640 1 --2- 172.21.15.69:0/1087549870 >> v2:172.21.15.121:3300/0 conn(0x7f7fbc15e980 0x7f7fbc162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.037+0000 7f7fc2800640 1 -- 172.21.15.69:0/1087549870 >> 172.21.15.69:0/1087549870 conn(0x7f7fbc09d920 msgr2=0x7f7fbc08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:01.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.037+0000 7f7fc2800640 1 -- 172.21.15.69:0/1087549870 shutdown_connections 2024-04-24T08:18:01.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.037+0000 7f7fc2800640 1 -- 172.21.15.69:0/1087549870 wait complete. 2024-04-24T08:18:01.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.038+0000 7f7fc2800640 1 Processor -- start 2024-04-24T08:18:01.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.039+0000 7f7fc2800640 1 -- start start 2024-04-24T08:18:01.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.039+0000 7f7fc2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0190 0x7f7fbc153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.039+0000 7f7fc2800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fbc0d0da0 0x7f7fbc0c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc2800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fbc15e980 0x7f7fbc0b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc2800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc0d1360 con 0x7f7fbc0d0190 2024-04-24T08:18:01.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc2800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7fbc0d11e0 con 0x7f7fbc15e980 2024-04-24T08:18:01.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc2800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7fbc0cf070 con 0x7f7fbc0d0da0 2024-04-24T08:18:01.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0190 0x7f7fbc153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:01.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0190 0x7f7fbc153010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33904/0 (socket says 172.21.15.69:33904) 2024-04-24T08:18:01.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc1400640 1 -- 172.21.15.69:0/1991628234 learned_addr learned my addr 172.21.15.69:0/1991628234 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc1400640 1 -- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fbc0d0da0 msgr2=0x7f7fbc0c8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc1400640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fbc0d0da0 0x7f7fbc0c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc1400640 1 -- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fbc15e980 msgr2=0x7f7fbc0b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc1400640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fbc15e980 0x7f7fbc0b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.040+0000 7f7fc1400640 1 -- 172.21.15.69:0/1991628234 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fbc0c9810 con 0x7f7fbc0d0190 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.041+0000 7f7fc1400640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0190 0x7f7fbc153010 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00ce20 tx=0x7f7fac00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.041+0000 7f7fb9600640 1 -- 172.21.15.69:0/1991628234 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac003b90 con 0x7f7fbc0d0190 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.041+0000 7f7fb9600640 1 -- 172.21.15.69:0/1991628234 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fac003d30 con 0x7f7fbc0d0190 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.041+0000 7f7fb9600640 1 -- 172.21.15.69:0/1991628234 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fac004260 con 0x7f7fbc0d0190 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.041+0000 7f7fc2800640 1 -- 172.21.15.69:0/1991628234 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fbc0c8f70 con 0x7f7fbc0d0190 2024-04-24T08:18:01.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.041+0000 7f7fc2800640 1 -- 172.21.15.69:0/1991628234 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fbc0c9530 con 0x7f7fbc0d0190 2024-04-24T08:18:01.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.042+0000 7f7fc2800640 1 -- 172.21.15.69:0/1991628234 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fbc075d50 con 0x7f7fbc0d0190 2024-04-24T08:18:01.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.043+0000 7f7fb9600640 1 -- 172.21.15.69:0/1991628234 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fac004400 con 0x7f7fbc0d0190 2024-04-24T08:18:01.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.044+0000 7f7fb9600640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f9c06c920 0x7f7f9c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.044+0000 7f7fb9600640 1 -- 172.21.15.69:0/1991628234 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5115+0+0 (secure 0 0 0) 0x7f7fac08c4a0 con 0x7f7fbc0d0190 2024-04-24T08:18:01.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.045+0000 7f7fc0a00640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f9c06c920 0x7f7f9c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:01.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.046+0000 7f7fc0a00640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f9c06c920 0x7f7f9c06ede0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8034000 tx=0x7f7fa802c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:01.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.049+0000 7f7fb9600640 1 -- 172.21.15.69:0/1991628234 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fac01d340 con 0x7f7fbc0d0190 2024-04-24T08:18:01.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.249+0000 7f7fc2800640 1 -- 172.21.15.69:0/1991628234 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f7fbc153860 con 0x7f7fbc0d0190 2024-04-24T08:18:01.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.256+0000 7f7fb9600640 1 -- 172.21.15.69:0/1991628234 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) ==== 104+0+0 (secure 0 0 0) 0x7f7fac05aa80 con 0x7f7fbc0d0190 2024-04-24T08:18:01.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 -- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f9c06c920 msgr2=0x7f7f9c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f9c06c920 0x7f7f9c06ede0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7fa8034000 tx=0x7f7fa802c000 comp rx=0 tx=0).stop 2024-04-24T08:18:01.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 -- 172.21.15.69:0/1991628234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0190 msgr2=0x7f7fbc153010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0190 0x7f7fbc153010 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00ce20 tx=0x7f7fac00a660 comp rx=0 tx=0).stop 2024-04-24T08:18:01.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 -- 172.21.15.69:0/1991628234 shutdown_connections 2024-04-24T08:18:01.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7f9c06c920 0x7f7f9c06ede0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7fbc15e980 0x7f7fbc0b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7fbc0d0da0 0x7f7fbc0c8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 --2- 172.21.15.69:0/1991628234 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7fbc0d0190 0x7f7fbc153010 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 -- 172.21.15.69:0/1991628234 >> 172.21.15.69:0/1991628234 conn(0x7f7fbc09d920 msgr2=0x7f7fbc074cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:01.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 -- 172.21.15.69:0/1991628234 shutdown_connections 2024-04-24T08:18:01.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.258+0000 7f7fc2800640 1 -- 172.21.15.69:0/1991628234 wait complete. 2024-04-24T08:18:01.278 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-04-24T08:18:01.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.472+0000 7f5b85800640 1 Processor -- start 2024-04-24T08:18:01.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.472+0000 7f5b85800640 1 -- start start 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.473+0000 7f5b85800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5b80145fe0 0x7f5b8006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.473+0000 7f5b85800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b800cf3d0 con 0x7f5b800d0300 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.473+0000 7f5b85800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b800cf6b0 con 0x7f5b80145a90 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.473+0000 7f5b85800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5b800cf990 con 0x7f5b80145fe0 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.474+0000 7f5b7ea00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5b800d0300 0x7f5b800d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47372/0 (socket says 172.21.15.69:47372) 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.474+0000 7f5b7ea00640 1 -- 172.21.15.69:0/650840691 learned_addr learned my addr 172.21.15.69:0/650840691 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.474+0000 7f5b7f400640 1 --2- 172.21.15.69:0/650840691 >> v2:172.21.15.69:3300/0 conn(0x7f5b80145fe0 0x7f5b8006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.474+0000 7f5b7e000640 1 -- 172.21.15.69:0/650840691 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4051109430 0 0) 0x7f5b800cf3d0 con 0x7f5b800d0300 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.474+0000 7f5b7e000640 1 -- 172.21.15.69:0/650840691 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b64003520 con 0x7f5b800d0300 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.474+0000 7f5b7f400640 1 -- 172.21.15.69:0/650840691 >> v1:172.21.15.121:6789/0 conn(0x7f5b80145a90 legacy=0x7f5b80160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.474+0000 7f5b7f400640 1 -- 172.21.15.69:0/650840691 >> v1:172.21.15.69:6789/0 conn(0x7f5b800d0300 legacy=0x7f5b800d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.474+0000 7f5b7f400640 1 -- 172.21.15.69:0/650840691 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b800cfc70 con 0x7f5b80145fe0 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.475+0000 7f5b7f400640 1 --2- 172.21.15.69:0/650840691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b80145fe0 0x7f5b8006e240 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009d90 tx=0x7f5b7802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5dff654adce81ea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.475+0000 7f5b7e000640 1 -- 172.21.15.69:0/650840691 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78007740 con 0x7f5b80145fe0 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.475+0000 7f5b7e000640 1 -- 172.21.15.69:0/650840691 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b780078e0 con 0x7f5b80145fe0 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.475+0000 7f5b7e000640 1 -- 172.21.15.69:0/650840691 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b78007cb0 con 0x7f5b80145fe0 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.476+0000 7f5b85800640 1 -- 172.21.15.69:0/650840691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b80145fe0 msgr2=0x7f5b8006e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.476+0000 7f5b85800640 1 --2- 172.21.15.69:0/650840691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b80145fe0 0x7f5b8006e240 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009d90 tx=0x7f5b7802a6a0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.476+0000 7f5b85800640 1 -- 172.21.15.69:0/650840691 shutdown_connections 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.476+0000 7f5b85800640 1 --2- 172.21.15.69:0/650840691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b80145fe0 0x7f5b8006e240 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.476+0000 7f5b85800640 1 -- 172.21.15.69:0/650840691 >> 172.21.15.69:0/650840691 conn(0x7f5b80067d90 msgr2=0x7f5b8006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.476+0000 7f5b85800640 1 -- 172.21.15.69:0/650840691 shutdown_connections 2024-04-24T08:18:01.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.476+0000 7f5b85800640 1 -- 172.21.15.69:0/650840691 wait complete. 2024-04-24T08:18:01.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.478+0000 7f5b85800640 1 Processor -- start 2024-04-24T08:18:01.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.478+0000 7f5b85800640 1 -- start start 2024-04-24T08:18:01.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.479+0000 7f5b85800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b800d0300 0x7f5b8013f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.479+0000 7f5b85800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b80145a90 0x7f5b8013d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.479+0000 7f5b85800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b80145fe0 0x7f5b8013df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.479+0000 7f5b85800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5b801458f0 con 0x7f5b800d0300 2024-04-24T08:18:01.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.479+0000 7f5b85800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5b80145770 con 0x7f5b80145a90 2024-04-24T08:18:01.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.479+0000 7f5b85800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5b800cdc30 con 0x7f5b80145fe0 2024-04-24T08:18:01.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.479+0000 7f5b7ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b800d0300 0x7f5b8013f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:01.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.479+0000 7f5b7ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b800d0300 0x7f5b8013f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33926/0 (socket says 172.21.15.69:33926) 2024-04-24T08:18:01.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.479+0000 7f5b7ea00640 1 -- 172.21.15.69:0/424823548 learned_addr learned my addr 172.21.15.69:0/424823548 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:01.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b7ea00640 1 -- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b80145fe0 msgr2=0x7f5b8013df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b7ea00640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b80145fe0 0x7f5b8013df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b7ea00640 1 -- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b80145a90 msgr2=0x7f5b8013d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b7ea00640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b80145a90 0x7f5b8013d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b7ea00640 1 -- 172.21.15.69:0/424823548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b80142c20 con 0x7f5b800d0300 2024-04-24T08:18:01.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b7ea00640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b800d0300 0x7f5b8013f240 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000c390 tx=0x7f5b7000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:01.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b76a00640 1 -- 172.21.15.69:0/424823548 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b70017070 con 0x7f5b800d0300 2024-04-24T08:18:01.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b76a00640 1 -- 172.21.15.69:0/424823548 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b70005ce0 con 0x7f5b800d0300 2024-04-24T08:18:01.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b76a00640 1 -- 172.21.15.69:0/424823548 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b700072d0 con 0x7f5b800d0300 2024-04-24T08:18:01.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b85800640 1 -- 172.21.15.69:0/424823548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b80142f00 con 0x7f5b800d0300 2024-04-24T08:18:01.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.480+0000 7f5b85800640 1 -- 172.21.15.69:0/424823548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b801434c0 con 0x7f5b800d0300 2024-04-24T08:18:01.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.481+0000 7f5b85800640 1 -- 172.21.15.69:0/424823548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b800cdf60 con 0x7f5b800d0300 2024-04-24T08:18:01.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.482+0000 7f5b76a00640 1 -- 172.21.15.69:0/424823548 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5b7001a070 con 0x7f5b800d0300 2024-04-24T08:18:01.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.483+0000 7f5b76a00640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5b5c06c8f0 0x7f5b5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:01.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.483+0000 7f5b76a00640 1 -- 172.21.15.69:0/424823548 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5115+0+0 (secure 0 0 0) 0x7f5b7008ac40 con 0x7f5b800d0300 2024-04-24T08:18:01.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.484+0000 7f5b77e00640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5b5c06c8f0 0x7f5b5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:01.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.485+0000 7f5b77e00640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5b5c06c8f0 0x7f5b5c06edb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5b8013eea0 tx=0x7f5b680079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:01.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.488+0000 7f5b76a00640 1 -- 172.21.15.69:0/424823548 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b800cdf60 con 0x7f5b800d0300 2024-04-24T08:18:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.756+0000 7f5b85800640 1 -- 172.21.15.69:0/424823548 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) -- 0x7f5b80068a80 con 0x7f5b800d0300 2024-04-24T08:18:01.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.828+0000 7f5b76a00640 1 -- 172.21.15.69:0/424823548 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v451) ==== 167+0+0 (secure 0 0 0) 0x7f5b70058bd0 con 0x7f5b800d0300 2024-04-24T08:18:01.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 -- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5b5c06c8f0 msgr2=0x7f5b5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5b5c06c8f0 0x7f5b5c06edb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5b8013eea0 tx=0x7f5b680079b0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 -- 172.21.15.69:0/424823548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b800d0300 msgr2=0x7f5b8013f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:01.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b800d0300 0x7f5b8013f240 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000c390 tx=0x7f5b7000c850 comp rx=0 tx=0).stop 2024-04-24T08:18:01.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 -- 172.21.15.69:0/424823548 shutdown_connections 2024-04-24T08:18:01.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5b5c06c8f0 0x7f5b5c06edb0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b80145fe0 0x7f5b8013df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b80145a90 0x7f5b8013d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 --2- 172.21.15.69:0/424823548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b800d0300 0x7f5b8013f240 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:01.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 -- 172.21.15.69:0/424823548 >> 172.21.15.69:0/424823548 conn(0x7f5b80067d90 msgr2=0x7f5b8006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:01.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 -- 172.21.15.69:0/424823548 shutdown_connections 2024-04-24T08:18:01.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:01.833+0000 7f5b85800640 1 -- 172.21.15.69:0/424823548 wait complete. 2024-04-24T08:18:01.853 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-04-24T08:18:01.853 INFO:tasks.cephfs_test_runner:Starting test: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-24T08:18:01.854 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:02.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.050+0000 7f897e800640 1 Processor -- start 2024-04-24T08:18:02.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.050+0000 7f897e800640 1 -- start start 2024-04-24T08:18:02.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.051+0000 7f897e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89780d0300 0x7f897806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.051+0000 7f897e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8978145fe0 0x7f8978160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.051+0000 7f897e800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89780cf3d0 con 0x7f8978145a90 2024-04-24T08:18:02.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.052+0000 7f897e800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f89780cf6b0 con 0x7f89780d0300 2024-04-24T08:18:02.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.052+0000 7f897e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f89780cf990 con 0x7f8978145fe0 2024-04-24T08:18:02.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.052+0000 7f8977400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89780d0300 0x7f897806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.052+0000 7f8977400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f89780d0300 0x7f897806e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33928/0 (socket says 172.21.15.69:33928) 2024-04-24T08:18:02.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.052+0000 7f8977400640 1 -- 172.21.15.69:0/2518080709 learned_addr learned my addr 172.21.15.69:0/2518080709 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:02.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.052+0000 7f8976a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8978145a90 0x7f89780d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47376/0 (socket says 172.21.15.69:47376) 2024-04-24T08:18:02.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.052+0000 7f8977e00640 1 --2- 172.21.15.69:0/2518080709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8978145fe0 0x7f8978160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.052+0000 7f8977400640 1 -- 172.21.15.69:0/2518080709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8978145fe0 msgr2=0x7f8978160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:02.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.053+0000 7f8977400640 1 --2- 172.21.15.69:0/2518080709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8978145fe0 0x7f8978160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.053+0000 7f8977400640 1 -- 172.21.15.69:0/2518080709 >> v1:172.21.15.69:6789/0 conn(0x7f8978145a90 legacy=0x7f89780d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.053+0000 7f8977400640 1 -- 172.21.15.69:0/2518080709 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89780cfc70 con 0x7f89780d0300 2024-04-24T08:18:02.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.054+0000 7f8977400640 1 --2- 172.21.15.69:0/2518080709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89780d0300 0x7f897806e240 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f8968009870 tx=0x7f896802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa74f59ebf833001 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.054+0000 7f8976000640 1 -- 172.21.15.69:0/2518080709 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8968037070 con 0x7f89780d0300 2024-04-24T08:18:02.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.054+0000 7f8976000640 1 -- 172.21.15.69:0/2518080709 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8968032440 con 0x7f89780d0300 2024-04-24T08:18:02.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.054+0000 7f8976000640 1 -- 172.21.15.69:0/2518080709 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8968032810 con 0x7f89780d0300 2024-04-24T08:18:02.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.054+0000 7f897e800640 1 -- 172.21.15.69:0/2518080709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89780d0300 msgr2=0x7f897806e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.054+0000 7f897e800640 1 --2- 172.21.15.69:0/2518080709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89780d0300 0x7f897806e240 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f8968009870 tx=0x7f896802a250 comp rx=0 tx=0).stop 2024-04-24T08:18:02.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.055+0000 7f897e800640 1 -- 172.21.15.69:0/2518080709 shutdown_connections 2024-04-24T08:18:02.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.055+0000 7f897e800640 1 --2- 172.21.15.69:0/2518080709 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8978145fe0 0x7f8978160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.055+0000 7f897e800640 1 --2- 172.21.15.69:0/2518080709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89780d0300 0x7f897806e240 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.055+0000 7f897e800640 1 -- 172.21.15.69:0/2518080709 >> 172.21.15.69:0/2518080709 conn(0x7f8978067d90 msgr2=0x7f897806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:02.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.055+0000 7f897e800640 1 -- 172.21.15.69:0/2518080709 shutdown_connections 2024-04-24T08:18:02.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.055+0000 7f897e800640 1 -- 172.21.15.69:0/2518080709 wait complete. 2024-04-24T08:18:02.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.056+0000 7f897e800640 1 Processor -- start 2024-04-24T08:18:02.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.057+0000 7f897e800640 1 -- start start 2024-04-24T08:18:02.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.057+0000 7f897e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89780d0300 0x7f89780b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.057+0000 7f897e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8978145a90 0x7f89781e9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.057+0000 7f897e800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8978145fe0 0x7f89781e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.057+0000 7f897e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89781458f0 con 0x7f8978145a90 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.057+0000 7f897e800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8978145770 con 0x7f8978145fe0 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.057+0000 7f897e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f89780cdc30 con 0x7f89780d0300 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.058+0000 7f8976a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8978145a90 0x7f89781e9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.058+0000 7f8976a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8978145a90 0x7f89781e9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33942/0 (socket says 172.21.15.69:33942) 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.058+0000 7f8976a00640 1 -- 172.21.15.69:0/2416744092 learned_addr learned my addr 172.21.15.69:0/2416744092 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.058+0000 7f8977400640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89780d0300 0x7f89780b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.058+0000 7f8976a00640 1 -- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89780d0300 msgr2=0x7f89780b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.058+0000 7f8976a00640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89780d0300 0x7f89780b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.058+0000 7f8976a00640 1 -- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8978145fe0 msgr2=0x7f89781e9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.058+0000 7f8977e00640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8978145fe0 0x7f89781e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.058+0000 7f8976a00640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8978145fe0 0x7f89781e9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.058+0000 7f8976a00640 1 -- 172.21.15.69:0/2416744092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89781f4e30 con 0x7f8978145a90 2024-04-24T08:18:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.059+0000 7f8976a00640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8978145a90 0x7f89781e9aa0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f896400c400 tx=0x7f896400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.059+0000 7f896f400640 1 -- 172.21.15.69:0/2416744092 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89640074e0 con 0x7f8978145a90 2024-04-24T08:18:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.059+0000 7f896f400640 1 -- 172.21.15.69:0/2416744092 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8964007680 con 0x7f8978145a90 2024-04-24T08:18:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.059+0000 7f896f400640 1 -- 172.21.15.69:0/2416744092 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8964004a00 con 0x7f8978145a90 2024-04-24T08:18:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.059+0000 7f897e800640 1 -- 172.21.15.69:0/2416744092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89781f5110 con 0x7f8978145a90 2024-04-24T08:18:02.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.059+0000 7f897e800640 1 -- 172.21.15.69:0/2416744092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89781f56d0 con 0x7f8978145a90 2024-04-24T08:18:02.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.060+0000 7f897e800640 1 -- 172.21.15.69:0/2416744092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8978068a80 con 0x7f8978145a90 2024-04-24T08:18:02.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.061+0000 7f896f400640 1 -- 172.21.15.69:0/2416744092 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f896401a020 con 0x7f8978145a90 2024-04-24T08:18:02.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.062+0000 7f896f400640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f895806c8f0 0x7f895806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.062+0000 7f896f400640 1 -- 172.21.15.69:0/2416744092 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5115+0+0 (secure 0 0 0) 0x7f896408b1b0 con 0x7f8978145a90 2024-04-24T08:18:02.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.063+0000 7f8977400640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f895806c8f0 0x7f895806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.064+0000 7f8977400640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f895806c8f0 0x7f895806edb0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8968034000 tx=0x7f896802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.066+0000 7f896f400640 1 -- 172.21.15.69:0/2416744092 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8964059830 con 0x7f8978145a90 2024-04-24T08:18:02.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.266+0000 7f897e800640 1 -- 172.21.15.69:0/2416744092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f897806eb60 con 0x7f8978145a90 2024-04-24T08:18:02.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.268+0000 7f896f400640 1 -- 172.21.15.69:0/2416744092 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+13259 (secure 0 0 0) 0x7f8964059130 con 0x7f8978145a90 2024-04-24T08:18:02.270 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:02.270 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:18:00.803796+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6833/4254348458":"2024-04-25T08:17:57.695996+0000","172.21.15.69:6832/4254348458":"2024-04-25T08:17:57.695996+0000","172.21.15.69:6832/2529876410":"2024-04-25T08:17:08.931681+0000","172.21.15.121:6837/1179683245":"2024-04-25T08:15:45.259762+0000","172.21.15.69:6833/2529876410":"2024-04-25T08:17:08.931681+0000","172.21.15.121:6835/1179683245":"2024-04-25T08:15:45.259762+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 -- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f895806c8f0 msgr2=0x7f895806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f895806c8f0 0x7f895806edb0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f8968034000 tx=0x7f896802e000 comp rx=0 tx=0).stop 2024-04-24T08:18:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 -- 172.21.15.69:0/2416744092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8978145a90 msgr2=0x7f89781e9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8978145a90 0x7f89781e9aa0 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f896400c400 tx=0x7f896400c8c0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 -- 172.21.15.69:0/2416744092 shutdown_connections 2024-04-24T08:18:02.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f895806c8f0 0x7f895806edb0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8978145a90 0x7f89781e9aa0 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8978145fe0 0x7f89781e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 --2- 172.21.15.69:0/2416744092 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89780d0300 0x7f89780b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 -- 172.21.15.69:0/2416744092 >> 172.21.15.69:0/2416744092 conn(0x7f8978067d90 msgr2=0x7f89780b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.270+0000 7f897e800640 1 -- 172.21.15.69:0/2416744092 shutdown_connections 2024-04-24T08:18:02.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.271+0000 7f897e800640 1 -- 172.21.15.69:0/2416744092 wait complete. 2024-04-24T08:18:02.291 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:18:02.291 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:18:02.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.495+0000 7f6611c00640 1 Processor -- start 2024-04-24T08:18:02.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.495+0000 7f6611c00640 1 -- start start 2024-04-24T08:18:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.495+0000 7f6611c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c15c820 0x7f660c0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.495+0000 7f6611c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f660c0b46a0 0x7f660c06f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.495+0000 7f6611c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f660c15cc00 con 0x7f660c06fd10 2024-04-24T08:18:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.495+0000 7f6611c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f660c06ed30 con 0x7f660c0b46a0 2024-04-24T08:18:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.495+0000 7f6611c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f660c0b4e10 con 0x7f660c15c820 2024-04-24T08:18:02.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.496+0000 7f660b400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f660c06fd10 0x7f660c0ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47384/0 (socket says 172.21.15.69:47384) 2024-04-24T08:18:02.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.496+0000 7f660b400640 1 -- 172.21.15.69:0/1125562965 learned_addr learned my addr 172.21.15.69:0/1125562965 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.496+0000 7f6603e00640 1 --2- 172.21.15.69:0/1125562965 >> v2:172.21.15.121:3300/0 conn(0x7f660c0b46a0 0x7f660c06f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.496+0000 7f660aa00640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c15c820 0x7f660c0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.496+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2792031056 0 0) 0x7f660c15cc00 con 0x7f660c06fd10 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.496+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65f0003520 con 0x7f660c06fd10 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3712163618 0 0) 0x7f65f0003520 con 0x7f660c06fd10 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c15c820 msgr2=0x7f660c0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c15c820 0x7f660c0b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 >> v2:172.21.15.121:3300/0 conn(0x7f660c0b46a0 msgr2=0x7f660c06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 --2- 172.21.15.69:0/1125562965 >> v2:172.21.15.121:3300/0 conn(0x7f660c0b46a0 0x7f660c06f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c0b50f0 con 0x7f660c06fd10 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f66040019f0 con 0x7f660c06fd10 2024-04-24T08:18:02.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 >> v1:172.21.15.69:6789/0 conn(0x7f660c06fd10 legacy=0x7f660c0ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c15c820 0x7f65f00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65f0003f30 0x7f65f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f0004d40 0x7f65f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f660c06ed30 con 0x7f65f0003f30 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f660c0b4e10 con 0x7f65f0004d40 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f65f000b9c0 con 0x7f660c15c820 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f6603e00640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65f0003f30 0x7f65f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.497+0000 7f660aa00640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c15c820 0x7f65f00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.498+0000 7f660b400640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f0004d40 0x7f65f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.498+0000 7f6603e00640 1 -- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c15c820 msgr2=0x7f65f00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.498+0000 7f6603e00640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c15c820 0x7f65f00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.498+0000 7f6603e00640 1 -- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f0004d40 msgr2=0x7f65f00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.498+0000 7f6603e00640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f0004d40 0x7f65f00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.498+0000 7f6603e00640 1 -- 172.21.15.69:0/1125562965 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f0003520 con 0x7f65f0003f30 2024-04-24T08:18:02.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.498+0000 7f660b400640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f0004d40 0x7f65f00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:02.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.499+0000 7f6603e00640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65f0003f30 0x7f65f0004800 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009ac0 tx=0x7f65f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6bdbd906ae9dba server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.499+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4036070 con 0x7f65f0003f30 2024-04-24T08:18:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.499+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65f4031400 con 0x7f65f0003f30 2024-04-24T08:18:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.499+0000 7f660a000640 1 -- 172.21.15.69:0/1125562965 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f4009690 con 0x7f65f0003f30 2024-04-24T08:18:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.499+0000 7f6611c00640 1 -- 172.21.15.69:0/1125562965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65f0003f30 msgr2=0x7f65f0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.499+0000 7f6611c00640 1 --2- 172.21.15.69:0/1125562965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65f0003f30 0x7f65f0004800 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f65f4009ac0 tx=0x7f65f0005830 comp rx=0 tx=0).stop 2024-04-24T08:18:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.499+0000 7f660b400640 1 -- 172.21.15.69:0/1125562965 reap_dead start 2024-04-24T08:18:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.500+0000 7f6611c00640 1 -- 172.21.15.69:0/1125562965 shutdown_connections 2024-04-24T08:18:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.500+0000 7f6611c00640 1 -- 172.21.15.69:0/1125562965 >> 172.21.15.69:0/1125562965 conn(0x7f660c0d0350 msgr2=0x7f660c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:02.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.500+0000 7f6611c00640 1 -- 172.21.15.69:0/1125562965 shutdown_connections 2024-04-24T08:18:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.500+0000 7f6611c00640 1 -- 172.21.15.69:0/1125562965 wait complete. 2024-04-24T08:18:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.501+0000 7f6611c00640 1 Processor -- start 2024-04-24T08:18:02.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.502+0000 7f6611c00640 1 -- start start 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.502+0000 7f6611c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f0003f30 0x7f660c147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.502+0000 7f6611c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c156e40 0x7f660c15d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f6611c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f660c09dad0 0x7f660c1e72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f6611c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f660c0b4b00 con 0x7f660c09dad0 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f6611c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f660c06f2b0 con 0x7f65f0003f30 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f6611c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f660c0ce960 con 0x7f660c156e40 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f660b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f660c09dad0 0x7f660c1e72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f660b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f660c09dad0 0x7f660c1e72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33958/0 (socket says 172.21.15.69:33958) 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f660b400640 1 -- 172.21.15.69:0/1854852232 learned_addr learned my addr 172.21.15.69:0/1854852232 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f6603e00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c156e40 0x7f660c15d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f660aa00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f0003f30 0x7f660c147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f660b400640 1 -- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c156e40 msgr2=0x7f660c15d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f660b400640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c156e40 0x7f660c15d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f660b400640 1 -- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f0003f30 msgr2=0x7f660c147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f660b400640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f0003f30 0x7f660c147230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.503+0000 7f660b400640 1 -- 172.21.15.69:0/1854852232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660c1576c0 con 0x7f660c09dad0 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.504+0000 7f660b400640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f660c09dad0 0x7f660c1e72e0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f660c0755e0 tx=0x7f660400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.504+0000 7f6602a00640 1 -- 172.21.15.69:0/1854852232 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604004d60 con 0x7f660c09dad0 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.504+0000 7f6602a00640 1 -- 172.21.15.69:0/1854852232 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660400bb80 con 0x7f660c09dad0 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.504+0000 7f6602a00640 1 -- 172.21.15.69:0/1854852232 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604007070 con 0x7f660c09dad0 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.504+0000 7f6611c00640 1 -- 172.21.15.69:0/1854852232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660c15df10 con 0x7f660c09dad0 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.504+0000 7f6611c00640 1 -- 172.21.15.69:0/1854852232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f660c15e4d0 con 0x7f660c09dad0 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.505+0000 7f6611c00640 1 -- 172.21.15.69:0/1854852232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f660c068a80 con 0x7f660c09dad0 2024-04-24T08:18:02.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.506+0000 7f6602a00640 1 -- 172.21.15.69:0/1854852232 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f660401b020 con 0x7f660c09dad0 2024-04-24T08:18:02.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.507+0000 7f6602a00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65e806c8f0 0x7f65e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.507+0000 7f6602a00640 1 -- 172.21.15.69:0/1854852232 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5115+0+0 (secure 0 0 0) 0x7f660408b9f0 con 0x7f660c09dad0 2024-04-24T08:18:02.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.508+0000 7f660aa00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65e806c8f0 0x7f65e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.509+0000 7f660aa00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65e806c8f0 0x7f65e806edb0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009870 tx=0x7f65fc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.512+0000 7f6602a00640 1 -- 172.21.15.69:0/1854852232 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f660c1576c0 con 0x7f660c09dad0 2024-04-24T08:18:02.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.697+0000 7f6611c00640 1 -- 172.21.15.69:0/1854852232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f660c0cddc0 con 0x7f660c09dad0 2024-04-24T08:18:02.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.698+0000 7f6602a00640 1 -- 172.21.15.69:0/1854852232 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f6604059fd0 con 0x7f660c09dad0 2024-04-24T08:18:02.700 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:18:02.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.700+0000 7f6611c00640 1 -- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65e806c8f0 msgr2=0x7f65e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.700+0000 7f6611c00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65e806c8f0 0x7f65e806edb0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f65fc009870 tx=0x7f65fc008040 comp rx=0 tx=0).stop 2024-04-24T08:18:02.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.700+0000 7f6611c00640 1 -- 172.21.15.69:0/1854852232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f660c09dad0 msgr2=0x7f660c1e72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.700+0000 7f6611c00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f660c09dad0 0x7f660c1e72e0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f660c0755e0 tx=0x7f660400b4a0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.700+0000 7f6611c00640 1 -- 172.21.15.69:0/1854852232 shutdown_connections 2024-04-24T08:18:02.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.700+0000 7f6611c00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f65e806c8f0 0x7f65e806edb0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.700+0000 7f6611c00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f660c09dad0 0x7f660c1e72e0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.700+0000 7f6611c00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f660c156e40 0x7f660c15d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.700+0000 7f6611c00640 1 --2- 172.21.15.69:0/1854852232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65f0003f30 0x7f660c147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.700+0000 7f6611c00640 1 -- 172.21.15.69:0/1854852232 >> 172.21.15.69:0/1854852232 conn(0x7f660c0d0350 msgr2=0x7f660c0d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:02.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.701+0000 7f6611c00640 1 -- 172.21.15.69:0/1854852232 shutdown_connections 2024-04-24T08:18:02.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.701+0000 7f6611c00640 1 -- 172.21.15.69:0/1854852232 wait complete. 2024-04-24T08:18:02.720 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-04-24T08:18:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.906+0000 7fdd51000640 1 Processor -- start 2024-04-24T08:18:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.906+0000 7fdd51000640 1 -- start start 2024-04-24T08:18:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.906+0000 7fdd51000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd4c06cfb0 0x7fdd4c0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.906+0000 7fdd51000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd4c15ed40 con 0x7fdd4c06d410 2024-04-24T08:18:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.907+0000 7fdd51000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd4c0cf050 con 0x7fdd4c15e960 2024-04-24T08:18:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.907+0000 7fdd51000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fdd4c162b50 con 0x7fdd4c06cfb0 2024-04-24T08:18:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.907+0000 7fdd4be00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdd4c06d410 0x7fdd4c0d2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47386/0 (socket says 172.21.15.69:47386) 2024-04-24T08:18:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.907+0000 7fdd4be00640 1 -- 172.21.15.69:0/316714296 learned_addr learned my addr 172.21.15.69:0/316714296 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.907+0000 7fdd42a00640 1 --2- 172.21.15.69:0/316714296 >> v2:172.21.15.69:3300/0 conn(0x7fdd4c06cfb0 0x7fdd4c0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.907+0000 7fdd4aa00640 1 -- 172.21.15.69:0/316714296 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 485864136 0 0) 0x7fdd4c15ed40 con 0x7fdd4c06d410 2024-04-24T08:18:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.907+0000 7fdd4aa00640 1 -- 172.21.15.69:0/316714296 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd30003520 con 0x7fdd4c06d410 2024-04-24T08:18:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.907+0000 7fdd42a00640 1 -- 172.21.15.69:0/316714296 >> v1:172.21.15.121:6789/0 conn(0x7fdd4c15e960 legacy=0x7fdd4c162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:02.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.908+0000 7fdd42a00640 1 -- 172.21.15.69:0/316714296 >> v1:172.21.15.69:6789/0 conn(0x7fdd4c06d410 legacy=0x7fdd4c0d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.908+0000 7fdd42a00640 1 -- 172.21.15.69:0/316714296 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4c162e20 con 0x7fdd4c06cfb0 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd42a00640 1 --2- 172.21.15.69:0/316714296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06cfb0 0x7fdd4c0cfaf0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fdd340097a0 tx=0x7fdd3402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191216a500d15610 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd4aa00640 1 -- 172.21.15.69:0/316714296 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd34037070 con 0x7fdd4c06cfb0 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd4aa00640 1 -- 172.21.15.69:0/316714296 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd34032440 con 0x7fdd4c06cfb0 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd4aa00640 1 -- 172.21.15.69:0/316714296 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd34032810 con 0x7fdd4c06cfb0 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd51000640 1 -- 172.21.15.69:0/316714296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06cfb0 msgr2=0x7fdd4c0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd51000640 1 --2- 172.21.15.69:0/316714296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06cfb0 0x7fdd4c0cfaf0 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fdd340097a0 tx=0x7fdd3402a230 comp rx=0 tx=0).stop 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd51000640 1 -- 172.21.15.69:0/316714296 shutdown_connections 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd51000640 1 --2- 172.21.15.69:0/316714296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06cfb0 0x7fdd4c0cfaf0 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd51000640 1 -- 172.21.15.69:0/316714296 >> 172.21.15.69:0/316714296 conn(0x7fdd4c09d900 msgr2=0x7fdd4c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd51000640 1 -- 172.21.15.69:0/316714296 shutdown_connections 2024-04-24T08:18:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.909+0000 7fdd51000640 1 -- 172.21.15.69:0/316714296 wait complete. 2024-04-24T08:18:02.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.911+0000 7fdd51000640 1 Processor -- start 2024-04-24T08:18:02.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.911+0000 7fdd51000640 1 -- start start 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.912+0000 7fdd51000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c06cfb0 0x7fdd4c1f4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.912+0000 7fdd51000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06d410 0x7fdd4c15ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.912+0000 7fdd51000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c15e960 0x7fdd4c15b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.912+0000 7fdd51000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c0cf5d0 con 0x7fdd4c06d410 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.912+0000 7fdd51000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4c0cf2f0 con 0x7fdd4c15e960 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.912+0000 7fdd51000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdd4c0b6170 con 0x7fdd4c06cfb0 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.912+0000 7fdd42a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06d410 0x7fdd4c15ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.912+0000 7fdd42a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06d410 0x7fdd4c15ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33984/0 (socket says 172.21.15.69:33984) 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.912+0000 7fdd42a00640 1 -- 172.21.15.69:0/1921625317 learned_addr learned my addr 172.21.15.69:0/1921625317 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.913+0000 7fdd42a00640 1 -- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c06cfb0 msgr2=0x7fdd4c1f4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.913+0000 7fdd42a00640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c06cfb0 0x7fdd4c1f4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.913+0000 7fdd42a00640 1 -- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c15e960 msgr2=0x7fdd4c15b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.913+0000 7fdd42a00640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c15e960 0x7fdd4c15b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.913+0000 7fdd42a00640 1 -- 172.21.15.69:0/1921625317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4c0ca170 con 0x7fdd4c06d410 2024-04-24T08:18:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.913+0000 7fdd42a00640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06d410 0x7fdd4c15ab10 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fdd34035040 tx=0x7fdd340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.913+0000 7fdd43e00640 1 -- 172.21.15.69:0/1921625317 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd34037040 con 0x7fdd4c06d410 2024-04-24T08:18:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.913+0000 7fdd43e00640 1 -- 172.21.15.69:0/1921625317 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd34030500 con 0x7fdd4c06d410 2024-04-24T08:18:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.913+0000 7fdd43e00640 1 -- 172.21.15.69:0/1921625317 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd34030910 con 0x7fdd4c06d410 2024-04-24T08:18:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.913+0000 7fdd51000640 1 -- 172.21.15.69:0/1921625317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4c0ca450 con 0x7fdd4c06d410 2024-04-24T08:18:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.914+0000 7fdd51000640 1 -- 172.21.15.69:0/1921625317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd4c0caa10 con 0x7fdd4c06d410 2024-04-24T08:18:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.915+0000 7fdd51000640 1 -- 172.21.15.69:0/1921625317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd4c075d50 con 0x7fdd4c06d410 2024-04-24T08:18:02.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.915+0000 7fdd43e00640 1 -- 172.21.15.69:0/1921625317 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdd34044020 con 0x7fdd4c06d410 2024-04-24T08:18:02.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.917+0000 7fdd43e00640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd2806c8f0 0x7fdd2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:02.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.917+0000 7fdd43e00640 1 -- 172.21.15.69:0/1921625317 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5115+0+0 (secure 0 0 0) 0x7fdd340abb80 con 0x7fdd4c06d410 2024-04-24T08:18:02.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.917+0000 7fdd4b400640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd2806c8f0 0x7fdd2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:02.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.918+0000 7fdd4b400640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd2806c8f0 0x7fdd2806edb0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c000f60 tx=0x7fdd3c0079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:02.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:02.921+0000 7fdd43e00640 1 -- 172.21.15.69:0/1921625317 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd3407a200 con 0x7fdd4c06d410 2024-04-24T08:18:03.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.197+0000 7fdd51000640 1 -- 172.21.15.69:0/1921625317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) -- 0x7fdd4c0d1dc0 con 0x7fdd4c06d410 2024-04-24T08:18:03.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.845+0000 7fdd43e00640 1 -- 172.21.15.69:0/1921625317 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v453) ==== 149+0+0 (secure 0 0 0) 0x7fdd34079b00 con 0x7fdd4c06d410 2024-04-24T08:18:03.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.849+0000 7fdd51000640 1 -- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd2806c8f0 msgr2=0x7fdd2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:03.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.849+0000 7fdd51000640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd2806c8f0 0x7fdd2806edb0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c000f60 tx=0x7fdd3c0079b0 comp rx=0 tx=0).stop 2024-04-24T08:18:03.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.849+0000 7fdd51000640 1 -- 172.21.15.69:0/1921625317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06d410 msgr2=0x7fdd4c15ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:03.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.849+0000 7fdd51000640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06d410 0x7fdd4c15ab10 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fdd34035040 tx=0x7fdd340046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:03.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.849+0000 7fdd51000640 1 -- 172.21.15.69:0/1921625317 shutdown_connections 2024-04-24T08:18:03.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.849+0000 7fdd51000640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd2806c8f0 0x7fdd2806edb0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:03.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.849+0000 7fdd51000640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd4c06d410 0x7fdd4c15ab10 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:03.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.849+0000 7fdd51000640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd4c15e960 0x7fdd4c15b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:03.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.849+0000 7fdd51000640 1 --2- 172.21.15.69:0/1921625317 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd4c06cfb0 0x7fdd4c1f4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:03.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.849+0000 7fdd51000640 1 -- 172.21.15.69:0/1921625317 >> 172.21.15.69:0/1921625317 conn(0x7fdd4c09d900 msgr2=0x7fdd4c0b4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:03.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.850+0000 7fdd51000640 1 -- 172.21.15.69:0/1921625317 shutdown_connections 2024-04-24T08:18:03.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:03.850+0000 7fdd51000640 1 -- 172.21.15.69:0/1921625317 wait complete. 2024-04-24T08:18:03.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-24T08:18:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.062+0000 7f084e400640 1 Processor -- start 2024-04-24T08:18:04.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.062+0000 7f084e400640 1 -- start start 2024-04-24T08:18:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.063+0000 7f084e400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f084815c820 0x7f08480b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.063+0000 7f084e400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08480b46a0 0x7f084806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.064+0000 7f084e400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f084815cc00 con 0x7f084806fd10 2024-04-24T08:18:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.064+0000 7f084e400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f084806ed30 con 0x7f08480b46a0 2024-04-24T08:18:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.064+0000 7f084e400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f08480b4e10 con 0x7f084815c820 2024-04-24T08:18:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.064+0000 7f0846a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08480b46a0 0x7f084806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.064+0000 7f0847e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f084806fd10 0x7f08480ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47390/0 (socket says 172.21.15.69:47390) 2024-04-24T08:18:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.064+0000 7f0847e00640 1 -- 172.21.15.69:0/3425265211 learned_addr learned my addr 172.21.15.69:0/3425265211 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:04.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.064+0000 7f0847400640 1 --2- 172.21.15.69:0/3425265211 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f084815c820 0x7f08480b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.065+0000 7f0846a00640 1 -- 172.21.15.69:0/3425265211 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f084815c820 msgr2=0x7f08480b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:04.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.065+0000 7f0846a00640 1 --2- 172.21.15.69:0/3425265211 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f084815c820 0x7f08480b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.065+0000 7f0846a00640 1 -- 172.21.15.69:0/3425265211 >> v1:172.21.15.69:6789/0 conn(0x7f084806fd10 legacy=0x7f08480ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.065+0000 7f0846a00640 1 -- 172.21.15.69:0/3425265211 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08480b50f0 con 0x7f08480b46a0 2024-04-24T08:18:04.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.065+0000 7f0846000640 1 -- 172.21.15.69:0/3425265211 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3570981638 0 0) 0x7f084815cc00 con 0x7f084806fd10 2024-04-24T08:18:04.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.066+0000 7f0846a00640 1 --2- 172.21.15.69:0/3425265211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08480b46a0 0x7f084806f7d0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f08340097a0 tx=0x7f083402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2327ab9f764ae1f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:04.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.066+0000 7f0846000640 1 -- 172.21.15.69:0/3425265211 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834037070 con 0x7f08480b46a0 2024-04-24T08:18:04.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.066+0000 7f0846000640 1 -- 172.21.15.69:0/3425265211 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0834032440 con 0x7f08480b46a0 2024-04-24T08:18:04.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.066+0000 7f0846000640 1 -- 172.21.15.69:0/3425265211 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834032810 con 0x7f08480b46a0 2024-04-24T08:18:04.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.066+0000 7f084e400640 1 -- 172.21.15.69:0/3425265211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08480b46a0 msgr2=0x7f084806f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.067+0000 7f084e400640 1 --2- 172.21.15.69:0/3425265211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08480b46a0 0x7f084806f7d0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f08340097a0 tx=0x7f083402a180 comp rx=0 tx=0).stop 2024-04-24T08:18:04.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.067+0000 7f084e400640 1 -- 172.21.15.69:0/3425265211 shutdown_connections 2024-04-24T08:18:04.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.067+0000 7f084e400640 1 --2- 172.21.15.69:0/3425265211 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08480b46a0 0x7f084806f7d0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.067+0000 7f084e400640 1 --2- 172.21.15.69:0/3425265211 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f084815c820 0x7f08480b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.067+0000 7f084e400640 1 -- 172.21.15.69:0/3425265211 >> 172.21.15.69:0/3425265211 conn(0x7f08480d0350 msgr2=0x7f08480d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:04.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.067+0000 7f084e400640 1 -- 172.21.15.69:0/3425265211 shutdown_connections 2024-04-24T08:18:04.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.067+0000 7f084e400640 1 -- 172.21.15.69:0/3425265211 wait complete. 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.068+0000 7f084e400640 1 Processor -- start 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.069+0000 7f084e400640 1 -- start start 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.069+0000 7f084e400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f084806fd10 0x7f08480ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.069+0000 7f084e400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08480b46a0 0x7f08481601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.069+0000 7f084e400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f084815c820 0x7f0848160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.069+0000 7f084e400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f08480b4b00 con 0x7f084806fd10 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.069+0000 7f084e400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f084806f2b0 con 0x7f084815c820 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.069+0000 7f084e400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f08480ce960 con 0x7f08480b46a0 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.070+0000 7f0847400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f084806fd10 0x7f08480ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.070+0000 7f0847400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f084806fd10 0x7f08480ca970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34014/0 (socket says 172.21.15.69:34014) 2024-04-24T08:18:04.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.070+0000 7f0847400640 1 -- 172.21.15.69:0/1728810771 learned_addr learned my addr 172.21.15.69:0/1728810771 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.070+0000 7f0847400640 1 -- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08480b46a0 msgr2=0x7f08481601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.070+0000 7f0847400640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08480b46a0 0x7f08481601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.070+0000 7f0847400640 1 -- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f084815c820 msgr2=0x7f0848160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.070+0000 7f0847400640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f084815c820 0x7f0848160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.070+0000 7f0847400640 1 -- 172.21.15.69:0/1728810771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f084815f320 con 0x7f084806fd10 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.070+0000 7f0847400640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f084806fd10 0x7f08480ca970 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f083800d800 tx=0x7f083800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.071+0000 7f083f400640 1 -- 172.21.15.69:0/1728810771 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083800b6c0 con 0x7f084806fd10 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.071+0000 7f083f400640 1 -- 172.21.15.69:0/1728810771 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0838019070 con 0x7f084806fd10 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.071+0000 7f084e400640 1 -- 172.21.15.69:0/1728810771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f084815f600 con 0x7f084806fd10 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.071+0000 7f083f400640 1 -- 172.21.15.69:0/1728810771 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083800ba80 con 0x7f084806fd10 2024-04-24T08:18:04.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.071+0000 7f084e400640 1 -- 172.21.15.69:0/1728810771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f084815fbc0 con 0x7f084806fd10 2024-04-24T08:18:04.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.072+0000 7f084e400640 1 -- 172.21.15.69:0/1728810771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0848068a80 con 0x7f084806fd10 2024-04-24T08:18:04.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.073+0000 7f083f400640 1 -- 172.21.15.69:0/1728810771 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f083800bc20 con 0x7f084806fd10 2024-04-24T08:18:04.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.074+0000 7f083f400640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f082806c920 0x7f082806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.074+0000 7f083f400640 1 -- 172.21.15.69:0/1728810771 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5115+0+0 (secure 0 0 0) 0x7f083808bde0 con 0x7f084806fd10 2024-04-24T08:18:04.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.074+0000 7f0846a00640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f082806c920 0x7f082806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.076+0000 7f0846a00640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f082806c920 0x7f082806ede0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0834034000 tx=0x7f083402c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:04.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.078+0000 7f083f400640 1 -- 172.21.15.69:0/1728810771 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f083801c360 con 0x7f084806fd10 2024-04-24T08:18:04.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.265+0000 7f084e400640 1 -- 172.21.15.69:0/1728810771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f08480cddc0 con 0x7f084806fd10 2024-04-24T08:18:04.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.272+0000 7f083f400640 1 -- 172.21.15.69:0/1728810771 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) ==== 122+0+0 (secure 0 0 0) 0x7f083805a3c0 con 0x7f084806fd10 2024-04-24T08:18:04.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 -- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f082806c920 msgr2=0x7f082806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f082806c920 0x7f082806ede0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f0834034000 tx=0x7f083402c000 comp rx=0 tx=0).stop 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 -- 172.21.15.69:0/1728810771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f084806fd10 msgr2=0x7f08480ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f084806fd10 0x7f08480ca970 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f083800d800 tx=0x7f083800dcc0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 -- 172.21.15.69:0/1728810771 shutdown_connections 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f082806c920 0x7f082806ede0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f084815c820 0x7f0848160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08480b46a0 0x7f08481601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 --2- 172.21.15.69:0/1728810771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f084806fd10 0x7f08480ca970 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 -- 172.21.15.69:0/1728810771 >> 172.21.15.69:0/1728810771 conn(0x7f08480d0350 msgr2=0x7f0848074d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 -- 172.21.15.69:0/1728810771 shutdown_connections 2024-04-24T08:18:04.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.275+0000 7f084e400640 1 -- 172.21.15.69:0/1728810771 wait complete. 2024-04-24T08:18:04.292 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:18:04.292 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:18:04.390 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:18:04.521 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:18:04.521 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:18:04.521 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:04.576 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:18:04.577 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:18:04.577 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:04.634 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:18:04.635 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:18:04.635 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:04.691 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:18:04.691 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:18:04.692 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:04.749 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:18:04.749 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-24T08:18:04.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:04.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.954+0000 7f00b4000640 1 Processor -- start 2024-04-24T08:18:04.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.954+0000 7f00b4000640 1 -- start start 2024-04-24T08:18:04.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.956+0000 7f00b4000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f00ac0af9a0 0x7f00ac1697e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.956+0000 7f00b4000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00ac0adab0 con 0x7f00ac15ea10 2024-04-24T08:18:04.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.956+0000 7f00b4000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00ac0add50 con 0x7f00ac1631a0 2024-04-24T08:18:04.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.956+0000 7f00b4000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f00ac0ae030 con 0x7f00ac0af9a0 2024-04-24T08:18:04.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.956+0000 7f00b1600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f00ac15ea10 0x7f00ac0d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47392/0 (socket says 172.21.15.69:47392) 2024-04-24T08:18:04.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.956+0000 7f00b1600640 1 -- 172.21.15.69:0/2632984062 learned_addr learned my addr 172.21.15.69:0/2632984062 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:04.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.957+0000 7f00b2000640 1 --2- 172.21.15.69:0/2632984062 >> v2:172.21.15.121:3300/0 conn(0x7f00ac0af9a0 0x7f00ac1697e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.957+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1258242647 0 0) 0x7f00ac0adab0 con 0x7f00ac15ea10 2024-04-24T08:18:04.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.957+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0090003520 con 0x7f00ac15ea10 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.957+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3933645747 0 0) 0x7f0090003520 con 0x7f00ac15ea10 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.957+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 >> v2:172.21.15.121:3300/0 conn(0x7f00ac0af9a0 msgr2=0x7f00ac1697e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.957+0000 7f00abe00640 1 --2- 172.21.15.69:0/2632984062 >> v2:172.21.15.121:3300/0 conn(0x7f00ac0af9a0 0x7f00ac1697e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.957+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 >> v1:172.21.15.121:6789/0 conn(0x7f00ac1631a0 legacy=0x7f00ac0af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.957+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ac0ae310 con 0x7f00ac15ea10 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.957+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f009c001670 con 0x7f00ac15ea10 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.957+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 >> v1:172.21.15.69:6789/0 conn(0x7f00ac15ea10 legacy=0x7f00ac0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00abe00640 1 --2- 172.21.15.69:0/2632984062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0090003a00 0x7f0090003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00abe00640 1 --2- 172.21.15.69:0/2632984062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0090004800 0x7f0090004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00abe00640 1 --2- 172.21.15.69:0/2632984062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00900051c0 0x7f009000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac0ae030 con 0x7f00900051c0 2024-04-24T08:18:04.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f009000c0c0 con 0x7f0090004800 2024-04-24T08:18:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f009000c2c0 con 0x7f0090003a00 2024-04-24T08:18:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00b2000640 1 --2- 172.21.15.69:0/2632984062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00900051c0 0x7f009000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00b2000640 1 -- 172.21.15.69:0/2632984062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0090003a00 msgr2=0x7f0090003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00b2000640 1 --2- 172.21.15.69:0/2632984062 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0090003a00 0x7f0090003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00b2000640 1 -- 172.21.15.69:0/2632984062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0090004800 msgr2=0x7f0090004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00b2000640 1 --2- 172.21.15.69:0/2632984062 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0090004800 0x7f0090004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.958+0000 7f00b2000640 1 -- 172.21.15.69:0/2632984062 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0090003520 con 0x7f00900051c0 2024-04-24T08:18:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.959+0000 7f00b2000640 1 -- 172.21.15.69:0/2632984062 reap_dead start 2024-04-24T08:18:04.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.959+0000 7f00b2000640 1 --2- 172.21.15.69:0/2632984062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00900051c0 0x7f009000b9c0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009fc0 tx=0x7f0090005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3b9de9290ca149 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.959+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4035020 con 0x7f00900051c0 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.960+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a4003bd0 con 0x7f00900051c0 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.960+0000 7f00abe00640 1 -- 172.21.15.69:0/2632984062 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a4009c10 con 0x7f00900051c0 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.960+0000 7f00b4000640 1 -- 172.21.15.69:0/2632984062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00900051c0 msgr2=0x7f009000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.960+0000 7f00b4000640 1 --2- 172.21.15.69:0/2632984062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00900051c0 0x7f009000b9c0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009fc0 tx=0x7f0090005830 comp rx=0 tx=0).stop 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.960+0000 7f00b4000640 1 -- 172.21.15.69:0/2632984062 shutdown_connections 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.960+0000 7f00b4000640 1 --2- 172.21.15.69:0/2632984062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00900051c0 0x7f009000b9c0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.960+0000 7f00b4000640 1 -- 172.21.15.69:0/2632984062 >> 172.21.15.69:0/2632984062 conn(0x7f00ac067d90 msgr2=0x7f00ac06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.960+0000 7f00b4000640 1 -- 172.21.15.69:0/2632984062 shutdown_connections 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.960+0000 7f00b4000640 1 -- 172.21.15.69:0/2632984062 wait complete. 2024-04-24T08:18:04.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.962+0000 7f00b4000640 1 Processor -- start 2024-04-24T08:18:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.962+0000 7f00b4000640 1 -- start start 2024-04-24T08:18:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.963+0000 7f00b4000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00900051c0 0x7f00ac14f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.963+0000 7f00b4000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac14f100 0x7f00ac151650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.963+0000 7f00b4000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00ac160a30 0x7f00ac1449d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.963+0000 7f00b4000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac0aedf0 con 0x7f00ac14f100 2024-04-24T08:18:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.963+0000 7f00b4000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f00ac0ae8d0 con 0x7f00ac160a30 2024-04-24T08:18:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.963+0000 7f00b4000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f00ac0af290 con 0x7f00900051c0 2024-04-24T08:18:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.963+0000 7f00b0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac14f100 0x7f00ac151650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.963+0000 7f00b0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac14f100 0x7f00ac151650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34034/0 (socket says 172.21.15.69:34034) 2024-04-24T08:18:04.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.963+0000 7f00b0c00640 1 -- 172.21.15.69:0/2507869012 learned_addr learned my addr 172.21.15.69:0/2507869012 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:04.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00b2000640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00ac160a30 0x7f00ac1449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00b1600640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00900051c0 0x7f00ac14f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00b0c00640 1 -- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00900051c0 msgr2=0x7f00ac14f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00b0c00640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00900051c0 0x7f00ac14f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00b0c00640 1 -- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00ac160a30 msgr2=0x7f00ac1449d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:04.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00b0c00640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00ac160a30 0x7f00ac1449d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:04.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00b0c00640 1 -- 172.21.15.69:0/2507869012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00ac1505b0 con 0x7f00ac14f100 2024-04-24T08:18:04.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00b2000640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00ac160a30 0x7f00ac1449d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:04.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00b0c00640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac14f100 0x7f00ac151650 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f009800c500 tx=0x7f009800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:04.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00a9600640 1 -- 172.21.15.69:0/2507869012 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00980052d0 con 0x7f00ac14f100 2024-04-24T08:18:04.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00a9600640 1 -- 172.21.15.69:0/2507869012 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0098005470 con 0x7f00ac14f100 2024-04-24T08:18:04.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.964+0000 7f00b4000640 1 -- 172.21.15.69:0/2507869012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00ac150890 con 0x7f00ac14f100 2024-04-24T08:18:04.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.965+0000 7f00a9600640 1 -- 172.21.15.69:0/2507869012 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0098012870 con 0x7f00ac14f100 2024-04-24T08:18:04.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.965+0000 7f00b4000640 1 -- 172.21.15.69:0/2507869012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00ac150e50 con 0x7f00ac14f100 2024-04-24T08:18:04.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.966+0000 7f00b4000640 1 -- 172.21.15.69:0/2507869012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00ac068a80 con 0x7f00ac14f100 2024-04-24T08:18:04.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.967+0000 7f00a9600640 1 -- 172.21.15.69:0/2507869012 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0098012a10 con 0x7f00ac14f100 2024-04-24T08:18:04.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.968+0000 7f00a9600640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f008c06c8f0 0x7f008c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:04.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.968+0000 7f00a9600640 1 -- 172.21.15.69:0/2507869012 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5115+0+0 (secure 0 0 0) 0x7f009808b660 con 0x7f00ac14f100 2024-04-24T08:18:04.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.968+0000 7f00b1600640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f008c06c8f0 0x7f008c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:04.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.969+0000 7f00b1600640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f008c06c8f0 0x7f008c06edb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f009c000ff0 tx=0x7f009c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:04.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:04.972+0000 7f00a9600640 1 -- 172.21.15.69:0/2507869012 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0098059cd0 con 0x7f00ac14f100 2024-04-24T08:18:05.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.211+0000 7f00b4000640 1 -- 172.21.15.69:0/2507869012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00ac159ab0 con 0x7f00ac14f100 2024-04-24T08:18:05.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.212+0000 7f00a9600640 1 -- 172.21.15.69:0/2507869012 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+3866 (secure 0 0 0) 0x7f00980595d0 con 0x7f00ac14f100 2024-04-24T08:18:05.213 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:05.213 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":76,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/70319430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":70319430},{"type":"v1","addr":"172.21.15.69:6835","nonce":70319430}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":5357,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3430481199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3430481199},{"type":"v1","addr":"172.21.15.121:6837","nonce":3430481199}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":63},{"gid":5686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/233519124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":233519124},{"type":"v1","addr":"172.21.15.121:6836","nonce":233519124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56},{"gid":8466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/4189042976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":4189042976},{"type":"v1","addr":"172.21.15.69:6833","nonce":4189042976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":76}],"filesystems":[]} 2024-04-24T08:18:05.213 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 76 2024-04-24T08:18:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.214+0000 7f00b4000640 1 -- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f008c06c8f0 msgr2=0x7f008c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.214+0000 7f00b4000640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f008c06c8f0 0x7f008c06edb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f009c000ff0 tx=0x7f009c009040 comp rx=0 tx=0).stop 2024-04-24T08:18:05.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.215+0000 7f00b4000640 1 -- 172.21.15.69:0/2507869012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac14f100 msgr2=0x7f00ac151650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.215+0000 7f00b4000640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac14f100 0x7f00ac151650 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f009800c500 tx=0x7f009800c9c0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.215+0000 7f00b4000640 1 -- 172.21.15.69:0/2507869012 shutdown_connections 2024-04-24T08:18:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.215+0000 7f00b4000640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f008c06c8f0 0x7f008c06edb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.215+0000 7f00b4000640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f00ac14f100 0x7f00ac151650 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.215+0000 7f00b4000640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f00ac160a30 0x7f00ac1449d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.215+0000 7f00b4000640 1 --2- 172.21.15.69:0/2507869012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f00900051c0 0x7f00ac14f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.215+0000 7f00b4000640 1 -- 172.21.15.69:0/2507869012 >> 172.21.15.69:0/2507869012 conn(0x7f00ac067d90 msgr2=0x7f00ac0b4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.215+0000 7f00b4000640 1 -- 172.21.15.69:0/2507869012 shutdown_connections 2024-04-24T08:18:05.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.215+0000 7f00b4000640 1 -- 172.21.15.69:0/2507869012 wait complete. 2024-04-24T08:18:05.235 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T08:18:05.235 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-24T08:18:05.235 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T08:18:05.235 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:18:05.235 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T08:18:05.235 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-24T08:18:05.235 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T08:18:05.236 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:18:05.236 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T08:18:05.236 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-24T08:18:05.236 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T08:18:05.236 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:18:05.236 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T08:18:05.236 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-24T08:18:05.236 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T08:18:05.237 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:18:05.259 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T08:18:05.259 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T08:18:05.263 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T08:18:05.263 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T08:18:05.265 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T08:18:05.265 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T08:18:05.268 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T08:18:05.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T08:18:05.310 INFO:tasks.ceph.mds.b:Started 2024-04-24T08:18:05.311 DEBUG:teuthology.parallel:result is None 2024-04-24T08:18:05.315 INFO:tasks.ceph.mds.c:Started 2024-04-24T08:18:05.315 DEBUG:teuthology.parallel:result is None 2024-04-24T08:18:05.317 INFO:tasks.ceph.mds.d:Started 2024-04-24T08:18:05.317 DEBUG:teuthology.parallel:result is None 2024-04-24T08:18:05.321 INFO:tasks.ceph.mds.a:Started 2024-04-24T08:18:05.322 DEBUG:teuthology.parallel:result is None 2024-04-24T08:18:05.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-24T08:18:05.461 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:18:05.463 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:18:05.468 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:18:05.468 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-24T08:18:05.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.494+0000 7f4f06800640 1 Processor -- start 2024-04-24T08:18:05.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.494+0000 7f4f06800640 1 -- start start 2024-04-24T08:18:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.495+0000 7f4f06800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d05d0 0x7f4f001699b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.495+0000 7f4f06800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f000d26c0 con 0x7f4f0015e890 2024-04-24T08:18:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.495+0000 7f4f06800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f000d29a0 con 0x7f4f000d18c0 2024-04-24T08:18:05.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.495+0000 7f4f06800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4f000d32d0 con 0x7f4f000d05d0 2024-04-24T08:18:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.495+0000 7f4eff400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4f0015e890 0x7f4f000d11b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47412/0 (socket says 172.21.15.69:47412) 2024-04-24T08:18:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.495+0000 7f4eff400640 1 -- 172.21.15.69:0/2746040260 learned_addr learned my addr 172.21.15.69:0/2746040260 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:05.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.495+0000 7f4effe00640 1 --2- 172.21.15.69:0/2746040260 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d05d0 0x7f4f001699b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.496+0000 7f4efe000640 1 -- 172.21.15.69:0/2746040260 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3690108499 0 0) 0x7f4f000d26c0 con 0x7f4f0015e890 2024-04-24T08:18:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.496+0000 7f4efe000640 1 -- 172.21.15.69:0/2746040260 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ee4003520 con 0x7f4f0015e890 2024-04-24T08:18:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.496+0000 7f4efe000640 1 -- 172.21.15.69:0/2746040260 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3387085307 0 0) 0x7f4f000d29a0 con 0x7f4f000d18c0 2024-04-24T08:18:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.496+0000 7f4efe000640 1 -- 172.21.15.69:0/2746040260 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f000d26c0 con 0x7f4f000d18c0 2024-04-24T08:18:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.496+0000 7f4effe00640 1 -- 172.21.15.69:0/2746040260 >> v1:172.21.15.121:6789/0 conn(0x7f4f000d18c0 legacy=0x7f4f000cfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.496+0000 7f4effe00640 1 -- 172.21.15.69:0/2746040260 >> v1:172.21.15.69:6789/0 conn(0x7f4f0015e890 legacy=0x7f4f000d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.496+0000 7f4effe00640 1 -- 172.21.15.69:0/2746040260 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f000d2c80 con 0x7f4f000d05d0 2024-04-24T08:18:05.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.497+0000 7f4effe00640 1 --2- 172.21.15.69:0/2746040260 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d05d0 0x7f4f001699b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8009dc0 tx=0x7f4ef802a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b861abcf474cc4b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:05.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.497+0000 7f4efe000640 1 -- 172.21.15.69:0/2746040260 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8007740 con 0x7f4f000d05d0 2024-04-24T08:18:05.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.497+0000 7f4efe000640 1 -- 172.21.15.69:0/2746040260 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ef80078e0 con 0x7f4f000d05d0 2024-04-24T08:18:05.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.497+0000 7f4efe000640 1 -- 172.21.15.69:0/2746040260 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8007cb0 con 0x7f4f000d05d0 2024-04-24T08:18:05.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.497+0000 7f4f06800640 1 -- 172.21.15.69:0/2746040260 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d05d0 msgr2=0x7f4f001699b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.497+0000 7f4f06800640 1 --2- 172.21.15.69:0/2746040260 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d05d0 0x7f4f001699b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8009dc0 tx=0x7f4ef802a760 comp rx=0 tx=0).stop 2024-04-24T08:18:05.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.498+0000 7f4f06800640 1 -- 172.21.15.69:0/2746040260 shutdown_connections 2024-04-24T08:18:05.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.498+0000 7f4f06800640 1 --2- 172.21.15.69:0/2746040260 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d05d0 0x7f4f001699b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.498+0000 7f4f06800640 1 -- 172.21.15.69:0/2746040260 >> 172.21.15.69:0/2746040260 conn(0x7f4f0008bf60 msgr2=0x7f4f000d47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:05.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.498+0000 7f4f06800640 1 -- 172.21.15.69:0/2746040260 shutdown_connections 2024-04-24T08:18:05.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.498+0000 7f4f06800640 1 -- 172.21.15.69:0/2746040260 wait complete. 2024-04-24T08:18:05.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.499+0000 7f4f06800640 1 Processor -- start 2024-04-24T08:18:05.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.499+0000 7f4f06800640 1 -- start start 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.500+0000 7f4f06800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f000d05d0 0x7f4f00158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.500+0000 7f4f06800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d18c0 0x7f4f0015b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.500+0000 7f4f06800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f0015e890 0x7f4f001595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.500+0000 7f4f06800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4f000d3760 con 0x7f4f0015e890 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.500+0000 7f4f06800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4f000b4b70 con 0x7f4f000d05d0 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.500+0000 7f4f06800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4f000d2370 con 0x7f4f000d18c0 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.500+0000 7f4effe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f0015e890 0x7f4f001595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4effe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f0015e890 0x7f4f001595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34068/0 (socket says 172.21.15.69:34068) 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4effe00640 1 -- 172.21.15.69:0/849139798 learned_addr learned my addr 172.21.15.69:0/849139798 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4efea00640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d18c0 0x7f4f0015b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4eff400640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f000d05d0 0x7f4f00158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4effe00640 1 -- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d18c0 msgr2=0x7f4f0015b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4effe00640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d18c0 0x7f4f0015b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4effe00640 1 -- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f000d05d0 msgr2=0x7f4f00158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4effe00640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f000d05d0 0x7f4f00158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4effe00640 1 -- 172.21.15.69:0/849139798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f0015a160 con 0x7f4f0015e890 2024-04-24T08:18:05.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4eff400640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f000d05d0 0x7f4f00158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:05.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4effe00640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f0015e890 0x7f4f001595c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f4ef802ad50 tx=0x7f4ef8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:05.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.501+0000 7f4ef7400640 1 -- 172.21.15.69:0/849139798 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8040070 con 0x7f4f0015e890 2024-04-24T08:18:05.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.502+0000 7f4ef7400640 1 -- 172.21.15.69:0/849139798 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ef8031bc0 con 0x7f4f0015e890 2024-04-24T08:18:05.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.502+0000 7f4ef7400640 1 -- 172.21.15.69:0/849139798 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8009450 con 0x7f4f0015e890 2024-04-24T08:18:05.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.502+0000 7f4f06800640 1 -- 172.21.15.69:0/849139798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f0015a440 con 0x7f4f0015e890 2024-04-24T08:18:05.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.502+0000 7f4f06800640 1 -- 172.21.15.69:0/849139798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f0015aa00 con 0x7f4f0015e890 2024-04-24T08:18:05.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.503+0000 7f4f06800640 1 -- 172.21.15.69:0/849139798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f00068a80 con 0x7f4f0015e890 2024-04-24T08:18:05.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.503+0000 7f4ef7400640 1 -- 172.21.15.69:0/849139798 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ef8039040 con 0x7f4f0015e890 2024-04-24T08:18:05.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.504+0000 7f4ef7400640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4ee006c8f0 0x7f4ee006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:05.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.504+0000 7f4ef7400640 1 -- 172.21.15.69:0/849139798 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 5115+0+0 (secure 0 0 0) 0x7f4ef80acc70 con 0x7f4f0015e890 2024-04-24T08:18:05.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.505+0000 7f4eff400640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4ee006c8f0 0x7f4ee006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:05.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.506+0000 7f4eff400640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4ee006c8f0 0x7f4ee006edb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0007fd0 tx=0x7f4ef0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:05.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.509+0000 7f4ef7400640 1 -- 172.21.15.69:0/849139798 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ef804c030 con 0x7f4f0015e890 2024-04-24T08:18:05.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.718+0000 7f4f06800640 1 -- 172.21.15.69:0/849139798 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f4f000c7eb0 con 0x7f4f0015e890 2024-04-24T08:18:05.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.877+0000 7f4ef7400640 1 -- 172.21.15.69:0/849139798 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v91) ==== 97+0+0 (secure 0 0 0) 0x7f4ef8081020 con 0x7f4f0015e890 2024-04-24T08:18:05.878 INFO:teuthology.orchestra.run.smithi069.stderr: removed all blocklist entries 2024-04-24T08:18:05.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.880+0000 7f4f06800640 1 -- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4ee006c8f0 msgr2=0x7f4ee006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.880+0000 7f4f06800640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4ee006c8f0 0x7f4ee006edb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0007fd0 tx=0x7f4ef0009040 comp rx=0 tx=0).stop 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.881+0000 7f4f06800640 1 -- 172.21.15.69:0/849139798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f0015e890 msgr2=0x7f4f001595c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.881+0000 7f4f06800640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f0015e890 0x7f4f001595c0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f4ef802ad50 tx=0x7f4ef8004a40 comp rx=0 tx=0).stop 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.881+0000 7f4f06800640 1 -- 172.21.15.69:0/849139798 shutdown_connections 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.881+0000 7f4f06800640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4ee006c8f0 0x7f4ee006edb0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.881+0000 7f4f06800640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f0015e890 0x7f4f001595c0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.881+0000 7f4f06800640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f000d18c0 0x7f4f0015b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.881+0000 7f4f06800640 1 --2- 172.21.15.69:0/849139798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f000d05d0 0x7f4f00158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.881+0000 7f4f06800640 1 -- 172.21.15.69:0/849139798 >> 172.21.15.69:0/849139798 conn(0x7f4f0008bf60 msgr2=0x7f4f0006e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.881+0000 7f4f06800640 1 -- 172.21.15.69:0/849139798 shutdown_connections 2024-04-24T08:18:05.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:05.881+0000 7f4f06800640 1 -- 172.21.15.69:0/849139798 wait complete. 2024-04-24T08:18:05.903 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-24T08:18:06.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.096+0000 7fe727000640 1 Processor -- start 2024-04-24T08:18:06.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.096+0000 7fe727000640 1 -- start start 2024-04-24T08:18:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.097+0000 7fe727000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe72015d1d0 0x7fe7200ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.097+0000 7fe727000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe72015d660 con 0x7fe7200b30f0 2024-04-24T08:18:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.097+0000 7fe727000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe72006dc30 con 0x7fe720156c60 2024-04-24T08:18:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.097+0000 7fe727000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe72006df10 con 0x7fe72015d1d0 2024-04-24T08:18:06.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.098+0000 7fe726600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe7200b30f0 0x7fe72006e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47422/0 (socket says 172.21.15.69:47422) 2024-04-24T08:18:06.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.098+0000 7fe726600640 1 -- 172.21.15.69:0/1647065530 learned_addr learned my addr 172.21.15.69:0/1647065530 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.098+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054720448 0 0) 0x7fe72015d660 con 0x7fe7200b30f0 2024-04-24T08:18:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.098+0000 7fe725200640 1 --2- 172.21.15.69:0/1647065530 >> v2:172.21.15.121:3300/0 conn(0x7fe72015d1d0 0x7fe7200ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.098+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe708003520 con 0x7fe7200b30f0 2024-04-24T08:18:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1793116286 0 0) 0x7fe708003520 con 0x7fe7200b30f0 2024-04-24T08:18:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 >> v2:172.21.15.121:3300/0 conn(0x7fe72015d1d0 msgr2=0x7fe7200ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:06.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 --2- 172.21.15.69:0/1647065530 >> v2:172.21.15.121:3300/0 conn(0x7fe72015d1d0 0x7fe7200ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 >> v1:172.21.15.121:6789/0 conn(0x7fe720156c60 legacy=0x7fe72015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72006e1f0 con 0x7fe7200b30f0 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe7180019f0 con 0x7fe7200b30f0 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 >> v1:172.21.15.69:6789/0 conn(0x7fe7200b30f0 legacy=0x7fe72006e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe708003a00 0x7fe708003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe708004800 0x7fe708004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe708005210 0x7fe70800fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe72006df10 con 0x7fe708004800 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe708010570 con 0x7fe708005210 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe708010770 con 0x7fe708003a00 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe725200640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe708004800 0x7fe708004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe725c00640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe708003a00 0x7fe708003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe726600640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe708005210 0x7fe70800fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.099+0000 7fe725200640 1 -- 172.21.15.69:0/1647065530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe708003a00 msgr2=0x7fe708003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.100+0000 7fe725200640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe708003a00 0x7fe708003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.100+0000 7fe725200640 1 -- 172.21.15.69:0/1647065530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe708005210 msgr2=0x7fe70800fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.100+0000 7fe725200640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe708005210 0x7fe70800fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.100+0000 7fe725200640 1 -- 172.21.15.69:0/1647065530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe708003520 con 0x7fe708004800 2024-04-24T08:18:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.100+0000 7fe726600640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe708005210 0x7fe70800fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:06.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.100+0000 7fe726600640 1 -- 172.21.15.69:0/1647065530 reap_dead start 2024-04-24T08:18:06.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.101+0000 7fe725200640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe708004800 0x7fe708004c80 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fe70c00ba30 tx=0x7fe708005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fa9da2371c7638 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.101+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c030070 con 0x7fe708004800 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.101+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe70c02ade0 con 0x7fe708004800 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.101+0000 7fe71fe00640 1 -- 172.21.15.69:0/1647065530 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c00b600 con 0x7fe708004800 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.101+0000 7fe727000640 1 -- 172.21.15.69:0/1647065530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe708004800 msgr2=0x7fe708004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.101+0000 7fe727000640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe708004800 0x7fe708004c80 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fe70c00ba30 tx=0x7fe708005830 comp rx=0 tx=0).stop 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.102+0000 7fe727000640 1 -- 172.21.15.69:0/1647065530 shutdown_connections 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.102+0000 7fe727000640 1 --2- 172.21.15.69:0/1647065530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe708004800 0x7fe708004c80 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.102+0000 7fe727000640 1 -- 172.21.15.69:0/1647065530 >> 172.21.15.69:0/1647065530 conn(0x7fe7200d0160 msgr2=0x7fe7200d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.102+0000 7fe727000640 1 -- 172.21.15.69:0/1647065530 shutdown_connections 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.102+0000 7fe727000640 1 -- 172.21.15.69:0/1647065530 wait complete. 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.103+0000 7fe727000640 1 Processor -- start 2024-04-24T08:18:06.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.103+0000 7fe727000640 1 -- start start 2024-04-24T08:18:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.104+0000 7fe727000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe708004800 0x7fe720152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe727000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe720145630 0x7fe7201437c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe727000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe720143d00 0x7fe720154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe727000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe72015cac0 con 0x7fe720145630 2024-04-24T08:18:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe727000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe72015d940 con 0x7fe720143d00 2024-04-24T08:18:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe727000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe72015cc40 con 0x7fe708004800 2024-04-24T08:18:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe725200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe720145630 0x7fe7201437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe725200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe720145630 0x7fe7201437c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34086/0 (socket says 172.21.15.69:34086) 2024-04-24T08:18:06.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe725200640 1 -- 172.21.15.69:0/1918795239 learned_addr learned my addr 172.21.15.69:0/1918795239 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe725c00640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe708004800 0x7fe720152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe725200640 1 -- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe708004800 msgr2=0x7fe720152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe725200640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe708004800 0x7fe720152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe725200640 1 -- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe720143d00 msgr2=0x7fe720154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe725200640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe720143d00 0x7fe720154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.105+0000 7fe725200640 1 -- 172.21.15.69:0/1918795239 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7201609b0 con 0x7fe720145630 2024-04-24T08:18:06.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.106+0000 7fe725200640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe720145630 0x7fe7201437c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe70c00bb20 tx=0x7fe70c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.106+0000 7fe71d600640 1 -- 172.21.15.69:0/1918795239 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c043030 con 0x7fe720145630 2024-04-24T08:18:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.106+0000 7fe71d600640 1 -- 172.21.15.69:0/1918795239 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe70c02abe0 con 0x7fe720145630 2024-04-24T08:18:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.106+0000 7fe71d600640 1 -- 172.21.15.69:0/1918795239 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe70c00b600 con 0x7fe720145630 2024-04-24T08:18:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.106+0000 7fe727000640 1 -- 172.21.15.69:0/1918795239 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe720098c20 con 0x7fe720145630 2024-04-24T08:18:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.106+0000 7fe727000640 1 -- 172.21.15.69:0/1918795239 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe720153310 con 0x7fe720145630 2024-04-24T08:18:06.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.107+0000 7fe727000640 1 -- 172.21.15.69:0/1918795239 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe720098a50 con 0x7fe720145630 2024-04-24T08:18:06.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.109+0000 7fe71d600640 1 -- 172.21.15.69:0/1918795239 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe70c046070 con 0x7fe720145630 2024-04-24T08:18:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.110+0000 7fe71d600640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe70006c8f0 0x7fe70006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.110+0000 7fe71d600640 1 -- 172.21.15.69:0/1918795239 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 4857+0+0 (secure 0 0 0) 0x7fe70c0ac350 con 0x7fe720145630 2024-04-24T08:18:06.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.110+0000 7fe725c00640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe70006c8f0 0x7fe70006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.112+0000 7fe725c00640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe70006c8f0 0x7fe70006edb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe714001550 tx=0x7fe714009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:06.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.114+0000 7fe71d600640 1 -- 172.21.15.69:0/1918795239 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe70c03c3a0 con 0x7fe720145630 2024-04-24T08:18:06.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.385+0000 7fe727000640 1 -- 172.21.15.69:0/1918795239 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fe720144120 con 0x7fe720145630 2024-04-24T08:18:06.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.386+0000 7fe71d600640 1 -- 172.21.15.69:0/1918795239 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) ==== 80+0+5304 (secure 0 0 0) 0x7fe70c07a9a0 con 0x7fe720145630 2024-04-24T08:18:06.387 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:06.387 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:18:06.387 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_dump": [ 2024-04-24T08:18:06.387 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.387 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.a", 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmsqMeLRAApLPtHjtq78HzlVcsjfqXOg==", 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.b", 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmkyd5ChAAGsJGU6GGMvvGzQnABuDx/g==", 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:18:06.388 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.c", 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmS+yLORAAHPCeBtq99IXEVTpcMT3c4A==", 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.389 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.d", 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShm0xmHFRAA3VfYoqbcV8kWF+4fCZMVLA==", 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.0", 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC5vShmMbvcERAANgogiCC/sXf1y0Ykv7ecNQ==", 2024-04-24T08:18:06.390 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.1", 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC8vShmDD+uLBAAJHXmxEm1cumJ3wjqe6S0ww==", 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.2", 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC/vShmS63kNhAAXec5HVuwo8THKstXiQOL8Q==", 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.3", 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDDvShmxcDJBhAA3V3I4C63EHPRUyhzBCZEzw==", 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.392 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.4", 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDJvShm1pX9ERAAL2gjv/+M915Cf5OoqBhZVw==", 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.393 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.5", 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDMvShmNIxbHRAA23gxiaGvagddVc2mE5DJJw==", 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.6", 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDPvShmb+x4NhAA6aWmAzU6c0AZomVgWcFwig==", 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:18:06.394 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.7", 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDTvShmgcKMBRAA6fkJvNxBjlNhK9CrZ/FMSw==", 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.0", 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw==", 2024-04-24T08:18:06.395 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow r", 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.1", 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmKAIHMhAAuTuFmTHvBteErsG23LMEyQ==", 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.2", 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmW0MnKRAASzxCAKuApVvfu9kUfK9BJw==", 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.3", 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC2vShmfgoaABAAniS9oubv92R7Qe2EITGKxg==", 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.397 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.admin", 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQCzvShmLh6yLxAAoH7NhoEAUKcdXp6n+ai2Ug==", 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.398 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.x", 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmv4rwIxAA3rp+yy09qrs7HXip8bZ8xA==", 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.y", 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShm2eatGxAAFQkL0GBsujypv4VnmSy5bw==", 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:18:06.399 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:18:06.400 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.400 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:06.400 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-24T08:18:06.400 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:18:06.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 -- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe70006c8f0 msgr2=0x7fe70006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe70006c8f0 0x7fe70006edb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe714001550 tx=0x7fe714009040 comp rx=0 tx=0).stop 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 -- 172.21.15.69:0/1918795239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe720145630 msgr2=0x7fe7201437c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe720145630 0x7fe7201437c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fe70c00bb20 tx=0x7fe70c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 -- 172.21.15.69:0/1918795239 shutdown_connections 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe70006c8f0 0x7fe70006edb0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe720145630 0x7fe7201437c0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe720143d00 0x7fe720154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 --2- 172.21.15.69:0/1918795239 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe708004800 0x7fe720152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 -- 172.21.15.69:0/1918795239 >> 172.21.15.69:0/1918795239 conn(0x7fe7200d0160 msgr2=0x7fe72006ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 -- 172.21.15.69:0/1918795239 shutdown_connections 2024-04-24T08:18:06.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.389+0000 7fe727000640 1 -- 172.21.15.69:0/1918795239 wait complete. 2024-04-24T08:18:06.408 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:06.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.597+0000 7f4f16e00640 1 Processor -- start 2024-04-24T08:18:06.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.597+0000 7f4f16e00640 1 -- start start 2024-04-24T08:18:06.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.598+0000 7f4f16e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4f1015b490 0x7f4f100b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.598+0000 7f4f16e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4f1006f2d0 0x7f4f1006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f16e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f100b46e0 con 0x7f4f100cfbe0 2024-04-24T08:18:06.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f16e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4f1006fc70 con 0x7f4f1006f2d0 2024-04-24T08:18:06.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f16e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4f100d0720 con 0x7f4f1015b490 2024-04-24T08:18:06.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f0f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4f1006f2d0 0x7f4f1006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f14e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4f100cfbe0 0x7f4f100d0010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47426/0 (socket says 172.21.15.69:47426) 2024-04-24T08:18:06.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f0f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4f1006f2d0 0x7f4f1006f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34096/0 (socket says 172.21.15.69:34096) 2024-04-24T08:18:06.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f14e00640 1 -- 172.21.15.69:0/3842043861 learned_addr learned my addr 172.21.15.69:0/3842043861 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:06.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f0f400640 1 -- 172.21.15.69:0/3842043861 >> v2:172.21.15.121:3300/0 conn(0x7f4f1015b490 msgr2=0x7f4f100b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:06.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f0fe00640 1 --2- 172.21.15.69:0/3842043861 >> v2:172.21.15.121:3300/0 conn(0x7f4f1015b490 0x7f4f100b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f0f400640 1 --2- 172.21.15.69:0/3842043861 >> v2:172.21.15.121:3300/0 conn(0x7f4f1015b490 0x7f4f100b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f0f400640 1 -- 172.21.15.69:0/3842043861 >> v1:172.21.15.69:6789/0 conn(0x7f4f100cfbe0 legacy=0x7f4f100d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f0f400640 1 -- 172.21.15.69:0/3842043861 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f100d0960 con 0x7f4f1006f2d0 2024-04-24T08:18:06.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.599+0000 7f4f0ea00640 1 -- 172.21.15.69:0/3842043861 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 469557659 0 0) 0x7f4f100b46e0 con 0x7f4f100cfbe0 2024-04-24T08:18:06.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.600+0000 7f4f0f400640 1 --2- 172.21.15.69:0/3842043861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 0x7f4f1006f730 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f4f10072fa0 tx=0x7f4f0002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2337c8fdbaaaa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:06.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.600+0000 7f4f0ea00640 1 -- 172.21.15.69:0/3842043861 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f00007740 con 0x7f4f1006f2d0 2024-04-24T08:18:06.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.601+0000 7f4f0ea00640 1 -- 172.21.15.69:0/3842043861 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f000078e0 con 0x7f4f1006f2d0 2024-04-24T08:18:06.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.601+0000 7f4f0ea00640 1 -- 172.21.15.69:0/3842043861 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f00007cb0 con 0x7f4f1006f2d0 2024-04-24T08:18:06.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.601+0000 7f4f16e00640 1 -- 172.21.15.69:0/3842043861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 msgr2=0x7f4f1006f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.601+0000 7f4f16e00640 1 --2- 172.21.15.69:0/3842043861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 0x7f4f1006f730 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f4f10072fa0 tx=0x7f4f0002a680 comp rx=0 tx=0).stop 2024-04-24T08:18:06.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.601+0000 7f4f16e00640 1 -- 172.21.15.69:0/3842043861 shutdown_connections 2024-04-24T08:18:06.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.601+0000 7f4f16e00640 1 --2- 172.21.15.69:0/3842043861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 0x7f4f1006f730 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.601+0000 7f4f16e00640 1 --2- 172.21.15.69:0/3842043861 >> v2:172.21.15.121:3300/0 conn(0x7f4f1015b490 0x7f4f100b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.601+0000 7f4f16e00640 1 -- 172.21.15.69:0/3842043861 >> 172.21.15.69:0/3842043861 conn(0x7f4f10067d90 msgr2=0x7f4f1006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:06.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.601+0000 7f4f16e00640 1 -- 172.21.15.69:0/3842043861 shutdown_connections 2024-04-24T08:18:06.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.601+0000 7f4f16e00640 1 -- 172.21.15.69:0/3842043861 wait complete. 2024-04-24T08:18:06.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.603+0000 7f4f16e00640 1 Processor -- start 2024-04-24T08:18:06.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.603+0000 7f4f16e00640 1 -- start start 2024-04-24T08:18:06.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.604+0000 7f4f16e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 0x7f4f101560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.604+0000 7f4f16e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f100cfbe0 0x7f4f1014cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.604+0000 7f4f16e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f1015b490 0x7f4f1015c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.604+0000 7f4f16e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4f100d0c40 con 0x7f4f1006f2d0 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.604+0000 7f4f16e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4f100b4c60 con 0x7f4f1015b490 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.604+0000 7f4f16e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4f1015b870 con 0x7f4f100cfbe0 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.604+0000 7f4f0fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 0x7f4f101560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.604+0000 7f4f0fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 0x7f4f101560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34104/0 (socket says 172.21.15.69:34104) 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.604+0000 7f4f0fe00640 1 -- 172.21.15.69:0/821655082 learned_addr learned my addr 172.21.15.69:0/821655082 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f0fe00640 1 -- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f100cfbe0 msgr2=0x7f4f1014cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f0fe00640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f100cfbe0 0x7f4f1014cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f0fe00640 1 -- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f1015b490 msgr2=0x7f4f1015c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f0fe00640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f1015b490 0x7f4f1015c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f0fe00640 1 -- 172.21.15.69:0/821655082 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f1015edf0 con 0x7f4f1006f2d0 2024-04-24T08:18:06.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f0fe00640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 0x7f4f101560a0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800c7f0 tx=0x7f4ef800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:06.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f07e00640 1 -- 172.21.15.69:0/821655082 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8015070 con 0x7f4f1006f2d0 2024-04-24T08:18:06.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f07e00640 1 -- 172.21.15.69:0/821655082 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ef8004220 con 0x7f4f1006f2d0 2024-04-24T08:18:06.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f07e00640 1 -- 172.21.15.69:0/821655082 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8007620 con 0x7f4f1006f2d0 2024-04-24T08:18:06.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f16e00640 1 -- 172.21.15.69:0/821655082 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f101e3dc0 con 0x7f4f1006f2d0 2024-04-24T08:18:06.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.605+0000 7f4f16e00640 1 -- 172.21.15.69:0/821655082 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f101e4380 con 0x7f4f1006f2d0 2024-04-24T08:18:06.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.606+0000 7f4f16e00640 1 -- 172.21.15.69:0/821655082 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f10068a80 con 0x7f4f1006f2d0 2024-04-24T08:18:06.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.607+0000 7f4f07e00640 1 -- 172.21.15.69:0/821655082 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4ef80079d0 con 0x7f4f1006f2d0 2024-04-24T08:18:06.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.608+0000 7f4f07e00640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eec06c920 0x7f4eec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:06.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.608+0000 7f4f07e00640 1 -- 172.21.15.69:0/821655082 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 4857+0+0 (secure 0 0 0) 0x7f4ef808b820 con 0x7f4f1006f2d0 2024-04-24T08:18:06.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.609+0000 7f4f0f400640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eec06c920 0x7f4eec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:06.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.610+0000 7f4f0f400640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eec06c920 0x7f4eec06ede0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4f00034000 tx=0x7f4f0002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:06.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.613+0000 7f4f07e00640 1 -- 172.21.15.69:0/821655082 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ef8059f00 con 0x7f4f1006f2d0 2024-04-24T08:18:06.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.816+0000 7f4f16e00640 1 -- 172.21.15.69:0/821655082 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f100b3530 con 0x7f4f1006f2d0 2024-04-24T08:18:06.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.818+0000 7f4f07e00640 1 -- 172.21.15.69:0/821655082 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+12868 (secure 0 0 0) 0x7f4ef8059800 con 0x7f4f1006f2d0 2024-04-24T08:18:06.819 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:06.819 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":93,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:18:05.891981+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:06.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.821+0000 7f4f16e00640 1 -- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eec06c920 msgr2=0x7f4eec06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.821+0000 7f4f16e00640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eec06c920 0x7f4eec06ede0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f4f00034000 tx=0x7f4f0002e000 comp rx=0 tx=0).stop 2024-04-24T08:18:06.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.821+0000 7f4f16e00640 1 -- 172.21.15.69:0/821655082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 msgr2=0x7f4f101560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:06.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.821+0000 7f4f16e00640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 0x7f4f101560a0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800c7f0 tx=0x7f4ef800ccb0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.821+0000 7f4f16e00640 1 -- 172.21.15.69:0/821655082 shutdown_connections 2024-04-24T08:18:06.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.821+0000 7f4f16e00640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4eec06c920 0x7f4eec06ede0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.821+0000 7f4f16e00640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f1015b490 0x7f4f1015c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.821+0000 7f4f16e00640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4f100cfbe0 0x7f4f1014cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.821+0000 7f4f16e00640 1 --2- 172.21.15.69:0/821655082 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4f1006f2d0 0x7f4f101560a0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:06.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.821+0000 7f4f16e00640 1 -- 172.21.15.69:0/821655082 >> 172.21.15.69:0/821655082 conn(0x7f4f10067d90 msgr2=0x7f4f100b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:06.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.822+0000 7f4f16e00640 1 -- 172.21.15.69:0/821655082 shutdown_connections 2024-04-24T08:18:06.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:06.822+0000 7f4f16e00640 1 -- 172.21.15.69:0/821655082 wait complete. 2024-04-24T08:18:06.842 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:18:06.843 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:18:07.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.038+0000 7fb8b5c00640 1 Processor -- start 2024-04-24T08:18:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.038+0000 7fb8b5c00640 1 -- start start 2024-04-24T08:18:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.039+0000 7fb8b5c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb8b015c9b0 0x7fb8b00b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.039+0000 7fb8b5c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b00b3f70 0x7fb8b00cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.039+0000 7fb8b5c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b00cd6d0 con 0x7fb8b0155f20 2024-04-24T08:18:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.039+0000 7fb8b5c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb8b006db90 con 0x7fb8b015c9b0 2024-04-24T08:18:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.039+0000 7fb8b5c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b0072070 con 0x7fb8b00b3f70 2024-04-24T08:18:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.040+0000 7fb8af400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b00b3f70 0x7fb8b00cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.040+0000 7fb8aea00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fb8b0155f20 0x7fb8b00ccd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:48282/0 (socket says 172.21.15.69:48282) 2024-04-24T08:18:07.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.040+0000 7fb8af400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b00b3f70 0x7fb8b00cf360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:54452/0 (socket says 172.21.15.69:54452) 2024-04-24T08:18:07.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.040+0000 7fb8aea00640 1 -- 172.21.15.69:0/3626196651 learned_addr learned my addr 172.21.15.69:0/3626196651 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:07.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.040+0000 7fb8a6000640 1 --2- 172.21.15.69:0/3626196651 >> v2:172.21.15.121:3300/0 conn(0x7fb8b015c9b0 0x7fb8b00b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.041+0000 7fb8a6000640 1 -- 172.21.15.69:0/3626196651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b00b3f70 msgr2=0x7fb8b00cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:07.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.041+0000 7fb8a6000640 1 --2- 172.21.15.69:0/3626196651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b00b3f70 0x7fb8b00cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.041+0000 7fb8a6000640 1 -- 172.21.15.69:0/3626196651 >> v1:172.21.15.121:6789/0 conn(0x7fb8b0155f20 legacy=0x7fb8b00ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.041+0000 7fb8a6000640 1 -- 172.21.15.69:0/3626196651 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b00cd9b0 con 0x7fb8b015c9b0 2024-04-24T08:18:07.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.041+0000 7fb8ae000640 1 -- 172.21.15.69:0/3626196651 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1474006588 0 0) 0x7fb8b00cd6d0 con 0x7fb8b0155f20 2024-04-24T08:18:07.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.041+0000 7fb8af400640 1 --2- 172.21.15.69:0/3626196651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b00b3f70 0x7fb8b00cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:07.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.042+0000 7fb8a6000640 1 --2- 172.21.15.69:0/3626196651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8b015c9b0 0x7fb8b00b3a30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb8980097a0 tx=0x7fb89802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb9803df03f00fab server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.043+0000 7fb8ae000640 1 -- 172.21.15.69:0/3626196651 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898037070 con 0x7fb8b015c9b0 2024-04-24T08:18:07.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.043+0000 7fb8ae000640 1 -- 172.21.15.69:0/3626196651 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb898032440 con 0x7fb8b015c9b0 2024-04-24T08:18:07.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.043+0000 7fb8ae000640 1 -- 172.21.15.69:0/3626196651 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898032810 con 0x7fb8b015c9b0 2024-04-24T08:18:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.043+0000 7fb8b5c00640 1 -- 172.21.15.69:0/3626196651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8b015c9b0 msgr2=0x7fb8b00b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.043+0000 7fb8b5c00640 1 --2- 172.21.15.69:0/3626196651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8b015c9b0 0x7fb8b00b3a30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb8980097a0 tx=0x7fb89802a210 comp rx=0 tx=0).stop 2024-04-24T08:18:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.044+0000 7fb8b5c00640 1 -- 172.21.15.69:0/3626196651 shutdown_connections 2024-04-24T08:18:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.044+0000 7fb8b5c00640 1 --2- 172.21.15.69:0/3626196651 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b00b3f70 0x7fb8b00cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.044+0000 7fb8b5c00640 1 --2- 172.21.15.69:0/3626196651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8b015c9b0 0x7fb8b00b3a30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.044+0000 7fb8b5c00640 1 -- 172.21.15.69:0/3626196651 >> 172.21.15.69:0/3626196651 conn(0x7fb8b00d0350 msgr2=0x7fb8b00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.044+0000 7fb8b5c00640 1 -- 172.21.15.69:0/3626196651 shutdown_connections 2024-04-24T08:18:07.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.044+0000 7fb8b5c00640 1 -- 172.21.15.69:0/3626196651 wait complete. 2024-04-24T08:18:07.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.045+0000 7fb8b5c00640 1 Processor -- start 2024-04-24T08:18:07.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.046+0000 7fb8b5c00640 1 -- start start 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.046+0000 7fb8b5c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8b00b3f70 0x7fb8b015b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.046+0000 7fb8b5c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8b0155f20 0x7fb8b0151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.046+0000 7fb8b5c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b015c9b0 0x7fb8b00b1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.046+0000 7fb8b5c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b00cf8a0 con 0x7fb8b0155f20 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.046+0000 7fb8b5c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b0071d00 con 0x7fb8b00b3f70 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8b5c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b00cfa20 con 0x7fb8b015c9b0 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8a6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8b0155f20 0x7fb8b0151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8a6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8b0155f20 0x7fb8b0151040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34114/0 (socket says 172.21.15.69:34114) 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8a6000640 1 -- 172.21.15.69:0/1485229416 learned_addr learned my addr 172.21.15.69:0/1485229416 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8af400640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b015c9b0 0x7fb8b00b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8a6000640 1 -- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b015c9b0 msgr2=0x7fb8b00b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8aea00640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8b00b3f70 0x7fb8b015b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8a6000640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b015c9b0 0x7fb8b00b1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8a6000640 1 -- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8b00b3f70 msgr2=0x7fb8b015b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8a6000640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8b00b3f70 0x7fb8b015b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.047+0000 7fb8a6000640 1 -- 172.21.15.69:0/1485229416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b01f8c40 con 0x7fb8b0155f20 2024-04-24T08:18:07.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.048+0000 7fb8a6000640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8b0155f20 0x7fb8b0151040 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fb898035040 tx=0x7fb8980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.048+0000 7fb8a7400640 1 -- 172.21.15.69:0/1485229416 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898037040 con 0x7fb8b0155f20 2024-04-24T08:18:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.048+0000 7fb8a7400640 1 -- 172.21.15.69:0/1485229416 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb898030500 con 0x7fb8b0155f20 2024-04-24T08:18:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.048+0000 7fb8b5c00640 1 -- 172.21.15.69:0/1485229416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b015a850 con 0x7fb8b0155f20 2024-04-24T08:18:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.048+0000 7fb8a7400640 1 -- 172.21.15.69:0/1485229416 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8980308d0 con 0x7fb8b0155f20 2024-04-24T08:18:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.048+0000 7fb8b5c00640 1 -- 172.21.15.69:0/1485229416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b00b1570 con 0x7fb8b0155f20 2024-04-24T08:18:07.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.049+0000 7fb8b5c00640 1 -- 172.21.15.69:0/1485229416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b0068a80 con 0x7fb8b0155f20 2024-04-24T08:18:07.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.050+0000 7fb8a7400640 1 -- 172.21.15.69:0/1485229416 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb898044020 con 0x7fb8b0155f20 2024-04-24T08:18:07.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.051+0000 7fb8a7400640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb88c06c8f0 0x7fb88c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.051+0000 7fb8a7400640 1 -- 172.21.15.69:0/1485229416 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 4857+0+0 (secure 0 0 0) 0x7fb8980aba20 con 0x7fb8b0155f20 2024-04-24T08:18:07.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.052+0000 7fb8aea00640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb88c06c8f0 0x7fb88c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.053+0000 7fb8aea00640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb88c06c8f0 0x7fb88c06edb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0007fd0 tx=0x7fb8a0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.055+0000 7fb8a7400640 1 -- 172.21.15.69:0/1485229416 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb89807a190 con 0x7fb8b0155f20 2024-04-24T08:18:07.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.262+0000 7fb8b5c00640 1 -- 172.21.15.69:0/1485229416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb8b0069240 con 0x7fb8b0155f20 2024-04-24T08:18:07.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.263+0000 7fb8a7400640 1 -- 172.21.15.69:0/1485229416 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fb898079a90 con 0x7fb8b0155f20 2024-04-24T08:18:07.264 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 -- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb88c06c8f0 msgr2=0x7fb88c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb88c06c8f0 0x7fb88c06edb0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0007fd0 tx=0x7fb8a0009040 comp rx=0 tx=0).stop 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 -- 172.21.15.69:0/1485229416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8b0155f20 msgr2=0x7fb8b0151040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8b0155f20 0x7fb8b0151040 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7fb898035040 tx=0x7fb8980046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 -- 172.21.15.69:0/1485229416 shutdown_connections 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb88c06c8f0 0x7fb88c06edb0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8b0155f20 0x7fb8b0151040 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8b015c9b0 0x7fb8b00b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 --2- 172.21.15.69:0/1485229416 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8b00b3f70 0x7fb8b015b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 -- 172.21.15.69:0/1485229416 >> 172.21.15.69:0/1485229416 conn(0x7fb8b00d0350 msgr2=0x7fb8b006dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:07.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 -- 172.21.15.69:0/1485229416 shutdown_connections 2024-04-24T08:18:07.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.266+0000 7fb8b5c00640 1 -- 172.21.15.69:0/1485229416 wait complete. 2024-04-24T08:18:07.286 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T08:18:07.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.481+0000 7ff24c400640 1 Processor -- start 2024-04-24T08:18:07.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.481+0000 7ff24c400640 1 -- start start 2024-04-24T08:18:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.483+0000 7ff24c400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff2440af9a0 0x7ff2441697e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.483+0000 7ff24c400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2440adab0 con 0x7ff24415ea10 2024-04-24T08:18:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.483+0000 7ff24c400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2440add50 con 0x7ff2441631a0 2024-04-24T08:18:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.483+0000 7ff24c400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff2440ae030 con 0x7ff2440af9a0 2024-04-24T08:18:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.483+0000 7ff249a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff24415ea10 0x7ff2440d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47440/0 (socket says 172.21.15.69:47440) 2024-04-24T08:18:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.483+0000 7ff249a00640 1 -- 172.21.15.69:0/2232095625 learned_addr learned my addr 172.21.15.69:0/2232095625 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:07.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.483+0000 7ff24a400640 1 --2- 172.21.15.69:0/2232095625 >> v2:172.21.15.69:3300/0 conn(0x7ff2440af9a0 0x7ff2441697e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.484+0000 7ff243e00640 1 -- 172.21.15.69:0/2232095625 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2598030750 0 0) 0x7ff2440adab0 con 0x7ff24415ea10 2024-04-24T08:18:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.484+0000 7ff243e00640 1 -- 172.21.15.69:0/2232095625 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff22c003520 con 0x7ff24415ea10 2024-04-24T08:18:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.484+0000 7ff24a400640 1 -- 172.21.15.69:0/2232095625 >> v1:172.21.15.121:6789/0 conn(0x7ff2441631a0 legacy=0x7ff2440af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.484+0000 7ff24a400640 1 -- 172.21.15.69:0/2232095625 >> v1:172.21.15.69:6789/0 conn(0x7ff24415ea10 legacy=0x7ff2440d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.484+0000 7ff24a400640 1 -- 172.21.15.69:0/2232095625 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2440ae310 con 0x7ff2440af9a0 2024-04-24T08:18:07.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.485+0000 7ff24a400640 1 --2- 172.21.15.69:0/2232095625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2440af9a0 0x7ff2441697e0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009d90 tx=0x7ff23c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac808a90a412bd67 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.486+0000 7ff243e00640 1 -- 172.21.15.69:0/2232095625 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c007740 con 0x7ff2440af9a0 2024-04-24T08:18:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.486+0000 7ff243e00640 1 -- 172.21.15.69:0/2232095625 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff23c0078e0 con 0x7ff2440af9a0 2024-04-24T08:18:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.486+0000 7ff243e00640 1 -- 172.21.15.69:0/2232095625 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c007cb0 con 0x7ff2440af9a0 2024-04-24T08:18:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.486+0000 7ff24c400640 1 -- 172.21.15.69:0/2232095625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2440af9a0 msgr2=0x7ff2441697e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.486+0000 7ff24c400640 1 --2- 172.21.15.69:0/2232095625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2440af9a0 0x7ff2441697e0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009d90 tx=0x7ff23c02a6f0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.486+0000 7ff24c400640 1 -- 172.21.15.69:0/2232095625 shutdown_connections 2024-04-24T08:18:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.486+0000 7ff24c400640 1 --2- 172.21.15.69:0/2232095625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2440af9a0 0x7ff2441697e0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.486+0000 7ff24c400640 1 -- 172.21.15.69:0/2232095625 >> 172.21.15.69:0/2232095625 conn(0x7ff244067d90 msgr2=0x7ff24406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.486+0000 7ff24c400640 1 -- 172.21.15.69:0/2232095625 shutdown_connections 2024-04-24T08:18:07.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.486+0000 7ff24c400640 1 -- 172.21.15.69:0/2232095625 wait complete. 2024-04-24T08:18:07.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.488+0000 7ff24c400640 1 Processor -- start 2024-04-24T08:18:07.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.488+0000 7ff24c400640 1 -- start start 2024-04-24T08:18:07.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.489+0000 7ff24c400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2440af9a0 0x7ff2440cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.489+0000 7ff24c400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff24415ea10 0x7ff2441534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.489+0000 7ff24c400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2441631a0 0x7ff244153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.489+0000 7ff24c400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff2440aedf0 con 0x7ff2441631a0 2024-04-24T08:18:07.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.489+0000 7ff24c400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff2440ae8d0 con 0x7ff2440af9a0 2024-04-24T08:18:07.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.489+0000 7ff24c400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff24406f000 con 0x7ff24415ea10 2024-04-24T08:18:07.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff24a400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2441631a0 0x7ff244153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff24a400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2441631a0 0x7ff244153a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34126/0 (socket says 172.21.15.69:34126) 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff24a400640 1 -- 172.21.15.69:0/1355992263 learned_addr learned my addr 172.21.15.69:0/1355992263 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff249000640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff24415ea10 0x7ff2441534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff249a00640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2440af9a0 0x7ff2440cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff24a400640 1 -- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff24415ea10 msgr2=0x7ff2441534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff24a400640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff24415ea10 0x7ff2441534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff24a400640 1 -- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2440af9a0 msgr2=0x7ff2440cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff24a400640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2440af9a0 0x7ff2440cc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff24a400640 1 -- 172.21.15.69:0/1355992263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2440cec90 con 0x7ff2441631a0 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.490+0000 7ff24a400640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2441631a0 0x7ff244153a00 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009eb0 tx=0x7ff23c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.491+0000 7ff241600640 1 -- 172.21.15.69:0/1355992263 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c040070 con 0x7ff2441631a0 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.491+0000 7ff241600640 1 -- 172.21.15.69:0/1355992263 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff23c031bc0 con 0x7ff2441631a0 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.491+0000 7ff241600640 1 -- 172.21.15.69:0/1355992263 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c009450 con 0x7ff2441631a0 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.491+0000 7ff24c400640 1 -- 172.21.15.69:0/1355992263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2441f51f0 con 0x7ff2441631a0 2024-04-24T08:18:07.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.491+0000 7ff24c400640 1 -- 172.21.15.69:0/1355992263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2440b1360 con 0x7ff2441631a0 2024-04-24T08:18:07.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.492+0000 7ff24c400640 1 -- 172.21.15.69:0/1355992263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff244068a80 con 0x7ff2441631a0 2024-04-24T08:18:07.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.493+0000 7ff241600640 1 -- 172.21.15.69:0/1355992263 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff23c039040 con 0x7ff2441631a0 2024-04-24T08:18:07.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.494+0000 7ff241600640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff22406c8f0 0x7ff22406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.494+0000 7ff241600640 1 -- 172.21.15.69:0/1355992263 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 4857+0+0 (secure 0 0 0) 0x7ff23c0ad0f0 con 0x7ff2441631a0 2024-04-24T08:18:07.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.494+0000 7ff249a00640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff22406c8f0 0x7ff22406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.496+0000 7ff249a00640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff22406c8f0 0x7ff22406edb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff2380011e0 tx=0x7ff238009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.498+0000 7ff241600640 1 -- 172.21.15.69:0/1355992263 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff2440cec90 con 0x7ff2441631a0 2024-04-24T08:18:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.698+0000 7ff24c400640 1 -- 172.21.15.69:0/1355992263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff2440af760 con 0x7ff2441631a0 2024-04-24T08:18:07.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.699+0000 7ff241600640 1 -- 172.21.15.69:0/1355992263 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v93) ==== 84+0+65 (secure 0 0 0) 0x7ff23c07b7d0 con 0x7ff2441631a0 2024-04-24T08:18:07.700 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:07.700 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:18:07.700 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:18:07.700 INFO:teuthology.orchestra.run.smithi069.stdout: "poolnum": 1, 2024-04-24T08:18:07.700 INFO:teuthology.orchestra.run.smithi069.stdout: "poolname": ".mgr" 2024-04-24T08:18:07.700 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:07.700 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:18:07.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.701+0000 7ff24c400640 1 -- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff22406c8f0 msgr2=0x7ff22406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.701+0000 7ff24c400640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff22406c8f0 0x7ff22406edb0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff2380011e0 tx=0x7ff238009040 comp rx=0 tx=0).stop 2024-04-24T08:18:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.701+0000 7ff24c400640 1 -- 172.21.15.69:0/1355992263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2441631a0 msgr2=0x7ff244153a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.701+0000 7ff24c400640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2441631a0 0x7ff244153a00 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009eb0 tx=0x7ff23c004a40 comp rx=0 tx=0).stop 2024-04-24T08:18:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.702+0000 7ff24c400640 1 -- 172.21.15.69:0/1355992263 shutdown_connections 2024-04-24T08:18:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.702+0000 7ff24c400640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff22406c8f0 0x7ff22406edb0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.702+0000 7ff24c400640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff2441631a0 0x7ff244153a00 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.702+0000 7ff24c400640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff24415ea10 0x7ff2441534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.702+0000 7ff24c400640 1 --2- 172.21.15.69:0/1355992263 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2440af9a0 0x7ff2440cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.702+0000 7ff24c400640 1 -- 172.21.15.69:0/1355992263 >> 172.21.15.69:0/1355992263 conn(0x7ff244067d90 msgr2=0x7ff2440b3fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.702+0000 7ff24c400640 1 -- 172.21.15.69:0/1355992263 shutdown_connections 2024-04-24T08:18:07.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.702+0000 7ff24c400640 1 -- 172.21.15.69:0/1355992263 wait complete. 2024-04-24T08:18:07.722 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T08:18:07.722 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T08:18:07.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.925+0000 7f8a05200640 1 Processor -- start 2024-04-24T08:18:07.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.926+0000 7f8a05200640 1 -- start start 2024-04-24T08:18:07.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.927+0000 7f8a05200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8a00070ce0 0x7f8a000d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.927+0000 7f8a05200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a00070390 con 0x7f8a0015cb70 2024-04-24T08:18:07.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.927+0000 7f8a05200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a0006fdd0 con 0x7f8a000ce2c0 2024-04-24T08:18:07.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.927+0000 7f8a05200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8a000700b0 con 0x7f8a00070ce0 2024-04-24T08:18:07.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.927+0000 7f89fde00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8a0015cb70 0x7f8a000705d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47456/0 (socket says 172.21.15.69:47456) 2024-04-24T08:18:07.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.927+0000 7f89fde00640 1 -- 172.21.15.69:0/2597432087 learned_addr learned my addr 172.21.15.69:0/2597432087 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:07.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.927+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4036099562 0 0) 0x7f8a00070390 con 0x7f8a0015cb70 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89e4003520 con 0x7f8a0015cb70 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fd400640 1 --2- 172.21.15.69:0/2597432087 >> v2:172.21.15.121:3300/0 conn(0x7f8a00070ce0 0x7f8a000d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3393718836 0 0) 0x7f89e4003520 con 0x7f8a0015cb70 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 >> v2:172.21.15.121:3300/0 conn(0x7f8a00070ce0 msgr2=0x7f8a000d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 --2- 172.21.15.69:0/2597432087 >> v2:172.21.15.121:3300/0 conn(0x7f8a00070ce0 0x7f8a000d0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 >> v1:172.21.15.121:6789/0 conn(0x7f8a000ce2c0 legacy=0x7f8a000d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a000d04d0 con 0x7f8a0015cb70 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f89f00019f0 con 0x7f8a0015cb70 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 >> v1:172.21.15.69:6789/0 conn(0x7f8a0015cb70 legacy=0x7f8a000705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89e4003a00 0x7f89e4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89e4004800 0x7f89e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89e4005210 0x7f89e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a000700b0 con 0x7f89e4003a00 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f89e400c160 con 0x7f89e4005210 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f89e400c360 con 0x7f89e4004800 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.928+0000 7f89fde00640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89e4003a00 0x7f89e4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.929+0000 7f89fd400640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89e4004800 0x7f89e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.929+0000 7f89fe800640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89e4005210 0x7f89e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.929+0000 7f89fde00640 1 -- 172.21.15.69:0/2597432087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89e4004800 msgr2=0x7f89e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.929+0000 7f89fde00640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f89e4004800 0x7f89e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.929+0000 7f89fde00640 1 -- 172.21.15.69:0/2597432087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89e4005210 msgr2=0x7f89e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.929+0000 7f89fde00640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89e4005210 0x7f89e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.929+0000 7f89fde00640 1 -- 172.21.15.69:0/2597432087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e4003520 con 0x7f89e4003a00 2024-04-24T08:18:07.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.929+0000 7f89fe800640 1 -- 172.21.15.69:0/2597432087 reap_dead start 2024-04-24T08:18:07.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.930+0000 7f89fde00640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89e4003a00 0x7f89e4003e20 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f89f000a790 tx=0x7f89e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c2e2ac38574995 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.930+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f000a3b0 con 0x7f89e4003a00 2024-04-24T08:18:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.930+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89f002f070 con 0x7f89e4003a00 2024-04-24T08:18:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.930+0000 7f89fca00640 1 -- 172.21.15.69:0/2597432087 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f002bc40 con 0x7f89e4003a00 2024-04-24T08:18:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.931+0000 7f8a05200640 1 -- 172.21.15.69:0/2597432087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89e4003a00 msgr2=0x7f89e4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.931+0000 7f8a05200640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89e4003a00 0x7f89e4003e20 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f89f000a790 tx=0x7f89e4005830 comp rx=0 tx=0).stop 2024-04-24T08:18:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.931+0000 7f8a05200640 1 -- 172.21.15.69:0/2597432087 shutdown_connections 2024-04-24T08:18:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.931+0000 7f8a05200640 1 --2- 172.21.15.69:0/2597432087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89e4003a00 0x7f89e4003e20 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.931+0000 7f8a05200640 1 -- 172.21.15.69:0/2597432087 >> 172.21.15.69:0/2597432087 conn(0x7f8a000d2530 msgr2=0x7f8a000d2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.931+0000 7f8a05200640 1 -- 172.21.15.69:0/2597432087 shutdown_connections 2024-04-24T08:18:07.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.931+0000 7f8a05200640 1 -- 172.21.15.69:0/2597432087 wait complete. 2024-04-24T08:18:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.933+0000 7f8a05200640 1 Processor -- start 2024-04-24T08:18:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.933+0000 7f8a05200640 1 -- start start 2024-04-24T08:18:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f8a05200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89e4003a00 0x7f8a0014aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f8a05200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a0015fdf0 0x7f8a0015cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f8a05200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a0014b170 0x7f8a00159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f8a05200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a000d31d0 con 0x7f8a0014b170 2024-04-24T08:18:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f8a05200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8a000d3050 con 0x7f89e4003a00 2024-04-24T08:18:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f8a05200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8a000b1ad0 con 0x7f8a0015fdf0 2024-04-24T08:18:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f89fe800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a0014b170 0x7f8a00159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f89fe800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a0014b170 0x7f8a00159ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34146/0 (socket says 172.21.15.69:34146) 2024-04-24T08:18:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f89fe800640 1 -- 172.21.15.69:0/784275242 learned_addr learned my addr 172.21.15.69:0/784275242 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f89fd400640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a0015fdf0 0x7f8a0015cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f89fe800640 1 -- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a0015fdf0 msgr2=0x7f8a0015cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f89fe800640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a0015fdf0 0x7f8a0015cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f89fe800640 1 -- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89e4003a00 msgr2=0x7f8a0014aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f89fe800640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89e4003a00 0x7f8a0014aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.934+0000 7f89fe800640 1 -- 172.21.15.69:0/784275242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a00160270 con 0x7f8a0014b170 2024-04-24T08:18:07.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.935+0000 7f89fe800640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a0014b170 0x7f8a00159ba0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f8a00073af0 tx=0x7f89f800e670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.935+0000 7f89f6000640 1 -- 172.21.15.69:0/784275242 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8019070 con 0x7f8a0014b170 2024-04-24T08:18:07.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.935+0000 7f89f6000640 1 -- 172.21.15.69:0/784275242 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89f80052e0 con 0x7f8a0014b170 2024-04-24T08:18:07.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.935+0000 7f89f6000640 1 -- 172.21.15.69:0/784275242 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f8005490 con 0x7f8a0014b170 2024-04-24T08:18:07.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.935+0000 7f8a05200640 1 -- 172.21.15.69:0/784275242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a000cb040 con 0x7f8a0014b170 2024-04-24T08:18:07.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.935+0000 7f8a05200640 1 -- 172.21.15.69:0/784275242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a000cb600 con 0x7f8a0014b170 2024-04-24T08:18:07.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.936+0000 7f8a05200640 1 -- 172.21.15.69:0/784275242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a00075dc0 con 0x7f8a0014b170 2024-04-24T08:18:07.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.937+0000 7f89f6000640 1 -- 172.21.15.69:0/784275242 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f89f801c070 con 0x7f8a0014b170 2024-04-24T08:18:07.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.938+0000 7f89f6000640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89dc06c8f0 0x7f89dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:07.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.938+0000 7f89f6000640 1 -- 172.21.15.69:0/784275242 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 4857+0+0 (secure 0 0 0) 0x7f89f808ab30 con 0x7f8a0014b170 2024-04-24T08:18:07.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.939+0000 7f89fde00640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89dc06c8f0 0x7f89dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:07.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.940+0000 7f89fde00640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89dc06c8f0 0x7f89dc06edb0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f89f0033000 tx=0x7f89f0007000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:07.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:07.943+0000 7f89f6000640 1 -- 172.21.15.69:0/784275242 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89f80592a0 con 0x7f8a0014b170 2024-04-24T08:18:08.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.142+0000 7f8a05200640 1 -- 172.21.15.69:0/784275242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8a00070390 con 0x7f8a0014b170 2024-04-24T08:18:08.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.905+0000 7f89f6000640 1 -- 172.21.15.69:0/784275242 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v94) ==== 138+0+0 (secure 0 0 0) 0x7f89f8058ba0 con 0x7f8a0014b170 2024-04-24T08:18:08.906 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' created 2024-04-24T08:18:08.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 -- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89dc06c8f0 msgr2=0x7f89dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:08.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89dc06c8f0 0x7f89dc06edb0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f89f0033000 tx=0x7f89f0007000 comp rx=0 tx=0).stop 2024-04-24T08:18:08.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 -- 172.21.15.69:0/784275242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a0014b170 msgr2=0x7f8a00159ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:08.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a0014b170 0x7f8a00159ba0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f8a00073af0 tx=0x7f89f800e670 comp rx=0 tx=0).stop 2024-04-24T08:18:08.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 -- 172.21.15.69:0/784275242 shutdown_connections 2024-04-24T08:18:08.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89dc06c8f0 0x7f89dc06edb0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:08.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a0014b170 0x7f8a00159ba0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:08.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a0015fdf0 0x7f8a0015cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:08.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 --2- 172.21.15.69:0/784275242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89e4003a00 0x7f8a0014aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:08.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 -- 172.21.15.69:0/784275242 >> 172.21.15.69:0/784275242 conn(0x7f8a000d2530 msgr2=0x7f8a0015c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:08.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 -- 172.21.15.69:0/784275242 shutdown_connections 2024-04-24T08:18:08.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:08.909+0000 7f8a05200640 1 -- 172.21.15.69:0/784275242 wait complete. 2024-04-24T08:18:08.939 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T08:18:09.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.134+0000 7f6694400640 1 Processor -- start 2024-04-24T08:18:09.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.135+0000 7f6694400640 1 -- start start 2024-04-24T08:18:09.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.136+0000 7f6694400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f668c15ea90 0x7f668c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:09.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.136+0000 7f6694400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f668c0d06b0 0x7f668c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:09.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.136+0000 7f6694400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f668c15ee70 0x7f668c0cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:09.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.136+0000 7f6694400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f668c0b8c90 con 0x7f668c15ea90 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.136+0000 7f6694400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f668c0b8080 con 0x7f668c15ee70 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.136+0000 7f6694400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f668c0b8360 con 0x7f668c0d06b0 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.136+0000 7f6691a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f668c15ea90 0x7f668c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.136+0000 7f6691a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f668c15ea90 0x7f668c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40216/0 (socket says 172.21.15.69:40216) 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.136+0000 7f6691a00640 1 -- 172.21.15.69:0/850508494 learned_addr learned my addr 172.21.15.69:0/850508494 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.137+0000 7f6692400640 1 --2- 172.21.15.69:0/850508494 >> v2:172.21.15.121:3300/0 conn(0x7f668c15ee70 0x7f668c0cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.137+0000 7f6691000640 1 --2- 172.21.15.69:0/850508494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f668c0d06b0 0x7f668c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.137+0000 7f6691a00640 1 -- 172.21.15.69:0/850508494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f668c0d06b0 msgr2=0x7f668c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.137+0000 7f6691a00640 1 --2- 172.21.15.69:0/850508494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f668c0d06b0 0x7f668c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.137+0000 7f6691a00640 1 -- 172.21.15.69:0/850508494 >> v2:172.21.15.121:3300/0 conn(0x7f668c15ee70 msgr2=0x7f668c0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:09.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.137+0000 7f6691a00640 1 --2- 172.21.15.69:0/850508494 >> v2:172.21.15.121:3300/0 conn(0x7f668c15ee70 0x7f668c0cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.137+0000 7f6691a00640 1 -- 172.21.15.69:0/850508494 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668c0b8640 con 0x7f668c15ea90 2024-04-24T08:18:09.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.138+0000 7f6691a00640 1 --2- 172.21.15.69:0/850508494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 0x7f668c0cf290 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f668c06c6a0 tx=0x7f667c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c250306333fa250 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:09.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.138+0000 7f668be00640 1 -- 172.21.15.69:0/850508494 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c037070 con 0x7f668c15ea90 2024-04-24T08:18:09.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.138+0000 7f668be00640 1 -- 172.21.15.69:0/850508494 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f667c032440 con 0x7f668c15ea90 2024-04-24T08:18:09.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.138+0000 7f668be00640 1 -- 172.21.15.69:0/850508494 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c032810 con 0x7f668c15ea90 2024-04-24T08:18:09.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.139+0000 7f6694400640 1 -- 172.21.15.69:0/850508494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 msgr2=0x7f668c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:09.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.139+0000 7f6694400640 1 --2- 172.21.15.69:0/850508494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 0x7f668c0cf290 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f668c06c6a0 tx=0x7f667c02a1c0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.139+0000 7f6694400640 1 -- 172.21.15.69:0/850508494 shutdown_connections 2024-04-24T08:18:09.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.139+0000 7f6694400640 1 --2- 172.21.15.69:0/850508494 >> v2:172.21.15.121:3300/0 conn(0x7f668c15ee70 0x7f668c0cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.139+0000 7f6694400640 1 --2- 172.21.15.69:0/850508494 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f668c0d06b0 0x7f668c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.139+0000 7f6694400640 1 --2- 172.21.15.69:0/850508494 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 0x7f668c0cf290 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.139+0000 7f6694400640 1 -- 172.21.15.69:0/850508494 >> 172.21.15.69:0/850508494 conn(0x7f668c08bf60 msgr2=0x7f668c0d4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:09.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.139+0000 7f6694400640 1 -- 172.21.15.69:0/850508494 shutdown_connections 2024-04-24T08:18:09.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.139+0000 7f6694400640 1 -- 172.21.15.69:0/850508494 wait complete. 2024-04-24T08:18:09.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.141+0000 7f6694400640 1 Processor -- start 2024-04-24T08:18:09.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.141+0000 7f6694400640 1 -- start start 2024-04-24T08:18:09.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.142+0000 7f6694400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c0d06b0 0x7f668c1f42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:09.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.142+0000 7f6694400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 0x7f668c09f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:09.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.142+0000 7f6694400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f668c15ee70 0x7f668c0a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:09.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.142+0000 7f6694400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f668c0d2650 con 0x7f668c15ea90 2024-04-24T08:18:09.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.142+0000 7f6694400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f668c0d24d0 con 0x7f668c0d06b0 2024-04-24T08:18:09.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.142+0000 7f6694400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f668c0d27d0 con 0x7f668c15ee70 2024-04-24T08:18:09.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.142+0000 7f6691000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 0x7f668c09f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:09.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.142+0000 7f6691000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 0x7f668c09f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40232/0 (socket says 172.21.15.69:40232) 2024-04-24T08:18:09.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.142+0000 7f6691000640 1 -- 172.21.15.69:0/2897482080 learned_addr learned my addr 172.21.15.69:0/2897482080 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:09.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6691a00640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c0d06b0 0x7f668c1f42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6692400640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f668c15ee70 0x7f668c0a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6691000640 1 -- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f668c15ee70 msgr2=0x7f668c0a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6691000640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f668c15ee70 0x7f668c0a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6691000640 1 -- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c0d06b0 msgr2=0x7f668c1f42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6691000640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c0d06b0 0x7f668c1f42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6691000640 1 -- 172.21.15.69:0/2897482080 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668c09fc70 con 0x7f668c15ea90 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6691a00640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c0d06b0 0x7f668c1f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6691000640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 0x7f668c09f420 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f667800d800 tx=0x7f667800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6689600640 1 -- 172.21.15.69:0/2897482080 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66780096a0 con 0x7f668c15ea90 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6689600640 1 -- 172.21.15.69:0/2897482080 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6678019070 con 0x7f668c15ea90 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6689600640 1 -- 172.21.15.69:0/2897482080 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678009a60 con 0x7f668c15ea90 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.143+0000 7f6694400640 1 -- 172.21.15.69:0/2897482080 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668c09ff50 con 0x7f668c15ea90 2024-04-24T08:18:09.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.144+0000 7f6694400640 1 -- 172.21.15.69:0/2897482080 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668c0a0510 con 0x7f668c15ea90 2024-04-24T08:18:09.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.145+0000 7f6694400640 1 -- 172.21.15.69:0/2897482080 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f668c068a80 con 0x7f668c15ea90 2024-04-24T08:18:09.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.146+0000 7f6689600640 1 -- 172.21.15.69:0/2897482080 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6678009c00 con 0x7f668c15ea90 2024-04-24T08:18:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.147+0000 7f6689600640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 0x7f666c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.148+0000 7f6689600640 1 -- 172.21.15.69:0/2897482080 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 5227+0+0 (secure 0 0 0) 0x7f667808be50 con 0x7f668c15ea90 2024-04-24T08:18:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.148+0000 7f6691a00640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 0x7f666c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.149+0000 7f6691a00640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 0x7f666c06edb0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f667c034000 tx=0x7f667c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:09.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.153+0000 7f6689600640 1 -- 172.21.15.69:0/2897482080 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f667805a3c0 con 0x7f668c15ea90 2024-04-24T08:18:09.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.354+0000 7f6694400640 1 -- 172.21.15.69:0/2897482080 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f668c09f960 con 0x7f668c15ea90 2024-04-24T08:18:09.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.924+0000 7f6689600640 1 -- 172.21.15.69:0/2897482080 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v95) ==== 170+0+0 (secure 0 0 0) 0x7f6678059cc0 con 0x7f668c15ea90 2024-04-24T08:18:09.925 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' created 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.927+0000 7f6694400640 1 -- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 msgr2=0x7f666c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 0x7f666c06edb0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f667c034000 tx=0x7f667c02e000 comp rx=0 tx=0).stop 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 -- 172.21.15.69:0/2897482080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 msgr2=0x7f668c09f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 0x7f668c09f420 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f667800d800 tx=0x7f667800dcc0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 -- 172.21.15.69:0/2897482080 shutdown_connections 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f666c06c8f0 0x7f666c06edb0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f668c15ea90 0x7f668c09f420 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f668c15ee70 0x7f668c0a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 --2- 172.21.15.69:0/2897482080 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f668c0d06b0 0x7f668c1f42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 -- 172.21.15.69:0/2897482080 >> 172.21.15.69:0/2897482080 conn(0x7f668c08bf60 msgr2=0x7f668c0d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 -- 172.21.15.69:0/2897482080 shutdown_connections 2024-04-24T08:18:09.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:09.928+0000 7f6694400640 1 -- 172.21.15.69:0/2897482080 wait complete. 2024-04-24T08:18:09.960 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T08:18:10.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.136+0000 7fec76c00640 1 Processor -- start 2024-04-24T08:18:10.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.136+0000 7fec76c00640 1 -- start start 2024-04-24T08:18:10.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.138+0000 7fec76c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fec700b46a0 0x7fec700b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:10.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.138+0000 7fec76c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec700739f0 con 0x7fec700b3de0 2024-04-24T08:18:10.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.138+0000 7fec76c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec70073cc0 con 0x7fec7015f4b0 2024-04-24T08:18:10.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.138+0000 7fec76c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fec700b5010 con 0x7fec700b46a0 2024-04-24T08:18:10.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.138+0000 7fec6f400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fec700b3de0 0x7fec700732e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51954/0 (socket says 172.21.15.69:51954) 2024-04-24T08:18:10.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.138+0000 7fec6f400640 1 -- 172.21.15.69:0/1181945585 learned_addr learned my addr 172.21.15.69:0/1181945585 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:10.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.138+0000 7fec74c00640 1 --2- 172.21.15.69:0/1181945585 >> v2:172.21.15.69:3300/0 conn(0x7fec700b46a0 0x7fec700b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.139+0000 7fec6ea00640 1 -- 172.21.15.69:0/1181945585 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1976447307 0 0) 0x7fec700739f0 con 0x7fec700b3de0 2024-04-24T08:18:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.139+0000 7fec74c00640 1 -- 172.21.15.69:0/1181945585 >> v1:172.21.15.121:6789/0 conn(0x7fec7015f4b0 legacy=0x7fec7015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.139+0000 7fec74c00640 1 -- 172.21.15.69:0/1181945585 >> v1:172.21.15.69:6789/0 conn(0x7fec700b3de0 legacy=0x7fec700732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.139+0000 7fec74c00640 1 -- 172.21.15.69:0/1181945585 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec70073fa0 con 0x7fec700b46a0 2024-04-24T08:18:10.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.140+0000 7fec74c00640 1 --2- 172.21.15.69:0/1181945585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec700b46a0 0x7fec700b4ad0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fec68009d90 tx=0x7fec6802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab81852edabeb3d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:10.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.140+0000 7fec6ea00640 1 -- 172.21.15.69:0/1181945585 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68007740 con 0x7fec700b46a0 2024-04-24T08:18:10.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.140+0000 7fec6ea00640 1 -- 172.21.15.69:0/1181945585 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec680078e0 con 0x7fec700b46a0 2024-04-24T08:18:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.140+0000 7fec6ea00640 1 -- 172.21.15.69:0/1181945585 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68007cb0 con 0x7fec700b46a0 2024-04-24T08:18:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.141+0000 7fec76c00640 1 -- 172.21.15.69:0/1181945585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec700b46a0 msgr2=0x7fec700b4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.141+0000 7fec76c00640 1 --2- 172.21.15.69:0/1181945585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec700b46a0 0x7fec700b4ad0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7fec68009d90 tx=0x7fec6802a6a0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.141+0000 7fec76c00640 1 -- 172.21.15.69:0/1181945585 shutdown_connections 2024-04-24T08:18:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.141+0000 7fec76c00640 1 --2- 172.21.15.69:0/1181945585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec700b46a0 0x7fec700b4ad0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.141+0000 7fec76c00640 1 -- 172.21.15.69:0/1181945585 >> 172.21.15.69:0/1181945585 conn(0x7fec700d0350 msgr2=0x7fec700d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.141+0000 7fec76c00640 1 -- 172.21.15.69:0/1181945585 shutdown_connections 2024-04-24T08:18:10.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.141+0000 7fec76c00640 1 -- 172.21.15.69:0/1181945585 wait complete. 2024-04-24T08:18:10.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.143+0000 7fec76c00640 1 Processor -- start 2024-04-24T08:18:10.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.143+0000 7fec76c00640 1 -- start start 2024-04-24T08:18:10.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec76c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec700b3de0 0x7fec701495e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:10.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec76c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec700b46a0 0x7fec7009a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec76c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7015f4b0 0x7fec7009a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec76c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fec700b38c0 con 0x7fec7015f4b0 2024-04-24T08:18:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec76c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fec700b35e0 con 0x7fec700b46a0 2024-04-24T08:18:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec76c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fec700ce250 con 0x7fec700b3de0 2024-04-24T08:18:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec74c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7015f4b0 0x7fec7009a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec74c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7015f4b0 0x7fec7009a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40242/0 (socket says 172.21.15.69:40242) 2024-04-24T08:18:10.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec74c00640 1 -- 172.21.15.69:0/2204232539 learned_addr learned my addr 172.21.15.69:0/2204232539 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:10.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec6f400640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec700b46a0 0x7fec7009a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:10.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec74c00640 1 -- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec700b3de0 msgr2=0x7fec701495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec74c00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec700b3de0 0x7fec701495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec74c00640 1 -- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec700b46a0 msgr2=0x7fec7009a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.144+0000 7fec74c00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec700b46a0 0x7fec7009a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.145+0000 7fec74c00640 1 -- 172.21.15.69:0/2204232539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec70154e70 con 0x7fec7015f4b0 2024-04-24T08:18:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.145+0000 7fec74c00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7015f4b0 0x7fec7009a890 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fec6802ac90 tx=0x7fec68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.145+0000 7fec67e00640 1 -- 172.21.15.69:0/2204232539 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68040070 con 0x7fec7015f4b0 2024-04-24T08:18:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.145+0000 7fec67e00640 1 -- 172.21.15.69:0/2204232539 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec68031c90 con 0x7fec7015f4b0 2024-04-24T08:18:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.145+0000 7fec67e00640 1 -- 172.21.15.69:0/2204232539 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec68045680 con 0x7fec7015f4b0 2024-04-24T08:18:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.145+0000 7fec76c00640 1 -- 172.21.15.69:0/2204232539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec70155150 con 0x7fec7015f4b0 2024-04-24T08:18:10.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.145+0000 7fec76c00640 1 -- 172.21.15.69:0/2204232539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec70155710 con 0x7fec7015f4b0 2024-04-24T08:18:10.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.147+0000 7fec76c00640 1 -- 172.21.15.69:0/2204232539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec70068a80 con 0x7fec7015f4b0 2024-04-24T08:18:10.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.148+0000 7fec67e00640 1 -- 172.21.15.69:0/2204232539 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fec68039040 con 0x7fec7015f4b0 2024-04-24T08:18:10.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.149+0000 7fec67e00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec5006c8f0 0x7fec5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:10.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.150+0000 7fec67e00640 1 -- 172.21.15.69:0/2204232539 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 5609+0+0 (secure 0 0 0) 0x7fec680ad5f0 con 0x7fec7015f4b0 2024-04-24T08:18:10.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.150+0000 7fec6fe00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec5006c8f0 0x7fec5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:10.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.151+0000 7fec6fe00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec5006c8f0 0x7fec5006edb0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fec60003cc0 tx=0x7fec600079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:10.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.155+0000 7fec67e00640 1 -- 172.21.15.69:0/2204232539 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec6807bbc0 con 0x7fec7015f4b0 2024-04-24T08:18:10.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.395+0000 7fec76c00640 1 -- 172.21.15.69:0/2204232539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fec70158800 con 0x7fec7015f4b0 2024-04-24T08:18:10.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.928+0000 7fec67e00640 1 -- 172.21.15.69:0/2204232539 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v79) ==== 174+0+0 (secure 0 0 0) 0x7fec6807b4c0 con 0x7fec7015f4b0 2024-04-24T08:18:10.929 INFO:teuthology.orchestra.run.smithi069.stderr:new fs with metadata pool 14 and data pool 15 2024-04-24T08:18:10.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.932+0000 7fec76c00640 1 -- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec5006c8f0 msgr2=0x7fec5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.932+0000 7fec76c00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec5006c8f0 0x7fec5006edb0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fec60003cc0 tx=0x7fec600079b0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.932+0000 7fec76c00640 1 -- 172.21.15.69:0/2204232539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7015f4b0 msgr2=0x7fec7009a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:10.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.932+0000 7fec76c00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7015f4b0 0x7fec7009a890 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fec6802ac90 tx=0x7fec68004a40 comp rx=0 tx=0).stop 2024-04-24T08:18:10.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.933+0000 7fec76c00640 1 -- 172.21.15.69:0/2204232539 shutdown_connections 2024-04-24T08:18:10.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.933+0000 7fec76c00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fec5006c8f0 0x7fec5006edb0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.933+0000 7fec76c00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fec7015f4b0 0x7fec7009a890 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.933+0000 7fec76c00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fec700b46a0 0x7fec7009a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.933+0000 7fec76c00640 1 --2- 172.21.15.69:0/2204232539 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fec700b3de0 0x7fec701495e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:10.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.933+0000 7fec76c00640 1 -- 172.21.15.69:0/2204232539 >> 172.21.15.69:0/2204232539 conn(0x7fec700d0350 msgr2=0x7fec700732a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:10.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.934+0000 7fec76c00640 1 -- 172.21.15.69:0/2204232539 shutdown_connections 2024-04-24T08:18:10.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:10.935+0000 7fec76c00640 1 -- 172.21.15.69:0/2204232539 wait complete. 2024-04-24T08:18:10.979 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:11.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.141+0000 7f8500400640 1 Processor -- start 2024-04-24T08:18:11.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.141+0000 7f8500400640 1 -- start start 2024-04-24T08:18:11.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.142+0000 7f8500400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f84f815ea10 0x7f84f80d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.142+0000 7f8500400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f84f81631a0 0x7f84f80af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.142+0000 7f8500400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84f80adab0 con 0x7f84f80af7d0 2024-04-24T08:18:11.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.142+0000 7f8500400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f84f80afbb0 con 0x7f84f815ea10 2024-04-24T08:18:11.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.142+0000 7f8500400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f84f80add50 con 0x7f84f81631a0 2024-04-24T08:18:11.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.143+0000 7f84fda00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f84f815ea10 0x7f84f80d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.143+0000 7f84fe400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f84f80af7d0 0x7f84f8071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51964/0 (socket says 172.21.15.69:51964) 2024-04-24T08:18:11.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.143+0000 7f84fda00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f84f815ea10 0x7f84f80d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40250/0 (socket says 172.21.15.69:40250) 2024-04-24T08:18:11.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.143+0000 7f84fe400640 1 -- 172.21.15.69:0/673605295 learned_addr learned my addr 172.21.15.69:0/673605295 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:11.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.143+0000 7f84fda00640 1 -- 172.21.15.69:0/673605295 >> v2:172.21.15.121:3300/0 conn(0x7f84f81631a0 msgr2=0x7f84f80af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:11.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.143+0000 7f84fda00640 1 --2- 172.21.15.69:0/673605295 >> v2:172.21.15.121:3300/0 conn(0x7f84f81631a0 0x7f84f80af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.143+0000 7f84fda00640 1 -- 172.21.15.69:0/673605295 >> v1:172.21.15.69:6789/0 conn(0x7f84f80af7d0 legacy=0x7f84f8071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.143+0000 7f84f7e00640 1 -- 172.21.15.69:0/673605295 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1453101305 0 0) 0x7f84f80adab0 con 0x7f84f80af7d0 2024-04-24T08:18:11.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.143+0000 7f84fda00640 1 -- 172.21.15.69:0/673605295 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f80ae030 con 0x7f84f815ea10 2024-04-24T08:18:11.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.144+0000 7f84fda00640 1 --2- 172.21.15.69:0/673605295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f815ea10 0x7f84f80d0830 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f84f806f5d0 tx=0x7f84ec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed3603c835903544 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:11.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.144+0000 7f84f7e00640 1 -- 172.21.15.69:0/673605295 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec037070 con 0x7f84f815ea10 2024-04-24T08:18:11.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.144+0000 7f84f7e00640 1 -- 172.21.15.69:0/673605295 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84ec032440 con 0x7f84f815ea10 2024-04-24T08:18:11.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.144+0000 7f84f7e00640 1 -- 172.21.15.69:0/673605295 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec032810 con 0x7f84f815ea10 2024-04-24T08:18:11.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.145+0000 7f8500400640 1 -- 172.21.15.69:0/673605295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f815ea10 msgr2=0x7f84f80d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.145+0000 7f8500400640 1 --2- 172.21.15.69:0/673605295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f815ea10 0x7f84f80d0830 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f84f806f5d0 tx=0x7f84ec02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.145+0000 7f8500400640 1 -- 172.21.15.69:0/673605295 shutdown_connections 2024-04-24T08:18:11.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.145+0000 7f8500400640 1 --2- 172.21.15.69:0/673605295 >> v2:172.21.15.121:3300/0 conn(0x7f84f81631a0 0x7f84f80af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.145+0000 7f8500400640 1 --2- 172.21.15.69:0/673605295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f815ea10 0x7f84f80d0830 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.145+0000 7f8500400640 1 -- 172.21.15.69:0/673605295 >> 172.21.15.69:0/673605295 conn(0x7f84f8067d90 msgr2=0x7f84f806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:11.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.145+0000 7f8500400640 1 -- 172.21.15.69:0/673605295 shutdown_connections 2024-04-24T08:18:11.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.145+0000 7f8500400640 1 -- 172.21.15.69:0/673605295 wait complete. 2024-04-24T08:18:11.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.147+0000 7f8500400640 1 Processor -- start 2024-04-24T08:18:11.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.147+0000 7f8500400640 1 -- start start 2024-04-24T08:18:11.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f8500400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f80af7d0 0x7f84f8156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f8500400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f815ea10 0x7f84f813e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f8500400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f81631a0 0x7f84f8153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f8500400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84f80ae8d0 con 0x7f84f81631a0 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f8500400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f84f80ae5f0 con 0x7f84f815ea10 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f8500400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f84f80aedf0 con 0x7f84f80af7d0 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f84fe400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f81631a0 0x7f84f8153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f84fe400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f81631a0 0x7f84f8153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40256/0 (socket says 172.21.15.69:40256) 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f84fe400640 1 -- 172.21.15.69:0/3064724331 learned_addr learned my addr 172.21.15.69:0/3064724331 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f84fd000640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f815ea10 0x7f84f813e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f84fe400640 1 -- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f80af7d0 msgr2=0x7f84f8156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f84fe400640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f80af7d0 0x7f84f8156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f84fe400640 1 -- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f815ea10 msgr2=0x7f84f813e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f84fe400640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f815ea10 0x7f84f813e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.148+0000 7f84fe400640 1 -- 172.21.15.69:0/3064724331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f81554a0 con 0x7f84f81631a0 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.149+0000 7f84fe400640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f81631a0 0x7f84f8153700 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f84f000cb30 tx=0x7f84f000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.149+0000 7f84f5600640 1 -- 172.21.15.69:0/3064724331 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0017070 con 0x7f84f81631a0 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.149+0000 7f84f5600640 1 -- 172.21.15.69:0/3064724331 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f000acc0 con 0x7f84f81631a0 2024-04-24T08:18:11.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.149+0000 7f84f5600640 1 -- 172.21.15.69:0/3064724331 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0012640 con 0x7f84f81631a0 2024-04-24T08:18:11.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.149+0000 7f8500400640 1 -- 172.21.15.69:0/3064724331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f8155780 con 0x7f84f81631a0 2024-04-24T08:18:11.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.149+0000 7f8500400640 1 -- 172.21.15.69:0/3064724331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f8155d40 con 0x7f84f81631a0 2024-04-24T08:18:11.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.150+0000 7f8500400640 1 -- 172.21.15.69:0/3064724331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84f8068a80 con 0x7f84f81631a0 2024-04-24T08:18:11.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.151+0000 7f84f5600640 1 -- 172.21.15.69:0/3064724331 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f84f001a070 con 0x7f84f81631a0 2024-04-24T08:18:11.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.152+0000 7f84f5600640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f84dc06c8f0 0x7f84dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.152+0000 7f84f5600640 1 -- 172.21.15.69:0/3064724331 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f84f008b990 con 0x7f84f81631a0 2024-04-24T08:18:11.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.153+0000 7f84fda00640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f84dc06c8f0 0x7f84dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.154+0000 7f84fda00640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f84dc06c8f0 0x7f84dc06edb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f84ec034000 tx=0x7f84ec02c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:11.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.157+0000 7f84f5600640 1 -- 172.21.15.69:0/3064724331 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f84f0059dc0 con 0x7f84f81631a0 2024-04-24T08:18:11.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.348+0000 7f8500400640 1 -- 172.21.15.69:0/3064724331 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84f806ddd0 con 0x7f84f81631a0 2024-04-24T08:18:11.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.351+0000 7f84f5600640 1 -- 172.21.15.69:0/3064724331 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+16597 (secure 0 0 0) 0x7f84f00596c0 con 0x7f84f81631a0 2024-04-24T08:18:11.352 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:11.352 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":96,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:18:10.396219+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:08.156864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:09.368519+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:11.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 -- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f84dc06c8f0 msgr2=0x7f84dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f84dc06c8f0 0x7f84dc06edb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f84ec034000 tx=0x7f84ec02c000 comp rx=0 tx=0).stop 2024-04-24T08:18:11.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 -- 172.21.15.69:0/3064724331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f81631a0 msgr2=0x7f84f8153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f81631a0 0x7f84f8153700 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f84f000cb30 tx=0x7f84f000a060 comp rx=0 tx=0).stop 2024-04-24T08:18:11.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 -- 172.21.15.69:0/3064724331 shutdown_connections 2024-04-24T08:18:11.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f84dc06c8f0 0x7f84dc06edb0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f81631a0 0x7f84f8153700 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84f815ea10 0x7f84f813e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 --2- 172.21.15.69:0/3064724331 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f84f80af7d0 0x7f84f8156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 -- 172.21.15.69:0/3064724331 >> 172.21.15.69:0/3064724331 conn(0x7f84f8067d90 msgr2=0x7f84f8160e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:11.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.353+0000 7f8500400640 1 -- 172.21.15.69:0/3064724331 shutdown_connections 2024-04-24T08:18:11.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.354+0000 7f8500400640 1 -- 172.21.15.69:0/3064724331 wait complete. 2024-04-24T08:18:11.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:11.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.554+0000 7f145be00640 1 Processor -- start 2024-04-24T08:18:11.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.555+0000 7f145be00640 1 -- start start 2024-04-24T08:18:11.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.556+0000 7f145be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f145c15ab10 0x7f145c144090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.556+0000 7f145be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f145c071be0 0x7f145c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.556+0000 7f145be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f145c0b3190 con 0x7f145c06e960 2024-04-24T08:18:11.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.556+0000 7f145be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f145c0ad590 con 0x7f145c15ab10 2024-04-24T08:18:11.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.556+0000 7f145be00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f145c0b3400 con 0x7f145c071be0 2024-04-24T08:18:11.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.556+0000 7f145aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f145c15ab10 0x7f145c144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.556+0000 7f145b400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f145c06e960 0x7f145c15e6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51970/0 (socket says 172.21.15.69:51970) 2024-04-24T08:18:11.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.556+0000 7f145b400640 1 -- 172.21.15.69:0/3496481545 learned_addr learned my addr 172.21.15.69:0/3496481545 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:11.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.557+0000 7f1452000640 1 --2- 172.21.15.69:0/3496481545 >> v2:172.21.15.121:3300/0 conn(0x7f145c071be0 0x7f145c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.557+0000 7f145a000640 1 -- 172.21.15.69:0/3496481545 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2302184726 0 0) 0x7f145c0b3190 con 0x7f145c06e960 2024-04-24T08:18:11.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.557+0000 7f145a000640 1 -- 172.21.15.69:0/3496481545 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1440004630 con 0x7f145c06e960 2024-04-24T08:18:11.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.557+0000 7f145aa00640 1 -- 172.21.15.69:0/3496481545 >> v2:172.21.15.121:3300/0 conn(0x7f145c071be0 msgr2=0x7f145c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:11.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.557+0000 7f145aa00640 1 --2- 172.21.15.69:0/3496481545 >> v2:172.21.15.121:3300/0 conn(0x7f145c071be0 0x7f145c06e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.557+0000 7f145aa00640 1 -- 172.21.15.69:0/3496481545 >> v1:172.21.15.69:6789/0 conn(0x7f145c06e960 legacy=0x7f145c15e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.557+0000 7f145aa00640 1 -- 172.21.15.69:0/3496481545 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145c0b36e0 con 0x7f145c15ab10 2024-04-24T08:18:11.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.558+0000 7f145aa00640 1 --2- 172.21.15.69:0/3496481545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 0x7f145c144090 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f145c06f6b0 tx=0x7f144c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774c1fa67038b36 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.558+0000 7f145a000640 1 -- 172.21.15.69:0/3496481545 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c037070 con 0x7f145c15ab10 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.558+0000 7f145a000640 1 -- 172.21.15.69:0/3496481545 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144c032440 con 0x7f145c15ab10 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.558+0000 7f145a000640 1 -- 172.21.15.69:0/3496481545 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c032810 con 0x7f145c15ab10 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.559+0000 7f145be00640 1 -- 172.21.15.69:0/3496481545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 msgr2=0x7f145c144090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.559+0000 7f145be00640 1 --2- 172.21.15.69:0/3496481545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 0x7f145c144090 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f145c06f6b0 tx=0x7f144c02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.559+0000 7f145be00640 1 -- 172.21.15.69:0/3496481545 shutdown_connections 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.559+0000 7f145be00640 1 --2- 172.21.15.69:0/3496481545 >> v2:172.21.15.121:3300/0 conn(0x7f145c071be0 0x7f145c06e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.559+0000 7f145be00640 1 --2- 172.21.15.69:0/3496481545 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 0x7f145c144090 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.559+0000 7f145be00640 1 -- 172.21.15.69:0/3496481545 >> 172.21.15.69:0/3496481545 conn(0x7f145c0d0160 msgr2=0x7f145c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.559+0000 7f145be00640 1 -- 172.21.15.69:0/3496481545 shutdown_connections 2024-04-24T08:18:11.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.559+0000 7f145be00640 1 -- 172.21.15.69:0/3496481545 wait complete. 2024-04-24T08:18:11.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.560+0000 7f145be00640 1 Processor -- start 2024-04-24T08:18:11.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.561+0000 7f145be00640 1 -- start start 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.561+0000 7f145be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f145c06e960 0x7f145c143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.561+0000 7f145be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f145c071be0 0x7f145c157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 0x7f145c1578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f145c15aef0 con 0x7f145c15ab10 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f145c072150 con 0x7f145c071be0 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f145c06f950 con 0x7f145c06e960 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 0x7f145c1578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 0x7f145c1578d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40274/0 (socket says 172.21.15.69:40274) 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145b400640 1 -- 172.21.15.69:0/2296680754 learned_addr learned my addr 172.21.15.69:0/2296680754 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f1452000640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f145c071be0 0x7f145c157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145b400640 1 -- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f145c06e960 msgr2=0x7f145c143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145b400640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f145c06e960 0x7f145c143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145b400640 1 -- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f145c071be0 msgr2=0x7f145c157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145b400640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f145c071be0 0x7f145c157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.562+0000 7f145b400640 1 -- 172.21.15.69:0/2296680754 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145c097540 con 0x7f145c15ab10 2024-04-24T08:18:11.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.563+0000 7f145b400640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 0x7f145c1578d0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f145400c390 tx=0x7f145400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:11.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.563+0000 7f1453400640 1 -- 172.21.15.69:0/2296680754 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454017070 con 0x7f145c15ab10 2024-04-24T08:18:11.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.563+0000 7f1453400640 1 -- 172.21.15.69:0/2296680754 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1454005ce0 con 0x7f145c15ab10 2024-04-24T08:18:11.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.563+0000 7f145be00640 1 -- 172.21.15.69:0/2296680754 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f145c097820 con 0x7f145c15ab10 2024-04-24T08:18:11.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.563+0000 7f1453400640 1 -- 172.21.15.69:0/2296680754 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1454007310 con 0x7f145c15ab10 2024-04-24T08:18:11.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.563+0000 7f145be00640 1 -- 172.21.15.69:0/2296680754 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f145c097de0 con 0x7f145c15ab10 2024-04-24T08:18:11.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.564+0000 7f145be00640 1 -- 172.21.15.69:0/2296680754 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f145c150ed0 con 0x7f145c15ab10 2024-04-24T08:18:11.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.565+0000 7f1453400640 1 -- 172.21.15.69:0/2296680754 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f145401a070 con 0x7f145c15ab10 2024-04-24T08:18:11.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.566+0000 7f1453400640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f143806c8f0 0x7f143806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.566+0000 7f1453400640 1 -- 172.21.15.69:0/2296680754 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 5709+0+0 (secure 0 0 0) 0x7f145408af00 con 0x7f145c15ab10 2024-04-24T08:18:11.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.567+0000 7f145aa00640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f143806c8f0 0x7f143806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.568+0000 7f145aa00640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f143806c8f0 0x7f143806edb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f144c034000 tx=0x7f144c02c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:11.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.570+0000 7f1453400640 1 -- 172.21.15.69:0/2296680754 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1454059330 con 0x7f145c15ab10 2024-04-24T08:18:11.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.770+0000 7f145be00640 1 -- 172.21.15.69:0/2296680754 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f145c06f020 con 0x7f145c15ab10 2024-04-24T08:18:11.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.773+0000 7f1453400640 1 -- 172.21.15.69:0/2296680754 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) ==== 74+0+16597 (secure 0 0 0) 0x7f1454058c30 con 0x7f145c15ab10 2024-04-24T08:18:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:11.774 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":96,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:18:10.396219+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:08.156864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:09.368519+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 -- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f143806c8f0 msgr2=0x7f143806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f143806c8f0 0x7f143806edb0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f144c034000 tx=0x7f144c02c000 comp rx=0 tx=0).stop 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 -- 172.21.15.69:0/2296680754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 msgr2=0x7f145c1578d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 0x7f145c1578d0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f145400c390 tx=0x7f145400c850 comp rx=0 tx=0).stop 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 -- 172.21.15.69:0/2296680754 shutdown_connections 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f143806c8f0 0x7f143806edb0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f145c15ab10 0x7f145c1578d0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f145c071be0 0x7f145c157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 --2- 172.21.15.69:0/2296680754 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f145c06e960 0x7f145c143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 -- 172.21.15.69:0/2296680754 >> 172.21.15.69:0/2296680754 conn(0x7f145c0d0160 msgr2=0x7f145c0997a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:11.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.775+0000 7f145be00640 1 -- 172.21.15.69:0/2296680754 shutdown_connections 2024-04-24T08:18:11.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.776+0000 7f145be00640 1 -- 172.21.15.69:0/2296680754 wait complete. 2024-04-24T08:18:11.795 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T08:18:11.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.982+0000 7f0d09600640 1 Processor -- start 2024-04-24T08:18:11.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.982+0000 7f0d09600640 1 -- start start 2024-04-24T08:18:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d09600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d0415c820 0x7f0d040b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d09600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0d040b46a0 0x7f0d0406f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d09600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d0406fd10 0x7f0d040ce250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d09600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0d0415cc00 con 0x7f0d0406fd10 2024-04-24T08:18:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d09600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0d0406ed30 con 0x7f0d040b46a0 2024-04-24T08:18:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d09600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0d040b4e10 con 0x7f0d0415c820 2024-04-24T08:18:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d03400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d0406fd10 0x7f0d040ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d03400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0d0406fd10 0x7f0d040ce250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40288/0 (socket says 172.21.15.69:40288) 2024-04-24T08:18:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d03400640 1 -- 172.21.15.69:0/1539139537 learned_addr learned my addr 172.21.15.69:0/1539139537 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d03400640 1 -- 172.21.15.69:0/1539139537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d0415c820 msgr2=0x7f0d040b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d03400640 1 --2- 172.21.15.69:0/1539139537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d0415c820 0x7f0d040b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d03400640 1 -- 172.21.15.69:0/1539139537 >> v2:172.21.15.121:3300/0 conn(0x7f0d040b46a0 msgr2=0x7f0d0406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d03400640 1 --2- 172.21.15.69:0/1539139537 >> v2:172.21.15.121:3300/0 conn(0x7f0d040b46a0 0x7f0d0406f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.984+0000 7f0d03400640 1 -- 172.21.15.69:0/1539139537 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d040b50f0 con 0x7f0d0406fd10 2024-04-24T08:18:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.985+0000 7f0d03400640 1 --2- 172.21.15.69:0/1539139537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d0406fd10 0x7f0d040ce250 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f0d040b2fe0 tx=0x7f0cfc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbefa9ec773df1d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:11.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.986+0000 7f0d01600640 1 -- 172.21.15.69:0/1539139537 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc036070 con 0x7f0d0406fd10 2024-04-24T08:18:11.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.986+0000 7f0d01600640 1 -- 172.21.15.69:0/1539139537 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cfc031600 con 0x7f0d0406fd10 2024-04-24T08:18:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.986+0000 7f0d01600640 1 -- 172.21.15.69:0/1539139537 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cfc0319d0 con 0x7f0d0406fd10 2024-04-24T08:18:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.986+0000 7f0d09600640 1 -- 172.21.15.69:0/1539139537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d0406fd10 msgr2=0x7f0d040ce250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.986+0000 7f0d09600640 1 --2- 172.21.15.69:0/1539139537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d0406fd10 0x7f0d040ce250 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f0d040b2fe0 tx=0x7f0cfc02a6a0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.986+0000 7f0d09600640 1 -- 172.21.15.69:0/1539139537 shutdown_connections 2024-04-24T08:18:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.986+0000 7f0d09600640 1 --2- 172.21.15.69:0/1539139537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d0406fd10 0x7f0d040ce250 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.986+0000 7f0d09600640 1 --2- 172.21.15.69:0/1539139537 >> v2:172.21.15.121:3300/0 conn(0x7f0d040b46a0 0x7f0d0406f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.986+0000 7f0d09600640 1 --2- 172.21.15.69:0/1539139537 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d0415c820 0x7f0d040b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.987+0000 7f0d09600640 1 -- 172.21.15.69:0/1539139537 >> 172.21.15.69:0/1539139537 conn(0x7f0d040d0350 msgr2=0x7f0d040d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.987+0000 7f0d09600640 1 -- 172.21.15.69:0/1539139537 shutdown_connections 2024-04-24T08:18:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.987+0000 7f0d09600640 1 -- 172.21.15.69:0/1539139537 wait complete. 2024-04-24T08:18:11.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.988+0000 7f0d09600640 1 Processor -- start 2024-04-24T08:18:11.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.988+0000 7f0d09600640 1 -- start start 2024-04-24T08:18:11.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.989+0000 7f0d09600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d0406fd10 0x7f0d041551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.989+0000 7f0d09600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d040b46a0 0x7f0d041f0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.989+0000 7f0d09600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d0415c820 0x7f0d0415d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.989+0000 7f0d09600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0d040b4b00 con 0x7f0d040b46a0 2024-04-24T08:18:11.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.989+0000 7f0d09600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0d0406f2b0 con 0x7f0d0406fd10 2024-04-24T08:18:11.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.989+0000 7f0d09600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0d040ce790 con 0x7f0d0415c820 2024-04-24T08:18:11.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.989+0000 7f0d02000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d040b46a0 0x7f0d041f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.989+0000 7f0d02000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d040b46a0 0x7f0d041f0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40290/0 (socket says 172.21.15.69:40290) 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.989+0000 7f0d02000640 1 -- 172.21.15.69:0/1054808103 learned_addr learned my addr 172.21.15.69:0/1054808103 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.990+0000 7f0d02a00640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d0406fd10 0x7f0d041551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.990+0000 7f0d03400640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d0415c820 0x7f0d0415d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.990+0000 7f0d02000640 1 -- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d0415c820 msgr2=0x7f0d0415d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.990+0000 7f0d02000640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d0415c820 0x7f0d0415d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.990+0000 7f0d02000640 1 -- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d0406fd10 msgr2=0x7f0d041551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.990+0000 7f0d02000640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d0406fd10 0x7f0d041551c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.990+0000 7f0d02000640 1 -- 172.21.15.69:0/1054808103 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d040ad620 con 0x7f0d040b46a0 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.990+0000 7f0d03400640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d0415c820 0x7f0d0415d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.990+0000 7f0d02000640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d040b46a0 0x7f0d041f0410 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000b4d0 tx=0x7f0cf000b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.990+0000 7f0cfaa00640 1 -- 172.21.15.69:0/1054808103 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0014070 con 0x7f0d040b46a0 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.991+0000 7f0cfaa00640 1 -- 172.21.15.69:0/1054808103 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cf0010870 con 0x7f0d040b46a0 2024-04-24T08:18:11.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.991+0000 7f0d09600640 1 -- 172.21.15.69:0/1054808103 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d040ad900 con 0x7f0d040b46a0 2024-04-24T08:18:11.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.991+0000 7f0cfaa00640 1 -- 172.21.15.69:0/1054808103 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0010c60 con 0x7f0d040b46a0 2024-04-24T08:18:11.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.991+0000 7f0d09600640 1 -- 172.21.15.69:0/1054808103 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d040adec0 con 0x7f0d040b46a0 2024-04-24T08:18:11.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.992+0000 7f0d09600640 1 -- 172.21.15.69:0/1054808103 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d04068a80 con 0x7f0d040b46a0 2024-04-24T08:18:11.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.993+0000 7f0cfaa00640 1 -- 172.21.15.69:0/1054808103 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0cf0017070 con 0x7f0d040b46a0 2024-04-24T08:18:11.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.994+0000 7f0cfaa00640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ce406c8f0 0x7f0ce406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:11.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.994+0000 7f0cfaa00640 1 -- 172.21.15.69:0/1054808103 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f0cf008b990 con 0x7f0d040b46a0 2024-04-24T08:18:11.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.995+0000 7f0d02a00640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ce406c8f0 0x7f0ce406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:11.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.996+0000 7f0d02a00640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ce406c8f0 0x7f0ce406edb0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0d04160d90 tx=0x7f0cf4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:11.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:11.998+0000 7f0cfaa00640 1 -- 172.21.15.69:0/1054808103 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0cf0059db0 con 0x7f0d040b46a0 2024-04-24T08:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.219+0000 7f0d09600640 1 -- 172.21.15.69:0/1054808103 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0d04155700 con 0x7f0d040b46a0 2024-04-24T08:18:12.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.987+0000 7f0cfaa00640 1 -- 172.21.15.69:0/1054808103 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v82) ==== 118+0+0 (secure 0 0 0) 0x7f0cf00596b0 con 0x7f0d040b46a0 2024-04-24T08:18:12.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.991+0000 7f0d09600640 1 -- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ce406c8f0 msgr2=0x7f0ce406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:12.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.991+0000 7f0d09600640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ce406c8f0 0x7f0ce406edb0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f0d04160d90 tx=0x7f0cf4008040 comp rx=0 tx=0).stop 2024-04-24T08:18:12.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.991+0000 7f0d09600640 1 -- 172.21.15.69:0/1054808103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d040b46a0 msgr2=0x7f0d041f0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:12.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.991+0000 7f0d09600640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d040b46a0 0x7f0d041f0410 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000b4d0 tx=0x7f0cf000b990 comp rx=0 tx=0).stop 2024-04-24T08:18:12.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.991+0000 7f0d09600640 1 -- 172.21.15.69:0/1054808103 shutdown_connections 2024-04-24T08:18:12.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.991+0000 7f0d09600640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0ce406c8f0 0x7f0ce406edb0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:12.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.991+0000 7f0d09600640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0d040b46a0 0x7f0d041f0410 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:12.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.991+0000 7f0d09600640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0d0415c820 0x7f0d0415d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:12.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.991+0000 7f0d09600640 1 --2- 172.21.15.69:0/1054808103 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0d0406fd10 0x7f0d041551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:12.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.991+0000 7f0d09600640 1 -- 172.21.15.69:0/1054808103 >> 172.21.15.69:0/1054808103 conn(0x7f0d040d0350 msgr2=0x7f0d040afdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:12.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.992+0000 7f0d09600640 1 -- 172.21.15.69:0/1054808103 shutdown_connections 2024-04-24T08:18:12.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:12.992+0000 7f0d09600640 1 -- 172.21.15.69:0/1054808103 wait complete. 2024-04-24T08:18:13.012 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-24T08:18:13.013 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T08:18:13.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.220+0000 7f6e7d200640 1 Processor -- start 2024-04-24T08:18:13.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.220+0000 7f6e7d200640 1 -- start start 2024-04-24T08:18:13.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.222+0000 7f6e7d200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6e7815f4b0 0x7f6e780b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:13.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.222+0000 7f6e7d200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b46a0 0x7f6e780b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:13.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.222+0000 7f6e7d200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e780739f0 con 0x7f6e7815f890 2024-04-24T08:18:13.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.222+0000 7f6e7d200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6e780b5010 con 0x7f6e7815f4b0 2024-04-24T08:18:13.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.222+0000 7f6e7d200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6e78073cc0 con 0x7f6e780b46a0 2024-04-24T08:18:13.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.222+0000 7f6e75400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f6e7815f890 0x7f6e780732e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51986/0 (socket says 172.21.15.69:51986) 2024-04-24T08:18:13.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.222+0000 7f6e75400640 1 -- 172.21.15.69:0/1308081551 learned_addr learned my addr 172.21.15.69:0/1308081551 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e76800640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b46a0 0x7f6e780b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.222+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3125992593 0 0) 0x7f6e780739f0 con 0x7f6e7815f890 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e75e00640 1 --2- 172.21.15.69:0/1308081551 >> v2:172.21.15.121:3300/0 conn(0x7f6e7815f4b0 0x7f6e780b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e5c003520 con 0x7f6e7815f890 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1639130209 0 0) 0x7f6e5c003520 con 0x7f6e7815f890 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b46a0 msgr2=0x7f6e780b4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b46a0 0x7f6e780b4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 >> v2:172.21.15.121:3300/0 conn(0x7f6e7815f4b0 msgr2=0x7f6e780b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 --2- 172.21.15.69:0/1308081551 >> v2:172.21.15.121:3300/0 conn(0x7f6e7815f4b0 0x7f6e780b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e78073fa0 con 0x7f6e7815f890 2024-04-24T08:18:13.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f6e64001690 con 0x7f6e7815f890 2024-04-24T08:18:13.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 >> v1:172.21.15.69:6789/0 conn(0x7f6e7815f890 legacy=0x7f6e780732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:13.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b46a0 0x7f6e5c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:13.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e5c003f80 0x7f6e5c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:13.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e5c004d40 0x7f6e5c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:13.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e780b5010 con 0x7f6e5c003f80 2024-04-24T08:18:13.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6e78073cc0 con 0x7f6e5c004d40 2024-04-24T08:18:13.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6e5c00b9c0 con 0x7f6e780b46a0 2024-04-24T08:18:13.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.223+0000 7f6e75400640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e5c003f80 0x7f6e5c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:13.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.224+0000 7f6e75e00640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b46a0 0x7f6e5c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:13.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.224+0000 7f6e75400640 1 -- 172.21.15.69:0/1308081551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b46a0 msgr2=0x7f6e5c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:13.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.224+0000 7f6e75400640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b46a0 0x7f6e5c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.224+0000 7f6e75400640 1 -- 172.21.15.69:0/1308081551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e5c004d40 msgr2=0x7f6e5c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:13.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.224+0000 7f6e75400640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e5c004d40 0x7f6e5c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.224+0000 7f6e75400640 1 -- 172.21.15.69:0/1308081551 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5c003520 con 0x7f6e5c003f80 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.225+0000 7f6e75400640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e5c003f80 0x7f6e5c004800 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f6e64007fd0 tx=0x7f6e5c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bac3c38029b6468 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.225+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64007ca0 con 0x7f6e5c003f80 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.225+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e64005af0 con 0x7f6e5c003f80 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.225+0000 7f6e74a00640 1 -- 172.21.15.69:0/1308081551 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64007070 con 0x7f6e5c003f80 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.226+0000 7f6e7d200640 1 -- 172.21.15.69:0/1308081551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e5c003f80 msgr2=0x7f6e5c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.226+0000 7f6e7d200640 1 --2- 172.21.15.69:0/1308081551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e5c003f80 0x7f6e5c004800 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f6e64007fd0 tx=0x7f6e5c005830 comp rx=0 tx=0).stop 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.226+0000 7f6e76800640 1 -- 172.21.15.69:0/1308081551 reap_dead start 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.226+0000 7f6e7d200640 1 -- 172.21.15.69:0/1308081551 shutdown_connections 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.226+0000 7f6e7d200640 1 -- 172.21.15.69:0/1308081551 >> 172.21.15.69:0/1308081551 conn(0x7f6e780d0350 msgr2=0x7f6e780d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.226+0000 7f6e7d200640 1 -- 172.21.15.69:0/1308081551 shutdown_connections 2024-04-24T08:18:13.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.226+0000 7f6e7d200640 1 -- 172.21.15.69:0/1308081551 wait complete. 2024-04-24T08:18:13.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.228+0000 7f6e7d200640 1 Processor -- start 2024-04-24T08:18:13.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.228+0000 7f6e7d200640 1 -- start start 2024-04-24T08:18:13.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.229+0000 7f6e7d200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e5c003f80 0x7f6e781456b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:13.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.229+0000 7f6e7d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e781e8300 0x7f6e781e8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:13.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.229+0000 7f6e7d200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b07f0 0x7f6e780b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:13.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.229+0000 7f6e7d200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e780b38c0 con 0x7f6e781e8300 2024-04-24T08:18:13.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.229+0000 7f6e7d200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6e780b35e0 con 0x7f6e5c003f80 2024-04-24T08:18:13.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.229+0000 7f6e7d200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6e7815fd40 con 0x7f6e780b07f0 2024-04-24T08:18:13.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.229+0000 7f6e75400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e781e8300 0x7f6e781e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:13.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.229+0000 7f6e75400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e781e8300 0x7f6e781e8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40304/0 (socket says 172.21.15.69:40304) 2024-04-24T08:18:13.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.229+0000 7f6e75400640 1 -- 172.21.15.69:0/1146579801 learned_addr learned my addr 172.21.15.69:0/1146579801 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e76800640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b07f0 0x7f6e780b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e75e00640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e5c003f80 0x7f6e781456b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e75400640 1 -- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b07f0 msgr2=0x7f6e780b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e75400640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b07f0 0x7f6e780b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e75400640 1 -- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e5c003f80 msgr2=0x7f6e781456b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e75400640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e5c003f80 0x7f6e781456b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e75400640 1 -- 172.21.15.69:0/1146579801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e781e8dc0 con 0x7f6e781e8300 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e75e00640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e5c003f80 0x7f6e781456b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e75400640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e781e8300 0x7f6e781e8780 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400c220 tx=0x7f6e7815eed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e6e000640 1 -- 172.21.15.69:0/1146579801 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e64037040 con 0x7f6e781e8300 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e6e000640 1 -- 172.21.15.69:0/1146579801 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e64001450 con 0x7f6e781e8300 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.230+0000 7f6e6e000640 1 -- 172.21.15.69:0/1146579801 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e640047e0 con 0x7f6e781e8300 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.231+0000 7f6e7d200640 1 -- 172.21.15.69:0/1146579801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e781e90a0 con 0x7f6e781e8300 2024-04-24T08:18:13.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.231+0000 7f6e7d200640 1 -- 172.21.15.69:0/1146579801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e781e9660 con 0x7f6e781e8300 2024-04-24T08:18:13.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.232+0000 7f6e7d200640 1 -- 172.21.15.69:0/1146579801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e78068a80 con 0x7f6e781e8300 2024-04-24T08:18:13.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.233+0000 7f6e6e000640 1 -- 172.21.15.69:0/1146579801 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6e64043020 con 0x7f6e781e8300 2024-04-24T08:18:13.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.234+0000 7f6e6e000640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6e5806c920 0x7f6e5806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:13.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.234+0000 7f6e6e000640 1 -- 172.21.15.69:0/1146579801 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f6e640aaec0 con 0x7f6e781e8300 2024-04-24T08:18:13.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.234+0000 7f6e75e00640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6e5806c920 0x7f6e5806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:13.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.236+0000 7f6e75e00640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6e5806c920 0x7f6e5806ede0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6e68009980 tx=0x7f6e68008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:13.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.239+0000 7f6e6e000640 1 -- 172.21.15.69:0/1146579801 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e6404a020 con 0x7f6e781e8300 2024-04-24T08:18:13.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.479+0000 7f6e7d200640 1 -- 172.21.15.69:0/1146579801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6e7806c760 con 0x7f6e781e8300 2024-04-24T08:18:13.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.990+0000 7f6e6e000640 1 -- 172.21.15.69:0/1146579801 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v83) ==== 122+0+0 (secure 0 0 0) 0x7f6e64079250 con 0x7f6e781e8300 2024-04-24T08:18:13.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 -- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6e5806c920 msgr2=0x7f6e5806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:13.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6e5806c920 0x7f6e5806ede0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6e68009980 tx=0x7f6e68008040 comp rx=0 tx=0).stop 2024-04-24T08:18:13.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 -- 172.21.15.69:0/1146579801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e781e8300 msgr2=0x7f6e781e8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:13.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e781e8300 0x7f6e781e8780 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400c220 tx=0x7f6e7815eed0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 -- 172.21.15.69:0/1146579801 shutdown_connections 2024-04-24T08:18:13.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6e5806c920 0x7f6e5806ede0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e781e8300 0x7f6e781e8780 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6e780b07f0 0x7f6e780b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 --2- 172.21.15.69:0/1146579801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e5c003f80 0x7f6e781456b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:13.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 -- 172.21.15.69:0/1146579801 >> 172.21.15.69:0/1146579801 conn(0x7f6e780d0350 msgr2=0x7f6e78160000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:13.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.995+0000 7f6e7d200640 1 -- 172.21.15.69:0/1146579801 shutdown_connections 2024-04-24T08:18:13.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:13.996+0000 7f6e7d200640 1 -- 172.21.15.69:0/1146579801 wait complete. 2024-04-24T08:18:14.016 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T08:18:14.016 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:14.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.216+0000 7f5d36600640 1 Processor -- start 2024-04-24T08:18:14.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.216+0000 7f5d36600640 1 -- start start 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.217+0000 7f5d36600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d3015e960 0x7f5d30162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.217+0000 7f5d36600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5d30162980 0x7f5d3006cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d36600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d3006d4f0 0x7f5d300cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d36600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5d3015ed40 con 0x7f5d3015e960 2024-04-24T08:18:14.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d36600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5d300cf050 con 0x7f5d30162980 2024-04-24T08:18:14.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d36600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5d30162e20 con 0x7f5d3006d4f0 2024-04-24T08:18:14.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d35200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d3015e960 0x7f5d30162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d35200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5d3015e960 0x7f5d30162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40312/0 (socket says 172.21.15.69:40312) 2024-04-24T08:18:14.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d35200640 1 -- 172.21.15.69:0/962046654 learned_addr learned my addr 172.21.15.69:0/962046654 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:14.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d2fe00640 1 --2- 172.21.15.69:0/962046654 >> v2:172.21.15.121:3300/0 conn(0x7f5d30162980 0x7f5d3006cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d35200640 1 -- 172.21.15.69:0/962046654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d3006d4f0 msgr2=0x7f5d300cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d35c00640 1 --2- 172.21.15.69:0/962046654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d3006d4f0 0x7f5d300cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d35200640 1 --2- 172.21.15.69:0/962046654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d3006d4f0 0x7f5d300cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d35200640 1 -- 172.21.15.69:0/962046654 >> v2:172.21.15.121:3300/0 conn(0x7f5d30162980 msgr2=0x7f5d3006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d35200640 1 --2- 172.21.15.69:0/962046654 >> v2:172.21.15.121:3300/0 conn(0x7f5d30162980 0x7f5d3006cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d35200640 1 -- 172.21.15.69:0/962046654 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d30163100 con 0x7f5d3015e960 2024-04-24T08:18:14.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.218+0000 7f5d2fe00640 1 --2- 172.21.15.69:0/962046654 >> v2:172.21.15.121:3300/0 conn(0x7f5d30162980 0x7f5d3006cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:14.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.219+0000 7f5d35200640 1 --2- 172.21.15.69:0/962046654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3015e960 0x7f5d30162440 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f5d30073c50 tx=0x7f5d2002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b26aa4ec2551c58 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.220+0000 7f5d2f400640 1 -- 172.21.15.69:0/962046654 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20037070 con 0x7f5d3015e960 2024-04-24T08:18:14.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.220+0000 7f5d2f400640 1 -- 172.21.15.69:0/962046654 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d20032440 con 0x7f5d3015e960 2024-04-24T08:18:14.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.220+0000 7f5d2f400640 1 -- 172.21.15.69:0/962046654 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20032810 con 0x7f5d3015e960 2024-04-24T08:18:14.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.220+0000 7f5d36600640 1 -- 172.21.15.69:0/962046654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3015e960 msgr2=0x7f5d30162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.220+0000 7f5d36600640 1 --2- 172.21.15.69:0/962046654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3015e960 0x7f5d30162440 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f5d30073c50 tx=0x7f5d2002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.220+0000 7f5d36600640 1 -- 172.21.15.69:0/962046654 shutdown_connections 2024-04-24T08:18:14.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.220+0000 7f5d36600640 1 --2- 172.21.15.69:0/962046654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d3006d4f0 0x7f5d300cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.220+0000 7f5d36600640 1 --2- 172.21.15.69:0/962046654 >> v2:172.21.15.121:3300/0 conn(0x7f5d30162980 0x7f5d3006cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.220+0000 7f5d36600640 1 --2- 172.21.15.69:0/962046654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3015e960 0x7f5d30162440 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.220+0000 7f5d36600640 1 -- 172.21.15.69:0/962046654 >> 172.21.15.69:0/962046654 conn(0x7f5d3009d900 msgr2=0x7f5d3008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.221+0000 7f5d36600640 1 -- 172.21.15.69:0/962046654 shutdown_connections 2024-04-24T08:18:14.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.221+0000 7f5d36600640 1 -- 172.21.15.69:0/962046654 wait complete. 2024-04-24T08:18:14.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.222+0000 7f5d36600640 1 Processor -- start 2024-04-24T08:18:14.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.222+0000 7f5d36600640 1 -- start start 2024-04-24T08:18:14.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.223+0000 7f5d36600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3006d4f0 0x7f5d300c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.223+0000 7f5d36600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d3015e960 0x7f5d301535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.223+0000 7f5d36600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d30162980 0x7f5d300b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.223+0000 7f5d36600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5d300b6170 con 0x7f5d3006d4f0 2024-04-24T08:18:14.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.223+0000 7f5d36600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5d300cf5d0 con 0x7f5d30162980 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.223+0000 7f5d36600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5d300b62f0 con 0x7f5d3015e960 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3006d4f0 0x7f5d300c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3006d4f0 0x7f5d300c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40314/0 (socket says 172.21.15.69:40314) 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35200640 1 -- 172.21.15.69:0/347625535 learned_addr learned my addr 172.21.15.69:0/347625535 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d2fe00640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d3015e960 0x7f5d301535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35c00640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d30162980 0x7f5d300b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35200640 1 -- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d3015e960 msgr2=0x7f5d301535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35200640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d3015e960 0x7f5d301535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35200640 1 -- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d30162980 msgr2=0x7f5d300b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35200640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d30162980 0x7f5d300b1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35200640 1 -- 172.21.15.69:0/347625535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d301f4d60 con 0x7f5d3006d4f0 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35c00640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d30162980 0x7f5d300b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.224+0000 7f5d35200640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3006d4f0 0x7f5d300c9c40 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f5d20000c00 tx=0x7f5d200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.225+0000 7f5d2cc00640 1 -- 172.21.15.69:0/347625535 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20037040 con 0x7f5d3006d4f0 2024-04-24T08:18:14.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.225+0000 7f5d2cc00640 1 -- 172.21.15.69:0/347625535 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d20042070 con 0x7f5d3006d4f0 2024-04-24T08:18:14.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.225+0000 7f5d2cc00640 1 -- 172.21.15.69:0/347625535 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20030bb0 con 0x7f5d3006d4f0 2024-04-24T08:18:14.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.225+0000 7f5d36600640 1 -- 172.21.15.69:0/347625535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d300ca2d0 con 0x7f5d3006d4f0 2024-04-24T08:18:14.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.225+0000 7f5d36600640 1 -- 172.21.15.69:0/347625535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d300ca890 con 0x7f5d3006d4f0 2024-04-24T08:18:14.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.226+0000 7f5d36600640 1 -- 172.21.15.69:0/347625535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d3006cf30 con 0x7f5d3006d4f0 2024-04-24T08:18:14.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.227+0000 7f5d2cc00640 1 -- 172.21.15.69:0/347625535 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d20047460 con 0x7f5d3006d4f0 2024-04-24T08:18:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.228+0000 7f5d2cc00640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0c06c8f0 0x7f5d0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.228+0000 7f5d2cc00640 1 -- 172.21.15.69:0/347625535 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f5d200ac730 con 0x7f5d3006d4f0 2024-04-24T08:18:14.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.228+0000 7f5d2fe00640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0c06c8f0 0x7f5d0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.230+0000 7f5d2fe00640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0c06c8f0 0x7f5d0c06edb0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5d3015c8a0 tx=0x7f5d18008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.232+0000 7f5d2cc00640 1 -- 172.21.15.69:0/347625535 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d301f4d60 con 0x7f5d3006d4f0 2024-04-24T08:18:14.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.474+0000 7f5d36600640 1 -- 172.21.15.69:0/347625535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d30075d50 con 0x7f5d3006d4f0 2024-04-24T08:18:14.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.476+0000 7f5d2cc00640 1 -- 172.21.15.69:0/347625535 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5186 (secure 0 0 0) 0x7f5d20045360 con 0x7f5d3006d4f0 2024-04-24T08:18:14.476 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:14.477 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:18:13.481156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6046":{"gid":6046,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/412016900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":412016900},{"type":"v1","addr":"172.21.15.69:6835","nonce":412016900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6046,"qdb_cluster":[6046]},"id":7}]} 2024-04-24T08:18:14.477 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 83 2024-04-24T08:18:14.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 -- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0c06c8f0 msgr2=0x7f5d0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0c06c8f0 0x7f5d0c06edb0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f5d3015c8a0 tx=0x7f5d18008040 comp rx=0 tx=0).stop 2024-04-24T08:18:14.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 -- 172.21.15.69:0/347625535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3006d4f0 msgr2=0x7f5d300c9c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3006d4f0 0x7f5d300c9c40 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f5d20000c00 tx=0x7f5d200046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 -- 172.21.15.69:0/347625535 shutdown_connections 2024-04-24T08:18:14.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5d0c06c8f0 0x7f5d0c06edb0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5d30162980 0x7f5d300b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5d3015e960 0x7f5d301535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 --2- 172.21.15.69:0/347625535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5d3006d4f0 0x7f5d300c9c40 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 -- 172.21.15.69:0/347625535 >> 172.21.15.69:0/347625535 conn(0x7f5d3009d900 msgr2=0x7f5d300d2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.478+0000 7f5d36600640 1 -- 172.21.15.69:0/347625535 shutdown_connections 2024-04-24T08:18:14.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.479+0000 7f5d36600640 1 -- 172.21.15.69:0/347625535 wait complete. 2024-04-24T08:18:14.498 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:14.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.702+0000 7f7adfe00640 1 Processor -- start 2024-04-24T08:18:14.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.702+0000 7f7adfe00640 1 -- start start 2024-04-24T08:18:14.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.704+0000 7f7adfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae0071e50 0x7f7ae015ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.704+0000 7f7adfe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7ae0072230 0x7f7ae01446a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.704+0000 7f7adfe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae015f540 con 0x7f7ae0162300 2024-04-24T08:18:14.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.704+0000 7f7adfe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7ae015f820 con 0x7f7ae0072230 2024-04-24T08:18:14.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.704+0000 7f7adfe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae015fb00 con 0x7f7ae0071e50 2024-04-24T08:18:14.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.704+0000 7f7ad6000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7ae0162300 0x7f7ae006e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:51992/0 (socket says 172.21.15.69:51992) 2024-04-24T08:18:14.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.704+0000 7f7ad6000640 1 -- 172.21.15.69:0/3529281216 learned_addr learned my addr 172.21.15.69:0/3529281216 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:14.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7adea00640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae0071e50 0x7f7ae015ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7adf400640 1 --2- 172.21.15.69:0/3529281216 >> v2:172.21.15.121:3300/0 conn(0x7f7ae0072230 0x7f7ae01446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3596598105 0 0) 0x7f7ae015f540 con 0x7f7ae0162300 2024-04-24T08:18:14.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ac4003520 con 0x7f7ae0162300 2024-04-24T08:18:14.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 561798431 0 0) 0x7f7ac4003520 con 0x7f7ae0162300 2024-04-24T08:18:14.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae0071e50 msgr2=0x7f7ae015ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae0071e50 0x7f7ae015ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 >> v2:172.21.15.121:3300/0 conn(0x7f7ae0072230 msgr2=0x7f7ae01446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 --2- 172.21.15.69:0/3529281216 >> v2:172.21.15.121:3300/0 conn(0x7f7ae0072230 0x7f7ae01446a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae006dda0 con 0x7f7ae0162300 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f7ac80019f0 con 0x7f7ae0162300 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7adea00640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae0071e50 0x7f7ae015ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 >> v1:172.21.15.69:6789/0 conn(0x7f7ae0162300 legacy=0x7f7ae006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ac40039b0 0x7f7ac4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 0x7f7ac4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae0071e50 0x7f7ac40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae015f820 con 0x7f7ac4004800 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae015fb00 con 0x7f7ac40039b0 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.705+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7ac400b9c0 con 0x7f7ae0071e50 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.706+0000 7f7ad6000640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 0x7f7ac4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.706+0000 7f7adea00640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ac40039b0 0x7f7ac4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.706+0000 7f7ad6000640 1 -- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae0071e50 msgr2=0x7f7ac40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.706+0000 7f7adf400640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae0071e50 0x7f7ac40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.706+0000 7f7ad6000640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae0071e50 0x7f7ac40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.706+0000 7f7ad6000640 1 -- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ac40039b0 msgr2=0x7f7ac4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.706+0000 7f7ad6000640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ac40039b0 0x7f7ac4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.706+0000 7f7ad6000640 1 -- 172.21.15.69:0/3529281216 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac4003520 con 0x7f7ac4004800 2024-04-24T08:18:14.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.706+0000 7f7adea00640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ac40039b0 0x7f7ac4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:14.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.707+0000 7f7ad6000640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 0x7f7ac4004c80 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800a950 tx=0x7f7ac4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ad13ef753f0f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.707+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac800a4c0 con 0x7f7ac4004800 2024-04-24T08:18:14.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.707+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ac802f070 con 0x7f7ac4004800 2024-04-24T08:18:14.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.707+0000 7f7ade000640 1 -- 172.21.15.69:0/3529281216 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac802bd00 con 0x7f7ac4004800 2024-04-24T08:18:14.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.708+0000 7f7adfe00640 1 -- 172.21.15.69:0/3529281216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 msgr2=0x7f7ac4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.708+0000 7f7adfe00640 1 --2- 172.21.15.69:0/3529281216 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 0x7f7ac4004c80 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800a950 tx=0x7f7ac4005830 comp rx=0 tx=0).stop 2024-04-24T08:18:14.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.708+0000 7f7adf400640 1 -- 172.21.15.69:0/3529281216 reap_dead start 2024-04-24T08:18:14.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.708+0000 7f7adfe00640 1 -- 172.21.15.69:0/3529281216 shutdown_connections 2024-04-24T08:18:14.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.708+0000 7f7adfe00640 1 -- 172.21.15.69:0/3529281216 >> 172.21.15.69:0/3529281216 conn(0x7f7ae00d0160 msgr2=0x7f7ae00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.708+0000 7f7adfe00640 1 -- 172.21.15.69:0/3529281216 shutdown_connections 2024-04-24T08:18:14.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.708+0000 7f7adfe00640 1 -- 172.21.15.69:0/3529281216 wait complete. 2024-04-24T08:18:14.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.710+0000 7f7adfe00640 1 Processor -- start 2024-04-24T08:18:14.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.710+0000 7f7adfe00640 1 -- start start 2024-04-24T08:18:14.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.711+0000 7f7adfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 0x7f7ae015a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.711+0000 7f7adfe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae00ad0d0 0x7f7ae00ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.711+0000 7f7adfe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae01436c0 0x7f7ae0143b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.711+0000 7f7adfe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae015ecc0 con 0x7f7ac4004800 2024-04-24T08:18:14.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.711+0000 7f7adfe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae0144e30 con 0x7f7ae01436c0 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.711+0000 7f7adfe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae015ee40 con 0x7f7ae00ad0d0 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.711+0000 7f7adea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 0x7f7ae015a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.711+0000 7f7adea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 0x7f7ae015a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40324/0 (socket says 172.21.15.69:40324) 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.711+0000 7f7adea00640 1 -- 172.21.15.69:0/521566879 learned_addr learned my addr 172.21.15.69:0/521566879 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.711+0000 7f7adf400640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae01436c0 0x7f7ae0143b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7adea00640 1 -- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae00ad0d0 msgr2=0x7f7ae00ad550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7ad6000640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae00ad0d0 0x7f7ae00ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7adea00640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae00ad0d0 0x7f7ae00ad550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7adea00640 1 -- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae01436c0 msgr2=0x7f7ae0143b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7adea00640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae01436c0 0x7f7ae0143b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7adea00640 1 -- 172.21.15.69:0/521566879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae0146520 con 0x7f7ac4004800 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7adf400640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae01436c0 0x7f7ae0143b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:14.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7adea00640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 0x7f7ae015a0c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000f650 tx=0x7f7ad000fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7ad7400640 1 -- 172.21.15.69:0/521566879 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0018070 con 0x7f7ac4004800 2024-04-24T08:18:14.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7ad7400640 1 -- 172.21.15.69:0/521566879 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ad00098e0 con 0x7f7ac4004800 2024-04-24T08:18:14.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7adfe00640 1 -- 172.21.15.69:0/521566879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae0161db0 con 0x7f7ac4004800 2024-04-24T08:18:14.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7ad7400640 1 -- 172.21.15.69:0/521566879 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0009cd0 con 0x7f7ac4004800 2024-04-24T08:18:14.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.712+0000 7f7adfe00640 1 -- 172.21.15.69:0/521566879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae00adf20 con 0x7f7ac4004800 2024-04-24T08:18:14.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.714+0000 7f7adfe00640 1 -- 172.21.15.69:0/521566879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ae0075d50 con 0x7f7ac4004800 2024-04-24T08:18:14.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.715+0000 7f7ad7400640 1 -- 172.21.15.69:0/521566879 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ad001b070 con 0x7f7ac4004800 2024-04-24T08:18:14.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.716+0000 7f7ad7400640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7abc06c920 0x7f7abc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.716+0000 7f7ad7400640 1 -- 172.21.15.69:0/521566879 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f7ad008bd90 con 0x7f7ac4004800 2024-04-24T08:18:14.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.716+0000 7f7ad6000640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7abc06c920 0x7f7abc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.717+0000 7f7ad6000640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7abc06c920 0x7f7abc06ede0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8033000 tx=0x7f7ac8007000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.720+0000 7f7ad7400640 1 -- 172.21.15.69:0/521566879 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ae0146520 con 0x7f7ac4004800 2024-04-24T08:18:14.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.906+0000 7f7adfe00640 1 -- 172.21.15.69:0/521566879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ae00d0d90 con 0x7f7ac4004800 2024-04-24T08:18:14.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.908+0000 7f7ad7400640 1 -- 172.21.15.69:0/521566879 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+16584 (secure 0 0 0) 0x7f7ad005a120 con 0x7f7ac4004800 2024-04-24T08:18:14.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:14.909 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":97,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:18:11.928428+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:08.156864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:09.368519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:14.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 -- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7abc06c920 msgr2=0x7f7abc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7abc06c920 0x7f7abc06ede0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8033000 tx=0x7f7ac8007000 comp rx=0 tx=0).stop 2024-04-24T08:18:14.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 -- 172.21.15.69:0/521566879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 msgr2=0x7f7ae015a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 0x7f7ae015a0c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7ad000f650 tx=0x7f7ad000fb10 comp rx=0 tx=0).stop 2024-04-24T08:18:14.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 -- 172.21.15.69:0/521566879 shutdown_connections 2024-04-24T08:18:14.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7abc06c920 0x7f7abc06ede0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ae01436c0 0x7f7ae0143b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ae00ad0d0 0x7f7ae00ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 --2- 172.21.15.69:0/521566879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ac4004800 0x7f7ae015a0c0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 -- 172.21.15.69:0/521566879 >> 172.21.15.69:0/521566879 conn(0x7f7ae00d0160 msgr2=0x7f7ae00b2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 -- 172.21.15.69:0/521566879 shutdown_connections 2024-04-24T08:18:14.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:14.911+0000 7f7adfe00640 1 -- 172.21.15.69:0/521566879 wait complete. 2024-04-24T08:18:14.931 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T08:18:14.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:18:15.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.127+0000 7f81cd000640 1 Processor -- start 2024-04-24T08:18:15.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.127+0000 7f81cd000640 1 -- start start 2024-04-24T08:18:15.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.128+0000 7f81cd000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f81c806db90 0x7f81c8073530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.128+0000 7f81cd000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f81c815b980 0x7f81c815be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.128+0000 7f81cd000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81c8073a70 con 0x7f81c81523f0 2024-04-24T08:18:15.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.128+0000 7f81cd000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f81c80aeac0 con 0x7f81c806db90 2024-04-24T08:18:15.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.128+0000 7f81cd000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f81c80aeda0 con 0x7f81c815b980 2024-04-24T08:18:15.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.128+0000 7f81bea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f81c806db90 0x7f81c8073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.128+0000 7f81bea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f81c806db90 0x7f81c8073530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40338/0 (socket says 172.21.15.69:40338) 2024-04-24T08:18:15.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.128+0000 7f81bea00640 1 -- 172.21.15.69:0/836105471 learned_addr learned my addr 172.21.15.69:0/836105471 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:15.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.129+0000 7f81bea00640 1 -- 172.21.15.69:0/836105471 >> v2:172.21.15.121:3300/0 conn(0x7f81c815b980 msgr2=0x7f81c815be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.129+0000 7f81bea00640 1 --2- 172.21.15.69:0/836105471 >> v2:172.21.15.121:3300/0 conn(0x7f81c815b980 0x7f81c815be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.129+0000 7f81bea00640 1 -- 172.21.15.69:0/836105471 >> v1:172.21.15.121:6789/0 conn(0x7f81c81523f0 legacy=0x7f81c8072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.129+0000 7f81bea00640 1 -- 172.21.15.69:0/836105471 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c80af080 con 0x7f81c806db90 2024-04-24T08:18:15.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.130+0000 7f81bea00640 1 --2- 172.21.15.69:0/836105471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 0x7f81c8073530 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f81c80ad970 tx=0x7f81b002c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e58eda41f4fe9aaf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.130+0000 7f81c6a00640 1 -- 172.21.15.69:0/836105471 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b0039070 con 0x7f81c806db90 2024-04-24T08:18:15.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.130+0000 7f81c6a00640 1 -- 172.21.15.69:0/836105471 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81b0034640 con 0x7f81c806db90 2024-04-24T08:18:15.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.130+0000 7f81c6a00640 1 -- 172.21.15.69:0/836105471 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b0034a10 con 0x7f81c806db90 2024-04-24T08:18:15.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.130+0000 7f81cd000640 1 -- 172.21.15.69:0/836105471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 msgr2=0x7f81c8073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.130+0000 7f81cd000640 1 --2- 172.21.15.69:0/836105471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 0x7f81c8073530 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f81c80ad970 tx=0x7f81b002c3b0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.131+0000 7f81cd000640 1 -- 172.21.15.69:0/836105471 shutdown_connections 2024-04-24T08:18:15.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.131+0000 7f81cd000640 1 --2- 172.21.15.69:0/836105471 >> v2:172.21.15.121:3300/0 conn(0x7f81c815b980 0x7f81c815be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.131+0000 7f81cd000640 1 --2- 172.21.15.69:0/836105471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 0x7f81c8073530 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.131+0000 7f81cd000640 1 -- 172.21.15.69:0/836105471 >> 172.21.15.69:0/836105471 conn(0x7f81c80d0160 msgr2=0x7f81c80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.131+0000 7f81cd000640 1 -- 172.21.15.69:0/836105471 shutdown_connections 2024-04-24T08:18:15.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.131+0000 7f81cd000640 1 -- 172.21.15.69:0/836105471 wait complete. 2024-04-24T08:18:15.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.132+0000 7f81cd000640 1 Processor -- start 2024-04-24T08:18:15.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.133+0000 7f81cd000640 1 -- start start 2024-04-24T08:18:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.133+0000 7f81cd000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 0x7f81c8099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.133+0000 7f81cd000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81c81523f0 0x7f81c8157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.133+0000 7f81cd000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81c815b980 0x7f81c8156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81cd000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f81c806dfd0 con 0x7f81c806db90 2024-04-24T08:18:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81cd000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f81c80b3190 con 0x7f81c81523f0 2024-04-24T08:18:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81cd000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f81c81527d0 con 0x7f81c815b980 2024-04-24T08:18:15.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81c7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 0x7f81c8099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81c7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 0x7f81c8099a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40352/0 (socket says 172.21.15.69:40352) 2024-04-24T08:18:15.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81c7400640 1 -- 172.21.15.69:0/4254413834 learned_addr learned my addr 172.21.15.69:0/4254413834 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81c7400640 1 -- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81c815b980 msgr2=0x7f81c8156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81c7400640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81c815b980 0x7f81c8156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81c7400640 1 -- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81c81523f0 msgr2=0x7f81c8157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81c7400640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81c81523f0 0x7f81c8157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81c7400640 1 -- 172.21.15.69:0/4254413834 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c8156870 con 0x7f81c806db90 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.134+0000 7f81c7400640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 0x7f81c8099a20 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f81b800c440 tx=0x7f81b800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.135+0000 7f81bfe00640 1 -- 172.21.15.69:0/4254413834 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b8017070 con 0x7f81c806db90 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.135+0000 7f81bfe00640 1 -- 172.21.15.69:0/4254413834 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81b80035f0 con 0x7f81c806db90 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.135+0000 7f81bfe00640 1 -- 172.21.15.69:0/4254413834 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b80039e0 con 0x7f81c806db90 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.135+0000 7f81cd000640 1 -- 172.21.15.69:0/4254413834 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c8156b50 con 0x7f81c806db90 2024-04-24T08:18:15.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.135+0000 7f81cd000640 1 -- 172.21.15.69:0/4254413834 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c8157110 con 0x7f81c806db90 2024-04-24T08:18:15.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.136+0000 7f81cd000640 1 -- 172.21.15.69:0/4254413834 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81c8075d50 con 0x7f81c806db90 2024-04-24T08:18:15.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.137+0000 7f81bfe00640 1 -- 172.21.15.69:0/4254413834 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f81b8003b80 con 0x7f81c806db90 2024-04-24T08:18:15.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.138+0000 7f81bfe00640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81a406c8f0 0x7f81a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.138+0000 7f81bfe00640 1 -- 172.21.15.69:0/4254413834 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f81b808aec0 con 0x7f81c806db90 2024-04-24T08:18:15.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.138+0000 7f81bea00640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81a406c8f0 0x7f81a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.140+0000 7f81bea00640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81a406c8f0 0x7f81a406edb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f81b0036000 tx=0x7f81b002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.142+0000 7f81bfe00640 1 -- 172.21.15.69:0/4254413834 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81b80592e0 con 0x7f81c806db90 2024-04-24T08:18:15.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.338+0000 7f81cd000640 1 -- 172.21.15.69:0/4254413834 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f81c8071d80 con 0x7f81a406c8f0 2024-04-24T08:18:15.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.358+0000 7f81bfe00640 1 -- 172.21.15.69:0/4254413834 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252085 (secure 0 0 0) 0x7f81c8071d80 con 0x7f81a406c8f0 2024-04-24T08:18:15.363 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:15.364 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:18:15.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.364+0000 7f81cd000640 1 -- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81a406c8f0 msgr2=0x7f81a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.364+0000 7f81cd000640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81a406c8f0 0x7f81a406edb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f81b0036000 tx=0x7f81b002e000 comp rx=0 tx=0).stop 2024-04-24T08:18:15.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.364+0000 7f81cd000640 1 -- 172.21.15.69:0/4254413834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 msgr2=0x7f81c8099a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.364+0000 7f81cd000640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 0x7f81c8099a20 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f81b800c440 tx=0x7f81b800c900 comp rx=0 tx=0).stop 2024-04-24T08:18:15.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.364+0000 7f81cd000640 1 -- 172.21.15.69:0/4254413834 shutdown_connections 2024-04-24T08:18:15.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.364+0000 7f81cd000640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81a406c8f0 0x7f81a406edb0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.364+0000 7f81cd000640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81c815b980 0x7f81c8156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.364+0000 7f81cd000640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81c81523f0 0x7f81c8157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.364+0000 7f81cd000640 1 --2- 172.21.15.69:0/4254413834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c806db90 0x7f81c8099a20 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.364+0000 7f81cd000640 1 -- 172.21.15.69:0/4254413834 >> 172.21.15.69:0/4254413834 conn(0x7f81c80d0160 msgr2=0x7f81c8072700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.365+0000 7f81cd000640 1 -- 172.21.15.69:0/4254413834 shutdown_connections 2024-04-24T08:18:15.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.365+0000 7f81cd000640 1 -- 172.21.15.69:0/4254413834 wait complete. 2024-04-24T08:18:15.384 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-04-24T08:18:14.884150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260056,"kb_used_data":10600,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714664,"statfs":{"total":773094113280,"available":772827815936,"internally_reserved":0,"allocated":10854400,"data_stored":5340292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.974995"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424088+0000","last_change":"2024-04-24T08:18:10.961565+0000","last_active":"2024-04-24T08:18:12.424088+0000","last_peered":"2024-04-24T08:18:12.424088+0000","last_clean":"2024-04-24T08:18:12.424088+0000","last_became_active":"2024-04-24T08:18:10.961283+0000","last_became_peered":"2024-04-24T08:18:10.961283+0000","last_unstale":"2024-04-24T08:18:12.424088+0000","last_undegraded":"2024-04-24T08:18:12.424088+0000","last_fullsized":"2024-04-24T08:18:12.424088+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:16:24.059564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937615+0000","last_change":"2024-04-24T08:18:10.963956+0000","last_active":"2024-04-24T08:18:11.937615+0000","last_peered":"2024-04-24T08:18:11.937615+0000","last_clean":"2024-04-24T08:18:11.937615+0000","last_became_active":"2024-04-24T08:18:10.963663+0000","last_became_peered":"2024-04-24T08:18:10.963663+0000","last_unstale":"2024-04-24T08:18:11.937615+0000","last_undegraded":"2024-04-24T08:18:11.937615+0000","last_fullsized":"2024-04-24T08:18:11.937615+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:55:25.744718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423922+0000","last_change":"2024-04-24T08:18:10.966965+0000","last_active":"2024-04-24T08:18:12.423922+0000","last_peered":"2024-04-24T08:18:12.423922+0000","last_clean":"2024-04-24T08:18:12.423922+0000","last_became_active":"2024-04-24T08:18:10.966702+0000","last_became_peered":"2024-04-24T08:18:10.966702+0000","last_unstale":"2024-04-24T08:18:12.423922+0000","last_undegraded":"2024-04-24T08:18:12.423922+0000","last_fullsized":"2024-04-24T08:18:12.423922+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:44:07.151307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424304+0000","last_change":"2024-04-24T08:18:10.945475+0000","last_active":"2024-04-24T08:18:12.424304+0000","last_peered":"2024-04-24T08:18:12.424304+0000","last_clean":"2024-04-24T08:18:12.424304+0000","last_became_active":"2024-04-24T08:18:10.944765+0000","last_became_peered":"2024-04-24T08:18:10.944765+0000","last_unstale":"2024-04-24T08:18:12.424304+0000","last_undegraded":"2024-04-24T08:18:12.424304+0000","last_fullsized":"2024-04-24T08:18:12.424304+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:24.288637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424358+0000","last_change":"2024-04-24T08:18:10.969312+0000","last_active":"2024-04-24T08:18:12.424358+0000","last_peered":"2024-04-24T08:18:12.424358+0000","last_clean":"2024-04-24T08:18:12.424358+0000","last_became_active":"2024-04-24T08:18:10.968932+0000","last_became_peered":"2024-04-24T08:18:10.968932+0000","last_unstale":"2024-04-24T08:18:12.424358+0000","last_undegraded":"2024-04-24T08:18:12.424358+0000","last_fullsized":"2024-04-24T08:18:12.424358+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:30:56.763387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424827+0000","last_change":"2024-04-24T08:18:10.971713+0000","last_active":"2024-04-24T08:18:12.424827+0000","last_peered":"2024-04-24T08:18:12.424827+0000","last_clean":"2024-04-24T08:18:12.424827+0000","last_became_active":"2024-04-24T08:18:10.971336+0000","last_became_peered":"2024-04-24T08:18:10.971336+0000","last_unstale":"2024-04-24T08:18:12.424827+0000","last_undegraded":"2024-04-24T08:18:12.424827+0000","last_fullsized":"2024-04-24T08:18:12.424827+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:55:48.779115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425957+0000","last_change":"2024-04-24T08:18:10.967624+0000","last_active":"2024-04-24T08:18:12.425957+0000","last_peered":"2024-04-24T08:18:12.425957+0000","last_clean":"2024-04-24T08:18:12.425957+0000","last_became_active":"2024-04-24T08:18:10.967364+0000","last_became_peered":"2024-04-24T08:18:10.967364+0000","last_unstale":"2024-04-24T08:18:12.425957+0000","last_undegraded":"2024-04-24T08:18:12.425957+0000","last_fullsized":"2024-04-24T08:18:12.425957+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:37.236811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937228+0000","last_change":"2024-04-24T08:18:10.968700+0000","last_active":"2024-04-24T08:18:11.937228+0000","last_peered":"2024-04-24T08:18:11.937228+0000","last_clean":"2024-04-24T08:18:11.937228+0000","last_became_active":"2024-04-24T08:18:10.968384+0000","last_became_peered":"2024-04-24T08:18:10.968384+0000","last_unstale":"2024-04-24T08:18:11.937228+0000","last_undegraded":"2024-04-24T08:18:11.937228+0000","last_fullsized":"2024-04-24T08:18:11.937228+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:13:58.550617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945751+0000","last_change":"2024-04-24T08:18:10.960428+0000","last_active":"2024-04-24T08:18:11.945751+0000","last_peered":"2024-04-24T08:18:11.945751+0000","last_clean":"2024-04-24T08:18:11.945751+0000","last_became_active":"2024-04-24T08:18:10.956831+0000","last_became_peered":"2024-04-24T08:18:10.956831+0000","last_unstale":"2024-04-24T08:18:11.945751+0000","last_undegraded":"2024-04-24T08:18:11.945751+0000","last_fullsized":"2024-04-24T08:18:11.945751+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:26:18.219342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426383+0000","last_change":"2024-04-24T08:18:10.972276+0000","last_active":"2024-04-24T08:18:12.426383+0000","last_peered":"2024-04-24T08:18:12.426383+0000","last_clean":"2024-04-24T08:18:12.426383+0000","last_became_active":"2024-04-24T08:18:10.972004+0000","last_became_peered":"2024-04-24T08:18:10.972004+0000","last_unstale":"2024-04-24T08:18:12.426383+0000","last_undegraded":"2024-04-24T08:18:12.426383+0000","last_fullsized":"2024-04-24T08:18:12.426383+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:56:22.407532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424889+0000","last_change":"2024-04-24T08:18:10.968740+0000","last_active":"2024-04-24T08:18:12.424889+0000","last_peered":"2024-04-24T08:18:12.424889+0000","last_clean":"2024-04-24T08:18:12.424889+0000","last_became_active":"2024-04-24T08:18:10.968058+0000","last_became_peered":"2024-04-24T08:18:10.968058+0000","last_unstale":"2024-04-24T08:18:12.424889+0000","last_undegraded":"2024-04-24T08:18:12.424889+0000","last_fullsized":"2024-04-24T08:18:12.424889+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:41:03.904194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945715+0000","last_change":"2024-04-24T08:18:10.964804+0000","last_active":"2024-04-24T08:18:11.945715+0000","last_peered":"2024-04-24T08:18:11.945715+0000","last_clean":"2024-04-24T08:18:11.945715+0000","last_became_active":"2024-04-24T08:18:10.964219+0000","last_became_peered":"2024-04-24T08:18:10.964219+0000","last_unstale":"2024-04-24T08:18:11.945715+0000","last_undegraded":"2024-04-24T08:18:11.945715+0000","last_fullsized":"2024-04-24T08:18:11.945715+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:31.570848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425162+0000","last_change":"2024-04-24T08:18:10.960719+0000","last_active":"2024-04-24T08:18:12.425162+0000","last_peered":"2024-04-24T08:18:12.425162+0000","last_clean":"2024-04-24T08:18:12.425162+0000","last_became_active":"2024-04-24T08:18:10.956874+0000","last_became_peered":"2024-04-24T08:18:10.956874+0000","last_unstale":"2024-04-24T08:18:12.425162+0000","last_undegraded":"2024-04-24T08:18:12.425162+0000","last_fullsized":"2024-04-24T08:18:12.425162+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:43:44.792654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424902+0000","last_change":"2024-04-24T08:18:10.963233+0000","last_active":"2024-04-24T08:18:12.424902+0000","last_peered":"2024-04-24T08:18:12.424902+0000","last_clean":"2024-04-24T08:18:12.424902+0000","last_became_active":"2024-04-24T08:18:10.962962+0000","last_became_peered":"2024-04-24T08:18:10.962962+0000","last_unstale":"2024-04-24T08:18:12.424902+0000","last_undegraded":"2024-04-24T08:18:12.424902+0000","last_fullsized":"2024-04-24T08:18:12.424902+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:55:22.703333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422863+0000","last_change":"2024-04-24T08:18:10.951541+0000","last_active":"2024-04-24T08:18:12.422863+0000","last_peered":"2024-04-24T08:18:12.422863+0000","last_clean":"2024-04-24T08:18:12.422863+0000","last_became_active":"2024-04-24T08:18:10.951239+0000","last_became_peered":"2024-04-24T08:18:10.951239+0000","last_unstale":"2024-04-24T08:18:12.422863+0000","last_undegraded":"2024-04-24T08:18:12.422863+0000","last_fullsized":"2024-04-24T08:18:12.422863+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:00:07.240078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423928+0000","last_change":"2024-04-24T08:18:10.966894+0000","last_active":"2024-04-24T08:18:12.423928+0000","last_peered":"2024-04-24T08:18:12.423928+0000","last_clean":"2024-04-24T08:18:12.423928+0000","last_became_active":"2024-04-24T08:18:10.966628+0000","last_became_peered":"2024-04-24T08:18:10.966628+0000","last_unstale":"2024-04-24T08:18:12.423928+0000","last_undegraded":"2024-04-24T08:18:12.423928+0000","last_fullsized":"2024-04-24T08:18:12.423928+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:01:36.187055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.938215+0000","last_change":"2024-04-24T08:18:10.949255+0000","last_active":"2024-04-24T08:18:11.938215+0000","last_peered":"2024-04-24T08:18:11.938215+0000","last_clean":"2024-04-24T08:18:11.938215+0000","last_became_active":"2024-04-24T08:18:10.948925+0000","last_became_peered":"2024-04-24T08:18:10.948925+0000","last_unstale":"2024-04-24T08:18:11.938215+0000","last_undegraded":"2024-04-24T08:18:11.938215+0000","last_fullsized":"2024-04-24T08:18:11.938215+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:14:30.218520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426760+0000","last_change":"2024-04-24T08:18:10.960378+0000","last_active":"2024-04-24T08:18:12.426760+0000","last_peered":"2024-04-24T08:18:12.426760+0000","last_clean":"2024-04-24T08:18:12.426760+0000","last_became_active":"2024-04-24T08:18:10.960054+0000","last_became_peered":"2024-04-24T08:18:10.960054+0000","last_unstale":"2024-04-24T08:18:12.426760+0000","last_undegraded":"2024-04-24T08:18:12.426760+0000","last_fullsized":"2024-04-24T08:18:12.426760+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:31.570848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426046+0000","last_change":"2024-04-24T08:18:10.972125+0000","last_active":"2024-04-24T08:18:12.426046+0000","last_peered":"2024-04-24T08:18:12.426046+0000","last_clean":"2024-04-24T08:18:12.426046+0000","last_became_active":"2024-04-24T08:18:10.969463+0000","last_became_peered":"2024-04-24T08:18:10.969463+0000","last_unstale":"2024-04-24T08:18:12.426046+0000","last_undegraded":"2024-04-24T08:18:12.426046+0000","last_fullsized":"2024-04-24T08:18:12.426046+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:59:40.923189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945749+0000","last_change":"2024-04-24T08:18:10.955613+0000","last_active":"2024-04-24T08:18:11.945749+0000","last_peered":"2024-04-24T08:18:11.945749+0000","last_clean":"2024-04-24T08:18:11.945749+0000","last_became_active":"2024-04-24T08:18:10.955239+0000","last_became_peered":"2024-04-24T08:18:10.955239+0000","last_unstale":"2024-04-24T08:18:11.945749+0000","last_undegraded":"2024-04-24T08:18:11.945749+0000","last_fullsized":"2024-04-24T08:18:11.945749+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:44:52.657162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425707+0000","last_change":"2024-04-24T08:18:10.964804+0000","last_active":"2024-04-24T08:18:12.425707+0000","last_peered":"2024-04-24T08:18:12.425707+0000","last_clean":"2024-04-24T08:18:12.425707+0000","last_became_active":"2024-04-24T08:18:10.964543+0000","last_became_peered":"2024-04-24T08:18:10.964543+0000","last_unstale":"2024-04-24T08:18:12.425707+0000","last_undegraded":"2024-04-24T08:18:12.425707+0000","last_fullsized":"2024-04-24T08:18:12.425707+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:30.856317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937249+0000","last_change":"2024-04-24T08:18:10.964957+0000","last_active":"2024-04-24T08:18:11.937249+0000","last_peered":"2024-04-24T08:18:11.937249+0000","last_clean":"2024-04-24T08:18:11.937249+0000","last_became_active":"2024-04-24T08:18:10.963117+0000","last_became_peered":"2024-04-24T08:18:10.963117+0000","last_unstale":"2024-04-24T08:18:11.937249+0000","last_undegraded":"2024-04-24T08:18:11.937249+0000","last_fullsized":"2024-04-24T08:18:11.937249+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:30:56.763387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937275+0000","last_change":"2024-04-24T08:18:10.969508+0000","last_active":"2024-04-24T08:18:11.937275+0000","last_peered":"2024-04-24T08:18:11.937275+0000","last_clean":"2024-04-24T08:18:11.937275+0000","last_became_active":"2024-04-24T08:18:10.967363+0000","last_became_peered":"2024-04-24T08:18:10.967363+0000","last_unstale":"2024-04-24T08:18:11.937275+0000","last_undegraded":"2024-04-24T08:18:11.937275+0000","last_fullsized":"2024-04-24T08:18:11.937275+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:55:49.106660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424975+0000","last_change":"2024-04-24T08:18:10.971483+0000","last_active":"2024-04-24T08:18:12.424975+0000","last_peered":"2024-04-24T08:18:12.424975+0000","last_clean":"2024-04-24T08:18:12.424975+0000","last_became_active":"2024-04-24T08:18:10.969672+0000","last_became_peered":"2024-04-24T08:18:10.969672+0000","last_unstale":"2024-04-24T08:18:12.424975+0000","last_undegraded":"2024-04-24T08:18:12.424975+0000","last_fullsized":"2024-04-24T08:18:12.424975+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:50:21.118405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425844+0000","last_change":"2024-04-24T08:18:10.975602+0000","last_active":"2024-04-24T08:18:12.425844+0000","last_peered":"2024-04-24T08:18:12.425844+0000","last_clean":"2024-04-24T08:18:12.425844+0000","last_became_active":"2024-04-24T08:18:10.975265+0000","last_became_peered":"2024-04-24T08:18:10.975265+0000","last_unstale":"2024-04-24T08:18:12.425844+0000","last_undegraded":"2024-04-24T08:18:12.425844+0000","last_fullsized":"2024-04-24T08:18:12.425844+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:25.063819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937751+0000","last_change":"2024-04-24T08:18:10.942883+0000","last_active":"2024-04-24T08:18:11.937751+0000","last_peered":"2024-04-24T08:18:11.937751+0000","last_clean":"2024-04-24T08:18:11.937751+0000","last_became_active":"2024-04-24T08:18:10.942585+0000","last_became_peered":"2024-04-24T08:18:10.942585+0000","last_unstale":"2024-04-24T08:18:11.937751+0000","last_undegraded":"2024-04-24T08:18:11.937751+0000","last_fullsized":"2024-04-24T08:18:11.937751+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:03:24.238839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425303+0000","last_change":"2024-04-24T08:18:10.969281+0000","last_active":"2024-04-24T08:18:12.425303+0000","last_peered":"2024-04-24T08:18:12.425303+0000","last_clean":"2024-04-24T08:18:12.425303+0000","last_became_active":"2024-04-24T08:18:10.968943+0000","last_became_peered":"2024-04-24T08:18:10.968943+0000","last_unstale":"2024-04-24T08:18:12.425303+0000","last_undegraded":"2024-04-24T08:18:12.425303+0000","last_fullsized":"2024-04-24T08:18:12.425303+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:00:07.240078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425070+0000","last_change":"2024-04-24T08:18:10.974369+0000","last_active":"2024-04-24T08:18:12.425070+0000","last_peered":"2024-04-24T08:18:12.425070+0000","last_clean":"2024-04-24T08:18:12.425070+0000","last_became_active":"2024-04-24T08:18:10.973666+0000","last_became_peered":"2024-04-24T08:18:10.973666+0000","last_unstale":"2024-04-24T08:18:12.425070+0000","last_undegraded":"2024-04-24T08:18:12.425070+0000","last_fullsized":"2024-04-24T08:18:12.425070+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:55:25.744718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937076+0000","last_change":"2024-04-24T08:18:10.943657+0000","last_active":"2024-04-24T08:18:11.937076+0000","last_peered":"2024-04-24T08:18:11.937076+0000","last_clean":"2024-04-24T08:18:11.937076+0000","last_became_active":"2024-04-24T08:18:10.943355+0000","last_became_peered":"2024-04-24T08:18:10.943355+0000","last_unstale":"2024-04-24T08:18:11.937076+0000","last_undegraded":"2024-04-24T08:18:11.937076+0000","last_fullsized":"2024-04-24T08:18:11.937076+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:52:05.222065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425135+0000","last_change":"2024-04-24T08:18:10.966574+0000","last_active":"2024-04-24T08:18:12.425135+0000","last_peered":"2024-04-24T08:18:12.425135+0000","last_clean":"2024-04-24T08:18:12.425135+0000","last_became_active":"2024-04-24T08:18:10.965002+0000","last_became_peered":"2024-04-24T08:18:10.965002+0000","last_unstale":"2024-04-24T08:18:12.425135+0000","last_undegraded":"2024-04-24T08:18:12.425135+0000","last_fullsized":"2024-04-24T08:18:12.425135+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:17.068910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423160+0000","last_change":"2024-04-24T08:18:10.968123+0000","last_active":"2024-04-24T08:18:12.423160+0000","last_peered":"2024-04-24T08:18:12.423160+0000","last_clean":"2024-04-24T08:18:12.423160+0000","last_became_active":"2024-04-24T08:18:10.967845+0000","last_became_peered":"2024-04-24T08:18:10.967845+0000","last_unstale":"2024-04-24T08:18:12.423160+0000","last_undegraded":"2024-04-24T08:18:12.423160+0000","last_fullsized":"2024-04-24T08:18:12.423160+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:53:18.191200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422098+0000","last_change":"2024-04-24T08:18:10.948105+0000","last_active":"2024-04-24T08:18:12.422098+0000","last_peered":"2024-04-24T08:18:12.422098+0000","last_clean":"2024-04-24T08:18:12.422098+0000","last_became_active":"2024-04-24T08:18:10.946467+0000","last_became_peered":"2024-04-24T08:18:10.946467+0000","last_unstale":"2024-04-24T08:18:12.422098+0000","last_undegraded":"2024-04-24T08:18:12.422098+0000","last_fullsized":"2024-04-24T08:18:12.422098+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:54.348306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425985+0000","last_change":"2024-04-24T08:18:10.965880+0000","last_active":"2024-04-24T08:18:12.425985+0000","last_peered":"2024-04-24T08:18:12.425985+0000","last_clean":"2024-04-24T08:18:12.425985+0000","last_became_active":"2024-04-24T08:18:10.965449+0000","last_became_peered":"2024-04-24T08:18:10.965449+0000","last_unstale":"2024-04-24T08:18:12.425985+0000","last_undegraded":"2024-04-24T08:18:12.425985+0000","last_fullsized":"2024-04-24T08:18:12.425985+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:06:21.094134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"96'4","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937821+0000","last_change":"2024-04-24T08:18:09.961054+0000","last_active":"2024-04-24T08:18:11.937821+0000","last_peered":"2024-04-24T08:18:11.937821+0000","last_clean":"2024-04-24T08:18:11.937821+0000","last_became_active":"2024-04-24T08:18:09.960763+0000","last_became_peered":"2024-04-24T08:18:09.960763+0000","last_unstale":"2024-04-24T08:18:11.937821+0000","last_undegraded":"2024-04-24T08:18:11.937821+0000","last_fullsized":"2024-04-24T08:18:11.937821+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:00:06.222300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.427165+0000","last_change":"2024-04-24T08:18:10.966892+0000","last_active":"2024-04-24T08:18:12.427165+0000","last_peered":"2024-04-24T08:18:12.427165+0000","last_clean":"2024-04-24T08:18:12.427165+0000","last_became_active":"2024-04-24T08:18:10.966578+0000","last_became_peered":"2024-04-24T08:18:10.966578+0000","last_unstale":"2024-04-24T08:18:12.427165+0000","last_undegraded":"2024-04-24T08:18:12.427165+0000","last_fullsized":"2024-04-24T08:18:12.427165+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:30:53.924950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"96'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426397+0000","last_change":"2024-04-24T08:18:09.962285+0000","last_active":"2024-04-24T08:18:12.426397+0000","last_peered":"2024-04-24T08:18:12.426397+0000","last_clean":"2024-04-24T08:18:12.426397+0000","last_became_active":"2024-04-24T08:18:09.961972+0000","last_became_peered":"2024-04-24T08:18:09.961972+0000","last_unstale":"2024-04-24T08:18:12.426397+0000","last_undegraded":"2024-04-24T08:18:12.426397+0000","last_fullsized":"2024-04-24T08:18:12.426397+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:05:58.842655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422990+0000","last_change":"2024-04-24T08:18:10.961857+0000","last_active":"2024-04-24T08:18:12.422990+0000","last_peered":"2024-04-24T08:18:12.422990+0000","last_clean":"2024-04-24T08:18:12.422990+0000","last_became_active":"2024-04-24T08:18:10.961544+0000","last_became_peered":"2024-04-24T08:18:10.961544+0000","last_unstale":"2024-04-24T08:18:12.422990+0000","last_undegraded":"2024-04-24T08:18:12.422990+0000","last_fullsized":"2024-04-24T08:18:12.422990+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:19:17.771354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.938212+0000","last_change":"2024-04-24T08:18:09.956166+0000","last_active":"2024-04-24T08:18:11.938212+0000","last_peered":"2024-04-24T08:18:11.938212+0000","last_clean":"2024-04-24T08:18:11.938212+0000","last_became_active":"2024-04-24T08:18:09.955195+0000","last_became_peered":"2024-04-24T08:18:09.955195+0000","last_unstale":"2024-04-24T08:18:11.938212+0000","last_undegraded":"2024-04-24T08:18:11.938212+0000","last_fullsized":"2024-04-24T08:18:11.938212+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:43:06.185599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945676+0000","last_change":"2024-04-24T08:18:10.970331+0000","last_active":"2024-04-24T08:18:11.945676+0000","last_peered":"2024-04-24T08:18:11.945676+0000","last_clean":"2024-04-24T08:18:11.945676+0000","last_became_active":"2024-04-24T08:18:10.968722+0000","last_became_peered":"2024-04-24T08:18:10.968722+0000","last_unstale":"2024-04-24T08:18:11.945676+0000","last_undegraded":"2024-04-24T08:18:11.945676+0000","last_fullsized":"2024-04-24T08:18:11.945676+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:43:44.792654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937768+0000","last_change":"2024-04-24T08:18:09.951148+0000","last_active":"2024-04-24T08:18:11.937768+0000","last_peered":"2024-04-24T08:18:11.937768+0000","last_clean":"2024-04-24T08:18:11.937768+0000","last_became_active":"2024-04-24T08:18:09.950870+0000","last_became_peered":"2024-04-24T08:18:09.950870+0000","last_unstale":"2024-04-24T08:18:11.937768+0000","last_undegraded":"2024-04-24T08:18:11.937768+0000","last_fullsized":"2024-04-24T08:18:11.937768+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:55:32.014758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.942592+0000","last_change":"2024-04-24T08:18:10.952307+0000","last_active":"2024-04-24T08:18:11.942592+0000","last_peered":"2024-04-24T08:18:11.942592+0000","last_clean":"2024-04-24T08:18:11.942592+0000","last_became_active":"2024-04-24T08:18:10.951464+0000","last_became_peered":"2024-04-24T08:18:10.951464+0000","last_unstale":"2024-04-24T08:18:11.942592+0000","last_undegraded":"2024-04-24T08:18:11.942592+0000","last_fullsized":"2024-04-24T08:18:11.942592+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:44:07.151307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"96'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423279+0000","last_change":"2024-04-24T08:18:09.960759+0000","last_active":"2024-04-24T08:18:12.423279+0000","last_peered":"2024-04-24T08:18:12.423279+0000","last_clean":"2024-04-24T08:18:12.423279+0000","last_became_active":"2024-04-24T08:18:09.960406+0000","last_became_peered":"2024-04-24T08:18:09.960406+0000","last_unstale":"2024-04-24T08:18:12.423279+0000","last_undegraded":"2024-04-24T08:18:12.423279+0000","last_fullsized":"2024-04-24T08:18:12.423279+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:03:39.760745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424914+0000","last_change":"2024-04-24T08:18:10.968582+0000","last_active":"2024-04-24T08:18:12.424914+0000","last_peered":"2024-04-24T08:18:12.424914+0000","last_clean":"2024-04-24T08:18:12.424914+0000","last_became_active":"2024-04-24T08:18:10.966087+0000","last_became_peered":"2024-04-24T08:18:10.966087+0000","last_unstale":"2024-04-24T08:18:12.424914+0000","last_undegraded":"2024-04-24T08:18:12.424914+0000","last_fullsized":"2024-04-24T08:18:12.424914+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:43:07.203377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"96'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937575+0000","last_change":"2024-04-24T08:18:09.956966+0000","last_active":"2024-04-24T08:18:11.937575+0000","last_peered":"2024-04-24T08:18:11.937575+0000","last_clean":"2024-04-24T08:18:11.937575+0000","last_became_active":"2024-04-24T08:18:09.955343+0000","last_became_peered":"2024-04-24T08:18:09.955343+0000","last_unstale":"2024-04-24T08:18:11.937575+0000","last_undegraded":"2024-04-24T08:18:11.937575+0000","last_fullsized":"2024-04-24T08:18:11.937575+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:50:20.100627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423776+0000","last_change":"2024-04-24T08:18:10.947122+0000","last_active":"2024-04-24T08:18:12.423776+0000","last_peered":"2024-04-24T08:18:12.423776+0000","last_clean":"2024-04-24T08:18:12.423776+0000","last_became_active":"2024-04-24T08:18:10.945155+0000","last_became_peered":"2024-04-24T08:18:10.945155+0000","last_unstale":"2024-04-24T08:18:12.423776+0000","last_undegraded":"2024-04-24T08:18:12.423776+0000","last_fullsized":"2024-04-24T08:18:12.423776+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:37.236811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424175+0000","last_change":"2024-04-24T08:18:09.963877+0000","last_active":"2024-04-24T08:18:12.424175+0000","last_peered":"2024-04-24T08:18:12.424175+0000","last_clean":"2024-04-24T08:18:12.424175+0000","last_became_active":"2024-04-24T08:18:09.963493+0000","last_became_peered":"2024-04-24T08:18:09.963493+0000","last_unstale":"2024-04-24T08:18:12.424175+0000","last_undegraded":"2024-04-24T08:18:12.424175+0000","last_fullsized":"2024-04-24T08:18:12.424175+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:43:06.185599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.944954+0000","last_change":"2024-04-24T08:18:10.975197+0000","last_active":"2024-04-24T08:18:11.944954+0000","last_peered":"2024-04-24T08:18:11.944954+0000","last_clean":"2024-04-24T08:18:11.944954+0000","last_became_active":"2024-04-24T08:18:10.971948+0000","last_became_peered":"2024-04-24T08:18:10.971948+0000","last_unstale":"2024-04-24T08:18:11.944954+0000","last_undegraded":"2024-04-24T08:18:11.944954+0000","last_fullsized":"2024-04-24T08:18:11.944954+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:47.129428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422067+0000","last_change":"2024-04-24T08:18:09.955900+0000","last_active":"2024-04-24T08:18:12.422067+0000","last_peered":"2024-04-24T08:18:12.422067+0000","last_clean":"2024-04-24T08:18:12.422067+0000","last_became_active":"2024-04-24T08:18:09.954789+0000","last_became_peered":"2024-04-24T08:18:09.954789+0000","last_unstale":"2024-04-24T08:18:12.422067+0000","last_undegraded":"2024-04-24T08:18:12.422067+0000","last_fullsized":"2024-04-24T08:18:12.422067+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:29.838539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.938225+0000","last_change":"2024-04-24T08:18:10.958985+0000","last_active":"2024-04-24T08:18:11.938225+0000","last_peered":"2024-04-24T08:18:11.938225+0000","last_clean":"2024-04-24T08:18:11.938225+0000","last_became_active":"2024-04-24T08:18:10.958452+0000","last_became_peered":"2024-04-24T08:18:10.958452+0000","last_unstale":"2024-04-24T08:18:11.938225+0000","last_undegraded":"2024-04-24T08:18:11.938225+0000","last_fullsized":"2024-04-24T08:18:11.938225+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:56:22.407532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425189+0000","last_change":"2024-04-24T08:18:09.951533+0000","last_active":"2024-04-24T08:18:12.425189+0000","last_peered":"2024-04-24T08:18:12.425189+0000","last_clean":"2024-04-24T08:18:12.425189+0000","last_became_active":"2024-04-24T08:18:09.950569+0000","last_became_peered":"2024-04-24T08:18:09.950569+0000","last_unstale":"2024-04-24T08:18:12.425189+0000","last_undegraded":"2024-04-24T08:18:12.425189+0000","last_fullsized":"2024-04-24T08:18:12.425189+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:59:39.905411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945741+0000","last_change":"2024-04-24T08:18:10.971167+0000","last_active":"2024-04-24T08:18:11.945741+0000","last_peered":"2024-04-24T08:18:11.945741+0000","last_clean":"2024-04-24T08:18:11.945741+0000","last_became_active":"2024-04-24T08:18:10.969277+0000","last_became_peered":"2024-04-24T08:18:10.969277+0000","last_unstale":"2024-04-24T08:18:11.945741+0000","last_undegraded":"2024-04-24T08:18:11.945741+0000","last_fullsized":"2024-04-24T08:18:11.945741+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:30:53.924950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426757+0000","last_change":"2024-04-24T08:18:09.953624+0000","last_active":"2024-04-24T08:18:12.426757+0000","last_peered":"2024-04-24T08:18:12.426757+0000","last_clean":"2024-04-24T08:18:12.426757+0000","last_became_active":"2024-04-24T08:18:09.953332+0000","last_became_peered":"2024-04-24T08:18:09.953332+0000","last_unstale":"2024-04-24T08:18:12.426757+0000","last_undegraded":"2024-04-24T08:18:12.426757+0000","last_fullsized":"2024-04-24T08:18:12.426757+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:23:20.288631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425275+0000","last_change":"2024-04-24T08:18:10.976693+0000","last_active":"2024-04-24T08:18:12.425275+0000","last_peered":"2024-04-24T08:18:12.425275+0000","last_clean":"2024-04-24T08:18:12.425275+0000","last_became_active":"2024-04-24T08:18:10.976379+0000","last_became_peered":"2024-04-24T08:18:10.976379+0000","last_unstale":"2024-04-24T08:18:12.425275+0000","last_undegraded":"2024-04-24T08:18:12.425275+0000","last_fullsized":"2024-04-24T08:18:12.425275+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:03:24.238839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"96'3","reported_seq":17,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426383+0000","last_change":"2024-04-24T08:18:09.959508+0000","last_active":"2024-04-24T08:18:12.426383+0000","last_peered":"2024-04-24T08:18:12.426383+0000","last_clean":"2024-04-24T08:18:12.426383+0000","last_became_active":"2024-04-24T08:18:09.959216+0000","last_became_peered":"2024-04-24T08:18:09.959216+0000","last_unstale":"2024-04-24T08:18:12.426383+0000","last_undegraded":"2024-04-24T08:18:12.426383+0000","last_fullsized":"2024-04-24T08:18:12.426383+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:59:36.083209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425689+0000","last_change":"2024-04-24T08:18:10.962994+0000","last_active":"2024-04-24T08:18:12.425689+0000","last_peered":"2024-04-24T08:18:12.425689+0000","last_clean":"2024-04-24T08:18:12.425689+0000","last_became_active":"2024-04-24T08:18:10.962668+0000","last_became_peered":"2024-04-24T08:18:10.962668+0000","last_unstale":"2024-04-24T08:18:12.425689+0000","last_undegraded":"2024-04-24T08:18:12.425689+0000","last_fullsized":"2024-04-24T08:18:12.425689+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:53:18.191200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"96'6","reported_seq":20,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945987+0000","last_change":"2024-04-24T08:18:09.948927+0000","last_active":"2024-04-24T08:18:11.945987+0000","last_peered":"2024-04-24T08:18:11.945987+0000","last_clean":"2024-04-24T08:18:11.945987+0000","last_became_active":"2024-04-24T08:18:09.948487+0000","last_became_peered":"2024-04-24T08:18:09.948487+0000","last_unstale":"2024-04-24T08:18:11.945987+0000","last_undegraded":"2024-04-24T08:18:11.945987+0000","last_fullsized":"2024-04-24T08:18:11.945987+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:14:26.159337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424112+0000","last_change":"2024-04-24T08:18:10.961987+0000","last_active":"2024-04-24T08:18:12.424112+0000","last_peered":"2024-04-24T08:18:12.424112+0000","last_clean":"2024-04-24T08:18:12.424112+0000","last_became_active":"2024-04-24T08:18:10.961725+0000","last_became_peered":"2024-04-24T08:18:10.961725+0000","last_unstale":"2024-04-24T08:18:12.424112+0000","last_undegraded":"2024-04-24T08:18:12.424112+0000","last_fullsized":"2024-04-24T08:18:12.424112+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:47.129428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425970+0000","last_change":"2024-04-24T08:18:09.948853+0000","last_active":"2024-04-24T08:18:12.425970+0000","last_peered":"2024-04-24T08:18:12.425970+0000","last_clean":"2024-04-24T08:18:12.425970+0000","last_became_active":"2024-04-24T08:18:09.948579+0000","last_became_peered":"2024-04-24T08:18:09.948579+0000","last_unstale":"2024-04-24T08:18:12.425970+0000","last_undegraded":"2024-04-24T08:18:12.425970+0000","last_fullsized":"2024-04-24T08:18:12.425970+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:13.441179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422989+0000","last_change":"2024-04-24T08:18:10.958159+0000","last_active":"2024-04-24T08:18:12.422989+0000","last_peered":"2024-04-24T08:18:12.422989+0000","last_clean":"2024-04-24T08:18:12.422989+0000","last_became_active":"2024-04-24T08:18:10.957886+0000","last_became_peered":"2024-04-24T08:18:10.957886+0000","last_unstale":"2024-04-24T08:18:12.422989+0000","last_undegraded":"2024-04-24T08:18:12.422989+0000","last_fullsized":"2024-04-24T08:18:12.422989+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:39.543199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425565+0000","last_change":"2024-04-24T08:18:09.955673+0000","last_active":"2024-04-24T08:18:12.425565+0000","last_peered":"2024-04-24T08:18:12.425565+0000","last_clean":"2024-04-24T08:18:12.425565+0000","last_became_active":"2024-04-24T08:18:09.955271+0000","last_became_peered":"2024-04-24T08:18:09.955271+0000","last_unstale":"2024-04-24T08:18:12.425565+0000","last_undegraded":"2024-04-24T08:18:12.425565+0000","last_fullsized":"2024-04-24T08:18:12.425565+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:21:00.326506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.944637+0000","last_change":"2024-04-24T08:18:10.967236+0000","last_active":"2024-04-24T08:18:11.944637+0000","last_peered":"2024-04-24T08:18:11.944637+0000","last_clean":"2024-04-24T08:18:11.944637+0000","last_became_active":"2024-04-24T08:18:10.966957+0000","last_became_peered":"2024-04-24T08:18:10.966957+0000","last_unstale":"2024-04-24T08:18:11.944637+0000","last_undegraded":"2024-04-24T08:18:11.944637+0000","last_fullsized":"2024-04-24T08:18:11.944637+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:37.236811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.944806+0000","last_change":"2024-04-24T08:18:09.961786+0000","last_active":"2024-04-24T08:18:11.944806+0000","last_peered":"2024-04-24T08:18:11.944806+0000","last_clean":"2024-04-24T08:18:11.944806+0000","last_became_active":"2024-04-24T08:18:09.960475+0000","last_became_peered":"2024-04-24T08:18:09.960475+0000","last_unstale":"2024-04-24T08:18:11.944806+0000","last_undegraded":"2024-04-24T08:18:11.944806+0000","last_fullsized":"2024-04-24T08:18:11.944806+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:47:14.219825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424412+0000","last_change":"2024-04-24T08:18:10.965745+0000","last_active":"2024-04-24T08:18:12.424412+0000","last_peered":"2024-04-24T08:18:12.424412+0000","last_clean":"2024-04-24T08:18:12.424412+0000","last_became_active":"2024-04-24T08:18:10.965111+0000","last_became_peered":"2024-04-24T08:18:10.965111+0000","last_unstale":"2024-04-24T08:18:12.424412+0000","last_undegraded":"2024-04-24T08:18:12.424412+0000","last_fullsized":"2024-04-24T08:18:12.424412+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:33:45.349270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"96'8","reported_seq":22,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422353+0000","last_change":"2024-04-24T08:18:09.958822+0000","last_active":"2024-04-24T08:18:12.422353+0000","last_peered":"2024-04-24T08:18:12.422353+0000","last_clean":"2024-04-24T08:18:12.422353+0000","last_became_active":"2024-04-24T08:18:09.958427+0000","last_became_peered":"2024-04-24T08:18:09.958427+0000","last_unstale":"2024-04-24T08:18:12.422353+0000","last_undegraded":"2024-04-24T08:18:12.422353+0000","last_fullsized":"2024-04-24T08:18:12.422353+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:06:20.076356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426256+0000","last_change":"2024-04-24T08:18:10.967330+0000","last_active":"2024-04-24T08:18:12.426256+0000","last_peered":"2024-04-24T08:18:12.426256+0000","last_clean":"2024-04-24T08:18:12.426256+0000","last_became_active":"2024-04-24T08:18:10.966987+0000","last_became_peered":"2024-04-24T08:18:10.966987+0000","last_unstale":"2024-04-24T08:18:12.426256+0000","last_undegraded":"2024-04-24T08:18:12.426256+0000","last_fullsized":"2024-04-24T08:18:12.426256+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:14:30.218520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945513+0000","last_change":"2024-04-24T08:18:09.955420+0000","last_active":"2024-04-24T08:18:11.945513+0000","last_peered":"2024-04-24T08:18:11.945513+0000","last_clean":"2024-04-24T08:18:11.945513+0000","last_became_active":"2024-04-24T08:18:09.955114+0000","last_became_peered":"2024-04-24T08:18:09.955114+0000","last_unstale":"2024-04-24T08:18:11.945513+0000","last_undegraded":"2024-04-24T08:18:11.945513+0000","last_fullsized":"2024-04-24T08:18:11.945513+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:07:36.574089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.941966+0000","last_change":"2024-04-24T08:18:10.967646+0000","last_active":"2024-04-24T08:18:11.941966+0000","last_peered":"2024-04-24T08:18:11.941966+0000","last_clean":"2024-04-24T08:18:11.941966+0000","last_became_active":"2024-04-24T08:18:10.966255+0000","last_became_peered":"2024-04-24T08:18:10.966255+0000","last_unstale":"2024-04-24T08:18:11.941966+0000","last_undegraded":"2024-04-24T08:18:11.941966+0000","last_fullsized":"2024-04-24T08:18:11.941966+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:04:56.555360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"96'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424857+0000","last_change":"2024-04-24T08:18:09.961530+0000","last_active":"2024-04-24T08:18:12.424857+0000","last_peered":"2024-04-24T08:18:12.424857+0000","last_clean":"2024-04-24T08:18:12.424857+0000","last_became_active":"2024-04-24T08:18:09.960225+0000","last_became_peered":"2024-04-24T08:18:09.960225+0000","last_unstale":"2024-04-24T08:18:12.424857+0000","last_undegraded":"2024-04-24T08:18:12.424857+0000","last_fullsized":"2024-04-24T08:18:12.424857+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:00:06.222300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426099+0000","last_change":"2024-04-24T08:18:10.969069+0000","last_active":"2024-04-24T08:18:12.426099+0000","last_peered":"2024-04-24T08:18:12.426099+0000","last_clean":"2024-04-24T08:18:12.426099+0000","last_became_active":"2024-04-24T08:18:10.968472+0000","last_became_peered":"2024-04-24T08:18:10.968472+0000","last_unstale":"2024-04-24T08:18:12.426099+0000","last_undegraded":"2024-04-24T08:18:12.426099+0000","last_fullsized":"2024-04-24T08:18:12.426099+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:47.129428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937931+0000","last_change":"2024-04-24T08:18:09.950987+0000","last_active":"2024-04-24T08:18:11.937931+0000","last_peered":"2024-04-24T08:18:11.937931+0000","last_clean":"2024-04-24T08:18:11.937931+0000","last_became_active":"2024-04-24T08:18:09.950701+0000","last_became_peered":"2024-04-24T08:18:09.950701+0000","last_unstale":"2024-04-24T08:18:11.937931+0000","last_undegraded":"2024-04-24T08:18:11.937931+0000","last_fullsized":"2024-04-24T08:18:11.937931+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:01:35.169277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425278+0000","last_change":"2024-04-24T08:18:10.951009+0000","last_active":"2024-04-24T08:18:12.425278+0000","last_peered":"2024-04-24T08:18:12.425278+0000","last_clean":"2024-04-24T08:18:12.425278+0000","last_became_active":"2024-04-24T08:18:10.950700+0000","last_became_peered":"2024-04-24T08:18:10.950700+0000","last_unstale":"2024-04-24T08:18:12.425278+0000","last_undegraded":"2024-04-24T08:18:12.425278+0000","last_fullsized":"2024-04-24T08:18:12.425278+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:26.192324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937867+0000","last_change":"2024-04-24T08:18:09.956430+0000","last_active":"2024-04-24T08:18:11.937867+0000","last_peered":"2024-04-24T08:18:11.937867+0000","last_clean":"2024-04-24T08:18:11.937867+0000","last_became_active":"2024-04-24T08:18:09.955296+0000","last_became_peered":"2024-04-24T08:18:09.955296+0000","last_unstale":"2024-04-24T08:18:11.937867+0000","last_undegraded":"2024-04-24T08:18:11.937867+0000","last_fullsized":"2024-04-24T08:18:11.937867+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:07:36.574089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425983+0000","last_change":"2024-04-24T08:18:10.953024+0000","last_active":"2024-04-24T08:18:12.425983+0000","last_peered":"2024-04-24T08:18:12.425983+0000","last_clean":"2024-04-24T08:18:12.425983+0000","last_became_active":"2024-04-24T08:18:10.952717+0000","last_became_peered":"2024-04-24T08:18:10.952717+0000","last_unstale":"2024-04-24T08:18:12.425983+0000","last_undegraded":"2024-04-24T08:18:12.425983+0000","last_fullsized":"2024-04-24T08:18:12.425983+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:03:40.778523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.942689+0000","last_change":"2024-04-24T08:18:09.957070+0000","last_active":"2024-04-24T08:18:11.942689+0000","last_peered":"2024-04-24T08:18:11.942689+0000","last_clean":"2024-04-24T08:18:11.942689+0000","last_became_active":"2024-04-24T08:18:09.955906+0000","last_became_peered":"2024-04-24T08:18:09.955906+0000","last_unstale":"2024-04-24T08:18:11.942689+0000","last_undegraded":"2024-04-24T08:18:11.942689+0000","last_fullsized":"2024-04-24T08:18:11.942689+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:16:16.444932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423429+0000","last_change":"2024-04-24T08:18:10.951718+0000","last_active":"2024-04-24T08:18:12.423429+0000","last_peered":"2024-04-24T08:18:12.423429+0000","last_clean":"2024-04-24T08:18:12.423429+0000","last_became_active":"2024-04-24T08:18:10.951451+0000","last_became_peered":"2024-04-24T08:18:10.951451+0000","last_unstale":"2024-04-24T08:18:12.423429+0000","last_undegraded":"2024-04-24T08:18:12.423429+0000","last_fullsized":"2024-04-24T08:18:12.423429+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:55:47.186547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.942174+0000","last_change":"2024-04-24T08:18:09.948999+0000","last_active":"2024-04-24T08:18:11.942174+0000","last_peered":"2024-04-24T08:18:11.942174+0000","last_clean":"2024-04-24T08:18:11.942174+0000","last_became_active":"2024-04-24T08:18:09.948659+0000","last_became_peered":"2024-04-24T08:18:09.948659+0000","last_unstale":"2024-04-24T08:18:11.942174+0000","last_undegraded":"2024-04-24T08:18:11.942174+0000","last_fullsized":"2024-04-24T08:18:11.942174+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:48:14.986504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423481+0000","last_change":"2024-04-24T08:18:10.967340+0000","last_active":"2024-04-24T08:18:12.423481+0000","last_peered":"2024-04-24T08:18:12.423481+0000","last_clean":"2024-04-24T08:18:12.423481+0000","last_became_active":"2024-04-24T08:18:10.967009+0000","last_became_peered":"2024-04-24T08:18:10.967009+0000","last_unstale":"2024-04-24T08:18:12.423481+0000","last_undegraded":"2024-04-24T08:18:12.423481+0000","last_fullsized":"2024-04-24T08:18:12.423481+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:52:05.222065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.938034+0000","last_change":"2024-04-24T08:18:09.952806+0000","last_active":"2024-04-24T08:18:11.938034+0000","last_peered":"2024-04-24T08:18:11.938034+0000","last_clean":"2024-04-24T08:18:11.938034+0000","last_became_active":"2024-04-24T08:18:09.952532+0000","last_became_peered":"2024-04-24T08:18:09.952532+0000","last_unstale":"2024-04-24T08:18:11.938034+0000","last_undegraded":"2024-04-24T08:18:11.938034+0000","last_fullsized":"2024-04-24T08:18:11.938034+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:55:47.761337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937999+0000","last_change":"2024-04-24T08:18:10.959066+0000","last_active":"2024-04-24T08:18:11.937999+0000","last_peered":"2024-04-24T08:18:11.937999+0000","last_clean":"2024-04-24T08:18:11.937999+0000","last_became_active":"2024-04-24T08:18:10.958765+0000","last_became_peered":"2024-04-24T08:18:10.958765+0000","last_unstale":"2024-04-24T08:18:11.937999+0000","last_undegraded":"2024-04-24T08:18:11.937999+0000","last_fullsized":"2024-04-24T08:18:11.937999+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:25.063819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.942474+0000","last_change":"2024-04-24T08:18:09.953042+0000","last_active":"2024-04-24T08:18:11.942474+0000","last_peered":"2024-04-24T08:18:11.942474+0000","last_clean":"2024-04-24T08:18:11.942474+0000","last_became_active":"2024-04-24T08:18:09.952616+0000","last_became_peered":"2024-04-24T08:18:09.952616+0000","last_unstale":"2024-04-24T08:18:11.942474+0000","last_undegraded":"2024-04-24T08:18:11.942474+0000","last_fullsized":"2024-04-24T08:18:11.942474+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:53.330528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424457+0000","last_change":"2024-04-24T08:18:10.952705+0000","last_active":"2024-04-24T08:18:12.424457+0000","last_peered":"2024-04-24T08:18:12.424457+0000","last_clean":"2024-04-24T08:18:12.424457+0000","last_became_active":"2024-04-24T08:18:10.952432+0000","last_became_peered":"2024-04-24T08:18:10.952432+0000","last_unstale":"2024-04-24T08:18:12.424457+0000","last_undegraded":"2024-04-24T08:18:12.424457+0000","last_fullsized":"2024-04-24T08:18:12.424457+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:43:56.909589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"97'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.959036+0000","last_change":"2024-04-24T08:18:09.956418+0000","last_active":"2024-04-24T08:18:12.959036+0000","last_peered":"2024-04-24T08:18:12.959036+0000","last_clean":"2024-04-24T08:18:12.959036+0000","last_became_active":"2024-04-24T08:18:09.956151+0000","last_became_peered":"2024-04-24T08:18:09.956151+0000","last_unstale":"2024-04-24T08:18:12.959036+0000","last_undegraded":"2024-04-24T08:18:12.959036+0000","last_fullsized":"2024-04-24T08:18:12.959036+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:53:17.173422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423400+0000","last_change":"2024-04-24T08:18:10.968128+0000","last_active":"2024-04-24T08:18:12.423400+0000","last_peered":"2024-04-24T08:18:12.423400+0000","last_clean":"2024-04-24T08:18:12.423400+0000","last_became_active":"2024-04-24T08:18:10.967679+0000","last_became_peered":"2024-04-24T08:18:10.967679+0000","last_unstale":"2024-04-24T08:18:12.423400+0000","last_undegraded":"2024-04-24T08:18:12.423400+0000","last_fullsized":"2024-04-24T08:18:12.423400+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:04:56.555360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.938183+0000","last_change":"2024-04-24T08:18:09.949493+0000","last_active":"2024-04-24T08:18:11.938183+0000","last_peered":"2024-04-24T08:18:11.938183+0000","last_clean":"2024-04-24T08:18:11.938183+0000","last_became_active":"2024-04-24T08:18:09.949230+0000","last_became_peered":"2024-04-24T08:18:09.949230+0000","last_unstale":"2024-04-24T08:18:11.938183+0000","last_undegraded":"2024-04-24T08:18:11.938183+0000","last_fullsized":"2024-04-24T08:18:11.938183+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:36.219033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424570+0000","last_change":"2024-04-24T08:18:10.974447+0000","last_active":"2024-04-24T08:18:12.424570+0000","last_peered":"2024-04-24T08:18:12.424570+0000","last_clean":"2024-04-24T08:18:12.424570+0000","last_became_active":"2024-04-24T08:18:10.974172+0000","last_became_peered":"2024-04-24T08:18:10.974172+0000","last_unstale":"2024-04-24T08:18:12.424570+0000","last_undegraded":"2024-04-24T08:18:12.424570+0000","last_fullsized":"2024-04-24T08:18:12.424570+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:55:49.106660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425253+0000","last_change":"2024-04-24T08:18:09.959792+0000","last_active":"2024-04-24T08:18:12.425253+0000","last_peered":"2024-04-24T08:18:12.425253+0000","last_clean":"2024-04-24T08:18:12.425253+0000","last_became_active":"2024-04-24T08:18:09.959379+0000","last_became_peered":"2024-04-24T08:18:09.959379+0000","last_unstale":"2024-04-24T08:18:12.425253+0000","last_undegraded":"2024-04-24T08:18:12.425253+0000","last_fullsized":"2024-04-24T08:18:12.425253+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:50:20.100627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425534+0000","last_change":"2024-04-24T08:18:10.967312+0000","last_active":"2024-04-24T08:18:12.425534+0000","last_peered":"2024-04-24T08:18:12.425534+0000","last_clean":"2024-04-24T08:18:12.425534+0000","last_became_active":"2024-04-24T08:18:10.966992+0000","last_became_peered":"2024-04-24T08:18:10.966992+0000","last_unstale":"2024-04-24T08:18:12.425534+0000","last_undegraded":"2024-04-24T08:18:12.425534+0000","last_fullsized":"2024-04-24T08:18:12.425534+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:59:15.762308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937174+0000","last_change":"2024-04-24T08:18:09.948078+0000","last_active":"2024-04-24T08:18:11.937174+0000","last_peered":"2024-04-24T08:18:11.937174+0000","last_clean":"2024-04-24T08:18:11.937174+0000","last_became_active":"2024-04-24T08:18:09.947776+0000","last_became_peered":"2024-04-24T08:18:09.947776+0000","last_unstale":"2024-04-24T08:18:11.937174+0000","last_undegraded":"2024-04-24T08:18:11.937174+0000","last_fullsized":"2024-04-24T08:18:11.937174+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:41:02.886416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.944978+0000","last_change":"2024-04-24T08:18:10.964531+0000","last_active":"2024-04-24T08:18:11.944978+0000","last_peered":"2024-04-24T08:18:11.944978+0000","last_clean":"2024-04-24T08:18:11.944978+0000","last_became_active":"2024-04-24T08:18:10.964207+0000","last_became_peered":"2024-04-24T08:18:10.964207+0000","last_unstale":"2024-04-24T08:18:11.944978+0000","last_undegraded":"2024-04-24T08:18:11.944978+0000","last_fullsized":"2024-04-24T08:18:11.944978+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:24.288637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423452+0000","last_change":"2024-04-24T08:18:09.960963+0000","last_active":"2024-04-24T08:18:12.423452+0000","last_peered":"2024-04-24T08:18:12.423452+0000","last_clean":"2024-04-24T08:18:12.423452+0000","last_became_active":"2024-04-24T08:18:09.960294+0000","last_became_peered":"2024-04-24T08:18:09.960294+0000","last_unstale":"2024-04-24T08:18:12.423452+0000","last_undegraded":"2024-04-24T08:18:12.423452+0000","last_fullsized":"2024-04-24T08:18:12.423452+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:41:02.886416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424054+0000","last_change":"2024-04-24T08:18:10.966570+0000","last_active":"2024-04-24T08:18:12.424054+0000","last_peered":"2024-04-24T08:18:12.424054+0000","last_clean":"2024-04-24T08:18:12.424054+0000","last_became_active":"2024-04-24T08:18:10.966281+0000","last_became_peered":"2024-04-24T08:18:10.966281+0000","last_unstale":"2024-04-24T08:18:12.424054+0000","last_undegraded":"2024-04-24T08:18:12.424054+0000","last_fullsized":"2024-04-24T08:18:12.424054+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:38:40.600148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423124+0000","last_change":"2024-04-24T08:18:09.951310+0000","last_active":"2024-04-24T08:18:12.423124+0000","last_peered":"2024-04-24T08:18:12.423124+0000","last_clean":"2024-04-24T08:18:12.423124+0000","last_became_active":"2024-04-24T08:18:09.951031+0000","last_became_peered":"2024-04-24T08:18:09.951031+0000","last_unstale":"2024-04-24T08:18:12.423124+0000","last_undegraded":"2024-04-24T08:18:12.423124+0000","last_fullsized":"2024-04-24T08:18:12.423124+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:14:26.159337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423884+0000","last_change":"2024-04-24T08:18:10.970908+0000","last_active":"2024-04-24T08:18:12.423884+0000","last_peered":"2024-04-24T08:18:12.423884+0000","last_clean":"2024-04-24T08:18:12.423884+0000","last_became_active":"2024-04-24T08:18:10.970647+0000","last_became_peered":"2024-04-24T08:18:10.970647+0000","last_unstale":"2024-04-24T08:18:12.423884+0000","last_undegraded":"2024-04-24T08:18:12.423884+0000","last_fullsized":"2024-04-24T08:18:12.423884+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:13:58.550617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424317+0000","last_change":"2024-04-24T08:18:09.946342+0000","last_active":"2024-04-24T08:18:12.424317+0000","last_peered":"2024-04-24T08:18:12.424317+0000","last_clean":"2024-04-24T08:18:12.424317+0000","last_became_active":"2024-04-24T08:18:09.946059+0000","last_became_peered":"2024-04-24T08:18:09.946059+0000","last_unstale":"2024-04-24T08:18:12.424317+0000","last_undegraded":"2024-04-24T08:18:12.424317+0000","last_fullsized":"2024-04-24T08:18:12.424317+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:38:03.998681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426165+0000","last_change":"2024-04-24T08:18:10.960855+0000","last_active":"2024-04-24T08:18:12.426165+0000","last_peered":"2024-04-24T08:18:12.426165+0000","last_clean":"2024-04-24T08:18:12.426165+0000","last_became_active":"2024-04-24T08:18:10.960521+0000","last_became_peered":"2024-04-24T08:18:10.960521+0000","last_unstale":"2024-04-24T08:18:12.426165+0000","last_undegraded":"2024-04-24T08:18:12.426165+0000","last_fullsized":"2024-04-24T08:18:12.426165+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:12:42.246156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"96'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424785+0000","last_change":"2024-04-24T08:18:09.960411+0000","last_active":"2024-04-24T08:18:12.424785+0000","last_peered":"2024-04-24T08:18:12.424785+0000","last_clean":"2024-04-24T08:18:12.424785+0000","last_became_active":"2024-04-24T08:18:09.960110+0000","last_became_peered":"2024-04-24T08:18:09.960110+0000","last_unstale":"2024-04-24T08:18:12.424785+0000","last_undegraded":"2024-04-24T08:18:12.424785+0000","last_fullsized":"2024-04-24T08:18:12.424785+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:21:52.773703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":314,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425492+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:18:12.425492+0000","last_peered":"2024-04-24T08:18:12.425492+0000","last_clean":"2024-04-24T08:18:12.425492+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:18:12.425492+0000","last_undegraded":"2024-04-24T08:18:12.425492+0000","last_fullsized":"2024-04-24T08:18:12.425492+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574974,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1802240,"data_stored":1109822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574974,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1208320,"data_stored":519994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574972,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32356,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339484,"statfs":{"total":96636764160,"available":96603631616,"internally_reserved":0,"allocated":1204224,"data_stored":519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574971,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1220608,"data_stored":521078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574972,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1216512,"data_stored":520976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574973,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1204224,"data_stored":519376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32356,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339484,"statfs":{"total":96636764160,"available":96603631616,"internally_reserved":0,"allocated":1200128,"data_stored":519342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574972,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32936,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338904,"statfs":{"total":96636764160,"available":96603037696,"internally_reserved":0,"allocated":1798144,"data_stored":1110292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:18:15.389 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:18:15.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.631+0000 7f9935400640 1 Processor -- start 2024-04-24T08:18:15.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.631+0000 7f9935400640 1 -- start start 2024-04-24T08:18:15.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.632+0000 7f9935400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f993015b490 0x7f99300b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.632+0000 7f9935400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99300d02f0 0x7f993006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.632+0000 7f9935400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99300b46e0 con 0x7f993006f2d0 2024-04-24T08:18:15.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.633+0000 7f9935400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f993006fc70 con 0x7f993015b490 2024-04-24T08:18:15.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.633+0000 7f9935400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99300d0960 con 0x7f99300d02f0 2024-04-24T08:18:15.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.633+0000 7f9925e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f993006f2d0 0x7f99300cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52004/0 (socket says 172.21.15.69:52004) 2024-04-24T08:18:15.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.633+0000 7f9925e00640 1 -- 172.21.15.69:0/1405442673 learned_addr learned my addr 172.21.15.69:0/1405442673 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:15.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.633+0000 7f992f200640 1 --2- 172.21.15.69:0/1405442673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99300d02f0 0x7f993006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.633+0000 7f992e800640 1 --2- 172.21.15.69:0/1405442673 >> v2:172.21.15.121:3300/0 conn(0x7f993015b490 0x7f99300b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.633+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1521327007 0 0) 0x7f99300b46e0 con 0x7f993006f2d0 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.633+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9914003520 con 0x7f993006f2d0 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4036900457 0 0) 0x7f9914003520 con 0x7f993006f2d0 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99300d02f0 msgr2=0x7f993006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 --2- 172.21.15.69:0/1405442673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99300d02f0 0x7f993006f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 >> v2:172.21.15.121:3300/0 conn(0x7f993015b490 msgr2=0x7f99300b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 --2- 172.21.15.69:0/1405442673 >> v2:172.21.15.121:3300/0 conn(0x7f993015b490 0x7f99300b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99300b4980 con 0x7f993006f2d0 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f9918001670 con 0x7f993006f2d0 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 >> v1:172.21.15.69:6789/0 conn(0x7f993006f2d0 legacy=0x7f99300cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 --2- 172.21.15.69:0/1405442673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99140039b0 0x7f9914003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 --2- 172.21.15.69:0/1405442673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99300d02f0 0x7f9914004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 --2- 172.21.15.69:0/1405442673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9914004d40 0x7f99140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f993006fc70 con 0x7f99140039b0 2024-04-24T08:18:15.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f99300d0960 con 0x7f9914004d40 2024-04-24T08:18:15.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f991400dbd0 con 0x7f99300d02f0 2024-04-24T08:18:15.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.634+0000 7f992e800640 1 --2- 172.21.15.69:0/1405442673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99140039b0 0x7f9914003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.635+0000 7f992e800640 1 -- 172.21.15.69:0/1405442673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99300d02f0 msgr2=0x7f9914004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.635+0000 7f992e800640 1 --2- 172.21.15.69:0/1405442673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99300d02f0 0x7f9914004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.635+0000 7f992e800640 1 -- 172.21.15.69:0/1405442673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9914004d40 msgr2=0x7f99140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.635+0000 7f992e800640 1 --2- 172.21.15.69:0/1405442673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9914004d40 0x7f99140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.635+0000 7f992e800640 1 -- 172.21.15.69:0/1405442673 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9914003520 con 0x7f99140039b0 2024-04-24T08:18:15.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.635+0000 7f992e800640 1 --2- 172.21.15.69:0/1405442673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99140039b0 0x7f9914003dd0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f9920009ac0 tx=0x7f9914005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7299c0f3ce822f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.636+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920036070 con 0x7f99140039b0 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.636+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9920031400 con 0x7f99140039b0 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.636+0000 7f992de00640 1 -- 172.21.15.69:0/1405442673 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9920009690 con 0x7f99140039b0 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.636+0000 7f9935400640 1 -- 172.21.15.69:0/1405442673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99140039b0 msgr2=0x7f9914003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.636+0000 7f9935400640 1 --2- 172.21.15.69:0/1405442673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99140039b0 0x7f9914003dd0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f9920009ac0 tx=0x7f9914005830 comp rx=0 tx=0).stop 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.636+0000 7f992f200640 1 -- 172.21.15.69:0/1405442673 reap_dead start 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.636+0000 7f9935400640 1 -- 172.21.15.69:0/1405442673 shutdown_connections 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.636+0000 7f9935400640 1 -- 172.21.15.69:0/1405442673 >> 172.21.15.69:0/1405442673 conn(0x7f9930067d90 msgr2=0x7f993006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.636+0000 7f9935400640 1 -- 172.21.15.69:0/1405442673 shutdown_connections 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.636+0000 7f9935400640 1 -- 172.21.15.69:0/1405442673 wait complete. 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.638+0000 7f9935400640 1 Processor -- start 2024-04-24T08:18:15.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.638+0000 7f9935400640 1 -- start start 2024-04-24T08:18:15.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.639+0000 7f9935400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99140039b0 0x7f993015c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.639+0000 7f9935400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9930149990 0x7f993015df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.639+0000 7f9935400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f993015e4c0 0x7f99300b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.639+0000 7f9935400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f993015b870 con 0x7f993015e4c0 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.639+0000 7f9935400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f99300d0c40 con 0x7f99140039b0 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.639+0000 7f9935400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99300b56c0 con 0x7f9930149990 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f993015e4c0 0x7f99300b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f993015e4c0 0x7f99300b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40374/0 (socket says 172.21.15.69:40374) 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992f200640 1 -- 172.21.15.69:0/435424869 learned_addr learned my addr 172.21.15.69:0/435424869 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f9925e00640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9930149990 0x7f993015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992e800640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99140039b0 0x7f993015c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992f200640 1 -- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9930149990 msgr2=0x7f993015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992f200640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9930149990 0x7f993015df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992f200640 1 -- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99140039b0 msgr2=0x7f993015c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992f200640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99140039b0 0x7f993015c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992f200640 1 -- 172.21.15.69:0/435424869 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99301e3e50 con 0x7f993015e4c0 2024-04-24T08:18:15.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992e800640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99140039b0 0x7f993015c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:15.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.640+0000 7f992f200640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f993015e4c0 0x7f99300b3560 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f99300ccfd0 tx=0x7f992800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.641+0000 7f9927400640 1 -- 172.21.15.69:0/435424869 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9928018020 con 0x7f993015e4c0 2024-04-24T08:18:15.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.641+0000 7f9927400640 1 -- 172.21.15.69:0/435424869 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f992800bab0 con 0x7f993015e4c0 2024-04-24T08:18:15.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.641+0000 7f9927400640 1 -- 172.21.15.69:0/435424869 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9928014400 con 0x7f993015e4c0 2024-04-24T08:18:15.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.641+0000 7f9935400640 1 -- 172.21.15.69:0/435424869 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99301e40f0 con 0x7f993015e4c0 2024-04-24T08:18:15.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.641+0000 7f9935400640 1 -- 172.21.15.69:0/435424869 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99301e46b0 con 0x7f993015e4c0 2024-04-24T08:18:15.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.642+0000 7f9935400640 1 -- 172.21.15.69:0/435424869 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9930068a80 con 0x7f993015e4c0 2024-04-24T08:18:15.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.643+0000 7f9927400640 1 -- 172.21.15.69:0/435424869 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9928005d10 con 0x7f993015e4c0 2024-04-24T08:18:15.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.644+0000 7f9927400640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990c06c8f0 0x7f990c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.644+0000 7f9927400640 1 -- 172.21.15.69:0/435424869 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f992808be00 con 0x7f993015e4c0 2024-04-24T08:18:15.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.645+0000 7f992e800640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990c06c8f0 0x7f990c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.646+0000 7f992e800640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990c06c8f0 0x7f990c06edb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9920033000 tx=0x7f9920008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.649+0000 7f9927400640 1 -- 172.21.15.69:0/435424869 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9928022070 con 0x7f993015e4c0 2024-04-24T08:18:15.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.829+0000 7f9935400640 1 -- 172.21.15.69:0/435424869 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f993006bda0 con 0x7f990c06c8f0 2024-04-24T08:18:15.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.847+0000 7f9927400640 1 -- 172.21.15.69:0/435424869 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252085 (secure 0 0 0) 0x7f993006bda0 con 0x7f990c06c8f0 2024-04-24T08:18:15.851 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:15.853 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:18:15.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 -- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990c06c8f0 msgr2=0x7f990c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990c06c8f0 0x7f990c06edb0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9920033000 tx=0x7f9920008030 comp rx=0 tx=0).stop 2024-04-24T08:18:15.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 -- 172.21.15.69:0/435424869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f993015e4c0 msgr2=0x7f99300b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f993015e4c0 0x7f99300b3560 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f99300ccfd0 tx=0x7f992800b480 comp rx=0 tx=0).stop 2024-04-24T08:18:15.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 -- 172.21.15.69:0/435424869 shutdown_connections 2024-04-24T08:18:15.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f990c06c8f0 0x7f990c06edb0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f993015e4c0 0x7f99300b3560 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9930149990 0x7f993015df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 --2- 172.21.15.69:0/435424869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99140039b0 0x7f993015c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 -- 172.21.15.69:0/435424869 >> 172.21.15.69:0/435424869 conn(0x7f9930067d90 msgr2=0x7f993015a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 -- 172.21.15.69:0/435424869 shutdown_connections 2024-04-24T08:18:15.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:15.853+0000 7f9935400640 1 -- 172.21.15.69:0/435424869 wait complete. 2024-04-24T08:18:15.872 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-04-24T08:18:14.884150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260056,"kb_used_data":10600,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754714664,"statfs":{"total":773094113280,"available":772827815936,"internally_reserved":0,"allocated":10854400,"data_stored":5340292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.974995"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424088+0000","last_change":"2024-04-24T08:18:10.961565+0000","last_active":"2024-04-24T08:18:12.424088+0000","last_peered":"2024-04-24T08:18:12.424088+0000","last_clean":"2024-04-24T08:18:12.424088+0000","last_became_active":"2024-04-24T08:18:10.961283+0000","last_became_peered":"2024-04-24T08:18:10.961283+0000","last_unstale":"2024-04-24T08:18:12.424088+0000","last_undegraded":"2024-04-24T08:18:12.424088+0000","last_fullsized":"2024-04-24T08:18:12.424088+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:16:24.059564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937615+0000","last_change":"2024-04-24T08:18:10.963956+0000","last_active":"2024-04-24T08:18:11.937615+0000","last_peered":"2024-04-24T08:18:11.937615+0000","last_clean":"2024-04-24T08:18:11.937615+0000","last_became_active":"2024-04-24T08:18:10.963663+0000","last_became_peered":"2024-04-24T08:18:10.963663+0000","last_unstale":"2024-04-24T08:18:11.937615+0000","last_undegraded":"2024-04-24T08:18:11.937615+0000","last_fullsized":"2024-04-24T08:18:11.937615+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:55:25.744718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423922+0000","last_change":"2024-04-24T08:18:10.966965+0000","last_active":"2024-04-24T08:18:12.423922+0000","last_peered":"2024-04-24T08:18:12.423922+0000","last_clean":"2024-04-24T08:18:12.423922+0000","last_became_active":"2024-04-24T08:18:10.966702+0000","last_became_peered":"2024-04-24T08:18:10.966702+0000","last_unstale":"2024-04-24T08:18:12.423922+0000","last_undegraded":"2024-04-24T08:18:12.423922+0000","last_fullsized":"2024-04-24T08:18:12.423922+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:44:07.151307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424304+0000","last_change":"2024-04-24T08:18:10.945475+0000","last_active":"2024-04-24T08:18:12.424304+0000","last_peered":"2024-04-24T08:18:12.424304+0000","last_clean":"2024-04-24T08:18:12.424304+0000","last_became_active":"2024-04-24T08:18:10.944765+0000","last_became_peered":"2024-04-24T08:18:10.944765+0000","last_unstale":"2024-04-24T08:18:12.424304+0000","last_undegraded":"2024-04-24T08:18:12.424304+0000","last_fullsized":"2024-04-24T08:18:12.424304+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:24.288637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424358+0000","last_change":"2024-04-24T08:18:10.969312+0000","last_active":"2024-04-24T08:18:12.424358+0000","last_peered":"2024-04-24T08:18:12.424358+0000","last_clean":"2024-04-24T08:18:12.424358+0000","last_became_active":"2024-04-24T08:18:10.968932+0000","last_became_peered":"2024-04-24T08:18:10.968932+0000","last_unstale":"2024-04-24T08:18:12.424358+0000","last_undegraded":"2024-04-24T08:18:12.424358+0000","last_fullsized":"2024-04-24T08:18:12.424358+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:30:56.763387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424827+0000","last_change":"2024-04-24T08:18:10.971713+0000","last_active":"2024-04-24T08:18:12.424827+0000","last_peered":"2024-04-24T08:18:12.424827+0000","last_clean":"2024-04-24T08:18:12.424827+0000","last_became_active":"2024-04-24T08:18:10.971336+0000","last_became_peered":"2024-04-24T08:18:10.971336+0000","last_unstale":"2024-04-24T08:18:12.424827+0000","last_undegraded":"2024-04-24T08:18:12.424827+0000","last_fullsized":"2024-04-24T08:18:12.424827+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:55:48.779115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425957+0000","last_change":"2024-04-24T08:18:10.967624+0000","last_active":"2024-04-24T08:18:12.425957+0000","last_peered":"2024-04-24T08:18:12.425957+0000","last_clean":"2024-04-24T08:18:12.425957+0000","last_became_active":"2024-04-24T08:18:10.967364+0000","last_became_peered":"2024-04-24T08:18:10.967364+0000","last_unstale":"2024-04-24T08:18:12.425957+0000","last_undegraded":"2024-04-24T08:18:12.425957+0000","last_fullsized":"2024-04-24T08:18:12.425957+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:37.236811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937228+0000","last_change":"2024-04-24T08:18:10.968700+0000","last_active":"2024-04-24T08:18:11.937228+0000","last_peered":"2024-04-24T08:18:11.937228+0000","last_clean":"2024-04-24T08:18:11.937228+0000","last_became_active":"2024-04-24T08:18:10.968384+0000","last_became_peered":"2024-04-24T08:18:10.968384+0000","last_unstale":"2024-04-24T08:18:11.937228+0000","last_undegraded":"2024-04-24T08:18:11.937228+0000","last_fullsized":"2024-04-24T08:18:11.937228+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:13:58.550617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945751+0000","last_change":"2024-04-24T08:18:10.960428+0000","last_active":"2024-04-24T08:18:11.945751+0000","last_peered":"2024-04-24T08:18:11.945751+0000","last_clean":"2024-04-24T08:18:11.945751+0000","last_became_active":"2024-04-24T08:18:10.956831+0000","last_became_peered":"2024-04-24T08:18:10.956831+0000","last_unstale":"2024-04-24T08:18:11.945751+0000","last_undegraded":"2024-04-24T08:18:11.945751+0000","last_fullsized":"2024-04-24T08:18:11.945751+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:26:18.219342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426383+0000","last_change":"2024-04-24T08:18:10.972276+0000","last_active":"2024-04-24T08:18:12.426383+0000","last_peered":"2024-04-24T08:18:12.426383+0000","last_clean":"2024-04-24T08:18:12.426383+0000","last_became_active":"2024-04-24T08:18:10.972004+0000","last_became_peered":"2024-04-24T08:18:10.972004+0000","last_unstale":"2024-04-24T08:18:12.426383+0000","last_undegraded":"2024-04-24T08:18:12.426383+0000","last_fullsized":"2024-04-24T08:18:12.426383+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:56:22.407532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424889+0000","last_change":"2024-04-24T08:18:10.968740+0000","last_active":"2024-04-24T08:18:12.424889+0000","last_peered":"2024-04-24T08:18:12.424889+0000","last_clean":"2024-04-24T08:18:12.424889+0000","last_became_active":"2024-04-24T08:18:10.968058+0000","last_became_peered":"2024-04-24T08:18:10.968058+0000","last_unstale":"2024-04-24T08:18:12.424889+0000","last_undegraded":"2024-04-24T08:18:12.424889+0000","last_fullsized":"2024-04-24T08:18:12.424889+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:41:03.904194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945715+0000","last_change":"2024-04-24T08:18:10.964804+0000","last_active":"2024-04-24T08:18:11.945715+0000","last_peered":"2024-04-24T08:18:11.945715+0000","last_clean":"2024-04-24T08:18:11.945715+0000","last_became_active":"2024-04-24T08:18:10.964219+0000","last_became_peered":"2024-04-24T08:18:10.964219+0000","last_unstale":"2024-04-24T08:18:11.945715+0000","last_undegraded":"2024-04-24T08:18:11.945715+0000","last_fullsized":"2024-04-24T08:18:11.945715+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:31.570848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425162+0000","last_change":"2024-04-24T08:18:10.960719+0000","last_active":"2024-04-24T08:18:12.425162+0000","last_peered":"2024-04-24T08:18:12.425162+0000","last_clean":"2024-04-24T08:18:12.425162+0000","last_became_active":"2024-04-24T08:18:10.956874+0000","last_became_peered":"2024-04-24T08:18:10.956874+0000","last_unstale":"2024-04-24T08:18:12.425162+0000","last_undegraded":"2024-04-24T08:18:12.425162+0000","last_fullsized":"2024-04-24T08:18:12.425162+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:43:44.792654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424902+0000","last_change":"2024-04-24T08:18:10.963233+0000","last_active":"2024-04-24T08:18:12.424902+0000","last_peered":"2024-04-24T08:18:12.424902+0000","last_clean":"2024-04-24T08:18:12.424902+0000","last_became_active":"2024-04-24T08:18:10.962962+0000","last_became_peered":"2024-04-24T08:18:10.962962+0000","last_unstale":"2024-04-24T08:18:12.424902+0000","last_undegraded":"2024-04-24T08:18:12.424902+0000","last_fullsized":"2024-04-24T08:18:12.424902+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:55:22.703333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422863+0000","last_change":"2024-04-24T08:18:10.951541+0000","last_active":"2024-04-24T08:18:12.422863+0000","last_peered":"2024-04-24T08:18:12.422863+0000","last_clean":"2024-04-24T08:18:12.422863+0000","last_became_active":"2024-04-24T08:18:10.951239+0000","last_became_peered":"2024-04-24T08:18:10.951239+0000","last_unstale":"2024-04-24T08:18:12.422863+0000","last_undegraded":"2024-04-24T08:18:12.422863+0000","last_fullsized":"2024-04-24T08:18:12.422863+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:00:07.240078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423928+0000","last_change":"2024-04-24T08:18:10.966894+0000","last_active":"2024-04-24T08:18:12.423928+0000","last_peered":"2024-04-24T08:18:12.423928+0000","last_clean":"2024-04-24T08:18:12.423928+0000","last_became_active":"2024-04-24T08:18:10.966628+0000","last_became_peered":"2024-04-24T08:18:10.966628+0000","last_unstale":"2024-04-24T08:18:12.423928+0000","last_undegraded":"2024-04-24T08:18:12.423928+0000","last_fullsized":"2024-04-24T08:18:12.423928+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:01:36.187055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.938215+0000","last_change":"2024-04-24T08:18:10.949255+0000","last_active":"2024-04-24T08:18:11.938215+0000","last_peered":"2024-04-24T08:18:11.938215+0000","last_clean":"2024-04-24T08:18:11.938215+0000","last_became_active":"2024-04-24T08:18:10.948925+0000","last_became_peered":"2024-04-24T08:18:10.948925+0000","last_unstale":"2024-04-24T08:18:11.938215+0000","last_undegraded":"2024-04-24T08:18:11.938215+0000","last_fullsized":"2024-04-24T08:18:11.938215+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:14:30.218520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426760+0000","last_change":"2024-04-24T08:18:10.960378+0000","last_active":"2024-04-24T08:18:12.426760+0000","last_peered":"2024-04-24T08:18:12.426760+0000","last_clean":"2024-04-24T08:18:12.426760+0000","last_became_active":"2024-04-24T08:18:10.960054+0000","last_became_peered":"2024-04-24T08:18:10.960054+0000","last_unstale":"2024-04-24T08:18:12.426760+0000","last_undegraded":"2024-04-24T08:18:12.426760+0000","last_fullsized":"2024-04-24T08:18:12.426760+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:39:31.570848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426046+0000","last_change":"2024-04-24T08:18:10.972125+0000","last_active":"2024-04-24T08:18:12.426046+0000","last_peered":"2024-04-24T08:18:12.426046+0000","last_clean":"2024-04-24T08:18:12.426046+0000","last_became_active":"2024-04-24T08:18:10.969463+0000","last_became_peered":"2024-04-24T08:18:10.969463+0000","last_unstale":"2024-04-24T08:18:12.426046+0000","last_undegraded":"2024-04-24T08:18:12.426046+0000","last_fullsized":"2024-04-24T08:18:12.426046+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:59:40.923189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945749+0000","last_change":"2024-04-24T08:18:10.955613+0000","last_active":"2024-04-24T08:18:11.945749+0000","last_peered":"2024-04-24T08:18:11.945749+0000","last_clean":"2024-04-24T08:18:11.945749+0000","last_became_active":"2024-04-24T08:18:10.955239+0000","last_became_peered":"2024-04-24T08:18:10.955239+0000","last_unstale":"2024-04-24T08:18:11.945749+0000","last_undegraded":"2024-04-24T08:18:11.945749+0000","last_fullsized":"2024-04-24T08:18:11.945749+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:44:52.657162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425707+0000","last_change":"2024-04-24T08:18:10.964804+0000","last_active":"2024-04-24T08:18:12.425707+0000","last_peered":"2024-04-24T08:18:12.425707+0000","last_clean":"2024-04-24T08:18:12.425707+0000","last_became_active":"2024-04-24T08:18:10.964543+0000","last_became_peered":"2024-04-24T08:18:10.964543+0000","last_unstale":"2024-04-24T08:18:12.425707+0000","last_undegraded":"2024-04-24T08:18:12.425707+0000","last_fullsized":"2024-04-24T08:18:12.425707+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:30.856317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937249+0000","last_change":"2024-04-24T08:18:10.964957+0000","last_active":"2024-04-24T08:18:11.937249+0000","last_peered":"2024-04-24T08:18:11.937249+0000","last_clean":"2024-04-24T08:18:11.937249+0000","last_became_active":"2024-04-24T08:18:10.963117+0000","last_became_peered":"2024-04-24T08:18:10.963117+0000","last_unstale":"2024-04-24T08:18:11.937249+0000","last_undegraded":"2024-04-24T08:18:11.937249+0000","last_fullsized":"2024-04-24T08:18:11.937249+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:30:56.763387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937275+0000","last_change":"2024-04-24T08:18:10.969508+0000","last_active":"2024-04-24T08:18:11.937275+0000","last_peered":"2024-04-24T08:18:11.937275+0000","last_clean":"2024-04-24T08:18:11.937275+0000","last_became_active":"2024-04-24T08:18:10.967363+0000","last_became_peered":"2024-04-24T08:18:10.967363+0000","last_unstale":"2024-04-24T08:18:11.937275+0000","last_undegraded":"2024-04-24T08:18:11.937275+0000","last_fullsized":"2024-04-24T08:18:11.937275+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:55:49.106660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424975+0000","last_change":"2024-04-24T08:18:10.971483+0000","last_active":"2024-04-24T08:18:12.424975+0000","last_peered":"2024-04-24T08:18:12.424975+0000","last_clean":"2024-04-24T08:18:12.424975+0000","last_became_active":"2024-04-24T08:18:10.969672+0000","last_became_peered":"2024-04-24T08:18:10.969672+0000","last_unstale":"2024-04-24T08:18:12.424975+0000","last_undegraded":"2024-04-24T08:18:12.424975+0000","last_fullsized":"2024-04-24T08:18:12.424975+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:50:21.118405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425844+0000","last_change":"2024-04-24T08:18:10.975602+0000","last_active":"2024-04-24T08:18:12.425844+0000","last_peered":"2024-04-24T08:18:12.425844+0000","last_clean":"2024-04-24T08:18:12.425844+0000","last_became_active":"2024-04-24T08:18:10.975265+0000","last_became_peered":"2024-04-24T08:18:10.975265+0000","last_unstale":"2024-04-24T08:18:12.425844+0000","last_undegraded":"2024-04-24T08:18:12.425844+0000","last_fullsized":"2024-04-24T08:18:12.425844+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:25.063819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937751+0000","last_change":"2024-04-24T08:18:10.942883+0000","last_active":"2024-04-24T08:18:11.937751+0000","last_peered":"2024-04-24T08:18:11.937751+0000","last_clean":"2024-04-24T08:18:11.937751+0000","last_became_active":"2024-04-24T08:18:10.942585+0000","last_became_peered":"2024-04-24T08:18:10.942585+0000","last_unstale":"2024-04-24T08:18:11.937751+0000","last_undegraded":"2024-04-24T08:18:11.937751+0000","last_fullsized":"2024-04-24T08:18:11.937751+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:03:24.238839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425303+0000","last_change":"2024-04-24T08:18:10.969281+0000","last_active":"2024-04-24T08:18:12.425303+0000","last_peered":"2024-04-24T08:18:12.425303+0000","last_clean":"2024-04-24T08:18:12.425303+0000","last_became_active":"2024-04-24T08:18:10.968943+0000","last_became_peered":"2024-04-24T08:18:10.968943+0000","last_unstale":"2024-04-24T08:18:12.425303+0000","last_undegraded":"2024-04-24T08:18:12.425303+0000","last_fullsized":"2024-04-24T08:18:12.425303+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:00:07.240078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425070+0000","last_change":"2024-04-24T08:18:10.974369+0000","last_active":"2024-04-24T08:18:12.425070+0000","last_peered":"2024-04-24T08:18:12.425070+0000","last_clean":"2024-04-24T08:18:12.425070+0000","last_became_active":"2024-04-24T08:18:10.973666+0000","last_became_peered":"2024-04-24T08:18:10.973666+0000","last_unstale":"2024-04-24T08:18:12.425070+0000","last_undegraded":"2024-04-24T08:18:12.425070+0000","last_fullsized":"2024-04-24T08:18:12.425070+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:55:25.744718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937076+0000","last_change":"2024-04-24T08:18:10.943657+0000","last_active":"2024-04-24T08:18:11.937076+0000","last_peered":"2024-04-24T08:18:11.937076+0000","last_clean":"2024-04-24T08:18:11.937076+0000","last_became_active":"2024-04-24T08:18:10.943355+0000","last_became_peered":"2024-04-24T08:18:10.943355+0000","last_unstale":"2024-04-24T08:18:11.937076+0000","last_undegraded":"2024-04-24T08:18:11.937076+0000","last_fullsized":"2024-04-24T08:18:11.937076+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:52:05.222065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425135+0000","last_change":"2024-04-24T08:18:10.966574+0000","last_active":"2024-04-24T08:18:12.425135+0000","last_peered":"2024-04-24T08:18:12.425135+0000","last_clean":"2024-04-24T08:18:12.425135+0000","last_became_active":"2024-04-24T08:18:10.965002+0000","last_became_peered":"2024-04-24T08:18:10.965002+0000","last_unstale":"2024-04-24T08:18:12.425135+0000","last_undegraded":"2024-04-24T08:18:12.425135+0000","last_fullsized":"2024-04-24T08:18:12.425135+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:17.068910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423160+0000","last_change":"2024-04-24T08:18:10.968123+0000","last_active":"2024-04-24T08:18:12.423160+0000","last_peered":"2024-04-24T08:18:12.423160+0000","last_clean":"2024-04-24T08:18:12.423160+0000","last_became_active":"2024-04-24T08:18:10.967845+0000","last_became_peered":"2024-04-24T08:18:10.967845+0000","last_unstale":"2024-04-24T08:18:12.423160+0000","last_undegraded":"2024-04-24T08:18:12.423160+0000","last_fullsized":"2024-04-24T08:18:12.423160+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:53:18.191200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422098+0000","last_change":"2024-04-24T08:18:10.948105+0000","last_active":"2024-04-24T08:18:12.422098+0000","last_peered":"2024-04-24T08:18:12.422098+0000","last_clean":"2024-04-24T08:18:12.422098+0000","last_became_active":"2024-04-24T08:18:10.946467+0000","last_became_peered":"2024-04-24T08:18:10.946467+0000","last_unstale":"2024-04-24T08:18:12.422098+0000","last_undegraded":"2024-04-24T08:18:12.422098+0000","last_fullsized":"2024-04-24T08:18:12.422098+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:54.348306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425985+0000","last_change":"2024-04-24T08:18:10.965880+0000","last_active":"2024-04-24T08:18:12.425985+0000","last_peered":"2024-04-24T08:18:12.425985+0000","last_clean":"2024-04-24T08:18:12.425985+0000","last_became_active":"2024-04-24T08:18:10.965449+0000","last_became_peered":"2024-04-24T08:18:10.965449+0000","last_unstale":"2024-04-24T08:18:12.425985+0000","last_undegraded":"2024-04-24T08:18:12.425985+0000","last_fullsized":"2024-04-24T08:18:12.425985+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:06:21.094134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"96'4","reported_seq":18,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937821+0000","last_change":"2024-04-24T08:18:09.961054+0000","last_active":"2024-04-24T08:18:11.937821+0000","last_peered":"2024-04-24T08:18:11.937821+0000","last_clean":"2024-04-24T08:18:11.937821+0000","last_became_active":"2024-04-24T08:18:09.960763+0000","last_became_peered":"2024-04-24T08:18:09.960763+0000","last_unstale":"2024-04-24T08:18:11.937821+0000","last_undegraded":"2024-04-24T08:18:11.937821+0000","last_fullsized":"2024-04-24T08:18:11.937821+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:00:06.222300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.427165+0000","last_change":"2024-04-24T08:18:10.966892+0000","last_active":"2024-04-24T08:18:12.427165+0000","last_peered":"2024-04-24T08:18:12.427165+0000","last_clean":"2024-04-24T08:18:12.427165+0000","last_became_active":"2024-04-24T08:18:10.966578+0000","last_became_peered":"2024-04-24T08:18:10.966578+0000","last_unstale":"2024-04-24T08:18:12.427165+0000","last_undegraded":"2024-04-24T08:18:12.427165+0000","last_fullsized":"2024-04-24T08:18:12.427165+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:30:53.924950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"96'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426397+0000","last_change":"2024-04-24T08:18:09.962285+0000","last_active":"2024-04-24T08:18:12.426397+0000","last_peered":"2024-04-24T08:18:12.426397+0000","last_clean":"2024-04-24T08:18:12.426397+0000","last_became_active":"2024-04-24T08:18:09.961972+0000","last_became_peered":"2024-04-24T08:18:09.961972+0000","last_unstale":"2024-04-24T08:18:12.426397+0000","last_undegraded":"2024-04-24T08:18:12.426397+0000","last_fullsized":"2024-04-24T08:18:12.426397+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:05:58.842655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422990+0000","last_change":"2024-04-24T08:18:10.961857+0000","last_active":"2024-04-24T08:18:12.422990+0000","last_peered":"2024-04-24T08:18:12.422990+0000","last_clean":"2024-04-24T08:18:12.422990+0000","last_became_active":"2024-04-24T08:18:10.961544+0000","last_became_peered":"2024-04-24T08:18:10.961544+0000","last_unstale":"2024-04-24T08:18:12.422990+0000","last_undegraded":"2024-04-24T08:18:12.422990+0000","last_fullsized":"2024-04-24T08:18:12.422990+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:19:17.771354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.938212+0000","last_change":"2024-04-24T08:18:09.956166+0000","last_active":"2024-04-24T08:18:11.938212+0000","last_peered":"2024-04-24T08:18:11.938212+0000","last_clean":"2024-04-24T08:18:11.938212+0000","last_became_active":"2024-04-24T08:18:09.955195+0000","last_became_peered":"2024-04-24T08:18:09.955195+0000","last_unstale":"2024-04-24T08:18:11.938212+0000","last_undegraded":"2024-04-24T08:18:11.938212+0000","last_fullsized":"2024-04-24T08:18:11.938212+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:43:06.185599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945676+0000","last_change":"2024-04-24T08:18:10.970331+0000","last_active":"2024-04-24T08:18:11.945676+0000","last_peered":"2024-04-24T08:18:11.945676+0000","last_clean":"2024-04-24T08:18:11.945676+0000","last_became_active":"2024-04-24T08:18:10.968722+0000","last_became_peered":"2024-04-24T08:18:10.968722+0000","last_unstale":"2024-04-24T08:18:11.945676+0000","last_undegraded":"2024-04-24T08:18:11.945676+0000","last_fullsized":"2024-04-24T08:18:11.945676+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:43:44.792654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937768+0000","last_change":"2024-04-24T08:18:09.951148+0000","last_active":"2024-04-24T08:18:11.937768+0000","last_peered":"2024-04-24T08:18:11.937768+0000","last_clean":"2024-04-24T08:18:11.937768+0000","last_became_active":"2024-04-24T08:18:09.950870+0000","last_became_peered":"2024-04-24T08:18:09.950870+0000","last_unstale":"2024-04-24T08:18:11.937768+0000","last_undegraded":"2024-04-24T08:18:11.937768+0000","last_fullsized":"2024-04-24T08:18:11.937768+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:55:32.014758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.942592+0000","last_change":"2024-04-24T08:18:10.952307+0000","last_active":"2024-04-24T08:18:11.942592+0000","last_peered":"2024-04-24T08:18:11.942592+0000","last_clean":"2024-04-24T08:18:11.942592+0000","last_became_active":"2024-04-24T08:18:10.951464+0000","last_became_peered":"2024-04-24T08:18:10.951464+0000","last_unstale":"2024-04-24T08:18:11.942592+0000","last_undegraded":"2024-04-24T08:18:11.942592+0000","last_fullsized":"2024-04-24T08:18:11.942592+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:44:07.151307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"96'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423279+0000","last_change":"2024-04-24T08:18:09.960759+0000","last_active":"2024-04-24T08:18:12.423279+0000","last_peered":"2024-04-24T08:18:12.423279+0000","last_clean":"2024-04-24T08:18:12.423279+0000","last_became_active":"2024-04-24T08:18:09.960406+0000","last_became_peered":"2024-04-24T08:18:09.960406+0000","last_unstale":"2024-04-24T08:18:12.423279+0000","last_undegraded":"2024-04-24T08:18:12.423279+0000","last_fullsized":"2024-04-24T08:18:12.423279+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:03:39.760745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424914+0000","last_change":"2024-04-24T08:18:10.968582+0000","last_active":"2024-04-24T08:18:12.424914+0000","last_peered":"2024-04-24T08:18:12.424914+0000","last_clean":"2024-04-24T08:18:12.424914+0000","last_became_active":"2024-04-24T08:18:10.966087+0000","last_became_peered":"2024-04-24T08:18:10.966087+0000","last_unstale":"2024-04-24T08:18:12.424914+0000","last_undegraded":"2024-04-24T08:18:12.424914+0000","last_fullsized":"2024-04-24T08:18:12.424914+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:43:07.203377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"96'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937575+0000","last_change":"2024-04-24T08:18:09.956966+0000","last_active":"2024-04-24T08:18:11.937575+0000","last_peered":"2024-04-24T08:18:11.937575+0000","last_clean":"2024-04-24T08:18:11.937575+0000","last_became_active":"2024-04-24T08:18:09.955343+0000","last_became_peered":"2024-04-24T08:18:09.955343+0000","last_unstale":"2024-04-24T08:18:11.937575+0000","last_undegraded":"2024-04-24T08:18:11.937575+0000","last_fullsized":"2024-04-24T08:18:11.937575+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:50:20.100627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423776+0000","last_change":"2024-04-24T08:18:10.947122+0000","last_active":"2024-04-24T08:18:12.423776+0000","last_peered":"2024-04-24T08:18:12.423776+0000","last_clean":"2024-04-24T08:18:12.423776+0000","last_became_active":"2024-04-24T08:18:10.945155+0000","last_became_peered":"2024-04-24T08:18:10.945155+0000","last_unstale":"2024-04-24T08:18:12.423776+0000","last_undegraded":"2024-04-24T08:18:12.423776+0000","last_fullsized":"2024-04-24T08:18:12.423776+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:37.236811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424175+0000","last_change":"2024-04-24T08:18:09.963877+0000","last_active":"2024-04-24T08:18:12.424175+0000","last_peered":"2024-04-24T08:18:12.424175+0000","last_clean":"2024-04-24T08:18:12.424175+0000","last_became_active":"2024-04-24T08:18:09.963493+0000","last_became_peered":"2024-04-24T08:18:09.963493+0000","last_unstale":"2024-04-24T08:18:12.424175+0000","last_undegraded":"2024-04-24T08:18:12.424175+0000","last_fullsized":"2024-04-24T08:18:12.424175+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:43:06.185599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.944954+0000","last_change":"2024-04-24T08:18:10.975197+0000","last_active":"2024-04-24T08:18:11.944954+0000","last_peered":"2024-04-24T08:18:11.944954+0000","last_clean":"2024-04-24T08:18:11.944954+0000","last_became_active":"2024-04-24T08:18:10.971948+0000","last_became_peered":"2024-04-24T08:18:10.971948+0000","last_unstale":"2024-04-24T08:18:11.944954+0000","last_undegraded":"2024-04-24T08:18:11.944954+0000","last_fullsized":"2024-04-24T08:18:11.944954+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:47.129428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422067+0000","last_change":"2024-04-24T08:18:09.955900+0000","last_active":"2024-04-24T08:18:12.422067+0000","last_peered":"2024-04-24T08:18:12.422067+0000","last_clean":"2024-04-24T08:18:12.422067+0000","last_became_active":"2024-04-24T08:18:09.954789+0000","last_became_peered":"2024-04-24T08:18:09.954789+0000","last_unstale":"2024-04-24T08:18:12.422067+0000","last_undegraded":"2024-04-24T08:18:12.422067+0000","last_fullsized":"2024-04-24T08:18:12.422067+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:32:29.838539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.938225+0000","last_change":"2024-04-24T08:18:10.958985+0000","last_active":"2024-04-24T08:18:11.938225+0000","last_peered":"2024-04-24T08:18:11.938225+0000","last_clean":"2024-04-24T08:18:11.938225+0000","last_became_active":"2024-04-24T08:18:10.958452+0000","last_became_peered":"2024-04-24T08:18:10.958452+0000","last_unstale":"2024-04-24T08:18:11.938225+0000","last_undegraded":"2024-04-24T08:18:11.938225+0000","last_fullsized":"2024-04-24T08:18:11.938225+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:56:22.407532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425189+0000","last_change":"2024-04-24T08:18:09.951533+0000","last_active":"2024-04-24T08:18:12.425189+0000","last_peered":"2024-04-24T08:18:12.425189+0000","last_clean":"2024-04-24T08:18:12.425189+0000","last_became_active":"2024-04-24T08:18:09.950569+0000","last_became_peered":"2024-04-24T08:18:09.950569+0000","last_unstale":"2024-04-24T08:18:12.425189+0000","last_undegraded":"2024-04-24T08:18:12.425189+0000","last_fullsized":"2024-04-24T08:18:12.425189+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:59:39.905411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945741+0000","last_change":"2024-04-24T08:18:10.971167+0000","last_active":"2024-04-24T08:18:11.945741+0000","last_peered":"2024-04-24T08:18:11.945741+0000","last_clean":"2024-04-24T08:18:11.945741+0000","last_became_active":"2024-04-24T08:18:10.969277+0000","last_became_peered":"2024-04-24T08:18:10.969277+0000","last_unstale":"2024-04-24T08:18:11.945741+0000","last_undegraded":"2024-04-24T08:18:11.945741+0000","last_fullsized":"2024-04-24T08:18:11.945741+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:30:53.924950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426757+0000","last_change":"2024-04-24T08:18:09.953624+0000","last_active":"2024-04-24T08:18:12.426757+0000","last_peered":"2024-04-24T08:18:12.426757+0000","last_clean":"2024-04-24T08:18:12.426757+0000","last_became_active":"2024-04-24T08:18:09.953332+0000","last_became_peered":"2024-04-24T08:18:09.953332+0000","last_unstale":"2024-04-24T08:18:12.426757+0000","last_undegraded":"2024-04-24T08:18:12.426757+0000","last_fullsized":"2024-04-24T08:18:12.426757+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:23:20.288631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425275+0000","last_change":"2024-04-24T08:18:10.976693+0000","last_active":"2024-04-24T08:18:12.425275+0000","last_peered":"2024-04-24T08:18:12.425275+0000","last_clean":"2024-04-24T08:18:12.425275+0000","last_became_active":"2024-04-24T08:18:10.976379+0000","last_became_peered":"2024-04-24T08:18:10.976379+0000","last_unstale":"2024-04-24T08:18:12.425275+0000","last_undegraded":"2024-04-24T08:18:12.425275+0000","last_fullsized":"2024-04-24T08:18:12.425275+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:03:24.238839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"96'3","reported_seq":17,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426383+0000","last_change":"2024-04-24T08:18:09.959508+0000","last_active":"2024-04-24T08:18:12.426383+0000","last_peered":"2024-04-24T08:18:12.426383+0000","last_clean":"2024-04-24T08:18:12.426383+0000","last_became_active":"2024-04-24T08:18:09.959216+0000","last_became_peered":"2024-04-24T08:18:09.959216+0000","last_unstale":"2024-04-24T08:18:12.426383+0000","last_undegraded":"2024-04-24T08:18:12.426383+0000","last_fullsized":"2024-04-24T08:18:12.426383+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:59:36.083209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425689+0000","last_change":"2024-04-24T08:18:10.962994+0000","last_active":"2024-04-24T08:18:12.425689+0000","last_peered":"2024-04-24T08:18:12.425689+0000","last_clean":"2024-04-24T08:18:12.425689+0000","last_became_active":"2024-04-24T08:18:10.962668+0000","last_became_peered":"2024-04-24T08:18:10.962668+0000","last_unstale":"2024-04-24T08:18:12.425689+0000","last_undegraded":"2024-04-24T08:18:12.425689+0000","last_fullsized":"2024-04-24T08:18:12.425689+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:53:18.191200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"96'6","reported_seq":20,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945987+0000","last_change":"2024-04-24T08:18:09.948927+0000","last_active":"2024-04-24T08:18:11.945987+0000","last_peered":"2024-04-24T08:18:11.945987+0000","last_clean":"2024-04-24T08:18:11.945987+0000","last_became_active":"2024-04-24T08:18:09.948487+0000","last_became_peered":"2024-04-24T08:18:09.948487+0000","last_unstale":"2024-04-24T08:18:11.945987+0000","last_undegraded":"2024-04-24T08:18:11.945987+0000","last_fullsized":"2024-04-24T08:18:11.945987+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:14:26.159337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424112+0000","last_change":"2024-04-24T08:18:10.961987+0000","last_active":"2024-04-24T08:18:12.424112+0000","last_peered":"2024-04-24T08:18:12.424112+0000","last_clean":"2024-04-24T08:18:12.424112+0000","last_became_active":"2024-04-24T08:18:10.961725+0000","last_became_peered":"2024-04-24T08:18:10.961725+0000","last_unstale":"2024-04-24T08:18:12.424112+0000","last_undegraded":"2024-04-24T08:18:12.424112+0000","last_fullsized":"2024-04-24T08:18:12.424112+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:47.129428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425970+0000","last_change":"2024-04-24T08:18:09.948853+0000","last_active":"2024-04-24T08:18:12.425970+0000","last_peered":"2024-04-24T08:18:12.425970+0000","last_clean":"2024-04-24T08:18:12.425970+0000","last_became_active":"2024-04-24T08:18:09.948579+0000","last_became_peered":"2024-04-24T08:18:09.948579+0000","last_unstale":"2024-04-24T08:18:12.425970+0000","last_undegraded":"2024-04-24T08:18:12.425970+0000","last_fullsized":"2024-04-24T08:18:12.425970+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:13.441179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422989+0000","last_change":"2024-04-24T08:18:10.958159+0000","last_active":"2024-04-24T08:18:12.422989+0000","last_peered":"2024-04-24T08:18:12.422989+0000","last_clean":"2024-04-24T08:18:12.422989+0000","last_became_active":"2024-04-24T08:18:10.957886+0000","last_became_peered":"2024-04-24T08:18:10.957886+0000","last_unstale":"2024-04-24T08:18:12.422989+0000","last_undegraded":"2024-04-24T08:18:12.422989+0000","last_fullsized":"2024-04-24T08:18:12.422989+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:39.543199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425565+0000","last_change":"2024-04-24T08:18:09.955673+0000","last_active":"2024-04-24T08:18:12.425565+0000","last_peered":"2024-04-24T08:18:12.425565+0000","last_clean":"2024-04-24T08:18:12.425565+0000","last_became_active":"2024-04-24T08:18:09.955271+0000","last_became_peered":"2024-04-24T08:18:09.955271+0000","last_unstale":"2024-04-24T08:18:12.425565+0000","last_undegraded":"2024-04-24T08:18:12.425565+0000","last_fullsized":"2024-04-24T08:18:12.425565+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:21:00.326506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.944637+0000","last_change":"2024-04-24T08:18:10.967236+0000","last_active":"2024-04-24T08:18:11.944637+0000","last_peered":"2024-04-24T08:18:11.944637+0000","last_clean":"2024-04-24T08:18:11.944637+0000","last_became_active":"2024-04-24T08:18:10.966957+0000","last_became_peered":"2024-04-24T08:18:10.966957+0000","last_unstale":"2024-04-24T08:18:11.944637+0000","last_undegraded":"2024-04-24T08:18:11.944637+0000","last_fullsized":"2024-04-24T08:18:11.944637+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:37.236811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.944806+0000","last_change":"2024-04-24T08:18:09.961786+0000","last_active":"2024-04-24T08:18:11.944806+0000","last_peered":"2024-04-24T08:18:11.944806+0000","last_clean":"2024-04-24T08:18:11.944806+0000","last_became_active":"2024-04-24T08:18:09.960475+0000","last_became_peered":"2024-04-24T08:18:09.960475+0000","last_unstale":"2024-04-24T08:18:11.944806+0000","last_undegraded":"2024-04-24T08:18:11.944806+0000","last_fullsized":"2024-04-24T08:18:11.944806+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:47:14.219825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424412+0000","last_change":"2024-04-24T08:18:10.965745+0000","last_active":"2024-04-24T08:18:12.424412+0000","last_peered":"2024-04-24T08:18:12.424412+0000","last_clean":"2024-04-24T08:18:12.424412+0000","last_became_active":"2024-04-24T08:18:10.965111+0000","last_became_peered":"2024-04-24T08:18:10.965111+0000","last_unstale":"2024-04-24T08:18:12.424412+0000","last_undegraded":"2024-04-24T08:18:12.424412+0000","last_fullsized":"2024-04-24T08:18:12.424412+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:33:45.349270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"96'8","reported_seq":22,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.422353+0000","last_change":"2024-04-24T08:18:09.958822+0000","last_active":"2024-04-24T08:18:12.422353+0000","last_peered":"2024-04-24T08:18:12.422353+0000","last_clean":"2024-04-24T08:18:12.422353+0000","last_became_active":"2024-04-24T08:18:09.958427+0000","last_became_peered":"2024-04-24T08:18:09.958427+0000","last_unstale":"2024-04-24T08:18:12.422353+0000","last_undegraded":"2024-04-24T08:18:12.422353+0000","last_fullsized":"2024-04-24T08:18:12.422353+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:06:20.076356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426256+0000","last_change":"2024-04-24T08:18:10.967330+0000","last_active":"2024-04-24T08:18:12.426256+0000","last_peered":"2024-04-24T08:18:12.426256+0000","last_clean":"2024-04-24T08:18:12.426256+0000","last_became_active":"2024-04-24T08:18:10.966987+0000","last_became_peered":"2024-04-24T08:18:10.966987+0000","last_unstale":"2024-04-24T08:18:12.426256+0000","last_undegraded":"2024-04-24T08:18:12.426256+0000","last_fullsized":"2024-04-24T08:18:12.426256+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:14:30.218520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.945513+0000","last_change":"2024-04-24T08:18:09.955420+0000","last_active":"2024-04-24T08:18:11.945513+0000","last_peered":"2024-04-24T08:18:11.945513+0000","last_clean":"2024-04-24T08:18:11.945513+0000","last_became_active":"2024-04-24T08:18:09.955114+0000","last_became_peered":"2024-04-24T08:18:09.955114+0000","last_unstale":"2024-04-24T08:18:11.945513+0000","last_undegraded":"2024-04-24T08:18:11.945513+0000","last_fullsized":"2024-04-24T08:18:11.945513+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:07:36.574089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.941966+0000","last_change":"2024-04-24T08:18:10.967646+0000","last_active":"2024-04-24T08:18:11.941966+0000","last_peered":"2024-04-24T08:18:11.941966+0000","last_clean":"2024-04-24T08:18:11.941966+0000","last_became_active":"2024-04-24T08:18:10.966255+0000","last_became_peered":"2024-04-24T08:18:10.966255+0000","last_unstale":"2024-04-24T08:18:11.941966+0000","last_undegraded":"2024-04-24T08:18:11.941966+0000","last_fullsized":"2024-04-24T08:18:11.941966+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:04:56.555360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"96'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424857+0000","last_change":"2024-04-24T08:18:09.961530+0000","last_active":"2024-04-24T08:18:12.424857+0000","last_peered":"2024-04-24T08:18:12.424857+0000","last_clean":"2024-04-24T08:18:12.424857+0000","last_became_active":"2024-04-24T08:18:09.960225+0000","last_became_peered":"2024-04-24T08:18:09.960225+0000","last_unstale":"2024-04-24T08:18:12.424857+0000","last_undegraded":"2024-04-24T08:18:12.424857+0000","last_fullsized":"2024-04-24T08:18:12.424857+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:00:06.222300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426099+0000","last_change":"2024-04-24T08:18:10.969069+0000","last_active":"2024-04-24T08:18:12.426099+0000","last_peered":"2024-04-24T08:18:12.426099+0000","last_clean":"2024-04-24T08:18:12.426099+0000","last_became_active":"2024-04-24T08:18:10.968472+0000","last_became_peered":"2024-04-24T08:18:10.968472+0000","last_unstale":"2024-04-24T08:18:12.426099+0000","last_undegraded":"2024-04-24T08:18:12.426099+0000","last_fullsized":"2024-04-24T08:18:12.426099+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:44:47.129428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937931+0000","last_change":"2024-04-24T08:18:09.950987+0000","last_active":"2024-04-24T08:18:11.937931+0000","last_peered":"2024-04-24T08:18:11.937931+0000","last_clean":"2024-04-24T08:18:11.937931+0000","last_became_active":"2024-04-24T08:18:09.950701+0000","last_became_peered":"2024-04-24T08:18:09.950701+0000","last_unstale":"2024-04-24T08:18:11.937931+0000","last_undegraded":"2024-04-24T08:18:11.937931+0000","last_fullsized":"2024-04-24T08:18:11.937931+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:01:35.169277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425278+0000","last_change":"2024-04-24T08:18:10.951009+0000","last_active":"2024-04-24T08:18:12.425278+0000","last_peered":"2024-04-24T08:18:12.425278+0000","last_clean":"2024-04-24T08:18:12.425278+0000","last_became_active":"2024-04-24T08:18:10.950700+0000","last_became_peered":"2024-04-24T08:18:10.950700+0000","last_unstale":"2024-04-24T08:18:12.425278+0000","last_undegraded":"2024-04-24T08:18:12.425278+0000","last_fullsized":"2024-04-24T08:18:12.425278+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:26.192324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937867+0000","last_change":"2024-04-24T08:18:09.956430+0000","last_active":"2024-04-24T08:18:11.937867+0000","last_peered":"2024-04-24T08:18:11.937867+0000","last_clean":"2024-04-24T08:18:11.937867+0000","last_became_active":"2024-04-24T08:18:09.955296+0000","last_became_peered":"2024-04-24T08:18:09.955296+0000","last_unstale":"2024-04-24T08:18:11.937867+0000","last_undegraded":"2024-04-24T08:18:11.937867+0000","last_fullsized":"2024-04-24T08:18:11.937867+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:07:36.574089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425983+0000","last_change":"2024-04-24T08:18:10.953024+0000","last_active":"2024-04-24T08:18:12.425983+0000","last_peered":"2024-04-24T08:18:12.425983+0000","last_clean":"2024-04-24T08:18:12.425983+0000","last_became_active":"2024-04-24T08:18:10.952717+0000","last_became_peered":"2024-04-24T08:18:10.952717+0000","last_unstale":"2024-04-24T08:18:12.425983+0000","last_undegraded":"2024-04-24T08:18:12.425983+0000","last_fullsized":"2024-04-24T08:18:12.425983+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:03:40.778523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.942689+0000","last_change":"2024-04-24T08:18:09.957070+0000","last_active":"2024-04-24T08:18:11.942689+0000","last_peered":"2024-04-24T08:18:11.942689+0000","last_clean":"2024-04-24T08:18:11.942689+0000","last_became_active":"2024-04-24T08:18:09.955906+0000","last_became_peered":"2024-04-24T08:18:09.955906+0000","last_unstale":"2024-04-24T08:18:11.942689+0000","last_undegraded":"2024-04-24T08:18:11.942689+0000","last_fullsized":"2024-04-24T08:18:11.942689+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:16:16.444932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423429+0000","last_change":"2024-04-24T08:18:10.951718+0000","last_active":"2024-04-24T08:18:12.423429+0000","last_peered":"2024-04-24T08:18:12.423429+0000","last_clean":"2024-04-24T08:18:12.423429+0000","last_became_active":"2024-04-24T08:18:10.951451+0000","last_became_peered":"2024-04-24T08:18:10.951451+0000","last_unstale":"2024-04-24T08:18:12.423429+0000","last_undegraded":"2024-04-24T08:18:12.423429+0000","last_fullsized":"2024-04-24T08:18:12.423429+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:55:47.186547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.942174+0000","last_change":"2024-04-24T08:18:09.948999+0000","last_active":"2024-04-24T08:18:11.942174+0000","last_peered":"2024-04-24T08:18:11.942174+0000","last_clean":"2024-04-24T08:18:11.942174+0000","last_became_active":"2024-04-24T08:18:09.948659+0000","last_became_peered":"2024-04-24T08:18:09.948659+0000","last_unstale":"2024-04-24T08:18:11.942174+0000","last_undegraded":"2024-04-24T08:18:11.942174+0000","last_fullsized":"2024-04-24T08:18:11.942174+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:48:14.986504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423481+0000","last_change":"2024-04-24T08:18:10.967340+0000","last_active":"2024-04-24T08:18:12.423481+0000","last_peered":"2024-04-24T08:18:12.423481+0000","last_clean":"2024-04-24T08:18:12.423481+0000","last_became_active":"2024-04-24T08:18:10.967009+0000","last_became_peered":"2024-04-24T08:18:10.967009+0000","last_unstale":"2024-04-24T08:18:12.423481+0000","last_undegraded":"2024-04-24T08:18:12.423481+0000","last_fullsized":"2024-04-24T08:18:12.423481+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:52:05.222065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.938034+0000","last_change":"2024-04-24T08:18:09.952806+0000","last_active":"2024-04-24T08:18:11.938034+0000","last_peered":"2024-04-24T08:18:11.938034+0000","last_clean":"2024-04-24T08:18:11.938034+0000","last_became_active":"2024-04-24T08:18:09.952532+0000","last_became_peered":"2024-04-24T08:18:09.952532+0000","last_unstale":"2024-04-24T08:18:11.938034+0000","last_undegraded":"2024-04-24T08:18:11.938034+0000","last_fullsized":"2024-04-24T08:18:11.938034+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:55:47.761337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937999+0000","last_change":"2024-04-24T08:18:10.959066+0000","last_active":"2024-04-24T08:18:11.937999+0000","last_peered":"2024-04-24T08:18:11.937999+0000","last_clean":"2024-04-24T08:18:11.937999+0000","last_became_active":"2024-04-24T08:18:10.958765+0000","last_became_peered":"2024-04-24T08:18:10.958765+0000","last_unstale":"2024-04-24T08:18:11.937999+0000","last_undegraded":"2024-04-24T08:18:11.937999+0000","last_fullsized":"2024-04-24T08:18:11.937999+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:45:25.063819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"96'2","reported_seq":16,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.942474+0000","last_change":"2024-04-24T08:18:09.953042+0000","last_active":"2024-04-24T08:18:11.942474+0000","last_peered":"2024-04-24T08:18:11.942474+0000","last_clean":"2024-04-24T08:18:11.942474+0000","last_became_active":"2024-04-24T08:18:09.952616+0000","last_became_peered":"2024-04-24T08:18:09.952616+0000","last_unstale":"2024-04-24T08:18:11.942474+0000","last_undegraded":"2024-04-24T08:18:11.942474+0000","last_fullsized":"2024-04-24T08:18:11.942474+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:12:53.330528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424457+0000","last_change":"2024-04-24T08:18:10.952705+0000","last_active":"2024-04-24T08:18:12.424457+0000","last_peered":"2024-04-24T08:18:12.424457+0000","last_clean":"2024-04-24T08:18:12.424457+0000","last_became_active":"2024-04-24T08:18:10.952432+0000","last_became_peered":"2024-04-24T08:18:10.952432+0000","last_unstale":"2024-04-24T08:18:12.424457+0000","last_undegraded":"2024-04-24T08:18:12.424457+0000","last_fullsized":"2024-04-24T08:18:12.424457+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:43:56.909589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"97'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.959036+0000","last_change":"2024-04-24T08:18:09.956418+0000","last_active":"2024-04-24T08:18:12.959036+0000","last_peered":"2024-04-24T08:18:12.959036+0000","last_clean":"2024-04-24T08:18:12.959036+0000","last_became_active":"2024-04-24T08:18:09.956151+0000","last_became_peered":"2024-04-24T08:18:09.956151+0000","last_unstale":"2024-04-24T08:18:12.959036+0000","last_undegraded":"2024-04-24T08:18:12.959036+0000","last_fullsized":"2024-04-24T08:18:12.959036+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:53:17.173422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423400+0000","last_change":"2024-04-24T08:18:10.968128+0000","last_active":"2024-04-24T08:18:12.423400+0000","last_peered":"2024-04-24T08:18:12.423400+0000","last_clean":"2024-04-24T08:18:12.423400+0000","last_became_active":"2024-04-24T08:18:10.967679+0000","last_became_peered":"2024-04-24T08:18:10.967679+0000","last_unstale":"2024-04-24T08:18:12.423400+0000","last_undegraded":"2024-04-24T08:18:12.423400+0000","last_fullsized":"2024-04-24T08:18:12.423400+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:04:56.555360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.938183+0000","last_change":"2024-04-24T08:18:09.949493+0000","last_active":"2024-04-24T08:18:11.938183+0000","last_peered":"2024-04-24T08:18:11.938183+0000","last_clean":"2024-04-24T08:18:11.938183+0000","last_became_active":"2024-04-24T08:18:09.949230+0000","last_became_peered":"2024-04-24T08:18:09.949230+0000","last_unstale":"2024-04-24T08:18:11.938183+0000","last_undegraded":"2024-04-24T08:18:11.938183+0000","last_fullsized":"2024-04-24T08:18:11.938183+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:36.219033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424570+0000","last_change":"2024-04-24T08:18:10.974447+0000","last_active":"2024-04-24T08:18:12.424570+0000","last_peered":"2024-04-24T08:18:12.424570+0000","last_clean":"2024-04-24T08:18:12.424570+0000","last_became_active":"2024-04-24T08:18:10.974172+0000","last_became_peered":"2024-04-24T08:18:10.974172+0000","last_unstale":"2024-04-24T08:18:12.424570+0000","last_undegraded":"2024-04-24T08:18:12.424570+0000","last_fullsized":"2024-04-24T08:18:12.424570+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:55:49.106660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425253+0000","last_change":"2024-04-24T08:18:09.959792+0000","last_active":"2024-04-24T08:18:12.425253+0000","last_peered":"2024-04-24T08:18:12.425253+0000","last_clean":"2024-04-24T08:18:12.425253+0000","last_became_active":"2024-04-24T08:18:09.959379+0000","last_became_peered":"2024-04-24T08:18:09.959379+0000","last_unstale":"2024-04-24T08:18:12.425253+0000","last_undegraded":"2024-04-24T08:18:12.425253+0000","last_fullsized":"2024-04-24T08:18:12.425253+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:50:20.100627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425534+0000","last_change":"2024-04-24T08:18:10.967312+0000","last_active":"2024-04-24T08:18:12.425534+0000","last_peered":"2024-04-24T08:18:12.425534+0000","last_clean":"2024-04-24T08:18:12.425534+0000","last_became_active":"2024-04-24T08:18:10.966992+0000","last_became_peered":"2024-04-24T08:18:10.966992+0000","last_unstale":"2024-04-24T08:18:12.425534+0000","last_undegraded":"2024-04-24T08:18:12.425534+0000","last_fullsized":"2024-04-24T08:18:12.425534+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:59:15.762308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.937174+0000","last_change":"2024-04-24T08:18:09.948078+0000","last_active":"2024-04-24T08:18:11.937174+0000","last_peered":"2024-04-24T08:18:11.937174+0000","last_clean":"2024-04-24T08:18:11.937174+0000","last_became_active":"2024-04-24T08:18:09.947776+0000","last_became_peered":"2024-04-24T08:18:09.947776+0000","last_unstale":"2024-04-24T08:18:11.937174+0000","last_undegraded":"2024-04-24T08:18:11.937174+0000","last_fullsized":"2024-04-24T08:18:11.937174+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:41:02.886416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:11.944978+0000","last_change":"2024-04-24T08:18:10.964531+0000","last_active":"2024-04-24T08:18:11.944978+0000","last_peered":"2024-04-24T08:18:11.944978+0000","last_clean":"2024-04-24T08:18:11.944978+0000","last_became_active":"2024-04-24T08:18:10.964207+0000","last_became_peered":"2024-04-24T08:18:10.964207+0000","last_unstale":"2024-04-24T08:18:11.944978+0000","last_undegraded":"2024-04-24T08:18:11.944978+0000","last_fullsized":"2024-04-24T08:18:11.944978+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:23:24.288637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423452+0000","last_change":"2024-04-24T08:18:09.960963+0000","last_active":"2024-04-24T08:18:12.423452+0000","last_peered":"2024-04-24T08:18:12.423452+0000","last_clean":"2024-04-24T08:18:12.423452+0000","last_became_active":"2024-04-24T08:18:09.960294+0000","last_became_peered":"2024-04-24T08:18:09.960294+0000","last_unstale":"2024-04-24T08:18:12.423452+0000","last_undegraded":"2024-04-24T08:18:12.423452+0000","last_fullsized":"2024-04-24T08:18:12.423452+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:41:02.886416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424054+0000","last_change":"2024-04-24T08:18:10.966570+0000","last_active":"2024-04-24T08:18:12.424054+0000","last_peered":"2024-04-24T08:18:12.424054+0000","last_clean":"2024-04-24T08:18:12.424054+0000","last_became_active":"2024-04-24T08:18:10.966281+0000","last_became_peered":"2024-04-24T08:18:10.966281+0000","last_unstale":"2024-04-24T08:18:12.424054+0000","last_undegraded":"2024-04-24T08:18:12.424054+0000","last_fullsized":"2024-04-24T08:18:12.424054+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:38:40.600148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423124+0000","last_change":"2024-04-24T08:18:09.951310+0000","last_active":"2024-04-24T08:18:12.423124+0000","last_peered":"2024-04-24T08:18:12.423124+0000","last_clean":"2024-04-24T08:18:12.423124+0000","last_became_active":"2024-04-24T08:18:09.951031+0000","last_became_peered":"2024-04-24T08:18:09.951031+0000","last_unstale":"2024-04-24T08:18:12.423124+0000","last_undegraded":"2024-04-24T08:18:12.423124+0000","last_fullsized":"2024-04-24T08:18:12.423124+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:14:26.159337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.423884+0000","last_change":"2024-04-24T08:18:10.970908+0000","last_active":"2024-04-24T08:18:12.423884+0000","last_peered":"2024-04-24T08:18:12.423884+0000","last_clean":"2024-04-24T08:18:12.423884+0000","last_became_active":"2024-04-24T08:18:10.970647+0000","last_became_peered":"2024-04-24T08:18:10.970647+0000","last_unstale":"2024-04-24T08:18:12.423884+0000","last_undegraded":"2024-04-24T08:18:12.423884+0000","last_fullsized":"2024-04-24T08:18:12.423884+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:13:58.550617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424317+0000","last_change":"2024-04-24T08:18:09.946342+0000","last_active":"2024-04-24T08:18:12.424317+0000","last_peered":"2024-04-24T08:18:12.424317+0000","last_clean":"2024-04-24T08:18:12.424317+0000","last_became_active":"2024-04-24T08:18:09.946059+0000","last_became_peered":"2024-04-24T08:18:09.946059+0000","last_unstale":"2024-04-24T08:18:12.424317+0000","last_undegraded":"2024-04-24T08:18:12.424317+0000","last_fullsized":"2024-04-24T08:18:12.424317+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:38:03.998681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.426165+0000","last_change":"2024-04-24T08:18:10.960855+0000","last_active":"2024-04-24T08:18:12.426165+0000","last_peered":"2024-04-24T08:18:12.426165+0000","last_clean":"2024-04-24T08:18:12.426165+0000","last_became_active":"2024-04-24T08:18:10.960521+0000","last_became_peered":"2024-04-24T08:18:10.960521+0000","last_unstale":"2024-04-24T08:18:12.426165+0000","last_undegraded":"2024-04-24T08:18:12.426165+0000","last_fullsized":"2024-04-24T08:18:12.426165+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:09.906866+0000","last_clean_scrub_stamp":"2024-04-24T08:18:09.906866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:12:42.246156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"96'1","reported_seq":15,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.424785+0000","last_change":"2024-04-24T08:18:09.960411+0000","last_active":"2024-04-24T08:18:12.424785+0000","last_peered":"2024-04-24T08:18:12.424785+0000","last_clean":"2024-04-24T08:18:12.424785+0000","last_became_active":"2024-04-24T08:18:09.960110+0000","last_became_peered":"2024-04-24T08:18:09.960110+0000","last_unstale":"2024-04-24T08:18:12.424785+0000","last_undegraded":"2024-04-24T08:18:12.424785+0000","last_fullsized":"2024-04-24T08:18:12.424785+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:18:08.889088+0000","last_clean_scrub_stamp":"2024-04-24T08:18:08.889088+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:21:52.773703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":314,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-04-24T08:18:12.425492+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:18:12.425492+0000","last_peered":"2024-04-24T08:18:12.425492+0000","last_clean":"2024-04-24T08:18:12.425492+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:18:12.425492+0000","last_undegraded":"2024-04-24T08:18:12.425492+0000","last_fullsized":"2024-04-24T08:18:12.425492+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574974,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32940,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338900,"statfs":{"total":96636764160,"available":96603033600,"internally_reserved":0,"allocated":1802240,"data_stored":1109822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574974,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1208320,"data_stored":519994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574972,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32356,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339484,"statfs":{"total":96636764160,"available":96603631616,"internally_reserved":0,"allocated":1204224,"data_stored":519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574971,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1220608,"data_stored":521078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574972,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1216512,"data_stored":520976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574973,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32360,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339480,"statfs":{"total":96636764160,"available":96603627520,"internally_reserved":0,"allocated":1204224,"data_stored":519376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32356,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339484,"statfs":{"total":96636764160,"available":96603631616,"internally_reserved":0,"allocated":1200128,"data_stored":519342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574972,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32936,"kb_used_data":1756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338904,"statfs":{"total":96636764160,"available":96603037696,"internally_reserved":0,"allocated":1798144,"data_stored":1110292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:18:15.876 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T08:18:15.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-24T08:18:16.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:18:16.018 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:18:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.125+0000 7f441d200640 1 Processor -- start 2024-04-24T08:18:16.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.125+0000 7f441d200640 1 -- start start 2024-04-24T08:18:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.127+0000 7f441d200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44180b2b00 0x7f44180d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.127+0000 7f441d200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f441806df30 con 0x7f441809b0e0 2024-04-24T08:18:16.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.127+0000 7f441d200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44180d0770 con 0x7f441806d750 2024-04-24T08:18:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.127+0000 7f441d200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f44180d1060 con 0x7f44180b2b00 2024-04-24T08:18:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.127+0000 7f4416800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f441809b0e0 0x7f44181627b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52014/0 (socket says 172.21.15.69:52014) 2024-04-24T08:18:16.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.127+0000 7f4416800640 1 -- 172.21.15.69:0/3375990395 learned_addr learned my addr 172.21.15.69:0/3375990395 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:16.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.127+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 166442766 0 0) 0x7f4410001110 con 0x7f441809b0e0 2024-04-24T08:18:16.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4415e00640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44180b2b00 0x7f44180d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43fc003520 con 0x7f441809b0e0 2024-04-24T08:18:16.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4185932066 0 0) 0x7f43fc003520 con 0x7f441809b0e0 2024-04-24T08:18:16.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44180b2b00 msgr2=0x7f44180d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44180b2b00 0x7f44180d14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 >> v1:172.21.15.121:6789/0 conn(0x7f441806d750 legacy=0x7f441809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441806dc50 con 0x7f441809b0e0 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f4410001840 con 0x7f441809b0e0 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 >> v1:172.21.15.69:6789/0 conn(0x7f441809b0e0 legacy=0x7f44181627b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc0039e0 0x7f43fc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43fc004800 0x7f43fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44180b2b00 0x7f43fc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44180d1060 con 0x7f43fc0039e0 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc00ba70 con 0x7f43fc004800 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f43fc00bc70 con 0x7f44180b2b00 2024-04-24T08:18:16.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.128+0000 7f4415e00640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc0039e0 0x7f43fc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.129+0000 7f4415400640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43fc004800 0x7f43fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.129+0000 7f4416800640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44180b2b00 0x7f43fc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.129+0000 7f4415e00640 1 -- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44180b2b00 msgr2=0x7f43fc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.129+0000 7f4415e00640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f44180b2b00 0x7f43fc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.129+0000 7f4415e00640 1 -- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43fc004800 msgr2=0x7f43fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.129+0000 7f4415e00640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43fc004800 0x7f43fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.129+0000 7f4415e00640 1 -- 172.21.15.69:0/3375990395 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc003520 con 0x7f43fc0039e0 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.129+0000 7f4415400640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f43fc004800 0x7f43fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.130+0000 7f4415e00640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc0039e0 0x7f43fc003e00 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f4408009f90 tx=0x7f43fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec8130167920368 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.130+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408009c10 con 0x7f43fc0039e0 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.130+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4408036070 con 0x7f43fc0039e0 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.130+0000 7f4414a00640 1 -- 172.21.15.69:0/3375990395 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408031830 con 0x7f43fc0039e0 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.130+0000 7f441d200640 1 -- 172.21.15.69:0/3375990395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc0039e0 msgr2=0x7f43fc003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.130+0000 7f441d200640 1 --2- 172.21.15.69:0/3375990395 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f43fc0039e0 0x7f43fc003e00 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f4408009f90 tx=0x7f43fc005830 comp rx=0 tx=0).stop 2024-04-24T08:18:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.130+0000 7f4416800640 1 -- 172.21.15.69:0/3375990395 reap_dead start 2024-04-24T08:18:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.131+0000 7f441d200640 1 -- 172.21.15.69:0/3375990395 shutdown_connections 2024-04-24T08:18:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.131+0000 7f441d200640 1 -- 172.21.15.69:0/3375990395 >> 172.21.15.69:0/3375990395 conn(0x7f44180b4040 msgr2=0x7f44180b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.131+0000 7f441d200640 1 -- 172.21.15.69:0/3375990395 shutdown_connections 2024-04-24T08:18:16.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.131+0000 7f441d200640 1 -- 172.21.15.69:0/3375990395 wait complete. 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.132+0000 7f441d200640 1 Processor -- start 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.133+0000 7f441d200640 1 -- start start 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.133+0000 7f441d200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43fc0039e0 0x7f44181592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.133+0000 7f441d200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44181597f0 0x7f44181f4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.133+0000 7f441d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44181f5330 0x7f44181f57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f441d200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44180b2ee0 con 0x7f44181f5330 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f441d200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f44180d0cf0 con 0x7f44181597f0 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f441d200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f44180b38e0 con 0x7f43fc0039e0 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f4416800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44181f5330 0x7f44181f57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f4416800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44181f5330 0x7f44181f57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40400/0 (socket says 172.21.15.69:40400) 2024-04-24T08:18:16.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f4416800640 1 -- 172.21.15.69:0/3356169011 learned_addr learned my addr 172.21.15.69:0/3356169011 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f4415400640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44181597f0 0x7f44181f4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f4416800640 1 -- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43fc0039e0 msgr2=0x7f44181592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f4416800640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43fc0039e0 0x7f44181592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f4416800640 1 -- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44181597f0 msgr2=0x7f44181f4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f4416800640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44181597f0 0x7f44181f4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f4416800640 1 -- 172.21.15.69:0/3356169011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44181f5cf0 con 0x7f44181f5330 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.134+0000 7f4416800640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44181f5330 0x7f44181f57b0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f4418072960 tx=0x7f441000d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.135+0000 7f440e000640 1 -- 172.21.15.69:0/3356169011 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4410007440 con 0x7f44181f5330 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.135+0000 7f440e000640 1 -- 172.21.15.69:0/3356169011 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4410007600 con 0x7f44181f5330 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.135+0000 7f441d200640 1 -- 172.21.15.69:0/3356169011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f441815a070 con 0x7f44181f5330 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.135+0000 7f440e000640 1 -- 172.21.15.69:0/3356169011 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4410004a20 con 0x7f44181f5330 2024-04-24T08:18:16.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.135+0000 7f441d200640 1 -- 172.21.15.69:0/3356169011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f441815a630 con 0x7f44181f5330 2024-04-24T08:18:16.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.136+0000 7f441d200640 1 -- 172.21.15.69:0/3356169011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4418075e30 con 0x7f44181f5330 2024-04-24T08:18:16.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.137+0000 7f440e000640 1 -- 172.21.15.69:0/3356169011 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f441001b060 con 0x7f44181f5330 2024-04-24T08:18:16.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.138+0000 7f440e000640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43f406c8f0 0x7f43f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.138+0000 7f440e000640 1 -- 172.21.15.69:0/3356169011 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f441008b440 con 0x7f44181f5330 2024-04-24T08:18:16.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.138+0000 7f4415e00640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43f406c8f0 0x7f43f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.140+0000 7f4415e00640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43f406c8f0 0x7f43f406edb0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4408035000 tx=0x7f440802d000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.142 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.142+0000 7f440e000640 1 -- 172.21.15.69:0/3356169011 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f441806df30 con 0x7f44181f5330 2024-04-24T08:18:16.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.414+0000 7f441d200640 1 -- 172.21.15.69:0/3356169011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f44180c9270 con 0x7f44181f5330 2024-04-24T08:18:16.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.418+0000 7f440e000640 1 -- 172.21.15.69:0/3356169011 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) ==== 190+0+147 (secure 0 0 0) 0x7f441000f2f0 con 0x7f44181f5330 2024-04-24T08:18:16.419 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-24T08:18:16.419 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw== 2024-04-24T08:18:16.419 INFO:teuthology.orchestra.run.smithi069.stdout: caps mds = "allow" 2024-04-24T08:18:16.419 INFO:teuthology.orchestra.run.smithi069.stdout: caps mon = "allow r" 2024-04-24T08:18:16.419 INFO:teuthology.orchestra.run.smithi069.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-24T08:18:16.419 INFO:teuthology.orchestra.run.smithi069.stderr:updated caps for client.0 2024-04-24T08:18:16.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 -- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43f406c8f0 msgr2=0x7f43f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43f406c8f0 0x7f43f406edb0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f4408035000 tx=0x7f440802d000 comp rx=0 tx=0).stop 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 -- 172.21.15.69:0/3356169011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44181f5330 msgr2=0x7f44181f57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44181f5330 0x7f44181f57b0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f4418072960 tx=0x7f441000d9d0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 -- 172.21.15.69:0/3356169011 shutdown_connections 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f43f406c8f0 0x7f43f406edb0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44181f5330 0x7f44181f57b0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44181597f0 0x7f44181f4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 --2- 172.21.15.69:0/3356169011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f43fc0039e0 0x7f44181592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 -- 172.21.15.69:0/3356169011 >> 172.21.15.69:0/3356169011 conn(0x7f44180b4040 msgr2=0x7f44180b09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 -- 172.21.15.69:0/3356169011 shutdown_connections 2024-04-24T08:18:16.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.421+0000 7f441d200640 1 -- 172.21.15.69:0/3356169011 wait complete. 2024-04-24T08:18:16.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:16.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.676+0000 7f3c30e00640 1 Processor -- start 2024-04-24T08:18:16.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.676+0000 7f3c30e00640 1 -- start start 2024-04-24T08:18:16.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c30e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c1523f0 0x7f3c2c06db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c30e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3c2c15b980 0x7f3c2c15be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c30e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c2c0aeac0 con 0x7f3c2c072e20 2024-04-24T08:18:16.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c30e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3c2c0aeda0 con 0x7f3c2c15b980 2024-04-24T08:18:16.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c30e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3c2c0af080 con 0x7f3c2c1523f0 2024-04-24T08:18:16.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c22a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3c2c072e20 0x7f3c2c073260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52026/0 (socket says 172.21.15.69:52026) 2024-04-24T08:18:16.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c22a00640 1 -- 172.21.15.69:0/1006216302 learned_addr learned my addr 172.21.15.69:0/1006216302 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:16.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3276367613 0 0) 0x7f3c2c0aeac0 con 0x7f3c2c072e20 2024-04-24T08:18:16.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c10005730 con 0x7f3c2c072e20 2024-04-24T08:18:16.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c2b400640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c1523f0 0x7f3c2c06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.678+0000 7f3c2be00640 1 --2- 172.21.15.69:0/1006216302 >> v2:172.21.15.121:3300/0 conn(0x7f3c2c15b980 0x7f3c2c15be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2372063174 0 0) 0x7f3c10005730 con 0x7f3c2c072e20 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c1523f0 msgr2=0x7f3c2c06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c1523f0 0x7f3c2c06db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 >> v2:172.21.15.121:3300/0 conn(0x7f3c2c15b980 msgr2=0x7f3c2c15be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 --2- 172.21.15.69:0/1006216302 >> v2:172.21.15.121:3300/0 conn(0x7f3c2c15b980 0x7f3c2c15be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c2c0af360 con 0x7f3c2c072e20 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f3c14001670 con 0x7f3c2c072e20 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 >> v1:172.21.15.69:6789/0 conn(0x7f3c2c072e20 legacy=0x7f3c2c073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c10005bc0 0x7f3c10005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c1523f0 0x7f3c10006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c10006f50 0x7f3c100073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c0aeda0 con 0x7f3c10006f50 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c0af080 con 0x7f3c10005bc0 2024-04-24T08:18:16.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3c1000dd60 con 0x7f3c2c1523f0 2024-04-24T08:18:16.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.679+0000 7f3c2be00640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c10006f50 0x7f3c100073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.680+0000 7f3c2b400640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c10005bc0 0x7f3c10005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.680+0000 7f3c22a00640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c1523f0 0x7f3c10006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.680+0000 7f3c2be00640 1 -- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c1523f0 msgr2=0x7f3c10006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.680+0000 7f3c2be00640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c1523f0 0x7f3c10006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.680+0000 7f3c2be00640 1 -- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c10005bc0 msgr2=0x7f3c10005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.680+0000 7f3c2be00640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c10005bc0 0x7f3c10005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.680+0000 7f3c2be00640 1 -- 172.21.15.69:0/1006216302 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c10005730 con 0x7f3c10006f50 2024-04-24T08:18:16.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.681+0000 7f3c2be00640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c10006f50 0x7f3c100073d0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f3c24007360 tx=0x7f3c10007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd7e327cfa68765 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.681+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c24009be0 con 0x7f3c10006f50 2024-04-24T08:18:16.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.681+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c2402e620 con 0x7f3c10006f50 2024-04-24T08:18:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.681+0000 7f3c2aa00640 1 -- 172.21.15.69:0/1006216302 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c2402e9f0 con 0x7f3c10006f50 2024-04-24T08:18:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.682+0000 7f3c30e00640 1 -- 172.21.15.69:0/1006216302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c10006f50 msgr2=0x7f3c100073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.682+0000 7f3c30e00640 1 --2- 172.21.15.69:0/1006216302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c10006f50 0x7f3c100073d0 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f3c24007360 tx=0x7f3c10007a40 comp rx=0 tx=0).stop 2024-04-24T08:18:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.682+0000 7f3c2be00640 1 -- 172.21.15.69:0/1006216302 reap_dead start 2024-04-24T08:18:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.682+0000 7f3c30e00640 1 -- 172.21.15.69:0/1006216302 shutdown_connections 2024-04-24T08:18:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.682+0000 7f3c30e00640 1 -- 172.21.15.69:0/1006216302 >> 172.21.15.69:0/1006216302 conn(0x7f3c2c0d0160 msgr2=0x7f3c2c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.682+0000 7f3c30e00640 1 -- 172.21.15.69:0/1006216302 shutdown_connections 2024-04-24T08:18:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.682+0000 7f3c30e00640 1 -- 172.21.15.69:0/1006216302 wait complete. 2024-04-24T08:18:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.684+0000 7f3c30e00640 1 Processor -- start 2024-04-24T08:18:16.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.684+0000 7f3c30e00640 1 -- start start 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c30e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c2c1f4140 0x7f3c2c14ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c30e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c2c1f4520 0x7f3c2c1586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c30e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c158c00 0x7f3c2c153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c30e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c073970 con 0x7f3c2c1f4520 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c30e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3c2c1527d0 con 0x7f3c2c1f4140 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c30e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3c2c073af0 con 0x7f3c2c158c00 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c22a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c2c1f4520 0x7f3c2c1586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c22a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c2c1f4520 0x7f3c2c1586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40420/0 (socket says 172.21.15.69:40420) 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c22a00640 1 -- 172.21.15.69:0/3204325524 learned_addr learned my addr 172.21.15.69:0/3204325524 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c22a00640 1 -- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c158c00 msgr2=0x7f3c2c153f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c22a00640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c158c00 0x7f3c2c153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c22a00640 1 -- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c2c1f4140 msgr2=0x7f3c2c14ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c22a00640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c2c1f4140 0x7f3c2c14ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.685+0000 7f3c22a00640 1 -- 172.21.15.69:0/3204325524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c2c150a80 con 0x7f3c2c1f4520 2024-04-24T08:18:16.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.686+0000 7f3c22a00640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c2c1f4520 0x7f3c2c1586c0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f3c14001200 tx=0x7f3c1400ef00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.686+0000 7f3c23e00640 1 -- 172.21.15.69:0/3204325524 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1400c850 con 0x7f3c2c1f4520 2024-04-24T08:18:16.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.686+0000 7f3c23e00640 1 -- 172.21.15.69:0/3204325524 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c14018070 con 0x7f3c2c1f4520 2024-04-24T08:18:16.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.686+0000 7f3c23e00640 1 -- 172.21.15.69:0/3204325524 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1400cc50 con 0x7f3c2c1f4520 2024-04-24T08:18:16.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.686+0000 7f3c30e00640 1 -- 172.21.15.69:0/3204325524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c2c150d60 con 0x7f3c2c1f4520 2024-04-24T08:18:16.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.686+0000 7f3c30e00640 1 -- 172.21.15.69:0/3204325524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c2c147270 con 0x7f3c2c1f4520 2024-04-24T08:18:16.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.687+0000 7f3c30e00640 1 -- 172.21.15.69:0/3204325524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c2c073660 con 0x7f3c2c1f4520 2024-04-24T08:18:16.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.688+0000 7f3c23e00640 1 -- 172.21.15.69:0/3204325524 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3c1401d460 con 0x7f3c2c1f4520 2024-04-24T08:18:16.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.689+0000 7f3c23e00640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3c0806c920 0x7f3c0806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.689+0000 7f3c23e00640 1 -- 172.21.15.69:0/3204325524 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f3c1408bae0 con 0x7f3c2c1f4520 2024-04-24T08:18:16.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.690+0000 7f3c2b400640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3c0806c920 0x7f3c0806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.691+0000 7f3c2b400640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3c0806c920 0x7f3c0806ede0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c006fd0 tx=0x7f3c1c00d280 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.693+0000 7f3c23e00640 1 -- 172.21.15.69:0/3204325524 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c2c073660 con 0x7f3c2c1f4520 2024-04-24T08:18:16.881 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:18:16.880+0000 7f7949200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:18:16.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.932+0000 7f3c30e00640 1 -- 172.21.15.69:0/3204325524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c2c075d50 con 0x7f3c2c1f4520 2024-04-24T08:18:16.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.933+0000 7f3c23e00640 1 -- 172.21.15.69:0/3204325524 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5186 (secure 0 0 0) 0x7f3c14059f00 con 0x7f3c2c1f4520 2024-04-24T08:18:16.934 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:16.934 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:18:13.481156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6046":{"gid":6046,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/412016900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":412016900},{"type":"v1","addr":"172.21.15.69:6835","nonce":412016900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6046,"qdb_cluster":[6046]},"id":7}]} 2024-04-24T08:18:16.934 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 83 2024-04-24T08:18:16.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 -- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3c0806c920 msgr2=0x7f3c0806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3c0806c920 0x7f3c0806ede0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c006fd0 tx=0x7f3c1c00d280 comp rx=0 tx=0).stop 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 -- 172.21.15.69:0/3204325524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c2c1f4520 msgr2=0x7f3c2c1586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c2c1f4520 0x7f3c2c1586c0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f3c14001200 tx=0x7f3c1400ef00 comp rx=0 tx=0).stop 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 -- 172.21.15.69:0/3204325524 shutdown_connections 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3c0806c920 0x7f3c0806ede0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c2c1f4520 0x7f3c2c1586c0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3c2c158c00 0x7f3c2c153f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 --2- 172.21.15.69:0/3204325524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c2c1f4140 0x7f3c2c14ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 -- 172.21.15.69:0/3204325524 >> 172.21.15.69:0/3204325524 conn(0x7f3c2c0d0160 msgr2=0x7f3c2c0b3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 -- 172.21.15.69:0/3204325524 shutdown_connections 2024-04-24T08:18:16.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:16.936+0000 7f3c30e00640 1 -- 172.21.15.69:0/3204325524 wait complete. 2024-04-24T08:18:17.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:18:10.396179+0000', 'modified': '2024-04-24T08:18:13.481156+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6046}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6046': {'gid': 6046, 'name': 'a', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.69:6835/412016900', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6833', 'nonce': 412016900}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 412016900}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6046, 'qdb_cluster': [6046]} 2024-04-24T08:18:17.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:18:17.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-24T08:18:17.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.570+0000 7f55c6e00640 1 Processor -- start 2024-04-24T08:18:17.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.571+0000 7f55c6e00640 1 -- start start 2024-04-24T08:18:17.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.572+0000 7f55c6e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f55c0160020 0x7f55c00d0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.572+0000 7f55c6e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 0x7f55c00d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.572+0000 7f55c6e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55c00cdc70 con 0x7f55c015fc40 2024-04-24T08:18:17.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.572+0000 7f55c6e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f55c00b3b30 con 0x7f55c0160020 2024-04-24T08:18:17.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.572+0000 7f55c6e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f55c00cdf50 con 0x7f55c00d0190 2024-04-24T08:18:17.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.573+0000 7f55c5000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f55c0160020 0x7f55c00d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.573+0000 7f55c5a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f55c015fc40 0x7f55c00b3370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:52038/0 (socket says 172.21.15.69:52038) 2024-04-24T08:18:17.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.573+0000 7f55c5a00640 1 -- 172.21.15.69:0/3634182460 learned_addr learned my addr 172.21.15.69:0/3634182460 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:17.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.573+0000 7f55c5000640 1 -- 172.21.15.69:0/3634182460 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 msgr2=0x7f55c00d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.573+0000 7f55c6400640 1 --2- 172.21.15.69:0/3634182460 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 0x7f55c00d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.573+0000 7f55bfe00640 1 -- 172.21.15.69:0/3634182460 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4108543716 0 0) 0x7f55c00cdc70 con 0x7f55c015fc40 2024-04-24T08:18:17.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.573+0000 7f55c5000640 1 --2- 172.21.15.69:0/3634182460 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 0x7f55c00d0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.573+0000 7f55c5000640 1 -- 172.21.15.69:0/3634182460 >> v1:172.21.15.69:6789/0 conn(0x7f55c015fc40 legacy=0x7f55c00b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.573+0000 7f55c5000640 1 -- 172.21.15.69:0/3634182460 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c00ce230 con 0x7f55c0160020 2024-04-24T08:18:17.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.573+0000 7f55c6400640 1 --2- 172.21.15.69:0/3634182460 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 0x7f55c00d0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:17.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.574+0000 7f55c5000640 1 --2- 172.21.15.69:0/3634182460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 0x7f55c00d0990 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009d90 tx=0x7f55ac02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c64305979b83f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.575+0000 7f55bfe00640 1 -- 172.21.15.69:0/3634182460 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac007740 con 0x7f55c0160020 2024-04-24T08:18:17.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.575+0000 7f55bfe00640 1 -- 172.21.15.69:0/3634182460 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55ac0078e0 con 0x7f55c0160020 2024-04-24T08:18:17.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.575+0000 7f55bfe00640 1 -- 172.21.15.69:0/3634182460 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac007cb0 con 0x7f55c0160020 2024-04-24T08:18:17.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.575+0000 7f55c6e00640 1 -- 172.21.15.69:0/3634182460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 msgr2=0x7f55c00d0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.575+0000 7f55c6e00640 1 --2- 172.21.15.69:0/3634182460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 0x7f55c00d0990 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009d90 tx=0x7f55ac02a660 comp rx=0 tx=0).stop 2024-04-24T08:18:17.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.575+0000 7f55c6e00640 1 -- 172.21.15.69:0/3634182460 shutdown_connections 2024-04-24T08:18:17.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.575+0000 7f55c6e00640 1 --2- 172.21.15.69:0/3634182460 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 0x7f55c00d0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.575+0000 7f55c6e00640 1 --2- 172.21.15.69:0/3634182460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 0x7f55c00d0990 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.575+0000 7f55c6e00640 1 -- 172.21.15.69:0/3634182460 >> 172.21.15.69:0/3634182460 conn(0x7f55c00b3ea0 msgr2=0x7f55c00b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.576+0000 7f55c6e00640 1 -- 172.21.15.69:0/3634182460 shutdown_connections 2024-04-24T08:18:17.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.576+0000 7f55c6e00640 1 -- 172.21.15.69:0/3634182460 wait complete. 2024-04-24T08:18:17.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.577+0000 7f55c6e00640 1 Processor -- start 2024-04-24T08:18:17.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.577+0000 7f55c6e00640 1 -- start start 2024-04-24T08:18:17.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.578+0000 7f55c6e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 0x7f55c00cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.578+0000 7f55c6e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55c015fc40 0x7f55c00c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.578+0000 7f55c6e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 0x7f55c01548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.578+0000 7f55c6e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55c00b5970 con 0x7f55c0160020 2024-04-24T08:18:17.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.578+0000 7f55c6e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f55c00b57f0 con 0x7f55c015fc40 2024-04-24T08:18:17.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.578+0000 7f55c6e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f55c00b5af0 con 0x7f55c00d0190 2024-04-24T08:18:17.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c6400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 0x7f55c01548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c6400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 0x7f55c01548e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40446/0 (socket says 172.21.15.69:40446) 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c6400640 1 -- 172.21.15.69:0/2346400354 learned_addr learned my addr 172.21.15.69:0/2346400354 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c5a00640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 0x7f55c00cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c5000640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55c015fc40 0x7f55c00c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c6400640 1 -- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 msgr2=0x7f55c00cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c6400640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 0x7f55c00cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c6400640 1 -- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55c015fc40 msgr2=0x7f55c00c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c6400640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55c015fc40 0x7f55c00c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c6400640 1 -- 172.21.15.69:0/2346400354 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c00c9bf0 con 0x7f55c0160020 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c5a00640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 0x7f55c00cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c5000640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55c015fc40 0x7f55c00c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.579+0000 7f55c6400640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 0x7f55c01548e0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f55b800c820 tx=0x7f55b800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.580+0000 7f55bd600640 1 -- 172.21.15.69:0/2346400354 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8015070 con 0x7f55c0160020 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.580+0000 7f55bd600640 1 -- 172.21.15.69:0/2346400354 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b8007a30 con 0x7f55c0160020 2024-04-24T08:18:17.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.580+0000 7f55bd600640 1 -- 172.21.15.69:0/2346400354 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8007e20 con 0x7f55c0160020 2024-04-24T08:18:17.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.580+0000 7f55c6e00640 1 -- 172.21.15.69:0/2346400354 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c00c9ed0 con 0x7f55c0160020 2024-04-24T08:18:17.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.580+0000 7f55c6e00640 1 -- 172.21.15.69:0/2346400354 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c00c5e40 con 0x7f55c0160020 2024-04-24T08:18:17.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.582+0000 7f55bd600640 1 -- 172.21.15.69:0/2346400354 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f55b8018070 con 0x7f55c0160020 2024-04-24T08:18:17.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.584+0000 7f55bd600640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55a006c8f0 0x7f55a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.584+0000 7f55bd600640 1 -- 172.21.15.69:0/2346400354 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f55b808bb20 con 0x7f55c0160020 2024-04-24T08:18:17.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.584+0000 7f55c5a00640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55a006c8f0 0x7f55a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.584+0000 7f55cb391740 1 Processor -- start 2024-04-24T08:18:17.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.585+0000 7f55cb391740 1 -- start start 2024-04-24T08:18:17.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.585+0000 7f55cb391740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c713980 0x56543c6c6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.585+0000 7f55cb391740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 0x56543c6f5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.585+0000 7f55cb391740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6360 0x56543c6cb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.585+0000 7f55cb391740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56543c630b10 con 0x56543c6f6360 2024-04-24T08:18:17.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.585+0000 7f55cb391740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56543c5d72b0 con 0x56543c6f6aa0 2024-04-24T08:18:17.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.585+0000 7f55cb391740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56543c62cad0 con 0x56543c713980 2024-04-24T08:18:17.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.585+0000 7f55c5a00640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55a006c8f0 0x7f55a006edb0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f55b0000f60 tx=0x7f55b0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c6400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6360 0x56543c6cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c6400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6360 0x56543c6cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40450/0 (socket says 172.21.15.69:40450) 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c6400640 1 -- 172.21.15.69:0/1474309274 learned_addr learned my addr 172.21.15.69:0/1474309274 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c5a00640 1 --2- 172.21.15.69:0/1474309274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c713980 0x56543c6c6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c5000640 1 --2- 172.21.15.69:0/1474309274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 0x56543c6f5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c6400640 1 -- 172.21.15.69:0/1474309274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c713980 msgr2=0x56543c6c6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c6400640 1 --2- 172.21.15.69:0/1474309274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c713980 0x56543c6c6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c6400640 1 -- 172.21.15.69:0/1474309274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 msgr2=0x56543c6f5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c6400640 1 --2- 172.21.15.69:0/1474309274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 0x56543c6f5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c6400640 1 -- 172.21.15.69:0/1474309274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56543c3432d0 con 0x56543c6f6360 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c5000640 1 --2- 172.21.15.69:0/1474309274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 0x56543c6f5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.586+0000 7f55c6400640 1 --2- 172.21.15.69:0/1474309274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6360 0x56543c6cb170 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f55b8089f20 tx=0x7f55b805db60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e35532f44fd23ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55b6000640 1 -- 172.21.15.69:0/1474309274 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b801f040 con 0x56543c6f6360 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55b6000640 1 -- 172.21.15.69:0/1474309274 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b800ce10 con 0x56543c6f6360 2024-04-24T08:18:17.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55b6000640 1 -- 172.21.15.69:0/1474309274 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b8060070 con 0x56543c6f6360 2024-04-24T08:18:17.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55cb391740 1 -- 172.21.15.69:0/1474309274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6360 msgr2=0x56543c6cb170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55cb391740 1 --2- 172.21.15.69:0/1474309274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6360 0x56543c6cb170 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f55b8089f20 tx=0x7f55b805db60 comp rx=0 tx=0).stop 2024-04-24T08:18:17.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55cb391740 1 -- 172.21.15.69:0/1474309274 shutdown_connections 2024-04-24T08:18:17.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55cb391740 1 --2- 172.21.15.69:0/1474309274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6360 0x56543c6cb170 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55cb391740 1 --2- 172.21.15.69:0/1474309274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 0x56543c6f5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55cb391740 1 --2- 172.21.15.69:0/1474309274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c713980 0x56543c6c6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55cb391740 1 -- 172.21.15.69:0/1474309274 >> 172.21.15.69:0/1474309274 conn(0x56543c6c8580 msgr2=0x56543c710e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55cb391740 1 -- 172.21.15.69:0/1474309274 shutdown_connections 2024-04-24T08:18:17.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.587+0000 7f55cb391740 1 -- 172.21.15.69:0/1474309274 wait complete. 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.588+0000 7f55cb391740 1 -- start start 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.588+0000 7f55cb391740 1 Processor -- start 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.589+0000 7f55cb391740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 0x56543c6f8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.589+0000 7f55cb391740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c711e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.589+0000 7f55cb391740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c6e38c0 0x56543c7123b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.589+0000 7f55cb391740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56543c6ca730 con 0x56543c713980 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.589+0000 7f55cb391740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56543c5ea590 con 0x56543c6f6aa0 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.589+0000 7f55cb391740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56543c6c9b20 con 0x56543c6e38c0 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.589+0000 7f55cb391740 10 client.0 _ll_register_callbacks cb 0x56543c5cd3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c711e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c711e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40458/0 (socket says 172.21.15.69:40458) 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c5000640 1 -- 172.21.15.69:0/3140187163 learned_addr learned my addr 172.21.15.69:0/3140187163 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c5a00640 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 0x56543c6f8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c6400640 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c6e38c0 0x56543c7123b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c5000640 1 -- 172.21.15.69:0/3140187163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c6e38c0 msgr2=0x56543c7123b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c5000640 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c6e38c0 0x56543c7123b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c5000640 1 -- 172.21.15.69:0/3140187163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 msgr2=0x56543c6f8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c5000640 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 0x56543c6f8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c5000640 1 -- 172.21.15.69:0/3140187163 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56543c40d6b0 con 0x56543c713980 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55c5000640 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c711e70 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009eb0 tx=0x7f55ac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df80b90c79ea58dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.590+0000 7f55aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55aaa00640 1 -- 172.21.15.69:0/3140187163 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac009450 con 0x56543c713980 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55cb391740 1 -- 172.21.15.69:0/3140187163 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55b800ce10 con 0x56543c713980 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55aaa00640 1 -- 172.21.15.69:0/3140187163 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f55ac007540 con 0x56543c713980 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55aaa00640 1 -- 172.21.15.69:0/3140187163 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f55ac04c050 con 0x56543c713980 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55cb391740 10 client.8706 fetch_fsmap learned FSMap version 83 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55cb391740 1 -- 172.21.15.69:0/3140187163 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x56543c6f8bc0 con 0x56543c713980 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55aaa00640 1 -- 172.21.15.69:0/3140187163 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b800ce10 con 0x56543c713980 2024-04-24T08:18:17.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55aaa00640 1 -- 172.21.15.69:0/3140187163 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7f55ac007740 con 0x56543c713980 2024-04-24T08:18:17.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55cb391740 10 client.8706 fetch_fsmap finished waiting for FSMap version 83 2024-04-24T08:18:17.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55cb391740 10 client.8706 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:18:17.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55cb391740 20 client.8706 populate_metadata read hostname 'smithi069' 2024-04-24T08:18:17.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55cb391740 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x56543c6f7a80 0x56543c72f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55cb391740 4 client.8706 mds_command: new command op to 6046 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:17.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.591+0000 7f55cb391740 1 -- 172.21.15.69:0/3140187163 --> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56543c63e530 con 0x56543c6f7a80 2024-04-24T08:18:17.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.592+0000 7f55c5a00640 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x56543c6f7a80 0x56543c72f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.592+0000 7f55c5a00640 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x56543c6f7a80 0x56543c72f5b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad0b029427833978 server_cookie=3df9fe2ad7543c4d in_seq=0 out_seq=0 2024-04-24T08:18:17.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.592+0000 7f55aaa00640 10 client.8706 ms_handle_connect on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.593+0000 7f55aaa00640 1 -- 172.21.15.69:0/3140187163 <== mds.0 v2:172.21.15.69:6833/412016900 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56543c63e530 con 0x56543c6f7a80 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.593+0000 7f55aaa00640 10 client.8706 handle_command_reply: tid=0 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.593+0000 7f55cb391740 1 client.8706 shutdown 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 1 -- 172.21.15.69:0/3140187163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 msgr2=0x56543c711e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c711e70 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009eb0 tx=0x7f55ac004ab0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 1 -- 172.21.15.69:0/3140187163 shutdown_connections 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c711e70 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x56543c6f7a80 0x56543c72f5b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c6e38c0 0x56543c7123b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 1 --2- 172.21.15.69:0/3140187163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6f6aa0 0x56543c6f8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55aaa00640 0 client.8706 ms_handle_reset on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 1 -- 172.21.15.69:0/3140187163 >> 172.21.15.69:0/3140187163 conn(0x56543c6f6360 msgr2=0x56543c6c7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 1 -- 172.21.15.69:0/3140187163 shutdown_connections 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 1 -- 172.21.15.69:0/3140187163 wait complete. 2024-04-24T08:18:17.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.594+0000 7f55cb391740 20 client.8706 trim_cache size 0 max 16384 2024-04-24T08:18:17.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.614+0000 7f55cb391740 1 Processor -- start 2024-04-24T08:18:17.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.614+0000 7f55cb391740 1 -- start start 2024-04-24T08:18:17.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.614+0000 7f55cb391740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6e38c0 0x56543c6cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.614+0000 7f55cb391740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c6f6aa0 0x56543c6cc070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.614+0000 7f55cb391740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c707fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.614+0000 7f55cb391740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55c0200aa0 con 0x56543c713980 2024-04-24T08:18:17.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.614+0000 7f55cb391740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f55c01ffa80 con 0x56543c6e38c0 2024-04-24T08:18:17.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.614+0000 7f55cb391740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f55c01f8c40 con 0x56543c6f6aa0 2024-04-24T08:18:17.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c6400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c707fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c6400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c707fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40474/0 (socket says 172.21.15.69:40474) 2024-04-24T08:18:17.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c6400640 1 -- 172.21.15.69:0/1540266527 learned_addr learned my addr 172.21.15.69:0/1540266527 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:17.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c5000640 1 --2- 172.21.15.69:0/1540266527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c6f6aa0 0x56543c6cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c5a00640 1 --2- 172.21.15.69:0/1540266527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6e38c0 0x56543c6cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c6400640 1 -- 172.21.15.69:0/1540266527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c6f6aa0 msgr2=0x56543c6cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c6400640 1 --2- 172.21.15.69:0/1540266527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c6f6aa0 0x56543c6cc070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c6400640 1 -- 172.21.15.69:0/1540266527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6e38c0 msgr2=0x56543c6cbb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c6400640 1 --2- 172.21.15.69:0/1540266527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6e38c0 0x56543c6cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c6400640 1 -- 172.21.15.69:0/1540266527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56543c712f20 con 0x56543c713980 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.615+0000 7f55c6400640 1 --2- 172.21.15.69:0/1540266527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c707fb0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f55b8055c10 tx=0x7f55b8060000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c94fff74a0dbd429 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55b5600640 1 -- 172.21.15.69:0/1540266527 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b801f040 con 0x56543c713980 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55b5600640 1 -- 172.21.15.69:0/1540266527 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b8060070 con 0x56543c713980 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55b5600640 1 -- 172.21.15.69:0/1540266527 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b800ce10 con 0x56543c713980 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55cb391740 1 -- 172.21.15.69:0/1540266527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 msgr2=0x56543c707fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55cb391740 1 --2- 172.21.15.69:0/1540266527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c707fb0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f55b8055c10 tx=0x7f55b8060000 comp rx=0 tx=0).stop 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55cb391740 1 -- 172.21.15.69:0/1540266527 shutdown_connections 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55cb391740 1 --2- 172.21.15.69:0/1540266527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c713980 0x56543c707fb0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55cb391740 1 --2- 172.21.15.69:0/1540266527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c6f6aa0 0x56543c6cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55cb391740 1 --2- 172.21.15.69:0/1540266527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6e38c0 0x56543c6cbb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55cb391740 1 -- 172.21.15.69:0/1540266527 >> 172.21.15.69:0/1540266527 conn(0x56543c6f6360 msgr2=0x56543c6f11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.616+0000 7f55cb391740 1 -- 172.21.15.69:0/1540266527 shutdown_connections 2024-04-24T08:18:17.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.617+0000 7f55cb391740 1 -- 172.21.15.69:0/1540266527 wait complete. 2024-04-24T08:18:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.617+0000 7f55cb391740 1 -- start start 2024-04-24T08:18:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.617+0000 7f55cb391740 1 Processor -- start 2024-04-24T08:18:17.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.618+0000 7f55cb391740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6e38c0 0x56543c6e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.618+0000 7f55cb391740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6aa0 0x56543c6cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.618+0000 7f55cb391740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c713980 0x56543c78b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.618+0000 7f55cb391740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56543c6f8ea0 con 0x56543c6f6aa0 2024-04-24T08:18:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.618+0000 7f55cb391740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f55c01ff3f0 con 0x56543c6e38c0 2024-04-24T08:18:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.618+0000 7f55cb391740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56543c6f94b0 con 0x56543c713980 2024-04-24T08:18:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55cb391740 10 client.0 _ll_register_callbacks cb 0x7f55ac003e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55c5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6aa0 0x56543c6cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55c5000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6aa0 0x56543c6cbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40486/0 (socket says 172.21.15.69:40486) 2024-04-24T08:18:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55c5000640 1 -- 172.21.15.69:0/3959974081 learned_addr learned my addr 172.21.15.69:0/3959974081 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55c5a00640 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6e38c0 0x56543c6e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55c5000640 1 -- 172.21.15.69:0/3959974081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c713980 msgr2=0x56543c78b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55c5000640 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c713980 0x56543c78b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55c5000640 1 -- 172.21.15.69:0/3959974081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6e38c0 msgr2=0x56543c6e59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55c5000640 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6e38c0 0x56543c6e59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55c5000640 1 -- 172.21.15.69:0/3959974081 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56543c6cc470 con 0x56543c6f6aa0 2024-04-24T08:18:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.619+0000 7f55c5000640 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6aa0 0x56543c6cbf30 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f55ac007300 tx=0x7f55ac030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9b26f2d3c7eada server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55aaa00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:18:17.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55aaa00640 1 -- 172.21.15.69:0/3959974081 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac009a30 con 0x56543c6f6aa0 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55aaa00640 1 -- 172.21.15.69:0/3959974081 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f55ac003fb0 con 0x56543c6f6aa0 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55cb391740 1 -- 172.21.15.69:0/3959974081 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55b8060070 con 0x56543c6f6aa0 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55aaa00640 1 -- 172.21.15.69:0/3959974081 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f55ac0041b0 con 0x56543c6f6aa0 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55cb391740 10 client.8712 fetch_fsmap learned FSMap version 83 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55cb391740 1 -- 172.21.15.69:0/3959974081 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x56543c723d30 con 0x56543c6f6aa0 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55aaa00640 1 -- 172.21.15.69:0/3959974081 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b8060070 con 0x56543c6f6aa0 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55aaa00640 1 -- 172.21.15.69:0/3959974081 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7f55ac064020 con 0x56543c6f6aa0 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55cb391740 10 client.8712 fetch_fsmap finished waiting for FSMap version 83 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55cb391740 10 client.8712 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.620+0000 7f55cb391740 20 client.8712 populate_metadata read hostname 'smithi069' 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.621+0000 7f55cb391740 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x56543c722cb0 0x56543c723090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.621+0000 7f55cb391740 4 client.8712 mds_command: new command op to 6046 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:18:17.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.621+0000 7f55cb391740 1 -- 172.21.15.69:0/3959974081 --> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] -- command(tid 0: {"prefix": "status"}) -- 0x56543c6383a0 con 0x56543c722cb0 2024-04-24T08:18:17.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.621+0000 7f55c5a00640 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x56543c722cb0 0x56543c723090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.621+0000 7f55c5a00640 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x56543c722cb0 0x56543c723090 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f17be42db397077d server_cookie=e51567e62c5e779a in_seq=0 out_seq=0 2024-04-24T08:18:17.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.621+0000 7f55aaa00640 10 client.8712 ms_handle_connect on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:17.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55aaa00640 1 -- 172.21.15.69:0/3959974081 <== mds.0 v2:172.21.15.69:6833/412016900 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x56543c6383a0 con 0x56543c722cb0 2024-04-24T08:18:17.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55aaa00640 10 client.8712 handle_command_reply: tid=0 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 client.8712 shutdown 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 -- 172.21.15.69:0/3959974081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6aa0 msgr2=0x56543c6cbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6aa0 0x56543c6cbf30 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f55ac007300 tx=0x7f55ac030000 comp rx=0 tx=0).stop 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 -- 172.21.15.69:0/3959974081 shutdown_connections 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56543c6f6aa0 0x56543c6cbf30 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x56543c722cb0 0x56543c723090 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56543c713980 0x56543c78b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 --2- 172.21.15.69:0/3959974081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56543c6e38c0 0x56543c6e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55aaa00640 0 client.8712 ms_handle_reset on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 -- 172.21.15.69:0/3959974081 >> 172.21.15.69:0/3959974081 conn(0x56543c6f6360 msgr2=0x56543c7120f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 -- 172.21.15.69:0/3959974081 shutdown_connections 2024-04-24T08:18:17.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 1 -- 172.21.15.69:0/3959974081 wait complete. 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.622+0000 7f55cb391740 20 client.8712 trim_cache size 0 max 16384 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 6046, 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 6.689964238, 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 83, 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 97, 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 97, 2024-04-24T08:18:17.628 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 12.155261634 2024-04-24T08:18:17.629 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:18:17.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 -- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55a006c8f0 msgr2=0x7f55a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55a006c8f0 0x7f55a006edb0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f55b0000f60 tx=0x7f55b0009040 comp rx=0 tx=0).stop 2024-04-24T08:18:17.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 -- 172.21.15.69:0/2346400354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 msgr2=0x7f55c01548e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 0x7f55c01548e0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f55b800c820 tx=0x7f55b800cce0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 -- 172.21.15.69:0/2346400354 shutdown_connections 2024-04-24T08:18:17.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f55a006c8f0 0x7f55a006edb0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55c0160020 0x7f55c01548e0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f55c015fc40 0x7f55c00c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 --2- 172.21.15.69:0/2346400354 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f55c00d0190 0x7f55c00cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 -- 172.21.15.69:0/2346400354 >> 172.21.15.69:0/2346400354 conn(0x7f55c00b3ea0 msgr2=0x7f55c006cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 -- 172.21.15.69:0/2346400354 shutdown_connections 2024-04-24T08:18:17.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.623+0000 7f55c6e00640 1 -- 172.21.15.69:0/2346400354 wait complete. 2024-04-24T08:18:17.640 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:18:17.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.835+0000 7f5af3e00640 1 Processor -- start 2024-04-24T08:18:17.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.835+0000 7f5af3e00640 1 -- start start 2024-04-24T08:18:17.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.837+0000 7f5af3e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5af415e990 0x7f5af4160210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.837+0000 7f5af3e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5af406e720 0x7f5af40d0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.837+0000 7f5af3e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af415ed70 con 0x7f5af40b3eb0 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.837+0000 7f5af3e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f5af4162810 con 0x7f5af406e720 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.837+0000 7f5af3e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5af4162af0 con 0x7f5af415e990 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.837+0000 7f5aebe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5af406e720 0x7f5af40d0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.838+0000 7f5aebe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f5af406e720 0x7f5af40d0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40492/0 (socket says 172.21.15.69:40492) 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.838+0000 7f5aebe00640 1 -- 172.21.15.69:0/3591326759 learned_addr learned my addr 172.21.15.69:0/3591326759 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.838+0000 7f5af2a00640 1 --2- 172.21.15.69:0/3591326759 >> v2:172.21.15.121:3300/0 conn(0x7f5af415e990 0x7f5af4160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.838+0000 7f5aebe00640 1 -- 172.21.15.69:0/3591326759 >> v2:172.21.15.121:3300/0 conn(0x7f5af415e990 msgr2=0x7f5af4160210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.838+0000 7f5aebe00640 1 --2- 172.21.15.69:0/3591326759 >> v2:172.21.15.121:3300/0 conn(0x7f5af415e990 0x7f5af4160210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.838+0000 7f5aebe00640 1 -- 172.21.15.69:0/3591326759 >> v1:172.21.15.121:6789/0 conn(0x7f5af40b3eb0 legacy=0x7f5af415f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.838+0000 7f5aebe00640 1 -- 172.21.15.69:0/3591326759 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af4160750 con 0x7f5af406e720 2024-04-24T08:18:17.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.840+0000 7f5aebe00640 1 --2- 172.21.15.69:0/3591326759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af406e720 0x7f5af40d0000 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f5af40686d0 tx=0x7f5adc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd45a0a75762734 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.840+0000 7f5af2000640 1 -- 172.21.15.69:0/3591326759 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc037070 con 0x7f5af406e720 2024-04-24T08:18:17.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.840+0000 7f5af2000640 1 -- 172.21.15.69:0/3591326759 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5adc032440 con 0x7f5af406e720 2024-04-24T08:18:17.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.840+0000 7f5af2000640 1 -- 172.21.15.69:0/3591326759 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc032810 con 0x7f5af406e720 2024-04-24T08:18:17.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.841+0000 7f5af3e00640 1 -- 172.21.15.69:0/3591326759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af406e720 msgr2=0x7f5af40d0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.841+0000 7f5af3e00640 1 --2- 172.21.15.69:0/3591326759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af406e720 0x7f5af40d0000 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f5af40686d0 tx=0x7f5adc02a230 comp rx=0 tx=0).stop 2024-04-24T08:18:17.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.841+0000 7f5af3e00640 1 -- 172.21.15.69:0/3591326759 shutdown_connections 2024-04-24T08:18:17.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.841+0000 7f5af3e00640 1 --2- 172.21.15.69:0/3591326759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af406e720 0x7f5af40d0000 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.841+0000 7f5af3e00640 1 --2- 172.21.15.69:0/3591326759 >> v2:172.21.15.121:3300/0 conn(0x7f5af415e990 0x7f5af4160210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.841+0000 7f5af3e00640 1 -- 172.21.15.69:0/3591326759 >> 172.21.15.69:0/3591326759 conn(0x7f5af406fd10 msgr2=0x7f5af4070120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.841+0000 7f5af3e00640 1 -- 172.21.15.69:0/3591326759 shutdown_connections 2024-04-24T08:18:17.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.841+0000 7f5af3e00640 1 -- 172.21.15.69:0/3591326759 wait complete. 2024-04-24T08:18:17.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.843+0000 7f5af3e00640 1 Processor -- start 2024-04-24T08:18:17.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.843+0000 7f5af3e00640 1 -- start start 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.844+0000 7f5af3e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5af406e720 0x7f5af4155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.844+0000 7f5af3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af40b3eb0 0x7f5af40c65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.844+0000 7f5af3e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5af415e990 0x7f5af40cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.844+0000 7f5af3e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5af4162430 con 0x7f5af40b3eb0 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.844+0000 7f5af3e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5af41630b0 con 0x7f5af406e720 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.844+0000 7f5af3e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5af41625b0 con 0x7f5af415e990 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.844+0000 7f5aebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af40b3eb0 0x7f5af40c65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.844+0000 7f5aebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af40b3eb0 0x7f5af40c65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40502/0 (socket says 172.21.15.69:40502) 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.844+0000 7f5aebe00640 1 -- 172.21.15.69:0/1138554830 learned_addr learned my addr 172.21.15.69:0/1138554830 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.844+0000 7f5af2a00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5af406e720 0x7f5af4155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5aebe00640 1 -- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5af415e990 msgr2=0x7f5af40cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5af3400640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5af415e990 0x7f5af40cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5aebe00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5af415e990 0x7f5af40cde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5aebe00640 1 -- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5af406e720 msgr2=0x7f5af4155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5aebe00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5af406e720 0x7f5af4155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5aebe00640 1 -- 172.21.15.69:0/1138554830 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af40c7c90 con 0x7f5af40b3eb0 2024-04-24T08:18:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5af2a00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5af406e720 0x7f5af4155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5aebe00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af40b3eb0 0x7f5af40c65c0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f5adc02a270 tx=0x7f5adc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5aeaa00640 1 -- 172.21.15.69:0/1138554830 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc041020 con 0x7f5af40b3eb0 2024-04-24T08:18:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5aeaa00640 1 -- 172.21.15.69:0/1138554830 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5adc0306c0 con 0x7f5af40b3eb0 2024-04-24T08:18:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5aeaa00640 1 -- 172.21.15.69:0/1138554830 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5adc037040 con 0x7f5af40b3eb0 2024-04-24T08:18:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.845+0000 7f5af3e00640 1 -- 172.21.15.69:0/1138554830 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af40cb130 con 0x7f5af40b3eb0 2024-04-24T08:18:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.846+0000 7f5af3e00640 1 -- 172.21.15.69:0/1138554830 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af40c5ae0 con 0x7f5af40b3eb0 2024-04-24T08:18:17.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.847+0000 7f5af3e00640 1 -- 172.21.15.69:0/1138554830 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af4075d50 con 0x7f5af40b3eb0 2024-04-24T08:18:17.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.847+0000 7f5aeaa00640 1 -- 172.21.15.69:0/1138554830 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5adc044020 con 0x7f5af40b3eb0 2024-04-24T08:18:17.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.848+0000 7f5aeaa00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ad006c8f0 0x7f5ad006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.849+0000 7f5aeaa00640 1 -- 172.21.15.69:0/1138554830 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f5adc0ac330 con 0x7f5af40b3eb0 2024-04-24T08:18:17.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.849+0000 7f5af2a00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ad006c8f0 0x7f5ad006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.850+0000 7f5af2a00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ad006c8f0 0x7f5ad006edb0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4006fd0 tx=0x7f5ae4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:17.853+0000 7f5aeaa00640 1 -- 172.21.15.69:0/1138554830 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5adc04b030 con 0x7f5af40b3eb0 2024-04-24T08:18:18.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.104+0000 7f5af3e00640 1 -- 172.21.15.69:0/1138554830 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f5af4070830 con 0x7f5af40b3eb0 2024-04-24T08:18:18.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.105+0000 7f5aeaa00640 1 -- 172.21.15.69:0/1138554830 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v83) ==== 53+0+73 (secure 0 0 0) 0x7f5adc07a760 con 0x7f5af40b3eb0 2024-04-24T08:18:18.105 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:18:18.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.107+0000 7f5af3e00640 1 -- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ad006c8f0 msgr2=0x7f5ad006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:18.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.107+0000 7f5af3e00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ad006c8f0 0x7f5ad006edb0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4006fd0 tx=0x7f5ae4008040 comp rx=0 tx=0).stop 2024-04-24T08:18:18.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.107+0000 7f5af3e00640 1 -- 172.21.15.69:0/1138554830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af40b3eb0 msgr2=0x7f5af40c65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:18.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.107+0000 7f5af3e00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af40b3eb0 0x7f5af40c65c0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f5adc02a270 tx=0x7f5adc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:18.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.108+0000 7f5af3e00640 1 -- 172.21.15.69:0/1138554830 shutdown_connections 2024-04-24T08:18:18.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.108+0000 7f5af3e00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ad006c8f0 0x7f5ad006edb0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:18.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.108+0000 7f5af3e00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5af40b3eb0 0x7f5af40c65c0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:18.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.108+0000 7f5af3e00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5af415e990 0x7f5af40cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:18.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.108+0000 7f5af3e00640 1 --2- 172.21.15.69:0/1138554830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5af406e720 0x7f5af4155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:18.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.108+0000 7f5af3e00640 1 -- 172.21.15.69:0/1138554830 >> 172.21.15.69:0/1138554830 conn(0x7f5af406fd10 msgr2=0x7f5af4156a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:18.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.108+0000 7f5af3e00640 1 -- 172.21.15.69:0/1138554830 shutdown_connections 2024-04-24T08:18:18.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18.108+0000 7f5af3e00640 1 -- 172.21.15.69:0/1138554830 wait complete. 2024-04-24T08:18:18.127 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:18:18.127 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:18:18.127 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:18:18.127 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:18:18.127 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:18.127 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:18:18.127 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:18:18.127 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:18:18.127 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:18:18.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:18.128 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:18:18.179 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:18:18.179 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:18:18.179 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:18:18.179 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:18.179 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:18:18.179 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42082sec preferred_lft 42082sec 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:18.180 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:18:18.181 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:18.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:18.182 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:18:18.260 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:18:18.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:18.264 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:18:18.315 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:18:18.315 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:18:18.315 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:18:18.316 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:18:18.316 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:18:18.316 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:18:18.316 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:18:18.316 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:18:18.316 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:18:18.316 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:18:18.316 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:18:18.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:18:18.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:18:18.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:18.526 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:18:18.572 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:18:18.572 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:18:18.573 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:18:18.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:18.573 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:18:18.646 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:18:18.646 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:18:18.646 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:18.695 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:18:18.695 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:18:18.695 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:18.745 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:18:18.828 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:18:18.879 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:18:18.880 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:18:18.880 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:18:18.880 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:18:18.880 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:18:18.880 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:18:18.880 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:18:18.880 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:18:18.880 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:18:18.880 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:18:18.880 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:18:18.881 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:18:18.882 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:18:18.882 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:18:18.882 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:18:18.882 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:18:18.882 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:18:18.882 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:18:18.882 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:18:18.882 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:18:18.882 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:18:18.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:18.883 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:18:18.929 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:18:18.930 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:18:18.934 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:18:19.003 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:18:19.043 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:18:19.043 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:18:19.043 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:18:19.043 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:18:19.043 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:18:19.043 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:18:19.043 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:18:19.043 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:18:19.043 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:18:19.044 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:18:19.045 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:18:19.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:19.046 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:18:19.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:18:19.101+0000 7fd38d1ea480 -1 init, newargv = 0x5593287414a0 newargc=15 2024-04-24T08:18:19.101 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[69871]: starting ceph client 2024-04-24T08:18:19.110 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[69871]: starting fuse 2024-04-24T08:18:19.130 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:18:19.130 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:18:19.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:18:19.130 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:19.131 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:18:19.132 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:18:19.132 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:18:19.132 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:18:19.132 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:18:19.132 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:18:19.132 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:18:19.132 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:18:19.132 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:19.132 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:18:19.132 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:18:19.330 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.69871.asok 2024-04-24T08:18:19.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:18:19.348 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.69871.asok 2024-04-24T08:18:19.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:19.348 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.69871.asok status 2024-04-24T08:18:19.572 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:18:19.615 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:18:19.615 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:18:19.615 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:18:19.615 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "69871", 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 8724, 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 8724 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:18:19.616 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3285991453 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3285991453 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.8724 192.168.0.1:0/3285991453", 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/3285991453", 2024-04-24T08:18:19.617 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:18:19.618 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 83, 2024-04-24T08:18:19.618 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 97, 2024-04-24T08:18:19.618 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:18:19.618 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:18:19.618 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:18:19.618 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:18:19.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:19.619 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:18:19.619 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:19.676 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:18:19.676 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:19.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:19.676 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:19.760 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:18:19.760 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:19.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:19.812 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file1_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-04-24T08:18:20.032 INFO:teuthology.orchestra.run.smithi069.stderr:6+0 records in 2024-04-24T08:18:20.032 INFO:teuthology.orchestra.run.smithi069.stderr:6+0 records out 2024-04-24T08:18:20.033 INFO:teuthology.orchestra.run.smithi069.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.159756 s, 39.4 MB/s 2024-04-24T08:18:20.035 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:18:20.035 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:20.092 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:20.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.291+0000 7f04f1400640 1 Processor -- start 2024-04-24T08:18:20.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.291+0000 7f04f1400640 1 -- start start 2024-04-24T08:18:20.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.292+0000 7f04f1400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04ec06db90 0x7f04ec073530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.292+0000 7f04f1400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec15b980 0x7f04ec15be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.292+0000 7f04f1400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ec073a70 con 0x7f04ec1523f0 2024-04-24T08:18:20.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.292+0000 7f04f1400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f04ec0aeac0 con 0x7f04ec06db90 2024-04-24T08:18:20.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.292+0000 7f04f1400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04ec0aeda0 con 0x7f04ec15b980 2024-04-24T08:18:20.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.293+0000 7f04f0a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec15b980 0x7f04ec15be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.293+0000 7f04ebe00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f04ec1523f0 0x7f04ec072e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:42966/0 (socket says 172.21.15.69:42966) 2024-04-24T08:18:20.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.293+0000 7f04ebe00640 1 -- 172.21.15.69:0/1318675087 learned_addr learned my addr 172.21.15.69:0/1318675087 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:20.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.293+0000 7f04eb400640 1 --2- 172.21.15.69:0/1318675087 >> v2:172.21.15.121:3300/0 conn(0x7f04ec06db90 0x7f04ec073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.294+0000 7f04eaa00640 1 -- 172.21.15.69:0/1318675087 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2684815964 0 0) 0x7f04ec073a70 con 0x7f04ec1523f0 2024-04-24T08:18:20.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.294+0000 7f04eaa00640 1 -- 172.21.15.69:0/1318675087 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04d0003520 con 0x7f04ec1523f0 2024-04-24T08:18:20.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.294+0000 7f04f0a00640 1 -- 172.21.15.69:0/1318675087 >> v2:172.21.15.121:3300/0 conn(0x7f04ec06db90 msgr2=0x7f04ec073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:20.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.294+0000 7f04f0a00640 1 --2- 172.21.15.69:0/1318675087 >> v2:172.21.15.121:3300/0 conn(0x7f04ec06db90 0x7f04ec073530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.294+0000 7f04f0a00640 1 -- 172.21.15.69:0/1318675087 >> v1:172.21.15.121:6789/0 conn(0x7f04ec1523f0 legacy=0x7f04ec072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.294+0000 7f04f0a00640 1 -- 172.21.15.69:0/1318675087 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec0af080 con 0x7f04ec15b980 2024-04-24T08:18:20.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.296+0000 7f04f0a00640 1 --2- 172.21.15.69:0/1318675087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec15b980 0x7f04ec15be10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04e40097a0 tx=0x7f04e402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=518225b028e94f14 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:20.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.296+0000 7f04eaa00640 1 -- 172.21.15.69:0/1318675087 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4037070 con 0x7f04ec15b980 2024-04-24T08:18:20.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.296+0000 7f04eaa00640 1 -- 172.21.15.69:0/1318675087 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04e402ae50 con 0x7f04ec15b980 2024-04-24T08:18:20.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.296+0000 7f04eaa00640 1 -- 172.21.15.69:0/1318675087 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4032660 con 0x7f04ec15b980 2024-04-24T08:18:20.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.297+0000 7f04f1400640 1 -- 172.21.15.69:0/1318675087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec15b980 msgr2=0x7f04ec15be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.297+0000 7f04f1400640 1 --2- 172.21.15.69:0/1318675087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec15b980 0x7f04ec15be10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f04e40097a0 tx=0x7f04e402a100 comp rx=0 tx=0).stop 2024-04-24T08:18:20.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.297+0000 7f04f1400640 1 -- 172.21.15.69:0/1318675087 shutdown_connections 2024-04-24T08:18:20.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.297+0000 7f04f1400640 1 --2- 172.21.15.69:0/1318675087 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec15b980 0x7f04ec15be10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.297+0000 7f04f1400640 1 --2- 172.21.15.69:0/1318675087 >> v2:172.21.15.121:3300/0 conn(0x7f04ec06db90 0x7f04ec073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.297+0000 7f04f1400640 1 -- 172.21.15.69:0/1318675087 >> 172.21.15.69:0/1318675087 conn(0x7f04ec0d0160 msgr2=0x7f04ec0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:20.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.297+0000 7f04f1400640 1 -- 172.21.15.69:0/1318675087 shutdown_connections 2024-04-24T08:18:20.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.297+0000 7f04f1400640 1 -- 172.21.15.69:0/1318675087 wait complete. 2024-04-24T08:18:20.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.299+0000 7f04f1400640 1 Processor -- start 2024-04-24T08:18:20.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.299+0000 7f04f1400640 1 -- start start 2024-04-24T08:18:20.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04f1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec06db90 0x7f04ec1466f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04f1400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec1523f0 0x7f04ec0b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04f1400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04ec15b980 0x7f04ec0b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04f1400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec06dfd0 con 0x7f04ec06db90 2024-04-24T08:18:20.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04f1400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec0b3190 con 0x7f04ec15b980 2024-04-24T08:18:20.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04f1400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04ec1527d0 con 0x7f04ec1523f0 2024-04-24T08:18:20.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04ebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec06db90 0x7f04ec1466f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04ebe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec06db90 0x7f04ec1466f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35200/0 (socket says 172.21.15.69:35200) 2024-04-24T08:18:20.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04ebe00640 1 -- 172.21.15.69:0/2037185619 learned_addr learned my addr 172.21.15.69:0/2037185619 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:20.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04eb400640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec1523f0 0x7f04ec0b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04f0a00640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04ec15b980 0x7f04ec0b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04ebe00640 1 -- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec1523f0 msgr2=0x7f04ec0b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04ebe00640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec1523f0 0x7f04ec0b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04ebe00640 1 -- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04ec15b980 msgr2=0x7f04ec0b0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04ebe00640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04ec15b980 0x7f04ec0b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04ebe00640 1 -- 172.21.15.69:0/2037185619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec06eee0 con 0x7f04ec06db90 2024-04-24T08:18:20.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.300+0000 7f04eb400640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec1523f0 0x7f04ec0b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:20.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.301+0000 7f04ebe00640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec06db90 0x7f04ec1466f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00c430 tx=0x7f04dc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:20.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.301+0000 7f04e3e00640 1 -- 172.21.15.69:0/2037185619 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc017070 con 0x7f04ec06db90 2024-04-24T08:18:20.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.301+0000 7f04e3e00640 1 -- 172.21.15.69:0/2037185619 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04dc005ce0 con 0x7f04ec06db90 2024-04-24T08:18:20.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.301+0000 7f04f1400640 1 -- 172.21.15.69:0/2037185619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ec1555e0 con 0x7f04ec06db90 2024-04-24T08:18:20.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.301+0000 7f04e3e00640 1 -- 172.21.15.69:0/2037185619 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc0072f0 con 0x7f04ec06db90 2024-04-24T08:18:20.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.301+0000 7f04f1400640 1 -- 172.21.15.69:0/2037185619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ec155ba0 con 0x7f04ec06db90 2024-04-24T08:18:20.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.302+0000 7f04f1400640 1 -- 172.21.15.69:0/2037185619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ec075d50 con 0x7f04ec06db90 2024-04-24T08:18:20.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.303+0000 7f04e3e00640 1 -- 172.21.15.69:0/2037185619 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04dc01a070 con 0x7f04ec06db90 2024-04-24T08:18:20.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.304+0000 7f04e3e00640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04cc06c940 0x7f04cc06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.304+0000 7f04e3e00640 1 -- 172.21.15.69:0/2037185619 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f04dc08af80 con 0x7f04ec06db90 2024-04-24T08:18:20.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.305+0000 7f04eb400640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04cc06c940 0x7f04cc06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.306+0000 7f04eb400640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04cc06c940 0x7f04cc06ee00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0b4620 tx=0x7f04d800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:20.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.309+0000 7f04e3e00640 1 -- 172.21.15.69:0/2037185619 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04dc059420 con 0x7f04ec06db90 2024-04-24T08:18:20.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.511+0000 7f04f1400640 1 -- 172.21.15.69:0/2037185619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04ec0cf6f0 con 0x7f04ec06db90 2024-04-24T08:18:20.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.513+0000 7f04e3e00640 1 -- 172.21.15.69:0/2037185619 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+16584 (secure 0 0 0) 0x7f04dc058d20 con 0x7f04ec06db90 2024-04-24T08:18:20.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:20.514 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":97,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:18:11.928428+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:08.156864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:09.368519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:20.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 -- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04cc06c940 msgr2=0x7f04cc06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04cc06c940 0x7f04cc06ee00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f04ec0b4620 tx=0x7f04d800b040 comp rx=0 tx=0).stop 2024-04-24T08:18:20.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 -- 172.21.15.69:0/2037185619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec06db90 msgr2=0x7f04ec1466f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec06db90 0x7f04ec1466f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00c430 tx=0x7f04dc00c8f0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 -- 172.21.15.69:0/2037185619 shutdown_connections 2024-04-24T08:18:20.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f04cc06c940 0x7f04cc06ee00 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04ec15b980 0x7f04ec0b0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04ec1523f0 0x7f04ec0b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 --2- 172.21.15.69:0/2037185619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04ec06db90 0x7f04ec1466f0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 -- 172.21.15.69:0/2037185619 >> 172.21.15.69:0/2037185619 conn(0x7f04ec0d0160 msgr2=0x7f04ec15a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:20.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 -- 172.21.15.69:0/2037185619 shutdown_connections 2024-04-24T08:18:20.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.516+0000 7f04f1400640 1 -- 172.21.15.69:0/2037185619 wait complete. 2024-04-24T08:18:20.536 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:08.156864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:09.368519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:18:20.537 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:18:20.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.748+0000 7fcdee800640 1 Processor -- start 2024-04-24T08:18:20.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.748+0000 7fcdee800640 1 -- start start 2024-04-24T08:18:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.749+0000 7fcdee800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcde81420a0 0x7fcde8071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.749+0000 7fcdee800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcde81606e0 0x7fcde80cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.749+0000 7fcdee800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcde8072450 con 0x7fcde8142480 2024-04-24T08:18:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.749+0000 7fcdee800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcde80cf1e0 con 0x7fcde81420a0 2024-04-24T08:18:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.749+0000 7fcdee800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcde8072730 con 0x7fcde81606e0 2024-04-24T08:18:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.749+0000 7fcded400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcde81420a0 0x7fcde8071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.750+0000 7fcded400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcde81420a0 0x7fcde8071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35210/0 (socket says 172.21.15.69:35210) 2024-04-24T08:18:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.750+0000 7fcded400640 1 -- 172.21.15.69:0/2176817972 learned_addr learned my addr 172.21.15.69:0/2176817972 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.750+0000 7fcded400640 1 -- 172.21.15.69:0/2176817972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcde81606e0 msgr2=0x7fcde80cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.750+0000 7fcdede00640 1 --2- 172.21.15.69:0/2176817972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcde81606e0 0x7fcde80cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.750+0000 7fcded400640 1 --2- 172.21.15.69:0/2176817972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcde81606e0 0x7fcde80cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.750+0000 7fcded400640 1 -- 172.21.15.69:0/2176817972 >> v1:172.21.15.121:6789/0 conn(0x7fcde8142480 legacy=0x7fcde815ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.750+0000 7fcded400640 1 -- 172.21.15.69:0/2176817972 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde8072a10 con 0x7fcde81420a0 2024-04-24T08:18:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.750+0000 7fcdede00640 1 --2- 172.21.15.69:0/2176817972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcde81606e0 0x7fcde80cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.751+0000 7fcded400640 1 --2- 172.21.15.69:0/2176817972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 0x7fcde8071d70 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009870 tx=0x7fcddc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7299d799da070df server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.751+0000 7fcde7e00640 1 -- 172.21.15.69:0/2176817972 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc037070 con 0x7fcde81420a0 2024-04-24T08:18:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.751+0000 7fcde7e00640 1 -- 172.21.15.69:0/2176817972 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcddc032440 con 0x7fcde81420a0 2024-04-24T08:18:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.751+0000 7fcde7e00640 1 -- 172.21.15.69:0/2176817972 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc032810 con 0x7fcde81420a0 2024-04-24T08:18:20.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.752+0000 7fcdee800640 1 -- 172.21.15.69:0/2176817972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 msgr2=0x7fcde8071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.752+0000 7fcdee800640 1 --2- 172.21.15.69:0/2176817972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 0x7fcde8071d70 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009870 tx=0x7fcddc02a210 comp rx=0 tx=0).stop 2024-04-24T08:18:20.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.752+0000 7fcdee800640 1 -- 172.21.15.69:0/2176817972 shutdown_connections 2024-04-24T08:18:20.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.752+0000 7fcdee800640 1 --2- 172.21.15.69:0/2176817972 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcde81606e0 0x7fcde80cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.752+0000 7fcdee800640 1 --2- 172.21.15.69:0/2176817972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 0x7fcde8071d70 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.752+0000 7fcdee800640 1 -- 172.21.15.69:0/2176817972 >> 172.21.15.69:0/2176817972 conn(0x7fcde8067b80 msgr2=0x7fcde806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:20.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.752+0000 7fcdee800640 1 -- 172.21.15.69:0/2176817972 shutdown_connections 2024-04-24T08:18:20.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.752+0000 7fcdee800640 1 -- 172.21.15.69:0/2176817972 wait complete. 2024-04-24T08:18:20.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.754+0000 7fcdee800640 1 Processor -- start 2024-04-24T08:18:20.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.754+0000 7fcdee800640 1 -- start start 2024-04-24T08:18:20.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.755+0000 7fcdee800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 0x7fcde80b3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.755+0000 7fcdee800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcde8142480 0x7fcde8136c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.755+0000 7fcdee800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcde81606e0 0x7fcde815b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.755+0000 7fcdee800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcde80cfbb0 con 0x7fcde81420a0 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.755+0000 7fcdee800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcde8142910 con 0x7fcde81606e0 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.755+0000 7fcdee800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcde80cfd30 con 0x7fcde8142480 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.755+0000 7fcded400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 0x7fcde80b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.755+0000 7fcded400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 0x7fcde80b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35220/0 (socket says 172.21.15.69:35220) 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.755+0000 7fcded400640 1 -- 172.21.15.69:0/1104932543 learned_addr learned my addr 172.21.15.69:0/1104932543 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.756+0000 7fcdede00640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcde81606e0 0x7fcde815b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.756+0000 7fcded400640 1 -- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcde8142480 msgr2=0x7fcde8136c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.756+0000 7fcded400640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcde8142480 0x7fcde8136c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.756+0000 7fcded400640 1 -- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcde81606e0 msgr2=0x7fcde815b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.756+0000 7fcded400640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcde81606e0 0x7fcde815b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.756+0000 7fcded400640 1 -- 172.21.15.69:0/1104932543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcde80b2180 con 0x7fcde81420a0 2024-04-24T08:18:20.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.756+0000 7fcded400640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 0x7fcde80b3220 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009870 tx=0x7fcddc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.756+0000 7fcde5600640 1 -- 172.21.15.69:0/1104932543 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc041020 con 0x7fcde81420a0 2024-04-24T08:18:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.757+0000 7fcde5600640 1 -- 172.21.15.69:0/1104932543 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcddc030970 con 0x7fcde81420a0 2024-04-24T08:18:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.757+0000 7fcde5600640 1 -- 172.21.15.69:0/1104932543 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc037040 con 0x7fcde81420a0 2024-04-24T08:18:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.757+0000 7fcdee800640 1 -- 172.21.15.69:0/1104932543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcde80b2460 con 0x7fcde81420a0 2024-04-24T08:18:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.757+0000 7fcdee800640 1 -- 172.21.15.69:0/1104932543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcde80b2a20 con 0x7fcde81420a0 2024-04-24T08:18:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.758+0000 7fcdee800640 1 -- 172.21.15.69:0/1104932543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcde8075d50 con 0x7fcde81420a0 2024-04-24T08:18:20.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.759+0000 7fcde5600640 1 -- 172.21.15.69:0/1104932543 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcddc044020 con 0x7fcde81420a0 2024-04-24T08:18:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.760+0000 7fcde5600640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcdc806c8f0 0x7fcdc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.760+0000 7fcde5600640 1 -- 172.21.15.69:0/1104932543 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fcddc0ac550 con 0x7fcde81420a0 2024-04-24T08:18:20.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.760+0000 7fcdeca00640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcdc806c8f0 0x7fcdc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:20.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.761+0000 7fcdeca00640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcdc806c8f0 0x7fcdc806edb0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcde8137460 tx=0x7fcdd40079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:20.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.764+0000 7fcde5600640 1 -- 172.21.15.69:0/1104932543 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcddc04b030 con 0x7fcde81420a0 2024-04-24T08:18:20.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.963+0000 7fcdee800640 1 -- 172.21.15.69:0/1104932543 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcde806c470 con 0x7fcde81420a0 2024-04-24T08:18:20.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.964+0000 7fcde5600640 1 -- 172.21.15.69:0/1104932543 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7fcddc07a970 con 0x7fcde81420a0 2024-04-24T08:18:20.964 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:18:20.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.966+0000 7fcdee800640 1 -- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcdc806c8f0 msgr2=0x7fcdc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.966+0000 7fcdee800640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcdc806c8f0 0x7fcdc806edb0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcde8137460 tx=0x7fcdd40079b0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.967+0000 7fcdee800640 1 -- 172.21.15.69:0/1104932543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 msgr2=0x7fcde80b3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.967+0000 7fcdee800640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 0x7fcde80b3220 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009870 tx=0x7fcddc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.967+0000 7fcdee800640 1 -- 172.21.15.69:0/1104932543 shutdown_connections 2024-04-24T08:18:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.967+0000 7fcdee800640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcdc806c8f0 0x7fcdc806edb0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.967+0000 7fcdee800640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcde81606e0 0x7fcde815b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.967+0000 7fcdee800640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcde8142480 0x7fcde8136c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.967+0000 7fcdee800640 1 --2- 172.21.15.69:0/1104932543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcde81420a0 0x7fcde80b3220 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.967+0000 7fcdee800640 1 -- 172.21.15.69:0/1104932543 >> 172.21.15.69:0/1104932543 conn(0x7fcde8067b80 msgr2=0x7fcde8141620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.967+0000 7fcdee800640 1 -- 172.21.15.69:0/1104932543 shutdown_connections 2024-04-24T08:18:20.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:20.967+0000 7fcdee800640 1 -- 172.21.15.69:0/1104932543 wait complete. 2024-04-24T08:18:20.986 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:18:21.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.192+0000 7f2981600640 1 Processor -- start 2024-04-24T08:18:21.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.192+0000 7f2981600640 1 -- start start 2024-04-24T08:18:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2981600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c1420a0 0x7f297c071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2981600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f297c142480 0x7f297c15ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2981600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f297c072450 con 0x7f297c160510 2024-04-24T08:18:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2981600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f297c160960 con 0x7f297c142480 2024-04-24T08:18:21.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2981600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f297c0cf1e0 con 0x7f297c1420a0 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2973e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f297c142480 0x7f297c15ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2980c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f297c160510 0x7f297c15c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60270/0 (socket says 172.21.15.69:60270) 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2980c00640 1 -- 172.21.15.69:0/993908150 learned_addr learned my addr 172.21.15.69:0/993908150 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2973e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f297c142480 0x7f297c15ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35228/0 (socket says 172.21.15.69:35228) 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f297be00640 1 --2- 172.21.15.69:0/993908150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c1420a0 0x7f297c071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2973e00640 1 -- 172.21.15.69:0/993908150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c1420a0 msgr2=0x7f297c071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2973e00640 1 --2- 172.21.15.69:0/993908150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c1420a0 0x7f297c071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.194+0000 7f2973e00640 1 -- 172.21.15.69:0/993908150 >> v1:172.21.15.69:6789/0 conn(0x7f297c160510 legacy=0x7f297c15c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.195+0000 7f2973e00640 1 -- 172.21.15.69:0/993908150 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297c072730 con 0x7f297c142480 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.195+0000 7f297b400640 1 -- 172.21.15.69:0/993908150 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 195364149 0 0) 0x7f297c072450 con 0x7f297c160510 2024-04-24T08:18:21.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.195+0000 7f297be00640 1 --2- 172.21.15.69:0/993908150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c1420a0 0x7f297c071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:21.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.196+0000 7f2973e00640 1 --2- 172.21.15.69:0/993908150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c142480 0x7f297c15ffd0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f29640097a0 tx=0x7f296402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51230e6d64858dfa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.196+0000 7f297b400640 1 -- 172.21.15.69:0/993908150 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964037070 con 0x7f297c142480 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.196+0000 7f297b400640 1 -- 172.21.15.69:0/993908150 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2964032440 con 0x7f297c142480 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.196+0000 7f297b400640 1 -- 172.21.15.69:0/993908150 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964032810 con 0x7f297c142480 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.196+0000 7f2981600640 1 -- 172.21.15.69:0/993908150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c142480 msgr2=0x7f297c15ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.196+0000 7f2981600640 1 --2- 172.21.15.69:0/993908150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c142480 0x7f297c15ffd0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f29640097a0 tx=0x7f296402a180 comp rx=0 tx=0).stop 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.197+0000 7f2981600640 1 -- 172.21.15.69:0/993908150 shutdown_connections 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.197+0000 7f2981600640 1 --2- 172.21.15.69:0/993908150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c142480 0x7f297c15ffd0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.197+0000 7f2981600640 1 --2- 172.21.15.69:0/993908150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c1420a0 0x7f297c071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.197+0000 7f2981600640 1 -- 172.21.15.69:0/993908150 >> 172.21.15.69:0/993908150 conn(0x7f297c067b80 msgr2=0x7f297c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.197+0000 7f2981600640 1 -- 172.21.15.69:0/993908150 shutdown_connections 2024-04-24T08:18:21.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.197+0000 7f2981600640 1 -- 172.21.15.69:0/993908150 wait complete. 2024-04-24T08:18:21.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.198+0000 7f2981600640 1 Processor -- start 2024-04-24T08:18:21.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.198+0000 7f2981600640 1 -- start start 2024-04-24T08:18:21.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.199+0000 7f2981600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c1420a0 0x7f297c0b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.199+0000 7f2981600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c142480 0x7f297c0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.199+0000 7f2981600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f297c160510 0x7f297c1f57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.199+0000 7f2981600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f297c1428e0 con 0x7f297c1420a0 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.199+0000 7f2981600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f297c072cf0 con 0x7f297c160510 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.199+0000 7f2981600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f297c155730 con 0x7f297c142480 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f297be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c1420a0 0x7f297c0b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f297be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c1420a0 0x7f297c0b4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35230/0 (socket says 172.21.15.69:35230) 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f297be00640 1 -- 172.21.15.69:0/3390746934 learned_addr learned my addr 172.21.15.69:0/3390746934 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f2973e00640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c142480 0x7f297c0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f297be00640 1 -- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c142480 msgr2=0x7f297c0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f297be00640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c142480 0x7f297c0b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f297be00640 1 -- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f297c160510 msgr2=0x7f297c1f57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f297be00640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f297c160510 0x7f297c1f57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f297be00640 1 -- 172.21.15.69:0/3390746934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297c0b3b90 con 0x7f297c1420a0 2024-04-24T08:18:21.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f2973e00640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c142480 0x7f297c0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:21.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.200+0000 7f297be00640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c1420a0 0x7f297c0b4fa0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f296c00c7f0 tx=0x7f296c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:21.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.201+0000 7f2978c00640 1 -- 172.21.15.69:0/3390746934 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c015070 con 0x7f297c1420a0 2024-04-24T08:18:21.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.201+0000 7f2978c00640 1 -- 172.21.15.69:0/3390746934 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f296c007a30 con 0x7f297c1420a0 2024-04-24T08:18:21.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.201+0000 7f2978c00640 1 -- 172.21.15.69:0/3390746934 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c007e00 con 0x7f297c1420a0 2024-04-24T08:18:21.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.201+0000 7f2981600640 1 -- 172.21.15.69:0/3390746934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f297c0b3e70 con 0x7f297c1420a0 2024-04-24T08:18:21.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.201+0000 7f2981600640 1 -- 172.21.15.69:0/3390746934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f297c1f4540 con 0x7f297c1420a0 2024-04-24T08:18:21.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.202+0000 7f2981600640 1 -- 172.21.15.69:0/3390746934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f297c075d50 con 0x7f297c1420a0 2024-04-24T08:18:21.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.203+0000 7f2978c00640 1 -- 172.21.15.69:0/3390746934 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f296c018070 con 0x7f297c1420a0 2024-04-24T08:18:21.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.204+0000 7f2978c00640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 0x7f295806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.204+0000 7f2978c00640 1 -- 172.21.15.69:0/3390746934 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f296c08bc60 con 0x7f297c1420a0 2024-04-24T08:18:21.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.205+0000 7f2973e00640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 0x7f295806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.206+0000 7f2973e00640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 0x7f295806edb0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2964034000 tx=0x7f296402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:21.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.208+0000 7f2978c00640 1 -- 172.21.15.69:0/3390746934 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f296c059f60 con 0x7f297c1420a0 2024-04-24T08:18:21.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.407+0000 7f2981600640 1 -- 172.21.15.69:0/3390746934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f297c06c470 con 0x7f297c1420a0 2024-04-24T08:18:21.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.408+0000 7f2978c00640 1 -- 172.21.15.69:0/3390746934 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7f296c059860 con 0x7f297c1420a0 2024-04-24T08:18:21.409 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:18:21.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.410+0000 7f2981600640 1 -- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 msgr2=0x7f295806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.410+0000 7f2981600640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 0x7f295806edb0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2964034000 tx=0x7f296402e000 comp rx=0 tx=0).stop 2024-04-24T08:18:21.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.410+0000 7f2981600640 1 -- 172.21.15.69:0/3390746934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c1420a0 msgr2=0x7f297c0b4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.410+0000 7f2981600640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c1420a0 0x7f297c0b4fa0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f296c00c7f0 tx=0x7f296c00cbf0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.410+0000 7f2981600640 1 -- 172.21.15.69:0/3390746934 shutdown_connections 2024-04-24T08:18:21.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.411+0000 7f2981600640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f295806c8f0 0x7f295806edb0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.411+0000 7f2981600640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f297c160510 0x7f297c1f57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.411+0000 7f2981600640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f297c142480 0x7f297c0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.411+0000 7f2981600640 1 --2- 172.21.15.69:0/3390746934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f297c1420a0 0x7f297c0b4fa0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.411+0000 7f2981600640 1 -- 172.21.15.69:0/3390746934 >> 172.21.15.69:0/3390746934 conn(0x7f297c067b80 msgr2=0x7f297c0b17b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:21.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.411+0000 7f2981600640 1 -- 172.21.15.69:0/3390746934 shutdown_connections 2024-04-24T08:18:21.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.411+0000 7f2981600640 1 -- 172.21.15.69:0/3390746934 wait complete. 2024-04-24T08:18:21.430 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:18:21.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.634+0000 7f0437c00640 1 Processor -- start 2024-04-24T08:18:21.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.635+0000 7f0437c00640 1 -- start start 2024-04-24T08:18:21.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.636+0000 7f0437c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04300d0850 0x7f04300cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.636+0000 7f0437c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04300b07a0 0x7f04300cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.636+0000 7f0437c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f043006bb50 con 0x7f04300d0c30 2024-04-24T08:18:21.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.636+0000 7f0437c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f04300cdc00 con 0x7f04300b07a0 2024-04-24T08:18:21.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.636+0000 7f0437c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04300cf320 con 0x7f04300d0850 2024-04-24T08:18:21.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.636+0000 7f0435c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f04300d0c30 0x7f04300ce110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60284/0 (socket says 172.21.15.69:60284) 2024-04-24T08:18:21.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.636+0000 7f0435c00640 1 -- 172.21.15.69:0/2582767259 learned_addr learned my addr 172.21.15.69:0/2582767259 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f0435200640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04300d0850 0x7f04300cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042fe00640 1 --2- 172.21.15.69:0/2582767259 >> v2:172.21.15.121:3300/0 conn(0x7f04300b07a0 0x7f04300cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1000666361 0 0) 0x7f043006bb50 con 0x7f04300d0c30 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0414003520 con 0x7f04300d0c30 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4263385162 0 0) 0x7f0414003520 con 0x7f04300d0c30 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04300d0850 msgr2=0x7f04300cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04300d0850 0x7f04300cbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 >> v2:172.21.15.121:3300/0 conn(0x7f04300b07a0 msgr2=0x7f04300cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 --2- 172.21.15.69:0/2582767259 >> v2:172.21.15.121:3300/0 conn(0x7f04300b07a0 0x7f04300cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043006be30 con 0x7f04300d0c30 2024-04-24T08:18:21.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f04280019f0 con 0x7f04300d0c30 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 >> v1:172.21.15.69:6789/0 conn(0x7f04300d0c30 legacy=0x7f04300ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04140039b0 0x7f0414003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414004800 0x7f0414004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04140051c0 0x7f041400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04300cdc00 con 0x7f04140051c0 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04300cf320 con 0x7f0414004800 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.637+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f041400c0e0 con 0x7f04140039b0 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.638+0000 7f0435c00640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04140051c0 0x7f041400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.638+0000 7f0435200640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04140039b0 0x7f0414003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.638+0000 7f042fe00640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414004800 0x7f0414004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.638+0000 7f0435c00640 1 -- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04140039b0 msgr2=0x7f0414003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.638+0000 7f0435c00640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04140039b0 0x7f0414003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.638+0000 7f0435c00640 1 -- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414004800 msgr2=0x7f0414004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.638+0000 7f0435c00640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414004800 0x7f0414004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.638+0000 7f0435c00640 1 -- 172.21.15.69:0/2582767259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0414003520 con 0x7f04140051c0 2024-04-24T08:18:21.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.639+0000 7f0435c00640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04140051c0 0x7f041400b9c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f042800a7a0 tx=0x7f0414005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778a2942410eb796 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:21.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.639+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042800a3f0 con 0x7f04140051c0 2024-04-24T08:18:21.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.639+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f042802f070 con 0x7f04140051c0 2024-04-24T08:18:21.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.639+0000 7f042f400640 1 -- 172.21.15.69:0/2582767259 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042802bc60 con 0x7f04140051c0 2024-04-24T08:18:21.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.640+0000 7f0437c00640 1 -- 172.21.15.69:0/2582767259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04140051c0 msgr2=0x7f041400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.640+0000 7f0437c00640 1 --2- 172.21.15.69:0/2582767259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04140051c0 0x7f041400b9c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f042800a7a0 tx=0x7f0414005830 comp rx=0 tx=0).stop 2024-04-24T08:18:21.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.640+0000 7f0435c00640 1 -- 172.21.15.69:0/2582767259 reap_dead start 2024-04-24T08:18:21.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.640+0000 7f0437c00640 1 -- 172.21.15.69:0/2582767259 shutdown_connections 2024-04-24T08:18:21.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.640+0000 7f0437c00640 1 -- 172.21.15.69:0/2582767259 >> 172.21.15.69:0/2582767259 conn(0x7f04300b3050 msgr2=0x7f04300b3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:21.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.640+0000 7f0437c00640 1 -- 172.21.15.69:0/2582767259 shutdown_connections 2024-04-24T08:18:21.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.640+0000 7f0437c00640 1 -- 172.21.15.69:0/2582767259 wait complete. 2024-04-24T08:18:21.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.642+0000 7f0437c00640 1 Processor -- start 2024-04-24T08:18:21.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.642+0000 7f0437c00640 1 -- start start 2024-04-24T08:18:21.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.643+0000 7f0437c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04140051c0 0x7f04301f74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.643+0000 7f0437c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04300afaa0 0x7f04300aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.643+0000 7f0437c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04300b6630 0x7f04300b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.643+0000 7f0437c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f043006c3f0 con 0x7f04300afaa0 2024-04-24T08:18:21.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.643+0000 7f0437c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f043006c6d0 con 0x7f04140051c0 2024-04-24T08:18:21.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.643+0000 7f0437c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04300d10c0 con 0x7f04300b6630 2024-04-24T08:18:21.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.643+0000 7f042fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04300afaa0 0x7f04300aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.643+0000 7f042fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04300afaa0 0x7f04300aff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35242/0 (socket says 172.21.15.69:35242) 2024-04-24T08:18:21.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.643+0000 7f042fe00640 1 -- 172.21.15.69:0/1279021804 learned_addr learned my addr 172.21.15.69:0/1279021804 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:21.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f0435c00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04300b6630 0x7f04300b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f0435200640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04140051c0 0x7f04301f74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f042fe00640 1 -- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04300b6630 msgr2=0x7f04300b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f042fe00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04300b6630 0x7f04300b6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f042fe00640 1 -- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04140051c0 msgr2=0x7f04301f74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f042fe00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04140051c0 0x7f04301f74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f042fe00640 1 -- 172.21.15.69:0/1279021804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043014a7f0 con 0x7f04300afaa0 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f0435200640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04140051c0 0x7f04301f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f042fe00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04300afaa0 0x7f04300aff20 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f041c00f600 tx=0x7f041c00fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f042cc00640 1 -- 172.21.15.69:0/1279021804 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c019020 con 0x7f04300afaa0 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f042cc00640 1 -- 172.21.15.69:0/1279021804 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f041c014870 con 0x7f04300afaa0 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f0437c00640 1 -- 172.21.15.69:0/1279021804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f043014aad0 con 0x7f04300afaa0 2024-04-24T08:18:21.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.644+0000 7f042cc00640 1 -- 172.21.15.69:0/1279021804 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c014c20 con 0x7f04300afaa0 2024-04-24T08:18:21.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.645+0000 7f0437c00640 1 -- 172.21.15.69:0/1279021804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f043014b090 con 0x7f04300afaa0 2024-04-24T08:18:21.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.646+0000 7f0437c00640 1 -- 172.21.15.69:0/1279021804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f043014b470 con 0x7f04300afaa0 2024-04-24T08:18:21.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.647+0000 7f042cc00640 1 -- 172.21.15.69:0/1279021804 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f041c01c020 con 0x7f04300afaa0 2024-04-24T08:18:21.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.647+0000 7f042cc00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041006c8f0 0x7f041006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:21.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.648+0000 7f042cc00640 1 -- 172.21.15.69:0/1279021804 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f041c08c4f0 con 0x7f04300afaa0 2024-04-24T08:18:21.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.648+0000 7f0435200640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041006c8f0 0x7f041006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:21.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.649+0000 7f0435200640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041006c8f0 0x7f041006edb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0420006fd0 tx=0x7f0420008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:21.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.652+0000 7f042cc00640 1 -- 172.21.15.69:0/1279021804 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f043014b470 con 0x7f04300afaa0 2024-04-24T08:18:21.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.853+0000 7f0437c00640 1 -- 172.21.15.69:0/1279021804 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0430068a80 con 0x7f04300afaa0 2024-04-24T08:18:21.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.854+0000 7f042cc00640 1 -- 172.21.15.69:0/1279021804 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f041c023020 con 0x7f04300afaa0 2024-04-24T08:18:21.855 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 -- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041006c8f0 msgr2=0x7f041006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041006c8f0 0x7f041006edb0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0420006fd0 tx=0x7f0420008040 comp rx=0 tx=0).stop 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 -- 172.21.15.69:0/1279021804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04300afaa0 msgr2=0x7f04300aff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04300afaa0 0x7f04300aff20 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f041c00f600 tx=0x7f041c00fac0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 -- 172.21.15.69:0/1279021804 shutdown_connections 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041006c8f0 0x7f041006edb0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f04300afaa0 0x7f04300aff20 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04300b6630 0x7f04300b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 --2- 172.21.15.69:0/1279021804 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04140051c0 0x7f04301f74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 -- 172.21.15.69:0/1279021804 >> 172.21.15.69:0/1279021804 conn(0x7f04300b3050 msgr2=0x7f043015dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 -- 172.21.15.69:0/1279021804 shutdown_connections 2024-04-24T08:18:21.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:21.857+0000 7f0437c00640 1 -- 172.21.15.69:0/1279021804 wait complete. 2024-04-24T08:18:21.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:18:22.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.075+0000 7ff963e00640 1 Processor -- start 2024-04-24T08:18:22.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.076+0000 7ff963e00640 1 -- start start 2024-04-24T08:18:22.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.077+0000 7ff963e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640cca80 0x7ff9640c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.077+0000 7ff963e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff9640cf4d0 0x7ff9640cce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.077+0000 7ff963e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff964161180 con 0x7ff9640d0630 2024-04-24T08:18:22.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.077+0000 7ff963e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff9640cd3a0 con 0x7ff9640cf4d0 2024-04-24T08:18:22.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.077+0000 7ff963e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff964160600 con 0x7ff9640cca80 2024-04-24T08:18:22.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.078+0000 7ff962a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640cca80 0x7ff9640c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.078+0000 7ff962000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff9640cf4d0 0x7ff9640cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.078+0000 7ff963400640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff9640d0630 0x7ff9640d0ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:42984/0 (socket says 172.21.15.69:42984) 2024-04-24T08:18:22.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.078+0000 7ff963400640 1 -- 172.21.15.69:0/1756045825 learned_addr learned my addr 172.21.15.69:0/1756045825 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:22.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.079+0000 7ff961600640 1 -- 172.21.15.69:0/1756045825 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 242511522 0 0) 0x7ff964161180 con 0x7ff9640d0630 2024-04-24T08:18:22.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.079+0000 7ff962a00640 1 -- 172.21.15.69:0/1756045825 >> v2:172.21.15.121:3300/0 conn(0x7ff9640cf4d0 msgr2=0x7ff9640cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:22.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.079+0000 7ff962a00640 1 --2- 172.21.15.69:0/1756045825 >> v2:172.21.15.121:3300/0 conn(0x7ff9640cf4d0 0x7ff9640cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.079+0000 7ff962a00640 1 -- 172.21.15.69:0/1756045825 >> v1:172.21.15.121:6789/0 conn(0x7ff9640d0630 legacy=0x7ff9640d0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.079+0000 7ff962a00640 1 -- 172.21.15.69:0/1756045825 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9641608e0 con 0x7ff9640cca80 2024-04-24T08:18:22.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.080+0000 7ff962a00640 1 --2- 172.21.15.69:0/1756045825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640cca80 0x7ff9640c5f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff9540097a0 tx=0x7ff95402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28361c1c500720a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:22.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.081+0000 7ff961600640 1 -- 172.21.15.69:0/1756045825 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff954037070 con 0x7ff9640cca80 2024-04-24T08:18:22.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.081+0000 7ff961600640 1 -- 172.21.15.69:0/1756045825 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff954032440 con 0x7ff9640cca80 2024-04-24T08:18:22.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.081+0000 7ff961600640 1 -- 172.21.15.69:0/1756045825 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff954032810 con 0x7ff9640cca80 2024-04-24T08:18:22.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.081+0000 7ff963e00640 1 -- 172.21.15.69:0/1756045825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640cca80 msgr2=0x7ff9640c5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.081+0000 7ff963e00640 1 --2- 172.21.15.69:0/1756045825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640cca80 0x7ff9640c5f80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff9540097a0 tx=0x7ff95402a180 comp rx=0 tx=0).stop 2024-04-24T08:18:22.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.081+0000 7ff963e00640 1 -- 172.21.15.69:0/1756045825 shutdown_connections 2024-04-24T08:18:22.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.081+0000 7ff963e00640 1 --2- 172.21.15.69:0/1756045825 >> v2:172.21.15.121:3300/0 conn(0x7ff9640cf4d0 0x7ff9640cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.081+0000 7ff963e00640 1 --2- 172.21.15.69:0/1756045825 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640cca80 0x7ff9640c5f80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.081+0000 7ff963e00640 1 -- 172.21.15.69:0/1756045825 >> 172.21.15.69:0/1756045825 conn(0x7ff96406fd10 msgr2=0x7ff964070120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:22.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.081+0000 7ff963e00640 1 -- 172.21.15.69:0/1756045825 shutdown_connections 2024-04-24T08:18:22.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.082+0000 7ff963e00640 1 -- 172.21.15.69:0/1756045825 wait complete. 2024-04-24T08:18:22.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.083+0000 7ff963e00640 1 Processor -- start 2024-04-24T08:18:22.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.083+0000 7ff963e00640 1 -- start start 2024-04-24T08:18:22.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.084+0000 7ff963e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9640cca80 0x7ff9640cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.084+0000 7ff963e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640cf4d0 0x7ff96415c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.084+0000 7ff963e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640d0630 0x7ff96415e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.084+0000 7ff963e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9640cf930 con 0x7ff9640cf4d0 2024-04-24T08:18:22.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.084+0000 7ff963e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff964160ea0 con 0x7ff9640cca80 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.084+0000 7ff963e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff9640d1e30 con 0x7ff9640d0630 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff962000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640cf4d0 0x7ff96415c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff962000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640cf4d0 0x7ff96415c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35252/0 (socket says 172.21.15.69:35252) 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff962000640 1 -- 172.21.15.69:0/3934865884 learned_addr learned my addr 172.21.15.69:0/3934865884 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff962000640 1 -- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640d0630 msgr2=0x7ff96415e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff963400640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640d0630 0x7ff96415e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff962a00640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9640cca80 0x7ff9640cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff962000640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640d0630 0x7ff96415e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff962000640 1 -- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9640cca80 msgr2=0x7ff9640cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff962000640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9640cca80 0x7ff9640cbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff962000640 1 -- 172.21.15.69:0/3934865884 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96415d270 con 0x7ff9640cf4d0 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff963400640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640d0630 0x7ff96415e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.085+0000 7ff962000640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640cf4d0 0x7ff96415c900 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7ff95000e9e0 tx=0x7ff95000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:22.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.086+0000 7ff95b400640 1 -- 172.21.15.69:0/3934865884 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950017070 con 0x7ff9640cf4d0 2024-04-24T08:18:22.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.086+0000 7ff95b400640 1 -- 172.21.15.69:0/3934865884 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff95000c9d0 con 0x7ff9640cf4d0 2024-04-24T08:18:22.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.086+0000 7ff963e00640 1 -- 172.21.15.69:0/3934865884 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff96415d550 con 0x7ff9640cf4d0 2024-04-24T08:18:22.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.086+0000 7ff95b400640 1 -- 172.21.15.69:0/3934865884 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95000cdc0 con 0x7ff9640cf4d0 2024-04-24T08:18:22.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.086+0000 7ff963e00640 1 -- 172.21.15.69:0/3934865884 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff96415db10 con 0x7ff9640cf4d0 2024-04-24T08:18:22.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.087+0000 7ff963e00640 1 -- 172.21.15.69:0/3934865884 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff964075d50 con 0x7ff9640cf4d0 2024-04-24T08:18:22.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.088+0000 7ff95b400640 1 -- 172.21.15.69:0/3934865884 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff95001a070 con 0x7ff9640cf4d0 2024-04-24T08:18:22.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.089+0000 7ff95b400640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff94406c8f0 0x7ff94406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.089+0000 7ff95b400640 1 -- 172.21.15.69:0/3934865884 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7ff95008be10 con 0x7ff9640cf4d0 2024-04-24T08:18:22.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.090+0000 7ff962a00640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff94406c8f0 0x7ff94406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.091+0000 7ff962a00640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff94406c8f0 0x7ff94406edb0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ff954034000 tx=0x7ff95402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:22.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.094+0000 7ff95b400640 1 -- 172.21.15.69:0/3934865884 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff95005a2d0 con 0x7ff9640cf4d0 2024-04-24T08:18:22.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.309+0000 7ff963e00640 1 -- 172.21.15.69:0/3934865884 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff964070830 con 0x7ff9640cf4d0 2024-04-24T08:18:22.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.310+0000 7ff95b400640 1 -- 172.21.15.69:0/3934865884 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v97) ==== 80+0+0 (secure 0 0 0) 0x7ff950059bd0 con 0x7ff9640cf4d0 2024-04-24T08:18:22.311 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:18:22.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.313+0000 7ff963e00640 1 -- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff94406c8f0 msgr2=0x7ff94406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.313+0000 7ff963e00640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff94406c8f0 0x7ff94406edb0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ff954034000 tx=0x7ff95402e000 comp rx=0 tx=0).stop 2024-04-24T08:18:22.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.313+0000 7ff963e00640 1 -- 172.21.15.69:0/3934865884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640cf4d0 msgr2=0x7ff96415c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.313+0000 7ff963e00640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640cf4d0 0x7ff96415c900 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7ff95000e9e0 tx=0x7ff95000eea0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.314+0000 7ff963e00640 1 -- 172.21.15.69:0/3934865884 shutdown_connections 2024-04-24T08:18:22.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.314+0000 7ff963e00640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff94406c8f0 0x7ff94406edb0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.314+0000 7ff963e00640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9640cf4d0 0x7ff96415c900 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.314+0000 7ff963e00640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9640d0630 0x7ff96415e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.314+0000 7ff963e00640 1 --2- 172.21.15.69:0/3934865884 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff9640cca80 0x7ff9640cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.314+0000 7ff963e00640 1 -- 172.21.15.69:0/3934865884 >> 172.21.15.69:0/3934865884 conn(0x7ff96406fd10 msgr2=0x7ff9640c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:22.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.314+0000 7ff963e00640 1 -- 172.21.15.69:0/3934865884 shutdown_connections 2024-04-24T08:18:22.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.314+0000 7ff963e00640 1 -- 172.21.15.69:0/3934865884 wait complete. 2024-04-24T08:18:22.334 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:18:22.334 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:22.384 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:22.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.626+0000 7f19de800640 1 Processor -- start 2024-04-24T08:18:22.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.626+0000 7f19de800640 1 -- start start 2024-04-24T08:18:22.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.628+0000 7f19de800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19d80b06f0 0x7f19d8070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.628+0000 7f19de800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f19d80b0ad0 0x7f19d8070a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.628+0000 7f19de800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19d806ef50 con 0x7f19d806ffd0 2024-04-24T08:18:22.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.628+0000 7f19de800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f19d806f230 con 0x7f19d80b06f0 2024-04-24T08:18:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.628+0000 7f19de800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f19d806f510 con 0x7f19d80b0ad0 2024-04-24T08:18:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.628+0000 7f19d7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19d80b06f0 0x7f19d8070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.628+0000 7f19d7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f19d80b06f0 0x7f19d8070520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35260/0 (socket says 172.21.15.69:35260) 2024-04-24T08:18:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.628+0000 7f19d7400640 1 -- 172.21.15.69:0/2343916915 learned_addr learned my addr 172.21.15.69:0/2343916915 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.629+0000 7f19d7400640 1 -- 172.21.15.69:0/2343916915 >> v2:172.21.15.121:3300/0 conn(0x7f19d80b0ad0 msgr2=0x7f19d8070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.629+0000 7f19cea00640 1 --2- 172.21.15.69:0/2343916915 >> v2:172.21.15.121:3300/0 conn(0x7f19d80b0ad0 0x7f19d8070a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.629+0000 7f19d7400640 1 --2- 172.21.15.69:0/2343916915 >> v2:172.21.15.121:3300/0 conn(0x7f19d80b0ad0 0x7f19d8070a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.629+0000 7f19d7400640 1 -- 172.21.15.69:0/2343916915 >> v1:172.21.15.121:6789/0 conn(0x7f19d806ffd0 legacy=0x7f19d806ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.629+0000 7f19d7400640 1 -- 172.21.15.69:0/2343916915 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d806f7f0 con 0x7f19d80b06f0 2024-04-24T08:18:22.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.629+0000 7f19cea00640 1 --2- 172.21.15.69:0/2343916915 >> v2:172.21.15.121:3300/0 conn(0x7f19d80b0ad0 0x7f19d8070a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:22.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.630+0000 7f19d7400640 1 --2- 172.21.15.69:0/2343916915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d80b06f0 0x7f19d8070520 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f19d80c8520 tx=0x7f19c802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859f459aa0aa82c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.630+0000 7f19d6a00640 1 -- 172.21.15.69:0/2343916915 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8037070 con 0x7f19d80b06f0 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.630+0000 7f19d6a00640 1 -- 172.21.15.69:0/2343916915 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c8032440 con 0x7f19d80b06f0 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.630+0000 7f19d6a00640 1 -- 172.21.15.69:0/2343916915 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8032810 con 0x7f19d80b06f0 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.631+0000 7f19de800640 1 -- 172.21.15.69:0/2343916915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d80b06f0 msgr2=0x7f19d8070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.631+0000 7f19de800640 1 --2- 172.21.15.69:0/2343916915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d80b06f0 0x7f19d8070520 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f19d80c8520 tx=0x7f19c802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.631+0000 7f19de800640 1 -- 172.21.15.69:0/2343916915 shutdown_connections 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.631+0000 7f19de800640 1 --2- 172.21.15.69:0/2343916915 >> v2:172.21.15.121:3300/0 conn(0x7f19d80b0ad0 0x7f19d8070a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.631+0000 7f19de800640 1 --2- 172.21.15.69:0/2343916915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d80b06f0 0x7f19d8070520 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.631+0000 7f19de800640 1 -- 172.21.15.69:0/2343916915 >> 172.21.15.69:0/2343916915 conn(0x7f19d809a9d0 msgr2=0x7f19d80d14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.631+0000 7f19de800640 1 -- 172.21.15.69:0/2343916915 shutdown_connections 2024-04-24T08:18:22.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.631+0000 7f19de800640 1 -- 172.21.15.69:0/2343916915 wait complete. 2024-04-24T08:18:22.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.633+0000 7f19de800640 1 Processor -- start 2024-04-24T08:18:22.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.633+0000 7f19de800640 1 -- start start 2024-04-24T08:18:22.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.634+0000 7f19de800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d806ffd0 0x7f19d814d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.634+0000 7f19de800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80b06f0 0x7f19d80b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.634+0000 7f19de800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80b0ad0 0x7f19d815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.634+0000 7f19de800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f19d806c130 con 0x7f19d806ffd0 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.634+0000 7f19de800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f19d80d1cb0 con 0x7f19d80b0ad0 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.634+0000 7f19de800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f19d806c2b0 con 0x7f19d80b06f0 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.634+0000 7f19d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d806ffd0 0x7f19d814d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.634+0000 7f19d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d806ffd0 0x7f19d814d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35276/0 (socket says 172.21.15.69:35276) 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.634+0000 7f19d7400640 1 -- 172.21.15.69:0/2675233481 learned_addr learned my addr 172.21.15.69:0/2675233481 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19d7400640 1 -- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80b06f0 msgr2=0x7f19d80b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19d7400640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80b06f0 0x7f19d80b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19d7400640 1 -- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80b0ad0 msgr2=0x7f19d815c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19d7400640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80b0ad0 0x7f19d815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19d7400640 1 -- 172.21.15.69:0/2675233481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d80b1910 con 0x7f19d806ffd0 2024-04-24T08:18:22.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19d7400640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d806ffd0 0x7f19d814d6d0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f19c8000c00 tx=0x7f19c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:22.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19cfe00640 1 -- 172.21.15.69:0/2675233481 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8041020 con 0x7f19d806ffd0 2024-04-24T08:18:22.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19cfe00640 1 -- 172.21.15.69:0/2675233481 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c8030990 con 0x7f19d806ffd0 2024-04-24T08:18:22.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19cfe00640 1 -- 172.21.15.69:0/2675233481 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8037040 con 0x7f19d806ffd0 2024-04-24T08:18:22.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19de800640 1 -- 172.21.15.69:0/2675233481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d80b1bb0 con 0x7f19d806ffd0 2024-04-24T08:18:22.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.635+0000 7f19de800640 1 -- 172.21.15.69:0/2675233481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d80b2170 con 0x7f19d806ffd0 2024-04-24T08:18:22.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.637+0000 7f19de800640 1 -- 172.21.15.69:0/2675233481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d8069190 con 0x7f19d806ffd0 2024-04-24T08:18:22.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.638+0000 7f19cfe00640 1 -- 172.21.15.69:0/2675233481 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f19c8044020 con 0x7f19d806ffd0 2024-04-24T08:18:22.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.638+0000 7f19cfe00640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19b406c8f0 0x7f19b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:22.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.639+0000 7f19cfe00640 1 -- 172.21.15.69:0/2675233481 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f19c80ac240 con 0x7f19d806ffd0 2024-04-24T08:18:22.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.639+0000 7f19cea00640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19b406c8f0 0x7f19b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:22.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.640+0000 7f19cea00640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19b406c8f0 0x7f19b406edb0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f19c0006fd0 tx=0x7f19c0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:22.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.643+0000 7f19cfe00640 1 -- 172.21.15.69:0/2675233481 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f19c804b070 con 0x7f19d806ffd0 2024-04-24T08:18:22.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.847+0000 7f19de800640 1 -- 172.21.15.69:0/2675233481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19d8074e60 con 0x7f19d806ffd0 2024-04-24T08:18:22.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.849+0000 7f19cfe00640 1 -- 172.21.15.69:0/2675233481 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+16584 (secure 0 0 0) 0x7f19c807a670 con 0x7f19d806ffd0 2024-04-24T08:18:22.851 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:22.851 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":97,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:18:11.928428+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:08.156864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:09.368519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:22.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.852+0000 7f19de800640 1 -- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19b406c8f0 msgr2=0x7f19b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.852+0000 7f19de800640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19b406c8f0 0x7f19b406edb0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f19c0006fd0 tx=0x7f19c0008040 comp rx=0 tx=0).stop 2024-04-24T08:18:22.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.852+0000 7f19de800640 1 -- 172.21.15.69:0/2675233481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d806ffd0 msgr2=0x7f19d814d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:22.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.852+0000 7f19de800640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d806ffd0 0x7f19d814d6d0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f19c8000c00 tx=0x7f19c80046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.853+0000 7f19de800640 1 -- 172.21.15.69:0/2675233481 shutdown_connections 2024-04-24T08:18:22.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.853+0000 7f19de800640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19b406c8f0 0x7f19b406edb0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.853+0000 7f19de800640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19d80b0ad0 0x7f19d815c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.853+0000 7f19de800640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f19d80b06f0 0x7f19d80b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.853+0000 7f19de800640 1 --2- 172.21.15.69:0/2675233481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19d806ffd0 0x7f19d814d6d0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:22.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.853+0000 7f19de800640 1 -- 172.21.15.69:0/2675233481 >> 172.21.15.69:0/2675233481 conn(0x7f19d809a9d0 msgr2=0x7f19d815dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:22.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.853+0000 7f19de800640 1 -- 172.21.15.69:0/2675233481 shutdown_connections 2024-04-24T08:18:22.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:22.853+0000 7f19de800640 1 -- 172.21.15.69:0/2675233481 wait complete. 2024-04-24T08:18:22.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:08.156864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:09.368519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:18:22.873 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:18:23.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.081+0000 7fd34be00640 1 Processor -- start 2024-04-24T08:18:23.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.082+0000 7fd34be00640 1 -- start start 2024-04-24T08:18:23.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.083+0000 7fd34be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd34c15e980 0x7fd34c162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.083+0000 7fd34be00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd34c0d0da0 0x7fd34c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.083+0000 7fd34be00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd34c15ed60 con 0x7fd34c0d0190 2024-04-24T08:18:23.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.083+0000 7fd34be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd34c0d1580 con 0x7fd34c15e980 2024-04-24T08:18:23.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.083+0000 7fd34be00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd34c0d1860 con 0x7fd34c0d0da0 2024-04-24T08:18:23.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.083+0000 7fd34aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd34c15e980 0x7fd34c162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.084+0000 7fd34aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd34c15e980 0x7fd34c162f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35290/0 (socket says 172.21.15.69:35290) 2024-04-24T08:18:23.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.084+0000 7fd34aa00640 1 -- 172.21.15.69:0/920095449 learned_addr learned my addr 172.21.15.69:0/920095449 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:23.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.084+0000 7fd34a000640 1 --2- 172.21.15.69:0/920095449 >> v2:172.21.15.121:3300/0 conn(0x7fd34c0d0da0 0x7fd34c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.084+0000 7fd34aa00640 1 -- 172.21.15.69:0/920095449 >> v2:172.21.15.121:3300/0 conn(0x7fd34c0d0da0 msgr2=0x7fd34c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:23.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.084+0000 7fd34aa00640 1 --2- 172.21.15.69:0/920095449 >> v2:172.21.15.121:3300/0 conn(0x7fd34c0d0da0 0x7fd34c0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.084+0000 7fd34aa00640 1 -- 172.21.15.69:0/920095449 >> v1:172.21.15.121:6789/0 conn(0x7fd34c0d0190 legacy=0x7fd34c06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:23.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.084+0000 7fd34aa00640 1 -- 172.21.15.69:0/920095449 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34c0cf450 con 0x7fd34c15e980 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.085+0000 7fd34aa00640 1 --2- 172.21.15.69:0/920095449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c15e980 0x7fd34c162f90 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fd34c06c480 tx=0x7fd33402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e4ef5448a62161 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.085+0000 7fd349600640 1 -- 172.21.15.69:0/920095449 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334037070 con 0x7fd34c15e980 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.085+0000 7fd349600640 1 -- 172.21.15.69:0/920095449 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd334032440 con 0x7fd34c15e980 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.085+0000 7fd349600640 1 -- 172.21.15.69:0/920095449 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334032810 con 0x7fd34c15e980 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.086+0000 7fd34be00640 1 -- 172.21.15.69:0/920095449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c15e980 msgr2=0x7fd34c162f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.086+0000 7fd34be00640 1 --2- 172.21.15.69:0/920095449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c15e980 0x7fd34c162f90 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fd34c06c480 tx=0x7fd33402a2c0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.086+0000 7fd34be00640 1 -- 172.21.15.69:0/920095449 shutdown_connections 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.086+0000 7fd34be00640 1 --2- 172.21.15.69:0/920095449 >> v2:172.21.15.121:3300/0 conn(0x7fd34c0d0da0 0x7fd34c0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.086+0000 7fd34be00640 1 --2- 172.21.15.69:0/920095449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c15e980 0x7fd34c162f90 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.086+0000 7fd34be00640 1 -- 172.21.15.69:0/920095449 >> 172.21.15.69:0/920095449 conn(0x7fd34c09d920 msgr2=0x7fd34c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:23.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.086+0000 7fd34be00640 1 -- 172.21.15.69:0/920095449 shutdown_connections 2024-04-24T08:18:23.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.086+0000 7fd34be00640 1 -- 172.21.15.69:0/920095449 wait complete. 2024-04-24T08:18:23.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.088+0000 7fd34be00640 1 Processor -- start 2024-04-24T08:18:23.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.088+0000 7fd34be00640 1 -- start start 2024-04-24T08:18:23.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.089+0000 7fd34be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c0d0190 0x7fd34c152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.089+0000 7fd34be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd34c0d0da0 0x7fd34c0b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.089+0000 7fd34be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd34c15e980 0x7fd34c0ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.089+0000 7fd34be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd34c0d1360 con 0x7fd34c0d0190 2024-04-24T08:18:23.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.089+0000 7fd34be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd34c0d11e0 con 0x7fd34c0d0da0 2024-04-24T08:18:23.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.089+0000 7fd34be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd34c0cf070 con 0x7fd34c15e980 2024-04-24T08:18:23.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.089+0000 7fd34aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c0d0190 0x7fd34c152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.089+0000 7fd34aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c0d0190 0x7fd34c152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35302/0 (socket says 172.21.15.69:35302) 2024-04-24T08:18:23.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.089+0000 7fd34aa00640 1 -- 172.21.15.69:0/792334654 learned_addr learned my addr 172.21.15.69:0/792334654 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:23.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd34aa00640 1 -- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd34c15e980 msgr2=0x7fd34c0ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd34aa00640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd34c15e980 0x7fd34c0ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd34aa00640 1 -- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd34c0d0da0 msgr2=0x7fd34c0b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd34aa00640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd34c0d0da0 0x7fd34c0b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd34aa00640 1 -- 172.21.15.69:0/792334654 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd34c0b71f0 con 0x7fd34c0d0190 2024-04-24T08:18:23.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd34aa00640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c0d0190 0x7fd34c152f10 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fd33402a300 tx=0x7fd3340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:23.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd342a00640 1 -- 172.21.15.69:0/792334654 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334041020 con 0x7fd34c0d0190 2024-04-24T08:18:23.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd342a00640 1 -- 172.21.15.69:0/792334654 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3340306c0 con 0x7fd34c0d0190 2024-04-24T08:18:23.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd34be00640 1 -- 172.21.15.69:0/792334654 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd34c0b74d0 con 0x7fd34c0d0190 2024-04-24T08:18:23.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd342a00640 1 -- 172.21.15.69:0/792334654 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd334037040 con 0x7fd34c0d0190 2024-04-24T08:18:23.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.090+0000 7fd34be00640 1 -- 172.21.15.69:0/792334654 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd34c0b7a90 con 0x7fd34c0d0190 2024-04-24T08:18:23.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.091+0000 7fd34be00640 1 -- 172.21.15.69:0/792334654 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd34c075d50 con 0x7fd34c0d0190 2024-04-24T08:18:23.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.092+0000 7fd342a00640 1 -- 172.21.15.69:0/792334654 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd334044020 con 0x7fd34c0d0190 2024-04-24T08:18:23.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.093+0000 7fd342a00640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd32806c920 0x7fd32806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.093+0000 7fd342a00640 1 -- 172.21.15.69:0/792334654 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fd3340ac3d0 con 0x7fd34c0d0190 2024-04-24T08:18:23.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.094+0000 7fd34a000640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd32806c920 0x7fd32806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.095+0000 7fd34a000640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd32806c920 0x7fd32806ede0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd33c006fd0 tx=0x7fd33c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:23.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.098+0000 7fd342a00640 1 -- 172.21.15.69:0/792334654 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd33404b030 con 0x7fd34c0d0190 2024-04-24T08:18:23.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.294+0000 7fd34be00640 1 -- 172.21.15.69:0/792334654 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd34c06d150 con 0x7fd34c0d0190 2024-04-24T08:18:23.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.295+0000 7fd342a00640 1 -- 172.21.15.69:0/792334654 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7fd33407a760 con 0x7fd34c0d0190 2024-04-24T08:18:23.296 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 -- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd32806c920 msgr2=0x7fd32806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd32806c920 0x7fd32806ede0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd33c006fd0 tx=0x7fd33c008040 comp rx=0 tx=0).stop 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 -- 172.21.15.69:0/792334654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c0d0190 msgr2=0x7fd34c152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c0d0190 0x7fd34c152f10 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fd33402a300 tx=0x7fd3340046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 -- 172.21.15.69:0/792334654 shutdown_connections 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd32806c920 0x7fd32806ede0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd34c15e980 0x7fd34c0ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd34c0d0da0 0x7fd34c0b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 --2- 172.21.15.69:0/792334654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd34c0d0190 0x7fd34c152f10 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 -- 172.21.15.69:0/792334654 >> 172.21.15.69:0/792334654 conn(0x7fd34c09d920 msgr2=0x7fd34c0b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:23.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 -- 172.21.15.69:0/792334654 shutdown_connections 2024-04-24T08:18:23.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.298+0000 7fd34be00640 1 -- 172.21.15.69:0/792334654 wait complete. 2024-04-24T08:18:23.317 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:18:23.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.517+0000 7f965f200640 1 Processor -- start 2024-04-24T08:18:23.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.517+0000 7f965f200640 1 -- start start 2024-04-24T08:18:23.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f965f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96580b3e90 0x7f96580b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f965f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f96580ce250 0x7f96580ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f965f200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f965815cc00 con 0x7f965815c820 2024-04-24T08:18:23.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f965f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f96580b4be0 con 0x7f96580b3e90 2024-04-24T08:18:23.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f965f200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f965806ed30 con 0x7f96580ce250 2024-04-24T08:18:23.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f9657400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96580b3e90 0x7f96580b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f9657400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f96580b3e90 0x7f96580b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35304/0 (socket says 172.21.15.69:35304) 2024-04-24T08:18:23.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f9657400640 1 -- 172.21.15.69:0/3903299723 learned_addr learned my addr 172.21.15.69:0/3903299723 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:23.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f9657400640 1 -- 172.21.15.69:0/3903299723 >> v2:172.21.15.121:3300/0 conn(0x7f96580ce250 msgr2=0x7f96580ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:23.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f9657400640 1 --2- 172.21.15.69:0/3903299723 >> v2:172.21.15.121:3300/0 conn(0x7f96580ce250 0x7f96580ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f9657400640 1 -- 172.21.15.69:0/3903299723 >> v1:172.21.15.121:6789/0 conn(0x7f965815c820 legacy=0x7f965806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:23.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.519+0000 7f9657400640 1 -- 172.21.15.69:0/3903299723 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96580b4e10 con 0x7f96580b3e90 2024-04-24T08:18:23.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.520+0000 7f9657400640 1 --2- 172.21.15.69:0/3903299723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96580b3e90 0x7f96580b46a0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f9658161420 tx=0x7f964402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c4d2f278d8d19e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:23.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f9656a00640 1 -- 172.21.15.69:0/3903299723 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9644037070 con 0x7f96580b3e90 2024-04-24T08:18:23.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f9656a00640 1 -- 172.21.15.69:0/3903299723 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9644032440 con 0x7f96580b3e90 2024-04-24T08:18:23.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f9656a00640 1 -- 172.21.15.69:0/3903299723 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9644032810 con 0x7f96580b3e90 2024-04-24T08:18:23.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f965f200640 1 -- 172.21.15.69:0/3903299723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96580b3e90 msgr2=0x7f96580b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f965f200640 1 --2- 172.21.15.69:0/3903299723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96580b3e90 0x7f96580b46a0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f9658161420 tx=0x7f964402a230 comp rx=0 tx=0).stop 2024-04-24T08:18:23.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f965f200640 1 -- 172.21.15.69:0/3903299723 shutdown_connections 2024-04-24T08:18:23.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f965f200640 1 --2- 172.21.15.69:0/3903299723 >> v2:172.21.15.121:3300/0 conn(0x7f96580ce250 0x7f96580ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f965f200640 1 --2- 172.21.15.69:0/3903299723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f96580b3e90 0x7f96580b46a0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f965f200640 1 -- 172.21.15.69:0/3903299723 >> 172.21.15.69:0/3903299723 conn(0x7f96580d0350 msgr2=0x7f96580d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:23.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f965f200640 1 -- 172.21.15.69:0/3903299723 shutdown_connections 2024-04-24T08:18:23.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.521+0000 7f965f200640 1 -- 172.21.15.69:0/3903299723 wait complete. 2024-04-24T08:18:23.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.523+0000 7f965f200640 1 Processor -- start 2024-04-24T08:18:23.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.523+0000 7f965f200640 1 -- start start 2024-04-24T08:18:23.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.524+0000 7f965f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96580b3e90 0x7f965814f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.524+0000 7f965f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96580ce250 0x7f96580cbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.524+0000 7f965f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965815c820 0x7f96580b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.524+0000 7f965f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f965806f2b0 con 0x7f965815c820 2024-04-24T08:18:23.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.524+0000 7f965f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f965806efd0 con 0x7f96580b3e90 2024-04-24T08:18:23.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.524+0000 7f965f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f965806fee0 con 0x7f96580ce250 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.524+0000 7f965d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965815c820 0x7f96580b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.524+0000 7f965d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965815c820 0x7f96580b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35310/0 (socket says 172.21.15.69:35310) 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.524+0000 7f965d200640 1 -- 172.21.15.69:0/3338012039 learned_addr learned my addr 172.21.15.69:0/3338012039 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f9657400640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96580ce250 0x7f96580cbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f9657e00640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96580b3e90 0x7f965814f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f965d200640 1 -- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96580ce250 msgr2=0x7f96580cbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f965d200640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96580ce250 0x7f96580cbdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f965d200640 1 -- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96580b3e90 msgr2=0x7f965814f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f965d200640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96580b3e90 0x7f965814f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f965d200640 1 -- 172.21.15.69:0/3338012039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965814cf50 con 0x7f965815c820 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f9657e00640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96580b3e90 0x7f965814f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f965d200640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965815c820 0x7f96580b02f0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f965000b970 tx=0x7f965000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:23.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f964fe00640 1 -- 172.21.15.69:0/3338012039 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650019070 con 0x7f965815c820 2024-04-24T08:18:23.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f964fe00640 1 -- 172.21.15.69:0/3338012039 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f965000cb60 con 0x7f965815c820 2024-04-24T08:18:23.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f964fe00640 1 -- 172.21.15.69:0/3338012039 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650014440 con 0x7f965815c820 2024-04-24T08:18:23.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.525+0000 7f965f200640 1 -- 172.21.15.69:0/3338012039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965814d230 con 0x7f965815c820 2024-04-24T08:18:23.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.526+0000 7f965f200640 1 -- 172.21.15.69:0/3338012039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96580b0830 con 0x7f965815c820 2024-04-24T08:18:23.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.527+0000 7f965f200640 1 -- 172.21.15.69:0/3338012039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9658068a80 con 0x7f965815c820 2024-04-24T08:18:23.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.527+0000 7f964fe00640 1 -- 172.21.15.69:0/3338012039 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f965001c070 con 0x7f965815c820 2024-04-24T08:18:23.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.528+0000 7f964fe00640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f963806c920 0x7f963806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.528+0000 7f964fe00640 1 -- 172.21.15.69:0/3338012039 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f965008b920 con 0x7f965815c820 2024-04-24T08:18:23.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.529+0000 7f9657e00640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f963806c920 0x7f963806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.530+0000 7f9657e00640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f963806c920 0x7f963806ede0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9648007fd0 tx=0x7f9648009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:23.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.533+0000 7f964fe00640 1 -- 172.21.15.69:0/3338012039 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9650059cb0 con 0x7f965815c820 2024-04-24T08:18:23.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.714+0000 7f965f200640 1 -- 172.21.15.69:0/3338012039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f96580cddc0 con 0x7f965815c820 2024-04-24T08:18:23.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.715+0000 7f964fe00640 1 -- 172.21.15.69:0/3338012039 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7f96500595b0 con 0x7f965815c820 2024-04-24T08:18:23.716 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 -- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f963806c920 msgr2=0x7f963806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f963806c920 0x7f963806ede0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f9648007fd0 tx=0x7f9648009040 comp rx=0 tx=0).stop 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 -- 172.21.15.69:0/3338012039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965815c820 msgr2=0x7f96580b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965815c820 0x7f96580b02f0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f965000b970 tx=0x7f965000be30 comp rx=0 tx=0).stop 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 -- 172.21.15.69:0/3338012039 shutdown_connections 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f963806c920 0x7f963806ede0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f965815c820 0x7f96580b02f0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f96580ce250 0x7f96580cbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 --2- 172.21.15.69:0/3338012039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96580b3e90 0x7f965814f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 -- 172.21.15.69:0/3338012039 >> 172.21.15.69:0/3338012039 conn(0x7f96580d0350 msgr2=0x7f96580b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:23.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 -- 172.21.15.69:0/3338012039 shutdown_connections 2024-04-24T08:18:23.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.718+0000 7f965f200640 1 -- 172.21.15.69:0/3338012039 wait complete. 2024-04-24T08:18:23.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:18:23.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.932+0000 7fa32b200640 1 Processor -- start 2024-04-24T08:18:23.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.932+0000 7fa32b200640 1 -- start start 2024-04-24T08:18:23.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.933+0000 7fa32b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c15ea60 0x7fa32c0b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.933+0000 7fa32b200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa32c0d09f0 0x7fa32c06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.933+0000 7fa32b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa32c0ce910 con 0x7fa32c0cdf30 2024-04-24T08:18:23.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.933+0000 7fa32b200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa32c0cebf0 con 0x7fa32c0d09f0 2024-04-24T08:18:23.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.933+0000 7fa32b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa32c0ceed0 con 0x7fa32c15ea60 2024-04-24T08:18:23.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.934+0000 7fa329400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa32c0cdf30 0x7fa32c0d02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:60294/0 (socket says 172.21.15.69:60294) 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.934+0000 7fa329400640 1 -- 172.21.15.69:0/957592364 learned_addr learned my addr 172.21.15.69:0/957592364 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.934+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2838777726 0 0) 0x7fa32c0ce910 con 0x7fa32c0cdf30 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.934+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa310003520 con 0x7fa32c0cdf30 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.934+0000 7fa329e00640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c15ea60 0x7fa32c0b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.934+0000 7fa32a800640 1 --2- 172.21.15.69:0/957592364 >> v2:172.21.15.121:3300/0 conn(0x7fa32c0d09f0 0x7fa32c06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.934+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1602078012 0 0) 0x7fa310003520 con 0x7fa32c0cdf30 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c15ea60 msgr2=0x7fa32c0b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c15ea60 0x7fa32c0b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 >> v2:172.21.15.121:3300/0 conn(0x7fa32c0d09f0 msgr2=0x7fa32c06df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 --2- 172.21.15.69:0/957592364 >> v2:172.21.15.121:3300/0 conn(0x7fa32c0d09f0 0x7fa32c06df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32c0cf1b0 con 0x7fa32c0cdf30 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fa318001670 con 0x7fa32c0cdf30 2024-04-24T08:18:23.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 >> v1:172.21.15.69:6789/0 conn(0x7fa32c0cdf30 legacy=0x7fa32c0d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3100039b0 0x7fa310003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c15ea60 0x7fa310004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa310004d40 0x7fa3100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c0cebf0 con 0x7fa310004d40 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c0ceed0 con 0x7fa3100039b0 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa31000bb50 con 0x7fa32c15ea60 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa32a800640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa310004d40 0x7fa3100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.935+0000 7fa329400640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c15ea60 0x7fa310004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.936+0000 7fa32a800640 1 -- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c15ea60 msgr2=0x7fa310004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.936+0000 7fa32a800640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c15ea60 0x7fa310004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.936+0000 7fa32a800640 1 -- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3100039b0 msgr2=0x7fa310003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.936+0000 7fa32a800640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3100039b0 0x7fa310003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.936+0000 7fa32a800640 1 -- 172.21.15.69:0/957592364 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa310003520 con 0x7fa310004d40 2024-04-24T08:18:23.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.936+0000 7fa329400640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c15ea60 0x7fa310004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:23.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.937+0000 7fa32a800640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa310004d40 0x7fa3100051c0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa324007330 tx=0x7fa310005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43c02054e6532c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:23.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.937+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324009be0 con 0x7fa310004d40 2024-04-24T08:18:23.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.937+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa324036600 con 0x7fa310004d40 2024-04-24T08:18:23.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.937+0000 7fa328a00640 1 -- 172.21.15.69:0/957592364 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3240369d0 con 0x7fa310004d40 2024-04-24T08:18:23.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.938+0000 7fa32b200640 1 -- 172.21.15.69:0/957592364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa310004d40 msgr2=0x7fa3100051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.938+0000 7fa32b200640 1 --2- 172.21.15.69:0/957592364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa310004d40 0x7fa3100051c0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fa324007330 tx=0x7fa310005830 comp rx=0 tx=0).stop 2024-04-24T08:18:23.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.938+0000 7fa32a800640 1 -- 172.21.15.69:0/957592364 reap_dead start 2024-04-24T08:18:23.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.938+0000 7fa32b200640 1 -- 172.21.15.69:0/957592364 shutdown_connections 2024-04-24T08:18:23.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.938+0000 7fa32b200640 1 -- 172.21.15.69:0/957592364 >> 172.21.15.69:0/957592364 conn(0x7fa32c067b80 msgr2=0x7fa32c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:23.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.938+0000 7fa32b200640 1 -- 172.21.15.69:0/957592364 shutdown_connections 2024-04-24T08:18:23.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.938+0000 7fa32b200640 1 -- 172.21.15.69:0/957592364 wait complete. 2024-04-24T08:18:23.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.940+0000 7fa32b200640 1 Processor -- start 2024-04-24T08:18:23.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.940+0000 7fa32b200640 1 -- start start 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa32b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa32c1eefc0 0x7fa32c1ef3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa32b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa32c0ae700 0x7fa32c1f92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa32b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c1f9a10 0x7fa32c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa32b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c0ce510 con 0x7fa32c0ae700 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa32b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c0ce390 con 0x7fa32c1eefc0 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa32b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa32c0ce690 con 0x7fa32c1f9a10 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa329400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa32c0ae700 0x7fa32c1f92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa329400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa32c0ae700 0x7fa32c1f92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35324/0 (socket says 172.21.15.69:35324) 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa329400640 1 -- 172.21.15.69:0/2445264345 learned_addr learned my addr 172.21.15.69:0/2445264345 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa32a800640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c1f9a10 0x7fa32c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa329400640 1 -- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c1f9a10 msgr2=0x7fa32c0b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa329400640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c1f9a10 0x7fa32c0b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa329400640 1 -- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa32c1eefc0 msgr2=0x7fa32c1ef3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:23.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.941+0000 7fa329400640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa32c1eefc0 0x7fa32c1ef3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:23.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.942+0000 7fa329400640 1 -- 172.21.15.69:0/2445264345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32c1f9810 con 0x7fa32c0ae700 2024-04-24T08:18:23.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.942+0000 7fa329400640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa32c0ae700 0x7fa32c1f92d0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fa31800db10 tx=0x7fa318007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:23.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.942+0000 7fa322000640 1 -- 172.21.15.69:0/2445264345 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318019070 con 0x7fa32c0ae700 2024-04-24T08:18:23.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.942+0000 7fa322000640 1 -- 172.21.15.69:0/2445264345 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa318004cb0 con 0x7fa32c0ae700 2024-04-24T08:18:23.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.942+0000 7fa32b200640 1 -- 172.21.15.69:0/2445264345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa32c0ab9f0 con 0x7fa32c0ae700 2024-04-24T08:18:23.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.942+0000 7fa322000640 1 -- 172.21.15.69:0/2445264345 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa318004e60 con 0x7fa32c0ae700 2024-04-24T08:18:23.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.942+0000 7fa32b200640 1 -- 172.21.15.69:0/2445264345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa32c0abfb0 con 0x7fa32c0ae700 2024-04-24T08:18:23.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.943+0000 7fa32b200640 1 -- 172.21.15.69:0/2445264345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa32c075d50 con 0x7fa32c0ae700 2024-04-24T08:18:23.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.944+0000 7fa322000640 1 -- 172.21.15.69:0/2445264345 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa31801c070 con 0x7fa32c0ae700 2024-04-24T08:18:23.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.945+0000 7fa322000640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa30c06c8f0 0x7fa30c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:23.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.945+0000 7fa322000640 1 -- 172.21.15.69:0/2445264345 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fa31808b720 con 0x7fa32c0ae700 2024-04-24T08:18:23.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.946+0000 7fa329e00640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa30c06c8f0 0x7fa30c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:23.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.947+0000 7fa329e00640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa30c06c8f0 0x7fa30c06edb0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00b7d0 tx=0x7fa31c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:23.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:23.950+0000 7fa322000640 1 -- 172.21.15.69:0/2445264345 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa32c1f9810 con 0x7fa32c0ae700 2024-04-24T08:18:24.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.147+0000 7fa32b200640 1 -- 172.21.15.69:0/2445264345 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa32c06bd00 con 0x7fa32c0ae700 2024-04-24T08:18:24.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.148+0000 7fa322000640 1 -- 172.21.15.69:0/2445264345 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7fa318059b40 con 0x7fa32c0ae700 2024-04-24T08:18:24.150 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:18:24.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.150+0000 7fa32b200640 1 -- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa30c06c8f0 msgr2=0x7fa30c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.150+0000 7fa32b200640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa30c06c8f0 0x7fa30c06edb0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00b7d0 tx=0x7fa31c008040 comp rx=0 tx=0).stop 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.150+0000 7fa32b200640 1 -- 172.21.15.69:0/2445264345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa32c0ae700 msgr2=0x7fa32c1f92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.151+0000 7fa32b200640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa32c0ae700 0x7fa32c1f92d0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fa31800db10 tx=0x7fa318007e30 comp rx=0 tx=0).stop 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.151+0000 7fa32b200640 1 -- 172.21.15.69:0/2445264345 shutdown_connections 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.151+0000 7fa32b200640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa30c06c8f0 0x7fa30c06edb0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.151+0000 7fa32b200640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa32c0ae700 0x7fa32c1f92d0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.151+0000 7fa32b200640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa32c1f9a10 0x7fa32c0b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.151+0000 7fa32b200640 1 --2- 172.21.15.69:0/2445264345 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa32c1eefc0 0x7fa32c1ef3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.151+0000 7fa32b200640 1 -- 172.21.15.69:0/2445264345 >> 172.21.15.69:0/2445264345 conn(0x7fa32c067b80 msgr2=0x7fa32c06de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.151+0000 7fa32b200640 1 -- 172.21.15.69:0/2445264345 shutdown_connections 2024-04-24T08:18:24.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.151+0000 7fa32b200640 1 -- 172.21.15.69:0/2445264345 wait complete. 2024-04-24T08:18:24.170 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:18:24.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.366+0000 7fe9a3200640 1 Processor -- start 2024-04-24T08:18:24.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.367+0000 7fe9a3200640 1 -- start start 2024-04-24T08:18:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.368+0000 7fe9a3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe99c15ea10 0x7fe99c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.368+0000 7fe9a3200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe99c0af9a0 0x7fe99c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.368+0000 7fe9a3200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe99c0adab0 con 0x7fe99c1631a0 2024-04-24T08:18:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.368+0000 7fe9a3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe99c0add50 con 0x7fe99c15ea10 2024-04-24T08:18:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.368+0000 7fe9a3200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe99c0ae030 con 0x7fe99c0af9a0 2024-04-24T08:18:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.369+0000 7fe99be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe99c15ea10 0x7fe99c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:24.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.369+0000 7fe99be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe99c15ea10 0x7fe99c0d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35334/0 (socket says 172.21.15.69:35334) 2024-04-24T08:18:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.369+0000 7fe99be00640 1 -- 172.21.15.69:0/3273390270 learned_addr learned my addr 172.21.15.69:0/3273390270 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.369+0000 7fe99be00640 1 -- 172.21.15.69:0/3273390270 >> v2:172.21.15.121:3300/0 conn(0x7fe99c0af9a0 msgr2=0x7fe99c06dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.369+0000 7fe9a1200640 1 --2- 172.21.15.69:0/3273390270 >> v2:172.21.15.121:3300/0 conn(0x7fe99c0af9a0 0x7fe99c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.369+0000 7fe99be00640 1 --2- 172.21.15.69:0/3273390270 >> v2:172.21.15.121:3300/0 conn(0x7fe99c0af9a0 0x7fe99c06dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.369+0000 7fe99be00640 1 -- 172.21.15.69:0/3273390270 >> v1:172.21.15.121:6789/0 conn(0x7fe99c1631a0 legacy=0x7fe99c0af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:24.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.369+0000 7fe99be00640 1 -- 172.21.15.69:0/3273390270 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99c0ae310 con 0x7fe99c15ea10 2024-04-24T08:18:24.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.371+0000 7fe99be00640 1 --2- 172.21.15.69:0/3273390270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c15ea10 0x7fe99c0d0830 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fe99c0cc4d0 tx=0x7fe98c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3167410b9e2265cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:24.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.371+0000 7fe99b400640 1 -- 172.21.15.69:0/3273390270 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c037070 con 0x7fe99c15ea10 2024-04-24T08:18:24.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.371+0000 7fe99b400640 1 -- 172.21.15.69:0/3273390270 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98c032440 con 0x7fe99c15ea10 2024-04-24T08:18:24.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.371+0000 7fe99b400640 1 -- 172.21.15.69:0/3273390270 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c032810 con 0x7fe99c15ea10 2024-04-24T08:18:24.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.372+0000 7fe9a3200640 1 -- 172.21.15.69:0/3273390270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c15ea10 msgr2=0x7fe99c0d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:24.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.372+0000 7fe9a3200640 1 --2- 172.21.15.69:0/3273390270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c15ea10 0x7fe99c0d0830 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7fe99c0cc4d0 tx=0x7fe98c02a230 comp rx=0 tx=0).stop 2024-04-24T08:18:24.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.372+0000 7fe9a3200640 1 -- 172.21.15.69:0/3273390270 shutdown_connections 2024-04-24T08:18:24.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.372+0000 7fe9a3200640 1 --2- 172.21.15.69:0/3273390270 >> v2:172.21.15.121:3300/0 conn(0x7fe99c0af9a0 0x7fe99c06dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.372+0000 7fe9a3200640 1 --2- 172.21.15.69:0/3273390270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c15ea10 0x7fe99c0d0830 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.372+0000 7fe9a3200640 1 -- 172.21.15.69:0/3273390270 >> 172.21.15.69:0/3273390270 conn(0x7fe99c067d90 msgr2=0x7fe99c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:24.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.372+0000 7fe9a3200640 1 -- 172.21.15.69:0/3273390270 shutdown_connections 2024-04-24T08:18:24.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.372+0000 7fe9a3200640 1 -- 172.21.15.69:0/3273390270 wait complete. 2024-04-24T08:18:24.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.374+0000 7fe9a3200640 1 Processor -- start 2024-04-24T08:18:24.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.374+0000 7fe9a3200640 1 -- start start 2024-04-24T08:18:24.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe9a3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c0af9a0 0x7fe99c145770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:24.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe9a3200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe99c15ea10 0x7fe99c0cf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:24.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe9a3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe99c1631a0 0x7fe99c148970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:24.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe9a3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c0aedf0 con 0x7fe99c0af9a0 2024-04-24T08:18:24.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe9a3200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe99c0ae8d0 con 0x7fe99c15ea10 2024-04-24T08:18:24.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe9a3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe99c0d0d70 con 0x7fe99c1631a0 2024-04-24T08:18:24.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe99be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c0af9a0 0x7fe99c145770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:24.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe99be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c0af9a0 0x7fe99c145770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35338/0 (socket says 172.21.15.69:35338) 2024-04-24T08:18:24.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe99be00640 1 -- 172.21.15.69:0/1560292595 learned_addr learned my addr 172.21.15.69:0/1560292595 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:24.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe99be00640 1 -- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe99c1631a0 msgr2=0x7fe99c148970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:24.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe9a1200640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe99c1631a0 0x7fe99c148970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:24.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe99be00640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe99c1631a0 0x7fe99c148970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe99be00640 1 -- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe99c15ea10 msgr2=0x7fe99c0cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:24.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.375+0000 7fe993400640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe99c15ea10 0x7fe99c0cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:24.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.376+0000 7fe99be00640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe99c15ea10 0x7fe99c0cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.376+0000 7fe99be00640 1 -- 172.21.15.69:0/1560292595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99c145cb0 con 0x7fe99c0af9a0 2024-04-24T08:18:24.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.376+0000 7fe993400640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe99c15ea10 0x7fe99c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:24.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.376+0000 7fe99be00640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c0af9a0 0x7fe99c145770 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fe98c000c00 tx=0x7fe98c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:24.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.376+0000 7fe998c00640 1 -- 172.21.15.69:0/1560292595 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c041020 con 0x7fe99c0af9a0 2024-04-24T08:18:24.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.376+0000 7fe998c00640 1 -- 172.21.15.69:0/1560292595 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98c032e50 con 0x7fe99c0af9a0 2024-04-24T08:18:24.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.376+0000 7fe998c00640 1 -- 172.21.15.69:0/1560292595 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c037040 con 0x7fe99c0af9a0 2024-04-24T08:18:24.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.376+0000 7fe9a3200640 1 -- 172.21.15.69:0/1560292595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe99c0cdfe0 con 0x7fe99c0af9a0 2024-04-24T08:18:24.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.376+0000 7fe9a3200640 1 -- 172.21.15.69:0/1560292595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe99c0ce5a0 con 0x7fe99c0af9a0 2024-04-24T08:18:24.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.378+0000 7fe9a3200640 1 -- 172.21.15.69:0/1560292595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe99c068a80 con 0x7fe99c0af9a0 2024-04-24T08:18:24.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.378+0000 7fe998c00640 1 -- 172.21.15.69:0/1560292595 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe98c044020 con 0x7fe99c0af9a0 2024-04-24T08:18:24.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.379+0000 7fe998c00640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe97806c920 0x7fe97806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:24.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.379+0000 7fe998c00640 1 -- 172.21.15.69:0/1560292595 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fe98c0ac410 con 0x7fe99c0af9a0 2024-04-24T08:18:24.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.380+0000 7fe993400640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe97806c920 0x7fe97806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:24.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.381+0000 7fe993400640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe97806c920 0x7fe97806ede0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe99c156020 tx=0x7fe984008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:24.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.384+0000 7fe998c00640 1 -- 172.21.15.69:0/1560292595 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe98c04b030 con 0x7fe99c0af9a0 2024-04-24T08:18:24.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.581+0000 7fe9a3200640 1 -- 172.21.15.69:0/1560292595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe99c06ddd0 con 0x7fe99c0af9a0 2024-04-24T08:18:24.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.582+0000 7fe998c00640 1 -- 172.21.15.69:0/1560292595 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v97) ==== 80+0+0 (secure 0 0 0) 0x7fe98c07a8d0 con 0x7fe99c0af9a0 2024-04-24T08:18:24.583 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:18:24.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 -- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe97806c920 msgr2=0x7fe97806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:24.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe97806c920 0x7fe97806ede0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fe99c156020 tx=0x7fe984008040 comp rx=0 tx=0).stop 2024-04-24T08:18:24.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 -- 172.21.15.69:0/1560292595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c0af9a0 msgr2=0x7fe99c145770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:24.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c0af9a0 0x7fe99c145770 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7fe98c000c00 tx=0x7fe98c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 -- 172.21.15.69:0/1560292595 shutdown_connections 2024-04-24T08:18:24.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe97806c920 0x7fe97806ede0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe99c1631a0 0x7fe99c148970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe99c15ea10 0x7fe99c0cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 --2- 172.21.15.69:0/1560292595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe99c0af9a0 0x7fe99c145770 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:24.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 -- 172.21.15.69:0/1560292595 >> 172.21.15.69:0/1560292595 conn(0x7fe99c067d90 msgr2=0x7fe99c0d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:24.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 -- 172.21.15.69:0/1560292595 shutdown_connections 2024-04-24T08:18:24.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:24.584+0000 7fe9a3200640 1 -- 172.21.15.69:0/1560292595 wait complete. 2024-04-24T08:18:24.603 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:18:24.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:24.603 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:24.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[69871]: fuse finished with error 0 and tester_r 0 2024-04-24T08:18:24.685 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:18:30.786 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:18:30.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:30.787 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:18:30.845 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:31.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.048+0000 7f68c3000640 1 Processor -- start 2024-04-24T08:18:31.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.048+0000 7f68c3000640 1 -- start start 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.049+0000 7f68c3000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f68bc071e50 0x7f68bc15ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c3000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68bc162300 0x7f68bc06e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c3000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68bc15f540 con 0x7f68bc072230 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c3000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f68bc15f820 con 0x7f68bc071e50 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c3000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f68bc15fb00 con 0x7f68bc162300 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c1c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f68bc071e50 0x7f68bc15ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c1c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f68bc071e50 0x7f68bc15ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52990/0 (socket says 172.21.15.69:52990) 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c1c00640 1 -- 172.21.15.69:0/4028401930 learned_addr learned my addr 172.21.15.69:0/4028401930 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c1200640 1 --2- 172.21.15.69:0/4028401930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68bc162300 0x7f68bc06e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c1c00640 1 -- 172.21.15.69:0/4028401930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68bc162300 msgr2=0x7f68bc06e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c1c00640 1 --2- 172.21.15.69:0/4028401930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68bc162300 0x7f68bc06e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c1c00640 1 -- 172.21.15.69:0/4028401930 >> v1:172.21.15.121:6789/0 conn(0x7f68bc072230 legacy=0x7f68bc06e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:31.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.050+0000 7f68c1c00640 1 -- 172.21.15.69:0/4028401930 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc06dda0 con 0x7f68bc071e50 2024-04-24T08:18:31.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.051+0000 7f68c1c00640 1 --2- 172.21.15.69:0/4028401930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc071e50 0x7f68bc15ecc0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009870 tx=0x7f68b002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126b44883713fd81 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:31.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.052+0000 7f68bbe00640 1 -- 172.21.15.69:0/4028401930 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0037070 con 0x7f68bc071e50 2024-04-24T08:18:31.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.052+0000 7f68bbe00640 1 -- 172.21.15.69:0/4028401930 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68b0032440 con 0x7f68bc071e50 2024-04-24T08:18:31.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.052+0000 7f68bbe00640 1 -- 172.21.15.69:0/4028401930 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b0032810 con 0x7f68bc071e50 2024-04-24T08:18:31.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.052+0000 7f68c3000640 1 -- 172.21.15.69:0/4028401930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc071e50 msgr2=0x7f68bc15ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:31.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.052+0000 7f68c3000640 1 --2- 172.21.15.69:0/4028401930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc071e50 0x7f68bc15ecc0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f68b0009870 tx=0x7f68b002a210 comp rx=0 tx=0).stop 2024-04-24T08:18:31.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.053+0000 7f68c3000640 1 -- 172.21.15.69:0/4028401930 shutdown_connections 2024-04-24T08:18:31.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.053+0000 7f68c3000640 1 --2- 172.21.15.69:0/4028401930 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68bc162300 0x7f68bc06e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.053+0000 7f68c3000640 1 --2- 172.21.15.69:0/4028401930 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc071e50 0x7f68bc15ecc0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.053+0000 7f68c3000640 1 -- 172.21.15.69:0/4028401930 >> 172.21.15.69:0/4028401930 conn(0x7f68bc0d0160 msgr2=0x7f68bc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:31.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.053+0000 7f68c3000640 1 -- 172.21.15.69:0/4028401930 shutdown_connections 2024-04-24T08:18:31.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.053+0000 7f68c3000640 1 -- 172.21.15.69:0/4028401930 wait complete. 2024-04-24T08:18:31.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.054+0000 7f68c3000640 1 Processor -- start 2024-04-24T08:18:31.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.054+0000 7f68c3000640 1 -- start start 2024-04-24T08:18:31.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.055+0000 7f68c3000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68bc071e50 0x7f68bc15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.055+0000 7f68c3000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68bc072230 0x7f68bc0b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.055+0000 7f68c3000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc162300 0x7f68bc0b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.055+0000 7f68c3000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc06f090 con 0x7f68bc162300 2024-04-24T08:18:31.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.055+0000 7f68c3000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc06ef10 con 0x7f68bc071e50 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.055+0000 7f68c3000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f68bc06f210 con 0x7f68bc072230 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c2600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc162300 0x7f68bc0b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c2600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc162300 0x7f68bc0b4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53006/0 (socket says 172.21.15.69:53006) 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c2600640 1 -- 172.21.15.69:0/4208008037 learned_addr learned my addr 172.21.15.69:0/4208008037 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c1200640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68bc072230 0x7f68bc0b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c1c00640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68bc071e50 0x7f68bc15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c2600640 1 -- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68bc072230 msgr2=0x7f68bc0b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c2600640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68bc072230 0x7f68bc0b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c2600640 1 -- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68bc071e50 msgr2=0x7f68bc15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c2600640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68bc071e50 0x7f68bc15c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c2600640 1 -- 172.21.15.69:0/4208008037 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc153750 con 0x7f68bc162300 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.056+0000 7f68c2600640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc162300 0x7f68bc0b4be0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f68b400c780 tx=0x7f68b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.057+0000 7f68b9600640 1 -- 172.21.15.69:0/4208008037 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b4017070 con 0x7f68bc162300 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.057+0000 7f68b9600640 1 -- 172.21.15.69:0/4208008037 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68b4003da0 con 0x7f68bc162300 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.057+0000 7f68c3000640 1 -- 172.21.15.69:0/4208008037 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68bc153a30 con 0x7f68bc162300 2024-04-24T08:18:31.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.057+0000 7f68b9600640 1 -- 172.21.15.69:0/4208008037 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b40052b0 con 0x7f68bc162300 2024-04-24T08:18:31.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.057+0000 7f68c3000640 1 -- 172.21.15.69:0/4208008037 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68bc153ff0 con 0x7f68bc162300 2024-04-24T08:18:31.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.058+0000 7f68c3000640 1 -- 172.21.15.69:0/4208008037 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68bc075d50 con 0x7f68bc162300 2024-04-24T08:18:31.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.059+0000 7f68b9600640 1 -- 172.21.15.69:0/4208008037 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f68b401a070 con 0x7f68bc162300 2024-04-24T08:18:31.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.060+0000 7f68b9600640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f689c06c8f0 0x7f689c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.060+0000 7f68b9600640 1 -- 172.21.15.69:0/4208008037 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f68b408b330 con 0x7f68bc162300 2024-04-24T08:18:31.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.060+0000 7f68c1c00640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f689c06c8f0 0x7f689c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.062+0000 7f68c1c00640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f689c06c8f0 0x7f689c06edb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f68b0034000 tx=0x7f68b002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:31.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.065+0000 7f68b9600640 1 -- 172.21.15.69:0/4208008037 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68b4059630 con 0x7f68bc162300 2024-04-24T08:18:31.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.305+0000 7f68c3000640 1 -- 172.21.15.69:0/4208008037 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68bc156a60 con 0x7f68bc162300 2024-04-24T08:18:31.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.307+0000 7f68b9600640 1 -- 172.21.15.69:0/4208008037 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5186 (secure 0 0 0) 0x7f68b4058f30 con 0x7f68bc162300 2024-04-24T08:18:31.308 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:31.308 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:18:13.481156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6046":{"gid":6046,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/412016900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":412016900},{"type":"v1","addr":"172.21.15.69:6835","nonce":412016900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6046,"qdb_cluster":[6046]},"id":7}]} 2024-04-24T08:18:31.309 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 83 2024-04-24T08:18:31.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.309+0000 7f68c3000640 1 -- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f689c06c8f0 msgr2=0x7f689c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:31.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.309+0000 7f68c3000640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f689c06c8f0 0x7f689c06edb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f68b0034000 tx=0x7f68b002e000 comp rx=0 tx=0).stop 2024-04-24T08:18:31.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.309+0000 7f68c3000640 1 -- 172.21.15.69:0/4208008037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc162300 msgr2=0x7f68bc0b4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:31.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.309+0000 7f68c3000640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc162300 0x7f68bc0b4be0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f68b400c780 tx=0x7f68b400cc40 comp rx=0 tx=0).stop 2024-04-24T08:18:31.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.309+0000 7f68c3000640 1 -- 172.21.15.69:0/4208008037 shutdown_connections 2024-04-24T08:18:31.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.309+0000 7f68c3000640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f689c06c8f0 0x7f689c06edb0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.309+0000 7f68c3000640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f68bc162300 0x7f68bc0b4be0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.309+0000 7f68c3000640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68bc072230 0x7f68bc0b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.309+0000 7f68c3000640 1 --2- 172.21.15.69:0/4208008037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68bc071e50 0x7f68bc15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.309+0000 7f68c3000640 1 -- 172.21.15.69:0/4208008037 >> 172.21.15.69:0/4208008037 conn(0x7f68bc0d0160 msgr2=0x7f68bc06e2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.310+0000 7f68c3000640 1 -- 172.21.15.69:0/4208008037 shutdown_connections 2024-04-24T08:18:31.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.310+0000 7f68c3000640 1 -- 172.21.15.69:0/4208008037 wait complete. 2024-04-24T08:18:31.329 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-04-24T08:18:31.555 INFO:teuthology.orchestra.run.smithi069.stdout:{"message":"","return_code":0} 2024-04-24T08:18:31.556 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-24T08:18:31.556 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.743+0000 7f3334000640 1 Processor -- start 2024-04-24T08:18:31.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.743+0000 7f3334000640 1 -- start start 2024-04-24T08:18:31.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.744+0000 7f3334000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f332c0d0300 0x7f332c06e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.745+0000 7f3334000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f332c145a90 0x7f332c0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.745+0000 7f3334000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f332c145fe0 0x7f332c160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.745+0000 7f3334000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f332c0d0c20 con 0x7f332c145fe0 2024-04-24T08:18:31.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.745+0000 7f3334000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f332c0cf3d0 con 0x7f332c145a90 2024-04-24T08:18:31.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.745+0000 7f3334000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f332c0cf6b0 con 0x7f332c0d0300 2024-04-24T08:18:31.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.745+0000 7f3332000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f332c145fe0 0x7f332c160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.745+0000 7f3332000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f332c145fe0 0x7f332c160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53010/0 (socket says 172.21.15.69:53010) 2024-04-24T08:18:31.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.745+0000 7f3332000640 1 -- 172.21.15.69:0/2857626209 learned_addr learned my addr 172.21.15.69:0/2857626209 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:31.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.745+0000 7f3330c00640 1 --2- 172.21.15.69:0/2857626209 >> v2:172.21.15.121:3300/0 conn(0x7f332c145a90 0x7f332c0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.745+0000 7f3331600640 1 --2- 172.21.15.69:0/2857626209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f332c0d0300 0x7f332c06e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.746+0000 7f3332000640 1 -- 172.21.15.69:0/2857626209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f332c0d0300 msgr2=0x7f332c06e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.746+0000 7f3332000640 1 --2- 172.21.15.69:0/2857626209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f332c0d0300 0x7f332c06e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.746+0000 7f3332000640 1 -- 172.21.15.69:0/2857626209 >> v2:172.21.15.121:3300/0 conn(0x7f332c145a90 msgr2=0x7f332c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.746+0000 7f3332000640 1 --2- 172.21.15.69:0/2857626209 >> v2:172.21.15.121:3300/0 conn(0x7f332c145a90 0x7f332c0d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.746+0000 7f3332000640 1 -- 172.21.15.69:0/2857626209 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332c0cf990 con 0x7f332c145fe0 2024-04-24T08:18:31.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.747+0000 7f3332000640 1 --2- 172.21.15.69:0/2857626209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c145fe0 0x7f332c160540 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f332c073210 tx=0x7f332402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dffe889fff92c7f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.747+0000 7f332be00640 1 -- 172.21.15.69:0/2857626209 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3324036070 con 0x7f332c145fe0 2024-04-24T08:18:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f332be00640 1 -- 172.21.15.69:0/2857626209 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3324031600 con 0x7f332c145fe0 2024-04-24T08:18:31.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f332be00640 1 -- 172.21.15.69:0/2857626209 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33240319d0 con 0x7f332c145fe0 2024-04-24T08:18:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f3334000640 1 -- 172.21.15.69:0/2857626209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c145fe0 msgr2=0x7f332c160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f3334000640 1 --2- 172.21.15.69:0/2857626209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c145fe0 0x7f332c160540 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f332c073210 tx=0x7f332402a6a0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f3334000640 1 -- 172.21.15.69:0/2857626209 shutdown_connections 2024-04-24T08:18:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f3334000640 1 --2- 172.21.15.69:0/2857626209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c145fe0 0x7f332c160540 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f3334000640 1 --2- 172.21.15.69:0/2857626209 >> v2:172.21.15.121:3300/0 conn(0x7f332c145a90 0x7f332c0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f3334000640 1 --2- 172.21.15.69:0/2857626209 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f332c0d0300 0x7f332c06e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f3334000640 1 -- 172.21.15.69:0/2857626209 >> 172.21.15.69:0/2857626209 conn(0x7f332c067d90 msgr2=0x7f332c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f3334000640 1 -- 172.21.15.69:0/2857626209 shutdown_connections 2024-04-24T08:18:31.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.748+0000 7f3334000640 1 -- 172.21.15.69:0/2857626209 wait complete. 2024-04-24T08:18:31.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.750+0000 7f3334000640 1 Processor -- start 2024-04-24T08:18:31.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.751+0000 7f3334000640 1 -- start start 2024-04-24T08:18:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3334000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c0d0300 0x7f332c13f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3334000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f332c145a90 0x7f332c0ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3334000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f332c145fe0 0x7f332c0ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3334000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f332c145770 con 0x7f332c0d0300 2024-04-24T08:18:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3334000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f332c0cff50 con 0x7f332c145fe0 2024-04-24T08:18:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3334000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f332c1458f0 con 0x7f332c145a90 2024-04-24T08:18:31.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3331600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c0d0300 0x7f332c13f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3331600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c0d0300 0x7f332c13f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53018/0 (socket says 172.21.15.69:53018) 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3331600640 1 -- 172.21.15.69:0/312948649 learned_addr learned my addr 172.21.15.69:0/312948649 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3330c00640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f332c145a90 0x7f332c0ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3331600640 1 -- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f332c145a90 msgr2=0x7f332c0ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.752+0000 7f3331600640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f332c145a90 0x7f332c0ae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.753+0000 7f3331600640 1 -- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f332c145fe0 msgr2=0x7f332c0ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.753+0000 7f3331600640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f332c145fe0 0x7f332c0ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.753+0000 7f3331600640 1 -- 172.21.15.69:0/312948649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332c0aea90 con 0x7f332c0d0300 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.753+0000 7f3331600640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c0d0300 0x7f332c13f030 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f332000d800 tx=0x7f332000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.753+0000 7f3329600640 1 -- 172.21.15.69:0/312948649 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3320018030 con 0x7f332c0d0300 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.753+0000 7f3329600640 1 -- 172.21.15.69:0/312948649 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33200096a0 con 0x7f332c0d0300 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.753+0000 7f3329600640 1 -- 172.21.15.69:0/312948649 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3320009870 con 0x7f332c0d0300 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.753+0000 7f3334000640 1 -- 172.21.15.69:0/312948649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332c0aed70 con 0x7f332c0d0300 2024-04-24T08:18:31.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.753+0000 7f3334000640 1 -- 172.21.15.69:0/312948649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f332c142900 con 0x7f332c0d0300 2024-04-24T08:18:31.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.754+0000 7f3334000640 1 -- 172.21.15.69:0/312948649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f332c0af290 con 0x7f332c0d0300 2024-04-24T08:18:31.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.756+0000 7f3329600640 1 -- 172.21.15.69:0/312948649 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f332001b070 con 0x7f332c0d0300 2024-04-24T08:18:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.757+0000 7f3329600640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f330c06c8f0 0x7f330c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:31.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.757+0000 7f3329600640 1 -- 172.21.15.69:0/312948649 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f332008c5f0 con 0x7f332c0d0300 2024-04-24T08:18:31.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.757+0000 7f3330c00640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f330c06c8f0 0x7f330c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:31.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.759+0000 7f3330c00640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f330c06c8f0 0x7f330c06edb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3318006fd0 tx=0x7f3318008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:31.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.760+0000 7f3329600640 1 -- 172.21.15.69:0/312948649 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f332005ab00 con 0x7f332c0d0300 2024-04-24T08:18:31.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.979+0000 7f3334000640 1 -- 172.21.15.69:0/312948649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f332c068a80 con 0x7f332c0d0300 2024-04-24T08:18:31.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.980+0000 7f3329600640 1 -- 172.21.15.69:0/312948649 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5186 (secure 0 0 0) 0x7f332005a400 con 0x7f332c0d0300 2024-04-24T08:18:31.982 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:31.982 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:18:13.481156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6046":{"gid":6046,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/412016900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":412016900},{"type":"v1","addr":"172.21.15.69:6835","nonce":412016900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6046,"qdb_cluster":[6046]},"id":7}]} 2024-04-24T08:18:31.982 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 83 2024-04-24T08:18:31.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 -- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f330c06c8f0 msgr2=0x7f330c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f330c06c8f0 0x7f330c06edb0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3318006fd0 tx=0x7f3318008040 comp rx=0 tx=0).stop 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 -- 172.21.15.69:0/312948649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c0d0300 msgr2=0x7f332c13f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c0d0300 0x7f332c13f030 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f332000d800 tx=0x7f332000dcc0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 -- 172.21.15.69:0/312948649 shutdown_connections 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f330c06c8f0 0x7f330c06edb0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f332c145fe0 0x7f332c0ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f332c145a90 0x7f332c0ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 --2- 172.21.15.69:0/312948649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f332c0d0300 0x7f332c13f030 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 -- 172.21.15.69:0/312948649 >> 172.21.15.69:0/312948649 conn(0x7f332c067d90 msgr2=0x7f332c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 -- 172.21.15.69:0/312948649 shutdown_connections 2024-04-24T08:18:31.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:31.983+0000 7f3334000640 1 -- 172.21.15.69:0/312948649 wait complete. 2024-04-24T08:18:32.002 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata get mds0_inotable - 2024-04-24T08:18:32.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.107+0000 7f7e57f08f40 1 Processor -- start 2024-04-24T08:18:32.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.107+0000 7f7e57f08f40 1 -- start start 2024-04-24T08:18:32.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.108+0000 7f7e57f08f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564d2efe7ad0 0x564d2f0112d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.108+0000 7f7e57f08f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564d2f014380 0x564d2f0b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.108+0000 7f7e57f08f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2f0b94b0 0x564d2f0bae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.108+0000 7f7e57f08f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564d2f0b9aa0 con 0x564d2f0b94b0 2024-04-24T08:18:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.108+0000 7f7e57f08f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564d2ef8bcc0 con 0x564d2efe7ad0 2024-04-24T08:18:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.108+0000 7f7e57f08f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564d2f0b9c20 con 0x564d2f014380 2024-04-24T08:18:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.108+0000 7f7e55c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2f0b94b0 0x564d2f0bae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.108+0000 7f7e55c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2f0b94b0 0x564d2f0bae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53030/0 (socket says 172.21.15.69:53030) 2024-04-24T08:18:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.108+0000 7f7e55c00640 1 -- 172.21.15.69:0/3359917840 learned_addr learned my addr 172.21.15.69:0/3359917840 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:32.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.108+0000 7f7e55200640 1 --2- 172.21.15.69:0/3359917840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564d2efe7ad0 0x564d2f0112d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.109+0000 7f7e54800640 1 --2- 172.21.15.69:0/3359917840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564d2f014380 0x564d2f0b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.109+0000 7f7e55c00640 1 -- 172.21.15.69:0/3359917840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564d2f014380 msgr2=0x564d2f0b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:32.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.109+0000 7f7e55c00640 1 --2- 172.21.15.69:0/3359917840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564d2f014380 0x564d2f0b2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.109+0000 7f7e55c00640 1 -- 172.21.15.69:0/3359917840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564d2efe7ad0 msgr2=0x564d2f0112d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:32.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.109+0000 7f7e55c00640 1 --2- 172.21.15.69:0/3359917840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564d2efe7ad0 0x564d2f0112d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.109+0000 7f7e55c00640 1 -- 172.21.15.69:0/3359917840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d2f0b35a0 con 0x564d2f0b94b0 2024-04-24T08:18:32.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.109+0000 7f7e55c00640 1 --2- 172.21.15.69:0/3359917840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2f0b94b0 0x564d2f0bae00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c002850 tx=0x7f7e4c002d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67fa5a952afd10 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:32.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.109+0000 7f7e53400640 1 -- 172.21.15.69:0/3359917840 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c016020 con 0x564d2f0b94b0 2024-04-24T08:18:32.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.109+0000 7f7e53400640 1 -- 172.21.15.69:0/3359917840 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e4c00d9c0 con 0x564d2f0b94b0 2024-04-24T08:18:32.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.109+0000 7f7e53400640 1 -- 172.21.15.69:0/3359917840 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c00ddb0 con 0x564d2f0b94b0 2024-04-24T08:18:32.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.110+0000 7f7e57f08f40 1 -- 172.21.15.69:0/3359917840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2f0b94b0 msgr2=0x564d2f0bae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.110+0000 7f7e57f08f40 1 --2- 172.21.15.69:0/3359917840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2f0b94b0 0x564d2f0bae00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c002850 tx=0x7f7e4c002d10 comp rx=0 tx=0).stop 2024-04-24T08:18:32.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.110+0000 7f7e57f08f40 1 -- 172.21.15.69:0/3359917840 shutdown_connections 2024-04-24T08:18:32.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.110+0000 7f7e57f08f40 1 --2- 172.21.15.69:0/3359917840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2f0b94b0 0x564d2f0bae00 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.110+0000 7f7e57f08f40 1 --2- 172.21.15.69:0/3359917840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564d2f014380 0x564d2f0b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.110+0000 7f7e57f08f40 1 --2- 172.21.15.69:0/3359917840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564d2efe7ad0 0x564d2f0112d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.110+0000 7f7e57f08f40 1 -- 172.21.15.69:0/3359917840 >> 172.21.15.69:0/3359917840 conn(0x564d2f011e40 msgr2=0x564d2f014760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.110+0000 7f7e57f08f40 1 -- 172.21.15.69:0/3359917840 shutdown_connections 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.110+0000 7f7e57f08f40 1 -- 172.21.15.69:0/3359917840 wait complete. 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.111+0000 7f7e57f08f40 1 Processor -- start 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.111+0000 7f7e57f08f40 1 -- start start 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.112+0000 7f7e57f08f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2efe7ad0 0x564d2f0bb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.112+0000 7f7e57f08f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564d2f014380 0x564d2f0b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.112+0000 7f7e57f08f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564d2f0b94b0 0x564d2efe47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.112+0000 7f7e57f08f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564d2f0b7db0 con 0x564d2efe7ad0 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.112+0000 7f7e57f08f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x564d2f0b32c0 con 0x564d2f0b94b0 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.112+0000 7f7e57f08f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564d2efe59e0 con 0x564d2f014380 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.112+0000 7f7e55200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2efe7ad0 0x564d2f0bb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.112+0000 7f7e55200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2efe7ad0 0x564d2f0bb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53038/0 (socket says 172.21.15.69:53038) 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.112+0000 7f7e55200640 1 -- 172.21.15.69:0/1664781151 learned_addr learned my addr 172.21.15.69:0/1664781151 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:32.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e54800640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564d2f014380 0x564d2f0b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e55c00640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564d2f0b94b0 0x564d2efe47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e55200640 1 -- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564d2f014380 msgr2=0x564d2f0b7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e55200640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564d2f014380 0x564d2f0b7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e55200640 1 -- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564d2f0b94b0 msgr2=0x564d2efe47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e55200640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564d2f0b94b0 0x564d2efe47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e55200640 1 -- 172.21.15.69:0/1664781151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d2f011810 con 0x564d2efe7ad0 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e55c00640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564d2f0b94b0 0x564d2efe47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e55200640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2efe7ad0 0x564d2f0bb000 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f7e48045f50 tx=0x7f7e48003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e52000640 1 -- 172.21.15.69:0/1664781151 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4806b070 con 0x564d2efe7ad0 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e52000640 1 -- 172.21.15.69:0/1664781151 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e480538a0 con 0x564d2efe7ad0 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.113+0000 7f7e52000640 1 -- 172.21.15.69:0/1664781151 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48053c90 con 0x564d2efe7ad0 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.114+0000 7f7e57f08f40 1 -- 172.21.15.69:0/1664781151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x564d2f0bb5e0 con 0x564d2efe7ad0 2024-04-24T08:18:32.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.114+0000 7f7e57f08f40 1 -- 172.21.15.69:0/1664781151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d2f0b6550 con 0x564d2efe7ad0 2024-04-24T08:18:32.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.115+0000 7f7e52000640 1 -- 172.21.15.69:0/1664781151 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e4805c3f0 con 0x564d2efe7ad0 2024-04-24T08:18:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.116+0000 7f7e52000640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3406b2d0 0x7f7e3406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.116+0000 7f7e52000640 1 -- 172.21.15.69:0/1664781151 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f7e480ce430 con 0x564d2efe7ad0 2024-04-24T08:18:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.117+0000 7f7e57f08f40 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x564d2f02a730 0x564d2f0a7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.117+0000 7f7e57f08f40 1 -- 172.21.15.69:0/1664781151 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:1 14.13 14:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e97) -- 0x564d2f152620 con 0x564d2f02a730 2024-04-24T08:18:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.117+0000 7f7e54800640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3406b2d0 0x7f7e3406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.117+0000 7f7e55c00640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x564d2f02a730 0x564d2f0a7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e55c00640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x564d2f02a730 0x564d2f0a7690 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e55c00640 1 -- 172.21.15.69:0/1664781151 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv3 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x7f7e4c012470 con 0x564d2f02a730 2024-04-24T08:18:32.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e57f08f40 1 -- 172.21.15.69:0/1664781151 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x564d2f02a730 msgr2=0x564d2f0a7690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e57f08f40 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x564d2f02a730 0x564d2f0a7690 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e54800640 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3406b2d0 0x7f7e3406d790 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x564d2f0bb8c0 tx=0x7f7e40003050 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e57f08f40 1 -- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3406b2d0 msgr2=0x7f7e3406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e57f08f40 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e3406b2d0 0x7f7e3406d790 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x564d2f0bb8c0 tx=0x7f7e40003050 comp rx=0 tx=0).stop 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e57f08f40 1 -- 172.21.15.69:0/1664781151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2efe7ad0 msgr2=0x564d2f0bb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e57f08f40 1 --2- 172.21.15.69:0/1664781151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564d2efe7ad0 0x564d2f0bb000 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f7e48045f50 tx=0x7f7e48003020 comp rx=0 tx=0).stop 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e55c00640 1 -- 172.21.15.69:0/1664781151 reap_dead start 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e57f08f40 1 -- 172.21.15.69:0/1664781151 shutdown_connections 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.118+0000 7f7e57f08f40 1 -- 172.21.15.69:0/1664781151 >> 172.21.15.69:0/1664781151 conn(0x564d2f011e40 msgr2=0x564d2f012220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.119+0000 7f7e57f08f40 1 -- 172.21.15.69:0/1664781151 shutdown_connections 2024-04-24T08:18:32.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.119+0000 7f7e57f08f40 1 -- 172.21.15.69:0/1664781151 wait complete. 2024-04-24T08:18:32.125 INFO:teuthology.orchestra.run.smithi069.stdout:����� 2024-04-24T08:18:32.125 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:18:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.302+0000 7fd741e00640 1 Processor -- start 2024-04-24T08:18:32.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.303+0000 7fd741e00640 1 -- start start 2024-04-24T08:18:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.304+0000 7fd741e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd73c156c60 0x7fd73c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.304+0000 7fd741e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd73c0ad7e0 0x7fd73c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.304+0000 7fd741e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd73c15d660 con 0x7fd73c0b30f0 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.304+0000 7fd741e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd73c06dc30 con 0x7fd73c156c60 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.304+0000 7fd741e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd73c06df10 con 0x7fd73c0ad7e0 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.304+0000 7fd740a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd73c156c60 0x7fd73c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.305+0000 7fd740a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd73c156c60 0x7fd73c15cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53042/0 (socket says 172.21.15.69:53042) 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.305+0000 7fd740a00640 1 -- 172.21.15.69:0/1312745833 learned_addr learned my addr 172.21.15.69:0/1312745833 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.305+0000 7fd740a00640 1 -- 172.21.15.69:0/1312745833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd73c0ad7e0 msgr2=0x7fd73c15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.305+0000 7fd741400640 1 --2- 172.21.15.69:0/1312745833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd73c0ad7e0 0x7fd73c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.305+0000 7fd740a00640 1 --2- 172.21.15.69:0/1312745833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd73c0ad7e0 0x7fd73c15d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.305+0000 7fd740a00640 1 -- 172.21.15.69:0/1312745833 >> v1:172.21.15.69:6789/0 conn(0x7fd73c0b30f0 legacy=0x7fd73c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.305+0000 7fd73be00640 1 -- 172.21.15.69:0/1312745833 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4077271428 0 0) 0x7fd73c15d660 con 0x7fd73c0b30f0 2024-04-24T08:18:32.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.305+0000 7fd740a00640 1 -- 172.21.15.69:0/1312745833 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c06e1f0 con 0x7fd73c156c60 2024-04-24T08:18:32.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.306+0000 7fd740a00640 1 --2- 172.21.15.69:0/1312745833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 0x7fd73c15cac0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009870 tx=0x7fd72c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ade1f6cae4ba2c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:32.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.306+0000 7fd73be00640 1 -- 172.21.15.69:0/1312745833 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c037070 con 0x7fd73c156c60 2024-04-24T08:18:32.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.306+0000 7fd73be00640 1 -- 172.21.15.69:0/1312745833 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd72c032440 con 0x7fd73c156c60 2024-04-24T08:18:32.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.306+0000 7fd73be00640 1 -- 172.21.15.69:0/1312745833 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd72c032810 con 0x7fd73c156c60 2024-04-24T08:18:32.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.307+0000 7fd741e00640 1 -- 172.21.15.69:0/1312745833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 msgr2=0x7fd73c15cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.307+0000 7fd741e00640 1 --2- 172.21.15.69:0/1312745833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 0x7fd73c15cac0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fd72c009870 tx=0x7fd72c02a270 comp rx=0 tx=0).stop 2024-04-24T08:18:32.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.307+0000 7fd741e00640 1 -- 172.21.15.69:0/1312745833 shutdown_connections 2024-04-24T08:18:32.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.307+0000 7fd741e00640 1 --2- 172.21.15.69:0/1312745833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd73c0ad7e0 0x7fd73c15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.307+0000 7fd741e00640 1 --2- 172.21.15.69:0/1312745833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 0x7fd73c15cac0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.307+0000 7fd741e00640 1 -- 172.21.15.69:0/1312745833 >> 172.21.15.69:0/1312745833 conn(0x7fd73c0d0160 msgr2=0x7fd73c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:32.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.307+0000 7fd741e00640 1 -- 172.21.15.69:0/1312745833 shutdown_connections 2024-04-24T08:18:32.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.307+0000 7fd741e00640 1 -- 172.21.15.69:0/1312745833 wait complete. 2024-04-24T08:18:32.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.309+0000 7fd741e00640 1 Processor -- start 2024-04-24T08:18:32.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.309+0000 7fd741e00640 1 -- start start 2024-04-24T08:18:32.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.310+0000 7fd741e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd73c0ad7e0 0x7fd73c0994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.310+0000 7fd741e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd73c0b30f0 0x7fd73c0b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.310+0000 7fd741e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 0x7fd73c0b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.310+0000 7fd741e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c06e710 con 0x7fd73c156c60 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.310+0000 7fd741e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd73c15d940 con 0x7fd73c0b30f0 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.310+0000 7fd741e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd73c06e890 con 0x7fd73c0ad7e0 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.310+0000 7fd741400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 0x7fd73c0b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.310+0000 7fd741400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 0x7fd73c0b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53048/0 (socket says 172.21.15.69:53048) 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.310+0000 7fd741400640 1 -- 172.21.15.69:0/1177033780 learned_addr learned my addr 172.21.15.69:0/1177033780 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd740a00640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd73c0ad7e0 0x7fd73c0994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd741400640 1 -- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd73c0ad7e0 msgr2=0x7fd73c0994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd741400640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd73c0ad7e0 0x7fd73c0994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd741400640 1 -- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd73c0b30f0 msgr2=0x7fd73c0b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd741400640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd73c0b30f0 0x7fd73c0b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd741400640 1 -- 172.21.15.69:0/1177033780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73c1e64a0 con 0x7fd73c156c60 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd741400640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 0x7fd73c0b0e70 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fd73400cd40 tx=0x7fd73400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd739600640 1 -- 172.21.15.69:0/1177033780 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734003b90 con 0x7fd73c156c60 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd739600640 1 -- 172.21.15.69:0/1177033780 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd734003d30 con 0x7fd73c156c60 2024-04-24T08:18:32.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd741e00640 1 -- 172.21.15.69:0/1177033780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd73c158cb0 con 0x7fd73c156c60 2024-04-24T08:18:32.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd739600640 1 -- 172.21.15.69:0/1177033780 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd734004240 con 0x7fd73c156c60 2024-04-24T08:18:32.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.311+0000 7fd741e00640 1 -- 172.21.15.69:0/1177033780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd73c159270 con 0x7fd73c156c60 2024-04-24T08:18:32.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.312+0000 7fd741e00640 1 -- 172.21.15.69:0/1177033780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd73c068e10 con 0x7fd73c156c60 2024-04-24T08:18:32.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.313+0000 7fd739600640 1 -- 172.21.15.69:0/1177033780 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd7340043e0 con 0x7fd73c156c60 2024-04-24T08:18:32.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.314+0000 7fd739600640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd71806c8f0 0x7fd71806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:32.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.314+0000 7fd739600640 1 -- 172.21.15.69:0/1177033780 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fd73408c740 con 0x7fd73c156c60 2024-04-24T08:18:32.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.315+0000 7fd740a00640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd71806c8f0 0x7fd71806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:32.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.316+0000 7fd740a00640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd71806c8f0 0x7fd71806edb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd72c034000 tx=0x7fd72c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:32.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.318+0000 7fd739600640 1 -- 172.21.15.69:0/1177033780 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd73c068e10 con 0x7fd73c156c60 2024-04-24T08:18:32.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.556+0000 7fd741e00640 1 -- 172.21.15.69:0/1177033780 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd73c075d50 con 0x7fd73c156c60 2024-04-24T08:18:32.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.556+0000 7fd739600640 1 -- 172.21.15.69:0/1177033780 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v83) ==== 53+0+73 (secure 0 0 0) 0x7fd73405aad0 con 0x7fd73c156c60 2024-04-24T08:18:32.558 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 -- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd71806c8f0 msgr2=0x7fd71806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd71806c8f0 0x7fd71806edb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd72c034000 tx=0x7fd72c02e000 comp rx=0 tx=0).stop 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 -- 172.21.15.69:0/1177033780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 msgr2=0x7fd73c0b0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 0x7fd73c0b0e70 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fd73400cd40 tx=0x7fd73400a660 comp rx=0 tx=0).stop 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 -- 172.21.15.69:0/1177033780 shutdown_connections 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd71806c8f0 0x7fd71806edb0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd73c156c60 0x7fd73c0b0e70 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd73c0b30f0 0x7fd73c0b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 --2- 172.21.15.69:0/1177033780 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd73c0ad7e0 0x7fd73c0994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 -- 172.21.15.69:0/1177033780 >> 172.21.15.69:0/1177033780 conn(0x7fd73c0d0160 msgr2=0x7fd73c15b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 -- 172.21.15.69:0/1177033780 shutdown_connections 2024-04-24T08:18:32.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32.559+0000 7fd741e00640 1 -- 172.21.15.69:0/1177033780 wait complete. 2024-04-24T08:18:32.579 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:18:32.579 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:18:32.579 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:18:32.579 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:18:32.579 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:32.579 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:18:32.579 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:18:32.579 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:18:32.579 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:18:32.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:32.580 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:18:32.628 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:18:32.629 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42067sec preferred_lft 42067sec 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:18:32.630 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:18:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:18:32.631 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:18:32.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:32.632 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:18:32.705 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:18:32.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:32.709 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:18:32.757 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:18:32.757 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:18:32.757 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:18:32.757 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:18:32.757 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:18:32.757 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:18:32.757 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:18:32.757 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:18:32.757 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:18:32.758 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:18:32.758 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:18:32.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:18:32.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:18:32.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:32.948 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:18:32.992 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:18:32.992 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:18:32.993 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:18:32.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:32.993 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:18:33.122 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:18:33.122 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:18:33.123 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:33.173 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:18:33.173 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:18:33.173 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:33.226 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:18:33.306 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:18:33.358 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:18:33.359 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:18:33.360 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:18:33.361 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:18:33.361 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:18:33.361 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:18:33.361 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:18:33.361 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:18:33.361 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:18:33.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:33.362 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:18:33.413 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:18:33.413 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:18:33.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:18:33.487 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:18:33.530 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:18:33.530 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:18:33.530 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:18:33.530 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:18:33.530 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:18:33.530 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:18:33.530 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:18:33.531 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:18:33.532 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:18:33.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:33.533 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:18:33.586 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:18:33.585+0000 7f629cf94480 -1 init, newargv = 0x561d9f4c7a30 newargc=15 2024-04-24T08:18:33.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[70951]: starting ceph client 2024-04-24T08:18:33.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[70951]: starting fuse 2024-04-24T08:18:33.618 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:18:33.618 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:18:33.618 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:18:33.618 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:18:33.618 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:18:33.618 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:18:33.618 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:18:33.618 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:18:33.619 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:18:33.620 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:18:33.620 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:33.620 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:18:33.620 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:18:33.811 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.70951.asok 2024-04-24T08:18:33.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:18:33.828 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.70951.asok 2024-04-24T08:18:33.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:33.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.70951.asok status 2024-04-24T08:18:34.051 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "70951", 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 8811, 2024-04-24T08:18:34.052 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 8811 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 285791952 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:18:34.053 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 285791952 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.8811 192.168.0.1:0/285791952", 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/285791952", 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 83, 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 97, 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:18:34.054 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:18:34.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:34.066 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:18:34.066 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:34.119 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:18:34.119 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:34.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:34.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:34.196 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:18:34.196 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:34.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:34.246 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file2_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-04-24T08:18:34.508 INFO:teuthology.orchestra.run.smithi069.stderr:6+0 records in 2024-04-24T08:18:34.508 INFO:teuthology.orchestra.run.smithi069.stderr:6+0 records out 2024-04-24T08:18:34.508 INFO:teuthology.orchestra.run.smithi069.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.153998 s, 40.9 MB/s 2024-04-24T08:18:34.510 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:18:34.510 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:34.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:34.563 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file3_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-04-24T08:18:34.770 INFO:teuthology.orchestra.run.smithi069.stderr:6+0 records in 2024-04-24T08:18:34.771 INFO:teuthology.orchestra.run.smithi069.stderr:6+0 records out 2024-04-24T08:18:34.771 INFO:teuthology.orchestra.run.smithi069.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.154861 s, 40.6 MB/s 2024-04-24T08:18:34.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:34.772 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-04-24T08:18:34.832 INFO:teuthology.orchestra.run.smithi069.stdout:./ 2024-04-24T08:18:34.833 INFO:teuthology.orchestra.run.smithi069.stdout:./file2_sixmegs 2024-04-24T08:18:34.833 INFO:teuthology.orchestra.run.smithi069.stdout:./file3_sixmegs 2024-04-24T08:18:34.833 INFO:teuthology.orchestra.run.smithi069.stdout:./file1_sixmegs 2024-04-24T08:18:34.834 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:18:34.834 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:18:34.834 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:18:34.834 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:34.834 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-04-24T08:18:34.834 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:18:34.976 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:18:34.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:18:34.990 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:18:34.990 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:18:34.990 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:18:34.990 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:34.990 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file2_sixmegs").st_ino) 2024-04-24T08:18:34.990 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:18:35.171 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627777 2024-04-24T08:18:35.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:18:35.185 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:18:35.185 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:18:35.185 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:18:35.185 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:35.185 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file3_sixmegs").st_ino) 2024-04-24T08:18:35.185 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:18:35.369 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627778 2024-04-24T08:18:35.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:18:35.383 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:18:35.383 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:18:35.383 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:18:35.383 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:18:35.383 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file1_sixmegs").st_ino) 2024-04-24T08:18:35.383 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:18:35.558 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627776 2024-04-24T08:18:35.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:18:35.571 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:18:35.571 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:35.667 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.870+0000 7f4954a00640 1 Processor -- start 2024-04-24T08:18:35.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.870+0000 7f4954a00640 1 -- start start 2024-04-24T08:18:35.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.871+0000 7f4954a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4950162c70 0x7f49500b3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.871+0000 7f4954a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49500b3d30 0x7f49500b4190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.871+0000 7f4954a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4950071d70 0x7f49500721a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.871+0000 7f4954a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f49500b46d0 con 0x7f4950162c70 2024-04-24T08:18:35.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.871+0000 7f4954a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f49501669d0 con 0x7f4950071d70 2024-04-24T08:18:35.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.871+0000 7f4954a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4950166cb0 con 0x7f49500b3d30 2024-04-24T08:18:35.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f494de00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4950162c70 0x7f49500b3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f494de00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4950162c70 0x7f49500b3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53050/0 (socket says 172.21.15.69:53050) 2024-04-24T08:18:35.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f494de00640 1 -- 172.21.15.69:0/488255408 learned_addr learned my addr 172.21.15.69:0/488255408 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:35.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f494e800640 1 --2- 172.21.15.69:0/488255408 >> v2:172.21.15.121:3300/0 conn(0x7f4950071d70 0x7f49500721a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f4945400640 1 --2- 172.21.15.69:0/488255408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49500b3d30 0x7f49500b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f494de00640 1 -- 172.21.15.69:0/488255408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49500b3d30 msgr2=0x7f49500b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:35.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f494de00640 1 --2- 172.21.15.69:0/488255408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49500b3d30 0x7f49500b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f494de00640 1 -- 172.21.15.69:0/488255408 >> v2:172.21.15.121:3300/0 conn(0x7f4950071d70 msgr2=0x7f49500721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:35.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f494de00640 1 --2- 172.21.15.69:0/488255408 >> v2:172.21.15.121:3300/0 conn(0x7f4950071d70 0x7f49500721a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f494de00640 1 -- 172.21.15.69:0/488255408 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4950166f90 con 0x7f4950162c70 2024-04-24T08:18:35.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f4945400640 1 --2- 172.21.15.69:0/488255408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49500b3d30 0x7f49500b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:35.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.872+0000 7f494e800640 1 --2- 172.21.15.69:0/488255408 >> v2:172.21.15.121:3300/0 conn(0x7f4950071d70 0x7f49500721a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:35.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.873+0000 7f494de00640 1 --2- 172.21.15.69:0/488255408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4950162c70 0x7f49500b3320 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f49500cc800 tx=0x7f494002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5314fbb240de74eb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:35.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f494d400640 1 -- 172.21.15.69:0/488255408 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940037070 con 0x7f4950162c70 2024-04-24T08:18:35.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f494d400640 1 -- 172.21.15.69:0/488255408 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4940032440 con 0x7f4950162c70 2024-04-24T08:18:35.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f494d400640 1 -- 172.21.15.69:0/488255408 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940032810 con 0x7f4950162c70 2024-04-24T08:18:35.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f4954a00640 1 -- 172.21.15.69:0/488255408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4950162c70 msgr2=0x7f49500b3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:35.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f4954a00640 1 --2- 172.21.15.69:0/488255408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4950162c70 0x7f49500b3320 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f49500cc800 tx=0x7f494002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f4954a00640 1 -- 172.21.15.69:0/488255408 shutdown_connections 2024-04-24T08:18:35.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f4954a00640 1 --2- 172.21.15.69:0/488255408 >> v2:172.21.15.121:3300/0 conn(0x7f4950071d70 0x7f49500721a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f4954a00640 1 --2- 172.21.15.69:0/488255408 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f49500b3d30 0x7f49500b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f4954a00640 1 --2- 172.21.15.69:0/488255408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4950162c70 0x7f49500b3320 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f4954a00640 1 -- 172.21.15.69:0/488255408 >> 172.21.15.69:0/488255408 conn(0x7f4950067d90 msgr2=0x7f495006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:35.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.874+0000 7f4954a00640 1 -- 172.21.15.69:0/488255408 shutdown_connections 2024-04-24T08:18:35.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.875+0000 7f4954a00640 1 -- 172.21.15.69:0/488255408 wait complete. 2024-04-24T08:18:35.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.876+0000 7f4954a00640 1 Processor -- start 2024-04-24T08:18:35.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.876+0000 7f4954a00640 1 -- start start 2024-04-24T08:18:35.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.877+0000 7f4954a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4950071d70 0x7f495015c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.877+0000 7f4954a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49500b3d30 0x7f495015dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.877+0000 7f4954a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4950162c70 0x7f495014e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.877+0000 7f4954a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4950163050 con 0x7f49500b3d30 2024-04-24T08:18:35.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.877+0000 7f4954a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4950167550 con 0x7f4950071d70 2024-04-24T08:18:35.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.877+0000 7f4954a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f49500cfc40 con 0x7f4950162c70 2024-04-24T08:18:35.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f4945400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49500b3d30 0x7f495015dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f4945400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49500b3d30 0x7f495015dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53060/0 (socket says 172.21.15.69:53060) 2024-04-24T08:18:35.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f4945400640 1 -- 172.21.15.69:0/2312424167 learned_addr learned my addr 172.21.15.69:0/2312424167 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:35.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f494de00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4950071d70 0x7f495015c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f494e800640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4950162c70 0x7f495014e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f4945400640 1 -- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4950162c70 msgr2=0x7f495014e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f4945400640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4950162c70 0x7f495014e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f4945400640 1 -- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4950071d70 msgr2=0x7f495015c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f4945400640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4950071d70 0x7f495015c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f4945400640 1 -- 172.21.15.69:0/2312424167 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495015a9c0 con 0x7f49500b3d30 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f494de00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4950071d70 0x7f495015c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.878+0000 7f4945400640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49500b3d30 0x7f495015dc20 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f493800c7f0 tx=0x7f493800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.879+0000 7f4946a00640 1 -- 172.21.15.69:0/2312424167 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938015070 con 0x7f49500b3d30 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.879+0000 7f4946a00640 1 -- 172.21.15.69:0/2312424167 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4938007a30 con 0x7f49500b3d30 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.879+0000 7f4946a00640 1 -- 172.21.15.69:0/2312424167 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938007e00 con 0x7f49500b3d30 2024-04-24T08:18:35.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.879+0000 7f4954a00640 1 -- 172.21.15.69:0/2312424167 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495015aca0 con 0x7f49500b3d30 2024-04-24T08:18:35.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.879+0000 7f4954a00640 1 -- 172.21.15.69:0/2312424167 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f495015e160 con 0x7f49500b3d30 2024-04-24T08:18:35.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.880+0000 7f4954a00640 1 -- 172.21.15.69:0/2312424167 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f495015ae70 con 0x7f49500b3d30 2024-04-24T08:18:35.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.881+0000 7f4946a00640 1 -- 172.21.15.69:0/2312424167 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4938018070 con 0x7f49500b3d30 2024-04-24T08:18:35.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.882+0000 7f4946a00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f492c06c8f0 0x7f492c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:35.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.882+0000 7f4946a00640 1 -- 172.21.15.69:0/2312424167 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f493808bb20 con 0x7f49500b3d30 2024-04-24T08:18:35.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.882+0000 7f494de00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f492c06c8f0 0x7f492c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:35.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.884+0000 7f494de00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f492c06c8f0 0x7f492c06edb0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f4940034000 tx=0x7f494002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:35.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:35.886+0000 7f4946a00640 1 -- 172.21.15.69:0/2312424167 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4938059f40 con 0x7f49500b3d30 2024-04-24T08:18:36.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.085+0000 7f4954a00640 1 -- 172.21.15.69:0/2312424167 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4950068a80 con 0x7f49500b3d30 2024-04-24T08:18:36.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.087+0000 7f4946a00640 1 -- 172.21.15.69:0/2312424167 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+16584 (secure 0 0 0) 0x7f4938059840 con 0x7f49500b3d30 2024-04-24T08:18:36.088 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:36.089 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":97,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:18:11.928428+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:08.156864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:09.368519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 -- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f492c06c8f0 msgr2=0x7f492c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f492c06c8f0 0x7f492c06edb0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f4940034000 tx=0x7f494002e000 comp rx=0 tx=0).stop 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 -- 172.21.15.69:0/2312424167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49500b3d30 msgr2=0x7f495015dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49500b3d30 0x7f495015dc20 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f493800c7f0 tx=0x7f493800cbf0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 -- 172.21.15.69:0/2312424167 shutdown_connections 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f492c06c8f0 0x7f492c06edb0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49500b3d30 0x7f495015dc20 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4950162c70 0x7f495014e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 --2- 172.21.15.69:0/2312424167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4950071d70 0x7f495015c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 -- 172.21.15.69:0/2312424167 >> 172.21.15.69:0/2312424167 conn(0x7f4950067d90 msgr2=0x7f495006e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:36.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 -- 172.21.15.69:0/2312424167 shutdown_connections 2024-04-24T08:18:36.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.089+0000 7f4954a00640 1 -- 172.21.15.69:0/2312424167 wait complete. 2024-04-24T08:18:36.109 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:08.156864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:09.368519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:18:36.109 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:18:36.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.315+0000 7fed97e00640 1 Processor -- start 2024-04-24T08:18:36.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.315+0000 7fed97e00640 1 -- start start 2024-04-24T08:18:36.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.316+0000 7fed97e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fed9815ea60 0x7fed980b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.317+0000 7fed97e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed980ce910 con 0x7fed980cdf30 2024-04-24T08:18:36.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.317+0000 7fed97e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed980cebf0 con 0x7fed980d09f0 2024-04-24T08:18:36.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.317+0000 7fed97e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fed980ceed0 con 0x7fed9815ea60 2024-04-24T08:18:36.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.317+0000 7fed96000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fed980cdf30 0x7fed980d02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46922/0 (socket says 172.21.15.69:46922) 2024-04-24T08:18:36.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.317+0000 7fed96000640 1 -- 172.21.15.69:0/1687823339 learned_addr learned my addr 172.21.15.69:0/1687823339 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:36.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.317+0000 7fed96a00640 1 --2- 172.21.15.69:0/1687823339 >> v2:172.21.15.69:3300/0 conn(0x7fed9815ea60 0x7fed980b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.317+0000 7fed95600640 1 -- 172.21.15.69:0/1687823339 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3065493346 0 0) 0x7fed980ce910 con 0x7fed980cdf30 2024-04-24T08:18:36.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.318+0000 7fed95600640 1 -- 172.21.15.69:0/1687823339 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed7c003520 con 0x7fed980cdf30 2024-04-24T08:18:36.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.318+0000 7fed96a00640 1 -- 172.21.15.69:0/1687823339 >> v1:172.21.15.121:6789/0 conn(0x7fed980d09f0 legacy=0x7fed9806df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.318+0000 7fed96a00640 1 -- 172.21.15.69:0/1687823339 >> v1:172.21.15.69:6789/0 conn(0x7fed980cdf30 legacy=0x7fed980d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.318+0000 7fed96a00640 1 -- 172.21.15.69:0/1687823339 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed980cf1b0 con 0x7fed9815ea60 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.319+0000 7fed96a00640 1 --2- 172.21.15.69:0/1687823339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed9815ea60 0x7fed980b5300 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fed880097a0 tx=0x7fed8802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d182cc5b7e654c58 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.319+0000 7fed95600640 1 -- 172.21.15.69:0/1687823339 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88037070 con 0x7fed9815ea60 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.319+0000 7fed95600640 1 -- 172.21.15.69:0/1687823339 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed88032440 con 0x7fed9815ea60 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.319+0000 7fed95600640 1 -- 172.21.15.69:0/1687823339 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88032810 con 0x7fed9815ea60 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.320+0000 7fed97e00640 1 -- 172.21.15.69:0/1687823339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed9815ea60 msgr2=0x7fed980b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.320+0000 7fed97e00640 1 --2- 172.21.15.69:0/1687823339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed9815ea60 0x7fed980b5300 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fed880097a0 tx=0x7fed8802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.320+0000 7fed97e00640 1 -- 172.21.15.69:0/1687823339 shutdown_connections 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.320+0000 7fed97e00640 1 --2- 172.21.15.69:0/1687823339 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed9815ea60 0x7fed980b5300 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.320+0000 7fed97e00640 1 -- 172.21.15.69:0/1687823339 >> 172.21.15.69:0/1687823339 conn(0x7fed98067b80 msgr2=0x7fed9806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.320+0000 7fed97e00640 1 -- 172.21.15.69:0/1687823339 shutdown_connections 2024-04-24T08:18:36.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.320+0000 7fed97e00640 1 -- 172.21.15.69:0/1687823339 wait complete. 2024-04-24T08:18:36.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.321+0000 7fed97e00640 1 Processor -- start 2024-04-24T08:18:36.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.322+0000 7fed97e00640 1 -- start start 2024-04-24T08:18:36.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.322+0000 7fed97e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed980cdf30 0x7fed980aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.322+0000 7fed97e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed980d09f0 0x7fed98149ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.322+0000 7fed97e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9815ea60 0x7fed980b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.322+0000 7fed97e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed980ce540 con 0x7fed980cdf30 2024-04-24T08:18:36.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.322+0000 7fed97e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fed980ce3c0 con 0x7fed980d09f0 2024-04-24T08:18:36.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.322+0000 7fed97e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed980ce6c0 con 0x7fed9815ea60 2024-04-24T08:18:36.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.323+0000 7fed96a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed980cdf30 0x7fed980aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.323+0000 7fed96a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed980cdf30 0x7fed980aecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53080/0 (socket says 172.21.15.69:53080) 2024-04-24T08:18:36.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.323+0000 7fed96a00640 1 -- 172.21.15.69:0/729628752 learned_addr learned my addr 172.21.15.69:0/729628752 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:36.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.323+0000 7fed96a00640 1 -- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9815ea60 msgr2=0x7fed980b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.323+0000 7fed96a00640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9815ea60 0x7fed980b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.323+0000 7fed96000640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed980d09f0 0x7fed98149ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.323+0000 7fed96a00640 1 -- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed980d09f0 msgr2=0x7fed98149ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.323+0000 7fed96a00640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed980d09f0 0x7fed98149ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.323+0000 7fed96a00640 1 -- 172.21.15.69:0/729628752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed980ae780 con 0x7fed980cdf30 2024-04-24T08:18:36.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.324+0000 7fed96a00640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed980cdf30 0x7fed980aecd0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fed88035040 tx=0x7fed880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:36.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.324+0000 7fed8ea00640 1 -- 172.21.15.69:0/729628752 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88037040 con 0x7fed980cdf30 2024-04-24T08:18:36.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.324+0000 7fed8ea00640 1 -- 172.21.15.69:0/729628752 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed88030500 con 0x7fed980cdf30 2024-04-24T08:18:36.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.324+0000 7fed97e00640 1 -- 172.21.15.69:0/729628752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed980adc00 con 0x7fed980cdf30 2024-04-24T08:18:36.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.324+0000 7fed8ea00640 1 -- 172.21.15.69:0/729628752 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed880308d0 con 0x7fed980cdf30 2024-04-24T08:18:36.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.324+0000 7fed97e00640 1 -- 172.21.15.69:0/729628752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed980ae1c0 con 0x7fed980cdf30 2024-04-24T08:18:36.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.325+0000 7fed97e00640 1 -- 172.21.15.69:0/729628752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed98148550 con 0x7fed980cdf30 2024-04-24T08:18:36.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.326+0000 7fed8ea00640 1 -- 172.21.15.69:0/729628752 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fed88044020 con 0x7fed980cdf30 2024-04-24T08:18:36.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.327+0000 7fed8ea00640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7406c8f0 0x7fed7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.327+0000 7fed8ea00640 1 -- 172.21.15.69:0/729628752 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fed880abe70 con 0x7fed980cdf30 2024-04-24T08:18:36.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.328+0000 7fed96000640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7406c8f0 0x7fed7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.329+0000 7fed96000640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7406c8f0 0x7fed7406edb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fed98146940 tx=0x7fed80009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:36.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.332+0000 7fed8ea00640 1 -- 172.21.15.69:0/729628752 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fed8807a2a0 con 0x7fed980cdf30 2024-04-24T08:18:36.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.533+0000 7fed97e00640 1 -- 172.21.15.69:0/729628752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fed98075d50 con 0x7fed980cdf30 2024-04-24T08:18:36.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.534+0000 7fed8ea00640 1 -- 172.21.15.69:0/729628752 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7fed88079ba0 con 0x7fed980cdf30 2024-04-24T08:18:36.536 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:18:36.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 -- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7406c8f0 msgr2=0x7fed7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7406c8f0 0x7fed7406edb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fed98146940 tx=0x7fed80009040 comp rx=0 tx=0).stop 2024-04-24T08:18:36.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 -- 172.21.15.69:0/729628752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed980cdf30 msgr2=0x7fed980aecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed980cdf30 0x7fed980aecd0 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fed88035040 tx=0x7fed880046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 -- 172.21.15.69:0/729628752 shutdown_connections 2024-04-24T08:18:36.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7406c8f0 0x7fed7406edb0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9815ea60 0x7fed980b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed980d09f0 0x7fed98149ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 --2- 172.21.15.69:0/729628752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed980cdf30 0x7fed980aecd0 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 -- 172.21.15.69:0/729628752 >> 172.21.15.69:0/729628752 conn(0x7fed98067b80 msgr2=0x7fed9806e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 -- 172.21.15.69:0/729628752 shutdown_connections 2024-04-24T08:18:36.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.537+0000 7fed97e00640 1 -- 172.21.15.69:0/729628752 wait complete. 2024-04-24T08:18:36.558 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:18:36.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.751+0000 7ff863200640 1 Processor -- start 2024-04-24T08:18:36.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.751+0000 7ff863200640 1 -- start start 2024-04-24T08:18:36.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.753+0000 7ff863200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff864071d70 0x7ff864072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.753+0000 7ff863200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86415edf0 0x7ff86415f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.753+0000 7ff863200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff864072690 con 0x7ff864162270 2024-04-24T08:18:36.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.753+0000 7ff863200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff8640d0a00 con 0x7ff864071d70 2024-04-24T08:18:36.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.753+0000 7ff863200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff86406f010 con 0x7ff86415edf0 2024-04-24T08:18:36.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.753+0000 7ff862800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff864162270 0x7ff864162700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46928/0 (socket says 172.21.15.69:46928) 2024-04-24T08:18:36.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.753+0000 7ff862800640 1 -- 172.21.15.69:0/318295906 learned_addr learned my addr 172.21.15.69:0/318295906 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:36.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff861400640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86415edf0 0x7ff86415f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff861e00640 1 --2- 172.21.15.69:0/318295906 >> v2:172.21.15.121:3300/0 conn(0x7ff864071d70 0x7ff864072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 661237001 0 0) 0x7ff864072690 con 0x7ff864162270 2024-04-24T08:18:36.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff848003520 con 0x7ff864162270 2024-04-24T08:18:36.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3733633267 0 0) 0x7ff848003520 con 0x7ff864162270 2024-04-24T08:18:36.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86415edf0 msgr2=0x7ff86415f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86415edf0 0x7ff86415f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 >> v2:172.21.15.121:3300/0 conn(0x7ff864071d70 msgr2=0x7ff864072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 --2- 172.21.15.69:0/318295906 >> v2:172.21.15.121:3300/0 conn(0x7ff864071d70 0x7ff864072150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff86415fbb0 con 0x7ff864162270 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7ff85c0019f0 con 0x7ff864162270 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 >> v1:172.21.15.69:6789/0 conn(0x7ff864162270 legacy=0x7ff864162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 0x7ff848003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff848004800 0x7ff848004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86415edf0 0x7ff8480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff8640d0a00 con 0x7ff8480039b0 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff86406f010 con 0x7ff848004800 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.754+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff84800b9c0 con 0x7ff86415edf0 2024-04-24T08:18:36.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.755+0000 7ff861e00640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 0x7ff848003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.755+0000 7ff861400640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff848004800 0x7ff848004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.755+0000 7ff862800640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86415edf0 0x7ff8480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.755+0000 7ff861e00640 1 -- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86415edf0 msgr2=0x7ff8480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:36.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.755+0000 7ff861e00640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86415edf0 0x7ff8480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.755+0000 7ff861e00640 1 -- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff848004800 msgr2=0x7ff848004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:36.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.755+0000 7ff861e00640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff848004800 0x7ff848004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.756+0000 7ff861e00640 1 -- 172.21.15.69:0/318295906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff848003520 con 0x7ff8480039b0 2024-04-24T08:18:36.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.756+0000 7ff861400640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff848004800 0x7ff848004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:36.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.757+0000 7ff861e00640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 0x7ff848003dd0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7ff854009ac0 tx=0x7ff848005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df79fe690676fcb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:36.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.757+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854036070 con 0x7ff8480039b0 2024-04-24T08:18:36.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.758+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff854031400 con 0x7ff8480039b0 2024-04-24T08:18:36.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.758+0000 7ff860a00640 1 -- 172.21.15.69:0/318295906 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854009690 con 0x7ff8480039b0 2024-04-24T08:18:36.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.758+0000 7ff863200640 1 -- 172.21.15.69:0/318295906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 msgr2=0x7ff848003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.758+0000 7ff863200640 1 --2- 172.21.15.69:0/318295906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 0x7ff848003dd0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7ff854009ac0 tx=0x7ff848005830 comp rx=0 tx=0).stop 2024-04-24T08:18:36.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.758+0000 7ff862800640 1 -- 172.21.15.69:0/318295906 reap_dead start 2024-04-24T08:18:36.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.758+0000 7ff863200640 1 -- 172.21.15.69:0/318295906 shutdown_connections 2024-04-24T08:18:36.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.758+0000 7ff863200640 1 -- 172.21.15.69:0/318295906 >> 172.21.15.69:0/318295906 conn(0x7ff864067b80 msgr2=0x7ff86406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:36.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.758+0000 7ff863200640 1 -- 172.21.15.69:0/318295906 shutdown_connections 2024-04-24T08:18:36.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.758+0000 7ff863200640 1 -- 172.21.15.69:0/318295906 wait complete. 2024-04-24T08:18:36.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.760+0000 7ff863200640 1 Processor -- start 2024-04-24T08:18:36.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.760+0000 7ff863200640 1 -- start start 2024-04-24T08:18:36.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.761+0000 7ff863200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 0x7ff86415dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.761+0000 7ff863200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86413dca0 0x7ff8640cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.761+0000 7ff863200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8641465f0 0x7ff864146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.761+0000 7ff863200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff8640d0690 con 0x7ff8480039b0 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.761+0000 7ff863200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff8640d03b0 con 0x7ff8641465f0 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.761+0000 7ff863200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff8640d0e90 con 0x7ff86413dca0 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff861e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 0x7ff86415dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff861e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 0x7ff86415dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53092/0 (socket says 172.21.15.69:53092) 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff861e00640 1 -- 172.21.15.69:0/3999752142 learned_addr learned my addr 172.21.15.69:0/3999752142 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff861e00640 1 -- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86413dca0 msgr2=0x7ff8640cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff862800640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8641465f0 0x7ff864146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff861400640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86413dca0 0x7ff8640cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff861e00640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86413dca0 0x7ff8640cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff861e00640 1 -- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8641465f0 msgr2=0x7ff864146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff861e00640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8641465f0 0x7ff864146a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff861e00640 1 -- 172.21.15.69:0/3999752142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8640ccdf0 con 0x7ff8480039b0 2024-04-24T08:18:36.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.762+0000 7ff861e00640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 0x7ff86415dd50 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7ff854009ac0 tx=0x7ff854004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:36.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.763+0000 7ff85a000640 1 -- 172.21.15.69:0/3999752142 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854040020 con 0x7ff8480039b0 2024-04-24T08:18:36.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.763+0000 7ff85a000640 1 -- 172.21.15.69:0/3999752142 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff854003b40 con 0x7ff8480039b0 2024-04-24T08:18:36.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.763+0000 7ff863200640 1 -- 172.21.15.69:0/3999752142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8640cd0d0 con 0x7ff8480039b0 2024-04-24T08:18:36.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.763+0000 7ff85a000640 1 -- 172.21.15.69:0/3999752142 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854036040 con 0x7ff8480039b0 2024-04-24T08:18:36.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.763+0000 7ff863200640 1 -- 172.21.15.69:0/3999752142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8640ccb00 con 0x7ff8480039b0 2024-04-24T08:18:36.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.764+0000 7ff863200640 1 -- 172.21.15.69:0/3999752142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8640cd5f0 con 0x7ff8480039b0 2024-04-24T08:18:36.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.765+0000 7ff85a000640 1 -- 172.21.15.69:0/3999752142 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff854043020 con 0x7ff8480039b0 2024-04-24T08:18:36.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.767+0000 7ff85a000640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff84406c8f0 0x7ff84406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:36.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.767+0000 7ff85a000640 1 -- 172.21.15.69:0/3999752142 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7ff8540abcc0 con 0x7ff8480039b0 2024-04-24T08:18:36.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.767+0000 7ff861400640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff84406c8f0 0x7ff84406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:36.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.769+0000 7ff861400640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff84406c8f0 0x7ff84406edb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff8640b3d40 tx=0x7ff850008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:36.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.772+0000 7ff85a000640 1 -- 172.21.15.69:0/3999752142 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff8540b2080 con 0x7ff8480039b0 2024-04-24T08:18:36.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.971+0000 7ff863200640 1 -- 172.21.15.69:0/3999752142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff864075d50 con 0x7ff8480039b0 2024-04-24T08:18:36.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.972+0000 7ff85a000640 1 -- 172.21.15.69:0/3999752142 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7ff85404a070 con 0x7ff8480039b0 2024-04-24T08:18:36.973 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:18:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.974+0000 7ff863200640 1 -- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff84406c8f0 msgr2=0x7ff84406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.974+0000 7ff863200640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff84406c8f0 0x7ff84406edb0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ff8640b3d40 tx=0x7ff850008040 comp rx=0 tx=0).stop 2024-04-24T08:18:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.974+0000 7ff863200640 1 -- 172.21.15.69:0/3999752142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 msgr2=0x7ff86415dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.974+0000 7ff863200640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 0x7ff86415dd50 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7ff854009ac0 tx=0x7ff854004730 comp rx=0 tx=0).stop 2024-04-24T08:18:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.975+0000 7ff863200640 1 -- 172.21.15.69:0/3999752142 shutdown_connections 2024-04-24T08:18:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.975+0000 7ff863200640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff84406c8f0 0x7ff84406edb0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.975+0000 7ff863200640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8641465f0 0x7ff864146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.975+0000 7ff863200640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff86413dca0 0x7ff8640cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.975+0000 7ff863200640 1 --2- 172.21.15.69:0/3999752142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8480039b0 0x7ff86415dd50 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.975+0000 7ff863200640 1 -- 172.21.15.69:0/3999752142 >> 172.21.15.69:0/3999752142 conn(0x7ff864067b80 msgr2=0x7ff86406f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.975+0000 7ff863200640 1 -- 172.21.15.69:0/3999752142 shutdown_connections 2024-04-24T08:18:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:36.975+0000 7ff863200640 1 -- 172.21.15.69:0/3999752142 wait complete. 2024-04-24T08:18:36.995 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:18:37.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.194+0000 7febfd600640 1 Processor -- start 2024-04-24T08:18:37.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.194+0000 7febfd600640 1 -- start start 2024-04-24T08:18:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.195+0000 7febfd600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febf815ea60 0x7febf80b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.195+0000 7febfd600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febf80d09f0 0x7febf806df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.195+0000 7febfd600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febf80ce910 con 0x7febf80cdf30 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.195+0000 7febfd600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7febf80cebf0 con 0x7febf80d09f0 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.195+0000 7febfd600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7febf80ceed0 con 0x7febf815ea60 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.196+0000 7febfcc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febf80d09f0 0x7febf806df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.196+0000 7febfcc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febf80d09f0 0x7febf806df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53096/0 (socket says 172.21.15.69:53096) 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.196+0000 7febfcc00640 1 -- 172.21.15.69:0/3029361276 learned_addr learned my addr 172.21.15.69:0/3029361276 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.196+0000 7febf7e00640 1 --2- 172.21.15.69:0/3029361276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febf815ea60 0x7febf80b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.196+0000 7febfcc00640 1 -- 172.21.15.69:0/3029361276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febf815ea60 msgr2=0x7febf80b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.196+0000 7febfcc00640 1 --2- 172.21.15.69:0/3029361276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febf815ea60 0x7febf80b5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.196+0000 7febfcc00640 1 -- 172.21.15.69:0/3029361276 >> v1:172.21.15.121:6789/0 conn(0x7febf80cdf30 legacy=0x7febf80d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:37.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.196+0000 7febfcc00640 1 -- 172.21.15.69:0/3029361276 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf80cf1b0 con 0x7febf80d09f0 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.197+0000 7febfcc00640 1 --2- 172.21.15.69:0/3029361276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80d09f0 0x7febf806df40 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7febf0009e50 tx=0x7febf002a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faccec1e5c13afb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febf6a00640 1 -- 172.21.15.69:0/3029361276 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf0036070 con 0x7febf80d09f0 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febf6a00640 1 -- 172.21.15.69:0/3029361276 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febf0031600 con 0x7febf80d09f0 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febf6a00640 1 -- 172.21.15.69:0/3029361276 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf00319d0 con 0x7febf80d09f0 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febfd600640 1 -- 172.21.15.69:0/3029361276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80d09f0 msgr2=0x7febf806df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febfd600640 1 --2- 172.21.15.69:0/3029361276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80d09f0 0x7febf806df40 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7febf0009e50 tx=0x7febf002a630 comp rx=0 tx=0).stop 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febfd600640 1 -- 172.21.15.69:0/3029361276 shutdown_connections 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febfd600640 1 --2- 172.21.15.69:0/3029361276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80d09f0 0x7febf806df40 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febfd600640 1 --2- 172.21.15.69:0/3029361276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febf815ea60 0x7febf80b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febfd600640 1 -- 172.21.15.69:0/3029361276 >> 172.21.15.69:0/3029361276 conn(0x7febf8067b80 msgr2=0x7febf806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febfd600640 1 -- 172.21.15.69:0/3029361276 shutdown_connections 2024-04-24T08:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.198+0000 7febfd600640 1 -- 172.21.15.69:0/3029361276 wait complete. 2024-04-24T08:18:37.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.200+0000 7febfd600640 1 Processor -- start 2024-04-24T08:18:37.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.200+0000 7febfd600640 1 -- start start 2024-04-24T08:18:37.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.201+0000 7febfd600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80cdf30 0x7febf8157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.201+0000 7febfd600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febf80d09f0 0x7febf8149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.201+0000 7febfd600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febf815ea60 0x7febf814acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.201+0000 7febfd600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7febf80ce510 con 0x7febf80cdf30 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.201+0000 7febfd600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7febf80ce390 con 0x7febf815ea60 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.201+0000 7febfd600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7febf80ce690 con 0x7febf80d09f0 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.201+0000 7febf7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80cdf30 0x7febf8157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.202+0000 7febf7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80cdf30 0x7febf8157a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53112/0 (socket says 172.21.15.69:53112) 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.202+0000 7febf7e00640 1 -- 172.21.15.69:0/3142438879 learned_addr learned my addr 172.21.15.69:0/3142438879 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.202+0000 7febf7e00640 1 -- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febf80d09f0 msgr2=0x7febf8149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.202+0000 7febf7e00640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febf80d09f0 0x7febf8149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.202+0000 7febf7e00640 1 -- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febf815ea60 msgr2=0x7febf814acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.202+0000 7febf7e00640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febf815ea60 0x7febf814acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.202+0000 7febf7e00640 1 -- 172.21.15.69:0/3142438879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf814b660 con 0x7febf80cdf30 2024-04-24T08:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.202+0000 7febf7e00640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80cdf30 0x7febf8157a80 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7febe800d800 tx=0x7febe800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:37.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.202+0000 7febefe00640 1 -- 172.21.15.69:0/3142438879 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe800b6c0 con 0x7febf80cdf30 2024-04-24T08:18:37.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.203+0000 7febefe00640 1 -- 172.21.15.69:0/3142438879 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe8019070 con 0x7febf80cdf30 2024-04-24T08:18:37.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.203+0000 7febefe00640 1 -- 172.21.15.69:0/3142438879 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe800ba80 con 0x7febf80cdf30 2024-04-24T08:18:37.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.203+0000 7febfd600640 1 -- 172.21.15.69:0/3142438879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf814a090 con 0x7febf80cdf30 2024-04-24T08:18:37.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.203+0000 7febfd600640 1 -- 172.21.15.69:0/3142438879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febf8156fa0 con 0x7febf80cdf30 2024-04-24T08:18:37.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.204+0000 7febfd600640 1 -- 172.21.15.69:0/3142438879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febf8075d50 con 0x7febf80cdf30 2024-04-24T08:18:37.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.205+0000 7febefe00640 1 -- 172.21.15.69:0/3142438879 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febe800bc20 con 0x7febf80cdf30 2024-04-24T08:18:37.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.206+0000 7febefe00640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febd806c8f0 0x7febd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.206+0000 7febefe00640 1 -- 172.21.15.69:0/3142438879 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7febe808bff0 con 0x7febf80cdf30 2024-04-24T08:18:37.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.207+0000 7febf7400640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febd806c8f0 0x7febd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.208+0000 7febf7400640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febd806c8f0 0x7febd806edb0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7febe4006fd0 tx=0x7febe4006bc0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:37.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.211+0000 7febefe00640 1 -- 172.21.15.69:0/3142438879 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febf814b660 con 0x7febf80cdf30 2024-04-24T08:18:37.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.412+0000 7febfd600640 1 -- 172.21.15.69:0/3142438879 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7febf8068d00 con 0x7febf80cdf30 2024-04-24T08:18:37.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.413+0000 7febefe00640 1 -- 172.21.15.69:0/3142438879 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7febe801c360 con 0x7febf80cdf30 2024-04-24T08:18:37.415 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 -- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febd806c8f0 msgr2=0x7febd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febd806c8f0 0x7febd806edb0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7febe4006fd0 tx=0x7febe4006bc0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 -- 172.21.15.69:0/3142438879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80cdf30 msgr2=0x7febf8157a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80cdf30 0x7febf8157a80 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7febe800d800 tx=0x7febe800dcc0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 -- 172.21.15.69:0/3142438879 shutdown_connections 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febd806c8f0 0x7febd806edb0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febf815ea60 0x7febf814acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febf80d09f0 0x7febf8149b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 --2- 172.21.15.69:0/3142438879 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febf80cdf30 0x7febf8157a80 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 -- 172.21.15.69:0/3142438879 >> 172.21.15.69:0/3142438879 conn(0x7febf8067b80 msgr2=0x7febf80aecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:37.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 -- 172.21.15.69:0/3142438879 shutdown_connections 2024-04-24T08:18:37.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.416+0000 7febfd600640 1 -- 172.21.15.69:0/3142438879 wait complete. 2024-04-24T08:18:37.437 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:18:37.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.634+0000 7fc78ec00640 1 Processor -- start 2024-04-24T08:18:37.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.635+0000 7fc78ec00640 1 -- start start 2024-04-24T08:18:37.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.636+0000 7fc78ec00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc78815e960 0x7fc788162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.636+0000 7fc78ec00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc788162980 0x7fc78806cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.636+0000 7fc78ec00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc78815ed40 con 0x7fc78806d4f0 2024-04-24T08:18:37.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.636+0000 7fc78ec00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc7880cf050 con 0x7fc788162980 2024-04-24T08:18:37.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.636+0000 7fc78ec00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc788162e20 con 0x7fc78815e960 2024-04-24T08:18:37.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.636+0000 7fc78ce00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc788162980 0x7fc78806cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.636+0000 7fc78ce00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc788162980 0x7fc78806cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53128/0 (socket says 172.21.15.69:53128) 2024-04-24T08:18:37.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.636+0000 7fc78e200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc78806d4f0 0x7fc7880cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46934/0 (socket says 172.21.15.69:46934) 2024-04-24T08:18:37.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.636+0000 7fc78ce00640 1 -- 172.21.15.69:0/878324783 learned_addr learned my addr 172.21.15.69:0/878324783 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:37.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.637+0000 7fc78d800640 1 --2- 172.21.15.69:0/878324783 >> v2:172.21.15.121:3300/0 conn(0x7fc78815e960 0x7fc788162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.637+0000 7fc78ce00640 1 -- 172.21.15.69:0/878324783 >> v2:172.21.15.121:3300/0 conn(0x7fc78815e960 msgr2=0x7fc788162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:37.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.637+0000 7fc78ce00640 1 --2- 172.21.15.69:0/878324783 >> v2:172.21.15.121:3300/0 conn(0x7fc78815e960 0x7fc788162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.637+0000 7fc78ce00640 1 -- 172.21.15.69:0/878324783 >> v1:172.21.15.69:6789/0 conn(0x7fc78806d4f0 legacy=0x7fc7880cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.637+0000 7fc78ce00640 1 -- 172.21.15.69:0/878324783 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc788163100 con 0x7fc788162980 2024-04-24T08:18:37.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.637+0000 7fc787e00640 1 -- 172.21.15.69:0/878324783 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1453617009 0 0) 0x7fc78815ed40 con 0x7fc78806d4f0 2024-04-24T08:18:37.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.638+0000 7fc78ce00640 1 --2- 172.21.15.69:0/878324783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc788162980 0x7fc78806cfb0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fc78808c6d0 tx=0x7fc77402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f369711d51a4ce server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:37.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.638+0000 7fc787e00640 1 -- 172.21.15.69:0/878324783 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774037070 con 0x7fc788162980 2024-04-24T08:18:37.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.638+0000 7fc787e00640 1 -- 172.21.15.69:0/878324783 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc774032440 con 0x7fc788162980 2024-04-24T08:18:37.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.638+0000 7fc787e00640 1 -- 172.21.15.69:0/878324783 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774032810 con 0x7fc788162980 2024-04-24T08:18:37.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.639+0000 7fc78ec00640 1 -- 172.21.15.69:0/878324783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc788162980 msgr2=0x7fc78806cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.639+0000 7fc78ec00640 1 --2- 172.21.15.69:0/878324783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc788162980 0x7fc78806cfb0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fc78808c6d0 tx=0x7fc77402a270 comp rx=0 tx=0).stop 2024-04-24T08:18:37.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.639+0000 7fc78ec00640 1 -- 172.21.15.69:0/878324783 shutdown_connections 2024-04-24T08:18:37.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.639+0000 7fc78ec00640 1 --2- 172.21.15.69:0/878324783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc788162980 0x7fc78806cfb0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.639+0000 7fc78ec00640 1 --2- 172.21.15.69:0/878324783 >> v2:172.21.15.121:3300/0 conn(0x7fc78815e960 0x7fc788162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.639+0000 7fc78ec00640 1 -- 172.21.15.69:0/878324783 >> 172.21.15.69:0/878324783 conn(0x7fc78809d900 msgr2=0x7fc78808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:37.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.639+0000 7fc78ec00640 1 -- 172.21.15.69:0/878324783 shutdown_connections 2024-04-24T08:18:37.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.639+0000 7fc78ec00640 1 -- 172.21.15.69:0/878324783 wait complete. 2024-04-24T08:18:37.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.641+0000 7fc78ec00640 1 Processor -- start 2024-04-24T08:18:37.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.641+0000 7fc78ec00640 1 -- start start 2024-04-24T08:18:37.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.642+0000 7fc78ec00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78806d4f0 0x7fc788152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.642+0000 7fc78ec00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc78815e960 0x7fc7880b9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.642+0000 7fc78ec00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc788162980 0x7fc7881f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.642+0000 7fc78ec00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc7880b6170 con 0x7fc78815e960 2024-04-24T08:18:37.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.642+0000 7fc78ec00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc7880cf5d0 con 0x7fc788162980 2024-04-24T08:18:37.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.642+0000 7fc78ec00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc7880b62f0 con 0x7fc78806d4f0 2024-04-24T08:18:37.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.642+0000 7fc78ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc78815e960 0x7fc7880b9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.642+0000 7fc78ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc78815e960 0x7fc7880b9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53138/0 (socket says 172.21.15.69:53138) 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.642+0000 7fc78ce00640 1 -- 172.21.15.69:0/2713515086 learned_addr learned my addr 172.21.15.69:0/2713515086 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.643+0000 7fc78ce00640 1 -- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78806d4f0 msgr2=0x7fc788152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.643+0000 7fc78ce00640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78806d4f0 0x7fc788152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.643+0000 7fc78ce00640 1 -- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc788162980 msgr2=0x7fc7881f52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.643+0000 7fc78ce00640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc788162980 0x7fc7881f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.643+0000 7fc78ce00640 1 -- 172.21.15.69:0/2713515086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7880b1950 con 0x7fc78815e960 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.643+0000 7fc78ce00640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc78815e960 0x7fc7880b9b90 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fc774000c00 tx=0x7fc7740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.643+0000 7fc785600640 1 -- 172.21.15.69:0/2713515086 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774041020 con 0x7fc78815e960 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.643+0000 7fc785600640 1 -- 172.21.15.69:0/2713515086 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7740306c0 con 0x7fc78815e960 2024-04-24T08:18:37.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.643+0000 7fc78ec00640 1 -- 172.21.15.69:0/2713515086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7880b1c30 con 0x7fc78815e960 2024-04-24T08:18:37.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.643+0000 7fc785600640 1 -- 172.21.15.69:0/2713515086 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc774037040 con 0x7fc78815e960 2024-04-24T08:18:37.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.644+0000 7fc78ec00640 1 -- 172.21.15.69:0/2713515086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7880b21f0 con 0x7fc78815e960 2024-04-24T08:18:37.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.645+0000 7fc78ec00640 1 -- 172.21.15.69:0/2713515086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc788075d50 con 0x7fc78815e960 2024-04-24T08:18:37.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.646+0000 7fc785600640 1 -- 172.21.15.69:0/2713515086 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc774044020 con 0x7fc78815e960 2024-04-24T08:18:37.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.647+0000 7fc785600640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc76806c8f0 0x7fc76806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:37.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.647+0000 7fc785600640 1 -- 172.21.15.69:0/2713515086 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fc7740ac420 con 0x7fc78815e960 2024-04-24T08:18:37.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.647+0000 7fc78d800640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc76806c8f0 0x7fc76806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:37.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.649+0000 7fc78d800640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc76806c8f0 0x7fc76806edb0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc77c0098f0 tx=0x7fc77c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:37.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.651+0000 7fc785600640 1 -- 172.21.15.69:0/2713515086 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc77404b030 con 0x7fc78815e960 2024-04-24T08:18:37.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.865+0000 7fc78ec00640 1 -- 172.21.15.69:0/2713515086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc7880d1dc0 con 0x7fc78815e960 2024-04-24T08:18:37.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.866+0000 7fc785600640 1 -- 172.21.15.69:0/2713515086 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v97) ==== 80+0+0 (secure 0 0 0) 0x7fc77407a7b0 con 0x7fc78815e960 2024-04-24T08:18:37.868 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:18:37.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 -- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc76806c8f0 msgr2=0x7fc76806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc76806c8f0 0x7fc76806edb0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc77c0098f0 tx=0x7fc77c008040 comp rx=0 tx=0).stop 2024-04-24T08:18:37.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 -- 172.21.15.69:0/2713515086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc78815e960 msgr2=0x7fc7880b9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:37.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc78815e960 0x7fc7880b9b90 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fc774000c00 tx=0x7fc7740046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 -- 172.21.15.69:0/2713515086 shutdown_connections 2024-04-24T08:18:37.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc76806c8f0 0x7fc76806edb0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc78815e960 0x7fc7880b9b90 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc788162980 0x7fc7881f52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 --2- 172.21.15.69:0/2713515086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc78806d4f0 0x7fc788152810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:37.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 -- 172.21.15.69:0/2713515086 >> 172.21.15.69:0/2713515086 conn(0x7fc78809d900 msgr2=0x7fc7880b4100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:37.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 -- 172.21.15.69:0/2713515086 shutdown_connections 2024-04-24T08:18:37.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:37.869+0000 7fc78ec00640 1 -- 172.21.15.69:0/2713515086 wait complete. 2024-04-24T08:18:37.891 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:18:37.891 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:37.935 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:18:38.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.174+0000 7fa683800640 1 Processor -- start 2024-04-24T08:18:38.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.174+0000 7fa683800640 1 -- start start 2024-04-24T08:18:38.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.176+0000 7fa683800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa67c15f520 0x7fa67c15f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.176+0000 7fa683800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa67c0b4ca0 0x7fa67c06e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.176+0000 7fa683800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa67c06d350 con 0x7fa67c06ba20 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.176+0000 7fa683800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa67c06c2b0 con 0x7fa67c0b4ca0 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.176+0000 7fa683800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa67c06c590 con 0x7fa67c15f520 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.176+0000 7fa681800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa67c0b4ca0 0x7fa67c06e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.176+0000 7fa681800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa67c0b4ca0 0x7fa67c06e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53144/0 (socket says 172.21.15.69:53144) 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.176+0000 7fa67be00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa67c06ba20 0x7fa67c0b4590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46950/0 (socket says 172.21.15.69:46950) 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.176+0000 7fa681800640 1 -- 172.21.15.69:0/2535358214 learned_addr learned my addr 172.21.15.69:0/2535358214 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.177+0000 7fa680e00640 1 --2- 172.21.15.69:0/2535358214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa67c15f520 0x7fa67c15f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.177+0000 7fa681800640 1 -- 172.21.15.69:0/2535358214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa67c15f520 msgr2=0x7fa67c15f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.177+0000 7fa681800640 1 --2- 172.21.15.69:0/2535358214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa67c15f520 0x7fa67c15f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.177+0000 7fa681800640 1 -- 172.21.15.69:0/2535358214 >> v1:172.21.15.69:6789/0 conn(0x7fa67c06ba20 legacy=0x7fa67c0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.177+0000 7fa681800640 1 -- 172.21.15.69:0/2535358214 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67c06c870 con 0x7fa67c0b4ca0 2024-04-24T08:18:38.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.177+0000 7fa67b400640 1 -- 172.21.15.69:0/2535358214 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3740341589 0 0) 0x7fa67c06d350 con 0x7fa67c06ba20 2024-04-24T08:18:38.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.178+0000 7fa681800640 1 --2- 172.21.15.69:0/2535358214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c0b4ca0 0x7fa67c06e880 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fa674009e50 tx=0x7fa67402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79fad0e69b6bb2a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:38.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.178+0000 7fa67b400640 1 -- 172.21.15.69:0/2535358214 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa674036070 con 0x7fa67c0b4ca0 2024-04-24T08:18:38.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.178+0000 7fa67b400640 1 -- 172.21.15.69:0/2535358214 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa674031600 con 0x7fa67c0b4ca0 2024-04-24T08:18:38.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.178+0000 7fa67b400640 1 -- 172.21.15.69:0/2535358214 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6740319d0 con 0x7fa67c0b4ca0 2024-04-24T08:18:38.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.179+0000 7fa683800640 1 -- 172.21.15.69:0/2535358214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c0b4ca0 msgr2=0x7fa67c06e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.179+0000 7fa683800640 1 --2- 172.21.15.69:0/2535358214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c0b4ca0 0x7fa67c06e880 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fa674009e50 tx=0x7fa67402a710 comp rx=0 tx=0).stop 2024-04-24T08:18:38.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.179+0000 7fa683800640 1 -- 172.21.15.69:0/2535358214 shutdown_connections 2024-04-24T08:18:38.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.179+0000 7fa683800640 1 --2- 172.21.15.69:0/2535358214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c0b4ca0 0x7fa67c06e880 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.179+0000 7fa683800640 1 --2- 172.21.15.69:0/2535358214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa67c15f520 0x7fa67c15f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.179+0000 7fa683800640 1 -- 172.21.15.69:0/2535358214 >> 172.21.15.69:0/2535358214 conn(0x7fa67c0b3230 msgr2=0x7fa67c0d0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:38.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.179+0000 7fa683800640 1 -- 172.21.15.69:0/2535358214 shutdown_connections 2024-04-24T08:18:38.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.179+0000 7fa683800640 1 -- 172.21.15.69:0/2535358214 wait complete. 2024-04-24T08:18:38.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.181+0000 7fa683800640 1 Processor -- start 2024-04-24T08:18:38.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.181+0000 7fa683800640 1 -- start start 2024-04-24T08:18:38.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.182+0000 7fa683800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa67c06ba20 0x7fa67c145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.182+0000 7fa683800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa67c0b4ca0 0x7fa67c1556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.182+0000 7fa683800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c15f520 0x7fa67c1f43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.182+0000 7fa683800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c06bea0 con 0x7fa67c15f520 2024-04-24T08:18:38.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.182+0000 7fa683800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa67c06ce30 con 0x7fa67c0b4ca0 2024-04-24T08:18:38.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.182+0000 7fa683800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa67c06c020 con 0x7fa67c06ba20 2024-04-24T08:18:38.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.182+0000 7fa681800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c15f520 0x7fa67c1f43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.182+0000 7fa681800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c15f520 0x7fa67c1f43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53156/0 (socket says 172.21.15.69:53156) 2024-04-24T08:18:38.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.182+0000 7fa681800640 1 -- 172.21.15.69:0/3416258846 learned_addr learned my addr 172.21.15.69:0/3416258846 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:38.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa67be00640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa67c0b4ca0 0x7fa67c1556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa680e00640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa67c06ba20 0x7fa67c145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa681800640 1 -- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa67c06ba20 msgr2=0x7fa67c145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa681800640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa67c06ba20 0x7fa67c145780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa681800640 1 -- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa67c0b4ca0 msgr2=0x7fa67c1556c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa681800640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa67c0b4ca0 0x7fa67c1556c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa681800640 1 -- 172.21.15.69:0/3416258846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa67c152c70 con 0x7fa67c15f520 2024-04-24T08:18:38.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa681800640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c15f520 0x7fa67c1f43b0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fa674034040 tx=0x7fa674004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:38.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa678c00640 1 -- 172.21.15.69:0/3416258846 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa674041020 con 0x7fa67c15f520 2024-04-24T08:18:38.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa678c00640 1 -- 172.21.15.69:0/3416258846 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa674037040 con 0x7fa67c15f520 2024-04-24T08:18:38.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa683800640 1 -- 172.21.15.69:0/3416258846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa67c150a40 con 0x7fa67c15f520 2024-04-24T08:18:38.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.183+0000 7fa678c00640 1 -- 172.21.15.69:0/3416258846 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa674036050 con 0x7fa67c15f520 2024-04-24T08:18:38.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.184+0000 7fa683800640 1 -- 172.21.15.69:0/3416258846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa67c1f48f0 con 0x7fa67c15f520 2024-04-24T08:18:38.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.185+0000 7fa683800640 1 -- 172.21.15.69:0/3416258846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa67c068a80 con 0x7fa67c15f520 2024-04-24T08:18:38.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.186+0000 7fa678c00640 1 -- 172.21.15.69:0/3416258846 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa674007c30 con 0x7fa67c15f520 2024-04-24T08:18:38.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.187+0000 7fa678c00640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa66006c970 0x7fa66006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.187+0000 7fa678c00640 1 -- 172.21.15.69:0/3416258846 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fa6740ac180 con 0x7fa67c15f520 2024-04-24T08:18:38.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.187+0000 7fa680e00640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa66006c970 0x7fa66006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.189+0000 7fa680e00640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa66006c970 0x7fa66006ee30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa66c006fd0 tx=0x7fa66c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:38.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.192+0000 7fa678c00640 1 -- 172.21.15.69:0/3416258846 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa67404b060 con 0x7fa67c15f520 2024-04-24T08:18:38.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.394+0000 7fa683800640 1 -- 172.21.15.69:0/3416258846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa67c0b1350 con 0x7fa67c15f520 2024-04-24T08:18:38.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.396+0000 7fa678c00640 1 -- 172.21.15.69:0/3416258846 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) ==== 74+0+16584 (secure 0 0 0) 0x7fa67407a720 con 0x7fa67c15f520 2024-04-24T08:18:38.398 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:38.398 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":97,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:18:11.928428+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:08.156864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:09.368519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:18:38.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.398+0000 7fa683800640 1 -- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa66006c970 msgr2=0x7fa66006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.398+0000 7fa683800640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa66006c970 0x7fa66006ee30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa66c006fd0 tx=0x7fa66c008040 comp rx=0 tx=0).stop 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.398+0000 7fa683800640 1 -- 172.21.15.69:0/3416258846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c15f520 msgr2=0x7fa67c1f43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.398+0000 7fa683800640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c15f520 0x7fa67c1f43b0 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fa674034040 tx=0x7fa674004a40 comp rx=0 tx=0).stop 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.399+0000 7fa683800640 1 -- 172.21.15.69:0/3416258846 shutdown_connections 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.399+0000 7fa683800640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa66006c970 0x7fa66006ee30 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.399+0000 7fa683800640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa67c15f520 0x7fa67c1f43b0 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.399+0000 7fa683800640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa67c0b4ca0 0x7fa67c1556c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.399+0000 7fa683800640 1 --2- 172.21.15.69:0/3416258846 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa67c06ba20 0x7fa67c145780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.399+0000 7fa683800640 1 -- 172.21.15.69:0/3416258846 >> 172.21.15.69:0/3416258846 conn(0x7fa67c0b3230 msgr2=0x7fa67c158940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.399+0000 7fa683800640 1 -- 172.21.15.69:0/3416258846 shutdown_connections 2024-04-24T08:18:38.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.399+0000 7fa683800640 1 -- 172.21.15.69:0/3416258846 wait complete. 2024-04-24T08:18:38.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:08.156864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:09.368519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:18:38.420 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:18:38.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.622+0000 7f4a06800640 1 Processor -- start 2024-04-24T08:18:38.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.623+0000 7f4a06800640 1 -- start start 2024-04-24T08:18:38.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.624+0000 7f4a06800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a0015c820 0x7f4a000b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.624+0000 7f4a06800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4a000ce250 0x7f4a000ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.624+0000 7f4a06800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0015cc00 con 0x7f4a000b46a0 2024-04-24T08:18:38.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.624+0000 7f4a06800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4a0006ed30 con 0x7f4a000ce250 2024-04-24T08:18:38.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.624+0000 7f4a06800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a000b4e10 con 0x7f4a0015c820 2024-04-24T08:18:38.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.624+0000 7f49fea00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4a000b46a0 0x7f4a0006f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46960/0 (socket says 172.21.15.69:46960) 2024-04-24T08:18:38.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.624+0000 7f49fea00640 1 -- 172.21.15.69:0/1843821681 learned_addr learned my addr 172.21.15.69:0/1843821681 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:38.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49ff400640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a0015c820 0x7f4a000b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49ffe00640 1 --2- 172.21.15.69:0/1843821681 >> v2:172.21.15.121:3300/0 conn(0x7f4a000ce250 0x7f4a000ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804126896 0 0) 0x7f4a0015cc00 con 0x7f4a000b46a0 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49e4003520 con 0x7f4a000b46a0 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3765439308 0 0) 0x7f49e4003520 con 0x7f4a000b46a0 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a0015c820 msgr2=0x7f4a000b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a0015c820 0x7f4a000b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 >> v2:172.21.15.121:3300/0 conn(0x7f4a000ce250 msgr2=0x7f4a000ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 --2- 172.21.15.69:0/1843821681 >> v2:172.21.15.121:3300/0 conn(0x7f4a000ce250 0x7f4a000ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a000b50f0 con 0x7f4a000b46a0 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f49e8001670 con 0x7f4a000b46a0 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 >> v1:172.21.15.69:6789/0 conn(0x7f4a000b46a0 legacy=0x7f4a0006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a0015c820 0x7f49e40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49e4003f30 0x7f49e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49e4004d40 0x7f49e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0006ed30 con 0x7f49e4003f30 2024-04-24T08:18:38.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4a000b4e10 con 0x7f49e4004d40 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.625+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f49e400ba10 con 0x7f4a0015c820 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.626+0000 7f49fea00640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49e4003f30 0x7f49e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.626+0000 7f49ff400640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a0015c820 0x7f49e40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.626+0000 7f49fea00640 1 -- 172.21.15.69:0/1843821681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a0015c820 msgr2=0x7f49e40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.626+0000 7f49fea00640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a0015c820 0x7f49e40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.626+0000 7f49fea00640 1 -- 172.21.15.69:0/1843821681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49e4004d40 msgr2=0x7f49e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.626+0000 7f49fea00640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49e4004d40 0x7f49e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.626+0000 7f49fea00640 1 -- 172.21.15.69:0/1843821681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49e4003520 con 0x7f49e4003f30 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.627+0000 7f49fea00640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49e4003f30 0x7f49e4004800 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f49e8007fd0 tx=0x7f49e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6bc6f031f55659 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.627+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8007d50 con 0x7f49e4003f30 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.627+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49e8005ad0 con 0x7f49e4003f30 2024-04-24T08:18:38.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.627+0000 7f49fe000640 1 -- 172.21.15.69:0/1843821681 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8007070 con 0x7f49e4003f30 2024-04-24T08:18:38.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.628+0000 7f4a06800640 1 -- 172.21.15.69:0/1843821681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49e4003f30 msgr2=0x7f49e4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.628+0000 7f4a06800640 1 --2- 172.21.15.69:0/1843821681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f49e4003f30 0x7f49e4004800 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f49e8007fd0 tx=0x7f49e4005830 comp rx=0 tx=0).stop 2024-04-24T08:18:38.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.628+0000 7f49ffe00640 1 -- 172.21.15.69:0/1843821681 reap_dead start 2024-04-24T08:18:38.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.628+0000 7f4a06800640 1 -- 172.21.15.69:0/1843821681 shutdown_connections 2024-04-24T08:18:38.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.628+0000 7f4a06800640 1 -- 172.21.15.69:0/1843821681 >> 172.21.15.69:0/1843821681 conn(0x7f4a000d0350 msgr2=0x7f4a000d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:38.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.628+0000 7f4a06800640 1 -- 172.21.15.69:0/1843821681 shutdown_connections 2024-04-24T08:18:38.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.628+0000 7f4a06800640 1 -- 172.21.15.69:0/1843821681 wait complete. 2024-04-24T08:18:38.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.630+0000 7f4a06800640 1 Processor -- start 2024-04-24T08:18:38.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.630+0000 7f4a06800640 1 -- start start 2024-04-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.631+0000 7f4a06800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49e4003f30 0x7f4a001498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.631+0000 7f4a06800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0015ab20 0x7f4a001f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.631+0000 7f4a06800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a001f2c20 0x7f4a00154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.631+0000 7f4a06800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a000b4b00 con 0x7f4a0015ab20 2024-04-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.631+0000 7f4a06800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0006f2b0 con 0x7f49e4003f30 2024-04-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.631+0000 7f4a06800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a0006fee0 con 0x7f4a001f2c20 2024-04-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.631+0000 7f49fea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0015ab20 0x7f4a001f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.631+0000 7f49fea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0015ab20 0x7f4a001f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53170/0 (socket says 172.21.15.69:53170) 2024-04-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.631+0000 7f49fea00640 1 -- 172.21.15.69:0/2309354120 learned_addr learned my addr 172.21.15.69:0/2309354120 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:38.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f49fea00640 1 -- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a001f2c20 msgr2=0x7f4a00154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f49fea00640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a001f2c20 0x7f4a00154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f49ff400640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49e4003f30 0x7f4a001498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f49fea00640 1 -- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49e4003f30 msgr2=0x7f4a001498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f49fea00640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49e4003f30 0x7f4a001498f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f49fea00640 1 -- 172.21.15.69:0/2309354120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0015e280 con 0x7f4a0015ab20 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f49fea00640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0015ab20 0x7f4a001f26e0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f49e800c300 tx=0x7f4a000b52c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f49f7400640 1 -- 172.21.15.69:0/2309354120 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8042020 con 0x7f4a0015ab20 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f49f7400640 1 -- 172.21.15.69:0/2309354120 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49e8007070 con 0x7f4a0015ab20 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f49f7400640 1 -- 172.21.15.69:0/2309354120 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49e8007d50 con 0x7f4a0015ab20 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f4a06800640 1 -- 172.21.15.69:0/2309354120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0015e560 con 0x7f4a0015ab20 2024-04-24T08:18:38.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.632+0000 7f4a06800640 1 -- 172.21.15.69:0/2309354120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0015eb20 con 0x7f4a0015ab20 2024-04-24T08:18:38.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.634+0000 7f4a06800640 1 -- 172.21.15.69:0/2309354120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a00068a80 con 0x7f4a0015ab20 2024-04-24T08:18:38.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.635+0000 7f49f7400640 1 -- 172.21.15.69:0/2309354120 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f49e8045020 con 0x7f4a0015ab20 2024-04-24T08:18:38.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.636+0000 7f49f7400640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49dc06c8f0 0x7f49dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:38.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.636+0000 7f49f7400640 1 -- 172.21.15.69:0/2309354120 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f49e80ac330 con 0x7f4a0015ab20 2024-04-24T08:18:38.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.637+0000 7f49ff400640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49dc06c8f0 0x7f49dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:38.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.638+0000 7f49ff400640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49dc06c8f0 0x7f49dc06edb0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f49f0009870 tx=0x7f49f0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:38.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.641+0000 7f49f7400640 1 -- 172.21.15.69:0/2309354120 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f49e804c030 con 0x7f4a0015ab20 2024-04-24T08:18:38.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.837+0000 7f4a06800640 1 -- 172.21.15.69:0/2309354120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4a000cddc0 con 0x7f4a0015ab20 2024-04-24T08:18:38.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.838+0000 7f49f7400640 1 -- 172.21.15.69:0/2309354120 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) ==== 93+0+10 (secure 0 0 0) 0x7f49e807a7f0 con 0x7f4a0015ab20 2024-04-24T08:18:38.839 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:18:38.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 -- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49dc06c8f0 msgr2=0x7f49dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49dc06c8f0 0x7f49dc06edb0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f49f0009870 tx=0x7f49f0008040 comp rx=0 tx=0).stop 2024-04-24T08:18:38.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 -- 172.21.15.69:0/2309354120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0015ab20 msgr2=0x7f4a001f26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:38.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0015ab20 0x7f4a001f26e0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f49e800c300 tx=0x7f4a000b52c0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 -- 172.21.15.69:0/2309354120 shutdown_connections 2024-04-24T08:18:38.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f49dc06c8f0 0x7f49dc06edb0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0015ab20 0x7f4a001f26e0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a001f2c20 0x7f4a00154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 --2- 172.21.15.69:0/2309354120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f49e4003f30 0x7f4a001498f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:38.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 -- 172.21.15.69:0/2309354120 >> 172.21.15.69:0/2309354120 conn(0x7f4a000d0350 msgr2=0x7f4a000b3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:38.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.840+0000 7f4a06800640 1 -- 172.21.15.69:0/2309354120 shutdown_connections 2024-04-24T08:18:38.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:38.841+0000 7f4a06800640 1 -- 172.21.15.69:0/2309354120 wait complete. 2024-04-24T08:18:38.860 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:18:39.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.059+0000 7f99a4a00640 1 Processor -- start 2024-04-24T08:18:39.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.059+0000 7f99a4a00640 1 -- start start 2024-04-24T08:18:39.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.061+0000 7f99a4a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f99a01420a0 0x7f99a0071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.061+0000 7f99a4a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99a01606e0 0x7f99a00cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.061+0000 7f99a4a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99a0072450 con 0x7f99a0142480 2024-04-24T08:18:39.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.061+0000 7f99a4a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f99a00cf1e0 con 0x7f99a01420a0 2024-04-24T08:18:39.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.061+0000 7f99a4a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99a0072730 con 0x7f99a01606e0 2024-04-24T08:18:39.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.062+0000 7f999ea00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f99a0142480 0x7f99a015ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:37042/0 (socket says 172.21.15.69:37042) 2024-04-24T08:18:39.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.062+0000 7f999ea00640 1 -- 172.21.15.69:0/2971415518 learned_addr learned my addr 172.21.15.69:0/2971415518 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:39.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.062+0000 7f999f400640 1 --2- 172.21.15.69:0/2971415518 >> v2:172.21.15.121:3300/0 conn(0x7f99a01420a0 0x7f99a0071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.062+0000 7f999fe00640 1 --2- 172.21.15.69:0/2971415518 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99a01606e0 0x7f99a00cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.063+0000 7f999f400640 1 -- 172.21.15.69:0/2971415518 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99a01606e0 msgr2=0x7f99a00cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:39.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.063+0000 7f999f400640 1 --2- 172.21.15.69:0/2971415518 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99a01606e0 0x7f99a00cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.063+0000 7f999f400640 1 -- 172.21.15.69:0/2971415518 >> v1:172.21.15.121:6789/0 conn(0x7f99a0142480 legacy=0x7f99a015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.063+0000 7f999f400640 1 -- 172.21.15.69:0/2971415518 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a0072a10 con 0x7f99a01420a0 2024-04-24T08:18:39.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.064+0000 7f999f400640 1 --2- 172.21.15.69:0/2971415518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99a01420a0 0x7f99a0071d70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f99900097a0 tx=0x7f999002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9f559cc2de2563 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:39.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.065+0000 7f999e000640 1 -- 172.21.15.69:0/2971415518 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990037070 con 0x7f99a01420a0 2024-04-24T08:18:39.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.065+0000 7f999e000640 1 -- 172.21.15.69:0/2971415518 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9990032440 con 0x7f99a01420a0 2024-04-24T08:18:39.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.065+0000 7f999e000640 1 -- 172.21.15.69:0/2971415518 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9990032810 con 0x7f99a01420a0 2024-04-24T08:18:39.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.066+0000 7f99a4a00640 1 -- 172.21.15.69:0/2971415518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99a01420a0 msgr2=0x7f99a0071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.066+0000 7f99a4a00640 1 --2- 172.21.15.69:0/2971415518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99a01420a0 0x7f99a0071d70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f99900097a0 tx=0x7f999002a180 comp rx=0 tx=0).stop 2024-04-24T08:18:39.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.066+0000 7f99a4a00640 1 -- 172.21.15.69:0/2971415518 shutdown_connections 2024-04-24T08:18:39.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.066+0000 7f99a4a00640 1 --2- 172.21.15.69:0/2971415518 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99a01606e0 0x7f99a00cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.066+0000 7f99a4a00640 1 --2- 172.21.15.69:0/2971415518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99a01420a0 0x7f99a0071d70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.066+0000 7f99a4a00640 1 -- 172.21.15.69:0/2971415518 >> 172.21.15.69:0/2971415518 conn(0x7f99a0067b80 msgr2=0x7f99a006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:39.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.066+0000 7f99a4a00640 1 -- 172.21.15.69:0/2971415518 shutdown_connections 2024-04-24T08:18:39.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.066+0000 7f99a4a00640 1 -- 172.21.15.69:0/2971415518 wait complete. 2024-04-24T08:18:39.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.068+0000 7f99a4a00640 1 Processor -- start 2024-04-24T08:18:39.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.069+0000 7f99a4a00640 1 -- start start 2024-04-24T08:18:39.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f99a4a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99a01420a0 0x7f99a006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f99a4a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99a0142480 0x7f99a0153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f99a4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a01606e0 0x7f99a0155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f99a4a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f99a00cfbb0 con 0x7f99a01606e0 2024-04-24T08:18:39.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f99a4a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f99a0142910 con 0x7f99a0142480 2024-04-24T08:18:39.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f99a4a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99a00cfd30 con 0x7f99a01420a0 2024-04-24T08:18:39.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f999fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a01606e0 0x7f99a0155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f999fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a01606e0 0x7f99a0155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37538/0 (socket says 172.21.15.69:37538) 2024-04-24T08:18:39.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f999fe00640 1 -- 172.21.15.69:0/3932665908 learned_addr learned my addr 172.21.15.69:0/3932665908 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:39.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f999ea00640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99a0142480 0x7f99a0153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.070+0000 7f999f400640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99a01420a0 0x7f99a006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f999fe00640 1 -- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99a01420a0 msgr2=0x7f99a006ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f999fe00640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99a01420a0 0x7f99a006ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f999fe00640 1 -- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99a0142480 msgr2=0x7f99a0153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f999fe00640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99a0142480 0x7f99a0153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f999fe00640 1 -- 172.21.15.69:0/3932665908 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a01544f0 con 0x7f99a01606e0 2024-04-24T08:18:39.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f999fe00640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a01606e0 0x7f99a0155a70 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f9998009cb0 tx=0x7f999800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:39.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f9997400640 1 -- 172.21.15.69:0/3932665908 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998018020 con 0x7f99a01606e0 2024-04-24T08:18:39.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f9997400640 1 -- 172.21.15.69:0/3932665908 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9998014480 con 0x7f99a01606e0 2024-04-24T08:18:39.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f9997400640 1 -- 172.21.15.69:0/3932665908 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998014870 con 0x7f99a01606e0 2024-04-24T08:18:39.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f99a4a00640 1 -- 172.21.15.69:0/3932665908 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a01547d0 con 0x7f99a01606e0 2024-04-24T08:18:39.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.071+0000 7f99a4a00640 1 -- 172.21.15.69:0/3932665908 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a0155070 con 0x7f99a01606e0 2024-04-24T08:18:39.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.072+0000 7f99a4a00640 1 -- 172.21.15.69:0/3932665908 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a01549a0 con 0x7f99a01606e0 2024-04-24T08:18:39.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.073+0000 7f9997400640 1 -- 172.21.15.69:0/3932665908 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9998005d10 con 0x7f99a01606e0 2024-04-24T08:18:39.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.074+0000 7f9997400640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f998006c8f0 0x7f998006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.074+0000 7f9997400640 1 -- 172.21.15.69:0/3932665908 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f999808c8d0 con 0x7f99a01606e0 2024-04-24T08:18:39.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.075+0000 7f999f400640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f998006c8f0 0x7f998006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.076+0000 7f999f400640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f998006c8f0 0x7f998006edb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9990034000 tx=0x7f999002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:39.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.079+0000 7f9997400640 1 -- 172.21.15.69:0/3932665908 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9998022030 con 0x7f99a01606e0 2024-04-24T08:18:39.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.277+0000 7f99a4a00640 1 -- 172.21.15.69:0/3932665908 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f99a0075d50 con 0x7f99a01606e0 2024-04-24T08:18:39.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.278+0000 7f9997400640 1 -- 172.21.15.69:0/3932665908 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) ==== 104+0+11 (secure 0 0 0) 0x7f9998010040 con 0x7f99a01606e0 2024-04-24T08:18:39.280 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:18:39.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.281+0000 7f99a4a00640 1 -- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f998006c8f0 msgr2=0x7f998006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.281+0000 7f99a4a00640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f998006c8f0 0x7f998006edb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f9990034000 tx=0x7f999002e000 comp rx=0 tx=0).stop 2024-04-24T08:18:39.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.281+0000 7f99a4a00640 1 -- 172.21.15.69:0/3932665908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a01606e0 msgr2=0x7f99a0155a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.281+0000 7f99a4a00640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a01606e0 0x7f99a0155a70 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f9998009cb0 tx=0x7f999800c490 comp rx=0 tx=0).stop 2024-04-24T08:18:39.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.281+0000 7f99a4a00640 1 -- 172.21.15.69:0/3932665908 shutdown_connections 2024-04-24T08:18:39.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.281+0000 7f99a4a00640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f998006c8f0 0x7f998006edb0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.281+0000 7f99a4a00640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f99a01606e0 0x7f99a0155a70 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.281+0000 7f99a4a00640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99a0142480 0x7f99a0153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.281+0000 7f99a4a00640 1 --2- 172.21.15.69:0/3932665908 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99a01420a0 0x7f99a006ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.281+0000 7f99a4a00640 1 -- 172.21.15.69:0/3932665908 >> 172.21.15.69:0/3932665908 conn(0x7f99a0067b80 msgr2=0x7f99a006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:39.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.282+0000 7f99a4a00640 1 -- 172.21.15.69:0/3932665908 shutdown_connections 2024-04-24T08:18:39.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.282+0000 7f99a4a00640 1 -- 172.21.15.69:0/3932665908 wait complete. 2024-04-24T08:18:39.300 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.494+0000 7f74e8e00640 1 Processor -- start 2024-04-24T08:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.495+0000 7f74e8e00640 1 -- start start 2024-04-24T08:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.496+0000 7f74e8e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 0x7f74e40b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.496+0000 7f74e8e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f74e40d09f0 0x7f74e406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.496+0000 7f74e8e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74e40ce910 con 0x7f74e40cdf30 2024-04-24T08:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.496+0000 7f74e8e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f74e40cebf0 con 0x7f74e40d09f0 2024-04-24T08:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.496+0000 7f74e8e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f74e40ceed0 con 0x7f74e415ea60 2024-04-24T08:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.497+0000 7f74e3400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 0x7f74e40b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.497+0000 7f74daa00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f74e40cdf30 0x7f74e40d02e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:37048/0 (socket says 172.21.15.69:37048) 2024-04-24T08:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.497+0000 7f74daa00640 1 -- 172.21.15.69:0/1115489500 learned_addr learned my addr 172.21.15.69:0/1115489500 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.497+0000 7f74e3400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 0x7f74e40b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:47680/0 (socket says 172.21.15.69:47680) 2024-04-24T08:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.497+0000 7f74e3e00640 1 --2- 172.21.15.69:0/1115489500 >> v2:172.21.15.121:3300/0 conn(0x7f74e40d09f0 0x7f74e406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.498+0000 7f74e3400640 1 -- 172.21.15.69:0/1115489500 >> v2:172.21.15.121:3300/0 conn(0x7f74e40d09f0 msgr2=0x7f74e406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.498+0000 7f74e3400640 1 --2- 172.21.15.69:0/1115489500 >> v2:172.21.15.121:3300/0 conn(0x7f74e40d09f0 0x7f74e406df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.498+0000 7f74e3400640 1 -- 172.21.15.69:0/1115489500 >> v1:172.21.15.121:6789/0 conn(0x7f74e40cdf30 legacy=0x7f74e40d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.498+0000 7f74e3400640 1 -- 172.21.15.69:0/1115489500 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e40cf1b0 con 0x7f74e415ea60 2024-04-24T08:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.499+0000 7f74e3400640 1 --2- 172.21.15.69:0/1115489500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 0x7f74e40b5300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f74d400ba40 tx=0x7f74d402c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecd57665314dad43 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.500+0000 7f74e2a00640 1 -- 172.21.15.69:0/1115489500 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4031710 con 0x7f74e415ea60 2024-04-24T08:18:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.500+0000 7f74e2a00640 1 -- 172.21.15.69:0/1115489500 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d40318b0 con 0x7f74e415ea60 2024-04-24T08:18:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.500+0000 7f74e2a00640 1 -- 172.21.15.69:0/1115489500 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d4031c80 con 0x7f74e415ea60 2024-04-24T08:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.500+0000 7f74e8e00640 1 -- 172.21.15.69:0/1115489500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 msgr2=0x7f74e40b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.500+0000 7f74e8e00640 1 --2- 172.21.15.69:0/1115489500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 0x7f74e40b5300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f74d400ba40 tx=0x7f74d402c7b0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.501+0000 7f74e8e00640 1 -- 172.21.15.69:0/1115489500 shutdown_connections 2024-04-24T08:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.501+0000 7f74e8e00640 1 --2- 172.21.15.69:0/1115489500 >> v2:172.21.15.121:3300/0 conn(0x7f74e40d09f0 0x7f74e406df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.501+0000 7f74e8e00640 1 --2- 172.21.15.69:0/1115489500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 0x7f74e40b5300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.501+0000 7f74e8e00640 1 -- 172.21.15.69:0/1115489500 >> 172.21.15.69:0/1115489500 conn(0x7f74e4067b80 msgr2=0x7f74e406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.501+0000 7f74e8e00640 1 -- 172.21.15.69:0/1115489500 shutdown_connections 2024-04-24T08:18:39.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.501+0000 7f74e8e00640 1 -- 172.21.15.69:0/1115489500 wait complete. 2024-04-24T08:18:39.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.503+0000 7f74e8e00640 1 Processor -- start 2024-04-24T08:18:39.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.503+0000 7f74e8e00640 1 -- start start 2024-04-24T08:18:39.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.504+0000 7f74e8e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74e40cdf30 0x7f74e4153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.504+0000 7f74e8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74e40d09f0 0x7f74e4153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.504+0000 7f74e8e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 0x7f74e415c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.504+0000 7f74e8e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f74e40ce510 con 0x7f74e40d09f0 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.504+0000 7f74e8e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f74e40ce390 con 0x7f74e40cdf30 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.504+0000 7f74e8e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f74e40ce690 con 0x7f74e415ea60 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.504+0000 7f74daa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74e40d09f0 0x7f74e4153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.504+0000 7f74daa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74e40d09f0 0x7f74e4153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37546/0 (socket says 172.21.15.69:37546) 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.504+0000 7f74daa00640 1 -- 172.21.15.69:0/1018341057 learned_addr learned my addr 172.21.15.69:0/1018341057 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74e3400640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74e40cdf30 0x7f74e4153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74e3e00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 0x7f74e415c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74daa00640 1 -- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 msgr2=0x7f74e415c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74daa00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 0x7f74e415c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74daa00640 1 -- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74e40cdf30 msgr2=0x7f74e4153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74daa00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74e40cdf30 0x7f74e4153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74daa00640 1 -- 172.21.15.69:0/1018341057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e4154cf0 con 0x7f74e40d09f0 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74e3400640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74e40cdf30 0x7f74e4153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:39.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74daa00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74e40d09f0 0x7f74e4153970 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c400 tx=0x7f74cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:39.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74dbe00640 1 -- 172.21.15.69:0/1018341057 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc017070 con 0x7f74e40d09f0 2024-04-24T08:18:39.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74dbe00640 1 -- 172.21.15.69:0/1018341057 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74cc005ce0 con 0x7f74e40d09f0 2024-04-24T08:18:39.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74dbe00640 1 -- 172.21.15.69:0/1018341057 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc0072f0 con 0x7f74e40d09f0 2024-04-24T08:18:39.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74e8e00640 1 -- 172.21.15.69:0/1018341057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e4154fd0 con 0x7f74e40d09f0 2024-04-24T08:18:39.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.505+0000 7f74e8e00640 1 -- 172.21.15.69:0/1018341057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e4155590 con 0x7f74e40d09f0 2024-04-24T08:18:39.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.507+0000 7f74e8e00640 1 -- 172.21.15.69:0/1018341057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e41557d0 con 0x7f74e40d09f0 2024-04-24T08:18:39.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.507+0000 7f74dbe00640 1 -- 172.21.15.69:0/1018341057 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f74cc01a070 con 0x7f74e40d09f0 2024-04-24T08:18:39.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.508+0000 7f74dbe00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74c006c920 0x7f74c006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.509+0000 7f74dbe00640 1 -- 172.21.15.69:0/1018341057 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f74cc08af80 con 0x7f74e40d09f0 2024-04-24T08:18:39.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.509+0000 7f74e3400640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74c006c920 0x7f74c006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.510+0000 7f74e3400640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74c006c920 0x7f74c006ede0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f74e4072ff0 tx=0x7f74d4035040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:39.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.513+0000 7f74dbe00640 1 -- 172.21.15.69:0/1018341057 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74e41557d0 con 0x7f74e40d09f0 2024-04-24T08:18:39.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.718+0000 7f74e8e00640 1 -- 172.21.15.69:0/1018341057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f74e406e780 con 0x7f74e40d09f0 2024-04-24T08:18:39.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.719+0000 7f74dbe00640 1 -- 172.21.15.69:0/1018341057 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) ==== 100+0+11 (secure 0 0 0) 0x7f74cc0593a0 con 0x7f74e40d09f0 2024-04-24T08:18:39.721 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:18:39.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.722+0000 7f74e8e00640 1 -- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74c006c920 msgr2=0x7f74c006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.722+0000 7f74e8e00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74c006c920 0x7f74c006ede0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f74e4072ff0 tx=0x7f74d4035040 comp rx=0 tx=0).stop 2024-04-24T08:18:39.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.722+0000 7f74e8e00640 1 -- 172.21.15.69:0/1018341057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74e40d09f0 msgr2=0x7f74e4153970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.722+0000 7f74e8e00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74e40d09f0 0x7f74e4153970 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c400 tx=0x7f74cc00c8c0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.722+0000 7f74e8e00640 1 -- 172.21.15.69:0/1018341057 shutdown_connections 2024-04-24T08:18:39.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.722+0000 7f74e8e00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f74c006c920 0x7f74c006ede0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.722+0000 7f74e8e00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74e40d09f0 0x7f74e4153970 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.722+0000 7f74e8e00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f74e415ea60 0x7f74e415c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.722+0000 7f74e8e00640 1 --2- 172.21.15.69:0/1018341057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74e40cdf30 0x7f74e4153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.722+0000 7f74e8e00640 1 -- 172.21.15.69:0/1018341057 >> 172.21.15.69:0/1018341057 conn(0x7f74e4067b80 msgr2=0x7f74e415a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:39.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.723+0000 7f74e8e00640 1 -- 172.21.15.69:0/1018341057 shutdown_connections 2024-04-24T08:18:39.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.723+0000 7f74e8e00640 1 -- 172.21.15.69:0/1018341057 wait complete. 2024-04-24T08:18:39.744 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:18:39.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.942+0000 7febd2e00640 1 Processor -- start 2024-04-24T08:18:39.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.942+0000 7febd2e00640 1 -- start start 2024-04-24T08:18:39.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.944+0000 7febd2e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febcc1523f0 0x7febcc06db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.944+0000 7febd2e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc072e20 0x7febcc073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.944+0000 7febd2e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febcc0aeac0 con 0x7febcc0737a0 2024-04-24T08:18:39.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.944+0000 7febd2e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7febcc0aeda0 con 0x7febcc1523f0 2024-04-24T08:18:39.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.944+0000 7febd2e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7febcc0af080 con 0x7febcc072e20 2024-04-24T08:18:39.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.944+0000 7febd1a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febcc1523f0 0x7febcc06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.944+0000 7febd1a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7febcc1523f0 0x7febcc06db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37554/0 (socket says 172.21.15.69:37554) 2024-04-24T08:18:39.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.944+0000 7febd1a00640 1 -- 172.21.15.69:0/827758404 learned_addr learned my addr 172.21.15.69:0/827758404 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:39.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.944+0000 7febd1000640 1 --2- 172.21.15.69:0/827758404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc072e20 0x7febcc073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.944+0000 7febd1a00640 1 -- 172.21.15.69:0/827758404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc072e20 msgr2=0x7febcc073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:39.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.945+0000 7febd1a00640 1 --2- 172.21.15.69:0/827758404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc072e20 0x7febcc073260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.945+0000 7febd1a00640 1 -- 172.21.15.69:0/827758404 >> v1:172.21.15.121:6789/0 conn(0x7febcc0737a0 legacy=0x7febcc15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:39.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.945+0000 7febd1a00640 1 -- 172.21.15.69:0/827758404 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc0af360 con 0x7febcc1523f0 2024-04-24T08:18:39.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.945+0000 7febd1000640 1 --2- 172.21.15.69:0/827758404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc072e20 0x7febcc073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:39.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.946+0000 7febd1a00640 1 --2- 172.21.15.69:0/827758404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 0x7febcc06db90 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7febc0009870 tx=0x7febc002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1933bafd41a42148 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:39.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.946+0000 7febcbe00640 1 -- 172.21.15.69:0/827758404 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc0037070 con 0x7febcc1523f0 2024-04-24T08:18:39.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.946+0000 7febcbe00640 1 -- 172.21.15.69:0/827758404 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febc0032440 con 0x7febcc1523f0 2024-04-24T08:18:39.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.946+0000 7febcbe00640 1 -- 172.21.15.69:0/827758404 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc0032810 con 0x7febcc1523f0 2024-04-24T08:18:39.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.946+0000 7febd2e00640 1 -- 172.21.15.69:0/827758404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 msgr2=0x7febcc06db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.946+0000 7febd2e00640 1 --2- 172.21.15.69:0/827758404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 0x7febcc06db90 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7febc0009870 tx=0x7febc002a180 comp rx=0 tx=0).stop 2024-04-24T08:18:39.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.947+0000 7febd2e00640 1 -- 172.21.15.69:0/827758404 shutdown_connections 2024-04-24T08:18:39.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.947+0000 7febd2e00640 1 --2- 172.21.15.69:0/827758404 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc072e20 0x7febcc073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.947+0000 7febd2e00640 1 --2- 172.21.15.69:0/827758404 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 0x7febcc06db90 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.947+0000 7febd2e00640 1 -- 172.21.15.69:0/827758404 >> 172.21.15.69:0/827758404 conn(0x7febcc0d0160 msgr2=0x7febcc0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:39.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.947+0000 7febd2e00640 1 -- 172.21.15.69:0/827758404 shutdown_connections 2024-04-24T08:18:39.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.947+0000 7febd2e00640 1 -- 172.21.15.69:0/827758404 wait complete. 2024-04-24T08:18:39.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.948+0000 7febd2e00640 1 Processor -- start 2024-04-24T08:18:39.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.949+0000 7febd2e00640 1 -- start start 2024-04-24T08:18:39.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.949+0000 7febd2e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febcc072e20 0x7febcc0ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.949+0000 7febd2e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc0737a0 0x7febcc072440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.949+0000 7febd2e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 0x7febcc14f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.949+0000 7febd2e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7febcc15f680 con 0x7febcc1523f0 2024-04-24T08:18:39.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.949+0000 7febd2e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7febcc1527d0 con 0x7febcc072e20 2024-04-24T08:18:39.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.949+0000 7febd2e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7febcc15f800 con 0x7febcc0737a0 2024-04-24T08:18:39.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd2400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 0x7febcc14f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd2400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 0x7febcc14f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37560/0 (socket says 172.21.15.69:37560) 2024-04-24T08:18:39.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd2400640 1 -- 172.21.15.69:0/2095067805 learned_addr learned my addr 172.21.15.69:0/2095067805 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd1a00640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febcc072e20 0x7febcc0ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd1000640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc0737a0 0x7febcc072440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd2400640 1 -- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc0737a0 msgr2=0x7febcc072440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd2400640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc0737a0 0x7febcc072440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd2400640 1 -- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febcc072e20 msgr2=0x7febcc0ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd2400640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febcc072e20 0x7febcc0ae060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd2400640 1 -- 172.21.15.69:0/2095067805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febcc1f4930 con 0x7febcc1523f0 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.950+0000 7febd2400640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 0x7febcc14f3b0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7febc400c780 tx=0x7febc400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.951+0000 7febc9600640 1 -- 172.21.15.69:0/2095067805 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc4004b60 con 0x7febcc1523f0 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.951+0000 7febc9600640 1 -- 172.21.15.69:0/2095067805 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febc4004d00 con 0x7febcc1523f0 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.951+0000 7febc9600640 1 -- 172.21.15.69:0/2095067805 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc400aa90 con 0x7febcc1523f0 2024-04-24T08:18:39.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.951+0000 7febd2e00640 1 -- 172.21.15.69:0/2095067805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febcc1f4c10 con 0x7febcc1523f0 2024-04-24T08:18:39.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.951+0000 7febd2e00640 1 -- 172.21.15.69:0/2095067805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febcc1f51d0 con 0x7febcc1523f0 2024-04-24T08:18:39.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.952+0000 7febd2e00640 1 -- 172.21.15.69:0/2095067805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febcc075d50 con 0x7febcc1523f0 2024-04-24T08:18:39.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.953+0000 7febc9600640 1 -- 172.21.15.69:0/2095067805 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febc400ac30 con 0x7febcc1523f0 2024-04-24T08:18:39.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.954+0000 7febc9600640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 0x7febac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:39.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.955+0000 7febc9600640 1 -- 172.21.15.69:0/2095067805 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7febc408bf90 con 0x7febcc1523f0 2024-04-24T08:18:39.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.955+0000 7febd1a00640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 0x7febac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:39.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.956+0000 7febd1a00640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 0x7febac06edb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7febc0034000 tx=0x7febc002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:39.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:39.959+0000 7febc9600640 1 -- 172.21.15.69:0/2095067805 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febc4020070 con 0x7febcc1523f0 2024-04-24T08:18:40.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.172+0000 7febd2e00640 1 -- 172.21.15.69:0/2095067805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7febcc073300 con 0x7febcc1523f0 2024-04-24T08:18:40.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.173+0000 7febc9600640 1 -- 172.21.15.69:0/2095067805 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v97) ==== 80+0+0 (secure 0 0 0) 0x7febc405a290 con 0x7febcc1523f0 2024-04-24T08:18:40.175 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:18:40.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.176+0000 7febd2e00640 1 -- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 msgr2=0x7febac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.176+0000 7febd2e00640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 0x7febac06edb0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7febc0034000 tx=0x7febc002e000 comp rx=0 tx=0).stop 2024-04-24T08:18:40.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.176+0000 7febd2e00640 1 -- 172.21.15.69:0/2095067805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 msgr2=0x7febcc14f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.176+0000 7febd2e00640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 0x7febcc14f3b0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7febc400c780 tx=0x7febc400cc40 comp rx=0 tx=0).stop 2024-04-24T08:18:40.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.177+0000 7febd2e00640 1 -- 172.21.15.69:0/2095067805 shutdown_connections 2024-04-24T08:18:40.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.177+0000 7febd2e00640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7febac06c8f0 0x7febac06edb0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.177+0000 7febd2e00640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febcc1523f0 0x7febcc14f3b0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.177+0000 7febd2e00640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7febcc0737a0 0x7febcc072440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.177+0000 7febd2e00640 1 --2- 172.21.15.69:0/2095067805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7febcc072e20 0x7febcc0ae060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.177+0000 7febd2e00640 1 -- 172.21.15.69:0/2095067805 >> 172.21.15.69:0/2095067805 conn(0x7febcc0d0160 msgr2=0x7febcc0b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:40.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.177+0000 7febd2e00640 1 -- 172.21.15.69:0/2095067805 shutdown_connections 2024-04-24T08:18:40.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.177+0000 7febd2e00640 1 -- 172.21.15.69:0/2095067805 wait complete. 2024-04-24T08:18:40.197 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:18:40.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:40.197 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:40.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[70951]: fuse finished with error 0 and tester_r 0 2024-04-24T08:18:40.282 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:18:40.376 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:18:40.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:40.376 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:18:40.414 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:40.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.628+0000 7f4a9d200640 1 Processor -- start 2024-04-24T08:18:40.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.628+0000 7f4a9d200640 1 -- start start 2024-04-24T08:18:40.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.630+0000 7f4a9d200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4a980d0300 0x7f4a9806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.630+0000 7f4a9d200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a980cf3d0 con 0x7f4a98145a90 2024-04-24T08:18:40.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.630+0000 7f4a9d200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a980cf6b0 con 0x7f4a98145fe0 2024-04-24T08:18:40.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.630+0000 7f4a9d200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4a980cf990 con 0x7f4a980d0300 2024-04-24T08:18:40.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.630+0000 7f4a95400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4a98145a90 0x7f4a980d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37020/0 (socket says 172.21.15.69:37020) 2024-04-24T08:18:40.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.630+0000 7f4a95400640 1 -- 172.21.15.69:0/3790218267 learned_addr learned my addr 172.21.15.69:0/3790218267 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:40.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.630+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1726200555 0 0) 0x7f4a980cf3d0 con 0x7f4a98145a90 2024-04-24T08:18:40.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.630+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a7c003520 con 0x7f4a98145a90 2024-04-24T08:18:40.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a95e00640 1 --2- 172.21.15.69:0/3790218267 >> v2:172.21.15.121:3300/0 conn(0x7f4a980d0300 0x7f4a9806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 99406993 0 0) 0x7f4a7c003520 con 0x7f4a98145a90 2024-04-24T08:18:40.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 >> v2:172.21.15.121:3300/0 conn(0x7f4a980d0300 msgr2=0x7f4a9806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:40.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 --2- 172.21.15.69:0/3790218267 >> v2:172.21.15.121:3300/0 conn(0x7f4a980d0300 0x7f4a9806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 >> v1:172.21.15.121:6789/0 conn(0x7f4a98145fe0 legacy=0x7f4a98160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a980cfc70 con 0x7f4a98145a90 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f4a84001670 con 0x7f4a98145a90 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 >> v1:172.21.15.69:6789/0 conn(0x7f4a98145a90 legacy=0x7f4a980d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a7c003a00 0x7f4a7c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a7c004800 0x7f4a7c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a7c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a980cf990 con 0x7f4a7c005210 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7c00c160 con 0x7f4a7c003a00 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7c00c360 con 0x7f4a7c004800 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.631+0000 7f4a96800640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a7c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.632+0000 7f4a95e00640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a7c003a00 0x7f4a7c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.632+0000 7f4a95400640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a7c004800 0x7f4a7c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.632+0000 7f4a96800640 1 -- 172.21.15.69:0/3790218267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a7c004800 msgr2=0x7f4a7c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:40.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.632+0000 7f4a96800640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a7c004800 0x7f4a7c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.632+0000 7f4a96800640 1 -- 172.21.15.69:0/3790218267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a7c003a00 msgr2=0x7f4a7c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:40.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.632+0000 7f4a96800640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a7c003a00 0x7f4a7c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.632+0000 7f4a96800640 1 -- 172.21.15.69:0/3790218267 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7c003520 con 0x7f4a7c005210 2024-04-24T08:18:40.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.632+0000 7f4a95400640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a7c004800 0x7f4a7c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:40.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.633+0000 7f4a96800640 1 -- 172.21.15.69:0/3790218267 reap_dead start 2024-04-24T08:18:40.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.633+0000 7f4a96800640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a7c00b9c0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f4a90001e50 tx=0x7f4a7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9166011f3a1190a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:40.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.633+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90001a70 con 0x7f4a7c005210 2024-04-24T08:18:40.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.633+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a90003970 con 0x7f4a7c005210 2024-04-24T08:18:40.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.633+0000 7f4a94a00640 1 -- 172.21.15.69:0/3790218267 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90003d60 con 0x7f4a7c005210 2024-04-24T08:18:40.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.634+0000 7f4a9d200640 1 -- 172.21.15.69:0/3790218267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 msgr2=0x7f4a7c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.634+0000 7f4a9d200640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a7c00b9c0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f4a90001e50 tx=0x7f4a7c005830 comp rx=0 tx=0).stop 2024-04-24T08:18:40.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.634+0000 7f4a9d200640 1 -- 172.21.15.69:0/3790218267 shutdown_connections 2024-04-24T08:18:40.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.634+0000 7f4a9d200640 1 --2- 172.21.15.69:0/3790218267 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a7c00b9c0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.634+0000 7f4a9d200640 1 -- 172.21.15.69:0/3790218267 >> 172.21.15.69:0/3790218267 conn(0x7f4a98067d90 msgr2=0x7f4a9806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:40.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.634+0000 7f4a9d200640 1 -- 172.21.15.69:0/3790218267 shutdown_connections 2024-04-24T08:18:40.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.634+0000 7f4a9d200640 1 -- 172.21.15.69:0/3790218267 wait complete. 2024-04-24T08:18:40.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.636+0000 7f4a9d200640 1 Processor -- start 2024-04-24T08:18:40.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.636+0000 7f4a9d200640 1 -- start start 2024-04-24T08:18:40.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.637+0000 7f4a9d200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a9815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.637+0000 7f4a9d200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a981f2c90 0x7f4a98136480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.637+0000 7f4a9d200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a981f30f0 0x7f4a981369c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.637+0000 7f4a9d200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a981458f0 con 0x7f4a7c005210 2024-04-24T08:18:40.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.637+0000 7f4a9d200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4a98145770 con 0x7f4a981f30f0 2024-04-24T08:18:40.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.637+0000 7f4a9d200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4a980cdc30 con 0x7f4a981f2c90 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a95e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a9815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a95e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a9815c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37590/0 (socket says 172.21.15.69:37590) 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a95e00640 1 -- 172.21.15.69:0/1104616048 learned_addr learned my addr 172.21.15.69:0/1104616048 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a96800640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a981f30f0 0x7f4a981369c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a95400640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a981f2c90 0x7f4a98136480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a95e00640 1 -- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a981f2c90 msgr2=0x7f4a98136480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a95e00640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a981f2c90 0x7f4a98136480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a95e00640 1 -- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a981f30f0 msgr2=0x7f4a981369c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a95e00640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a981f30f0 0x7f4a981369c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a95e00640 1 -- 172.21.15.69:0/1104616048 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a981f2220 con 0x7f4a7c005210 2024-04-24T08:18:40.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.638+0000 7f4a95e00640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a9815c2a0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800f580 tx=0x7f4a8800fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:40.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.639+0000 7f4a8e000640 1 -- 172.21.15.69:0/1104616048 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a88013810 con 0x7f4a7c005210 2024-04-24T08:18:40.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.639+0000 7f4a8e000640 1 -- 172.21.15.69:0/1104616048 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a8801a070 con 0x7f4a7c005210 2024-04-24T08:18:40.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.639+0000 7f4a8e000640 1 -- 172.21.15.69:0/1104616048 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a88013bb0 con 0x7f4a7c005210 2024-04-24T08:18:40.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.639+0000 7f4a9d200640 1 -- 172.21.15.69:0/1104616048 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a98137020 con 0x7f4a7c005210 2024-04-24T08:18:40.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.639+0000 7f4a9d200640 1 -- 172.21.15.69:0/1104616048 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a98137300 con 0x7f4a7c005210 2024-04-24T08:18:40.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.640+0000 7f4a9d200640 1 -- 172.21.15.69:0/1104616048 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a981f2050 con 0x7f4a7c005210 2024-04-24T08:18:40.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.641+0000 7f4a8e000640 1 -- 172.21.15.69:0/1104616048 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a88012070 con 0x7f4a7c005210 2024-04-24T08:18:40.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.642+0000 7f4a8e000640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a7806c8f0 0x7f4a7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:40.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.642+0000 7f4a8e000640 1 -- 172.21.15.69:0/1104616048 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f4a8808bbf0 con 0x7f4a7c005210 2024-04-24T08:18:40.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.642+0000 7f4a95400640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a7806c8f0 0x7f4a7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:40.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.644+0000 7f4a95400640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a7806c8f0 0x7f4a7806edb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4a84001200 tx=0x7f4a84009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:40.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.646+0000 7f4a8e000640 1 -- 172.21.15.69:0/1104616048 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a8801c310 con 0x7f4a7c005210 2024-04-24T08:18:40.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.886+0000 7f4a9d200640 1 -- 172.21.15.69:0/1104616048 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a98068a80 con 0x7f4a7c005210 2024-04-24T08:18:40.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.887+0000 7f4a8e000640 1 -- 172.21.15.69:0/1104616048 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5186 (secure 0 0 0) 0x7f4a88059f90 con 0x7f4a7c005210 2024-04-24T08:18:40.889 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:40.889 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:18:13.481156+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6046},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6046":{"gid":6046,"name":"a","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.69:6835/412016900","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":412016900},{"type":"v1","addr":"172.21.15.69:6835","nonce":412016900}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6046,"qdb_cluster":[6046]},"id":7}]} 2024-04-24T08:18:40.889 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 83 2024-04-24T08:18:40.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 -- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a7806c8f0 msgr2=0x7f4a7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a7806c8f0 0x7f4a7806edb0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4a84001200 tx=0x7f4a84009040 comp rx=0 tx=0).stop 2024-04-24T08:18:40.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 -- 172.21.15.69:0/1104616048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 msgr2=0x7f4a9815c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:40.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a9815c2a0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800f580 tx=0x7f4a8800fa40 comp rx=0 tx=0).stop 2024-04-24T08:18:40.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 -- 172.21.15.69:0/1104616048 shutdown_connections 2024-04-24T08:18:40.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4a7806c8f0 0x7f4a7806edb0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4a981f30f0 0x7f4a981369c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4a981f2c90 0x7f4a98136480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 --2- 172.21.15.69:0/1104616048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a7c005210 0x7f4a9815c2a0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:40.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 -- 172.21.15.69:0/1104616048 >> 172.21.15.69:0/1104616048 conn(0x7f4a98067d90 msgr2=0x7f4a980afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:40.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 -- 172.21.15.69:0/1104616048 shutdown_connections 2024-04-24T08:18:40.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:40.890+0000 7f4a9d200640 1 -- 172.21.15.69:0/1104616048 wait complete. 2024-04-24T08:18:40.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-04-24T08:18:41.138 INFO:teuthology.orchestra.run.smithi069.stdout:{"message":"","return_code":0} 2024-04-24T08:18:41.139 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-24T08:18:41.139 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:18:41.139 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:18:41.189 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:18:41.189 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi069.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-04-24T08:18:41.189 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:18:41.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:41.190 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:18:41.239 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-24T08:18:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:18:41.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo daemon-helper kill ceph --cluster ceph -w 2024-04-24T08:18:41.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:41.246 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-04-24T08:18:41.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.428+0000 7f9711800640 1 Processor -- start 2024-04-24T08:18:41.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.428+0000 7f9711800640 1 -- start start 2024-04-24T08:18:41.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.429+0000 7f9711800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f970c0d02c0 0x7f970c0cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f9711800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f970c06c430 0x7f970c1604a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f9711800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f970c15cf50 con 0x7f970c0d0850 2024-04-24T08:18:41.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f9711800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f970c15db60 con 0x7f970c0d02c0 2024-04-24T08:18:41.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f9711800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f970c0cd5b0 con 0x7f970c06c430 2024-04-24T08:18:41.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f9702000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f970c0d02c0 0x7f970c0cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f970aa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f970c0d0850 0x7f970c0d0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37032/0 (socket says 172.21.15.69:37032) 2024-04-24T08:18:41.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f970aa00640 1 -- 172.21.15.69:0/2255957325 learned_addr learned my addr 172.21.15.69:0/2255957325 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:41.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f9702000640 1 -- 172.21.15.69:0/2255957325 >> v2:172.21.15.121:3300/0 conn(0x7f970c06c430 msgr2=0x7f970c1604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f970b400640 1 --2- 172.21.15.69:0/2255957325 >> v2:172.21.15.121:3300/0 conn(0x7f970c06c430 0x7f970c1604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f9702000640 1 --2- 172.21.15.69:0/2255957325 >> v2:172.21.15.121:3300/0 conn(0x7f970c06c430 0x7f970c1604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f9702000640 1 -- 172.21.15.69:0/2255957325 >> v1:172.21.15.69:6789/0 conn(0x7f970c0d0850 legacy=0x7f970c0d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f9702000640 1 -- 172.21.15.69:0/2255957325 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f970c15d230 con 0x7f970c0d02c0 2024-04-24T08:18:41.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.430+0000 7f970a000640 1 -- 172.21.15.69:0/2255957325 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1707675345 0 0) 0x7f970c15cf50 con 0x7f970c0d0850 2024-04-24T08:18:41.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.431+0000 7f970b400640 1 --2- 172.21.15.69:0/2255957325 >> v2:172.21.15.121:3300/0 conn(0x7f970c06c430 0x7f970c1604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:41.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.432+0000 7f9702000640 1 --2- 172.21.15.69:0/2255957325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d02c0 0x7f970c0cd070 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f970c0cce40 tx=0x7f96f402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e638cf5d9e953fe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.432+0000 7f970a000640 1 -- 172.21.15.69:0/2255957325 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4037070 con 0x7f970c0d02c0 2024-04-24T08:18:41.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.432+0000 7f970a000640 1 -- 172.21.15.69:0/2255957325 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f4032440 con 0x7f970c0d02c0 2024-04-24T08:18:41.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.432+0000 7f970a000640 1 -- 172.21.15.69:0/2255957325 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4032810 con 0x7f970c0d02c0 2024-04-24T08:18:41.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.433+0000 7f9711800640 1 -- 172.21.15.69:0/2255957325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d02c0 msgr2=0x7f970c0cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.433+0000 7f9711800640 1 --2- 172.21.15.69:0/2255957325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d02c0 0x7f970c0cd070 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f970c0cce40 tx=0x7f96f402a1a0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.433+0000 7f9711800640 1 -- 172.21.15.69:0/2255957325 shutdown_connections 2024-04-24T08:18:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.433+0000 7f9711800640 1 --2- 172.21.15.69:0/2255957325 >> v2:172.21.15.121:3300/0 conn(0x7f970c06c430 0x7f970c1604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.433+0000 7f9711800640 1 --2- 172.21.15.69:0/2255957325 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d02c0 0x7f970c0cd070 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.433+0000 7f9711800640 1 -- 172.21.15.69:0/2255957325 >> 172.21.15.69:0/2255957325 conn(0x7f970c0b2f70 msgr2=0x7f970c0b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.433+0000 7f9711800640 1 -- 172.21.15.69:0/2255957325 shutdown_connections 2024-04-24T08:18:41.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.433+0000 7f9711800640 1 -- 172.21.15.69:0/2255957325 wait complete. 2024-04-24T08:18:41.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.435+0000 7f9711800640 1 Processor -- start 2024-04-24T08:18:41.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.435+0000 7f9711800640 1 -- start start 2024-04-24T08:18:41.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.436+0000 7f9711800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f970c06c430 0x7f970c150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.436+0000 7f9711800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f970c0d02c0 0x7f970c0b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.436+0000 7f9711800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d0850 0x7f970c0c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.436+0000 7f9711800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f970c06c8e0 con 0x7f970c0d0850 2024-04-24T08:18:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.436+0000 7f9711800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f970c15d7f0 con 0x7f970c0d02c0 2024-04-24T08:18:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.436+0000 7f9711800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f970c15cb20 con 0x7f970c06c430 2024-04-24T08:18:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.436+0000 7f970b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d0850 0x7f970c0c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.436+0000 7f970b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d0850 0x7f970c0c5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37604/0 (socket says 172.21.15.69:37604) 2024-04-24T08:18:41.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.436+0000 7f970b400640 1 -- 172.21.15.69:0/1796146259 learned_addr learned my addr 172.21.15.69:0/1796146259 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.436+0000 7f970aa00640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f970c06c430 0x7f970c150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f970b400640 1 -- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f970c06c430 msgr2=0x7f970c150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f9702000640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f970c0d02c0 0x7f970c0b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f970b400640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f970c06c430 0x7f970c150860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f970b400640 1 -- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f970c0d02c0 msgr2=0x7f970c0b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f970b400640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f970c0d02c0 0x7f970c0b5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f970b400640 1 -- 172.21.15.69:0/1796146259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f970c0b62b0 con 0x7f970c0d0850 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f970b400640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d0850 0x7f970c0c5b20 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f970400c7f0 tx=0x7f970400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f9703400640 1 -- 172.21.15.69:0/1796146259 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704015070 con 0x7f970c0d0850 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f9703400640 1 -- 172.21.15.69:0/1796146259 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9704007820 con 0x7f970c0d0850 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f9703400640 1 -- 172.21.15.69:0/1796146259 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704007c50 con 0x7f970c0d0850 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f9711800640 1 -- 172.21.15.69:0/1796146259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f970c0b6590 con 0x7f970c0d0850 2024-04-24T08:18:41.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.437+0000 7f9711800640 1 -- 172.21.15.69:0/1796146259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f970c150da0 con 0x7f970c0d0850 2024-04-24T08:18:41.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.440+0000 7f9703400640 1 -- 172.21.15.69:0/1796146259 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9704018070 con 0x7f970c0d0850 2024-04-24T08:18:41.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.441+0000 7f9703400640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f96e806c8f0 0x7f96e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.441+0000 7f9703400640 1 -- 172.21.15.69:0/1796146259 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f970408bd60 con 0x7f970c0d0850 2024-04-24T08:18:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.442+0000 7f9714322740 1 Processor -- start 2024-04-24T08:18:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.442+0000 7f970aa00640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f96e806c8f0 0x7f96e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.442+0000 7f9714322740 1 -- start start 2024-04-24T08:18:41.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.443+0000 7f9714322740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed17e460 0x55dbed130df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.443+0000 7f9714322740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed160900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.443+0000 7f9714322740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed160e40 0x55dbed135c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.443+0000 7f9714322740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dbed09ae70 con 0x55dbed161580 2024-04-24T08:18:41.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.443+0000 7f9714322740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55dbed041380 con 0x55dbed160e40 2024-04-24T08:18:41.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.443+0000 7f9714322740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55dbed096e30 con 0x55dbed17e460 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.443+0000 7f9702000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.443+0000 7f970aa00640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f96e806c8f0 0x7f96e806edb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0012e0 tx=0x7f96fc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.443+0000 7f9702000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed160900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37616/0 (socket says 172.21.15.69:37616) 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.443+0000 7f9702000640 1 -- 172.21.15.69:0/3859548920 learned_addr learned my addr 172.21.15.69:0/3859548920 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.444+0000 7f970aa00640 1 --2- 172.21.15.69:0/3859548920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed17e460 0x55dbed130df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.444+0000 7f970b400640 1 --2- 172.21.15.69:0/3859548920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed160e40 0x55dbed135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.444+0000 7f9702000640 1 -- 172.21.15.69:0/3859548920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed17e460 msgr2=0x55dbed130df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.444+0000 7f9702000640 1 --2- 172.21.15.69:0/3859548920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed17e460 0x55dbed130df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.444+0000 7f9702000640 1 -- 172.21.15.69:0/3859548920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed160e40 msgr2=0x55dbed135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.444+0000 7f9702000640 1 --2- 172.21.15.69:0/3859548920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed160e40 0x55dbed135c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.444+0000 7f9702000640 1 -- 172.21.15.69:0/3859548920 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbece757e0 con 0x55dbed161580 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.444+0000 7f9702000640 1 --2- 172.21.15.69:0/3859548920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed160900 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f96f4009860 tx=0x7f96f402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc2b9bbd3585a5b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.445+0000 7f96fb400640 1 -- 172.21.15.69:0/3859548920 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4037040 con 0x55dbed161580 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.445+0000 7f96fb400640 1 -- 172.21.15.69:0/3859548920 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f403d640 con 0x55dbed161580 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.445+0000 7f96fb400640 1 -- 172.21.15.69:0/3859548920 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f403d9f0 con 0x55dbed161580 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.445+0000 7f9714322740 1 -- 172.21.15.69:0/3859548920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 msgr2=0x55dbed160900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.445+0000 7f9714322740 1 --2- 172.21.15.69:0/3859548920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed160900 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f96f4009860 tx=0x7f96f402c000 comp rx=0 tx=0).stop 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.446+0000 7f9714322740 1 -- 172.21.15.69:0/3859548920 shutdown_connections 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.446+0000 7f9714322740 1 --2- 172.21.15.69:0/3859548920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed160900 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.446+0000 7f9714322740 1 --2- 172.21.15.69:0/3859548920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed160e40 0x55dbed135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.446+0000 7f9714322740 1 --2- 172.21.15.69:0/3859548920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed17e460 0x55dbed130df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.446+0000 7f9714322740 1 -- 172.21.15.69:0/3859548920 >> 172.21.15.69:0/3859548920 conn(0x55dbed133060 msgr2=0x55dbed17b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.446+0000 7f9714322740 1 -- 172.21.15.69:0/3859548920 shutdown_connections 2024-04-24T08:18:41.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.446+0000 7f9714322740 1 -- 172.21.15.69:0/3859548920 wait complete. 2024-04-24T08:18:41.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.446+0000 7f9714322740 1 -- start start 2024-04-24T08:18:41.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.447+0000 7f9714322740 1 Processor -- start 2024-04-24T08:18:41.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f9714322740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed151240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f9714322740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed17e460 0x55dbed163ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f9714322740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17ed20 0x55dbed17f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f9714322740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dbed135210 con 0x55dbed161580 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f9714322740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55dbed054660 con 0x55dbed17ed20 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f9714322740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55dbed134600 con 0x55dbed17e460 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f9714322740 10 client.0 _ll_register_callbacks cb 0x55dbed042aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f970aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f970aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed151240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37622/0 (socket says 172.21.15.69:37622) 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f970aa00640 1 -- 172.21.15.69:0/1959437051 learned_addr learned my addr 172.21.15.69:0/1959437051 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f970b400640 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17ed20 0x55dbed17f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f9702000640 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed17e460 0x55dbed163ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f970aa00640 1 -- 172.21.15.69:0/1959437051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed17e460 msgr2=0x55dbed163ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f970aa00640 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed17e460 0x55dbed163ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f970aa00640 1 -- 172.21.15.69:0/1959437051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17ed20 msgr2=0x55dbed17f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f970aa00640 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17ed20 0x55dbed17f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.448+0000 7f970aa00640 1 -- 172.21.15.69:0/1959437051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbeced5f70 con 0x55dbed161580 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.449+0000 7f970aa00640 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed151240 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0073e0 tx=0x7f96fc001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3af003fbdbbc61 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.449+0000 7f96efe00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.449+0000 7f96efe00640 1 -- 172.21.15.69:0/1959437051 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96fc026060 con 0x55dbed161580 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.449+0000 7f96efe00640 1 -- 172.21.15.69:0/1959437051 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f96fc005cd0 con 0x55dbed161580 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.449+0000 7f9714322740 1 -- 172.21.15.69:0/1959437051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96f403d640 con 0x55dbed161580 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.449+0000 7f96efe00640 1 -- 172.21.15.69:0/1959437051 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f96fc0078a0 con 0x55dbed161580 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.449+0000 7f9714322740 10 client.8895 fetch_fsmap learned FSMap version 83 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.449+0000 7f9714322740 1 -- 172.21.15.69:0/1959437051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x55dbecd6d170 con 0x55dbed161580 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.450+0000 7f96efe00640 1 -- 172.21.15.69:0/1959437051 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96f403d640 con 0x55dbed161580 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.450+0000 7f96efe00640 1 -- 172.21.15.69:0/1959437051 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7f96fc004b60 con 0x55dbed161580 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.450+0000 7f9714322740 10 client.8895 fetch_fsmap finished waiting for FSMap version 83 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.450+0000 7f9714322740 10 client.8895 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:18:41.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.450+0000 7f9714322740 20 client.8895 populate_metadata read hostname 'smithi069' 2024-04-24T08:18:41.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.450+0000 7f9714322740 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55dbed18bb20 0x55dbed18bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.450+0000 7f9714322740 4 client.8895 mds_command: new command op to 6046 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:41.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.450+0000 7f9714322740 1 -- 172.21.15.69:0/1959437051 --> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbed0a8890 con 0x55dbed18bb20 2024-04-24T08:18:41.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.450+0000 7f9702000640 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55dbed18bb20 0x55dbed18bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.451+0000 7f9702000640 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55dbed18bb20 0x55dbed18bf50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cd2a707205d7de3 server_cookie=94012a5f61078177 in_seq=0 out_seq=0 2024-04-24T08:18:41.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.451+0000 7f96efe00640 10 client.8895 ms_handle_connect on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:41.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.451+0000 7f96efe00640 1 -- 172.21.15.69:0/1959437051 <== mds.0 v2:172.21.15.69:6833/412016900 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbed0a8890 con 0x55dbed18bb20 2024-04-24T08:18:41.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.451+0000 7f96efe00640 10 client.8895 handle_command_reply: tid=0 2024-04-24T08:18:41.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.451+0000 7f9714322740 1 client.8895 shutdown 2024-04-24T08:18:41.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 1 -- 172.21.15.69:0/1959437051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 msgr2=0x55dbed151240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed151240 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0073e0 tx=0x7f96fc001250 comp rx=0 tx=0).stop 2024-04-24T08:18:41.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 1 -- 172.21.15.69:0/1959437051 shutdown_connections 2024-04-24T08:18:41.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55dbed18bb20 0x55dbed18bf50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17ed20 0x55dbed17f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed17e460 0x55dbed163ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 1 --2- 172.21.15.69:0/1959437051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed161580 0x55dbed151240 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f96efe00640 0 client.8895 ms_handle_reset on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:41.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 1 -- 172.21.15.69:0/1959437051 >> 172.21.15.69:0/1959437051 conn(0x55dbed160e40 msgr2=0x55dbed17ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 1 -- 172.21.15.69:0/1959437051 shutdown_connections 2024-04-24T08:18:41.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 1 -- 172.21.15.69:0/1959437051 wait complete. 2024-04-24T08:18:41.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.452+0000 7f9714322740 20 client.8895 trim_cache size 0 max 16384 2024-04-24T08:18:41.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.455+0000 7fd8fd800640 1 Processor -- start 2024-04-24T08:18:41.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.455+0000 7fd8fd800640 1 -- start start 2024-04-24T08:18:41.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.456+0000 7fd8fd800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd8f815eaf0 0x7fd8f80d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.456+0000 7fd8fd800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8f80d10c0 0x7fd8f80cf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.456+0000 7fd8fd800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd8f80d2e50 0x7fd8f80d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.456+0000 7fd8fd800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd8f80d3800 con 0x7fd8f80d2e50 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.456+0000 7fd8fd800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd8f80d2c20 con 0x7fd8f815eaf0 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.456+0000 7fd8fd800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f80cf950 con 0x7fd8f80d10c0 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.456+0000 7fd8f7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd8f80d2e50 0x7fd8f80d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.456+0000 7fd8f7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd8f80d2e50 0x7fd8f80d32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37626/0 (socket says 172.21.15.69:37626) 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.456+0000 7fd8f7400640 1 -- 172.21.15.69:0/2687000957 learned_addr learned my addr 172.21.15.69:0/2687000957 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.457+0000 7fd8f6a00640 1 --2- 172.21.15.69:0/2687000957 >> v2:172.21.15.121:3300/0 conn(0x7fd8f815eaf0 0x7fd8f80d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.457+0000 7fd8f6000640 1 --2- 172.21.15.69:0/2687000957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8f80d10c0 0x7fd8f80cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.457+0000 7fd8f7400640 1 -- 172.21.15.69:0/2687000957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8f80d10c0 msgr2=0x7fd8f80cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.457+0000 7fd8f7400640 1 --2- 172.21.15.69:0/2687000957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8f80d10c0 0x7fd8f80cf410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.457+0000 7fd8f7400640 1 -- 172.21.15.69:0/2687000957 >> v2:172.21.15.121:3300/0 conn(0x7fd8f815eaf0 msgr2=0x7fd8f80d26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.457+0000 7fd8f7400640 1 --2- 172.21.15.69:0/2687000957 >> v2:172.21.15.121:3300/0 conn(0x7fd8f815eaf0 0x7fd8f80d26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.457+0000 7fd8f7400640 1 -- 172.21.15.69:0/2687000957 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f80d1680 con 0x7fd8f80d2e50 2024-04-24T08:18:41.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.458+0000 7fd8f7400640 1 --2- 172.21.15.69:0/2687000957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f80d2e50 0x7fd8f80d32c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fd8f808c900 tx=0x7fd8f002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b72c2f668f245a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.459+0000 7fd8f5600640 1 -- 172.21.15.69:0/2687000957 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0036070 con 0x7fd8f80d2e50 2024-04-24T08:18:41.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.459+0000 7fd8f5600640 1 -- 172.21.15.69:0/2687000957 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8f0031600 con 0x7fd8f80d2e50 2024-04-24T08:18:41.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.459+0000 7fd8f5600640 1 -- 172.21.15.69:0/2687000957 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f00319d0 con 0x7fd8f80d2e50 2024-04-24T08:18:41.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.459+0000 7fd8fd800640 1 -- 172.21.15.69:0/2687000957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f80d2e50 msgr2=0x7fd8f80d32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.459+0000 7fd8fd800640 1 --2- 172.21.15.69:0/2687000957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f80d2e50 0x7fd8f80d32c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fd8f808c900 tx=0x7fd8f002a6a0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.460+0000 7fd8fd800640 1 -- 172.21.15.69:0/2687000957 shutdown_connections 2024-04-24T08:18:41.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.460+0000 7fd8fd800640 1 --2- 172.21.15.69:0/2687000957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f80d2e50 0x7fd8f80d32c0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.460+0000 7fd8fd800640 1 --2- 172.21.15.69:0/2687000957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8f80d10c0 0x7fd8f80cf410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.460+0000 7fd8fd800640 1 --2- 172.21.15.69:0/2687000957 >> v2:172.21.15.121:3300/0 conn(0x7fd8f815eaf0 0x7fd8f80d26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.460+0000 7fd8fd800640 1 -- 172.21.15.69:0/2687000957 >> 172.21.15.69:0/2687000957 conn(0x7fd8f809dad0 msgr2=0x7fd8f808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.460+0000 7fd8fd800640 1 -- 172.21.15.69:0/2687000957 shutdown_connections 2024-04-24T08:18:41.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.460+0000 7fd8fd800640 1 -- 172.21.15.69:0/2687000957 wait complete. 2024-04-24T08:18:41.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.461+0000 7fd8fd800640 1 Processor -- start 2024-04-24T08:18:41.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.462+0000 7fd8fd800640 1 -- start start 2024-04-24T08:18:41.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.462+0000 7fd8fd800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f80d10c0 0x7fd8f81f1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.462+0000 7fd8fd800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8f80d2e50 0x7fd8f8161460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.462+0000 7fd8fd800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8f815eaf0 0x7fd8f80b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.462+0000 7fd8fd800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f80d1c40 con 0x7fd8f80d10c0 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.462+0000 7fd8fd800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f80d1960 con 0x7fd8f815eaf0 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.462+0000 7fd8fd800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd8f80d1f20 con 0x7fd8f80d2e50 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f80d10c0 0x7fd8f81f1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f80d10c0 0x7fd8f81f1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37636/0 (socket says 172.21.15.69:37636) 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f6a00640 1 -- 172.21.15.69:0/3833438189 learned_addr learned my addr 172.21.15.69:0/3833438189 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f6000640 1 --2- 172.21.15.69:0/3833438189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8f80d2e50 0x7fd8f8161460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f7400640 1 --2- 172.21.15.69:0/3833438189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8f815eaf0 0x7fd8f80b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f6a00640 1 -- 172.21.15.69:0/3833438189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8f80d2e50 msgr2=0x7fd8f8161460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f6a00640 1 --2- 172.21.15.69:0/3833438189 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8f80d2e50 0x7fd8f8161460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f6a00640 1 -- 172.21.15.69:0/3833438189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8f815eaf0 msgr2=0x7fd8f80b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f6a00640 1 --2- 172.21.15.69:0/3833438189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8f815eaf0 0x7fd8f80b78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f6a00640 1 -- 172.21.15.69:0/3833438189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f8160710 con 0x7fd8f80d10c0 2024-04-24T08:18:41.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.463+0000 7fd8f6a00640 1 --2- 172.21.15.69:0/3833438189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8f80d10c0 0x7fd8f81f1a50 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd8e800d800 tx=0x7fd8e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.464+0000 7fd8eea00640 1 -- 172.21.15.69:0/3833438189 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e800b6c0 con 0x7fd8f80d10c0 2024-04-24T08:18:41.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.464+0000 7fd8eea00640 1 -- 172.21.15.69:0/3833438189 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e8019070 con 0x7fd8f80d10c0 2024-04-24T08:18:41.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.464+0000 7fd8eea00640 1 -- 172.21.15.69:0/3833438189 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e800ba80 con 0x7fd8f80d10c0 2024-04-24T08:18:41.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.464+0000 7fd8fd800640 1 -- 172.21.15.69:0/3833438189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f815fd50 con 0x7fd8f80d10c0 2024-04-24T08:18:41.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.464+0000 7fd8fd800640 1 -- 172.21.15.69:0/3833438189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f8160c60 con 0x7fd8f80d10c0 2024-04-24T08:18:41.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.465+0000 7fd8fd800640 1 -- 172.21.15.69:0/3833438189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fd8f8068a80 con 0x7fd8f80d10c0 2024-04-24T08:18:41.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.466+0000 7fd8eea00640 1 -- 172.21.15.69:0/3833438189 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd8e800bc20 con 0x7fd8f80d10c0 2024-04-24T08:18:41.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.467+0000 7fd8eea00640 1 --2- 172.21.15.69:0/3833438189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd8d806c8f0 0x7fd8d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.467+0000 7fd8eea00640 1 -- 172.21.15.69:0/3833438189 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fd8e808ca80 con 0x7fd8f80d10c0 2024-04-24T08:18:41.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.467+0000 7fd8eea00640 1 -- 172.21.15.69:0/3833438189 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+515 (secure 0 0 0) 0x7fd8e808d2c0 con 0x7fd8f80d10c0 2024-04-24T08:18:41.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.468+0000 7fd8fd800640 1 -- 172.21.15.69:0/3833438189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7fd8f80d5780 con 0x7fd8f80d10c0 2024-04-24T08:18:41.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.468+0000 7fd8f6000640 1 --2- 172.21.15.69:0/3833438189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd8d806c8f0 0x7fd8d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.469+0000 7fd8f6000640 1 --2- 172.21.15.69:0/3833438189 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd8d806c8f0 0x7fd8d806edb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd8e4006fd0 tx=0x7fd8e4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.474+0000 7f9714322740 1 Processor -- start 2024-04-24T08:18:41.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.474+0000 7f9714322740 1 -- start start 2024-04-24T08:18:41.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.475+0000 7f9714322740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed161580 0x55dbed17afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.475+0000 7f9714322740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17e460 0x55dbed194b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.475+0000 7f9714322740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed1722a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.475+0000 7f9714322740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f970c200ba0 con 0x55dbed17ed20 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.475+0000 7f9714322740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f970c1ffc70 con 0x55dbed17e460 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.475+0000 7f9714322740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f970c1f86e0 con 0x55dbed161580 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f970b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed1722a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f970b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed1722a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37648/0 (socket says 172.21.15.69:37648) 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f970b400640 1 -- 172.21.15.69:0/2430118519 learned_addr learned my addr 172.21.15.69:0/2430118519 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f970b400640 1 -- 172.21.15.69:0/2430118519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed161580 msgr2=0x55dbed17afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f970b400640 1 --2- 172.21.15.69:0/2430118519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed161580 0x55dbed17afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f970b400640 1 -- 172.21.15.69:0/2430118519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17e460 msgr2=0x55dbed194b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f970b400640 1 --2- 172.21.15.69:0/2430118519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17e460 0x55dbed194b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f970b400640 1 -- 172.21.15.69:0/2430118519 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbed1630f0 con 0x55dbed17ed20 2024-04-24T08:18:41.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f970b400640 1 --2- 172.21.15.69:0/2430118519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed1722a0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f9704059980 tx=0x7f970405dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0d0786e6a2064a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f96faa00640 1 -- 172.21.15.69:0/2430118519 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970401f070 con 0x55dbed17ed20 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f96faa00640 1 -- 172.21.15.69:0/2430118519 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9704056060 con 0x55dbed17ed20 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.476+0000 7f96faa00640 1 -- 172.21.15.69:0/2430118519 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9704062360 con 0x55dbed17ed20 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.477+0000 7f9714322740 1 -- 172.21.15.69:0/2430118519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 msgr2=0x55dbed1722a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.477+0000 7f9714322740 1 --2- 172.21.15.69:0/2430118519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed1722a0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f9704059980 tx=0x7f970405dda0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.477+0000 7f9714322740 1 -- 172.21.15.69:0/2430118519 shutdown_connections 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.477+0000 7f9714322740 1 --2- 172.21.15.69:0/2430118519 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed1722a0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.477+0000 7f9714322740 1 --2- 172.21.15.69:0/2430118519 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17e460 0x55dbed194b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.477+0000 7f9714322740 1 --2- 172.21.15.69:0/2430118519 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed161580 0x55dbed17afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.477+0000 7f9714322740 1 -- 172.21.15.69:0/2430118519 >> 172.21.15.69:0/2430118519 conn(0x55dbed160e40 msgr2=0x55dbed164690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.477+0000 7f9714322740 1 -- 172.21.15.69:0/2430118519 shutdown_connections 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.477+0000 7f9714322740 1 -- 172.21.15.69:0/2430118519 wait complete. 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.478+0000 7f9714322740 1 -- start start 2024-04-24T08:18:41.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.478+0000 7f9714322740 1 Processor -- start 2024-04-24T08:18:41.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.479+0000 7f9714322740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed161580 0x55dbed130730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.479+0000 7f9714322740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17e460 0x55dbed15b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.479+0000 7f9714322740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.479+0000 7f9714322740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f970c0709e0 con 0x55dbed17ed20 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.479+0000 7f9714322740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55dbed195530 con 0x55dbed17e460 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.479+0000 7f9714322740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55dbed131210 con 0x55dbed161580 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.479+0000 7f9714322740 10 client.0 _ll_register_callbacks cb 0x7f96fc005bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.479+0000 7f970b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.479+0000 7f970b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed15bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37658/0 (socket says 172.21.15.69:37658) 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.479+0000 7f970b400640 1 -- 172.21.15.69:0/449797828 learned_addr learned my addr 172.21.15.69:0/449797828 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f970aa00640 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed161580 0x55dbed130730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f970b400640 1 -- 172.21.15.69:0/449797828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed161580 msgr2=0x55dbed130730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f970b400640 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed161580 0x55dbed130730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f9702000640 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17e460 0x55dbed15b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f970b400640 1 -- 172.21.15.69:0/449797828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17e460 msgr2=0x55dbed15b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f970b400640 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17e460 0x55dbed15b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f970b400640 1 -- 172.21.15.69:0/449797828 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbed1f6580 con 0x55dbed17ed20 2024-04-24T08:18:41.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f970b400640 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed15bda0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f970400c7c0 tx=0x7f970408a0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19608783bb731bf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f96efe00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f96efe00640 1 -- 172.21.15.69:0/449797828 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970408a3c0 con 0x55dbed17ed20 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f96efe00640 1 -- 172.21.15.69:0/449797828 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f9704083d60 con 0x55dbed17ed20 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.480+0000 7f9714322740 1 -- 172.21.15.69:0/449797828 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9704056060 con 0x55dbed17ed20 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f96efe00640 1 -- 172.21.15.69:0/449797828 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f9704058e40 con 0x55dbed17ed20 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f9714322740 10 client.8907 fetch_fsmap learned FSMap version 83 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f9714322740 1 -- 172.21.15.69:0/449797828 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x55dbed1dc540 con 0x55dbed17ed20 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f96efe00640 1 -- 172.21.15.69:0/449797828 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9704056060 con 0x55dbed17ed20 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f96efe00640 1 -- 172.21.15.69:0/449797828 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7f9704058fe0 con 0x55dbed17ed20 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f9714322740 10 client.8907 fetch_fsmap finished waiting for FSMap version 83 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f9714322740 10 client.8907 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f9714322740 20 client.8907 populate_metadata read hostname 'smithi069' 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f9714322740 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55dbed161ac0 0x55dbed161ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f9714322740 4 client.8907 mds_command: new command op to 6046 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-04-24T08:18:41.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f9714322740 1 -- 172.21.15.69:0/449797828 --> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) -- 0x55dbed0a2700 con 0x55dbed161ac0 2024-04-24T08:18:41.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.481+0000 7f970aa00640 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55dbed161ac0 0x55dbed161ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.482+0000 7f970aa00640 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55dbed161ac0 0x55dbed161ed0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c36a0cccc137b18c server_cookie=1a11b33c17d62162 in_seq=0 out_seq=0 2024-04-24T08:18:41.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.482+0000 7f96efe00640 10 client.8907 ms_handle_connect on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:41.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f96efe00640 1 -- 172.21.15.69:0/449797828 <== mds.0 v2:172.21.15.69:6833/412016900 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dbed0a2700 con 0x55dbed161ac0 2024-04-24T08:18:41.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f96efe00640 10 client.8907 handle_command_reply: tid=0 2024-04-24T08:18:41.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 client.8907 shutdown 2024-04-24T08:18:41.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 -- 172.21.15.69:0/449797828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 msgr2=0x55dbed15bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed15bda0 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f970400c7c0 tx=0x7f970408a0c0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 -- 172.21.15.69:0/449797828 shutdown_connections 2024-04-24T08:18:41.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbed17ed20 0x55dbed15bda0 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55dbed161ac0 0x55dbed161ed0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55dbed17e460 0x55dbed15b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 --2- 172.21.15.69:0/449797828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55dbed161580 0x55dbed130730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 -- 172.21.15.69:0/449797828 >> 172.21.15.69:0/449797828 conn(0x55dbed160e40 msgr2=0x55dbed17b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f96efe00640 0 client.8907 ms_handle_reset on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 -- 172.21.15.69:0/449797828 shutdown_connections 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 1 -- 172.21.15.69:0/449797828 wait complete. 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.483+0000 7f9714322740 20 client.8907 trim_cache size 0 max 16384 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "9f1d1be3-f321-4275-b629-caf9c8c5fcf6", 2024-04-24T08:18:41.488 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-04-24T08:18:41.489 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:18:41.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 -- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f96e806c8f0 msgr2=0x7f96e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f96e806c8f0 0x7f96e806edb0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f96fc0012e0 tx=0x7f96fc009040 comp rx=0 tx=0).stop 2024-04-24T08:18:41.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 -- 172.21.15.69:0/1796146259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d0850 msgr2=0x7f970c0c5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d0850 0x7f970c0c5b20 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f970400c7f0 tx=0x7f970400ccb0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 -- 172.21.15.69:0/1796146259 shutdown_connections 2024-04-24T08:18:41.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f96e806c8f0 0x7f96e806edb0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f970c0d0850 0x7f970c0c5b20 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f970c0d02c0 0x7f970c0b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 --2- 172.21.15.69:0/1796146259 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f970c06c430 0x7f970c150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 -- 172.21.15.69:0/1796146259 >> 172.21.15.69:0/1796146259 conn(0x7f970c0b2f70 msgr2=0x7f970c0b0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 -- 172.21.15.69:0/1796146259 shutdown_connections 2024-04-24T08:18:41.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.484+0000 7f9711800640 1 -- 172.21.15.69:0/1796146259 wait complete. 2024-04-24T08:18:41.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:41.502 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-24T08:18:41.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.679+0000 7fbbfbe00640 1 Processor -- start 2024-04-24T08:18:41.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.680+0000 7fbbfbe00640 1 -- start start 2024-04-24T08:18:41.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.681+0000 7fbbfbe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbbf40b3e90 0x7fbbf40b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.681+0000 7fbbfbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbf415cc00 con 0x7fbbf415c820 2024-04-24T08:18:41.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.681+0000 7fbbfbe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbf406ed30 con 0x7fbbf40ce250 2024-04-24T08:18:41.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.681+0000 7fbbfbe00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fbbf40b4be0 con 0x7fbbf40b3e90 2024-04-24T08:18:41.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.681+0000 7fbbf9400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fbbf415c820 0x7fbbf406f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37048/0 (socket says 172.21.15.69:37048) 2024-04-24T08:18:41.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.681+0000 7fbbf9400640 1 -- 172.21.15.69:0/3515122881 learned_addr learned my addr 172.21.15.69:0/3515122881 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:18:41.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 423006054 0 0) 0x7fbbf415cc00 con 0x7fbbf415c820 2024-04-24T08:18:41.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbdc003520 con 0x7fbbf415c820 2024-04-24T08:18:41.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/3515122881 >> v2:172.21.15.121:3300/0 conn(0x7fbbf40b3e90 0x7fbbf40b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 166701002 0 0) 0x7fbbdc003520 con 0x7fbbf415c820 2024-04-24T08:18:41.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 >> v2:172.21.15.121:3300/0 conn(0x7fbbf40b3e90 msgr2=0x7fbbf40b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 --2- 172.21.15.69:0/3515122881 >> v2:172.21.15.121:3300/0 conn(0x7fbbf40b3e90 0x7fbbf40b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 >> v1:172.21.15.121:6789/0 conn(0x7fbbf40ce250 legacy=0x7fbbf40ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf40b4e10 con 0x7fbbf415c820 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fbbe8001670 con 0x7fbbf415c820 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 >> v1:172.21.15.69:6789/0 conn(0x7fbbf415c820 legacy=0x7fbbf406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 --2- 172.21.15.69:0/3515122881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbdc003a00 0x7fbbdc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 --2- 172.21.15.69:0/3515122881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbdc004800 0x7fbbdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 --2- 172.21.15.69:0/3515122881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbdc005230 0x7fbbdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbbdc00c160 con 0x7fbbdc003a00 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbbdc00c310 con 0x7fbbdc004800 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.682+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbbdc00c510 con 0x7fbbdc005230 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.683+0000 7fbbf9400640 1 --2- 172.21.15.69:0/3515122881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbdc003a00 0x7fbbdc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.683+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/3515122881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbdc004800 0x7fbbdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.683+0000 7fbbf9400640 1 -- 172.21.15.69:0/3515122881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbdc005230 msgr2=0x7fbbdc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.683+0000 7fbbf9400640 1 --2- 172.21.15.69:0/3515122881 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbdc005230 0x7fbbdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.683+0000 7fbbf9400640 1 -- 172.21.15.69:0/3515122881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbdc004800 msgr2=0x7fbbdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.683+0000 7fbbf9400640 1 --2- 172.21.15.69:0/3515122881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbdc004800 0x7fbbdc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.683+0000 7fbbf9400640 1 -- 172.21.15.69:0/3515122881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbdc003520 con 0x7fbbdc003a00 2024-04-24T08:18:41.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.683+0000 7fbbf9e00640 1 -- 172.21.15.69:0/3515122881 reap_dead start 2024-04-24T08:18:41.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.684+0000 7fbbf9400640 1 --2- 172.21.15.69:0/3515122881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbdc003a00 0x7fbbdc003e40 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fbbe800c110 tx=0x7fbbdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18839be9eed42a70 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.684+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8007c40 con 0x7fbbdc003a00 2024-04-24T08:18:41.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.684+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe8005af0 con 0x7fbbdc003a00 2024-04-24T08:18:41.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.684+0000 7fbbf3e00640 1 -- 172.21.15.69:0/3515122881 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8007070 con 0x7fbbdc003a00 2024-04-24T08:18:41.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.685+0000 7fbbfbe00640 1 -- 172.21.15.69:0/3515122881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbdc003a00 msgr2=0x7fbbdc003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.685+0000 7fbbfbe00640 1 --2- 172.21.15.69:0/3515122881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbdc003a00 0x7fbbdc003e40 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fbbe800c110 tx=0x7fbbdc005830 comp rx=0 tx=0).stop 2024-04-24T08:18:41.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.685+0000 7fbbfbe00640 1 -- 172.21.15.69:0/3515122881 shutdown_connections 2024-04-24T08:18:41.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.685+0000 7fbbfbe00640 1 --2- 172.21.15.69:0/3515122881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbdc003a00 0x7fbbdc003e40 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.685+0000 7fbbfbe00640 1 -- 172.21.15.69:0/3515122881 >> 172.21.15.69:0/3515122881 conn(0x7fbbf40d0350 msgr2=0x7fbbf40d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.685+0000 7fbbfbe00640 1 -- 172.21.15.69:0/3515122881 shutdown_connections 2024-04-24T08:18:41.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.685+0000 7fbbfbe00640 1 -- 172.21.15.69:0/3515122881 wait complete. 2024-04-24T08:18:41.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.687+0000 7fbbfbe00640 1 Processor -- start 2024-04-24T08:18:41.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.687+0000 7fbbfbe00640 1 -- start start 2024-04-24T08:18:41.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.688+0000 7fbbfbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbdc003a00 0x7fbbf415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.688+0000 7fbbfbe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbf41506f0 0x7fbbf4156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.688+0000 7fbbfbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbf4156e90 0x7fbbf40b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.688+0000 7fbbfbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf406f2b0 con 0x7fbbf4156e90 2024-04-24T08:18:41.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.688+0000 7fbbfbe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf406efd0 con 0x7fbbf41506f0 2024-04-24T08:18:41.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.688+0000 7fbbfbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf40b46a0 con 0x7fbbdc003a00 2024-04-24T08:18:41.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.688+0000 7fbbf9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbf4156e90 0x7fbbf40b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.688+0000 7fbbf9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbf4156e90 0x7fbbf40b06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37678/0 (socket says 172.21.15.69:37678) 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.688+0000 7fbbf9e00640 1 -- 172.21.15.69:0/4264633186 learned_addr learned my addr 172.21.15.69:0/4264633186 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf9400640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbdc003a00 0x7fbbf415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbf41506f0 0x7fbbf4156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf9e00640 1 -- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbdc003a00 msgr2=0x7fbbf415d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf9e00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbdc003a00 0x7fbbf415d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf9e00640 1 -- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbf41506f0 msgr2=0x7fbbf4156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf9e00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbf41506f0 0x7fbbf4156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf9e00640 1 -- 172.21.15.69:0/4264633186 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf415df60 con 0x7fbbf4156e90 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbf41506f0 0x7fbbf4156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf9e00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbf4156e90 0x7fbbf40b06c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40b2f30 tx=0x7fbbec00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf1600640 1 -- 172.21.15.69:0/4264633186 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec017070 con 0x7fbbf4156e90 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf1600640 1 -- 172.21.15.69:0/4264633186 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbec005d40 con 0x7fbbf4156e90 2024-04-24T08:18:41.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbf1600640 1 -- 172.21.15.69:0/4264633186 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbec00a850 con 0x7fbbf4156e90 2024-04-24T08:18:41.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.689+0000 7fbbfbe00640 1 -- 172.21.15.69:0/4264633186 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf415e240 con 0x7fbbf4156e90 2024-04-24T08:18:41.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.690+0000 7fbbfbe00640 1 -- 172.21.15.69:0/4264633186 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf415e800 con 0x7fbbf4156e90 2024-04-24T08:18:41.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.692+0000 7fbbf1600640 1 -- 172.21.15.69:0/4264633186 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbbec00a9f0 con 0x7fbbf4156e90 2024-04-24T08:18:41.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.693+0000 7fbbf1600640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbbd406c8f0 0x7fbbd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.693+0000 7fbbf1600640 1 -- 172.21.15.69:0/4264633186 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fbbec08ad40 con 0x7fbbf4156e90 2024-04-24T08:18:41.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.693+0000 7fbbfec24740 1 Processor -- start 2024-04-24T08:18:41.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.693+0000 7fbbfec24740 1 -- start start 2024-04-24T08:18:41.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.693+0000 7fbbf9400640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbbd406c8f0 0x7fbbd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.694+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab048c30 0x55aeaaffb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.694+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab02b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.694+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab02b610 0x55aeab000420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.694+0000 7fbbfec24740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55aeaaf651f0 con 0x55aeab02bd50 2024-04-24T08:18:41.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.694+0000 7fbbfec24740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55aeaaf0b350 con 0x55aeab048c30 2024-04-24T08:18:41.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.694+0000 7fbbfec24740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55aeaaf611b0 con 0x55aeab02b610 2024-04-24T08:18:41.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab02b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab02b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37690/0 (socket says 172.21.15.69:37690) 2024-04-24T08:18:41.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf8a00640 1 -- 172.21.15.69:0/2637203462 learned_addr learned my addr 172.21.15.69:0/2637203462 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf9400640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbbd406c8f0 0x7fbbd406edb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8032000 tx=0x7fbbe8009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf9e00640 1 --2- 172.21.15.69:0/2637203462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab02b610 0x55aeab000420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf8a00640 1 -- 172.21.15.69:0/2637203462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab02b610 msgr2=0x55aeab000420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/2637203462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab02b610 0x55aeab000420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf9400640 1 --2- 172.21.15.69:0/2637203462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab048c30 0x55aeaaffb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf8a00640 1 -- 172.21.15.69:0/2637203462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab048c30 msgr2=0x55aeaaffb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/2637203462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab048c30 0x55aeaaffb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf8a00640 1 -- 172.21.15.69:0/2637203462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeaaf5ff50 con 0x55aeab02bd50 2024-04-24T08:18:41.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf9e00640 1 --2- 172.21.15.69:0/2637203462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab02b610 0x55aeab000420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:41.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.695+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/2637203462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab02b0d0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fbbe000e970 tx=0x7fbbe000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1473be75241dc84 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.696+0000 7fbbe5c00640 1 -- 172.21.15.69:0/2637203462 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe000f040 con 0x55aeab02bd50 2024-04-24T08:18:41.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.696+0000 7fbbe5c00640 1 -- 172.21.15.69:0/2637203462 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe0010ca0 con 0x55aeab02bd50 2024-04-24T08:18:41.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.696+0000 7fbbe5c00640 1 -- 172.21.15.69:0/2637203462 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe001c640 con 0x55aeab02bd50 2024-04-24T08:18:41.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.696+0000 7fbbfec24740 1 -- 172.21.15.69:0/2637203462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 msgr2=0x55aeab02b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.696+0000 7fbbfec24740 1 --2- 172.21.15.69:0/2637203462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab02b0d0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fbbe000e970 tx=0x7fbbe000ee30 comp rx=0 tx=0).stop 2024-04-24T08:18:41.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.696+0000 7fbbfec24740 1 -- 172.21.15.69:0/2637203462 shutdown_connections 2024-04-24T08:18:41.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.696+0000 7fbbfec24740 1 --2- 172.21.15.69:0/2637203462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab02b0d0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.696+0000 7fbbfec24740 1 --2- 172.21.15.69:0/2637203462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab02b610 0x55aeab000420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.696+0000 7fbbfec24740 1 --2- 172.21.15.69:0/2637203462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab048c30 0x55aeaaffb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.696+0000 7fbbfec24740 1 -- 172.21.15.69:0/2637203462 >> 172.21.15.69:0/2637203462 conn(0x55aeaaffd830 msgr2=0x55aeab0460d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.697+0000 7fbbfec24740 1 -- 172.21.15.69:0/2637203462 shutdown_connections 2024-04-24T08:18:41.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.697+0000 7fbbfec24740 1 -- 172.21.15.69:0/2637203462 wait complete. 2024-04-24T08:18:41.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.697+0000 7fbbfec24740 1 -- start start 2024-04-24T08:18:41.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.697+0000 7fbbfec24740 1 Processor -- start 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.698+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02bd50 0x55aeab0282a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.698+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab048c30 0x55aeab02a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.698+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab02aeb0 0x55aeab026370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbfec24740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55aeaafff9e0 con 0x55aeab048c30 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbfec24740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55aeaaf1e690 con 0x55aeab02bd50 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbfec24740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55aeaaffedd0 con 0x55aeab02aeb0 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbfec24740 10 client.0 _ll_register_callbacks cb 0x55aeaaf5a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbf8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab048c30 0x55aeab02a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbf8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab048c30 0x55aeab02a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37700/0 (socket says 172.21.15.69:37700) 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbf8a00640 1 -- 172.21.15.69:0/139017649 learned_addr learned my addr 172.21.15.69:0/139017649 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbf8a00640 1 -- 172.21.15.69:0/139017649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab02aeb0 msgr2=0x55aeab026370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab02aeb0 0x55aeab026370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbf8a00640 1 -- 172.21.15.69:0/139017649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02bd50 msgr2=0x55aeab0282a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02bd50 0x55aeab0282a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbf8a00640 1 -- 172.21.15.69:0/139017649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeaaf1d920 con 0x55aeab048c30 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab048c30 0x55aeab02a970 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0018000 tx=0x7fbbe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a87b7297df78823 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.699+0000 7fbbda000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbda000640 1 -- 172.21.15.69:0/139017649 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0010aa0 con 0x55aeab048c30 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbda000640 1 -- 172.21.15.69:0/139017649 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fbbe001cbe0 con 0x55aeab048c30 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbfec24740 1 -- 172.21.15.69:0/139017649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe0010ca0 con 0x55aeab048c30 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbda000640 1 -- 172.21.15.69:0/139017649 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fbbe002d050 con 0x55aeab048c30 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbfec24740 10 client.8919 fetch_fsmap learned FSMap version 83 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbfec24740 1 -- 172.21.15.69:0/139017649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x55aeaad417d0 con 0x55aeab048c30 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbda000640 1 -- 172.21.15.69:0/139017649 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe0010ca0 con 0x55aeab048c30 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbda000640 1 -- 172.21.15.69:0/139017649 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7fbbe0030020 con 0x55aeab048c30 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbfec24740 10 client.8919 fetch_fsmap finished waiting for FSMap version 83 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbfec24740 10 client.8919 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:18:41.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbfec24740 20 client.8919 populate_metadata read hostname 'smithi069' 2024-04-24T08:18:41.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbfec24740 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55aeab05a020 0x55aeab05a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.700+0000 7fbbfec24740 4 client.8919 mds_command: new command op to 6046 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:41.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.701+0000 7fbbfec24740 1 -- 172.21.15.69:0/139017649 --> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aeaaf72c10 con 0x55aeab05a020 2024-04-24T08:18:41.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.701+0000 7fbbf9400640 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55aeab05a020 0x55aeab05a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.701+0000 7fbbf9400640 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55aeab05a020 0x55aeab05a420 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb7819f286025b44 server_cookie=7fc607a3c528ecb7 in_seq=0 out_seq=0 2024-04-24T08:18:41.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.701+0000 7fbbda000640 10 client.8919 ms_handle_connect on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:41.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbda000640 1 -- 172.21.15.69:0/139017649 <== mds.0 v2:172.21.15.69:6833/412016900 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aeaaf72c10 con 0x55aeab05a020 2024-04-24T08:18:41.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbda000640 10 client.8919 handle_command_reply: tid=0 2024-04-24T08:18:41.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbfec24740 1 client.8919 shutdown 2024-04-24T08:18:41.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbfec24740 1 -- 172.21.15.69:0/139017649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab048c30 msgr2=0x55aeab02a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbfec24740 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab048c30 0x55aeab02a970 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0018000 tx=0x7fbbe00046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbfec24740 1 -- 172.21.15.69:0/139017649 shutdown_connections 2024-04-24T08:18:41.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbfec24740 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab048c30 0x55aeab02a970 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbfec24740 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55aeab05a020 0x55aeab05a420 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbfec24740 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab02aeb0 0x55aeab026370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbfec24740 1 --2- 172.21.15.69:0/139017649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02bd50 0x55aeab0282a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbda000640 0 client.8919 ms_handle_reset on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:41.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.702+0000 7fbbfec24740 1 -- 172.21.15.69:0/139017649 >> 172.21.15.69:0/139017649 conn(0x55aeab02b610 msgr2=0x55aeaaffac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.703+0000 7fbbfec24740 1 -- 172.21.15.69:0/139017649 shutdown_connections 2024-04-24T08:18:41.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.703+0000 7fbbfec24740 1 -- 172.21.15.69:0/139017649 wait complete. 2024-04-24T08:18:41.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.703+0000 7fbbfec24740 20 client.8919 trim_cache size 0 max 16384 2024-04-24T08:18:41.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.722+0000 7fbbfec24740 1 Processor -- start 2024-04-24T08:18:41.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.722+0000 7fbbfec24740 1 -- start start 2024-04-24T08:18:41.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02aeb0 0x55aeab045160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab046f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab048c30 0x55aeab047440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbfec24740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf41ff3d0 con 0x55aeab02bd50 2024-04-24T08:18:41.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbfec24740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf41fe430 con 0x55aeab02aeb0 2024-04-24T08:18:41.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbfec24740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf4155880 con 0x55aeab048c30 2024-04-24T08:18:41.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbf8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab046f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbf8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab046f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37714/0 (socket says 172.21.15.69:37714) 2024-04-24T08:18:41.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbf8a00640 1 -- 172.21.15.69:0/3181695110 learned_addr learned my addr 172.21.15.69:0/3181695110 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbf8a00640 1 -- 172.21.15.69:0/3181695110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab048c30 msgr2=0x55aeab047440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/3181695110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab048c30 0x55aeab047440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbf8a00640 1 -- 172.21.15.69:0/3181695110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02aeb0 msgr2=0x55aeab045160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/3181695110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02aeb0 0x55aeab045160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.723+0000 7fbbf8a00640 1 -- 172.21.15.69:0/3181695110 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aeab02f340 con 0x55aeab02bd50 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.724+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/3181695110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab046f00 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0010780 tx=0x7fbbe0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e16688571f6ba77d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.724+0000 7fbbe5200640 1 -- 172.21.15.69:0/3181695110 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe001cbc0 con 0x55aeab02bd50 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.724+0000 7fbbe5200640 1 -- 172.21.15.69:0/3181695110 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe0036070 con 0x55aeab02bd50 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.724+0000 7fbbe5200640 1 -- 172.21.15.69:0/3181695110 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0010ca0 con 0x55aeab02bd50 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.725+0000 7fbbfec24740 1 -- 172.21.15.69:0/3181695110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 msgr2=0x55aeab046f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.725+0000 7fbbfec24740 1 --2- 172.21.15.69:0/3181695110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab046f00 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0010780 tx=0x7fbbe0021000 comp rx=0 tx=0).stop 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.725+0000 7fbbfec24740 1 -- 172.21.15.69:0/3181695110 shutdown_connections 2024-04-24T08:18:41.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.725+0000 7fbbfec24740 1 --2- 172.21.15.69:0/3181695110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab046f00 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.725+0000 7fbbfec24740 1 --2- 172.21.15.69:0/3181695110 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab048c30 0x55aeab047440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.725+0000 7fbbfec24740 1 --2- 172.21.15.69:0/3181695110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02aeb0 0x55aeab045160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.725+0000 7fbbfec24740 1 -- 172.21.15.69:0/3181695110 >> 172.21.15.69:0/3181695110 conn(0x55aeab02b610 msgr2=0x55aeab0280b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.725+0000 7fbbfec24740 1 -- 172.21.15.69:0/3181695110 shutdown_connections 2024-04-24T08:18:41.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.725+0000 7fbbfec24740 1 -- 172.21.15.69:0/3181695110 wait complete. 2024-04-24T08:18:41.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.725+0000 7fbbfec24740 1 -- start start 2024-04-24T08:18:41.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.726+0000 7fbbfec24740 1 Processor -- start 2024-04-24T08:18:41.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02aeb0 0x55aeab024fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab041c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbfec24740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab048c30 0x55aeab042190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbfec24740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55aeab045b40 con 0x55aeab02bd50 2024-04-24T08:18:41.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbfec24740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf4155ed0 con 0x55aeab02aeb0 2024-04-24T08:18:41.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbfec24740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55aeab02a3f0 con 0x55aeab048c30 2024-04-24T08:18:41.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbfec24740 10 client.0 _ll_register_callbacks cb 0x7fbbe000e790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:41.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbf8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab041c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbf8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab041c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37728/0 (socket says 172.21.15.69:37728) 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbf8a00640 1 -- 172.21.15.69:0/2446922705 learned_addr learned my addr 172.21.15.69:0/2446922705 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbf9e00640 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab048c30 0x55aeab042190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbf8a00640 1 -- 172.21.15.69:0/2446922705 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab048c30 msgr2=0x55aeab042190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab048c30 0x55aeab042190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbf8a00640 1 -- 172.21.15.69:0/2446922705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02aeb0 msgr2=0x55aeab024fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02aeb0 0x55aeab024fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.727+0000 7fbbf8a00640 1 -- 172.21.15.69:0/2446922705 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aeaac773f0 con 0x55aeab02bd50 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.728+0000 7fbbf8a00640 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab041c50 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0010720 tx=0x7fbbe0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9becd24b423b55b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.728+0000 7fbbda000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.728+0000 7fbbda000640 1 -- 172.21.15.69:0/2446922705 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0003610 con 0x55aeab02bd50 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.728+0000 7fbbda000640 1 -- 172.21.15.69:0/2446922705 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fbbe00037b0 con 0x55aeab02bd50 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.728+0000 7fbbfec24740 1 -- 172.21.15.69:0/2446922705 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe0036070 con 0x55aeab02bd50 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.728+0000 7fbbda000640 1 -- 172.21.15.69:0/2446922705 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fbbe00039b0 con 0x55aeab02bd50 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.728+0000 7fbbfec24740 10 client.8925 fetch_fsmap learned FSMap version 83 2024-04-24T08:18:41.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.728+0000 7fbbfec24740 1 -- 172.21.15.69:0/2446922705 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x55aeaac370f0 con 0x55aeab02bd50 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.728+0000 7fbbda000640 1 -- 172.21.15.69:0/2446922705 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe0048070 con 0x55aeab02bd50 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.729+0000 7fbbda000640 1 -- 172.21.15.69:0/2446922705 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7fbbe0030070 con 0x55aeab02bd50 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.729+0000 7fbbfec24740 10 client.8925 fetch_fsmap finished waiting for FSMap version 83 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.729+0000 7fbbfec24740 10 client.8925 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.729+0000 7fbbfec24740 20 client.8925 populate_metadata read hostname 'smithi069' 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.729+0000 7fbbfec24740 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55aeaaffb6b0 0x55aeab045b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.729+0000 7fbbfec24740 4 client.8925 mds_command: new command op to 6046 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.729+0000 7fbbfec24740 1 -- 172.21.15.69:0/2446922705 --> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55aeab03d600 con 0x55aeaaffb6b0 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.729+0000 7fbbf9400640 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55aeaaffb6b0 0x55aeab045b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.729+0000 7fbbf9400640 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55aeaaffb6b0 0x55aeab045b40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8a953e7e6e5b9da server_cookie=2491b988a41123c4 in_seq=0 out_seq=0 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.729+0000 7fbbda000640 10 client.8925 ms_handle_connect on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbda000640 1 -- 172.21.15.69:0/2446922705 <== mds.0 v2:172.21.15.69:6833/412016900 1 ==== command_reply(tid 0: 0 ) ==== 8+0+260 (crc 0 0 0) 0x55aeab03d600 con 0x55aeaaffb6b0 2024-04-24T08:18:41.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbda000640 10 client.8925 handle_command_reply: tid=0 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 client.8925 shutdown 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 -- 172.21.15.69:0/2446922705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 msgr2=0x55aeab041c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab041c50 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0010720 tx=0x7fbbe0008040 comp rx=0 tx=0).stop 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 -- 172.21.15.69:0/2446922705 shutdown_connections 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55aeab02bd50 0x55aeab041c50 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55aeaaffb6b0 0x55aeab045b40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55aeab048c30 0x55aeab042190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbda000640 0 client.8925 ms_handle_reset on v2:172.21.15.69:6833/412016900 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 --2- 172.21.15.69:0/2446922705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55aeab02aeb0 0x55aeab024fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 -- 172.21.15.69:0/2446922705 >> 172.21.15.69:0/2446922705 conn(0x55aeab02b610 msgr2=0x55aeab02de10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 -- 172.21.15.69:0/2446922705 shutdown_connections 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 1 -- 172.21.15.69:0/2446922705 wait complete. 2024-04-24T08:18:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.730+0000 7fbbfec24740 20 client.8925 trim_cache size 0 max 16384 2024-04-24T08:18:41.736 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:18:41.736 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:18:41.736 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-04-24T08:18:41.736 INFO:teuthology.orchestra.run.smithi069.stdout: "9f1d1be3-f321-4275-b629-caf9c8c5fcf6": { 2024-04-24T08:18:41.736 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/", 2024-04-24T08:18:41.736 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "9f1d1be3-f321-4275-b629-caf9c8c5fcf6", 2024-04-24T08:18:41.736 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,repair" 2024-04-24T08:18:41.736 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:41.736 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:18:41.736 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:18:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 -- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbbd406c8f0 msgr2=0x7fbbd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbbd406c8f0 0x7fbbd406edb0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8032000 tx=0x7fbbe8009030 comp rx=0 tx=0).stop 2024-04-24T08:18:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 -- 172.21.15.69:0/4264633186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbf4156e90 msgr2=0x7fbbf40b06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbf4156e90 0x7fbbf40b06c0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40b2f30 tx=0x7fbbec00cb90 comp rx=0 tx=0).stop 2024-04-24T08:18:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 -- 172.21.15.69:0/4264633186 shutdown_connections 2024-04-24T08:18:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbbd406c8f0 0x7fbbd406edb0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbbf4156e90 0x7fbbf40b06c0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbbf41506f0 0x7fbbf4156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 --2- 172.21.15.69:0/4264633186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbbdc003a00 0x7fbbf415d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 -- 172.21.15.69:0/4264633186 >> 172.21.15.69:0/4264633186 conn(0x7fbbf40d0350 msgr2=0x7fbbf415b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 -- 172.21.15.69:0/4264633186 shutdown_connections 2024-04-24T08:18:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:41.731+0000 7fbbfbe00640 1 -- 172.21.15.69:0/4264633186 wait complete. 2024-04-24T08:18:41.749 INFO:tasks.cephfs.filesystem:scrub status for tag:9f1d1be3-f321-4275-b629-caf9c8c5fcf6 - {'path': '/', 'tag': '9f1d1be3-f321-4275-b629-caf9c8c5fcf6', 'options': 'recursive,repair'} 2024-04-24T08:18:42.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:42.564+0000 7fd8eea00640 1 -- 172.21.15.69:0/3833438189 <== mon.0 v2:172.21.15.69:3300/0 7 ==== log(4 entries from seq 1 at 2024-04-24T08:18:41.484051+0000) ==== 695+0+0 (secure 0 0 0) 0x7fd8e808d000 con 0x7fd8f80d10c0 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: cluster: 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: id: 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: health: HEALTH_OK 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: services: 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: mon: 3 daemons, quorum a,b,c (age 10m) 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: mgr: x(active, since 10m), standbys: y 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: mds: 1/1 daemons up, 3 standby 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: osd: 8 osds: 8 up (since 10m), 8 in (since 10m) 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:42.566 INFO:teuthology.orchestra.run.smithi069.stdout: data: 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout: volumes: 1/1 healthy 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout: pools: 3 pools, 97 pgs 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout: objects: 30 objects, 19 MiB 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout: usage: 290 MiB used, 720 GiB / 720 GiB avail 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout: pgs: 97 active+clean 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout: io: 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout: client: 85 B/s rd, 1.0 MiB/s wr, 0 op/s rd, 2 op/s wr 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:18:41.484051+0000 mds.a [INF] scrub queued for path: / 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:18:41.484054+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:18:41.484059+0000 mds.a [INF] scrub summary: active paths [/] 2024-04-24T08:18:42.567 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:18:41.486076+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-04-24T08:18:45.582 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:18:45.471616+0000 mds.a [INF] scrub summary: idle 2024-04-24T08:18:45.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:18:45.580+0000 7fd8eea00640 1 -- 172.21.15.69:0/3833438189 <== mon.0 v2:172.21.15.69:3300/0 8 ==== log(1 entries from seq 5 at 2024-04-24T08:18:45.471616+0000) ==== 189+0+0 (secure 0 0 0) 0x7fd8e8058840 con 0x7fd8f80d10c0 2024-04-24T08:18:46.140 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:18:46.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:19:11.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:19:11.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-24T08:19:11.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.961+0000 7fc5f1e00640 1 Processor -- start 2024-04-24T08:19:11.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.961+0000 7fc5f1e00640 1 -- start start 2024-04-24T08:19:11.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.962+0000 7fc5f1e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec155f20 0x7fc5ec0ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.962+0000 7fc5f1e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc5ec15c9b0 0x7fc5ec0b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.962+0000 7fc5f1e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5ec0cd6d0 con 0x7fc5ec0b3f70 2024-04-24T08:19:11.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.962+0000 7fc5f1e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc5ec06db90 con 0x7fc5ec15c9b0 2024-04-24T08:19:11.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.962+0000 7fc5f1e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ec072070 con 0x7fc5ec155f20 2024-04-24T08:19:11.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5eb400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc5ec0b3f70 0x7fc5ec0cf360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37606/0 (socket says 172.21.15.69:37606) 2024-04-24T08:19:11.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5eb400640 1 -- 172.21.15.69:0/177456486 learned_addr learned my addr 172.21.15.69:0/177456486 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:11.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5e2000640 1 --2- 172.21.15.69:0/177456486 >> v2:172.21.15.121:3300/0 conn(0x7fc5ec15c9b0 0x7fc5ec0b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/177456486 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec155f20 0x7fc5ec0ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1365289367 0 0) 0x7fc5ec0cd6d0 con 0x7fc5ec0b3f70 2024-04-24T08:19:11.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5d0003520 con 0x7fc5ec0b3f70 2024-04-24T08:19:11.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3356290518 0 0) 0x7fc5d0003520 con 0x7fc5ec0b3f70 2024-04-24T08:19:11.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec155f20 msgr2=0x7fc5ec0ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:11.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5ea000640 1 --2- 172.21.15.69:0/177456486 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec155f20 0x7fc5ec0ccd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 >> v2:172.21.15.121:3300/0 conn(0x7fc5ec15c9b0 msgr2=0x7fc5ec0b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5ea000640 1 --2- 172.21.15.69:0/177456486 >> v2:172.21.15.121:3300/0 conn(0x7fc5ec15c9b0 0x7fc5ec0b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.963+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ec0cd9b0 con 0x7fc5ec0b3f70 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fc5e40019f0 con 0x7fc5ec0b3f70 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 >> v1:172.21.15.69:6789/0 conn(0x7fc5ec0b3f70 legacy=0x7fc5ec0cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5ea000640 1 --2- 172.21.15.69:0/177456486 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec155f20 0x7fc5d00039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5ea000640 1 --2- 172.21.15.69:0/177456486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5d0003f50 0x7fc5d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5ea000640 1 --2- 172.21.15.69:0/177456486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5d0004d40 0x7fc5d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec072070 con 0x7fc5d0003f50 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d000b9c0 con 0x7fc5d0004d40 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc5d000bb90 con 0x7fc5ec155f20 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5e2000640 1 --2- 172.21.15.69:0/177456486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5d0003f50 0x7fc5d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5e2000640 1 -- 172.21.15.69:0/177456486 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec155f20 msgr2=0x7fc5d00039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5e2000640 1 --2- 172.21.15.69:0/177456486 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec155f20 0x7fc5d00039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5e2000640 1 -- 172.21.15.69:0/177456486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5d0004d40 msgr2=0x7fc5d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:11.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5e2000640 1 --2- 172.21.15.69:0/177456486 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5d0004d40 0x7fc5d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.964+0000 7fc5e2000640 1 -- 172.21.15.69:0/177456486 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d0003520 con 0x7fc5d0003f50 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.965+0000 7fc5e2000640 1 --2- 172.21.15.69:0/177456486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5d0003f50 0x7fc5d0004800 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4009a90 tx=0x7fc5d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d048bdf26e3d65bc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.966+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4036070 con 0x7fc5d0003f50 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.966+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d4031400 con 0x7fc5d0003f50 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.966+0000 7fc5ea000640 1 -- 172.21.15.69:0/177456486 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4009690 con 0x7fc5d0003f50 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.966+0000 7fc5f1e00640 1 -- 172.21.15.69:0/177456486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5d0003f50 msgr2=0x7fc5d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.966+0000 7fc5f1e00640 1 --2- 172.21.15.69:0/177456486 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5d0003f50 0x7fc5d0004800 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4009a90 tx=0x7fc5d0005830 comp rx=0 tx=0).stop 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.966+0000 7fc5eb400640 1 -- 172.21.15.69:0/177456486 reap_dead start 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.966+0000 7fc5f1e00640 1 -- 172.21.15.69:0/177456486 shutdown_connections 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.966+0000 7fc5f1e00640 1 -- 172.21.15.69:0/177456486 >> 172.21.15.69:0/177456486 conn(0x7fc5ec0d0350 msgr2=0x7fc5ec0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.966+0000 7fc5f1e00640 1 -- 172.21.15.69:0/177456486 shutdown_connections 2024-04-24T08:19:11.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.966+0000 7fc5f1e00640 1 -- 172.21.15.69:0/177456486 wait complete. 2024-04-24T08:19:11.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.968+0000 7fc5f1e00640 1 Processor -- start 2024-04-24T08:19:11.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.968+0000 7fc5f1e00640 1 -- start start 2024-04-24T08:19:11.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.969+0000 7fc5f1e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5d0003f50 0x7fc5ec06db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.969+0000 7fc5f1e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec0b1ae0 0x7fc5ec06e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.969+0000 7fc5f1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5ec1efbd0 0x7fc5ec1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.969+0000 7fc5f1e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec0cfa70 con 0x7fc5ec1efbd0 2024-04-24T08:19:11.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.969+0000 7fc5f1e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec071d00 con 0x7fc5d0003f50 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.969+0000 7fc5f1e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ec0cfbf0 con 0x7fc5ec0b1ae0 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.969+0000 7fc5eb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5ec1efbd0 0x7fc5ec1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.969+0000 7fc5eb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5ec1efbd0 0x7fc5ec1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42018/0 (socket says 172.21.15.69:42018) 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.969+0000 7fc5eb400640 1 -- 172.21.15.69:0/3224665308 learned_addr learned my addr 172.21.15.69:0/3224665308 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.970+0000 7fc5e2000640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec0b1ae0 0x7fc5ec06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.970+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5d0003f50 0x7fc5ec06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.970+0000 7fc5eb400640 1 -- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec0b1ae0 msgr2=0x7fc5ec06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.970+0000 7fc5eb400640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec0b1ae0 0x7fc5ec06e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.970+0000 7fc5eb400640 1 -- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5d0003f50 msgr2=0x7fc5ec06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.970+0000 7fc5eb400640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5d0003f50 0x7fc5ec06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.970+0000 7fc5eb400640 1 -- 172.21.15.69:0/3224665308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ec0b09d0 con 0x7fc5ec1efbd0 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.970+0000 7fc5eb400640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5ec1efbd0 0x7fc5ec1563d0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0d1010 tx=0x7fc5e400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.970+0000 7fc5e3e00640 1 -- 172.21.15.69:0/3224665308 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e400a610 con 0x7fc5ec1efbd0 2024-04-24T08:19:11.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.971+0000 7fc5e3e00640 1 -- 172.21.15.69:0/3224665308 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5e400a7d0 con 0x7fc5ec1efbd0 2024-04-24T08:19:11.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.971+0000 7fc5e3e00640 1 -- 172.21.15.69:0/3224665308 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e400abc0 con 0x7fc5ec1efbd0 2024-04-24T08:19:11.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.971+0000 7fc5f1e00640 1 -- 172.21.15.69:0/3224665308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5ec0b0cb0 con 0x7fc5ec1efbd0 2024-04-24T08:19:11.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.971+0000 7fc5f1e00640 1 -- 172.21.15.69:0/3224665308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5ec0b1270 con 0x7fc5ec1efbd0 2024-04-24T08:19:11.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.973+0000 7fc5e3e00640 1 -- 172.21.15.69:0/3224665308 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc5e4011070 con 0x7fc5ec1efbd0 2024-04-24T08:19:11.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.974+0000 7fc5e3e00640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc5c8070d30 0x7fc5c80731f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.974+0000 7fc5e3e00640 1 -- 172.21.15.69:0/3224665308 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fc5e408b1d0 con 0x7fc5ec1efbd0 2024-04-24T08:19:11.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.974+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc5c8070d30 0x7fc5c80731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.975+0000 7fc5f4de8740 1 Processor -- start 2024-04-24T08:19:11.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.975+0000 7fc5f4de8740 1 -- start start 2024-04-24T08:19:11.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896dbd50 0x55a6896db0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896db610 0x55a6896b0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a6896151f0 con 0x55a6896f8c30 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a6895bb350 con 0x55a6896db610 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a6896111b0 con 0x55a6896dbd50 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc5c8070d30 0x7fc5c80731f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0047b0 tx=0x7fc5dc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5eaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5eaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896ab5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42028/0 (socket says 172.21.15.69:42028) 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5eaa00640 1 -- 172.21.15.69:0/4171150071 learned_addr learned my addr 172.21.15.69:0/4171150071 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5e2000640 1 --2- 172.21.15.69:0/4171150071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896dbd50 0x55a6896db0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5eaa00640 1 -- 172.21.15.69:0/4171150071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896dbd50 msgr2=0x55a6896db0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5eb400640 1 --2- 172.21.15.69:0/4171150071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896db610 0x55a6896b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/4171150071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896dbd50 0x55a6896db0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5eaa00640 1 -- 172.21.15.69:0/4171150071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896db610 msgr2=0x55a6896b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/4171150071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896db610 0x55a6896b0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.976+0000 7fc5eaa00640 1 -- 172.21.15.69:0/4171150071 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a68960ff50 con 0x55a6896f8c30 2024-04-24T08:19:11.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.977+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/4171150071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896ab5c0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc011b80 tx=0x7fc5dc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8276377634bb121 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:11.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.977+0000 7fc5db400640 1 -- 172.21.15.69:0/4171150071 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc015a80 con 0x55a6896f8c30 2024-04-24T08:19:11.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.977+0000 7fc5db400640 1 -- 172.21.15.69:0/4171150071 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc026070 con 0x55a6896f8c30 2024-04-24T08:19:11.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.977+0000 7fc5db400640 1 -- 172.21.15.69:0/4171150071 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc015e20 con 0x55a6896f8c30 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.978+0000 7fc5f4de8740 1 -- 172.21.15.69:0/4171150071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 msgr2=0x55a6896ab5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.978+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/4171150071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896ab5c0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc011b80 tx=0x7fc5dc014000 comp rx=0 tx=0).stop 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.978+0000 7fc5f4de8740 1 -- 172.21.15.69:0/4171150071 shutdown_connections 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.978+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/4171150071 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896db610 0x55a6896b0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.978+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/4171150071 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896dbd50 0x55a6896db0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.978+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/4171150071 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896ab5c0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.978+0000 7fc5f4de8740 1 -- 172.21.15.69:0/4171150071 >> 172.21.15.69:0/4171150071 conn(0x55a6896ad830 msgr2=0x55a6896f60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.978+0000 7fc5f4de8740 1 -- 172.21.15.69:0/4171150071 shutdown_connections 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.978+0000 7fc5f4de8740 1 -- 172.21.15.69:0/4171150071 wait complete. 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.979+0000 7fc5f4de8740 1 -- start start 2024-04-24T08:19:11.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.979+0000 7fc5f4de8740 1 Processor -- start 2024-04-24T08:19:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.980+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896d6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.980+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896f8c30 0x55a689706520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.980+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896de500 0x55a6896de980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.980+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a6896af9e0 con 0x55a6896dbd50 2024-04-24T08:19:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.980+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55a6895ce690 con 0x55a6896de500 2024-04-24T08:19:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.980+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a6896aedd0 con 0x55a6896f8c30 2024-04-24T08:19:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.980+0000 7fc5f4de8740 10 client.0 _ll_register_callbacks cb 0x55a68960a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.980+0000 7fc5eaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896d6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.980+0000 7fc5eaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896d6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42036/0 (socket says 172.21.15.69:42036) 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.980+0000 7fc5eaa00640 1 -- 172.21.15.69:0/546592272 learned_addr learned my addr 172.21.15.69:0/546592272 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5eaa00640 1 -- 172.21.15.69:0/546592272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896f8c30 msgr2=0x55a689706520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896f8c30 0x55a689706520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5eaa00640 1 -- 172.21.15.69:0/546592272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896de500 msgr2=0x55a6896de980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896de500 0x55a6896de980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5eaa00640 1 -- 172.21.15.69:0/546592272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6895cd920 con 0x55a6896dbd50 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896d6bc0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0096f0 tx=0x7fc5dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b45dd3902a1a78 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5cfe00640 1 -- 172.21.15.69:0/546592272 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc021790 con 0x55a6896dbd50 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5cfe00640 1 -- 172.21.15.69:0/546592272 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fc5dc021930 con 0x55a6896dbd50 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5f4de8740 1 -- 172.21.15.69:0/546592272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5dc026070 con 0x55a6896dbd50 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.981+0000 7fc5cfe00640 1 -- 172.21.15.69:0/546592272 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fc5dc021b30 con 0x55a6896dbd50 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5f4de8740 10 client.8937 fetch_fsmap learned FSMap version 83 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5f4de8740 1 -- 172.21.15.69:0/546592272 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x55a6893f17d0 con 0x55a6896dbd50 2024-04-24T08:19:11.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5cfe00640 1 -- 172.21.15.69:0/546592272 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc026070 con 0x55a6896dbd50 2024-04-24T08:19:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5cfe00640 1 -- 172.21.15.69:0/546592272 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7fc5dc040020 con 0x55a6896dbd50 2024-04-24T08:19:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5f4de8740 10 client.8937 fetch_fsmap finished waiting for FSMap version 83 2024-04-24T08:19:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5f4de8740 10 client.8937 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:19:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5f4de8740 20 client.8937 populate_metadata read hostname 'smithi069' 2024-04-24T08:19:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55a6896edbe0 0x55a6896ee040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5f4de8740 4 client.8937 mds_command: new command op to 6046 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5f4de8740 1 -- 172.21.15.69:0/546592272 --> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a689622c10 con 0x55a6896edbe0 2024-04-24T08:19:11.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.982+0000 7fc5e2000640 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55a6896edbe0 0x55a6896ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:11.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.983+0000 7fc5e2000640 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55a6896edbe0 0x55a6896ee040 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e69940819452a89b server_cookie=f51aec66a0c496ae in_seq=0 out_seq=0 2024-04-24T08:19:11.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.983+0000 7fc5cfe00640 10 client.8937 ms_handle_connect on v2:172.21.15.69:6833/412016900 2024-04-24T08:19:11.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5cfe00640 1 -- 172.21.15.69:0/546592272 <== mds.0 v2:172.21.15.69:6833/412016900 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a689622c10 con 0x55a6896edbe0 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5cfe00640 10 client.8937 handle_command_reply: tid=0 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 client.8937 shutdown 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 -- 172.21.15.69:0/546592272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 msgr2=0x55a6896d6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896d6bc0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0096f0 tx=0x7fc5dc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 -- 172.21.15.69:0/546592272 shutdown_connections 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55a6896edbe0 0x55a6896ee040 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896de500 0x55a6896de980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896f8c30 0x55a689706520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/546592272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896d6bc0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5cfe00640 0 client.8937 ms_handle_reset on v2:172.21.15.69:6833/412016900 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 -- 172.21.15.69:0/546592272 >> 172.21.15.69:0/546592272 conn(0x55a6896db610 msgr2=0x55a6896ac300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 -- 172.21.15.69:0/546592272 shutdown_connections 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.984+0000 7fc5f4de8740 1 -- 172.21.15.69:0/546592272 wait complete. 2024-04-24T08:19:11.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:11.985+0000 7fc5f4de8740 20 client.8937 trim_cache size 0 max 16384 2024-04-24T08:19:12.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.004+0000 7fc5f4de8740 1 Processor -- start 2024-04-24T08:19:12.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.004+0000 7fc5f4de8740 1 -- start start 2024-04-24T08:19:12.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.004+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896ddd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.004+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896de500 0x55a6896f94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.004+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896f8c30 0x55a6896f9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.004+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec1fcf80 con 0x55a6896dbd50 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.005+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec1fc030 con 0x55a6896de500 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.005+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ec15ac00 con 0x55a6896f8c30 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.005+0000 7fc5eaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896ddd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.005+0000 7fc5eaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896ddd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42044/0 (socket says 172.21.15.69:42044) 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.005+0000 7fc5eaa00640 1 -- 172.21.15.69:0/4232703276 learned_addr learned my addr 172.21.15.69:0/4232703276 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.005+0000 7fc5eaa00640 1 -- 172.21.15.69:0/4232703276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896f8c30 msgr2=0x55a6896f9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.005+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/4232703276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896f8c30 0x55a6896f9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.005+0000 7fc5eaa00640 1 -- 172.21.15.69:0/4232703276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896de500 msgr2=0x55a6896f94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.005+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/4232703276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896de500 0x55a6896f94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.005+0000 7fc5eaa00640 1 -- 172.21.15.69:0/4232703276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6893273f0 con 0x55a6896dbd50 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/4232703276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896ddd10 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc021eb0 tx=0x7fc5dc02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6de063c99621f77 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5daa00640 1 -- 172.21.15.69:0/4232703276 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc046070 con 0x55a6896dbd50 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5daa00640 1 -- 172.21.15.69:0/4232703276 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc026040 con 0x55a6896dbd50 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5daa00640 1 -- 172.21.15.69:0/4232703276 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc004040 con 0x55a6896dbd50 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5f4de8740 1 -- 172.21.15.69:0/4232703276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 msgr2=0x55a6896ddd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/4232703276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896ddd10 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc021eb0 tx=0x7fc5dc02e070 comp rx=0 tx=0).stop 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5f4de8740 1 -- 172.21.15.69:0/4232703276 shutdown_connections 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/4232703276 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896f8c30 0x55a6896f9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/4232703276 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896de500 0x55a6896f94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/4232703276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896dbd50 0x55a6896ddd10 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.006+0000 7fc5f4de8740 1 -- 172.21.15.69:0/4232703276 >> 172.21.15.69:0/4232703276 conn(0x55a6896db610 msgr2=0x55a6896df030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.007+0000 7fc5f4de8740 1 -- 172.21.15.69:0/4232703276 shutdown_connections 2024-04-24T08:19:12.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.007+0000 7fc5f4de8740 1 -- 172.21.15.69:0/4232703276 wait complete. 2024-04-24T08:19:12.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.007+0000 7fc5f4de8740 1 -- start start 2024-04-24T08:19:12.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.007+0000 7fc5f4de8740 1 Processor -- start 2024-04-24T08:19:12.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.008+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896dbd50 0x55a68976f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.008+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896de500 0x55a6896f94f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.008+0000 7fc5f4de8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896f9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.008+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a6896eea50 con 0x55a6896f8c30 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.008+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec1fb9e0 con 0x55a6896dbd50 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.008+0000 7fc5f4de8740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55a6896fa0c0 con 0x55a6896de500 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.008+0000 7fc5f4de8740 10 client.0 _ll_register_callbacks cb 0x7fc5dc011a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5eb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896f9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5eb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896f9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42052/0 (socket says 172.21.15.69:42052) 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5eb400640 1 -- 172.21.15.69:0/2466277306 learned_addr learned my addr 172.21.15.69:0/2466277306 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5e2000640 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896de500 0x55a6896f94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5eb400640 1 -- 172.21.15.69:0/2466277306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896de500 msgr2=0x55a6896f94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5eb400640 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896de500 0x55a6896f94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5eb400640 1 -- 172.21.15.69:0/2466277306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896dbd50 msgr2=0x55a68976f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896dbd50 0x55a68976f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5eb400640 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896dbd50 0x55a68976f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5eb400640 1 -- 172.21.15.69:0/2466277306 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6892e70f0 con 0x55a6896f8c30 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5eb400640 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896f9a30 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4058c00 tx=0x7fc5e4059600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2899d849f6271056 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5cfe00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5cfe00640 1 -- 172.21.15.69:0/2466277306 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5e401a070 con 0x55a6896f8c30 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5cfe00640 1 -- 172.21.15.69:0/2466277306 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7fc5e4058350 con 0x55a6896f8c30 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.009+0000 7fc5f4de8740 1 -- 172.21.15.69:0/2466277306 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc5dc026040 con 0x55a6896f8c30 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5cfe00640 1 -- 172.21.15.69:0/2466277306 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fc5e4061ae0 con 0x55a6896f8c30 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5f4de8740 10 client.8943 fetch_fsmap learned FSMap version 83 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5f4de8740 1 -- 172.21.15.69:0/2466277306 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) -- 0x55a6895da810 con 0x55a6896f8c30 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5cfe00640 1 -- 172.21.15.69:0/2466277306 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc026040 con 0x55a6896f8c30 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5cfe00640 1 -- 172.21.15.69:0/2466277306 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 83) ==== 3494+0+0 (secure 0 0 0) 0x7fc5e4089b20 con 0x55a6896f8c30 2024-04-24T08:19:12.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5f4de8740 10 client.8943 fetch_fsmap finished waiting for FSMap version 83 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5f4de8740 10 client.8943 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5f4de8740 20 client.8943 populate_metadata read hostname 'smithi069' 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55a689705f50 0x55a6897730a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5f4de8740 4 client.8943 mds_command: new command op to 6046 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5f4de8740 1 -- 172.21.15.69:0/2466277306 --> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a68961ca80 con 0x55a689705f50 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.010+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55a689705f50 0x55a6897730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5eaa00640 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55a689705f50 0x55a6897730a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd7db2295884302e server_cookie=9512e0534041f718 in_seq=0 out_seq=0 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5cfe00640 10 client.8943 ms_handle_connect on v2:172.21.15.69:6833/412016900 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5cfe00640 1 -- 172.21.15.69:0/2466277306 <== mds.0 v2:172.21.15.69:6833/412016900 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a68961ca80 con 0x55a689705f50 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5cfe00640 10 client.8943 handle_command_reply: tid=0 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5f4de8740 1 client.8943 shutdown 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5f4de8740 1 -- 172.21.15.69:0/2466277306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 msgr2=0x55a6896f9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896f9a30 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4058c00 tx=0x7fc5e4059600 comp rx=0 tx=0).stop 2024-04-24T08:19:12.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5f4de8740 1 -- 172.21.15.69:0/2466277306 shutdown_connections 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6896f8c30 0x55a6896f9a30 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.69:6833/412016900,v1:172.21.15.69:6835/412016900] conn(0x55a689705f50 0x55a6897730a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55a6896de500 0x55a6896f94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5f4de8740 1 --2- 172.21.15.69:0/2466277306 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a6896dbd50 0x55a68976f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5cfe00640 0 client.8943 ms_handle_reset on v2:172.21.15.69:6833/412016900 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.011+0000 7fc5f4de8740 1 -- 172.21.15.69:0/2466277306 >> 172.21.15.69:0/2466277306 conn(0x55a6896db610 msgr2=0x55a6896eebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f4de8740 1 -- 172.21.15.69:0/2466277306 shutdown_connections 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f4de8740 1 -- 172.21.15.69:0/2466277306 wait complete. 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f4de8740 20 client.8943 trim_cache size 0 max 16384 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:19:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-04-24T08:19:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-04-24T08:19:12.018 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:19:12.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f1e00640 1 -- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc5c8070d30 msgr2=0x7fc5c80731f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f1e00640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc5c8070d30 0x7fc5c80731f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0047b0 tx=0x7fc5dc008040 comp rx=0 tx=0).stop 2024-04-24T08:19:12.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f1e00640 1 -- 172.21.15.69:0/3224665308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5ec1efbd0 msgr2=0x7fc5ec1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f1e00640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5ec1efbd0 0x7fc5ec1563d0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec0d1010 tx=0x7fc5e400cd50 comp rx=0 tx=0).stop 2024-04-24T08:19:12.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f1e00640 1 -- 172.21.15.69:0/3224665308 shutdown_connections 2024-04-24T08:19:12.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f1e00640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc5c8070d30 0x7fc5c80731f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f1e00640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc5ec1efbd0 0x7fc5ec1563d0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f1e00640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc5ec0b1ae0 0x7fc5ec06e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f1e00640 1 --2- 172.21.15.69:0/3224665308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5d0003f50 0x7fc5ec06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.012+0000 7fc5f1e00640 1 -- 172.21.15.69:0/3224665308 >> 172.21.15.69:0/3224665308 conn(0x7fc5ec0d0350 msgr2=0x7fc5ec15bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.013+0000 7fc5f1e00640 1 -- 172.21.15.69:0/3224665308 shutdown_connections 2024-04-24T08:19:12.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.013+0000 7fc5f1e00640 1 -- 172.21.15.69:0/3224665308 wait complete. 2024-04-24T08:19:12.032 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:19:12.133 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:19:12.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.331+0000 7f2703200640 1 Processor -- start 2024-04-24T08:19:12.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.332+0000 7f2703200640 1 -- start start 2024-04-24T08:19:12.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2703200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f270415e960 0x7f2704162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2703200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040cfaf0 0x7f27040cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2703200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270415ed40 con 0x7f2704162980 2024-04-24T08:19:12.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2703200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f27040cf050 con 0x7f270415e960 2024-04-24T08:19:12.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2703200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2704162e20 con 0x7f27040cfaf0 2024-04-24T08:19:12.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2701400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2704162980 0x7f270406cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37622/0 (socket says 172.21.15.69:37622) 2024-04-24T08:19:12.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2701400640 1 -- 172.21.15.69:0/1910065337 learned_addr learned my addr 172.21.15.69:0/1910065337 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3084156029 0 0) 0x7f270415ed40 con 0x7f2704162980 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2701e00640 1 --2- 172.21.15.69:0/1910065337 >> v2:172.21.15.121:3300/0 conn(0x7f270415e960 0x7f2704162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26e4003520 con 0x7f2704162980 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.333+0000 7f2702800640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040cfaf0 0x7f27040cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3300365814 0 0) 0x7f26e4003520 con 0x7f2704162980 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040cfaf0 msgr2=0x7f27040cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040cfaf0 0x7f27040cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 >> v2:172.21.15.121:3300/0 conn(0x7f270415e960 msgr2=0x7f2704162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 --2- 172.21.15.69:0/1910065337 >> v2:172.21.15.121:3300/0 conn(0x7f270415e960 0x7f2704162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2704163100 con 0x7f2704162980 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f26ec0019f0 con 0x7f2704162980 2024-04-24T08:19:12.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 >> v1:172.21.15.69:6789/0 conn(0x7f2704162980 legacy=0x7f270406cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 0x7f26e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26e4004800 0x7f26e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040cfaf0 0x7f26e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f27040cf050 con 0x7f26e40039b0 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2704162e20 con 0x7f26e4004800 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f26e400ba70 con 0x7f27040cfaf0 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.334+0000 7f2701e00640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 0x7f26e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.335+0000 7f2702800640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040cfaf0 0x7f26e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.335+0000 7f2701400640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26e4004800 0x7f26e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.335+0000 7f2701e00640 1 -- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040cfaf0 msgr2=0x7f26e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.335+0000 7f2701e00640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040cfaf0 0x7f26e40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.335+0000 7f2701e00640 1 -- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26e4004800 msgr2=0x7f26e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.335+0000 7f2701e00640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26e4004800 0x7f26e4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.335+0000 7f2701e00640 1 -- 172.21.15.69:0/1910065337 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26e4003520 con 0x7f26e40039b0 2024-04-24T08:19:12.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.336+0000 7f2701e00640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 0x7f26e4003dd0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f26f400ba00 tx=0x7f26e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9797c6447c56c457 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.336+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4030070 con 0x7f26e40039b0 2024-04-24T08:19:12.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.336+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f402adb0 con 0x7f26e40039b0 2024-04-24T08:19:12.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.336+0000 7f2700a00640 1 -- 172.21.15.69:0/1910065337 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f400b600 con 0x7f26e40039b0 2024-04-24T08:19:12.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.337+0000 7f2703200640 1 -- 172.21.15.69:0/1910065337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 msgr2=0x7f26e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.337+0000 7f2703200640 1 --2- 172.21.15.69:0/1910065337 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 0x7f26e4003dd0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f26f400ba00 tx=0x7f26e4005830 comp rx=0 tx=0).stop 2024-04-24T08:19:12.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.337+0000 7f2702800640 1 -- 172.21.15.69:0/1910065337 reap_dead start 2024-04-24T08:19:12.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.337+0000 7f2703200640 1 -- 172.21.15.69:0/1910065337 shutdown_connections 2024-04-24T08:19:12.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.337+0000 7f2703200640 1 -- 172.21.15.69:0/1910065337 >> 172.21.15.69:0/1910065337 conn(0x7f270409d900 msgr2=0x7f270408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.337+0000 7f2703200640 1 -- 172.21.15.69:0/1910065337 shutdown_connections 2024-04-24T08:19:12.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.337+0000 7f2703200640 1 -- 172.21.15.69:0/1910065337 wait complete. 2024-04-24T08:19:12.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.339+0000 7f2703200640 1 Processor -- start 2024-04-24T08:19:12.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.339+0000 7f2703200640 1 -- start start 2024-04-24T08:19:12.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2703200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 0x7f2704159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2703200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270406dc60 0x7f270406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2703200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040c9780 0x7f27040c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2703200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f270406d6c0 con 0x7f26e40039b0 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2703200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f27040cf5d0 con 0x7f270406dc60 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2703200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f27040b6170 con 0x7f27040c9780 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2701e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 0x7f2704159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2701e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 0x7f2704159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42070/0 (socket says 172.21.15.69:42070) 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2701e00640 1 -- 172.21.15.69:0/1446610673 learned_addr learned my addr 172.21.15.69:0/1446610673 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2701e00640 1 -- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040c9780 msgr2=0x7f27040c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2701e00640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040c9780 0x7f27040c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2701e00640 1 -- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270406dc60 msgr2=0x7f270406e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2701e00640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270406dc60 0x7f270406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.340+0000 7f2701e00640 1 -- 172.21.15.69:0/1446610673 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27040c8190 con 0x7f26e40039b0 2024-04-24T08:19:12.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.341+0000 7f2701e00640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 0x7f2704159880 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f26f400bb20 tx=0x7f26f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.341+0000 7f26fa000640 1 -- 172.21.15.69:0/1446610673 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4043020 con 0x7f26e40039b0 2024-04-24T08:19:12.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.341+0000 7f26fa000640 1 -- 172.21.15.69:0/1446610673 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f403f680 con 0x7f26e40039b0 2024-04-24T08:19:12.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.341+0000 7f26fa000640 1 -- 172.21.15.69:0/1446610673 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f400b600 con 0x7f26e40039b0 2024-04-24T08:19:12.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.341+0000 7f2703200640 1 -- 172.21.15.69:0/1446610673 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27040c8470 con 0x7f26e40039b0 2024-04-24T08:19:12.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.341+0000 7f2703200640 1 -- 172.21.15.69:0/1446610673 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27040c8a30 con 0x7f26e40039b0 2024-04-24T08:19:12.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.342+0000 7f2703200640 1 -- 172.21.15.69:0/1446610673 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f270406e600 con 0x7f26e40039b0 2024-04-24T08:19:12.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.343+0000 7f26fa000640 1 -- 172.21.15.69:0/1446610673 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f26f4046020 con 0x7f26e40039b0 2024-04-24T08:19:12.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.344+0000 7f26fa000640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f26e006c8f0 0x7f26e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.344+0000 7f26fa000640 1 -- 172.21.15.69:0/1446610673 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 5709+0+0 (secure 0 0 0) 0x7f26f40ac780 con 0x7f26e40039b0 2024-04-24T08:19:12.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.344+0000 7f2701400640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f26e006c8f0 0x7f26e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.345+0000 7f2701400640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f26e006c8f0 0x7f26e006edb0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f26ec001370 tx=0x7f26ec009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.348+0000 7f26fa000640 1 -- 172.21.15.69:0/1446610673 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26f404d030 con 0x7f26e40039b0 2024-04-24T08:19:12.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.586+0000 7f2703200640 1 -- 172.21.15.69:0/1446610673 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f2704075d50 con 0x7f26e40039b0 2024-04-24T08:19:12.589 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:19:12.587+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:19:12.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.591+0000 7f26fa000640 1 -- 172.21.15.69:0/1446610673 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) ==== 161+0+0 (secure 0 0 0) 0x7f26f407aa80 con 0x7f26e40039b0 2024-04-24T08:19:12.593 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:19:12.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 -- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f26e006c8f0 msgr2=0x7f26e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f26e006c8f0 0x7f26e006edb0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f26ec001370 tx=0x7f26ec009040 comp rx=0 tx=0).stop 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 -- 172.21.15.69:0/1446610673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 msgr2=0x7f2704159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 0x7f2704159880 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f26f400bb20 tx=0x7f26f4004730 comp rx=0 tx=0).stop 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 -- 172.21.15.69:0/1446610673 shutdown_connections 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f26e006c8f0 0x7f26e006edb0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f27040c9780 0x7f27040c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f270406dc60 0x7f270406e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 --2- 172.21.15.69:0/1446610673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26e40039b0 0x7f2704159880 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 -- 172.21.15.69:0/1446610673 >> 172.21.15.69:0/1446610673 conn(0x7f270409d900 msgr2=0x7f270415cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 -- 172.21.15.69:0/1446610673 shutdown_connections 2024-04-24T08:19:12.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.594+0000 7f2703200640 1 -- 172.21.15.69:0/1446610673 wait complete. 2024-04-24T08:19:12.617 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627777 summary 2024-04-24T08:19:12.625 INFO:tasks.ceph.mds.a.smithi069.stderr: -6313> 2024-04-24T08:18:16.880+0000 7f7949200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:19:12.684 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:19:12.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.718+0000 7ffb42521280 1 -- start start 2024-04-24T08:19:12.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.718+0000 7ffb42521280 1 Processor -- start 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.719+0000 7ffb42521280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddbe4ec00 0x556ddcc16000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.719+0000 7ffb42521280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddbe4f800 0x556ddcc16580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.719+0000 7ffb42521280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddbe4fc00 0x556ddcc16b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.719+0000 7ffb42521280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556ddcc32180 con 0x556ddbe4fc00 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.719+0000 7ffb42521280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556ddcc32300 con 0x556ddbe4ec00 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.719+0000 7ffb42521280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556ddcc32480 con 0x556ddbe4f800 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb40600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddbe4fc00 0x556ddcc16b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb40600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddbe4fc00 0x556ddcc16b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42086/0 (socket says 172.21.15.69:42086) 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb40600640 1 -- 172.21.15.69:0/1563485233 learned_addr learned my addr 172.21.15.69:0/1563485233 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb3fc00640 1 --2- 172.21.15.69:0/1563485233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddbe4ec00 0x556ddcc16000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb3f200640 1 --2- 172.21.15.69:0/1563485233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddbe4f800 0x556ddcc16580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb40600640 1 -- 172.21.15.69:0/1563485233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddbe4f800 msgr2=0x556ddcc16580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb40600640 1 --2- 172.21.15.69:0/1563485233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddbe4f800 0x556ddcc16580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb40600640 1 -- 172.21.15.69:0/1563485233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddbe4ec00 msgr2=0x556ddcc16000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb40600640 1 --2- 172.21.15.69:0/1563485233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddbe4ec00 0x556ddcc16000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb40600640 1 -- 172.21.15.69:0/1563485233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ddbe82780 con 0x556ddbe4fc00 2024-04-24T08:19:12.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb3fc00640 1 --2- 172.21.15.69:0/1563485233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddbe4ec00 0x556ddcc16000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:12.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.720+0000 7ffb40600640 1 --2- 172.21.15.69:0/1563485233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddbe4fc00 0x556ddcc16b00 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x556ddcc793b0 tx=0x556ddcbe8780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33dee36d8ccf7ea8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb3ca00640 1 -- 172.21.15.69:0/1563485233 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556ddbe74680 con 0x556ddbe4fc00 2024-04-24T08:19:12.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb3ca00640 1 -- 172.21.15.69:0/1563485233 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x556ddbe7a700 con 0x556ddbe4fc00 2024-04-24T08:19:12.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb3ca00640 1 -- 172.21.15.69:0/1563485233 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556ddbe74b60 con 0x556ddbe4fc00 2024-04-24T08:19:12.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb42521280 1 -- 172.21.15.69:0/1563485233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ddbe834a0 con 0x556ddbe4fc00 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb42521280 1 -- 172.21.15.69:0/1563485233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x556ddbe82000 con 0x556ddbe4fc00 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb42521280 4 waiting for MDS map... 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb3ca00640 1 -- 172.21.15.69:0/1563485233 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x556ddcd22600 con 0x556ddbe4fc00 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb3ca00640 1 -- 172.21.15.69:0/1563485233 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 84) ==== 2885+0+0 (secure 0 0 0) 0x556ddbea9180 con 0x556ddbe4fc00 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb42521280 4 Got MDS map 84 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb42521280 10 main: JournalTool::main 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.721+0000 7ffb42521280 4 main: JournalTool: connecting to RADOS... 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.722+0000 7ffb42521280 1 Processor -- start 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.722+0000 7ffb42521280 1 -- start start 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.723+0000 7ffb42521280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddcd25400 0x556ddcc17080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.723+0000 7ffb42521280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddcd25800 0x556ddcc17600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.723+0000 7ffb42521280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25c00 0x556ddcc17b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.723+0000 7ffb42521280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556ddcc32600 con 0x556ddcd25c00 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.723+0000 7ffb42521280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556ddcc32780 con 0x556ddcd25800 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.723+0000 7ffb42521280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556ddcc32900 con 0x556ddcd25400 2024-04-24T08:19:12.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.723+0000 7ffb3f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25c00 0x556ddcc17b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.723+0000 7ffb3f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25c00 0x556ddcc17b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42088/0 (socket says 172.21.15.69:42088) 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.723+0000 7ffb3f200640 1 -- 172.21.15.69:0/3017799522 learned_addr learned my addr 172.21.15.69:0/3017799522 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3fc00640 1 --2- 172.21.15.69:0/3017799522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddcd25800 0x556ddcc17600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3f200640 1 --2- 172.21.15.69:0/3017799522 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddcd25400 0x556ddcc17080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3f200640 1 -- 172.21.15.69:0/3017799522 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddcd25400 msgr2=0x556ddcc17080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3f200640 1 --2- 172.21.15.69:0/3017799522 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddcd25400 0x556ddcc17080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3f200640 1 -- 172.21.15.69:0/3017799522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddcd25800 msgr2=0x556ddcc17600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3f200640 1 --2- 172.21.15.69:0/3017799522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddcd25800 0x556ddcc17600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3f200640 1 -- 172.21.15.69:0/3017799522 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ddbe823c0 con 0x556ddcd25c00 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3f200640 1 --2- 172.21.15.69:0/3017799522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25c00 0x556ddcc17b80 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x556ddcd38690 tx=0x556ddcd44cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18478e23a29ee51 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3a200640 1 -- 172.21.15.69:0/3017799522 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556ddbe74680 con 0x556ddcd25c00 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3a200640 1 -- 172.21.15.69:0/3017799522 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x556ddbe7afc0 con 0x556ddcd25c00 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.724+0000 7ffb3a200640 1 -- 172.21.15.69:0/3017799522 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556ddbe74b60 con 0x556ddcd25c00 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.725+0000 7ffb42521280 1 -- 172.21.15.69:0/3017799522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25c00 msgr2=0x556ddcc17b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.725+0000 7ffb42521280 1 --2- 172.21.15.69:0/3017799522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25c00 0x556ddcc17b80 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x556ddcd38690 tx=0x556ddcd44cc0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.725+0000 7ffb42521280 1 -- 172.21.15.69:0/3017799522 shutdown_connections 2024-04-24T08:19:12.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.725+0000 7ffb42521280 1 --2- 172.21.15.69:0/3017799522 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25c00 0x556ddcc17b80 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.725+0000 7ffb42521280 1 --2- 172.21.15.69:0/3017799522 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddcd25800 0x556ddcc17600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.725+0000 7ffb42521280 1 --2- 172.21.15.69:0/3017799522 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddcd25400 0x556ddcc17080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.725+0000 7ffb42521280 1 -- 172.21.15.69:0/3017799522 >> 172.21.15.69:0/3017799522 conn(0x556ddcd25000 msgr2=0x556ddbf41000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.725+0000 7ffb42521280 1 -- 172.21.15.69:0/3017799522 shutdown_connections 2024-04-24T08:19:12.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.725+0000 7ffb42521280 1 -- 172.21.15.69:0/3017799522 wait complete. 2024-04-24T08:19:12.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.726+0000 7ffb42521280 1 Processor -- start 2024-04-24T08:19:12.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.726+0000 7ffb42521280 1 -- start start 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb42521280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddcd25400 0x556ddcc17080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb42521280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25800 0x556ddcc17600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb42521280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddcd25c00 0x556ddcc17b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb42521280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556ddcc33c80 con 0x556ddcd25800 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb42521280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556ddcc33b00 con 0x556ddcd25400 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb42521280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556ddcc33980 con 0x556ddcd25c00 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb3fc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25800 0x556ddcc17600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb3fc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25800 0x556ddcc17600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42098/0 (socket says 172.21.15.69:42098) 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb3fc00640 1 -- 172.21.15.69:0/1303942057 learned_addr learned my addr 172.21.15.69:0/1303942057 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb3fc00640 1 -- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddcd25c00 msgr2=0x556ddcc17b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb3fc00640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddcd25c00 0x556ddcc17b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb3fc00640 1 -- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddcd25400 msgr2=0x556ddcc17080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb3fc00640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddcd25400 0x556ddcc17080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.727+0000 7ffb3fc00640 1 -- 172.21.15.69:0/1303942057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ddbe82780 con 0x556ddcd25800 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.728+0000 7ffb3fc00640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25800 0x556ddcc17600 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x556ddcd39290 tx=0x556ddcbe8a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.728+0000 7ffb38400640 1 -- 172.21.15.69:0/1303942057 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556ddbe749c0 con 0x556ddcd25800 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.728+0000 7ffb38400640 1 -- 172.21.15.69:0/1303942057 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x556ddbe7afc0 con 0x556ddcd25800 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.728+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556ddbe834a0 con 0x556ddcd25800 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.728+0000 7ffb38400640 1 -- 172.21.15.69:0/1303942057 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556ddbe74680 con 0x556ddcd25800 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.728+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ddbe82960 con 0x556ddcd25800 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.728+0000 7ffb42521280 4 main: JournalTool: resolving pool 14 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.730+0000 7ffb38400640 1 -- 172.21.15.69:0/1303942057 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x556ddcda0c00 con 0x556ddcd25800 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.730+0000 7ffb38400640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556ddcd9cc00 0x556ddcc18100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.730+0000 7ffb38400640 1 -- 172.21.15.69:0/1303942057 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x556ddcd9ec00 con 0x556ddcd25800 2024-04-24T08:19:12.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.730+0000 7ffb42521280 4 main: JournalTool: creating IoCtx.. 2024-04-24T08:19:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.730+0000 7ffb42521280 4 main: Executing for rank 0 2024-04-24T08:19:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.730+0000 7ffb42521280 4 Filtering by inode '1099511627777' 2024-04-24T08:19:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.731+0000 7ffb42521280 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x556ddcd9d000 0x556ddcc18680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.731+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:1 14.f 14:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98) -- 0x556ddcd9d400 con 0x556ddcd9d000 2024-04-24T08:19:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.731+0000 7ffb3f200640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556ddcd9cc00 0x556ddcc18100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.731+0000 7ffb3f200640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x556ddcd9d000 0x556ddcc18680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.732+0000 7ffb3f200640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556ddcd9cc00 0x556ddcc18100 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x556ddcd387e0 tx=0x556ddcd44c60 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.732+0000 7ffb3f200640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x556ddcd9d000 0x556ddcc18680 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.739+0000 7ffb3f200640 1 -- 172.21.15.69:0/1303942057 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x556ddbf48000 con 0x556ddcd9d000 2024-04-24T08:19:12.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.739+0000 7ffb42521280 4 Pointer 400.00000000 is readable 2024-04-24T08:19:12.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.739+0000 7ffb42521280 4 JournalScanner::scan: reading header object '200.00000000' 2024-04-24T08:19:12.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.739+0000 7ffb42521280 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x556ddcd9d800 0x556ddcc18c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.739+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:2 14.14 14:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98) -- 0x556ddcd9dc00 con 0x556ddcd9d800 2024-04-24T08:19:12.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.739+0000 7ffb40600640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x556ddcd9d800 0x556ddcc18c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.740+0000 7ffb40600640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x556ddcd9d800 0x556ddcc18c00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.742+0000 7ffb40600640 1 -- 172.21.15.69:0/1303942057 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv31 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x556ddbf48500 con 0x556ddcd9d800 2024-04-24T08:19:12.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.742+0000 7ffb42521280 10 Header 0x400000 0x405f0e 0x40628a 2024-04-24T08:19:12.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.742+0000 7ffb42521280 10 Starting journal scan from offset 0x405f0e 2024-04-24T08:19:12.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.742+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:3 14.14 14:2e2fa760:::200.00000001:head [read 24334~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98) -- 0x556ddcd96000 con 0x556ddcd9d800 2024-04-24T08:19:12.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.743+0000 7ffb40600640 1 -- 172.21.15.69:0/1303942057 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== osd_op_reply(3 200.00000001 [read 24334~892 out=892b] v0'0 uv28 ondisk = 0) ==== 156+0+892 (crc 0 0 0) 0x556ddbf48500 con 0x556ddcd9d800 2024-04-24T08:19:12.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.743+0000 7ffb42521280 4 Read 0x37c bytes from 200.00000001 gap=0 2024-04-24T08:19:12.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.743+0000 7ffb42521280 10 Parsing data, 0x37c bytes available 2024-04-24T08:19:12.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.743+0000 7ffb42521280 10 Attempting decode at 0x405f0e 2024-04-24T08:19:12.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.743+0000 7ffb42521280 10 Consumed 0x37c bytes 2024-04-24T08:19:12.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.743+0000 7ffb42521280 10 Valid entry at 0x405f0e 2024-04-24T08:19:12.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.743+0000 7ffb42521280 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x556ddcd96400 0x556ddcc19180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.743+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:4 14.1b 14:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98) -- 0x556ddcd96800 con 0x556ddcd96400 2024-04-24T08:19:12.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.744+0000 7ffb3fc00640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x556ddcd96400 0x556ddcc19180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.744+0000 7ffb3fc00640 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x556ddcd96400 0x556ddcc19180 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb3fc00640 1 -- 172.21.15.69:0/1303942057 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x556ddbf48780 con 0x556ddcd96400 2024-04-24T08:19:12.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 4 Reached end of journal objects 2024-04-24T08:19:12.749 INFO:teuthology.orchestra.run.smithi069.stdout:Events by type: 2024-04-24T08:19:12.749 INFO:teuthology.orchestra.run.smithi069.stdout:Errors: 0 2024-04-24T08:19:12.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 4 Scanned objects, 0 missing, 1 valid 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 4 Events scanned, 0 gaps 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 4 Found 1 valid events 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 4 Selected 0 events events for processing 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 4 0 events 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x556ddcd96400 msgr2=0x556ddcc19180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x556ddcd96400 0x556ddcc19180 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x556ddcd9d800 msgr2=0x556ddcc18c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x556ddcd9d800 0x556ddcc18c00 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x556ddcd9d000 msgr2=0x556ddcc18680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.747+0000 7ffb42521280 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x556ddcd9d000 0x556ddcc18680 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.748+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556ddcd9cc00 msgr2=0x556ddcc18100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.748+0000 7ffb42521280 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556ddcd9cc00 0x556ddcc18100 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x556ddcd387e0 tx=0x556ddcd44c60 comp rx=0 tx=0).stop 2024-04-24T08:19:12.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.748+0000 7ffb3fc00640 1 -- 172.21.15.69:0/1303942057 reap_dead start 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.748+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25800 msgr2=0x556ddcc17600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.748+0000 7ffb42521280 1 --2- 172.21.15.69:0/1303942057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddcd25800 0x556ddcc17600 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x556ddcd39290 tx=0x556ddcbe8a20 comp rx=0 tx=0).stop 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.748+0000 7ffb3fc00640 1 -- 172.21.15.69:0/1303942057 reap_dead start 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.748+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 shutdown_connections 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.748+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 >> 172.21.15.69:0/1303942057 conn(0x556ddcd25000 msgr2=0x556ddbf41000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.748+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 shutdown_connections 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.748+0000 7ffb42521280 1 -- 172.21.15.69:0/1303942057 wait complete. 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.749+0000 7ffb42521280 1 -- 172.21.15.69:0/1563485233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddbe4fc00 msgr2=0x556ddcc16b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.749+0000 7ffb42521280 1 --2- 172.21.15.69:0/1563485233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddbe4fc00 0x556ddcc16b00 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x556ddcc793b0 tx=0x556ddcbe8780 comp rx=0 tx=0).stop 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.749+0000 7ffb42521280 1 -- 172.21.15.69:0/1563485233 shutdown_connections 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.749+0000 7ffb42521280 1 --2- 172.21.15.69:0/1563485233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556ddbe4fc00 0x556ddcc16b00 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.749+0000 7ffb42521280 1 --2- 172.21.15.69:0/1563485233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556ddbe4f800 0x556ddcc16580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.749+0000 7ffb42521280 1 --2- 172.21.15.69:0/1563485233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556ddbe4ec00 0x556ddcc16000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.749+0000 7ffb42521280 1 -- 172.21.15.69:0/1563485233 >> 172.21.15.69:0/1563485233 conn(0x556ddbe4f000 msgr2=0x556ddbf40800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.749+0000 7ffb42521280 1 -- 172.21.15.69:0/1563485233 shutdown_connections 2024-04-24T08:19:12.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.749+0000 7ffb42521280 1 -- 172.21.15.69:0/1563485233 wait complete. 2024-04-24T08:19:12.758 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627777', 'summary'] in time 0:00:00.141085, result: Events by type: Errors: 0 2024-04-24T08:19:12.758 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627778 summary 2024-04-24T08:19:12.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.850+0000 7f4bb5f1c280 1 -- start start 2024-04-24T08:19:12.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.850+0000 7f4bb5f1c280 1 Processor -- start 2024-04-24T08:19:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb5f1c280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51b448c00 0x55b51b5b8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb5f1c280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51b449800 0x55b51b5b8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb5f1c280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51b449c00 0x55b51b5b8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb5f1c280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b51c1b4180 con 0x55b51b449c00 2024-04-24T08:19:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb5f1c280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b51c1b4300 con 0x55b51b448c00 2024-04-24T08:19:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb5f1c280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55b51c1b4480 con 0x55b51b449800 2024-04-24T08:19:12.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb4000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51b449c00 0x55b51b5b8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb4000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51b449c00 0x55b51b5b8b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42116/0 (socket says 172.21.15.69:42116) 2024-04-24T08:19:12.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb4000640 1 -- 172.21.15.69:0/3361767810 learned_addr learned my addr 172.21.15.69:0/3361767810 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb4000640 1 -- 172.21.15.69:0/3361767810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51b449800 msgr2=0x55b51b5b8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb4000640 1 --2- 172.21.15.69:0/3361767810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51b449800 0x55b51b5b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb4000640 1 -- 172.21.15.69:0/3361767810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51b448c00 msgr2=0x55b51b5b8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb4000640 1 --2- 172.21.15.69:0/3361767810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51b448c00 0x55b51b5b8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.851+0000 7f4bb4000640 1 -- 172.21.15.69:0/3361767810 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b51b47c780 con 0x55b51b449c00 2024-04-24T08:19:12.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb4000640 1 --2- 172.21.15.69:0/3361767810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51b449c00 0x55b51b5b8b00 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x55b51c2732f0 tx=0x55b51c1e2660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51b00286ae6304a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb0400640 1 -- 172.21.15.69:0/3361767810 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55b51b46e9c0 con 0x55b51b449c00 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb0400640 1 -- 172.21.15.69:0/3361767810 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55b51b474700 con 0x55b51b449c00 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/3361767810 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b51b47c3c0 con 0x55b51b449c00 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/3361767810 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55b51b47d680 con 0x55b51b449c00 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb5f1c280 4 waiting for MDS map... 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb0400640 1 -- 172.21.15.69:0/3361767810 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55b51b46e680 con 0x55b51b449c00 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb0400640 1 -- 172.21.15.69:0/3361767810 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x55b51c22a200 con 0x55b51b449c00 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb0400640 1 -- 172.21.15.69:0/3361767810 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 84) ==== 2885+0+0 (secure 0 0 0) 0x55b51b4a3180 con 0x55b51b449c00 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb5f1c280 4 Got MDS map 84 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.852+0000 7f4bb5f1c280 10 main: JournalTool::main 2024-04-24T08:19:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.853+0000 7f4bb5f1c280 4 main: JournalTool: connecting to RADOS... 2024-04-24T08:19:12.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.853+0000 7f4bb5f1c280 1 Processor -- start 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.853+0000 7f4bb5f1c280 1 -- start start 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.854+0000 7f4bb5f1c280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.854+0000 7f4bb5f1c280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51c36e400 0x55b51b5b9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.854+0000 7f4bb5f1c280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51c36e800 0x55b51b5b9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.854+0000 7f4bb5f1c280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b51c1b4600 con 0x55b51c36e000 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.854+0000 7f4bb5f1c280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b51c1b4780 con 0x55b51c36e800 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.854+0000 7f4bb5f1c280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55b51c1b4900 con 0x55b51c36e400 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.854+0000 7f4bb2c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4bb2c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42124/0 (socket says 172.21.15.69:42124) 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4bb2c00640 1 -- 172.21.15.69:0/2350246010 learned_addr learned my addr 172.21.15.69:0/2350246010 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4bb3600640 1 --2- 172.21.15.69:0/2350246010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51c36e400 0x55b51b5b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4bb2c00640 1 -- 172.21.15.69:0/2350246010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51c36e400 msgr2=0x55b51b5b9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4bb2c00640 1 --2- 172.21.15.69:0/2350246010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51c36e400 0x55b51b5b9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4bb2c00640 1 -- 172.21.15.69:0/2350246010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51c36e800 msgr2=0x55b51b5b9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4bb2c00640 1 --2- 172.21.15.69:0/2350246010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51c36e800 0x55b51b5b9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4bb2c00640 1 -- 172.21.15.69:0/2350246010 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b51b47d4a0 con 0x55b51c36e000 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4bb2c00640 1 --2- 172.21.15.69:0/2350246010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x55b51c31a930 tx=0x55b51c31ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37079218c7f3c1f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4badc00640 1 -- 172.21.15.69:0/2350246010 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55b51b46e9c0 con 0x55b51c36e000 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4badc00640 1 -- 172.21.15.69:0/2350246010 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55b51b474e00 con 0x55b51c36e000 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.855+0000 7f4badc00640 1 -- 172.21.15.69:0/2350246010 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55b51b46e680 con 0x55b51c36e000 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.856+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/2350246010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 msgr2=0x55b51b5b9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.856+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/2350246010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x55b51c31a930 tx=0x55b51c31ed20 comp rx=0 tx=0).stop 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.856+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/2350246010 shutdown_connections 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.856+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/2350246010 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51c36e800 0x55b51b5b9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.856+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/2350246010 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51c36e400 0x55b51b5b9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.856+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/2350246010 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.856+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/2350246010 >> 172.21.15.69:0/2350246010 conn(0x55b51c1e1c00 msgr2=0x55b51c1cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.856+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/2350246010 shutdown_connections 2024-04-24T08:19:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.856+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/2350246010 wait complete. 2024-04-24T08:19:12.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.857+0000 7f4bb5f1c280 1 Processor -- start 2024-04-24T08:19:12.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.857+0000 7f4bb5f1c280 1 -- start start 2024-04-24T08:19:12.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb5f1c280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb5f1c280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51c36e400 0x55b51b5b9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb5f1c280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51c36e800 0x55b51b5b9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb5f1c280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b51c1b5c80 con 0x55b51c36e000 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb5f1c280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b51c1b5b00 con 0x55b51c36e400 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb5f1c280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55b51c1b5980 con 0x55b51c36e800 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb2c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb2c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42132/0 (socket says 172.21.15.69:42132) 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb2c00640 1 -- 172.21.15.69:0/96994917 learned_addr learned my addr 172.21.15.69:0/96994917 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb2c00640 1 -- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51c36e800 msgr2=0x55b51b5b9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb2c00640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51c36e800 0x55b51b5b9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb2c00640 1 -- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51c36e400 msgr2=0x55b51b5b9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb2c00640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51c36e400 0x55b51b5b9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.858+0000 7f4bb2c00640 1 -- 172.21.15.69:0/96994917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b51b47c000 con 0x55b51c36e000 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.859+0000 7f4bb2c00640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x55b51c31b5f0 tx=0x55b51c31ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.859+0000 7f4babe00640 1 -- 172.21.15.69:0/96994917 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55b51b46e4e0 con 0x55b51c36e000 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.859+0000 7f4babe00640 1 -- 172.21.15.69:0/96994917 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55b51b474c40 con 0x55b51c36e000 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.859+0000 7f4babe00640 1 -- 172.21.15.69:0/96994917 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55b51b46e9c0 con 0x55b51c36e000 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.859+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55b51b47c960 con 0x55b51c36e000 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.859+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b51b47d860 con 0x55b51c36e000 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.859+0000 7f4bb5f1c280 4 main: JournalTool: resolving pool 14 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.861+0000 7f4babe00640 1 -- 172.21.15.69:0/96994917 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55b51c37ac00 con 0x55b51c36e000 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.861+0000 7f4babe00640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55b51b7b0000 0x55b51b5ba100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.861+0000 7f4babe00640 1 -- 172.21.15.69:0/96994917 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x55b51c378a00 con 0x55b51c36e000 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.861+0000 7f4bb5f1c280 4 main: JournalTool: creating IoCtx.. 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.861+0000 7f4bb5f1c280 4 main: Executing for rank 0 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.861+0000 7f4bb5f1c280 4 Filtering by inode '1099511627778' 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.862+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55b51c36ec00 0x55b51b5ba680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.862+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:1 14.f 14:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98) -- 0x55b51c36f000 con 0x55b51c36ec00 2024-04-24T08:19:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.862+0000 7f4bb3600640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55b51b7b0000 0x55b51b5ba100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.862+0000 7f4bb2c00640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55b51c36ec00 0x55b51b5ba680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.863+0000 7f4bb2c00640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55b51c36ec00 0x55b51b5ba680 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.864+0000 7f4bb3600640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55b51b7b0000 0x55b51b5ba100 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x55b51c1da7e0 tx=0x55b51c31f080 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.864+0000 7f4bb2c00640 1 -- 172.21.15.69:0/96994917 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) ==== 156+0+22 (crc 0 0 0) 0x55b51c1d2000 con 0x55b51c36ec00 2024-04-24T08:19:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.864+0000 7f4bb5f1c280 4 Pointer 400.00000000 is readable 2024-04-24T08:19:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.864+0000 7f4bb5f1c280 4 JournalScanner::scan: reading header object '200.00000000' 2024-04-24T08:19:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.864+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55b51b7b1000 0x55b51b5bac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.864+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:2 14.14 14:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98) -- 0x55b51b7b1400 con 0x55b51b7b1000 2024-04-24T08:19:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.865+0000 7f4bb4000640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55b51b7b1000 0x55b51b5bac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.866+0000 7f4bb4000640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55b51b7b1000 0x55b51b5bac00 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.867+0000 7f4bb4000640 1 -- 172.21.15.69:0/96994917 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv31 ondisk = 0) ==== 156+0+90 (crc 0 0 0) 0x55b51c1d2500 con 0x55b51b7b1000 2024-04-24T08:19:12.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.867+0000 7f4bb5f1c280 10 Header 0x400000 0x405f0e 0x40628a 2024-04-24T08:19:12.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.867+0000 7f4bb5f1c280 10 Starting journal scan from offset 0x405f0e 2024-04-24T08:19:12.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.867+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:3 14.14 14:2e2fa760:::200.00000001:head [read 24334~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98) -- 0x55b51b7b1800 con 0x55b51b7b1000 2024-04-24T08:19:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.868+0000 7f4bb4000640 1 -- 172.21.15.69:0/96994917 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== osd_op_reply(3 200.00000001 [read 24334~892 out=892b] v0'0 uv28 ondisk = 0) ==== 156+0+892 (crc 0 0 0) 0x55b51c1d2500 con 0x55b51b7b1000 2024-04-24T08:19:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.868+0000 7f4bb5f1c280 4 Read 0x37c bytes from 200.00000001 gap=0 2024-04-24T08:19:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.868+0000 7f4bb5f1c280 10 Parsing data, 0x37c bytes available 2024-04-24T08:19:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.868+0000 7f4bb5f1c280 10 Attempting decode at 0x405f0e 2024-04-24T08:19:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.868+0000 7f4bb5f1c280 10 Consumed 0x37c bytes 2024-04-24T08:19:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.868+0000 7f4bb5f1c280 10 Valid entry at 0x405f0e 2024-04-24T08:19:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.868+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55b51b7b1c00 0x55b51b5bb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.868+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:4 14.1b 14:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e98) -- 0x55b51b7d2000 con 0x55b51b7b1c00 2024-04-24T08:19:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.868+0000 7f4bb3600640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55b51b7b1c00 0x55b51b5bb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb3600640 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55b51b7b1c00 0x55b51b5bb180 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb3600640 1 -- 172.21.15.69:0/96994917 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55b51c1d2780 con 0x55b51b7b1c00 2024-04-24T08:19:12.872 INFO:teuthology.orchestra.run.smithi069.stdout:Events by type: 2024-04-24T08:19:12.872 INFO:teuthology.orchestra.run.smithi069.stdout:Errors: 0 2024-04-24T08:19:12.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 4 Reached end of journal objects 2024-04-24T08:19:12.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 4 Scanned objects, 0 missing, 1 valid 2024-04-24T08:19:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 4 Events scanned, 0 gaps 2024-04-24T08:19:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 4 Found 1 valid events 2024-04-24T08:19:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 4 Selected 0 events events for processing 2024-04-24T08:19:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 4 0 events 2024-04-24T08:19:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55b51b7b1c00 msgr2=0x55b51b5bb180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55b51b7b1c00 0x55b51b5bb180 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55b51b7b1000 msgr2=0x55b51b5bac00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55b51b7b1000 0x55b51b5bac00 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55b51c36ec00 msgr2=0x55b51b5ba680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.869+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55b51c36ec00 0x55b51b5ba680 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb3600640 1 -- 172.21.15.69:0/96994917 reap_dead start 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55b51b7b0000 msgr2=0x55b51b5ba100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55b51b7b0000 0x55b51b5ba100 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x55b51c1da7e0 tx=0x55b51c31f080 comp rx=0 tx=0).stop 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 msgr2=0x55b51b5b9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x55b51c31b5f0 tx=0x55b51c31ec00 comp rx=0 tx=0).stop 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 shutdown_connections 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55b51b7b0000 0x55b51b5ba100 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/96994917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51c36e000 0x55b51b5b9080 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 >> 172.21.15.69:0/96994917 conn(0x55b51c1e1c00 msgr2=0x55b51c1cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 shutdown_connections 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/96994917 wait complete. 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/3361767810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51b449c00 msgr2=0x55b51b5b8b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/3361767810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51b449c00 0x55b51b5b8b00 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x55b51c2732f0 tx=0x55b51c1e2660 comp rx=0 tx=0).stop 2024-04-24T08:19:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/3361767810 shutdown_connections 2024-04-24T08:19:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/3361767810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b51b449c00 0x55b51b5b8b00 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/3361767810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b51b449800 0x55b51b5b8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 --2- 172.21.15.69:0/3361767810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b51b448c00 0x55b51b5b8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.870+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/3361767810 >> 172.21.15.69:0/3361767810 conn(0x55b51b449000 msgr2=0x55b51c1ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.871+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/3361767810 shutdown_connections 2024-04-24T08:19:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.871+0000 7f4bb5f1c280 1 -- 172.21.15.69:0/3361767810 wait complete. 2024-04-24T08:19:12.877 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627778', 'summary'] in time 0:00:00.118495, result: Events by type: Errors: 0 2024-04-24T08:19:12.877 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds0_inotable - 2024-04-24T08:19:12.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.974+0000 7fb02294df40 1 Processor -- start 2024-04-24T08:19:12.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.974+0000 7fb02294df40 1 -- start start 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.975+0000 7fb02294df40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdca5f5a0 0x560bdcb2cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.975+0000 7fb02294df40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdcaaa940 0x560bdca8c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.975+0000 7fb02294df40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560bdcb30f50 0x560bdca8c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.975+0000 7fb02294df40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560bdca06cc0 con 0x560bdca5f5a0 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.975+0000 7fb02294df40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x560bdca0ad00 con 0x560bdcaaa940 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.975+0000 7fb02294df40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x560bdca90180 con 0x560bdcb30f50 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb01fc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdca5f5a0 0x560bdcb2cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb01fc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdca5f5a0 0x560bdcb2cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42142/0 (socket says 172.21.15.69:42142) 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb01fc00640 1 -- 172.21.15.69:0/3486063687 learned_addr learned my addr 172.21.15.69:0/3486063687 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb01f200640 1 --2- 172.21.15.69:0/3486063687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdcaaa940 0x560bdca8c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb01fc00640 1 -- 172.21.15.69:0/3486063687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560bdcb30f50 msgr2=0x560bdca8c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb020600640 1 --2- 172.21.15.69:0/3486063687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560bdcb30f50 0x560bdca8c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb01fc00640 1 --2- 172.21.15.69:0/3486063687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560bdcb30f50 0x560bdca8c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb01fc00640 1 -- 172.21.15.69:0/3486063687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdcaaa940 msgr2=0x560bdca8c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb01fc00640 1 --2- 172.21.15.69:0/3486063687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdcaaa940 0x560bdca8c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb01fc00640 1 -- 172.21.15.69:0/3486063687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bdca929c0 con 0x560bdca5f5a0 2024-04-24T08:19:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.976+0000 7fb01f200640 1 --2- 172.21.15.69:0/3486063687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdcaaa940 0x560bdca8c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb01fc00640 1 --2- 172.21.15.69:0/3486063687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdca5f5a0 0x560bdcb2cd20 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fb0100017b0 tx=0x7fb010001c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdce38bee4a8f418 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb01de00640 1 -- 172.21.15.69:0/3486063687 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010016070 con 0x560bdca5f5a0 2024-04-24T08:19:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb01de00640 1 -- 172.21.15.69:0/3486063687 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb010004520 con 0x560bdca5f5a0 2024-04-24T08:19:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb01de00640 1 -- 172.21.15.69:0/3486063687 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb010004950 con 0x560bdca5f5a0 2024-04-24T08:19:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb02294df40 1 -- 172.21.15.69:0/3486063687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdca5f5a0 msgr2=0x560bdcb2cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb02294df40 1 --2- 172.21.15.69:0/3486063687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdca5f5a0 0x560bdcb2cd20 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fb0100017b0 tx=0x7fb010001c70 comp rx=0 tx=0).stop 2024-04-24T08:19:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb02294df40 1 -- 172.21.15.69:0/3486063687 shutdown_connections 2024-04-24T08:19:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb02294df40 1 --2- 172.21.15.69:0/3486063687 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560bdcb30f50 0x560bdca8c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb02294df40 1 --2- 172.21.15.69:0/3486063687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdcaaa940 0x560bdca8c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb02294df40 1 --2- 172.21.15.69:0/3486063687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdca5f5a0 0x560bdcb2cd20 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb02294df40 1 -- 172.21.15.69:0/3486063687 >> 172.21.15.69:0/3486063687 conn(0x560bdca8eb50 msgr2=0x560bdca8e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.977+0000 7fb02294df40 1 -- 172.21.15.69:0/3486063687 shutdown_connections 2024-04-24T08:19:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.978+0000 7fb02294df40 1 -- 172.21.15.69:0/3486063687 wait complete. 2024-04-24T08:19:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.978+0000 7fb02294df40 1 Processor -- start 2024-04-24T08:19:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.978+0000 7fb02294df40 1 -- start start 2024-04-24T08:19:12.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.979+0000 7fb02294df40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdca5f5a0 0x560bdcb24920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.979+0000 7fb02294df40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdcaaa940 0x560bdcb2cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.979+0000 7fb02294df40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560bdcb30f50 0x560bdcb2d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.979+0000 7fb02294df40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560bdca92c60 con 0x560bdcaaa940 2024-04-24T08:19:12.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.979+0000 7fb02294df40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x560bdca8df50 con 0x560bdca5f5a0 2024-04-24T08:19:12.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.979+0000 7fb02294df40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x560bdca8d2d0 con 0x560bdcb30f50 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb01f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdcaaa940 0x560bdcb2cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb01f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdcaaa940 0x560bdcb2cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42146/0 (socket says 172.21.15.69:42146) 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb01f200640 1 -- 172.21.15.69:0/683662832 learned_addr learned my addr 172.21.15.69:0/683662832 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb01fc00640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdca5f5a0 0x560bdcb24920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb020600640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560bdcb30f50 0x560bdcb2d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb01f200640 1 -- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560bdcb30f50 msgr2=0x560bdcb2d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb01f200640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x560bdcb30f50 0x560bdcb2d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb01f200640 1 -- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdca5f5a0 msgr2=0x560bdcb24920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb01f200640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdca5f5a0 0x560bdcb24920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb01f200640 1 -- 172.21.15.69:0/683662832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560bdcb2c330 con 0x560bdcaaa940 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.980+0000 7fb01fc00640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560bdca5f5a0 0x560bdcb24920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.981+0000 7fb01f200640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdcaaa940 0x560bdcb2cbc0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fb00c058040 tx=0x7fb00c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.981+0000 7fb01ca00640 1 -- 172.21.15.69:0/683662832 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c065020 con 0x560bdcaaa940 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.981+0000 7fb01ca00640 1 -- 172.21.15.69:0/683662832 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00c060470 con 0x560bdcaaa940 2024-04-24T08:19:12.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.981+0000 7fb01ca00640 1 -- 172.21.15.69:0/683662832 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c05a040 con 0x560bdcaaa940 2024-04-24T08:19:12.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.981+0000 7fb02294df40 1 -- 172.21.15.69:0/683662832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560bdcb2dda0 con 0x560bdcaaa940 2024-04-24T08:19:12.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.981+0000 7fb02294df40 1 -- 172.21.15.69:0/683662832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560bdcb38680 con 0x560bdcaaa940 2024-04-24T08:19:12.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.983+0000 7fb01ca00640 1 -- 172.21.15.69:0/683662832 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb00c068020 con 0x560bdcaaa940 2024-04-24T08:19:12.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.984+0000 7fb01ca00640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb00006b2d0 0x7fb00006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.984+0000 7fb01ca00640 1 -- 172.21.15.69:0/683662832 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fb00c0ce570 con 0x560bdcaaa940 2024-04-24T08:19:12.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.984+0000 7fb02294df40 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x560bdcb30570 0x560bdcfd7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:12.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.984+0000 7fb02294df40 1 -- 172.21.15.69:0/683662832 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:1 14.13 14:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98) -- 0x560bdcbccda0 con 0x560bdcb30570 2024-04-24T08:19:12.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.985+0000 7fb01fc00640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb00006b2d0 0x7fb00006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.985+0000 7fb020600640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x560bdcb30570 0x560bdcfd7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:12.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.985+0000 7fb020600640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x560bdcb30570 0x560bdcfd7da0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.986+0000 7fb01fc00640 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb00006b2d0 0x7fb00006d790 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb0100017b0 tx=0x7fb010009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:12.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.991+0000 7fb020600640 1 -- 172.21.15.69:0/683662832 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~34] v98'5 uv5 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7fb01800e330 con 0x560bdcb30570 2024-04-24T08:19:12.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.991+0000 7fb02294df40 1 -- 172.21.15.69:0/683662832 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x560bdcb30570 msgr2=0x560bdcfd7da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.991+0000 7fb02294df40 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x560bdcb30570 0x560bdcfd7da0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.992+0000 7fb02294df40 1 -- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb00006b2d0 msgr2=0x7fb00006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.992+0000 7fb02294df40 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb00006b2d0 0x7fb00006d790 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb0100017b0 tx=0x7fb010009000 comp rx=0 tx=0).stop 2024-04-24T08:19:12.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.992+0000 7fb02294df40 1 -- 172.21.15.69:0/683662832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdcaaa940 msgr2=0x560bdcb2cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:12.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.992+0000 7fb02294df40 1 --2- 172.21.15.69:0/683662832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560bdcaaa940 0x560bdcb2cbc0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fb00c058040 tx=0x7fb00c002fb0 comp rx=0 tx=0).stop 2024-04-24T08:19:12.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.992+0000 7fb020600640 1 -- 172.21.15.69:0/683662832 reap_dead start 2024-04-24T08:19:12.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.992+0000 7fb02294df40 1 -- 172.21.15.69:0/683662832 shutdown_connections 2024-04-24T08:19:12.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.992+0000 7fb02294df40 1 -- 172.21.15.69:0/683662832 >> 172.21.15.69:0/683662832 conn(0x560bdca8eb50 msgr2=0x560bdcb26180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:12.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.992+0000 7fb02294df40 1 -- 172.21.15.69:0/683662832 shutdown_connections 2024-04-24T08:19:12.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:12.992+0000 7fb02294df40 1 -- 172.21.15.69:0/683662832 wait complete. 2024-04-24T08:19:12.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-24T08:19:13.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.188+0000 7f9a8d600640 1 Processor -- start 2024-04-24T08:19:13.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.188+0000 7f9a8d600640 1 -- start start 2024-04-24T08:19:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.190+0000 7f9a8d600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9a88071d70 0x7f9a88072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.190+0000 7f9a8d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a8806e000 0x7f9a8806e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.190+0000 7f9a8d600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a88072690 con 0x7f9a8815edf0 2024-04-24T08:19:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.190+0000 7f9a8d600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9a880d0a00 con 0x7f9a88071d70 2024-04-24T08:19:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.190+0000 7f9a8d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8815f960 con 0x7f9a8806e000 2024-04-24T08:19:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.190+0000 7f9a87e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9a88071d70 0x7f9a88072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.190+0000 7f9a7f400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9a8815edf0 0x7f9a8815f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37662/0 (socket says 172.21.15.69:37662) 2024-04-24T08:19:13.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.190+0000 7f9a7f400640 1 -- 172.21.15.69:0/3784426947 learned_addr learned my addr 172.21.15.69:0/3784426947 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.191+0000 7f9a8cc00640 1 --2- 172.21.15.69:0/3784426947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a8806e000 0x7f9a8806e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.191+0000 7f9a87e00640 1 -- 172.21.15.69:0/3784426947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a8806e000 msgr2=0x7f9a8806e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.191+0000 7f9a87e00640 1 --2- 172.21.15.69:0/3784426947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a8806e000 0x7f9a8806e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.191+0000 7f9a87e00640 1 -- 172.21.15.69:0/3784426947 >> v1:172.21.15.69:6789/0 conn(0x7f9a8815edf0 legacy=0x7f9a8815f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.191+0000 7f9a87e00640 1 -- 172.21.15.69:0/3784426947 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8815fbb0 con 0x7f9a88071d70 2024-04-24T08:19:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.191+0000 7f9a87400640 1 -- 172.21.15.69:0/3784426947 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323853139 0 0) 0x7f9a88072690 con 0x7f9a8815edf0 2024-04-24T08:19:13.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.191+0000 7f9a8cc00640 1 --2- 172.21.15.69:0/3784426947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a8806e000 0x7f9a8806e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:13.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.192+0000 7f9a87e00640 1 --2- 172.21.15.69:0/3784426947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a88071d70 0x7f9a88072150 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f9a780097a0 tx=0x7f9a7802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87c0d9c0ed631a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:13.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.193+0000 7f9a87400640 1 -- 172.21.15.69:0/3784426947 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78037070 con 0x7f9a88071d70 2024-04-24T08:19:13.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.193+0000 7f9a87400640 1 -- 172.21.15.69:0/3784426947 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a78032440 con 0x7f9a88071d70 2024-04-24T08:19:13.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.193+0000 7f9a87400640 1 -- 172.21.15.69:0/3784426947 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78032810 con 0x7f9a88071d70 2024-04-24T08:19:13.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.193+0000 7f9a8d600640 1 -- 172.21.15.69:0/3784426947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a88071d70 msgr2=0x7f9a88072150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.193+0000 7f9a8d600640 1 --2- 172.21.15.69:0/3784426947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a88071d70 0x7f9a88072150 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f9a780097a0 tx=0x7f9a7802a180 comp rx=0 tx=0).stop 2024-04-24T08:19:13.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.193+0000 7f9a8d600640 1 -- 172.21.15.69:0/3784426947 shutdown_connections 2024-04-24T08:19:13.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.193+0000 7f9a8d600640 1 --2- 172.21.15.69:0/3784426947 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a8806e000 0x7f9a8806e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.193+0000 7f9a8d600640 1 --2- 172.21.15.69:0/3784426947 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a88071d70 0x7f9a88072150 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.193+0000 7f9a8d600640 1 -- 172.21.15.69:0/3784426947 >> 172.21.15.69:0/3784426947 conn(0x7f9a88067b80 msgr2=0x7f9a8806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:13.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.193+0000 7f9a8d600640 1 -- 172.21.15.69:0/3784426947 shutdown_connections 2024-04-24T08:19:13.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.194+0000 7f9a8d600640 1 -- 172.21.15.69:0/3784426947 wait complete. 2024-04-24T08:19:13.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.195+0000 7f9a8d600640 1 Processor -- start 2024-04-24T08:19:13.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.195+0000 7f9a8d600640 1 -- start start 2024-04-24T08:19:13.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.196+0000 7f9a8d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a8806e000 0x7f9a881f49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.196+0000 7f9a8d600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a88071d70 0x7f9a88138bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.196+0000 7f9a8d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a8815edf0 0x7f9a8813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.196+0000 7f9a8d600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9a880d0690 con 0x7f9a8806e000 2024-04-24T08:19:13.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.196+0000 7f9a8d600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9a880d03b0 con 0x7f9a88071d70 2024-04-24T08:19:13.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.196+0000 7f9a8d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a880d0e90 con 0x7f9a8815edf0 2024-04-24T08:19:13.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.197+0000 7f9a87e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a8806e000 0x7f9a881f49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.197+0000 7f9a87e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a8806e000 0x7f9a881f49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42158/0 (socket says 172.21.15.69:42158) 2024-04-24T08:19:13.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.197+0000 7f9a87e00640 1 -- 172.21.15.69:0/1169855294 learned_addr learned my addr 172.21.15.69:0/1169855294 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.197+0000 7f9a7f400640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a88071d70 0x7f9a88138bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.197+0000 7f9a87e00640 1 -- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a8815edf0 msgr2=0x7f9a8813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.197+0000 7f9a87e00640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a8815edf0 0x7f9a8813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.197+0000 7f9a87e00640 1 -- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a88071d70 msgr2=0x7f9a88138bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.197+0000 7f9a87e00640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a88071d70 0x7f9a88138bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.197+0000 7f9a87e00640 1 -- 172.21.15.69:0/1169855294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8813d0b0 con 0x7f9a8806e000 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.197+0000 7f9a87e00640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a8806e000 0x7f9a881f49a0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f9a780096f0 tx=0x7f9a780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.198+0000 7f9a84c00640 1 -- 172.21.15.69:0/1169855294 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78037040 con 0x7f9a8806e000 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.198+0000 7f9a84c00640 1 -- 172.21.15.69:0/1169855294 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a780307b0 con 0x7f9a8806e000 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.198+0000 7f9a8d600640 1 -- 172.21.15.69:0/1169855294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8814f880 con 0x7f9a8806e000 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.198+0000 7f9a84c00640 1 -- 172.21.15.69:0/1169855294 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78030b80 con 0x7f9a8806e000 2024-04-24T08:19:13.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.198+0000 7f9a8d600640 1 -- 172.21.15.69:0/1169855294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8814fe40 con 0x7f9a8806e000 2024-04-24T08:19:13.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.199+0000 7f9a8d600640 1 -- 172.21.15.69:0/1169855294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a88075d50 con 0x7f9a8806e000 2024-04-24T08:19:13.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.200+0000 7f9a84c00640 1 -- 172.21.15.69:0/1169855294 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a78044020 con 0x7f9a8806e000 2024-04-24T08:19:13.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.201+0000 7f9a84c00640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a6406c8f0 0x7f9a6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.201+0000 7f9a84c00640 1 -- 172.21.15.69:0/1169855294 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7f9a780abff0 con 0x7f9a8806e000 2024-04-24T08:19:13.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.202+0000 7f9a7f400640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a6406c8f0 0x7f9a6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.203+0000 7f9a7f400640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a6406c8f0 0x7f9a6406edb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9a88143660 tx=0x7f9a70009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:13.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.205+0000 7f9a84c00640 1 -- 172.21.15.69:0/1169855294 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a7804b020 con 0x7f9a8806e000 2024-04-24T08:19:13.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.447+0000 7f9a8d600640 1 -- 172.21.15.69:0/1169855294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f9a8815e6f0 con 0x7f9a8806e000 2024-04-24T08:19:13.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.677+0000 7f9a84c00640 1 -- 172.21.15.69:0/1169855294 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v85) ==== 162+0+0 (secure 0 0 0) 0x7f9a8813d0b0 con 0x7f9a8806e000 2024-04-24T08:19:13.679 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-24T08:19:13.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.681+0000 7f9a8d600640 1 -- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a6406c8f0 msgr2=0x7f9a6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.681+0000 7f9a8d600640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a6406c8f0 0x7f9a6406edb0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f9a88143660 tx=0x7f9a70009040 comp rx=0 tx=0).stop 2024-04-24T08:19:13.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.681+0000 7f9a8d600640 1 -- 172.21.15.69:0/1169855294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a8806e000 msgr2=0x7f9a881f49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.681+0000 7f9a8d600640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a8806e000 0x7f9a881f49a0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f9a780096f0 tx=0x7f9a780046c0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.681+0000 7f9a8d600640 1 -- 172.21.15.69:0/1169855294 shutdown_connections 2024-04-24T08:19:13.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.681+0000 7f9a8d600640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a6406c8f0 0x7f9a6406edb0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.681+0000 7f9a8d600640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a8815edf0 0x7f9a8813ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.681+0000 7f9a8d600640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a88071d70 0x7f9a88138bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.681+0000 7f9a8d600640 1 --2- 172.21.15.69:0/1169855294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a8806e000 0x7f9a881f49a0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.681+0000 7f9a8d600640 1 -- 172.21.15.69:0/1169855294 >> 172.21.15.69:0/1169855294 conn(0x7f9a88067b80 msgr2=0x7f9a8806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:13.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.682+0000 7f9a8d600640 1 -- 172.21.15.69:0/1169855294 shutdown_connections 2024-04-24T08:19:13.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.682+0000 7f9a8d600640 1 -- 172.21.15.69:0/1169855294 wait complete. 2024-04-24T08:19:13.708 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:19:13.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.885+0000 7fc450a00640 1 Processor -- start 2024-04-24T08:19:13.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.885+0000 7fc450a00640 1 -- start start 2024-04-24T08:19:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.886+0000 7fc450a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c071be0 0x7fc44c144090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.886+0000 7fc450a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc44c0b3190 con 0x7fc44c15ab10 2024-04-24T08:19:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.886+0000 7fc450a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc44c0ad590 con 0x7fc44c06eb30 2024-04-24T08:19:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.886+0000 7fc450a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc44c0b3400 con 0x7fc44c071be0 2024-04-24T08:19:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.887+0000 7fc44a800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc44c15ab10 0x7fc44c06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37664/0 (socket says 172.21.15.69:37664) 2024-04-24T08:19:13.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.887+0000 7fc44a800640 1 -- 172.21.15.69:0/1681028707 learned_addr learned my addr 172.21.15.69:0/1681028707 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:13.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.887+0000 7fc44b200640 1 --2- 172.21.15.69:0/1681028707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c071be0 0x7fc44c144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.887+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 850352459 0 0) 0x7fc44c0b3190 con 0x7fc44c15ab10 2024-04-24T08:19:13.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.887+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc430005730 con 0x7fc44c15ab10 2024-04-24T08:19:13.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.887+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2065412205 0 0) 0x7fc430005730 con 0x7fc44c15ab10 2024-04-24T08:19:13.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.887+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c071be0 msgr2=0x7fc44c144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:13.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.887+0000 7fc449400640 1 --2- 172.21.15.69:0/1681028707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c071be0 0x7fc44c144090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.887+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 >> v1:172.21.15.121:6789/0 conn(0x7fc44c06eb30 legacy=0x7fc44c0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.887+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44c0b36e0 con 0x7fc44c15ab10 2024-04-24T08:19:13.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fc43c001670 con 0x7fc44c15ab10 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 >> v1:172.21.15.69:6789/0 conn(0x7fc44c15ab10 legacy=0x7fc44c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc449400640 1 --2- 172.21.15.69:0/1681028707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc430005bf0 0x7fc430006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc449400640 1 --2- 172.21.15.69:0/1681028707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c071be0 0x7fc430006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc449400640 1 --2- 172.21.15.69:0/1681028707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 0x7fc4300073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c0b3400 con 0x7fc430006f50 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc43000dc80 con 0x7fc430005bf0 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc43000de80 con 0x7fc44c071be0 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc44b200640 1 --2- 172.21.15.69:0/1681028707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 0x7fc4300073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc44b200640 1 -- 172.21.15.69:0/1681028707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c071be0 msgr2=0x7fc430006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc44b200640 1 --2- 172.21.15.69:0/1681028707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c071be0 0x7fc430006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc44b200640 1 -- 172.21.15.69:0/1681028707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc430005bf0 msgr2=0x7fc430006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc44b200640 1 --2- 172.21.15.69:0/1681028707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc430005bf0 0x7fc430006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.888+0000 7fc44b200640 1 -- 172.21.15.69:0/1681028707 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc430005730 con 0x7fc430006f50 2024-04-24T08:19:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.889+0000 7fc44b200640 1 --2- 172.21.15.69:0/1681028707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 0x7fc4300073d0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fc444009f90 tx=0x7fc430007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98740f1ff54c272c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.890+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc444009c10 con 0x7fc430006f50 2024-04-24T08:19:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.890+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc444036070 con 0x7fc430006f50 2024-04-24T08:19:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.890+0000 7fc449400640 1 -- 172.21.15.69:0/1681028707 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc444031880 con 0x7fc430006f50 2024-04-24T08:19:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.890+0000 7fc450a00640 1 -- 172.21.15.69:0/1681028707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 msgr2=0x7fc4300073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.890+0000 7fc450a00640 1 --2- 172.21.15.69:0/1681028707 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 0x7fc4300073d0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fc444009f90 tx=0x7fc430007a40 comp rx=0 tx=0).stop 2024-04-24T08:19:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.890+0000 7fc44b200640 1 -- 172.21.15.69:0/1681028707 reap_dead start 2024-04-24T08:19:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.890+0000 7fc450a00640 1 -- 172.21.15.69:0/1681028707 shutdown_connections 2024-04-24T08:19:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.890+0000 7fc450a00640 1 -- 172.21.15.69:0/1681028707 >> 172.21.15.69:0/1681028707 conn(0x7fc44c0d0160 msgr2=0x7fc44c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.890+0000 7fc450a00640 1 -- 172.21.15.69:0/1681028707 shutdown_connections 2024-04-24T08:19:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.890+0000 7fc450a00640 1 -- 172.21.15.69:0/1681028707 wait complete. 2024-04-24T08:19:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.892+0000 7fc450a00640 1 Processor -- start 2024-04-24T08:19:13.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.892+0000 7fc450a00640 1 -- start start 2024-04-24T08:19:13.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.893+0000 7fc450a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 0x7fc44c1434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.893+0000 7fc450a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc44c1e86c0 0x7fc44c1e8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.893+0000 7fc450a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c15b1b0 0x7fc44c15b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.893+0000 7fc450a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c15aef0 con 0x7fc430006f50 2024-04-24T08:19:13.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.893+0000 7fc450a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc44c072150 con 0x7fc44c1e86c0 2024-04-24T08:19:13.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.893+0000 7fc450a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc44c06f950 con 0x7fc44c15b1b0 2024-04-24T08:19:13.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.893+0000 7fc44a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 0x7fc44c1434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.893+0000 7fc44a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 0x7fc44c1434c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42166/0 (socket says 172.21.15.69:42166) 2024-04-24T08:19:13.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.893+0000 7fc44a800640 1 -- 172.21.15.69:0/180562899 learned_addr learned my addr 172.21.15.69:0/180562899 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc44a800640 1 -- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c15b1b0 msgr2=0x7fc44c15b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc44b200640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c15b1b0 0x7fc44c15b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc449e00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc44c1e86c0 0x7fc44c1e8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc44a800640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c15b1b0 0x7fc44c15b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc44a800640 1 -- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc44c1e86c0 msgr2=0x7fc44c1e8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc44a800640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc44c1e86c0 0x7fc44c1e8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc44a800640 1 -- 172.21.15.69:0/180562899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44c08cbe0 con 0x7fc430006f50 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc449e00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc44c1e86c0 0x7fc44c1e8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc44a800640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 0x7fc44c1434c0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00c7d0 tx=0x7fc43c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc442a00640 1 -- 172.21.15.69:0/180562899 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c018070 con 0x7fc430006f50 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc442a00640 1 -- 172.21.15.69:0/180562899 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc43c0074e0 con 0x7fc430006f50 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.894+0000 7fc442a00640 1 -- 172.21.15.69:0/180562899 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43c004a20 con 0x7fc430006f50 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.895+0000 7fc450a00640 1 -- 172.21.15.69:0/180562899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc44c1e9180 con 0x7fc430006f50 2024-04-24T08:19:13.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.895+0000 7fc450a00640 1 -- 172.21.15.69:0/180562899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc44c08c3e0 con 0x7fc430006f50 2024-04-24T08:19:13.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.896+0000 7fc450a00640 1 -- 172.21.15.69:0/180562899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc44c15bb70 con 0x7fc430006f50 2024-04-24T08:19:13.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.897+0000 7fc442a00640 1 -- 172.21.15.69:0/180562899 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc43c01b070 con 0x7fc430006f50 2024-04-24T08:19:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.898+0000 7fc442a00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc42c06c920 0x7fc42c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.898+0000 7fc442a00640 1 -- 172.21.15.69:0/180562899 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fc43c08b120 con 0x7fc430006f50 2024-04-24T08:19:13.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.898+0000 7fc449e00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc42c06c920 0x7fc42c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:13.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.899+0000 7fc449e00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc42c06c920 0x7fc42c06ede0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc44c155340 tx=0x7fc438009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:13.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:13.902+0000 7fc442a00640 1 -- 172.21.15.69:0/180562899 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc43c059450 con 0x7fc430006f50 2024-04-24T08:19:14.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.139+0000 7fc450a00640 1 -- 172.21.15.69:0/180562899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc44c075d50 con 0x7fc430006f50 2024-04-24T08:19:14.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.140+0000 7fc442a00640 1 -- 172.21.15.69:0/180562899 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5183 (secure 0 0 0) 0x7fc43c058d50 con 0x7fc430006f50 2024-04-24T08:19:14.142 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:14.142 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":86,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:19:13.683891+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.69:6835/3642138906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3642138906},{"type":"v1","addr":"172.21.15.69:6835","nonce":3642138906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-24T08:19:14.142 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 86 2024-04-24T08:19:14.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 -- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc42c06c920 msgr2=0x7fc42c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:14.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc42c06c920 0x7fc42c06ede0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc44c155340 tx=0x7fc438009040 comp rx=0 tx=0).stop 2024-04-24T08:19:14.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 -- 172.21.15.69:0/180562899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 msgr2=0x7fc44c1434c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:14.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 0x7fc44c1434c0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fc43c00c7d0 tx=0x7fc43c00cc90 comp rx=0 tx=0).stop 2024-04-24T08:19:14.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 -- 172.21.15.69:0/180562899 shutdown_connections 2024-04-24T08:19:14.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc42c06c920 0x7fc42c06ede0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:14.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc44c15b1b0 0x7fc44c15b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:14.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc44c1e86c0 0x7fc44c1e8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:14.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 --2- 172.21.15.69:0/180562899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc430006f50 0x7fc44c1434c0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:14.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 -- 172.21.15.69:0/180562899 >> 172.21.15.69:0/180562899 conn(0x7fc44c0d0160 msgr2=0x7fc44c153ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:14.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 -- 172.21.15.69:0/180562899 shutdown_connections 2024-04-24T08:19:14.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:14.143+0000 7fc450a00640 1 -- 172.21.15.69:0/180562899 wait complete. 2024-04-24T08:19:14.565 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:18:10.396179+0000', 'modified': '2024-04-24T08:19:13.683891+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8955}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8955': {'gid': 8955, 'name': 'a', 'rank': 0, 'incarnation': 86, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.69:6835/3642138906', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6833', 'nonce': 3642138906}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 3642138906}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:19:14.565 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8955:up:replay 2024-04-24T08:19:15.567 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:19:15.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.777+0000 7f2267000640 1 Processor -- start 2024-04-24T08:19:15.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.777+0000 7f2267000640 1 -- start start 2024-04-24T08:19:15.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.778+0000 7f2267000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f226006db90 0x7f2260073530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:15.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.778+0000 7f2267000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f226015b980 0x7f226015be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:15.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.778+0000 7f2267000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2260073a70 con 0x7f22601523f0 2024-04-24T08:19:15.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.778+0000 7f2267000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f22600aeac0 con 0x7f226015b980 2024-04-24T08:19:15.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.778+0000 7f2267000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f22600aeda0 con 0x7f226006db90 2024-04-24T08:19:15.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.779+0000 7f2265c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f22601523f0 0x7f2260072e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37678/0 (socket says 172.21.15.69:37678) 2024-04-24T08:19:15.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.779+0000 7f2265c00640 1 -- 172.21.15.69:0/2106251567 learned_addr learned my addr 172.21.15.69:0/2106251567 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:15.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.779+0000 7f2266600640 1 --2- 172.21.15.69:0/2106251567 >> v2:172.21.15.121:3300/0 conn(0x7f226015b980 0x7f226015be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.779+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 899768641 0 0) 0x7f2260073a70 con 0x7f22601523f0 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.779+0000 7f2265200640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f226006db90 0x7f2260073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.779+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2248003520 con 0x7f22601523f0 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.779+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 941244314 0 0) 0x7f2248003520 con 0x7f22601523f0 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f226006db90 msgr2=0x7f2260073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f226006db90 0x7f2260073530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 >> v2:172.21.15.121:3300/0 conn(0x7f226015b980 msgr2=0x7f226015be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 --2- 172.21.15.69:0/2106251567 >> v2:172.21.15.121:3300/0 conn(0x7f226015b980 0x7f226015be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22600af080 con 0x7f22601523f0 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f2254001670 con 0x7f22601523f0 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 >> v1:172.21.15.69:6789/0 conn(0x7f22601523f0 legacy=0x7f2260072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22480039b0 0x7f2248003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f226006db90 0x7f2248004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:15.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2248004d40 0x7f22480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f22600aeac0 con 0x7f2248004d40 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f22600aeda0 con 0x7f22480039b0 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f224800dd30 con 0x7f226006db90 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f2266600640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2248004d40 0x7f22480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f2265c00640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22480039b0 0x7f2248003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f2265200640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f226006db90 0x7f2248004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f2266600640 1 -- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f226006db90 msgr2=0x7f2248004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f2266600640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f226006db90 0x7f2248004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f2266600640 1 -- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22480039b0 msgr2=0x7f2248003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f2266600640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22480039b0 0x7f2248003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f2266600640 1 -- 172.21.15.69:0/2106251567 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2248003520 con 0x7f2248004d40 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.780+0000 7f2265c00640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f22480039b0 0x7f2248003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:15.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.781+0000 7f2266600640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2248004d40 0x7f22480051c0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f2258007360 tx=0x7f2248005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986dc39a3ad5020f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.782+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2258009be0 con 0x7f2248004d40 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.782+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f225802e620 con 0x7f2248004d40 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.782+0000 7f225fe00640 1 -- 172.21.15.69:0/2106251567 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225802e9f0 con 0x7f2248004d40 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.782+0000 7f2267000640 1 -- 172.21.15.69:0/2106251567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2248004d40 msgr2=0x7f22480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.782+0000 7f2267000640 1 --2- 172.21.15.69:0/2106251567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2248004d40 0x7f22480051c0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f2258007360 tx=0x7f2248005830 comp rx=0 tx=0).stop 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.782+0000 7f2266600640 1 -- 172.21.15.69:0/2106251567 reap_dead start 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.782+0000 7f2267000640 1 -- 172.21.15.69:0/2106251567 shutdown_connections 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.782+0000 7f2267000640 1 -- 172.21.15.69:0/2106251567 >> 172.21.15.69:0/2106251567 conn(0x7f22600d0160 msgr2=0x7f22600d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.782+0000 7f2267000640 1 -- 172.21.15.69:0/2106251567 shutdown_connections 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.783+0000 7f2267000640 1 -- 172.21.15.69:0/2106251567 wait complete. 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.784+0000 7f2267000640 1 Processor -- start 2024-04-24T08:19:15.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.784+0000 7f2267000640 1 -- start start 2024-04-24T08:19:15.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.785+0000 7f2267000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2248004d40 0x7f226009a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:15.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.785+0000 7f2267000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f226013af80 0x7f226014b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:15.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.785+0000 7f2267000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f226009c120 0x7f226009c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.785+0000 7f2267000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f226006dfd0 con 0x7f226013af80 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.785+0000 7f2267000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f22600b3190 con 0x7f226009c120 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.785+0000 7f2267000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f22601527d0 con 0x7f2248004d40 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2265200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f226013af80 0x7f226014b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2265200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f226013af80 0x7f226014b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42182/0 (socket says 172.21.15.69:42182) 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2265200640 1 -- 172.21.15.69:0/964303957 learned_addr learned my addr 172.21.15.69:0/964303957 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2265c00640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2248004d40 0x7f226009a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2266600640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f226009c120 0x7f226009c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2265200640 1 -- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2248004d40 msgr2=0x7f226009a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2265200640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2248004d40 0x7f226009a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2265200640 1 -- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f226009c120 msgr2=0x7f226009c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2265200640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f226009c120 0x7f226009c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2265200640 1 -- 172.21.15.69:0/964303957 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226013b410 con 0x7f226013af80 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.786+0000 7f2265200640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f226013af80 0x7f226014b690 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f224c00e7e0 tx=0x7f224c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:15.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.787+0000 7f225d600640 1 -- 172.21.15.69:0/964303957 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c017070 con 0x7f226013af80 2024-04-24T08:19:15.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.787+0000 7f225d600640 1 -- 172.21.15.69:0/964303957 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f224c00c6d0 con 0x7f226013af80 2024-04-24T08:19:15.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.787+0000 7f225d600640 1 -- 172.21.15.69:0/964303957 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c00f040 con 0x7f226013af80 2024-04-24T08:19:15.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.787+0000 7f2267000640 1 -- 172.21.15.69:0/964303957 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f226013b6f0 con 0x7f226013af80 2024-04-24T08:19:15.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.787+0000 7f2267000640 1 -- 172.21.15.69:0/964303957 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f226013bf90 con 0x7f226013af80 2024-04-24T08:19:15.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.788+0000 7f2267000640 1 -- 172.21.15.69:0/964303957 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f226013b8c0 con 0x7f226013af80 2024-04-24T08:19:15.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.789+0000 7f225d600640 1 -- 172.21.15.69:0/964303957 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f224c01a070 con 0x7f226013af80 2024-04-24T08:19:15.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.790+0000 7f225d600640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f224006c830 0x7f224006ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:15.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.790+0000 7f225d600640 1 -- 172.21.15.69:0/964303957 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7f224c08b940 con 0x7f226013af80 2024-04-24T08:19:15.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.791+0000 7f2265c00640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f224006c830 0x7f224006ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:15.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.792+0000 7f2265c00640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f224006c830 0x7f224006ecf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f22540017e0 tx=0x7f2254009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:15.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:15.795+0000 7f225d600640 1 -- 172.21.15.69:0/964303957 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f224c059c70 con 0x7f226013af80 2024-04-24T08:19:16.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.034+0000 7f2267000640 1 -- 172.21.15.69:0/964303957 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2260075d50 con 0x7f226013af80 2024-04-24T08:19:16.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.034+0000 7f225d600640 1 -- 172.21.15.69:0/964303957 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5183 (secure 0 0 0) 0x7f224c059570 con 0x7f226013af80 2024-04-24T08:19:16.036 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:16.036 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":88,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:19:14.694834+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.69:6835/3642138906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3642138906},{"type":"v1","addr":"172.21.15.69:6835","nonce":3642138906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-24T08:19:16.036 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 88 2024-04-24T08:19:16.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 -- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f224006c830 msgr2=0x7f224006ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f224006c830 0x7f224006ecf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f22540017e0 tx=0x7f2254009040 comp rx=0 tx=0).stop 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 -- 172.21.15.69:0/964303957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f226013af80 msgr2=0x7f226014b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f226013af80 0x7f226014b690 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f224c00e7e0 tx=0x7f224c00eca0 comp rx=0 tx=0).stop 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 -- 172.21.15.69:0/964303957 shutdown_connections 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f224006c830 0x7f224006ecf0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f226013af80 0x7f226014b690 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f226009c120 0x7f226009c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 --2- 172.21.15.69:0/964303957 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2248004d40 0x7f226009a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 -- 172.21.15.69:0/964303957 >> 172.21.15.69:0/964303957 conn(0x7f22600d0160 msgr2=0x7f226015be10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 -- 172.21.15.69:0/964303957 shutdown_connections 2024-04-24T08:19:16.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:16.037+0000 7f2267000640 1 -- 172.21.15.69:0/964303957 wait complete. 2024-04-24T08:19:16.275 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:19:16.282 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:19:16.459 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 88, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:18:10.396179+0000', 'modified': '2024-04-24T08:19:14.694834+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8955}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8955': {'gid': 8955, 'name': 'a', 'rank': 0, 'incarnation': 86, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.69:6835/3642138906', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6833', 'nonce': 3642138906}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 3642138906}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:19:16.460 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8955:up:rejoin 2024-04-24T08:19:17.460 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:19:17.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.663+0000 7fa9b6600640 1 Processor -- start 2024-04-24T08:19:17.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.663+0000 7fa9b6600640 1 -- start start 2024-04-24T08:19:17.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.665+0000 7fa9b6600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0156c60 0x7fa9b015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:17.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.665+0000 7fa9b6600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa9b00ad7e0 0x7fa9b015d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:17.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.665+0000 7fa9b6600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9b015d660 con 0x7fa9b00b30f0 2024-04-24T08:19:17.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.665+0000 7fa9b6600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa9b006dc30 con 0x7fa9b00ad7e0 2024-04-24T08:19:17.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.665+0000 7fa9b6600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b006df10 con 0x7fa9b0156c60 2024-04-24T08:19:17.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.665+0000 7fa9afe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa9b00b30f0 0x7fa9b00ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37682/0 (socket says 172.21.15.69:37682) 2024-04-24T08:19:17.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.665+0000 7fa9afe00640 1 -- 172.21.15.69:0/368406993 learned_addr learned my addr 172.21.15.69:0/368406993 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:17.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.665+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298475953 0 0) 0x7fa9b015d660 con 0x7fa9b00b30f0 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.665+0000 7fa9b5200640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0156c60 0x7fa9b015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa998003520 con 0x7fa9b00b30f0 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.665+0000 7fa9b5c00640 1 --2- 172.21.15.69:0/368406993 >> v2:172.21.15.121:3300/0 conn(0x7fa9b00ad7e0 0x7fa9b015d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3396477609 0 0) 0x7fa998003520 con 0x7fa9b00b30f0 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0156c60 msgr2=0x7fa9b015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0156c60 0x7fa9b015cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 >> v2:172.21.15.121:3300/0 conn(0x7fa9b00ad7e0 msgr2=0x7fa9b015d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 --2- 172.21.15.69:0/368406993 >> v2:172.21.15.121:3300/0 conn(0x7fa9b00ad7e0 0x7fa9b015d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b006e1f0 con 0x7fa9b00b30f0 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fa99c001670 con 0x7fa9b00b30f0 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9b5200640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0156c60 0x7fa9b015cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 >> v1:172.21.15.69:6789/0 conn(0x7fa9b00b30f0 legacy=0x7fa9b00ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 0x7fa998003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0156c60 0x7fa998004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:17.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa998004d40 0x7fa9980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b006dc30 con 0x7fa9980039b0 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b006df10 con 0x7fa998004d40 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa99800ba40 con 0x7fa9b0156c60 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.666+0000 7fa9b5200640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 0x7fa998003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.667+0000 7fa9afe00640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0156c60 0x7fa998004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.667+0000 7fa9b5c00640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa998004d40 0x7fa9980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.667+0000 7fa9b5200640 1 -- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0156c60 msgr2=0x7fa998004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.667+0000 7fa9b5200640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0156c60 0x7fa998004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.667+0000 7fa9b5200640 1 -- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa998004d40 msgr2=0x7fa9980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.667+0000 7fa9b5200640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa998004d40 0x7fa9980051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.667+0000 7fa9b5200640 1 -- 172.21.15.69:0/368406993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa998003520 con 0x7fa9980039b0 2024-04-24T08:19:17.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.667+0000 7fa9afe00640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0156c60 0x7fa998004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:17.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.668+0000 7fa9b5200640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 0x7fa998003dd0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fa9a40098a0 tx=0x7fa998005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d53521d351ae6cd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:17.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.668+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a400f040 con 0x7fa9980039b0 2024-04-24T08:19:17.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.668+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a402ace0 con 0x7fa9980039b0 2024-04-24T08:19:17.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.668+0000 7fa9af400640 1 -- 172.21.15.69:0/368406993 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4004250 con 0x7fa9980039b0 2024-04-24T08:19:17.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.669+0000 7fa9b6600640 1 -- 172.21.15.69:0/368406993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 msgr2=0x7fa998003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:17.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.669+0000 7fa9b6600640 1 --2- 172.21.15.69:0/368406993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 0x7fa998003dd0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fa9a40098a0 tx=0x7fa998005830 comp rx=0 tx=0).stop 2024-04-24T08:19:17.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.669+0000 7fa9b5c00640 1 -- 172.21.15.69:0/368406993 reap_dead start 2024-04-24T08:19:17.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.669+0000 7fa9b6600640 1 -- 172.21.15.69:0/368406993 shutdown_connections 2024-04-24T08:19:17.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.669+0000 7fa9b6600640 1 -- 172.21.15.69:0/368406993 >> 172.21.15.69:0/368406993 conn(0x7fa9b00d0160 msgr2=0x7fa9b00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:17.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.669+0000 7fa9b6600640 1 -- 172.21.15.69:0/368406993 shutdown_connections 2024-04-24T08:19:17.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.669+0000 7fa9b6600640 1 -- 172.21.15.69:0/368406993 wait complete. 2024-04-24T08:19:17.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.671+0000 7fa9b6600640 1 Processor -- start 2024-04-24T08:19:17.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.671+0000 7fa9b6600640 1 -- start start 2024-04-24T08:19:17.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.672+0000 7fa9b6600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 0x7fa9b01456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:17.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.672+0000 7fa9b6600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9b0073050 0x7fa9b0155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:17.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.672+0000 7fa9b6600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0072630 0x7fa9b006ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:17.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.672+0000 7fa9b6600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b015cac0 con 0x7fa9980039b0 2024-04-24T08:19:17.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.672+0000 7fa9b6600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b015d940 con 0x7fa9b0073050 2024-04-24T08:19:17.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.672+0000 7fa9b6600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b015cc40 con 0x7fa9b0072630 2024-04-24T08:19:17.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.672+0000 7fa9b5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 0x7fa9b01456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:17.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.672+0000 7fa9b5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 0x7fa9b01456f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42200/0 (socket says 172.21.15.69:42200) 2024-04-24T08:19:17.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.672+0000 7fa9b5200640 1 -- 172.21.15.69:0/2434081892 learned_addr learned my addr 172.21.15.69:0/2434081892 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:17.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.673+0000 7fa9afe00640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9b0073050 0x7fa9b0155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:17.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.673+0000 7fa9b5c00640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0072630 0x7fa9b006ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:17.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.673+0000 7fa9b5200640 1 -- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0072630 msgr2=0x7fa9b006ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:17.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.673+0000 7fa9b5200640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0072630 0x7fa9b006ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.673+0000 7fa9b5200640 1 -- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9b0073050 msgr2=0x7fa9b0155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.673+0000 7fa9b5200640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9b0073050 0x7fa9b0155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.673+0000 7fa9b5200640 1 -- 172.21.15.69:0/2434081892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b00736c0 con 0x7fa9980039b0 2024-04-24T08:19:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.673+0000 7fa9b5200640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 0x7fa9b01456f0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4034040 tx=0x7fa9a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.674+0000 7fa9acc00640 1 -- 172.21.15.69:0/2434081892 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4041070 con 0x7fa9980039b0 2024-04-24T08:19:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.674+0000 7fa9acc00640 1 -- 172.21.15.69:0/2434081892 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a400f000 con 0x7fa9980039b0 2024-04-24T08:19:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.674+0000 7fa9acc00640 1 -- 172.21.15.69:0/2434081892 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4003c50 con 0x7fa9980039b0 2024-04-24T08:19:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.674+0000 7fa9b6600640 1 -- 172.21.15.69:0/2434081892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b00739a0 con 0x7fa9980039b0 2024-04-24T08:19:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.674+0000 7fa9b6600640 1 -- 172.21.15.69:0/2434081892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b0072110 con 0x7fa9980039b0 2024-04-24T08:19:17.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.675+0000 7fa9b6600640 1 -- 172.21.15.69:0/2434081892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b0073430 con 0x7fa9980039b0 2024-04-24T08:19:17.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.676+0000 7fa9acc00640 1 -- 172.21.15.69:0/2434081892 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa9a4044070 con 0x7fa9980039b0 2024-04-24T08:19:17.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.677+0000 7fa9acc00640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa99006c8f0 0x7fa99006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:17.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.677+0000 7fa9acc00640 1 -- 172.21.15.69:0/2434081892 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fa9a40ac390 con 0x7fa9980039b0 2024-04-24T08:19:17.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.678+0000 7fa9afe00640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa99006c8f0 0x7fa99006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:17.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.679+0000 7fa9afe00640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa99006c8f0 0x7fa99006edb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fa99c001eb0 tx=0x7fa99c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:17.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.682+0000 7fa9acc00640 1 -- 172.21.15.69:0/2434081892 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9a407a630 con 0x7fa9980039b0 2024-04-24T08:19:17.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.923+0000 7fa9b6600640 1 -- 172.21.15.69:0/2434081892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9b0075d50 con 0x7fa9980039b0 2024-04-24T08:19:17.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.924+0000 7fa9acc00640 1 -- 172.21.15.69:0/2434081892 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+5190 (secure 0 0 0) 0x7fa9a4079f30 con 0x7fa9980039b0 2024-04-24T08:19:17.925 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:17.925 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:19:16.707297+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":98,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8955},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8955":{"gid":8955,"name":"a","rank":0,"incarnation":86,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/3642138906","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3642138906},{"type":"v1","addr":"172.21.15.69:6835","nonce":3642138906}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8955,"qdb_cluster":[8955]},"id":7}]} 2024-04-24T08:19:17.926 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 89 2024-04-24T08:19:17.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.926+0000 7fa9b6600640 1 -- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa99006c8f0 msgr2=0x7fa99006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:17.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.926+0000 7fa9b6600640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa99006c8f0 0x7fa99006edb0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fa99c001eb0 tx=0x7fa99c009040 comp rx=0 tx=0).stop 2024-04-24T08:19:17.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.926+0000 7fa9b6600640 1 -- 172.21.15.69:0/2434081892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 msgr2=0x7fa9b01456f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:17.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.926+0000 7fa9b6600640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 0x7fa9b01456f0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4034040 tx=0x7fa9a4004730 comp rx=0 tx=0).stop 2024-04-24T08:19:17.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.926+0000 7fa9b6600640 1 -- 172.21.15.69:0/2434081892 shutdown_connections 2024-04-24T08:19:17.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.926+0000 7fa9b6600640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa99006c8f0 0x7fa99006edb0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:17.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.926+0000 7fa9b6600640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa9b0072630 0x7fa9b006ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:17.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.926+0000 7fa9b6600640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa9b0073050 0x7fa9b0155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:17.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.926+0000 7fa9b6600640 1 --2- 172.21.15.69:0/2434081892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9980039b0 0x7fa9b01456f0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:17.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.926+0000 7fa9b6600640 1 -- 172.21.15.69:0/2434081892 >> 172.21.15.69:0/2434081892 conn(0x7fa9b00d0160 msgr2=0x7fa9b00b3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:17.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.927+0000 7fa9b6600640 1 -- 172.21.15.69:0/2434081892 shutdown_connections 2024-04-24T08:19:17.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:17.927+0000 7fa9b6600640 1 -- 172.21.15.69:0/2434081892 wait complete. 2024-04-24T08:19:18.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 89, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:18:10.396179+0000', 'modified': '2024-04-24T08:19:16.707297+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 98, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8955}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8955': {'gid': 8955, 'name': 'a', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.69:6835/3642138906', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6833', 'nonce': 3642138906}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 3642138906}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8955, 'qdb_cluster': [8955]} 2024-04-24T08:19:18.347 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:19:18.347 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-24T08:19:18.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.553+0000 7fe4dd600640 1 Processor -- start 2024-04-24T08:19:18.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.554+0000 7fe4dd600640 1 -- start start 2024-04-24T08:19:18.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.555+0000 7fe4dd600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe4d80710e0 0x7fe4d8162250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.555+0000 7fe4dd600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d815f320 0x7fe4d815f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.555+0000 7fe4dd600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4d8070860 con 0x7fe4d80cf9a0 2024-04-24T08:19:18.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.555+0000 7fe4dd600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe4d815ee90 con 0x7fe4d80710e0 2024-04-24T08:19:18.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.555+0000 7fe4dd600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d8163180 con 0x7fe4d815f320 2024-04-24T08:19:18.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.555+0000 7fe4d7e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe4d80cf9a0 0x7fe4d8072af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:37694/0 (socket says 172.21.15.69:37694) 2024-04-24T08:19:18.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.555+0000 7fe4d7e00640 1 -- 172.21.15.69:0/3335268255 learned_addr learned my addr 172.21.15.69:0/3335268255 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:18.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4dcc00640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d815f320 0x7fe4d815f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d7400640 1 --2- 172.21.15.69:0/3335268255 >> v2:172.21.15.121:3300/0 conn(0x7fe4d80710e0 0x7fe4d8162250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 812035165 0 0) 0x7fe4d8070860 con 0x7fe4d80cf9a0 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4bc003520 con 0x7fe4d80cf9a0 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3723046320 0 0) 0x7fe4bc003520 con 0x7fe4d80cf9a0 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d815f320 msgr2=0x7fe4d815f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d815f320 0x7fe4d815f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 >> v2:172.21.15.121:3300/0 conn(0x7fe4d80710e0 msgr2=0x7fe4d8162250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 --2- 172.21.15.69:0/3335268255 >> v2:172.21.15.121:3300/0 conn(0x7fe4d80710e0 0x7fe4d8162250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d81628e0 con 0x7fe4d80cf9a0 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe4c80019f0 con 0x7fe4d80cf9a0 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d7400640 1 --2- 172.21.15.69:0/3335268255 >> v2:172.21.15.121:3300/0 conn(0x7fe4d80710e0 0x7fe4d8162250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 >> v1:172.21.15.69:6789/0 conn(0x7fe4d80cf9a0 legacy=0x7fe4d8072af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4bc0039b0 0x7fe4bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d815f320 0x7fe4bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 0x7fe4bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d815ee90 con 0x7fe4bc004d40 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d8163180 con 0x7fe4bc0039b0 2024-04-24T08:19:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.556+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe4bc00baf0 con 0x7fe4d815f320 2024-04-24T08:19:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.557+0000 7fe4dcc00640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 0x7fe4bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.557+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4bc0039b0 0x7fe4bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.557+0000 7fe4dcc00640 1 -- 172.21.15.69:0/3335268255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d815f320 msgr2=0x7fe4bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.557+0000 7fe4dcc00640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d815f320 0x7fe4bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.557+0000 7fe4dcc00640 1 -- 172.21.15.69:0/3335268255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4bc0039b0 msgr2=0x7fe4bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.557+0000 7fe4dcc00640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4bc0039b0 0x7fe4bc003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.557+0000 7fe4dcc00640 1 -- 172.21.15.69:0/3335268255 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4bc003520 con 0x7fe4bc004d40 2024-04-24T08:19:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.558+0000 7fe4dcc00640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 0x7fe4bc0051c0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000ba30 tx=0x7fe4bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8552ad69d8d93e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.558+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0030070 con 0x7fe4bc004d40 2024-04-24T08:19:18.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.558+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d002ae10 con 0x7fe4bc004d40 2024-04-24T08:19:18.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.558+0000 7fe4d6a00640 1 -- 172.21.15.69:0/3335268255 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d000b600 con 0x7fe4bc004d40 2024-04-24T08:19:18.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.559+0000 7fe4dd600640 1 -- 172.21.15.69:0/3335268255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 msgr2=0x7fe4bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.559+0000 7fe4dd600640 1 --2- 172.21.15.69:0/3335268255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 0x7fe4bc0051c0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fe4d000ba30 tx=0x7fe4bc005830 comp rx=0 tx=0).stop 2024-04-24T08:19:18.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.559+0000 7fe4dcc00640 1 -- 172.21.15.69:0/3335268255 reap_dead start 2024-04-24T08:19:18.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.559+0000 7fe4dd600640 1 -- 172.21.15.69:0/3335268255 shutdown_connections 2024-04-24T08:19:18.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.559+0000 7fe4dd600640 1 -- 172.21.15.69:0/3335268255 >> 172.21.15.69:0/3335268255 conn(0x7fe4d80d12c0 msgr2=0x7fe4d80d16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.559+0000 7fe4dd600640 1 -- 172.21.15.69:0/3335268255 shutdown_connections 2024-04-24T08:19:18.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.559+0000 7fe4dd600640 1 -- 172.21.15.69:0/3335268255 wait complete. 2024-04-24T08:19:18.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.561+0000 7fe4dd600640 1 Processor -- start 2024-04-24T08:19:18.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.561+0000 7fe4dd600640 1 -- start start 2024-04-24T08:19:18.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4dd600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 0x7fe4d80ae260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4dd600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d80caf20 0x7fe4d81f43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4dd600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4d8159260 0x7fe4d80af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4dd600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d8073200 con 0x7fe4bc004d40 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4dd600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d8162ea0 con 0x7fe4d8159260 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4dd600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d80c8390 con 0x7fe4d80caf20 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4d7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 0x7fe4d80ae260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4d7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 0x7fe4d80ae260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42228/0 (socket says 172.21.15.69:42228) 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4d7e00640 1 -- 172.21.15.69:0/1916649150 learned_addr learned my addr 172.21.15.69:0/1916649150 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4d7e00640 1 -- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d80caf20 msgr2=0x7fe4d81f43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d80caf20 0x7fe4d81f43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4d7e00640 1 -- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4d8159260 msgr2=0x7fe4d80af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.562+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4d8159260 0x7fe4d80af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.563+0000 7fe4d7e00640 1 -- 172.21.15.69:0/1916649150 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d80c8670 con 0x7fe4bc004d40 2024-04-24T08:19:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.563+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 0x7fe4d80ae260 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8002130 tx=0x7fe4c800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.563+0000 7fe4cfe00640 1 -- 172.21.15.69:0/1916649150 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8019070 con 0x7fe4bc004d40 2024-04-24T08:19:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.563+0000 7fe4cfe00640 1 -- 172.21.15.69:0/1916649150 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c8004d60 con 0x7fe4bc004d40 2024-04-24T08:19:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.563+0000 7fe4cfe00640 1 -- 172.21.15.69:0/1916649150 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c8001f60 con 0x7fe4bc004d40 2024-04-24T08:19:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.563+0000 7fe4dd600640 1 -- 172.21.15.69:0/1916649150 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d80c8950 con 0x7fe4bc004d40 2024-04-24T08:19:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.563+0000 7fe4dd600640 1 -- 172.21.15.69:0/1916649150 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d80c8f10 con 0x7fe4bc004d40 2024-04-24T08:19:18.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.565+0000 7fe4cfe00640 1 -- 172.21.15.69:0/1916649150 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe4c80072d0 con 0x7fe4bc004d40 2024-04-24T08:19:18.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.566+0000 7fe4cfe00640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4b406c8f0 0x7fe4b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.567+0000 7fe4cfe00640 1 -- 172.21.15.69:0/1916649150 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fe4c808bc40 con 0x7fe4bc004d40 2024-04-24T08:19:18.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.567+0000 7fe4e1a71740 1 Processor -- start 2024-04-24T08:19:18.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.567+0000 7fe4e1a71740 1 -- start start 2024-04-24T08:19:18.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.567+0000 7fe4d7400640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4b406c8f0 0x7fe4b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d3986baa0 0x555d3986ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986b360 0x555d39840170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555d397a5b10 con 0x555d3986baa0 2024-04-24T08:19:18.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555d3974c2b0 con 0x555d39888980 2024-04-24T08:19:18.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x555d397a1ad0 con 0x555d3986b360 2024-04-24T08:19:18.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d3986baa0 0x555d3986ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d3986baa0 0x555d3986ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42236/0 (socket says 172.21.15.69:42236) 2024-04-24T08:19:18.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4d7400640 1 -- 172.21.15.69:0/2803662226 learned_addr learned my addr 172.21.15.69:0/2803662226 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.568+0000 7fe4dcc00640 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986b360 0x555d39840170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4d7400640 1 -- 172.21.15.69:0/2803662226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986b360 msgr2=0x555d39840170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4d7400640 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986b360 0x555d39840170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4d7400640 1 -- 172.21.15.69:0/2803662226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 msgr2=0x555d3983b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4d7400640 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4d7400640 1 -- 172.21.15.69:0/2803662226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d394b82d0 con 0x555d3986baa0 2024-04-24T08:19:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4dcc00640 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986b360 0x555d39840170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4d7400640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4b406c8f0 0x7fe4b406edb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0006fd0 tx=0x7fe4c0009b90 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4d7400640 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d3986baa0 0x555d3986ae20 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000e930 tx=0x7fe4c0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a7792f476c44709 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4c7e00640 1 -- 172.21.15.69:0/2803662226 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0025070 con 0x555d3986baa0 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4c7e00640 1 -- 172.21.15.69:0/2803662226 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c001c040 con 0x555d3986baa0 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.569+0000 7fe4c7e00640 1 -- 172.21.15.69:0/2803662226 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c000f040 con 0x555d3986baa0 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.570+0000 7fe4e1a71740 1 -- 172.21.15.69:0/2803662226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d3986baa0 msgr2=0x555d3986ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.570+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d3986baa0 0x555d3986ae20 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000e930 tx=0x7fe4c0019000 comp rx=0 tx=0).stop 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.570+0000 7fe4e1a71740 1 -- 172.21.15.69:0/2803662226 shutdown_connections 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.570+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d3986baa0 0x555d3986ae20 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.570+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986b360 0x555d39840170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.570+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/2803662226 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.570+0000 7fe4e1a71740 1 -- 172.21.15.69:0/2803662226 >> 172.21.15.69:0/2803662226 conn(0x555d3983d580 msgr2=0x555d39885e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.570+0000 7fe4e1a71740 1 -- 172.21.15.69:0/2803662226 shutdown_connections 2024-04-24T08:19:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.570+0000 7fe4e1a71740 1 -- 172.21.15.69:0/2803662226 wait complete. 2024-04-24T08:19:18.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.571+0000 7fe4e1a71740 1 -- start start 2024-04-24T08:19:18.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.571+0000 7fe4e1a71740 1 Processor -- start 2024-04-24T08:19:18.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d39896120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d39888980 0x555d39867ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d398839d0 0x555d398680e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555d3983f730 con 0x555d39888980 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555d3975f590 con 0x555d398839d0 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x555d3983eb20 con 0x555d3986baa0 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4e1a71740 10 client.0 _ll_register_callbacks cb 0x555d397423a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d39888980 0x555d39867ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d39888980 0x555d39867ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42244/0 (socket says 172.21.15.69:42244) 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4d7400640 1 -- 172.21.15.69:0/148481596 learned_addr learned my addr 172.21.15.69:0/148481596 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d39896120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.572+0000 7fe4dcc00640 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d398839d0 0x555d398680e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4d7400640 1 -- 172.21.15.69:0/148481596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 msgr2=0x555d39896120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4d7400640 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d39896120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4d7400640 1 -- 172.21.15.69:0/148481596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d398839d0 msgr2=0x555d398680e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4d7400640 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d398839d0 0x555d398680e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4d7400640 1 -- 172.21.15.69:0/148481596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d395826b0 con 0x555d39888980 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4d7400640 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d39888980 0x555d39867ba0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00176f0 tx=0x7fe4c0006a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223e450ec357081d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4c4c00640 1 -- 172.21.15.69:0/148481596 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0030020 con 0x555d39888980 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4e1a71740 1 -- 172.21.15.69:0/148481596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4c001c040 con 0x555d39888980 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4c4c00640 1 -- 172.21.15.69:0/148481596 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fe4c00178d0 con 0x555d39888980 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.573+0000 7fe4c4c00640 1 -- 172.21.15.69:0/148481596 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fe4c003a050 con 0x555d39888980 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4e1a71740 10 client.9030 fetch_fsmap learned FSMap version 89 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4e1a71740 1 -- 172.21.15.69:0/148481596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x555d3986eeb0 con 0x555d39888980 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4c4c00640 1 -- 172.21.15.69:0/148481596 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c001c040 con 0x555d39888980 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4c4c00640 1 -- 172.21.15.69:0/148481596 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7fe4c003d020 con 0x555d39888980 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4e1a71740 10 client.9030 fetch_fsmap finished waiting for FSMap version 89 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4e1a71740 10 client.9030 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4e1a71740 20 client.9030 populate_metadata read hostname 'smithi069' 2024-04-24T08:19:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x555d39899fe0 0x555d3989a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4e1a71740 4 client.9030 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:18.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4e1a71740 1 -- 172.21.15.69:0/148481596 --> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d397b3530 con 0x555d39899fe0 2024-04-24T08:19:18.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.574+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x555d39899fe0 0x555d3989a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.575+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x555d39899fe0 0x555d3989a3c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=660b011320bcb968 server_cookie=5e0aead3c553c458 in_seq=0 out_seq=0 2024-04-24T08:19:18.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.575+0000 7fe4c4c00640 10 client.9030 ms_handle_connect on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:18.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4c4c00640 1 -- 172.21.15.69:0/148481596 <== mds.0 v2:172.21.15.69:6833/3642138906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d397b3530 con 0x555d39899fe0 2024-04-24T08:19:18.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4c4c00640 10 client.9030 handle_command_reply: tid=0 2024-04-24T08:19:18.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 client.9030 shutdown 2024-04-24T08:19:18.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 -- 172.21.15.69:0/148481596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d39888980 msgr2=0x555d39867ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d39888980 0x555d39867ba0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fe4c00176f0 tx=0x7fe4c0006a20 comp rx=0 tx=0).stop 2024-04-24T08:19:18.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 -- 172.21.15.69:0/148481596 shutdown_connections 2024-04-24T08:19:18.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x555d39899fe0 0x555d3989a3c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d39888980 0x555d39867ba0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d398839d0 0x555d398680e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4c4c00640 0 client.9030 ms_handle_reset on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:18.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/148481596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d39896120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 -- 172.21.15.69:0/148481596 >> 172.21.15.69:0/148481596 conn(0x555d3986b360 msgr2=0x555d3983aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 -- 172.21.15.69:0/148481596 shutdown_connections 2024-04-24T08:19:18.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 1 -- 172.21.15.69:0/148481596 wait complete. 2024-04-24T08:19:18.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.576+0000 7fe4e1a71740 20 client.9030 trim_cache size 0 max 16384 2024-04-24T08:19:18.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.595+0000 7fe4e1a71740 1 Processor -- start 2024-04-24T08:19:18.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.595+0000 7fe4e1a71740 1 -- start start 2024-04-24T08:19:18.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d39895f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d39896470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d8200420 con 0x555d398839d0 2024-04-24T08:19:18.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d81f8530 con 0x555d39888980 2024-04-24T08:19:18.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d81f8b80 con 0x555d3986baa0 2024-04-24T08:19:18.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d39896470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d39896470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42254/0 (socket says 172.21.15.69:42254) 2024-04-24T08:19:18.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4d7400640 1 -- 172.21.15.69:0/3070917587 learned_addr learned my addr 172.21.15.69:0/3070917587 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4d7400640 1 -- 172.21.15.69:0/3070917587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 msgr2=0x555d39895f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.596+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/3070917587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d39895f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.597+0000 7fe4d7400640 1 --2- 172.21.15.69:0/3070917587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d39895f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.597+0000 7fe4d7400640 1 -- 172.21.15.69:0/3070917587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 msgr2=0x555d3983ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.597+0000 7fe4dcc00640 1 --2- 172.21.15.69:0/3070917587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.597+0000 7fe4d7400640 1 --2- 172.21.15.69:0/3070917587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.597+0000 7fe4d7400640 1 -- 172.21.15.69:0/3070917587 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d39884530 con 0x555d398839d0 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.597+0000 7fe4d7400640 1 --2- 172.21.15.69:0/3070917587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d39896470 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0006a90 tx=0x7fe4c0029000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f519ba7503f1d4a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.597+0000 7fe4c7400640 1 -- 172.21.15.69:0/3070917587 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0030070 con 0x555d398839d0 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.597+0000 7fe4c7400640 1 -- 172.21.15.69:0/3070917587 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c001c040 con 0x555d398839d0 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.597+0000 7fe4c7400640 1 -- 172.21.15.69:0/3070917587 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0025040 con 0x555d398839d0 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.598+0000 7fe4e1a71740 1 -- 172.21.15.69:0/3070917587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 msgr2=0x555d39896470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.598+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/3070917587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d39896470 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0006a90 tx=0x7fe4c0029000 comp rx=0 tx=0).stop 2024-04-24T08:19:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.598+0000 7fe4e1a71740 1 -- 172.21.15.69:0/3070917587 shutdown_connections 2024-04-24T08:19:18.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.598+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/3070917587 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d39896470 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.598+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/3070917587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.598+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/3070917587 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d39895f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.598+0000 7fe4e1a71740 1 -- 172.21.15.69:0/3070917587 >> 172.21.15.69:0/3070917587 conn(0x555d3986b360 msgr2=0x555d39866500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.598+0000 7fe4e1a71740 1 -- 172.21.15.69:0/3070917587 shutdown_connections 2024-04-24T08:19:18.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.598+0000 7fe4e1a71740 1 -- 172.21.15.69:0/3070917587 wait complete. 2024-04-24T08:19:18.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.598+0000 7fe4e1a71740 1 -- start start 2024-04-24T08:19:18.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.599+0000 7fe4e1a71740 1 Processor -- start 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.599+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d398fdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.599+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d3983ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.599+0000 7fe4e1a71740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.599+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555d3986c660 con 0x555d398839d0 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.599+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x555d398843b0 con 0x555d39888980 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.599+0000 7fe4e1a71740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x555d3986d140 con 0x555d3986baa0 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4e1a71740 10 client.0 _ll_register_callbacks cb 0x7fe4c0019d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d3983ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4d7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d3983ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42258/0 (socket says 172.21.15.69:42258) 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4d7400640 1 -- 172.21.15.69:0/485110218 learned_addr learned my addr 172.21.15.69:0/485110218 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d398fdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4dcc00640 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4d7400640 1 -- 172.21.15.69:0/485110218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 msgr2=0x555d398fdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4d7400640 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d398fdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4d7400640 1 -- 172.21.15.69:0/485110218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 msgr2=0x555d3983f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4d7400640 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4d7400640 1 -- 172.21.15.69:0/485110218 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d39886790 con 0x555d398839d0 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4d7400640 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d3983ed10 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fe4c002e040 tx=0x7fe4c001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e287a724e5e59f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.600+0000 7fe4c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4c4c00640 1 -- 172.21.15.69:0/485110218 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0025040 con 0x555d398839d0 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4c4c00640 1 -- 172.21.15.69:0/485110218 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fe4c002b880 con 0x555d398839d0 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4e1a71740 1 -- 172.21.15.69:0/485110218 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4c001c040 con 0x555d398839d0 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4c4c00640 1 -- 172.21.15.69:0/485110218 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fe4c004d050 con 0x555d398839d0 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4e1a71740 10 client.9036 fetch_fsmap learned FSMap version 89 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4e1a71740 1 -- 172.21.15.69:0/485110218 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x555d399004b0 con 0x555d398839d0 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4c4c00640 1 -- 172.21.15.69:0/485110218 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c001c040 con 0x555d398839d0 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4c4c00640 1 -- 172.21.15.69:0/485110218 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7fe4c003d040 con 0x555d398839d0 2024-04-24T08:19:18.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4e1a71740 10 client.9036 fetch_fsmap finished waiting for FSMap version 89 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4e1a71740 10 client.9036 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4e1a71740 20 client.9036 populate_metadata read hostname 'smithi069' 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x555d3983aa10 0x555d398a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4e1a71740 4 client.9036 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.601+0000 7fe4e1a71740 1 -- 172.21.15.69:0/485110218 --> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] -- command(tid 0: {"prefix": "status"}) -- 0x555d398863e0 con 0x555d3983aa10 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.602+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x555d3983aa10 0x555d398a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.602+0000 7fe4d7e00640 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x555d3983aa10 0x555d398a8e10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87878b8d0999d452 server_cookie=8c7a8e6a525005ed in_seq=0 out_seq=0 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.602+0000 7fe4c4c00640 10 client.9036 ms_handle_connect on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.602+0000 7fe4c4c00640 1 -- 172.21.15.69:0/485110218 <== mds.0 v2:172.21.15.69:6833/3642138906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x555d398863e0 con 0x555d3983aa10 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.602+0000 7fe4c4c00640 10 client.9036 handle_command_reply: tid=0 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.602+0000 7fe4e1a71740 1 client.9036 shutdown 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.602+0000 7fe4e1a71740 1 -- 172.21.15.69:0/485110218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 msgr2=0x555d3983ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.602+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d3983ed10 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fe4c002e040 tx=0x7fe4c001a000 comp rx=0 tx=0).stop 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4e1a71740 1 -- 172.21.15.69:0/485110218 shutdown_connections 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x555d3983aa10 0x555d398a8e10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d398839d0 0x555d3983ed10 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555d39888980 0x555d3983f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4c4c00640 0 client.9036 ms_handle_reset on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4e1a71740 1 --2- 172.21.15.69:0/485110218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x555d3986baa0 0x555d398fdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4e1a71740 1 -- 172.21.15.69:0/485110218 >> 172.21.15.69:0/485110218 conn(0x555d3986b360 msgr2=0x555d3987cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4e1a71740 1 -- 172.21.15.69:0/485110218 shutdown_connections 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4e1a71740 1 -- 172.21.15.69:0/485110218 wait complete. 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4e1a71740 20 client.9036 trim_cache size 0 max 16384 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:19:18.608 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 8955, 2024-04-24T08:19:18.609 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:19:18.609 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:19:18.609 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:19:18.609 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 4.9113943520000003, 2024-04-24T08:19:18.609 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 89, 2024-04-24T08:19:18.609 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 98, 2024-04-24T08:19:18.609 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 98, 2024-04-24T08:19:18.609 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 5.9157672469999998 2024-04-24T08:19:18.609 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:19:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4dd600640 1 -- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4b406c8f0 msgr2=0x7fe4b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4dd600640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4b406c8f0 0x7fe4b406edb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fe4c0006fd0 tx=0x7fe4c0009b90 comp rx=0 tx=0).stop 2024-04-24T08:19:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4dd600640 1 -- 172.21.15.69:0/1916649150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 msgr2=0x7fe4d80ae260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4dd600640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 0x7fe4d80ae260 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8002130 tx=0x7fe4c800b4a0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.603+0000 7fe4dd600640 1 -- 172.21.15.69:0/1916649150 shutdown_connections 2024-04-24T08:19:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.604+0000 7fe4dd600640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4b406c8f0 0x7fe4b406edb0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.604+0000 7fe4dd600640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4d8159260 0x7fe4d80af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.604+0000 7fe4dd600640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4d80caf20 0x7fe4d81f43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.604+0000 7fe4dd600640 1 --2- 172.21.15.69:0/1916649150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4bc004d40 0x7fe4d80ae260 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.604+0000 7fe4dd600640 1 -- 172.21.15.69:0/1916649150 >> 172.21.15.69:0/1916649150 conn(0x7fe4d80d12c0 msgr2=0x7fe4d806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.604+0000 7fe4dd600640 1 -- 172.21.15.69:0/1916649150 shutdown_connections 2024-04-24T08:19:18.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.604+0000 7fe4dd600640 1 -- 172.21.15.69:0/1916649150 wait complete. 2024-04-24T08:19:18.622 DEBUG:teuthology.orchestra.run.smithi069:> sudo daemon-helper kill ceph --cluster ceph -w 2024-04-24T08:19:18.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:19:18.629 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-04-24T08:19:18.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.806+0000 7f4fb4e00640 1 Processor -- start 2024-04-24T08:19:18.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.806+0000 7f4fb4e00640 1 -- start start 2024-04-24T08:19:18.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.808+0000 7f4fb4e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4fb00ccfd0 0x7f4fb00ce600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.808+0000 7f4fb4e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4fb006c130 0x7f4fb00d0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.808+0000 7f4fb4e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fb00cd840 con 0x7f4fb00d07c0 2024-04-24T08:19:18.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.808+0000 7f4fb4e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4fb00ce170 con 0x7f4fb006c130 2024-04-24T08:19:18.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.808+0000 7f4fb4e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb00cdb20 con 0x7f4fb00ccfd0 2024-04-24T08:19:18.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.808+0000 7f4fae800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4fb006c130 0x7f4fb00d0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.808+0000 7f4fae800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4fb006c130 0x7f4fb00d0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42260/0 (socket says 172.21.15.69:42260) 2024-04-24T08:19:18.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.808+0000 7f4fae800640 1 -- 172.21.15.69:0/3440155158 learned_addr learned my addr 172.21.15.69:0/3440155158 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.809+0000 7f4fade00640 1 --2- 172.21.15.69:0/3440155158 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4fb00ccfd0 0x7f4fb00ce600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.809+0000 7f4fae800640 1 -- 172.21.15.69:0/3440155158 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4fb00ccfd0 msgr2=0x7f4fb00ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.809+0000 7f4fae800640 1 --2- 172.21.15.69:0/3440155158 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4fb00ccfd0 0x7f4fb00ce600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.809+0000 7f4fae800640 1 -- 172.21.15.69:0/3440155158 >> v1:172.21.15.121:6789/0 conn(0x7f4fb00d07c0 legacy=0x7f4fb006ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.809+0000 7f4fae800640 1 -- 172.21.15.69:0/3440155158 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb00cde00 con 0x7f4fb006c130 2024-04-24T08:19:18.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.810+0000 7f4fae800640 1 --2- 172.21.15.69:0/3440155158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 0x7f4fb00d0c20 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8009e50 tx=0x7f4fa802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597e83041b9e8f33 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.811+0000 7f4faca00640 1 -- 172.21.15.69:0/3440155158 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8036070 con 0x7f4fb006c130 2024-04-24T08:19:18.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.811+0000 7f4faca00640 1 -- 172.21.15.69:0/3440155158 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa8031600 con 0x7f4fb006c130 2024-04-24T08:19:18.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.811+0000 7f4faca00640 1 -- 172.21.15.69:0/3440155158 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa80319d0 con 0x7f4fb006c130 2024-04-24T08:19:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.811+0000 7f4fb4e00640 1 -- 172.21.15.69:0/3440155158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 msgr2=0x7f4fb00d0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.811+0000 7f4fb4e00640 1 --2- 172.21.15.69:0/3440155158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 0x7f4fb00d0c20 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8009e50 tx=0x7f4fa802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.812+0000 7f4fb4e00640 1 -- 172.21.15.69:0/3440155158 shutdown_connections 2024-04-24T08:19:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.812+0000 7f4fb4e00640 1 --2- 172.21.15.69:0/3440155158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 0x7f4fb00d0c20 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.812+0000 7f4fb4e00640 1 --2- 172.21.15.69:0/3440155158 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4fb00ccfd0 0x7f4fb00ce600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.812+0000 7f4fb4e00640 1 -- 172.21.15.69:0/3440155158 >> 172.21.15.69:0/3440155158 conn(0x7f4fb00d1410 msgr2=0x7f4fb00d1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.812+0000 7f4fb4e00640 1 -- 172.21.15.69:0/3440155158 shutdown_connections 2024-04-24T08:19:18.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.812+0000 7f4fb4e00640 1 -- 172.21.15.69:0/3440155158 wait complete. 2024-04-24T08:19:18.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.814+0000 7f4fb4e00640 1 Processor -- start 2024-04-24T08:19:18.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.815+0000 7f4fb4e00640 1 -- start start 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.816+0000 7f4fb4e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 0x7f4fb01557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.816+0000 7f4fb4e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00ccfd0 0x7f4fb00c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.816+0000 7f4fb4e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4fb00d07c0 0x7f4fb00c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.816+0000 7f4fb4e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb00d2200 con 0x7f4fb006c130 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.816+0000 7f4fb4e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb00d2080 con 0x7f4fb00ccfd0 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.816+0000 7f4fb4e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb00cba60 con 0x7f4fb00d07c0 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.816+0000 7f4fade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 0x7f4fb01557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.817+0000 7f4fade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 0x7f4fb01557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50650/0 (socket says 172.21.15.69:50650) 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.817+0000 7f4fade00640 1 -- 172.21.15.69:0/2111021378 learned_addr learned my addr 172.21.15.69:0/2111021378 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.817+0000 7f4fad400640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00ccfd0 0x7f4fb00c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.817+0000 7f4fade00640 1 -- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4fb00d07c0 msgr2=0x7f4fb00c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.817+0000 7f4fade00640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4fb00d07c0 0x7f4fb00c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.817+0000 7f4fade00640 1 -- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00ccfd0 msgr2=0x7f4fb00c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.817+0000 7f4fade00640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00ccfd0 0x7f4fb00c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.817+0000 7f4fade00640 1 -- 172.21.15.69:0/2111021378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb0155d00 con 0x7f4fb006c130 2024-04-24T08:19:18.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.817+0000 7f4fad400640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00ccfd0 0x7f4fb00c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:18.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.817+0000 7f4fade00640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 0x7f4fb01557c0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000d830 tx=0x7f4fa000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.818+0000 7f4fa6000640 1 -- 172.21.15.69:0/2111021378 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa000b6c0 con 0x7f4fb006c130 2024-04-24T08:19:18.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.818+0000 7f4fa6000640 1 -- 172.21.15.69:0/2111021378 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa0019070 con 0x7f4fb006c130 2024-04-24T08:19:18.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.818+0000 7f4fa6000640 1 -- 172.21.15.69:0/2111021378 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa000ba80 con 0x7f4fb006c130 2024-04-24T08:19:18.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.818+0000 7f4fb4e00640 1 -- 172.21.15.69:0/2111021378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb00c8ec0 con 0x7f4fb006c130 2024-04-24T08:19:18.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.818+0000 7f4fb4e00640 1 -- 172.21.15.69:0/2111021378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb00c9480 con 0x7f4fb006c130 2024-04-24T08:19:18.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.820+0000 7f4fa6000640 1 -- 172.21.15.69:0/2111021378 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4fa001e460 con 0x7f4fb006c130 2024-04-24T08:19:18.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.821+0000 7f4fa6000640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4f9006c8f0 0x7f4f9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.821+0000 7f4fa6000640 1 -- 172.21.15.69:0/2111021378 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7f4fa008c330 con 0x7f4fb006c130 2024-04-24T08:19:18.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.821+0000 7f4fad400640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4f9006c8f0 0x7f4f9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.822+0000 7f4fb7d9c740 1 Processor -- start 2024-04-24T08:19:18.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.822+0000 7f4fb7d9c740 1 -- start start 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fad400640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4f9006c8f0 0x7f4f9006edb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00cd460 tx=0x7f4f9c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a799df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7c9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7c9e40 0x55e88a79ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e88a703e70 con 0x55e88a7ca580 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e88a6aa380 con 0x55e88a7e7460 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55e88a6ffe30 con 0x55e88a7c9e40 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fad400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7c9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fad400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7c9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50660/0 (socket says 172.21.15.69:50660) 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fad400640 1 -- 172.21.15.69:0/1660172906 learned_addr learned my addr 172.21.15.69:0/1660172906 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fade00640 1 --2- 172.21.15.69:0/1660172906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a799df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fae800640 1 --2- 172.21.15.69:0/1660172906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7c9e40 0x55e88a79ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fad400640 1 -- 172.21.15.69:0/1660172906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7c9e40 msgr2=0x55e88a79ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fad400640 1 --2- 172.21.15.69:0/1660172906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7c9e40 0x55e88a79ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fad400640 1 -- 172.21.15.69:0/1660172906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 msgr2=0x55e88a799df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fad400640 1 --2- 172.21.15.69:0/1660172906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a799df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.823+0000 7f4fad400640 1 -- 172.21.15.69:0/1660172906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e88a4de7e0 con 0x55e88a7ca580 2024-04-24T08:19:18.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.824+0000 7f4fad400640 1 --2- 172.21.15.69:0/1660172906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7c9900 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00ca90 tx=0x7f4f9c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b836a6813714e1e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.824+0000 7f4f9aa00640 1 -- 172.21.15.69:0/1660172906 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c013db0 con 0x55e88a7ca580 2024-04-24T08:19:18.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.824+0000 7f4f9aa00640 1 -- 172.21.15.69:0/1660172906 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f9c00ae30 con 0x55e88a7ca580 2024-04-24T08:19:18.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.824+0000 7f4f9aa00640 1 -- 172.21.15.69:0/1660172906 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c004240 con 0x55e88a7ca580 2024-04-24T08:19:18.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.825+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/1660172906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 msgr2=0x55e88a7c9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.825+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/1660172906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7c9900 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00ca90 tx=0x7f4f9c00f000 comp rx=0 tx=0).stop 2024-04-24T08:19:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.825+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/1660172906 shutdown_connections 2024-04-24T08:19:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.825+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/1660172906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7c9900 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.825+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/1660172906 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7c9e40 0x55e88a79ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.825+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/1660172906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a799df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.825+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/1660172906 >> 172.21.15.69:0/1660172906 conn(0x55e88a79c060 msgr2=0x55e88a7e4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.825+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/1660172906 shutdown_connections 2024-04-24T08:19:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.825+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/1660172906 wait complete. 2024-04-24T08:19:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.826+0000 7f4fb7d9c740 1 -- start start 2024-04-24T08:19:18.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.826+0000 7f4fb7d9c740 1 Processor -- start 2024-04-24T08:19:18.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.827+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7ca580 0x55e88a7e4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.827+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a7cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.827+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.827+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e88a79e210 con 0x55e88a7cd5e0 2024-04-24T08:19:18.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.827+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e88a6bd660 con 0x55e88a7e7460 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.827+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55e88a79d600 con 0x55e88a7ca580 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.827+0000 7f4fb7d9c740 10 client.0 _ll_register_callbacks cb 0x55e88a6abaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fae800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fae800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7b3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50664/0 (socket says 172.21.15.69:50664) 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fae800640 1 -- 172.21.15.69:0/4207178390 learned_addr learned my addr 172.21.15.69:0/4207178390 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fad400640 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a7cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fade00640 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7ca580 0x55e88a7e4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fae800640 1 -- 172.21.15.69:0/4207178390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7ca580 msgr2=0x55e88a7e4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fae800640 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7ca580 0x55e88a7e4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fae800640 1 -- 172.21.15.69:0/4207178390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 msgr2=0x55e88a7cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fae800640 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a7cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fae800640 1 -- 172.21.15.69:0/4207178390 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e88a53ef70 con 0x55e88a7cd5e0 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fad400640 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a7cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4fae800640 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7b3ab0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8009f70 tx=0x7f4fa8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82295d9ecadf7c5f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.828+0000 7f4f8f400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:19:18.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.829+0000 7f4f8f400640 1 -- 172.21.15.69:0/4207178390 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8041070 con 0x55e88a7cd5e0 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.829+0000 7f4f8f400640 1 -- 172.21.15.69:0/4207178390 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7f4fa8031d40 con 0x55e88a7cd5e0 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.829+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/4207178390 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4f9c00ae30 con 0x55e88a7cd5e0 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.829+0000 7f4f8f400640 1 -- 172.21.15.69:0/4207178390 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f4fa804c050 con 0x55e88a7cd5e0 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.829+0000 7f4fb7d9c740 10 client.9048 fetch_fsmap learned FSMap version 89 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.829+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/4207178390 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x55e88a3d6170 con 0x55e88a7cd5e0 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.829+0000 7f4f8f400640 1 -- 172.21.15.69:0/4207178390 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f9c00ae30 con 0x55e88a7cd5e0 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.829+0000 7f4f8f400640 1 -- 172.21.15.69:0/4207178390 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7f4fa804f020 con 0x55e88a7cd5e0 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.829+0000 7f4fb7d9c740 10 client.9048 fetch_fsmap finished waiting for FSMap version 89 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.829+0000 7f4fb7d9c740 10 client.9048 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.830+0000 7f4fb7d9c740 20 client.9048 populate_metadata read hostname 'smithi069' 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.830+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55e88a7e2980 0x55e88a7e2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.830+0000 7f4fb7d9c740 4 client.9048 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.830+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/4207178390 --> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e88a711890 con 0x55e88a7e2980 2024-04-24T08:19:18.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.830+0000 7f4fade00640 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55e88a7e2980 0x55e88a7e2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.830+0000 7f4627e00640 1 Processor -- start 2024-04-24T08:19:18.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.830+0000 7f4627e00640 1 -- start start 2024-04-24T08:19:18.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.830+0000 7f4fade00640 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55e88a7e2980 0x55e88a7e2d90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36d65a318a1e225b server_cookie=265c22fabc88f68e in_seq=0 out_seq=0 2024-04-24T08:19:18.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.830+0000 7f4f8f400640 10 client.9048 ms_handle_connect on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:18.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.831+0000 7f4f8f400640 1 -- 172.21.15.69:0/4207178390 <== mds.0 v2:172.21.15.69:6833/3642138906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e88a711890 con 0x55e88a7e2980 2024-04-24T08:19:18.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.831+0000 7f4f8f400640 10 client.9048 handle_command_reply: tid=0 2024-04-24T08:19:18.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.831+0000 7f4fb7d9c740 1 client.9048 shutdown 2024-04-24T08:19:18.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/4207178390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 msgr2=0x55e88a7b3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4627e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f462815e8e0 0x7f46280d0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7b3ab0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8009f70 tx=0x7f4fa8004a40 comp rx=0 tx=0).stop 2024-04-24T08:19:18.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4627e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f462815ecc0 con 0x7f4628162ff0 2024-04-24T08:19:18.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4627e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f462806cd50 con 0x7f46280d2b80 2024-04-24T08:19:18.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4627e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f462806d080 con 0x7f462815e8e0 2024-04-24T08:19:18.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/4207178390 shutdown_connections 2024-04-24T08:19:18.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55e88a7e2980 0x55e88a7e2d90 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7b3ab0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a7cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/4207178390 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7ca580 0x55e88a7e4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4f8f400640 0 client.9048 ms_handle_reset on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:18.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/4207178390 >> 172.21.15.69:0/4207178390 conn(0x55e88a7c9e40 msgr2=0x55e88a799580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4626000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f4628162ff0 0x7f46280cf150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47892/0 (socket says 172.21.15.69:47892) 2024-04-24T08:19:18.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4626000640 1 -- 172.21.15.69:0/83438632 learned_addr learned my addr 172.21.15.69:0/83438632 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:18.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/4207178390 shutdown_connections 2024-04-24T08:19:18.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/4207178390 wait complete. 2024-04-24T08:19:18.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4fb7d9c740 20 client.9048 trim_cache size 0 max 16384 2024-04-24T08:19:18.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.832+0000 7f4626a00640 1 --2- 172.21.15.69:0/83438632 >> v2:172.21.15.121:3300/0 conn(0x7f462815e8e0 0x7f46280d0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 620098996 0 0) 0x7f462815ecc0 con 0x7f4628162ff0 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f460c003520 con 0x7f4628162ff0 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 477332136 0 0) 0x7f460c003520 con 0x7f4628162ff0 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 >> v2:172.21.15.121:3300/0 conn(0x7f462815e8e0 msgr2=0x7f46280d0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 --2- 172.21.15.69:0/83438632 >> v2:172.21.15.121:3300/0 conn(0x7f462815e8e0 0x7f46280d0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 >> v1:172.21.15.121:6789/0 conn(0x7f46280d2b80 legacy=0x7f462806d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46280d17f0 con 0x7f4628162ff0 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f4614001670 con 0x7f4628162ff0 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 >> v1:172.21.15.69:6789/0 conn(0x7f4628162ff0 legacy=0x7f46280cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f460c0039b0 0x7f460c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f460c004800 0x7f460c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460c0051c0 0x7f460c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f462806d080 con 0x7f460c004800 2024-04-24T08:19:18.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f460c00c060 con 0x7f460c0051c0 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f460c00c260 con 0x7f460c0039b0 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.833+0000 7f4626000640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f460c004800 0x7f460c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.834+0000 7f4626a00640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f460c0039b0 0x7f460c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.834+0000 7f4627400640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460c0051c0 0x7f460c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.834+0000 7f4626000640 1 -- 172.21.15.69:0/83438632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f460c0039b0 msgr2=0x7f460c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.834+0000 7f4626000640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f460c0039b0 0x7f460c003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.834+0000 7f4626000640 1 -- 172.21.15.69:0/83438632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460c0051c0 msgr2=0x7f460c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.834+0000 7f4626000640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460c0051c0 0x7f460c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.834+0000 7f4626000640 1 -- 172.21.15.69:0/83438632 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460c003520 con 0x7f460c004800 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.834+0000 7f4627400640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f460c0051c0 0x7f460c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.834+0000 7f4627400640 1 -- 172.21.15.69:0/83438632 reap_dead start 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.835+0000 7f4626000640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f460c004800 0x7f460c004c80 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f4628163470 tx=0x7f460c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe38789be24a067 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.835+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614007d30 con 0x7f460c004800 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.835+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46140058d0 con 0x7f460c004800 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.835+0000 7f4625600640 1 -- 172.21.15.69:0/83438632 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4614005d60 con 0x7f460c004800 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.836+0000 7f4627e00640 1 -- 172.21.15.69:0/83438632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f460c004800 msgr2=0x7f460c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.836+0000 7f4627e00640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f460c004800 0x7f460c004c80 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f4628163470 tx=0x7f460c005830 comp rx=0 tx=0).stop 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.836+0000 7f4627e00640 1 -- 172.21.15.69:0/83438632 shutdown_connections 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.836+0000 7f4627e00640 1 --2- 172.21.15.69:0/83438632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f460c004800 0x7f460c004c80 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.836+0000 7f4627e00640 1 -- 172.21.15.69:0/83438632 >> 172.21.15.69:0/83438632 conn(0x7f462808e030 msgr2=0x7f462809d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.836+0000 7f4627e00640 1 -- 172.21.15.69:0/83438632 shutdown_connections 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.836+0000 7f4627e00640 1 -- 172.21.15.69:0/83438632 wait complete. 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.838+0000 7f4627e00640 1 Processor -- start 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.838+0000 7f4627e00640 1 -- start start 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f460c004800 0x7f46280cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46280b43d0 0x7f46280b4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46280b6470 0x7f46280b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f46280d1db0 con 0x7f46280b6470 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f46280d1ad0 con 0x7f46280b43d0 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f46280d3030 con 0x7f460c004800 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46280b6470 0x7f46280b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46280b6470 0x7f46280b68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50692/0 (socket says 172.21.15.69:50692) 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627400640 1 -- 172.21.15.69:0/3420949237 learned_addr learned my addr 172.21.15.69:0/3420949237 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4626a00640 1 --2- 172.21.15.69:0/3420949237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f460c004800 0x7f46280cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4626000640 1 --2- 172.21.15.69:0/3420949237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46280b43d0 0x7f46280b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627400640 1 -- 172.21.15.69:0/3420949237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f460c004800 msgr2=0x7f46280cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.839+0000 7f4627400640 1 --2- 172.21.15.69:0/3420949237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f460c004800 0x7f46280cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.840+0000 7f4627400640 1 -- 172.21.15.69:0/3420949237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46280b43d0 msgr2=0x7f46280b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.840+0000 7f4627400640 1 --2- 172.21.15.69:0/3420949237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46280b43d0 0x7f46280b4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.840+0000 7f4627400640 1 -- 172.21.15.69:0/3420949237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4628152610 con 0x7f46280b6470 2024-04-24T08:19:18.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.840+0000 7f4626a00640 1 --2- 172.21.15.69:0/3420949237 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f460c004800 0x7f46280cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:18.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.840+0000 7f4627400640 1 --2- 172.21.15.69:0/3420949237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46280b6470 0x7f46280b68f0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f462808daa0 tx=0x7f462000d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.840+0000 7f461ea00640 1 -- 172.21.15.69:0/3420949237 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620018070 con 0x7f46280b6470 2024-04-24T08:19:18.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.840+0000 7f461ea00640 1 -- 172.21.15.69:0/3420949237 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4620003da0 con 0x7f46280b6470 2024-04-24T08:19:18.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.840+0000 7f461ea00640 1 -- 172.21.15.69:0/3420949237 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4620005310 con 0x7f46280b6470 2024-04-24T08:19:18.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.840+0000 7f4627e00640 1 -- 172.21.15.69:0/3420949237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46281528f0 con 0x7f46280b6470 2024-04-24T08:19:18.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.840+0000 7f4627e00640 1 -- 172.21.15.69:0/3420949237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4628152eb0 con 0x7f46280b6470 2024-04-24T08:19:18.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.841+0000 7f4627e00640 1 -- 172.21.15.69:0/3420949237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f4628075d50 con 0x7f46280b6470 2024-04-24T08:19:18.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.842+0000 7f461ea00640 1 -- 172.21.15.69:0/3420949237 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f462001b070 con 0x7f46280b6470 2024-04-24T08:19:18.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.843+0000 7f461ea00640 1 --2- 172.21.15.69:0/3420949237 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f460806c8f0 0x7f460806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.843+0000 7f461ea00640 1 -- 172.21.15.69:0/3420949237 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7f462008baf0 con 0x7f46280b6470 2024-04-24T08:19:18.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.843+0000 7f461ea00640 1 -- 172.21.15.69:0/3420949237 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+516 (secure 0 0 0) 0x7f462008be20 con 0x7f46280b6470 2024-04-24T08:19:18.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.844+0000 7f4627e00640 1 -- 172.21.15.69:0/3420949237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({log-info=0+}) -- 0x7f462806cb30 con 0x7f46280b6470 2024-04-24T08:19:18.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.844+0000 7f4626a00640 1 --2- 172.21.15.69:0/3420949237 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f460806c8f0 0x7f460806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.845+0000 7f4626a00640 1 --2- 172.21.15.69:0/3420949237 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f460806c8f0 0x7f460806edb0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4618009a60 tx=0x7f4618008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.852+0000 7f4fb7d9c740 1 Processor -- start 2024-04-24T08:19:18.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.853+0000 7f4fb7d9c740 1 -- start start 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.853+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7e3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.853+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7cd5e0 0x55e88a809860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.853+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a7e7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.853+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb0200f80 con 0x55e88a7ca580 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.853+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb01fffc0 con 0x55e88a7e7460 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.853+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb00b6740 con 0x55e88a7cd5e0 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4fade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7e3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4fade00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7e3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50706/0 (socket says 172.21.15.69:50706) 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4fade00640 1 -- 172.21.15.69:0/1280485703 learned_addr learned my addr 172.21.15.69:0/1280485703 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4fade00640 1 -- 172.21.15.69:0/1280485703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7cd5e0 msgr2=0x55e88a809860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4fade00640 1 --2- 172.21.15.69:0/1280485703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7cd5e0 0x55e88a809860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4fade00640 1 -- 172.21.15.69:0/1280485703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 msgr2=0x55e88a7e7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4fade00640 1 --2- 172.21.15.69:0/1280485703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a7e7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4fade00640 1 -- 172.21.15.69:0/1280485703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e88a79d1a0 con 0x55e88a7ca580 2024-04-24T08:19:18.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4fade00640 1 --2- 172.21.15.69:0/1280485703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7e3810 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f4fa008a760 tx=0x7f4fa0007400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f764bf18110e702 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4f9a000640 1 -- 172.21.15.69:0/1280485703 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0059f60 con 0x55e88a7ca580 2024-04-24T08:19:18.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4f9a000640 1 -- 172.21.15.69:0/1280485703 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa005e310 con 0x55e88a7ca580 2024-04-24T08:19:18.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.854+0000 7f4f9a000640 1 -- 172.21.15.69:0/1280485703 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa0062b40 con 0x55e88a7ca580 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.855+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/1280485703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 msgr2=0x55e88a7e3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.855+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/1280485703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7e3810 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f4fa008a760 tx=0x7f4fa0007400 comp rx=0 tx=0).stop 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.855+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/1280485703 shutdown_connections 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.855+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/1280485703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a7e7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.855+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/1280485703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7cd5e0 0x55e88a809860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.855+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/1280485703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7ca580 0x55e88a7e3810 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.855+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/1280485703 >> 172.21.15.69:0/1280485703 conn(0x55e88a7c9e40 msgr2=0x55e88a7e4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.855+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/1280485703 shutdown_connections 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.855+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/1280485703 wait complete. 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.856+0000 7f4fb7d9c740 1 -- start start 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.856+0000 7f4fb7d9c740 1 Processor -- start 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.857+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7ca580 0x55e88a85ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.857+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7daca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.857+0000 7f4fb7d9c740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a798c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.857+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e88a7dc6e0 con 0x55e88a7cd5e0 2024-04-24T08:19:18.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.857+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb00c5b60 con 0x55e88a7e7460 2024-04-24T08:19:18.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.857+0000 7f4fb7d9c740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55e88a7db4b0 con 0x55e88a7ca580 2024-04-24T08:19:18.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.857+0000 7f4fb7d9c740 10 client.0 _ll_register_callbacks cb 0x7f4fa8034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:18.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.857+0000 7f4fad400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7daca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.857+0000 7f4fad400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7daca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50708/0 (socket says 172.21.15.69:50708) 2024-04-24T08:19:18.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.857+0000 7f4fad400640 1 -- 172.21.15.69:0/159661298 learned_addr learned my addr 172.21.15.69:0/159661298 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:18.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4fade00640 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7ca580 0x55e88a85ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4fad400640 1 -- 172.21.15.69:0/159661298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7ca580 msgr2=0x55e88a85ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4fae800640 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a798c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4fad400640 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7ca580 0x55e88a85ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4fad400640 1 -- 172.21.15.69:0/159661298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 msgr2=0x55e88a798c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:18.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4fad400640 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a798c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4fad400640 1 -- 172.21.15.69:0/159661298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e88a7f7910 con 0x55e88a7cd5e0 2024-04-24T08:19:18.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4fad400640 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7daca0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00ca60 tx=0x7f4f9c004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb36b9a6bed53e90 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4f8f400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4f8f400640 1 -- 172.21.15.69:0/159661298 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c004030 con 0x55e88a7cd5e0 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4f8f400640 1 -- 172.21.15.69:0/159661298 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7f4f9c0041d0 con 0x55e88a7cd5e0 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/159661298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fa005e310 con 0x55e88a7cd5e0 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.858+0000 7f4f8f400640 1 -- 172.21.15.69:0/159661298 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f4f9c0043d0 con 0x55e88a7cd5e0 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4fb7d9c740 10 client.9060 fetch_fsmap learned FSMap version 89 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/159661298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x55e88a7db1e0 con 0x55e88a7cd5e0 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4f8f400640 1 -- 172.21.15.69:0/159661298 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa005e310 con 0x55e88a7cd5e0 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4f8f400640 1 -- 172.21.15.69:0/159661298 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7f4f9c03a080 con 0x55e88a7cd5e0 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4fb7d9c740 10 client.9060 fetch_fsmap finished waiting for FSMap version 89 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4fb7d9c740 10 client.9060 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4fb7d9c740 20 client.9060 populate_metadata read hostname 'smithi069' 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55e88a7c6d90 0x55e88a7991c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:18.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4fb7d9c740 4 client.9060 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-04-24T08:19:18.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/159661298 --> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) -- 0x55e88a70b700 con 0x55e88a7c6d90 2024-04-24T08:19:18.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.859+0000 7f4fade00640 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55e88a7c6d90 0x55e88a7991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:18.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.860+0000 7f4fade00640 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55e88a7c6d90 0x55e88a7991c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54b246b7b5b9f6aa server_cookie=ac5ee50b62a5d5d3 in_seq=0 out_seq=0 2024-04-24T08:19:18.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.860+0000 7f4f8f400640 10 client.9060 ms_handle_connect on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:18.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.860+0000 7f4f8f400640 1 -- 172.21.15.69:0/159661298 <== mds.0 v2:172.21.15.69:6833/3642138906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e88a70b700 con 0x55e88a7c6d90 2024-04-24T08:19:18.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.860+0000 7f4f8f400640 10 client.9060 handle_command_reply: tid=0 2024-04-24T08:19:18.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.860+0000 7f4fb7d9c740 1 client.9060 shutdown 2024-04-24T08:19:18.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/159661298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 msgr2=0x55e88a7daca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7daca0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c00ca60 tx=0x7f4f9c004680 comp rx=0 tx=0).stop 2024-04-24T08:19:18.865 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:19:18.861+0000 7fe159200640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-04-24T08:19:18.865 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:19:18.861+0000 7fe159200640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-04-24T08:19:18.866 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:19:18.861+0000 7fe159200640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-04-24T08:19:18.866 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:19:18.861+0000 7fe159200640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-04-24T08:19:18.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/159661298 shutdown_connections 2024-04-24T08:19:18.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55e88a7c6d90 0x55e88a7991c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e88a7cd5e0 0x55e88a7daca0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e88a7e7460 0x55e88a798c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4fb7d9c740 1 --2- 172.21.15.69:0/159661298 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55e88a7ca580 0x55e88a85ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/159661298 >> 172.21.15.69:0/159661298 conn(0x55e88a7c9e40 msgr2=0x55e88a7b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4f8f400640 0 client.9060 ms_handle_reset on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:18.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/159661298 shutdown_connections 2024-04-24T08:19:18.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.861+0000 7f4fb7d9c740 1 -- 172.21.15.69:0/159661298 wait complete. 2024-04-24T08:19:18.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb7d9c740 20 client.9060 trim_cache size 0 max 16384 2024-04-24T08:19:18.867 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:19:18.867 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-04-24T08:19:18.868 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "76c45099-d189-4cc9-8efe-7618923fdc69", 2024-04-24T08:19:18.868 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-04-24T08:19:18.868 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:19:18.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb4e00640 1 -- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4f9006c8f0 msgr2=0x7f4f9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb4e00640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4f9006c8f0 0x7f4f9006edb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00cd460 tx=0x7f4f9c008040 comp rx=0 tx=0).stop 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb4e00640 1 -- 172.21.15.69:0/2111021378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 msgr2=0x7f4fb01557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb4e00640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 0x7f4fb01557c0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f4fa000d830 tx=0x7f4fa000dcf0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb4e00640 1 -- 172.21.15.69:0/2111021378 shutdown_connections 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb4e00640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4f9006c8f0 0x7f4f9006edb0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb4e00640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4fb00d07c0 0x7f4fb00c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb4e00640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00ccfd0 0x7f4fb00c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb4e00640 1 --2- 172.21.15.69:0/2111021378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fb006c130 0x7f4fb01557c0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.862+0000 7f4fb4e00640 1 -- 172.21.15.69:0/2111021378 >> 172.21.15.69:0/2111021378 conn(0x7f4fb00d1410 msgr2=0x7f4fb00cbe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.863+0000 7f4fb4e00640 1 -- 172.21.15.69:0/2111021378 shutdown_connections 2024-04-24T08:19:18.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:18.863+0000 7f4fb4e00640 1 -- 172.21.15.69:0/2111021378 wait complete. 2024-04-24T08:19:18.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:19:18.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-24T08:19:19.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.054+0000 7fd96d800640 1 Processor -- start 2024-04-24T08:19:19.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.055+0000 7fd96d800640 1 -- start start 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.056+0000 7fd96d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9681420a0 0x7fd968071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.056+0000 7fd96d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd968142480 0x7fd96815ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.056+0000 7fd96d800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd968072450 con 0x7fd968160510 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.056+0000 7fd96d800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd968160960 con 0x7fd9681420a0 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.056+0000 7fd96d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9680cf1e0 con 0x7fd968142480 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.057+0000 7fd967e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9681420a0 0x7fd968071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.057+0000 7fd96ce00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd968160510 0x7fd96815c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:47906/0 (socket says 172.21.15.69:47906) 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.057+0000 7fd96ce00640 1 -- 172.21.15.69:0/2121876933 learned_addr learned my addr 172.21.15.69:0/2121876933 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.057+0000 7fd967400640 1 --2- 172.21.15.69:0/2121876933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd968142480 0x7fd96815ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.057+0000 7fd967e00640 1 -- 172.21.15.69:0/2121876933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd968142480 msgr2=0x7fd96815ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.057+0000 7fd967e00640 1 --2- 172.21.15.69:0/2121876933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd968142480 0x7fd96815ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.057+0000 7fd967e00640 1 -- 172.21.15.69:0/2121876933 >> v1:172.21.15.69:6789/0 conn(0x7fd968160510 legacy=0x7fd96815c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:19.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.057+0000 7fd967e00640 1 -- 172.21.15.69:0/2121876933 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd968072730 con 0x7fd9681420a0 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.058+0000 7fd967e00640 1 --2- 172.21.15.69:0/2121876933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9681420a0 0x7fd968071d70 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fd9580097a0 tx=0x7fd95802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3acd73fe231766 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd966a00640 1 -- 172.21.15.69:0/2121876933 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958037070 con 0x7fd9681420a0 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd966a00640 1 -- 172.21.15.69:0/2121876933 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd958032440 con 0x7fd9681420a0 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd966a00640 1 -- 172.21.15.69:0/2121876933 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958032810 con 0x7fd9681420a0 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd96d800640 1 -- 172.21.15.69:0/2121876933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9681420a0 msgr2=0x7fd968071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd96d800640 1 --2- 172.21.15.69:0/2121876933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9681420a0 0x7fd968071d70 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7fd9580097a0 tx=0x7fd95802a180 comp rx=0 tx=0).stop 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd96d800640 1 -- 172.21.15.69:0/2121876933 shutdown_connections 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd96d800640 1 --2- 172.21.15.69:0/2121876933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd968142480 0x7fd96815ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd96d800640 1 --2- 172.21.15.69:0/2121876933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9681420a0 0x7fd968071d70 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd96d800640 1 -- 172.21.15.69:0/2121876933 >> 172.21.15.69:0/2121876933 conn(0x7fd968067b80 msgr2=0x7fd96806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd96d800640 1 -- 172.21.15.69:0/2121876933 shutdown_connections 2024-04-24T08:19:19.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.059+0000 7fd96d800640 1 -- 172.21.15.69:0/2121876933 wait complete. 2024-04-24T08:19:19.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.061+0000 7fd96d800640 1 Processor -- start 2024-04-24T08:19:19.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.061+0000 7fd96d800640 1 -- start start 2024-04-24T08:19:19.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.062+0000 7fd96d800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9681420a0 0x7fd968154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.062+0000 7fd96d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd968142480 0x7fd9681f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.062+0000 7fd96d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd968160510 0x7fd968140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.062+0000 7fd96d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681428e0 con 0x7fd968160510 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.062+0000 7fd96d800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd968072cf0 con 0x7fd9681420a0 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.062+0000 7fd96d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd96815d450 con 0x7fd968142480 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.062+0000 7fd96ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd968160510 0x7fd968140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.062+0000 7fd96ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd968160510 0x7fd968140c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50718/0 (socket says 172.21.15.69:50718) 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.062+0000 7fd96ce00640 1 -- 172.21.15.69:0/2376111823 learned_addr learned my addr 172.21.15.69:0/2376111823 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd967400640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd968142480 0x7fd9681f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd967e00640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9681420a0 0x7fd968154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd96ce00640 1 -- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd968142480 msgr2=0x7fd9681f3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd96ce00640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd968142480 0x7fd9681f3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd96ce00640 1 -- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9681420a0 msgr2=0x7fd968154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd96ce00640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9681420a0 0x7fd968154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd96ce00640 1 -- 172.21.15.69:0/2376111823 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96815c840 con 0x7fd968160510 2024-04-24T08:19:19.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd96ce00640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd968160510 0x7fd968140c40 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fd96000ca80 tx=0x7fd96000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:19.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd95fe00640 1 -- 172.21.15.69:0/2376111823 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960017070 con 0x7fd968160510 2024-04-24T08:19:19.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd95fe00640 1 -- 172.21.15.69:0/2376111823 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd96000ac30 con 0x7fd968160510 2024-04-24T08:19:19.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd95fe00640 1 -- 172.21.15.69:0/2376111823 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960012600 con 0x7fd968160510 2024-04-24T08:19:19.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.063+0000 7fd96d800640 1 -- 172.21.15.69:0/2376111823 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96815cb20 con 0x7fd968160510 2024-04-24T08:19:19.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.064+0000 7fd96d800640 1 -- 172.21.15.69:0/2376111823 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96815d0e0 con 0x7fd968160510 2024-04-24T08:19:19.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.066+0000 7fd95fe00640 1 -- 172.21.15.69:0/2376111823 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd96001a070 con 0x7fd968160510 2024-04-24T08:19:19.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.067+0000 7fd95fe00640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd94806c8f0 0x7fd94806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.067+0000 7fd95fe00640 1 -- 172.21.15.69:0/2376111823 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fd96008b870 con 0x7fd968160510 2024-04-24T08:19:19.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.068+0000 7fd967e00640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd94806c8f0 0x7fd94806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.068+0000 7fd971de9740 1 Processor -- start 2024-04-24T08:19:19.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.068+0000 7fd971de9740 1 -- start start 2024-04-24T08:19:19.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.069+0000 7fd971de9740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557228424e00 0x5572283d7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.069+0000 7fd971de9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284072a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.069+0000 7fd971de9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5572284077e0 0x5572283dc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.069+0000 7fd971de9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5572283411f0 con 0x557228407f20 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.069+0000 7fd971de9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5572282e7350 con 0x5572284077e0 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.069+0000 7fd971de9740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55722833d1b0 con 0x557228424e00 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.069+0000 7fd967e00640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd94806c8f0 0x7fd94806edb0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd958034000 tx=0x7fd95802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.069+0000 7fd967400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284072a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.069+0000 7fd967400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284072a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50728/0 (socket says 172.21.15.69:50728) 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.069+0000 7fd967400640 1 -- 172.21.15.69:0/2871452462 learned_addr learned my addr 172.21.15.69:0/2871452462 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd96ce00640 1 --2- 172.21.15.69:0/2871452462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5572284077e0 0x5572283dc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd967e00640 1 --2- 172.21.15.69:0/2871452462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557228424e00 0x5572283d7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd967400640 1 -- 172.21.15.69:0/2871452462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557228424e00 msgr2=0x5572283d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:19.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd967400640 1 --2- 172.21.15.69:0/2871452462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557228424e00 0x5572283d7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd967400640 1 -- 172.21.15.69:0/2871452462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5572284077e0 msgr2=0x5572283dc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:19.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd967400640 1 --2- 172.21.15.69:0/2871452462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5572284077e0 0x5572283dc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd967400640 1 -- 172.21.15.69:0/2871452462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55722833bf50 con 0x557228407f20 2024-04-24T08:19:19.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd96ce00640 1 --2- 172.21.15.69:0/2871452462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5572284077e0 0x5572283dc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:19.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd967400640 1 --2- 172.21.15.69:0/2871452462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284072a0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fd95400ca30 tx=0x7fd95400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda9b76fcbf0b9d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:19.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd95cc00640 1 -- 172.21.15.69:0/2871452462 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954018030 con 0x557228407f20 2024-04-24T08:19:19.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd95cc00640 1 -- 172.21.15.69:0/2871452462 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95400eb40 con 0x557228407f20 2024-04-24T08:19:19.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.070+0000 7fd95cc00640 1 -- 172.21.15.69:0/2871452462 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95400ed10 con 0x557228407f20 2024-04-24T08:19:19.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.071+0000 7fd971de9740 1 -- 172.21.15.69:0/2871452462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 msgr2=0x5572284072a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:19.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.071+0000 7fd971de9740 1 --2- 172.21.15.69:0/2871452462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284072a0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fd95400ca30 tx=0x7fd95400cef0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.071+0000 7fd971de9740 1 -- 172.21.15.69:0/2871452462 shutdown_connections 2024-04-24T08:19:19.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.071+0000 7fd971de9740 1 --2- 172.21.15.69:0/2871452462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284072a0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.071+0000 7fd971de9740 1 --2- 172.21.15.69:0/2871452462 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5572284077e0 0x5572283dc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.071+0000 7fd971de9740 1 --2- 172.21.15.69:0/2871452462 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557228424e00 0x5572283d7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.071+0000 7fd971de9740 1 -- 172.21.15.69:0/2871452462 >> 172.21.15.69:0/2871452462 conn(0x5572283d9a00 msgr2=0x5572284222a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:19.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.071+0000 7fd971de9740 1 -- 172.21.15.69:0/2871452462 shutdown_connections 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.071+0000 7fd971de9740 1 -- 172.21.15.69:0/2871452462 wait complete. 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.072+0000 7fd971de9740 1 -- start start 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.072+0000 7fd971de9740 1 Processor -- start 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.073+0000 7fd971de9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x55722840a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.073+0000 7fd971de9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228424e00 0x557228425c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.073+0000 7fd971de9740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5572283f7b50 0x5572284261a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.073+0000 7fd971de9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5572283dbbb0 con 0x557228407f20 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.073+0000 7fd971de9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5572282fa690 con 0x557228424e00 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.073+0000 7fd971de9740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5572283dafa0 con 0x5572283f7b50 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.073+0000 7fd971de9740 10 client.0 _ll_register_callbacks cb 0x557228336650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.073+0000 7fd967e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x55722840a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.073+0000 7fd967e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x55722840a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50740/0 (socket says 172.21.15.69:50740) 2024-04-24T08:19:19.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.073+0000 7fd967e00640 1 -- 172.21.15.69:0/1741053383 learned_addr learned my addr 172.21.15.69:0/1741053383 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd967e00640 1 -- 172.21.15.69:0/1741053383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5572283f7b50 msgr2=0x5572284261a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd96ce00640 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5572283f7b50 0x5572284261a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd967e00640 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5572283f7b50 0x5572284261a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd967e00640 1 -- 172.21.15.69:0/1741053383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228424e00 msgr2=0x557228425c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd967e00640 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228424e00 0x557228425c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd967e00640 1 -- 172.21.15.69:0/1741053383 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572282f9920 con 0x557228407f20 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd967e00640 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x55722840a950 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fd958035040 tx=0x7fd958004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83041b4af929969d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd951600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd951600640 1 -- 172.21.15.69:0/1741053383 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958037040 con 0x557228407f20 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd951600640 1 -- 172.21.15.69:0/1741053383 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fd95803eb30 con 0x557228407f20 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd971de9740 1 -- 172.21.15.69:0/1741053383 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd95400eb40 con 0x557228407f20 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.074+0000 7fd951600640 1 -- 172.21.15.69:0/1741053383 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fd958055050 con 0x557228407f20 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd971de9740 10 client.9075 fetch_fsmap learned FSMap version 89 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd971de9740 1 -- 172.21.15.69:0/1741053383 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x55722811d7d0 con 0x557228407f20 2024-04-24T08:19:19.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd951600640 1 -- 172.21.15.69:0/1741053383 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95400eb40 con 0x557228407f20 2024-04-24T08:19:19.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd951600640 1 -- 172.21.15.69:0/1741053383 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7fd958058020 con 0x557228407f20 2024-04-24T08:19:19.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd971de9740 10 client.9075 fetch_fsmap finished waiting for FSMap version 89 2024-04-24T08:19:19.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd971de9740 10 client.9075 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:19:19.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd971de9740 20 client.9075 populate_metadata read hostname 'smithi069' 2024-04-24T08:19:19.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd971de9740 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x557228419980 0x557228419e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd971de9740 4 client.9075 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:19.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd971de9740 1 -- 172.21.15.69:0/1741053383 --> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55722834ec10 con 0x557228419980 2024-04-24T08:19:19.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.075+0000 7fd967400640 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x557228419980 0x557228419e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.076+0000 7fd967400640 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x557228419980 0x557228419e00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c28861f205aab35 server_cookie=6ea9985bc5fed85a in_seq=0 out_seq=0 2024-04-24T08:19:19.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.076+0000 7fd951600640 10 client.9075 ms_handle_connect on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:19.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.076+0000 7fd951600640 1 -- 172.21.15.69:0/1741053383 <== mds.0 v2:172.21.15.69:6833/3642138906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55722834ec10 con 0x557228419980 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.076+0000 7fd951600640 10 client.9075 handle_command_reply: tid=0 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.076+0000 7fd971de9740 1 client.9075 shutdown 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 1 -- 172.21.15.69:0/1741053383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 msgr2=0x55722840a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x55722840a950 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7fd958035040 tx=0x7fd958004730 comp rx=0 tx=0).stop 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 1 -- 172.21.15.69:0/1741053383 shutdown_connections 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x557228419980 0x557228419e00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5572283f7b50 0x5572284261a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228424e00 0x557228425c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd951600640 0 client.9075 ms_handle_reset on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 1 --2- 172.21.15.69:0/1741053383 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x55722840a950 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 1 -- 172.21.15.69:0/1741053383 >> 172.21.15.69:0/1741053383 conn(0x5572284077e0 msgr2=0x557228422250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 1 -- 172.21.15.69:0/1741053383 shutdown_connections 2024-04-24T08:19:19.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 1 -- 172.21.15.69:0/1741053383 wait complete. 2024-04-24T08:19:19.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.077+0000 7fd971de9740 20 client.9075 trim_cache size 0 max 16384 2024-04-24T08:19:19.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.096+0000 7fd971de9740 1 Processor -- start 2024-04-24T08:19:19.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.096+0000 7fd971de9740 1 -- start start 2024-04-24T08:19:19.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.096+0000 7fd971de9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5572283f7b50 0x5572283d72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.096+0000 7fd971de9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228407f20 0x5572284217b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.096+0000 7fd971de9740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557228424e00 0x5572284347b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd971de9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd968200df0 con 0x5572283f7b50 2024-04-24T08:19:19.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd971de9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd9681ffe40 con 0x557228407f20 2024-04-24T08:19:19.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd971de9740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd968156480 con 0x557228424e00 2024-04-24T08:19:19.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd967e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5572283f7b50 0x5572283d72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd967e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5572283f7b50 0x5572283d72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50744/0 (socket says 172.21.15.69:50744) 2024-04-24T08:19:19.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd967e00640 1 -- 172.21.15.69:0/1856315725 learned_addr learned my addr 172.21.15.69:0/1856315725 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:19.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd967e00640 1 -- 172.21.15.69:0/1856315725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557228424e00 msgr2=0x5572284347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:19.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd967e00640 1 --2- 172.21.15.69:0/1856315725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557228424e00 0x5572284347b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd967e00640 1 -- 172.21.15.69:0/1856315725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228407f20 msgr2=0x5572284217b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:19.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd967e00640 1 --2- 172.21.15.69:0/1856315725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228407f20 0x5572284217b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd967e00640 1 -- 172.21.15.69:0/1856315725 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572280533f0 con 0x5572283f7b50 2024-04-24T08:19:19.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.097+0000 7fd967e00640 1 --2- 172.21.15.69:0/1856315725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5572283f7b50 0x5572283d72a0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fd95802a770 tx=0x7fd958040070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70fd186b2ca98c12 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:19.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd953e00640 1 -- 172.21.15.69:0/1856315725 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958037070 con 0x5572283f7b50 2024-04-24T08:19:19.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd953e00640 1 -- 172.21.15.69:0/1856315725 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95804d070 con 0x5572283f7b50 2024-04-24T08:19:19.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd953e00640 1 -- 172.21.15.69:0/1856315725 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95803c630 con 0x5572283f7b50 2024-04-24T08:19:19.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd971de9740 1 -- 172.21.15.69:0/1856315725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5572283f7b50 msgr2=0x5572283d72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:19.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd971de9740 1 --2- 172.21.15.69:0/1856315725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5572283f7b50 0x5572283d72a0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7fd95802a770 tx=0x7fd958040070 comp rx=0 tx=0).stop 2024-04-24T08:19:19.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd971de9740 1 -- 172.21.15.69:0/1856315725 shutdown_connections 2024-04-24T08:19:19.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd971de9740 1 --2- 172.21.15.69:0/1856315725 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557228424e00 0x5572284347b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd971de9740 1 --2- 172.21.15.69:0/1856315725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228407f20 0x5572284217b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd971de9740 1 --2- 172.21.15.69:0/1856315725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5572283f7b50 0x5572283d72a0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd971de9740 1 -- 172.21.15.69:0/1856315725 >> 172.21.15.69:0/1856315725 conn(0x5572284077e0 msgr2=0x557228419d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd971de9740 1 -- 172.21.15.69:0/1856315725 shutdown_connections 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.098+0000 7fd971de9740 1 -- 172.21.15.69:0/1856315725 wait complete. 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.099+0000 7fd971de9740 1 -- start start 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.099+0000 7fd971de9740 1 Processor -- start 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.100+0000 7fd971de9740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5572283f7b50 0x557228417c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.100+0000 7fd971de9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284233d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.100+0000 7fd971de9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228424e00 0x5572284347b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.100+0000 7fd971de9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557228421cf0 con 0x557228407f20 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.100+0000 7fd971de9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd96815e580 con 0x557228424e00 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.100+0000 7fd971de9740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5572283d78e0 con 0x5572283f7b50 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.100+0000 7fd971de9740 10 client.0 _ll_register_callbacks cb 0x7fd958035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:19.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.100+0000 7fd967400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284233d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.100+0000 7fd967400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284233d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50748/0 (socket says 172.21.15.69:50748) 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.100+0000 7fd967400640 1 -- 172.21.15.69:0/1945313496 learned_addr learned my addr 172.21.15.69:0/1945313496 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd967e00640 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5572283f7b50 0x557228417c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd967400640 1 -- 172.21.15.69:0/1945313496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5572283f7b50 msgr2=0x557228417c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd967400640 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5572283f7b50 0x557228417c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd967400640 1 -- 172.21.15.69:0/1945313496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228424e00 msgr2=0x5572284347b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd96ce00640 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228424e00 0x5572284347b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd967400640 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228424e00 0x5572284347b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd967400640 1 -- 172.21.15.69:0/1945313496 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572280130f0 con 0x557228407f20 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd967400640 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284233d0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fd954006fd0 tx=0x7fd95400c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99fac059d323b74 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd951600640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd951600640 1 -- 172.21.15.69:0/1945313496 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95400fe80 con 0x557228407f20 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd951600640 1 -- 172.21.15.69:0/1945313496 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fd954004030 con 0x557228407f20 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd971de9740 1 -- 172.21.15.69:0/1945313496 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd95804d070 con 0x557228407f20 2024-04-24T08:19:19.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd951600640 1 -- 172.21.15.69:0/1945313496 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fd95402d050 con 0x557228407f20 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd971de9740 10 client.9081 fetch_fsmap learned FSMap version 89 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.101+0000 7fd971de9740 1 -- 172.21.15.69:0/1945313496 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x557228306810 con 0x557228407f20 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd951600640 1 -- 172.21.15.69:0/1945313496 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95804d070 con 0x557228407f20 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd951600640 1 -- 172.21.15.69:0/1945313496 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7fd954030020 con 0x557228407f20 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd971de9740 10 client.9081 fetch_fsmap finished waiting for FSMap version 89 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd971de9740 10 client.9081 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd971de9740 20 client.9081 populate_metadata read hostname 'smithi069' 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd971de9740 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55722849b700 0x557228434cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd971de9740 4 client.9081 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd971de9740 1 -- 172.21.15.69:0/1945313496 --> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55722841b790 con 0x55722849b700 2024-04-24T08:19:19.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd967e00640 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55722849b700 0x557228434cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:19.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd967e00640 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55722849b700 0x557228434cf0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e99bb8fa36b5914 server_cookie=ef4d244225de08ad in_seq=0 out_seq=0 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.102+0000 7fd951600640 10 client.9081 ms_handle_connect on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd951600640 1 -- 172.21.15.69:0/1945313496 <== mds.0 v2:172.21.15.69:6833/3642138906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+260 (crc 0 0 0) 0x55722841b790 con 0x55722849b700 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd951600640 10 client.9081 handle_command_reply: tid=0 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 client.9081 shutdown 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 -- 172.21.15.69:0/1945313496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 msgr2=0x5572284233d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284233d0 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7fd954006fd0 tx=0x7fd95400c720 comp rx=0 tx=0).stop 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 -- 172.21.15.69:0/1945313496 shutdown_connections 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55722849b700 0x557228434cf0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557228407f20 0x5572284233d0 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557228424e00 0x5572284347b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 --2- 172.21.15.69:0/1945313496 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5572283f7b50 0x557228417c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd951600640 0 client.9081 ms_handle_reset on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 -- 172.21.15.69:0/1945313496 >> 172.21.15.69:0/1945313496 conn(0x5572284077e0 msgr2=0x5572283d6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 -- 172.21.15.69:0/1945313496 shutdown_connections 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 1 -- 172.21.15.69:0/1945313496 wait complete. 2024-04-24T08:19:19.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.103+0000 7fd971de9740 20 client.9081 trim_cache size 0 max 16384 2024-04-24T08:19:19.109 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:19:19.109 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:19:19.109 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-04-24T08:19:19.109 INFO:teuthology.orchestra.run.smithi069.stdout: "76c45099-d189-4cc9-8efe-7618923fdc69": { 2024-04-24T08:19:19.109 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/", 2024-04-24T08:19:19.109 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "76c45099-d189-4cc9-8efe-7618923fdc69", 2024-04-24T08:19:19.109 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,repair" 2024-04-24T08:19:19.109 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:19:19.109 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:19:19.109 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:19:19.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 -- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd94806c8f0 msgr2=0x7fd94806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:19.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd94806c8f0 0x7fd94806edb0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd958034000 tx=0x7fd95802e000 comp rx=0 tx=0).stop 2024-04-24T08:19:19.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 -- 172.21.15.69:0/2376111823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd968160510 msgr2=0x7fd968140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:19.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd968160510 0x7fd968140c40 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fd96000ca80 tx=0x7fd96000cf40 comp rx=0 tx=0).stop 2024-04-24T08:19:19.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 -- 172.21.15.69:0/2376111823 shutdown_connections 2024-04-24T08:19:19.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd94806c8f0 0x7fd94806edb0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd968160510 0x7fd968140c40 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd968142480 0x7fd9681f3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 --2- 172.21.15.69:0/2376111823 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9681420a0 0x7fd968154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:19.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 -- 172.21.15.69:0/2376111823 >> 172.21.15.69:0/2376111823 conn(0x7fd968067b80 msgr2=0x7fd9680b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:19.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 -- 172.21.15.69:0/2376111823 shutdown_connections 2024-04-24T08:19:19.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.104+0000 7fd96d800640 1 -- 172.21.15.69:0/2376111823 wait complete. 2024-04-24T08:19:19.122 INFO:tasks.cephfs.filesystem:scrub status for tag:76c45099-d189-4cc9-8efe-7618923fdc69 - {'path': '/', 'tag': '76c45099-d189-4cc9-8efe-7618923fdc69', 'options': 'recursive,repair'} 2024-04-24T08:19:19.694 INFO:tasks.ceph.mds.a.smithi069.stderr:2024-04-24T08:19:19.693+0000 7fe15e200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:19:19.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:19.734+0000 7f461ea00640 1 -- 172.21.15.69:0/3420949237 <== mon.0 v2:172.21.15.69:3300/0 7 ==== log(8 entries from seq 1 at 2024-04-24T08:19:18.861850+0000) ==== 1407+0+0 (secure 0 0 0) 0x7f46200577c0 con 0x7f46280b6470 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: cluster: 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: id: 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: health: HEALTH_OK 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: services: 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: mon: 3 daemons, quorum a,b,c (age 11m) 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: mgr: x(active, since 11m), standbys: y 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: mds: 1/1 daemons up, 3 standby 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: osd: 8 osds: 8 up (since 11m), 8 in (since 11m) 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: data: 2024-04-24T08:19:19.737 INFO:teuthology.orchestra.run.smithi069.stdout: volumes: 1/1 healthy 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout: pools: 3 pools, 97 pgs 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout: objects: 30 objects, 19 MiB 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout: usage: 290 MiB used, 720 GiB / 720 GiB avail 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout: pgs: 97 active+clean 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout: io: 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout: client: 1.8 KiB/s rd, 102 B/s wr, 1 op/s rd, 0 op/s wr 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:19:18.861850+0000 mds.a [INF] scrub queued for path: / 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:19:18.861853+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:19:18.861857+0000 mds.a [INF] scrub summary: active paths [/] 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:19:18.862647+0000 mds.a [ERR] scrub: inode wrongly marked free: 0x10000000001 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:19:18.862654+0000 mds.a [ERR] inode table repaired for inode: 0x10000000001 2024-04-24T08:19:19.738 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:19:18.862770+0000 mds.a [ERR] scrub: inode wrongly marked free: 0x10000000002 2024-04-24T08:19:19.739 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:19:18.862776+0000 mds.a [ERR] inode table repaired for inode: 0x10000000002 2024-04-24T08:19:19.739 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:19:18.863709+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-04-24T08:19:23.755 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:19:22.691716+0000 mds.a [INF] scrub summary: idle 2024-04-24T08:19:23.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:23.753+0000 7f461ea00640 1 -- 172.21.15.69:0/3420949237 <== mon.0 v2:172.21.15.69:3300/0 8 ==== log(1 entries from seq 9 at 2024-04-24T08:19:22.691716+0000) ==== 189+0+0 (secure 0 0 0) 0x7f462008c000 con 0x7f46280b6470 2024-04-24T08:19:46.404 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:19:46.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:19:49.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:19:49.124 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-24T08:19:49.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.336+0000 7fa5c2c00640 1 Processor -- start 2024-04-24T08:19:49.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.336+0000 7fa5c2c00640 1 -- start start 2024-04-24T08:19:49.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.337+0000 7fa5c2c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc15e050 0x7fa5bc06dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.337+0000 7fa5c2c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa5bc15f120 0x7fa5bc06f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.338+0000 7fa5c2c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5bc15e430 con 0x7fa5bc071d70 2024-04-24T08:19:49.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.338+0000 7fa5c2c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa5bc06e490 con 0x7fa5bc15f120 2024-04-24T08:19:49.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.338+0000 7fa5c2c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa5bc06e770 con 0x7fa5bc15e050 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.338+0000 7fa5c2200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa5bc071d70 0x7fa5bc0cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33620/0 (socket says 172.21.15.69:33620) 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.338+0000 7fa5c2200640 1 -- 172.21.15.69:0/1207246852 learned_addr learned my addr 172.21.15.69:0/1207246852 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.338+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1207246852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc15e050 0x7fa5bc06dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.338+0000 7fa5c0e00640 1 --2- 172.21.15.69:0/1207246852 >> v2:172.21.15.121:3300/0 conn(0x7fa5bc15f120 0x7fa5bc06f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 384100355 0 0) 0x7fa5bc15e430 con 0x7fa5bc071d70 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5a4003520 con 0x7fa5bc071d70 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 922921259 0 0) 0x7fa5a4003520 con 0x7fa5bc071d70 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc15e050 msgr2=0x7fa5bc06dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 --2- 172.21.15.69:0/1207246852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc15e050 0x7fa5bc06dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 >> v2:172.21.15.121:3300/0 conn(0x7fa5bc15f120 msgr2=0x7fa5bc06f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 --2- 172.21.15.69:0/1207246852 >> v2:172.21.15.121:3300/0 conn(0x7fa5bc15f120 0x7fa5bc06f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5bc06ea50 con 0x7fa5bc071d70 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fa5b40019f0 con 0x7fa5bc071d70 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 >> v1:172.21.15.69:6789/0 conn(0x7fa5bc071d70 legacy=0x7fa5bc0cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 --2- 172.21.15.69:0/1207246852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 0x7fa5a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 --2- 172.21.15.69:0/1207246852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc15e050 0x7fa5a4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 --2- 172.21.15.69:0/1207246852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5a4004d40 0x7fa5a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc06e490 con 0x7fa5a40039b0 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc06e770 con 0x7fa5a4004d40 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.339+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa5a400b9c0 con 0x7fa5bc15e050 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.340+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1207246852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 0x7fa5a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.340+0000 7fa5c1800640 1 -- 172.21.15.69:0/1207246852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc15e050 msgr2=0x7fa5a4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.340+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1207246852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc15e050 0x7fa5a4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.340+0000 7fa5c1800640 1 -- 172.21.15.69:0/1207246852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5a4004d40 msgr2=0x7fa5a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.340+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1207246852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5a4004d40 0x7fa5a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.340+0000 7fa5c1800640 1 -- 172.21.15.69:0/1207246852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a4003520 con 0x7fa5a40039b0 2024-04-24T08:19:49.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.341+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1207246852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 0x7fa5a4003dd0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000a7f0 tx=0x7fa5a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55a738a05451de5a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:49.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.341+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b003a070 con 0x7fa5a40039b0 2024-04-24T08:19:49.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.341+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b002d840 con 0x7fa5a40039b0 2024-04-24T08:19:49.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.341+0000 7fa5bbe00640 1 -- 172.21.15.69:0/1207246852 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000b040 con 0x7fa5a40039b0 2024-04-24T08:19:49.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.342+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1207246852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 msgr2=0x7fa5a4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.342+0000 7fa5c2c00640 1 --2- 172.21.15.69:0/1207246852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 0x7fa5a4003dd0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000a7f0 tx=0x7fa5a4005830 comp rx=0 tx=0).stop 2024-04-24T08:19:49.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.342+0000 7fa5c2200640 1 -- 172.21.15.69:0/1207246852 reap_dead start 2024-04-24T08:19:49.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.342+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1207246852 shutdown_connections 2024-04-24T08:19:49.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.342+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1207246852 >> 172.21.15.69:0/1207246852 conn(0x7fa5bc067b80 msgr2=0x7fa5bc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:49.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.342+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1207246852 shutdown_connections 2024-04-24T08:19:49.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.342+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1207246852 wait complete. 2024-04-24T08:19:49.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.344+0000 7fa5c2c00640 1 Processor -- start 2024-04-24T08:19:49.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.344+0000 7fa5c2c00640 1 -- start start 2024-04-24T08:19:49.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c2c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 0x7fa5bc146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c2c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5bc0b3c10 0x7fa5bc0b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c2c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc148010 0x7fa5bc148490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c2c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc15f580 con 0x7fa5a40039b0 2024-04-24T08:19:49.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c2c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc06f010 con 0x7fa5bc0b3c10 2024-04-24T08:19:49.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c2c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa5bc06fa70 con 0x7fa5bc148010 2024-04-24T08:19:49.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 0x7fa5bc146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 0x7fa5bc146bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39922/0 (socket says 172.21.15.69:39922) 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c1800640 1 -- 172.21.15.69:0/1464177294 learned_addr learned my addr 172.21.15.69:0/1464177294 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c1800640 1 -- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc148010 msgr2=0x7fa5bc148490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc148010 0x7fa5bc148490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c0e00640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5bc0b3c10 0x7fa5bc0b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c1800640 1 -- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5bc0b3c10 msgr2=0x7fa5bc0b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5bc0b3c10 0x7fa5bc0b4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.345+0000 7fa5c1800640 1 -- 172.21.15.69:0/1464177294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5bc148d70 con 0x7fa5a40039b0 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.346+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 0x7fa5bc146bb0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000abb0 tx=0x7fa5b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.346+0000 7fa5b9600640 1 -- 172.21.15.69:0/1464177294 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b003a050 con 0x7fa5a40039b0 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.346+0000 7fa5b9600640 1 -- 172.21.15.69:0/1464177294 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b0046070 con 0x7fa5a40039b0 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.346+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1464177294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5bc149050 con 0x7fa5a40039b0 2024-04-24T08:19:49.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.346+0000 7fa5b9600640 1 -- 172.21.15.69:0/1464177294 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000b040 con 0x7fa5a40039b0 2024-04-24T08:19:49.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.346+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1464177294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5bc1470f0 con 0x7fa5a40039b0 2024-04-24T08:19:49.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.348+0000 7fa5b9600640 1 -- 172.21.15.69:0/1464177294 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa5b0003830 con 0x7fa5a40039b0 2024-04-24T08:19:49.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.349+0000 7fa5b9600640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa59c06c8f0 0x7fa59c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.349+0000 7fa5b9600640 1 -- 172.21.15.69:0/1464177294 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fa5b00ae750 con 0x7fa5a40039b0 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.350+0000 7fa5c700b740 1 Processor -- start 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.350+0000 7fa5c0e00640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa59c06c8f0 0x7fa59c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.350+0000 7fa5c700b740 1 -- start start 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f2e00 0x55ac6a4a5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4d5f20 0x55ac6a4d52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d57e0 0x55ac6a4aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c700b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ac6a40f1f0 con 0x55ac6a4d57e0 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c700b740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ac6a3b5350 con 0x55ac6a4f2e00 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c700b740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ac6a40b1b0 con 0x55ac6a4d5f20 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c2200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d57e0 0x55ac6a4aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c2200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d57e0 0x55ac6a4aa5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39934/0 (socket says 172.21.15.69:39934) 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c2200640 1 -- 172.21.15.69:0/2492007425 learned_addr learned my addr 172.21.15.69:0/2492007425 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:49.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c0e00640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa59c06c8f0 0x7fa59c06edb0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc14db80 tx=0x7fa5a8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c0e00640 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4d5f20 0x55ac6a4d52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c1800640 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f2e00 0x55ac6a4a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c2200640 1 -- 172.21.15.69:0/2492007425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4d5f20 msgr2=0x55ac6a4d52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c2200640 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4d5f20 0x55ac6a4d52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c2200640 1 -- 172.21.15.69:0/2492007425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f2e00 msgr2=0x55ac6a4a5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.351+0000 7fa5c2200640 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f2e00 0x55ac6a4a5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.352+0000 7fa5c2200640 1 -- 172.21.15.69:0/2492007425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac6a409f50 con 0x55ac6a4d57e0 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.352+0000 7fa5c0e00640 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4d5f20 0x55ac6a4d52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.352+0000 7fa5c1800640 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f2e00 0x55ac6a4a5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.352+0000 7fa5c2200640 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d57e0 0x55ac6a4aa5f0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc06c830 tx=0x7fa5b400c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b47ed12f124356 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.352+0000 7fa5ae000640 1 -- 172.21.15.69:0/2492007425 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4004d60 con 0x55ac6a4d57e0 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.352+0000 7fa5ae000640 1 -- 172.21.15.69:0/2492007425 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b400cb90 con 0x55ac6a4d57e0 2024-04-24T08:19:49.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.352+0000 7fa5ae000640 1 -- 172.21.15.69:0/2492007425 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4007030 con 0x55ac6a4d57e0 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.353+0000 7fa5c700b740 1 -- 172.21.15.69:0/2492007425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d57e0 msgr2=0x55ac6a4aa5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.353+0000 7fa5c700b740 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d57e0 0x55ac6a4aa5f0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc06c830 tx=0x7fa5b400c4b0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.353+0000 7fa5c700b740 1 -- 172.21.15.69:0/2492007425 shutdown_connections 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.353+0000 7fa5c700b740 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d57e0 0x55ac6a4aa5f0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.353+0000 7fa5c700b740 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4d5f20 0x55ac6a4d52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.353+0000 7fa5c700b740 1 --2- 172.21.15.69:0/2492007425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f2e00 0x55ac6a4a5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.353+0000 7fa5c700b740 1 -- 172.21.15.69:0/2492007425 >> 172.21.15.69:0/2492007425 conn(0x55ac6a4a7a00 msgr2=0x55ac6a4f02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.353+0000 7fa5c700b740 1 -- 172.21.15.69:0/2492007425 shutdown_connections 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.353+0000 7fa5c700b740 1 -- 172.21.15.69:0/2492007425 wait complete. 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.354+0000 7fa5c700b740 1 -- start start 2024-04-24T08:19:49.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.354+0000 7fa5c700b740 1 Processor -- start 2024-04-24T08:19:49.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4c2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f2e00 0x55ac6a4d7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f0250 0x55ac6a4f06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c700b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ac6a4a9bb0 con 0x55ac6a4d5f20 2024-04-24T08:19:49.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c700b740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ac6a3c8690 con 0x55ac6a4f0250 2024-04-24T08:19:49.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c700b740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ac6a4a8fa0 con 0x55ac6a4f2e00 2024-04-24T08:19:49.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c700b740 10 client.0 _ll_register_callbacks cb 0x55ac6a404650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4c2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4c2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39942/0 (socket says 172.21.15.69:39942) 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c1800640 1 -- 172.21.15.69:0/1505526892 learned_addr learned my addr 172.21.15.69:0/1505526892 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c1800640 1 -- 172.21.15.69:0/1505526892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f2e00 msgr2=0x55ac6a4d7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f2e00 0x55ac6a4d7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c1800640 1 -- 172.21.15.69:0/1505526892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f0250 msgr2=0x55ac6a4f06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f0250 0x55ac6a4f06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.355+0000 7fa5c1800640 1 -- 172.21.15.69:0/1505526892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac6a3c7920 con 0x55ac6a4d5f20 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5c1800640 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4c2cb0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000ab80 tx=0x7fa5b007cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca3a8a7c69f8740 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5a2a00640 1 -- 172.21.15.69:0/1505526892 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0094020 con 0x55ac6a4d5f20 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5a2a00640 1 -- 172.21.15.69:0/1505526892 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fa5b00a7c80 con 0x55ac6a4d5f20 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5c700b740 1 -- 172.21.15.69:0/1505526892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5b400cb90 con 0x55ac6a4d5f20 2024-04-24T08:19:49.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5a2a00640 1 -- 172.21.15.69:0/1505526892 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fa5b0084ff0 con 0x55ac6a4d5f20 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5c700b740 10 client.9093 fetch_fsmap learned FSMap version 89 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5c700b740 1 -- 172.21.15.69:0/1505526892 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x55ac6a1eb7d0 con 0x55ac6a4d5f20 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5a2a00640 1 -- 172.21.15.69:0/1505526892 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b400cb90 con 0x55ac6a4d5f20 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5a2a00640 1 -- 172.21.15.69:0/1505526892 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7fa5b0084b60 con 0x55ac6a4d5f20 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.356+0000 7fa5c700b740 10 client.9093 fetch_fsmap finished waiting for FSMap version 89 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.357+0000 7fa5c700b740 10 client.9093 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.357+0000 7fa5c700b740 20 client.9093 populate_metadata read hostname 'smithi069' 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.357+0000 7fa5c700b740 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55ac6a4ee320 0x55ac6a4ee7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.357+0000 7fa5c700b740 4 client.9093 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.357+0000 7fa5c700b740 1 -- 172.21.15.69:0/1505526892 --> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac6a41cc10 con 0x55ac6a4ee320 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.357+0000 7fa5c0e00640 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55ac6a4ee320 0x55ac6a4ee7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.357+0000 7fa5a2a00640 10 client.9093 ms_handle_connect on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:49.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.357+0000 7fa5c0e00640 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55ac6a4ee320 0x55ac6a4ee7d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24a4eca77cc6c7ce server_cookie=e0b1893acc8e5306 in_seq=0 out_seq=0 2024-04-24T08:19:49.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.358+0000 7fa5a2a00640 1 -- 172.21.15.69:0/1505526892 <== mds.0 v2:172.21.15.69:6833/3642138906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ac6a41cc10 con 0x55ac6a4ee320 2024-04-24T08:19:49.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.358+0000 7fa5a2a00640 10 client.9093 handle_command_reply: tid=0 2024-04-24T08:19:49.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.358+0000 7fa5c700b740 1 client.9093 shutdown 2024-04-24T08:19:49.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.358+0000 7fa5c700b740 1 -- 172.21.15.69:0/1505526892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 msgr2=0x55ac6a4c2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.358+0000 7fa5c700b740 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4c2cb0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000ab80 tx=0x7fa5b007cc50 comp rx=0 tx=0).stop 2024-04-24T08:19:49.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.358+0000 7fa5c700b740 1 -- 172.21.15.69:0/1505526892 shutdown_connections 2024-04-24T08:19:49.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.358+0000 7fa5c700b740 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55ac6a4ee320 0x55ac6a4ee7d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.358+0000 7fa5c700b740 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f0250 0x55ac6a4f06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.359+0000 7fa5a2a00640 0 client.9093 ms_handle_reset on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:49.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.359+0000 7fa5c700b740 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f2e00 0x55ac6a4d7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.359+0000 7fa5c700b740 1 --2- 172.21.15.69:0/1505526892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4c2cb0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.359+0000 7fa5c700b740 1 -- 172.21.15.69:0/1505526892 >> 172.21.15.69:0/1505526892 conn(0x55ac6a4d57e0 msgr2=0x55ac6a4d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:49.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.359+0000 7fa5c700b740 1 -- 172.21.15.69:0/1505526892 shutdown_connections 2024-04-24T08:19:49.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.359+0000 7fa5c700b740 1 -- 172.21.15.69:0/1505526892 wait complete. 2024-04-24T08:19:49.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.359+0000 7fa5c700b740 20 client.9093 trim_cache size 0 max 16384 2024-04-24T08:19:49.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.378+0000 7fa5c700b740 1 Processor -- start 2024-04-24T08:19:49.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.378+0000 7fa5c700b740 1 -- start start 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a50a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f0250 0x55ac6a4c2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f2e00 0x55ac6a4ecfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c700b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc1fd0f0 con 0x55ac6a4d5f20 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c700b740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc1fc130 con 0x55ac6a4f0250 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c700b740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa5bc14c470 con 0x55ac6a4f2e00 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a50a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a50a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39946/0 (socket says 172.21.15.69:39946) 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c1800640 1 -- 172.21.15.69:0/2225074436 learned_addr learned my addr 172.21.15.69:0/2225074436 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c1800640 1 -- 172.21.15.69:0/2225074436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f2e00 msgr2=0x55ac6a4ecfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c1800640 1 --2- 172.21.15.69:0/2225074436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f2e00 0x55ac6a4ecfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c1800640 1 -- 172.21.15.69:0/2225074436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f0250 msgr2=0x55ac6a4c2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c1800640 1 --2- 172.21.15.69:0/2225074436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f0250 0x55ac6a4c2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.379+0000 7fa5c1800640 1 -- 172.21.15.69:0/2225074436 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac6a1213f0 con 0x55ac6a4d5f20 2024-04-24T08:19:49.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.380+0000 7fa5c1800640 1 --2- 172.21.15.69:0/2225074436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a50a350 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00adfd0 tx=0x7fa5b00ae000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad42559f072966f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:49.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.380+0000 7fa5ad600640 1 -- 172.21.15.69:0/2225074436 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0094040 con 0x55ac6a4d5f20 2024-04-24T08:19:49.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.380+0000 7fa5ad600640 1 -- 172.21.15.69:0/2225074436 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b003b040 con 0x55ac6a4d5f20 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.380+0000 7fa5ad600640 1 -- 172.21.15.69:0/2225074436 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b007b860 con 0x55ac6a4d5f20 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.380+0000 7fa5c700b740 1 -- 172.21.15.69:0/2225074436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 msgr2=0x55ac6a50a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.380+0000 7fa5c700b740 1 --2- 172.21.15.69:0/2225074436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a50a350 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00adfd0 tx=0x7fa5b00ae000 comp rx=0 tx=0).stop 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.381+0000 7fa5c700b740 1 -- 172.21.15.69:0/2225074436 shutdown_connections 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.381+0000 7fa5c700b740 1 --2- 172.21.15.69:0/2225074436 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f2e00 0x55ac6a4ecfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.381+0000 7fa5c700b740 1 --2- 172.21.15.69:0/2225074436 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f0250 0x55ac6a4c2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.381+0000 7fa5c700b740 1 --2- 172.21.15.69:0/2225074436 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a50a350 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.381+0000 7fa5c700b740 1 -- 172.21.15.69:0/2225074436 >> 172.21.15.69:0/2225074436 conn(0x55ac6a4d57e0 msgr2=0x55ac6a4a0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.381+0000 7fa5c700b740 1 -- 172.21.15.69:0/2225074436 shutdown_connections 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.381+0000 7fa5c700b740 1 -- 172.21.15.69:0/2225074436 wait complete. 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.381+0000 7fa5c700b740 1 -- start start 2024-04-24T08:19:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.382+0000 7fa5c700b740 1 Processor -- start 2024-04-24T08:19:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.382+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4ff3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.382+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f0250 0x55ac6a4c1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c700b740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f2e00 0x55ac6a4aa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c700b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ac6a3b4790 con 0x55ac6a4d5f20 2024-04-24T08:19:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c700b740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa5bc1fbb10 con 0x55ac6a4f2e00 2024-04-24T08:19:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c700b740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ac6a4d64b0 con 0x55ac6a4f0250 2024-04-24T08:19:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c700b740 10 client.0 _ll_register_callbacks cb 0x7fa5b0004580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4ff3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4ff3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39960/0 (socket says 172.21.15.69:39960) 2024-04-24T08:19:49.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c1800640 1 -- 172.21.15.69:0/512445552 learned_addr learned my addr 172.21.15.69:0/512445552 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c1800640 1 -- 172.21.15.69:0/512445552 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f0250 msgr2=0x55ac6a4c1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c1800640 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f0250 0x55ac6a4c1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c1800640 1 -- 172.21.15.69:0/512445552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f2e00 msgr2=0x55ac6a4aa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c1800640 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f2e00 0x55ac6a4aa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c1800640 1 -- 172.21.15.69:0/512445552 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac6a0e10f0 con 0x55ac6a4d5f20 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5c1800640 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4ff3b0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00a8040 tx=0x7fa5b00acae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2acb856086e1db server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.383+0000 7fa5a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5a2a00640 1 -- 172.21.15.69:0/512445552 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b00a0060 con 0x55ac6a4d5f20 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5a2a00640 1 -- 172.21.15.69:0/512445552 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7fa5b007ba60 con 0x55ac6a4d5f20 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5c700b740 1 -- 172.21.15.69:0/512445552 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5b003b040 con 0x55ac6a4d5f20 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5a2a00640 1 -- 172.21.15.69:0/512445552 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fa5b0094040 con 0x55ac6a4d5f20 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5c700b740 10 client.9099 fetch_fsmap learned FSMap version 89 2024-04-24T08:19:49.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5c700b740 1 -- 172.21.15.69:0/512445552 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=89,monmap=2+}) -- 0x55ac6a3d4810 con 0x55ac6a4d5f20 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5a2a00640 1 -- 172.21.15.69:0/512445552 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b003b040 con 0x55ac6a4d5f20 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5a2a00640 1 -- 172.21.15.69:0/512445552 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 89) ==== 3494+0+0 (secure 0 0 0) 0x7fa5b00a6020 con 0x55ac6a4d5f20 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5c700b740 10 client.9099 fetch_fsmap finished waiting for FSMap version 89 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5c700b740 10 client.9099 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5c700b740 20 client.9099 populate_metadata read hostname 'smithi069' 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5c700b740 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55ac6a4aaa20 0x55ac6a4e5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5c700b740 4 client.9099 mds_command: new command op to 8955 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5c700b740 1 -- 172.21.15.69:0/512445552 --> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55ac6a416a80 con 0x55ac6a4aaa20 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.384+0000 7fa5c0e00640 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55ac6a4aaa20 0x55ac6a4e5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5c0e00640 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55ac6a4aaa20 0x55ac6a4e5c30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62b2fa731de6ebba server_cookie=453beb6caccbe07a in_seq=0 out_seq=0 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5a2a00640 10 client.9099 ms_handle_connect on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5a2a00640 1 -- 172.21.15.69:0/512445552 <== mds.0 v2:172.21.15.69:6833/3642138906 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55ac6a416a80 con 0x55ac6a4aaa20 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5a2a00640 10 client.9099 handle_command_reply: tid=0 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5c700b740 1 client.9099 shutdown 2024-04-24T08:19:49.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5c700b740 1 -- 172.21.15.69:0/512445552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 msgr2=0x55ac6a4ff3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5c700b740 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4ff3b0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fa5b00a8040 tx=0x7fa5b00acae0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5c700b740 1 -- 172.21.15.69:0/512445552 shutdown_connections 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5c700b740 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.69:6833/3642138906,v1:172.21.15.69:6835/3642138906] conn(0x55ac6a4aaa20 0x55ac6a4e5c30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5c700b740 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ac6a4f2e00 0x55ac6a4aa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5a2a00640 0 client.9099 ms_handle_reset on v2:172.21.15.69:6833/3642138906 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.385+0000 7fa5c700b740 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ac6a4f0250 0x55ac6a4c1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c700b740 1 --2- 172.21.15.69:0/512445552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ac6a4d5f20 0x55ac6a4ff3b0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c700b740 1 -- 172.21.15.69:0/512445552 >> 172.21.15.69:0/512445552 conn(0x55ac6a4d57e0 msgr2=0x55ac6a4ecfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c700b740 1 -- 172.21.15.69:0/512445552 shutdown_connections 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c700b740 1 -- 172.21.15.69:0/512445552 wait complete. 2024-04-24T08:19:49.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c700b740 20 client.9099 trim_cache size 0 max 16384 2024-04-24T08:19:49.393 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:19:49.393 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-04-24T08:19:49.393 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-04-24T08:19:49.393 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa59c06c8f0 msgr2=0x7fa59c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c2c00640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa59c06c8f0 0x7fa59c06edb0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc14db80 tx=0x7fa5a8008040 comp rx=0 tx=0).stop 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1464177294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 msgr2=0x7fa5bc146bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c2c00640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 0x7fa5bc146bb0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000abb0 tx=0x7fa5b0004730 comp rx=0 tx=0).stop 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1464177294 shutdown_connections 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c2c00640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa59c06c8f0 0x7fa59c06edb0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c2c00640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5bc148010 0x7fa5bc148490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.386+0000 7fa5c2c00640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5bc0b3c10 0x7fa5bc0b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.387+0000 7fa5c2c00640 1 --2- 172.21.15.69:0/1464177294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa5a40039b0 0x7fa5bc146bb0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.387+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1464177294 >> 172.21.15.69:0/1464177294 conn(0x7fa5bc067b80 msgr2=0x7fa5bc0d0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.387+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1464177294 shutdown_connections 2024-04-24T08:19:49.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.387+0000 7fa5c2c00640 1 -- 172.21.15.69:0/1464177294 wait complete. 2024-04-24T08:19:49.405 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:19:49.506 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:19:49.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.709+0000 7f1a77e00640 1 Processor -- start 2024-04-24T08:19:49.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.710+0000 7f1a77e00640 1 -- start start 2024-04-24T08:19:49.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.711+0000 7f1a77e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1a7015e890 0x7f1a700d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.711+0000 7f1a77e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a7015ec70 0x7f1a700d16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.711+0000 7f1a77e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a700d26c0 con 0x7f1a700cfec0 2024-04-24T08:19:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.711+0000 7f1a77e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1a700d32d0 con 0x7f1a7015e890 2024-04-24T08:19:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.711+0000 7f1a77e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1a700d29a0 con 0x7f1a7015ec70 2024-04-24T08:19:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.711+0000 7f1a75400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1a7015e890 0x7f1a700d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.711+0000 7f1a75e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1a700cfec0 0x7f1a70162680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33628/0 (socket says 172.21.15.69:33628) 2024-04-24T08:19:49.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.712+0000 7f1a75e00640 1 -- 172.21.15.69:0/2928270508 learned_addr learned my addr 172.21.15.69:0/2928270508 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.712+0000 7f1a74a00640 1 --2- 172.21.15.69:0/2928270508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a7015ec70 0x7f1a700d16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.712+0000 7f1a75400640 1 -- 172.21.15.69:0/2928270508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a7015ec70 msgr2=0x7f1a700d16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.712+0000 7f1a75400640 1 --2- 172.21.15.69:0/2928270508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a7015ec70 0x7f1a700d16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.712+0000 7f1a75400640 1 -- 172.21.15.69:0/2928270508 >> v1:172.21.15.69:6789/0 conn(0x7f1a700cfec0 legacy=0x7f1a70162680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.712+0000 7f1a75400640 1 -- 172.21.15.69:0/2928270508 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a700d2c80 con 0x7f1a7015e890 2024-04-24T08:19:49.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.712+0000 7f1a6fe00640 1 -- 172.21.15.69:0/2928270508 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3647857712 0 0) 0x7f1a700d26c0 con 0x7f1a700cfec0 2024-04-24T08:19:49.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.713+0000 7f1a75400640 1 --2- 172.21.15.69:0/2928270508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 0x7f1a700d11b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f1a640097a0 tx=0x7f1a6402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cd450523a9d52f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:49.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a6fe00640 1 -- 172.21.15.69:0/2928270508 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64037070 con 0x7f1a7015e890 2024-04-24T08:19:49.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a6fe00640 1 -- 172.21.15.69:0/2928270508 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a64032440 con 0x7f1a7015e890 2024-04-24T08:19:49.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a6fe00640 1 -- 172.21.15.69:0/2928270508 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64032810 con 0x7f1a7015e890 2024-04-24T08:19:49.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a77e00640 1 -- 172.21.15.69:0/2928270508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 msgr2=0x7f1a700d11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a77e00640 1 --2- 172.21.15.69:0/2928270508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 0x7f1a700d11b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f1a640097a0 tx=0x7f1a6402a210 comp rx=0 tx=0).stop 2024-04-24T08:19:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a77e00640 1 -- 172.21.15.69:0/2928270508 shutdown_connections 2024-04-24T08:19:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a77e00640 1 --2- 172.21.15.69:0/2928270508 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a7015ec70 0x7f1a700d16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a77e00640 1 --2- 172.21.15.69:0/2928270508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 0x7f1a700d11b0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a77e00640 1 -- 172.21.15.69:0/2928270508 >> 172.21.15.69:0/2928270508 conn(0x7f1a7008bf60 msgr2=0x7f1a700d47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a77e00640 1 -- 172.21.15.69:0/2928270508 shutdown_connections 2024-04-24T08:19:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.714+0000 7f1a77e00640 1 -- 172.21.15.69:0/2928270508 wait complete. 2024-04-24T08:19:49.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.716+0000 7f1a77e00640 1 Processor -- start 2024-04-24T08:19:49.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.716+0000 7f1a77e00640 1 -- start start 2024-04-24T08:19:49.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.717+0000 7f1a77e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a700cfec0 0x7f1a700c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.717+0000 7f1a77e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 0x7f1a70151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.717+0000 7f1a77e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a7015ec70 0x7f1a70151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.717+0000 7f1a77e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1a700d3760 con 0x7f1a7015e890 2024-04-24T08:19:49.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.717+0000 7f1a77e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1a700b4b70 con 0x7f1a7015ec70 2024-04-24T08:19:49.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.717+0000 7f1a77e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1a700d2370 con 0x7f1a700cfec0 2024-04-24T08:19:49.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.717+0000 7f1a74a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 0x7f1a70151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.717+0000 7f1a74a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 0x7f1a70151800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39982/0 (socket says 172.21.15.69:39982) 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.717+0000 7f1a74a00640 1 -- 172.21.15.69:0/2042645236 learned_addr learned my addr 172.21.15.69:0/2042645236 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.718+0000 7f1a75400640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a700cfec0 0x7f1a700c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.718+0000 7f1a75e00640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a7015ec70 0x7f1a70151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.718+0000 7f1a74a00640 1 -- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a700cfec0 msgr2=0x7f1a700c89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.718+0000 7f1a74a00640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a700cfec0 0x7f1a700c89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.718+0000 7f1a74a00640 1 -- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a7015ec70 msgr2=0x7f1a70151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.718+0000 7f1a74a00640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a7015ec70 0x7f1a70151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.718+0000 7f1a74a00640 1 -- 172.21.15.69:0/2042645236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a700b9310 con 0x7f1a7015e890 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.718+0000 7f1a74a00640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 0x7f1a70151800 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00c7f0 tx=0x7f1a5c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.718+0000 7f1a6d600640 1 -- 172.21.15.69:0/2042645236 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c015070 con 0x7f1a7015e890 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.719+0000 7f1a6d600640 1 -- 172.21.15.69:0/2042645236 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5c004220 con 0x7f1a7015e890 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.719+0000 7f1a6d600640 1 -- 172.21.15.69:0/2042645236 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c007620 con 0x7f1a7015e890 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.719+0000 7f1a77e00640 1 -- 172.21.15.69:0/2042645236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a700b95f0 con 0x7f1a7015e890 2024-04-24T08:19:49.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.719+0000 7f1a77e00640 1 -- 172.21.15.69:0/2042645236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a700b9bb0 con 0x7f1a7015e890 2024-04-24T08:19:49.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.720+0000 7f1a77e00640 1 -- 172.21.15.69:0/2042645236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a70068a80 con 0x7f1a7015e890 2024-04-24T08:19:49.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.721+0000 7f1a6d600640 1 -- 172.21.15.69:0/2042645236 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a5c0079d0 con 0x7f1a7015e890 2024-04-24T08:19:49.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.722+0000 7f1a6d600640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a5006c8f0 0x7f1a5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:49.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.722+0000 7f1a6d600640 1 -- 172.21.15.69:0/2042645236 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 5795+0+0 (secure 0 0 0) 0x7f1a5c08bf70 con 0x7f1a7015e890 2024-04-24T08:19:49.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.722+0000 7f1a75400640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a5006c8f0 0x7f1a5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:49.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.724+0000 7f1a75400640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a5006c8f0 0x7f1a5006edb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1a64034000 tx=0x7f1a6402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:49.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.726+0000 7f1a6d600640 1 -- 172.21.15.69:0/2042645236 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a5c05a2a0 con 0x7f1a7015e890 2024-04-24T08:19:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.972+0000 7f1a77e00640 1 -- 172.21.15.69:0/2042645236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f1a701617a0 con 0x7f1a7015e890 2024-04-24T08:19:49.975 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:19:49.973+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:19:49.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.977+0000 7f1a6d600640 1 -- 172.21.15.69:0/2042645236 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v90) ==== 161+0+0 (secure 0 0 0) 0x7f1a5c059ba0 con 0x7f1a7015e890 2024-04-24T08:19:49.979 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:19:49.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 -- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a5006c8f0 msgr2=0x7f1a5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a5006c8f0 0x7f1a5006edb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f1a64034000 tx=0x7f1a6402e000 comp rx=0 tx=0).stop 2024-04-24T08:19:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 -- 172.21.15.69:0/2042645236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 msgr2=0x7f1a70151800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 0x7f1a70151800 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00c7f0 tx=0x7f1a5c00ccb0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 -- 172.21.15.69:0/2042645236 shutdown_connections 2024-04-24T08:19:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a5006c8f0 0x7f1a5006edb0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7015e890 0x7f1a70151800 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a7015ec70 0x7f1a70151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 --2- 172.21.15.69:0/2042645236 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a700cfec0 0x7f1a700c89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 -- 172.21.15.69:0/2042645236 >> 172.21.15.69:0/2042645236 conn(0x7f1a7008bf60 msgr2=0x7f1a70161d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 -- 172.21.15.69:0/2042645236 shutdown_connections 2024-04-24T08:19:49.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:49.981+0000 7f1a77e00640 1 -- 172.21.15.69:0/2042645236 wait complete. 2024-04-24T08:19:49.989 INFO:tasks.ceph.mds.a.smithi069.stderr: -1149> 2024-04-24T08:19:18.861+0000 7fe159200640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-04-24T08:19:49.989 INFO:tasks.ceph.mds.a.smithi069.stderr: -1146> 2024-04-24T08:19:18.861+0000 7fe159200640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-04-24T08:19:49.989 INFO:tasks.ceph.mds.a.smithi069.stderr: -1129> 2024-04-24T08:19:18.861+0000 7fe159200640 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-04-24T08:19:49.989 INFO:tasks.ceph.mds.a.smithi069.stderr: -1126> 2024-04-24T08:19:18.861+0000 7fe159200640 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-04-24T08:19:49.989 INFO:tasks.ceph.mds.a.smithi069.stderr: -981> 2024-04-24T08:19:19.693+0000 7fe15e200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:19:50.002 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 0 show inode 2024-04-24T08:19:50.037 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:19:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.082+0000 7ff13df20280 1 -- start start 2024-04-24T08:19:50.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.083+0000 7ff13df20280 1 Processor -- start 2024-04-24T08:19:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.083+0000 7ff13df20280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d24824c00 0x557d255fc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.083+0000 7ff13df20280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d24825800 0x557d255fc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.083+0000 7ff13df20280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d24825c00 0x557d255fcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.083+0000 7ff13df20280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557d255bc180 con 0x557d24825c00 2024-04-24T08:19:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13df20280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557d255bc300 con 0x557d24824c00 2024-04-24T08:19:50.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13df20280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557d255bc480 con 0x557d24825800 2024-04-24T08:19:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13c000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d24825c00 0x557d255fcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13c000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d24825c00 0x557d255fcb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39998/0 (socket says 172.21.15.69:39998) 2024-04-24T08:19:50.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13c000640 1 -- 172.21.15.69:0/450845296 learned_addr learned my addr 172.21.15.69:0/450845296 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:50.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13c000640 1 -- 172.21.15.69:0/450845296 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d24825800 msgr2=0x557d255fc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13c000640 1 --2- 172.21.15.69:0/450845296 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d24825800 0x557d255fc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13b600640 1 --2- 172.21.15.69:0/450845296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d24824c00 0x557d255fc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13c000640 1 -- 172.21.15.69:0/450845296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d24824c00 msgr2=0x557d255fc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13c000640 1 --2- 172.21.15.69:0/450845296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d24824c00 0x557d255fc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.084+0000 7ff13c000640 1 -- 172.21.15.69:0/450845296 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d24858780 con 0x557d24825c00 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.085+0000 7ff13c000640 1 --2- 172.21.15.69:0/450845296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d24825c00 0x557d255fcb00 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x557d256510e0 tx=0x557d255be8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e438c64906b568 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.085+0000 7ff138400640 1 -- 172.21.15.69:0/450845296 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d2484a9c0 con 0x557d24825c00 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.085+0000 7ff138400640 1 -- 172.21.15.69:0/450845296 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x557d24850fc0 con 0x557d24825c00 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.085+0000 7ff13df20280 1 -- 172.21.15.69:0/450845296 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d248594a0 con 0x557d24825c00 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.085+0000 7ff13df20280 1 -- 172.21.15.69:0/450845296 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x557d24858000 con 0x557d24825c00 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.085+0000 7ff13df20280 4 waiting for MDS map... 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.085+0000 7ff138400640 1 -- 172.21.15.69:0/450845296 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d2484a680 con 0x557d24825c00 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.085+0000 7ff138400640 1 -- 172.21.15.69:0/450845296 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x557d256e6200 con 0x557d24825c00 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.085+0000 7ff138400640 1 -- 172.21.15.69:0/450845296 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 90) ==== 2885+0+0 (secure 0 0 0) 0x557d2487f180 con 0x557d24825c00 2024-04-24T08:19:50.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.085+0000 7ff13df20280 4 Got MDS map 90 2024-04-24T08:19:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.086+0000 7ff13df20280 10 main: main 2024-04-24T08:19:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.086+0000 7ff13df20280 4 main: connecting to RADOS... 2024-04-24T08:19:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.086+0000 7ff13df20280 1 Processor -- start 2024-04-24T08:19:50.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.086+0000 7ff13df20280 1 -- start start 2024-04-24T08:19:50.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.087+0000 7ff13df20280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d25744000 0x557d255fd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.087+0000 7ff13df20280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.087+0000 7ff13df20280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d25744800 0x557d255fdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.087+0000 7ff13df20280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557d255bc600 con 0x557d25744400 2024-04-24T08:19:50.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.087+0000 7ff13df20280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557d255bc780 con 0x557d25744800 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.087+0000 7ff13df20280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557d255bc900 con 0x557d25744000 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.087+0000 7ff13b600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.087+0000 7ff13b600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40014/0 (socket says 172.21.15.69:40014) 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.087+0000 7ff13b600640 1 -- 172.21.15.69:0/1787858704 learned_addr learned my addr 172.21.15.69:0/1787858704 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.088+0000 7ff13b600640 1 -- 172.21.15.69:0/1787858704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d25744000 msgr2=0x557d255fd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.088+0000 7ff13b600640 1 --2- 172.21.15.69:0/1787858704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d25744000 0x557d255fd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.088+0000 7ff13b600640 1 -- 172.21.15.69:0/1787858704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d25744800 msgr2=0x557d255fdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.088+0000 7ff13b600640 1 --2- 172.21.15.69:0/1787858704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d25744800 0x557d255fdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.088+0000 7ff13b600640 1 -- 172.21.15.69:0/1787858704 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d248583c0 con 0x557d25744400 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.088+0000 7ff13b600640 1 --2- 172.21.15.69:0/1787858704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x557d256f20f0 tx=0x557d256f4cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21246a1db6d46a52 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.088+0000 7ff135c00640 1 -- 172.21.15.69:0/1787858704 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d2484a9c0 con 0x557d25744400 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.088+0000 7ff135c00640 1 -- 172.21.15.69:0/1787858704 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x557d24850a80 con 0x557d25744400 2024-04-24T08:19:50.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.088+0000 7ff135c00640 1 -- 172.21.15.69:0/1787858704 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d2484a680 con 0x557d25744400 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.089+0000 7ff13df20280 1 -- 172.21.15.69:0/1787858704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 msgr2=0x557d255fd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.089+0000 7ff13df20280 1 --2- 172.21.15.69:0/1787858704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x557d256f20f0 tx=0x557d256f4cc0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.089+0000 7ff13df20280 1 -- 172.21.15.69:0/1787858704 shutdown_connections 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.089+0000 7ff13df20280 1 --2- 172.21.15.69:0/1787858704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.089+0000 7ff13df20280 1 --2- 172.21.15.69:0/1787858704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d25744800 0x557d255fdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.089+0000 7ff13df20280 1 --2- 172.21.15.69:0/1787858704 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d25744000 0x557d255fd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.089+0000 7ff13df20280 1 -- 172.21.15.69:0/1787858704 >> 172.21.15.69:0/1787858704 conn(0x557d25601c00 msgr2=0x557d255ab000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.089+0000 7ff13df20280 1 -- 172.21.15.69:0/1787858704 shutdown_connections 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.089+0000 7ff13df20280 1 -- 172.21.15.69:0/1787858704 wait complete. 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.089+0000 7ff13df20280 1 Processor -- start 2024-04-24T08:19:50.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.090+0000 7ff13df20280 1 -- start start 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.090+0000 7ff13df20280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d25744000 0x557d255fd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.090+0000 7ff13df20280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.090+0000 7ff13df20280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d25744800 0x557d255fdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.090+0000 7ff13df20280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557d255bdc80 con 0x557d25744400 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.090+0000 7ff13df20280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557d255bdb00 con 0x557d25744800 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.090+0000 7ff13df20280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557d255bd980 con 0x557d25744000 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13b600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13b600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40024/0 (socket says 172.21.15.69:40024) 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13b600640 1 -- 172.21.15.69:0/1639509269 learned_addr learned my addr 172.21.15.69:0/1639509269 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13ac00640 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d25744000 0x557d255fd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13b600640 1 -- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d25744000 msgr2=0x557d255fd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13b600640 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d25744000 0x557d255fd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13b600640 1 -- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d25744800 msgr2=0x557d255fdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13b600640 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d25744800 0x557d255fdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13b600640 1 -- 172.21.15.69:0/1639509269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d24858960 con 0x557d25744400 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13ac00640 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d25744000 0x557d255fd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13b600640 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x557d256f3410 tx=0x557d256f4c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff133e00640 1 -- 172.21.15.69:0/1639509269 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d2484a4e0 con 0x557d25744400 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff133e00640 1 -- 172.21.15.69:0/1639509269 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x557d24850c40 con 0x557d25744400 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff133e00640 1 -- 172.21.15.69:0/1639509269 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d2484a9c0 con 0x557d25744400 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.091+0000 7ff13df20280 1 -- 172.21.15.69:0/1639509269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557d24859680 con 0x557d25744400 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.092+0000 7ff13df20280 1 -- 172.21.15.69:0/1639509269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d24859860 con 0x557d25744400 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.092+0000 7ff13df20280 4 main: resolving pool 14 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.093+0000 7ff133e00640 1 -- 172.21.15.69:0/1639509269 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x557d25758c00 con 0x557d25744400 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.094+0000 7ff133e00640 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x557d24b8a000 0x557d255fe100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.094+0000 7ff133e00640 1 -- 172.21.15.69:0/1639509269 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x557d2574aa00 con 0x557d25744400 2024-04-24T08:19:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.094+0000 7ff13df20280 4 main: creating IoCtx.. 2024-04-24T08:19:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.094+0000 7ff13df20280 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x557d25744c00 0x557d255fe680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.094+0000 7ff13df20280 1 -- 172.21.15.69:0/1639509269 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:1 14.13 14:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) -- 0x557d25745000 con 0x557d25744c00 2024-04-24T08:19:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.094+0000 7ff13ac00640 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x557d25744c00 0x557d255fe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.094+0000 7ff13ac00640 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x557d24b8a000 0x557d255fe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.095+0000 7ff13ac00640 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x557d25744c00 0x557d255fe680 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:50.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.096+0000 7ff13ac00640 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x557d24b8a000 0x557d255fe100 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x557d256f32f0 tx=0x557d256f4fc0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:50.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13ac00640 1 -- 172.21.15.69:0/1639509269 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv7 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x557d255ae000 con 0x557d25744c00 2024-04-24T08:19:50.100 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:19:50.100 INFO:teuthology.orchestra.run.smithi069.stdout: "0": { 2024-04-24T08:19:50.100 INFO:teuthology.orchestra.run.smithi069.stdout: "data": { 2024-04-24T08:19:50.100 INFO:teuthology.orchestra.run.smithi069.stdout: "version": 5, 2024-04-24T08:19:50.100 INFO:teuthology.orchestra.run.smithi069.stdout: "inotable": { 2024-04-24T08:19:50.100 INFO:teuthology.orchestra.run.smithi069.stdout: "projected_free": [ 2024-04-24T08:19:50.100 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:19:50.100 INFO:teuthology.orchestra.run.smithi069.stdout: "start": 1099511627779, 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: "len": 1099511627773 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: "free": [ 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: "start": 1099511627779, 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: "len": 1099511627773 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: "result": 0 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:19:50.101 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:19:50.102 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13df20280 1 -- 172.21.15.69:0/1639509269 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x557d25744c00 msgr2=0x557d255fe680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13df20280 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x557d25744c00 0x557d255fe680 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13df20280 1 -- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x557d24b8a000 msgr2=0x557d255fe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13df20280 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x557d24b8a000 0x557d255fe100 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x557d256f32f0 tx=0x557d256f4fc0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13df20280 1 -- 172.21.15.69:0/1639509269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 msgr2=0x557d255fd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13df20280 1 --2- 172.21.15.69:0/1639509269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d25744400 0x557d255fd600 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x557d256f3410 tx=0x557d256f4c00 comp rx=0 tx=0).stop 2024-04-24T08:19:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13b600640 1 -- 172.21.15.69:0/1639509269 reap_dead start 2024-04-24T08:19:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13df20280 1 -- 172.21.15.69:0/1639509269 shutdown_connections 2024-04-24T08:19:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13df20280 1 -- 172.21.15.69:0/1639509269 >> 172.21.15.69:0/1639509269 conn(0x557d25601c00 msgr2=0x557d255ab000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:50.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13df20280 1 -- 172.21.15.69:0/1639509269 shutdown_connections 2024-04-24T08:19:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.098+0000 7ff13df20280 1 -- 172.21.15.69:0/1639509269 wait complete. 2024-04-24T08:19:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.099+0000 7ff13df20280 1 -- 172.21.15.69:0/450845296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d24825c00 msgr2=0x557d255fcb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.099+0000 7ff13df20280 1 --2- 172.21.15.69:0/450845296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d24825c00 0x557d255fcb00 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x557d256510e0 tx=0x557d255be8a0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.099+0000 7ff13df20280 1 -- 172.21.15.69:0/450845296 shutdown_connections 2024-04-24T08:19:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.099+0000 7ff13df20280 1 --2- 172.21.15.69:0/450845296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d24825c00 0x557d255fcb00 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.099+0000 7ff13df20280 1 --2- 172.21.15.69:0/450845296 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557d24825800 0x557d255fc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.099+0000 7ff13df20280 1 --2- 172.21.15.69:0/450845296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557d24824c00 0x557d255fc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.099+0000 7ff13df20280 1 -- 172.21.15.69:0/450845296 >> 172.21.15.69:0/450845296 conn(0x557d24825000 msgr2=0x557d255aa800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.099+0000 7ff13df20280 1 -- 172.21.15.69:0/450845296 shutdown_connections 2024-04-24T08:19:50.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.099+0000 7ff13df20280 1 -- 172.21.15.69:0/450845296 wait complete. 2024-04-24T08:19:50.106 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '0', 'show', 'inode'] in time 0:00:00.103939, result: { "0": { "data": { "version": 5, "inotable": { "projected_free": [ { "start": 1099511627779, "len": 1099511627773 } ], "free": [ { "start": 1099511627779, "len": 1099511627773 } ] } }, "result": 0 } } 2024-04-24T08:19:50.106 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:19:50.106 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:19:50.249 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:19:50.351 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:19:50.351 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:19:50.351 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:19:50.351 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:19:50.411 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:19:50.411 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:19:50.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:19:50.412 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:19:50.460 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-24T08:19:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:19:50.461 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:19:50.461 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:19:50.461 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:19:50.516 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:19:50.517 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:19:50.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:19:50.517 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-24T08:19:50.568 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-24T08:19:50.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:19:50.570 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:19:50.570 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:19:50.571 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:19:50.628 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:19:50.628 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:19:50.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:19:50.628 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-24T08:19:50.678 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-24T08:19:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:19:50.680 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:19:50.680 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:19:50.680 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:19:50.736 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:19:50.736 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:19:50.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:19:50.736 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-24T08:19:50.781 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-24T08:19:50.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:19:50.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:19:50.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.990+0000 7ff5df200640 1 Processor -- start 2024-04-24T08:19:50.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.990+0000 7ff5df200640 1 -- start start 2024-04-24T08:19:50.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.991+0000 7ff5df200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff5d81523f0 0x7ff5d806db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.991+0000 7ff5df200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff5d8072e20 0x7ff5d8073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.991+0000 7ff5df200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5d80737a0 0x7ff5d815ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:50.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.991+0000 7ff5df200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff5d80aeac0 con 0x7ff5d8072e20 2024-04-24T08:19:50.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.991+0000 7ff5df200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff5d80aeda0 con 0x7ff5d81523f0 2024-04-24T08:19:50.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.991+0000 7ff5df200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d80af080 con 0x7ff5d80737a0 2024-04-24T08:19:50.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.991+0000 7ff5dd400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff5d8072e20 0x7ff5d8073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:50.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.991+0000 7ff5dd400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff5d8072e20 0x7ff5d8073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40034/0 (socket says 172.21.15.69:40034) 2024-04-24T08:19:50.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.991+0000 7ff5dd400640 1 -- 172.21.15.69:0/4011591483 learned_addr learned my addr 172.21.15.69:0/4011591483 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:50.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.992+0000 7ff5dd400640 1 -- 172.21.15.69:0/4011591483 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5d80737a0 msgr2=0x7ff5d815ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:50.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.992+0000 7ff5dd400640 1 --2- 172.21.15.69:0/4011591483 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5d80737a0 0x7ff5d815ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.992+0000 7ff5dd400640 1 -- 172.21.15.69:0/4011591483 >> v2:172.21.15.121:3300/0 conn(0x7ff5d81523f0 msgr2=0x7ff5d806db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:50.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.992+0000 7ff5dd400640 1 --2- 172.21.15.69:0/4011591483 >> v2:172.21.15.121:3300/0 conn(0x7ff5d81523f0 0x7ff5d806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.992+0000 7ff5dd400640 1 -- 172.21.15.69:0/4011591483 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d80af360 con 0x7ff5d8072e20 2024-04-24T08:19:50.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.993+0000 7ff5dd400640 1 --2- 172.21.15.69:0/4011591483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 0x7ff5d8073260 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7ff5d806cb50 tx=0x7ff5c802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=797e74e3545ff921 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.993+0000 7ff5dca00640 1 -- 172.21.15.69:0/4011591483 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8037070 con 0x7ff5d8072e20 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.993+0000 7ff5dca00640 1 -- 172.21.15.69:0/4011591483 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5c8032440 con 0x7ff5d8072e20 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.993+0000 7ff5dca00640 1 -- 172.21.15.69:0/4011591483 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8032810 con 0x7ff5d8072e20 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.994+0000 7ff5df200640 1 -- 172.21.15.69:0/4011591483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 msgr2=0x7ff5d8073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.994+0000 7ff5df200640 1 --2- 172.21.15.69:0/4011591483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 0x7ff5d8073260 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7ff5d806cb50 tx=0x7ff5c802a140 comp rx=0 tx=0).stop 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.994+0000 7ff5df200640 1 -- 172.21.15.69:0/4011591483 shutdown_connections 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.994+0000 7ff5df200640 1 --2- 172.21.15.69:0/4011591483 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5d80737a0 0x7ff5d815ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.994+0000 7ff5df200640 1 --2- 172.21.15.69:0/4011591483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 0x7ff5d8073260 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.994+0000 7ff5df200640 1 --2- 172.21.15.69:0/4011591483 >> v2:172.21.15.121:3300/0 conn(0x7ff5d81523f0 0x7ff5d806db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.994+0000 7ff5df200640 1 -- 172.21.15.69:0/4011591483 >> 172.21.15.69:0/4011591483 conn(0x7ff5d80d0160 msgr2=0x7ff5d80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.994+0000 7ff5df200640 1 -- 172.21.15.69:0/4011591483 shutdown_connections 2024-04-24T08:19:50.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.994+0000 7ff5df200640 1 -- 172.21.15.69:0/4011591483 wait complete. 2024-04-24T08:19:50.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.996+0000 7ff5df200640 1 Processor -- start 2024-04-24T08:19:50.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.996+0000 7ff5df200640 1 -- start start 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.997+0000 7ff5df200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 0x7ff5d8156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.997+0000 7ff5df200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5d80737a0 0x7ff5d8158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.997+0000 7ff5df200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5d81523f0 0x7ff5d8147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.997+0000 7ff5df200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d815f4b0 con 0x7ff5d8072e20 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.997+0000 7ff5df200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d81527d0 con 0x7ff5d80737a0 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.997+0000 7ff5df200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff5d815f630 con 0x7ff5d81523f0 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.997+0000 7ff5dde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 0x7ff5d8156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.997+0000 7ff5dde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 0x7ff5d8156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40036/0 (socket says 172.21.15.69:40036) 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.997+0000 7ff5dde00640 1 -- 172.21.15.69:0/963343232 learned_addr learned my addr 172.21.15.69:0/963343232 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5dde00640 1 -- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5d81523f0 msgr2=0x7ff5d8147460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5dde00640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5d81523f0 0x7ff5d8147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5dde00640 1 -- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5d80737a0 msgr2=0x7ff5d8158af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5dde00640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5d80737a0 0x7ff5d8158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5dde00640 1 -- 172.21.15.69:0/963343232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d8147d90 con 0x7ff5d8072e20 2024-04-24T08:19:51.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5dde00640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 0x7ff5d8156040 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00b410 tx=0x7ff5cc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:51.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5d2000640 1 -- 172.21.15.69:0/963343232 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc015020 con 0x7ff5d8072e20 2024-04-24T08:19:51.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5d2000640 1 -- 172.21.15.69:0/963343232 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5cc011850 con 0x7ff5d8072e20 2024-04-24T08:19:51.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5df200640 1 -- 172.21.15.69:0/963343232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d8152b40 con 0x7ff5d8072e20 2024-04-24T08:19:51.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5d2000640 1 -- 172.21.15.69:0/963343232 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc011c40 con 0x7ff5d8072e20 2024-04-24T08:19:51.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:50.998+0000 7ff5df200640 1 -- 172.21.15.69:0/963343232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d8153100 con 0x7ff5d8072e20 2024-04-24T08:19:51.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.000+0000 7ff5df200640 1 -- 172.21.15.69:0/963343232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5d8075d50 con 0x7ff5d8072e20 2024-04-24T08:19:51.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.000+0000 7ff5d2000640 1 -- 172.21.15.69:0/963343232 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff5cc018020 con 0x7ff5d8072e20 2024-04-24T08:19:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.002+0000 7ff5d2000640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5bc06c8f0 0x7ff5bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.002+0000 7ff5d2000640 1 -- 172.21.15.69:0/963343232 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7ff5cc08bba0 con 0x7ff5d8072e20 2024-04-24T08:19:51.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.002+0000 7ff5dd400640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5bc06c8f0 0x7ff5bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.004+0000 7ff5dd400640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5bc06c8f0 0x7ff5bc06edb0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8034000 tx=0x7ff5c802c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:51.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.007+0000 7ff5d2000640 1 -- 172.21.15.69:0/963343232 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff5cc01f020 con 0x7ff5d8072e20 2024-04-24T08:19:51.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.247+0000 7ff5df200640 1 -- 172.21.15.69:0/963343232 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5d80726a0 con 0x7ff5d8072e20 2024-04-24T08:19:51.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.249+0000 7ff5d2000640 1 -- 172.21.15.69:0/963343232 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5175 (secure 0 0 0) 0x7ff5cc0172c0 con 0x7ff5d8072e20 2024-04-24T08:19:51.250 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:51.250 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9111,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3686931568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3686931568},{"type":"v1","addr":"172.21.15.69:6835","nonce":3686931568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:19:49.975280+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-24T08:19:51.251 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 91 2024-04-24T08:19:51.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.251+0000 7ff5df200640 1 -- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5bc06c8f0 msgr2=0x7ff5bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.251+0000 7ff5df200640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5bc06c8f0 0x7ff5bc06edb0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8034000 tx=0x7ff5c802c000 comp rx=0 tx=0).stop 2024-04-24T08:19:51.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.251+0000 7ff5df200640 1 -- 172.21.15.69:0/963343232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 msgr2=0x7ff5d8156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.251+0000 7ff5df200640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 0x7ff5d8156040 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00b410 tx=0x7ff5cc00b8d0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.251+0000 7ff5df200640 1 -- 172.21.15.69:0/963343232 shutdown_connections 2024-04-24T08:19:51.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.251+0000 7ff5df200640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff5bc06c8f0 0x7ff5bc06edb0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.251+0000 7ff5df200640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff5d81523f0 0x7ff5d8147460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.251+0000 7ff5df200640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5d80737a0 0x7ff5d8158af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.251+0000 7ff5df200640 1 --2- 172.21.15.69:0/963343232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8072e20 0x7ff5d8156040 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.251+0000 7ff5df200640 1 -- 172.21.15.69:0/963343232 >> 172.21.15.69:0/963343232 conn(0x7ff5d80d0160 msgr2=0x7ff5d80b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:51.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.252+0000 7ff5df200640 1 -- 172.21.15.69:0/963343232 shutdown_connections 2024-04-24T08:19:51.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.252+0000 7ff5df200640 1 -- 172.21.15.69:0/963343232 wait complete. 2024-04-24T08:19:51.272 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:19:51.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.465+0000 7f039f200640 1 Processor -- start 2024-04-24T08:19:51.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.465+0000 7f039f200640 1 -- start start 2024-04-24T08:19:51.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.467+0000 7f039f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03a0162300 0x7f03a0143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.467+0000 7f039f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f03a0072230 0x7f03a0143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.467+0000 7f039f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03a015f540 con 0x7f03a0071e50 2024-04-24T08:19:51.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.467+0000 7f039f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f03a015f820 con 0x7f03a0072230 2024-04-24T08:19:51.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.467+0000 7f039f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f03a015fb00 con 0x7f03a0162300 2024-04-24T08:19:51.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.467+0000 7f039de00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f03a0071e50 0x7f03a015ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33630/0 (socket says 172.21.15.69:33630) 2024-04-24T08:19:51.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.467+0000 7f039de00640 1 -- 172.21.15.69:0/3332142275 learned_addr learned my addr 172.21.15.69:0/3332142275 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:51.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.467+0000 7f039e800640 1 --2- 172.21.15.69:0/3332142275 >> v2:172.21.15.69:3300/0 conn(0x7f03a0072230 0x7f03a0143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.468+0000 7f039d400640 1 --2- 172.21.15.69:0/3332142275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03a0162300 0x7f03a0143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.468+0000 7f039ca00640 1 -- 172.21.15.69:0/3332142275 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 803113639 0 0) 0x7f03a015f540 con 0x7f03a0071e50 2024-04-24T08:19:51.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.468+0000 7f039ca00640 1 -- 172.21.15.69:0/3332142275 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0384003520 con 0x7f03a0071e50 2024-04-24T08:19:51.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.468+0000 7f039e800640 1 -- 172.21.15.69:0/3332142275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03a0162300 msgr2=0x7f03a0143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:51.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.468+0000 7f039e800640 1 --2- 172.21.15.69:0/3332142275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03a0162300 0x7f03a0143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.468+0000 7f039e800640 1 -- 172.21.15.69:0/3332142275 >> v1:172.21.15.69:6789/0 conn(0x7f03a0071e50 legacy=0x7f03a015ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.468+0000 7f039e800640 1 -- 172.21.15.69:0/3332142275 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a006dda0 con 0x7f03a0072230 2024-04-24T08:19:51.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.469+0000 7f039e800640 1 --2- 172.21.15.69:0/3332142275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0072230 0x7f03a0143bd0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f0398009d90 tx=0x7f039802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1fff7d7efecd02 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:51.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.469+0000 7f039ca00640 1 -- 172.21.15.69:0/3332142275 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0398007740 con 0x7f03a0072230 2024-04-24T08:19:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.469+0000 7f039ca00640 1 -- 172.21.15.69:0/3332142275 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03980078e0 con 0x7f03a0072230 2024-04-24T08:19:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.469+0000 7f039ca00640 1 -- 172.21.15.69:0/3332142275 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0398007cb0 con 0x7f03a0072230 2024-04-24T08:19:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.470+0000 7f039f200640 1 -- 172.21.15.69:0/3332142275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0072230 msgr2=0x7f03a0143bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.470+0000 7f039f200640 1 --2- 172.21.15.69:0/3332142275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0072230 0x7f03a0143bd0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f0398009d90 tx=0x7f039802a660 comp rx=0 tx=0).stop 2024-04-24T08:19:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.470+0000 7f039f200640 1 -- 172.21.15.69:0/3332142275 shutdown_connections 2024-04-24T08:19:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.470+0000 7f039f200640 1 --2- 172.21.15.69:0/3332142275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0072230 0x7f03a0143bd0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.470+0000 7f039f200640 1 --2- 172.21.15.69:0/3332142275 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03a0162300 0x7f03a0143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.470+0000 7f039f200640 1 -- 172.21.15.69:0/3332142275 >> 172.21.15.69:0/3332142275 conn(0x7f03a00d0160 msgr2=0x7f03a00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.470+0000 7f039f200640 1 -- 172.21.15.69:0/3332142275 shutdown_connections 2024-04-24T08:19:51.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.470+0000 7f039f200640 1 -- 172.21.15.69:0/3332142275 wait complete. 2024-04-24T08:19:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.471+0000 7f039f200640 1 Processor -- start 2024-04-24T08:19:51.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.472+0000 7f039f200640 1 -- start start 2024-04-24T08:19:51.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03a0071e50 0x7f03a009c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03a0072230 0x7f03a0142620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0162300 0x7f03a009a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f03a0144290 con 0x7f03a0162300 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f03a0144110 con 0x7f03a0071e50 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f03a0144410 con 0x7f03a0072230 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0162300 0x7f03a009a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0162300 0x7f03a009a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40064/0 (socket says 172.21.15.69:40064) 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039e800640 1 -- 172.21.15.69:0/2745868642 learned_addr learned my addr 172.21.15.69:0/2745868642 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039d400640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03a0072230 0x7f03a0142620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039de00640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03a0071e50 0x7f03a009c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039e800640 1 -- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03a0072230 msgr2=0x7f03a0142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039e800640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03a0072230 0x7f03a0142620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039e800640 1 -- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03a0071e50 msgr2=0x7f03a009c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039e800640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03a0071e50 0x7f03a009c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.473+0000 7f039e800640 1 -- 172.21.15.69:0/2745868642 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a01f44b0 con 0x7f03a0162300 2024-04-24T08:19:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.474+0000 7f039e800640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0162300 0x7f03a009a810 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f0398009eb0 tx=0x7f0398004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.474+0000 7f0396000640 1 -- 172.21.15.69:0/2745868642 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0398040070 con 0x7f03a0162300 2024-04-24T08:19:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.474+0000 7f0396000640 1 -- 172.21.15.69:0/2745868642 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0398031bc0 con 0x7f03a0162300 2024-04-24T08:19:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.474+0000 7f039f200640 1 -- 172.21.15.69:0/2745868642 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a01f4790 con 0x7f03a0162300 2024-04-24T08:19:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.474+0000 7f0396000640 1 -- 172.21.15.69:0/2745868642 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0398009450 con 0x7f03a0162300 2024-04-24T08:19:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.474+0000 7f039f200640 1 -- 172.21.15.69:0/2745868642 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a01f4d50 con 0x7f03a0162300 2024-04-24T08:19:51.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.475+0000 7f039f200640 1 -- 172.21.15.69:0/2745868642 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03a0075d50 con 0x7f03a0162300 2024-04-24T08:19:51.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.476+0000 7f0396000640 1 -- 172.21.15.69:0/2745868642 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0398039040 con 0x7f03a0162300 2024-04-24T08:19:51.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.478+0000 7f0396000640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f038006c8f0 0x7f038006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.478+0000 7f0396000640 1 -- 172.21.15.69:0/2745868642 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7f03980ad400 con 0x7f03a0162300 2024-04-24T08:19:51.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.478+0000 7f039de00640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f038006c8f0 0x7f038006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.479+0000 7f039de00640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f038006c8f0 0x7f038006edb0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0390000f60 tx=0x7f0390009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:51.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.482+0000 7f0396000640 1 -- 172.21.15.69:0/2745868642 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f039807b7b0 con 0x7f03a0162300 2024-04-24T08:19:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.683+0000 7f039f200640 1 -- 172.21.15.69:0/2745868642 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f03a00d0d90 con 0x7f03a0162300 2024-04-24T08:19:51.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.686+0000 7f0396000640 1 -- 172.21.15.69:0/2745868642 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+16841 (secure 0 0 0) 0x7f039807b0b0 con 0x7f03a0162300 2024-04-24T08:19:51.688 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:51.688 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":99,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:19:49.975068+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:08.156864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:09.368519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/3642138906":"2024-04-25T08:19:49.975039+0000","172.21.15.69:6833/3642138906":"2024-04-25T08:19:49.975039+0000","172.21.15.69:6835/412016900":"2024-04-25T08:19:12.588993+0000","172.21.15.69:6833/412016900":"2024-04-25T08:19:12.588993+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:19:51.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.688+0000 7f039f200640 1 -- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f038006c8f0 msgr2=0x7f038006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f038006c8f0 0x7f038006edb0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0390000f60 tx=0x7f0390009040 comp rx=0 tx=0).stop 2024-04-24T08:19:51.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 -- 172.21.15.69:0/2745868642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0162300 msgr2=0x7f03a009a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0162300 0x7f03a009a810 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7f0398009eb0 tx=0x7f0398004a40 comp rx=0 tx=0).stop 2024-04-24T08:19:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 -- 172.21.15.69:0/2745868642 shutdown_connections 2024-04-24T08:19:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f038006c8f0 0x7f038006edb0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03a0162300 0x7f03a009a810 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f03a0072230 0x7f03a0142620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 --2- 172.21.15.69:0/2745868642 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03a0071e50 0x7f03a009c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 -- 172.21.15.69:0/2745868642 >> 172.21.15.69:0/2745868642 conn(0x7f03a00d0160 msgr2=0x7f03a0072e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 -- 172.21.15.69:0/2745868642 shutdown_connections 2024-04-24T08:19:51.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.689+0000 7f039f200640 1 -- 172.21.15.69:0/2745868642 wait complete. 2024-04-24T08:19:51.710 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:18:08.156864+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:18:09.368519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:19:51.710 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:19:51.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.912+0000 7f8d70600640 1 Processor -- start 2024-04-24T08:19:51.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.913+0000 7f8d70600640 1 -- start start 2024-04-24T08:19:51.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.914+0000 7f8d70600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d6815c820 0x7f8d680b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.914+0000 7f8d70600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8d680b46a0 0x7f8d6806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.914+0000 7f8d70600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d6815cc00 con 0x7f8d6806fd10 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.914+0000 7f8d70600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8d6806ed30 con 0x7f8d6815c820 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.914+0000 7f8d70600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8d680b4e10 con 0x7f8d680b46a0 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.915+0000 7f8d6dc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d6815c820 0x7f8d680b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.915+0000 7f8d6e600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8d6806fd10 0x7f8d680ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33634/0 (socket says 172.21.15.69:33634) 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.915+0000 7f8d6dc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8d6815c820 0x7f8d680b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40074/0 (socket says 172.21.15.69:40074) 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.915+0000 7f8d6e600640 1 -- 172.21.15.69:0/2653717730 learned_addr learned my addr 172.21.15.69:0/2653717730 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.915+0000 7f8d6d200640 1 --2- 172.21.15.69:0/2653717730 >> v2:172.21.15.121:3300/0 conn(0x7f8d680b46a0 0x7f8d6806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.915+0000 7f8d6dc00640 1 -- 172.21.15.69:0/2653717730 >> v2:172.21.15.121:3300/0 conn(0x7f8d680b46a0 msgr2=0x7f8d6806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.915+0000 7f8d6dc00640 1 --2- 172.21.15.69:0/2653717730 >> v2:172.21.15.121:3300/0 conn(0x7f8d680b46a0 0x7f8d6806f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.915+0000 7f8d6dc00640 1 -- 172.21.15.69:0/2653717730 >> v1:172.21.15.69:6789/0 conn(0x7f8d6806fd10 legacy=0x7f8d680ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.915+0000 7f8d6dc00640 1 -- 172.21.15.69:0/2653717730 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d680b50f0 con 0x7f8d6815c820 2024-04-24T08:19:51.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.915+0000 7f8d67e00640 1 -- 172.21.15.69:0/2653717730 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2799845471 0 0) 0x7f8d6815cc00 con 0x7f8d6806fd10 2024-04-24T08:19:51.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.916+0000 7f8d6dc00640 1 --2- 172.21.15.69:0/2653717730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 0x7f8d680b3e90 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f8d6806e200 tx=0x7f8d5c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9df388534761710 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:51.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.917+0000 7f8d67e00640 1 -- 172.21.15.69:0/2653717730 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c037070 con 0x7f8d6815c820 2024-04-24T08:19:51.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.917+0000 7f8d67e00640 1 -- 172.21.15.69:0/2653717730 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d5c032440 con 0x7f8d6815c820 2024-04-24T08:19:51.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.917+0000 7f8d67e00640 1 -- 172.21.15.69:0/2653717730 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c032810 con 0x7f8d6815c820 2024-04-24T08:19:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.917+0000 7f8d70600640 1 -- 172.21.15.69:0/2653717730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 msgr2=0x7f8d680b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.917+0000 7f8d70600640 1 --2- 172.21.15.69:0/2653717730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 0x7f8d680b3e90 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f8d6806e200 tx=0x7f8d5c02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.917+0000 7f8d70600640 1 -- 172.21.15.69:0/2653717730 shutdown_connections 2024-04-24T08:19:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.917+0000 7f8d70600640 1 --2- 172.21.15.69:0/2653717730 >> v2:172.21.15.121:3300/0 conn(0x7f8d680b46a0 0x7f8d6806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.917+0000 7f8d70600640 1 --2- 172.21.15.69:0/2653717730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 0x7f8d680b3e90 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.917+0000 7f8d70600640 1 -- 172.21.15.69:0/2653717730 >> 172.21.15.69:0/2653717730 conn(0x7f8d680d0350 msgr2=0x7f8d680d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.917+0000 7f8d70600640 1 -- 172.21.15.69:0/2653717730 shutdown_connections 2024-04-24T08:19:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.918+0000 7f8d70600640 1 -- 172.21.15.69:0/2653717730 wait complete. 2024-04-24T08:19:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.919+0000 7f8d70600640 1 Processor -- start 2024-04-24T08:19:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.919+0000 7f8d70600640 1 -- start start 2024-04-24T08:19:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.920+0000 7f8d70600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d6806fd10 0x7f8d680cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.920+0000 7f8d70600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d680b46a0 0x7f8d6815f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.920+0000 7f8d70600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 0x7f8d681f7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.920+0000 7f8d70600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d680b4b30 con 0x7f8d6815c820 2024-04-24T08:19:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.920+0000 7f8d70600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6806f2b0 con 0x7f8d680b46a0 2024-04-24T08:19:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.920+0000 7f8d70600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8d680ce960 con 0x7f8d6806fd10 2024-04-24T08:19:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 0x7f8d681f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 0x7f8d681f7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40078/0 (socket says 172.21.15.69:40078) 2024-04-24T08:19:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6e600640 1 -- 172.21.15.69:0/807814018 learned_addr learned my addr 172.21.15.69:0/807814018 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6dc00640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d6806fd10 0x7f8d680cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6e600640 1 -- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d6806fd10 msgr2=0x7f8d680cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6d200640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d680b46a0 0x7f8d6815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6e600640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d6806fd10 0x7f8d680cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6e600640 1 -- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d680b46a0 msgr2=0x7f8d6815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6e600640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d680b46a0 0x7f8d6815f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6e600640 1 -- 172.21.15.69:0/807814018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d6809bd00 con 0x7f8d6815c820 2024-04-24T08:19:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.921+0000 7f8d6e600640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 0x7f8d681f7990 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c730 tx=0x7f8d6000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.922+0000 7f8d65600640 1 -- 172.21.15.69:0/807814018 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d60018070 con 0x7f8d6815c820 2024-04-24T08:19:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.922+0000 7f8d65600640 1 -- 172.21.15.69:0/807814018 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d6000ce40 con 0x7f8d6815c820 2024-04-24T08:19:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.922+0000 7f8d65600640 1 -- 172.21.15.69:0/807814018 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000a860 con 0x7f8d6815c820 2024-04-24T08:19:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.922+0000 7f8d70600640 1 -- 172.21.15.69:0/807814018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d680cb740 con 0x7f8d6815c820 2024-04-24T08:19:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.922+0000 7f8d70600640 1 -- 172.21.15.69:0/807814018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d680cbd00 con 0x7f8d6815c820 2024-04-24T08:19:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.923+0000 7f8d70600640 1 -- 172.21.15.69:0/807814018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d680cbf40 con 0x7f8d6815c820 2024-04-24T08:19:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.924+0000 7f8d65600640 1 -- 172.21.15.69:0/807814018 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8d6000aa00 con 0x7f8d6815c820 2024-04-24T08:19:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.925+0000 7f8d65600640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d4806c8f0 0x7f8d4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.925+0000 7f8d65600640 1 -- 172.21.15.69:0/807814018 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7f8d6008b210 con 0x7f8d6815c820 2024-04-24T08:19:51.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.926+0000 7f8d6dc00640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d4806c8f0 0x7f8d4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:51.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.927+0000 7f8d6dc00640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d4806c8f0 0x7f8d4806edb0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c034000 tx=0x7f8d5c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:51.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:51.930+0000 7f8d65600640 1 -- 172.21.15.69:0/807814018 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d680cbf40 con 0x7f8d6815c820 2024-04-24T08:19:52.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.131+0000 7f8d70600640 1 -- 172.21.15.69:0/807814018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d6806f7d0 con 0x7f8d6815c820 2024-04-24T08:19:52.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.132+0000 7f8d65600640 1 -- 172.21.15.69:0/807814018 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7f8d6806f7d0 con 0x7f8d6815c820 2024-04-24T08:19:52.134 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:19:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 -- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d4806c8f0 msgr2=0x7f8d4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d4806c8f0 0x7f8d4806edb0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c034000 tx=0x7f8d5c02e000 comp rx=0 tx=0).stop 2024-04-24T08:19:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 -- 172.21.15.69:0/807814018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 msgr2=0x7f8d681f7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 0x7f8d681f7990 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f8d6000c730 tx=0x7f8d6000cbf0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 -- 172.21.15.69:0/807814018 shutdown_connections 2024-04-24T08:19:52.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8d4806c8f0 0x7f8d4806edb0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d6815c820 0x7f8d681f7990 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d680b46a0 0x7f8d6815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 --2- 172.21.15.69:0/807814018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8d6806fd10 0x7f8d680cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 -- 172.21.15.69:0/807814018 >> 172.21.15.69:0/807814018 conn(0x7f8d680d0350 msgr2=0x7f8d68071d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:52.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 -- 172.21.15.69:0/807814018 shutdown_connections 2024-04-24T08:19:52.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.135+0000 7f8d70600640 1 -- 172.21.15.69:0/807814018 wait complete. 2024-04-24T08:19:52.157 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:19:52.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.353+0000 7f9329e00640 1 Processor -- start 2024-04-24T08:19:52.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.353+0000 7f9329e00640 1 -- start start 2024-04-24T08:19:52.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.354+0000 7f9329e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f932415faf0 0x7f93240d4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.354+0000 7f9329e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f932406cc30 con 0x7f932415e870 2024-04-24T08:19:52.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.354+0000 7f9329e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f932406ced0 con 0x7f93240d3350 2024-04-24T08:19:52.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.354+0000 7f9329e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f932406e9c0 con 0x7f932415faf0 2024-04-24T08:19:52.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.354+0000 7f9322a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f932415e870 0x7f93240d2c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33648/0 (socket says 172.21.15.69:33648) 2024-04-24T08:19:52.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.354+0000 7f9322a00640 1 -- 172.21.15.69:0/3957057901 learned_addr learned my addr 172.21.15.69:0/3957057901 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9323400640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f932415faf0 0x7f93240d4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2410360181 0 0) 0x7f932406cc30 con 0x7f932415e870 2024-04-24T08:19:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9308003520 con 0x7f932415e870 2024-04-24T08:19:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2985054673 0 0) 0x7f9308003520 con 0x7f932415e870 2024-04-24T08:19:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f932415faf0 msgr2=0x7f93240d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f932415faf0 0x7f93240d4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 >> v1:172.21.15.121:6789/0 conn(0x7f93240d3350 legacy=0x7f93241615a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932406d1b0 con 0x7f932415e870 2024-04-24T08:19:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f9314001690 con 0x7f932415e870 2024-04-24T08:19:52.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 >> v1:172.21.15.69:6789/0 conn(0x7f932415e870 legacy=0x7f93240d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f932415faf0 0x7f93080039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9308004060 0x7f9308004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9308004d40 0x7f93080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f930800bbd0 con 0x7f9308004060 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f930800bd80 con 0x7f9308004d40 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.355+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f930800bf80 con 0x7f932415faf0 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.356+0000 7f9322000640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9308004060 0x7f9308004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.356+0000 7f9323400640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9308004d40 0x7f93080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.356+0000 7f9322a00640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f932415faf0 0x7f93080039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.356+0000 7f9322000640 1 -- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f932415faf0 msgr2=0x7f93080039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.356+0000 7f9322000640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f932415faf0 0x7f93080039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.356+0000 7f9322000640 1 -- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9308004d40 msgr2=0x7f93080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.356+0000 7f9322000640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9308004d40 0x7f93080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.356+0000 7f9322000640 1 -- 172.21.15.69:0/3957057901 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9308003520 con 0x7f9308004060 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.357+0000 7f9322000640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9308004060 0x7f9308004800 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f930c001750 tx=0x7f9308005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15761d0024fbb9aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.357+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c001370 con 0x7f9308004060 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.357+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f930c0052d0 con 0x7f9308004060 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.357+0000 7f9321600640 1 -- 172.21.15.69:0/3957057901 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c02abe0 con 0x7f9308004060 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.358+0000 7f9329e00640 1 -- 172.21.15.69:0/3957057901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9308004060 msgr2=0x7f9308004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.358+0000 7f9329e00640 1 --2- 172.21.15.69:0/3957057901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9308004060 0x7f9308004800 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f930c001750 tx=0x7f9308005830 comp rx=0 tx=0).stop 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.358+0000 7f9323400640 1 -- 172.21.15.69:0/3957057901 reap_dead start 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.358+0000 7f9329e00640 1 -- 172.21.15.69:0/3957057901 shutdown_connections 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.358+0000 7f9329e00640 1 -- 172.21.15.69:0/3957057901 >> 172.21.15.69:0/3957057901 conn(0x7f932408bf60 msgr2=0x7f93240d46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.358+0000 7f9329e00640 1 -- 172.21.15.69:0/3957057901 shutdown_connections 2024-04-24T08:19:52.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.358+0000 7f9329e00640 1 -- 172.21.15.69:0/3957057901 wait complete. 2024-04-24T08:19:52.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.360+0000 7f9329e00640 1 Processor -- start 2024-04-24T08:19:52.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.360+0000 7f9329e00640 1 -- start start 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.361+0000 7f9329e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9308004060 0x7f93240b9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.361+0000 7f9329e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324155870 0x7f93240b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.361+0000 7f9329e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93240b4430 0x7f93241576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.361+0000 7f9329e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f93241602d0 con 0x7f9324155870 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.361+0000 7f9329e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f93240d55a0 con 0x7f93240b4430 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.361+0000 7f9329e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f932415ec50 con 0x7f9308004060 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.361+0000 7f9322000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324155870 0x7f93240b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.361+0000 7f9322000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324155870 0x7f93240b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40102/0 (socket says 172.21.15.69:40102) 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.361+0000 7f9322000640 1 -- 172.21.15.69:0/4118547974 learned_addr learned my addr 172.21.15.69:0/4118547974 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f9322a00640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9308004060 0x7f93240b9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f9322000640 1 -- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9308004060 msgr2=0x7f93240b9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f9322000640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9308004060 0x7f93240b9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f9322000640 1 -- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93240b4430 msgr2=0x7f93241576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f9322000640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93240b4430 0x7f93241576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f9322000640 1 -- 172.21.15.69:0/4118547974 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93240ba180 con 0x7f9324155870 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f9322000640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324155870 0x7f93240b3ef0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f930c000c00 tx=0x7f930c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f931b400640 1 -- 172.21.15.69:0/4118547974 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c042070 con 0x7f9324155870 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f931b400640 1 -- 172.21.15.69:0/4118547974 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f930c005480 con 0x7f9324155870 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f931b400640 1 -- 172.21.15.69:0/4118547974 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c037040 con 0x7f9324155870 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.362+0000 7f9329e00640 1 -- 172.21.15.69:0/4118547974 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93240b7dc0 con 0x7f9324155870 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.363+0000 7f9329e00640 1 -- 172.21.15.69:0/4118547974 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93240b8380 con 0x7f9324155870 2024-04-24T08:19:52.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.364+0000 7f9329e00640 1 -- 172.21.15.69:0/4118547974 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93240b4850 con 0x7f9324155870 2024-04-24T08:19:52.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.365+0000 7f931b400640 1 -- 172.21.15.69:0/4118547974 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f930c0069f0 con 0x7f9324155870 2024-04-24T08:19:52.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.366+0000 7f931b400640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f930006c8f0 0x7f930006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.366+0000 7f931b400640 1 -- 172.21.15.69:0/4118547974 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7f930c0ab4f0 con 0x7f9324155870 2024-04-24T08:19:52.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.367+0000 7f9322a00640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f930006c8f0 0x7f930006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.368+0000 7f9322a00640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f930006c8f0 0x7f930006edb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9314001c10 tx=0x7f9314000f30 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:52.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.371+0000 7f931b400640 1 -- 172.21.15.69:0/4118547974 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f930c079740 con 0x7f9324155870 2024-04-24T08:19:52.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.565+0000 7f9329e00640 1 -- 172.21.15.69:0/4118547974 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9324069190 con 0x7f9324155870 2024-04-24T08:19:52.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.566+0000 7f931b400640 1 -- 172.21.15.69:0/4118547974 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) ==== 104+0+11 (secure 0 0 0) 0x7f930c079040 con 0x7f9324155870 2024-04-24T08:19:52.567 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:19:52.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 -- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f930006c8f0 msgr2=0x7f930006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f930006c8f0 0x7f930006edb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9314001c10 tx=0x7f9314000f30 comp rx=0 tx=0).stop 2024-04-24T08:19:52.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 -- 172.21.15.69:0/4118547974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324155870 msgr2=0x7f93240b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324155870 0x7f93240b3ef0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f930c000c00 tx=0x7f930c001120 comp rx=0 tx=0).stop 2024-04-24T08:19:52.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 -- 172.21.15.69:0/4118547974 shutdown_connections 2024-04-24T08:19:52.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f930006c8f0 0x7f930006edb0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9324155870 0x7f93240b3ef0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93240b4430 0x7f93241576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 --2- 172.21.15.69:0/4118547974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9308004060 0x7f93240b9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 -- 172.21.15.69:0/4118547974 >> 172.21.15.69:0/4118547974 conn(0x7f932408bf60 msgr2=0x7f93240b2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:52.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.568+0000 7f9329e00640 1 -- 172.21.15.69:0/4118547974 shutdown_connections 2024-04-24T08:19:52.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.569+0000 7f9329e00640 1 -- 172.21.15.69:0/4118547974 wait complete. 2024-04-24T08:19:52.587 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:19:52.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.772+0000 7fcd5b200640 1 Processor -- start 2024-04-24T08:19:52.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.772+0000 7fcd5b200640 1 -- start start 2024-04-24T08:19:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.773+0000 7fcd5b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcd5c1606e0 0x7fcd5c167360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.773+0000 7fcd5b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd5c072450 con 0x7fcd5c071d70 2024-04-24T08:19:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.773+0000 7fcd5b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd5c072730 con 0x7fcd5c1420a0 2024-04-24T08:19:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.773+0000 7fcd5b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcd5c0cf1e0 con 0x7fcd5c1606e0 2024-04-24T08:19:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.774+0000 7fcd59400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcd5c071d70 0x7fcd5c15ffd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33664/0 (socket says 172.21.15.69:33664) 2024-04-24T08:19:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.774+0000 7fcd59400640 1 -- 172.21.15.69:0/1317652141 learned_addr learned my addr 172.21.15.69:0/1317652141 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:52.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.774+0000 7fcd5a800640 1 --2- 172.21.15.69:0/1317652141 >> v2:172.21.15.69:3300/0 conn(0x7fcd5c1606e0 0x7fcd5c167360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.774+0000 7fcd5a800640 1 -- 172.21.15.69:0/1317652141 >> v1:172.21.15.121:6789/0 conn(0x7fcd5c1420a0 legacy=0x7fcd5c142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.774+0000 7fcd58a00640 1 -- 172.21.15.69:0/1317652141 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1840755535 0 0) 0x7fcd5c072450 con 0x7fcd5c071d70 2024-04-24T08:19:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.774+0000 7fcd5a800640 1 -- 172.21.15.69:0/1317652141 >> v1:172.21.15.69:6789/0 conn(0x7fcd5c071d70 legacy=0x7fcd5c15ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.774+0000 7fcd5a800640 1 -- 172.21.15.69:0/1317652141 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5c072a10 con 0x7fcd5c1606e0 2024-04-24T08:19:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.775+0000 7fcd5a800640 1 --2- 172.21.15.69:0/1317652141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1606e0 0x7fcd5c167360 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009d90 tx=0x7fcd5402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7db3e80165efee7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.775+0000 7fcd58a00640 1 -- 172.21.15.69:0/1317652141 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54007740 con 0x7fcd5c1606e0 2024-04-24T08:19:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.775+0000 7fcd58a00640 1 -- 172.21.15.69:0/1317652141 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd540078e0 con 0x7fcd5c1606e0 2024-04-24T08:19:52.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.775+0000 7fcd58a00640 1 -- 172.21.15.69:0/1317652141 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54007cb0 con 0x7fcd5c1606e0 2024-04-24T08:19:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.776+0000 7fcd5b200640 1 -- 172.21.15.69:0/1317652141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1606e0 msgr2=0x7fcd5c167360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.776+0000 7fcd5b200640 1 --2- 172.21.15.69:0/1317652141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1606e0 0x7fcd5c167360 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009d90 tx=0x7fcd5402a6a0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.776+0000 7fcd5b200640 1 -- 172.21.15.69:0/1317652141 shutdown_connections 2024-04-24T08:19:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.776+0000 7fcd5b200640 1 --2- 172.21.15.69:0/1317652141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1606e0 0x7fcd5c167360 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.776+0000 7fcd5b200640 1 -- 172.21.15.69:0/1317652141 >> 172.21.15.69:0/1317652141 conn(0x7fcd5c067b80 msgr2=0x7fcd5c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.776+0000 7fcd5b200640 1 -- 172.21.15.69:0/1317652141 shutdown_connections 2024-04-24T08:19:52.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.776+0000 7fcd5b200640 1 -- 172.21.15.69:0/1317652141 wait complete. 2024-04-24T08:19:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.778+0000 7fcd5b200640 1 Processor -- start 2024-04-24T08:19:52.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.778+0000 7fcd5b200640 1 -- start start 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.779+0000 7fcd5b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c071d70 0x7fcd5c156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.779+0000 7fcd5b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1420a0 0x7fcd5c1e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.779+0000 7fcd5b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd5c1606e0 0x7fcd5c1e6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.779+0000 7fcd5b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c0d0680 con 0x7fcd5c1420a0 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.779+0000 7fcd5b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5c072200 con 0x7fcd5c071d70 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.779+0000 7fcd5b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcd5c0d0800 con 0x7fcd5c1606e0 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.779+0000 7fcd59400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1420a0 0x7fcd5c1e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.779+0000 7fcd59400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1420a0 0x7fcd5c1e5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40126/0 (socket says 172.21.15.69:40126) 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.779+0000 7fcd59400640 1 -- 172.21.15.69:0/493109081 learned_addr learned my addr 172.21.15.69:0/493109081 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd59400640 1 -- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd5c1606e0 msgr2=0x7fcd5c1e6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd59400640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd5c1606e0 0x7fcd5c1e6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd59400640 1 -- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c071d70 msgr2=0x7fcd5c156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd59e00640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c071d70 0x7fcd5c156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd59400640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c071d70 0x7fcd5c156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd59400640 1 -- 172.21.15.69:0/493109081 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd5c1e6a70 con 0x7fcd5c1420a0 2024-04-24T08:19:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd59e00640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c071d70 0x7fcd5c156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd59400640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1420a0 0x7fcd5c1e5df0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800c4d0 tx=0x7fcd4800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd52000640 1 -- 172.21.15.69:0/493109081 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48018070 con 0x7fcd5c1420a0 2024-04-24T08:19:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd52000640 1 -- 172.21.15.69:0/493109081 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd48007500 con 0x7fcd5c1420a0 2024-04-24T08:19:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd52000640 1 -- 172.21.15.69:0/493109081 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd48004a20 con 0x7fcd5c1420a0 2024-04-24T08:19:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd5b200640 1 -- 172.21.15.69:0/493109081 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd5c1e6d50 con 0x7fcd5c1420a0 2024-04-24T08:19:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.780+0000 7fcd5b200640 1 -- 172.21.15.69:0/493109081 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd5c154030 con 0x7fcd5c1420a0 2024-04-24T08:19:52.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.782+0000 7fcd5b200640 1 -- 172.21.15.69:0/493109081 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd5c153b60 con 0x7fcd5c1420a0 2024-04-24T08:19:52.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.782+0000 7fcd52000640 1 -- 172.21.15.69:0/493109081 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd48004bc0 con 0x7fcd5c1420a0 2024-04-24T08:19:52.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.783+0000 7fcd52000640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd3c06c8f0 0x7fcd3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:52.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.783+0000 7fcd52000640 1 -- 172.21.15.69:0/493109081 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7fcd4808ae40 con 0x7fcd5c1420a0 2024-04-24T08:19:52.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.784+0000 7fcd59e00640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd3c06c8f0 0x7fcd3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:52.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.785+0000 7fcd59e00640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd3c06c8f0 0x7fcd3c06edb0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c003eb0 tx=0x7fcd4c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:52.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.788+0000 7fcd52000640 1 -- 172.21.15.69:0/493109081 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd48059250 con 0x7fcd5c1420a0 2024-04-24T08:19:52.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.986+0000 7fcd5b200640 1 -- 172.21.15.69:0/493109081 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcd5c075d50 con 0x7fcd5c1420a0 2024-04-24T08:19:52.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.987+0000 7fcd52000640 1 -- 172.21.15.69:0/493109081 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) ==== 100+0+11 (secure 0 0 0) 0x7fcd48058b50 con 0x7fcd5c1420a0 2024-04-24T08:19:52.989 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 -- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd3c06c8f0 msgr2=0x7fcd3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd3c06c8f0 0x7fcd3c06edb0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c003eb0 tx=0x7fcd4c009040 comp rx=0 tx=0).stop 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 -- 172.21.15.69:0/493109081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1420a0 msgr2=0x7fcd5c1e5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1420a0 0x7fcd5c1e5df0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800c4d0 tx=0x7fcd4800c990 comp rx=0 tx=0).stop 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 -- 172.21.15.69:0/493109081 shutdown_connections 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcd3c06c8f0 0x7fcd3c06edb0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcd5c1420a0 0x7fcd5c1e5df0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcd5c1606e0 0x7fcd5c1e6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 --2- 172.21.15.69:0/493109081 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c071d70 0x7fcd5c156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 -- 172.21.15.69:0/493109081 >> 172.21.15.69:0/493109081 conn(0x7fcd5c067b80 msgr2=0x7fcd5c15c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 -- 172.21.15.69:0/493109081 shutdown_connections 2024-04-24T08:19:52.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:52.990+0000 7fcd5b200640 1 -- 172.21.15.69:0/493109081 wait complete. 2024-04-24T08:19:53.010 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:19:53.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.209+0000 7ff805000640 1 Processor -- start 2024-04-24T08:19:53.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.209+0000 7ff805000640 1 -- start start 2024-04-24T08:19:53.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.210+0000 7ff805000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff800071e50 0x7ff80015ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.210+0000 7ff805000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff800162300 0x7ff80006e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.210+0000 7ff805000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff800072230 0x7ff80006e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.210+0000 7ff805000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff80015f540 con 0x7ff800071e50 2024-04-24T08:19:53.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.210+0000 7ff805000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff80015f820 con 0x7ff800072230 2024-04-24T08:19:53.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.210+0000 7ff805000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff80015fb00 con 0x7ff800162300 2024-04-24T08:19:53.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7ff400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff800071e50 0x7ff80015ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7ff400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff800071e50 0x7ff80015ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40140/0 (socket says 172.21.15.69:40140) 2024-04-24T08:19:53.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7ff400640 1 -- 172.21.15.69:0/1550305717 learned_addr learned my addr 172.21.15.69:0/1550305717 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:53.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7ff400640 1 -- 172.21.15.69:0/1550305717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff800162300 msgr2=0x7ff80006e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7ffe00640 1 --2- 172.21.15.69:0/1550305717 >> v2:172.21.15.121:3300/0 conn(0x7ff800072230 0x7ff80006e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7f7e00640 1 --2- 172.21.15.69:0/1550305717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff800162300 0x7ff80006e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7ff400640 1 --2- 172.21.15.69:0/1550305717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff800162300 0x7ff80006e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7ff400640 1 -- 172.21.15.69:0/1550305717 >> v2:172.21.15.121:3300/0 conn(0x7ff800072230 msgr2=0x7ff80006e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7ff400640 1 --2- 172.21.15.69:0/1550305717 >> v2:172.21.15.121:3300/0 conn(0x7ff800072230 0x7ff80006e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7ff400640 1 -- 172.21.15.69:0/1550305717 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80006dda0 con 0x7ff800071e50 2024-04-24T08:19:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7ffe00640 1 --2- 172.21.15.69:0/1550305717 >> v2:172.21.15.121:3300/0 conn(0x7ff800072230 0x7ff80006e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.211+0000 7ff7f7e00640 1 --2- 172.21.15.69:0/1550305717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff800162300 0x7ff80006e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.212+0000 7ff7ff400640 1 --2- 172.21.15.69:0/1550305717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 0x7ff80015ecc0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7ff800141e10 tx=0x7ff7f002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59bc600a559a852e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.213+0000 7ff7fea00640 1 -- 172.21.15.69:0/1550305717 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0037070 con 0x7ff800071e50 2024-04-24T08:19:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.213+0000 7ff7fea00640 1 -- 172.21.15.69:0/1550305717 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f0032440 con 0x7ff800071e50 2024-04-24T08:19:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.213+0000 7ff7fea00640 1 -- 172.21.15.69:0/1550305717 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0032810 con 0x7ff800071e50 2024-04-24T08:19:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.213+0000 7ff805000640 1 -- 172.21.15.69:0/1550305717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 msgr2=0x7ff80015ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.213+0000 7ff805000640 1 --2- 172.21.15.69:0/1550305717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 0x7ff80015ecc0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7ff800141e10 tx=0x7ff7f002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.214+0000 7ff805000640 1 -- 172.21.15.69:0/1550305717 shutdown_connections 2024-04-24T08:19:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.214+0000 7ff805000640 1 --2- 172.21.15.69:0/1550305717 >> v2:172.21.15.121:3300/0 conn(0x7ff800072230 0x7ff80006e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.214+0000 7ff805000640 1 --2- 172.21.15.69:0/1550305717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff800162300 0x7ff80006e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.214+0000 7ff805000640 1 --2- 172.21.15.69:0/1550305717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 0x7ff80015ecc0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.214+0000 7ff805000640 1 -- 172.21.15.69:0/1550305717 >> 172.21.15.69:0/1550305717 conn(0x7ff8000d0160 msgr2=0x7ff8000d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.214+0000 7ff805000640 1 -- 172.21.15.69:0/1550305717 shutdown_connections 2024-04-24T08:19:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.214+0000 7ff805000640 1 -- 172.21.15.69:0/1550305717 wait complete. 2024-04-24T08:19:53.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.215+0000 7ff805000640 1 Processor -- start 2024-04-24T08:19:53.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.216+0000 7ff805000640 1 -- start start 2024-04-24T08:19:53.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.216+0000 7ff805000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 0x7ff800156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.216+0000 7ff805000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff800072230 0x7ff8000b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.216+0000 7ff805000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff800162300 0x7ff8000b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.216+0000 7ff805000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff80006eec0 con 0x7ff800071e50 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.216+0000 7ff805000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff80006ed40 con 0x7ff800162300 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.216+0000 7ff805000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff80006f040 con 0x7ff800072230 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.217+0000 7ff7ff400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 0x7ff800156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.217+0000 7ff7ff400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 0x7ff800156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40144/0 (socket says 172.21.15.69:40144) 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.217+0000 7ff7ff400640 1 -- 172.21.15.69:0/4234231762 learned_addr learned my addr 172.21.15.69:0/4234231762 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.217+0000 7ff7ff400640 1 -- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff800072230 msgr2=0x7ff8000b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.217+0000 7ff7ff400640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff800072230 0x7ff8000b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.217+0000 7ff7ff400640 1 -- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff800162300 msgr2=0x7ff8000b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.217+0000 7ff7ff400640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff800162300 0x7ff8000b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.217+0000 7ff7ff400640 1 -- 172.21.15.69:0/4234231762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff800158700 con 0x7ff800071e50 2024-04-24T08:19:53.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.217+0000 7ff7ff400640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 0x7ff800156d60 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0000c00 tx=0x7ff7f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.218+0000 7ff7f7400640 1 -- 172.21.15.69:0/4234231762 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0041020 con 0x7ff800071e50 2024-04-24T08:19:53.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.218+0000 7ff7f7400640 1 -- 172.21.15.69:0/4234231762 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7f0030800 con 0x7ff800071e50 2024-04-24T08:19:53.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.218+0000 7ff7f7400640 1 -- 172.21.15.69:0/4234231762 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0037040 con 0x7ff800071e50 2024-04-24T08:19:53.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.218+0000 7ff805000640 1 -- 172.21.15.69:0/4234231762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8001589e0 con 0x7ff800071e50 2024-04-24T08:19:53.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.218+0000 7ff805000640 1 -- 172.21.15.69:0/4234231762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff800158fa0 con 0x7ff800071e50 2024-04-24T08:19:53.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.219+0000 7ff805000640 1 -- 172.21.15.69:0/4234231762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff800075d50 con 0x7ff800071e50 2024-04-24T08:19:53.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.220+0000 7ff7f7400640 1 -- 172.21.15.69:0/4234231762 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff7f0044020 con 0x7ff800071e50 2024-04-24T08:19:53.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.221+0000 7ff7f7400640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7dc06c8f0 0x7ff7dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.221+0000 7ff7f7400640 1 -- 172.21.15.69:0/4234231762 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7ff7f00ac580 con 0x7ff800071e50 2024-04-24T08:19:53.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.221+0000 7ff7f7e00640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7dc06c8f0 0x7ff7dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.223+0000 7ff7f7e00640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7dc06c8f0 0x7ff7dc06edb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8006b10 tx=0x7ff7e8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.225+0000 7ff7f7400640 1 -- 172.21.15.69:0/4234231762 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff7f004b030 con 0x7ff800071e50 2024-04-24T08:19:53.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.459+0000 7ff805000640 1 -- 172.21.15.69:0/4234231762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8000d0d90 con 0x7ff800071e50 2024-04-24T08:19:53.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.460+0000 7ff7f7400640 1 -- 172.21.15.69:0/4234231762 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5175 (secure 0 0 0) 0x7ff7f007a860 con 0x7ff800071e50 2024-04-24T08:19:53.463 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:53.463 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9111,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3686931568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3686931568},{"type":"v1","addr":"172.21.15.69:6835","nonce":3686931568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:19:49.975280+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-24T08:19:53.463 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 91 2024-04-24T08:19:53.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 -- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7dc06c8f0 msgr2=0x7ff7dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7dc06c8f0 0x7ff7dc06edb0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8006b10 tx=0x7ff7e8008040 comp rx=0 tx=0).stop 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 -- 172.21.15.69:0/4234231762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 msgr2=0x7ff800156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 0x7ff800156d60 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0000c00 tx=0x7ff7f00046c0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 -- 172.21.15.69:0/4234231762 shutdown_connections 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7dc06c8f0 0x7ff7dc06edb0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff800162300 0x7ff8000b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff800072230 0x7ff8000b0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 --2- 172.21.15.69:0/4234231762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff800071e50 0x7ff800156d60 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 -- 172.21.15.69:0/4234231762 >> 172.21.15.69:0/4234231762 conn(0x7ff8000d0160 msgr2=0x7ff80015ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 -- 172.21.15.69:0/4234231762 shutdown_connections 2024-04-24T08:19:53.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.464+0000 7ff805000640 1 -- 172.21.15.69:0/4234231762 wait complete. 2024-04-24T08:19:53.485 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:19:53.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.679+0000 7f11cbc00640 1 Processor -- start 2024-04-24T08:19:53.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.680+0000 7f11cbc00640 1 -- start start 2024-04-24T08:19:53.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.681+0000 7f11cbc00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f11c406c510 0x7f11c415cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.681+0000 7f11cbc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c40cd0e0 0x7f11c40d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.681+0000 7f11cbc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11c40b3c70 con 0x7f11c406ba20 2024-04-24T08:19:53.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.681+0000 7f11cbc00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f11c4161200 con 0x7f11c406c510 2024-04-24T08:19:53.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.681+0000 7f11cbc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f11c4160310 con 0x7f11c40cd0e0 2024-04-24T08:19:53.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.682+0000 7f11c3e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f11c406c510 0x7f11c415cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.682+0000 7f11c9c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c40cd0e0 0x7f11c40d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.682+0000 7f11c9c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c40cd0e0 0x7f11c40d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:51022/0 (socket says 172.21.15.69:51022) 2024-04-24T08:19:53.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.682+0000 7f11c9c00640 1 -- 172.21.15.69:0/839905921 learned_addr learned my addr 172.21.15.69:0/839905921 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:53.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.683+0000 7f11c9c00640 1 -- 172.21.15.69:0/839905921 >> v2:172.21.15.121:3300/0 conn(0x7f11c406c510 msgr2=0x7f11c415cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.683+0000 7f11c9c00640 1 --2- 172.21.15.69:0/839905921 >> v2:172.21.15.121:3300/0 conn(0x7f11c406c510 0x7f11c415cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.683+0000 7f11c9c00640 1 -- 172.21.15.69:0/839905921 >> v1:172.21.15.121:6789/0 conn(0x7f11c406ba20 legacy=0x7f11c406be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.683+0000 7f11c9c00640 1 -- 172.21.15.69:0/839905921 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c41605f0 con 0x7f11c40cd0e0 2024-04-24T08:19:53.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.683+0000 7f11c3e00640 1 --2- 172.21.15.69:0/839905921 >> v2:172.21.15.121:3300/0 conn(0x7f11c406c510 0x7f11c415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:53.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.684+0000 7f11c9c00640 1 --2- 172.21.15.69:0/839905921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c40cd0e0 0x7f11c40d0470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009870 tx=0x7f11bc02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee0d38c313e1f7af server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.684+0000 7f11c3400640 1 -- 172.21.15.69:0/839905921 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc037070 con 0x7f11c40cd0e0 2024-04-24T08:19:53.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.685+0000 7f11c3400640 1 -- 172.21.15.69:0/839905921 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11bc032440 con 0x7f11c40cd0e0 2024-04-24T08:19:53.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.685+0000 7f11c3400640 1 -- 172.21.15.69:0/839905921 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11bc032810 con 0x7f11c40cd0e0 2024-04-24T08:19:53.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.685+0000 7f11cbc00640 1 -- 172.21.15.69:0/839905921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c40cd0e0 msgr2=0x7f11c40d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.685+0000 7f11cbc00640 1 --2- 172.21.15.69:0/839905921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c40cd0e0 0x7f11c40d0470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009870 tx=0x7f11bc02a250 comp rx=0 tx=0).stop 2024-04-24T08:19:53.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.685+0000 7f11cbc00640 1 -- 172.21.15.69:0/839905921 shutdown_connections 2024-04-24T08:19:53.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.685+0000 7f11cbc00640 1 --2- 172.21.15.69:0/839905921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c40cd0e0 0x7f11c40d0470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.685+0000 7f11cbc00640 1 --2- 172.21.15.69:0/839905921 >> v2:172.21.15.121:3300/0 conn(0x7f11c406c510 0x7f11c415cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.685+0000 7f11cbc00640 1 -- 172.21.15.69:0/839905921 >> 172.21.15.69:0/839905921 conn(0x7f11c40b2f50 msgr2=0x7f11c40b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.686+0000 7f11cbc00640 1 -- 172.21.15.69:0/839905921 shutdown_connections 2024-04-24T08:19:53.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.686+0000 7f11cbc00640 1 -- 172.21.15.69:0/839905921 wait complete. 2024-04-24T08:19:53.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.688+0000 7f11cbc00640 1 Processor -- start 2024-04-24T08:19:53.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.688+0000 7f11cbc00640 1 -- start start 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.689+0000 7f11cbc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c406ba20 0x7f11c40afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.689+0000 7f11cbc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c406c510 0x7f11c40b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.689+0000 7f11cbc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11c40cd0e0 0x7f11c40b1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.689+0000 7f11cbc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f11c4160e90 con 0x7f11c406c510 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.689+0000 7f11cbc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f11c4160bb0 con 0x7f11c40cd0e0 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.689+0000 7f11cbc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f11c40cd590 con 0x7f11c406ba20 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.689+0000 7f11c3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c406c510 0x7f11c40b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.689+0000 7f11c3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c406c510 0x7f11c40b65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40152/0 (socket says 172.21.15.69:40152) 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.689+0000 7f11c3e00640 1 -- 172.21.15.69:0/537457663 learned_addr learned my addr 172.21.15.69:0/537457663 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:53.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.690+0000 7f11c9200640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c406ba20 0x7f11c40afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.690+0000 7f11c3e00640 1 -- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c406ba20 msgr2=0x7f11c40afc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.690+0000 7f11c3e00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c406ba20 0x7f11c40afc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.690+0000 7f11c3e00640 1 -- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11c40cd0e0 msgr2=0x7f11c40b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.690+0000 7f11c9c00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11c40cd0e0 0x7f11c40b1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.690+0000 7f11c3e00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11c40cd0e0 0x7f11c40b1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.690+0000 7f11c3e00640 1 -- 172.21.15.69:0/537457663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c40b0180 con 0x7f11c406c510 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.690+0000 7f11c3e00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c406c510 0x7f11c40b65a0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f11b400efc0 tx=0x7f11b4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.691+0000 7f11c0c00640 1 -- 172.21.15.69:0/537457663 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b4018020 con 0x7f11c406c510 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.691+0000 7f11c0c00640 1 -- 172.21.15.69:0/537457663 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b4014480 con 0x7f11c406c510 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.691+0000 7f11c0c00640 1 -- 172.21.15.69:0/537457663 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b4014850 con 0x7f11c406c510 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.691+0000 7f11cbc00640 1 -- 172.21.15.69:0/537457663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11c40c5780 con 0x7f11c406c510 2024-04-24T08:19:53.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.691+0000 7f11cbc00640 1 -- 172.21.15.69:0/537457663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11c40c5d40 con 0x7f11c406c510 2024-04-24T08:19:53.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.692+0000 7f11cbc00640 1 -- 172.21.15.69:0/537457663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c4069190 con 0x7f11c406c510 2024-04-24T08:19:53.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.693+0000 7f11c0c00640 1 -- 172.21.15.69:0/537457663 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f11b4005d10 con 0x7f11c406c510 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.694+0000 7f11c0c00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11a006c8f0 0x7f11a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.694+0000 7f11c0c00640 1 -- 172.21.15.69:0/537457663 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7f11b408ca10 con 0x7f11c406c510 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.695+0000 7f11c9200640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11a006c8f0 0x7f11a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.696+0000 7f11c9200640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11a006c8f0 0x7f11a006edb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f11ac007fd0 tx=0x7f11ac009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.698+0000 7f11c0c00640 1 -- 172.21.15.69:0/537457663 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f11b4022030 con 0x7f11c406c510 2024-04-24T08:19:53.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.889+0000 7f11cbc00640 1 -- 172.21.15.69:0/537457663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f11c40b73c0 con 0x7f11c406c510 2024-04-24T08:19:53.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.890+0000 7f11c0c00640 1 -- 172.21.15.69:0/537457663 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+16841 (secure 0 0 0) 0x7f11b4010040 con 0x7f11c406c510 2024-04-24T08:19:53.892 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:53.892 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":99,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:19:49.975068+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:08.156864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:09.368519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/3642138906":"2024-04-25T08:19:49.975039+0000","172.21.15.69:6833/3642138906":"2024-04-25T08:19:49.975039+0000","172.21.15.69:6835/412016900":"2024-04-25T08:19:12.588993+0000","172.21.15.69:6833/412016900":"2024-04-25T08:19:12.588993+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:19:53.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.892+0000 7f11cbc00640 1 -- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11a006c8f0 msgr2=0x7f11a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.892+0000 7f11cbc00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11a006c8f0 0x7f11a006edb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f11ac007fd0 tx=0x7f11ac009040 comp rx=0 tx=0).stop 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.893+0000 7f11cbc00640 1 -- 172.21.15.69:0/537457663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c406c510 msgr2=0x7f11c40b65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.893+0000 7f11cbc00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c406c510 0x7f11c40b65a0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f11b400efc0 tx=0x7f11b4009f90 comp rx=0 tx=0).stop 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.893+0000 7f11cbc00640 1 -- 172.21.15.69:0/537457663 shutdown_connections 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.893+0000 7f11cbc00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f11a006c8f0 0x7f11a006edb0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.893+0000 7f11cbc00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f11c406c510 0x7f11c40b65a0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.893+0000 7f11cbc00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f11c40cd0e0 0x7f11c40b1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.893+0000 7f11cbc00640 1 --2- 172.21.15.69:0/537457663 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f11c406ba20 0x7f11c40afc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.893+0000 7f11cbc00640 1 -- 172.21.15.69:0/537457663 >> 172.21.15.69:0/537457663 conn(0x7f11c40b2f50 msgr2=0x7f11c40cde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.893+0000 7f11cbc00640 1 -- 172.21.15.69:0/537457663 shutdown_connections 2024-04-24T08:19:53.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:53.893+0000 7f11cbc00640 1 -- 172.21.15.69:0/537457663 wait complete. 2024-04-24T08:19:53.915 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:19:53.915 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:19:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.098+0000 7f626a400640 1 Processor -- start 2024-04-24T08:19:54.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.098+0000 7f626a400640 1 -- start start 2024-04-24T08:19:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.100+0000 7f626a400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6264156c60 0x7f626415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.100+0000 7f626a400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f62640b30f0 0x7f626415d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.100+0000 7f626a400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62640ad0d0 0x7f62640ad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.100+0000 7f626a400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f626415d660 con 0x7f6264156c60 2024-04-24T08:19:54.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.100+0000 7f626a400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f626406dc30 con 0x7f62640b30f0 2024-04-24T08:19:54.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.100+0000 7f626a400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f626406df10 con 0x7f62640ad0d0 2024-04-24T08:19:54.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.100+0000 7f6269000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6264156c60 0x7f626415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.100+0000 7f6269000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f6264156c60 0x7f626415cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40154/0 (socket says 172.21.15.69:40154) 2024-04-24T08:19:54.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.100+0000 7f6269000640 1 -- 172.21.15.69:0/2516518059 learned_addr learned my addr 172.21.15.69:0/2516518059 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.100+0000 7f6263e00640 1 --2- 172.21.15.69:0/2516518059 >> v2:172.21.15.121:3300/0 conn(0x7f62640b30f0 0x7f626415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.101+0000 7f6269a00640 1 --2- 172.21.15.69:0/2516518059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62640ad0d0 0x7f62640ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.101+0000 7f6269000640 1 -- 172.21.15.69:0/2516518059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62640ad0d0 msgr2=0x7f62640ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.101+0000 7f6269000640 1 --2- 172.21.15.69:0/2516518059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62640ad0d0 0x7f62640ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.101+0000 7f6269000640 1 -- 172.21.15.69:0/2516518059 >> v2:172.21.15.121:3300/0 conn(0x7f62640b30f0 msgr2=0x7f626415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.101+0000 7f6269000640 1 --2- 172.21.15.69:0/2516518059 >> v2:172.21.15.121:3300/0 conn(0x7f62640b30f0 0x7f626415d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.101+0000 7f6269000640 1 -- 172.21.15.69:0/2516518059 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626406e1f0 con 0x7f6264156c60 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.102+0000 7f6269000640 1 --2- 172.21.15.69:0/2516518059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6264156c60 0x7f626415cac0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f626406d900 tx=0x7f625802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2417c1c41705f6fa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.102+0000 7f6263400640 1 -- 172.21.15.69:0/2516518059 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258037070 con 0x7f6264156c60 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f6263400640 1 -- 172.21.15.69:0/2516518059 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6258032440 con 0x7f6264156c60 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f6263400640 1 -- 172.21.15.69:0/2516518059 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258032810 con 0x7f6264156c60 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f626a400640 1 -- 172.21.15.69:0/2516518059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6264156c60 msgr2=0x7f626415cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f626a400640 1 --2- 172.21.15.69:0/2516518059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6264156c60 0x7f626415cac0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f626406d900 tx=0x7f625802a1e0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f626a400640 1 -- 172.21.15.69:0/2516518059 shutdown_connections 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f626a400640 1 --2- 172.21.15.69:0/2516518059 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62640ad0d0 0x7f62640ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f626a400640 1 --2- 172.21.15.69:0/2516518059 >> v2:172.21.15.121:3300/0 conn(0x7f62640b30f0 0x7f626415d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f626a400640 1 --2- 172.21.15.69:0/2516518059 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6264156c60 0x7f626415cac0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f626a400640 1 -- 172.21.15.69:0/2516518059 >> 172.21.15.69:0/2516518059 conn(0x7f62640d0160 msgr2=0x7f62640d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f626a400640 1 -- 172.21.15.69:0/2516518059 shutdown_connections 2024-04-24T08:19:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.103+0000 7f626a400640 1 -- 172.21.15.69:0/2516518059 wait complete. 2024-04-24T08:19:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.105+0000 7f626a400640 1 Processor -- start 2024-04-24T08:19:54.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.105+0000 7f626a400640 1 -- start start 2024-04-24T08:19:54.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.106+0000 7f626a400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62640ad0d0 0x7f62641f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.106+0000 7f626a400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62640b30f0 0x7f6264155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.106+0000 7f626a400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264156c60 0x7f6264143490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.106+0000 7f626a400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62641600e0 con 0x7f62640ad0d0 2024-04-24T08:19:54.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.106+0000 7f626a400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f626415d940 con 0x7f6264156c60 2024-04-24T08:19:54.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.106+0000 7f626a400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6264160260 con 0x7f62640b30f0 2024-04-24T08:19:54.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.106+0000 7f6269000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62640ad0d0 0x7f62641f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.106+0000 7f6269000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62640ad0d0 0x7f62641f42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40160/0 (socket says 172.21.15.69:40160) 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.106+0000 7f6269000640 1 -- 172.21.15.69:0/2552046635 learned_addr learned my addr 172.21.15.69:0/2552046635 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6269000640 1 -- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62640b30f0 msgr2=0x7f6264155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6263e00640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62640b30f0 0x7f6264155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6269a00640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264156c60 0x7f6264143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6269000640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62640b30f0 0x7f6264155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6269000640 1 -- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264156c60 msgr2=0x7f6264143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6269000640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264156c60 0x7f6264143490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6269000640 1 -- 172.21.15.69:0/2552046635 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62640c9e50 con 0x7f62640ad0d0 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6269000640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62640ad0d0 0x7f62641f42f0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f625802a220 tx=0x7f62580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6260c00640 1 -- 172.21.15.69:0/2552046635 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258041020 con 0x7f62640ad0d0 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6260c00640 1 -- 172.21.15.69:0/2552046635 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62580306c0 con 0x7f62640ad0d0 2024-04-24T08:19:54.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f6260c00640 1 -- 172.21.15.69:0/2552046635 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6258037040 con 0x7f62640ad0d0 2024-04-24T08:19:54.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.107+0000 7f626a400640 1 -- 172.21.15.69:0/2552046635 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626415a9d0 con 0x7f62640ad0d0 2024-04-24T08:19:54.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.108+0000 7f626a400640 1 -- 172.21.15.69:0/2552046635 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62640ca3b0 con 0x7f62640ad0d0 2024-04-24T08:19:54.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.109+0000 7f626a400640 1 -- 172.21.15.69:0/2552046635 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6264075d50 con 0x7f62640ad0d0 2024-04-24T08:19:54.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.110+0000 7f6260c00640 1 -- 172.21.15.69:0/2552046635 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6258044020 con 0x7f62640ad0d0 2024-04-24T08:19:54.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.111+0000 7f6260c00640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f624406c8f0 0x7f624406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.111+0000 7f6260c00640 1 -- 172.21.15.69:0/2552046635 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7f62580ac3c0 con 0x7f62640ad0d0 2024-04-24T08:19:54.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.111+0000 7f6263e00640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f624406c8f0 0x7f624406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.112+0000 7f6263e00640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f624406c8f0 0x7f624406edb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f62640b01b0 tx=0x7f6250008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.115+0000 7f6260c00640 1 -- 172.21.15.69:0/2552046635 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f625804b030 con 0x7f62640ad0d0 2024-04-24T08:19:54.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.350+0000 7f626a400640 1 -- 172.21.15.69:0/2552046635 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f626406f9e0 con 0x7f62640ad0d0 2024-04-24T08:19:54.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.352+0000 7f6260c00640 1 -- 172.21.15.69:0/2552046635 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5175 (secure 0 0 0) 0x7f62640c9e50 con 0x7f62640ad0d0 2024-04-24T08:19:54.354 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:54.354 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9111,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3686931568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3686931568},{"type":"v1","addr":"172.21.15.69:6835","nonce":3686931568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:19:49.975280+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-24T08:19:54.354 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 91 2024-04-24T08:19:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.354+0000 7f626a400640 1 -- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f624406c8f0 msgr2=0x7f624406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.354+0000 7f626a400640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f624406c8f0 0x7f624406edb0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f62640b01b0 tx=0x7f6250008040 comp rx=0 tx=0).stop 2024-04-24T08:19:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.354+0000 7f626a400640 1 -- 172.21.15.69:0/2552046635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62640ad0d0 msgr2=0x7f62641f42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.354+0000 7f626a400640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62640ad0d0 0x7f62641f42f0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f625802a220 tx=0x7f62580046c0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.354+0000 7f626a400640 1 -- 172.21.15.69:0/2552046635 shutdown_connections 2024-04-24T08:19:54.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.354+0000 7f626a400640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f624406c8f0 0x7f624406edb0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.354+0000 7f626a400640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6264156c60 0x7f6264143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.354+0000 7f626a400640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62640b30f0 0x7f6264155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.354+0000 7f626a400640 1 --2- 172.21.15.69:0/2552046635 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62640ad0d0 0x7f62641f42f0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.354+0000 7f626a400640 1 -- 172.21.15.69:0/2552046635 >> 172.21.15.69:0/2552046635 conn(0x7f62640d0160 msgr2=0x7f6264160560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.355+0000 7f626a400640 1 -- 172.21.15.69:0/2552046635 shutdown_connections 2024-04-24T08:19:54.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.355+0000 7f626a400640 1 -- 172.21.15.69:0/2552046635 wait complete. 2024-04-24T08:19:54.374 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:19:54.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.558+0000 7fd0ee600640 1 Processor -- start 2024-04-24T08:19:54.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.559+0000 7fd0ee600640 1 -- start start 2024-04-24T08:19:54.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.560+0000 7fd0ee600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0e80d0300 0x7fd0e806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.560+0000 7fd0ee600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd0e8145fe0 0x7fd0e8160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.560+0000 7fd0ee600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0e80cf3d0 con 0x7fd0e8145a90 2024-04-24T08:19:54.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.560+0000 7fd0ee600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd0e80cf6b0 con 0x7fd0e80d0300 2024-04-24T08:19:54.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.560+0000 7fd0ee600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd0e80cf990 con 0x7fd0e8145fe0 2024-04-24T08:19:54.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.560+0000 7fd0e7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0e80d0300 0x7fd0e806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.560+0000 7fd0e7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0e80d0300 0x7fd0e806e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40172/0 (socket says 172.21.15.69:40172) 2024-04-24T08:19:54.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.560+0000 7fd0e7400640 1 -- 172.21.15.69:0/2606360346 learned_addr learned my addr 172.21.15.69:0/2606360346 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:54.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.561+0000 7fd0e7e00640 1 --2- 172.21.15.69:0/2606360346 >> v2:172.21.15.121:3300/0 conn(0x7fd0e8145fe0 0x7fd0e8160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.561+0000 7fd0e7400640 1 -- 172.21.15.69:0/2606360346 >> v2:172.21.15.121:3300/0 conn(0x7fd0e8145fe0 msgr2=0x7fd0e8160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.561+0000 7fd0e7400640 1 --2- 172.21.15.69:0/2606360346 >> v2:172.21.15.121:3300/0 conn(0x7fd0e8145fe0 0x7fd0e8160540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.561+0000 7fd0e7400640 1 -- 172.21.15.69:0/2606360346 >> v1:172.21.15.121:6789/0 conn(0x7fd0e8145a90 legacy=0x7fd0e80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.561+0000 7fd0e7400640 1 -- 172.21.15.69:0/2606360346 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e80cfc70 con 0x7fd0e80d0300 2024-04-24T08:19:54.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.562+0000 7fd0e7400640 1 --2- 172.21.15.69:0/2606360346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e80d0300 0x7fd0e806e240 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fd0e80d1050 tx=0x7fd0d802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10e35ab0b920db7e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.562+0000 7fd0e6a00640 1 -- 172.21.15.69:0/2606360346 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8036070 con 0x7fd0e80d0300 2024-04-24T08:19:54.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.562+0000 7fd0e6a00640 1 -- 172.21.15.69:0/2606360346 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d8031600 con 0x7fd0e80d0300 2024-04-24T08:19:54.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.562+0000 7fd0e6a00640 1 -- 172.21.15.69:0/2606360346 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d80319d0 con 0x7fd0e80d0300 2024-04-24T08:19:54.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.563+0000 7fd0ee600640 1 -- 172.21.15.69:0/2606360346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e80d0300 msgr2=0x7fd0e806e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.563+0000 7fd0ee600640 1 --2- 172.21.15.69:0/2606360346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e80d0300 0x7fd0e806e240 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fd0e80d1050 tx=0x7fd0d802a6f0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.563+0000 7fd0ee600640 1 -- 172.21.15.69:0/2606360346 shutdown_connections 2024-04-24T08:19:54.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.563+0000 7fd0ee600640 1 --2- 172.21.15.69:0/2606360346 >> v2:172.21.15.121:3300/0 conn(0x7fd0e8145fe0 0x7fd0e8160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.563+0000 7fd0ee600640 1 --2- 172.21.15.69:0/2606360346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e80d0300 0x7fd0e806e240 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.563+0000 7fd0ee600640 1 -- 172.21.15.69:0/2606360346 >> 172.21.15.69:0/2606360346 conn(0x7fd0e8067d90 msgr2=0x7fd0e806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.563+0000 7fd0ee600640 1 -- 172.21.15.69:0/2606360346 shutdown_connections 2024-04-24T08:19:54.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.563+0000 7fd0ee600640 1 -- 172.21.15.69:0/2606360346 wait complete. 2024-04-24T08:19:54.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.564+0000 7fd0ee600640 1 Processor -- start 2024-04-24T08:19:54.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.565+0000 7fd0ee600640 1 -- start start 2024-04-24T08:19:54.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.565+0000 7fd0ee600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e80d0300 0x7fd0e81438d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.565+0000 7fd0ee600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e8145a90 0x7fd0e80c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.565+0000 7fd0ee600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e8145fe0 0x7fd0e80ca270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.565+0000 7fd0ee600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e81458f0 con 0x7fd0e8145a90 2024-04-24T08:19:54.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.565+0000 7fd0ee600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e8145770 con 0x7fd0e8145fe0 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.565+0000 7fd0ee600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e80cdc30 con 0x7fd0e80d0300 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0dea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e8145a90 0x7fd0e80c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0dea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e8145a90 0x7fd0e80c6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40184/0 (socket says 172.21.15.69:40184) 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0dea00640 1 -- 172.21.15.69:0/3616426330 learned_addr learned my addr 172.21.15.69:0/3616426330 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0e7e00640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e8145fe0 0x7fd0e80ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0e7400640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e80d0300 0x7fd0e81438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0dea00640 1 -- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e80d0300 msgr2=0x7fd0e81438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0dea00640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e80d0300 0x7fd0e81438d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0dea00640 1 -- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e8145fe0 msgr2=0x7fd0e80ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0dea00640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e8145fe0 0x7fd0e80ca270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0dea00640 1 -- 172.21.15.69:0/3616426330 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e80b35c0 con 0x7fd0e8145a90 2024-04-24T08:19:54.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.566+0000 7fd0dea00640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e8145a90 0x7fd0e80c6c20 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000b3e0 tx=0x7fd0d000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.567+0000 7fd0dfe00640 1 -- 172.21.15.69:0/3616426330 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0015020 con 0x7fd0e8145a90 2024-04-24T08:19:54.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.567+0000 7fd0dfe00640 1 -- 172.21.15.69:0/3616426330 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d0011850 con 0x7fd0e8145a90 2024-04-24T08:19:54.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.567+0000 7fd0dfe00640 1 -- 172.21.15.69:0/3616426330 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0011c40 con 0x7fd0e8145a90 2024-04-24T08:19:54.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.567+0000 7fd0ee600640 1 -- 172.21.15.69:0/3616426330 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e80b38a0 con 0x7fd0e8145a90 2024-04-24T08:19:54.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.567+0000 7fd0ee600640 1 -- 172.21.15.69:0/3616426330 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e80b3e60 con 0x7fd0e8145a90 2024-04-24T08:19:54.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.568+0000 7fd0ee600640 1 -- 172.21.15.69:0/3616426330 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0e80afcb0 con 0x7fd0e8145a90 2024-04-24T08:19:54.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.569+0000 7fd0dfe00640 1 -- 172.21.15.69:0/3616426330 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd0d0018020 con 0x7fd0e8145a90 2024-04-24T08:19:54.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.570+0000 7fd0dfe00640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0c406c8f0 0x7fd0c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.570+0000 7fd0dfe00640 1 -- 172.21.15.69:0/3616426330 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7fd0d008bc60 con 0x7fd0e8145a90 2024-04-24T08:19:54.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.570+0000 7fd0e7400640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0c406c8f0 0x7fd0c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.573 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.571+0000 7fd0e7400640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0c406c8f0 0x7fd0c406edb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8033000 tx=0x7fd0d8009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.574+0000 7fd0dfe00640 1 -- 172.21.15.69:0/3616426330 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0e80afcb0 con 0x7fd0e8145a90 2024-04-24T08:19:54.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.817+0000 7fd0ee600640 1 -- 172.21.15.69:0/3616426330 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0e8068a80 con 0x7fd0e8145a90 2024-04-24T08:19:54.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.818+0000 7fd0dfe00640 1 -- 172.21.15.69:0/3616426330 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5175 (secure 0 0 0) 0x7fd0d001f020 con 0x7fd0e8145a90 2024-04-24T08:19:54.821 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:54.821 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":91,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9111,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3686931568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3686931568},{"type":"v1","addr":"172.21.15.69:6835","nonce":3686931568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:18:10.396179+0000","modified":"2024-04-24T08:19:49.975280+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":7}]} 2024-04-24T08:19:54.821 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 91 2024-04-24T08:19:54.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.820+0000 7fd0ee600640 1 -- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0c406c8f0 msgr2=0x7fd0c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.820+0000 7fd0ee600640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0c406c8f0 0x7fd0c406edb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8033000 tx=0x7fd0d8009000 comp rx=0 tx=0).stop 2024-04-24T08:19:54.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.821+0000 7fd0ee600640 1 -- 172.21.15.69:0/3616426330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e8145a90 msgr2=0x7fd0e80c6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.821+0000 7fd0ee600640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e8145a90 0x7fd0e80c6c20 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000b3e0 tx=0x7fd0d000b8a0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.821+0000 7fd0ee600640 1 -- 172.21.15.69:0/3616426330 shutdown_connections 2024-04-24T08:19:54.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.821+0000 7fd0ee600640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0c406c8f0 0x7fd0c406edb0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.821+0000 7fd0ee600640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0e8145a90 0x7fd0e80c6c20 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.821+0000 7fd0ee600640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0e8145fe0 0x7fd0e80ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.821+0000 7fd0ee600640 1 --2- 172.21.15.69:0/3616426330 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0e80d0300 0x7fd0e81438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.821+0000 7fd0ee600640 1 -- 172.21.15.69:0/3616426330 >> 172.21.15.69:0/3616426330 conn(0x7fd0e8067d90 msgr2=0x7fd0e806f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.821+0000 7fd0ee600640 1 -- 172.21.15.69:0/3616426330 shutdown_connections 2024-04-24T08:19:54.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:54.821+0000 7fd0ee600640 1 -- 172.21.15.69:0/3616426330 wait complete. 2024-04-24T08:19:54.840 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:19:55.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.041+0000 7f011d600640 1 Processor -- start 2024-04-24T08:19:55.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.041+0000 7f011d600640 1 -- start start 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.042+0000 7f011d600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f011815e960 0x7f0118162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.043+0000 7f011d600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f011815ed40 con 0x7f01180cfaf0 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.043+0000 7f011d600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01180cf050 con 0x7f0118162980 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.043+0000 7f011d600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0118162e20 con 0x7f011815e960 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.043+0000 7f0117e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f011815e960 0x7f0118162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.043+0000 7f011cc00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f01180cfaf0 0x7f01180d2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33668/0 (socket says 172.21.15.69:33668) 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.043+0000 7f011cc00640 1 -- 172.21.15.69:0/794411204 learned_addr learned my addr 172.21.15.69:0/794411204 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.043+0000 7f0117e00640 1 -- 172.21.15.69:0/794411204 >> v1:172.21.15.121:6789/0 conn(0x7f0118162980 legacy=0x7f011806cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.043+0000 7f0117e00640 1 -- 172.21.15.69:0/794411204 >> v1:172.21.15.69:6789/0 conn(0x7f01180cfaf0 legacy=0x7f01180d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.043+0000 7f0117e00640 1 -- 172.21.15.69:0/794411204 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0118163100 con 0x7f011815e960 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.043+0000 7f0116a00640 1 -- 172.21.15.69:0/794411204 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2138923287 0 0) 0x7f011815ed40 con 0x7f01180cfaf0 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.044+0000 7f0117e00640 1 --2- 172.21.15.69:0/794411204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f011815e960 0x7f0118162440 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f011815ef10 tx=0x7f010802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3f470335a265b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.045+0000 7f0116a00640 1 -- 172.21.15.69:0/794411204 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0108037070 con 0x7f011815e960 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.045+0000 7f0116a00640 1 -- 172.21.15.69:0/794411204 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0108032440 con 0x7f011815e960 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.045+0000 7f0116a00640 1 -- 172.21.15.69:0/794411204 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0108032810 con 0x7f011815e960 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.045+0000 7f011d600640 1 -- 172.21.15.69:0/794411204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f011815e960 msgr2=0x7f0118162440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.045+0000 7f011d600640 1 --2- 172.21.15.69:0/794411204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f011815e960 0x7f0118162440 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f011815ef10 tx=0x7f010802a1c0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.045+0000 7f011d600640 1 -- 172.21.15.69:0/794411204 shutdown_connections 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.045+0000 7f011d600640 1 --2- 172.21.15.69:0/794411204 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f011815e960 0x7f0118162440 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.045+0000 7f011d600640 1 -- 172.21.15.69:0/794411204 >> 172.21.15.69:0/794411204 conn(0x7f011809d900 msgr2=0x7f011808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.046+0000 7f011d600640 1 -- 172.21.15.69:0/794411204 shutdown_connections 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.046+0000 7f011d600640 1 -- 172.21.15.69:0/794411204 wait complete. 2024-04-24T08:19:55.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.047+0000 7f011d600640 1 Processor -- start 2024-04-24T08:19:55.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.047+0000 7f011d600640 1 -- start start 2024-04-24T08:19:55.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.048+0000 7f011d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01180cfaf0 0x7f01180ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.048+0000 7f011d600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f011815e960 0x7f011806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.048+0000 7f011d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0118162980 0x7f011806e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.048+0000 7f011d600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f011806d6c0 con 0x7f0118162980 2024-04-24T08:19:55.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.048+0000 7f011d600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f01180cf5d0 con 0x7f011815e960 2024-04-24T08:19:55.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.048+0000 7f011d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f01180b6170 con 0x7f01180cfaf0 2024-04-24T08:19:55.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f011cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0118162980 0x7f011806e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f011cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0118162980 0x7f011806e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40196/0 (socket says 172.21.15.69:40196) 2024-04-24T08:19:55.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f011cc00640 1 -- 172.21.15.69:0/472730601 learned_addr learned my addr 172.21.15.69:0/472730601 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:55.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f0117e00640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01180cfaf0 0x7f01180ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f0117400640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f011815e960 0x7f011806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f011cc00640 1 -- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01180cfaf0 msgr2=0x7f01180ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f011cc00640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01180cfaf0 0x7f01180ce0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f011cc00640 1 -- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f011815e960 msgr2=0x7f011806dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f011cc00640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f011815e960 0x7f011806dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f011cc00640 1 -- 172.21.15.69:0/472730601 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01180cd000 con 0x7f0118162980 2024-04-24T08:19:55.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f0117400640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f011815e960 0x7f011806dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:55.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.049+0000 7f011cc00640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0118162980 0x7f011806e210 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f011000c730 tx=0x7f011000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.050+0000 7f010fe00640 1 -- 172.21.15.69:0/472730601 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0110018070 con 0x7f0118162980 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.050+0000 7f010fe00640 1 -- 172.21.15.69:0/472730601 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f011000ce40 con 0x7f0118162980 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.050+0000 7f010fe00640 1 -- 172.21.15.69:0/472730601 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011000a860 con 0x7f0118162980 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.050+0000 7f011d600640 1 -- 172.21.15.69:0/472730601 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01180cd2e0 con 0x7f0118162980 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.050+0000 7f011d600640 1 -- 172.21.15.69:0/472730601 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01180cd8a0 con 0x7f0118162980 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.051+0000 7f011d600640 1 -- 172.21.15.69:0/472730601 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0118075d50 con 0x7f0118162980 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.052+0000 7f010fe00640 1 -- 172.21.15.69:0/472730601 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f011000aa00 con 0x7f0118162980 2024-04-24T08:19:55.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.053+0000 7f010fe00640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00f406c920 0x7f00f406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.053+0000 7f010fe00640 1 -- 172.21.15.69:0/472730601 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7f011008b1e0 con 0x7f0118162980 2024-04-24T08:19:55.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.053+0000 7f0117e00640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00f406c920 0x7f00f406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.055+0000 7f0117e00640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00f406c920 0x7f00f406ede0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0108034000 tx=0x7f010802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.057+0000 7f010fe00640 1 -- 172.21.15.69:0/472730601 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0110059430 con 0x7f0118162980 2024-04-24T08:19:55.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.262+0000 7f011d600640 1 -- 172.21.15.69:0/472730601 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01180d1dc0 con 0x7f0118162980 2024-04-24T08:19:55.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.264+0000 7f010fe00640 1 -- 172.21.15.69:0/472730601 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+16841 (secure 0 0 0) 0x7f0110058d30 con 0x7f0118162980 2024-04-24T08:19:55.266 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:19:55.266 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":99,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:19:49.975068+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:18:08.156864+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-04-24T08:18:09.368519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/3642138906":"2024-04-25T08:19:49.975039+0000","172.21.15.69:6833/3642138906":"2024-04-25T08:19:49.975039+0000","172.21.15.69:6835/412016900":"2024-04-25T08:19:12.588993+0000","172.21.15.69:6833/412016900":"2024-04-25T08:19:12.588993+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 -- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00f406c920 msgr2=0x7f00f406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00f406c920 0x7f00f406ede0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0108034000 tx=0x7f010802e000 comp rx=0 tx=0).stop 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 -- 172.21.15.69:0/472730601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0118162980 msgr2=0x7f011806e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0118162980 0x7f011806e210 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f011000c730 tx=0x7f011000cbf0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 -- 172.21.15.69:0/472730601 shutdown_connections 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f00f406c920 0x7f00f406ede0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0118162980 0x7f011806e210 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f011815e960 0x7f011806dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 --2- 172.21.15.69:0/472730601 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01180cfaf0 0x7f01180ce0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 -- 172.21.15.69:0/472730601 >> 172.21.15.69:0/472730601 conn(0x7f011809d900 msgr2=0x7f01180b56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 -- 172.21.15.69:0/472730601 shutdown_connections 2024-04-24T08:19:55.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.267+0000 7f011d600640 1 -- 172.21.15.69:0/472730601 wait complete. 2024-04-24T08:19:55.288 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:19:55.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.487+0000 7f6980c00640 1 Processor -- start 2024-04-24T08:19:55.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.487+0000 7f6980c00640 1 -- start start 2024-04-24T08:19:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.488+0000 7f6980c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c162300 0x7f697c143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.488+0000 7f6980c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f697c15f540 con 0x7f697c071e50 2024-04-24T08:19:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.488+0000 7f6980c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f697c15f820 con 0x7f697c072230 2024-04-24T08:19:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.488+0000 7f6980c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f697c15fb00 con 0x7f697c162300 2024-04-24T08:19:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.489+0000 7f697b400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f697c071e50 0x7f697c15ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33672/0 (socket says 172.21.15.69:33672) 2024-04-24T08:19:55.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.489+0000 7f697b400640 1 -- 172.21.15.69:0/1193769186 learned_addr learned my addr 172.21.15.69:0/1193769186 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.489+0000 7f697aa00640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c162300 0x7f697c143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.489+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1747193490 0 0) 0x7f697c15f540 con 0x7f697c071e50 2024-04-24T08:19:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.489+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6960003520 con 0x7f697c071e50 2024-04-24T08:19:55.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 400692492 0 0) 0x7f6960003520 con 0x7f697c071e50 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c162300 msgr2=0x7f697c143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c162300 0x7f697c143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 >> v1:172.21.15.121:6789/0 conn(0x7f697c072230 legacy=0x7f697c143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c06dda0 con 0x7f697c071e50 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f696c001670 con 0x7f697c071e50 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 >> v1:172.21.15.69:6789/0 conn(0x7f697c071e50 legacy=0x7f697c15ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c162300 0x7f69600039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6960003f60 0x7f6960004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6960004d40 0x7f69600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f697c15fb00 con 0x7f6960004d40 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f696000baa0 con 0x7f6960003f60 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f696000bca0 con 0x7f697c162300 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697be00640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6960004d40 0x7f69600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.490+0000 7f697b400640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c162300 0x7f69600039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.491+0000 7f697aa00640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6960003f60 0x7f6960004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.491+0000 7f697be00640 1 -- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c162300 msgr2=0x7f69600039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.491+0000 7f697be00640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c162300 0x7f69600039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.491+0000 7f697be00640 1 -- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6960003f60 msgr2=0x7f6960004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.491+0000 7f697be00640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6960003f60 0x7f6960004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.491+0000 7f697be00640 1 -- 172.21.15.69:0/1193769186 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6960003520 con 0x7f6960004d40 2024-04-24T08:19:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.492+0000 7f697be00640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6960004d40 0x7f69600051c0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f6974001e50 tx=0x7f6960005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1f02570b2ae73f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.492+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974001a70 con 0x7f6960004d40 2024-04-24T08:19:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.492+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6974003970 con 0x7f6960004d40 2024-04-24T08:19:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.492+0000 7f697a000640 1 -- 172.21.15.69:0/1193769186 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974003d60 con 0x7f6960004d40 2024-04-24T08:19:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.492+0000 7f6980c00640 1 -- 172.21.15.69:0/1193769186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6960004d40 msgr2=0x7f69600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.492+0000 7f6980c00640 1 --2- 172.21.15.69:0/1193769186 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6960004d40 0x7f69600051c0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f6974001e50 tx=0x7f6960005830 comp rx=0 tx=0).stop 2024-04-24T08:19:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.492+0000 7f697be00640 1 -- 172.21.15.69:0/1193769186 reap_dead start 2024-04-24T08:19:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.493+0000 7f6980c00640 1 -- 172.21.15.69:0/1193769186 shutdown_connections 2024-04-24T08:19:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.493+0000 7f6980c00640 1 -- 172.21.15.69:0/1193769186 >> 172.21.15.69:0/1193769186 conn(0x7f697c0d0160 msgr2=0x7f697c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.493+0000 7f6980c00640 1 -- 172.21.15.69:0/1193769186 shutdown_connections 2024-04-24T08:19:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.493+0000 7f6980c00640 1 -- 172.21.15.69:0/1193769186 wait complete. 2024-04-24T08:19:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.495+0000 7f6980c00640 1 Processor -- start 2024-04-24T08:19:55.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.495+0000 7f6980c00640 1 -- start start 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f6980c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6960004d40 0x7f697c09d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f6980c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c1f5470 0x7f697c0b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f6980c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f697c1570e0 0x7f697c1f47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f6980c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f697c143690 con 0x7f697c1570e0 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f6980c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f697c143810 con 0x7f6960004d40 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f6980c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f697c143990 con 0x7f697c1f5470 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f697be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f697c1570e0 0x7f697c1f47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f697be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f697c1570e0 0x7f697c1f47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40218/0 (socket says 172.21.15.69:40218) 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f697be00640 1 -- 172.21.15.69:0/1842496585 learned_addr learned my addr 172.21.15.69:0/1842496585 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f697aa00640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c1f5470 0x7f697c0b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f697be00640 1 -- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c1f5470 msgr2=0x7f697c0b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f697be00640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c1f5470 0x7f697c0b48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f697be00640 1 -- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6960004d40 msgr2=0x7f697c09d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f697be00640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6960004d40 0x7f697c09d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.496+0000 7f697be00640 1 -- 172.21.15.69:0/1842496585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f697c09dcd0 con 0x7f697c1570e0 2024-04-24T08:19:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.497+0000 7f697be00640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f697c1570e0 0x7f697c1f47e0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f6974001f70 tx=0x7f69740014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.497+0000 7f6973400640 1 -- 172.21.15.69:0/1842496585 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974037040 con 0x7f697c1570e0 2024-04-24T08:19:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.497+0000 7f6973400640 1 -- 172.21.15.69:0/1842496585 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6974003e50 con 0x7f697c1570e0 2024-04-24T08:19:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.497+0000 7f6980c00640 1 -- 172.21.15.69:0/1842496585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f697c160530 con 0x7f697c1570e0 2024-04-24T08:19:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.497+0000 7f6973400640 1 -- 172.21.15.69:0/1842496585 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6974007db0 con 0x7f697c1570e0 2024-04-24T08:19:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.497+0000 7f6980c00640 1 -- 172.21.15.69:0/1842496585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f697c160af0 con 0x7f697c1570e0 2024-04-24T08:19:55.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.498+0000 7f6980c00640 1 -- 172.21.15.69:0/1842496585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f697c0b5180 con 0x7f697c1570e0 2024-04-24T08:19:55.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.499+0000 7f6973400640 1 -- 172.21.15.69:0/1842496585 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6974042020 con 0x7f697c1570e0 2024-04-24T08:19:55.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.500+0000 7f6973400640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f695c06c8f0 0x7f695c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.500+0000 7f6973400640 1 -- 172.21.15.69:0/1842496585 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7f69740ab2a0 con 0x7f697c1570e0 2024-04-24T08:19:55.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.501+0000 7f697b400640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f695c06c8f0 0x7f695c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.502+0000 7f697b400640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f695c06c8f0 0x7f695c06edb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f697c0b3c60 tx=0x7f696c001d50 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.504+0000 7f6973400640 1 -- 172.21.15.69:0/1842496585 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f697c0b5180 con 0x7f697c1570e0 2024-04-24T08:19:55.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:55.744+0000 7f6980c00640 1 -- 172.21.15.69:0/1842496585 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f697c075d50 con 0x7f697c1570e0 2024-04-24T08:19:56.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.617+0000 7f6973400640 1 -- 172.21.15.69:0/1842496585 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v92) ==== 161+0+0 (secure 0 0 0) 0x7f6974079690 con 0x7f697c1570e0 2024-04-24T08:19:56.619 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:19:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.620+0000 7f6980c00640 1 -- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f695c06c8f0 msgr2=0x7f695c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.621+0000 7f6980c00640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f695c06c8f0 0x7f695c06edb0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f697c0b3c60 tx=0x7f696c001d50 comp rx=0 tx=0).stop 2024-04-24T08:19:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.621+0000 7f6980c00640 1 -- 172.21.15.69:0/1842496585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f697c1570e0 msgr2=0x7f697c1f47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.621+0000 7f6980c00640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f697c1570e0 0x7f697c1f47e0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f6974001f70 tx=0x7f69740014a0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.621+0000 7f6980c00640 1 -- 172.21.15.69:0/1842496585 shutdown_connections 2024-04-24T08:19:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.621+0000 7f6980c00640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f695c06c8f0 0x7f695c06edb0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.621+0000 7f6980c00640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f697c1570e0 0x7f697c1f47e0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.621+0000 7f6980c00640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f697c1f5470 0x7f697c0b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.621+0000 7f6980c00640 1 --2- 172.21.15.69:0/1842496585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6960004d40 0x7f697c09d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.621+0000 7f6980c00640 1 -- 172.21.15.69:0/1842496585 >> 172.21.15.69:0/1842496585 conn(0x7f697c0d0160 msgr2=0x7f697c0ad0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.621+0000 7f6980c00640 1 -- 172.21.15.69:0/1842496585 shutdown_connections 2024-04-24T08:19:56.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.622+0000 7f6980c00640 1 -- 172.21.15.69:0/1842496585 wait complete. 2024-04-24T08:19:56.644 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T08:19:56.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.846+0000 7f75d6200640 1 Processor -- start 2024-04-24T08:19:56.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.846+0000 7f75d6200640 1 -- start start 2024-04-24T08:19:56.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.847+0000 7f75d6200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75d015e050 0x7f75d006dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.847+0000 7f75d6200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75d015f120 0x7f75d006f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.847+0000 7f75d6200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75d015e430 con 0x7f75d0071d70 2024-04-24T08:19:56.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.847+0000 7f75d6200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f75d006e490 con 0x7f75d015f120 2024-04-24T08:19:56.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.847+0000 7f75d6200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75d006e770 con 0x7f75d015e050 2024-04-24T08:19:56.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.848+0000 7f75cfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75d015f120 0x7f75d006f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.848+0000 7f75cfe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f75d015f120 0x7f75d006f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40234/0 (socket says 172.21.15.69:40234) 2024-04-24T08:19:56.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.848+0000 7f75d5800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f75d0071d70 0x7f75d00cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33676/0 (socket says 172.21.15.69:33676) 2024-04-24T08:19:56.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.848+0000 7f75cfe00640 1 -- 172.21.15.69:0/259606078 learned_addr learned my addr 172.21.15.69:0/259606078 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:56.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.848+0000 7f75d4e00640 1 --2- 172.21.15.69:0/259606078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75d015e050 0x7f75d006dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.848+0000 7f75cfe00640 1 -- 172.21.15.69:0/259606078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75d015e050 msgr2=0x7f75d006dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.848+0000 7f75cfe00640 1 --2- 172.21.15.69:0/259606078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75d015e050 0x7f75d006dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.848+0000 7f75cfe00640 1 -- 172.21.15.69:0/259606078 >> v1:172.21.15.69:6789/0 conn(0x7f75d0071d70 legacy=0x7f75d00cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.848+0000 7f75cfe00640 1 -- 172.21.15.69:0/259606078 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d006ea50 con 0x7f75d015f120 2024-04-24T08:19:56.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.848+0000 7f75cf400640 1 -- 172.21.15.69:0/259606078 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 331589033 0 0) 0x7f75d015e430 con 0x7f75d0071d70 2024-04-24T08:19:56.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.849+0000 7f75cfe00640 1 --2- 172.21.15.69:0/259606078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 0x7f75d006f530 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009e50 tx=0x7f75b802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ffc2a07e891848 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75cf400640 1 -- 172.21.15.69:0/259606078 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8036070 con 0x7f75d015f120 2024-04-24T08:19:56.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75cf400640 1 -- 172.21.15.69:0/259606078 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75b8031600 con 0x7f75d015f120 2024-04-24T08:19:56.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75cf400640 1 -- 172.21.15.69:0/259606078 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b80319d0 con 0x7f75d015f120 2024-04-24T08:19:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75d6200640 1 -- 172.21.15.69:0/259606078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 msgr2=0x7f75d006f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75d6200640 1 --2- 172.21.15.69:0/259606078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 0x7f75d006f530 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009e50 tx=0x7f75b802a680 comp rx=0 tx=0).stop 2024-04-24T08:19:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75d6200640 1 -- 172.21.15.69:0/259606078 shutdown_connections 2024-04-24T08:19:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75d6200640 1 --2- 172.21.15.69:0/259606078 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 0x7f75d006f530 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75d6200640 1 --2- 172.21.15.69:0/259606078 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75d015e050 0x7f75d006dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75d6200640 1 -- 172.21.15.69:0/259606078 >> 172.21.15.69:0/259606078 conn(0x7f75d0067b80 msgr2=0x7f75d006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75d6200640 1 -- 172.21.15.69:0/259606078 shutdown_connections 2024-04-24T08:19:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.850+0000 7f75d6200640 1 -- 172.21.15.69:0/259606078 wait complete. 2024-04-24T08:19:56.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.852+0000 7f75d6200640 1 Processor -- start 2024-04-24T08:19:56.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.852+0000 7f75d6200640 1 -- start start 2024-04-24T08:19:56.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.853+0000 7f75d6200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75d0071d70 0x7f75d00ce030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.853+0000 7f75d6200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75d015e050 0x7f75d0151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.853+0000 7f75d6200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 0x7f75d0154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.853+0000 7f75d6200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f75d015f580 con 0x7f75d015f120 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.853+0000 7f75d6200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f75d006f010 con 0x7f75d0071d70 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.853+0000 7f75d6200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75d006fa70 con 0x7f75d015e050 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.853+0000 7f75d5800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 0x7f75d0154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.853+0000 7f75d5800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 0x7f75d0154b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40238/0 (socket says 172.21.15.69:40238) 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.853+0000 7f75d5800640 1 -- 172.21.15.69:0/1502740945 learned_addr learned my addr 172.21.15.69:0/1502740945 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75d4e00640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75d0071d70 0x7f75d00ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75cfe00640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75d015e050 0x7f75d0151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75d5800640 1 -- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75d015e050 msgr2=0x7f75d0151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75d5800640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75d015e050 0x7f75d0151ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75d5800640 1 -- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75d0071d70 msgr2=0x7f75d00ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75d5800640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75d0071d70 0x7f75d00ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75d5800640 1 -- 172.21.15.69:0/1502740945 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d00ce610 con 0x7f75d015f120 2024-04-24T08:19:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75d5800640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 0x7f75d0154b20 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f75c800c440 tx=0x7f75c800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75ccc00640 1 -- 172.21.15.69:0/1502740945 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c80074e0 con 0x7f75d015f120 2024-04-24T08:19:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75ccc00640 1 -- 172.21.15.69:0/1502740945 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c80047e0 con 0x7f75d015f120 2024-04-24T08:19:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.854+0000 7f75ccc00640 1 -- 172.21.15.69:0/1502740945 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8004bd0 con 0x7f75d015f120 2024-04-24T08:19:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.855+0000 7f75d6200640 1 -- 172.21.15.69:0/1502740945 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d015c590 con 0x7f75d015f120 2024-04-24T08:19:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.855+0000 7f75d6200640 1 -- 172.21.15.69:0/1502740945 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d015cb50 con 0x7f75d015f120 2024-04-24T08:19:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.856+0000 7f75d6200640 1 -- 172.21.15.69:0/1502740945 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75d0075d50 con 0x7f75d015f120 2024-04-24T08:19:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.857+0000 7f75ccc00640 1 -- 172.21.15.69:0/1502740945 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f75c8011070 con 0x7f75d015f120 2024-04-24T08:19:56.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.858+0000 7f75ccc00640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f75ac06c8f0 0x7f75ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.858+0000 7f75ccc00640 1 -- 172.21.15.69:0/1502740945 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7f75c808b7b0 con 0x7f75d015f120 2024-04-24T08:19:56.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.859+0000 7f75d4e00640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f75ac06c8f0 0x7f75ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.860+0000 7f75d4e00640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f75ac06c8f0 0x7f75ac06edb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f75c0006fd0 tx=0x7f75c0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:56.863+0000 7f75ccc00640 1 -- 172.21.15.69:0/1502740945 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75c8059b20 con 0x7f75d015f120 2024-04-24T08:19:57.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.105+0000 7f75d6200640 1 -- 172.21.15.69:0/1502740945 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f75d00728a0 con 0x7f75d015f120 2024-04-24T08:19:57.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.627+0000 7f75ccc00640 1 -- 172.21.15.69:0/1502740945 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v93) ==== 104+0+0 (secure 0 0 0) 0x7f75c8059420 con 0x7f75d015f120 2024-04-24T08:19:57.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.631+0000 7f75d6200640 1 -- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f75ac06c8f0 msgr2=0x7f75ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.631+0000 7f75d6200640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f75ac06c8f0 0x7f75ac06edb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f75c0006fd0 tx=0x7f75c0008040 comp rx=0 tx=0).stop 2024-04-24T08:19:57.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.631+0000 7f75d6200640 1 -- 172.21.15.69:0/1502740945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 msgr2=0x7f75d0154b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.631+0000 7f75d6200640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 0x7f75d0154b20 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f75c800c440 tx=0x7f75c800c900 comp rx=0 tx=0).stop 2024-04-24T08:19:57.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.631+0000 7f75d6200640 1 -- 172.21.15.69:0/1502740945 shutdown_connections 2024-04-24T08:19:57.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.631+0000 7f75d6200640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f75ac06c8f0 0x7f75ac06edb0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.631+0000 7f75d6200640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f75d015f120 0x7f75d0154b20 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.631+0000 7f75d6200640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f75d015e050 0x7f75d0151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.631+0000 7f75d6200640 1 --2- 172.21.15.69:0/1502740945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75d0071d70 0x7f75d00ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.631+0000 7f75d6200640 1 -- 172.21.15.69:0/1502740945 >> 172.21.15.69:0/1502740945 conn(0x7f75d0067b80 msgr2=0x7f75d00af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:57.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.632+0000 7f75d6200640 1 -- 172.21.15.69:0/1502740945 shutdown_connections 2024-04-24T08:19:57.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.632+0000 7f75d6200640 1 -- 172.21.15.69:0/1502740945 wait complete. 2024-04-24T08:19:57.658 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T08:19:57.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.865+0000 7f624e000640 1 Processor -- start 2024-04-24T08:19:57.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.866+0000 7f624e000640 1 -- start start 2024-04-24T08:19:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.867+0000 7f624e000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f624815c690 0x7f62480d3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.867+0000 7f624e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62481625b0 0x7f6248162a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.867+0000 7f624e000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f624815f0d0 con 0x7f62480d1070 2024-04-24T08:19:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.867+0000 7f624e000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f624815fa00 con 0x7f624815c690 2024-04-24T08:19:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.867+0000 7f624e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f624815f3b0 con 0x7f62481625b0 2024-04-24T08:19:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.867+0000 7f6246a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f62480d1070 0x7f62480d14f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:33688/0 (socket says 172.21.15.69:33688) 2024-04-24T08:19:57.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.867+0000 7f6246a00640 1 -- 172.21.15.69:0/1425511201 learned_addr learned my addr 172.21.15.69:0/1425511201 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6247e00640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62481625b0 0x7f6248162a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6247400640 1 --2- 172.21.15.69:0/1425511201 >> v2:172.21.15.121:3300/0 conn(0x7f624815c690 0x7f62480d3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141932283 0 0) 0x7f624815f0d0 con 0x7f62480d1070 2024-04-24T08:19:57.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6228003520 con 0x7f62480d1070 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3835377459 0 0) 0x7f6228003520 con 0x7f62480d1070 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62481625b0 msgr2=0x7f6248162a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62481625b0 0x7f6248162a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 >> v2:172.21.15.121:3300/0 conn(0x7f624815c690 msgr2=0x7f62480d3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 --2- 172.21.15.69:0/1425511201 >> v2:172.21.15.121:3300/0 conn(0x7f624815c690 0x7f62480d3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f624815f690 con 0x7f62480d1070 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f6230001670 con 0x7f62480d1070 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 >> v1:172.21.15.69:6789/0 conn(0x7f62480d1070 legacy=0x7f62480d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62280039b0 0x7f6228003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6228004800 0x7f6228004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 0x7f622800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f624815fa00 con 0x7f62280051c0 2024-04-24T08:19:57.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f624815f3b0 con 0x7f6228004800 2024-04-24T08:19:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f622800c000 con 0x7f62280039b0 2024-04-24T08:19:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.868+0000 7f6247e00640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62481625b0 0x7f6248162a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.869+0000 7f6247e00640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 0x7f622800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.869+0000 7f6247400640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62280039b0 0x7f6228003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.869+0000 7f6246a00640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6228004800 0x7f6228004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.869+0000 7f6247e00640 1 -- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62280039b0 msgr2=0x7f6228003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.869+0000 7f6247e00640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f62280039b0 0x7f6228003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.869+0000 7f6247e00640 1 -- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6228004800 msgr2=0x7f6228004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.869+0000 7f6247e00640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6228004800 0x7f6228004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.869+0000 7f6247e00640 1 -- 172.21.15.69:0/1425511201 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6228003520 con 0x7f62280051c0 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.870+0000 7f6247e00640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 0x7f622800b9c0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f6240009f00 tx=0x7f6228005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58bc018d5b253d05 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.871+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240009b50 con 0x7f62280051c0 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.871+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6240003990 con 0x7f62280051c0 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.871+0000 7f6246000640 1 -- 172.21.15.69:0/1425511201 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6240003d60 con 0x7f62280051c0 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.871+0000 7f624e000640 1 -- 172.21.15.69:0/1425511201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 msgr2=0x7f622800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.871+0000 7f624e000640 1 --2- 172.21.15.69:0/1425511201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 0x7f622800b9c0 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f6240009f00 tx=0x7f6228005830 comp rx=0 tx=0).stop 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.871+0000 7f6247e00640 1 -- 172.21.15.69:0/1425511201 reap_dead start 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.871+0000 7f624e000640 1 -- 172.21.15.69:0/1425511201 shutdown_connections 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.871+0000 7f624e000640 1 -- 172.21.15.69:0/1425511201 >> 172.21.15.69:0/1425511201 conn(0x7f6248067760 msgr2=0x7f6248067b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.871+0000 7f624e000640 1 -- 172.21.15.69:0/1425511201 shutdown_connections 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.871+0000 7f624e000640 1 -- 172.21.15.69:0/1425511201 wait complete. 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.873+0000 7f624e000640 1 Processor -- start 2024-04-24T08:19:57.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.873+0000 7f624e000640 1 -- start start 2024-04-24T08:19:57.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.874+0000 7f624e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 0x7f62480cad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.874+0000 7f624e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6248147a60 0x7f6248158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.874+0000 7f624e000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6248147ee0 0x7f6248158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.874+0000 7f624e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f62480b6da0 con 0x7f62280051c0 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.874+0000 7f624e000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f62480b7140 con 0x7f6248147ee0 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.874+0000 7f624e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f62480b6f20 con 0x7f6248147a60 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.874+0000 7f6247400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 0x7f62480cad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f6247400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 0x7f62480cad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40258/0 (socket says 172.21.15.69:40258) 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f6247400640 1 -- 172.21.15.69:0/1562300407 learned_addr learned my addr 172.21.15.69:0/1562300407 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f6247400640 1 -- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6248147a60 msgr2=0x7f6248158590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f6246a00640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6248147a60 0x7f6248158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f6247e00640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6248147ee0 0x7f6248158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f6247400640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6248147a60 0x7f6248158590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f6247400640 1 -- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6248147ee0 msgr2=0x7f6248158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f6247400640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6248147ee0 0x7f6248158ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:57.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f6247400640 1 -- 172.21.15.69:0/1562300407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62480ccbb0 con 0x7f62280051c0 2024-04-24T08:19:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f6247400640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 0x7f62480cad60 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f623800f660 tx=0x7f623800fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.875+0000 7f623f400640 1 -- 172.21.15.69:0/1562300407 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6238019020 con 0x7f62280051c0 2024-04-24T08:19:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.876+0000 7f623f400640 1 -- 172.21.15.69:0/1562300407 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f623800d940 con 0x7f62280051c0 2024-04-24T08:19:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.876+0000 7f624e000640 1 -- 172.21.15.69:0/1562300407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62480cce90 con 0x7f62280051c0 2024-04-24T08:19:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.876+0000 7f623f400640 1 -- 172.21.15.69:0/1562300407 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623800dcf0 con 0x7f62280051c0 2024-04-24T08:19:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.876+0000 7f624e000640 1 -- 172.21.15.69:0/1562300407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62480cd450 con 0x7f62280051c0 2024-04-24T08:19:57.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.877+0000 7f624e000640 1 -- 172.21.15.69:0/1562300407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6248068a80 con 0x7f62280051c0 2024-04-24T08:19:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.878+0000 7f623f400640 1 -- 172.21.15.69:0/1562300407 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f623801c020 con 0x7f62280051c0 2024-04-24T08:19:57.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.879+0000 7f623f400640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f622406c920 0x7f622406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:57.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.879+0000 7f623f400640 1 -- 172.21.15.69:0/1562300407 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 5881+0+0 (secure 0 0 0) 0x7f623808c4f0 con 0x7f62280051c0 2024-04-24T08:19:57.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.880+0000 7f6246a00640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f622406c920 0x7f622406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:57.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.881+0000 7f6246a00640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f622406c920 0x7f622406ede0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f624815c0a0 tx=0x7f6230001db0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:57.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:57.884+0000 7f623f400640 1 -- 172.21.15.69:0/1562300407 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6238023020 con 0x7f62280051c0 2024-04-24T08:19:58.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.083+0000 7f624e000640 1 -- 172.21.15.69:0/1562300407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f62480d41b0 con 0x7f62280051c0 2024-04-24T08:19:58.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.639+0000 7f623f400640 1 -- 172.21.15.69:0/1562300407 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) ==== 181+0+0 (secure 0 0 0) 0x7f623805a7d0 con 0x7f62280051c0 2024-04-24T08:19:58.641 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' removed 2024-04-24T08:19:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.644+0000 7f624e000640 1 -- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f622406c920 msgr2=0x7f622406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.644+0000 7f624e000640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f622406c920 0x7f622406ede0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f624815c0a0 tx=0x7f6230001db0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.644+0000 7f624e000640 1 -- 172.21.15.69:0/1562300407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 msgr2=0x7f62480cad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:58.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.644+0000 7f624e000640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 0x7f62480cad60 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f623800f660 tx=0x7f623800fb20 comp rx=0 tx=0).stop 2024-04-24T08:19:58.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.645+0000 7f624e000640 1 -- 172.21.15.69:0/1562300407 shutdown_connections 2024-04-24T08:19:58.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.645+0000 7f624e000640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f622406c920 0x7f622406ede0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.645+0000 7f624e000640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6248147ee0 0x7f6248158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.645+0000 7f624e000640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6248147a60 0x7f6248158590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.645+0000 7f624e000640 1 --2- 172.21.15.69:0/1562300407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f62280051c0 0x7f62480cad60 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.645+0000 7f624e000640 1 -- 172.21.15.69:0/1562300407 >> 172.21.15.69:0/1562300407 conn(0x7f6248067760 msgr2=0x7f624815cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:58.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.645+0000 7f624e000640 1 -- 172.21.15.69:0/1562300407 shutdown_connections 2024-04-24T08:19:58.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.645+0000 7f624e000640 1 -- 172.21.15.69:0/1562300407 wait complete. 2024-04-24T08:19:58.670 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T08:19:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.866+0000 7f6627000640 1 Processor -- start 2024-04-24T08:19:58.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.866+0000 7f6627000640 1 -- start start 2024-04-24T08:19:58.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f6627000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f662006f5e0 0x7f66201623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:58.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f6627000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f66200b32d0 0x7f662015f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:58.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f6627000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f662015edb0 con 0x7f662015fad0 2024-04-24T08:19:58.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f6627000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f662006f9c0 con 0x7f662006f5e0 2024-04-24T08:19:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f6627000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f66201628e0 con 0x7f66200b32d0 2024-04-24T08:19:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f661fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f662006f5e0 0x7f66201623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f6625000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f662015fad0 0x7f66200b45f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56256/0 (socket says 172.21.15.69:56256) 2024-04-24T08:19:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f6625000640 1 -- 172.21.15.69:0/2404501034 learned_addr learned my addr 172.21.15.69:0/2404501034 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f661fe00640 1 -- 172.21.15.69:0/2404501034 >> v2:172.21.15.121:3300/0 conn(0x7f66200b32d0 msgr2=0x7f662015f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f661fe00640 1 --2- 172.21.15.69:0/2404501034 >> v2:172.21.15.121:3300/0 conn(0x7f66200b32d0 0x7f662015f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f661fe00640 1 -- 172.21.15.69:0/2404501034 >> v1:172.21.15.69:6789/0 conn(0x7f662015fad0 legacy=0x7f66200b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.868+0000 7f661fe00640 1 -- 172.21.15.69:0/2404501034 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6620162bb0 con 0x7f662006f5e0 2024-04-24T08:19:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.870+0000 7f661fe00640 1 --2- 172.21.15.69:0/2404501034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f662006f5e0 0x7f66201623a0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f66200b3e90 tx=0x7f661802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86568d799f6c66f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.870+0000 7f661ea00640 1 -- 172.21.15.69:0/2404501034 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618037070 con 0x7f662006f5e0 2024-04-24T08:19:58.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.870+0000 7f661ea00640 1 -- 172.21.15.69:0/2404501034 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6618032440 con 0x7f662006f5e0 2024-04-24T08:19:58.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.870+0000 7f661ea00640 1 -- 172.21.15.69:0/2404501034 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618032810 con 0x7f662006f5e0 2024-04-24T08:19:58.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.871+0000 7f6627000640 1 -- 172.21.15.69:0/2404501034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f662006f5e0 msgr2=0x7f66201623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:58.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.871+0000 7f6627000640 1 --2- 172.21.15.69:0/2404501034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f662006f5e0 0x7f66201623a0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f66200b3e90 tx=0x7f661802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.871+0000 7f6627000640 1 -- 172.21.15.69:0/2404501034 shutdown_connections 2024-04-24T08:19:58.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.871+0000 7f6627000640 1 --2- 172.21.15.69:0/2404501034 >> v2:172.21.15.121:3300/0 conn(0x7f66200b32d0 0x7f662015f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.871+0000 7f6627000640 1 --2- 172.21.15.69:0/2404501034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f662006f5e0 0x7f66201623a0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.871+0000 7f6627000640 1 -- 172.21.15.69:0/2404501034 >> 172.21.15.69:0/2404501034 conn(0x7f66200b2380 msgr2=0x7f66200b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:58.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.871+0000 7f6627000640 1 -- 172.21.15.69:0/2404501034 shutdown_connections 2024-04-24T08:19:58.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.871+0000 7f6627000640 1 -- 172.21.15.69:0/2404501034 wait complete. 2024-04-24T08:19:58.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.872+0000 7f6627000640 1 Processor -- start 2024-04-24T08:19:58.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.873+0000 7f6627000640 1 -- start start 2024-04-24T08:19:58.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f6627000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f662006f5e0 0x7f66200ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:58.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f6627000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66200b32d0 0x7f6620159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:58.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f6627000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f662015a230 0x7f6620159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:58.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f6627000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f662015f070 con 0x7f66200b32d0 2024-04-24T08:19:58.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f6627000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6620163170 con 0x7f662006f5e0 2024-04-24T08:19:58.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f6627000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f66200b4d00 con 0x7f662015a230 2024-04-24T08:19:58.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f661f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66200b32d0 0x7f6620159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:58.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f661f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66200b32d0 0x7f6620159670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59404/0 (socket says 172.21.15.69:59404) 2024-04-24T08:19:58.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f661f400640 1 -- 172.21.15.69:0/372266086 learned_addr learned my addr 172.21.15.69:0/372266086 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:58.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f661f400640 1 -- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f662015a230 msgr2=0x7f6620159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:58.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f661f400640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f662015a230 0x7f6620159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f661f400640 1 -- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f662006f5e0 msgr2=0x7f66200ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:58.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f661f400640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f662006f5e0 0x7f66200ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:58.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.874+0000 7f661f400640 1 -- 172.21.15.69:0/372266086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66201e8810 con 0x7f66200b32d0 2024-04-24T08:19:58.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.875+0000 7f661f400640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66200b32d0 0x7f6620159670 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f660c00b4a0 tx=0x7f660c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:58.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.875+0000 7f6617e00640 1 -- 172.21.15.69:0/372266086 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c014070 con 0x7f66200b32d0 2024-04-24T08:19:58.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.875+0000 7f6617e00640 1 -- 172.21.15.69:0/372266086 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f660c010610 con 0x7f66200b32d0 2024-04-24T08:19:58.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.875+0000 7f6617e00640 1 -- 172.21.15.69:0/372266086 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f660c010a00 con 0x7f66200b32d0 2024-04-24T08:19:58.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.875+0000 7f6627000640 1 -- 172.21.15.69:0/372266086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f662015cbb0 con 0x7f66200b32d0 2024-04-24T08:19:58.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.875+0000 7f6627000640 1 -- 172.21.15.69:0/372266086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66201e8310 con 0x7f66200b32d0 2024-04-24T08:19:58.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.876+0000 7f6627000640 1 -- 172.21.15.69:0/372266086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f662006e8d0 con 0x7f66200b32d0 2024-04-24T08:19:58.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.877+0000 7f6617e00640 1 -- 172.21.15.69:0/372266086 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f660c017070 con 0x7f66200b32d0 2024-04-24T08:19:58.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.878+0000 7f6617e00640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f660006c8f0 0x7f660006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:58.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.878+0000 7f6617e00640 1 -- 172.21.15.69:0/372266086 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 5443+0+0 (secure 0 0 0) 0x7f660c08b3d0 con 0x7f66200b32d0 2024-04-24T08:19:58.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.879+0000 7f661fe00640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f660006c8f0 0x7f660006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:58.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.880+0000 7f661fe00640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f660006c8f0 0x7f660006edb0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6618034000 tx=0x7f661802c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:58.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:58.883+0000 7f6617e00640 1 -- 172.21.15.69:0/372266086 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f660c059900 con 0x7f66200b32d0 2024-04-24T08:19:59.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.079+0000 7f6627000640 1 -- 172.21.15.69:0/372266086 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6620068a80 con 0x7f66200b32d0 2024-04-24T08:19:59.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.649+0000 7f6617e00640 1 -- 172.21.15.69:0/372266086 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) ==== 169+0+0 (secure 0 0 0) 0x7f660c089e20 con 0x7f66200b32d0 2024-04-24T08:19:59.651 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' removed 2024-04-24T08:19:59.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.652+0000 7f6627000640 1 -- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f660006c8f0 msgr2=0x7f660006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.652+0000 7f6627000640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f660006c8f0 0x7f660006edb0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f6618034000 tx=0x7f661802c000 comp rx=0 tx=0).stop 2024-04-24T08:19:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.653+0000 7f6627000640 1 -- 172.21.15.69:0/372266086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66200b32d0 msgr2=0x7f6620159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.653+0000 7f6627000640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66200b32d0 0x7f6620159670 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f660c00b4a0 tx=0x7f660c00b960 comp rx=0 tx=0).stop 2024-04-24T08:19:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.653+0000 7f6627000640 1 -- 172.21.15.69:0/372266086 shutdown_connections 2024-04-24T08:19:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.653+0000 7f6627000640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f660006c8f0 0x7f660006edb0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.653+0000 7f6627000640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66200b32d0 0x7f6620159670 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.653+0000 7f6627000640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f662015a230 0x7f6620159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.653+0000 7f6627000640 1 --2- 172.21.15.69:0/372266086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f662006f5e0 0x7f66200ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.653+0000 7f6627000640 1 -- 172.21.15.69:0/372266086 >> 172.21.15.69:0/372266086 conn(0x7f66200b2380 msgr2=0x7f66200d0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:59.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.654+0000 7f6627000640 1 -- 172.21.15.69:0/372266086 shutdown_connections 2024-04-24T08:19:59.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.655+0000 7f6627000640 1 -- 172.21.15.69:0/372266086 wait complete. 2024-04-24T08:19:59.681 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-24T08:19:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.880+0000 7f8437200640 1 Processor -- start 2024-04-24T08:19:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.880+0000 7f8437200640 1 -- start start 2024-04-24T08:19:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.881+0000 7f8437200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843815e6d0 0x7f843806cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.881+0000 7f8437200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f843815eab0 0x7f84380d4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.881+0000 7f8437200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84380d4bf0 con 0x7f84380d4760 2024-04-24T08:19:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.881+0000 7f8437200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f84380d2ea0 con 0x7f843815eab0 2024-04-24T08:19:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.881+0000 7f8437200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f84380d3180 con 0x7f843815e6d0 2024-04-24T08:19:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.882+0000 7f8435400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f843815eab0 0x7f84380d4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.882+0000 7f8435e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843815e6d0 0x7f843806cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.882+0000 7f8436800640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f84380d4760 0x7f84380d21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:37266/0 (socket says 172.21.15.69:37266) 2024-04-24T08:19:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.882+0000 7f8436800640 1 -- 172.21.15.69:0/1363003953 learned_addr learned my addr 172.21.15.69:0/1363003953 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:19:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.883+0000 7f8435400640 1 -- 172.21.15.69:0/1363003953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843815e6d0 msgr2=0x7f843806cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.883+0000 7f8435400640 1 --2- 172.21.15.69:0/1363003953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843815e6d0 0x7f843806cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.883+0000 7f8435400640 1 -- 172.21.15.69:0/1363003953 >> v1:172.21.15.121:6789/0 conn(0x7f84380d4760 legacy=0x7f84380d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.883+0000 7f8434a00640 1 -- 172.21.15.69:0/1363003953 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 747292090 0 0) 0x7f84380d4bf0 con 0x7f84380d4760 2024-04-24T08:19:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.883+0000 7f8435400640 1 -- 172.21.15.69:0/1363003953 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84380cf220 con 0x7f843815eab0 2024-04-24T08:19:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.884+0000 7f8435400640 1 --2- 172.21.15.69:0/1363003953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f843815eab0 0x7f84380d4220 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f84240097a0 tx=0x7f842402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a96797f8facb076 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:59.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8434a00640 1 -- 172.21.15.69:0/1363003953 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424037070 con 0x7f843815eab0 2024-04-24T08:19:59.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8434a00640 1 -- 172.21.15.69:0/1363003953 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f842402ae50 con 0x7f843815eab0 2024-04-24T08:19:59.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8434a00640 1 -- 172.21.15.69:0/1363003953 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424032660 con 0x7f843815eab0 2024-04-24T08:19:59.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8437200640 1 -- 172.21.15.69:0/1363003953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f843815eab0 msgr2=0x7f84380d4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:59.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8437200640 1 --2- 172.21.15.69:0/1363003953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f843815eab0 0x7f84380d4220 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f84240097a0 tx=0x7f842402a100 comp rx=0 tx=0).stop 2024-04-24T08:19:59.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8437200640 1 -- 172.21.15.69:0/1363003953 shutdown_connections 2024-04-24T08:19:59.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8437200640 1 --2- 172.21.15.69:0/1363003953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f843815eab0 0x7f84380d4220 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:59.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8437200640 1 --2- 172.21.15.69:0/1363003953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843815e6d0 0x7f843806cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:59.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8437200640 1 -- 172.21.15.69:0/1363003953 >> 172.21.15.69:0/1363003953 conn(0x7f843809d8e0 msgr2=0x7f843808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:59.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8437200640 1 -- 172.21.15.69:0/1363003953 shutdown_connections 2024-04-24T08:19:59.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.885+0000 7f8437200640 1 -- 172.21.15.69:0/1363003953 wait complete. 2024-04-24T08:19:59.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.887+0000 7f8437200640 1 Processor -- start 2024-04-24T08:19:59.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.887+0000 7f8437200640 1 -- start start 2024-04-24T08:19:59.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.888+0000 7f8437200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84380d4760 0x7f84380b8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.888+0000 7f8437200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843815e6d0 0x7f8438154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.888+0000 7f8437200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843815eab0 0x7f84380ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.888+0000 7f8437200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84380d3460 con 0x7f843815e6d0 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.888+0000 7f8437200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f843806d300 con 0x7f84380d4760 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.888+0000 7f8437200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f84380d28c0 con 0x7f843815eab0 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.888+0000 7f8435400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843815e6d0 0x7f8438154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8435400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843815e6d0 0x7f8438154a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59416/0 (socket says 172.21.15.69:59416) 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8435400640 1 -- 172.21.15.69:0/1129655853 learned_addr learned my addr 172.21.15.69:0/1129655853 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8435e00640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84380d4760 0x7f84380b8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8436800640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843815eab0 0x7f84380ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8435400640 1 -- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843815eab0 msgr2=0x7f84380ca2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8435400640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843815eab0 0x7f84380ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:59.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8435400640 1 -- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84380d4760 msgr2=0x7f84380b8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8435400640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84380d4760 0x7f84380b8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8435400640 1 -- 172.21.15.69:0/1129655853 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f843815fee0 con 0x7f843815e6d0 2024-04-24T08:19:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8435e00640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84380d4760 0x7f84380b8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:59.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.889+0000 7f8435400640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843815e6d0 0x7f8438154a80 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f84240097a0 tx=0x7f84240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:59.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.890+0000 7f842e000640 1 -- 172.21.15.69:0/1129655853 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424037040 con 0x7f843815e6d0 2024-04-24T08:19:59.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.890+0000 7f842e000640 1 -- 172.21.15.69:0/1129655853 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8424004030 con 0x7f843815e6d0 2024-04-24T08:19:59.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.890+0000 7f842e000640 1 -- 172.21.15.69:0/1129655853 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424004420 con 0x7f843815e6d0 2024-04-24T08:19:59.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.890+0000 7f8437200640 1 -- 172.21.15.69:0/1129655853 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8438154fc0 con 0x7f843815e6d0 2024-04-24T08:19:59.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.890+0000 7f8437200640 1 -- 172.21.15.69:0/1129655853 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84380caed0 con 0x7f843815e6d0 2024-04-24T08:19:59.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.891+0000 7f8437200640 1 -- 172.21.15.69:0/1129655853 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8438075d50 con 0x7f843815e6d0 2024-04-24T08:19:59.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.892+0000 7f842e000640 1 -- 172.21.15.69:0/1129655853 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8424003610 con 0x7f843815e6d0 2024-04-24T08:19:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.892+0000 7f842e000640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 0x7f841806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.893+0000 7f842e000640 1 -- 172.21.15.69:0/1129655853 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5029+0+0 (secure 0 0 0) 0x7f84240ac670 con 0x7f843815e6d0 2024-04-24T08:19:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.893+0000 7f8435e00640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 0x7f841806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.894+0000 7f8435e00640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 0x7f841806edb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8428009ac0 tx=0x7f8428008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:59.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:19:59.897+0000 7f842e000640 1 -- 172.21.15.69:0/1129655853 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f842407adc0 con 0x7f843815e6d0 2024-04-24T08:20:00.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.100+0000 7f8437200640 1 -- 172.21.15.69:0/1129655853 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f843806d060 con 0x7f843815e6d0 2024-04-24T08:20:00.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.108+0000 7f842e000640 1 -- 172.21.15.69:0/1129655853 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) ==== 104+0+0 (secure 0 0 0) 0x7f842407a6c0 con 0x7f843815e6d0 2024-04-24T08:20:00.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.110+0000 7f8437200640 1 -- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 msgr2=0x7f841806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.110+0000 7f8437200640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 0x7f841806edb0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8428009ac0 tx=0x7f8428008040 comp rx=0 tx=0).stop 2024-04-24T08:20:00.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.110+0000 7f8437200640 1 -- 172.21.15.69:0/1129655853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843815e6d0 msgr2=0x7f8438154a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.110+0000 7f8437200640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843815e6d0 0x7f8438154a80 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f84240097a0 tx=0x7f84240046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.110+0000 7f8437200640 1 -- 172.21.15.69:0/1129655853 shutdown_connections 2024-04-24T08:20:00.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.110+0000 7f8437200640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f841806c8f0 0x7f841806edb0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.110+0000 7f8437200640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f843815e6d0 0x7f8438154a80 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.110+0000 7f8437200640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f843815eab0 0x7f84380ca2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.110+0000 7f8437200640 1 --2- 172.21.15.69:0/1129655853 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f84380d4760 0x7f84380b8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.110+0000 7f8437200640 1 -- 172.21.15.69:0/1129655853 >> 172.21.15.69:0/1129655853 conn(0x7f843809d8e0 msgr2=0x7f84380cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:00.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.111+0000 7f8437200640 1 -- 172.21.15.69:0/1129655853 shutdown_connections 2024-04-24T08:20:00.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.111+0000 7f8437200640 1 -- 172.21.15.69:0/1129655853 wait complete. 2024-04-24T08:20:00.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-04-24T08:20:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.322+0000 7ff67e800640 1 Processor -- start 2024-04-24T08:20:00.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.322+0000 7ff67e800640 1 -- start start 2024-04-24T08:20:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.323+0000 7ff67e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 0x7ff678072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.323+0000 7ff67e800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff67806f730 0x7ff67806dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.323+0000 7ff67e800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff678072690 con 0x7ff6780b3320 2024-04-24T08:20:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.324+0000 7ff67e800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff6780b4f80 con 0x7ff67806f730 2024-04-24T08:20:00.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.324+0000 7ff67e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff6780b4370 con 0x7ff678071d70 2024-04-24T08:20:00.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.324+0000 7ff676a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff67806f730 0x7ff67806dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:00.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.324+0000 7ff677400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 0x7ff678072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:00.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.324+0000 7ff676a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff67806f730 0x7ff67806dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:34886/0 (socket says 172.21.15.69:34886) 2024-04-24T08:20:00.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.324+0000 7ff676a00640 1 -- 172.21.15.69:0/2380311145 learned_addr learned my addr 172.21.15.69:0/2380311145 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:00.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.324+0000 7ff677e00640 1 --1- 172.21.15.69:0/2380311145 >> v1:172.21.15.121:6789/0 conn(0x7ff6780b3320 0x7ff67806e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:37276/0 (socket says 172.21.15.69:37276) 2024-04-24T08:20:00.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.325+0000 7ff676a00640 1 -- 172.21.15.69:0/2380311145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 msgr2=0x7ff678072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:00.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.325+0000 7ff676a00640 1 --2- 172.21.15.69:0/2380311145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 0x7ff678072150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.325+0000 7ff676a00640 1 -- 172.21.15.69:0/2380311145 >> v1:172.21.15.121:6789/0 conn(0x7ff6780b3320 legacy=0x7ff67806e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.325+0000 7ff676a00640 1 -- 172.21.15.69:0/2380311145 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6780b4650 con 0x7ff67806f730 2024-04-24T08:20:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.326+0000 7ff676a00640 1 --2- 172.21.15.69:0/2380311145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67806f730 0x7ff67806dd70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff660009870 tx=0x7ff66002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3b2e9921f40936e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:00.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.327+0000 7ff676000640 1 -- 172.21.15.69:0/2380311145 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660037070 con 0x7ff67806f730 2024-04-24T08:20:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.327+0000 7ff676000640 1 -- 172.21.15.69:0/2380311145 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff660032440 con 0x7ff67806f730 2024-04-24T08:20:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.327+0000 7ff676000640 1 -- 172.21.15.69:0/2380311145 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660032810 con 0x7ff67806f730 2024-04-24T08:20:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.328+0000 7ff67e800640 1 -- 172.21.15.69:0/2380311145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67806f730 msgr2=0x7ff67806dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.328+0000 7ff67e800640 1 --2- 172.21.15.69:0/2380311145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67806f730 0x7ff67806dd70 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff660009870 tx=0x7ff66002a270 comp rx=0 tx=0).stop 2024-04-24T08:20:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.328+0000 7ff67e800640 1 -- 172.21.15.69:0/2380311145 shutdown_connections 2024-04-24T08:20:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.328+0000 7ff67e800640 1 --2- 172.21.15.69:0/2380311145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67806f730 0x7ff67806dd70 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.328+0000 7ff67e800640 1 --2- 172.21.15.69:0/2380311145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 0x7ff678072150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.328+0000 7ff67e800640 1 -- 172.21.15.69:0/2380311145 >> 172.21.15.69:0/2380311145 conn(0x7ff678067d90 msgr2=0x7ff67806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.328+0000 7ff67e800640 1 -- 172.21.15.69:0/2380311145 shutdown_connections 2024-04-24T08:20:00.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.328+0000 7ff67e800640 1 -- 172.21.15.69:0/2380311145 wait complete. 2024-04-24T08:20:00.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.329+0000 7ff67e800640 1 Processor -- start 2024-04-24T08:20:00.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.330+0000 7ff67e800640 1 -- start start 2024-04-24T08:20:00.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.330+0000 7ff67e800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67806f730 0x7ff67813a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.330+0000 7ff67e800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 0x7ff678136b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff67e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6780b3320 0x7ff678153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff67e800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff67806ee60 con 0x7ff6780b3320 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff67e800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff6780b4c10 con 0x7ff67806f730 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff67e800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff67815b040 con 0x7ff678071d70 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff677e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6780b3320 0x7ff678153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff677e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6780b3320 0x7ff678153aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59432/0 (socket says 172.21.15.69:59432) 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff677e00640 1 -- 172.21.15.69:0/4104998311 learned_addr learned my addr 172.21.15.69:0/4104998311 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff677400640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67806f730 0x7ff67813a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff676a00640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 0x7ff678136b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff677e00640 1 -- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 msgr2=0x7ff678136b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff677e00640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 0x7ff678136b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff677e00640 1 -- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67806f730 msgr2=0x7ff67813a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff677e00640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67806f730 0x7ff67813a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff677e00640 1 -- 172.21.15.69:0/4104998311 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff678137430 con 0x7ff6780b3320 2024-04-24T08:20:00.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.331+0000 7ff676a00640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 0x7ff678136b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:00.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.332+0000 7ff677e00640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6780b3320 0x7ff678153aa0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7ff670001eb0 tx=0x7ff67000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:00.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.332+0000 7ff66f400640 1 -- 172.21.15.69:0/4104998311 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6700074e0 con 0x7ff6780b3320 2024-04-24T08:20:00.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.332+0000 7ff66f400640 1 -- 172.21.15.69:0/4104998311 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff670007680 con 0x7ff6780b3320 2024-04-24T08:20:00.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.332+0000 7ff66f400640 1 -- 172.21.15.69:0/4104998311 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6700049e0 con 0x7ff6780b3320 2024-04-24T08:20:00.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.332+0000 7ff67e800640 1 -- 172.21.15.69:0/4104998311 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff678137710 con 0x7ff6780b3320 2024-04-24T08:20:00.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.332+0000 7ff67e800640 1 -- 172.21.15.69:0/4104998311 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff678137cd0 con 0x7ff6780b3320 2024-04-24T08:20:00.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.333+0000 7ff67e800640 1 -- 172.21.15.69:0/4104998311 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff678068a80 con 0x7ff6780b3320 2024-04-24T08:20:00.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.334+0000 7ff66f400640 1 -- 172.21.15.69:0/4104998311 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff67001a020 con 0x7ff6780b3320 2024-04-24T08:20:00.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.335+0000 7ff66f400640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff65406c8f0 0x7ff65406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.335+0000 7ff66f400640 1 -- 172.21.15.69:0/4104998311 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5029+0+0 (secure 0 0 0) 0x7ff67008aff0 con 0x7ff6780b3320 2024-04-24T08:20:00.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.336+0000 7ff677400640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff65406c8f0 0x7ff65406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:00.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.337+0000 7ff677400640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff65406c8f0 0x7ff65406edb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff6780b1f00 tx=0x7ff66800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:00.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.340+0000 7ff66f400640 1 -- 172.21.15.69:0/4104998311 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff6700596f0 con 0x7ff6780b3320 2024-04-24T08:20:00.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.618+0000 7ff67e800640 1 -- 172.21.15.69:0/4104998311 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) -- 0x7ff67815b780 con 0x7ff6780b3320 2024-04-24T08:20:00.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.652+0000 7ff66f400640 1 -- 172.21.15.69:0/4104998311 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v536) ==== 146+0+0 (secure 0 0 0) 0x7ff670058ff0 con 0x7ff6780b3320 2024-04-24T08:20:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 -- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff65406c8f0 msgr2=0x7ff65406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff65406c8f0 0x7ff65406edb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff6780b1f00 tx=0x7ff66800b040 comp rx=0 tx=0).stop 2024-04-24T08:20:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 -- 172.21.15.69:0/4104998311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6780b3320 msgr2=0x7ff678153aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6780b3320 0x7ff678153aa0 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7ff670001eb0 tx=0x7ff67000c820 comp rx=0 tx=0).stop 2024-04-24T08:20:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 -- 172.21.15.69:0/4104998311 shutdown_connections 2024-04-24T08:20:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff65406c8f0 0x7ff65406edb0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6780b3320 0x7ff678153aa0 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff678071d70 0x7ff678136b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 --2- 172.21.15.69:0/4104998311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67806f730 0x7ff67813a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 -- 172.21.15.69:0/4104998311 >> 172.21.15.69:0/4104998311 conn(0x7ff678067d90 msgr2=0x7ff67806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 -- 172.21.15.69:0/4104998311 shutdown_connections 2024-04-24T08:20:00.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.656+0000 7ff67e800640 1 -- 172.21.15.69:0/4104998311 wait complete. 2024-04-24T08:20:00.676 INFO:tasks.cephfs_test_runner:test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-04-24T08:20:00.676 INFO:tasks.cephfs_test_runner:Starting test: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-04-24T08:20:00.677 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:00.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.868+0000 7ff59a800640 1 Processor -- start 2024-04-24T08:20:00.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.868+0000 7ff59a800640 1 -- start start 2024-04-24T08:20:00.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.869+0000 7ff59a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff59415b490 0x7ff5940b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.869+0000 7ff59a800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5940b46e0 con 0x7ff59406f2d0 2024-04-24T08:20:00.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.869+0000 7ff59a800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5940d0960 con 0x7ff5940d02f0 2024-04-24T08:20:00.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.869+0000 7ff59a800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff5940d0720 con 0x7ff59415b490 2024-04-24T08:20:00.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.870+0000 7ff593400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff59415b490 0x7ff5940b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:00.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.870+0000 7ff58aa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff59406f2d0 0x7ff5940cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56270/0 (socket says 172.21.15.69:56270) 2024-04-24T08:20:00.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.870+0000 7ff58aa00640 1 -- 172.21.15.69:0/561529166 learned_addr learned my addr 172.21.15.69:0/561529166 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.870+0000 7ff593400640 1 -- 172.21.15.69:0/561529166 >> v1:172.21.15.121:6789/0 conn(0x7ff5940d02f0 legacy=0x7ff5940ce0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.870+0000 7ff593400640 1 -- 172.21.15.69:0/561529166 >> v1:172.21.15.69:6789/0 conn(0x7ff59406f2d0 legacy=0x7ff5940cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.870+0000 7ff593400640 1 -- 172.21.15.69:0/561529166 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5940b4980 con 0x7ff59415b490 2024-04-24T08:20:00.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.871+0000 7ff593400640 1 --2- 172.21.15.69:0/561529166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59415b490 0x7ff5940b5180 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7ff5840097a0 tx=0x7ff58402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1bb094a960bf6a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:00.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.871+0000 7ff592a00640 1 -- 172.21.15.69:0/561529166 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584037070 con 0x7ff59415b490 2024-04-24T08:20:00.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.871+0000 7ff592a00640 1 -- 172.21.15.69:0/561529166 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff584032440 con 0x7ff59415b490 2024-04-24T08:20:00.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.871+0000 7ff592a00640 1 -- 172.21.15.69:0/561529166 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584032810 con 0x7ff59415b490 2024-04-24T08:20:00.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.872+0000 7ff59a800640 1 -- 172.21.15.69:0/561529166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59415b490 msgr2=0x7ff5940b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.872+0000 7ff59a800640 1 --2- 172.21.15.69:0/561529166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59415b490 0x7ff5940b5180 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7ff5840097a0 tx=0x7ff58402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.872+0000 7ff59a800640 1 -- 172.21.15.69:0/561529166 shutdown_connections 2024-04-24T08:20:00.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.872+0000 7ff59a800640 1 --2- 172.21.15.69:0/561529166 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59415b490 0x7ff5940b5180 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.872+0000 7ff59a800640 1 -- 172.21.15.69:0/561529166 >> 172.21.15.69:0/561529166 conn(0x7ff594067d90 msgr2=0x7ff59406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:00.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.872+0000 7ff59a800640 1 -- 172.21.15.69:0/561529166 shutdown_connections 2024-04-24T08:20:00.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.872+0000 7ff59a800640 1 -- 172.21.15.69:0/561529166 wait complete. 2024-04-24T08:20:00.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.874+0000 7ff59a800640 1 Processor -- start 2024-04-24T08:20:00.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.874+0000 7ff59a800640 1 -- start start 2024-04-24T08:20:00.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.875+0000 7ff59a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59406f2d0 0x7ff59414f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.875+0000 7ff59a800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5940d02f0 0x7ff59414f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.875+0000 7ff59a800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff59415b490 0x7ff5941511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.875+0000 7ff59a800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff59415b870 con 0x7ff59406f2d0 2024-04-24T08:20:00.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.875+0000 7ff59a800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff5940d0c40 con 0x7ff5940d02f0 2024-04-24T08:20:00.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.875+0000 7ff59a800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff5940b56c0 con 0x7ff59415b490 2024-04-24T08:20:00.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.875+0000 7ff593400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59406f2d0 0x7ff59414f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:00.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.875+0000 7ff593400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59406f2d0 0x7ff59414f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59448/0 (socket says 172.21.15.69:59448) 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.875+0000 7ff593400640 1 -- 172.21.15.69:0/2283330172 learned_addr learned my addr 172.21.15.69:0/2283330172 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff58aa00640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5940d02f0 0x7ff59414f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff593400640 1 -- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff59415b490 msgr2=0x7ff5941511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff593400640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff59415b490 0x7ff5941511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff593400640 1 -- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5940d02f0 msgr2=0x7ff59414f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff593400640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5940d02f0 0x7ff59414f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff593400640 1 -- 172.21.15.69:0/2283330172 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59414f550 con 0x7ff59406f2d0 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff58aa00640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5940d02f0 0x7ff59414f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff593400640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59406f2d0 0x7ff59414f010 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7ff5840098c0 tx=0x7ff5840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff58be00640 1 -- 172.21.15.69:0/2283330172 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584041070 con 0x7ff59406f2d0 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff58be00640 1 -- 172.21.15.69:0/2283330172 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff58403c930 con 0x7ff59406f2d0 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff58be00640 1 -- 172.21.15.69:0/2283330172 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584037040 con 0x7ff59406f2d0 2024-04-24T08:20:00.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.876+0000 7ff59a800640 1 -- 172.21.15.69:0/2283330172 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5941503f0 con 0x7ff59406f2d0 2024-04-24T08:20:00.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.877+0000 7ff59a800640 1 -- 172.21.15.69:0/2283330172 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5941509b0 con 0x7ff59406f2d0 2024-04-24T08:20:00.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.878+0000 7ff59a800640 1 -- 172.21.15.69:0/2283330172 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5940ccc30 con 0x7ff59406f2d0 2024-04-24T08:20:00.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.878+0000 7ff58be00640 1 -- 172.21.15.69:0/2283330172 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff584044070 con 0x7ff59406f2d0 2024-04-24T08:20:00.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.879+0000 7ff58be00640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff57006c8f0 0x7ff57006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:00.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.879+0000 7ff58be00640 1 -- 172.21.15.69:0/2283330172 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5029+0+0 (secure 0 0 0) 0x7ff5840ac4c0 con 0x7ff59406f2d0 2024-04-24T08:20:00.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.880+0000 7ff58aa00640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff57006c8f0 0x7ff57006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:00.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.881+0000 7ff58aa00640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff57006c8f0 0x7ff57006edb0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff5941f8010 tx=0x7ff57c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:00.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:00.884+0000 7ff58be00640 1 -- 172.21.15.69:0/2283330172 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff5940ccc30 con 0x7ff59406f2d0 2024-04-24T08:20:01.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.084+0000 7ff59a800640 1 -- 172.21.15.69:0/2283330172 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff594068a80 con 0x7ff59406f2d0 2024-04-24T08:20:01.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.086+0000 7ff58be00640 1 -- 172.21.15.69:0/2283330172 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+13126 (secure 0 0 0) 0x7ff59414f550 con 0x7ff59406f2d0 2024-04-24T08:20:01.088 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:01.088 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":101,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:19:59.641031+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/3642138906":"2024-04-25T08:19:49.975039+0000","172.21.15.69:6833/3642138906":"2024-04-25T08:19:49.975039+0000","172.21.15.69:6835/412016900":"2024-04-25T08:19:12.588993+0000","172.21.15.69:6833/412016900":"2024-04-25T08:19:12.588993+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 -- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff57006c8f0 msgr2=0x7ff57006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff57006c8f0 0x7ff57006edb0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff5941f8010 tx=0x7ff57c009040 comp rx=0 tx=0).stop 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 -- 172.21.15.69:0/2283330172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59406f2d0 msgr2=0x7ff59414f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59406f2d0 0x7ff59414f010 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7ff5840098c0 tx=0x7ff5840046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 -- 172.21.15.69:0/2283330172 shutdown_connections 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff57006c8f0 0x7ff57006edb0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff59415b490 0x7ff5941511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5940d02f0 0x7ff59414f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 --2- 172.21.15.69:0/2283330172 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff59406f2d0 0x7ff59414f010 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 -- 172.21.15.69:0/2283330172 >> 172.21.15.69:0/2283330172 conn(0x7ff594067d90 msgr2=0x7ff5940ce0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 -- 172.21.15.69:0/2283330172 shutdown_connections 2024-04-24T08:20:01.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.089+0000 7ff59a800640 1 -- 172.21.15.69:0/2283330172 wait complete. 2024-04-24T08:20:01.109 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:20:01.109 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:20:01.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.308+0000 7f02a4a00640 1 Processor -- start 2024-04-24T08:20:01.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.308+0000 7f02a4a00640 1 -- start start 2024-04-24T08:20:01.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.309+0000 7f02a4a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f02a015ea60 0x7f02a00b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.309+0000 7f02a4a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02a00cdf30 0x7f02a00d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f02a4a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f02a00ce390 0x7f02a00d0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f02a4a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f02a00ce910 con 0x7f02a015ea60 2024-04-24T08:20:01.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f02a4a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f02a00cebf0 con 0x7f02a00ce390 2024-04-24T08:20:01.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f02a4a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f02a00ceed0 con 0x7f02a00cdf30 2024-04-24T08:20:01.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f029e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f02a015ea60 0x7f02a00b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f029e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f02a015ea60 0x7f02a00b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59456/0 (socket says 172.21.15.69:59456) 2024-04-24T08:20:01.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f029e800640 1 -- 172.21.15.69:0/741929144 learned_addr learned my addr 172.21.15.69:0/741929144 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:01.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f029e800640 1 -- 172.21.15.69:0/741929144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02a00cdf30 msgr2=0x7f02a00d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:01.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f029e800640 1 --2- 172.21.15.69:0/741929144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02a00cdf30 0x7f02a00d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f029e800640 1 -- 172.21.15.69:0/741929144 >> v2:172.21.15.121:3300/0 conn(0x7f02a00ce390 msgr2=0x7f02a00d0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:01.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f029e800640 1 --2- 172.21.15.69:0/741929144 >> v2:172.21.15.121:3300/0 conn(0x7f02a00ce390 0x7f02a00d0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.310+0000 7f029e800640 1 -- 172.21.15.69:0/741929144 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a00cf1b0 con 0x7f02a015ea60 2024-04-24T08:20:01.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.311+0000 7f029e800640 1 --2- 172.21.15.69:0/741929144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 0x7f02a00b5300 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f02a00cf800 tx=0x7f029002c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae53bf133f26326 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:01.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.312+0000 7f029de00640 1 -- 172.21.15.69:0/741929144 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0290032700 con 0x7f02a015ea60 2024-04-24T08:20:01.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.312+0000 7f029de00640 1 -- 172.21.15.69:0/741929144 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02900328a0 con 0x7f02a015ea60 2024-04-24T08:20:01.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.312+0000 7f029de00640 1 -- 172.21.15.69:0/741929144 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0290032c70 con 0x7f02a015ea60 2024-04-24T08:20:01.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.312+0000 7f02a4a00640 1 -- 172.21.15.69:0/741929144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 msgr2=0x7f02a00b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.312+0000 7f02a4a00640 1 --2- 172.21.15.69:0/741929144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 0x7f02a00b5300 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f02a00cf800 tx=0x7f029002c800 comp rx=0 tx=0).stop 2024-04-24T08:20:01.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.312+0000 7f02a4a00640 1 -- 172.21.15.69:0/741929144 shutdown_connections 2024-04-24T08:20:01.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.312+0000 7f02a4a00640 1 --2- 172.21.15.69:0/741929144 >> v2:172.21.15.121:3300/0 conn(0x7f02a00ce390 0x7f02a00d0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.312+0000 7f02a4a00640 1 --2- 172.21.15.69:0/741929144 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02a00cdf30 0x7f02a00d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.312+0000 7f02a4a00640 1 --2- 172.21.15.69:0/741929144 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 0x7f02a00b5300 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.312+0000 7f02a4a00640 1 -- 172.21.15.69:0/741929144 >> 172.21.15.69:0/741929144 conn(0x7f02a0067b80 msgr2=0x7f02a006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:01.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.313+0000 7f02a4a00640 1 -- 172.21.15.69:0/741929144 shutdown_connections 2024-04-24T08:20:01.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.313+0000 7f02a4a00640 1 -- 172.21.15.69:0/741929144 wait complete. 2024-04-24T08:20:01.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.314+0000 7f02a4a00640 1 Processor -- start 2024-04-24T08:20:01.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.314+0000 7f02a4a00640 1 -- start start 2024-04-24T08:20:01.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.315+0000 7f02a4a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02a00cdf30 0x7f02a014e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.315+0000 7f02a4a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02a00ce390 0x7f02a01e4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.315+0000 7f02a4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 0x7f02a014ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.315+0000 7f02a4a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f02a00b34a0 con 0x7f02a015ea60 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.315+0000 7f02a4a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f02a00b3320 con 0x7f02a00cdf30 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.315+0000 7f02a4a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f02a00b3620 con 0x7f02a00ce390 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.315+0000 7f029f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 0x7f02a014ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.316+0000 7f029f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 0x7f02a014ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59458/0 (socket says 172.21.15.69:59458) 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.316+0000 7f029f200640 1 -- 172.21.15.69:0/2978118652 learned_addr learned my addr 172.21.15.69:0/2978118652 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.316+0000 7f029e800640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02a00cdf30 0x7f02a014e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.316+0000 7f0295e00640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02a00ce390 0x7f02a01e4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.316+0000 7f029f200640 1 -- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02a00ce390 msgr2=0x7f02a01e4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.316+0000 7f029f200640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02a00ce390 0x7f02a01e4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.316+0000 7f029f200640 1 -- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02a00cdf30 msgr2=0x7f02a014e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.316+0000 7f029f200640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02a00cdf30 0x7f02a014e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.316+0000 7f029f200640 1 -- 172.21.15.69:0/2978118652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a014e740 con 0x7f02a015ea60 2024-04-24T08:20:01.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.316+0000 7f029f200640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 0x7f02a014ff40 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f029800b970 tx=0x7f029800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:01.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.317+0000 7f0297400640 1 -- 172.21.15.69:0/2978118652 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298019070 con 0x7f02a015ea60 2024-04-24T08:20:01.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.317+0000 7f0297400640 1 -- 172.21.15.69:0/2978118652 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f029800cb60 con 0x7f02a015ea60 2024-04-24T08:20:01.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.317+0000 7f0297400640 1 -- 172.21.15.69:0/2978118652 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298014440 con 0x7f02a015ea60 2024-04-24T08:20:01.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.317+0000 7f02a4a00640 1 -- 172.21.15.69:0/2978118652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a015ca20 con 0x7f02a015ea60 2024-04-24T08:20:01.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.317+0000 7f02a4a00640 1 -- 172.21.15.69:0/2978118652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a015cfe0 con 0x7f02a015ea60 2024-04-24T08:20:01.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.318+0000 7f02a4a00640 1 -- 172.21.15.69:0/2978118652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02a0075d50 con 0x7f02a015ea60 2024-04-24T08:20:01.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.319+0000 7f0297400640 1 -- 172.21.15.69:0/2978118652 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f029801c070 con 0x7f02a015ea60 2024-04-24T08:20:01.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.320+0000 7f0297400640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f027c06c8f0 0x7f027c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.320+0000 7f0297400640 1 -- 172.21.15.69:0/2978118652 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5029+0+0 (secure 0 0 0) 0x7f029808b5c0 con 0x7f02a015ea60 2024-04-24T08:20:01.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.321+0000 7f029e800640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f027c06c8f0 0x7f027c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.322+0000 7f029e800640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f027c06c8f0 0x7f027c06edb0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f0290037000 tx=0x7f0290008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:01.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.324+0000 7f0297400640 1 -- 172.21.15.69:0/2978118652 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0298059c90 con 0x7f02a015ea60 2024-04-24T08:20:01.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.528+0000 7f02a4a00640 1 -- 172.21.15.69:0/2978118652 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f02a006e630 con 0x7f02a015ea60 2024-04-24T08:20:01.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.529+0000 7f0297400640 1 -- 172.21.15.69:0/2978118652 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) ==== 93+0+10 (secure 0 0 0) 0x7f029808b030 con 0x7f02a015ea60 2024-04-24T08:20:01.530 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:20:01.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 -- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f027c06c8f0 msgr2=0x7f027c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f027c06c8f0 0x7f027c06edb0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f0290037000 tx=0x7f0290008040 comp rx=0 tx=0).stop 2024-04-24T08:20:01.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 -- 172.21.15.69:0/2978118652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 msgr2=0x7f02a014ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 0x7f02a014ff40 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f029800b970 tx=0x7f029800be30 comp rx=0 tx=0).stop 2024-04-24T08:20:01.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 -- 172.21.15.69:0/2978118652 shutdown_connections 2024-04-24T08:20:01.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f027c06c8f0 0x7f027c06edb0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02a015ea60 0x7f02a014ff40 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02a00ce390 0x7f02a01e4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 --2- 172.21.15.69:0/2978118652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02a00cdf30 0x7f02a014e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.534 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 -- 172.21.15.69:0/2978118652 >> 172.21.15.69:0/2978118652 conn(0x7f02a0067b80 msgr2=0x7f02a006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:01.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 -- 172.21.15.69:0/2978118652 shutdown_connections 2024-04-24T08:20:01.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.532+0000 7f02a4a00640 1 -- 172.21.15.69:0/2978118652 wait complete. 2024-04-24T08:20:01.551 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-04-24T08:20:01.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.749+0000 7f3609200640 1 Processor -- start 2024-04-24T08:20:01.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.750+0000 7f3609200640 1 -- start start 2024-04-24T08:20:01.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.751+0000 7f3609200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414c650 0x7f3604071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.751+0000 7f3609200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f36041666e0 0x7f36040721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.751+0000 7f3609200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3604163fb0 con 0x7f360406db90 2024-04-24T08:20:01.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.751+0000 7f3609200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f360406e130 con 0x7f36041666e0 2024-04-24T08:20:01.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.751+0000 7f3609200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f360406e410 con 0x7f360414c650 2024-04-24T08:20:01.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.751+0000 7f3603e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f360406db90 0x7f36040b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56274/0 (socket says 172.21.15.69:56274) 2024-04-24T08:20:01.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.751+0000 7f3603e00640 1 -- 172.21.15.69:0/2229570954 learned_addr learned my addr 172.21.15.69:0/2229570954 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3603400640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414c650 0x7f3604071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f35faa00640 1 --2- 172.21.15.69:0/2229570954 >> v2:172.21.15.121:3300/0 conn(0x7f36041666e0 0x7f36040721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3558043635 0 0) 0x7f3604163fb0 con 0x7f360406db90 2024-04-24T08:20:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35e8003520 con 0x7f360406db90 2024-04-24T08:20:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2691258627 0 0) 0x7f35e8003520 con 0x7f360406db90 2024-04-24T08:20:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414c650 msgr2=0x7f3604071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:01.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414c650 0x7f3604071c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 >> v2:172.21.15.121:3300/0 conn(0x7f36041666e0 msgr2=0x7f36040721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 --2- 172.21.15.69:0/2229570954 >> v2:172.21.15.121:3300/0 conn(0x7f36041666e0 0x7f36040721d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360406e6f0 con 0x7f360406db90 2024-04-24T08:20:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f35fc001a10 con 0x7f360406db90 2024-04-24T08:20:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 >> v1:172.21.15.69:6789/0 conn(0x7f360406db90 legacy=0x7f36040b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e80039b0 0x7f35e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.752+0000 7f3602a00640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414c650 0x7f35e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3602a00640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e8004d40 0x7f35e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f360406e130 con 0x7f35e8004d40 2024-04-24T08:20:01.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f360406e410 con 0x7f35e80039b0 2024-04-24T08:20:01.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f35e800b9c0 con 0x7f360414c650 2024-04-24T08:20:01.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3603e00640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e8004d40 0x7f35e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3603400640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e80039b0 0x7f35e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f35faa00640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414c650 0x7f35e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3603e00640 1 -- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414c650 msgr2=0x7f35e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3603e00640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414c650 0x7f35e8004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3603e00640 1 -- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e80039b0 msgr2=0x7f35e8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3603e00640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e80039b0 0x7f35e8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3603e00640 1 -- 172.21.15.69:0/2229570954 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e8003520 con 0x7f35e8004d40 2024-04-24T08:20:01.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.753+0000 7f3603400640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e80039b0 0x7f35e8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.754+0000 7f3603e00640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e8004d40 0x7f35e80051c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00a880 tx=0x7f35e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a119583389788bd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.755+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc00a4d0 con 0x7f35e8004d40 2024-04-24T08:20:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.755+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35fc02f070 con 0x7f35e8004d40 2024-04-24T08:20:01.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.755+0000 7f3602a00640 1 -- 172.21.15.69:0/2229570954 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc02bcc0 con 0x7f35e8004d40 2024-04-24T08:20:01.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.755+0000 7f3609200640 1 -- 172.21.15.69:0/2229570954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e8004d40 msgr2=0x7f35e80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.755+0000 7f3609200640 1 --2- 172.21.15.69:0/2229570954 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35e8004d40 0x7f35e80051c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f35fc00a880 tx=0x7f35e8005830 comp rx=0 tx=0).stop 2024-04-24T08:20:01.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.755+0000 7f3603e00640 1 -- 172.21.15.69:0/2229570954 reap_dead start 2024-04-24T08:20:01.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.755+0000 7f3609200640 1 -- 172.21.15.69:0/2229570954 shutdown_connections 2024-04-24T08:20:01.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.755+0000 7f3609200640 1 -- 172.21.15.69:0/2229570954 >> 172.21.15.69:0/2229570954 conn(0x7f36040d0160 msgr2=0x7f36040d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:01.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.755+0000 7f3609200640 1 -- 172.21.15.69:0/2229570954 shutdown_connections 2024-04-24T08:20:01.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.755+0000 7f3609200640 1 -- 172.21.15.69:0/2229570954 wait complete. 2024-04-24T08:20:01.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.757+0000 7f3609200640 1 Processor -- start 2024-04-24T08:20:01.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.757+0000 7f3609200640 1 -- start start 2024-04-24T08:20:01.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.758+0000 7f3609200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e8004d40 0x7f36041f84a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.758+0000 7f3609200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414b6c0 0x7f360415c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.758+0000 7f3609200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040cc8b0 0x7f36040ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.758+0000 7f3609200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f360414ca30 con 0x7f36040cc8b0 2024-04-24T08:20:01.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.758+0000 7f3609200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f360406ecb0 con 0x7f35e8004d40 2024-04-24T08:20:01.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.758+0000 7f3609200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3604072830 con 0x7f360414b6c0 2024-04-24T08:20:01.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.758+0000 7f3603e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040cc8b0 0x7f36040ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.758+0000 7f3603e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040cc8b0 0x7f36040ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59474/0 (socket says 172.21.15.69:59474) 2024-04-24T08:20:01.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.758+0000 7f3603e00640 1 -- 172.21.15.69:0/3741344012 learned_addr learned my addr 172.21.15.69:0/3741344012 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:01.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.759+0000 7f3603400640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e8004d40 0x7f36041f84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.759+0000 7f35faa00640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414b6c0 0x7f360415c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.759+0000 7f3603e00640 1 -- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414b6c0 msgr2=0x7f360415c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.759+0000 7f3603e00640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414b6c0 0x7f360415c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.759+0000 7f3603e00640 1 -- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e8004d40 msgr2=0x7f36041f84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:01.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.759+0000 7f3603e00640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e8004d40 0x7f36041f84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:01.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.759+0000 7f3603e00640 1 -- 172.21.15.69:0/3741344012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360415ef40 con 0x7f36040cc8b0 2024-04-24T08:20:01.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.759+0000 7f3603e00640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040cc8b0 0x7f36040ccd30 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f35fc02b530 tx=0x7f360406e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:01.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.760+0000 7f35fbe00640 1 -- 172.21.15.69:0/3741344012 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc00a4d0 con 0x7f36040cc8b0 2024-04-24T08:20:01.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.760+0000 7f35fbe00640 1 -- 172.21.15.69:0/3741344012 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35fc02bcc0 con 0x7f36040cc8b0 2024-04-24T08:20:01.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.760+0000 7f35fbe00640 1 -- 172.21.15.69:0/3741344012 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc002020 con 0x7f36040cc8b0 2024-04-24T08:20:01.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.760+0000 7f3609200640 1 -- 172.21.15.69:0/3741344012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f360415f1e0 con 0x7f36040cc8b0 2024-04-24T08:20:01.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.760+0000 7f3609200640 1 -- 172.21.15.69:0/3741344012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f360415f7a0 con 0x7f36040cc8b0 2024-04-24T08:20:01.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.761+0000 7f3609200640 1 -- 172.21.15.69:0/3741344012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3604075d50 con 0x7f36040cc8b0 2024-04-24T08:20:01.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.762+0000 7f35fbe00640 1 -- 172.21.15.69:0/3741344012 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f35fc03e030 con 0x7f36040cc8b0 2024-04-24T08:20:01.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.763+0000 7f35fbe00640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35e006c920 0x7f35e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:01.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.763+0000 7f35fbe00640 1 -- 172.21.15.69:0/3741344012 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5029+0+0 (secure 0 0 0) 0x7f35fc0aaf00 con 0x7f36040cc8b0 2024-04-24T08:20:01.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.763+0000 7f3603400640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35e006c920 0x7f35e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:01.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.764+0000 7f3603400640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35e006c920 0x7f35e006ede0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009aa0 tx=0x7f35f4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:01.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:01.767+0000 7f35fbe00640 1 -- 172.21.15.69:0/3741344012 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f35fc0795c0 con 0x7f36040cc8b0 2024-04-24T08:20:02.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.037+0000 7f3609200640 1 -- 172.21.15.69:0/3741344012 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) -- 0x7f36041f89e0 con 0x7f36040cc8b0 2024-04-24T08:20:02.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.665+0000 7f35fbe00640 1 -- 172.21.15.69:0/3741344012 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v538) ==== 147+0+0 (secure 0 0 0) 0x7f35fc078ec0 con 0x7f36040cc8b0 2024-04-24T08:20:02.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.668+0000 7f3609200640 1 -- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35e006c920 msgr2=0x7f35e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:02.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.668+0000 7f3609200640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35e006c920 0x7f35e006ede0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f35f4009aa0 tx=0x7f35f4008040 comp rx=0 tx=0).stop 2024-04-24T08:20:02.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.668+0000 7f3609200640 1 -- 172.21.15.69:0/3741344012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040cc8b0 msgr2=0x7f36040ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:02.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.668+0000 7f3609200640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040cc8b0 0x7f36040ccd30 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f35fc02b530 tx=0x7f360406e8c0 comp rx=0 tx=0).stop 2024-04-24T08:20:02.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.668+0000 7f3609200640 1 -- 172.21.15.69:0/3741344012 shutdown_connections 2024-04-24T08:20:02.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.668+0000 7f3609200640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f35e006c920 0x7f35e006ede0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:02.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.668+0000 7f3609200640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36040cc8b0 0x7f36040ccd30 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:02.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.668+0000 7f3609200640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f360414b6c0 0x7f360415c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:02.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.668+0000 7f3609200640 1 --2- 172.21.15.69:0/3741344012 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e8004d40 0x7f36041f84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:02.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.668+0000 7f3609200640 1 -- 172.21.15.69:0/3741344012 >> 172.21.15.69:0/3741344012 conn(0x7f36040d0160 msgr2=0x7f3604074b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:02.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.669+0000 7f3609200640 1 -- 172.21.15.69:0/3741344012 shutdown_connections 2024-04-24T08:20:02.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.669+0000 7f3609200640 1 -- 172.21.15.69:0/3741344012 wait complete. 2024-04-24T08:20:02.692 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-24T08:20:02.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.895+0000 7fd99dc00640 1 Processor -- start 2024-04-24T08:20:02.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.895+0000 7fd99dc00640 1 -- start start 2024-04-24T08:20:02.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.897+0000 7fd99dc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9980d1860 0x7fd998169900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:02.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.897+0000 7fd99dc00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9980d2710 con 0x7fd99815ec30 2024-04-24T08:20:02.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.897+0000 7fd99dc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9980d29f0 con 0x7fd99815e850 2024-04-24T08:20:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.897+0000 7fd99dc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd9980d37b0 con 0x7fd9980d1860 2024-04-24T08:20:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.897+0000 7fd98e000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd99815ec30 0x7fd9980d1150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56290/0 (socket says 172.21.15.69:56290) 2024-04-24T08:20:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.897+0000 7fd98e000640 1 -- 172.21.15.69:0/1311016178 learned_addr learned my addr 172.21.15.69:0/1311016178 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.897+0000 7fd997400640 1 --2- 172.21.15.69:0/1311016178 >> v2:172.21.15.69:3300/0 conn(0x7fd9980d1860 0x7fd998169900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.897+0000 7fd996000640 1 -- 172.21.15.69:0/1311016178 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409490496 0 0) 0x7fd9980d2710 con 0x7fd99815ec30 2024-04-24T08:20:02.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.897+0000 7fd996000640 1 -- 172.21.15.69:0/1311016178 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd97c003520 con 0x7fd99815ec30 2024-04-24T08:20:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.897+0000 7fd997400640 1 -- 172.21.15.69:0/1311016178 >> v1:172.21.15.121:6789/0 conn(0x7fd99815e850 legacy=0x7fd9980cff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.898+0000 7fd997400640 1 -- 172.21.15.69:0/1311016178 >> v1:172.21.15.69:6789/0 conn(0x7fd99815ec30 legacy=0x7fd9980d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:02.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.898+0000 7fd997400640 1 -- 172.21.15.69:0/1311016178 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9980d2cd0 con 0x7fd9980d1860 2024-04-24T08:20:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.899+0000 7fd997400640 1 --2- 172.21.15.69:0/1311016178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 0x7fd998169900 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fd9900097a0 tx=0x7fd99002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6040ddd4844587e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.899+0000 7fd996000640 1 -- 172.21.15.69:0/1311016178 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd990037070 con 0x7fd9980d1860 2024-04-24T08:20:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.899+0000 7fd996000640 1 -- 172.21.15.69:0/1311016178 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd99002ae50 con 0x7fd9980d1860 2024-04-24T08:20:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.899+0000 7fd996000640 1 -- 172.21.15.69:0/1311016178 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd990032660 con 0x7fd9980d1860 2024-04-24T08:20:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.899+0000 7fd99dc00640 1 -- 172.21.15.69:0/1311016178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 msgr2=0x7fd998169900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.899+0000 7fd99dc00640 1 --2- 172.21.15.69:0/1311016178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 0x7fd998169900 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fd9900097a0 tx=0x7fd99002a140 comp rx=0 tx=0).stop 2024-04-24T08:20:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.900+0000 7fd99dc00640 1 -- 172.21.15.69:0/1311016178 shutdown_connections 2024-04-24T08:20:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.900+0000 7fd99dc00640 1 --2- 172.21.15.69:0/1311016178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 0x7fd998169900 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.900+0000 7fd99dc00640 1 -- 172.21.15.69:0/1311016178 >> 172.21.15.69:0/1311016178 conn(0x7fd99808bf60 msgr2=0x7fd9980d4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.900+0000 7fd99dc00640 1 -- 172.21.15.69:0/1311016178 shutdown_connections 2024-04-24T08:20:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.900+0000 7fd99dc00640 1 -- 172.21.15.69:0/1311016178 wait complete. 2024-04-24T08:20:02.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.901+0000 7fd99dc00640 1 Processor -- start 2024-04-24T08:20:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.901+0000 7fd99dc00640 1 -- start start 2024-04-24T08:20:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.902+0000 7fd99dc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 0x7fd9980caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.902+0000 7fd99dc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd99815e850 0x7fd998158210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.902+0000 7fd99dc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd99815ec30 0x7fd998158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.902+0000 7fd99dc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9980d3290 con 0x7fd9980d1860 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.902+0000 7fd99dc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd99806eba0 con 0x7fd99815e850 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.902+0000 7fd99dc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9980d0620 con 0x7fd99815ec30 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.902+0000 7fd996a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 0x7fd9980caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd996a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 0x7fd9980caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59482/0 (socket says 172.21.15.69:59482) 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd996a00640 1 -- 172.21.15.69:0/2248327547 learned_addr learned my addr 172.21.15.69:0/2248327547 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd996a00640 1 -- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd99815ec30 msgr2=0x7fd998158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd996a00640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd99815ec30 0x7fd998158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd996a00640 1 -- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd99815e850 msgr2=0x7fd998158210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd996a00640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd99815e850 0x7fd998158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd996a00640 1 -- 172.21.15.69:0/2248327547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9980c9700 con 0x7fd9980d1860 2024-04-24T08:20:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd996a00640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 0x7fd9980caa80 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd98800c7c0 tx=0x7fd98800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd98f400640 1 -- 172.21.15.69:0/2248327547 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988016020 con 0x7fd9980d1860 2024-04-24T08:20:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd98f400640 1 -- 172.21.15.69:0/2248327547 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98800aa50 con 0x7fd9980d1860 2024-04-24T08:20:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.903+0000 7fd98f400640 1 -- 172.21.15.69:0/2248327547 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988012420 con 0x7fd9980d1860 2024-04-24T08:20:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.904+0000 7fd99dc00640 1 -- 172.21.15.69:0/2248327547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9980c99e0 con 0x7fd9980d1860 2024-04-24T08:20:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.904+0000 7fd99dc00640 1 -- 172.21.15.69:0/2248327547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9980c9fa0 con 0x7fd9980d1860 2024-04-24T08:20:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.905+0000 7fd99dc00640 1 -- 172.21.15.69:0/2248327547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd998068a80 con 0x7fd9980d1860 2024-04-24T08:20:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.906+0000 7fd98f400640 1 -- 172.21.15.69:0/2248327547 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd988019020 con 0x7fd9980d1860 2024-04-24T08:20:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.907+0000 7fd98f400640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd97406c8f0 0x7fd97406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.907+0000 7fd98f400640 1 -- 172.21.15.69:0/2248327547 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5029+0+0 (secure 0 0 0) 0x7fd98808b6a0 con 0x7fd9980d1860 2024-04-24T08:20:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.907+0000 7fd98e000640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd97406c8f0 0x7fd97406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:02.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.908+0000 7fd98e000640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd97406c8f0 0x7fd97406edb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd980000f60 tx=0x7fd9800078f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:02.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:02.911+0000 7fd98f400640 1 -- 172.21.15.69:0/2248327547 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd988020070 con 0x7fd9980d1860 2024-04-24T08:20:03.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.109+0000 7fd99dc00640 1 -- 172.21.15.69:0/2248327547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fd998161da0 con 0x7fd9980d1860 2024-04-24T08:20:03.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.117+0000 7fd98f400640 1 -- 172.21.15.69:0/2248327547 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) ==== 122+0+0 (secure 0 0 0) 0x7fd988059cd0 con 0x7fd9980d1860 2024-04-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 -- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd97406c8f0 msgr2=0x7fd97406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd97406c8f0 0x7fd97406edb0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd980000f60 tx=0x7fd9800078f0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 -- 172.21.15.69:0/2248327547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 msgr2=0x7fd9980caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 0x7fd9980caa80 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd98800c7c0 tx=0x7fd98800cc80 comp rx=0 tx=0).stop 2024-04-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 -- 172.21.15.69:0/2248327547 shutdown_connections 2024-04-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd97406c8f0 0x7fd97406edb0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd99815ec30 0x7fd998158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd99815e850 0x7fd998158210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 --2- 172.21.15.69:0/2248327547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9980d1860 0x7fd9980caa80 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 -- 172.21.15.69:0/2248327547 >> 172.21.15.69:0/2248327547 conn(0x7fd99808bf60 msgr2=0x7fd998161300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:03.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 -- 172.21.15.69:0/2248327547 shutdown_connections 2024-04-24T08:20:03.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.120+0000 7fd99dc00640 1 -- 172.21.15.69:0/2248327547 wait complete. 2024-04-24T08:20:03.140 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:20:03.140 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:20:03.244 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:20:03.355 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:20:03.355 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:20:03.355 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:20:03.413 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:20:03.413 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:20:03.413 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:20:03.468 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:20:03.468 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:20:03.468 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:20:03.527 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:20:03.527 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:20:03.527 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:20:03.584 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:20:03.585 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-24T08:20:03.585 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:03.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.790+0000 7efceb200640 1 Processor -- start 2024-04-24T08:20:03.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.791+0000 7efceb200640 1 -- start start 2024-04-24T08:20:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.792+0000 7efceb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec156c60 0x7efcec15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.792+0000 7efceb200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efcec0b30f0 0x7efcec15d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.792+0000 7efceb200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcec15d660 con 0x7efcec0ad0d0 2024-04-24T08:20:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.792+0000 7efceb200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7efcec06dc30 con 0x7efcec0b30f0 2024-04-24T08:20:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.792+0000 7efceb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efcec06df10 con 0x7efcec156c60 2024-04-24T08:20:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.792+0000 7efcea800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7efcec0ad0d0 0x7efcec1600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56298/0 (socket says 172.21.15.69:56298) 2024-04-24T08:20:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.792+0000 7efcea800640 1 -- 172.21.15.69:0/3812450717 learned_addr learned my addr 172.21.15.69:0/3812450717 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:03.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9e00640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec156c60 0x7efcec15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:03.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce3e00640 1 --2- 172.21.15.69:0/3812450717 >> v2:172.21.15.121:3300/0 conn(0x7efcec0b30f0 0x7efcec15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1006323320 0 0) 0x7efcec15d660 con 0x7efcec0ad0d0 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efcd0003520 con 0x7efcec0ad0d0 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 917826745 0 0) 0x7efcd0003520 con 0x7efcec0ad0d0 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec156c60 msgr2=0x7efcec15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec156c60 0x7efcec15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 >> v2:172.21.15.121:3300/0 conn(0x7efcec0b30f0 msgr2=0x7efcec15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 --2- 172.21.15.69:0/3812450717 >> v2:172.21.15.121:3300/0 conn(0x7efcec0b30f0 0x7efcec15d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcec06e1f0 con 0x7efcec0ad0d0 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7efce40019f0 con 0x7efcec0ad0d0 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 >> v1:172.21.15.69:6789/0 conn(0x7efcec0ad0d0 legacy=0x7efcec1600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec156c60 0x7efcd00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcd0003f30 0x7efcd0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0004d40 0x7efcd00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efcec06dc30 con 0x7efcd0004d40 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efcec06df10 con 0x7efcd0003f30 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.793+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efcd000b9c0 con 0x7efcec156c60 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.794+0000 7efcea800640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0004d40 0x7efcd00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.794+0000 7efce9e00640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec156c60 0x7efcd00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.794+0000 7efce3e00640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcd0003f30 0x7efcd0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.794+0000 7efcea800640 1 -- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec156c60 msgr2=0x7efcd00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.794+0000 7efcea800640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec156c60 0x7efcd00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.794+0000 7efcea800640 1 -- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcd0003f30 msgr2=0x7efcd0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.794+0000 7efcea800640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcd0003f30 0x7efcd0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.794+0000 7efcea800640 1 -- 172.21.15.69:0/3812450717 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcd0003520 con 0x7efcd0004d40 2024-04-24T08:20:03.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.795+0000 7efcea800640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0004d40 0x7efcd00051c0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7efce400a8a0 tx=0x7efcd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae67d05f89bfc0b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.796+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce400a4c0 con 0x7efcd0004d40 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.796+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce402f070 con 0x7efcd0004d40 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.796+0000 7efce9400640 1 -- 172.21.15.69:0/3812450717 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce402bd50 con 0x7efcd0004d40 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.796+0000 7efceb200640 1 -- 172.21.15.69:0/3812450717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0004d40 msgr2=0x7efcd00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.796+0000 7efceb200640 1 --2- 172.21.15.69:0/3812450717 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcd0004d40 0x7efcd00051c0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7efce400a8a0 tx=0x7efcd0005830 comp rx=0 tx=0).stop 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.796+0000 7efcea800640 1 -- 172.21.15.69:0/3812450717 reap_dead start 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.796+0000 7efceb200640 1 -- 172.21.15.69:0/3812450717 shutdown_connections 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.796+0000 7efceb200640 1 -- 172.21.15.69:0/3812450717 >> 172.21.15.69:0/3812450717 conn(0x7efcec0d0160 msgr2=0x7efcec0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.796+0000 7efceb200640 1 -- 172.21.15.69:0/3812450717 shutdown_connections 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.796+0000 7efceb200640 1 -- 172.21.15.69:0/3812450717 wait complete. 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.798+0000 7efceb200640 1 Processor -- start 2024-04-24T08:20:03.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.798+0000 7efceb200640 1 -- start start 2024-04-24T08:20:03.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.799+0000 7efceb200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcd0004d40 0x7efcec0c9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:03.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.799+0000 7efceb200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcec155b80 0x7efcec1555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:03.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.799+0000 7efceb200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0c9f80 0x7efcec158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.799+0000 7efceb200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efcec15cac0 con 0x7efcec155b80 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.799+0000 7efceb200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efcec15d940 con 0x7efcd0004d40 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.799+0000 7efceb200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efcec15cc40 con 0x7efcec0c9f80 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.799+0000 7efce3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcec155b80 0x7efcec1555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.799+0000 7efce3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcec155b80 0x7efcec1555d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59514/0 (socket says 172.21.15.69:59514) 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.799+0000 7efce3e00640 1 -- 172.21.15.69:0/176594427 learned_addr learned my addr 172.21.15.69:0/176594427 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.800+0000 7efce3e00640 1 -- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0c9f80 msgr2=0x7efcec158090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.800+0000 7efcea800640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0c9f80 0x7efcec158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.800+0000 7efce9e00640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcd0004d40 0x7efcec0c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.800+0000 7efce3e00640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0c9f80 0x7efcec158090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.800+0000 7efce3e00640 1 -- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcd0004d40 msgr2=0x7efcec0c9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.800+0000 7efce3e00640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcd0004d40 0x7efcec0c9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.800+0000 7efce3e00640 1 -- 172.21.15.69:0/176594427 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcec073400 con 0x7efcec155b80 2024-04-24T08:20:03.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.800+0000 7efce3e00640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcec155b80 0x7efcec1555d0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7efcd4011560 tx=0x7efcd4011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.800+0000 7efce2000640 1 -- 172.21.15.69:0/176594427 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4004220 con 0x7efcec155b80 2024-04-24T08:20:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.800+0000 7efce2000640 1 -- 172.21.15.69:0/176594427 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcd40043c0 con 0x7efcec155b80 2024-04-24T08:20:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.801+0000 7efceb200640 1 -- 172.21.15.69:0/176594427 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcec1f8b00 con 0x7efcec155b80 2024-04-24T08:20:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.801+0000 7efce2000640 1 -- 172.21.15.69:0/176594427 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcd4015870 con 0x7efcec155b80 2024-04-24T08:20:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.801+0000 7efceb200640 1 -- 172.21.15.69:0/176594427 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcec073980 con 0x7efcec155b80 2024-04-24T08:20:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.802+0000 7efceb200640 1 -- 172.21.15.69:0/176594427 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcec075d50 con 0x7efcec155b80 2024-04-24T08:20:03.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.803+0000 7efce2000640 1 -- 172.21.15.69:0/176594427 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efcd4015a10 con 0x7efcec155b80 2024-04-24T08:20:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.803+0000 7efce2000640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efcc806c920 0x7efcc806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.804+0000 7efce2000640 1 -- 172.21.15.69:0/176594427 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5029+0+0 (secure 0 0 0) 0x7efcd408bef0 con 0x7efcec155b80 2024-04-24T08:20:03.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.804+0000 7efce9e00640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efcc806c920 0x7efcc806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:03.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.805+0000 7efce9e00640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efcc806c920 0x7efcc806ede0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009950 tx=0x7efcdc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:03.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:03.808+0000 7efce2000640 1 -- 172.21.15.69:0/176594427 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efcd4019030 con 0x7efcec155b80 2024-04-24T08:20:04.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.047+0000 7efceb200640 1 -- 172.21.15.69:0/176594427 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcec06f9e0 con 0x7efcec155b80 2024-04-24T08:20:04.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.048+0000 7efce2000640 1 -- 172.21.15.69:0/176594427 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+3875 (secure 0 0 0) 0x7efcd405a5b0 con 0x7efcec155b80 2024-04-24T08:20:04.049 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:04.050 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":93,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5549,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/2696425972","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":2696425972},{"type":"v1","addr":"172.21.15.121:6836","nonce":2696425972}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":5552,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1154426276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":1154426276},{"type":"v1","addr":"172.21.15.121:6837","nonce":1154426276}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":78},{"gid":6043,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6834/2705057816","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2705057816},{"type":"v1","addr":"172.21.15.69:6834","nonce":2705057816}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77},{"gid":9111,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3686931568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6833","nonce":3686931568},{"type":"v1","addr":"172.21.15.69:6835","nonce":3686931568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":91}],"filesystems":[]} 2024-04-24T08:20:04.050 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 93 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 -- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efcc806c920 msgr2=0x7efcc806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efcc806c920 0x7efcc806ede0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7efcdc009950 tx=0x7efcdc008040 comp rx=0 tx=0).stop 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 -- 172.21.15.69:0/176594427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcec155b80 msgr2=0x7efcec1555d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcec155b80 0x7efcec1555d0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7efcd4011560 tx=0x7efcd4011a20 comp rx=0 tx=0).stop 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 -- 172.21.15.69:0/176594427 shutdown_connections 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efcc806c920 0x7efcc806ede0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efcec155b80 0x7efcec1555d0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efcec0c9f80 0x7efcec158090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 --2- 172.21.15.69:0/176594427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efcd0004d40 0x7efcec0c9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 -- 172.21.15.69:0/176594427 >> 172.21.15.69:0/176594427 conn(0x7efcec0d0160 msgr2=0x7efcec15a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:04.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 -- 172.21.15.69:0/176594427 shutdown_connections 2024-04-24T08:20:04.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.050+0000 7efceb200640 1 -- 172.21.15.69:0/176594427 wait complete. 2024-04-24T08:20:04.071 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T08:20:04.071 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-24T08:20:04.071 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T08:20:04.071 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:20:04.072 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T08:20:04.072 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-24T08:20:04.072 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T08:20:04.072 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:20:04.072 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T08:20:04.072 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-24T08:20:04.072 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T08:20:04.073 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:20:04.073 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T08:20:04.073 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-24T08:20:04.073 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T08:20:04.073 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:20:04.093 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T08:20:04.093 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T08:20:04.094 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T08:20:04.094 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T08:20:04.095 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T08:20:04.096 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T08:20:04.100 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T08:20:04.100 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T08:20:04.141 INFO:tasks.ceph.mds.c:Started 2024-04-24T08:20:04.141 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:04.146 INFO:tasks.ceph.mds.a:Started 2024-04-24T08:20:04.146 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:04.147 INFO:tasks.ceph.mds.b:Started 2024-04-24T08:20:04.147 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:04.153 INFO:tasks.ceph.mds.d:Started 2024-04-24T08:20:04.154 DEBUG:teuthology.parallel:result is None 2024-04-24T08:20:04.154 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-24T08:20:04.286 INFO:tasks.ceph.mds.a.smithi069.stdout:starting mds.a at 2024-04-24T08:20:04.291 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:20:04.296 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:20:04.297 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2024-04-24T08:20:04.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.322+0000 7fdd9ce00640 1 Processor -- start 2024-04-24T08:20:04.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.323+0000 7fdd9ce00640 1 -- start start 2024-04-24T08:20:04.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd9ce00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd98162150 0x7fdd980d5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd9ce00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 0x7fdd980d1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd9ce00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd980d22d0 con 0x7fdd980cfe40 2024-04-24T08:20:04.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd9ce00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fdd980d2900 con 0x7fdd98162150 2024-04-24T08:20:04.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd9ce00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdd980d03b0 con 0x7fdd980d10f0 2024-04-24T08:20:04.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd8d400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdd980cfe40 0x7fdd980b7550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56320/0 (socket says 172.21.15.69:56320) 2024-04-24T08:20:04.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd8d400640 1 -- 172.21.15.69:0/2655484069 learned_addr learned my addr 172.21.15.69:0/2655484069 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:04.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd96800640 1 --2- 172.21.15.69:0/2655484069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 0x7fdd980d1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd95e00640 1 --2- 172.21.15.69:0/2655484069 >> v2:172.21.15.121:3300/0 conn(0x7fdd98162150 0x7fdd980d5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd95400640 1 -- 172.21.15.69:0/2655484069 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3957506984 0 0) 0x7fdd980d22d0 con 0x7fdd980cfe40 2024-04-24T08:20:04.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd95400640 1 -- 172.21.15.69:0/2655484069 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd7c003520 con 0x7fdd980cfe40 2024-04-24T08:20:04.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd96800640 1 -- 172.21.15.69:0/2655484069 >> v2:172.21.15.121:3300/0 conn(0x7fdd98162150 msgr2=0x7fdd980d5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:04.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.324+0000 7fdd96800640 1 --2- 172.21.15.69:0/2655484069 >> v2:172.21.15.121:3300/0 conn(0x7fdd98162150 0x7fdd980d5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.325+0000 7fdd96800640 1 -- 172.21.15.69:0/2655484069 >> v1:172.21.15.69:6789/0 conn(0x7fdd980cfe40 legacy=0x7fdd980b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.325+0000 7fdd96800640 1 -- 172.21.15.69:0/2655484069 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd980d0690 con 0x7fdd980d10f0 2024-04-24T08:20:04.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.325+0000 7fdd95e00640 1 --2- 172.21.15.69:0/2655484069 >> v2:172.21.15.121:3300/0 conn(0x7fdd98162150 0x7fdd980d5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:04.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.326+0000 7fdd96800640 1 --2- 172.21.15.69:0/2655484069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 0x7fdd980d1580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdd900097a0 tx=0x7fdd9002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=644a64170ebdb84 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:04.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.326+0000 7fdd95400640 1 -- 172.21.15.69:0/2655484069 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90037070 con 0x7fdd980d10f0 2024-04-24T08:20:04.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.326+0000 7fdd95400640 1 -- 172.21.15.69:0/2655484069 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd90032440 con 0x7fdd980d10f0 2024-04-24T08:20:04.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.326+0000 7fdd95400640 1 -- 172.21.15.69:0/2655484069 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90032810 con 0x7fdd980d10f0 2024-04-24T08:20:04.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.326+0000 7fdd9ce00640 1 -- 172.21.15.69:0/2655484069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 msgr2=0x7fdd980d1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.326+0000 7fdd9ce00640 1 --2- 172.21.15.69:0/2655484069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 0x7fdd980d1580 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdd900097a0 tx=0x7fdd9002a180 comp rx=0 tx=0).stop 2024-04-24T08:20:04.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.326+0000 7fdd9ce00640 1 -- 172.21.15.69:0/2655484069 shutdown_connections 2024-04-24T08:20:04.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.326+0000 7fdd9ce00640 1 --2- 172.21.15.69:0/2655484069 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 0x7fdd980d1580 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.326+0000 7fdd9ce00640 1 --2- 172.21.15.69:0/2655484069 >> v2:172.21.15.121:3300/0 conn(0x7fdd98162150 0x7fdd980d5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.326+0000 7fdd9ce00640 1 -- 172.21.15.69:0/2655484069 >> 172.21.15.69:0/2655484069 conn(0x7fdd9808bf60 msgr2=0x7fdd980d4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:04.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.327+0000 7fdd9ce00640 1 -- 172.21.15.69:0/2655484069 shutdown_connections 2024-04-24T08:20:04.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.327+0000 7fdd9ce00640 1 -- 172.21.15.69:0/2655484069 wait complete. 2024-04-24T08:20:04.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.328+0000 7fdd9ce00640 1 Processor -- start 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.328+0000 7fdd9ce00640 1 -- start start 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.329+0000 7fdd9ce00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd980cfe40 0x7fdd98155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.329+0000 7fdd9ce00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 0x7fdd98159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.329+0000 7fdd9ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd98162150 0x7fdd9815a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.329+0000 7fdd9ce00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdd980d5560 con 0x7fdd98162150 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.329+0000 7fdd9ce00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdd980b7c60 con 0x7fdd980cfe40 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.329+0000 7fdd9ce00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdd980d2d90 con 0x7fdd980d10f0 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.329+0000 7fdd96800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd98162150 0x7fdd9815a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.329+0000 7fdd96800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd98162150 0x7fdd9815a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59554/0 (socket says 172.21.15.69:59554) 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.329+0000 7fdd96800640 1 -- 172.21.15.69:0/4213466722 learned_addr learned my addr 172.21.15.69:0/4213466722 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd95e00640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd980cfe40 0x7fdd98155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd8d400640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 0x7fdd98159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd96800640 1 -- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 msgr2=0x7fdd98159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd96800640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 0x7fdd98159ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd96800640 1 -- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd980cfe40 msgr2=0x7fdd98155790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd96800640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd980cfe40 0x7fdd98155790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd96800640 1 -- 172.21.15.69:0/4213466722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd980c8360 con 0x7fdd98162150 2024-04-24T08:20:04.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd96800640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd98162150 0x7fdd9815a530 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fdd90035040 tx=0x7fdd900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:04.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd8ea00640 1 -- 172.21.15.69:0/4213466722 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90041020 con 0x7fdd98162150 2024-04-24T08:20:04.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd8ea00640 1 -- 172.21.15.69:0/4213466722 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9003d640 con 0x7fdd98162150 2024-04-24T08:20:04.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd8ea00640 1 -- 172.21.15.69:0/4213466722 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd90037040 con 0x7fdd98162150 2024-04-24T08:20:04.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.330+0000 7fdd9ce00640 1 -- 172.21.15.69:0/4213466722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd980c8600 con 0x7fdd98162150 2024-04-24T08:20:04.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.331+0000 7fdd9ce00640 1 -- 172.21.15.69:0/4213466722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd980c8bc0 con 0x7fdd98162150 2024-04-24T08:20:04.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.332+0000 7fdd9ce00640 1 -- 172.21.15.69:0/4213466722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd98068a80 con 0x7fdd98162150 2024-04-24T08:20:04.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.333+0000 7fdd8ea00640 1 -- 172.21.15.69:0/4213466722 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdd90044020 con 0x7fdd98162150 2024-04-24T08:20:04.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.334+0000 7fdd8ea00640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd7406c920 0x7fdd7406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.334+0000 7fdd8ea00640 1 -- 172.21.15.69:0/4213466722 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 5029+0+0 (secure 0 0 0) 0x7fdd900ac540 con 0x7fdd98162150 2024-04-24T08:20:04.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.334+0000 7fdd95e00640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd7406c920 0x7fdd7406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.335+0000 7fdd95e00640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd7406c920 0x7fdd7406ede0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fdd9815dbf0 tx=0x7fdd8800b040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:04.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.337+0000 7fdd8ea00640 1 -- 172.21.15.69:0/4213466722 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd9004b030 con 0x7fdd98162150 2024-04-24T08:20:04.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.548+0000 7fdd9ce00640 1 -- 172.21.15.69:0/4213466722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fdd9806e8f0 con 0x7fdd98162150 2024-04-24T08:20:04.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.697+0000 7fdd8ea00640 1 -- 172.21.15.69:0/4213466722 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v103) ==== 97+0+0 (secure 0 0 0) 0x7fdd90030e20 con 0x7fdd98162150 2024-04-24T08:20:04.699 INFO:teuthology.orchestra.run.smithi069.stderr: removed all blocklist entries 2024-04-24T08:20:04.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.700+0000 7fdd9ce00640 1 -- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd7406c920 msgr2=0x7fdd7406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.700+0000 7fdd9ce00640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd7406c920 0x7fdd7406ede0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fdd9815dbf0 tx=0x7fdd8800b040 comp rx=0 tx=0).stop 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.700+0000 7fdd9ce00640 1 -- 172.21.15.69:0/4213466722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd98162150 msgr2=0x7fdd9815a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.700+0000 7fdd9ce00640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd98162150 0x7fdd9815a530 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fdd90035040 tx=0x7fdd900046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.700+0000 7fdd9ce00640 1 -- 172.21.15.69:0/4213466722 shutdown_connections 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.700+0000 7fdd9ce00640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdd7406c920 0x7fdd7406ede0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.700+0000 7fdd9ce00640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd98162150 0x7fdd9815a530 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.700+0000 7fdd9ce00640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd980d10f0 0x7fdd98159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.700+0000 7fdd9ce00640 1 --2- 172.21.15.69:0/4213466722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd980cfe40 0x7fdd98155790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.700+0000 7fdd9ce00640 1 -- 172.21.15.69:0/4213466722 >> 172.21.15.69:0/4213466722 conn(0x7fdd9808bf60 msgr2=0x7fdd981598e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.701+0000 7fdd9ce00640 1 -- 172.21.15.69:0/4213466722 shutdown_connections 2024-04-24T08:20:04.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.701+0000 7fdd9ce00640 1 -- 172.21.15.69:0/4213466722 wait complete. 2024-04-24T08:20:04.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-24T08:20:04.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.912+0000 7fcadf200640 1 Processor -- start 2024-04-24T08:20:04.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.912+0000 7fcadf200640 1 -- start start 2024-04-24T08:20:04.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.913+0000 7fcadf200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae01666e0 0x7fcae00b3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.913+0000 7fcadf200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcae0163fb0 con 0x7fcae00723a0 2024-04-24T08:20:04.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.913+0000 7fcadf200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcae006e130 con 0x7fcae014c650 2024-04-24T08:20:04.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.913+0000 7fcadf200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcae006e410 con 0x7fcae01666e0 2024-04-24T08:20:04.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.914+0000 7fcad7e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcae00723a0 0x7fcae00b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56330/0 (socket says 172.21.15.69:56330) 2024-04-24T08:20:04.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.914+0000 7fcad7e00640 1 -- 172.21.15.69:0/2558354495 learned_addr learned my addr 172.21.15.69:0/2558354495 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:04.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.914+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3602121723 0 0) 0x7fcae0163fb0 con 0x7fcae00723a0 2024-04-24T08:20:04.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.914+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcac4003520 con 0x7fcae00723a0 2024-04-24T08:20:04.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.914+0000 7fcade800640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae01666e0 0x7fcae00b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 752771368 0 0) 0x7fcac4003520 con 0x7fcae00723a0 2024-04-24T08:20:04.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae01666e0 msgr2=0x7fcae00b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae01666e0 0x7fcae00b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 >> v1:172.21.15.121:6789/0 conn(0x7fcae014c650 legacy=0x7fcae0071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae006e6f0 con 0x7fcae00723a0 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fcac8001690 con 0x7fcae00723a0 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 >> v1:172.21.15.69:6789/0 conn(0x7fcae00723a0 legacy=0x7fcae00b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcac40039e0 0x7fcac4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcac4004800 0x7fcac4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcac4005210 0x7fcac400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcae006e410 con 0x7fcac4005210 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcac400c110 con 0x7fcac4004800 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcac400c310 con 0x7fcac40039e0 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcade800640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcac4005210 0x7fcac400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcadde00640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcac40039e0 0x7fcac4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcad7e00640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcac4004800 0x7fcac4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcade800640 1 -- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcac40039e0 msgr2=0x7fcac4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:04.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcade800640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcac40039e0 0x7fcac4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcade800640 1 -- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcac4004800 msgr2=0x7fcac4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:04.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcade800640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcac4004800 0x7fcac4004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.915+0000 7fcade800640 1 -- 172.21.15.69:0/2558354495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac4003520 con 0x7fcac4005210 2024-04-24T08:20:04.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.916+0000 7fcade800640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcac4005210 0x7fcac400b9c0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fcae0166ac0 tx=0x7fcac4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add18af178ae7dcb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:04.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.917+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad8009b20 con 0x7fcac4005210 2024-04-24T08:20:04.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.917+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcad8036070 con 0x7fcac4005210 2024-04-24T08:20:04.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.917+0000 7fcadd400640 1 -- 172.21.15.69:0/2558354495 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad802d070 con 0x7fcac4005210 2024-04-24T08:20:04.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.917+0000 7fcadf200640 1 -- 172.21.15.69:0/2558354495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcac4005210 msgr2=0x7fcac400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.917+0000 7fcadf200640 1 --2- 172.21.15.69:0/2558354495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcac4005210 0x7fcac400b9c0 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fcae0166ac0 tx=0x7fcac4005830 comp rx=0 tx=0).stop 2024-04-24T08:20:04.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.917+0000 7fcade800640 1 -- 172.21.15.69:0/2558354495 reap_dead start 2024-04-24T08:20:04.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.917+0000 7fcadf200640 1 -- 172.21.15.69:0/2558354495 shutdown_connections 2024-04-24T08:20:04.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.917+0000 7fcadf200640 1 -- 172.21.15.69:0/2558354495 >> 172.21.15.69:0/2558354495 conn(0x7fcae00d0160 msgr2=0x7fcae00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:04.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.917+0000 7fcadf200640 1 -- 172.21.15.69:0/2558354495 shutdown_connections 2024-04-24T08:20:04.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.917+0000 7fcadf200640 1 -- 172.21.15.69:0/2558354495 wait complete. 2024-04-24T08:20:04.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.919+0000 7fcadf200640 1 Processor -- start 2024-04-24T08:20:04.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.919+0000 7fcadf200640 1 -- start start 2024-04-24T08:20:04.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.920+0000 7fcadf200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcac4005210 0x7fcae01650e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.920+0000 7fcadf200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae014b480 0x7fcae01ec520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.920+0000 7fcadf200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae01eca60 0x7fcae00b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.920+0000 7fcadf200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcae014ca30 con 0x7fcae014b480 2024-04-24T08:20:04.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.920+0000 7fcadf200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcae006ecb0 con 0x7fcac4005210 2024-04-24T08:20:04.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.920+0000 7fcadf200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcae0072830 con 0x7fcae01eca60 2024-04-24T08:20:04.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.920+0000 7fcad7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae014b480 0x7fcae01ec520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.920+0000 7fcad7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae014b480 0x7fcae01ec520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59562/0 (socket says 172.21.15.69:59562) 2024-04-24T08:20:04.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.920+0000 7fcad7e00640 1 -- 172.21.15.69:0/3952206770 learned_addr learned my addr 172.21.15.69:0/3952206770 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcad7e00640 1 -- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae01eca60 msgr2=0x7fcae00b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcade800640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae01eca60 0x7fcae00b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcad7e00640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae01eca60 0x7fcae00b0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcad7e00640 1 -- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcac4005210 msgr2=0x7fcae01650e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcad7e00640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcac4005210 0x7fcae01650e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcad7e00640 1 -- 172.21.15.69:0/3952206770 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae01ecf70 con 0x7fcae014b480 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcad7e00640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae014b480 0x7fcae01ec520 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d7a0 tx=0x7fcac800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcad6000640 1 -- 172.21.15.69:0/3952206770 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8019070 con 0x7fcae014b480 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcad6000640 1 -- 172.21.15.69:0/3952206770 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcac80074e0 con 0x7fcae014b480 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcadf200640 1 -- 172.21.15.69:0/3952206770 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae01ed250 con 0x7fcae014b480 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcad6000640 1 -- 172.21.15.69:0/3952206770 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8004a20 con 0x7fcae014b480 2024-04-24T08:20:04.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.921+0000 7fcadf200640 1 -- 172.21.15.69:0/3952206770 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae0165f30 con 0x7fcae014b480 2024-04-24T08:20:04.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.922+0000 7fcadf200640 1 -- 172.21.15.69:0/3952206770 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae01646d0 con 0x7fcae014b480 2024-04-24T08:20:04.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.923+0000 7fcad6000640 1 -- 172.21.15.69:0/3952206770 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcac801c070 con 0x7fcae014b480 2024-04-24T08:20:04.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.925+0000 7fcad6000640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcabc06c8f0 0x7fcabc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:04.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.925+0000 7fcad6000640 1 -- 172.21.15.69:0/3952206770 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4857+0+0 (secure 0 0 0) 0x7fcac808b310 con 0x7fcae014b480 2024-04-24T08:20:04.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.925+0000 7fcadde00640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcabc06c8f0 0x7fcabc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:04.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.927+0000 7fcadde00640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcabc06c8f0 0x7fcabc06edb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fcad0004d00 tx=0x7fcad0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:04.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:04.930+0000 7fcad6000640 1 -- 172.21.15.69:0/3952206770 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcac80599f0 con 0x7fcae014b480 2024-04-24T08:20:05.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.184+0000 7fcadf200640 1 -- 172.21.15.69:0/3952206770 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fcae0075d50 con 0x7fcae014b480 2024-04-24T08:20:05.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.185+0000 7fcad6000640 1 -- 172.21.15.69:0/3952206770 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) ==== 80+0+5304 (secure 0 0 0) 0x7fcac80592f0 con 0x7fcae014b480 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: "auth_dump": [ 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.a", 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmsqMeLRAApLPtHjtq78HzlVcsjfqXOg==", 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:20:05.187 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.b", 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmkyd5ChAAGsJGU6GGMvvGzQnABuDx/g==", 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.188 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.c", 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmS+yLORAAHPCeBtq99IXEVTpcMT3c4A==", 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mds.d", 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShm0xmHFRAA3VfYoqbcV8kWF+4fCZMVLA==", 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:20:05.189 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.0", 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC5vShmMbvcERAANgogiCC/sXf1y0Ykv7ecNQ==", 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.190 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.1", 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC8vShmDD+uLBAAJHXmxEm1cumJ3wjqe6S0ww==", 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.2", 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC/vShmS63kNhAAXec5HVuwo8THKstXiQOL8Q==", 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:20:05.191 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.3", 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDDvShmxcDJBhAA3V3I4C63EHPRUyhzBCZEzw==", 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.4", 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDJvShm1pX9ERAAL2gjv/+M915Cf5OoqBhZVw==", 2024-04-24T08:20:05.192 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.5", 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDMvShmNIxbHRAA23gxiaGvagddVc2mE5DJJw==", 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.193 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.6", 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDPvShmb+x4NhAA6aWmAzU6c0AZomVgWcFwig==", 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "osd.7", 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQDTvShmgcKMBRAA6fkJvNxBjlNhK9CrZ/FMSw==", 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow profile osd", 2024-04-24T08:20:05.194 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile osd", 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.0", 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw==", 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow r", 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.195 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.1", 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmKAIHMhAAuTuFmTHvBteErsG23LMEyQ==", 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.2", 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC1vShmW0MnKRAASzxCAKuApVvfu9kUfK9BJw==", 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:20:05.196 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.3", 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC2vShmfgoaABAAniS9oubv92R7Qe2EITGKxg==", 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow", 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow r", 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow rw", 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow rwx" 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "client.admin", 2024-04-24T08:20:05.197 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQCzvShmLh6yLxAAoH7NhoEAUKcdXp6n+ai2Ug==", 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr": "allow *", 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow *", 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.x", 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShmv4rwIxAA3rp+yy09qrs7HXip8bZ8xA==", 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:20:05.198 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: "entity": "mgr.y", 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: "key": "AQC0vShm2eatGxAAFQkL0GBsujypv4VnmSy5bw==", 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: "caps": { 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: "mds": "allow *", 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: "mon": "allow profile mgr", 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: "osd": "allow *" 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-04-24T08:20:05.199 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:20:05.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 -- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcabc06c8f0 msgr2=0x7fcabc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcabc06c8f0 0x7fcabc06edb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fcad0004d00 tx=0x7fcad0009040 comp rx=0 tx=0).stop 2024-04-24T08:20:05.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 -- 172.21.15.69:0/3952206770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae014b480 msgr2=0x7fcae01ec520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae014b480 0x7fcae01ec520 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d7a0 tx=0x7fcac800dc60 comp rx=0 tx=0).stop 2024-04-24T08:20:05.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 -- 172.21.15.69:0/3952206770 shutdown_connections 2024-04-24T08:20:05.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcabc06c8f0 0x7fcabc06edb0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcae014b480 0x7fcae01ec520 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcae01eca60 0x7fcae00b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 --2- 172.21.15.69:0/3952206770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcac4005210 0x7fcae01650e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 -- 172.21.15.69:0/3952206770 >> 172.21.15.69:0/3952206770 conn(0x7fcae00d0160 msgr2=0x7fcae00b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:05.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 -- 172.21.15.69:0/3952206770 shutdown_connections 2024-04-24T08:20:05.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.188+0000 7fcadf200640 1 -- 172.21.15.69:0/3952206770 wait complete. 2024-04-24T08:20:05.207 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:05.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.393+0000 7fd595600640 1 Processor -- start 2024-04-24T08:20:05.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.394+0000 7fd595600640 1 -- start start 2024-04-24T08:20:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.395+0000 7fd595600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd5901523f0 0x7fd59006db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.395+0000 7fd595600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5900aeac0 con 0x7fd590072e20 2024-04-24T08:20:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.395+0000 7fd595600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5900aeda0 con 0x7fd59015b980 2024-04-24T08:20:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.395+0000 7fd595600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd5900af080 con 0x7fd5901523f0 2024-04-24T08:20:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.395+0000 7fd58fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd5901523f0 0x7fd59006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.395+0000 7fd587400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd590072e20 0x7fd590073260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56346/0 (socket says 172.21.15.69:56346) 2024-04-24T08:20:05.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.395+0000 7fd587400640 1 -- 172.21.15.69:0/387759586 learned_addr learned my addr 172.21.15.69:0/387759586 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.396+0000 7fd58fe00640 1 -- 172.21.15.69:0/387759586 >> v1:172.21.15.121:6789/0 conn(0x7fd59015b980 legacy=0x7fd59015be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.396+0000 7fd58fe00640 1 -- 172.21.15.69:0/387759586 >> v1:172.21.15.69:6789/0 conn(0x7fd590072e20 legacy=0x7fd590073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.396+0000 7fd58f400640 1 -- 172.21.15.69:0/387759586 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1981022273 0 0) 0x7fd5900aeac0 con 0x7fd590072e20 2024-04-24T08:20:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.396+0000 7fd58fe00640 1 -- 172.21.15.69:0/387759586 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5900af360 con 0x7fd5901523f0 2024-04-24T08:20:05.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.397+0000 7fd58fe00640 1 --2- 172.21.15.69:0/387759586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5901523f0 0x7fd59006db90 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fd580009d90 tx=0x7fd58002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c36d816186f0d22 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:05.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.398+0000 7fd58f400640 1 -- 172.21.15.69:0/387759586 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580031600 con 0x7fd5901523f0 2024-04-24T08:20:05.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.398+0000 7fd58f400640 1 -- 172.21.15.69:0/387759586 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5800317a0 con 0x7fd5901523f0 2024-04-24T08:20:05.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.398+0000 7fd58f400640 1 -- 172.21.15.69:0/387759586 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580031b70 con 0x7fd5901523f0 2024-04-24T08:20:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.398+0000 7fd595600640 1 -- 172.21.15.69:0/387759586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5901523f0 msgr2=0x7fd59006db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.398+0000 7fd595600640 1 --2- 172.21.15.69:0/387759586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5901523f0 0x7fd59006db90 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fd580009d90 tx=0x7fd58002a610 comp rx=0 tx=0).stop 2024-04-24T08:20:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.398+0000 7fd595600640 1 -- 172.21.15.69:0/387759586 shutdown_connections 2024-04-24T08:20:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.398+0000 7fd595600640 1 --2- 172.21.15.69:0/387759586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5901523f0 0x7fd59006db90 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.398+0000 7fd595600640 1 -- 172.21.15.69:0/387759586 >> 172.21.15.69:0/387759586 conn(0x7fd5900d0160 msgr2=0x7fd5900d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.398+0000 7fd595600640 1 -- 172.21.15.69:0/387759586 shutdown_connections 2024-04-24T08:20:05.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.399+0000 7fd595600640 1 -- 172.21.15.69:0/387759586 wait complete. 2024-04-24T08:20:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.400+0000 7fd595600640 1 Processor -- start 2024-04-24T08:20:05.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.400+0000 7fd595600640 1 -- start start 2024-04-24T08:20:05.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.401+0000 7fd595600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd590072e20 0x7fd590148580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.401+0000 7fd595600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5901523f0 0x7fd590152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.401+0000 7fd595600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59015b980 0x7fd590158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.401+0000 7fd595600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd590073970 con 0x7fd59015b980 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.401+0000 7fd595600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd5901527d0 con 0x7fd590072e20 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.401+0000 7fd595600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd590073af0 con 0x7fd5901523f0 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd594c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59015b980 0x7fd590158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd594c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59015b980 0x7fd590158090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59580/0 (socket says 172.21.15.69:59580) 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd594c00640 1 -- 172.21.15.69:0/864431003 learned_addr learned my addr 172.21.15.69:0/864431003 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd587400640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5901523f0 0x7fd590152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd58fe00640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd590072e20 0x7fd590148580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd594c00640 1 -- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5901523f0 msgr2=0x7fd590152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd594c00640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5901523f0 0x7fd590152950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd594c00640 1 -- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd590072e20 msgr2=0x7fd590148580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd594c00640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd590072e20 0x7fd590148580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd594c00640 1 -- 172.21.15.69:0/864431003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd590153330 con 0x7fd59015b980 2024-04-24T08:20:05.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.402+0000 7fd594c00640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59015b980 0x7fd590158090 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fd58800b410 tx=0x7fd58800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:05.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.403+0000 7fd58cc00640 1 -- 172.21.15.69:0/864431003 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588015030 con 0x7fd59015b980 2024-04-24T08:20:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.403+0000 7fd58cc00640 1 -- 172.21.15.69:0/864431003 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd588004020 con 0x7fd59015b980 2024-04-24T08:20:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.403+0000 7fd58cc00640 1 -- 172.21.15.69:0/864431003 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588004410 con 0x7fd59015b980 2024-04-24T08:20:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.403+0000 7fd595600640 1 -- 172.21.15.69:0/864431003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd590153610 con 0x7fd59015b980 2024-04-24T08:20:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.403+0000 7fd595600640 1 -- 172.21.15.69:0/864431003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd590153bd0 con 0x7fd59015b980 2024-04-24T08:20:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.404+0000 7fd595600640 1 -- 172.21.15.69:0/864431003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd590075d50 con 0x7fd59015b980 2024-04-24T08:20:05.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.405+0000 7fd58cc00640 1 -- 172.21.15.69:0/864431003 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd588018070 con 0x7fd59015b980 2024-04-24T08:20:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.406+0000 7fd58cc00640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd56c06c8f0 0x7fd56c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.406+0000 7fd58cc00640 1 -- 172.21.15.69:0/864431003 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4857+0+0 (secure 0 0 0) 0x7fd58808b2d0 con 0x7fd59015b980 2024-04-24T08:20:05.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.406+0000 7fd58fe00640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd56c06c8f0 0x7fd56c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.407+0000 7fd58fe00640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd56c06c8f0 0x7fd56c06edb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd580034000 tx=0x7fd58002f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:05.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.410+0000 7fd58cc00640 1 -- 172.21.15.69:0/864431003 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd588059a40 con 0x7fd59015b980 2024-04-24T08:20:05.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.613+0000 7fd595600640 1 -- 172.21.15.69:0/864431003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd590151880 con 0x7fd59015b980 2024-04-24T08:20:05.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.615+0000 7fd58cc00640 1 -- 172.21.15.69:0/864431003 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) ==== 74+0+12869 (secure 0 0 0) 0x7fd58808b0d0 con 0x7fd59015b980 2024-04-24T08:20:05.617 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:05.617 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":105,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:20:04.712451+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:05.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.617+0000 7fd595600640 1 -- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd56c06c8f0 msgr2=0x7fd56c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.617+0000 7fd595600640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd56c06c8f0 0x7fd56c06edb0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd580034000 tx=0x7fd58002f000 comp rx=0 tx=0).stop 2024-04-24T08:20:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.617+0000 7fd595600640 1 -- 172.21.15.69:0/864431003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59015b980 msgr2=0x7fd590158090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.617+0000 7fd595600640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59015b980 0x7fd590158090 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fd58800b410 tx=0x7fd58800b8d0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.617+0000 7fd595600640 1 -- 172.21.15.69:0/864431003 shutdown_connections 2024-04-24T08:20:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.617+0000 7fd595600640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd56c06c8f0 0x7fd56c06edb0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.617+0000 7fd595600640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd59015b980 0x7fd590158090 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.617+0000 7fd595600640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd5901523f0 0x7fd590152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.617+0000 7fd595600640 1 --2- 172.21.15.69:0/864431003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd590072e20 0x7fd590148580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.617+0000 7fd595600640 1 -- 172.21.15.69:0/864431003 >> 172.21.15.69:0/864431003 conn(0x7fd5900d0160 msgr2=0x7fd5900adec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.618+0000 7fd595600640 1 -- 172.21.15.69:0/864431003 shutdown_connections 2024-04-24T08:20:05.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.618+0000 7fd595600640 1 -- 172.21.15.69:0/864431003 wait complete. 2024-04-24T08:20:05.638 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:20:05.638 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:20:05.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.828+0000 7f3152400640 1 Processor -- start 2024-04-24T08:20:05.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.828+0000 7f3152400640 1 -- start start 2024-04-24T08:20:05.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.829+0000 7f3152400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f314c15c820 0x7f314c0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.829+0000 7f3152400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f314c0ce250 0x7f314c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.829+0000 7f3152400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314c15cc00 con 0x7f314c0b46a0 2024-04-24T08:20:05.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.830+0000 7f3152400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f314c06ed30 con 0x7f314c0ce250 2024-04-24T08:20:05.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.830+0000 7f3152400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f314c0b4e10 con 0x7f314c15c820 2024-04-24T08:20:05.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.830+0000 7f314be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f314c0ce250 0x7f314c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.830+0000 7f314aa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f314c0b46a0 0x7f314c06f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56356/0 (socket says 172.21.15.69:56356) 2024-04-24T08:20:05.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.830+0000 7f314aa00640 1 -- 172.21.15.69:0/4288829041 learned_addr learned my addr 172.21.15.69:0/4288829041 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:05.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.830+0000 7f314b400640 1 --2- 172.21.15.69:0/4288829041 >> v2:172.21.15.121:3300/0 conn(0x7f314c15c820 0x7f314c0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.830+0000 7f314be00640 1 -- 172.21.15.69:0/4288829041 >> v2:172.21.15.121:3300/0 conn(0x7f314c15c820 msgr2=0x7f314c0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:05.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.830+0000 7f314be00640 1 --2- 172.21.15.69:0/4288829041 >> v2:172.21.15.121:3300/0 conn(0x7f314c15c820 0x7f314c0b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.830+0000 7f314be00640 1 -- 172.21.15.69:0/4288829041 >> v1:172.21.15.69:6789/0 conn(0x7f314c0b46a0 legacy=0x7f314c06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.831+0000 7f314be00640 1 -- 172.21.15.69:0/4288829041 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c0b50f0 con 0x7f314c0ce250 2024-04-24T08:20:05.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.830+0000 7f314a000640 1 -- 172.21.15.69:0/4288829041 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149620306 0 0) 0x7f314c15cc00 con 0x7f314c0b46a0 2024-04-24T08:20:05.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.832+0000 7f314be00640 1 --2- 172.21.15.69:0/4288829041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ce250 0x7f314c0ce6c0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f314c0b2f60 tx=0x7f314402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7e862a573a9edc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.832+0000 7f314a000640 1 -- 172.21.15.69:0/4288829041 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144007740 con 0x7f314c0ce250 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.832+0000 7f314a000640 1 -- 172.21.15.69:0/4288829041 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31440078e0 con 0x7f314c0ce250 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.832+0000 7f314a000640 1 -- 172.21.15.69:0/4288829041 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3144007cb0 con 0x7f314c0ce250 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.833+0000 7f3152400640 1 -- 172.21.15.69:0/4288829041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ce250 msgr2=0x7f314c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.833+0000 7f3152400640 1 --2- 172.21.15.69:0/4288829041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ce250 0x7f314c0ce6c0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f314c0b2f60 tx=0x7f314402a680 comp rx=0 tx=0).stop 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.833+0000 7f3152400640 1 -- 172.21.15.69:0/4288829041 shutdown_connections 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.833+0000 7f3152400640 1 --2- 172.21.15.69:0/4288829041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ce250 0x7f314c0ce6c0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.833+0000 7f3152400640 1 --2- 172.21.15.69:0/4288829041 >> v2:172.21.15.121:3300/0 conn(0x7f314c15c820 0x7f314c0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.833+0000 7f3152400640 1 -- 172.21.15.69:0/4288829041 >> 172.21.15.69:0/4288829041 conn(0x7f314c0d0350 msgr2=0x7f314c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.833+0000 7f3152400640 1 -- 172.21.15.69:0/4288829041 shutdown_connections 2024-04-24T08:20:05.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.833+0000 7f3152400640 1 -- 172.21.15.69:0/4288829041 wait complete. 2024-04-24T08:20:05.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.835+0000 7f3152400640 1 Processor -- start 2024-04-24T08:20:05.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.835+0000 7f3152400640 1 -- start start 2024-04-24T08:20:05.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f3152400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0b46a0 0x7f314c15f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f3152400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c0ce250 0x7f314c15d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f3152400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f314c15c820 0x7f314c14d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f3152400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f314c0b4b30 con 0x7f314c0b46a0 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f3152400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f314c06f2b0 con 0x7f314c15c820 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f3152400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f314c06fee0 con 0x7f314c0ce250 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f314b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0b46a0 0x7f314c15f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f314b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0b46a0 0x7f314c15f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59596/0 (socket says 172.21.15.69:59596) 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f314b400640 1 -- 172.21.15.69:0/2308508672 learned_addr learned my addr 172.21.15.69:0/2308508672 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f314b400640 1 -- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c0ce250 msgr2=0x7f314c15d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f314b400640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c0ce250 0x7f314c15d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f314b400640 1 -- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f314c15c820 msgr2=0x7f314c14d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f314b400640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f314c15c820 0x7f314c14d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.836+0000 7f314b400640 1 -- 172.21.15.69:0/2308508672 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c1f7ed0 con 0x7f314c0b46a0 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.837+0000 7f314b400640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0b46a0 0x7f314c15f660 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f313c00d800 tx=0x7f313c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.837+0000 7f3143400640 1 -- 172.21.15.69:0/2308508672 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313c00b6c0 con 0x7f314c0b46a0 2024-04-24T08:20:05.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.837+0000 7f3143400640 1 -- 172.21.15.69:0/2308508672 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f313c019070 con 0x7f314c0b46a0 2024-04-24T08:20:05.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.837+0000 7f3143400640 1 -- 172.21.15.69:0/2308508672 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313c00ba80 con 0x7f314c0b46a0 2024-04-24T08:20:05.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.837+0000 7f3152400640 1 -- 172.21.15.69:0/2308508672 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314c1f81b0 con 0x7f314c0b46a0 2024-04-24T08:20:05.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.837+0000 7f3152400640 1 -- 172.21.15.69:0/2308508672 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314c1f8770 con 0x7f314c0b46a0 2024-04-24T08:20:05.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.838+0000 7f3152400640 1 -- 172.21.15.69:0/2308508672 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f314c068a80 con 0x7f314c0b46a0 2024-04-24T08:20:05.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.839+0000 7f3143400640 1 -- 172.21.15.69:0/2308508672 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f313c00bc20 con 0x7f314c0b46a0 2024-04-24T08:20:05.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.840+0000 7f3143400640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f312c06c8f0 0x7f312c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:05.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.840+0000 7f3143400640 1 -- 172.21.15.69:0/2308508672 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4857+0+0 (secure 0 0 0) 0x7f313c08bcb0 con 0x7f314c0b46a0 2024-04-24T08:20:05.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.841+0000 7f314aa00640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f312c06c8f0 0x7f312c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:05.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.842+0000 7f314aa00640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f312c06c8f0 0x7f312c06edb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3138007fd0 tx=0x7f3138007960 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:05.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:05.844+0000 7f3143400640 1 -- 172.21.15.69:0/2308508672 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f313c05a420 con 0x7f314c0b46a0 2024-04-24T08:20:06.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.046+0000 7f3152400640 1 -- 172.21.15.69:0/2308508672 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f314c15bca0 con 0x7f314c0b46a0 2024-04-24T08:20:06.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.047+0000 7f3143400640 1 -- 172.21.15.69:0/2308508672 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) ==== 93+0+10 (secure 0 0 0) 0x7f313c059d20 con 0x7f314c0b46a0 2024-04-24T08:20:06.049 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:20:06.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 -- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f312c06c8f0 msgr2=0x7f312c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f312c06c8f0 0x7f312c06edb0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f3138007fd0 tx=0x7f3138007960 comp rx=0 tx=0).stop 2024-04-24T08:20:06.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 -- 172.21.15.69:0/2308508672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0b46a0 msgr2=0x7f314c15f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0b46a0 0x7f314c15f660 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f313c00d800 tx=0x7f313c00dcc0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 -- 172.21.15.69:0/2308508672 shutdown_connections 2024-04-24T08:20:06.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f312c06c8f0 0x7f312c06edb0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f314c15c820 0x7f314c14d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f314c0ce250 0x7f314c15d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 --2- 172.21.15.69:0/2308508672 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0b46a0 0x7f314c15f660 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 -- 172.21.15.69:0/2308508672 >> 172.21.15.69:0/2308508672 conn(0x7f314c0d0350 msgr2=0x7f314c0cd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:06.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 -- 172.21.15.69:0/2308508672 shutdown_connections 2024-04-24T08:20:06.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.050+0000 7f3152400640 1 -- 172.21.15.69:0/2308508672 wait complete. 2024-04-24T08:20:06.070 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T08:20:06.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.273+0000 7fb4f7000640 1 Processor -- start 2024-04-24T08:20:06.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.273+0000 7fb4f7000640 1 -- start start 2024-04-24T08:20:06.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.275+0000 7fb4f7000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f015ea10 0x7fb4f00d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.275+0000 7fb4f7000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb4f01631a0 0x7fb4f00af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.275+0000 7fb4f7000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4f00adab0 con 0x7fb4f00af7d0 2024-04-24T08:20:06.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.275+0000 7fb4f7000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb4f00afc40 con 0x7fb4f01631a0 2024-04-24T08:20:06.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.275+0000 7fb4f7000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f00add50 con 0x7fb4f015ea10 2024-04-24T08:20:06.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.275+0000 7fb4f5000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb4f00af7d0 0x7fb4f0071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56358/0 (socket says 172.21.15.69:56358) 2024-04-24T08:20:06.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.275+0000 7fb4f5000640 1 -- 172.21.15.69:0/2553127995 learned_addr learned my addr 172.21.15.69:0/2553127995 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.275+0000 7fb4efe00640 1 --2- 172.21.15.69:0/2553127995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f015ea10 0x7fb4f00d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.275+0000 7fb4ef400640 1 --2- 172.21.15.69:0/2553127995 >> v2:172.21.15.121:3300/0 conn(0x7fb4f01631a0 0x7fb4f00af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.276+0000 7fb4eea00640 1 -- 172.21.15.69:0/2553127995 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 652543331 0 0) 0x7fb4f00adab0 con 0x7fb4f00af7d0 2024-04-24T08:20:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.276+0000 7fb4eea00640 1 -- 172.21.15.69:0/2553127995 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb4d4003520 con 0x7fb4f00af7d0 2024-04-24T08:20:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.276+0000 7fb4efe00640 1 -- 172.21.15.69:0/2553127995 >> v2:172.21.15.121:3300/0 conn(0x7fb4f01631a0 msgr2=0x7fb4f00af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.276+0000 7fb4efe00640 1 --2- 172.21.15.69:0/2553127995 >> v2:172.21.15.121:3300/0 conn(0x7fb4f01631a0 0x7fb4f00af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.276+0000 7fb4eea00640 1 -- 172.21.15.69:0/2553127995 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2749682241 0 0) 0x7fb4d4003520 con 0x7fb4f00af7d0 2024-04-24T08:20:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.276+0000 7fb4efe00640 1 -- 172.21.15.69:0/2553127995 >> v1:172.21.15.69:6789/0 conn(0x7fb4f00af7d0 legacy=0x7fb4f0071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.276+0000 7fb4efe00640 1 -- 172.21.15.69:0/2553127995 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f00ae030 con 0x7fb4f015ea10 2024-04-24T08:20:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.277+0000 7fb4efe00640 1 --2- 172.21.15.69:0/2553127995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f015ea10 0x7fb4f00d0830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00097a0 tx=0x7fb4e002a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53d3b1c18f28c304 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:06.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.278+0000 7fb4eea00640 1 -- 172.21.15.69:0/2553127995 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0037070 con 0x7fb4f015ea10 2024-04-24T08:20:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.278+0000 7fb4eea00640 1 -- 172.21.15.69:0/2553127995 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4e002aba0 con 0x7fb4f015ea10 2024-04-24T08:20:06.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.278+0000 7fb4eea00640 1 -- 172.21.15.69:0/2553127995 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4e80019f0 con 0x7fb4f015ea10 2024-04-24T08:20:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.279+0000 7fb4f7000640 1 -- 172.21.15.69:0/2553127995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f015ea10 msgr2=0x7fb4f00d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.279+0000 7fb4f7000640 1 --2- 172.21.15.69:0/2553127995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f015ea10 0x7fb4f00d0830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00097a0 tx=0x7fb4e002a110 comp rx=0 tx=0).stop 2024-04-24T08:20:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.279+0000 7fb4f7000640 1 -- 172.21.15.69:0/2553127995 shutdown_connections 2024-04-24T08:20:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.279+0000 7fb4f7000640 1 --2- 172.21.15.69:0/2553127995 >> v2:172.21.15.121:3300/0 conn(0x7fb4f01631a0 0x7fb4f00af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.279+0000 7fb4f7000640 1 --2- 172.21.15.69:0/2553127995 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f015ea10 0x7fb4f00d0830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.279+0000 7fb4f7000640 1 -- 172.21.15.69:0/2553127995 >> 172.21.15.69:0/2553127995 conn(0x7fb4f0067d90 msgr2=0x7fb4f006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.279+0000 7fb4f7000640 1 -- 172.21.15.69:0/2553127995 shutdown_connections 2024-04-24T08:20:06.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.279+0000 7fb4f7000640 1 -- 172.21.15.69:0/2553127995 wait complete. 2024-04-24T08:20:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.280+0000 7fb4f7000640 1 Processor -- start 2024-04-24T08:20:06.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.281+0000 7fb4f7000640 1 -- start start 2024-04-24T08:20:06.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.281+0000 7fb4f7000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f00af7d0 0x7fb4f00cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.281+0000 7fb4f7000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4f015ea10 0x7fb4f0148cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.281+0000 7fb4f7000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f01631a0 0x7fb4f00b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.281+0000 7fb4f7000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f00ae8d0 con 0x7fb4f015ea10 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.281+0000 7fb4f7000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb4f00ae5f0 con 0x7fb4f00af7d0 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.281+0000 7fb4f7000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb4f00aedf0 con 0x7fb4f01631a0 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.282+0000 7fb4ef400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4f015ea10 0x7fb4f0148cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.282+0000 7fb4ef400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4f015ea10 0x7fb4f0148cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59602/0 (socket says 172.21.15.69:59602) 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.282+0000 7fb4ef400640 1 -- 172.21.15.69:0/2148167020 learned_addr learned my addr 172.21.15.69:0/2148167020 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.282+0000 7fb4ef400640 1 -- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f01631a0 msgr2=0x7fb4f00b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.282+0000 7fb4f5000640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f01631a0 0x7fb4f00b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.282+0000 7fb4efe00640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f00af7d0 0x7fb4f00cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.282+0000 7fb4ef400640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f01631a0 0x7fb4f00b0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.282+0000 7fb4ef400640 1 -- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f00af7d0 msgr2=0x7fb4f00cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.282+0000 7fb4ef400640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f00af7d0 0x7fb4f00cc860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.282+0000 7fb4ef400640 1 -- 172.21.15.69:0/2148167020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4f00ccda0 con 0x7fb4f015ea10 2024-04-24T08:20:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.283+0000 7fb4ef400640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4f015ea10 0x7fb4f0148cf0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00e870 tx=0x7fb4dc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.283+0000 7fb4e7e00640 1 -- 172.21.15.69:0/2148167020 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc017070 con 0x7fb4f015ea10 2024-04-24T08:20:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.283+0000 7fb4e7e00640 1 -- 172.21.15.69:0/2148167020 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4dc00c8e0 con 0x7fb4f015ea10 2024-04-24T08:20:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.283+0000 7fb4e7e00640 1 -- 172.21.15.69:0/2148167020 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc00ccd0 con 0x7fb4f015ea10 2024-04-24T08:20:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.283+0000 7fb4f7000640 1 -- 172.21.15.69:0/2148167020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4f015d220 con 0x7fb4f015ea10 2024-04-24T08:20:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.283+0000 7fb4f7000640 1 -- 172.21.15.69:0/2148167020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4f015a760 con 0x7fb4f015ea10 2024-04-24T08:20:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.284+0000 7fb4f7000640 1 -- 172.21.15.69:0/2148167020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4f015d610 con 0x7fb4f015ea10 2024-04-24T08:20:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.285+0000 7fb4e7e00640 1 -- 172.21.15.69:0/2148167020 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb4dc01a070 con 0x7fb4f015ea10 2024-04-24T08:20:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.286+0000 7fb4e7e00640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb4d006c920 0x7fb4d006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.286+0000 7fb4e7e00640 1 -- 172.21.15.69:0/2148167020 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4857+0+0 (secure 0 0 0) 0x7fb4dc08b620 con 0x7fb4f015ea10 2024-04-24T08:20:06.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.287+0000 7fb4efe00640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb4d006c920 0x7fb4d006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.288+0000 7fb4efe00640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb4d006c920 0x7fb4d006ede0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0034000 tx=0x7fb4e002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:06.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.290+0000 7fb4e7e00640 1 -- 172.21.15.69:0/2148167020 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb4f00ccda0 con 0x7fb4f015ea10 2024-04-24T08:20:06.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.495+0000 7fb4f7000640 1 -- 172.21.15.69:0/2148167020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb4f0068a80 con 0x7fb4f015ea10 2024-04-24T08:20:06.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.496+0000 7fb4e7e00640 1 -- 172.21.15.69:0/2148167020 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v105) ==== 84+0+65 (secure 0 0 0) 0x7fb4dc059d80 con 0x7fb4f015ea10 2024-04-24T08:20:06.498 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:06.498 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-04-24T08:20:06.498 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-04-24T08:20:06.498 INFO:teuthology.orchestra.run.smithi069.stdout: "poolnum": 1, 2024-04-24T08:20:06.498 INFO:teuthology.orchestra.run.smithi069.stdout: "poolname": ".mgr" 2024-04-24T08:20:06.499 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:06.499 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-04-24T08:20:06.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.499+0000 7fb4f7000640 1 -- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb4d006c920 msgr2=0x7fb4d006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.499+0000 7fb4f7000640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb4d006c920 0x7fb4d006ede0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0034000 tx=0x7fb4e002e000 comp rx=0 tx=0).stop 2024-04-24T08:20:06.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.499+0000 7fb4f7000640 1 -- 172.21.15.69:0/2148167020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4f015ea10 msgr2=0x7fb4f0148cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.499+0000 7fb4f7000640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4f015ea10 0x7fb4f0148cf0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00e870 tx=0x7fb4dc00ed30 comp rx=0 tx=0).stop 2024-04-24T08:20:06.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.499+0000 7fb4f7000640 1 -- 172.21.15.69:0/2148167020 shutdown_connections 2024-04-24T08:20:06.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.499+0000 7fb4f7000640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb4d006c920 0x7fb4d006ede0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.499+0000 7fb4f7000640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4f015ea10 0x7fb4f0148cf0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.499+0000 7fb4f7000640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb4f01631a0 0x7fb4f00b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.499+0000 7fb4f7000640 1 --2- 172.21.15.69:0/2148167020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb4f00af7d0 0x7fb4f00cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.499+0000 7fb4f7000640 1 -- 172.21.15.69:0/2148167020 >> 172.21.15.69:0/2148167020 conn(0x7fb4f0067d90 msgr2=0x7fb4f0159040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:06.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.500+0000 7fb4f7000640 1 -- 172.21.15.69:0/2148167020 shutdown_connections 2024-04-24T08:20:06.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.500+0000 7fb4f7000640 1 -- 172.21.15.69:0/2148167020 wait complete. 2024-04-24T08:20:06.520 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T08:20:06.520 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T08:20:06.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.730+0000 7ffbddc00640 1 Processor -- start 2024-04-24T08:20:06.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.730+0000 7ffbddc00640 1 -- start start 2024-04-24T08:20:06.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.731+0000 7ffbddc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 0x7ffbd80d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.731+0000 7ffbddc00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ffbd80b3fb0 0x7ffbd80d0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.732+0000 7ffbddc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd815ca40 con 0x7ffbd815bcd0 2024-04-24T08:20:06.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.732+0000 7ffbddc00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ffbd815c7e0 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.732+0000 7ffbddc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd80b31f0 con 0x7ffbd80aeb00 2024-04-24T08:20:06.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.732+0000 7ffbd6a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 0x7ffbd80d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.732+0000 7ffbd6000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ffbd80b3fb0 0x7ffbd80d0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.732+0000 7ffbd6a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 0x7ffbd80d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:33154/0 (socket says 172.21.15.69:33154) 2024-04-24T08:20:06.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.732+0000 7ffbd6a00640 1 -- 172.21.15.69:0/3583510321 learned_addr learned my addr 172.21.15.69:0/3583510321 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:06.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.733+0000 7ffbd6a00640 1 -- 172.21.15.69:0/3583510321 >> v2:172.21.15.121:3300/0 conn(0x7ffbd80b3fb0 msgr2=0x7ffbd80d0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:06.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.733+0000 7ffbd6a00640 1 --2- 172.21.15.69:0/3583510321 >> v2:172.21.15.121:3300/0 conn(0x7ffbd80b3fb0 0x7ffbd80d0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.733+0000 7ffbd6a00640 1 -- 172.21.15.69:0/3583510321 >> v1:172.21.15.121:6789/0 conn(0x7ffbd815bcd0 legacy=0x7ffbd815c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.733+0000 7ffbd6a00640 1 -- 172.21.15.69:0/3583510321 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd80b34d0 con 0x7ffbd80aeb00 2024-04-24T08:20:06.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.733+0000 7ffbd6000640 1 --2- 172.21.15.69:0/3583510321 >> v2:172.21.15.121:3300/0 conn(0x7ffbd80b3fb0 0x7ffbd80d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:06.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.734+0000 7ffbd6a00640 1 --2- 172.21.15.69:0/3583510321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 0x7ffbd80d07c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009870 tx=0x7ffbc802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aef1447507c86975 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:06.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.735+0000 7ffbd5600640 1 -- 172.21.15.69:0/3583510321 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8037070 con 0x7ffbd80aeb00 2024-04-24T08:20:06.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.735+0000 7ffbd5600640 1 -- 172.21.15.69:0/3583510321 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc8032440 con 0x7ffbd80aeb00 2024-04-24T08:20:06.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.735+0000 7ffbd5600640 1 -- 172.21.15.69:0/3583510321 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8032810 con 0x7ffbd80aeb00 2024-04-24T08:20:06.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.735+0000 7ffbddc00640 1 -- 172.21.15.69:0/3583510321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 msgr2=0x7ffbd80d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.735+0000 7ffbddc00640 1 --2- 172.21.15.69:0/3583510321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 0x7ffbd80d07c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009870 tx=0x7ffbc802a250 comp rx=0 tx=0).stop 2024-04-24T08:20:06.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.735+0000 7ffbddc00640 1 -- 172.21.15.69:0/3583510321 shutdown_connections 2024-04-24T08:20:06.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.735+0000 7ffbddc00640 1 --2- 172.21.15.69:0/3583510321 >> v2:172.21.15.121:3300/0 conn(0x7ffbd80b3fb0 0x7ffbd80d0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.735+0000 7ffbddc00640 1 --2- 172.21.15.69:0/3583510321 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 0x7ffbd80d07c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.735+0000 7ffbddc00640 1 -- 172.21.15.69:0/3583510321 >> 172.21.15.69:0/3583510321 conn(0x7ffbd806be40 msgr2=0x7ffbd806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:06.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.735+0000 7ffbddc00640 1 -- 172.21.15.69:0/3583510321 shutdown_connections 2024-04-24T08:20:06.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.736+0000 7ffbddc00640 1 -- 172.21.15.69:0/3583510321 wait complete. 2024-04-24T08:20:06.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.737+0000 7ffbddc00640 1 Processor -- start 2024-04-24T08:20:06.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.737+0000 7ffbddc00640 1 -- start start 2024-04-24T08:20:06.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.738+0000 7ffbddc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 0x7ffbd815b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.738+0000 7ffbddc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd80b3fb0 0x7ffbd8159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.738+0000 7ffbddc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd815bcd0 0x7ffbd8156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.738+0000 7ffbddc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd80aeee0 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.738+0000 7ffbddc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd80b3a90 con 0x7ffbd815bcd0 2024-04-24T08:20:06.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.738+0000 7ffbddc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd8099890 con 0x7ffbd80aeb00 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd80b3fb0 0x7ffbd8159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd80b3fb0 0x7ffbd8159540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59604/0 (socket says 172.21.15.69:59604) 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd6000640 1 -- 172.21.15.69:0/857264072 learned_addr learned my addr 172.21.15.69:0/857264072 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd6a00640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 0x7ffbd815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd7400640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd815bcd0 0x7ffbd8156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd6000640 1 -- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 msgr2=0x7ffbd815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd6000640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 0x7ffbd815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd6000640 1 -- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd815bcd0 msgr2=0x7ffbd8156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd6000640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd815bcd0 0x7ffbd8156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd6000640 1 -- 172.21.15.69:0/857264072 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd8157350 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd7400640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd815bcd0 0x7ffbd8156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.739+0000 7ffbd6000640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd80b3fb0 0x7ffbd8159540 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000efc0 tx=0x7ffbc0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:06.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.740+0000 7ffbcea00640 1 -- 172.21.15.69:0/857264072 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0018020 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.740+0000 7ffbcea00640 1 -- 172.21.15.69:0/857264072 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc0014680 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.740+0000 7ffbcea00640 1 -- 172.21.15.69:0/857264072 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000ce60 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.740+0000 7ffbddc00640 1 -- 172.21.15.69:0/857264072 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd8099240 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.740+0000 7ffbddc00640 1 -- 172.21.15.69:0/857264072 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd80b1d10 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.741+0000 7ffbddc00640 1 -- 172.21.15.69:0/857264072 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd8099070 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.742+0000 7ffbcea00640 1 -- 172.21.15.69:0/857264072 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffbc0005d10 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.743+0000 7ffbcea00640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffbb406c8f0 0x7ffbb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:06.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.743+0000 7ffbcea00640 1 -- 172.21.15.69:0/857264072 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4857+0+0 (secure 0 0 0) 0x7ffbc008c420 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.743+0000 7ffbd6a00640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffbb406c8f0 0x7ffbb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:06.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.745+0000 7ffbd6a00640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffbb406c8f0 0x7ffbb406edb0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8034000 tx=0x7ffbc802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:06.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.747+0000 7ffbcea00640 1 -- 172.21.15.69:0/857264072 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffbc0022070 con 0x7ffbd80b3fb0 2024-04-24T08:20:06.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:06.947+0000 7ffbddc00640 1 -- 172.21.15.69:0/857264072 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ffbd8075dc0 con 0x7ffbd80b3fb0 2024-04-24T08:20:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.724+0000 7ffbcea00640 1 -- 172.21.15.69:0/857264072 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v106) ==== 138+0+0 (secure 0 0 0) 0x7ffbc0010040 con 0x7ffbd80b3fb0 2024-04-24T08:20:07.726 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' created 2024-04-24T08:20:07.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.728+0000 7ffbddc00640 1 -- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffbb406c8f0 msgr2=0x7ffbb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.728+0000 7ffbddc00640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffbb406c8f0 0x7ffbb406edb0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8034000 tx=0x7ffbc802e000 comp rx=0 tx=0).stop 2024-04-24T08:20:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.728+0000 7ffbddc00640 1 -- 172.21.15.69:0/857264072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd80b3fb0 msgr2=0x7ffbd8159540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.728+0000 7ffbddc00640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd80b3fb0 0x7ffbd8159540 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000efc0 tx=0x7ffbc0009f90 comp rx=0 tx=0).stop 2024-04-24T08:20:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.728+0000 7ffbddc00640 1 -- 172.21.15.69:0/857264072 shutdown_connections 2024-04-24T08:20:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.728+0000 7ffbddc00640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ffbb406c8f0 0x7ffbb406edb0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.729+0000 7ffbddc00640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffbd80b3fb0 0x7ffbd8159540 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.729+0000 7ffbddc00640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffbd815bcd0 0x7ffbd8156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.729+0000 7ffbddc00640 1 --2- 172.21.15.69:0/857264072 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ffbd80aeb00 0x7ffbd815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.729+0000 7ffbddc00640 1 -- 172.21.15.69:0/857264072 >> 172.21.15.69:0/857264072 conn(0x7ffbd806be40 msgr2=0x7ffbd80cd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:07.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.729+0000 7ffbddc00640 1 -- 172.21.15.69:0/857264072 shutdown_connections 2024-04-24T08:20:07.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.729+0000 7ffbddc00640 1 -- 172.21.15.69:0/857264072 wait complete. 2024-04-24T08:20:07.759 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T08:20:07.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.956+0000 7f4e9f600640 1 Processor -- start 2024-04-24T08:20:07.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.957+0000 7f4e9f600640 1 -- start start 2024-04-24T08:20:07.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.958+0000 7f4e9f600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4e9815e3b0 0x7f4e980ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:07.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.958+0000 7f4e9f600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4e9815e9a0 0x7f4e9815ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:07.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.958+0000 7f4e9f600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e980b3150 0x7f4e980b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:07.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.958+0000 7f4e9f600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9815f340 con 0x7f4e9815e9a0 2024-04-24T08:20:07.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.958+0000 7f4e9f600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9815e790 con 0x7f4e9815e3b0 2024-04-24T08:20:07.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.958+0000 7f4e9f600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4e98072330 con 0x7f4e980b3150 2024-04-24T08:20:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.959+0000 7f4e97e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4e9815e9a0 0x7f4e9815ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.959+0000 7f4e9cc00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4e9815e3b0 0x7f4e980ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.959+0000 7f4e97e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f4e9815e9a0 0x7f4e9815ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59606/0 (socket says 172.21.15.69:59606) 2024-04-24T08:20:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.959+0000 7f4e97e00640 1 -- 172.21.15.69:0/3886159689 learned_addr learned my addr 172.21.15.69:0/3886159689 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.959+0000 7f4e9d600640 1 --2- 172.21.15.69:0/3886159689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e980b3150 0x7f4e980b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.959+0000 7f4e97e00640 1 -- 172.21.15.69:0/3886159689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e980b3150 msgr2=0x7f4e980b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.959+0000 7f4e97e00640 1 --2- 172.21.15.69:0/3886159689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e980b3150 0x7f4e980b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.959+0000 7f4e97e00640 1 -- 172.21.15.69:0/3886159689 >> v2:172.21.15.121:3300/0 conn(0x7f4e9815e3b0 msgr2=0x7f4e980ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:07.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.959+0000 7f4e97e00640 1 --2- 172.21.15.69:0/3886159689 >> v2:172.21.15.121:3300/0 conn(0x7f4e9815e3b0 0x7f4e980ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.959+0000 7f4e97e00640 1 -- 172.21.15.69:0/3886159689 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e980b5170 con 0x7f4e9815e9a0 2024-04-24T08:20:07.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.960+0000 7f4e97e00640 1 --2- 172.21.15.69:0/3886159689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e9815e9a0 0x7f4e9815ee00 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f4e980cda80 tx=0x7f4e8002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fac0cb0523bb750 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:07.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.961+0000 7f4e97400640 1 -- 172.21.15.69:0/3886159689 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80037070 con 0x7f4e9815e9a0 2024-04-24T08:20:07.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.961+0000 7f4e97400640 1 -- 172.21.15.69:0/3886159689 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e80008500 con 0x7f4e9815e9a0 2024-04-24T08:20:07.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.961+0000 7f4e97400640 1 -- 172.21.15.69:0/3886159689 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e800088d0 con 0x7f4e9815e9a0 2024-04-24T08:20:07.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.961+0000 7f4e9f600640 1 -- 172.21.15.69:0/3886159689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e9815e9a0 msgr2=0x7f4e9815ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:07.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.961+0000 7f4e9f600640 1 --2- 172.21.15.69:0/3886159689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e9815e9a0 0x7f4e9815ee00 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f4e980cda80 tx=0x7f4e8002b7a0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.962+0000 7f4e9f600640 1 -- 172.21.15.69:0/3886159689 shutdown_connections 2024-04-24T08:20:07.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.962+0000 7f4e9f600640 1 --2- 172.21.15.69:0/3886159689 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e980b3150 0x7f4e980b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.962+0000 7f4e9f600640 1 --2- 172.21.15.69:0/3886159689 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e9815e9a0 0x7f4e9815ee00 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.962+0000 7f4e9f600640 1 --2- 172.21.15.69:0/3886159689 >> v2:172.21.15.121:3300/0 conn(0x7f4e9815e3b0 0x7f4e980ca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.962+0000 7f4e9f600640 1 -- 172.21.15.69:0/3886159689 >> 172.21.15.69:0/3886159689 conn(0x7f4e980d0720 msgr2=0x7f4e980d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.962+0000 7f4e9f600640 1 -- 172.21.15.69:0/3886159689 shutdown_connections 2024-04-24T08:20:07.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.962+0000 7f4e9f600640 1 -- 172.21.15.69:0/3886159689 wait complete. 2024-04-24T08:20:07.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.963+0000 7f4e9f600640 1 Processor -- start 2024-04-24T08:20:07.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.964+0000 7f4e9f600640 1 -- start start 2024-04-24T08:20:07.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.964+0000 7f4e9f600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e980b3150 0x7f4e981555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:07.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.964+0000 7f4e9f600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e9815e3b0 0x7f4e98155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:07.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.964+0000 7f4e9f600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9815e9a0 0x7f4e980ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:07.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.964+0000 7f4e9f600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e98071ce0 con 0x7f4e980b3150 2024-04-24T08:20:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.964+0000 7f4e9f600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4e980cd5d0 con 0x7f4e9815e9a0 2024-04-24T08:20:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.964+0000 7f4e9f600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4e98071fc0 con 0x7f4e9815e3b0 2024-04-24T08:20:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.965+0000 7f4e9cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e980b3150 0x7f4e981555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.965+0000 7f4e9cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e980b3150 0x7f4e981555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59618/0 (socket says 172.21.15.69:59618) 2024-04-24T08:20:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.965+0000 7f4e9cc00640 1 -- 172.21.15.69:0/1902184840 learned_addr learned my addr 172.21.15.69:0/1902184840 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.965+0000 7f4e9d600640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9815e9a0 0x7f4e980ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:07.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.965+0000 7f4e97e00640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e9815e3b0 0x7f4e98155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:07.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.965+0000 7f4e9cc00640 1 -- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e9815e3b0 msgr2=0x7f4e98155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:07.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.965+0000 7f4e9cc00640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e9815e3b0 0x7f4e98155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.965+0000 7f4e9cc00640 1 -- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9815e9a0 msgr2=0x7f4e980ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:07.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.965+0000 7f4e9cc00640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9815e9a0 0x7f4e980ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:07.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.965+0000 7f4e9cc00640 1 -- 172.21.15.69:0/1902184840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e98155af0 con 0x7f4e980b3150 2024-04-24T08:20:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.966+0000 7f4e9cc00640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e980b3150 0x7f4e981555b0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800e9e0 tx=0x7f4e8800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.966+0000 7f4e94c00640 1 -- 172.21.15.69:0/1902184840 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88017070 con 0x7f4e980b3150 2024-04-24T08:20:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.966+0000 7f4e94c00640 1 -- 172.21.15.69:0/1902184840 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8800c9d0 con 0x7f4e980b3150 2024-04-24T08:20:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.966+0000 7f4e9f600640 1 -- 172.21.15.69:0/1902184840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e98073170 con 0x7f4e980b3150 2024-04-24T08:20:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.966+0000 7f4e94c00640 1 -- 172.21.15.69:0/1902184840 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8800cdc0 con 0x7f4e980b3150 2024-04-24T08:20:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.966+0000 7f4e9f600640 1 -- 172.21.15.69:0/1902184840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e980af370 con 0x7f4e980b3150 2024-04-24T08:20:07.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.967+0000 7f4e9f600640 1 -- 172.21.15.69:0/1902184840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e98068a80 con 0x7f4e980b3150 2024-04-24T08:20:07.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.968+0000 7f4e94c00640 1 -- 172.21.15.69:0/1902184840 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4e8801a070 con 0x7f4e980b3150 2024-04-24T08:20:07.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.969+0000 7f4e94c00640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e7406c920 0x7f4e7406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:07.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.969+0000 7f4e94c00640 1 -- 172.21.15.69:0/1902184840 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 5227+0+0 (secure 0 0 0) 0x7f4e8808b8a0 con 0x7f4e980b3150 2024-04-24T08:20:07.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.970+0000 7f4e97e00640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e7406c920 0x7f4e7406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:07.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.971+0000 7f4e97e00640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e7406c920 0x7f4e7406ede0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4e80034000 tx=0x7f4e8000a000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:07.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:07.974+0000 7f4e94c00640 1 -- 172.21.15.69:0/1902184840 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e88059860 con 0x7f4e980b3150 2024-04-24T08:20:08.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.175+0000 7f4e9f600640 1 -- 172.21.15.69:0/1902184840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f4e980b49d0 con 0x7f4e980b3150 2024-04-24T08:20:08.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.730+0000 7f4e94c00640 1 -- 172.21.15.69:0/1902184840 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v107) ==== 170+0+0 (secure 0 0 0) 0x7f4e8805db60 con 0x7f4e980b3150 2024-04-24T08:20:08.732 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' created 2024-04-24T08:20:08.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.734+0000 7f4e9f600640 1 -- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e7406c920 msgr2=0x7f4e7406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:08.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.734+0000 7f4e9f600640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e7406c920 0x7f4e7406ede0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f4e80034000 tx=0x7f4e8000a000 comp rx=0 tx=0).stop 2024-04-24T08:20:08.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.734+0000 7f4e9f600640 1 -- 172.21.15.69:0/1902184840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e980b3150 msgr2=0x7f4e981555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:08.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.734+0000 7f4e9f600640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e980b3150 0x7f4e981555b0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800e9e0 tx=0x7f4e8800eea0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.734+0000 7f4e9f600640 1 -- 172.21.15.69:0/1902184840 shutdown_connections 2024-04-24T08:20:08.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.734+0000 7f4e9f600640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4e7406c920 0x7f4e7406ede0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.734+0000 7f4e9f600640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e9815e9a0 0x7f4e980ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.734+0000 7f4e9f600640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e9815e3b0 0x7f4e98155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.734+0000 7f4e9f600640 1 --2- 172.21.15.69:0/1902184840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e980b3150 0x7f4e981555b0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.734+0000 7f4e9f600640 1 -- 172.21.15.69:0/1902184840 >> 172.21.15.69:0/1902184840 conn(0x7f4e980d0720 msgr2=0x7f4e980b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:08.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.735+0000 7f4e9f600640 1 -- 172.21.15.69:0/1902184840 shutdown_connections 2024-04-24T08:20:08.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.735+0000 7f4e9f600640 1 -- 172.21.15.69:0/1902184840 wait complete. 2024-04-24T08:20:08.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T08:20:08.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.946+0000 7fb657e00640 1 Processor -- start 2024-04-24T08:20:08.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.946+0000 7fb657e00640 1 -- start start 2024-04-24T08:20:08.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.947+0000 7fb657e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb658071d70 0x7fb658072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:08.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.947+0000 7fb657e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb658072690 con 0x7fb65815edf0 2024-04-24T08:20:08.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.947+0000 7fb657e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb65815fbb0 con 0x7fb65806e000 2024-04-24T08:20:08.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.947+0000 7fb657e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb6580d0a00 con 0x7fb658071d70 2024-04-24T08:20:08.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb64e000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb65815edf0 0x7fb65815f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43192/0 (socket says 172.21.15.69:43192) 2024-04-24T08:20:08.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb64e000640 1 -- 172.21.15.69:0/4040771564 learned_addr learned my addr 172.21.15.69:0/4040771564 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:08.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3331422056 0 0) 0x7fb658072690 con 0x7fb65815edf0 2024-04-24T08:20:08.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb656a00640 1 --2- 172.21.15.69:0/4040771564 >> v2:172.21.15.121:3300/0 conn(0x7fb658071d70 0x7fb658072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb63c003520 con 0x7fb65815edf0 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1278076653 0 0) 0x7fb63c003520 con 0x7fb65815edf0 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 >> v2:172.21.15.121:3300/0 conn(0x7fb658071d70 msgr2=0x7fb658072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb656000640 1 --2- 172.21.15.69:0/4040771564 >> v2:172.21.15.121:3300/0 conn(0x7fb658071d70 0x7fb658072150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 >> v1:172.21.15.121:6789/0 conn(0x7fb65806e000 legacy=0x7fb65806e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6580d00d0 con 0x7fb65815edf0 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fb640001670 con 0x7fb65815edf0 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.948+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 >> v1:172.21.15.69:6789/0 conn(0x7fb65815edf0 legacy=0x7fb65815f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb656000640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb63c0039b0 0x7fb63c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb656000640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb63c004800 0x7fb63c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb656000640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb63c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb6580d0a00 con 0x7fb63c0051c0 2024-04-24T08:20:08.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb63c00c0f0 con 0x7fb63c004800 2024-04-24T08:20:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb63c00c2f0 con 0x7fb63c0039b0 2024-04-24T08:20:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb657400640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb63c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb64e000640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb63c004800 0x7fb63c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb656a00640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb63c0039b0 0x7fb63c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb657400640 1 -- 172.21.15.69:0/4040771564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb63c0039b0 msgr2=0x7fb63c003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb657400640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb63c0039b0 0x7fb63c003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb657400640 1 -- 172.21.15.69:0/4040771564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb63c004800 msgr2=0x7fb63c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.949+0000 7fb657400640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb63c004800 0x7fb63c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.950+0000 7fb657400640 1 -- 172.21.15.69:0/4040771564 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63c003520 con 0x7fb63c0051c0 2024-04-24T08:20:08.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.950+0000 7fb657400640 1 -- 172.21.15.69:0/4040771564 reap_dead start 2024-04-24T08:20:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.951+0000 7fb657400640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb63c00b9c0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fb650001d20 tx=0x7fb63c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a459175e98b8d296 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.951+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650001920 con 0x7fb63c0051c0 2024-04-24T08:20:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.951+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65002e070 con 0x7fb63c0051c0 2024-04-24T08:20:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.951+0000 7fb656000640 1 -- 172.21.15.69:0/4040771564 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb650003970 con 0x7fb63c0051c0 2024-04-24T08:20:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.951+0000 7fb657e00640 1 -- 172.21.15.69:0/4040771564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 msgr2=0x7fb63c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:08.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.951+0000 7fb657e00640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb63c00b9c0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fb650001d20 tx=0x7fb63c005830 comp rx=0 tx=0).stop 2024-04-24T08:20:08.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.952+0000 7fb657e00640 1 -- 172.21.15.69:0/4040771564 shutdown_connections 2024-04-24T08:20:08.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.952+0000 7fb657e00640 1 --2- 172.21.15.69:0/4040771564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb63c00b9c0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.952+0000 7fb657e00640 1 -- 172.21.15.69:0/4040771564 >> 172.21.15.69:0/4040771564 conn(0x7fb658067b80 msgr2=0x7fb65806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:08.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.952+0000 7fb657e00640 1 -- 172.21.15.69:0/4040771564 shutdown_connections 2024-04-24T08:20:08.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.952+0000 7fb657e00640 1 -- 172.21.15.69:0/4040771564 wait complete. 2024-04-24T08:20:08.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.953+0000 7fb657e00640 1 Processor -- start 2024-04-24T08:20:08.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.954+0000 7fb657e00640 1 -- start start 2024-04-24T08:20:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.954+0000 7fb657e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb658146300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb657e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb65813edb0 0x7fb6580cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb657e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6581f53f0 0x7fb65806e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:08.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb657e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb6580d0e90 con 0x7fb63c0051c0 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb657e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb6580d0690 con 0x7fb6581f53f0 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb657e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb6580d1010 con 0x7fb65813edb0 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb656a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb658146300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb656a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb658146300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54388/0 (socket says 172.21.15.69:54388) 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb656a00640 1 -- 172.21.15.69:0/1095918982 learned_addr learned my addr 172.21.15.69:0/1095918982 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb657400640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6581f53f0 0x7fb65806e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb64e000640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb65813edb0 0x7fb6580cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb656a00640 1 -- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb65813edb0 msgr2=0x7fb6580cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb656a00640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb65813edb0 0x7fb6580cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb656a00640 1 -- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6581f53f0 msgr2=0x7fb65806e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb656a00640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6581f53f0 0x7fb65806e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb656a00640 1 -- 172.21.15.69:0/1095918982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6580cde90 con 0x7fb63c0051c0 2024-04-24T08:20:08.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.955+0000 7fb657400640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6581f53f0 0x7fb65806e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:08.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.956+0000 7fb656a00640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb658146300 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fb64800f580 tx=0x7fb64800fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:08.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.956+0000 7fb64fe00640 1 -- 172.21.15.69:0/1095918982 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648013810 con 0x7fb63c0051c0 2024-04-24T08:20:08.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.956+0000 7fb64fe00640 1 -- 172.21.15.69:0/1095918982 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb64801a070 con 0x7fb63c0051c0 2024-04-24T08:20:08.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.956+0000 7fb64fe00640 1 -- 172.21.15.69:0/1095918982 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648013bb0 con 0x7fb63c0051c0 2024-04-24T08:20:08.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.956+0000 7fb657e00640 1 -- 172.21.15.69:0/1095918982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6580ce170 con 0x7fb63c0051c0 2024-04-24T08:20:08.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.956+0000 7fb657e00640 1 -- 172.21.15.69:0/1095918982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb658143d40 con 0x7fb63c0051c0 2024-04-24T08:20:08.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.958+0000 7fb657e00640 1 -- 172.21.15.69:0/1095918982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6580ce690 con 0x7fb63c0051c0 2024-04-24T08:20:08.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.959+0000 7fb64fe00640 1 -- 172.21.15.69:0/1095918982 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb648012070 con 0x7fb63c0051c0 2024-04-24T08:20:08.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.960+0000 7fb64fe00640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb63406c920 0x7fb63406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:08.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.960+0000 7fb64fe00640 1 -- 172.21.15.69:0/1095918982 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 5609+0+0 (secure 0 0 0) 0x7fb64808c590 con 0x7fb63c0051c0 2024-04-24T08:20:08.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.960+0000 7fb64e000640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb63406c920 0x7fb63406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:08.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.962+0000 7fb64e000640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb63406c920 0x7fb63406ede0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb640000f60 tx=0x7fb640000ed0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:08.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:08.964+0000 7fb64fe00640 1 -- 172.21.15.69:0/1095918982 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6580ce690 con 0x7fb63c0051c0 2024-04-24T08:20:09.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.202+0000 7fb657e00640 1 -- 172.21.15.69:0/1095918982 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb658075d50 con 0x7fb63c0051c0 2024-04-24T08:20:09.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.756+0000 7fb64fe00640 1 -- 172.21.15.69:0/1095918982 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v96) ==== 174+0+0 (secure 0 0 0) 0x7fb64805aa10 con 0x7fb63c0051c0 2024-04-24T08:20:09.758 INFO:teuthology.orchestra.run.smithi069.stderr:new fs with metadata pool 16 and data pool 17 2024-04-24T08:20:09.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.759+0000 7fb657e00640 1 -- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb63406c920 msgr2=0x7fb63406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.759+0000 7fb657e00640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb63406c920 0x7fb63406ede0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fb640000f60 tx=0x7fb640000ed0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.759+0000 7fb657e00640 1 -- 172.21.15.69:0/1095918982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 msgr2=0x7fb658146300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.759+0000 7fb657e00640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb658146300 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fb64800f580 tx=0x7fb64800fa40 comp rx=0 tx=0).stop 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.760+0000 7fb657e00640 1 -- 172.21.15.69:0/1095918982 shutdown_connections 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.760+0000 7fb657e00640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb63406c920 0x7fb63406ede0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.760+0000 7fb657e00640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6581f53f0 0x7fb65806e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.760+0000 7fb657e00640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb65813edb0 0x7fb6580cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.760+0000 7fb657e00640 1 --2- 172.21.15.69:0/1095918982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb63c0051c0 0x7fb658146300 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.760+0000 7fb657e00640 1 -- 172.21.15.69:0/1095918982 >> 172.21.15.69:0/1095918982 conn(0x7fb658067b80 msgr2=0x7fb65815fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.760+0000 7fb657e00640 1 -- 172.21.15.69:0/1095918982 shutdown_connections 2024-04-24T08:20:09.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.760+0000 7fb657e00640 1 -- 172.21.15.69:0/1095918982 wait complete. 2024-04-24T08:20:09.799 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:09.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.970+0000 7f7550600640 1 Processor -- start 2024-04-24T08:20:09.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.970+0000 7f7550600640 1 -- start start 2024-04-24T08:20:09.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.971+0000 7f7550600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548145fe0 0x7f754806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.971+0000 7f7550600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75480cf3d0 con 0x7f7548145a90 2024-04-24T08:20:09.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.971+0000 7f7550600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75480cf6b0 con 0x7f75480d0300 2024-04-24T08:20:09.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.971+0000 7f7550600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75480cf990 con 0x7f7548145fe0 2024-04-24T08:20:09.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.972+0000 7f754d200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7548145a90 0x7f7548160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43208/0 (socket says 172.21.15.69:43208) 2024-04-24T08:20:09.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.972+0000 7f754d200640 1 -- 172.21.15.69:0/4257257428 learned_addr learned my addr 172.21.15.69:0/4257257428 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:09.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.972+0000 7f754e600640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548145fe0 0x7f754806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.972+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3790712051 0 0) 0x7f75480cf3d0 con 0x7f7548145a90 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.972+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7530003520 con 0x7f7548145a90 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1017781296 0 0) 0x7f7530003520 con 0x7f7548145a90 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548145fe0 msgr2=0x7f754806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548145fe0 0x7f754806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 >> v1:172.21.15.121:6789/0 conn(0x7f75480d0300 legacy=0x7f75480d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75480cfc70 con 0x7f7548145a90 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f7534001670 con 0x7f7548145a90 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 >> v1:172.21.15.69:6789/0 conn(0x7f7548145a90 legacy=0x7f7548160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548145fe0 0x7f75300039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7530004010 0x7f7530004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 0x7f75300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f75480cf990 con 0x7f7530004d40 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f753000bb80 con 0x7f7530004010 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f753000bd80 con 0x7f7548145fe0 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f754e600640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 0x7f75300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f754d200640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7530004010 0x7f7530004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f754e600640 1 -- 172.21.15.69:0/4257257428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548145fe0 msgr2=0x7f75300039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f754e600640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548145fe0 0x7f75300039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f754e600640 1 -- 172.21.15.69:0/4257257428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7530004010 msgr2=0x7f7530004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f754e600640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7530004010 0x7f7530004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.973+0000 7f754e600640 1 -- 172.21.15.69:0/4257257428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7530003520 con 0x7f7530004d40 2024-04-24T08:20:09.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.974+0000 7f754e600640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 0x7f75300051c0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f7540007360 tx=0x7f7530005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfe38e02e9bcb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.975+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7540009be0 con 0x7f7530004d40 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.975+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f754002e620 con 0x7f7530004d40 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.975+0000 7f7547e00640 1 -- 172.21.15.69:0/4257257428 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f754002e9f0 con 0x7f7530004d40 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.975+0000 7f7550600640 1 -- 172.21.15.69:0/4257257428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 msgr2=0x7f75300051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.975+0000 7f7550600640 1 --2- 172.21.15.69:0/4257257428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 0x7f75300051c0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f7540007360 tx=0x7f7530005830 comp rx=0 tx=0).stop 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.975+0000 7f754e600640 1 -- 172.21.15.69:0/4257257428 reap_dead start 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.975+0000 7f7550600640 1 -- 172.21.15.69:0/4257257428 shutdown_connections 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.975+0000 7f7550600640 1 -- 172.21.15.69:0/4257257428 >> 172.21.15.69:0/4257257428 conn(0x7f7548067d90 msgr2=0x7f754806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.975+0000 7f7550600640 1 -- 172.21.15.69:0/4257257428 shutdown_connections 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.975+0000 7f7550600640 1 -- 172.21.15.69:0/4257257428 wait complete. 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.977+0000 7f7550600640 1 Processor -- start 2024-04-24T08:20:09.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.977+0000 7f7550600640 1 -- start start 2024-04-24T08:20:09.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.978+0000 7f7550600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 0x7f754815c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.978+0000 7f7550600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548141f10 0x7f7548142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.978+0000 7f7550600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75480ae220 0x7f75480ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.978+0000 7f7550600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f75481458f0 con 0x7f7530004d40 2024-04-24T08:20:09.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.978+0000 7f7550600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7548145770 con 0x7f75480ae220 2024-04-24T08:20:09.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.978+0000 7f7550600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f75480cdc30 con 0x7f7548141f10 2024-04-24T08:20:09.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754dc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 0x7f754815c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754dc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 0x7f754815c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54398/0 (socket says 172.21.15.69:54398) 2024-04-24T08:20:09.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754dc00640 1 -- 172.21.15.69:0/610577931 learned_addr learned my addr 172.21.15.69:0/610577931 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754e600640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75480ae220 0x7f75480ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754d200640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548141f10 0x7f7548142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754dc00640 1 -- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548141f10 msgr2=0x7f7548142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754dc00640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548141f10 0x7f7548142390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754dc00640 1 -- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75480ae220 msgr2=0x7f75480ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754dc00640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75480ae220 0x7f75480ae6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754dc00640 1 -- 172.21.15.69:0/610577931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f754815c750 con 0x7f7530004d40 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.979+0000 7f754dc00640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 0x7f754815c210 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f753c00c3f0 tx=0x7f753c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.980+0000 7f7545600640 1 -- 172.21.15.69:0/610577931 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c017070 con 0x7f7530004d40 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.980+0000 7f7545600640 1 -- 172.21.15.69:0/610577931 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f753c0052f0 con 0x7f7530004d40 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.980+0000 7f7550600640 1 -- 172.21.15.69:0/610577931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75480aeed0 con 0x7f7530004d40 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.980+0000 7f7545600640 1 -- 172.21.15.69:0/610577931 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c003830 con 0x7f7530004d40 2024-04-24T08:20:09.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.980+0000 7f7550600640 1 -- 172.21.15.69:0/610577931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f754812f4a0 con 0x7f7530004d40 2024-04-24T08:20:09.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.981+0000 7f7550600640 1 -- 172.21.15.69:0/610577931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7548068a80 con 0x7f7530004d40 2024-04-24T08:20:09.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.982+0000 7f7545600640 1 -- 172.21.15.69:0/610577931 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f753c01a070 con 0x7f7530004d40 2024-04-24T08:20:09.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.983+0000 7f7545600640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f752806c8f0 0x7f752806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:09.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.983+0000 7f7545600640 1 -- 172.21.15.69:0/610577931 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f753c08af10 con 0x7f7530004d40 2024-04-24T08:20:09.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.984+0000 7f754d200640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f752806c8f0 0x7f752806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:09.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.985+0000 7f754d200640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f752806c8f0 0x7f752806edb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7534000fd0 tx=0x7f7534009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:09.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:09.987+0000 7f7545600640 1 -- 172.21.15.69:0/610577931 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f753c0592a0 con 0x7f7530004d40 2024-04-24T08:20:10.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.172+0000 7f7550600640 1 -- 172.21.15.69:0/610577931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75480d1120 con 0x7f7530004d40 2024-04-24T08:20:10.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.175+0000 7f7545600640 1 -- 172.21.15.69:0/610577931 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+16616 (secure 0 0 0) 0x7f753c058ba0 con 0x7f7530004d40 2024-04-24T08:20:10.176 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:10.177 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":108,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:20:09.204208+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:10.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 -- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f752806c8f0 msgr2=0x7f752806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f752806c8f0 0x7f752806edb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f7534000fd0 tx=0x7f7534009040 comp rx=0 tx=0).stop 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 -- 172.21.15.69:0/610577931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 msgr2=0x7f754815c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 0x7f754815c210 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f753c00c3f0 tx=0x7f753c00c8b0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 -- 172.21.15.69:0/610577931 shutdown_connections 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f752806c8f0 0x7f752806edb0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75480ae220 0x7f75480ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7548141f10 0x7f7548142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 --2- 172.21.15.69:0/610577931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7530004d40 0x7f754815c210 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 -- 172.21.15.69:0/610577931 >> 172.21.15.69:0/610577931 conn(0x7f7548067d90 msgr2=0x7f7548072600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.177+0000 7f7550600640 1 -- 172.21.15.69:0/610577931 shutdown_connections 2024-04-24T08:20:10.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.178+0000 7f7550600640 1 -- 172.21.15.69:0/610577931 wait complete. 2024-04-24T08:20:10.197 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:10.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.394+0000 7f736b200640 1 Processor -- start 2024-04-24T08:20:10.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.394+0000 7f736b200640 1 -- start start 2024-04-24T08:20:10.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.394+0000 7f736b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736406eb30 0x7f7364144090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.394+0000 7f736b200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7364071be0 0x7f73640b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.394+0000 7f736b200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73640b3190 con 0x7f736415ab10 2024-04-24T08:20:10.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.394+0000 7f736b200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f73640ad590 con 0x7f7364071be0 2024-04-24T08:20:10.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.394+0000 7f736b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f73640b3400 con 0x7f736406eb30 2024-04-24T08:20:10.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.395+0000 7f7369400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736406eb30 0x7f7364144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.395+0000 7f736a800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7364071be0 0x7f73640b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.395+0000 7f7369400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736406eb30 0x7f7364144090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:33984/0 (socket says 172.21.15.69:33984) 2024-04-24T08:20:10.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.395+0000 7f7369400640 1 -- 172.21.15.69:0/1545945886 learned_addr learned my addr 172.21.15.69:0/1545945886 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:10.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.396+0000 7f7369400640 1 -- 172.21.15.69:0/1545945886 >> v2:172.21.15.121:3300/0 conn(0x7f7364071be0 msgr2=0x7f73640b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:10.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.396+0000 7f7369400640 1 --2- 172.21.15.69:0/1545945886 >> v2:172.21.15.121:3300/0 conn(0x7f7364071be0 0x7f73640b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.396+0000 7f7369400640 1 -- 172.21.15.69:0/1545945886 >> v1:172.21.15.121:6789/0 conn(0x7f736415ab10 legacy=0x7f736406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.396+0000 7f7369400640 1 -- 172.21.15.69:0/1545945886 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73640b36e0 con 0x7f736406eb30 2024-04-24T08:20:10.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.398+0000 7f7369400640 1 --2- 172.21.15.69:0/1545945886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736406eb30 0x7f7364144090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7354009870 tx=0x7f735402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9639a5556c46f0da server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:10.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.398+0000 7f7368a00640 1 -- 172.21.15.69:0/1545945886 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354037070 con 0x7f736406eb30 2024-04-24T08:20:10.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.398+0000 7f7368a00640 1 -- 172.21.15.69:0/1545945886 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7354032440 con 0x7f736406eb30 2024-04-24T08:20:10.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.398+0000 7f7368a00640 1 -- 172.21.15.69:0/1545945886 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7354032810 con 0x7f736406eb30 2024-04-24T08:20:10.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.399+0000 7f736b200640 1 -- 172.21.15.69:0/1545945886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736406eb30 msgr2=0x7f7364144090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.399+0000 7f736b200640 1 --2- 172.21.15.69:0/1545945886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736406eb30 0x7f7364144090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7354009870 tx=0x7f735402a270 comp rx=0 tx=0).stop 2024-04-24T08:20:10.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.399+0000 7f736b200640 1 -- 172.21.15.69:0/1545945886 shutdown_connections 2024-04-24T08:20:10.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.399+0000 7f736b200640 1 --2- 172.21.15.69:0/1545945886 >> v2:172.21.15.121:3300/0 conn(0x7f7364071be0 0x7f73640b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.399+0000 7f736b200640 1 --2- 172.21.15.69:0/1545945886 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736406eb30 0x7f7364144090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.399+0000 7f736b200640 1 -- 172.21.15.69:0/1545945886 >> 172.21.15.69:0/1545945886 conn(0x7f73640d0160 msgr2=0x7f73640d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:10.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.399+0000 7f736b200640 1 -- 172.21.15.69:0/1545945886 shutdown_connections 2024-04-24T08:20:10.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.399+0000 7f736b200640 1 -- 172.21.15.69:0/1545945886 wait complete. 2024-04-24T08:20:10.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.401+0000 7f736b200640 1 Processor -- start 2024-04-24T08:20:10.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.401+0000 7f736b200640 1 -- start start 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f736b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f736406eb30 0x7f73640b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f736b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7364071be0 0x7f736406bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f736b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736415ab10 0x7f73641f20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f736b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f736415aef0 con 0x7f736406eb30 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f736b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7364072150 con 0x7f7364071be0 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f736b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f736406f950 con 0x7f736415ab10 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f7369e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f736406eb30 0x7f73640b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f7369e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f736406eb30 0x7f73640b1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54406/0 (socket says 172.21.15.69:54406) 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f7369e00640 1 -- 172.21.15.69:0/1521796254 learned_addr learned my addr 172.21.15.69:0/1521796254 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f736a800640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736415ab10 0x7f73641f20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f7369400640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7364071be0 0x7f736406bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f7369e00640 1 -- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736415ab10 msgr2=0x7f73641f20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f7369e00640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736415ab10 0x7f73641f20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f7369e00640 1 -- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7364071be0 msgr2=0x7f736406bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f7369e00640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7364071be0 0x7f736406bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f7369e00640 1 -- 172.21.15.69:0/1521796254 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7364152ae0 con 0x7f736406eb30 2024-04-24T08:20:10.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.402+0000 7f7369400640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7364071be0 0x7f736406bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:10.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.403+0000 7f7369e00640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f736406eb30 0x7f73640b1640 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f735800c340 tx=0x7f735800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:10.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.403+0000 7f735e000640 1 -- 172.21.15.69:0/1521796254 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7358017070 con 0x7f736406eb30 2024-04-24T08:20:10.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.403+0000 7f735e000640 1 -- 172.21.15.69:0/1521796254 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73580052d0 con 0x7f736406eb30 2024-04-24T08:20:10.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.403+0000 7f736b200640 1 -- 172.21.15.69:0/1521796254 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7364152dc0 con 0x7f736406eb30 2024-04-24T08:20:10.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.403+0000 7f735e000640 1 -- 172.21.15.69:0/1521796254 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7358003850 con 0x7f736406eb30 2024-04-24T08:20:10.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.403+0000 7f736b200640 1 -- 172.21.15.69:0/1521796254 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7364155ae0 con 0x7f736406eb30 2024-04-24T08:20:10.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.405+0000 7f736b200640 1 -- 172.21.15.69:0/1521796254 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7364072bd0 con 0x7f736406eb30 2024-04-24T08:20:10.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.405+0000 7f735e000640 1 -- 172.21.15.69:0/1521796254 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f735801a070 con 0x7f736406eb30 2024-04-24T08:20:10.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.406+0000 7f735e000640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f734806c8f0 0x7f734806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.406+0000 7f735e000640 1 -- 172.21.15.69:0/1521796254 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 5709+0+0 (secure 0 0 0) 0x7f735808ae60 con 0x7f736406eb30 2024-04-24T08:20:10.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.407+0000 7f7369400640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f734806c8f0 0x7f734806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.408+0000 7f7369400640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f734806c8f0 0x7f734806edb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7354034000 tx=0x7f735402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:10.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.411+0000 7f735e000640 1 -- 172.21.15.69:0/1521796254 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7364072bd0 con 0x7f736406eb30 2024-04-24T08:20:10.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.601+0000 7f736b200640 1 -- 172.21.15.69:0/1521796254 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7364075d50 con 0x7f736406eb30 2024-04-24T08:20:10.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.604+0000 7f735e000640 1 -- 172.21.15.69:0/1521796254 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) ==== 74+0+16616 (secure 0 0 0) 0x7f7358059280 con 0x7f736406eb30 2024-04-24T08:20:10.606 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:10.606 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":108,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:20:09.204208+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.606+0000 7f736b200640 1 -- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f734806c8f0 msgr2=0x7f734806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.606+0000 7f736b200640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f734806c8f0 0x7f734806edb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7354034000 tx=0x7f735402e000 comp rx=0 tx=0).stop 2024-04-24T08:20:10.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.606+0000 7f736b200640 1 -- 172.21.15.69:0/1521796254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f736406eb30 msgr2=0x7f73640b1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.606+0000 7f736b200640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f736406eb30 0x7f73640b1640 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f735800c340 tx=0x7f735800c800 comp rx=0 tx=0).stop 2024-04-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.607+0000 7f736b200640 1 -- 172.21.15.69:0/1521796254 shutdown_connections 2024-04-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.607+0000 7f736b200640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f734806c8f0 0x7f734806edb0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.607+0000 7f736b200640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f736415ab10 0x7f73641f20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.607+0000 7f736b200640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7364071be0 0x7f736406bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.607+0000 7f736b200640 1 --2- 172.21.15.69:0/1521796254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f736406eb30 0x7f73640b1640 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.607+0000 7f736b200640 1 -- 172.21.15.69:0/1521796254 >> 172.21.15.69:0/1521796254 conn(0x7f73640d0160 msgr2=0x7f73640b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.607+0000 7f736b200640 1 -- 172.21.15.69:0/1521796254 shutdown_connections 2024-04-24T08:20:10.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.607+0000 7f736b200640 1 -- 172.21.15.69:0/1521796254 wait complete. 2024-04-24T08:20:10.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T08:20:10.714 INFO:tasks.ceph.mds.b.smithi121.stderr:2024-04-24T08:20:10.713+0000 7fe60c400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:20:10.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.821+0000 7f2127400640 1 Processor -- start 2024-04-24T08:20:10.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.822+0000 7f2127400640 1 -- start start 2024-04-24T08:20:10.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.823+0000 7f2127400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f21200b46a0 0x7f21200ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.823+0000 7f2127400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f212015cc00 con 0x7f212015c820 2024-04-24T08:20:10.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.823+0000 7f2127400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f212006ed30 con 0x7f21200b3e90 2024-04-24T08:20:10.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.823+0000 7f2127400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f21200b4e10 con 0x7f21200b46a0 2024-04-24T08:20:10.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.823+0000 7f2124a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f212015c820 0x7f212006f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43222/0 (socket says 172.21.15.69:43222) 2024-04-24T08:20:10.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.823+0000 7f2124a00640 1 -- 172.21.15.69:0/4060885915 learned_addr learned my addr 172.21.15.69:0/4060885915 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:10.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.823+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1288864482 0 0) 0x7f212015cc00 con 0x7f212015c820 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.823+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2104003520 con 0x7f212015c820 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.823+0000 7f2125400640 1 --2- 172.21.15.69:0/4060885915 >> v2:172.21.15.121:3300/0 conn(0x7f21200b46a0 0x7f21200ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3589363462 0 0) 0x7f2104003520 con 0x7f212015c820 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 >> v2:172.21.15.121:3300/0 conn(0x7f21200b46a0 msgr2=0x7f21200ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 --2- 172.21.15.69:0/4060885915 >> v2:172.21.15.121:3300/0 conn(0x7f21200b46a0 0x7f21200ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 >> v1:172.21.15.121:6789/0 conn(0x7f21200b3e90 legacy=0x7f21200ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21200b50f0 con 0x7f212015c820 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f2110001670 con 0x7f212015c820 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 >> v1:172.21.15.69:6789/0 conn(0x7f212015c820 legacy=0x7f212006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 --2- 172.21.15.69:0/4060885915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21040039b0 0x7f2104003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 --2- 172.21.15.69:0/4060885915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2104004800 0x7f2104004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 --2- 172.21.15.69:0/4060885915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040051c0 0x7f210400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f21200b4e10 con 0x7f21040039b0 2024-04-24T08:20:10.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f210400c110 con 0x7f2104004800 2024-04-24T08:20:10.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f210400c310 con 0x7f21040051c0 2024-04-24T08:20:10.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.824+0000 7f2124a00640 1 --2- 172.21.15.69:0/4060885915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21040039b0 0x7f2104003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.825+0000 7f2117e00640 1 --2- 172.21.15.69:0/4060885915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2104004800 0x7f2104004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.825+0000 7f2124a00640 1 -- 172.21.15.69:0/4060885915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040051c0 msgr2=0x7f210400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:10.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.825+0000 7f2124a00640 1 --2- 172.21.15.69:0/4060885915 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f21040051c0 0x7f210400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.825+0000 7f2124a00640 1 -- 172.21.15.69:0/4060885915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2104004800 msgr2=0x7f2104004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:10.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.825+0000 7f2124a00640 1 --2- 172.21.15.69:0/4060885915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2104004800 0x7f2104004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.825+0000 7f2124a00640 1 -- 172.21.15.69:0/4060885915 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2104003520 con 0x7f21040039b0 2024-04-24T08:20:10.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.825+0000 7f2125400640 1 -- 172.21.15.69:0/4060885915 reap_dead start 2024-04-24T08:20:10.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.826+0000 7f2124a00640 1 --2- 172.21.15.69:0/4060885915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21040039b0 0x7f2104003dd0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f211000c240 tx=0x7f2104005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7c346cfebeb545 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:10.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.826+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211002e070 con 0x7f21040039b0 2024-04-24T08:20:10.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.826+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2110005af0 con 0x7f21040039b0 2024-04-24T08:20:10.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.826+0000 7f211fe00640 1 -- 172.21.15.69:0/4060885915 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2110007d30 con 0x7f21040039b0 2024-04-24T08:20:10.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.826+0000 7f2127400640 1 -- 172.21.15.69:0/4060885915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21040039b0 msgr2=0x7f2104003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.826+0000 7f2127400640 1 --2- 172.21.15.69:0/4060885915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21040039b0 0x7f2104003dd0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f211000c240 tx=0x7f2104005830 comp rx=0 tx=0).stop 2024-04-24T08:20:10.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.827+0000 7f2127400640 1 -- 172.21.15.69:0/4060885915 shutdown_connections 2024-04-24T08:20:10.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.827+0000 7f2127400640 1 --2- 172.21.15.69:0/4060885915 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21040039b0 0x7f2104003dd0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.827+0000 7f2127400640 1 -- 172.21.15.69:0/4060885915 >> 172.21.15.69:0/4060885915 conn(0x7f21200d0350 msgr2=0x7f21200d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:10.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.827+0000 7f2127400640 1 -- 172.21.15.69:0/4060885915 shutdown_connections 2024-04-24T08:20:10.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.827+0000 7f2127400640 1 -- 172.21.15.69:0/4060885915 wait complete. 2024-04-24T08:20:10.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.828+0000 7f2127400640 1 Processor -- start 2024-04-24T08:20:10.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.828+0000 7f2127400640 1 -- start start 2024-04-24T08:20:10.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.829+0000 7f2127400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040039b0 0x7f212015a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.829+0000 7f2127400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21200994f0 0x7f2120099970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.829+0000 7f2127400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f212015e8c0 0x7f212015ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.829+0000 7f2127400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f21200b4b10 con 0x7f21200994f0 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.829+0000 7f2127400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f212006f2b0 con 0x7f21040039b0 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.829+0000 7f2127400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f212006f7d0 con 0x7f212015e8c0 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2117e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21200994f0 0x7f2120099970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2117e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21200994f0 0x7f2120099970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54410/0 (socket says 172.21.15.69:54410) 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2117e00640 1 -- 172.21.15.69:0/2434351927 learned_addr learned my addr 172.21.15.69:0/2434351927 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2124a00640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040039b0 0x7f212015a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2125400640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f212015e8c0 0x7f212015ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2117e00640 1 -- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f212015e8c0 msgr2=0x7f212015ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2117e00640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f212015e8c0 0x7f212015ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2117e00640 1 -- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040039b0 msgr2=0x7f212015a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:10.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2117e00640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040039b0 0x7f212015a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:10.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2117e00640 1 -- 172.21.15.69:0/2434351927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f212015f5f0 con 0x7f21200994f0 2024-04-24T08:20:10.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.830+0000 7f2117e00640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21200994f0 0x7f2120099970 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f210800e410 tx=0x7f210800e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:10.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.831+0000 7f211d600640 1 -- 172.21.15.69:0/2434351927 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108003a70 con 0x7f21200994f0 2024-04-24T08:20:10.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.831+0000 7f211d600640 1 -- 172.21.15.69:0/2434351927 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2108018070 con 0x7f21200994f0 2024-04-24T08:20:10.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.831+0000 7f2127400640 1 -- 172.21.15.69:0/2434351927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f212015f8d0 con 0x7f21200994f0 2024-04-24T08:20:10.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.831+0000 7f211d600640 1 -- 172.21.15.69:0/2434351927 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108003e50 con 0x7f21200994f0 2024-04-24T08:20:10.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.831+0000 7f2127400640 1 -- 172.21.15.69:0/2434351927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21201409f0 con 0x7f21200994f0 2024-04-24T08:20:10.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.832+0000 7f2127400640 1 -- 172.21.15.69:0/2434351927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2120068a80 con 0x7f21200994f0 2024-04-24T08:20:10.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.833+0000 7f211d600640 1 -- 172.21.15.69:0/2434351927 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2108004050 con 0x7f21200994f0 2024-04-24T08:20:10.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.834+0000 7f211d600640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20fc06ca90 0x7f20fc06ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:10.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.834+0000 7f211d600640 1 -- 172.21.15.69:0/2434351927 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f210808b8d0 con 0x7f21200994f0 2024-04-24T08:20:10.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.835+0000 7f2124a00640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20fc06ca90 0x7f20fc06ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:10.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.836+0000 7f2124a00640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20fc06ca90 0x7f20fc06ef50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2110033000 tx=0x7f211002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:10.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:10.838+0000 7f211d600640 1 -- 172.21.15.69:0/2434351927 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2108059c60 con 0x7f21200994f0 2024-04-24T08:20:11.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.080+0000 7f2127400640 1 -- 172.21.15.69:0/2434351927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f21200cddc0 con 0x7f21200994f0 2024-04-24T08:20:11.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.793+0000 7f211d600640 1 -- 172.21.15.69:0/2434351927 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v98) ==== 118+0+0 (secure 0 0 0) 0x7f2108059560 con 0x7f21200994f0 2024-04-24T08:20:11.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.796+0000 7f2127400640 1 -- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20fc06ca90 msgr2=0x7f20fc06ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:11.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.796+0000 7f2127400640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20fc06ca90 0x7f20fc06ef50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2110033000 tx=0x7f211002e000 comp rx=0 tx=0).stop 2024-04-24T08:20:11.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.796+0000 7f2127400640 1 -- 172.21.15.69:0/2434351927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21200994f0 msgr2=0x7f2120099970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:11.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.796+0000 7f2127400640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21200994f0 0x7f2120099970 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f210800e410 tx=0x7f210800e8d0 comp rx=0 tx=0).stop 2024-04-24T08:20:11.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.797+0000 7f2127400640 1 -- 172.21.15.69:0/2434351927 shutdown_connections 2024-04-24T08:20:11.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.797+0000 7f2127400640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f20fc06ca90 0x7f20fc06ef50 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:11.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.797+0000 7f2127400640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f21200994f0 0x7f2120099970 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:11.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.797+0000 7f2127400640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f212015e8c0 0x7f212015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:11.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.797+0000 7f2127400640 1 --2- 172.21.15.69:0/2434351927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21040039b0 0x7f212015a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:11.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.797+0000 7f2127400640 1 -- 172.21.15.69:0/2434351927 >> 172.21.15.69:0/2434351927 conn(0x7f21200d0350 msgr2=0x7f21200b08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:11.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.797+0000 7f2127400640 1 -- 172.21.15.69:0/2434351927 shutdown_connections 2024-04-24T08:20:11.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:11.797+0000 7f2127400640 1 -- 172.21.15.69:0/2434351927 wait complete. 2024-04-24T08:20:11.821 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-24T08:20:11.821 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T08:20:12.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.028+0000 7f884de00640 1 Processor -- start 2024-04-24T08:20:12.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.029+0000 7f884de00640 1 -- start start 2024-04-24T08:20:12.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.030+0000 7f884de00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f884815ea70 0x7f88480d5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:12.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.030+0000 7f884de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 0x7f884806cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:12.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.030+0000 7f884de00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88480d55a0 con 0x7f884806d190 2024-04-24T08:20:12.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.030+0000 7f884de00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f88480d1f20 con 0x7f884815ea70 2024-04-24T08:20:12.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.030+0000 7f884de00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f88480cf810 con 0x7f88480d1960 2024-04-24T08:20:12.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.031+0000 7f8846000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 0x7f884806cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:12.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.031+0000 7f8846a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f884815ea70 0x7f88480d5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.031+0000 7f8846000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 0x7f884806cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:34012/0 (socket says 172.21.15.69:34012) 2024-04-24T08:20:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.031+0000 7f8846000640 1 -- 172.21.15.69:0/377821711 learned_addr learned my addr 172.21.15.69:0/377821711 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.032+0000 7f8846000640 1 -- 172.21.15.69:0/377821711 >> v2:172.21.15.121:3300/0 conn(0x7f884815ea70 msgr2=0x7f88480d5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.032+0000 7f8846000640 1 --2- 172.21.15.69:0/377821711 >> v2:172.21.15.121:3300/0 conn(0x7f884815ea70 0x7f88480d5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.032+0000 7f8846000640 1 -- 172.21.15.69:0/377821711 >> v1:172.21.15.121:6789/0 conn(0x7f884806d190 legacy=0x7f88480d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.032+0000 7f8846000640 1 -- 172.21.15.69:0/377821711 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88480cfaf0 con 0x7f88480d1960 2024-04-24T08:20:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.032+0000 7f8846a00640 1 --2- 172.21.15.69:0/377821711 >> v2:172.21.15.121:3300/0 conn(0x7f884815ea70 0x7f88480d5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:12.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.033+0000 7f8846000640 1 --2- 172.21.15.69:0/377821711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 0x7f884806cc50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e50 tx=0x7f883002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33145ea9b054d130 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:12.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.033+0000 7f8845600640 1 -- 172.21.15.69:0/377821711 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8830036070 con 0x7f88480d1960 2024-04-24T08:20:12.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.033+0000 7f8845600640 1 -- 172.21.15.69:0/377821711 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8830031600 con 0x7f88480d1960 2024-04-24T08:20:12.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.033+0000 7f8845600640 1 -- 172.21.15.69:0/377821711 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88300319d0 con 0x7f88480d1960 2024-04-24T08:20:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.034+0000 7f884de00640 1 -- 172.21.15.69:0/377821711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 msgr2=0x7f884806cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.034+0000 7f884de00640 1 --2- 172.21.15.69:0/377821711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 0x7f884806cc50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e50 tx=0x7f883002a710 comp rx=0 tx=0).stop 2024-04-24T08:20:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.034+0000 7f884de00640 1 -- 172.21.15.69:0/377821711 shutdown_connections 2024-04-24T08:20:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.034+0000 7f884de00640 1 --2- 172.21.15.69:0/377821711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 0x7f884806cc50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.034+0000 7f884de00640 1 --2- 172.21.15.69:0/377821711 >> v2:172.21.15.121:3300/0 conn(0x7f884815ea70 0x7f88480d5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.034+0000 7f884de00640 1 -- 172.21.15.69:0/377821711 >> 172.21.15.69:0/377821711 conn(0x7f884808bf60 msgr2=0x7f88480d46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.034+0000 7f884de00640 1 -- 172.21.15.69:0/377821711 shutdown_connections 2024-04-24T08:20:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.034+0000 7f884de00640 1 -- 172.21.15.69:0/377821711 wait complete. 2024-04-24T08:20:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.036+0000 7f884de00640 1 Processor -- start 2024-04-24T08:20:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.036+0000 7f884de00640 1 -- start start 2024-04-24T08:20:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.037+0000 7f884de00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f884806d190 0x7f884815f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.037+0000 7f884de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 0x7f8848161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.037+0000 7f884de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f884815ea70 0x7f88481616c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.037+0000 7f884de00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f88480cf4c0 con 0x7f884815ea70 2024-04-24T08:20:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.037+0000 7f884de00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f884815ee50 con 0x7f884806d190 2024-04-24T08:20:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.037+0000 7f884de00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f88480d0a00 con 0x7f88480d1960 2024-04-24T08:20:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.037+0000 7f8847400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f884815ea70 0x7f88481616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.037+0000 7f8847400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f884815ea70 0x7f88481616c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54414/0 (socket says 172.21.15.69:54414) 2024-04-24T08:20:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.037+0000 7f8847400640 1 -- 172.21.15.69:0/2281021015 learned_addr learned my addr 172.21.15.69:0/2281021015 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f8846000640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 0x7f8848161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f8846a00640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f884806d190 0x7f884815f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f8847400640 1 -- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 msgr2=0x7f8848161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f8847400640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 0x7f8848161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f8847400640 1 -- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f884806d190 msgr2=0x7f884815f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f8847400640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f884806d190 0x7f884815f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f8847400640 1 -- 172.21.15.69:0/2281021015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8848161d00 con 0x7f884815ea70 2024-04-24T08:20:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f8846a00640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f884806d190 0x7f884815f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:12.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f8847400640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f884815ea70 0x7f88481616c0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f884000c3c0 tx=0x7f884000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:12.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f883f400640 1 -- 172.21.15.69:0/2281021015 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88400052d0 con 0x7f884815ea70 2024-04-24T08:20:12.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f883f400640 1 -- 172.21.15.69:0/2281021015 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8840005470 con 0x7f884815ea70 2024-04-24T08:20:12.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f884de00640 1 -- 172.21.15.69:0/2281021015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88480c82f0 con 0x7f884815ea70 2024-04-24T08:20:12.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.038+0000 7f883f400640 1 -- 172.21.15.69:0/2281021015 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8840007260 con 0x7f884815ea70 2024-04-24T08:20:12.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.039+0000 7f884de00640 1 -- 172.21.15.69:0/2281021015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88480c88b0 con 0x7f884815ea70 2024-04-24T08:20:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.040+0000 7f884de00640 1 -- 172.21.15.69:0/2281021015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8848068a80 con 0x7f884815ea70 2024-04-24T08:20:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.041+0000 7f883f400640 1 -- 172.21.15.69:0/2281021015 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8840007400 con 0x7f884815ea70 2024-04-24T08:20:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.042+0000 7f883f400640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882406c8f0 0x7f882406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.042+0000 7f883f400640 1 -- 172.21.15.69:0/2281021015 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f884008af00 con 0x7f884815ea70 2024-04-24T08:20:12.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.042+0000 7f8846a00640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882406c8f0 0x7f882406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:12.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.043+0000 7f8846a00640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882406c8f0 0x7f882406edb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8838009ac0 tx=0x7f8838008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.046+0000 7f883f400640 1 -- 172.21.15.69:0/2281021015 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8840059330 con 0x7f884815ea70 2024-04-24T08:20:12.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.287+0000 7f884de00640 1 -- 172.21.15.69:0/2281021015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f884815db70 con 0x7f884815ea70 2024-04-24T08:20:12.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.805+0000 7f883f400640 1 -- 172.21.15.69:0/2281021015 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v99) ==== 122+0+0 (secure 0 0 0) 0x7f8840058c30 con 0x7f884815ea70 2024-04-24T08:20:12.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.809+0000 7f884de00640 1 -- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882406c8f0 msgr2=0x7f882406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:12.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.809+0000 7f884de00640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882406c8f0 0x7f882406edb0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f8838009ac0 tx=0x7f8838008040 comp rx=0 tx=0).stop 2024-04-24T08:20:12.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.809+0000 7f884de00640 1 -- 172.21.15.69:0/2281021015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f884815ea70 msgr2=0x7f88481616c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:12.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.809+0000 7f884de00640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f884815ea70 0x7f88481616c0 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f884000c3c0 tx=0x7f884000c880 comp rx=0 tx=0).stop 2024-04-24T08:20:12.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.810+0000 7f884de00640 1 -- 172.21.15.69:0/2281021015 shutdown_connections 2024-04-24T08:20:12.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.810+0000 7f884de00640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f882406c8f0 0x7f882406edb0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:12.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.810+0000 7f884de00640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f884815ea70 0x7f88481616c0 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:12.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.810+0000 7f884de00640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88480d1960 0x7f8848161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:12.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.810+0000 7f884de00640 1 --2- 172.21.15.69:0/2281021015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f884806d190 0x7f884815f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:12.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.810+0000 7f884de00640 1 -- 172.21.15.69:0/2281021015 >> 172.21.15.69:0/2281021015 conn(0x7f884808bf60 msgr2=0x7f884806d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:12.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.810+0000 7f884de00640 1 -- 172.21.15.69:0/2281021015 shutdown_connections 2024-04-24T08:20:12.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:12.810+0000 7f884de00640 1 -- 172.21.15.69:0/2281021015 wait complete. 2024-04-24T08:20:12.832 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T08:20:12.833 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:13.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.032+0000 7f78bf200640 1 Processor -- start 2024-04-24T08:20:13.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.032+0000 7f78bf200640 1 -- start start 2024-04-24T08:20:13.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.033+0000 7f78bf200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c01523f0 0x7f78c006db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.033+0000 7f78bf200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78c0072e20 0x7f78c0073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.033+0000 7f78bf200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78c00aeac0 con 0x7f78c00737a0 2024-04-24T08:20:13.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.033+0000 7f78bf200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f78c00aeda0 con 0x7f78c0072e20 2024-04-24T08:20:13.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.033+0000 7f78bf200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f78c00af080 con 0x7f78c01523f0 2024-04-24T08:20:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.034+0000 7f78be800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f78c00737a0 0x7f78c015ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43224/0 (socket says 172.21.15.69:43224) 2024-04-24T08:20:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.034+0000 7f78be800640 1 -- 172.21.15.69:0/3706272463 learned_addr learned my addr 172.21.15.69:0/3706272463 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.034+0000 7f78bd400640 1 --2- 172.21.15.69:0/3706272463 >> v2:172.21.15.69:3300/0 conn(0x7f78c0072e20 0x7f78c0073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.034+0000 7f78bde00640 1 --2- 172.21.15.69:0/3706272463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c01523f0 0x7f78c006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.034+0000 7f78bd400640 1 -- 172.21.15.69:0/3706272463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c01523f0 msgr2=0x7f78c006db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.034+0000 7f78bd400640 1 --2- 172.21.15.69:0/3706272463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c01523f0 0x7f78c006db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.034+0000 7f78bd400640 1 -- 172.21.15.69:0/3706272463 >> v1:172.21.15.69:6789/0 conn(0x7f78c00737a0 legacy=0x7f78c015ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.034+0000 7f78bca00640 1 -- 172.21.15.69:0/3706272463 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2078275645 0 0) 0x7f78c00aeac0 con 0x7f78c00737a0 2024-04-24T08:20:13.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.034+0000 7f78bd400640 1 -- 172.21.15.69:0/3706272463 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c00af360 con 0x7f78c0072e20 2024-04-24T08:20:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.034+0000 7f78bde00640 1 --2- 172.21.15.69:0/3706272463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c01523f0 0x7f78c006db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.035+0000 7f78bd400640 1 --2- 172.21.15.69:0/3706272463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c0072e20 0x7f78c0073260 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0097a0 tx=0x7f78ac02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f96a40717b8590 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bca00640 1 -- 172.21.15.69:0/3706272463 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac037070 con 0x7f78c0072e20 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bca00640 1 -- 172.21.15.69:0/3706272463 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ac02ae50 con 0x7f78c0072e20 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bca00640 1 -- 172.21.15.69:0/3706272463 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ac032660 con 0x7f78c0072e20 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bf200640 1 -- 172.21.15.69:0/3706272463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c0072e20 msgr2=0x7f78c0073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bf200640 1 --2- 172.21.15.69:0/3706272463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c0072e20 0x7f78c0073260 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0097a0 tx=0x7f78ac02a100 comp rx=0 tx=0).stop 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bf200640 1 -- 172.21.15.69:0/3706272463 shutdown_connections 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bf200640 1 --2- 172.21.15.69:0/3706272463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c0072e20 0x7f78c0073260 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bf200640 1 --2- 172.21.15.69:0/3706272463 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c01523f0 0x7f78c006db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bf200640 1 -- 172.21.15.69:0/3706272463 >> 172.21.15.69:0/3706272463 conn(0x7f78c00d0160 msgr2=0x7f78c00d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bf200640 1 -- 172.21.15.69:0/3706272463 shutdown_connections 2024-04-24T08:20:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.036+0000 7f78bf200640 1 -- 172.21.15.69:0/3706272463 wait complete. 2024-04-24T08:20:13.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.038+0000 7f78bf200640 1 Processor -- start 2024-04-24T08:20:13.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.038+0000 7f78bf200640 1 -- start start 2024-04-24T08:20:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.039+0000 7f78bf200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c0072e20 0x7f78c014d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.039+0000 7f78bf200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f78c00737a0 0x7f78c0156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.039+0000 7f78bf200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c01523f0 0x7f78c0159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.039+0000 7f78bf200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f78c015f680 con 0x7f78c01523f0 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.039+0000 7f78bf200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f78c01527d0 con 0x7f78c00737a0 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.039+0000 7f78bf200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f78c015f800 con 0x7f78c0072e20 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.039+0000 7f78be800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c01523f0 0x7f78c0159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.039+0000 7f78be800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c01523f0 0x7f78c0159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54438/0 (socket says 172.21.15.69:54438) 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.039+0000 7f78be800640 1 -- 172.21.15.69:0/2702516074 learned_addr learned my addr 172.21.15.69:0/2702516074 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78bd400640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f78c00737a0 0x7f78c0156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78bde00640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c0072e20 0x7f78c014d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78be800640 1 -- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c0072e20 msgr2=0x7f78c014d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78be800640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c0072e20 0x7f78c014d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78be800640 1 -- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f78c00737a0 msgr2=0x7f78c0156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78be800640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f78c00737a0 0x7f78c0156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78be800640 1 -- 172.21.15.69:0/2702516074 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c0156500 con 0x7f78c01523f0 2024-04-24T08:20:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78bde00640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c0072e20 0x7f78c014d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:13.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78be800640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c01523f0 0x7f78c0159b20 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c7e0 tx=0x7f78b800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78b6000640 1 -- 172.21.15.69:0/2702516074 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b8018070 con 0x7f78c01523f0 2024-04-24T08:20:13.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78b6000640 1 -- 172.21.15.69:0/2702516074 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78b800a640 con 0x7f78c01523f0 2024-04-24T08:20:13.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.040+0000 7f78b6000640 1 -- 172.21.15.69:0/2702516074 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78b800aa30 con 0x7f78c01523f0 2024-04-24T08:20:13.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.041+0000 7f78bf200640 1 -- 172.21.15.69:0/2702516074 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78c01f4af0 con 0x7f78c01523f0 2024-04-24T08:20:13.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.041+0000 7f78bf200640 1 -- 172.21.15.69:0/2702516074 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78c014da80 con 0x7f78c01523f0 2024-04-24T08:20:13.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.042+0000 7f78bf200640 1 -- 172.21.15.69:0/2702516074 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78c0075d50 con 0x7f78c01523f0 2024-04-24T08:20:13.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.043+0000 7f78b6000640 1 -- 172.21.15.69:0/2702516074 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f78b801b070 con 0x7f78c01523f0 2024-04-24T08:20:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.044+0000 7f78b6000640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78a006c970 0x7f78a006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.044+0000 7f78b6000640 1 -- 172.21.15.69:0/2702516074 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f78b808b130 con 0x7f78c01523f0 2024-04-24T08:20:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.044+0000 7f78bde00640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78a006c970 0x7f78a006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.046+0000 7f78bde00640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78a006c970 0x7f78a006ee30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f78b0006fd0 tx=0x7f78b0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.048+0000 7f78b6000640 1 -- 172.21.15.69:0/2702516074 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78b80596d0 con 0x7f78c01523f0 2024-04-24T08:20:13.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.285+0000 7f78bf200640 1 -- 172.21.15.69:0/2702516074 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78c00b35b0 con 0x7f78c01523f0 2024-04-24T08:20:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.286+0000 7f78b6000640 1 -- 172.21.15.69:0/2702516074 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5186 (secure 0 0 0) 0x7f78b8058fd0 con 0x7f78c01523f0 2024-04-24T08:20:13.287 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:13.287 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2194021008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2194021008},{"type":"v1","addr":"172.21.15.69:6835","nonce":2194021008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:20:12.288476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9288":{"gid":9288,"name":"b","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6836/1515102940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1515102940},{"type":"v1","addr":"172.21.15.121:6836","nonce":1515102940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9288,"qdb_cluster":[9288]},"id":8}]} 2024-04-24T08:20:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 99 2024-04-24T08:20:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 -- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78a006c970 msgr2=0x7f78a006ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78a006c970 0x7f78a006ee30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f78b0006fd0 tx=0x7f78b0008040 comp rx=0 tx=0).stop 2024-04-24T08:20:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 -- 172.21.15.69:0/2702516074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c01523f0 msgr2=0x7f78c0159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c01523f0 0x7f78c0159b20 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c7e0 tx=0x7f78b800cca0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 -- 172.21.15.69:0/2702516074 shutdown_connections 2024-04-24T08:20:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f78a006c970 0x7f78a006ee30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78c01523f0 0x7f78c0159b20 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f78c00737a0 0x7f78c0156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 --2- 172.21.15.69:0/2702516074 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78c0072e20 0x7f78c014d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 -- 172.21.15.69:0/2702516074 >> 172.21.15.69:0/2702516074 conn(0x7f78c00d0160 msgr2=0x7f78c0071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:13.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 -- 172.21.15.69:0/2702516074 shutdown_connections 2024-04-24T08:20:13.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.289+0000 7f78bf200640 1 -- 172.21.15.69:0/2702516074 wait complete. 2024-04-24T08:20:13.309 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:13.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.506+0000 7f955fe00640 1 Processor -- start 2024-04-24T08:20:13.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.506+0000 7f955fe00640 1 -- start start 2024-04-24T08:20:13.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.507+0000 7f955fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95601523f0 0x7f956006db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.507+0000 7f955fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9560072e20 0x7f9560073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.507+0000 7f955fe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95600aeac0 con 0x7f95600737a0 2024-04-24T08:20:13.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.507+0000 7f955fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f95600aeda0 con 0x7f9560072e20 2024-04-24T08:20:13.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.507+0000 7f955fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f95600af080 con 0x7f95601523f0 2024-04-24T08:20:13.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.508+0000 7f955e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9560072e20 0x7f9560073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.508+0000 7f955e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9560072e20 0x7f9560073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54454/0 (socket says 172.21.15.69:54454) 2024-04-24T08:20:13.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.508+0000 7f955e000640 1 -- 172.21.15.69:0/3418272248 learned_addr learned my addr 172.21.15.69:0/3418272248 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:13.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.508+0000 7f955ea00640 1 --2- 172.21.15.69:0/3418272248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95601523f0 0x7f956006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.508+0000 7f955e000640 1 -- 172.21.15.69:0/3418272248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95601523f0 msgr2=0x7f956006db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:13.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.508+0000 7f955e000640 1 --2- 172.21.15.69:0/3418272248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95601523f0 0x7f956006db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.508+0000 7f955e000640 1 -- 172.21.15.69:0/3418272248 >> v1:172.21.15.121:6789/0 conn(0x7f95600737a0 legacy=0x7f956015ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:13.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.508+0000 7f955e000640 1 -- 172.21.15.69:0/3418272248 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95600af360 con 0x7f9560072e20 2024-04-24T08:20:13.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.509+0000 7f955e000640 1 --2- 172.21.15.69:0/3418272248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9560072e20 0x7f9560073260 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f954c009870 tx=0x7f954c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16356ab7410d17fa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.509+0000 7f955d600640 1 -- 172.21.15.69:0/3418272248 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c037070 con 0x7f9560072e20 2024-04-24T08:20:13.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.510+0000 7f955d600640 1 -- 172.21.15.69:0/3418272248 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f954c032440 con 0x7f9560072e20 2024-04-24T08:20:13.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.510+0000 7f955d600640 1 -- 172.21.15.69:0/3418272248 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c032810 con 0x7f9560072e20 2024-04-24T08:20:13.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.510+0000 7f955fe00640 1 -- 172.21.15.69:0/3418272248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9560072e20 msgr2=0x7f9560073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.510+0000 7f955fe00640 1 --2- 172.21.15.69:0/3418272248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9560072e20 0x7f9560073260 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f954c009870 tx=0x7f954c02a180 comp rx=0 tx=0).stop 2024-04-24T08:20:13.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.510+0000 7f955fe00640 1 -- 172.21.15.69:0/3418272248 shutdown_connections 2024-04-24T08:20:13.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.510+0000 7f955fe00640 1 --2- 172.21.15.69:0/3418272248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9560072e20 0x7f9560073260 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.510+0000 7f955fe00640 1 --2- 172.21.15.69:0/3418272248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f95601523f0 0x7f956006db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.510+0000 7f955fe00640 1 -- 172.21.15.69:0/3418272248 >> 172.21.15.69:0/3418272248 conn(0x7f95600d0160 msgr2=0x7f95600d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:13.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.510+0000 7f955fe00640 1 -- 172.21.15.69:0/3418272248 shutdown_connections 2024-04-24T08:20:13.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.510+0000 7f955fe00640 1 -- 172.21.15.69:0/3418272248 wait complete. 2024-04-24T08:20:13.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.512+0000 7f955fe00640 1 Processor -- start 2024-04-24T08:20:13.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.512+0000 7f955fe00640 1 -- start start 2024-04-24T08:20:13.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.513+0000 7f955fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9560072e20 0x7f95601e7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.513+0000 7f955fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95600737a0 0x7f95600b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.513+0000 7f955fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95601523f0 0x7f9560099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.513+0000 7f955fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f956015f680 con 0x7f95601523f0 2024-04-24T08:20:13.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.513+0000 7f955fe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f95601527d0 con 0x7f95600737a0 2024-04-24T08:20:13.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.513+0000 7f955fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f956015f800 con 0x7f9560072e20 2024-04-24T08:20:13.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.513+0000 7f955f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95601523f0 0x7f9560099510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f955f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95601523f0 0x7f9560099510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54464/0 (socket says 172.21.15.69:54464) 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f955f400640 1 -- 172.21.15.69:0/2219562988 learned_addr learned my addr 172.21.15.69:0/2219562988 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f955ea00640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9560072e20 0x7f95601e7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f955e000640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95600737a0 0x7f95600b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f955f400640 1 -- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9560072e20 msgr2=0x7f95601e7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f955f400640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9560072e20 0x7f95601e7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f955f400640 1 -- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95600737a0 msgr2=0x7f95600b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f955f400640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95600737a0 0x7f95600b3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f955f400640 1 -- 172.21.15.69:0/2219562988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95601e6590 con 0x7f95601523f0 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f955f400640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95601523f0 0x7f9560099510 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f955800c7b0 tx=0x7f955800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.514+0000 7f9557400640 1 -- 172.21.15.69:0/2219562988 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9558004b60 con 0x7f95601523f0 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.515+0000 7f9557400640 1 -- 172.21.15.69:0/2219562988 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9558004d00 con 0x7f95601523f0 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.515+0000 7f9557400640 1 -- 172.21.15.69:0/2219562988 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955800aa30 con 0x7f95601523f0 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.515+0000 7f955fe00640 1 -- 172.21.15.69:0/2219562988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95601e6870 con 0x7f95601523f0 2024-04-24T08:20:13.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.515+0000 7f955fe00640 1 -- 172.21.15.69:0/2219562988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95601e6e30 con 0x7f95601523f0 2024-04-24T08:20:13.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.516+0000 7f955fe00640 1 -- 172.21.15.69:0/2219562988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9560075d50 con 0x7f95601523f0 2024-04-24T08:20:13.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.517+0000 7f9557400640 1 -- 172.21.15.69:0/2219562988 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f955800abd0 con 0x7f95601523f0 2024-04-24T08:20:13.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.518+0000 7f9557400640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f954006c8f0 0x7f954006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.518+0000 7f9557400640 1 -- 172.21.15.69:0/2219562988 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f955808bd80 con 0x7f95601523f0 2024-04-24T08:20:13.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.519+0000 7f955ea00640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f954006c8f0 0x7f954006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.520+0000 7f955ea00640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f954006c8f0 0x7f954006edb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9550006fd0 tx=0x7f9550008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.523+0000 7f9557400640 1 -- 172.21.15.69:0/2219562988 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9558020070 con 0x7f95601523f0 2024-04-24T08:20:13.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.724+0000 7f955fe00640 1 -- 172.21.15.69:0/2219562988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f956015a930 con 0x7f95601523f0 2024-04-24T08:20:13.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.726+0000 7f9557400640 1 -- 172.21.15.69:0/2219562988 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+16603 (secure 0 0 0) 0x7f955805a1a0 con 0x7f95601523f0 2024-04-24T08:20:13.727 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:13.727 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":109,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:20:10.770819+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:13.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 -- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f954006c8f0 msgr2=0x7f954006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f954006c8f0 0x7f954006edb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9550006fd0 tx=0x7f9550008040 comp rx=0 tx=0).stop 2024-04-24T08:20:13.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 -- 172.21.15.69:0/2219562988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95601523f0 msgr2=0x7f9560099510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95601523f0 0x7f9560099510 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f955800c7b0 tx=0x7f955800cc70 comp rx=0 tx=0).stop 2024-04-24T08:20:13.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 -- 172.21.15.69:0/2219562988 shutdown_connections 2024-04-24T08:20:13.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f954006c8f0 0x7f954006edb0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f95601523f0 0x7f9560099510 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95600737a0 0x7f95600b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 --2- 172.21.15.69:0/2219562988 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9560072e20 0x7f95601e7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 -- 172.21.15.69:0/2219562988 >> 172.21.15.69:0/2219562988 conn(0x7f95600d0160 msgr2=0x7f9560072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:13.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 -- 172.21.15.69:0/2219562988 shutdown_connections 2024-04-24T08:20:13.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.729+0000 7f955fe00640 1 -- 172.21.15.69:0/2219562988 wait complete. 2024-04-24T08:20:13.750 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T08:20:13.750 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:20:13.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.945+0000 7f54ba400640 1 Processor -- start 2024-04-24T08:20:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.945+0000 7f54ba400640 1 -- start start 2024-04-24T08:20:13.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.947+0000 7f54ba400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b41631a0 0x7f54b40af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.947+0000 7f54ba400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f54b40af7d0 0x7f54b406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.947+0000 7f54ba400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54b40adab0 con 0x7f54b415ea10 2024-04-24T08:20:13.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.947+0000 7f54ba400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f54b40afc40 con 0x7f54b40af7d0 2024-04-24T08:20:13.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.947+0000 7f54ba400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f54b40add50 con 0x7f54b41631a0 2024-04-24T08:20:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.947+0000 7f54abe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b41631a0 0x7f54b40af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.947+0000 7f54b3e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f54b40af7d0 0x7f54b406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.948+0000 7f54b3e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f54b40af7d0 0x7f54b406dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:53870/0 (socket says 172.21.15.69:53870) 2024-04-24T08:20:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.948+0000 7f54abe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b41631a0 0x7f54b40af290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:34058/0 (socket says 172.21.15.69:34058) 2024-04-24T08:20:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.948+0000 7f54b3e00640 1 -- 172.21.15.69:0/1197239500 learned_addr learned my addr 172.21.15.69:0/1197239500 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.948+0000 7f54b3400640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f54b415ea10 0x7f54b40d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:36774/0 (socket says 172.21.15.69:36774) 2024-04-24T08:20:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.949+0000 7f54abe00640 1 -- 172.21.15.69:0/1197239500 >> v2:172.21.15.121:3300/0 conn(0x7f54b40af7d0 msgr2=0x7f54b406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.949+0000 7f54abe00640 1 --2- 172.21.15.69:0/1197239500 >> v2:172.21.15.121:3300/0 conn(0x7f54b40af7d0 0x7f54b406dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.949+0000 7f54abe00640 1 -- 172.21.15.69:0/1197239500 >> v1:172.21.15.121:6789/0 conn(0x7f54b415ea10 legacy=0x7f54b40d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.949+0000 7f54abe00640 1 -- 172.21.15.69:0/1197239500 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b40ae030 con 0x7f54b41631a0 2024-04-24T08:20:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.949+0000 7f54b3e00640 1 --2- 172.21.15.69:0/1197239500 >> v2:172.21.15.121:3300/0 conn(0x7f54b40af7d0 0x7f54b406dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.950+0000 7f54abe00640 1 --2- 172.21.15.69:0/1197239500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b41631a0 0x7f54b40af290 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f549c0097a0 tx=0x7f549c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e50023f44473fd90 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.950+0000 7f54b2a00640 1 -- 172.21.15.69:0/1197239500 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c037070 con 0x7f54b41631a0 2024-04-24T08:20:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.950+0000 7f54b2a00640 1 -- 172.21.15.69:0/1197239500 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f549c032440 con 0x7f54b41631a0 2024-04-24T08:20:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.950+0000 7f54b2a00640 1 -- 172.21.15.69:0/1197239500 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f549c032810 con 0x7f54b41631a0 2024-04-24T08:20:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.951+0000 7f54ba400640 1 -- 172.21.15.69:0/1197239500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b41631a0 msgr2=0x7f54b40af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.951+0000 7f54ba400640 1 --2- 172.21.15.69:0/1197239500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b41631a0 0x7f54b40af290 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f549c0097a0 tx=0x7f549c02a180 comp rx=0 tx=0).stop 2024-04-24T08:20:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.951+0000 7f54ba400640 1 -- 172.21.15.69:0/1197239500 shutdown_connections 2024-04-24T08:20:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.951+0000 7f54ba400640 1 --2- 172.21.15.69:0/1197239500 >> v2:172.21.15.121:3300/0 conn(0x7f54b40af7d0 0x7f54b406dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.951+0000 7f54ba400640 1 --2- 172.21.15.69:0/1197239500 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b41631a0 0x7f54b40af290 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.951+0000 7f54ba400640 1 -- 172.21.15.69:0/1197239500 >> 172.21.15.69:0/1197239500 conn(0x7f54b4067d90 msgr2=0x7f54b406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.951+0000 7f54ba400640 1 -- 172.21.15.69:0/1197239500 shutdown_connections 2024-04-24T08:20:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.951+0000 7f54ba400640 1 -- 172.21.15.69:0/1197239500 wait complete. 2024-04-24T08:20:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.953+0000 7f54ba400640 1 Processor -- start 2024-04-24T08:20:13.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.953+0000 7f54ba400640 1 -- start start 2024-04-24T08:20:13.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.954+0000 7f54ba400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b40af7d0 0x7f54b41f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.954+0000 7f54ba400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b415ea10 0x7f54b414d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.954+0000 7f54ba400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54b41631a0 0x7f54b41f5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.954+0000 7f54ba400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f54b40ae8d0 con 0x7f54b40af7d0 2024-04-24T08:20:13.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.954+0000 7f54ba400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f54b40ae5f0 con 0x7f54b41631a0 2024-04-24T08:20:13.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.954+0000 7f54ba400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f54b40aedf0 con 0x7f54b415ea10 2024-04-24T08:20:13.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.954+0000 7f54b3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b40af7d0 0x7f54b41f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.954+0000 7f54b3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b40af7d0 0x7f54b41f50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54466/0 (socket says 172.21.15.69:54466) 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.954+0000 7f54b3400640 1 -- 172.21.15.69:0/4027185791 learned_addr learned my addr 172.21.15.69:0/4027185791 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54b3400640 1 -- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b415ea10 msgr2=0x7f54b414d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54b3400640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b415ea10 0x7f54b414d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54b3400640 1 -- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54b41631a0 msgr2=0x7f54b41f5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54b3400640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54b41631a0 0x7f54b41f5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54b3400640 1 -- 172.21.15.69:0/4027185791 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54b41f5bc0 con 0x7f54b40af7d0 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54b3400640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b40af7d0 0x7f54b41f50f0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f54a400c400 tx=0x7f54a400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54ab400640 1 -- 172.21.15.69:0/4027185791 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a40074e0 con 0x7f54b40af7d0 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54ab400640 1 -- 172.21.15.69:0/4027185791 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54a4007680 con 0x7f54b40af7d0 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54ab400640 1 -- 172.21.15.69:0/4027185791 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a4004a00 con 0x7f54b40af7d0 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54ba400640 1 -- 172.21.15.69:0/4027185791 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54b40b1d70 con 0x7f54b40af7d0 2024-04-24T08:20:13.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.955+0000 7f54ba400640 1 -- 172.21.15.69:0/4027185791 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54b40cdee0 con 0x7f54b40af7d0 2024-04-24T08:20:13.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.956+0000 7f54ba400640 1 -- 172.21.15.69:0/4027185791 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54b4068a80 con 0x7f54b40af7d0 2024-04-24T08:20:13.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.957+0000 7f54ab400640 1 -- 172.21.15.69:0/4027185791 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f54a401a020 con 0x7f54b40af7d0 2024-04-24T08:20:13.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.959+0000 7f54ab400640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f549006c8f0 0x7f549006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:13.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.959+0000 7f54ab400640 1 -- 172.21.15.69:0/4027185791 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f54a408b3e0 con 0x7f54b40af7d0 2024-04-24T08:20:13.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.959+0000 7f54abe00640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f549006c8f0 0x7f549006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:13.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.960+0000 7f54abe00640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f549006c8f0 0x7f549006edb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f549c034000 tx=0x7f549c02c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:13.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:13.963+0000 7f54ab400640 1 -- 172.21.15.69:0/4027185791 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54a4010300 con 0x7f54b40af7d0 2024-04-24T08:20:14.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.160+0000 7f54ba400640 1 -- 172.21.15.69:0/4027185791 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f54b406ddd0 con 0x7f549006c8f0 2024-04-24T08:20:14.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.180+0000 7f54ab400640 1 -- 172.21.15.69:0/4027185791 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247087 (secure 0 0 0) 0x7f54b406ddd0 con 0x7f549006c8f0 2024-04-24T08:20:14.185 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:14.186 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:20:14.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.186+0000 7f54ba400640 1 -- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f549006c8f0 msgr2=0x7f549006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.186+0000 7f54ba400640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f549006c8f0 0x7f549006edb0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f549c034000 tx=0x7f549c02c000 comp rx=0 tx=0).stop 2024-04-24T08:20:14.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.186+0000 7f54ba400640 1 -- 172.21.15.69:0/4027185791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b40af7d0 msgr2=0x7f54b41f50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.186+0000 7f54ba400640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b40af7d0 0x7f54b41f50f0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f54a400c400 tx=0x7f54a400c8c0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.187+0000 7f54ba400640 1 -- 172.21.15.69:0/4027185791 shutdown_connections 2024-04-24T08:20:14.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.187+0000 7f54ba400640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f549006c8f0 0x7f549006edb0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.187+0000 7f54ba400640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54b41631a0 0x7f54b41f5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.187+0000 7f54ba400640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f54b415ea10 0x7f54b414d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.187+0000 7f54ba400640 1 --2- 172.21.15.69:0/4027185791 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54b40af7d0 0x7f54b41f50f0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.187+0000 7f54ba400640 1 -- 172.21.15.69:0/4027185791 >> 172.21.15.69:0/4027185791 conn(0x7f54b4067d90 msgr2=0x7f54b4158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:14.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.187+0000 7f54ba400640 1 -- 172.21.15.69:0/4027185791 shutdown_connections 2024-04-24T08:20:14.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.187+0000 7f54ba400640 1 -- 172.21.15.69:0/4027185791 wait complete. 2024-04-24T08:20:14.206 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-04-24T08:20:12.930388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260932,"kb_used_data":11488,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713788,"statfs":{"total":773094113280,"available":772826918912,"internally_reserved":0,"allocated":11763712,"data_stored":5750782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.200298"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.761304+0000","last_change":"2024-04-24T08:20:08.761304+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.761304+0000","last_undegraded":"2024-04-24T08:20:08.761304+0000","last_fullsized":"2024-04-24T08:20:08.761304+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.740349+0000","last_change":"2024-04-24T08:20:08.740349+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.740349+0000","last_undegraded":"2024-04-24T08:20:08.740349+0000","last_fullsized":"2024-04-24T08:20:08.740349+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.755775+0000","last_change":"2024-04-24T08:20:08.755775+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.755775+0000","last_undegraded":"2024-04-24T08:20:08.755775+0000","last_fullsized":"2024-04-24T08:20:08.755775+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.756093+0000","last_change":"2024-04-24T08:20:08.756093+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.756093+0000","last_undegraded":"2024-04-24T08:20:08.756093+0000","last_fullsized":"2024-04-24T08:20:08.756093+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.755449+0000","last_change":"2024-04-24T08:20:08.755449+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.755449+0000","last_undegraded":"2024-04-24T08:20:08.755449+0000","last_fullsized":"2024-04-24T08:20:08.755449+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.759258+0000","last_change":"2024-04-24T08:20:08.759258+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.759258+0000","last_undegraded":"2024-04-24T08:20:08.759258+0000","last_fullsized":"2024-04-24T08:20:08.759258+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.741095+0000","last_change":"2024-04-24T08:20:08.741095+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.741095+0000","last_undegraded":"2024-04-24T08:20:08.741095+0000","last_fullsized":"2024-04-24T08:20:08.741095+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.759172+0000","last_change":"2024-04-24T08:20:08.759172+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.759172+0000","last_undegraded":"2024-04-24T08:20:08.759172+0000","last_fullsized":"2024-04-24T08:20:08.759172+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.756459+0000","last_change":"2024-04-24T08:20:08.756459+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.756459+0000","last_undegraded":"2024-04-24T08:20:08.756459+0000","last_fullsized":"2024-04-24T08:20:08.756459+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.757303+0000","last_change":"2024-04-24T08:20:08.757303+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.757303+0000","last_undegraded":"2024-04-24T08:20:08.757303+0000","last_fullsized":"2024-04-24T08:20:08.757303+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.742570+0000","last_change":"2024-04-24T08:20:08.742570+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.742570+0000","last_undegraded":"2024-04-24T08:20:08.742570+0000","last_fullsized":"2024-04-24T08:20:08.742570+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754956+0000","last_change":"2024-04-24T08:20:08.754956+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754956+0000","last_undegraded":"2024-04-24T08:20:08.754956+0000","last_fullsized":"2024-04-24T08:20:08.754956+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.752401+0000","last_change":"2024-04-24T08:20:08.752401+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.752401+0000","last_undegraded":"2024-04-24T08:20:08.752401+0000","last_fullsized":"2024-04-24T08:20:08.752401+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753720+0000","last_change":"2024-04-24T08:20:08.753720+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753720+0000","last_undegraded":"2024-04-24T08:20:08.753720+0000","last_fullsized":"2024-04-24T08:20:08.753720+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.757701+0000","last_change":"2024-04-24T08:20:08.757701+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.757701+0000","last_undegraded":"2024-04-24T08:20:08.757701+0000","last_fullsized":"2024-04-24T08:20:08.757701+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.755783+0000","last_change":"2024-04-24T08:20:08.755783+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.755783+0000","last_undegraded":"2024-04-24T08:20:08.755783+0000","last_fullsized":"2024-04-24T08:20:08.755783+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.741980+0000","last_change":"2024-04-24T08:20:08.741980+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.741980+0000","last_undegraded":"2024-04-24T08:20:08.741980+0000","last_fullsized":"2024-04-24T08:20:08.741980+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.751997+0000","last_change":"2024-04-24T08:20:08.751997+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.751997+0000","last_undegraded":"2024-04-24T08:20:08.751997+0000","last_fullsized":"2024-04-24T08:20:08.751997+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754860+0000","last_change":"2024-04-24T08:20:08.754860+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754860+0000","last_undegraded":"2024-04-24T08:20:08.754860+0000","last_fullsized":"2024-04-24T08:20:08.754860+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.740346+0000","last_change":"2024-04-24T08:20:08.740346+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.740346+0000","last_undegraded":"2024-04-24T08:20:08.740346+0000","last_fullsized":"2024-04-24T08:20:08.740346+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.741581+0000","last_change":"2024-04-24T08:20:08.741581+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.741581+0000","last_undegraded":"2024-04-24T08:20:08.741581+0000","last_fullsized":"2024-04-24T08:20:08.741581+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753020+0000","last_change":"2024-04-24T08:20:08.753020+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753020+0000","last_undegraded":"2024-04-24T08:20:08.753020+0000","last_fullsized":"2024-04-24T08:20:08.753020+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.740388+0000","last_change":"2024-04-24T08:20:08.740388+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.740388+0000","last_undegraded":"2024-04-24T08:20:08.740388+0000","last_fullsized":"2024-04-24T08:20:08.740388+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.750565+0000","last_change":"2024-04-24T08:20:08.750565+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.750565+0000","last_undegraded":"2024-04-24T08:20:08.750565+0000","last_fullsized":"2024-04-24T08:20:08.750565+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.740719+0000","last_change":"2024-04-24T08:20:08.740719+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.740719+0000","last_undegraded":"2024-04-24T08:20:08.740719+0000","last_fullsized":"2024-04-24T08:20:08.740719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754870+0000","last_change":"2024-04-24T08:20:08.754870+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754870+0000","last_undegraded":"2024-04-24T08:20:08.754870+0000","last_fullsized":"2024-04-24T08:20:08.754870+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.758148+0000","last_change":"2024-04-24T08:20:08.758148+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.758148+0000","last_undegraded":"2024-04-24T08:20:08.758148+0000","last_fullsized":"2024-04-24T08:20:08.758148+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.755369+0000","last_change":"2024-04-24T08:20:08.755369+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.755369+0000","last_undegraded":"2024-04-24T08:20:08.755369+0000","last_fullsized":"2024-04-24T08:20:08.755369+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.745347+0000","last_change":"2024-04-24T08:20:08.745347+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.745347+0000","last_undegraded":"2024-04-24T08:20:08.745347+0000","last_fullsized":"2024-04-24T08:20:08.745347+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.743311+0000","last_change":"2024-04-24T08:20:08.743311+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.743311+0000","last_undegraded":"2024-04-24T08:20:08.743311+0000","last_fullsized":"2024-04-24T08:20:08.743311+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.755577+0000","last_change":"2024-04-24T08:20:08.755577+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.755577+0000","last_undegraded":"2024-04-24T08:20:08.755577+0000","last_fullsized":"2024-04-24T08:20:08.755577+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.756839+0000","last_change":"2024-04-24T08:20:08.756839+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.756839+0000","last_undegraded":"2024-04-24T08:20:08.756839+0000","last_fullsized":"2024-04-24T08:20:08.756839+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.761419+0000","last_change":"2024-04-24T08:20:08.761419+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.761419+0000","last_undegraded":"2024-04-24T08:20:08.761419+0000","last_fullsized":"2024-04-24T08:20:08.761419+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.768078+0000","last_change":"2024-04-24T08:20:08.765030+0000","last_active":"2024-04-24T08:20:08.768078+0000","last_peered":"2024-04-24T08:20:08.768078+0000","last_clean":"2024-04-24T08:20:08.768078+0000","last_became_active":"2024-04-24T08:20:08.764730+0000","last_became_peered":"2024-04-24T08:20:08.764730+0000","last_unstale":"2024-04-24T08:20:08.768078+0000","last_undegraded":"2024-04-24T08:20:08.768078+0000","last_fullsized":"2024-04-24T08:20:08.768078+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:12:45.289874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754916+0000","last_change":"2024-04-24T08:20:08.754916+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754916+0000","last_undegraded":"2024-04-24T08:20:08.754916+0000","last_fullsized":"2024-04-24T08:20:08.754916+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.767267+0000","last_change":"2024-04-24T08:20:08.764950+0000","last_active":"2024-04-24T08:20:08.767267+0000","last_peered":"2024-04-24T08:20:08.767267+0000","last_clean":"2024-04-24T08:20:08.767267+0000","last_became_active":"2024-04-24T08:20:08.764618+0000","last_became_peered":"2024-04-24T08:20:08.764618+0000","last_unstale":"2024-04-24T08:20:08.767267+0000","last_undegraded":"2024-04-24T08:20:08.767267+0000","last_fullsized":"2024-04-24T08:20:08.767267+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:33.902279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.756452+0000","last_change":"2024-04-24T08:20:08.756452+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.756452+0000","last_undegraded":"2024-04-24T08:20:08.756452+0000","last_fullsized":"2024-04-24T08:20:08.756452+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.774371+0000","last_change":"2024-04-24T08:20:08.772932+0000","last_active":"2024-04-24T08:20:08.774371+0000","last_peered":"2024-04-24T08:20:08.774371+0000","last_clean":"2024-04-24T08:20:08.774371+0000","last_became_active":"2024-04-24T08:20:08.772234+0000","last_became_peered":"2024-04-24T08:20:08.772234+0000","last_unstale":"2024-04-24T08:20:08.774371+0000","last_undegraded":"2024-04-24T08:20:08.774371+0000","last_fullsized":"2024-04-24T08:20:08.774371+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:54.564513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753481+0000","last_change":"2024-04-24T08:20:08.753481+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753481+0000","last_undegraded":"2024-04-24T08:20:08.753481+0000","last_fullsized":"2024-04-24T08:20:08.753481+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772429+0000","last_change":"2024-04-24T08:20:08.770132+0000","last_active":"2024-04-24T08:20:08.772429+0000","last_peered":"2024-04-24T08:20:08.772429+0000","last_clean":"2024-04-24T08:20:08.772429+0000","last_became_active":"2024-04-24T08:20:08.769756+0000","last_became_peered":"2024-04-24T08:20:08.769756+0000","last_unstale":"2024-04-24T08:20:08.772429+0000","last_undegraded":"2024-04-24T08:20:08.772429+0000","last_fullsized":"2024-04-24T08:20:08.772429+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:15:56.351743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.749058+0000","last_change":"2024-04-24T08:20:08.749058+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.749058+0000","last_undegraded":"2024-04-24T08:20:08.749058+0000","last_fullsized":"2024-04-24T08:20:08.749058+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.768047+0000","last_change":"2024-04-24T08:20:08.763740+0000","last_active":"2024-04-24T08:20:08.768047+0000","last_peered":"2024-04-24T08:20:08.768047+0000","last_clean":"2024-04-24T08:20:08.768047+0000","last_became_active":"2024-04-24T08:20:08.763472+0000","last_became_peered":"2024-04-24T08:20:08.763472+0000","last_unstale":"2024-04-24T08:20:08.768047+0000","last_undegraded":"2024-04-24T08:20:08.768047+0000","last_fullsized":"2024-04-24T08:20:08.768047+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:52:34.264489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.744129+0000","last_change":"2024-04-24T08:20:08.744129+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.744129+0000","last_undegraded":"2024-04-24T08:20:08.744129+0000","last_fullsized":"2024-04-24T08:20:08.744129+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.764276+0000","last_change":"2024-04-24T08:20:08.762014+0000","last_active":"2024-04-24T08:20:08.764276+0000","last_peered":"2024-04-24T08:20:08.764276+0000","last_clean":"2024-04-24T08:20:08.764276+0000","last_became_active":"2024-04-24T08:20:08.761675+0000","last_became_peered":"2024-04-24T08:20:08.761675+0000","last_unstale":"2024-04-24T08:20:08.764276+0000","last_undegraded":"2024-04-24T08:20:08.764276+0000","last_fullsized":"2024-04-24T08:20:08.764276+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:02:41.460096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.745815+0000","last_change":"2024-04-24T08:20:08.745815+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.745815+0000","last_undegraded":"2024-04-24T08:20:08.745815+0000","last_fullsized":"2024-04-24T08:20:08.745815+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.759411+0000","last_change":"2024-04-24T08:20:08.755417+0000","last_active":"2024-04-24T08:20:08.759411+0000","last_peered":"2024-04-24T08:20:08.759411+0000","last_clean":"2024-04-24T08:20:08.759411+0000","last_became_active":"2024-04-24T08:20:08.755131+0000","last_became_peered":"2024-04-24T08:20:08.755131+0000","last_unstale":"2024-04-24T08:20:08.759411+0000","last_undegraded":"2024-04-24T08:20:08.759411+0000","last_fullsized":"2024-04-24T08:20:08.759411+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:34:35.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.742263+0000","last_change":"2024-04-24T08:20:08.742263+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.742263+0000","last_undegraded":"2024-04-24T08:20:08.742263+0000","last_fullsized":"2024-04-24T08:20:08.742263+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772216+0000","last_change":"2024-04-24T08:20:08.764423+0000","last_active":"2024-04-24T08:20:08.772216+0000","last_peered":"2024-04-24T08:20:08.772216+0000","last_clean":"2024-04-24T08:20:08.772216+0000","last_became_active":"2024-04-24T08:20:08.764017+0000","last_became_peered":"2024-04-24T08:20:08.764017+0000","last_unstale":"2024-04-24T08:20:08.772216+0000","last_undegraded":"2024-04-24T08:20:08.772216+0000","last_fullsized":"2024-04-24T08:20:08.772216+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:26:30.942475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.747438+0000","last_change":"2024-04-24T08:20:08.747438+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.747438+0000","last_undegraded":"2024-04-24T08:20:08.747438+0000","last_fullsized":"2024-04-24T08:20:08.747438+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.771544+0000","last_change":"2024-04-24T08:20:08.768692+0000","last_active":"2024-04-24T08:20:08.771544+0000","last_peered":"2024-04-24T08:20:08.771544+0000","last_clean":"2024-04-24T08:20:08.771544+0000","last_became_active":"2024-04-24T08:20:08.768352+0000","last_became_peered":"2024-04-24T08:20:08.768352+0000","last_unstale":"2024-04-24T08:20:08.771544+0000","last_undegraded":"2024-04-24T08:20:08.771544+0000","last_fullsized":"2024-04-24T08:20:08.771544+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:48.970516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753718+0000","last_change":"2024-04-24T08:20:08.753718+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753718+0000","last_undegraded":"2024-04-24T08:20:08.753718+0000","last_fullsized":"2024-04-24T08:20:08.753718+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.770760+0000","last_change":"2024-04-24T08:20:08.767491+0000","last_active":"2024-04-24T08:20:08.770760+0000","last_peered":"2024-04-24T08:20:08.770760+0000","last_clean":"2024-04-24T08:20:08.770760+0000","last_became_active":"2024-04-24T08:20:08.767200+0000","last_became_peered":"2024-04-24T08:20:08.767200+0000","last_unstale":"2024-04-24T08:20:08.770760+0000","last_undegraded":"2024-04-24T08:20:08.770760+0000","last_fullsized":"2024-04-24T08:20:08.770760+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:41.544350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754342+0000","last_change":"2024-04-24T08:20:08.754342+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754342+0000","last_undegraded":"2024-04-24T08:20:08.754342+0000","last_fullsized":"2024-04-24T08:20:08.754342+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.775190+0000","last_change":"2024-04-24T08:20:08.771750+0000","last_active":"2024-04-24T08:20:08.775190+0000","last_peered":"2024-04-24T08:20:08.775190+0000","last_clean":"2024-04-24T08:20:08.775190+0000","last_became_active":"2024-04-24T08:20:08.771450+0000","last_became_peered":"2024-04-24T08:20:08.771450+0000","last_unstale":"2024-04-24T08:20:08.775190+0000","last_undegraded":"2024-04-24T08:20:08.775190+0000","last_fullsized":"2024-04-24T08:20:08.775190+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:46.509284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.744328+0000","last_change":"2024-04-24T08:20:08.744328+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.744328+0000","last_undegraded":"2024-04-24T08:20:08.744328+0000","last_fullsized":"2024-04-24T08:20:08.744328+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.771114+0000","last_change":"2024-04-24T08:20:08.763054+0000","last_active":"2024-04-24T08:20:08.771114+0000","last_peered":"2024-04-24T08:20:08.771114+0000","last_clean":"2024-04-24T08:20:08.771114+0000","last_became_active":"2024-04-24T08:20:08.762785+0000","last_became_peered":"2024-04-24T08:20:08.762785+0000","last_unstale":"2024-04-24T08:20:08.771114+0000","last_undegraded":"2024-04-24T08:20:08.771114+0000","last_fullsized":"2024-04-24T08:20:08.771114+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:52:34.850318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.752357+0000","last_change":"2024-04-24T08:20:08.752357+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.752357+0000","last_undegraded":"2024-04-24T08:20:08.752357+0000","last_fullsized":"2024-04-24T08:20:08.752357+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.774116+0000","last_change":"2024-04-24T08:20:08.771246+0000","last_active":"2024-04-24T08:20:08.774116+0000","last_peered":"2024-04-24T08:20:08.774116+0000","last_clean":"2024-04-24T08:20:08.774116+0000","last_became_active":"2024-04-24T08:20:08.770915+0000","last_became_peered":"2024-04-24T08:20:08.770915+0000","last_unstale":"2024-04-24T08:20:08.774116+0000","last_undegraded":"2024-04-24T08:20:08.774116+0000","last_fullsized":"2024-04-24T08:20:08.774116+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:01.949509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753899+0000","last_change":"2024-04-24T08:20:08.753899+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753899+0000","last_undegraded":"2024-04-24T08:20:08.753899+0000","last_fullsized":"2024-04-24T08:20:08.753899+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.771979+0000","last_change":"2024-04-24T08:20:08.770641+0000","last_active":"2024-04-24T08:20:08.771979+0000","last_peered":"2024-04-24T08:20:08.771979+0000","last_clean":"2024-04-24T08:20:08.771979+0000","last_became_active":"2024-04-24T08:20:08.770281+0000","last_became_peered":"2024-04-24T08:20:08.770281+0000","last_unstale":"2024-04-24T08:20:08.771979+0000","last_undegraded":"2024-04-24T08:20:08.771979+0000","last_fullsized":"2024-04-24T08:20:08.771979+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:52:34.850318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.751348+0000","last_change":"2024-04-24T08:20:08.751348+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.751348+0000","last_undegraded":"2024-04-24T08:20:08.751348+0000","last_fullsized":"2024-04-24T08:20:08.751348+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.765894+0000","last_change":"2024-04-24T08:20:08.760398+0000","last_active":"2024-04-24T08:20:08.765894+0000","last_peered":"2024-04-24T08:20:08.765894+0000","last_clean":"2024-04-24T08:20:08.765894+0000","last_became_active":"2024-04-24T08:20:08.760075+0000","last_became_peered":"2024-04-24T08:20:08.760075+0000","last_unstale":"2024-04-24T08:20:08.765894+0000","last_undegraded":"2024-04-24T08:20:08.765894+0000","last_fullsized":"2024-04-24T08:20:08.765894+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:46:50.458288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.752990+0000","last_change":"2024-04-24T08:20:08.752990+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.752990+0000","last_undegraded":"2024-04-24T08:20:08.752990+0000","last_fullsized":"2024-04-24T08:20:08.752990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.775165+0000","last_change":"2024-04-24T08:20:08.769979+0000","last_active":"2024-04-24T08:20:08.775165+0000","last_peered":"2024-04-24T08:20:08.775165+0000","last_clean":"2024-04-24T08:20:08.775165+0000","last_became_active":"2024-04-24T08:20:08.769667+0000","last_became_peered":"2024-04-24T08:20:08.769667+0000","last_unstale":"2024-04-24T08:20:08.775165+0000","last_undegraded":"2024-04-24T08:20:08.775165+0000","last_fullsized":"2024-04-24T08:20:08.775165+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:20.634892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.761130+0000","last_change":"2024-04-24T08:20:08.761130+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.761130+0000","last_undegraded":"2024-04-24T08:20:08.761130+0000","last_fullsized":"2024-04-24T08:20:08.761130+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.765621+0000","last_change":"2024-04-24T08:20:08.762510+0000","last_active":"2024-04-24T08:20:08.765621+0000","last_peered":"2024-04-24T08:20:08.765621+0000","last_clean":"2024-04-24T08:20:08.765621+0000","last_became_active":"2024-04-24T08:20:08.762168+0000","last_became_peered":"2024-04-24T08:20:08.762168+0000","last_unstale":"2024-04-24T08:20:08.765621+0000","last_undegraded":"2024-04-24T08:20:08.765621+0000","last_fullsized":"2024-04-24T08:20:08.765621+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:28:16.020468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754069+0000","last_change":"2024-04-24T08:20:08.754069+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754069+0000","last_undegraded":"2024-04-24T08:20:08.754069+0000","last_fullsized":"2024-04-24T08:20:08.754069+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.763758+0000","last_change":"2024-04-24T08:20:08.761477+0000","last_active":"2024-04-24T08:20:08.763758+0000","last_peered":"2024-04-24T08:20:08.763758+0000","last_clean":"2024-04-24T08:20:08.763758+0000","last_became_active":"2024-04-24T08:20:08.759812+0000","last_became_peered":"2024-04-24T08:20:08.759812+0000","last_unstale":"2024-04-24T08:20:08.763758+0000","last_undegraded":"2024-04-24T08:20:08.763758+0000","last_fullsized":"2024-04-24T08:20:08.763758+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:10:49.534393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.752461+0000","last_change":"2024-04-24T08:20:08.752461+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.752461+0000","last_undegraded":"2024-04-24T08:20:08.752461+0000","last_fullsized":"2024-04-24T08:20:08.752461+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.768460+0000","last_change":"2024-04-24T08:20:08.767284+0000","last_active":"2024-04-24T08:20:08.768460+0000","last_peered":"2024-04-24T08:20:08.768460+0000","last_clean":"2024-04-24T08:20:08.768460+0000","last_became_active":"2024-04-24T08:20:08.766880+0000","last_became_peered":"2024-04-24T08:20:08.766880+0000","last_unstale":"2024-04-24T08:20:08.768460+0000","last_undegraded":"2024-04-24T08:20:08.768460+0000","last_fullsized":"2024-04-24T08:20:08.768460+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:27:25.539395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753173+0000","last_change":"2024-04-24T08:20:08.753173+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753173+0000","last_undegraded":"2024-04-24T08:20:08.753173+0000","last_fullsized":"2024-04-24T08:20:08.753173+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.775773+0000","last_change":"2024-04-24T08:20:08.773991+0000","last_active":"2024-04-24T08:20:08.775773+0000","last_peered":"2024-04-24T08:20:08.775773+0000","last_clean":"2024-04-24T08:20:08.775773+0000","last_became_active":"2024-04-24T08:20:08.773703+0000","last_became_peered":"2024-04-24T08:20:08.773703+0000","last_unstale":"2024-04-24T08:20:08.775773+0000","last_undegraded":"2024-04-24T08:20:08.775773+0000","last_fullsized":"2024-04-24T08:20:08.775773+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:08.415565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.746401+0000","last_change":"2024-04-24T08:20:08.746401+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.746401+0000","last_undegraded":"2024-04-24T08:20:08.746401+0000","last_fullsized":"2024-04-24T08:20:08.746401+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772260+0000","last_change":"2024-04-24T08:20:08.769543+0000","last_active":"2024-04-24T08:20:08.772260+0000","last_peered":"2024-04-24T08:20:08.772260+0000","last_clean":"2024-04-24T08:20:08.772260+0000","last_became_active":"2024-04-24T08:20:08.767940+0000","last_became_peered":"2024-04-24T08:20:08.767940+0000","last_unstale":"2024-04-24T08:20:08.772260+0000","last_undegraded":"2024-04-24T08:20:08.772260+0000","last_fullsized":"2024-04-24T08:20:08.772260+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:54:03.023190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754926+0000","last_change":"2024-04-24T08:20:08.754926+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754926+0000","last_undegraded":"2024-04-24T08:20:08.754926+0000","last_fullsized":"2024-04-24T08:20:08.754926+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772376+0000","last_change":"2024-04-24T08:20:08.769318+0000","last_active":"2024-04-24T08:20:08.772376+0000","last_peered":"2024-04-24T08:20:08.772376+0000","last_clean":"2024-04-24T08:20:08.772376+0000","last_became_active":"2024-04-24T08:20:08.769050+0000","last_became_peered":"2024-04-24T08:20:08.769050+0000","last_unstale":"2024-04-24T08:20:08.772376+0000","last_undegraded":"2024-04-24T08:20:08.772376+0000","last_fullsized":"2024-04-24T08:20:08.772376+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:27:25.539395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.749824+0000","last_change":"2024-04-24T08:20:08.749824+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.749824+0000","last_undegraded":"2024-04-24T08:20:08.749824+0000","last_fullsized":"2024-04-24T08:20:08.749824+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772258+0000","last_change":"2024-04-24T08:20:08.767252+0000","last_active":"2024-04-24T08:20:08.772258+0000","last_peered":"2024-04-24T08:20:08.772258+0000","last_clean":"2024-04-24T08:20:08.772258+0000","last_became_active":"2024-04-24T08:20:08.766235+0000","last_became_peered":"2024-04-24T08:20:08.766235+0000","last_unstale":"2024-04-24T08:20:08.772258+0000","last_undegraded":"2024-04-24T08:20:08.772258+0000","last_fullsized":"2024-04-24T08:20:08.772258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:27:26.888390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.750732+0000","last_change":"2024-04-24T08:20:08.750732+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.750732+0000","last_undegraded":"2024-04-24T08:20:08.750732+0000","last_fullsized":"2024-04-24T08:20:08.750732+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.775584+0000","last_change":"2024-04-24T08:20:08.774854+0000","last_active":"2024-04-24T08:20:08.775584+0000","last_peered":"2024-04-24T08:20:08.775584+0000","last_clean":"2024-04-24T08:20:08.775584+0000","last_became_active":"2024-04-24T08:20:08.773896+0000","last_became_peered":"2024-04-24T08:20:08.773896+0000","last_unstale":"2024-04-24T08:20:08.775584+0000","last_undegraded":"2024-04-24T08:20:08.775584+0000","last_fullsized":"2024-04-24T08:20:08.775584+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:56.617029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.747005+0000","last_change":"2024-04-24T08:20:08.747005+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.747005+0000","last_undegraded":"2024-04-24T08:20:08.747005+0000","last_fullsized":"2024-04-24T08:20:08.747005+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.767713+0000","last_change":"2024-04-24T08:20:08.763470+0000","last_active":"2024-04-24T08:20:08.767713+0000","last_peered":"2024-04-24T08:20:08.767713+0000","last_clean":"2024-04-24T08:20:08.767713+0000","last_became_active":"2024-04-24T08:20:08.762839+0000","last_became_peered":"2024-04-24T08:20:08.762839+0000","last_unstale":"2024-04-24T08:20:08.767713+0000","last_undegraded":"2024-04-24T08:20:08.767713+0000","last_fullsized":"2024-04-24T08:20:08.767713+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:23:50.528789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.747108+0000","last_change":"2024-04-24T08:20:08.747108+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.747108+0000","last_undegraded":"2024-04-24T08:20:08.747108+0000","last_fullsized":"2024-04-24T08:20:08.747108+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.764788+0000","last_change":"2024-04-24T08:20:08.761630+0000","last_active":"2024-04-24T08:20:08.764788+0000","last_peered":"2024-04-24T08:20:08.764788+0000","last_clean":"2024-04-24T08:20:08.764788+0000","last_became_active":"2024-04-24T08:20:08.761355+0000","last_became_peered":"2024-04-24T08:20:08.761355+0000","last_unstale":"2024-04-24T08:20:08.764788+0000","last_undegraded":"2024-04-24T08:20:08.764788+0000","last_fullsized":"2024-04-24T08:20:08.764788+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:23:50.528789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.751382+0000","last_change":"2024-04-24T08:20:08.751382+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.751382+0000","last_undegraded":"2024-04-24T08:20:08.751382+0000","last_fullsized":"2024-04-24T08:20:08.751382+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.774398+0000","last_change":"2024-04-24T08:20:08.770890+0000","last_active":"2024-04-24T08:20:08.774398+0000","last_peered":"2024-04-24T08:20:08.774398+0000","last_clean":"2024-04-24T08:20:08.774398+0000","last_became_active":"2024-04-24T08:20:08.770587+0000","last_became_peered":"2024-04-24T08:20:08.770587+0000","last_unstale":"2024-04-24T08:20:08.774398+0000","last_undegraded":"2024-04-24T08:20:08.774398+0000","last_fullsized":"2024-04-24T08:20:08.774398+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:34:35.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.741263+0000","last_change":"2024-04-24T08:20:08.741263+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.741263+0000","last_undegraded":"2024-04-24T08:20:08.741263+0000","last_fullsized":"2024-04-24T08:20:08.741263+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.761163+0000","last_change":"2024-04-24T08:20:08.757528+0000","last_active":"2024-04-24T08:20:08.761163+0000","last_peered":"2024-04-24T08:20:08.761163+0000","last_clean":"2024-04-24T08:20:08.761163+0000","last_became_active":"2024-04-24T08:20:08.757135+0000","last_became_peered":"2024-04-24T08:20:08.757135+0000","last_unstale":"2024-04-24T08:20:08.761163+0000","last_undegraded":"2024-04-24T08:20:08.761163+0000","last_fullsized":"2024-04-24T08:20:08.761163+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:20.208658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.742895+0000","last_change":"2024-04-24T08:20:08.742895+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.742895+0000","last_undegraded":"2024-04-24T08:20:08.742895+0000","last_fullsized":"2024-04-24T08:20:08.742895+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.767903+0000","last_change":"2024-04-24T08:20:08.766731+0000","last_active":"2024-04-24T08:20:08.767903+0000","last_peered":"2024-04-24T08:20:08.767903+0000","last_clean":"2024-04-24T08:20:08.767903+0000","last_became_active":"2024-04-24T08:20:08.766406+0000","last_became_peered":"2024-04-24T08:20:08.766406+0000","last_unstale":"2024-04-24T08:20:08.767903+0000","last_undegraded":"2024-04-24T08:20:08.767903+0000","last_fullsized":"2024-04-24T08:20:08.767903+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:56.617029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754413+0000","last_change":"2024-04-24T08:20:08.754413+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754413+0000","last_undegraded":"2024-04-24T08:20:08.754413+0000","last_fullsized":"2024-04-24T08:20:08.754413+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.771612+0000","last_change":"2024-04-24T08:20:08.768906+0000","last_active":"2024-04-24T08:20:08.771612+0000","last_peered":"2024-04-24T08:20:08.771612+0000","last_clean":"2024-04-24T08:20:08.771612+0000","last_became_active":"2024-04-24T08:20:08.768593+0000","last_became_peered":"2024-04-24T08:20:08.768593+0000","last_unstale":"2024-04-24T08:20:08.771612+0000","last_undegraded":"2024-04-24T08:20:08.771612+0000","last_fullsized":"2024-04-24T08:20:08.771612+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:45:04.610966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.749425+0000","last_change":"2024-04-24T08:20:08.749425+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.749425+0000","last_undegraded":"2024-04-24T08:20:08.749425+0000","last_fullsized":"2024-04-24T08:20:08.749425+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772240+0000","last_change":"2024-04-24T08:20:08.767036+0000","last_active":"2024-04-24T08:20:08.772240+0000","last_peered":"2024-04-24T08:20:08.772240+0000","last_clean":"2024-04-24T08:20:08.772240+0000","last_became_active":"2024-04-24T08:20:08.765832+0000","last_became_peered":"2024-04-24T08:20:08.765832+0000","last_unstale":"2024-04-24T08:20:08.772240+0000","last_undegraded":"2024-04-24T08:20:08.772240+0000","last_fullsized":"2024-04-24T08:20:08.772240+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:34.854054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.750103+0000","last_change":"2024-04-24T08:20:08.750103+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.750103+0000","last_undegraded":"2024-04-24T08:20:08.750103+0000","last_fullsized":"2024-04-24T08:20:08.750103+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.768810+0000","last_change":"2024-04-24T08:20:08.767872+0000","last_active":"2024-04-24T08:20:08.768810+0000","last_peered":"2024-04-24T08:20:08.768810+0000","last_clean":"2024-04-24T08:20:08.768810+0000","last_became_active":"2024-04-24T08:20:08.767538+0000","last_became_peered":"2024-04-24T08:20:08.767538+0000","last_unstale":"2024-04-24T08:20:08.768810+0000","last_undegraded":"2024-04-24T08:20:08.768810+0000","last_fullsized":"2024-04-24T08:20:08.768810+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:01.949509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":331,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.761279+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:20:08.761279+0000","last_peered":"2024-04-24T08:20:08.761279+0000","last_clean":"2024-04-24T08:20:08.761279+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:20:08.761279+0000","last_undegraded":"2024-04-24T08:20:08.761279+0000","last_fullsized":"2024-04-24T08:20:08.761279+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1925120,"data_stored":1166330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":575962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":569687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":569687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":569687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":569687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":569687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574995,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1912832,"data_stored":1160055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:20:14.211 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:20:14.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.441+0000 7fb371a00640 1 Processor -- start 2024-04-24T08:20:14.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.441+0000 7fb371a00640 1 -- start start 2024-04-24T08:20:14.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb371a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb36c0709f0 0x7fb36c06ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb371a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb36c15ed60 0x7fb36c06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb371a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb36c15f340 con 0x7fb36c15bd50 2024-04-24T08:20:14.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb371a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb36c070dd0 con 0x7fb36c0709f0 2024-04-24T08:20:14.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb371a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb36c15b270 con 0x7fb36c15ed60 2024-04-24T08:20:14.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb36b400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fb36c15bd50 0x7fb36c15e650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43238/0 (socket says 172.21.15.69:43238) 2024-04-24T08:20:14.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb36b400640 1 -- 172.21.15.69:0/3580072842 learned_addr learned my addr 172.21.15.69:0/3580072842 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:14.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb36be00640 1 --2- 172.21.15.69:0/3580072842 >> v2:172.21.15.69:3300/0 conn(0x7fb36c0709f0 0x7fb36c06ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb371000640 1 --2- 172.21.15.69:0/3580072842 >> v2:172.21.15.121:3300/0 conn(0x7fb36c15ed60 0x7fb36c06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb36be00640 1 -- 172.21.15.69:0/3580072842 >> v2:172.21.15.121:3300/0 conn(0x7fb36c15ed60 msgr2=0x7fb36c06cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:14.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb36be00640 1 --2- 172.21.15.69:0/3580072842 >> v2:172.21.15.121:3300/0 conn(0x7fb36c15ed60 0x7fb36c06cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb36be00640 1 -- 172.21.15.69:0/3580072842 >> v1:172.21.15.69:6789/0 conn(0x7fb36c15bd50 legacy=0x7fb36c15e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.443+0000 7fb36aa00640 1 -- 172.21.15.69:0/3580072842 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335694018 0 0) 0x7fb36c15f340 con 0x7fb36c15bd50 2024-04-24T08:20:14.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.444+0000 7fb36be00640 1 -- 172.21.15.69:0/3580072842 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c15b550 con 0x7fb36c0709f0 2024-04-24T08:20:14.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.445+0000 7fb36be00640 1 --2- 172.21.15.69:0/3580072842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c0709f0 0x7fb36c06ca30 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fb36c15e5b0 tx=0x7fb35c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea544f12e1159a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:14.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.445+0000 7fb36aa00640 1 -- 172.21.15.69:0/3580072842 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c037070 con 0x7fb36c0709f0 2024-04-24T08:20:14.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.445+0000 7fb36aa00640 1 -- 172.21.15.69:0/3580072842 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb35c032440 con 0x7fb36c0709f0 2024-04-24T08:20:14.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.445+0000 7fb36aa00640 1 -- 172.21.15.69:0/3580072842 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c02ae60 con 0x7fb36c0709f0 2024-04-24T08:20:14.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.445+0000 7fb371a00640 1 -- 172.21.15.69:0/3580072842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c0709f0 msgr2=0x7fb36c06ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.445+0000 7fb371a00640 1 --2- 172.21.15.69:0/3580072842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c0709f0 0x7fb36c06ca30 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fb36c15e5b0 tx=0x7fb35c02a110 comp rx=0 tx=0).stop 2024-04-24T08:20:14.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.445+0000 7fb371a00640 1 -- 172.21.15.69:0/3580072842 shutdown_connections 2024-04-24T08:20:14.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.445+0000 7fb371a00640 1 --2- 172.21.15.69:0/3580072842 >> v2:172.21.15.121:3300/0 conn(0x7fb36c15ed60 0x7fb36c06cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.445+0000 7fb371a00640 1 --2- 172.21.15.69:0/3580072842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c0709f0 0x7fb36c06ca30 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.445+0000 7fb371a00640 1 -- 172.21.15.69:0/3580072842 >> 172.21.15.69:0/3580072842 conn(0x7fb36c072580 msgr2=0x7fb36c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:14.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.446+0000 7fb371a00640 1 -- 172.21.15.69:0/3580072842 shutdown_connections 2024-04-24T08:20:14.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.446+0000 7fb371a00640 1 -- 172.21.15.69:0/3580072842 wait complete. 2024-04-24T08:20:14.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.447+0000 7fb371a00640 1 Processor -- start 2024-04-24T08:20:14.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.447+0000 7fb371a00640 1 -- start start 2024-04-24T08:20:14.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.448+0000 7fb371a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb36c0709f0 0x7fb36c144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.448+0000 7fb371a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15bd50 0x7fb36c14aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.448+0000 7fb371a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c15ed60 0x7fb36c0cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.448+0000 7fb371a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c067af0 con 0x7fb36c15bd50 2024-04-24T08:20:14.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.448+0000 7fb371a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c067970 con 0x7fb36c15ed60 2024-04-24T08:20:14.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.448+0000 7fb371a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb36c067c70 con 0x7fb36c0709f0 2024-04-24T08:20:14.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.448+0000 7fb36b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15bd50 0x7fb36c14aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.448+0000 7fb36b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15bd50 0x7fb36c14aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54486/0 (socket says 172.21.15.69:54486) 2024-04-24T08:20:14.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.448+0000 7fb36b400640 1 -- 172.21.15.69:0/1562222107 learned_addr learned my addr 172.21.15.69:0/1562222107 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb36be00640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb36c0709f0 0x7fb36c144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb371000640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c15ed60 0x7fb36c0cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb36b400640 1 -- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb36c0709f0 msgr2=0x7fb36c144f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb36b400640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb36c0709f0 0x7fb36c144f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb36b400640 1 -- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c15ed60 msgr2=0x7fb36c0cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb36b400640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c15ed60 0x7fb36c0cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb36b400640 1 -- 172.21.15.69:0/1562222107 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c148ae0 con 0x7fb36c15bd50 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb36b400640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15bd50 0x7fb36c14aff0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fb35800c460 tx=0x7fb35800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb363e00640 1 -- 172.21.15.69:0/1562222107 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3580074e0 con 0x7fb36c15bd50 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb363e00640 1 -- 172.21.15.69:0/1562222107 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3580047e0 con 0x7fb36c15bd50 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb363e00640 1 -- 172.21.15.69:0/1562222107 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358004bd0 con 0x7fb36c15bd50 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.449+0000 7fb371a00640 1 -- 172.21.15.69:0/1562222107 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36c1454c0 con 0x7fb36c15bd50 2024-04-24T08:20:14.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.450+0000 7fb371a00640 1 -- 172.21.15.69:0/1562222107 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb36c149690 con 0x7fb36c15bd50 2024-04-24T08:20:14.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.450+0000 7fb371a00640 1 -- 172.21.15.69:0/1562222107 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb36c075d50 con 0x7fb36c15bd50 2024-04-24T08:20:14.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.452+0000 7fb363e00640 1 -- 172.21.15.69:0/1562222107 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb358011070 con 0x7fb36c15bd50 2024-04-24T08:20:14.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.452+0000 7fb363e00640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c8f0 0x7fb34c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.452+0000 7fb363e00640 1 -- 172.21.15.69:0/1562222107 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fb35808b770 con 0x7fb36c15bd50 2024-04-24T08:20:14.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.453+0000 7fb36be00640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c8f0 0x7fb34c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.454+0000 7fb36be00640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c8f0 0x7fb34c06edb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb35c034000 tx=0x7fb35c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:14.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.457+0000 7fb363e00640 1 -- 172.21.15.69:0/1562222107 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb358059b90 con 0x7fb36c15bd50 2024-04-24T08:20:14.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.649+0000 7fb371a00640 1 -- 172.21.15.69:0/1562222107 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb36c0ca900 con 0x7fb34c06c8f0 2024-04-24T08:20:14.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.667+0000 7fb363e00640 1 -- 172.21.15.69:0/1562222107 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+247087 (secure 0 0 0) 0x7fb36c0ca900 con 0x7fb34c06c8f0 2024-04-24T08:20:14.671 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:14.673 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 -- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c8f0 msgr2=0x7fb34c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c8f0 0x7fb34c06edb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb35c034000 tx=0x7fb35c02e000 comp rx=0 tx=0).stop 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 -- 172.21.15.69:0/1562222107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15bd50 msgr2=0x7fb36c14aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15bd50 0x7fb36c14aff0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fb35800c460 tx=0x7fb35800c920 comp rx=0 tx=0).stop 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 -- 172.21.15.69:0/1562222107 shutdown_connections 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb34c06c8f0 0x7fb34c06edb0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb36c15bd50 0x7fb36c14aff0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb36c15ed60 0x7fb36c0cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 --2- 172.21.15.69:0/1562222107 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb36c0709f0 0x7fb36c144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 -- 172.21.15.69:0/1562222107 >> 172.21.15.69:0/1562222107 conn(0x7fb36c072580 msgr2=0x7fb36c06f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:14.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.673+0000 7fb371a00640 1 -- 172.21.15.69:0/1562222107 shutdown_connections 2024-04-24T08:20:14.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.674+0000 7fb371a00640 1 -- 172.21.15.69:0/1562222107 wait complete. 2024-04-24T08:20:14.693 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":464,"stamp":"2024-04-24T08:20:12.930388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260932,"kb_used_data":11488,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713788,"statfs":{"total":773094113280,"available":772826918912,"internally_reserved":0,"allocated":11763712,"data_stored":5750782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.200298"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.761304+0000","last_change":"2024-04-24T08:20:08.761304+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.761304+0000","last_undegraded":"2024-04-24T08:20:08.761304+0000","last_fullsized":"2024-04-24T08:20:08.761304+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.740349+0000","last_change":"2024-04-24T08:20:08.740349+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.740349+0000","last_undegraded":"2024-04-24T08:20:08.740349+0000","last_fullsized":"2024-04-24T08:20:08.740349+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.755775+0000","last_change":"2024-04-24T08:20:08.755775+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.755775+0000","last_undegraded":"2024-04-24T08:20:08.755775+0000","last_fullsized":"2024-04-24T08:20:08.755775+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.756093+0000","last_change":"2024-04-24T08:20:08.756093+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.756093+0000","last_undegraded":"2024-04-24T08:20:08.756093+0000","last_fullsized":"2024-04-24T08:20:08.756093+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.755449+0000","last_change":"2024-04-24T08:20:08.755449+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.755449+0000","last_undegraded":"2024-04-24T08:20:08.755449+0000","last_fullsized":"2024-04-24T08:20:08.755449+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.759258+0000","last_change":"2024-04-24T08:20:08.759258+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.759258+0000","last_undegraded":"2024-04-24T08:20:08.759258+0000","last_fullsized":"2024-04-24T08:20:08.759258+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.741095+0000","last_change":"2024-04-24T08:20:08.741095+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.741095+0000","last_undegraded":"2024-04-24T08:20:08.741095+0000","last_fullsized":"2024-04-24T08:20:08.741095+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.759172+0000","last_change":"2024-04-24T08:20:08.759172+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.759172+0000","last_undegraded":"2024-04-24T08:20:08.759172+0000","last_fullsized":"2024-04-24T08:20:08.759172+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.756459+0000","last_change":"2024-04-24T08:20:08.756459+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.756459+0000","last_undegraded":"2024-04-24T08:20:08.756459+0000","last_fullsized":"2024-04-24T08:20:08.756459+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.757303+0000","last_change":"2024-04-24T08:20:08.757303+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.757303+0000","last_undegraded":"2024-04-24T08:20:08.757303+0000","last_fullsized":"2024-04-24T08:20:08.757303+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.742570+0000","last_change":"2024-04-24T08:20:08.742570+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.742570+0000","last_undegraded":"2024-04-24T08:20:08.742570+0000","last_fullsized":"2024-04-24T08:20:08.742570+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754956+0000","last_change":"2024-04-24T08:20:08.754956+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754956+0000","last_undegraded":"2024-04-24T08:20:08.754956+0000","last_fullsized":"2024-04-24T08:20:08.754956+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.752401+0000","last_change":"2024-04-24T08:20:08.752401+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.752401+0000","last_undegraded":"2024-04-24T08:20:08.752401+0000","last_fullsized":"2024-04-24T08:20:08.752401+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753720+0000","last_change":"2024-04-24T08:20:08.753720+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753720+0000","last_undegraded":"2024-04-24T08:20:08.753720+0000","last_fullsized":"2024-04-24T08:20:08.753720+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.757701+0000","last_change":"2024-04-24T08:20:08.757701+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.757701+0000","last_undegraded":"2024-04-24T08:20:08.757701+0000","last_fullsized":"2024-04-24T08:20:08.757701+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.755783+0000","last_change":"2024-04-24T08:20:08.755783+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.755783+0000","last_undegraded":"2024-04-24T08:20:08.755783+0000","last_fullsized":"2024-04-24T08:20:08.755783+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.741980+0000","last_change":"2024-04-24T08:20:08.741980+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.741980+0000","last_undegraded":"2024-04-24T08:20:08.741980+0000","last_fullsized":"2024-04-24T08:20:08.741980+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.751997+0000","last_change":"2024-04-24T08:20:08.751997+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.751997+0000","last_undegraded":"2024-04-24T08:20:08.751997+0000","last_fullsized":"2024-04-24T08:20:08.751997+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754860+0000","last_change":"2024-04-24T08:20:08.754860+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754860+0000","last_undegraded":"2024-04-24T08:20:08.754860+0000","last_fullsized":"2024-04-24T08:20:08.754860+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.740346+0000","last_change":"2024-04-24T08:20:08.740346+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.740346+0000","last_undegraded":"2024-04-24T08:20:08.740346+0000","last_fullsized":"2024-04-24T08:20:08.740346+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.741581+0000","last_change":"2024-04-24T08:20:08.741581+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.741581+0000","last_undegraded":"2024-04-24T08:20:08.741581+0000","last_fullsized":"2024-04-24T08:20:08.741581+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753020+0000","last_change":"2024-04-24T08:20:08.753020+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753020+0000","last_undegraded":"2024-04-24T08:20:08.753020+0000","last_fullsized":"2024-04-24T08:20:08.753020+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.740388+0000","last_change":"2024-04-24T08:20:08.740388+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.740388+0000","last_undegraded":"2024-04-24T08:20:08.740388+0000","last_fullsized":"2024-04-24T08:20:08.740388+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.750565+0000","last_change":"2024-04-24T08:20:08.750565+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.750565+0000","last_undegraded":"2024-04-24T08:20:08.750565+0000","last_fullsized":"2024-04-24T08:20:08.750565+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.740719+0000","last_change":"2024-04-24T08:20:08.740719+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.740719+0000","last_undegraded":"2024-04-24T08:20:08.740719+0000","last_fullsized":"2024-04-24T08:20:08.740719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754870+0000","last_change":"2024-04-24T08:20:08.754870+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754870+0000","last_undegraded":"2024-04-24T08:20:08.754870+0000","last_fullsized":"2024-04-24T08:20:08.754870+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.758148+0000","last_change":"2024-04-24T08:20:08.758148+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.758148+0000","last_undegraded":"2024-04-24T08:20:08.758148+0000","last_fullsized":"2024-04-24T08:20:08.758148+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.755369+0000","last_change":"2024-04-24T08:20:08.755369+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.755369+0000","last_undegraded":"2024-04-24T08:20:08.755369+0000","last_fullsized":"2024-04-24T08:20:08.755369+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.745347+0000","last_change":"2024-04-24T08:20:08.745347+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.745347+0000","last_undegraded":"2024-04-24T08:20:08.745347+0000","last_fullsized":"2024-04-24T08:20:08.745347+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.743311+0000","last_change":"2024-04-24T08:20:08.743311+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.743311+0000","last_undegraded":"2024-04-24T08:20:08.743311+0000","last_fullsized":"2024-04-24T08:20:08.743311+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.755577+0000","last_change":"2024-04-24T08:20:08.755577+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.755577+0000","last_undegraded":"2024-04-24T08:20:08.755577+0000","last_fullsized":"2024-04-24T08:20:08.755577+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.756839+0000","last_change":"2024-04-24T08:20:08.756839+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.756839+0000","last_undegraded":"2024-04-24T08:20:08.756839+0000","last_fullsized":"2024-04-24T08:20:08.756839+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.761419+0000","last_change":"2024-04-24T08:20:08.761419+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.761419+0000","last_undegraded":"2024-04-24T08:20:08.761419+0000","last_fullsized":"2024-04-24T08:20:08.761419+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.768078+0000","last_change":"2024-04-24T08:20:08.765030+0000","last_active":"2024-04-24T08:20:08.768078+0000","last_peered":"2024-04-24T08:20:08.768078+0000","last_clean":"2024-04-24T08:20:08.768078+0000","last_became_active":"2024-04-24T08:20:08.764730+0000","last_became_peered":"2024-04-24T08:20:08.764730+0000","last_unstale":"2024-04-24T08:20:08.768078+0000","last_undegraded":"2024-04-24T08:20:08.768078+0000","last_fullsized":"2024-04-24T08:20:08.768078+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:12:45.289874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754916+0000","last_change":"2024-04-24T08:20:08.754916+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754916+0000","last_undegraded":"2024-04-24T08:20:08.754916+0000","last_fullsized":"2024-04-24T08:20:08.754916+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.767267+0000","last_change":"2024-04-24T08:20:08.764950+0000","last_active":"2024-04-24T08:20:08.767267+0000","last_peered":"2024-04-24T08:20:08.767267+0000","last_clean":"2024-04-24T08:20:08.767267+0000","last_became_active":"2024-04-24T08:20:08.764618+0000","last_became_peered":"2024-04-24T08:20:08.764618+0000","last_unstale":"2024-04-24T08:20:08.767267+0000","last_undegraded":"2024-04-24T08:20:08.767267+0000","last_fullsized":"2024-04-24T08:20:08.767267+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:33.902279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.756452+0000","last_change":"2024-04-24T08:20:08.756452+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.756452+0000","last_undegraded":"2024-04-24T08:20:08.756452+0000","last_fullsized":"2024-04-24T08:20:08.756452+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.774371+0000","last_change":"2024-04-24T08:20:08.772932+0000","last_active":"2024-04-24T08:20:08.774371+0000","last_peered":"2024-04-24T08:20:08.774371+0000","last_clean":"2024-04-24T08:20:08.774371+0000","last_became_active":"2024-04-24T08:20:08.772234+0000","last_became_peered":"2024-04-24T08:20:08.772234+0000","last_unstale":"2024-04-24T08:20:08.774371+0000","last_undegraded":"2024-04-24T08:20:08.774371+0000","last_fullsized":"2024-04-24T08:20:08.774371+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:32:54.564513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753481+0000","last_change":"2024-04-24T08:20:08.753481+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753481+0000","last_undegraded":"2024-04-24T08:20:08.753481+0000","last_fullsized":"2024-04-24T08:20:08.753481+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772429+0000","last_change":"2024-04-24T08:20:08.770132+0000","last_active":"2024-04-24T08:20:08.772429+0000","last_peered":"2024-04-24T08:20:08.772429+0000","last_clean":"2024-04-24T08:20:08.772429+0000","last_became_active":"2024-04-24T08:20:08.769756+0000","last_became_peered":"2024-04-24T08:20:08.769756+0000","last_unstale":"2024-04-24T08:20:08.772429+0000","last_undegraded":"2024-04-24T08:20:08.772429+0000","last_fullsized":"2024-04-24T08:20:08.772429+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:15:56.351743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.749058+0000","last_change":"2024-04-24T08:20:08.749058+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.749058+0000","last_undegraded":"2024-04-24T08:20:08.749058+0000","last_fullsized":"2024-04-24T08:20:08.749058+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.768047+0000","last_change":"2024-04-24T08:20:08.763740+0000","last_active":"2024-04-24T08:20:08.768047+0000","last_peered":"2024-04-24T08:20:08.768047+0000","last_clean":"2024-04-24T08:20:08.768047+0000","last_became_active":"2024-04-24T08:20:08.763472+0000","last_became_peered":"2024-04-24T08:20:08.763472+0000","last_unstale":"2024-04-24T08:20:08.768047+0000","last_undegraded":"2024-04-24T08:20:08.768047+0000","last_fullsized":"2024-04-24T08:20:08.768047+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:52:34.264489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.744129+0000","last_change":"2024-04-24T08:20:08.744129+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.744129+0000","last_undegraded":"2024-04-24T08:20:08.744129+0000","last_fullsized":"2024-04-24T08:20:08.744129+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.764276+0000","last_change":"2024-04-24T08:20:08.762014+0000","last_active":"2024-04-24T08:20:08.764276+0000","last_peered":"2024-04-24T08:20:08.764276+0000","last_clean":"2024-04-24T08:20:08.764276+0000","last_became_active":"2024-04-24T08:20:08.761675+0000","last_became_peered":"2024-04-24T08:20:08.761675+0000","last_unstale":"2024-04-24T08:20:08.764276+0000","last_undegraded":"2024-04-24T08:20:08.764276+0000","last_fullsized":"2024-04-24T08:20:08.764276+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:02:41.460096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.745815+0000","last_change":"2024-04-24T08:20:08.745815+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.745815+0000","last_undegraded":"2024-04-24T08:20:08.745815+0000","last_fullsized":"2024-04-24T08:20:08.745815+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.759411+0000","last_change":"2024-04-24T08:20:08.755417+0000","last_active":"2024-04-24T08:20:08.759411+0000","last_peered":"2024-04-24T08:20:08.759411+0000","last_clean":"2024-04-24T08:20:08.759411+0000","last_became_active":"2024-04-24T08:20:08.755131+0000","last_became_peered":"2024-04-24T08:20:08.755131+0000","last_unstale":"2024-04-24T08:20:08.759411+0000","last_undegraded":"2024-04-24T08:20:08.759411+0000","last_fullsized":"2024-04-24T08:20:08.759411+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:34:35.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.742263+0000","last_change":"2024-04-24T08:20:08.742263+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.742263+0000","last_undegraded":"2024-04-24T08:20:08.742263+0000","last_fullsized":"2024-04-24T08:20:08.742263+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772216+0000","last_change":"2024-04-24T08:20:08.764423+0000","last_active":"2024-04-24T08:20:08.772216+0000","last_peered":"2024-04-24T08:20:08.772216+0000","last_clean":"2024-04-24T08:20:08.772216+0000","last_became_active":"2024-04-24T08:20:08.764017+0000","last_became_peered":"2024-04-24T08:20:08.764017+0000","last_unstale":"2024-04-24T08:20:08.772216+0000","last_undegraded":"2024-04-24T08:20:08.772216+0000","last_fullsized":"2024-04-24T08:20:08.772216+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:26:30.942475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.747438+0000","last_change":"2024-04-24T08:20:08.747438+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.747438+0000","last_undegraded":"2024-04-24T08:20:08.747438+0000","last_fullsized":"2024-04-24T08:20:08.747438+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.771544+0000","last_change":"2024-04-24T08:20:08.768692+0000","last_active":"2024-04-24T08:20:08.771544+0000","last_peered":"2024-04-24T08:20:08.771544+0000","last_clean":"2024-04-24T08:20:08.771544+0000","last_became_active":"2024-04-24T08:20:08.768352+0000","last_became_peered":"2024-04-24T08:20:08.768352+0000","last_unstale":"2024-04-24T08:20:08.771544+0000","last_undegraded":"2024-04-24T08:20:08.771544+0000","last_fullsized":"2024-04-24T08:20:08.771544+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:58:48.970516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753718+0000","last_change":"2024-04-24T08:20:08.753718+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753718+0000","last_undegraded":"2024-04-24T08:20:08.753718+0000","last_fullsized":"2024-04-24T08:20:08.753718+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.770760+0000","last_change":"2024-04-24T08:20:08.767491+0000","last_active":"2024-04-24T08:20:08.770760+0000","last_peered":"2024-04-24T08:20:08.770760+0000","last_clean":"2024-04-24T08:20:08.770760+0000","last_became_active":"2024-04-24T08:20:08.767200+0000","last_became_peered":"2024-04-24T08:20:08.767200+0000","last_unstale":"2024-04-24T08:20:08.770760+0000","last_undegraded":"2024-04-24T08:20:08.770760+0000","last_fullsized":"2024-04-24T08:20:08.770760+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:41.544350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754342+0000","last_change":"2024-04-24T08:20:08.754342+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754342+0000","last_undegraded":"2024-04-24T08:20:08.754342+0000","last_fullsized":"2024-04-24T08:20:08.754342+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.775190+0000","last_change":"2024-04-24T08:20:08.771750+0000","last_active":"2024-04-24T08:20:08.775190+0000","last_peered":"2024-04-24T08:20:08.775190+0000","last_clean":"2024-04-24T08:20:08.775190+0000","last_became_active":"2024-04-24T08:20:08.771450+0000","last_became_peered":"2024-04-24T08:20:08.771450+0000","last_unstale":"2024-04-24T08:20:08.775190+0000","last_undegraded":"2024-04-24T08:20:08.775190+0000","last_fullsized":"2024-04-24T08:20:08.775190+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:46.509284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.744328+0000","last_change":"2024-04-24T08:20:08.744328+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.744328+0000","last_undegraded":"2024-04-24T08:20:08.744328+0000","last_fullsized":"2024-04-24T08:20:08.744328+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.771114+0000","last_change":"2024-04-24T08:20:08.763054+0000","last_active":"2024-04-24T08:20:08.771114+0000","last_peered":"2024-04-24T08:20:08.771114+0000","last_clean":"2024-04-24T08:20:08.771114+0000","last_became_active":"2024-04-24T08:20:08.762785+0000","last_became_peered":"2024-04-24T08:20:08.762785+0000","last_unstale":"2024-04-24T08:20:08.771114+0000","last_undegraded":"2024-04-24T08:20:08.771114+0000","last_fullsized":"2024-04-24T08:20:08.771114+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:52:34.850318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.752357+0000","last_change":"2024-04-24T08:20:08.752357+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.752357+0000","last_undegraded":"2024-04-24T08:20:08.752357+0000","last_fullsized":"2024-04-24T08:20:08.752357+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.774116+0000","last_change":"2024-04-24T08:20:08.771246+0000","last_active":"2024-04-24T08:20:08.774116+0000","last_peered":"2024-04-24T08:20:08.774116+0000","last_clean":"2024-04-24T08:20:08.774116+0000","last_became_active":"2024-04-24T08:20:08.770915+0000","last_became_peered":"2024-04-24T08:20:08.770915+0000","last_unstale":"2024-04-24T08:20:08.774116+0000","last_undegraded":"2024-04-24T08:20:08.774116+0000","last_fullsized":"2024-04-24T08:20:08.774116+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:01.949509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753899+0000","last_change":"2024-04-24T08:20:08.753899+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753899+0000","last_undegraded":"2024-04-24T08:20:08.753899+0000","last_fullsized":"2024-04-24T08:20:08.753899+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.771979+0000","last_change":"2024-04-24T08:20:08.770641+0000","last_active":"2024-04-24T08:20:08.771979+0000","last_peered":"2024-04-24T08:20:08.771979+0000","last_clean":"2024-04-24T08:20:08.771979+0000","last_became_active":"2024-04-24T08:20:08.770281+0000","last_became_peered":"2024-04-24T08:20:08.770281+0000","last_unstale":"2024-04-24T08:20:08.771979+0000","last_undegraded":"2024-04-24T08:20:08.771979+0000","last_fullsized":"2024-04-24T08:20:08.771979+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:52:34.850318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.751348+0000","last_change":"2024-04-24T08:20:08.751348+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.751348+0000","last_undegraded":"2024-04-24T08:20:08.751348+0000","last_fullsized":"2024-04-24T08:20:08.751348+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.765894+0000","last_change":"2024-04-24T08:20:08.760398+0000","last_active":"2024-04-24T08:20:08.765894+0000","last_peered":"2024-04-24T08:20:08.765894+0000","last_clean":"2024-04-24T08:20:08.765894+0000","last_became_active":"2024-04-24T08:20:08.760075+0000","last_became_peered":"2024-04-24T08:20:08.760075+0000","last_unstale":"2024-04-24T08:20:08.765894+0000","last_undegraded":"2024-04-24T08:20:08.765894+0000","last_fullsized":"2024-04-24T08:20:08.765894+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:46:50.458288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.752990+0000","last_change":"2024-04-24T08:20:08.752990+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.752990+0000","last_undegraded":"2024-04-24T08:20:08.752990+0000","last_fullsized":"2024-04-24T08:20:08.752990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.775165+0000","last_change":"2024-04-24T08:20:08.769979+0000","last_active":"2024-04-24T08:20:08.775165+0000","last_peered":"2024-04-24T08:20:08.775165+0000","last_clean":"2024-04-24T08:20:08.775165+0000","last_became_active":"2024-04-24T08:20:08.769667+0000","last_became_peered":"2024-04-24T08:20:08.769667+0000","last_unstale":"2024-04-24T08:20:08.775165+0000","last_undegraded":"2024-04-24T08:20:08.775165+0000","last_fullsized":"2024-04-24T08:20:08.775165+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:20.634892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.761130+0000","last_change":"2024-04-24T08:20:08.761130+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.761130+0000","last_undegraded":"2024-04-24T08:20:08.761130+0000","last_fullsized":"2024-04-24T08:20:08.761130+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.765621+0000","last_change":"2024-04-24T08:20:08.762510+0000","last_active":"2024-04-24T08:20:08.765621+0000","last_peered":"2024-04-24T08:20:08.765621+0000","last_clean":"2024-04-24T08:20:08.765621+0000","last_became_active":"2024-04-24T08:20:08.762168+0000","last_became_peered":"2024-04-24T08:20:08.762168+0000","last_unstale":"2024-04-24T08:20:08.765621+0000","last_undegraded":"2024-04-24T08:20:08.765621+0000","last_fullsized":"2024-04-24T08:20:08.765621+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:28:16.020468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754069+0000","last_change":"2024-04-24T08:20:08.754069+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754069+0000","last_undegraded":"2024-04-24T08:20:08.754069+0000","last_fullsized":"2024-04-24T08:20:08.754069+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.763758+0000","last_change":"2024-04-24T08:20:08.761477+0000","last_active":"2024-04-24T08:20:08.763758+0000","last_peered":"2024-04-24T08:20:08.763758+0000","last_clean":"2024-04-24T08:20:08.763758+0000","last_became_active":"2024-04-24T08:20:08.759812+0000","last_became_peered":"2024-04-24T08:20:08.759812+0000","last_unstale":"2024-04-24T08:20:08.763758+0000","last_undegraded":"2024-04-24T08:20:08.763758+0000","last_fullsized":"2024-04-24T08:20:08.763758+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:10:49.534393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.752461+0000","last_change":"2024-04-24T08:20:08.752461+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.752461+0000","last_undegraded":"2024-04-24T08:20:08.752461+0000","last_fullsized":"2024-04-24T08:20:08.752461+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.768460+0000","last_change":"2024-04-24T08:20:08.767284+0000","last_active":"2024-04-24T08:20:08.768460+0000","last_peered":"2024-04-24T08:20:08.768460+0000","last_clean":"2024-04-24T08:20:08.768460+0000","last_became_active":"2024-04-24T08:20:08.766880+0000","last_became_peered":"2024-04-24T08:20:08.766880+0000","last_unstale":"2024-04-24T08:20:08.768460+0000","last_undegraded":"2024-04-24T08:20:08.768460+0000","last_fullsized":"2024-04-24T08:20:08.768460+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:27:25.539395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.753173+0000","last_change":"2024-04-24T08:20:08.753173+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.753173+0000","last_undegraded":"2024-04-24T08:20:08.753173+0000","last_fullsized":"2024-04-24T08:20:08.753173+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.775773+0000","last_change":"2024-04-24T08:20:08.773991+0000","last_active":"2024-04-24T08:20:08.775773+0000","last_peered":"2024-04-24T08:20:08.775773+0000","last_clean":"2024-04-24T08:20:08.775773+0000","last_became_active":"2024-04-24T08:20:08.773703+0000","last_became_peered":"2024-04-24T08:20:08.773703+0000","last_unstale":"2024-04-24T08:20:08.775773+0000","last_undegraded":"2024-04-24T08:20:08.775773+0000","last_fullsized":"2024-04-24T08:20:08.775773+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:08.415565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.746401+0000","last_change":"2024-04-24T08:20:08.746401+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.746401+0000","last_undegraded":"2024-04-24T08:20:08.746401+0000","last_fullsized":"2024-04-24T08:20:08.746401+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772260+0000","last_change":"2024-04-24T08:20:08.769543+0000","last_active":"2024-04-24T08:20:08.772260+0000","last_peered":"2024-04-24T08:20:08.772260+0000","last_clean":"2024-04-24T08:20:08.772260+0000","last_became_active":"2024-04-24T08:20:08.767940+0000","last_became_peered":"2024-04-24T08:20:08.767940+0000","last_unstale":"2024-04-24T08:20:08.772260+0000","last_undegraded":"2024-04-24T08:20:08.772260+0000","last_fullsized":"2024-04-24T08:20:08.772260+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:54:03.023190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754926+0000","last_change":"2024-04-24T08:20:08.754926+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754926+0000","last_undegraded":"2024-04-24T08:20:08.754926+0000","last_fullsized":"2024-04-24T08:20:08.754926+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772376+0000","last_change":"2024-04-24T08:20:08.769318+0000","last_active":"2024-04-24T08:20:08.772376+0000","last_peered":"2024-04-24T08:20:08.772376+0000","last_clean":"2024-04-24T08:20:08.772376+0000","last_became_active":"2024-04-24T08:20:08.769050+0000","last_became_peered":"2024-04-24T08:20:08.769050+0000","last_unstale":"2024-04-24T08:20:08.772376+0000","last_undegraded":"2024-04-24T08:20:08.772376+0000","last_fullsized":"2024-04-24T08:20:08.772376+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:27:25.539395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.749824+0000","last_change":"2024-04-24T08:20:08.749824+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.749824+0000","last_undegraded":"2024-04-24T08:20:08.749824+0000","last_fullsized":"2024-04-24T08:20:08.749824+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772258+0000","last_change":"2024-04-24T08:20:08.767252+0000","last_active":"2024-04-24T08:20:08.772258+0000","last_peered":"2024-04-24T08:20:08.772258+0000","last_clean":"2024-04-24T08:20:08.772258+0000","last_became_active":"2024-04-24T08:20:08.766235+0000","last_became_peered":"2024-04-24T08:20:08.766235+0000","last_unstale":"2024-04-24T08:20:08.772258+0000","last_undegraded":"2024-04-24T08:20:08.772258+0000","last_fullsized":"2024-04-24T08:20:08.772258+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:27:26.888390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.750732+0000","last_change":"2024-04-24T08:20:08.750732+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.750732+0000","last_undegraded":"2024-04-24T08:20:08.750732+0000","last_fullsized":"2024-04-24T08:20:08.750732+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.775584+0000","last_change":"2024-04-24T08:20:08.774854+0000","last_active":"2024-04-24T08:20:08.775584+0000","last_peered":"2024-04-24T08:20:08.775584+0000","last_clean":"2024-04-24T08:20:08.775584+0000","last_became_active":"2024-04-24T08:20:08.773896+0000","last_became_peered":"2024-04-24T08:20:08.773896+0000","last_unstale":"2024-04-24T08:20:08.775584+0000","last_undegraded":"2024-04-24T08:20:08.775584+0000","last_fullsized":"2024-04-24T08:20:08.775584+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:56.617029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.747005+0000","last_change":"2024-04-24T08:20:08.747005+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.747005+0000","last_undegraded":"2024-04-24T08:20:08.747005+0000","last_fullsized":"2024-04-24T08:20:08.747005+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.767713+0000","last_change":"2024-04-24T08:20:08.763470+0000","last_active":"2024-04-24T08:20:08.767713+0000","last_peered":"2024-04-24T08:20:08.767713+0000","last_clean":"2024-04-24T08:20:08.767713+0000","last_became_active":"2024-04-24T08:20:08.762839+0000","last_became_peered":"2024-04-24T08:20:08.762839+0000","last_unstale":"2024-04-24T08:20:08.767713+0000","last_undegraded":"2024-04-24T08:20:08.767713+0000","last_fullsized":"2024-04-24T08:20:08.767713+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:23:50.528789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.747108+0000","last_change":"2024-04-24T08:20:08.747108+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.747108+0000","last_undegraded":"2024-04-24T08:20:08.747108+0000","last_fullsized":"2024-04-24T08:20:08.747108+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.764788+0000","last_change":"2024-04-24T08:20:08.761630+0000","last_active":"2024-04-24T08:20:08.764788+0000","last_peered":"2024-04-24T08:20:08.764788+0000","last_clean":"2024-04-24T08:20:08.764788+0000","last_became_active":"2024-04-24T08:20:08.761355+0000","last_became_peered":"2024-04-24T08:20:08.761355+0000","last_unstale":"2024-04-24T08:20:08.764788+0000","last_undegraded":"2024-04-24T08:20:08.764788+0000","last_fullsized":"2024-04-24T08:20:08.764788+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:23:50.528789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.751382+0000","last_change":"2024-04-24T08:20:08.751382+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.751382+0000","last_undegraded":"2024-04-24T08:20:08.751382+0000","last_fullsized":"2024-04-24T08:20:08.751382+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.774398+0000","last_change":"2024-04-24T08:20:08.770890+0000","last_active":"2024-04-24T08:20:08.774398+0000","last_peered":"2024-04-24T08:20:08.774398+0000","last_clean":"2024-04-24T08:20:08.774398+0000","last_became_active":"2024-04-24T08:20:08.770587+0000","last_became_peered":"2024-04-24T08:20:08.770587+0000","last_unstale":"2024-04-24T08:20:08.774398+0000","last_undegraded":"2024-04-24T08:20:08.774398+0000","last_fullsized":"2024-04-24T08:20:08.774398+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:34:35.090753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.741263+0000","last_change":"2024-04-24T08:20:08.741263+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.741263+0000","last_undegraded":"2024-04-24T08:20:08.741263+0000","last_fullsized":"2024-04-24T08:20:08.741263+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.761163+0000","last_change":"2024-04-24T08:20:08.757528+0000","last_active":"2024-04-24T08:20:08.761163+0000","last_peered":"2024-04-24T08:20:08.761163+0000","last_clean":"2024-04-24T08:20:08.761163+0000","last_became_active":"2024-04-24T08:20:08.757135+0000","last_became_peered":"2024-04-24T08:20:08.757135+0000","last_unstale":"2024-04-24T08:20:08.761163+0000","last_undegraded":"2024-04-24T08:20:08.761163+0000","last_fullsized":"2024-04-24T08:20:08.761163+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:20.208658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.742895+0000","last_change":"2024-04-24T08:20:08.742895+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.742895+0000","last_undegraded":"2024-04-24T08:20:08.742895+0000","last_fullsized":"2024-04-24T08:20:08.742895+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.767903+0000","last_change":"2024-04-24T08:20:08.766731+0000","last_active":"2024-04-24T08:20:08.767903+0000","last_peered":"2024-04-24T08:20:08.767903+0000","last_clean":"2024-04-24T08:20:08.767903+0000","last_became_active":"2024-04-24T08:20:08.766406+0000","last_became_peered":"2024-04-24T08:20:08.766406+0000","last_unstale":"2024-04-24T08:20:08.767903+0000","last_undegraded":"2024-04-24T08:20:08.767903+0000","last_fullsized":"2024-04-24T08:20:08.767903+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:29:56.617029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.754413+0000","last_change":"2024-04-24T08:20:08.754413+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.754413+0000","last_undegraded":"2024-04-24T08:20:08.754413+0000","last_fullsized":"2024-04-24T08:20:08.754413+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.771612+0000","last_change":"2024-04-24T08:20:08.768906+0000","last_active":"2024-04-24T08:20:08.771612+0000","last_peered":"2024-04-24T08:20:08.771612+0000","last_clean":"2024-04-24T08:20:08.771612+0000","last_became_active":"2024-04-24T08:20:08.768593+0000","last_became_peered":"2024-04-24T08:20:08.768593+0000","last_unstale":"2024-04-24T08:20:08.771612+0000","last_undegraded":"2024-04-24T08:20:08.771612+0000","last_fullsized":"2024-04-24T08:20:08.771612+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:45:04.610966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.749425+0000","last_change":"2024-04-24T08:20:08.749425+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.749425+0000","last_undegraded":"2024-04-24T08:20:08.749425+0000","last_fullsized":"2024-04-24T08:20:08.749425+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.772240+0000","last_change":"2024-04-24T08:20:08.767036+0000","last_active":"2024-04-24T08:20:08.772240+0000","last_peered":"2024-04-24T08:20:08.772240+0000","last_clean":"2024-04-24T08:20:08.772240+0000","last_became_active":"2024-04-24T08:20:08.765832+0000","last_became_peered":"2024-04-24T08:20:08.765832+0000","last_unstale":"2024-04-24T08:20:08.772240+0000","last_undegraded":"2024-04-24T08:20:08.772240+0000","last_fullsized":"2024-04-24T08:20:08.772240+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:34.854054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":1,"reported_epoch":107,"state":"creating+peering","last_fresh":"2024-04-24T08:20:08.750103+0000","last_change":"2024-04-24T08:20:08.750103+0000","last_active":"2024-04-24T08:20:08.725174+0000","last_peered":"2024-04-24T08:20:08.725174+0000","last_clean":"2024-04-24T08:20:08.725174+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T08:20:08.750103+0000","last_undegraded":"2024-04-24T08:20:08.750103+0000","last_fullsized":"2024-04-24T08:20:08.750103+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":9,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.768810+0000","last_change":"2024-04-24T08:20:08.767872+0000","last_active":"2024-04-24T08:20:08.768810+0000","last_peered":"2024-04-24T08:20:08.768810+0000","last_clean":"2024-04-24T08:20:08.768810+0000","last_became_active":"2024-04-24T08:20:08.767538+0000","last_became_peered":"2024-04-24T08:20:08.767538+0000","last_unstale":"2024-04-24T08:20:08.768810+0000","last_undegraded":"2024-04-24T08:20:08.768810+0000","last_fullsized":"2024-04-24T08:20:08.768810+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:42:01.949509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":331,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-24T08:20:08.761279+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:20:08.761279+0000","last_peered":"2024-04-24T08:20:08.761279+0000","last_clean":"2024-04-24T08:20:08.761279+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:20:08.761279+0000","last_undegraded":"2024-04-24T08:20:08.761279+0000","last_fullsized":"2024-04-24T08:20:08.761279+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574997,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33060,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338780,"statfs":{"total":96636764160,"available":96602910720,"internally_reserved":0,"allocated":1925120,"data_stored":1166330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574997,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32484,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339356,"statfs":{"total":96636764160,"available":96603500544,"internally_reserved":0,"allocated":1331200,"data_stored":575962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":569687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574994,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":569687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574995,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":569687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":569687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574996,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1318912,"data_stored":569687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574995,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1912832,"data_stored":1160055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:20:14.698 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:20:14.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.933+0000 7fe635600640 1 Processor -- start 2024-04-24T08:20:14.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.933+0000 7fe635600640 1 -- start start 2024-04-24T08:20:14.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.934+0000 7fe635600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe63015d1d0 0x7fe6300ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.934+0000 7fe635600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6300b30f0 0x7fe6300ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.934+0000 7fe635600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe63015d660 con 0x7fe630156c60 2024-04-24T08:20:14.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.934+0000 7fe635600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe63006dc30 con 0x7fe63015d1d0 2024-04-24T08:20:14.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.934+0000 7fe635600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe63006df10 con 0x7fe6300b30f0 2024-04-24T08:20:14.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.935+0000 7fe62fe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe630156c60 0x7fe63015cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43246/0 (socket says 172.21.15.69:43246) 2024-04-24T08:20:14.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.935+0000 7fe62fe00640 1 -- 172.21.15.69:0/2307556829 learned_addr learned my addr 172.21.15.69:0/2307556829 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:14.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.935+0000 7fe627e00640 1 --2- 172.21.15.69:0/2307556829 >> v2:172.21.15.121:3300/0 conn(0x7fe63015d1d0 0x7fe6300ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.935+0000 7fe634c00640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6300b30f0 0x7fe6300ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.935+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2468610689 0 0) 0x7fe63015d660 con 0x7fe630156c60 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.935+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe614003520 con 0x7fe630156c60 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 674862486 0 0) 0x7fe614003520 con 0x7fe630156c60 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6300b30f0 msgr2=0x7fe6300ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6300b30f0 0x7fe6300ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 >> v2:172.21.15.121:3300/0 conn(0x7fe63015d1d0 msgr2=0x7fe6300ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 --2- 172.21.15.69:0/2307556829 >> v2:172.21.15.121:3300/0 conn(0x7fe63015d1d0 0x7fe6300ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe63006e1f0 con 0x7fe630156c60 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe620001670 con 0x7fe630156c60 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 >> v1:172.21.15.69:6789/0 conn(0x7fe630156c60 legacy=0x7fe63015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 0x7fe614003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614004800 0x7fe614004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6300b30f0 0x7fe6140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe63006dc30 con 0x7fe6140039b0 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe63006df10 con 0x7fe614004800 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe61400b9c0 con 0x7fe6300b30f0 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.936+0000 7fe62fe00640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 0x7fe614003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.937+0000 7fe627e00640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614004800 0x7fe614004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.937+0000 7fe634c00640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6300b30f0 0x7fe6140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.937+0000 7fe62fe00640 1 -- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6300b30f0 msgr2=0x7fe6140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.937+0000 7fe62fe00640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe6300b30f0 0x7fe6140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.937+0000 7fe62fe00640 1 -- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614004800 msgr2=0x7fe614004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.937+0000 7fe62fe00640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe614004800 0x7fe614004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.937+0000 7fe62fe00640 1 -- 172.21.15.69:0/2307556829 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe614003520 con 0x7fe6140039b0 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.938+0000 7fe62fe00640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 0x7fe614003dd0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fe62000c210 tx=0x7fe614005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ef6396652c33cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.938+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620007d50 con 0x7fe6140039b0 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.938+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe620005af0 con 0x7fe6140039b0 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.938+0000 7fe62f400640 1 -- 172.21.15.69:0/2307556829 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620007070 con 0x7fe6140039b0 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.939+0000 7fe635600640 1 -- 172.21.15.69:0/2307556829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 msgr2=0x7fe614003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.939+0000 7fe635600640 1 --2- 172.21.15.69:0/2307556829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 0x7fe614003dd0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fe62000c210 tx=0x7fe614005830 comp rx=0 tx=0).stop 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.939+0000 7fe634c00640 1 -- 172.21.15.69:0/2307556829 reap_dead start 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.939+0000 7fe635600640 1 -- 172.21.15.69:0/2307556829 shutdown_connections 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.939+0000 7fe635600640 1 -- 172.21.15.69:0/2307556829 >> 172.21.15.69:0/2307556829 conn(0x7fe6300d0160 msgr2=0x7fe6300d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.939+0000 7fe635600640 1 -- 172.21.15.69:0/2307556829 shutdown_connections 2024-04-24T08:20:14.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.939+0000 7fe635600640 1 -- 172.21.15.69:0/2307556829 wait complete. 2024-04-24T08:20:14.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.941+0000 7fe635600640 1 Processor -- start 2024-04-24T08:20:14.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.941+0000 7fe635600640 1 -- start start 2024-04-24T08:20:14.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe635600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 0x7fe630073070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe635600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe63015ac70 0x7fe63015ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe635600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe630099b40 0x7fe63015f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe635600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe6300ad610 con 0x7fe6140039b0 2024-04-24T08:20:14.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe635600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe63015d940 con 0x7fe63015ac70 2024-04-24T08:20:14.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe635600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe6300ad790 con 0x7fe630099b40 2024-04-24T08:20:14.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe62fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 0x7fe630073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe62fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 0x7fe630073070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54504/0 (socket says 172.21.15.69:54504) 2024-04-24T08:20:14.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe62fe00640 1 -- 172.21.15.69:0/1463337640 learned_addr learned my addr 172.21.15.69:0/1463337640 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:14.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe627e00640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe63015ac70 0x7fe63015ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.942+0000 7fe634c00640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe630099b40 0x7fe63015f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe62fe00640 1 -- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe630099b40 msgr2=0x7fe63015f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe62fe00640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe630099b40 0x7fe63015f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe62fe00640 1 -- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe63015ac70 msgr2=0x7fe63015ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:14.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe62fe00640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe63015ac70 0x7fe63015ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:14.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe62fe00640 1 -- 172.21.15.69:0/1463337640 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe63015b590 con 0x7fe6140039b0 2024-04-24T08:20:14.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe62fe00640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 0x7fe630073070 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fe620034040 tx=0x7fe620001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:14.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe62cc00640 1 -- 172.21.15.69:0/1463337640 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620042070 con 0x7fe6140039b0 2024-04-24T08:20:14.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe62cc00640 1 -- 172.21.15.69:0/1463337640 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe620007480 con 0x7fe6140039b0 2024-04-24T08:20:14.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe62cc00640 1 -- 172.21.15.69:0/1463337640 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe620001c90 con 0x7fe6140039b0 2024-04-24T08:20:14.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe635600640 1 -- 172.21.15.69:0/1463337640 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe63015b870 con 0x7fe6140039b0 2024-04-24T08:20:14.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.943+0000 7fe635600640 1 -- 172.21.15.69:0/1463337640 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe63015be30 con 0x7fe6140039b0 2024-04-24T08:20:14.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.944+0000 7fe635600640 1 -- 172.21.15.69:0/1463337640 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6301e6810 con 0x7fe6140039b0 2024-04-24T08:20:14.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.945+0000 7fe62cc00640 1 -- 172.21.15.69:0/1463337640 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe620045070 con 0x7fe6140039b0 2024-04-24T08:20:14.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.947+0000 7fe62cc00640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe60c06c8f0 0x7fe60c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:14.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.947+0000 7fe62cc00640 1 -- 172.21.15.69:0/1463337640 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fe6200ac790 con 0x7fe6140039b0 2024-04-24T08:20:14.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.948+0000 7fe627e00640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe60c06c8f0 0x7fe60c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:14.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.949+0000 7fe627e00640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe60c06c8f0 0x7fe60c06edb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe63015a120 tx=0x7fe618008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:14.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:14.952+0000 7fe62cc00640 1 -- 172.21.15.69:0/1463337640 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe6301e6810 con 0x7fe6140039b0 2024-04-24T08:20:15.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.148+0000 7fe635600640 1 -- 172.21.15.69:0/1463337640 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe630075d50 con 0x7fe60c06c8f0 2024-04-24T08:20:15.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.167+0000 7fe62cc00640 1 -- 172.21.15.69:0/1463337640 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252483 (secure 0 0 0) 0x7fe630075d50 con 0x7fe60c06c8f0 2024-04-24T08:20:15.172 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:15.173 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:20:15.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 -- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe60c06c8f0 msgr2=0x7fe60c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe60c06c8f0 0x7fe60c06edb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe63015a120 tx=0x7fe618008040 comp rx=0 tx=0).stop 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 -- 172.21.15.69:0/1463337640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 msgr2=0x7fe630073070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 0x7fe630073070 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7fe620034040 tx=0x7fe620001a40 comp rx=0 tx=0).stop 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 -- 172.21.15.69:0/1463337640 shutdown_connections 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe60c06c8f0 0x7fe60c06edb0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe630099b40 0x7fe63015f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe63015ac70 0x7fe63015ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 --2- 172.21.15.69:0/1463337640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe6140039b0 0x7fe630073070 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 -- 172.21.15.69:0/1463337640 >> 172.21.15.69:0/1463337640 conn(0x7fe6300d0160 msgr2=0x7fe6300aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 -- 172.21.15.69:0/1463337640 shutdown_connections 2024-04-24T08:20:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.174+0000 7fe635600640 1 -- 172.21.15.69:0/1463337640 wait complete. 2024-04-24T08:20:15.193 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-04-24T08:20:14.931006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261288,"kb_used_data":11816,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713432,"statfs":{"total":773094113280,"available":772826554368,"internally_reserved":0,"allocated":12099584,"data_stored":5897852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.197126"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896427+0000","last_change":"2024-04-24T08:20:09.799542+0000","last_active":"2024-04-24T08:20:10.896427+0000","last_peered":"2024-04-24T08:20:10.896427+0000","last_clean":"2024-04-24T08:20:10.896427+0000","last_became_active":"2024-04-24T08:20:09.799243+0000","last_became_peered":"2024-04-24T08:20:09.799243+0000","last_unstale":"2024-04-24T08:20:10.896427+0000","last_undegraded":"2024-04-24T08:20:10.896427+0000","last_fullsized":"2024-04-24T08:20:10.896427+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:19:43.824329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791360+0000","last_change":"2024-04-24T08:20:09.794708+0000","last_active":"2024-04-24T08:20:10.791360+0000","last_peered":"2024-04-24T08:20:10.791360+0000","last_clean":"2024-04-24T08:20:10.791360+0000","last_became_active":"2024-04-24T08:20:09.793835+0000","last_became_peered":"2024-04-24T08:20:09.793835+0000","last_unstale":"2024-04-24T08:20:10.791360+0000","last_undegraded":"2024-04-24T08:20:10.791360+0000","last_fullsized":"2024-04-24T08:20:10.791360+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:13.818493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897294+0000","last_change":"2024-04-24T08:20:09.788114+0000","last_active":"2024-04-24T08:20:10.897294+0000","last_peered":"2024-04-24T08:20:10.897294+0000","last_clean":"2024-04-24T08:20:10.897294+0000","last_became_active":"2024-04-24T08:20:09.787809+0000","last_became_peered":"2024-04-24T08:20:09.787809+0000","last_unstale":"2024-04-24T08:20:10.897294+0000","last_undegraded":"2024-04-24T08:20:10.897294+0000","last_fullsized":"2024-04-24T08:20:10.897294+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:26:31.959658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895708+0000","last_change":"2024-04-24T08:20:09.794858+0000","last_active":"2024-04-24T08:20:10.895708+0000","last_peered":"2024-04-24T08:20:10.895708+0000","last_clean":"2024-04-24T08:20:10.895708+0000","last_became_active":"2024-04-24T08:20:09.794047+0000","last_became_peered":"2024-04-24T08:20:09.794047+0000","last_unstale":"2024-04-24T08:20:10.895708+0000","last_undegraded":"2024-04-24T08:20:10.895708+0000","last_fullsized":"2024-04-24T08:20:10.895708+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:52:00.465750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788117+0000","last_change":"2024-04-24T08:20:09.780027+0000","last_active":"2024-04-24T08:20:10.788117+0000","last_peered":"2024-04-24T08:20:10.788117+0000","last_clean":"2024-04-24T08:20:10.788117+0000","last_became_active":"2024-04-24T08:20:09.779744+0000","last_became_peered":"2024-04-24T08:20:09.779744+0000","last_unstale":"2024-04-24T08:20:10.788117+0000","last_undegraded":"2024-04-24T08:20:10.788117+0000","last_fullsized":"2024-04-24T08:20:10.788117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:09.432747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897475+0000","last_change":"2024-04-24T08:20:09.784098+0000","last_active":"2024-04-24T08:20:10.897475+0000","last_peered":"2024-04-24T08:20:10.897475+0000","last_clean":"2024-04-24T08:20:10.897475+0000","last_became_active":"2024-04-24T08:20:09.783835+0000","last_became_peered":"2024-04-24T08:20:09.783835+0000","last_unstale":"2024-04-24T08:20:10.897475+0000","last_undegraded":"2024-04-24T08:20:10.897475+0000","last_fullsized":"2024-04-24T08:20:10.897475+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:56.817425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789432+0000","last_change":"2024-04-24T08:20:09.785141+0000","last_active":"2024-04-24T08:20:10.789432+0000","last_peered":"2024-04-24T08:20:10.789432+0000","last_clean":"2024-04-24T08:20:10.789432+0000","last_became_active":"2024-04-24T08:20:09.784585+0000","last_became_peered":"2024-04-24T08:20:09.784585+0000","last_unstale":"2024-04-24T08:20:10.789432+0000","last_undegraded":"2024-04-24T08:20:10.789432+0000","last_fullsized":"2024-04-24T08:20:10.789432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:44:55.266192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896771+0000","last_change":"2024-04-24T08:20:09.792355+0000","last_active":"2024-04-24T08:20:10.896771+0000","last_peered":"2024-04-24T08:20:10.896771+0000","last_clean":"2024-04-24T08:20:10.896771+0000","last_became_active":"2024-04-24T08:20:09.791438+0000","last_became_peered":"2024-04-24T08:20:09.791438+0000","last_unstale":"2024-04-24T08:20:10.896771+0000","last_undegraded":"2024-04-24T08:20:10.896771+0000","last_fullsized":"2024-04-24T08:20:10.896771+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:59:41.162621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897198+0000","last_change":"2024-04-24T08:20:09.803001+0000","last_active":"2024-04-24T08:20:10.897198+0000","last_peered":"2024-04-24T08:20:10.897198+0000","last_clean":"2024-04-24T08:20:10.897198+0000","last_became_active":"2024-04-24T08:20:09.801984+0000","last_became_peered":"2024-04-24T08:20:09.801984+0000","last_unstale":"2024-04-24T08:20:10.897198+0000","last_undegraded":"2024-04-24T08:20:10.897198+0000","last_fullsized":"2024-04-24T08:20:10.897198+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:51:22.573683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896823+0000","last_change":"2024-04-24T08:20:09.801533+0000","last_active":"2024-04-24T08:20:10.896823+0000","last_peered":"2024-04-24T08:20:10.896823+0000","last_clean":"2024-04-24T08:20:10.896823+0000","last_became_active":"2024-04-24T08:20:09.801142+0000","last_became_peered":"2024-04-24T08:20:09.801142+0000","last_unstale":"2024-04-24T08:20:10.896823+0000","last_undegraded":"2024-04-24T08:20:10.896823+0000","last_fullsized":"2024-04-24T08:20:10.896823+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:05.572616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791049+0000","last_change":"2024-04-24T08:20:09.791103+0000","last_active":"2024-04-24T08:20:10.791049+0000","last_peered":"2024-04-24T08:20:10.791049+0000","last_clean":"2024-04-24T08:20:10.791049+0000","last_became_active":"2024-04-24T08:20:09.787354+0000","last_became_peered":"2024-04-24T08:20:09.787354+0000","last_unstale":"2024-04-24T08:20:10.791049+0000","last_undegraded":"2024-04-24T08:20:10.791049+0000","last_fullsized":"2024-04-24T08:20:10.791049+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:48.090822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896003+0000","last_change":"2024-04-24T08:20:09.802741+0000","last_active":"2024-04-24T08:20:10.896003+0000","last_peered":"2024-04-24T08:20:10.896003+0000","last_clean":"2024-04-24T08:20:10.896003+0000","last_became_active":"2024-04-24T08:20:09.802402+0000","last_became_peered":"2024-04-24T08:20:09.802402+0000","last_unstale":"2024-04-24T08:20:10.896003+0000","last_undegraded":"2024-04-24T08:20:10.896003+0000","last_fullsized":"2024-04-24T08:20:10.896003+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:52.645695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897895+0000","last_change":"2024-04-24T08:20:09.800630+0000","last_active":"2024-04-24T08:20:10.897895+0000","last_peered":"2024-04-24T08:20:10.897895+0000","last_clean":"2024-04-24T08:20:10.897895+0000","last_became_active":"2024-04-24T08:20:09.800140+0000","last_became_peered":"2024-04-24T08:20:09.800140+0000","last_unstale":"2024-04-24T08:20:10.897895+0000","last_undegraded":"2024-04-24T08:20:10.897895+0000","last_fullsized":"2024-04-24T08:20:10.897895+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:57.166226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896796+0000","last_change":"2024-04-24T08:20:09.789152+0000","last_active":"2024-04-24T08:20:10.896796+0000","last_peered":"2024-04-24T08:20:10.896796+0000","last_clean":"2024-04-24T08:20:10.896796+0000","last_became_active":"2024-04-24T08:20:09.788646+0000","last_became_peered":"2024-04-24T08:20:09.788646+0000","last_unstale":"2024-04-24T08:20:10.896796+0000","last_undegraded":"2024-04-24T08:20:10.896796+0000","last_fullsized":"2024-04-24T08:20:10.896796+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:10:50.551575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897147+0000","last_change":"2024-04-24T08:20:09.782866+0000","last_active":"2024-04-24T08:20:10.897147+0000","last_peered":"2024-04-24T08:20:10.897147+0000","last_clean":"2024-04-24T08:20:10.897147+0000","last_became_active":"2024-04-24T08:20:09.781963+0000","last_became_peered":"2024-04-24T08:20:09.781963+0000","last_unstale":"2024-04-24T08:20:10.897147+0000","last_undegraded":"2024-04-24T08:20:10.897147+0000","last_fullsized":"2024-04-24T08:20:10.897147+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:42:59.649790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.894889+0000","last_change":"2024-04-24T08:20:09.799141+0000","last_active":"2024-04-24T08:20:10.894889+0000","last_peered":"2024-04-24T08:20:10.894889+0000","last_clean":"2024-04-24T08:20:10.894889+0000","last_became_active":"2024-04-24T08:20:09.798679+0000","last_became_peered":"2024-04-24T08:20:09.798679+0000","last_unstale":"2024-04-24T08:20:10.894889+0000","last_undegraded":"2024-04-24T08:20:10.894889+0000","last_fullsized":"2024-04-24T08:20:10.894889+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:06:29.713888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791716+0000","last_change":"2024-04-24T08:20:09.789678+0000","last_active":"2024-04-24T08:20:10.791716+0000","last_peered":"2024-04-24T08:20:10.791716+0000","last_clean":"2024-04-24T08:20:10.791716+0000","last_became_active":"2024-04-24T08:20:09.787273+0000","last_became_peered":"2024-04-24T08:20:09.787273+0000","last_unstale":"2024-04-24T08:20:10.791716+0000","last_undegraded":"2024-04-24T08:20:10.791716+0000","last_fullsized":"2024-04-24T08:20:10.791716+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:29.553044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898926+0000","last_change":"2024-04-24T08:20:09.785296+0000","last_active":"2024-04-24T08:20:10.898926+0000","last_peered":"2024-04-24T08:20:10.898926+0000","last_clean":"2024-04-24T08:20:10.898926+0000","last_became_active":"2024-04-24T08:20:09.784996+0000","last_became_peered":"2024-04-24T08:20:09.784996+0000","last_unstale":"2024-04-24T08:20:10.898926+0000","last_undegraded":"2024-04-24T08:20:10.898926+0000","last_fullsized":"2024-04-24T08:20:10.898926+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:09:43.173248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897367+0000","last_change":"2024-04-24T08:20:09.797840+0000","last_active":"2024-04-24T08:20:10.897367+0000","last_peered":"2024-04-24T08:20:10.897367+0000","last_clean":"2024-04-24T08:20:10.897367+0000","last_became_active":"2024-04-24T08:20:09.797575+0000","last_became_peered":"2024-04-24T08:20:09.797575+0000","last_unstale":"2024-04-24T08:20:10.897367+0000","last_undegraded":"2024-04-24T08:20:10.897367+0000","last_fullsized":"2024-04-24T08:20:10.897367+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:35.871237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791599+0000","last_change":"2024-04-24T08:20:09.789338+0000","last_active":"2024-04-24T08:20:10.791599+0000","last_peered":"2024-04-24T08:20:10.791599+0000","last_clean":"2024-04-24T08:20:10.791599+0000","last_became_active":"2024-04-24T08:20:09.787008+0000","last_became_peered":"2024-04-24T08:20:09.787008+0000","last_unstale":"2024-04-24T08:20:10.791599+0000","last_undegraded":"2024-04-24T08:20:10.791599+0000","last_fullsized":"2024-04-24T08:20:10.791599+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:03.358776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791614+0000","last_change":"2024-04-24T08:20:09.785546+0000","last_active":"2024-04-24T08:20:10.791614+0000","last_peered":"2024-04-24T08:20:10.791614+0000","last_clean":"2024-04-24T08:20:10.791614+0000","last_became_active":"2024-04-24T08:20:09.785285+0000","last_became_peered":"2024-04-24T08:20:09.785285+0000","last_unstale":"2024-04-24T08:20:10.791614+0000","last_undegraded":"2024-04-24T08:20:10.791614+0000","last_fullsized":"2024-04-24T08:20:10.791614+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:47.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899951+0000","last_change":"2024-04-24T08:20:09.792270+0000","last_active":"2024-04-24T08:20:10.899951+0000","last_peered":"2024-04-24T08:20:10.899951+0000","last_clean":"2024-04-24T08:20:10.899951+0000","last_became_active":"2024-04-24T08:20:09.791977+0000","last_became_peered":"2024-04-24T08:20:09.791977+0000","last_unstale":"2024-04-24T08:20:10.899951+0000","last_undegraded":"2024-04-24T08:20:10.899951+0000","last_fullsized":"2024-04-24T08:20:10.899951+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:47.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788610+0000","last_change":"2024-04-24T08:20:09.788141+0000","last_active":"2024-04-24T08:20:10.788610+0000","last_peered":"2024-04-24T08:20:10.788610+0000","last_clean":"2024-04-24T08:20:10.788610+0000","last_became_active":"2024-04-24T08:20:09.787507+0000","last_became_peered":"2024-04-24T08:20:09.787507+0000","last_unstale":"2024-04-24T08:20:10.788610+0000","last_undegraded":"2024-04-24T08:20:10.788610+0000","last_fullsized":"2024-04-24T08:20:10.788610+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:21.652074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897965+0000","last_change":"2024-04-24T08:20:09.787331+0000","last_active":"2024-04-24T08:20:10.897965+0000","last_peered":"2024-04-24T08:20:10.897965+0000","last_clean":"2024-04-24T08:20:10.897965+0000","last_became_active":"2024-04-24T08:20:09.786840+0000","last_became_peered":"2024-04-24T08:20:09.786840+0000","last_unstale":"2024-04-24T08:20:10.897965+0000","last_undegraded":"2024-04-24T08:20:10.897965+0000","last_fullsized":"2024-04-24T08:20:10.897965+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:47.526466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791717+0000","last_change":"2024-04-24T08:20:09.787529+0000","last_active":"2024-04-24T08:20:10.791717+0000","last_peered":"2024-04-24T08:20:10.791717+0000","last_clean":"2024-04-24T08:20:10.791717+0000","last_became_active":"2024-04-24T08:20:09.787236+0000","last_became_peered":"2024-04-24T08:20:09.787236+0000","last_unstale":"2024-04-24T08:20:10.791717+0000","last_undegraded":"2024-04-24T08:20:10.791717+0000","last_fullsized":"2024-04-24T08:20:10.791717+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:09:43.173248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896884+0000","last_change":"2024-04-24T08:20:09.788225+0000","last_active":"2024-04-24T08:20:10.896884+0000","last_peered":"2024-04-24T08:20:10.896884+0000","last_clean":"2024-04-24T08:20:10.896884+0000","last_became_active":"2024-04-24T08:20:09.787936+0000","last_became_peered":"2024-04-24T08:20:09.787936+0000","last_unstale":"2024-04-24T08:20:10.896884+0000","last_undegraded":"2024-04-24T08:20:10.896884+0000","last_fullsized":"2024-04-24T08:20:10.896884+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:20:18.482020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899461+0000","last_change":"2024-04-24T08:20:09.794928+0000","last_active":"2024-04-24T08:20:10.899461+0000","last_peered":"2024-04-24T08:20:10.899461+0000","last_clean":"2024-04-24T08:20:10.899461+0000","last_became_active":"2024-04-24T08:20:09.794289+0000","last_became_peered":"2024-04-24T08:20:09.794289+0000","last_unstale":"2024-04-24T08:20:10.899461+0000","last_undegraded":"2024-04-24T08:20:10.899461+0000","last_fullsized":"2024-04-24T08:20:10.899461+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:51:07.962266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898786+0000","last_change":"2024-04-24T08:20:09.794776+0000","last_active":"2024-04-24T08:20:10.898786+0000","last_peered":"2024-04-24T08:20:10.898786+0000","last_clean":"2024-04-24T08:20:10.898786+0000","last_became_active":"2024-04-24T08:20:09.794151+0000","last_became_peered":"2024-04-24T08:20:09.794151+0000","last_unstale":"2024-04-24T08:20:10.898786+0000","last_undegraded":"2024-04-24T08:20:10.898786+0000","last_fullsized":"2024-04-24T08:20:10.898786+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:03.358776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790533+0000","last_change":"2024-04-24T08:20:09.791933+0000","last_active":"2024-04-24T08:20:10.790533+0000","last_peered":"2024-04-24T08:20:10.790533+0000","last_clean":"2024-04-24T08:20:10.790533+0000","last_became_active":"2024-04-24T08:20:09.789144+0000","last_became_peered":"2024-04-24T08:20:09.789144+0000","last_unstale":"2024-04-24T08:20:10.790533+0000","last_undegraded":"2024-04-24T08:20:10.790533+0000","last_fullsized":"2024-04-24T08:20:10.790533+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:57.166226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789744+0000","last_change":"2024-04-24T08:20:09.779514+0000","last_active":"2024-04-24T08:20:10.789744+0000","last_peered":"2024-04-24T08:20:10.789744+0000","last_clean":"2024-04-24T08:20:10.789744+0000","last_became_active":"2024-04-24T08:20:09.779002+0000","last_became_peered":"2024-04-24T08:20:09.779002+0000","last_unstale":"2024-04-24T08:20:10.789744+0000","last_undegraded":"2024-04-24T08:20:10.789744+0000","last_fullsized":"2024-04-24T08:20:10.789744+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:42:59.649790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895194+0000","last_change":"2024-04-24T08:20:09.789034+0000","last_active":"2024-04-24T08:20:10.895194+0000","last_peered":"2024-04-24T08:20:10.895194+0000","last_clean":"2024-04-24T08:20:10.895194+0000","last_became_active":"2024-04-24T08:20:09.788764+0000","last_became_peered":"2024-04-24T08:20:09.788764+0000","last_unstale":"2024-04-24T08:20:10.895194+0000","last_undegraded":"2024-04-24T08:20:10.895194+0000","last_fullsized":"2024-04-24T08:20:10.895194+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:29.411232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896727+0000","last_change":"2024-04-24T08:20:09.796857+0000","last_active":"2024-04-24T08:20:10.896727+0000","last_peered":"2024-04-24T08:20:10.896727+0000","last_clean":"2024-04-24T08:20:10.896727+0000","last_became_active":"2024-04-24T08:20:09.795564+0000","last_became_peered":"2024-04-24T08:20:09.795564+0000","last_unstale":"2024-04-24T08:20:10.896727+0000","last_undegraded":"2024-04-24T08:20:10.896727+0000","last_fullsized":"2024-04-24T08:20:10.896727+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:44:55.266192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899863+0000","last_change":"2024-04-24T08:20:09.779341+0000","last_active":"2024-04-24T08:20:10.899863+0000","last_peered":"2024-04-24T08:20:10.899863+0000","last_clean":"2024-04-24T08:20:10.899863+0000","last_became_active":"2024-04-24T08:20:09.779056+0000","last_became_peered":"2024-04-24T08:20:09.779056+0000","last_unstale":"2024-04-24T08:20:10.899863+0000","last_undegraded":"2024-04-24T08:20:10.899863+0000","last_fullsized":"2024-04-24T08:20:10.899863+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:34.919462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"108'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790412+0000","last_change":"2024-04-24T08:20:08.765030+0000","last_active":"2024-04-24T08:20:10.790412+0000","last_peered":"2024-04-24T08:20:10.790412+0000","last_clean":"2024-04-24T08:20:10.790412+0000","last_became_active":"2024-04-24T08:20:08.764730+0000","last_became_peered":"2024-04-24T08:20:08.764730+0000","last_unstale":"2024-04-24T08:20:10.790412+0000","last_undegraded":"2024-04-24T08:20:10.790412+0000","last_fullsized":"2024-04-24T08:20:10.790412+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:19:40.821738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898429+0000","last_change":"2024-04-24T08:20:09.793872+0000","last_active":"2024-04-24T08:20:10.898429+0000","last_peered":"2024-04-24T08:20:10.898429+0000","last_clean":"2024-04-24T08:20:10.898429+0000","last_became_active":"2024-04-24T08:20:09.793548+0000","last_became_peered":"2024-04-24T08:20:09.793548+0000","last_unstale":"2024-04-24T08:20:10.898429+0000","last_undegraded":"2024-04-24T08:20:10.898429+0000","last_fullsized":"2024-04-24T08:20:10.898429+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:03.358776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898085+0000","last_change":"2024-04-24T08:20:08.764950+0000","last_active":"2024-04-24T08:20:10.898085+0000","last_peered":"2024-04-24T08:20:10.898085+0000","last_clean":"2024-04-24T08:20:10.898085+0000","last_became_active":"2024-04-24T08:20:08.764618+0000","last_became_peered":"2024-04-24T08:20:08.764618+0000","last_unstale":"2024-04-24T08:20:10.898085+0000","last_undegraded":"2024-04-24T08:20:10.898085+0000","last_fullsized":"2024-04-24T08:20:10.898085+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:02.260282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899378+0000","last_change":"2024-04-24T08:20:09.796732+0000","last_active":"2024-04-24T08:20:10.899378+0000","last_peered":"2024-04-24T08:20:10.899378+0000","last_clean":"2024-04-24T08:20:10.899378+0000","last_became_active":"2024-04-24T08:20:09.795416+0000","last_became_peered":"2024-04-24T08:20:09.795416+0000","last_unstale":"2024-04-24T08:20:10.899378+0000","last_undegraded":"2024-04-24T08:20:10.899378+0000","last_fullsized":"2024-04-24T08:20:10.899378+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:38.016468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787234+0000","last_change":"2024-04-24T08:20:08.772932+0000","last_active":"2024-04-24T08:20:10.787234+0000","last_peered":"2024-04-24T08:20:10.787234+0000","last_clean":"2024-04-24T08:20:10.787234+0000","last_became_active":"2024-04-24T08:20:08.772234+0000","last_became_peered":"2024-04-24T08:20:08.772234+0000","last_unstale":"2024-04-24T08:20:10.787234+0000","last_undegraded":"2024-04-24T08:20:10.787234+0000","last_fullsized":"2024-04-24T08:20:10.787234+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:12:45.289874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897735+0000","last_change":"2024-04-24T08:20:09.797817+0000","last_active":"2024-04-24T08:20:10.897735+0000","last_peered":"2024-04-24T08:20:10.897735+0000","last_clean":"2024-04-24T08:20:10.897735+0000","last_became_active":"2024-04-24T08:20:09.797517+0000","last_became_peered":"2024-04-24T08:20:09.797517+0000","last_unstale":"2024-04-24T08:20:10.897735+0000","last_undegraded":"2024-04-24T08:20:10.897735+0000","last_fullsized":"2024-04-24T08:20:10.897735+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:21.652074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787417+0000","last_change":"2024-04-24T08:20:08.770132+0000","last_active":"2024-04-24T08:20:10.787417+0000","last_peered":"2024-04-24T08:20:10.787417+0000","last_clean":"2024-04-24T08:20:10.787417+0000","last_became_active":"2024-04-24T08:20:08.769756+0000","last_became_peered":"2024-04-24T08:20:08.769756+0000","last_unstale":"2024-04-24T08:20:10.787417+0000","last_undegraded":"2024-04-24T08:20:10.787417+0000","last_fullsized":"2024-04-24T08:20:10.787417+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:57:23.545844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896040+0000","last_change":"2024-04-24T08:20:09.804520+0000","last_active":"2024-04-24T08:20:10.896040+0000","last_peered":"2024-04-24T08:20:10.896040+0000","last_clean":"2024-04-24T08:20:10.896040+0000","last_became_active":"2024-04-24T08:20:09.804216+0000","last_became_peered":"2024-04-24T08:20:09.804216+0000","last_unstale":"2024-04-24T08:20:10.896040+0000","last_undegraded":"2024-04-24T08:20:10.896040+0000","last_fullsized":"2024-04-24T08:20:10.896040+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:45:39.680748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789936+0000","last_change":"2024-04-24T08:20:08.763740+0000","last_active":"2024-04-24T08:20:10.789936+0000","last_peered":"2024-04-24T08:20:10.789936+0000","last_clean":"2024-04-24T08:20:10.789936+0000","last_became_active":"2024-04-24T08:20:08.763472+0000","last_became_peered":"2024-04-24T08:20:08.763472+0000","last_unstale":"2024-04-24T08:20:10.789936+0000","last_undegraded":"2024-04-24T08:20:10.789936+0000","last_fullsized":"2024-04-24T08:20:10.789936+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:13:02.412872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790984+0000","last_change":"2024-04-24T08:20:09.796520+0000","last_active":"2024-04-24T08:20:10.790984+0000","last_peered":"2024-04-24T08:20:10.790984+0000","last_clean":"2024-04-24T08:20:10.790984+0000","last_became_active":"2024-04-24T08:20:09.793445+0000","last_became_peered":"2024-04-24T08:20:09.793445+0000","last_unstale":"2024-04-24T08:20:10.790984+0000","last_undegraded":"2024-04-24T08:20:10.790984+0000","last_fullsized":"2024-04-24T08:20:10.790984+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:34.067483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.900109+0000","last_change":"2024-04-24T08:20:08.762014+0000","last_active":"2024-04-24T08:20:10.900109+0000","last_peered":"2024-04-24T08:20:10.900109+0000","last_clean":"2024-04-24T08:20:10.900109+0000","last_became_active":"2024-04-24T08:20:08.761675+0000","last_became_peered":"2024-04-24T08:20:08.761675+0000","last_unstale":"2024-04-24T08:20:10.900109+0000","last_undegraded":"2024-04-24T08:20:10.900109+0000","last_fullsized":"2024-04-24T08:20:10.900109+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:23.173525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895119+0000","last_change":"2024-04-24T08:20:09.795885+0000","last_active":"2024-04-24T08:20:10.895119+0000","last_peered":"2024-04-24T08:20:10.895119+0000","last_clean":"2024-04-24T08:20:10.895119+0000","last_became_active":"2024-04-24T08:20:09.795631+0000","last_became_peered":"2024-04-24T08:20:09.795631+0000","last_unstale":"2024-04-24T08:20:10.895119+0000","last_undegraded":"2024-04-24T08:20:10.895119+0000","last_fullsized":"2024-04-24T08:20:10.895119+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:19:33.323423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788665+0000","last_change":"2024-04-24T08:20:08.755417+0000","last_active":"2024-04-24T08:20:10.788665+0000","last_peered":"2024-04-24T08:20:10.788665+0000","last_clean":"2024-04-24T08:20:10.788665+0000","last_became_active":"2024-04-24T08:20:08.755131+0000","last_became_peered":"2024-04-24T08:20:08.755131+0000","last_unstale":"2024-04-24T08:20:10.788665+0000","last_undegraded":"2024-04-24T08:20:10.788665+0000","last_fullsized":"2024-04-24T08:20:10.788665+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:23.173525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789296+0000","last_change":"2024-04-24T08:20:09.785134+0000","last_active":"2024-04-24T08:20:10.789296+0000","last_peered":"2024-04-24T08:20:10.789296+0000","last_clean":"2024-04-24T08:20:10.789296+0000","last_became_active":"2024-04-24T08:20:09.784131+0000","last_became_peered":"2024-04-24T08:20:09.784131+0000","last_unstale":"2024-04-24T08:20:10.789296+0000","last_undegraded":"2024-04-24T08:20:10.789296+0000","last_fullsized":"2024-04-24T08:20:10.789296+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:16:12.799342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895001+0000","last_change":"2024-04-24T08:20:08.764423+0000","last_active":"2024-04-24T08:20:10.895001+0000","last_peered":"2024-04-24T08:20:10.895001+0000","last_clean":"2024-04-24T08:20:10.895001+0000","last_became_active":"2024-04-24T08:20:08.764017+0000","last_became_peered":"2024-04-24T08:20:08.764017+0000","last_unstale":"2024-04-24T08:20:10.895001+0000","last_undegraded":"2024-04-24T08:20:10.895001+0000","last_fullsized":"2024-04-24T08:20:10.895001+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:47.073639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790527+0000","last_change":"2024-04-24T08:20:09.775184+0000","last_active":"2024-04-24T08:20:10.790527+0000","last_peered":"2024-04-24T08:20:10.790527+0000","last_clean":"2024-04-24T08:20:10.790527+0000","last_became_active":"2024-04-24T08:20:09.774870+0000","last_became_peered":"2024-04-24T08:20:09.774870+0000","last_unstale":"2024-04-24T08:20:10.790527+0000","last_undegraded":"2024-04-24T08:20:10.790527+0000","last_fullsized":"2024-04-24T08:20:10.790527+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:38.016468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787733+0000","last_change":"2024-04-24T08:20:08.768692+0000","last_active":"2024-04-24T08:20:10.787733+0000","last_peered":"2024-04-24T08:20:10.787733+0000","last_clean":"2024-04-24T08:20:10.787733+0000","last_became_active":"2024-04-24T08:20:08.768352+0000","last_became_peered":"2024-04-24T08:20:08.768352+0000","last_unstale":"2024-04-24T08:20:10.787733+0000","last_undegraded":"2024-04-24T08:20:10.787733+0000","last_fullsized":"2024-04-24T08:20:10.787733+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:52:34.264489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898831+0000","last_change":"2024-04-24T08:20:09.799946+0000","last_active":"2024-04-24T08:20:10.898831+0000","last_peered":"2024-04-24T08:20:10.898831+0000","last_clean":"2024-04-24T08:20:10.898831+0000","last_became_active":"2024-04-24T08:20:09.799680+0000","last_became_peered":"2024-04-24T08:20:09.799680+0000","last_unstale":"2024-04-24T08:20:10.898831+0000","last_undegraded":"2024-04-24T08:20:10.898831+0000","last_fullsized":"2024-04-24T08:20:10.898831+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:47.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899134+0000","last_change":"2024-04-24T08:20:08.767491+0000","last_active":"2024-04-24T08:20:10.899134+0000","last_peered":"2024-04-24T08:20:10.899134+0000","last_clean":"2024-04-24T08:20:10.899134+0000","last_became_active":"2024-04-24T08:20:08.767200+0000","last_became_peered":"2024-04-24T08:20:08.767200+0000","last_unstale":"2024-04-24T08:20:10.899134+0000","last_undegraded":"2024-04-24T08:20:10.899134+0000","last_fullsized":"2024-04-24T08:20:10.899134+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:29:39.437886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898703+0000","last_change":"2024-04-24T08:20:09.784394+0000","last_active":"2024-04-24T08:20:10.898703+0000","last_peered":"2024-04-24T08:20:10.898703+0000","last_clean":"2024-04-24T08:20:10.898703+0000","last_became_active":"2024-04-24T08:20:09.784125+0000","last_became_peered":"2024-04-24T08:20:09.784125+0000","last_unstale":"2024-04-24T08:20:10.898703+0000","last_undegraded":"2024-04-24T08:20:10.898703+0000","last_fullsized":"2024-04-24T08:20:10.898703+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:13.818493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"108'3","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896283+0000","last_change":"2024-04-24T08:20:08.771750+0000","last_active":"2024-04-24T08:20:10.896283+0000","last_peered":"2024-04-24T08:20:10.896283+0000","last_clean":"2024-04-24T08:20:10.896283+0000","last_became_active":"2024-04-24T08:20:08.771450+0000","last_became_peered":"2024-04-24T08:20:08.771450+0000","last_unstale":"2024-04-24T08:20:10.896283+0000","last_undegraded":"2024-04-24T08:20:10.896283+0000","last_fullsized":"2024-04-24T08:20:10.896283+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:56.149044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790276+0000","last_change":"2024-04-24T08:20:09.784652+0000","last_active":"2024-04-24T08:20:10.790276+0000","last_peered":"2024-04-24T08:20:10.790276+0000","last_clean":"2024-04-24T08:20:10.790276+0000","last_became_active":"2024-04-24T08:20:09.784367+0000","last_became_peered":"2024-04-24T08:20:09.784367+0000","last_unstale":"2024-04-24T08:20:10.790276+0000","last_undegraded":"2024-04-24T08:20:10.790276+0000","last_fullsized":"2024-04-24T08:20:10.790276+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:56.817425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"108'6","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896458+0000","last_change":"2024-04-24T08:20:08.763054+0000","last_active":"2024-04-24T08:20:10.896458+0000","last_peered":"2024-04-24T08:20:10.896458+0000","last_clean":"2024-04-24T08:20:10.896458+0000","last_became_active":"2024-04-24T08:20:08.762785+0000","last_became_peered":"2024-04-24T08:20:08.762785+0000","last_unstale":"2024-04-24T08:20:10.896458+0000","last_undegraded":"2024-04-24T08:20:10.896458+0000","last_fullsized":"2024-04-24T08:20:10.896458+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:02.341594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.900015+0000","last_change":"2024-04-24T08:20:09.775131+0000","last_active":"2024-04-24T08:20:10.900015+0000","last_peered":"2024-04-24T08:20:10.900015+0000","last_clean":"2024-04-24T08:20:10.900015+0000","last_became_active":"2024-04-24T08:20:09.774820+0000","last_became_peered":"2024-04-24T08:20:09.774820+0000","last_unstale":"2024-04-24T08:20:10.900015+0000","last_undegraded":"2024-04-24T08:20:10.900015+0000","last_fullsized":"2024-04-24T08:20:10.900015+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:09:43.173248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898575+0000","last_change":"2024-04-24T08:20:08.771246+0000","last_active":"2024-04-24T08:20:10.898575+0000","last_peered":"2024-04-24T08:20:10.898575+0000","last_clean":"2024-04-24T08:20:10.898575+0000","last_became_active":"2024-04-24T08:20:08.770915+0000","last_became_peered":"2024-04-24T08:20:08.770915+0000","last_unstale":"2024-04-24T08:20:10.898575+0000","last_undegraded":"2024-04-24T08:20:10.898575+0000","last_fullsized":"2024-04-24T08:20:10.898575+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:19:40.821738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897970+0000","last_change":"2024-04-24T08:20:09.787926+0000","last_active":"2024-04-24T08:20:10.897970+0000","last_peered":"2024-04-24T08:20:10.897970+0000","last_clean":"2024-04-24T08:20:10.897970+0000","last_became_active":"2024-04-24T08:20:09.787610+0000","last_became_peered":"2024-04-24T08:20:09.787610+0000","last_unstale":"2024-04-24T08:20:10.897970+0000","last_undegraded":"2024-04-24T08:20:10.897970+0000","last_fullsized":"2024-04-24T08:20:10.897970+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:57.166226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898305+0000","last_change":"2024-04-24T08:20:08.770641+0000","last_active":"2024-04-24T08:20:10.898305+0000","last_peered":"2024-04-24T08:20:10.898305+0000","last_clean":"2024-04-24T08:20:10.898305+0000","last_became_active":"2024-04-24T08:20:08.770281+0000","last_became_peered":"2024-04-24T08:20:08.770281+0000","last_unstale":"2024-04-24T08:20:10.898305+0000","last_undegraded":"2024-04-24T08:20:10.898305+0000","last_fullsized":"2024-04-24T08:20:10.898305+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:34.854054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898155+0000","last_change":"2024-04-24T08:20:09.789809+0000","last_active":"2024-04-24T08:20:10.898155+0000","last_peered":"2024-04-24T08:20:10.898155+0000","last_clean":"2024-04-24T08:20:10.898155+0000","last_became_active":"2024-04-24T08:20:09.789248+0000","last_became_peered":"2024-04-24T08:20:09.789248+0000","last_unstale":"2024-04-24T08:20:10.898155+0000","last_undegraded":"2024-04-24T08:20:10.898155+0000","last_fullsized":"2024-04-24T08:20:10.898155+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:34.067483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"108'8","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.786954+0000","last_change":"2024-04-24T08:20:08.760398+0000","last_active":"2024-04-24T08:20:10.786954+0000","last_peered":"2024-04-24T08:20:10.786954+0000","last_clean":"2024-04-24T08:20:10.786954+0000","last_became_active":"2024-04-24T08:20:08.760075+0000","last_became_peered":"2024-04-24T08:20:08.760075+0000","last_unstale":"2024-04-24T08:20:10.786954+0000","last_undegraded":"2024-04-24T08:20:10.786954+0000","last_fullsized":"2024-04-24T08:20:10.786954+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:20.208658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788203+0000","last_change":"2024-04-24T08:20:09.797624+0000","last_active":"2024-04-24T08:20:10.788203+0000","last_peered":"2024-04-24T08:20:10.788203+0000","last_clean":"2024-04-24T08:20:10.788203+0000","last_became_active":"2024-04-24T08:20:09.796587+0000","last_became_peered":"2024-04-24T08:20:09.796587+0000","last_unstale":"2024-04-24T08:20:10.788203+0000","last_undegraded":"2024-04-24T08:20:10.788203+0000","last_fullsized":"2024-04-24T08:20:10.788203+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:10:52.836766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897475+0000","last_change":"2024-04-24T08:20:08.769979+0000","last_active":"2024-04-24T08:20:10.897475+0000","last_peered":"2024-04-24T08:20:10.897475+0000","last_clean":"2024-04-24T08:20:10.897475+0000","last_became_active":"2024-04-24T08:20:08.769667+0000","last_became_peered":"2024-04-24T08:20:08.769667+0000","last_unstale":"2024-04-24T08:20:10.897475+0000","last_undegraded":"2024-04-24T08:20:10.897475+0000","last_fullsized":"2024-04-24T08:20:10.897475+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:33.050301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899848+0000","last_change":"2024-04-24T08:20:09.795745+0000","last_active":"2024-04-24T08:20:10.899848+0000","last_peered":"2024-04-24T08:20:10.899848+0000","last_clean":"2024-04-24T08:20:10.899848+0000","last_became_active":"2024-04-24T08:20:09.795459+0000","last_became_peered":"2024-04-24T08:20:09.795459+0000","last_unstale":"2024-04-24T08:20:10.899848+0000","last_undegraded":"2024-04-24T08:20:10.899848+0000","last_fullsized":"2024-04-24T08:20:10.899848+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:07:38.079929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788129+0000","last_change":"2024-04-24T08:20:08.762510+0000","last_active":"2024-04-24T08:20:10.788129+0000","last_peered":"2024-04-24T08:20:10.788129+0000","last_clean":"2024-04-24T08:20:10.788129+0000","last_became_active":"2024-04-24T08:20:08.762168+0000","last_became_peered":"2024-04-24T08:20:08.762168+0000","last_unstale":"2024-04-24T08:20:10.788129+0000","last_undegraded":"2024-04-24T08:20:10.788129+0000","last_fullsized":"2024-04-24T08:20:10.788129+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:05:22.039965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898404+0000","last_change":"2024-04-24T08:20:09.800369+0000","last_active":"2024-04-24T08:20:10.898404+0000","last_peered":"2024-04-24T08:20:10.898404+0000","last_clean":"2024-04-24T08:20:10.898404+0000","last_became_active":"2024-04-24T08:20:09.799535+0000","last_became_peered":"2024-04-24T08:20:09.799535+0000","last_unstale":"2024-04-24T08:20:10.898404+0000","last_undegraded":"2024-04-24T08:20:10.898404+0000","last_fullsized":"2024-04-24T08:20:10.898404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:25.726242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898135+0000","last_change":"2024-04-24T08:20:08.761477+0000","last_active":"2024-04-24T08:20:10.898135+0000","last_peered":"2024-04-24T08:20:10.898135+0000","last_clean":"2024-04-24T08:20:10.898135+0000","last_became_active":"2024-04-24T08:20:08.759812+0000","last_became_peered":"2024-04-24T08:20:08.759812+0000","last_unstale":"2024-04-24T08:20:10.898135+0000","last_undegraded":"2024-04-24T08:20:10.898135+0000","last_fullsized":"2024-04-24T08:20:10.898135+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:27:26.888390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897806+0000","last_change":"2024-04-24T08:20:09.795821+0000","last_active":"2024-04-24T08:20:10.897806+0000","last_peered":"2024-04-24T08:20:10.897806+0000","last_clean":"2024-04-24T08:20:10.897806+0000","last_became_active":"2024-04-24T08:20:09.795409+0000","last_became_peered":"2024-04-24T08:20:09.795409+0000","last_unstale":"2024-04-24T08:20:10.897806+0000","last_undegraded":"2024-04-24T08:20:10.897806+0000","last_fullsized":"2024-04-24T08:20:10.897806+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:47.526466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790642+0000","last_change":"2024-04-24T08:20:08.767284+0000","last_active":"2024-04-24T08:20:10.790642+0000","last_peered":"2024-04-24T08:20:10.790642+0000","last_clean":"2024-04-24T08:20:10.790642+0000","last_became_active":"2024-04-24T08:20:08.766880+0000","last_became_peered":"2024-04-24T08:20:08.766880+0000","last_unstale":"2024-04-24T08:20:10.790642+0000","last_undegraded":"2024-04-24T08:20:10.790642+0000","last_fullsized":"2024-04-24T08:20:10.790642+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:08.415565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896838+0000","last_change":"2024-04-24T08:20:09.787635+0000","last_active":"2024-04-24T08:20:10.896838+0000","last_peered":"2024-04-24T08:20:10.896838+0000","last_clean":"2024-04-24T08:20:10.896838+0000","last_became_active":"2024-04-24T08:20:09.787270+0000","last_became_peered":"2024-04-24T08:20:09.787270+0000","last_unstale":"2024-04-24T08:20:10.896838+0000","last_undegraded":"2024-04-24T08:20:10.896838+0000","last_fullsized":"2024-04-24T08:20:10.896838+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:23:51.545972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899062+0000","last_change":"2024-04-24T08:20:08.773991+0000","last_active":"2024-04-24T08:20:10.899062+0000","last_peered":"2024-04-24T08:20:10.899062+0000","last_clean":"2024-04-24T08:20:10.899062+0000","last_became_active":"2024-04-24T08:20:08.773703+0000","last_became_peered":"2024-04-24T08:20:08.773703+0000","last_unstale":"2024-04-24T08:20:10.899062+0000","last_undegraded":"2024-04-24T08:20:10.899062+0000","last_fullsized":"2024-04-24T08:20:10.899062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:13:02.412872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895443+0000","last_change":"2024-04-24T08:20:09.797747+0000","last_active":"2024-04-24T08:20:10.895443+0000","last_peered":"2024-04-24T08:20:10.895443+0000","last_clean":"2024-04-24T08:20:10.895443+0000","last_became_active":"2024-04-24T08:20:09.797468+0000","last_became_peered":"2024-04-24T08:20:09.797468+0000","last_unstale":"2024-04-24T08:20:10.895443+0000","last_undegraded":"2024-04-24T08:20:10.895443+0000","last_fullsized":"2024-04-24T08:20:10.895443+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:10:12.534666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.786965+0000","last_change":"2024-04-24T08:20:08.769543+0000","last_active":"2024-04-24T08:20:10.786965+0000","last_peered":"2024-04-24T08:20:10.786965+0000","last_clean":"2024-04-24T08:20:10.786965+0000","last_became_active":"2024-04-24T08:20:08.767940+0000","last_became_peered":"2024-04-24T08:20:08.767940+0000","last_unstale":"2024-04-24T08:20:10.786965+0000","last_undegraded":"2024-04-24T08:20:10.786965+0000","last_fullsized":"2024-04-24T08:20:10.786965+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:16:28.019646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897242+0000","last_change":"2024-04-24T08:20:09.790241+0000","last_active":"2024-04-24T08:20:10.897242+0000","last_peered":"2024-04-24T08:20:10.897242+0000","last_clean":"2024-04-24T08:20:10.897242+0000","last_became_active":"2024-04-24T08:20:09.789926+0000","last_became_peered":"2024-04-24T08:20:09.789926+0000","last_unstale":"2024-04-24T08:20:10.897242+0000","last_undegraded":"2024-04-24T08:20:10.897242+0000","last_fullsized":"2024-04-24T08:20:10.897242+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:52:35.867500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897241+0000","last_change":"2024-04-24T08:20:08.769318+0000","last_active":"2024-04-24T08:20:10.897241+0000","last_peered":"2024-04-24T08:20:10.897241+0000","last_clean":"2024-04-24T08:20:10.897241+0000","last_became_active":"2024-04-24T08:20:08.769050+0000","last_became_peered":"2024-04-24T08:20:08.769050+0000","last_unstale":"2024-04-24T08:20:10.897241+0000","last_undegraded":"2024-04-24T08:20:10.897241+0000","last_fullsized":"2024-04-24T08:20:10.897241+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:10:51.819583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898432+0000","last_change":"2024-04-24T08:20:09.792723+0000","last_active":"2024-04-24T08:20:10.898432+0000","last_peered":"2024-04-24T08:20:10.898432+0000","last_clean":"2024-04-24T08:20:10.898432+0000","last_became_active":"2024-04-24T08:20:09.792405+0000","last_became_peered":"2024-04-24T08:20:09.792405+0000","last_unstale":"2024-04-24T08:20:10.898432+0000","last_undegraded":"2024-04-24T08:20:10.898432+0000","last_fullsized":"2024-04-24T08:20:10.898432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:48.090822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895133+0000","last_change":"2024-04-24T08:20:08.767252+0000","last_active":"2024-04-24T08:20:10.895133+0000","last_peered":"2024-04-24T08:20:10.895133+0000","last_clean":"2024-04-24T08:20:10.895133+0000","last_became_active":"2024-04-24T08:20:08.766235+0000","last_became_peered":"2024-04-24T08:20:08.766235+0000","last_unstale":"2024-04-24T08:20:10.895133+0000","last_undegraded":"2024-04-24T08:20:10.895133+0000","last_fullsized":"2024-04-24T08:20:10.895133+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:12.801310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899088+0000","last_change":"2024-04-24T08:20:09.799590+0000","last_active":"2024-04-24T08:20:10.899088+0000","last_peered":"2024-04-24T08:20:10.899088+0000","last_clean":"2024-04-24T08:20:10.899088+0000","last_became_active":"2024-04-24T08:20:09.799255+0000","last_became_peered":"2024-04-24T08:20:09.799255+0000","last_unstale":"2024-04-24T08:20:10.899088+0000","last_undegraded":"2024-04-24T08:20:10.899088+0000","last_fullsized":"2024-04-24T08:20:10.899088+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:29.553044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899509+0000","last_change":"2024-04-24T08:20:08.774854+0000","last_active":"2024-04-24T08:20:10.899509+0000","last_peered":"2024-04-24T08:20:10.899509+0000","last_clean":"2024-04-24T08:20:10.899509+0000","last_became_active":"2024-04-24T08:20:08.773896+0000","last_became_peered":"2024-04-24T08:20:08.773896+0000","last_unstale":"2024-04-24T08:20:10.899509+0000","last_undegraded":"2024-04-24T08:20:10.899509+0000","last_fullsized":"2024-04-24T08:20:10.899509+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:12:47.591419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897513+0000","last_change":"2024-04-24T08:20:09.793423+0000","last_active":"2024-04-24T08:20:10.897513+0000","last_peered":"2024-04-24T08:20:10.897513+0000","last_clean":"2024-04-24T08:20:10.897513+0000","last_became_active":"2024-04-24T08:20:09.793148+0000","last_became_peered":"2024-04-24T08:20:09.793148+0000","last_unstale":"2024-04-24T08:20:10.897513+0000","last_undegraded":"2024-04-24T08:20:10.897513+0000","last_fullsized":"2024-04-24T08:20:10.897513+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:39:47.452847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899236+0000","last_change":"2024-04-24T08:20:08.763470+0000","last_active":"2024-04-24T08:20:10.899236+0000","last_peered":"2024-04-24T08:20:10.899236+0000","last_clean":"2024-04-24T08:20:10.899236+0000","last_became_active":"2024-04-24T08:20:08.762839+0000","last_became_peered":"2024-04-24T08:20:08.762839+0000","last_unstale":"2024-04-24T08:20:10.899236+0000","last_undegraded":"2024-04-24T08:20:10.899236+0000","last_fullsized":"2024-04-24T08:20:10.899236+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:20.634892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896284+0000","last_change":"2024-04-24T08:20:09.788152+0000","last_active":"2024-04-24T08:20:10.896284+0000","last_peered":"2024-04-24T08:20:10.896284+0000","last_clean":"2024-04-24T08:20:10.896284+0000","last_became_active":"2024-04-24T08:20:09.787847+0000","last_became_peered":"2024-04-24T08:20:09.787847+0000","last_unstale":"2024-04-24T08:20:10.896284+0000","last_undegraded":"2024-04-24T08:20:10.896284+0000","last_fullsized":"2024-04-24T08:20:10.896284+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:59:19.646168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896389+0000","last_change":"2024-04-24T08:20:08.761630+0000","last_active":"2024-04-24T08:20:10.896389+0000","last_peered":"2024-04-24T08:20:10.896389+0000","last_clean":"2024-04-24T08:20:10.896389+0000","last_became_active":"2024-04-24T08:20:08.761355+0000","last_became_peered":"2024-04-24T08:20:08.761355+0000","last_unstale":"2024-04-24T08:20:10.896389+0000","last_undegraded":"2024-04-24T08:20:10.896389+0000","last_fullsized":"2024-04-24T08:20:10.896389+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:24.709059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897649+0000","last_change":"2024-04-24T08:20:09.800757+0000","last_active":"2024-04-24T08:20:10.897649+0000","last_peered":"2024-04-24T08:20:10.897649+0000","last_clean":"2024-04-24T08:20:10.897649+0000","last_became_active":"2024-04-24T08:20:09.800064+0000","last_became_peered":"2024-04-24T08:20:09.800064+0000","last_unstale":"2024-04-24T08:20:10.897649+0000","last_undegraded":"2024-04-24T08:20:10.897649+0000","last_fullsized":"2024-04-24T08:20:10.897649+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:13.818493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898874+0000","last_change":"2024-04-24T08:20:08.770890+0000","last_active":"2024-04-24T08:20:10.898874+0000","last_peered":"2024-04-24T08:20:10.898874+0000","last_clean":"2024-04-24T08:20:10.898874+0000","last_became_active":"2024-04-24T08:20:08.770587+0000","last_became_peered":"2024-04-24T08:20:08.770587+0000","last_unstale":"2024-04-24T08:20:10.898874+0000","last_undegraded":"2024-04-24T08:20:10.898874+0000","last_fullsized":"2024-04-24T08:20:10.898874+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:24:17.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790143+0000","last_change":"2024-04-24T08:20:09.773964+0000","last_active":"2024-04-24T08:20:10.790143+0000","last_peered":"2024-04-24T08:20:10.790143+0000","last_clean":"2024-04-24T08:20:10.790143+0000","last_became_active":"2024-04-24T08:20:09.773691+0000","last_became_peered":"2024-04-24T08:20:09.773691+0000","last_unstale":"2024-04-24T08:20:10.790143+0000","last_undegraded":"2024-04-24T08:20:10.790143+0000","last_fullsized":"2024-04-24T08:20:10.790143+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:47.526466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790239+0000","last_change":"2024-04-24T08:20:08.757528+0000","last_active":"2024-04-24T08:20:10.790239+0000","last_peered":"2024-04-24T08:20:10.790239+0000","last_clean":"2024-04-24T08:20:10.790239+0000","last_became_active":"2024-04-24T08:20:08.757135+0000","last_became_peered":"2024-04-24T08:20:08.757135+0000","last_unstale":"2024-04-24T08:20:10.790239+0000","last_undegraded":"2024-04-24T08:20:10.790239+0000","last_fullsized":"2024-04-24T08:20:10.790239+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:35:49.116094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790951+0000","last_change":"2024-04-24T08:20:09.793813+0000","last_active":"2024-04-24T08:20:10.790951+0000","last_peered":"2024-04-24T08:20:10.790951+0000","last_clean":"2024-04-24T08:20:10.790951+0000","last_became_active":"2024-04-24T08:20:09.793103+0000","last_became_peered":"2024-04-24T08:20:09.793103+0000","last_unstale":"2024-04-24T08:20:10.790951+0000","last_undegraded":"2024-04-24T08:20:10.790951+0000","last_fullsized":"2024-04-24T08:20:10.790951+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:25.726242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789590+0000","last_change":"2024-04-24T08:20:08.766731+0000","last_active":"2024-04-24T08:20:10.789590+0000","last_peered":"2024-04-24T08:20:10.789590+0000","last_clean":"2024-04-24T08:20:10.789590+0000","last_became_active":"2024-04-24T08:20:08.766406+0000","last_became_peered":"2024-04-24T08:20:08.766406+0000","last_unstale":"2024-04-24T08:20:10.789590+0000","last_undegraded":"2024-04-24T08:20:10.789590+0000","last_fullsized":"2024-04-24T08:20:10.789590+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:37:01.856004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897684+0000","last_change":"2024-04-24T08:20:09.789425+0000","last_active":"2024-04-24T08:20:10.897684+0000","last_peered":"2024-04-24T08:20:10.897684+0000","last_clean":"2024-04-24T08:20:10.897684+0000","last_became_active":"2024-04-24T08:20:09.788643+0000","last_became_peered":"2024-04-24T08:20:09.788643+0000","last_unstale":"2024-04-24T08:20:10.897684+0000","last_undegraded":"2024-04-24T08:20:10.897684+0000","last_fullsized":"2024-04-24T08:20:10.897684+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:27:27.905572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787134+0000","last_change":"2024-04-24T08:20:08.768906+0000","last_active":"2024-04-24T08:20:10.787134+0000","last_peered":"2024-04-24T08:20:10.787134+0000","last_clean":"2024-04-24T08:20:10.787134+0000","last_became_active":"2024-04-24T08:20:08.768593+0000","last_became_peered":"2024-04-24T08:20:08.768593+0000","last_unstale":"2024-04-24T08:20:10.787134+0000","last_undegraded":"2024-04-24T08:20:10.787134+0000","last_fullsized":"2024-04-24T08:20:10.787134+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:47:22.864945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897148+0000","last_change":"2024-04-24T08:20:09.795062+0000","last_active":"2024-04-24T08:20:10.897148+0000","last_peered":"2024-04-24T08:20:10.897148+0000","last_clean":"2024-04-24T08:20:10.897148+0000","last_became_active":"2024-04-24T08:20:09.794686+0000","last_became_peered":"2024-04-24T08:20:09.794686+0000","last_unstale":"2024-04-24T08:20:10.897148+0000","last_undegraded":"2024-04-24T08:20:10.897148+0000","last_fullsized":"2024-04-24T08:20:10.897148+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:08:08.087220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895229+0000","last_change":"2024-04-24T08:20:08.767036+0000","last_active":"2024-04-24T08:20:10.895229+0000","last_peered":"2024-04-24T08:20:10.895229+0000","last_clean":"2024-04-24T08:20:10.895229+0000","last_became_active":"2024-04-24T08:20:08.765832+0000","last_became_peered":"2024-04-24T08:20:08.765832+0000","last_unstale":"2024-04-24T08:20:10.895229+0000","last_undegraded":"2024-04-24T08:20:10.895229+0000","last_fullsized":"2024-04-24T08:20:10.895229+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:46.310731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898004+0000","last_change":"2024-04-24T08:20:09.787121+0000","last_active":"2024-04-24T08:20:10.898004+0000","last_peered":"2024-04-24T08:20:10.898004+0000","last_clean":"2024-04-24T08:20:10.898004+0000","last_became_active":"2024-04-24T08:20:09.786467+0000","last_became_peered":"2024-04-24T08:20:09.786467+0000","last_unstale":"2024-04-24T08:20:10.898004+0000","last_undegraded":"2024-04-24T08:20:10.898004+0000","last_fullsized":"2024-04-24T08:20:10.898004+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:25.726242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789569+0000","last_change":"2024-04-24T08:20:08.767872+0000","last_active":"2024-04-24T08:20:10.789569+0000","last_peered":"2024-04-24T08:20:10.789569+0000","last_clean":"2024-04-24T08:20:10.789569+0000","last_became_active":"2024-04-24T08:20:08.767538+0000","last_became_peered":"2024-04-24T08:20:08.767538+0000","last_unstale":"2024-04-24T08:20:10.789569+0000","last_undegraded":"2024-04-24T08:20:10.789569+0000","last_fullsized":"2024-04-24T08:20:10.789569+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:10:51.819583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898446+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:20:10.898446+0000","last_peered":"2024-04-24T08:20:10.898446+0000","last_clean":"2024-04-24T08:20:10.898446+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:20:10.898446+0000","last_undegraded":"2024-04-24T08:20:10.898446+0000","last_fullsized":"2024-04-24T08:20:10.898446+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574998,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1957888,"data_stored":1180021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574998,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":589107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574996,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1363968,"data_stored":589641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574995,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":589037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574996,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1372160,"data_stored":590719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574997,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":589037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574997,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1363968,"data_stored":589161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574996,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33112,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338728,"statfs":{"total":96636764160,"available":96602857472,"internally_reserved":0,"allocated":1970176,"data_stored":1181129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:20:15.198 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-24T08:20:15.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.437+0000 7f8c45e00640 1 Processor -- start 2024-04-24T08:20:15.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.438+0000 7f8c45e00640 1 -- start start 2024-04-24T08:20:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.439+0000 7f8c45e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8c4015c7a0 0x7f8c40072ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.439+0000 7f8c45e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4015cb80 con 0x7f8c40073430 2024-04-24T08:20:15.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.439+0000 7f8c45e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c40073a40 con 0x7f8c400d0ea0 2024-04-24T08:20:15.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.439+0000 7f8c45e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8c40074000 con 0x7f8c4015c7a0 2024-04-24T08:20:15.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.439+0000 7f8c3e000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f8c40073430 0x7f8c400d0790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43260/0 (socket says 172.21.15.69:43260) 2024-04-24T08:20:15.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.439+0000 7f8c3e000640 1 -- 172.21.15.69:0/3457644094 learned_addr learned my addr 172.21.15.69:0/3457644094 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:15.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3ea00640 1 --2- 172.21.15.69:0/3457644094 >> v2:172.21.15.121:3300/0 conn(0x7f8c4015c7a0 0x7f8c40072ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 700670572 0 0) 0x7f8c4015cb80 con 0x7f8c40073430 2024-04-24T08:20:15.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c24003520 con 0x7f8c40073430 2024-04-24T08:20:15.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1388312484 0 0) 0x7f8c24003520 con 0x7f8c40073430 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 >> v2:172.21.15.121:3300/0 conn(0x7f8c4015c7a0 msgr2=0x7f8c40072ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 --2- 172.21.15.69:0/3457644094 >> v2:172.21.15.121:3300/0 conn(0x7f8c4015c7a0 0x7f8c40072ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 >> v1:172.21.15.121:6789/0 conn(0x7f8c400d0ea0 legacy=0x7f8c400adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c40073d20 con 0x7f8c40073430 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f8c2c001670 con 0x7f8c40073430 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 >> v1:172.21.15.69:6789/0 conn(0x7f8c40073430 legacy=0x7f8c400d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c240039b0 0x7f8c24003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c24004800 0x7f8c24004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8c40074000 con 0x7f8c24005210 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2400c140 con 0x7f8c24004800 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.440+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8c2400c340 con 0x7f8c240039b0 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.441+0000 7f8c3f400640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.441+0000 7f8c3e000640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c24004800 0x7f8c24004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.441+0000 7f8c3ea00640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c240039b0 0x7f8c24003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.441+0000 7f8c3f400640 1 -- 172.21.15.69:0/3457644094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c240039b0 msgr2=0x7f8c24003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:15.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.441+0000 7f8c3f400640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c240039b0 0x7f8c24003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.441+0000 7f8c3f400640 1 -- 172.21.15.69:0/3457644094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c24004800 msgr2=0x7f8c24004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:15.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.441+0000 7f8c3f400640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c24004800 0x7f8c24004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.441+0000 7f8c3f400640 1 -- 172.21.15.69:0/3457644094 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c24003520 con 0x7f8c24005210 2024-04-24T08:20:15.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.441+0000 7f8c3ea00640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c240039b0 0x7f8c24003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:15.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.442+0000 7f8c3f400640 1 -- 172.21.15.69:0/3457644094 reap_dead start 2024-04-24T08:20:15.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.442+0000 7f8c3f400640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c2400b9c0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f8c400ca9a0 tx=0x7f8c24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06bff3302d24405 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:15.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.442+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c38001a70 con 0x7f8c24005210 2024-04-24T08:20:15.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.442+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3802ae10 con 0x7f8c24005210 2024-04-24T08:20:15.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.442+0000 7f8c3d600640 1 -- 172.21.15.69:0/3457644094 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c38003bb0 con 0x7f8c24005210 2024-04-24T08:20:15.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.443+0000 7f8c45e00640 1 -- 172.21.15.69:0/3457644094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 msgr2=0x7f8c2400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.443+0000 7f8c45e00640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c2400b9c0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f8c400ca9a0 tx=0x7f8c24005830 comp rx=0 tx=0).stop 2024-04-24T08:20:15.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.443+0000 7f8c45e00640 1 -- 172.21.15.69:0/3457644094 shutdown_connections 2024-04-24T08:20:15.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.443+0000 7f8c45e00640 1 --2- 172.21.15.69:0/3457644094 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c2400b9c0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.443+0000 7f8c45e00640 1 -- 172.21.15.69:0/3457644094 >> 172.21.15.69:0/3457644094 conn(0x7f8c4006be40 msgr2=0x7f8c4006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:15.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.443+0000 7f8c45e00640 1 -- 172.21.15.69:0/3457644094 shutdown_connections 2024-04-24T08:20:15.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.443+0000 7f8c45e00640 1 -- 172.21.15.69:0/3457644094 wait complete. 2024-04-24T08:20:15.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.445+0000 7f8c45e00640 1 Processor -- start 2024-04-24T08:20:15.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.445+0000 7f8c45e00640 1 -- start start 2024-04-24T08:20:15.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.446+0000 7f8c45e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c4014f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.446+0000 7f8c45e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4015bce0 0x7f8c401f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.446+0000 7f8c45e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c40156290 0x7f8c401581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.446+0000 7f8c45e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8c400b4cc0 con 0x7f8c24005210 2024-04-24T08:20:15.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.446+0000 7f8c45e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8c400ad670 con 0x7f8c4015bce0 2024-04-24T08:20:15.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.446+0000 7f8c45e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8c400b2640 con 0x7f8c40156290 2024-04-24T08:20:15.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.446+0000 7f8c3ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c4014f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.446+0000 7f8c3ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c4014f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54528/0 (socket says 172.21.15.69:54528) 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.446+0000 7f8c3ea00640 1 -- 172.21.15.69:0/844791075 learned_addr learned my addr 172.21.15.69:0/844791075 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c3e000640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4015bce0 0x7f8c401f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c3f400640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c40156290 0x7f8c401581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c3ea00640 1 -- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c40156290 msgr2=0x7f8c401581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c3ea00640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c40156290 0x7f8c401581e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c3ea00640 1 -- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4015bce0 msgr2=0x7f8c401f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c3ea00640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4015bce0 0x7f8c401f4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c3ea00640 1 -- 172.21.15.69:0/844791075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c400d0ad0 con 0x7f8c24005210 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c3e000640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4015bce0 0x7f8c401f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c3f400640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c40156290 0x7f8c401581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c3ea00640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c4014f1e0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f8c300108b0 tx=0x7f8c30010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c37400640 1 -- 172.21.15.69:0/844791075 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3000f040 con 0x7f8c24005210 2024-04-24T08:20:15.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c37400640 1 -- 172.21.15.69:0/844791075 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3000eb00 con 0x7f8c24005210 2024-04-24T08:20:15.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c37400640 1 -- 172.21.15.69:0/844791075 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3001c400 con 0x7f8c24005210 2024-04-24T08:20:15.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.447+0000 7f8c45e00640 1 -- 172.21.15.69:0/844791075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c400cf850 con 0x7f8c24005210 2024-04-24T08:20:15.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.448+0000 7f8c45e00640 1 -- 172.21.15.69:0/844791075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c400b2ba0 con 0x7f8c24005210 2024-04-24T08:20:15.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.449+0000 7f8c45e00640 1 -- 172.21.15.69:0/844791075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c400af9e0 con 0x7f8c24005210 2024-04-24T08:20:15.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.450+0000 7f8c37400640 1 -- 172.21.15.69:0/844791075 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8c3001c5a0 con 0x7f8c24005210 2024-04-24T08:20:15.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.451+0000 7f8c37400640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c2006c920 0x7f8c2006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.451+0000 7f8c37400640 1 -- 172.21.15.69:0/844791075 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f8c3008c1f0 con 0x7f8c24005210 2024-04-24T08:20:15.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.452+0000 7f8c3e000640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c2006c920 0x7f8c2006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.453+0000 7f8c3e000640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c2006c920 0x7f8c2006ede0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c001200 tx=0x7f8c2c001150 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:15.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.456+0000 7f8c37400640 1 -- 172.21.15.69:0/844791075 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c400af9e0 con 0x7f8c24005210 2024-04-24T08:20:15.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.730+0000 7f8c45e00640 1 -- 172.21.15.69:0/844791075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f8c40075dc0 con 0x7f8c24005210 2024-04-24T08:20:15.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.730+0000 7f8c37400640 1 -- 172.21.15.69:0/844791075 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1036 (secure 0 0 0) 0x7f8c30018030 con 0x7f8c24005210 2024-04-24T08:20:15.731 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:15.731 INFO:teuthology.orchestra.run.smithi069.stdout:{"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":740,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":109,"num_osds":8,"num_up_osds":8,"osd_up_since":1713946087,"num_in_osds":8,"osd_in_since":1713946078,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":267194368,"bytes_avail":772826918912,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":99,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"b","status":"up:active","gid":9288}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":34,"modified":"2024-04-24T08:19:50.922796+0000","services":{}},"progress_events":{}} 2024-04-24T08:20:15.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.732+0000 7f8c45e00640 1 -- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c2006c920 msgr2=0x7f8c2006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c2006c920 0x7f8c2006ede0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c001200 tx=0x7f8c2c001150 comp rx=0 tx=0).stop 2024-04-24T08:20:15.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 -- 172.21.15.69:0/844791075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 msgr2=0x7f8c4014f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c4014f1e0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f8c300108b0 tx=0x7f8c30010d70 comp rx=0 tx=0).stop 2024-04-24T08:20:15.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 -- 172.21.15.69:0/844791075 shutdown_connections 2024-04-24T08:20:15.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f8c2006c920 0x7f8c2006ede0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8c40156290 0x7f8c401581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8c4015bce0 0x7f8c401f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 --2- 172.21.15.69:0/844791075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c24005210 0x7f8c4014f1e0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 -- 172.21.15.69:0/844791075 >> 172.21.15.69:0/844791075 conn(0x7f8c4006be40 msgr2=0x7f8c4015adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:15.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 -- 172.21.15.69:0/844791075 shutdown_connections 2024-04-24T08:20:15.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.733+0000 7f8c45e00640 1 -- 172.21.15.69:0/844791075 wait complete. 2024-04-24T08:20:15.751 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-24T08:20:15.751 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:20:15.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.949+0000 7f312aa00640 1 Processor -- start 2024-04-24T08:20:15.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.949+0000 7f312aa00640 1 -- start start 2024-04-24T08:20:15.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.951+0000 7f312aa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31240d0300 0x7f312406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.951+0000 7f312aa00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31240cf3d0 con 0x7f3124145fe0 2024-04-24T08:20:15.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.951+0000 7f312aa00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31240cf6b0 con 0x7f3124145a90 2024-04-24T08:20:15.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.951+0000 7f312aa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f31240cf990 con 0x7f31240d0300 2024-04-24T08:20:15.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.951+0000 7f3128a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3124145fe0 0x7f3124160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:43272/0 (socket says 172.21.15.69:43272) 2024-04-24T08:20:15.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.951+0000 7f3128a00640 1 -- 172.21.15.69:0/3579197097 learned_addr learned my addr 172.21.15.69:0/3579197097 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:15.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123e00640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31240d0300 0x7f312406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3718791760 0 0) 0x7f31240cf3d0 con 0x7f3124145fe0 2024-04-24T08:20:15.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3108003520 con 0x7f3124145fe0 2024-04-24T08:20:15.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2850256385 0 0) 0x7f3108003520 con 0x7f3124145fe0 2024-04-24T08:20:15.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31240d0300 msgr2=0x7f312406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:15.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31240d0300 0x7f312406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 >> v1:172.21.15.121:6789/0 conn(0x7f3124145a90 legacy=0x7f31240d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31240cfc70 con 0x7f3124145fe0 2024-04-24T08:20:15.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f311c0019f0 con 0x7f3124145fe0 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 >> v1:172.21.15.69:6789/0 conn(0x7f3124145fe0 legacy=0x7f3124160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31240d0300 0x7f31080039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3108003f60 0x7f3108004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3108004d40 0x7f31080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31240cf990 con 0x7f3108004d40 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f310800baa0 con 0x7f3108003f60 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.952+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f310800bca0 con 0x7f31240d0300 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.953+0000 7f3128a00640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3108004d40 0x7f31080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.953+0000 7f311b400640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3108003f60 0x7f3108004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.953+0000 7f3128a00640 1 -- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31240d0300 msgr2=0x7f31080039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.953+0000 7f3128a00640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f31240d0300 0x7f31080039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.953+0000 7f3128a00640 1 -- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3108003f60 msgr2=0x7f3108004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:15.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.953+0000 7f3128a00640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3108003f60 0x7f3108004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.953+0000 7f3128a00640 1 -- 172.21.15.69:0/3579197097 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3108003520 con 0x7f3108004d40 2024-04-24T08:20:15.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.953+0000 7f311b400640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3108003f60 0x7f3108004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:15.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.954+0000 7f3128a00640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3108004d40 0x7f31080051c0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f311c00a860 tx=0x7f3108005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3292bf3336435690 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:15.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.954+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c00a480 con 0x7f3108004d40 2024-04-24T08:20:15.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.954+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f311c02f070 con 0x7f3108004d40 2024-04-24T08:20:15.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.954+0000 7f3123400640 1 -- 172.21.15.69:0/3579197097 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c02bd40 con 0x7f3108004d40 2024-04-24T08:20:15.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.955+0000 7f312aa00640 1 -- 172.21.15.69:0/3579197097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3108004d40 msgr2=0x7f31080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.955+0000 7f312aa00640 1 --2- 172.21.15.69:0/3579197097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3108004d40 0x7f31080051c0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f311c00a860 tx=0x7f3108005830 comp rx=0 tx=0).stop 2024-04-24T08:20:15.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.955+0000 7f3128a00640 1 -- 172.21.15.69:0/3579197097 reap_dead start 2024-04-24T08:20:15.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.955+0000 7f312aa00640 1 -- 172.21.15.69:0/3579197097 shutdown_connections 2024-04-24T08:20:15.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.955+0000 7f312aa00640 1 -- 172.21.15.69:0/3579197097 >> 172.21.15.69:0/3579197097 conn(0x7f3124067d90 msgr2=0x7f312406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:15.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.955+0000 7f312aa00640 1 -- 172.21.15.69:0/3579197097 shutdown_connections 2024-04-24T08:20:15.960 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.955+0000 7f312aa00640 1 -- 172.21.15.69:0/3579197097 wait complete. 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.957+0000 7f312aa00640 1 Processor -- start 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.957+0000 7f312aa00640 1 -- start start 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.958+0000 7f312aa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3108004d40 0x7f31241f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.958+0000 7f312aa00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31240ab1b0 0x7f31241f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.958+0000 7f312aa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31241f2a40 0x7f312415ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.958+0000 7f312aa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31241458f0 con 0x7f31241f2a40 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.958+0000 7f312aa00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3124145770 con 0x7f31240ab1b0 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.958+0000 7f312aa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f31240cdc30 con 0x7f3108004d40 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.958+0000 7f3128a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31241f2a40 0x7f312415ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.958+0000 7f3128a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31241f2a40 0x7f312415ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54552/0 (socket says 172.21.15.69:54552) 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.958+0000 7f3128a00640 1 -- 172.21.15.69:0/2886503128 learned_addr learned my addr 172.21.15.69:0/2886503128 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:15.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f3128a00640 1 -- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3108004d40 msgr2=0x7f31241f33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f3128a00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3108004d40 0x7f31241f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f3128a00640 1 -- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31240ab1b0 msgr2=0x7f31241f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f311b400640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31240ab1b0 0x7f31241f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f3128a00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31240ab1b0 0x7f31241f1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f3128a00640 1 -- 172.21.15.69:0/2886503128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3124132810 con 0x7f31241f2a40 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f3128a00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31241f2a40 0x7f312415ecc0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f311c00a980 tx=0x7f31240cf5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f3120c00640 1 -- 172.21.15.69:0/2886503128 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c00a480 con 0x7f31241f2a40 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f3120c00640 1 -- 172.21.15.69:0/2886503128 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f311c004b60 con 0x7f31241f2a40 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f3120c00640 1 -- 172.21.15.69:0/2886503128 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c034470 con 0x7f31241f2a40 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.959+0000 7f312aa00640 1 -- 172.21.15.69:0/2886503128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3124132af0 con 0x7f31241f2a40 2024-04-24T08:20:15.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.960+0000 7f312aa00640 1 -- 172.21.15.69:0/2886503128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31241330b0 con 0x7f31241f2a40 2024-04-24T08:20:15.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.961+0000 7f312aa00640 1 -- 172.21.15.69:0/2886503128 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3124068a80 con 0x7f31241f2a40 2024-04-24T08:20:15.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.962+0000 7f3120c00640 1 -- 172.21.15.69:0/2886503128 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f311c034610 con 0x7f31241f2a40 2024-04-24T08:20:15.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.963+0000 7f3120c00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f310006c8f0 0x7f310006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:15.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.963+0000 7f3120c00640 1 -- 172.21.15.69:0/2886503128 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f311c0ab830 con 0x7f31241f2a40 2024-04-24T08:20:15.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.963+0000 7f3123e00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f310006c8f0 0x7f310006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:15.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.964+0000 7f3123e00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f310006c8f0 0x7f310006edb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3114009a40 tx=0x7f3114009500 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:15.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:15.968+0000 7f3120c00640 1 -- 172.21.15.69:0/2886503128 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f311c047030 con 0x7f31241f2a40 2024-04-24T08:20:16.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.163+0000 7f312aa00640 1 -- 172.21.15.69:0/2886503128 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f312406eeb0 con 0x7f310006c8f0 2024-04-24T08:20:16.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.181+0000 7f3120c00640 1 -- 172.21.15.69:0/2886503128 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252483 (secure 0 0 0) 0x7f312406eeb0 con 0x7f310006c8f0 2024-04-24T08:20:16.185 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:16.187 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:20:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.186+0000 7f312aa00640 1 -- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f310006c8f0 msgr2=0x7f310006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.186+0000 7f312aa00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f310006c8f0 0x7f310006edb0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3114009a40 tx=0x7f3114009500 comp rx=0 tx=0).stop 2024-04-24T08:20:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.186+0000 7f312aa00640 1 -- 172.21.15.69:0/2886503128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31241f2a40 msgr2=0x7f312415ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.186+0000 7f312aa00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31241f2a40 0x7f312415ecc0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f311c00a980 tx=0x7f31240cf5a0 comp rx=0 tx=0).stop 2024-04-24T08:20:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.186+0000 7f312aa00640 1 -- 172.21.15.69:0/2886503128 shutdown_connections 2024-04-24T08:20:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.186+0000 7f312aa00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f310006c8f0 0x7f310006edb0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.186+0000 7f312aa00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31241f2a40 0x7f312415ecc0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.186+0000 7f312aa00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31240ab1b0 0x7f31241f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.186+0000 7f312aa00640 1 --2- 172.21.15.69:0/2886503128 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3108004d40 0x7f31241f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:16.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.186+0000 7f312aa00640 1 -- 172.21.15.69:0/2886503128 >> 172.21.15.69:0/2886503128 conn(0x7f3124067d90 msgr2=0x7f31240aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:16.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.187+0000 7f312aa00640 1 -- 172.21.15.69:0/2886503128 shutdown_connections 2024-04-24T08:20:16.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:16.187+0000 7f312aa00640 1 -- 172.21.15.69:0/2886503128 wait complete. 2024-04-24T08:20:16.205 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-04-24T08:20:14.931006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261288,"kb_used_data":11816,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713432,"statfs":{"total":773094113280,"available":772826554368,"internally_reserved":0,"allocated":12099584,"data_stored":5897852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.197126"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896427+0000","last_change":"2024-04-24T08:20:09.799542+0000","last_active":"2024-04-24T08:20:10.896427+0000","last_peered":"2024-04-24T08:20:10.896427+0000","last_clean":"2024-04-24T08:20:10.896427+0000","last_became_active":"2024-04-24T08:20:09.799243+0000","last_became_peered":"2024-04-24T08:20:09.799243+0000","last_unstale":"2024-04-24T08:20:10.896427+0000","last_undegraded":"2024-04-24T08:20:10.896427+0000","last_fullsized":"2024-04-24T08:20:10.896427+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:19:43.824329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791360+0000","last_change":"2024-04-24T08:20:09.794708+0000","last_active":"2024-04-24T08:20:10.791360+0000","last_peered":"2024-04-24T08:20:10.791360+0000","last_clean":"2024-04-24T08:20:10.791360+0000","last_became_active":"2024-04-24T08:20:09.793835+0000","last_became_peered":"2024-04-24T08:20:09.793835+0000","last_unstale":"2024-04-24T08:20:10.791360+0000","last_undegraded":"2024-04-24T08:20:10.791360+0000","last_fullsized":"2024-04-24T08:20:10.791360+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:13.818493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897294+0000","last_change":"2024-04-24T08:20:09.788114+0000","last_active":"2024-04-24T08:20:10.897294+0000","last_peered":"2024-04-24T08:20:10.897294+0000","last_clean":"2024-04-24T08:20:10.897294+0000","last_became_active":"2024-04-24T08:20:09.787809+0000","last_became_peered":"2024-04-24T08:20:09.787809+0000","last_unstale":"2024-04-24T08:20:10.897294+0000","last_undegraded":"2024-04-24T08:20:10.897294+0000","last_fullsized":"2024-04-24T08:20:10.897294+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:26:31.959658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895708+0000","last_change":"2024-04-24T08:20:09.794858+0000","last_active":"2024-04-24T08:20:10.895708+0000","last_peered":"2024-04-24T08:20:10.895708+0000","last_clean":"2024-04-24T08:20:10.895708+0000","last_became_active":"2024-04-24T08:20:09.794047+0000","last_became_peered":"2024-04-24T08:20:09.794047+0000","last_unstale":"2024-04-24T08:20:10.895708+0000","last_undegraded":"2024-04-24T08:20:10.895708+0000","last_fullsized":"2024-04-24T08:20:10.895708+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:52:00.465750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788117+0000","last_change":"2024-04-24T08:20:09.780027+0000","last_active":"2024-04-24T08:20:10.788117+0000","last_peered":"2024-04-24T08:20:10.788117+0000","last_clean":"2024-04-24T08:20:10.788117+0000","last_became_active":"2024-04-24T08:20:09.779744+0000","last_became_peered":"2024-04-24T08:20:09.779744+0000","last_unstale":"2024-04-24T08:20:10.788117+0000","last_undegraded":"2024-04-24T08:20:10.788117+0000","last_fullsized":"2024-04-24T08:20:10.788117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:09.432747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897475+0000","last_change":"2024-04-24T08:20:09.784098+0000","last_active":"2024-04-24T08:20:10.897475+0000","last_peered":"2024-04-24T08:20:10.897475+0000","last_clean":"2024-04-24T08:20:10.897475+0000","last_became_active":"2024-04-24T08:20:09.783835+0000","last_became_peered":"2024-04-24T08:20:09.783835+0000","last_unstale":"2024-04-24T08:20:10.897475+0000","last_undegraded":"2024-04-24T08:20:10.897475+0000","last_fullsized":"2024-04-24T08:20:10.897475+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:56.817425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789432+0000","last_change":"2024-04-24T08:20:09.785141+0000","last_active":"2024-04-24T08:20:10.789432+0000","last_peered":"2024-04-24T08:20:10.789432+0000","last_clean":"2024-04-24T08:20:10.789432+0000","last_became_active":"2024-04-24T08:20:09.784585+0000","last_became_peered":"2024-04-24T08:20:09.784585+0000","last_unstale":"2024-04-24T08:20:10.789432+0000","last_undegraded":"2024-04-24T08:20:10.789432+0000","last_fullsized":"2024-04-24T08:20:10.789432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:44:55.266192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896771+0000","last_change":"2024-04-24T08:20:09.792355+0000","last_active":"2024-04-24T08:20:10.896771+0000","last_peered":"2024-04-24T08:20:10.896771+0000","last_clean":"2024-04-24T08:20:10.896771+0000","last_became_active":"2024-04-24T08:20:09.791438+0000","last_became_peered":"2024-04-24T08:20:09.791438+0000","last_unstale":"2024-04-24T08:20:10.896771+0000","last_undegraded":"2024-04-24T08:20:10.896771+0000","last_fullsized":"2024-04-24T08:20:10.896771+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:59:41.162621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897198+0000","last_change":"2024-04-24T08:20:09.803001+0000","last_active":"2024-04-24T08:20:10.897198+0000","last_peered":"2024-04-24T08:20:10.897198+0000","last_clean":"2024-04-24T08:20:10.897198+0000","last_became_active":"2024-04-24T08:20:09.801984+0000","last_became_peered":"2024-04-24T08:20:09.801984+0000","last_unstale":"2024-04-24T08:20:10.897198+0000","last_undegraded":"2024-04-24T08:20:10.897198+0000","last_fullsized":"2024-04-24T08:20:10.897198+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:51:22.573683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896823+0000","last_change":"2024-04-24T08:20:09.801533+0000","last_active":"2024-04-24T08:20:10.896823+0000","last_peered":"2024-04-24T08:20:10.896823+0000","last_clean":"2024-04-24T08:20:10.896823+0000","last_became_active":"2024-04-24T08:20:09.801142+0000","last_became_peered":"2024-04-24T08:20:09.801142+0000","last_unstale":"2024-04-24T08:20:10.896823+0000","last_undegraded":"2024-04-24T08:20:10.896823+0000","last_fullsized":"2024-04-24T08:20:10.896823+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:05.572616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791049+0000","last_change":"2024-04-24T08:20:09.791103+0000","last_active":"2024-04-24T08:20:10.791049+0000","last_peered":"2024-04-24T08:20:10.791049+0000","last_clean":"2024-04-24T08:20:10.791049+0000","last_became_active":"2024-04-24T08:20:09.787354+0000","last_became_peered":"2024-04-24T08:20:09.787354+0000","last_unstale":"2024-04-24T08:20:10.791049+0000","last_undegraded":"2024-04-24T08:20:10.791049+0000","last_fullsized":"2024-04-24T08:20:10.791049+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:48.090822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896003+0000","last_change":"2024-04-24T08:20:09.802741+0000","last_active":"2024-04-24T08:20:10.896003+0000","last_peered":"2024-04-24T08:20:10.896003+0000","last_clean":"2024-04-24T08:20:10.896003+0000","last_became_active":"2024-04-24T08:20:09.802402+0000","last_became_peered":"2024-04-24T08:20:09.802402+0000","last_unstale":"2024-04-24T08:20:10.896003+0000","last_undegraded":"2024-04-24T08:20:10.896003+0000","last_fullsized":"2024-04-24T08:20:10.896003+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:52.645695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897895+0000","last_change":"2024-04-24T08:20:09.800630+0000","last_active":"2024-04-24T08:20:10.897895+0000","last_peered":"2024-04-24T08:20:10.897895+0000","last_clean":"2024-04-24T08:20:10.897895+0000","last_became_active":"2024-04-24T08:20:09.800140+0000","last_became_peered":"2024-04-24T08:20:09.800140+0000","last_unstale":"2024-04-24T08:20:10.897895+0000","last_undegraded":"2024-04-24T08:20:10.897895+0000","last_fullsized":"2024-04-24T08:20:10.897895+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:57.166226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896796+0000","last_change":"2024-04-24T08:20:09.789152+0000","last_active":"2024-04-24T08:20:10.896796+0000","last_peered":"2024-04-24T08:20:10.896796+0000","last_clean":"2024-04-24T08:20:10.896796+0000","last_became_active":"2024-04-24T08:20:09.788646+0000","last_became_peered":"2024-04-24T08:20:09.788646+0000","last_unstale":"2024-04-24T08:20:10.896796+0000","last_undegraded":"2024-04-24T08:20:10.896796+0000","last_fullsized":"2024-04-24T08:20:10.896796+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:10:50.551575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897147+0000","last_change":"2024-04-24T08:20:09.782866+0000","last_active":"2024-04-24T08:20:10.897147+0000","last_peered":"2024-04-24T08:20:10.897147+0000","last_clean":"2024-04-24T08:20:10.897147+0000","last_became_active":"2024-04-24T08:20:09.781963+0000","last_became_peered":"2024-04-24T08:20:09.781963+0000","last_unstale":"2024-04-24T08:20:10.897147+0000","last_undegraded":"2024-04-24T08:20:10.897147+0000","last_fullsized":"2024-04-24T08:20:10.897147+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:42:59.649790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.894889+0000","last_change":"2024-04-24T08:20:09.799141+0000","last_active":"2024-04-24T08:20:10.894889+0000","last_peered":"2024-04-24T08:20:10.894889+0000","last_clean":"2024-04-24T08:20:10.894889+0000","last_became_active":"2024-04-24T08:20:09.798679+0000","last_became_peered":"2024-04-24T08:20:09.798679+0000","last_unstale":"2024-04-24T08:20:10.894889+0000","last_undegraded":"2024-04-24T08:20:10.894889+0000","last_fullsized":"2024-04-24T08:20:10.894889+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:06:29.713888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791716+0000","last_change":"2024-04-24T08:20:09.789678+0000","last_active":"2024-04-24T08:20:10.791716+0000","last_peered":"2024-04-24T08:20:10.791716+0000","last_clean":"2024-04-24T08:20:10.791716+0000","last_became_active":"2024-04-24T08:20:09.787273+0000","last_became_peered":"2024-04-24T08:20:09.787273+0000","last_unstale":"2024-04-24T08:20:10.791716+0000","last_undegraded":"2024-04-24T08:20:10.791716+0000","last_fullsized":"2024-04-24T08:20:10.791716+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:29.553044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898926+0000","last_change":"2024-04-24T08:20:09.785296+0000","last_active":"2024-04-24T08:20:10.898926+0000","last_peered":"2024-04-24T08:20:10.898926+0000","last_clean":"2024-04-24T08:20:10.898926+0000","last_became_active":"2024-04-24T08:20:09.784996+0000","last_became_peered":"2024-04-24T08:20:09.784996+0000","last_unstale":"2024-04-24T08:20:10.898926+0000","last_undegraded":"2024-04-24T08:20:10.898926+0000","last_fullsized":"2024-04-24T08:20:10.898926+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:09:43.173248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897367+0000","last_change":"2024-04-24T08:20:09.797840+0000","last_active":"2024-04-24T08:20:10.897367+0000","last_peered":"2024-04-24T08:20:10.897367+0000","last_clean":"2024-04-24T08:20:10.897367+0000","last_became_active":"2024-04-24T08:20:09.797575+0000","last_became_peered":"2024-04-24T08:20:09.797575+0000","last_unstale":"2024-04-24T08:20:10.897367+0000","last_undegraded":"2024-04-24T08:20:10.897367+0000","last_fullsized":"2024-04-24T08:20:10.897367+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:35.871237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791599+0000","last_change":"2024-04-24T08:20:09.789338+0000","last_active":"2024-04-24T08:20:10.791599+0000","last_peered":"2024-04-24T08:20:10.791599+0000","last_clean":"2024-04-24T08:20:10.791599+0000","last_became_active":"2024-04-24T08:20:09.787008+0000","last_became_peered":"2024-04-24T08:20:09.787008+0000","last_unstale":"2024-04-24T08:20:10.791599+0000","last_undegraded":"2024-04-24T08:20:10.791599+0000","last_fullsized":"2024-04-24T08:20:10.791599+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:03.358776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791614+0000","last_change":"2024-04-24T08:20:09.785546+0000","last_active":"2024-04-24T08:20:10.791614+0000","last_peered":"2024-04-24T08:20:10.791614+0000","last_clean":"2024-04-24T08:20:10.791614+0000","last_became_active":"2024-04-24T08:20:09.785285+0000","last_became_peered":"2024-04-24T08:20:09.785285+0000","last_unstale":"2024-04-24T08:20:10.791614+0000","last_undegraded":"2024-04-24T08:20:10.791614+0000","last_fullsized":"2024-04-24T08:20:10.791614+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:47.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899951+0000","last_change":"2024-04-24T08:20:09.792270+0000","last_active":"2024-04-24T08:20:10.899951+0000","last_peered":"2024-04-24T08:20:10.899951+0000","last_clean":"2024-04-24T08:20:10.899951+0000","last_became_active":"2024-04-24T08:20:09.791977+0000","last_became_peered":"2024-04-24T08:20:09.791977+0000","last_unstale":"2024-04-24T08:20:10.899951+0000","last_undegraded":"2024-04-24T08:20:10.899951+0000","last_fullsized":"2024-04-24T08:20:10.899951+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:47.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788610+0000","last_change":"2024-04-24T08:20:09.788141+0000","last_active":"2024-04-24T08:20:10.788610+0000","last_peered":"2024-04-24T08:20:10.788610+0000","last_clean":"2024-04-24T08:20:10.788610+0000","last_became_active":"2024-04-24T08:20:09.787507+0000","last_became_peered":"2024-04-24T08:20:09.787507+0000","last_unstale":"2024-04-24T08:20:10.788610+0000","last_undegraded":"2024-04-24T08:20:10.788610+0000","last_fullsized":"2024-04-24T08:20:10.788610+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:21.652074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897965+0000","last_change":"2024-04-24T08:20:09.787331+0000","last_active":"2024-04-24T08:20:10.897965+0000","last_peered":"2024-04-24T08:20:10.897965+0000","last_clean":"2024-04-24T08:20:10.897965+0000","last_became_active":"2024-04-24T08:20:09.786840+0000","last_became_peered":"2024-04-24T08:20:09.786840+0000","last_unstale":"2024-04-24T08:20:10.897965+0000","last_undegraded":"2024-04-24T08:20:10.897965+0000","last_fullsized":"2024-04-24T08:20:10.897965+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:47.526466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791717+0000","last_change":"2024-04-24T08:20:09.787529+0000","last_active":"2024-04-24T08:20:10.791717+0000","last_peered":"2024-04-24T08:20:10.791717+0000","last_clean":"2024-04-24T08:20:10.791717+0000","last_became_active":"2024-04-24T08:20:09.787236+0000","last_became_peered":"2024-04-24T08:20:09.787236+0000","last_unstale":"2024-04-24T08:20:10.791717+0000","last_undegraded":"2024-04-24T08:20:10.791717+0000","last_fullsized":"2024-04-24T08:20:10.791717+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:09:43.173248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896884+0000","last_change":"2024-04-24T08:20:09.788225+0000","last_active":"2024-04-24T08:20:10.896884+0000","last_peered":"2024-04-24T08:20:10.896884+0000","last_clean":"2024-04-24T08:20:10.896884+0000","last_became_active":"2024-04-24T08:20:09.787936+0000","last_became_peered":"2024-04-24T08:20:09.787936+0000","last_unstale":"2024-04-24T08:20:10.896884+0000","last_undegraded":"2024-04-24T08:20:10.896884+0000","last_fullsized":"2024-04-24T08:20:10.896884+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:20:18.482020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899461+0000","last_change":"2024-04-24T08:20:09.794928+0000","last_active":"2024-04-24T08:20:10.899461+0000","last_peered":"2024-04-24T08:20:10.899461+0000","last_clean":"2024-04-24T08:20:10.899461+0000","last_became_active":"2024-04-24T08:20:09.794289+0000","last_became_peered":"2024-04-24T08:20:09.794289+0000","last_unstale":"2024-04-24T08:20:10.899461+0000","last_undegraded":"2024-04-24T08:20:10.899461+0000","last_fullsized":"2024-04-24T08:20:10.899461+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:51:07.962266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898786+0000","last_change":"2024-04-24T08:20:09.794776+0000","last_active":"2024-04-24T08:20:10.898786+0000","last_peered":"2024-04-24T08:20:10.898786+0000","last_clean":"2024-04-24T08:20:10.898786+0000","last_became_active":"2024-04-24T08:20:09.794151+0000","last_became_peered":"2024-04-24T08:20:09.794151+0000","last_unstale":"2024-04-24T08:20:10.898786+0000","last_undegraded":"2024-04-24T08:20:10.898786+0000","last_fullsized":"2024-04-24T08:20:10.898786+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:03.358776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790533+0000","last_change":"2024-04-24T08:20:09.791933+0000","last_active":"2024-04-24T08:20:10.790533+0000","last_peered":"2024-04-24T08:20:10.790533+0000","last_clean":"2024-04-24T08:20:10.790533+0000","last_became_active":"2024-04-24T08:20:09.789144+0000","last_became_peered":"2024-04-24T08:20:09.789144+0000","last_unstale":"2024-04-24T08:20:10.790533+0000","last_undegraded":"2024-04-24T08:20:10.790533+0000","last_fullsized":"2024-04-24T08:20:10.790533+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:57.166226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789744+0000","last_change":"2024-04-24T08:20:09.779514+0000","last_active":"2024-04-24T08:20:10.789744+0000","last_peered":"2024-04-24T08:20:10.789744+0000","last_clean":"2024-04-24T08:20:10.789744+0000","last_became_active":"2024-04-24T08:20:09.779002+0000","last_became_peered":"2024-04-24T08:20:09.779002+0000","last_unstale":"2024-04-24T08:20:10.789744+0000","last_undegraded":"2024-04-24T08:20:10.789744+0000","last_fullsized":"2024-04-24T08:20:10.789744+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:42:59.649790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895194+0000","last_change":"2024-04-24T08:20:09.789034+0000","last_active":"2024-04-24T08:20:10.895194+0000","last_peered":"2024-04-24T08:20:10.895194+0000","last_clean":"2024-04-24T08:20:10.895194+0000","last_became_active":"2024-04-24T08:20:09.788764+0000","last_became_peered":"2024-04-24T08:20:09.788764+0000","last_unstale":"2024-04-24T08:20:10.895194+0000","last_undegraded":"2024-04-24T08:20:10.895194+0000","last_fullsized":"2024-04-24T08:20:10.895194+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:29.411232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896727+0000","last_change":"2024-04-24T08:20:09.796857+0000","last_active":"2024-04-24T08:20:10.896727+0000","last_peered":"2024-04-24T08:20:10.896727+0000","last_clean":"2024-04-24T08:20:10.896727+0000","last_became_active":"2024-04-24T08:20:09.795564+0000","last_became_peered":"2024-04-24T08:20:09.795564+0000","last_unstale":"2024-04-24T08:20:10.896727+0000","last_undegraded":"2024-04-24T08:20:10.896727+0000","last_fullsized":"2024-04-24T08:20:10.896727+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:44:55.266192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899863+0000","last_change":"2024-04-24T08:20:09.779341+0000","last_active":"2024-04-24T08:20:10.899863+0000","last_peered":"2024-04-24T08:20:10.899863+0000","last_clean":"2024-04-24T08:20:10.899863+0000","last_became_active":"2024-04-24T08:20:09.779056+0000","last_became_peered":"2024-04-24T08:20:09.779056+0000","last_unstale":"2024-04-24T08:20:10.899863+0000","last_undegraded":"2024-04-24T08:20:10.899863+0000","last_fullsized":"2024-04-24T08:20:10.899863+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:34.919462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"108'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790412+0000","last_change":"2024-04-24T08:20:08.765030+0000","last_active":"2024-04-24T08:20:10.790412+0000","last_peered":"2024-04-24T08:20:10.790412+0000","last_clean":"2024-04-24T08:20:10.790412+0000","last_became_active":"2024-04-24T08:20:08.764730+0000","last_became_peered":"2024-04-24T08:20:08.764730+0000","last_unstale":"2024-04-24T08:20:10.790412+0000","last_undegraded":"2024-04-24T08:20:10.790412+0000","last_fullsized":"2024-04-24T08:20:10.790412+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:19:40.821738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898429+0000","last_change":"2024-04-24T08:20:09.793872+0000","last_active":"2024-04-24T08:20:10.898429+0000","last_peered":"2024-04-24T08:20:10.898429+0000","last_clean":"2024-04-24T08:20:10.898429+0000","last_became_active":"2024-04-24T08:20:09.793548+0000","last_became_peered":"2024-04-24T08:20:09.793548+0000","last_unstale":"2024-04-24T08:20:10.898429+0000","last_undegraded":"2024-04-24T08:20:10.898429+0000","last_fullsized":"2024-04-24T08:20:10.898429+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:03.358776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898085+0000","last_change":"2024-04-24T08:20:08.764950+0000","last_active":"2024-04-24T08:20:10.898085+0000","last_peered":"2024-04-24T08:20:10.898085+0000","last_clean":"2024-04-24T08:20:10.898085+0000","last_became_active":"2024-04-24T08:20:08.764618+0000","last_became_peered":"2024-04-24T08:20:08.764618+0000","last_unstale":"2024-04-24T08:20:10.898085+0000","last_undegraded":"2024-04-24T08:20:10.898085+0000","last_fullsized":"2024-04-24T08:20:10.898085+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:02.260282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899378+0000","last_change":"2024-04-24T08:20:09.796732+0000","last_active":"2024-04-24T08:20:10.899378+0000","last_peered":"2024-04-24T08:20:10.899378+0000","last_clean":"2024-04-24T08:20:10.899378+0000","last_became_active":"2024-04-24T08:20:09.795416+0000","last_became_peered":"2024-04-24T08:20:09.795416+0000","last_unstale":"2024-04-24T08:20:10.899378+0000","last_undegraded":"2024-04-24T08:20:10.899378+0000","last_fullsized":"2024-04-24T08:20:10.899378+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:38.016468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787234+0000","last_change":"2024-04-24T08:20:08.772932+0000","last_active":"2024-04-24T08:20:10.787234+0000","last_peered":"2024-04-24T08:20:10.787234+0000","last_clean":"2024-04-24T08:20:10.787234+0000","last_became_active":"2024-04-24T08:20:08.772234+0000","last_became_peered":"2024-04-24T08:20:08.772234+0000","last_unstale":"2024-04-24T08:20:10.787234+0000","last_undegraded":"2024-04-24T08:20:10.787234+0000","last_fullsized":"2024-04-24T08:20:10.787234+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:12:45.289874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897735+0000","last_change":"2024-04-24T08:20:09.797817+0000","last_active":"2024-04-24T08:20:10.897735+0000","last_peered":"2024-04-24T08:20:10.897735+0000","last_clean":"2024-04-24T08:20:10.897735+0000","last_became_active":"2024-04-24T08:20:09.797517+0000","last_became_peered":"2024-04-24T08:20:09.797517+0000","last_unstale":"2024-04-24T08:20:10.897735+0000","last_undegraded":"2024-04-24T08:20:10.897735+0000","last_fullsized":"2024-04-24T08:20:10.897735+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:21.652074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787417+0000","last_change":"2024-04-24T08:20:08.770132+0000","last_active":"2024-04-24T08:20:10.787417+0000","last_peered":"2024-04-24T08:20:10.787417+0000","last_clean":"2024-04-24T08:20:10.787417+0000","last_became_active":"2024-04-24T08:20:08.769756+0000","last_became_peered":"2024-04-24T08:20:08.769756+0000","last_unstale":"2024-04-24T08:20:10.787417+0000","last_undegraded":"2024-04-24T08:20:10.787417+0000","last_fullsized":"2024-04-24T08:20:10.787417+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:57:23.545844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896040+0000","last_change":"2024-04-24T08:20:09.804520+0000","last_active":"2024-04-24T08:20:10.896040+0000","last_peered":"2024-04-24T08:20:10.896040+0000","last_clean":"2024-04-24T08:20:10.896040+0000","last_became_active":"2024-04-24T08:20:09.804216+0000","last_became_peered":"2024-04-24T08:20:09.804216+0000","last_unstale":"2024-04-24T08:20:10.896040+0000","last_undegraded":"2024-04-24T08:20:10.896040+0000","last_fullsized":"2024-04-24T08:20:10.896040+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:45:39.680748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789936+0000","last_change":"2024-04-24T08:20:08.763740+0000","last_active":"2024-04-24T08:20:10.789936+0000","last_peered":"2024-04-24T08:20:10.789936+0000","last_clean":"2024-04-24T08:20:10.789936+0000","last_became_active":"2024-04-24T08:20:08.763472+0000","last_became_peered":"2024-04-24T08:20:08.763472+0000","last_unstale":"2024-04-24T08:20:10.789936+0000","last_undegraded":"2024-04-24T08:20:10.789936+0000","last_fullsized":"2024-04-24T08:20:10.789936+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:13:02.412872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790984+0000","last_change":"2024-04-24T08:20:09.796520+0000","last_active":"2024-04-24T08:20:10.790984+0000","last_peered":"2024-04-24T08:20:10.790984+0000","last_clean":"2024-04-24T08:20:10.790984+0000","last_became_active":"2024-04-24T08:20:09.793445+0000","last_became_peered":"2024-04-24T08:20:09.793445+0000","last_unstale":"2024-04-24T08:20:10.790984+0000","last_undegraded":"2024-04-24T08:20:10.790984+0000","last_fullsized":"2024-04-24T08:20:10.790984+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:34.067483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.900109+0000","last_change":"2024-04-24T08:20:08.762014+0000","last_active":"2024-04-24T08:20:10.900109+0000","last_peered":"2024-04-24T08:20:10.900109+0000","last_clean":"2024-04-24T08:20:10.900109+0000","last_became_active":"2024-04-24T08:20:08.761675+0000","last_became_peered":"2024-04-24T08:20:08.761675+0000","last_unstale":"2024-04-24T08:20:10.900109+0000","last_undegraded":"2024-04-24T08:20:10.900109+0000","last_fullsized":"2024-04-24T08:20:10.900109+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:23.173525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895119+0000","last_change":"2024-04-24T08:20:09.795885+0000","last_active":"2024-04-24T08:20:10.895119+0000","last_peered":"2024-04-24T08:20:10.895119+0000","last_clean":"2024-04-24T08:20:10.895119+0000","last_became_active":"2024-04-24T08:20:09.795631+0000","last_became_peered":"2024-04-24T08:20:09.795631+0000","last_unstale":"2024-04-24T08:20:10.895119+0000","last_undegraded":"2024-04-24T08:20:10.895119+0000","last_fullsized":"2024-04-24T08:20:10.895119+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:19:33.323423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788665+0000","last_change":"2024-04-24T08:20:08.755417+0000","last_active":"2024-04-24T08:20:10.788665+0000","last_peered":"2024-04-24T08:20:10.788665+0000","last_clean":"2024-04-24T08:20:10.788665+0000","last_became_active":"2024-04-24T08:20:08.755131+0000","last_became_peered":"2024-04-24T08:20:08.755131+0000","last_unstale":"2024-04-24T08:20:10.788665+0000","last_undegraded":"2024-04-24T08:20:10.788665+0000","last_fullsized":"2024-04-24T08:20:10.788665+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:23.173525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789296+0000","last_change":"2024-04-24T08:20:09.785134+0000","last_active":"2024-04-24T08:20:10.789296+0000","last_peered":"2024-04-24T08:20:10.789296+0000","last_clean":"2024-04-24T08:20:10.789296+0000","last_became_active":"2024-04-24T08:20:09.784131+0000","last_became_peered":"2024-04-24T08:20:09.784131+0000","last_unstale":"2024-04-24T08:20:10.789296+0000","last_undegraded":"2024-04-24T08:20:10.789296+0000","last_fullsized":"2024-04-24T08:20:10.789296+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:16:12.799342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895001+0000","last_change":"2024-04-24T08:20:08.764423+0000","last_active":"2024-04-24T08:20:10.895001+0000","last_peered":"2024-04-24T08:20:10.895001+0000","last_clean":"2024-04-24T08:20:10.895001+0000","last_became_active":"2024-04-24T08:20:08.764017+0000","last_became_peered":"2024-04-24T08:20:08.764017+0000","last_unstale":"2024-04-24T08:20:10.895001+0000","last_undegraded":"2024-04-24T08:20:10.895001+0000","last_fullsized":"2024-04-24T08:20:10.895001+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:47.073639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790527+0000","last_change":"2024-04-24T08:20:09.775184+0000","last_active":"2024-04-24T08:20:10.790527+0000","last_peered":"2024-04-24T08:20:10.790527+0000","last_clean":"2024-04-24T08:20:10.790527+0000","last_became_active":"2024-04-24T08:20:09.774870+0000","last_became_peered":"2024-04-24T08:20:09.774870+0000","last_unstale":"2024-04-24T08:20:10.790527+0000","last_undegraded":"2024-04-24T08:20:10.790527+0000","last_fullsized":"2024-04-24T08:20:10.790527+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:38.016468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787733+0000","last_change":"2024-04-24T08:20:08.768692+0000","last_active":"2024-04-24T08:20:10.787733+0000","last_peered":"2024-04-24T08:20:10.787733+0000","last_clean":"2024-04-24T08:20:10.787733+0000","last_became_active":"2024-04-24T08:20:08.768352+0000","last_became_peered":"2024-04-24T08:20:08.768352+0000","last_unstale":"2024-04-24T08:20:10.787733+0000","last_undegraded":"2024-04-24T08:20:10.787733+0000","last_fullsized":"2024-04-24T08:20:10.787733+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:52:34.264489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898831+0000","last_change":"2024-04-24T08:20:09.799946+0000","last_active":"2024-04-24T08:20:10.898831+0000","last_peered":"2024-04-24T08:20:10.898831+0000","last_clean":"2024-04-24T08:20:10.898831+0000","last_became_active":"2024-04-24T08:20:09.799680+0000","last_became_peered":"2024-04-24T08:20:09.799680+0000","last_unstale":"2024-04-24T08:20:10.898831+0000","last_undegraded":"2024-04-24T08:20:10.898831+0000","last_fullsized":"2024-04-24T08:20:10.898831+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:47.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899134+0000","last_change":"2024-04-24T08:20:08.767491+0000","last_active":"2024-04-24T08:20:10.899134+0000","last_peered":"2024-04-24T08:20:10.899134+0000","last_clean":"2024-04-24T08:20:10.899134+0000","last_became_active":"2024-04-24T08:20:08.767200+0000","last_became_peered":"2024-04-24T08:20:08.767200+0000","last_unstale":"2024-04-24T08:20:10.899134+0000","last_undegraded":"2024-04-24T08:20:10.899134+0000","last_fullsized":"2024-04-24T08:20:10.899134+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:29:39.437886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898703+0000","last_change":"2024-04-24T08:20:09.784394+0000","last_active":"2024-04-24T08:20:10.898703+0000","last_peered":"2024-04-24T08:20:10.898703+0000","last_clean":"2024-04-24T08:20:10.898703+0000","last_became_active":"2024-04-24T08:20:09.784125+0000","last_became_peered":"2024-04-24T08:20:09.784125+0000","last_unstale":"2024-04-24T08:20:10.898703+0000","last_undegraded":"2024-04-24T08:20:10.898703+0000","last_fullsized":"2024-04-24T08:20:10.898703+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:13.818493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"108'3","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896283+0000","last_change":"2024-04-24T08:20:08.771750+0000","last_active":"2024-04-24T08:20:10.896283+0000","last_peered":"2024-04-24T08:20:10.896283+0000","last_clean":"2024-04-24T08:20:10.896283+0000","last_became_active":"2024-04-24T08:20:08.771450+0000","last_became_peered":"2024-04-24T08:20:08.771450+0000","last_unstale":"2024-04-24T08:20:10.896283+0000","last_undegraded":"2024-04-24T08:20:10.896283+0000","last_fullsized":"2024-04-24T08:20:10.896283+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:56.149044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790276+0000","last_change":"2024-04-24T08:20:09.784652+0000","last_active":"2024-04-24T08:20:10.790276+0000","last_peered":"2024-04-24T08:20:10.790276+0000","last_clean":"2024-04-24T08:20:10.790276+0000","last_became_active":"2024-04-24T08:20:09.784367+0000","last_became_peered":"2024-04-24T08:20:09.784367+0000","last_unstale":"2024-04-24T08:20:10.790276+0000","last_undegraded":"2024-04-24T08:20:10.790276+0000","last_fullsized":"2024-04-24T08:20:10.790276+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:56.817425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"108'6","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896458+0000","last_change":"2024-04-24T08:20:08.763054+0000","last_active":"2024-04-24T08:20:10.896458+0000","last_peered":"2024-04-24T08:20:10.896458+0000","last_clean":"2024-04-24T08:20:10.896458+0000","last_became_active":"2024-04-24T08:20:08.762785+0000","last_became_peered":"2024-04-24T08:20:08.762785+0000","last_unstale":"2024-04-24T08:20:10.896458+0000","last_undegraded":"2024-04-24T08:20:10.896458+0000","last_fullsized":"2024-04-24T08:20:10.896458+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:02.341594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.900015+0000","last_change":"2024-04-24T08:20:09.775131+0000","last_active":"2024-04-24T08:20:10.900015+0000","last_peered":"2024-04-24T08:20:10.900015+0000","last_clean":"2024-04-24T08:20:10.900015+0000","last_became_active":"2024-04-24T08:20:09.774820+0000","last_became_peered":"2024-04-24T08:20:09.774820+0000","last_unstale":"2024-04-24T08:20:10.900015+0000","last_undegraded":"2024-04-24T08:20:10.900015+0000","last_fullsized":"2024-04-24T08:20:10.900015+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:09:43.173248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898575+0000","last_change":"2024-04-24T08:20:08.771246+0000","last_active":"2024-04-24T08:20:10.898575+0000","last_peered":"2024-04-24T08:20:10.898575+0000","last_clean":"2024-04-24T08:20:10.898575+0000","last_became_active":"2024-04-24T08:20:08.770915+0000","last_became_peered":"2024-04-24T08:20:08.770915+0000","last_unstale":"2024-04-24T08:20:10.898575+0000","last_undegraded":"2024-04-24T08:20:10.898575+0000","last_fullsized":"2024-04-24T08:20:10.898575+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:19:40.821738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897970+0000","last_change":"2024-04-24T08:20:09.787926+0000","last_active":"2024-04-24T08:20:10.897970+0000","last_peered":"2024-04-24T08:20:10.897970+0000","last_clean":"2024-04-24T08:20:10.897970+0000","last_became_active":"2024-04-24T08:20:09.787610+0000","last_became_peered":"2024-04-24T08:20:09.787610+0000","last_unstale":"2024-04-24T08:20:10.897970+0000","last_undegraded":"2024-04-24T08:20:10.897970+0000","last_fullsized":"2024-04-24T08:20:10.897970+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:57.166226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898305+0000","last_change":"2024-04-24T08:20:08.770641+0000","last_active":"2024-04-24T08:20:10.898305+0000","last_peered":"2024-04-24T08:20:10.898305+0000","last_clean":"2024-04-24T08:20:10.898305+0000","last_became_active":"2024-04-24T08:20:08.770281+0000","last_became_peered":"2024-04-24T08:20:08.770281+0000","last_unstale":"2024-04-24T08:20:10.898305+0000","last_undegraded":"2024-04-24T08:20:10.898305+0000","last_fullsized":"2024-04-24T08:20:10.898305+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:34.854054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898155+0000","last_change":"2024-04-24T08:20:09.789809+0000","last_active":"2024-04-24T08:20:10.898155+0000","last_peered":"2024-04-24T08:20:10.898155+0000","last_clean":"2024-04-24T08:20:10.898155+0000","last_became_active":"2024-04-24T08:20:09.789248+0000","last_became_peered":"2024-04-24T08:20:09.789248+0000","last_unstale":"2024-04-24T08:20:10.898155+0000","last_undegraded":"2024-04-24T08:20:10.898155+0000","last_fullsized":"2024-04-24T08:20:10.898155+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:34.067483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"108'8","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.786954+0000","last_change":"2024-04-24T08:20:08.760398+0000","last_active":"2024-04-24T08:20:10.786954+0000","last_peered":"2024-04-24T08:20:10.786954+0000","last_clean":"2024-04-24T08:20:10.786954+0000","last_became_active":"2024-04-24T08:20:08.760075+0000","last_became_peered":"2024-04-24T08:20:08.760075+0000","last_unstale":"2024-04-24T08:20:10.786954+0000","last_undegraded":"2024-04-24T08:20:10.786954+0000","last_fullsized":"2024-04-24T08:20:10.786954+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:20.208658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788203+0000","last_change":"2024-04-24T08:20:09.797624+0000","last_active":"2024-04-24T08:20:10.788203+0000","last_peered":"2024-04-24T08:20:10.788203+0000","last_clean":"2024-04-24T08:20:10.788203+0000","last_became_active":"2024-04-24T08:20:09.796587+0000","last_became_peered":"2024-04-24T08:20:09.796587+0000","last_unstale":"2024-04-24T08:20:10.788203+0000","last_undegraded":"2024-04-24T08:20:10.788203+0000","last_fullsized":"2024-04-24T08:20:10.788203+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:10:52.836766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897475+0000","last_change":"2024-04-24T08:20:08.769979+0000","last_active":"2024-04-24T08:20:10.897475+0000","last_peered":"2024-04-24T08:20:10.897475+0000","last_clean":"2024-04-24T08:20:10.897475+0000","last_became_active":"2024-04-24T08:20:08.769667+0000","last_became_peered":"2024-04-24T08:20:08.769667+0000","last_unstale":"2024-04-24T08:20:10.897475+0000","last_undegraded":"2024-04-24T08:20:10.897475+0000","last_fullsized":"2024-04-24T08:20:10.897475+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:33.050301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899848+0000","last_change":"2024-04-24T08:20:09.795745+0000","last_active":"2024-04-24T08:20:10.899848+0000","last_peered":"2024-04-24T08:20:10.899848+0000","last_clean":"2024-04-24T08:20:10.899848+0000","last_became_active":"2024-04-24T08:20:09.795459+0000","last_became_peered":"2024-04-24T08:20:09.795459+0000","last_unstale":"2024-04-24T08:20:10.899848+0000","last_undegraded":"2024-04-24T08:20:10.899848+0000","last_fullsized":"2024-04-24T08:20:10.899848+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:07:38.079929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788129+0000","last_change":"2024-04-24T08:20:08.762510+0000","last_active":"2024-04-24T08:20:10.788129+0000","last_peered":"2024-04-24T08:20:10.788129+0000","last_clean":"2024-04-24T08:20:10.788129+0000","last_became_active":"2024-04-24T08:20:08.762168+0000","last_became_peered":"2024-04-24T08:20:08.762168+0000","last_unstale":"2024-04-24T08:20:10.788129+0000","last_undegraded":"2024-04-24T08:20:10.788129+0000","last_fullsized":"2024-04-24T08:20:10.788129+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:05:22.039965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898404+0000","last_change":"2024-04-24T08:20:09.800369+0000","last_active":"2024-04-24T08:20:10.898404+0000","last_peered":"2024-04-24T08:20:10.898404+0000","last_clean":"2024-04-24T08:20:10.898404+0000","last_became_active":"2024-04-24T08:20:09.799535+0000","last_became_peered":"2024-04-24T08:20:09.799535+0000","last_unstale":"2024-04-24T08:20:10.898404+0000","last_undegraded":"2024-04-24T08:20:10.898404+0000","last_fullsized":"2024-04-24T08:20:10.898404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:25.726242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898135+0000","last_change":"2024-04-24T08:20:08.761477+0000","last_active":"2024-04-24T08:20:10.898135+0000","last_peered":"2024-04-24T08:20:10.898135+0000","last_clean":"2024-04-24T08:20:10.898135+0000","last_became_active":"2024-04-24T08:20:08.759812+0000","last_became_peered":"2024-04-24T08:20:08.759812+0000","last_unstale":"2024-04-24T08:20:10.898135+0000","last_undegraded":"2024-04-24T08:20:10.898135+0000","last_fullsized":"2024-04-24T08:20:10.898135+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:27:26.888390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897806+0000","last_change":"2024-04-24T08:20:09.795821+0000","last_active":"2024-04-24T08:20:10.897806+0000","last_peered":"2024-04-24T08:20:10.897806+0000","last_clean":"2024-04-24T08:20:10.897806+0000","last_became_active":"2024-04-24T08:20:09.795409+0000","last_became_peered":"2024-04-24T08:20:09.795409+0000","last_unstale":"2024-04-24T08:20:10.897806+0000","last_undegraded":"2024-04-24T08:20:10.897806+0000","last_fullsized":"2024-04-24T08:20:10.897806+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:47.526466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790642+0000","last_change":"2024-04-24T08:20:08.767284+0000","last_active":"2024-04-24T08:20:10.790642+0000","last_peered":"2024-04-24T08:20:10.790642+0000","last_clean":"2024-04-24T08:20:10.790642+0000","last_became_active":"2024-04-24T08:20:08.766880+0000","last_became_peered":"2024-04-24T08:20:08.766880+0000","last_unstale":"2024-04-24T08:20:10.790642+0000","last_undegraded":"2024-04-24T08:20:10.790642+0000","last_fullsized":"2024-04-24T08:20:10.790642+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:08.415565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896838+0000","last_change":"2024-04-24T08:20:09.787635+0000","last_active":"2024-04-24T08:20:10.896838+0000","last_peered":"2024-04-24T08:20:10.896838+0000","last_clean":"2024-04-24T08:20:10.896838+0000","last_became_active":"2024-04-24T08:20:09.787270+0000","last_became_peered":"2024-04-24T08:20:09.787270+0000","last_unstale":"2024-04-24T08:20:10.896838+0000","last_undegraded":"2024-04-24T08:20:10.896838+0000","last_fullsized":"2024-04-24T08:20:10.896838+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:23:51.545972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899062+0000","last_change":"2024-04-24T08:20:08.773991+0000","last_active":"2024-04-24T08:20:10.899062+0000","last_peered":"2024-04-24T08:20:10.899062+0000","last_clean":"2024-04-24T08:20:10.899062+0000","last_became_active":"2024-04-24T08:20:08.773703+0000","last_became_peered":"2024-04-24T08:20:08.773703+0000","last_unstale":"2024-04-24T08:20:10.899062+0000","last_undegraded":"2024-04-24T08:20:10.899062+0000","last_fullsized":"2024-04-24T08:20:10.899062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:13:02.412872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895443+0000","last_change":"2024-04-24T08:20:09.797747+0000","last_active":"2024-04-24T08:20:10.895443+0000","last_peered":"2024-04-24T08:20:10.895443+0000","last_clean":"2024-04-24T08:20:10.895443+0000","last_became_active":"2024-04-24T08:20:09.797468+0000","last_became_peered":"2024-04-24T08:20:09.797468+0000","last_unstale":"2024-04-24T08:20:10.895443+0000","last_undegraded":"2024-04-24T08:20:10.895443+0000","last_fullsized":"2024-04-24T08:20:10.895443+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:10:12.534666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.786965+0000","last_change":"2024-04-24T08:20:08.769543+0000","last_active":"2024-04-24T08:20:10.786965+0000","last_peered":"2024-04-24T08:20:10.786965+0000","last_clean":"2024-04-24T08:20:10.786965+0000","last_became_active":"2024-04-24T08:20:08.767940+0000","last_became_peered":"2024-04-24T08:20:08.767940+0000","last_unstale":"2024-04-24T08:20:10.786965+0000","last_undegraded":"2024-04-24T08:20:10.786965+0000","last_fullsized":"2024-04-24T08:20:10.786965+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:16:28.019646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897242+0000","last_change":"2024-04-24T08:20:09.790241+0000","last_active":"2024-04-24T08:20:10.897242+0000","last_peered":"2024-04-24T08:20:10.897242+0000","last_clean":"2024-04-24T08:20:10.897242+0000","last_became_active":"2024-04-24T08:20:09.789926+0000","last_became_peered":"2024-04-24T08:20:09.789926+0000","last_unstale":"2024-04-24T08:20:10.897242+0000","last_undegraded":"2024-04-24T08:20:10.897242+0000","last_fullsized":"2024-04-24T08:20:10.897242+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:52:35.867500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897241+0000","last_change":"2024-04-24T08:20:08.769318+0000","last_active":"2024-04-24T08:20:10.897241+0000","last_peered":"2024-04-24T08:20:10.897241+0000","last_clean":"2024-04-24T08:20:10.897241+0000","last_became_active":"2024-04-24T08:20:08.769050+0000","last_became_peered":"2024-04-24T08:20:08.769050+0000","last_unstale":"2024-04-24T08:20:10.897241+0000","last_undegraded":"2024-04-24T08:20:10.897241+0000","last_fullsized":"2024-04-24T08:20:10.897241+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:10:51.819583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898432+0000","last_change":"2024-04-24T08:20:09.792723+0000","last_active":"2024-04-24T08:20:10.898432+0000","last_peered":"2024-04-24T08:20:10.898432+0000","last_clean":"2024-04-24T08:20:10.898432+0000","last_became_active":"2024-04-24T08:20:09.792405+0000","last_became_peered":"2024-04-24T08:20:09.792405+0000","last_unstale":"2024-04-24T08:20:10.898432+0000","last_undegraded":"2024-04-24T08:20:10.898432+0000","last_fullsized":"2024-04-24T08:20:10.898432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:48.090822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895133+0000","last_change":"2024-04-24T08:20:08.767252+0000","last_active":"2024-04-24T08:20:10.895133+0000","last_peered":"2024-04-24T08:20:10.895133+0000","last_clean":"2024-04-24T08:20:10.895133+0000","last_became_active":"2024-04-24T08:20:08.766235+0000","last_became_peered":"2024-04-24T08:20:08.766235+0000","last_unstale":"2024-04-24T08:20:10.895133+0000","last_undegraded":"2024-04-24T08:20:10.895133+0000","last_fullsized":"2024-04-24T08:20:10.895133+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:12.801310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899088+0000","last_change":"2024-04-24T08:20:09.799590+0000","last_active":"2024-04-24T08:20:10.899088+0000","last_peered":"2024-04-24T08:20:10.899088+0000","last_clean":"2024-04-24T08:20:10.899088+0000","last_became_active":"2024-04-24T08:20:09.799255+0000","last_became_peered":"2024-04-24T08:20:09.799255+0000","last_unstale":"2024-04-24T08:20:10.899088+0000","last_undegraded":"2024-04-24T08:20:10.899088+0000","last_fullsized":"2024-04-24T08:20:10.899088+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:29.553044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899509+0000","last_change":"2024-04-24T08:20:08.774854+0000","last_active":"2024-04-24T08:20:10.899509+0000","last_peered":"2024-04-24T08:20:10.899509+0000","last_clean":"2024-04-24T08:20:10.899509+0000","last_became_active":"2024-04-24T08:20:08.773896+0000","last_became_peered":"2024-04-24T08:20:08.773896+0000","last_unstale":"2024-04-24T08:20:10.899509+0000","last_undegraded":"2024-04-24T08:20:10.899509+0000","last_fullsized":"2024-04-24T08:20:10.899509+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:12:47.591419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897513+0000","last_change":"2024-04-24T08:20:09.793423+0000","last_active":"2024-04-24T08:20:10.897513+0000","last_peered":"2024-04-24T08:20:10.897513+0000","last_clean":"2024-04-24T08:20:10.897513+0000","last_became_active":"2024-04-24T08:20:09.793148+0000","last_became_peered":"2024-04-24T08:20:09.793148+0000","last_unstale":"2024-04-24T08:20:10.897513+0000","last_undegraded":"2024-04-24T08:20:10.897513+0000","last_fullsized":"2024-04-24T08:20:10.897513+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:39:47.452847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899236+0000","last_change":"2024-04-24T08:20:08.763470+0000","last_active":"2024-04-24T08:20:10.899236+0000","last_peered":"2024-04-24T08:20:10.899236+0000","last_clean":"2024-04-24T08:20:10.899236+0000","last_became_active":"2024-04-24T08:20:08.762839+0000","last_became_peered":"2024-04-24T08:20:08.762839+0000","last_unstale":"2024-04-24T08:20:10.899236+0000","last_undegraded":"2024-04-24T08:20:10.899236+0000","last_fullsized":"2024-04-24T08:20:10.899236+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:20.634892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896284+0000","last_change":"2024-04-24T08:20:09.788152+0000","last_active":"2024-04-24T08:20:10.896284+0000","last_peered":"2024-04-24T08:20:10.896284+0000","last_clean":"2024-04-24T08:20:10.896284+0000","last_became_active":"2024-04-24T08:20:09.787847+0000","last_became_peered":"2024-04-24T08:20:09.787847+0000","last_unstale":"2024-04-24T08:20:10.896284+0000","last_undegraded":"2024-04-24T08:20:10.896284+0000","last_fullsized":"2024-04-24T08:20:10.896284+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:59:19.646168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896389+0000","last_change":"2024-04-24T08:20:08.761630+0000","last_active":"2024-04-24T08:20:10.896389+0000","last_peered":"2024-04-24T08:20:10.896389+0000","last_clean":"2024-04-24T08:20:10.896389+0000","last_became_active":"2024-04-24T08:20:08.761355+0000","last_became_peered":"2024-04-24T08:20:08.761355+0000","last_unstale":"2024-04-24T08:20:10.896389+0000","last_undegraded":"2024-04-24T08:20:10.896389+0000","last_fullsized":"2024-04-24T08:20:10.896389+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:24.709059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897649+0000","last_change":"2024-04-24T08:20:09.800757+0000","last_active":"2024-04-24T08:20:10.897649+0000","last_peered":"2024-04-24T08:20:10.897649+0000","last_clean":"2024-04-24T08:20:10.897649+0000","last_became_active":"2024-04-24T08:20:09.800064+0000","last_became_peered":"2024-04-24T08:20:09.800064+0000","last_unstale":"2024-04-24T08:20:10.897649+0000","last_undegraded":"2024-04-24T08:20:10.897649+0000","last_fullsized":"2024-04-24T08:20:10.897649+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:13.818493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898874+0000","last_change":"2024-04-24T08:20:08.770890+0000","last_active":"2024-04-24T08:20:10.898874+0000","last_peered":"2024-04-24T08:20:10.898874+0000","last_clean":"2024-04-24T08:20:10.898874+0000","last_became_active":"2024-04-24T08:20:08.770587+0000","last_became_peered":"2024-04-24T08:20:08.770587+0000","last_unstale":"2024-04-24T08:20:10.898874+0000","last_undegraded":"2024-04-24T08:20:10.898874+0000","last_fullsized":"2024-04-24T08:20:10.898874+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:24:17.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790143+0000","last_change":"2024-04-24T08:20:09.773964+0000","last_active":"2024-04-24T08:20:10.790143+0000","last_peered":"2024-04-24T08:20:10.790143+0000","last_clean":"2024-04-24T08:20:10.790143+0000","last_became_active":"2024-04-24T08:20:09.773691+0000","last_became_peered":"2024-04-24T08:20:09.773691+0000","last_unstale":"2024-04-24T08:20:10.790143+0000","last_undegraded":"2024-04-24T08:20:10.790143+0000","last_fullsized":"2024-04-24T08:20:10.790143+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:47.526466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790239+0000","last_change":"2024-04-24T08:20:08.757528+0000","last_active":"2024-04-24T08:20:10.790239+0000","last_peered":"2024-04-24T08:20:10.790239+0000","last_clean":"2024-04-24T08:20:10.790239+0000","last_became_active":"2024-04-24T08:20:08.757135+0000","last_became_peered":"2024-04-24T08:20:08.757135+0000","last_unstale":"2024-04-24T08:20:10.790239+0000","last_undegraded":"2024-04-24T08:20:10.790239+0000","last_fullsized":"2024-04-24T08:20:10.790239+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:35:49.116094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790951+0000","last_change":"2024-04-24T08:20:09.793813+0000","last_active":"2024-04-24T08:20:10.790951+0000","last_peered":"2024-04-24T08:20:10.790951+0000","last_clean":"2024-04-24T08:20:10.790951+0000","last_became_active":"2024-04-24T08:20:09.793103+0000","last_became_peered":"2024-04-24T08:20:09.793103+0000","last_unstale":"2024-04-24T08:20:10.790951+0000","last_undegraded":"2024-04-24T08:20:10.790951+0000","last_fullsized":"2024-04-24T08:20:10.790951+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:25.726242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789590+0000","last_change":"2024-04-24T08:20:08.766731+0000","last_active":"2024-04-24T08:20:10.789590+0000","last_peered":"2024-04-24T08:20:10.789590+0000","last_clean":"2024-04-24T08:20:10.789590+0000","last_became_active":"2024-04-24T08:20:08.766406+0000","last_became_peered":"2024-04-24T08:20:08.766406+0000","last_unstale":"2024-04-24T08:20:10.789590+0000","last_undegraded":"2024-04-24T08:20:10.789590+0000","last_fullsized":"2024-04-24T08:20:10.789590+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:37:01.856004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897684+0000","last_change":"2024-04-24T08:20:09.789425+0000","last_active":"2024-04-24T08:20:10.897684+0000","last_peered":"2024-04-24T08:20:10.897684+0000","last_clean":"2024-04-24T08:20:10.897684+0000","last_became_active":"2024-04-24T08:20:09.788643+0000","last_became_peered":"2024-04-24T08:20:09.788643+0000","last_unstale":"2024-04-24T08:20:10.897684+0000","last_undegraded":"2024-04-24T08:20:10.897684+0000","last_fullsized":"2024-04-24T08:20:10.897684+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:27:27.905572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787134+0000","last_change":"2024-04-24T08:20:08.768906+0000","last_active":"2024-04-24T08:20:10.787134+0000","last_peered":"2024-04-24T08:20:10.787134+0000","last_clean":"2024-04-24T08:20:10.787134+0000","last_became_active":"2024-04-24T08:20:08.768593+0000","last_became_peered":"2024-04-24T08:20:08.768593+0000","last_unstale":"2024-04-24T08:20:10.787134+0000","last_undegraded":"2024-04-24T08:20:10.787134+0000","last_fullsized":"2024-04-24T08:20:10.787134+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:47:22.864945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897148+0000","last_change":"2024-04-24T08:20:09.795062+0000","last_active":"2024-04-24T08:20:10.897148+0000","last_peered":"2024-04-24T08:20:10.897148+0000","last_clean":"2024-04-24T08:20:10.897148+0000","last_became_active":"2024-04-24T08:20:09.794686+0000","last_became_peered":"2024-04-24T08:20:09.794686+0000","last_unstale":"2024-04-24T08:20:10.897148+0000","last_undegraded":"2024-04-24T08:20:10.897148+0000","last_fullsized":"2024-04-24T08:20:10.897148+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:08:08.087220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895229+0000","last_change":"2024-04-24T08:20:08.767036+0000","last_active":"2024-04-24T08:20:10.895229+0000","last_peered":"2024-04-24T08:20:10.895229+0000","last_clean":"2024-04-24T08:20:10.895229+0000","last_became_active":"2024-04-24T08:20:08.765832+0000","last_became_peered":"2024-04-24T08:20:08.765832+0000","last_unstale":"2024-04-24T08:20:10.895229+0000","last_undegraded":"2024-04-24T08:20:10.895229+0000","last_fullsized":"2024-04-24T08:20:10.895229+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:46.310731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898004+0000","last_change":"2024-04-24T08:20:09.787121+0000","last_active":"2024-04-24T08:20:10.898004+0000","last_peered":"2024-04-24T08:20:10.898004+0000","last_clean":"2024-04-24T08:20:10.898004+0000","last_became_active":"2024-04-24T08:20:09.786467+0000","last_became_peered":"2024-04-24T08:20:09.786467+0000","last_unstale":"2024-04-24T08:20:10.898004+0000","last_undegraded":"2024-04-24T08:20:10.898004+0000","last_fullsized":"2024-04-24T08:20:10.898004+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:25.726242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789569+0000","last_change":"2024-04-24T08:20:08.767872+0000","last_active":"2024-04-24T08:20:10.789569+0000","last_peered":"2024-04-24T08:20:10.789569+0000","last_clean":"2024-04-24T08:20:10.789569+0000","last_became_active":"2024-04-24T08:20:08.767538+0000","last_became_peered":"2024-04-24T08:20:08.767538+0000","last_unstale":"2024-04-24T08:20:10.789569+0000","last_undegraded":"2024-04-24T08:20:10.789569+0000","last_fullsized":"2024-04-24T08:20:10.789569+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:10:51.819583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898446+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:20:10.898446+0000","last_peered":"2024-04-24T08:20:10.898446+0000","last_clean":"2024-04-24T08:20:10.898446+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:20:10.898446+0000","last_undegraded":"2024-04-24T08:20:10.898446+0000","last_fullsized":"2024-04-24T08:20:10.898446+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574998,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1957888,"data_stored":1180021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574998,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":589107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574996,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1363968,"data_stored":589641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574995,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":589037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574996,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1372160,"data_stored":590719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574997,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":589037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574997,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1363968,"data_stored":589161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574996,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33112,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338728,"statfs":{"total":96636764160,"available":96602857472,"internally_reserved":0,"allocated":1970176,"data_stored":1181129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:20:16.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:20:16.533 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:20:19.210 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:20:19.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.409+0000 7fcfff200640 1 Processor -- start 2024-04-24T08:20:19.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.409+0000 7fcfff200640 1 -- start start 2024-04-24T08:20:19.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.410+0000 7fcfff200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0001523f0 0x7fd00006db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.410+0000 7fcfff200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd000072e20 0x7fd000073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.410+0000 7fcfff200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0000aeac0 con 0x7fd0000737a0 2024-04-24T08:20:19.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.411+0000 7fcfff200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd0000aeda0 con 0x7fd0001523f0 2024-04-24T08:20:19.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.411+0000 7fcfff200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd0000af080 con 0x7fd000072e20 2024-04-24T08:20:19.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.411+0000 7fcffe800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd0000737a0 0x7fd00015ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56946/0 (socket says 172.21.15.69:56946) 2024-04-24T08:20:19.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.411+0000 7fcffe800640 1 -- 172.21.15.69:0/2825732070 learned_addr learned my addr 172.21.15.69:0/2825732070 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:19.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.411+0000 7fcffde00640 1 --2- 172.21.15.69:0/2825732070 >> v2:172.21.15.69:3300/0 conn(0x7fd0001523f0 0x7fd00006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.411+0000 7fcffd400640 1 --2- 172.21.15.69:0/2825732070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd000072e20 0x7fd000073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.411+0000 7fcffde00640 1 -- 172.21.15.69:0/2825732070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd000072e20 msgr2=0x7fd000073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:19.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.411+0000 7fcffde00640 1 --2- 172.21.15.69:0/2825732070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd000072e20 0x7fd000073260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.412+0000 7fcffde00640 1 -- 172.21.15.69:0/2825732070 >> v1:172.21.15.69:6789/0 conn(0x7fd0000737a0 legacy=0x7fd00015ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:19.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.412+0000 7fcffde00640 1 -- 172.21.15.69:0/2825732070 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0000af360 con 0x7fd0001523f0 2024-04-24T08:20:19.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.412+0000 7fcffca00640 1 -- 172.21.15.69:0/2825732070 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399044186 0 0) 0x7fd0000aeac0 con 0x7fd0000737a0 2024-04-24T08:20:19.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.413+0000 7fcffde00640 1 --2- 172.21.15.69:0/2825732070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 0x7fd00006db90 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fcff00097a0 tx=0x7fcff002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c687d1e663aeba9d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:19.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.413+0000 7fcffca00640 1 -- 172.21.15.69:0/2825732070 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff0037070 con 0x7fd0001523f0 2024-04-24T08:20:19.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.413+0000 7fcffca00640 1 -- 172.21.15.69:0/2825732070 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcff002ae50 con 0x7fd0001523f0 2024-04-24T08:20:19.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.413+0000 7fcffca00640 1 -- 172.21.15.69:0/2825732070 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff0032660 con 0x7fd0001523f0 2024-04-24T08:20:19.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.413+0000 7fcfff200640 1 -- 172.21.15.69:0/2825732070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 msgr2=0x7fd00006db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:19.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.413+0000 7fcfff200640 1 --2- 172.21.15.69:0/2825732070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 0x7fd00006db90 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7fcff00097a0 tx=0x7fcff002a100 comp rx=0 tx=0).stop 2024-04-24T08:20:19.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.414+0000 7fcfff200640 1 -- 172.21.15.69:0/2825732070 shutdown_connections 2024-04-24T08:20:19.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.414+0000 7fcfff200640 1 --2- 172.21.15.69:0/2825732070 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd000072e20 0x7fd000073260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.414+0000 7fcfff200640 1 --2- 172.21.15.69:0/2825732070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 0x7fd00006db90 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.414+0000 7fcfff200640 1 -- 172.21.15.69:0/2825732070 >> 172.21.15.69:0/2825732070 conn(0x7fd0000d0160 msgr2=0x7fd0000d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:19.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.414+0000 7fcfff200640 1 -- 172.21.15.69:0/2825732070 shutdown_connections 2024-04-24T08:20:19.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.414+0000 7fcfff200640 1 -- 172.21.15.69:0/2825732070 wait complete. 2024-04-24T08:20:19.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.415+0000 7fcfff200640 1 Processor -- start 2024-04-24T08:20:19.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.416+0000 7fcfff200640 1 -- start start 2024-04-24T08:20:19.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.416+0000 7fcfff200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd000072e20 0x7fd000152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.416+0000 7fcfff200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0000737a0 0x7fd00015c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcfff200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 0x7fd0001491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcfff200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd00015f680 con 0x7fd0001523f0 2024-04-24T08:20:19.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcfff200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd0001527d0 con 0x7fd0000737a0 2024-04-24T08:20:19.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcfff200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd00015f800 con 0x7fd000072e20 2024-04-24T08:20:19.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcffe800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 0x7fd0001491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcffe800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 0x7fd0001491f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40796/0 (socket says 172.21.15.69:40796) 2024-04-24T08:20:19.418 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcffe800640 1 -- 172.21.15.69:0/3555279993 learned_addr learned my addr 172.21.15.69:0/3555279993 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:19.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcffd400640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0000737a0 0x7fd00015c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcffe800640 1 -- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd000072e20 msgr2=0x7fd000152950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:19.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcffde00640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd000072e20 0x7fd000152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcffe800640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd000072e20 0x7fd000152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcffe800640 1 -- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0000737a0 msgr2=0x7fd00015c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:19.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcffe800640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0000737a0 0x7fd00015c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.417+0000 7fcffe800640 1 -- 172.21.15.69:0/3555279993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0001577b0 con 0x7fd0001523f0 2024-04-24T08:20:19.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.418+0000 7fcffe800640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 0x7fd0001491f0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fcff800c7a0 tx=0x7fcff800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:19.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.418+0000 7fcff6000640 1 -- 172.21.15.69:0/3555279993 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff8018070 con 0x7fd0001523f0 2024-04-24T08:20:19.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.418+0000 7fcff6000640 1 -- 172.21.15.69:0/3555279993 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcff800a640 con 0x7fd0001523f0 2024-04-24T08:20:19.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.418+0000 7fcff6000640 1 -- 172.21.15.69:0/3555279993 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff800aab0 con 0x7fd0001523f0 2024-04-24T08:20:19.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.418+0000 7fcfff200640 1 -- 172.21.15.69:0/3555279993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0001f83b0 con 0x7fd0001523f0 2024-04-24T08:20:19.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.418+0000 7fcfff200640 1 -- 172.21.15.69:0/3555279993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd000157d30 con 0x7fd0001523f0 2024-04-24T08:20:19.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.419+0000 7fcfff200640 1 -- 172.21.15.69:0/3555279993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd000075d50 con 0x7fd0001523f0 2024-04-24T08:20:19.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.420+0000 7fcff6000640 1 -- 172.21.15.69:0/3555279993 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcff800ac50 con 0x7fd0001523f0 2024-04-24T08:20:19.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.421+0000 7fcff6000640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 0x7fcfe006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.421+0000 7fcff6000640 1 -- 172.21.15.69:0/3555279993 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fcff808b1d0 con 0x7fd0001523f0 2024-04-24T08:20:19.422 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.422+0000 7fcffde00640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 0x7fcfe006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.423+0000 7fcffde00640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 0x7fcfe006edb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fcff0034000 tx=0x7fcff002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:19.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.425+0000 7fcff6000640 1 -- 172.21.15.69:0/3555279993 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcff8059600 con 0x7fd0001523f0 2024-04-24T08:20:19.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.622+0000 7fcfff200640 1 -- 172.21.15.69:0/3555279993 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd0000cfb90 con 0x7fcfe006c8f0 2024-04-24T08:20:19.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.641+0000 7fcff6000640 1 -- 172.21.15.69:0/3555279993 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252371 (secure 0 0 0) 0x7fd0000cfb90 con 0x7fcfe006c8f0 2024-04-24T08:20:19.643 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:19.647 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:20:19.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.647+0000 7fcfff200640 1 -- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 msgr2=0x7fcfe006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:19.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.647+0000 7fcfff200640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 0x7fcfe006edb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fcff0034000 tx=0x7fcff002e000 comp rx=0 tx=0).stop 2024-04-24T08:20:19.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.647+0000 7fcfff200640 1 -- 172.21.15.69:0/3555279993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 msgr2=0x7fd0001491f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:19.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.647+0000 7fcfff200640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 0x7fd0001491f0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7fcff800c7a0 tx=0x7fcff800cc60 comp rx=0 tx=0).stop 2024-04-24T08:20:19.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.648+0000 7fcfff200640 1 -- 172.21.15.69:0/3555279993 shutdown_connections 2024-04-24T08:20:19.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.648+0000 7fcfff200640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 0x7fcfe006edb0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.648+0000 7fcfff200640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0001523f0 0x7fd0001491f0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.648+0000 7fcfff200640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0000737a0 0x7fd00015c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.648+0000 7fcfff200640 1 --2- 172.21.15.69:0/3555279993 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd000072e20 0x7fd000152950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.648+0000 7fcfff200640 1 -- 172.21.15.69:0/3555279993 >> 172.21.15.69:0/3555279993 conn(0x7fd0000d0160 msgr2=0x7fd0000b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:19.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.648+0000 7fcfff200640 1 -- 172.21.15.69:0/3555279993 shutdown_connections 2024-04-24T08:20:19.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.648+0000 7fcfff200640 1 -- 172.21.15.69:0/3555279993 wait complete. 2024-04-24T08:20:19.668 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-04-24T08:20:18.932749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":261288,"kb_used_data":11816,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754713432,"statfs":{"total":773094113280,"available":772826554368,"internally_reserved":0,"allocated":12099584,"data_stored":5897852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.172112"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896427+0000","last_change":"2024-04-24T08:20:09.799542+0000","last_active":"2024-04-24T08:20:10.896427+0000","last_peered":"2024-04-24T08:20:10.896427+0000","last_clean":"2024-04-24T08:20:10.896427+0000","last_became_active":"2024-04-24T08:20:09.799243+0000","last_became_peered":"2024-04-24T08:20:09.799243+0000","last_unstale":"2024-04-24T08:20:10.896427+0000","last_undegraded":"2024-04-24T08:20:10.896427+0000","last_fullsized":"2024-04-24T08:20:10.896427+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:19:43.824329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791360+0000","last_change":"2024-04-24T08:20:09.794708+0000","last_active":"2024-04-24T08:20:10.791360+0000","last_peered":"2024-04-24T08:20:10.791360+0000","last_clean":"2024-04-24T08:20:10.791360+0000","last_became_active":"2024-04-24T08:20:09.793835+0000","last_became_peered":"2024-04-24T08:20:09.793835+0000","last_unstale":"2024-04-24T08:20:10.791360+0000","last_undegraded":"2024-04-24T08:20:10.791360+0000","last_fullsized":"2024-04-24T08:20:10.791360+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:13.818493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897294+0000","last_change":"2024-04-24T08:20:09.788114+0000","last_active":"2024-04-24T08:20:10.897294+0000","last_peered":"2024-04-24T08:20:10.897294+0000","last_clean":"2024-04-24T08:20:10.897294+0000","last_became_active":"2024-04-24T08:20:09.787809+0000","last_became_peered":"2024-04-24T08:20:09.787809+0000","last_unstale":"2024-04-24T08:20:10.897294+0000","last_undegraded":"2024-04-24T08:20:10.897294+0000","last_fullsized":"2024-04-24T08:20:10.897294+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:26:31.959658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895708+0000","last_change":"2024-04-24T08:20:09.794858+0000","last_active":"2024-04-24T08:20:10.895708+0000","last_peered":"2024-04-24T08:20:10.895708+0000","last_clean":"2024-04-24T08:20:10.895708+0000","last_became_active":"2024-04-24T08:20:09.794047+0000","last_became_peered":"2024-04-24T08:20:09.794047+0000","last_unstale":"2024-04-24T08:20:10.895708+0000","last_undegraded":"2024-04-24T08:20:10.895708+0000","last_fullsized":"2024-04-24T08:20:10.895708+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:52:00.465750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788117+0000","last_change":"2024-04-24T08:20:09.780027+0000","last_active":"2024-04-24T08:20:10.788117+0000","last_peered":"2024-04-24T08:20:10.788117+0000","last_clean":"2024-04-24T08:20:10.788117+0000","last_became_active":"2024-04-24T08:20:09.779744+0000","last_became_peered":"2024-04-24T08:20:09.779744+0000","last_unstale":"2024-04-24T08:20:10.788117+0000","last_undegraded":"2024-04-24T08:20:10.788117+0000","last_fullsized":"2024-04-24T08:20:10.788117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:09.432747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897475+0000","last_change":"2024-04-24T08:20:09.784098+0000","last_active":"2024-04-24T08:20:10.897475+0000","last_peered":"2024-04-24T08:20:10.897475+0000","last_clean":"2024-04-24T08:20:10.897475+0000","last_became_active":"2024-04-24T08:20:09.783835+0000","last_became_peered":"2024-04-24T08:20:09.783835+0000","last_unstale":"2024-04-24T08:20:10.897475+0000","last_undegraded":"2024-04-24T08:20:10.897475+0000","last_fullsized":"2024-04-24T08:20:10.897475+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:56.817425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789432+0000","last_change":"2024-04-24T08:20:09.785141+0000","last_active":"2024-04-24T08:20:10.789432+0000","last_peered":"2024-04-24T08:20:10.789432+0000","last_clean":"2024-04-24T08:20:10.789432+0000","last_became_active":"2024-04-24T08:20:09.784585+0000","last_became_peered":"2024-04-24T08:20:09.784585+0000","last_unstale":"2024-04-24T08:20:10.789432+0000","last_undegraded":"2024-04-24T08:20:10.789432+0000","last_fullsized":"2024-04-24T08:20:10.789432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:44:55.266192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896771+0000","last_change":"2024-04-24T08:20:09.792355+0000","last_active":"2024-04-24T08:20:10.896771+0000","last_peered":"2024-04-24T08:20:10.896771+0000","last_clean":"2024-04-24T08:20:10.896771+0000","last_became_active":"2024-04-24T08:20:09.791438+0000","last_became_peered":"2024-04-24T08:20:09.791438+0000","last_unstale":"2024-04-24T08:20:10.896771+0000","last_undegraded":"2024-04-24T08:20:10.896771+0000","last_fullsized":"2024-04-24T08:20:10.896771+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:59:41.162621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897198+0000","last_change":"2024-04-24T08:20:09.803001+0000","last_active":"2024-04-24T08:20:10.897198+0000","last_peered":"2024-04-24T08:20:10.897198+0000","last_clean":"2024-04-24T08:20:10.897198+0000","last_became_active":"2024-04-24T08:20:09.801984+0000","last_became_peered":"2024-04-24T08:20:09.801984+0000","last_unstale":"2024-04-24T08:20:10.897198+0000","last_undegraded":"2024-04-24T08:20:10.897198+0000","last_fullsized":"2024-04-24T08:20:10.897198+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:51:22.573683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896823+0000","last_change":"2024-04-24T08:20:09.801533+0000","last_active":"2024-04-24T08:20:10.896823+0000","last_peered":"2024-04-24T08:20:10.896823+0000","last_clean":"2024-04-24T08:20:10.896823+0000","last_became_active":"2024-04-24T08:20:09.801142+0000","last_became_peered":"2024-04-24T08:20:09.801142+0000","last_unstale":"2024-04-24T08:20:10.896823+0000","last_undegraded":"2024-04-24T08:20:10.896823+0000","last_fullsized":"2024-04-24T08:20:10.896823+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:05.572616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791049+0000","last_change":"2024-04-24T08:20:09.791103+0000","last_active":"2024-04-24T08:20:10.791049+0000","last_peered":"2024-04-24T08:20:10.791049+0000","last_clean":"2024-04-24T08:20:10.791049+0000","last_became_active":"2024-04-24T08:20:09.787354+0000","last_became_peered":"2024-04-24T08:20:09.787354+0000","last_unstale":"2024-04-24T08:20:10.791049+0000","last_undegraded":"2024-04-24T08:20:10.791049+0000","last_fullsized":"2024-04-24T08:20:10.791049+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:48.090822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896003+0000","last_change":"2024-04-24T08:20:09.802741+0000","last_active":"2024-04-24T08:20:10.896003+0000","last_peered":"2024-04-24T08:20:10.896003+0000","last_clean":"2024-04-24T08:20:10.896003+0000","last_became_active":"2024-04-24T08:20:09.802402+0000","last_became_peered":"2024-04-24T08:20:09.802402+0000","last_unstale":"2024-04-24T08:20:10.896003+0000","last_undegraded":"2024-04-24T08:20:10.896003+0000","last_fullsized":"2024-04-24T08:20:10.896003+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:52.645695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897895+0000","last_change":"2024-04-24T08:20:09.800630+0000","last_active":"2024-04-24T08:20:10.897895+0000","last_peered":"2024-04-24T08:20:10.897895+0000","last_clean":"2024-04-24T08:20:10.897895+0000","last_became_active":"2024-04-24T08:20:09.800140+0000","last_became_peered":"2024-04-24T08:20:09.800140+0000","last_unstale":"2024-04-24T08:20:10.897895+0000","last_undegraded":"2024-04-24T08:20:10.897895+0000","last_fullsized":"2024-04-24T08:20:10.897895+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:57.166226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896796+0000","last_change":"2024-04-24T08:20:09.789152+0000","last_active":"2024-04-24T08:20:10.896796+0000","last_peered":"2024-04-24T08:20:10.896796+0000","last_clean":"2024-04-24T08:20:10.896796+0000","last_became_active":"2024-04-24T08:20:09.788646+0000","last_became_peered":"2024-04-24T08:20:09.788646+0000","last_unstale":"2024-04-24T08:20:10.896796+0000","last_undegraded":"2024-04-24T08:20:10.896796+0000","last_fullsized":"2024-04-24T08:20:10.896796+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:10:50.551575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897147+0000","last_change":"2024-04-24T08:20:09.782866+0000","last_active":"2024-04-24T08:20:10.897147+0000","last_peered":"2024-04-24T08:20:10.897147+0000","last_clean":"2024-04-24T08:20:10.897147+0000","last_became_active":"2024-04-24T08:20:09.781963+0000","last_became_peered":"2024-04-24T08:20:09.781963+0000","last_unstale":"2024-04-24T08:20:10.897147+0000","last_undegraded":"2024-04-24T08:20:10.897147+0000","last_fullsized":"2024-04-24T08:20:10.897147+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:42:59.649790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.894889+0000","last_change":"2024-04-24T08:20:09.799141+0000","last_active":"2024-04-24T08:20:10.894889+0000","last_peered":"2024-04-24T08:20:10.894889+0000","last_clean":"2024-04-24T08:20:10.894889+0000","last_became_active":"2024-04-24T08:20:09.798679+0000","last_became_peered":"2024-04-24T08:20:09.798679+0000","last_unstale":"2024-04-24T08:20:10.894889+0000","last_undegraded":"2024-04-24T08:20:10.894889+0000","last_fullsized":"2024-04-24T08:20:10.894889+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:06:29.713888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791716+0000","last_change":"2024-04-24T08:20:09.789678+0000","last_active":"2024-04-24T08:20:10.791716+0000","last_peered":"2024-04-24T08:20:10.791716+0000","last_clean":"2024-04-24T08:20:10.791716+0000","last_became_active":"2024-04-24T08:20:09.787273+0000","last_became_peered":"2024-04-24T08:20:09.787273+0000","last_unstale":"2024-04-24T08:20:10.791716+0000","last_undegraded":"2024-04-24T08:20:10.791716+0000","last_fullsized":"2024-04-24T08:20:10.791716+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:29.553044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898926+0000","last_change":"2024-04-24T08:20:09.785296+0000","last_active":"2024-04-24T08:20:10.898926+0000","last_peered":"2024-04-24T08:20:10.898926+0000","last_clean":"2024-04-24T08:20:10.898926+0000","last_became_active":"2024-04-24T08:20:09.784996+0000","last_became_peered":"2024-04-24T08:20:09.784996+0000","last_unstale":"2024-04-24T08:20:10.898926+0000","last_undegraded":"2024-04-24T08:20:10.898926+0000","last_fullsized":"2024-04-24T08:20:10.898926+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:09:43.173248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897367+0000","last_change":"2024-04-24T08:20:09.797840+0000","last_active":"2024-04-24T08:20:10.897367+0000","last_peered":"2024-04-24T08:20:10.897367+0000","last_clean":"2024-04-24T08:20:10.897367+0000","last_became_active":"2024-04-24T08:20:09.797575+0000","last_became_peered":"2024-04-24T08:20:09.797575+0000","last_unstale":"2024-04-24T08:20:10.897367+0000","last_undegraded":"2024-04-24T08:20:10.897367+0000","last_fullsized":"2024-04-24T08:20:10.897367+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:35.871237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791599+0000","last_change":"2024-04-24T08:20:09.789338+0000","last_active":"2024-04-24T08:20:10.791599+0000","last_peered":"2024-04-24T08:20:10.791599+0000","last_clean":"2024-04-24T08:20:10.791599+0000","last_became_active":"2024-04-24T08:20:09.787008+0000","last_became_peered":"2024-04-24T08:20:09.787008+0000","last_unstale":"2024-04-24T08:20:10.791599+0000","last_undegraded":"2024-04-24T08:20:10.791599+0000","last_fullsized":"2024-04-24T08:20:10.791599+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:03.358776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791614+0000","last_change":"2024-04-24T08:20:09.785546+0000","last_active":"2024-04-24T08:20:10.791614+0000","last_peered":"2024-04-24T08:20:10.791614+0000","last_clean":"2024-04-24T08:20:10.791614+0000","last_became_active":"2024-04-24T08:20:09.785285+0000","last_became_peered":"2024-04-24T08:20:09.785285+0000","last_unstale":"2024-04-24T08:20:10.791614+0000","last_undegraded":"2024-04-24T08:20:10.791614+0000","last_fullsized":"2024-04-24T08:20:10.791614+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:47.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899951+0000","last_change":"2024-04-24T08:20:09.792270+0000","last_active":"2024-04-24T08:20:10.899951+0000","last_peered":"2024-04-24T08:20:10.899951+0000","last_clean":"2024-04-24T08:20:10.899951+0000","last_became_active":"2024-04-24T08:20:09.791977+0000","last_became_peered":"2024-04-24T08:20:09.791977+0000","last_unstale":"2024-04-24T08:20:10.899951+0000","last_undegraded":"2024-04-24T08:20:10.899951+0000","last_fullsized":"2024-04-24T08:20:10.899951+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:47.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788610+0000","last_change":"2024-04-24T08:20:09.788141+0000","last_active":"2024-04-24T08:20:10.788610+0000","last_peered":"2024-04-24T08:20:10.788610+0000","last_clean":"2024-04-24T08:20:10.788610+0000","last_became_active":"2024-04-24T08:20:09.787507+0000","last_became_peered":"2024-04-24T08:20:09.787507+0000","last_unstale":"2024-04-24T08:20:10.788610+0000","last_undegraded":"2024-04-24T08:20:10.788610+0000","last_fullsized":"2024-04-24T08:20:10.788610+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:21.652074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897965+0000","last_change":"2024-04-24T08:20:09.787331+0000","last_active":"2024-04-24T08:20:10.897965+0000","last_peered":"2024-04-24T08:20:10.897965+0000","last_clean":"2024-04-24T08:20:10.897965+0000","last_became_active":"2024-04-24T08:20:09.786840+0000","last_became_peered":"2024-04-24T08:20:09.786840+0000","last_unstale":"2024-04-24T08:20:10.897965+0000","last_undegraded":"2024-04-24T08:20:10.897965+0000","last_fullsized":"2024-04-24T08:20:10.897965+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:47.526466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.791717+0000","last_change":"2024-04-24T08:20:09.787529+0000","last_active":"2024-04-24T08:20:10.791717+0000","last_peered":"2024-04-24T08:20:10.791717+0000","last_clean":"2024-04-24T08:20:10.791717+0000","last_became_active":"2024-04-24T08:20:09.787236+0000","last_became_peered":"2024-04-24T08:20:09.787236+0000","last_unstale":"2024-04-24T08:20:10.791717+0000","last_undegraded":"2024-04-24T08:20:10.791717+0000","last_fullsized":"2024-04-24T08:20:10.791717+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:09:43.173248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896884+0000","last_change":"2024-04-24T08:20:09.788225+0000","last_active":"2024-04-24T08:20:10.896884+0000","last_peered":"2024-04-24T08:20:10.896884+0000","last_clean":"2024-04-24T08:20:10.896884+0000","last_became_active":"2024-04-24T08:20:09.787936+0000","last_became_peered":"2024-04-24T08:20:09.787936+0000","last_unstale":"2024-04-24T08:20:10.896884+0000","last_undegraded":"2024-04-24T08:20:10.896884+0000","last_fullsized":"2024-04-24T08:20:10.896884+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:20:18.482020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899461+0000","last_change":"2024-04-24T08:20:09.794928+0000","last_active":"2024-04-24T08:20:10.899461+0000","last_peered":"2024-04-24T08:20:10.899461+0000","last_clean":"2024-04-24T08:20:10.899461+0000","last_became_active":"2024-04-24T08:20:09.794289+0000","last_became_peered":"2024-04-24T08:20:09.794289+0000","last_unstale":"2024-04-24T08:20:10.899461+0000","last_undegraded":"2024-04-24T08:20:10.899461+0000","last_fullsized":"2024-04-24T08:20:10.899461+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:51:07.962266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898786+0000","last_change":"2024-04-24T08:20:09.794776+0000","last_active":"2024-04-24T08:20:10.898786+0000","last_peered":"2024-04-24T08:20:10.898786+0000","last_clean":"2024-04-24T08:20:10.898786+0000","last_became_active":"2024-04-24T08:20:09.794151+0000","last_became_peered":"2024-04-24T08:20:09.794151+0000","last_unstale":"2024-04-24T08:20:10.898786+0000","last_undegraded":"2024-04-24T08:20:10.898786+0000","last_fullsized":"2024-04-24T08:20:10.898786+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:03.358776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790533+0000","last_change":"2024-04-24T08:20:09.791933+0000","last_active":"2024-04-24T08:20:10.790533+0000","last_peered":"2024-04-24T08:20:10.790533+0000","last_clean":"2024-04-24T08:20:10.790533+0000","last_became_active":"2024-04-24T08:20:09.789144+0000","last_became_peered":"2024-04-24T08:20:09.789144+0000","last_unstale":"2024-04-24T08:20:10.790533+0000","last_undegraded":"2024-04-24T08:20:10.790533+0000","last_fullsized":"2024-04-24T08:20:10.790533+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:57.166226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789744+0000","last_change":"2024-04-24T08:20:09.779514+0000","last_active":"2024-04-24T08:20:10.789744+0000","last_peered":"2024-04-24T08:20:10.789744+0000","last_clean":"2024-04-24T08:20:10.789744+0000","last_became_active":"2024-04-24T08:20:09.779002+0000","last_became_peered":"2024-04-24T08:20:09.779002+0000","last_unstale":"2024-04-24T08:20:10.789744+0000","last_undegraded":"2024-04-24T08:20:10.789744+0000","last_fullsized":"2024-04-24T08:20:10.789744+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:42:59.649790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895194+0000","last_change":"2024-04-24T08:20:09.789034+0000","last_active":"2024-04-24T08:20:10.895194+0000","last_peered":"2024-04-24T08:20:10.895194+0000","last_clean":"2024-04-24T08:20:10.895194+0000","last_became_active":"2024-04-24T08:20:09.788764+0000","last_became_peered":"2024-04-24T08:20:09.788764+0000","last_unstale":"2024-04-24T08:20:10.895194+0000","last_undegraded":"2024-04-24T08:20:10.895194+0000","last_fullsized":"2024-04-24T08:20:10.895194+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:54:29.411232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896727+0000","last_change":"2024-04-24T08:20:09.796857+0000","last_active":"2024-04-24T08:20:10.896727+0000","last_peered":"2024-04-24T08:20:10.896727+0000","last_clean":"2024-04-24T08:20:10.896727+0000","last_became_active":"2024-04-24T08:20:09.795564+0000","last_became_peered":"2024-04-24T08:20:09.795564+0000","last_unstale":"2024-04-24T08:20:10.896727+0000","last_undegraded":"2024-04-24T08:20:10.896727+0000","last_fullsized":"2024-04-24T08:20:10.896727+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:44:55.266192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899863+0000","last_change":"2024-04-24T08:20:09.779341+0000","last_active":"2024-04-24T08:20:10.899863+0000","last_peered":"2024-04-24T08:20:10.899863+0000","last_clean":"2024-04-24T08:20:10.899863+0000","last_became_active":"2024-04-24T08:20:09.779056+0000","last_became_peered":"2024-04-24T08:20:09.779056+0000","last_unstale":"2024-04-24T08:20:10.899863+0000","last_undegraded":"2024-04-24T08:20:10.899863+0000","last_fullsized":"2024-04-24T08:20:10.899863+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:36:34.919462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"108'4","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790412+0000","last_change":"2024-04-24T08:20:08.765030+0000","last_active":"2024-04-24T08:20:10.790412+0000","last_peered":"2024-04-24T08:20:10.790412+0000","last_clean":"2024-04-24T08:20:10.790412+0000","last_became_active":"2024-04-24T08:20:08.764730+0000","last_became_peered":"2024-04-24T08:20:08.764730+0000","last_unstale":"2024-04-24T08:20:10.790412+0000","last_undegraded":"2024-04-24T08:20:10.790412+0000","last_fullsized":"2024-04-24T08:20:10.790412+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:19:40.821738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898429+0000","last_change":"2024-04-24T08:20:09.793872+0000","last_active":"2024-04-24T08:20:10.898429+0000","last_peered":"2024-04-24T08:20:10.898429+0000","last_clean":"2024-04-24T08:20:10.898429+0000","last_became_active":"2024-04-24T08:20:09.793548+0000","last_became_peered":"2024-04-24T08:20:09.793548+0000","last_unstale":"2024-04-24T08:20:10.898429+0000","last_undegraded":"2024-04-24T08:20:10.898429+0000","last_fullsized":"2024-04-24T08:20:10.898429+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:03.358776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898085+0000","last_change":"2024-04-24T08:20:08.764950+0000","last_active":"2024-04-24T08:20:10.898085+0000","last_peered":"2024-04-24T08:20:10.898085+0000","last_clean":"2024-04-24T08:20:10.898085+0000","last_became_active":"2024-04-24T08:20:08.764618+0000","last_became_peered":"2024-04-24T08:20:08.764618+0000","last_unstale":"2024-04-24T08:20:10.898085+0000","last_undegraded":"2024-04-24T08:20:10.898085+0000","last_fullsized":"2024-04-24T08:20:10.898085+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:02.260282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899378+0000","last_change":"2024-04-24T08:20:09.796732+0000","last_active":"2024-04-24T08:20:10.899378+0000","last_peered":"2024-04-24T08:20:10.899378+0000","last_clean":"2024-04-24T08:20:10.899378+0000","last_became_active":"2024-04-24T08:20:09.795416+0000","last_became_peered":"2024-04-24T08:20:09.795416+0000","last_unstale":"2024-04-24T08:20:10.899378+0000","last_undegraded":"2024-04-24T08:20:10.899378+0000","last_fullsized":"2024-04-24T08:20:10.899378+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:38.016468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787234+0000","last_change":"2024-04-24T08:20:08.772932+0000","last_active":"2024-04-24T08:20:10.787234+0000","last_peered":"2024-04-24T08:20:10.787234+0000","last_clean":"2024-04-24T08:20:10.787234+0000","last_became_active":"2024-04-24T08:20:08.772234+0000","last_became_peered":"2024-04-24T08:20:08.772234+0000","last_unstale":"2024-04-24T08:20:10.787234+0000","last_undegraded":"2024-04-24T08:20:10.787234+0000","last_fullsized":"2024-04-24T08:20:10.787234+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:12:45.289874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897735+0000","last_change":"2024-04-24T08:20:09.797817+0000","last_active":"2024-04-24T08:20:10.897735+0000","last_peered":"2024-04-24T08:20:10.897735+0000","last_clean":"2024-04-24T08:20:10.897735+0000","last_became_active":"2024-04-24T08:20:09.797517+0000","last_became_peered":"2024-04-24T08:20:09.797517+0000","last_unstale":"2024-04-24T08:20:10.897735+0000","last_undegraded":"2024-04-24T08:20:10.897735+0000","last_fullsized":"2024-04-24T08:20:10.897735+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:21.652074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787417+0000","last_change":"2024-04-24T08:20:08.770132+0000","last_active":"2024-04-24T08:20:10.787417+0000","last_peered":"2024-04-24T08:20:10.787417+0000","last_clean":"2024-04-24T08:20:10.787417+0000","last_became_active":"2024-04-24T08:20:08.769756+0000","last_became_peered":"2024-04-24T08:20:08.769756+0000","last_unstale":"2024-04-24T08:20:10.787417+0000","last_undegraded":"2024-04-24T08:20:10.787417+0000","last_fullsized":"2024-04-24T08:20:10.787417+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:57:23.545844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896040+0000","last_change":"2024-04-24T08:20:09.804520+0000","last_active":"2024-04-24T08:20:10.896040+0000","last_peered":"2024-04-24T08:20:10.896040+0000","last_clean":"2024-04-24T08:20:10.896040+0000","last_became_active":"2024-04-24T08:20:09.804216+0000","last_became_peered":"2024-04-24T08:20:09.804216+0000","last_unstale":"2024-04-24T08:20:10.896040+0000","last_undegraded":"2024-04-24T08:20:10.896040+0000","last_fullsized":"2024-04-24T08:20:10.896040+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:45:39.680748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789936+0000","last_change":"2024-04-24T08:20:08.763740+0000","last_active":"2024-04-24T08:20:10.789936+0000","last_peered":"2024-04-24T08:20:10.789936+0000","last_clean":"2024-04-24T08:20:10.789936+0000","last_became_active":"2024-04-24T08:20:08.763472+0000","last_became_peered":"2024-04-24T08:20:08.763472+0000","last_unstale":"2024-04-24T08:20:10.789936+0000","last_undegraded":"2024-04-24T08:20:10.789936+0000","last_fullsized":"2024-04-24T08:20:10.789936+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:13:02.412872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790984+0000","last_change":"2024-04-24T08:20:09.796520+0000","last_active":"2024-04-24T08:20:10.790984+0000","last_peered":"2024-04-24T08:20:10.790984+0000","last_clean":"2024-04-24T08:20:10.790984+0000","last_became_active":"2024-04-24T08:20:09.793445+0000","last_became_peered":"2024-04-24T08:20:09.793445+0000","last_unstale":"2024-04-24T08:20:10.790984+0000","last_undegraded":"2024-04-24T08:20:10.790984+0000","last_fullsized":"2024-04-24T08:20:10.790984+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:34.067483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.900109+0000","last_change":"2024-04-24T08:20:08.762014+0000","last_active":"2024-04-24T08:20:10.900109+0000","last_peered":"2024-04-24T08:20:10.900109+0000","last_clean":"2024-04-24T08:20:10.900109+0000","last_became_active":"2024-04-24T08:20:08.761675+0000","last_became_peered":"2024-04-24T08:20:08.761675+0000","last_unstale":"2024-04-24T08:20:10.900109+0000","last_undegraded":"2024-04-24T08:20:10.900109+0000","last_fullsized":"2024-04-24T08:20:10.900109+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:23.173525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895119+0000","last_change":"2024-04-24T08:20:09.795885+0000","last_active":"2024-04-24T08:20:10.895119+0000","last_peered":"2024-04-24T08:20:10.895119+0000","last_clean":"2024-04-24T08:20:10.895119+0000","last_became_active":"2024-04-24T08:20:09.795631+0000","last_became_peered":"2024-04-24T08:20:09.795631+0000","last_unstale":"2024-04-24T08:20:10.895119+0000","last_undegraded":"2024-04-24T08:20:10.895119+0000","last_fullsized":"2024-04-24T08:20:10.895119+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:19:33.323423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788665+0000","last_change":"2024-04-24T08:20:08.755417+0000","last_active":"2024-04-24T08:20:10.788665+0000","last_peered":"2024-04-24T08:20:10.788665+0000","last_clean":"2024-04-24T08:20:10.788665+0000","last_became_active":"2024-04-24T08:20:08.755131+0000","last_became_peered":"2024-04-24T08:20:08.755131+0000","last_unstale":"2024-04-24T08:20:10.788665+0000","last_undegraded":"2024-04-24T08:20:10.788665+0000","last_fullsized":"2024-04-24T08:20:10.788665+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:02:23.173525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789296+0000","last_change":"2024-04-24T08:20:09.785134+0000","last_active":"2024-04-24T08:20:10.789296+0000","last_peered":"2024-04-24T08:20:10.789296+0000","last_clean":"2024-04-24T08:20:10.789296+0000","last_became_active":"2024-04-24T08:20:09.784131+0000","last_became_peered":"2024-04-24T08:20:09.784131+0000","last_unstale":"2024-04-24T08:20:10.789296+0000","last_undegraded":"2024-04-24T08:20:10.789296+0000","last_fullsized":"2024-04-24T08:20:10.789296+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:16:12.799342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895001+0000","last_change":"2024-04-24T08:20:08.764423+0000","last_active":"2024-04-24T08:20:10.895001+0000","last_peered":"2024-04-24T08:20:10.895001+0000","last_clean":"2024-04-24T08:20:10.895001+0000","last_became_active":"2024-04-24T08:20:08.764017+0000","last_became_peered":"2024-04-24T08:20:08.764017+0000","last_unstale":"2024-04-24T08:20:10.895001+0000","last_undegraded":"2024-04-24T08:20:10.895001+0000","last_fullsized":"2024-04-24T08:20:10.895001+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:47.073639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790527+0000","last_change":"2024-04-24T08:20:09.775184+0000","last_active":"2024-04-24T08:20:10.790527+0000","last_peered":"2024-04-24T08:20:10.790527+0000","last_clean":"2024-04-24T08:20:10.790527+0000","last_became_active":"2024-04-24T08:20:09.774870+0000","last_became_peered":"2024-04-24T08:20:09.774870+0000","last_unstale":"2024-04-24T08:20:10.790527+0000","last_undegraded":"2024-04-24T08:20:10.790527+0000","last_fullsized":"2024-04-24T08:20:10.790527+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:11:38.016468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787733+0000","last_change":"2024-04-24T08:20:08.768692+0000","last_active":"2024-04-24T08:20:10.787733+0000","last_peered":"2024-04-24T08:20:10.787733+0000","last_clean":"2024-04-24T08:20:10.787733+0000","last_became_active":"2024-04-24T08:20:08.768352+0000","last_became_peered":"2024-04-24T08:20:08.768352+0000","last_unstale":"2024-04-24T08:20:10.787733+0000","last_undegraded":"2024-04-24T08:20:10.787733+0000","last_fullsized":"2024-04-24T08:20:10.787733+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:52:34.264489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898831+0000","last_change":"2024-04-24T08:20:09.799946+0000","last_active":"2024-04-24T08:20:10.898831+0000","last_peered":"2024-04-24T08:20:10.898831+0000","last_clean":"2024-04-24T08:20:10.898831+0000","last_became_active":"2024-04-24T08:20:09.799680+0000","last_became_peered":"2024-04-24T08:20:09.799680+0000","last_unstale":"2024-04-24T08:20:10.898831+0000","last_undegraded":"2024-04-24T08:20:10.898831+0000","last_fullsized":"2024-04-24T08:20:10.898831+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:47.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899134+0000","last_change":"2024-04-24T08:20:08.767491+0000","last_active":"2024-04-24T08:20:10.899134+0000","last_peered":"2024-04-24T08:20:10.899134+0000","last_clean":"2024-04-24T08:20:10.899134+0000","last_became_active":"2024-04-24T08:20:08.767200+0000","last_became_peered":"2024-04-24T08:20:08.767200+0000","last_unstale":"2024-04-24T08:20:10.899134+0000","last_undegraded":"2024-04-24T08:20:10.899134+0000","last_fullsized":"2024-04-24T08:20:10.899134+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:29:39.437886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898703+0000","last_change":"2024-04-24T08:20:09.784394+0000","last_active":"2024-04-24T08:20:10.898703+0000","last_peered":"2024-04-24T08:20:10.898703+0000","last_clean":"2024-04-24T08:20:10.898703+0000","last_became_active":"2024-04-24T08:20:09.784125+0000","last_became_peered":"2024-04-24T08:20:09.784125+0000","last_unstale":"2024-04-24T08:20:10.898703+0000","last_undegraded":"2024-04-24T08:20:10.898703+0000","last_fullsized":"2024-04-24T08:20:10.898703+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:13.818493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"108'3","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896283+0000","last_change":"2024-04-24T08:20:08.771750+0000","last_active":"2024-04-24T08:20:10.896283+0000","last_peered":"2024-04-24T08:20:10.896283+0000","last_clean":"2024-04-24T08:20:10.896283+0000","last_became_active":"2024-04-24T08:20:08.771450+0000","last_became_peered":"2024-04-24T08:20:08.771450+0000","last_unstale":"2024-04-24T08:20:10.896283+0000","last_undegraded":"2024-04-24T08:20:10.896283+0000","last_fullsized":"2024-04-24T08:20:10.896283+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:56.149044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790276+0000","last_change":"2024-04-24T08:20:09.784652+0000","last_active":"2024-04-24T08:20:10.790276+0000","last_peered":"2024-04-24T08:20:10.790276+0000","last_clean":"2024-04-24T08:20:10.790276+0000","last_became_active":"2024-04-24T08:20:09.784367+0000","last_became_peered":"2024-04-24T08:20:09.784367+0000","last_unstale":"2024-04-24T08:20:10.790276+0000","last_undegraded":"2024-04-24T08:20:10.790276+0000","last_fullsized":"2024-04-24T08:20:10.790276+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:56.817425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"108'6","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896458+0000","last_change":"2024-04-24T08:20:08.763054+0000","last_active":"2024-04-24T08:20:10.896458+0000","last_peered":"2024-04-24T08:20:10.896458+0000","last_clean":"2024-04-24T08:20:10.896458+0000","last_became_active":"2024-04-24T08:20:08.762785+0000","last_became_peered":"2024-04-24T08:20:08.762785+0000","last_unstale":"2024-04-24T08:20:10.896458+0000","last_undegraded":"2024-04-24T08:20:10.896458+0000","last_fullsized":"2024-04-24T08:20:10.896458+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:11:02.341594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.900015+0000","last_change":"2024-04-24T08:20:09.775131+0000","last_active":"2024-04-24T08:20:10.900015+0000","last_peered":"2024-04-24T08:20:10.900015+0000","last_clean":"2024-04-24T08:20:10.900015+0000","last_became_active":"2024-04-24T08:20:09.774820+0000","last_became_peered":"2024-04-24T08:20:09.774820+0000","last_unstale":"2024-04-24T08:20:10.900015+0000","last_undegraded":"2024-04-24T08:20:10.900015+0000","last_fullsized":"2024-04-24T08:20:10.900015+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:09:43.173248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898575+0000","last_change":"2024-04-24T08:20:08.771246+0000","last_active":"2024-04-24T08:20:10.898575+0000","last_peered":"2024-04-24T08:20:10.898575+0000","last_clean":"2024-04-24T08:20:10.898575+0000","last_became_active":"2024-04-24T08:20:08.770915+0000","last_became_peered":"2024-04-24T08:20:08.770915+0000","last_unstale":"2024-04-24T08:20:10.898575+0000","last_undegraded":"2024-04-24T08:20:10.898575+0000","last_fullsized":"2024-04-24T08:20:10.898575+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:19:40.821738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897970+0000","last_change":"2024-04-24T08:20:09.787926+0000","last_active":"2024-04-24T08:20:10.897970+0000","last_peered":"2024-04-24T08:20:10.897970+0000","last_clean":"2024-04-24T08:20:10.897970+0000","last_became_active":"2024-04-24T08:20:09.787610+0000","last_became_peered":"2024-04-24T08:20:09.787610+0000","last_unstale":"2024-04-24T08:20:10.897970+0000","last_undegraded":"2024-04-24T08:20:10.897970+0000","last_fullsized":"2024-04-24T08:20:10.897970+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:28:57.166226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898305+0000","last_change":"2024-04-24T08:20:08.770641+0000","last_active":"2024-04-24T08:20:10.898305+0000","last_peered":"2024-04-24T08:20:10.898305+0000","last_clean":"2024-04-24T08:20:10.898305+0000","last_became_active":"2024-04-24T08:20:08.770281+0000","last_became_peered":"2024-04-24T08:20:08.770281+0000","last_unstale":"2024-04-24T08:20:10.898305+0000","last_undegraded":"2024-04-24T08:20:10.898305+0000","last_fullsized":"2024-04-24T08:20:10.898305+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:55:34.854054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898155+0000","last_change":"2024-04-24T08:20:09.789809+0000","last_active":"2024-04-24T08:20:10.898155+0000","last_peered":"2024-04-24T08:20:10.898155+0000","last_clean":"2024-04-24T08:20:10.898155+0000","last_became_active":"2024-04-24T08:20:09.789248+0000","last_became_peered":"2024-04-24T08:20:09.789248+0000","last_unstale":"2024-04-24T08:20:10.898155+0000","last_undegraded":"2024-04-24T08:20:10.898155+0000","last_fullsized":"2024-04-24T08:20:10.898155+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:34.067483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"108'8","reported_seq":22,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.786954+0000","last_change":"2024-04-24T08:20:08.760398+0000","last_active":"2024-04-24T08:20:10.786954+0000","last_peered":"2024-04-24T08:20:10.786954+0000","last_clean":"2024-04-24T08:20:10.786954+0000","last_became_active":"2024-04-24T08:20:08.760075+0000","last_became_peered":"2024-04-24T08:20:08.760075+0000","last_unstale":"2024-04-24T08:20:10.786954+0000","last_undegraded":"2024-04-24T08:20:10.786954+0000","last_fullsized":"2024-04-24T08:20:10.786954+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:20.208658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788203+0000","last_change":"2024-04-24T08:20:09.797624+0000","last_active":"2024-04-24T08:20:10.788203+0000","last_peered":"2024-04-24T08:20:10.788203+0000","last_clean":"2024-04-24T08:20:10.788203+0000","last_became_active":"2024-04-24T08:20:09.796587+0000","last_became_peered":"2024-04-24T08:20:09.796587+0000","last_unstale":"2024-04-24T08:20:10.788203+0000","last_undegraded":"2024-04-24T08:20:10.788203+0000","last_fullsized":"2024-04-24T08:20:10.788203+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:10:52.836766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897475+0000","last_change":"2024-04-24T08:20:08.769979+0000","last_active":"2024-04-24T08:20:10.897475+0000","last_peered":"2024-04-24T08:20:10.897475+0000","last_clean":"2024-04-24T08:20:10.897475+0000","last_became_active":"2024-04-24T08:20:08.769667+0000","last_became_peered":"2024-04-24T08:20:08.769667+0000","last_unstale":"2024-04-24T08:20:10.897475+0000","last_undegraded":"2024-04-24T08:20:10.897475+0000","last_fullsized":"2024-04-24T08:20:10.897475+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:33.050301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899848+0000","last_change":"2024-04-24T08:20:09.795745+0000","last_active":"2024-04-24T08:20:10.899848+0000","last_peered":"2024-04-24T08:20:10.899848+0000","last_clean":"2024-04-24T08:20:10.899848+0000","last_became_active":"2024-04-24T08:20:09.795459+0000","last_became_peered":"2024-04-24T08:20:09.795459+0000","last_unstale":"2024-04-24T08:20:10.899848+0000","last_undegraded":"2024-04-24T08:20:10.899848+0000","last_fullsized":"2024-04-24T08:20:10.899848+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:07:38.079929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.788129+0000","last_change":"2024-04-24T08:20:08.762510+0000","last_active":"2024-04-24T08:20:10.788129+0000","last_peered":"2024-04-24T08:20:10.788129+0000","last_clean":"2024-04-24T08:20:10.788129+0000","last_became_active":"2024-04-24T08:20:08.762168+0000","last_became_peered":"2024-04-24T08:20:08.762168+0000","last_unstale":"2024-04-24T08:20:10.788129+0000","last_undegraded":"2024-04-24T08:20:10.788129+0000","last_fullsized":"2024-04-24T08:20:10.788129+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:05:22.039965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898404+0000","last_change":"2024-04-24T08:20:09.800369+0000","last_active":"2024-04-24T08:20:10.898404+0000","last_peered":"2024-04-24T08:20:10.898404+0000","last_clean":"2024-04-24T08:20:10.898404+0000","last_became_active":"2024-04-24T08:20:09.799535+0000","last_became_peered":"2024-04-24T08:20:09.799535+0000","last_unstale":"2024-04-24T08:20:10.898404+0000","last_undegraded":"2024-04-24T08:20:10.898404+0000","last_fullsized":"2024-04-24T08:20:10.898404+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:25.726242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898135+0000","last_change":"2024-04-24T08:20:08.761477+0000","last_active":"2024-04-24T08:20:10.898135+0000","last_peered":"2024-04-24T08:20:10.898135+0000","last_clean":"2024-04-24T08:20:10.898135+0000","last_became_active":"2024-04-24T08:20:08.759812+0000","last_became_peered":"2024-04-24T08:20:08.759812+0000","last_unstale":"2024-04-24T08:20:10.898135+0000","last_undegraded":"2024-04-24T08:20:10.898135+0000","last_fullsized":"2024-04-24T08:20:10.898135+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:27:26.888390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897806+0000","last_change":"2024-04-24T08:20:09.795821+0000","last_active":"2024-04-24T08:20:10.897806+0000","last_peered":"2024-04-24T08:20:10.897806+0000","last_clean":"2024-04-24T08:20:10.897806+0000","last_became_active":"2024-04-24T08:20:09.795409+0000","last_became_peered":"2024-04-24T08:20:09.795409+0000","last_unstale":"2024-04-24T08:20:10.897806+0000","last_undegraded":"2024-04-24T08:20:10.897806+0000","last_fullsized":"2024-04-24T08:20:10.897806+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:47.526466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790642+0000","last_change":"2024-04-24T08:20:08.767284+0000","last_active":"2024-04-24T08:20:10.790642+0000","last_peered":"2024-04-24T08:20:10.790642+0000","last_clean":"2024-04-24T08:20:10.790642+0000","last_became_active":"2024-04-24T08:20:08.766880+0000","last_became_peered":"2024-04-24T08:20:08.766880+0000","last_unstale":"2024-04-24T08:20:10.790642+0000","last_undegraded":"2024-04-24T08:20:10.790642+0000","last_fullsized":"2024-04-24T08:20:10.790642+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:08.415565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896838+0000","last_change":"2024-04-24T08:20:09.787635+0000","last_active":"2024-04-24T08:20:10.896838+0000","last_peered":"2024-04-24T08:20:10.896838+0000","last_clean":"2024-04-24T08:20:10.896838+0000","last_became_active":"2024-04-24T08:20:09.787270+0000","last_became_peered":"2024-04-24T08:20:09.787270+0000","last_unstale":"2024-04-24T08:20:10.896838+0000","last_undegraded":"2024-04-24T08:20:10.896838+0000","last_fullsized":"2024-04-24T08:20:10.896838+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:23:51.545972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899062+0000","last_change":"2024-04-24T08:20:08.773991+0000","last_active":"2024-04-24T08:20:10.899062+0000","last_peered":"2024-04-24T08:20:10.899062+0000","last_clean":"2024-04-24T08:20:10.899062+0000","last_became_active":"2024-04-24T08:20:08.773703+0000","last_became_peered":"2024-04-24T08:20:08.773703+0000","last_unstale":"2024-04-24T08:20:10.899062+0000","last_undegraded":"2024-04-24T08:20:10.899062+0000","last_fullsized":"2024-04-24T08:20:10.899062+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:13:02.412872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895443+0000","last_change":"2024-04-24T08:20:09.797747+0000","last_active":"2024-04-24T08:20:10.895443+0000","last_peered":"2024-04-24T08:20:10.895443+0000","last_clean":"2024-04-24T08:20:10.895443+0000","last_became_active":"2024-04-24T08:20:09.797468+0000","last_became_peered":"2024-04-24T08:20:09.797468+0000","last_unstale":"2024-04-24T08:20:10.895443+0000","last_undegraded":"2024-04-24T08:20:10.895443+0000","last_fullsized":"2024-04-24T08:20:10.895443+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:10:12.534666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.786965+0000","last_change":"2024-04-24T08:20:08.769543+0000","last_active":"2024-04-24T08:20:10.786965+0000","last_peered":"2024-04-24T08:20:10.786965+0000","last_clean":"2024-04-24T08:20:10.786965+0000","last_became_active":"2024-04-24T08:20:08.767940+0000","last_became_peered":"2024-04-24T08:20:08.767940+0000","last_unstale":"2024-04-24T08:20:10.786965+0000","last_undegraded":"2024-04-24T08:20:10.786965+0000","last_fullsized":"2024-04-24T08:20:10.786965+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:16:28.019646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897242+0000","last_change":"2024-04-24T08:20:09.790241+0000","last_active":"2024-04-24T08:20:10.897242+0000","last_peered":"2024-04-24T08:20:10.897242+0000","last_clean":"2024-04-24T08:20:10.897242+0000","last_became_active":"2024-04-24T08:20:09.789926+0000","last_became_peered":"2024-04-24T08:20:09.789926+0000","last_unstale":"2024-04-24T08:20:10.897242+0000","last_undegraded":"2024-04-24T08:20:10.897242+0000","last_fullsized":"2024-04-24T08:20:10.897242+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:52:35.867500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897241+0000","last_change":"2024-04-24T08:20:08.769318+0000","last_active":"2024-04-24T08:20:10.897241+0000","last_peered":"2024-04-24T08:20:10.897241+0000","last_clean":"2024-04-24T08:20:10.897241+0000","last_became_active":"2024-04-24T08:20:08.769050+0000","last_became_peered":"2024-04-24T08:20:08.769050+0000","last_unstale":"2024-04-24T08:20:10.897241+0000","last_undegraded":"2024-04-24T08:20:10.897241+0000","last_fullsized":"2024-04-24T08:20:10.897241+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:10:51.819583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898432+0000","last_change":"2024-04-24T08:20:09.792723+0000","last_active":"2024-04-24T08:20:10.898432+0000","last_peered":"2024-04-24T08:20:10.898432+0000","last_clean":"2024-04-24T08:20:10.898432+0000","last_became_active":"2024-04-24T08:20:09.792405+0000","last_became_peered":"2024-04-24T08:20:09.792405+0000","last_unstale":"2024-04-24T08:20:10.898432+0000","last_undegraded":"2024-04-24T08:20:10.898432+0000","last_fullsized":"2024-04-24T08:20:10.898432+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:51:48.090822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895133+0000","last_change":"2024-04-24T08:20:08.767252+0000","last_active":"2024-04-24T08:20:10.895133+0000","last_peered":"2024-04-24T08:20:10.895133+0000","last_clean":"2024-04-24T08:20:10.895133+0000","last_became_active":"2024-04-24T08:20:08.766235+0000","last_became_peered":"2024-04-24T08:20:08.766235+0000","last_unstale":"2024-04-24T08:20:10.895133+0000","last_undegraded":"2024-04-24T08:20:10.895133+0000","last_fullsized":"2024-04-24T08:20:10.895133+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:12.801310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899088+0000","last_change":"2024-04-24T08:20:09.799590+0000","last_active":"2024-04-24T08:20:10.899088+0000","last_peered":"2024-04-24T08:20:10.899088+0000","last_clean":"2024-04-24T08:20:10.899088+0000","last_became_active":"2024-04-24T08:20:09.799255+0000","last_became_peered":"2024-04-24T08:20:09.799255+0000","last_unstale":"2024-04-24T08:20:10.899088+0000","last_undegraded":"2024-04-24T08:20:10.899088+0000","last_fullsized":"2024-04-24T08:20:10.899088+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:27:29.553044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"108'2","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899509+0000","last_change":"2024-04-24T08:20:08.774854+0000","last_active":"2024-04-24T08:20:10.899509+0000","last_peered":"2024-04-24T08:20:10.899509+0000","last_clean":"2024-04-24T08:20:10.899509+0000","last_became_active":"2024-04-24T08:20:08.773896+0000","last_became_peered":"2024-04-24T08:20:08.773896+0000","last_unstale":"2024-04-24T08:20:10.899509+0000","last_undegraded":"2024-04-24T08:20:10.899509+0000","last_fullsized":"2024-04-24T08:20:10.899509+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:12:47.591419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897513+0000","last_change":"2024-04-24T08:20:09.793423+0000","last_active":"2024-04-24T08:20:10.897513+0000","last_peered":"2024-04-24T08:20:10.897513+0000","last_clean":"2024-04-24T08:20:10.897513+0000","last_became_active":"2024-04-24T08:20:09.793148+0000","last_became_peered":"2024-04-24T08:20:09.793148+0000","last_unstale":"2024-04-24T08:20:10.897513+0000","last_undegraded":"2024-04-24T08:20:10.897513+0000","last_fullsized":"2024-04-24T08:20:10.897513+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:39:47.452847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"108'1","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.899236+0000","last_change":"2024-04-24T08:20:08.763470+0000","last_active":"2024-04-24T08:20:10.899236+0000","last_peered":"2024-04-24T08:20:10.899236+0000","last_clean":"2024-04-24T08:20:10.899236+0000","last_became_active":"2024-04-24T08:20:08.762839+0000","last_became_peered":"2024-04-24T08:20:08.762839+0000","last_unstale":"2024-04-24T08:20:10.899236+0000","last_undegraded":"2024-04-24T08:20:10.899236+0000","last_fullsized":"2024-04-24T08:20:10.899236+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:17:20.634892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896284+0000","last_change":"2024-04-24T08:20:09.788152+0000","last_active":"2024-04-24T08:20:10.896284+0000","last_peered":"2024-04-24T08:20:10.896284+0000","last_clean":"2024-04-24T08:20:10.896284+0000","last_became_active":"2024-04-24T08:20:09.787847+0000","last_became_peered":"2024-04-24T08:20:09.787847+0000","last_unstale":"2024-04-24T08:20:10.896284+0000","last_undegraded":"2024-04-24T08:20:10.896284+0000","last_fullsized":"2024-04-24T08:20:10.896284+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:59:19.646168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.896389+0000","last_change":"2024-04-24T08:20:08.761630+0000","last_active":"2024-04-24T08:20:10.896389+0000","last_peered":"2024-04-24T08:20:10.896389+0000","last_clean":"2024-04-24T08:20:10.896389+0000","last_became_active":"2024-04-24T08:20:08.761355+0000","last_became_peered":"2024-04-24T08:20:08.761355+0000","last_unstale":"2024-04-24T08:20:10.896389+0000","last_undegraded":"2024-04-24T08:20:10.896389+0000","last_fullsized":"2024-04-24T08:20:10.896389+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:24.709059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897649+0000","last_change":"2024-04-24T08:20:09.800757+0000","last_active":"2024-04-24T08:20:10.897649+0000","last_peered":"2024-04-24T08:20:10.897649+0000","last_clean":"2024-04-24T08:20:10.897649+0000","last_became_active":"2024-04-24T08:20:09.800064+0000","last_became_peered":"2024-04-24T08:20:09.800064+0000","last_unstale":"2024-04-24T08:20:10.897649+0000","last_undegraded":"2024-04-24T08:20:10.897649+0000","last_fullsized":"2024-04-24T08:20:10.897649+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:22:13.818493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898874+0000","last_change":"2024-04-24T08:20:08.770890+0000","last_active":"2024-04-24T08:20:10.898874+0000","last_peered":"2024-04-24T08:20:10.898874+0000","last_clean":"2024-04-24T08:20:10.898874+0000","last_became_active":"2024-04-24T08:20:08.770587+0000","last_became_peered":"2024-04-24T08:20:08.770587+0000","last_unstale":"2024-04-24T08:20:10.898874+0000","last_undegraded":"2024-04-24T08:20:10.898874+0000","last_fullsized":"2024-04-24T08:20:10.898874+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:24:17.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790143+0000","last_change":"2024-04-24T08:20:09.773964+0000","last_active":"2024-04-24T08:20:10.790143+0000","last_peered":"2024-04-24T08:20:10.790143+0000","last_clean":"2024-04-24T08:20:10.790143+0000","last_became_active":"2024-04-24T08:20:09.773691+0000","last_became_peered":"2024-04-24T08:20:09.773691+0000","last_unstale":"2024-04-24T08:20:10.790143+0000","last_undegraded":"2024-04-24T08:20:10.790143+0000","last_fullsized":"2024-04-24T08:20:10.790143+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:54:47.526466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790239+0000","last_change":"2024-04-24T08:20:08.757528+0000","last_active":"2024-04-24T08:20:10.790239+0000","last_peered":"2024-04-24T08:20:10.790239+0000","last_clean":"2024-04-24T08:20:10.790239+0000","last_became_active":"2024-04-24T08:20:08.757135+0000","last_became_peered":"2024-04-24T08:20:08.757135+0000","last_unstale":"2024-04-24T08:20:10.790239+0000","last_undegraded":"2024-04-24T08:20:10.790239+0000","last_fullsized":"2024-04-24T08:20:10.790239+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:35:49.116094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.790951+0000","last_change":"2024-04-24T08:20:09.793813+0000","last_active":"2024-04-24T08:20:10.790951+0000","last_peered":"2024-04-24T08:20:10.790951+0000","last_clean":"2024-04-24T08:20:10.790951+0000","last_became_active":"2024-04-24T08:20:09.793103+0000","last_became_peered":"2024-04-24T08:20:09.793103+0000","last_unstale":"2024-04-24T08:20:10.790951+0000","last_undegraded":"2024-04-24T08:20:10.790951+0000","last_fullsized":"2024-04-24T08:20:10.790951+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:25.726242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789590+0000","last_change":"2024-04-24T08:20:08.766731+0000","last_active":"2024-04-24T08:20:10.789590+0000","last_peered":"2024-04-24T08:20:10.789590+0000","last_clean":"2024-04-24T08:20:10.789590+0000","last_became_active":"2024-04-24T08:20:08.766406+0000","last_became_peered":"2024-04-24T08:20:08.766406+0000","last_unstale":"2024-04-24T08:20:10.789590+0000","last_undegraded":"2024-04-24T08:20:10.789590+0000","last_fullsized":"2024-04-24T08:20:10.789590+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:37:01.856004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897684+0000","last_change":"2024-04-24T08:20:09.789425+0000","last_active":"2024-04-24T08:20:10.897684+0000","last_peered":"2024-04-24T08:20:10.897684+0000","last_clean":"2024-04-24T08:20:10.897684+0000","last_became_active":"2024-04-24T08:20:09.788643+0000","last_became_peered":"2024-04-24T08:20:09.788643+0000","last_unstale":"2024-04-24T08:20:10.897684+0000","last_undegraded":"2024-04-24T08:20:10.897684+0000","last_fullsized":"2024-04-24T08:20:10.897684+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:27:27.905572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.787134+0000","last_change":"2024-04-24T08:20:08.768906+0000","last_active":"2024-04-24T08:20:10.787134+0000","last_peered":"2024-04-24T08:20:10.787134+0000","last_clean":"2024-04-24T08:20:10.787134+0000","last_became_active":"2024-04-24T08:20:08.768593+0000","last_became_peered":"2024-04-24T08:20:08.768593+0000","last_unstale":"2024-04-24T08:20:10.787134+0000","last_undegraded":"2024-04-24T08:20:10.787134+0000","last_fullsized":"2024-04-24T08:20:10.787134+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:47:22.864945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.897148+0000","last_change":"2024-04-24T08:20:09.795062+0000","last_active":"2024-04-24T08:20:10.897148+0000","last_peered":"2024-04-24T08:20:10.897148+0000","last_clean":"2024-04-24T08:20:10.897148+0000","last_became_active":"2024-04-24T08:20:09.794686+0000","last_became_peered":"2024-04-24T08:20:09.794686+0000","last_unstale":"2024-04-24T08:20:10.897148+0000","last_undegraded":"2024-04-24T08:20:10.897148+0000","last_fullsized":"2024-04-24T08:20:10.897148+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:08:08.087220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.895229+0000","last_change":"2024-04-24T08:20:08.767036+0000","last_active":"2024-04-24T08:20:10.895229+0000","last_peered":"2024-04-24T08:20:10.895229+0000","last_clean":"2024-04-24T08:20:10.895229+0000","last_became_active":"2024-04-24T08:20:08.765832+0000","last_became_peered":"2024-04-24T08:20:08.765832+0000","last_unstale":"2024-04-24T08:20:10.895229+0000","last_undegraded":"2024-04-24T08:20:10.895229+0000","last_fullsized":"2024-04-24T08:20:10.895229+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:22:46.310731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898004+0000","last_change":"2024-04-24T08:20:09.787121+0000","last_active":"2024-04-24T08:20:10.898004+0000","last_peered":"2024-04-24T08:20:10.898004+0000","last_clean":"2024-04-24T08:20:10.898004+0000","last_became_active":"2024-04-24T08:20:09.786467+0000","last_became_peered":"2024-04-24T08:20:09.786467+0000","last_unstale":"2024-04-24T08:20:10.898004+0000","last_undegraded":"2024-04-24T08:20:10.898004+0000","last_fullsized":"2024-04-24T08:20:10.898004+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:08.725174+0000","last_clean_scrub_stamp":"2024-04-24T08:20:08.725174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:13:25.726242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.789569+0000","last_change":"2024-04-24T08:20:08.767872+0000","last_active":"2024-04-24T08:20:10.789569+0000","last_peered":"2024-04-24T08:20:10.789569+0000","last_clean":"2024-04-24T08:20:10.789569+0000","last_became_active":"2024-04-24T08:20:08.767538+0000","last_became_peered":"2024-04-24T08:20:08.767538+0000","last_unstale":"2024-04-24T08:20:10.789569+0000","last_undegraded":"2024-04-24T08:20:10.789569+0000","last_fullsized":"2024-04-24T08:20:10.789569+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:20:07.707992+0000","last_clean_scrub_stamp":"2024-04-24T08:20:07.707992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:10:51.819583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-04-24T08:20:10.898446+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:20:10.898446+0000","last_peered":"2024-04-24T08:20:10.898446+0000","last_clean":"2024-04-24T08:20:10.898446+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:20:10.898446+0000","last_undegraded":"2024-04-24T08:20:10.898446+0000","last_fullsized":"2024-04-24T08:20:10.898446+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574999,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33100,"kb_used_data":1912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338740,"statfs":{"total":96636764160,"available":96602869760,"internally_reserved":0,"allocated":1957888,"data_stored":1180021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574999,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32508,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339332,"statfs":{"total":96636764160,"available":96603475968,"internally_reserved":0,"allocated":1359872,"data_stored":589107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574997,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1363968,"data_stored":589641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574996,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":589037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574997,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32520,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339320,"statfs":{"total":96636764160,"available":96603463680,"internally_reserved":0,"allocated":1372160,"data_stored":590719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574998,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1355776,"data_stored":589037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574998,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1363968,"data_stored":589161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574997,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33112,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338728,"statfs":{"total":96636764160,"available":96602857472,"internally_reserved":0,"allocated":1970176,"data_stored":1181129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:20:19.672 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T08:20:19.673 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-24T08:20:19.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.919+0000 7ff89c400640 1 Processor -- start 2024-04-24T08:20:19.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.919+0000 7ff89c400640 1 -- start start 2024-04-24T08:20:19.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.920+0000 7ff89c400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8940b3320 0x7ff8940d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.920+0000 7ff89c400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8940d0840 0x7ff8940b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.920+0000 7ff89c400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8940b3780 con 0x7ff8941591d0 2024-04-24T08:20:19.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.920+0000 7ff89c400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff8940d0cb0 con 0x7ff8940d0840 2024-04-24T08:20:19.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.920+0000 7ff89c400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff89415beb0 con 0x7ff8940b3320 2024-04-24T08:20:19.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.921+0000 7ff89a400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8940d0840 0x7ff8940b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.921+0000 7ff89a400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff8940d0840 0x7ff8940b49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40800/0 (socket says 172.21.15.69:40800) 2024-04-24T08:20:19.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.921+0000 7ff89a400640 1 -- 172.21.15.69:0/1391395322 learned_addr learned my addr 172.21.15.69:0/1391395322 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:19.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.921+0000 7ff899000640 1 --2- 172.21.15.69:0/1391395322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8940b3320 0x7ff8940d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.921+0000 7ff89a400640 1 -- 172.21.15.69:0/1391395322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8940b3320 msgr2=0x7ff8940d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:19.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.921+0000 7ff89a400640 1 --2- 172.21.15.69:0/1391395322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8940b3320 0x7ff8940d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.921+0000 7ff89a400640 1 -- 172.21.15.69:0/1391395322 >> v1:172.21.15.121:6789/0 conn(0x7ff8941591d0 legacy=0x7ff89406f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:19.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.921+0000 7ff89a400640 1 -- 172.21.15.69:0/1391395322 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff89415c190 con 0x7ff8940d0840 2024-04-24T08:20:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.922+0000 7ff89a400640 1 --2- 172.21.15.69:0/1391395322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8940d0840 0x7ff8940b49b0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7ff8940d1450 tx=0x7ff88c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8947b4d5641d0f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff893e00640 1 -- 172.21.15.69:0/1391395322 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c036070 con 0x7ff8940d0840 2024-04-24T08:20:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff893e00640 1 -- 172.21.15.69:0/1391395322 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff88c031600 con 0x7ff8940d0840 2024-04-24T08:20:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff893e00640 1 -- 172.21.15.69:0/1391395322 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c0319d0 con 0x7ff8940d0840 2024-04-24T08:20:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff89c400640 1 -- 172.21.15.69:0/1391395322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8940d0840 msgr2=0x7ff8940b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff89c400640 1 --2- 172.21.15.69:0/1391395322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8940d0840 0x7ff8940b49b0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7ff8940d1450 tx=0x7ff88c02a6a0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff89c400640 1 -- 172.21.15.69:0/1391395322 shutdown_connections 2024-04-24T08:20:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff89c400640 1 --2- 172.21.15.69:0/1391395322 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8940d0840 0x7ff8940b49b0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff89c400640 1 --2- 172.21.15.69:0/1391395322 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8940b3320 0x7ff8940d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff89c400640 1 -- 172.21.15.69:0/1391395322 >> 172.21.15.69:0/1391395322 conn(0x7ff89406ba20 msgr2=0x7ff89406be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:19.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff89c400640 1 -- 172.21.15.69:0/1391395322 shutdown_connections 2024-04-24T08:20:19.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.923+0000 7ff89c400640 1 -- 172.21.15.69:0/1391395322 wait complete. 2024-04-24T08:20:19.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.925+0000 7ff89c400640 1 Processor -- start 2024-04-24T08:20:19.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.925+0000 7ff89c400640 1 -- start start 2024-04-24T08:20:19.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.926+0000 7ff89c400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8940b3320 0x7ff89414ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.926+0000 7ff89c400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8940d0840 0x7ff8940b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.926+0000 7ff89c400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8941591d0 0x7ff8940b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.926+0000 7ff89c400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff89415ca30 con 0x7ff8941591d0 2024-04-24T08:20:19.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.926+0000 7ff89c400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff89415c750 con 0x7ff8940d0840 2024-04-24T08:20:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.926+0000 7ff89c400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff89415bb60 con 0x7ff8940b3320 2024-04-24T08:20:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.926+0000 7ff89a400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8941591d0 0x7ff8940b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.926+0000 7ff89a400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8941591d0 0x7ff8940b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40804/0 (socket says 172.21.15.69:40804) 2024-04-24T08:20:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.926+0000 7ff89a400640 1 -- 172.21.15.69:0/2772905510 learned_addr learned my addr 172.21.15.69:0/2772905510 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff899000640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8940d0840 0x7ff8940b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff89a400640 1 -- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8940b3320 msgr2=0x7ff89414ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff89a400640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8940b3320 0x7ff89414ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff89a400640 1 -- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8940d0840 msgr2=0x7ff8940b15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff89a400640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8940d0840 0x7ff8940b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:19.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff89a400640 1 -- 172.21.15.69:0/2772905510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8940afd30 con 0x7ff8941591d0 2024-04-24T08:20:19.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff89a400640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8941591d0 0x7ff8940b0af0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7ff88c02ac90 tx=0x7ff88c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:19.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff891600640 1 -- 172.21.15.69:0/2772905510 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c036050 con 0x7ff8941591d0 2024-04-24T08:20:19.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff891600640 1 -- 172.21.15.69:0/2772905510 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff88c042070 con 0x7ff8941591d0 2024-04-24T08:20:19.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff891600640 1 -- 172.21.15.69:0/2772905510 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c009510 con 0x7ff8941591d0 2024-04-24T08:20:19.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.927+0000 7ff89c400640 1 -- 172.21.15.69:0/2772905510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8940b0010 con 0x7ff8941591d0 2024-04-24T08:20:19.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.928+0000 7ff89c400640 1 -- 172.21.15.69:0/2772905510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8940b1b00 con 0x7ff8941591d0 2024-04-24T08:20:19.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.929+0000 7ff89c400640 1 -- 172.21.15.69:0/2772905510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff894075e30 con 0x7ff8941591d0 2024-04-24T08:20:19.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.930+0000 7ff891600640 1 -- 172.21.15.69:0/2772905510 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff88c007bd0 con 0x7ff8941591d0 2024-04-24T08:20:19.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.931+0000 7ff891600640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff87406c920 0x7ff87406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:19.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.931+0000 7ff891600640 1 -- 172.21.15.69:0/2772905510 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7ff88c0aca70 con 0x7ff8941591d0 2024-04-24T08:20:19.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.931+0000 7ff899a00640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff87406c920 0x7ff87406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:19.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.933+0000 7ff899a00640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff87406c920 0x7ff87406ede0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff888006fd0 tx=0x7ff888006bc0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:19.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:19.935+0000 7ff891600640 1 -- 172.21.15.69:0/2772905510 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff88c03a360 con 0x7ff8941591d0 2024-04-24T08:20:20.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.206+0000 7ff89c400640 1 -- 172.21.15.69:0/2772905510 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7ff8940d00e0 con 0x7ff8941591d0 2024-04-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.212+0000 7ff891600640 1 -- 172.21.15.69:0/2772905510 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) ==== 190+0+147 (secure 0 0 0) 0x7ff88c07ae90 con 0x7ff8941591d0 2024-04-24T08:20:20.212 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-04-24T08:20:20.213 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQC1vShmPqbJIBAA1VYcOIQwejOX1QUBcOiDuw== 2024-04-24T08:20:20.213 INFO:teuthology.orchestra.run.smithi069.stdout: caps mds = "allow" 2024-04-24T08:20:20.213 INFO:teuthology.orchestra.run.smithi069.stdout: caps mon = "allow r" 2024-04-24T08:20:20.213 INFO:teuthology.orchestra.run.smithi069.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-24T08:20:20.213 INFO:teuthology.orchestra.run.smithi069.stderr:updated caps for client.0 2024-04-24T08:20:20.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 -- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff87406c920 msgr2=0x7ff87406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:20.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff87406c920 0x7ff87406ede0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff888006fd0 tx=0x7ff888006bc0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 -- 172.21.15.69:0/2772905510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8941591d0 msgr2=0x7ff8940b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:20.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8941591d0 0x7ff8940b0af0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7ff88c02ac90 tx=0x7ff88c004a40 comp rx=0 tx=0).stop 2024-04-24T08:20:20.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 -- 172.21.15.69:0/2772905510 shutdown_connections 2024-04-24T08:20:20.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff87406c920 0x7ff87406ede0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8941591d0 0x7ff8940b0af0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8940d0840 0x7ff8940b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 --2- 172.21.15.69:0/2772905510 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8940b3320 0x7ff89414ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 -- 172.21.15.69:0/2772905510 >> 172.21.15.69:0/2772905510 conn(0x7ff89406ba20 msgr2=0x7ff89415f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:20.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.215+0000 7ff89c400640 1 -- 172.21.15.69:0/2772905510 shutdown_connections 2024-04-24T08:20:20.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.216+0000 7ff89c400640 1 -- 172.21.15.69:0/2772905510 wait complete. 2024-04-24T08:20:20.237 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:20.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.429+0000 7f5b06400640 1 Processor -- start 2024-04-24T08:20:20.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.430+0000 7f5b06400640 1 -- start start 2024-04-24T08:20:20.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.431+0000 7f5b06400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f5b000cf9a0 0x7f5b00162250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:20.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.431+0000 7f5b06400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b00070860 con 0x7f5b0015f320 2024-04-24T08:20:20.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.431+0000 7f5b06400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b00163180 con 0x7f5b000710e0 2024-04-24T08:20:20.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.431+0000 7f5b06400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f5b0015ee90 con 0x7f5b000cf9a0 2024-04-24T08:20:20.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.432+0000 7f5b05a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f5b0015f320 0x7f5b0006ba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56962/0 (socket says 172.21.15.69:56962) 2024-04-24T08:20:20.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.432+0000 7f5b05a00640 1 -- 172.21.15.69:0/1537298981 learned_addr learned my addr 172.21.15.69:0/1537298981 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:20.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.432+0000 7f5b05000640 1 --2- 172.21.15.69:0/1537298981 >> v2:172.21.15.121:3300/0 conn(0x7f5b000cf9a0 0x7f5b00162250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:20.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.432+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2046003750 0 0) 0x7f5b00070860 con 0x7f5b0015f320 2024-04-24T08:20:20.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.432+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ae4003520 con 0x7f5b0015f320 2024-04-24T08:20:20.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2883575335 0 0) 0x7f5ae4003520 con 0x7f5b0015f320 2024-04-24T08:20:20.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 >> v2:172.21.15.121:3300/0 conn(0x7f5b000cf9a0 msgr2=0x7f5b00162250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:20.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 --2- 172.21.15.69:0/1537298981 >> v2:172.21.15.121:3300/0 conn(0x7f5b000cf9a0 0x7f5b00162250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 >> v1:172.21.15.121:6789/0 conn(0x7f5b000710e0 legacy=0x7f5b00072af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:20.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b001628e0 con 0x7f5b0015f320 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f5af80019f0 con 0x7f5b0015f320 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 >> v1:172.21.15.69:6789/0 conn(0x7f5b0015f320 legacy=0x7f5b0006ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae4003a00 0x7f5ae4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4004800 0x7f5ae4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ae40051c0 0x7f5ae400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0015ee90 con 0x7f5ae4003a00 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5ae400c0c0 con 0x7f5ae4004800 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5ae400c2c0 con 0x7f5ae40051c0 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.433+0000 7f5b05000640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae4003a00 0x7f5ae4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.434+0000 7f5affe00640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4004800 0x7f5ae4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.434+0000 7f5b05000640 1 -- 172.21.15.69:0/1537298981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ae40051c0 msgr2=0x7f5ae400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.434+0000 7f5b05000640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5ae40051c0 0x7f5ae400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.434+0000 7f5b05000640 1 -- 172.21.15.69:0/1537298981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4004800 msgr2=0x7f5ae4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.434+0000 7f5b05000640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4004800 0x7f5ae4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.434+0000 7f5b05000640 1 -- 172.21.15.69:0/1537298981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae4003520 con 0x7f5ae4003a00 2024-04-24T08:20:20.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.434+0000 7f5b05a00640 1 -- 172.21.15.69:0/1537298981 reap_dead start 2024-04-24T08:20:20.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.434+0000 7f5affe00640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4004800 0x7f5ae4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.435+0000 7f5b05000640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae4003a00 0x7f5ae4003e20 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009ac0 tx=0x7f5ae4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4e7c3a566674 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.435+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0036070 con 0x7f5ae4003a00 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.435+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af0031400 con 0x7f5ae4003a00 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.435+0000 7f5aff400640 1 -- 172.21.15.69:0/1537298981 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0009690 con 0x7f5ae4003a00 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.436+0000 7f5b06400640 1 -- 172.21.15.69:0/1537298981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae4003a00 msgr2=0x7f5ae4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.436+0000 7f5b06400640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae4003a00 0x7f5ae4003e20 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009ac0 tx=0x7f5ae4005830 comp rx=0 tx=0).stop 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.436+0000 7f5b06400640 1 -- 172.21.15.69:0/1537298981 shutdown_connections 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.436+0000 7f5b06400640 1 --2- 172.21.15.69:0/1537298981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5ae4003a00 0x7f5ae4003e20 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.436+0000 7f5b06400640 1 -- 172.21.15.69:0/1537298981 >> 172.21.15.69:0/1537298981 conn(0x7f5b000d12c0 msgr2=0x7f5b000d16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.436+0000 7f5b06400640 1 -- 172.21.15.69:0/1537298981 shutdown_connections 2024-04-24T08:20:20.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.436+0000 7f5b06400640 1 -- 172.21.15.69:0/1537298981 wait complete. 2024-04-24T08:20:20.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.438+0000 7f5b06400640 1 Processor -- start 2024-04-24T08:20:20.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.438+0000 7f5b06400640 1 -- start start 2024-04-24T08:20:20.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5b06400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4003a00 0x7f5b000c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:20.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5b06400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b00154b30 0x7f5b000c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:20.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5b06400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b00154110 0x7f5b001607c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:20.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5b06400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5b00162250 con 0x7f5b00154b30 2024-04-24T08:20:20.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5b06400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f5b00162ea0 con 0x7f5ae4003a00 2024-04-24T08:20:20.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5b06400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f5b001623d0 con 0x7f5b00154110 2024-04-24T08:20:20.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5affe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b00154b30 0x7f5b000c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:20.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5affe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b00154b30 0x7f5b000c96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40824/0 (socket says 172.21.15.69:40824) 2024-04-24T08:20:20.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5affe00640 1 -- 172.21.15.69:0/1201225061 learned_addr learned my addr 172.21.15.69:0/1201225061 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:20.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5b05000640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4003a00 0x7f5b000c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:20.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5affe00640 1 -- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b00154110 msgr2=0x7f5b001607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:20.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5affe00640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b00154110 0x7f5b001607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5affe00640 1 -- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4003a00 msgr2=0x7f5b000c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:20.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.439+0000 7f5affe00640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4003a00 0x7f5b000c9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.440+0000 7f5affe00640 1 -- 172.21.15.69:0/1201225061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b00156380 con 0x7f5b00154b30 2024-04-24T08:20:20.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.440+0000 7f5affe00640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b00154b30 0x7f5b000c96a0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00d480 tx=0x7f5aec00d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:20.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.440+0000 7f5afcc00640 1 -- 172.21.15.69:0/1201225061 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec005480 con 0x7f5b00154b30 2024-04-24T08:20:20.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.440+0000 7f5afcc00640 1 -- 172.21.15.69:0/1201225061 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aec017070 con 0x7f5b00154b30 2024-04-24T08:20:20.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.440+0000 7f5afcc00640 1 -- 172.21.15.69:0/1201225061 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aec003830 con 0x7f5b00154b30 2024-04-24T08:20:20.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.440+0000 7f5b06400640 1 -- 172.21.15.69:0/1201225061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b00156660 con 0x7f5b00154b30 2024-04-24T08:20:20.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.440+0000 7f5b06400640 1 -- 172.21.15.69:0/1201225061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b00156c20 con 0x7f5b00154b30 2024-04-24T08:20:20.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.441+0000 7f5b06400640 1 -- 172.21.15.69:0/1201225061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b00075d50 con 0x7f5b00154b30 2024-04-24T08:20:20.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.442+0000 7f5afcc00640 1 -- 172.21.15.69:0/1201225061 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5aec0039d0 con 0x7f5b00154b30 2024-04-24T08:20:20.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.443+0000 7f5afcc00640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ae006c8f0 0x7f5ae006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:20.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.443+0000 7f5afcc00640 1 -- 172.21.15.69:0/1201225061 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f5aec08b9a0 con 0x7f5b00154b30 2024-04-24T08:20:20.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.444+0000 7f5b05000640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ae006c8f0 0x7f5ae006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:20.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.445+0000 7f5b05000640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ae006c8f0 0x7f5ae006edb0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5af0033000 tx=0x7f5af0008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:20.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.448+0000 7f5afcc00640 1 -- 172.21.15.69:0/1201225061 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5aec059e80 con 0x7f5b00154b30 2024-04-24T08:20:20.688 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.687+0000 7f5b06400640 1 -- 172.21.15.69:0/1201225061 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b000703a0 con 0x7f5b00154b30 2024-04-24T08:20:20.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.688+0000 7f5afcc00640 1 -- 172.21.15.69:0/1201225061 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5186 (secure 0 0 0) 0x7f5aec059780 con 0x7f5b00154b30 2024-04-24T08:20:20.689 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:20.690 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":99,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":9285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/2194021008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2194021008},{"type":"v1","addr":"172.21.15.69:6835","nonce":2194021008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:20:12.288476+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9288},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9288":{"gid":9288,"name":"b","rank":0,"incarnation":96,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6836/1515102940","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":1515102940},{"type":"v1","addr":"172.21.15.121:6836","nonce":1515102940}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9288,"qdb_cluster":[9288]},"id":8}]} 2024-04-24T08:20:20.690 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 99 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 -- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ae006c8f0 msgr2=0x7f5ae006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ae006c8f0 0x7f5ae006edb0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f5af0033000 tx=0x7f5af0008030 comp rx=0 tx=0).stop 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 -- 172.21.15.69:0/1201225061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b00154b30 msgr2=0x7f5b000c96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b00154b30 0x7f5b000c96a0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00d480 tx=0x7f5aec00d940 comp rx=0 tx=0).stop 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 -- 172.21.15.69:0/1201225061 shutdown_connections 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f5ae006c8f0 0x7f5ae006edb0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b00154b30 0x7f5b000c96a0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5b00154110 0x7f5b001607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 --2- 172.21.15.69:0/1201225061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ae4003a00 0x7f5b000c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 -- 172.21.15.69:0/1201225061 >> 172.21.15.69:0/1201225061 conn(0x7f5b000d12c0 msgr2=0x7f5b00161ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 -- 172.21.15.69:0/1201225061 shutdown_connections 2024-04-24T08:20:20.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:20.691+0000 7f5b06400640 1 -- 172.21.15.69:0/1201225061 wait complete. 2024-04-24T08:20:21.112 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:20:09.204175+0000', 'modified': '2024-04-24T08:20:12.288476+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9288}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9288': {'gid': 9288, 'name': 'b', 'rank': 0, 'incarnation': 96, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6836/1515102940', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6834', 'nonce': 1515102940}, {'type': 'v1', 'addr': '172.21.15.121:6836', 'nonce': 1515102940}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9288, 'qdb_cluster': [9288]} 2024-04-24T08:20:21.112 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:20:21.112 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T08:20:21.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.321+0000 7f04a9000640 1 Processor -- start 2024-04-24T08:20:21.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.322+0000 7f04a9000640 1 -- start start 2024-04-24T08:20:21.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.323+0000 7f04a9000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f04a406f2d0 0x7f04a406f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.323+0000 7f04a9000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a40cfbe0 0x7f04a40d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.323+0000 7f04a9000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04a40b46e0 con 0x7f04a415b490 2024-04-24T08:20:21.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.323+0000 7f04a9000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f04a406fc70 con 0x7f04a406f2d0 2024-04-24T08:20:21.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.323+0000 7f04a9000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04a40b5890 con 0x7f04a40cfbe0 2024-04-24T08:20:21.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a1e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f04a415b490 0x7f04a40b5180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56968/0 (socket says 172.21.15.69:56968) 2024-04-24T08:20:21.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a1e00640 1 -- 172.21.15.69:0/4256536645 learned_addr learned my addr 172.21.15.69:0/4256536645 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:21.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1907205446 0 0) 0x7f04a40b46e0 con 0x7f04a415b490 2024-04-24T08:20:21.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0488003520 con 0x7f04a415b490 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a2800640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a40cfbe0 0x7f04a40d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a1400640 1 --2- 172.21.15.69:0/4256536645 >> v2:172.21.15.121:3300/0 conn(0x7f04a406f2d0 0x7f04a406f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1170975325 0 0) 0x7f0488003520 con 0x7f04a415b490 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a40cfbe0 msgr2=0x7f04a40d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a0a00640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a40cfbe0 0x7f04a40d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 >> v2:172.21.15.121:3300/0 conn(0x7f04a406f2d0 msgr2=0x7f04a406f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a0a00640 1 --2- 172.21.15.69:0/4256536645 >> v2:172.21.15.121:3300/0 conn(0x7f04a406f2d0 0x7f04a406f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a40d0960 con 0x7f04a415b490 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.324+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f0494001670 con 0x7f04a415b490 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 >> v1:172.21.15.69:6789/0 conn(0x7f04a415b490 legacy=0x7f04a40b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a0a00640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04880039b0 0x7f0488003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a0a00640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 0x7f0488004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a0a00640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a40cfbe0 0x7f04880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04a406fc70 con 0x7f0488004800 2024-04-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04a40b5890 con 0x7f04880039b0 2024-04-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f048800ba70 con 0x7f04a40cfbe0 2024-04-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a1400640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 0x7f0488004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a1e00640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04880039b0 0x7f0488003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a1400640 1 -- 172.21.15.69:0/4256536645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a40cfbe0 msgr2=0x7f04880051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a1400640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a40cfbe0 0x7f04880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a1400640 1 -- 172.21.15.69:0/4256536645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04880039b0 msgr2=0x7f0488003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a1400640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04880039b0 0x7f0488003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.325+0000 7f04a1400640 1 -- 172.21.15.69:0/4256536645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0488003520 con 0x7f0488004800 2024-04-24T08:20:21.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.326+0000 7f04a1400640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 0x7f0488004c80 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f049000ba00 tx=0x7f0488005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1956579c81f869d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.327+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490030070 con 0x7f0488004800 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.327+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049002adb0 con 0x7f0488004800 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.327+0000 7f04a0a00640 1 -- 172.21.15.69:0/4256536645 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049000b600 con 0x7f0488004800 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.327+0000 7f04a9000640 1 -- 172.21.15.69:0/4256536645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 msgr2=0x7f0488004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.327+0000 7f04a9000640 1 --2- 172.21.15.69:0/4256536645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 0x7f0488004c80 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f049000ba00 tx=0x7f0488005830 comp rx=0 tx=0).stop 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.327+0000 7f04a2800640 1 -- 172.21.15.69:0/4256536645 reap_dead start 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.327+0000 7f04a9000640 1 -- 172.21.15.69:0/4256536645 shutdown_connections 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.327+0000 7f04a9000640 1 -- 172.21.15.69:0/4256536645 >> 172.21.15.69:0/4256536645 conn(0x7f04a4067d90 msgr2=0x7f04a406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.327+0000 7f04a9000640 1 -- 172.21.15.69:0/4256536645 shutdown_connections 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.328+0000 7f04a9000640 1 -- 172.21.15.69:0/4256536645 wait complete. 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.329+0000 7f04a9000640 1 Processor -- start 2024-04-24T08:20:21.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.329+0000 7f04a9000640 1 -- start start 2024-04-24T08:20:21.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.330+0000 7f04a9000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 0x7f04a414e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.330+0000 7f04a9000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a4151550 0x7f04a414c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.330+0000 7f04a9000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a40af050 0x7f04a4150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.330+0000 7f04a9000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04a40d0c40 con 0x7f0488004800 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.330+0000 7f04a9000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04a40b4c60 con 0x7f04a40af050 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.330+0000 7f04a9000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04a415b870 con 0x7f04a4151550 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 0x7f04a414e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 0x7f04a414e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40834/0 (socket says 172.21.15.69:40834) 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a1e00640 1 -- 172.21.15.69:0/1602261039 learned_addr learned my addr 172.21.15.69:0/1602261039 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a1400640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a4151550 0x7f04a414c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a2800640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a40af050 0x7f04a4150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a1e00640 1 -- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a4151550 msgr2=0x7f04a414c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a1e00640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a4151550 0x7f04a414c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a1e00640 1 -- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a40af050 msgr2=0x7f04a4150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a1e00640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a40af050 0x7f04a4150970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a1e00640 1 -- 172.21.15.69:0/1602261039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a414cfe0 con 0x7f0488004800 2024-04-24T08:20:21.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.331+0000 7f04a1e00640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 0x7f04a414e080 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f049400d800 tx=0x7f049400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.332+0000 7f049a000640 1 -- 172.21.15.69:0/1602261039 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0494019070 con 0x7f0488004800 2024-04-24T08:20:21.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.332+0000 7f049a000640 1 -- 172.21.15.69:0/1602261039 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04940074e0 con 0x7f0488004800 2024-04-24T08:20:21.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.332+0000 7f04a9000640 1 -- 172.21.15.69:0/1602261039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a414d2c0 con 0x7f0488004800 2024-04-24T08:20:21.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.332+0000 7f049a000640 1 -- 172.21.15.69:0/1602261039 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04940049e0 con 0x7f0488004800 2024-04-24T08:20:21.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.332+0000 7f04a9000640 1 -- 172.21.15.69:0/1602261039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a414d880 con 0x7f0488004800 2024-04-24T08:20:21.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.334+0000 7f049a000640 1 -- 172.21.15.69:0/1602261039 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f049401c070 con 0x7f0488004800 2024-04-24T08:20:21.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.336+0000 7f049a000640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 0x7f048406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.336+0000 7f049a000640 1 -- 172.21.15.69:0/1602261039 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f049408b740 con 0x7f0488004800 2024-04-24T08:20:21.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.336+0000 7f04abdc4740 1 Processor -- start 2024-04-24T08:20:21.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.336+0000 7f04abdc4740 1 -- start start 2024-04-24T08:20:21.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.336+0000 7f04a1400640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 0x7f048406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.337+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589de6a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.337+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589de99c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.337+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589de9a190 0x55589de6efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.337+0000 7f04abdc4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55589ddd4b10 con 0x55589deb77b0 2024-04-24T08:20:21.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.337+0000 7f04abdc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55589dd7b2b0 con 0x55589de9a8d0 2024-04-24T08:20:21.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.337+0000 7f04abdc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55589ddd0ad0 con 0x55589de9a190 2024-04-24T08:20:21.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.337+0000 7f04a1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589de6a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589de6a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40846/0 (socket says 172.21.15.69:40846) 2024-04-24T08:20:21.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1e00640 1 -- 172.21.15.69:0/983736011 learned_addr learned my addr 172.21.15.69:0/983736011 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:21.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1400640 1 --2- 172.21.15.69:0/983736011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589de99c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a2800640 1 --2- 172.21.15.69:0/983736011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589de9a190 0x55589de6efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1e00640 1 -- 172.21.15.69:0/983736011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589de9a190 msgr2=0x55589de6efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1e00640 1 --2- 172.21.15.69:0/983736011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589de9a190 0x55589de6efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1e00640 1 -- 172.21.15.69:0/983736011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 msgr2=0x55589de99c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1e00640 1 --2- 172.21.15.69:0/983736011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589de99c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1e00640 1 -- 172.21.15.69:0/983736011 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55589dae72d0 con 0x55589deb77b0 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1400640 1 --2- 172.21.15.69:0/983736011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589de99c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1400640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 0x7f048406edb0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f04880057e0 tx=0x7f0490030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f04a1e00640 1 --2- 172.21.15.69:0/983736011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589de6a140 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f0494061a80 tx=0x7f04940555e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafe8d3f4fe02673 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f048ea00640 1 -- 172.21.15.69:0/983736011 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0494061fb0 con 0x55589deb77b0 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f048ea00640 1 -- 172.21.15.69:0/983736011 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0494057640 con 0x55589deb77b0 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.338+0000 7f048ea00640 1 -- 172.21.15.69:0/983736011 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0494004b80 con 0x55589deb77b0 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.339+0000 7f04abdc4740 1 -- 172.21.15.69:0/983736011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 msgr2=0x55589de6a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.339+0000 7f04abdc4740 1 --2- 172.21.15.69:0/983736011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589de6a140 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f0494061a80 tx=0x7f04940555e0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.339+0000 7f04abdc4740 1 -- 172.21.15.69:0/983736011 shutdown_connections 2024-04-24T08:20:21.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.339+0000 7f04abdc4740 1 --2- 172.21.15.69:0/983736011 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589de9a190 0x55589de6efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.339+0000 7f04abdc4740 1 --2- 172.21.15.69:0/983736011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589de99c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.339+0000 7f04abdc4740 1 --2- 172.21.15.69:0/983736011 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589de6a140 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.339+0000 7f04abdc4740 1 -- 172.21.15.69:0/983736011 >> 172.21.15.69:0/983736011 conn(0x55589de6c3b0 msgr2=0x55589deb4c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:21.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.339+0000 7f04abdc4740 1 -- 172.21.15.69:0/983736011 shutdown_connections 2024-04-24T08:20:21.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.339+0000 7f04abdc4740 1 -- 172.21.15.69:0/983736011 wait complete. 2024-04-24T08:20:21.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.340+0000 7f04abdc4740 1 -- start start 2024-04-24T08:20:21.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.340+0000 7f04abdc4740 1 Processor -- start 2024-04-24T08:20:21.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.341+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589de95860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.341+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589dead7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.341+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589dec5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.341+0000 7f04abdc4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55589de6e560 con 0x55589deb77b0 2024-04-24T08:20:21.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.341+0000 7f04abdc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55589dd8e590 con 0x55589de9a8d0 2024-04-24T08:20:21.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.341+0000 7f04abdc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55589de6d950 con 0x55589dec50f0 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.341+0000 7f04abdc4740 10 client.0 _ll_register_callbacks cb 0x55589dd713a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589dead7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589dead7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40852/0 (socket says 172.21.15.69:40852) 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1400640 1 -- 172.21.15.69:0/4226327801 learned_addr learned my addr 172.21.15.69:0/4226327801 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1e00640 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589de95860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1400640 1 -- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 msgr2=0x55589dec5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1400640 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589dec5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1400640 1 -- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 msgr2=0x55589de95860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1400640 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589de95860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1400640 1 -- 172.21.15.69:0/4226327801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55589dbb16b0 con 0x55589deb77b0 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1e00640 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589de95860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:21.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f04a1400640 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589dead7c0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f0490006b10 tx=0x7f04900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c910445371add95e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.342+0000 7f0483400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f0483400640 1 -- 172.21.15.69:0/4226327801 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490050060 con 0x55589deb77b0 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f0483400640 1 -- 172.21.15.69:0/4226327801 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f049003d8e0 con 0x55589deb77b0 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f04abdc4740 1 -- 172.21.15.69:0/4226327801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0494057640 con 0x55589deb77b0 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f0483400640 1 -- 172.21.15.69:0/4226327801 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f0490056050 con 0x55589deb77b0 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f04abdc4740 10 client.9438 fetch_fsmap learned FSMap version 99 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f04abdc4740 1 -- 172.21.15.69:0/4226327801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x55589deadd00 con 0x55589deb77b0 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f0483400640 1 -- 172.21.15.69:0/4226327801 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0494057640 con 0x55589deb77b0 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f0483400640 1 -- 172.21.15.69:0/4226327801 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f0490059020 con 0x55589deb77b0 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f04abdc4740 10 client.9438 fetch_fsmap finished waiting for FSMap version 99 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f04abdc4740 10 client.9438 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f04abdc4740 20 client.9438 populate_metadata read hostname 'smithi069' 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f04abdc4740 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x55589de9b6f0 0x55589ded3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f04abdc4740 4 client.9438 mds_command: new command op to 9288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:21.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.343+0000 7f04abdc4740 1 -- 172.21.15.69:0/4226327801 --> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55589dde2530 con 0x55589de9b6f0 2024-04-24T08:20:21.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.344+0000 7f04a1e00640 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x55589de9b6f0 0x55589ded3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.345+0000 7f04a1e00640 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x55589de9b6f0 0x55589ded3a30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=502dc85e1ac55e03 server_cookie=539d673a375422c7 in_seq=0 out_seq=0 2024-04-24T08:20:21.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.345+0000 7f0483400640 10 client.9438 ms_handle_connect on v2:172.21.15.121:6834/1515102940 2024-04-24T08:20:21.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f0483400640 1 -- 172.21.15.69:0/4226327801 <== mds.0 v2:172.21.15.121:6834/1515102940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55589dde2530 con 0x55589de9b6f0 2024-04-24T08:20:21.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f0483400640 10 client.9438 handle_command_reply: tid=0 2024-04-24T08:20:21.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 client.9438 shutdown 2024-04-24T08:20:21.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 -- 172.21.15.69:0/4226327801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 msgr2=0x55589dead7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589dead7c0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f0490006b10 tx=0x7f04900046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 -- 172.21.15.69:0/4226327801 shutdown_connections 2024-04-24T08:20:21.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x55589de9b6f0 0x55589ded3a30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589dead7c0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589dec5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 --2- 172.21.15.69:0/4226327801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589de95860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f0483400640 0 client.9438 ms_handle_reset on v2:172.21.15.121:6834/1515102940 2024-04-24T08:20:21.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 -- 172.21.15.69:0/4226327801 >> 172.21.15.69:0/4226327801 conn(0x55589de9a190 msgr2=0x55589de698c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:21.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 -- 172.21.15.69:0/4226327801 shutdown_connections 2024-04-24T08:20:21.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 1 -- 172.21.15.69:0/4226327801 wait complete. 2024-04-24T08:20:21.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.347+0000 7f04abdc4740 20 client.9438 trim_cache size 0 max 16384 2024-04-24T08:20:21.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.366+0000 7f04abdc4740 1 Processor -- start 2024-04-24T08:20:21.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.366+0000 7f04abdc4740 1 -- start start 2024-04-24T08:20:21.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.367+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589ded90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.367+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589deb5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.367+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589deb6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.367+0000 7f04abdc4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04a4201430 con 0x55589deb77b0 2024-04-24T08:20:21.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.367+0000 7f04abdc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04a4200470 con 0x55589de9a8d0 2024-04-24T08:20:21.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.367+0000 7f04abdc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04a40ae1d0 con 0x55589dec50f0 2024-04-24T08:20:21.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a2800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589deb6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a2800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589deb6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:52058/0 (socket says 172.21.15.69:52058) 2024-04-24T08:20:21.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a2800640 1 -- 172.21.15.69:0/998091836 learned_addr learned my addr 172.21.15.69:0/998091836 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:21.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a1400640 1 --2- 172.21.15.69:0/998091836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589deb5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a1e00640 1 --2- 172.21.15.69:0/998091836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589ded90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a1400640 1 -- 172.21.15.69:0/998091836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 msgr2=0x55589deb6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a1400640 1 --2- 172.21.15.69:0/998091836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589deb6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a1400640 1 -- 172.21.15.69:0/998091836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 msgr2=0x55589ded90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a1400640 1 --2- 172.21.15.69:0/998091836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589ded90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a1400640 1 -- 172.21.15.69:0/998091836 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55589de710f0 con 0x55589deb77b0 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.368+0000 7f04a1400640 1 --2- 172.21.15.69:0/998091836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589deb5af0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f0490040fd0 tx=0x7f0490042050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b8f92158f7a556 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f048e000640 1 -- 172.21.15.69:0/998091836 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049005f070 con 0x55589deb77b0 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f048e000640 1 -- 172.21.15.69:0/998091836 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049003dde0 con 0x55589deb77b0 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f048e000640 1 -- 172.21.15.69:0/998091836 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049003db30 con 0x55589deb77b0 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f04abdc4740 1 -- 172.21.15.69:0/998091836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 msgr2=0x55589deb5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f04abdc4740 1 --2- 172.21.15.69:0/998091836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589deb5af0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f0490040fd0 tx=0x7f0490042050 comp rx=0 tx=0).stop 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f04abdc4740 1 -- 172.21.15.69:0/998091836 shutdown_connections 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f04abdc4740 1 --2- 172.21.15.69:0/998091836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589deb77b0 0x55589deb5af0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f04abdc4740 1 --2- 172.21.15.69:0/998091836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589deb6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f04abdc4740 1 --2- 172.21.15.69:0/998091836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589de9a8d0 0x55589ded90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f04abdc4740 1 -- 172.21.15.69:0/998091836 >> 172.21.15.69:0/998091836 conn(0x55589de9a190 msgr2=0x55589deb6fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.369+0000 7f04abdc4740 1 -- 172.21.15.69:0/998091836 shutdown_connections 2024-04-24T08:20:21.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.370+0000 7f04abdc4740 1 -- 172.21.15.69:0/998091836 wait complete. 2024-04-24T08:20:21.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.370+0000 7f04abdc4740 1 -- start start 2024-04-24T08:20:21.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.370+0000 7f04abdc4740 1 Processor -- start 2024-04-24T08:20:21.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.371+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589de9a8d0 0x55589df35200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.371+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589deb77b0 0x55589ded8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.371+0000 7f04abdc4740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589deb5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.371+0000 7f04abdc4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55589dec5b50 con 0x55589de9a8d0 2024-04-24T08:20:21.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.371+0000 7f04abdc4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f04a41ffe70 con 0x55589deb77b0 2024-04-24T08:20:21.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.371+0000 7f04abdc4740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55589deb6d60 con 0x55589dec50f0 2024-04-24T08:20:21.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.371+0000 7f04abdc4740 10 client.0 _ll_register_callbacks cb 0x7f0488004700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:21.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.371+0000 7f04a1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589de9a8d0 0x55589df35200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589de9a8d0 0x55589df35200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40874/0 (socket says 172.21.15.69:40874) 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a1e00640 1 -- 172.21.15.69:0/2263869323 learned_addr learned my addr 172.21.15.69:0/2263869323 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a1400640 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589deb77b0 0x55589ded8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a2800640 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589deb5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a1e00640 1 -- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 msgr2=0x55589deb5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a1e00640 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589deb5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a1e00640 1 -- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589deb77b0 msgr2=0x55589ded8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a1e00640 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589deb77b0 0x55589ded8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a1e00640 1 -- 172.21.15.69:0/2263869323 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55589dec5980 con 0x55589de9a8d0 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a1400640 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589deb77b0 0x55589ded8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a2800640 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589deb5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04a1e00640 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589de9a8d0 0x55589df35200 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f049400dfd0 tx=0x7f0494001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea08b4f01b3197f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f0483400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:20:21.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f0483400640 1 -- 172.21.15.69:0/2263869323 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04940559b0 con 0x55589de9a8d0 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f0483400640 1 -- 172.21.15.69:0/2263869323 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f0494004b80 con 0x55589de9a8d0 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.372+0000 7f04abdc4740 1 -- 172.21.15.69:0/2263869323 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f049003dde0 con 0x55589de9a8d0 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f0483400640 1 -- 172.21.15.69:0/2263869323 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f0494055b50 con 0x55589de9a8d0 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f04abdc4740 10 client.9444 fetch_fsmap learned FSMap version 99 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f04abdc4740 1 -- 172.21.15.69:0/2263869323 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x55589ded8900 con 0x55589de9a8d0 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f0483400640 1 -- 172.21.15.69:0/2263869323 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049003dde0 con 0x55589de9a8d0 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f0483400640 1 -- 172.21.15.69:0/2263869323 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f0494071bb0 con 0x55589de9a8d0 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f04abdc4740 10 client.9444 fetch_fsmap finished waiting for FSMap version 99 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f04abdc4740 10 client.9444 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f04abdc4740 20 client.9444 populate_metadata read hostname 'smithi069' 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f04abdc4740 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x55589de94270 0x55589df37bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f04abdc4740 4 client.9444 mds_command: new command op to 9288 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.373+0000 7f04abdc4740 1 -- 172.21.15.69:0/2263869323 --> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] -- command(tid 0: {"prefix": "status"}) -- 0x55589dddc3a0 con 0x55589de94270 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.374+0000 7f04a1400640 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x55589de94270 0x55589df37bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.374+0000 7f04a1400640 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x55589de94270 0x55589df37bd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d30c5e7dd5f13365 server_cookie=1d36b1a86eb8f3d3 in_seq=0 out_seq=0 2024-04-24T08:20:21.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.374+0000 7f0483400640 10 client.9444 ms_handle_connect on v2:172.21.15.121:6834/1515102940 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f0483400640 1 -- 172.21.15.69:0/2263869323 <== mds.0 v2:172.21.15.121:6834/1515102940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x55589dddc3a0 con 0x55589de94270 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f0483400640 10 client.9444 handle_command_reply: tid=0 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 client.9444 shutdown 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 -- 172.21.15.69:0/2263869323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589de9a8d0 msgr2=0x55589df35200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589de9a8d0 0x55589df35200 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f049400dfd0 tx=0x7f0494001a40 comp rx=0 tx=0).stop 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 -- 172.21.15.69:0/2263869323 shutdown_connections 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x55589de94270 0x55589df37bd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55589dec50f0 0x55589deb5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55589deb77b0 0x55589ded8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 --2- 172.21.15.69:0/2263869323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55589de9a8d0 0x55589df35200 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f0483400640 0 client.9444 ms_handle_reset on v2:172.21.15.121:6834/1515102940 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 -- 172.21.15.69:0/2263869323 >> 172.21.15.69:0/2263869323 conn(0x55589de9a190 msgr2=0x55589deb3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 -- 172.21.15.69:0/2263869323 shutdown_connections 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 1 -- 172.21.15.69:0/2263869323 wait complete. 2024-04-24T08:20:21.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.375+0000 7f04abdc4740 20 client.9444 trim_cache size 0 max 16384 2024-04-24T08:20:21.380 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:20:21.380 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:20:21.380 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:20:21.380 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 9288, 2024-04-24T08:20:21.380 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:20:21.380 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:20:21.380 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:20:21.380 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 11.616421159, 2024-04-24T08:20:21.380 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 99, 2024-04-24T08:20:21.380 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 109, 2024-04-24T08:20:21.381 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 109, 2024-04-24T08:20:21.381 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 17.074672305 2024-04-24T08:20:21.381 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:20:21.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 -- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 msgr2=0x7f048406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 0x7f048406edb0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f04880057e0 tx=0x7f0490030000 comp rx=0 tx=0).stop 2024-04-24T08:20:21.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 -- 172.21.15.69:0/1602261039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 msgr2=0x7f04a414e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 0x7f04a414e080 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f049400d800 tx=0x7f049400dcc0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 -- 172.21.15.69:0/1602261039 shutdown_connections 2024-04-24T08:20:21.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f048406c8f0 0x7f048406edb0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f04a40af050 0x7f04a4150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f04a4151550 0x7f04a414c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 --2- 172.21.15.69:0/1602261039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0488004800 0x7f04a414e080 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 -- 172.21.15.69:0/1602261039 >> 172.21.15.69:0/1602261039 conn(0x7f04a4067d90 msgr2=0x7f04a4072600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:21.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 -- 172.21.15.69:0/1602261039 shutdown_connections 2024-04-24T08:20:21.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.376+0000 7f04a9000640 1 -- 172.21.15.69:0/1602261039 wait complete. 2024-04-24T08:20:21.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:20:21.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.575+0000 7ff172e00640 1 Processor -- start 2024-04-24T08:20:21.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.575+0000 7ff172e00640 1 -- start start 2024-04-24T08:20:21.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.576+0000 7ff172e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff16c156c60 0x7ff16c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.576+0000 7ff172e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff16c0ad7e0 0x7ff16c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.576+0000 7ff172e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff16c15d660 con 0x7ff16c0b30f0 2024-04-24T08:20:21.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.576+0000 7ff172e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff16c06dc30 con 0x7ff16c156c60 2024-04-24T08:20:21.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.576+0000 7ff172e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff16c06df10 con 0x7ff16c0ad7e0 2024-04-24T08:20:21.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.577+0000 7ff171000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff16c0b30f0 0x7ff16c0ad0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56974/0 (socket says 172.21.15.69:56974) 2024-04-24T08:20:21.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.577+0000 7ff171000640 1 -- 172.21.15.69:0/3762866897 learned_addr learned my addr 172.21.15.69:0/3762866897 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:21.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.577+0000 7ff171a00640 1 --2- 172.21.15.69:0/3762866897 >> v2:172.21.15.69:3300/0 conn(0x7ff16c156c60 0x7ff16c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.577+0000 7ff172400640 1 --2- 172.21.15.69:0/3762866897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff16c0ad7e0 0x7ff16c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.577+0000 7ff171a00640 1 -- 172.21.15.69:0/3762866897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff16c0ad7e0 msgr2=0x7ff16c15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:21.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.577+0000 7ff171a00640 1 --2- 172.21.15.69:0/3762866897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff16c0ad7e0 0x7ff16c15d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.577+0000 7ff171a00640 1 -- 172.21.15.69:0/3762866897 >> v1:172.21.15.69:6789/0 conn(0x7ff16c0b30f0 legacy=0x7ff16c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.577+0000 7ff171a00640 1 -- 172.21.15.69:0/3762866897 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16c06e1f0 con 0x7ff16c156c60 2024-04-24T08:20:21.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.578+0000 7ff171a00640 1 --2- 172.21.15.69:0/3762866897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 0x7ff16c15cac0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7ff1600097a0 tx=0x7ff16002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ec4ac4e892d560 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.579+0000 7ff16be00640 1 -- 172.21.15.69:0/3762866897 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160037070 con 0x7ff16c156c60 2024-04-24T08:20:21.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.579+0000 7ff16be00640 1 -- 172.21.15.69:0/3762866897 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff160032440 con 0x7ff16c156c60 2024-04-24T08:20:21.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.579+0000 7ff16be00640 1 -- 172.21.15.69:0/3762866897 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160032810 con 0x7ff16c156c60 2024-04-24T08:20:21.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.579+0000 7ff172e00640 1 -- 172.21.15.69:0/3762866897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 msgr2=0x7ff16c15cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.579+0000 7ff172e00640 1 --2- 172.21.15.69:0/3762866897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 0x7ff16c15cac0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7ff1600097a0 tx=0x7ff16002a180 comp rx=0 tx=0).stop 2024-04-24T08:20:21.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.579+0000 7ff172e00640 1 -- 172.21.15.69:0/3762866897 shutdown_connections 2024-04-24T08:20:21.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.579+0000 7ff172e00640 1 --2- 172.21.15.69:0/3762866897 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff16c0ad7e0 0x7ff16c15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.580+0000 7ff172e00640 1 --2- 172.21.15.69:0/3762866897 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 0x7ff16c15cac0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.580+0000 7ff172e00640 1 -- 172.21.15.69:0/3762866897 >> 172.21.15.69:0/3762866897 conn(0x7ff16c0d0160 msgr2=0x7ff16c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:21.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.580+0000 7ff172e00640 1 -- 172.21.15.69:0/3762866897 shutdown_connections 2024-04-24T08:20:21.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.580+0000 7ff172e00640 1 -- 172.21.15.69:0/3762866897 wait complete. 2024-04-24T08:20:21.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.581+0000 7ff172e00640 1 Processor -- start 2024-04-24T08:20:21.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.581+0000 7ff172e00640 1 -- start start 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.582+0000 7ff172e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff16c0ad7e0 0x7ff16c1f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.582+0000 7ff172e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff16c0b30f0 0x7ff16c0735c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.582+0000 7ff172e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 0x7ff16c15c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.582+0000 7ff172e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff16c06e710 con 0x7ff16c156c60 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.582+0000 7ff172e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff16c15d940 con 0x7ff16c0b30f0 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.582+0000 7ff172e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff16c06e890 con 0x7ff16c0ad7e0 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.583+0000 7ff172400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 0x7ff16c15c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.583+0000 7ff172400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 0x7ff16c15c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40898/0 (socket says 172.21.15.69:40898) 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.583+0000 7ff172400640 1 -- 172.21.15.69:0/37465096 learned_addr learned my addr 172.21.15.69:0/37465096 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.583+0000 7ff171000640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff16c0b30f0 0x7ff16c0735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.583+0000 7ff172400640 1 -- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff16c0ad7e0 msgr2=0x7ff16c1f4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.583+0000 7ff172400640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff16c0ad7e0 0x7ff16c1f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.583+0000 7ff172400640 1 -- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff16c0b30f0 msgr2=0x7ff16c0735c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.583+0000 7ff172400640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff16c0b30f0 0x7ff16c0735c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.583+0000 7ff172400640 1 -- 172.21.15.69:0/37465096 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff16c0720c0 con 0x7ff16c156c60 2024-04-24T08:20:21.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.583+0000 7ff172400640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 0x7ff16c15c950 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7ff164007e60 tx=0x7ff16400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.584+0000 7ff169600640 1 -- 172.21.15.69:0/37465096 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164014620 con 0x7ff16c156c60 2024-04-24T08:20:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.584+0000 7ff169600640 1 -- 172.21.15.69:0/37465096 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff164019070 con 0x7ff16c156c60 2024-04-24T08:20:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.584+0000 7ff172e00640 1 -- 172.21.15.69:0/37465096 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff16c0723a0 con 0x7ff16c156c60 2024-04-24T08:20:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.584+0000 7ff169600640 1 -- 172.21.15.69:0/37465096 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff164014a00 con 0x7ff16c156c60 2024-04-24T08:20:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.584+0000 7ff172e00640 1 -- 172.21.15.69:0/37465096 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff16c072960 con 0x7ff16c156c60 2024-04-24T08:20:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.585+0000 7ff172e00640 1 -- 172.21.15.69:0/37465096 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff16c075d50 con 0x7ff16c156c60 2024-04-24T08:20:21.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.586+0000 7ff169600640 1 -- 172.21.15.69:0/37465096 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff164014ba0 con 0x7ff16c156c60 2024-04-24T08:20:21.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.587+0000 7ff169600640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff14c06c8f0 0x7ff14c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:21.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.587+0000 7ff169600640 1 -- 172.21.15.69:0/37465096 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7ff16408c510 con 0x7ff16c156c60 2024-04-24T08:20:21.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.588+0000 7ff171a00640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff14c06c8f0 0x7ff14c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:21.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.589+0000 7ff171a00640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff14c06c8f0 0x7ff14c06edb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff160034000 tx=0x7ff16002c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:21.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.592+0000 7ff169600640 1 -- 172.21.15.69:0/37465096 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff16405a8a0 con 0x7ff16c156c60 2024-04-24T08:20:21.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.829+0000 7ff172e00640 1 -- 172.21.15.69:0/37465096 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff16c06f9e0 con 0x7ff16c156c60 2024-04-24T08:20:21.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.829+0000 7ff169600640 1 -- 172.21.15.69:0/37465096 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) ==== 53+0+73 (secure 0 0 0) 0x7ff16405a1a0 con 0x7ff16c156c60 2024-04-24T08:20:21.830 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:20:21.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 -- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff14c06c8f0 msgr2=0x7ff14c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff14c06c8f0 0x7ff14c06edb0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7ff160034000 tx=0x7ff16002c000 comp rx=0 tx=0).stop 2024-04-24T08:20:21.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 -- 172.21.15.69:0/37465096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 msgr2=0x7ff16c15c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:21.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 0x7ff16c15c950 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7ff164007e60 tx=0x7ff16400e4b0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 -- 172.21.15.69:0/37465096 shutdown_connections 2024-04-24T08:20:21.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff14c06c8f0 0x7ff14c06edb0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff16c156c60 0x7ff16c15c950 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff16c0b30f0 0x7ff16c0735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 --2- 172.21.15.69:0/37465096 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff16c0ad7e0 0x7ff16c1f4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:21.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 -- 172.21.15.69:0/37465096 >> 172.21.15.69:0/37465096 conn(0x7ff16c0d0160 msgr2=0x7ff16c155c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:21.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 -- 172.21.15.69:0/37465096 shutdown_connections 2024-04-24T08:20:21.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:21.832+0000 7ff172e00640 1 -- 172.21.15.69:0/37465096 wait complete. 2024-04-24T08:20:21.851 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:20:21.851 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:20:21.851 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:20:21.851 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:20:21.851 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:21.851 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:20:21.851 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:20:21.851 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:20:21.852 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:20:21.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:21.852 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:20:21.903 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:20:21.903 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:20:21.903 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:20:21.903 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:21.903 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:20:21.903 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:21.903 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:20:21.903 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 41958sec preferred_lft 41958sec 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:21.904 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:20:21.905 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:21.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:21.906 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:20:21.981 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:20:21.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:21.985 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:20:22.038 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:20:22.038 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:20:22.038 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:20:22.038 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:20:22.038 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:20:22.038 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:20:22.038 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:20:22.038 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:20:22.039 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:20:22.039 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:20:22.039 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:20:22.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:20:22.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:22.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:22.248 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:20:22.293 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:20:22.293 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:20:22.294 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:20:22.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:22.294 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:20:22.414 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:20:22.414 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:20:22.414 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:22.468 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:20:22.649 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:20:22.649 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:22.707 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:20:22.796 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:20:22.852 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:20:22.852 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:20:22.852 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:20:22.852 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:20:22.852 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:20:22.852 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:20:22.852 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:20:22.852 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:20:22.852 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:20:22.853 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:20:22.854 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:20:22.854 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:20:22.854 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:20:22.854 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:20:22.854 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:20:22.854 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:22.854 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:22.854 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:22.854 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:22.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:22.855 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:20:22.905 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:20:22.905 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:20:22.910 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:20:22.976 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:20:23.019 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:20:23.020 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:23.021 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:23.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:23.022 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:20:23.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:20:23.074+0000 7fcb4f00d480 -1 init, newargv = 0x55ab3ce2c450 newargc=15 2024-04-24T08:20:23.075 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[75282]: starting ceph client 2024-04-24T08:20:23.084 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[75282]: starting fuse 2024-04-24T08:20:23.105 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:20:23.105 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:20:23.105 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:20:23.105 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:20:23.105 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:20:23.106 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:20:23.107 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:20:23.107 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:20:23.107 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:20:23.107 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:20:23.107 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:23.107 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:20:23.107 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:23.290 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.75282.asok 2024-04-24T08:20:23.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:23.306 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.75282.asok 2024-04-24T08:20:23.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:23.307 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.75282.asok status 2024-04-24T08:20:23.529 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:20:23.529 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:20:23.529 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:20:23.529 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "75282", 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 9459, 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 9459 2024-04-24T08:20:23.530 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2660563346 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2660563346 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.9459 192.168.0.1:0/2660563346", 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2660563346", 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 99, 2024-04-24T08:20:23.531 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 109, 2024-04-24T08:20:23.532 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:20:23.532 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:20:23.532 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:20:23.532 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:20:23.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:23.544 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:20:23.544 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:23.598 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:20:23.598 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:23.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:23.599 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:23.681 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:23.682 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:23.682 DEBUG:teuthology.orchestra.run.smithi069:> mount_point = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:20:23.682 DEBUG:teuthology.orchestra.run.smithi069:> parent = os.path.join(mount_point, "parent") 2024-04-24T08:20:23.682 DEBUG:teuthology.orchestra.run.smithi069:> os.mkdir(parent) 2024-04-24T08:20:23.682 DEBUG:teuthology.orchestra.run.smithi069:> flushed = os.path.join(parent, "flushed") 2024-04-24T08:20:23.682 DEBUG:teuthology.orchestra.run.smithi069:> os.mkdir(flushed) 2024-04-24T08:20:23.682 DEBUG:teuthology.orchestra.run.smithi069:> for f in ["alpha", "bravo", "charlie"]: 2024-04-24T08:20:23.682 DEBUG:teuthology.orchestra.run.smithi069:> open(os.path.join(flushed, f), '"'"'w'"'"').write(f) 2024-04-24T08:20:23.682 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:23.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:23.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:23.891 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-04-24T08:20:24.025 INFO:teuthology.orchestra.run.smithi069.stdout:./ 2024-04-24T08:20:24.025 INFO:teuthology.orchestra.run.smithi069.stdout:./parent 2024-04-24T08:20:24.025 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed 2024-04-24T08:20:24.025 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed/charlie 2024-04-24T08:20:24.025 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed/alpha 2024-04-24T08:20:24.025 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed/bravo 2024-04-24T08:20:24.027 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:24.027 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:24.027 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:24.027 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:24.027 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-04-24T08:20:24.027 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:24.168 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:20:24.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:24.182 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:24.182 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:24.182 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:24.182 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:24.183 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-04-24T08:20:24.183 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:24.363 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627776 2024-04-24T08:20:24.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:24.376 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:24.376 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:24.376 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:24.376 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:24.376 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-04-24T08:20:24.376 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:24.548 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627777 2024-04-24T08:20:24.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:24.561 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:24.561 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:24.561 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:24.561 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:24.561 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-04-24T08:20:24.561 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:24.745 INFO:teuthology.orchestra.run.smithi069.stdout:1099511628280 2024-04-24T08:20:24.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:24.759 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:24.759 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:24.759 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:24.759 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:24.759 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-04-24T08:20:24.759 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:24.941 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627778 2024-04-24T08:20:24.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:24.954 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:24.954 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:24.954 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:24.954 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:24.954 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-04-24T08:20:24.955 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:25.140 INFO:teuthology.orchestra.run.smithi069.stdout:1099511628279 2024-04-24T08:20:25.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:25.154 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:20:25.154 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:20:25.251 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:25.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.450+0000 7fd33ae00640 1 Processor -- start 2024-04-24T08:20:25.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.450+0000 7fd33ae00640 1 -- start start 2024-04-24T08:20:25.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.451+0000 7fd33ae00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd33415e960 0x7fd334162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.451+0000 7fd33ae00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd33415ed40 con 0x7fd3340cfaf0 2024-04-24T08:20:25.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.451+0000 7fd33ae00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3340cf050 con 0x7fd334162980 2024-04-24T08:20:25.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.451+0000 7fd33ae00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd334162e20 con 0x7fd33415e960 2024-04-24T08:20:25.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.452+0000 7fd33a400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd3340cfaf0 0x7fd3340d2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56978/0 (socket says 172.21.15.69:56978) 2024-04-24T08:20:25.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.452+0000 7fd33a400640 1 -- 172.21.15.69:0/2640897054 learned_addr learned my addr 172.21.15.69:0/2640897054 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:25.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.452+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2121198978 0 0) 0x7fd33415ed40 con 0x7fd3340cfaf0 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.452+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd31c003520 con 0x7fd3340cfaf0 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.452+0000 7fd339a00640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd33415e960 0x7fd334162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 293260576 0 0) 0x7fd31c003520 con 0x7fd3340cfaf0 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd33415e960 msgr2=0x7fd334162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd33415e960 0x7fd334162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 >> v1:172.21.15.121:6789/0 conn(0x7fd334162980 legacy=0x7fd33406cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd334163100 con 0x7fd3340cfaf0 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fd32c0019f0 con 0x7fd3340cfaf0 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 >> v1:172.21.15.69:6789/0 conn(0x7fd3340cfaf0 legacy=0x7fd3340d2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd33415e960 0x7fd31c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 0x7fd31c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd31c004d40 0x7fd31c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd334162e20 con 0x7fd31c004010 2024-04-24T08:20:25.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c00bb80 con 0x7fd31c004d40 2024-04-24T08:20:25.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd31c00bd80 con 0x7fd33415e960 2024-04-24T08:20:25.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd339000640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 0x7fd31c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd33a400640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd31c004d40 0x7fd31c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd339000640 1 -- 172.21.15.69:0/2640897054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd33415e960 msgr2=0x7fd31c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:25.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd339000640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd33415e960 0x7fd31c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.453+0000 7fd339000640 1 -- 172.21.15.69:0/2640897054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd31c004d40 msgr2=0x7fd31c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:25.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.454+0000 7fd339000640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd31c004d40 0x7fd31c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.454+0000 7fd339000640 1 -- 172.21.15.69:0/2640897054 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c003520 con 0x7fd31c004010 2024-04-24T08:20:25.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.455+0000 7fd339000640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 0x7fd31c004800 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fd320001750 tx=0x7fd31c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d314f2c2bc507f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.455+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd320001370 con 0x7fd31c004010 2024-04-24T08:20:25.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.455+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3200052d0 con 0x7fd31c004010 2024-04-24T08:20:25.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.455+0000 7fd333e00640 1 -- 172.21.15.69:0/2640897054 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32002abe0 con 0x7fd31c004010 2024-04-24T08:20:25.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.455+0000 7fd33ae00640 1 -- 172.21.15.69:0/2640897054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 msgr2=0x7fd31c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.455+0000 7fd33ae00640 1 --2- 172.21.15.69:0/2640897054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 0x7fd31c004800 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fd320001750 tx=0x7fd31c005830 comp rx=0 tx=0).stop 2024-04-24T08:20:25.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.456+0000 7fd33a400640 1 -- 172.21.15.69:0/2640897054 reap_dead start 2024-04-24T08:20:25.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.456+0000 7fd33ae00640 1 -- 172.21.15.69:0/2640897054 shutdown_connections 2024-04-24T08:20:25.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.456+0000 7fd33ae00640 1 -- 172.21.15.69:0/2640897054 >> 172.21.15.69:0/2640897054 conn(0x7fd33409d900 msgr2=0x7fd33408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:25.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.456+0000 7fd33ae00640 1 -- 172.21.15.69:0/2640897054 shutdown_connections 2024-04-24T08:20:25.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.456+0000 7fd33ae00640 1 -- 172.21.15.69:0/2640897054 wait complete. 2024-04-24T08:20:25.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.457+0000 7fd33ae00640 1 Processor -- start 2024-04-24T08:20:25.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.458+0000 7fd33ae00640 1 -- start start 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.458+0000 7fd33ae00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 0x7fd334157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.458+0000 7fd33ae00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd334157ed0 0x7fd33406dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.458+0000 7fd33ae00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd33406e170 0x7fd3340ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.458+0000 7fd33ae00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd33406d6c0 con 0x7fd31c004010 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.458+0000 7fd33ae00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd3340cf5d0 con 0x7fd33406e170 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.458+0000 7fd33ae00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd3340b6170 con 0x7fd334157ed0 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.459+0000 7fd339a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 0x7fd334157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.459+0000 7fd339a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 0x7fd334157990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40912/0 (socket says 172.21.15.69:40912) 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.459+0000 7fd339a00640 1 -- 172.21.15.69:0/3065160775 learned_addr learned my addr 172.21.15.69:0/3065160775 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.459+0000 7fd33a400640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd33406e170 0x7fd3340ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.459+0000 7fd339a00640 1 -- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd334157ed0 msgr2=0x7fd33406dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.459+0000 7fd339a00640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd334157ed0 0x7fd33406dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.459+0000 7fd339a00640 1 -- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd33406e170 msgr2=0x7fd3340ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.459+0000 7fd339a00640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd33406e170 0x7fd3340ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.459+0000 7fd339a00640 1 -- 172.21.15.69:0/3065160775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3340c9700 con 0x7fd31c004010 2024-04-24T08:20:25.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.460+0000 7fd339a00640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 0x7fd334157990 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fd32800d620 tx=0x7fd32800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.460+0000 7fd331600640 1 -- 172.21.15.69:0/3065160775 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32800b620 con 0x7fd31c004010 2024-04-24T08:20:25.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.460+0000 7fd331600640 1 -- 172.21.15.69:0/3065160775 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd328019070 con 0x7fd31c004010 2024-04-24T08:20:25.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.460+0000 7fd331600640 1 -- 172.21.15.69:0/3065160775 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32800b9e0 con 0x7fd31c004010 2024-04-24T08:20:25.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.460+0000 7fd33ae00640 1 -- 172.21.15.69:0/3065160775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3340b7290 con 0x7fd31c004010 2024-04-24T08:20:25.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.460+0000 7fd33ae00640 1 -- 172.21.15.69:0/3065160775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3340b7850 con 0x7fd31c004010 2024-04-24T08:20:25.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.461+0000 7fd33ae00640 1 -- 172.21.15.69:0/3065160775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd334075d50 con 0x7fd31c004010 2024-04-24T08:20:25.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.462+0000 7fd331600640 1 -- 172.21.15.69:0/3065160775 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd32800bb80 con 0x7fd31c004010 2024-04-24T08:20:25.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.463+0000 7fd331600640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd31406c920 0x7fd31406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.463+0000 7fd331600640 1 -- 172.21.15.69:0/3065160775 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fd32808bfe0 con 0x7fd31c004010 2024-04-24T08:20:25.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.464+0000 7fd339000640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd31406c920 0x7fd31406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.465+0000 7fd339000640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd31406c920 0x7fd31406ede0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd320032000 tx=0x7fd32002c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.468+0000 7fd331600640 1 -- 172.21.15.69:0/3065160775 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd3340c9700 con 0x7fd31c004010 2024-04-24T08:20:25.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.669+0000 7fd33ae00640 1 -- 172.21.15.69:0/3065160775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd3340d1dc0 con 0x7fd31c004010 2024-04-24T08:20:25.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.671+0000 7fd331600640 1 -- 172.21.15.69:0/3065160775 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+16603 (secure 0 0 0) 0x7fd32805a390 con 0x7fd31c004010 2024-04-24T08:20:25.673 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:25.673 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":109,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:20:10.770819+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 -- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd31406c920 msgr2=0x7fd31406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd31406c920 0x7fd31406ede0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd320032000 tx=0x7fd32002c000 comp rx=0 tx=0).stop 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 -- 172.21.15.69:0/3065160775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 msgr2=0x7fd334157990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 0x7fd334157990 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fd32800d620 tx=0x7fd32800dae0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 -- 172.21.15.69:0/3065160775 shutdown_connections 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd31406c920 0x7fd31406ede0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd33406e170 0x7fd3340ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd334157ed0 0x7fd33406dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 --2- 172.21.15.69:0/3065160775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd31c004010 0x7fd334157990 secure :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fd32800d620 tx=0x7fd32800dae0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 -- 172.21.15.69:0/3065160775 >> 172.21.15.69:0/3065160775 conn(0x7fd33409d900 msgr2=0x7fd3340cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 -- 172.21.15.69:0/3065160775 shutdown_connections 2024-04-24T08:20:25.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.674+0000 7fd33ae00640 1 -- 172.21.15.69:0/3065160775 wait complete. 2024-04-24T08:20:25.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:20:06.963817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:20:08.190440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:20:25.695 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:20:25.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.896+0000 7f2a03200640 1 Processor -- start 2024-04-24T08:20:25.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.896+0000 7f2a03200640 1 -- start start 2024-04-24T08:20:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.897+0000 7f2a03200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2a04072480 0x7f2a0415f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.897+0000 7f2a03200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a0406e220 con 0x7f2a0415e050 2024-04-24T08:20:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.897+0000 7f2a03200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a0415e430 con 0x7f2a0406dd90 2024-04-24T08:20:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.897+0000 7f2a03200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2a0415f660 con 0x7f2a04072480 2024-04-24T08:20:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a01e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2a0415e050 0x7f2a0406f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56982/0 (socket says 172.21.15.69:56982) 2024-04-24T08:20:25.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a01e00640 1 -- 172.21.15.69:0/253511723 learned_addr learned my addr 172.21.15.69:0/253511723 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a02800640 1 --2- 172.21.15.69:0/253511723 >> v2:172.21.15.121:3300/0 conn(0x7f2a04072480 0x7f2a0415f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1268138547 0 0) 0x7f2a0406e220 con 0x7f2a0415e050 2024-04-24T08:20:25.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29e8003520 con 0x7f2a0415e050 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3939297048 0 0) 0x7f29e8003520 con 0x7f2a0415e050 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 >> v2:172.21.15.121:3300/0 conn(0x7f2a04072480 msgr2=0x7f2a0415f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a00a00640 1 --2- 172.21.15.69:0/253511723 >> v2:172.21.15.121:3300/0 conn(0x7f2a04072480 0x7f2a0415f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 >> v1:172.21.15.121:6789/0 conn(0x7f2a0406dd90 legacy=0x7f2a04071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0406e490 con 0x7f2a0415e050 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.898+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f29f40019f0 con 0x7f2a0415e050 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 >> v1:172.21.15.69:6789/0 conn(0x7f2a0415e050 legacy=0x7f2a0406f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a00a00640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e80039b0 0x7f29e8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a00a00640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a00a00640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80051c0 0x7f29e800dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29e800e2d0 con 0x7f29e80051c0 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f29e800e480 con 0x7f29e8004800 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f29e800e680 con 0x7f29e80039b0 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a02800640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80051c0 0x7f29e800dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a01e00640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e80039b0 0x7f29e8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a01400640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a02800640 1 -- 172.21.15.69:0/253511723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e80039b0 msgr2=0x7f29e8003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:25.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a02800640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e80039b0 0x7f29e8003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a02800640 1 -- 172.21.15.69:0/253511723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e8004800 msgr2=0x7f29e8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:25.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a02800640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29e8004800 0x7f29e8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.899+0000 7f2a02800640 1 -- 172.21.15.69:0/253511723 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e8003520 con 0x7f29e80051c0 2024-04-24T08:20:25.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.900+0000 7f2a02800640 1 -- 172.21.15.69:0/253511723 reap_dead start 2024-04-24T08:20:25.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.900+0000 7f2a02800640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80051c0 0x7f29e800dbd0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009ac0 tx=0x7f29e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c701279b1db3b18 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.901+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc009690 con 0x7f29e80051c0 2024-04-24T08:20:25.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.901+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29fc004020 con 0x7f29e80051c0 2024-04-24T08:20:25.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.901+0000 7f2a00a00640 1 -- 172.21.15.69:0/253511723 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc004410 con 0x7f29e80051c0 2024-04-24T08:20:25.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.901+0000 7f2a03200640 1 -- 172.21.15.69:0/253511723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80051c0 msgr2=0x7f29e800dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.901+0000 7f2a03200640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80051c0 0x7f29e800dbd0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009ac0 tx=0x7f29e8005830 comp rx=0 tx=0).stop 2024-04-24T08:20:25.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.901+0000 7f2a03200640 1 -- 172.21.15.69:0/253511723 shutdown_connections 2024-04-24T08:20:25.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.901+0000 7f2a03200640 1 --2- 172.21.15.69:0/253511723 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f29e80051c0 0x7f29e800dbd0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.901+0000 7f2a03200640 1 -- 172.21.15.69:0/253511723 >> 172.21.15.69:0/253511723 conn(0x7f2a04067b80 msgr2=0x7f2a0406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:25.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.902+0000 7f2a03200640 1 -- 172.21.15.69:0/253511723 shutdown_connections 2024-04-24T08:20:25.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.902+0000 7f2a03200640 1 -- 172.21.15.69:0/253511723 wait complete. 2024-04-24T08:20:25.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.903+0000 7f2a03200640 1 Processor -- start 2024-04-24T08:20:25.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.903+0000 7f2a03200640 1 -- start start 2024-04-24T08:20:25.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a03200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e80051c0 0x7f2a041f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a03200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a04156bd0 0x7f2a04159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a03200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a04157030 0x7f2a041598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a03200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0406ed30 con 0x7f2a04157030 2024-04-24T08:20:25.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a03200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0406ea50 con 0x7f2a04156bd0 2024-04-24T08:20:25.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a03200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a0406f010 con 0x7f29e80051c0 2024-04-24T08:20:25.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a02800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a04157030 0x7f2a041598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a02800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a04157030 0x7f2a041598b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40936/0 (socket says 172.21.15.69:40936) 2024-04-24T08:20:25.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a02800640 1 -- 172.21.15.69:0/1297082453 learned_addr learned my addr 172.21.15.69:0/1297082453 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:25.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a01e00640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e80051c0 0x7f2a041f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a01400640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a04156bd0 0x7f2a04159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a02800640 1 -- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e80051c0 msgr2=0x7f2a041f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a02800640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e80051c0 0x7f2a041f3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a02800640 1 -- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a04156bd0 msgr2=0x7f2a04159370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:25.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a02800640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a04156bd0 0x7f2a04159370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:25.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.905+0000 7f2a02800640 1 -- 172.21.15.69:0/1297082453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a04151200 con 0x7f2a04157030 2024-04-24T08:20:25.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.906+0000 7f2a02800640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a04157030 0x7f2a041598b0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009bb0 tx=0x7f29fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.906+0000 7f29fa000640 1 -- 172.21.15.69:0/1297082453 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc036040 con 0x7f2a04157030 2024-04-24T08:20:25.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.906+0000 7f29fa000640 1 -- 172.21.15.69:0/1297082453 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29fc040070 con 0x7f2a04157030 2024-04-24T08:20:25.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.906+0000 7f29fa000640 1 -- 172.21.15.69:0/1297082453 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc03b410 con 0x7f2a04157030 2024-04-24T08:20:25.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.906+0000 7f2a03200640 1 -- 172.21.15.69:0/1297082453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a041514e0 con 0x7f2a04157030 2024-04-24T08:20:25.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.906+0000 7f2a03200640 1 -- 172.21.15.69:0/1297082453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a04151aa0 con 0x7f2a04157030 2024-04-24T08:20:25.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.908+0000 7f2a03200640 1 -- 172.21.15.69:0/1297082453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a04075d50 con 0x7f2a04157030 2024-04-24T08:20:25.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.909+0000 7f29fa000640 1 -- 172.21.15.69:0/1297082453 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f29fc03b5b0 con 0x7f2a04157030 2024-04-24T08:20:25.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.909+0000 7f29fa000640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f29e006c8f0 0x7f29e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:25.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.910+0000 7f29fa000640 1 -- 172.21.15.69:0/1297082453 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f29fc0ab690 con 0x7f2a04157030 2024-04-24T08:20:25.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.910+0000 7f2a01e00640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f29e006c8f0 0x7f29e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:25.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.911+0000 7f2a01e00640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f29e006c8f0 0x7f29e006edb0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f29f4005d30 tx=0x7f29f4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:25.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:25.914+0000 7f29fa000640 1 -- 172.21.15.69:0/1297082453 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f29fc049020 con 0x7f2a04157030 2024-04-24T08:20:26.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.113+0000 7f2a03200640 1 -- 172.21.15.69:0/1297082453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2a040728a0 con 0x7f2a04157030 2024-04-24T08:20:26.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.114+0000 7f29fa000640 1 -- 172.21.15.69:0/1297082453 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) ==== 93+0+10 (secure 0 0 0) 0x7f29fc079a20 con 0x7f2a04157030 2024-04-24T08:20:26.114 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 -- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f29e006c8f0 msgr2=0x7f29e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f29e006c8f0 0x7f29e006edb0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f29f4005d30 tx=0x7f29f4009040 comp rx=0 tx=0).stop 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 -- 172.21.15.69:0/1297082453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a04157030 msgr2=0x7f2a041598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a04157030 0x7f2a041598b0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f29fc009bb0 tx=0x7f29fc004730 comp rx=0 tx=0).stop 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 -- 172.21.15.69:0/1297082453 shutdown_connections 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f29e006c8f0 0x7f29e006edb0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a04157030 0x7f2a041598b0 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a04156bd0 0x7f2a04159370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 --2- 172.21.15.69:0/1297082453 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f29e80051c0 0x7f2a041f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 -- 172.21.15.69:0/1297082453 >> 172.21.15.69:0/1297082453 conn(0x7f2a04067b80 msgr2=0x7f2a04156330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 -- 172.21.15.69:0/1297082453 shutdown_connections 2024-04-24T08:20:26.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.116+0000 7f2a03200640 1 -- 172.21.15.69:0/1297082453 wait complete. 2024-04-24T08:20:26.137 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:20:26.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.341+0000 7fc46de00640 1 Processor -- start 2024-04-24T08:20:26.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.341+0000 7fc46de00640 1 -- start start 2024-04-24T08:20:26.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.343+0000 7fc46de00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4680b3e90 0x7fc4680b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.343+0000 7fc46de00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc4680ce250 0x7fc4680ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.343+0000 7fc46de00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc46815cc00 con 0x7fc46815c820 2024-04-24T08:20:26.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.343+0000 7fc46de00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc4680b4be0 con 0x7fc4680b3e90 2024-04-24T08:20:26.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.343+0000 7fc46de00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc46806ed30 con 0x7fc4680ce250 2024-04-24T08:20:26.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.343+0000 7fc466000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4680b3e90 0x7fc4680b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.343+0000 7fc466000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc4680b3e90 0x7fc4680b46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40950/0 (socket says 172.21.15.69:40950) 2024-04-24T08:20:26.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.343+0000 7fc466000640 1 -- 172.21.15.69:0/606701573 learned_addr learned my addr 172.21.15.69:0/606701573 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:26.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.343+0000 7fc467400640 1 --2- 172.21.15.69:0/606701573 >> v2:172.21.15.121:3300/0 conn(0x7fc4680ce250 0x7fc4680ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.344+0000 7fc466000640 1 -- 172.21.15.69:0/606701573 >> v2:172.21.15.121:3300/0 conn(0x7fc4680ce250 msgr2=0x7fc4680ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.344+0000 7fc466000640 1 --2- 172.21.15.69:0/606701573 >> v2:172.21.15.121:3300/0 conn(0x7fc4680ce250 0x7fc4680ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.344+0000 7fc466000640 1 -- 172.21.15.69:0/606701573 >> v1:172.21.15.121:6789/0 conn(0x7fc46815c820 legacy=0x7fc46806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.344+0000 7fc466000640 1 -- 172.21.15.69:0/606701573 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4680b4e10 con 0x7fc4680b3e90 2024-04-24T08:20:26.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.345+0000 7fc466000640 1 --2- 172.21.15.69:0/606701573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4680b3e90 0x7fc4680b46a0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fc468161420 tx=0x7fc45002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea4d92c16cbaf2c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.345+0000 7fc465600640 1 -- 172.21.15.69:0/606701573 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc450036070 con 0x7fc4680b3e90 2024-04-24T08:20:26.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.345+0000 7fc465600640 1 -- 172.21.15.69:0/606701573 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc450031600 con 0x7fc4680b3e90 2024-04-24T08:20:26.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.345+0000 7fc465600640 1 -- 172.21.15.69:0/606701573 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4500319d0 con 0x7fc4680b3e90 2024-04-24T08:20:26.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.346+0000 7fc46de00640 1 -- 172.21.15.69:0/606701573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4680b3e90 msgr2=0x7fc4680b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.346+0000 7fc46de00640 1 --2- 172.21.15.69:0/606701573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4680b3e90 0x7fc4680b46a0 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7fc468161420 tx=0x7fc45002a6f0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.346+0000 7fc46de00640 1 -- 172.21.15.69:0/606701573 shutdown_connections 2024-04-24T08:20:26.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.346+0000 7fc46de00640 1 --2- 172.21.15.69:0/606701573 >> v2:172.21.15.121:3300/0 conn(0x7fc4680ce250 0x7fc4680ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.346+0000 7fc46de00640 1 --2- 172.21.15.69:0/606701573 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4680b3e90 0x7fc4680b46a0 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.346+0000 7fc46de00640 1 -- 172.21.15.69:0/606701573 >> 172.21.15.69:0/606701573 conn(0x7fc4680d0350 msgr2=0x7fc4680d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.346+0000 7fc46de00640 1 -- 172.21.15.69:0/606701573 shutdown_connections 2024-04-24T08:20:26.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.346+0000 7fc46de00640 1 -- 172.21.15.69:0/606701573 wait complete. 2024-04-24T08:20:26.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.348+0000 7fc46de00640 1 Processor -- start 2024-04-24T08:20:26.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.348+0000 7fc46de00640 1 -- start start 2024-04-24T08:20:26.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.349+0000 7fc46de00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4680b3e90 0x7fc468149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.349+0000 7fc46de00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4680ce250 0x7fc46815e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.349+0000 7fc46de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc46815c820 0x7fc46815ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.349+0000 7fc46de00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc46806f2b0 con 0x7fc46815c820 2024-04-24T08:20:26.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.349+0000 7fc46de00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc46806efd0 con 0x7fc4680b3e90 2024-04-24T08:20:26.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.349+0000 7fc46de00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc46806fee0 con 0x7fc4680ce250 2024-04-24T08:20:26.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.349+0000 7fc467400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc46815c820 0x7fc46815ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.349+0000 7fc467400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc46815c820 0x7fc46815ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40956/0 (socket says 172.21.15.69:40956) 2024-04-24T08:20:26.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.349+0000 7fc467400640 1 -- 172.21.15.69:0/1546251121 learned_addr learned my addr 172.21.15.69:0/1546251121 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:26.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc466a00640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4680b3e90 0x7fc468149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc466000640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4680ce250 0x7fc46815e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc467400640 1 -- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4680ce250 msgr2=0x7fc46815e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc467400640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4680ce250 0x7fc46815e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc467400640 1 -- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4680b3e90 msgr2=0x7fc468149c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc467400640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4680b3e90 0x7fc468149c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc467400640 1 -- 172.21.15.69:0/1546251121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc46815e5f0 con 0x7fc46815c820 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc466a00640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4680b3e90 0x7fc468149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc467400640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc46815c820 0x7fc46815ef30 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fc46000d830 tx=0x7fc46000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc45f400640 1 -- 172.21.15.69:0/1546251121 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46000b6c0 con 0x7fc46815c820 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc45f400640 1 -- 172.21.15.69:0/1546251121 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc460019070 con 0x7fc46815c820 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc45f400640 1 -- 172.21.15.69:0/1546251121 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46000bac0 con 0x7fc46815c820 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.350+0000 7fc46de00640 1 -- 172.21.15.69:0/1546251121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc468157250 con 0x7fc46815c820 2024-04-24T08:20:26.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.351+0000 7fc46de00640 1 -- 172.21.15.69:0/1546251121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc468157810 con 0x7fc46815c820 2024-04-24T08:20:26.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.352+0000 7fc46de00640 1 -- 172.21.15.69:0/1546251121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc468068a80 con 0x7fc46815c820 2024-04-24T08:20:26.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.353+0000 7fc45f400640 1 -- 172.21.15.69:0/1546251121 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc46001e460 con 0x7fc46815c820 2024-04-24T08:20:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.354+0000 7fc45f400640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc44406c8f0 0x7fc44406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.354+0000 7fc45f400640 1 -- 172.21.15.69:0/1546251121 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fc46008c2d0 con 0x7fc46815c820 2024-04-24T08:20:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.354+0000 7fc466a00640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc44406c8f0 0x7fc44406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.356+0000 7fc466a00640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc44406c8f0 0x7fc44406edb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc458003eb0 tx=0x7fc458009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.358+0000 7fc45f400640 1 -- 172.21.15.69:0/1546251121 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc46005a660 con 0x7fc46815c820 2024-04-24T08:20:26.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.560+0000 7fc46de00640 1 -- 172.21.15.69:0/1546251121 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc4680cddc0 con 0x7fc46815c820 2024-04-24T08:20:26.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.561+0000 7fc45f400640 1 -- 172.21.15.69:0/1546251121 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) ==== 104+0+11 (secure 0 0 0) 0x7fc460059f60 con 0x7fc46815c820 2024-04-24T08:20:26.561 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:20:26.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 -- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc44406c8f0 msgr2=0x7fc44406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc44406c8f0 0x7fc44406edb0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fc458003eb0 tx=0x7fc458009040 comp rx=0 tx=0).stop 2024-04-24T08:20:26.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 -- 172.21.15.69:0/1546251121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc46815c820 msgr2=0x7fc46815ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc46815c820 0x7fc46815ef30 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fc46000d830 tx=0x7fc46000dcf0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 -- 172.21.15.69:0/1546251121 shutdown_connections 2024-04-24T08:20:26.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc44406c8f0 0x7fc44406edb0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc46815c820 0x7fc46815ef30 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc4680ce250 0x7fc46815e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 --2- 172.21.15.69:0/1546251121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc4680b3e90 0x7fc468149c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 -- 172.21.15.69:0/1546251121 >> 172.21.15.69:0/1546251121 conn(0x7fc4680d0350 msgr2=0x7fc4680b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 -- 172.21.15.69:0/1546251121 shutdown_connections 2024-04-24T08:20:26.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.563+0000 7fc46de00640 1 -- 172.21.15.69:0/1546251121 wait complete. 2024-04-24T08:20:26.581 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:20:26.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.785+0000 7fe4e1e00640 1 Processor -- start 2024-04-24T08:20:26.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.785+0000 7fe4e1e00640 1 -- start start 2024-04-24T08:20:26.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.786+0000 7fe4e1e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe4dc0b5300 0x7fe4dc0d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.786+0000 7fe4e1e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe4dc0d0820 0x7fe4dc06df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.786+0000 7fe4e1e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4dc0d0c40 con 0x7fe4dc15ea60 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.786+0000 7fe4e1e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe4dc0ce640 con 0x7fe4dc0d0820 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.786+0000 7fe4e1e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe4dc0ce910 con 0x7fe4dc0b5300 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.787+0000 7fe4e1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe4dc0d0820 0x7fe4dc06df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.787+0000 7fe4e1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe4dc0d0820 0x7fe4dc06df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40972/0 (socket says 172.21.15.69:40972) 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.787+0000 7fe4e1400640 1 -- 172.21.15.69:0/231225049 learned_addr learned my addr 172.21.15.69:0/231225049 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.787+0000 7fe4e1400640 1 -- 172.21.15.69:0/231225049 >> v2:172.21.15.121:3300/0 conn(0x7fe4dc0b5300 msgr2=0x7fe4dc0d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.787+0000 7fe4e1400640 1 --2- 172.21.15.69:0/231225049 >> v2:172.21.15.121:3300/0 conn(0x7fe4dc0b5300 0x7fe4dc0d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.787+0000 7fe4e1400640 1 -- 172.21.15.69:0/231225049 >> v1:172.21.15.121:6789/0 conn(0x7fe4dc15ea60 legacy=0x7fe4dc0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.787+0000 7fe4e1400640 1 -- 172.21.15.69:0/231225049 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4dc0cebf0 con 0x7fe4dc0d0820 2024-04-24T08:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.789+0000 7fe4e1400640 1 --2- 172.21.15.69:0/231225049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0d0820 0x7fe4dc06df40 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0b5a10 tx=0x7fe4d402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0efa5b36713d522 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.789+0000 7fe4db400640 1 -- 172.21.15.69:0/231225049 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4037070 con 0x7fe4dc0d0820 2024-04-24T08:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.789+0000 7fe4db400640 1 -- 172.21.15.69:0/231225049 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d4032440 con 0x7fe4dc0d0820 2024-04-24T08:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.789+0000 7fe4db400640 1 -- 172.21.15.69:0/231225049 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4032810 con 0x7fe4dc0d0820 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.790+0000 7fe4e1e00640 1 -- 172.21.15.69:0/231225049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0d0820 msgr2=0x7fe4dc06df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.790+0000 7fe4e1e00640 1 --2- 172.21.15.69:0/231225049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0d0820 0x7fe4dc06df40 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0b5a10 tx=0x7fe4d402a230 comp rx=0 tx=0).stop 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.790+0000 7fe4e1e00640 1 -- 172.21.15.69:0/231225049 shutdown_connections 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.790+0000 7fe4e1e00640 1 --2- 172.21.15.69:0/231225049 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0d0820 0x7fe4dc06df40 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.790+0000 7fe4e1e00640 1 --2- 172.21.15.69:0/231225049 >> v2:172.21.15.121:3300/0 conn(0x7fe4dc0b5300 0x7fe4dc0d02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.790+0000 7fe4e1e00640 1 -- 172.21.15.69:0/231225049 >> 172.21.15.69:0/231225049 conn(0x7fe4dc067b80 msgr2=0x7fe4dc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.790+0000 7fe4e1e00640 1 -- 172.21.15.69:0/231225049 shutdown_connections 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.790+0000 7fe4e1e00640 1 -- 172.21.15.69:0/231225049 wait complete. 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.792+0000 7fe4e1e00640 1 Processor -- start 2024-04-24T08:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.792+0000 7fe4e1e00640 1 -- start start 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.793+0000 7fe4e1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0b5300 0x7fe4dc0b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.793+0000 7fe4e1e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4dc0d0820 0x7fe4dc14c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.793+0000 7fe4e1e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4dc15ea60 0x7fe4dc0cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.793+0000 7fe4e1e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc0cf490 con 0x7fe4dc0b5300 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.793+0000 7fe4e1e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe4dc0cf1b0 con 0x7fe4dc15ea60 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.793+0000 7fe4e1e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe4dc06e480 con 0x7fe4dc0d0820 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.793+0000 7fe4e0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0b5300 0x7fe4dc0b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.793+0000 7fe4e0a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0b5300 0x7fe4dc0b4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40986/0 (socket says 172.21.15.69:40986) 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.793+0000 7fe4e0a00640 1 -- 172.21.15.69:0/1870846382 learned_addr learned my addr 172.21.15.69:0/1870846382 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4e1400640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4dc15ea60 0x7fe4dc0cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4dbe00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4dc0d0820 0x7fe4dc14c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4e0a00640 1 -- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4dc0d0820 msgr2=0x7fe4dc14c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4e0a00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4dc0d0820 0x7fe4dc14c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4e0a00640 1 -- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4dc15ea60 msgr2=0x7fe4dc0cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4e0a00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4dc15ea60 0x7fe4dc0cba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4e0a00640 1 -- 172.21.15.69:0/1870846382 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4dc0b3360 con 0x7fe4dc0b5300 2024-04-24T08:20:26.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4e0a00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0b5300 0x7fe4dc0b4400 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00ba50 tx=0x7fe4cc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4d8c00640 1 -- 172.21.15.69:0/1870846382 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc019070 con 0x7fe4dc0b5300 2024-04-24T08:20:26.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4d8c00640 1 -- 172.21.15.69:0/1870846382 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4cc00cc20 con 0x7fe4dc0b5300 2024-04-24T08:20:26.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4d8c00640 1 -- 172.21.15.69:0/1870846382 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc014680 con 0x7fe4dc0b5300 2024-04-24T08:20:26.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4e1e00640 1 -- 172.21.15.69:0/1870846382 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4dc0b3640 con 0x7fe4dc0b5300 2024-04-24T08:20:26.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.794+0000 7fe4e1e00640 1 -- 172.21.15.69:0/1870846382 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4dc0b3c00 con 0x7fe4dc0b5300 2024-04-24T08:20:26.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.796+0000 7fe4e1e00640 1 -- 172.21.15.69:0/1870846382 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4dc15bc00 con 0x7fe4dc0b5300 2024-04-24T08:20:26.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.797+0000 7fe4d8c00640 1 -- 172.21.15.69:0/1870846382 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe4cc01c070 con 0x7fe4dc0b5300 2024-04-24T08:20:26.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.798+0000 7fe4d8c00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4bc06c8f0 0x7fe4bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.798+0000 7fe4d8c00640 1 -- 172.21.15.69:0/1870846382 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fe4cc08b880 con 0x7fe4dc0b5300 2024-04-24T08:20:26.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.798+0000 7fe4dbe00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4bc06c8f0 0x7fe4bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.799+0000 7fe4dbe00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4bc06c8f0 0x7fe4bc06edb0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0caef0 tx=0x7fe4c8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:26.802+0000 7fe4d8c00640 1 -- 172.21.15.69:0/1870846382 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe4dc15bc00 con 0x7fe4dc0b5300 2024-04-24T08:20:27.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.000+0000 7fe4e1e00640 1 -- 172.21.15.69:0/1870846382 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe4dc0b4040 con 0x7fe4dc0b5300 2024-04-24T08:20:27.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.001+0000 7fe4d8c00640 1 -- 172.21.15.69:0/1870846382 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) ==== 100+0+11 (secure 0 0 0) 0x7fe4cc059650 con 0x7fe4dc0b5300 2024-04-24T08:20:27.002 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 -- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4bc06c8f0 msgr2=0x7fe4bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4bc06c8f0 0x7fe4bc06edb0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0caef0 tx=0x7fe4c8008040 comp rx=0 tx=0).stop 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 -- 172.21.15.69:0/1870846382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0b5300 msgr2=0x7fe4dc0b4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0b5300 0x7fe4dc0b4400 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc00ba50 tx=0x7fe4cc00bf10 comp rx=0 tx=0).stop 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 -- 172.21.15.69:0/1870846382 shutdown_connections 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4bc06c8f0 0x7fe4bc06edb0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4dc15ea60 0x7fe4dc0cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4dc0d0820 0x7fe4dc14c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 --2- 172.21.15.69:0/1870846382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4dc0b5300 0x7fe4dc0b4400 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 -- 172.21.15.69:0/1870846382 >> 172.21.15.69:0/1870846382 conn(0x7fe4dc067b80 msgr2=0x7fe4dc15dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 -- 172.21.15.69:0/1870846382 shutdown_connections 2024-04-24T08:20:27.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.004+0000 7fe4e1e00640 1 -- 172.21.15.69:0/1870846382 wait complete. 2024-04-24T08:20:27.023 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:20:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.225+0000 7ff14e400640 1 Processor -- start 2024-04-24T08:20:27.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.226+0000 7ff14e400640 1 -- start start 2024-04-24T08:20:27.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.227+0000 7ff14e400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff14815d1d0 0x7ff1480ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.227+0000 7ff14e400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff1480b30f0 0x7ff1480ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.227+0000 7ff14e400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff14815d660 con 0x7ff148156c60 2024-04-24T08:20:27.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.227+0000 7ff14e400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7ff14806dc30 con 0x7ff1480b30f0 2024-04-24T08:20:27.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.227+0000 7ff14e400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff14806df10 con 0x7ff14815d1d0 2024-04-24T08:20:27.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.228+0000 7ff14da00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff1480b30f0 0x7ff1480ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.228+0000 7ff13fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff14815d1d0 0x7ff1480ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.228+0000 7ff13fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff14815d1d0 0x7ff1480ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:52134/0 (socket says 172.21.15.69:52134) 2024-04-24T08:20:27.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.228+0000 7ff13fe00640 1 -- 172.21.15.69:0/3091702112 learned_addr learned my addr 172.21.15.69:0/3091702112 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:27.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.229+0000 7ff14da00640 1 -- 172.21.15.69:0/3091702112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff14815d1d0 msgr2=0x7ff1480ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.229+0000 7ff14da00640 1 --2- 172.21.15.69:0/3091702112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff14815d1d0 0x7ff1480ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.229+0000 7ff14da00640 1 -- 172.21.15.69:0/3091702112 >> v1:172.21.15.121:6789/0 conn(0x7ff148156c60 legacy=0x7ff14815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.229+0000 7ff14da00640 1 -- 172.21.15.69:0/3091702112 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff14806e1f0 con 0x7ff1480b30f0 2024-04-24T08:20:27.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.230+0000 7ff14da00640 1 --2- 172.21.15.69:0/3091702112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1480b30f0 0x7ff1480ad610 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff140009d90 tx=0x7ff14002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79298df449a8eab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.231+0000 7ff147e00640 1 -- 172.21.15.69:0/3091702112 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140007740 con 0x7ff1480b30f0 2024-04-24T08:20:27.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.238+0000 7ff147e00640 1 -- 172.21.15.69:0/3091702112 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1400078e0 con 0x7ff1480b30f0 2024-04-24T08:20:27.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.238+0000 7ff147e00640 1 -- 172.21.15.69:0/3091702112 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140007cb0 con 0x7ff1480b30f0 2024-04-24T08:20:27.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.238+0000 7ff14e400640 1 -- 172.21.15.69:0/3091702112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1480b30f0 msgr2=0x7ff1480ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.238+0000 7ff14e400640 1 --2- 172.21.15.69:0/3091702112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1480b30f0 0x7ff1480ad610 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff140009d90 tx=0x7ff14002a660 comp rx=0 tx=0).stop 2024-04-24T08:20:27.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.239+0000 7ff14e400640 1 -- 172.21.15.69:0/3091702112 shutdown_connections 2024-04-24T08:20:27.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.239+0000 7ff14e400640 1 --2- 172.21.15.69:0/3091702112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff1480b30f0 0x7ff1480ad610 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.239+0000 7ff14e400640 1 --2- 172.21.15.69:0/3091702112 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff14815d1d0 0x7ff1480ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.239+0000 7ff14e400640 1 -- 172.21.15.69:0/3091702112 >> 172.21.15.69:0/3091702112 conn(0x7ff1480d0160 msgr2=0x7ff1480d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.239+0000 7ff14e400640 1 -- 172.21.15.69:0/3091702112 shutdown_connections 2024-04-24T08:20:27.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.239+0000 7ff14e400640 1 -- 172.21.15.69:0/3091702112 wait complete. 2024-04-24T08:20:27.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.240+0000 7ff14e400640 1 Processor -- start 2024-04-24T08:20:27.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.241+0000 7ff14e400640 1 -- start start 2024-04-24T08:20:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.241+0000 7ff14e400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1480b30f0 0x7ff148142e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.241+0000 7ff14e400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff148156c60 0x7ff148155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.241+0000 7ff14e400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1480cc2e0 0x7ff148072c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.241+0000 7ff14e400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff14806e710 con 0x7ff1480cc2e0 2024-04-24T08:20:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.241+0000 7ff14e400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff14815d940 con 0x7ff148156c60 2024-04-24T08:20:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.241+0000 7ff14e400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff14806e890 con 0x7ff1480b30f0 2024-04-24T08:20:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff14da00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1480cc2e0 0x7ff148072c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff14da00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1480cc2e0 0x7ff148072c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40994/0 (socket says 172.21.15.69:40994) 2024-04-24T08:20:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff14da00640 1 -- 172.21.15.69:0/4059496143 learned_addr learned my addr 172.21.15.69:0/4059496143 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:27.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff14d000640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1480b30f0 0x7ff148142e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff14da00640 1 -- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1480b30f0 msgr2=0x7ff148142e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff14da00640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1480b30f0 0x7ff148142e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff13fe00640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff148156c60 0x7ff148155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff14da00640 1 -- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff148156c60 msgr2=0x7ff148155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff14da00640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff148156c60 0x7ff148155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff14da00640 1 -- 172.21.15.69:0/4059496143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1480cc7a0 con 0x7ff1480cc2e0 2024-04-24T08:20:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff14d000640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1480b30f0 0x7ff148142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.242+0000 7ff13fe00640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff148156c60 0x7ff148155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:27.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.243+0000 7ff14da00640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1480cc2e0 0x7ff148072c30 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7ff14002ac50 tx=0x7ff140004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.243+0000 7ff145600640 1 -- 172.21.15.69:0/4059496143 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140040070 con 0x7ff1480cc2e0 2024-04-24T08:20:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.243+0000 7ff145600640 1 -- 172.21.15.69:0/4059496143 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff140031bc0 con 0x7ff1480cc2e0 2024-04-24T08:20:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.243+0000 7ff145600640 1 -- 172.21.15.69:0/4059496143 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140009450 con 0x7ff1480cc2e0 2024-04-24T08:20:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.243+0000 7ff14e400640 1 -- 172.21.15.69:0/4059496143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1480ca6b0 con 0x7ff1480cc2e0 2024-04-24T08:20:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.243+0000 7ff14e400640 1 -- 172.21.15.69:0/4059496143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1480cac70 con 0x7ff1480cc2e0 2024-04-24T08:20:27.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.244+0000 7ff14e400640 1 -- 172.21.15.69:0/4059496143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff148075d50 con 0x7ff1480cc2e0 2024-04-24T08:20:27.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.245+0000 7ff145600640 1 -- 172.21.15.69:0/4059496143 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff140039040 con 0x7ff1480cc2e0 2024-04-24T08:20:27.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.246+0000 7ff145600640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff12406c8f0 0x7ff12406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.246+0000 7ff145600640 1 -- 172.21.15.69:0/4059496143 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7ff1400ad4e0 con 0x7ff1480cc2e0 2024-04-24T08:20:27.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.247+0000 7ff14d000640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff12406c8f0 0x7ff12406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.248+0000 7ff14d000640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff12406c8f0 0x7ff12406edb0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff138000f60 tx=0x7ff138009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.251+0000 7ff145600640 1 -- 172.21.15.69:0/4059496143 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff14007b7f0 con 0x7ff1480cc2e0 2024-04-24T08:20:27.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.469+0000 7ff14e400640 1 -- 172.21.15.69:0/4059496143 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff1480cae40 con 0x7ff1480cc2e0 2024-04-24T08:20:27.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.470+0000 7ff145600640 1 -- 172.21.15.69:0/4059496143 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v109) ==== 80+0+0 (secure 0 0 0) 0x7ff14007b0f0 con 0x7ff1480cc2e0 2024-04-24T08:20:27.471 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:20:27.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 -- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff12406c8f0 msgr2=0x7ff12406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff12406c8f0 0x7ff12406edb0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7ff138000f60 tx=0x7ff138009040 comp rx=0 tx=0).stop 2024-04-24T08:20:27.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 -- 172.21.15.69:0/4059496143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1480cc2e0 msgr2=0x7ff148072c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1480cc2e0 0x7ff148072c30 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7ff14002ac50 tx=0x7ff140004a40 comp rx=0 tx=0).stop 2024-04-24T08:20:27.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 -- 172.21.15.69:0/4059496143 shutdown_connections 2024-04-24T08:20:27.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff12406c8f0 0x7ff12406edb0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1480cc2e0 0x7ff148072c30 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff148156c60 0x7ff148155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 --2- 172.21.15.69:0/4059496143 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff1480b30f0 0x7ff148142e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 -- 172.21.15.69:0/4059496143 >> 172.21.15.69:0/4059496143 conn(0x7ff1480d0160 msgr2=0x7ff1480ad0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 -- 172.21.15.69:0/4059496143 shutdown_connections 2024-04-24T08:20:27.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.473+0000 7ff14e400640 1 -- 172.21.15.69:0/4059496143 wait complete. 2024-04-24T08:20:27.492 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:20:27.492 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:20:27.543 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:27.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.744+0000 7f7076600640 1 Processor -- start 2024-04-24T08:20:27.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.744+0000 7f7076600640 1 -- start start 2024-04-24T08:20:27.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.746+0000 7f7076600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70700d0300 0x7f707006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.746+0000 7f7076600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7070145a90 0x7f70700d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.746+0000 7f7076600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7070145fe0 0x7f7070160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.746+0000 7f7076600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f70700d0c20 con 0x7f70700d0300 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.746+0000 7f7076600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f70700cf3d0 con 0x7f7070145a90 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.746+0000 7f7076600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f70700cf6b0 con 0x7f7070145fe0 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.746+0000 7f706f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70700d0300 0x7f707006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.746+0000 7f706f400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70700d0300 0x7f707006e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40998/0 (socket says 172.21.15.69:40998) 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.746+0000 7f706f400640 1 -- 172.21.15.69:0/3499432548 learned_addr learned my addr 172.21.15.69:0/3499432548 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.746+0000 7f706ea00640 1 --2- 172.21.15.69:0/3499432548 >> v2:172.21.15.121:3300/0 conn(0x7f7070145a90 0x7f70700d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.747+0000 7f706fe00640 1 --2- 172.21.15.69:0/3499432548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7070145fe0 0x7f7070160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.747+0000 7f706f400640 1 -- 172.21.15.69:0/3499432548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7070145fe0 msgr2=0x7f7070160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.747+0000 7f706f400640 1 --2- 172.21.15.69:0/3499432548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7070145fe0 0x7f7070160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.747+0000 7f706f400640 1 -- 172.21.15.69:0/3499432548 >> v2:172.21.15.121:3300/0 conn(0x7f7070145a90 msgr2=0x7f70700d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.747+0000 7f706f400640 1 --2- 172.21.15.69:0/3499432548 >> v2:172.21.15.121:3300/0 conn(0x7f7070145a90 0x7f70700d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.747+0000 7f706f400640 1 -- 172.21.15.69:0/3499432548 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70700cf990 con 0x7f70700d0300 2024-04-24T08:20:27.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.747+0000 7f706fe00640 1 --2- 172.21.15.69:0/3499432548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7070145fe0 0x7f7070160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:27.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.748+0000 7f706f400640 1 --2- 172.21.15.69:0/3499432548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70700d0300 0x7f707006e240 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f70700b5780 tx=0x7f706002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bdea954ff05dcff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.748+0000 7f706e000640 1 -- 172.21.15.69:0/3499432548 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060037070 con 0x7f70700d0300 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.748+0000 7f706e000640 1 -- 172.21.15.69:0/3499432548 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7060032440 con 0x7f70700d0300 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.748+0000 7f706e000640 1 -- 172.21.15.69:0/3499432548 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7060032810 con 0x7f70700d0300 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.749+0000 7f7076600640 1 -- 172.21.15.69:0/3499432548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70700d0300 msgr2=0x7f707006e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.749+0000 7f7076600640 1 --2- 172.21.15.69:0/3499432548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70700d0300 0x7f707006e240 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f70700b5780 tx=0x7f706002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.749+0000 7f7076600640 1 -- 172.21.15.69:0/3499432548 shutdown_connections 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.749+0000 7f7076600640 1 --2- 172.21.15.69:0/3499432548 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7070145fe0 0x7f7070160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.749+0000 7f7076600640 1 --2- 172.21.15.69:0/3499432548 >> v2:172.21.15.121:3300/0 conn(0x7f7070145a90 0x7f70700d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.749+0000 7f7076600640 1 --2- 172.21.15.69:0/3499432548 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70700d0300 0x7f707006e240 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.749+0000 7f7076600640 1 -- 172.21.15.69:0/3499432548 >> 172.21.15.69:0/3499432548 conn(0x7f7070067d90 msgr2=0x7f707006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.749+0000 7f7076600640 1 -- 172.21.15.69:0/3499432548 shutdown_connections 2024-04-24T08:20:27.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.749+0000 7f7076600640 1 -- 172.21.15.69:0/3499432548 wait complete. 2024-04-24T08:20:27.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.751+0000 7f7076600640 1 Processor -- start 2024-04-24T08:20:27.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.751+0000 7f7076600640 1 -- start start 2024-04-24T08:20:27.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.752+0000 7f7076600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70700d0300 0x7f707015c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.752+0000 7f7076600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7070145a90 0x7f70701f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.752+0000 7f7076600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7070145fe0 0x7f707015eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.752+0000 7f7076600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7070145770 con 0x7f7070145a90 2024-04-24T08:20:27.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.752+0000 7f7076600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f70700cff50 con 0x7f70700d0300 2024-04-24T08:20:27.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.752+0000 7f7076600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f70701458f0 con 0x7f7070145fe0 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.752+0000 7f706ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7070145a90 0x7f70701f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.752+0000 7f706ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7070145a90 0x7f70701f4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41002/0 (socket says 172.21.15.69:41002) 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.752+0000 7f706ea00640 1 -- 172.21.15.69:0/3401426127 learned_addr learned my addr 172.21.15.69:0/3401426127 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f706f400640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70700d0300 0x7f707015c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f706ea00640 1 -- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7070145fe0 msgr2=0x7f707015eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f706ea00640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7070145fe0 0x7f707015eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f706ea00640 1 -- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70700d0300 msgr2=0x7f707015c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f706ea00640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70700d0300 0x7f707015c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f706ea00640 1 -- 172.21.15.69:0/3401426127 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f707015c630 con 0x7f7070145a90 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f706ea00640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7070145a90 0x7f70701f4bb0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f705c00d800 tx=0x7f705c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f7067400640 1 -- 172.21.15.69:0/3401426127 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c018030 con 0x7f7070145a90 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f7067400640 1 -- 172.21.15.69:0/3401426127 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f705c0096a0 con 0x7f7070145a90 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f7076600640 1 -- 172.21.15.69:0/3401426127 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70701e5b90 con 0x7f7070145a90 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.753+0000 7f7067400640 1 -- 172.21.15.69:0/3401426127 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f705c009870 con 0x7f7070145a90 2024-04-24T08:20:27.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.754+0000 7f7076600640 1 -- 172.21.15.69:0/3401426127 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70701e6430 con 0x7f7070145a90 2024-04-24T08:20:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.755+0000 7f7076600640 1 -- 172.21.15.69:0/3401426127 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7070068a80 con 0x7f7070145a90 2024-04-24T08:20:27.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.756+0000 7f7067400640 1 -- 172.21.15.69:0/3401426127 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f705c01b070 con 0x7f7070145a90 2024-04-24T08:20:27.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.756+0000 7f7067400640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 0x7f705006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.757+0000 7f7067400640 1 -- 172.21.15.69:0/3401426127 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f705c08be50 con 0x7f7070145a90 2024-04-24T08:20:27.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.757+0000 7f706f400640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 0x7f705006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.758+0000 7f706f400640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 0x7f705006edb0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7060034000 tx=0x7f706002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.761+0000 7f7067400640 1 -- 172.21.15.69:0/3401426127 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f707015c630 con 0x7f7070145a90 2024-04-24T08:20:27.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.962+0000 7f7076600640 1 -- 172.21.15.69:0/3401426127 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70700b3670 con 0x7f7070145a90 2024-04-24T08:20:27.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.964+0000 7f7067400640 1 -- 172.21.15.69:0/3401426127 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+16603 (secure 0 0 0) 0x7f705c05a1e0 con 0x7f7070145a90 2024-04-24T08:20:27.965 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:27.965 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":109,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:20:10.770819+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:27.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 -- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 msgr2=0x7f705006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 0x7f705006edb0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f7060034000 tx=0x7f706002e000 comp rx=0 tx=0).stop 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 -- 172.21.15.69:0/3401426127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7070145a90 msgr2=0x7f70701f4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7070145a90 0x7f70701f4bb0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f705c00d800 tx=0x7f705c00dcc0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 -- 172.21.15.69:0/3401426127 shutdown_connections 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f705006c8f0 0x7f705006edb0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7070145a90 0x7f70701f4bb0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7070145fe0 0x7f707015eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 --2- 172.21.15.69:0/3401426127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70700d0300 0x7f707015c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 -- 172.21.15.69:0/3401426127 >> 172.21.15.69:0/3401426127 conn(0x7f7070067d90 msgr2=0x7f707006e810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.967+0000 7f7076600640 1 -- 172.21.15.69:0/3401426127 shutdown_connections 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:27.968+0000 7f7076600640 1 -- 172.21.15.69:0/3401426127 wait complete. 2024-04-24T08:20:27.988 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:20:06.963817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:20:08.190440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:20:27.988 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.189+0000 7fe1ef600640 1 Processor -- start 2024-04-24T08:20:28.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.189+0000 7fe1ef600640 1 -- start start 2024-04-24T08:20:28.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.191+0000 7fe1ef600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe1e815c820 0x7fe1e80b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.191+0000 7fe1ef600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1e80b46a0 0x7fe1e806f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.191+0000 7fe1ef600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1e815cc00 con 0x7fe1e806fd10 2024-04-24T08:20:28.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.191+0000 7fe1ef600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe1e806ed30 con 0x7fe1e815c820 2024-04-24T08:20:28.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.191+0000 7fe1ef600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe1e80b4e10 con 0x7fe1e80b46a0 2024-04-24T08:20:28.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.191+0000 7fe1ecc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe1e815c820 0x7fe1e80b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.191+0000 7fe1ecc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe1e815c820 0x7fe1e80b3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41004/0 (socket says 172.21.15.69:41004) 2024-04-24T08:20:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.191+0000 7fe1ecc00640 1 -- 172.21.15.69:0/2048937925 learned_addr learned my addr 172.21.15.69:0/2048937925 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.192+0000 7fe1dfe00640 1 --2- 172.21.15.69:0/2048937925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1e80b46a0 0x7fe1e806f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.192+0000 7fe1ecc00640 1 -- 172.21.15.69:0/2048937925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1e80b46a0 msgr2=0x7fe1e806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.192+0000 7fe1ecc00640 1 --2- 172.21.15.69:0/2048937925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1e80b46a0 0x7fe1e806f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.192+0000 7fe1ecc00640 1 -- 172.21.15.69:0/2048937925 >> v1:172.21.15.121:6789/0 conn(0x7fe1e806fd10 legacy=0x7fe1e80ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.192+0000 7fe1ecc00640 1 -- 172.21.15.69:0/2048937925 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e80b50f0 con 0x7fe1e815c820 2024-04-24T08:20:28.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.192+0000 7fe1dfe00640 1 --2- 172.21.15.69:0/2048937925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1e80b46a0 0x7fe1e806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:28.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.193+0000 7fe1ecc00640 1 --2- 172.21.15.69:0/2048937925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e815c820 0x7fe1e80b3e90 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009e50 tx=0x7fe1d802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9ef4171cde0816 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.193+0000 7fe1e7e00640 1 -- 172.21.15.69:0/2048937925 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8036070 con 0x7fe1e815c820 2024-04-24T08:20:28.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.193+0000 7fe1e7e00640 1 -- 172.21.15.69:0/2048937925 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1d8031600 con 0x7fe1e815c820 2024-04-24T08:20:28.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.193+0000 7fe1e7e00640 1 -- 172.21.15.69:0/2048937925 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d80319d0 con 0x7fe1e815c820 2024-04-24T08:20:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.194+0000 7fe1ef600640 1 -- 172.21.15.69:0/2048937925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e815c820 msgr2=0x7fe1e80b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.194+0000 7fe1ef600640 1 --2- 172.21.15.69:0/2048937925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e815c820 0x7fe1e80b3e90 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009e50 tx=0x7fe1d802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.194+0000 7fe1ef600640 1 -- 172.21.15.69:0/2048937925 shutdown_connections 2024-04-24T08:20:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.194+0000 7fe1ef600640 1 --2- 172.21.15.69:0/2048937925 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1e80b46a0 0x7fe1e806f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.194+0000 7fe1ef600640 1 --2- 172.21.15.69:0/2048937925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e815c820 0x7fe1e80b3e90 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.194+0000 7fe1ef600640 1 -- 172.21.15.69:0/2048937925 >> 172.21.15.69:0/2048937925 conn(0x7fe1e80d0350 msgr2=0x7fe1e80d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.194+0000 7fe1ef600640 1 -- 172.21.15.69:0/2048937925 shutdown_connections 2024-04-24T08:20:28.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.194+0000 7fe1ef600640 1 -- 172.21.15.69:0/2048937925 wait complete. 2024-04-24T08:20:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.196+0000 7fe1ef600640 1 Processor -- start 2024-04-24T08:20:28.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.196+0000 7fe1ef600640 1 -- start start 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ef600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e806fd10 0x7fe1e81579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ef600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1e80b46a0 0x7fe1e815ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ef600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1e815c820 0x7fe1e8160460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ef600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e80b4b00 con 0x7fe1e806fd10 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ef600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e806f2b0 con 0x7fe1e815c820 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ef600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe1e80ce960 con 0x7fe1e80b46a0 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ecc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e806fd10 0x7fe1e81579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ecc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e806fd10 0x7fe1e81579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41014/0 (socket says 172.21.15.69:41014) 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ecc00640 1 -- 172.21.15.69:0/2299595854 learned_addr learned my addr 172.21.15.69:0/2299595854 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ecc00640 1 -- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1e80b46a0 msgr2=0x7fe1e815ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ecc00640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1e80b46a0 0x7fe1e815ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ecc00640 1 -- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1e815c820 msgr2=0x7fe1e8160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.197+0000 7fe1ecc00640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1e815c820 0x7fe1e8160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.198+0000 7fe1ecc00640 1 -- 172.21.15.69:0/2299595854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e815f5a0 con 0x7fe1e806fd10 2024-04-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.198+0000 7fe1ecc00640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e806fd10 0x7fe1e81579d0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8034040 tx=0x7fe1d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.198+0000 7fe1e5600640 1 -- 172.21.15.69:0/2299595854 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8041020 con 0x7fe1e806fd10 2024-04-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.198+0000 7fe1e5600640 1 -- 172.21.15.69:0/2299595854 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1d8007800 con 0x7fe1e806fd10 2024-04-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.198+0000 7fe1ef600640 1 -- 172.21.15.69:0/2299595854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e815f880 con 0x7fe1e806fd10 2024-04-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.198+0000 7fe1e5600640 1 -- 172.21.15.69:0/2299595854 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d8036040 con 0x7fe1e806fd10 2024-04-24T08:20:28.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.198+0000 7fe1ef600640 1 -- 172.21.15.69:0/2299595854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e81e6640 con 0x7fe1e806fd10 2024-04-24T08:20:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.199+0000 7fe1ef600640 1 -- 172.21.15.69:0/2299595854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e8068a80 con 0x7fe1e806fd10 2024-04-24T08:20:28.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.200+0000 7fe1e5600640 1 -- 172.21.15.69:0/2299595854 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe1d8044020 con 0x7fe1e806fd10 2024-04-24T08:20:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.201+0000 7fe1e5600640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe1c406c8f0 0x7fe1c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.201+0000 7fe1e5600640 1 -- 172.21.15.69:0/2299595854 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fe1d80acbb0 con 0x7fe1e806fd10 2024-04-24T08:20:28.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.202+0000 7fe1dfe00640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe1c406c8f0 0x7fe1c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.203+0000 7fe1dfe00640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe1c406c8f0 0x7fe1c406edb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0006fd0 tx=0x7fe1d0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.206+0000 7fe1e5600640 1 -- 172.21.15.69:0/2299595854 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe1d804b030 con 0x7fe1e806fd10 2024-04-24T08:20:28.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.406+0000 7fe1ef600640 1 -- 172.21.15.69:0/2299595854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe1e80cddc0 con 0x7fe1e806fd10 2024-04-24T08:20:28.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.407+0000 7fe1e5600640 1 -- 172.21.15.69:0/2299595854 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) ==== 93+0+10 (secure 0 0 0) 0x7fe1d807b000 con 0x7fe1e806fd10 2024-04-24T08:20:28.408 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:20:28.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 -- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe1c406c8f0 msgr2=0x7fe1c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe1c406c8f0 0x7fe1c406edb0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0006fd0 tx=0x7fe1d0008040 comp rx=0 tx=0).stop 2024-04-24T08:20:28.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 -- 172.21.15.69:0/2299595854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e806fd10 msgr2=0x7fe1e81579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e806fd10 0x7fe1e81579d0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8034040 tx=0x7fe1d8004a40 comp rx=0 tx=0).stop 2024-04-24T08:20:28.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 -- 172.21.15.69:0/2299595854 shutdown_connections 2024-04-24T08:20:28.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe1c406c8f0 0x7fe1c406edb0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1e815c820 0x7fe1e8160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe1e80b46a0 0x7fe1e815ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 --2- 172.21.15.69:0/2299595854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e806fd10 0x7fe1e81579d0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 -- 172.21.15.69:0/2299595854 >> 172.21.15.69:0/2299595854 conn(0x7fe1e80d0350 msgr2=0x7fe1e80cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 -- 172.21.15.69:0/2299595854 shutdown_connections 2024-04-24T08:20:28.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.410+0000 7fe1ef600640 1 -- 172.21.15.69:0/2299595854 wait complete. 2024-04-24T08:20:28.430 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:20:28.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.633+0000 7f2e81a00640 1 Processor -- start 2024-04-24T08:20:28.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.634+0000 7f2e81a00640 1 -- start start 2024-04-24T08:20:28.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.635+0000 7f2e81a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c15e980 0x7f2e7c162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.635+0000 7f2e81a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e7c15ed60 con 0x7f2e7c06cfb0 2024-04-24T08:20:28.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.635+0000 7f2e81a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e7c0d1580 con 0x7f2e7c0d0da0 2024-04-24T08:20:28.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.635+0000 7f2e81a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2e7c0d1860 con 0x7f2e7c15e980 2024-04-24T08:20:28.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.635+0000 7f2e81000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2e7c06cfb0 0x7f2e7c0d29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:56984/0 (socket says 172.21.15.69:56984) 2024-04-24T08:20:28.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e81000640 1 -- 172.21.15.69:0/3128053249 learned_addr learned my addr 172.21.15.69:0/3128053249 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7be00640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c15e980 0x7f2e7c162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693799450 0 0) 0x7f2e7c15ed60 con 0x7f2e7c06cfb0 2024-04-24T08:20:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e60003520 con 0x7f2e7c06cfb0 2024-04-24T08:20:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 199628648 0 0) 0x7f2e60003520 con 0x7f2e7c06cfb0 2024-04-24T08:20:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c15e980 msgr2=0x7f2e7c162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7aa00640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c15e980 0x7f2e7c162f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 >> v1:172.21.15.121:6789/0 conn(0x7f2e7c0d0da0 legacy=0x7f2e7c0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7c0cf450 con 0x7f2e7c06cfb0 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f2e740019f0 con 0x7f2e7c06cfb0 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 >> v1:172.21.15.69:6789/0 conn(0x7f2e7c06cfb0 legacy=0x7f2e7c0d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.636+0000 7f2e7aa00640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e600039e0 0x7f2e60003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e7aa00640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c15e980 0x7f2e60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e7aa00640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e60004d40 0x7f2e600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c0d1860 con 0x7f2e60004d40 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2e6000bb00 con 0x7f2e600039e0 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2e6000bd00 con 0x7f2e7c15e980 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e81000640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e60004d40 0x7f2e600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e7be00640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e600039e0 0x7f2e60003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e7b400640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c15e980 0x7f2e60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e81000640 1 -- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c15e980 msgr2=0x7f2e60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e81000640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c15e980 0x7f2e60004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e81000640 1 -- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e600039e0 msgr2=0x7f2e60003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e81000640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e600039e0 0x7f2e60003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.637+0000 7f2e81000640 1 -- 172.21.15.69:0/3128053249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e60003520 con 0x7f2e60004d40 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.638+0000 7f2e81000640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e60004d40 0x7f2e600051c0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f2e74002230 tx=0x7f2e60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88eeaa27b611bdeb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.639+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7400a4a0 con 0x7f2e60004d40 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.639+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e7402f070 con 0x7f2e60004d40 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.639+0000 7f2e7aa00640 1 -- 172.21.15.69:0/3128053249 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7402bcb0 con 0x7f2e60004d40 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.639+0000 7f2e81a00640 1 -- 172.21.15.69:0/3128053249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e60004d40 msgr2=0x7f2e600051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.639+0000 7f2e81a00640 1 --2- 172.21.15.69:0/3128053249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e60004d40 0x7f2e600051c0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f2e74002230 tx=0x7f2e60005830 comp rx=0 tx=0).stop 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.639+0000 7f2e81000640 1 -- 172.21.15.69:0/3128053249 reap_dead start 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.639+0000 7f2e81a00640 1 -- 172.21.15.69:0/3128053249 shutdown_connections 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.639+0000 7f2e81a00640 1 -- 172.21.15.69:0/3128053249 >> 172.21.15.69:0/3128053249 conn(0x7f2e7c09d920 msgr2=0x7f2e7c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.640+0000 7f2e81a00640 1 -- 172.21.15.69:0/3128053249 shutdown_connections 2024-04-24T08:20:28.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.640+0000 7f2e81a00640 1 -- 172.21.15.69:0/3128053249 wait complete. 2024-04-24T08:20:28.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.641+0000 7f2e81a00640 1 Processor -- start 2024-04-24T08:20:28.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.642+0000 7f2e81a00640 1 -- start start 2024-04-24T08:20:28.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.642+0000 7f2e81a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c153c20 0x7f2e7c0cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.642+0000 7f2e81a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e7c154000 0x7f2e7c0b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c0cb630 0x7f2e7c0b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c0d11e0 con 0x7f2e7c0cb630 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c0d0360 con 0x7f2e7c154000 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2e7c0d1360 con 0x7f2e7c153c20 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c0cb630 0x7f2e7c0b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c0cb630 0x7f2e7c0b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41038/0 (socket says 172.21.15.69:41038) 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81000640 1 -- 172.21.15.69:0/67525935 learned_addr learned my addr 172.21.15.69:0/67525935 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e7b400640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e7c154000 0x7f2e7c0b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e7be00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c153c20 0x7f2e7c0cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81000640 1 -- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c153c20 msgr2=0x7f2e7c0cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81000640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c153c20 0x7f2e7c0cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81000640 1 -- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e7c154000 msgr2=0x7f2e7c0b3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81000640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e7c154000 0x7f2e7c0b3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e81000640 1 -- 172.21.15.69:0/67525935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7c0cbbd0 con 0x7f2e7c0cb630 2024-04-24T08:20:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e7b400640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e7c154000 0x7f2e7c0b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.643+0000 7f2e7be00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c153c20 0x7f2e7c0cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.644+0000 7f2e81000640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c0cb630 0x7f2e7c0b3560 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f2e7402b500 tx=0x7f2e7402d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.644+0000 7f2e73e00640 1 -- 172.21.15.69:0/67525935 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7400a480 con 0x7f2e7c0cb630 2024-04-24T08:20:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.644+0000 7f2e73e00640 1 -- 172.21.15.69:0/67525935 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e74004b60 con 0x7f2e7c0cb630 2024-04-24T08:20:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.644+0000 7f2e81a00640 1 -- 172.21.15.69:0/67525935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7c0cbeb0 con 0x7f2e7c0cb630 2024-04-24T08:20:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.644+0000 7f2e73e00640 1 -- 172.21.15.69:0/67525935 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e74034470 con 0x7f2e7c0cb630 2024-04-24T08:20:28.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.644+0000 7f2e81a00640 1 -- 172.21.15.69:0/67525935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e7c0cc470 con 0x7f2e7c0cb630 2024-04-24T08:20:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.645+0000 7f2e81a00640 1 -- 172.21.15.69:0/67525935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e7c075d50 con 0x7f2e7c0cb630 2024-04-24T08:20:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.646+0000 7f2e73e00640 1 -- 172.21.15.69:0/67525935 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2e74034610 con 0x7f2e7c0cb630 2024-04-24T08:20:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.647+0000 7f2e73e00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2e5c06c8f0 0x7f2e5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.647+0000 7f2e73e00640 1 -- 172.21.15.69:0/67525935 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f2e740abb60 con 0x7f2e7c0cb630 2024-04-24T08:20:28.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.648+0000 7f2e7be00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2e5c06c8f0 0x7f2e5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.649+0000 7f2e7be00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2e5c06c8f0 0x7f2e5c06edb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009a00 tx=0x7f2e6c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.651+0000 7f2e73e00640 1 -- 172.21.15.69:0/67525935 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e74079e60 con 0x7f2e7c0cb630 2024-04-24T08:20:28.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.851+0000 7f2e81a00640 1 -- 172.21.15.69:0/67525935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2e7c0d1e40 con 0x7f2e7c0cb630 2024-04-24T08:20:28.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.852+0000 7f2e73e00640 1 -- 172.21.15.69:0/67525935 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) ==== 104+0+11 (secure 0 0 0) 0x7f2e74079760 con 0x7f2e7c0cb630 2024-04-24T08:20:28.852 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 -- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2e5c06c8f0 msgr2=0x7f2e5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2e5c06c8f0 0x7f2e5c06edb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c009a00 tx=0x7f2e6c008040 comp rx=0 tx=0).stop 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 -- 172.21.15.69:0/67525935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c0cb630 msgr2=0x7f2e7c0b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c0cb630 0x7f2e7c0b3560 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f2e7402b500 tx=0x7f2e7402d070 comp rx=0 tx=0).stop 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 -- 172.21.15.69:0/67525935 shutdown_connections 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2e5c06c8f0 0x7f2e5c06edb0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e7c0cb630 0x7f2e7c0b3560 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2e7c154000 0x7f2e7c0b3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 --2- 172.21.15.69:0/67525935 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2e7c153c20 0x7f2e7c0cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 -- 172.21.15.69:0/67525935 >> 172.21.15.69:0/67525935 conn(0x7f2e7c09d920 msgr2=0x7f2e7c1545d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 -- 172.21.15.69:0/67525935 shutdown_connections 2024-04-24T08:20:28.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:28.855+0000 7f2e81a00640 1 -- 172.21.15.69:0/67525935 wait complete. 2024-04-24T08:20:28.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:20:29.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.075+0000 7f2a4f200640 1 Processor -- start 2024-04-24T08:20:29.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.075+0000 7f2a4f200640 1 -- start start 2024-04-24T08:20:29.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.077+0000 7f2a4f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480d1960 0x7f2a4806cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.077+0000 7f2a4f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2a4806d190 0x7f2a480d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.077+0000 7f2a4f200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a480d1f20 con 0x7f2a4815ea70 2024-04-24T08:20:29.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.077+0000 7f2a4f200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f2a480cf810 con 0x7f2a4806d190 2024-04-24T08:20:29.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.077+0000 7f2a4f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a480cfaf0 con 0x7f2a480d1960 2024-04-24T08:20:29.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.077+0000 7f2a47e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2a4815ea70 0x7f2a480d5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39624/0 (socket says 172.21.15.69:39624) 2024-04-24T08:20:29.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.077+0000 7f2a47e00640 1 -- 172.21.15.69:0/688218924 learned_addr learned my addr 172.21.15.69:0/688218924 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.077+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374763502 0 0) 0x7f2a480d1f20 con 0x7f2a4815ea70 2024-04-24T08:20:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a2c003520 con 0x7f2a4815ea70 2024-04-24T08:20:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a47400640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480d1960 0x7f2a4806cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a4d200640 1 --2- 172.21.15.69:0/688218924 >> v2:172.21.15.121:3300/0 conn(0x7f2a4806d190 0x7f2a480d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 858413189 0 0) 0x7f2a2c003520 con 0x7f2a4815ea70 2024-04-24T08:20:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480d1960 msgr2=0x7f2a4806cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480d1960 0x7f2a4806cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 >> v2:172.21.15.121:3300/0 conn(0x7f2a4806d190 msgr2=0x7f2a480d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 --2- 172.21.15.69:0/688218924 >> v2:172.21.15.121:3300/0 conn(0x7f2a4806d190 0x7f2a480d02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a480cfdd0 con 0x7f2a4815ea70 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f2a38001670 con 0x7f2a4815ea70 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 >> v1:172.21.15.69:6789/0 conn(0x7f2a4815ea70 legacy=0x7f2a480d5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a2c0039b0 0x7f2a2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 0x7f2a2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480d1960 0x7f2a2c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a480cf810 con 0x7f2a2c004800 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2a480cfaf0 con 0x7f2a2c0039b0 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a2c00bac0 con 0x7f2a480d1960 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a4d200640 1 --2- 172.21.15.69:0/688218924 >> v2:172.21.15.121:3300/0 conn(0x7f2a4806d190 0x7f2a480d02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.078+0000 7f2a47400640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 0x7f2a2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.079+0000 7f2a47e00640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a2c0039b0 0x7f2a2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.079+0000 7f2a47400640 1 -- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480d1960 msgr2=0x7f2a2c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.079+0000 7f2a4d200640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480d1960 0x7f2a2c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.079+0000 7f2a47400640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480d1960 0x7f2a2c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.079+0000 7f2a47400640 1 -- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a2c0039b0 msgr2=0x7f2a2c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.079+0000 7f2a47400640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a2c0039b0 0x7f2a2c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.079+0000 7f2a47400640 1 -- 172.21.15.69:0/688218924 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2c003520 con 0x7f2a2c004800 2024-04-24T08:20:29.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.079+0000 7f2a47e00640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a2c0039b0 0x7f2a2c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:29.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.080+0000 7f2a47400640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 0x7f2a2c004c80 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400ba00 tx=0x7f2a2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881e55a7de9bd015 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.080+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a34030070 con 0x7f2a2c004800 2024-04-24T08:20:29.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.080+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a3402ada0 con 0x7f2a2c004800 2024-04-24T08:20:29.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.080+0000 7f2a46a00640 1 -- 172.21.15.69:0/688218924 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3400b600 con 0x7f2a2c004800 2024-04-24T08:20:29.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.081+0000 7f2a4f200640 1 -- 172.21.15.69:0/688218924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 msgr2=0x7f2a2c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.081+0000 7f2a4f200640 1 --2- 172.21.15.69:0/688218924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 0x7f2a2c004c80 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400ba00 tx=0x7f2a2c005830 comp rx=0 tx=0).stop 2024-04-24T08:20:29.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.081+0000 7f2a4d200640 1 -- 172.21.15.69:0/688218924 reap_dead start 2024-04-24T08:20:29.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.081+0000 7f2a4f200640 1 -- 172.21.15.69:0/688218924 shutdown_connections 2024-04-24T08:20:29.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.081+0000 7f2a4f200640 1 -- 172.21.15.69:0/688218924 >> 172.21.15.69:0/688218924 conn(0x7f2a4808bf60 msgr2=0x7f2a480d46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.081+0000 7f2a4f200640 1 -- 172.21.15.69:0/688218924 shutdown_connections 2024-04-24T08:20:29.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.081+0000 7f2a4f200640 1 -- 172.21.15.69:0/688218924 wait complete. 2024-04-24T08:20:29.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.083+0000 7f2a4f200640 1 Processor -- start 2024-04-24T08:20:29.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.083+0000 7f2a4f200640 1 -- start start 2024-04-24T08:20:29.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.084+0000 7f2a4f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 0x7f2a480b9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.084+0000 7f2a4f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a480cc820 0x7f2a48153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.084+0000 7f2a4f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480cd8e0 0x7f2a480cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.084+0000 7f2a4f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4806cc50 con 0x7f2a2c004800 2024-04-24T08:20:29.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.084+0000 7f2a4f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2a480cf4c0 con 0x7f2a480cc820 2024-04-24T08:20:29.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.084+0000 7f2a4f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2a4806cdd0 con 0x7f2a480cd8e0 2024-04-24T08:20:29.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.084+0000 7f2a47e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 0x7f2a480b9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.084+0000 7f2a47e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 0x7f2a480b9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42676/0 (socket says 172.21.15.69:42676) 2024-04-24T08:20:29.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.084+0000 7f2a47e00640 1 -- 172.21.15.69:0/592606560 learned_addr learned my addr 172.21.15.69:0/592606560 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:29.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a47400640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a480cc820 0x7f2a48153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a47e00640 1 -- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480cd8e0 msgr2=0x7f2a480cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a47e00640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480cd8e0 0x7f2a480cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a47e00640 1 -- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a480cc820 msgr2=0x7f2a48153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a47e00640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a480cc820 0x7f2a48153f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a47e00640 1 -- 172.21.15.69:0/592606560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a480ca220 con 0x7f2a2c004800 2024-04-24T08:20:29.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a47e00640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 0x7f2a480b9c80 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f2a3800d860 tx=0x7f2a3800dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a3fe00640 1 -- 172.21.15.69:0/592606560 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38019070 con 0x7f2a2c004800 2024-04-24T08:20:29.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a3fe00640 1 -- 172.21.15.69:0/592606560 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a380074e0 con 0x7f2a2c004800 2024-04-24T08:20:29.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a4f200640 1 -- 172.21.15.69:0/592606560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a480ca500 con 0x7f2a2c004800 2024-04-24T08:20:29.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.085+0000 7f2a3fe00640 1 -- 172.21.15.69:0/592606560 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a380049e0 con 0x7f2a2c004800 2024-04-24T08:20:29.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.086+0000 7f2a4f200640 1 -- 172.21.15.69:0/592606560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a480caac0 con 0x7f2a2c004800 2024-04-24T08:20:29.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.086+0000 7f2a4f200640 1 -- 172.21.15.69:0/592606560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a48068a80 con 0x7f2a2c004800 2024-04-24T08:20:29.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.088+0000 7f2a3fe00640 1 -- 172.21.15.69:0/592606560 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a3801c070 con 0x7f2a2c004800 2024-04-24T08:20:29.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.088+0000 7f2a3fe00640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a2806c8f0 0x7f2a2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.089+0000 7f2a3fe00640 1 -- 172.21.15.69:0/592606560 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f2a3808b600 con 0x7f2a2c004800 2024-04-24T08:20:29.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.089+0000 7f2a47400640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a2806c8f0 0x7f2a2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.090+0000 7f2a47400640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a2806c8f0 0x7f2a2806edb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0057e0 tx=0x7f2a34030000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.093+0000 7f2a3fe00640 1 -- 172.21.15.69:0/592606560 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a38059a20 con 0x7f2a2c004800 2024-04-24T08:20:29.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.276+0000 7f2a4f200640 1 -- 172.21.15.69:0/592606560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2a480b3ee0 con 0x7f2a2c004800 2024-04-24T08:20:29.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.277+0000 7f2a3fe00640 1 -- 172.21.15.69:0/592606560 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) ==== 100+0+11 (secure 0 0 0) 0x7f2a38059320 con 0x7f2a2c004800 2024-04-24T08:20:29.278 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:20:29.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.279+0000 7f2a4f200640 1 -- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a2806c8f0 msgr2=0x7f2a2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a2806c8f0 0x7f2a2806edb0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c0057e0 tx=0x7f2a34030000 comp rx=0 tx=0).stop 2024-04-24T08:20:29.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 -- 172.21.15.69:0/592606560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 msgr2=0x7f2a480b9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 0x7f2a480b9c80 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f2a3800d860 tx=0x7f2a3800dd20 comp rx=0 tx=0).stop 2024-04-24T08:20:29.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 -- 172.21.15.69:0/592606560 shutdown_connections 2024-04-24T08:20:29.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2a2806c8f0 0x7f2a2806edb0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2a480cd8e0 0x7f2a480cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2a480cc820 0x7f2a48153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 --2- 172.21.15.69:0/592606560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a2c004800 0x7f2a480b9c80 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 -- 172.21.15.69:0/592606560 >> 172.21.15.69:0/592606560 conn(0x7f2a4808bf60 msgr2=0x7f2a4815d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 -- 172.21.15.69:0/592606560 shutdown_connections 2024-04-24T08:20:29.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.280+0000 7f2a4f200640 1 -- 172.21.15.69:0/592606560 wait complete. 2024-04-24T08:20:29.298 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:20:29.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.486+0000 7ff7e4200640 1 Processor -- start 2024-04-24T08:20:29.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.486+0000 7ff7e4200640 1 -- start start 2024-04-24T08:20:29.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.487+0000 7ff7e4200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc06f730 0x7ff7dc0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.487+0000 7ff7e4200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7dc0b4370 con 0x7ff7dc0b3320 2024-04-24T08:20:29.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.487+0000 7ff7e4200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7dc0b4650 con 0x7ff7dc071d70 2024-04-24T08:20:29.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.487+0000 7ff7e4200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff7dc0b4f80 con 0x7ff7dc06f730 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.488+0000 7ff7e2200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff7dc0b3320 0x7ff7dc0ce180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:39630/0 (socket says 172.21.15.69:39630) 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.488+0000 7ff7e2200640 1 -- 172.21.15.69:0/751035933 learned_addr learned my addr 172.21.15.69:0/751035933 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.488+0000 7ff7e0e00640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc06f730 0x7ff7dc0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.488+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 255277996 0 0) 0x7ff7dc0b4370 con 0x7ff7dc0b3320 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.488+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7c4004630 con 0x7ff7dc0b3320 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3806387693 0 0) 0x7ff7c4004630 con 0x7ff7dc0b3320 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc06f730 msgr2=0x7ff7dc0cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc06f730 0x7ff7dc0cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 >> v1:172.21.15.121:6789/0 conn(0x7ff7dc071d70 legacy=0x7ff7dc072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7dc0b4930 con 0x7ff7dc0b3320 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7ff7d40019f0 con 0x7ff7dc0b3320 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7e0e00640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc06f730 0x7ff7dc0cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 >> v1:172.21.15.69:6789/0 conn(0x7ff7dc0b3320 legacy=0x7ff7dc0ce180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 0x7ff7c4004f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc06f730 0x7ff7c4005910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7c4005e50 0x7ff7c40062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc0b4f80 con 0x7ff7c4004af0 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c400cc60 con 0x7ff7c4005e50 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff7c400ce30 con 0x7ff7dc06f730 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7e1800640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 0x7ff7c4004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7e2200640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7c4005e50 0x7ff7c40062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7e1800640 1 -- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc06f730 msgr2=0x7ff7c4005910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7e1800640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc06f730 0x7ff7c4005910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.489+0000 7ff7e1800640 1 -- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7c4005e50 msgr2=0x7ff7c40062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.490+0000 7ff7e1800640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7c4005e50 0x7ff7c40062d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.490+0000 7ff7e1800640 1 -- 172.21.15.69:0/751035933 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c4004630 con 0x7ff7c4004af0 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.491+0000 7ff7e1800640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 0x7ff7c4004f10 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0001750 tx=0x7ff7c4006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb5cff8f3fb4da0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.491+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d0001370 con 0x7ff7c4004af0 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.491+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7d00052d0 con 0x7ff7c4004af0 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.491+0000 7ff7dbe00640 1 -- 172.21.15.69:0/751035933 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d002aba0 con 0x7ff7c4004af0 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.491+0000 7ff7e4200640 1 -- 172.21.15.69:0/751035933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 msgr2=0x7ff7c4004f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.491+0000 7ff7e4200640 1 --2- 172.21.15.69:0/751035933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 0x7ff7c4004f10 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0001750 tx=0x7ff7c4006940 comp rx=0 tx=0).stop 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.491+0000 7ff7e2200640 1 -- 172.21.15.69:0/751035933 reap_dead start 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.492+0000 7ff7e4200640 1 -- 172.21.15.69:0/751035933 shutdown_connections 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.492+0000 7ff7e4200640 1 -- 172.21.15.69:0/751035933 >> 172.21.15.69:0/751035933 conn(0x7ff7dc067d90 msgr2=0x7ff7dc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.492+0000 7ff7e4200640 1 -- 172.21.15.69:0/751035933 shutdown_connections 2024-04-24T08:20:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.492+0000 7ff7e4200640 1 -- 172.21.15.69:0/751035933 wait complete. 2024-04-24T08:20:29.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.493+0000 7ff7e4200640 1 Processor -- start 2024-04-24T08:20:29.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.494+0000 7ff7e4200640 1 -- start start 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e4200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 0x7ff7dc15bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e4200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc15dca0 0x7ff7dc0cac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e4200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7dc1563f0 0x7ff7dc156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e4200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc15b040 con 0x7ff7c4004af0 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e4200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff7dc06ee60 con 0x7ff7dc1563f0 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e4200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff7dc15b1c0 con 0x7ff7dc15dca0 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 0x7ff7dc15bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 0x7ff7dc15bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42688/0 (socket says 172.21.15.69:42688) 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e1800640 1 -- 172.21.15.69:0/2864881680 learned_addr learned my addr 172.21.15.69:0/2864881680 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e1800640 1 -- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc15dca0 msgr2=0x7ff7dc0cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e1800640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc15dca0 0x7ff7dc0cac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e1800640 1 -- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7dc1563f0 msgr2=0x7ff7dc156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e1800640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7dc1563f0 0x7ff7dc156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.495+0000 7ff7e1800640 1 -- 172.21.15.69:0/2864881680 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7dc15c2f0 con 0x7ff7c4004af0 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.496+0000 7ff7e1800640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 0x7ff7dc15bdb0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0008fd0 tx=0x7ff7d0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.496+0000 7ff7d9600640 1 -- 172.21.15.69:0/2864881680 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d0042070 con 0x7ff7c4004af0 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.496+0000 7ff7d9600640 1 -- 172.21.15.69:0/2864881680 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7d0005480 con 0x7ff7c4004af0 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.496+0000 7ff7e4200640 1 -- 172.21.15.69:0/2864881680 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7dc15c590 con 0x7ff7c4004af0 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.496+0000 7ff7d9600640 1 -- 172.21.15.69:0/2864881680 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d0037040 con 0x7ff7c4004af0 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.496+0000 7ff7e4200640 1 -- 172.21.15.69:0/2864881680 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7dc15cb50 con 0x7ff7c4004af0 2024-04-24T08:20:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.497+0000 7ff7e4200640 1 -- 172.21.15.69:0/2864881680 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7dc068a80 con 0x7ff7c4004af0 2024-04-24T08:20:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.498+0000 7ff7d9600640 1 -- 172.21.15.69:0/2864881680 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff7d0045070 con 0x7ff7c4004af0 2024-04-24T08:20:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.499+0000 7ff7d9600640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7bc06c8f0 0x7ff7bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.499+0000 7ff7d9600640 1 -- 172.21.15.69:0/2864881680 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7ff7d00ab320 con 0x7ff7c4004af0 2024-04-24T08:20:29.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.500+0000 7ff7e0e00640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7bc06c8f0 0x7ff7bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.501+0000 7ff7e0e00640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7bc06c8f0 0x7ff7bc06edb0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8006ce0 tx=0x7ff7c8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.503+0000 7ff7d9600640 1 -- 172.21.15.69:0/2864881680 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff7d00796c0 con 0x7ff7c4004af0 2024-04-24T08:20:29.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.715+0000 7ff7e4200640 1 -- 172.21.15.69:0/2864881680 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff7dc073020 con 0x7ff7c4004af0 2024-04-24T08:20:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.716+0000 7ff7d9600640 1 -- 172.21.15.69:0/2864881680 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v109) ==== 80+0+0 (secure 0 0 0) 0x7ff7d0078fc0 con 0x7ff7c4004af0 2024-04-24T08:20:29.717 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 -- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7bc06c8f0 msgr2=0x7ff7bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7bc06c8f0 0x7ff7bc06edb0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8006ce0 tx=0x7ff7c8008040 comp rx=0 tx=0).stop 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 -- 172.21.15.69:0/2864881680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 msgr2=0x7ff7dc15bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 0x7ff7dc15bdb0 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7ff7d0008fd0 tx=0x7ff7d0001120 comp rx=0 tx=0).stop 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 -- 172.21.15.69:0/2864881680 shutdown_connections 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff7bc06c8f0 0x7ff7bc06edb0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff7dc1563f0 0x7ff7dc156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff7dc15dca0 0x7ff7dc0cac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 --2- 172.21.15.69:0/2864881680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff7c4004af0 0x7ff7dc15bdb0 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 -- 172.21.15.69:0/2864881680 >> 172.21.15.69:0/2864881680 conn(0x7ff7dc067d90 msgr2=0x7ff7dc0c5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 -- 172.21.15.69:0/2864881680 shutdown_connections 2024-04-24T08:20:29.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:29.719+0000 7ff7e4200640 1 -- 172.21.15.69:0/2864881680 wait complete. 2024-04-24T08:20:29.737 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:20:29.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:29.738 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:29.815 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[75282]: fuse finished with error 0 and tester_r 0 2024-04-24T08:20:29.819 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:20:35.920 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:20:35.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:35.921 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:20:35.980 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-04-24T08:20:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.187+0000 7f8a1f200640 1 Processor -- start 2024-04-24T08:20:36.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.187+0000 7f8a1f200640 1 -- start start 2024-04-24T08:20:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.188+0000 7f8a1f200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8a2015d030 0x7f8a20163b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1f200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a20072220 0x7f8a20166de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1f200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a201640a0 con 0x7f8a20071d70 2024-04-24T08:20:36.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1f200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f8a200b39e0 con 0x7f8a20072220 2024-04-24T08:20:36.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1f200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f8a20072690 con 0x7f8a2015d030 2024-04-24T08:20:36.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a20072220 0x7f8a20166de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1e800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8a20072220 0x7f8a20166de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42704/0 (socket says 172.21.15.69:42704) 2024-04-24T08:20:36.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1e800640 1 -- 172.21.15.69:0/3496761584 learned_addr learned my addr 172.21.15.69:0/3496761584 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:36.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1e800640 1 -- 172.21.15.69:0/3496761584 >> v2:172.21.15.121:3300/0 conn(0x7f8a2015d030 msgr2=0x7f8a20163b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1e800640 1 --2- 172.21.15.69:0/3496761584 >> v2:172.21.15.121:3300/0 conn(0x7f8a2015d030 0x7f8a20163b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1e800640 1 -- 172.21.15.69:0/3496761584 >> v1:172.21.15.121:6789/0 conn(0x7f8a20071d70 legacy=0x7f8a201666d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.189+0000 7f8a1e800640 1 -- 172.21.15.69:0/3496761584 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a200b3700 con 0x7f8a20072220 2024-04-24T08:20:36.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.190+0000 7f8a1e800640 1 --2- 172.21.15.69:0/3496761584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20072220 0x7f8a20166de0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f8a2006ffa0 tx=0x7f8a1802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc5eed59070116b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:36.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.191+0000 7f8a1d400640 1 -- 172.21.15.69:0/3496761584 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a18036070 con 0x7f8a20072220 2024-04-24T08:20:36.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.191+0000 7f8a1d400640 1 -- 172.21.15.69:0/3496761584 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a18031600 con 0x7f8a20072220 2024-04-24T08:20:36.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.191+0000 7f8a1d400640 1 -- 172.21.15.69:0/3496761584 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180319d0 con 0x7f8a20072220 2024-04-24T08:20:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.191+0000 7f8a1f200640 1 -- 172.21.15.69:0/3496761584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20072220 msgr2=0x7f8a20166de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.191+0000 7f8a1f200640 1 --2- 172.21.15.69:0/3496761584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20072220 0x7f8a20166de0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f8a2006ffa0 tx=0x7f8a1802a6f0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.191+0000 7f8a1f200640 1 -- 172.21.15.69:0/3496761584 shutdown_connections 2024-04-24T08:20:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.191+0000 7f8a1f200640 1 --2- 172.21.15.69:0/3496761584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20072220 0x7f8a20166de0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.191+0000 7f8a1f200640 1 --2- 172.21.15.69:0/3496761584 >> v2:172.21.15.121:3300/0 conn(0x7f8a2015d030 0x7f8a20163b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.192+0000 7f8a1f200640 1 -- 172.21.15.69:0/3496761584 >> 172.21.15.69:0/3496761584 conn(0x7f8a20067b80 msgr2=0x7f8a2006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:36.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.192+0000 7f8a1f200640 1 -- 172.21.15.69:0/3496761584 shutdown_connections 2024-04-24T08:20:36.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.192+0000 7f8a1f200640 1 -- 172.21.15.69:0/3496761584 wait complete. 2024-04-24T08:20:36.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.193+0000 7f8a1f200640 1 Processor -- start 2024-04-24T08:20:36.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.193+0000 7f8a1f200640 1 -- start start 2024-04-24T08:20:36.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.194+0000 7f8a1f200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a20071d70 0x7f8a20156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.194+0000 7f8a1f200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a20072220 0x7f8a200b1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.194+0000 7f8a1f200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a2015d030 0x7f8a20150260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.194+0000 7f8a1f200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a200b43e0 con 0x7f8a2015d030 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.194+0000 7f8a1f200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8a200b4100 con 0x7f8a20071d70 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.194+0000 7f8a1f200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8a2006f7b0 con 0x7f8a20072220 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a2015d030 0x7f8a20150260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a2015d030 0x7f8a20150260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42708/0 (socket says 172.21.15.69:42708) 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1e800640 1 -- 172.21.15.69:0/2256401407 learned_addr learned my addr 172.21.15.69:0/2256401407 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1de00640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a20071d70 0x7f8a20156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1e800640 1 -- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a20072220 msgr2=0x7f8a200b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a15400640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a20072220 0x7f8a200b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1e800640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a20072220 0x7f8a200b1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1e800640 1 -- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a20071d70 msgr2=0x7f8a20156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1e800640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a20071d70 0x7f8a20156030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1e800640 1 -- 172.21.15.69:0/2256401407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a200b0090 con 0x7f8a2015d030 2024-04-24T08:20:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1de00640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a20071d70 0x7f8a20156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.195+0000 7f8a1e800640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a2015d030 0x7f8a20150260 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f8a18034040 tx=0x7f8a18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.196+0000 7f8a16a00640 1 -- 172.21.15.69:0/2256401407 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a18041020 con 0x7f8a2015d030 2024-04-24T08:20:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.196+0000 7f8a16a00640 1 -- 172.21.15.69:0/2256401407 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a18007780 con 0x7f8a2015d030 2024-04-24T08:20:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.196+0000 7f8a16a00640 1 -- 172.21.15.69:0/2256401407 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a18036040 con 0x7f8a2015d030 2024-04-24T08:20:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.196+0000 7f8a1f200640 1 -- 172.21.15.69:0/2256401407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a200b0370 con 0x7f8a2015d030 2024-04-24T08:20:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.196+0000 7f8a1f200640 1 -- 172.21.15.69:0/2256401407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a200b0930 con 0x7f8a2015d030 2024-04-24T08:20:36.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.198+0000 7f8a16a00640 1 -- 172.21.15.69:0/2256401407 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8a18044020 con 0x7f8a2015d030 2024-04-24T08:20:36.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.199+0000 7f8a16a00640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89fc06c920 0x7f89fc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.199+0000 7f8a16a00640 1 -- 172.21.15.69:0/2256401407 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f8a180ac660 con 0x7f8a2015d030 2024-04-24T08:20:36.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.200+0000 7f8a1de00640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89fc06c920 0x7f89fc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.200+0000 7f8a27c23740 1 Processor -- start 2024-04-24T08:20:36.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.200+0000 7f8a27c23740 1 -- start start 2024-04-24T08:20:36.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564887579e00 0x56488752c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x56488755c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488755c7e0 0x5648875315f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a27c23740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5648874961f0 con 0x56488755cf20 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a27c23740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56488743c350 con 0x56488755c7e0 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a27c23740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5648874921b0 con 0x564887579e00 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a1de00640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89fc06c920 0x7f89fc06ede0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f8a10006fd0 tx=0x7f8a10008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a15400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x56488755c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a15400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x56488755c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42710/0 (socket says 172.21.15.69:42710) 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a15400640 1 -- 172.21.15.69:0/1362118243 learned_addr learned my addr 172.21.15.69:0/1362118243 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.201+0000 7f8a1e800640 1 --2- 172.21.15.69:0/1362118243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488755c7e0 0x5648875315f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a1de00640 1 --2- 172.21.15.69:0/1362118243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564887579e00 0x56488752c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a15400640 1 -- 172.21.15.69:0/1362118243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564887579e00 msgr2=0x56488752c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a15400640 1 --2- 172.21.15.69:0/1362118243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564887579e00 0x56488752c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a15400640 1 -- 172.21.15.69:0/1362118243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488755c7e0 msgr2=0x5648875315f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a15400640 1 --2- 172.21.15.69:0/1362118243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488755c7e0 0x5648875315f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a15400640 1 -- 172.21.15.69:0/1362118243 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564887490f50 con 0x56488755cf20 2024-04-24T08:20:36.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a1e800640 1 --2- 172.21.15.69:0/1362118243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488755c7e0 0x5648875315f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:36.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a15400640 1 --2- 172.21.15.69:0/1362118243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x56488755c2a0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800a790 tx=0x7f8a0800ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e57cda8f3d4017 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:36.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a0ea00640 1 -- 172.21.15.69:0/1362118243 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08015070 con 0x56488755cf20 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a0ea00640 1 -- 172.21.15.69:0/1362118243 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a0800cb40 con 0x56488755cf20 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.202+0000 7f8a0ea00640 1 -- 172.21.15.69:0/1362118243 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a080113f0 con 0x56488755cf20 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.203+0000 7f8a27c23740 1 -- 172.21.15.69:0/1362118243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 msgr2=0x56488755c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.203+0000 7f8a27c23740 1 --2- 172.21.15.69:0/1362118243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x56488755c2a0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800a790 tx=0x7f8a0800ac50 comp rx=0 tx=0).stop 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.203+0000 7f8a27c23740 1 -- 172.21.15.69:0/1362118243 shutdown_connections 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.203+0000 7f8a27c23740 1 --2- 172.21.15.69:0/1362118243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x56488755c2a0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.203+0000 7f8a27c23740 1 --2- 172.21.15.69:0/1362118243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488755c7e0 0x5648875315f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.203+0000 7f8a27c23740 1 --2- 172.21.15.69:0/1362118243 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564887579e00 0x56488752c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.203+0000 7f8a27c23740 1 -- 172.21.15.69:0/1362118243 >> 172.21.15.69:0/1362118243 conn(0x56488752ea00 msgr2=0x5648875772a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.203+0000 7f8a27c23740 1 -- 172.21.15.69:0/1362118243 shutdown_connections 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.203+0000 7f8a27c23740 1 -- 172.21.15.69:0/1362118243 wait complete. 2024-04-24T08:20:36.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.204+0000 7f8a27c23740 1 -- start start 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.204+0000 7f8a27c23740 1 Processor -- start 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.205+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56488755cf20 0x56488756f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.205+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564887579e00 0x564887557d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.205+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488758b6b0 0x56488757ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.205+0000 7f8a27c23740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564887530bb0 con 0x564887579e00 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.205+0000 7f8a27c23740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x56488744f690 con 0x56488758b6b0 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.205+0000 7f8a27c23740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x56488752ffa0 con 0x56488755cf20 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.205+0000 7f8a27c23740 10 client.0 _ll_register_callbacks cb 0x56488748b650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.205+0000 7f8a15400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564887579e00 0x564887557d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.205+0000 7f8a15400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564887579e00 0x564887557d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42720/0 (socket says 172.21.15.69:42720) 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.205+0000 7f8a15400640 1 -- 172.21.15.69:0/2248154196 learned_addr learned my addr 172.21.15.69:0/2248154196 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a1e800640 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488758b6b0 0x56488757ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a1de00640 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56488755cf20 0x56488756f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a15400640 1 -- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56488755cf20 msgr2=0x56488756f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a15400640 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56488755cf20 0x56488756f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a15400640 1 -- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488758b6b0 msgr2=0x56488757ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a15400640 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488758b6b0 0x56488757ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a15400640 1 -- 172.21.15.69:0/2248154196 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56488744e920 con 0x564887579e00 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a15400640 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564887579e00 0x564887557d90 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800a140 tx=0x7f8a08004640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfff9029e6b45bd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a03400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a03400640 1 -- 172.21.15.69:0/2248154196 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0802a060 con 0x564887579e00 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a03400640 1 -- 172.21.15.69:0/2248154196 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f8a0800c940 con 0x564887579e00 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.206+0000 7f8a27c23740 1 -- 172.21.15.69:0/2248154196 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a0800cb40 con 0x564887579e00 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a03400640 1 -- 172.21.15.69:0/2248154196 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f8a08030050 con 0x564887579e00 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a27c23740 10 client.9528 fetch_fsmap learned FSMap version 99 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a27c23740 1 -- 172.21.15.69:0/2248154196 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x5648872727d0 con 0x564887579e00 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a03400640 1 -- 172.21.15.69:0/2248154196 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a0800cb40 con 0x564887579e00 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a03400640 1 -- 172.21.15.69:0/2248154196 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f8a08033020 con 0x564887579e00 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a27c23740 10 client.9528 fetch_fsmap finished waiting for FSMap version 99 2024-04-24T08:20:36.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a27c23740 10 client.9528 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.b 2024-04-24T08:20:36.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a27c23740 20 client.9528 populate_metadata read hostname 'smithi069' 2024-04-24T08:20:36.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x56488756df60 0x56488756e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a27c23740 4 client.9528 mds_command: new command op to 9288 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:36.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.207+0000 7f8a27c23740 1 -- 172.21.15.69:0/2248154196 --> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648874a3c10 con 0x56488756df60 2024-04-24T08:20:36.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.208+0000 7f8a1de00640 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x56488756df60 0x56488756e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.209+0000 7f8a1de00640 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x56488756df60 0x56488756e3a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d51728ba7292032 server_cookie=ee853c26204e9bc8 in_seq=0 out_seq=0 2024-04-24T08:20:36.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.209+0000 7f8a03400640 10 client.9528 ms_handle_connect on v2:172.21.15.121:6834/1515102940 2024-04-24T08:20:36.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a03400640 1 -- 172.21.15.69:0/2248154196 <== mds.0 v2:172.21.15.121:6834/1515102940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5648874a3c10 con 0x56488756df60 2024-04-24T08:20:36.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a03400640 10 client.9528 handle_command_reply: tid=0 2024-04-24T08:20:36.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 client.9528 shutdown 2024-04-24T08:20:36.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 -- 172.21.15.69:0/2248154196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564887579e00 msgr2=0x564887557d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564887579e00 0x564887557d90 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800a140 tx=0x7f8a08004640 comp rx=0 tx=0).stop 2024-04-24T08:20:36.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 -- 172.21.15.69:0/2248154196 shutdown_connections 2024-04-24T08:20:36.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x56488756df60 0x56488756e3a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564887579e00 0x564887557d90 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488758b6b0 0x56488757ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a03400640 0 client.9528 ms_handle_reset on v2:172.21.15.121:6834/1515102940 2024-04-24T08:20:36.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2248154196 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56488755cf20 0x56488756f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 -- 172.21.15.69:0/2248154196 >> 172.21.15.69:0/2248154196 conn(0x56488755c7e0 msgr2=0x564887577250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:36.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 -- 172.21.15.69:0/2248154196 shutdown_connections 2024-04-24T08:20:36.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 1 -- 172.21.15.69:0/2248154196 wait complete. 2024-04-24T08:20:36.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.211+0000 7f8a27c23740 20 client.9528 trim_cache size 0 max 16384 2024-04-24T08:20:36.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.227+0000 7f8a27c23740 1 Processor -- start 2024-04-24T08:20:36.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.228+0000 7f8a27c23740 1 -- start start 2024-04-24T08:20:36.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.228+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x5648875322a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.228+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564887579e00 0x5648875327e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.228+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56488758b6b0 0x5648875770a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.228+0000 7f8a27c23740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a202051d0 con 0x56488755cf20 2024-04-24T08:20:36.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.228+0000 7f8a27c23740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8a202042c0 con 0x564887579e00 2024-04-24T08:20:36.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.228+0000 7f8a27c23740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8a201ec3d0 con 0x56488758b6b0 2024-04-24T08:20:36.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.229+0000 7f8a1de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x5648875322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.229+0000 7f8a1de00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x5648875322a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42736/0 (socket says 172.21.15.69:42736) 2024-04-24T08:20:36.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.229+0000 7f8a1de00640 1 -- 172.21.15.69:0/694622109 learned_addr learned my addr 172.21.15.69:0/694622109 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:36.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.229+0000 7f8a1de00640 1 -- 172.21.15.69:0/694622109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56488758b6b0 msgr2=0x5648875770a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.229+0000 7f8a15400640 1 --2- 172.21.15.69:0/694622109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564887579e00 0x5648875327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.229+0000 7f8a1de00640 1 --2- 172.21.15.69:0/694622109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56488758b6b0 0x5648875770a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.229+0000 7f8a1de00640 1 -- 172.21.15.69:0/694622109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564887579e00 msgr2=0x5648875327e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.229+0000 7f8a1de00640 1 --2- 172.21.15.69:0/694622109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564887579e00 0x5648875327e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.229+0000 7f8a1de00640 1 -- 172.21.15.69:0/694622109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56488756fca0 con 0x56488755cf20 2024-04-24T08:20:36.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.230+0000 7f8a1de00640 1 --2- 172.21.15.69:0/694622109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x5648875322a0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x56488756ed90 tx=0x7f8a10012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e385ba0b3642ad server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:36.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.230+0000 7f8a0e000640 1 -- 172.21.15.69:0/694622109 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a10013d30 con 0x56488755cf20 2024-04-24T08:20:36.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.230+0000 7f8a0e000640 1 -- 172.21.15.69:0/694622109 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a10029070 con 0x56488755cf20 2024-04-24T08:20:36.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.230+0000 7f8a0e000640 1 -- 172.21.15.69:0/694622109 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a10024350 con 0x56488755cf20 2024-04-24T08:20:36.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.231+0000 7f8a27c23740 1 -- 172.21.15.69:0/694622109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 msgr2=0x5648875322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.231+0000 7f8a27c23740 1 --2- 172.21.15.69:0/694622109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x5648875322a0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x56488756ed90 tx=0x7f8a10012000 comp rx=0 tx=0).stop 2024-04-24T08:20:36.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.231+0000 7f8a27c23740 1 -- 172.21.15.69:0/694622109 shutdown_connections 2024-04-24T08:20:36.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.231+0000 7f8a27c23740 1 --2- 172.21.15.69:0/694622109 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x56488758b6b0 0x5648875770a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.231+0000 7f8a27c23740 1 --2- 172.21.15.69:0/694622109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564887579e00 0x5648875327e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.231+0000 7f8a27c23740 1 --2- 172.21.15.69:0/694622109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488755cf20 0x5648875322a0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.231+0000 7f8a27c23740 1 -- 172.21.15.69:0/694622109 >> 172.21.15.69:0/694622109 conn(0x56488755c7e0 msgr2=0x564887557ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:36.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.231+0000 7f8a27c23740 1 -- 172.21.15.69:0/694622109 shutdown_connections 2024-04-24T08:20:36.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.231+0000 7f8a27c23740 1 -- 172.21.15.69:0/694622109 wait complete. 2024-04-24T08:20:36.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.231+0000 7f8a27c23740 1 -- start start 2024-04-24T08:20:36.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.232+0000 7f8a27c23740 1 Processor -- start 2024-04-24T08:20:36.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.232+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488755cf20 0x56488752f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564887579e00 0x56488752c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a27c23740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488758b6b0 0x5648875322a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a27c23740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56488743b790 con 0x56488758b6b0 2024-04-24T08:20:36.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a27c23740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201eca20 con 0x56488755cf20 2024-04-24T08:20:36.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a27c23740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x564887533580 con 0x564887579e00 2024-04-24T08:20:36.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a27c23740 10 client.0 _ll_register_callbacks cb 0x7f8a0800a5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a1e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488758b6b0 0x5648875322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a1e800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488758b6b0 0x5648875322a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42752/0 (socket says 172.21.15.69:42752) 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a1e800640 1 -- 172.21.15.69:0/2069160296 learned_addr learned my addr 172.21.15.69:0/2069160296 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a1e800640 1 -- 172.21.15.69:0/2069160296 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564887579e00 msgr2=0x56488752c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a1e800640 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564887579e00 0x56488752c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a1e800640 1 -- 172.21.15.69:0/2069160296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488755cf20 msgr2=0x56488752f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a1e800640 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488755cf20 0x56488752f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a1e800640 1 -- 172.21.15.69:0/2069160296 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648871a83f0 con 0x56488758b6b0 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a1e800640 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488758b6b0 0x5648875322a0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f8a180768a0 tx=0x7f8a1807ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c34e76059c6fb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.233+0000 7f8a03400640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a03400640 1 -- 172.21.15.69:0/2069160296 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180a8020 con 0x56488758b6b0 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a03400640 1 -- 172.21.15.69:0/2069160296 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f8a18092d60 con 0x56488758b6b0 2024-04-24T08:20:36.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a27c23740 1 -- 172.21.15.69:0/2069160296 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a10029070 con 0x56488758b6b0 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a03400640 1 -- 172.21.15.69:0/2069160296 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=99) ==== 24+0+0 (secure 0 0 0) 0x7f8a18083020 con 0x56488758b6b0 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a27c23740 10 client.9534 fetch_fsmap learned FSMap version 99 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a27c23740 1 -- 172.21.15.69:0/2069160296 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=99,monmap=2+}) -- 0x5648871680f0 con 0x56488758b6b0 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a03400640 1 -- 172.21.15.69:0/2069160296 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a10029070 con 0x56488758b6b0 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a03400640 1 -- 172.21.15.69:0/2069160296 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 99) ==== 3494+0+0 (secure 0 0 0) 0x7f8a18007b50 con 0x56488758b6b0 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a27c23740 10 client.9534 fetch_fsmap finished waiting for FSMap version 99 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a27c23740 10 client.9534 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.b 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a27c23740 20 client.9534 populate_metadata read hostname 'smithi069' 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x56488758a3e0 0x5648875327e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a27c23740 4 client.9534 mds_command: new command op to 9288 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-24T08:20:36.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.234+0000 7f8a27c23740 1 -- 172.21.15.69:0/2069160296 --> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56488749da80 con 0x56488758a3e0 2024-04-24T08:20:36.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.235+0000 7f8a1de00640 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x56488758a3e0 0x5648875327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.236+0000 7f8a1de00640 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x56488758a3e0 0x5648875327e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2a0b2a7bde31584 server_cookie=baafec8f7cea9caa in_seq=0 out_seq=0 2024-04-24T08:20:36.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.236+0000 7f8a03400640 10 client.9534 ms_handle_connect on v2:172.21.15.121:6834/1515102940 2024-04-24T08:20:36.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a03400640 1 -- 172.21.15.69:0/2069160296 <== mds.0 v2:172.21.15.121:6834/1515102940 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56488749da80 con 0x56488758a3e0 2024-04-24T08:20:36.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a03400640 10 client.9534 handle_command_reply: tid=0 2024-04-24T08:20:36.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 client.9534 shutdown 2024-04-24T08:20:36.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 -- 172.21.15.69:0/2069160296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488758b6b0 msgr2=0x5648875322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488758b6b0 0x5648875322a0 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f8a180768a0 tx=0x7f8a1807ac00 comp rx=0 tx=0).stop 2024-04-24T08:20:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 -- 172.21.15.69:0/2069160296 shutdown_connections 2024-04-24T08:20:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.121:6834/1515102940,v1:172.21.15.121:6836/1515102940] conn(0x56488758a3e0 0x5648875327e0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56488758b6b0 0x5648875322a0 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x564887579e00 0x56488752c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a03400640 0 client.9534 ms_handle_reset on v2:172.21.15.121:6834/1515102940 2024-04-24T08:20:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 --2- 172.21.15.69:0/2069160296 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56488755cf20 0x56488752f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 -- 172.21.15.69:0/2069160296 >> 172.21.15.69:0/2069160296 conn(0x56488755c7e0 msgr2=0x56488756f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 -- 172.21.15.69:0/2069160296 shutdown_connections 2024-04-24T08:20:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 1 -- 172.21.15.69:0/2069160296 wait complete. 2024-04-24T08:20:36.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.252+0000 7f8a27c23740 20 client.9534 trim_cache size 0 max 16384 2024-04-24T08:20:36.255 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:20:36.255 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-24T08:20:36.255 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0 2024-04-24T08:20:36.255 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:20:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 -- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89fc06c920 msgr2=0x7f89fc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89fc06c920 0x7f89fc06ede0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f8a10006fd0 tx=0x7f8a10008040 comp rx=0 tx=0).stop 2024-04-24T08:20:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 -- 172.21.15.69:0/2256401407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a2015d030 msgr2=0x7f8a20150260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a2015d030 0x7f8a20150260 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f8a18034040 tx=0x7f8a18004a40 comp rx=0 tx=0).stop 2024-04-24T08:20:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 -- 172.21.15.69:0/2256401407 shutdown_connections 2024-04-24T08:20:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f89fc06c920 0x7f89fc06ede0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a2015d030 0x7f8a20150260 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a20072220 0x7f8a200b1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 --2- 172.21.15.69:0/2256401407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a20071d70 0x7f8a20156030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 -- 172.21.15.69:0/2256401407 >> 172.21.15.69:0/2256401407 conn(0x7f8a20067b80 msgr2=0x7f8a201592b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 -- 172.21.15.69:0/2256401407 shutdown_connections 2024-04-24T08:20:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.253+0000 7f8a1f200640 1 -- 172.21.15.69:0/2256401407 wait complete. 2024-04-24T08:20:36.274 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:20:36.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.459+0000 7fdf4fc00640 1 Processor -- start 2024-04-24T08:20:36.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.459+0000 7fdf4fc00640 1 -- start start 2024-04-24T08:20:36.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.460+0000 7fdf4fc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf480d0300 0x7fdf4806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.460+0000 7fdf4fc00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdf48145a90 0x7fdf480d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.460+0000 7fdf4fc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf480d0c20 con 0x7fdf48145fe0 2024-04-24T08:20:36.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.460+0000 7fdf4fc00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fdf480cf3d0 con 0x7fdf48145a90 2024-04-24T08:20:36.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.460+0000 7fdf4fc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdf480cf6b0 con 0x7fdf480d0300 2024-04-24T08:20:36.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.461+0000 7fdf3fe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdf48145a90 0x7fdf480d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.461+0000 7fdf4d200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf480d0300 0x7fdf4806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.461+0000 7fdf3fe00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdf48145a90 0x7fdf480d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.69:51122/0 (socket says 172.21.15.69:51122) 2024-04-24T08:20:36.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.461+0000 7fdf3fe00640 1 -- 172.21.15.69:0/1729287855 learned_addr learned my addr 172.21.15.69:0/1729287855 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:36.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.462+0000 7fdf3fe00640 1 -- 172.21.15.69:0/1729287855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf480d0300 msgr2=0x7fdf4806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:36.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.462+0000 7fdf3fe00640 1 --2- 172.21.15.69:0/1729287855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf480d0300 0x7fdf4806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.462+0000 7fdf3fe00640 1 -- 172.21.15.69:0/1729287855 >> v1:172.21.15.121:6789/0 conn(0x7fdf48145fe0 legacy=0x7fdf48160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.462+0000 7fdf3fe00640 1 -- 172.21.15.69:0/1729287855 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf480cf990 con 0x7fdf48145a90 2024-04-24T08:20:36.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.462+0000 7fdf4d200640 1 --2- 172.21.15.69:0/1729287855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf480d0300 0x7fdf4806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:36.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.464+0000 7fdf3fe00640 1 --2- 172.21.15.69:0/1729287855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf48145a90 0x7fdf480d06e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009870 tx=0x7fdf3002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b32805e544a266d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:36.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.464+0000 7fdf47e00640 1 -- 172.21.15.69:0/1729287855 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30037070 con 0x7fdf48145a90 2024-04-24T08:20:36.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.464+0000 7fdf47e00640 1 -- 172.21.15.69:0/1729287855 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf30032440 con 0x7fdf48145a90 2024-04-24T08:20:36.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.464+0000 7fdf47e00640 1 -- 172.21.15.69:0/1729287855 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30032810 con 0x7fdf48145a90 2024-04-24T08:20:36.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.465+0000 7fdf4fc00640 1 -- 172.21.15.69:0/1729287855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf48145a90 msgr2=0x7fdf480d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.465+0000 7fdf4fc00640 1 --2- 172.21.15.69:0/1729287855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf48145a90 0x7fdf480d06e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009870 tx=0x7fdf3002a250 comp rx=0 tx=0).stop 2024-04-24T08:20:36.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.465+0000 7fdf4fc00640 1 -- 172.21.15.69:0/1729287855 shutdown_connections 2024-04-24T08:20:36.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.465+0000 7fdf4fc00640 1 --2- 172.21.15.69:0/1729287855 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf48145a90 0x7fdf480d06e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.465+0000 7fdf4fc00640 1 --2- 172.21.15.69:0/1729287855 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf480d0300 0x7fdf4806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.465+0000 7fdf4fc00640 1 -- 172.21.15.69:0/1729287855 >> 172.21.15.69:0/1729287855 conn(0x7fdf48067d90 msgr2=0x7fdf4806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:36.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.465+0000 7fdf4fc00640 1 -- 172.21.15.69:0/1729287855 shutdown_connections 2024-04-24T08:20:36.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.465+0000 7fdf4fc00640 1 -- 172.21.15.69:0/1729287855 wait complete. 2024-04-24T08:20:36.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.467+0000 7fdf4fc00640 1 Processor -- start 2024-04-24T08:20:36.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.467+0000 7fdf4fc00640 1 -- start start 2024-04-24T08:20:36.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.468+0000 7fdf4fc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf480d0300 0x7fdf48139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.468+0000 7fdf4fc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf48145a90 0x7fdf480b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.468+0000 7fdf4fc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf48145fe0 0x7fdf4815bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.468+0000 7fdf4fc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48145770 con 0x7fdf48145a90 2024-04-24T08:20:36.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.468+0000 7fdf4fc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdf480cff50 con 0x7fdf480d0300 2024-04-24T08:20:36.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.468+0000 7fdf4fc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdf481458f0 con 0x7fdf48145fe0 2024-04-24T08:20:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.468+0000 7fdf3fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf48145a90 0x7fdf480b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.468+0000 7fdf3fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf48145a90 0x7fdf480b3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42758/0 (socket says 172.21.15.69:42758) 2024-04-24T08:20:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.468+0000 7fdf3fe00640 1 -- 172.21.15.69:0/445081671 learned_addr learned my addr 172.21.15.69:0/445081671 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.468+0000 7fdf4d200640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf480d0300 0x7fdf48139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf4dc00640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf48145fe0 0x7fdf4815bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf3fe00640 1 -- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf48145fe0 msgr2=0x7fdf4815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf3fe00640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf48145fe0 0x7fdf4815bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf3fe00640 1 -- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf480d0300 msgr2=0x7fdf48139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf3fe00640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf480d0300 0x7fdf48139970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf3fe00640 1 -- 172.21.15.69:0/445081671 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf48139eb0 con 0x7fdf48145a90 2024-04-24T08:20:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf3fe00640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf48145a90 0x7fdf480b3690 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fdf30000c00 tx=0x7fdf300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf45600640 1 -- 172.21.15.69:0/445081671 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30037040 con 0x7fdf48145a90 2024-04-24T08:20:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf45600640 1 -- 172.21.15.69:0/445081671 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf30042070 con 0x7fdf48145a90 2024-04-24T08:20:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf45600640 1 -- 172.21.15.69:0/445081671 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30030880 con 0x7fdf48145a90 2024-04-24T08:20:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.469+0000 7fdf4fc00640 1 -- 172.21.15.69:0/445081671 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf480adb70 con 0x7fdf48145a90 2024-04-24T08:20:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.470+0000 7fdf4fc00640 1 -- 172.21.15.69:0/445081671 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf480ae130 con 0x7fdf48145a90 2024-04-24T08:20:36.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.471+0000 7fdf4fc00640 1 -- 172.21.15.69:0/445081671 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf48068a80 con 0x7fdf48145a90 2024-04-24T08:20:36.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.472+0000 7fdf45600640 1 -- 172.21.15.69:0/445081671 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdf30030a20 con 0x7fdf48145a90 2024-04-24T08:20:36.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.472+0000 7fdf45600640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf2406c8f0 0x7fdf2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:36.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.473+0000 7fdf45600640 1 -- 172.21.15.69:0/445081671 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fdf300ac4a0 con 0x7fdf48145a90 2024-04-24T08:20:36.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.473+0000 7fdf4d200640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf2406c8f0 0x7fdf2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:36.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.474+0000 7fdf4d200640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf2406c8f0 0x7fdf2406edb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009a30 tx=0x7fdf38008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:36.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.478+0000 7fdf45600640 1 -- 172.21.15.69:0/445081671 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdf3007a830 con 0x7fdf48145a90 2024-04-24T08:20:36.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.723+0000 7fdf4fc00640 1 -- 172.21.15.69:0/445081671 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdf48144600 con 0x7fdf48145a90 2024-04-24T08:20:36.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.724+0000 7fdf45600640 1 -- 172.21.15.69:0/445081671 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v99) ==== 53+0+73 (secure 0 0 0) 0x7fdf3007a130 con 0x7fdf48145a90 2024-04-24T08:20:36.725 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:20:36.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 -- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf2406c8f0 msgr2=0x7fdf2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf2406c8f0 0x7fdf2406edb0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009a30 tx=0x7fdf38008040 comp rx=0 tx=0).stop 2024-04-24T08:20:36.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 -- 172.21.15.69:0/445081671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf48145a90 msgr2=0x7fdf480b3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:36.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf48145a90 0x7fdf480b3690 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fdf30000c00 tx=0x7fdf300046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 -- 172.21.15.69:0/445081671 shutdown_connections 2024-04-24T08:20:36.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdf2406c8f0 0x7fdf2406edb0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdf48145a90 0x7fdf480b3690 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdf48145fe0 0x7fdf4815bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 --2- 172.21.15.69:0/445081671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf480d0300 0x7fdf48139970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:36.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 -- 172.21.15.69:0/445081671 >> 172.21.15.69:0/445081671 conn(0x7fdf48067d90 msgr2=0x7fdf4806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:36.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 -- 172.21.15.69:0/445081671 shutdown_connections 2024-04-24T08:20:36.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:36.727+0000 7fdf4fc00640 1 -- 172.21.15.69:0/445081671 wait complete. 2024-04-24T08:20:36.749 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:20:36.749 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:20:36.749 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:20:36.749 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:20:36.749 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:36.749 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:20:36.749 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:20:36.749 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:20:36.749 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:20:36.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:36.750 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:36.795 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 41943sec preferred_lft 41943sec 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:20:36.796 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:36.797 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:20:36.797 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:36.797 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:20:36.797 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:36.797 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:20:36.797 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:36.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:36.798 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:20:36.920 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:20:36.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:36.924 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:20:36.976 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:20:36.976 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:20:36.977 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:20:36.977 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:20:36.977 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:20:36.978 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:20:36.978 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:20:36.978 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:20:36.978 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:20:36.978 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:20:36.978 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:20:37.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:20:37.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:37.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:37.184 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:20:37.230 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:20:37.231 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:20:37.231 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:20:37.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:37.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:20:37.359 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:20:37.359 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:20:37.359 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:37.411 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:20:37.411 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:20:37.411 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:37.460 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:20:37.543 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:20:37.598 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:20:37.598 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:20:37.599 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:20:37.600 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:20:37.601 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:20:37.601 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:20:37.601 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:37.601 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:37.601 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:37.601 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:37.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:37.602 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:20:37.655 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:20:37.655 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:20:37.659 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:20:37.726 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:20:37.769 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:20:37.769 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:20:37.769 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:20:37.770 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:37.771 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:37.772 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:37.772 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:37.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:37.772 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:20:37.836 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:20:37.835+0000 7f6b5af72480 -1 init, newargv = 0x5647b4c903e0 newargc=15 2024-04-24T08:20:37.836 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[76391]: starting ceph client 2024-04-24T08:20:37.846 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[76391]: starting fuse 2024-04-24T08:20:37.860 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:20:37.860 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:20:37.860 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:20:37.860 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:20:37.860 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:20:37.860 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:37.860 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:20:37.860 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:37.860 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:20:37.860 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:20:37.860 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:20:37.860 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:20:37.861 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:38.046 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.76391.asok 2024-04-24T08:20:38.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:38.062 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.76391.asok 2024-04-24T08:20:38.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:38.063 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.76391.asok status 2024-04-24T08:20:38.288 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "76391", 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 9546, 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:20:38.289 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 9546 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2542681025 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2542681025 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:38.290 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.9546 192.168.0.1:0/2542681025", 2024-04-24T08:20:38.291 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/2542681025", 2024-04-24T08:20:38.291 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:20:38.291 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 99, 2024-04-24T08:20:38.291 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 109, 2024-04-24T08:20:38.291 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:20:38.291 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:20:38.291 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:20:38.291 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:20:38.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:38.304 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:20:38.304 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:38.357 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:20:38.358 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:38.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:38.358 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:38.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:38.438 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent/unflushed) 2024-04-24T08:20:38.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:38.501 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=./parent/unflushed/jfile bs=1M count=8) 2024-04-24T08:20:38.706 INFO:teuthology.orchestra.run.smithi069.stderr:8+0 records in 2024-04-24T08:20:38.706 INFO:teuthology.orchestra.run.smithi069.stderr:8+0 records out 2024-04-24T08:20:38.707 INFO:teuthology.orchestra.run.smithi069.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.146924 s, 57.1 MB/s 2024-04-24T08:20:38.710 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:38.710 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:38.711 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:38.711 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:38.711 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-04-24T08:20:38.711 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:38.845 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627779 2024-04-24T08:20:38.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:38.858 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:38.858 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:38.858 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:38.859 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:38.859 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-04-24T08:20:38.859 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:39.034 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627780 2024-04-24T08:20:39.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:39.048 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:20:39.048 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:20:39.141 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:39.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.339+0000 7f1a6fe00640 1 Processor -- start 2024-04-24T08:20:39.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.339+0000 7f1a6fe00640 1 -- start start 2024-04-24T08:20:39.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.341+0000 7f1a6fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a7015e960 0x7f1a70162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.341+0000 7f1a6fe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1a70162980 0x7f1a7006cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.341+0000 7f1a6fe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a7015ed40 con 0x7f1a7006d4f0 2024-04-24T08:20:39.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.341+0000 7f1a6fe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1a700cf050 con 0x7f1a70162980 2024-04-24T08:20:39.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.341+0000 7f1a6fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1a70162e20 con 0x7f1a7015e960 2024-04-24T08:20:39.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.341+0000 7f1a66000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1a70162980 0x7f1a7006cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.341+0000 7f1a6f400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1a7006d4f0 0x7f1a700cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46350/0 (socket says 172.21.15.69:46350) 2024-04-24T08:20:39.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.341+0000 7f1a6f400640 1 -- 172.21.15.69:0/3904183232 learned_addr learned my addr 172.21.15.69:0/3904183232 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:39.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.342+0000 7f1a6ea00640 1 --2- 172.21.15.69:0/3904183232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a7015e960 0x7f1a70162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.342+0000 7f1a66000640 1 -- 172.21.15.69:0/3904183232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a7015e960 msgr2=0x7f1a70162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:39.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.342+0000 7f1a66000640 1 --2- 172.21.15.69:0/3904183232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a7015e960 0x7f1a70162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.342+0000 7f1a66000640 1 -- 172.21.15.69:0/3904183232 >> v1:172.21.15.69:6789/0 conn(0x7f1a7006d4f0 legacy=0x7f1a700cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.342+0000 7f1a66000640 1 -- 172.21.15.69:0/3904183232 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a70163100 con 0x7f1a70162980 2024-04-24T08:20:39.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.342+0000 7f1a6e000640 1 -- 172.21.15.69:0/3904183232 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3740849391 0 0) 0x7f1a7015ed40 con 0x7f1a7006d4f0 2024-04-24T08:20:39.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.343+0000 7f1a66000640 1 --2- 172.21.15.69:0/3904183232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a70162980 0x7f1a7006cfb0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f1a580097a0 tx=0x7f1a5802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc8d9f6eb98955cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:39.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.343+0000 7f1a6e000640 1 -- 172.21.15.69:0/3904183232 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a58037070 con 0x7f1a70162980 2024-04-24T08:20:39.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.343+0000 7f1a6e000640 1 -- 172.21.15.69:0/3904183232 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a58032440 con 0x7f1a70162980 2024-04-24T08:20:39.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.343+0000 7f1a6e000640 1 -- 172.21.15.69:0/3904183232 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a58032810 con 0x7f1a70162980 2024-04-24T08:20:39.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.344+0000 7f1a6fe00640 1 -- 172.21.15.69:0/3904183232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a70162980 msgr2=0x7f1a7006cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.344+0000 7f1a6fe00640 1 --2- 172.21.15.69:0/3904183232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a70162980 0x7f1a7006cfb0 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f1a580097a0 tx=0x7f1a5802a180 comp rx=0 tx=0).stop 2024-04-24T08:20:39.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.344+0000 7f1a6fe00640 1 -- 172.21.15.69:0/3904183232 shutdown_connections 2024-04-24T08:20:39.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.344+0000 7f1a6fe00640 1 --2- 172.21.15.69:0/3904183232 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a70162980 0x7f1a7006cfb0 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.344+0000 7f1a6fe00640 1 --2- 172.21.15.69:0/3904183232 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a7015e960 0x7f1a70162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.344+0000 7f1a6fe00640 1 -- 172.21.15.69:0/3904183232 >> 172.21.15.69:0/3904183232 conn(0x7f1a7009d900 msgr2=0x7f1a7008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:39.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.344+0000 7f1a6fe00640 1 -- 172.21.15.69:0/3904183232 shutdown_connections 2024-04-24T08:20:39.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.344+0000 7f1a6fe00640 1 -- 172.21.15.69:0/3904183232 wait complete. 2024-04-24T08:20:39.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.346+0000 7f1a6fe00640 1 Processor -- start 2024-04-24T08:20:39.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.346+0000 7f1a6fe00640 1 -- start start 2024-04-24T08:20:39.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.347+0000 7f1a6fe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7006d4f0 0x7f1a70157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.347+0000 7f1a6fe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a7015e960 0x7f1a700cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.347+0000 7f1a6fe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a70162980 0x7f1a700c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.347+0000 7f1a6fe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1a700b6170 con 0x7f1a7006d4f0 2024-04-24T08:20:39.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.347+0000 7f1a6fe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1a700cf5d0 con 0x7f1a7015e960 2024-04-24T08:20:39.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.347+0000 7f1a6fe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1a700b62f0 con 0x7f1a70162980 2024-04-24T08:20:39.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.347+0000 7f1a6ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7006d4f0 0x7f1a70157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.347+0000 7f1a6ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7006d4f0 0x7f1a70157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39394/0 (socket says 172.21.15.69:39394) 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.347+0000 7f1a6ea00640 1 -- 172.21.15.69:0/728594699 learned_addr learned my addr 172.21.15.69:0/728594699 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.348+0000 7f1a6f400640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a70162980 0x7f1a700c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.348+0000 7f1a66000640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a7015e960 0x7f1a700cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.348+0000 7f1a6ea00640 1 -- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a70162980 msgr2=0x7f1a700c8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.348+0000 7f1a6ea00640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a70162980 0x7f1a700c8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.348+0000 7f1a6ea00640 1 -- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a7015e960 msgr2=0x7f1a700cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.348+0000 7f1a6ea00640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a7015e960 0x7f1a700cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.348+0000 7f1a6ea00640 1 -- 172.21.15.69:0/728594699 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a70158050 con 0x7f1a7006d4f0 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.348+0000 7f1a6ea00640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7006d4f0 0x7f1a70157b10 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f1a6000d830 tx=0x7f1a6000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.348+0000 7f1a67400640 1 -- 172.21.15.69:0/728594699 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000b6c0 con 0x7f1a7006d4f0 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.349+0000 7f1a67400640 1 -- 172.21.15.69:0/728594699 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a60019070 con 0x7f1a7006d4f0 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.349+0000 7f1a6fe00640 1 -- 172.21.15.69:0/728594699 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a700cc150 con 0x7f1a7006d4f0 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.349+0000 7f1a67400640 1 -- 172.21.15.69:0/728594699 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6000ba80 con 0x7f1a7006d4f0 2024-04-24T08:20:39.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.349+0000 7f1a6fe00640 1 -- 172.21.15.69:0/728594699 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a700cc710 con 0x7f1a7006d4f0 2024-04-24T08:20:39.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.350+0000 7f1a6fe00640 1 -- 172.21.15.69:0/728594699 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a70075d50 con 0x7f1a7006d4f0 2024-04-24T08:20:39.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.351+0000 7f1a67400640 1 -- 172.21.15.69:0/728594699 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a6001e460 con 0x7f1a7006d4f0 2024-04-24T08:20:39.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.352+0000 7f1a67400640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a4c06c8f0 0x7f1a4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.352+0000 7f1a67400640 1 -- 172.21.15.69:0/728594699 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f1a6008c230 con 0x7f1a7006d4f0 2024-04-24T08:20:39.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.352+0000 7f1a66000640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a4c06c8f0 0x7f1a4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.353+0000 7f1a66000640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a4c06c8f0 0x7f1a4c06edb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1a58034000 tx=0x7f1a5802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:39.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.356+0000 7f1a67400640 1 -- 172.21.15.69:0/728594699 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a6005a650 con 0x7f1a7006d4f0 2024-04-24T08:20:39.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.555+0000 7f1a6fe00640 1 -- 172.21.15.69:0/728594699 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a700ba180 con 0x7f1a7006d4f0 2024-04-24T08:20:39.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.557+0000 7f1a67400640 1 -- 172.21.15.69:0/728594699 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+16603 (secure 0 0 0) 0x7f1a60059f50 con 0x7f1a7006d4f0 2024-04-24T08:20:39.559 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:39.560 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":109,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:20:10.770819+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:39.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.560+0000 7f1a6fe00640 1 -- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a4c06c8f0 msgr2=0x7f1a4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.560+0000 7f1a6fe00640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a4c06c8f0 0x7f1a4c06edb0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f1a58034000 tx=0x7f1a5802e000 comp rx=0 tx=0).stop 2024-04-24T08:20:39.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.560+0000 7f1a6fe00640 1 -- 172.21.15.69:0/728594699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7006d4f0 msgr2=0x7f1a70157b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.560+0000 7f1a6fe00640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7006d4f0 0x7f1a70157b10 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f1a6000d830 tx=0x7f1a6000dcf0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.560+0000 7f1a6fe00640 1 -- 172.21.15.69:0/728594699 shutdown_connections 2024-04-24T08:20:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.560+0000 7f1a6fe00640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1a4c06c8f0 0x7f1a4c06edb0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.560+0000 7f1a6fe00640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a70162980 0x7f1a700c8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.560+0000 7f1a6fe00640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a7015e960 0x7f1a700cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.560+0000 7f1a6fe00640 1 --2- 172.21.15.69:0/728594699 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a7006d4f0 0x7f1a70157b10 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.560+0000 7f1a6fe00640 1 -- 172.21.15.69:0/728594699 >> 172.21.15.69:0/728594699 conn(0x7f1a7009d900 msgr2=0x7f1a700cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.561+0000 7f1a6fe00640 1 -- 172.21.15.69:0/728594699 shutdown_connections 2024-04-24T08:20:39.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.561+0000 7f1a6fe00640 1 -- 172.21.15.69:0/728594699 wait complete. 2024-04-24T08:20:39.580 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:20:06.963817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:20:08.190440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:20:39.580 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:20:39.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.780+0000 7f70f5a00640 1 Processor -- start 2024-04-24T08:20:39.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.780+0000 7f70f5a00640 1 -- start start 2024-04-24T08:20:39.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.782+0000 7f70f5a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 0x7f70f00b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.782+0000 7f70f5a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70f00b46a0 0x7f70f006f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.782+0000 7f70f5a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70f015cc00 con 0x7f70f006fd10 2024-04-24T08:20:39.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.782+0000 7f70f5a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f70f006ed30 con 0x7f70f00b46a0 2024-04-24T08:20:39.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.782+0000 7f70f5a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f70f00b4e10 con 0x7f70f015c820 2024-04-24T08:20:39.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.782+0000 7f70ee000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70f00b46a0 0x7f70f006f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.782+0000 7f70ee000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f70f00b46a0 0x7f70f006f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39402/0 (socket says 172.21.15.69:39402) 2024-04-24T08:20:39.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.782+0000 7f70ee000640 1 -- 172.21.15.69:0/1283083698 learned_addr learned my addr 172.21.15.69:0/1283083698 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:39.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.782+0000 7f70eea00640 1 --2- 172.21.15.69:0/1283083698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 0x7f70f00b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.783+0000 7f70ee000640 1 -- 172.21.15.69:0/1283083698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 msgr2=0x7f70f00b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:39.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.783+0000 7f70ee000640 1 --2- 172.21.15.69:0/1283083698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 0x7f70f00b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.783+0000 7f70ee000640 1 -- 172.21.15.69:0/1283083698 >> v1:172.21.15.121:6789/0 conn(0x7f70f006fd10 legacy=0x7f70f00ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.783+0000 7f70ee000640 1 -- 172.21.15.69:0/1283083698 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f00b50f0 con 0x7f70f00b46a0 2024-04-24T08:20:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.783+0000 7f70eea00640 1 --2- 172.21.15.69:0/1283083698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 0x7f70f00b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.784+0000 7f70ee000640 1 --2- 172.21.15.69:0/1283083698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 0x7f70f006f7d0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009870 tx=0x7f70d802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbbc7ab857631da server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.785+0000 7f70ed600640 1 -- 172.21.15.69:0/1283083698 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8037070 con 0x7f70f00b46a0 2024-04-24T08:20:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.785+0000 7f70ed600640 1 -- 172.21.15.69:0/1283083698 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d8032440 con 0x7f70f00b46a0 2024-04-24T08:20:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.785+0000 7f70ed600640 1 -- 172.21.15.69:0/1283083698 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8032810 con 0x7f70f00b46a0 2024-04-24T08:20:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.785+0000 7f70f5a00640 1 -- 172.21.15.69:0/1283083698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 msgr2=0x7f70f006f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.785+0000 7f70f5a00640 1 --2- 172.21.15.69:0/1283083698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 0x7f70f006f7d0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009870 tx=0x7f70d802a270 comp rx=0 tx=0).stop 2024-04-24T08:20:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.785+0000 7f70f5a00640 1 -- 172.21.15.69:0/1283083698 shutdown_connections 2024-04-24T08:20:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.785+0000 7f70f5a00640 1 --2- 172.21.15.69:0/1283083698 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 0x7f70f006f7d0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.785+0000 7f70f5a00640 1 --2- 172.21.15.69:0/1283083698 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 0x7f70f00b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.785+0000 7f70f5a00640 1 -- 172.21.15.69:0/1283083698 >> 172.21.15.69:0/1283083698 conn(0x7f70f00d0350 msgr2=0x7f70f00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.786+0000 7f70f5a00640 1 -- 172.21.15.69:0/1283083698 shutdown_connections 2024-04-24T08:20:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.786+0000 7f70f5a00640 1 -- 172.21.15.69:0/1283083698 wait complete. 2024-04-24T08:20:39.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.787+0000 7f70f5a00640 1 Processor -- start 2024-04-24T08:20:39.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.787+0000 7f70f5a00640 1 -- start start 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.788+0000 7f70f5a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f006fd10 0x7f70f015eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.788+0000 7f70f5a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 0x7f70f0151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.788+0000 7f70f5a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 0x7f70f01573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.788+0000 7f70f5a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f70f00b4b00 con 0x7f70f00b46a0 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.788+0000 7f70f5a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f70f006f2b0 con 0x7f70f006fd10 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.788+0000 7f70f5a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f70f00ce960 con 0x7f70f015c820 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70ee000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 0x7f70f0151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70ee000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 0x7f70f0151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39412/0 (socket says 172.21.15.69:39412) 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70ee000640 1 -- 172.21.15.69:0/1543667447 learned_addr learned my addr 172.21.15.69:0/1543667447 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70eea00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f006fd10 0x7f70f015eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70ef400640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 0x7f70f01573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70ee000640 1 -- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 msgr2=0x7f70f01573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70ee000640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 0x7f70f01573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70ee000640 1 -- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f006fd10 msgr2=0x7f70f015eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:39.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70ee000640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f006fd10 0x7f70f015eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:39.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70ee000640 1 -- 172.21.15.69:0/1543667447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f015de40 con 0x7f70f00b46a0 2024-04-24T08:20:39.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.789+0000 7f70eea00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f006fd10 0x7f70f015eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:39.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.790+0000 7f70ee000640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 0x7f70f0151540 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009870 tx=0x7f70d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:39.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.790+0000 7f70e6a00640 1 -- 172.21.15.69:0/1543667447 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8041070 con 0x7f70f00b46a0 2024-04-24T08:20:39.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.790+0000 7f70e6a00640 1 -- 172.21.15.69:0/1543667447 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d803c890 con 0x7f70f00b46a0 2024-04-24T08:20:39.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.790+0000 7f70e6a00640 1 -- 172.21.15.69:0/1543667447 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8037040 con 0x7f70f00b46a0 2024-04-24T08:20:39.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.790+0000 7f70f5a00640 1 -- 172.21.15.69:0/1543667447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70f015e120 con 0x7f70f00b46a0 2024-04-24T08:20:39.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.790+0000 7f70f5a00640 1 -- 172.21.15.69:0/1543667447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70f015e6e0 con 0x7f70f00b46a0 2024-04-24T08:20:39.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.791+0000 7f70f5a00640 1 -- 172.21.15.69:0/1543667447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70f0068a80 con 0x7f70f00b46a0 2024-04-24T08:20:39.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.792+0000 7f70e6a00640 1 -- 172.21.15.69:0/1543667447 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f70d8044070 con 0x7f70f00b46a0 2024-04-24T08:20:39.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.793+0000 7f70e6a00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f70cc06c8f0 0x7f70cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:39.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.793+0000 7f70e6a00640 1 -- 172.21.15.69:0/1543667447 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f70d80ac6d0 con 0x7f70f00b46a0 2024-04-24T08:20:39.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.794+0000 7f70eea00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f70cc06c8f0 0x7f70cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.795+0000 7f70eea00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f70cc06c8f0 0x7f70cc06edb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f70f015f7e0 tx=0x7f70e0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.798+0000 7f70e6a00640 1 -- 172.21.15.69:0/1543667447 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70d807ab80 con 0x7f70f00b46a0 2024-04-24T08:20:40.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:39.999+0000 7f70f5a00640 1 -- 172.21.15.69:0/1543667447 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f70f00cddc0 con 0x7f70f00b46a0 2024-04-24T08:20:40.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.000+0000 7f70e6a00640 1 -- 172.21.15.69:0/1543667447 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) ==== 93+0+10 (secure 0 0 0) 0x7f70d807a480 con 0x7f70f00b46a0 2024-04-24T08:20:40.001 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:20:40.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 -- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f70cc06c8f0 msgr2=0x7f70cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f70cc06c8f0 0x7f70cc06edb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f70f015f7e0 tx=0x7f70e0008040 comp rx=0 tx=0).stop 2024-04-24T08:20:40.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 -- 172.21.15.69:0/1543667447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 msgr2=0x7f70f0151540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 0x7f70f0151540 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009870 tx=0x7f70d80046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 -- 172.21.15.69:0/1543667447 shutdown_connections 2024-04-24T08:20:40.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f70cc06c8f0 0x7f70cc06edb0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70f00b46a0 0x7f70f0151540 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f70f015c820 0x7f70f01573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 --2- 172.21.15.69:0/1543667447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f70f006fd10 0x7f70f015eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 -- 172.21.15.69:0/1543667447 >> 172.21.15.69:0/1543667447 conn(0x7f70f00d0350 msgr2=0x7f70f00af560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:40.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 -- 172.21.15.69:0/1543667447 shutdown_connections 2024-04-24T08:20:40.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.002+0000 7f70f5a00640 1 -- 172.21.15.69:0/1543667447 wait complete. 2024-04-24T08:20:40.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:20:40.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.225+0000 7fbfcaa00640 1 Processor -- start 2024-04-24T08:20:40.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.225+0000 7fbfcaa00640 1 -- start start 2024-04-24T08:20:40.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.227+0000 7fbfcaa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbfc415ea60 0x7fbfc40b5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.227+0000 7fbfcaa00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fbfc40d09f0 0x7fbfc406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.227+0000 7fbfcaa00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfc40ce910 con 0x7fbfc40cdf30 2024-04-24T08:20:40.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.227+0000 7fbfcaa00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fbfc40cebf0 con 0x7fbfc415ea60 2024-04-24T08:20:40.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.227+0000 7fbfcaa00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fbfc40ceed0 con 0x7fbfc40d09f0 2024-04-24T08:20:40.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.227+0000 7fbfc9600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbfc415ea60 0x7fbfc40b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:40.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.227+0000 7fbfc9600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fbfc415ea60 0x7fbfc40b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39416/0 (socket says 172.21.15.69:39416) 2024-04-24T08:20:40.229 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.227+0000 7fbfc9600640 1 -- 172.21.15.69:0/2624928693 learned_addr learned my addr 172.21.15.69:0/2624928693 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:40.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.227+0000 7fbfca000640 1 --2- 172.21.15.69:0/2624928693 >> v2:172.21.15.121:3300/0 conn(0x7fbfc40d09f0 0x7fbfc406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:40.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.228+0000 7fbfc9600640 1 -- 172.21.15.69:0/2624928693 >> v2:172.21.15.121:3300/0 conn(0x7fbfc40d09f0 msgr2=0x7fbfc406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:40.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.228+0000 7fbfc9600640 1 --2- 172.21.15.69:0/2624928693 >> v2:172.21.15.121:3300/0 conn(0x7fbfc40d09f0 0x7fbfc406df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.228+0000 7fbfc9600640 1 -- 172.21.15.69:0/2624928693 >> v1:172.21.15.121:6789/0 conn(0x7fbfc40cdf30 legacy=0x7fbfc40d02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:40.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.228+0000 7fbfc9600640 1 -- 172.21.15.69:0/2624928693 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc40cf1b0 con 0x7fbfc415ea60 2024-04-24T08:20:40.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.229+0000 7fbfc9600640 1 --2- 172.21.15.69:0/2624928693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 0x7fbfc40b5300 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40aff30 tx=0x7fbfb802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514d4206891fc35f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:40.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.229+0000 7fbfc3e00640 1 -- 172.21.15.69:0/2624928693 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8037070 con 0x7fbfc415ea60 2024-04-24T08:20:40.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.229+0000 7fbfc3e00640 1 -- 172.21.15.69:0/2624928693 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfb8032440 con 0x7fbfc415ea60 2024-04-24T08:20:40.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.229+0000 7fbfc3e00640 1 -- 172.21.15.69:0/2624928693 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb8032810 con 0x7fbfc415ea60 2024-04-24T08:20:40.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.230+0000 7fbfcaa00640 1 -- 172.21.15.69:0/2624928693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 msgr2=0x7fbfc40b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.230+0000 7fbfcaa00640 1 --2- 172.21.15.69:0/2624928693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 0x7fbfc40b5300 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40aff30 tx=0x7fbfb802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.230+0000 7fbfcaa00640 1 -- 172.21.15.69:0/2624928693 shutdown_connections 2024-04-24T08:20:40.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.230+0000 7fbfcaa00640 1 --2- 172.21.15.69:0/2624928693 >> v2:172.21.15.121:3300/0 conn(0x7fbfc40d09f0 0x7fbfc406df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.230+0000 7fbfcaa00640 1 --2- 172.21.15.69:0/2624928693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 0x7fbfc40b5300 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.230+0000 7fbfcaa00640 1 -- 172.21.15.69:0/2624928693 >> 172.21.15.69:0/2624928693 conn(0x7fbfc4067b80 msgr2=0x7fbfc406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:40.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.230+0000 7fbfcaa00640 1 -- 172.21.15.69:0/2624928693 shutdown_connections 2024-04-24T08:20:40.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.230+0000 7fbfcaa00640 1 -- 172.21.15.69:0/2624928693 wait complete. 2024-04-24T08:20:40.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.232+0000 7fbfcaa00640 1 Processor -- start 2024-04-24T08:20:40.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.232+0000 7fbfcaa00640 1 -- start start 2024-04-24T08:20:40.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.233+0000 7fbfcaa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc40cdf30 0x7fbfc413dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.233+0000 7fbfcaa00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc40d09f0 0x7fbfc4153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.233+0000 7fbfcaa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 0x7fbfc4152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.233+0000 7fbfcaa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc40ce540 con 0x7fbfc415ea60 2024-04-24T08:20:40.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.233+0000 7fbfcaa00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fbfc40ce3c0 con 0x7fbfc40d09f0 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.233+0000 7fbfcaa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fbfc40ce6c0 con 0x7fbfc40cdf30 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.233+0000 7fbfca000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 0x7fbfc4152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.233+0000 7fbfca000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 0x7fbfc4152410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39422/0 (socket says 172.21.15.69:39422) 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.233+0000 7fbfca000640 1 -- 172.21.15.69:0/504181817 learned_addr learned my addr 172.21.15.69:0/504181817 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.234+0000 7fbfc8c00640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc40d09f0 0x7fbfc4153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.234+0000 7fbfc9600640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc40cdf30 0x7fbfc413dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.234+0000 7fbfca000640 1 -- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc40cdf30 msgr2=0x7fbfc413dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.234+0000 7fbfca000640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc40cdf30 0x7fbfc413dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.234+0000 7fbfca000640 1 -- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc40d09f0 msgr2=0x7fbfc4153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.234+0000 7fbfca000640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc40d09f0 0x7fbfc4153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.234+0000 7fbfca000640 1 -- 172.21.15.69:0/504181817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfc413e290 con 0x7fbfc415ea60 2024-04-24T08:20:40.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.234+0000 7fbfc9600640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc40cdf30 0x7fbfc413dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:40.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.234+0000 7fbfca000640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 0x7fbfc4152410 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc007e40 tx=0x7fbfbc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:40.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.235+0000 7fbfc1600640 1 -- 172.21.15.69:0/504181817 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc014620 con 0x7fbfc415ea60 2024-04-24T08:20:40.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.235+0000 7fbfc1600640 1 -- 172.21.15.69:0/504181817 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfbc019070 con 0x7fbfc415ea60 2024-04-24T08:20:40.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.235+0000 7fbfc1600640 1 -- 172.21.15.69:0/504181817 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc014a20 con 0x7fbfc415ea60 2024-04-24T08:20:40.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.235+0000 7fbfcaa00640 1 -- 172.21.15.69:0/504181817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfc4152c60 con 0x7fbfc415ea60 2024-04-24T08:20:40.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.235+0000 7fbfcaa00640 1 -- 172.21.15.69:0/504181817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfc4153220 con 0x7fbfc415ea60 2024-04-24T08:20:40.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.236+0000 7fbfcaa00640 1 -- 172.21.15.69:0/504181817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfc4075d50 con 0x7fbfc415ea60 2024-04-24T08:20:40.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.237+0000 7fbfc1600640 1 -- 172.21.15.69:0/504181817 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbfbc014bc0 con 0x7fbfc415ea60 2024-04-24T08:20:40.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.238+0000 7fbfc1600640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbfa406c8f0 0x7fbfa406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.238+0000 7fbfc1600640 1 -- 172.21.15.69:0/504181817 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fbfbc08c590 con 0x7fbfc415ea60 2024-04-24T08:20:40.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.239+0000 7fbfc9600640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbfa406c8f0 0x7fbfa406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:40.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.240+0000 7fbfc9600640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbfa406c8f0 0x7fbfa406edb0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8034000 tx=0x7fbfb802e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:40.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.243+0000 7fbfc1600640 1 -- 172.21.15.69:0/504181817 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbfc413e290 con 0x7fbfc415ea60 2024-04-24T08:20:40.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.443+0000 7fbfcaa00640 1 -- 172.21.15.69:0/504181817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbfc40727f0 con 0x7fbfc415ea60 2024-04-24T08:20:40.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.444+0000 7fbfc1600640 1 -- 172.21.15.69:0/504181817 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) ==== 104+0+11 (secure 0 0 0) 0x7fbfbc05a890 con 0x7fbfc415ea60 2024-04-24T08:20:40.446 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:20:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.447+0000 7fbfcaa00640 1 -- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbfa406c8f0 msgr2=0x7fbfa406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.447+0000 7fbfcaa00640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbfa406c8f0 0x7fbfa406edb0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8034000 tx=0x7fbfb802e000 comp rx=0 tx=0).stop 2024-04-24T08:20:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.447+0000 7fbfcaa00640 1 -- 172.21.15.69:0/504181817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 msgr2=0x7fbfc4152410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.447+0000 7fbfcaa00640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 0x7fbfc4152410 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc007e40 tx=0x7fbfbc00e4b0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.447+0000 7fbfcaa00640 1 -- 172.21.15.69:0/504181817 shutdown_connections 2024-04-24T08:20:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.447+0000 7fbfcaa00640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fbfa406c8f0 0x7fbfa406edb0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.447+0000 7fbfcaa00640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfc415ea60 0x7fbfc4152410 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.447+0000 7fbfcaa00640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbfc40d09f0 0x7fbfc4153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.447+0000 7fbfcaa00640 1 --2- 172.21.15.69:0/504181817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fbfc40cdf30 0x7fbfc413dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.447+0000 7fbfcaa00640 1 -- 172.21.15.69:0/504181817 >> 172.21.15.69:0/504181817 conn(0x7fbfc4067b80 msgr2=0x7fbfc415dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:40.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.448+0000 7fbfcaa00640 1 -- 172.21.15.69:0/504181817 shutdown_connections 2024-04-24T08:20:40.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.448+0000 7fbfcaa00640 1 -- 172.21.15.69:0/504181817 wait complete. 2024-04-24T08:20:40.468 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:20:40.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.671+0000 7f780d600640 1 Processor -- start 2024-04-24T08:20:40.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.671+0000 7f780d600640 1 -- start start 2024-04-24T08:20:40.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.673+0000 7f780d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f780815f4b0 0x7f78080b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.673+0000 7f780d600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78080b46a0 0x7f78080b4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.673+0000 7f780d600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78080739f0 con 0x7f780815f890 2024-04-24T08:20:40.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.673+0000 7f780d600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f78080b5030 con 0x7f78080b46a0 2024-04-24T08:20:40.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.673+0000 7f780d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7808073cc0 con 0x7f780815f4b0 2024-04-24T08:20:40.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.673+0000 7f7807400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78080b46a0 0x7f78080b4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:40.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.673+0000 7f7807400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f78080b46a0 0x7f78080b4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39432/0 (socket says 172.21.15.69:39432) 2024-04-24T08:20:40.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.673+0000 7f7807400640 1 -- 172.21.15.69:0/1437065380 learned_addr learned my addr 172.21.15.69:0/1437065380 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:40.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.673+0000 7f7806a00640 1 --2- 172.21.15.69:0/1437065380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f780815f4b0 0x7f78080b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:40.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.674+0000 7f7807400640 1 -- 172.21.15.69:0/1437065380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f780815f4b0 msgr2=0x7f78080b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:40.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.674+0000 7f7807400640 1 --2- 172.21.15.69:0/1437065380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f780815f4b0 0x7f78080b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.674+0000 7f7807400640 1 -- 172.21.15.69:0/1437065380 >> v1:172.21.15.121:6789/0 conn(0x7f780815f890 legacy=0x7f78080732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:40.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.674+0000 7f7807400640 1 -- 172.21.15.69:0/1437065380 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7808073fa0 con 0x7f78080b46a0 2024-04-24T08:20:40.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.675+0000 7f7807400640 1 --2- 172.21.15.69:0/1437065380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78080b46a0 0x7f78080b4af0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f7800009e50 tx=0x7f780002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6653cfa677a87c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:40.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.675+0000 7f7806000640 1 -- 172.21.15.69:0/1437065380 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7800036070 con 0x7f78080b46a0 2024-04-24T08:20:40.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.675+0000 7f7806000640 1 -- 172.21.15.69:0/1437065380 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7800031600 con 0x7f78080b46a0 2024-04-24T08:20:40.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.675+0000 7f7806000640 1 -- 172.21.15.69:0/1437065380 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78000319d0 con 0x7f78080b46a0 2024-04-24T08:20:40.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.676+0000 7f780d600640 1 -- 172.21.15.69:0/1437065380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78080b46a0 msgr2=0x7f78080b4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.676+0000 7f780d600640 1 --2- 172.21.15.69:0/1437065380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78080b46a0 0x7f78080b4af0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f7800009e50 tx=0x7f780002a6c0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.676+0000 7f780d600640 1 -- 172.21.15.69:0/1437065380 shutdown_connections 2024-04-24T08:20:40.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.676+0000 7f780d600640 1 --2- 172.21.15.69:0/1437065380 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f78080b46a0 0x7f78080b4af0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.676+0000 7f780d600640 1 --2- 172.21.15.69:0/1437065380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f780815f4b0 0x7f78080b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.676+0000 7f780d600640 1 -- 172.21.15.69:0/1437065380 >> 172.21.15.69:0/1437065380 conn(0x7f78080d0350 msgr2=0x7f78080d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:40.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.676+0000 7f780d600640 1 -- 172.21.15.69:0/1437065380 shutdown_connections 2024-04-24T08:20:40.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.676+0000 7f780d600640 1 -- 172.21.15.69:0/1437065380 wait complete. 2024-04-24T08:20:40.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.677+0000 7f780d600640 1 Processor -- start 2024-04-24T08:20:40.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.678+0000 7f780d600640 1 -- start start 2024-04-24T08:20:40.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.678+0000 7f780d600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78080b46a0 0x7f7808143410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.678+0000 7f780d600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f780815f4b0 0x7f780806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.678+0000 7f780d600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f780815f890 0x7f78081f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.678+0000 7f780d600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f78080b38c0 con 0x7f780815f4b0 2024-04-24T08:20:40.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.678+0000 7f780d600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f78080b35e0 con 0x7f780815f890 2024-04-24T08:20:40.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.678+0000 7f780d600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f780815fd10 con 0x7f78080b46a0 2024-04-24T08:20:40.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.679+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f780815f4b0 0x7f780806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.679+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f780815f4b0 0x7f780806c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39434/0 (socket says 172.21.15.69:39434) 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.679+0000 7f77ffe00640 1 -- 172.21.15.69:0/4110658738 learned_addr learned my addr 172.21.15.69:0/4110658738 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.679+0000 7f77ffe00640 1 -- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78080b46a0 msgr2=0x7f7808143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.679+0000 7f77ffe00640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78080b46a0 0x7f7808143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.679+0000 7f77ffe00640 1 -- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f780815f890 msgr2=0x7f78081f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.679+0000 7f77ffe00640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f780815f890 0x7f78081f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.679+0000 7f77ffe00640 1 -- 172.21.15.69:0/4110658738 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7808141420 con 0x7f780815f4b0 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.679+0000 7f77ffe00640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f780815f4b0 0x7f780806c5d0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f77f000c430 tx=0x7f77f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.680+0000 7f77fea00640 1 -- 172.21.15.69:0/4110658738 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0011790 con 0x7f780815f4b0 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.680+0000 7f77fea00640 1 -- 172.21.15.69:0/4110658738 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f0017070 con 0x7f780815f4b0 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.680+0000 7f780d600640 1 -- 172.21.15.69:0/4110658738 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78081439f0 con 0x7f780815f4b0 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.680+0000 7f77fea00640 1 -- 172.21.15.69:0/4110658738 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f0011bb0 con 0x7f780815f4b0 2024-04-24T08:20:40.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.680+0000 7f780d600640 1 -- 172.21.15.69:0/4110658738 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7808143fb0 con 0x7f780815f4b0 2024-04-24T08:20:40.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.681+0000 7f780d600640 1 -- 172.21.15.69:0/4110658738 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7808068a80 con 0x7f780815f4b0 2024-04-24T08:20:40.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.682+0000 7f77fea00640 1 -- 172.21.15.69:0/4110658738 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f77f001c460 con 0x7f780815f4b0 2024-04-24T08:20:40.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.683+0000 7f77fea00640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77e406c8f0 0x7f77e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:40.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.683+0000 7f77fea00640 1 -- 172.21.15.69:0/4110658738 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f77f008bba0 con 0x7f780815f4b0 2024-04-24T08:20:40.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.683+0000 7f7806a00640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77e406c8f0 0x7f77e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:40.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.685+0000 7f7806a00640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77e406c8f0 0x7f77e406edb0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f77f8006fd0 tx=0x7f77f8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:40.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.689+0000 7f77fea00640 1 -- 172.21.15.69:0/4110658738 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f77f0059ea0 con 0x7f780815f4b0 2024-04-24T08:20:40.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.892+0000 7f780d600640 1 -- 172.21.15.69:0/4110658738 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7808158190 con 0x7f780815f4b0 2024-04-24T08:20:40.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.893+0000 7f77fea00640 1 -- 172.21.15.69:0/4110658738 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) ==== 100+0+11 (secure 0 0 0) 0x7f77f00597a0 con 0x7f780815f4b0 2024-04-24T08:20:40.895 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 -- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77e406c8f0 msgr2=0x7f77e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77e406c8f0 0x7f77e406edb0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f77f8006fd0 tx=0x7f77f8008040 comp rx=0 tx=0).stop 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 -- 172.21.15.69:0/4110658738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f780815f4b0 msgr2=0x7f780806c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f780815f4b0 0x7f780806c5d0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f77f000c430 tx=0x7f77f000c8f0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 -- 172.21.15.69:0/4110658738 shutdown_connections 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f77e406c8f0 0x7f77e406edb0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f780815f4b0 0x7f780806c5d0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f780815f890 0x7f78081f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 --2- 172.21.15.69:0/4110658738 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f78080b46a0 0x7f7808143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 -- 172.21.15.69:0/4110658738 >> 172.21.15.69:0/4110658738 conn(0x7f78080d0350 msgr2=0x7f78080b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 -- 172.21.15.69:0/4110658738 shutdown_connections 2024-04-24T08:20:40.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:40.896+0000 7f780d600640 1 -- 172.21.15.69:0/4110658738 wait complete. 2024-04-24T08:20:40.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:20:41.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.114+0000 7f7eeda00640 1 Processor -- start 2024-04-24T08:20:41.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.114+0000 7f7eeda00640 1 -- start start 2024-04-24T08:20:41.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.116+0000 7f7eeda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee8071e50 0x7f7ee815ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.116+0000 7f7eeda00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ee8144db0 con 0x7f7ee8162300 2024-04-24T08:20:41.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.116+0000 7f7eeda00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ee815f540 con 0x7f7ee8072230 2024-04-24T08:20:41.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.116+0000 7f7eeda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee815f820 con 0x7f7ee8071e50 2024-04-24T08:20:41.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.116+0000 7f7ee7400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7ee8162300 0x7f7ee806e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46360/0 (socket says 172.21.15.69:46360) 2024-04-24T08:20:41.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.116+0000 7f7ee7400640 1 -- 172.21.15.69:0/106840450 learned_addr learned my addr 172.21.15.69:0/106840450 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:41.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee7e00640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee8071e50 0x7f7ee815ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1466265514 0 0) 0x7f7ee8144db0 con 0x7f7ee8162300 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ecc003520 con 0x7f7ee8162300 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1144604250 0 0) 0x7f7ecc003520 con 0x7f7ee8162300 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee8071e50 msgr2=0x7f7ee815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee8071e50 0x7f7ee815ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 >> v1:172.21.15.121:6789/0 conn(0x7f7ee8072230 legacy=0x7f7ee81446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee815fb00 con 0x7f7ee8162300 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f7ed4001670 con 0x7f7ee8162300 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 >> v1:172.21.15.69:6789/0 conn(0x7f7ee8162300 legacy=0x7f7ee806e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ecc0039e0 0x7f7ecc003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee8071e50 0x7f7ecc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 0x7f7ecc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee815f820 con 0x7f7ecc004d40 2024-04-24T08:20:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7ecc00ba70 con 0x7f7ecc0039e0 2024-04-24T08:20:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.117+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7ecc00bc70 con 0x7f7ee8071e50 2024-04-24T08:20:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.118+0000 7f7eed000640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 0x7f7ecc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.118+0000 7f7ee7e00640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ecc0039e0 0x7f7ecc003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.118+0000 7f7eed000640 1 -- 172.21.15.69:0/106840450 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee8071e50 msgr2=0x7f7ecc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.118+0000 7f7eed000640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee8071e50 0x7f7ecc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.118+0000 7f7eed000640 1 -- 172.21.15.69:0/106840450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ecc0039e0 msgr2=0x7f7ecc003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.118+0000 7f7eed000640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ecc0039e0 0x7f7ecc003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.118+0000 7f7eed000640 1 -- 172.21.15.69:0/106840450 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ecc003520 con 0x7f7ecc004d40 2024-04-24T08:20:41.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.119+0000 7f7eed000640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 0x7f7ecc0051c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0001e20 tx=0x7f7ecc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad688d60a0a73a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:41.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.119+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0001a70 con 0x7f7ecc004d40 2024-04-24T08:20:41.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.119+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee002ae40 con 0x7f7ecc004d40 2024-04-24T08:20:41.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.119+0000 7f7ee6a00640 1 -- 172.21.15.69:0/106840450 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee0003bb0 con 0x7f7ecc004d40 2024-04-24T08:20:41.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.120+0000 7f7eeda00640 1 -- 172.21.15.69:0/106840450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 msgr2=0x7f7ecc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.120+0000 7f7eeda00640 1 --2- 172.21.15.69:0/106840450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 0x7f7ecc0051c0 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7f7ee0001e20 tx=0x7f7ecc005830 comp rx=0 tx=0).stop 2024-04-24T08:20:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.120+0000 7f7eed000640 1 -- 172.21.15.69:0/106840450 reap_dead start 2024-04-24T08:20:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.120+0000 7f7eeda00640 1 -- 172.21.15.69:0/106840450 shutdown_connections 2024-04-24T08:20:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.120+0000 7f7eeda00640 1 -- 172.21.15.69:0/106840450 >> 172.21.15.69:0/106840450 conn(0x7f7ee80d0160 msgr2=0x7f7ee80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.120+0000 7f7eeda00640 1 -- 172.21.15.69:0/106840450 shutdown_connections 2024-04-24T08:20:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.120+0000 7f7eeda00640 1 -- 172.21.15.69:0/106840450 wait complete. 2024-04-24T08:20:41.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.122+0000 7f7eeda00640 1 Processor -- start 2024-04-24T08:20:41.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.122+0000 7f7eeda00640 1 -- start start 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.123+0000 7f7eeda00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 0x7f7ee8099a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.123+0000 7f7eeda00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee80ad300 0x7f7ee80aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.123+0000 7f7eeda00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee814a130 0x7f7ee815bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.123+0000 7f7eeda00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee815ee40 con 0x7f7ecc004d40 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.123+0000 7f7eeda00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee815ecc0 con 0x7f7ee814a130 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.123+0000 7f7eeda00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee815efc0 con 0x7f7ee80ad300 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.124+0000 7f7ee7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 0x7f7ee8099a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.124+0000 7f7ee7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 0x7f7ee8099a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39452/0 (socket says 172.21.15.69:39452) 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.124+0000 7f7ee7e00640 1 -- 172.21.15.69:0/2436323836 learned_addr learned my addr 172.21.15.69:0/2436323836 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.124+0000 7f7ee7e00640 1 -- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee80ad300 msgr2=0x7f7ee80aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.124+0000 7f7ee7e00640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee80ad300 0x7f7ee80aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.124+0000 7f7ee7e00640 1 -- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee814a130 msgr2=0x7f7ee815bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.124+0000 7f7ee7e00640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee814a130 0x7f7ee815bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.124+0000 7f7ee7e00640 1 -- 172.21.15.69:0/2436323836 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee80b0700 con 0x7f7ecc004d40 2024-04-24T08:20:41.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.124+0000 7f7ee7e00640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 0x7f7ee8099a20 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80116a0 tx=0x7f7ed8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:41.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.125+0000 7f7edfe00640 1 -- 172.21.15.69:0/2436323836 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed800f940 con 0x7f7ecc004d40 2024-04-24T08:20:41.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.125+0000 7f7edfe00640 1 -- 172.21.15.69:0/2436323836 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ed800fae0 con 0x7f7ecc004d40 2024-04-24T08:20:41.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.125+0000 7f7edfe00640 1 -- 172.21.15.69:0/2436323836 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8015460 con 0x7f7ecc004d40 2024-04-24T08:20:41.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.125+0000 7f7eeda00640 1 -- 172.21.15.69:0/2436323836 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee80b0440 con 0x7f7ecc004d40 2024-04-24T08:20:41.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.125+0000 7f7eeda00640 1 -- 172.21.15.69:0/2436323836 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee8159290 con 0x7f7ecc004d40 2024-04-24T08:20:41.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.126+0000 7f7eeda00640 1 -- 172.21.15.69:0/2436323836 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee8075d50 con 0x7f7ecc004d40 2024-04-24T08:20:41.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.127+0000 7f7edfe00640 1 -- 172.21.15.69:0/2436323836 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7ed8015600 con 0x7f7ecc004d40 2024-04-24T08:20:41.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.128+0000 7f7edfe00640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ec806c8f0 0x7f7ec806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.128+0000 7f7edfe00640 1 -- 172.21.15.69:0/2436323836 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f7ed808c110 con 0x7f7ecc004d40 2024-04-24T08:20:41.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.129+0000 7f7ee7400640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ec806c8f0 0x7f7ec806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.130+0000 7f7ee7400640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ec806c8f0 0x7f7ec806edb0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7ee81587a0 tx=0x7f7ed4001520 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:41.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.132+0000 7f7edfe00640 1 -- 172.21.15.69:0/2436323836 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7ed8019030 con 0x7f7ecc004d40 2024-04-24T08:20:41.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.348+0000 7f7eeda00640 1 -- 172.21.15.69:0/2436323836 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7ee80d0d90 con 0x7f7ecc004d40 2024-04-24T08:20:41.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.349+0000 7f7edfe00640 1 -- 172.21.15.69:0/2436323836 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v109) ==== 80+0+0 (secure 0 0 0) 0x7f7ed8059e50 con 0x7f7ecc004d40 2024-04-24T08:20:41.350 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:20:41.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 -- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ec806c8f0 msgr2=0x7f7ec806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ec806c8f0 0x7f7ec806edb0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7ee81587a0 tx=0x7f7ed4001520 comp rx=0 tx=0).stop 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 -- 172.21.15.69:0/2436323836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 msgr2=0x7f7ee8099a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 0x7f7ee8099a20 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80116a0 tx=0x7f7ed8011b60 comp rx=0 tx=0).stop 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 -- 172.21.15.69:0/2436323836 shutdown_connections 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7ec806c8f0 0x7f7ec806edb0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ee814a130 0x7f7ee815bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7ee80ad300 0x7f7ee80aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 --2- 172.21.15.69:0/2436323836 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7ecc004d40 0x7f7ee8099a20 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 -- 172.21.15.69:0/2436323836 >> 172.21.15.69:0/2436323836 conn(0x7f7ee80d0160 msgr2=0x7f7ee80b0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 -- 172.21.15.69:0/2436323836 shutdown_connections 2024-04-24T08:20:41.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.352+0000 7f7eeda00640 1 -- 172.21.15.69:0/2436323836 wait complete. 2024-04-24T08:20:41.371 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:20:41.372 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:20:41.419 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:20:41.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.662+0000 7fe269600640 1 Processor -- start 2024-04-24T08:20:41.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.662+0000 7fe269600640 1 -- start start 2024-04-24T08:20:41.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.663+0000 7fe269600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2640d0300 0x7fe26406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.664+0000 7fe269600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe264145a90 0x7fe2640d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.664+0000 7fe269600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2640d0c20 con 0x7fe264145fe0 2024-04-24T08:20:41.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.664+0000 7fe269600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe2640cf3d0 con 0x7fe264145a90 2024-04-24T08:20:41.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.664+0000 7fe269600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe2640cf6b0 con 0x7fe2640d0300 2024-04-24T08:20:41.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.664+0000 7fe263400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe264145fe0 0x7fe264160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46374/0 (socket says 172.21.15.69:46374) 2024-04-24T08:20:41.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.664+0000 7fe263400640 1 -- 172.21.15.69:0/2244955249 learned_addr learned my addr 172.21.15.69:0/2244955249 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:41.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.664+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2160903401 0 0) 0x7fe2640d0c20 con 0x7fe264145fe0 2024-04-24T08:20:41.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.664+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe248003520 con 0x7fe264145fe0 2024-04-24T08:20:41.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.664+0000 7fe262000640 1 --2- 172.21.15.69:0/2244955249 >> v2:172.21.15.121:3300/0 conn(0x7fe264145a90 0x7fe2640d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.664+0000 7fe262a00640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2640d0300 0x7fe26406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3453531602 0 0) 0x7fe248003520 con 0x7fe264145fe0 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2640d0300 msgr2=0x7fe26406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2640d0300 0x7fe26406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 >> v2:172.21.15.121:3300/0 conn(0x7fe264145a90 msgr2=0x7fe2640d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 --2- 172.21.15.69:0/2244955249 >> v2:172.21.15.121:3300/0 conn(0x7fe264145a90 0x7fe2640d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2640cf990 con 0x7fe264145fe0 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fe25c0019f0 con 0x7fe264145fe0 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 >> v1:172.21.15.69:6789/0 conn(0x7fe264145fe0 legacy=0x7fe264160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2640d0300 0x7fe2480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe248003fe0 0x7fe248004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe248004d40 0x7fe2480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe2640cf3d0 con 0x7fe248004d40 2024-04-24T08:20:41.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2640cf6b0 con 0x7fe248003fe0 2024-04-24T08:20:41.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe24800bac0 con 0x7fe2640d0300 2024-04-24T08:20:41.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.665+0000 7fe263400640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe248004d40 0x7fe2480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.666+0000 7fe262a00640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2640d0300 0x7fe2480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.666+0000 7fe263400640 1 -- 172.21.15.69:0/2244955249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2640d0300 msgr2=0x7fe2480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:41.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.666+0000 7fe263400640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe2640d0300 0x7fe2480039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.666+0000 7fe263400640 1 -- 172.21.15.69:0/2244955249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe248003fe0 msgr2=0x7fe248004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:41.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.666+0000 7fe263400640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe248003fe0 0x7fe248004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.666+0000 7fe263400640 1 -- 172.21.15.69:0/2244955249 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe248003520 con 0x7fe248004d40 2024-04-24T08:20:41.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.667+0000 7fe263400640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe248004d40 0x7fe2480051c0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00a920 tx=0x7fe248005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90430065fc94af01 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:41.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.667+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00a4c0 con 0x7fe248004d40 2024-04-24T08:20:41.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.667+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe25c02f070 con 0x7fe248004d40 2024-04-24T08:20:41.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.667+0000 7fe261600640 1 -- 172.21.15.69:0/2244955249 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c02bd40 con 0x7fe248004d40 2024-04-24T08:20:41.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.667+0000 7fe269600640 1 -- 172.21.15.69:0/2244955249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe248004d40 msgr2=0x7fe2480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.667+0000 7fe269600640 1 --2- 172.21.15.69:0/2244955249 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe248004d40 0x7fe2480051c0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00a920 tx=0x7fe248005830 comp rx=0 tx=0).stop 2024-04-24T08:20:41.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.667+0000 7fe263400640 1 -- 172.21.15.69:0/2244955249 reap_dead start 2024-04-24T08:20:41.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.668+0000 7fe269600640 1 -- 172.21.15.69:0/2244955249 shutdown_connections 2024-04-24T08:20:41.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.668+0000 7fe269600640 1 -- 172.21.15.69:0/2244955249 >> 172.21.15.69:0/2244955249 conn(0x7fe264067d90 msgr2=0x7fe26406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:41.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.668+0000 7fe269600640 1 -- 172.21.15.69:0/2244955249 shutdown_connections 2024-04-24T08:20:41.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.668+0000 7fe269600640 1 -- 172.21.15.69:0/2244955249 wait complete. 2024-04-24T08:20:41.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.669+0000 7fe269600640 1 Processor -- start 2024-04-24T08:20:41.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.669+0000 7fe269600640 1 -- start start 2024-04-24T08:20:41.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.670+0000 7fe269600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe248004d40 0x7fe2641438f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.670+0000 7fe269600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2640ac8f0 0x7fe2640acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.670+0000 7fe269600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2640c8170 0x7fe2640c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.670+0000 7fe269600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe264145770 con 0x7fe2640c8170 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.670+0000 7fe269600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe2640cff50 con 0x7fe2640ac8f0 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.670+0000 7fe269600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe2641458f0 con 0x7fe248004d40 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe263400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2640c8170 0x7fe2640c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe263400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2640c8170 0x7fe2640c8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39468/0 (socket says 172.21.15.69:39468) 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe263400640 1 -- 172.21.15.69:0/4224658864 learned_addr learned my addr 172.21.15.69:0/4224658864 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe262a00640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe248004d40 0x7fe2641438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe262000640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2640ac8f0 0x7fe2640acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe263400640 1 -- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe248004d40 msgr2=0x7fe2641438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe263400640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe248004d40 0x7fe2641438f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe263400640 1 -- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2640ac8f0 msgr2=0x7fe2640acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe263400640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2640ac8f0 0x7fe2640acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe263400640 1 -- 172.21.15.69:0/4224658864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe264143e30 con 0x7fe2640c8170 2024-04-24T08:20:41.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe262a00640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe248004d40 0x7fe2641438f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:41.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.671+0000 7fe263400640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2640c8170 0x7fe2640c8600 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe25c000f80 tx=0x7fe2640cfb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:41.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.672+0000 7fe25aa00640 1 -- 172.21.15.69:0/4224658864 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c044070 con 0x7fe2640c8170 2024-04-24T08:20:41.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.672+0000 7fe25aa00640 1 -- 172.21.15.69:0/4224658864 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe25c034470 con 0x7fe2640c8170 2024-04-24T08:20:41.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.672+0000 7fe269600640 1 -- 172.21.15.69:0/4224658864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2640ab850 con 0x7fe2640c8170 2024-04-24T08:20:41.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.672+0000 7fe25aa00640 1 -- 172.21.15.69:0/4224658864 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00a4c0 con 0x7fe2640c8170 2024-04-24T08:20:41.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.672+0000 7fe269600640 1 -- 172.21.15.69:0/4224658864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2640abe10 con 0x7fe2640c8170 2024-04-24T08:20:41.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.673+0000 7fe269600640 1 -- 172.21.15.69:0/4224658864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2640afcb0 con 0x7fe2640c8170 2024-04-24T08:20:41.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.674+0000 7fe25aa00640 1 -- 172.21.15.69:0/4224658864 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe25c047070 con 0x7fe2640c8170 2024-04-24T08:20:41.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.675+0000 7fe25aa00640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe24406c8f0 0x7fe24406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:41.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.675+0000 7fe25aa00640 1 -- 172.21.15.69:0/4224658864 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fe25c0abc30 con 0x7fe2640c8170 2024-04-24T08:20:41.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.676+0000 7fe262a00640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe24406c8f0 0x7fe24406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:41.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.677+0000 7fe262a00640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe24406c8f0 0x7fe24406edb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe254006fd0 tx=0x7fe254008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:41.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.679+0000 7fe25aa00640 1 -- 172.21.15.69:0/4224658864 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2640afcb0 con 0x7fe2640c8170 2024-04-24T08:20:41.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.880+0000 7fe269600640 1 -- 172.21.15.69:0/4224658864 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe264068a80 con 0x7fe2640c8170 2024-04-24T08:20:41.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.883+0000 7fe25aa00640 1 -- 172.21.15.69:0/4224658864 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+16603 (secure 0 0 0) 0x7fe25c07a050 con 0x7fe2640c8170 2024-04-24T08:20:41.885 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:41.885 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":109,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:20:10.770819+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:20:41.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.885+0000 7fe269600640 1 -- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe24406c8f0 msgr2=0x7fe24406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.885+0000 7fe269600640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe24406c8f0 0x7fe24406edb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe254006fd0 tx=0x7fe254008040 comp rx=0 tx=0).stop 2024-04-24T08:20:41.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.885+0000 7fe269600640 1 -- 172.21.15.69:0/4224658864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2640c8170 msgr2=0x7fe2640c8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:41.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.885+0000 7fe269600640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2640c8170 0x7fe2640c8600 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe25c000f80 tx=0x7fe2640cfb60 comp rx=0 tx=0).stop 2024-04-24T08:20:41.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.885+0000 7fe269600640 1 -- 172.21.15.69:0/4224658864 shutdown_connections 2024-04-24T08:20:41.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.885+0000 7fe269600640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe24406c8f0 0x7fe24406edb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.885+0000 7fe269600640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2640c8170 0x7fe2640c8600 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.885+0000 7fe269600640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2640ac8f0 0x7fe2640acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.885+0000 7fe269600640 1 --2- 172.21.15.69:0/4224658864 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe248004d40 0x7fe2641438f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:41.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.886+0000 7fe269600640 1 -- 172.21.15.69:0/4224658864 >> 172.21.15.69:0/4224658864 conn(0x7fe264067d90 msgr2=0x7fe2640aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:41.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.886+0000 7fe269600640 1 -- 172.21.15.69:0/4224658864 shutdown_connections 2024-04-24T08:20:41.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:41.886+0000 7fe269600640 1 -- 172.21.15.69:0/4224658864 wait complete. 2024-04-24T08:20:41.905 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:20:06.963817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:20:08.190440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:20:41.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:20:42.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.101+0000 7efdc6600640 1 Processor -- start 2024-04-24T08:20:42.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.101+0000 7efdc6600640 1 -- start start 2024-04-24T08:20:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.102+0000 7efdc6600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7efdc00d0850 0x7efdc00cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.102+0000 7efdc6600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc00cd6c0 0x7efdc00ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.102+0000 7efdc6600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdc00cdb50 con 0x7efdc00b07a0 2024-04-24T08:20:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.102+0000 7efdc6600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7efdc00cf320 con 0x7efdc00d0850 2024-04-24T08:20:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.102+0000 7efdc6600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efdc006c6d0 con 0x7efdc00cd6c0 2024-04-24T08:20:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.103+0000 7efdb6a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7efdc00b07a0 0x7efdc00d0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46382/0 (socket says 172.21.15.69:46382) 2024-04-24T08:20:42.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.103+0000 7efdb6a00640 1 -- 172.21.15.69:0/2522851981 learned_addr learned my addr 172.21.15.69:0/2522851981 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.103+0000 7efdbf400640 1 --2- 172.21.15.69:0/2522851981 >> v2:172.21.15.121:3300/0 conn(0x7efdc00d0850 0x7efdc00cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.103+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3180147881 0 0) 0x7efdc00cdb50 con 0x7efdc00b07a0 2024-04-24T08:20:42.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.103+0000 7efdbfe00640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc00cd6c0 0x7efdc00ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.103+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efda4003520 con 0x7efdc00b07a0 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.103+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4214506046 0 0) 0x7efda4003520 con 0x7efdc00b07a0 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc00cd6c0 msgr2=0x7efdc00ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc00cd6c0 0x7efdc00ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 >> v2:172.21.15.121:3300/0 conn(0x7efdc00d0850 msgr2=0x7efdc00cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 --2- 172.21.15.69:0/2522851981 >> v2:172.21.15.121:3300/0 conn(0x7efdc00d0850 0x7efdc00cbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc006bb50 con 0x7efdc00b07a0 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7efda80019f0 con 0x7efdc00b07a0 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 >> v1:172.21.15.69:6789/0 conn(0x7efdc00b07a0 legacy=0x7efdc00d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc00cd6c0 0x7efda40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4003fe0 0x7efda4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda4004d40 0x7efda40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efdc00cf320 con 0x7efda4003fe0 2024-04-24T08:20:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efdc006c6d0 con 0x7efda4004d40 2024-04-24T08:20:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efda400bc30 con 0x7efdc00cd6c0 2024-04-24T08:20:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdb6a00640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4003fe0 0x7efda4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbfe00640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda4004d40 0x7efda40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdb6a00640 1 -- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc00cd6c0 msgr2=0x7efda40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdbf400640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc00cd6c0 0x7efda40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdb6a00640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efdc00cd6c0 0x7efda40039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdb6a00640 1 -- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda4004d40 msgr2=0x7efda40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdb6a00640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efda4004d40 0x7efda40051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.104+0000 7efdb6a00640 1 -- 172.21.15.69:0/2522851981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda4003520 con 0x7efda4003fe0 2024-04-24T08:20:42.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.105+0000 7efdb6a00640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4003fe0 0x7efda4004800 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7efda800a8a0 tx=0x7efda4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c07214a403cc2a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.106+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda800a4c0 con 0x7efda4003fe0 2024-04-24T08:20:42.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.106+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efda802f070 con 0x7efda4003fe0 2024-04-24T08:20:42.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.106+0000 7efdbea00640 1 -- 172.21.15.69:0/2522851981 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda802bd70 con 0x7efda4003fe0 2024-04-24T08:20:42.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.106+0000 7efdc6600640 1 -- 172.21.15.69:0/2522851981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4003fe0 msgr2=0x7efda4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.106+0000 7efdc6600640 1 --2- 172.21.15.69:0/2522851981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efda4003fe0 0x7efda4004800 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7efda800a8a0 tx=0x7efda4005830 comp rx=0 tx=0).stop 2024-04-24T08:20:42.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.106+0000 7efdbfe00640 1 -- 172.21.15.69:0/2522851981 reap_dead start 2024-04-24T08:20:42.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.106+0000 7efdc6600640 1 -- 172.21.15.69:0/2522851981 shutdown_connections 2024-04-24T08:20:42.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.106+0000 7efdc6600640 1 -- 172.21.15.69:0/2522851981 >> 172.21.15.69:0/2522851981 conn(0x7efdc00b3050 msgr2=0x7efdc00b3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:42.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.106+0000 7efdc6600640 1 -- 172.21.15.69:0/2522851981 shutdown_connections 2024-04-24T08:20:42.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.106+0000 7efdc6600640 1 -- 172.21.15.69:0/2522851981 wait complete. 2024-04-24T08:20:42.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.108+0000 7efdc6600640 1 Processor -- start 2024-04-24T08:20:42.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.108+0000 7efdc6600640 1 -- start start 2024-04-24T08:20:42.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.109+0000 7efdc6600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efda4003fe0 0x7efdc014c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.109+0000 7efdc6600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc015d070 0x7efdc015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.109+0000 7efdc6600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdc01f7d70 0x7efdc01f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.109+0000 7efdc6600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efdc006c3f0 con 0x7efdc01f7d70 2024-04-24T08:20:42.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.109+0000 7efdc6600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7efdc006c110 con 0x7efdc015d070 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.109+0000 7efdc6600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7efdc00ce110 con 0x7efda4003fe0 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.109+0000 7efdbfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdc01f7d70 0x7efdc01f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdbfe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdc01f7d70 0x7efdc01f81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39486/0 (socket says 172.21.15.69:39486) 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdbfe00640 1 -- 172.21.15.69:0/3001816265 learned_addr learned my addr 172.21.15.69:0/3001816265 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdbf400640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efda4003fe0 0x7efdc014c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdb6a00640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc015d070 0x7efdc015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdbfe00640 1 -- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efda4003fe0 msgr2=0x7efdc014c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdbfe00640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efda4003fe0 0x7efdc014c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdbfe00640 1 -- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc015d070 msgr2=0x7efdc015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdbfe00640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc015d070 0x7efdc015d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdbfe00640 1 -- 172.21.15.69:0/3001816265 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc014fb80 con 0x7efdc01f7d70 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdbf400640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efda4003fe0 0x7efdc014c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:42.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdb6a00640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc015d070 0x7efdc015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:42.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.110+0000 7efdbfe00640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdc01f7d70 0x7efdc01f81f0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7efdb8006fd0 tx=0x7efdb800e770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.111+0000 7efdb7e00640 1 -- 172.21.15.69:0/3001816265 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb8017020 con 0x7efdc01f7d70 2024-04-24T08:20:42.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.111+0000 7efdb7e00640 1 -- 172.21.15.69:0/3001816265 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdb8013630 con 0x7efdc01f7d70 2024-04-24T08:20:42.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.111+0000 7efdb7e00640 1 -- 172.21.15.69:0/3001816265 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb800f040 con 0x7efdc01f7d70 2024-04-24T08:20:42.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.111+0000 7efdc6600640 1 -- 172.21.15.69:0/3001816265 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc014edc0 con 0x7efdc01f7d70 2024-04-24T08:20:42.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.111+0000 7efdc6600640 1 -- 172.21.15.69:0/3001816265 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc014f380 con 0x7efdc01f7d70 2024-04-24T08:20:42.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.112+0000 7efdc6600640 1 -- 172.21.15.69:0/3001816265 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdc0068a80 con 0x7efdc01f7d70 2024-04-24T08:20:42.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.113+0000 7efdb7e00640 1 -- 172.21.15.69:0/3001816265 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efdb801a020 con 0x7efdc01f7d70 2024-04-24T08:20:42.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.113+0000 7efdb7e00640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efd9c06c920 0x7efd9c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.114+0000 7efdb7e00640 1 -- 172.21.15.69:0/3001816265 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7efdb808b520 con 0x7efdc01f7d70 2024-04-24T08:20:42.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.114+0000 7efdbf400640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efd9c06c920 0x7efd9c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.116+0000 7efdbf400640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efd9c06c920 0x7efd9c06ede0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7efdb0006fd0 tx=0x7efdb0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.118+0000 7efdb7e00640 1 -- 172.21.15.69:0/3001816265 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efdb8021020 con 0x7efdc01f7d70 2024-04-24T08:20:42.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.317+0000 7efdc6600640 1 -- 172.21.15.69:0/3001816265 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efdc00ae4d0 con 0x7efdc01f7d70 2024-04-24T08:20:42.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.317+0000 7efdb7e00640 1 -- 172.21.15.69:0/3001816265 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v109) ==== 93+0+10 (secure 0 0 0) 0x7efdb80592f0 con 0x7efdc01f7d70 2024-04-24T08:20:42.319 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:20:42.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 -- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efd9c06c920 msgr2=0x7efd9c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efd9c06c920 0x7efd9c06ede0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7efdb0006fd0 tx=0x7efdb0008040 comp rx=0 tx=0).stop 2024-04-24T08:20:42.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 -- 172.21.15.69:0/3001816265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdc01f7d70 msgr2=0x7efdc01f81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdc01f7d70 0x7efdc01f81f0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7efdb8006fd0 tx=0x7efdb800e770 comp rx=0 tx=0).stop 2024-04-24T08:20:42.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 -- 172.21.15.69:0/3001816265 shutdown_connections 2024-04-24T08:20:42.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7efd9c06c920 0x7efd9c06ede0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efdc01f7d70 0x7efdc01f81f0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efdc015d070 0x7efdc015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 --2- 172.21.15.69:0/3001816265 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7efda4003fe0 0x7efdc014c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 -- 172.21.15.69:0/3001816265 >> 172.21.15.69:0/3001816265 conn(0x7efdc00b3050 msgr2=0x7efdc00b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:42.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 -- 172.21.15.69:0/3001816265 shutdown_connections 2024-04-24T08:20:42.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.320+0000 7efdc6600640 1 -- 172.21.15.69:0/3001816265 wait complete. 2024-04-24T08:20:42.340 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:20:42.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.539+0000 7fd7d8a00640 1 Processor -- start 2024-04-24T08:20:42.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.539+0000 7fd7d8a00640 1 -- start start 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.540+0000 7fd7d8a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd7d41420a0 0x7fd7d4071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.540+0000 7fd7d8a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd7d4142480 0x7fd7d415ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.540+0000 7fd7d8a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7d4160510 0x7fd7d415c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.540+0000 7fd7d8a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd7d4160960 con 0x7fd7d41420a0 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.540+0000 7fd7d8a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd7d40cf1e0 con 0x7fd7d4142480 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.541+0000 7fd7d8a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd7d4072450 con 0x7fd7d4160510 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.541+0000 7fd7d3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd7d41420a0 0x7fd7d4071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.541+0000 7fd7d3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd7d41420a0 0x7fd7d4071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39492/0 (socket says 172.21.15.69:39492) 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.541+0000 7fd7d3400640 1 -- 172.21.15.69:0/2929878412 learned_addr learned my addr 172.21.15.69:0/2929878412 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.541+0000 7fd7d3400640 1 -- 172.21.15.69:0/2929878412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7d4160510 msgr2=0x7fd7d415c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.541+0000 7fd7d3400640 1 --2- 172.21.15.69:0/2929878412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7d4160510 0x7fd7d415c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.541+0000 7fd7d3400640 1 -- 172.21.15.69:0/2929878412 >> v2:172.21.15.121:3300/0 conn(0x7fd7d4142480 msgr2=0x7fd7d415ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:42.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.541+0000 7fd7d3400640 1 --2- 172.21.15.69:0/2929878412 >> v2:172.21.15.121:3300/0 conn(0x7fd7d4142480 0x7fd7d415ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.541+0000 7fd7d3400640 1 -- 172.21.15.69:0/2929878412 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d4072730 con 0x7fd7d41420a0 2024-04-24T08:20:42.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.542+0000 7fd7d3400640 1 --2- 172.21.15.69:0/2929878412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 0x7fd7d4071d70 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40d13d0 tx=0x7fd7c402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3859beee36d777c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d2000640 1 -- 172.21.15.69:0/2929878412 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4037070 con 0x7fd7d41420a0 2024-04-24T08:20:42.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d2000640 1 -- 172.21.15.69:0/2929878412 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c4032440 con 0x7fd7d41420a0 2024-04-24T08:20:42.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d2000640 1 -- 172.21.15.69:0/2929878412 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4032810 con 0x7fd7d41420a0 2024-04-24T08:20:42.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d8a00640 1 -- 172.21.15.69:0/2929878412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 msgr2=0x7fd7d4071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d8a00640 1 --2- 172.21.15.69:0/2929878412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 0x7fd7d4071d70 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40d13d0 tx=0x7fd7c402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d8a00640 1 -- 172.21.15.69:0/2929878412 shutdown_connections 2024-04-24T08:20:42.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d8a00640 1 --2- 172.21.15.69:0/2929878412 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7d4160510 0x7fd7d415c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d8a00640 1 --2- 172.21.15.69:0/2929878412 >> v2:172.21.15.121:3300/0 conn(0x7fd7d4142480 0x7fd7d415ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d8a00640 1 --2- 172.21.15.69:0/2929878412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 0x7fd7d4071d70 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d8a00640 1 -- 172.21.15.69:0/2929878412 >> 172.21.15.69:0/2929878412 conn(0x7fd7d4067b80 msgr2=0x7fd7d406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:42.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d8a00640 1 -- 172.21.15.69:0/2929878412 shutdown_connections 2024-04-24T08:20:42.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.543+0000 7fd7d8a00640 1 -- 172.21.15.69:0/2929878412 wait complete. 2024-04-24T08:20:42.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.545+0000 7fd7d8a00640 1 Processor -- start 2024-04-24T08:20:42.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.546+0000 7fd7d8a00640 1 -- start start 2024-04-24T08:20:42.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d8a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 0x7fd7d406e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d8a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7d4142480 0x7fd7d41f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d8a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7d4160510 0x7fd7d415b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d8a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d4142910 con 0x7fd7d41420a0 2024-04-24T08:20:42.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d8a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d4072cf0 con 0x7fd7d4160510 2024-04-24T08:20:42.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d8a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd7d415cef0 con 0x7fd7d4142480 2024-04-24T08:20:42.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 0x7fd7d406e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 0x7fd7d406e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39502/0 (socket says 172.21.15.69:39502) 2024-04-24T08:20:42.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d3400640 1 -- 172.21.15.69:0/4108260346 learned_addr learned my addr 172.21.15.69:0/4108260346 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:42.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d2a00640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7d4142480 0x7fd7d41f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.547+0000 7fd7d3400640 1 -- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7d4142480 msgr2=0x7fd7d41f3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7d3400640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7d4142480 0x7fd7d41f3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7d3400640 1 -- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7d4160510 msgr2=0x7fd7d415b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7d3e00640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7d4160510 0x7fd7d415b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7d3400640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7d4160510 0x7fd7d415b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7d3400640 1 -- 172.21.15.69:0/4108260346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d406f180 con 0x7fd7d41420a0 2024-04-24T08:20:42.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7d3400640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 0x7fd7d406e460 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4000c00 tx=0x7fd7c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7cb400640 1 -- 172.21.15.69:0/4108260346 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4041070 con 0x7fd7d41420a0 2024-04-24T08:20:42.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7cb400640 1 -- 172.21.15.69:0/4108260346 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c403c870 con 0x7fd7d41420a0 2024-04-24T08:20:42.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7cb400640 1 -- 172.21.15.69:0/4108260346 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4037040 con 0x7fd7d41420a0 2024-04-24T08:20:42.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7d8a00640 1 -- 172.21.15.69:0/4108260346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7d406f460 con 0x7fd7d41420a0 2024-04-24T08:20:42.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.548+0000 7fd7d8a00640 1 -- 172.21.15.69:0/4108260346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7d4154b50 con 0x7fd7d41420a0 2024-04-24T08:20:42.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.550+0000 7fd7d8a00640 1 -- 172.21.15.69:0/4108260346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7d4075d50 con 0x7fd7d41420a0 2024-04-24T08:20:42.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.551+0000 7fd7cb400640 1 -- 172.21.15.69:0/4108260346 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd7c4044070 con 0x7fd7d41420a0 2024-04-24T08:20:42.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.552+0000 7fd7cb400640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7b406c8f0 0x7fd7b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.552+0000 7fd7cb400640 1 -- 172.21.15.69:0/4108260346 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fd7c40ac730 con 0x7fd7d41420a0 2024-04-24T08:20:42.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.552+0000 7fd7d2a00640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7b406c8f0 0x7fd7b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.554+0000 7fd7d2a00640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7b406c8f0 0x7fd7b406edb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4135790 tx=0x7fd7c00072f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.558+0000 7fd7cb400640 1 -- 172.21.15.69:0/4108260346 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7c407ad10 con 0x7fd7d41420a0 2024-04-24T08:20:42.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.742+0000 7fd7d8a00640 1 -- 172.21.15.69:0/4108260346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd7d4154f30 con 0x7fd7d41420a0 2024-04-24T08:20:42.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.743+0000 7fd7cb400640 1 -- 172.21.15.69:0/4108260346 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v109) ==== 104+0+11 (secure 0 0 0) 0x7fd7c407a610 con 0x7fd7d41420a0 2024-04-24T08:20:42.744 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 -- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7b406c8f0 msgr2=0x7fd7b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7b406c8f0 0x7fd7b406edb0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4135790 tx=0x7fd7c00072f0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 -- 172.21.15.69:0/4108260346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 msgr2=0x7fd7d406e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 0x7fd7d406e460 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4000c00 tx=0x7fd7c40046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 -- 172.21.15.69:0/4108260346 shutdown_connections 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd7b406c8f0 0x7fd7b406edb0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7d4160510 0x7fd7d415b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd7d4142480 0x7fd7d41f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 --2- 172.21.15.69:0/4108260346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7d41420a0 0x7fd7d406e460 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 -- 172.21.15.69:0/4108260346 >> 172.21.15.69:0/4108260346 conn(0x7fd7d4067b80 msgr2=0x7fd7d406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:42.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.745+0000 7fd7d8a00640 1 -- 172.21.15.69:0/4108260346 shutdown_connections 2024-04-24T08:20:42.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.746+0000 7fd7d8a00640 1 -- 172.21.15.69:0/4108260346 wait complete. 2024-04-24T08:20:42.764 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:20:42.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.951+0000 7f1822c00640 1 Processor -- start 2024-04-24T08:20:42.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.951+0000 7f1822c00640 1 -- start start 2024-04-24T08:20:42.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.953+0000 7f1822c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f181c0b3e90 0x7f181c0b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.953+0000 7f1822c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f181c15cc00 con 0x7f181c15c820 2024-04-24T08:20:42.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.953+0000 7f1822c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f181c06ed30 con 0x7f181c0ce250 2024-04-24T08:20:42.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.953+0000 7f1822c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f181c0b4be0 con 0x7f181c0b3e90 2024-04-24T08:20:42.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.953+0000 7f1813400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f181c0b3e90 0x7f181c0b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.953+0000 7f181be00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f181c15c820 0x7f181c06f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46386/0 (socket says 172.21.15.69:46386) 2024-04-24T08:20:42.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.953+0000 7f181be00640 1 -- 172.21.15.69:0/3461736136 learned_addr learned my addr 172.21.15.69:0/3461736136 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:42.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.954+0000 7f181b400640 1 -- 172.21.15.69:0/3461736136 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2261363935 0 0) 0x7f181c15cc00 con 0x7f181c15c820 2024-04-24T08:20:42.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.954+0000 7f181b400640 1 -- 172.21.15.69:0/3461736136 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1800003520 con 0x7f181c15c820 2024-04-24T08:20:42.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.954+0000 7f1813400640 1 -- 172.21.15.69:0/3461736136 >> v1:172.21.15.121:6789/0 conn(0x7f181c0ce250 legacy=0x7f181c0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.954+0000 7f1813400640 1 -- 172.21.15.69:0/3461736136 >> v1:172.21.15.69:6789/0 conn(0x7f181c15c820 legacy=0x7f181c06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.954+0000 7f1813400640 1 -- 172.21.15.69:0/3461736136 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181c0b4e10 con 0x7f181c0b3e90 2024-04-24T08:20:42.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.955+0000 7f1813400640 1 --2- 172.21.15.69:0/3461736136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0b3e90 0x7f181c0b46a0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f181c15cdd0 tx=0x7f180402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9442833ae46383ad server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.955+0000 7f181b400640 1 -- 172.21.15.69:0/3461736136 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804037070 con 0x7f181c0b3e90 2024-04-24T08:20:42.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.955+0000 7f181b400640 1 -- 172.21.15.69:0/3461736136 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1804032440 con 0x7f181c0b3e90 2024-04-24T08:20:42.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.955+0000 7f181b400640 1 -- 172.21.15.69:0/3461736136 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18040327f0 con 0x7f181c0b3e90 2024-04-24T08:20:42.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.956+0000 7f1822c00640 1 -- 172.21.15.69:0/3461736136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0b3e90 msgr2=0x7f181c0b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.956+0000 7f1822c00640 1 --2- 172.21.15.69:0/3461736136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0b3e90 0x7f181c0b46a0 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7f181c15cdd0 tx=0x7f180402a210 comp rx=0 tx=0).stop 2024-04-24T08:20:42.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.956+0000 7f1822c00640 1 -- 172.21.15.69:0/3461736136 shutdown_connections 2024-04-24T08:20:42.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.956+0000 7f1822c00640 1 --2- 172.21.15.69:0/3461736136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0b3e90 0x7f181c0b46a0 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.956+0000 7f1822c00640 1 -- 172.21.15.69:0/3461736136 >> 172.21.15.69:0/3461736136 conn(0x7f181c0d0350 msgr2=0x7f181c0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:42.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.956+0000 7f1822c00640 1 -- 172.21.15.69:0/3461736136 shutdown_connections 2024-04-24T08:20:42.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.956+0000 7f1822c00640 1 -- 172.21.15.69:0/3461736136 wait complete. 2024-04-24T08:20:42.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.958+0000 7f1822c00640 1 Processor -- start 2024-04-24T08:20:42.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.958+0000 7f1822c00640 1 -- start start 2024-04-24T08:20:42.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.959+0000 7f1822c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f181c0b3e90 0x7f181c151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.959+0000 7f1822c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0ce250 0x7f181c15cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.959+0000 7f1822c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f181c15c820 0x7f181c15f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.959+0000 7f1822c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f181c06f2b0 con 0x7f181c0ce250 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.959+0000 7f1822c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f181c06efd0 con 0x7f181c0b3e90 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.959+0000 7f1822c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f181c06fee0 con 0x7f181c15c820 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.959+0000 7f1813400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0ce250 0x7f181c15cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.959+0000 7f1813400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0ce250 0x7f181c15cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39514/0 (socket says 172.21.15.69:39514) 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.959+0000 7f1813400640 1 -- 172.21.15.69:0/930698949 learned_addr learned my addr 172.21.15.69:0/930698949 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.960+0000 7f181be00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f181c0b3e90 0x7f181c151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.960+0000 7f1820c00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f181c15c820 0x7f181c15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.960+0000 7f1813400640 1 -- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f181c15c820 msgr2=0x7f181c15f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.960+0000 7f1813400640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f181c15c820 0x7f181c15f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.960+0000 7f1813400640 1 -- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f181c0b3e90 msgr2=0x7f181c151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.960+0000 7f1813400640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f181c0b3e90 0x7f181c151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:42.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.960+0000 7f1813400640 1 -- 172.21.15.69:0/930698949 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181c1557f0 con 0x7f181c0ce250 2024-04-24T08:20:42.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.960+0000 7f1820c00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f181c15c820 0x7f181c15f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:42.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.960+0000 7f1813400640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0ce250 0x7f181c15cc00 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f1804035040 tx=0x7f18040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.960+0000 7f1818c00640 1 -- 172.21.15.69:0/930698949 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804037050 con 0x7f181c0ce250 2024-04-24T08:20:42.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.961+0000 7f1818c00640 1 -- 172.21.15.69:0/930698949 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1804038040 con 0x7f181c0ce250 2024-04-24T08:20:42.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.961+0000 7f1822c00640 1 -- 172.21.15.69:0/930698949 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f181c15d570 con 0x7f181c0ce250 2024-04-24T08:20:42.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.961+0000 7f1818c00640 1 -- 172.21.15.69:0/930698949 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1804004450 con 0x7f181c0ce250 2024-04-24T08:20:42.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.961+0000 7f1822c00640 1 -- 172.21.15.69:0/930698949 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f181c151d40 con 0x7f181c0ce250 2024-04-24T08:20:42.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.962+0000 7f1822c00640 1 -- 172.21.15.69:0/930698949 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f181c068a80 con 0x7f181c0ce250 2024-04-24T08:20:42.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.963+0000 7f1818c00640 1 -- 172.21.15.69:0/930698949 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f180403c6d0 con 0x7f181c0ce250 2024-04-24T08:20:42.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.964+0000 7f1818c00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f17f806c8f0 0x7f17f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:42.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.964+0000 7f1818c00640 1 -- 172.21.15.69:0/930698949 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f18040ac600 con 0x7f181c0ce250 2024-04-24T08:20:42.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.964+0000 7f181be00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f17f806c8f0 0x7f17f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:42.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.965+0000 7f181be00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f17f806c8f0 0x7f17f806edb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f180c0011e0 tx=0x7f180c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:42.969 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:42.968+0000 7f1818c00640 1 -- 172.21.15.69:0/930698949 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f180407aa20 con 0x7f181c0ce250 2024-04-24T08:20:43.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.165+0000 7f1822c00640 1 -- 172.21.15.69:0/930698949 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f181c0cddc0 con 0x7f181c0ce250 2024-04-24T08:20:43.167 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.166+0000 7f1818c00640 1 -- 172.21.15.69:0/930698949 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v109) ==== 100+0+11 (secure 0 0 0) 0x7f180407a320 con 0x7f181c0ce250 2024-04-24T08:20:43.168 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:20:43.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.168+0000 7f1822c00640 1 -- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f17f806c8f0 msgr2=0x7f17f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.168+0000 7f1822c00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f17f806c8f0 0x7f17f806edb0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f180c0011e0 tx=0x7f180c009040 comp rx=0 tx=0).stop 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.169+0000 7f1822c00640 1 -- 172.21.15.69:0/930698949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0ce250 msgr2=0x7f181c15cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.169+0000 7f1822c00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0ce250 0x7f181c15cc00 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7f1804035040 tx=0x7f18040046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.169+0000 7f1822c00640 1 -- 172.21.15.69:0/930698949 shutdown_connections 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.169+0000 7f1822c00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f17f806c8f0 0x7f17f806edb0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.169+0000 7f1822c00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f181c0ce250 0x7f181c15cc00 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.169+0000 7f1822c00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f181c15c820 0x7f181c15f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.169+0000 7f1822c00640 1 --2- 172.21.15.69:0/930698949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f181c0b3e90 0x7f181c151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.169+0000 7f1822c00640 1 -- 172.21.15.69:0/930698949 >> 172.21.15.69:0/930698949 conn(0x7f181c0d0350 msgr2=0x7f181c15a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.169+0000 7f1822c00640 1 -- 172.21.15.69:0/930698949 shutdown_connections 2024-04-24T08:20:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.169+0000 7f1822c00640 1 -- 172.21.15.69:0/930698949 wait complete. 2024-04-24T08:20:43.188 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:20:43.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.381+0000 7f9268a00640 1 Processor -- start 2024-04-24T08:20:43.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.381+0000 7f9268a00640 1 -- start start 2024-04-24T08:20:43.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.383+0000 7f9268a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f926415b490 0x7f92640b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:43.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.383+0000 7f9268a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926406f2d0 0x7f926406f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:43.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.383+0000 7f9268a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f92640cfbe0 0x7f92640d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:43.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.383+0000 7f9268a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f926406fc70 con 0x7f926415b490 2024-04-24T08:20:43.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.383+0000 7f9268a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f92640b46e0 con 0x7f92640cfbe0 2024-04-24T08:20:43.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.383+0000 7f9268a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f92640d0960 con 0x7f926406f2d0 2024-04-24T08:20:43.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.383+0000 7f9261e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f926415b490 0x7f92640b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:43.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.383+0000 7f9261e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f926415b490 0x7f92640b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39518/0 (socket says 172.21.15.69:39518) 2024-04-24T08:20:43.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.383+0000 7f9261e00640 1 -- 172.21.15.69:0/482228586 learned_addr learned my addr 172.21.15.69:0/482228586 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:43.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.384+0000 7f9261400640 1 --2- 172.21.15.69:0/482228586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926406f2d0 0x7f926406f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:43.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.384+0000 7f9262800640 1 --2- 172.21.15.69:0/482228586 >> v2:172.21.15.121:3300/0 conn(0x7f92640cfbe0 0x7f92640d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:43.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.384+0000 7f9261e00640 1 -- 172.21.15.69:0/482228586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926406f2d0 msgr2=0x7f926406f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:43.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.384+0000 7f9261e00640 1 --2- 172.21.15.69:0/482228586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926406f2d0 0x7f926406f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.384+0000 7f9261e00640 1 -- 172.21.15.69:0/482228586 >> v2:172.21.15.121:3300/0 conn(0x7f92640cfbe0 msgr2=0x7f92640d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:43.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.384+0000 7f9261e00640 1 --2- 172.21.15.69:0/482228586 >> v2:172.21.15.121:3300/0 conn(0x7f92640cfbe0 0x7f92640d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.384+0000 7f9261e00640 1 -- 172.21.15.69:0/482228586 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92640b4980 con 0x7f926415b490 2024-04-24T08:20:43.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.385+0000 7f9261e00640 1 --2- 172.21.15.69:0/482228586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926415b490 0x7f92640b5180 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f926406e230 tx=0x7f925402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d021948d58fd1e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:43.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.386+0000 7f9260a00640 1 -- 172.21.15.69:0/482228586 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254037070 con 0x7f926415b490 2024-04-24T08:20:43.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.386+0000 7f9260a00640 1 -- 172.21.15.69:0/482228586 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9254032440 con 0x7f926415b490 2024-04-24T08:20:43.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.386+0000 7f9260a00640 1 -- 172.21.15.69:0/482228586 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254032810 con 0x7f926415b490 2024-04-24T08:20:43.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.387+0000 7f9268a00640 1 -- 172.21.15.69:0/482228586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926415b490 msgr2=0x7f92640b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:43.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.387+0000 7f9268a00640 1 --2- 172.21.15.69:0/482228586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926415b490 0x7f92640b5180 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f926406e230 tx=0x7f925402a1e0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.387+0000 7f9268a00640 1 -- 172.21.15.69:0/482228586 shutdown_connections 2024-04-24T08:20:43.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.387+0000 7f9268a00640 1 --2- 172.21.15.69:0/482228586 >> v2:172.21.15.121:3300/0 conn(0x7f92640cfbe0 0x7f92640d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.387+0000 7f9268a00640 1 --2- 172.21.15.69:0/482228586 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926406f2d0 0x7f926406f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.387+0000 7f9268a00640 1 --2- 172.21.15.69:0/482228586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926415b490 0x7f92640b5180 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.387+0000 7f9268a00640 1 -- 172.21.15.69:0/482228586 >> 172.21.15.69:0/482228586 conn(0x7f9264067d90 msgr2=0x7f926406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:43.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.387+0000 7f9268a00640 1 -- 172.21.15.69:0/482228586 shutdown_connections 2024-04-24T08:20:43.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.387+0000 7f9268a00640 1 -- 172.21.15.69:0/482228586 wait complete. 2024-04-24T08:20:43.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.389+0000 7f9268a00640 1 Processor -- start 2024-04-24T08:20:43.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.389+0000 7f9268a00640 1 -- start start 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9268a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926406f2d0 0x7f926415d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9268a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92640cfbe0 0x7f92640b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9268a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926415b490 0x7f926414da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9268a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f926415b870 con 0x7f926406f2d0 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9268a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f92640d0c40 con 0x7f92640cfbe0 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9268a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f92640d05b0 con 0x7f926415b490 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9261e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926406f2d0 0x7f926415d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9261e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926406f2d0 0x7f926415d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39520/0 (socket says 172.21.15.69:39520) 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9261e00640 1 -- 172.21.15.69:0/2937313623 learned_addr learned my addr 172.21.15.69:0/2937313623 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9261e00640 1 -- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926415b490 msgr2=0x7f926414da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9261e00640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926415b490 0x7f926414da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9261e00640 1 -- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92640cfbe0 msgr2=0x7f92640b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9261e00640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92640cfbe0 0x7f92640b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.391+0000 7f9261e00640 1 -- 172.21.15.69:0/2937313623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926415c7d0 con 0x7f926406f2d0 2024-04-24T08:20:43.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.392+0000 7f9261e00640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926406f2d0 0x7f926415d870 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f9254000c00 tx=0x7f92540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:43.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.392+0000 7f925a000640 1 -- 172.21.15.69:0/2937313623 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254041070 con 0x7f926406f2d0 2024-04-24T08:20:43.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.392+0000 7f925a000640 1 -- 172.21.15.69:0/2937313623 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f925403c870 con 0x7f926406f2d0 2024-04-24T08:20:43.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.392+0000 7f925a000640 1 -- 172.21.15.69:0/2937313623 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9254037040 con 0x7f926406f2d0 2024-04-24T08:20:43.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.392+0000 7f9268a00640 1 -- 172.21.15.69:0/2937313623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926415cab0 con 0x7f926406f2d0 2024-04-24T08:20:43.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.392+0000 7f9268a00640 1 -- 172.21.15.69:0/2937313623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926415d070 con 0x7f926406f2d0 2024-04-24T08:20:43.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.393+0000 7f9268a00640 1 -- 172.21.15.69:0/2937313623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9264068a80 con 0x7f926406f2d0 2024-04-24T08:20:43.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.395+0000 7f925a000640 1 -- 172.21.15.69:0/2937313623 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9254044070 con 0x7f926406f2d0 2024-04-24T08:20:43.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.395+0000 7f925a000640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f924406c940 0x7f924406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:43.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.396+0000 7f925a000640 1 -- 172.21.15.69:0/2937313623 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7f92540ac7f0 con 0x7f926406f2d0 2024-04-24T08:20:43.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.396+0000 7f9261400640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f924406c940 0x7f924406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:43.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.397+0000 7f9261400640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f924406c940 0x7f924406ee00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f92640adfe0 tx=0x7f9250008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:43.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.400+0000 7f925a000640 1 -- 172.21.15.69:0/2937313623 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f925407ad20 con 0x7f926406f2d0 2024-04-24T08:20:43.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.610+0000 7f9268a00640 1 -- 172.21.15.69:0/2937313623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f92640b31c0 con 0x7f926406f2d0 2024-04-24T08:20:43.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.611+0000 7f925a000640 1 -- 172.21.15.69:0/2937313623 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v109) ==== 80+0+0 (secure 0 0 0) 0x7f925407a620 con 0x7f926406f2d0 2024-04-24T08:20:43.612 INFO:teuthology.orchestra.run.smithi069.stderr:listed 0 entries 2024-04-24T08:20:43.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 -- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f924406c940 msgr2=0x7f924406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:43.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f924406c940 0x7f924406ee00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f92640adfe0 tx=0x7f9250008040 comp rx=0 tx=0).stop 2024-04-24T08:20:43.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 -- 172.21.15.69:0/2937313623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926406f2d0 msgr2=0x7f926415d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:43.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926406f2d0 0x7f926415d870 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f9254000c00 tx=0x7f92540046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 -- 172.21.15.69:0/2937313623 shutdown_connections 2024-04-24T08:20:43.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f924406c940 0x7f924406ee00 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f926415b490 0x7f926414da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f92640cfbe0 0x7f92640b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 --2- 172.21.15.69:0/2937313623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926406f2d0 0x7f926415d870 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:43.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 -- 172.21.15.69:0/2937313623 >> 172.21.15.69:0/2937313623 conn(0x7f9264067d90 msgr2=0x7f92640ce090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:43.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.613+0000 7f9268a00640 1 -- 172.21.15.69:0/2937313623 shutdown_connections 2024-04-24T08:20:43.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:43.614+0000 7f9268a00640 1 -- 172.21.15.69:0/2937313623 wait complete. 2024-04-24T08:20:43.632 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:20:43.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:43.632 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:43.706 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[76391]: fuse finished with error 0 and tester_r 0 2024-04-24T08:20:43.713 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:20:46.686 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:20:46.694 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:20:49.815 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:20:49.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:49.815 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:20:49.873 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:20:49.873 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:20:49.924 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:20:49.924 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi069.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:49.924 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:20:49.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:49.925 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:20:49.974 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-24T08:20:49.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:20:49.976 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:20:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.177+0000 7fe40b200640 1 Processor -- start 2024-04-24T08:20:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.177+0000 7fe40b200640 1 -- start start 2024-04-24T08:20:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.178+0000 7fe40b200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe404156c60 0x7fe40415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.178+0000 7fe40b200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe4040b30f0 0x7fe40415d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.178+0000 7fe40b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe40415d660 con 0x7fe4040ad0d0 2024-04-24T08:20:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.178+0000 7fe40b200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe40406dc30 con 0x7fe404156c60 2024-04-24T08:20:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.178+0000 7fe40b200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe40406df10 con 0x7fe4040b30f0 2024-04-24T08:20:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.179+0000 7fe409e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe404156c60 0x7fe40415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.179+0000 7fe40a800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe4040ad0d0 0x7fe4041600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46024/0 (socket says 172.21.15.69:46024) 2024-04-24T08:20:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.179+0000 7fe40a800640 1 -- 172.21.15.69:0/1724756851 learned_addr learned my addr 172.21.15.69:0/1724756851 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.179+0000 7fe409400640 1 --2- 172.21.15.69:0/1724756851 >> v2:172.21.15.121:3300/0 conn(0x7fe4040b30f0 0x7fe40415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.179+0000 7fe409e00640 1 -- 172.21.15.69:0/1724756851 >> v2:172.21.15.121:3300/0 conn(0x7fe4040b30f0 msgr2=0x7fe40415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.179+0000 7fe409e00640 1 --2- 172.21.15.69:0/1724756851 >> v2:172.21.15.121:3300/0 conn(0x7fe4040b30f0 0x7fe40415d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.179+0000 7fe409e00640 1 -- 172.21.15.69:0/1724756851 >> v1:172.21.15.69:6789/0 conn(0x7fe4040ad0d0 legacy=0x7fe4041600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.179+0000 7fe409e00640 1 -- 172.21.15.69:0/1724756851 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe40406e1f0 con 0x7fe404156c60 2024-04-24T08:20:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.179+0000 7fe408a00640 1 -- 172.21.15.69:0/1724756851 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3018054284 0 0) 0x7fe40415d660 con 0x7fe4040ad0d0 2024-04-24T08:20:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.180+0000 7fe409e00640 1 --2- 172.21.15.69:0/1724756851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe404156c60 0x7fe40415cac0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fe40406d900 tx=0x7fe3f802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d0059722fe9a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.181+0000 7fe408a00640 1 -- 172.21.15.69:0/1724756851 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8037070 con 0x7fe404156c60 2024-04-24T08:20:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.181+0000 7fe408a00640 1 -- 172.21.15.69:0/1724756851 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f8032440 con 0x7fe404156c60 2024-04-24T08:20:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.181+0000 7fe408a00640 1 -- 172.21.15.69:0/1724756851 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8032810 con 0x7fe404156c60 2024-04-24T08:20:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.181+0000 7fe40b200640 1 -- 172.21.15.69:0/1724756851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe404156c60 msgr2=0x7fe40415cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.181+0000 7fe40b200640 1 --2- 172.21.15.69:0/1724756851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe404156c60 0x7fe40415cac0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7fe40406d900 tx=0x7fe3f802a1a0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.181+0000 7fe40b200640 1 -- 172.21.15.69:0/1724756851 shutdown_connections 2024-04-24T08:20:50.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.181+0000 7fe40b200640 1 --2- 172.21.15.69:0/1724756851 >> v2:172.21.15.121:3300/0 conn(0x7fe4040b30f0 0x7fe40415d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.181+0000 7fe40b200640 1 --2- 172.21.15.69:0/1724756851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe404156c60 0x7fe40415cac0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.181+0000 7fe40b200640 1 -- 172.21.15.69:0/1724756851 >> 172.21.15.69:0/1724756851 conn(0x7fe4040d0160 msgr2=0x7fe4040d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.181+0000 7fe40b200640 1 -- 172.21.15.69:0/1724756851 shutdown_connections 2024-04-24T08:20:50.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.182+0000 7fe40b200640 1 -- 172.21.15.69:0/1724756851 wait complete. 2024-04-24T08:20:50.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.183+0000 7fe40b200640 1 Processor -- start 2024-04-24T08:20:50.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.184+0000 7fe40b200640 1 -- start start 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe40b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4040ad0d0 0x7fe4040cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe40b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4040b30f0 0x7fe404155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe40b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe404156c60 0x7fe4041558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe40b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4040ad540 con 0x7fe4040ad0d0 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe40b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe40415d940 con 0x7fe4040b30f0 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe40b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe4040ad6c0 con 0x7fe404156c60 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe409e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4040ad0d0 0x7fe4040cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe409e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4040ad0d0 0x7fe4040cb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40678/0 (socket says 172.21.15.69:40678) 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe409e00640 1 -- 172.21.15.69:0/3284514841 learned_addr learned my addr 172.21.15.69:0/3284514841 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe409400640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4040b30f0 0x7fe404155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe409e00640 1 -- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe404156c60 msgr2=0x7fe4041558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe409e00640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe404156c60 0x7fe4041558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe409e00640 1 -- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4040b30f0 msgr2=0x7fe404155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe409e00640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4040b30f0 0x7fe404155380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.185+0000 7fe409e00640 1 -- 172.21.15.69:0/3284514841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4040728a0 con 0x7fe4040ad0d0 2024-04-24T08:20:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.186+0000 7fe409e00640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4040ad0d0 0x7fe4040cb810 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8000c00 tx=0x7fe3f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.186+0000 7fe3fe000640 1 -- 172.21.15.69:0/3284514841 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8037040 con 0x7fe4040ad0d0 2024-04-24T08:20:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.186+0000 7fe3fe000640 1 -- 172.21.15.69:0/3284514841 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f8042070 con 0x7fe4040ad0d0 2024-04-24T08:20:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.186+0000 7fe40b200640 1 -- 172.21.15.69:0/3284514841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe404072b80 con 0x7fe4040ad0d0 2024-04-24T08:20:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.186+0000 7fe3fe000640 1 -- 172.21.15.69:0/3284514841 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8030a50 con 0x7fe4040ad0d0 2024-04-24T08:20:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.186+0000 7fe40b200640 1 -- 172.21.15.69:0/3284514841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe404073140 con 0x7fe4040ad0d0 2024-04-24T08:20:50.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.187+0000 7fe40b200640 1 -- 172.21.15.69:0/3284514841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4041460c0 con 0x7fe4040ad0d0 2024-04-24T08:20:50.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.189+0000 7fe3fe000640 1 -- 172.21.15.69:0/3284514841 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe3f8030bf0 con 0x7fe4040ad0d0 2024-04-24T08:20:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.189+0000 7fe3fe000640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3e806c8f0 0x7fe3e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.190+0000 7fe3fe000640 1 -- 172.21.15.69:0/3284514841 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 5709+0+0 (secure 0 0 0) 0x7fe3f80ac5f0 con 0x7fe4040ad0d0 2024-04-24T08:20:50.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.190+0000 7fe409400640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3e806c8f0 0x7fe3e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.192+0000 7fe409400640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3e806c8f0 0x7fe3e806edb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4006fd0 tx=0x7fe3f4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:50.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.195+0000 7fe3fe000640 1 -- 172.21.15.69:0/3284514841 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe4041460c0 con 0x7fe4040ad0d0 2024-04-24T08:20:50.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.434+0000 7fe40b200640 1 -- 172.21.15.69:0/3284514841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe404075d50 con 0x7fe4040ad0d0 2024-04-24T08:20:50.436 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:20:50.435+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:20:50.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.439+0000 7fe3fe000640 1 -- 172.21.15.69:0/3284514841 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v100) ==== 161+0+0 (secure 0 0 0) 0x7fe3f807aa10 con 0x7fe4040ad0d0 2024-04-24T08:20:50.441 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:20:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 -- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3e806c8f0 msgr2=0x7fe3e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3e806c8f0 0x7fe3e806edb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4006fd0 tx=0x7fe3f4008040 comp rx=0 tx=0).stop 2024-04-24T08:20:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 -- 172.21.15.69:0/3284514841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4040ad0d0 msgr2=0x7fe4040cb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4040ad0d0 0x7fe4040cb810 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8000c00 tx=0x7fe3f80046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 -- 172.21.15.69:0/3284514841 shutdown_connections 2024-04-24T08:20:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe3e806c8f0 0x7fe3e806edb0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe404156c60 0x7fe4041558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe4040b30f0 0x7fe404155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 --2- 172.21.15.69:0/3284514841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4040ad0d0 0x7fe4040cb810 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 -- 172.21.15.69:0/3284514841 >> 172.21.15.69:0/3284514841 conn(0x7fe4040d0160 msgr2=0x7fe40415ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 -- 172.21.15.69:0/3284514841 shutdown_connections 2024-04-24T08:20:50.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.442+0000 7fe40b200640 1 -- 172.21.15.69:0/3284514841 wait complete. 2024-04-24T08:20:50.465 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T08:20:50.472 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T08:20:50.482 INFO:tasks.ceph.mds.b.smithi121.stderr: -8561> 2024-04-24T08:20:10.713+0000 7fe60c400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:20:50.560 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2024-04-24T08:20:50.602 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.603 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-04-24T08:20:50.604 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-04-24T08:20:50.605 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.606 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.607 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: bdev async discard = True 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: bdev enable discard = True 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore allocator = bitmap 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 1/20 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 1/20 2024-04-24T08:20:50.608 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 4/10 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-24T08:20:50.609 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:20:50.610 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-04-24T08:20:50.611 INFO:teuthology.orchestra.run.smithi121.stdout: mon down mkfs grace = 300 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = "" 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-24T08:20:50.612 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds balancer = 20 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = False 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-24T08:20:50.613 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-24T08:20:50.614 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-24T08:20:50.614 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-24T08:20:50.614 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-24T08:20:50.614 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T08:20:50.614 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-24T08:20:50.614 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.614 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-24T08:20:50.614 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.614 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-24T08:20:50.615 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.616 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-24T08:20:50.617 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.618 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-04-24T08:20:50.619 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-24T08:20:50.620 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-24T08:20:50.621 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-24T08:20:50.622 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: mon down mkfs grace = 300 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:20:50.623 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-04-24T08:20:50.624 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-04-24T08:20:50.625 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = False 2024-04-24T08:20:50.625 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-24T08:20:50.625 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-24T08:20:50.625 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-24T08:20:50.639 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T08:20:50.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T08:20:50.762 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.763 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-04-24T08:20:50.764 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:20:50.765 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-04-24T08:20:50.766 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-04-24T08:20:50.767 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: bdev async discard = True 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: bdev enable discard = True 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore allocator = bitmap 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 1/20 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 1/20 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 4/10 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-24T08:20:50.768 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-04-24T08:20:50.769 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T08:20:50.770 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: mon down mkfs grace = 300 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T08:20:50.771 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = "" 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2024-04-24T08:20:50.772 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2024-04-24T08:20:50.773 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds balancer = 20 2024-04-24T08:20:50.773 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:20:50.773 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2024-04-24T08:20:50.773 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = False 2024-04-24T08:20:50.773 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2024-04-24T08:20:50.773 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = False 2024-04-24T08:20:50.773 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-24T08:20:50.773 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-24T08:20:50.773 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-24T08:20:50.773 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.774 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-24T08:20:50.775 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.776 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-24T08:20:50.777 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.778 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:20:50.779 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-24T08:20:50.780 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:20:50.781 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: mon down mkfs grace = 300 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:50.782 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-24T08:20:50.783 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = False 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = False 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-24T08:20:50.784 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-24T08:20:50.803 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmomapkey 10000000001.00000000 bravo_head 2024-04-24T08:20:50.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.909+0000 7f22c8c0bf40 1 Processor -- start 2024-04-24T08:20:50.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.909+0000 7f22c8c0bf40 1 -- start start 2024-04-24T08:20:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.910+0000 7f22c8c0bf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a8003ab0 0x55e673208740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.910+0000 7f22c8c0bf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673292610 0x55e673294a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.910+0000 7f22c8c0bf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e673288ff0 0x55e673294f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.910+0000 7f22c8c0bf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e673162cc0 con 0x55e673292610 2024-04-24T08:20:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.910+0000 7f22c8c0bf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e673166d00 con 0x55e673288ff0 2024-04-24T08:20:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.910+0000 7f22c8c0bf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55e6731ec180 con 0x7f22a8003ab0 2024-04-24T08:20:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.910+0000 7f22c4800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673292610 0x55e673294a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.910+0000 7f22c4800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673292610 0x55e673294a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40700/0 (socket says 172.21.15.69:40700) 2024-04-24T08:20:50.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.910+0000 7f22c4800640 1 -- 172.21.15.69:0/978648675 learned_addr learned my addr 172.21.15.69:0/978648675 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:50.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c5200640 1 --2- 172.21.15.69:0/978648675 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a8003ab0 0x55e673208740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c5c00640 1 --2- 172.21.15.69:0/978648675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e673288ff0 0x55e673294f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c4800640 1 -- 172.21.15.69:0/978648675 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a8003ab0 msgr2=0x55e673208740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:50.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c4800640 1 --2- 172.21.15.69:0/978648675 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a8003ab0 0x55e673208740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c4800640 1 -- 172.21.15.69:0/978648675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e673288ff0 msgr2=0x55e673294f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c4800640 1 --2- 172.21.15.69:0/978648675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e673288ff0 0x55e673294f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c4800640 1 -- 172.21.15.69:0/978648675 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6732894c0 con 0x55e673292610 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c4800640 1 --2- 172.21.15.69:0/978648675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673292610 0x55e673294a10 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f22b0003e00 tx=0x7f22b000d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba84e7b7ed02fcf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c3400640 1 -- 172.21.15.69:0/978648675 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b000de60 con 0x55e673292610 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c3400640 1 -- 172.21.15.69:0/978648675 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22b0003110 con 0x55e673292610 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.911+0000 7f22c3400640 1 -- 172.21.15.69:0/978648675 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0003300 con 0x55e673292610 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.912+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/978648675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673292610 msgr2=0x55e673294a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.912+0000 7f22c8c0bf40 1 --2- 172.21.15.69:0/978648675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673292610 0x55e673294a10 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f22b0003e00 tx=0x7f22b000d630 comp rx=0 tx=0).stop 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.912+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/978648675 shutdown_connections 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.912+0000 7f22c8c0bf40 1 --2- 172.21.15.69:0/978648675 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673292610 0x55e673294a10 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.912+0000 7f22c8c0bf40 1 --2- 172.21.15.69:0/978648675 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e673288ff0 0x55e673294f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.912+0000 7f22c8c0bf40 1 --2- 172.21.15.69:0/978648675 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a8003ab0 0x55e673208740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.912+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/978648675 >> 172.21.15.69:0/978648675 conn(0x55e6731eab50 msgr2=0x55e6731ba5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:50.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.912+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/978648675 shutdown_connections 2024-04-24T08:20:50.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.912+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/978648675 wait complete. 2024-04-24T08:20:50.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.913+0000 7f22c8c0bf40 1 Processor -- start 2024-04-24T08:20:50.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.913+0000 7f22c8c0bf40 1 -- start start 2024-04-24T08:20:50.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.914+0000 7f22c8c0bf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673288ff0 0x55e67327aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.914+0000 7f22c8c0bf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a8003ab0 0x55e673277770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.914+0000 7f22c8c0bf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e67328d3c0 0x55e673279520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.914+0000 7f22c8c0bf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e6731ea7a0 con 0x55e673288ff0 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.914+0000 7f22c8c0bf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55e67328a040 con 0x55e67328d3c0 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.914+0000 7f22c8c0bf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55e673295a80 con 0x7f22a8003ab0 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.914+0000 7f22c5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673288ff0 0x55e67327aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.914+0000 7f22c5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673288ff0 0x55e67327aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40704/0 (socket says 172.21.15.69:40704) 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.914+0000 7f22c5200640 1 -- 172.21.15.69:0/3888325914 learned_addr learned my addr 172.21.15.69:0/3888325914 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c4800640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a8003ab0 0x55e673277770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c5200640 1 -- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a8003ab0 msgr2=0x55e673277770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c5c00640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e67328d3c0 0x55e673279520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c5200640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f22a8003ab0 0x55e673277770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c5200640 1 -- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e67328d3c0 msgr2=0x55e673279520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c5200640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e67328d3c0 0x55e673279520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c5200640 1 -- 172.21.15.69:0/3888325914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e67327afc0 con 0x55e673288ff0 2024-04-24T08:20:50.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c5c00640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e67328d3c0 0x55e673279520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:50.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c5200640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673288ff0 0x55e67327aa80 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f22b4005fd0 tx=0x7f22b4009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:50.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c2000640 1 -- 172.21.15.69:0/3888325914 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b401b070 con 0x55e673288ff0 2024-04-24T08:20:50.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c2000640 1 -- 172.21.15.69:0/3888325914 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22b4016470 con 0x55e673288ff0 2024-04-24T08:20:50.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c2000640 1 -- 172.21.15.69:0/3888325914 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b4016860 con 0x55e673288ff0 2024-04-24T08:20:50.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/3888325914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e673278480 con 0x55e673288ff0 2024-04-24T08:20:50.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.915+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/3888325914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e673278a40 con 0x55e673288ff0 2024-04-24T08:20:50.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.917+0000 7f22c2000640 1 -- 172.21.15.69:0/3888325914 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f22b401e070 con 0x55e673288ff0 2024-04-24T08:20:50.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.918+0000 7f22c2000640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f22a406b2d0 0x7f22a406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.918+0000 7f22c2000640 1 -- 172.21.15.69:0/3888325914 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f22b408a570 con 0x55e673288ff0 2024-04-24T08:20:50.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.919+0000 7f22c8c0bf40 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55e673207aa0 0x55e673207ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:50.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.919+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/3888325914 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:1 16.7 16:e2e07930:::10000000001.00000000:head [omap-rm-keys in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110) -- 0x55e673278030 con 0x55e673207aa0 2024-04-24T08:20:50.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.919+0000 7f22c4800640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f22a406b2d0 0x7f22a406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.919+0000 7f22c5c00640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55e673207aa0 0x55e673207ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:50.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.920+0000 7f22c5c00640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55e673207aa0 0x55e673207ea0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:50.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.920+0000 7f22c4800640 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f22a406b2d0 0x7f22a406d790 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f22b0003e00 tx=0x7f22b0002000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:50.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c5c00640 1 -- 172.21.15.69:0/3888325914 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(1 10000000001.00000000 [omap-rm-keys] v110'5 uv5 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7f22bc0060f0 con 0x55e673207aa0 2024-04-24T08:20:50.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/3888325914 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55e673207aa0 msgr2=0x55e673207ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c8c0bf40 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55e673207aa0 0x55e673207ea0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f22a406b2d0 msgr2=0x7f22a406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c8c0bf40 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f22a406b2d0 0x7f22a406d790 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f22b0003e00 tx=0x7f22b0002000 comp rx=0 tx=0).stop 2024-04-24T08:20:50.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/3888325914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673288ff0 msgr2=0x55e67327aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:50.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c8c0bf40 1 --2- 172.21.15.69:0/3888325914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e673288ff0 0x55e67327aa80 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f22b4005fd0 tx=0x7f22b4009eb0 comp rx=0 tx=0).stop 2024-04-24T08:20:50.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c5c00640 1 -- 172.21.15.69:0/3888325914 reap_dead start 2024-04-24T08:20:50.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/3888325914 shutdown_connections 2024-04-24T08:20:50.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/3888325914 >> 172.21.15.69:0/3888325914 conn(0x55e6731eab50 msgr2=0x55e673291b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:50.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/3888325914 shutdown_connections 2024-04-24T08:20:50.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:50.924+0000 7f22c8c0bf40 1 -- 172.21.15.69:0/3888325914 wait complete. 2024-04-24T08:20:50.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-24T08:20:51.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.116+0000 7fc3e1800640 1 Processor -- start 2024-04-24T08:20:51.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.116+0000 7fc3e1800640 1 -- start start 2024-04-24T08:20:51.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.117+0000 7fc3e1800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc3dc071d70 0x7fc3dc072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.117+0000 7fc3e1800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3dc06e000 0x7fc3dc06e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.117+0000 7fc3e1800640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3dc072690 con 0x7fc3dc15edf0 2024-04-24T08:20:51.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.117+0000 7fc3e1800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc3dc0d0a00 con 0x7fc3dc071d70 2024-04-24T08:20:51.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.117+0000 7fc3e1800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc3dc15f960 con 0x7fc3dc06e000 2024-04-24T08:20:51.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.118+0000 7fc3dbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc3dc071d70 0x7fc3dc072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:51.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.118+0000 7fc3d3400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc3dc15edf0 0x7fc3dc15f250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46038/0 (socket says 172.21.15.69:46038) 2024-04-24T08:20:51.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.118+0000 7fc3d3400640 1 -- 172.21.15.69:0/765900980 learned_addr learned my addr 172.21.15.69:0/765900980 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:51.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.118+0000 7fc3dbe00640 1 -- 172.21.15.69:0/765900980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3dc06e000 msgr2=0x7fc3dc06e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:51.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.118+0000 7fc3dbe00640 1 --2- 172.21.15.69:0/765900980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3dc06e000 0x7fc3dc06e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.118+0000 7fc3dbe00640 1 -- 172.21.15.69:0/765900980 >> v1:172.21.15.69:6789/0 conn(0x7fc3dc15edf0 legacy=0x7fc3dc15f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.118+0000 7fc3dbe00640 1 -- 172.21.15.69:0/765900980 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc15fbb0 con 0x7fc3dc071d70 2024-04-24T08:20:51.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.119+0000 7fc3dbe00640 1 --2- 172.21.15.69:0/765900980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc071d70 0x7fc3dc072150 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0097a0 tx=0x7fc3cc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74306e6e5e8436de server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:51.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3db400640 1 -- 172.21.15.69:0/765900980 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc037070 con 0x7fc3dc071d70 2024-04-24T08:20:51.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3db400640 1 -- 172.21.15.69:0/765900980 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3cc032440 con 0x7fc3dc071d70 2024-04-24T08:20:51.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3db400640 1 -- 172.21.15.69:0/765900980 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc032810 con 0x7fc3dc071d70 2024-04-24T08:20:51.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3e1800640 1 -- 172.21.15.69:0/765900980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc071d70 msgr2=0x7fc3dc072150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3e1800640 1 --2- 172.21.15.69:0/765900980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc071d70 0x7fc3dc072150 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0097a0 tx=0x7fc3cc02a180 comp rx=0 tx=0).stop 2024-04-24T08:20:51.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3e1800640 1 -- 172.21.15.69:0/765900980 shutdown_connections 2024-04-24T08:20:51.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3e1800640 1 --2- 172.21.15.69:0/765900980 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3dc06e000 0x7fc3dc06e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3e1800640 1 --2- 172.21.15.69:0/765900980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc071d70 0x7fc3dc072150 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3e1800640 1 -- 172.21.15.69:0/765900980 >> 172.21.15.69:0/765900980 conn(0x7fc3dc067b80 msgr2=0x7fc3dc06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:51.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3e1800640 1 -- 172.21.15.69:0/765900980 shutdown_connections 2024-04-24T08:20:51.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.120+0000 7fc3e1800640 1 -- 172.21.15.69:0/765900980 wait complete. 2024-04-24T08:20:51.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.122+0000 7fc3e1800640 1 Processor -- start 2024-04-24T08:20:51.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.122+0000 7fc3e1800640 1 -- start start 2024-04-24T08:20:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.123+0000 7fc3e1800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3dc06e000 0x7fc3dc15d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.123+0000 7fc3e1800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3dc071d70 0x7fc3dc146b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.123+0000 7fc3e1800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc15edf0 0x7fc3dc147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.123+0000 7fc3e1800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc0d0690 con 0x7fc3dc15edf0 2024-04-24T08:20:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.123+0000 7fc3e1800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc0d03b0 con 0x7fc3dc071d70 2024-04-24T08:20:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.123+0000 7fc3e1800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc3dc0d0e90 con 0x7fc3dc06e000 2024-04-24T08:20:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.123+0000 7fc3e0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc15edf0 0x7fc3dc147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.123+0000 7fc3e0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc15edf0 0x7fc3dc147060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40720/0 (socket says 172.21.15.69:40720) 2024-04-24T08:20:51.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.123+0000 7fc3e0e00640 1 -- 172.21.15.69:0/2542442810 learned_addr learned my addr 172.21.15.69:0/2542442810 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3d3400640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3dc071d70 0x7fc3dc146b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3dbe00640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3dc06e000 0x7fc3dc15d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3e0e00640 1 -- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3dc06e000 msgr2=0x7fc3dc15d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3e0e00640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3dc06e000 0x7fc3dc15d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3e0e00640 1 -- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3dc071d70 msgr2=0x7fc3dc146b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3e0e00640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3dc071d70 0x7fc3dc146b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3e0e00640 1 -- 172.21.15.69:0/2542442810 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc139d70 con 0x7fc3dc15edf0 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3d3400640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3dc071d70 0x7fc3dc146b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3e0e00640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc15edf0 0x7fc3dc147060 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400b970 tx=0x7fc3d400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3d8c00640 1 -- 172.21.15.69:0/2542442810 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4019070 con 0x7fc3dc15edf0 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3d8c00640 1 -- 172.21.15.69:0/2542442810 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d400cb60 con 0x7fc3dc15edf0 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3e1800640 1 -- 172.21.15.69:0/2542442810 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3dc13a050 con 0x7fc3dc15edf0 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.124+0000 7fc3d8c00640 1 -- 172.21.15.69:0/2542442810 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4014440 con 0x7fc3dc15edf0 2024-04-24T08:20:51.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.125+0000 7fc3e1800640 1 -- 172.21.15.69:0/2542442810 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3dc13d380 con 0x7fc3dc15edf0 2024-04-24T08:20:51.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.126+0000 7fc3e1800640 1 -- 172.21.15.69:0/2542442810 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3dc13d640 con 0x7fc3dc15edf0 2024-04-24T08:20:51.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.127+0000 7fc3d8c00640 1 -- 172.21.15.69:0/2542442810 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc3d401c070 con 0x7fc3dc15edf0 2024-04-24T08:20:51.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.127+0000 7fc3d8c00640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc3b806c8f0 0x7fc3b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.128+0000 7fc3d8c00640 1 -- 172.21.15.69:0/2542442810 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7fc3d408b900 con 0x7fc3dc15edf0 2024-04-24T08:20:51.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.128+0000 7fc3dbe00640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc3b806c8f0 0x7fc3b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:51.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.129+0000 7fc3dbe00640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc3b806c8f0 0x7fc3b806edb0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc034000 tx=0x7fc3cc02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:51.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.133+0000 7fc3d8c00640 1 -- 172.21.15.69:0/2542442810 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3dc13d640 con 0x7fc3dc15edf0 2024-04-24T08:20:51.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.382+0000 7fc3e1800640 1 -- 172.21.15.69:0/2542442810 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fc3dc075d50 con 0x7fc3dc15edf0 2024-04-24T08:20:51.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.451+0000 7fc3d8c00640 1 -- 172.21.15.69:0/2542442810 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v101) ==== 162+0+0 (secure 0 0 0) 0x7fc3d4059cc0 con 0x7fc3dc15edf0 2024-04-24T08:20:51.453 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-24T08:20:51.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.454+0000 7fc3e1800640 1 -- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc3b806c8f0 msgr2=0x7fc3b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.454+0000 7fc3e1800640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc3b806c8f0 0x7fc3b806edb0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc034000 tx=0x7fc3cc02e000 comp rx=0 tx=0).stop 2024-04-24T08:20:51.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.454+0000 7fc3e1800640 1 -- 172.21.15.69:0/2542442810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc15edf0 msgr2=0x7fc3dc147060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.454+0000 7fc3e1800640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc15edf0 0x7fc3dc147060 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400b970 tx=0x7fc3d400be30 comp rx=0 tx=0).stop 2024-04-24T08:20:51.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.454+0000 7fc3e1800640 1 -- 172.21.15.69:0/2542442810 shutdown_connections 2024-04-24T08:20:51.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.454+0000 7fc3e1800640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc3b806c8f0 0x7fc3b806edb0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.454+0000 7fc3e1800640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc3dc15edf0 0x7fc3dc147060 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.454+0000 7fc3e1800640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3dc071d70 0x7fc3dc146b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.454+0000 7fc3e1800640 1 --2- 172.21.15.69:0/2542442810 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc3dc06e000 0x7fc3dc15d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.454+0000 7fc3e1800640 1 -- 172.21.15.69:0/2542442810 >> 172.21.15.69:0/2542442810 conn(0x7fc3dc067b80 msgr2=0x7fc3dc06e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:51.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.455+0000 7fc3e1800640 1 -- 172.21.15.69:0/2542442810 shutdown_connections 2024-04-24T08:20:51.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.455+0000 7fc3e1800640 1 -- 172.21.15.69:0/2542442810 wait complete. 2024-04-24T08:20:51.477 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:51.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.651+0000 7fd9cde00640 1 Processor -- start 2024-04-24T08:20:51.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.651+0000 7fd9cde00640 1 -- start start 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.652+0000 7fd9cde00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9c815b490 0x7fd9c80b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.652+0000 7fd9cde00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9c80d02f0 0x7fd9c806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.652+0000 7fd9cde00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c80b46e0 con 0x7fd9c806f2d0 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.652+0000 7fd9cde00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd9c806fc70 con 0x7fd9c815b490 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.652+0000 7fd9cde00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c80d0960 con 0x7fd9c80d02f0 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.653+0000 7fd9c6a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9c815b490 0x7fd9c80b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.653+0000 7fd9c6a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd9c815b490 0x7fd9c80b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40722/0 (socket says 172.21.15.69:40722) 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.653+0000 7fd9c6a00640 1 -- 172.21.15.69:0/743838961 learned_addr learned my addr 172.21.15.69:0/743838961 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.653+0000 7fd9c6a00640 1 -- 172.21.15.69:0/743838961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9c80d02f0 msgr2=0x7fd9c806f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.653+0000 7fd9c7400640 1 --2- 172.21.15.69:0/743838961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9c80d02f0 0x7fd9c806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.653+0000 7fd9c6a00640 1 --2- 172.21.15.69:0/743838961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9c80d02f0 0x7fd9c806f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.653+0000 7fd9c6a00640 1 -- 172.21.15.69:0/743838961 >> v1:172.21.15.121:6789/0 conn(0x7fd9c806f2d0 legacy=0x7fd9c80cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:51.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.653+0000 7fd9c6a00640 1 -- 172.21.15.69:0/743838961 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c80b4980 con 0x7fd9c815b490 2024-04-24T08:20:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.654+0000 7fd9c6a00640 1 --2- 172.21.15.69:0/743838961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c815b490 0x7fd9c80b5180 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009870 tx=0x7fd9b802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83912af150f3dac server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.654+0000 7fd9c5600640 1 -- 172.21.15.69:0/743838961 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8037070 con 0x7fd9c815b490 2024-04-24T08:20:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.654+0000 7fd9c5600640 1 -- 172.21.15.69:0/743838961 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b8032440 con 0x7fd9c815b490 2024-04-24T08:20:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.654+0000 7fd9c5600640 1 -- 172.21.15.69:0/743838961 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8032810 con 0x7fd9c815b490 2024-04-24T08:20:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.655+0000 7fd9cde00640 1 -- 172.21.15.69:0/743838961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c815b490 msgr2=0x7fd9c80b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.655+0000 7fd9cde00640 1 --2- 172.21.15.69:0/743838961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c815b490 0x7fd9c80b5180 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009870 tx=0x7fd9b802a200 comp rx=0 tx=0).stop 2024-04-24T08:20:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.655+0000 7fd9cde00640 1 -- 172.21.15.69:0/743838961 shutdown_connections 2024-04-24T08:20:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.655+0000 7fd9cde00640 1 --2- 172.21.15.69:0/743838961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9c80d02f0 0x7fd9c806f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.655+0000 7fd9cde00640 1 --2- 172.21.15.69:0/743838961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c815b490 0x7fd9c80b5180 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.655+0000 7fd9cde00640 1 -- 172.21.15.69:0/743838961 >> 172.21.15.69:0/743838961 conn(0x7fd9c8067d90 msgr2=0x7fd9c806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.655+0000 7fd9cde00640 1 -- 172.21.15.69:0/743838961 shutdown_connections 2024-04-24T08:20:51.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.655+0000 7fd9cde00640 1 -- 172.21.15.69:0/743838961 wait complete. 2024-04-24T08:20:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.657+0000 7fd9cde00640 1 Processor -- start 2024-04-24T08:20:51.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.657+0000 7fd9cde00640 1 -- start start 2024-04-24T08:20:51.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.658+0000 7fd9cde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c806f2d0 0x7fd9c8147a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.658+0000 7fd9cde00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9c80d02f0 0x7fd9c814df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9cde00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9c815b490 0x7fd9c814e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9cde00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c815b870 con 0x7fd9c806f2d0 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9cde00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c80d0c40 con 0x7fd9c815b490 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9cde00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c80b56c0 con 0x7fd9c80d02f0 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9c6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c806f2d0 0x7fd9c8147a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9c6a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c806f2d0 0x7fd9c8147a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40724/0 (socket says 172.21.15.69:40724) 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9c6a00640 1 -- 172.21.15.69:0/2437189636 learned_addr learned my addr 172.21.15.69:0/2437189636 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9c6a00640 1 -- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9c80d02f0 msgr2=0x7fd9c814df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9c6a00640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9c80d02f0 0x7fd9c814df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9c6a00640 1 -- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9c815b490 msgr2=0x7fd9c814e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9c6a00640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9c815b490 0x7fd9c814e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.659+0000 7fd9c6a00640 1 -- 172.21.15.69:0/2437189636 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c814c720 con 0x7fd9c806f2d0 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.660+0000 7fd9c6a00640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c806f2d0 0x7fd9c8147a50 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009870 tx=0x7fd9b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:51.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.660+0000 7fd9bf400640 1 -- 172.21.15.69:0/2437189636 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8041020 con 0x7fd9c806f2d0 2024-04-24T08:20:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.660+0000 7fd9bf400640 1 -- 172.21.15.69:0/2437189636 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b8030970 con 0x7fd9c806f2d0 2024-04-24T08:20:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.660+0000 7fd9cde00640 1 -- 172.21.15.69:0/2437189636 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c8159b40 con 0x7fd9c806f2d0 2024-04-24T08:20:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.660+0000 7fd9bf400640 1 -- 172.21.15.69:0/2437189636 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8037040 con 0x7fd9c806f2d0 2024-04-24T08:20:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.660+0000 7fd9cde00640 1 -- 172.21.15.69:0/2437189636 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c8148510 con 0x7fd9c806f2d0 2024-04-24T08:20:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.661+0000 7fd9cde00640 1 -- 172.21.15.69:0/2437189636 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c8068a80 con 0x7fd9c806f2d0 2024-04-24T08:20:51.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.662+0000 7fd9bf400640 1 -- 172.21.15.69:0/2437189636 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9b8044020 con 0x7fd9c806f2d0 2024-04-24T08:20:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.663+0000 7fd9bf400640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c8f0 0x7fd9a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.664+0000 7fd9bf400640 1 -- 172.21.15.69:0/2437189636 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7fd9b80ac640 con 0x7fd9c806f2d0 2024-04-24T08:20:51.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.664+0000 7fd9c6000640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c8f0 0x7fd9a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:51.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.665+0000 7fd9c6000640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c8f0 0x7fd9a806edb0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd9c814f680 tx=0x7fd9b40079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:51.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.668+0000 7fd9bf400640 1 -- 172.21.15.69:0/2437189636 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9b804b030 con 0x7fd9c806f2d0 2024-04-24T08:20:51.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.906+0000 7fd9cde00640 1 -- 172.21.15.69:0/2437189636 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9c806eeb0 con 0x7fd9c806f2d0 2024-04-24T08:20:51.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.908+0000 7fd9bf400640 1 -- 172.21.15.69:0/2437189636 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+5185 (secure 0 0 0) 0x7fd9b807a970 con 0x7fd9c806f2d0 2024-04-24T08:20:51.909 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:51.909 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":102,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:20:51.457514+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9285":{"gid":9285,"name":"c","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.69:6835/2194021008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2194021008},{"type":"v1","addr":"172.21.15.69:6835","nonce":2194021008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-04-24T08:20:51.910 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 102 2024-04-24T08:20:51.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 -- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c8f0 msgr2=0x7fd9a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c8f0 0x7fd9a806edb0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd9c814f680 tx=0x7fd9b40079b0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 -- 172.21.15.69:0/2437189636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c806f2d0 msgr2=0x7fd9c8147a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:51.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c806f2d0 0x7fd9c8147a50 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009870 tx=0x7fd9b80046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 -- 172.21.15.69:0/2437189636 shutdown_connections 2024-04-24T08:20:51.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c8f0 0x7fd9a806edb0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9c815b490 0x7fd9c814e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9c80d02f0 0x7fd9c814df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 --2- 172.21.15.69:0/2437189636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9c806f2d0 0x7fd9c8147a50 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:51.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 -- 172.21.15.69:0/2437189636 >> 172.21.15.69:0/2437189636 conn(0x7fd9c8067d90 msgr2=0x7fd9c80b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:51.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.910+0000 7fd9cde00640 1 -- 172.21.15.69:0/2437189636 shutdown_connections 2024-04-24T08:20:51.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:51.911+0000 7fd9cde00640 1 -- 172.21.15.69:0/2437189636 wait complete. 2024-04-24T08:20:52.331 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:20:09.204175+0000', 'modified': '2024-04-24T08:20:51.457514+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 110, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9285': {'gid': 9285, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.69:6835/2194021008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 2194021008}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 2194021008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:20:52.332 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9285:up:replay 2024-04-24T08:20:53.332 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:53.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.542+0000 7f8006600640 1 Processor -- start 2024-04-24T08:20:53.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.542+0000 7f8006600640 1 -- start start 2024-04-24T08:20:53.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.544+0000 7f8006600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f800015e960 0x7f8000162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.544+0000 7f8006600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8000162980 0x7f800006cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.544+0000 7f8006600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f800006d4f0 0x7f80000cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.544+0000 7f8006600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f800015ed40 con 0x7f800006d4f0 2024-04-24T08:20:53.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.544+0000 7f8006600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f80000cf050 con 0x7f8000162980 2024-04-24T08:20:53.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.544+0000 7f8006600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f8000162e20 con 0x7f800015e960 2024-04-24T08:20:53.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.544+0000 7f8005c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f800006d4f0 0x7f80000cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:53.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.544+0000 7f8005c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f800006d4f0 0x7f80000cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40732/0 (socket says 172.21.15.69:40732) 2024-04-24T08:20:53.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.544+0000 7f8005c00640 1 -- 172.21.15.69:0/3469817400 learned_addr learned my addr 172.21.15.69:0/3469817400 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:53.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f8005c00640 1 -- 172.21.15.69:0/3469817400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f800015e960 msgr2=0x7f8000162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:53.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f8005c00640 1 --2- 172.21.15.69:0/3469817400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f800015e960 0x7f8000162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f8005c00640 1 -- 172.21.15.69:0/3469817400 >> v2:172.21.15.121:3300/0 conn(0x7f8000162980 msgr2=0x7f800006cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:53.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f8005c00640 1 --2- 172.21.15.69:0/3469817400 >> v2:172.21.15.121:3300/0 conn(0x7f8000162980 0x7f800006cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f8005c00640 1 -- 172.21.15.69:0/3469817400 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8000163100 con 0x7f800006d4f0 2024-04-24T08:20:53.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f8005c00640 1 --2- 172.21.15.69:0/3469817400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800006d4f0 0x7f80000cfaf0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f800009e030 tx=0x7f7ff802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379591331a89ce0e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:53.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f7fffe00640 1 -- 172.21.15.69:0/3469817400 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8036070 con 0x7f800006d4f0 2024-04-24T08:20:53.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f7fffe00640 1 -- 172.21.15.69:0/3469817400 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ff8031600 con 0x7f800006d4f0 2024-04-24T08:20:53.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f7fffe00640 1 -- 172.21.15.69:0/3469817400 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff80319d0 con 0x7f800006d4f0 2024-04-24T08:20:53.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f8006600640 1 -- 172.21.15.69:0/3469817400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800006d4f0 msgr2=0x7f80000cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:53.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.545+0000 7f8006600640 1 --2- 172.21.15.69:0/3469817400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800006d4f0 0x7f80000cfaf0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f800009e030 tx=0x7f7ff802a6a0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.546+0000 7f8006600640 1 -- 172.21.15.69:0/3469817400 shutdown_connections 2024-04-24T08:20:53.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.546+0000 7f8006600640 1 --2- 172.21.15.69:0/3469817400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800006d4f0 0x7f80000cfaf0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.546+0000 7f8006600640 1 --2- 172.21.15.69:0/3469817400 >> v2:172.21.15.121:3300/0 conn(0x7f8000162980 0x7f800006cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.546+0000 7f8006600640 1 --2- 172.21.15.69:0/3469817400 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f800015e960 0x7f8000162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.546+0000 7f8006600640 1 -- 172.21.15.69:0/3469817400 >> 172.21.15.69:0/3469817400 conn(0x7f800009d900 msgr2=0x7f800008bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:53.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.546+0000 7f8006600640 1 -- 172.21.15.69:0/3469817400 shutdown_connections 2024-04-24T08:20:53.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.546+0000 7f8006600640 1 -- 172.21.15.69:0/3469817400 wait complete. 2024-04-24T08:20:53.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.547+0000 7f8006600640 1 Processor -- start 2024-04-24T08:20:53.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.547+0000 7f8006600640 1 -- start start 2024-04-24T08:20:53.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.548+0000 7f8006600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800006d4f0 0x7f80001619b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.548+0000 7f8006600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800015e960 0x7f80000cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.548+0000 7f8006600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8000162980 0x7f80000cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.548+0000 7f8006600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80000b6170 con 0x7f800015e960 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.548+0000 7f8006600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f80000cf5d0 con 0x7f800006d4f0 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.548+0000 7f8006600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f80000b62f0 con 0x7f8000162980 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.549+0000 7f7ff7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800015e960 0x7f80000cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.549+0000 7f7ff7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800015e960 0x7f80000cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40748/0 (socket says 172.21.15.69:40748) 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.549+0000 7f7ff7e00640 1 -- 172.21.15.69:0/2996270840 learned_addr learned my addr 172.21.15.69:0/2996270840 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.549+0000 7f7ff7e00640 1 -- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8000162980 msgr2=0x7f80000cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.549+0000 7f7ff7e00640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8000162980 0x7f80000cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.549+0000 7f7ff7e00640 1 -- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800006d4f0 msgr2=0x7f80001619b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.549+0000 7f7ff7e00640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800006d4f0 0x7f80001619b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.549+0000 7f7ff7e00640 1 -- 172.21.15.69:0/2996270840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80001e83d0 con 0x7f800015e960 2024-04-24T08:20:53.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.549+0000 7f7ff7e00640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800015e960 0x7f80000cd1a0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800b440 tx=0x7f7fe800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:53.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.550+0000 7f7ffd600640 1 -- 172.21.15.69:0/2996270840 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8015020 con 0x7f800015e960 2024-04-24T08:20:53.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.550+0000 7f7ffd600640 1 -- 172.21.15.69:0/2996270840 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fe8011890 con 0x7f800015e960 2024-04-24T08:20:53.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.550+0000 7f7ffd600640 1 -- 172.21.15.69:0/2996270840 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe8011c80 con 0x7f800015e960 2024-04-24T08:20:53.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.550+0000 7f8006600640 1 -- 172.21.15.69:0/2996270840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80000cdd10 con 0x7f800015e960 2024-04-24T08:20:53.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.550+0000 7f8006600640 1 -- 172.21.15.69:0/2996270840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8000148c00 con 0x7f800015e960 2024-04-24T08:20:53.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.552+0000 7f8006600640 1 -- 172.21.15.69:0/2996270840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8000075d50 con 0x7f800015e960 2024-04-24T08:20:53.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.553+0000 7f7ffd600640 1 -- 172.21.15.69:0/2996270840 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7fe8018020 con 0x7f800015e960 2024-04-24T08:20:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.553+0000 7f7ffd600640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7fdc06c8f0 0x7f7fdc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.553+0000 7f7ffd600640 1 -- 172.21.15.69:0/2996270840 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f7fe808bd40 con 0x7f800015e960 2024-04-24T08:20:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.553+0000 7f8005200640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7fdc06c8f0 0x7f7fdc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.556+0000 7f8005200640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7fdc06c8f0 0x7f7fdc06edb0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009710 tx=0x7f7ff0006c90 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:53.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.558+0000 7f7ffd600640 1 -- 172.21.15.69:0/2996270840 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fe801f020 con 0x7f800015e960 2024-04-24T08:20:53.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.795+0000 7f8006600640 1 -- 172.21.15.69:0/2996270840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80000d24d0 con 0x7f800015e960 2024-04-24T08:20:53.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.796+0000 7f7ffd600640 1 -- 172.21.15.69:0/2996270840 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) ==== 95+0+5186 (secure 0 0 0) 0x7f7fe800d040 con 0x7f800015e960 2024-04-24T08:20:53.797 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:53.797 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":104,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:20:52.470230+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9285":{"gid":9285,"name":"c","rank":0,"incarnation":102,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.69:6835/2194021008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2194021008},{"type":"v1","addr":"172.21.15.69:6835","nonce":2194021008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-04-24T08:20:53.797 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 104 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.798+0000 7f8006600640 1 -- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7fdc06c8f0 msgr2=0x7f7fdc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.798+0000 7f8006600640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7fdc06c8f0 0x7f7fdc06edb0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f7ff0009710 tx=0x7f7ff0006c90 comp rx=0 tx=0).stop 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.798+0000 7f8006600640 1 -- 172.21.15.69:0/2996270840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800015e960 msgr2=0x7f80000cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.798+0000 7f8006600640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800015e960 0x7f80000cd1a0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f7fe800b440 tx=0x7f7fe800b900 comp rx=0 tx=0).stop 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.798+0000 7f8006600640 1 -- 172.21.15.69:0/2996270840 shutdown_connections 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.798+0000 7f8006600640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7fdc06c8f0 0x7f7fdc06edb0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.798+0000 7f8006600640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f800015e960 0x7f80000cd1a0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.798+0000 7f8006600640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8000162980 0x7f80000cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.798+0000 7f8006600640 1 --2- 172.21.15.69:0/2996270840 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f800006d4f0 0x7f80001619b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.798+0000 7f8006600640 1 -- 172.21.15.69:0/2996270840 >> 172.21.15.69:0/2996270840 conn(0x7f800009d900 msgr2=0x7f80000d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:53.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.799+0000 7f8006600640 1 -- 172.21.15.69:0/2996270840 shutdown_connections 2024-04-24T08:20:53.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:53.799+0000 7f8006600640 1 -- 172.21.15.69:0/2996270840 wait complete. 2024-04-24T08:20:54.220 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:20:09.204175+0000', 'modified': '2024-04-24T08:20:52.470230+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 110, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9285': {'gid': 9285, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:rejoin', 'state_seq': 14, 'addr': '172.21.15.69:6835/2194021008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 2194021008}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 2194021008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:20:54.220 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9285:up:rejoin 2024-04-24T08:20:55.220 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:55.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.430+0000 7f8621c00640 1 Processor -- start 2024-04-24T08:20:55.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.430+0000 7f8621c00640 1 -- start start 2024-04-24T08:20:55.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.431+0000 7f8621c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c06f730 0x7f861c0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.432+0000 7f8621c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f861c072480 0x7f861c165920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.432+0000 7f8621c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f861c0b4370 con 0x7f861c0b3320 2024-04-24T08:20:55.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.432+0000 7f8621c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f861c0b4f80 con 0x7f861c072480 2024-04-24T08:20:55.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.432+0000 7f8621c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f861c0b4650 con 0x7f861c06f730 2024-04-24T08:20:55.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.432+0000 7f861aa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c06f730 0x7f861c0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.432+0000 7f861b400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f861c072480 0x7f861c165920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.433+0000 7f861a000640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f861c0b3320 0x7f861c071d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:39614/0 (socket says 172.21.15.69:39614) 2024-04-24T08:20:55.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.433+0000 7f861aa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c06f730 0x7f861c0d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:38374/0 (socket says 172.21.15.69:38374) 2024-04-24T08:20:55.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.433+0000 7f861a000640 1 -- 172.21.15.69:0/4251260833 learned_addr learned my addr 172.21.15.69:0/4251260833 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:55.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.434+0000 7f861aa00640 1 -- 172.21.15.69:0/4251260833 >> v2:172.21.15.121:3300/0 conn(0x7f861c072480 msgr2=0x7f861c165920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:55.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.434+0000 7f861aa00640 1 --2- 172.21.15.69:0/4251260833 >> v2:172.21.15.121:3300/0 conn(0x7f861c072480 0x7f861c165920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.434+0000 7f861aa00640 1 -- 172.21.15.69:0/4251260833 >> v1:172.21.15.121:6789/0 conn(0x7f861c0b3320 legacy=0x7f861c071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.434+0000 7f861aa00640 1 -- 172.21.15.69:0/4251260833 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861c0b4930 con 0x7f861c06f730 2024-04-24T08:20:55.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.434+0000 7f8619600640 1 -- 172.21.15.69:0/4251260833 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3415345507 0 0) 0x7f861c0b4370 con 0x7f861c0b3320 2024-04-24T08:20:55.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.434+0000 7f861b400640 1 --2- 172.21.15.69:0/4251260833 >> v2:172.21.15.121:3300/0 conn(0x7f861c072480 0x7f861c165920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:55.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.435+0000 7f861aa00640 1 --2- 172.21.15.69:0/4251260833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c06f730 0x7f861c0d0370 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f86140097a0 tx=0x7f861402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33d757e0ae3800de server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:55.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.436+0000 7f8619600640 1 -- 172.21.15.69:0/4251260833 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614037070 con 0x7f861c06f730 2024-04-24T08:20:55.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.436+0000 7f8619600640 1 -- 172.21.15.69:0/4251260833 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8614032440 con 0x7f861c06f730 2024-04-24T08:20:55.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.436+0000 7f8619600640 1 -- 172.21.15.69:0/4251260833 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614032810 con 0x7f861c06f730 2024-04-24T08:20:55.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.436+0000 7f8621c00640 1 -- 172.21.15.69:0/4251260833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c06f730 msgr2=0x7f861c0d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.436+0000 7f8621c00640 1 --2- 172.21.15.69:0/4251260833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c06f730 0x7f861c0d0370 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f86140097a0 tx=0x7f861402a180 comp rx=0 tx=0).stop 2024-04-24T08:20:55.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.436+0000 7f8621c00640 1 -- 172.21.15.69:0/4251260833 shutdown_connections 2024-04-24T08:20:55.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.436+0000 7f8621c00640 1 --2- 172.21.15.69:0/4251260833 >> v2:172.21.15.121:3300/0 conn(0x7f861c072480 0x7f861c165920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.436+0000 7f8621c00640 1 --2- 172.21.15.69:0/4251260833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c06f730 0x7f861c0d0370 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.436+0000 7f8621c00640 1 -- 172.21.15.69:0/4251260833 >> 172.21.15.69:0/4251260833 conn(0x7f861c067d90 msgr2=0x7f861c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:55.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.437+0000 7f8621c00640 1 -- 172.21.15.69:0/4251260833 shutdown_connections 2024-04-24T08:20:55.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.437+0000 7f8621c00640 1 -- 172.21.15.69:0/4251260833 wait complete. 2024-04-24T08:20:55.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.438+0000 7f8621c00640 1 Processor -- start 2024-04-24T08:20:55.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.438+0000 7f8621c00640 1 -- start start 2024-04-24T08:20:55.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.439+0000 7f8621c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f861c06f730 0x7f861c13a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.439+0000 7f8621c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c072480 0x7f861c15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.439+0000 7f8621c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f861c0b3320 0x7f861c0c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.439+0000 7f8621c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f861c0d08b0 con 0x7f861c0b3320 2024-04-24T08:20:55.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.439+0000 7f8621c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f861c06ee60 con 0x7f861c06f730 2024-04-24T08:20:55.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.439+0000 7f8621c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f861c0d0a30 con 0x7f861c072480 2024-04-24T08:20:55.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f861c0b3320 0x7f861c0c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f861c0b3320 0x7f861c0c96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40762/0 (socket says 172.21.15.69:40762) 2024-04-24T08:20:55.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861b400640 1 -- 172.21.15.69:0/2549652329 learned_addr learned my addr 172.21.15.69:0/2549652329 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:55.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861aa00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f861c06f730 0x7f861c13a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861a000640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c072480 0x7f861c15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861b400640 1 -- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c072480 msgr2=0x7f861c15b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861b400640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c072480 0x7f861c15b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861b400640 1 -- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f861c06f730 msgr2=0x7f861c13a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861b400640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f861c06f730 0x7f861c13a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861b400640 1 -- 172.21.15.69:0/2549652329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861c15d1b0 con 0x7f861c0b3320 2024-04-24T08:20:55.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861aa00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f861c06f730 0x7f861c13a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:55.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861a000640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c072480 0x7f861c15b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:55.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.440+0000 7f861b400640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f861c0b3320 0x7f861c0c96a0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f860c00e960 tx=0x7f860c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:55.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.441+0000 7f8612a00640 1 -- 172.21.15.69:0/2549652329 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c017070 con 0x7f861c0b3320 2024-04-24T08:20:55.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.441+0000 7f8612a00640 1 -- 172.21.15.69:0/2549652329 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f860c004020 con 0x7f861c0b3320 2024-04-24T08:20:55.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.441+0000 7f8612a00640 1 -- 172.21.15.69:0/2549652329 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c0043f0 con 0x7f861c0b3320 2024-04-24T08:20:55.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.441+0000 7f8621c00640 1 -- 172.21.15.69:0/2549652329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861c15d490 con 0x7f861c0b3320 2024-04-24T08:20:55.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.441+0000 7f8621c00640 1 -- 172.21.15.69:0/2549652329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861c15da50 con 0x7f861c0b3320 2024-04-24T08:20:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.442+0000 7f8621c00640 1 -- 172.21.15.69:0/2549652329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f861c068a80 con 0x7f861c0b3320 2024-04-24T08:20:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.443+0000 7f8612a00640 1 -- 172.21.15.69:0/2549652329 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f860c00ca00 con 0x7f861c0b3320 2024-04-24T08:20:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.444+0000 7f8612a00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85f806c8f0 0x7f85f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.444+0000 7f8612a00640 1 -- 172.21.15.69:0/2549652329 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f860c08bb00 con 0x7f861c0b3320 2024-04-24T08:20:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.445+0000 7f861aa00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85f806c8f0 0x7f85f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.446+0000 7f861aa00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85f806c8f0 0x7f85f806edb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f8614034000 tx=0x7f861402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:55.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.449+0000 7f8612a00640 1 -- 172.21.15.69:0/2549652329 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f860c059e30 con 0x7f861c0b3320 2024-04-24T08:20:55.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.692+0000 7f8621c00640 1 -- 172.21.15.69:0/2549652329 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f861c073020 con 0x7f861c0b3320 2024-04-24T08:20:55.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.694+0000 7f8612a00640 1 -- 172.21.15.69:0/2549652329 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5193 (secure 0 0 0) 0x7f860c059730 con 0x7f861c0b3320 2024-04-24T08:20:55.694 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:20:55.694 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:20:54.481212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9285":{"gid":9285,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":15,"addr":"172.21.15.69:6835/2194021008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2194021008},{"type":"v1","addr":"172.21.15.69:6835","nonce":2194021008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9285,"qdb_cluster":[9285]},"id":8}]} 2024-04-24T08:20:55.695 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 105 2024-04-24T08:20:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.696+0000 7f8621c00640 1 -- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85f806c8f0 msgr2=0x7f85f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.696+0000 7f8621c00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85f806c8f0 0x7f85f806edb0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f8614034000 tx=0x7f861402e000 comp rx=0 tx=0).stop 2024-04-24T08:20:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.696+0000 7f8621c00640 1 -- 172.21.15.69:0/2549652329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f861c0b3320 msgr2=0x7f861c0c96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.696+0000 7f8621c00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f861c0b3320 0x7f861c0c96a0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f860c00e960 tx=0x7f860c00ee20 comp rx=0 tx=0).stop 2024-04-24T08:20:55.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.696+0000 7f8621c00640 1 -- 172.21.15.69:0/2549652329 shutdown_connections 2024-04-24T08:20:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.696+0000 7f8621c00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f85f806c8f0 0x7f85f806edb0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.696+0000 7f8621c00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f861c0b3320 0x7f861c0c96a0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.696+0000 7f8621c00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f861c072480 0x7f861c15b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.696+0000 7f8621c00640 1 --2- 172.21.15.69:0/2549652329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f861c06f730 0x7f861c13a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.696+0000 7f8621c00640 1 -- 172.21.15.69:0/2549652329 >> 172.21.15.69:0/2549652329 conn(0x7f861c067d90 msgr2=0x7f861c0cdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.697+0000 7f8621c00640 1 -- 172.21.15.69:0/2549652329 shutdown_connections 2024-04-24T08:20:55.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:55.697+0000 7f8621c00640 1 -- 172.21.15.69:0/2549652329 wait complete. 2024-04-24T08:20:55.714 INFO:tasks.ceph.mds.c.smithi069.stderr:2024-04-24T08:20:55.714+0000 7f56b2200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:20:56.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:20:09.204175+0000', 'modified': '2024-04-24T08:20:54.481212+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 110, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9285': {'gid': 9285, 'name': 'c', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.69:6835/2194021008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 2194021008}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 2194021008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9285, 'qdb_cluster': [9285]} 2024-04-24T08:20:56.119 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:20:56.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-24T08:20:56.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.329+0000 7f13f3200640 1 Processor -- start 2024-04-24T08:20:56.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.330+0000 7f13f3200640 1 -- start start 2024-04-24T08:20:56.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.331+0000 7f13f3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406cfb0 0x7f13f40cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.331+0000 7f13f3200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f13f406d410 0x7f13f40d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.331+0000 7f13f3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13f415ed40 con 0x7f13f415e960 2024-04-24T08:20:56.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.331+0000 7f13f3200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f13f40cf050 con 0x7f13f406d410 2024-04-24T08:20:56.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.331+0000 7f13f3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f13f4162b50 con 0x7f13f406cfb0 2024-04-24T08:20:56.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.331+0000 7f13f1e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f13f415e960 0x7f13f4162440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:46046/0 (socket says 172.21.15.69:46046) 2024-04-24T08:20:56.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.331+0000 7f13f1e00640 1 -- 172.21.15.69:0/1967857866 learned_addr learned my addr 172.21.15.69:0/1967857866 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:20:56.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.331+0000 7f13f2800640 1 --2- 172.21.15.69:0/1967857866 >> v2:172.21.15.69:3300/0 conn(0x7f13f406d410 0x7f13f40d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.332+0000 7f13f1400640 1 --2- 172.21.15.69:0/1967857866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406cfb0 0x7f13f40cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.332+0000 7f13f0a00640 1 -- 172.21.15.69:0/1967857866 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2564980014 0 0) 0x7f13f415ed40 con 0x7f13f415e960 2024-04-24T08:20:56.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.332+0000 7f13f2800640 1 -- 172.21.15.69:0/1967857866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406cfb0 msgr2=0x7f13f40cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.332+0000 7f13f2800640 1 --2- 172.21.15.69:0/1967857866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406cfb0 0x7f13f40cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.332+0000 7f13f2800640 1 -- 172.21.15.69:0/1967857866 >> v1:172.21.15.69:6789/0 conn(0x7f13f415e960 legacy=0x7f13f4162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.332+0000 7f13f2800640 1 -- 172.21.15.69:0/1967857866 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f4162e20 con 0x7f13f406d410 2024-04-24T08:20:56.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.333+0000 7f13f2800640 1 --2- 172.21.15.69:0/1967857866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f406d410 0x7f13f40d0030 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0097a0 tx=0x7f13ec02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159a487f31b28b2b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.333+0000 7f13f0a00640 1 -- 172.21.15.69:0/1967857866 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec037070 con 0x7f13f406d410 2024-04-24T08:20:56.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.333+0000 7f13f0a00640 1 -- 172.21.15.69:0/1967857866 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13ec032440 con 0x7f13f406d410 2024-04-24T08:20:56.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.333+0000 7f13f0a00640 1 -- 172.21.15.69:0/1967857866 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec032810 con 0x7f13f406d410 2024-04-24T08:20:56.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.334+0000 7f13f3200640 1 -- 172.21.15.69:0/1967857866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f406d410 msgr2=0x7f13f40d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.334+0000 7f13f3200640 1 --2- 172.21.15.69:0/1967857866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f406d410 0x7f13f40d0030 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0097a0 tx=0x7f13ec02a180 comp rx=0 tx=0).stop 2024-04-24T08:20:56.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.334+0000 7f13f3200640 1 -- 172.21.15.69:0/1967857866 shutdown_connections 2024-04-24T08:20:56.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.334+0000 7f13f3200640 1 --2- 172.21.15.69:0/1967857866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f406d410 0x7f13f40d0030 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.334+0000 7f13f3200640 1 --2- 172.21.15.69:0/1967857866 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406cfb0 0x7f13f40cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.334+0000 7f13f3200640 1 -- 172.21.15.69:0/1967857866 >> 172.21.15.69:0/1967857866 conn(0x7f13f409d900 msgr2=0x7f13f408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.334+0000 7f13f3200640 1 -- 172.21.15.69:0/1967857866 shutdown_connections 2024-04-24T08:20:56.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.334+0000 7f13f3200640 1 -- 172.21.15.69:0/1967857866 wait complete. 2024-04-24T08:20:56.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.336+0000 7f13f3200640 1 Processor -- start 2024-04-24T08:20:56.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.336+0000 7f13f3200640 1 -- start start 2024-04-24T08:20:56.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.337+0000 7f13f3200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13f406cfb0 0x7f13f414eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.337+0000 7f13f3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406d410 0x7f13f40c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.337+0000 7f13f3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f415e960 0x7f13f40cd620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.337+0000 7f13f3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f13f40cf5d0 con 0x7f13f415e960 2024-04-24T08:20:56.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.337+0000 7f13f3200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f13f40cf2f0 con 0x7f13f406cfb0 2024-04-24T08:20:56.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.337+0000 7f13f3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f13f40b6170 con 0x7f13f406d410 2024-04-24T08:20:56.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.337+0000 7f13f2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f415e960 0x7f13f40cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.337+0000 7f13f2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f415e960 0x7f13f40cd620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40772/0 (socket says 172.21.15.69:40772) 2024-04-24T08:20:56.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.337+0000 7f13f2800640 1 -- 172.21.15.69:0/2399502022 learned_addr learned my addr 172.21.15.69:0/2399502022 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:56.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f1400640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406d410 0x7f13f40c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f2800640 1 -- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406d410 msgr2=0x7f13f40c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f2800640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406d410 0x7f13f40c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f2800640 1 -- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13f406cfb0 msgr2=0x7f13f414eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f1e00640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13f406cfb0 0x7f13f414eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f2800640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13f406cfb0 0x7f13f414eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f2800640 1 -- 172.21.15.69:0/2399502022 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f40c96d0 con 0x7f13f415e960 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f1400640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406d410 0x7f13f40c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f2800640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f415e960 0x7f13f40cd620 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f13ec035040 tx=0x7f13ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13ea000640 1 -- 172.21.15.69:0/2399502022 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec037040 con 0x7f13f415e960 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13ea000640 1 -- 172.21.15.69:0/2399502022 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13ec042070 con 0x7f13f415e960 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13ea000640 1 -- 172.21.15.69:0/2399502022 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec030b50 con 0x7f13f415e960 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f3200640 1 -- 172.21.15.69:0/2399502022 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f41602e0 con 0x7f13f415e960 2024-04-24T08:20:56.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.338+0000 7f13f3200640 1 -- 172.21.15.69:0/2399502022 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f41608a0 con 0x7f13f415e960 2024-04-24T08:20:56.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.341+0000 7f13ea000640 1 -- 172.21.15.69:0/2399502022 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f13ec030cf0 con 0x7f13f415e960 2024-04-24T08:20:56.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.342+0000 7f13ea000640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f13d006c8f0 0x7f13d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.342+0000 7f13ea000640 1 -- 172.21.15.69:0/2399502022 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f13ec0ac6f0 con 0x7f13f415e960 2024-04-24T08:20:56.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.342+0000 7f13fb7a7740 1 Processor -- start 2024-04-24T08:20:56.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.343+0000 7f13fb7a7740 1 -- start start 2024-04-24T08:20:56.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.343+0000 7f13f1e00640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f13d006c8f0 0x7f13d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.343+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b355d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b35602aa0 0x557b35601e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b35602360 0x557b355d7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557b3553cb10 con 0x557b3561f980 2024-04-24T08:20:56.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557b354e32b0 con 0x557b35602aa0 2024-04-24T08:20:56.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557b35538ad0 con 0x557b35602360 2024-04-24T08:20:56.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13f1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b355d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13f1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b355d2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40784/0 (socket says 172.21.15.69:40784) 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13f1e00640 1 -- 172.21.15.69:0/2316997502 learned_addr learned my addr 172.21.15.69:0/2316997502 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13f1e00640 1 -- 172.21.15.69:0/2316997502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b35602360 msgr2=0x557b355d7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13f1e00640 1 --2- 172.21.15.69:0/2316997502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b35602360 0x557b355d7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13f1e00640 1 -- 172.21.15.69:0/2316997502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b35602aa0 msgr2=0x557b35601e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13f1e00640 1 --2- 172.21.15.69:0/2316997502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b35602aa0 0x557b35601e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13f1e00640 1 -- 172.21.15.69:0/2316997502 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b3524f2d0 con 0x557b3561f980 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.344+0000 7f13f1e00640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f13d006c8f0 0x7f13d006edb0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f13dc000f60 tx=0x7f13dc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.345+0000 7f13f1e00640 1 --2- 172.21.15.69:0/2316997502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b355d2310 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00d840 tx=0x7f13dc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cacdab595e25b4e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.345+0000 7f13e2600640 1 -- 172.21.15.69:0/2316997502 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc023070 con 0x557b3561f980 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.345+0000 7f13e2600640 1 -- 172.21.15.69:0/2316997502 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13dc007580 con 0x557b3561f980 2024-04-24T08:20:56.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.345+0000 7f13e2600640 1 -- 172.21.15.69:0/2316997502 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc014aa0 con 0x557b3561f980 2024-04-24T08:20:56.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.345+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2316997502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 msgr2=0x557b355d2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.345+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/2316997502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b355d2310 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00d840 tx=0x7f13dc018000 comp rx=0 tx=0).stop 2024-04-24T08:20:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.346+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2316997502 shutdown_connections 2024-04-24T08:20:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.346+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/2316997502 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b35602360 0x557b355d7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.346+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/2316997502 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b35602aa0 0x557b35601e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.346+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/2316997502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b355d2310 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.346+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2316997502 >> 172.21.15.69:0/2316997502 conn(0x557b355d4580 msgr2=0x557b3561ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.346+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2316997502 shutdown_connections 2024-04-24T08:20:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.346+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2316997502 wait complete. 2024-04-24T08:20:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.346+0000 7f13fb7a7740 1 -- start start 2024-04-24T08:20:56.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.347+0000 7f13fb7a7740 1 Processor -- start 2024-04-24T08:20:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b35602aa0 0x557b355fd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b35605110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b356207e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557b355d6730 con 0x557b3561f980 2024-04-24T08:20:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557b354f6590 con 0x557b3562dcc0 2024-04-24T08:20:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557b355d5b20 con 0x557b35602aa0 2024-04-24T08:20:56.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13fb7a7740 10 client.0 _ll_register_callbacks cb 0x557b354d93a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b35605110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b35605110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40802/0 (socket says 172.21.15.69:40802) 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f1400640 1 -- 172.21.15.69:0/796710900 learned_addr learned my addr 172.21.15.69:0/796710900 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f1e00640 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b35602aa0 0x557b355fd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f1400640 1 -- 172.21.15.69:0/796710900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b35602aa0 msgr2=0x557b355fd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f1400640 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b35602aa0 0x557b355fd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f1400640 1 -- 172.21.15.69:0/796710900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 msgr2=0x557b356207e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f2800640 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b356207e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f1400640 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b356207e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f1400640 1 -- 172.21.15.69:0/796710900 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b353196b0 con 0x557b3561f980 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.348+0000 7f13f1e00640 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b35602aa0 0x557b355fd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.349+0000 7f13f1400640 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b35605110 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f13e4007a40 tx=0x7f13e4007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d10b6655431208 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.349+0000 7f13d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:20:56.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.349+0000 7f13d6a00640 1 -- 172.21.15.69:0/796710900 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4018020 con 0x557b3561f980 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.349+0000 7f13d6a00640 1 -- 172.21.15.69:0/796710900 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f13e400fc60 con 0x557b3561f980 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.349+0000 7f13fb7a7740 1 -- 172.21.15.69:0/796710900 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13dc007580 con 0x557b3561f980 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.349+0000 7f13d6a00640 1 -- 172.21.15.69:0/796710900 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f13e4022050 con 0x557b3561f980 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.349+0000 7f13fb7a7740 10 client.9678 fetch_fsmap learned FSMap version 105 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.349+0000 7f13fb7a7740 1 -- 172.21.15.69:0/796710900 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x557b3562d4c0 con 0x557b3561f980 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.349+0000 7f13d6a00640 1 -- 172.21.15.69:0/796710900 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13dc007580 con 0x557b3561f980 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.350+0000 7f13d6a00640 1 -- 172.21.15.69:0/796710900 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f13e4025020 con 0x557b3561f980 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.350+0000 7f13fb7a7740 10 client.9678 fetch_fsmap finished waiting for FSMap version 105 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.350+0000 7f13fb7a7740 10 client.9678 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.350+0000 7f13fb7a7740 20 client.9678 populate_metadata read hostname 'smithi069' 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.350+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x557b356038e0 0x557b3563ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.350+0000 7f13fb7a7740 4 client.9678 mds_command: new command op to 9285 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.350+0000 7f13fb7a7740 1 -- 172.21.15.69:0/796710900 --> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b3554a530 con 0x557b356038e0 2024-04-24T08:20:56.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.350+0000 7f13f1e00640 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x557b356038e0 0x557b3563ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.351+0000 7f13f1e00640 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x557b356038e0 0x557b3563ba80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21448588ae241861 server_cookie=bb6b0b2873a9c0a4 in_seq=0 out_seq=0 2024-04-24T08:20:56.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.351+0000 7f13d6a00640 10 client.9678 ms_handle_connect on v2:172.21.15.69:6834/2194021008 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.351+0000 7f13d6a00640 1 -- 172.21.15.69:0/796710900 <== mds.0 v2:172.21.15.69:6834/2194021008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557b3554a530 con 0x557b356038e0 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.351+0000 7f13d6a00640 10 client.9678 handle_command_reply: tid=0 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 client.9678 shutdown 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 -- 172.21.15.69:0/796710900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 msgr2=0x557b35605110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b35605110 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f13e4007a40 tx=0x7f13e4007f00 comp rx=0 tx=0).stop 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 -- 172.21.15.69:0/796710900 shutdown_connections 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x557b356038e0 0x557b3563ba80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b3561f980 0x557b35605110 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b356207e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13d6a00640 0 client.9678 ms_handle_reset on v2:172.21.15.69:6834/2194021008 2024-04-24T08:20:56.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/796710900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b35602aa0 0x557b355fd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 -- 172.21.15.69:0/796710900 >> 172.21.15.69:0/796710900 conn(0x557b35602360 msgr2=0x557b3561f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 -- 172.21.15.69:0/796710900 shutdown_connections 2024-04-24T08:20:56.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 1 -- 172.21.15.69:0/796710900 wait complete. 2024-04-24T08:20:56.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.352+0000 7f13fb7a7740 20 client.9678 trim_cache size 0 max 16384 2024-04-24T08:20:56.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.372+0000 7f13fb7a7740 1 Processor -- start 2024-04-24T08:20:56.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.372+0000 7f13fb7a7740 1 -- start start 2024-04-24T08:20:56.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.373+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b3562cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.373+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b3561f980 0x557b3561bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.373+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b3562b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.373+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f13f41ffc70 con 0x557b35602aa0 2024-04-24T08:20:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.373+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f13f415f110 con 0x557b3562dcc0 2024-04-24T08:20:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.373+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f13f415f760 con 0x557b3561f980 2024-04-24T08:20:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.373+0000 7f13f1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b3562cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.373+0000 7f13f1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b3562cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40810/0 (socket says 172.21.15.69:40810) 2024-04-24T08:20:56.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.373+0000 7f13f1e00640 1 -- 172.21.15.69:0/1529696386 learned_addr learned my addr 172.21.15.69:0/1529696386 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.373+0000 7f13f1400640 1 --2- 172.21.15.69:0/1529696386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b3561f980 0x557b3561bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13f1e00640 1 -- 172.21.15.69:0/1529696386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b3561f980 msgr2=0x557b3561bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13f1e00640 1 --2- 172.21.15.69:0/1529696386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b3561f980 0x557b3561bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13f1e00640 1 -- 172.21.15.69:0/1529696386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 msgr2=0x557b3562b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13f2800640 1 --2- 172.21.15.69:0/1529696386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b3562b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13f1e00640 1 --2- 172.21.15.69:0/1529696386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b3562b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13f1e00640 1 -- 172.21.15.69:0/1529696386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b355d9100 con 0x557b35602aa0 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13f1400640 1 --2- 172.21.15.69:0/1529696386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b3561f980 0x557b3561bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13f1e00640 1 --2- 172.21.15.69:0/1529696386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b3562cba0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f13dc007db0 tx=0x7f13dc01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c12fb4ad417355 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13e1c00640 1 -- 172.21.15.69:0/1529696386 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc023040 con 0x557b35602aa0 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13e1c00640 1 -- 172.21.15.69:0/1529696386 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13dc014800 con 0x557b35602aa0 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.374+0000 7f13e1c00640 1 -- 172.21.15.69:0/1529696386 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc014bb0 con 0x557b35602aa0 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.375+0000 7f13fb7a7740 1 -- 172.21.15.69:0/1529696386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 msgr2=0x557b3562cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.375+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/1529696386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b3562cba0 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f13dc007db0 tx=0x7f13dc01c000 comp rx=0 tx=0).stop 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.375+0000 7f13fb7a7740 1 -- 172.21.15.69:0/1529696386 shutdown_connections 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.375+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/1529696386 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b3562b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.375+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/1529696386 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b3561f980 0x557b3561bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.375+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/1529696386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b3562cba0 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.375+0000 7f13fb7a7740 1 -- 172.21.15.69:0/1529696386 >> 172.21.15.69:0/1529696386 conn(0x557b35602360 msgr2=0x557b355fbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.375+0000 7f13fb7a7740 1 -- 172.21.15.69:0/1529696386 shutdown_connections 2024-04-24T08:20:56.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.375+0000 7f13fb7a7740 1 -- 172.21.15.69:0/1529696386 wait complete. 2024-04-24T08:20:56.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.376+0000 7f13fb7a7740 1 -- start start 2024-04-24T08:20:56.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.376+0000 7f13fb7a7740 1 Processor -- start 2024-04-24T08:20:56.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b35600e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b3561f980 0x557b3561ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13fb7a7740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b35697140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557b356311e0 con 0x557b35602aa0 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557b3561c620 con 0x557b3562dcc0 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13fb7a7740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557b35630e70 con 0x557b3561f980 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13fb7a7740 10 client.0 _ll_register_callbacks cb 0x7f13e40075d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13f1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b35600e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13f1e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b35600e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40822/0 (socket says 172.21.15.69:40822) 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13f1e00640 1 -- 172.21.15.69:0/2871812596 learned_addr learned my addr 172.21.15.69:0/2871812596 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13f1e00640 1 -- 172.21.15.69:0/2871812596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b3561f980 msgr2=0x557b3561ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13f1e00640 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b3561f980 0x557b3561ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13f1e00640 1 -- 172.21.15.69:0/2871812596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 msgr2=0x557b35697140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13f1e00640 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b35697140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.377+0000 7f13f1e00640 1 -- 172.21.15.69:0/2871812596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b35600290 con 0x557b35602aa0 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.378+0000 7f13f1e00640 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b35600e50 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f13dc008fd0 tx=0x7f13dc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee2f9169318480 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.378+0000 7f13d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.378+0000 7f13d6a00640 1 -- 172.21.15.69:0/2871812596 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc023070 con 0x557b35602aa0 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.378+0000 7f13d6a00640 1 -- 172.21.15.69:0/2871812596 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f13dc014bb0 con 0x557b35602aa0 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.378+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2871812596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13dc014800 con 0x557b35602aa0 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.378+0000 7f13d6a00640 1 -- 172.21.15.69:0/2871812596 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f13dc007640 con 0x557b35602aa0 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.378+0000 7f13fb7a7740 10 client.9684 fetch_fsmap learned FSMap version 105 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.378+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2871812596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x557b3561d350 con 0x557b35602aa0 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.379+0000 7f13d6a00640 1 -- 172.21.15.69:0/2871812596 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13dc014800 con 0x557b35602aa0 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.379+0000 7f13d6a00640 1 -- 172.21.15.69:0/2871812596 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f13dc03e020 con 0x557b35602aa0 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.379+0000 7f13fb7a7740 10 client.9684 fetch_fsmap finished waiting for FSMap version 105 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.379+0000 7f13fb7a7740 10 client.9684 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:20:56.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.379+0000 7f13fb7a7740 20 client.9684 populate_metadata read hostname 'smithi069' 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.379+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x557b35640fd0 0x557b356413b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.379+0000 7f13fb7a7740 4 client.9684 mds_command: new command op to 9285 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.379+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2871812596 --> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] -- command(tid 0: {"prefix": "status"}) -- 0x557b355443a0 con 0x557b35640fd0 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.379+0000 7f13f1400640 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x557b35640fd0 0x557b356413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13f1400640 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x557b35640fd0 0x557b356413b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c831c869d7f548d5 server_cookie=c3ab17341c57aa3a in_seq=0 out_seq=0 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13d6a00640 10 client.9684 ms_handle_connect on v2:172.21.15.69:6834/2194021008 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13d6a00640 1 -- 172.21.15.69:0/2871812596 <== mds.0 v2:172.21.15.69:6834/2194021008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x557b355443a0 con 0x557b35640fd0 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13d6a00640 10 client.9684 handle_command_reply: tid=0 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 client.9684 shutdown 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2871812596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 msgr2=0x557b35600e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b35600e50 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f13dc008fd0 tx=0x7f13dc000ed0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2871812596 shutdown_connections 2024-04-24T08:20:56.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x557b35640fd0 0x557b356413b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557b3562dcc0 0x557b35697140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557b3561f980 0x557b3561ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 --2- 172.21.15.69:0/2871812596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557b35602aa0 0x557b35600e50 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13d6a00640 0 client.9684 ms_handle_reset on v2:172.21.15.69:6834/2194021008 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2871812596 >> 172.21.15.69:0/2871812596 conn(0x557b35602360 msgr2=0x557b355d10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2871812596 shutdown_connections 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 1 -- 172.21.15.69:0/2871812596 wait complete. 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.380+0000 7f13fb7a7740 20 client.9684 trim_cache size 0 max 16384 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:20:56.385 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:20:56.386 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 9285, 2024-04-24T08:20:56.386 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:20:56.386 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:20:56.386 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:20:56.386 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 4.9174665790000001, 2024-04-24T08:20:56.386 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 105, 2024-04-24T08:20:56.386 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 110, 2024-04-24T08:20:56.386 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 110, 2024-04-24T08:20:56.386 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 52.085123430000003 2024-04-24T08:20:56.386 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:20:56.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 -- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f13d006c8f0 msgr2=0x7f13d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f13d006c8f0 0x7f13d006edb0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f13dc000f60 tx=0x7f13dc009040 comp rx=0 tx=0).stop 2024-04-24T08:20:56.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 -- 172.21.15.69:0/2399502022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f415e960 msgr2=0x7f13f40cd620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f415e960 0x7f13f40cd620 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f13ec035040 tx=0x7f13ec0046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 -- 172.21.15.69:0/2399502022 shutdown_connections 2024-04-24T08:20:56.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f13d006c8f0 0x7f13d006edb0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f13f415e960 0x7f13f40cd620 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f13f406d410 0x7f13f40c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 --2- 172.21.15.69:0/2399502022 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13f406cfb0 0x7f13f414eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 -- 172.21.15.69:0/2399502022 >> 172.21.15.69:0/2399502022 conn(0x7f13f409d900 msgr2=0x7f13f40b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 -- 172.21.15.69:0/2399502022 shutdown_connections 2024-04-24T08:20:56.388 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.381+0000 7f13f3200640 1 -- 172.21.15.69:0/2399502022 wait complete. 2024-04-24T08:20:56.399 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:20:56.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.592+0000 7f924be00640 1 Processor -- start 2024-04-24T08:20:56.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.592+0000 7f924be00640 1 -- start start 2024-04-24T08:20:56.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.593+0000 7f924be00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f924c156c60 0x7f924c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.594+0000 7f924be00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f924c15d660 con 0x7f924c0b30f0 2024-04-24T08:20:56.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.594+0000 7f924be00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f924c06dc30 con 0x7f924c0ad7e0 2024-04-24T08:20:56.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.594+0000 7f924be00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f924c06df10 con 0x7f924c156c60 2024-04-24T08:20:56.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.594+0000 7f924aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f924c156c60 0x7f924c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.594+0000 7f924aa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f924c156c60 0x7f924c15cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40826/0 (socket says 172.21.15.69:40826) 2024-04-24T08:20:56.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.594+0000 7f924aa00640 1 -- 172.21.15.69:0/2547980553 learned_addr learned my addr 172.21.15.69:0/2547980553 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:56.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.595+0000 7f924aa00640 1 -- 172.21.15.69:0/2547980553 >> v1:172.21.15.121:6789/0 conn(0x7f924c0ad7e0 legacy=0x7f924c06e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.595+0000 7f924aa00640 1 -- 172.21.15.69:0/2547980553 >> v1:172.21.15.69:6789/0 conn(0x7f924c0b30f0 legacy=0x7f924c0ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:56.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.595+0000 7f924aa00640 1 -- 172.21.15.69:0/2547980553 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c06e1f0 con 0x7f924c156c60 2024-04-24T08:20:56.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.596+0000 7f924aa00640 1 --2- 172.21.15.69:0/2547980553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 0x7f924c15cac0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f923c009870 tx=0x7f923c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72b4e2a9b8720dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.597+0000 7f924a000640 1 -- 172.21.15.69:0/2547980553 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c037070 con 0x7f924c156c60 2024-04-24T08:20:56.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.597+0000 7f924a000640 1 -- 172.21.15.69:0/2547980553 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f923c032440 con 0x7f924c156c60 2024-04-24T08:20:56.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.597+0000 7f924a000640 1 -- 172.21.15.69:0/2547980553 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c032810 con 0x7f924c156c60 2024-04-24T08:20:56.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.597+0000 7f924be00640 1 -- 172.21.15.69:0/2547980553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 msgr2=0x7f924c15cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.597+0000 7f924be00640 1 --2- 172.21.15.69:0/2547980553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 0x7f924c15cac0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f923c009870 tx=0x7f923c02a2a0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.597+0000 7f924be00640 1 -- 172.21.15.69:0/2547980553 shutdown_connections 2024-04-24T08:20:56.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.597+0000 7f924be00640 1 --2- 172.21.15.69:0/2547980553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 0x7f924c15cac0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.597+0000 7f924be00640 1 -- 172.21.15.69:0/2547980553 >> 172.21.15.69:0/2547980553 conn(0x7f924c0d0160 msgr2=0x7f924c0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.597+0000 7f924be00640 1 -- 172.21.15.69:0/2547980553 shutdown_connections 2024-04-24T08:20:56.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.597+0000 7f924be00640 1 -- 172.21.15.69:0/2547980553 wait complete. 2024-04-24T08:20:56.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.599+0000 7f924be00640 1 Processor -- start 2024-04-24T08:20:56.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.599+0000 7f924be00640 1 -- start start 2024-04-24T08:20:56.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.600+0000 7f924be00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f924c0ad7e0 0x7f924c0998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.600+0000 7f924be00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f924c0b30f0 0x7f924c0b0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.600+0000 7f924be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 0x7f924c0b1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.600+0000 7f924be00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f924c15d000 con 0x7f924c156c60 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.600+0000 7f924be00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f924c15d940 con 0x7f924c0b30f0 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.600+0000 7f924be00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f924c15d180 con 0x7f924c0ad7e0 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.600+0000 7f924b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 0x7f924c0b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.600+0000 7f924b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 0x7f924c0b1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40830/0 (socket says 172.21.15.69:40830) 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.600+0000 7f924b400640 1 -- 172.21.15.69:0/1595580141 learned_addr learned my addr 172.21.15.69:0/1595580141 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.601+0000 7f9242000640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f924c0b30f0 0x7f924c0b0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.601+0000 7f924b400640 1 -- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f924c0ad7e0 msgr2=0x7f924c0998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.601+0000 7f924aa00640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f924c0ad7e0 0x7f924c0998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.601+0000 7f924b400640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f924c0ad7e0 0x7f924c0998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.601+0000 7f924b400640 1 -- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f924c0b30f0 msgr2=0x7f924c0b0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.601+0000 7f924b400640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f924c0b30f0 0x7f924c0b0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.601+0000 7f924b400640 1 -- 172.21.15.69:0/1595580141 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c159230 con 0x7f924c156c60 2024-04-24T08:20:56.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.601+0000 7f924b400640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 0x7f924c0b1420 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f924400c6d0 tx=0x7f924400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.601+0000 7f9243400640 1 -- 172.21.15.69:0/1595580141 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244016020 con 0x7f924c156c60 2024-04-24T08:20:56.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.602+0000 7f9243400640 1 -- 172.21.15.69:0/1595580141 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f924400a840 con 0x7f924c156c60 2024-04-24T08:20:56.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.602+0000 7f9243400640 1 -- 172.21.15.69:0/1595580141 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924400ac70 con 0x7f924c156c60 2024-04-24T08:20:56.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.602+0000 7f924be00640 1 -- 172.21.15.69:0/1595580141 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924c157cb0 con 0x7f924c156c60 2024-04-24T08:20:56.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.602+0000 7f924be00640 1 -- 172.21.15.69:0/1595580141 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924c158270 con 0x7f924c156c60 2024-04-24T08:20:56.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.603+0000 7f924be00640 1 -- 172.21.15.69:0/1595580141 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924c075d50 con 0x7f924c156c60 2024-04-24T08:20:56.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.604+0000 7f9243400640 1 -- 172.21.15.69:0/1595580141 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9244019020 con 0x7f924c156c60 2024-04-24T08:20:56.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.605+0000 7f9243400640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f922806c8f0 0x7f922806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:56.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.605+0000 7f9243400640 1 -- 172.21.15.69:0/1595580141 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f924408bff0 con 0x7f924c156c60 2024-04-24T08:20:56.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.606+0000 7f924aa00640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f922806c8f0 0x7f922806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:56.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.607+0000 7f924aa00640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f922806c8f0 0x7f922806edb0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f923c034000 tx=0x7f923c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:56.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.610+0000 7f9243400640 1 -- 172.21.15.69:0/1595580141 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9244020020 con 0x7f924c156c60 2024-04-24T08:20:56.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.853+0000 7f924be00640 1 -- 172.21.15.69:0/1595580141 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f924c0b3cf0 con 0x7f924c156c60 2024-04-24T08:20:56.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.854+0000 7f9243400640 1 -- 172.21.15.69:0/1595580141 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v105) ==== 53+0+73 (secure 0 0 0) 0x7f924c159230 con 0x7f924c156c60 2024-04-24T08:20:56.854 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:20:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 -- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f922806c8f0 msgr2=0x7f922806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f922806c8f0 0x7f922806edb0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f923c034000 tx=0x7f923c02e000 comp rx=0 tx=0).stop 2024-04-24T08:20:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 -- 172.21.15.69:0/1595580141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 msgr2=0x7f924c0b1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 0x7f924c0b1420 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f924400c6d0 tx=0x7f924400cb90 comp rx=0 tx=0).stop 2024-04-24T08:20:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 -- 172.21.15.69:0/1595580141 shutdown_connections 2024-04-24T08:20:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f922806c8f0 0x7f922806edb0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f924c156c60 0x7f924c0b1420 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f924c0b30f0 0x7f924c0b0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 --2- 172.21.15.69:0/1595580141 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f924c0ad7e0 0x7f924c0998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:56.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 -- 172.21.15.69:0/1595580141 >> 172.21.15.69:0/1595580141 conn(0x7f924c0d0160 msgr2=0x7f924c160480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 -- 172.21.15.69:0/1595580141 shutdown_connections 2024-04-24T08:20:56.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:56.857+0000 7f924be00640 1 -- 172.21.15.69:0/1595580141 wait complete. 2024-04-24T08:20:56.877 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:20:56.877 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:20:56.877 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:20:56.877 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:20:56.877 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:56.877 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:20:56.877 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:20:56.877 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:20:56.877 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:20:56.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:56.878 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:20:56.927 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:20:56.927 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:20:56.927 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:20:56.927 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:56.927 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:20:56.927 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 41923sec preferred_lft 41923sec 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:20:56.928 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:20:56.929 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:20:56.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:56.930 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:20:57.000 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:20:57.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:57.005 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:20:57.058 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:20:57.058 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:20:57.059 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:20:57.059 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:20:57.059 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:20:57.059 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:20:57.059 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:20:57.060 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:20:57.060 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:20:57.060 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:20:57.060 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:20:57.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:20:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:57.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:57.271 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:20:57.322 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:20:57.322 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:20:57.322 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:20:57.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:57.322 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:20:57.402 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:20:57.402 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:20:57.402 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:57.455 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:20:57.455 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:20:57.455 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:57.505 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:20:57.585 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:20:57.641 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:20:57.641 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:20:57.641 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:20:57.641 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:20:57.642 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:20:57.643 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:20:57.644 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:20:57.644 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:20:57.644 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:57.644 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:57.644 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:57.644 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:57.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:57.645 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:20:57.692 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:20:57.692 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:20:57.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:20:57.770 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:20:57.815 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:20:57.816 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:20:57.816 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:20:57.816 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:20:57.816 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:20:57.816 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:20:57.816 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:20:57.817 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:20:57.817 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:20:57.817 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:20:57.817 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:20:57.817 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:20:57.817 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:20:57.817 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:20:57.817 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:20:57.817 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:20:57.818 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:20:57.818 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:20:57.818 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:20:57.818 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:20:57.818 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:20:57.818 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:20:57.818 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:20:57.818 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:20:57.819 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:20:57.819 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:20:57.819 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:20:57.819 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:20:57.819 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:20:57.819 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:20:57.819 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:57.819 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:20:57.820 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:57.820 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:20:57.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:57.821 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:20:57.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:20:57.869+0000 7fed5148d480 -1 init, newargv = 0x560498d156d0 newargc=15 2024-04-24T08:20:57.870 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[77799]: starting ceph client 2024-04-24T08:20:57.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[77799]: starting fuse 2024-04-24T08:20:57.902 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:20:57.902 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:20:57.902 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:20:57.902 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:20:57.902 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:20:57.902 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:57.902 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:20:57.903 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:57.903 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:20:57.903 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:20:57.903 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:20:57.903 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:20:57.903 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:57.904 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:20:57.904 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:20:57.904 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:20:57.904 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:20:57.904 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:57.904 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:20:57.904 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:20:57.905 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:20:57.905 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:20:57.905 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:20:57.905 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:20:57.905 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:20:57.905 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:20:57.905 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:57.905 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:20:57.905 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:58.103 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.77799.asok 2024-04-24T08:20:58.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:58.119 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.77799.asok 2024-04-24T08:20:58.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:58.120 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.77799.asok status 2024-04-24T08:20:58.346 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:20:58.346 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "77799", 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 9699, 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:20:58.347 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 9699 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1864718919 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1864718919 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:20:58.348 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.9699 192.168.0.1:0/1864718919", 2024-04-24T08:20:58.349 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/1864718919", 2024-04-24T08:20:58.349 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:20:58.349 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 105, 2024-04-24T08:20:58.349 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 110, 2024-04-24T08:20:58.349 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:20:58.349 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:20:58.349 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:20:58.349 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:20:58.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:58.361 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:20:58.361 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:58.414 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:20:58.414 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:58.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:58.414 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:20:58.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:58.488 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-04-24T08:20:58.557 INFO:teuthology.orchestra.run.smithi069.stdout:./ 2024-04-24T08:20:58.557 INFO:teuthology.orchestra.run.smithi069.stdout:./parent 2024-04-24T08:20:58.557 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed 2024-04-24T08:20:58.557 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed/charlie 2024-04-24T08:20:58.557 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed/alpha 2024-04-24T08:20:58.557 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/unflushed 2024-04-24T08:20:58.557 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/unflushed/jfile 2024-04-24T08:20:58.559 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:58.559 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:58.559 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:58.559 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:58.559 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-04-24T08:20:58.559 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:58.691 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:20:58.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:58.705 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:58.705 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:58.705 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:58.705 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:58.705 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-04-24T08:20:58.705 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:58.888 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627776 2024-04-24T08:20:58.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:58.903 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:58.903 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:58.903 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:58.903 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:58.903 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-04-24T08:20:58.903 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:59.086 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627777 2024-04-24T08:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:59.099 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:59.100 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:59.100 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:59.100 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:59.100 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-04-24T08:20:59.100 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:59.283 INFO:teuthology.orchestra.run.smithi069.stdout:1099511628280 2024-04-24T08:20:59.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:59.297 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:59.297 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:59.298 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:59.298 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:59.298 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-04-24T08:20:59.298 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:59.481 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627778 2024-04-24T08:20:59.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:59.495 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:59.496 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:59.496 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:59.496 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:59.496 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-04-24T08:20:59.496 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:59.680 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627779 2024-04-24T08:20:59.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:59.695 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:20:59.695 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:20:59.695 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:20:59.695 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:20:59.695 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-04-24T08:20:59.695 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:20:59.874 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627780 2024-04-24T08:20:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:20:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:20:59.888 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:20:59.888 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:20:59.984 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:00.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.171+0000 7fc708000640 1 Processor -- start 2024-04-24T08:21:00.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.172+0000 7fc708000640 1 -- start start 2024-04-24T08:21:00.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.173+0000 7fc708000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc70006f2d0 0x7fc70006f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.173+0000 7fc708000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7000b46e0 con 0x7fc70015b490 2024-04-24T08:21:00.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.173+0000 7fc708000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7000d0960 con 0x7fc7000cfbe0 2024-04-24T08:21:00.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.173+0000 7fc708000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc70006fc70 con 0x7fc70006f2d0 2024-04-24T08:21:00.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.174+0000 7fc705600640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fc70015b490 0x7fc7000b5180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49556/0 (socket says 172.21.15.69:49556) 2024-04-24T08:21:00.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.174+0000 7fc705600640 1 -- 172.21.15.69:0/3211170021 learned_addr learned my addr 172.21.15.69:0/3211170021 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:00.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.174+0000 7fc704c00640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc70006f2d0 0x7fc70006f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.174+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3929883563 0 0) 0x7fc7000b46e0 con 0x7fc70015b490 2024-04-24T08:21:00.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.174+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6e8003520 con 0x7fc70015b490 2024-04-24T08:21:00.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1863457784 0 0) 0x7fc6e8003520 con 0x7fc70015b490 2024-04-24T08:21:00.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc70006f2d0 msgr2=0x7fc70006f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc70006f2d0 0x7fc70006f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 >> v1:172.21.15.121:6789/0 conn(0x7fc7000cfbe0 legacy=0x7fc7000d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7000b4980 con 0x7fc70015b490 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fc6f4001670 con 0x7fc70015b490 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 >> v1:172.21.15.69:6789/0 conn(0x7fc70015b490 legacy=0x7fc7000b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6e80039e0 0x7fc6e8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6e8004800 0x7fc6e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc70006f2d0 0x7fc6e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc70006fc70 con 0x7fc6e8004800 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e800ba70 con 0x7fc6e80039e0 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e800bc70 con 0x7fc70006f2d0 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.175+0000 7fc704c00640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6e8004800 0x7fc6e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.176+0000 7fc705600640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6e80039e0 0x7fc6e8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.176+0000 7fc706000640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc70006f2d0 0x7fc6e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.176+0000 7fc704c00640 1 -- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc70006f2d0 msgr2=0x7fc6e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.176+0000 7fc704c00640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc70006f2d0 0x7fc6e80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.176+0000 7fc704c00640 1 -- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6e80039e0 msgr2=0x7fc6e8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.176+0000 7fc704c00640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6e80039e0 0x7fc6e8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.176+0000 7fc704c00640 1 -- 172.21.15.69:0/3211170021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e8003520 con 0x7fc6e8004800 2024-04-24T08:21:00.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.176+0000 7fc705600640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6e80039e0 0x7fc6e8003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.177+0000 7fc704c00640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6e8004800 0x7fc6e8004c80 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009a90 tx=0x7fc6e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e288ed928d2a5e5a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.177+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec036070 con 0x7fc6e8004800 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.177+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6ec031400 con 0x7fc6e8004800 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.177+0000 7fc6ffe00640 1 -- 172.21.15.69:0/3211170021 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec009690 con 0x7fc6e8004800 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.178+0000 7fc708000640 1 -- 172.21.15.69:0/3211170021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6e8004800 msgr2=0x7fc6e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.178+0000 7fc708000640 1 --2- 172.21.15.69:0/3211170021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6e8004800 0x7fc6e8004c80 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009a90 tx=0x7fc6e8005830 comp rx=0 tx=0).stop 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.178+0000 7fc706000640 1 -- 172.21.15.69:0/3211170021 reap_dead start 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.178+0000 7fc708000640 1 -- 172.21.15.69:0/3211170021 shutdown_connections 2024-04-24T08:21:00.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.178+0000 7fc708000640 1 -- 172.21.15.69:0/3211170021 >> 172.21.15.69:0/3211170021 conn(0x7fc700067d90 msgr2=0x7fc70006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.178+0000 7fc708000640 1 -- 172.21.15.69:0/3211170021 shutdown_connections 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.178+0000 7fc708000640 1 -- 172.21.15.69:0/3211170021 wait complete. 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.180+0000 7fc708000640 1 Processor -- start 2024-04-24T08:21:00.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.180+0000 7fc708000640 1 -- start start 2024-04-24T08:21:00.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc708000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc700071fe0 0x7fc7001e58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc708000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7000723c0 0x7fc7001e4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc708000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7001e7010 0x7fc700155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc708000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc7000d0c40 con 0x7fc7001e7010 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc708000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc7000b5890 con 0x7fc700071fe0 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc708000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc70015b870 con 0x7fc7000723c0 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc706000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7001e7010 0x7fc700155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc706000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7001e7010 0x7fc700155090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34004/0 (socket says 172.21.15.69:34004) 2024-04-24T08:21:00.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc706000640 1 -- 172.21.15.69:0/1685094766 learned_addr learned my addr 172.21.15.69:0/1685094766 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:00.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc706000640 1 -- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7000723c0 msgr2=0x7fc7001e4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.181+0000 7fc706000640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7000723c0 0x7fc7001e4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.182+0000 7fc706000640 1 -- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc700071fe0 msgr2=0x7fc7001e58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.182+0000 7fc706000640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc700071fe0 0x7fc7001e58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.182+0000 7fc706000640 1 -- 172.21.15.69:0/1685094766 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7001e4d70 con 0x7fc7001e7010 2024-04-24T08:21:00.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.182+0000 7fc706000640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7001e7010 0x7fc700155090 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fc700073140 tx=0x7fc6f800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.182+0000 7fc6fd600640 1 -- 172.21.15.69:0/1685094766 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8018070 con 0x7fc7001e7010 2024-04-24T08:21:00.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.182+0000 7fc6fd600640 1 -- 172.21.15.69:0/1685094766 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6f8003b70 con 0x7fc7001e7010 2024-04-24T08:21:00.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.182+0000 7fc6fd600640 1 -- 172.21.15.69:0/1685094766 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6f8005070 con 0x7fc7001e7010 2024-04-24T08:21:00.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.182+0000 7fc708000640 1 -- 172.21.15.69:0/1685094766 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7001e5f70 con 0x7fc7001e7010 2024-04-24T08:21:00.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.182+0000 7fc708000640 1 -- 172.21.15.69:0/1685094766 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7001e6530 con 0x7fc7001e7010 2024-04-24T08:21:00.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.183+0000 7fc708000640 1 -- 172.21.15.69:0/1685094766 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc700068a80 con 0x7fc7001e7010 2024-04-24T08:21:00.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.184+0000 7fc6fd600640 1 -- 172.21.15.69:0/1685094766 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc6f801b070 con 0x7fc7001e7010 2024-04-24T08:21:00.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.185+0000 7fc6fd600640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6e006c920 0x7fc6e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.185+0000 7fc6fd600640 1 -- 172.21.15.69:0/1685094766 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7fc6f808b360 con 0x7fc7001e7010 2024-04-24T08:21:00.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.186+0000 7fc705600640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6e006c920 0x7fc6e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.187+0000 7fc705600640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6e006c920 0x7fc6e006ede0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40017e0 tx=0x7fc6f4009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.190+0000 7fc6fd600640 1 -- 172.21.15.69:0/1685094766 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6f8059690 con 0x7fc7001e7010 2024-04-24T08:21:00.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.392+0000 7fc708000640 1 -- 172.21.15.69:0/1685094766 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc70006cd50 con 0x7fc7001e7010 2024-04-24T08:21:00.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.394+0000 7fc6fd600640 1 -- 172.21.15.69:0/1685094766 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+16734 (secure 0 0 0) 0x7fc6f8058f90 con 0x7fc7001e7010 2024-04-24T08:21:00.395 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:00.395 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":110,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:20:50.436238+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6836/1515102940":"2024-04-25T08:20:50.436176+0000","172.21.15.121:6834/1515102940":"2024-04-25T08:20:50.436176+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.396+0000 7fc708000640 1 -- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6e006c920 msgr2=0x7fc6e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.396+0000 7fc708000640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6e006c920 0x7fc6e006ede0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40017e0 tx=0x7fc6f4009040 comp rx=0 tx=0).stop 2024-04-24T08:21:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.396+0000 7fc708000640 1 -- 172.21.15.69:0/1685094766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7001e7010 msgr2=0x7fc700155090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.396+0000 7fc708000640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7001e7010 0x7fc700155090 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fc700073140 tx=0x7fc6f800dc00 comp rx=0 tx=0).stop 2024-04-24T08:21:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.397+0000 7fc708000640 1 -- 172.21.15.69:0/1685094766 shutdown_connections 2024-04-24T08:21:00.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.397+0000 7fc708000640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6e006c920 0x7fc6e006ede0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.397+0000 7fc708000640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc7001e7010 0x7fc700155090 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.397+0000 7fc708000640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc7000723c0 0x7fc7001e4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.397+0000 7fc708000640 1 --2- 172.21.15.69:0/1685094766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc700071fe0 0x7fc7001e58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.397+0000 7fc708000640 1 -- 172.21.15.69:0/1685094766 >> 172.21.15.69:0/1685094766 conn(0x7fc700067d90 msgr2=0x7fc70015dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.397+0000 7fc708000640 1 -- 172.21.15.69:0/1685094766 shutdown_connections 2024-04-24T08:21:00.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.397+0000 7fc708000640 1 -- 172.21.15.69:0/1685094766 wait complete. 2024-04-24T08:21:00.417 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:20:06.963817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:20:08.190440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:21:00.418 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:21:00.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.622+0000 7f79dbe00640 1 Processor -- start 2024-04-24T08:21:00.623 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.623+0000 7f79dbe00640 1 -- start start 2024-04-24T08:21:00.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.624+0000 7f79dbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79dc15e980 0x7f79dc162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.624+0000 7f79dbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc15ed60 con 0x7f79dc0d0da0 2024-04-24T08:21:00.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.624+0000 7f79dbe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc0d1580 con 0x7f79dc06cfb0 2024-04-24T08:21:00.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.624+0000 7f79dbe00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f79dc0d1860 con 0x7f79dc15e980 2024-04-24T08:21:00.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.624+0000 7f79daa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79dc15e980 0x7f79dc162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.625+0000 7f79d2000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f79dc0d0da0 0x7f79dc0cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49562/0 (socket says 172.21.15.69:49562) 2024-04-24T08:21:00.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.625+0000 7f79d2000640 1 -- 172.21.15.69:0/685546796 learned_addr learned my addr 172.21.15.69:0/685546796 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.625+0000 7f79daa00640 1 -- 172.21.15.69:0/685546796 >> v1:172.21.15.121:6789/0 conn(0x7f79dc06cfb0 legacy=0x7f79dc0d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.625+0000 7f79daa00640 1 -- 172.21.15.69:0/685546796 >> v1:172.21.15.69:6789/0 conn(0x7f79dc0d0da0 legacy=0x7f79dc0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.625+0000 7f79daa00640 1 -- 172.21.15.69:0/685546796 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc0cf450 con 0x7f79dc15e980 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.625+0000 7f79da000640 1 -- 172.21.15.69:0/685546796 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1853804451 0 0) 0x7f79dc15ed60 con 0x7f79dc0d0da0 2024-04-24T08:21:00.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.626+0000 7f79daa00640 1 --2- 172.21.15.69:0/685546796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15e980 0x7f79dc162f90 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0097a0 tx=0x7f79cc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439456a6f1398b8b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.627+0000 7f79da000640 1 -- 172.21.15.69:0/685546796 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc037070 con 0x7f79dc15e980 2024-04-24T08:21:00.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.627+0000 7f79da000640 1 -- 172.21.15.69:0/685546796 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79cc032440 con 0x7f79dc15e980 2024-04-24T08:21:00.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.627+0000 7f79da000640 1 -- 172.21.15.69:0/685546796 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc032810 con 0x7f79dc15e980 2024-04-24T08:21:00.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.627+0000 7f79dbe00640 1 -- 172.21.15.69:0/685546796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15e980 msgr2=0x7f79dc162f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.627+0000 7f79dbe00640 1 --2- 172.21.15.69:0/685546796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15e980 0x7f79dc162f90 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0097a0 tx=0x7f79cc02a270 comp rx=0 tx=0).stop 2024-04-24T08:21:00.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.627+0000 7f79dbe00640 1 -- 172.21.15.69:0/685546796 shutdown_connections 2024-04-24T08:21:00.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.627+0000 7f79dbe00640 1 --2- 172.21.15.69:0/685546796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc15e980 0x7f79dc162f90 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.627+0000 7f79dbe00640 1 -- 172.21.15.69:0/685546796 >> 172.21.15.69:0/685546796 conn(0x7f79dc09d920 msgr2=0x7f79dc08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.627+0000 7f79dbe00640 1 -- 172.21.15.69:0/685546796 shutdown_connections 2024-04-24T08:21:00.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.628+0000 7f79dbe00640 1 -- 172.21.15.69:0/685546796 wait complete. 2024-04-24T08:21:00.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.629+0000 7f79dbe00640 1 Processor -- start 2024-04-24T08:21:00.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.629+0000 7f79dbe00640 1 -- start start 2024-04-24T08:21:00.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.630+0000 7f79dbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc06cfb0 0x7f79dc0c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.630+0000 7f79dbe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79dc0d0da0 0x7f79dc0cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.630+0000 7f79dbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79dc15e980 0x7f79dc0c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.630+0000 7f79dbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc0d11e0 con 0x7f79dc06cfb0 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.630+0000 7f79dbe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc0d0360 con 0x7f79dc0d0da0 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.630+0000 7f79dbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc0d1360 con 0x7f79dc15e980 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.631+0000 7f79daa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc06cfb0 0x7f79dc0c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.631+0000 7f79daa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc06cfb0 0x7f79dc0c8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34028/0 (socket says 172.21.15.69:34028) 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.631+0000 7f79daa00640 1 -- 172.21.15.69:0/2489198018 learned_addr learned my addr 172.21.15.69:0/2489198018 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.631+0000 7f79daa00640 1 -- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79dc15e980 msgr2=0x7f79dc0c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.631+0000 7f79daa00640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79dc15e980 0x7f79dc0c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.631+0000 7f79daa00640 1 -- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79dc0d0da0 msgr2=0x7f79dc0cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.631+0000 7f79daa00640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79dc0d0da0 0x7f79dc0cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.631+0000 7f79daa00640 1 -- 172.21.15.69:0/2489198018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc1f8210 con 0x7f79dc06cfb0 2024-04-24T08:21:00.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.631+0000 7f79daa00640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc06cfb0 0x7f79dc0c8310 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f79cc000c00 tx=0x7f79cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.632+0000 7f79d3400640 1 -- 172.21.15.69:0/2489198018 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc041070 con 0x7f79dc06cfb0 2024-04-24T08:21:00.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.632+0000 7f79d3400640 1 -- 172.21.15.69:0/2489198018 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79cc03c930 con 0x7f79dc06cfb0 2024-04-24T08:21:00.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.632+0000 7f79d3400640 1 -- 172.21.15.69:0/2489198018 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79cc037040 con 0x7f79dc06cfb0 2024-04-24T08:21:00.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.632+0000 7f79dbe00640 1 -- 172.21.15.69:0/2489198018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79dc1f84f0 con 0x7f79dc06cfb0 2024-04-24T08:21:00.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.632+0000 7f79dbe00640 1 -- 172.21.15.69:0/2489198018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79dc1f8ab0 con 0x7f79dc06cfb0 2024-04-24T08:21:00.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.633+0000 7f79dbe00640 1 -- 172.21.15.69:0/2489198018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79dc075d50 con 0x7f79dc06cfb0 2024-04-24T08:21:00.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.634+0000 7f79d3400640 1 -- 172.21.15.69:0/2489198018 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f79cc044070 con 0x7f79dc06cfb0 2024-04-24T08:21:00.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.635+0000 7f79d3400640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79b806c8f0 0x7f79b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.635+0000 7f79d3400640 1 -- 172.21.15.69:0/2489198018 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f79cc0ac900 con 0x7f79dc06cfb0 2024-04-24T08:21:00.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.635+0000 7f79d2000640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79b806c8f0 0x7f79b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.637+0000 7f79d2000640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79b806c8f0 0x7f79b806edb0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f79c4007fd0 tx=0x7f79c4007920 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.640+0000 7f79d3400640 1 -- 172.21.15.69:0/2489198018 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79cc07ad50 con 0x7f79dc06cfb0 2024-04-24T08:21:00.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.838+0000 7f79dbe00640 1 -- 172.21.15.69:0/2489198018 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79dc0c8880 con 0x7f79dc06cfb0 2024-04-24T08:21:00.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.839+0000 7f79d3400640 1 -- 172.21.15.69:0/2489198018 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) ==== 93+0+10 (secure 0 0 0) 0x7f79cc07a650 con 0x7f79dc06cfb0 2024-04-24T08:21:00.840 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 -- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79b806c8f0 msgr2=0x7f79b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79b806c8f0 0x7f79b806edb0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f79c4007fd0 tx=0x7f79c4007920 comp rx=0 tx=0).stop 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 -- 172.21.15.69:0/2489198018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc06cfb0 msgr2=0x7f79dc0c8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc06cfb0 0x7f79dc0c8310 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f79cc000c00 tx=0x7f79cc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 -- 172.21.15.69:0/2489198018 shutdown_connections 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79b806c8f0 0x7f79b806edb0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79dc15e980 0x7f79dc0c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79dc0d0da0 0x7f79dc0cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 --2- 172.21.15.69:0/2489198018 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79dc06cfb0 0x7f79dc0c8310 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 -- 172.21.15.69:0/2489198018 >> 172.21.15.69:0/2489198018 conn(0x7f79dc09d920 msgr2=0x7f79dc0cfc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 -- 172.21.15.69:0/2489198018 shutdown_connections 2024-04-24T08:21:00.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:00.842+0000 7f79dbe00640 1 -- 172.21.15.69:0/2489198018 wait complete. 2024-04-24T08:21:00.862 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:21:01.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.064+0000 7f9a6ac00640 1 Processor -- start 2024-04-24T08:21:01.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.064+0000 7f9a6ac00640 1 -- start start 2024-04-24T08:21:01.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.066+0000 7f9a6ac00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9a6415c820 0x7f9a640b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.066+0000 7f9a6ac00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6415cc00 con 0x7f9a640ce250 2024-04-24T08:21:01.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.066+0000 7f9a6ac00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a6406ed30 con 0x7f9a640b46a0 2024-04-24T08:21:01.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.066+0000 7f9a6ac00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9a640b4e10 con 0x7f9a6415c820 2024-04-24T08:21:01.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.066+0000 7f9a68c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9a640ce250 0x7f9a640ccd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49566/0 (socket says 172.21.15.69:49566) 2024-04-24T08:21:01.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.066+0000 7f9a68c00640 1 -- 172.21.15.69:0/511487616 learned_addr learned my addr 172.21.15.69:0/511487616 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:01.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63e00640 1 --2- 172.21.15.69:0/511487616 >> v2:172.21.15.121:3300/0 conn(0x7f9a6415c820 0x7f9a640b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 415443699 0 0) 0x7f9a6415cc00 con 0x7f9a640ce250 2024-04-24T08:21:01.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a48003520 con 0x7f9a640ce250 2024-04-24T08:21:01.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 668129993 0 0) 0x7f9a48003520 con 0x7f9a640ce250 2024-04-24T08:21:01.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 >> v2:172.21.15.121:3300/0 conn(0x7f9a6415c820 msgr2=0x7f9a640b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 --2- 172.21.15.69:0/511487616 >> v2:172.21.15.121:3300/0 conn(0x7f9a6415c820 0x7f9a640b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 >> v1:172.21.15.121:6789/0 conn(0x7f9a640b46a0 legacy=0x7f9a6406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a640b50f0 con 0x7f9a640ce250 2024-04-24T08:21:01.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f9a5c0019f0 con 0x7f9a640ce250 2024-04-24T08:21:01.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 >> v1:172.21.15.69:6789/0 conn(0x7f9a640ce250 legacy=0x7f9a640ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a48003a00 0x7f9a48003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a48004800 0x7f9a48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a480051c0 0x7f9a4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9a640b4e10 con 0x7f9a48004800 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.067+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4800c0f0 con 0x7f9a480051c0 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.068+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a4800c2f0 con 0x7f9a48003a00 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.068+0000 7f9a5b400640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a48004800 0x7f9a48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.068+0000 7f9a5b400640 1 -- 172.21.15.69:0/511487616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a48003a00 msgr2=0x7f9a48003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.068+0000 7f9a63e00640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a48003a00 0x7f9a48003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.068+0000 7f9a68c00640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a480051c0 0x7f9a4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.068+0000 7f9a5b400640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a48003a00 0x7f9a48003e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.068+0000 7f9a5b400640 1 -- 172.21.15.69:0/511487616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a480051c0 msgr2=0x7f9a4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.068+0000 7f9a5b400640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a480051c0 0x7f9a4800b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.068+0000 7f9a5b400640 1 -- 172.21.15.69:0/511487616 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a48003520 con 0x7f9a48004800 2024-04-24T08:21:01.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.068+0000 7f9a68c00640 1 -- 172.21.15.69:0/511487616 reap_dead start 2024-04-24T08:21:01.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.069+0000 7f9a5b400640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a48004800 0x7f9a48004c80 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c001750 tx=0x7f9a48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e08b23e61bb4b17 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.069+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c001370 con 0x7f9a48004800 2024-04-24T08:21:01.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.069+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a4c0052d0 con 0x7f9a48004800 2024-04-24T08:21:01.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.069+0000 7f9a63400640 1 -- 172.21.15.69:0/511487616 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4c02abc0 con 0x7f9a48004800 2024-04-24T08:21:01.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.070+0000 7f9a6ac00640 1 -- 172.21.15.69:0/511487616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a48004800 msgr2=0x7f9a48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.070+0000 7f9a6ac00640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a48004800 0x7f9a48004c80 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c001750 tx=0x7f9a48005830 comp rx=0 tx=0).stop 2024-04-24T08:21:01.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.070+0000 7f9a6ac00640 1 -- 172.21.15.69:0/511487616 shutdown_connections 2024-04-24T08:21:01.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.070+0000 7f9a6ac00640 1 --2- 172.21.15.69:0/511487616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a48004800 0x7f9a48004c80 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.070+0000 7f9a6ac00640 1 -- 172.21.15.69:0/511487616 >> 172.21.15.69:0/511487616 conn(0x7f9a640d0350 msgr2=0x7f9a640d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.070+0000 7f9a6ac00640 1 -- 172.21.15.69:0/511487616 shutdown_connections 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.070+0000 7f9a6ac00640 1 -- 172.21.15.69:0/511487616 wait complete. 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.072+0000 7f9a6ac00640 1 Processor -- start 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.072+0000 7f9a6ac00640 1 -- start start 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.073+0000 7f9a6ac00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48004800 0x7f9a6415e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.073+0000 7f9a6ac00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a641e61c0 0x7f9a6414dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.073+0000 7f9a6ac00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a641f3340 0x7f9a641f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.073+0000 7f9a6ac00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9a640b4b30 con 0x7f9a641f3340 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.073+0000 7f9a6ac00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6406f2b0 con 0x7f9a48004800 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.073+0000 7f9a6ac00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9a640b3e90 con 0x7f9a641e61c0 2024-04-24T08:21:01.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.073+0000 7f9a68c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a641f3340 0x7f9a641f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.073+0000 7f9a68c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a641f3340 0x7f9a641f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34038/0 (socket says 172.21.15.69:34038) 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.073+0000 7f9a68c00640 1 -- 172.21.15.69:0/1186204693 learned_addr learned my addr 172.21.15.69:0/1186204693 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a68c00640 1 -- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a641e61c0 msgr2=0x7f9a6414dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a68c00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a641e61c0 0x7f9a6414dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a68c00640 1 -- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48004800 msgr2=0x7f9a6415e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a68c00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48004800 0x7f9a6415e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a68c00640 1 -- 172.21.15.69:0/1186204693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a641e6640 con 0x7f9a641f3340 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a68c00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a641f3340 0x7f9a641f37c0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f9a640afd00 tx=0x7f9a5c00b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a60c00640 1 -- 172.21.15.69:0/1186204693 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c00b900 con 0x7f9a641f3340 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a60c00640 1 -- 172.21.15.69:0/1186204693 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a5c00baa0 con 0x7f9a641f3340 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a6ac00640 1 -- 172.21.15.69:0/1186204693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a6415fab0 con 0x7f9a641f3340 2024-04-24T08:21:01.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a60c00640 1 -- 172.21.15.69:0/1186204693 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c00be90 con 0x7f9a641f3340 2024-04-24T08:21:01.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.074+0000 7f9a6ac00640 1 -- 172.21.15.69:0/1186204693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a6415de60 con 0x7f9a641f3340 2024-04-24T08:21:01.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.076+0000 7f9a6ac00640 1 -- 172.21.15.69:0/1186204693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a6415d970 con 0x7f9a641f3340 2024-04-24T08:21:01.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.076+0000 7f9a60c00640 1 -- 172.21.15.69:0/1186204693 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9a5c01b020 con 0x7f9a641f3340 2024-04-24T08:21:01.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.077+0000 7f9a60c00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a4006ca90 0x7f9a4006ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.078+0000 7f9a60c00640 1 -- 172.21.15.69:0/1186204693 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f9a5c08bde0 con 0x7f9a641f3340 2024-04-24T08:21:01.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.078+0000 7f9a63e00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a4006ca90 0x7f9a4006ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.079+0000 7f9a63e00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a4006ca90 0x7f9a4006ef50 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9a64051740 tx=0x7f9a54008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.082+0000 7f9a60c00640 1 -- 172.21.15.69:0/1186204693 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a5c05a110 con 0x7f9a641f3340 2024-04-24T08:21:01.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.282+0000 7f9a6ac00640 1 -- 172.21.15.69:0/1186204693 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9a640cddc0 con 0x7f9a641f3340 2024-04-24T08:21:01.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.283+0000 7f9a60c00640 1 -- 172.21.15.69:0/1186204693 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v110) ==== 104+0+11 (secure 0 0 0) 0x7f9a5c059a10 con 0x7f9a641f3340 2024-04-24T08:21:01.284 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:21:01.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 -- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a4006ca90 msgr2=0x7f9a4006ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a4006ca90 0x7f9a4006ef50 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f9a64051740 tx=0x7f9a54008040 comp rx=0 tx=0).stop 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 -- 172.21.15.69:0/1186204693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a641f3340 msgr2=0x7f9a641f37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a641f3340 0x7f9a641f37c0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f9a640afd00 tx=0x7f9a5c00b0d0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 -- 172.21.15.69:0/1186204693 shutdown_connections 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9a4006ca90 0x7f9a4006ef50 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9a641f3340 0x7f9a641f37c0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9a641e61c0 0x7f9a6414dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 --2- 172.21.15.69:0/1186204693 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a48004800 0x7f9a6415e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 -- 172.21.15.69:0/1186204693 >> 172.21.15.69:0/1186204693 conn(0x7f9a640d0350 msgr2=0x7f9a640afdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 -- 172.21.15.69:0/1186204693 shutdown_connections 2024-04-24T08:21:01.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.286+0000 7f9a6ac00640 1 -- 172.21.15.69:0/1186204693 wait complete. 2024-04-24T08:21:01.305 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:21:01.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.507+0000 7f09b1400640 1 Processor -- start 2024-04-24T08:21:01.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.508+0000 7f09b1400640 1 -- start start 2024-04-24T08:21:01.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.509+0000 7f09b1400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09ac0b3e90 0x7f09ac0b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.509+0000 7f09b1400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f09ac0ce250 0x7f09ac0ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.509+0000 7f09b1400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09ac15cc00 con 0x7f09ac15c820 2024-04-24T08:21:01.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.509+0000 7f09b1400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f09ac0b4be0 con 0x7f09ac0ce250 2024-04-24T08:21:01.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.509+0000 7f09b1400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09ac0b42f0 con 0x7f09ac0b3e90 2024-04-24T08:21:01.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.509+0000 7f09aa800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f09ac15c820 0x7f09ac06f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49576/0 (socket says 172.21.15.69:49576) 2024-04-24T08:21:01.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.509+0000 7f09aa800640 1 -- 172.21.15.69:0/673750150 learned_addr learned my addr 172.21.15.69:0/673750150 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:01.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.509+0000 7f09ab200640 1 --2- 172.21.15.69:0/673750150 >> v2:172.21.15.69:3300/0 conn(0x7f09ac0ce250 0x7f09ac0ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.510+0000 7f09a9e00640 1 --2- 172.21.15.69:0/673750150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09ac0b3e90 0x7f09ac0b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.510+0000 7f09ab200640 1 -- 172.21.15.69:0/673750150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09ac0b3e90 msgr2=0x7f09ac0b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.510+0000 7f09ab200640 1 --2- 172.21.15.69:0/673750150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09ac0b3e90 0x7f09ac0b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.510+0000 7f09ab200640 1 -- 172.21.15.69:0/673750150 >> v1:172.21.15.69:6789/0 conn(0x7f09ac15c820 legacy=0x7f09ac06f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.510+0000 7f09ab200640 1 -- 172.21.15.69:0/673750150 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac06ed30 con 0x7f09ac0ce250 2024-04-24T08:21:01.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.510+0000 7f09a9400640 1 -- 172.21.15.69:0/673750150 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1955798895 0 0) 0x7f09ac15cc00 con 0x7f09ac15c820 2024-04-24T08:21:01.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.511+0000 7f09ab200640 1 --2- 172.21.15.69:0/673750150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 0x7f09ac0ce6e0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f09a40097a0 tx=0x7f09a402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0b449a0b5e67f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.511+0000 7f09a9400640 1 -- 172.21.15.69:0/673750150 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4037070 con 0x7f09ac0ce250 2024-04-24T08:21:01.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.511+0000 7f09a9400640 1 -- 172.21.15.69:0/673750150 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a4032440 con 0x7f09ac0ce250 2024-04-24T08:21:01.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.511+0000 7f09a9400640 1 -- 172.21.15.69:0/673750150 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4032810 con 0x7f09ac0ce250 2024-04-24T08:21:01.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.512+0000 7f09b1400640 1 -- 172.21.15.69:0/673750150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 msgr2=0x7f09ac0ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.512+0000 7f09b1400640 1 --2- 172.21.15.69:0/673750150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 0x7f09ac0ce6e0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f09a40097a0 tx=0x7f09a402a180 comp rx=0 tx=0).stop 2024-04-24T08:21:01.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.512+0000 7f09b1400640 1 -- 172.21.15.69:0/673750150 shutdown_connections 2024-04-24T08:21:01.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.512+0000 7f09b1400640 1 --2- 172.21.15.69:0/673750150 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 0x7f09ac0ce6e0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.512+0000 7f09b1400640 1 --2- 172.21.15.69:0/673750150 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09ac0b3e90 0x7f09ac0b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.512+0000 7f09b1400640 1 -- 172.21.15.69:0/673750150 >> 172.21.15.69:0/673750150 conn(0x7f09ac0d0350 msgr2=0x7f09ac0d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.512+0000 7f09b1400640 1 -- 172.21.15.69:0/673750150 shutdown_connections 2024-04-24T08:21:01.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.512+0000 7f09b1400640 1 -- 172.21.15.69:0/673750150 wait complete. 2024-04-24T08:21:01.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.514+0000 7f09b1400640 1 Processor -- start 2024-04-24T08:21:01.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.514+0000 7f09b1400640 1 -- start start 2024-04-24T08:21:01.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.515+0000 7f09b1400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09ac0b3e90 0x7f09ac146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.515+0000 7f09b1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 0x7f09ac073460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.515+0000 7f09b1400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09ac15c820 0x7f09ac0b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.515+0000 7f09b1400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac06efd0 con 0x7f09ac0ce250 2024-04-24T08:21:01.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.515+0000 7f09b1400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac0b50f0 con 0x7f09ac15c820 2024-04-24T08:21:01.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.515+0000 7f09b1400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f09ac06f2b0 con 0x7f09ac0b3e90 2024-04-24T08:21:01.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.515+0000 7f09a9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 0x7f09ac073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.515+0000 7f09a9e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 0x7f09ac073460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34048/0 (socket says 172.21.15.69:34048) 2024-04-24T08:21:01.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.515+0000 7f09a9e00640 1 -- 172.21.15.69:0/2567446711 learned_addr learned my addr 172.21.15.69:0/2567446711 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:01.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.516+0000 7f09a9e00640 1 -- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09ac0b3e90 msgr2=0x7f09ac146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.516+0000 7f09a9e00640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09ac0b3e90 0x7f09ac146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.516+0000 7f09a9e00640 1 -- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09ac15c820 msgr2=0x7f09ac0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.516+0000 7f09a9e00640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09ac15c820 0x7f09ac0b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.516+0000 7f09a9e00640 1 -- 172.21.15.69:0/2567446711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac0cb970 con 0x7f09ac0ce250 2024-04-24T08:21:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.516+0000 7f09a9e00640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 0x7f09ac073460 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f0994007ef0 tx=0x7f099400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.516+0000 7f09a2a00640 1 -- 172.21.15.69:0/2567446711 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994014620 con 0x7f09ac0ce250 2024-04-24T08:21:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.516+0000 7f09a2a00640 1 -- 172.21.15.69:0/2567446711 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0994019070 con 0x7f09ac0ce250 2024-04-24T08:21:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.516+0000 7f09a2a00640 1 -- 172.21.15.69:0/2567446711 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0994014a00 con 0x7f09ac0ce250 2024-04-24T08:21:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.516+0000 7f09b1400640 1 -- 172.21.15.69:0/2567446711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09ac0af450 con 0x7f09ac0ce250 2024-04-24T08:21:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.517+0000 7f09b1400640 1 -- 172.21.15.69:0/2567446711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09ac0aef00 con 0x7f09ac0ce250 2024-04-24T08:21:01.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.518+0000 7f09b1400640 1 -- 172.21.15.69:0/2567446711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09ac068a80 con 0x7f09ac0ce250 2024-04-24T08:21:01.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.519+0000 7f09a2a00640 1 -- 172.21.15.69:0/2567446711 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0994014ba0 con 0x7f09ac0ce250 2024-04-24T08:21:01.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.519+0000 7f09a2a00640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098806c8f0 0x7f098806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.520+0000 7f09a2a00640 1 -- 172.21.15.69:0/2567446711 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f099408c530 con 0x7f09ac0ce250 2024-04-24T08:21:01.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.520+0000 7f09aa800640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098806c8f0 0x7f098806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.521+0000 7f09aa800640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098806c8f0 0x7f098806edb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f099c007fd0 tx=0x7f099c007960 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.524+0000 7f09a2a00640 1 -- 172.21.15.69:0/2567446711 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f099405a8f0 con 0x7f09ac0ce250 2024-04-24T08:21:01.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.726+0000 7f09b1400640 1 -- 172.21.15.69:0/2567446711 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f09ac0cddc0 con 0x7f09ac0ce250 2024-04-24T08:21:01.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.727+0000 7f09a2a00640 1 -- 172.21.15.69:0/2567446711 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v110) ==== 100+0+11 (secure 0 0 0) 0x7f099405a1f0 con 0x7f09ac0ce250 2024-04-24T08:21:01.727 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:21:01.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 -- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098806c8f0 msgr2=0x7f098806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098806c8f0 0x7f098806edb0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f099c007fd0 tx=0x7f099c007960 comp rx=0 tx=0).stop 2024-04-24T08:21:01.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 -- 172.21.15.69:0/2567446711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 msgr2=0x7f09ac073460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 0x7f09ac073460 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f0994007ef0 tx=0x7f099400e4b0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 -- 172.21.15.69:0/2567446711 shutdown_connections 2024-04-24T08:21:01.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f098806c8f0 0x7f098806edb0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09ac0ce250 0x7f09ac073460 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09ac15c820 0x7f09ac0b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 --2- 172.21.15.69:0/2567446711 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f09ac0b3e90 0x7f09ac146850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 -- 172.21.15.69:0/2567446711 >> 172.21.15.69:0/2567446711 conn(0x7f09ac0d0350 msgr2=0x7f09ac0d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.729+0000 7f09b1400640 1 -- 172.21.15.69:0/2567446711 shutdown_connections 2024-04-24T08:21:01.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.730+0000 7f09b1400640 1 -- 172.21.15.69:0/2567446711 wait complete. 2024-04-24T08:21:01.747 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:21:01.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.943+0000 7f8585e00640 1 Processor -- start 2024-04-24T08:21:01.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.943+0000 7f8585e00640 1 -- start start 2024-04-24T08:21:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.945+0000 7f8585e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580071d70 0x7f8580072150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.945+0000 7f8585e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f858006eb90 con 0x7f858006f730 2024-04-24T08:21:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.945+0000 7f8585e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8580072690 con 0x7f85800b3320 2024-04-24T08:21:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.945+0000 7f8585e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f85800b4f80 con 0x7f8580071d70 2024-04-24T08:21:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.945+0000 7f8576000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f858006f730 0x7f858006dd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49590/0 (socket says 172.21.15.69:49590) 2024-04-24T08:21:01.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.945+0000 7f8576000640 1 -- 172.21.15.69:0/922350064 learned_addr learned my addr 172.21.15.69:0/922350064 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857ea00640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580071d70 0x7f8580072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1110619180 0 0) 0x7f858006eb90 con 0x7f858006f730 2024-04-24T08:21:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8564003520 con 0x7f858006f730 2024-04-24T08:21:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 272451014 0 0) 0x7f8564003520 con 0x7f858006f730 2024-04-24T08:21:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580071d70 msgr2=0x7f8580072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580071d70 0x7f8580072150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 >> v1:172.21.15.121:6789/0 conn(0x7f85800b3320 legacy=0x7f858006e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85800b4370 con 0x7f858006f730 2024-04-24T08:21:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f8568001670 con 0x7f858006f730 2024-04-24T08:21:01.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 >> v1:172.21.15.69:6789/0 conn(0x7f858006f730 legacy=0x7f858006dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580071d70 0x7f85640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8564004010 0x7f8564004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 0x7f85640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85800b4f80 con 0x7f8564004d40 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f856400bb80 con 0x7f8564004010 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.946+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f856400bd80 con 0x7f8580071d70 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.947+0000 7f857f400640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 0x7f85640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.947+0000 7f8576000640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8564004010 0x7f8564004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.947+0000 7f857ea00640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580071d70 0x7f85640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.947+0000 7f857f400640 1 -- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580071d70 msgr2=0x7f85640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.947+0000 7f857f400640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580071d70 0x7f85640039e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.947+0000 7f857f400640 1 -- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8564004010 msgr2=0x7f8564004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.947+0000 7f857f400640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8564004010 0x7f8564004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.947+0000 7f857f400640 1 -- 172.21.15.69:0/922350064 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8564003520 con 0x7f8564004d40 2024-04-24T08:21:01.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.947+0000 7f8576000640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8564004010 0x7f8564004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:01.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.948+0000 7f857f400640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 0x7f85640051c0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f8578001e50 tx=0x7f8564005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8695aa8b7d8b046b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.948+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8578001a70 con 0x7f8564004d40 2024-04-24T08:21:01.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.948+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8578003970 con 0x7f8564004d40 2024-04-24T08:21:01.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.948+0000 7f857e000640 1 -- 172.21.15.69:0/922350064 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8578003d60 con 0x7f8564004d40 2024-04-24T08:21:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.949+0000 7f8585e00640 1 -- 172.21.15.69:0/922350064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 msgr2=0x7f85640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.949+0000 7f8585e00640 1 --2- 172.21.15.69:0/922350064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 0x7f85640051c0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f8578001e50 tx=0x7f8564005830 comp rx=0 tx=0).stop 2024-04-24T08:21:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.949+0000 7f857f400640 1 -- 172.21.15.69:0/922350064 reap_dead start 2024-04-24T08:21:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.949+0000 7f8585e00640 1 -- 172.21.15.69:0/922350064 shutdown_connections 2024-04-24T08:21:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.949+0000 7f8585e00640 1 -- 172.21.15.69:0/922350064 >> 172.21.15.69:0/922350064 conn(0x7f8580067d90 msgr2=0x7f858006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.949+0000 7f8585e00640 1 -- 172.21.15.69:0/922350064 shutdown_connections 2024-04-24T08:21:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.949+0000 7f8585e00640 1 -- 172.21.15.69:0/922350064 wait complete. 2024-04-24T08:21:01.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.951+0000 7f8585e00640 1 Processor -- start 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.951+0000 7f8585e00640 1 -- start start 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.952+0000 7f8585e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 0x7f85800b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.952+0000 7f8585e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580155550 0x7f85801559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.952+0000 7f8585e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f858015c8d0 0x7f858015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.952+0000 7f8585e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85800b4c10 con 0x7f8564004d40 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.952+0000 7f8585e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f85800b4930 con 0x7f858015c8d0 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.952+0000 7f8585e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f858006ee60 con 0x7f8580155550 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.952+0000 7f857ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 0x7f85800b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.952+0000 7f857ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 0x7f85800b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34070/0 (socket says 172.21.15.69:34070) 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.952+0000 7f857ea00640 1 -- 172.21.15.69:0/422077396 learned_addr learned my addr 172.21.15.69:0/422077396 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f857ea00640 1 -- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580155550 msgr2=0x7f85801559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f857ea00640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580155550 0x7f85801559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f857ea00640 1 -- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f858015c8d0 msgr2=0x7f858015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f857ea00640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f858015c8d0 0x7f858015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f857ea00640 1 -- 172.21.15.69:0/422077396 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858015d290 con 0x7f8564004d40 2024-04-24T08:21:01.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f857ea00640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 0x7f85800b18a0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f857000ed40 tx=0x7f857000c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f8577e00640 1 -- 172.21.15.69:0/422077396 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570019480 con 0x7f8564004d40 2024-04-24T08:21:01.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f8577e00640 1 -- 172.21.15.69:0/422077396 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8570019620 con 0x7f8564004d40 2024-04-24T08:21:01.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f8577e00640 1 -- 172.21.15.69:0/422077396 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85700199d0 con 0x7f8564004d40 2024-04-24T08:21:01.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f8585e00640 1 -- 172.21.15.69:0/422077396 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f858015b290 con 0x7f8564004d40 2024-04-24T08:21:01.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.953+0000 7f8585e00640 1 -- 172.21.15.69:0/422077396 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f858015b850 con 0x7f8564004d40 2024-04-24T08:21:01.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.955+0000 7f8585e00640 1 -- 172.21.15.69:0/422077396 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85800aed50 con 0x7f8564004d40 2024-04-24T08:21:01.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.956+0000 7f8577e00640 1 -- 172.21.15.69:0/422077396 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f857000b040 con 0x7f8564004d40 2024-04-24T08:21:01.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.956+0000 7f8577e00640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f855c06c8f0 0x7f855c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.957+0000 7f8577e00640 1 -- 172.21.15.69:0/422077396 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f857008c4b0 con 0x7f8564004d40 2024-04-24T08:21:01.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.957+0000 7f8576000640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f855c06c8f0 0x7f855c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.958+0000 7f8576000640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f855c06c8f0 0x7f855c06edb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8568001200 tx=0x7f8568009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:01.961+0000 7f8577e00640 1 -- 172.21.15.69:0/422077396 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f85800aed50 con 0x7f8564004d40 2024-04-24T08:21:02.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.174+0000 7f8585e00640 1 -- 172.21.15.69:0/422077396 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8580068a80 con 0x7f8564004d40 2024-04-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.175+0000 7f8577e00640 1 -- 172.21.15.69:0/422077396 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v110) ==== 80+0+124 (secure 0 0 0) 0x7f857005a7f0 con 0x7f8564004d40 2024-04-24T08:21:02.175 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6836/1515102940 2024-04-25T08:20:50.436176+0000 2024-04-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6834/1515102940 2024-04-25T08:20:50.436176+0000 2024-04-24T08:21:02.176 INFO:teuthology.orchestra.run.smithi069.stderr:listed 2 entries 2024-04-24T08:21:02.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.177+0000 7f8585e00640 1 -- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f855c06c8f0 msgr2=0x7f855c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.177+0000 7f8585e00640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f855c06c8f0 0x7f855c06edb0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f8568001200 tx=0x7f8568009040 comp rx=0 tx=0).stop 2024-04-24T08:21:02.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.177+0000 7f8585e00640 1 -- 172.21.15.69:0/422077396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 msgr2=0x7f85800b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.177+0000 7f8585e00640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 0x7f85800b18a0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f857000ed40 tx=0x7f857000c6b0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.178+0000 7f8585e00640 1 -- 172.21.15.69:0/422077396 shutdown_connections 2024-04-24T08:21:02.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.178+0000 7f8585e00640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f855c06c8f0 0x7f855c06edb0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.178+0000 7f8585e00640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f858015c8d0 0x7f858015cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.178+0000 7f8585e00640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8580155550 0x7f85801559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.178+0000 7f8585e00640 1 --2- 172.21.15.69:0/422077396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8564004d40 0x7f85800b18a0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.178+0000 7f8585e00640 1 -- 172.21.15.69:0/422077396 >> 172.21.15.69:0/422077396 conn(0x7f8580067d90 msgr2=0x7f858015a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:02.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.178+0000 7f8585e00640 1 -- 172.21.15.69:0/422077396 shutdown_connections 2024-04-24T08:21:02.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.178+0000 7f8585e00640 1 -- 172.21.15.69:0/422077396 wait complete. 2024-04-24T08:21:02.197 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:21:02.197 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:21:02.249 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:02.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.451+0000 7fb3e7200640 1 Processor -- start 2024-04-24T08:21:02.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.451+0000 7fb3e7200640 1 -- start start 2024-04-24T08:21:02.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.452+0000 7fb3e7200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb3e8156c60 0x7fb3e815cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.452+0000 7fb3e7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e80b30f0 0x7fb3e815d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.453+0000 7fb3e7200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3e815d660 con 0x7fb3e80ad0d0 2024-04-24T08:21:02.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.453+0000 7fb3e7200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb3e806dc30 con 0x7fb3e8156c60 2024-04-24T08:21:02.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.453+0000 7fb3e7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb3e806df10 con 0x7fb3e80b30f0 2024-04-24T08:21:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.453+0000 7fb3e5400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e80b30f0 0x7fb3e815d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.453+0000 7fb3e5400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e80b30f0 0x7fb3e815d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:47754/0 (socket says 172.21.15.69:47754) 2024-04-24T08:21:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.454+0000 7fb3e5400640 1 -- 172.21.15.69:0/3760746151 learned_addr learned my addr 172.21.15.69:0/3760746151 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:02.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.454+0000 7fb3e5e00640 1 --2- 172.21.15.69:0/3760746151 >> v2:172.21.15.121:3300/0 conn(0x7fb3e8156c60 0x7fb3e815cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.454+0000 7fb3e5400640 1 -- 172.21.15.69:0/3760746151 >> v2:172.21.15.121:3300/0 conn(0x7fb3e8156c60 msgr2=0x7fb3e815cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.454+0000 7fb3e5400640 1 --2- 172.21.15.69:0/3760746151 >> v2:172.21.15.121:3300/0 conn(0x7fb3e8156c60 0x7fb3e815cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.454+0000 7fb3e5400640 1 -- 172.21.15.69:0/3760746151 >> v1:172.21.15.121:6789/0 conn(0x7fb3e80ad0d0 legacy=0x7fb3e81600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.454+0000 7fb3e5400640 1 -- 172.21.15.69:0/3760746151 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e806e1f0 con 0x7fb3e80b30f0 2024-04-24T08:21:02.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.456+0000 7fb3e5400640 1 --2- 172.21.15.69:0/3760746151 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e80b30f0 0x7fb3e815d000 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009870 tx=0x7fb3d402a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=546316bf9949ef03 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:02.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.456+0000 7fb3e4a00640 1 -- 172.21.15.69:0/3760746151 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4037070 con 0x7fb3e80b30f0 2024-04-24T08:21:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.456+0000 7fb3e4a00640 1 -- 172.21.15.69:0/3760746151 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3d402adf0 con 0x7fb3e80b30f0 2024-04-24T08:21:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.457+0000 7fb3e4a00640 1 -- 172.21.15.69:0/3760746151 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4032660 con 0x7fb3e80b30f0 2024-04-24T08:21:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.457+0000 7fb3e7200640 1 -- 172.21.15.69:0/3760746151 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e80b30f0 msgr2=0x7fb3e815d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.457+0000 7fb3e7200640 1 --2- 172.21.15.69:0/3760746151 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e80b30f0 0x7fb3e815d000 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009870 tx=0x7fb3d402a160 comp rx=0 tx=0).stop 2024-04-24T08:21:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.457+0000 7fb3e7200640 1 -- 172.21.15.69:0/3760746151 shutdown_connections 2024-04-24T08:21:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.457+0000 7fb3e7200640 1 --2- 172.21.15.69:0/3760746151 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e80b30f0 0x7fb3e815d000 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.457+0000 7fb3e7200640 1 --2- 172.21.15.69:0/3760746151 >> v2:172.21.15.121:3300/0 conn(0x7fb3e8156c60 0x7fb3e815cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.457+0000 7fb3e7200640 1 -- 172.21.15.69:0/3760746151 >> 172.21.15.69:0/3760746151 conn(0x7fb3e80d0160 msgr2=0x7fb3e80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.457+0000 7fb3e7200640 1 -- 172.21.15.69:0/3760746151 shutdown_connections 2024-04-24T08:21:02.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.457+0000 7fb3e7200640 1 -- 172.21.15.69:0/3760746151 wait complete. 2024-04-24T08:21:02.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.459+0000 7fb3e7200640 1 Processor -- start 2024-04-24T08:21:02.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.459+0000 7fb3e7200640 1 -- start start 2024-04-24T08:21:02.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.460+0000 7fb3e7200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3e80ad0d0 0x7fb3e81f5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.461 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.460+0000 7fb3e7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3e80b30f0 0x7fb3e81f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.460+0000 7fb3e7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e8156c60 0x7fb3e8159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.460+0000 7fb3e7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e80ad530 con 0x7fb3e80b30f0 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.460+0000 7fb3e7200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e815d940 con 0x7fb3e80ad0d0 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.460+0000 7fb3e7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb3e80ad6b0 con 0x7fb3e8156c60 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.460+0000 7fb3e5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3e80b30f0 0x7fb3e81f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.461+0000 7fb3e5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3e80b30f0 0x7fb3e81f3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34082/0 (socket says 172.21.15.69:34082) 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.461+0000 7fb3e5400640 1 -- 172.21.15.69:0/1736084921 learned_addr learned my addr 172.21.15.69:0/1736084921 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.461+0000 7fb3e5e00640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3e80ad0d0 0x7fb3e81f5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.461+0000 7fb3e6800640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e8156c60 0x7fb3e8159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.461+0000 7fb3e5400640 1 -- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e8156c60 msgr2=0x7fb3e8159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.461+0000 7fb3e5400640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e8156c60 0x7fb3e8159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.461+0000 7fb3e5400640 1 -- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3e80ad0d0 msgr2=0x7fb3e81f5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.461+0000 7fb3e5400640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3e80ad0d0 0x7fb3e81f5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.461+0000 7fb3e5400640 1 -- 172.21.15.69:0/1736084921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e81f7d90 con 0x7fb3e80b30f0 2024-04-24T08:21:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.461+0000 7fb3e5400640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3e80b30f0 0x7fb3e81f3fe0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009870 tx=0x7fb3d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.462+0000 7fb3de000640 1 -- 172.21.15.69:0/1736084921 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4037040 con 0x7fb3e80b30f0 2024-04-24T08:21:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.462+0000 7fb3de000640 1 -- 172.21.15.69:0/1736084921 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3d4004030 con 0x7fb3e80b30f0 2024-04-24T08:21:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.462+0000 7fb3de000640 1 -- 172.21.15.69:0/1736084921 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4004220 con 0x7fb3e80b30f0 2024-04-24T08:21:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.462+0000 7fb3e7200640 1 -- 172.21.15.69:0/1736084921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e81f8070 con 0x7fb3e80b30f0 2024-04-24T08:21:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.462+0000 7fb3e7200640 1 -- 172.21.15.69:0/1736084921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e81f8630 con 0x7fb3e80b30f0 2024-04-24T08:21:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.463+0000 7fb3e7200640 1 -- 172.21.15.69:0/1736084921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3e8075d50 con 0x7fb3e80b30f0 2024-04-24T08:21:02.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.464+0000 7fb3de000640 1 -- 172.21.15.69:0/1736084921 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb3d403a040 con 0x7fb3e80b30f0 2024-04-24T08:21:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.465+0000 7fb3de000640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3c806c940 0x7fb3c806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.465+0000 7fb3de000640 1 -- 172.21.15.69:0/1736084921 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7fb3d40ac450 con 0x7fb3e80b30f0 2024-04-24T08:21:02.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.466+0000 7fb3e5e00640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3c806c940 0x7fb3c806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.467+0000 7fb3e5e00640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3c806c940 0x7fb3c806ee00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80b1540 tx=0x7fb3d8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:02.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.469+0000 7fb3de000640 1 -- 172.21.15.69:0/1736084921 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3d407a920 con 0x7fb3e80b30f0 2024-04-24T08:21:02.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.672+0000 7fb3e7200640 1 -- 172.21.15.69:0/1736084921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb3e80d05b0 con 0x7fb3e80b30f0 2024-04-24T08:21:02.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.673+0000 7fb3de000640 1 -- 172.21.15.69:0/1736084921 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+16734 (secure 0 0 0) 0x7fb3d407a220 con 0x7fb3e80b30f0 2024-04-24T08:21:02.674 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:02.675 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":110,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:20:50.436238+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.121:6836/1515102940":"2024-04-25T08:20:50.436176+0000","172.21.15.121:6834/1515102940":"2024-04-25T08:20:50.436176+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:02.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 -- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3c806c940 msgr2=0x7fb3c806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3c806c940 0x7fb3c806ee00 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb3e80b1540 tx=0x7fb3d8008040 comp rx=0 tx=0).stop 2024-04-24T08:21:02.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 -- 172.21.15.69:0/1736084921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3e80b30f0 msgr2=0x7fb3e81f3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3e80b30f0 0x7fb3e81f3fe0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4009870 tx=0x7fb3d40046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 -- 172.21.15.69:0/1736084921 shutdown_connections 2024-04-24T08:21:02.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb3c806c940 0x7fb3c806ee00 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb3e80b30f0 0x7fb3e81f3fe0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb3e8156c60 0x7fb3e8159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 --2- 172.21.15.69:0/1736084921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3e80ad0d0 0x7fb3e81f5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 -- 172.21.15.69:0/1736084921 >> 172.21.15.69:0/1736084921 conn(0x7fb3e80d0160 msgr2=0x7fb3e80aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:02.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.676+0000 7fb3e7200640 1 -- 172.21.15.69:0/1736084921 shutdown_connections 2024-04-24T08:21:02.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.677+0000 7fb3e7200640 1 -- 172.21.15.69:0/1736084921 wait complete. 2024-04-24T08:21:02.696 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:20:06.963817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:20:08.190440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:21:02.696 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:21:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.900+0000 7f035d800640 1 Processor -- start 2024-04-24T08:21:02.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.900+0000 7f035d800640 1 -- start start 2024-04-24T08:21:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.902+0000 7f035d800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f035815c820 0x7f03580b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.902+0000 7f035d800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f03580ce250 0x7f03580ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.902+0000 7f035d800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f035815cc00 con 0x7f03580b46a0 2024-04-24T08:21:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.902+0000 7f035d800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f035806ed30 con 0x7f03580ce250 2024-04-24T08:21:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.902+0000 7f035d800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f03580b4e10 con 0x7f035815c820 2024-04-24T08:21:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.902+0000 7f0357400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f03580ce250 0x7f03580ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.903+0000 7f0357400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f03580ce250 0x7f03580ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34086/0 (socket says 172.21.15.69:34086) 2024-04-24T08:21:02.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.903+0000 7f0357400640 1 -- 172.21.15.69:0/1337915830 learned_addr learned my addr 172.21.15.69:0/1337915830 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.903+0000 7f0356a00640 1 --2- 172.21.15.69:0/1337915830 >> v2:172.21.15.121:3300/0 conn(0x7f035815c820 0x7f03580b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.903+0000 7f0357400640 1 -- 172.21.15.69:0/1337915830 >> v2:172.21.15.121:3300/0 conn(0x7f035815c820 msgr2=0x7f03580b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.903+0000 7f0357400640 1 --2- 172.21.15.69:0/1337915830 >> v2:172.21.15.121:3300/0 conn(0x7f035815c820 0x7f03580b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.903+0000 7f0357400640 1 -- 172.21.15.69:0/1337915830 >> v1:172.21.15.121:6789/0 conn(0x7f03580b46a0 legacy=0x7f035806f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.903+0000 7f0357400640 1 -- 172.21.15.69:0/1337915830 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03580b50f0 con 0x7f03580ce250 2024-04-24T08:21:02.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.903+0000 7f0356a00640 1 --2- 172.21.15.69:0/1337915830 >> v2:172.21.15.121:3300/0 conn(0x7f035815c820 0x7f03580b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.904+0000 7f0357400640 1 --2- 172.21.15.69:0/1337915830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580ce250 0x7f03580ce6c0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f03580b2f60 tx=0x7f035002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8556831f501d1846 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.904+0000 7f0355600640 1 -- 172.21.15.69:0/1337915830 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0350037070 con 0x7f03580ce250 2024-04-24T08:21:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.904+0000 7f0355600640 1 -- 172.21.15.69:0/1337915830 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0350032440 con 0x7f03580ce250 2024-04-24T08:21:02.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.904+0000 7f0355600640 1 -- 172.21.15.69:0/1337915830 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0350032810 con 0x7f03580ce250 2024-04-24T08:21:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.905+0000 7f035d800640 1 -- 172.21.15.69:0/1337915830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580ce250 msgr2=0x7f03580ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.905+0000 7f035d800640 1 --2- 172.21.15.69:0/1337915830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580ce250 0x7f03580ce6c0 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f03580b2f60 tx=0x7f035002a230 comp rx=0 tx=0).stop 2024-04-24T08:21:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.905+0000 7f035d800640 1 -- 172.21.15.69:0/1337915830 shutdown_connections 2024-04-24T08:21:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.905+0000 7f035d800640 1 --2- 172.21.15.69:0/1337915830 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580ce250 0x7f03580ce6c0 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.905+0000 7f035d800640 1 --2- 172.21.15.69:0/1337915830 >> v2:172.21.15.121:3300/0 conn(0x7f035815c820 0x7f03580b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.905+0000 7f035d800640 1 -- 172.21.15.69:0/1337915830 >> 172.21.15.69:0/1337915830 conn(0x7f03580d0350 msgr2=0x7f03580d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.905+0000 7f035d800640 1 -- 172.21.15.69:0/1337915830 shutdown_connections 2024-04-24T08:21:02.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.905+0000 7f035d800640 1 -- 172.21.15.69:0/1337915830 wait complete. 2024-04-24T08:21:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.908+0000 7f035d800640 1 Processor -- start 2024-04-24T08:21:02.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.908+0000 7f035d800640 1 -- start start 2024-04-24T08:21:02.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.909+0000 7f035d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580b46a0 0x7f035815aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.909+0000 7f035d800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03580ce250 0x7f0358159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.909+0000 7f035d800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f035815c820 0x7f03581f42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.909+0000 7f035d800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f03580b4b30 con 0x7f03580b46a0 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.909+0000 7f035d800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f035806f2b0 con 0x7f03580ce250 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.909+0000 7f035d800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f035806fee0 con 0x7f035815c820 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.909+0000 7f0356a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580b46a0 0x7f035815aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.909+0000 7f0356a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580b46a0 0x7f035815aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34090/0 (socket says 172.21.15.69:34090) 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.909+0000 7f0356a00640 1 -- 172.21.15.69:0/2616732001 learned_addr learned my addr 172.21.15.69:0/2616732001 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.910+0000 7f0356000640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03580ce250 0x7f0358159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.910+0000 7f0357400640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f035815c820 0x7f03581f42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.910+0000 7f0356a00640 1 -- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f035815c820 msgr2=0x7f03581f42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.910+0000 7f0356a00640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f035815c820 0x7f03581f42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.910+0000 7f0356a00640 1 -- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03580ce250 msgr2=0x7f0358159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.910+0000 7f0356a00640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03580ce250 0x7f0358159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.910+0000 7f0356a00640 1 -- 172.21.15.69:0/2616732001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0358159a50 con 0x7f03580b46a0 2024-04-24T08:21:02.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.910+0000 7f0356a00640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580b46a0 0x7f035815aaf0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f034800c7f0 tx=0x7f034800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:02.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.911+0000 7f034ea00640 1 -- 172.21.15.69:0/2616732001 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0348015070 con 0x7f03580b46a0 2024-04-24T08:21:02.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.911+0000 7f034ea00640 1 -- 172.21.15.69:0/2616732001 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03480079d0 con 0x7f03580b46a0 2024-04-24T08:21:02.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.911+0000 7f034ea00640 1 -- 172.21.15.69:0/2616732001 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0348007da0 con 0x7f03580b46a0 2024-04-24T08:21:02.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.911+0000 7f035d800640 1 -- 172.21.15.69:0/2616732001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0358159d30 con 0x7f03580b46a0 2024-04-24T08:21:02.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.911+0000 7f035d800640 1 -- 172.21.15.69:0/2616732001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f035815a2f0 con 0x7f03580b46a0 2024-04-24T08:21:02.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.912+0000 7f035d800640 1 -- 172.21.15.69:0/2616732001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0358068a80 con 0x7f03580b46a0 2024-04-24T08:21:02.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.913+0000 7f034ea00640 1 -- 172.21.15.69:0/2616732001 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0348018070 con 0x7f03580b46a0 2024-04-24T08:21:02.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.914+0000 7f034ea00640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f033406c920 0x7f033406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.914+0000 7f034ea00640 1 -- 172.21.15.69:0/2616732001 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f034808bba0 con 0x7f03580b46a0 2024-04-24T08:21:02.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.915+0000 7f0356000640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f033406c920 0x7f033406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.916+0000 7f0356000640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f033406c920 0x7f033406ede0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0358146060 tx=0x7f03400094d0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:02.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:02.920+0000 7f034ea00640 1 -- 172.21.15.69:0/2616732001 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0348059ed0 con 0x7f03580b46a0 2024-04-24T08:21:03.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.121+0000 7f035d800640 1 -- 172.21.15.69:0/2616732001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f03580b0250 con 0x7f03580b46a0 2024-04-24T08:21:03.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.122+0000 7f034ea00640 1 -- 172.21.15.69:0/2616732001 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) ==== 93+0+10 (secure 0 0 0) 0x7f03480597d0 con 0x7f03580b46a0 2024-04-24T08:21:03.122 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:21:03.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 -- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f033406c920 msgr2=0x7f033406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f033406c920 0x7f033406ede0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f0358146060 tx=0x7f03400094d0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 -- 172.21.15.69:0/2616732001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580b46a0 msgr2=0x7f035815aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580b46a0 0x7f035815aaf0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f034800c7f0 tx=0x7f034800cbf0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 -- 172.21.15.69:0/2616732001 shutdown_connections 2024-04-24T08:21:03.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f033406c920 0x7f033406ede0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f035815c820 0x7f03581f42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f03580ce250 0x7f0358159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 --2- 172.21.15.69:0/2616732001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03580b46a0 0x7f035815aaf0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 -- 172.21.15.69:0/2616732001 >> 172.21.15.69:0/2616732001 conn(0x7f03580d0350 msgr2=0x7f03580b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:03.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 -- 172.21.15.69:0/2616732001 shutdown_connections 2024-04-24T08:21:03.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.124+0000 7f035d800640 1 -- 172.21.15.69:0/2616732001 wait complete. 2024-04-24T08:21:03.143 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:21:03.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.337+0000 7f6b99e00640 1 Processor -- start 2024-04-24T08:21:03.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.337+0000 7f6b99e00640 1 -- start start 2024-04-24T08:21:03.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.338+0000 7f6b99e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b9415c820 0x7f6b940b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.338+0000 7f6b99e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6b940ce250 0x7f6b940ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.339+0000 7f6b99e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b9415cc00 con 0x7f6b940b46a0 2024-04-24T08:21:03.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.339+0000 7f6b99e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f6b9406ed30 con 0x7f6b940ce250 2024-04-24T08:21:03.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.339+0000 7f6b99e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6b940b4e10 con 0x7f6b9415c820 2024-04-24T08:21:03.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.339+0000 7f6b92a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b9415c820 0x7f6b940b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.339+0000 7f6b8a000640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f6b940b46a0 0x7f6b9406f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:40006/0 (socket says 172.21.15.69:40006) 2024-04-24T08:21:03.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.339+0000 7f6b8a000640 1 -- 172.21.15.69:0/1549474694 learned_addr learned my addr 172.21.15.69:0/1549474694 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:03.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.339+0000 7f6b93400640 1 --2- 172.21.15.69:0/1549474694 >> v2:172.21.15.121:3300/0 conn(0x7f6b940ce250 0x7f6b940ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.340+0000 7f6b92000640 1 -- 172.21.15.69:0/1549474694 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 934325639 0 0) 0x7f6b9415cc00 con 0x7f6b940b46a0 2024-04-24T08:21:03.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.340+0000 7f6b92000640 1 -- 172.21.15.69:0/1549474694 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b78003520 con 0x7f6b940b46a0 2024-04-24T08:21:03.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.340+0000 7f6b92a00640 1 -- 172.21.15.69:0/1549474694 >> v2:172.21.15.121:3300/0 conn(0x7f6b940ce250 msgr2=0x7f6b940ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:03.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.340+0000 7f6b92a00640 1 --2- 172.21.15.69:0/1549474694 >> v2:172.21.15.121:3300/0 conn(0x7f6b940ce250 0x7f6b940ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.340+0000 7f6b92a00640 1 -- 172.21.15.69:0/1549474694 >> v1:172.21.15.121:6789/0 conn(0x7f6b940b46a0 legacy=0x7f6b9406f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.340+0000 7f6b92a00640 1 -- 172.21.15.69:0/1549474694 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b940b50f0 con 0x7f6b9415c820 2024-04-24T08:21:03.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.342+0000 7f6b92a00640 1 --2- 172.21.15.69:0/1549474694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b9415c820 0x7f6b940b3e90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009d90 tx=0x7f6b8402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ac77c4bd24635af server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:03.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.342+0000 7f6b92000640 1 -- 172.21.15.69:0/1549474694 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b84007740 con 0x7f6b9415c820 2024-04-24T08:21:03.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.342+0000 7f6b92000640 1 -- 172.21.15.69:0/1549474694 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b840078e0 con 0x7f6b9415c820 2024-04-24T08:21:03.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.342+0000 7f6b92000640 1 -- 172.21.15.69:0/1549474694 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b84007cb0 con 0x7f6b9415c820 2024-04-24T08:21:03.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.343+0000 7f6b99e00640 1 -- 172.21.15.69:0/1549474694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b9415c820 msgr2=0x7f6b940b3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.343+0000 7f6b99e00640 1 --2- 172.21.15.69:0/1549474694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b9415c820 0x7f6b940b3e90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009d90 tx=0x7f6b8402a660 comp rx=0 tx=0).stop 2024-04-24T08:21:03.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.343+0000 7f6b99e00640 1 -- 172.21.15.69:0/1549474694 shutdown_connections 2024-04-24T08:21:03.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.343+0000 7f6b99e00640 1 --2- 172.21.15.69:0/1549474694 >> v2:172.21.15.121:3300/0 conn(0x7f6b940ce250 0x7f6b940ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.343+0000 7f6b99e00640 1 --2- 172.21.15.69:0/1549474694 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b9415c820 0x7f6b940b3e90 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.343+0000 7f6b99e00640 1 -- 172.21.15.69:0/1549474694 >> 172.21.15.69:0/1549474694 conn(0x7f6b940d0350 msgr2=0x7f6b940d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:03.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.343+0000 7f6b99e00640 1 -- 172.21.15.69:0/1549474694 shutdown_connections 2024-04-24T08:21:03.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.343+0000 7f6b99e00640 1 -- 172.21.15.69:0/1549474694 wait complete. 2024-04-24T08:21:03.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.345+0000 7f6b99e00640 1 Processor -- start 2024-04-24T08:21:03.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.345+0000 7f6b99e00640 1 -- start start 2024-04-24T08:21:03.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.346+0000 7f6b99e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b940b46a0 0x7f6b9415e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.346+0000 7f6b99e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b940ce250 0x7f6b94151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.346+0000 7f6b99e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b9415c820 0x7f6b940ae850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.346+0000 7f6b99e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6b940b4b00 con 0x7f6b940ce250 2024-04-24T08:21:03.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.346+0000 7f6b99e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9406f2b0 con 0x7f6b940b46a0 2024-04-24T08:21:03.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.346+0000 7f6b99e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9406fee0 con 0x7f6b9415c820 2024-04-24T08:21:03.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.346+0000 7f6b8a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b940ce250 0x7f6b94151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.346+0000 7f6b8a000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b940ce250 0x7f6b94151870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34094/0 (socket says 172.21.15.69:34094) 2024-04-24T08:21:03.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.346+0000 7f6b8a000640 1 -- 172.21.15.69:0/1224175922 learned_addr learned my addr 172.21.15.69:0/1224175922 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:03.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b92a00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b940b46a0 0x7f6b9415e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b8a000640 1 -- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b9415c820 msgr2=0x7f6b940ae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b8a000640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b9415c820 0x7f6b940ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b8a000640 1 -- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b940b46a0 msgr2=0x7f6b9415e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b8a000640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b940b46a0 0x7f6b9415e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b8a000640 1 -- 172.21.15.69:0/1224175922 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9415d8b0 con 0x7f6b940ce250 2024-04-24T08:21:03.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b92a00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b940b46a0 0x7f6b9415e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:03.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b8a000640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b940ce250 0x7f6b94151870 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00c430 tx=0x7f6b7c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:03.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b8be00640 1 -- 172.21.15.69:0/1224175922 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c017070 con 0x7f6b940ce250 2024-04-24T08:21:03.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b8be00640 1 -- 172.21.15.69:0/1224175922 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b7c005ce0 con 0x7f6b940ce250 2024-04-24T08:21:03.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b8be00640 1 -- 172.21.15.69:0/1224175922 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b7c0072f0 con 0x7f6b940ce250 2024-04-24T08:21:03.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.347+0000 7f6b99e00640 1 -- 172.21.15.69:0/1224175922 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9415db90 con 0x7f6b940ce250 2024-04-24T08:21:03.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.348+0000 7f6b99e00640 1 -- 172.21.15.69:0/1224175922 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9415e150 con 0x7f6b940ce250 2024-04-24T08:21:03.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.349+0000 7f6b99e00640 1 -- 172.21.15.69:0/1224175922 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b94068a80 con 0x7f6b940ce250 2024-04-24T08:21:03.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.350+0000 7f6b8be00640 1 -- 172.21.15.69:0/1224175922 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6b7c01a070 con 0x7f6b940ce250 2024-04-24T08:21:03.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.351+0000 7f6b8be00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6b7006c8f0 0x7f6b7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.351+0000 7f6b8be00640 1 -- 172.21.15.69:0/1224175922 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(110..110 src has 1..110) ==== 5795+0+0 (secure 0 0 0) 0x7f6b7c08b130 con 0x7f6b940ce250 2024-04-24T08:21:03.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.351+0000 7f6b92a00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6b7006c8f0 0x7f6b7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.352+0000 7f6b92a00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6b7006c8f0 0x7f6b7006edb0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f6b84034000 tx=0x7f6b8402f000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:03.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.355+0000 7f6b8be00640 1 -- 172.21.15.69:0/1224175922 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6b7c059460 con 0x7f6b940ce250 2024-04-24T08:21:03.562 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:21:03.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.561+0000 7f6b99e00640 1 -- 172.21.15.69:0/1224175922 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6b9415e580 con 0x7f6b940ce250 2024-04-24T08:21:03.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.562+0000 7f6b8be00640 1 -- 172.21.15.69:0/1224175922 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) ==== 104+0+11 (secure 0 0 0) 0x7f6b7c058d60 con 0x7f6b940ce250 2024-04-24T08:21:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.565+0000 7f6b99e00640 1 -- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6b7006c8f0 msgr2=0x7f6b7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.565+0000 7f6b99e00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6b7006c8f0 0x7f6b7006edb0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f6b84034000 tx=0x7f6b8402f000 comp rx=0 tx=0).stop 2024-04-24T08:21:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.565+0000 7f6b99e00640 1 -- 172.21.15.69:0/1224175922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b940ce250 msgr2=0x7f6b94151870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.565+0000 7f6b99e00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b940ce250 0x7f6b94151870 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c00c430 tx=0x7f6b7c00c8f0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.565+0000 7f6b99e00640 1 -- 172.21.15.69:0/1224175922 shutdown_connections 2024-04-24T08:21:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.565+0000 7f6b99e00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f6b7006c8f0 0x7f6b7006edb0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.565+0000 7f6b99e00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b940ce250 0x7f6b94151870 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.565+0000 7f6b99e00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b9415c820 0x7f6b940ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.565+0000 7f6b99e00640 1 --2- 172.21.15.69:0/1224175922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b940b46a0 0x7f6b9415e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.565+0000 7f6b99e00640 1 -- 172.21.15.69:0/1224175922 >> 172.21.15.69:0/1224175922 conn(0x7f6b940d0350 msgr2=0x7f6b940b1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:03.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.566+0000 7f6b99e00640 1 -- 172.21.15.69:0/1224175922 shutdown_connections 2024-04-24T08:21:03.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.566+0000 7f6b99e00640 1 -- 172.21.15.69:0/1224175922 wait complete. 2024-04-24T08:21:03.585 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:21:03.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.779+0000 7f80c7200640 1 Processor -- start 2024-04-24T08:21:03.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.779+0000 7f80c7200640 1 -- start start 2024-04-24T08:21:03.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.780+0000 7f80c7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00b3e90 0x7f80c00b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.780+0000 7f80c7200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f80c00ce250 0x7f80c00ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.780+0000 7f80c7200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80c015cc00 con 0x7f80c015c820 2024-04-24T08:21:03.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.780+0000 7f80c7200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f80c00b4be0 con 0x7f80c00ce250 2024-04-24T08:21:03.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.780+0000 7f80c7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f80c00b42f0 con 0x7f80c00b3e90 2024-04-24T08:21:03.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.781+0000 7f80b7e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00b3e90 0x7f80c00b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.781+0000 7f80bfe00640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f80c015c820 0x7f80c006f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:40012/0 (socket says 172.21.15.69:40012) 2024-04-24T08:21:03.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.781+0000 7f80bfe00640 1 -- 172.21.15.69:0/4082687482 learned_addr learned my addr 172.21.15.69:0/4082687482 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:03.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.782+0000 7f80c5200640 1 --2- 172.21.15.69:0/4082687482 >> v2:172.21.15.121:3300/0 conn(0x7f80c00ce250 0x7f80c00ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.782+0000 7f80bf400640 1 -- 172.21.15.69:0/4082687482 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2339860005 0 0) 0x7f80c015cc00 con 0x7f80c015c820 2024-04-24T08:21:03.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.782+0000 7f80bf400640 1 -- 172.21.15.69:0/4082687482 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80a4003520 con 0x7f80c015c820 2024-04-24T08:21:03.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.782+0000 7f80b7e00640 1 -- 172.21.15.69:0/4082687482 >> v2:172.21.15.121:3300/0 conn(0x7f80c00ce250 msgr2=0x7f80c00ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:03.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.782+0000 7f80b7e00640 1 --2- 172.21.15.69:0/4082687482 >> v2:172.21.15.121:3300/0 conn(0x7f80c00ce250 0x7f80c00ce6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.782+0000 7f80b7e00640 1 -- 172.21.15.69:0/4082687482 >> v1:172.21.15.121:6789/0 conn(0x7f80c015c820 legacy=0x7f80c006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.782+0000 7f80b7e00640 1 -- 172.21.15.69:0/4082687482 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c006ed30 con 0x7f80c00b3e90 2024-04-24T08:21:03.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.784+0000 7f80b7e00640 1 --2- 172.21.15.69:0/4082687482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00b3e90 0x7f80c00b46a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f80a80097a0 tx=0x7f80a802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10eb9e0f75fdf1fd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:03.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.784+0000 7f80bf400640 1 -- 172.21.15.69:0/4082687482 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8037070 con 0x7f80c00b3e90 2024-04-24T08:21:03.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.784+0000 7f80bf400640 1 -- 172.21.15.69:0/4082687482 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80a8032440 con 0x7f80c00b3e90 2024-04-24T08:21:03.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.784+0000 7f80bf400640 1 -- 172.21.15.69:0/4082687482 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a8032810 con 0x7f80c00b3e90 2024-04-24T08:21:03.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.785+0000 7f80c7200640 1 -- 172.21.15.69:0/4082687482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00b3e90 msgr2=0x7f80c00b46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.785+0000 7f80c7200640 1 --2- 172.21.15.69:0/4082687482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00b3e90 0x7f80c00b46a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f80a80097a0 tx=0x7f80a802a180 comp rx=0 tx=0).stop 2024-04-24T08:21:03.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.785+0000 7f80c7200640 1 -- 172.21.15.69:0/4082687482 shutdown_connections 2024-04-24T08:21:03.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.785+0000 7f80c7200640 1 --2- 172.21.15.69:0/4082687482 >> v2:172.21.15.121:3300/0 conn(0x7f80c00ce250 0x7f80c00ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.785+0000 7f80c7200640 1 --2- 172.21.15.69:0/4082687482 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00b3e90 0x7f80c00b46a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.785+0000 7f80c7200640 1 -- 172.21.15.69:0/4082687482 >> 172.21.15.69:0/4082687482 conn(0x7f80c00d0350 msgr2=0x7f80c00d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:03.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.785+0000 7f80c7200640 1 -- 172.21.15.69:0/4082687482 shutdown_connections 2024-04-24T08:21:03.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.785+0000 7f80c7200640 1 -- 172.21.15.69:0/4082687482 wait complete. 2024-04-24T08:21:03.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.786+0000 7f80c7200640 1 Processor -- start 2024-04-24T08:21:03.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.787+0000 7f80c7200640 1 -- start start 2024-04-24T08:21:03.789 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.787+0000 7f80c7200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c00b3e90 0x7f80c0156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.787+0000 7f80c7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00ce250 0x7f80c00ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80c015c820 0x7f80c00cd000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80c006efd0 con 0x7f80c015c820 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c7200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f80c00b50f0 con 0x7f80c00b3e90 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f80c006f2b0 con 0x7f80c00ce250 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80c015c820 0x7f80c00cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c5200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80c015c820 0x7f80c00cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34104/0 (socket says 172.21.15.69:34104) 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c5200640 1 -- 172.21.15.69:0/2166182842 learned_addr learned my addr 172.21.15.69:0/2166182842 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80bfe00640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c00b3e90 0x7f80c0156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80b7e00640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00ce250 0x7f80c00ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c5200640 1 -- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00ce250 msgr2=0x7f80c00ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.790 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c5200640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00ce250 0x7f80c00ccac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c5200640 1 -- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c00b3e90 msgr2=0x7f80c0156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c5200640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c00b3e90 0x7f80c0156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80c5200640 1 -- 172.21.15.69:0/2166182842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c015de50 con 0x7f80c015c820 2024-04-24T08:21:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.788+0000 7f80bfe00640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c00b3e90 0x7f80c0156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.789+0000 7f80c5200640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80c015c820 0x7f80c00cd000 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f80b800efc0 tx=0x7f80b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.789+0000 7f80bcc00640 1 -- 172.21.15.69:0/2166182842 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8018020 con 0x7f80c015c820 2024-04-24T08:21:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.789+0000 7f80bcc00640 1 -- 172.21.15.69:0/2166182842 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80b8014480 con 0x7f80c015c820 2024-04-24T08:21:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.789+0000 7f80c7200640 1 -- 172.21.15.69:0/2166182842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c00cb850 con 0x7f80c015c820 2024-04-24T08:21:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.789+0000 7f80bcc00640 1 -- 172.21.15.69:0/2166182842 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8014870 con 0x7f80c015c820 2024-04-24T08:21:03.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.789+0000 7f80c7200640 1 -- 172.21.15.69:0/2166182842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c015e110 con 0x7f80c015c820 2024-04-24T08:21:03.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.790+0000 7f80c7200640 1 -- 172.21.15.69:0/2166182842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80c0068a80 con 0x7f80c015c820 2024-04-24T08:21:03.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.791+0000 7f80bcc00640 1 -- 172.21.15.69:0/2166182842 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f80b8005d10 con 0x7f80c015c820 2024-04-24T08:21:03.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.792+0000 7f80bcc00640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f809c06c8f0 0x7f809c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.792+0000 7f80bcc00640 1 -- 172.21.15.69:0/2166182842 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5853+0+0 (secure 0 0 0) 0x7f80b808c6f0 con 0x7f80c015c820 2024-04-24T08:21:03.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.793+0000 7f80bfe00640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f809c06c8f0 0x7f809c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.794+0000 7f80bfe00640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f809c06c8f0 0x7f809c06edb0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f80b0005880 tx=0x7f80b0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:03.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.796+0000 7f80bcc00640 1 -- 172.21.15.69:0/2166182842 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f80b8022070 con 0x7f80c015c820 2024-04-24T08:21:03.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.996+0000 7f80c7200640 1 -- 172.21.15.69:0/2166182842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f80c00cddc0 con 0x7f80c015c820 2024-04-24T08:21:03.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:03.997+0000 7f80bcc00640 1 -- 172.21.15.69:0/2166182842 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) ==== 100+0+11 (secure 0 0 0) 0x7f80b8010040 con 0x7f80c015c820 2024-04-24T08:21:03.998 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:21:04.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 -- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f809c06c8f0 msgr2=0x7f809c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f809c06c8f0 0x7f809c06edb0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f80b0005880 tx=0x7f80b0009040 comp rx=0 tx=0).stop 2024-04-24T08:21:04.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 -- 172.21.15.69:0/2166182842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80c015c820 msgr2=0x7f80c00cd000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80c015c820 0x7f80c00cd000 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f80b800efc0 tx=0x7f80b800c490 comp rx=0 tx=0).stop 2024-04-24T08:21:04.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 -- 172.21.15.69:0/2166182842 shutdown_connections 2024-04-24T08:21:04.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f809c06c8f0 0x7f809c06edb0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80c015c820 0x7f80c00cd000 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f80c00ce250 0x7f80c00ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 --2- 172.21.15.69:0/2166182842 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80c00b3e90 0x7f80c0156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 -- 172.21.15.69:0/2166182842 >> 172.21.15.69:0/2166182842 conn(0x7f80c00d0350 msgr2=0x7f80c00d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:04.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 -- 172.21.15.69:0/2166182842 shutdown_connections 2024-04-24T08:21:04.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.000+0000 7f80c7200640 1 -- 172.21.15.69:0/2166182842 wait complete. 2024-04-24T08:21:04.021 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:21:04.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.224+0000 7f9c59a00640 1 Processor -- start 2024-04-24T08:21:04.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.224+0000 7f9c59a00640 1 -- start start 2024-04-24T08:21:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.226+0000 7f9c59a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c540cf9a0 0x7f9c54162250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.226+0000 7f9c59a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c540710e0 0x7f9c54072af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.226+0000 7f9c59a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c54070860 con 0x7f9c54073030 2024-04-24T08:21:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.226+0000 7f9c59a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f9c5415ee90 con 0x7f9c540710e0 2024-04-24T08:21:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.226+0000 7f9c59a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9c54163180 con 0x7f9c540cf9a0 2024-04-24T08:21:04.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.226+0000 7f9c53400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f9c540710e0 0x7f9c54072af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.226+0000 7f9c59000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9c54073030 0x7f9c5415f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49602/0 (socket says 172.21.15.69:49602) 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.226+0000 7f9c59000640 1 -- 172.21.15.69:0/3645550113 learned_addr learned my addr 172.21.15.69:0/3645550113 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.226+0000 7f9c53e00640 1 --2- 172.21.15.69:0/3645550113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c540cf9a0 0x7f9c54162250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.227+0000 7f9c52a00640 1 -- 172.21.15.69:0/3645550113 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1700243946 0 0) 0x7f9c54070860 con 0x7f9c54073030 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.227+0000 7f9c52a00640 1 -- 172.21.15.69:0/3645550113 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c38003520 con 0x7f9c54073030 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.227+0000 7f9c53400640 1 -- 172.21.15.69:0/3645550113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c540cf9a0 msgr2=0x7f9c54162250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.227+0000 7f9c53400640 1 --2- 172.21.15.69:0/3645550113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c540cf9a0 0x7f9c54162250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.227+0000 7f9c53400640 1 -- 172.21.15.69:0/3645550113 >> v1:172.21.15.69:6789/0 conn(0x7f9c54073030 legacy=0x7f9c5415f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.227+0000 7f9c53400640 1 -- 172.21.15.69:0/3645550113 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c541628e0 con 0x7f9c540710e0 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.228+0000 7f9c53400640 1 --2- 172.21.15.69:0/3645550113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c540710e0 0x7f9c54072af0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f9c400097a0 tx=0x7f9c4002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f519604bd5a08784 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:04.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.228+0000 7f9c52a00640 1 -- 172.21.15.69:0/3645550113 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c40037070 con 0x7f9c540710e0 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.228+0000 7f9c52a00640 1 -- 172.21.15.69:0/3645550113 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c40032440 con 0x7f9c540710e0 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.228+0000 7f9c52a00640 1 -- 172.21.15.69:0/3645550113 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c40032810 con 0x7f9c540710e0 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.229+0000 7f9c59a00640 1 -- 172.21.15.69:0/3645550113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c540710e0 msgr2=0x7f9c54072af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.229+0000 7f9c59a00640 1 --2- 172.21.15.69:0/3645550113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c540710e0 0x7f9c54072af0 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f9c400097a0 tx=0x7f9c4002a180 comp rx=0 tx=0).stop 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.229+0000 7f9c59a00640 1 -- 172.21.15.69:0/3645550113 shutdown_connections 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.229+0000 7f9c59a00640 1 --2- 172.21.15.69:0/3645550113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c540710e0 0x7f9c54072af0 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.229+0000 7f9c59a00640 1 --2- 172.21.15.69:0/3645550113 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c540cf9a0 0x7f9c54162250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.229+0000 7f9c59a00640 1 -- 172.21.15.69:0/3645550113 >> 172.21.15.69:0/3645550113 conn(0x7f9c540d12c0 msgr2=0x7f9c540d16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.229+0000 7f9c59a00640 1 -- 172.21.15.69:0/3645550113 shutdown_connections 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.229+0000 7f9c59a00640 1 -- 172.21.15.69:0/3645550113 wait complete. 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.230+0000 7f9c59a00640 1 Processor -- start 2024-04-24T08:21:04.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.231+0000 7f9c59a00640 1 -- start start 2024-04-24T08:21:04.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.231+0000 7f9c59a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c540710e0 0x7f9c54158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.231+0000 7f9c59a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c54073030 0x7f9c541612b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.231+0000 7f9c59a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c540cf9a0 0x7f9c541516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c59a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c5415fa30 con 0x7f9c54073030 2024-04-24T08:21:04.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c59a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9c54162ea0 con 0x7f9c540710e0 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c59a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9c5415fbb0 con 0x7f9c540cf9a0 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c53400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c54073030 0x7f9c541612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c53400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c54073030 0x7f9c541612b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34132/0 (socket says 172.21.15.69:34132) 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c53400640 1 -- 172.21.15.69:0/3102445624 learned_addr learned my addr 172.21.15.69:0/3102445624 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c53400640 1 -- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c540cf9a0 msgr2=0x7f9c541516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c53400640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c540cf9a0 0x7f9c541516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c53400640 1 -- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c540710e0 msgr2=0x7f9c54158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c53400640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c540710e0 0x7f9c54158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c53400640 1 -- 172.21.15.69:0/3102445624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c540ca1a0 con 0x7f9c54073030 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.232+0000 7f9c53400640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c54073030 0x7f9c541612b0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f9c40000c00 tx=0x7f9c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.233+0000 7f9c4be00640 1 -- 172.21.15.69:0/3102445624 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c40041070 con 0x7f9c54073030 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.233+0000 7f9c4be00640 1 -- 172.21.15.69:0/3102445624 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c4003c870 con 0x7f9c54073030 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.233+0000 7f9c4be00640 1 -- 172.21.15.69:0/3102445624 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c40037040 con 0x7f9c54073030 2024-04-24T08:21:04.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.233+0000 7f9c59a00640 1 -- 172.21.15.69:0/3102445624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c540ca480 con 0x7f9c54073030 2024-04-24T08:21:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.233+0000 7f9c59a00640 1 -- 172.21.15.69:0/3102445624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c540caa40 con 0x7f9c54073030 2024-04-24T08:21:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.234+0000 7f9c59a00640 1 -- 172.21.15.69:0/3102445624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c540cac80 con 0x7f9c54073030 2024-04-24T08:21:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.235+0000 7f9c4be00640 1 -- 172.21.15.69:0/3102445624 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9c40044070 con 0x7f9c54073030 2024-04-24T08:21:04.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.236+0000 7f9c4be00640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9c3406c8f0 0x7f9c3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.236+0000 7f9c4be00640 1 -- 172.21.15.69:0/3102445624 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(111..111 src has 1..111) ==== 5853+0+0 (secure 0 0 0) 0x7f9c400ac7a0 con 0x7f9c54073030 2024-04-24T08:21:04.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.237+0000 7f9c53e00640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9c3406c8f0 0x7f9c3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.238+0000 7f9c53e00640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9c3406c8f0 0x7f9c3406edb0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9c44006fd0 tx=0x7f9c44008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:04.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.240+0000 7f9c4be00640 1 -- 172.21.15.69:0/3102445624 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9c4007abc0 con 0x7f9c54073030 2024-04-24T08:21:04.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.453+0000 7f9c59a00640 1 -- 172.21.15.69:0/3102445624 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9c540703a0 con 0x7f9c54073030 2024-04-24T08:21:04.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.454+0000 7f9c4be00640 1 -- 172.21.15.69:0/3102445624 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v111) ==== 80+0+124 (secure 0 0 0) 0x7f9c4007a4c0 con 0x7f9c54073030 2024-04-24T08:21:04.455 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6836/1515102940 2024-04-25T08:20:50.436176+0000 2024-04-24T08:21:04.455 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6834/1515102940 2024-04-25T08:20:50.436176+0000 2024-04-24T08:21:04.455 INFO:teuthology.orchestra.run.smithi069.stderr:listed 2 entries 2024-04-24T08:21:04.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 -- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9c3406c8f0 msgr2=0x7f9c3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9c3406c8f0 0x7f9c3406edb0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9c44006fd0 tx=0x7f9c44008040 comp rx=0 tx=0).stop 2024-04-24T08:21:04.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 -- 172.21.15.69:0/3102445624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c54073030 msgr2=0x7f9c541612b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c54073030 0x7f9c541612b0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f9c40000c00 tx=0x7f9c400046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 -- 172.21.15.69:0/3102445624 shutdown_connections 2024-04-24T08:21:04.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9c3406c8f0 0x7f9c3406edb0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c54073030 0x7f9c541612b0 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9c540cf9a0 0x7f9c541516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 --2- 172.21.15.69:0/3102445624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c540710e0 0x7f9c54158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 -- 172.21.15.69:0/3102445624 >> 172.21.15.69:0/3102445624 conn(0x7f9c540d12c0 msgr2=0x7f9c540cd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:04.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 -- 172.21.15.69:0/3102445624 shutdown_connections 2024-04-24T08:21:04.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:04.457+0000 7f9c59a00640 1 -- 172.21.15.69:0/3102445624 wait complete. 2024-04-24T08:21:04.477 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:21:04.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:04.477 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:04.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[77799]: fuse finished with error 0 and tester_r 0 2024-04-24T08:21:04.562 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:21:10.663 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:21:10.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:10.664 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:21:10.720 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:10.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.914+0000 7f79f9200640 1 Processor -- start 2024-04-24T08:21:10.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.914+0000 7f79f9200640 1 -- start start 2024-04-24T08:21:10.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.915+0000 7f79f9200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f79f41523f0 0x7f79f406db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.915+0000 7f79f9200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79f4072e20 0x7f79f4073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.915+0000 7f79f9200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f40737a0 0x7f79f415ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.915+0000 7f79f9200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f79f40aeac0 con 0x7f79f4072e20 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.915+0000 7f79f9200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f79f40aeda0 con 0x7f79f41523f0 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.915+0000 7f79f9200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f79f40af080 con 0x7f79f40737a0 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.916+0000 7f79ebe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79f4072e20 0x7f79f4073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.916+0000 7f79ebe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f79f4072e20 0x7f79f4073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52930/0 (socket says 172.21.15.69:52930) 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.916+0000 7f79ebe00640 1 -- 172.21.15.69:0/1726012852 learned_addr learned my addr 172.21.15.69:0/1726012852 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.916+0000 7f79f3e00640 1 --2- 172.21.15.69:0/1726012852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f40737a0 0x7f79f415ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.916+0000 7f79f3400640 1 --2- 172.21.15.69:0/1726012852 >> v2:172.21.15.121:3300/0 conn(0x7f79f41523f0 0x7f79f406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.916+0000 7f79ebe00640 1 -- 172.21.15.69:0/1726012852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f40737a0 msgr2=0x7f79f415ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.916+0000 7f79ebe00640 1 --2- 172.21.15.69:0/1726012852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f40737a0 0x7f79f415ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.916+0000 7f79ebe00640 1 -- 172.21.15.69:0/1726012852 >> v2:172.21.15.121:3300/0 conn(0x7f79f41523f0 msgr2=0x7f79f406db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.916+0000 7f79ebe00640 1 --2- 172.21.15.69:0/1726012852 >> v2:172.21.15.121:3300/0 conn(0x7f79f41523f0 0x7f79f406db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.916+0000 7f79ebe00640 1 -- 172.21.15.69:0/1726012852 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f40af360 con 0x7f79f4072e20 2024-04-24T08:21:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.917+0000 7f79ebe00640 1 --2- 172.21.15.69:0/1726012852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 0x7f79f4073260 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f79f406cb50 tx=0x7f79dc02c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76999232de361da4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.918+0000 7f79f2a00640 1 -- 172.21.15.69:0/1726012852 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc032700 con 0x7f79f4072e20 2024-04-24T08:21:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.918+0000 7f79f2a00640 1 -- 172.21.15.69:0/1726012852 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79dc0328a0 con 0x7f79f4072e20 2024-04-24T08:21:10.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.918+0000 7f79f2a00640 1 -- 172.21.15.69:0/1726012852 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79dc032c70 con 0x7f79f4072e20 2024-04-24T08:21:10.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.918+0000 7f79f9200640 1 -- 172.21.15.69:0/1726012852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 msgr2=0x7f79f4073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:10.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.918+0000 7f79f9200640 1 --2- 172.21.15.69:0/1726012852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 0x7f79f4073260 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f79f406cb50 tx=0x7f79dc02c800 comp rx=0 tx=0).stop 2024-04-24T08:21:10.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.918+0000 7f79f9200640 1 -- 172.21.15.69:0/1726012852 shutdown_connections 2024-04-24T08:21:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.918+0000 7f79f9200640 1 --2- 172.21.15.69:0/1726012852 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f40737a0 0x7f79f415ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.918+0000 7f79f9200640 1 --2- 172.21.15.69:0/1726012852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 0x7f79f4073260 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.918+0000 7f79f9200640 1 --2- 172.21.15.69:0/1726012852 >> v2:172.21.15.121:3300/0 conn(0x7f79f41523f0 0x7f79f406db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.918+0000 7f79f9200640 1 -- 172.21.15.69:0/1726012852 >> 172.21.15.69:0/1726012852 conn(0x7f79f40d0160 msgr2=0x7f79f40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.919+0000 7f79f9200640 1 -- 172.21.15.69:0/1726012852 shutdown_connections 2024-04-24T08:21:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.919+0000 7f79f9200640 1 -- 172.21.15.69:0/1726012852 wait complete. 2024-04-24T08:21:10.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.920+0000 7f79f9200640 1 Processor -- start 2024-04-24T08:21:10.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.920+0000 7f79f9200640 1 -- start start 2024-04-24T08:21:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.921+0000 7f79f9200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 0x7f79f414f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.921+0000 7f79f9200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f40737a0 0x7f79f40b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.921+0000 7f79f9200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79f41523f0 0x7f79f415a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.921+0000 7f79f9200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f79f415f4b0 con 0x7f79f4072e20 2024-04-24T08:21:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.921+0000 7f79f9200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f79f41527d0 con 0x7f79f41523f0 2024-04-24T08:21:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.921+0000 7f79f9200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f79f415f630 con 0x7f79f40737a0 2024-04-24T08:21:10.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79f3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 0x7f79f414f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79f3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 0x7f79f414f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52946/0 (socket says 172.21.15.69:52946) 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79f3400640 1 -- 172.21.15.69:0/3515468439 learned_addr learned my addr 172.21.15.69:0/3515468439 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79ebe00640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f40737a0 0x7f79f40b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79f3400640 1 -- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f40737a0 msgr2=0x7f79f40b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79f3400640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f40737a0 0x7f79f40b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79f3400640 1 -- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79f41523f0 msgr2=0x7f79f415a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79f3400640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79f41523f0 0x7f79f415a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79f3400640 1 -- 172.21.15.69:0/3515468439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f41f43e0 con 0x7f79f4072e20 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79f3400640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 0x7f79f414f4c0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f79e400d5f0 tx=0x7f79e400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.922+0000 7f79eb400640 1 -- 172.21.15.69:0/3515468439 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e40125f0 con 0x7f79f4072e20 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.923+0000 7f79eb400640 1 -- 172.21.15.69:0/3515468439 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e4019070 con 0x7f79f4072e20 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.923+0000 7f79eb400640 1 -- 172.21.15.69:0/3515468439 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e40129d0 con 0x7f79f4072e20 2024-04-24T08:21:10.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.923+0000 7f79f9200640 1 -- 172.21.15.69:0/3515468439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f41f46c0 con 0x7f79f4072e20 2024-04-24T08:21:10.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.923+0000 7f79f9200640 1 -- 172.21.15.69:0/3515468439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f41f4c80 con 0x7f79f4072e20 2024-04-24T08:21:10.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.924+0000 7f79f9200640 1 -- 172.21.15.69:0/3515468439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f4075d50 con 0x7f79f4072e20 2024-04-24T08:21:10.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.925+0000 7f79eb400640 1 -- 172.21.15.69:0/3515468439 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f79e4012b70 con 0x7f79f4072e20 2024-04-24T08:21:10.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.926+0000 7f79eb400640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79d006c920 0x7f79d006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:10.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.926+0000 7f79eb400640 1 -- 172.21.15.69:0/3515468439 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f79e408bfc0 con 0x7f79f4072e20 2024-04-24T08:21:10.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.926+0000 7f79ebe00640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79d006c920 0x7f79d006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:10.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.928+0000 7f79ebe00640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79d006c920 0x7f79d006ede0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f79dc037000 tx=0x7f79dc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:10.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:10.931+0000 7f79eb400640 1 -- 172.21.15.69:0/3515468439 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79e405a360 con 0x7f79f4072e20 2024-04-24T08:21:11.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.168+0000 7f79f9200640 1 -- 172.21.15.69:0/3515468439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79f4157f20 con 0x7f79f4072e20 2024-04-24T08:21:11.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.170+0000 7f79eb400640 1 -- 172.21.15.69:0/3515468439 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+5193 (secure 0 0 0) 0x7f79e4059c60 con 0x7f79f4072e20 2024-04-24T08:21:11.170 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:11.171 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:20:54.481212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":110,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9285":{"gid":9285,"name":"c","rank":0,"incarnation":102,"state":"up:active","state_seq":15,"addr":"172.21.15.69:6835/2194021008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":2194021008},{"type":"v1","addr":"172.21.15.69:6835","nonce":2194021008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9285,"qdb_cluster":[9285]},"id":8}]} 2024-04-24T08:21:11.171 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 105 2024-04-24T08:21:11.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.172+0000 7f79f9200640 1 -- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79d006c920 msgr2=0x7f79d006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.172+0000 7f79f9200640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79d006c920 0x7f79d006ede0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f79dc037000 tx=0x7f79dc008040 comp rx=0 tx=0).stop 2024-04-24T08:21:11.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.172+0000 7f79f9200640 1 -- 172.21.15.69:0/3515468439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 msgr2=0x7f79f414f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.172+0000 7f79f9200640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 0x7f79f414f4c0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f79e400d5f0 tx=0x7f79e400dab0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.172+0000 7f79f9200640 1 -- 172.21.15.69:0/3515468439 shutdown_connections 2024-04-24T08:21:11.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.172+0000 7f79f9200640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f79d006c920 0x7f79d006ede0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.172+0000 7f79f9200640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79f41523f0 0x7f79f415a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.172+0000 7f79f9200640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f79f40737a0 0x7f79f40b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.172+0000 7f79f9200640 1 --2- 172.21.15.69:0/3515468439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f79f4072e20 0x7f79f414f4c0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.172+0000 7f79f9200640 1 -- 172.21.15.69:0/3515468439 >> 172.21.15.69:0/3515468439 conn(0x7f79f40d0160 msgr2=0x7f79f4150580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.173+0000 7f79f9200640 1 -- 172.21.15.69:0/3515468439 shutdown_connections 2024-04-24T08:21:11.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.173+0000 7f79f9200640 1 -- 172.21.15.69:0/3515468439 wait complete. 2024-04-24T08:21:11.192 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json tag path /parent mytag123 2024-04-24T08:21:11.418 INFO:teuthology.orchestra.run.smithi069.stdout:{"return_code":0,"scrub_tag":"mytag123","mode":"asynchronous"} 2024-04-24T08:21:11.418 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag123" } 2024-04-24T08:21:11.418 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-04-24T08:21:11.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.520+0000 7fc94151df40 1 Processor -- start 2024-04-24T08:21:11.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.520+0000 7fc94151df40 1 -- start start 2024-04-24T08:21:11.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.520+0000 7fc94151df40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de1fb435a0 0x55de1fc1c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.520+0000 7fc94151df40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de1fb8e940 0x55de1fc09e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc94151df40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc1d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc94151df40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55de1fb74180 con 0x55de1fc14f50 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc94151df40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55de1faeacc0 con 0x55de1fb8e940 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc94151df40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55de1fc1bd60 con 0x55de1fb435a0 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc1d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc1d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52968/0 (socket says 172.21.15.69:52968) 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93e600640 1 -- 172.21.15.69:0/2299061883 learned_addr learned my addr 172.21.15.69:0/2299061883 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93dc00640 1 --2- 172.21.15.69:0/2299061883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de1fb435a0 0x55de1fc1c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93e600640 1 -- 172.21.15.69:0/2299061883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de1fb435a0 msgr2=0x55de1fc1c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93d200640 1 --2- 172.21.15.69:0/2299061883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de1fb8e940 0x55de1fc09e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93e600640 1 --2- 172.21.15.69:0/2299061883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de1fb435a0 0x55de1fc1c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93e600640 1 -- 172.21.15.69:0/2299061883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de1fb8e940 msgr2=0x55de1fc09e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93e600640 1 --2- 172.21.15.69:0/2299061883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de1fb8e940 0x55de1fc09e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93e600640 1 -- 172.21.15.69:0/2299061883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de1fc13290 con 0x55de1fc14f50 2024-04-24T08:21:11.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.521+0000 7fc93dc00640 1 --2- 172.21.15.69:0/2299061883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de1fb435a0 0x55de1fc1c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:11.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.522+0000 7fc93e600640 1 --2- 172.21.15.69:0/2299061883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc1d2d0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fc93800a4b0 tx=0x7fc93800a970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7290c41db23586c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.522+0000 7fc937400640 1 -- 172.21.15.69:0/2299061883 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc938015070 con 0x55de1fc14f50 2024-04-24T08:21:11.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.522+0000 7fc937400640 1 -- 172.21.15.69:0/2299061883 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc93800e730 con 0x55de1fc14f50 2024-04-24T08:21:11.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.522+0000 7fc937400640 1 -- 172.21.15.69:0/2299061883 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93800eb20 con 0x55de1fc14f50 2024-04-24T08:21:11.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.522+0000 7fc94151df40 1 -- 172.21.15.69:0/2299061883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 msgr2=0x55de1fc1d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.522+0000 7fc94151df40 1 --2- 172.21.15.69:0/2299061883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc1d2d0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fc93800a4b0 tx=0x7fc93800a970 comp rx=0 tx=0).stop 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.523+0000 7fc94151df40 1 -- 172.21.15.69:0/2299061883 shutdown_connections 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.523+0000 7fc94151df40 1 --2- 172.21.15.69:0/2299061883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc1d2d0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.523+0000 7fc94151df40 1 --2- 172.21.15.69:0/2299061883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de1fb8e940 0x55de1fc09e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.523+0000 7fc94151df40 1 --2- 172.21.15.69:0/2299061883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de1fb435a0 0x55de1fc1c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.523+0000 7fc94151df40 1 -- 172.21.15.69:0/2299061883 >> 172.21.15.69:0/2299061883 conn(0x55de1fb72b50 msgr2=0x55de1fb72360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.523+0000 7fc94151df40 1 -- 172.21.15.69:0/2299061883 shutdown_connections 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.523+0000 7fc94151df40 1 -- 172.21.15.69:0/2299061883 wait complete. 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.523+0000 7fc94151df40 1 Processor -- start 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.524+0000 7fc94151df40 1 -- start start 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.524+0000 7fc94151df40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de1fb435a0 0x55de1fcab8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.524+0000 7fc94151df40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de1fb8e940 0x55de1fb8faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.524+0000 7fc94151df40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc0e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.524+0000 7fc94151df40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55de1fb906e0 con 0x55de1fc14f50 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.524+0000 7fc94151df40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55de1fc1bf40 con 0x55de1fb435a0 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.524+0000 7fc94151df40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55de1fb909c0 con 0x55de1fb8e940 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.525+0000 7fc93e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc0e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.525+0000 7fc93e600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc0e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52980/0 (socket says 172.21.15.69:52980) 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.525+0000 7fc93e600640 1 -- 172.21.15.69:0/2828027348 learned_addr learned my addr 172.21.15.69:0/2828027348 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.525+0000 7fc93d200640 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de1fb8e940 0x55de1fb8faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.525+0000 7fc93e600640 1 -- 172.21.15.69:0/2828027348 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de1fb8e940 msgr2=0x55de1fb8faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.525+0000 7fc93e600640 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de1fb8e940 0x55de1fb8faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.525+0000 7fc93e600640 1 -- 172.21.15.69:0/2828027348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de1fb435a0 msgr2=0x55de1fcab8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.525+0000 7fc93e600640 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de1fb435a0 0x55de1fcab8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.525+0000 7fc93e600640 1 -- 172.21.15.69:0/2828027348 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de1fcabe00 con 0x55de1fc14f50 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.526+0000 7fc93e600640 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc0e280 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fc93800a9b0 tx=0x7fc938003350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.526+0000 7fc936000640 1 -- 172.21.15.69:0/2828027348 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc93801f030 con 0x55de1fc14f50 2024-04-24T08:21:11.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.526+0000 7fc936000640 1 -- 172.21.15.69:0/2828027348 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc938010480 con 0x55de1fc14f50 2024-04-24T08:21:11.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.526+0000 7fc936000640 1 -- 172.21.15.69:0/2828027348 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc938015040 con 0x55de1fc14f50 2024-04-24T08:21:11.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.526+0000 7fc94151df40 1 -- 172.21.15.69:0/2828027348 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55de1fc12d10 con 0x55de1fc14f50 2024-04-24T08:21:11.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.526+0000 7fc94151df40 1 -- 172.21.15.69:0/2828027348 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de1fc067d0 con 0x55de1fc14f50 2024-04-24T08:21:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.528+0000 7fc936000640 1 -- 172.21.15.69:0/2828027348 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc938022030 con 0x55de1fc14f50 2024-04-24T08:21:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.529+0000 7fc936000640 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc91c06b2d0 0x7fc91c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.529+0000 7fc936000640 1 -- 172.21.15.69:0/2828027348 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fc93808b740 con 0x55de1fc14f50 2024-04-24T08:21:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.529+0000 7fc94151df40 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55de1fb8ffe0 0x55de1fcb4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.529+0000 7fc94151df40 1 -- 172.21.15.69:0/2828027348 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:1 17.36 17:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) -- 0x55de1fb8ed70 con 0x55de1fb8ffe0 2024-04-24T08:21:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.530+0000 7fc93d200640 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55de1fb8ffe0 0x55de1fcb4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.530+0000 7fc93dc00640 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc91c06b2d0 0x7fc91c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.530+0000 7fc93d200640 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55de1fb8ffe0 0x55de1fcb4c80 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc93d200640 1 -- 172.21.15.69:0/2828027348 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) ==== 164+0+0 (crc 0 0 0) 0x7fc928007720 con 0x55de1fb8ffe0 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:error getting xattr cephfs_data/100000001f7.00000000/scrub_tag: (61) No data available 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc94151df40 1 -- 172.21.15.69:0/2828027348 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55de1fb8ffe0 msgr2=0x55de1fcb4c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc94151df40 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55de1fb8ffe0 0x55de1fcb4c80 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc94151df40 1 -- 172.21.15.69:0/2828027348 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc91c06b2d0 msgr2=0x7fc91c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc94151df40 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc91c06b2d0 0x7fc91c06d790 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fc930057000 tx=0x7fc930051000 comp rx=0 tx=0).stop 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc94151df40 1 -- 172.21.15.69:0/2828027348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 msgr2=0x55de1fc0e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc94151df40 1 --2- 172.21.15.69:0/2828027348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de1fc14f50 0x55de1fc0e280 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fc93800a9b0 tx=0x7fc938003350 comp rx=0 tx=0).stop 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc93e600640 1 -- 172.21.15.69:0/2828027348 reap_dead start 2024-04-24T08:21:11.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc94151df40 1 -- 172.21.15.69:0/2828027348 shutdown_connections 2024-04-24T08:21:11.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc94151df40 1 -- 172.21.15.69:0/2828027348 >> 172.21.15.69:0/2828027348 conn(0x55de1fb72b50 msgr2=0x55de1fb76360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc94151df40 1 -- 172.21.15.69:0/2828027348 shutdown_connections 2024-04-24T08:21:11.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.531+0000 7fc94151df40 1 -- 172.21.15.69:0/2828027348 wait complete. 2024-04-24T08:21:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:21:11.536 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-04-24T08:21:11.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.634+0000 7fd0f840df40 1 Processor -- start 2024-04-24T08:21:11.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.634+0000 7fd0f840df40 1 -- start start 2024-04-24T08:21:11.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.635+0000 7fd0f840df40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3945a0 0x55ca6c3c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.635+0000 7fd0f840df40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ca6c3df940 0x55ca6c392680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.635+0000 7fd0f840df40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ca6c465f50 0x55ca6c3c2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.635+0000 7fd0f840df40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ca6c46cd60 con 0x55ca6c3945a0 2024-04-24T08:21:11.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.635+0000 7fd0f840df40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ca6c3c5180 con 0x55ca6c3df940 2024-04-24T08:21:11.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.635+0000 7fd0f840df40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ca6c3949b0 con 0x55ca6c465f50 2024-04-24T08:21:11.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.635+0000 7fd0f4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3945a0 0x55ca6c3c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.635+0000 7fd0f4a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3945a0 0x55ca6c3c3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53002/0 (socket says 172.21.15.69:53002) 2024-04-24T08:21:11.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.635+0000 7fd0f4a00640 1 -- 172.21.15.69:0/3564745841 learned_addr learned my addr 172.21.15.69:0/3564745841 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:11.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.636+0000 7fd0f4a00640 1 -- 172.21.15.69:0/3564745841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ca6c465f50 msgr2=0x55ca6c3c2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:11.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.636+0000 7fd0f4a00640 1 --2- 172.21.15.69:0/3564745841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ca6c465f50 0x55ca6c3c2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.636+0000 7fd0f4a00640 1 -- 172.21.15.69:0/3564745841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ca6c3df940 msgr2=0x55ca6c392680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:11.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.636+0000 7fd0f4000640 1 --2- 172.21.15.69:0/3564745841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ca6c3df940 0x55ca6c392680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.636+0000 7fd0f4a00640 1 --2- 172.21.15.69:0/3564745841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ca6c3df940 0x55ca6c392680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.636+0000 7fd0f4a00640 1 -- 172.21.15.69:0/3564745841 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca6c2bfb30 con 0x55ca6c3945a0 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.636+0000 7fd0f4a00640 1 --2- 172.21.15.69:0/3564745841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3945a0 0x55ca6c3c3f30 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40095d0 tx=0x7fd0e4009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a328bf4f03d3d88 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.636+0000 7fd0f2c00640 1 -- 172.21.15.69:0/3564745841 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4017070 con 0x55ca6c3945a0 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.636+0000 7fd0f2c00640 1 -- 172.21.15.69:0/3564745841 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e4006680 con 0x55ca6c3945a0 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.636+0000 7fd0f2c00640 1 -- 172.21.15.69:0/3564745841 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4006a30 con 0x55ca6c3945a0 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.637+0000 7fd0f840df40 1 -- 172.21.15.69:0/3564745841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3945a0 msgr2=0x55ca6c3c3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.637+0000 7fd0f840df40 1 --2- 172.21.15.69:0/3564745841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3945a0 0x55ca6c3c3f30 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40095d0 tx=0x7fd0e4009a90 comp rx=0 tx=0).stop 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.637+0000 7fd0f840df40 1 -- 172.21.15.69:0/3564745841 shutdown_connections 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.637+0000 7fd0f840df40 1 --2- 172.21.15.69:0/3564745841 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ca6c465f50 0x55ca6c3c2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.637+0000 7fd0f840df40 1 --2- 172.21.15.69:0/3564745841 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ca6c3df940 0x55ca6c392680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.637+0000 7fd0f840df40 1 --2- 172.21.15.69:0/3564745841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3945a0 0x55ca6c3c3f30 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.637+0000 7fd0f840df40 1 -- 172.21.15.69:0/3564745841 >> 172.21.15.69:0/3564745841 conn(0x55ca6c3c3b50 msgr2=0x55ca6c3c1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.637+0000 7fd0f840df40 1 -- 172.21.15.69:0/3564745841 shutdown_connections 2024-04-24T08:21:11.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.637+0000 7fd0f840df40 1 -- 172.21.15.69:0/3564745841 wait complete. 2024-04-24T08:21:11.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.638+0000 7fd0f840df40 1 Processor -- start 2024-04-24T08:21:11.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.638+0000 7fd0f840df40 1 -- start start 2024-04-24T08:21:11.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f840df40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ca6c3945a0 0x55ca6c46ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f840df40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3df940 0x55ca6c45dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f840df40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ca6c465f50 0x55ca6c4597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f840df40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ca6c46d870 con 0x55ca6c3df940 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f840df40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55ca6c3c4470 con 0x55ca6c3945a0 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f840df40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55ca6c462b10 con 0x55ca6c465f50 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f4000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3df940 0x55ca6c45dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f4000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3df940 0x55ca6c45dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53010/0 (socket says 172.21.15.69:53010) 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f4000640 1 -- 172.21.15.69:0/1427960282 learned_addr learned my addr 172.21.15.69:0/1427960282 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f5400640 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ca6c465f50 0x55ca6c4597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f4000640 1 -- 172.21.15.69:0/1427960282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ca6c465f50 msgr2=0x55ca6c4597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f4000640 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55ca6c465f50 0x55ca6c4597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f4000640 1 -- 172.21.15.69:0/1427960282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ca6c3945a0 msgr2=0x55ca6c46ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f4000640 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ca6c3945a0 0x55ca6c46ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.639+0000 7fd0f4000640 1 -- 172.21.15.69:0/1427960282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca6c3c2df0 con 0x55ca6c3df940 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.640+0000 7fd0f4000640 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3df940 0x55ca6c45dc00 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000be30 tx=0x7fd0e000bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.640+0000 7fd0f1800640 1 -- 172.21.15.69:0/1427960282 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0019070 con 0x55ca6c3df940 2024-04-24T08:21:11.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.640+0000 7fd0f1800640 1 -- 172.21.15.69:0/1427960282 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e0007b50 con 0x55ca6c3df940 2024-04-24T08:21:11.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.640+0000 7fd0f1800640 1 -- 172.21.15.69:0/1427960282 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0014410 con 0x55ca6c3df940 2024-04-24T08:21:11.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.640+0000 7fd0f840df40 1 -- 172.21.15.69:0/1427960282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55ca6c45a520 con 0x55ca6c3df940 2024-04-24T08:21:11.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.640+0000 7fd0f840df40 1 -- 172.21.15.69:0/1427960282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca6c462850 con 0x55ca6c3df940 2024-04-24T08:21:11.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.642+0000 7fd0f1800640 1 -- 172.21.15.69:0/1427960282 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd0e001c070 con 0x55ca6c3df940 2024-04-24T08:21:11.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.643+0000 7fd0f1800640 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0d406b2d0 0x7fd0d406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.643+0000 7fd0f1800640 1 -- 172.21.15.69:0/1427960282 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fd0e0089920 con 0x55ca6c3df940 2024-04-24T08:21:11.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.643+0000 7fd0f840df40 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55ca6c45e140 0x55ca6c460110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.643+0000 7fd0f840df40 1 -- 172.21.15.69:0/1427960282 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:1 17.12 17:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) -- 0x55ca6c3c4da0 con 0x55ca6c45e140 2024-04-24T08:21:11.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.644+0000 7fd0f5400640 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55ca6c45e140 0x55ca6c460110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.644+0000 7fd0f4a00640 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0d406b2d0 0x7fd0d406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.644+0000 7fd0f5400640 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55ca6c45e140 0x55ca6c460110 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f5400640 1 -- 172.21.15.69:0/1427960282 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr (12) out=12b] v0'0 uv22 ondisk = 0) ==== 164+0+12 (crc 0 0 0) 0x7fd0ec051860 con 0x55ca6c45e140 2024-04-24T08:21:11.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f840df40 1 -- 172.21.15.69:0/1427960282 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55ca6c45e140 msgr2=0x55ca6c460110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f840df40 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55ca6c45e140 0x55ca6c460110 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f840df40 1 -- 172.21.15.69:0/1427960282 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0d406b2d0 msgr2=0x7fd0d406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f840df40 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0d406b2d0 0x7fd0d406d790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4009ad0 tx=0x7fd0e400e000 comp rx=0 tx=0).stop 2024-04-24T08:21:11.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f840df40 1 -- 172.21.15.69:0/1427960282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3df940 msgr2=0x55ca6c45dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f840df40 1 --2- 172.21.15.69:0/1427960282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ca6c3df940 0x55ca6c45dc00 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fd0e000be30 tx=0x7fd0e000bf20 comp rx=0 tx=0).stop 2024-04-24T08:21:11.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f5400640 1 -- 172.21.15.69:0/1427960282 reap_dead start 2024-04-24T08:21:11.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f840df40 1 -- 172.21.15.69:0/1427960282 shutdown_connections 2024-04-24T08:21:11.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f840df40 1 -- 172.21.15.69:0/1427960282 >> 172.21.15.69:0/1427960282 conn(0x55ca6c3c3b50 msgr2=0x55ca6c3c1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f840df40 1 -- 172.21.15.69:0/1427960282 shutdown_connections 2024-04-24T08:21:11.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.645+0000 7fd0f840df40 1 -- 172.21.15.69:0/1427960282 wait complete. 2024-04-24T08:21:11.650 INFO:teuthology.orchestra.run.smithi069.stdout:mytag123 2024-04-24T08:21:11.651 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-04-24T08:21:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.735+0000 7fc95f8ebf40 1 Processor -- start 2024-04-24T08:21:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.735+0000 7fc95f8ebf40 1 -- start start 2024-04-24T08:21:11.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc95f8ebf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562b18ab35a0 0x562b18b7dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc95f8ebf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18afe940 0x562b18ae1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc95f8ebf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b18b84f50 0x562b18b7ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc95f8ebf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562b18b8bd60 con 0x562b18afe940 2024-04-24T08:21:11.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc95f8ebf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562b18ae4180 con 0x562b18b84f50 2024-04-24T08:21:11.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc95f8ebf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x562b18ab39b0 con 0x562b18ab35a0 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc954000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18afe940 0x562b18ae1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc954000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18afe940 0x562b18ae1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53020/0 (socket says 172.21.15.69:53020) 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc954000640 1 -- 172.21.15.69:0/2507287396 learned_addr learned my addr 172.21.15.69:0/2507287396 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc954000640 1 -- 172.21.15.69:0/2507287396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562b18ab35a0 msgr2=0x562b18b7dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc954000640 1 --2- 172.21.15.69:0/2507287396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562b18ab35a0 0x562b18b7dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc954000640 1 -- 172.21.15.69:0/2507287396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b18b84f50 msgr2=0x562b18b7ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc954000640 1 --2- 172.21.15.69:0/2507287396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b18b84f50 0x562b18b7ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.736+0000 7fc954000640 1 -- 172.21.15.69:0/2507287396 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b189deb30 con 0x562b18afe940 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc954000640 1 --2- 172.21.15.69:0/2507287396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18afe940 0x562b18ae1cb0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fc944009760 tx=0x7fc944009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b40cdb7aa009e6c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc957400640 1 -- 172.21.15.69:0/2507287396 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944017070 con 0x562b18afe940 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc957400640 1 -- 172.21.15.69:0/2507287396 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9440090d0 con 0x562b18afe940 2024-04-24T08:21:11.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc957400640 1 -- 172.21.15.69:0/2507287396 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc944006b60 con 0x562b18afe940 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/2507287396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18afe940 msgr2=0x562b18ae1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc95f8ebf40 1 --2- 172.21.15.69:0/2507287396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18afe940 0x562b18ae1cb0 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fc944009760 tx=0x7fc944009c20 comp rx=0 tx=0).stop 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/2507287396 shutdown_connections 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc95f8ebf40 1 --2- 172.21.15.69:0/2507287396 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18afe940 0x562b18ae1cb0 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc95f8ebf40 1 --2- 172.21.15.69:0/2507287396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b18b84f50 0x562b18b7ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc95f8ebf40 1 --2- 172.21.15.69:0/2507287396 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562b18ab35a0 0x562b18b7dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/2507287396 >> 172.21.15.69:0/2507287396 conn(0x562b18ae2b50 msgr2=0x562b18ae0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/2507287396 shutdown_connections 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.737+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/2507287396 wait complete. 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.738+0000 7fc95f8ebf40 1 Processor -- start 2024-04-24T08:21:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.738+0000 7fc95f8ebf40 1 -- start start 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.739+0000 7fc95f8ebf40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b18ab35a0 0x562b18b7d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.739+0000 7fc95f8ebf40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562b18afe940 0x562b18b88a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.739+0000 7fc95f8ebf40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18b84f50 0x562b18b86390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.739+0000 7fc95f8ebf40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562b18b7e290 con 0x562b18b84f50 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.739+0000 7fc95f8ebf40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x562b18b8c7e0 con 0x562b18ab35a0 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.739+0000 7fc95f8ebf40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x562b18b8a670 con 0x562b18afe940 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.739+0000 7fc95d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18b84f50 0x562b18b86390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.739+0000 7fc95d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18b84f50 0x562b18b86390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53028/0 (socket says 172.21.15.69:53028) 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.739+0000 7fc95d400640 1 -- 172.21.15.69:0/287918808 learned_addr learned my addr 172.21.15.69:0/287918808 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc95ca00640 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b18ab35a0 0x562b18b7d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc95d400640 1 -- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562b18afe940 msgr2=0x562b18b88a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc95d400640 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x562b18afe940 0x562b18b88a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc95d400640 1 -- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b18ab35a0 msgr2=0x562b18b7d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc95d400640 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x562b18ab35a0 0x562b18b7d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc95d400640 1 -- 172.21.15.69:0/287918808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b18b89510 con 0x562b18b84f50 2024-04-24T08:21:11.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc95d400640 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18b84f50 0x562b18b86390 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fc958009ac0 tx=0x7fc958009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc956000640 1 -- 172.21.15.69:0/287918808 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc958017070 con 0x562b18b84f50 2024-04-24T08:21:11.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc956000640 1 -- 172.21.15.69:0/287918808 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc958006b40 con 0x562b18b84f50 2024-04-24T08:21:11.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/287918808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x562b18b897f0 con 0x562b18b84f50 2024-04-24T08:21:11.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc956000640 1 -- 172.21.15.69:0/287918808 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc958012410 con 0x562b18b84f50 2024-04-24T08:21:11.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.740+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/287918808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b18b7c630 con 0x562b18b84f50 2024-04-24T08:21:11.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.742+0000 7fc956000640 1 -- 172.21.15.69:0/287918808 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc95801a070 con 0x562b18b84f50 2024-04-24T08:21:11.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.743+0000 7fc956000640 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93806b300 0x7fc93806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.743+0000 7fc956000640 1 -- 172.21.15.69:0/287918808 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7fc9580897a0 con 0x562b18b84f50 2024-04-24T08:21:11.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.743+0000 7fc95f8ebf40 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x562b18b7e670 0x562b18b868d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.744+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/287918808 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:1 17.19 17:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) -- 0x562b18b88f70 con 0x562b18b7e670 2024-04-24T08:21:11.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.744+0000 7fc95ca00640 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93806b300 0x7fc93806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.744+0000 7fc954000640 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x562b18b7e670 0x562b18b868d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.745+0000 7fc954000640 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x562b18b7e670 0x562b18b868d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95ca00640 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93806b300 0x7fc93806d7c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc94c056000 tx=0x7fc94c050000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc954000640 1 -- 172.21.15.69:0/287918808 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (12) out=12b] v0'0 uv23 ondisk = 0) ==== 164+0+12 (crc 0 0 0) 0x7fc944006960 con 0x562b18b7e670 2024-04-24T08:21:11.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x562b18b7e670 msgr2=0x562b18b868d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95f8ebf40 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x562b18b7e670 0x562b18b868d0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93806b300 msgr2=0x7fc93806d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95f8ebf40 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc93806b300 0x7fc93806d7c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fc94c056000 tx=0x7fc94c050000 comp rx=0 tx=0).stop 2024-04-24T08:21:11.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/287918808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18b84f50 msgr2=0x562b18b86390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95f8ebf40 1 --2- 172.21.15.69:0/287918808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562b18b84f50 0x562b18b86390 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fc958009ac0 tx=0x7fc958009f80 comp rx=0 tx=0).stop 2024-04-24T08:21:11.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95d400640 1 -- 172.21.15.69:0/287918808 reap_dead start 2024-04-24T08:21:11.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/287918808 shutdown_connections 2024-04-24T08:21:11.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/287918808 >> 172.21.15.69:0/287918808 conn(0x562b18ae2b50 msgr2=0x562b18ae0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.746+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/287918808 shutdown_connections 2024-04-24T08:21:11.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.747+0000 7fc95f8ebf40 1 -- 172.21.15.69:0/287918808 wait complete. 2024-04-24T08:21:11.751 INFO:teuthology.orchestra.run.smithi069.stdout:mytag123 2024-04-24T08:21:11.751 DEBUG:teuthology.orchestra.run.smithi069:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000004.00000000 scrub_tag 2024-04-24T08:21:11.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.831+0000 7f043d480f40 1 Processor -- start 2024-04-24T08:21:11.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.831+0000 7f043d480f40 1 -- start start 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.832+0000 7f043d480f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.832+0000 7f043d480f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563fb72b0940 0x563fb73303b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.832+0000 7f043d480f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x563fb7336f50 0x563fb732c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.832+0000 7f043d480f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563fb7296180 con 0x563fb7265770 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.832+0000 7f043d480f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x563fb720ccc0 con 0x563fb72b0940 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.832+0000 7f043d480f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x563fb733dd60 con 0x563fb7336f50 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.832+0000 7f043a600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.832+0000 7f043a600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53034/0 (socket says 172.21.15.69:53034) 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.832+0000 7f043a600640 1 -- 172.21.15.69:0/3074115439 learned_addr learned my addr 172.21.15.69:0/3074115439 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f043b000640 1 --2- 172.21.15.69:0/3074115439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x563fb7336f50 0x563fb732c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f043a600640 1 -- 172.21.15.69:0/3074115439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x563fb7336f50 msgr2=0x563fb732c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:11.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f043a600640 1 --2- 172.21.15.69:0/3074115439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x563fb7336f50 0x563fb732c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f043a600640 1 -- 172.21.15.69:0/3074115439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563fb72b0940 msgr2=0x563fb73303b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:11.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f0439c00640 1 --2- 172.21.15.69:0/3074115439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563fb72b0940 0x563fb73303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f043a600640 1 --2- 172.21.15.69:0/3074115439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563fb72b0940 0x563fb73303b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f043a600640 1 -- 172.21.15.69:0/3074115439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fb732ca60 con 0x563fb7265770 2024-04-24T08:21:11.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f043a600640 1 --2- 172.21.15.69:0/3074115439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f042c00c4b0 tx=0x7f042c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981a99e61473960c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f0433e00640 1 -- 172.21.15.69:0/3074115439 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c0116b0 con 0x563fb7265770 2024-04-24T08:21:11.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f0433e00640 1 -- 172.21.15.69:0/3074115439 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f042c017070 con 0x563fb7265770 2024-04-24T08:21:11.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.833+0000 7f0433e00640 1 -- 172.21.15.69:0/3074115439 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c011a90 con 0x563fb7265770 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.834+0000 7f043d480f40 1 -- 172.21.15.69:0/3074115439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 msgr2=0x563fb7292b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.834+0000 7f043d480f40 1 --2- 172.21.15.69:0/3074115439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f042c00c4b0 tx=0x7f042c00c970 comp rx=0 tx=0).stop 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.834+0000 7f043d480f40 1 -- 172.21.15.69:0/3074115439 shutdown_connections 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.834+0000 7f043d480f40 1 --2- 172.21.15.69:0/3074115439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x563fb7336f50 0x563fb732c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.834+0000 7f043d480f40 1 --2- 172.21.15.69:0/3074115439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563fb72b0940 0x563fb73303b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.834+0000 7f043d480f40 1 --2- 172.21.15.69:0/3074115439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.834+0000 7f043d480f40 1 -- 172.21.15.69:0/3074115439 >> 172.21.15.69:0/3074115439 conn(0x563fb7294b50 msgr2=0x563fb7265040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.834+0000 7f043d480f40 1 -- 172.21.15.69:0/3074115439 shutdown_connections 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.834+0000 7f043d480f40 1 -- 172.21.15.69:0/3074115439 wait complete. 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.835+0000 7f043d480f40 1 Processor -- start 2024-04-24T08:21:11.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.835+0000 7f043d480f40 1 -- start start 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043d480f40 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043d480f40 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563fb72b0940 0x563fb72635f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043d480f40 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x563fb7336f50 0x563fb732a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043d480f40 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563fb72950d0 con 0x563fb7265770 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043d480f40 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x563fb7264750 con 0x563fb72b0940 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043d480f40 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x563fb7293070 con 0x563fb7336f50 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043a600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043a600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53036/0 (socket says 172.21.15.69:53036) 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043a600640 1 -- 172.21.15.69:0/942020484 learned_addr learned my addr 172.21.15.69:0/942020484 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043a600640 1 -- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x563fb7336f50 msgr2=0x563fb732a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043a600640 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x563fb7336f50 0x563fb732a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043a600640 1 -- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563fb72b0940 msgr2=0x563fb72635f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043a600640 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x563fb72b0940 0x563fb72635f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.836+0000 7f043a600640 1 -- 172.21.15.69:0/942020484 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fb73288f0 con 0x563fb7265770 2024-04-24T08:21:11.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.837+0000 7f043a600640 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f042c014040 tx=0x7f042c0033a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.837+0000 7f0432a00640 1 -- 172.21.15.69:0/942020484 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c012ac0 con 0x563fb7265770 2024-04-24T08:21:11.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.837+0000 7f0432a00640 1 -- 172.21.15.69:0/942020484 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f042c012c60 con 0x563fb7265770 2024-04-24T08:21:11.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.837+0000 7f043d480f40 1 -- 172.21.15.69:0/942020484 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x563fb73394b0 con 0x563fb7265770 2024-04-24T08:21:11.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.837+0000 7f0432a00640 1 -- 172.21.15.69:0/942020484 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042c026650 con 0x563fb7265770 2024-04-24T08:21:11.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.837+0000 7f043d480f40 1 -- 172.21.15.69:0/942020484 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fb7263b30 con 0x563fb7265770 2024-04-24T08:21:11.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.839+0000 7f0432a00640 1 -- 172.21.15.69:0/942020484 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f042c024020 con 0x563fb7265770 2024-04-24T08:21:11.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.840+0000 7f0432a00640 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06b2d0 0x7f041c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.840+0000 7f0432a00640 1 -- 172.21.15.69:0/942020484 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f042c08c980 con 0x563fb7265770 2024-04-24T08:21:11.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.840+0000 7f043d480f40 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x563fb732acf0 0x563fb73279e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:11.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.840+0000 7f043d480f40 1 -- 172.21.15.69:0/942020484 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:1 17.5 17:a367a3e8:::10000000004.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112) -- 0x563fb733aba0 con 0x563fb732acf0 2024-04-24T08:21:11.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.840+0000 7f0439c00640 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06b2d0 0x7f041c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.841+0000 7f043b000640 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x563fb732acf0 0x563fb73279e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:11.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.842+0000 7f0439c00640 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06b2d0 0x7f041c06d790 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x563fb733a990 tx=0x7f04280013a0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.842+0000 7f043b000640 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x563fb732acf0 0x563fb73279e0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043b000640 1 -- 172.21.15.69:0/942020484 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(1 10000000004.00000000 [getxattr (12) out=12b] v0'0 uv12 ondisk = 0) ==== 164+0+12 (crc 0 0 0) 0x7f0434051070 con 0x563fb732acf0 2024-04-24T08:21:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043d480f40 1 -- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x563fb732acf0 msgr2=0x563fb73279e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043d480f40 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x563fb732acf0 0x563fb73279e0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043d480f40 1 -- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06b2d0 msgr2=0x7f041c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043d480f40 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06b2d0 0x7f041c06d790 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x563fb733a990 tx=0x7f04280013a0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043d480f40 1 -- 172.21.15.69:0/942020484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 msgr2=0x563fb7292b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043d480f40 1 --2- 172.21.15.69:0/942020484 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563fb7265770 0x563fb7292b30 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f042c014040 tx=0x7f042c0033a0 comp rx=0 tx=0).stop 2024-04-24T08:21:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043b000640 1 -- 172.21.15.69:0/942020484 reap_dead start 2024-04-24T08:21:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043d480f40 1 -- 172.21.15.69:0/942020484 shutdown_connections 2024-04-24T08:21:11.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043d480f40 1 -- 172.21.15.69:0/942020484 >> 172.21.15.69:0/942020484 conn(0x563fb7294b50 msgr2=0x563fb7265040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:11.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043d480f40 1 -- 172.21.15.69:0/942020484 shutdown_connections 2024-04-24T08:21:11.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:11.843+0000 7f043d480f40 1 -- 172.21.15.69:0/942020484 wait complete. 2024-04-24T08:21:11.848 INFO:teuthology.orchestra.run.smithi069.stdout:mytag123 2024-04-24T08:21:11.848 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-04-24T08:21:12.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.025+0000 7f4d42800640 1 Processor -- start 2024-04-24T08:21:12.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.025+0000 7f4d42800640 1 -- start start 2024-04-24T08:21:12.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.027+0000 7f4d42800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4d3c071d70 0x7f4d3c072220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.027+0000 7f4d42800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 0x7f4d3c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.027+0000 7f4d42800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d3c0b3700 con 0x7f4d3c15d030 2024-04-24T08:21:12.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.027+0000 7f4d42800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f4d3c0b39e0 con 0x7f4d3c071d70 2024-04-24T08:21:12.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.027+0000 7f4d42800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3c0b3e20 con 0x7f4d3c06e240 2024-04-24T08:21:12.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.027+0000 7f4d41e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 0x7f4d3c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.027+0000 7f4d40a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f4d3c071d70 0x7f4d3c072220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.028+0000 7f4d41400640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f4d3c15d030 0x7f4d3c163b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:56162/0 (socket says 172.21.15.69:56162) 2024-04-24T08:21:12.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.028+0000 7f4d41400640 1 -- 172.21.15.69:0/2936269963 learned_addr learned my addr 172.21.15.69:0/2936269963 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:12.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.028+0000 7f4d40a00640 1 -- 172.21.15.69:0/2936269963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 msgr2=0x7f4d3c06e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.028+0000 7f4d40a00640 1 --2- 172.21.15.69:0/2936269963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 0x7f4d3c06e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.028+0000 7f4d40a00640 1 -- 172.21.15.69:0/2936269963 >> v1:172.21.15.121:6789/0 conn(0x7f4d3c15d030 legacy=0x7f4d3c163b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.028+0000 7f4d40a00640 1 -- 172.21.15.69:0/2936269963 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c0b4100 con 0x7f4d3c071d70 2024-04-24T08:21:12.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.030+0000 7f4d40a00640 1 --2- 172.21.15.69:0/2936269963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3c071d70 0x7f4d3c072220 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4d280097a0 tx=0x7f4d2802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5afc6dea79382936 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.030+0000 7f4d3be00640 1 -- 172.21.15.69:0/2936269963 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28037070 con 0x7f4d3c071d70 2024-04-24T08:21:12.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.030+0000 7f4d3be00640 1 -- 172.21.15.69:0/2936269963 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d28032440 con 0x7f4d3c071d70 2024-04-24T08:21:12.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.030+0000 7f4d3be00640 1 -- 172.21.15.69:0/2936269963 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28032810 con 0x7f4d3c071d70 2024-04-24T08:21:12.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.031+0000 7f4d42800640 1 -- 172.21.15.69:0/2936269963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3c071d70 msgr2=0x7f4d3c072220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.031+0000 7f4d42800640 1 --2- 172.21.15.69:0/2936269963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3c071d70 0x7f4d3c072220 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4d280097a0 tx=0x7f4d2802a180 comp rx=0 tx=0).stop 2024-04-24T08:21:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.031+0000 7f4d42800640 1 -- 172.21.15.69:0/2936269963 shutdown_connections 2024-04-24T08:21:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.031+0000 7f4d42800640 1 --2- 172.21.15.69:0/2936269963 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 0x7f4d3c06e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.031+0000 7f4d42800640 1 --2- 172.21.15.69:0/2936269963 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3c071d70 0x7f4d3c072220 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.031+0000 7f4d42800640 1 -- 172.21.15.69:0/2936269963 >> 172.21.15.69:0/2936269963 conn(0x7f4d3c067b80 msgr2=0x7f4d3c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.031+0000 7f4d42800640 1 -- 172.21.15.69:0/2936269963 shutdown_connections 2024-04-24T08:21:12.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.031+0000 7f4d42800640 1 -- 172.21.15.69:0/2936269963 wait complete. 2024-04-24T08:21:12.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.033+0000 7f4d42800640 1 Processor -- start 2024-04-24T08:21:12.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.033+0000 7f4d42800640 1 -- start start 2024-04-24T08:21:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d42800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 0x7f4d3c154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d42800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3c071d70 0x7f4d3c153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d42800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3c15d030 0x7f4d3c1f8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d42800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c0b3320 con 0x7f4d3c15d030 2024-04-24T08:21:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d42800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c06f7b0 con 0x7f4d3c071d70 2024-04-24T08:21:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d42800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3c0b34a0 con 0x7f4d3c06e240 2024-04-24T08:21:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d41e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3c15d030 0x7f4d3c1f8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d41e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3c15d030 0x7f4d3c1f8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53040/0 (socket says 172.21.15.69:53040) 2024-04-24T08:21:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d41e00640 1 -- 172.21.15.69:0/1859159890 learned_addr learned my addr 172.21.15.69:0/1859159890 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d41400640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 0x7f4d3c154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d40a00640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3c071d70 0x7f4d3c153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d41e00640 1 -- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 msgr2=0x7f4d3c154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.034+0000 7f4d41e00640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 0x7f4d3c154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d41e00640 1 -- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3c071d70 msgr2=0x7f4d3c153d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d41e00640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3c071d70 0x7f4d3c153d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d41e00640 1 -- 172.21.15.69:0/1859159890 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c155100 con 0x7f4d3c15d030 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d40a00640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3c071d70 0x7f4d3c153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d41400640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 0x7f4d3c154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d41e00640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3c15d030 0x7f4d3c1f8bc0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400efc0 tx=0x7f4d3400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d39600640 1 -- 172.21.15.69:0/1859159890 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34018020 con 0x7f4d3c15d030 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d39600640 1 -- 172.21.15.69:0/1859159890 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d34014480 con 0x7f4d3c15d030 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d39600640 1 -- 172.21.15.69:0/1859159890 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34014870 con 0x7f4d3c15d030 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d42800640 1 -- 172.21.15.69:0/1859159890 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3c1553e0 con 0x7f4d3c15d030 2024-04-24T08:21:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.035+0000 7f4d42800640 1 -- 172.21.15.69:0/1859159890 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3c1559a0 con 0x7f4d3c15d030 2024-04-24T08:21:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.038+0000 7f4d39600640 1 -- 172.21.15.69:0/1859159890 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d34005d10 con 0x7f4d3c15d030 2024-04-24T08:21:12.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.039+0000 7f4d39600640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d1c06c8f0 0x7f4d1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.039+0000 7f4d39600640 1 -- 172.21.15.69:0/1859159890 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f4d3408c7e0 con 0x7f4d3c15d030 2024-04-24T08:21:12.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.039+0000 7f4d46b72740 1 Processor -- start 2024-04-24T08:21:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.039+0000 7f4d46b72740 1 -- start start 2024-04-24T08:21:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.039+0000 7f4d41400640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d1c06c8f0 0x7f4d1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.040+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684ca1e00 0x55b684c54790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.040+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c84f20 0x55b684c842a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.040+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c847e0 0x55b684c595f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.040+0000 7f4d46b72740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b684bbe1f0 con 0x55b684c847e0 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.040+0000 7f4d46b72740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b684b64350 con 0x55b684c84f20 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.040+0000 7f4d46b72740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55b684bba1b0 con 0x55b684ca1e00 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.040+0000 7f4d41400640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d1c06c8f0 0x7f4d1c06edb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4d30005880 tx=0x7f4d30009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.040+0000 7f4d41e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c847e0 0x55b684c595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.040+0000 7f4d41e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c847e0 0x55b684c595f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53050/0 (socket says 172.21.15.69:53050) 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.040+0000 7f4d41e00640 1 -- 172.21.15.69:0/3201261338 learned_addr learned my addr 172.21.15.69:0/3201261338 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d40a00640 1 --2- 172.21.15.69:0/3201261338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c84f20 0x55b684c842a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d41400640 1 --2- 172.21.15.69:0/3201261338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684ca1e00 0x55b684c54790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d41e00640 1 -- 172.21.15.69:0/3201261338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684ca1e00 msgr2=0x55b684c54790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d41e00640 1 --2- 172.21.15.69:0/3201261338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684ca1e00 0x55b684c54790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d41e00640 1 -- 172.21.15.69:0/3201261338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c84f20 msgr2=0x55b684c842a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d41e00640 1 --2- 172.21.15.69:0/3201261338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c84f20 0x55b684c842a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d41e00640 1 -- 172.21.15.69:0/3201261338 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b684bb8f50 con 0x55b684c847e0 2024-04-24T08:21:12.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d41e00640 1 --2- 172.21.15.69:0/3201261338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c847e0 0x55b684c595f0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f4d3405a370 tx=0x7f4d3405a4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1056887180d5d7b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d2dc00640 1 -- 172.21.15.69:0/3201261338 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34089070 con 0x55b684c847e0 2024-04-24T08:21:12.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d2dc00640 1 -- 172.21.15.69:0/3201261338 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3408acf0 con 0x55b684c847e0 2024-04-24T08:21:12.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.041+0000 7f4d2dc00640 1 -- 172.21.15.69:0/3201261338 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34072070 con 0x55b684c847e0 2024-04-24T08:21:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.042+0000 7f4d46b72740 1 -- 172.21.15.69:0/3201261338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c847e0 msgr2=0x55b684c595f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.042+0000 7f4d46b72740 1 --2- 172.21.15.69:0/3201261338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c847e0 0x55b684c595f0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f4d3405a370 tx=0x7f4d3405a4a0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.042+0000 7f4d46b72740 1 -- 172.21.15.69:0/3201261338 shutdown_connections 2024-04-24T08:21:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.042+0000 7f4d46b72740 1 --2- 172.21.15.69:0/3201261338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c847e0 0x55b684c595f0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.042+0000 7f4d46b72740 1 --2- 172.21.15.69:0/3201261338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c84f20 0x55b684c842a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.042+0000 7f4d46b72740 1 --2- 172.21.15.69:0/3201261338 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684ca1e00 0x55b684c54790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.042+0000 7f4d46b72740 1 -- 172.21.15.69:0/3201261338 >> 172.21.15.69:0/3201261338 conn(0x55b684c56a00 msgr2=0x55b684c9f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.042+0000 7f4d46b72740 1 -- 172.21.15.69:0/3201261338 shutdown_connections 2024-04-24T08:21:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.042+0000 7f4d46b72740 1 -- 172.21.15.69:0/3201261338 wait complete. 2024-04-24T08:21:12.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.043+0000 7f4d46b72740 1 -- start start 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.043+0000 7f4d46b72740 1 Processor -- start 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c84f20 0x55b684c7fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c96e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c95f60 0x55b684c963e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d46b72740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b684c58bb0 con 0x55b684ca1e00 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d46b72740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55b684b77690 con 0x55b684c95f60 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d46b72740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55b684c57fa0 con 0x55b684c84f20 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d46b72740 10 client.0 _ll_register_callbacks cb 0x55b684bb3650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d40a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c96e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d40a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c96e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53054/0 (socket says 172.21.15.69:53054) 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d40a00640 1 -- 172.21.15.69:0/2569166190 learned_addr learned my addr 172.21.15.69:0/2569166190 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.044+0000 7f4d41400640 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c84f20 0x55b684c7fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.045+0000 7f4d41e00640 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c95f60 0x55b684c963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.045+0000 7f4d40a00640 1 -- 172.21.15.69:0/2569166190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c84f20 msgr2=0x55b684c7fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.045+0000 7f4d40a00640 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c84f20 0x55b684c7fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.045+0000 7f4d40a00640 1 -- 172.21.15.69:0/2569166190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c95f60 msgr2=0x55b684c963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.045+0000 7f4d40a00640 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c95f60 0x55b684c963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.045+0000 7f4d40a00640 1 -- 172.21.15.69:0/2569166190 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b684b76920 con 0x55b684ca1e00 2024-04-24T08:21:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.045+0000 7f4d40a00640 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c96e60 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f4d280097a0 tx=0x7f4d28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb206c882f2a985 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.045+0000 7f4d22000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:21:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d22000640 1 -- 172.21.15.69:0/2569166190 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28037040 con 0x55b684ca1e00 2024-04-24T08:21:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d22000640 1 -- 172.21.15.69:0/2569166190 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f4d28032dd0 con 0x55b684ca1e00 2024-04-24T08:21:12.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d46b72740 1 -- 172.21.15.69:0/2569166190 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d3408acf0 con 0x55b684ca1e00 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d22000640 1 -- 172.21.15.69:0/2569166190 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f4d2804d050 con 0x55b684ca1e00 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d46b72740 10 client.9810 fetch_fsmap learned FSMap version 105 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d46b72740 1 -- 172.21.15.69:0/2569166190 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x55b68499a7d0 con 0x55b684ca1e00 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d22000640 1 -- 172.21.15.69:0/2569166190 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3408acf0 con 0x55b684ca1e00 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d22000640 1 -- 172.21.15.69:0/2569166190 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f4d28050020 con 0x55b684ca1e00 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d46b72740 10 client.9810 fetch_fsmap finished waiting for FSMap version 105 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d46b72740 10 client.9810 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d46b72740 20 client.9810 populate_metadata read hostname 'smithi069' 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x55b684c86e50 0x55b684c87230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d46b72740 4 client.9810 mds_command: new command op to 9285 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.046+0000 7f4d46b72740 1 -- 172.21.15.69:0/2569166190 --> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b684bcbc10 con 0x55b684c86e50 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.047+0000 7f4d41400640 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x55b684c86e50 0x55b684c87230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.047+0000 7f4d41400640 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x55b684c86e50 0x55b684c87230 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9025bbc6fa36913 server_cookie=c19e960c9ea31c35 in_seq=0 out_seq=0 2024-04-24T08:21:12.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.047+0000 7f4d22000640 10 client.9810 ms_handle_connect on v2:172.21.15.69:6834/2194021008 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d22000640 1 -- 172.21.15.69:0/2569166190 <== mds.0 v2:172.21.15.69:6834/2194021008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b684bcbc10 con 0x55b684c86e50 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d22000640 10 client.9810 handle_command_reply: tid=0 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 client.9810 shutdown 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 -- 172.21.15.69:0/2569166190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 msgr2=0x55b684c96e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c96e60 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f4d280097a0 tx=0x7f4d28004730 comp rx=0 tx=0).stop 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 -- 172.21.15.69:0/2569166190 shutdown_connections 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x55b684c86e50 0x55b684c87230 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c96e60 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c95f60 0x55b684c963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2569166190 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c84f20 0x55b684c7fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d22000640 0 client.9810 ms_handle_reset on v2:172.21.15.69:6834/2194021008 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 -- 172.21.15.69:0/2569166190 >> 172.21.15.69:0/2569166190 conn(0x55b684c847e0 msgr2=0x55b684c59040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 -- 172.21.15.69:0/2569166190 shutdown_connections 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 1 -- 172.21.15.69:0/2569166190 wait complete. 2024-04-24T08:21:12.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.048+0000 7f4d46b72740 20 client.9810 trim_cache size 0 max 16384 2024-04-24T08:21:12.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.064+0000 7f4d46b72740 1 Processor -- start 2024-04-24T08:21:12.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.064+0000 7f4d46b72740 1 -- start start 2024-04-24T08:21:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.065+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c84f20 0x55b684c94ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.065+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c95f60 0x55b684caedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.065+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c9ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.065+0000 7f4d46b72740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b684c88840 con 0x55b684ca1e00 2024-04-24T08:21:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.065+0000 7f4d46b72740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c2052f0 con 0x55b684c84f20 2024-04-24T08:21:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.065+0000 7f4d46b72740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3c2042f0 con 0x55b684c95f60 2024-04-24T08:21:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d41e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c9ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d41e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c9ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53062/0 (socket says 172.21.15.69:53062) 2024-04-24T08:21:12.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d41e00640 1 -- 172.21.15.69:0/1027138634 learned_addr learned my addr 172.21.15.69:0/1027138634 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d41e00640 1 -- 172.21.15.69:0/1027138634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c95f60 msgr2=0x55b684caedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d41e00640 1 --2- 172.21.15.69:0/1027138634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c95f60 0x55b684caedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d41e00640 1 -- 172.21.15.69:0/1027138634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c84f20 msgr2=0x55b684c94ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d41e00640 1 --2- 172.21.15.69:0/1027138634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c84f20 0x55b684c94ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d41e00640 1 -- 172.21.15.69:0/1027138634 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b684c971d0 con 0x55b684ca1e00 2024-04-24T08:21:12.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d41e00640 1 --2- 172.21.15.69:0/1027138634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c9ceb0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400c990 tx=0x7f4d3405e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8501d1cde0e521e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d2d200640 1 -- 172.21.15.69:0/1027138634 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34078070 con 0x55b684ca1e00 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d2d200640 1 -- 172.21.15.69:0/1027138634 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3408a040 con 0x55b684ca1e00 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.066+0000 7f4d2d200640 1 -- 172.21.15.69:0/1027138634 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34089070 con 0x55b684ca1e00 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.067+0000 7f4d46b72740 1 -- 172.21.15.69:0/1027138634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 msgr2=0x55b684c9ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.067+0000 7f4d46b72740 1 --2- 172.21.15.69:0/1027138634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c9ceb0 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400c990 tx=0x7f4d3405e000 comp rx=0 tx=0).stop 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.067+0000 7f4d46b72740 1 -- 172.21.15.69:0/1027138634 shutdown_connections 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.067+0000 7f4d46b72740 1 --2- 172.21.15.69:0/1027138634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684ca1e00 0x55b684c9ceb0 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.067+0000 7f4d46b72740 1 --2- 172.21.15.69:0/1027138634 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c95f60 0x55b684caedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.067+0000 7f4d46b72740 1 --2- 172.21.15.69:0/1027138634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684c84f20 0x55b684c94ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.067+0000 7f4d46b72740 1 -- 172.21.15.69:0/1027138634 >> 172.21.15.69:0/1027138634 conn(0x55b684c847e0 msgr2=0x55b684c4fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.067+0000 7f4d46b72740 1 -- 172.21.15.69:0/1027138634 shutdown_connections 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.067+0000 7f4d46b72740 1 -- 172.21.15.69:0/1027138634 wait complete. 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.068+0000 7f4d46b72740 1 -- start start 2024-04-24T08:21:12.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.068+0000 7f4d46b72740 1 Processor -- start 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c84f20 0x55b684c581e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c95f60 0x55b684c53740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d46b72740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684ca1e00 0x55b684c53c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d46b72740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c203cd0 con 0x55b684c95f60 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d46b72740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c203680 con 0x55b684ca1e00 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d46b72740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55b684c88840 con 0x55b684c84f20 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d46b72740 10 client.0 _ll_register_callbacks cb 0x7f4d28035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d40a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c95f60 0x55b684c53740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d40a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c95f60 0x55b684c53740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53066/0 (socket says 172.21.15.69:53066) 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d40a00640 1 -- 172.21.15.69:0/2085905987 learned_addr learned my addr 172.21.15.69:0/2085905987 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d40a00640 1 -- 172.21.15.69:0/2085905987 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c84f20 msgr2=0x55b684c581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d40a00640 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c84f20 0x55b684c581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.069+0000 7f4d41e00640 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684ca1e00 0x55b684c53c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d40a00640 1 -- 172.21.15.69:0/2085905987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684ca1e00 msgr2=0x55b684c53c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d40a00640 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684ca1e00 0x55b684c53c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d40a00640 1 -- 172.21.15.69:0/2085905987 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b684c875a0 con 0x55b684c95f60 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d41e00640 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684ca1e00 0x55b684c53c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d40a00640 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c95f60 0x55b684c53740 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f4d28000c00 tx=0x7f4d280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b4cd33b74542e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d22000640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d22000640 1 -- 172.21.15.69:0/2085905987 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d28037070 con 0x55b684c95f60 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d22000640 1 -- 172.21.15.69:0/2085905987 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f4d28004450 con 0x55b684c95f60 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d46b72740 1 -- 172.21.15.69:0/2085905987 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d3408a040 con 0x55b684c95f60 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d22000640 1 -- 172.21.15.69:0/2085905987 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) ==== 24+0+0 (secure 0 0 0) 0x7f4d28059050 con 0x55b684c95f60 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d46b72740 10 client.9816 fetch_fsmap learned FSMap version 105 2024-04-24T08:21:12.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.070+0000 7f4d46b72740 1 -- 172.21.15.69:0/2085905987 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) -- 0x55b6848d03f0 con 0x55b684c95f60 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.071+0000 7f4d22000640 1 -- 172.21.15.69:0/2085905987 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3408a040 con 0x55b684c95f60 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.071+0000 7f4d22000640 1 -- 172.21.15.69:0/2085905987 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 105) ==== 3494+0+0 (secure 0 0 0) 0x7f4d2805c020 con 0x55b684c95f60 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.071+0000 7f4d46b72740 10 client.9816 fetch_fsmap finished waiting for FSMap version 105 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.071+0000 7f4d46b72740 10 client.9816 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.071+0000 7f4d46b72740 20 client.9816 populate_metadata read hostname 'smithi069' 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.071+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x55b684cc1d70 0x55b684cc2150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.071+0000 7f4d46b72740 4 client.9816 mds_command: new command op to 9285 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.071+0000 7f4d46b72740 1 -- 172.21.15.69:0/2085905987 --> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b684bc5a80 con 0x55b684cc1d70 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.071+0000 7f4d41400640 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x55b684cc1d70 0x55b684cc2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.072+0000 7f4d41400640 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x55b684cc1d70 0x55b684cc2150 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c70f584e64b5c74 server_cookie=55103d90e6345cc4 in_seq=0 out_seq=0 2024-04-24T08:21:12.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.072+0000 7f4d22000640 10 client.9816 ms_handle_connect on v2:172.21.15.69:6834/2194021008 2024-04-24T08:21:12.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d22000640 1 -- 172.21.15.69:0/2085905987 <== mds.0 v2:172.21.15.69:6834/2194021008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b684bc5a80 con 0x55b684cc1d70 2024-04-24T08:21:12.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d22000640 10 client.9816 handle_command_reply: tid=0 2024-04-24T08:21:12.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d46b72740 1 client.9816 shutdown 2024-04-24T08:21:12.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d46b72740 1 -- 172.21.15.69:0/2085905987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c95f60 msgr2=0x55b684c53740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c95f60 0x55b684c53740 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7f4d28000c00 tx=0x7f4d280046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d46b72740 1 -- 172.21.15.69:0/2085905987 shutdown_connections 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.69:6834/2194021008,v1:172.21.15.69:6835/2194021008] conn(0x55b684cc1d70 0x55b684cc2150 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b684c95f60 0x55b684c53740 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b684ca1e00 0x55b684c53c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d46b72740 1 --2- 172.21.15.69:0/2085905987 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55b684c84f20 0x55b684c581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d46b72740 1 -- 172.21.15.69:0/2085905987 >> 172.21.15.69:0/2085905987 conn(0x55b684c847e0 msgr2=0x55b684caedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.084+0000 7f4d22000640 0 client.9816 ms_handle_reset on v2:172.21.15.69:6834/2194021008 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.085+0000 7f4d46b72740 1 -- 172.21.15.69:0/2085905987 shutdown_connections 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.085+0000 7f4d46b72740 1 -- 172.21.15.69:0/2085905987 wait complete. 2024-04-24T08:21:12.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.085+0000 7f4d46b72740 20 client.9816 trim_cache size 0 max 16384 2024-04-24T08:21:12.087 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "message": "", 2024-04-24T08:21:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0 2024-04-24T08:21:12.087 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.085+0000 7f4d42800640 1 -- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d1c06c8f0 msgr2=0x7f4d1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.085+0000 7f4d42800640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d1c06c8f0 0x7f4d1c06edb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f4d30005880 tx=0x7f4d30009040 comp rx=0 tx=0).stop 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.085+0000 7f4d42800640 1 -- 172.21.15.69:0/1859159890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3c15d030 msgr2=0x7f4d3c1f8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.085+0000 7f4d42800640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3c15d030 0x7f4d3c1f8bc0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400efc0 tx=0x7f4d3400c490 comp rx=0 tx=0).stop 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.086+0000 7f4d42800640 1 -- 172.21.15.69:0/1859159890 shutdown_connections 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.086+0000 7f4d42800640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f4d1c06c8f0 0x7f4d1c06edb0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.086+0000 7f4d42800640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4d3c15d030 0x7f4d3c1f8bc0 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.086+0000 7f4d42800640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d3c071d70 0x7f4d3c153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.086+0000 7f4d42800640 1 --2- 172.21.15.69:0/1859159890 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4d3c06e240 0x7f4d3c154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.086+0000 7f4d42800640 1 -- 172.21.15.69:0/1859159890 >> 172.21.15.69:0/1859159890 conn(0x7f4d3c067b80 msgr2=0x7f4d3c0b2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.086+0000 7f4d42800640 1 -- 172.21.15.69:0/1859159890 shutdown_connections 2024-04-24T08:21:12.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.086+0000 7f4d42800640 1 -- 172.21.15.69:0/1859159890 wait complete. 2024-04-24T08:21:12.102 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:21:12.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.276+0000 7f3dc9200640 1 Processor -- start 2024-04-24T08:21:12.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.277+0000 7f3dc9200640 1 -- start start 2024-04-24T08:21:12.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.278+0000 7f3dc9200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f3dc415d1d0 0x7f3dc40ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.278+0000 7f3dc9200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc415d660 con 0x7f3dc4156c60 2024-04-24T08:21:12.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.278+0000 7f3dc9200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc406dc30 con 0x7f3dc40b30f0 2024-04-24T08:21:12.279 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.278+0000 7f3dc9200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f3dc406df10 con 0x7f3dc415d1d0 2024-04-24T08:21:12.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc3400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3dc4156c60 0x7f3dc415cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48664/0 (socket says 172.21.15.69:48664) 2024-04-24T08:21:12.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc3400640 1 -- 172.21.15.69:0/3657034055 learned_addr learned my addr 172.21.15.69:0/3657034055 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:12.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc2a00640 1 --2- 172.21.15.69:0/3657034055 >> v2:172.21.15.121:3300/0 conn(0x7f3dc415d1d0 0x7f3dc40ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3765863936 0 0) 0x7f3dc415d660 con 0x7f3dc4156c60 2024-04-24T08:21:12.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3da8003520 con 0x7f3dc4156c60 2024-04-24T08:21:12.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2684570246 0 0) 0x7f3da8003520 con 0x7f3dc4156c60 2024-04-24T08:21:12.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 >> v2:172.21.15.121:3300/0 conn(0x7f3dc415d1d0 msgr2=0x7f3dc40ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc2000640 1 --2- 172.21.15.69:0/3657034055 >> v2:172.21.15.121:3300/0 conn(0x7f3dc415d1d0 0x7f3dc40ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 >> v1:172.21.15.121:6789/0 conn(0x7f3dc40b30f0 legacy=0x7f3dc406e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc406e1f0 con 0x7f3dc4156c60 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.279+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f3db40019f0 con 0x7f3dc4156c60 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 >> v1:172.21.15.69:6789/0 conn(0x7f3dc4156c60 legacy=0x7f3dc415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc2000640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3da8003a00 0x7f3da8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc2000640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3da8004800 0x7f3da8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc2000640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3da80051c0 0x7f3da800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc406df10 con 0x7f3da80051c0 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3da800c0c0 con 0x7f3da8003a00 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3da800c2c0 con 0x7f3da8004800 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc3e00640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3da80051c0 0x7f3da800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc3e00640 1 -- 172.21.15.69:0/3657034055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3da8004800 msgr2=0x7f3da8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc3e00640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3da8004800 0x7f3da8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc3400640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3da8003a00 0x7f3da8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc3e00640 1 -- 172.21.15.69:0/3657034055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3da8003a00 msgr2=0x7f3da8003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc3e00640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3da8003a00 0x7f3da8003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc3e00640 1 -- 172.21.15.69:0/3657034055 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3da8003520 con 0x7f3da80051c0 2024-04-24T08:21:12.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.280+0000 7f3dc3400640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3da8003a00 0x7f3da8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.281+0000 7f3dc3e00640 1 -- 172.21.15.69:0/3657034055 reap_dead start 2024-04-24T08:21:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.281+0000 7f3dc3e00640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3da80051c0 0x7f3da800b9c0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40d11e0 tx=0x7f3da8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48e758ef758de52 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.282+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc001370 con 0x7f3da80051c0 2024-04-24T08:21:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.282+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dbc0052d0 con 0x7f3da80051c0 2024-04-24T08:21:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.282+0000 7f3dc2000640 1 -- 172.21.15.69:0/3657034055 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc02ab90 con 0x7f3da80051c0 2024-04-24T08:21:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.282+0000 7f3dc9200640 1 -- 172.21.15.69:0/3657034055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3da80051c0 msgr2=0x7f3da800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.282+0000 7f3dc9200640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3da80051c0 0x7f3da800b9c0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f3dc40d11e0 tx=0x7f3da8005830 comp rx=0 tx=0).stop 2024-04-24T08:21:12.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.282+0000 7f3dc9200640 1 -- 172.21.15.69:0/3657034055 shutdown_connections 2024-04-24T08:21:12.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.282+0000 7f3dc9200640 1 --2- 172.21.15.69:0/3657034055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3da80051c0 0x7f3da800b9c0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.282+0000 7f3dc9200640 1 -- 172.21.15.69:0/3657034055 >> 172.21.15.69:0/3657034055 conn(0x7f3dc40d0160 msgr2=0x7f3dc40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.282+0000 7f3dc9200640 1 -- 172.21.15.69:0/3657034055 shutdown_connections 2024-04-24T08:21:12.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.282+0000 7f3dc9200640 1 -- 172.21.15.69:0/3657034055 wait complete. 2024-04-24T08:21:12.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.284+0000 7f3dc9200640 1 Processor -- start 2024-04-24T08:21:12.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.284+0000 7f3dc9200640 1 -- start start 2024-04-24T08:21:12.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.285+0000 7f3dc9200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3da80051c0 0x7f3dc41f8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.285+0000 7f3dc9200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dc4154ef0 0x7f3dc4157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.285+0000 7f3dc9200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dc4157e70 0x7f3dc415dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.285+0000 7f3dc9200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc415cac0 con 0x7f3dc4157e70 2024-04-24T08:21:12.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.285+0000 7f3dc9200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc415d940 con 0x7f3dc4154ef0 2024-04-24T08:21:12.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.285+0000 7f3dc9200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc415cc40 con 0x7f3da80051c0 2024-04-24T08:21:12.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.285+0000 7f3dc3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dc4157e70 0x7f3dc415dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dc4157e70 0x7f3dc415dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53076/0 (socket says 172.21.15.69:53076) 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc3e00640 1 -- 172.21.15.69:0/299934527 learned_addr learned my addr 172.21.15.69:0/299934527 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc2a00640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dc4154ef0 0x7f3dc4157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc3400640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3da80051c0 0x7f3dc41f8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc3e00640 1 -- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3da80051c0 msgr2=0x7f3dc41f8330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc3e00640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3da80051c0 0x7f3dc41f8330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc3e00640 1 -- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dc4154ef0 msgr2=0x7f3dc4157930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc3e00640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dc4154ef0 0x7f3dc4157930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc3e00640 1 -- 172.21.15.69:0/299934527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc41556f0 con 0x7f3dc4157e70 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc2a00640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dc4154ef0 0x7f3dc4157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc3400640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3da80051c0 0x7f3dc41f8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dc3e00640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dc4157e70 0x7f3dc415dda0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc02a540 tx=0x7f3dbc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.286+0000 7f3dbb400640 1 -- 172.21.15.69:0/299934527 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc001370 con 0x7f3dc4157e70 2024-04-24T08:21:12.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.287+0000 7f3dbb400640 1 -- 172.21.15.69:0/299934527 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dbc005480 con 0x7f3dc4157e70 2024-04-24T08:21:12.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.287+0000 7f3dbb400640 1 -- 172.21.15.69:0/299934527 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc007a30 con 0x7f3dc4157e70 2024-04-24T08:21:12.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.287+0000 7f3dc9200640 1 -- 172.21.15.69:0/299934527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc41559d0 con 0x7f3dc4157e70 2024-04-24T08:21:12.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.287+0000 7f3dc9200640 1 -- 172.21.15.69:0/299934527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc4155f90 con 0x7f3dc4157e70 2024-04-24T08:21:12.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.288+0000 7f3dc9200640 1 -- 172.21.15.69:0/299934527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc4075d50 con 0x7f3dc4157e70 2024-04-24T08:21:12.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.289+0000 7f3dbb400640 1 -- 172.21.15.69:0/299934527 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3dbc03a030 con 0x7f3dc4157e70 2024-04-24T08:21:12.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.290+0000 7f3dbb400640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3da406c920 0x7f3da406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.290+0000 7f3dbb400640 1 -- 172.21.15.69:0/299934527 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(112..112 src has 1..112) ==== 5853+0+0 (secure 0 0 0) 0x7f3dbc0aa680 con 0x7f3dc4157e70 2024-04-24T08:21:12.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.290+0000 7f3dc3400640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3da406c920 0x7f3da406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.291+0000 7f3dc3400640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3da406c920 0x7f3da406ede0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3db4002230 tx=0x7f3db40017f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.294+0000 7f3dbb400640 1 -- 172.21.15.69:0/299934527 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3dbc078a20 con 0x7f3dc4157e70 2024-04-24T08:21:12.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.531+0000 7f3dc9200640 1 -- 172.21.15.69:0/299934527 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f3dc406f1c0 con 0x7f3dc4157e70 2024-04-24T08:21:12.532 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:21:12.532+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:21:12.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.535+0000 7f3dbb400640 1 -- 172.21.15.69:0/299934527 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v106) ==== 161+0+0 (secure 0 0 0) 0x7f3dbc078320 con 0x7f3dc4157e70 2024-04-24T08:21:12.536 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:21:12.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 -- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3da406c920 msgr2=0x7f3da406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3da406c920 0x7f3da406ede0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3db4002230 tx=0x7f3db40017f0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 -- 172.21.15.69:0/299934527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dc4157e70 msgr2=0x7f3dc415dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dc4157e70 0x7f3dc415dda0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc02a540 tx=0x7f3dbc001120 comp rx=0 tx=0).stop 2024-04-24T08:21:12.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 -- 172.21.15.69:0/299934527 shutdown_connections 2024-04-24T08:21:12.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f3da406c920 0x7f3da406ede0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3dc4157e70 0x7f3dc415dda0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3dc4154ef0 0x7f3dc4157930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 --2- 172.21.15.69:0/299934527 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f3da80051c0 0x7f3dc41f8330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 -- 172.21.15.69:0/299934527 >> 172.21.15.69:0/299934527 conn(0x7f3dc40d0160 msgr2=0x7f3dc40b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 -- 172.21.15.69:0/299934527 shutdown_connections 2024-04-24T08:21:12.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.538+0000 7f3dc9200640 1 -- 172.21.15.69:0/299934527 wait complete. 2024-04-24T08:21:12.547 INFO:tasks.ceph.mds.c.smithi069.stderr: -2491> 2024-04-24T08:20:55.714+0000 7f56b2200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:21:12.558 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset --force 2024-04-24T08:21:12.598 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:21:12.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.647+0000 7f931b926280 1 -- start start 2024-04-24T08:21:12.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.647+0000 7f931b926280 1 Processor -- start 2024-04-24T08:21:12.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.648+0000 7f931b926280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718eaa800 0x556718287080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.648+0000 7f931b926280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718113c00 0x556718286b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.648+0000 7f931b926280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718eabc00 0x556718286580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.648+0000 7f931b926280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556718ef8000 con 0x556718113c00 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.648+0000 7f931b926280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556718ef8180 con 0x556718eabc00 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.648+0000 7f931b926280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556718ef8300 con 0x556718eaa800 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.648+0000 7f9318600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718113c00 0x556718286b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9318600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718113c00 0x556718286b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53112/0 (socket says 172.21.15.69:53112) 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9318600640 1 -- 172.21.15.69:0/1825560346 learned_addr learned my addr 172.21.15.69:0/1825560346 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9319000640 1 --2- 172.21.15.69:0/1825560346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718eaa800 0x556718287080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9318600640 1 -- 172.21.15.69:0/1825560346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718eaa800 msgr2=0x556718287080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9318600640 1 --2- 172.21.15.69:0/1825560346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718eaa800 0x556718287080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9319a00640 1 --2- 172.21.15.69:0/1825560346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718eabc00 0x556718286580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9318600640 1 -- 172.21.15.69:0/1825560346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718eabc00 msgr2=0x556718286580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9318600640 1 --2- 172.21.15.69:0/1825560346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718eabc00 0x556718286580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9318600640 1 -- 172.21.15.69:0/1825560346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556718146780 con 0x556718113c00 2024-04-24T08:21:12.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9318600640 1 --2- 172.21.15.69:0/1825560346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718113c00 0x556718286b00 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x556718f3f3e0 tx=0x556718eae180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68483134160d883e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9315e00640 1 -- 172.21.15.69:0/1825560346 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556718138d00 con 0x556718113c00 2024-04-24T08:21:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9315e00640 1 -- 172.21.15.69:0/1825560346 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55671813e700 con 0x556718113c00 2024-04-24T08:21:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.649+0000 7f9315e00640 1 -- 172.21.15.69:0/1825560346 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556718138680 con 0x556718113c00 2024-04-24T08:21:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.650+0000 7f931b926280 1 -- 172.21.15.69:0/1825560346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556718147680 con 0x556718113c00 2024-04-24T08:21:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.650+0000 7f931b926280 1 -- 172.21.15.69:0/1825560346 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x556718146000 con 0x556718113c00 2024-04-24T08:21:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.650+0000 7f931b926280 4 waiting for MDS map... 2024-04-24T08:21:12.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.650+0000 7f9315e00640 1 -- 172.21.15.69:0/1825560346 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x556718fe4600 con 0x556718113c00 2024-04-24T08:21:12.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.650+0000 7f9315e00640 1 -- 172.21.15.69:0/1825560346 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x55671816d180 con 0x556718113c00 2024-04-24T08:21:12.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.650+0000 7f931b926280 4 Got MDS map 106 2024-04-24T08:21:12.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.650+0000 7f931b926280 10 main: JournalTool::main 2024-04-24T08:21:12.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.650+0000 7f931b926280 4 main: JournalTool: connecting to RADOS... 2024-04-24T08:21:12.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.651+0000 7f931b926280 1 Processor -- start 2024-04-24T08:21:12.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.651+0000 7f931b926280 1 -- start start 2024-04-24T08:21:12.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f931b926280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718eaa000 0x556718287600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f931b926280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718feb800 0x556718287b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f931b926280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718febc00 0x556718288100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f931b926280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556718ef8480 con 0x556718feb800 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f931b926280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556718ef8600 con 0x556718eaa000 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f931b926280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556718ef8780 con 0x556718febc00 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f9319000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718feb800 0x556718287b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f9319000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718feb800 0x556718287b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53118/0 (socket says 172.21.15.69:53118) 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f9319000640 1 -- 172.21.15.69:0/4063413607 learned_addr learned my addr 172.21.15.69:0/4063413607 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f9319000640 1 -- 172.21.15.69:0/4063413607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718febc00 msgr2=0x556718288100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f9319000640 1 --2- 172.21.15.69:0/4063413607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718febc00 0x556718288100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f9319000640 1 -- 172.21.15.69:0/4063413607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718eaa000 msgr2=0x556718287600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f9319000640 1 --2- 172.21.15.69:0/4063413607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718eaa000 0x556718287600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f9319000640 1 -- 172.21.15.69:0/4063413607 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567181463c0 con 0x556718feb800 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.652+0000 7f9319000640 1 --2- 172.21.15.69:0/4063413607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718feb800 0x556718287b80 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x556719002660 tx=0x556719004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d41a3164b6f70bb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f9313600640 1 -- 172.21.15.69:0/4063413607 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556718138d00 con 0x556718feb800 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f9313600640 1 -- 172.21.15.69:0/4063413607 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55671813ec40 con 0x556718feb800 2024-04-24T08:21:12.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f9313600640 1 -- 172.21.15.69:0/4063413607 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556718138680 con 0x556718feb800 2024-04-24T08:21:12.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f931b926280 1 -- 172.21.15.69:0/4063413607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718feb800 msgr2=0x556718287b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f931b926280 1 --2- 172.21.15.69:0/4063413607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718feb800 0x556718287b80 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x556719002660 tx=0x556719004cc0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f931b926280 1 -- 172.21.15.69:0/4063413607 shutdown_connections 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f931b926280 1 --2- 172.21.15.69:0/4063413607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718feb800 0x556718287b80 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f931b926280 1 --2- 172.21.15.69:0/4063413607 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718febc00 0x556718288100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f931b926280 1 --2- 172.21.15.69:0/4063413607 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718eaa000 0x556718287600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f931b926280 1 -- 172.21.15.69:0/4063413607 >> 172.21.15.69:0/4063413607 conn(0x556718eaa400 msgr2=0x556718e99800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f931b926280 1 -- 172.21.15.69:0/4063413607 shutdown_connections 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.653+0000 7f931b926280 1 -- 172.21.15.69:0/4063413607 wait complete. 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.654+0000 7f931b926280 1 Processor -- start 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.654+0000 7f931b926280 1 -- start start 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f931b926280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718eaa000 0x556718287600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f931b926280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718feb800 0x556718287b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f931b926280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718febc00 0x556718288100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f931b926280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556718ef9e00 con 0x556718eaa000 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f931b926280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556718ef9c80 con 0x556718feb800 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f931b926280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556718ef9b00 con 0x556718febc00 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f9319a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718eaa000 0x556718287600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f9319a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718eaa000 0x556718287600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53122/0 (socket says 172.21.15.69:53122) 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f9319a00640 1 -- 172.21.15.69:0/2877911732 learned_addr learned my addr 172.21.15.69:0/2877911732 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f9318600640 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718febc00 0x556718288100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f9319a00640 1 -- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718febc00 msgr2=0x556718288100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f9319a00640 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718febc00 0x556718288100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.655+0000 7f9319a00640 1 -- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718feb800 msgr2=0x556718287b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.656+0000 7f9319a00640 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718feb800 0x556718287b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.656+0000 7f9319a00640 1 -- 172.21.15.69:0/2877911732 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556718146780 con 0x556718eaa000 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.656+0000 7f9319a00640 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718eaa000 0x556718287600 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x556718142e70 tx=0x556718eac6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.656+0000 7f9311800640 1 -- 172.21.15.69:0/2877911732 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556718138d00 con 0x556718eaa000 2024-04-24T08:21:12.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.656+0000 7f9311800640 1 -- 172.21.15.69:0/2877911732 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55671813ee00 con 0x556718eaa000 2024-04-24T08:21:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.656+0000 7f931b926280 1 -- 172.21.15.69:0/2877911732 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x556718147680 con 0x556718eaa000 2024-04-24T08:21:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.656+0000 7f9311800640 1 -- 172.21.15.69:0/2877911732 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556718138680 con 0x556718eaa000 2024-04-24T08:21:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.656+0000 7f931b926280 1 -- 172.21.15.69:0/2877911732 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567181474a0 con 0x556718eaa000 2024-04-24T08:21:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.656+0000 7f931b926280 4 main: JournalTool: resolving pool 16 2024-04-24T08:21:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.658+0000 7f9311800640 1 -- 172.21.15.69:0/2877911732 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x556719058c00 con 0x556718eaa000 2024-04-24T08:21:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.658+0000 7f9311800640 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556719050c00 0x556718288680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.658+0000 7f9311800640 1 -- 172.21.15.69:0/2877911732 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x556719052a00 con 0x556718eaa000 2024-04-24T08:21:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.658+0000 7f931b926280 4 main: JournalTool: creating IoCtx.. 2024-04-24T08:21:12.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.658+0000 7f931b926280 4 main: Executing for rank 0 2024-04-24T08:21:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.659+0000 7f9319000640 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556719050c00 0x556718288680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.659+0000 7f931b926280 1 -- start start 2024-04-24T08:21:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.659+0000 7f931b926280 1 Processor -- start 2024-04-24T08:21:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.660+0000 7f9319000640 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556719050c00 0x556718288680 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x556719002660 tx=0x556719004c00 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.660+0000 7f931b926280 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556719051400 0x556718288c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.660+0000 7f931b926280 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556719051800 0x556718289180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.660+0000 7f931b926280 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556719051c00 0x556718289700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.660+0000 7f931b926280 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556718ef9800 con 0x556719051800 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.660+0000 7f931b926280 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x556718ef9680 con 0x556719051c00 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.660+0000 7f931b926280 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x556718ef8480 con 0x556719051400 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.660+0000 7f9319000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556719051800 0x556718289180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.660+0000 7f9319000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556719051800 0x556718289180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53132/0 (socket says 172.21.15.69:53132) 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.660+0000 7f9319000640 1 -- 172.21.15.69:0/2089703857 learned_addr learned my addr 172.21.15.69:0/2089703857 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f9319a00640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556719051400 0x556718288c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f9319000640 1 -- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556719051400 msgr2=0x556718288c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f9319000640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556719051400 0x556718288c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f9319000640 1 -- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556719051c00 msgr2=0x556718289700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f9319000640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556719051c00 0x556718289700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f9319000640 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556718146960 con 0x556719051800 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f9319000640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556719051800 0x556718289180 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x55671843d920 tx=0x556719004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2dc65b9dbee4b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f930dc00640 1 -- 172.21.15.69:0/2089703857 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x556718138820 con 0x556719051800 2024-04-24T08:21:12.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f930dc00640 1 -- 172.21.15.69:0/2089703857 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55671813e1c0 con 0x556719051800 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f930dc00640 1 -- 172.21.15.69:0/2089703857 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55671907e1a0 con 0x556719051800 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567181463c0 con 0x556719051800 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x556718146960 con 0x556719051800 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f931b926280 4 waiting for MDS map... 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f930dc00640 1 -- 172.21.15.69:0/2089703857 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x5567184e8000 con 0x556719051800 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.661+0000 7f930dc00640 1 -- 172.21.15.69:0/2089703857 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x55671816d500 con 0x556719051800 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.662+0000 7f931b926280 4 Got MDS map 106 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.662+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x5567184e7400 0x5567184fe000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.662+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:1 16.f 16:f1f69726:::400.00000000:head [writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x5567184e7800 con 0x5567184e7400 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.662+0000 7f9319a00640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x5567184e7400 0x5567184fe000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.662+0000 7f9319a00640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x5567184e7400 0x5567184fe000 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=cca06fdc3fa14bfa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.666+0000 7f9319a00640 1 -- 172.21.15.69:0/2089703857 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== osd_op_reply(1 400.00000000 [writefull 0~22] v113'2 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x556718e9c000 con 0x5567184e7400 2024-04-24T08:21:12.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.666+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:2 16.14 16:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x5567184e7c00 con 0x5567184e7400 2024-04-24T08:21:12.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.667+0000 7f9319a00640 1 -- 172.21.15.69:0/2089703857 <== osd.0 v2:172.21.15.69:6800/146737935 2 ==== osd_op_reply(2 200.00000000 [writefull 0~90] v113'42 uv42 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x556718e9c000 con 0x5567184e7400 2024-04-24T08:21:12.668 INFO:teuthology.orchestra.run.smithi069.stdout:writing EResetJournal entry 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.667+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:3 16.14 16:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x556718f14000 con 0x5567184e7400 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.667+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556718f14400 0x5567184fe580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.667+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:4 16.1b 16:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x556718f14800 con 0x556718f14400 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.667+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x556718f14c00 0x5567184feb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.667+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:5 16.b 16:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x556718f15000 con 0x556718f14c00 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.668+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:6 16.1f 16:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x556718f15400 con 0x556718f14400 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.668+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x556718f15800 0x5567184ff080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.668+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:7 16.1a 16:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x556718f15c00 con 0x556718f15800 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.668+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55671850a000 0x5567184ff600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.668+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:8 16.1e 16:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55671850a400 con 0x55671850a000 2024-04-24T08:21:12.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.668+0000 7f9319a00640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x556718f14c00 0x5567184feb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.668+0000 7f9319000640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x556718f15800 0x5567184ff080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.668+0000 7f9318600640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556718f14400 0x5567184fe580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.668+0000 7f9319a00640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x556718f14c00 0x5567184feb00 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=f39ef74dc4843294 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.669+0000 7f9318600640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55671850a000 0x5567184ff600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.669+0000 7f9319000640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x556718f15800 0x5567184ff080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=14feebe204652c84 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.669+0000 7f9319a00640 1 -- 172.21.15.69:0/2089703857 <== osd.0 v2:172.21.15.69:6800/146737935 3 ==== osd_op_reply(3 200.00000001 [delete] v113'43 uv43 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x556718e9c000 con 0x5567184e7400 2024-04-24T08:21:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.670+0000 7f9318600640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55671850a000 0x5567184ff600 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=e66b3bf0faaf6527 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.670+0000 7f9318600640 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556718f14400 0x5567184fe580 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=93039e1f58f835f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.670+0000 7f9319a00640 1 -- 172.21.15.69:0/2089703857 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(5 200.00000003 [delete] v113'6 uv5 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x556718e9c000 con 0x556718f14c00 2024-04-24T08:21:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.671+0000 7f9319000640 1 -- 172.21.15.69:0/2089703857 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(7 200.00000005 [delete] v113'3 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x556718e9c500 con 0x556718f15800 2024-04-24T08:21:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.673+0000 7f9318600640 1 -- 172.21.15.69:0/2089703857 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(4 200.00000002 [delete] v113'3 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x556718e9c780 con 0x556718f14400 2024-04-24T08:21:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.673+0000 7f930be00640 1 -- 172.21.15.69:0/2089703857 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:9 16.14 16:2e2fa760:::200.00000001:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x556718feb000 con 0x5567184e7400 2024-04-24T08:21:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f9318600640 1 -- 172.21.15.69:0/2089703857 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== osd_op_reply(8 200.00000006 [delete] v113'3 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x556718e9c780 con 0x55671850a000 2024-04-24T08:21:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f9319a00640 1 -- 172.21.15.69:0/2089703857 <== osd.0 v2:172.21.15.69:6800/146737935 4 ==== osd_op_reply(9 200.00000001 [write 0~48] v113'44 uv44 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x556718e9c000 con 0x5567184e7400 2024-04-24T08:21:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f9318600640 1 -- 172.21.15.69:0/2089703857 <== osd.5 v2:172.21.15.121:6822/2187246263 2 ==== osd_op_reply(6 200.00000004 [delete] v113'9 uv8 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x556718e9c780 con 0x556718f14400 2024-04-24T08:21:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 4 Successfully wrote new journal pointer and header for rank 8:0 2024-04-24T08:21:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x5567184e7400 msgr2=0x5567184fe000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x5567184e7400 0x5567184fe000 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x556718f14c00 msgr2=0x5567184feb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x556718f14c00 0x5567184feb00 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x556718f15800 msgr2=0x5567184ff080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x556718f15800 0x5567184ff080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556718f14400 msgr2=0x5567184fe580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f9318600640 1 -- 172.21.15.69:0/2089703857 reap_dead start 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556718f14400 0x5567184fe580 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55671850a000 msgr2=0x5567184ff600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55671850a000 0x5567184ff600 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556719051800 msgr2=0x556718289180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556719051800 0x556718289180 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x55671843d920 tx=0x556719004c60 comp rx=0 tx=0).stop 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 shutdown_connections 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55671850a000 0x5567184ff600 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x556718f14400 0x5567184fe580 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 --2- 172.21.15.69:0/2089703857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556719051800 0x556718289180 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.674+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 >> 172.21.15.69:0/2089703857 conn(0x556719051000 msgr2=0x556718e99000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 shutdown_connections 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 -- 172.21.15.69:0/2089703857 wait complete. 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 -- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556719050c00 msgr2=0x556718288680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556719050c00 0x556718288680 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x556719002660 tx=0x556719004c00 comp rx=0 tx=0).stop 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 -- 172.21.15.69:0/2877911732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718eaa000 msgr2=0x556718287600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718eaa000 0x556718287600 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x556718142e70 tx=0x556718eac6c0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 -- 172.21.15.69:0/2877911732 shutdown_connections 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x556719050c00 0x556718288680 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718febc00 0x556718288100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718feb800 0x556718287b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 --2- 172.21.15.69:0/2877911732 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718eaa000 0x556718287600 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 -- 172.21.15.69:0/2877911732 >> 172.21.15.69:0/2877911732 conn(0x556718eaa400 msgr2=0x556718e99800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 -- 172.21.15.69:0/2877911732 shutdown_connections 2024-04-24T08:21:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.675+0000 7f931b926280 1 -- 172.21.15.69:0/2877911732 wait complete. 2024-04-24T08:21:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.676+0000 7f931b926280 1 -- 172.21.15.69:0/1825560346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718113c00 msgr2=0x556718286b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.676+0000 7f931b926280 1 --2- 172.21.15.69:0/1825560346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718113c00 0x556718286b00 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x556718f3f3e0 tx=0x556718eae180 comp rx=0 tx=0).stop 2024-04-24T08:21:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.676+0000 7f931b926280 1 -- 172.21.15.69:0/1825560346 shutdown_connections 2024-04-24T08:21:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.676+0000 7f931b926280 1 --2- 172.21.15.69:0/1825560346 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556718113c00 0x556718286b00 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.676+0000 7f931b926280 1 --2- 172.21.15.69:0/1825560346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556718eabc00 0x556718286580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.676+0000 7f931b926280 1 --2- 172.21.15.69:0/1825560346 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x556718eaa800 0x556718287080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.676+0000 7f931b926280 1 -- 172.21.15.69:0/1825560346 >> 172.21.15.69:0/1825560346 conn(0x556718113000 msgr2=0x556718e98000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.676+0000 7f931b926280 1 -- 172.21.15.69:0/1825560346 shutdown_connections 2024-04-24T08:21:12.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.676+0000 7f931b926280 1 -- 172.21.15.69:0/1825560346 wait complete. 2024-04-24T08:21:12.681 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset', '--force'] in time 0:00:00.122802, result: writing EResetJournal entry 2024-04-24T08:21:12.682 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-04-24T08:21:12.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.818+0000 7f1be401a740 1 -- start start 2024-04-24T08:21:12.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.818+0000 7f1be401a740 1 Processor -- start 2024-04-24T08:21:12.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.819+0000 7f1be401a740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701cb0800 0x55d701ce7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.819+0000 7f1be401a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1c00 0x55d701ce6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.819+0000 7f1be401a740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701cb1400 0x55d701ce6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.819+0000 7f1be401a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d701d06000 con 0x55d701cb1c00 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.819+0000 7f1be401a740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d701d06180 con 0x55d701cb1400 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.819+0000 7f1be401a740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55d701d06300 con 0x55d701cb0800 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1c00 0x55d701ce6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1c00 0x55d701ce6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53144/0 (socket says 172.21.15.69:53144) 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be0c00640 1 -- 172.21.15.69:0/993521612 learned_addr learned my addr 172.21.15.69:0/993521612 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be1600640 1 --2- 172.21.15.69:0/993521612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701cb0800 0x55d701ce7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be0c00640 1 -- 172.21.15.69:0/993521612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701cb0800 msgr2=0x55d701ce7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be0c00640 1 --2- 172.21.15.69:0/993521612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701cb0800 0x55d701ce7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be0c00640 1 -- 172.21.15.69:0/993521612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701cb1400 msgr2=0x55d701ce6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be2000640 1 --2- 172.21.15.69:0/993521612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701cb1400 0x55d701ce6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be0c00640 1 --2- 172.21.15.69:0/993521612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701cb1400 0x55d701ce6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be0c00640 1 -- 172.21.15.69:0/993521612 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d700f3f680 con 0x55d701cb1c00 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be1600640 1 --2- 172.21.15.69:0/993521612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701cb0800 0x55d701ce7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.820+0000 7f1be0c00640 1 --2- 172.21.15.69:0/993521612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1c00 0x55d701ce6b00 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x55d701d3fc80 tx=0x55d701cb60c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927f203be6406728 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1bde400640 1 -- 172.21.15.69:0/993521612 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d700f38d00 con 0x55d701cb1c00 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1bde400640 1 -- 172.21.15.69:0/993521612 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d700f5c540 con 0x55d701cb1c00 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1bde400640 1 -- 172.21.15.69:0/993521612 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d700f38680 con 0x55d701cb1c00 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1be401a740 1 -- 172.21.15.69:0/993521612 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d700f3f2c0 con 0x55d701cb1c00 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1be401a740 1 -- 172.21.15.69:0/993521612 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55d700f3eb40 con 0x55d701cb1c00 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1be401a740 4 waiting for MDS map... 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1bde400640 1 -- 172.21.15.69:0/993521612 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x55d701dea600 con 0x55d701cb1c00 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1bde400640 1 -- 172.21.15.69:0/993521612 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x55d700f6b180 con 0x55d701cb1c00 2024-04-24T08:21:12.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1be401a740 4 Got MDS map 106 2024-04-24T08:21:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1be401a740 4 datascan.main: Using metadata pool output 2024-04-24T08:21:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.821+0000 7f1be401a740 4 datascan.main: connecting to RADOS... 2024-04-24T08:21:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.822+0000 7f1be401a740 1 Processor -- start 2024-04-24T08:21:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.822+0000 7f1be401a740 1 -- start start 2024-04-24T08:21:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be401a740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701cb0400 0x55d701ce7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be401a740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701decc00 0x55d701ce7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be401a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701ded000 0x55d701ce8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be401a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d701d06480 con 0x55d701ded000 2024-04-24T08:21:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be401a740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d701d06600 con 0x55d701cb0400 2024-04-24T08:21:12.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be401a740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55d701d06780 con 0x55d701decc00 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701ded000 0x55d701ce8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701ded000 0x55d701ce8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53148/0 (socket says 172.21.15.69:53148) 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be0c00640 1 -- 172.21.15.69:0/878125308 learned_addr learned my addr 172.21.15.69:0/878125308 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be0c00640 1 -- 172.21.15.69:0/878125308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701decc00 msgr2=0x55d701ce7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be0c00640 1 --2- 172.21.15.69:0/878125308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701decc00 0x55d701ce7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be0c00640 1 -- 172.21.15.69:0/878125308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701cb0400 msgr2=0x55d701ce7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be0c00640 1 --2- 172.21.15.69:0/878125308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701cb0400 0x55d701ce7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.823+0000 7f1be0c00640 1 -- 172.21.15.69:0/878125308 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d700f3e3c0 con 0x55d701ded000 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1be0c00640 1 --2- 172.21.15.69:0/878125308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701ded000 0x55d701ce8100 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x55d701d0ab10 tx=0x55d701cb29c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2385470a77e19045 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1bdbc00640 1 -- 172.21.15.69:0/878125308 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d700f38820 con 0x55d701ded000 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1bdbc00640 1 -- 172.21.15.69:0/878125308 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d700f5cc40 con 0x55d701ded000 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1bdbc00640 1 -- 172.21.15.69:0/878125308 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d700f38d00 con 0x55d701ded000 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1be401a740 1 -- 172.21.15.69:0/878125308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701ded000 msgr2=0x55d701ce8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1be401a740 1 --2- 172.21.15.69:0/878125308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701ded000 0x55d701ce8100 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x55d701d0ab10 tx=0x55d701cb29c0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1be401a740 1 -- 172.21.15.69:0/878125308 shutdown_connections 2024-04-24T08:21:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1be401a740 1 --2- 172.21.15.69:0/878125308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701ded000 0x55d701ce8100 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1be401a740 1 --2- 172.21.15.69:0/878125308 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701decc00 0x55d701ce7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1be401a740 1 --2- 172.21.15.69:0/878125308 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701cb0400 0x55d701ce7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.824+0000 7f1be401a740 1 -- 172.21.15.69:0/878125308 >> 172.21.15.69:0/878125308 conn(0x55d701cb1800 msgr2=0x55d700fe1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.825+0000 7f1be401a740 1 -- 172.21.15.69:0/878125308 shutdown_connections 2024-04-24T08:21:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.825+0000 7f1be401a740 1 -- 172.21.15.69:0/878125308 wait complete. 2024-04-24T08:21:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.825+0000 7f1be401a740 1 Processor -- start 2024-04-24T08:21:12.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.825+0000 7f1be401a740 1 -- start start 2024-04-24T08:21:12.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.826+0000 7f1be401a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1800 0x55d701ce7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.826+0000 7f1be401a740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701dedc00 0x55d701ce7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.826+0000 7f1be401a740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701ded800 0x55d701ce8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.826+0000 7f1be401a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d701d07e00 con 0x55d701cb1800 2024-04-24T08:21:12.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.826+0000 7f1be401a740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55d701d07c80 con 0x55d701dedc00 2024-04-24T08:21:12.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.826+0000 7f1be401a740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55d701d07b00 con 0x55d701ded800 2024-04-24T08:21:12.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.826+0000 7f1be2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1800 0x55d701ce7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.826+0000 7f1be2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1800 0x55d701ce7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53156/0 (socket says 172.21.15.69:53156) 2024-04-24T08:21:12.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.826+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 learned_addr learned my addr 172.21.15.69:0/1027581426 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:12.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701ded800 msgr2=0x55d701ce8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1be2000640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701ded800 0x55d701ce8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701dedc00 msgr2=0x55d701ce7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1be2000640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701dedc00 0x55d701ce7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d700f3f680 con 0x55d701cb1800 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1be2000640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1800 0x55d701ce7600 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x55d701cb4630 tx=0x55d701cb3f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1bd9e00640 1 -- 172.21.15.69:0/1027581426 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d700f38b60 con 0x55d701cb1800 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1bd9e00640 1 -- 172.21.15.69:0/1027581426 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d700f5d340 con 0x55d701cb1800 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d700f3f2c0 con 0x55d701cb1800 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1bd9e00640 1 -- 172.21.15.69:0/1027581426 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d700f38820 con 0x55d701cb1800 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d700f3fa40 con 0x55d701cb1800 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.827+0000 7f1be401a740 4 datascan.init: resolving metadata pool 16 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.829+0000 7f1bd9e00640 1 -- 172.21.15.69:0/1027581426 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55d701e52c00 con 0x55d701cb1800 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.829+0000 7f1bd9e00640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55d701284000 0x55d701ce8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.829+0000 7f1bd9e00640 1 -- 172.21.15.69:0/1027581426 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x55d701e5e800 con 0x55d701cb1800 2024-04-24T08:21:12.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.829+0000 7f1be401a740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-24T08:21:12.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.829+0000 7f1be401a740 4 datascan.main: data pool 'cephfs_data' has ID 17 2024-04-24T08:21:12.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.829+0000 7f1be401a740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-24T08:21:12.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.830+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55d701ded400 0x55d701ce8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.830+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701ded000 con 0x55d701ded400 2024-04-24T08:21:12.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.830+0000 7f1be1600640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55d701284000 0x55d701ce8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.830+0000 7f1be0c00640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55d701ded400 0x55d701ce8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.831+0000 7f1be0c00640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55d701ded400 0x55d701ce8c00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.831+0000 7f1be1600640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55d701284000 0x55d701ce8680 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x55d701cb51a0 tx=0x55d701dfd0e0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.832+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== osd_op_reply(1 [pgnls start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.832+0000 7f1bdb200640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55d701285000 0x55d70129e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.832+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:2 17.20 17:04000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701285400 con 0x55d701285000 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.833+0000 7f1be2000640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55d701285000 0x55d70129e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.833+0000 7f1be2000640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55d701285000 0x55d70129e000 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.834+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(2 [pgnls start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d701285000 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.834+0000 7f1bdbc00640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55d701285c00 0x55d70129e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.834+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:3 17.10 17:08000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2000 con 0x55d701285c00 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.834+0000 7f1be1600640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55d701285c00 0x55d70129e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.834+0000 7f1be1600640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55d701285c00 0x55d70129e580 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.835+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(3 [pgnls start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.835+0000 7f1bdb200640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55d701285800 0x55d70129eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.835+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:4 17.30 17:0c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2800 con 0x55d701285800 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.835+0000 7f1be0c00640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55d701285800 0x55d70129eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.836+0000 7f1be0c00640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55d701285800 0x55d70129eb00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.836+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== osd_op_reply(4 [pgnls start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.836+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:5 17.8 17:10000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2400 con 0x55d701ded400 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.837+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 2 ==== osd_op_reply(5 [pgnls start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.837+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:6 17.28 17:14000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2c00 con 0x55d701285c00 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.837+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 2 ==== osd_op_reply(6 [pgnls start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.837+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:7 17.18 17:18000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e50800 con 0x55d701285800 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.838+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 2 ==== osd_op_reply(7 [pgnls start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.838+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:8 17.38 17:1c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a3000 con 0x55d701ded400 2024-04-24T08:21:12.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.839+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 3 ==== osd_op_reply(8 [pgnls start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.839+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:9 17.4 17:20000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e50000 con 0x55d701285800 2024-04-24T08:21:12.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.839+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 3 ==== osd_op_reply(9 [pgnls start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.839+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:10 17.24 17:24000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2800 con 0x55d701285c00 2024-04-24T08:21:12.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.839+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 3 ==== osd_op_reply(10 [pgnls start_epoch 113 out=49b] v109'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.839+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:11 17.14 17:28000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e50400 con 0x55d701285c00 2024-04-24T08:21:12.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.840+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 4 ==== osd_op_reply(11 [pgnls start_epoch 113 out=49b] v109'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.840+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:12 17.34 17:2c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a3400 con 0x55d701285800 2024-04-24T08:21:12.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.840+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 4 ==== osd_op_reply(12 [pgnls start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.840+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:13 17.c 17:30000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2000 con 0x55d701ded400 2024-04-24T08:21:12.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.841+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 4 ==== osd_op_reply(13 [pgnls start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.841+0000 7f1bdb200640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55d7012a3800 0x55d70129f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.841+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:14 17.2c 17:34000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701ded000 con 0x55d7012a3800 2024-04-24T08:21:12.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.842+0000 7f1be2000640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55d7012a3800 0x55d70129f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.843+0000 7f1be2000640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55d7012a3800 0x55d70129f080 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.843+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(14 [pgnls start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.843+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:15 17.1c 17:38000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2c00 con 0x55d701285000 2024-04-24T08:21:12.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.844+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.1 v2:172.21.15.69:6815/153736297 2 ==== osd_op_reply(15 [pgnls start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d701285000 2024-04-24T08:21:12.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.844+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:16 17.3c 17:3c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e51c00 con 0x55d701285800 2024-04-24T08:21:12.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.844+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 5 ==== osd_op_reply(16 [pgnls start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.844+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:17 17.2 17:40000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2800 con 0x55d7012a3800 2024-04-24T08:21:12.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.845+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== osd_op_reply(17 [pgnls start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.845+0000 7f1bdb200640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55d701e51800 0x55d70129f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.845+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:18 17.22 17:44000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e51400 con 0x55d701e51800 2024-04-24T08:21:12.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.846+0000 7f1be1600640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55d701e51800 0x55d70129f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.847+0000 7f1be1600640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55d701e51800 0x55d70129f600 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.847+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(18 [pgnls start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701e51800 2024-04-24T08:21:12.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.848+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:19 17.12 17:48000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701284c00 con 0x55d701285c00 2024-04-24T08:21:12.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.848+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 5 ==== osd_op_reply(19 [pgnls start_epoch 113 out=81b] v112'22 uv22 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.848+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:20 17.12 17:4bec3886:::10000000002.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55d701decc00 con 0x55d701285c00 2024-04-24T08:21:12.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.848+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 6 ==== osd_op_reply(20 10000000002.00000000 [stat out=16b] v0'0 uv22 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.848+0000 7f1be401a740 10 datascan.operator(): handling object 1099511627778.0 2024-04-24T08:21:12.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.848+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:21 17.12 17:4bec3886:::10000000002.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2c00 con 0x55d701285c00 2024-04-24T08:21:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.852+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 7 ==== osd_op_reply(21 10000000002.00000000 [call] v113'23 uv23 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.852+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:22 17.32 17:4c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2400 con 0x55d701ded400 2024-04-24T08:21:12.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.853+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 5 ==== osd_op_reply(22 [pgnls start_epoch 113 out=49b] v109'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.853+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:23 17.a 17:50000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2c00 con 0x55d701ded400 2024-04-24T08:21:12.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.854+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 6 ==== osd_op_reply(23 [pgnls start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.854+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:24 17.2a 17:54000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701ded000 con 0x55d701285c00 2024-04-24T08:21:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.855+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 8 ==== osd_op_reply(24 [pgnls start_epoch 113 out=49b] v109'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.855+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:25 17.1a 17:58000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701decc00 con 0x55d701285800 2024-04-24T08:21:12.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.855+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 6 ==== osd_op_reply(25 [pgnls start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.855+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:26 17.3a 17:5c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701285400 con 0x55d701285000 2024-04-24T08:21:12.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.856+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.1 v2:172.21.15.69:6815/153736297 3 ==== osd_op_reply(26 [pgnls start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d701285000 2024-04-24T08:21:12.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.856+0000 7f1bdbc00640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55d701284c00 0x55d7012a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.856+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:27 17.6 17:60000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e50400 con 0x55d701284c00 2024-04-24T08:21:12.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.856+0000 7f1be0c00640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55d701284c00 0x55d7012a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.857+0000 7f1be0c00640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55d701284c00 0x55d7012a0680 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.858+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(27 [pgnls start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701284c00 2024-04-24T08:21:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.858+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:28 17.26 17:64000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a3c00 con 0x55d701285800 2024-04-24T08:21:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.858+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 7 ==== osd_op_reply(28 [pgnls start_epoch 113 out=49b] v109'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.859+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:29 17.16 17:68000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e51400 con 0x55d701e51800 2024-04-24T08:21:12.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.859+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.5 v2:172.21.15.121:6822/2187246263 2 ==== osd_op_reply(29 [pgnls start_epoch 113 out=49b] v109'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701e51800 2024-04-24T08:21:12.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.860+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:30 17.36 17:6c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c6000 con 0x55d701285800 2024-04-24T08:21:12.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.860+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 8 ==== osd_op_reply(30 [pgnls start_epoch 113 out=81b] v109'33 uv17 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.860+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:31 17.36 17:6fe38dc2:::100000001f7.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a3000 con 0x55d701285800 2024-04-24T08:21:12.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.861+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 9 ==== osd_op_reply(31 100000001f7.00000000 [stat out=16b] v0'0 uv17 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.861+0000 7f1be401a740 10 datascan.operator(): handling object 1099511628279.0 2024-04-24T08:21:12.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.861+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:32 17.36 17:6fe38dc2:::100000001f7.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a3400 con 0x55d701285800 2024-04-24T08:21:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.863+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 10 ==== osd_op_reply(32 100000001f7.00000000 [call] v113'34 uv34 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.863+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55d7012a2000 0x55d701ce9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.863+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:33 17.e 17:70000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e51000 con 0x55d7012a2000 2024-04-24T08:21:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.863+0000 7f1be2000640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55d7012a2000 0x55d701ce9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:12.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.864+0000 7f1be2000640 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55d7012a2000 0x55d701ce9180 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.864+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== osd_op_reply(33 [pgnls start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a2000 2024-04-24T08:21:12.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.864+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:34 17.2e 17:74000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c6400 con 0x55d701284c00 2024-04-24T08:21:12.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.865+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.6 v2:172.21.15.121:6807/3068198325 2 ==== osd_op_reply(34 [pgnls start_epoch 113 out=49b] v109'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701284c00 2024-04-24T08:21:12.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.866+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:35 17.1e 17:78000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2400 con 0x55d701285000 2024-04-24T08:21:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.866+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.1 v2:172.21.15.69:6815/153736297 4 ==== osd_op_reply(35 [pgnls start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d701285000 2024-04-24T08:21:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.866+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:36 17.3e 17:7c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c6800 con 0x55d7012a3800 2024-04-24T08:21:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.867+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 3 ==== osd_op_reply(36 [pgnls start_epoch 113 out=49b] v109'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.867+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:37 17.1 17:80000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2c00 con 0x55d701285800 2024-04-24T08:21:12.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.868+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 11 ==== osd_op_reply(37 [pgnls start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.868+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:38 17.21 17:84000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c6c00 con 0x55d701285800 2024-04-24T08:21:12.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.868+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 12 ==== osd_op_reply(38 [pgnls start_epoch 113 out=49b] v109'24 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.868+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:39 17.11 17:88000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a3c00 con 0x55d701285c00 2024-04-24T08:21:12.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.869+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 9 ==== osd_op_reply(39 [pgnls start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.869+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:40 17.31 17:8c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c7000 con 0x55d701285000 2024-04-24T08:21:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.869+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.1 v2:172.21.15.69:6815/153736297 5 ==== osd_op_reply(40 [pgnls start_epoch 113 out=81b] v109'15 uv8 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55d701ca0500 con 0x55d701285000 2024-04-24T08:21:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.869+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:41 17.31 17:8de29a03:::10000000004.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55d701e50800 con 0x55d701285000 2024-04-24T08:21:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.870+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.1 v2:172.21.15.69:6815/153736297 6 ==== osd_op_reply(41 10000000004.00000001 [stat out=16b] v0'0 uv8 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x55d701ca0500 con 0x55d701285000 2024-04-24T08:21:12.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.870+0000 7f1be401a740 10 datascan.operator(): handling object 1099511627780.1 2024-04-24T08:21:12.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.870+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:42 17.5 17:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55d701e50000 con 0x55d701284c00 2024-04-24T08:21:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.874+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.6 v2:172.21.15.121:6807/3068198325 3 ==== osd_op_reply(42 10000000004.00000000 [call] v113'14 uv14 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55d701ca0000 con 0x55d701284c00 2024-04-24T08:21:12.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.874+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:43 17.9 17:90000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c6000 con 0x55d701285c00 2024-04-24T08:21:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.874+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 10 ==== osd_op_reply(43 [pgnls start_epoch 113 out=49b] v109'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.874+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:44 17.29 17:94000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c7400 con 0x55d7012a3800 2024-04-24T08:21:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.875+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 4 ==== osd_op_reply(44 [pgnls start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.875+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:45 17.19 17:98000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e51c00 con 0x55d701e51800 2024-04-24T08:21:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.877+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.5 v2:172.21.15.121:6822/2187246263 3 ==== osd_op_reply(45 [pgnls start_epoch 113 out=81b] v112'23 uv23 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55d701ca0780 con 0x55d701e51800 2024-04-24T08:21:12.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.877+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:46 17.19 17:997b7129:::100000001f8.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c7c00 con 0x55d701e51800 2024-04-24T08:21:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.878+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.5 v2:172.21.15.121:6822/2187246263 4 ==== osd_op_reply(46 100000001f8.00000000 [stat out=16b] v0'0 uv23 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x55d701ca0780 con 0x55d701e51800 2024-04-24T08:21:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.878+0000 7f1be401a740 10 datascan.operator(): handling object 1099511628280.0 2024-04-24T08:21:12.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.878+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:47 17.19 17:997b7129:::100000001f8.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c6000 con 0x55d701e51800 2024-04-24T08:21:12.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.882+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.5 v2:172.21.15.121:6822/2187246263 5 ==== osd_op_reply(47 100000001f8.00000000 [call] v113'24 uv24 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55d701ca0780 con 0x55d701e51800 2024-04-24T08:21:12.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.882+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:48 17.39 17:9c000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e51400 con 0x55d701e51800 2024-04-24T08:21:12.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.883+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.5 v2:172.21.15.121:6822/2187246263 6 ==== osd_op_reply(48 [pgnls start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701e51800 2024-04-24T08:21:12.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.883+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:49 17.5 17:a0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e50400 con 0x55d701284c00 2024-04-24T08:21:12.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.885+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.6 v2:172.21.15.121:6807/3068198325 4 ==== osd_op_reply(49 [pgnls start_epoch 113 out=81b] v113'14 uv14 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55d701ca0000 con 0x55d701284c00 2024-04-24T08:21:12.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.885+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:50 17.5 17:a367a3e8:::10000000004.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55d701e51c00 con 0x55d701284c00 2024-04-24T08:21:12.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.886+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.6 v2:172.21.15.121:6807/3068198325 5 ==== osd_op_reply(50 10000000004.00000000 [stat out=16b] v0'0 uv14 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x55d701ca0000 con 0x55d701284c00 2024-04-24T08:21:12.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.886+0000 7f1be401a740 10 datascan.operator(): handling object 1099511627780.0 2024-04-24T08:21:12.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.886+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:51 17.5 17:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55d701e51000 con 0x55d701284c00 2024-04-24T08:21:12.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.890+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.6 v2:172.21.15.121:6807/3068198325 6 ==== osd_op_reply(51 10000000004.00000000 [call] v113'15 uv15 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55d701ca0000 con 0x55d701284c00 2024-04-24T08:21:12.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.890+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:52 17.25 17:a4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e50800 con 0x55d701285000 2024-04-24T08:21:12.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.891+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.1 v2:172.21.15.69:6815/153736297 7 ==== osd_op_reply(52 [pgnls start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d701285000 2024-04-24T08:21:12.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.891+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:53 17.15 17:a8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701decc00 con 0x55d701e51800 2024-04-24T08:21:12.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.892+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.5 v2:172.21.15.121:6822/2187246263 7 ==== osd_op_reply(53 [pgnls start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701e51800 2024-04-24T08:21:12.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.892+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:54 17.35 17:ac000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c7800 con 0x55d7012a3800 2024-04-24T08:21:12.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.893+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 5 ==== osd_op_reply(54 [pgnls start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.893+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:55 17.d 17:b0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c6400 con 0x55d701ded400 2024-04-24T08:21:12.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.894+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 7 ==== osd_op_reply(55 [pgnls start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.894+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:56 17.2d 17:b4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c6c00 con 0x55d701ded400 2024-04-24T08:21:12.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.895+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 8 ==== osd_op_reply(56 [pgnls start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.895+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:57 17.1d 17:b8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e50000 con 0x55d701ded400 2024-04-24T08:21:12.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.896+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 9 ==== osd_op_reply(57 [pgnls start_epoch 113 out=49b] v109'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.896+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:58 17.3d 17:bc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c7c00 con 0x55d701ded400 2024-04-24T08:21:12.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.897+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 10 ==== osd_op_reply(58 [pgnls start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.898+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:59 17.3 17:c0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e50400 con 0x55d7012a3800 2024-04-24T08:21:12.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.899+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 6 ==== osd_op_reply(59 [pgnls start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.899+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:60 17.23 17:c4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c6800 con 0x55d7012a3800 2024-04-24T08:21:12.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.900+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 7 ==== osd_op_reply(60 [pgnls start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.900+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:61 17.13 17:c8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d701e51c00 con 0x55d701285c00 2024-04-24T08:21:12.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.901+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 <== osd.3 v2:172.21.15.69:6811/1875112303 11 ==== osd_op_reply(61 [pgnls start_epoch 113 out=49b] v109'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0780 con 0x55d701285c00 2024-04-24T08:21:12.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.901+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:62 17.33 17:cc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c7000 con 0x55d701ded400 2024-04-24T08:21:12.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.902+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.7 v2:172.21.15.121:6811/176144805 11 ==== osd_op_reply(62 [pgnls start_epoch 113 out=49b] v109'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701ded400 2024-04-24T08:21:12.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.902+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:63 17.b 17:d0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a3000 con 0x55d701285800 2024-04-24T08:21:12.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.903+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 13 ==== osd_op_reply(63 [pgnls start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.903+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:64 17.2b 17:d4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012c7400 con 0x55d7012a3800 2024-04-24T08:21:12.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.904+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 8 ==== osd_op_reply(64 [pgnls start_epoch 113 out=49b] v109'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.905+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:65 17.1b 17:d8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a3400 con 0x55d7012a3800 2024-04-24T08:21:12.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.906+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 9 ==== osd_op_reply(65 [pgnls start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.906+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:66 17.3b 17:dc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012dc000 con 0x55d7012a2000 2024-04-24T08:21:12.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.906+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.2 v2:172.21.15.69:6808/2630808717 2 ==== osd_op_reply(66 [pgnls start_epoch 113 out=49b] v109'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a2000 2024-04-24T08:21:12.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.907+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:67 17.7 17:e0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012a2c00 con 0x55d701285800 2024-04-24T08:21:12.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.908+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 14 ==== osd_op_reply(67 [pgnls start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.908+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:68 17.27 17:e4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012dc400 con 0x55d7012a3800 2024-04-24T08:21:12.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.909+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 10 ==== osd_op_reply(68 [pgnls start_epoch 113 out=49b] v109'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.909+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:69 17.17 17:e8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012de000 con 0x55d701284c00 2024-04-24T08:21:12.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.910+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.6 v2:172.21.15.121:6807/3068198325 7 ==== osd_op_reply(69 [pgnls start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701284c00 2024-04-24T08:21:12.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.910+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:70 17.37 17:ec000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012dc800 con 0x55d7012a2000 2024-04-24T08:21:12.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.911+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.2 v2:172.21.15.69:6808/2630808717 3 ==== osd_op_reply(70 [pgnls start_epoch 113 out=49b] v109'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a2000 2024-04-24T08:21:12.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.911+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:71 17.f 17:f0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012de400 con 0x55d701285000 2024-04-24T08:21:12.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.912+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.1 v2:172.21.15.69:6815/153736297 8 ==== osd_op_reply(71 [pgnls start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d701285000 2024-04-24T08:21:12.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.912+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:72 17.2f 17:f4000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012dcc00 con 0x55d7012a3800 2024-04-24T08:21:12.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.913+0000 7f1be2000640 1 -- 172.21.15.69:0/1027581426 <== osd.4 v2:172.21.15.121:6800/2274643300 11 ==== osd_op_reply(72 [pgnls start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0500 con 0x55d7012a3800 2024-04-24T08:21:12.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.913+0000 7f1bdbc00640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:73 17.1f 17:f8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012de800 con 0x55d701284c00 2024-04-24T08:21:12.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.915+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.6 v2:172.21.15.121:6807/3068198325 8 ==== osd_op_reply(73 [pgnls start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701284c00 2024-04-24T08:21:12.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.915+0000 7f1bdb200640 1 -- 172.21.15.69:0/1027581426 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:74 17.3f 17:fc000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55d7012dd000 con 0x55d701285800 2024-04-24T08:21:12.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be0c00640 1 -- 172.21.15.69:0/1027581426 <== osd.0 v2:172.21.15.69:6800/146737935 15 ==== osd_op_reply(74 [pgnls start_epoch 113 out=49b] v109'17 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55d701ca0000 con 0x55d701285800 2024-04-24T08:21:12.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55d701285800 msgr2=0x55d70129eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55d701285800 0x55d70129eb00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55d701285000 msgr2=0x55d70129e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55d701285000 0x55d70129e000 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55d7012a2000 msgr2=0x55d701ce9180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55d7012a2000 0x55d701ce9180 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55d701285c00 msgr2=0x55d70129e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55d701285c00 0x55d70129e580 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55d7012a3800 msgr2=0x55d70129f080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55d7012a3800 0x55d70129f080 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55d701e51800 msgr2=0x55d70129f600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55d701e51800 0x55d70129f600 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 reap_dead start 2024-04-24T08:21:12.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55d701284c00 msgr2=0x55d7012a0680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55d701284c00 0x55d7012a0680 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.916+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 reap_dead start 2024-04-24T08:21:12.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 reap_dead start 2024-04-24T08:21:12.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be1600640 1 -- 172.21.15.69:0/1027581426 reap_dead start 2024-04-24T08:21:12.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55d701ded400 msgr2=0x55d701ce8c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55d701ded400 0x55d701ce8c00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55d701284000 msgr2=0x55d701ce8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55d701284000 0x55d701ce8680 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x55d701cb51a0 tx=0x55d701dfd0e0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1800 msgr2=0x55d701ce7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1800 0x55d701ce7600 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x55d701cb4630 tx=0x55d701cb3f20 comp rx=0 tx=0).stop 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 shutdown_connections 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55d701ded400 0x55d701ce8c00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55d701284000 0x55d701ce8680 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 --2- 172.21.15.69:0/1027581426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1800 0x55d701ce7600 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.917+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 >> 172.21.15.69:0/1027581426 conn(0x55d701cb0400 msgr2=0x55d700fe1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.918+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 shutdown_connections 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.918+0000 7f1be401a740 1 -- 172.21.15.69:0/1027581426 wait complete. 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.919+0000 7f1be401a740 1 -- 172.21.15.69:0/993521612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1c00 msgr2=0x55d701ce6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.919+0000 7f1be401a740 1 --2- 172.21.15.69:0/993521612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1c00 0x55d701ce6b00 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x55d701d3fc80 tx=0x55d701cb60c0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.919+0000 7f1be401a740 1 -- 172.21.15.69:0/993521612 shutdown_connections 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.919+0000 7f1be401a740 1 --2- 172.21.15.69:0/993521612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d701cb1c00 0x55d701ce6b00 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.919+0000 7f1be401a740 1 --2- 172.21.15.69:0/993521612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55d701cb1400 0x55d701ce6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.919+0000 7f1be401a740 1 --2- 172.21.15.69:0/993521612 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55d701cb0800 0x55d701ce7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:12.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.919+0000 7f1be401a740 1 -- 172.21.15.69:0/993521612 >> 172.21.15.69:0/993521612 conn(0x55d700f13000 msgr2=0x55d700fe0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:12.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.919+0000 7f1be401a740 1 -- 172.21.15.69:0/993521612 shutdown_connections 2024-04-24T08:21:12.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:12.919+0000 7f1be401a740 1 -- 172.21.15.69:0/993521612 wait complete. 2024-04-24T08:21:12.931 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.249099, result: 2024-04-24T08:21:12.931 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes --filter-tag mytag123 cephfs_data 2024-04-24T08:21:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.035+0000 7f95eab1b740 1 -- start start 2024-04-24T08:21:13.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.035+0000 7f95eab1b740 1 Processor -- start 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.036+0000 7f95eab1b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de32d86c00 0x55de33afc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.036+0000 7f95eab1b740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de32d87800 0x55de33afc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.036+0000 7f95eab1b740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de32d87c00 0x55de33afcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.036+0000 7f95eab1b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55de33b00180 con 0x55de32d86c00 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.036+0000 7f95eab1b740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55de33b00300 con 0x55de32d87800 2024-04-24T08:21:13.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.036+0000 7f95eab1b740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55de33b00480 con 0x55de32d87c00 2024-04-24T08:21:13.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e8200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de32d86c00 0x55de33afc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e8200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de32d86c00 0x55de33afc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53172/0 (socket says 172.21.15.69:53172) 2024-04-24T08:21:13.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e8200640 1 -- 172.21.15.69:0/945994833 learned_addr learned my addr 172.21.15.69:0/945994833 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:13.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e8c00640 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de32d87c00 0x55de33afcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e7800640 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de32d87800 0x55de33afc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e8200640 1 -- 172.21.15.69:0/945994833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de32d87c00 msgr2=0x55de33afcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:13.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e8200640 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de32d87c00 0x55de33afcb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e8200640 1 -- 172.21.15.69:0/945994833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de32d87800 msgr2=0x55de33afc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e8200640 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de32d87800 0x55de33afc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e8200640 1 -- 172.21.15.69:0/945994833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de32db3680 con 0x55de32d86c00 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e7800640 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de32d87800 0x55de33afc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.037+0000 7f95e8c00640 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de32d87c00 0x55de33afcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.038+0000 7f95e8200640 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de32d86c00 0x55de33afc000 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x55de33ba7c50 tx=0x55de33b2a120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6913f3159529f8be server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.038+0000 7f95e5000640 1 -- 172.21.15.69:0/945994833 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55de32dac680 con 0x55de32d86c00 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.038+0000 7f95e5000640 1 -- 172.21.15.69:0/945994833 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55de32dd0e00 con 0x55de32d86c00 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.038+0000 7f95e5000640 1 -- 172.21.15.69:0/945994833 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55de32dacb60 con 0x55de32d86c00 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.038+0000 7f95eab1b740 1 -- 172.21.15.69:0/945994833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de32db3860 con 0x55de32d86c00 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.038+0000 7f95eab1b740 1 -- 172.21.15.69:0/945994833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55de32db32c0 con 0x55de32d86c00 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.038+0000 7f95eab1b740 4 waiting for MDS map... 2024-04-24T08:21:13.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.038+0000 7f95e5000640 1 -- 172.21.15.69:0/945994833 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x55de33c62600 con 0x55de32d86c00 2024-04-24T08:21:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.039+0000 7f95e5000640 1 -- 172.21.15.69:0/945994833 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x55de32ddf180 con 0x55de32d86c00 2024-04-24T08:21:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.039+0000 7f95eab1b740 4 Got MDS map 106 2024-04-24T08:21:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.039+0000 7f95eab1b740 10 datascan.parse_kwarg: Applying tag filter: 'mytag123' 2024-04-24T08:21:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.039+0000 7f95eab1b740 4 datascan.main: Using metadata pool output 2024-04-24T08:21:13.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.039+0000 7f95eab1b740 4 datascan.main: connecting to RADOS... 2024-04-24T08:21:13.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.039+0000 7f95eab1b740 1 Processor -- start 2024-04-24T08:21:13.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.040+0000 7f95eab1b740 1 -- start start 2024-04-24T08:21:13.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.040+0000 7f95eab1b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65400 0x55de33afd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.040+0000 7f95eab1b740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de33c65800 0x55de33afd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.040+0000 7f95eab1b740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de33c65c00 0x55de33afdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.040+0000 7f95eab1b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55de33b00600 con 0x55de33c65400 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.040+0000 7f95eab1b740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55de33b00780 con 0x55de33c65c00 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.040+0000 7f95eab1b740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55de33b00900 con 0x55de33c65800 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e8c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65400 0x55de33afd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e8c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65400 0x55de33afd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53186/0 (socket says 172.21.15.69:53186) 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e8c00640 1 -- 172.21.15.69:0/59589312 learned_addr learned my addr 172.21.15.69:0/59589312 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e8c00640 1 -- 172.21.15.69:0/59589312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de33c65800 msgr2=0x55de33afd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e8c00640 1 --2- 172.21.15.69:0/59589312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de33c65800 0x55de33afd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e8c00640 1 -- 172.21.15.69:0/59589312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de33c65c00 msgr2=0x55de33afdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e8c00640 1 --2- 172.21.15.69:0/59589312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de33c65c00 0x55de33afdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e8c00640 1 -- 172.21.15.69:0/59589312 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de32db34a0 con 0x55de33c65400 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e8c00640 1 --2- 172.21.15.69:0/59589312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65400 0x55de33afd080 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x55de33ba71d0 tx=0x55de33b26a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d3beabfdc7de48 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e2800640 1 -- 172.21.15.69:0/59589312 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55de32dac9c0 con 0x55de33c65400 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e2800640 1 -- 172.21.15.69:0/59589312 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55de32dd0c40 con 0x55de33c65400 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.041+0000 7f95e2800640 1 -- 172.21.15.69:0/59589312 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55de32dac680 con 0x55de33c65400 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.042+0000 7f95eab1b740 1 -- 172.21.15.69:0/59589312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65400 msgr2=0x55de33afd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.042+0000 7f95eab1b740 1 --2- 172.21.15.69:0/59589312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65400 0x55de33afd080 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x55de33ba71d0 tx=0x55de33b26a80 comp rx=0 tx=0).stop 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.042+0000 7f95eab1b740 1 -- 172.21.15.69:0/59589312 shutdown_connections 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.042+0000 7f95eab1b740 1 --2- 172.21.15.69:0/59589312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de33c65c00 0x55de33afdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.042+0000 7f95eab1b740 1 --2- 172.21.15.69:0/59589312 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de33c65800 0x55de33afd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.042+0000 7f95eab1b740 1 --2- 172.21.15.69:0/59589312 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65400 0x55de33afd080 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.042+0000 7f95eab1b740 1 -- 172.21.15.69:0/59589312 >> 172.21.15.69:0/59589312 conn(0x55de33c65000 msgr2=0x55de32e73000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.042+0000 7f95eab1b740 1 -- 172.21.15.69:0/59589312 shutdown_connections 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.042+0000 7f95eab1b740 1 -- 172.21.15.69:0/59589312 wait complete. 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.043+0000 7f95eab1b740 1 Processor -- start 2024-04-24T08:21:13.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.043+0000 7f95eab1b740 1 -- start start 2024-04-24T08:21:13.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.043+0000 7f95eab1b740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de33c65400 0x55de33afd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.043+0000 7f95eab1b740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de33c65800 0x55de33afd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.043+0000 7f95eab1b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65c00 0x55de33afdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.043+0000 7f95eab1b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55de33b01b00 con 0x55de33c65c00 2024-04-24T08:21:13.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.043+0000 7f95eab1b740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x55de33b01980 con 0x55de33c65800 2024-04-24T08:21:13.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.043+0000 7f95eab1b740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x55de33b01800 con 0x55de33c65400 2024-04-24T08:21:13.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.044+0000 7f95e7800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65c00 0x55de33afdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.044+0000 7f95e7800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65c00 0x55de33afdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53200/0 (socket says 172.21.15.69:53200) 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.044+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 learned_addr learned my addr 172.21.15.69:0/1449101575 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.044+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de33c65400 msgr2=0x55de33afd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.044+0000 7f95e7800640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de33c65400 0x55de33afd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.044+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de33c65800 msgr2=0x55de33afd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.044+0000 7f95e7800640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de33c65800 0x55de33afd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.044+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de32db3680 con 0x55de33c65c00 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.044+0000 7f95e7800640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65c00 0x55de33afdb80 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x55de33c73a10 tx=0x55de33c82c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.045+0000 7f95e0a00640 1 -- 172.21.15.69:0/1449101575 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55de32dac9c0 con 0x55de33c65c00 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.045+0000 7f95e0a00640 1 -- 172.21.15.69:0/1449101575 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55de32dd0540 con 0x55de33c65c00 2024-04-24T08:21:13.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.045+0000 7f95e0a00640 1 -- 172.21.15.69:0/1449101575 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55de32dac680 con 0x55de33c65c00 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.045+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55de32db3860 con 0x55de33c65c00 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.045+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de32db30e0 con 0x55de33c65c00 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.045+0000 7f95eab1b740 4 datascan.init: resolving metadata pool 16 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.046+0000 7f95e0a00640 1 -- 172.21.15.69:0/1449101575 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x55de33b86c00 con 0x55de33c65c00 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.047+0000 7f95e0a00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55de33cd6c00 0x55de33afe100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.047+0000 7f95e0a00640 1 -- 172.21.15.69:0/1449101575 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x55de33cd8800 con 0x55de33c65c00 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.047+0000 7f95eab1b740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.047+0000 7f95eab1b740 4 datascan.main: data pool 'cephfs_data' has ID 17 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.047+0000 7f95eab1b740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.047+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55de33cd7000 0x55de33afe680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.047+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:1 16.14 16:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55de33cd7400 con 0x55de33cd7000 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.048+0000 7f95e8200640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55de33cd7000 0x55de33afe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.048+0000 7f95e8c00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55de33cd6c00 0x55de33afe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.048+0000 7f95e8200640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55de33cd7000 0x55de33afe680 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.049+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv38 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.049+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55de33cd7800 0x55de33afec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.049+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:2 16.1f 16:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55de33cd7c00 con 0x55de33cd7800 2024-04-24T08:21:13.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.049+0000 7f95e8c00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55de33cd6c00 0x55de33afe100 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x55de33ba71d0 tx=0x55de33b26a20 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.049+0000 7f95e8c00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55de33cd7800 0x55de33afec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.050+0000 7f95e8c00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55de33cd7800 0x55de33afec00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.051+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x55de33b14500 con 0x55de33cd7800 2024-04-24T08:21:13.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.051+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55de3311c000 0x55de33aff180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.051+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:3 17.0 17:00000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311c400 con 0x55de3311c000 2024-04-24T08:21:13.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.052+0000 7f95e8200640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55de3311c000 0x55de33aff180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.053+0000 7f95e8200640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55de3311c000 0x55de33aff180 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.054+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.054+0000 7f95e1e00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55de3311d800 0x55de3312e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.054+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:4 17.20 17:04000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311dc00 con 0x55de3311d800 2024-04-24T08:21:13.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.055+0000 7f95e7800640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55de3311d800 0x55de3312e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.055+0000 7f95e7800640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55de3311d800 0x55de3312e000 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.055+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(4 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de3311d800 2024-04-24T08:21:13.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.056+0000 7f95e2800640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55de33122000 0x55de3312e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.056+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:5 17.10 17:08000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122400 con 0x55de33122000 2024-04-24T08:21:13.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.056+0000 7f95e8c00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55de33122000 0x55de3312e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.056+0000 7f95e8c00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55de33122000 0x55de3312e580 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.057+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(5 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.057+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:6 17.30 17:0c000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122c00 con 0x55de33cd7000 2024-04-24T08:21:13.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.057+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 2 ==== osd_op_reply(6 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.057+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:7 17.8 17:10000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122800 con 0x55de3311c000 2024-04-24T08:21:13.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.058+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 2 ==== osd_op_reply(7 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.058+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:8 17.28 17:14000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123000 con 0x55de33122000 2024-04-24T08:21:13.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.059+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 2 ==== osd_op_reply(8 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.059+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:9 17.18 17:18000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33c64800 con 0x55de33cd7000 2024-04-24T08:21:13.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.059+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 3 ==== osd_op_reply(9 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.059+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:10 17.38 17:1c000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123400 con 0x55de3311c000 2024-04-24T08:21:13.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.060+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 3 ==== osd_op_reply(10 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.060+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:11 17.4 17:20000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33b25400 con 0x55de33cd7000 2024-04-24T08:21:13.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.061+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 4 ==== osd_op_reply(11 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.061+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:12 17.24 17:24000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122400 con 0x55de33122000 2024-04-24T08:21:13.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.061+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 3 ==== osd_op_reply(12 [pgnls-filter start_epoch 113 out=49b] v109'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.061+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:13 17.14 17:28000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33b25000 con 0x55de33122000 2024-04-24T08:21:13.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.061+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 4 ==== osd_op_reply(13 [pgnls-filter start_epoch 113 out=49b] v109'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.061+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:14 17.34 17:2c000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311c400 con 0x55de33cd7000 2024-04-24T08:21:13.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.062+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 5 ==== osd_op_reply(14 [pgnls-filter start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.062+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:15 17.c 17:30000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123000 con 0x55de3311c000 2024-04-24T08:21:13.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.063+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 4 ==== osd_op_reply(15 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.063+0000 7f95e1e00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55de33cd7c00 0x55de3312eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.063+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:16 17.2c 17:34000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33cd7400 con 0x55de33cd7c00 2024-04-24T08:21:13.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.063+0000 7f95e8200640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55de33cd7c00 0x55de3312eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.064+0000 7f95e8200640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55de33cd7c00 0x55de3312eb00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.065+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(16 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.065+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:17 17.1c 17:38000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122c00 con 0x55de3311d800 2024-04-24T08:21:13.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.065+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.1 v2:172.21.15.69:6815/153736297 2 ==== osd_op_reply(17 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de3311d800 2024-04-24T08:21:13.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.066+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:18 17.3c 17:3c000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123400 con 0x55de33cd7000 2024-04-24T08:21:13.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.066+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 6 ==== osd_op_reply(18 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.066+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:19 17.2 17:40000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122800 con 0x55de33cd7c00 2024-04-24T08:21:13.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.067+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== osd_op_reply(19 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.067+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:20 17.22 17:44000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33cd7400 con 0x55de33cd7800 2024-04-24T08:21:13.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.068+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.5 v2:172.21.15.121:6822/2187246263 2 ==== osd_op_reply(20 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33cd7800 2024-04-24T08:21:13.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.068+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:21 17.12 17:48000000::::head [pgnls-filter start_epoch 113 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123800 con 0x55de33122000 2024-04-24T08:21:13.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.069+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 5 ==== osd_op_reply(21 [pgnls-filter start_epoch 113 out=81b] v113'23 uv23 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.069+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:22 17.0 17:00000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311c800 con 0x55de3311c000 2024-04-24T08:21:13.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.070+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 5 ==== osd_op_reply(22 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.070+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:23 17.20 17:04000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123000 con 0x55de3311d800 2024-04-24T08:21:13.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.070+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.1 v2:172.21.15.69:6815/153736297 3 ==== osd_op_reply(23 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de3311d800 2024-04-24T08:21:13.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.070+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:24 17.10 17:08000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311c400 con 0x55de33122000 2024-04-24T08:21:13.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.071+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 6 ==== osd_op_reply(24 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.071+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:25 17.30 17:0c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123c00 con 0x55de33cd7000 2024-04-24T08:21:13.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.071+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 7 ==== osd_op_reply(25 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.071+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:26 17.8 17:10000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33b25400 con 0x55de3311c000 2024-04-24T08:21:13.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.072+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 6 ==== osd_op_reply(26 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.072+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:27 17.28 17:14000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142000 con 0x55de33122000 2024-04-24T08:21:13.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.073+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 7 ==== osd_op_reply(27 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.073+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:28 17.18 17:18000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33c64800 con 0x55de33cd7000 2024-04-24T08:21:13.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.073+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 8 ==== osd_op_reply(28 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.073+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:29 17.38 17:1c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142400 con 0x55de3311c000 2024-04-24T08:21:13.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.074+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 7 ==== osd_op_reply(29 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.074+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:30 17.4 17:20000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123400 con 0x55de33cd7000 2024-04-24T08:21:13.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.074+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 9 ==== osd_op_reply(30 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.075+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:31 17.24 17:24000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142800 con 0x55de33122000 2024-04-24T08:21:13.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.075+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 8 ==== osd_op_reply(31 [pgnls-filter start_epoch 113 out=49b] v109'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.075+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:32 17.14 17:28000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122800 con 0x55de33122000 2024-04-24T08:21:13.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.075+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 9 ==== osd_op_reply(32 [pgnls-filter start_epoch 113 out=49b] v109'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.075+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:33 17.34 17:2c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142c00 con 0x55de33cd7000 2024-04-24T08:21:13.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.076+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 10 ==== osd_op_reply(33 [pgnls-filter start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.076+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:34 17.c 17:30000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33cd7400 con 0x55de3311c000 2024-04-24T08:21:13.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.077+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 8 ==== osd_op_reply(34 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.077+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:35 17.2c 17:34000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33143000 con 0x55de33cd7c00 2024-04-24T08:21:13.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.078+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 3 ==== osd_op_reply(35 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.078+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:36 17.1c 17:38000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33b25400 con 0x55de3311d800 2024-04-24T08:21:13.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.078+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.1 v2:172.21.15.69:6815/153736297 4 ==== osd_op_reply(36 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de3311d800 2024-04-24T08:21:13.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.078+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:37 17.3c 17:3c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33143400 con 0x55de33cd7000 2024-04-24T08:21:13.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.079+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 11 ==== osd_op_reply(37 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.079+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:38 17.2 17:40000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33b25000 con 0x55de33cd7c00 2024-04-24T08:21:13.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.080+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 4 ==== osd_op_reply(38 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.080+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:39 17.22 17:44000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311dc00 con 0x55de33cd7800 2024-04-24T08:21:13.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.081+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.5 v2:172.21.15.121:6822/2187246263 3 ==== osd_op_reply(39 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33cd7800 2024-04-24T08:21:13.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.081+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:40 17.12 17:48000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33c64800 con 0x55de33122000 2024-04-24T08:21:13.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.081+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 10 ==== osd_op_reply(40 [pgnls-filter start_epoch 113 out=49b] v113'23 uv23 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.082+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:41 17.32 17:4c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122c00 con 0x55de3311c000 2024-04-24T08:21:13.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.083+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 9 ==== osd_op_reply(41 [pgnls-filter start_epoch 113 out=49b] v109'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.083+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:42 17.a 17:50000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142400 con 0x55de3311c000 2024-04-24T08:21:13.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.083+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 10 ==== osd_op_reply(42 [pgnls-filter start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.084+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:43 17.2a 17:54000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123000 con 0x55de33122000 2024-04-24T08:21:13.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.084+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 11 ==== osd_op_reply(43 [pgnls-filter start_epoch 113 out=49b] v109'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.084+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:44 17.1a 17:58000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142000 con 0x55de33cd7000 2024-04-24T08:21:13.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.085+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 12 ==== osd_op_reply(44 [pgnls-filter start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.085+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:45 17.3a 17:5c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123c00 con 0x55de3311d800 2024-04-24T08:21:13.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.085+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.1 v2:172.21.15.69:6815/153736297 5 ==== osd_op_reply(45 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de3311d800 2024-04-24T08:21:13.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.085+0000 7f95e1e00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55de33143800 0x55de3312f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.085+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:46 17.6 17:60000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33143c00 con 0x55de33143800 2024-04-24T08:21:13.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.086+0000 7f95e7800640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55de33143800 0x55de3312f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.087+0000 7f95e7800640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55de33143800 0x55de3312f080 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.088+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(46 [pgnls-filter start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de33143800 2024-04-24T08:21:13.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.088+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:47 17.26 17:64000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122400 con 0x55de33cd7000 2024-04-24T08:21:13.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.089+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 13 ==== osd_op_reply(47 [pgnls-filter start_epoch 113 out=49b] v109'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.089+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:48 17.16 17:68000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311c800 con 0x55de33cd7800 2024-04-24T08:21:13.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.090+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.5 v2:172.21.15.121:6822/2187246263 4 ==== osd_op_reply(48 [pgnls-filter start_epoch 113 out=49b] v109'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33cd7800 2024-04-24T08:21:13.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.090+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:49 17.36 17:6c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123800 con 0x55de33cd7000 2024-04-24T08:21:13.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.091+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 14 ==== osd_op_reply(49 [pgnls-filter start_epoch 113 out=81b] v113'34 uv34 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.091+0000 7f95eab1b740 20 datascan.forall_objects: OSD matched oid 100000001f7.00000000 2024-04-24T08:21:13.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.091+0000 7f95eab1b740 10 datascan.operator(): handling object 100000001f7.0 2024-04-24T08:21:13.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.091+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:50 17.36 17:6fe38dc2:::100000001f7.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55de3311d000 con 0x55de33cd7000 2024-04-24T08:21:13.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.092+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 15 ==== osd_op_reply(50 100000001f7.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (126) out=126b,getxattr (30) out=30b,getxattr] v0'0 uv34 ondisk = 0) ==== 416+0+194 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.092+0000 7f95eab1b740 10 datascan.inject_with_backtrace: inode: 0x0x100000001f7 2024-04-24T08:21:13.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.092+0000 7f95eab1b740 10 datascan.inject_with_backtrace: backptr: 0x0x10000000001/bravo 2024-04-24T08:21:13.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.092+0000 7f95eab1b740 20 datascan.get_frag_of: dirino=0x10000000001 target_dname=bravo 2024-04-24T08:21:13.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.092+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:51 16.7 16:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55de3311d400 con 0x55de33122000 2024-04-24T08:21:13.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.092+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 12 ==== osd_op_reply(51 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv8 ondisk = 0) ==== 164+0+95 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.092+0000 7f95eab1b740 20 datascan.get_frag_of: got backtrace parent 0x10000000000/flushed 2024-04-24T08:21:13.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.092+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55de33122800 0x55de33aff700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.092+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:52 16.10 16:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55de33b25000 con 0x55de33122800 2024-04-24T08:21:13.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.093+0000 7f95e8c00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55de33122800 0x55de33aff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.094+0000 7f95e8c00640 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55de33122800 0x55de33aff700 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.094+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== osd_op_reply(52 10000000000.00000000 [omap-get-vals-by-keys out=524b] v0'0 uv12 ondisk = 0) ==== 164+0+524 (crc 0 0 0) 0x55de33b14500 con 0x55de33122800 2024-04-24T08:21:13.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.094+0000 7f95eab1b740 20 datascan.read_dentry: oid=10000000000.00000000 dname=flushed frag=*, r=0 2024-04-24T08:21:13.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.095+0000 7f95eab1b740 20 datascan.get_frag_of: fast path, fragtree is fragtree_t(*) 2024-04-24T08:21:13.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.095+0000 7f95eab1b740 20 datascan.get_frag_of: frag is * 2024-04-24T08:21:13.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.095+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:53 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55de33cd7400 con 0x55de33122000 2024-04-24T08:21:13.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.095+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 13 ==== osd_op_reply(53 10000000001.00000000 [omap-get-header out=274b] v0'0 uv8 ondisk = 0) ==== 164+0+274 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.095+0000 7f95eab1b740 10 datascan.find_or_create_dirfrag: read_version = 8 2024-04-24T08:21:13.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.095+0000 7f95eab1b740 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000001 * 2024-04-24T08:21:13.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.095+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:54 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x55de3311dc00 con 0x55de33122000 2024-04-24T08:21:13.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.096+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 14 ==== osd_op_reply(54 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv8 ondisk = 0) ==== 164+0+4 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.096+0000 7f95eab1b740 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-04-24T08:21:13.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.096+0000 7f95eab1b740 20 datascan.read_dentry: bravo_head not found in result 2024-04-24T08:21:13.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.096+0000 7f95eab1b740 10 datascan.inject_with_backtrace: Linking inode 0x0x100000001f7 at 0x0x10000000001/bravo with size=5 bytes 2024-04-24T08:21:13.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.096+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:55 16.7 16:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x55de3311c800 con 0x55de33122000 2024-04-24T08:21:13.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.098+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 15 ==== osd_op_reply(55 10000000001.00000000 [omap-set-vals] v113'9 uv9 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.098+0000 7f95eab1b740 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-04-24T08:21:13.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.098+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:56 17.e 17:70000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311d400 con 0x55de33122800 2024-04-24T08:21:13.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.099+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.2 v2:172.21.15.69:6808/2630808717 2 ==== osd_op_reply(56 [pgnls-filter start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122800 2024-04-24T08:21:13.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.099+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:57 17.2e 17:74000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33b25400 con 0x55de33143800 2024-04-24T08:21:13.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.100+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.6 v2:172.21.15.121:6807/3068198325 2 ==== osd_op_reply(57 [pgnls-filter start_epoch 113 out=49b] v109'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de33143800 2024-04-24T08:21:13.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.100+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:58 17.1e 17:78000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311c400 con 0x55de3311d800 2024-04-24T08:21:13.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.101+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.1 v2:172.21.15.69:6815/153736297 6 ==== osd_op_reply(58 [pgnls-filter start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de3311d800 2024-04-24T08:21:13.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.101+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:59 17.3e 17:7c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142800 con 0x55de33cd7c00 2024-04-24T08:21:13.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.102+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 5 ==== osd_op_reply(59 [pgnls-filter start_epoch 113 out=49b] v109'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.102+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:60 17.1 17:80000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33154000 con 0x55de33cd7000 2024-04-24T08:21:13.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.103+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 16 ==== osd_op_reply(60 [pgnls-filter start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.103+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:61 17.21 17:84000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142c00 con 0x55de33cd7000 2024-04-24T08:21:13.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.104+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 17 ==== osd_op_reply(61 [pgnls-filter start_epoch 113 out=49b] v109'24 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.104+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:62 17.11 17:88000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33154400 con 0x55de33122000 2024-04-24T08:21:13.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.104+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 16 ==== osd_op_reply(62 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.104+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:63 17.31 17:8c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33143000 con 0x55de3311d800 2024-04-24T08:21:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.105+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.1 v2:172.21.15.69:6815/153736297 7 ==== osd_op_reply(63 [pgnls-filter start_epoch 113 out=49b] v109'15 uv8 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de3311d800 2024-04-24T08:21:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.105+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:64 17.9 17:90000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33154800 con 0x55de33122000 2024-04-24T08:21:13.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.106+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 17 ==== osd_op_reply(64 [pgnls-filter start_epoch 113 out=49b] v109'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.106+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:65 17.29 17:94000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33143400 con 0x55de33cd7c00 2024-04-24T08:21:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.107+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 6 ==== osd_op_reply(65 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.107+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:66 17.19 17:98000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33154c00 con 0x55de33cd7800 2024-04-24T08:21:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.108+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.5 v2:172.21.15.121:6822/2187246263 5 ==== osd_op_reply(66 [pgnls-filter start_epoch 113 out=49b] v113'24 uv24 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33cd7800 2024-04-24T08:21:13.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.108+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:67 17.39 17:9c000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142400 con 0x55de33cd7800 2024-04-24T08:21:13.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.110+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.5 v2:172.21.15.121:6822/2187246263 6 ==== osd_op_reply(67 [pgnls-filter start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33cd7800 2024-04-24T08:21:13.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.110+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:68 17.5 17:a0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33155000 con 0x55de33143800 2024-04-24T08:21:13.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.111+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.6 v2:172.21.15.121:6807/3068198325 3 ==== osd_op_reply(68 [pgnls-filter start_epoch 113 out=49b] v113'15 uv15 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de33143800 2024-04-24T08:21:13.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.111+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:69 17.25 17:a4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123000 con 0x55de3311d800 2024-04-24T08:21:13.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.112+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.1 v2:172.21.15.69:6815/153736297 8 ==== osd_op_reply(69 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de3311d800 2024-04-24T08:21:13.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.112+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:70 17.15 17:a8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33155400 con 0x55de33cd7800 2024-04-24T08:21:13.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.113+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.5 v2:172.21.15.121:6822/2187246263 7 ==== osd_op_reply(70 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33cd7800 2024-04-24T08:21:13.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.113+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:71 17.35 17:ac000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123400 con 0x55de33cd7c00 2024-04-24T08:21:13.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.114+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 7 ==== osd_op_reply(71 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.114+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:72 17.d 17:b0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33155800 con 0x55de3311c000 2024-04-24T08:21:13.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.115+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 11 ==== osd_op_reply(72 [pgnls-filter start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.115+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:73 17.2d 17:b4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122c00 con 0x55de3311c000 2024-04-24T08:21:13.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.117+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 12 ==== osd_op_reply(73 [pgnls-filter start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.117+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:74 17.1d 17:b8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33155c00 con 0x55de3311c000 2024-04-24T08:21:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.118+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 13 ==== osd_op_reply(74 [pgnls-filter start_epoch 113 out=49b] v109'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.118+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:75 17.3d 17:bc000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33b25000 con 0x55de3311c000 2024-04-24T08:21:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.119+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 14 ==== osd_op_reply(75 [pgnls-filter start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.119+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:76 17.3 17:c0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142000 con 0x55de33cd7c00 2024-04-24T08:21:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.120+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 8 ==== osd_op_reply(76 [pgnls-filter start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.121+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:77 17.23 17:c4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33122400 con 0x55de33cd7c00 2024-04-24T08:21:13.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.122+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 9 ==== osd_op_reply(77 [pgnls-filter start_epoch 113 out=49b] v109'14 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.122+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:78 17.13 17:c8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33142800 con 0x55de33122000 2024-04-24T08:21:13.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.123+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.3 v2:172.21.15.69:6811/1875112303 18 ==== osd_op_reply(78 [pgnls-filter start_epoch 113 out=49b] v109'30 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122000 2024-04-24T08:21:13.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.123+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:79 17.33 17:cc000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123800 con 0x55de3311c000 2024-04-24T08:21:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.124+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.7 v2:172.21.15.121:6811/176144805 15 ==== osd_op_reply(79 [pgnls-filter start_epoch 113 out=49b] v109'20 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de3311c000 2024-04-24T08:21:13.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.124+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:80 17.b 17:d0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311dc00 con 0x55de33cd7000 2024-04-24T08:21:13.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.125+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 18 ==== osd_op_reply(80 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.125+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:81 17.2b 17:d4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33123c00 con 0x55de33cd7c00 2024-04-24T08:21:13.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.127+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 10 ==== osd_op_reply(81 [pgnls-filter start_epoch 113 out=49b] v109'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.127+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:82 17.1b 17:d8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311c800 con 0x55de33cd7c00 2024-04-24T08:21:13.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.128+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 11 ==== osd_op_reply(82 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.128+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:83 17.3b 17:dc000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33154400 con 0x55de33122800 2024-04-24T08:21:13.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.129+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.2 v2:172.21.15.69:6808/2630808717 3 ==== osd_op_reply(83 [pgnls-filter start_epoch 113 out=49b] v109'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122800 2024-04-24T08:21:13.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.129+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:84 17.7 17:e0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311d400 con 0x55de33cd7000 2024-04-24T08:21:13.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.130+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 19 ==== osd_op_reply(84 [pgnls-filter start_epoch 113 out=49b] v109'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.130+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:85 17.27 17:e4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33154800 con 0x55de33cd7c00 2024-04-24T08:21:13.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.131+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 12 ==== osd_op_reply(85 [pgnls-filter start_epoch 113 out=49b] v109'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.131+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:86 17.17 17:e8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311d000 con 0x55de33143800 2024-04-24T08:21:13.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.133+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.6 v2:172.21.15.121:6807/3068198325 4 ==== osd_op_reply(86 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de33143800 2024-04-24T08:21:13.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.133+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:87 17.37 17:ec000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33154c00 con 0x55de33122800 2024-04-24T08:21:13.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.134+0000 7f95e8c00640 1 -- 172.21.15.69:0/1449101575 <== osd.2 v2:172.21.15.69:6808/2630808717 4 ==== osd_op_reply(87 [pgnls-filter start_epoch 113 out=49b] v109'22 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14500 con 0x55de33122800 2024-04-24T08:21:13.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.134+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:88 17.f 17:f0000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311cc00 con 0x55de3311d800 2024-04-24T08:21:13.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.135+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.1 v2:172.21.15.69:6815/153736297 9 ==== osd_op_reply(88 [pgnls-filter start_epoch 113 out=49b] v109'18 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de3311d800 2024-04-24T08:21:13.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.135+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:89 17.2f 17:f4000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33154000 con 0x55de33cd7c00 2024-04-24T08:21:13.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.136+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.4 v2:172.21.15.121:6800/2274643300 13 ==== osd_op_reply(89 [pgnls-filter start_epoch 113 out=49b] v109'16 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7c00 2024-04-24T08:21:13.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.136+0000 7f95e1e00640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:90 17.1f 17:f8000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de3311c400 con 0x55de33143800 2024-04-24T08:21:13.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.138+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 <== osd.6 v2:172.21.15.121:6807/3068198325 5 ==== osd_op_reply(90 [pgnls-filter start_epoch 113 out=49b] v109'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14780 con 0x55de33143800 2024-04-24T08:21:13.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.138+0000 7f95e2800640 1 -- 172.21.15.69:0/1449101575 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:91 17.3f 17:fc000000::::head [pgnls-filter start_epoch 113 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x55de33cd7400 con 0x55de33cd7000 2024-04-24T08:21:13.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95e8200640 1 -- 172.21.15.69:0/1449101575 <== osd.0 v2:172.21.15.69:6800/146737935 20 ==== osd_op_reply(91 [pgnls-filter start_epoch 113 out=49b] v109'17 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55de33b14000 con 0x55de33cd7000 2024-04-24T08:21:13.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55de33cd7000 msgr2=0x55de33afe680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x55de33cd7000 0x55de33afe680 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55de3311d800 msgr2=0x55de3312e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x55de3311d800 0x55de3312e000 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55de33122800 msgr2=0x55de33aff700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x55de33122800 0x55de33aff700 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.140 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55de33122000 msgr2=0x55de3312e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x55de33122000 0x55de3312e580 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55de33cd7c00 msgr2=0x55de3312eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x55de33cd7c00 0x55de3312eb00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55de33cd7800 msgr2=0x55de33afec00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x55de33cd7800 0x55de33afec00 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55de33143800 msgr2=0x55de3312f080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x55de33143800 0x55de3312f080 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55de3311c000 msgr2=0x55de33aff180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.141 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.139+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x55de3311c000 0x55de33aff180 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 reap_dead start 2024-04-24T08:21:13.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 reap_dead start 2024-04-24T08:21:13.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95e7800640 1 -- 172.21.15.69:0/1449101575 reap_dead start 2024-04-24T08:21:13.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55de33cd6c00 msgr2=0x55de33afe100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55de33cd6c00 0x55de33afe100 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x55de33ba71d0 tx=0x55de33b26a20 comp rx=0 tx=0).stop 2024-04-24T08:21:13.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65c00 msgr2=0x55de33afdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65c00 0x55de33afdb80 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x55de33c73a10 tx=0x55de33c82c00 comp rx=0 tx=0).stop 2024-04-24T08:21:13.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 shutdown_connections 2024-04-24T08:21:13.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x55de33cd6c00 0x55de33afe100 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95eab1b740 1 --2- 172.21.15.69:0/1449101575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de33c65c00 0x55de33afdb80 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.140+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 >> 172.21.15.69:0/1449101575 conn(0x55de33c65000 msgr2=0x55de32e73000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.141+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 shutdown_connections 2024-04-24T08:21:13.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.141+0000 7f95eab1b740 1 -- 172.21.15.69:0/1449101575 wait complete. 2024-04-24T08:21:13.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.142+0000 7f95eab1b740 1 -- 172.21.15.69:0/945994833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de32d86c00 msgr2=0x55de33afc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.142+0000 7f95eab1b740 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de32d86c00 0x55de33afc000 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x55de33ba7c50 tx=0x55de33b2a120 comp rx=0 tx=0).stop 2024-04-24T08:21:13.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.142+0000 7f95eab1b740 1 -- 172.21.15.69:0/945994833 shutdown_connections 2024-04-24T08:21:13.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.142+0000 7f95eab1b740 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x55de32d87c00 0x55de33afcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.142+0000 7f95eab1b740 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55de32d87800 0x55de33afc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.142+0000 7f95eab1b740 1 --2- 172.21.15.69:0/945994833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55de32d86c00 0x55de33afc000 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.142+0000 7f95eab1b740 1 -- 172.21.15.69:0/945994833 >> 172.21.15.69:0/945994833 conn(0x55de32d87000 msgr2=0x55de32e72800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.142+0000 7f95eab1b740 1 -- 172.21.15.69:0/945994833 shutdown_connections 2024-04-24T08:21:13.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.143+0000 7f95eab1b740 1 -- 172.21.15.69:0/945994833 wait complete. 2024-04-24T08:21:13.157 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', '--filter-tag', 'mytag123', 'cephfs_data'] in time 0:00:00.226136, result: 2024-04-24T08:21:13.158 DEBUG:teuthology.orchestra.run.smithi069:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-04-24T08:21:13.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.262+0000 7f733021a740 1 -- start start 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.262+0000 7f733021a740 1 Processor -- start 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.263+0000 7f733021a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0f68400 0x5621a0f40b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.263+0000 7f733021a740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0f68800 0x5621a0f41080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.263+0000 7f733021a740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0f69c00 0x5621a0f40580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.263+0000 7f733021a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5621a0fc0000 con 0x5621a0f68400 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.263+0000 7f733021a740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5621a0fc0180 con 0x5621a0f69c00 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.263+0000 7f733021a740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5621a0fc0300 con 0x5621a0f68800 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.263+0000 7f732d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0f68400 0x5621a0f40b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.264+0000 7f732d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0f68400 0x5621a0f40b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53216/0 (socket says 172.21.15.69:53216) 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.264+0000 7f732d800640 1 -- 172.21.15.69:0/2459086427 learned_addr learned my addr 172.21.15.69:0/2459086427 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.264+0000 7f732d800640 1 -- 172.21.15.69:0/2459086427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0f68800 msgr2=0x5621a0f41080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.264+0000 7f732d800640 1 --2- 172.21.15.69:0/2459086427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0f68800 0x5621a0f41080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.264+0000 7f732d800640 1 -- 172.21.15.69:0/2459086427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0f69c00 msgr2=0x5621a0f40580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:13.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.264+0000 7f732e200640 1 --2- 172.21.15.69:0/2459086427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0f69c00 0x5621a0f40580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.264+0000 7f732d800640 1 --2- 172.21.15.69:0/2459086427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0f69c00 0x5621a0f40580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.264+0000 7f732d800640 1 -- 172.21.15.69:0/2459086427 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a01f7680 con 0x5621a0f68400 2024-04-24T08:21:13.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.264+0000 7f732d800640 1 --2- 172.21.15.69:0/2459086427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0f68400 0x5621a0f40b00 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x5621a1001a10 tx=0x5621a0f6e120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a35fc78b3ce491c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f732a600640 1 -- 172.21.15.69:0/2459086427 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5621a01f0d00 con 0x5621a0f68400 2024-04-24T08:21:13.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f732a600640 1 -- 172.21.15.69:0/2459086427 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5621a0214540 con 0x5621a0f68400 2024-04-24T08:21:13.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f732a600640 1 -- 172.21.15.69:0/2459086427 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5621a01f0680 con 0x5621a0f68400 2024-04-24T08:21:13.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f733021a740 1 -- 172.21.15.69:0/2459086427 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621a01f72c0 con 0x5621a0f68400 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f733021a740 1 -- 172.21.15.69:0/2459086427 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=0}) -- 0x5621a01f6b40 con 0x5621a0f68400 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f733021a740 4 waiting for MDS map... 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f732a600640 1 -- 172.21.15.69:0/2459086427 <== mon.0 v2:172.21.15.69:3300/0 4 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x5621a0f3a600 con 0x5621a0f68400 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f732a600640 1 -- 172.21.15.69:0/2459086427 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 106) ==== 2885+0+0 (secure 0 0 0) 0x5621a0223180 con 0x5621a0f68400 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f733021a740 4 Got MDS map 106 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f733021a740 4 datascan.main: Using metadata pool output 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.265+0000 7f733021a740 4 datascan.main: connecting to RADOS... 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.266+0000 7f733021a740 1 Processor -- start 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.266+0000 7f733021a740 1 -- start start 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.267+0000 7f733021a740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0f69800 0x5621a0f40000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.267+0000 7f733021a740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0fdd000 0x5621a0f41600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.267+0000 7f733021a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd400 0x5621a0f41b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.267+0000 7f733021a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5621a0fc0480 con 0x5621a0fdd400 2024-04-24T08:21:13.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.267+0000 7f733021a740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5621a0fc0600 con 0x5621a0fdd000 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.267+0000 7f733021a740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5621a0fc0780 con 0x5621a0f69800 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.267+0000 7f732ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd400 0x5621a0f41b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.267+0000 7f732ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd400 0x5621a0f41b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53224/0 (socket says 172.21.15.69:53224) 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.267+0000 7f732ce00640 1 -- 172.21.15.69:0/1528905476 learned_addr learned my addr 172.21.15.69:0/1528905476 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f732ce00640 1 -- 172.21.15.69:0/1528905476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0f69800 msgr2=0x5621a0f40000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f732ce00640 1 --2- 172.21.15.69:0/1528905476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0f69800 0x5621a0f40000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f732ce00640 1 -- 172.21.15.69:0/1528905476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0fdd000 msgr2=0x5621a0f41600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f732ce00640 1 --2- 172.21.15.69:0/1528905476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0fdd000 0x5621a0f41600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f732ce00640 1 -- 172.21.15.69:0/1528905476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a01f63c0 con 0x5621a0fdd400 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f732ce00640 1 --2- 172.21.15.69:0/1528905476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd400 0x5621a0f41b80 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x5621a0f38720 tx=0x5621a10aad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2026ff9d4014e55 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f7327e00640 1 -- 172.21.15.69:0/1528905476 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5621a01f0d00 con 0x5621a0fdd400 2024-04-24T08:21:13.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f7327e00640 1 -- 172.21.15.69:0/1528905476 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5621a0214c40 con 0x5621a0fdd400 2024-04-24T08:21:13.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f7327e00640 1 -- 172.21.15.69:0/1528905476 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5621a01f0680 con 0x5621a0fdd400 2024-04-24T08:21:13.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f733021a740 1 -- 172.21.15.69:0/1528905476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd400 msgr2=0x5621a0f41b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.273 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.268+0000 7f733021a740 1 --2- 172.21.15.69:0/1528905476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd400 0x5621a0f41b80 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x5621a0f38720 tx=0x5621a10aad20 comp rx=0 tx=0).stop 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.269+0000 7f733021a740 1 -- 172.21.15.69:0/1528905476 shutdown_connections 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.269+0000 7f733021a740 1 --2- 172.21.15.69:0/1528905476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd400 0x5621a0f41b80 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.269+0000 7f733021a740 1 --2- 172.21.15.69:0/1528905476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0fdd000 0x5621a0f41600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.269+0000 7f733021a740 1 --2- 172.21.15.69:0/1528905476 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0f69800 0x5621a0f40000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.269+0000 7f733021a740 1 -- 172.21.15.69:0/1528905476 >> 172.21.15.69:0/1528905476 conn(0x5621a0f69400 msgr2=0x5621a02b7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.269+0000 7f733021a740 1 -- 172.21.15.69:0/1528905476 shutdown_connections 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.269+0000 7f733021a740 1 -- 172.21.15.69:0/1528905476 wait complete. 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.269+0000 7f733021a740 1 Processor -- start 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.270+0000 7f733021a740 1 -- start start 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.270+0000 7f733021a740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0f69800 0x5621a0f40000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.270+0000 7f733021a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd000 0x5621a0f41600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.270+0000 7f733021a740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0fdd400 0x5621a0f41b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.270+0000 7f733021a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5621a0fc1e00 con 0x5621a0fdd000 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.270+0000 7f733021a740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x5621a0fc1c80 con 0x5621a0f69800 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.270+0000 7f733021a740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x5621a0fc1b00 con 0x5621a0fdd400 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.270+0000 7f732d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd000 0x5621a0f41600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f732d800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd000 0x5621a0f41600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53240/0 (socket says 172.21.15.69:53240) 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 learned_addr learned my addr 172.21.15.69:0/4259505759 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0f69800 0x5621a0f40000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0fdd400 msgr2=0x5621a0f41b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0fdd400 0x5621a0f41b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f732d800640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0fdd400 0x5621a0f41b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0f69800 msgr2=0x5621a0f40000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f732d800640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0f69800 0x5621a0f40000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621a01f7680 con 0x5621a0fdd000 2024-04-24T08:21:13.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f732d800640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd000 0x5621a0f41600 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x5621a0f39a40 tx=0x5621a0f6aa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f7326000640 1 -- 172.21.15.69:0/4259505759 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5621a01f0820 con 0x5621a0fdd000 2024-04-24T08:21:13.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f7326000640 1 -- 172.21.15.69:0/4259505759 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5621a0215340 con 0x5621a0fdd000 2024-04-24T08:21:13.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.271+0000 7f7326000640 1 -- 172.21.15.69:0/4259505759 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5621a01f0d00 con 0x5621a0fdd000 2024-04-24T08:21:13.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.272+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5621a01f72c0 con 0x5621a0fdd000 2024-04-24T08:21:13.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.272+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621a01f7a40 con 0x5621a0fdd000 2024-04-24T08:21:13.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.272+0000 7f733021a740 4 datascan.init: resolving metadata pool 16 2024-04-24T08:21:13.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.273+0000 7f7326000640 1 -- 172.21.15.69:0/4259505759 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x5621a110cc00 con 0x5621a0fdd000 2024-04-24T08:21:13.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.274+0000 7f7326000640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x5621a1105c00 0x5621a0f42100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.274+0000 7f7326000640 1 -- 172.21.15.69:0/4259505759 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x5621a110aa00 con 0x5621a0fdd000 2024-04-24T08:21:13.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.274+0000 7f733021a740 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-04-24T08:21:13.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.274+0000 7f733021a740 4 datascan.main: resolving metadata pool 16 2024-04-24T08:21:13.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.274+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x5621a0fdd800 0x5621a0f42680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.274+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:1 16.0 16:00000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0fddc00 con 0x5621a0fdd800 2024-04-24T08:21:13.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.274+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x5621a0fdd800 0x5621a0f42680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.275+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x5621a1105c00 0x5621a0f42100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.275+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x5621a0fdd800 0x5621a0f42680 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.275+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== osd_op_reply(1 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.275+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x5621a0fddc00 0x5621a0554000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.275+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:2 16.10 16:08000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a1104800 con 0x5621a0fddc00 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.276+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x5621a0fddc00 0x5621a0554000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.276+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x5621a1105c00 0x5621a0f42100 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x5621a0f6cf30 tx=0x5621a0f6a720 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.276+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x5621a0fddc00 0x5621a0554000 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.277+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== osd_op_reply(2 [pgnls start_epoch 113 out=177b] v112'16 uv16 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.277+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:3 16.8 16:10000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a1104800 con 0x5621a0fdd800 2024-04-24T08:21:13.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.277+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 2 ==== osd_op_reply(3 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.277+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:4 16.18 16:18000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a1104800 con 0x5621a0fdd800 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.277+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 3 ==== osd_op_reply(4 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.277+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x5621a1104800 0x5621a0554580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.277+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:5 16.4 16:20000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a1104000 con 0x5621a1104800 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.278+0000 7f732d800640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x5621a1104800 0x5621a0554580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.278+0000 7f732d800640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x5621a1104800 0x5621a0554580 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.278+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== osd_op_reply(5 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.279+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:6 16.14 16:28000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a1104000 con 0x5621a0fdd800 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.279+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 4 ==== osd_op_reply(6 [pgnls start_epoch 113 out=149b] v113'44 uv44 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.279+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x5621a1104000 0x5621a0554b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.279+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:7 16.c 16:30000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a1104400 con 0x5621a1104000 2024-04-24T08:21:13.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.280+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x5621a1104000 0x5621a0554b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.281+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x5621a1104000 0x5621a0554b00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.282+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== osd_op_reply(7 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.282+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:8 16.1c 16:38000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a1104400 con 0x5621a0fddc00 2024-04-24T08:21:13.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.282+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 2 ==== osd_op_reply(8 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.282+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x5621a1104400 0x5621a0555080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.282+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:9 16.2 16:40000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2000 con 0x5621a1104400 2024-04-24T08:21:13.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.283+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x5621a1104400 0x5621a0555080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.284+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x5621a1104400 0x5621a0555080 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.284+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== osd_op_reply(9 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a1104400 2024-04-24T08:21:13.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.284+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x5621a04d2000 0x5621a0555600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.285+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:10 16.12 16:48000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2400 con 0x5621a04d2000 2024-04-24T08:21:13.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.285+0000 7f732d800640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x5621a04d2000 0x5621a0555600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.286+0000 7f732d800640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x5621a04d2000 0x5621a0555600 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.287+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== osd_op_reply(10 [pgnls start_epoch 113 out=81b] v112'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x5621a0f58780 con 0x5621a04d2000 2024-04-24T08:21:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.287+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:11 16.a 16:50000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2400 con 0x5621a0fddc00 2024-04-24T08:21:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.288+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 3 ==== osd_op_reply(11 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.288+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:12 16.1a 16:58000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2400 con 0x5621a1104800 2024-04-24T08:21:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.288+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 2 ==== osd_op_reply(12 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.288+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:13 16.6 16:60000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2400 con 0x5621a04d2000 2024-04-24T08:21:13.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.289+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.6 v2:172.21.15.121:6807/3068198325 2 ==== osd_op_reply(13 [pgnls start_epoch 113 out=77b] v112'11 uv11 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x5621a0f58780 con 0x5621a04d2000 2024-04-24T08:21:13.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.289+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:14 16.16 16:68000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2400 con 0x5621a1104800 2024-04-24T08:21:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.290+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 3 ==== osd_op_reply(14 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.290+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:15 16.e 16:70000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2400 con 0x5621a0fddc00 2024-04-24T08:21:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.290+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 4 ==== osd_op_reply(15 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.290+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x5621a04d2400 0x5621a0555b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.290+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:16 16.1e 16:78000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2800 con 0x5621a04d2400 2024-04-24T08:21:13.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.291+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x5621a04d2400 0x5621a0555b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.292+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x5621a04d2400 0x5621a0555b80 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.293+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== osd_op_reply(16 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a04d2400 2024-04-24T08:21:13.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.293+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:17 16.1 16:80000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2800 con 0x5621a0fddc00 2024-04-24T08:21:13.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.293+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 5 ==== osd_op_reply(17 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.293+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:18 16.11 16:88000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2800 con 0x5621a1104000 2024-04-24T08:21:13.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.294+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 2 ==== osd_op_reply(18 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.294+0000 7f732ce00640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5621a04d2800 0x5621a0556c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.294+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:19 16.9 16:90000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a04d2800 2024-04-24T08:21:13.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.294+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5621a04d2800 0x5621a0556c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.295+0000 7f732e200640 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5621a04d2800 0x5621a0556c00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.296+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== osd_op_reply(19 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a04d2800 2024-04-24T08:21:13.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.296+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:20 16.19 16:98000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a1104400 2024-04-24T08:21:13.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.297+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== osd_op_reply(20 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a1104400 2024-04-24T08:21:13.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.297+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:21 16.5 16:a0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a0fdd800 2024-04-24T08:21:13.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.297+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 5 ==== osd_op_reply(21 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.297+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:22 16.15 16:a8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a0fdd800 2024-04-24T08:21:13.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.298+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 6 ==== osd_op_reply(22 [pgnls start_epoch 113 out=97b] v112'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.298+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:23 16.d 16:b0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a04d2000 2024-04-24T08:21:13.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.299+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.6 v2:172.21.15.121:6807/3068198325 3 ==== osd_op_reply(23 [pgnls start_epoch 113 out=74b] v108'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x5621a0f58780 con 0x5621a04d2000 2024-04-24T08:21:13.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.299+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:24 16.1d 16:b8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a0fddc00 2024-04-24T08:21:13.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.299+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 6 ==== osd_op_reply(24 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.299+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:25 16.3 16:c0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a1104000 2024-04-24T08:21:13.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.300+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 3 ==== osd_op_reply(25 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.300+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:26 16.13 16:c8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a04d2800 2024-04-24T08:21:13.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.300+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.1 v2:172.21.15.69:6815/153736297 2 ==== osd_op_reply(26 [pgnls start_epoch 113 out=98b] v112'5 uv5 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x5621a0f58500 con 0x5621a04d2800 2024-04-24T08:21:13.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.301+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:27 16.b 16:d0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a04d2800 2024-04-24T08:21:13.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.301+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.1 v2:172.21.15.69:6815/153736297 3 ==== osd_op_reply(27 [pgnls start_epoch 113 out=76b] v113'6 uv5 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x5621a0f58500 con 0x5621a04d2800 2024-04-24T08:21:13.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.301+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:28 16.1b 16:d8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a1104000 2024-04-24T08:21:13.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.302+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 4 ==== osd_op_reply(28 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.302+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:29 16.7 16:e0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a1104800 2024-04-24T08:21:13.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.303+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 4 ==== osd_op_reply(29 [pgnls start_epoch 113 out=105b] v113'9 uv9 ondisk = 1) ==== 144+0+105 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.303+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:30 16.17 16:e8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a0fddc00 2024-04-24T08:21:13.303 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.303+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 7 ==== osd_op_reply(30 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.303+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:31 16.f 16:f0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a0fdd800 2024-04-24T08:21:13.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.304+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 7 ==== osd_op_reply(31 [pgnls start_epoch 113 out=73b] v113'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.304+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:32 16.1f 16:f8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3400 con 0x5621a1104000 2024-04-24T08:21:13.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.305+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 5 ==== osd_op_reply(32 [pgnls start_epoch 113 out=101b] v113'9 uv8 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.305+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 601.00000000 2024-04-24T08:21:13.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.305+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:33 16.10 16:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578000 con 0x5621a0fddc00 2024-04-24T08:21:13.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.305+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 8 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.305+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 602.00000000 2024-04-24T08:21:13.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.305+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:34 16.10 16:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578400 con 0x5621a0fddc00 2024-04-24T08:21:13.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.306+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 9 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.306+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-04-24T08:21:13.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.306+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:35 16.10 16:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578800 con 0x5621a0fddc00 2024-04-24T08:21:13.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.306+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 10 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=1047b] v0'0 uv12 ondisk = 0) ==== 164+0+1047 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.306+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 600.00000000 2024-04-24T08:21:13.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.306+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:36 16.10 16:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578c00 con 0x5621a0fddc00 2024-04-24T08:21:13.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 11 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 603.00000000 2024-04-24T08:21:13.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:37 16.10 16:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0579000 con 0x5621a0fddc00 2024-04-24T08:21:13.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 12 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-04-24T08:21:13.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-24T08:21:13.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 200.00000000 2024-04-24T08:21:13.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-04-24T08:21:13.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 200.00000001 2024-04-24T08:21:13.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-04-24T08:21:13.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 606.00000000 2024-04-24T08:21:13.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:38 16.14 16:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0579400 con 0x5621a0fdd800 2024-04-24T08:21:13.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.307+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 8 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv39 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.308+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 607.00000000 2024-04-24T08:21:13.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.308+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:39 16.1c 16:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578000 con 0x5621a0fddc00 2024-04-24T08:21:13.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.308+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 13 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.308+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 10000000003.00000000 2024-04-24T08:21:13.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.308+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:40 16.12 16:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578400 con 0x5621a04d2000 2024-04-24T08:21:13.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.309+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.6 v2:172.21.15.121:6807/3068198325 4 ==== osd_op_reply(40 10000000003.00000000 [omap-get-vals out=523b] v0'0 uv3 ondisk = 0) ==== 164+0+523 (crc 0 0 0) 0x5621a0f58780 con 0x5621a04d2000 2024-04-24T08:21:13.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.309+0000 7f733021a740 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-04-24T08:21:13.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.309+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-04-24T08:21:13.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.309+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 608.00000000 2024-04-24T08:21:13.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.309+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:41 16.e 16:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578800 con 0x5621a0fddc00 2024-04-24T08:21:13.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.310+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 14 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.310+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 604.00000000 2024-04-24T08:21:13.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.310+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:42 16.15 16:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578c00 con 0x5621a0fdd800 2024-04-24T08:21:13.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.310+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 9 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.310+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 500.00000000 2024-04-24T08:21:13.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.310+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-04-24T08:21:13.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.310+0000 7f733021a740 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-04-24T08:21:13.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.310+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-24T08:21:13.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.310+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 605.00000000 2024-04-24T08:21:13.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.310+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:43 16.1d 16:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0579000 con 0x5621a0fddc00 2024-04-24T08:21:13.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.311+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 15 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.311+0000 7f733021a740 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-04-24T08:21:13.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.311+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-24T08:21:13.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.311+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 100.00000000 2024-04-24T08:21:13.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.311+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:44 16.13 16:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3000 con 0x5621a04d2800 2024-04-24T08:21:13.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.311+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.1 v2:172.21.15.69:6815/153736297 4 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x5621a0f58500 con 0x5621a04d2800 2024-04-24T08:21:13.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.311+0000 7f733021a740 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-04-24T08:21:13.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.311+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-24T08:21:13.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.311+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 609.00000000 2024-04-24T08:21:13.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.311+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:45 16.7 16:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d2c00 con 0x5621a1104800 2024-04-24T08:21:13.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 5 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv7 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 10000000001.00000000 2024-04-24T08:21:13.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:46 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0579800 con 0x5621a1104800 2024-04-24T08:21:13.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 6 ==== osd_op_reply(46 10000000001.00000000 [omap-get-vals out=1545b] v0'0 uv9 ondisk = 0) ==== 164+0+1545 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f733021a740 20 datascan.scan_links: injected ino detected 2024-04-24T08:21:13.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 400.00000000 2024-04-24T08:21:13.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-04-24T08:21:13.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-04-24T08:21:13.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-24T08:21:13.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f733021a740 10 datascan.scan_links: step 1: handling object 1.00000000 2024-04-24T08:21:13.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.312+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:47 16.1f 16:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0579c00 con 0x5621a1104000 2024-04-24T08:21:13.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.313+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 6 ==== osd_op_reply(47 1.00000000 [omap-get-vals out=524b] v0'0 uv8 ondisk = 0) ==== 154+0+524 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.314+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:48 16.0 16:00000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fdd800 2024-04-24T08:21:13.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.314+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 10 ==== osd_op_reply(48 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.314+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:49 16.10 16:08000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fddc00 2024-04-24T08:21:13.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.314+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 16 ==== osd_op_reply(49 [pgnls start_epoch 113 out=177b] v112'16 uv16 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.314+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:50 16.8 16:10000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fdd800 2024-04-24T08:21:13.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.315+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 11 ==== osd_op_reply(50 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.315+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:51 16.18 16:18000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fdd800 2024-04-24T08:21:13.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.315+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 12 ==== osd_op_reply(51 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.315+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:52 16.4 16:20000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104800 2024-04-24T08:21:13.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.316+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 7 ==== osd_op_reply(52 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.316+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:53 16.14 16:28000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fdd800 2024-04-24T08:21:13.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.316+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 13 ==== osd_op_reply(53 [pgnls start_epoch 113 out=149b] v113'44 uv44 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.316+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:54 16.c 16:30000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104000 2024-04-24T08:21:13.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.317+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 7 ==== osd_op_reply(54 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.317+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:55 16.1c 16:38000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fddc00 2024-04-24T08:21:13.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.317+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 17 ==== osd_op_reply(55 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.318+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:56 16.2 16:40000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104400 2024-04-24T08:21:13.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.319+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.4 v2:172.21.15.121:6800/2274643300 3 ==== osd_op_reply(56 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a1104400 2024-04-24T08:21:13.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.319+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:57 16.12 16:48000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a04d2000 2024-04-24T08:21:13.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.320+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.6 v2:172.21.15.121:6807/3068198325 5 ==== osd_op_reply(57 [pgnls start_epoch 113 out=81b] v112'3 uv3 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x5621a0f58780 con 0x5621a04d2000 2024-04-24T08:21:13.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.320+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:58 16.a 16:50000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fddc00 2024-04-24T08:21:13.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.320+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 18 ==== osd_op_reply(58 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.320+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:59 16.1a 16:58000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104800 2024-04-24T08:21:13.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.321+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 8 ==== osd_op_reply(59 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.321+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:60 16.6 16:60000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a04d2000 2024-04-24T08:21:13.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.322+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.6 v2:172.21.15.121:6807/3068198325 6 ==== osd_op_reply(60 [pgnls start_epoch 113 out=77b] v112'11 uv11 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x5621a0f58780 con 0x5621a04d2000 2024-04-24T08:21:13.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.322+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:61 16.16 16:68000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104800 2024-04-24T08:21:13.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.323+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 9 ==== osd_op_reply(61 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.323+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:62 16.e 16:70000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fddc00 2024-04-24T08:21:13.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.323+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 19 ==== osd_op_reply(62 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.323+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- osd_op(unknown.0.0:63 16.1e 16:78000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a04d2400 2024-04-24T08:21:13.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.324+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.7 v2:172.21.15.121:6811/176144805 2 ==== osd_op_reply(63 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a04d2400 2024-04-24T08:21:13.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.324+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:64 16.1 16:80000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fddc00 2024-04-24T08:21:13.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.325+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 20 ==== osd_op_reply(64 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.325+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:65 16.11 16:88000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104000 2024-04-24T08:21:13.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.326+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 8 ==== osd_op_reply(65 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.326+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:66 16.9 16:90000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a04d2800 2024-04-24T08:21:13.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.327+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.1 v2:172.21.15.69:6815/153736297 5 ==== osd_op_reply(66 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a04d2800 2024-04-24T08:21:13.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.327+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- osd_op(unknown.0.0:67 16.19 16:98000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104400 2024-04-24T08:21:13.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.328+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.4 v2:172.21.15.121:6800/2274643300 4 ==== osd_op_reply(67 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a1104400 2024-04-24T08:21:13.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.328+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:68 16.5 16:a0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fdd800 2024-04-24T08:21:13.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.328+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 14 ==== osd_op_reply(68 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.328+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:69 16.15 16:a8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fdd800 2024-04-24T08:21:13.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.329+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 15 ==== osd_op_reply(69 [pgnls start_epoch 113 out=97b] v112'4 uv4 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.329+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:70 16.d 16:b0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a04d2000 2024-04-24T08:21:13.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.330+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.6 v2:172.21.15.121:6807/3068198325 7 ==== osd_op_reply(70 [pgnls start_epoch 113 out=74b] v108'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x5621a0f58780 con 0x5621a04d2000 2024-04-24T08:21:13.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.330+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:71 16.1d 16:b8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fddc00 2024-04-24T08:21:13.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.331+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 21 ==== osd_op_reply(71 [pgnls start_epoch 113 out=73b] v112'3 uv3 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.331+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:72 16.3 16:c0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104000 2024-04-24T08:21:13.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.332+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 9 ==== osd_op_reply(72 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.332+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:73 16.13 16:c8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a04d2800 2024-04-24T08:21:13.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.333+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.1 v2:172.21.15.69:6815/153736297 6 ==== osd_op_reply(73 [pgnls start_epoch 113 out=98b] v112'5 uv5 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x5621a0f58500 con 0x5621a04d2800 2024-04-24T08:21:13.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.333+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:74 16.b 16:d0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a04d2800 2024-04-24T08:21:13.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.334+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.1 v2:172.21.15.69:6815/153736297 7 ==== osd_op_reply(74 [pgnls start_epoch 113 out=76b] v113'6 uv5 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x5621a0f58500 con 0x5621a04d2800 2024-04-24T08:21:13.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.334+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:75 16.1b 16:d8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104000 2024-04-24T08:21:13.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.335+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 10 ==== osd_op_reply(75 [pgnls start_epoch 113 out=49b] v113'3 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.335+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:76 16.7 16:e0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104800 2024-04-24T08:21:13.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.335+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 10 ==== osd_op_reply(76 [pgnls start_epoch 113 out=105b] v113'9 uv9 ondisk = 1) ==== 144+0+105 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.335+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:77 16.17 16:e8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fddc00 2024-04-24T08:21:13.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.336+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 22 ==== osd_op_reply(77 [pgnls start_epoch 113 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.336+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:78 16.f 16:f0000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a0fdd800 2024-04-24T08:21:13.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.337+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 16 ==== osd_op_reply(78 [pgnls start_epoch 113 out=73b] v113'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.337+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:79 16.1f 16:f8000000::::head [pgnls start_epoch 113 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582000 con 0x5621a1104000 2024-04-24T08:21:13.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.338+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 11 ==== osd_op_reply(79 [pgnls start_epoch 113 out=101b] v113'9 uv8 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.338+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 601.00000000 2024-04-24T08:21:13.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.338+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:80 16.10 16:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582400 con 0x5621a0fddc00 2024-04-24T08:21:13.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.339+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 23 ==== osd_op_reply(80 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.339+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 602.00000000 2024-04-24T08:21:13.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.339+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:81 16.10 16:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582800 con 0x5621a0fddc00 2024-04-24T08:21:13.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.339+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 24 ==== osd_op_reply(81 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.339+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-04-24T08:21:13.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.340+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:82 16.10 16:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582c00 con 0x5621a0fddc00 2024-04-24T08:21:13.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.340+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 25 ==== osd_op_reply(82 10000000000.00000000 [omap-get-vals out=1047b] v0'0 uv12 ondisk = 0) ==== 164+0+1047 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.340+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 600.00000000 2024-04-24T08:21:13.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.340+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:83 16.10 16:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0583000 con 0x5621a0fddc00 2024-04-24T08:21:13.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 26 ==== osd_op_reply(83 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 603.00000000 2024-04-24T08:21:13.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:84 16.10 16:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0583400 con 0x5621a0fddc00 2024-04-24T08:21:13.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 27 ==== osd_op_reply(84 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-04-24T08:21:13.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-24T08:21:13.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 200.00000000 2024-04-24T08:21:13.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-04-24T08:21:13.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 200.00000001 2024-04-24T08:21:13.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-04-24T08:21:13.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 606.00000000 2024-04-24T08:21:13.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.341+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:85 16.14 16:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0583800 con 0x5621a0fdd800 2024-04-24T08:21:13.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.342+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 17 ==== osd_op_reply(85 606.00000000 [omap-get-vals out=5b] v0'0 uv39 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.342+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 607.00000000 2024-04-24T08:21:13.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.342+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:86 16.1c 16:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0583c00 con 0x5621a0fddc00 2024-04-24T08:21:13.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.343+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 28 ==== osd_op_reply(86 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.343+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 10000000003.00000000 2024-04-24T08:21:13.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.343+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:87 16.12 16:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582400 con 0x5621a04d2000 2024-04-24T08:21:13.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.344+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.6 v2:172.21.15.121:6807/3068198325 8 ==== osd_op_reply(87 10000000003.00000000 [omap-get-vals out=523b] v0'0 uv3 ondisk = 0) ==== 164+0+523 (crc 0 0 0) 0x5621a0f58780 con 0x5621a04d2000 2024-04-24T08:21:13.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.344+0000 7f733021a740 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-04-24T08:21:13.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.344+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-04-24T08:21:13.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.344+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 608.00000000 2024-04-24T08:21:13.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.344+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:88 16.e 16:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582800 con 0x5621a0fddc00 2024-04-24T08:21:13.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.345+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 29 ==== osd_op_reply(88 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.345+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 604.00000000 2024-04-24T08:21:13.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.345+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- osd_op(unknown.0.0:89 16.15 16:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0582c00 con 0x5621a0fdd800 2024-04-24T08:21:13.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.346+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.0 v2:172.21.15.69:6800/146737935 18 ==== osd_op_reply(89 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58000 con 0x5621a0fdd800 2024-04-24T08:21:13.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.346+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 500.00000000 2024-04-24T08:21:13.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.346+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-04-24T08:21:13.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.346+0000 7f733021a740 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-04-24T08:21:13.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.346+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-24T08:21:13.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.346+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 605.00000000 2024-04-24T08:21:13.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.346+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- osd_op(unknown.0.0:90 16.1d 16:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0583000 con 0x5621a0fddc00 2024-04-24T08:21:13.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.347+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.2 v2:172.21.15.69:6808/2630808717 30 ==== osd_op_reply(90 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58500 con 0x5621a0fddc00 2024-04-24T08:21:13.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.347+0000 7f733021a740 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-04-24T08:21:13.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.347+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-24T08:21:13.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.347+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 100.00000000 2024-04-24T08:21:13.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.347+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:91 16.13 16:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a04d3000 con 0x5621a04d2800 2024-04-24T08:21:13.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.348+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.1 v2:172.21.15.69:6815/153736297 8 ==== osd_op_reply(91 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) ==== 156+0+5195 (crc 0 0 0) 0x5621a0f58500 con 0x5621a04d2800 2024-04-24T08:21:13.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.348+0000 7f733021a740 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-04-24T08:21:13.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.348+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-24T08:21:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.348+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 609.00000000 2024-04-24T08:21:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.348+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:92 16.7 16:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578000 con 0x5621a1104800 2024-04-24T08:21:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 11 ==== osd_op_reply(92 609.00000000 [omap-get-vals out=5b] v0'0 uv7 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 10000000001.00000000 2024-04-24T08:21:13.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:93 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578800 con 0x5621a1104800 2024-04-24T08:21:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 12 ==== osd_op_reply(93 10000000001.00000000 [omap-get-vals out=1545b] v0'0 uv9 ondisk = 0) ==== 164+0+1545 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f733021a740 20 datascan.scan_links: injected ino detected 2024-04-24T08:21:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f733021a740 20 datascan.scan_links: adding 0x100000001f7 for future processing to fix dnfirst 2024-04-24T08:21:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 400.00000000 2024-04-24T08:21:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-04-24T08:21:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-04-24T08:21:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f733021a740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-24T08:21:13.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f733021a740 10 datascan.scan_links: step 2: handling object 1.00000000 2024-04-24T08:21:13.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.349+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- osd_op(unknown.0.0:94 16.1f 16:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0579000 con 0x5621a1104000 2024-04-24T08:21:13.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.351+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 <== osd.5 v2:172.21.15.121:6822/2187246263 12 ==== osd_op_reply(94 1.00000000 [omap-get-vals out=524b] v0'0 uv8 ondisk = 0) ==== 154+0+524 (crc 0 0 0) 0x5621a0f58000 con 0x5621a1104000 2024-04-24T08:21:13.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.351+0000 7f733021a740 10 datascan.scan_links: processing 0 dup_primaries, 0 remote_links 2024-04-24T08:21:13.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.351+0000 7f733021a740 10 datascan.scan_links: removing dup dentries from 0 objects 2024-04-24T08:21:13.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.351+0000 7f733021a740 10 datascan.scan_links: processing 0 bad_nlink_inos 2024-04-24T08:21:13.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.351+0000 7f733021a740 10 datascan.scan_links: processing 1 injected_inos 2024-04-24T08:21:13.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.351+0000 7f733021a740 10 datascan.scan_links: handling injected_ino 0x100000001f7 2024-04-24T08:21:13.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.351+0000 7f733021a740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000001:bravo 2024-04-24T08:21:13.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.351+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:95 16.7 16:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0579800 con 0x5621a1104800 2024-04-24T08:21:13.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.351+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 13 ==== osd_op_reply(95 10000000001.00000000 [omap-get-vals-by-keys out=506b] v0'0 uv9 ondisk = 0) ==== 164+0+506 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.352+0000 7f733021a740 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-04-24T08:21:13.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.352+0000 7f733021a740 20 datascan.scan_links: first is now 2 2024-04-24T08:21:13.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.352+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- osd_op(unknown.0.0:96 16.7 16:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578400 con 0x5621a1104800 2024-04-24T08:21:13.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.355+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.3 v2:172.21.15.69:6811/1875112303 14 ==== osd_op_reply(96 10000000001.00000000 [omap-set-vals] v113'10 uv10 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x5621a0f58780 con 0x5621a1104800 2024-04-24T08:21:13.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.355+0000 7f733021a740 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-04-24T08:21:13.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.355+0000 7f733021a740 10 datascan.scan_links: updating inotable 2024-04-24T08:21:13.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.355+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:97 16.13 16:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578000 con 0x5621a04d2800 2024-04-24T08:21:13.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.356+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.1 v2:172.21.15.69:6815/153736297 9 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv4 ondisk = 0) ==== 157+0+50 (crc 0 0 0) 0x5621a0f58500 con 0x5621a04d2800 2024-04-24T08:21:13.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.356+0000 7f733021a740 10 mds.0.inotable: skip_inos was [0x10000000005~0x1f2,0x100000001f9~0xfffffffe07] 2024-04-24T08:21:13.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.356+0000 7f733021a740 10 mds.0.inotable: skip_inos now [0x100000001f9~0xfffffffe07] 2024-04-24T08:21:13.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.356+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- osd_op(unknown.0.0:98 16.13 16:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113) -- 0x5621a0578800 con 0x5621a04d2800 2024-04-24T08:21:13.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.360+0000 7f732e200640 1 -- 172.21.15.69:0/4259505759 <== osd.1 v2:172.21.15.69:6815/153736297 10 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v113'6 uv6 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x5621a0f58500 con 0x5621a04d2800 2024-04-24T08:21:13.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.360+0000 7f733021a740 10 datascan.scan_links: updating snaptable 2024-04-24T08:21:13.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.360+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- osd_op(unknown.0.0:99 16.d 16:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) -- 0x5621a0579c00 con 0x5621a04d2000 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.361+0000 7f732d800640 1 -- 172.21.15.69:0/4259505759 <== osd.6 v2:172.21.15.121:6807/3068198325 9 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x5621a0f58780 con 0x5621a04d2000 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x5621a0fdd800 msgr2=0x5621a0f42680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x5621a0fdd800 0x5621a0f42680 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5621a04d2800 msgr2=0x5621a0556c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x5621a04d2800 0x5621a0556c00 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x5621a0fddc00 msgr2=0x5621a0554000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x5621a0fddc00 0x5621a0554000 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x5621a1104800 msgr2=0x5621a0554580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x5621a1104800 0x5621a0554580 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x5621a1104400 msgr2=0x5621a0555080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x5621a1104400 0x5621a0555080 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x5621a1104000 msgr2=0x5621a0554b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x5621a1104000 0x5621a0554b00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x5621a04d2000 msgr2=0x5621a0555600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x5621a04d2000 0x5621a0555600 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x5621a04d2400 msgr2=0x5621a0555b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x5621a04d2400 0x5621a0555b80 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 reap_dead start 2024-04-24T08:21:13.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 reap_dead start 2024-04-24T08:21:13.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 reap_dead start 2024-04-24T08:21:13.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f732ce00640 1 -- 172.21.15.69:0/4259505759 reap_dead start 2024-04-24T08:21:13.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.362+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x5621a1105c00 msgr2=0x5621a0f42100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.363+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x5621a1105c00 0x5621a0f42100 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x5621a0f6cf30 tx=0x5621a0f6a720 comp rx=0 tx=0).stop 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.363+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd000 msgr2=0x5621a0f41600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.363+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd000 0x5621a0f41600 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x5621a0f39a40 tx=0x5621a0f6aa80 comp rx=0 tx=0).stop 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.363+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 shutdown_connections 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.363+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x5621a1105c00 0x5621a0f42100 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.363+0000 7f733021a740 1 --2- 172.21.15.69:0/4259505759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0fdd000 0x5621a0f41600 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.363+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 >> 172.21.15.69:0/4259505759 conn(0x5621a0f69400 msgr2=0x5621a02b7800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.363+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 shutdown_connections 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.363+0000 7f733021a740 1 -- 172.21.15.69:0/4259505759 wait complete. 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.364+0000 7f733021a740 1 -- 172.21.15.69:0/2459086427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0f68400 msgr2=0x5621a0f40b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.364+0000 7f733021a740 1 --2- 172.21.15.69:0/2459086427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0f68400 0x5621a0f40b00 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x5621a1001a10 tx=0x5621a0f6e120 comp rx=0 tx=0).stop 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.364+0000 7f733021a740 1 -- 172.21.15.69:0/2459086427 shutdown_connections 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.364+0000 7f733021a740 1 --2- 172.21.15.69:0/2459086427 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5621a0f69c00 0x5621a0f40580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.364+0000 7f733021a740 1 --2- 172.21.15.69:0/2459086427 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x5621a0f68800 0x5621a0f41080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.364+0000 7f733021a740 1 --2- 172.21.15.69:0/2459086427 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5621a0f68400 0x5621a0f40b00 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.364+0000 7f733021a740 1 -- 172.21.15.69:0/2459086427 >> 172.21.15.69:0/2459086427 conn(0x5621a01cb000 msgr2=0x5621a02b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.365+0000 7f733021a740 1 -- 172.21.15.69:0/2459086427 shutdown_connections 2024-04-24T08:21:13.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.365+0000 7f733021a740 1 -- 172.21.15.69:0/2459086427 wait complete. 2024-04-24T08:21:13.374 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.215977, result: 2024-04-24T08:21:13.375 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T08:21:13.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T08:21:13.511 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-24T08:21:13.511 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-24T08:21:13.511 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T08:21:13.511 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-24T08:21:13.511 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-24T08:21:13.512 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T08:21:13.513 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.514 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.515 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-24T08:21:13.516 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-04-24T08:21:13.517 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-24T08:21:13.518 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T08:21:13.519 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon down mkfs grace = 300 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-24T08:21:13.520 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-24T08:21:13.521 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = False 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-24T08:21:13.522 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-24T08:21:13.530 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-04-24T08:21:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-04-24T08:21:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T08:21:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-04-24T08:21:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-04-24T08:21:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-04-24T08:21:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.530 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-04-24T08:21:13.531 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.532 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-04-24T08:21:13.533 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-04-24T08:21:13.534 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: bdev async discard = True 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: bdev enable discard = True 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore allocator = bitmap 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 1/20 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 1/20 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 4/10 2024-04-24T08:21:13.535 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-24T08:21:13.536 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T08:21:13.537 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: mon down mkfs grace = 300 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-04-24T08:21:13.538 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = "" 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2024-04-24T08:21:13.539 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2024-04-24T08:21:13.540 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2024-04-24T08:21:13.540 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2024-04-24T08:21:13.540 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2024-04-24T08:21:13.541 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds balancer = 20 2024-04-24T08:21:13.541 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:21:13.541 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2024-04-24T08:21:13.541 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = False 2024-04-24T08:21:13.541 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2024-04-24T08:21:13.541 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2024-04-24T08:21:13.541 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-24T08:21:13.541 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-24T08:21:13.541 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-24T08:21:13.564 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T08:21:13.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T08:21:13.701 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: chdir = "" 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: auth supported = cephx 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: filestore xattr use omap = true 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: mon clock drift allowed = 1.000 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: osd crush chooseleaf type = 0 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: auth debug = true 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on old message = true 2024-04-24T08:21:13.702 INFO:teuthology.orchestra.run.smithi069.stdout: ms die on bug = true 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: mon pg warn max object skew = 0 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default size = 2 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow primary affinity = true 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd allow pg remap = true 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on legacy crush tunables = false 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on crush straw calc version zero = false 2024-04-24T08:21:13.703 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on no sortbitwise = false 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on osd down out interval zero = false 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: mon warn on too few osds = false 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: mon_allow_pool_size_one = true 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: osd default data pool replay window = 5 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: mon allow pool delete = true 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: mon cluster log file level = debug 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: debug asserts on shutdown = true 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: mon health detail to clog = false 2024-04-24T08:21:13.704 INFO:teuthology.orchestra.run.smithi069.stdout: mon host = "172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout:[osd] 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: osd journal size = 100 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub load threshold = 5.0 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: osd scrub max interval = 600 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: osd mclock profile = high_recovery_ops 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: osd recover clone overlap = true 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: osd recovery max chunk = 1048576 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug shutdown = true 2024-04-24T08:21:13.705 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug op order = true 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug verify stray on activate = true 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug trim objects = true 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: osd open classes on start = true 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug pg log writeout = true 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: osd deep scrub update digest min age = 30 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: osd map max advance = 10 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: journal zero on create = true 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: filestore ondisk finisher threads = 3 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: filestore apply finisher threads = 3 2024-04-24T08:21:13.706 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: bdev debug aio = true 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: osd debug misdirected ops = true 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: bdev async discard = True 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: bdev enable discard = True 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore allocator = bitmap 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore block size = 96636764160 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: bluestore fsck on mount = True 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluefs = 1/20 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: debug bluestore = 1/20 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: debug osd = 20 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: debug rocksdb = 4/10 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd full ratio = 0.9 2024-04-24T08:21:13.707 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd nearfull ratio = 0.8 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: osd failsafe full ratio = 0.95 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: osd objectstore = bluestore 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout:[mgr] 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: debug mgr = 20 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: mgr/telemetry/nag = false 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout:[mon] 2024-04-24T08:21:13.708 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: debug mon = 20 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: debug paxos = 20 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: debug auth = 20 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: mon data avail warn = 5 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: mon mgr mkfs grace = 240 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd reporter subtree level = osd 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: mon osd prime pg temp = true 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.709 INFO:teuthology.orchestra.run.smithi069.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: # 1m isn't quite enough 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: mon_down_mkfs_grace = 2m 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: mon_warn_on_filestore_osds = false 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: mon down mkfs grace = 300 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: mon op complaint time = 120 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout:[client] 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: rgw cache enabled = true 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable ops log = true 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: rgw enable usage log = true 2024-04-24T08:21:13.710 INFO:teuthology.orchestra.run.smithi069.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout: client mount timeout = 600 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout: debug client = 20 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout: pid file = "" 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.a] 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.b] 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout:[mon.c] 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a] 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c] 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b] 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d] 2024-04-24T08:21:13.711 INFO:teuthology.orchestra.run.smithi069.stdout:[mds] 2024-04-24T08:21:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds = 20 2024-04-24T08:21:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: debug mds balancer = 20 2024-04-24T08:21:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: debug ms = 1 2024-04-24T08:21:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug frag = True 2024-04-24T08:21:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: mds debug scatterstat = True 2024-04-24T08:21:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: mds op complaint time = 180 2024-04-24T08:21:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: mds verify scatter = True 2024-04-24T08:21:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: osd op complaint time = 180 2024-04-24T08:21:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: rados mon op timeout = 900 2024-04-24T08:21:13.712 INFO:teuthology.orchestra.run.smithi069.stdout: rados osd op timeout = 900 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2024-04-24T08:21:13.713 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2024-04-24T08:21:13.714 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.715 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.69,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 3422a353-af9e-4ed5-a4b2-7365b9c531c6 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: osd mclock profile = high_recovery_ops 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2024-04-24T08:21:13.716 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = true 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug trim objects = true 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2024-04-24T08:21:13.717 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: bdev async discard = True 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: bdev enable discard = True 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore allocator = bitmap 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 1/20 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 1/20 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 4/10 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T08:21:13.718 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-24T08:21:13.719 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T08:21:13.720 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: mon down mkfs grace = 300 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-04-24T08:21:13.721 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = "" 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2024-04-24T08:21:13.722 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds balancer = 20 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2024-04-24T08:21:13.723 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2024-04-24T08:21:13.724 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2024-04-24T08:21:13.741 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-24T08:21:13.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.941+0000 7fab82800640 1 Processor -- start 2024-04-24T08:21:13.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.941+0000 7fab82800640 1 -- start start 2024-04-24T08:21:13.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.943+0000 7fab82800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fab7c162f90 0x7fab7c0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.943+0000 7fab82800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab7c0d0190 0x7fab7c06cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.943+0000 7fab82800640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab7c15ed60 con 0x7fab7c15e980 2024-04-24T08:21:13.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.943+0000 7fab82800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fab7c1633d0 con 0x7fab7c162f90 2024-04-24T08:21:13.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.943+0000 7fab82800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fab7c0d1580 con 0x7fab7c0d0190 2024-04-24T08:21:13.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.943+0000 7fab80a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fab7c162f90 0x7fab7c0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.944+0000 7fab80a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fab7c162f90 0x7fab7c0cfc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53256/0 (socket says 172.21.15.69:53256) 2024-04-24T08:21:13.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.944+0000 7fab80a00640 1 -- 172.21.15.69:0/1279106619 learned_addr learned my addr 172.21.15.69:0/1279106619 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:13.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.944+0000 7fab81e00640 1 --2- 172.21.15.69:0/1279106619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab7c0d0190 0x7fab7c06cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.944+0000 7fab80a00640 1 -- 172.21.15.69:0/1279106619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab7c0d0190 msgr2=0x7fab7c06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:13.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.944+0000 7fab80a00640 1 --2- 172.21.15.69:0/1279106619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab7c0d0190 0x7fab7c06cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.944+0000 7fab80a00640 1 -- 172.21.15.69:0/1279106619 >> v1:172.21.15.121:6789/0 conn(0x7fab7c15e980 legacy=0x7fab7c0d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.944+0000 7fab80a00640 1 -- 172.21.15.69:0/1279106619 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7c0d1860 con 0x7fab7c162f90 2024-04-24T08:21:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.945+0000 7fab80a00640 1 --2- 172.21.15.69:0/1279106619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 0x7fab7c0cfc50 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fab68009870 tx=0x7fab6802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0daa92a03c826bd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.946+0000 7fab7be00640 1 -- 172.21.15.69:0/1279106619 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab68037070 con 0x7fab7c162f90 2024-04-24T08:21:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.946+0000 7fab7be00640 1 -- 172.21.15.69:0/1279106619 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab6802ae50 con 0x7fab7c162f90 2024-04-24T08:21:13.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.946+0000 7fab7be00640 1 -- 172.21.15.69:0/1279106619 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab68032860 con 0x7fab7c162f90 2024-04-24T08:21:13.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.946+0000 7fab82800640 1 -- 172.21.15.69:0/1279106619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 msgr2=0x7fab7c0cfc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.946+0000 7fab82800640 1 --2- 172.21.15.69:0/1279106619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 0x7fab7c0cfc50 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7fab68009870 tx=0x7fab6802a300 comp rx=0 tx=0).stop 2024-04-24T08:21:13.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.946+0000 7fab82800640 1 -- 172.21.15.69:0/1279106619 shutdown_connections 2024-04-24T08:21:13.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.946+0000 7fab82800640 1 --2- 172.21.15.69:0/1279106619 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab7c0d0190 0x7fab7c06cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.946+0000 7fab82800640 1 --2- 172.21.15.69:0/1279106619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 0x7fab7c0cfc50 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.946+0000 7fab82800640 1 -- 172.21.15.69:0/1279106619 >> 172.21.15.69:0/1279106619 conn(0x7fab7c09d920 msgr2=0x7fab7c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:13.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.947+0000 7fab82800640 1 -- 172.21.15.69:0/1279106619 shutdown_connections 2024-04-24T08:21:13.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.947+0000 7fab82800640 1 -- 172.21.15.69:0/1279106619 wait complete. 2024-04-24T08:21:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.948+0000 7fab82800640 1 Processor -- start 2024-04-24T08:21:13.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.948+0000 7fab82800640 1 -- start start 2024-04-24T08:21:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.949+0000 7fab82800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab7c0d0190 0x7fab7c150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.949+0000 7fab82800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab7c15e980 0x7fab7c0cb630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.949+0000 7fab82800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 0x7fab7c0cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.949+0000 7fab82800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c0cf070 con 0x7fab7c162f90 2024-04-24T08:21:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.949+0000 7fab82800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fab7c0cf730 con 0x7fab7c15e980 2024-04-24T08:21:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.949+0000 7fab82800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fab7c0cf1f0 con 0x7fab7c0d0190 2024-04-24T08:21:13.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab81e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 0x7fab7c0cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab81e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 0x7fab7c0cbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53272/0 (socket says 172.21.15.69:53272) 2024-04-24T08:21:13.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab81e00640 1 -- 172.21.15.69:0/4079811920 learned_addr learned my addr 172.21.15.69:0/4079811920 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab80a00640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab7c15e980 0x7fab7c0cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab81400640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab7c0d0190 0x7fab7c150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab81e00640 1 -- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab7c0d0190 msgr2=0x7fab7c150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab81e00640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab7c0d0190 0x7fab7c150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab81e00640 1 -- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab7c15e980 msgr2=0x7fab7c0cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab81e00640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab7c15e980 0x7fab7c0cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab81e00640 1 -- 172.21.15.69:0/4079811920 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab7c15fe50 con 0x7fab7c162f90 2024-04-24T08:21:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.950+0000 7fab81e00640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 0x7fab7c0cbb70 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fab74007e70 tx=0x7fab7400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.951+0000 7fab79600640 1 -- 172.21.15.69:0/4079811920 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab74014620 con 0x7fab7c162f90 2024-04-24T08:21:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.951+0000 7fab79600640 1 -- 172.21.15.69:0/4079811920 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab74019070 con 0x7fab7c162f90 2024-04-24T08:21:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.951+0000 7fab79600640 1 -- 172.21.15.69:0/4079811920 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab74014a20 con 0x7fab7c162f90 2024-04-24T08:21:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.951+0000 7fab82800640 1 -- 172.21.15.69:0/4079811920 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab7c160130 con 0x7fab7c162f90 2024-04-24T08:21:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.951+0000 7fab82800640 1 -- 172.21.15.69:0/4079811920 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab7c1606f0 con 0x7fab7c162f90 2024-04-24T08:21:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.952+0000 7fab82800640 1 -- 172.21.15.69:0/4079811920 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab7c075d50 con 0x7fab7c162f90 2024-04-24T08:21:13.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.953+0000 7fab79600640 1 -- 172.21.15.69:0/4079811920 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fab74014bc0 con 0x7fab7c162f90 2024-04-24T08:21:13.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.954+0000 7fab79600640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab5c06c920 0x7fab5c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:13.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.954+0000 7fab79600640 1 -- 172.21.15.69:0/4079811920 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fab7408c5f0 con 0x7fab7c162f90 2024-04-24T08:21:13.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.955+0000 7fab81400640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab5c06c920 0x7fab5c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:13.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.956+0000 7fab81400640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab5c06c920 0x7fab5c06ede0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fab70000f60 tx=0x7fab70009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:13.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:13.958+0000 7fab79600640 1 -- 172.21.15.69:0/4079811920 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fab74011260 con 0x7fab7c162f90 2024-04-24T08:21:14.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.198+0000 7fab82800640 1 -- 172.21.15.69:0/4079811920 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7fab7c06e630 con 0x7fab7c162f90 2024-04-24T08:21:14.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.617+0000 7fab79600640 1 -- 172.21.15.69:0/4079811920 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v108) ==== 162+0+0 (secure 0 0 0) 0x7fab7405a890 con 0x7fab7c162f90 2024-04-24T08:21:14.619 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-24T08:21:14.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 -- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab5c06c920 msgr2=0x7fab5c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab5c06c920 0x7fab5c06ede0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fab70000f60 tx=0x7fab70009040 comp rx=0 tx=0).stop 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 -- 172.21.15.69:0/4079811920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 msgr2=0x7fab7c0cbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 0x7fab7c0cbb70 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7fab74007e70 tx=0x7fab7400e4b0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 -- 172.21.15.69:0/4079811920 shutdown_connections 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fab5c06c920 0x7fab5c06ede0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab7c162f90 0x7fab7c0cbb70 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fab7c15e980 0x7fab7c0cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 --2- 172.21.15.69:0/4079811920 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fab7c0d0190 0x7fab7c150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 -- 172.21.15.69:0/4079811920 >> 172.21.15.69:0/4079811920 conn(0x7fab7c09d920 msgr2=0x7fab7c0b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 -- 172.21.15.69:0/4079811920 shutdown_connections 2024-04-24T08:21:14.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.623+0000 7fab82800640 1 -- 172.21.15.69:0/4079811920 wait complete. 2024-04-24T08:21:14.650 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:14.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.830+0000 7f281ce00640 1 Processor -- start 2024-04-24T08:21:14.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.830+0000 7f281ce00640 1 -- start start 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.832+0000 7f281ce00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f28180afde0 0x7f281806ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.832+0000 7f281ce00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f281815cbb0 con 0x7f281806cf70 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.832+0000 7f281ce00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f281815ce90 con 0x7f281815fb30 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.832+0000 7f281ce00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f281815d970 con 0x7f28180afde0 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.832+0000 7f2816a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f28180afde0 0x7f281806ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.832+0000 7f2816a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f28180afde0 0x7f281806ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53284/0 (socket says 172.21.15.69:53284) 2024-04-24T08:21:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.832+0000 7f2816a00640 1 -- 172.21.15.69:0/1194728706 learned_addr learned my addr 172.21.15.69:0/1194728706 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.833+0000 7f2816a00640 1 -- 172.21.15.69:0/1194728706 >> v1:172.21.15.121:6789/0 conn(0x7f281815fb30 legacy=0x7f2818073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.833+0000 7f2816a00640 1 -- 172.21.15.69:0/1194728706 >> v1:172.21.15.69:6789/0 conn(0x7f281806cf70 legacy=0x7f2818152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.833+0000 7f2816a00640 1 -- 172.21.15.69:0/1194728706 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f281815d170 con 0x7f28180afde0 2024-04-24T08:21:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.834+0000 7f2816a00640 1 --2- 172.21.15.69:0/1194728706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28180afde0 0x7f281806ca30 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f2800009870 tx=0x7f280002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad9e2b4c20091b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.834+0000 7f2816000640 1 -- 172.21.15.69:0/1194728706 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800037070 con 0x7f28180afde0 2024-04-24T08:21:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.834+0000 7f2816000640 1 -- 172.21.15.69:0/1194728706 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2800032440 con 0x7f28180afde0 2024-04-24T08:21:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.834+0000 7f2816000640 1 -- 172.21.15.69:0/1194728706 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2800032810 con 0x7f28180afde0 2024-04-24T08:21:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.835+0000 7f281ce00640 1 -- 172.21.15.69:0/1194728706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28180afde0 msgr2=0x7f281806ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.835+0000 7f281ce00640 1 --2- 172.21.15.69:0/1194728706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28180afde0 0x7f281806ca30 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f2800009870 tx=0x7f280002a210 comp rx=0 tx=0).stop 2024-04-24T08:21:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.835+0000 7f281ce00640 1 -- 172.21.15.69:0/1194728706 shutdown_connections 2024-04-24T08:21:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.835+0000 7f281ce00640 1 --2- 172.21.15.69:0/1194728706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28180afde0 0x7f281806ca30 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.835+0000 7f281ce00640 1 -- 172.21.15.69:0/1194728706 >> 172.21.15.69:0/1194728706 conn(0x7f28180722b0 msgr2=0x7f28180726c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.835+0000 7f281ce00640 1 -- 172.21.15.69:0/1194728706 shutdown_connections 2024-04-24T08:21:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.835+0000 7f281ce00640 1 -- 172.21.15.69:0/1194728706 wait complete. 2024-04-24T08:21:14.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.837+0000 7f281ce00640 1 Processor -- start 2024-04-24T08:21:14.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.837+0000 7f281ce00640 1 -- start start 2024-04-24T08:21:14.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.838+0000 7f281ce00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f281806cf70 0x7f281814e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.838+0000 7f281ce00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28180afde0 0x7f28181e7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.838+0000 7f281ce00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f281815fb30 0x7f281806f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.838+0000 7f281ce00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f281806d3c0 con 0x7f281815fb30 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.838+0000 7f281ce00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f28180ce3c0 con 0x7f28180afde0 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.838+0000 7f281ce00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2818073850 con 0x7f281806cf70 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.838+0000 7f2817e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f281815fb30 0x7f281806f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.838+0000 7f2817e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f281815fb30 0x7f281806f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53288/0 (socket says 172.21.15.69:53288) 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.838+0000 7f2817e00640 1 -- 172.21.15.69:0/546127903 learned_addr learned my addr 172.21.15.69:0/546127903 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.839+0000 7f2816a00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28180afde0 0x7f28181e7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.839+0000 7f2817400640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f281806cf70 0x7f281814e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.839+0000 7f2817e00640 1 -- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f281806cf70 msgr2=0x7f281814e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.839+0000 7f2817e00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f281806cf70 0x7f281814e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.839+0000 7f2817e00640 1 -- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28180afde0 msgr2=0x7f28181e7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.839+0000 7f2817e00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28180afde0 0x7f28181e7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.839+0000 7f2817e00640 1 -- 172.21.15.69:0/546127903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28181e8820 con 0x7f281815fb30 2024-04-24T08:21:14.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.839+0000 7f2816a00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28180afde0 0x7f28181e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:14.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.839+0000 7f2817e00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f281815fb30 0x7f281806f380 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f281000c7d0 tx=0x7f281000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:14.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.840+0000 7f280f400640 1 -- 172.21.15.69:0/546127903 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2810003b70 con 0x7f281815fb30 2024-04-24T08:21:14.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.840+0000 7f280f400640 1 -- 172.21.15.69:0/546127903 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2810003d30 con 0x7f281815fb30 2024-04-24T08:21:14.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.840+0000 7f280f400640 1 -- 172.21.15.69:0/546127903 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281000aa90 con 0x7f281815fb30 2024-04-24T08:21:14.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.840+0000 7f281ce00640 1 -- 172.21.15.69:0/546127903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28181e8b00 con 0x7f281815fb30 2024-04-24T08:21:14.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.840+0000 7f281ce00640 1 -- 172.21.15.69:0/546127903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28181e90c0 con 0x7f281815fb30 2024-04-24T08:21:14.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.841+0000 7f281ce00640 1 -- 172.21.15.69:0/546127903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2818075d50 con 0x7f281815fb30 2024-04-24T08:21:14.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.842+0000 7f280f400640 1 -- 172.21.15.69:0/546127903 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f281000ac30 con 0x7f281815fb30 2024-04-24T08:21:14.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.843+0000 7f280f400640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f27f406c8f0 0x7f27f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:14.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.843+0000 7f280f400640 1 -- 172.21.15.69:0/546127903 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f281008bb10 con 0x7f281815fb30 2024-04-24T08:21:14.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.844+0000 7f2817400640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f27f406c8f0 0x7f27f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:14.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.845+0000 7f2817400640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f27f406c8f0 0x7f27f406edb0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2808006fd0 tx=0x7f2808008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:14.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:14.848+0000 7f280f400640 1 -- 172.21.15.69:0/546127903 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2810059ee0 con 0x7f281815fb30 2024-04-24T08:21:15.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.092+0000 7f281ce00640 1 -- 172.21.15.69:0/546127903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2818067c60 con 0x7f281815fb30 2024-04-24T08:21:15.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.093+0000 7f280f400640 1 -- 172.21.15.69:0/546127903 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) ==== 95+0+5179 (secure 0 0 0) 0x7f28100597e0 con 0x7f281815fb30 2024-04-24T08:21:15.095 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:15.096 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":109,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:21:14.622620+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9825},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9825":{"gid":9825,"name":"c","rank":0,"incarnation":109,"state":"up:replay","state_seq":1,"addr":"172.21.15.69:6835/99442307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":99442307},{"type":"v1","addr":"172.21.15.69:6835","nonce":99442307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-04-24T08:21:15.096 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 109 2024-04-24T08:21:15.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.097+0000 7f281ce00640 1 -- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f27f406c8f0 msgr2=0x7f27f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.097+0000 7f281ce00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f27f406c8f0 0x7f27f406edb0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2808006fd0 tx=0x7f2808008040 comp rx=0 tx=0).stop 2024-04-24T08:21:15.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.097+0000 7f281ce00640 1 -- 172.21.15.69:0/546127903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f281815fb30 msgr2=0x7f281806f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:15.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.097+0000 7f281ce00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f281815fb30 0x7f281806f380 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f281000c7d0 tx=0x7f281000cc90 comp rx=0 tx=0).stop 2024-04-24T08:21:15.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.097+0000 7f281ce00640 1 -- 172.21.15.69:0/546127903 shutdown_connections 2024-04-24T08:21:15.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.097+0000 7f281ce00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f27f406c8f0 0x7f27f406edb0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.097+0000 7f281ce00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f281815fb30 0x7f281806f380 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.097+0000 7f281ce00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28180afde0 0x7f28181e7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.097+0000 7f281ce00640 1 --2- 172.21.15.69:0/546127903 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f281806cf70 0x7f281814e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:15.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.097+0000 7f281ce00640 1 -- 172.21.15.69:0/546127903 >> 172.21.15.69:0/546127903 conn(0x7f28180722b0 msgr2=0x7f2818073070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:15.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.098+0000 7f281ce00640 1 -- 172.21.15.69:0/546127903 shutdown_connections 2024-04-24T08:21:15.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:15.098+0000 7f281ce00640 1 -- 172.21.15.69:0/546127903 wait complete. 2024-04-24T08:21:15.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 109, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:20:09.204175+0000', 'modified': '2024-04-24T08:21:14.622620+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 113, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9825}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9825': {'gid': 9825, 'name': 'c', 'rank': 0, 'incarnation': 109, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.69:6835/99442307', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 99442307}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 99442307}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:21:15.521 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9825:up:replay 2024-04-24T08:21:16.522 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:16.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.733+0000 7f08fa000640 1 Processor -- start 2024-04-24T08:21:16.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.733+0000 7f08fa000640 1 -- start start 2024-04-24T08:21:16.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08fa000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08f415d1d0 0x7f08f40ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08fa000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08f40b30f0 0x7f08f40ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08fa000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08f415d660 con 0x7f08f4156c60 2024-04-24T08:21:16.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08fa000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f08f406dc30 con 0x7f08f40b30f0 2024-04-24T08:21:16.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08fa000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f08f406df10 con 0x7f08f415d1d0 2024-04-24T08:21:16.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08f9600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08f40b30f0 0x7f08f40ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08f9600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f08f40b30f0 0x7f08f40ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53304/0 (socket says 172.21.15.69:53304) 2024-04-24T08:21:16.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08f9600640 1 -- 172.21.15.69:0/1611888181 learned_addr learned my addr 172.21.15.69:0/1611888181 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:16.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08ebe00640 1 --2- 172.21.15.69:0/1611888181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08f415d1d0 0x7f08f40ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08f9600640 1 -- 172.21.15.69:0/1611888181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08f415d1d0 msgr2=0x7f08f40ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:16.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.735+0000 7f08f9600640 1 --2- 172.21.15.69:0/1611888181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08f415d1d0 0x7f08f40ad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.736+0000 7f08f9600640 1 -- 172.21.15.69:0/1611888181 >> v1:172.21.15.121:6789/0 conn(0x7f08f4156c60 legacy=0x7f08f415cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:16.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.736+0000 7f08f9600640 1 -- 172.21.15.69:0/1611888181 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f406e1f0 con 0x7f08f40b30f0 2024-04-24T08:21:16.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.736+0000 7f08ebe00640 1 --2- 172.21.15.69:0/1611888181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08f415d1d0 0x7f08f40ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:16.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.737+0000 7f08f9600640 1 --2- 172.21.15.69:0/1611888181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f40b30f0 0x7f08f40ad610 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009e50 tx=0x7f08ec02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5ebcf5e21847c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:16.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.737+0000 7f08f3e00640 1 -- 172.21.15.69:0/1611888181 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec036070 con 0x7f08f40b30f0 2024-04-24T08:21:16.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.737+0000 7f08f3e00640 1 -- 172.21.15.69:0/1611888181 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08ec031600 con 0x7f08f40b30f0 2024-04-24T08:21:16.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.737+0000 7f08f3e00640 1 -- 172.21.15.69:0/1611888181 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec0319d0 con 0x7f08f40b30f0 2024-04-24T08:21:16.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.737+0000 7f08fa000640 1 -- 172.21.15.69:0/1611888181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f40b30f0 msgr2=0x7f08f40ad610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.737+0000 7f08fa000640 1 --2- 172.21.15.69:0/1611888181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f40b30f0 0x7f08f40ad610 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f08ec009e50 tx=0x7f08ec02a6c0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.738+0000 7f08fa000640 1 -- 172.21.15.69:0/1611888181 shutdown_connections 2024-04-24T08:21:16.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.738+0000 7f08fa000640 1 --2- 172.21.15.69:0/1611888181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f40b30f0 0x7f08f40ad610 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.738+0000 7f08fa000640 1 --2- 172.21.15.69:0/1611888181 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08f415d1d0 0x7f08f40ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.738+0000 7f08fa000640 1 -- 172.21.15.69:0/1611888181 >> 172.21.15.69:0/1611888181 conn(0x7f08f40d0160 msgr2=0x7f08f40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:16.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.738+0000 7f08fa000640 1 -- 172.21.15.69:0/1611888181 shutdown_connections 2024-04-24T08:21:16.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.738+0000 7f08fa000640 1 -- 172.21.15.69:0/1611888181 wait complete. 2024-04-24T08:21:16.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.739+0000 7f08fa000640 1 Processor -- start 2024-04-24T08:21:16.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.739+0000 7f08fa000640 1 -- start start 2024-04-24T08:21:16.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.740+0000 7f08fa000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08f40b30f0 0x7f08f41f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.740+0000 7f08fa000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08f4156c60 0x7f08f41ef730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.740+0000 7f08fa000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f4155a10 0x7f08f4140280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.740+0000 7f08fa000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f08f406e710 con 0x7f08f4155a10 2024-04-24T08:21:16.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.740+0000 7f08fa000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f08f415d940 con 0x7f08f4156c60 2024-04-24T08:21:16.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.740+0000 7f08fa000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f08f406e890 con 0x7f08f40b30f0 2024-04-24T08:21:16.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.741+0000 7f08f9600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f4155a10 0x7f08f4140280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.741+0000 7f08f9600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f4155a10 0x7f08f4140280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53310/0 (socket says 172.21.15.69:53310) 2024-04-24T08:21:16.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.741+0000 7f08f9600640 1 -- 172.21.15.69:0/3379158837 learned_addr learned my addr 172.21.15.69:0/3379158837 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:16.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.741+0000 7f08ebe00640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08f4156c60 0x7f08f41ef730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.741+0000 7f08f9600640 1 -- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08f40b30f0 msgr2=0x7f08f41f2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.741+0000 7f08f9600640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08f40b30f0 0x7f08f41f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.741+0000 7f08f9600640 1 -- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08f4156c60 msgr2=0x7f08f41ef730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.741+0000 7f08f9600640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08f4156c60 0x7f08f41ef730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.741+0000 7f08f9600640 1 -- 172.21.15.69:0/3379158837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08f41f32e0 con 0x7f08f4155a10 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.741+0000 7f08f9600640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f4155a10 0x7f08f4140280 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f08ec034040 tx=0x7f08ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.742+0000 7f08f1600640 1 -- 172.21.15.69:0/3379158837 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec036050 con 0x7f08f4155a10 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.742+0000 7f08f1600640 1 -- 172.21.15.69:0/3379158837 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08ec042070 con 0x7f08f4155a10 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.742+0000 7f08f1600640 1 -- 172.21.15.69:0/3379158837 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec009510 con 0x7f08f4155a10 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.742+0000 7f08fa000640 1 -- 172.21.15.69:0/3379158837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08f41570e0 con 0x7f08f4155a10 2024-04-24T08:21:16.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.742+0000 7f08fa000640 1 -- 172.21.15.69:0/3379158837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08f4158e30 con 0x7f08f4155a10 2024-04-24T08:21:16.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.743+0000 7f08fa000640 1 -- 172.21.15.69:0/3379158837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08f4075d50 con 0x7f08f4155a10 2024-04-24T08:21:16.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.744+0000 7f08f1600640 1 -- 172.21.15.69:0/3379158837 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f08ec007bf0 con 0x7f08f4155a10 2024-04-24T08:21:16.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.745+0000 7f08f1600640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08d006c8f0 0x7f08d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:16.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.745+0000 7f08f1600640 1 -- 172.21.15.69:0/3379158837 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f08ec0acc90 con 0x7f08f4155a10 2024-04-24T08:21:16.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.745+0000 7f08f8c00640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08d006c8f0 0x7f08d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:16.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.746+0000 7f08f8c00640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08d006c8f0 0x7f08d006edb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f08f40cf5e0 tx=0x7f08e40079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:16.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.749+0000 7f08f1600640 1 -- 172.21.15.69:0/3379158837 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f08f41f32e0 con 0x7f08f4155a10 2024-04-24T08:21:16.813 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:21:16.821 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:21:16.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.990+0000 7f08fa000640 1 -- 172.21.15.69:0/3379158837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08f4156270 con 0x7f08f4155a10 2024-04-24T08:21:16.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.991+0000 7f08f1600640 1 -- 172.21.15.69:0/3379158837 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5179 (secure 0 0 0) 0x7f08ec07af30 con 0x7f08f4155a10 2024-04-24T08:21:16.993 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:16.993 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":111,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:21:15.643208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9825},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9825":{"gid":9825,"name":"c","rank":0,"incarnation":109,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.69:6835/99442307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":99442307},{"type":"v1","addr":"172.21.15.69:6835","nonce":99442307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":8}]} 2024-04-24T08:21:16.994 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 111 2024-04-24T08:21:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 -- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08d006c8f0 msgr2=0x7f08d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08d006c8f0 0x7f08d006edb0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f08f40cf5e0 tx=0x7f08e40079b0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 -- 172.21.15.69:0/3379158837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f4155a10 msgr2=0x7f08f4140280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f4155a10 0x7f08f4140280 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f08ec034040 tx=0x7f08ec004a40 comp rx=0 tx=0).stop 2024-04-24T08:21:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 -- 172.21.15.69:0/3379158837 shutdown_connections 2024-04-24T08:21:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f08d006c8f0 0x7f08d006edb0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08f4155a10 0x7f08f4140280 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08f4156c60 0x7f08f41ef730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 --2- 172.21.15.69:0/3379158837 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f08f40b30f0 0x7f08f41f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 -- 172.21.15.69:0/3379158837 >> 172.21.15.69:0/3379158837 conn(0x7f08f40d0160 msgr2=0x7f08f415aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.994+0000 7f08fa000640 1 -- 172.21.15.69:0/3379158837 shutdown_connections 2024-04-24T08:21:16.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:16.995+0000 7f08fa000640 1 -- 172.21.15.69:0/3379158837 wait complete. 2024-04-24T08:21:17.419 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 111, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:20:09.204175+0000', 'modified': '2024-04-24T08:21:15.643208+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 113, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9825}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9825': {'gid': 9825, 'name': 'c', 'rank': 0, 'incarnation': 109, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.69:6835/99442307', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 99442307}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 99442307}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T08:21:17.419 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9825:up:rejoin 2024-04-24T08:21:18.421 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:18.617 INFO:tasks.ceph.mds.c.smithi069.stderr:2024-04-24T08:21:18.615+0000 7f4367c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:21:18.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.628+0000 7fa4caa00640 1 Processor -- start 2024-04-24T08:21:18.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.628+0000 7fa4caa00640 1 -- start start 2024-04-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.629+0000 7fa4caa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4c41523f0 0x7fa4c406db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.629+0000 7fa4caa00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa4c415b980 0x7fa4c415be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.629+0000 7fa4caa00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4c40aeac0 con 0x7fa4c4072e20 2024-04-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.629+0000 7fa4caa00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c40aeda0 con 0x7fa4c415b980 2024-04-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.629+0000 7fa4caa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c40af080 con 0x7fa4c41523f0 2024-04-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.630+0000 7fa4c8c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa4c4072e20 0x7fa4c4073260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:48700/0 (socket says 172.21.15.69:48700) 2024-04-24T08:21:18.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.630+0000 7fa4c8c00640 1 -- 172.21.15.69:0/3874261918 learned_addr learned my addr 172.21.15.69:0/3874261918 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:18.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.630+0000 7fa4ca000640 1 --2- 172.21.15.69:0/3874261918 >> v2:172.21.15.69:3300/0 conn(0x7fa4c415b980 0x7fa4c415be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.630+0000 7fa4c9600640 1 --2- 172.21.15.69:0/3874261918 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4c41523f0 0x7fa4c406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.630+0000 7fa4ca000640 1 -- 172.21.15.69:0/3874261918 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4c41523f0 msgr2=0x7fa4c406db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:18.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.630+0000 7fa4ca000640 1 --2- 172.21.15.69:0/3874261918 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4c41523f0 0x7fa4c406db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.630+0000 7fa4ca000640 1 -- 172.21.15.69:0/3874261918 >> v1:172.21.15.69:6789/0 conn(0x7fa4c4072e20 legacy=0x7fa4c4073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.630+0000 7fa4ca000640 1 -- 172.21.15.69:0/3874261918 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c40af360 con 0x7fa4c415b980 2024-04-24T08:21:18.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.630+0000 7fa4c3e00640 1 -- 172.21.15.69:0/3874261918 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3182914885 0 0) 0x7fa4c40aeac0 con 0x7fa4c4072e20 2024-04-24T08:21:18.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.631+0000 7fa4ca000640 1 --2- 172.21.15.69:0/3874261918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 0x7fa4c415be10 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009d90 tx=0x7fa4bc02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e92bacb2c19417 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:18.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4c3e00640 1 -- 172.21.15.69:0/3874261918 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc031600 con 0x7fa4c415b980 2024-04-24T08:21:18.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4c3e00640 1 -- 172.21.15.69:0/3874261918 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4bc0317a0 con 0x7fa4c415b980 2024-04-24T08:21:18.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4c3e00640 1 -- 172.21.15.69:0/3874261918 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc031b70 con 0x7fa4c415b980 2024-04-24T08:21:18.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4caa00640 1 -- 172.21.15.69:0/3874261918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 msgr2=0x7fa4c415be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4caa00640 1 --2- 172.21.15.69:0/3874261918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 0x7fa4c415be10 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009d90 tx=0x7fa4bc02a5d0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4caa00640 1 -- 172.21.15.69:0/3874261918 shutdown_connections 2024-04-24T08:21:18.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4caa00640 1 --2- 172.21.15.69:0/3874261918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 0x7fa4c415be10 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4caa00640 1 --2- 172.21.15.69:0/3874261918 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4c41523f0 0x7fa4c406db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4caa00640 1 -- 172.21.15.69:0/3874261918 >> 172.21.15.69:0/3874261918 conn(0x7fa4c40d0160 msgr2=0x7fa4c40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:18.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4caa00640 1 -- 172.21.15.69:0/3874261918 shutdown_connections 2024-04-24T08:21:18.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.632+0000 7fa4caa00640 1 -- 172.21.15.69:0/3874261918 wait complete. 2024-04-24T08:21:18.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.634+0000 7fa4caa00640 1 Processor -- start 2024-04-24T08:21:18.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.634+0000 7fa4caa00640 1 -- start start 2024-04-24T08:21:18.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.635+0000 7fa4caa00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4c4072e20 0x7fa4c41595a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.635+0000 7fa4caa00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4c41523f0 0x7fa4c414f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.635+0000 7fa4caa00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 0x7fa4c414fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.635+0000 7fa4caa00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c4073970 con 0x7fa4c415b980 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.635+0000 7fa4caa00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c41527d0 con 0x7fa4c41523f0 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.635+0000 7fa4caa00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c4073af0 con 0x7fa4c4072e20 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.635+0000 7fa4ca000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 0x7fa4c414fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.635+0000 7fa4ca000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 0x7fa4c414fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53340/0 (socket says 172.21.15.69:53340) 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.635+0000 7fa4ca000640 1 -- 172.21.15.69:0/1159117792 learned_addr learned my addr 172.21.15.69:0/1159117792 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.636+0000 7fa4c9600640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4c4072e20 0x7fa4c41595a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.636+0000 7fa4ca000640 1 -- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4c4072e20 msgr2=0x7fa4c41595a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.636+0000 7fa4ca000640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4c4072e20 0x7fa4c41595a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.636+0000 7fa4c8c00640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4c41523f0 0x7fa4c414f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.636+0000 7fa4ca000640 1 -- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4c41523f0 msgr2=0x7fa4c414f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.636+0000 7fa4ca000640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4c41523f0 0x7fa4c414f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.636+0000 7fa4ca000640 1 -- 172.21.15.69:0/1159117792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c415a520 con 0x7fa4c415b980 2024-04-24T08:21:18.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.636+0000 7fa4ca000640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 0x7fa4c414fa70 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009eb0 tx=0x7fa4bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:18.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.636+0000 7fa4c1600640 1 -- 172.21.15.69:0/1159117792 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc03f070 con 0x7fa4c415b980 2024-04-24T08:21:18.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.637+0000 7fa4c1600640 1 -- 172.21.15.69:0/1159117792 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4bc030b60 con 0x7fa4c415b980 2024-04-24T08:21:18.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.637+0000 7fa4c1600640 1 -- 172.21.15.69:0/1159117792 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc044460 con 0x7fa4c415b980 2024-04-24T08:21:18.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.637+0000 7fa4caa00640 1 -- 172.21.15.69:0/1159117792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c4159c80 con 0x7fa4c415b980 2024-04-24T08:21:18.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.637+0000 7fa4caa00640 1 -- 172.21.15.69:0/1159117792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c415a240 con 0x7fa4c415b980 2024-04-24T08:21:18.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.638+0000 7fa4caa00640 1 -- 172.21.15.69:0/1159117792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c40aebc0 con 0x7fa4c415b980 2024-04-24T08:21:18.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.639+0000 7fa4c1600640 1 -- 172.21.15.69:0/1159117792 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa4bc03b410 con 0x7fa4c415b980 2024-04-24T08:21:18.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.640+0000 7fa4c1600640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4a406c8f0 0x7fa4a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:18.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.640+0000 7fa4c1600640 1 -- 172.21.15.69:0/1159117792 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fa4bc0ad460 con 0x7fa4c415b980 2024-04-24T08:21:18.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.640+0000 7fa4c9600640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4a406c8f0 0x7fa4a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:18.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.641+0000 7fa4c9600640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4a406c8f0 0x7fa4a406edb0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800a970 tx=0x7fa4b8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:18.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.644+0000 7fa4c1600640 1 -- 172.21.15.69:0/1159117792 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4c40aebc0 con 0x7fa4c415b980 2024-04-24T08:21:18.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.894+0000 7fa4caa00640 1 -- 172.21.15.69:0/1159117792 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4c4075d50 con 0x7fa4c415b980 2024-04-24T08:21:18.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.895+0000 7fa4c1600640 1 -- 172.21.15.69:0/1159117792 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5186 (secure 0 0 0) 0x7fa4bc07b0e0 con 0x7fa4c415b980 2024-04-24T08:21:18.897 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:18.897 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:21:17.653434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9825},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9825":{"gid":9825,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/99442307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":99442307},{"type":"v1","addr":"172.21.15.69:6835","nonce":99442307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9825,"qdb_cluster":[9825]},"id":8}]} 2024-04-24T08:21:18.897 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 112 2024-04-24T08:21:18.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.899+0000 7fa4caa00640 1 -- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4a406c8f0 msgr2=0x7fa4a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.899+0000 7fa4caa00640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4a406c8f0 0x7fa4a406edb0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa4b800a970 tx=0x7fa4b8008040 comp rx=0 tx=0).stop 2024-04-24T08:21:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.899+0000 7fa4caa00640 1 -- 172.21.15.69:0/1159117792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 msgr2=0x7fa4c414fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.899+0000 7fa4caa00640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 0x7fa4c414fa70 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009eb0 tx=0x7fa4bc004a40 comp rx=0 tx=0).stop 2024-04-24T08:21:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.900+0000 7fa4caa00640 1 -- 172.21.15.69:0/1159117792 shutdown_connections 2024-04-24T08:21:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.900+0000 7fa4caa00640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa4a406c8f0 0x7fa4a406edb0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.900+0000 7fa4caa00640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa4c415b980 0x7fa4c414fa70 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.900+0000 7fa4caa00640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4c41523f0 0x7fa4c414f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.900+0000 7fa4caa00640 1 --2- 172.21.15.69:0/1159117792 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa4c4072e20 0x7fa4c41595a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.900+0000 7fa4caa00640 1 -- 172.21.15.69:0/1159117792 >> 172.21.15.69:0/1159117792 conn(0x7fa4c40d0160 msgr2=0x7fa4c4073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.900+0000 7fa4caa00640 1 -- 172.21.15.69:0/1159117792 shutdown_connections 2024-04-24T08:21:18.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:18.900+0000 7fa4caa00640 1 -- 172.21.15.69:0/1159117792 wait complete. 2024-04-24T08:21:19.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 112, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T08:20:09.204175+0000', 'modified': '2024-04-24T08:21:17.653434+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 113, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9825}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9825': {'gid': 9825, 'name': 'c', 'rank': 0, 'incarnation': 109, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.69:6835/99442307', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.69:6834', 'nonce': 99442307}, {'type': 'v1', 'addr': '172.21.15.69:6835', 'nonce': 99442307}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 9825, 'qdb_cluster': [9825]} 2024-04-24T08:21:19.323 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T08:21:19.323 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-24T08:21:19.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.534+0000 7f0b6ec00640 1 Processor -- start 2024-04-24T08:21:19.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.535+0000 7f0b6ec00640 1 -- start start 2024-04-24T08:21:19.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.536+0000 7f0b6ec00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0b68145a90 0x7f0b6806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.536+0000 7f0b6ec00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f0b68145fe0 0x7f0b68160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.536+0000 7f0b6ec00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b680cf3d0 con 0x7f0b680d0300 2024-04-24T08:21:19.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.536+0000 7f0b6ec00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f0b680cf6b0 con 0x7f0b68145fe0 2024-04-24T08:21:19.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.536+0000 7f0b6ec00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f0b680cf990 con 0x7f0b68145a90 2024-04-24T08:21:19.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.536+0000 7f0b67e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f0b680d0300 0x7f0b680d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50734/0 (socket says 172.21.15.69:50734) 2024-04-24T08:21:19.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.536+0000 7f0b67e00640 1 -- 172.21.15.69:0/931273817 learned_addr learned my addr 172.21.15.69:0/931273817 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:19.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.536+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/931273817 >> v2:172.21.15.69:3300/0 conn(0x7f0b68145fe0 0x7f0b68160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.537+0000 7f0b67400640 1 -- 172.21.15.69:0/931273817 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 382571010 0 0) 0x7f0b680cf3d0 con 0x7f0b680d0300 2024-04-24T08:21:19.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.537+0000 7f0b5f400640 1 --2- 172.21.15.69:0/931273817 >> v2:172.21.15.121:3300/0 conn(0x7f0b68145a90 0x7f0b6806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.537+0000 7f0b67400640 1 -- 172.21.15.69:0/931273817 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b4c003520 con 0x7f0b680d0300 2024-04-24T08:21:19.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.537+0000 7f0b6cc00640 1 -- 172.21.15.69:0/931273817 >> v2:172.21.15.121:3300/0 conn(0x7f0b68145a90 msgr2=0x7f0b6806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.537+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/931273817 >> v2:172.21.15.121:3300/0 conn(0x7f0b68145a90 0x7f0b6806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.537+0000 7f0b6cc00640 1 -- 172.21.15.69:0/931273817 >> v1:172.21.15.69:6789/0 conn(0x7f0b680d0300 legacy=0x7f0b680d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.537+0000 7f0b6cc00640 1 -- 172.21.15.69:0/931273817 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b680cfc70 con 0x7f0b68145fe0 2024-04-24T08:21:19.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.537+0000 7f0b5f400640 1 --2- 172.21.15.69:0/931273817 >> v2:172.21.15.121:3300/0 conn(0x7f0b68145a90 0x7f0b6806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:19.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.538+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/931273817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 0x7f0b68160540 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f0b680b5780 tx=0x7f0b6002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcae672c4943afa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.538+0000 7f0b67400640 1 -- 172.21.15.69:0/931273817 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b60007740 con 0x7f0b68145fe0 2024-04-24T08:21:19.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.538+0000 7f0b67400640 1 -- 172.21.15.69:0/931273817 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b600078e0 con 0x7f0b68145fe0 2024-04-24T08:21:19.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.538+0000 7f0b67400640 1 -- 172.21.15.69:0/931273817 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b60007cb0 con 0x7f0b68145fe0 2024-04-24T08:21:19.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.538+0000 7f0b6ec00640 1 -- 172.21.15.69:0/931273817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 msgr2=0x7f0b68160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.538+0000 7f0b6ec00640 1 --2- 172.21.15.69:0/931273817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 0x7f0b68160540 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f0b680b5780 tx=0x7f0b6002a680 comp rx=0 tx=0).stop 2024-04-24T08:21:19.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.539+0000 7f0b6ec00640 1 -- 172.21.15.69:0/931273817 shutdown_connections 2024-04-24T08:21:19.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.539+0000 7f0b6ec00640 1 --2- 172.21.15.69:0/931273817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 0x7f0b68160540 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.539+0000 7f0b6ec00640 1 --2- 172.21.15.69:0/931273817 >> v2:172.21.15.121:3300/0 conn(0x7f0b68145a90 0x7f0b6806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.539+0000 7f0b6ec00640 1 -- 172.21.15.69:0/931273817 >> 172.21.15.69:0/931273817 conn(0x7f0b68067d90 msgr2=0x7f0b6806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.539+0000 7f0b6ec00640 1 -- 172.21.15.69:0/931273817 shutdown_connections 2024-04-24T08:21:19.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.539+0000 7f0b6ec00640 1 -- 172.21.15.69:0/931273817 wait complete. 2024-04-24T08:21:19.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.540+0000 7f0b6ec00640 1 Processor -- start 2024-04-24T08:21:19.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.541+0000 7f0b6ec00640 1 -- start start 2024-04-24T08:21:19.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.541+0000 7f0b6ec00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b680d0300 0x7f0b6815bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.541+0000 7f0b6ec00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b68145a90 0x7f0b681f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.541+0000 7f0b6ec00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 0x7f0b681f7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.541+0000 7f0b6ec00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0b681458f0 con 0x7f0b68145fe0 2024-04-24T08:21:19.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.541+0000 7f0b6ec00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0b68145770 con 0x7f0b680d0300 2024-04-24T08:21:19.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.541+0000 7f0b6ec00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0b680cdc30 con 0x7f0b68145a90 2024-04-24T08:21:19.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b6cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 0x7f0b681f7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b6cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 0x7f0b681f7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54750/0 (socket says 172.21.15.69:54750) 2024-04-24T08:21:19.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b6cc00640 1 -- 172.21.15.69:0/4273557302 learned_addr learned my addr 172.21.15.69:0/4273557302 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:19.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b67e00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b680d0300 0x7f0b6815bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b5f400640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b68145a90 0x7f0b681f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b6cc00640 1 -- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b68145a90 msgr2=0x7f0b681f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b68145a90 0x7f0b681f7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b6cc00640 1 -- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b680d0300 msgr2=0x7f0b6815bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b680d0300 0x7f0b6815bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b6cc00640 1 -- 172.21.15.69:0/4273557302 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b681414f0 con 0x7f0b68145fe0 2024-04-24T08:21:19.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b67e00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b680d0300 0x7f0b6815bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:19.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.542+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 0x7f0b681f7f80 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f0b6002ac70 tx=0x7f0b60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.543+0000 7f0b64c00640 1 -- 172.21.15.69:0/4273557302 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b60040070 con 0x7f0b68145fe0 2024-04-24T08:21:19.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.543+0000 7f0b64c00640 1 -- 172.21.15.69:0/4273557302 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b60031bc0 con 0x7f0b68145fe0 2024-04-24T08:21:19.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.543+0000 7f0b6ec00640 1 -- 172.21.15.69:0/4273557302 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b681f85b0 con 0x7f0b68145fe0 2024-04-24T08:21:19.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.543+0000 7f0b64c00640 1 -- 172.21.15.69:0/4273557302 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b60009450 con 0x7f0b68145fe0 2024-04-24T08:21:19.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.543+0000 7f0b6ec00640 1 -- 172.21.15.69:0/4273557302 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b6815da80 con 0x7f0b68145fe0 2024-04-24T08:21:19.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.545+0000 7f0b64c00640 1 -- 172.21.15.69:0/4273557302 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b60039040 con 0x7f0b68145fe0 2024-04-24T08:21:19.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.546+0000 7f0b64c00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b4406c8f0 0x7f0b4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.546+0000 7f0b64c00640 1 -- 172.21.15.69:0/4273557302 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f0b600ad540 con 0x7f0b68145fe0 2024-04-24T08:21:19.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.547+0000 7f0b67e00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b4406c8f0 0x7f0b4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.547+0000 7f0b71988740 1 Processor -- start 2024-04-24T08:21:19.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.547+0000 7f0b71988740 1 -- start start 2024-04-24T08:21:19.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b67e00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b4406c8f0 0x7f0b4406edb0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0b58005880 tx=0x7f0b58009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b71988740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b0d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b71988740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b3cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b71988740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3d190 0x557699b11fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b71988740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557699a77b10 con 0x557699b3d8d0 2024-04-24T08:21:19.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b71988740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557699a1e2b0 con 0x557699b3d190 2024-04-24T08:21:19.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b71988740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557699a73ad0 con 0x557699b5a7b0 2024-04-24T08:21:19.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b5f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b3cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b5f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b3cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54756/0 (socket says 172.21.15.69:54756) 2024-04-24T08:21:19.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b5f400640 1 -- 172.21.15.69:0/3159598758 learned_addr learned my addr 172.21.15.69:0/3159598758 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:19.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.548+0000 7f0b67e00640 1 --2- 172.21.15.69:0/3159598758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b0d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.549+0000 7f0b5f400640 1 -- 172.21.15.69:0/3159598758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 msgr2=0x557699b0d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.549+0000 7f0b5f400640 1 --2- 172.21.15.69:0/3159598758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b0d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.549+0000 7f0b5f400640 1 -- 172.21.15.69:0/3159598758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3d190 msgr2=0x557699b11fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.549+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/3159598758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3d190 0x557699b11fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.549+0000 7f0b5f400640 1 --2- 172.21.15.69:0/3159598758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3d190 0x557699b11fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.549+0000 7f0b5f400640 1 -- 172.21.15.69:0/3159598758 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55769978a2d0 con 0x557699b3d8d0 2024-04-24T08:21:19.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.549+0000 7f0b5f400640 1 --2- 172.21.15.69:0/3159598758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b3cc50 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000b8c0 tx=0x7f0b5000bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b633e2d638161d97 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.549+0000 7f0b57e00640 1 -- 172.21.15.69:0/3159598758 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50016070 con 0x557699b3d8d0 2024-04-24T08:21:19.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.549+0000 7f0b57e00640 1 -- 172.21.15.69:0/3159598758 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5000dbb0 con 0x557699b3d8d0 2024-04-24T08:21:19.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.549+0000 7f0b57e00640 1 -- 172.21.15.69:0/3159598758 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50012460 con 0x557699b3d8d0 2024-04-24T08:21:19.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.550+0000 7f0b71988740 1 -- 172.21.15.69:0/3159598758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 msgr2=0x557699b3cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.550+0000 7f0b71988740 1 --2- 172.21.15.69:0/3159598758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b3cc50 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000b8c0 tx=0x7f0b5000bd80 comp rx=0 tx=0).stop 2024-04-24T08:21:19.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.550+0000 7f0b71988740 1 -- 172.21.15.69:0/3159598758 shutdown_connections 2024-04-24T08:21:19.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.550+0000 7f0b71988740 1 --2- 172.21.15.69:0/3159598758 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b3cc50 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.550+0000 7f0b71988740 1 --2- 172.21.15.69:0/3159598758 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3d190 0x557699b11fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.550+0000 7f0b71988740 1 --2- 172.21.15.69:0/3159598758 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b0d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.550+0000 7f0b71988740 1 -- 172.21.15.69:0/3159598758 >> 172.21.15.69:0/3159598758 conn(0x557699b0f3b0 msgr2=0x557699b57c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.550+0000 7f0b71988740 1 -- 172.21.15.69:0/3159598758 shutdown_connections 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.550+0000 7f0b71988740 1 -- 172.21.15.69:0/3159598758 wait complete. 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.551+0000 7f0b71988740 1 -- start start 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.551+0000 7f0b71988740 1 Processor -- start 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.552+0000 7f0b71988740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b3d8d0 0x557699b38840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.552+0000 7f0b71988740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b5a7b0 0x557699b3ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.552+0000 7f0b71988740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b12a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.552+0000 7f0b71988740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557699b11560 con 0x557699b3e300 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.552+0000 7f0b71988740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557699a31590 con 0x557699b5a7b0 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.552+0000 7f0b71988740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557699b10950 con 0x557699b3d8d0 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.552+0000 7f0b71988740 10 client.0 _ll_register_callbacks cb 0x557699a143a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.552+0000 7f0b6cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b12a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.552+0000 7f0b6cc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b12a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54766/0 (socket says 172.21.15.69:54766) 2024-04-24T08:21:19.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.552+0000 7f0b6cc00640 1 -- 172.21.15.69:0/2649905218 learned_addr learned my addr 172.21.15.69:0/2649905218 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b67e00640 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b3d8d0 0x557699b38840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b5f400640 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b5a7b0 0x557699b3ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b6cc00640 1 -- 172.21.15.69:0/2649905218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b3d8d0 msgr2=0x557699b38840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b3d8d0 0x557699b38840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b6cc00640 1 -- 172.21.15.69:0/2649905218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b5a7b0 msgr2=0x557699b3ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b5a7b0 0x557699b3ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b6cc00640 1 -- 172.21.15.69:0/2649905218 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576998546b0 con 0x557699b3e300 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b12a40 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f0b6003f040 tx=0x7f0b6007b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3bca3ff7c75ebe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b54c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b54c00640 1 -- 172.21.15.69:0/2649905218 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b60093030 con 0x557699b3e300 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b54c00640 1 -- 172.21.15.69:0/2649905218 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f0b600ad0c0 con 0x557699b3e300 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.553+0000 7f0b71988740 1 -- 172.21.15.69:0/2649905218 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b5000dbb0 con 0x557699b3e300 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b54c00640 1 -- 172.21.15.69:0/2649905218 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f0b60083cc0 con 0x557699b3e300 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b71988740 10 client.9918 fetch_fsmap learned FSMap version 112 2024-04-24T08:21:19.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b71988740 1 -- 172.21.15.69:0/2649905218 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) -- 0x557699b4eed0 con 0x557699b3e300 2024-04-24T08:21:19.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b54c00640 1 -- 172.21.15.69:0/2649905218 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5000dbb0 con 0x557699b3e300 2024-04-24T08:21:19.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b54c00640 1 -- 172.21.15.69:0/2649905218 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 112) ==== 3494+0+0 (secure 0 0 0) 0x7f0b6004d030 con 0x557699b3e300 2024-04-24T08:21:19.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b71988740 10 client.9918 fetch_fsmap finished waiting for FSMap version 112 2024-04-24T08:21:19.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b71988740 10 client.9918 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:21:19.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b71988740 20 client.9918 populate_metadata read hostname 'smithi069' 2024-04-24T08:21:19.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b71988740 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.69:6834/99442307,v1:172.21.15.69:6835/99442307] conn(0x557699b40530 0x557699b68850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b71988740 4 client.9918 mds_command: new command op to 9825 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:19.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b71988740 1 -- 172.21.15.69:0/2649905218 --> [v2:172.21.15.69:6834/99442307,v1:172.21.15.69:6835/99442307] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557699a85530 con 0x557699b40530 2024-04-24T08:21:19.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.554+0000 7f0b67e00640 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.69:6834/99442307,v1:172.21.15.69:6835/99442307] conn(0x557699b40530 0x557699b68850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.555+0000 7f0b67e00640 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.69:6834/99442307,v1:172.21.15.69:6835/99442307] conn(0x557699b40530 0x557699b68850 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1ede0db88d646f server_cookie=7343372b36aaf38d in_seq=0 out_seq=0 2024-04-24T08:21:19.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.555+0000 7f0b54c00640 10 client.9918 ms_handle_connect on v2:172.21.15.69:6834/99442307 2024-04-24T08:21:19.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b54c00640 1 -- 172.21.15.69:0/2649905218 <== mds.0 v2:172.21.15.69:6834/99442307 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557699a85530 con 0x557699b40530 2024-04-24T08:21:19.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b54c00640 10 client.9918 handle_command_reply: tid=0 2024-04-24T08:21:19.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 client.9918 shutdown 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 -- 172.21.15.69:0/2649905218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 msgr2=0x557699b12a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b12a40 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f0b6003f040 tx=0x7f0b6007b8a0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 -- 172.21.15.69:0/2649905218 shutdown_connections 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.69:6834/99442307,v1:172.21.15.69:6835/99442307] conn(0x557699b40530 0x557699b68850 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b12a40 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b5a7b0 0x557699b3ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b54c00640 0 client.9918 ms_handle_reset on v2:172.21.15.69:6834/99442307 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 --2- 172.21.15.69:0/2649905218 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b3d8d0 0x557699b38840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 -- 172.21.15.69:0/2649905218 >> 172.21.15.69:0/2649905218 conn(0x557699b3d190 msgr2=0x557699b57c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 -- 172.21.15.69:0/2649905218 shutdown_connections 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 1 -- 172.21.15.69:0/2649905218 wait complete. 2024-04-24T08:21:19.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.556+0000 7f0b71988740 20 client.9918 trim_cache size 0 max 16384 2024-04-24T08:21:19.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.576+0000 7f0b71988740 1 Processor -- start 2024-04-24T08:21:19.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.577+0000 7f0b71988740 1 -- start start 2024-04-24T08:21:19.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.577+0000 7f0b71988740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3d8d0 0x557699b50c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.577+0000 7f0b71988740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b511c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.577+0000 7f0b71988740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b37780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.577+0000 7f0b71988740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0b681fff50 con 0x557699b3e300 2024-04-24T08:21:19.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.577+0000 7f0b71988740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0b68143570 con 0x557699b3d8d0 2024-04-24T08:21:19.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.577+0000 7f0b71988740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6815efc0 con 0x557699b5a7b0 2024-04-24T08:21:19.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.578+0000 7f0b5f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b511c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.578+0000 7f0b67e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3d8d0 0x557699b50c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.578+0000 7f0b5f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b511c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54776/0 (socket says 172.21.15.69:54776) 2024-04-24T08:21:19.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.578+0000 7f0b5f400640 1 -- 172.21.15.69:0/1645775197 learned_addr learned my addr 172.21.15.69:0/1645775197 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:19.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.578+0000 7f0b6cc00640 1 --2- 172.21.15.69:0/1645775197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b37780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.578+0000 7f0b5f400640 1 -- 172.21.15.69:0/1645775197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 msgr2=0x557699b37780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.578+0000 7f0b5f400640 1 --2- 172.21.15.69:0/1645775197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b37780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.578+0000 7f0b5f400640 1 -- 172.21.15.69:0/1645775197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3d8d0 msgr2=0x557699b50c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.578+0000 7f0b5f400640 1 --2- 172.21.15.69:0/1645775197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3d8d0 0x557699b50c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.578+0000 7f0b5f400640 1 -- 172.21.15.69:0/1645775197 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557699b3e130 con 0x557699b3e300 2024-04-24T08:21:19.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.579+0000 7f0b5f400640 1 --2- 172.21.15.69:0/1645775197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b511c0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000bdc0 tx=0x7f0b50012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9a26c0b2656abc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.579+0000 7f0b57400640 1 -- 172.21.15.69:0/1645775197 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b5002b070 con 0x557699b3e300 2024-04-24T08:21:19.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.579+0000 7f0b57400640 1 -- 172.21.15.69:0/1645775197 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5000b040 con 0x557699b3e300 2024-04-24T08:21:19.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.579+0000 7f0b57400640 1 -- 172.21.15.69:0/1645775197 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b50016040 con 0x557699b3e300 2024-04-24T08:21:19.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.580+0000 7f0b71988740 1 -- 172.21.15.69:0/1645775197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 msgr2=0x557699b511c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.580+0000 7f0b71988740 1 --2- 172.21.15.69:0/1645775197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b511c0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7f0b5000bdc0 tx=0x7f0b50012000 comp rx=0 tx=0).stop 2024-04-24T08:21:19.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.580+0000 7f0b71988740 1 -- 172.21.15.69:0/1645775197 shutdown_connections 2024-04-24T08:21:19.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.580+0000 7f0b71988740 1 --2- 172.21.15.69:0/1645775197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3e300 0x557699b511c0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.580+0000 7f0b71988740 1 --2- 172.21.15.69:0/1645775197 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b37780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.580+0000 7f0b71988740 1 --2- 172.21.15.69:0/1645775197 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3d8d0 0x557699b50c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.580+0000 7f0b71988740 1 -- 172.21.15.69:0/1645775197 >> 172.21.15.69:0/1645775197 conn(0x557699b3d190 msgr2=0x557699b58d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.580+0000 7f0b71988740 1 -- 172.21.15.69:0/1645775197 shutdown_connections 2024-04-24T08:21:19.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.580+0000 7f0b71988740 1 -- 172.21.15.69:0/1645775197 wait complete. 2024-04-24T08:21:19.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.581+0000 7f0b71988740 1 -- start start 2024-04-24T08:21:19.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.581+0000 7f0b71988740 1 Processor -- start 2024-04-24T08:21:19.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b71988740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b7aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b71988740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3e300 0x557699b50c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b71988740 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b51150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b71988740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557699b403b0 con 0x557699b3d8d0 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b71988740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x557699b56d50 con 0x557699b3e300 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b71988740 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x557699b40a00 con 0x557699b5a7b0 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b71988740 10 client.0 _ll_register_callbacks cb 0x7f0b60007840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b67e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b7aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b67e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b7aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54780/0 (socket says 172.21.15.69:54780) 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b67e00640 1 -- 172.21.15.69:0/3240245441 learned_addr learned my addr 172.21.15.69:0/3240245441 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b67e00640 1 -- 172.21.15.69:0/3240245441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 msgr2=0x557699b51150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.582+0000 7f0b67e00640 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b51150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b67e00640 1 -- 172.21.15.69:0/3240245441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3e300 msgr2=0x557699b50c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b67e00640 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3e300 0x557699b50c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b67e00640 1 -- 172.21.15.69:0/3240245441 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557699b7b5e0 con 0x557699b3d8d0 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b67e00640 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b7aa70 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f0b58005a90 tx=0x7f0b58010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d41c2ad1ba274e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b54c00640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b54c00640 1 -- 172.21.15.69:0/3240245441 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b58007550 con 0x557699b3d8d0 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b54c00640 1 -- 172.21.15.69:0/3240245441 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f0b580076f0 con 0x557699b3d8d0 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b71988740 1 -- 172.21.15.69:0/3240245441 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0b5000b040 con 0x557699b3d8d0 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b54c00640 1 -- 172.21.15.69:0/3240245441 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f0b5800d4f0 con 0x557699b3d8d0 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b71988740 10 client.9924 fetch_fsmap learned FSMap version 112 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.583+0000 7f0b71988740 1 -- 172.21.15.69:0/3240245441 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) -- 0x557699bded20 con 0x557699b3d8d0 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.584+0000 7f0b54c00640 1 -- 172.21.15.69:0/3240245441 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b5000b040 con 0x557699b3d8d0 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.584+0000 7f0b54c00640 1 -- 172.21.15.69:0/3240245441 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 112) ==== 3494+0+0 (secure 0 0 0) 0x7f0b58036050 con 0x557699b3d8d0 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.584+0000 7f0b71988740 10 client.9924 fetch_fsmap finished waiting for FSMap version 112 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.584+0000 7f0b71988740 10 client.9924 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-24T08:21:19.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.584+0000 7f0b71988740 20 client.9924 populate_metadata read hostname 'smithi069' 2024-04-24T08:21:19.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.584+0000 7f0b71988740 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.69:6834/99442307,v1:172.21.15.69:6835/99442307] conn(0x557699b39260 0x557699b39640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.584+0000 7f0b71988740 4 client.9924 mds_command: new command op to 9825 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:21:19.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.584+0000 7f0b71988740 1 -- 172.21.15.69:0/3240245441 --> [v2:172.21.15.69:6834/99442307,v1:172.21.15.69:6835/99442307] -- command(tid 0: {"prefix": "status"}) -- 0x557699a7f3a0 con 0x557699b39260 2024-04-24T08:21:19.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.584+0000 7f0b5f400640 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.69:6834/99442307,v1:172.21.15.69:6835/99442307] conn(0x557699b39260 0x557699b39640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b5f400640 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.69:6834/99442307,v1:172.21.15.69:6835/99442307] conn(0x557699b39260 0x557699b39640 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4a774aea920708f server_cookie=26067d1ecfe7709e in_seq=0 out_seq=0 2024-04-24T08:21:19.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b54c00640 10 client.9924 ms_handle_connect on v2:172.21.15.69:6834/99442307 2024-04-24T08:21:19.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b54c00640 1 -- 172.21.15.69:0/3240245441 <== mds.0 v2:172.21.15.69:6834/99442307 1 ==== command_reply(tid 0: 0 ) ==== 8+0+334 (crc 0 0 0) 0x557699a7f3a0 con 0x557699b39260 2024-04-24T08:21:19.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b54c00640 10 client.9924 handle_command_reply: tid=0 2024-04-24T08:21:19.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 client.9924 shutdown 2024-04-24T08:21:19.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 -- 172.21.15.69:0/3240245441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 msgr2=0x557699b7aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b7aa70 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f0b58005a90 tx=0x7f0b58010000 comp rx=0 tx=0).stop 2024-04-24T08:21:19.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 -- 172.21.15.69:0/3240245441 shutdown_connections 2024-04-24T08:21:19.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.69:6834/99442307,v1:172.21.15.69:6835/99442307] conn(0x557699b39260 0x557699b39640 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x557699b5a7b0 0x557699b51150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x557699b3e300 0x557699b50c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b54c00640 0 client.9924 ms_handle_reset on v2:172.21.15.69:6834/99442307 2024-04-24T08:21:19.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 --2- 172.21.15.69:0/3240245441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557699b3d8d0 0x557699b7aa70 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 -- 172.21.15.69:0/3240245441 >> 172.21.15.69:0/3240245441 conn(0x557699b3d190 msgr2=0x557699b0cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 -- 172.21.15.69:0/3240245441 shutdown_connections 2024-04-24T08:21:19.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 1 -- 172.21.15.69:0/3240245441 wait complete. 2024-04-24T08:21:19.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.585+0000 7f0b71988740 20 client.9924 trim_cache size 0 max 16384 2024-04-24T08:21:19.590 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:19.590 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "3422a353-af9e-4ed5-a4b2-7365b9c531c6", 2024-04-24T08:21:19.590 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-04-24T08:21:19.591 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 9825, 2024-04-24T08:21:19.591 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-04-24T08:21:19.591 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-04-24T08:21:19.591 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-04-24T08:21:19.591 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 4.9570292299999998, 2024-04-24T08:21:19.591 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 112, 2024-04-24T08:21:19.591 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 113, 2024-04-24T08:21:19.591 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 113, 2024-04-24T08:21:19.591 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 6.9851052730000003 2024-04-24T08:21:19.591 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:19.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 -- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b4406c8f0 msgr2=0x7f0b4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b4406c8f0 0x7f0b4406edb0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f0b58005880 tx=0x7f0b58009040 comp rx=0 tx=0).stop 2024-04-24T08:21:19.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 -- 172.21.15.69:0/4273557302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 msgr2=0x7f0b681f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 0x7f0b681f7f80 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f0b6002ac70 tx=0x7f0b60004a40 comp rx=0 tx=0).stop 2024-04-24T08:21:19.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 -- 172.21.15.69:0/4273557302 shutdown_connections 2024-04-24T08:21:19.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f0b4406c8f0 0x7f0b4406edb0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0b68145fe0 0x7f0b681f7f80 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0b68145a90 0x7f0b681f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 --2- 172.21.15.69:0/4273557302 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0b680d0300 0x7f0b6815bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 -- 172.21.15.69:0/4273557302 >> 172.21.15.69:0/4273557302 conn(0x7f0b68067d90 msgr2=0x7f0b6806f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 -- 172.21.15.69:0/4273557302 shutdown_connections 2024-04-24T08:21:19.593 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.586+0000 7f0b6ec00640 1 -- 172.21.15.69:0/4273557302 wait complete. 2024-04-24T08:21:19.606 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T08:21:19.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.793+0000 7fdab3200640 1 Processor -- start 2024-04-24T08:21:19.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.793+0000 7fdab3200640 1 -- start start 2024-04-24T08:21:19.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.794+0000 7fdab3200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdab414c650 0x7fdab4071c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.794+0000 7fdab3200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab4163fb0 con 0x7fdab406db90 2024-04-24T08:21:19.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.794+0000 7fdab3200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab406e130 con 0x7fdab41666e0 2024-04-24T08:21:19.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.794+0000 7fdab3200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fdab406e410 con 0x7fdab414c650 2024-04-24T08:21:19.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.795+0000 7fdab2800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdab406db90 0x7fdab40b3800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50748/0 (socket says 172.21.15.69:50748) 2024-04-24T08:21:19.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.795+0000 7fdab2800640 1 -- 172.21.15.69:0/2959459778 learned_addr learned my addr 172.21.15.69:0/2959459778 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:19.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.795+0000 7fdab1e00640 1 --2- 172.21.15.69:0/2959459778 >> v2:172.21.15.121:3300/0 conn(0x7fdab414c650 0x7fdab4071c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.795+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890187243 0 0) 0x7fdab4163fb0 con 0x7fdab406db90 2024-04-24T08:21:19.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.795+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda98003520 con 0x7fdab406db90 2024-04-24T08:21:19.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2222710665 0 0) 0x7fda98003520 con 0x7fdab406db90 2024-04-24T08:21:19.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 >> v2:172.21.15.121:3300/0 conn(0x7fdab414c650 msgr2=0x7fdab4071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 --2- 172.21.15.69:0/2959459778 >> v2:172.21.15.121:3300/0 conn(0x7fdab414c650 0x7fdab4071c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 >> v1:172.21.15.121:6789/0 conn(0x7fdab41666e0 legacy=0x7fdab40b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab406e6f0 con 0x7fdab406db90 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fdaac0019f0 con 0x7fdab406db90 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 >> v1:172.21.15.69:6789/0 conn(0x7fdab406db90 legacy=0x7fdab40b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 --2- 172.21.15.69:0/2959459778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda98003a00 0x7fda98003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 --2- 172.21.15.69:0/2959459778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda98004800 0x7fda98004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 --2- 172.21.15.69:0/2959459778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda98005210 0x7fda9800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdab406e410 con 0x7fda98005210 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fda9800c160 con 0x7fda98003a00 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fda9800c360 con 0x7fda98004800 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.796+0000 7fdab2800640 1 --2- 172.21.15.69:0/2959459778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda98005210 0x7fda9800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.797+0000 7fdab1400640 1 --2- 172.21.15.69:0/2959459778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda98004800 0x7fda98004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.797+0000 7fdab2800640 1 -- 172.21.15.69:0/2959459778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda98004800 msgr2=0x7fda98004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.797+0000 7fdab2800640 1 --2- 172.21.15.69:0/2959459778 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fda98004800 0x7fda98004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.797+0000 7fdab2800640 1 -- 172.21.15.69:0/2959459778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda98003a00 msgr2=0x7fda98003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:19.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.797+0000 7fdab2800640 1 --2- 172.21.15.69:0/2959459778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda98003a00 0x7fda98003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.797+0000 7fdab2800640 1 -- 172.21.15.69:0/2959459778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda98003520 con 0x7fda98005210 2024-04-24T08:21:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.798+0000 7fdab2800640 1 -- 172.21.15.69:0/2959459778 reap_dead start 2024-04-24T08:21:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.798+0000 7fdab2800640 1 --2- 172.21.15.69:0/2959459778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda98005210 0x7fda9800b9c0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fdaac00a740 tx=0x7fda98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f510d199f2223207 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.798+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac00a390 con 0x7fda98005210 2024-04-24T08:21:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.798+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaac02f070 con 0x7fda98005210 2024-04-24T08:21:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.798+0000 7fdab0a00640 1 -- 172.21.15.69:0/2959459778 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac02bb90 con 0x7fda98005210 2024-04-24T08:21:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.799+0000 7fdab3200640 1 -- 172.21.15.69:0/2959459778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda98005210 msgr2=0x7fda9800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.799+0000 7fdab3200640 1 --2- 172.21.15.69:0/2959459778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda98005210 0x7fda9800b9c0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7fdaac00a740 tx=0x7fda98005830 comp rx=0 tx=0).stop 2024-04-24T08:21:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.799+0000 7fdab3200640 1 -- 172.21.15.69:0/2959459778 shutdown_connections 2024-04-24T08:21:19.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.799+0000 7fdab3200640 1 --2- 172.21.15.69:0/2959459778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fda98005210 0x7fda9800b9c0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.799+0000 7fdab3200640 1 -- 172.21.15.69:0/2959459778 >> 172.21.15.69:0/2959459778 conn(0x7fdab40d0160 msgr2=0x7fdab40d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.799+0000 7fdab3200640 1 -- 172.21.15.69:0/2959459778 shutdown_connections 2024-04-24T08:21:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.799+0000 7fdab3200640 1 -- 172.21.15.69:0/2959459778 wait complete. 2024-04-24T08:21:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.801+0000 7fdab3200640 1 Processor -- start 2024-04-24T08:21:19.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.801+0000 7fdab3200640 1 -- start start 2024-04-24T08:21:19.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.802+0000 7fdab3200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda98005210 0x7fdab40cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.802+0000 7fdab3200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdab414b6b0 0x7fdab40cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.802+0000 7fdab3200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdab40ce3d0 0x7fdab40b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.802+0000 7fdab3200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdab414ca30 con 0x7fdab414b6b0 2024-04-24T08:21:19.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.802+0000 7fdab3200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdab406ecb0 con 0x7fda98005210 2024-04-24T08:21:19.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.802+0000 7fdab3200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdab4072830 con 0x7fdab40ce3d0 2024-04-24T08:21:19.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.802+0000 7fdab1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdab414b6b0 0x7fdab40cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.802+0000 7fdab1400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdab414b6b0 0x7fdab40cde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54802/0 (socket says 172.21.15.69:54802) 2024-04-24T08:21:19.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.802+0000 7fdab1400640 1 -- 172.21.15.69:0/1219171283 learned_addr learned my addr 172.21.15.69:0/1219171283 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:19.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdab2800640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdab40ce3d0 0x7fdab40b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdab1e00640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda98005210 0x7fdab40cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdab1400640 1 -- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdab40ce3d0 msgr2=0x7fdab40b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdab1400640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdab40ce3d0 0x7fdab40b3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdab1400640 1 -- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda98005210 msgr2=0x7fdab40cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:19.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdab1400640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda98005210 0x7fdab40cd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:19.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdab1400640 1 -- 172.21.15.69:0/1219171283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab40ce8e0 con 0x7fdab414b6b0 2024-04-24T08:21:19.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdab1400640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdab414b6b0 0x7fdab40cde90 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000e4b0 tx=0x7fdaa000e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdaaa000640 1 -- 172.21.15.69:0/1219171283 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa00052e0 con 0x7fdab414b6b0 2024-04-24T08:21:19.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdaaa000640 1 -- 172.21.15.69:0/1219171283 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa0018070 con 0x7fdab414b6b0 2024-04-24T08:21:19.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdab3200640 1 -- 172.21.15.69:0/1219171283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab40ccae0 con 0x7fdab414b6b0 2024-04-24T08:21:19.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdaaa000640 1 -- 172.21.15.69:0/1219171283 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0003810 con 0x7fdab414b6b0 2024-04-24T08:21:19.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.803+0000 7fdab3200640 1 -- 172.21.15.69:0/1219171283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab40cd0a0 con 0x7fdab414b6b0 2024-04-24T08:21:19.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.805+0000 7fdab3200640 1 -- 172.21.15.69:0/1219171283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab4075d50 con 0x7fdab414b6b0 2024-04-24T08:21:19.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.806+0000 7fdaaa000640 1 -- 172.21.15.69:0/1219171283 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdaa00039b0 con 0x7fdab414b6b0 2024-04-24T08:21:19.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.808+0000 7fdaaa000640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda9406c8f0 0x7fda9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:19.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.808+0000 7fdaaa000640 1 -- 172.21.15.69:0/1219171283 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fdaa008b960 con 0x7fdab414b6b0 2024-04-24T08:21:19.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.808+0000 7fdab1e00640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda9406c8f0 0x7fda9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:19.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.809+0000 7fdab1e00640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda9406c8f0 0x7fda9406edb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009970 tx=0x7fdaa4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:19.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:19.812+0000 7fdaaa000640 1 -- 172.21.15.69:0/1219171283 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdaa0059cc0 con 0x7fdab414b6b0 2024-04-24T08:21:20.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.051+0000 7fdab3200640 1 -- 172.21.15.69:0/1219171283 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdab40d0d90 con 0x7fdab414b6b0 2024-04-24T08:21:20.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.052+0000 7fdaaa000640 1 -- 172.21.15.69:0/1219171283 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v112) ==== 53+0+73 (secure 0 0 0) 0x7fdaa00595c0 con 0x7fdab414b6b0 2024-04-24T08:21:20.054 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:21:20.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 -- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda9406c8f0 msgr2=0x7fda9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda9406c8f0 0x7fda9406edb0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009970 tx=0x7fdaa4008040 comp rx=0 tx=0).stop 2024-04-24T08:21:20.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 -- 172.21.15.69:0/1219171283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdab414b6b0 msgr2=0x7fdab40cde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:20.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdab414b6b0 0x7fdab40cde90 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000e4b0 tx=0x7fdaa000e970 comp rx=0 tx=0).stop 2024-04-24T08:21:20.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 -- 172.21.15.69:0/1219171283 shutdown_connections 2024-04-24T08:21:20.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fda9406c8f0 0x7fda9406edb0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdab414b6b0 0x7fdab40cde90 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdab40ce3d0 0x7fdab40b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 --2- 172.21.15.69:0/1219171283 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda98005210 0x7fdab40cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:20.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 -- 172.21.15.69:0/1219171283 >> 172.21.15.69:0/1219171283 conn(0x7fdab40d0160 msgr2=0x7fdab4166bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:20.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 -- 172.21.15.69:0/1219171283 shutdown_connections 2024-04-24T08:21:20.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20.055+0000 7fdab3200640 1 -- 172.21.15.69:0/1219171283 wait complete. 2024-04-24T08:21:20.076 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:21:20.076 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:21:20.076 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-04-24T08:21:20.076 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:21:20.076 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:20.076 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:21:20.076 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:21:20.076 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:21:20.076 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:21:20.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:20.076 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-04-24T08:21:20.125 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:21:20.125 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:21:20.125 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:21:20.125 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:21:20.125 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-04-24T08:21:20.125 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:21:20.125 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 41900sec preferred_lft 41900sec 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-04-24T08:21:20.126 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:21:20.127 INFO:teuthology.orchestra.run.smithi069.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T08:21:20.127 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:21:20.127 INFO:teuthology.orchestra.run.smithi069.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T08:21:20.127 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:21:20.127 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::a8be:e1ff:fe73:6e8a/64 scope link 2024-04-24T08:21:20.127 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:21:20.127 INFO:teuthology.orchestra.run.smithi069.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:21:20.128 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 9a:80:27:7c:69:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:21:20.128 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::9880:27ff:fe7c:6933/64 scope link 2024-04-24T08:21:20.128 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:21:20.128 INFO:teuthology.orchestra.run.smithi069.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T08:21:20.128 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether b2:53:90:45:57:2c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:21:20.128 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::b053:90ff:fe45:572c/64 scope link 2024-04-24T08:21:20.128 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:21:20.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:20.130 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:21:20.210 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:21:20.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:20.215 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-04-24T08:21:20.267 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-04-24T08:21:20.267 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:21:20.267 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T08:21:20.268 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T08:21:20.268 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:21:20.268 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:21:20.269 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-04-24T08:21:20.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T08:21:20.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T08:21:20.269 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T08:21:20.269 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-04-24T08:21:20.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:21:20.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:20.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:20.480 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:21:20.524 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:21:20.525 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:21:20.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:20.526 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-04-24T08:21:20.655 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T08:21:20.655 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:21:20.655 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:20.711 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:21:20.711 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:21:20.711 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:20.759 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:21:20.838 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:21:20.895 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:21:20.896 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:21:20.897 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:21:20.897 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:21:20.897 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:21:20.897 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:21:20.897 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:21:20.897 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:21:20.897 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:21:20.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:20.898 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:21:20.950 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T08:21:20.950 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T08:21:20.957 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-04-24T08:21:21.029 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T08:21:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-04-24T08:21:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-04-24T08:21:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-04-24T08:21:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-04-24T08:21:21.073 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-04-24T08:21:21.074 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:21:21.075 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:21:21.076 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:21:21.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:21.077 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-04-24T08:21:21.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:2024-04-24T08:21:21.134+0000 7f2f9d11c480 -1 init, newargv = 0x561e3b8535d0 newargc=15 2024-04-24T08:21:21.135 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[79676]: starting ceph client 2024-04-24T08:21:21.143 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[79676]: starting fuse 2024-04-24T08:21:21.161 INFO:teuthology.orchestra.run.smithi069.stdout:40 2024-04-24T08:21:21.162 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-04-24T08:21:21.162 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-04-24T08:21:21.162 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-04-24T08:21:21.162 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-04-24T08:21:21.162 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:21:21.162 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-04-24T08:21:21.162 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:21.162 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-04-24T08:21:21.162 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T08:21:21.162 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:21.163 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.0")) 2024-04-24T08:21:21.164 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:21:21.346 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.0.79676.asok 2024-04-24T08:21:21.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:21.363 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.79676.asok 2024-04-24T08:21:21.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:21.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.79676.asok status 2024-04-24T08:21:21.591 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:21.591 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-04-24T08:21:21.591 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "1f81fb397ae98da7563d451a78c61574c8f4e6e0", 2024-04-24T08:21:21.591 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T08:21:21.591 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "0", 2024-04-24T08:21:21.591 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "79676", 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 9936, 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 9936 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:21:21.592 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 47672667 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 47672667 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.9936 192.168.0.1:0/47672667", 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/47672667", 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 112, 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 113, 2024-04-24T08:21:21.593 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-04-24T08:21:21.594 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-04-24T08:21:21.594 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-04-24T08:21:21.594 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:21.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:21.605 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-04-24T08:21:21.606 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:21.659 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-04-24T08:21:21.660 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:21.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:21.660 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:21.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:21.744 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-04-24T08:21:21.826 INFO:teuthology.orchestra.run.smithi069.stdout:./ 2024-04-24T08:21:21.826 INFO:teuthology.orchestra.run.smithi069.stdout:./parent 2024-04-24T08:21:21.826 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed 2024-04-24T08:21:21.826 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed/charlie 2024-04-24T08:21:21.826 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed/alpha 2024-04-24T08:21:21.826 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/flushed/bravo 2024-04-24T08:21:21.827 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/unflushed 2024-04-24T08:21:21.827 INFO:teuthology.orchestra.run.smithi069.stdout:./parent/unflushed/jfile 2024-04-24T08:21:21.828 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:21:21.828 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:21:21.828 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:21:21.828 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:21.828 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-04-24T08:21:21.828 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:21:21.955 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-04-24T08:21:21.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:21.968 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:21:21.968 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:21:21.968 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:21:21.968 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:21.969 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-04-24T08:21:21.969 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:21:22.149 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627776 2024-04-24T08:21:22.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:22.163 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:21:22.163 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:21:22.163 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:21:22.163 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:22.163 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-04-24T08:21:22.163 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:21:22.345 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627777 2024-04-24T08:21:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:22.358 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:21:22.358 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:21:22.358 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:21:22.358 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:22.359 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-04-24T08:21:22.359 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:21:22.543 INFO:teuthology.orchestra.run.smithi069.stdout:1099511628280 2024-04-24T08:21:22.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:22.557 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:21:22.557 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:21:22.557 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:21:22.557 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:22.557 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-04-24T08:21:22.557 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:21:22.745 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627778 2024-04-24T08:21:22.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:22.758 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:21:22.759 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:21:22.759 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:21:22.759 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:22.759 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-04-24T08:21:22.759 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:21:22.943 INFO:teuthology.orchestra.run.smithi069.stdout:1099511628279 2024-04-24T08:21:22.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:22.957 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:21:22.957 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:21:22.957 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:21:22.957 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:22.957 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-04-24T08:21:22.957 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:21:23.144 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627779 2024-04-24T08:21:23.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:23.158 DEBUG:teuthology.orchestra.run.smithi069:> stdin-killer -- python3 -c ' 2024-04-24T08:21:23.158 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-04-24T08:21:23.158 DEBUG:teuthology.orchestra.run.smithi069:> import stat 2024-04-24T08:21:23.158 DEBUG:teuthology.orchestra.run.smithi069:> 2024-04-24T08:21:23.158 DEBUG:teuthology.orchestra.run.smithi069:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-04-24T08:21:23.158 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-04-24T08:21:23.342 INFO:teuthology.orchestra.run.smithi069.stdout:1099511627780 2024-04-24T08:21:23.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:21:23.356 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T08:21:23.356 DEBUG:teuthology.orchestra.run.smithi069:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:21:23.512 DEBUG:teuthology.orchestra.run.smithi121:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T08:21:23.615 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T08:21:23.615 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:21:23.615 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:21:23.615 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:21:23.669 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:23.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.871+0000 7effde200640 1 Processor -- start 2024-04-24T08:21:23.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.872+0000 7effde200640 1 -- start start 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.873+0000 7effde200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7effd815b490 0x7effd80b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.873+0000 7effde200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 0x7effd806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.873+0000 7effde200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effd80b46e0 con 0x7effd80cfbe0 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.873+0000 7effde200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7effd806fc70 con 0x7effd815b490 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.873+0000 7effde200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7effd80d0960 con 0x7effd806f2d0 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.874+0000 7effd7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7effd815b490 0x7effd80b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.874+0000 7effd7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7effd815b490 0x7effd80b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54810/0 (socket says 172.21.15.69:54810) 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.874+0000 7effd7400640 1 -- 172.21.15.69:0/3610826512 learned_addr learned my addr 172.21.15.69:0/3610826512 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.874+0000 7effcea00640 1 --2- 172.21.15.69:0/3610826512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 0x7effd806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.874+0000 7effd7400640 1 -- 172.21.15.69:0/3610826512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 msgr2=0x7effd806f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.874+0000 7effd7400640 1 --2- 172.21.15.69:0/3610826512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 0x7effd806f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:23.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.874+0000 7effd7400640 1 -- 172.21.15.69:0/3610826512 >> v1:172.21.15.121:6789/0 conn(0x7effd80cfbe0 legacy=0x7effd80d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:23.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.874+0000 7effd7400640 1 -- 172.21.15.69:0/3610826512 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd80b4980 con 0x7effd815b490 2024-04-24T08:21:23.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.875+0000 7effd7400640 1 --2- 172.21.15.69:0/3610826512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 0x7effd80b5180 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7effc8009e50 tx=0x7effc802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d980066be24e8ecf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:23.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.875+0000 7effd6a00640 1 -- 172.21.15.69:0/3610826512 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc8036070 con 0x7effd815b490 2024-04-24T08:21:23.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.876+0000 7effd6a00640 1 -- 172.21.15.69:0/3610826512 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc8031600 con 0x7effd815b490 2024-04-24T08:21:23.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.876+0000 7effd6a00640 1 -- 172.21.15.69:0/3610826512 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc80319d0 con 0x7effd815b490 2024-04-24T08:21:23.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.876+0000 7effde200640 1 -- 172.21.15.69:0/3610826512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 msgr2=0x7effd80b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:23.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.876+0000 7effde200640 1 --2- 172.21.15.69:0/3610826512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 0x7effd80b5180 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7effc8009e50 tx=0x7effc802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:21:23.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.876+0000 7effde200640 1 -- 172.21.15.69:0/3610826512 shutdown_connections 2024-04-24T08:21:23.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.876+0000 7effde200640 1 --2- 172.21.15.69:0/3610826512 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 0x7effd806f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:23.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.876+0000 7effde200640 1 --2- 172.21.15.69:0/3610826512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 0x7effd80b5180 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:23.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.876+0000 7effde200640 1 -- 172.21.15.69:0/3610826512 >> 172.21.15.69:0/3610826512 conn(0x7effd8067d90 msgr2=0x7effd806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:23.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.876+0000 7effde200640 1 -- 172.21.15.69:0/3610826512 shutdown_connections 2024-04-24T08:21:23.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.876+0000 7effde200640 1 -- 172.21.15.69:0/3610826512 wait complete. 2024-04-24T08:21:23.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.878+0000 7effde200640 1 Processor -- start 2024-04-24T08:21:23.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.879+0000 7effde200640 1 -- start start 2024-04-24T08:21:23.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.880+0000 7effde200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 0x7effd81f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:23.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.880+0000 7effde200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd80cfbe0 0x7effd81595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:23.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.880+0000 7effde200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 0x7effd8159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:23.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.880+0000 7effde200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7effd815b870 con 0x7effd815b490 2024-04-24T08:21:23.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.880+0000 7effde200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7effd80d0c40 con 0x7effd80cfbe0 2024-04-24T08:21:23.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.880+0000 7effde200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7effd80b56c0 con 0x7effd806f2d0 2024-04-24T08:21:23.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.880+0000 7effd7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 0x7effd8159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:23.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.880+0000 7effd7e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 0x7effd8159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54824/0 (socket says 172.21.15.69:54824) 2024-04-24T08:21:23.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.880+0000 7effd7e00640 1 -- 172.21.15.69:0/2078134377 learned_addr learned my addr 172.21.15.69:0/2078134377 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:23.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.880+0000 7effd7400640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 0x7effd81f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:23.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effd7e00640 1 -- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 msgr2=0x7effd81f3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:23.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effd7e00640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 0x7effd81f3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:23.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effd7e00640 1 -- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd80cfbe0 msgr2=0x7effd81595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:23.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effcea00640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd80cfbe0 0x7effd81595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:23.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effd7e00640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd80cfbe0 0x7effd81595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:23.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effd7e00640 1 -- 172.21.15.69:0/2078134377 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd81469b0 con 0x7effd815b490 2024-04-24T08:21:23.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effd7400640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 0x7effd81f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:23.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effd7e00640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 0x7effd8159b20 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7effd000c410 tx=0x7effd000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:23.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effcfe00640 1 -- 172.21.15.69:0/2078134377 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0011790 con 0x7effd815b490 2024-04-24T08:21:23.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effcfe00640 1 -- 172.21.15.69:0/2078134377 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effd0017070 con 0x7effd815b490 2024-04-24T08:21:23.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effde200640 1 -- 172.21.15.69:0/2078134377 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd8146c90 con 0x7effd815b490 2024-04-24T08:21:23.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effcfe00640 1 -- 172.21.15.69:0/2078134377 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0011bb0 con 0x7effd815b490 2024-04-24T08:21:23.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.881+0000 7effde200640 1 -- 172.21.15.69:0/2078134377 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd8147250 con 0x7effd815b490 2024-04-24T08:21:23.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.883+0000 7effde200640 1 -- 172.21.15.69:0/2078134377 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effd8147450 con 0x7effd815b490 2024-04-24T08:21:23.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.883+0000 7effcfe00640 1 -- 172.21.15.69:0/2078134377 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7effd001c460 con 0x7effd815b490 2024-04-24T08:21:23.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.884+0000 7effcfe00640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effb406c8f0 0x7effb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:23.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.884+0000 7effcfe00640 1 -- 172.21.15.69:0/2078134377 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7effd008b880 con 0x7effd815b490 2024-04-24T08:21:23.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.885+0000 7effd7400640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effb406c8f0 0x7effb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:23.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.886+0000 7effd7400640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effb406c8f0 0x7effb406edb0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7effc8033000 tx=0x7effc8009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:23.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:23.889+0000 7effcfe00640 1 -- 172.21.15.69:0/2078134377 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7effd8147450 con 0x7effd815b490 2024-04-24T08:21:24.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.092+0000 7effde200640 1 -- 172.21.15.69:0/2078134377 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7effd8068a80 con 0x7effd815b490 2024-04-24T08:21:24.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.095+0000 7effcfe00640 1 -- 172.21.15.69:0/2078134377 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+17017 (secure 0 0 0) 0x7effd0021020 con 0x7effd815b490 2024-04-24T08:21:24.097 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:24.097 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":113,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:21:12.532402+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.37","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/2194021008":"2024-04-25T08:21:12.532354+0000","172.21.15.69:6834/2194021008":"2024-04-25T08:21:12.532354+0000","172.21.15.121:6836/1515102940":"2024-04-25T08:20:50.436176+0000","172.21.15.121:6834/1515102940":"2024-04-25T08:20:50.436176+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:24.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.097+0000 7effde200640 1 -- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effb406c8f0 msgr2=0x7effb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.097+0000 7effde200640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effb406c8f0 0x7effb406edb0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7effc8033000 tx=0x7effc8009000 comp rx=0 tx=0).stop 2024-04-24T08:21:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.097+0000 7effde200640 1 -- 172.21.15.69:0/2078134377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 msgr2=0x7effd8159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.097+0000 7effde200640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 0x7effd8159b20 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7effd000c410 tx=0x7effd000c8d0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.098+0000 7effde200640 1 -- 172.21.15.69:0/2078134377 shutdown_connections 2024-04-24T08:21:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.098+0000 7effde200640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7effb406c8f0 0x7effb406edb0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.098+0000 7effde200640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effd815b490 0x7effd8159b20 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.098+0000 7effde200640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7effd80cfbe0 0x7effd81595e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.098+0000 7effde200640 1 --2- 172.21.15.69:0/2078134377 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7effd806f2d0 0x7effd81f3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.098+0000 7effde200640 1 -- 172.21.15.69:0/2078134377 >> 172.21.15.69:0/2078134377 conn(0x7effd8067d90 msgr2=0x7effd80b0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.098+0000 7effde200640 1 -- 172.21.15.69:0/2078134377 shutdown_connections 2024-04-24T08:21:24.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.098+0000 7effde200640 1 -- 172.21.15.69:0/2078134377 wait complete. 2024-04-24T08:21:24.118 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:20:06.963817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:20:08.190440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:21:24.118 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:21:24.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.322+0000 7fa8c6000640 1 Processor -- start 2024-04-24T08:21:24.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.322+0000 7fa8c6000640 1 -- start start 2024-04-24T08:21:24.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.324+0000 7fa8c6000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa8c0071d70 0x7fa8c015ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.324+0000 7fa8c6000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c0160920 con 0x7fa8c01420a0 2024-04-24T08:21:24.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.324+0000 7fa8c6000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c0072450 con 0x7fa8c0160510 2024-04-24T08:21:24.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.324+0000 7fa8c6000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa8c00cf1e0 con 0x7fa8c0071d70 2024-04-24T08:21:24.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.325+0000 7fa8c4c00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fa8c01420a0 0x7fa8c0142480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50760/0 (socket says 172.21.15.69:50760) 2024-04-24T08:21:24.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.325+0000 7fa8c4c00640 1 -- 172.21.15.69:0/2928868414 learned_addr learned my addr 172.21.15.69:0/2928868414 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:24.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.325+0000 7fa8b7e00640 1 --2- 172.21.15.69:0/2928868414 >> v2:172.21.15.69:3300/0 conn(0x7fa8c0071d70 0x7fa8c015ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.325+0000 7fa8bfe00640 1 -- 172.21.15.69:0/2928868414 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753479157 0 0) 0x7fa8b0001110 con 0x7fa8c01420a0 2024-04-24T08:21:24.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.325+0000 7fa8bfe00640 1 -- 172.21.15.69:0/2928868414 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8a4003520 con 0x7fa8c01420a0 2024-04-24T08:21:24.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.325+0000 7fa8b7e00640 1 -- 172.21.15.69:0/2928868414 >> v1:172.21.15.121:6789/0 conn(0x7fa8c0160510 legacy=0x7fa8c00cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.325+0000 7fa8b7e00640 1 -- 172.21.15.69:0/2928868414 >> v1:172.21.15.69:6789/0 conn(0x7fa8c01420a0 legacy=0x7fa8c0142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.325+0000 7fa8b7e00640 1 -- 172.21.15.69:0/2928868414 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c0072730 con 0x7fa8c0071d70 2024-04-24T08:21:24.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.326+0000 7fa8b7e00640 1 --2- 172.21.15.69:0/2928868414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0071d70 0x7fa8c015ffd0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80097a0 tx=0x7fa8a802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4313b5cdf9bc42f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:24.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.327+0000 7fa8bfe00640 1 -- 172.21.15.69:0/2928868414 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8037070 con 0x7fa8c0071d70 2024-04-24T08:21:24.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.327+0000 7fa8bfe00640 1 -- 172.21.15.69:0/2928868414 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8a8032440 con 0x7fa8c0071d70 2024-04-24T08:21:24.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.327+0000 7fa8bfe00640 1 -- 172.21.15.69:0/2928868414 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8032810 con 0x7fa8c0071d70 2024-04-24T08:21:24.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.327+0000 7fa8c6000640 1 -- 172.21.15.69:0/2928868414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0071d70 msgr2=0x7fa8c015ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.327+0000 7fa8c6000640 1 --2- 172.21.15.69:0/2928868414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0071d70 0x7fa8c015ffd0 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80097a0 tx=0x7fa8a802a230 comp rx=0 tx=0).stop 2024-04-24T08:21:24.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.327+0000 7fa8c6000640 1 -- 172.21.15.69:0/2928868414 shutdown_connections 2024-04-24T08:21:24.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.327+0000 7fa8c6000640 1 --2- 172.21.15.69:0/2928868414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0071d70 0x7fa8c015ffd0 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.327+0000 7fa8c6000640 1 -- 172.21.15.69:0/2928868414 >> 172.21.15.69:0/2928868414 conn(0x7fa8c0067b80 msgr2=0x7fa8c006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:24.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.327+0000 7fa8c6000640 1 -- 172.21.15.69:0/2928868414 shutdown_connections 2024-04-24T08:21:24.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.327+0000 7fa8c6000640 1 -- 172.21.15.69:0/2928868414 wait complete. 2024-04-24T08:21:24.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.329+0000 7fa8c6000640 1 Processor -- start 2024-04-24T08:21:24.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.329+0000 7fa8c6000640 1 -- start start 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.330+0000 7fa8c6000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c0071d70 0x7fa8c01529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.330+0000 7fa8c6000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8c01420a0 0x7fa8c00b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.330+0000 7fa8c6000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0160510 0x7fa8c00b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.330+0000 7fa8c6000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c0072cf0 con 0x7fa8c0160510 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.330+0000 7fa8c6000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c0072200 con 0x7fa8c0071d70 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.330+0000 7fa8c6000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c00cfd80 con 0x7fa8c01420a0 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.331+0000 7fa8c5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0160510 0x7fa8c00b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.331+0000 7fa8c5600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0160510 0x7fa8c00b4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54840/0 (socket says 172.21.15.69:54840) 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.331+0000 7fa8c5600640 1 -- 172.21.15.69:0/2825374286 learned_addr learned my addr 172.21.15.69:0/2825374286 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.331+0000 7fa8b7e00640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8c01420a0 0x7fa8c00b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.331+0000 7fa8c4c00640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c0071d70 0x7fa8c01529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.331+0000 7fa8c5600640 1 -- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8c01420a0 msgr2=0x7fa8c00b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.331+0000 7fa8c5600640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8c01420a0 0x7fa8c00b4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.331+0000 7fa8c5600640 1 -- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c0071d70 msgr2=0x7fa8c01529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.331+0000 7fa8c5600640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c0071d70 0x7fa8c01529f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.331+0000 7fa8c5600640 1 -- 172.21.15.69:0/2825374286 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c00aeeb0 con 0x7fa8c0160510 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.332+0000 7fa8c5600640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0160510 0x7fa8c00b4a50 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800c6b0 tx=0x7fa8b800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.332+0000 7fa8bd600640 1 -- 172.21.15.69:0/2825374286 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8003b70 con 0x7fa8c0160510 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.332+0000 7fa8bd600640 1 -- 172.21.15.69:0/2825374286 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b8003d30 con 0x7fa8c0160510 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.332+0000 7fa8c6000640 1 -- 172.21.15.69:0/2825374286 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c00af190 con 0x7fa8c0160510 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.332+0000 7fa8bd600640 1 -- 172.21.15.69:0/2825374286 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8005310 con 0x7fa8c0160510 2024-04-24T08:21:24.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.332+0000 7fa8c6000640 1 -- 172.21.15.69:0/2825374286 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c00af470 con 0x7fa8c0160510 2024-04-24T08:21:24.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.333+0000 7fa8c6000640 1 -- 172.21.15.69:0/2825374286 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c0075d50 con 0x7fa8c0160510 2024-04-24T08:21:24.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.334+0000 7fa8bd600640 1 -- 172.21.15.69:0/2825374286 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa8b80054b0 con 0x7fa8c0160510 2024-04-24T08:21:24.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.335+0000 7fa8bd600640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa89c06c8f0 0x7fa89c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.335+0000 7fa8bd600640 1 -- 172.21.15.69:0/2825374286 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fa8b808bb40 con 0x7fa8c0160510 2024-04-24T08:21:24.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.336+0000 7fa8c4c00640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa89c06c8f0 0x7fa89c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.337+0000 7fa8c4c00640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa89c06c8f0 0x7fa89c06edb0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00012b0 tx=0x7fa8b0009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:24.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.340+0000 7fa8bd600640 1 -- 172.21.15.69:0/2825374286 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa8b8059de0 con 0x7fa8c0160510 2024-04-24T08:21:24.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.539+0000 7fa8c6000640 1 -- 172.21.15.69:0/2825374286 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa8c006c470 con 0x7fa8c0160510 2024-04-24T08:21:24.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.539+0000 7fa8bd600640 1 -- 172.21.15.69:0/2825374286 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7fa8b80596e0 con 0x7fa8c0160510 2024-04-24T08:21:24.541 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:21:24.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 -- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa89c06c8f0 msgr2=0x7fa89c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa89c06c8f0 0x7fa89c06edb0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00012b0 tx=0x7fa8b0009040 comp rx=0 tx=0).stop 2024-04-24T08:21:24.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 -- 172.21.15.69:0/2825374286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0160510 msgr2=0x7fa8c00b4a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0160510 0x7fa8c00b4a50 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800c6b0 tx=0x7fa8b800cb70 comp rx=0 tx=0).stop 2024-04-24T08:21:24.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 -- 172.21.15.69:0/2825374286 shutdown_connections 2024-04-24T08:21:24.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa89c06c8f0 0x7fa89c06edb0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8c0160510 0x7fa8c00b4a50 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa8c01420a0 0x7fa8c00b4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 --2- 172.21.15.69:0/2825374286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c0071d70 0x7fa8c01529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 -- 172.21.15.69:0/2825374286 >> 172.21.15.69:0/2825374286 conn(0x7fa8c0067b80 msgr2=0x7fa8c00b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:24.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 -- 172.21.15.69:0/2825374286 shutdown_connections 2024-04-24T08:21:24.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.542+0000 7fa8c6000640 1 -- 172.21.15.69:0/2825374286 wait complete. 2024-04-24T08:21:24.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:21:24.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.770+0000 7f4737600640 1 Processor -- start 2024-04-24T08:21:24.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.770+0000 7f4737600640 1 -- start start 2024-04-24T08:21:24.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.772+0000 7f4737600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f47300d2350 0x7f47300d2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.772+0000 7f4737600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473015c9c0 0x7f47300d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.772+0000 7f4737600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47300d0c00 con 0x7f47300d06c0 2024-04-24T08:21:24.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.772+0000 7f4737600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f47300d0ee0 con 0x7f47300d2350 2024-04-24T08:21:24.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.772+0000 7f4737600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f47300d11c0 con 0x7f473015c9c0 2024-04-24T08:21:24.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.772+0000 7f4734c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f47300d2350 0x7f47300d2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.772+0000 7f4734c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f47300d2350 0x7f47300d2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54856/0 (socket says 172.21.15.69:54856) 2024-04-24T08:21:24.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.772+0000 7f4734c00640 1 -- 172.21.15.69:0/125575439 learned_addr learned my addr 172.21.15.69:0/125575439 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:24.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.773+0000 7f4727e00640 1 --2- 172.21.15.69:0/125575439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473015c9c0 0x7f47300d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.773+0000 7f4734c00640 1 -- 172.21.15.69:0/125575439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473015c9c0 msgr2=0x7f47300d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:24.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.773+0000 7f4734c00640 1 --2- 172.21.15.69:0/125575439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473015c9c0 0x7f47300d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.773+0000 7f4734c00640 1 -- 172.21.15.69:0/125575439 >> v1:172.21.15.121:6789/0 conn(0x7f47300d06c0 legacy=0x7f47300cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:24.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.773+0000 7f4734c00640 1 -- 172.21.15.69:0/125575439 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47300d14a0 con 0x7f47300d2350 2024-04-24T08:21:24.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.774+0000 7f4734c00640 1 --2- 172.21.15.69:0/125575439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 0x7f47300d2730 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f4720009ea0 tx=0x7f472002a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e917a904802455 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:24.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.774+0000 7f472fe00640 1 -- 172.21.15.69:0/125575439 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4720036070 con 0x7f47300d2350 2024-04-24T08:21:24.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.774+0000 7f472fe00640 1 -- 172.21.15.69:0/125575439 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4720031600 con 0x7f47300d2350 2024-04-24T08:21:24.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.774+0000 7f472fe00640 1 -- 172.21.15.69:0/125575439 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47200319d0 con 0x7f47300d2350 2024-04-24T08:21:24.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.775+0000 7f4737600640 1 -- 172.21.15.69:0/125575439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 msgr2=0x7f47300d2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.775+0000 7f4737600640 1 --2- 172.21.15.69:0/125575439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 0x7f47300d2730 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7f4720009ea0 tx=0x7f472002a760 comp rx=0 tx=0).stop 2024-04-24T08:21:24.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.775+0000 7f4737600640 1 -- 172.21.15.69:0/125575439 shutdown_connections 2024-04-24T08:21:24.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.775+0000 7f4737600640 1 --2- 172.21.15.69:0/125575439 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f473015c9c0 0x7f47300d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.775+0000 7f4737600640 1 --2- 172.21.15.69:0/125575439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 0x7f47300d2730 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.775+0000 7f4737600640 1 -- 172.21.15.69:0/125575439 >> 172.21.15.69:0/125575439 conn(0x7f473008bf60 msgr2=0x7f473008c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:24.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.775+0000 7f4737600640 1 -- 172.21.15.69:0/125575439 shutdown_connections 2024-04-24T08:21:24.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.775+0000 7f4737600640 1 -- 172.21.15.69:0/125575439 wait complete. 2024-04-24T08:21:24.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.777+0000 7f4737600640 1 Processor -- start 2024-04-24T08:21:24.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.777+0000 7f4737600640 1 -- start start 2024-04-24T08:21:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.778+0000 7f4737600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300d06c0 0x7f47300b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.778+0000 7f4737600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 0x7f47300b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.778+0000 7f4737600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47300b12f0 0x7f4730150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.778+0000 7f4737600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f47300cd1c0 con 0x7f47300d2350 2024-04-24T08:21:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.778+0000 7f4737600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f47300b2490 con 0x7f47300b12f0 2024-04-24T08:21:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.778+0000 7f4737600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f47300cd340 con 0x7f47300d06c0 2024-04-24T08:21:24.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.778+0000 7f4727e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 0x7f47300b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.778+0000 7f4727e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 0x7f47300b0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54872/0 (socket says 172.21.15.69:54872) 2024-04-24T08:21:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.778+0000 7f4727e00640 1 -- 172.21.15.69:0/3150150703 learned_addr learned my addr 172.21.15.69:0/3150150703 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.778+0000 7f4734c00640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300d06c0 0x7f47300b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f4727e00640 1 -- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300d06c0 msgr2=0x7f47300b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f4727e00640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300d06c0 0x7f47300b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f4727e00640 1 -- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47300b12f0 msgr2=0x7f4730150370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f4735600640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47300b12f0 0x7f4730150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f4727e00640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47300b12f0 0x7f4730150370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f4727e00640 1 -- 172.21.15.69:0/3150150703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47300af1e0 con 0x7f47300d2350 2024-04-24T08:21:24.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f4735600640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47300b12f0 0x7f4730150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:24.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f4727e00640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 0x7f47300b0db0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f471800d5f0 tx=0x7f471800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:24.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f472d600640 1 -- 172.21.15.69:0/3150150703 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47180125f0 con 0x7f47300d2350 2024-04-24T08:21:24.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f472d600640 1 -- 172.21.15.69:0/3150150703 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4718019070 con 0x7f47300d2350 2024-04-24T08:21:24.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f4737600640 1 -- 172.21.15.69:0/3150150703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47300af480 con 0x7f47300d2350 2024-04-24T08:21:24.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.779+0000 7f472d600640 1 -- 172.21.15.69:0/3150150703 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47180129d0 con 0x7f47300d2350 2024-04-24T08:21:24.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.780+0000 7f4737600640 1 -- 172.21.15.69:0/3150150703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47300afa40 con 0x7f47300d2350 2024-04-24T08:21:24.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.781+0000 7f4737600640 1 -- 172.21.15.69:0/3150150703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4730069190 con 0x7f47300d2350 2024-04-24T08:21:24.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.782+0000 7f472d600640 1 -- 172.21.15.69:0/3150150703 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4718012b70 con 0x7f47300d2350 2024-04-24T08:21:24.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.782+0000 7f472d600640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f470c06c920 0x7f470c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:24.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.783+0000 7f472d600640 1 -- 172.21.15.69:0/3150150703 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f471808c080 con 0x7f47300d2350 2024-04-24T08:21:24.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.783+0000 7f4734c00640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f470c06c920 0x7f470c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:24.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.784+0000 7f4734c00640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f470c06c920 0x7f470c06ede0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4720033000 tx=0x7f4720009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:24.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.787+0000 7f472d600640 1 -- 172.21.15.69:0/3150150703 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f471805a360 con 0x7f47300d2350 2024-04-24T08:21:24.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.984+0000 7f4737600640 1 -- 172.21.15.69:0/3150150703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4730070ff0 con 0x7f47300d2350 2024-04-24T08:21:24.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.985+0000 7f472d600640 1 -- 172.21.15.69:0/3150150703 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) ==== 104+0+11 (secure 0 0 0) 0x7f4718059c60 con 0x7f47300d2350 2024-04-24T08:21:24.987 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:21:24.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.987+0000 7f4737600640 1 -- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f470c06c920 msgr2=0x7f470c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.987+0000 7f4737600640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f470c06c920 0x7f470c06ede0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f4720033000 tx=0x7f4720009000 comp rx=0 tx=0).stop 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.987+0000 7f4737600640 1 -- 172.21.15.69:0/3150150703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 msgr2=0x7f47300b0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.987+0000 7f4737600640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 0x7f47300b0db0 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f471800d5f0 tx=0x7f471800dab0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.988+0000 7f4737600640 1 -- 172.21.15.69:0/3150150703 shutdown_connections 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.988+0000 7f4737600640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f470c06c920 0x7f470c06ede0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.988+0000 7f4737600640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47300d2350 0x7f47300b0db0 unknown :-1 s=CLOSED pgs=2113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.988+0000 7f4737600640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f47300b12f0 0x7f4730150370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.988+0000 7f4737600640 1 --2- 172.21.15.69:0/3150150703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f47300d06c0 0x7f47300b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.988+0000 7f4737600640 1 -- 172.21.15.69:0/3150150703 >> 172.21.15.69:0/3150150703 conn(0x7f473008bf60 msgr2=0x7f47300cecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.988+0000 7f4737600640 1 -- 172.21.15.69:0/3150150703 shutdown_connections 2024-04-24T08:21:24.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:24.988+0000 7f4737600640 1 -- 172.21.15.69:0/3150150703 wait complete. 2024-04-24T08:21:25.009 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:21:25.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.209+0000 7f1d69600640 1 Processor -- start 2024-04-24T08:21:25.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.210+0000 7f1d69600640 1 -- start start 2024-04-24T08:21:25.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.211+0000 7f1d69600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d64071d70 0x7f1d6415ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.211+0000 7f1d69600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64160510 0x7f1d640cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.211+0000 7f1d69600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d64072450 con 0x7f1d641420a0 2024-04-24T08:21:25.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.211+0000 7f1d69600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1d64160960 con 0x7f1d64071d70 2024-04-24T08:21:25.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.211+0000 7f1d69600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1d640cf1e0 con 0x7f1d64160510 2024-04-24T08:21:25.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.212+0000 7f1d5b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d64071d70 0x7f1d6415ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.212+0000 7f1d5b400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d64071d70 0x7f1d6415ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54876/0 (socket says 172.21.15.69:54876) 2024-04-24T08:21:25.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.212+0000 7f1d5b400640 1 -- 172.21.15.69:0/582599248 learned_addr learned my addr 172.21.15.69:0/582599248 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:25.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.212+0000 7f1d68c00640 1 --2- 172.21.15.69:0/582599248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64160510 0x7f1d640cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.212+0000 7f1d5b400640 1 -- 172.21.15.69:0/582599248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64160510 msgr2=0x7f1d640cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:25.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.212+0000 7f1d5b400640 1 --2- 172.21.15.69:0/582599248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64160510 0x7f1d640cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.212+0000 7f1d5b400640 1 -- 172.21.15.69:0/582599248 >> v1:172.21.15.121:6789/0 conn(0x7f1d641420a0 legacy=0x7f1d64142480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.212+0000 7f1d5b400640 1 -- 172.21.15.69:0/582599248 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d64072730 con 0x7f1d64071d70 2024-04-24T08:21:25.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.213+0000 7f1d5b400640 1 --2- 172.21.15.69:0/582599248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d64071d70 0x7f1d6415ffd0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c009870 tx=0x7f1d4c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3303e4a3bd58a27 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:25.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.214+0000 7f1d63400640 1 -- 172.21.15.69:0/582599248 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c037070 con 0x7f1d64071d70 2024-04-24T08:21:25.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.214+0000 7f1d63400640 1 -- 172.21.15.69:0/582599248 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d4c032440 con 0x7f1d64071d70 2024-04-24T08:21:25.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.214+0000 7f1d63400640 1 -- 172.21.15.69:0/582599248 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c032810 con 0x7f1d64071d70 2024-04-24T08:21:25.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.214+0000 7f1d69600640 1 -- 172.21.15.69:0/582599248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d64071d70 msgr2=0x7f1d6415ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.214+0000 7f1d69600640 1 --2- 172.21.15.69:0/582599248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d64071d70 0x7f1d6415ffd0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c009870 tx=0x7f1d4c02a270 comp rx=0 tx=0).stop 2024-04-24T08:21:25.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.214+0000 7f1d69600640 1 -- 172.21.15.69:0/582599248 shutdown_connections 2024-04-24T08:21:25.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.214+0000 7f1d69600640 1 --2- 172.21.15.69:0/582599248 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64160510 0x7f1d640cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.214+0000 7f1d69600640 1 --2- 172.21.15.69:0/582599248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d64071d70 0x7f1d6415ffd0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.214+0000 7f1d69600640 1 -- 172.21.15.69:0/582599248 >> 172.21.15.69:0/582599248 conn(0x7f1d64067b80 msgr2=0x7f1d6406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:25.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.215+0000 7f1d69600640 1 -- 172.21.15.69:0/582599248 shutdown_connections 2024-04-24T08:21:25.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.215+0000 7f1d69600640 1 -- 172.21.15.69:0/582599248 wait complete. 2024-04-24T08:21:25.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.216+0000 7f1d69600640 1 Processor -- start 2024-04-24T08:21:25.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.216+0000 7f1d69600640 1 -- start start 2024-04-24T08:21:25.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.217+0000 7f1d69600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64071d70 0x7f1d640b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.217+0000 7f1d69600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d641420a0 0x7f1d64130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.217+0000 7f1d69600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d64160510 0x7f1d6406eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.217+0000 7f1d69600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d64072cf0 con 0x7f1d641420a0 2024-04-24T08:21:25.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.217+0000 7f1d69600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1d64072200 con 0x7f1d64160510 2024-04-24T08:21:25.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.217+0000 7f1d69600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1d640cfbb0 con 0x7f1d64071d70 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.217+0000 7f1d5b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d641420a0 0x7f1d64130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d5b400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d641420a0 0x7f1d64130c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54892/0 (socket says 172.21.15.69:54892) 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d5b400640 1 -- 172.21.15.69:0/2114011730 learned_addr learned my addr 172.21.15.69:0/2114011730 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d5b400640 1 -- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64071d70 msgr2=0x7f1d640b4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d68c00640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d64160510 0x7f1d6406eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d63e00640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64071d70 0x7f1d640b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d5b400640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64071d70 0x7f1d640b4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d5b400640 1 -- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d64160510 msgr2=0x7f1d6406eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d5b400640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d64160510 0x7f1d6406eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d5b400640 1 -- 172.21.15.69:0/2114011730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d641316d0 con 0x7f1d641420a0 2024-04-24T08:21:25.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d68c00640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d64160510 0x7f1d6406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:25.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d63e00640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64071d70 0x7f1d640b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:25.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.218+0000 7f1d5b400640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d641420a0 0x7f1d64130c10 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c000c00 tx=0x7f1d4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:25.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.219+0000 7f1d60c00640 1 -- 172.21.15.69:0/2114011730 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c041020 con 0x7f1d641420a0 2024-04-24T08:21:25.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.219+0000 7f1d60c00640 1 -- 172.21.15.69:0/2114011730 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d4c0306c0 con 0x7f1d641420a0 2024-04-24T08:21:25.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.219+0000 7f1d69600640 1 -- 172.21.15.69:0/2114011730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d641319b0 con 0x7f1d641420a0 2024-04-24T08:21:25.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.219+0000 7f1d60c00640 1 -- 172.21.15.69:0/2114011730 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c037040 con 0x7f1d641420a0 2024-04-24T08:21:25.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.219+0000 7f1d69600640 1 -- 172.21.15.69:0/2114011730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d6415bac0 con 0x7f1d641420a0 2024-04-24T08:21:25.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.220+0000 7f1d69600640 1 -- 172.21.15.69:0/2114011730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d6412eb80 con 0x7f1d641420a0 2024-04-24T08:21:25.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.221+0000 7f1d60c00640 1 -- 172.21.15.69:0/2114011730 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d4c044020 con 0x7f1d641420a0 2024-04-24T08:21:25.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.222+0000 7f1d60c00640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d4006c8f0 0x7f1d4006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.222+0000 7f1d60c00640 1 -- 172.21.15.69:0/2114011730 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f1d4c0ac4c0 con 0x7f1d641420a0 2024-04-24T08:21:25.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.222+0000 7f1d63e00640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d4006c8f0 0x7f1d4006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.223+0000 7f1d63e00640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d4006c8f0 0x7f1d4006edb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1d54001680 tx=0x7f1d54009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:25.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.226+0000 7f1d60c00640 1 -- 172.21.15.69:0/2114011730 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d4c04b030 con 0x7f1d641420a0 2024-04-24T08:21:25.427 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.425+0000 7f1d69600640 1 -- 172.21.15.69:0/2114011730 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1d64075d50 con 0x7f1d641420a0 2024-04-24T08:21:25.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.426+0000 7f1d60c00640 1 -- 172.21.15.69:0/2114011730 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) ==== 100+0+11 (secure 0 0 0) 0x7f1d4c07a760 con 0x7f1d641420a0 2024-04-24T08:21:25.428 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:21:25.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.429+0000 7f1d69600640 1 -- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d4006c8f0 msgr2=0x7f1d4006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.429+0000 7f1d69600640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d4006c8f0 0x7f1d4006edb0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f1d54001680 tx=0x7f1d54009040 comp rx=0 tx=0).stop 2024-04-24T08:21:25.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.429+0000 7f1d69600640 1 -- 172.21.15.69:0/2114011730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d641420a0 msgr2=0x7f1d64130c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.429+0000 7f1d69600640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d641420a0 0x7f1d64130c10 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c000c00 tx=0x7f1d4c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.430+0000 7f1d69600640 1 -- 172.21.15.69:0/2114011730 shutdown_connections 2024-04-24T08:21:25.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.430+0000 7f1d69600640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d4006c8f0 0x7f1d4006edb0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.430+0000 7f1d69600640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d641420a0 0x7f1d64130c10 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.430+0000 7f1d69600640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d64160510 0x7f1d6406eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.430+0000 7f1d69600640 1 --2- 172.21.15.69:0/2114011730 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d64071d70 0x7f1d640b4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.430+0000 7f1d69600640 1 -- 172.21.15.69:0/2114011730 >> 172.21.15.69:0/2114011730 conn(0x7f1d64067b80 msgr2=0x7f1d64142480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:25.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.430+0000 7f1d69600640 1 -- 172.21.15.69:0/2114011730 shutdown_connections 2024-04-24T08:21:25.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.430+0000 7f1d69600640 1 -- 172.21.15.69:0/2114011730 wait complete. 2024-04-24T08:21:25.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:21:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.650+0000 7f7e71400640 1 Processor -- start 2024-04-24T08:21:25.652 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.650+0000 7f7e71400640 1 -- start start 2024-04-24T08:21:25.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.651+0000 7f7e71400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e6c15b490 0x7f7e6c0b5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e71400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c0d02f0 0x7f7e6c06f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e71400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e6c0b46e0 con 0x7f7e6c06f2d0 2024-04-24T08:21:25.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e71400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f7e6c06fc70 con 0x7f7e6c15b490 2024-04-24T08:21:25.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e71400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6c0d0960 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e6a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e6c15b490 0x7f7e6c0b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e6a800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f7e6c15b490 0x7f7e6c0b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54904/0 (socket says 172.21.15.69:54904) 2024-04-24T08:21:25.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e6a800640 1 -- 172.21.15.69:0/3407491839 learned_addr learned my addr 172.21.15.69:0/3407491839 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:25.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e6b200640 1 --2- 172.21.15.69:0/3407491839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c0d02f0 0x7f7e6c06f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e6a800640 1 -- 172.21.15.69:0/3407491839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c0d02f0 msgr2=0x7f7e6c06f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e6a800640 1 --2- 172.21.15.69:0/3407491839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c0d02f0 0x7f7e6c06f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e6a800640 1 -- 172.21.15.69:0/3407491839 >> v1:172.21.15.121:6789/0 conn(0x7f7e6c06f2d0 legacy=0x7f7e6c0cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.652+0000 7f7e6a800640 1 -- 172.21.15.69:0/3407491839 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c0b4980 con 0x7f7e6c15b490 2024-04-24T08:21:25.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.653+0000 7f7e6a800640 1 --2- 172.21.15.69:0/3407491839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c15b490 0x7f7e6c0b5180 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009e50 tx=0x7f7e5c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310d7acd13bd1bdf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:25.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.654+0000 7f7e69400640 1 -- 172.21.15.69:0/3407491839 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c036070 con 0x7f7e6c15b490 2024-04-24T08:21:25.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.654+0000 7f7e69400640 1 -- 172.21.15.69:0/3407491839 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e5c031600 con 0x7f7e6c15b490 2024-04-24T08:21:25.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.654+0000 7f7e69400640 1 -- 172.21.15.69:0/3407491839 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c0319d0 con 0x7f7e6c15b490 2024-04-24T08:21:25.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.654+0000 7f7e71400640 1 -- 172.21.15.69:0/3407491839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c15b490 msgr2=0x7f7e6c0b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.654+0000 7f7e71400640 1 --2- 172.21.15.69:0/3407491839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c15b490 0x7f7e6c0b5180 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009e50 tx=0x7f7e5c02a6c0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.654+0000 7f7e71400640 1 -- 172.21.15.69:0/3407491839 shutdown_connections 2024-04-24T08:21:25.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.654+0000 7f7e71400640 1 --2- 172.21.15.69:0/3407491839 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c0d02f0 0x7f7e6c06f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.654+0000 7f7e71400640 1 --2- 172.21.15.69:0/3407491839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c15b490 0x7f7e6c0b5180 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.655+0000 7f7e71400640 1 -- 172.21.15.69:0/3407491839 >> 172.21.15.69:0/3407491839 conn(0x7f7e6c067d90 msgr2=0x7f7e6c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:25.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.655+0000 7f7e71400640 1 -- 172.21.15.69:0/3407491839 shutdown_connections 2024-04-24T08:21:25.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.655+0000 7f7e71400640 1 -- 172.21.15.69:0/3407491839 wait complete. 2024-04-24T08:21:25.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.656+0000 7f7e71400640 1 Processor -- start 2024-04-24T08:21:25.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.656+0000 7f7e71400640 1 -- start start 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.657+0000 7f7e71400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c06f2d0 0x7f7e6c14a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.657+0000 7f7e71400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0d02f0 0x7f7e6c0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.657+0000 7f7e71400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c15b490 0x7f7e6c0cea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.657+0000 7f7e71400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c15b870 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.657+0000 7f7e71400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c0d0c40 con 0x7f7e6c06f2d0 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.657+0000 7f7e71400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6c0b56c0 con 0x7f7e6c15b490 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.658+0000 7f7e69e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0d02f0 0x7f7e6c0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.658+0000 7f7e69e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0d02f0 0x7f7e6c0ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54914/0 (socket says 172.21.15.69:54914) 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.658+0000 7f7e69e00640 1 -- 172.21.15.69:0/3854424202 learned_addr learned my addr 172.21.15.69:0/3854424202 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.658+0000 7f7e69e00640 1 -- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c15b490 msgr2=0x7f7e6c0cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.658+0000 7f7e6a800640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c06f2d0 0x7f7e6c14a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.658+0000 7f7e69e00640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c15b490 0x7f7e6c0cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.658+0000 7f7e69e00640 1 -- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c06f2d0 msgr2=0x7f7e6c14a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.658+0000 7f7e69e00640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c06f2d0 0x7f7e6c14a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.658+0000 7f7e69e00640 1 -- 172.21.15.69:0/3854424202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c14b280 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.658+0000 7f7e69e00640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0d02f0 0x7f7e6c0ce4c0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f7e5400c460 tx=0x7f7e5400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:25.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.659+0000 7f7e62a00640 1 -- 172.21.15.69:0/3854424202 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e54016020 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.659+0000 7f7e62a00640 1 -- 172.21.15.69:0/3854424202 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e54012870 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.659+0000 7f7e62a00640 1 -- 172.21.15.69:0/3854424202 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e54012c60 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.659+0000 7f7e71400640 1 -- 172.21.15.69:0/3854424202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e6c15a820 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.659+0000 7f7e71400640 1 -- 172.21.15.69:0/3854424202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e6c159f80 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.660+0000 7f7e71400640 1 -- 172.21.15.69:0/3854424202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e6c06bdc0 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.661+0000 7f7e62a00640 1 -- 172.21.15.69:0/3854424202 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e54019020 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.662+0000 7f7e62a00640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4806c8f0 0x7f7e4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:25.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.662+0000 7f7e62a00640 1 -- 172.21.15.69:0/3854424202 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f7e5408b630 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.663+0000 7f7e6a800640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4806c8f0 0x7f7e4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:25.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.664+0000 7f7e6a800640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4806c8f0 0x7f7e4806edb0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c033000 tx=0x7f7e5c009000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:25.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.667+0000 7f7e62a00640 1 -- 172.21.15.69:0/3854424202 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7e6c06bdc0 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.880+0000 7f7e71400640 1 -- 172.21.15.69:0/3854424202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7e6c072b80 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.882+0000 7f7e62a00640 1 -- 172.21.15.69:0/3854424202 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v113) ==== 80+0+246 (secure 0 0 0) 0x7f7e54020070 con 0x7f7e6c0d02f0 2024-04-24T08:21:25.883 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6835/2194021008 2024-04-25T08:21:12.532354+0000 2024-04-24T08:21:25.883 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6834/2194021008 2024-04-25T08:21:12.532354+0000 2024-04-24T08:21:25.883 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6836/1515102940 2024-04-25T08:20:50.436176+0000 2024-04-24T08:21:25.884 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.121:6834/1515102940 2024-04-25T08:20:50.436176+0000 2024-04-24T08:21:25.884 INFO:teuthology.orchestra.run.smithi069.stderr:listed 4 entries 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 -- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4806c8f0 msgr2=0x7f7e4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4806c8f0 0x7f7e4806edb0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c033000 tx=0x7f7e5c009000 comp rx=0 tx=0).stop 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 -- 172.21.15.69:0/3854424202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0d02f0 msgr2=0x7f7e6c0ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0d02f0 0x7f7e6c0ce4c0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f7e5400c460 tx=0x7f7e5400c920 comp rx=0 tx=0).stop 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 -- 172.21.15.69:0/3854424202 shutdown_connections 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e4806c8f0 0x7f7e4806edb0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e6c0d02f0 0x7f7e6c0ce4c0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e6c15b490 0x7f7e6c0cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 --2- 172.21.15.69:0/3854424202 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e6c06f2d0 0x7f7e6c14a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 -- 172.21.15.69:0/3854424202 >> 172.21.15.69:0/3854424202 conn(0x7f7e6c067d90 msgr2=0x7f7e6c0b1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:25.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 -- 172.21.15.69:0/3854424202 shutdown_connections 2024-04-24T08:21:25.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:25.884+0000 7f7e71400640 1 -- 172.21.15.69:0/3854424202 wait complete. 2024-04-24T08:21:25.903 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-04-24T08:21:25.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:25.903 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T08:21:25.972 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi069.stderr:ceph-fuse[79676]: fuse finished with error 0 and tester_r 0 2024-04-24T08:21:25.979 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:21:25.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:25.979 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:21:26.129 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:21:26.129 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:21:26.130 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:21:26.189 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:21:26.189 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:21:26.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:26.189 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-24T08:21:26.244 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-24T08:21:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:21:26.245 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:21:26.245 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:21:26.245 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:21:26.303 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:21:26.303 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:21:26.303 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:26.303 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-24T08:21:26.354 INFO:teuthology.orchestra.run.smithi069.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-24T08:21:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:21:26.355 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T08:21:26.355 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:21:26.355 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:21:26.411 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:21:26.411 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:21:26.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:26.411 DEBUG:teuthology.orchestra.run.smithi121:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-24T08:21:26.463 INFO:teuthology.orchestra.run.smithi121.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-24T08:21:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:21:26.465 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:26.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.671+0000 7f36c0c00640 1 Processor -- start 2024-04-24T08:21:26.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.672+0000 7f36c0c00640 1 -- start start 2024-04-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.673+0000 7f36c0c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36bc15e960 0x7f36bc162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.673+0000 7f36c0c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36bc0cfaf0 0x7f36bc0cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.673+0000 7f36c0c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36bc15ed40 con 0x7f36bc162980 2024-04-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.673+0000 7f36c0c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f36bc0cf050 con 0x7f36bc0cfaf0 2024-04-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.673+0000 7f36c0c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f36bc162e20 con 0x7f36bc15e960 2024-04-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.673+0000 7f36bbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36bc0cfaf0 0x7f36bc0cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.673+0000 7f36bbe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f36bc0cfaf0 0x7f36bc0cffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54918/0 (socket says 172.21.15.69:54918) 2024-04-24T08:21:26.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.673+0000 7f36bbe00640 1 -- 172.21.15.69:0/3867044210 learned_addr learned my addr 172.21.15.69:0/3867044210 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.674+0000 7f36bbe00640 1 -- 172.21.15.69:0/3867044210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36bc15e960 msgr2=0x7f36bc162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.674+0000 7f36bb400640 1 --2- 172.21.15.69:0/3867044210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36bc15e960 0x7f36bc162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.674+0000 7f36bbe00640 1 --2- 172.21.15.69:0/3867044210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36bc15e960 0x7f36bc162440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.674+0000 7f36bbe00640 1 -- 172.21.15.69:0/3867044210 >> v1:172.21.15.121:6789/0 conn(0x7f36bc162980 legacy=0x7f36bc06cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.674+0000 7f36bbe00640 1 -- 172.21.15.69:0/3867044210 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36bc163100 con 0x7f36bc0cfaf0 2024-04-24T08:21:26.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.675+0000 7f36bbe00640 1 --2- 172.21.15.69:0/3867044210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc0cfaf0 0x7f36bc0cffa0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009e50 tx=0x7f36b402a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6cb00a8981fc0c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.675+0000 7f36baa00640 1 -- 172.21.15.69:0/3867044210 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b4036070 con 0x7f36bc0cfaf0 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.675+0000 7f36baa00640 1 -- 172.21.15.69:0/3867044210 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36b4031600 con 0x7f36bc0cfaf0 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.675+0000 7f36baa00640 1 -- 172.21.15.69:0/3867044210 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b40319d0 con 0x7f36bc0cfaf0 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.676+0000 7f36c0c00640 1 -- 172.21.15.69:0/3867044210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc0cfaf0 msgr2=0x7f36bc0cffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.676+0000 7f36c0c00640 1 --2- 172.21.15.69:0/3867044210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc0cfaf0 0x7f36bc0cffa0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009e50 tx=0x7f36b402a6c0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.676+0000 7f36c0c00640 1 -- 172.21.15.69:0/3867044210 shutdown_connections 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.676+0000 7f36c0c00640 1 --2- 172.21.15.69:0/3867044210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc0cfaf0 0x7f36bc0cffa0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.676+0000 7f36c0c00640 1 --2- 172.21.15.69:0/3867044210 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36bc15e960 0x7f36bc162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.676+0000 7f36c0c00640 1 -- 172.21.15.69:0/3867044210 >> 172.21.15.69:0/3867044210 conn(0x7f36bc09d900 msgr2=0x7f36bc08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.676+0000 7f36c0c00640 1 -- 172.21.15.69:0/3867044210 shutdown_connections 2024-04-24T08:21:26.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.676+0000 7f36c0c00640 1 -- 172.21.15.69:0/3867044210 wait complete. 2024-04-24T08:21:26.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.678+0000 7f36c0c00640 1 Processor -- start 2024-04-24T08:21:26.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.678+0000 7f36c0c00640 1 -- start start 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.679+0000 7f36c0c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36bc0cfaf0 0x7f36bc14cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.679+0000 7f36c0c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc15e960 0x7f36bc15ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.679+0000 7f36c0c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36bc162980 0x7f36bc14d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.679+0000 7f36c0c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc06d6c0 con 0x7f36bc15e960 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.679+0000 7f36c0c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f36bc0cf5d0 con 0x7f36bc162980 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.679+0000 7f36c0c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f36bc0b6170 con 0x7f36bc0cfaf0 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.679+0000 7f36b2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc15e960 0x7f36bc15ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.679+0000 7f36b2a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc15e960 0x7f36bc15ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54932/0 (socket says 172.21.15.69:54932) 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.679+0000 7f36b2a00640 1 -- 172.21.15.69:0/2469182233 learned_addr learned my addr 172.21.15.69:0/2469182233 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.679+0000 7f36bbe00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36bc162980 0x7f36bc14d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:26.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36bb400640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36bc0cfaf0 0x7f36bc14cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36b2a00640 1 -- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36bc0cfaf0 msgr2=0x7f36bc14cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36b2a00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36bc0cfaf0 0x7f36bc14cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36b2a00640 1 -- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36bc162980 msgr2=0x7f36bc14d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36b2a00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36bc162980 0x7f36bc14d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36b2a00640 1 -- 172.21.15.69:0/2469182233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36bc14d140 con 0x7f36bc15e960 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36b2a00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc15e960 0x7f36bc15ef10 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f36a400b420 tx=0x7f36a400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36b3e00640 1 -- 172.21.15.69:0/2469182233 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a4011830 con 0x7f36bc15e960 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36b3e00640 1 -- 172.21.15.69:0/2469182233 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a4016070 con 0x7f36bc15e960 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36b3e00640 1 -- 172.21.15.69:0/2469182233 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a4011c10 con 0x7f36bc15e960 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36c0c00640 1 -- 172.21.15.69:0/2469182233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36bc0c9320 con 0x7f36bc15e960 2024-04-24T08:21:26.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.680+0000 7f36c0c00640 1 -- 172.21.15.69:0/2469182233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36bc0c98e0 con 0x7f36bc15e960 2024-04-24T08:21:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.681+0000 7f36c0c00640 1 -- 172.21.15.69:0/2469182233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36bc075d50 con 0x7f36bc15e960 2024-04-24T08:21:26.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.682+0000 7f36b3e00640 1 -- 172.21.15.69:0/2469182233 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f36a401b460 con 0x7f36bc15e960 2024-04-24T08:21:26.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.683+0000 7f36b3e00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f369806c8f0 0x7f369806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:26.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.683+0000 7f36b3e00640 1 -- 172.21.15.69:0/2469182233 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f36a408b7e0 con 0x7f36bc15e960 2024-04-24T08:21:26.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.684+0000 7f36bb400640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f369806c8f0 0x7f369806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:26.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.685+0000 7f36bb400640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f369806c8f0 0x7f369806edb0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00a970 tx=0x7f36ac008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:26.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.688+0000 7f36b3e00640 1 -- 172.21.15.69:0/2469182233 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f36a4020020 con 0x7f36bc15e960 2024-04-24T08:21:26.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.930+0000 7f36c0c00640 1 -- 172.21.15.69:0/2469182233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36bc0d3770 con 0x7f36bc15e960 2024-04-24T08:21:26.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.932+0000 7f36b3e00640 1 -- 172.21.15.69:0/2469182233 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5186 (secure 0 0 0) 0x7f36bc14d140 con 0x7f36bc15e960 2024-04-24T08:21:26.933 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:26.933 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:21:17.653434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9825},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9825":{"gid":9825,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/99442307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":99442307},{"type":"v1","addr":"172.21.15.69:6835","nonce":99442307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9825,"qdb_cluster":[9825]},"id":8}]} 2024-04-24T08:21:26.934 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 112 2024-04-24T08:21:26.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 -- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f369806c8f0 msgr2=0x7f369806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:26.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f369806c8f0 0x7f369806edb0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00a970 tx=0x7f36ac008040 comp rx=0 tx=0).stop 2024-04-24T08:21:26.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 -- 172.21.15.69:0/2469182233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc15e960 msgr2=0x7f36bc15ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:26.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc15e960 0x7f36bc15ef10 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f36a400b420 tx=0x7f36a400b8e0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 -- 172.21.15.69:0/2469182233 shutdown_connections 2024-04-24T08:21:26.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f369806c8f0 0x7f369806edb0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f36bc15e960 0x7f36bc15ef10 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36bc162980 0x7f36bc14d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 --2- 172.21.15.69:0/2469182233 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f36bc0cfaf0 0x7f36bc14cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:26.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 -- 172.21.15.69:0/2469182233 >> 172.21.15.69:0/2469182233 conn(0x7f36bc09d900 msgr2=0x7f36bc0b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:26.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.934+0000 7f36c0c00640 1 -- 172.21.15.69:0/2469182233 shutdown_connections 2024-04-24T08:21:26.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:26.935+0000 7f36c0c00640 1 -- 172.21.15.69:0/2469182233 wait complete. 2024-04-24T08:21:26.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:27.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.150+0000 7fe498c00640 1 Processor -- start 2024-04-24T08:21:27.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.150+0000 7fe498c00640 1 -- start start 2024-04-24T08:21:27.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.151+0000 7fe498c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe494156c60 0x7fe49415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.151+0000 7fe498c00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe4940b30f0 0x7fe49415d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.151+0000 7fe498c00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe49415d660 con 0x7fe4940ad0d0 2024-04-24T08:21:27.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.151+0000 7fe498c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe49406dc30 con 0x7fe494156c60 2024-04-24T08:21:27.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.151+0000 7fe498c00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe49406df10 con 0x7fe4940b30f0 2024-04-24T08:21:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.152+0000 7fe493400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe494156c60 0x7fe49415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.152+0000 7fe493400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe494156c60 0x7fe49415cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54942/0 (socket says 172.21.15.69:54942) 2024-04-24T08:21:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.152+0000 7fe493400640 1 -- 172.21.15.69:0/429141022 learned_addr learned my addr 172.21.15.69:0/429141022 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.152+0000 7fe493e00640 1 --1- 172.21.15.69:0/429141022 >> v1:172.21.15.69:6789/0 conn(0x7fe4940ad0d0 0x7fe4941600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50766/0 (socket says 172.21.15.69:50766) 2024-04-24T08:21:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.152+0000 7fe48be00640 1 --2- 172.21.15.69:0/429141022 >> v2:172.21.15.121:3300/0 conn(0x7fe4940b30f0 0x7fe49415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.152+0000 7fe493400640 1 -- 172.21.15.69:0/429141022 >> v2:172.21.15.121:3300/0 conn(0x7fe4940b30f0 msgr2=0x7fe49415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.152+0000 7fe493400640 1 --2- 172.21.15.69:0/429141022 >> v2:172.21.15.121:3300/0 conn(0x7fe4940b30f0 0x7fe49415d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.152+0000 7fe493400640 1 -- 172.21.15.69:0/429141022 >> v1:172.21.15.69:6789/0 conn(0x7fe4940ad0d0 legacy=0x7fe4941600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.152+0000 7fe493400640 1 -- 172.21.15.69:0/429141022 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe49406e1f0 con 0x7fe494156c60 2024-04-24T08:21:27.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.153+0000 7fe493400640 1 --2- 172.21.15.69:0/429141022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe494156c60 0x7fe49415cac0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fe49406d900 tx=0x7fe48402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68da62840fc903f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:27.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.154+0000 7fe492a00640 1 -- 172.21.15.69:0/429141022 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe484037070 con 0x7fe494156c60 2024-04-24T08:21:27.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.154+0000 7fe492a00640 1 -- 172.21.15.69:0/429141022 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe484032440 con 0x7fe494156c60 2024-04-24T08:21:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.154+0000 7fe492a00640 1 -- 172.21.15.69:0/429141022 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe484032810 con 0x7fe494156c60 2024-04-24T08:21:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.154+0000 7fe498c00640 1 -- 172.21.15.69:0/429141022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe494156c60 msgr2=0x7fe49415cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.154+0000 7fe498c00640 1 --2- 172.21.15.69:0/429141022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe494156c60 0x7fe49415cac0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7fe49406d900 tx=0x7fe48402a270 comp rx=0 tx=0).stop 2024-04-24T08:21:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.155+0000 7fe498c00640 1 -- 172.21.15.69:0/429141022 shutdown_connections 2024-04-24T08:21:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.155+0000 7fe498c00640 1 --2- 172.21.15.69:0/429141022 >> v2:172.21.15.121:3300/0 conn(0x7fe4940b30f0 0x7fe49415d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.155+0000 7fe498c00640 1 --2- 172.21.15.69:0/429141022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe494156c60 0x7fe49415cac0 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.155+0000 7fe498c00640 1 -- 172.21.15.69:0/429141022 >> 172.21.15.69:0/429141022 conn(0x7fe4940d0160 msgr2=0x7fe4940d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.155+0000 7fe498c00640 1 -- 172.21.15.69:0/429141022 shutdown_connections 2024-04-24T08:21:27.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.155+0000 7fe498c00640 1 -- 172.21.15.69:0/429141022 wait complete. 2024-04-24T08:21:27.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.156+0000 7fe498c00640 1 Processor -- start 2024-04-24T08:21:27.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.156+0000 7fe498c00640 1 -- start start 2024-04-24T08:21:27.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.157+0000 7fe498c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4940ad0d0 0x7fe4940997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.157+0000 7fe498c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4940b30f0 0x7fe4941589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.157+0000 7fe498c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe494156c60 0x7fe494159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.157+0000 7fe498c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4940ad540 con 0x7fe4940b30f0 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.157+0000 7fe498c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe49415d940 con 0x7fe494156c60 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.157+0000 7fe498c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe4940ad6c0 con 0x7fe4940ad0d0 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.158+0000 7fe48be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4940b30f0 0x7fe4941589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.158+0000 7fe48be00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4940b30f0 0x7fe4941589d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54954/0 (socket says 172.21.15.69:54954) 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.158+0000 7fe48be00640 1 -- 172.21.15.69:0/3091969562 learned_addr learned my addr 172.21.15.69:0/3091969562 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.158+0000 7fe493400640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4940ad0d0 0x7fe4940997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.158+0000 7fe48be00640 1 -- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4940ad0d0 msgr2=0x7fe4940997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.158+0000 7fe48be00640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4940ad0d0 0x7fe4940997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.158+0000 7fe493e00640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe494156c60 0x7fe494159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.158+0000 7fe48be00640 1 -- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe494156c60 msgr2=0x7fe494159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.158+0000 7fe48be00640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe494156c60 0x7fe494159520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.158+0000 7fe48be00640 1 -- 172.21.15.69:0/3091969562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe494154cf0 con 0x7fe4940b30f0 2024-04-24T08:21:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.159+0000 7fe48be00640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4940b30f0 0x7fe4941589d0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00d7d0 tx=0x7fe47c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.159+0000 7fe48b400640 1 -- 172.21.15.69:0/3091969562 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00b6c0 con 0x7fe4940b30f0 2024-04-24T08:21:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.159+0000 7fe48b400640 1 -- 172.21.15.69:0/3091969562 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe47c019070 con 0x7fe4940b30f0 2024-04-24T08:21:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.159+0000 7fe498c00640 1 -- 172.21.15.69:0/3091969562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe494154fd0 con 0x7fe4940b30f0 2024-04-24T08:21:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.159+0000 7fe48b400640 1 -- 172.21.15.69:0/3091969562 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00ba80 con 0x7fe4940b30f0 2024-04-24T08:21:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.159+0000 7fe498c00640 1 -- 172.21.15.69:0/3091969562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe494155590 con 0x7fe4940b30f0 2024-04-24T08:21:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.160+0000 7fe498c00640 1 -- 172.21.15.69:0/3091969562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe494075d50 con 0x7fe4940b30f0 2024-04-24T08:21:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.161+0000 7fe48b400640 1 -- 172.21.15.69:0/3091969562 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe47c00bc20 con 0x7fe4940b30f0 2024-04-24T08:21:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.162+0000 7fe48b400640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe47006c8f0 0x7fe47006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.162+0000 7fe48b400640 1 -- 172.21.15.69:0/3091969562 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fe47c08c160 con 0x7fe4940b30f0 2024-04-24T08:21:27.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.163+0000 7fe493400640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe47006c8f0 0x7fe47006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.164+0000 7fe493400640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe47006c8f0 0x7fe47006edb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe484034000 tx=0x7fe48402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:27.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.166+0000 7fe48b400640 1 -- 172.21.15.69:0/3091969562 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe47c01c360 con 0x7fe4940b30f0 2024-04-24T08:21:27.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.365+0000 7fe498c00640 1 -- 172.21.15.69:0/3091969562 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4940b4570 con 0x7fe4940b30f0 2024-04-24T08:21:27.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.367+0000 7fe48b400640 1 -- 172.21.15.69:0/3091969562 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+17017 (secure 0 0 0) 0x7fe47c05a400 con 0x7fe4940b30f0 2024-04-24T08:21:27.369 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:27.369 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":113,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:21:12.532402+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.37","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/2194021008":"2024-04-25T08:21:12.532354+0000","172.21.15.69:6834/2194021008":"2024-04-25T08:21:12.532354+0000","172.21.15.121:6836/1515102940":"2024-04-25T08:20:50.436176+0000","172.21.15.121:6834/1515102940":"2024-04-25T08:20:50.436176+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:27.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.369+0000 7fe498c00640 1 -- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe47006c8f0 msgr2=0x7fe47006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.369+0000 7fe498c00640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe47006c8f0 0x7fe47006edb0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe484034000 tx=0x7fe48402e000 comp rx=0 tx=0).stop 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.370+0000 7fe498c00640 1 -- 172.21.15.69:0/3091969562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4940b30f0 msgr2=0x7fe4941589d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.370+0000 7fe498c00640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4940b30f0 0x7fe4941589d0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00d7d0 tx=0x7fe47c00dc90 comp rx=0 tx=0).stop 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.370+0000 7fe498c00640 1 -- 172.21.15.69:0/3091969562 shutdown_connections 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.370+0000 7fe498c00640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe47006c8f0 0x7fe47006edb0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.370+0000 7fe498c00640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4940b30f0 0x7fe4941589d0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.370+0000 7fe498c00640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe494156c60 0x7fe494159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.370+0000 7fe498c00640 1 --2- 172.21.15.69:0/3091969562 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe4940ad0d0 0x7fe4940997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.370+0000 7fe498c00640 1 -- 172.21.15.69:0/3091969562 >> 172.21.15.69:0/3091969562 conn(0x7fe4940d0160 msgr2=0x7fe4940b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.370+0000 7fe498c00640 1 -- 172.21.15.69:0/3091969562 shutdown_connections 2024-04-24T08:21:27.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.370+0000 7fe498c00640 1 -- 172.21.15.69:0/3091969562 wait complete. 2024-04-24T08:21:27.391 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:08:08.695404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:20:06.963817+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:20:08.190440+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:21:27.391 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:21:27.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.592+0000 7f2cb5400640 1 Processor -- start 2024-04-24T08:21:27.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.592+0000 7f2cb5400640 1 -- start start 2024-04-24T08:21:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2cb5400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 0x7f2cb006f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2cb5400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2cb00cd050 0x7f2cb00cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2cb5400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cb006e680 con 0x7f2cb0072130 2024-04-24T08:21:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2cb5400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2cb00cd9f0 con 0x7f2cb00cd050 2024-04-24T08:21:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2cb5400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2cb006f570 con 0x7f2cb015e8e0 2024-04-24T08:21:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2ca5e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2cb00cd050 0x7f2cb00cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2caf200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2cb0072130 0x7f2cb00725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50780/0 (socket says 172.21.15.69:50780) 2024-04-24T08:21:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2ca5e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2cb00cd050 0x7f2cb00cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54968/0 (socket says 172.21.15.69:54968) 2024-04-24T08:21:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2caf200640 1 -- 172.21.15.69:0/2969929658 learned_addr learned my addr 172.21.15.69:0/2969929658 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2cae800640 1 --2- 172.21.15.69:0/2969929658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 0x7f2cb006f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.594+0000 7f2ca5e00640 1 -- 172.21.15.69:0/2969929658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 msgr2=0x7f2cb006f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.595+0000 7f2ca5e00640 1 --2- 172.21.15.69:0/2969929658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 0x7f2cb006f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.595+0000 7f2ca5e00640 1 -- 172.21.15.69:0/2969929658 >> v1:172.21.15.69:6789/0 conn(0x7f2cb0072130 legacy=0x7f2cb00725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.595+0000 7f2cade00640 1 -- 172.21.15.69:0/2969929658 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200662164 0 0) 0x7f2cb006e680 con 0x7f2cb0072130 2024-04-24T08:21:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.595+0000 7f2ca5e00640 1 -- 172.21.15.69:0/2969929658 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb006e960 con 0x7f2cb00cd050 2024-04-24T08:21:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.595+0000 7f2cae800640 1 --2- 172.21.15.69:0/2969929658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 0x7f2cb006f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.596+0000 7f2ca5e00640 1 --2- 172.21.15.69:0/2969929658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 0x7f2cb00cd4b0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f2c980097a0 tx=0x7f2c9802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eddf429e01b69a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.596+0000 7f2cade00640 1 -- 172.21.15.69:0/2969929658 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98037070 con 0x7f2cb00cd050 2024-04-24T08:21:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.596+0000 7f2cade00640 1 -- 172.21.15.69:0/2969929658 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c98032440 con 0x7f2cb00cd050 2024-04-24T08:21:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.596+0000 7f2cade00640 1 -- 172.21.15.69:0/2969929658 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98032810 con 0x7f2cb00cd050 2024-04-24T08:21:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.596+0000 7f2cb5400640 1 -- 172.21.15.69:0/2969929658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 msgr2=0x7f2cb00cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.596+0000 7f2cb5400640 1 --2- 172.21.15.69:0/2969929658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 0x7f2cb00cd4b0 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f2c980097a0 tx=0x7f2c9802a180 comp rx=0 tx=0).stop 2024-04-24T08:21:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.597+0000 7f2cb5400640 1 -- 172.21.15.69:0/2969929658 shutdown_connections 2024-04-24T08:21:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.597+0000 7f2cb5400640 1 --2- 172.21.15.69:0/2969929658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 0x7f2cb00cd4b0 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.597+0000 7f2cb5400640 1 --2- 172.21.15.69:0/2969929658 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 0x7f2cb006f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.597+0000 7f2cb5400640 1 -- 172.21.15.69:0/2969929658 >> 172.21.15.69:0/2969929658 conn(0x7f2cb00b2380 msgr2=0x7f2cb00b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.597+0000 7f2cb5400640 1 -- 172.21.15.69:0/2969929658 shutdown_connections 2024-04-24T08:21:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.597+0000 7f2cb5400640 1 -- 172.21.15.69:0/2969929658 wait complete. 2024-04-24T08:21:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.598+0000 7f2cb5400640 1 Processor -- start 2024-04-24T08:21:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.599+0000 7f2cb5400640 1 -- start start 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.599+0000 7f2cb5400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0072130 0x7f2cb0153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.599+0000 7f2cb5400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 0x7f2cb00b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.599+0000 7f2cb5400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 0x7f2cb00c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.599+0000 7f2cb5400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb006f200 con 0x7f2cb00cd050 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.599+0000 7f2cb5400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2cb006ef20 con 0x7f2cb0072130 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.599+0000 7f2cb5400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2cb00cf020 con 0x7f2cb015e8e0 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2ca5e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 0x7f2cb00b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2ca5e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 0x7f2cb00b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54984/0 (socket says 172.21.15.69:54984) 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2ca5e00640 1 -- 172.21.15.69:0/1328141759 learned_addr learned my addr 172.21.15.69:0/1328141759 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2caf200640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 0x7f2cb00c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2cae800640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0072130 0x7f2cb0153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2ca5e00640 1 -- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 msgr2=0x7f2cb00c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2ca5e00640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 0x7f2cb00c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2ca5e00640 1 -- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0072130 msgr2=0x7f2cb0153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2ca5e00640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0072130 0x7f2cb0153b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2ca5e00640 1 -- 172.21.15.69:0/1328141759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cb0161750 con 0x7f2cb00cd050 2024-04-24T08:21:27.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2caf200640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 0x7f2cb00c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:27.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.600+0000 7f2ca5e00640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 0x7f2cb00b3560 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f2c980097a0 tx=0x7f2c980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.601+0000 7f2ca7400640 1 -- 172.21.15.69:0/1328141759 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98037050 con 0x7f2cb00cd050 2024-04-24T08:21:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.601+0000 7f2ca7400640 1 -- 172.21.15.69:0/1328141759 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c98038040 con 0x7f2cb00cd050 2024-04-24T08:21:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.601+0000 7f2ca7400640 1 -- 172.21.15.69:0/1328141759 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98004490 con 0x7f2cb00cd050 2024-04-24T08:21:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.601+0000 7f2cb5400640 1 -- 172.21.15.69:0/1328141759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cb015edf0 con 0x7f2cb00cd050 2024-04-24T08:21:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.601+0000 7f2cb5400640 1 -- 172.21.15.69:0/1328141759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cb015f3b0 con 0x7f2cb00cd050 2024-04-24T08:21:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.602+0000 7f2cb5400640 1 -- 172.21.15.69:0/1328141759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cb0068a80 con 0x7f2cb00cd050 2024-04-24T08:21:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.603+0000 7f2ca7400640 1 -- 172.21.15.69:0/1328141759 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2c98030980 con 0x7f2cb00cd050 2024-04-24T08:21:27.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.604+0000 7f2ca7400640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c8c06c8f0 0x7f2c8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:27.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.604+0000 7f2ca7400640 1 -- 172.21.15.69:0/1328141759 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f2c980ac920 con 0x7f2cb00cd050 2024-04-24T08:21:27.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.605+0000 7f2cae800640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c8c06c8f0 0x7f2c8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:27.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.606+0000 7f2cae800640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c8c06c8f0 0x7f2c8c06edb0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0008ee0 tx=0x7f2ca0008430 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:27.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.608+0000 7f2ca7400640 1 -- 172.21.15.69:0/1328141759 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2c9807ae10 con 0x7f2cb00cd050 2024-04-24T08:21:27.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.808+0000 7f2cb5400640 1 -- 172.21.15.69:0/1328141759 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2cb00ce620 con 0x7f2cb00cd050 2024-04-24T08:21:27.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.809+0000 7f2ca7400640 1 -- 172.21.15.69:0/1328141759 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) ==== 93+0+10 (secure 0 0 0) 0x7f2c9807a710 con 0x7f2cb00cd050 2024-04-24T08:21:27.810 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-04-24T08:21:27.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 -- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c8c06c8f0 msgr2=0x7f2c8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c8c06c8f0 0x7f2c8c06edb0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0008ee0 tx=0x7f2ca0008430 comp rx=0 tx=0).stop 2024-04-24T08:21:27.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 -- 172.21.15.69:0/1328141759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 msgr2=0x7f2cb00b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:27.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 0x7f2cb00b3560 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f2c980097a0 tx=0x7f2c980046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 -- 172.21.15.69:0/1328141759 shutdown_connections 2024-04-24T08:21:27.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2c8c06c8f0 0x7f2c8c06edb0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cb00cd050 0x7f2cb00b3560 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2cb015e8e0 0x7f2cb00c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 --2- 172.21.15.69:0/1328141759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2cb0072130 0x7f2cb0153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:27.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 -- 172.21.15.69:0/1328141759 >> 172.21.15.69:0/1328141759 conn(0x7f2cb00b2380 msgr2=0x7f2cb00ad850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:27.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 -- 172.21.15.69:0/1328141759 shutdown_connections 2024-04-24T08:21:27.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:27.811+0000 7f2cb5400640 1 -- 172.21.15.69:0/1328141759 wait complete. 2024-04-24T08:21:27.831 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:21:28.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.032+0000 7f0675a00640 1 Processor -- start 2024-04-24T08:21:28.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.032+0000 7f0675a00640 1 -- start start 2024-04-24T08:21:28.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.034+0000 7f0675a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06701420a0 0x7f0670071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.034+0000 7f0675a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0670072450 con 0x7f06701606e0 2024-04-24T08:21:28.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.034+0000 7f0675a00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0670072730 con 0x7f0670142480 2024-04-24T08:21:28.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.034+0000 7f0675a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f06700cf1e0 con 0x7f06701420a0 2024-04-24T08:21:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.034+0000 7f0675000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f06701606e0 0x7f06700cf670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50784/0 (socket says 172.21.15.69:50784) 2024-04-24T08:21:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.034+0000 7f0675000640 1 -- 172.21.15.69:0/2765064003 learned_addr learned my addr 172.21.15.69:0/2765064003 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.035+0000 7f066fe00640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06701420a0 0x7f0670071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.035+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1770860114 0 0) 0x7f0670072450 con 0x7f06701606e0 2024-04-24T08:21:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.035+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0654003520 con 0x7f06701606e0 2024-04-24T08:21:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.035+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 227658595 0 0) 0x7f0654003520 con 0x7f06701606e0 2024-04-24T08:21:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.035+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06701420a0 msgr2=0x7f0670071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:28.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.035+0000 7f066ea00640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06701420a0 0x7f0670071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.035+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 >> v1:172.21.15.121:6789/0 conn(0x7f0670142480 legacy=0x7f067015ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.035+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0670072a10 con 0x7f06701606e0 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.035+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f06680019f0 con 0x7f06701606e0 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.035+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 >> v1:172.21.15.69:6789/0 conn(0x7f06701606e0 legacy=0x7f06700cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066ea00640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06701420a0 0x7f06540039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066ea00640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0654004060 0x7f0654004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066ea00640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0654004d40 0x7f06540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f06700cf1e0 con 0x7f0654004060 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f065400bc00 con 0x7f0654004d40 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f065400be00 con 0x7f06701420a0 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066f400640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0654004060 0x7f0654004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f0675000640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0654004d40 0x7f06540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066f400640 1 -- 172.21.15.69:0/2765064003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06701420a0 msgr2=0x7f06540039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:28.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066f400640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06701420a0 0x7f06540039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066f400640 1 -- 172.21.15.69:0/2765064003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0654004d40 msgr2=0x7f06540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:28.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066f400640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0654004d40 0x7f06540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.036+0000 7f066f400640 1 -- 172.21.15.69:0/2765064003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0654003520 con 0x7f0654004060 2024-04-24T08:21:28.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.037+0000 7f066f400640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0654004060 0x7f0654004800 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f0658001750 tx=0x7f0654005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1391e356cf8191e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:28.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.038+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0658001370 con 0x7f0654004060 2024-04-24T08:21:28.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.038+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06580052d0 con 0x7f0654004060 2024-04-24T08:21:28.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.038+0000 7f066ea00640 1 -- 172.21.15.69:0/2765064003 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f065802abe0 con 0x7f0654004060 2024-04-24T08:21:28.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.038+0000 7f0675a00640 1 -- 172.21.15.69:0/2765064003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0654004060 msgr2=0x7f0654004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.038+0000 7f0675a00640 1 --2- 172.21.15.69:0/2765064003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0654004060 0x7f0654004800 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7f0658001750 tx=0x7f0654005830 comp rx=0 tx=0).stop 2024-04-24T08:21:28.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.038+0000 7f0675000640 1 -- 172.21.15.69:0/2765064003 reap_dead start 2024-04-24T08:21:28.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.038+0000 7f0675a00640 1 -- 172.21.15.69:0/2765064003 shutdown_connections 2024-04-24T08:21:28.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.038+0000 7f0675a00640 1 -- 172.21.15.69:0/2765064003 >> 172.21.15.69:0/2765064003 conn(0x7f0670067b80 msgr2=0x7f067006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:28.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.038+0000 7f0675a00640 1 -- 172.21.15.69:0/2765064003 shutdown_connections 2024-04-24T08:21:28.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.038+0000 7f0675a00640 1 -- 172.21.15.69:0/2765064003 wait complete. 2024-04-24T08:21:28.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.040+0000 7f0675a00640 1 Processor -- start 2024-04-24T08:21:28.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.040+0000 7f0675a00640 1 -- start start 2024-04-24T08:21:28.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.041+0000 7f0675a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0654004060 0x7f0670140d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.041+0000 7f0675a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06700b22a0 0x7f06700b2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.041+0000 7f0675a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06700b2c50 0x7f0670155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.041+0000 7f0675a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f06701428e0 con 0x7f06700b2c50 2024-04-24T08:21:28.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.041+0000 7f0675a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0670071d70 con 0x7f0654004060 2024-04-24T08:21:28.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.041+0000 7f0675a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f0670157cc0 con 0x7f06700b22a0 2024-04-24T08:21:28.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.041+0000 7f0675000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06700b2c50 0x7f0670155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.041+0000 7f0675000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06700b2c50 0x7f0670155ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55012/0 (socket says 172.21.15.69:55012) 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.041+0000 7f0675000640 1 -- 172.21.15.69:0/3623669629 learned_addr learned my addr 172.21.15.69:0/3623669629 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f066fe00640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0654004060 0x7f0670140d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f066f400640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06700b22a0 0x7f06700b2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f0675000640 1 -- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06700b22a0 msgr2=0x7f06700b2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f0675000640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06700b22a0 0x7f06700b2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f0675000640 1 -- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0654004060 msgr2=0x7f0670140d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f0675000640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0654004060 0x7f0670140d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f0675000640 1 -- 172.21.15.69:0/3623669629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06701412a0 con 0x7f06700b2c50 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f0675000640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06700b2c50 0x7f0670155ae0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f0670153aa0 tx=0x7f066800e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f0667e00640 1 -- 172.21.15.69:0/3623669629 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066800ec30 con 0x7f06700b2c50 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f0667e00640 1 -- 172.21.15.69:0/3623669629 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0668018070 con 0x7f06700b2c50 2024-04-24T08:21:28.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f0667e00640 1 -- 172.21.15.69:0/3623669629 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668013420 con 0x7f06700b2c50 2024-04-24T08:21:28.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.042+0000 7f0675a00640 1 -- 172.21.15.69:0/3623669629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0670158150 con 0x7f06700b2c50 2024-04-24T08:21:28.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.043+0000 7f0675a00640 1 -- 172.21.15.69:0/3623669629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0670157390 con 0x7f06700b2c50 2024-04-24T08:21:28.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.044+0000 7f0675a00640 1 -- 172.21.15.69:0/3623669629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0670075d50 con 0x7f06700b2c50 2024-04-24T08:21:28.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.045+0000 7f0667e00640 1 -- 172.21.15.69:0/3623669629 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f06680135e0 con 0x7f06700b2c50 2024-04-24T08:21:28.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.046+0000 7f0667e00640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f064c06c8c0 0x7f064c06ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.046+0000 7f0667e00640 1 -- 172.21.15.69:0/3623669629 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f066808bff0 con 0x7f06700b2c50 2024-04-24T08:21:28.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.046+0000 7f066fe00640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f064c06c8c0 0x7f064c06ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.047+0000 7f066fe00640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f064c06c8c0 0x7f064c06ed80 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0660004790 tx=0x7f0660008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:28.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.050+0000 7f0667e00640 1 -- 172.21.15.69:0/3623669629 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06701412a0 con 0x7f06700b2c50 2024-04-24T08:21:28.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.253+0000 7f0675a00640 1 -- 172.21.15.69:0/3623669629 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f067006c470 con 0x7f06700b2c50 2024-04-24T08:21:28.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.254+0000 7f0667e00640 1 -- 172.21.15.69:0/3623669629 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) ==== 104+0+11 (secure 0 0 0) 0x7f066805a2b0 con 0x7f06700b2c50 2024-04-24T08:21:28.255 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-04-24T08:21:28.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.256+0000 7f0675a00640 1 -- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f064c06c8c0 msgr2=0x7f064c06ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.256+0000 7f0675a00640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f064c06c8c0 0x7f064c06ed80 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0660004790 tx=0x7f0660008040 comp rx=0 tx=0).stop 2024-04-24T08:21:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.256+0000 7f0675a00640 1 -- 172.21.15.69:0/3623669629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06700b2c50 msgr2=0x7f0670155ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.256+0000 7f0675a00640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06700b2c50 0x7f0670155ae0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f0670153aa0 tx=0x7f066800e160 comp rx=0 tx=0).stop 2024-04-24T08:21:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.257+0000 7f0675a00640 1 -- 172.21.15.69:0/3623669629 shutdown_connections 2024-04-24T08:21:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.257+0000 7f0675a00640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f064c06c8c0 0x7f064c06ed80 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.257+0000 7f0675a00640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f06700b2c50 0x7f0670155ae0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.257+0000 7f0675a00640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f06700b22a0 0x7f06700b2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.257+0000 7f0675a00640 1 --2- 172.21.15.69:0/3623669629 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0654004060 0x7f0670140d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.257+0000 7f0675a00640 1 -- 172.21.15.69:0/3623669629 >> 172.21.15.69:0/3623669629 conn(0x7f0670067b80 msgr2=0x7f067006eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.257+0000 7f0675a00640 1 -- 172.21.15.69:0/3623669629 shutdown_connections 2024-04-24T08:21:28.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.257+0000 7f0675a00640 1 -- 172.21.15.69:0/3623669629 wait complete. 2024-04-24T08:21:28.278 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:21:28.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.483+0000 7fe505600640 1 Processor -- start 2024-04-24T08:21:28.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.483+0000 7fe505600640 1 -- start start 2024-04-24T08:21:28.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.485+0000 7fe505600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe5000b3260 0x7fe5000b4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.485+0000 7fe505600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5001631c0 con 0x7fe5000cd3d0 2024-04-24T08:21:28.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.485+0000 7fe505600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5000cf4d0 con 0x7fe500073db0 2024-04-24T08:21:28.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.485+0000 7fe505600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe5000cf7b0 con 0x7fe5000b3260 2024-04-24T08:21:28.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.485+0000 7fe4f6000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe5000cd3d0 0x7fe5000b4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:50796/0 (socket says 172.21.15.69:50796) 2024-04-24T08:21:28.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.485+0000 7fe4f6000640 1 -- 172.21.15.69:0/1643863306 learned_addr learned my addr 172.21.15.69:0/1643863306 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:28.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.485+0000 7fe4ff400640 1 --2- 172.21.15.69:0/1643863306 >> v2:172.21.15.69:3300/0 conn(0x7fe5000b3260 0x7fe5000b4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.486+0000 7fe4fe000640 1 -- 172.21.15.69:0/1643863306 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240085994 0 0) 0x7fe5001631c0 con 0x7fe5000cd3d0 2024-04-24T08:21:28.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.486+0000 7fe4fe000640 1 -- 172.21.15.69:0/1643863306 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4e4003520 con 0x7fe5000cd3d0 2024-04-24T08:21:28.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.486+0000 7fe4ff400640 1 -- 172.21.15.69:0/1643863306 >> v1:172.21.15.121:6789/0 conn(0x7fe500073db0 legacy=0x7fe5000cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.486+0000 7fe4ff400640 1 -- 172.21.15.69:0/1643863306 >> v1:172.21.15.69:6789/0 conn(0x7fe5000cd3d0 legacy=0x7fe5000b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.486+0000 7fe4ff400640 1 -- 172.21.15.69:0/1643863306 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5000cfa90 con 0x7fe5000b3260 2024-04-24T08:21:28.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.487+0000 7fe4ff400640 1 --2- 172.21.15.69:0/1643863306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5000b3260 0x7fe5000b4e90 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8009d90 tx=0x7fe4f802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8b2d0a6e4824e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:28.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.488+0000 7fe4fe000640 1 -- 172.21.15.69:0/1643863306 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f8007740 con 0x7fe5000b3260 2024-04-24T08:21:28.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.488+0000 7fe4fe000640 1 -- 172.21.15.69:0/1643863306 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f80078e0 con 0x7fe5000b3260 2024-04-24T08:21:28.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.488+0000 7fe4fe000640 1 -- 172.21.15.69:0/1643863306 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f8007c90 con 0x7fe5000b3260 2024-04-24T08:21:28.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.488+0000 7fe505600640 1 -- 172.21.15.69:0/1643863306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5000b3260 msgr2=0x7fe5000b4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.488+0000 7fe505600640 1 --2- 172.21.15.69:0/1643863306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5000b3260 0x7fe5000b4e90 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8009d90 tx=0x7fe4f802a6f0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.488+0000 7fe505600640 1 -- 172.21.15.69:0/1643863306 shutdown_connections 2024-04-24T08:21:28.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.488+0000 7fe505600640 1 --2- 172.21.15.69:0/1643863306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5000b3260 0x7fe5000b4e90 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.488+0000 7fe505600640 1 -- 172.21.15.69:0/1643863306 >> 172.21.15.69:0/1643863306 conn(0x7fe5000d0350 msgr2=0x7fe5000d0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:28.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.488+0000 7fe505600640 1 -- 172.21.15.69:0/1643863306 shutdown_connections 2024-04-24T08:21:28.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.488+0000 7fe505600640 1 -- 172.21.15.69:0/1643863306 wait complete. 2024-04-24T08:21:28.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.490+0000 7fe505600640 1 Processor -- start 2024-04-24T08:21:28.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.490+0000 7fe505600640 1 -- start start 2024-04-24T08:21:28.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.491+0000 7fe505600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe500073db0 0x7fe500161d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.491+0000 7fe505600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5000b3260 0x7fe500164f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.491+0000 7fe505600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5000cd3d0 0x7fe500165470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.491+0000 7fe505600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5000ce470 con 0x7fe500073db0 2024-04-24T08:21:28.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.491+0000 7fe505600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe5000d0050 con 0x7fe5000cd3d0 2024-04-24T08:21:28.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.491+0000 7fe505600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe5000ce5f0 con 0x7fe5000b3260 2024-04-24T08:21:28.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.491+0000 7fe4fea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe500073db0 0x7fe500161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.491+0000 7fe4fea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe500073db0 0x7fe500161d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55034/0 (socket says 172.21.15.69:55034) 2024-04-24T08:21:28.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.491+0000 7fe4fea00640 1 -- 172.21.15.69:0/3484407481 learned_addr learned my addr 172.21.15.69:0/3484407481 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:28.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4ff400640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5000cd3d0 0x7fe500165470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4fea00640 1 -- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5000b3260 msgr2=0x7fe500164f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4fea00640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5000b3260 0x7fe500164f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4fea00640 1 -- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5000cd3d0 msgr2=0x7fe500165470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4fea00640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5000cd3d0 0x7fe500165470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4fea00640 1 -- 172.21.15.69:0/3484407481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe500163620 con 0x7fe500073db0 2024-04-24T08:21:28.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4ff400640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5000cd3d0 0x7fe500165470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:28.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4fea00640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe500073db0 0x7fe500161d00 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000c400 tx=0x7fe4f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:28.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4f7400640 1 -- 172.21.15.69:0/3484407481 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00052d0 con 0x7fe500073db0 2024-04-24T08:21:28.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4f7400640 1 -- 172.21.15.69:0/3484407481 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f0005490 con 0x7fe500073db0 2024-04-24T08:21:28.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe4f7400640 1 -- 172.21.15.69:0/3484407481 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0007260 con 0x7fe500073db0 2024-04-24T08:21:28.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe505600640 1 -- 172.21.15.69:0/3484407481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe500163900 con 0x7fe500073db0 2024-04-24T08:21:28.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.492+0000 7fe505600640 1 -- 172.21.15.69:0/3484407481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe500163ec0 con 0x7fe500073db0 2024-04-24T08:21:28.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.493+0000 7fe505600640 1 -- 172.21.15.69:0/3484407481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe500068a80 con 0x7fe500073db0 2024-04-24T08:21:28.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.494+0000 7fe4f7400640 1 -- 172.21.15.69:0/3484407481 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe4f0007400 con 0x7fe500073db0 2024-04-24T08:21:28.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.495+0000 7fe4f7400640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4dc06c920 0x7fe4dc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.495+0000 7fe4f7400640 1 -- 172.21.15.69:0/3484407481 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fe4f008b160 con 0x7fe500073db0 2024-04-24T08:21:28.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.496+0000 7fe4f6000640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4dc06c920 0x7fe4dc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.497+0000 7fe4f6000640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4dc06c920 0x7fe4dc06ede0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe500165c80 tx=0x7fe4e8007930 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:28.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.500+0000 7fe4f7400640 1 -- 172.21.15.69:0/3484407481 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe4f0059490 con 0x7fe500073db0 2024-04-24T08:21:28.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.704+0000 7fe505600640 1 -- 172.21.15.69:0/3484407481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe50006f530 con 0x7fe500073db0 2024-04-24T08:21:28.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.705+0000 7fe4f7400640 1 -- 172.21.15.69:0/3484407481 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) ==== 100+0+11 (secure 0 0 0) 0x7fe4f0058d90 con 0x7fe500073db0 2024-04-24T08:21:28.706 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-04-24T08:21:28.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.707+0000 7fe505600640 1 -- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4dc06c920 msgr2=0x7fe4dc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.707+0000 7fe505600640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4dc06c920 0x7fe4dc06ede0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fe500165c80 tx=0x7fe4e8007930 comp rx=0 tx=0).stop 2024-04-24T08:21:28.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.707+0000 7fe505600640 1 -- 172.21.15.69:0/3484407481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe500073db0 msgr2=0x7fe500161d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.707+0000 7fe505600640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe500073db0 0x7fe500161d00 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000c400 tx=0x7fe4f000c8c0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.708+0000 7fe505600640 1 -- 172.21.15.69:0/3484407481 shutdown_connections 2024-04-24T08:21:28.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.708+0000 7fe505600640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe4dc06c920 0x7fe4dc06ede0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.708+0000 7fe505600640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5000cd3d0 0x7fe500165470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.708+0000 7fe505600640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5000b3260 0x7fe500164f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.708+0000 7fe505600640 1 --2- 172.21.15.69:0/3484407481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe500073db0 0x7fe500161d00 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.708+0000 7fe505600640 1 -- 172.21.15.69:0/3484407481 >> 172.21.15.69:0/3484407481 conn(0x7fe5000d0350 msgr2=0x7fe5000ce920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:28.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.708+0000 7fe505600640 1 -- 172.21.15.69:0/3484407481 shutdown_connections 2024-04-24T08:21:28.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.708+0000 7fe505600640 1 -- 172.21.15.69:0/3484407481 wait complete. 2024-04-24T08:21:28.727 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.920+0000 7fd0fd600640 1 Processor -- start 2024-04-24T08:21:28.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.920+0000 7fd0fd600640 1 -- start start 2024-04-24T08:21:28.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.921+0000 7fd0fd600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0f8162300 0x7fd0f8143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.921+0000 7fd0fd600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd0f8072230 0x7fd0f8143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0fd600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0f815f540 con 0x7fd0f8071e50 2024-04-24T08:21:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0fd600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f815f820 con 0x7fd0f8162300 2024-04-24T08:21:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0fd600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f815fb00 con 0x7fd0f8072230 2024-04-24T08:21:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0f7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0f8162300 0x7fd0f8143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0f7400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fd0f8162300 0x7fd0f8143690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52884/0 (socket says 172.21.15.69:52884) 2024-04-24T08:21:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0f7e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd0f8071e50 0x7fd0f815ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49862/0 (socket says 172.21.15.69:49862) 2024-04-24T08:21:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0f7400640 1 -- 172.21.15.69:0/1697399558 learned_addr learned my addr 172.21.15.69:0/1697399558 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0f7400640 1 -- 172.21.15.69:0/1697399558 >> v2:172.21.15.121:3300/0 conn(0x7fd0f8072230 msgr2=0x7fd0f8143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0f7400640 1 --2- 172.21.15.69:0/1697399558 >> v2:172.21.15.121:3300/0 conn(0x7fd0f8072230 0x7fd0f8143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0f7400640 1 -- 172.21.15.69:0/1697399558 >> v1:172.21.15.69:6789/0 conn(0x7fd0f8071e50 legacy=0x7fd0f815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.922+0000 7fd0f7400640 1 -- 172.21.15.69:0/1697399558 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f806dda0 con 0x7fd0f8162300 2024-04-24T08:21:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.924+0000 7fd0f7400640 1 --2- 172.21.15.69:0/1697399558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8162300 0x7fd0f8143690 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8141de0 tx=0x7fd0e402a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45dcb31a217361c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.924+0000 7fd0f6a00640 1 -- 172.21.15.69:0/1697399558 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4037070 con 0x7fd0f8162300 2024-04-24T08:21:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.924+0000 7fd0f6a00640 1 -- 172.21.15.69:0/1697399558 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e4032440 con 0x7fd0f8162300 2024-04-24T08:21:28.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.924+0000 7fd0f6a00640 1 -- 172.21.15.69:0/1697399558 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4032810 con 0x7fd0f8162300 2024-04-24T08:21:28.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.925+0000 7fd0fd600640 1 -- 172.21.15.69:0/1697399558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8162300 msgr2=0x7fd0f8143690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.925+0000 7fd0fd600640 1 --2- 172.21.15.69:0/1697399558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8162300 0x7fd0f8143690 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8141de0 tx=0x7fd0e402a280 comp rx=0 tx=0).stop 2024-04-24T08:21:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.925+0000 7fd0fd600640 1 -- 172.21.15.69:0/1697399558 shutdown_connections 2024-04-24T08:21:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.925+0000 7fd0fd600640 1 --2- 172.21.15.69:0/1697399558 >> v2:172.21.15.121:3300/0 conn(0x7fd0f8072230 0x7fd0f8143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.925+0000 7fd0fd600640 1 --2- 172.21.15.69:0/1697399558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8162300 0x7fd0f8143690 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.925+0000 7fd0fd600640 1 -- 172.21.15.69:0/1697399558 >> 172.21.15.69:0/1697399558 conn(0x7fd0f80d0160 msgr2=0x7fd0f80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.925+0000 7fd0fd600640 1 -- 172.21.15.69:0/1697399558 shutdown_connections 2024-04-24T08:21:28.927 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.925+0000 7fd0fd600640 1 -- 172.21.15.69:0/1697399558 wait complete. 2024-04-24T08:21:28.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.927+0000 7fd0fd600640 1 Processor -- start 2024-04-24T08:21:28.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.927+0000 7fd0fd600640 1 -- start start 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0fd600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0f8071e50 0x7fd0f814a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0fd600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8072230 0x7fd0f80ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0fd600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0f8162300 0x7fd0f80ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0fd600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f8144110 con 0x7fd0f8072230 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0fd600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f8144290 con 0x7fd0f8162300 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0fd600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f8144410 con 0x7fd0f8071e50 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0f7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8072230 0x7fd0f80ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0f7400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8072230 0x7fd0f80ae120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52888/0 (socket says 172.21.15.69:52888) 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0f7400640 1 -- 172.21.15.69:0/345082733 learned_addr learned my addr 172.21.15.69:0/345082733 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0f7400640 1 -- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0f8071e50 msgr2=0x7fd0f814a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0f7400640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0f8071e50 0x7fd0f814a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0f7400640 1 -- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0f8162300 msgr2=0x7fd0f80ae660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0f7400640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0f8162300 0x7fd0f80ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:28.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.928+0000 7fd0f7400640 1 -- 172.21.15.69:0/345082733 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f814a550 con 0x7fd0f8072230 2024-04-24T08:21:28.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.929+0000 7fd0f7400640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8072230 0x7fd0f80ae120 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4000c00 tx=0x7fd0e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:28.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.929+0000 7fd0efe00640 1 -- 172.21.15.69:0/345082733 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4041020 con 0x7fd0f8072230 2024-04-24T08:21:28.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.929+0000 7fd0efe00640 1 -- 172.21.15.69:0/345082733 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e4030970 con 0x7fd0f8072230 2024-04-24T08:21:28.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.929+0000 7fd0efe00640 1 -- 172.21.15.69:0/345082733 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4037040 con 0x7fd0f8072230 2024-04-24T08:21:28.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.929+0000 7fd0fd600640 1 -- 172.21.15.69:0/345082733 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f81e8140 con 0x7fd0f8072230 2024-04-24T08:21:28.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.929+0000 7fd0fd600640 1 -- 172.21.15.69:0/345082733 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f81e8700 con 0x7fd0f8072230 2024-04-24T08:21:28.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.930+0000 7fd0fd600640 1 -- 172.21.15.69:0/345082733 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f8075d50 con 0x7fd0f8072230 2024-04-24T08:21:28.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.931+0000 7fd0efe00640 1 -- 172.21.15.69:0/345082733 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd0e4044020 con 0x7fd0f8072230 2024-04-24T08:21:28.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.932+0000 7fd0efe00640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0d806c8f0 0x7fd0d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:28.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.932+0000 7fd0efe00640 1 -- 172.21.15.69:0/345082733 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fd0e40ac360 con 0x7fd0f8072230 2024-04-24T08:21:28.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.933+0000 7fd0f7e00640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0d806c8f0 0x7fd0d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:28.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.934+0000 7fd0f7e00640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0d806c8f0 0x7fd0d806edb0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8000f60 tx=0x7fd0e8007a00 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:28.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:28.937+0000 7fd0efe00640 1 -- 172.21.15.69:0/345082733 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0e404b070 con 0x7fd0f8072230 2024-04-24T08:21:29.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.177+0000 7fd0fd600640 1 -- 172.21.15.69:0/345082733 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0f80d0d90 con 0x7fd0f8072230 2024-04-24T08:21:29.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.179+0000 7fd0efe00640 1 -- 172.21.15.69:0/345082733 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5186 (secure 0 0 0) 0x7fd0f814a550 con 0x7fd0f8072230 2024-04-24T08:21:29.180 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:29.180 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:21:17.653434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9825},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9825":{"gid":9825,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/99442307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":99442307},{"type":"v1","addr":"172.21.15.69:6835","nonce":99442307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9825,"qdb_cluster":[9825]},"id":8}]} 2024-04-24T08:21:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 112 2024-04-24T08:21:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 -- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0d806c8f0 msgr2=0x7fd0d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0d806c8f0 0x7fd0d806edb0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8000f60 tx=0x7fd0e8007a00 comp rx=0 tx=0).stop 2024-04-24T08:21:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 -- 172.21.15.69:0/345082733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8072230 msgr2=0x7fd0f80ae120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8072230 0x7fd0f80ae120 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4000c00 tx=0x7fd0e40046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 -- 172.21.15.69:0/345082733 shutdown_connections 2024-04-24T08:21:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd0d806c8f0 0x7fd0d806edb0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0f8072230 0x7fd0f80ae120 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0f8162300 0x7fd0f80ae660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 --2- 172.21.15.69:0/345082733 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0f8071e50 0x7fd0f814a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 -- 172.21.15.69:0/345082733 >> 172.21.15.69:0/345082733 conn(0x7fd0f80d0160 msgr2=0x7fd0f80b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 -- 172.21.15.69:0/345082733 shutdown_connections 2024-04-24T08:21:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.181+0000 7fd0fd600640 1 -- 172.21.15.69:0/345082733 wait complete. 2024-04-24T08:21:29.203 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:29.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.404+0000 7f241ee00640 1 Processor -- start 2024-04-24T08:21:29.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.404+0000 7f241ee00640 1 -- start start 2024-04-24T08:21:29.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.406+0000 7f241ee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418162300 0x7f2418143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.406+0000 7f241ee00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2418072230 0x7f2418143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.406+0000 7f241ee00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f241815f540 con 0x7f2418071e50 2024-04-24T08:21:29.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.406+0000 7f241ee00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f241815f820 con 0x7f2418072230 2024-04-24T08:21:29.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.406+0000 7f241ee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f241815fb00 con 0x7f2418162300 2024-04-24T08:21:29.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.406+0000 7f241da00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f2418071e50 0x7f241815ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49868/0 (socket says 172.21.15.69:49868) 2024-04-24T08:21:29.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.406+0000 7f241da00640 1 -- 172.21.15.69:0/308164392 learned_addr learned my addr 172.21.15.69:0/308164392 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:29.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.406+0000 7f241d000640 1 --2- 172.21.15.69:0/308164392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418162300 0x7f2418143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.406+0000 7f241e400640 1 --2- 172.21.15.69:0/308164392 >> v2:172.21.15.121:3300/0 conn(0x7f2418072230 0x7f2418143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2846620873 0 0) 0x7f241815f540 con 0x7f2418071e50 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2400003520 con 0x7f2418071e50 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2523942013 0 0) 0x7f2400003520 con 0x7f2418071e50 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418162300 msgr2=0x7f2418143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 --2- 172.21.15.69:0/308164392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418162300 0x7f2418143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 >> v2:172.21.15.121:3300/0 conn(0x7f2418072230 msgr2=0x7f2418143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 --2- 172.21.15.69:0/308164392 >> v2:172.21.15.121:3300/0 conn(0x7f2418072230 0x7f2418143bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241806dda0 con 0x7f2418071e50 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f240c001670 con 0x7f2418071e50 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 >> v1:172.21.15.69:6789/0 conn(0x7f2418071e50 legacy=0x7f241815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 --2- 172.21.15.69:0/308164392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418162300 0x7f24000039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 --2- 172.21.15.69:0/308164392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 0x7f2400004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 --2- 172.21.15.69:0/308164392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2400004d40 0x7f24000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.410 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f241815f820 con 0x7f2400003f30 2024-04-24T08:21:29.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f241815fb00 con 0x7f2400004d40 2024-04-24T08:21:29.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f240000b9c0 con 0x7f2418162300 2024-04-24T08:21:29.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.407+0000 7f241d000640 1 --2- 172.21.15.69:0/308164392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 0x7f2400004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.408+0000 7f241d000640 1 -- 172.21.15.69:0/308164392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418162300 msgr2=0x7f24000039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:29.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.408+0000 7f241d000640 1 --2- 172.21.15.69:0/308164392 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418162300 0x7f24000039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.408+0000 7f241d000640 1 -- 172.21.15.69:0/308164392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2400004d40 msgr2=0x7f24000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:29.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.408+0000 7f241d000640 1 --2- 172.21.15.69:0/308164392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2400004d40 0x7f24000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.408+0000 7f241d000640 1 -- 172.21.15.69:0/308164392 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2400003520 con 0x7f2400003f30 2024-04-24T08:21:29.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.409+0000 7f241d000640 1 --2- 172.21.15.69:0/308164392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 0x7f2400004800 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f2404009ac0 tx=0x7f2400005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bef2a955615607 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:29.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.409+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2404036070 con 0x7f2400003f30 2024-04-24T08:21:29.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.409+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2404031400 con 0x7f2400003f30 2024-04-24T08:21:29.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.409+0000 7f2417e00640 1 -- 172.21.15.69:0/308164392 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2404009690 con 0x7f2400003f30 2024-04-24T08:21:29.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.410+0000 7f241ee00640 1 -- 172.21.15.69:0/308164392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 msgr2=0x7f2400004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.410+0000 7f241ee00640 1 --2- 172.21.15.69:0/308164392 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 0x7f2400004800 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f2404009ac0 tx=0x7f2400005830 comp rx=0 tx=0).stop 2024-04-24T08:21:29.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.410+0000 7f241e400640 1 -- 172.21.15.69:0/308164392 reap_dead start 2024-04-24T08:21:29.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.410+0000 7f241ee00640 1 -- 172.21.15.69:0/308164392 shutdown_connections 2024-04-24T08:21:29.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.410+0000 7f241ee00640 1 -- 172.21.15.69:0/308164392 >> 172.21.15.69:0/308164392 conn(0x7f24180d0160 msgr2=0x7f24180d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:29.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.410+0000 7f241ee00640 1 -- 172.21.15.69:0/308164392 shutdown_connections 2024-04-24T08:21:29.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.410+0000 7f241ee00640 1 -- 172.21.15.69:0/308164392 wait complete. 2024-04-24T08:21:29.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.412+0000 7f241ee00640 1 Processor -- start 2024-04-24T08:21:29.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.412+0000 7f241ee00640 1 -- start start 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.413+0000 7f241ee00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 0x7f24181596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.413+0000 7f241ee00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418159d80 0x7f241815a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.413+0000 7f241ee00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24181422a0 0x7f2418142720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.413+0000 7f241ee00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2418143690 con 0x7f2400003f30 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.413+0000 7f241ee00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2418143810 con 0x7f24181422a0 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.413+0000 7f241ee00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2418143990 con 0x7f2418159d80 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.413+0000 7f241da00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 0x7f24181596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.413+0000 7f241da00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 0x7f24181596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52918/0 (socket says 172.21.15.69:52918) 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.413+0000 7f241da00640 1 -- 172.21.15.69:0/421634943 learned_addr learned my addr 172.21.15.69:0/421634943 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f241da00640 1 -- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418159d80 msgr2=0x7f241815a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f241da00640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418159d80 0x7f241815a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f241da00640 1 -- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24181422a0 msgr2=0x7f2418142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f241da00640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24181422a0 0x7f2418142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f241da00640 1 -- 172.21.15.69:0/421634943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f241815a840 con 0x7f2400003f30 2024-04-24T08:21:29.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f241da00640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 0x7f24181596f0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f240c00c8a0 tx=0x7f240c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:29.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f2415600640 1 -- 172.21.15.69:0/421634943 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c018070 con 0x7f2400003f30 2024-04-24T08:21:29.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f2415600640 1 -- 172.21.15.69:0/421634943 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f240c0074e0 con 0x7f2400003f30 2024-04-24T08:21:29.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f241ee00640 1 -- 172.21.15.69:0/421634943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f241815ab20 con 0x7f2400003f30 2024-04-24T08:21:29.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f2415600640 1 -- 172.21.15.69:0/421634943 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f240c004a00 con 0x7f2400003f30 2024-04-24T08:21:29.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.414+0000 7f241ee00640 1 -- 172.21.15.69:0/421634943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f241815b0e0 con 0x7f2400003f30 2024-04-24T08:21:29.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.415+0000 7f241ee00640 1 -- 172.21.15.69:0/421634943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2418142c60 con 0x7f2400003f30 2024-04-24T08:21:29.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.416+0000 7f2415600640 1 -- 172.21.15.69:0/421634943 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f240c01b070 con 0x7f2400003f30 2024-04-24T08:21:29.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.418+0000 7f2415600640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f23f806c8f0 0x7f23f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.418+0000 7f2415600640 1 -- 172.21.15.69:0/421634943 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f240c08b2e0 con 0x7f2400003f30 2024-04-24T08:21:29.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.418+0000 7f241d000640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f23f806c8f0 0x7f23f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.419+0000 7f241d000640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f23f806c8f0 0x7f23f806edb0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2404033000 tx=0x7f240402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:29.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.422+0000 7f2415600640 1 -- 172.21.15.69:0/421634943 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f240c0594f0 con 0x7f2400003f30 2024-04-24T08:21:29.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.622+0000 7f241ee00640 1 -- 172.21.15.69:0/421634943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2418075d50 con 0x7f2400003f30 2024-04-24T08:21:29.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.625+0000 7f2415600640 1 -- 172.21.15.69:0/421634943 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+17017 (secure 0 0 0) 0x7f240c058df0 con 0x7f2400003f30 2024-04-24T08:21:29.627 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:29.627 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":113,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:21:12.532402+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.37","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/2194021008":"2024-04-25T08:21:12.532354+0000","172.21.15.69:6834/2194021008":"2024-04-25T08:21:12.532354+0000","172.21.15.121:6836/1515102940":"2024-04-25T08:20:50.436176+0000","172.21.15.121:6834/1515102940":"2024-04-25T08:20:50.436176+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.627+0000 7f241ee00640 1 -- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f23f806c8f0 msgr2=0x7f23f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.627+0000 7f241ee00640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f23f806c8f0 0x7f23f806edb0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2404033000 tx=0x7f240402e000 comp rx=0 tx=0).stop 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.627+0000 7f241ee00640 1 -- 172.21.15.69:0/421634943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 msgr2=0x7f24181596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.627+0000 7f241ee00640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 0x7f24181596f0 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f240c00c8a0 tx=0x7f240c00cd60 comp rx=0 tx=0).stop 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.628+0000 7f241ee00640 1 -- 172.21.15.69:0/421634943 shutdown_connections 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.628+0000 7f241ee00640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f23f806c8f0 0x7f23f806edb0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.628+0000 7f241ee00640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24181422a0 0x7f2418142720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.628+0000 7f241ee00640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2418159d80 0x7f241815a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.628+0000 7f241ee00640 1 --2- 172.21.15.69:0/421634943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2400003f30 0x7f24181596f0 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.628+0000 7f241ee00640 1 -- 172.21.15.69:0/421634943 >> 172.21.15.69:0/421634943 conn(0x7f24180d0160 msgr2=0x7f24180b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.628+0000 7f241ee00640 1 -- 172.21.15.69:0/421634943 shutdown_connections 2024-04-24T08:21:29.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.628+0000 7f241ee00640 1 -- 172.21.15.69:0/421634943 wait complete. 2024-04-24T08:21:29.649 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:21:29.649 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:29.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.848+0000 7f2d10c00640 1 Processor -- start 2024-04-24T08:21:29.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.849+0000 7f2d10c00640 1 -- start start 2024-04-24T08:21:29.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.850+0000 7f2d10c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d0c06f730 0x7f2d0c0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.850+0000 7f2d10c00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d0c0b3320 0x7f2d0c071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.850+0000 7f2d10c00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d0c0b4370 con 0x7f2d0c0722b0 2024-04-24T08:21:29.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.850+0000 7f2d10c00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0c072710 con 0x7f2d0c0b3320 2024-04-24T08:21:29.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.850+0000 7f2d10c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0c0b4f80 con 0x7f2d0c06f730 2024-04-24T08:21:29.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.850+0000 7f2d01400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d0c0b3320 0x7f2d0c071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.850+0000 7f2d01400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f2d0c0b3320 0x7f2d0c071d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52920/0 (socket says 172.21.15.69:52920) 2024-04-24T08:21:29.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.850+0000 7f2d01400640 1 -- 172.21.15.69:0/3229124192 learned_addr learned my addr 172.21.15.69:0/3229124192 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:29.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.851+0000 7f2d01400640 1 -- 172.21.15.69:0/3229124192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d0c06f730 msgr2=0x7f2d0c0d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:29.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.851+0000 7f2d09e00640 1 --2- 172.21.15.69:0/3229124192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d0c06f730 0x7f2d0c0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.851+0000 7f2d01400640 1 --2- 172.21.15.69:0/3229124192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d0c06f730 0x7f2d0c0d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.851+0000 7f2d01400640 1 -- 172.21.15.69:0/3229124192 >> v1:172.21.15.121:6789/0 conn(0x7f2d0c0722b0 legacy=0x7f2d0c06dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:29.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.851+0000 7f2d01400640 1 -- 172.21.15.69:0/3229124192 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0c0b4650 con 0x7f2d0c0b3320 2024-04-24T08:21:29.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.852+0000 7f2d01400640 1 --2- 172.21.15.69:0/3229124192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0b3320 0x7f2d0c071d70 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4009870 tx=0x7f2cf402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada856d3d71a549a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.852+0000 7f2d09400640 1 -- 172.21.15.69:0/3229124192 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4037070 con 0x7f2d0c0b3320 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.852+0000 7f2d09400640 1 -- 172.21.15.69:0/3229124192 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf4032440 con 0x7f2d0c0b3320 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.852+0000 7f2d09400640 1 -- 172.21.15.69:0/3229124192 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4032810 con 0x7f2d0c0b3320 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.853+0000 7f2d10c00640 1 -- 172.21.15.69:0/3229124192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0b3320 msgr2=0x7f2d0c071d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.853+0000 7f2d10c00640 1 --2- 172.21.15.69:0/3229124192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0b3320 0x7f2d0c071d70 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4009870 tx=0x7f2cf402a210 comp rx=0 tx=0).stop 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.853+0000 7f2d10c00640 1 -- 172.21.15.69:0/3229124192 shutdown_connections 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.853+0000 7f2d10c00640 1 --2- 172.21.15.69:0/3229124192 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0b3320 0x7f2d0c071d70 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.853+0000 7f2d10c00640 1 --2- 172.21.15.69:0/3229124192 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d0c06f730 0x7f2d0c0d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.853+0000 7f2d10c00640 1 -- 172.21.15.69:0/3229124192 >> 172.21.15.69:0/3229124192 conn(0x7f2d0c067d90 msgr2=0x7f2d0c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.853+0000 7f2d10c00640 1 -- 172.21.15.69:0/3229124192 shutdown_connections 2024-04-24T08:21:29.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.853+0000 7f2d10c00640 1 -- 172.21.15.69:0/3229124192 wait complete. 2024-04-24T08:21:29.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.855+0000 7f2d10c00640 1 Processor -- start 2024-04-24T08:21:29.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.855+0000 7f2d10c00640 1 -- start start 2024-04-24T08:21:29.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.856+0000 7f2d10c00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d0c06f730 0x7f2d0c1e61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.856+0000 7f2d10c00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0722b0 0x7f2d0c156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.856+0000 7f2d10c00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d0c0b3320 0x7f2d0c1f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.856+0000 7f2d10c00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0c06ee60 con 0x7f2d0c0722b0 2024-04-24T08:21:29.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.856+0000 7f2d10c00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0c0b4c10 con 0x7f2d0c06f730 2024-04-24T08:21:29.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.856+0000 7f2d10c00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0c0d08b0 con 0x7f2d0c0b3320 2024-04-24T08:21:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.856+0000 7f2d01400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0722b0 0x7f2d0c156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.856+0000 7f2d01400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0722b0 0x7f2d0c156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52936/0 (socket says 172.21.15.69:52936) 2024-04-24T08:21:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.856+0000 7f2d01400640 1 -- 172.21.15.69:0/2083712849 learned_addr learned my addr 172.21.15.69:0/2083712849 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.857+0000 7f2d09e00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d0c06f730 0x7f2d0c1e61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.857+0000 7f2d01400640 1 -- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d0c0b3320 msgr2=0x7f2d0c1f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.857+0000 7f2d0a800640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d0c0b3320 0x7f2d0c1f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.857+0000 7f2d01400640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d0c0b3320 0x7f2d0c1f46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.857+0000 7f2d01400640 1 -- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d0c06f730 msgr2=0x7f2d0c1e61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.857+0000 7f2d01400640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d0c06f730 0x7f2d0c1e61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.857+0000 7f2d01400640 1 -- 172.21.15.69:0/2083712849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0c15b040 con 0x7f2d0c0722b0 2024-04-24T08:21:29.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.857+0000 7f2d09e00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d0c06f730 0x7f2d0c1e61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:29.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.857+0000 7f2d01400640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0722b0 0x7f2d0c156280 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40097c0 tx=0x7f2cf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:29.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.857+0000 7f2d02a00640 1 -- 172.21.15.69:0/2083712849 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4041020 con 0x7f2d0c0722b0 2024-04-24T08:21:29.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.858+0000 7f2d02a00640 1 -- 172.21.15.69:0/2083712849 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf4030970 con 0x7f2d0c0722b0 2024-04-24T08:21:29.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.858+0000 7f2d10c00640 1 -- 172.21.15.69:0/2083712849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d0c1f4fc0 con 0x7f2d0c0722b0 2024-04-24T08:21:29.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.858+0000 7f2d02a00640 1 -- 172.21.15.69:0/2083712849 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4037040 con 0x7f2d0c0722b0 2024-04-24T08:21:29.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.858+0000 7f2d10c00640 1 -- 172.21.15.69:0/2083712849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d0c1f5580 con 0x7f2d0c0722b0 2024-04-24T08:21:29.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.859+0000 7f2d10c00640 1 -- 172.21.15.69:0/2083712849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d0c068a80 con 0x7f2d0c0722b0 2024-04-24T08:21:29.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.860+0000 7f2d02a00640 1 -- 172.21.15.69:0/2083712849 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2cf4044020 con 0x7f2d0c0722b0 2024-04-24T08:21:29.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.861+0000 7f2d02a00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ce806c8f0 0x7f2ce806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:29.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.861+0000 7f2d02a00640 1 -- 172.21.15.69:0/2083712849 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f2cf40ac310 con 0x7f2d0c0722b0 2024-04-24T08:21:29.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.861+0000 7f2d09e00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ce806c8f0 0x7f2ce806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:29.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.863+0000 7f2d09e00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ce806c8f0 0x7f2ce806edb0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc008890 tx=0x7f2cfc008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:29.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:29.865+0000 7f2d02a00640 1 -- 172.21.15.69:0/2083712849 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2cf404b070 con 0x7f2d0c0722b0 2024-04-24T08:21:30.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.105+0000 7f2d10c00640 1 -- 172.21.15.69:0/2083712849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d0c0cbb30 con 0x7f2d0c0722b0 2024-04-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.106+0000 7f2d02a00640 1 -- 172.21.15.69:0/2083712849 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5186 (secure 0 0 0) 0x7f2cf407a5f0 con 0x7f2d0c0722b0 2024-04-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:21:17.653434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9825},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9825":{"gid":9825,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/99442307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":99442307},{"type":"v1","addr":"172.21.15.69:6835","nonce":99442307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9825,"qdb_cluster":[9825]},"id":8}]} 2024-04-24T08:21:30.108 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 112 2024-04-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 -- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ce806c8f0 msgr2=0x7f2ce806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ce806c8f0 0x7f2ce806edb0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc008890 tx=0x7f2cfc008040 comp rx=0 tx=0).stop 2024-04-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 -- 172.21.15.69:0/2083712849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0722b0 msgr2=0x7f2d0c156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0722b0 0x7f2d0c156280 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40097c0 tx=0x7f2cf40046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 -- 172.21.15.69:0/2083712849 shutdown_connections 2024-04-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f2ce806c8f0 0x7f2ce806edb0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d0c0722b0 0x7f2d0c156280 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2d0c0b3320 0x7f2d0c1f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 --2- 172.21.15.69:0/2083712849 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2d0c06f730 0x7f2d0c1e61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 -- 172.21.15.69:0/2083712849 >> 172.21.15.69:0/2083712849 conn(0x7f2d0c067d90 msgr2=0x7f2d0c15a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:30.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 -- 172.21.15.69:0/2083712849 shutdown_connections 2024-04-24T08:21:30.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.109+0000 7f2d10c00640 1 -- 172.21.15.69:0/2083712849 wait complete. 2024-04-24T08:21:30.130 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:30.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.331+0000 7f382e000640 1 Processor -- start 2024-04-24T08:21:30.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.331+0000 7f382e000640 1 -- start start 2024-04-24T08:21:30.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.332+0000 7f382e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38280d0300 0x7f382806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.332+0000 7f382e000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38280cf3d0 con 0x7f3828145fe0 2024-04-24T08:21:30.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.332+0000 7f382e000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38280cf6b0 con 0x7f3828145a90 2024-04-24T08:21:30.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.332+0000 7f382e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f38280cf990 con 0x7f38280d0300 2024-04-24T08:21:30.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.333+0000 7f3827e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f3828145fe0 0x7f3828160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49884/0 (socket says 172.21.15.69:49884) 2024-04-24T08:21:30.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.333+0000 7f3827e00640 1 -- 172.21.15.69:0/3148793451 learned_addr learned my addr 172.21.15.69:0/3148793451 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:30.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.333+0000 7f3827400640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38280d0300 0x7f382806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.333+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 548323215 0 0) 0x7f38280cf3d0 con 0x7f3828145fe0 2024-04-24T08:21:30.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.333+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f380c003520 con 0x7f3828145fe0 2024-04-24T08:21:30.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1466306394 0 0) 0x7f380c003520 con 0x7f3828145fe0 2024-04-24T08:21:30.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38280d0300 msgr2=0x7f382806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:30.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38280d0300 0x7f382806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 >> v1:172.21.15.121:6789/0 conn(0x7f3828145a90 legacy=0x7f38280d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38280cfc70 con 0x7f3828145fe0 2024-04-24T08:21:30.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f3818001670 con 0x7f3828145fe0 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 >> v1:172.21.15.69:6789/0 conn(0x7f3828145fe0 legacy=0x7f3828160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38280d0300 0x7f380c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c003f60 0x7f380c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c004d40 0x7f380c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f38280cf990 con 0x7f380c004d40 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f380c00baa0 con 0x7f380c003f60 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f380c00bca0 con 0x7f38280d0300 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3827e00640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c004d40 0x7f380c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3827400640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38280d0300 0x7f380c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3827e00640 1 -- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38280d0300 msgr2=0x7f380c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3827e00640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38280d0300 0x7f380c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3827e00640 1 -- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c003f60 msgr2=0x7f380c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3827e00640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f380c003f60 0x7f380c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.334+0000 7f3827e00640 1 -- 172.21.15.69:0/3148793451 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380c003520 con 0x7f380c004d40 2024-04-24T08:21:30.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.335+0000 7f3827400640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f38280d0300 0x7f380c0039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.335+0000 7f3827e00640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c004d40 0x7f380c0051c0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f381800c240 tx=0x7f380c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc11bff5f293bf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.336+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3818007d10 con 0x7f380c004d40 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.336+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3818005ad0 con 0x7f380c004d40 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.336+0000 7f3826000640 1 -- 172.21.15.69:0/3148793451 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3818007070 con 0x7f380c004d40 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.336+0000 7f382e000640 1 -- 172.21.15.69:0/3148793451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c004d40 msgr2=0x7f380c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.336+0000 7f382e000640 1 --2- 172.21.15.69:0/3148793451 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f380c004d40 0x7f380c0051c0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f381800c240 tx=0x7f380c005830 comp rx=0 tx=0).stop 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.336+0000 7f3827e00640 1 -- 172.21.15.69:0/3148793451 reap_dead start 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.336+0000 7f382e000640 1 -- 172.21.15.69:0/3148793451 shutdown_connections 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.336+0000 7f382e000640 1 -- 172.21.15.69:0/3148793451 >> 172.21.15.69:0/3148793451 conn(0x7f3828067d90 msgr2=0x7f382806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.337+0000 7f382e000640 1 -- 172.21.15.69:0/3148793451 shutdown_connections 2024-04-24T08:21:30.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.337+0000 7f382e000640 1 -- 172.21.15.69:0/3148793451 wait complete. 2024-04-24T08:21:30.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.338+0000 7f382e000640 1 Processor -- start 2024-04-24T08:21:30.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.338+0000 7f382e000640 1 -- start start 2024-04-24T08:21:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.339+0000 7f382e000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f380c004d40 0x7f382815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.339+0000 7f382e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38280cc2d0 0x7f38280ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.339+0000 7f382e000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38281440c0 0x7f38280c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.339+0000 7f382e000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f38281458f0 con 0x7f38280cc2d0 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.339+0000 7f382e000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f3828145770 con 0x7f38281440c0 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.339+0000 7f382e000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f38280cdc30 con 0x7f380c004d40 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3826a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38280cc2d0 0x7f38280ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3826a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38280cc2d0 0x7f38280ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52964/0 (socket says 172.21.15.69:52964) 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3826a00640 1 -- 172.21.15.69:0/2150064297 learned_addr learned my addr 172.21.15.69:0/2150064297 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3827400640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f380c004d40 0x7f382815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3826a00640 1 -- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f380c004d40 msgr2=0x7f382815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3826a00640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f380c004d40 0x7f382815c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3827e00640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38281440c0 0x7f38280c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3826a00640 1 -- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38281440c0 msgr2=0x7f38280c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3826a00640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38281440c0 0x7f38280c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3826a00640 1 -- 172.21.15.69:0/2150064297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38280cb230 con 0x7f38280cc2d0 2024-04-24T08:21:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.340+0000 7f3826a00640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38280cc2d0 0x7f38280ca8e0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f381000c480 tx=0x7f381000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.341+0000 7f381f400640 1 -- 172.21.15.69:0/2150064297 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38100052d0 con 0x7f38280cc2d0 2024-04-24T08:21:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.341+0000 7f381f400640 1 -- 172.21.15.69:0/2150064297 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3810005470 con 0x7f38280cc2d0 2024-04-24T08:21:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.341+0000 7f381f400640 1 -- 172.21.15.69:0/2150064297 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810012870 con 0x7f38280cc2d0 2024-04-24T08:21:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.341+0000 7f382e000640 1 -- 172.21.15.69:0/2150064297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38280cb510 con 0x7f38280cc2d0 2024-04-24T08:21:30.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.341+0000 7f382e000640 1 -- 172.21.15.69:0/2150064297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38280cbad0 con 0x7f38280cc2d0 2024-04-24T08:21:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.342+0000 7f382e000640 1 -- 172.21.15.69:0/2150064297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3828068a80 con 0x7f38280cc2d0 2024-04-24T08:21:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.343+0000 7f381f400640 1 -- 172.21.15.69:0/2150064297 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3810012a10 con 0x7f38280cc2d0 2024-04-24T08:21:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.344+0000 7f381f400640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f380406c920 0x7f380406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.344+0000 7f381f400640 1 -- 172.21.15.69:0/2150064297 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7f381008b900 con 0x7f38280cc2d0 2024-04-24T08:21:30.346 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.345+0000 7f3827400640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f380406c920 0x7f380406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.347 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.346+0000 7f3827400640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f380406c920 0x7f380406ede0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3828144990 tx=0x7f382000d040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:30.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.348+0000 7f381f400640 1 -- 172.21.15.69:0/2150064297 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3810059c30 con 0x7f38280cc2d0 2024-04-24T08:21:30.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.588+0000 7f382e000640 1 -- 172.21.15.69:0/2150064297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38280b5180 con 0x7f38280cc2d0 2024-04-24T08:21:30.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.589+0000 7f381f400640 1 -- 172.21.15.69:0/2150064297 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+5186 (secure 0 0 0) 0x7f3810059530 con 0x7f38280cc2d0 2024-04-24T08:21:30.591 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:30.591 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T08:20:09.204175+0000","modified":"2024-04-24T08:21:17.653434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":113,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":9825},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9825":{"gid":9825,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":4,"addr":"172.21.15.69:6835/99442307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":99442307},{"type":"v1","addr":"172.21.15.69:6835","nonce":99442307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":9825,"qdb_cluster":[9825]},"id":8}]} 2024-04-24T08:21:30.591 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 112 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 -- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f380406c920 msgr2=0x7f380406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f380406c920 0x7f380406ede0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3828144990 tx=0x7f382000d040 comp rx=0 tx=0).stop 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 -- 172.21.15.69:0/2150064297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38280cc2d0 msgr2=0x7f38280ca8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38280cc2d0 0x7f38280ca8e0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f381000c480 tx=0x7f381000c940 comp rx=0 tx=0).stop 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 -- 172.21.15.69:0/2150064297 shutdown_connections 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f380406c920 0x7f380406ede0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38280cc2d0 0x7f38280ca8e0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38281440c0 0x7f38280c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 --2- 172.21.15.69:0/2150064297 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f380c004d40 0x7f382815c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 -- 172.21.15.69:0/2150064297 >> 172.21.15.69:0/2150064297 conn(0x7f3828067d90 msgr2=0x7f38280b25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 -- 172.21.15.69:0/2150064297 shutdown_connections 2024-04-24T08:21:30.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.592+0000 7f382e000640 1 -- 172.21.15.69:0/2150064297 wait complete. 2024-04-24T08:21:30.613 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:30.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.810+0000 7fe174600640 1 Processor -- start 2024-04-24T08:21:30.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.810+0000 7fe174600640 1 -- start start 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.812+0000 7fe174600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe16c15ea10 0x7fe16c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.812+0000 7fe174600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe16c0af9a0 0x7fe16c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.812+0000 7fe174600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe16c0adab0 con 0x7fe16c1631a0 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.812+0000 7fe174600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe16c0add50 con 0x7fe16c0af9a0 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.812+0000 7fe174600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe16c0ae030 con 0x7fe16c15ea10 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.813+0000 7fe172600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe16c0af9a0 0x7fe16c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.813+0000 7fe172600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe16c0af9a0 0x7fe16c06dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52972/0 (socket says 172.21.15.69:52972) 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.813+0000 7fe172600640 1 -- 172.21.15.69:0/1985211529 learned_addr learned my addr 172.21.15.69:0/1985211529 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.813+0000 7fe171200640 1 --1- 172.21.15.69:0/1985211529 >> v1:172.21.15.69:6789/0 conn(0x7fe16c1631a0 0x7fe16c0af290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49898/0 (socket says 172.21.15.69:49898) 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.813+0000 7fe172600640 1 -- 172.21.15.69:0/1985211529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe16c15ea10 msgr2=0x7fe16c0d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:30.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.813+0000 7fe171c00640 1 --2- 172.21.15.69:0/1985211529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe16c15ea10 0x7fe16c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.813+0000 7fe172600640 1 --2- 172.21.15.69:0/1985211529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe16c15ea10 0x7fe16c0d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.813+0000 7fe172600640 1 -- 172.21.15.69:0/1985211529 >> v1:172.21.15.69:6789/0 conn(0x7fe16c1631a0 legacy=0x7fe16c0af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.813+0000 7fe172600640 1 -- 172.21.15.69:0/1985211529 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe16c0ae310 con 0x7fe16c0af9a0 2024-04-24T08:21:30.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.813+0000 7fe16be00640 1 -- 172.21.15.69:0/1985211529 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464339489 0 0) 0x7fe16c0adab0 con 0x7fe16c1631a0 2024-04-24T08:21:30.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.814+0000 7fe172600640 1 --2- 172.21.15.69:0/1985211529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c0af9a0 0x7fe16c06dff0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fe164009e50 tx=0x7fe16402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aab3a45b452c25f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:30.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe16be00640 1 -- 172.21.15.69:0/1985211529 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164036070 con 0x7fe16c0af9a0 2024-04-24T08:21:30.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe16be00640 1 -- 172.21.15.69:0/1985211529 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe164031600 con 0x7fe16c0af9a0 2024-04-24T08:21:30.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe16be00640 1 -- 172.21.15.69:0/1985211529 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1640319d0 con 0x7fe16c0af9a0 2024-04-24T08:21:30.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe174600640 1 -- 172.21.15.69:0/1985211529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c0af9a0 msgr2=0x7fe16c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe174600640 1 --2- 172.21.15.69:0/1985211529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c0af9a0 0x7fe16c06dff0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7fe164009e50 tx=0x7fe16402a730 comp rx=0 tx=0).stop 2024-04-24T08:21:30.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe174600640 1 -- 172.21.15.69:0/1985211529 shutdown_connections 2024-04-24T08:21:30.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe174600640 1 --2- 172.21.15.69:0/1985211529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c0af9a0 0x7fe16c06dff0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe174600640 1 --2- 172.21.15.69:0/1985211529 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe16c15ea10 0x7fe16c0d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe174600640 1 -- 172.21.15.69:0/1985211529 >> 172.21.15.69:0/1985211529 conn(0x7fe16c067d90 msgr2=0x7fe16c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:30.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe174600640 1 -- 172.21.15.69:0/1985211529 shutdown_connections 2024-04-24T08:21:30.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.815+0000 7fe174600640 1 -- 172.21.15.69:0/1985211529 wait complete. 2024-04-24T08:21:30.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.817+0000 7fe174600640 1 Processor -- start 2024-04-24T08:21:30.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.817+0000 7fe174600640 1 -- start start 2024-04-24T08:21:30.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.818+0000 7fe174600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe16c0af9a0 0x7fe16c0aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.818+0000 7fe174600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c15ea10 0x7fe16c14e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.818+0000 7fe174600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c1631a0 0x7fe16c152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.818+0000 7fe174600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe16c0aedf0 con 0x7fe16c1631a0 2024-04-24T08:21:30.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.818+0000 7fe174600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe16c0ae8d0 con 0x7fe16c15ea10 2024-04-24T08:21:30.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.818+0000 7fe174600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe16c0d0d70 con 0x7fe16c0af9a0 2024-04-24T08:21:30.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe172600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c1631a0 0x7fe16c152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe172600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c1631a0 0x7fe16c152e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52986/0 (socket says 172.21.15.69:52986) 2024-04-24T08:21:30.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe172600640 1 -- 172.21.15.69:0/943360214 learned_addr learned my addr 172.21.15.69:0/943360214 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:30.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe171c00640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe16c0af9a0 0x7fe16c0aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe171200640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c15ea10 0x7fe16c14e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe172600640 1 -- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe16c0af9a0 msgr2=0x7fe16c0aff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe172600640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe16c0af9a0 0x7fe16c0aff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe172600640 1 -- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c15ea10 msgr2=0x7fe16c14e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:30.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe172600640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c15ea10 0x7fe16c14e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:30.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe172600640 1 -- 172.21.15.69:0/943360214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe16c0b0ec0 con 0x7fe16c1631a0 2024-04-24T08:21:30.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.819+0000 7fe172600640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c1631a0 0x7fe16c152e80 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fe164009e50 tx=0x7fe164004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:30.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.820+0000 7fe169600640 1 -- 172.21.15.69:0/943360214 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164041020 con 0x7fe16c1631a0 2024-04-24T08:21:30.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.820+0000 7fe169600640 1 -- 172.21.15.69:0/943360214 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe164037040 con 0x7fe16c1631a0 2024-04-24T08:21:30.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.820+0000 7fe169600640 1 -- 172.21.15.69:0/943360214 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164036050 con 0x7fe16c1631a0 2024-04-24T08:21:30.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.820+0000 7fe174600640 1 -- 172.21.15.69:0/943360214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe16c0b0620 con 0x7fe16c1631a0 2024-04-24T08:21:30.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.820+0000 7fe174600640 1 -- 172.21.15.69:0/943360214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe16c0b0be0 con 0x7fe16c1631a0 2024-04-24T08:21:30.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.821+0000 7fe174600640 1 -- 172.21.15.69:0/943360214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe16c068a80 con 0x7fe16c1631a0 2024-04-24T08:21:30.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.822+0000 7fe169600640 1 -- 172.21.15.69:0/943360214 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe164007bf0 con 0x7fe16c1631a0 2024-04-24T08:21:30.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.823+0000 7fe169600640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe15006c8f0 0x7fe15006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:30.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.823+0000 7fe169600640 1 -- 172.21.15.69:0/943360214 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fe1640ac440 con 0x7fe16c1631a0 2024-04-24T08:21:30.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.824+0000 7fe171c00640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe15006c8f0 0x7fe15006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:30.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.825+0000 7fe171c00640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe15006c8f0 0x7fe15006edb0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fe16000a970 tx=0x7fe160008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:30.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:30.828+0000 7fe169600640 1 -- 172.21.15.69:0/943360214 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe16404b060 con 0x7fe16c1631a0 2024-04-24T08:21:31.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.029+0000 7fe174600640 1 -- 172.21.15.69:0/943360214 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe16c06e590 con 0x7fe16c1631a0 2024-04-24T08:21:31.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.031+0000 7fe169600640 1 -- 172.21.15.69:0/943360214 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) ==== 74+0+17017 (secure 0 0 0) 0x7fe16407a810 con 0x7fe16c1631a0 2024-04-24T08:21:31.033 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:31.033 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":113,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:21:12.532402+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:20:06.963817+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-04-24T08:20:08.190440+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.37","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6835/2194021008":"2024-04-25T08:21:12.532354+0000","172.21.15.69:6834/2194021008":"2024-04-25T08:21:12.532354+0000","172.21.15.121:6836/1515102940":"2024-04-25T08:20:50.436176+0000","172.21.15.121:6834/1515102940":"2024-04-25T08:20:50.436176+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 -- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe15006c8f0 msgr2=0x7fe15006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe15006c8f0 0x7fe15006edb0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fe16000a970 tx=0x7fe160008040 comp rx=0 tx=0).stop 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 -- 172.21.15.69:0/943360214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c1631a0 msgr2=0x7fe16c152e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c1631a0 0x7fe16c152e80 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fe164009e50 tx=0x7fe164004a40 comp rx=0 tx=0).stop 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 -- 172.21.15.69:0/943360214 shutdown_connections 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe15006c8f0 0x7fe15006edb0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe16c1631a0 0x7fe16c152e80 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe16c15ea10 0x7fe16c14e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 --2- 172.21.15.69:0/943360214 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe16c0af9a0 0x7fe16c0aff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 -- 172.21.15.69:0/943360214 >> 172.21.15.69:0/943360214 conn(0x7fe16c067d90 msgr2=0x7fe16c0d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 -- 172.21.15.69:0/943360214 shutdown_connections 2024-04-24T08:21:31.036 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.034+0000 7fe174600640 1 -- 172.21.15.69:0/943360214 wait complete. 2024-04-24T08:21:31.054 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:21:31.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.254+0000 7fcfff200640 1 Processor -- start 2024-04-24T08:21:31.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.255+0000 7fcfff200640 1 -- start start 2024-04-24T08:21:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.256+0000 7fcfff200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd0000723a0 0x7fd0000b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.256+0000 7fcfff200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 0x7fd0000b3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.256+0000 7fcfff200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd000163fb0 con 0x7fd00014c650 2024-04-24T08:21:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.256+0000 7fcfff200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd00006e130 con 0x7fd0000723a0 2024-04-24T08:21:31.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.256+0000 7fcfff200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd00006e410 con 0x7fd0001666e0 2024-04-24T08:21:31.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.257+0000 7fcffe800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 0x7fd0000b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.257+0000 7fcffe800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 0x7fd0000b3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:47866/0 (socket says 172.21.15.69:47866) 2024-04-24T08:21:31.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.257+0000 7fcffe800640 1 -- 172.21.15.69:0/3470166620 learned_addr learned my addr 172.21.15.69:0/3470166620 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:31.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.257+0000 7fcffd400640 1 --2- 172.21.15.69:0/3470166620 >> v2:172.21.15.121:3300/0 conn(0x7fd0000723a0 0x7fd0000b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.258+0000 7fcffe800640 1 -- 172.21.15.69:0/3470166620 >> v2:172.21.15.121:3300/0 conn(0x7fd0000723a0 msgr2=0x7fd0000b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.258+0000 7fcffe800640 1 --2- 172.21.15.69:0/3470166620 >> v2:172.21.15.121:3300/0 conn(0x7fd0000723a0 0x7fd0000b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.258+0000 7fcffe800640 1 -- 172.21.15.69:0/3470166620 >> v1:172.21.15.121:6789/0 conn(0x7fd00014c650 legacy=0x7fd000071c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.258+0000 7fcffe800640 1 -- 172.21.15.69:0/3470166620 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00006e6f0 con 0x7fd0001666e0 2024-04-24T08:21:31.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.259+0000 7fcffe800640 1 --2- 172.21.15.69:0/3470166620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 0x7fd0000b3630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009870 tx=0x7fcff802a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26be347b07d6c798 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcffca00640 1 -- 172.21.15.69:0/3470166620 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff8037070 con 0x7fd0001666e0 2024-04-24T08:21:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcffca00640 1 -- 172.21.15.69:0/3470166620 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcff802ad60 con 0x7fd0001666e0 2024-04-24T08:21:31.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcffca00640 1 -- 172.21.15.69:0/3470166620 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff8032660 con 0x7fd0001666e0 2024-04-24T08:21:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcfff200640 1 -- 172.21.15.69:0/3470166620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 msgr2=0x7fd0000b3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcfff200640 1 --2- 172.21.15.69:0/3470166620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 0x7fd0000b3630 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009870 tx=0x7fcff802a0d0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcfff200640 1 -- 172.21.15.69:0/3470166620 shutdown_connections 2024-04-24T08:21:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcfff200640 1 --2- 172.21.15.69:0/3470166620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 0x7fd0000b3630 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcfff200640 1 --2- 172.21.15.69:0/3470166620 >> v2:172.21.15.121:3300/0 conn(0x7fd0000723a0 0x7fd0000b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcfff200640 1 -- 172.21.15.69:0/3470166620 >> 172.21.15.69:0/3470166620 conn(0x7fd0000d0160 msgr2=0x7fd0000d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcfff200640 1 -- 172.21.15.69:0/3470166620 shutdown_connections 2024-04-24T08:21:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.260+0000 7fcfff200640 1 -- 172.21.15.69:0/3470166620 wait complete. 2024-04-24T08:21:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.262+0000 7fcfff200640 1 Processor -- start 2024-04-24T08:21:31.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.262+0000 7fcfff200640 1 -- start start 2024-04-24T08:21:31.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.263+0000 7fcfff200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0000723a0 0x7fd0001f8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.263+0000 7fcfff200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd00014c650 0x7fd0000cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.263+0000 7fcfff200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 0x7fd0000ad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.263+0000 7fcfff200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd00014ca30 con 0x7fd0000723a0 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.263+0000 7fcfff200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd00006ecb0 con 0x7fd00014c650 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.263+0000 7fcfff200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd000072830 con 0x7fd0001666e0 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.263+0000 7fcffde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0000723a0 0x7fd0001f8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.264+0000 7fcffde00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0000723a0 0x7fd0001f8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52992/0 (socket says 172.21.15.69:52992) 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.264+0000 7fcffde00640 1 -- 172.21.15.69:0/3573178481 learned_addr learned my addr 172.21.15.69:0/3573178481 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.264+0000 7fcffd400640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd00014c650 0x7fd0000cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.264+0000 7fcffe800640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 0x7fd0000ad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.264+0000 7fcffde00640 1 -- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 msgr2=0x7fd0000ad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.264+0000 7fcffde00640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 0x7fd0000ad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.264+0000 7fcffde00640 1 -- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd00014c650 msgr2=0x7fd0000cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.264+0000 7fcffde00640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd00014c650 0x7fd0000cd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.264+0000 7fcffde00640 1 -- 172.21.15.69:0/3573178481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0001f89a0 con 0x7fd0000723a0 2024-04-24T08:21:31.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.264+0000 7fcffde00640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0000723a0 0x7fd0001f8460 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fcff000caf0 tx=0x7fcff00020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:31.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.265+0000 7fcff6000640 1 -- 172.21.15.69:0/3573178481 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff0017070 con 0x7fd0000723a0 2024-04-24T08:21:31.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.265+0000 7fcff6000640 1 -- 172.21.15.69:0/3573178481 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcff0003b90 con 0x7fd0000723a0 2024-04-24T08:21:31.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.265+0000 7fcff6000640 1 -- 172.21.15.69:0/3573178481 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff000a910 con 0x7fd0000723a0 2024-04-24T08:21:31.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.265+0000 7fcfff200640 1 -- 172.21.15.69:0/3573178481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0000cdbc0 con 0x7fd0000723a0 2024-04-24T08:21:31.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.265+0000 7fcfff200640 1 -- 172.21.15.69:0/3573178481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0000ce180 con 0x7fd0000723a0 2024-04-24T08:21:31.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.266+0000 7fcfff200640 1 -- 172.21.15.69:0/3573178481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd000075d50 con 0x7fd0000723a0 2024-04-24T08:21:31.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.267+0000 7fcff6000640 1 -- 172.21.15.69:0/3573178481 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcff001a070 con 0x7fd0000723a0 2024-04-24T08:21:31.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.268+0000 7fcff6000640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 0x7fcfe006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.268+0000 7fcff6000640 1 -- 172.21.15.69:0/3573178481 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 5939+0+0 (secure 0 0 0) 0x7fcff008b420 con 0x7fd0000723a0 2024-04-24T08:21:31.270 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.268+0000 7fcffd400640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 0x7fcfe006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.270+0000 7fcffd400640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 0x7fcfe006edb0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd0000ae4c0 tx=0x7fcfec008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:31.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.273+0000 7fcff6000640 1 -- 172.21.15.69:0/3573178481 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcff0059750 con 0x7fd0000723a0 2024-04-24T08:21:31.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.510+0000 7fcfff200640 1 -- 172.21.15.69:0/3573178481 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd0000d0d90 con 0x7fd0000723a0 2024-04-24T08:21:31.513 INFO:tasks.ceph.mon.a.smithi069.stderr:2024-04-24T08:21:31.511+0000 7fa5d5400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:21:31.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.515+0000 7fcff6000640 1 -- 172.21.15.69:0/3573178481 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v113) ==== 161+0+0 (secure 0 0 0) 0x7fcff0059050 con 0x7fd0000723a0 2024-04-24T08:21:31.516 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 -- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 msgr2=0x7fcfe006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 0x7fcfe006edb0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd0000ae4c0 tx=0x7fcfec008040 comp rx=0 tx=0).stop 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 -- 172.21.15.69:0/3573178481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0000723a0 msgr2=0x7fd0001f8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0000723a0 0x7fd0001f8460 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fcff000caf0 tx=0x7fcff00020c0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 -- 172.21.15.69:0/3573178481 shutdown_connections 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcfe006c8f0 0x7fcfe006edb0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd0001666e0 0x7fd0000ad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd00014c650 0x7fd0000cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 --2- 172.21.15.69:0/3573178481 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd0000723a0 0x7fd0001f8460 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 -- 172.21.15.69:0/3573178481 >> 172.21.15.69:0/3573178481 conn(0x7fd0000d0160 msgr2=0x7fd000074b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 -- 172.21.15.69:0/3573178481 shutdown_connections 2024-04-24T08:21:31.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.518+0000 7fcfff200640 1 -- 172.21.15.69:0/3573178481 wait complete. 2024-04-24T08:21:31.538 INFO:tasks.ceph.mds.c.smithi069.stderr: -1497> 2024-04-24T08:21:18.615+0000 7f4367c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:21:31.541 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T08:21:31.604 INFO:tasks.ceph.mds.c.smithi069.stdout:starting mds.c at 2024-04-24T08:21:31.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.721+0000 7fb1eb400640 1 Processor -- start 2024-04-24T08:21:31.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.721+0000 7fb1eb400640 1 -- start start 2024-04-24T08:21:31.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.722+0000 7fb1eb400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb1e415c840 0x7fb1e40cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.722+0000 7fb1eb400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1e4160430 0x7fb1e4160810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.722+0000 7fb1eb400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1e40ced40 con 0x7fb1e40b3a70 2024-04-24T08:21:31.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.722+0000 7fb1eb400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb1e40ce4a0 con 0x7fb1e4160430 2024-04-24T08:21:31.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.722+0000 7fb1eb400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb1e40ce780 con 0x7fb1e415c840 2024-04-24T08:21:31.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.723+0000 7fb1e9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1e4160430 0x7fb1e4160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.723+0000 7fb1e9400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb1e4160430 0x7fb1e4160810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53022/0 (socket says 172.21.15.69:53022) 2024-04-24T08:21:31.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.723+0000 7fb1e9400640 1 -- 172.21.15.69:0/1077631455 learned_addr learned my addr 172.21.15.69:0/1077631455 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:31.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.723+0000 7fb1e9400640 1 -- 172.21.15.69:0/1077631455 >> v2:172.21.15.121:3300/0 conn(0x7fb1e415c840 msgr2=0x7fb1e40cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:31.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.723+0000 7fb1e9400640 1 --2- 172.21.15.69:0/1077631455 >> v2:172.21.15.121:3300/0 conn(0x7fb1e415c840 0x7fb1e40cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.723+0000 7fb1e9400640 1 -- 172.21.15.69:0/1077631455 >> v1:172.21.15.121:6789/0 conn(0x7fb1e40b3a70 legacy=0x7fb1e40d08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:31.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.723+0000 7fb1e9400640 1 -- 172.21.15.69:0/1077631455 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e40cea60 con 0x7fb1e4160430 2024-04-24T08:21:31.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.724+0000 7fb1e9400640 1 --2- 172.21.15.69:0/1077631455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 0x7fb1e4160810 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4073d80 tx=0x7fb1dc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd932fd85e345b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:31.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.725+0000 7fb1e3400640 1 -- 172.21.15.69:0/1077631455 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc037070 con 0x7fb1e4160430 2024-04-24T08:21:31.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.725+0000 7fb1e3400640 1 -- 172.21.15.69:0/1077631455 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1dc032440 con 0x7fb1e4160430 2024-04-24T08:21:31.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.725+0000 7fb1e3400640 1 -- 172.21.15.69:0/1077631455 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc032810 con 0x7fb1e4160430 2024-04-24T08:21:31.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.725+0000 7fb1eb400640 1 -- 172.21.15.69:0/1077631455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 msgr2=0x7fb1e4160810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.725+0000 7fb1eb400640 1 --2- 172.21.15.69:0/1077631455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 0x7fb1e4160810 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fb1e4073d80 tx=0x7fb1dc02a1a0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.725+0000 7fb1eb400640 1 -- 172.21.15.69:0/1077631455 shutdown_connections 2024-04-24T08:21:31.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.725+0000 7fb1eb400640 1 --2- 172.21.15.69:0/1077631455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 0x7fb1e4160810 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.725+0000 7fb1eb400640 1 --2- 172.21.15.69:0/1077631455 >> v2:172.21.15.121:3300/0 conn(0x7fb1e415c840 0x7fb1e40cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.725+0000 7fb1eb400640 1 -- 172.21.15.69:0/1077631455 >> 172.21.15.69:0/1077631455 conn(0x7fb1e40b2f50 msgr2=0x7fb1e40b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:31.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.726+0000 7fb1eb400640 1 -- 172.21.15.69:0/1077631455 shutdown_connections 2024-04-24T08:21:31.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.726+0000 7fb1eb400640 1 -- 172.21.15.69:0/1077631455 wait complete. 2024-04-24T08:21:31.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.727+0000 7fb1eb400640 1 Processor -- start 2024-04-24T08:21:31.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.727+0000 7fb1eb400640 1 -- start start 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.728+0000 7fb1eb400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e40b3a70 0x7fb1e40c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.728+0000 7fb1eb400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e415c840 0x7fb1e4153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.728+0000 7fb1eb400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 0x7fb1e40cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.728+0000 7fb1eb400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e415cc20 con 0x7fb1e4160430 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.728+0000 7fb1eb400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb1e406c6d0 con 0x7fb1e40b3a70 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.728+0000 7fb1eb400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb1e40d0fe0 con 0x7fb1e415c840 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.729+0000 7fb1e9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 0x7fb1e40cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.729+0000 7fb1e9400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 0x7fb1e40cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53038/0 (socket says 172.21.15.69:53038) 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.729+0000 7fb1e9400640 1 -- 172.21.15.69:0/2543271206 learned_addr learned my addr 172.21.15.69:0/2543271206 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.729+0000 7fb1e3e00640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e415c840 0x7fb1e4153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.729+0000 7fb1e8a00640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e40b3a70 0x7fb1e40c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.729+0000 7fb1e9400640 1 -- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e415c840 msgr2=0x7fb1e4153370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.729+0000 7fb1e9400640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e415c840 0x7fb1e4153370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.729+0000 7fb1e9400640 1 -- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e40b3a70 msgr2=0x7fb1e40c9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:31.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.729+0000 7fb1e9400640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e40b3a70 0x7fb1e40c9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:31.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.729+0000 7fb1e9400640 1 -- 172.21.15.69:0/2543271206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1e40c5410 con 0x7fb1e4160430 2024-04-24T08:21:31.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.730+0000 7fb1e9400640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 0x7fb1e40cad00 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc02a1e0 tx=0x7fb1dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.732+0000 7fb1e0c00640 1 -- 172.21.15.69:0/2543271206 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc041070 con 0x7fb1e4160430 2024-04-24T08:21:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.732+0000 7fb1e0c00640 1 -- 172.21.15.69:0/2543271206 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1dc03c890 con 0x7fb1e4160430 2024-04-24T08:21:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.733+0000 7fb1eb400640 1 -- 172.21.15.69:0/2543271206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1e40c9460 con 0x7fb1e4160430 2024-04-24T08:21:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.733+0000 7fb1e0c00640 1 -- 172.21.15.69:0/2543271206 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc037040 con 0x7fb1e4160430 2024-04-24T08:21:31.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.733+0000 7fb1eb400640 1 -- 172.21.15.69:0/2543271206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1e40c9a20 con 0x7fb1e4160430 2024-04-24T08:21:31.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.734+0000 7fb1eb400640 1 -- 172.21.15.69:0/2543271206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1e4069190 con 0x7fb1e4160430 2024-04-24T08:21:31.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.735+0000 7fb1e0c00640 1 -- 172.21.15.69:0/2543271206 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb1dc044070 con 0x7fb1e4160430 2024-04-24T08:21:31.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.735+0000 7fb1e0c00640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c006c8f0 0x7fb1c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:31.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.736+0000 7fb1e0c00640 1 -- 172.21.15.69:0/2543271206 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6025+0+0 (secure 0 0 0) 0x7fb1dc0ac800 con 0x7fb1e4160430 2024-04-24T08:21:31.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.736+0000 7fb1e8a00640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c006c8f0 0x7fb1c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:31.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.737+0000 7fb1e8a00640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c006c8f0 0x7fb1c006edb0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4007dc0 tx=0x7fb1d40072f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:31.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.740+0000 7fb1e0c00640 1 -- 172.21.15.69:0/2543271206 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb1dc07ac00 con 0x7fb1e4160430 2024-04-24T08:21:31.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:31.983+0000 7fb1eb400640 1 -- 172.21.15.69:0/2543271206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb1e4073080 con 0x7fb1e4160430 2024-04-24T08:21:32.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.519+0000 7fb1e0c00640 1 -- 172.21.15.69:0/2543271206 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v114) ==== 104+0+0 (secure 0 0 0) 0x7fb1dc07a500 con 0x7fb1e4160430 2024-04-24T08:21:32.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.523+0000 7fb1eb400640 1 -- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c006c8f0 msgr2=0x7fb1c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:32.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.523+0000 7fb1eb400640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c006c8f0 0x7fb1c006edb0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4007dc0 tx=0x7fb1d40072f0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.523+0000 7fb1eb400640 1 -- 172.21.15.69:0/2543271206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 msgr2=0x7fb1e40cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:32.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.523+0000 7fb1eb400640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 0x7fb1e40cad00 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc02a1e0 tx=0x7fb1dc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.523+0000 7fb1eb400640 1 -- 172.21.15.69:0/2543271206 shutdown_connections 2024-04-24T08:21:32.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.523+0000 7fb1eb400640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb1c006c8f0 0x7fb1c006edb0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.523+0000 7fb1eb400640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb1e4160430 0x7fb1e40cad00 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.523+0000 7fb1eb400640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb1e415c840 0x7fb1e4153370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.523+0000 7fb1eb400640 1 --2- 172.21.15.69:0/2543271206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e40b3a70 0x7fb1e40c9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.523+0000 7fb1eb400640 1 -- 172.21.15.69:0/2543271206 >> 172.21.15.69:0/2543271206 conn(0x7fb1e40b2f50 msgr2=0x7fb1e40d08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:32.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.524+0000 7fb1eb400640 1 -- 172.21.15.69:0/2543271206 shutdown_connections 2024-04-24T08:21:32.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.524+0000 7fb1eb400640 1 -- 172.21.15.69:0/2543271206 wait complete. 2024-04-24T08:21:32.547 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T08:21:32.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.745+0000 7fc6f9600640 1 Processor -- start 2024-04-24T08:21:32.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.746+0000 7fc6f9600640 1 -- start start 2024-04-24T08:21:32.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.747+0000 7fc6f9600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc6f40cda80 0x7fc6f40cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:32.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.747+0000 7fc6f9600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6f40ceea0 0x7fc6f40ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:32.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.747+0000 7fc6f9600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6f40b3cc0 con 0x7fc6f40cf350 2024-04-24T08:21:32.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.747+0000 7fc6f9600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fc6f415bce0 con 0x7fc6f40ceea0 2024-04-24T08:21:32.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.747+0000 7fc6f9600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fc6f415bfc0 con 0x7fc6f40cda80 2024-04-24T08:21:32.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.747+0000 7fc6f3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6f40ceea0 0x7fc6f40ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:32.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.748+0000 7fc6f3400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fc6f40ceea0 0x7fc6f40ce3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53042/0 (socket says 172.21.15.69:53042) 2024-04-24T08:21:32.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.748+0000 7fc6f3400640 1 -- 172.21.15.69:0/4157230817 learned_addr learned my addr 172.21.15.69:0/4157230817 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:32.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.748+0000 7fc6f3e00640 1 --2- 172.21.15.69:0/4157230817 >> v2:172.21.15.121:3300/0 conn(0x7fc6f40cda80 0x7fc6f40cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:32.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.748+0000 7fc6f3400640 1 -- 172.21.15.69:0/4157230817 >> v2:172.21.15.121:3300/0 conn(0x7fc6f40cda80 msgr2=0x7fc6f40cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:32.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.748+0000 7fc6f3400640 1 --2- 172.21.15.69:0/4157230817 >> v2:172.21.15.121:3300/0 conn(0x7fc6f40cda80 0x7fc6f40cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.748+0000 7fc6f3400640 1 -- 172.21.15.69:0/4157230817 >> v1:172.21.15.69:6789/0 conn(0x7fc6f40cf350 legacy=0x7fc6f40cf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:32.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.748+0000 7fc6f3400640 1 -- 172.21.15.69:0/4157230817 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f415c2a0 con 0x7fc6f40ceea0 2024-04-24T08:21:32.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.749+0000 7fc6f3400640 1 --2- 172.21.15.69:0/4157230817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 0x7fc6f40ce3a0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40cee20 tx=0x7fc6dc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663240f641c65a2e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:32.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.749+0000 7fc6f2a00640 1 -- 172.21.15.69:0/4157230817 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc037070 con 0x7fc6f40ceea0 2024-04-24T08:21:32.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.749+0000 7fc6f2a00640 1 -- 172.21.15.69:0/4157230817 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6dc032440 con 0x7fc6f40ceea0 2024-04-24T08:21:32.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.749+0000 7fc6f2a00640 1 -- 172.21.15.69:0/4157230817 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc032810 con 0x7fc6f40ceea0 2024-04-24T08:21:32.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.750+0000 7fc6f9600640 1 -- 172.21.15.69:0/4157230817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 msgr2=0x7fc6f40ce3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:32.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.750+0000 7fc6f9600640 1 --2- 172.21.15.69:0/4157230817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 0x7fc6f40ce3a0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40cee20 tx=0x7fc6dc02a270 comp rx=0 tx=0).stop 2024-04-24T08:21:32.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.750+0000 7fc6f9600640 1 -- 172.21.15.69:0/4157230817 shutdown_connections 2024-04-24T08:21:32.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.750+0000 7fc6f9600640 1 --2- 172.21.15.69:0/4157230817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 0x7fc6f40ce3a0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.750+0000 7fc6f9600640 1 --2- 172.21.15.69:0/4157230817 >> v2:172.21.15.121:3300/0 conn(0x7fc6f40cda80 0x7fc6f40cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.750+0000 7fc6f9600640 1 -- 172.21.15.69:0/4157230817 >> 172.21.15.69:0/4157230817 conn(0x7fc6f409a9d0 msgr2=0x7fc6f40b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:32.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.750+0000 7fc6f9600640 1 -- 172.21.15.69:0/4157230817 shutdown_connections 2024-04-24T08:21:32.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.750+0000 7fc6f9600640 1 -- 172.21.15.69:0/4157230817 wait complete. 2024-04-24T08:21:32.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.752+0000 7fc6f9600640 1 Processor -- start 2024-04-24T08:21:32.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.752+0000 7fc6f9600640 1 -- start start 2024-04-24T08:21:32.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.753+0000 7fc6f9600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f40cda80 0x7fc6f40b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:32.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.753+0000 7fc6f9600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 0x7fc6f4150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:32.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.753+0000 7fc6f9600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6f40cf350 0x7fc6f4150c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:32.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.753+0000 7fc6f9600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f415f0f0 con 0x7fc6f40ceea0 2024-04-24T08:21:32.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.753+0000 7fc6f9600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f415f400 con 0x7fc6f40cda80 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.753+0000 7fc6f9600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f415caa0 con 0x7fc6f40cf350 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.753+0000 7fc6f3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 0x7fc6f4150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.753+0000 7fc6f3400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 0x7fc6f4150710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53044/0 (socket says 172.21.15.69:53044) 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.753+0000 7fc6f3400640 1 -- 172.21.15.69:0/4023447284 learned_addr learned my addr 172.21.15.69:0/4023447284 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6f3400640 1 -- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6f40cf350 msgr2=0x7fc6f4150c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6f3400640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6f40cf350 0x7fc6f4150c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6f3400640 1 -- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f40cda80 msgr2=0x7fc6f40b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6f3e00640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f40cda80 0x7fc6f40b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6f3400640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f40cda80 0x7fc6f40b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6f3400640 1 -- 172.21.15.69:0/4023447284 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f40b5310 con 0x7fc6f40ceea0 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6f3e00640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f40cda80 0x7fc6f40b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:32.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6f3400640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 0x7fc6f4150710 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc000c00 tx=0x7fc6dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:32.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6ebe00640 1 -- 172.21.15.69:0/4023447284 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc037040 con 0x7fc6f40ceea0 2024-04-24T08:21:32.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6ebe00640 1 -- 172.21.15.69:0/4023447284 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6dc042070 con 0x7fc6f40ceea0 2024-04-24T08:21:32.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6ebe00640 1 -- 172.21.15.69:0/4023447284 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc030bb0 con 0x7fc6f40ceea0 2024-04-24T08:21:32.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6f9600640 1 -- 172.21.15.69:0/4023447284 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6f40b6ef0 con 0x7fc6f40ceea0 2024-04-24T08:21:32.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.754+0000 7fc6f9600640 1 -- 172.21.15.69:0/4023447284 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6f415ad70 con 0x7fc6f40ceea0 2024-04-24T08:21:32.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.756+0000 7fc6f9600640 1 -- 172.21.15.69:0/4023447284 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6f415a8a0 con 0x7fc6f40ceea0 2024-04-24T08:21:32.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.757+0000 7fc6ebe00640 1 -- 172.21.15.69:0/4023447284 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc6dc047460 con 0x7fc6f40ceea0 2024-04-24T08:21:32.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.757+0000 7fc6ebe00640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6d006c8f0 0x7fc6d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:32.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.758+0000 7fc6ebe00640 1 -- 172.21.15.69:0/4023447284 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 6025+0+0 (secure 0 0 0) 0x7fc6dc0ac8c0 con 0x7fc6f40ceea0 2024-04-24T08:21:32.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.758+0000 7fc6f3e00640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6d006c8f0 0x7fc6d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:32.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.759+0000 7fc6f3e00640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6d006c8f0 0x7fc6d006edb0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc6f41f7f30 tx=0x7fc6e40074c0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:32.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.762+0000 7fc6ebe00640 1 -- 172.21.15.69:0/4023447284 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6dc045360 con 0x7fc6f40ceea0 2024-04-24T08:21:32.965 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:32.964+0000 7fc6f9600640 1 -- 172.21.15.69:0/4023447284 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc6f4069190 con 0x7fc6f40ceea0 2024-04-24T08:21:33.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.755+0000 7fc6ebe00640 1 -- 172.21.15.69:0/4023447284 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v115) ==== 181+0+0 (secure 0 0 0) 0x7fc6dc07aba0 con 0x7fc6f40ceea0 2024-04-24T08:21:33.757 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' removed 2024-04-24T08:21:33.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.760+0000 7fc6f9600640 1 -- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6d006c8f0 msgr2=0x7fc6d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.761+0000 7fc6f9600640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6d006c8f0 0x7fc6d006edb0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc6f41f7f30 tx=0x7fc6e40074c0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.761+0000 7fc6f9600640 1 -- 172.21.15.69:0/4023447284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 msgr2=0x7fc6f4150710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.761+0000 7fc6f9600640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 0x7fc6f4150710 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc000c00 tx=0x7fc6dc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.761+0000 7fc6f9600640 1 -- 172.21.15.69:0/4023447284 shutdown_connections 2024-04-24T08:21:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.761+0000 7fc6f9600640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fc6d006c8f0 0x7fc6d006edb0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.761+0000 7fc6f9600640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc6f40ceea0 0x7fc6f4150710 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.761+0000 7fc6f9600640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc6f40cf350 0x7fc6f4150c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.761+0000 7fc6f9600640 1 --2- 172.21.15.69:0/4023447284 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc6f40cda80 0x7fc6f40b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.761+0000 7fc6f9600640 1 -- 172.21.15.69:0/4023447284 >> 172.21.15.69:0/4023447284 conn(0x7fc6f409a9d0 msgr2=0x7fc6f415cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.762+0000 7fc6f9600640 1 -- 172.21.15.69:0/4023447284 shutdown_connections 2024-04-24T08:21:33.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.762+0000 7fc6f9600640 1 -- 172.21.15.69:0/4023447284 wait complete. 2024-04-24T08:21:33.785 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T08:21:33.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.985+0000 7f8203400640 1 Processor -- start 2024-04-24T08:21:33.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.986+0000 7f8203400640 1 -- start start 2024-04-24T08:21:33.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.987+0000 7f8203400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f81fc159210 0x7f81fc0cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:33.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.987+0000 7f8203400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81fc0cd940 0x7f81fc0ad240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:33.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.987+0000 7f8203400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81fc0b3300 con 0x7f81fc0b40c0 2024-04-24T08:21:33.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.987+0000 7f8203400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f81fc1595f0 con 0x7f81fc159210 2024-04-24T08:21:33.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.987+0000 7f8203400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc0b3c30 con 0x7f81fc0cd940 2024-04-24T08:21:33.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.987+0000 7f8200a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f81fc159210 0x7f81fc0cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:33.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.987+0000 7f8200a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f81fc159210 0x7f81fc0cdf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53056/0 (socket says 172.21.15.69:53056) 2024-04-24T08:21:33.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.987+0000 7f8200a00640 1 -- 172.21.15.69:0/1067965582 learned_addr learned my addr 172.21.15.69:0/1067965582 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:33.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.988+0000 7f8200a00640 1 -- 172.21.15.69:0/1067965582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81fc0cd940 msgr2=0x7f81fc0ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:33.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.988+0000 7f8200a00640 1 --2- 172.21.15.69:0/1067965582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81fc0cd940 0x7f81fc0ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.988+0000 7f8200a00640 1 -- 172.21.15.69:0/1067965582 >> v1:172.21.15.121:6789/0 conn(0x7f81fc0b40c0 legacy=0x7f81fc0cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:33.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.988+0000 7f8200a00640 1 -- 172.21.15.69:0/1067965582 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc0b35e0 con 0x7f81fc159210 2024-04-24T08:21:33.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.989+0000 7f8200a00640 1 --2- 172.21.15.69:0/1067965582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 0x7f81fc0cdf60 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0b4500 tx=0x7f81ec02c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58603926c910db86 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:33.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.989+0000 7f81fb400640 1 -- 172.21.15.69:0/1067965582 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ec038070 con 0x7f81fc159210 2024-04-24T08:21:33.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.989+0000 7f81fb400640 1 -- 172.21.15.69:0/1067965582 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81ec031710 con 0x7f81fc159210 2024-04-24T08:21:33.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.989+0000 7f81fb400640 1 -- 172.21.15.69:0/1067965582 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ec031ae0 con 0x7f81fc159210 2024-04-24T08:21:33.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.990+0000 7f8203400640 1 -- 172.21.15.69:0/1067965582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 msgr2=0x7f81fc0cdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:33.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.990+0000 7f8203400640 1 --2- 172.21.15.69:0/1067965582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 0x7f81fc0cdf60 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0b4500 tx=0x7f81ec02c410 comp rx=0 tx=0).stop 2024-04-24T08:21:33.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.990+0000 7f8203400640 1 -- 172.21.15.69:0/1067965582 shutdown_connections 2024-04-24T08:21:33.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.990+0000 7f8203400640 1 --2- 172.21.15.69:0/1067965582 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81fc0cd940 0x7f81fc0ad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.990+0000 7f8203400640 1 --2- 172.21.15.69:0/1067965582 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 0x7f81fc0cdf60 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.990+0000 7f8203400640 1 -- 172.21.15.69:0/1067965582 >> 172.21.15.69:0/1067965582 conn(0x7f81fc0b2380 msgr2=0x7f81fc0b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:33.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.990+0000 7f8203400640 1 -- 172.21.15.69:0/1067965582 shutdown_connections 2024-04-24T08:21:33.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.990+0000 7f8203400640 1 -- 172.21.15.69:0/1067965582 wait complete. 2024-04-24T08:21:33.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.992+0000 7f8203400640 1 Processor -- start 2024-04-24T08:21:33.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.992+0000 7f8203400640 1 -- start start 2024-04-24T08:21:33.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.993+0000 7f8203400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81fc0b40c0 0x7f81fc158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:33.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.993+0000 7f8203400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81fc0cd940 0x7f81fc0b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:33.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.993+0000 7f8203400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 0x7f81fc159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:33.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.993+0000 7f8203400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc15cb30 con 0x7f81fc159210 2024-04-24T08:21:33.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.993+0000 7f8203400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc15c9b0 con 0x7f81fc0cd940 2024-04-24T08:21:33.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.993+0000 7f8203400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc0ce9b0 con 0x7f81fc0b40c0 2024-04-24T08:21:33.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.993+0000 7f8201400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 0x7f81fc159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:33.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.993+0000 7f8201400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 0x7f81fc159e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53070/0 (socket says 172.21.15.69:53070) 2024-04-24T08:21:33.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.993+0000 7f8201400640 1 -- 172.21.15.69:0/887984953 learned_addr learned my addr 172.21.15.69:0/887984953 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:33.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f81fbe00640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81fc0cd940 0x7f81fc0b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:33.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f8200a00640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81fc0b40c0 0x7f81fc158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:33.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f8201400640 1 -- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81fc0b40c0 msgr2=0x7f81fc158390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:33.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f8201400640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81fc0b40c0 0x7f81fc158390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f8201400640 1 -- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81fc0cd940 msgr2=0x7f81fc0b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:33.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f8201400640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81fc0cd940 0x7f81fc0b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:33.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f8201400640 1 -- 172.21.15.69:0/887984953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc0b1740 con 0x7f81fc159210 2024-04-24T08:21:33.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f81fbe00640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81fc0cd940 0x7f81fc0b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:33.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f8201400640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 0x7f81fc159e70 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f81f400ba10 tx=0x7f81f400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:33.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f81f8c00640 1 -- 172.21.15.69:0/887984953 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4019070 con 0x7f81fc159210 2024-04-24T08:21:33.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f81f8c00640 1 -- 172.21.15.69:0/887984953 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f400cb40 con 0x7f81fc159210 2024-04-24T08:21:33.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f81f8c00640 1 -- 172.21.15.69:0/887984953 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4014440 con 0x7f81fc159210 2024-04-24T08:21:33.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f8203400640 1 -- 172.21.15.69:0/887984953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81fc0726a0 con 0x7f81fc159210 2024-04-24T08:21:33.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.994+0000 7f8203400640 1 -- 172.21.15.69:0/887984953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81fc0b0a00 con 0x7f81fc159210 2024-04-24T08:21:33.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.996+0000 7f8203400640 1 -- 172.21.15.69:0/887984953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81fc068a80 con 0x7f81fc159210 2024-04-24T08:21:33.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.997+0000 7f81f8c00640 1 -- 172.21.15.69:0/887984953 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f81f401c070 con 0x7f81fc159210 2024-04-24T08:21:33.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.997+0000 7f81f8c00640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81d806c8f0 0x7f81d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:33.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.997+0000 7f81f8c00640 1 -- 172.21.15.69:0/887984953 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 5587+0+0 (secure 0 0 0) 0x7f81f408b8d0 con 0x7f81fc159210 2024-04-24T08:21:33.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.998+0000 7f8200a00640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81d806c8f0 0x7f81d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:34.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:33.999+0000 7f8200a00640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81d806c8f0 0x7f81d806edb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f81ec035000 tx=0x7f81ec008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:34.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.002+0000 7f81f8c00640 1 -- 172.21.15.69:0/887984953 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81f4059c40 con 0x7f81fc159210 2024-04-24T08:21:34.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.206+0000 7f8203400640 1 -- 172.21.15.69:0/887984953 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f81fc0b4fd0 con 0x7f81fc159210 2024-04-24T08:21:34.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.759+0000 7f81f8c00640 1 -- 172.21.15.69:0/887984953 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v116) ==== 169+0+0 (secure 0 0 0) 0x7f81f4059540 con 0x7f81fc159210 2024-04-24T08:21:34.761 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' removed 2024-04-24T08:21:34.764 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 -- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81d806c8f0 msgr2=0x7f81d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:34.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81d806c8f0 0x7f81d806edb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f81ec035000 tx=0x7f81ec008030 comp rx=0 tx=0).stop 2024-04-24T08:21:34.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 -- 172.21.15.69:0/887984953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 msgr2=0x7f81fc159e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:34.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 0x7f81fc159e70 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f81f400ba10 tx=0x7f81f400bed0 comp rx=0 tx=0).stop 2024-04-24T08:21:34.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 -- 172.21.15.69:0/887984953 shutdown_connections 2024-04-24T08:21:34.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f81d806c8f0 0x7f81d806edb0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:34.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81fc159210 0x7f81fc159e70 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:34.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f81fc0cd940 0x7f81fc0b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:34.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 --2- 172.21.15.69:0/887984953 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f81fc0b40c0 0x7f81fc158390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:34.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 -- 172.21.15.69:0/887984953 >> 172.21.15.69:0/887984953 conn(0x7f81fc0b2380 msgr2=0x7f81fc15d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:34.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.763+0000 7f8203400640 1 -- 172.21.15.69:0/887984953 shutdown_connections 2024-04-24T08:21:34.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.764+0000 7f8203400640 1 -- 172.21.15.69:0/887984953 wait complete. 2024-04-24T08:21:34.794 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-24T08:21:34.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.991+0000 7f1a20e00640 1 Processor -- start 2024-04-24T08:21:34.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.991+0000 7f1a20e00640 1 -- start start 2024-04-24T08:21:34.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.992+0000 7f1a20e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1a1c0cbb10 0x7f1a1c0cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:34.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.992+0000 7f1a20e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a1c0cbf80 con 0x7f1a1c0b07a0 2024-04-24T08:21:34.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.992+0000 7f1a20e00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a1c06bb50 con 0x7f1a1c0d0850 2024-04-24T08:21:34.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.992+0000 7f1a20e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1a1c0cdc00 con 0x7f1a1c0cbb10 2024-04-24T08:21:34.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.992+0000 7f1a19400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f1a1c0b07a0 0x7f1a1c0ce110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49924/0 (socket says 172.21.15.69:49924) 2024-04-24T08:21:34.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.992+0000 7f1a19400640 1 -- 172.21.15.69:0/169852748 learned_addr learned my addr 172.21.15.69:0/169852748 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:34.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.992+0000 7f1a1a800640 1 --2- 172.21.15.69:0/169852748 >> v2:172.21.15.121:3300/0 conn(0x7f1a1c0cbb10 0x7f1a1c0cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:34.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.992+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 623731407 0 0) 0x7f1a1c0cbf80 con 0x7f1a1c0b07a0 2024-04-24T08:21:34.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a00003520 con 0x7f1a1c0b07a0 2024-04-24T08:21:34.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 337956705 0 0) 0x7f1a00003520 con 0x7f1a1c0b07a0 2024-04-24T08:21:34.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 >> v2:172.21.15.121:3300/0 conn(0x7f1a1c0cbb10 msgr2=0x7f1a1c0cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:34.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 --2- 172.21.15.69:0/169852748 >> v2:172.21.15.121:3300/0 conn(0x7f1a1c0cbb10 0x7f1a1c0cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:34.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 >> v1:172.21.15.121:6789/0 conn(0x7f1a1c0d0850 legacy=0x7f1a1c0d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:34.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1c06be30 con 0x7f1a1c0b07a0 2024-04-24T08:21:34.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f1a08001670 con 0x7f1a1c0b07a0 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 >> v1:172.21.15.69:6789/0 conn(0x7f1a1c0b07a0 legacy=0x7f1a1c0ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a000039b0 0x7f1a00003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a00004800 0x7f1a00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a0000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1c0cdc00 con 0x7f1a000052e0 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1a0000c1b0 con 0x7f1a00004800 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1a0000c3b0 con 0x7f1a000039b0 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.993+0000 7f1a1a800640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a0000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.994+0000 7f1a19400640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a00004800 0x7f1a00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.994+0000 7f1a19e00640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a000039b0 0x7f1a00003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.994+0000 7f1a1a800640 1 -- 172.21.15.69:0/169852748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a000039b0 msgr2=0x7f1a00003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:34.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.994+0000 7f1a1a800640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a000039b0 0x7f1a00003dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:34.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.994+0000 7f1a1a800640 1 -- 172.21.15.69:0/169852748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a00004800 msgr2=0x7f1a00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:34.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.994+0000 7f1a1a800640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a00004800 0x7f1a00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:34.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.994+0000 7f1a1a800640 1 -- 172.21.15.69:0/169852748 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a00003520 con 0x7f1a000052e0 2024-04-24T08:21:34.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.995+0000 7f1a1a800640 1 -- 172.21.15.69:0/169852748 reap_dead start 2024-04-24T08:21:34.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.995+0000 7f1a1a800640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a0000b9c0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009f90 tx=0x7f1a00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369372543a210535 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:34.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.995+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a14035020 con 0x7f1a000052e0 2024-04-24T08:21:34.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.995+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a14003b90 con 0x7f1a000052e0 2024-04-24T08:21:34.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.995+0000 7f1a18a00640 1 -- 172.21.15.69:0/169852748 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a14009c10 con 0x7f1a000052e0 2024-04-24T08:21:34.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.996+0000 7f1a20e00640 1 -- 172.21.15.69:0/169852748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 msgr2=0x7f1a0000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:34.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.996+0000 7f1a20e00640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a0000b9c0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009f90 tx=0x7f1a00005830 comp rx=0 tx=0).stop 2024-04-24T08:21:34.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.996+0000 7f1a20e00640 1 -- 172.21.15.69:0/169852748 shutdown_connections 2024-04-24T08:21:34.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.996+0000 7f1a20e00640 1 --2- 172.21.15.69:0/169852748 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a0000b9c0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:34.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.996+0000 7f1a20e00640 1 -- 172.21.15.69:0/169852748 >> 172.21.15.69:0/169852748 conn(0x7f1a1c0b3050 msgr2=0x7f1a1c0b3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:35.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.996+0000 7f1a20e00640 1 -- 172.21.15.69:0/169852748 shutdown_connections 2024-04-24T08:21:35.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.996+0000 7f1a20e00640 1 -- 172.21.15.69:0/169852748 wait complete. 2024-04-24T08:21:35.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.998+0000 7f1a20e00640 1 Processor -- start 2024-04-24T08:21:35.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.998+0000 7f1a20e00640 1 -- start start 2024-04-24T08:21:35.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.998+0000 7f1a20e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a1c0b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a20e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a1c0c97d0 0x7f1a1c0c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a20e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a1c155550 0x7f1a1c1559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a20e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1c06c6d0 con 0x7f1a000052e0 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a20e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1c0cf320 con 0x7f1a1c155550 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a20e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1c06c3f0 con 0x7f1a1c0c97d0 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a19e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a1c0b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a19e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a1c0b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53084/0 (socket says 172.21.15.69:53084) 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a19e00640 1 -- 172.21.15.69:0/1192675051 learned_addr learned my addr 172.21.15.69:0/1192675051 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a19400640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a1c0c97d0 0x7f1a1c0c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a19e00640 1 -- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a1c0c97d0 msgr2=0x7f1a1c0c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a1a800640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a1c155550 0x7f1a1c1559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:35.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a19e00640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a1c0c97d0 0x7f1a1c0c9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a19e00640 1 -- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a1c155550 msgr2=0x7f1a1c1559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a19e00640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a1c155550 0x7f1a1c1559d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:34.999+0000 7f1a19e00640 1 -- 172.21.15.69:0/1192675051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1c0b41c0 con 0x7f1a000052e0 2024-04-24T08:21:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.000+0000 7f1a19e00640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a1c0b2440 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00d570 tx=0x7f1a0c00da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.000+0000 7f1a12000640 1 -- 172.21.15.69:0/1192675051 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c003610 con 0x7f1a000052e0 2024-04-24T08:21:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.000+0000 7f1a12000640 1 -- 172.21.15.69:0/1192675051 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c017070 con 0x7f1a000052e0 2024-04-24T08:21:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.000+0000 7f1a12000640 1 -- 172.21.15.69:0/1192675051 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c0039f0 con 0x7f1a000052e0 2024-04-24T08:21:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.000+0000 7f1a20e00640 1 -- 172.21.15.69:0/1192675051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a1c155f10 con 0x7f1a000052e0 2024-04-24T08:21:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.000+0000 7f1a20e00640 1 -- 172.21.15.69:0/1192675051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a1c0b4480 con 0x7f1a000052e0 2024-04-24T08:21:35.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.001+0000 7f1a20e00640 1 -- 172.21.15.69:0/1192675051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a1c0b3fc0 con 0x7f1a000052e0 2024-04-24T08:21:35.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.002+0000 7f1a12000640 1 -- 172.21.15.69:0/1192675051 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a0c003b90 con 0x7f1a000052e0 2024-04-24T08:21:35.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.004+0000 7f1a12000640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19fc06c8f0 0x7f19fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:35.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.004+0000 7f1a12000640 1 -- 172.21.15.69:0/1192675051 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f1a0c08b780 con 0x7f1a000052e0 2024-04-24T08:21:35.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.004+0000 7f1a19400640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19fc06c8f0 0x7f19fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:35.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.005+0000 7f1a19400640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19fc06c8f0 0x7f19fc06edb0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f1a08001eb0 tx=0x7f1a08001d80 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:35.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.009+0000 7f1a12000640 1 -- 172.21.15.69:0/1192675051 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a0c059eb0 con 0x7f1a000052e0 2024-04-24T08:21:35.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.210+0000 7f1a20e00640 1 -- 172.21.15.69:0/1192675051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f1a1c068a80 con 0x7f1a000052e0 2024-04-24T08:21:35.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.216+0000 7f1a12000640 1 -- 172.21.15.69:0/1192675051 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) ==== 104+0+0 (secure 0 0 0) 0x7f1a0c0597b0 con 0x7f1a000052e0 2024-04-24T08:21:35.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.218+0000 7f1a20e00640 1 -- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19fc06c8f0 msgr2=0x7f19fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.218+0000 7f1a20e00640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19fc06c8f0 0x7f19fc06edb0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f1a08001eb0 tx=0x7f1a08001d80 comp rx=0 tx=0).stop 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.219+0000 7f1a20e00640 1 -- 172.21.15.69:0/1192675051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 msgr2=0x7f1a1c0b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.219+0000 7f1a20e00640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a1c0b2440 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00d570 tx=0x7f1a0c00da30 comp rx=0 tx=0).stop 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.219+0000 7f1a20e00640 1 -- 172.21.15.69:0/1192675051 shutdown_connections 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.219+0000 7f1a20e00640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f19fc06c8f0 0x7f19fc06edb0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.219+0000 7f1a20e00640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1a1c155550 0x7f1a1c1559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.219+0000 7f1a20e00640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1a1c0c97d0 0x7f1a1c0c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.219+0000 7f1a20e00640 1 --2- 172.21.15.69:0/1192675051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1a000052e0 0x7f1a1c0b2440 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.219+0000 7f1a20e00640 1 -- 172.21.15.69:0/1192675051 >> 172.21.15.69:0/1192675051 conn(0x7f1a1c0b3050 msgr2=0x7f1a1c0d1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.219+0000 7f1a20e00640 1 -- 172.21.15.69:0/1192675051 shutdown_connections 2024-04-24T08:21:35.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.219+0000 7f1a20e00640 1 -- 172.21.15.69:0/1192675051 wait complete. 2024-04-24T08:21:35.239 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-04-24T08:21:35.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.430+0000 7f1b53200640 1 Processor -- start 2024-04-24T08:21:35.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.430+0000 7f1b53200640 1 -- start start 2024-04-24T08:21:35.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b53200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1b541523f0 0x7f1b5406db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:35.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b53200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1b5415b980 0x7f1b5415be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:35.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b53200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b540aeac0 con 0x7f1b54072e20 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b53200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1b540aeda0 con 0x7f1b541523f0 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b53200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f1b540af080 con 0x7f1b5415b980 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b51e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1b541523f0 0x7f1b5406db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b51e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1b541523f0 0x7f1b5406db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53100/0 (socket says 172.21.15.69:53100) 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b51e00640 1 -- 172.21.15.69:0/2339969008 learned_addr learned my addr 172.21.15.69:0/2339969008 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b52800640 1 --2- 172.21.15.69:0/2339969008 >> v2:172.21.15.121:3300/0 conn(0x7f1b5415b980 0x7f1b5415be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b51e00640 1 -- 172.21.15.69:0/2339969008 >> v2:172.21.15.121:3300/0 conn(0x7f1b5415b980 msgr2=0x7f1b5415be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b51e00640 1 --2- 172.21.15.69:0/2339969008 >> v2:172.21.15.121:3300/0 conn(0x7f1b5415b980 0x7f1b5415be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b51e00640 1 -- 172.21.15.69:0/2339969008 >> v1:172.21.15.121:6789/0 conn(0x7f1b54072e20 legacy=0x7f1b54073260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b51e00640 1 -- 172.21.15.69:0/2339969008 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b540af360 con 0x7f1b541523f0 2024-04-24T08:21:35.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.432+0000 7f1b52800640 1 --2- 172.21.15.69:0/2339969008 >> v2:172.21.15.121:3300/0 conn(0x7f1b5415b980 0x7f1b5415be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.433+0000 7f1b51e00640 1 --2- 172.21.15.69:0/2339969008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b541523f0 0x7f1b5406db90 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f1b540ad970 tx=0x7f1b4402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10857d5795ba5464 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.433+0000 7f1b51400640 1 -- 172.21.15.69:0/2339969008 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44037070 con 0x7f1b541523f0 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.433+0000 7f1b51400640 1 -- 172.21.15.69:0/2339969008 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b44032440 con 0x7f1b541523f0 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.433+0000 7f1b51400640 1 -- 172.21.15.69:0/2339969008 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44032810 con 0x7f1b541523f0 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.434+0000 7f1b53200640 1 -- 172.21.15.69:0/2339969008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b541523f0 msgr2=0x7f1b5406db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.434+0000 7f1b53200640 1 --2- 172.21.15.69:0/2339969008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b541523f0 0x7f1b5406db90 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f1b540ad970 tx=0x7f1b4402a1c0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.434+0000 7f1b53200640 1 -- 172.21.15.69:0/2339969008 shutdown_connections 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.434+0000 7f1b53200640 1 --2- 172.21.15.69:0/2339969008 >> v2:172.21.15.121:3300/0 conn(0x7f1b5415b980 0x7f1b5415be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.434+0000 7f1b53200640 1 --2- 172.21.15.69:0/2339969008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b541523f0 0x7f1b5406db90 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.434+0000 7f1b53200640 1 -- 172.21.15.69:0/2339969008 >> 172.21.15.69:0/2339969008 conn(0x7f1b540d0160 msgr2=0x7f1b540d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.434+0000 7f1b53200640 1 -- 172.21.15.69:0/2339969008 shutdown_connections 2024-04-24T08:21:35.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.434+0000 7f1b53200640 1 -- 172.21.15.69:0/2339969008 wait complete. 2024-04-24T08:21:35.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.436+0000 7f1b53200640 1 Processor -- start 2024-04-24T08:21:35.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.436+0000 7f1b53200640 1 -- start start 2024-04-24T08:21:35.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.437+0000 7f1b53200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54072e20 0x7f1b54147800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:35.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.437+0000 7f1b53200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b541523f0 0x7f1b541f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:35.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.437+0000 7f1b53200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5415b980 0x7f1b54071be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:35.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.437+0000 7f1b53200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1b54073970 con 0x7f1b54072e20 2024-04-24T08:21:35.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.437+0000 7f1b53200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1b541527d0 con 0x7f1b541523f0 2024-04-24T08:21:35.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.437+0000 7f1b53200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1b54073af0 con 0x7f1b5415b980 2024-04-24T08:21:35.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.437+0000 7f1b51e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54072e20 0x7f1b54147800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:35.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.437+0000 7f1b51e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54072e20 0x7f1b54147800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53116/0 (socket says 172.21.15.69:53116) 2024-04-24T08:21:35.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.437+0000 7f1b51e00640 1 -- 172.21.15.69:0/1990706669 learned_addr learned my addr 172.21.15.69:0/1990706669 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:35.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b52800640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5415b980 0x7f1b54071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:35.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b51e00640 1 -- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5415b980 msgr2=0x7f1b54071be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:35.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b51e00640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5415b980 0x7f1b54071be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b51e00640 1 -- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b541523f0 msgr2=0x7f1b541f21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:35.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b51e00640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b541523f0 0x7f1b541f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b51e00640 1 -- 172.21.15.69:0/1990706669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b541f2b40 con 0x7f1b54072e20 2024-04-24T08:21:35.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b51e00640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54072e20 0x7f1b54147800 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f1b44000c00 tx=0x7f1b440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:35.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b4aa00640 1 -- 172.21.15.69:0/1990706669 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44037040 con 0x7f1b54072e20 2024-04-24T08:21:35.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b4aa00640 1 -- 172.21.15.69:0/1990706669 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b44004250 con 0x7f1b54072e20 2024-04-24T08:21:35.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b53200640 1 -- 172.21.15.69:0/1990706669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b541f2e20 con 0x7f1b54072e20 2024-04-24T08:21:35.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b4aa00640 1 -- 172.21.15.69:0/1990706669 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44030770 con 0x7f1b54072e20 2024-04-24T08:21:35.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.438+0000 7f1b53200640 1 -- 172.21.15.69:0/1990706669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b54152ad0 con 0x7f1b54072e20 2024-04-24T08:21:35.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.439+0000 7f1b53200640 1 -- 172.21.15.69:0/1990706669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b541f3340 con 0x7f1b54072e20 2024-04-24T08:21:35.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.440+0000 7f1b4aa00640 1 -- 172.21.15.69:0/1990706669 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1b44030910 con 0x7f1b54072e20 2024-04-24T08:21:35.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.441+0000 7f1b4aa00640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3006c8f0 0x7f1b3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:35.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.442+0000 7f1b4aa00640 1 -- 172.21.15.69:0/1990706669 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f1b440ac830 con 0x7f1b54072e20 2024-04-24T08:21:35.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.442+0000 7f1b49400640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3006c8f0 0x7f1b3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:35.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.443+0000 7f1b49400640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3006c8f0 0x7f1b3006edb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1b54158480 tx=0x7f1b3c0079b0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:35.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.446+0000 7f1b4aa00640 1 -- 172.21.15.69:0/1990706669 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b44045e50 con 0x7f1b54072e20 2024-04-24T08:21:35.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.725+0000 7f1b53200640 1 -- 172.21.15.69:0/1990706669 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) -- 0x7f1b54075d50 con 0x7f1b54072e20 2024-04-24T08:21:35.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.783+0000 7f1b4aa00640 1 -- 172.21.15.69:0/1990706669 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v614) ==== 144+0+0 (secure 0 0 0) 0x7f1b4407aea0 con 0x7f1b54072e20 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 -- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3006c8f0 msgr2=0x7f1b3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3006c8f0 0x7f1b3006edb0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f1b54158480 tx=0x7f1b3c0079b0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 -- 172.21.15.69:0/1990706669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54072e20 msgr2=0x7f1b54147800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54072e20 0x7f1b54147800 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f1b44000c00 tx=0x7f1b440046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 -- 172.21.15.69:0/1990706669 shutdown_connections 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1b3006c8f0 0x7f1b3006edb0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1b5415b980 0x7f1b54071be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b541523f0 0x7f1b541f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 --2- 172.21.15.69:0/1990706669 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b54072e20 0x7f1b54147800 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 -- 172.21.15.69:0/1990706669 >> 172.21.15.69:0/1990706669 conn(0x7f1b540d0160 msgr2=0x7f1b540b3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:35.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.786+0000 7f1b53200640 1 -- 172.21.15.69:0/1990706669 shutdown_connections 2024-04-24T08:21:35.788 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:35.787+0000 7f1b53200640 1 -- 172.21.15.69:0/1990706669 wait complete. 2024-04-24T08:21:35.807 INFO:tasks.cephfs_test_runner:test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-04-24T08:21:35.807 INFO:tasks.cephfs_test_runner: 2024-04-24T08:21:35.807 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-24T08:21:35.807 INFO:tasks.cephfs_test_runner:Ran 7 tests in 756.564s 2024-04-24T08:21:35.808 INFO:tasks.cephfs_test_runner: 2024-04-24T08:21:35.808 INFO:tasks.cephfs_test_runner:OK 2024-04-24T08:21:35.808 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-24T08:21:35.819 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-24T08:21:35.853 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-24T08:21:35.853 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:21:35.853 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:21:35.911 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:21:35.912 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-04-24T08:21:35.912 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:21:35.963 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T08:21:35.964 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:21:35.964 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:21:36.020 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:21:36.020 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-04-24T08:21:36.020 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:21:36.069 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052691,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248964k,nr_inodes=812241,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497936k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T08:21:36.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:36.070 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-04-24T08:21:36.126 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:21:36.126 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T08:21:36.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:36.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:21:36.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:36.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T08:21:36.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:36.288 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-04-24T08:21:36.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:36.407 DEBUG:teuthology.orchestra.run.smithi121:> ip netns list 2024-04-24T08:21:36.465 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-24T08:21:36.465 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-24T08:21:36.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:36.466 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T08:21:36.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:36.560 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T08:21:36.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:21:36.628 DEBUG:teuthology.orchestra.run.smithi121:> sudo ip link delete ceph-brx 2024-04-24T08:21:36.756 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T08:21:36.765 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T08:21:36.766 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:21:36.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.970+0000 7fd8ee600640 1 Processor -- start 2024-04-24T08:21:36.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.971+0000 7fd8ee600640 1 -- start start 2024-04-24T08:21:36.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.972+0000 7fd8ee600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fd8e8162300 0x7fd8e8143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:36.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.972+0000 7fd8ee600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e815f540 con 0x7fd8e8071e50 2024-04-24T08:21:36.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.972+0000 7fd8ee600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e815f820 con 0x7fd8e8072230 2024-04-24T08:21:36.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.972+0000 7fd8ee600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fd8e815fb00 con 0x7fd8e8162300 2024-04-24T08:21:36.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.972+0000 7fd8ed200640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd8e8071e50 0x7fd8e815ecc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49926/0 (socket says 172.21.15.69:49926) 2024-04-24T08:21:36.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.972+0000 7fd8ed200640 1 -- 172.21.15.69:0/3340115157 learned_addr learned my addr 172.21.15.69:0/3340115157 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:36.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7e00640 1 --2- 172.21.15.69:0/3340115157 >> v2:172.21.15.121:3300/0 conn(0x7fd8e8162300 0x7fd8e8143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:36.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1979024803 0 0) 0x7fd8e815f540 con 0x7fd8e8071e50 2024-04-24T08:21:36.973 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8d0003520 con 0x7fd8e8071e50 2024-04-24T08:21:36.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2681669671 0 0) 0x7fd8d0003520 con 0x7fd8e8071e50 2024-04-24T08:21:36.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 >> v2:172.21.15.121:3300/0 conn(0x7fd8e8162300 msgr2=0x7fd8e8143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:36.975 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 --2- 172.21.15.69:0/3340115157 >> v2:172.21.15.121:3300/0 conn(0x7fd8e8162300 0x7fd8e8143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 >> v1:172.21.15.121:6789/0 conn(0x7fd8e8072230 legacy=0x7fd8e8143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e806dda0 con 0x7fd8e8071e50 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fd8dc001670 con 0x7fd8e8071e50 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 >> v1:172.21.15.69:6789/0 conn(0x7fd8e8071e50 legacy=0x7fd8e815ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8d0003a00 0x7fd8d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8d0004800 0x7fd8d0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e815fb00 con 0x7fd8d00051c0 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d000c0c0 con 0x7fd8d0004800 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.973+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd8d000c2c0 con 0x7fd8d0003a00 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.974+0000 7fd8edc00640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.974+0000 7fd8ed200640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8d0003a00 0x7fd8d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:36.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.974+0000 7fd8edc00640 1 -- 172.21.15.69:0/3340115157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8d0003a00 msgr2=0x7fd8d0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.974+0000 7fd8e7e00640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8d0004800 0x7fd8d0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.974+0000 7fd8edc00640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8d0003a00 0x7fd8d0003e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.974+0000 7fd8edc00640 1 -- 172.21.15.69:0/3340115157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8d0004800 msgr2=0x7fd8d0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.974+0000 7fd8edc00640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8d0004800 0x7fd8d0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.974+0000 7fd8edc00640 1 -- 172.21.15.69:0/3340115157 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d0003520 con 0x7fd8d00051c0 2024-04-24T08:21:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.974+0000 7fd8e7e00640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8d0004800 0x7fd8d0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.975+0000 7fd8edc00640 1 -- 172.21.15.69:0/3340115157 reap_dead start 2024-04-24T08:21:36.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.975+0000 7fd8edc00640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8d000b9c0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0001e50 tx=0x7fd8d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71211e4b2614def7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:36.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.975+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0001a70 con 0x7fd8d00051c0 2024-04-24T08:21:36.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.975+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8e0003970 con 0x7fd8d00051c0 2024-04-24T08:21:36.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.975+0000 7fd8e7400640 1 -- 172.21.15.69:0/3340115157 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e0003d60 con 0x7fd8d00051c0 2024-04-24T08:21:36.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.976+0000 7fd8ee600640 1 -- 172.21.15.69:0/3340115157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 msgr2=0x7fd8d000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:36.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.976+0000 7fd8ee600640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8d000b9c0 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fd8e0001e50 tx=0x7fd8d0005830 comp rx=0 tx=0).stop 2024-04-24T08:21:36.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.976+0000 7fd8ee600640 1 -- 172.21.15.69:0/3340115157 shutdown_connections 2024-04-24T08:21:36.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.976+0000 7fd8ee600640 1 --2- 172.21.15.69:0/3340115157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8d000b9c0 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:36.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.976+0000 7fd8ee600640 1 -- 172.21.15.69:0/3340115157 >> 172.21.15.69:0/3340115157 conn(0x7fd8e80d0160 msgr2=0x7fd8e80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:36.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.976+0000 7fd8ee600640 1 -- 172.21.15.69:0/3340115157 shutdown_connections 2024-04-24T08:21:36.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.976+0000 7fd8ee600640 1 -- 172.21.15.69:0/3340115157 wait complete. 2024-04-24T08:21:36.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.978+0000 7fd8ee600640 1 Processor -- start 2024-04-24T08:21:36.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.978+0000 7fd8ee600640 1 -- start start 2024-04-24T08:21:36.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.979+0000 7fd8ee600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8e80994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:36.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.979+0000 7fd8ee600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8e814ed70 0x7fd8e814f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:36.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.979+0000 7fd8ee600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8e81f4950 0x7fd8e81f4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:36.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.979+0000 7fd8ee600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e815f050 con 0x7fd8d00051c0 2024-04-24T08:21:36.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.979+0000 7fd8ee600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd8e815eed0 con 0x7fd8e814ed70 2024-04-24T08:21:36.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.979+0000 7fd8ee600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd8e815f1d0 con 0x7fd8e81f4950 2024-04-24T08:21:36.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.979+0000 7fd8ed200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8e80994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.979+0000 7fd8ed200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8e80994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53124/0 (socket says 172.21.15.69:53124) 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.979+0000 7fd8ed200640 1 -- 172.21.15.69:0/2376583703 learned_addr learned my addr 172.21.15.69:0/2376583703 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.980+0000 7fd8e7e00640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8e814ed70 0x7fd8e814f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.980+0000 7fd8edc00640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8e81f4950 0x7fd8e81f4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.980+0000 7fd8ed200640 1 -- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8e81f4950 msgr2=0x7fd8e81f4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.980+0000 7fd8ed200640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8e81f4950 0x7fd8e81f4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.980+0000 7fd8ed200640 1 -- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8e814ed70 msgr2=0x7fd8e814f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.980+0000 7fd8ed200640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8e814ed70 0x7fd8e814f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.980+0000 7fd8ed200640 1 -- 172.21.15.69:0/2376583703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e814faa0 con 0x7fd8d00051c0 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.980+0000 7fd8ed200640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8e80994d0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00d7d0 tx=0x7fd8dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.980+0000 7fd8e4c00640 1 -- 172.21.15.69:0/2376583703 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc019070 con 0x7fd8d00051c0 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.981+0000 7fd8e4c00640 1 -- 172.21.15.69:0/2376583703 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8dc0074e0 con 0x7fd8d00051c0 2024-04-24T08:21:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.981+0000 7fd8ee600640 1 -- 172.21.15.69:0/2376583703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e814fd80 con 0x7fd8d00051c0 2024-04-24T08:21:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.981+0000 7fd8e4c00640 1 -- 172.21.15.69:0/2376583703 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc0049e0 con 0x7fd8d00051c0 2024-04-24T08:21:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.981+0000 7fd8ee600640 1 -- 172.21.15.69:0/2376583703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8e8150340 con 0x7fd8d00051c0 2024-04-24T08:21:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.982+0000 7fd8ee600640 1 -- 172.21.15.69:0/2376583703 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8e8075d50 con 0x7fd8d00051c0 2024-04-24T08:21:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.983+0000 7fd8e4c00640 1 -- 172.21.15.69:0/2376583703 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd8dc01c070 con 0x7fd8d00051c0 2024-04-24T08:21:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.984+0000 7fd8e4c00640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd8c806c920 0x7fd8c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.984+0000 7fd8e4c00640 1 -- 172.21.15.69:0/2376583703 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fd8dc08b2b0 con 0x7fd8d00051c0 2024-04-24T08:21:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.984+0000 7fd8e7e00640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd8c806c920 0x7fd8c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.986+0000 7fd8e7e00640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd8c806c920 0x7fd8c806ede0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd8e80b00b0 tx=0x7fd8d4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:36.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:36.988+0000 7fd8e4c00640 1 -- 172.21.15.69:0/2376583703 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd8dc0599a0 con 0x7fd8d00051c0 2024-04-24T08:21:37.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.178+0000 7fd8ee600640 1 -- 172.21.15.69:0/2376583703 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd8e80d0d90 con 0x7fd8c806c920 2024-04-24T08:21:37.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.180+0000 7fd8e4c00640 1 -- 172.21.15.69:0/2376583703 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13523 (secure 0 0 0) 0x7fd8e80d0d90 con 0x7fd8c806c920 2024-04-24T08:21:37.181 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:37.182 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:21:37.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 -- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd8c806c920 msgr2=0x7fd8c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd8c806c920 0x7fd8c806ede0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fd8e80b00b0 tx=0x7fd8d4008040 comp rx=0 tx=0).stop 2024-04-24T08:21:37.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 -- 172.21.15.69:0/2376583703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 msgr2=0x7fd8e80994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8e80994d0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc00d7d0 tx=0x7fd8dc00dc90 comp rx=0 tx=0).stop 2024-04-24T08:21:37.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 -- 172.21.15.69:0/2376583703 shutdown_connections 2024-04-24T08:21:37.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd8c806c920 0x7fd8c806ede0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd8e81f4950 0x7fd8e81f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8e814ed70 0x7fd8e814f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 --2- 172.21.15.69:0/2376583703 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8d00051c0 0x7fd8e80994d0 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 -- 172.21.15.69:0/2376583703 >> 172.21.15.69:0/2376583703 conn(0x7fd8e80d0160 msgr2=0x7fd8e80b3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:37.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 -- 172.21.15.69:0/2376583703 shutdown_connections 2024-04-24T08:21:37.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.183+0000 7fd8ee600640 1 -- 172.21.15.69:0/2376583703 wait complete. 2024-04-24T08:21:37.201 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-04-24T08:21:36.963653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":278144,"kb_used_data":28704,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754696576,"statfs":{"total":773094113280,"available":772809293824,"internally_reserved":0,"allocated":29392896,"data_stored":22920858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390725,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-264,"num_write":-181,"num_write_kb":-8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001879"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":346,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T08:21:32.192490+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:21:32.192490+0000","last_peered":"2024-04-24T08:21:32.192490+0000","last_clean":"2024-04-24T08:21:32.192490+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:21:32.192490+0000","last_undegraded":"2024-04-24T08:21:32.192490+0000","last_fullsized":"2024-04-24T08:21:32.192490+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575014,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33152,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338688,"statfs":{"total":96636764160,"available":96602816512,"internally_reserved":0,"allocated":2019328,"data_stored":1210818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575014,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":5488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":5619712,"data_stored":4814215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1433600,"data_stored":620450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1417216,"data_stored":619834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575012,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36680,"kb_used_data":5500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335160,"statfs":{"total":96636764160,"available":96599203840,"internally_reserved":0,"allocated":5632000,"data_stored":4815515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":619839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575013,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":5488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":5619712,"data_stored":4814262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575012,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":6084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":6230016,"data_stored":5405925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:21:37.202 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:21:37.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.396+0000 7fb8f4200640 1 Processor -- start 2024-04-24T08:21:37.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.396+0000 7fb8f4200640 1 -- start start 2024-04-24T08:21:37.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.398+0000 7fb8f4200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb8ec0b3ba0 0x7fb8ec163180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.398+0000 7fb8f4200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb8ec09b0e0 0x7fb8ec0b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.398+0000 7fb8f4200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8ec0d0d60 con 0x7fb8ec0d0880 2024-04-24T08:21:37.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.398+0000 7fb8f4200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb8ec1636c0 con 0x7fb8ec09b0e0 2024-04-24T08:21:37.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.398+0000 7fb8f4200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb8ec0d1040 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.398+0000 7fb8f2200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb8ec09b0e0 0x7fb8ec0b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:37.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.398+0000 7fb8f2200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb8ec09b0e0 0x7fb8ec0b1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53134/0 (socket says 172.21.15.69:53134) 2024-04-24T08:21:37.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.398+0000 7fb8f2200640 1 -- 172.21.15.69:0/3885725277 learned_addr learned my addr 172.21.15.69:0/3885725277 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:37.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.399+0000 7fb8f1800640 1 --2- 172.21.15.69:0/3885725277 >> v2:172.21.15.121:3300/0 conn(0x7fb8ec0b3ba0 0x7fb8ec163180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:37.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.399+0000 7fb8f2200640 1 -- 172.21.15.69:0/3885725277 >> v2:172.21.15.121:3300/0 conn(0x7fb8ec0b3ba0 msgr2=0x7fb8ec163180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:37.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.399+0000 7fb8f2200640 1 --2- 172.21.15.69:0/3885725277 >> v2:172.21.15.121:3300/0 conn(0x7fb8ec0b3ba0 0x7fb8ec163180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.399+0000 7fb8f2200640 1 -- 172.21.15.69:0/3885725277 >> v1:172.21.15.121:6789/0 conn(0x7fb8ec0d0880 legacy=0x7fb8ec09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:37.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.399+0000 7fb8f2200640 1 -- 172.21.15.69:0/3885725277 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec0d1320 con 0x7fb8ec09b0e0 2024-04-24T08:21:37.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.400+0000 7fb8f2200640 1 --2- 172.21.15.69:0/3885725277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec09b0e0 0x7fb8ec0b1f10 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec0d2580 tx=0x7fb8e402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ccc399d749c475 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:37.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.400+0000 7fb8ebe00640 1 -- 172.21.15.69:0/3885725277 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4037070 con 0x7fb8ec09b0e0 2024-04-24T08:21:37.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.400+0000 7fb8ebe00640 1 -- 172.21.15.69:0/3885725277 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8e4032440 con 0x7fb8ec09b0e0 2024-04-24T08:21:37.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.400+0000 7fb8ebe00640 1 -- 172.21.15.69:0/3885725277 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8e4032810 con 0x7fb8ec09b0e0 2024-04-24T08:21:37.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.401+0000 7fb8f4200640 1 -- 172.21.15.69:0/3885725277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec09b0e0 msgr2=0x7fb8ec0b1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.401+0000 7fb8f4200640 1 --2- 172.21.15.69:0/3885725277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec09b0e0 0x7fb8ec0b1f10 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec0d2580 tx=0x7fb8e402a230 comp rx=0 tx=0).stop 2024-04-24T08:21:37.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.401+0000 7fb8f4200640 1 -- 172.21.15.69:0/3885725277 shutdown_connections 2024-04-24T08:21:37.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.401+0000 7fb8f4200640 1 --2- 172.21.15.69:0/3885725277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec09b0e0 0x7fb8ec0b1f10 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.401+0000 7fb8f4200640 1 --2- 172.21.15.69:0/3885725277 >> v2:172.21.15.121:3300/0 conn(0x7fb8ec0b3ba0 0x7fb8ec163180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.401+0000 7fb8f4200640 1 -- 172.21.15.69:0/3885725277 >> 172.21.15.69:0/3885725277 conn(0x7fb8ec0b40a0 msgr2=0x7fb8ec0b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:37.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.401+0000 7fb8f4200640 1 -- 172.21.15.69:0/3885725277 shutdown_connections 2024-04-24T08:21:37.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.401+0000 7fb8f4200640 1 -- 172.21.15.69:0/3885725277 wait complete. 2024-04-24T08:21:37.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.402+0000 7fb8f4200640 1 Processor -- start 2024-04-24T08:21:37.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.403+0000 7fb8f4200640 1 -- start start 2024-04-24T08:21:37.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.403+0000 7fb8f4200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec09b0e0 0x7fb8ec0b67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.403+0000 7fb8f4200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec0b3ba0 0x7fb8ec0b7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f4200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8ec0d0880 0x7fb8ec1e8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f4200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec09b940 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f4200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ec0d18e0 con 0x7fb8ec09b0e0 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f4200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb8ec0b2450 con 0x7fb8ec0d0880 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec0b3ba0 0x7fb8ec0b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f0e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec0b3ba0 0x7fb8ec0b7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53142/0 (socket says 172.21.15.69:53142) 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f0e00640 1 -- 172.21.15.69:0/3912083574 learned_addr learned my addr 172.21.15.69:0/3912083574 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f0e00640 1 -- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8ec0d0880 msgr2=0x7fb8ec1e8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f0e00640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8ec0d0880 0x7fb8ec1e8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f1800640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec09b0e0 0x7fb8ec0b67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f0e00640 1 -- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec09b0e0 msgr2=0x7fb8ec0b67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.406 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f0e00640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec09b0e0 0x7fb8ec0b67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.404+0000 7fb8f0e00640 1 -- 172.21.15.69:0/3912083574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec0b5740 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.405+0000 7fb8f0e00640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec0b3ba0 0x7fb8ec0b7740 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00ba80 tx=0x7fb8dc00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:37.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.405+0000 7fb8e9600640 1 -- 172.21.15.69:0/3912083574 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc00cc00 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.405+0000 7fb8e9600640 1 -- 172.21.15.69:0/3912083574 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8dc00cda0 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.405+0000 7fb8e9600640 1 -- 172.21.15.69:0/3912083574 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc003bb0 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.405+0000 7fb8f4200640 1 -- 172.21.15.69:0/3912083574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ec0b5a20 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.407 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.405+0000 7fb8f4200640 1 -- 172.21.15.69:0/3912083574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ec0b5fe0 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.406+0000 7fb8f4200640 1 -- 172.21.15.69:0/3912083574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8ec068a80 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.408 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.407+0000 7fb8e9600640 1 -- 172.21.15.69:0/3912083574 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb8dc014440 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.408+0000 7fb8e9600640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8d006c8f0 0x7fb8d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.409+0000 7fb8e9600640 1 -- 172.21.15.69:0/3912083574 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fb8dc08b850 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.409 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.409+0000 7fb8f1800640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8d006c8f0 0x7fb8d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:37.411 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.410+0000 7fb8f1800640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8d006c8f0 0x7fb8d006edb0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0006fd0 tx=0x7fb8e0008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:37.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.413+0000 7fb8e9600640 1 -- 172.21.15.69:0/3912083574 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb8dc059ed0 con 0x7fb8ec0b3ba0 2024-04-24T08:21:37.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.594+0000 7fb8f4200640 1 -- 172.21.15.69:0/3912083574 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb8ec0b4cc0 con 0x7fb8d006c8f0 2024-04-24T08:21:37.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.597+0000 7fb8e9600640 1 -- 172.21.15.69:0/3912083574 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13523 (secure 0 0 0) 0x7fb8ec0b4cc0 con 0x7fb8d006c8f0 2024-04-24T08:21:37.598 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:37.598 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:21:37.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 -- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8d006c8f0 msgr2=0x7fb8d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8d006c8f0 0x7fb8d006edb0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0006fd0 tx=0x7fb8e0008040 comp rx=0 tx=0).stop 2024-04-24T08:21:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 -- 172.21.15.69:0/3912083574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec0b3ba0 msgr2=0x7fb8ec0b7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec0b3ba0 0x7fb8ec0b7740 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00ba80 tx=0x7fb8dc00bf40 comp rx=0 tx=0).stop 2024-04-24T08:21:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 -- 172.21.15.69:0/3912083574 shutdown_connections 2024-04-24T08:21:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb8d006c8f0 0x7fb8d006edb0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb8ec0b3ba0 0x7fb8ec0b7740 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8ec0d0880 0x7fb8ec1e8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 --2- 172.21.15.69:0/3912083574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8ec09b0e0 0x7fb8ec0b67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 -- 172.21.15.69:0/3912083574 >> 172.21.15.69:0/3912083574 conn(0x7fb8ec0b40a0 msgr2=0x7fb8ec070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 -- 172.21.15.69:0/3912083574 shutdown_connections 2024-04-24T08:21:37.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.599+0000 7fb8f4200640 1 -- 172.21.15.69:0/3912083574 wait complete. 2024-04-24T08:21:37.617 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-04-24T08:21:36.963653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":278144,"kb_used_data":28704,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754696576,"statfs":{"total":773094113280,"available":772809293824,"internally_reserved":0,"allocated":29392896,"data_stored":22920858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390725,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-264,"num_write":-181,"num_write_kb":-8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001879"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":346,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T08:21:32.192490+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:21:32.192490+0000","last_peered":"2024-04-24T08:21:32.192490+0000","last_clean":"2024-04-24T08:21:32.192490+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:21:32.192490+0000","last_undegraded":"2024-04-24T08:21:32.192490+0000","last_fullsized":"2024-04-24T08:21:32.192490+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575014,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33152,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338688,"statfs":{"total":96636764160,"available":96602816512,"internally_reserved":0,"allocated":2019328,"data_stored":1210818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575014,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":5488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":5619712,"data_stored":4814215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1433600,"data_stored":620450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1417216,"data_stored":619834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575012,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36680,"kb_used_data":5500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335160,"statfs":{"total":96636764160,"available":96599203840,"internally_reserved":0,"allocated":5632000,"data_stored":4815515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":619839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575013,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":5488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":5619712,"data_stored":4814262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575012,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":6084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":6230016,"data_stored":5405925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:21:37.617 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T08:21:37.618 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:21:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.809+0000 7fe5d0600640 1 Processor -- start 2024-04-24T08:21:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.809+0000 7fe5d0600640 1 -- start start 2024-04-24T08:21:37.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.811+0000 7fe5d0600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fe5c8145a90 0x7fe5c806e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.811+0000 7fe5d0600640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fe5c8145fe0 0x7fe5c8160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.811+0000 7fe5d0600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5c80cf3d0 con 0x7fe5c80d0300 2024-04-24T08:21:37.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.811+0000 7fe5d0600640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fe5c80cf6b0 con 0x7fe5c8145fe0 2024-04-24T08:21:37.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.811+0000 7fe5d0600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fe5c80cf990 con 0x7fe5c8145a90 2024-04-24T08:21:37.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.811+0000 7fe5cdc00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fe5c80d0300 0x7fe5c80d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49938/0 (socket says 172.21.15.69:49938) 2024-04-24T08:21:37.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.811+0000 7fe5cdc00640 1 -- 172.21.15.69:0/3634685905 learned_addr learned my addr 172.21.15.69:0/3634685905 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:37.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.811+0000 7fe5ce600640 1 --2- 172.21.15.69:0/3634685905 >> v2:172.21.15.69:3300/0 conn(0x7fe5c8145fe0 0x7fe5c8160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:37.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.812+0000 7fe5cd200640 1 --2- 172.21.15.69:0/3634685905 >> v2:172.21.15.121:3300/0 conn(0x7fe5c8145a90 0x7fe5c806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:37.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.812+0000 7fe5ce600640 1 -- 172.21.15.69:0/3634685905 >> v2:172.21.15.121:3300/0 conn(0x7fe5c8145a90 msgr2=0x7fe5c806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:37.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.812+0000 7fe5ce600640 1 --2- 172.21.15.69:0/3634685905 >> v2:172.21.15.121:3300/0 conn(0x7fe5c8145a90 0x7fe5c806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.812+0000 7fe5ce600640 1 -- 172.21.15.69:0/3634685905 >> v1:172.21.15.69:6789/0 conn(0x7fe5c80d0300 legacy=0x7fe5c80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.812+0000 7fe5c7e00640 1 -- 172.21.15.69:0/3634685905 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1127943227 0 0) 0x7fe5c80cf3d0 con 0x7fe5c80d0300 2024-04-24T08:21:37.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.812+0000 7fe5ce600640 1 -- 172.21.15.69:0/3634685905 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c80cfc70 con 0x7fe5c8145fe0 2024-04-24T08:21:37.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.812+0000 7fe5cd200640 1 --2- 172.21.15.69:0/3634685905 >> v2:172.21.15.121:3300/0 conn(0x7fe5c8145a90 0x7fe5c806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:37.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.813+0000 7fe5ce600640 1 --2- 172.21.15.69:0/3634685905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c8145fe0 0x7fe5c8160540 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80b5780 tx=0x7fe5c002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f62e7334ef3f76 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:37.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.813+0000 7fe5c7e00640 1 -- 172.21.15.69:0/3634685905 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0007740 con 0x7fe5c8145fe0 2024-04-24T08:21:37.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.813+0000 7fe5c7e00640 1 -- 172.21.15.69:0/3634685905 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c00078e0 con 0x7fe5c8145fe0 2024-04-24T08:21:37.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.813+0000 7fe5c7e00640 1 -- 172.21.15.69:0/3634685905 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0007cb0 con 0x7fe5c8145fe0 2024-04-24T08:21:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.813+0000 7fe5d0600640 1 -- 172.21.15.69:0/3634685905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c8145fe0 msgr2=0x7fe5c8160540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.813+0000 7fe5d0600640 1 --2- 172.21.15.69:0/3634685905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c8145fe0 0x7fe5c8160540 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80b5780 tx=0x7fe5c002a680 comp rx=0 tx=0).stop 2024-04-24T08:21:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.814+0000 7fe5d0600640 1 -- 172.21.15.69:0/3634685905 shutdown_connections 2024-04-24T08:21:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.814+0000 7fe5d0600640 1 --2- 172.21.15.69:0/3634685905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c8145fe0 0x7fe5c8160540 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.814+0000 7fe5d0600640 1 --2- 172.21.15.69:0/3634685905 >> v2:172.21.15.121:3300/0 conn(0x7fe5c8145a90 0x7fe5c806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.814+0000 7fe5d0600640 1 -- 172.21.15.69:0/3634685905 >> 172.21.15.69:0/3634685905 conn(0x7fe5c8067d90 msgr2=0x7fe5c806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.814+0000 7fe5d0600640 1 -- 172.21.15.69:0/3634685905 shutdown_connections 2024-04-24T08:21:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.814+0000 7fe5d0600640 1 -- 172.21.15.69:0/3634685905 wait complete. 2024-04-24T08:21:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.815+0000 7fe5d0600640 1 Processor -- start 2024-04-24T08:21:37.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.815+0000 7fe5d0600640 1 -- start start 2024-04-24T08:21:37.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.816+0000 7fe5d0600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c80d0300 0x7fe5c81f44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.816+0000 7fe5d0600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c8145a90 0x7fe5c80cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.816+0000 7fe5d0600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c8145fe0 0x7fe5c8140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.816+0000 7fe5d0600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c81458f0 con 0x7fe5c80d0300 2024-04-24T08:21:37.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.816+0000 7fe5d0600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fe5c8145770 con 0x7fe5c8145a90 2024-04-24T08:21:37.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.816+0000 7fe5d0600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fe5c80cdc30 con 0x7fe5c8145fe0 2024-04-24T08:21:37.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.817+0000 7fe5cdc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c80d0300 0x7fe5c81f44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.817+0000 7fe5cdc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c80d0300 0x7fe5c81f44d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53158/0 (socket says 172.21.15.69:53158) 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.817+0000 7fe5cdc00640 1 -- 172.21.15.69:0/1304296817 learned_addr learned my addr 172.21.15.69:0/1304296817 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.817+0000 7fe5cdc00640 1 -- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c8145fe0 msgr2=0x7fe5c8140d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.817+0000 7fe5cdc00640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c8145fe0 0x7fe5c8140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.817+0000 7fe5cdc00640 1 -- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c8145a90 msgr2=0x7fe5c80cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.817+0000 7fe5cdc00640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c8145a90 0x7fe5c80cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.817+0000 7fe5cdc00640 1 -- 172.21.15.69:0/1304296817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5c80b3a90 con 0x7fe5c80d0300 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.817+0000 7fe5cdc00640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c80d0300 0x7fe5c81f44d0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00c430 tx=0x7fe5bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.817+0000 7fe5c5600640 1 -- 172.21.15.69:0/1304296817 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc0074e0 con 0x7fe5c80d0300 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.818+0000 7fe5c5600640 1 -- 172.21.15.69:0/1304296817 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5bc0047e0 con 0x7fe5c80d0300 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.818+0000 7fe5c5600640 1 -- 172.21.15.69:0/1304296817 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5bc004bd0 con 0x7fe5c80d0300 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.818+0000 7fe5d0600640 1 -- 172.21.15.69:0/1304296817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5c80b3d70 con 0x7fe5c80d0300 2024-04-24T08:21:37.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.818+0000 7fe5d0600640 1 -- 172.21.15.69:0/1304296817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5c81f4a10 con 0x7fe5c80d0300 2024-04-24T08:21:37.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.819+0000 7fe5d0600640 1 -- 172.21.15.69:0/1304296817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5c8068a80 con 0x7fe5c80d0300 2024-04-24T08:21:37.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.820+0000 7fe5c5600640 1 -- 172.21.15.69:0/1304296817 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe5bc011070 con 0x7fe5c80d0300 2024-04-24T08:21:37.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.821+0000 7fe5c5600640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe5a806c8f0 0x7fe5a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:37.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.821+0000 7fe5c5600640 1 -- 172.21.15.69:0/1304296817 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fe5bc08b5f0 con 0x7fe5c80d0300 2024-04-24T08:21:37.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.822+0000 7fe5cd200640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe5a806c8f0 0x7fe5a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:37.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.823+0000 7fe5cd200640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe5a806c8f0 0x7fe5a806edb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4006fd0 tx=0x7fe5b4008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:37.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:37.825+0000 7fe5c5600640 1 -- 172.21.15.69:0/1304296817 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe5bc059c60 con 0x7fe5c80d0300 2024-04-24T08:21:38.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.023+0000 7fe5d0600640 1 -- 172.21.15.69:0/1304296817 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe5c80d1120 con 0x7fe5a806c8f0 2024-04-24T08:21:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.026+0000 7fe5c5600640 1 -- 172.21.15.69:0/1304296817 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13523 (secure 0 0 0) 0x7fe5c80d1120 con 0x7fe5a806c8f0 2024-04-24T08:21:38.027 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:21:38.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 -- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe5a806c8f0 msgr2=0x7fe5a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe5a806c8f0 0x7fe5a806edb0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fe5b4006fd0 tx=0x7fe5b4008040 comp rx=0 tx=0).stop 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 -- 172.21.15.69:0/1304296817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c80d0300 msgr2=0x7fe5c81f44d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c80d0300 0x7fe5c81f44d0 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc00c430 tx=0x7fe5bc00c8f0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 -- 172.21.15.69:0/1304296817 shutdown_connections 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fe5a806c8f0 0x7fe5a806edb0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fe5c8145fe0 0x7fe5c8140d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5c8145a90 0x7fe5c80cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 --2- 172.21.15.69:0/1304296817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe5c80d0300 0x7fe5c81f44d0 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 -- 172.21.15.69:0/1304296817 >> 172.21.15.69:0/1304296817 conn(0x7fe5c8067d90 msgr2=0x7fe5c80d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 -- 172.21.15.69:0/1304296817 shutdown_connections 2024-04-24T08:21:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.029+0000 7fe5d0600640 1 -- 172.21.15.69:0/1304296817 wait complete. 2024-04-24T08:21:38.049 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-04-24T08:21:36.963653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":278144,"kb_used_data":28704,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754696576,"statfs":{"total":773094113280,"available":772809293824,"internally_reserved":0,"allocated":29392896,"data_stored":22920858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390725,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-264,"num_write":-181,"num_write_kb":-8296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001879"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":346,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-04-24T08:21:32.192490+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:21:32.192490+0000","last_peered":"2024-04-24T08:21:32.192490+0000","last_clean":"2024-04-24T08:21:32.192490+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:21:32.192490+0000","last_undegraded":"2024-04-24T08:21:32.192490+0000","last_fullsized":"2024-04-24T08:21:32.192490+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575014,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33152,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338688,"statfs":{"total":96636764160,"available":96602816512,"internally_reserved":0,"allocated":2019328,"data_stored":1210818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575014,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":5488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":5619712,"data_stored":4814215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575012,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32580,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339260,"statfs":{"total":96636764160,"available":96603402240,"internally_reserved":0,"allocated":1433600,"data_stored":620450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575011,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32564,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339276,"statfs":{"total":96636764160,"available":96603418624,"internally_reserved":0,"allocated":1417216,"data_stored":619834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575012,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36680,"kb_used_data":5500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335160,"statfs":{"total":96636764160,"available":96599203840,"internally_reserved":0,"allocated":5632000,"data_stored":4815515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575013,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339272,"statfs":{"total":96636764160,"available":96603414528,"internally_reserved":0,"allocated":1421312,"data_stored":619839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575013,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":5488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":5619712,"data_stored":4814262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575012,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37264,"kb_used_data":6084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334576,"statfs":{"total":96636764160,"available":96598605824,"internally_reserved":0,"allocated":6230016,"data_stored":5405925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:21:38.049 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:38.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.236+0000 7f9fa1000640 1 Processor -- start 2024-04-24T08:21:38.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.237+0000 7f9fa1000640 1 -- start start 2024-04-24T08:21:38.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.238+0000 7f9fa1000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c06cfb0 0x7f9f9c0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:38.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.238+0000 7f9fa1000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f9f9c06d410 0x7f9f9c0d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:38.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.238+0000 7f9fa1000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f9c15ed40 con 0x7f9f9c15e960 2024-04-24T08:21:38.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.238+0000 7f9fa1000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f9f9c0cf050 con 0x7f9f9c06d410 2024-04-24T08:21:38.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.238+0000 7f9fa1000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9f9c162b50 con 0x7f9f9c06cfb0 2024-04-24T08:21:38.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.238+0000 7f9f9b400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f9f9c15e960 0x7f9f9c162440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49944/0 (socket says 172.21.15.69:49944) 2024-04-24T08:21:38.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.238+0000 7f9f9b400640 1 -- 172.21.15.69:0/3364436659 learned_addr learned my addr 172.21.15.69:0/3364436659 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9be00640 1 --2- 172.21.15.69:0/3364436659 >> v2:172.21.15.121:3300/0 conn(0x7f9f9c06d410 0x7f9f9c0d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f93e00640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c06cfb0 0x7f9f9c0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3023618098 0 0) 0x7f9f9c15ed40 con 0x7f9f9c15e960 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f80003520 con 0x7f9f9c15e960 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2699546813 0 0) 0x7f9f80003520 con 0x7f9f9c15e960 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c06cfb0 msgr2=0x7f9f9c0cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c06cfb0 0x7f9f9c0cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 >> v2:172.21.15.121:3300/0 conn(0x7f9f9c06d410 msgr2=0x7f9f9c0d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 --2- 172.21.15.69:0/3364436659 >> v2:172.21.15.121:3300/0 conn(0x7f9f9c06d410 0x7f9f9c0d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f9c162e20 con 0x7f9f9c15e960 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f9f8c001670 con 0x7f9f9c15e960 2024-04-24T08:21:38.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 >> v1:172.21.15.69:6789/0 conn(0x7f9f9c15e960 legacy=0x7f9f9c162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f800039b0 0x7f9f80003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f80004800 0x7f9f80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c06cfb0 0x7f9f800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c0cf050 con 0x7f9f80004800 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c162b50 con 0x7f9f800039b0 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.239+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9f8000b9c0 con 0x7f9f9c06cfb0 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.240+0000 7f9f93e00640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f80004800 0x7f9f80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.240+0000 7f9f9b400640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f800039b0 0x7f9f80003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.240+0000 7f9f93e00640 1 -- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c06cfb0 msgr2=0x7f9f800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.240+0000 7f9f93e00640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c06cfb0 0x7f9f800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.240+0000 7f9f93e00640 1 -- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f800039b0 msgr2=0x7f9f80003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.240+0000 7f9f93e00640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f800039b0 0x7f9f80003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.240+0000 7f9f93e00640 1 -- 172.21.15.69:0/3364436659 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f80003520 con 0x7f9f80004800 2024-04-24T08:21:38.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.240+0000 7f9f9b400640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f800039b0 0x7f9f80003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:38.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.241+0000 7f9f93e00640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f80004800 0x7f9f80004c80 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009ac0 tx=0x7f9f80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc5854a2e4e073a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:38.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.241+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84036070 con 0x7f9f80004800 2024-04-24T08:21:38.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.241+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f84031400 con 0x7f9f80004800 2024-04-24T08:21:38.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.241+0000 7f9f9aa00640 1 -- 172.21.15.69:0/3364436659 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84009690 con 0x7f9f80004800 2024-04-24T08:21:38.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.242+0000 7f9fa1000640 1 -- 172.21.15.69:0/3364436659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f80004800 msgr2=0x7f9f80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:38.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.242+0000 7f9fa1000640 1 --2- 172.21.15.69:0/3364436659 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f80004800 0x7f9f80004c80 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009ac0 tx=0x7f9f80005830 comp rx=0 tx=0).stop 2024-04-24T08:21:38.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.242+0000 7f9f9be00640 1 -- 172.21.15.69:0/3364436659 reap_dead start 2024-04-24T08:21:38.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.242+0000 7f9fa1000640 1 -- 172.21.15.69:0/3364436659 shutdown_connections 2024-04-24T08:21:38.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.242+0000 7f9fa1000640 1 -- 172.21.15.69:0/3364436659 >> 172.21.15.69:0/3364436659 conn(0x7f9f9c09d900 msgr2=0x7f9f9c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:38.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.242+0000 7f9fa1000640 1 -- 172.21.15.69:0/3364436659 shutdown_connections 2024-04-24T08:21:38.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.242+0000 7f9fa1000640 1 -- 172.21.15.69:0/3364436659 wait complete. 2024-04-24T08:21:38.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.244+0000 7f9fa1000640 1 Processor -- start 2024-04-24T08:21:38.245 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.244+0000 7f9fa1000640 1 -- start start 2024-04-24T08:21:38.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.245+0000 7f9fa1000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f80004800 0x7f9f9c15d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:38.246 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.245+0000 7f9fa1000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c160fe0 0x7f9f9c0cb610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.245+0000 7f9fa1000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c0ccfb0 0x7f9f9c0cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.245+0000 7f9fa1000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c0cf5d0 con 0x7f9f9c160fe0 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.245+0000 7f9fa1000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f9f9c0cf2f0 con 0x7f9f80004800 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.245+0000 7f9fa1000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9f9c0b6170 con 0x7f9f9c0ccfb0 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.245+0000 7f9f93e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c160fe0 0x7f9f9c0cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.245+0000 7f9f93e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c160fe0 0x7f9f9c0cb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53178/0 (socket says 172.21.15.69:53178) 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.245+0000 7f9f93e00640 1 -- 172.21.15.69:0/1371793198 learned_addr learned my addr 172.21.15.69:0/1371793198 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.246+0000 7f9f9b400640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f80004800 0x7f9f9c15d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.246+0000 7f9f9be00640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c0ccfb0 0x7f9f9c0cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.246+0000 7f9f93e00640 1 -- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c0ccfb0 msgr2=0x7f9f9c0cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.246+0000 7f9f93e00640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c0ccfb0 0x7f9f9c0cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.246+0000 7f9f93e00640 1 -- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f80004800 msgr2=0x7f9f9c15d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.246+0000 7f9f93e00640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f80004800 0x7f9f9c15d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.246+0000 7f9f93e00640 1 -- 172.21.15.69:0/1371793198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f9c0cbf10 con 0x7f9f9c160fe0 2024-04-24T08:21:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.246+0000 7f9f9be00640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c0ccfb0 0x7f9f9c0cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:38.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.246+0000 7f9f93e00640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c160fe0 0x7f9f9c0cb610 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f9f84034040 tx=0x7f9f84004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:38.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.247+0000 7f9f93400640 1 -- 172.21.15.69:0/1371793198 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84040020 con 0x7f9f9c160fe0 2024-04-24T08:21:38.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.247+0000 7f9f93400640 1 -- 172.21.15.69:0/1371793198 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f840038d0 con 0x7f9f9c160fe0 2024-04-24T08:21:38.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.247+0000 7f9f93400640 1 -- 172.21.15.69:0/1371793198 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f84036040 con 0x7f9f9c160fe0 2024-04-24T08:21:38.248 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.247+0000 7f9fa1000640 1 -- 172.21.15.69:0/1371793198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f9c0cc1f0 con 0x7f9f9c160fe0 2024-04-24T08:21:38.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.247+0000 7f9fa1000640 1 -- 172.21.15.69:0/1371793198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f9c0cc7b0 con 0x7f9f9c160fe0 2024-04-24T08:21:38.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.248+0000 7f9fa1000640 1 -- 172.21.15.69:0/1371793198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f9c075d50 con 0x7f9f9c160fe0 2024-04-24T08:21:38.249 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.249+0000 7f9f93400640 1 -- 172.21.15.69:0/1371793198 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9f84043020 con 0x7f9f9c160fe0 2024-04-24T08:21:38.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.250+0000 7f9f93400640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f7806c8f0 0x7f9f7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:38.250 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.250+0000 7f9f93400640 1 -- 172.21.15.69:0/1371793198 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f9f840ab9f0 con 0x7f9f9c160fe0 2024-04-24T08:21:38.251 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.250+0000 7f9f9b400640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f7806c8f0 0x7f9f7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:38.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.251+0000 7f9f9b400640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f7806c8f0 0x7f9f7806edb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c001eb0 tx=0x7f9f8c009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:38.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.254+0000 7f9f93400640 1 -- 172.21.15.69:0/1371793198 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f8404a070 con 0x7f9f9c160fe0 2024-04-24T08:21:38.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.457+0000 7f9fa1000640 1 -- 172.21.15.69:0/1371793198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f9c0d1dc0 con 0x7f9f9c160fe0 2024-04-24T08:21:38.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.459+0000 7f9f93400640 1 -- 172.21.15.69:0/1371793198 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) ==== 74+0+13264 (secure 0 0 0) 0x7f9f84079fd0 con 0x7f9f9c160fe0 2024-04-24T08:21:38.459 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:38.460 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":116,"fsid":"3422a353-af9e-4ed5-a4b2-7365b9c531c6","created":"2024-04-24T08:07:55.443960+0000","modified":"2024-04-24T08:21:34.756570+0000","last_up_change":"2024-04-24T08:08:07.954297+0000","last_in_change":"2024-04-24T08:07:58.727306+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:08:08.695404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f892d466-0add-47f5-9758-28353c69b922","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6801","nonce":146737935}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6803","nonce":146737935}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6807","nonce":146737935}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":146737935},{"type":"v1","addr":"172.21.15.69:6805","nonce":146737935}]},"public_addr":"172.21.15.69:6801/146737935","cluster_addr":"172.21.15.69:6803/146737935","heartbeat_back_addr":"172.21.15.69:6807/146737935","heartbeat_front_addr":"172.21.15.69:6805/146737935","state":["exists","up"]},{"osd":1,"uuid":"40de3817-1e3a-467e-850b-0027ceccdcb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6815","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6817","nonce":153736297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6821","nonce":153736297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6831","nonce":153736297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6825","nonce":153736297},{"type":"v1","addr":"172.21.15.69:6827","nonce":153736297}]},"public_addr":"172.21.15.69:6817/153736297","cluster_addr":"172.21.15.69:6821/153736297","heartbeat_back_addr":"172.21.15.69:6831/153736297","heartbeat_front_addr":"172.21.15.69:6827/153736297","state":["exists","up"]},{"osd":2,"uuid":"e952a5f2-4652-482d-baa9-8193e47b28dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6809","nonce":2630808717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6812","nonce":2630808717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6819","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6822","nonce":2630808717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":2630808717},{"type":"v1","addr":"172.21.15.69:6816","nonce":2630808717}]},"public_addr":"172.21.15.69:6809/2630808717","cluster_addr":"172.21.15.69:6812/2630808717","heartbeat_back_addr":"172.21.15.69:6822/2630808717","heartbeat_front_addr":"172.21.15.69:6816/2630808717","state":["exists","up"]},{"osd":3,"uuid":"7fa75d5f-6148-4e0c-9e4e-b5041eaf66ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6811","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6813","nonce":1875112303}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6823","nonce":1875112303}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6829","nonce":1875112303}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":1875112303},{"type":"v1","addr":"172.21.15.69:6826","nonce":1875112303}]},"public_addr":"172.21.15.69:6813/1875112303","cluster_addr":"172.21.15.69:6823/1875112303","heartbeat_back_addr":"172.21.15.69:6829/1875112303","heartbeat_front_addr":"172.21.15.69:6826/1875112303","state":["exists","up"]},{"osd":4,"uuid":"027a943d-d007-4adf-8e00-8261810feaa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6801","nonce":2274643300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6803","nonce":2274643300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6809","nonce":2274643300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":2274643300},{"type":"v1","addr":"172.21.15.121:6805","nonce":2274643300}]},"public_addr":"172.21.15.121:6801/2274643300","cluster_addr":"172.21.15.121:6803/2274643300","heartbeat_back_addr":"172.21.15.121:6809/2274643300","heartbeat_front_addr":"172.21.15.121:6805/2274643300","state":["exists","up"]},{"osd":5,"uuid":"9bbf8edc-5742-43a4-ba34-afe0e3f880bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6824","nonce":2187246263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6827","nonce":2187246263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6831","nonce":2187246263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":2187246263},{"type":"v1","addr":"172.21.15.121:6829","nonce":2187246263}]},"public_addr":"172.21.15.121:6824/2187246263","cluster_addr":"172.21.15.121:6827/2187246263","heartbeat_back_addr":"172.21.15.121:6831/2187246263","heartbeat_front_addr":"172.21.15.121:6829/2187246263","state":["exists","up"]},{"osd":6,"uuid":"2edf233a-d123-4810-b90d-1a04dd625042","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6807","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6808","nonce":3068198325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6812","nonce":3068198325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6818","nonce":3068198325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6813","nonce":3068198325},{"type":"v1","addr":"172.21.15.121:6815","nonce":3068198325}]},"public_addr":"172.21.15.121:6808/3068198325","cluster_addr":"172.21.15.121:6812/3068198325","heartbeat_back_addr":"172.21.15.121:6818/3068198325","heartbeat_front_addr":"172.21.15.121:6815/3068198325","state":["exists","up"]},{"osd":7,"uuid":"bd79e417-76c0-4e4f-b82e-6fdf8c01c7a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6811","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6814","nonce":176144805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6817","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6819","nonce":176144805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6823","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6825","nonce":176144805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":176144805},{"type":"v1","addr":"172.21.15.121:6821","nonce":176144805}]},"public_addr":"172.21.15.121:6814/176144805","cluster_addr":"172.21.15.121:6819/176144805","heartbeat_back_addr":"172.21.15.121:6825/176144805","heartbeat_front_addr":"172.21.15.121:6821/176144805","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.051246+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.202561+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.172343+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.121790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.076640+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:03.902234+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.140192+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:08:04.129331+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:6834/99442307":"2024-04-25T08:21:31.512925+0000","172.21.15.69:6835/99442307":"2024-04-25T08:21:31.512925+0000","172.21.15.69:6835/2194021008":"2024-04-25T08:21:12.532354+0000","172.21.15.69:6834/2194021008":"2024-04-25T08:21:12.532354+0000","172.21.15.121:6836/1515102940":"2024-04-25T08:20:50.436176+0000","172.21.15.121:6834/1515102940":"2024-04-25T08:20:50.436176+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:38.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 -- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f7806c8f0 msgr2=0x7f9f7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:38.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f7806c8f0 0x7f9f7806edb0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c001eb0 tx=0x7f9f8c009040 comp rx=0 tx=0).stop 2024-04-24T08:21:38.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 -- 172.21.15.69:0/1371793198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c160fe0 msgr2=0x7f9f9c0cb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:38.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c160fe0 0x7f9f9c0cb610 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f9f84034040 tx=0x7f9f84004730 comp rx=0 tx=0).stop 2024-04-24T08:21:38.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 -- 172.21.15.69:0/1371793198 shutdown_connections 2024-04-24T08:21:38.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f9f7806c8f0 0x7f9f7806edb0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9f9c160fe0 0x7f9f9c0cb610 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.462 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f9f9c0ccfb0 0x7f9f9c0cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 --2- 172.21.15.69:0/1371793198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f80004800 0x7f9f9c15d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:38.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 -- 172.21.15.69:0/1371793198 >> 172.21.15.69:0/1371793198 conn(0x7f9f9c09d900 msgr2=0x7f9f9c0b44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:38.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 -- 172.21.15.69:0/1371793198 shutdown_connections 2024-04-24T08:21:38.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:38.461+0000 7f9fa1000640 1 -- 172.21.15.69:0/1371793198 wait complete. 2024-04-24T08:21:39.480 INFO:tasks.ceph:Scrubbing osd.0 2024-04-24T08:21:39.481 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T08:21:39.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.685+0000 7fd9cbe00640 1 Processor -- start 2024-04-24T08:21:39.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.685+0000 7fd9cbe00640 1 -- start start 2024-04-24T08:21:39.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.686+0000 7fd9cbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9cc0d32b0 0x7fd9cc0d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.686+0000 7fd9cbe00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9cc0d1e80 con 0x7fd9cc15e860 2024-04-24T08:21:39.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.686+0000 7fd9cbe00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9cc0cf350 con 0x7fd9cc0d4d30 2024-04-24T08:21:39.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.686+0000 7fd9cbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9cc0cfc80 con 0x7fd9cc0d32b0 2024-04-24T08:21:39.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.687+0000 7fd9caa00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fd9cc15e860 0x7fd9cc0d1670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42682/0 (socket says 172.21.15.69:42682) 2024-04-24T08:21:39.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.687+0000 7fd9caa00640 1 -- 172.21.15.69:0/1087625437 learned_addr learned my addr 172.21.15.69:0/1087625437 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:39.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.687+0000 7fd9cb400640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9cc0d32b0 0x7fd9cc0d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.687+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004462709 0 0) 0x7fd9cc0d1e80 con 0x7fd9cc15e860 2024-04-24T08:21:39.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.687+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9b0003520 con 0x7fd9cc15e860 2024-04-24T08:21:39.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1549261448 0 0) 0x7fd9b0003520 con 0x7fd9cc15e860 2024-04-24T08:21:39.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9cc0d32b0 msgr2=0x7fd9cc0d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:39.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9cc0d32b0 0x7fd9cc0d0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 >> v1:172.21.15.121:6789/0 conn(0x7fd9cc0d4d30 legacy=0x7fd9cc0d2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9cc0cf630 con 0x7fd9cc15e860 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fd9bc001670 con 0x7fd9cc15e860 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 >> v1:172.21.15.69:6789/0 conn(0x7fd9cc15e860 legacy=0x7fd9cc0d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b00039e0 0x7fd9b0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9b00043d0 0x7fd9b00048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b00050a0 0x7fd9b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc0cfc80 con 0x7fd9b00050a0 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b000bfb0 con 0x7fd9b00043d0 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9b000c1b0 con 0x7fd9b00039e0 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.688+0000 7fd9cb400640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9cc0d32b0 0x7fd9cc0d0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.689+0000 7fd9cb400640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b00050a0 0x7fd9b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.689+0000 7fd9c2000640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9b00043d0 0x7fd9b00048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.689+0000 7fd9caa00640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b00039e0 0x7fd9b0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.689+0000 7fd9cb400640 1 -- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b00039e0 msgr2=0x7fd9b0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:39.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.689+0000 7fd9cb400640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b00039e0 0x7fd9b0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.691 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.689+0000 7fd9cb400640 1 -- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9b00043d0 msgr2=0x7fd9b00048a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:39.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.689+0000 7fd9cb400640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9b00043d0 0x7fd9b00048a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.689+0000 7fd9cb400640 1 -- 172.21.15.69:0/1087625437 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b0003520 con 0x7fd9b00050a0 2024-04-24T08:21:39.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.690+0000 7fd9cb400640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b00050a0 0x7fd9b000b9c0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0d0900 tx=0x7fd9b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c7fce52c394655 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:39.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.691+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4036070 con 0x7fd9b00050a0 2024-04-24T08:21:39.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.691+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c4003b90 con 0x7fd9b00050a0 2024-04-24T08:21:39.692 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.691+0000 7fd9ca000640 1 -- 172.21.15.69:0/1087625437 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4009d60 con 0x7fd9b00050a0 2024-04-24T08:21:39.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.691+0000 7fd9cbe00640 1 -- 172.21.15.69:0/1087625437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b00050a0 msgr2=0x7fd9b000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.691+0000 7fd9cbe00640 1 --2- 172.21.15.69:0/1087625437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9b00050a0 0x7fd9b000b9c0 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc0d0900 tx=0x7fd9b0005830 comp rx=0 tx=0).stop 2024-04-24T08:21:39.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.691+0000 7fd9cb400640 1 -- 172.21.15.69:0/1087625437 reap_dead start 2024-04-24T08:21:39.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.691+0000 7fd9cbe00640 1 -- 172.21.15.69:0/1087625437 shutdown_connections 2024-04-24T08:21:39.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.692+0000 7fd9cbe00640 1 -- 172.21.15.69:0/1087625437 >> 172.21.15.69:0/1087625437 conn(0x7fd9cc06c210 msgr2=0x7fd9cc09d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:39.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.692+0000 7fd9cbe00640 1 -- 172.21.15.69:0/1087625437 shutdown_connections 2024-04-24T08:21:39.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.692+0000 7fd9cbe00640 1 -- 172.21.15.69:0/1087625437 wait complete. 2024-04-24T08:21:39.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.693+0000 7fd9cbe00640 1 Processor -- start 2024-04-24T08:21:39.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.693+0000 7fd9cbe00640 1 -- start start 2024-04-24T08:21:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.694+0000 7fd9cbe00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b00050a0 0x7fd9cc14d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.694+0000 7fd9cbe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9cc14e270 0x7fd9cc0cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.694+0000 7fd9cbe00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9cc0c9270 0x7fd9cc0c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.694+0000 7fd9cbe00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc15edc0 con 0x7fd9cc14e270 2024-04-24T08:21:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.694+0000 7fd9cbe00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fd9cc15ec40 con 0x7fd9cc0c9270 2024-04-24T08:21:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.694+0000 7fd9cbe00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fd9cc0d0110 con 0x7fd9b00050a0 2024-04-24T08:21:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9c2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9cc14e270 0x7fd9cc0cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9c2000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9cc14e270 0x7fd9cc0cdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52386/0 (socket says 172.21.15.69:52386) 2024-04-24T08:21:39.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9c2000640 1 -- 172.21.15.69:0/2519909503 learned_addr learned my addr 172.21.15.69:0/2519909503 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:39.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9caa00640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b00050a0 0x7fd9cc14d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9c2000640 1 -- 172.21.15.69:0/2519909503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b00050a0 msgr2=0x7fd9cc14d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9c2000640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fd9b00050a0 0x7fd9cc14d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9c2000640 1 -- 172.21.15.69:0/2519909503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9cc0c9270 msgr2=0x7fd9cc0c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9c2000640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd9cc0c9270 0x7fd9cc0c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9c2000640 1 -- 172.21.15.69:0/2519909503 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9cc0b9440 con 0x7fd9cc14e270 2024-04-24T08:21:39.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9c2000640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9cc14e270 0x7fd9cc0cdd80 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400f820 tx=0x7fd9b400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:39.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.695+0000 7fd9c3400640 1 -- 172.21.15.69:0/2519909503 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b40050d0 con 0x7fd9cc14e270 2024-04-24T08:21:39.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.696+0000 7fd9c3400640 1 -- 172.21.15.69:0/2519909503 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b4018070 con 0x7fd9cc14e270 2024-04-24T08:21:39.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.696+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9cc0b9720 con 0x7fd9cc14e270 2024-04-24T08:21:39.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.696+0000 7fd9c3400640 1 -- 172.21.15.69:0/2519909503 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4005490 con 0x7fd9cc14e270 2024-04-24T08:21:39.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.696+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9cc0b9ce0 con 0x7fd9cc14e270 2024-04-24T08:21:39.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.697+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd9cc075d50 con 0x7fd9cc14e270 2024-04-24T08:21:39.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.698+0000 7fd9c3400640 1 -- 172.21.15.69:0/2519909503 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9b400d650 con 0x7fd9cc14e270 2024-04-24T08:21:39.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.699+0000 7fd9c3400640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c920 0x7fd9a806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.699+0000 7fd9c3400640 1 -- 172.21.15.69:0/2519909503 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fd9b408b640 con 0x7fd9cc14e270 2024-04-24T08:21:39.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.699+0000 7fd9c3400640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x7fd9a8072920 0x7fd9a8074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.699+0000 7fd9c3400640 1 -- 172.21.15.69:0/2519909503 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9b400f020 con 0x7fd9a8072920 2024-04-24T08:21:39.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.699+0000 7fd9c3400640 1 -- 172.21.15.69:0/2519909503 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fd9b4092050 con 0x7fd9cc14e270 2024-04-24T08:21:39.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.699+0000 7fd9cb400640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x7fd9a8072920 0x7fd9a8074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.699+0000 7fd9caa00640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c920 0x7fd9a806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.700+0000 7fd9cb400640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x7fd9a8072920 0x7fd9a8074d40 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:39.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.700+0000 7fd9caa00640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c920 0x7fd9a806ede0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc001eb0 tx=0x7fd9bc009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:39.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.701+0000 7fd9c3400640 1 -- 172.21.15.69:0/2519909503 <== osd.0 v2:172.21.15.69:6800/146737935 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd9b400f020 con 0x7fd9a8072920 2024-04-24T08:21:39.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.716+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 --> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd9cc068f10 con 0x7fd9a8072920 2024-04-24T08:21:39.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.716+0000 7fd9c3400640 1 -- 172.21.15.69:0/2519909503 <== osd.0 v2:172.21.15.69:6800/146737935 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd9cc068f10 con 0x7fd9a8072920 2024-04-24T08:21:39.716 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:39.717 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T08:21:39.717 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:39.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.716+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x7fd9a8072920 msgr2=0x7fd9a8074d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.716+0000 7fd9cbe00640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.69:6800/146737935,v1:172.21.15.69:6801/146737935] conn(0x7fd9a8072920 0x7fd9a8074d40 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.717+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c920 msgr2=0x7fd9a806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.717+0000 7fd9cbe00640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fd9a806c920 0x7fd9a806ede0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc001eb0 tx=0x7fd9bc009040 comp rx=0 tx=0).stop 2024-04-24T08:21:39.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.717+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9cc14e270 msgr2=0x7fd9cc0cdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.717+0000 7fd9cbe00640 1 --2- 172.21.15.69:0/2519909503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd9cc14e270 0x7fd9cc0cdd80 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400f820 tx=0x7fd9b400fce0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.717+0000 7fd9cb400640 1 -- 172.21.15.69:0/2519909503 reap_dead start 2024-04-24T08:21:39.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.717+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 shutdown_connections 2024-04-24T08:21:39.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.717+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 >> 172.21.15.69:0/2519909503 conn(0x7fd9cc06c210 msgr2=0x7fd9cc0b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:39.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.717+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 shutdown_connections 2024-04-24T08:21:39.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.717+0000 7fd9cbe00640 1 -- 172.21.15.69:0/2519909503 wait complete. 2024-04-24T08:21:39.735 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-24T08:21:39.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.929+0000 7f7eb8e00640 1 Processor -- start 2024-04-24T08:21:39.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.929+0000 7f7eb8e00640 1 -- start start 2024-04-24T08:21:39.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb8e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7eb40d0300 0x7f7eb406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb8e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7eb4145fe0 0x7f7eb4160540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb8e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7eb40cf3d0 con 0x7f7eb4145a90 2024-04-24T08:21:39.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb8e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7eb40cf6b0 con 0x7f7eb40d0300 2024-04-24T08:21:39.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb8e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7eb40cf990 con 0x7f7eb4145fe0 2024-04-24T08:21:39.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb1400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7eb4145a90 0x7f7eb40d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42688/0 (socket says 172.21.15.69:42688) 2024-04-24T08:21:39.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb1400640 1 -- 172.21.15.69:0/1035821188 learned_addr learned my addr 172.21.15.69:0/1035821188 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:39.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4128608948 0 0) 0x7f7eb40cf3d0 con 0x7f7eb4145a90 2024-04-24T08:21:39.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb2800640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7eb4145fe0 0x7f7eb4160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.933 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e98003520 con 0x7f7eb4145a90 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.931+0000 7f7eb1e00640 1 --2- 172.21.15.69:0/1035821188 >> v2:172.21.15.121:3300/0 conn(0x7f7eb40d0300 0x7f7eb406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1290240239 0 0) 0x7f7e98003520 con 0x7f7eb4145a90 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7eb4145fe0 msgr2=0x7f7eb4160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7eb4145fe0 0x7f7eb4160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 >> v2:172.21.15.121:3300/0 conn(0x7f7eb40d0300 msgr2=0x7f7eb406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 --2- 172.21.15.69:0/1035821188 >> v2:172.21.15.121:3300/0 conn(0x7f7eb40d0300 0x7f7eb406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb40cfc70 con 0x7f7eb4145a90 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f7ea0001670 con 0x7f7eb4145a90 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 >> v1:172.21.15.69:6789/0 conn(0x7f7eb4145a90 legacy=0x7f7eb40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e980039b0 0x7f7e98003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7eb4145fe0 0x7f7e98004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e98004d40 0x7f7e980051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb40cf6b0 con 0x7f7e980039b0 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb40cf990 con 0x7f7e98004d40 2024-04-24T08:21:39.934 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7e9800b9f0 con 0x7f7eb4145fe0 2024-04-24T08:21:39.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb1e00640 1 --2- 172.21.15.69:0/1035821188 >> v2:172.21.15.121:3300/0 conn(0x7f7eb40d0300 0x7f7eb406e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:39.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.932+0000 7f7eb1e00640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e980039b0 0x7f7e98003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.933+0000 7f7eb1400640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7eb4145fe0 0x7f7e98004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.933+0000 7f7eb1e00640 1 -- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7eb4145fe0 msgr2=0x7f7e98004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:39.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.933+0000 7f7eb2800640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e98004d40 0x7f7e980051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.933+0000 7f7eb1e00640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7eb4145fe0 0x7f7e98004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.933+0000 7f7eb1e00640 1 -- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e98004d40 msgr2=0x7f7e980051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:39.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.933+0000 7f7eb1e00640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7e98004d40 0x7f7e980051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.935 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.933+0000 7f7eb1e00640 1 -- 172.21.15.69:0/1035821188 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e98003520 con 0x7f7e980039b0 2024-04-24T08:21:39.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.934+0000 7f7eb1e00640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e980039b0 0x7f7e98003dd0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f7ea40098a0 tx=0x7f7e98005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e61aeb344a275732 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:39.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.934+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4037070 con 0x7f7e980039b0 2024-04-24T08:21:39.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.934+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ea402ad20 con 0x7f7e980039b0 2024-04-24T08:21:39.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.934+0000 7f7eb0a00640 1 -- 172.21.15.69:0/1035821188 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ea400f040 con 0x7f7e980039b0 2024-04-24T08:21:39.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.935+0000 7f7eb8e00640 1 -- 172.21.15.69:0/1035821188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e980039b0 msgr2=0x7f7e98003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.935+0000 7f7eb8e00640 1 --2- 172.21.15.69:0/1035821188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e980039b0 0x7f7e98003dd0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f7ea40098a0 tx=0x7f7e98005830 comp rx=0 tx=0).stop 2024-04-24T08:21:39.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.935+0000 7f7eb2800640 1 -- 172.21.15.69:0/1035821188 reap_dead start 2024-04-24T08:21:39.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.935+0000 7f7eb8e00640 1 -- 172.21.15.69:0/1035821188 shutdown_connections 2024-04-24T08:21:39.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.935+0000 7f7eb8e00640 1 -- 172.21.15.69:0/1035821188 >> 172.21.15.69:0/1035821188 conn(0x7f7eb4067d90 msgr2=0x7f7eb406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:39.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.935+0000 7f7eb8e00640 1 -- 172.21.15.69:0/1035821188 shutdown_connections 2024-04-24T08:21:39.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.935+0000 7f7eb8e00640 1 -- 172.21.15.69:0/1035821188 wait complete. 2024-04-24T08:21:39.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.937+0000 7f7eb8e00640 1 Processor -- start 2024-04-24T08:21:39.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.937+0000 7f7eb8e00640 1 -- start start 2024-04-24T08:21:39.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb8e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e980039b0 0x7f7eb415c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb8e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7eb4141640 0x7f7eb4141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb8e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb413c5c0 0x7f7eb413ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb8e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb41458f0 con 0x7f7eb413c5c0 2024-04-24T08:21:39.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb8e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7eb4145770 con 0x7f7eb4141640 2024-04-24T08:21:39.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb8e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7eb40cdc30 con 0x7f7e980039b0 2024-04-24T08:21:39.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb413c5c0 0x7f7eb413ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb2800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb413c5c0 0x7f7eb413ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52402/0 (socket says 172.21.15.69:52402) 2024-04-24T08:21:39.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb2800640 1 -- 172.21.15.69:0/2495813776 learned_addr learned my addr 172.21.15.69:0/2495813776 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:39.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb1e00640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e980039b0 0x7f7eb415c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb2800640 1 -- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e980039b0 msgr2=0x7f7eb415c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb2800640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e980039b0 0x7f7eb415c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb2800640 1 -- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7eb4141640 msgr2=0x7f7eb4141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:39.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb2800640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7eb4141640 0x7f7eb4141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:39.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.938+0000 7f7eb2800640 1 -- 172.21.15.69:0/2495813776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7eb40ccc70 con 0x7f7eb413c5c0 2024-04-24T08:21:39.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.939+0000 7f7eb2800640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb413c5c0 0x7f7eb413ca40 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009fd0 tx=0x7f7eac00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:39.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.939+0000 7f7eaa000640 1 -- 172.21.15.69:0/2495813776 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eac01c070 con 0x7f7eb413c5c0 2024-04-24T08:21:39.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.939+0000 7f7eaa000640 1 -- 172.21.15.69:0/2495813776 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7eac017400 con 0x7f7eb413c5c0 2024-04-24T08:21:39.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.939+0000 7f7eb8e00640 1 -- 172.21.15.69:0/2495813776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7eb413cf80 con 0x7f7eb413c5c0 2024-04-24T08:21:39.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.939+0000 7f7eaa000640 1 -- 172.21.15.69:0/2495813776 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eac0177f0 con 0x7f7eb413c5c0 2024-04-24T08:21:39.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.939+0000 7f7eb8e00640 1 -- 172.21.15.69:0/2495813776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7eb413d1f0 con 0x7f7eb413c5c0 2024-04-24T08:21:39.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.940+0000 7f7eb8e00640 1 -- 172.21.15.69:0/2495813776 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7eb4068a80 con 0x7f7eb413c5c0 2024-04-24T08:21:39.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.941+0000 7f7eaa000640 1 -- 172.21.15.69:0/2495813776 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7eac005d10 con 0x7f7eb413c5c0 2024-04-24T08:21:39.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.942+0000 7f7eaa000640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e9406c8f0 0x7f7e9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:39.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.942+0000 7f7eaa000640 1 -- 172.21.15.69:0/2495813776 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f7eac08bb80 con 0x7f7eb413c5c0 2024-04-24T08:21:39.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.943+0000 7f7eb1e00640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e9406c8f0 0x7f7e9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:39.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.944+0000 7f7eb1e00640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e9406c8f0 0x7f7e9406edb0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7ea40098a0 tx=0x7f7ea402e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:39.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:39.946+0000 7f7eaa000640 1 -- 172.21.15.69:0/2495813776 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7eac012070 con 0x7f7eb413c5c0 2024-04-24T08:21:40.143 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.143+0000 7f7eb8e00640 1 -- 172.21.15.69:0/2495813776 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f7eb40d1120 con 0x7f7e9406c8f0 2024-04-24T08:21:40.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.144+0000 7f7eaa000640 1 -- 172.21.15.69:0/2495813776 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7eb40d1120 con 0x7f7e9406c8f0 2024-04-24T08:21:40.144 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T08:21:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 -- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e9406c8f0 msgr2=0x7f7e9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e9406c8f0 0x7f7e9406edb0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7ea40098a0 tx=0x7f7ea402e000 comp rx=0 tx=0).stop 2024-04-24T08:21:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 -- 172.21.15.69:0/2495813776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb413c5c0 msgr2=0x7f7eb413ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb413c5c0 0x7f7eb413ca40 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f7eac009fd0 tx=0x7f7eac00f4a0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 -- 172.21.15.69:0/2495813776 shutdown_connections 2024-04-24T08:21:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7e9406c8f0 0x7f7e9406edb0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7eb413c5c0 0x7f7eb413ca40 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7eb4141640 0x7f7eb4141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 --2- 172.21.15.69:0/2495813776 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7e980039b0 0x7f7eb415c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 -- 172.21.15.69:0/2495813776 >> 172.21.15.69:0/2495813776 conn(0x7f7eb4067d90 msgr2=0x7f7eb415f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:40.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.146+0000 7f7eb8e00640 1 -- 172.21.15.69:0/2495813776 shutdown_connections 2024-04-24T08:21:40.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.147+0000 7f7eb8e00640 1 -- 172.21.15.69:0/2495813776 wait complete. 2024-04-24T08:21:40.165 INFO:tasks.ceph:Scrubbing osd.1 2024-04-24T08:21:40.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T08:21:40.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.361+0000 7f7d2d400640 1 Processor -- start 2024-04-24T08:21:40.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.362+0000 7f7d2d400640 1 -- start start 2024-04-24T08:21:40.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.363+0000 7f7d2d400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d280d4d30 0x7f7d280d0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.363+0000 7f7d2d400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280d2ba0 0x7f7d280d3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.363+0000 7f7d2d400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d280d1e80 con 0x7f7d2815e860 2024-04-24T08:21:40.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.363+0000 7f7d2d400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f7d280d3550 con 0x7f7d280d4d30 2024-04-24T08:21:40.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.363+0000 7f7d2d400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d280cfc80 con 0x7f7d280d2ba0 2024-04-24T08:21:40.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f7d2815e860 0x7f7d280d1670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42702/0 (socket says 172.21.15.69:42702) 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27e00640 1 -- 172.21.15.69:0/2197700534 learned_addr learned my addr 172.21.15.69:0/2197700534 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d2ca00640 1 --2- 172.21.15.69:0/2197700534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280d2ba0 0x7f7d280d3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d1f400640 1 --2- 172.21.15.69:0/2197700534 >> v2:172.21.15.121:3300/0 conn(0x7f7d280d4d30 0x7f7d280d0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1712031027 0 0) 0x7f7d280d1e80 con 0x7f7d2815e860 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d0c003520 con 0x7f7d2815e860 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3541009562 0 0) 0x7f7d0c003520 con 0x7f7d2815e860 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280d2ba0 msgr2=0x7f7d280d3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 --2- 172.21.15.69:0/2197700534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280d2ba0 0x7f7d280d3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 >> v2:172.21.15.121:3300/0 conn(0x7f7d280d4d30 msgr2=0x7f7d280d0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 --2- 172.21.15.69:0/2197700534 >> v2:172.21.15.121:3300/0 conn(0x7f7d280d4d30 0x7f7d280d0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d280cf350 con 0x7f7d2815e860 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f7d10001670 con 0x7f7d2815e860 2024-04-24T08:21:40.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 >> v1:172.21.15.69:6789/0 conn(0x7f7d2815e860 legacy=0x7f7d280d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 --2- 172.21.15.69:0/2197700534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280d2ba0 0x7f7d0c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.364+0000 7f7d27400640 1 --2- 172.21.15.69:0/2197700534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 0x7f7d0c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.365+0000 7f7d27400640 1 --2- 172.21.15.69:0/2197700534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d0c004d40 0x7f7d0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.365+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d280d3550 con 0x7f7d0c003f30 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.365+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7d280cfc80 con 0x7f7d0c004d40 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.365+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0c00b9c0 con 0x7f7d280d2ba0 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.365+0000 7f7d1f400640 1 --2- 172.21.15.69:0/2197700534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 0x7f7d0c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.365+0000 7f7d1f400640 1 -- 172.21.15.69:0/2197700534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280d2ba0 msgr2=0x7f7d0c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.365+0000 7f7d1f400640 1 --2- 172.21.15.69:0/2197700534 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280d2ba0 0x7f7d0c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.365+0000 7f7d1f400640 1 -- 172.21.15.69:0/2197700534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d0c004d40 msgr2=0x7f7d0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.365+0000 7f7d1f400640 1 --2- 172.21.15.69:0/2197700534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d0c004d40 0x7f7d0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.365+0000 7f7d1f400640 1 -- 172.21.15.69:0/2197700534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0c003520 con 0x7f7d0c003f30 2024-04-24T08:21:40.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.366+0000 7f7d1f400640 1 --2- 172.21.15.69:0/2197700534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 0x7f7d0c004800 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009ac0 tx=0x7f7d0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a97312d8ddfe32c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:40.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.366+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18036070 con 0x7f7d0c003f30 2024-04-24T08:21:40.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.366+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d18031400 con 0x7f7d0c003f30 2024-04-24T08:21:40.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.366+0000 7f7d27400640 1 -- 172.21.15.69:0/2197700534 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d18009690 con 0x7f7d0c003f30 2024-04-24T08:21:40.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.367+0000 7f7d2d400640 1 -- 172.21.15.69:0/2197700534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 msgr2=0x7f7d0c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.367+0000 7f7d2d400640 1 --2- 172.21.15.69:0/2197700534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 0x7f7d0c004800 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009ac0 tx=0x7f7d0c005830 comp rx=0 tx=0).stop 2024-04-24T08:21:40.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.367+0000 7f7d2ca00640 1 -- 172.21.15.69:0/2197700534 reap_dead start 2024-04-24T08:21:40.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.367+0000 7f7d2d400640 1 -- 172.21.15.69:0/2197700534 shutdown_connections 2024-04-24T08:21:40.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.367+0000 7f7d2d400640 1 -- 172.21.15.69:0/2197700534 >> 172.21.15.69:0/2197700534 conn(0x7f7d2806c210 msgr2=0x7f7d2809d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:40.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.367+0000 7f7d2d400640 1 -- 172.21.15.69:0/2197700534 shutdown_connections 2024-04-24T08:21:40.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.367+0000 7f7d2d400640 1 -- 172.21.15.69:0/2197700534 wait complete. 2024-04-24T08:21:40.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.369+0000 7f7d2d400640 1 Processor -- start 2024-04-24T08:21:40.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.369+0000 7f7d2d400640 1 -- start start 2024-04-24T08:21:40.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.370+0000 7f7d2d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 0x7f7d28150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.370+0000 7f7d2d400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280cd750 0x7f7d280c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.370+0000 7f7d2d400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d280cdbd0 0x7f7d280c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.370+0000 7f7d2d400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2815ec40 con 0x7f7d0c003f30 2024-04-24T08:21:40.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.370+0000 7f7d2d400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f7d280cf910 con 0x7f7d280cdbd0 2024-04-24T08:21:40.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.370+0000 7f7d2d400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f7d2815edc0 con 0x7f7d280cd750 2024-04-24T08:21:40.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.370+0000 7f7d27e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 0x7f7d28150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.370+0000 7f7d27e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 0x7f7d28150b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52420/0 (socket says 172.21.15.69:52420) 2024-04-24T08:21:40.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.370+0000 7f7d27e00640 1 -- 172.21.15.69:0/3323520097 learned_addr learned my addr 172.21.15.69:0/3323520097 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:40.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d1f400640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280cd750 0x7f7d280c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d27e00640 1 -- 172.21.15.69:0/3323520097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280cd750 msgr2=0x7f7d280c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d27e00640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d280cd750 0x7f7d280c91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d27e00640 1 -- 172.21.15.69:0/3323520097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d280cdbd0 msgr2=0x7f7d280c96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d27e00640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d280cdbd0 0x7f7d280c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d27e00640 1 -- 172.21.15.69:0/3323520097 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d280b4ef0 con 0x7f7d0c003f30 2024-04-24T08:21:40.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d27e00640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 0x7f7d28150b00 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000c820 tx=0x7f7d1000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:40.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d24c00640 1 -- 172.21.15.69:0/3323520097 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10018070 con 0x7f7d0c003f30 2024-04-24T08:21:40.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d24c00640 1 -- 172.21.15.69:0/3323520097 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d100074e0 con 0x7f7d0c003f30 2024-04-24T08:21:40.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d280b51d0 con 0x7f7d0c003f30 2024-04-24T08:21:40.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d24c00640 1 -- 172.21.15.69:0/3323520097 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d100016f0 con 0x7f7d0c003f30 2024-04-24T08:21:40.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.371+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d280b5790 con 0x7f7d0c003f30 2024-04-24T08:21:40.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.372+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7d28075d50 con 0x7f7d0c003f30 2024-04-24T08:21:40.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.374+0000 7f7d24c00640 1 -- 172.21.15.69:0/3323520097 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d1001b070 con 0x7f7d0c003f30 2024-04-24T08:21:40.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.374+0000 7f7d24c00640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d0406c8f0 0x7f7d0406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.375+0000 7f7d24c00640 1 -- 172.21.15.69:0/3323520097 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f7d1008b4f0 con 0x7f7d0c003f30 2024-04-24T08:21:40.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.375+0000 7f7d24c00640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x7f7d040728f0 0x7f7d04074d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.375+0000 7f7d24c00640 1 -- 172.21.15.69:0/3323520097 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d10014e50 con 0x7f7d040728f0 2024-04-24T08:21:40.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.375+0000 7f7d24c00640 1 -- 172.21.15.69:0/3323520097 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f7d1008b8a0 con 0x7f7d0c003f30 2024-04-24T08:21:40.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.375+0000 7f7d1f400640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d0406c8f0 0x7f7d0406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.375+0000 7f7d2ca00640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x7f7d040728f0 0x7f7d04074d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.376+0000 7f7d2ca00640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x7f7d040728f0 0x7f7d04074d10 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:40.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.376+0000 7f7d1f400640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d0406c8f0 0x7f7d0406edb0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7d18033000 tx=0x7f7d18008030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:40.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.376+0000 7f7d24c00640 1 -- 172.21.15.69:0/3323520097 <== osd.1 v2:172.21.15.69:6815/153736297 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d10014e50 con 0x7f7d040728f0 2024-04-24T08:21:40.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.390+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 --> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7d2806dbb0 con 0x7f7d040728f0 2024-04-24T08:21:40.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.391+0000 7f7d24c00640 1 -- 172.21.15.69:0/3323520097 <== osd.1 v2:172.21.15.69:6815/153736297 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7d2806dbb0 con 0x7f7d040728f0 2024-04-24T08:21:40.391 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:40.391 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T08:21:40.392 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:40.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.391+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x7f7d040728f0 msgr2=0x7f7d04074d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.391+0000 7f7d2d400640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.69:6815/153736297,v1:172.21.15.69:6817/153736297] conn(0x7f7d040728f0 0x7f7d04074d10 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.392+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d0406c8f0 msgr2=0x7f7d0406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.392+0000 7f7d2d400640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f7d0406c8f0 0x7f7d0406edb0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7d18033000 tx=0x7f7d18008030 comp rx=0 tx=0).stop 2024-04-24T08:21:40.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.392+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 msgr2=0x7f7d28150b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.392+0000 7f7d2d400640 1 --2- 172.21.15.69:0/3323520097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d0c003f30 0x7f7d28150b00 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f7d1000c820 tx=0x7f7d1000cce0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.392+0000 7f7d2ca00640 1 -- 172.21.15.69:0/3323520097 reap_dead start 2024-04-24T08:21:40.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.392+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 shutdown_connections 2024-04-24T08:21:40.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.392+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 >> 172.21.15.69:0/3323520097 conn(0x7f7d2806c210 msgr2=0x7f7d280c9f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:40.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.392+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 shutdown_connections 2024-04-24T08:21:40.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.392+0000 7f7d2d400640 1 -- 172.21.15.69:0/3323520097 wait complete. 2024-04-24T08:21:40.409 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-24T08:21:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.601+0000 7f48a6200640 1 Processor -- start 2024-04-24T08:21:40.602 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.602+0000 7f48a6200640 1 -- start start 2024-04-24T08:21:40.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.603+0000 7f48a6200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f48a00c7ad0 0x7f48a009b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.603+0000 7f48a6200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a015cc30 con 0x7f48a015c850 2024-04-24T08:21:40.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.603+0000 7f48a6200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48a009b950 con 0x7f48a00d08b0 2024-04-24T08:21:40.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.603+0000 7f48a6200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f48a00c6630 con 0x7f48a00c7ad0 2024-04-24T08:21:40.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.603+0000 7f489f400640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f48a015c850 0x7f48a00d01a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42712/0 (socket says 172.21.15.69:42712) 2024-04-24T08:21:40.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.603+0000 7f489f400640 1 -- 172.21.15.69:0/1203921077 learned_addr learned my addr 172.21.15.69:0/1203921077 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1033893129 0 0) 0x7f48a015cc30 con 0x7f48a015c850 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4884003520 con 0x7f48a015c850 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489fe00640 1 --2- 172.21.15.69:0/1203921077 >> v2:172.21.15.121:3300/0 conn(0x7f48a00c7ad0 0x7f48a009b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3205789506 0 0) 0x7f4884003520 con 0x7f48a015c850 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 >> v2:172.21.15.121:3300/0 conn(0x7f48a00c7ad0 msgr2=0x7f48a009b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 --2- 172.21.15.69:0/1203921077 >> v2:172.21.15.121:3300/0 conn(0x7f48a00c7ad0 0x7f48a009b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 >> v1:172.21.15.121:6789/0 conn(0x7f48a00d08b0 legacy=0x7f48a009ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a009bc30 con 0x7f48a015c850 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f4890001670 con 0x7f48a015c850 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 >> v1:172.21.15.69:6789/0 conn(0x7f48a015c850 legacy=0x7f48a00d01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 --2- 172.21.15.69:0/1203921077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48840039b0 0x7f4884003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.606 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 --2- 172.21.15.69:0/1203921077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f4884004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 --2- 172.21.15.69:0/1203921077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48840051c0 0x7f488400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f48a00c6630 con 0x7f4884004800 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f488400c110 con 0x7f48840039b0 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f488400c310 con 0x7f48840051c0 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.604+0000 7f489ea00640 1 --2- 172.21.15.69:0/1203921077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f4884004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.605+0000 7f489ea00640 1 -- 172.21.15.69:0/1203921077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48840051c0 msgr2=0x7f488400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.605+0000 7f489ea00640 1 --2- 172.21.15.69:0/1203921077 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48840051c0 0x7f488400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.605+0000 7f489ea00640 1 -- 172.21.15.69:0/1203921077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48840039b0 msgr2=0x7f4884003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.605+0000 7f489ea00640 1 --2- 172.21.15.69:0/1203921077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48840039b0 0x7f4884003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.605+0000 7f489ea00640 1 -- 172.21.15.69:0/1203921077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4884003520 con 0x7f4884004800 2024-04-24T08:21:40.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.605+0000 7f489fe00640 1 -- 172.21.15.69:0/1203921077 reap_dead start 2024-04-24T08:21:40.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.606+0000 7f489ea00640 1 --2- 172.21.15.69:0/1203921077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f4884004c80 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f488c001750 tx=0x7f4884005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274469a3dbd5df56 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:40.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.606+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c001370 con 0x7f4884004800 2024-04-24T08:21:40.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.606+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f488c0052d0 con 0x7f4884004800 2024-04-24T08:21:40.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.606+0000 7f489e000640 1 -- 172.21.15.69:0/1203921077 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488c02abc0 con 0x7f4884004800 2024-04-24T08:21:40.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.606+0000 7f48a6200640 1 -- 172.21.15.69:0/1203921077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 msgr2=0x7f4884004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.606+0000 7f48a6200640 1 --2- 172.21.15.69:0/1203921077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f4884004c80 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f488c001750 tx=0x7f4884005830 comp rx=0 tx=0).stop 2024-04-24T08:21:40.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.607+0000 7f48a6200640 1 -- 172.21.15.69:0/1203921077 shutdown_connections 2024-04-24T08:21:40.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.607+0000 7f48a6200640 1 --2- 172.21.15.69:0/1203921077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f4884004c80 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.607+0000 7f48a6200640 1 -- 172.21.15.69:0/1203921077 >> 172.21.15.69:0/1203921077 conn(0x7f48a00b40c0 msgr2=0x7f48a00b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:40.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.607+0000 7f48a6200640 1 -- 172.21.15.69:0/1203921077 shutdown_connections 2024-04-24T08:21:40.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.607+0000 7f48a6200640 1 -- 172.21.15.69:0/1203921077 wait complete. 2024-04-24T08:21:40.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.608+0000 7f48a6200640 1 Processor -- start 2024-04-24T08:21:40.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.608+0000 7f48a6200640 1 -- start start 2024-04-24T08:21:40.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.609+0000 7f48a6200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f48a0156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.609+0000 7f48a6200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a00ca150 0x7f48a00b5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.609+0000 7f48a6200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48a00b7660 0x7f48a00b7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.609+0000 7f48a6200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f48a00d0320 con 0x7f4884004800 2024-04-24T08:21:40.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.609+0000 7f48a6200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f48a00d01a0 con 0x7f48a00ca150 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.609+0000 7f48a6200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f48a00d04a0 con 0x7f48a00b7660 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.609+0000 7f489f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f48a0156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.610+0000 7f489f400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f48a0156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52434/0 (socket says 172.21.15.69:52434) 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.610+0000 7f489f400640 1 -- 172.21.15.69:0/1659501843 learned_addr learned my addr 172.21.15.69:0/1659501843 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.610+0000 7f489fe00640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48a00b7660 0x7f48a00b7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.610+0000 7f489f400640 1 -- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48a00b7660 msgr2=0x7f48a00b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.610+0000 7f489ea00640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a00ca150 0x7f48a00b5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.610+0000 7f489f400640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48a00b7660 0x7f48a00b7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.610+0000 7f489f400640 1 -- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a00ca150 msgr2=0x7f48a00b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.610+0000 7f489f400640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a00ca150 0x7f48a00b5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.610+0000 7f489f400640 1 -- 172.21.15.69:0/1659501843 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48a00b5580 con 0x7f4884004800 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.610+0000 7f489f400640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f48a0156d40 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f489000c760 tx=0x7f489000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.611+0000 7f4897400640 1 -- 172.21.15.69:0/1659501843 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4890018070 con 0x7f4884004800 2024-04-24T08:21:40.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.611+0000 7f4897400640 1 -- 172.21.15.69:0/1659501843 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48900074e0 con 0x7f4884004800 2024-04-24T08:21:40.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.611+0000 7f4897400640 1 -- 172.21.15.69:0/1659501843 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4890004a20 con 0x7f4884004800 2024-04-24T08:21:40.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.611+0000 7f48a6200640 1 -- 172.21.15.69:0/1659501843 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48a00c4ec0 con 0x7f4884004800 2024-04-24T08:21:40.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.611+0000 7f48a6200640 1 -- 172.21.15.69:0/1659501843 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48a015a620 con 0x7f4884004800 2024-04-24T08:21:40.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.612+0000 7f48a6200640 1 -- 172.21.15.69:0/1659501843 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48a0069190 con 0x7f4884004800 2024-04-24T08:21:40.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.613+0000 7f4897400640 1 -- 172.21.15.69:0/1659501843 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f489001b070 con 0x7f4884004800 2024-04-24T08:21:40.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.614+0000 7f4897400640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f488006c8f0 0x7f488006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:40.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.614+0000 7f4897400640 1 -- 172.21.15.69:0/1659501843 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f489008ae10 con 0x7f4884004800 2024-04-24T08:21:40.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.614+0000 7f489ea00640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f488006c8f0 0x7f488006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:40.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.615+0000 7f489ea00640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f488006c8f0 0x7f488006edb0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f488c032000 tx=0x7f488c02e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:40.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.618+0000 7f4897400640 1 -- 172.21.15.69:0/1659501843 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f489008e030 con 0x7f4884004800 2024-04-24T08:21:40.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.823+0000 7f48a6200640 1 -- 172.21.15.69:0/1659501843 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f48a0075770 con 0x7f488006c8f0 2024-04-24T08:21:40.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.825+0000 7f4897400640 1 -- 172.21.15.69:0/1659501843 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f488c0051d0 con 0x7f488006c8f0 2024-04-24T08:21:40.825 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T08:21:40.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.827+0000 7f48a6200640 1 -- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f488006c8f0 msgr2=0x7f488006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.827+0000 7f48a6200640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f488006c8f0 0x7f488006edb0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f488c032000 tx=0x7f488c02e000 comp rx=0 tx=0).stop 2024-04-24T08:21:40.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.827+0000 7f48a6200640 1 -- 172.21.15.69:0/1659501843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 msgr2=0x7f48a0156d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:40.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.827+0000 7f48a6200640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f48a0156d40 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f489000c760 tx=0x7f489000cc20 comp rx=0 tx=0).stop 2024-04-24T08:21:40.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.827+0000 7f48a6200640 1 -- 172.21.15.69:0/1659501843 shutdown_connections 2024-04-24T08:21:40.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.827+0000 7f48a6200640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f488006c8f0 0x7f488006edb0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.827+0000 7f48a6200640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f48a00b7660 0x7f48a00b7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.827+0000 7f48a6200640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48a00ca150 0x7f48a00b5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.827+0000 7f48a6200640 1 --2- 172.21.15.69:0/1659501843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4884004800 0x7f48a0156d40 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:40.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.827+0000 7f48a6200640 1 -- 172.21.15.69:0/1659501843 >> 172.21.15.69:0/1659501843 conn(0x7f48a00b40c0 msgr2=0x7f48a00d0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:40.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.828+0000 7f48a6200640 1 -- 172.21.15.69:0/1659501843 shutdown_connections 2024-04-24T08:21:40.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:40.828+0000 7f48a6200640 1 -- 172.21.15.69:0/1659501843 wait complete. 2024-04-24T08:21:40.847 INFO:tasks.ceph:Scrubbing osd.2 2024-04-24T08:21:40.847 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T08:21:41.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.042+0000 7fed9b200640 1 Processor -- start 2024-04-24T08:21:41.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.042+0000 7fed9b200640 1 -- start start 2024-04-24T08:21:41.046 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.044+0000 7fed9b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c15e930 0x7fed9c1623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.044+0000 7fed9b200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fed9c0af5b0 0x7fed9c0cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.044+0000 7fed9b200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed9c1632b0 con 0x7fed9c0cce30 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.044+0000 7fed9b200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fed9c15ed10 con 0x7fed9c0af5b0 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.044+0000 7fed9b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed9c0cacf0 con 0x7fed9c15e930 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.044+0000 7fed9a800640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fed9c0cce30 0x7fed9c0ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42720/0 (socket says 172.21.15.69:42720) 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.044+0000 7fed9a800640 1 -- 172.21.15.69:0/3348484883 learned_addr learned my addr 172.21.15.69:0/3348484883 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.044+0000 7fed99e00640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c15e930 0x7fed9c1623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.044+0000 7fed93e00640 1 --2- 172.21.15.69:0/3348484883 >> v2:172.21.15.121:3300/0 conn(0x7fed9c0af5b0 0x7fed9c0cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.044+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2029253236 0 0) 0x7fed9c1632b0 con 0x7fed9c0cce30 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed7c003520 con 0x7fed9c0cce30 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4272716455 0 0) 0x7fed7c003520 con 0x7fed9c0cce30 2024-04-24T08:21:41.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c15e930 msgr2=0x7fed9c1623a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c15e930 0x7fed9c1623a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 >> v2:172.21.15.121:3300/0 conn(0x7fed9c0af5b0 msgr2=0x7fed9c0cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 --2- 172.21.15.69:0/3348484883 >> v2:172.21.15.121:3300/0 conn(0x7fed9c0af5b0 0x7fed9c0cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9c162a10 con 0x7fed9c0cce30 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fed940019f0 con 0x7fed9c0cce30 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 >> v1:172.21.15.69:6789/0 conn(0x7fed9c0cce30 legacy=0x7fed9c0ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c0039b0 0x7fed7c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c15e930 0x7fed7c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed7c004d40 0x7fed7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c0cacf0 con 0x7fed7c004d40 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fed7c00ba70 con 0x7fed7c0039b0 2024-04-24T08:21:41.048 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed7c00bc20 con 0x7fed9c15e930 2024-04-24T08:21:41.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.045+0000 7fed9a800640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed7c004d40 0x7fed7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.046+0000 7fed99e00640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c0039b0 0x7fed7c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.046+0000 7fed93e00640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c15e930 0x7fed7c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.046+0000 7fed9a800640 1 -- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c15e930 msgr2=0x7fed7c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:41.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.046+0000 7fed9a800640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c15e930 0x7fed7c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.046+0000 7fed9a800640 1 -- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c0039b0 msgr2=0x7fed7c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:41.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.046+0000 7fed9a800640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c0039b0 0x7fed7c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.046+0000 7fed9a800640 1 -- 172.21.15.69:0/3348484883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7c003520 con 0x7fed7c004d40 2024-04-24T08:21:41.049 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.046+0000 7fed93e00640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c15e930 0x7fed7c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:41.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.047+0000 7fed9a800640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed7c004d40 0x7fed7c0051c0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fed9400a7a0 tx=0x7fed7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64666359ea713399 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.047+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9400a3f0 con 0x7fed7c004d40 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.047+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed9402f070 con 0x7fed7c004d40 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.047+0000 7fed99400640 1 -- 172.21.15.69:0/3348484883 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9402bc20 con 0x7fed7c004d40 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.047+0000 7fed9b200640 1 -- 172.21.15.69:0/3348484883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed7c004d40 msgr2=0x7fed7c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.047+0000 7fed9b200640 1 --2- 172.21.15.69:0/3348484883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed7c004d40 0x7fed7c0051c0 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7fed9400a7a0 tx=0x7fed7c005830 comp rx=0 tx=0).stop 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.047+0000 7fed9a800640 1 -- 172.21.15.69:0/3348484883 reap_dead start 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.048+0000 7fed9b200640 1 -- 172.21.15.69:0/3348484883 shutdown_connections 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.048+0000 7fed9b200640 1 -- 172.21.15.69:0/3348484883 >> 172.21.15.69:0/3348484883 conn(0x7fed9c0b1c50 msgr2=0x7fed9c0b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.048+0000 7fed9b200640 1 -- 172.21.15.69:0/3348484883 shutdown_connections 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.048+0000 7fed9b200640 1 -- 172.21.15.69:0/3348484883 wait complete. 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.049+0000 7fed9b200640 1 Processor -- start 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.049+0000 7fed9b200640 1 -- start start 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.050+0000 7fed9b200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c004d40 0x7fed9c158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.050+0000 7fed9b200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c09a9d0 0x7fed9c09ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.050+0000 7fed9b200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed9c1e9cd0 0x7fed9c1ea150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.050+0000 7fed9b200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c162fd0 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.050+0000 7fed9b200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fed9c0cd2c0 con 0x7fed7c004d40 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.050+0000 7fed9b200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fed9c0ca920 con 0x7fed9c09a9d0 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed9a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed9c1e9cd0 0x7fed9c1ea150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed9a800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed9c1e9cd0 0x7fed9c1ea150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52452/0 (socket says 172.21.15.69:52452) 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed9a800640 1 -- 172.21.15.69:0/1181831789 learned_addr learned my addr 172.21.15.69:0/1181831789 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed99e00640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c004d40 0x7fed9c158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed93e00640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c09a9d0 0x7fed9c09ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed9a800640 1 -- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c09a9d0 msgr2=0x7fed9c09ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed9a800640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fed9c09a9d0 0x7fed9c09ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed9a800640 1 -- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c004d40 msgr2=0x7fed9c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed9a800640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c004d40 0x7fed9c158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed9a800640 1 -- 172.21.15.69:0/1181831789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9c15a610 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed99e00640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed7c004d40 0x7fed9c158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:41.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.051+0000 7fed9a800640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed9c1e9cd0 0x7fed9c1ea150 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fed9402b470 tx=0x7fed9c162be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.052+0000 7fed92000640 1 -- 172.21.15.69:0/1181831789 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9402bbd0 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.052+0000 7fed92000640 1 -- 172.21.15.69:0/1181831789 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed9402bd70 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.052+0000 7fed92000640 1 -- 172.21.15.69:0/1181831789 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed94034670 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.052+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed9c1eaa50 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.052+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed9c15ab90 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.053+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fed9c15af70 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.054+0000 7fed92000640 1 -- 172.21.15.69:0/1181831789 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fed94034810 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.054+0000 7fed92000640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7806c920 0x7fed7806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.055+0000 7fed92000640 1 -- 172.21.15.69:0/1181831789 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fed940abb30 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.055+0000 7fed92000640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x7fed78072920 0x7fed78074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.055+0000 7fed92000640 1 -- 172.21.15.69:0/1181831789 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed9400bf30 con 0x7fed78072920 2024-04-24T08:21:41.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.055+0000 7fed92000640 1 -- 172.21.15.69:0/1181831789 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fed940b02a0 con 0x7fed9c1e9cd0 2024-04-24T08:21:41.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.055+0000 7fed93e00640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x7fed78072920 0x7fed78074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.055+0000 7fed99e00640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7806c920 0x7fed7806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.056+0000 7fed93e00640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x7fed78072920 0x7fed78074d40 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.056+0000 7fed99e00640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7806c920 0x7fed7806ede0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009a00 tx=0x7fed8c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.056+0000 7fed92000640 1 -- 172.21.15.69:0/1181831789 <== osd.2 v2:172.21.15.69:6808/2630808717 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed9400bf30 con 0x7fed78072920 2024-04-24T08:21:41.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.071+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 --> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fed9c075d50 con 0x7fed78072920 2024-04-24T08:21:41.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.071+0000 7fed92000640 1 -- 172.21.15.69:0/1181831789 <== osd.2 v2:172.21.15.69:6808/2630808717 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fed9c075d50 con 0x7fed78072920 2024-04-24T08:21:41.072 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:41.072 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T08:21:41.072 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:41.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.071+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x7fed78072920 msgr2=0x7fed78074d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.071+0000 7fed9b200640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.69:6808/2630808717,v1:172.21.15.69:6809/2630808717] conn(0x7fed78072920 0x7fed78074d40 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.072+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7806c920 msgr2=0x7fed7806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.072+0000 7fed9b200640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fed7806c920 0x7fed7806ede0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fed8c009a00 tx=0x7fed8c008040 comp rx=0 tx=0).stop 2024-04-24T08:21:41.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.072+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed9c1e9cd0 msgr2=0x7fed9c1ea150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.072+0000 7fed9b200640 1 --2- 172.21.15.69:0/1181831789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed9c1e9cd0 0x7fed9c1ea150 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7fed9402b470 tx=0x7fed9c162be0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.072+0000 7fed9a800640 1 -- 172.21.15.69:0/1181831789 reap_dead start 2024-04-24T08:21:41.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.072+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 shutdown_connections 2024-04-24T08:21:41.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.072+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 >> 172.21.15.69:0/1181831789 conn(0x7fed9c0b1c50 msgr2=0x7fed9c06db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:41.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.072+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 shutdown_connections 2024-04-24T08:21:41.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.072+0000 7fed9b200640 1 -- 172.21.15.69:0/1181831789 wait complete. 2024-04-24T08:21:41.089 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-24T08:21:41.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.288+0000 7f8835800640 1 Processor -- start 2024-04-24T08:21:41.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.288+0000 7f8835800640 1 -- start start 2024-04-24T08:21:41.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.289+0000 7f8835800640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f88301523f0 0x7f883006db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.289+0000 7f8835800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8830072e20 0x7f8830073260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.289+0000 7f8835800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88300737a0 0x7f883015ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.289+0000 7f8835800640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f88300aeac0 con 0x7f8830072e20 2024-04-24T08:21:41.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.289+0000 7f8835800640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f88300aeda0 con 0x7f88301523f0 2024-04-24T08:21:41.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.289+0000 7f8835800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f88300af080 con 0x7f88300737a0 2024-04-24T08:21:41.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.289+0000 7f8827400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8830072e20 0x7f8830073260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.290+0000 7f8827400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f8830072e20 0x7f8830073260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52466/0 (socket says 172.21.15.69:52466) 2024-04-24T08:21:41.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.290+0000 7f8827400640 1 -- 172.21.15.69:0/1192299005 learned_addr learned my addr 172.21.15.69:0/1192299005 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:41.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.290+0000 7f8834e00640 1 --2- 172.21.15.69:0/1192299005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88300737a0 0x7f883015ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.290+0000 7f882fe00640 1 --2- 172.21.15.69:0/1192299005 >> v2:172.21.15.121:3300/0 conn(0x7f88301523f0 0x7f883006db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.290+0000 7f8827400640 1 -- 172.21.15.69:0/1192299005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88300737a0 msgr2=0x7f883015ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:41.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.290+0000 7f8827400640 1 --2- 172.21.15.69:0/1192299005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88300737a0 0x7f883015ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.290+0000 7f8827400640 1 -- 172.21.15.69:0/1192299005 >> v2:172.21.15.121:3300/0 conn(0x7f88301523f0 msgr2=0x7f883006db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:41.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.290+0000 7f8827400640 1 --2- 172.21.15.69:0/1192299005 >> v2:172.21.15.121:3300/0 conn(0x7f88301523f0 0x7f883006db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.290+0000 7f8827400640 1 -- 172.21.15.69:0/1192299005 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88300af360 con 0x7f8830072e20 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.291+0000 7f8827400640 1 --2- 172.21.15.69:0/1192299005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8830072e20 0x7f8830073260 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f883006cb50 tx=0x7f881802c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d588113a5a3a012 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.291+0000 7f882f400640 1 -- 172.21.15.69:0/1192299005 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8818039070 con 0x7f8830072e20 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.291+0000 7f882f400640 1 -- 172.21.15.69:0/1192299005 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8818034640 con 0x7f8830072e20 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.291+0000 7f882f400640 1 -- 172.21.15.69:0/1192299005 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8818034a10 con 0x7f8830072e20 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.292+0000 7f8835800640 1 -- 172.21.15.69:0/1192299005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8830072e20 msgr2=0x7f8830073260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.292+0000 7f8835800640 1 --2- 172.21.15.69:0/1192299005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8830072e20 0x7f8830073260 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f883006cb50 tx=0x7f881802c350 comp rx=0 tx=0).stop 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.292+0000 7f8835800640 1 -- 172.21.15.69:0/1192299005 shutdown_connections 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.292+0000 7f8835800640 1 --2- 172.21.15.69:0/1192299005 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f88300737a0 0x7f883015ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.292+0000 7f8835800640 1 --2- 172.21.15.69:0/1192299005 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8830072e20 0x7f8830073260 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.292+0000 7f8835800640 1 --2- 172.21.15.69:0/1192299005 >> v2:172.21.15.121:3300/0 conn(0x7f88301523f0 0x7f883006db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.292+0000 7f8835800640 1 -- 172.21.15.69:0/1192299005 >> 172.21.15.69:0/1192299005 conn(0x7f88300d0160 msgr2=0x7f88300d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:41.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.292+0000 7f8835800640 1 -- 172.21.15.69:0/1192299005 shutdown_connections 2024-04-24T08:21:41.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.292+0000 7f8835800640 1 -- 172.21.15.69:0/1192299005 wait complete. 2024-04-24T08:21:41.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.294+0000 7f8835800640 1 Processor -- start 2024-04-24T08:21:41.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.294+0000 7f8835800640 1 -- start start 2024-04-24T08:21:41.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8835800640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8830072e20 0x7f88301530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8835800640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88300737a0 0x7f8830155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8835800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88301523f0 0x7f883006e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8835800640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f883015f4b0 con 0x7f88301523f0 2024-04-24T08:21:41.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8835800640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f88301527d0 con 0x7f88300737a0 2024-04-24T08:21:41.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8835800640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f883015f630 con 0x7f8830072e20 2024-04-24T08:21:41.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8834e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88301523f0 0x7f883006e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8834e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88301523f0 0x7f883006e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52480/0 (socket says 172.21.15.69:52480) 2024-04-24T08:21:41.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8834e00640 1 -- 172.21.15.69:0/150833274 learned_addr learned my addr 172.21.15.69:0/150833274 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:41.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8827400640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88300737a0 0x7f8830155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f882fe00640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8830072e20 0x7f88301530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8834e00640 1 -- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8830072e20 msgr2=0x7f88301530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8834e00640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8830072e20 0x7f88301530d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8834e00640 1 -- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88300737a0 msgr2=0x7f8830155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8834e00640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88300737a0 0x7f8830155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.295+0000 7f8834e00640 1 -- 172.21.15.69:0/150833274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8830155c60 con 0x7f88301523f0 2024-04-24T08:21:41.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.296+0000 7f8834e00640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88301523f0 0x7f883006e690 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f8828007e60 tx=0x7f882800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.296+0000 7f882cc00640 1 -- 172.21.15.69:0/150833274 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828014620 con 0x7f88301523f0 2024-04-24T08:21:41.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.296+0000 7f882cc00640 1 -- 172.21.15.69:0/150833274 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8828019070 con 0x7f88301523f0 2024-04-24T08:21:41.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.296+0000 7f882cc00640 1 -- 172.21.15.69:0/150833274 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828014a20 con 0x7f88301523f0 2024-04-24T08:21:41.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.296+0000 7f8835800640 1 -- 172.21.15.69:0/150833274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8830156d00 con 0x7f88301523f0 2024-04-24T08:21:41.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.296+0000 7f8835800640 1 -- 172.21.15.69:0/150833274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88301572c0 con 0x7f88301523f0 2024-04-24T08:21:41.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.297+0000 7f8835800640 1 -- 172.21.15.69:0/150833274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8830156af0 con 0x7f88301523f0 2024-04-24T08:21:41.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.298+0000 7f882cc00640 1 -- 172.21.15.69:0/150833274 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8828014bc0 con 0x7f88301523f0 2024-04-24T08:21:41.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.299+0000 7f882cc00640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f880c06c8f0 0x7f880c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.299+0000 7f882cc00640 1 -- 172.21.15.69:0/150833274 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f882808c240 con 0x7f88301523f0 2024-04-24T08:21:41.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.300+0000 7f882fe00640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f880c06c8f0 0x7f880c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.301+0000 7f882fe00640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f880c06c8f0 0x7f880c06edb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8820006fd0 tx=0x7f8820008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.304+0000 7f882cc00640 1 -- 172.21.15.69:0/150833274 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f882805a820 con 0x7f88301523f0 2024-04-24T08:21:41.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.505+0000 7f8835800640 1 -- 172.21.15.69:0/150833274 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f8830075d50 con 0x7f880c06c8f0 2024-04-24T08:21:41.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.506+0000 7f882cc00640 1 -- 172.21.15.69:0/150833274 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8830075d50 con 0x7f880c06c8f0 2024-04-24T08:21:41.506 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 -- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f880c06c8f0 msgr2=0x7f880c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f880c06c8f0 0x7f880c06edb0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8820006fd0 tx=0x7f8820008040 comp rx=0 tx=0).stop 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 -- 172.21.15.69:0/150833274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88301523f0 msgr2=0x7f883006e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88301523f0 0x7f883006e690 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f8828007e60 tx=0x7f882800e4b0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 -- 172.21.15.69:0/150833274 shutdown_connections 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f880c06c8f0 0x7f880c06edb0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f88301523f0 0x7f883006e690 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88300737a0 0x7f8830155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 --2- 172.21.15.69:0/150833274 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8830072e20 0x7f88301530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 -- 172.21.15.69:0/150833274 >> 172.21.15.69:0/150833274 conn(0x7f88300d0160 msgr2=0x7f88300d0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:41.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.508+0000 7f8835800640 1 -- 172.21.15.69:0/150833274 shutdown_connections 2024-04-24T08:21:41.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.509+0000 7f8835800640 1 -- 172.21.15.69:0/150833274 wait complete. 2024-04-24T08:21:41.528 INFO:tasks.ceph:Scrubbing osd.3 2024-04-24T08:21:41.528 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-24T08:21:41.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.735+0000 7f9983200640 1 Processor -- start 2024-04-24T08:21:41.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.735+0000 7f9983200640 1 -- start start 2024-04-24T08:21:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.736+0000 7f9983200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f99840c30f0 0x7f99840af460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.736+0000 7f9983200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f998406f150 0x7f99840c34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.736+0000 7f9983200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9984073de0 con 0x7f9984166e60 2024-04-24T08:21:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.736+0000 7f9983200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f99840af9a0 con 0x7f99840c30f0 2024-04-24T08:21:41.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.736+0000 7f9983200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f99841666f0 con 0x7f998406f150 2024-04-24T08:21:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.737+0000 7f9979400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f998406f150 0x7f99840c34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.737+0000 7f9979400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f998406f150 0x7f99840c34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.69:48634/0 (socket says 172.21.15.69:48634) 2024-04-24T08:21:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.737+0000 7f9979400640 1 -- 172.21.15.69:0/3752895447 learned_addr learned my addr 172.21.15.69:0/3752895447 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.737+0000 7f9982800640 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f9984166e60 0x7f9984071b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.69:38300/0 (socket says 172.21.15.69:38300) 2024-04-24T08:21:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.737+0000 7f9981e00640 1 --2- 172.21.15.69:0/3752895447 >> v2:172.21.15.121:3300/0 conn(0x7f99840c30f0 0x7f99840af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.738+0000 7f9979400640 1 -- 172.21.15.69:0/3752895447 >> v2:172.21.15.121:3300/0 conn(0x7f99840c30f0 msgr2=0x7f99840af460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:41.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.738+0000 7f9979400640 1 --2- 172.21.15.69:0/3752895447 >> v2:172.21.15.121:3300/0 conn(0x7f99840c30f0 0x7f99840af460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.738+0000 7f9979400640 1 -- 172.21.15.69:0/3752895447 >> v1:172.21.15.121:6789/0 conn(0x7f9984166e60 legacy=0x7f9984071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.738+0000 7f9979400640 1 -- 172.21.15.69:0/3752895447 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99840c3a10 con 0x7f998406f150 2024-04-24T08:21:41.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.739+0000 7f9979400640 1 --2- 172.21.15.69:0/3752895447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f998406f150 0x7f99840c34d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f996c009870 tx=0x7f996c02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b42aab1369e33422 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.740+0000 7f9981400640 1 -- 172.21.15.69:0/3752895447 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c037070 con 0x7f998406f150 2024-04-24T08:21:41.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.740+0000 7f9981400640 1 -- 172.21.15.69:0/3752895447 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f996c032440 con 0x7f998406f150 2024-04-24T08:21:41.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.740+0000 7f9981400640 1 -- 172.21.15.69:0/3752895447 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f996c032810 con 0x7f998406f150 2024-04-24T08:21:41.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.740+0000 7f9983200640 1 -- 172.21.15.69:0/3752895447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f998406f150 msgr2=0x7f99840c34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.740+0000 7f9983200640 1 --2- 172.21.15.69:0/3752895447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f998406f150 0x7f99840c34d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f996c009870 tx=0x7f996c02a250 comp rx=0 tx=0).stop 2024-04-24T08:21:41.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.740+0000 7f9983200640 1 -- 172.21.15.69:0/3752895447 shutdown_connections 2024-04-24T08:21:41.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.740+0000 7f9983200640 1 --2- 172.21.15.69:0/3752895447 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f998406f150 0x7f99840c34d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.740+0000 7f9983200640 1 --2- 172.21.15.69:0/3752895447 >> v2:172.21.15.121:3300/0 conn(0x7f99840c30f0 0x7f99840af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.740+0000 7f9983200640 1 -- 172.21.15.69:0/3752895447 >> 172.21.15.69:0/3752895447 conn(0x7f99840b1c50 msgr2=0x7f99840b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:41.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.741+0000 7f9983200640 1 -- 172.21.15.69:0/3752895447 shutdown_connections 2024-04-24T08:21:41.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.741+0000 7f9983200640 1 -- 172.21.15.69:0/3752895447 wait complete. 2024-04-24T08:21:41.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.742+0000 7f9983200640 1 Processor -- start 2024-04-24T08:21:41.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.742+0000 7f9983200640 1 -- start start 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.743+0000 7f9983200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f998406f150 0x7f99841f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.743+0000 7f9983200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99840c30f0 0x7f99840b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.743+0000 7f9983200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9984166e60 0x7f998409d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.743+0000 7f9983200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f998406fc80 con 0x7f9984166e60 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.743+0000 7f9983200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f998406f9a0 con 0x7f998406f150 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.743+0000 7f9983200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f9984166940 con 0x7f99840c30f0 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9982800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9984166e60 0x7f998409d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9982800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9984166e60 0x7f998409d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52486/0 (socket says 172.21.15.69:52486) 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9982800640 1 -- 172.21.15.69:0/2946963803 learned_addr learned my addr 172.21.15.69:0/2946963803 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9981e00640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f998406f150 0x7f99841f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9979400640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99840c30f0 0x7f99840b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9982800640 1 -- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99840c30f0 msgr2=0x7f99840b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9982800640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f99840c30f0 0x7f99840b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9982800640 1 -- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f998406f150 msgr2=0x7f99841f9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9982800640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f998406f150 0x7f99841f9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9982800640 1 -- 172.21.15.69:0/2946963803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99840b3500 con 0x7f9984166e60 2024-04-24T08:21:41.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.744+0000 7f9981e00640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f998406f150 0x7f99841f9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:41.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.745+0000 7f9982800640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9984166e60 0x7f998409d270 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f997c00c730 tx=0x7f997c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.745+0000 7f997aa00640 1 -- 172.21.15.69:0/2946963803 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997c017070 con 0x7f9984166e60 2024-04-24T08:21:41.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.745+0000 7f997aa00640 1 -- 172.21.15.69:0/2946963803 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f997c00ce40 con 0x7f9984166e60 2024-04-24T08:21:41.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.745+0000 7f997aa00640 1 -- 172.21.15.69:0/2946963803 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f997c00a840 con 0x7f9984166e60 2024-04-24T08:21:41.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.745+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99840b37e0 con 0x7f9984166e60 2024-04-24T08:21:41.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.745+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99840b3da0 con 0x7f9984166e60 2024-04-24T08:21:41.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.746+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f99840b3fe0 con 0x7f9984166e60 2024-04-24T08:21:41.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.747+0000 7f997aa00640 1 -- 172.21.15.69:0/2946963803 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f997c01a070 con 0x7f9984166e60 2024-04-24T08:21:41.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.748+0000 7f997aa00640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f996006c8f0 0x7f996006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.748+0000 7f997aa00640 1 -- 172.21.15.69:0/2946963803 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f997c08b9e0 con 0x7f9984166e60 2024-04-24T08:21:41.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.749+0000 7f997aa00640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x7f99600728f0 0x7f9960074d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.749+0000 7f997aa00640 1 -- 172.21.15.69:0/2946963803 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f99600753c0 con 0x7f99600728f0 2024-04-24T08:21:41.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.749+0000 7f997aa00640 1 -- 172.21.15.69:0/2946963803 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f997c08bd90 con 0x7f9984166e60 2024-04-24T08:21:41.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.749+0000 7f9979400640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x7f99600728f0 0x7f9960074d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.749+0000 7f9981e00640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f996006c8f0 0x7f996006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.750+0000 7f9979400640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x7f99600728f0 0x7f9960074d10 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.750+0000 7f9981e00640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f996006c8f0 0x7f996006edb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f998409d200 tx=0x7f9974008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.750+0000 7f997aa00640 1 -- 172.21.15.69:0/2946963803 <== osd.3 v2:172.21.15.69:6811/1875112303 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f99600753c0 con 0x7f99600728f0 2024-04-24T08:21:41.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.765+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 --> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f998409ba80 con 0x7f99600728f0 2024-04-24T08:21:41.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.765+0000 7f997aa00640 1 -- 172.21.15.69:0/2946963803 <== osd.3 v2:172.21.15.69:6811/1875112303 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f998409ba80 con 0x7f99600728f0 2024-04-24T08:21:41.766 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:41.766 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T08:21:41.766 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:41.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x7f99600728f0 msgr2=0x7f9960074d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9983200640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.69:6811/1875112303,v1:172.21.15.69:6813/1875112303] conn(0x7f99600728f0 0x7f9960074d10 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f996006c8f0 msgr2=0x7f996006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9983200640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f996006c8f0 0x7f996006edb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f998409d200 tx=0x7f9974008040 comp rx=0 tx=0).stop 2024-04-24T08:21:41.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9984166e60 msgr2=0x7f998409d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9983200640 1 --2- 172.21.15.69:0/2946963803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9984166e60 0x7f998409d270 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f997c00c730 tx=0x7f997c00cbf0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9982800640 1 -- 172.21.15.69:0/2946963803 reap_dead start 2024-04-24T08:21:41.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 shutdown_connections 2024-04-24T08:21:41.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 >> 172.21.15.69:0/2946963803 conn(0x7f99840b1c50 msgr2=0x7f99840adce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:41.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 shutdown_connections 2024-04-24T08:21:41.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.766+0000 7f9983200640 1 -- 172.21.15.69:0/2946963803 wait complete. 2024-04-24T08:21:41.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-24T08:21:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.973+0000 7fa3a7200640 1 Processor -- start 2024-04-24T08:21:41.974 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.974+0000 7fa3a7200640 1 -- start start 2024-04-24T08:21:41.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.975+0000 7fa3a7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a815ab10 0x7fa3a8144090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.975+0000 7fa3a7200640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fa3a8071be0 0x7fa3a806e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.975+0000 7fa3a7200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3a806e960 0x7fa3a815e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.975+0000 7fa3a7200640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a806edd0 con 0x7fa3a806e960 2024-04-24T08:21:41.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.975+0000 7fa3a7200640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fa3a80b3190 con 0x7fa3a8071be0 2024-04-24T08:21:41.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.975+0000 7fa3a7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a80ad590 con 0x7fa3a815ab10 2024-04-24T08:21:41.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.976+0000 7fa3a6800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3a806e960 0x7fa3a815e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.976+0000 7fa3a6800640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fa3a806e960 0x7fa3a815e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52488/0 (socket says 172.21.15.69:52488) 2024-04-24T08:21:41.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.976+0000 7fa3a6800640 1 -- 172.21.15.69:0/1977067117 learned_addr learned my addr 172.21.15.69:0/1977067117 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:41.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.976+0000 7fa3a5e00640 1 --2- 172.21.15.69:0/1977067117 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a815ab10 0x7fa3a8144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.976+0000 7fa39d400640 1 --2- 172.21.15.69:0/1977067117 >> v2:172.21.15.121:3300/0 conn(0x7fa3a8071be0 0x7fa3a806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.976+0000 7fa3a6800640 1 -- 172.21.15.69:0/1977067117 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a815ab10 msgr2=0x7fa3a8144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:41.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.976+0000 7fa3a6800640 1 --2- 172.21.15.69:0/1977067117 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a815ab10 0x7fa3a8144090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.976+0000 7fa3a6800640 1 -- 172.21.15.69:0/1977067117 >> v2:172.21.15.121:3300/0 conn(0x7fa3a8071be0 msgr2=0x7fa3a806e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:41.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.976+0000 7fa3a6800640 1 --2- 172.21.15.69:0/1977067117 >> v2:172.21.15.121:3300/0 conn(0x7fa3a8071be0 0x7fa3a806e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.976+0000 7fa3a6800640 1 -- 172.21.15.69:0/1977067117 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a80b3400 con 0x7fa3a806e960 2024-04-24T08:21:41.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.977+0000 7fa3a6800640 1 --2- 172.21.15.69:0/1977067117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a806e960 0x7fa3a815e6b0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80ad740 tx=0x7fa3a002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1ab65195060fce server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.978+0000 7fa3a5400640 1 -- 172.21.15.69:0/1977067117 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0037070 con 0x7fa3a806e960 2024-04-24T08:21:41.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.978+0000 7fa3a5400640 1 -- 172.21.15.69:0/1977067117 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3a0032440 con 0x7fa3a806e960 2024-04-24T08:21:41.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.978+0000 7fa3a5400640 1 -- 172.21.15.69:0/1977067117 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0032810 con 0x7fa3a806e960 2024-04-24T08:21:41.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.978+0000 7fa3a7200640 1 -- 172.21.15.69:0/1977067117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a806e960 msgr2=0x7fa3a815e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.978+0000 7fa3a7200640 1 --2- 172.21.15.69:0/1977067117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a806e960 0x7fa3a815e6b0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80ad740 tx=0x7fa3a002a1c0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.978+0000 7fa3a7200640 1 -- 172.21.15.69:0/1977067117 shutdown_connections 2024-04-24T08:21:41.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.978+0000 7fa3a7200640 1 --2- 172.21.15.69:0/1977067117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a806e960 0x7fa3a815e6b0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.978+0000 7fa3a7200640 1 --2- 172.21.15.69:0/1977067117 >> v2:172.21.15.121:3300/0 conn(0x7fa3a8071be0 0x7fa3a806e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.978+0000 7fa3a7200640 1 --2- 172.21.15.69:0/1977067117 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a815ab10 0x7fa3a8144090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.978+0000 7fa3a7200640 1 -- 172.21.15.69:0/1977067117 >> 172.21.15.69:0/1977067117 conn(0x7fa3a80d0160 msgr2=0x7fa3a80d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:41.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.979+0000 7fa3a7200640 1 -- 172.21.15.69:0/1977067117 shutdown_connections 2024-04-24T08:21:41.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.979+0000 7fa3a7200640 1 -- 172.21.15.69:0/1977067117 wait complete. 2024-04-24T08:21:41.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.980+0000 7fa3a7200640 1 Processor -- start 2024-04-24T08:21:41.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.980+0000 7fa3a7200640 1 -- start start 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.981+0000 7fa3a7200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a806e960 0x7fa3a81f7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.981+0000 7fa3a7200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a8071be0 0x7fa3a806db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.981+0000 7fa3a7200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a815ab10 0x7fa3a806e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.981+0000 7fa3a7200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a8072150 con 0x7fa3a8071be0 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.981+0000 7fa3a7200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a80b39c0 con 0x7fa3a815ab10 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.981+0000 7fa3a7200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fa3a815aef0 con 0x7fa3a806e960 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa39d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a8071be0 0x7fa3a806db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa39d400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a8071be0 0x7fa3a806db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52504/0 (socket says 172.21.15.69:52504) 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa39d400640 1 -- 172.21.15.69:0/3024426895 learned_addr learned my addr 172.21.15.69:0/3024426895 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa3a5e00640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a806e960 0x7fa3a81f7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa3a6800640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a815ab10 0x7fa3a806e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa39d400640 1 -- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a806e960 msgr2=0x7fa3a81f7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa39d400640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a806e960 0x7fa3a81f7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa39d400640 1 -- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a815ab10 msgr2=0x7fa3a806e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa39d400640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a815ab10 0x7fa3a806e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa39d400640 1 -- 172.21.15.69:0/3024426895 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a81f8110 con 0x7fa3a8071be0 2024-04-24T08:21:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa3a6800640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a815ab10 0x7fa3a806e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:41.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.982+0000 7fa39d400640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a8071be0 0x7fa3a806db90 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fa39000d800 tx=0x7fa39000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.983+0000 7fa39ea00640 1 -- 172.21.15.69:0/3024426895 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39000b6c0 con 0x7fa3a8071be0 2024-04-24T08:21:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.983+0000 7fa39ea00640 1 -- 172.21.15.69:0/3024426895 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa390019070 con 0x7fa3a8071be0 2024-04-24T08:21:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.983+0000 7fa39ea00640 1 -- 172.21.15.69:0/3024426895 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa39000ba80 con 0x7fa3a8071be0 2024-04-24T08:21:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.983+0000 7fa3a7200640 1 -- 172.21.15.69:0/3024426895 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a8151ff0 con 0x7fa3a8071be0 2024-04-24T08:21:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.983+0000 7fa3a7200640 1 -- 172.21.15.69:0/3024426895 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a81525b0 con 0x7fa3a8071be0 2024-04-24T08:21:41.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.984+0000 7fa3a7200640 1 -- 172.21.15.69:0/3024426895 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a80b1c10 con 0x7fa3a8071be0 2024-04-24T08:21:41.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.985+0000 7fa39ea00640 1 -- 172.21.15.69:0/3024426895 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa39000bc20 con 0x7fa3a8071be0 2024-04-24T08:21:41.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.986+0000 7fa39ea00640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa38406c8f0 0x7fa38406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:41.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.986+0000 7fa39ea00640 1 -- 172.21.15.69:0/3024426895 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fa39008be90 con 0x7fa3a8071be0 2024-04-24T08:21:41.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.987+0000 7fa3a5e00640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa38406c8f0 0x7fa38406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:41.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.988+0000 7fa3a5e00640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa38406c8f0 0x7fa38406edb0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa398009940 tx=0x7fa398003cd0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:41.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:41.990+0000 7fa39ea00640 1 -- 172.21.15.69:0/3024426895 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3a80b1c10 con 0x7fa3a8071be0 2024-04-24T08:21:42.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.190+0000 7fa3a7200640 1 -- 172.21.15.69:0/3024426895 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fa3a8075d50 con 0x7fa38406c8f0 2024-04-24T08:21:42.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.192+0000 7fa39ea00640 1 -- 172.21.15.69:0/3024426895 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa3a8075d50 con 0x7fa38406c8f0 2024-04-24T08:21:42.192 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 3 to deep-scrub 2024-04-24T08:21:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 -- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa38406c8f0 msgr2=0x7fa38406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa38406c8f0 0x7fa38406edb0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa398009940 tx=0x7fa398003cd0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 -- 172.21.15.69:0/3024426895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a8071be0 msgr2=0x7fa3a806db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a8071be0 0x7fa3a806db90 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fa39000d800 tx=0x7fa39000dcc0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 -- 172.21.15.69:0/3024426895 shutdown_connections 2024-04-24T08:21:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fa38406c8f0 0x7fa38406edb0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa3a8071be0 0x7fa3a806db90 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3a815ab10 0x7fa3a806e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 --2- 172.21.15.69:0/3024426895 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa3a806e960 0x7fa3a81f7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 -- 172.21.15.69:0/3024426895 >> 172.21.15.69:0/3024426895 conn(0x7fa3a80d0160 msgr2=0x7fa3a80cf990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 -- 172.21.15.69:0/3024426895 shutdown_connections 2024-04-24T08:21:42.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.194+0000 7fa3a7200640 1 -- 172.21.15.69:0/3024426895 wait complete. 2024-04-24T08:21:42.214 INFO:tasks.ceph:Scrubbing osd.4 2024-04-24T08:21:42.214 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-24T08:21:42.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.420+0000 7fdd05a00640 1 Processor -- start 2024-04-24T08:21:42.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.420+0000 7fdd05a00640 1 -- start start 2024-04-24T08:21:42.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.422+0000 7fdd05a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd0015e930 0x7fdd001623a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.422+0000 7fdd05a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fdd000af5b0 0x7fdd000cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.422+0000 7fdd05a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd000cd280 con 0x7fdd000cce30 2024-04-24T08:21:42.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.422+0000 7fdd05a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fdd0015ed10 con 0x7fdd0015e930 2024-04-24T08:21:42.423 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.422+0000 7fdd05a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fdd000cacf0 con 0x7fdd000af5b0 2024-04-24T08:21:42.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.422+0000 7fdcffe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd0015e930 0x7fdd001623a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.422+0000 7fdcffe00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fdd0015e930 0x7fdd001623a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52506/0 (socket says 172.21.15.69:52506) 2024-04-24T08:21:42.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.422+0000 7fdd05000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdd000cce30 0x7fdd000ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42728/0 (socket says 172.21.15.69:42728) 2024-04-24T08:21:42.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.422+0000 7fdcffe00640 1 -- 172.21.15.69:0/1253665386 learned_addr learned my addr 172.21.15.69:0/1253665386 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:42.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.423+0000 7fdcf7400640 1 --2- 172.21.15.69:0/1253665386 >> v2:172.21.15.121:3300/0 conn(0x7fdd000af5b0 0x7fdd000cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.423+0000 7fdcffe00640 1 -- 172.21.15.69:0/1253665386 >> v2:172.21.15.121:3300/0 conn(0x7fdd000af5b0 msgr2=0x7fdd000cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:42.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.423+0000 7fdcffe00640 1 --2- 172.21.15.69:0/1253665386 >> v2:172.21.15.121:3300/0 conn(0x7fdd000af5b0 0x7fdd000cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.423+0000 7fdcffe00640 1 -- 172.21.15.69:0/1253665386 >> v1:172.21.15.69:6789/0 conn(0x7fdd000cce30 legacy=0x7fdd000ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.423+0000 7fdcffe00640 1 -- 172.21.15.69:0/1253665386 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd001632b0 con 0x7fdd0015e930 2024-04-24T08:21:42.424 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.423+0000 7fdcff400640 1 -- 172.21.15.69:0/1253665386 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4167728179 0 0) 0x7fdd000cd280 con 0x7fdd000cce30 2024-04-24T08:21:42.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.424+0000 7fdcffe00640 1 --2- 172.21.15.69:0/1253665386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd0015e930 0x7fdd001623a0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fdd00162300 tx=0x7fdcf002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7418712910fa702e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:42.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.424+0000 7fdcff400640 1 -- 172.21.15.69:0/1253665386 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0037070 con 0x7fdd0015e930 2024-04-24T08:21:42.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.424+0000 7fdcff400640 1 -- 172.21.15.69:0/1253665386 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf0032440 con 0x7fdd0015e930 2024-04-24T08:21:42.425 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.424+0000 7fdcff400640 1 -- 172.21.15.69:0/1253665386 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0032810 con 0x7fdd0015e930 2024-04-24T08:21:42.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.425+0000 7fdd05a00640 1 -- 172.21.15.69:0/1253665386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd0015e930 msgr2=0x7fdd001623a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.425+0000 7fdd05a00640 1 --2- 172.21.15.69:0/1253665386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd0015e930 0x7fdd001623a0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fdd00162300 tx=0x7fdcf002a1e0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.425+0000 7fdd05a00640 1 -- 172.21.15.69:0/1253665386 shutdown_connections 2024-04-24T08:21:42.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.425+0000 7fdd05a00640 1 --2- 172.21.15.69:0/1253665386 >> v2:172.21.15.121:3300/0 conn(0x7fdd000af5b0 0x7fdd000cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.425+0000 7fdd05a00640 1 --2- 172.21.15.69:0/1253665386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd0015e930 0x7fdd001623a0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.425+0000 7fdd05a00640 1 -- 172.21.15.69:0/1253665386 >> 172.21.15.69:0/1253665386 conn(0x7fdd000b1c50 msgr2=0x7fdd000b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:42.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.425+0000 7fdd05a00640 1 -- 172.21.15.69:0/1253665386 shutdown_connections 2024-04-24T08:21:42.426 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.425+0000 7fdd05a00640 1 -- 172.21.15.69:0/1253665386 wait complete. 2024-04-24T08:21:42.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.427+0000 7fdd05a00640 1 Processor -- start 2024-04-24T08:21:42.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.427+0000 7fdd05a00640 1 -- start start 2024-04-24T08:21:42.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.428+0000 7fdd05a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd000af5b0 0x7fdd00159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.428+0000 7fdd05a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd000cce30 0x7fdd000b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.428+0000 7fdd05a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd0015e930 0x7fdd000b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.428+0000 7fdd05a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdd00162fd0 con 0x7fdd0015e930 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.428+0000 7fdd05a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdd00162cf0 con 0x7fdd000af5b0 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.428+0000 7fdd05a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdd000ca920 con 0x7fdd000cce30 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdd05000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd0015e930 0x7fdd000b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdd05000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd0015e930 0x7fdd000b1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52508/0 (socket says 172.21.15.69:52508) 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdd05000640 1 -- 172.21.15.69:0/1291995222 learned_addr learned my addr 172.21.15.69:0/1291995222 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdcffe00640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd000af5b0 0x7fdd00159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdcf7400640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd000cce30 0x7fdd000b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdd05000640 1 -- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd000cce30 msgr2=0x7fdd000b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdd05000640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdd000cce30 0x7fdd000b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdd05000640 1 -- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd000af5b0 msgr2=0x7fdd00159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdd05000640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd000af5b0 0x7fdd00159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdd05000640 1 -- 172.21.15.69:0/1291995222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd0015ef40 con 0x7fdd0015e930 2024-04-24T08:21:42.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.429+0000 7fdd05000640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd0015e930 0x7fdd000b1000 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800c7e0 tx=0x7fdcf800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:42.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.430+0000 7fdcfcc00640 1 -- 172.21.15.69:0/1291995222 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8018070 con 0x7fdd0015e930 2024-04-24T08:21:42.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.430+0000 7fdcfcc00640 1 -- 172.21.15.69:0/1291995222 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf800a640 con 0x7fdd0015e930 2024-04-24T08:21:42.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.430+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd0015f1e0 con 0x7fdd0015e930 2024-04-24T08:21:42.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.430+0000 7fdcfcc00640 1 -- 172.21.15.69:0/1291995222 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf800aa30 con 0x7fdd0015e930 2024-04-24T08:21:42.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.430+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd0015f7a0 con 0x7fdd0015e930 2024-04-24T08:21:42.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.431+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdd00075d50 con 0x7fdd0015e930 2024-04-24T08:21:42.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.432+0000 7fdcfcc00640 1 -- 172.21.15.69:0/1291995222 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdcf801b070 con 0x7fdd0015e930 2024-04-24T08:21:42.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.433+0000 7fdcfcc00640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcdc06c8f0 0x7fdcdc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.433+0000 7fdcfcc00640 1 -- 172.21.15.69:0/1291995222 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fdcf808afb0 con 0x7fdd0015e930 2024-04-24T08:21:42.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.433+0000 7fdcfcc00640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x7fdcdc0728f0 0x7fdcdc074d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.433+0000 7fdcfcc00640 1 -- 172.21.15.69:0/1291995222 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdcdc0753c0 con 0x7fdcdc0728f0 2024-04-24T08:21:42.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.433+0000 7fdcfcc00640 1 -- 172.21.15.69:0/1291995222 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fdcf808d570 con 0x7fdd0015e930 2024-04-24T08:21:42.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.434+0000 7fdcffe00640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcdc06c8f0 0x7fdcdc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.434+0000 7fdcf7400640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x7fdcdc0728f0 0x7fdcdc074d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.435+0000 7fdcffe00640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcdc06c8f0 0x7fdcdc06edb0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0034000 tx=0x7fdcf002e000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:42.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.435+0000 7fdcf7400640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x7fdcdc0728f0 0x7fdcdc074d10 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:42.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.437+0000 7fdcfcc00640 1 -- 172.21.15.69:0/1291995222 <== osd.4 v2:172.21.15.121:6800/2274643300 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdcdc0753c0 con 0x7fdcdc0728f0 2024-04-24T08:21:42.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.452+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 --> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdd000cfcf0 con 0x7fdcdc0728f0 2024-04-24T08:21:42.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.453+0000 7fdcfcc00640 1 -- 172.21.15.69:0/1291995222 <== osd.4 v2:172.21.15.121:6800/2274643300 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdd000cfcf0 con 0x7fdcdc0728f0 2024-04-24T08:21:42.454 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:42.454 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T08:21:42.454 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:42.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x7fdcdc0728f0 msgr2=0x7fdcdc074d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05a00640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:6800/2274643300,v1:172.21.15.121:6801/2274643300] conn(0x7fdcdc0728f0 0x7fdcdc074d10 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcdc06c8f0 msgr2=0x7fdcdc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05a00640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdcdc06c8f0 0x7fdcdc06edb0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fdcf0034000 tx=0x7fdcf002e000 comp rx=0 tx=0).stop 2024-04-24T08:21:42.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd0015e930 msgr2=0x7fdd000b1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05a00640 1 --2- 172.21.15.69:0/1291995222 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd0015e930 0x7fdd000b1000 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800c7e0 tx=0x7fdcf800cca0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05000640 1 -- 172.21.15.69:0/1291995222 reap_dead start 2024-04-24T08:21:42.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 shutdown_connections 2024-04-24T08:21:42.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 >> 172.21.15.69:0/1291995222 conn(0x7fdd000b1c50 msgr2=0x7fdd00071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:42.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 shutdown_connections 2024-04-24T08:21:42.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.454+0000 7fdd05a00640 1 -- 172.21.15.69:0/1291995222 wait complete. 2024-04-24T08:21:42.474 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-24T08:21:42.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.676+0000 7f1d5dc00640 1 Processor -- start 2024-04-24T08:21:42.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.676+0000 7f1d5dc00640 1 -- start start 2024-04-24T08:21:42.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d5dc00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d5806f2d0 0x7f1d5806f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d5dc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d580cfbe0 0x7f1d580d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d5dc00640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d580b46e0 con 0x7f1d5815b490 2024-04-24T08:21:42.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d5dc00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f1d5806fc70 con 0x7f1d5806f2d0 2024-04-24T08:21:42.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d5dc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1d580b5890 con 0x7f1d580cfbe0 2024-04-24T08:21:42.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d4e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d5806f2d0 0x7f1d5806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d4e000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f1d5806f2d0 0x7f1d5806f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52514/0 (socket says 172.21.15.69:52514) 2024-04-24T08:21:42.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d4e000640 1 -- 172.21.15.69:0/3174335603 learned_addr learned my addr 172.21.15.69:0/3174335603 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:42.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d57400640 1 --2- 172.21.15.69:0/3174335603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d580cfbe0 0x7f1d580d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d4e000640 1 -- 172.21.15.69:0/3174335603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d580cfbe0 msgr2=0x7f1d580d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:42.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d4e000640 1 --2- 172.21.15.69:0/3174335603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d580cfbe0 0x7f1d580d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.678+0000 7f1d4e000640 1 -- 172.21.15.69:0/3174335603 >> v1:172.21.15.121:6789/0 conn(0x7f1d5815b490 legacy=0x7f1d580b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:42.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.679+0000 7f1d4e000640 1 -- 172.21.15.69:0/3174335603 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d580d0960 con 0x7f1d5806f2d0 2024-04-24T08:21:42.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.679+0000 7f1d4e000640 1 --2- 172.21.15.69:0/3174335603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d5806f2d0 0x7f1d5806f730 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009870 tx=0x7f1d4002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1be3766f7625bc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:42.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.680+0000 7f1d56000640 1 -- 172.21.15.69:0/3174335603 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d40037070 con 0x7f1d5806f2d0 2024-04-24T08:21:42.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.680+0000 7f1d56000640 1 -- 172.21.15.69:0/3174335603 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d40032440 con 0x7f1d5806f2d0 2024-04-24T08:21:42.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.680+0000 7f1d56000640 1 -- 172.21.15.69:0/3174335603 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d40032810 con 0x7f1d5806f2d0 2024-04-24T08:21:42.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.680+0000 7f1d5dc00640 1 -- 172.21.15.69:0/3174335603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d5806f2d0 msgr2=0x7f1d5806f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.680+0000 7f1d5dc00640 1 --2- 172.21.15.69:0/3174335603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d5806f2d0 0x7f1d5806f730 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009870 tx=0x7f1d4002a200 comp rx=0 tx=0).stop 2024-04-24T08:21:42.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.681+0000 7f1d5dc00640 1 -- 172.21.15.69:0/3174335603 shutdown_connections 2024-04-24T08:21:42.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.681+0000 7f1d5dc00640 1 --2- 172.21.15.69:0/3174335603 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d580cfbe0 0x7f1d580d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.681+0000 7f1d5dc00640 1 --2- 172.21.15.69:0/3174335603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d5806f2d0 0x7f1d5806f730 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.681+0000 7f1d5dc00640 1 -- 172.21.15.69:0/3174335603 >> 172.21.15.69:0/3174335603 conn(0x7f1d58067d90 msgr2=0x7f1d5806ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:42.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.681+0000 7f1d5dc00640 1 -- 172.21.15.69:0/3174335603 shutdown_connections 2024-04-24T08:21:42.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.681+0000 7f1d5dc00640 1 -- 172.21.15.69:0/3174335603 wait complete. 2024-04-24T08:21:42.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.682+0000 7f1d5dc00640 1 Processor -- start 2024-04-24T08:21:42.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.683+0000 7f1d5dc00640 1 -- start start 2024-04-24T08:21:42.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.683+0000 7f1d5dc00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d5806f2d0 0x7f1d581f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.683+0000 7f1d5dc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d580cfbe0 0x7f1d5815a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.683+0000 7f1d5dc00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d5815b490 0x7f1d5815a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.683+0000 7f1d5dc00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d580d0c40 con 0x7f1d580cfbe0 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.683+0000 7f1d5dc00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f1d580b4c60 con 0x7f1d5806f2d0 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.683+0000 7f1d5dc00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f1d5815b870 con 0x7f1d5815b490 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d4e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d580cfbe0 0x7f1d5815a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d4e000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d580cfbe0 0x7f1d5815a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52516/0 (socket says 172.21.15.69:52516) 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d4e000640 1 -- 172.21.15.69:0/2179991198 learned_addr learned my addr 172.21.15.69:0/2179991198 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d57400640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d5815b490 0x7f1d5815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d56a00640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d5806f2d0 0x7f1d581f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d4e000640 1 -- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d5815b490 msgr2=0x7f1d5815a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d4e000640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d5815b490 0x7f1d5815a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d4e000640 1 -- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d5806f2d0 msgr2=0x7f1d581f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d4e000640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d5806f2d0 0x7f1d581f7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d4e000640 1 -- 172.21.15.69:0/2179991198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d581f5340 con 0x7f1d580cfbe0 2024-04-24T08:21:42.686 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.684+0000 7f1d56a00640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d5806f2d0 0x7f1d581f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:42.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.685+0000 7f1d4e000640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d580cfbe0 0x7f1d5815a150 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009870 tx=0x7f1d400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:42.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.685+0000 7f1d4f400640 1 -- 172.21.15.69:0/2179991198 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d40041020 con 0x7f1d580cfbe0 2024-04-24T08:21:42.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.685+0000 7f1d4f400640 1 -- 172.21.15.69:0/2179991198 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d400306c0 con 0x7f1d580cfbe0 2024-04-24T08:21:42.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.685+0000 7f1d4f400640 1 -- 172.21.15.69:0/2179991198 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d40037040 con 0x7f1d580cfbe0 2024-04-24T08:21:42.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.685+0000 7f1d5dc00640 1 -- 172.21.15.69:0/2179991198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d5815be90 con 0x7f1d580cfbe0 2024-04-24T08:21:42.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.685+0000 7f1d5dc00640 1 -- 172.21.15.69:0/2179991198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d5815c450 con 0x7f1d580cfbe0 2024-04-24T08:21:42.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.686+0000 7f1d5dc00640 1 -- 172.21.15.69:0/2179991198 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d58068a80 con 0x7f1d580cfbe0 2024-04-24T08:21:42.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.687+0000 7f1d4f400640 1 -- 172.21.15.69:0/2179991198 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d40044020 con 0x7f1d580cfbe0 2024-04-24T08:21:42.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.688+0000 7f1d4f400640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d3406c8f0 0x7f1d3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:42.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.688+0000 7f1d4f400640 1 -- 172.21.15.69:0/2179991198 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f1d400ac0a0 con 0x7f1d580cfbe0 2024-04-24T08:21:42.689 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.689+0000 7f1d56a00640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d3406c8f0 0x7f1d3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:42.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.690+0000 7f1d56a00640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d3406c8f0 0x7f1d3406edb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f1d48003eb0 tx=0x7f1d48009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:42.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.692+0000 7f1d4f400640 1 -- 172.21.15.69:0/2179991198 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d4004b030 con 0x7f1d580cfbe0 2024-04-24T08:21:42.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.893+0000 7f1d5dc00640 1 -- 172.21.15.69:0/2179991198 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f1d581f7750 con 0x7f1d3406c8f0 2024-04-24T08:21:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.895+0000 7f1d4f400640 1 -- 172.21.15.69:0/2179991198 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1d581f7750 con 0x7f1d3406c8f0 2024-04-24T08:21:42.895 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 4 to deep-scrub 2024-04-24T08:21:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.897+0000 7f1d5dc00640 1 -- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d3406c8f0 msgr2=0x7f1d3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.897+0000 7f1d5dc00640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d3406c8f0 0x7f1d3406edb0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f1d48003eb0 tx=0x7f1d48009040 comp rx=0 tx=0).stop 2024-04-24T08:21:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.897+0000 7f1d5dc00640 1 -- 172.21.15.69:0/2179991198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d580cfbe0 msgr2=0x7f1d5815a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:42.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.897+0000 7f1d5dc00640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d580cfbe0 0x7f1d5815a150 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7f1d40009870 tx=0x7f1d400046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.898+0000 7f1d5dc00640 1 -- 172.21.15.69:0/2179991198 shutdown_connections 2024-04-24T08:21:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.898+0000 7f1d5dc00640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f1d3406c8f0 0x7f1d3406edb0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.898+0000 7f1d5dc00640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d580cfbe0 0x7f1d5815a150 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.898+0000 7f1d5dc00640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1d5815b490 0x7f1d5815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.898+0000 7f1d5dc00640 1 --2- 172.21.15.69:0/2179991198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d5806f2d0 0x7f1d581f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.898+0000 7f1d5dc00640 1 -- 172.21.15.69:0/2179991198 >> 172.21.15.69:0/2179991198 conn(0x7f1d58067d90 msgr2=0x7f1d580b3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.898+0000 7f1d5dc00640 1 -- 172.21.15.69:0/2179991198 shutdown_connections 2024-04-24T08:21:42.899 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:42.898+0000 7f1d5dc00640 1 -- 172.21.15.69:0/2179991198 wait complete. 2024-04-24T08:21:42.917 INFO:tasks.ceph:Scrubbing osd.5 2024-04-24T08:21:42.917 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-24T08:21:43.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.122+0000 7fcbf5a00640 1 Processor -- start 2024-04-24T08:21:43.123 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.122+0000 7fcbf5a00640 1 -- start start 2024-04-24T08:21:43.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.123+0000 7fcbf5a00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fcbf0073b40 0x7fcbf0071b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.123+0000 7fcbf5a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fcbf0072050 0x7fcbf00ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.124+0000 7fcbf5a00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf00ce8c0 con 0x7fcbf015c5f0 2024-04-24T08:21:43.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.124+0000 7fcbf5a00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fcbf00acf30 con 0x7fcbf0072050 2024-04-24T08:21:43.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.124+0000 7fcbf5a00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fcbf015c9d0 con 0x7fcbf0073b40 2024-04-24T08:21:43.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.124+0000 7fcbefe00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fcbf015c5f0 0x7fcbf00ce020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42740/0 (socket says 172.21.15.69:42740) 2024-04-24T08:21:43.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.124+0000 7fcbefe00640 1 -- 172.21.15.69:0/1561393201 learned_addr learned my addr 172.21.15.69:0/1561393201 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:43.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.124+0000 7fcbf5000640 1 --2- 172.21.15.69:0/1561393201 >> v2:172.21.15.69:3300/0 conn(0x7fcbf0072050 0x7fcbf00ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.124+0000 7fcbe7400640 1 --2- 172.21.15.69:0/1561393201 >> v2:172.21.15.121:3300/0 conn(0x7fcbf0073b40 0x7fcbf0071b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.124+0000 7fcbf5000640 1 -- 172.21.15.69:0/1561393201 >> v2:172.21.15.121:3300/0 conn(0x7fcbf0073b40 msgr2=0x7fcbf0071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:43.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.124+0000 7fcbf5000640 1 --2- 172.21.15.69:0/1561393201 >> v2:172.21.15.121:3300/0 conn(0x7fcbf0073b40 0x7fcbf0071b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.124+0000 7fcbf5000640 1 -- 172.21.15.69:0/1561393201 >> v1:172.21.15.69:6789/0 conn(0x7fcbf015c5f0 legacy=0x7fcbf00ce020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.125+0000 7fcbf5000640 1 -- 172.21.15.69:0/1561393201 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf00ceba0 con 0x7fcbf0072050 2024-04-24T08:21:43.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.125+0000 7fcbef400640 1 -- 172.21.15.69:0/1561393201 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2105265197 0 0) 0x7fcbf00ce8c0 con 0x7fcbf015c5f0 2024-04-24T08:21:43.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.126+0000 7fcbf5000640 1 --2- 172.21.15.69:0/1561393201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbf0072050 0x7fcbf00ad6a0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009d90 tx=0x7fcbe802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a4dad86da42a7d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.126+0000 7fcbef400640 1 -- 172.21.15.69:0/1561393201 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8031600 con 0x7fcbf0072050 2024-04-24T08:21:43.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.126+0000 7fcbef400640 1 -- 172.21.15.69:0/1561393201 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbe80317a0 con 0x7fcbf0072050 2024-04-24T08:21:43.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.126+0000 7fcbef400640 1 -- 172.21.15.69:0/1561393201 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8031b70 con 0x7fcbf0072050 2024-04-24T08:21:43.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.126+0000 7fcbf5a00640 1 -- 172.21.15.69:0/1561393201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbf0072050 msgr2=0x7fcbf00ad6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.126+0000 7fcbf5a00640 1 --2- 172.21.15.69:0/1561393201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbf0072050 0x7fcbf00ad6a0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009d90 tx=0x7fcbe802a610 comp rx=0 tx=0).stop 2024-04-24T08:21:43.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.127+0000 7fcbf5a00640 1 -- 172.21.15.69:0/1561393201 shutdown_connections 2024-04-24T08:21:43.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.127+0000 7fcbf5a00640 1 --2- 172.21.15.69:0/1561393201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbf0072050 0x7fcbf00ad6a0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.127+0000 7fcbf5a00640 1 --2- 172.21.15.69:0/1561393201 >> v2:172.21.15.121:3300/0 conn(0x7fcbf0073b40 0x7fcbf0071b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.127+0000 7fcbf5a00640 1 -- 172.21.15.69:0/1561393201 >> 172.21.15.69:0/1561393201 conn(0x7fcbf00b1c50 msgr2=0x7fcbf00b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:43.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.127+0000 7fcbf5a00640 1 -- 172.21.15.69:0/1561393201 shutdown_connections 2024-04-24T08:21:43.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.127+0000 7fcbf5a00640 1 -- 172.21.15.69:0/1561393201 wait complete. 2024-04-24T08:21:43.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.128+0000 7fcbf5a00640 1 Processor -- start 2024-04-24T08:21:43.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.129+0000 7fcbf5a00640 1 -- start start 2024-04-24T08:21:43.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.129+0000 7fcbf5a00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbf0072050 0x7fcbf01f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.129+0000 7fcbf5a00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcbf0073b40 0x7fcbf0158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.129+0000 7fcbf5a00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf015c5f0 0x7fcbf01f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.129+0000 7fcbf5a00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf00724a0 con 0x7fcbf0072050 2024-04-24T08:21:43.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.129+0000 7fcbf5a00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf00ad180 con 0x7fcbf015c5f0 2024-04-24T08:21:43.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.129+0000 7fcbf5a00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf00adbe0 con 0x7fcbf0073b40 2024-04-24T08:21:43.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.130+0000 7fcbefe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbf0072050 0x7fcbf01f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.130+0000 7fcbefe00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbf0072050 0x7fcbf01f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52530/0 (socket says 172.21.15.69:52530) 2024-04-24T08:21:43.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.130+0000 7fcbefe00640 1 -- 172.21.15.69:0/913758647 learned_addr learned my addr 172.21.15.69:0/913758647 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:43.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.130+0000 7fcbefe00640 1 -- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcbf0073b40 msgr2=0x7fcbf0158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.130+0000 7fcbe7400640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcbf0073b40 0x7fcbf0158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.130+0000 7fcbf5000640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf015c5f0 0x7fcbf01f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.130+0000 7fcbefe00640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcbf0073b40 0x7fcbf0158410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.130+0000 7fcbefe00640 1 -- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf015c5f0 msgr2=0x7fcbf01f5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.130+0000 7fcbefe00640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbf015c5f0 0x7fcbf01f5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.130+0000 7fcbefe00640 1 -- 172.21.15.69:0/913758647 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf01f63d0 con 0x7fcbf0072050 2024-04-24T08:21:43.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.131+0000 7fcbefe00640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbf0072050 0x7fcbf01f1e50 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000c430 tx=0x7fcbe000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.131+0000 7fcbecc00640 1 -- 172.21.15.69:0/913758647 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe00074e0 con 0x7fcbf0072050 2024-04-24T08:21:43.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.131+0000 7fcbecc00640 1 -- 172.21.15.69:0/913758647 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbe00047e0 con 0x7fcbf0072050 2024-04-24T08:21:43.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.131+0000 7fcbecc00640 1 -- 172.21.15.69:0/913758647 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0004bd0 con 0x7fcbf0072050 2024-04-24T08:21:43.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.131+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf01f24c0 con 0x7fcbf0072050 2024-04-24T08:21:43.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.131+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf01f2a80 con 0x7fcbf0072050 2024-04-24T08:21:43.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.132+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcbf0075d50 con 0x7fcbf0072050 2024-04-24T08:21:43.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.133+0000 7fcbecc00640 1 -- 172.21.15.69:0/913758647 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcbe0011070 con 0x7fcbf0072050 2024-04-24T08:21:43.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.134+0000 7fcbecc00640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcbcc06c8f0 0x7fcbcc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.134+0000 7fcbecc00640 1 -- 172.21.15.69:0/913758647 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fcbe008b560 con 0x7fcbf0072050 2024-04-24T08:21:43.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.134+0000 7fcbecc00640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x7fcbcc0728f0 0x7fcbcc074d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.134+0000 7fcbecc00640 1 -- 172.21.15.69:0/913758647 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbe0018e50 con 0x7fcbcc0728f0 2024-04-24T08:21:43.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.135+0000 7fcbecc00640 1 -- 172.21.15.69:0/913758647 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fcbe0092050 con 0x7fcbf0072050 2024-04-24T08:21:43.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.135+0000 7fcbe7400640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcbcc06c8f0 0x7fcbcc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.135+0000 7fcbf5000640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x7fcbcc0728f0 0x7fcbcc074d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.136+0000 7fcbe7400640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcbcc06c8f0 0x7fcbcc06edb0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fcbf01e8350 tx=0x7fcbd8008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.136+0000 7fcbf5000640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x7fcbcc0728f0 0x7fcbcc074d10 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.138+0000 7fcbecc00640 1 -- 172.21.15.69:0/913758647 <== osd.5 v2:172.21.15.121:6822/2187246263 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcbe0018e50 con 0x7fcbcc0728f0 2024-04-24T08:21:43.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.153+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 --> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcbf00c8b10 con 0x7fcbcc0728f0 2024-04-24T08:21:43.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.154+0000 7fcbecc00640 1 -- 172.21.15.69:0/913758647 <== osd.5 v2:172.21.15.121:6822/2187246263 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcbf00c8b10 con 0x7fcbcc0728f0 2024-04-24T08:21:43.155 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:43.155 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T08:21:43.155 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x7fcbcc0728f0 msgr2=0x7fcbcc074d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5a00640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:6822/2187246263,v1:172.21.15.121:6824/2187246263] conn(0x7fcbcc0728f0 0x7fcbcc074d10 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcbcc06c8f0 msgr2=0x7fcbcc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5a00640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fcbcc06c8f0 0x7fcbcc06edb0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fcbf01e8350 tx=0x7fcbd8008040 comp rx=0 tx=0).stop 2024-04-24T08:21:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbf0072050 msgr2=0x7fcbf01f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5a00640 1 --2- 172.21.15.69:0/913758647 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbf0072050 0x7fcbf01f1e50 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fcbe000c430 tx=0x7fcbe000c8f0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5000640 1 -- 172.21.15.69:0/913758647 reap_dead start 2024-04-24T08:21:43.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 shutdown_connections 2024-04-24T08:21:43.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 >> 172.21.15.69:0/913758647 conn(0x7fcbf00b1c50 msgr2=0x7fcbf006f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:43.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 shutdown_connections 2024-04-24T08:21:43.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.155+0000 7fcbf5a00640 1 -- 172.21.15.69:0/913758647 wait complete. 2024-04-24T08:21:43.172 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-24T08:21:43.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.364+0000 7ff91f000640 1 Processor -- start 2024-04-24T08:21:43.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.365+0000 7ff91f000640 1 -- start start 2024-04-24T08:21:43.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.366+0000 7ff91f000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7ff91809b0e0 0x7ff91815e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.366+0000 7ff91f000640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff91806df60 con 0x7ff9180cea60 2024-04-24T08:21:43.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.366+0000 7ff91f000640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9180d0db0 con 0x7ff9180d0820 2024-04-24T08:21:43.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.366+0000 7ff91f000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7ff9180cee40 con 0x7ff91809b0e0 2024-04-24T08:21:43.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.366+0000 7ff917e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7ff9180cea60 0x7ff91806d850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42750/0 (socket says 172.21.15.69:42750) 2024-04-24T08:21:43.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.366+0000 7ff917e00640 1 -- 172.21.15.69:0/653720423 learned_addr learned my addr 172.21.15.69:0/653720423 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:43.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.366+0000 7ff91d000640 1 --2- 172.21.15.69:0/653720423 >> v2:172.21.15.69:3300/0 conn(0x7ff91809b0e0 0x7ff91815e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.366+0000 7ff916a00640 1 -- 172.21.15.69:0/653720423 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2111055654 0 0) 0x7ff91806df60 con 0x7ff9180cea60 2024-04-24T08:21:43.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.366+0000 7ff916a00640 1 -- 172.21.15.69:0/653720423 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8fc003520 con 0x7ff9180cea60 2024-04-24T08:21:43.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.367+0000 7ff91d000640 1 -- 172.21.15.69:0/653720423 >> v1:172.21.15.121:6789/0 conn(0x7ff9180d0820 legacy=0x7ff91809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:43.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.367+0000 7ff91d000640 1 -- 172.21.15.69:0/653720423 >> v1:172.21.15.69:6789/0 conn(0x7ff9180cea60 legacy=0x7ff91806d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.367+0000 7ff91d000640 1 -- 172.21.15.69:0/653720423 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9180d1090 con 0x7ff91809b0e0 2024-04-24T08:21:43.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.368+0000 7ff91d000640 1 --2- 172.21.15.69:0/653720423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff91809b0e0 0x7ff91815e1c0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7ff910009d90 tx=0x7ff91002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83489f931ca8abb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.368+0000 7ff916a00640 1 -- 172.21.15.69:0/653720423 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff910031600 con 0x7ff91809b0e0 2024-04-24T08:21:43.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.368+0000 7ff916a00640 1 -- 172.21.15.69:0/653720423 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9100317a0 con 0x7ff91809b0e0 2024-04-24T08:21:43.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.368+0000 7ff916a00640 1 -- 172.21.15.69:0/653720423 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff910031b70 con 0x7ff91809b0e0 2024-04-24T08:21:43.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.369+0000 7ff91f000640 1 -- 172.21.15.69:0/653720423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff91809b0e0 msgr2=0x7ff91815e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.369+0000 7ff91f000640 1 --2- 172.21.15.69:0/653720423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff91809b0e0 0x7ff91815e1c0 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7ff910009d90 tx=0x7ff91002a610 comp rx=0 tx=0).stop 2024-04-24T08:21:43.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.369+0000 7ff91f000640 1 -- 172.21.15.69:0/653720423 shutdown_connections 2024-04-24T08:21:43.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.369+0000 7ff91f000640 1 --2- 172.21.15.69:0/653720423 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff91809b0e0 0x7ff91815e1c0 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.369+0000 7ff91f000640 1 -- 172.21.15.69:0/653720423 >> 172.21.15.69:0/653720423 conn(0x7ff9180b4040 msgr2=0x7ff9180b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:43.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.369+0000 7ff91f000640 1 -- 172.21.15.69:0/653720423 shutdown_connections 2024-04-24T08:21:43.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.369+0000 7ff91f000640 1 -- 172.21.15.69:0/653720423 wait complete. 2024-04-24T08:21:43.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.370+0000 7ff91f000640 1 Processor -- start 2024-04-24T08:21:43.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.371+0000 7ff91f000640 1 -- start start 2024-04-24T08:21:43.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.371+0000 7ff91f000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff91809b0e0 0x7ff9180b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.371+0000 7ff91f000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9180cea60 0x7ff91815ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.371+0000 7ff91f000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9180d0820 0x7ff91815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.371+0000 7ff91f000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff9180d1930 con 0x7ff9180d0820 2024-04-24T08:21:43.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.371+0000 7ff91f000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7ff9180d1650 con 0x7ff91809b0e0 2024-04-24T08:21:43.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.371+0000 7ff91f000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7ff9180cdc40 con 0x7ff9180cea60 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.372+0000 7ff91d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9180d0820 0x7ff91815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.372+0000 7ff91d000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9180d0820 0x7ff91815a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52542/0 (socket says 172.21.15.69:52542) 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.372+0000 7ff91d000640 1 -- 172.21.15.69:0/50507145 learned_addr learned my addr 172.21.15.69:0/50507145 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.372+0000 7ff91d000640 1 -- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9180cea60 msgr2=0x7ff91815ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.372+0000 7ff91d000640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9180cea60 0x7ff91815ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.372+0000 7ff91d000640 1 -- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff91809b0e0 msgr2=0x7ff9180b2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.372+0000 7ff91d000640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff91809b0e0 0x7ff9180b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.372+0000 7ff91d000640 1 -- 172.21.15.69:0/50507145 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff91815af20 con 0x7ff9180d0820 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.373+0000 7ff91d000640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9180d0820 0x7ff91815a360 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7ff910009d90 tx=0x7ff910004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.373+0000 7ff90fe00640 1 -- 172.21.15.69:0/50507145 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff910040070 con 0x7ff9180d0820 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.373+0000 7ff90fe00640 1 -- 172.21.15.69:0/50507145 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff910030b60 con 0x7ff9180d0820 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.373+0000 7ff90fe00640 1 -- 172.21.15.69:0/50507145 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff910009450 con 0x7ff9180d0820 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.373+0000 7ff91f000640 1 -- 172.21.15.69:0/50507145 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9181586d0 con 0x7ff9180d0820 2024-04-24T08:21:43.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.373+0000 7ff91f000640 1 -- 172.21.15.69:0/50507145 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff918158c90 con 0x7ff9180d0820 2024-04-24T08:21:43.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.374+0000 7ff91f000640 1 -- 172.21.15.69:0/50507145 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff918068a80 con 0x7ff9180d0820 2024-04-24T08:21:43.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.375+0000 7ff90fe00640 1 -- 172.21.15.69:0/50507145 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff910039040 con 0x7ff9180d0820 2024-04-24T08:21:43.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.376+0000 7ff90fe00640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff8f406c8f0 0x7ff8f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.376+0000 7ff90fe00640 1 -- 172.21.15.69:0/50507145 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7ff9100ac780 con 0x7ff9180d0820 2024-04-24T08:21:43.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.377+0000 7ff917e00640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff8f406c8f0 0x7ff8f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.378+0000 7ff917e00640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff8f406c8f0 0x7ff8f406edb0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff908000f60 tx=0x7ff9080078f0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.380+0000 7ff90fe00640 1 -- 172.21.15.69:0/50507145 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff91007ae90 con 0x7ff9180d0820 2024-04-24T08:21:43.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.581+0000 7ff91f000640 1 -- 172.21.15.69:0/50507145 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7ff91815ea60 con 0x7ff8f406c8f0 2024-04-24T08:21:43.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.583+0000 7ff90fe00640 1 -- 172.21.15.69:0/50507145 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff908005890 con 0x7ff8f406c8f0 2024-04-24T08:21:43.583 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 5 to deep-scrub 2024-04-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 -- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff8f406c8f0 msgr2=0x7ff8f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff8f406c8f0 0x7ff8f406edb0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff908000f60 tx=0x7ff9080078f0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 -- 172.21.15.69:0/50507145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9180d0820 msgr2=0x7ff91815a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9180d0820 0x7ff91815a360 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7ff910009d90 tx=0x7ff910004a40 comp rx=0 tx=0).stop 2024-04-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 -- 172.21.15.69:0/50507145 shutdown_connections 2024-04-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7ff8f406c8f0 0x7ff8f406edb0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff9180d0820 0x7ff91815a360 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff9180cea60 0x7ff91815ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 --2- 172.21.15.69:0/50507145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff91809b0e0 0x7ff9180b2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 -- 172.21.15.69:0/50507145 >> 172.21.15.69:0/50507145 conn(0x7ff9180b4040 msgr2=0x7ff91806d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:43.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 -- 172.21.15.69:0/50507145 shutdown_connections 2024-04-24T08:21:43.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.585+0000 7ff91f000640 1 -- 172.21.15.69:0/50507145 wait complete. 2024-04-24T08:21:43.604 INFO:tasks.ceph:Scrubbing osd.6 2024-04-24T08:21:43.604 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-24T08:21:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.808+0000 7fba69600640 1 Processor -- start 2024-04-24T08:21:43.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.809+0000 7fba69600640 1 -- start start 2024-04-24T08:21:43.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.810+0000 7fba69600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fba640d0ed0 0x7fba6415bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.810+0000 7fba69600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba640b3fb0 0x7fba6415c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.810+0000 7fba69600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba6415ca40 con 0x7fba640aeb00 2024-04-24T08:21:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.810+0000 7fba69600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fba640b31f0 con 0x7fba640d0ed0 2024-04-24T08:21:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.810+0000 7fba69600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fba640b34d0 con 0x7fba640b3fb0 2024-04-24T08:21:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.810+0000 7fba62a00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fba640aeb00 0x7fba640d07c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42760/0 (socket says 172.21.15.69:42760) 2024-04-24T08:21:43.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.810+0000 7fba62a00640 1 -- 172.21.15.69:0/3318550155 learned_addr learned my addr 172.21.15.69:0/3318550155 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba63400640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba640b3fb0 0x7fba6415c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3036384725 0 0) 0x7fba6415ca40 con 0x7fba640aeb00 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba5a000640 1 --2- 172.21.15.69:0/3318550155 >> v2:172.21.15.121:3300/0 conn(0x7fba640d0ed0 0x7fba6415bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba48003520 con 0x7fba640aeb00 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 59568354 0 0) 0x7fba48003520 con 0x7fba640aeb00 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba640b3fb0 msgr2=0x7fba6415c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba640b3fb0 0x7fba6415c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 >> v2:172.21.15.121:3300/0 conn(0x7fba640d0ed0 msgr2=0x7fba6415bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 --2- 172.21.15.69:0/3318550155 >> v2:172.21.15.121:3300/0 conn(0x7fba640d0ed0 0x7fba6415bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba640b37b0 con 0x7fba640aeb00 2024-04-24T08:21:43.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fba54001690 con 0x7fba640aeb00 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 >> v1:172.21.15.69:6789/0 conn(0x7fba640aeb00 legacy=0x7fba640d07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba640b3fb0 0x7fba480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba48003f30 0x7fba48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba48004d40 0x7fba480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba640b31f0 con 0x7fba48004d40 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fba640b34d0 con 0x7fba48003f30 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.811+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fba4800ba10 con 0x7fba640b3fb0 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.812+0000 7fba63400640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba48004d40 0x7fba480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.812+0000 7fba5a000640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba48003f30 0x7fba48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.812+0000 7fba63400640 1 -- 172.21.15.69:0/3318550155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba640b3fb0 msgr2=0x7fba480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.812+0000 7fba63400640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba640b3fb0 0x7fba480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.812+0000 7fba63400640 1 -- 172.21.15.69:0/3318550155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba48003f30 msgr2=0x7fba48004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.812+0000 7fba63400640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba48003f30 0x7fba48004800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.812+0000 7fba63400640 1 -- 172.21.15.69:0/3318550155 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba48003520 con 0x7fba48004d40 2024-04-24T08:21:43.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.813+0000 7fba63400640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba48004d40 0x7fba480051c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009f90 tx=0x7fba48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a89562d52cd430f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.813+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c009c10 con 0x7fba48004d40 2024-04-24T08:21:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.813+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba5c036070 con 0x7fba48004d40 2024-04-24T08:21:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.813+0000 7fba62000640 1 -- 172.21.15.69:0/3318550155 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c031880 con 0x7fba48004d40 2024-04-24T08:21:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.814+0000 7fba69600640 1 -- 172.21.15.69:0/3318550155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba48004d40 msgr2=0x7fba480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.814+0000 7fba69600640 1 --2- 172.21.15.69:0/3318550155 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba48004d40 0x7fba480051c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009f90 tx=0x7fba48005830 comp rx=0 tx=0).stop 2024-04-24T08:21:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.814+0000 7fba63400640 1 -- 172.21.15.69:0/3318550155 reap_dead start 2024-04-24T08:21:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.814+0000 7fba69600640 1 -- 172.21.15.69:0/3318550155 shutdown_connections 2024-04-24T08:21:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.814+0000 7fba69600640 1 -- 172.21.15.69:0/3318550155 >> 172.21.15.69:0/3318550155 conn(0x7fba6406be40 msgr2=0x7fba6406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.814+0000 7fba69600640 1 -- 172.21.15.69:0/3318550155 shutdown_connections 2024-04-24T08:21:43.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.814+0000 7fba69600640 1 -- 172.21.15.69:0/3318550155 wait complete. 2024-04-24T08:21:43.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.816+0000 7fba69600640 1 Processor -- start 2024-04-24T08:21:43.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.816+0000 7fba69600640 1 -- start start 2024-04-24T08:21:43.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.817+0000 7fba69600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba48004d40 0x7fba640ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.817+0000 7fba69600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba640b25f0 0x7fba640cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.817+0000 7fba69600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba640c51c0 0x7fba640c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.817+0000 7fba69600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba640aeee0 con 0x7fba640c51c0 2024-04-24T08:21:43.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.817+0000 7fba69600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fba6415c210 con 0x7fba640b25f0 2024-04-24T08:21:43.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.817+0000 7fba69600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fba640b17a0 con 0x7fba48004d40 2024-04-24T08:21:43.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.817+0000 7fba63400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba640c51c0 0x7fba640c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.817+0000 7fba63400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba640c51c0 0x7fba640c5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52554/0 (socket says 172.21.15.69:52554) 2024-04-24T08:21:43.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.817+0000 7fba63400640 1 -- 172.21.15.69:0/3913437818 learned_addr learned my addr 172.21.15.69:0/3913437818 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:43.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.818+0000 7fba63400640 1 -- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba48004d40 msgr2=0x7fba640ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.818+0000 7fba63400640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fba48004d40 0x7fba640ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.818+0000 7fba63400640 1 -- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba640b25f0 msgr2=0x7fba640cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.818+0000 7fba63400640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fba640b25f0 0x7fba640cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.818+0000 7fba63400640 1 -- 172.21.15.69:0/3913437818 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba640b1a10 con 0x7fba640c51c0 2024-04-24T08:21:43.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.818+0000 7fba63400640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba640c51c0 0x7fba640c5640 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009f90 tx=0x7fba5c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.818+0000 7fba5b400640 1 -- 172.21.15.69:0/3913437818 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c009c10 con 0x7fba640c51c0 2024-04-24T08:21:43.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.818+0000 7fba5b400640 1 -- 172.21.15.69:0/3913437818 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba5c031dd0 con 0x7fba640c51c0 2024-04-24T08:21:43.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.818+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba640b1cf0 con 0x7fba640c51c0 2024-04-24T08:21:43.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.818+0000 7fba5b400640 1 -- 172.21.15.69:0/3913437818 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c003bd0 con 0x7fba640c51c0 2024-04-24T08:21:43.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.819+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba640b2a70 con 0x7fba640c51c0 2024-04-24T08:21:43.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.820+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fba64075dc0 con 0x7fba640c51c0 2024-04-24T08:21:43.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.821+0000 7fba5b400640 1 -- 172.21.15.69:0/3913437818 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fba5c005d10 con 0x7fba640c51c0 2024-04-24T08:21:43.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.822+0000 7fba5b400640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba4006c920 0x7fba4006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.822+0000 7fba5b400640 1 -- 172.21.15.69:0/3913437818 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fba5c0ad080 con 0x7fba640c51c0 2024-04-24T08:21:43.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.822+0000 7fba5b400640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x7fba40072920 0x7fba40074d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:43.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.822+0000 7fba5b400640 1 -- 172.21.15.69:0/3913437818 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba5c030ec0 con 0x7fba40072920 2024-04-24T08:21:43.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.822+0000 7fba5b400640 1 -- 172.21.15.69:0/3913437818 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7fba5c0b4050 con 0x7fba640c51c0 2024-04-24T08:21:43.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.823+0000 7fba62a00640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba4006c920 0x7fba4006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.823+0000 7fba5a000640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x7fba40072920 0x7fba40074d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:43.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.824+0000 7fba5a000640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x7fba40072920 0x7fba40074d40 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.824+0000 7fba62a00640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba4006c920 0x7fba4006ede0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fba640ae1e0 tx=0x7fba54001500 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:43.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.827+0000 7fba5b400640 1 -- 172.21.15.69:0/3913437818 <== osd.6 v2:172.21.15.121:6807/3068198325 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba5c030ec0 con 0x7fba40072920 2024-04-24T08:21:43.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.841+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 --> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fba640b4ff0 con 0x7fba40072920 2024-04-24T08:21:43.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.842+0000 7fba5b400640 1 -- 172.21.15.69:0/3913437818 <== osd.6 v2:172.21.15.121:6807/3068198325 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fba640b4ff0 con 0x7fba40072920 2024-04-24T08:21:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:43.843 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T08:21:43.843 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:43.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x7fba40072920 msgr2=0x7fba40074d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba69600640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:6807/3068198325,v1:172.21.15.121:6808/3068198325] conn(0x7fba40072920 0x7fba40074d40 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:43.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba4006c920 msgr2=0x7fba4006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba69600640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fba4006c920 0x7fba4006ede0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fba640ae1e0 tx=0x7fba54001500 comp rx=0 tx=0).stop 2024-04-24T08:21:43.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba640c51c0 msgr2=0x7fba640c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:43.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba69600640 1 --2- 172.21.15.69:0/3913437818 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba640c51c0 0x7fba640c5640 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fba5c009f90 tx=0x7fba5c009750 comp rx=0 tx=0).stop 2024-04-24T08:21:43.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba63400640 1 -- 172.21.15.69:0/3913437818 reap_dead start 2024-04-24T08:21:43.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 shutdown_connections 2024-04-24T08:21:43.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 >> 172.21.15.69:0/3913437818 conn(0x7fba6406be40 msgr2=0x7fba6406f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:43.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 shutdown_connections 2024-04-24T08:21:43.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:43.843+0000 7fba69600640 1 -- 172.21.15.69:0/3913437818 wait complete. 2024-04-24T08:21:43.860 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-24T08:21:44.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.051+0000 7f02b1200640 1 Processor -- start 2024-04-24T08:21:44.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.052+0000 7f02b1200640 1 -- start start 2024-04-24T08:21:44.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.053+0000 7f02b1200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac072230 0x7f02ac169690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.053+0000 7f02b1200640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02ac144db0 con 0x7f02ac162300 2024-04-24T08:21:44.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.053+0000 7f02b1200640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02ac15f540 con 0x7f02ac071e50 2024-04-24T08:21:44.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.053+0000 7f02b1200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f02ac15f820 con 0x7f02ac072230 2024-04-24T08:21:44.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.054+0000 7f02a3e00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f02ac162300 0x7f02ac143690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42774/0 (socket says 172.21.15.69:42774) 2024-04-24T08:21:44.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.054+0000 7f02a3e00640 1 -- 172.21.15.69:0/1150140938 learned_addr learned my addr 172.21.15.69:0/1150140938 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:44.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.054+0000 7f02abe00640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac072230 0x7f02ac169690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.054+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 961513024 0 0) 0x7f02ac144db0 con 0x7f02ac162300 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.054+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0290003520 con 0x7f02ac162300 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.054+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2384158614 0 0) 0x7f0290003520 con 0x7f02ac162300 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.054+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac072230 msgr2=0x7f02ac169690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac072230 0x7f02ac169690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 >> v1:172.21.15.121:6789/0 conn(0x7f02ac071e50 legacy=0x7f02ac15ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ac15fb00 con 0x7f02ac162300 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f0294001670 con 0x7f02ac162300 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 >> v1:172.21.15.69:6789/0 conn(0x7f02ac162300 legacy=0x7f02ac143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02900039e0 0x7f0290003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0290004390 0x7f0290004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac072230 0x7f02900050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac15f820 con 0x7f02900039e0 2024-04-24T08:21:44.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f029000b9c0 con 0x7f0290004390 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f029000bbc0 con 0x7f02ac072230 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.055+0000 7f02ab400640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02900039e0 0x7f0290003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.056+0000 7f02abe00640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac072230 0x7f02900050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.056+0000 7f02a3e00640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0290004390 0x7f0290004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.056+0000 7f02ab400640 1 -- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac072230 msgr2=0x7f02900050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.056+0000 7f02ab400640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac072230 0x7f02900050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.056+0000 7f02ab400640 1 -- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0290004390 msgr2=0x7f0290004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.056+0000 7f02ab400640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0290004390 0x7f0290004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.056+0000 7f02ab400640 1 -- 172.21.15.69:0/1150140938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0290003520 con 0x7f02900039e0 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.056+0000 7f02a3e00640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0290004390 0x7f0290004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.056+0000 7f02abe00640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac072230 0x7f02900050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.057+0000 7f02ab400640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02900039e0 0x7f0290003dc0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f029c001860 tx=0x7f0290005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caea66159a01e358 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.057+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c037070 con 0x7f02900039e0 2024-04-24T08:21:44.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.057+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f029c02ac30 con 0x7f02900039e0 2024-04-24T08:21:44.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.057+0000 7f02aaa00640 1 -- 172.21.15.69:0/1150140938 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f029c001430 con 0x7f02900039e0 2024-04-24T08:21:44.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.057+0000 7f02b1200640 1 -- 172.21.15.69:0/1150140938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02900039e0 msgr2=0x7f0290003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.057+0000 7f02b1200640 1 --2- 172.21.15.69:0/1150140938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02900039e0 0x7f0290003dc0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f029c001860 tx=0x7f0290005830 comp rx=0 tx=0).stop 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.058+0000 7f02abe00640 1 -- 172.21.15.69:0/1150140938 reap_dead start 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.058+0000 7f02b1200640 1 -- 172.21.15.69:0/1150140938 shutdown_connections 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.058+0000 7f02b1200640 1 -- 172.21.15.69:0/1150140938 >> 172.21.15.69:0/1150140938 conn(0x7f02ac0d0160 msgr2=0x7f02ac0d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.058+0000 7f02b1200640 1 -- 172.21.15.69:0/1150140938 shutdown_connections 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.058+0000 7f02b1200640 1 -- 172.21.15.69:0/1150140938 wait complete. 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.059+0000 7f02b1200640 1 Processor -- start 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.060+0000 7f02b1200640 1 -- start start 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.060+0000 7f02b1200640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02900039e0 0x7f02ac0aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.060+0000 7f02b1200640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02ac144830 0x7f02ac15fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.060+0000 7f02b1200640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac0999b0 0x7f02ac1603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.060+0000 7f02b1200640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac06f450 con 0x7f02ac144830 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.060+0000 7f02b1200640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac06f2d0 con 0x7f02900039e0 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02b1200640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f02ac167680 con 0x7f02ac0999b0 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02a3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02ac144830 0x7f02ac15fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02a3e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02ac144830 0x7f02ac15fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52572/0 (socket says 172.21.15.69:52572) 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02a3e00640 1 -- 172.21.15.69:0/583218796 learned_addr learned my addr 172.21.15.69:0/583218796 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02ab400640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02900039e0 0x7f02ac0aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02abe00640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac0999b0 0x7f02ac1603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02a3e00640 1 -- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac0999b0 msgr2=0x7f02ac1603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02a3e00640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac0999b0 0x7f02ac1603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02a3e00640 1 -- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02900039e0 msgr2=0x7f02ac0aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02a3e00640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02900039e0 0x7f02ac0aeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.061+0000 7f02a3e00640 1 -- 172.21.15.69:0/583218796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ac15dbb0 con 0x7f02ac144830 2024-04-24T08:21:44.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.062+0000 7f02a3e00640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02ac144830 0x7f02ac15fe70 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f029400d840 tx=0x7f029400dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:44.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.062+0000 7f02a3400640 1 -- 172.21.15.69:0/583218796 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0294019070 con 0x7f02ac144830 2024-04-24T08:21:44.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.062+0000 7f02a3400640 1 -- 172.21.15.69:0/583218796 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02940074e0 con 0x7f02ac144830 2024-04-24T08:21:44.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.062+0000 7f02a3400640 1 -- 172.21.15.69:0/583218796 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02940049e0 con 0x7f02ac144830 2024-04-24T08:21:44.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.062+0000 7f02b1200640 1 -- 172.21.15.69:0/583218796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ac15de90 con 0x7f02ac144830 2024-04-24T08:21:44.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.062+0000 7f02b1200640 1 -- 172.21.15.69:0/583218796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ac15e450 con 0x7f02ac144830 2024-04-24T08:21:44.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.063+0000 7f02b1200640 1 -- 172.21.15.69:0/583218796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02ac075d50 con 0x7f02ac144830 2024-04-24T08:21:44.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.064+0000 7f02a3400640 1 -- 172.21.15.69:0/583218796 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0294004b80 con 0x7f02ac144830 2024-04-24T08:21:44.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.065+0000 7f02a3400640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f028806c920 0x7f028806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.065+0000 7f02a3400640 1 -- 172.21.15.69:0/583218796 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f029408b070 con 0x7f02ac144830 2024-04-24T08:21:44.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.066+0000 7f02ab400640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f028806c920 0x7f028806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.067+0000 7f02ab400640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f028806c920 0x7f028806ede0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f029c001860 tx=0x7f029c009030 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:44.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.069+0000 7f02a3400640 1 -- 172.21.15.69:0/583218796 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0294059780 con 0x7f02ac144830 2024-04-24T08:21:44.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.266+0000 7f02b1200640 1 -- 172.21.15.69:0/583218796 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f02ac0d0d90 con 0x7f028806c920 2024-04-24T08:21:44.268 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.268+0000 7f02a3400640 1 -- 172.21.15.69:0/583218796 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f02ac0d0d90 con 0x7f028806c920 2024-04-24T08:21:44.269 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 6 to deep-scrub 2024-04-24T08:21:44.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.270+0000 7f02b1200640 1 -- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f028806c920 msgr2=0x7f028806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.271 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.270+0000 7f02b1200640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f028806c920 0x7f028806ede0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f029c001860 tx=0x7f029c009030 comp rx=0 tx=0).stop 2024-04-24T08:21:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.270+0000 7f02b1200640 1 -- 172.21.15.69:0/583218796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02ac144830 msgr2=0x7f02ac15fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.270+0000 7f02b1200640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02ac144830 0x7f02ac15fe70 secure :-1 s=READY pgs=2209 cs=0 l=1 rev1=1 crypto rx=0x7f029400d840 tx=0x7f029400dd00 comp rx=0 tx=0).stop 2024-04-24T08:21:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.271+0000 7f02b1200640 1 -- 172.21.15.69:0/583218796 shutdown_connections 2024-04-24T08:21:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.271+0000 7f02b1200640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f028806c920 0x7f028806ede0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.271+0000 7f02b1200640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f02ac144830 0x7f02ac15fe70 unknown :-1 s=CLOSED pgs=2209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.271+0000 7f02b1200640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f02ac0999b0 0x7f02ac1603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.271+0000 7f02b1200640 1 --2- 172.21.15.69:0/583218796 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02900039e0 0x7f02ac0aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.271+0000 7f02b1200640 1 -- 172.21.15.69:0/583218796 >> 172.21.15.69:0/583218796 conn(0x7f02ac0d0160 msgr2=0x7f02ac15c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.271+0000 7f02b1200640 1 -- 172.21.15.69:0/583218796 shutdown_connections 2024-04-24T08:21:44.272 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.271+0000 7f02b1200640 1 -- 172.21.15.69:0/583218796 wait complete. 2024-04-24T08:21:44.290 INFO:tasks.ceph:Scrubbing osd.7 2024-04-24T08:21:44.290 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-24T08:21:44.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.498+0000 7f6469400640 1 Processor -- start 2024-04-24T08:21:44.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.498+0000 7f6469400640 1 -- start start 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.499+0000 7f6469400640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f64640d07c0 0x7f64640d0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.499+0000 7f6469400640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f64640addc0 0x7f64640ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.499+0000 7f6469400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64640b4840 con 0x7f646415c590 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.499+0000 7f6469400640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f64640b4560 con 0x7f64640addc0 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.499+0000 7f6469400640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f64640ad2e0 con 0x7f64640d07c0 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.499+0000 7f6463200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f64640addc0 0x7f64640ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.500+0000 7f6463200640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f64640addc0 0x7f64640ae1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52580/0 (socket says 172.21.15.69:52580) 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.500+0000 7f6463200640 1 -- 172.21.15.69:0/2467611412 learned_addr learned my addr 172.21.15.69:0/2467611412 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.500+0000 7f6461e00640 1 --2- 172.21.15.69:0/2467611412 >> v2:172.21.15.121:3300/0 conn(0x7f64640d07c0 0x7f64640d0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.500+0000 7f6463200640 1 -- 172.21.15.69:0/2467611412 >> v2:172.21.15.121:3300/0 conn(0x7f64640d07c0 msgr2=0x7f64640d0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.500+0000 7f6463200640 1 --2- 172.21.15.69:0/2467611412 >> v2:172.21.15.121:3300/0 conn(0x7f64640d07c0 0x7f64640d0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.500+0000 7f6463200640 1 -- 172.21.15.69:0/2467611412 >> v1:172.21.15.121:6789/0 conn(0x7f646415c590 legacy=0x7f64640b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.500+0000 7f6463200640 1 -- 172.21.15.69:0/2467611412 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64640ad5c0 con 0x7f64640addc0 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.501+0000 7f6463200640 1 --2- 172.21.15.69:0/2467611412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64640addc0 0x7f64640ae1d0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f64640700b0 tx=0x7f645c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17c7ab8e0ca95aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.501+0000 7f6461400640 1 -- 172.21.15.69:0/2467611412 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c037070 con 0x7f64640addc0 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.502+0000 7f6461400640 1 -- 172.21.15.69:0/2467611412 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f645c032440 con 0x7f64640addc0 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.502+0000 7f6461400640 1 -- 172.21.15.69:0/2467611412 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c032810 con 0x7f64640addc0 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.502+0000 7f6469400640 1 -- 172.21.15.69:0/2467611412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64640addc0 msgr2=0x7f64640ae1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.502+0000 7f6469400640 1 --2- 172.21.15.69:0/2467611412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64640addc0 0x7f64640ae1d0 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7f64640700b0 tx=0x7f645c02a270 comp rx=0 tx=0).stop 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.502+0000 7f6469400640 1 -- 172.21.15.69:0/2467611412 shutdown_connections 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.502+0000 7f6469400640 1 --2- 172.21.15.69:0/2467611412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64640addc0 0x7f64640ae1d0 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.502+0000 7f6469400640 1 --2- 172.21.15.69:0/2467611412 >> v2:172.21.15.121:3300/0 conn(0x7f64640d07c0 0x7f64640d0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.502+0000 7f6469400640 1 -- 172.21.15.69:0/2467611412 >> 172.21.15.69:0/2467611412 conn(0x7f646406be40 msgr2=0x7f646406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.502+0000 7f6469400640 1 -- 172.21.15.69:0/2467611412 shutdown_connections 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.502+0000 7f6469400640 1 -- 172.21.15.69:0/2467611412 wait complete. 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.504+0000 7f6469400640 1 Processor -- start 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.504+0000 7f6469400640 1 -- start start 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.505+0000 7f6469400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64640addc0 0x7f64640b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.505+0000 7f6469400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64640d07c0 0x7f646415c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.505+0000 7f6469400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f646415c590 0x7f646415ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.505+0000 7f6469400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f64640b5160 con 0x7f64640addc0 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.505+0000 7f6469400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f64640b4240 con 0x7f646415c590 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.505+0000 7f6469400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f64640b52e0 con 0x7f64640d07c0 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.505+0000 7f6462800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64640addc0 0x7f64640b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.505+0000 7f6462800640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64640addc0 0x7f64640b2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52584/0 (socket says 172.21.15.69:52584) 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.505+0000 7f6462800640 1 -- 172.21.15.69:0/4106950039 learned_addr learned my addr 172.21.15.69:0/4106950039 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f6461e00640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64640d07c0 0x7f646415c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f6463200640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f646415c590 0x7f646415ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f6462800640 1 -- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64640d07c0 msgr2=0x7f646415c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f6462800640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f64640d07c0 0x7f646415c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f6462800640 1 -- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f646415c590 msgr2=0x7f646415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f6462800640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f646415c590 0x7f646415ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f6462800640 1 -- 172.21.15.69:0/4106950039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6464072a70 con 0x7f64640addc0 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f6462800640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64640addc0 0x7f64640b2810 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f645400c730 tx=0x7f645400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f645aa00640 1 -- 172.21.15.69:0/4106950039 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6454016020 con 0x7f64640addc0 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f645aa00640 1 -- 172.21.15.69:0/4106950039 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6454006d50 con 0x7f64640addc0 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f645aa00640 1 -- 172.21.15.69:0/4106950039 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6454003bb0 con 0x7f64640addc0 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.506+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f646415e6d0 con 0x7f64640addc0 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.507+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f646415ec50 con 0x7f64640addc0 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.507+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6464075dc0 con 0x7f64640addc0 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.509+0000 7f645aa00640 1 -- 172.21.15.69:0/4106950039 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f645400a850 con 0x7f64640addc0 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.509+0000 7f645aa00640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f644006c8f0 0x7f644006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.510+0000 7f645aa00640 1 -- 172.21.15.69:0/4106950039 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f645408b550 con 0x7f64640addc0 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.510+0000 7f645aa00640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x7f64400728f0 0x7f6440074d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.510+0000 7f645aa00640 1 -- 172.21.15.69:0/4106950039 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f645401eec0 con 0x7f64400728f0 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.510+0000 7f645aa00640 1 -- 172.21.15.69:0/4106950039 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) ==== 24+0+0 (secure 0 0 0) 0x7f6454092050 con 0x7f64640addc0 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.510+0000 7f6461e00640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f644006c8f0 0x7f644006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.510+0000 7f6463200640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x7f64400728f0 0x7f6440074d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.512+0000 7f6461e00640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f644006c8f0 0x7f644006edb0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f64640c3db0 tx=0x7f644c008040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.512+0000 7f6463200640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x7f64400728f0 0x7f6440074d10 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:44.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.514+0000 7f645aa00640 1 -- 172.21.15.69:0/4106950039 <== osd.7 v2:172.21.15.121:6811/176144805 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f645401eec0 con 0x7f64400728f0 2024-04-24T08:21:44.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.528+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 --> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f64640af740 con 0x7f64400728f0 2024-04-24T08:21:44.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.529+0000 7f645aa00640 1 -- 172.21.15.69:0/4106950039 <== osd.7 v2:172.21.15.121:6811/176144805 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f64640af740 con 0x7f64400728f0 2024-04-24T08:21:44.530 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-04-24T08:21:44.530 INFO:teuthology.orchestra.run.smithi069.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T08:21:44.530 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-04-24T08:21:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x7f64400728f0 msgr2=0x7f6440074d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6469400640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:6811/176144805,v1:172.21.15.121:6814/176144805] conn(0x7f64400728f0 0x7f6440074d10 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f644006c8f0 msgr2=0x7f644006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6469400640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f644006c8f0 0x7f644006edb0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f64640c3db0 tx=0x7f644c008040 comp rx=0 tx=0).stop 2024-04-24T08:21:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64640addc0 msgr2=0x7f64640b2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6469400640 1 --2- 172.21.15.69:0/4106950039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64640addc0 0x7f64640b2810 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7f645400c730 tx=0x7f645400cbf0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6463200640 1 -- 172.21.15.69:0/4106950039 reap_dead start 2024-04-24T08:21:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 shutdown_connections 2024-04-24T08:21:44.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 >> 172.21.15.69:0/4106950039 conn(0x7f646406be40 msgr2=0x7f6464073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:44.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 shutdown_connections 2024-04-24T08:21:44.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.530+0000 7f6469400640 1 -- 172.21.15.69:0/4106950039 wait complete. 2024-04-24T08:21:44.548 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-24T08:21:44.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.747+0000 7f20a1000640 1 Processor -- start 2024-04-24T08:21:44.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.747+0000 7f20a1000640 1 -- start start 2024-04-24T08:21:44.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.749+0000 7f20a1000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f209c15ea10 0x7f209c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.749+0000 7f20a1000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f209c1631a0 0x7f209c0af290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.749+0000 7f20a1000640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f209c0af7d0 0x7f209c071d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.749+0000 7f20a1000640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f209c0afc40 con 0x7f209c15ea10 2024-04-24T08:21:44.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.749+0000 7f20a1000640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f209c0adab0 con 0x7f209c0af7d0 2024-04-24T08:21:44.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.749+0000 7f20a1000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f209c0add50 con 0x7f209c1631a0 2024-04-24T08:21:44.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.749+0000 7f2099e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f209c15ea10 0x7f209c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.749+0000 7f2099e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f209c15ea10 0x7f209c0d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52594/0 (socket says 172.21.15.69:52594) 2024-04-24T08:21:44.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.749+0000 7f2099e00640 1 -- 172.21.15.69:0/2565900347 learned_addr learned my addr 172.21.15.69:0/2565900347 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:44.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.749+0000 7f2099400640 1 --2- 172.21.15.69:0/2565900347 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f209c1631a0 0x7f209c0af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.750+0000 7f209a800640 1 --2- 172.21.15.69:0/2565900347 >> v2:172.21.15.121:3300/0 conn(0x7f209c0af7d0 0x7f209c071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.750+0000 7f2099e00640 1 -- 172.21.15.69:0/2565900347 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f209c1631a0 msgr2=0x7f209c0af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:44.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.750+0000 7f2099e00640 1 --2- 172.21.15.69:0/2565900347 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f209c1631a0 0x7f209c0af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.750+0000 7f2099e00640 1 -- 172.21.15.69:0/2565900347 >> v2:172.21.15.121:3300/0 conn(0x7f209c0af7d0 msgr2=0x7f209c071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:44.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.750+0000 7f2099e00640 1 --2- 172.21.15.69:0/2565900347 >> v2:172.21.15.121:3300/0 conn(0x7f209c0af7d0 0x7f209c071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.750+0000 7f2099e00640 1 -- 172.21.15.69:0/2565900347 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209c0ae030 con 0x7f209c15ea10 2024-04-24T08:21:44.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.751+0000 7f2099e00640 1 --2- 172.21.15.69:0/2565900347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c15ea10 0x7f209c0d0830 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f209c071d00 tx=0x7f208c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc691d40405f103 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:44.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.752+0000 7f2098a00640 1 -- 172.21.15.69:0/2565900347 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208c037070 con 0x7f209c15ea10 2024-04-24T08:21:44.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.752+0000 7f2098a00640 1 -- 172.21.15.69:0/2565900347 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f208c032440 con 0x7f209c15ea10 2024-04-24T08:21:44.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.752+0000 7f2098a00640 1 -- 172.21.15.69:0/2565900347 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208c032810 con 0x7f209c15ea10 2024-04-24T08:21:44.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.752+0000 7f20a1000640 1 -- 172.21.15.69:0/2565900347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c15ea10 msgr2=0x7f209c0d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.752+0000 7f20a1000640 1 --2- 172.21.15.69:0/2565900347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c15ea10 0x7f209c0d0830 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7f209c071d00 tx=0x7f208c02a1e0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.753+0000 7f20a1000640 1 -- 172.21.15.69:0/2565900347 shutdown_connections 2024-04-24T08:21:44.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.753+0000 7f20a1000640 1 --2- 172.21.15.69:0/2565900347 >> v2:172.21.15.121:3300/0 conn(0x7f209c0af7d0 0x7f209c071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.753+0000 7f20a1000640 1 --2- 172.21.15.69:0/2565900347 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f209c1631a0 0x7f209c0af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.753+0000 7f20a1000640 1 --2- 172.21.15.69:0/2565900347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c15ea10 0x7f209c0d0830 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.753+0000 7f20a1000640 1 -- 172.21.15.69:0/2565900347 >> 172.21.15.69:0/2565900347 conn(0x7f209c067d90 msgr2=0x7f209c06ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:44.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.753+0000 7f20a1000640 1 -- 172.21.15.69:0/2565900347 shutdown_connections 2024-04-24T08:21:44.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.753+0000 7f20a1000640 1 -- 172.21.15.69:0/2565900347 wait complete. 2024-04-24T08:21:44.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.754+0000 7f20a1000640 1 Processor -- start 2024-04-24T08:21:44.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.755+0000 7f20a1000640 1 -- start start 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.755+0000 7f20a1000640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c0af7d0 0x7f209c1f4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.755+0000 7f20a1000640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f209c15ea10 0x7f209c160850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.755+0000 7f20a1000640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f209c1631a0 0x7f209c0cc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.755+0000 7f20a1000640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f209c0ae8d0 con 0x7f209c0af7d0 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.755+0000 7f20a1000640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f209c0ae5f0 con 0x7f209c1631a0 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.756+0000 7f20a1000640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f209c0aedf0 con 0x7f209c15ea10 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.756+0000 7f2099e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c0af7d0 0x7f209c1f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.756+0000 7f2099e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c0af7d0 0x7f209c1f4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52596/0 (socket says 172.21.15.69:52596) 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.756+0000 7f2099e00640 1 -- 172.21.15.69:0/2466313229 learned_addr learned my addr 172.21.15.69:0/2466313229 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.756+0000 7f2099e00640 1 -- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f209c15ea10 msgr2=0x7f209c160850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.756+0000 7f2099e00640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f209c15ea10 0x7f209c160850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.756+0000 7f2099e00640 1 -- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f209c1631a0 msgr2=0x7f209c0cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.756+0000 7f2099e00640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f209c1631a0 0x7f209c0cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.756+0000 7f2099e00640 1 -- 172.21.15.69:0/2466313229 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209c161310 con 0x7f209c0af7d0 2024-04-24T08:21:44.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.757+0000 7f2099e00640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c0af7d0 0x7f209c1f4360 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f208c02a220 tx=0x7f208c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:44.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.757+0000 7f2092000640 1 -- 172.21.15.69:0/2466313229 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208c037040 con 0x7f209c0af7d0 2024-04-24T08:21:44.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.757+0000 7f2092000640 1 -- 172.21.15.69:0/2466313229 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f208c042070 con 0x7f209c0af7d0 2024-04-24T08:21:44.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.757+0000 7f2092000640 1 -- 172.21.15.69:0/2466313229 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208c03d8a0 con 0x7f209c0af7d0 2024-04-24T08:21:44.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.757+0000 7f20a1000640 1 -- 172.21.15.69:0/2466313229 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f209c1615f0 con 0x7f209c0af7d0 2024-04-24T08:21:44.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.757+0000 7f20a1000640 1 -- 172.21.15.69:0/2466313229 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f209c153cf0 con 0x7f209c0af7d0 2024-04-24T08:21:44.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.758+0000 7f20a1000640 1 -- 172.21.15.69:0/2466313229 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f209c068a80 con 0x7f209c0af7d0 2024-04-24T08:21:44.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.759+0000 7f2092000640 1 -- 172.21.15.69:0/2466313229 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f208c03da40 con 0x7f209c0af7d0 2024-04-24T08:21:44.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.760+0000 7f2092000640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f207c06c8f0 0x7f207c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:44.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.760+0000 7f2092000640 1 -- 172.21.15.69:0/2466313229 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f208c0ac120 con 0x7f209c0af7d0 2024-04-24T08:21:44.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.761+0000 7f2099400640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f207c06c8f0 0x7f207c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:44.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.762+0000 7f2099400640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f207c06c8f0 0x7f207c06edb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2088006a90 tx=0x7f2088006a20 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:44.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.765+0000 7f2092000640 1 -- 172.21.15.69:0/2466313229 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f208c030e20 con 0x7f209c0af7d0 2024-04-24T08:21:44.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.963+0000 7f20a1000640 1 -- 172.21.15.69:0/2466313229 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f209c1617c0 con 0x7f207c06c8f0 2024-04-24T08:21:44.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.964+0000 7f2092000640 1 -- 172.21.15.69:0/2466313229 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f209c1617c0 con 0x7f207c06c8f0 2024-04-24T08:21:44.964 INFO:teuthology.orchestra.run.smithi069.stderr:instructed osd(s) 7 to deep-scrub 2024-04-24T08:21:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 -- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f207c06c8f0 msgr2=0x7f207c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f207c06c8f0 0x7f207c06edb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2088006a90 tx=0x7f2088006a20 comp rx=0 tx=0).stop 2024-04-24T08:21:44.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 -- 172.21.15.69:0/2466313229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c0af7d0 msgr2=0x7f209c1f4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:44.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c0af7d0 0x7f209c1f4360 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7f208c02a220 tx=0x7f208c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.967 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 -- 172.21.15.69:0/2466313229 shutdown_connections 2024-04-24T08:21:44.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f207c06c8f0 0x7f207c06edb0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f209c1631a0 0x7f209c0cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f209c15ea10 0x7f209c160850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 --2- 172.21.15.69:0/2466313229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f209c0af7d0 0x7f209c1f4360 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:44.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 -- 172.21.15.69:0/2466313229 >> 172.21.15.69:0/2466313229 conn(0x7f209c067d90 msgr2=0x7f209c162110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:44.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 -- 172.21.15.69:0/2466313229 shutdown_connections 2024-04-24T08:21:44.968 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:44.966+0000 7f20a1000640 1 -- 172.21.15.69:0/2466313229 wait complete. 2024-04-24T08:21:44.985 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:21:45.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.184+0000 7fdb3c400640 1 Processor -- start 2024-04-24T08:21:45.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.184+0000 7fdb3c400640 1 -- start start 2024-04-24T08:21:45.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.185+0000 7fdb3c400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb34145fe0 0x7fdb3406e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:45.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.185+0000 7fdb3c400640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb340cf3d0 con 0x7fdb34145a90 2024-04-24T08:21:45.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.185+0000 7fdb3c400640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb340cf6b0 con 0x7fdb340d0300 2024-04-24T08:21:45.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.185+0000 7fdb3c400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdb340cf990 con 0x7fdb34145fe0 2024-04-24T08:21:45.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.186+0000 7fdb39000640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7fdb34145a90 0x7fdb34160540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:42786/0 (socket says 172.21.15.69:42786) 2024-04-24T08:21:45.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.186+0000 7fdb39000640 1 -- 172.21.15.69:0/332572101 learned_addr learned my addr 172.21.15.69:0/332572101 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:21:45.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.186+0000 7fdb3a400640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb34145fe0 0x7fdb3406e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:45.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.186+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846580657 0 0) 0x7fdb340cf3d0 con 0x7fdb34145a90 2024-04-24T08:21:45.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.186+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb1c003520 con 0x7fdb34145a90 2024-04-24T08:21:45.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 616946461 0 0) 0x7fdb1c003520 con 0x7fdb34145a90 2024-04-24T08:21:45.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb34145fe0 msgr2=0x7fdb3406e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:45.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb34145fe0 0x7fdb3406e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:45.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 >> v1:172.21.15.121:6789/0 conn(0x7fdb340d0300 legacy=0x7fdb340d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb340cfc70 con 0x7fdb34145a90 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7fdb20001670 con 0x7fdb34145a90 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 >> v1:172.21.15.69:6789/0 conn(0x7fdb34145a90 legacy=0x7fdb34160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0039e0 0x7fdb1c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb34145fe0 0x7fdb1c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb1c004d40 0x7fdb1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb340cf990 con 0x7fdb1c004d40 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1c00baa0 con 0x7fdb1c0039e0 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdb1c00bca0 con 0x7fdb34145fe0 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb3a400640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb1c004d40 0x7fdb1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.187+0000 7fdb39a00640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0039e0 0x7fdb1c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.188+0000 7fdb39000640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb34145fe0 0x7fdb1c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.188+0000 7fdb3a400640 1 -- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb34145fe0 msgr2=0x7fdb1c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:45.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.188+0000 7fdb3a400640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb34145fe0 0x7fdb1c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:45.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.188+0000 7fdb3a400640 1 -- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0039e0 msgr2=0x7fdb1c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:45.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.188+0000 7fdb3a400640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0039e0 0x7fdb1c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:45.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.188+0000 7fdb3a400640 1 -- 172.21.15.69:0/332572101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1c003520 con 0x7fdb1c004d40 2024-04-24T08:21:45.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.188+0000 7fdb39a00640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0039e0 0x7fdb1c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:45.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.189+0000 7fdb3a400640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb1c004d40 0x7fdb1c0051c0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c009f90 tx=0x7fdb1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828013a5c315afbe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.189+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c009c10 con 0x7fdb1c004d40 2024-04-24T08:21:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.189+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb2c036070 con 0x7fdb1c004d40 2024-04-24T08:21:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.189+0000 7fdb33e00640 1 -- 172.21.15.69:0/332572101 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c031850 con 0x7fdb1c004d40 2024-04-24T08:21:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.190+0000 7fdb3c400640 1 -- 172.21.15.69:0/332572101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb1c004d40 msgr2=0x7fdb1c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.190+0000 7fdb3c400640 1 --2- 172.21.15.69:0/332572101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb1c004d40 0x7fdb1c0051c0 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c009f90 tx=0x7fdb1c005830 comp rx=0 tx=0).stop 2024-04-24T08:21:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.190+0000 7fdb3a400640 1 -- 172.21.15.69:0/332572101 reap_dead start 2024-04-24T08:21:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.190+0000 7fdb3c400640 1 -- 172.21.15.69:0/332572101 shutdown_connections 2024-04-24T08:21:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.190+0000 7fdb3c400640 1 -- 172.21.15.69:0/332572101 >> 172.21.15.69:0/332572101 conn(0x7fdb34067d90 msgr2=0x7fdb3406ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.190+0000 7fdb3c400640 1 -- 172.21.15.69:0/332572101 shutdown_connections 2024-04-24T08:21:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.190+0000 7fdb3c400640 1 -- 172.21.15.69:0/332572101 wait complete. 2024-04-24T08:21:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.192+0000 7fdb3c400640 1 Processor -- start 2024-04-24T08:21:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.192+0000 7fdb3c400640 1 -- start start 2024-04-24T08:21:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb3c400640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb1c004d40 0x7fdb340c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb3c400640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb340c5290 0x7fdb3415c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb3c400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb340ae5f0 0x7fdb340aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb3c400640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341458f0 con 0x7fdb340ae5f0 2024-04-24T08:21:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb3c400640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fdb34145770 con 0x7fdb340c5290 2024-04-24T08:21:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb3c400640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fdb340cdc30 con 0x7fdb1c004d40 2024-04-24T08:21:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb3a400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb340ae5f0 0x7fdb340aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb3a400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb340ae5f0 0x7fdb340aeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52616/0 (socket says 172.21.15.69:52616) 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb3a400640 1 -- 172.21.15.69:0/2710914994 learned_addr learned my addr 172.21.15.69:0/2710914994 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb39000640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb340c5290 0x7fdb3415c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.193+0000 7fdb39a00640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb1c004d40 0x7fdb340c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb3a400640 1 -- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb1c004d40 msgr2=0x7fdb340c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb3a400640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb1c004d40 0x7fdb340c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb3a400640 1 -- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb340c5290 msgr2=0x7fdb3415c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb3a400640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb340c5290 0x7fdb3415c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb3a400640 1 -- 172.21.15.69:0/2710914994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb340c5850 con 0x7fdb340ae5f0 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb3a400640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb340ae5f0 0x7fdb340aeaa0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c033040 tx=0x7fdb2c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb31600640 1 -- 172.21.15.69:0/2710914994 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c009c10 con 0x7fdb340ae5f0 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb31600640 1 -- 172.21.15.69:0/2710914994 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb2c003990 con 0x7fdb340ae5f0 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb31600640 1 -- 172.21.15.69:0/2710914994 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c003d80 con 0x7fdb340ae5f0 2024-04-24T08:21:45.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb3c400640 1 -- 172.21.15.69:0/2710914994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb340c5b30 con 0x7fdb340ae5f0 2024-04-24T08:21:45.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.194+0000 7fdb3c400640 1 -- 172.21.15.69:0/2710914994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb340c60f0 con 0x7fdb340ae5f0 2024-04-24T08:21:45.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.196+0000 7fdb3c400640 1 -- 172.21.15.69:0/2710914994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb34068a80 con 0x7fdb340ae5f0 2024-04-24T08:21:45.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.197+0000 7fdb31600640 1 -- 172.21.15.69:0/2710914994 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb2c005d10 con 0x7fdb340ae5f0 2024-04-24T08:21:45.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.197+0000 7fdb31600640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb1406c8f0 0x7fdb1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:45.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.198+0000 7fdb31600640 1 -- 172.21.15.69:0/2710914994 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fdb2c0acc30 con 0x7fdb340ae5f0 2024-04-24T08:21:45.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.198+0000 7fdb39a00640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb1406c8f0 0x7fdb1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:45.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.199+0000 7fdb39a00640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb1406c8f0 0x7fdb1406edb0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fdb280054d0 tx=0x7fdb28009040 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:45.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.202+0000 7fdb31600640 1 -- 172.21.15.69:0/2710914994 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb2c02e050 con 0x7fdb340ae5f0 2024-04-24T08:21:45.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.398+0000 7fdb3c400640 1 -- 172.21.15.69:0/2710914994 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdb340d1120 con 0x7fdb1406c8f0 2024-04-24T08:21:45.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.400+0000 7fdb31600640 1 -- 172.21.15.69:0/2710914994 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13370 (secure 0 0 0) 0x7fdb340d1120 con 0x7fdb1406c8f0 2024-04-24T08:21:45.401 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:21:45.401 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:21:45.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.402+0000 7fdb3c400640 1 -- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb1406c8f0 msgr2=0x7fdb1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:45.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.402+0000 7fdb3c400640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb1406c8f0 0x7fdb1406edb0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fdb280054d0 tx=0x7fdb28009040 comp rx=0 tx=0).stop 2024-04-24T08:21:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.402+0000 7fdb3c400640 1 -- 172.21.15.69:0/2710914994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb340ae5f0 msgr2=0x7fdb340aeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.402+0000 7fdb3c400640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb340ae5f0 0x7fdb340aeaa0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c033040 tx=0x7fdb2c009750 comp rx=0 tx=0).stop 2024-04-24T08:21:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.402+0000 7fdb3c400640 1 -- 172.21.15.69:0/2710914994 shutdown_connections 2024-04-24T08:21:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.402+0000 7fdb3c400640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fdb1406c8f0 0x7fdb1406edb0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.402+0000 7fdb3c400640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdb340ae5f0 0x7fdb340aeaa0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.402+0000 7fdb3c400640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb340c5290 0x7fdb3415c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.403+0000 7fdb3c400640 1 --2- 172.21.15.69:0/2710914994 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fdb1c004d40 0x7fdb340c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.403+0000 7fdb3c400640 1 -- 172.21.15.69:0/2710914994 >> 172.21.15.69:0/2710914994 conn(0x7fdb34067d90 msgr2=0x7fdb3415cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.403+0000 7fdb3c400640 1 -- 172.21.15.69:0/2710914994 shutdown_connections 2024-04-24T08:21:45.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:21:45.403+0000 7fdb3c400640 1 -- 172.21.15.69:0/2710914994 wait complete. 2024-04-24T08:21:45.423 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-04-24T08:21:44.966068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":261940,"kb_used_data":12488,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712780,"statfs":{"total":773094113280,"available":772825886720,"internally_reserved":0,"allocated":12787712,"data_stored":6235440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.203578"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":350,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-24T08:21:35.057468+0000","last_change":"2024-04-24T08:08:10.189168+0000","last_active":"2024-04-24T08:21:35.057468+0000","last_peered":"2024-04-24T08:21:35.057468+0000","last_clean":"2024-04-24T08:21:35.057468+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:21:35.057468+0000","last_undegraded":"2024-04-24T08:21:35.057468+0000","last_fullsized":"2024-04-24T08:21:35.057468+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:08:08.990431+0000","last_clean_scrub_stamp":"2024-04-24T08:08:08.990431+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:11.086365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33180,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338660,"statfs":{"total":96636764160,"available":96602787840,"internally_reserved":0,"allocated":2043904,"data_stored":1222206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575016,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32596,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32600,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339240,"statfs":{"total":96636764160,"available":96603381760,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32600,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339240,"statfs":{"total":96636764160,"available":96603381760,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32596,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32596,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32596,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575014,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33176,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338664,"statfs":{"total":96636764160,"available":96602791936,"internally_reserved":0,"allocated":2043904,"data_stored":1222206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:21:45.427 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T08:08:08.990431+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=8, tm_min=8, tm_sec=8, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=8, tm_min=21, tm_sec=38, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T08:21:45.427 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T08:21:46.962 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:21:46.970 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:22:05.428 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:22:05.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.630+0000 7f0445600640 1 Processor -- start 2024-04-24T08:22:05.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.630+0000 7f0445600640 1 -- start start 2024-04-24T08:22:05.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.632+0000 7f0445600640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0440145a90 0x7f044006e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.632+0000 7f0445600640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04400cf3d0 con 0x7f04400d0300 2024-04-24T08:22:05.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.632+0000 7f0445600640 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04400cf6b0 con 0x7f0440145fe0 2024-04-24T08:22:05.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.632+0000 7f0445600640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7f04400cf990 con 0x7f0440145a90 2024-04-24T08:22:05.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.632+0000 7f043ea00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f04400d0300 0x7f04400d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49880/0 (socket says 172.21.15.69:49880) 2024-04-24T08:22:05.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.632+0000 7f043ea00640 1 -- 172.21.15.69:0/4171708984 learned_addr learned my addr 172.21.15.69:0/4171708984 (peer_addr_for_me v1:172.21.15.69:0/0) 2024-04-24T08:22:05.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f0436000640 1 --2- 172.21.15.69:0/4171708984 >> v2:172.21.15.121:3300/0 conn(0x7f0440145a90 0x7f044006e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3858020739 0 0) 0x7f04400cf3d0 con 0x7f04400d0300 2024-04-24T08:22:05.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 --> v1:172.21.15.69:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0424003520 con 0x7f04400d0300 2024-04-24T08:22:05.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 <== mon.0 v1:172.21.15.69:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1892261744 0 0) 0x7f0424003520 con 0x7f04400d0300 2024-04-24T08:22:05.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 >> v2:172.21.15.121:3300/0 conn(0x7f0440145a90 msgr2=0x7f044006e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:05.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 --2- 172.21.15.69:0/4171708984 >> v2:172.21.15.121:3300/0 conn(0x7f0440145a90 0x7f044006e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.636 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 >> v1:172.21.15.121:6789/0 conn(0x7f0440145fe0 legacy=0x7f0440160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 --> v1:172.21.15.69:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04400cfc70 con 0x7f04400d0300 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 <== mon.0 v1:172.21.15.69:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 383934454 0 0) 0x7f0430001670 con 0x7f04400d0300 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 >> v1:172.21.15.69:6789/0 conn(0x7f04400d0300 legacy=0x7f04400d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 --2- 172.21.15.69:0/4171708984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0424003a00 0x7f0424003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 --2- 172.21.15.69:0/4171708984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f0424004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 --2- 172.21.15.69:0/4171708984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0424005210 0x7f042400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04400cf990 con 0x7f0424004800 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f042400c160 con 0x7f0424005210 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.633+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f042400c360 con 0x7f0424003a00 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.634+0000 7f0436000640 1 --2- 172.21.15.69:0/4171708984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f0424004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.634+0000 7f0436000640 1 -- 172.21.15.69:0/4171708984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0424003a00 msgr2=0x7f0424003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.634+0000 7f0436000640 1 --2- 172.21.15.69:0/4171708984 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0424003a00 0x7f0424003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.634+0000 7f0436000640 1 -- 172.21.15.69:0/4171708984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0424005210 msgr2=0x7f042400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.634+0000 7f0436000640 1 --2- 172.21.15.69:0/4171708984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0424005210 0x7f042400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.637 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.634+0000 7f0436000640 1 -- 172.21.15.69:0/4171708984 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0424003520 con 0x7f0424004800 2024-04-24T08:22:05.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.634+0000 7f043f400640 1 -- 172.21.15.69:0/4171708984 reap_dead start 2024-04-24T08:22:05.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.635+0000 7f0436000640 1 --2- 172.21.15.69:0/4171708984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f0424004c80 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f042800ba30 tx=0x7f0424005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417a09af603ca358 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:05.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.635+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428030070 con 0x7f0424004800 2024-04-24T08:22:05.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.635+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f042802ae10 con 0x7f0424004800 2024-04-24T08:22:05.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.635+0000 7f043e000640 1 -- 172.21.15.69:0/4171708984 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042800b600 con 0x7f0424004800 2024-04-24T08:22:05.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.636+0000 7f0445600640 1 -- 172.21.15.69:0/4171708984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 msgr2=0x7f0424004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.636+0000 7f0445600640 1 --2- 172.21.15.69:0/4171708984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f0424004c80 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f042800ba30 tx=0x7f0424005830 comp rx=0 tx=0).stop 2024-04-24T08:22:05.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.636+0000 7f0445600640 1 -- 172.21.15.69:0/4171708984 shutdown_connections 2024-04-24T08:22:05.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.636+0000 7f0445600640 1 --2- 172.21.15.69:0/4171708984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f0424004c80 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.636+0000 7f0445600640 1 -- 172.21.15.69:0/4171708984 >> 172.21.15.69:0/4171708984 conn(0x7f0440067d90 msgr2=0x7f044006ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:05.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.636+0000 7f0445600640 1 -- 172.21.15.69:0/4171708984 shutdown_connections 2024-04-24T08:22:05.639 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.636+0000 7f0445600640 1 -- 172.21.15.69:0/4171708984 wait complete. 2024-04-24T08:22:05.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.638+0000 7f0445600640 1 Processor -- start 2024-04-24T08:22:05.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.638+0000 7f0445600640 1 -- start start 2024-04-24T08:22:05.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.639+0000 7f0445600640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f04401415f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.639+0000 7f0445600640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f044013d150 0x7f044013b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.639+0000 7f0445600640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f044013e0e0 0x7f044013adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.639+0000 7f0445600640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04401458f0 con 0x7f0424004800 2024-04-24T08:22:05.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.639+0000 7f0445600640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f0440145770 con 0x7f044013e0e0 2024-04-24T08:22:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.639+0000 7f0445600640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f04400cdc30 con 0x7f044013d150 2024-04-24T08:22:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.639+0000 7f043ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f04401415f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.639+0000 7f043ea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f04401415f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47104/0 (socket says 172.21.15.69:47104) 2024-04-24T08:22:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.639+0000 7f043ea00640 1 -- 172.21.15.69:0/3489993547 learned_addr learned my addr 172.21.15.69:0/3489993547 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:22:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f043ea00640 1 -- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f044013d150 msgr2=0x7f044013b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f043ea00640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f044013d150 0x7f044013b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f043ea00640 1 -- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f044013e0e0 msgr2=0x7f044013adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f043ea00640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f044013e0e0 0x7f044013adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f043ea00640 1 -- 172.21.15.69:0/3489993547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044013c0b0 con 0x7f0424004800 2024-04-24T08:22:05.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f043ea00640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f04401415f0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f043000c790 tx=0x7f043000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:05.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f0437400640 1 -- 172.21.15.69:0/3489993547 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0430018070 con 0x7f0424004800 2024-04-24T08:22:05.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f0437400640 1 -- 172.21.15.69:0/3489993547 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04300072f0 con 0x7f0424004800 2024-04-24T08:22:05.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f0445600640 1 -- 172.21.15.69:0/3489993547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044013c390 con 0x7f0424004800 2024-04-24T08:22:05.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f0437400640 1 -- 172.21.15.69:0/3489993547 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04300047e0 con 0x7f0424004800 2024-04-24T08:22:05.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.640+0000 7f0445600640 1 -- 172.21.15.69:0/3489993547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044013c950 con 0x7f0424004800 2024-04-24T08:22:05.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.641+0000 7f0445600640 1 -- 172.21.15.69:0/3489993547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0440068a80 con 0x7f0424004800 2024-04-24T08:22:05.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.642+0000 7f0437400640 1 -- 172.21.15.69:0/3489993547 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f043001b070 con 0x7f0424004800 2024-04-24T08:22:05.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.643+0000 7f0437400640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06c920 0x7f041c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.643+0000 7f0437400640 1 -- 172.21.15.69:0/3489993547 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f043008af30 con 0x7f0424004800 2024-04-24T08:22:05.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.644+0000 7f0436000640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06c920 0x7f041c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.645+0000 7f0436000640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06c920 0x7f041c06ede0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f042800ba00 tx=0x7f042802c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:05.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.648+0000 7f0437400640 1 -- 172.21.15.69:0/3489993547 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0430059510 con 0x7f0424004800 2024-04-24T08:22:05.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.849+0000 7f0445600640 1 -- 172.21.15.69:0/3489993547 --> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f04400d1120 con 0x7f041c06c920 2024-04-24T08:22:05.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.852+0000 7f0437400640 1 -- 172.21.15.69:0/3489993547 <== mgr.4109 v2:172.21.15.121:6832/2070338101 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13379 (secure 0 0 0) 0x7f04400d1120 con 0x7f041c06c920 2024-04-24T08:22:05.854 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:22:05.854 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.854+0000 7f0445600640 1 -- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06c920 msgr2=0x7f041c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.854+0000 7f0445600640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06c920 0x7f041c06ede0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f042800ba00 tx=0x7f042802c000 comp rx=0 tx=0).stop 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.854+0000 7f0445600640 1 -- 172.21.15.69:0/3489993547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 msgr2=0x7f04401415f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.854+0000 7f0445600640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f04401415f0 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f043000c790 tx=0x7f043000cc50 comp rx=0 tx=0).stop 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.855+0000 7f0445600640 1 -- 172.21.15.69:0/3489993547 shutdown_connections 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.855+0000 7f0445600640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f041c06c920 0x7f041c06ede0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.855+0000 7f0445600640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f044013e0e0 0x7f044013adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.855+0000 7f0445600640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f044013d150 0x7f044013b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.855+0000 7f0445600640 1 --2- 172.21.15.69:0/3489993547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0424004800 0x7f04401415f0 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.855+0000 7f0445600640 1 -- 172.21.15.69:0/3489993547 >> 172.21.15.69:0/3489993547 conn(0x7f0440067d90 msgr2=0x7f04400d06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:05.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.855+0000 7f0445600640 1 -- 172.21.15.69:0/3489993547 shutdown_connections 2024-04-24T08:22:05.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:05.855+0000 7f0445600640 1 -- 172.21.15.69:0/3489993547 wait complete. 2024-04-24T08:22:05.875 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2024-04-24T08:22:04.971709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":261928,"kb_used_data":12488,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754712792,"statfs":{"total":773094113280,"available":772825899008,"internally_reserved":0,"allocated":12787712,"data_stored":6235440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003375"},"pg_stats":[{"pgid":"1.0","version":"116'119","reported_seq":361,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-04-24T08:21:45.183251+0000","last_change":"2024-04-24T08:21:45.183326+0000","last_active":"2024-04-24T08:21:45.183251+0000","last_peered":"2024-04-24T08:21:45.183251+0000","last_clean":"2024-04-24T08:21:45.183251+0000","last_became_active":"2024-04-24T08:08:10.186782+0000","last_became_peered":"2024-04-24T08:08:10.186782+0000","last_unstale":"2024-04-24T08:21:45.183251+0000","last_undegraded":"2024-04-24T08:21:45.183251+0000","last_fullsized":"2024-04-24T08:21:45.183251+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'119","last_scrub_stamp":"2024-04-24T08:21:45.183036+0000","last_deep_scrub":"116'119","last_deep_scrub_stamp":"2024-04-24T08:21:45.183036+0000","last_clean_scrub_stamp":"2024-04-24T08:21:45.183036+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:21:54.939881+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575020,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33172,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338668,"statfs":{"total":96636764160,"available":96602796032,"internally_reserved":0,"allocated":2043904,"data_stored":1222206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834575020,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32596,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834575018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32600,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339240,"statfs":{"total":96636764160,"available":96603381760,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575017,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32600,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339240,"statfs":{"total":96636764160,"available":96603381760,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32596,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32596,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32596,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339244,"statfs":{"total":96636764160,"available":96603385856,"internally_reserved":0,"allocated":1449984,"data_stored":631838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834575018,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33172,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338668,"statfs":{"total":96636764160,"available":96602796032,"internally_reserved":0,"allocated":2043904,"data_stored":1222206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:22:05.876 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-24T08:22:06.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.062+0000 7f90fea00640 1 Processor -- start 2024-04-24T08:22:06.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.062+0000 7f90fea00640 1 -- start start 2024-04-24T08:22:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.063+0000 7f90fea00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f815e980 0x7f90f8162f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fea00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f90f806cfb0 0x7f90f806d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fea00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90f815ed60 con 0x7f90f80d0da0 2024-04-24T08:22:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fea00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7f90f80d1580 con 0x7f90f806cfb0 2024-04-24T08:22:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fea00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f90f80d1860 con 0x7f90f815e980 2024-04-24T08:22:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fe000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f90f806cfb0 0x7f90f806d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fe000640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7f90f806cfb0 0x7f90f806d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47110/0 (socket says 172.21.15.69:47110) 2024-04-24T08:22:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fcc00640 1 --1- >> v1:172.21.15.69:6789/0 conn(0x7f90f80d0da0 0x7f90f80cfc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.69:6789/0 says I am v1:172.21.15.69:49886/0 (socket says 172.21.15.69:49886) 2024-04-24T08:22:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fe000640 1 -- 172.21.15.69:0/4050997961 learned_addr learned my addr 172.21.15.69:0/4050997961 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:22:06.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fe000640 1 -- 172.21.15.69:0/4050997961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f815e980 msgr2=0x7f90f8162f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:06.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fe000640 1 --2- 172.21.15.69:0/4050997961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f815e980 0x7f90f8162f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fe000640 1 -- 172.21.15.69:0/4050997961 >> v1:172.21.15.69:6789/0 conn(0x7f90f80d0da0 legacy=0x7f90f80cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90fe000640 1 -- 172.21.15.69:0/4050997961 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f80cf450 con 0x7f90f806cfb0 2024-04-24T08:22:06.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.064+0000 7f90f7e00640 1 -- 172.21.15.69:0/4050997961 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 602411433 0 0) 0x7f90f815ed60 con 0x7f90f80d0da0 2024-04-24T08:22:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.065+0000 7f90fe000640 1 --2- 172.21.15.69:0/4050997961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f806cfb0 0x7f90f806d440 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009e50 tx=0x7f90f002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13cff1d7231a09d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90f7e00640 1 -- 172.21.15.69:0/4050997961 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f0036070 con 0x7f90f806cfb0 2024-04-24T08:22:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90f7e00640 1 -- 172.21.15.69:0/4050997961 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90f0031600 con 0x7f90f806cfb0 2024-04-24T08:22:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90f7e00640 1 -- 172.21.15.69:0/4050997961 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f00319d0 con 0x7f90f806cfb0 2024-04-24T08:22:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90fea00640 1 -- 172.21.15.69:0/4050997961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f806cfb0 msgr2=0x7f90f806d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90fea00640 1 --2- 172.21.15.69:0/4050997961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f806cfb0 0x7f90f806d440 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f90f0009e50 tx=0x7f90f002a7a0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90fea00640 1 -- 172.21.15.69:0/4050997961 shutdown_connections 2024-04-24T08:22:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90fea00640 1 --2- 172.21.15.69:0/4050997961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f806cfb0 0x7f90f806d440 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90fea00640 1 --2- 172.21.15.69:0/4050997961 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f815e980 0x7f90f8162f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90fea00640 1 -- 172.21.15.69:0/4050997961 >> 172.21.15.69:0/4050997961 conn(0x7f90f809d920 msgr2=0x7f90f808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:06.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90fea00640 1 -- 172.21.15.69:0/4050997961 shutdown_connections 2024-04-24T08:22:06.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.066+0000 7f90fea00640 1 -- 172.21.15.69:0/4050997961 wait complete. 2024-04-24T08:22:06.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.068+0000 7f90fea00640 1 Processor -- start 2024-04-24T08:22:06.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.068+0000 7f90fea00640 1 -- start start 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.069+0000 7f90fea00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f806cfb0 0x7f90f80cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.069+0000 7f90fea00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f80d0da0 0x7f90f81541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.069+0000 7f90fea00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f815e980 0x7f90f80cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.069+0000 7f90fea00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f90f80d11e0 con 0x7f90f80d0da0 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.069+0000 7f90fea00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7f90f80d0360 con 0x7f90f806cfb0 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.069+0000 7f90fea00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7f90f80d1360 con 0x7f90f815e980 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.069+0000 7f90fcc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f80d0da0 0x7f90f81541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.069+0000 7f90fcc00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f80d0da0 0x7f90f81541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47124/0 (socket says 172.21.15.69:47124) 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.069+0000 7f90fcc00640 1 -- 172.21.15.69:0/2670590035 learned_addr learned my addr 172.21.15.69:0/2670590035 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90fcc00640 1 -- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f815e980 msgr2=0x7f90f80cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90fcc00640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f815e980 0x7f90f80cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90fcc00640 1 -- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f806cfb0 msgr2=0x7f90f80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90fcc00640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f806cfb0 0x7f90f80cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90fcc00640 1 -- 172.21.15.69:0/2670590035 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f80ccb00 con 0x7f90f80d0da0 2024-04-24T08:22:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90fcc00640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f80d0da0 0x7f90f81541e0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f90e400c500 tx=0x7f90e400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90f5600640 1 -- 172.21.15.69:0/2670590035 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4018070 con 0x7f90f80d0da0 2024-04-24T08:22:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90f5600640 1 -- 172.21.15.69:0/2670590035 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90e4007580 con 0x7f90f80d0da0 2024-04-24T08:22:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90f5600640 1 -- 172.21.15.69:0/2670590035 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4004a20 con 0x7f90f80d0da0 2024-04-24T08:22:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90fea00640 1 -- 172.21.15.69:0/2670590035 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f80ccde0 con 0x7f90f80d0da0 2024-04-24T08:22:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.070+0000 7f90fea00640 1 -- 172.21.15.69:0/2670590035 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f80cd3a0 con 0x7f90f80d0da0 2024-04-24T08:22:06.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.071+0000 7f90fea00640 1 -- 172.21.15.69:0/2670590035 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90f8075d50 con 0x7f90f80d0da0 2024-04-24T08:22:06.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.072+0000 7f90f5600640 1 -- 172.21.15.69:0/2670590035 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f90e4004bc0 con 0x7f90f80d0da0 2024-04-24T08:22:06.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.073+0000 7f90f5600640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f90d806c8f0 0x7f90d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.073+0000 7f90f5600640 1 -- 172.21.15.69:0/2670590035 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7f90e408ab60 con 0x7f90f80d0da0 2024-04-24T08:22:06.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.074+0000 7f90fd600640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f90d806c8f0 0x7f90d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.075+0000 7f90fd600640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f90d806c8f0 0x7f90d806edb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009670 tx=0x7f90ec006bf0 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:06.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.078+0000 7f90f5600640 1 -- 172.21.15.69:0/2670590035 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f90e40591d0 con 0x7f90f80d0da0 2024-04-24T08:22:06.276 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.275+0000 7f90fea00640 1 -- 172.21.15.69:0/2670590035 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f90f80d1e40 con 0x7f90f80d0da0 2024-04-24T08:22:06.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.282+0000 7f90f5600640 1 -- 172.21.15.69:0/2670590035 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v24)=0 v24) ==== 123+0+0 (secure 0 0 0) 0x7f90e4058ad0 con 0x7f90f80d0da0 2024-04-24T08:22:06.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.284+0000 7f90f5600640 1 -- 172.21.15.69:0/2670590035 <== mon.0 v2:172.21.15.69:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f90e4090080 con 0x7f90f80d0da0 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.286+0000 7f90fea00640 1 -- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f90d806c8f0 msgr2=0x7f90d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.286+0000 7f90fea00640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f90d806c8f0 0x7f90d806edb0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f90ec009670 tx=0x7f90ec006bf0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.286+0000 7f90fea00640 1 -- 172.21.15.69:0/2670590035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f80d0da0 msgr2=0x7f90f81541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.287+0000 7f90fea00640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f80d0da0 0x7f90f81541e0 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7f90e400c500 tx=0x7f90e400c9c0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.287+0000 7f90fea00640 1 -- 172.21.15.69:0/2670590035 shutdown_connections 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.287+0000 7f90fea00640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7f90d806c8f0 0x7f90d806edb0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.287+0000 7f90fea00640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f90f80d0da0 0x7f90f81541e0 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.287+0000 7f90fea00640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f90f815e980 0x7f90f80cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.287+0000 7f90fea00640 1 --2- 172.21.15.69:0/2670590035 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90f806cfb0 0x7f90f80cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.287+0000 7f90fea00640 1 -- 172.21.15.69:0/2670590035 >> 172.21.15.69:0/2670590035 conn(0x7f90f809d920 msgr2=0x7f90f80cfc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:06.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.287+0000 7f90fea00640 1 -- 172.21.15.69:0/2670590035 shutdown_connections 2024-04-24T08:22:06.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.287+0000 7f90fea00640 1 -- 172.21.15.69:0/2670590035 wait complete. 2024-04-24T08:22:06.306 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:22:06.306 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:22:06.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.499+0000 7fb873e00640 1 Processor -- start 2024-04-24T08:22:06.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.499+0000 7fb873e00640 1 -- start start 2024-04-24T08:22:06.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.500+0000 7fb873e00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb874156c60 0x7fb87415cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.500+0000 7fb873e00640 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fb8740b30f0 0x7fb87415d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.500+0000 7fb873e00640 1 -- --> v1:172.21.15.69:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb87415d660 con 0x7fb8740ad0d0 2024-04-24T08:22:06.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.500+0000 7fb873e00640 1 -- --> v2:172.21.15.69:3300/0 -- mon_getmap magic: 0 -- 0x7fb87406dc30 con 0x7fb874156c60 2024-04-24T08:22:06.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.500+0000 7fb873e00640 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 -- 0x7fb87406df10 con 0x7fb8740b30f0 2024-04-24T08:22:06.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.501+0000 7fb872a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb874156c60 0x7fb87415cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.501+0000 7fb872a00640 1 --2- >> v2:172.21.15.69:3300/0 conn(0x7fb874156c60 0x7fb87415cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47128/0 (socket says 172.21.15.69:47128) 2024-04-24T08:22:06.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.501+0000 7fb872a00640 1 -- 172.21.15.69:0/1514340253 learned_addr learned my addr 172.21.15.69:0/1514340253 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:22:06.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.501+0000 7fb872000640 1 --2- 172.21.15.69:0/1514340253 >> v2:172.21.15.121:3300/0 conn(0x7fb8740b30f0 0x7fb87415d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.501+0000 7fb872a00640 1 -- 172.21.15.69:0/1514340253 >> v2:172.21.15.121:3300/0 conn(0x7fb8740b30f0 msgr2=0x7fb87415d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:22:06.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.501+0000 7fb872a00640 1 --2- 172.21.15.69:0/1514340253 >> v2:172.21.15.121:3300/0 conn(0x7fb8740b30f0 0x7fb87415d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.501+0000 7fb872a00640 1 -- 172.21.15.69:0/1514340253 >> v1:172.21.15.69:6789/0 conn(0x7fb8740ad0d0 legacy=0x7fb8741600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.501+0000 7fb872a00640 1 -- 172.21.15.69:0/1514340253 --> v2:172.21.15.69:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87406e1f0 con 0x7fb874156c60 2024-04-24T08:22:06.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.501+0000 7fb871600640 1 -- 172.21.15.69:0/1514340253 <== mon.0 v1:172.21.15.69:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1298348307 0 0) 0x7fb87415d660 con 0x7fb8740ad0d0 2024-04-24T08:22:06.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.501+0000 7fb872000640 1 --2- 172.21.15.69:0/1514340253 >> v2:172.21.15.121:3300/0 conn(0x7fb8740b30f0 0x7fb87415d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:22:06.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.502+0000 7fb872a00640 1 --2- 172.21.15.69:0/1514340253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 0x7fb87415cac0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fb87406d900 tx=0x7fb86402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c79c3f9aa9fea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:06.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb871600640 1 -- 172.21.15.69:0/1514340253 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb864037070 con 0x7fb874156c60 2024-04-24T08:22:06.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb871600640 1 -- 172.21.15.69:0/1514340253 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb864032440 con 0x7fb874156c60 2024-04-24T08:22:06.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb871600640 1 -- 172.21.15.69:0/1514340253 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb864032830 con 0x7fb874156c60 2024-04-24T08:22:06.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb873e00640 1 -- 172.21.15.69:0/1514340253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 msgr2=0x7fb87415cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb873e00640 1 --2- 172.21.15.69:0/1514340253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 0x7fb87415cac0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7fb87406d900 tx=0x7fb86402a270 comp rx=0 tx=0).stop 2024-04-24T08:22:06.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb873e00640 1 -- 172.21.15.69:0/1514340253 shutdown_connections 2024-04-24T08:22:06.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb873e00640 1 --2- 172.21.15.69:0/1514340253 >> v2:172.21.15.121:3300/0 conn(0x7fb8740b30f0 0x7fb87415d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb873e00640 1 --2- 172.21.15.69:0/1514340253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 0x7fb87415cac0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb873e00640 1 -- 172.21.15.69:0/1514340253 >> 172.21.15.69:0/1514340253 conn(0x7fb8740d0160 msgr2=0x7fb8740d0540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:06.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb873e00640 1 -- 172.21.15.69:0/1514340253 shutdown_connections 2024-04-24T08:22:06.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.503+0000 7fb873e00640 1 -- 172.21.15.69:0/1514340253 wait complete. 2024-04-24T08:22:06.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.505+0000 7fb873e00640 1 Processor -- start 2024-04-24T08:22:06.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.505+0000 7fb873e00640 1 -- start start 2024-04-24T08:22:06.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.506+0000 7fb873e00640 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8740ad0d0 0x7fb874145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.506+0000 7fb873e00640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8740b30f0 0x7fb874159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.506+0000 7fb873e00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 0x7fb8740994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.506+0000 7fb873e00640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb8740ad540 con 0x7fb874156c60 2024-04-24T08:22:06.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.506+0000 7fb873e00640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 -- 0x7fb87415d940 con 0x7fb8740b30f0 2024-04-24T08:22:06.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.506+0000 7fb873e00640 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 -- 0x7fb8740ad6c0 con 0x7fb8740ad0d0 2024-04-24T08:22:06.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.506+0000 7fb873400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 0x7fb8740994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.506+0000 7fb873400640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 0x7fb8740994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47134/0 (socket says 172.21.15.69:47134) 2024-04-24T08:22:06.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.506+0000 7fb873400640 1 -- 172.21.15.69:0/2678644834 learned_addr learned my addr 172.21.15.69:0/2678644834 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-04-24T08:22:06.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb873400640 1 -- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8740ad0d0 msgr2=0x7fb874145910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb873400640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8740ad0d0 0x7fb874145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb873400640 1 -- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8740b30f0 msgr2=0x7fb874159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb873400640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8740b30f0 0x7fb874159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb873400640 1 -- 172.21.15.69:0/2678644834 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb874155160 con 0x7fb874156c60 2024-04-24T08:22:06.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb873400640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 0x7fb8740994f0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00c790 tx=0x7fb86c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:06.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb86aa00640 1 -- 172.21.15.69:0/2678644834 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c003da0 con 0x7fb874156c60 2024-04-24T08:22:06.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb86aa00640 1 -- 172.21.15.69:0/2678644834 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb86c005070 con 0x7fb874156c60 2024-04-24T08:22:06.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb873e00640 1 -- 172.21.15.69:0/2678644834 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb874155440 con 0x7fb874156c60 2024-04-24T08:22:06.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb86aa00640 1 -- 172.21.15.69:0/2678644834 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c005480 con 0x7fb874156c60 2024-04-24T08:22:06.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.507+0000 7fb873e00640 1 -- 172.21.15.69:0/2678644834 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb874155a00 con 0x7fb874156c60 2024-04-24T08:22:06.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.508+0000 7fb873e00640 1 -- 172.21.15.69:0/2678644834 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb874142f40 con 0x7fb874156c60 2024-04-24T08:22:06.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.509+0000 7fb86aa00640 1 -- 172.21.15.69:0/2678644834 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb86c005620 con 0x7fb874156c60 2024-04-24T08:22:06.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.510+0000 7fb86aa00640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb85406c920 0x7fb85406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.510+0000 7fb86aa00640 1 -- 172.21.15.69:0/2678644834 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(116..116 src has 1..116) ==== 5115+0+0 (secure 0 0 0) 0x7fb86c08b9a0 con 0x7fb874156c60 2024-04-24T08:22:06.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.511+0000 7fb872a00640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb85406c920 0x7fb85406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.512+0000 7fb872a00640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb85406c920 0x7fb85406ede0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb864034000 tx=0x7fb86402c000 comp rx=0 tx=0).ready entity=mgr.4109 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:06.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.514+0000 7fb86aa00640 1 -- 172.21.15.69:0/2678644834 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb86c059ef0 con 0x7fb874156c60 2024-04-24T08:22:06.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.748+0000 7fb873e00640 1 -- 172.21.15.69:0/2678644834 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb874075d50 con 0x7fb874156c60 2024-04-24T08:22:06.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.749+0000 7fb86aa00640 1 -- 172.21.15.69:0/2678644834 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) ==== 95+0+3876 (secure 0 0 0) 0x7fb86c0597f0 con 0x7fb874156c60 2024-04-24T08:22:06.751 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-04-24T08:22:06.751 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":114,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5750,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6833/361644980","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":361644980},{"type":"v1","addr":"172.21.15.69:6833","nonce":361644980}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":94},{"gid":6325,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/3320453946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6835","nonce":3320453946},{"type":"v1","addr":"172.21.15.121:6837","nonce":3320453946}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":95},{"gid":6478,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6836/3201516349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3201516349},{"type":"v1","addr":"172.21.15.121:6836","nonce":3201516349}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":101},{"gid":10053,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3821350354","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3821350354},{"type":"v1","addr":"172.21.15.69:6835","nonce":3821350354}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":114}],"filesystems":[]} 2024-04-24T08:22:06.751 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 114 2024-04-24T08:22:06.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 -- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb85406c920 msgr2=0x7fb85406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb85406c920 0x7fb85406ede0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb864034000 tx=0x7fb86402c000 comp rx=0 tx=0).stop 2024-04-24T08:22:06.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 -- 172.21.15.69:0/2678644834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 msgr2=0x7fb8740994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 0x7fb8740994f0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00c790 tx=0x7fb86c00cc50 comp rx=0 tx=0).stop 2024-04-24T08:22:06.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 -- 172.21.15.69:0/2678644834 shutdown_connections 2024-04-24T08:22:06.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:6832/2070338101,v1:172.21.15.121:6833/2070338101] conn(0x7fb85406c920 0x7fb85406ede0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb874156c60 0x7fb8740994f0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8740b30f0 0x7fb874159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 --2- 172.21.15.69:0/2678644834 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fb8740ad0d0 0x7fb874145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 -- 172.21.15.69:0/2678644834 >> 172.21.15.69:0/2678644834 conn(0x7fb8740d0160 msgr2=0x7fb8740afd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:06.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 -- 172.21.15.69:0/2678644834 shutdown_connections 2024-04-24T08:22:06.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-04-24T08:22:06.752+0000 7fb873e00640 1 -- 172.21.15.69:0/2678644834 wait complete. 2024-04-24T08:22:06.773 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-24T08:22:06.773 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T08:22:06.773 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T08:22:06.773 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:06.791 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T08:22:06.791 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T08:22:06.791 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:06.857 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T08:22:06.858 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T08:22:06.858 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:06.892 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T08:22:06.892 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T08:22:06.892 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:06.958 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T08:22:06.959 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T08:22:06.959 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-24T08:22:06.959 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:07.005 INFO:tasks.ceph.osd.0:Stopped 2024-04-24T08:22:07.006 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-24T08:22:07.006 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:07.078 INFO:tasks.ceph.osd.1:Stopped 2024-04-24T08:22:07.078 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-24T08:22:07.078 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:07.128 INFO:tasks.ceph.osd.2:Stopped 2024-04-24T08:22:07.128 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-24T08:22:07.128 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:07.201 INFO:tasks.ceph.osd.3:Stopped 2024-04-24T08:22:07.201 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-24T08:22:07.201 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:07.283 INFO:tasks.ceph.osd.4:Stopped 2024-04-24T08:22:07.283 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-24T08:22:07.283 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:07.338 INFO:tasks.ceph.osd.5:Stopped 2024-04-24T08:22:07.339 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-24T08:22:07.339 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:07.411 INFO:tasks.ceph.osd.6:Stopped 2024-04-24T08:22:07.411 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-24T08:22:07.411 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:07.478 INFO:tasks.ceph.osd.7:Stopped 2024-04-24T08:22:07.478 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T08:22:07.478 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-24T08:22:07.478 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:07.575 INFO:tasks.ceph.mgr.y:Stopped 2024-04-24T08:22:07.576 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-24T08:22:07.576 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:10.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-04-24T08:22:13.677 INFO:tasks.ceph.mgr.x:Stopped 2024-04-24T08:22:13.677 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T08:22:13.677 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-24T08:22:13.677 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:13.713 INFO:tasks.ceph.mon.a:Stopped 2024-04-24T08:22:13.714 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-24T08:22:13.714 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:13.743 INFO:tasks.ceph.mon.b:Stopped 2024-04-24T08:22:13.743 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-24T08:22:13.744 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T08:22:13.817 INFO:tasks.ceph.mon.c:Stopped 2024-04-24T08:22:13.817 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T08:22:13.817 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-04-24T08:22:13.923 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:13:45.304707+0000 mds.a (mds.0) 1 : cluster [WRN] Corrupt dentry 'file_to_be_damaged' in dir frag 0x10000000000: Invalid tag char 'd' pos 0: Malformed input [buffer:3](/subdir) 2024-04-24T08:22:13.924 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-04-24T08:22:13.924 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-04-24T08:22:14.008 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | head -n 1 2024-04-24T08:22:14.086 INFO:teuthology.orchestra.run.smithi069.stdout:2024-04-24T08:15:53.345601+0000 mds.a (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-04-24T08:22:14.087 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:22:14.087 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-24T08:22:14.356 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:22:14.356 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-24T08:22:14.496 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:22:14.496 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-24T08:22:14.618 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:22:14.619 DEBUG:teuthology.orchestra.run.smithi069:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-24T08:22:14.775 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:22:14.775 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-24T08:22:15.054 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:22:15.054 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-24T08:22:15.177 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:22:15.177 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-24T08:22:15.303 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:22:15.303 DEBUG:teuthology.orchestra.run.smithi121:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-24T08:22:15.454 INFO:tasks.ceph:Archiving mon data... 2024-04-24T08:22:15.455 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210/data/mon.a.tgz 2024-04-24T08:22:15.455 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-04-24T08:22:15.512 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.BXSHU2FJPT 2024-04-24T08:22:15.512 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.BXSHU2FJPT 2024-04-24T08:22:15.687 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.BXSHU2FJPT 2024-04-24T08:22:15.823 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.BXSHU2FJPT is 398KB 2024-04-24T08:22:15.930 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.BXSHU2FJPT 2024-04-24T08:22:15.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-04-24T08:22:15.987 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210/data/mon.b.tgz 2024-04-24T08:22:15.987 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-04-24T08:22:16.042 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.4mI7t8n7LG 2024-04-24T08:22:16.043 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.4mI7t8n7LG 2024-04-24T08:22:16.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.4mI7t8n7LG 2024-04-24T08:22:16.357 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.4mI7t8n7LG is 398KB 2024-04-24T08:22:16.456 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.4mI7t8n7LG 2024-04-24T08:22:16.513 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210/data/mon.c.tgz 2024-04-24T08:22:16.513 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-04-24T08:22:16.564 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.FIMZdlcH3T 2024-04-24T08:22:16.564 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.FIMZdlcH3T 2024-04-24T08:22:16.730 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.FIMZdlcH3T 2024-04-24T08:22:16.865 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.FIMZdlcH3T is 398KB 2024-04-24T08:22:16.933 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.FIMZdlcH3T 2024-04-24T08:22:16.988 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T08:22:16.988 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T08:22:16.996 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T08:22:17.098 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:22:17.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:22:17.132 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T08:22:17.183 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T08:22:17.183 INFO:tasks.ceph:Archiving crash dumps... 2024-04-24T08:22:17.185 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210/remote/smithi069/crash 2024-04-24T08:22:17.186 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T08:22:17.258 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210/remote/smithi121/crash 2024-04-24T08:22:17.260 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T08:22:17.342 INFO:tasks.ceph:Shutting down logrotate 2024-04-24T08:22:17.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:22:17.350 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T08:22:17.460 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-24T08:22:17.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-24T08:22:17.636 INFO:tasks.ceph:Compressing logs... 2024-04-24T08:22:17.637 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T08:22:17.644 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T08:22:17.711 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68904.log 2024-04-24T08:22:17.711 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67956.log 2024-04-24T08:22:17.713 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80412.log 2024-04-24T08:22:17.713 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68904.log.gz 2024-04-24T08:22:17.714 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78718.log 2024-04-24T08:22:17.715 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80412.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67956.log.gz 2024-04-24T08:22:17.715 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70185.log 2024-04-24T08:22:17.715 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80412.log.gz 2024-04-24T08:22:17.716 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50205.log 2024-04-24T08:22:17.716 INFO:teuthology.orchestra.run.smithi069.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.78718.log.gz 2024-04-24T08:22:17.716 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80580.log 2024-04-24T08:22:17.717 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.70185.log.gz 2024-04-24T08:22:17.717 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44026.log 2024-04-24T08:22:17.718 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50205.log.gz 2024-04-24T08:22:17.718 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80580.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.80580.log.gz 2024-04-24T08:22:17.718 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51254.log 2024-04-24T08:22:17.719 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66337.log 2024-04-24T08:22:17.720 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44026.log.gz 2024-04-24T08:22:17.720 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52616.log 2024-04-24T08:22:17.720 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51254.log.gz 2024-04-24T08:22:17.721 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.66337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74332.log 2024-04-24T08:22:17.721 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66337.log.gz 2024-04-24T08:22:17.722 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53772.log 2024-04-24T08:22:17.723 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52616.log.gz 2024-04-24T08:22:17.723 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79398.log 2024-04-24T08:22:17.723 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74332.log.gz 2024-04-24T08:22:17.724 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58663.log 2024-04-24T08:22:17.724 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53772.log.gz 2024-04-24T08:22:17.725 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T08:22:17.725 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.79398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54716.log 2024-04-24T08:22:17.726 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79398.log.gz 2024-04-24T08:22:17.726 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49440.log 2024-04-24T08:22:17.726 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58663.log.gz 2024-04-24T08:22:17.726 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.39806.log 2024-04-24T08:22:17.728 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T08:22:17.728 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48875.log 2024-04-24T08:22:17.728 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54716.log.gz 2024-04-24T08:22:17.728 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48833.log 2024-04-24T08:22:17.730 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49440.log.gz 2024-04-24T08:22:17.730 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65169.log 2024-04-24T08:22:17.730 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48875.log.gz 2024-04-24T08:22:17.730 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43165.log 2024-04-24T08:22:17.730 INFO:teuthology.orchestra.run.smithi069.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48833.log.gz 2024-04-24T08:22:17.730 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.3.39806.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-24T08:22:17.730 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph.audit.log: 91.8% -- replaced with /var/log/ceph/ceph-client.3.39806.log.gz 2024-04-24T08:22:17.730 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-24T08:22:17.732 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-24T08:22:17.732 INFO:teuthology.orchestra.run.smithi121.stderr: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T08:22:17.732 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51986.log 2024-04-24T08:22:17.733 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40268.log 2024-04-24T08:22:17.733 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.65169.log.gz 2024-04-24T08:22:17.733 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43165.log.gz 2024-04-24T08:22:17.733 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51986.log: gzip 87.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.51986.log.gz 2024-04-24T08:22:17.733 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39124.log 2024-04-24T08:22:17.735 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33443.log 2024-04-24T08:22:17.735 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.5.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T08:22:17.735 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-24T08:22:17.735 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33283.log/var/log/ceph/ceph-client.admin.33443.log: 2024-04-24T08:22:17.736 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57041.log 2024-04-24T08:22:17.736 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40268.log.gz 2024-04-24T08:22:17.736 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61342.log 2024-04-24T08:22:17.736 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39124.log.gz 2024-04-24T08:22:17.736 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43207.log 2024-04-24T08:22:17.736 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57041.log.gz 2024-04-24T08:22:17.738 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.4.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33443.log.gz 2024-04-24T08:22:17.738 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33323.log 2024-04-24T08:22:17.738 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.33283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33192.log 2024-04-24T08:22:17.739 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61342.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61342.log.gz 2024-04-24T08:22:17.739 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82087.log 2024-04-24T08:22:17.739 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73438.log 2024-04-24T08:22:17.739 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43207.log.gz 2024-04-24T08:22:17.739 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.82087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46411.log 2024-04-24T08:22:17.739 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.82087.log.gz 2024-04-24T08:22:17.739 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70101.log 2024-04-24T08:22:17.739 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73438.log.gz 2024-04-24T08:22:17.742 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59560.log 2024-04-24T08:22:17.742 INFO:teuthology.orchestra.run.smithi069.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.46411.log.gz 2024-04-24T08:22:17.742 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69383.log 2024-04-24T08:22:17.742 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70101.log.gz 2024-04-24T08:22:17.742 INFO:teuthology.orchestra.run.smithi121.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33283.log.gz 2024-04-24T08:22:17.742 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.33323.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-24T08:22:17.742 INFO:teuthology.orchestra.run.smithi121.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33323.log.gz 2024-04-24T08:22:17.742 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.33192.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-24T08:22:17.743 INFO:teuthology.orchestra.run.smithi121.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33192.log.gz 2024-04-24T08:22:17.743 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-24T08:22:17.744 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59560.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59560.log.gz 2024-04-24T08:22:17.744 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77216.log 2024-04-24T08:22:17.744 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56047.log 2024-04-24T08:22:17.744 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69383.log.gz 2024-04-24T08:22:17.745 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33403.log 2024-04-24T08:22:17.745 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.77216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39292.log 2024-04-24T08:22:17.746 INFO:teuthology.orchestra.run.smithi069.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.77216.log.gz 2024-04-24T08:22:17.746 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72262.log 2024-04-24T08:22:17.746 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56047.log.gz 2024-04-24T08:22:17.747 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.33403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33363.log 2024-04-24T08:22:17.748 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69341.log 2024-04-24T08:22:17.748 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39292.log.gz 2024-04-24T08:22:17.748 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39936.log 2024-04-24T08:22:17.748 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.72262.log.gz 2024-04-24T08:22:17.749 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69341.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69341.log.gz 2024-04-24T08:22:17.749 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75575.log 2024-04-24T08:22:17.750 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-client.admin.33363.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.33403.log.gz 2024-04-24T08:22:17.750 INFO:teuthology.orchestra.run.smithi121.stderr: -5 --verbose -- /var/log/ceph/ceph-client.1.39283.log 2024-04-24T08:22:17.751 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39936.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39936.log.gz 2024-04-24T08:22:17.751 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41098.log 2024-04-24T08:22:17.752 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33242.log 2024-04-24T08:22:17.753 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75575.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75575.log.gz 2024-04-24T08:22:17.753 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81944.log 2024-04-24T08:22:17.753 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49230.log 2024-04-24T08:22:17.754 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81944.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.41098.log.gz 2024-04-24T08:22:17.754 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81944.log.gz 2024-04-24T08:22:17.754 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73692.log 2024-04-24T08:22:17.755 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-04-24T08:22:17.755 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49230.log.gz 2024-04-24T08:22:17.756 INFO:teuthology.orchestra.run.smithi121.stderr: 88.1%/var/log/ceph/ceph-client.1.39283.log: /var/log/ceph/ceph-client.admin.33242.log: -- replaced with /var/log/ceph/ceph-client.admin.33363.log.gz 2024-04-24T08:22:17.757 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42257.log 2024-04-24T08:22:17.757 INFO:teuthology.orchestra.run.smithi069.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.73692.log.gz 2024-04-24T08:22:17.757 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39934.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-04-24T08:22:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48088.log 2024-04-24T08:22:17.758 INFO:teuthology.orchestra.run.smithi121.stderr: 90.7%gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-24T08:22:17.759 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59475.log 2024-04-24T08:22:17.759 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42257.log.gz 2024-04-24T08:22:17.760 INFO:teuthology.orchestra.run.smithi121.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33242.log.gz 2024-04-24T08:22:17.761 INFO:teuthology.orchestra.run.smithi121.stderr: -- replaced with /var/log/ceph/ceph-client.1.39283.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-24T08:22:17.761 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-04-24T08:22:17.761 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43123.log 2024-04-24T08:22:17.762 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48088.log.gz 2024-04-24T08:22:17.762 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59475.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59475.log.gz 2024-04-24T08:22:17.762 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46808.log 2024-04-24T08:22:17.763 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33152.log 2024-04-24T08:22:17.763 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38771.log 2024-04-24T08:22:17.763 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43123.log.gz 2024-04-24T08:22:17.764 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46808.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46808.log.gz 2024-04-24T08:22:17.764 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76614.log 2024-04-24T08:22:17.765 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65785.log 2024-04-24T08:22:17.765 INFO:teuthology.orchestra.run.smithi069.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38771.log.gz 2024-04-24T08:22:17.766 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-client.admin.33152.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33152.log.gz 2024-04-24T08:22:17.766 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.76614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73144.log 2024-04-24T08:22:17.766 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76614.log.gz 2024-04-24T08:22:17.767 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65785.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65785.log.gz 2024-04-24T08:22:17.767 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57863.log 2024-04-24T08:22:17.768 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42745.log 2024-04-24T08:22:17.768 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73144.log.gz 2024-04-24T08:22:17.769 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43470.log 2024-04-24T08:22:17.769 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57863.log.gz 2024-04-24T08:22:17.773 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42745.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42745.log.gz 2024-04-24T08:22:17.773 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78176.log 2024-04-24T08:22:17.775 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80809.log 2024-04-24T08:22:17.775 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43470.log.gz 2024-04-24T08:22:17.775 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49043.log 2024-04-24T08:22:17.775 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78176.log.gz 2024-04-24T08:22:17.775 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69169.log 2024-04-24T08:22:17.775 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80809.log.gz 2024-04-24T08:22:17.775 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49043.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49043.log.gz 2024-04-24T08:22:17.775 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.42135.log 2024-04-24T08:22:17.776 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47032.log 2024-04-24T08:22:17.776 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69169.log.gz 2024-04-24T08:22:17.777 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.2.42135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62522.log 2024-04-24T08:22:17.778 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.47032.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47032.log.gz 2024-04-24T08:22:17.778 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56353.log 2024-04-24T08:22:17.778 INFO:teuthology.orchestra.run.smithi069.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.2.42135.log.gz 2024-04-24T08:22:17.779 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70453.log 2024-04-24T08:22:17.779 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62522.log.gz 2024-04-24T08:22:17.780 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45640.log 2024-04-24T08:22:17.781 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56353.log.gz 2024-04-24T08:22:17.781 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70453.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70453.log.gz 2024-04-24T08:22:17.781 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81566.log 2024-04-24T08:22:17.783 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68616.log 2024-04-24T08:22:17.783 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45640.log.gz 2024-04-24T08:22:17.784 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43900.log 2024-04-24T08:22:17.784 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81566.log.gz 2024-04-24T08:22:17.785 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65617.log 2024-04-24T08:22:17.785 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68616.log.gz 2024-04-24T08:22:17.786 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50121.log 2024-04-24T08:22:17.786 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43900.log.gz 2024-04-24T08:22:17.787 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48355.log 2024-04-24T08:22:17.787 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65617.log.gz 2024-04-24T08:22:17.788 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38813.log 2024-04-24T08:22:17.788 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50121.log.gz 2024-04-24T08:22:17.789 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68696.log 2024-04-24T08:22:17.790 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48355.log.gz 2024-04-24T08:22:17.790 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68082.log 2024-04-24T08:22:17.791 INFO:teuthology.orchestra.run.smithi069.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38813.log.gz 2024-04-24T08:22:17.791 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68696.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68696.log.gz 2024-04-24T08:22:17.791 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48490.log 2024-04-24T08:22:17.793 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45346.log 2024-04-24T08:22:17.793 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68082.log.gz 2024-04-24T08:22:17.794 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61823.log 2024-04-24T08:22:17.794 INFO:teuthology.orchestra.run.smithi069.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48490.log.gz 2024-04-24T08:22:17.795 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45346.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45346.log.gz 2024-04-24T08:22:17.795 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46325.log 2024-04-24T08:22:17.796 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76924.log 2024-04-24T08:22:17.796 INFO:teuthology.orchestra.run.smithi069.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.61823.log.gz 2024-04-24T08:22:17.797 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78840.log 2024-04-24T08:22:17.797 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46325.log.gz 2024-04-24T08:22:17.798 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.76924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78402.log 2024-04-24T08:22:17.798 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76924.log.gz 2024-04-24T08:22:17.799 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78840.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.78840.log.gz 2024-04-24T08:22:17.800 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72636.log 2024-04-24T08:22:17.801 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74290.log 2024-04-24T08:22:17.801 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78402.log.gz 2024-04-24T08:22:17.802 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74938.log 2024-04-24T08:22:17.802 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74290.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.72636.log.gz 2024-04-24T08:22:17.803 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74290.log.gz 2024-04-24T08:22:17.803 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52028.log 2024-04-24T08:22:17.804 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52700.log 2024-04-24T08:22:17.804 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.74938.log.gz 2024-04-24T08:22:17.805 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52028.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T08:22:17.805 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52028.log.gz 2024-04-24T08:22:17.806 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52700.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52700.log.gz 2024-04-24T08:22:17.806 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44596.log 2024-04-24T08:22:17.807 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60802.log 2024-04-24T08:22:17.808 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44596.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44596.log.gz 2024-04-24T08:22:17.809 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53125.log 2024-04-24T08:22:17.809 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76740.log 2024-04-24T08:22:17.810 INFO:teuthology.orchestra.run.smithi069.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.60802.log.gz 2024-04-24T08:22:17.810 INFO:teuthology.orchestra.run.smithi069.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T08:22:17.811 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53125.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53125.log.gz 2024-04-24T08:22:17.811 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57318.log 2024-04-24T08:22:17.811 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.76740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.63328.log 2024-04-24T08:22:17.812 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76740.log.gz 2024-04-24T08:22:17.812 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57318.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.57318.log.gz 2024-04-24T08:22:17.813 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43512.log 2024-04-24T08:22:17.814 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.63328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59966.log 2024-04-24T08:22:17.815 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71730.log 2024-04-24T08:22:17.815 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43512.log.gz 2024-04-24T08:22:17.816 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78360.log 2024-04-24T08:22:17.816 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59966.log.gz 2024-04-24T08:22:17.817 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39208.log 2024-04-24T08:22:17.817 INFO:teuthology.orchestra.run.smithi069.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.0.63328.log.gz 2024-04-24T08:22:17.817 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71730.log.gz 2024-04-24T08:22:17.818 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38899.log 2024-04-24T08:22:17.818 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.78360.log.gz 2024-04-24T08:22:17.819 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39208.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39208.log.gz 2024-04-24T08:22:17.819 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48172.log 2024-04-24T08:22:17.820 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63836.log 2024-04-24T08:22:17.820 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38899.log.gz 2024-04-24T08:22:17.821 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76782.log 2024-04-24T08:22:17.821 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48172.log.gz 2024-04-24T08:22:17.822 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67998.log 2024-04-24T08:22:17.822 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63836.log.gz 2024-04-24T08:22:17.823 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.76782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42661.log 2024-04-24T08:22:17.823 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76782.log.gz 2024-04-24T08:22:17.824 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49949.log 2024-04-24T08:22:17.824 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67998.log.gz 2024-04-24T08:22:17.825 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42661.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42661.log.gz 2024-04-24T08:22:17.825 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50289.log 2024-04-24T08:22:17.826 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69034.log 2024-04-24T08:22:17.826 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49949.log.gz 2024-04-24T08:22:17.827 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81092.log 2024-04-24T08:22:17.827 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50289.log.gz 2024-04-24T08:22:17.828 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69034.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.69034.log.gz 2024-04-24T08:22:17.828 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-24T08:22:17.829 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69085.log 2024-04-24T08:22:17.829 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81092.log.gz 2024-04-24T08:22:17.830 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80328.log 2024-04-24T08:22:17.831 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69085.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69085.log.gz 2024-04-24T08:22:17.831 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62307.log 2024-04-24T08:22:17.832 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51791.log 2024-04-24T08:22:17.832 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80328.log.gz 2024-04-24T08:22:17.833 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62307.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62307.log.gz 2024-04-24T08:22:17.833 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79332.log 2024-04-24T08:22:17.834 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75617.log 2024-04-24T08:22:17.834 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51791.log.gz 2024-04-24T08:22:17.835 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.79332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43942.log 2024-04-24T08:22:17.836 INFO:teuthology.orchestra.run.smithi069.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.79332.log.gz 2024-04-24T08:22:17.836 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46241.log 2024-04-24T08:22:17.837 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8%/var/log/ceph/ceph-client.admin.43942.log: -- replaced with /var/log/ceph/ceph-client.admin.75617.log.gz 2024-04-24T08:22:17.838 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43942.log.gz 2024-04-24T08:22:17.838 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.64754.log 2024-04-24T08:22:17.839 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43428.log 2024-04-24T08:22:17.839 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46241.log.gz 2024-04-24T08:22:17.840 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.64754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73650.log 2024-04-24T08:22:17.841 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60171.log 2024-04-24T08:22:17.842 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43428.log.gz 2024-04-24T08:22:17.842 INFO:teuthology.orchestra.run.smithi069.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.0.64754.log.gz 2024-04-24T08:22:17.843 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70594.log 2024-04-24T08:22:17.843 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73650.log.gz 2024-04-24T08:22:17.844 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64910.log 2024-04-24T08:22:17.845 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70594.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.60171.log.gz 2024-04-24T08:22:17.845 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77008.log 89.2% 2024-04-24T08:22:17.845 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70594.log.gz 2024-04-24T08:22:17.846 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67280.log 2024-04-24T08:22:17.847 INFO:teuthology.orchestra.run.smithi069.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.64910.log.gz 2024-04-24T08:22:17.847 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.77008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55287.log 2024-04-24T08:22:17.848 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.77008.log.gz 2024-04-24T08:22:17.848 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70143.log 2024-04-24T08:22:17.850 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55287.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.67280.log.gz 2024-04-24T08:22:17.850 INFO:teuthology.orchestra.run.smithi069.stderr:gzip 88.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.55287.log.gz --verbose 2024-04-24T08:22:17.850 INFO:teuthology.orchestra.run.smithi069.stderr: -- /var/log/ceph/ceph-client.admin.62858.log 2024-04-24T08:22:17.851 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78644.log 2024-04-24T08:22:17.851 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70143.log.gz 2024-04-24T08:22:17.852 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57083.log 2024-04-24T08:22:17.852 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62858.log.gz 2024-04-24T08:22:17.853 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81219.log 2024-04-24T08:22:17.853 INFO:teuthology.orchestra.run.smithi069.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.78644.log.gz 2024-04-24T08:22:17.854 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72422.log 2024-04-24T08:22:17.855 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.57083.log.gz 2024-04-24T08:22:17.855 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55650.log 2024-04-24T08:22:17.855 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81219.log.gz 2024-04-24T08:22:17.856 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47182.log 2024-04-24T08:22:17.856 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72422.log.gz 2024-04-24T08:22:17.857 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73060.log 2024-04-24T08:22:17.857 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55650.log.gz 2024-04-24T08:22:17.858 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.47182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61967.log 2024-04-24T08:22:17.859 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73060.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47182.log.gz 2024-04-24T08:22:17.860 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73060.log.gz 2024-04-24T08:22:17.860 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57125.log 2024-04-24T08:22:17.860 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80622.log 2024-04-24T08:22:17.861 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61967.log.gz 2024-04-24T08:22:17.862 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81524.log 2024-04-24T08:22:17.862 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57125.log.gz 2024-04-24T08:22:17.862 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81314.log 2024-04-24T08:22:17.863 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80622.log.gz 2024-04-24T08:22:17.864 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62009.log 2024-04-24T08:22:17.864 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81524.log.gz 2024-04-24T08:22:17.864 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/ceph-mon.b.log: 91.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-24T08:22:17.865 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67593.log 2024-04-24T08:22:17.865 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81314.log.gz 2024-04-24T08:22:17.866 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42871.log 2024-04-24T08:22:17.866 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62009.log.gz 2024-04-24T08:22:17.867 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40930.log 2024-04-24T08:22:17.867 INFO:teuthology.orchestra.run.smithi069.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67593.log.gz 2024-04-24T08:22:17.868 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74033.log 2024-04-24T08:22:17.869 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42871.log.gz 2024-04-24T08:22:17.869 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78486.log 2024-04-24T08:22:17.869 INFO:teuthology.orchestra.run.smithi069.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40930.log.gz 2024-04-24T08:22:17.871 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78929.log 2024-04-24T08:22:17.871 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74033.log.gz 2024-04-24T08:22:17.872 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65493.log 2024-04-24T08:22:17.872 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.78486.log.gz 2024-04-24T08:22:17.873 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70227.log 2024-04-24T08:22:17.874 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39418.log 2024-04-24T08:22:17.874 INFO:teuthology.orchestra.run.smithi069.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.78929.log.gz 2024-04-24T08:22:17.874 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65493.log.gz 2024-04-24T08:22:17.875 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54542.log 2024-04-24T08:22:17.875 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70227.log.gz 2024-04-24T08:22:17.876 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59433.log 2024-04-24T08:22:17.876 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39418.log.gz 2024-04-24T08:22:17.877 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66174.log 2024-04-24T08:22:17.877 INFO:teuthology.orchestra.run.smithi069.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.54542.log.gz 2024-04-24T08:22:17.878 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43554.log 2024-04-24T08:22:17.879 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59433.log.gz 2024-04-24T08:22:17.879 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.66174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56089.log 2024-04-24T08:22:17.881 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43554.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.66174.log.gz 2024-04-24T08:22:17.881 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71526.log 2024-04-24T08:22:17.881 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43554.log.gz 2024-04-24T08:22:17.882 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59704.log 2024-04-24T08:22:17.882 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56089.log.gz 2024-04-24T08:22:17.883 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45556.log 2024-04-24T08:22:17.883 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71526.log.gz 2024-04-24T08:22:17.884 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63962.log 2024-04-24T08:22:17.884 INFO:teuthology.orchestra.run.smithi069.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.59704.log.gz 2024-04-24T08:22:17.885 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79013.log 2024-04-24T08:22:17.885 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45556.log.gz 2024-04-24T08:22:17.886 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68353.log 2024-04-24T08:22:17.887 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63962.log.gz 2024-04-24T08:22:17.887 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.79013.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T08:22:17.889 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68353.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.79013.log.gz 2024-04-24T08:22:17.889 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62564.log 2024-04-24T08:22:17.889 INFO:teuthology.orchestra.run.smithi069.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.68353.log.gz 2024-04-24T08:22:17.890 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62434.log 2024-04-24T08:22:17.891 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62564.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62564.log.gz 2024-04-24T08:22:17.891 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48959.log 2024-04-24T08:22:17.892 INFO:teuthology.orchestra.run.smithi121.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-24T08:22:17.893 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79226.log 2024-04-24T08:22:17.893 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62434.log.gz 2024-04-24T08:22:17.893 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48959.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48959.log.gz 2024-04-24T08:22:17.893 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54283.log 2024-04-24T08:22:17.894 INFO:teuthology.orchestra.run.smithi069.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T08:22:17.894 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.79226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80496.log 2024-04-24T08:22:17.895 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79226.log.gz 2024-04-24T08:22:17.895 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47074.log 2024-04-24T08:22:17.896 INFO:teuthology.orchestra.run.smithi069.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.54283.log.gz 2024-04-24T08:22:17.896 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80370.log 2024-04-24T08:22:17.897 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80496.log.gz 2024-04-24T08:22:17.897 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.47074.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47074.log.gz 2024-04-24T08:22:17.898 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63568.log 2024-04-24T08:22:17.899 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.77799.log 2024-04-24T08:22:17.899 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80370.log.gz 2024-04-24T08:22:17.900 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78607.log 2024-04-24T08:22:17.900 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63568.log.gz 2024-04-24T08:22:17.901 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.77799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50415.log 2024-04-24T08:22:17.902 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44068.log 2024-04-24T08:22:17.902 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78607.log.gz 2024-04-24T08:22:17.903 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50415.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50415.log.gz 2024-04-24T08:22:17.904 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68479.log 2024-04-24T08:22:17.905 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55440.log 2024-04-24T08:22:17.905 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44068.log.gz 2024-04-24T08:22:17.905 INFO:teuthology.orchestra.run.smithi069.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.0.77799.log.gz 2024-04-24T08:22:17.906 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68479.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.68479.log.gz 2024-04-24T08:22:17.906 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58467.log 2024-04-24T08:22:17.906 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75004.log 2024-04-24T08:22:17.907 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55440.log.gz 2024-04-24T08:22:17.908 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46506.log 2024-04-24T08:22:17.908 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58467.log.gz 2024-04-24T08:22:17.909 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59391.log 2024-04-24T08:22:17.909 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75004.log.gz 2024-04-24T08:22:17.910 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53580.log 2024-04-24T08:22:17.910 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46506.log.gz 2024-04-24T08:22:17.911 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75843.log 2024-04-24T08:22:17.911 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59391.log.gz 2024-04-24T08:22:17.912 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57947.log 2024-04-24T08:22:17.913 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53580.log.gz 2024-04-24T08:22:17.913 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55920.log 2024-04-24T08:22:17.914 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75843.log.gz 2024-04-24T08:22:17.915 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61300.log 2024-04-24T08:22:17.915 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57947.log.gz 2024-04-24T08:22:17.916 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65235.log 2024-04-24T08:22:17.916 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55920.log.gz 2024-04-24T08:22:17.917 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61781.log 2024-04-24T08:22:17.917 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61300.log.gz 2024-04-24T08:22:17.918 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64368.log 2024-04-24T08:22:17.918 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65235.log.gz 2024-04-24T08:22:17.919 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49085.log 2024-04-24T08:22:17.919 INFO:teuthology.orchestra.run.smithi069.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61781.log.gz 2024-04-24T08:22:17.920 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73186.log 2024-04-24T08:22:17.920 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64368.log.gz 2024-04-24T08:22:17.921 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.66883.log 2024-04-24T08:22:17.922 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49085.log.gz 2024-04-24T08:22:17.923 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48917.log 2024-04-24T08:22:17.923 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.73186.log.gz 2024-04-24T08:22:17.924 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.66883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50079.log 2024-04-24T08:22:17.925 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67746.log 2024-04-24T08:22:17.925 INFO:teuthology.orchestra.run.smithi069.stderr: 92.2% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48917.log.gz -- replaced with /var/log/ceph/ceph-client.0.66883.log.gz 2024-04-24T08:22:17.925 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-24T08:22:17.926 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50247.log 2024-04-24T08:22:17.926 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50079.log.gz 2024-04-24T08:22:17.927 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44194.log 2024-04-24T08:22:17.927 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67746.log.gz 2024-04-24T08:22:17.928 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74075.log 2024-04-24T08:22:17.928 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50247.log.gz 2024-04-24T08:22:17.930 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66479.log 2024-04-24T08:22:17.930 INFO:teuthology.orchestra.run.smithi069.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.44194.log.gz 2024-04-24T08:22:17.930 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61865.log 2024-04-24T08:22:17.931 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.74075.log.gz 2024-04-24T08:22:17.932 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.66479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42451.log 2024-04-24T08:22:17.932 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66479.log.gz 2024-04-24T08:22:17.933 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56599.log 2024-04-24T08:22:17.933 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61865.log.gz 2024-04-24T08:22:17.934 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71484.log 2024-04-24T08:22:17.934 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42451.log.gz 2024-04-24T08:22:17.935 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74160.log 2024-04-24T08:22:17.935 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56599.log.gz 2024-04-24T08:22:17.936 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33433.log 2024-04-24T08:22:17.936 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71484.log.gz 2024-04-24T08:22:17.937 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71610.log 2024-04-24T08:22:17.938 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74160.log.gz 2024-04-24T08:22:17.938 INFO:teuthology.orchestra.run.smithi121.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-24T08:22:17.939 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.33433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77325.log 2024-04-24T08:22:17.939 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33433.log.gz 2024-04-24T08:22:17.939 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67551.log 2024-04-24T08:22:17.939 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71610.log.gz 2024-04-24T08:22:17.940 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.77325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44236.log 2024-04-24T08:22:17.940 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77325.log.gz 2024-04-24T08:22:17.941 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54650.log 2024-04-24T08:22:17.941 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67551.log.gz 2024-04-24T08:22:17.942 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78260.log 2024-04-24T08:22:17.943 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44236.log.gz 2024-04-24T08:22:17.943 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44496.log 2024-04-24T08:22:17.944 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78260.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54650.log.gz 2024-04-24T08:22:17.945 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48046.log 2024-04-24T08:22:17.945 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.78260.log.gz 2024-04-24T08:22:17.946 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61739.log 2024-04-24T08:22:17.946 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44496.log.gz 2024-04-24T08:22:17.947 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47988.log 2024-04-24T08:22:17.947 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48046.log.gz 2024-04-24T08:22:17.948 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51944.log 2024-04-24T08:22:17.948 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61739.log.gz 2024-04-24T08:22:17.949 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.47988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50373.log 2024-04-24T08:22:17.949 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47988.log.gz 2024-04-24T08:22:17.950 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39915.log 2024-04-24T08:22:17.951 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51944.log.gz 2024-04-24T08:22:17.951 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50373.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50373.log.gz 2024-04-24T08:22:17.951 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71342.log 2024-04-24T08:22:17.952 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51103.log 2024-04-24T08:22:17.952 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39915.log.gz 2024-04-24T08:22:17.953 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62392.log 2024-04-24T08:22:17.953 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.71342.log.gz 2024-04-24T08:22:17.954 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51103.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-24T08:22:17.955 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51103.log.gz 2024-04-24T08:22:17.956 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70059.log 2024-04-24T08:22:17.956 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62392.log.gz 2024-04-24T08:22:17.956 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50331.log 2024-04-24T08:22:17.958 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39594.log 2024-04-24T08:22:17.958 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70059.log.gz 2024-04-24T08:22:17.959 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66089.log 2024-04-24T08:22:17.959 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50331.log.gz 2024-04-24T08:22:17.960 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39594.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39594.log.gz 2024-04-24T08:22:17.960 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44152.log 2024-04-24T08:22:17.961 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.66089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56957.log 2024-04-24T08:22:17.962 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66089.log.gz 2024-04-24T08:22:17.963 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44152.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44152.log.gz 2024-04-24T08:22:17.963 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81692.log 2024-04-24T08:22:17.964 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56173.log 2024-04-24T08:22:17.964 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56957.log.gz 2024-04-24T08:22:17.966 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81692.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81692.log.gz 2024-04-24T08:22:17.966 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66234.log 2024-04-24T08:22:17.967 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78218.log 2024-04-24T08:22:17.967 INFO:teuthology.orchestra.run.smithi069.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56173.log.gz 2024-04-24T08:22:17.968 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.66234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38563.log 2024-04-24T08:22:17.970 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78218.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78218.log.gz 2024-04-24T08:22:17.970 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59207.log 2024-04-24T08:22:17.970 INFO:teuthology.orchestra.run.smithi069.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.66234.log.gz 2024-04-24T08:22:17.971 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76656.log 2024-04-24T08:22:17.971 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38563.log.gz 2024-04-24T08:22:17.972 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-04-24T08:22:17.972 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59207.log.gz 2024-04-24T08:22:17.973 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.76656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42997.log 2024-04-24T08:22:17.973 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76656.log.gz 2024-04-24T08:22:17.974 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58414.log 2024-04-24T08:22:17.975 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-04-24T08:22:17.978 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64280.log 2024-04-24T08:22:17.978 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42997.log.gz 2024-04-24T08:22:17.978 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55161.log 2024-04-24T08:22:17.978 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58414.log.gz 2024-04-24T08:22:17.979 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64280.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64280.log.gzgzip 2024-04-24T08:22:17.979 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58047.log 2024-04-24T08:22:17.979 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58335.log 2024-04-24T08:22:17.980 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55161.log.gz 2024-04-24T08:22:17.981 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55524.log 2024-04-24T08:22:17.981 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58047.log.gz 2024-04-24T08:22:17.982 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66521.log 2024-04-24T08:22:17.982 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58335.log.gz 2024-04-24T08:22:17.983 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41200.log 2024-04-24T08:22:17.984 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55524.log.gz 2024-04-24T08:22:17.985 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.66521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80664.log 2024-04-24T08:22:17.985 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41200.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.66521.log.gz 2024-04-24T08:22:17.986 INFO:teuthology.orchestra.run.smithi069.stderr:gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.41200.log.gz --verbose -- 2024-04-24T08:22:17.986 INFO:teuthology.orchestra.run.smithi069.stderr: /var/log/ceph/ceph-client.admin.45094.log 2024-04-24T08:22:17.987 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53083.log 2024-04-24T08:22:17.987 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80664.log.gz 2024-04-24T08:22:17.989 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46073.log 2024-04-24T08:22:17.989 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45094.log.gz 2024-04-24T08:22:17.990 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61925.log 2024-04-24T08:22:17.990 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53083.log.gz 2024-04-24T08:22:17.991 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49001.log 2024-04-24T08:22:17.991 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46073.log.gz 2024-04-24T08:22:17.992 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.60665.log 2024-04-24T08:22:17.992 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61925.log.gz 2024-04-24T08:22:17.993 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58293.log 2024-04-24T08:22:17.994 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49001.log.gz 2024-04-24T08:22:17.995 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.60665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47820.log 2024-04-24T08:22:17.996 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58293.log: 90.4% -- replaced with /var/log/ceph/ceph-client.0.60665.log.gz 2024-04-24T08:22:17.996 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58293.log.gz 2024-04-24T08:22:17.996 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68184.log 2024-04-24T08:22:17.997 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.47820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74752.log 2024-04-24T08:22:17.998 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47820.log.gz 2024-04-24T08:22:17.998 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48527.log 2024-04-24T08:22:17.999 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68184.log.gz 2024-04-24T08:22:18.000 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74752.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74752.log.gz 2024-04-24T08:22:18.000 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73480.log 2024-04-24T08:22:18.001 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49822.log 2024-04-24T08:22:18.001 INFO:teuthology.orchestra.run.smithi069.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48527.log.gz 2024-04-24T08:22:18.003 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45743.log 2024-04-24T08:22:18.003 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73480.log.gz 2024-04-24T08:22:18.004 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80893.log 2024-04-24T08:22:18.004 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49822.log.gz 2024-04-24T08:22:18.005 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45743.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45743.log.gz 2024-04-24T08:22:18.005 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-24T08:22:18.006 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38994.log 2024-04-24T08:22:18.007 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80893.log.gz 2024-04-24T08:22:18.008 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53041.log 2024-04-24T08:22:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38994.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38994.log.gz 2024-04-24T08:22:18.009 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81650.log 2024-04-24T08:22:18.010 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73565.log 2024-04-24T08:22:18.011 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53041.log: /var/log/ceph/ceph-client.admin.81650.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53041.log.gz 2024-04-24T08:22:18.012 INFO:teuthology.orchestra.run.smithi069.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81650.log.gz -5 2024-04-24T08:22:18.012 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56789.log 2024-04-24T08:22:18.013 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54824.log 2024-04-24T08:22:18.013 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73565.log.gz 2024-04-24T08:22:18.014 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56789.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56789.log.gz 2024-04-24T08:22:18.014 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41637.log 2024-04-24T08:22:18.015 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46636.log 2024-04-24T08:22:18.017 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.41637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60937.log 2024-04-24T08:22:18.017 INFO:teuthology.orchestra.run.smithi069.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.54824.log.gz 2024-04-24T08:22:18.018 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46636.log: 91.3% -- replaced with /var/log/ceph/ceph-client.0.41637.log.gz 2024-04-24T08:22:18.018 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78564.log 2024-04-24T08:22:18.018 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46636.log.gz 2024-04-24T08:22:18.019 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68521.log 2024-04-24T08:22:18.020 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60937.log.gz 2024-04-24T08:22:18.021 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74542.log 2024-04-24T08:22:18.021 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.78564.log.gz 2024-04-24T08:22:18.022 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77521.log 2024-04-24T08:22:18.022 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68521.log.gz 2024-04-24T08:22:18.023 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74542.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74542.log.gz 2024-04-24T08:22:18.023 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43738.log 2024-04-24T08:22:18.024 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.77521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52280.log 2024-04-24T08:22:18.025 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77521.log.gz 2024-04-24T08:22:18.026 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60345.log 2024-04-24T08:22:18.026 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43738.log.gz 2024-04-24T08:22:18.027 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73776.log 2024-04-24T08:22:18.027 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52280.log.gz 2024-04-24T08:22:18.028 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60345.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60345.log.gzgzip 2024-04-24T08:22:18.028 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.81356.log 2024-04-24T08:22:18.029 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82045.log 2024-04-24T08:22:18.029 INFO:teuthology.orchestra.run.smithi069.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.73776.log.gz 2024-04-24T08:22:18.031 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54392.log 2024-04-24T08:22:18.031 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81356.log.gz 2024-04-24T08:22:18.032 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.82045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46455.log 2024-04-24T08:22:18.032 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82045.log.gz 2024-04-24T08:22:18.033 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77272.log 2024-04-24T08:22:18.034 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54392.log.gz 2024-04-24T08:22:18.034 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57252.log 2024-04-24T08:22:18.035 INFO:teuthology.orchestra.run.smithi069.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.46455.log.gz 2024-04-24T08:22:18.036 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.77272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54608.log 2024-04-24T08:22:18.036 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77272.log.gz 2024-04-24T08:22:18.037 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57252.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-24T08:22:18.038 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.57252.log.gz 2024-04-24T08:22:18.038 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54608.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-24T08:22:18.039 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54608.log.gz 2024-04-24T08:22:18.039 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64476.log 2024-04-24T08:22:18.040 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46990.log 2024-04-24T08:22:18.042 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80175.log 2024-04-24T08:22:18.042 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64476.log.gz 2024-04-24T08:22:18.044 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55482.log 2024-04-24T08:22:18.044 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46990.log.gz 2024-04-24T08:22:18.045 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80175.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80175.log.gz 2024-04-24T08:22:18.046 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55482.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55482.log.gz 2024-04-24T08:22:18.047 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44412.log 2024-04-24T08:22:18.048 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74710.log 2024-04-24T08:22:18.050 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68124.log 2024-04-24T08:22:18.050 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44412.log.gz 2024-04-24T08:22:18.051 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74710.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74710.log.gz 2024-04-24T08:22:18.052 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68124.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.68124.log.gz 2024-04-24T08:22:18.052 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62900.log 2024-04-24T08:22:18.053 INFO:teuthology.orchestra.run.smithi069.stderr: 94.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-24T08:22:18.053 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52998.log 2024-04-24T08:22:18.055 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62900.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62900.log.gz 2024-04-24T08:22:18.055 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65127.log 2024-04-24T08:22:18.056 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5/var/log/ceph/ceph-client.admin.52998.log: --verbose -- /var/log/ceph/ceph-client.admin.53664.log 2024-04-24T08:22:18.056 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52998.log.gz 2024-04-24T08:22:18.057 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65127.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65127.log.gz 2024-04-24T08:22:18.057 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45052.log 2024-04-24T08:22:18.058 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64040.log 2024-04-24T08:22:18.059 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53664.log.gz 2024-04-24T08:22:18.059 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45052.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45052.log.gz 2024-04-24T08:22:18.060 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40270.log 2024-04-24T08:22:18.060 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81176.log 2024-04-24T08:22:18.061 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64040.log.gz 2024-04-24T08:22:18.062 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72358.log 2024-04-24T08:22:18.062 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40270.log.gz 2024-04-24T08:22:18.063 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81176.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-24T08:22:18.063 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81176.log.gz 2024-04-24T08:22:18.064 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81608.log 2024-04-24T08:22:18.064 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72358.log.gz 2024-04-24T08:22:18.065 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49907.log 2024-04-24T08:22:18.066 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81608.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60008.log 2024-04-24T08:22:18.066 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81608.log.gz 2024-04-24T08:22:18.068 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68437.log 2024-04-24T08:22:18.068 INFO:teuthology.orchestra.run.smithi069.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49907.log.gz 2024-04-24T08:22:18.069 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60008.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60008.log.gz 2024-04-24T08:22:18.070 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42299.log 2024-04-24T08:22:18.070 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68437.log.gz 2024-04-24T08:22:18.071 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40762.log 2024-04-24T08:22:18.072 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42299.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42299.log.gz 2024-04-24T08:22:18.073 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59858.log 2024-04-24T08:22:18.074 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48749.log 2024-04-24T08:22:18.074 INFO:teuthology.orchestra.run.smithi069.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40762.log.gz 2024-04-24T08:22:18.075 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.75282.log 2024-04-24T08:22:18.076 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59858.log.gz 2024-04-24T08:22:18.077 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53706.log 2024-04-24T08:22:18.077 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48749.log.gz 2024-04-24T08:22:18.077 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.75282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55734.log 2024-04-24T08:22:18.080 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53706.log: 90.8%/var/log/ceph/ceph-client.admin.55734.log: -- replaced with /var/log/ceph/ceph-client.admin.53706.log.gz 2024-04-24T08:22:18.081 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55734.log.gz 2024-04-24T08:22:18.081 INFO:teuthology.orchestra.run.smithi069.stderr: -- /var/log/ceph/ceph-client.admin.49780.log 2024-04-24T08:22:18.082 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56831.log 2024-04-24T08:22:18.083 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49780.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49780.log.gz 2024-04-24T08:22:18.084 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.56831.log: -- /var/log/ceph/ceph-client.admin.52490.log 2024-04-24T08:22:18.084 INFO:teuthology.orchestra.run.smithi069.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.0.75282.log.gz 2024-04-24T08:22:18.085 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56831.log.gz 2024-04-24T08:22:18.085 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50457.log 2024-04-24T08:22:18.086 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68040.log 2024-04-24T08:22:18.087 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52490.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52490.log.gz 2024-04-24T08:22:18.087 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50457.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50457.log.gz 2024-04-24T08:22:18.088 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46031.log 2024-04-24T08:22:18.089 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68040.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68040.log.gz 2024-04-24T08:22:18.089 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59249.log 2024-04-24T08:22:18.091 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.46031.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.69527.log 2024-04-24T08:22:18.091 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46031.log.gz 2024-04-24T08:22:18.092 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59249.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59249.log.gz 2024-04-24T08:22:18.092 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39582.log 2024-04-24T08:22:18.094 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58358.log 2024-04-24T08:22:18.094 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.69527.log.gz 2024-04-24T08:22:18.095 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51749.log 2024-04-24T08:22:18.095 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39582.log.gz 2024-04-24T08:22:18.097 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58358.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58358.log.gz 2024-04-24T08:22:18.098 INFO:teuthology.orchestra.run.smithi121.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-24T08:22:18.098 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.51749.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60279.log 2024-04-24T08:22:18.098 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51749.log.gz 2024-04-24T08:22:18.098 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40552.log 2024-04-24T08:22:18.099 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66433.log 2024-04-24T08:22:18.101 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40552.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60279.log.gz 2024-04-24T08:22:18.101 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40552.log.gz 2024-04-24T08:22:18.102 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66391.log 2024-04-24T08:22:18.102 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.66433.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66433.log.gz 2024-04-24T08:22:18.102 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47904.log 2024-04-24T08:22:18.104 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.66391.log: /var/log/ceph/ceph-client.admin.61021.log 2024-04-24T08:22:18.104 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66391.log.gz 2024-04-24T08:22:18.105 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.47904.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47904.log.gz 2024-04-24T08:22:18.106 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61216.log 2024-04-24T08:22:18.106 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61021.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61021.log.gz 2024-04-24T08:22:18.107 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50163.log 2024-04-24T08:22:18.108 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59517.log 2024-04-24T08:22:18.108 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61216.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61216.log.gz 2024-04-24T08:22:18.109 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50163.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50163.log.gz 2024-04-24T08:22:18.110 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52070.log 2024-04-24T08:22:18.110 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59517.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59517.log.gz 2024-04-24T08:22:18.111 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48453.log 2024-04-24T08:22:18.112 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76113.log 2024-04-24T08:22:18.113 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52070.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52070.log.gz 2024-04-24T08:22:18.114 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48453.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.48453.log.gz 2024-04-24T08:22:18.114 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72316.log 2024-04-24T08:22:18.115 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.76113.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76113.log.gz 2024-04-24T08:22:18.115 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48707.log 2024-04-24T08:22:18.116 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58509.log 2024-04-24T08:22:18.117 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72316.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72316.log.gz 2024-04-24T08:22:18.118 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63736.log 2024-04-24T08:22:18.118 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48707.log.gz 2024-04-24T08:22:18.119 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58509.log: gzip -5 89.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.58509.log.gz 2024-04-24T08:22:18.119 INFO:teuthology.orchestra.run.smithi069.stderr: /var/log/ceph/ceph-client.admin.81818.log 2024-04-24T08:22:18.121 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67088.log 2024-04-24T08:22:18.121 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63736.log.gz 2024-04-24T08:22:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81818.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81818.log.gz 2024-04-24T08:22:18.122 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72206.log 2024-04-24T08:22:18.123 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67088.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.67088.log.gz 2024-04-24T08:22:18.123 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40888.log 2024-04-24T08:22:18.125 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.72206.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46720.log 2024-04-24T08:22:18.125 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72206.log.gz 2024-04-24T08:22:18.126 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40888.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40888.log.gz 2024-04-24T08:22:18.126 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52364.log 2024-04-24T08:22:18.127 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46720.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46720.log.gz 2024-04-24T08:22:18.127 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48293.log 2024-04-24T08:22:18.129 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5/var/log/ceph/ceph-client.admin.52364.log: --verbose -- /var/log/ceph/ceph-client.admin.39925.log 2024-04-24T08:22:18.129 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52364.log.gz 2024-04-24T08:22:18.130 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48293.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48293.log.gz 2024-04-24T08:22:18.131 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56005.log 2024-04-24T08:22:18.131 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39925.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39925.log.gz 2024-04-24T08:22:18.132 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45430.log 2024-04-24T08:22:18.133 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39334.log 2024-04-24T08:22:18.133 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56005.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56005.log.gz 2024-04-24T08:22:18.134 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45430.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45430.log.gz 2024-04-24T08:22:18.135 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76698.log 2024-04-24T08:22:18.135 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39334.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39334.log.gz 2024-04-24T08:22:18.136 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41741.log 2024-04-24T08:22:18.137 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.76698.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.70369.log 2024-04-24T08:22:18.137 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76698.log.gz 2024-04-24T08:22:18.138 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41741.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41741.log.gz 2024-04-24T08:22:18.139 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61468.log 2024-04-24T08:22:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70369.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70369.log.gz 2024-04-24T08:22:18.140 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74812.log 2024-04-24T08:22:18.141 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61468.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61468.log.gz 2024-04-24T08:22:18.141 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57597.log 2024-04-24T08:22:18.143 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71426.log 2024-04-24T08:22:18.143 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74812.log.gz 2024-04-24T08:22:18.145 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.57597.log: /var/log/ceph/ceph-client.admin.71426.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71426.log.gz 2024-04-24T08:22:18.146 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69871.log 2024-04-24T08:22:18.147 INFO:teuthology.orchestra.run.smithi069.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.0.57597.log.gz 2024-04-24T08:22:18.147 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56556.log 2024-04-24T08:22:18.148 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.0.69871.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.79131.log 2024-04-24T08:22:18.150 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56556.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56556.log.gz 2024-04-24T08:22:18.150 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56683.log 2024-04-24T08:22:18.151 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.79131.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79131.log.gz 2024-04-24T08:22:18.152 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51902.log 2024-04-24T08:22:18.153 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56683.log: 94.0% -- replaced with /var/log/ceph/ceph-client.0.69871.log.gz 2024-04-24T08:22:18.153 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56683.log.gz 2024-04-24T08:22:18.154 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55082.log 2024-04-24T08:22:18.155 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51902.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51902.log.gz 2024-04-24T08:22:18.155 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45178.log 2024-04-24T08:22:18.157 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.55082.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42955.log 2024-04-24T08:22:18.157 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55082.log.gz 2024-04-24T08:22:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45178.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45178.log.gz 2024-04-24T08:22:18.158 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45598.log 2024-04-24T08:22:18.159 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42955.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42955.log.gz 2024-04-24T08:22:18.160 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71931.log 2024-04-24T08:22:18.161 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66276.log 2024-04-24T08:22:18.161 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45598.log.gz 2024-04-24T08:22:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71931.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.71931.log.gz 2024-04-24T08:22:18.163 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.66276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67872.log 2024-04-24T08:22:18.165 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65343.log 2024-04-24T08:22:18.165 INFO:teuthology.orchestra.run.smithi069.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.66276.log.gz 2024-04-24T08:22:18.166 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61552.log 2024-04-24T08:22:18.166 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67872.log.gz 2024-04-24T08:22:18.168 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74668.log 2024-04-24T08:22:18.168 INFO:teuthology.orchestra.run.smithi069.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.65343.log.gz 2024-04-24T08:22:18.169 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54050.log 2024-04-24T08:22:18.169 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61552.log.gz 2024-04-24T08:22:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74668.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.74668.log.gz 2024-04-24T08:22:18.171 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.54050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40263.log 2024-04-24T08:22:18.173 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78444.log 2024-04-24T08:22:18.174 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40263.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40263.log.gz 2024-04-24T08:22:18.174 INFO:teuthology.orchestra.run.smithi069.stderr: 93.8%/var/log/ceph/ceph-client.admin.78444.log: -- replaced with /var/log/ceph/ceph-client.0.54050.log.gz 2024-04-24T08:22:18.175 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51514.log 2024-04-24T08:22:18.175 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.78444.log.gz 2024-04-24T08:22:18.176 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48214.log 2024-04-24T08:22:18.177 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51514.log: 89.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45136.log 2024-04-24T08:22:18.178 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51514.log.gz 2024-04-24T08:22:18.179 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40510.log 2024-04-24T08:22:18.179 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48214.log.gz 2024-04-24T08:22:18.180 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45136.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45136.log.gz 2024-04-24T08:22:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49398.log 2024-04-24T08:22:18.181 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40510.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40510.log.gz 2024-04-24T08:22:18.182 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-24T08:22:18.183 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52196.log 2024-04-24T08:22:18.184 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49398.log.gz 2024-04-24T08:22:18.186 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.52196.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52196.log.gz 2024-04-24T08:22:18.187 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42829.log 2024-04-24T08:22:18.188 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-24T08:22:18.189 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56131.log 2024-04-24T08:22:18.190 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78092.log 2024-04-24T08:22:18.190 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42829.log.gz 2024-04-24T08:22:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56131.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56131.log.gz 2024-04-24T08:22:18.192 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78092.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78092.log.gz 2024-04-24T08:22:18.193 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78863.log 2024-04-24T08:22:18.194 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59638.log 2024-04-24T08:22:18.196 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46283.log 2024-04-24T08:22:18.196 INFO:teuthology.orchestra.run.smithi069.stderr: 91.2%/var/log/ceph/ceph-client.admin.59638.log: -- replaced with /var/log/ceph/ceph-client.admin.78863.log.gz 2024-04-24T08:22:18.197 INFO:teuthology.orchestra.run.smithi069.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.59638.log.gz 2024-04-24T08:22:18.198 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46283.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46283.log.gz 2024-04-24T08:22:18.199 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70285.log 2024-04-24T08:22:18.200 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81860.log 2024-04-24T08:22:18.202 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75885.log 2024-04-24T08:22:18.202 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70285.log.gz 2024-04-24T08:22:18.203 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81860.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81860.log.gz 2024-04-24T08:22:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75885.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75885.log.gz 2024-04-24T08:22:18.204 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51623.log 2024-04-24T08:22:18.206 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43612.log 2024-04-24T08:22:18.207 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62265.log 2024-04-24T08:22:18.208 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51623.log.gz 2024-04-24T08:22:18.208 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43612.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43612.log.gz 2024-04-24T08:22:18.210 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62265.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62265.log.gz 2024-04-24T08:22:18.210 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81734.log 2024-04-24T08:22:18.211 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71300.log 2024-04-24T08:22:18.213 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71568.log 2024-04-24T08:22:18.213 INFO:teuthology.orchestra.run.smithi069.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.81734.log.gz 2024-04-24T08:22:18.214 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71300.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71300.log.gz 2024-04-24T08:22:18.216 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71568.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71568.log.gz 2024-04-24T08:22:18.216 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62690.log 2024-04-24T08:22:18.217 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73228.log 2024-04-24T08:22:18.219 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60387.log 2024-04-24T08:22:18.219 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62690.log.gz 2024-04-24T08:22:18.220 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63878.log 2024-04-24T08:22:18.220 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73228.log.gz 2024-04-24T08:22:18.222 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60387.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60387.log.gz 2024-04-24T08:22:18.222 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49699.log 2024-04-24T08:22:18.223 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63878.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63878.log.gz 2024-04-24T08:22:18.223 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43039.log 2024-04-24T08:22:18.225 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50565.log 2024-04-24T08:22:18.225 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49699.log.gz 2024-04-24T08:22:18.226 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43039.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39250.log 2024-04-24T08:22:18.226 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43039.log.gz 2024-04-24T08:22:18.227 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50565.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50565.log.gz 2024-04-24T08:22:18.229 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39250.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39250.log.gz 2024-04-24T08:22:18.229 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70532.log 2024-04-24T08:22:18.230 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80091.log 2024-04-24T08:22:18.232 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43081.log 2024-04-24T08:22:18.232 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70532.log.gz 2024-04-24T08:22:18.233 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80091.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80091.log.gz 2024-04-24T08:22:18.234 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69425.log 2024-04-24T08:22:18.235 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72015.log 2024-04-24T08:22:18.235 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43081.log.gz 2024-04-24T08:22:18.236 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69425.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.69425.log.gz 2024-04-24T08:22:18.238 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72015.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.72015.log.gz 2024-04-24T08:22:18.239 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73018.log 2024-04-24T08:22:18.240 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71844.log 2024-04-24T08:22:18.242 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38943.log 2024-04-24T08:22:18.242 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73018.log.gz 2024-04-24T08:22:18.242 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71844.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71844.log.gz 2024-04-24T08:22:18.244 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38943.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38943.log.gz 2024-04-24T08:22:18.244 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82005.log 2024-04-24T08:22:18.246 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46864.log 2024-04-24T08:22:18.248 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.82005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49993.log 2024-04-24T08:22:18.248 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82005.log.gz 2024-04-24T08:22:18.248 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46864.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46864.log.gz 2024-04-24T08:22:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49993.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49993.log.gz 2024-04-24T08:22:18.250 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53264.log 2024-04-24T08:22:18.251 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75801.log 2024-04-24T08:22:18.253 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80538.log 2024-04-24T08:22:18.254 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53264.log.gz 2024-04-24T08:22:18.254 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75801.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75801.log.gz 2024-04-24T08:22:18.256 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80538.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80538.log.gz 2024-04-24T08:22:18.256 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42409.log 2024-04-24T08:22:18.257 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61697.log 2024-04-24T08:22:18.259 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62774.log 2024-04-24T08:22:18.259 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42409.log.gz 2024-04-24T08:22:18.260 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61697.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61697.log.gz -5 2024-04-24T08:22:18.261 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56873.log 2024-04-24T08:22:18.262 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46548.log 2024-04-24T08:22:18.262 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.62774.log.gz 2024-04-24T08:22:18.263 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44968.log 2024-04-24T08:22:18.263 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56873.log.gz 2024-04-24T08:22:18.264 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46548.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46548.log.gz 2024-04-24T08:22:18.265 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44842.log 2024-04-24T08:22:18.265 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44968.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44968.log.gz 2024-04-24T08:22:18.266 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72786.log 2024-04-24T08:22:18.267 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43654.log 2024-04-24T08:22:18.267 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44842.log.gz 2024-04-24T08:22:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72786.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72786.log.gz 2024-04-24T08:22:18.269 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65427.log 2024-04-24T08:22:18.270 INFO:teuthology.orchestra.run.smithi069.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43654.log.gz 2024-04-24T08:22:18.270 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39376.log 2024-04-24T08:22:18.272 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78134.log 2024-04-24T08:22:18.273 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0%/var/log/ceph/ceph-client.admin.39376.log: -- replaced with /var/log/ceph/ceph-client.admin.65427.log.gz 2024-04-24T08:22:18.274 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39376.log.gz 2024-04-24T08:22:18.274 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42787.log 2024-04-24T08:22:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78134.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.78134.log.gz 2024-04-24T08:22:18.275 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81482.log 2024-04-24T08:22:18.277 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45262.log 2024-04-24T08:22:18.277 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42787.log.gz 2024-04-24T08:22:18.278 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81440.log 2024-04-24T08:22:18.278 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81482.log.gz 2024-04-24T08:22:18.279 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47116.log 2024-04-24T08:22:18.279 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45262.log.gz 2024-04-24T08:22:18.281 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81440.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81440.log.gz 2024-04-24T08:22:18.282 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.47116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67704.log 2024-04-24T08:22:18.283 INFO:teuthology.orchestra.run.smithi069.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.47116.log.gz 2024-04-24T08:22:18.283 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56514.log 2024-04-24T08:22:18.285 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60979.log 2024-04-24T08:22:18.285 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67704.log.gz 2024-04-24T08:22:18.287 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56514.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56514.log.gz 2024-04-24T08:22:18.287 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44884.log 2024-04-24T08:22:18.287 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60979.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60979.log.gz 2024-04-24T08:22:18.288 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64322.log 2024-04-24T08:22:18.289 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57821.log 2024-04-24T08:22:18.289 INFO:teuthology.orchestra.run.smithi069.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44884.log.gz 2024-04-24T08:22:18.291 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64322.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64322.log.gz 2024-04-24T08:22:18.291 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77137.log 2024-04-24T08:22:18.292 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57821.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57821.log.gz 2024-04-24T08:22:18.292 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61510.log 2024-04-24T08:22:18.294 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.77137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61174.log 2024-04-24T08:22:18.294 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77137.log.gz 2024-04-24T08:22:18.295 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62104.log 2024-04-24T08:22:18.296 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61510.log.gz 2024-04-24T08:22:18.296 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61174.log: gzip -5 88.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.61174.log.gz -- 2024-04-24T08:22:18.297 INFO:teuthology.orchestra.run.smithi069.stderr: /var/log/ceph/ceph-client.admin.42577.log 2024-04-24T08:22:18.298 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53213.log 2024-04-24T08:22:18.298 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62104.log.gz 2024-04-24T08:22:18.299 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41056.log 2024-04-24T08:22:18.299 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42577.log.gz 2024-04-24T08:22:18.301 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53213.log: gzip -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53213.log.gz 2024-04-24T08:22:18.301 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43858.log 2024-04-24T08:22:18.302 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41056.log: gzip 89.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.41056.log.gz --verbose 2024-04-24T08:22:18.302 INFO:teuthology.orchestra.run.smithi069.stderr: -- /var/log/ceph/ceph-client.admin.80286.log 2024-04-24T08:22:18.303 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70411.log 2024-04-24T08:22:18.304 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43858.log.gz 2024-04-24T08:22:18.305 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80286.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80286.log.gz 2024-04-24T08:22:18.305 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46115.log 2024-04-24T08:22:18.306 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70411.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70411.log.gz 2024-04-24T08:22:18.306 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55692.log 2024-04-24T08:22:18.308 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.70951.log 2024-04-24T08:22:18.308 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46115.log.gz 2024-04-24T08:22:18.309 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55692.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55692.log.gz 2024-04-24T08:22:18.309 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61063.log 2024-04-24T08:22:18.311 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.70951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52532.log 2024-04-24T08:22:18.312 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61063.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61063.log.gz 2024-04-24T08:22:18.313 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44680.log 2024-04-24T08:22:18.313 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52532.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52532.log.gz 2024-04-24T08:22:18.314 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53169.log 2024-04-24T08:22:18.315 INFO:teuthology.orchestra.run.smithi121.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-24T08:22:18.316 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44680.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44680.log.gz 2024-04-24T08:22:18.316 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45946.log 2024-04-24T08:22:18.318 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53169.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53169.log.gz 2024-04-24T08:22:18.319 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59165.log 2024-04-24T08:22:18.320 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45946.log.gz 2024-04-24T08:22:18.320 INFO:teuthology.orchestra.run.smithi069.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.0.70951.log.gz 2024-04-24T08:22:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46367.log 2024-04-24T08:22:18.321 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59165.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59165.log.gz 2024-04-24T08:22:18.322 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62606.log 2024-04-24T08:22:18.323 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40261.log 2024-04-24T08:22:18.323 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46367.log.gz 2024-04-24T08:22:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62606.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62606.log.gz 2024-04-24T08:22:18.325 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40261.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40261.log.gz 2024-04-24T08:22:18.326 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62350.log 2024-04-24T08:22:18.327 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49272.log 2024-04-24T08:22:18.329 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55860.log 2024-04-24T08:22:18.329 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62350.log.gz 2024-04-24T08:22:18.330 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49272.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49272.log.gz 2024-04-24T08:22:18.330 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67509.log 2024-04-24T08:22:18.332 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55860.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55860.log.gz 2024-04-24T08:22:18.332 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67346.log 2024-04-24T08:22:18.333 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80706.log 2024-04-24T08:22:18.333 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67509.log.gz 2024-04-24T08:22:18.335 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67346.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67346.log.gz 2024-04-24T08:22:18.336 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.80706.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.65927.log 2024-04-24T08:22:18.336 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80706.log.gz 2024-04-24T08:22:18.336 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39598.log 2024-04-24T08:22:18.338 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67830.log 2024-04-24T08:22:18.338 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65927.log.gz 2024-04-24T08:22:18.339 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39598.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39598.log.gz 2024-04-24T08:22:18.340 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63920.log 2024-04-24T08:22:18.340 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67830.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67830.log.gz 2024-04-24T08:22:18.341 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51665.log 2024-04-24T08:22:18.342 INFO:teuthology.orchestra.run.smithi121.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-24T08:22:18.342 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63920.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63920.log.gz 2024-04-24T08:22:18.343 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55608.log 2024-04-24T08:22:18.344 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71843.log 2024-04-24T08:22:18.344 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51665.log.gz 2024-04-24T08:22:18.346 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60237.log 2024-04-24T08:22:18.346 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55608.log.gz 2024-04-24T08:22:18.348 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39935.log 2024-04-24T08:22:18.348 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60237.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71843.log.gz 2024-04-24T08:22:18.348 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60237.log.gz 2024-04-24T08:22:18.348 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74500.log 2024-04-24T08:22:18.350 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73734.log 2024-04-24T08:22:18.350 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39935.log.gz 2024-04-24T08:22:18.351 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74500.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74500.log.gz 2024-04-24T08:22:18.351 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74246.log 2024-04-24T08:22:18.353 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49127.log 2024-04-24T08:22:18.353 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73734.log.gz 2024-04-24T08:22:18.354 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64105.log 2024-04-24T08:22:18.354 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74246.log.gz 2024-04-24T08:22:18.355 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62816.log 2024-04-24T08:22:18.356 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49127.log.gz 2024-04-24T08:22:18.357 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56641.log 2024-04-24T08:22:18.357 INFO:teuthology.orchestra.run.smithi069.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.64105.log.gz 2024-04-24T08:22:18.358 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77455.log 2024-04-24T08:22:18.359 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62816.log.gz 2024-04-24T08:22:18.359 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56641.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56641.log.gz 2024-04-24T08:22:18.360 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46762.log 2024-04-24T08:22:18.362 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.77455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53436.log 2024-04-24T08:22:18.362 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.77455.log.gz 2024-04-24T08:22:18.362 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46762.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46762.log.gz 2024-04-24T08:22:18.363 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65659.log 2024-04-24T08:22:18.364 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53310.log 2024-04-24T08:22:18.364 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53436.log.gz 2024-04-24T08:22:18.365 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65659.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65659.log.gz 2024-04-24T08:22:18.366 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73608.log 2024-04-24T08:22:18.367 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52875.log 2024-04-24T08:22:18.367 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53310.log.gz 2024-04-24T08:22:18.368 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68819.log 2024-04-24T08:22:18.368 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.73608.log.gz 2024-04-24T08:22:18.369 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52875.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52875.log.gz 2024-04-24T08:22:18.371 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68819.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68819.log.gz 2024-04-24T08:22:18.371 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49187.log 2024-04-24T08:22:18.372 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78681.log 2024-04-24T08:22:18.374 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62942.log 2024-04-24T08:22:18.374 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49187.log.gz 2024-04-24T08:22:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78681.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.78681.log.gz 2024-04-24T08:22:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62184.log 2024-04-24T08:22:18.377 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75969.log 2024-04-24T08:22:18.377 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62942.log.gz 2024-04-24T08:22:18.378 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81902.log 2024-04-24T08:22:18.378 INFO:teuthology.orchestra.run.smithi069.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62184.log.gz 2024-04-24T08:22:18.380 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68395.log 2024-04-24T08:22:18.380 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75969.log.gz 2024-04-24T08:22:18.380 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81902.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81902.log.gz 2024-04-24T08:22:18.381 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53478.log 2024-04-24T08:22:18.382 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68395.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68395.log.gz 2024-04-24T08:22:18.382 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69593.log 2024-04-24T08:22:18.383 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39597.log 2024-04-24T08:22:18.384 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53478.log.gz 2024-04-24T08:22:18.385 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69593.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69593.log.gz 2024-04-24T08:22:18.385 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56727.log 2024-04-24T08:22:18.386 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39597.log.gz 2024-04-24T08:22:18.386 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64227.log 2024-04-24T08:22:18.388 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74896.log 2024-04-24T08:22:18.388 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56727.log.gz 2024-04-24T08:22:18.389 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55329.log 2024-04-24T08:22:18.389 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64227.log.gz 2024-04-24T08:22:18.391 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54458.log 2024-04-24T08:22:18.391 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74896.log.gz 2024-04-24T08:22:18.392 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66151.log 2024-04-24T08:22:18.392 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55329.log.gz 2024-04-24T08:22:18.394 INFO:teuthology.orchestra.run.smithi121.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-24T08:22:18.394 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54458.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.54458.log.gz 2024-04-24T08:22:18.394 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64410.log 2024-04-24T08:22:18.395 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.66151.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66151.log.gz 2024-04-24T08:22:18.395 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58555.log 2024-04-24T08:22:18.397 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66587.log 2024-04-24T08:22:18.397 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1%/var/log/ceph/ceph-client.admin.58555.log: -- replaced with /var/log/ceph/ceph-client.admin.64410.log.gz 2024-04-24T08:22:18.398 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58555.log.gz 2024-04-24T08:22:18.398 INFO:teuthology.orchestra.run.smithi069.stderr: -- /var/log/ceph/ceph-client.admin.78774.log 2024-04-24T08:22:18.400 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.66587.log: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73270.log 2024-04-24T08:22:18.400 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66587.log.gz 2024-04-24T08:22:18.401 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40804.log 2024-04-24T08:22:18.401 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.78774.log.gz 2024-04-24T08:22:18.403 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73270.log: gzip -5 --verbose 89.4% -- replaced with /var/log/ceph/ceph-client.admin.73270.log.gz 2024-04-24T08:22:18.403 INFO:teuthology.orchestra.run.smithi069.stderr: -- /var/log/ceph/ceph-client.admin.48416.log 2024-04-24T08:22:18.403 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40804.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40804.log.gz 2024-04-24T08:22:18.404 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80049.log 2024-04-24T08:22:18.405 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69127.log 2024-04-24T08:22:18.405 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48416.log.gz 2024-04-24T08:22:18.406 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80049.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80049.log.gz 2024-04-24T08:22:18.407 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76840.log 2024-04-24T08:22:18.407 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69127.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.69127.log.gz 2024-04-24T08:22:18.408 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54240.log 2024-04-24T08:22:18.409 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.76840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68777.log 2024-04-24T08:22:18.410 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76840.log.gz 2024-04-24T08:22:18.411 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54240.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54240.log.gz 2024-04-24T08:22:18.411 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68946.log 2024-04-24T08:22:18.412 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68777.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68777.log.gz 2024-04-24T08:22:18.412 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49524.log 2024-04-24T08:22:18.414 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2024-04-24T08:22:18.414 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68946.log.gz 2024-04-24T08:22:18.415 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73396.log 2024-04-24T08:22:18.416 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49524.log.gz 2024-04-24T08:22:18.416 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40594.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz 2024-04-24T08:22:18.418 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73396.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73396.log.gz 2024-04-24T08:22:18.418 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40846.log 2024-04-24T08:22:18.419 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38467.log 2024-04-24T08:22:18.421 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61384.log 2024-04-24T08:22:18.421 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40846.log.gz 2024-04-24T08:22:18.422 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38467.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.38467.log.gz -5 2024-04-24T08:22:18.422 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.58131.log 2024-04-24T08:22:18.424 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42493.log 2024-04-24T08:22:18.424 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61384.log.gz 2024-04-24T08:22:18.425 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58131.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58131.log.gz 2024-04-24T08:22:18.426 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.72549.log 2024-04-24T08:22:18.426 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42493.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42493.log.gz 2024-04-24T08:22:18.427 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48791.log 2024-04-24T08:22:18.428 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56999.log 2024-04-24T08:22:18.428 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.72549.log.gz 2024-04-24T08:22:18.429 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48791.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48791.log.gz 2024-04-24T08:22:18.430 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5/var/log/ceph/ceph-client.admin.56999.log: --verbose -- /var/log/ceph/ceph-client.admin.72720.log 2024-04-24T08:22:18.431 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.56999.log.gz 2024-04-24T08:22:18.431 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47862.log 2024-04-24T08:22:18.433 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47946.log 2024-04-24T08:22:18.434 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.47862.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.72720.log.gz 2024-04-24T08:22:18.434 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74458.log 2024-04-24T08:22:18.434 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47862.log.gz 2024-04-24T08:22:18.435 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.47946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74118.log 2024-04-24T08:22:18.435 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47946.log.gz 2024-04-24T08:22:18.437 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58173.log 2024-04-24T08:22:18.437 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74458.log.gz 2024-04-24T08:22:18.439 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74118.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74118.log.gz 2024-04-24T08:22:18.439 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43386.log 2024-04-24T08:22:18.439 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58173.log.gz 2024-04-24T08:22:18.440 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49482.log 2024-04-24T08:22:18.441 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51341.log 2024-04-24T08:22:18.441 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43386.log.gz 2024-04-24T08:22:18.443 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49482.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49482.log.gz 2024-04-24T08:22:18.443 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73354.log 2024-04-24T08:22:18.444 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33330.log 2024-04-24T08:22:18.444 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.51341.log.gz 2024-04-24T08:22:18.446 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57167.log 2024-04-24T08:22:18.446 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73354.log.gz 2024-04-24T08:22:18.447 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.33330.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33330.log.gz 2024-04-24T08:22:18.449 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57167.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57167.log.gz 2024-04-24T08:22:18.449 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72975.log 2024-04-24T08:22:18.450 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57989.log 2024-04-24T08:22:18.451 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44638.log 2024-04-24T08:22:18.452 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.72975.log.gz 2024-04-24T08:22:18.453 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57989.log: 88.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40678.log 2024-04-24T08:22:18.453 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57989.log.gz 2024-04-24T08:22:18.454 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42703.log 2024-04-24T08:22:18.454 INFO:teuthology.orchestra.run.smithi069.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44638.log.gz 2024-04-24T08:22:18.456 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40678.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40678.log.gz 2024-04-24T08:22:18.456 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68269.log 2024-04-24T08:22:18.457 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42703.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42703.log.gz 2024-04-24T08:22:18.458 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40265.log 2024-04-24T08:22:18.459 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69485.log 2024-04-24T08:22:18.459 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68269.log.gz 2024-04-24T08:22:18.461 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40265.log: gzip -5 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40265.log.gz 2024-04-24T08:22:18.461 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.65969.log 2024-04-24T08:22:18.461 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69485.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69485.log.gz 2024-04-24T08:22:18.462 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73523.log 2024-04-24T08:22:18.463 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74202.log 2024-04-24T08:22:18.463 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65969.log.gz 2024-04-24T08:22:18.464 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73523.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73523.log.gz 2024-04-24T08:22:18.465 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.74202.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46678.log 2024-04-24T08:22:18.466 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74202.log.gz 2024-04-24T08:22:18.466 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81398.log 2024-04-24T08:22:18.468 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69299.log 2024-04-24T08:22:18.468 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46678.log.gz 2024-04-24T08:22:18.469 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81398.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81398.log.gz 2024-04-24T08:22:18.470 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38729.log 2024-04-24T08:22:18.470 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69299.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69299.log.gz 2024-04-24T08:22:18.471 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64976.log 2024-04-24T08:22:18.472 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45388.log 2024-04-24T08:22:18.472 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38729.log.gz 2024-04-24T08:22:18.473 INFO:teuthology.orchestra.run.smithi121.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-24T08:22:18.474 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64976.log: gzip 90.9% -5 --verbose -- /var/log/ceph/ceph-client.admin.39040.log 2024-04-24T08:22:18.474 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64976.log.gz/var/log/ceph/ceph-client.admin.45388.log: 2024-04-24T08:22:18.475 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45388.log.gz 2024-04-24T08:22:18.475 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44454.log 2024-04-24T08:22:18.477 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42913.log 2024-04-24T08:22:18.477 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39040.log.gz 2024-04-24T08:22:18.478 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47460.log 2024-04-24T08:22:18.479 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44454.log.gz 2024-04-24T08:22:18.480 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42913.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42913.log.gz 2024-04-24T08:22:18.480 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45865.log 2024-04-24T08:22:18.482 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.47460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38644.log 2024-04-24T08:22:18.483 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45865.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45865.log.gz 2024-04-24T08:22:18.483 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39593.log 2024-04-24T08:22:18.485 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33065.log 2024-04-24T08:22:18.485 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38644.log.gz 2024-04-24T08:22:18.487 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39593.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39593.log.gz 2024-04-24T08:22:18.487 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.33065.log: 93.0% -- replaced with /var/log/ceph/ceph-client.0.47460.log.gz 2024-04-24T08:22:18.488 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33065.log.gz 2024-04-24T08:22:18.488 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51253.log 2024-04-24T08:22:18.489 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43291.log 2024-04-24T08:22:18.491 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78318.log 2024-04-24T08:22:18.492 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0%/var/log/ceph/ceph-client.admin.43291.log: -- replaced with /var/log/ceph/ceph-client.admin.51253.log.gz 2024-04-24T08:22:18.492 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43291.log.gz 2024-04-24T08:22:18.493 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62648.log 2024-04-24T08:22:18.493 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78318.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78318.log.gz 2024-04-24T08:22:18.494 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42619.log 2024-04-24T08:22:18.495 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52448.log 2024-04-24T08:22:18.495 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62648.log.gz 2024-04-24T08:22:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42619.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42619.log.gz 2024-04-24T08:22:18.497 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66011.log 2024-04-24T08:22:18.498 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72464.log 2024-04-24T08:22:18.500 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8%/var/log/ceph/ceph-client.admin.66011.log: -- replaced with /var/log/ceph/ceph-client.admin.52448.log.gz 2024-04-24T08:22:18.500 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66011.log.gz 2024-04-24T08:22:18.500 INFO:teuthology.orchestra.run.smithi069.stderr: /var/log/ceph/ceph-client.admin.57905.log 2024-04-24T08:22:18.501 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61426.log 2024-04-24T08:22:18.501 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.72464.log.gz 2024-04-24T08:22:18.503 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50855.log 2024-04-24T08:22:18.503 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57905.log.gz 2024-04-24T08:22:18.504 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62732.log 2024-04-24T08:22:18.504 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61426.log.gz 2024-04-24T08:22:18.505 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.50855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45304.log 2024-04-24T08:22:18.507 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62732.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62732.log.gz 2024-04-24T08:22:18.508 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46157.log/var/log/ceph/ceph-client.admin.45304.log: 2024-04-24T08:22:18.508 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% 92.8% -- replaced with /var/log/ceph/ceph-client.0.50855.log.gz 2024-04-24T08:22:18.508 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45304.log.gz 2024-04-24T08:22:18.509 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52956.log 2024-04-24T08:22:18.510 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46157.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46157.log.gz 2024-04-24T08:22:18.511 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64160.log 2024-04-24T08:22:18.511 INFO:teuthology.orchestra.run.smithi069.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-24T08:22:18.512 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52956.log: /var/log/ceph/ceph-client.admin.64160.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52956.log.gz 2024-04-24T08:22:18.513 INFO:teuthology.orchestra.run.smithi069.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.64160.log.gz 2024-04-24T08:22:18.513 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65843.log 2024-04-24T08:22:18.514 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74584.log 2024-04-24T08:22:18.515 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65843.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65843.log.gz 2024-04-24T08:22:18.515 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45988.log 2024-04-24T08:22:18.516 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39929.log 2024-04-24T08:22:18.516 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74584.log.gz 2024-04-24T08:22:18.517 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72104.log 2024-04-24T08:22:18.518 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45988.log.gz 2024-04-24T08:22:18.518 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68227.log 2024-04-24T08:22:18.519 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39929.log.gz 2024-04-24T08:22:18.519 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45700.log 2024-04-24T08:22:18.520 INFO:teuthology.orchestra.run.smithi069.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.72104.log.gz 2024-04-24T08:22:18.521 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52238.log 2024-04-24T08:22:18.521 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68227.log.gz 2024-04-24T08:22:18.522 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54974.log 2024-04-24T08:22:18.522 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45700.log.gz 2024-04-24T08:22:18.523 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51491.log 2024-04-24T08:22:18.523 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52238.log.gz 2024-04-24T08:22:18.524 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38855.log 2024-04-24T08:22:18.524 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54974.log.gz 2024-04-24T08:22:18.525 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59291.log 2024-04-24T08:22:18.525 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51491.log.gz 2024-04-24T08:22:18.526 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38855.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38855.log.gz 2024-04-24T08:22:18.527 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75659.log 2024-04-24T08:22:18.527 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72548.log 2024-04-24T08:22:18.528 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59291.log.gz 2024-04-24T08:22:18.529 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80935.log 2024-04-24T08:22:18.529 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75659.log.gz 2024-04-24T08:22:18.530 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77413.log 2024-04-24T08:22:18.530 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.72548.log.gz 2024-04-24T08:22:18.531 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44352.log 2024-04-24T08:22:18.531 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80935.log.gz 2024-04-24T08:22:18.533 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.77413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54336.log 2024-04-24T08:22:18.533 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77413.log.gz 2024-04-24T08:22:18.533 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44352.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44352.log.gz 2024-04-24T08:22:18.534 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72809.log 2024-04-24T08:22:18.535 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46590.log 2024-04-24T08:22:18.535 INFO:teuthology.orchestra.run.smithi069.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.54336.log.gz 2024-04-24T08:22:18.536 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44110.log 2024-04-24T08:22:18.536 INFO:teuthology.orchestra.run.smithi069.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.72809.log.gz 2024-04-24T08:22:18.537 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76391.log 2024-04-24T08:22:18.537 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46590.log.gz 2024-04-24T08:22:18.538 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44110.log: gzip -5 --verbose 89.5% -- -- replaced with /var/log/ceph/ceph-client.admin.44110.log.gz /var/log/ceph/ceph-client.admin.63610.log 2024-04-24T08:22:18.538 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-24T08:22:18.539 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.76391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77367.log 2024-04-24T08:22:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63610.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63610.log.gz 2024-04-24T08:22:18.540 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44800.log 2024-04-24T08:22:18.542 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.77367.log: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26687.log 2024-04-24T08:22:18.542 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77367.log.gz 2024-04-24T08:22:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44800.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44800.log.gz 2024-04-24T08:22:18.543 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73102.log 2024-04-24T08:22:18.544 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph.tmp-client.admin.26687.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26687.log.gz 2024-04-24T08:22:18.544 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43696.log 2024-04-24T08:22:18.545 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59333.log 2024-04-24T08:22:18.546 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73102.log: 94.1% -- replaced with /var/log/ceph/ceph-client.0.76391.log.gz 2024-04-24T08:22:18.546 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73102.log.gz 2024-04-24T08:22:18.547 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43696.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43696.log.gz 2024-04-24T08:22:18.547 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39922.log 2024-04-24T08:22:18.548 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59333.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59333.log.gz 2024-04-24T08:22:18.548 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74854.log 2024-04-24T08:22:18.549 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.39922.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53352.log 2024-04-24T08:22:18.550 INFO:teuthology.orchestra.run.smithi069.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39922.log.gz 2024-04-24T08:22:18.550 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74854.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74854.log.gz 2024-04-24T08:22:18.551 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39082.log 2024-04-24T08:22:18.552 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54908.log 2024-04-24T08:22:18.553 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39082.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53352.log.gz 2024-04-24T08:22:18.553 INFO:teuthology.orchestra.run.smithi069.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39082.log.gz 2024-04-24T08:22:18.553 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48130.log 2024-04-24T08:22:18.554 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81134.log 2024-04-24T08:22:18.555 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48130.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.54908.log.gz 2024-04-24T08:22:18.555 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48130.log.gz 2024-04-24T08:22:18.555 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58941.log 2024-04-24T08:22:18.556 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55016.log 2024-04-24T08:22:18.557 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.58941.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81134.log.gz 2024-04-24T08:22:18.557 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39166.log 2024-04-24T08:22:18.559 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61594.log 2024-04-24T08:22:18.559 INFO:teuthology.orchestra.run.smithi069.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.58941.log.gz 2024-04-24T08:22:18.559 INFO:teuthology.orchestra.run.smithi069.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.55016.log.gz 2024-04-24T08:22:18.560 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56258.log 2024-04-24T08:22:18.560 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39166.log.gz 2024-04-24T08:22:18.561 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63794.log 2024-04-24T08:22:18.562 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61594.log.gz 2024-04-24T08:22:18.562 INFO:teuthology.orchestra.run.smithi121.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-24T08:22:18.563 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62984.log 2024-04-24T08:22:18.563 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56258.log.gz 2024-04-24T08:22:18.563 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63794.log: gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.63794.log.gz 2024-04-24T08:22:18.564 INFO:teuthology.orchestra.run.smithi069.stderr: /var/log/ceph/ceph-client.admin.65061.log 2024-04-24T08:22:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2024-04-24T08:22:18.565 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65061.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.62984.log.gz 2024-04-24T08:22:18.566 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.79676.log 2024-04-24T08:22:18.566 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.65061.log.gz 2024-04-24T08:22:18.567 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-04-24T08:22:18.567 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m0.871s 2024-04-24T08:22:18.567 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m3.781s 2024-04-24T08:22:18.567 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m0.441s 2024-04-24T08:22:18.568 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65743.log 2024-04-24T08:22:18.568 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz 2024-04-24T08:22:18.570 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.0.79676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49619.log 2024-04-24T08:22:18.570 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57209.log 2024-04-24T08:22:18.570 INFO:teuthology.orchestra.run.smithi069.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.65743.log.gz 2024-04-24T08:22:18.571 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52112.log 2024-04-24T08:22:18.571 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49619.log.gz 2024-04-24T08:22:18.572 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.57209.log: 92.9% -- replaced with /var/log/ceph/ceph-client.0.79676.log.gz 2024-04-24T08:22:18.572 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57209.log.gz 2024-04-24T08:22:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70617.log 2024-04-24T08:22:18.573 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52112.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52112.log.gz 2024-04-24T08:22:18.574 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39599.log 2024-04-24T08:22:18.575 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39600.log 2024-04-24T08:22:18.575 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70617.log: /var/log/ceph/ceph-client.admin.39599.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.70617.log.gz 2024-04-24T08:22:18.576 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69215.log 2024-04-24T08:22:18.576 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39599.log.gz 2024-04-24T08:22:18.577 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74626.log 2024-04-24T08:22:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69215.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39600.log.gz 2024-04-24T08:22:18.578 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69215.log.gz 2024-04-24T08:22:18.578 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56433.log 2024-04-24T08:22:18.579 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61654.log 2024-04-24T08:22:18.579 INFO:teuthology.orchestra.run.smithi069.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74626.log.gz 2024-04-24T08:22:18.580 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56433.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80007.log 2024-04-24T08:22:18.580 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56433.log.gz 2024-04-24T08:22:18.581 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51707.log 2024-04-24T08:22:18.582 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61654.log.gz 2024-04-24T08:22:18.583 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80007.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80007.log.gz 2024-04-24T08:22:18.583 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65885.log 2024-04-24T08:22:18.583 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.51707.log: /var/log/ceph/ceph-client.admin.43249.log 2024-04-24T08:22:18.584 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65885.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51707.log.gz 2024-04-24T08:22:18.585 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52154.log 2024-04-24T08:22:18.585 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65885.log.gz 2024-04-24T08:22:18.586 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63694.log 2024-04-24T08:22:18.586 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43249.log.gz 2024-04-24T08:22:18.587 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65277.log 2024-04-24T08:22:18.587 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52154.log.gz 2024-04-24T08:22:18.588 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63694.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63694.log.gz 2024-04-24T08:22:18.588 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65701.log 2024-04-24T08:22:18.589 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78971.log 2024-04-24T08:22:18.591 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65701.log: gzip 90.8% -- replaced with /var/log/ceph/ceph-client.admin.65277.log.gz -5 2024-04-24T08:22:18.591 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48564.log 2024-04-24T08:22:18.591 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65701.log.gz 2024-04-24T08:22:18.592 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.78971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51179.log 2024-04-24T08:22:18.593 INFO:teuthology.orchestra.run.smithi069.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.78971.log.gz 2024-04-24T08:22:18.593 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.48564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76047.log 2024-04-24T08:22:18.593 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48564.log.gz 2024-04-24T08:22:18.594 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51179.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51179.log.gz 2024-04-24T08:22:18.594 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-24T08:22:18.595 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74416.log 2024-04-24T08:22:18.596 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.76047.log: /var/log/ceph/ceph-mds.a.log: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73312.log 2024-04-24T08:22:18.597 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.76047.log.gz 2024-04-24T08:22:18.598 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46199.log 2024-04-24T08:22:18.598 INFO:teuthology.orchestra.run.smithi069.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74416.log.gz 2024-04-24T08:22:18.599 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73312.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.73312.log.gz 2024-04-24T08:22:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40468.log 2024-04-24T08:22:18.600 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46199.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46199.log.gz 2024-04-24T08:22:18.601 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58215.log 2024-04-24T08:22:18.602 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60895.log 2024-04-24T08:22:18.603 INFO:teuthology.orchestra.run.smithi069.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40468.log.gz 2024-04-24T08:22:18.604 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58215.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.58215.log.gz 2024-04-24T08:22:18.604 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79184.log 2024-04-24T08:22:18.605 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60895.log.gz 2024-04-24T08:22:18.605 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81272.log 2024-04-24T08:22:18.607 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.79184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53538.log 2024-04-24T08:22:18.607 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79184.log.gz 2024-04-24T08:22:18.609 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81272.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81272.log.gz 2024-04-24T08:22:18.609 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40972.log 2024-04-24T08:22:18.609 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53538.log.gz 2024-04-24T08:22:18.610 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55566.log 2024-04-24T08:22:18.612 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80851.log 2024-04-24T08:22:18.612 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40972.log.gz 2024-04-24T08:22:18.613 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55566.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55566.log.gz 2024-04-24T08:22:18.614 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55963.log 2024-04-24T08:22:18.614 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80851.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80851.log.gz 2024-04-24T08:22:18.615 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51425.log 2024-04-24T08:22:18.616 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55963.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.55963.log.gz 2024-04-24T08:22:18.616 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43780.log 2024-04-24T08:22:18.618 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67425.log 2024-04-24T08:22:18.618 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.51425.log.gz 2024-04-24T08:22:18.619 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75927.log 2024-04-24T08:22:18.620 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43780.log.gz 2024-04-24T08:22:18.620 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63050.log 2024-04-24T08:22:18.620 INFO:teuthology.orchestra.run.smithi069.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67425.log.gz 2024-04-24T08:22:18.622 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67788.log 2024-04-24T08:22:18.622 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75927.log.gz 2024-04-24T08:22:18.623 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40271.log 2024-04-24T08:22:18.623 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63050.log.gz 2024-04-24T08:22:18.625 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67788.log: gzip -5 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67788.log.gz 2024-04-24T08:22:18.625 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55776.log 2024-04-24T08:22:18.626 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61258.log 2024-04-24T08:22:18.626 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40271.log.gz 2024-04-24T08:22:18.628 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55203.log 2024-04-24T08:22:18.628 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55776.log.gz 2024-04-24T08:22:18.629 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.61258.log: gzip -5 --verbose -- 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61258.log.gz 2024-04-24T08:22:18.629 INFO:teuthology.orchestra.run.smithi069.stderr: /var/log/ceph/ceph-client.admin.71652.log 2024-04-24T08:22:18.630 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55203.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55203.log.gz 2024-04-24T08:22:18.630 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59792.log 2024-04-24T08:22:18.632 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52322.log 2024-04-24T08:22:18.632 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71652.log.gz 2024-04-24T08:22:18.634 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59792.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.59792.log.gz/var/log/ceph/ceph-client.admin.52322.log: 2024-04-24T08:22:18.635 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52322.log.gz 2024-04-24T08:22:18.635 INFO:teuthology.orchestra.run.smithi069.stderr: /var/log/ceph/ceph-client.admin.52795.log 2024-04-24T08:22:18.636 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44310.log 2024-04-24T08:22:18.637 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52795.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52795.log.gz 2024-04-24T08:22:18.637 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80454.log 2024-04-24T08:22:18.639 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67467.log 2024-04-24T08:22:18.639 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44310.log.gz 2024-04-24T08:22:18.640 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80454.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.80454.log.gz 2024-04-24T08:22:18.640 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73872.log 2024-04-24T08:22:18.642 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67467.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67467.log.gz 2024-04-24T08:22:18.642 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41014.log 2024-04-24T08:22:18.643 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73872.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73872.log.gz -5 2024-04-24T08:22:18.643 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.59900.log 2024-04-24T08:22:18.645 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.41014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81776.log 2024-04-24T08:22:18.645 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41014.log.gz 2024-04-24T08:22:18.646 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.59900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50499.log 2024-04-24T08:22:18.646 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.59900.log.gz 2024-04-24T08:22:18.647 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.81776.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81776.log.gz 2024-04-24T08:22:18.648 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52658.log/var/log/ceph/ceph-client.admin.50499.log: 2024-04-24T08:22:18.649 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.50499.log.gz 2024-04-24T08:22:18.650 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49864.log 2024-04-24T08:22:18.651 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38686.log 2024-04-24T08:22:18.651 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52658.log.gz 2024-04-24T08:22:18.652 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49864.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49864.log.gz 2024-04-24T08:22:18.653 INFO:teuthology.orchestra.run.smithi069.stderr:gzip/var/log/ceph/ceph-client.admin.38686.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.76882.log 2024-04-24T08:22:18.653 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38686.log.gz 2024-04-24T08:22:18.654 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73952.log 2024-04-24T08:22:18.656 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.76882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45785.log 2024-04-24T08:22:18.656 INFO:teuthology.orchestra.run.smithi069.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.76882.log.gz 2024-04-24T08:22:18.657 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.73952.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73952.log.gz 2024-04-24T08:22:18.658 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67172.log 2024-04-24T08:22:18.658 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45785.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45785.log.gz 2024-04-24T08:22:18.659 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42535.log 2024-04-24T08:22:18.661 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76966.log 2024-04-24T08:22:18.661 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.67172.log.gz 2024-04-24T08:22:18.662 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.42535.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42535.log.gz -5 2024-04-24T08:22:18.662 INFO:teuthology.orchestra.run.smithi069.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53394.log 2024-04-24T08:22:18.663 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.76966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46906.log 2024-04-24T08:22:18.664 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.76966.log.gz 2024-04-24T08:22:18.665 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70327.log 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53394.log.gz 2024-04-24T08:22:18.665 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-24T08:22:18.666 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46906.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39930.log 2024-04-24T08:22:18.666 INFO:teuthology.orchestra.run.smithi069.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46906.log.gz 2024-04-24T08:22:18.668 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67914.log 2024-04-24T08:22:18.668 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70327.log.gz 2024-04-24T08:22:18.669 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39930.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39930.log.gz 2024-04-24T08:22:18.669 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68311.log 2024-04-24T08:22:18.671 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55818.log 2024-04-24T08:22:18.671 INFO:teuthology.orchestra.run.smithi069.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.67914.log.gz 2024-04-24T08:22:18.672 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68862.log 2024-04-24T08:22:18.672 INFO:teuthology.orchestra.run.smithi069.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68311.log.gz 2024-04-24T08:22:18.673 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68990.log 2024-04-24T08:22:18.673 INFO:teuthology.orchestra.run.smithi069.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.55818.log.gz 2024-04-24T08:22:18.675 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45010.log 2024-04-24T08:22:18.675 INFO:teuthology.orchestra.run.smithi069.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68862.log.gz 2024-04-24T08:22:18.676 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.68990.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68990.log.gz 2024-04-24T08:22:18.676 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43984.log 2024-04-24T08:22:18.677 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45010.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45010.log.gz 2024-04-24T08:22:18.678 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75701.log 2024-04-24T08:22:18.679 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.43984.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43984.log.gz 2024-04-24T08:22:18.679 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64082.log 2024-04-24T08:22:18.680 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75701.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75701.log.gz 2024-04-24T08:22:18.680 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50037.log 2024-04-24T08:22:18.682 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.64082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39589.log 2024-04-24T08:22:18.682 INFO:teuthology.orchestra.run.smithi069.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64082.log.gz 2024-04-24T08:22:18.683 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.50037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45220.log 2024-04-24T08:22:18.684 INFO:teuthology.orchestra.run.smithi069.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50037.log.gz 2024-04-24T08:22:18.684 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.39589.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39589.log.gz 2024-04-24T08:22:18.685 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56215.log 2024-04-24T08:22:18.685 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45220.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45220.log.gz 2024-04-24T08:22:18.686 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49314.log 2024-04-24T08:22:18.687 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72164.log 2024-04-24T08:22:18.688 INFO:teuthology.orchestra.run.smithi069.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56215.log.gz 2024-04-24T08:22:18.689 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49314.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49314.log.gz 2024-04-24T08:22:18.690 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5/var/log/ceph/ceph-client.admin.72164.log: --verbose -- /var/log/ceph/ceph-client.admin.54758.log 2024-04-24T08:22:18.690 INFO:teuthology.orchestra.run.smithi069.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.72164.log.gz 2024-04-24T08:22:18.691 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44722.log 2024-04-24T08:22:18.692 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.54758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49356.log 2024-04-24T08:22:18.693 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54758.log.gz 2024-04-24T08:22:18.693 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44722.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44722.log.gz 2024-04-24T08:22:18.694 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44926.log 2024-04-24T08:22:18.695 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.49356.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49356.log.gz 2024-04-24T08:22:18.695 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52406.log 2024-04-24T08:22:18.697 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72081.log 2024-04-24T08:22:18.697 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44926.log.gz 2024-04-24T08:22:18.698 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2024-04-24T08:22:18.698 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52406.log.gz 2024-04-24T08:22:18.699 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.72081.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72081.log.gz 2024-04-24T08:22:18.700 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40272.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gz 2024-04-24T08:22:18.701 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71384.log 2024-04-24T08:22:18.702 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44554.log 2024-04-24T08:22:18.704 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70673.log 2024-04-24T08:22:18.704 INFO:teuthology.orchestra.run.smithi069.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71384.log.gz 2024-04-24T08:22:18.705 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.44554.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44554.log.gz 2024-04-24T08:22:18.705 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46948.log 2024-04-24T08:22:18.706 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.70673.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70673.log.gz 2024-04-24T08:22:18.706 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80766.log 2024-04-24T08:22:18.708 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.46948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75743.log 2024-04-24T08:22:18.708 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46948.log.gz 2024-04-24T08:22:18.710 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80766.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.80766.log.gz 2024-04-24T08:22:18.710 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60074.log 2024-04-24T08:22:18.710 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.75743.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.75743.log.gz 2024-04-24T08:22:18.711 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56915.log 2024-04-24T08:22:18.712 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.60074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80133.log 2024-04-24T08:22:18.714 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.56915.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.60074.log.gz 2024-04-24T08:22:18.714 INFO:teuthology.orchestra.run.smithi069.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56915.log.gz 2024-04-24T08:22:18.714 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.80133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74374.log 2024-04-24T08:22:18.715 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80133.log.gz 2024-04-24T08:22:18.715 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45514.log 2024-04-24T08:22:18.718 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.74374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67022.log 2024-04-24T08:22:18.718 INFO:teuthology.orchestra.run.smithi069.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74374.log.gz 2024-04-24T08:22:18.718 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45472.log 2024-04-24T08:22:18.718 INFO:teuthology.orchestra.run.smithi069.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45514.log.gz 2024-04-24T08:22:18.719 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55245.log 2024-04-24T08:22:18.720 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.45472.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.67022.log.gz 2024-04-24T08:22:18.721 INFO:teuthology.orchestra.run.smithi069.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45472.log.gz 2024-04-24T08:22:18.721 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69257.log 2024-04-24T08:22:18.722 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.55245.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55245.log.gz 2024-04-24T08:22:18.722 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63652.log 2024-04-24T08:22:18.723 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.69257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67238.log 2024-04-24T08:22:18.724 INFO:teuthology.orchestra.run.smithi069.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69257.log.gz 2024-04-24T08:22:18.725 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.63652.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63652.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65535.log 2024-04-24T08:22:18.725 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-24T08:22:18.727 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.67238.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67238.log.gz 2024-04-24T08:22:18.727 INFO:teuthology.orchestra.run.smithi069.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.79290.log 2024-04-24T08:22:18.728 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.65535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53622.log 2024-04-24T08:22:18.728 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.65535.log.gz 2024-04-24T08:22:18.729 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.79290.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79290.log.gz 2024-04-24T08:22:18.730 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.53622.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53622.log.gz 2024-04-24T08:22:18.731 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51126.log 2024-04-24T08:22:18.732 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58597.log 2024-04-24T08:22:18.734 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.51126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52574.log 2024-04-24T08:22:18.734 INFO:teuthology.orchestra.run.smithi069.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51126.log.gz 2024-04-24T08:22:18.735 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.58597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58089.log 2024-04-24T08:22:18.735 INFO:teuthology.orchestra.run.smithi069.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.58597.log.gz 2024-04-24T08:22:18.736 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.52574.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52574.log.gz 2024-04-24T08:22:18.738 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62478.log/var/log/ceph/ceph-client.admin.58089.log: 2024-04-24T08:22:18.738 INFO:teuthology.orchestra.run.smithi069.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58089.log.gz 2024-04-24T08:22:18.739 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71258.log 2024-04-24T08:22:18.740 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.62478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40220.log 2024-04-24T08:22:18.740 INFO:teuthology.orchestra.run.smithi069.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62478.log.gz 2024-04-24T08:22:18.741 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.71258.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.71258.log.gz 2024-04-24T08:22:18.742 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.40220.log: 91.1% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-24T08:22:18.742 INFO:teuthology.orchestra.run.smithi069.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40220.log.gz 2024-04-24T08:22:18.743 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38368.log 2024-04-24T08:22:18.745 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/ceph-client.admin.38368.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38368.log.gz 2024-04-24T08:22:18.854 INFO:teuthology.orchestra.run.smithi069.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-24T08:22:18.889 INFO:teuthology.orchestra.run.smithi069.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-24T08:22:18.932 INFO:teuthology.orchestra.run.smithi069.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-24T08:22:19.077 INFO:teuthology.orchestra.run.smithi069.stderr: 90.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-24T08:22:19.081 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-04-24T08:22:19.081 INFO:teuthology.orchestra.run.smithi069.stderr:real 0m1.397s 2024-04-24T08:22:19.081 INFO:teuthology.orchestra.run.smithi069.stderr:user 0m4.544s 2024-04-24T08:22:19.081 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m2.386s 2024-04-24T08:22:19.082 INFO:tasks.ceph:Archiving logs... 2024-04-24T08:22:19.083 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210/remote/smithi069/log 2024-04-24T08:22:19.084 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T08:22:20.369 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210/remote/smithi121/log 2024-04-24T08:22:20.369 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T08:22:20.956 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T08:22:20.965 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T08:22:20.966 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T08:22:20.974 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T08:22:21.110 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T08:22:21.122 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T08:22:21.122 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:22:21.130 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:22:21.177 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:22:21.183 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:22:21.187 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:22:21.187 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-04-24T08:22:21.187 INFO:teuthology.orchestra.run.smithi069.stdout:^- hv01.front.sepia.ceph.com 2 7 377 8 +1259us[+1275us] +/- 578ms 2024-04-24T08:22:21.187 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -115us[ -100us] +/- 71ms 2024-04-24T08:22:21.187 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +130us[ +130us] +/- 73ms 2024-04-24T08:22:21.187 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:22:21.193 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:22:21.194 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-04-24T08:22:21.194 INFO:teuthology.orchestra.run.smithi121.stdout:^- hv01.front.sepia.ceph.com 2 6 377 7 +1254us[+1245us] +/- 578ms 2024-04-24T08:22:21.194 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -140us[ -149us] +/- 71ms 2024-04-24T08:22:21.194 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +138us[ +138us] +/- 74ms 2024-04-24T08:22:21.194 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:22:21.195 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T08:22:21.207 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T08:22:21.208 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T08:22:21.222 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T08:22:21.444 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T08:22:21.623 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:22:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:22:21.721 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:22:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:22:21.814 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T08:22:21.824 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T08:22:21.839 INFO:teuthology.task.internal:Duration was 1356.806177 seconds 2024-04-24T08:22:21.839 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T08:22:21.854 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T08:22:21.854 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T08:22:21.861 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T08:22:21.994 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:22:22.002 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:22:22.524 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T08:22:22.524 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-04-24T08:22:22.525 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T08:22:22.589 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-04-24T08:22:22.590 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T08:22:22.662 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T08:22:22.662 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T08:22:22.669 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T08:22:22.749 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T08:22:22.749 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T08:22:22.754 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T08:22:22.957 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T08:22:22.970 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T08:22:22.970 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T08:22:23.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T08:22:23.139 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T08:22:23.159 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:22:23.167 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:22:23.260 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-04-24T08:22:23.267 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-04-24T08:22:23.306 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:22:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:22:23.353 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:22:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:22:23.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T08:22:23.414 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T08:22:23.415 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210/remote/smithi069 2024-04-24T08:22:23.415 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T08:22:23.513 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210/remote/smithi121 2024-04-24T08:22:23.513 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T08:22:23.609 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T08:22:23.609 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T08:22:23.617 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T08:22:23.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T08:22:23.720 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T08:22:23.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T08:22:23.735 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T08:22:23.735 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T08:22:23.743 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T08:22:23.791 INFO:teuthology.orchestra.run.smithi069.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 08:22 /home/ubuntu/cephtest 2024-04-24T08:22:23.798 INFO:teuthology.orchestra.run.smithi121.stdout: 393949 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 08:22 /home/ubuntu/cephtest 2024-04-24T08:22:23.801 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T08:22:23.814 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T08:22:23.844 INFO:teuthology.nuke:Checking targets against current locks 2024-04-24T08:22:23.870 DEBUG:teuthology.nuke:shortname: smithi069 2024-04-24T08:22:23.870 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T08:22:23.891 DEBUG:teuthology.nuke:shortname: smithi121 2024-04-24T08:22:23.891 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T08:22:23.893 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:51:07.883968', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK20pgGhDUG25YTktz5xwXObPjKvwJGn16p16aQ/0z0+mcDD2fI1t/7FaYdn3Uc0KXME5axBn4v+GA4VH3lFBc8='} 2024-04-24T08:22:23.915 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671210', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:51:07.885359', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO0oFFrIDtsdmDzD5a+fGpNSauqw+ofxvqcq6IPtvi5gvN4205o5tQ8kZoJSeqobbF0W03bDbUz/qnqVr6oBns4='} 2024-04-24T08:22:23.944 INFO:teuthology.orchestra.console.smithi069:Power off 2024-04-24T08:22:23.944 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T08:22:23.970 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-04-24T08:22:23.970 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:22:23.986 INFO:teuthology.orchestra.console.smithi121:Power off 2024-04-24T08:22:23.986 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T08:22:23.995 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-24T08:22:24.007 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-04-24T08:22:24.007 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:22:24.027 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-04-24T08:22:27.996 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:22:28.029 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:22:28.107 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-04-24T08:22:28.138 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-04-24T08:22:32.110 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:22:32.139 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:22:32.219 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-04-24T08:22:32.249 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-04-24T08:22:32.321 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-04-24T08:22:32.350 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-04-24T08:22:32.455 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/forward-scrub} duration: 1356.8061773777008 failure_reason: '"2024-04-24T08:15:53.345601+0000 mds.a (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log' flavor: default mon.a-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 mon.b-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 owner: scheduled_rishabh@teuthology success: false 2024-04-24T08:22:32.456 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T08:22:32.530 INFO:teuthology.run:FAIL